Merge lp:~allanlesage/unity-scope-click/enable-coverage-reporting-refreshed into lp:unity-scope-click

[Allan LeSage]

Here we're adding some CMake tooling to aid in the generation of coverage reports. As described in the top-level CMakeLists.txt, after you've make and make test'ed, try make coverage and look for a coveragereport dir with html files in your build dir.

Note that you'll need a couple of dependencies, esp. lcov and gcovr --note that unity8 and mir have elected not to make these part of the debian/control deps. Also note that this method is borrowed wholesale from those projects and others, with some adaptation for your project.

We have some good tests to build on here :) , maybe the html view will help.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:209
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~allanlesage/unity-scope-click/enable-coverage-reporting/+merge/215319/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/unity-scope-click-ci/428/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scope-click-trusty-amd64-ci/329
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scope-click-trusty-armhf-ci/326
        deb: http://jenkins.qa.ubuntu.com/job/unity-scope-click-trusty-armhf-ci/326/artifact/work/output/*zip*/output.zip

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scope-click-ci/428/rebuild

review: Needs Fixing (continuous-integration)

[dobey (dobey)]

Please use "bzr commit unchanged --fixes=lp:1305180" so that bug #1305180 is linked to the branch.

[dobey (dobey)]

review: Needs Fixing

[Allan LeSage (allanlesage)]

Done; sorry for the delay :) .

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:210
http://jenkins.qa.ubuntu.com/job/unity-scope-click-ci/432/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scope-click-trusty-amd64-ci/333
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scope-click-trusty-armhf-ci/330
        deb: http://jenkins.qa.ubuntu.com/job/unity-scope-click-trusty-armhf-ci/330/artifact/work/output/*zip*/output.zip

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scope-click-ci/432/rebuild

review: Approve (continuous-integration)

[dobey (dobey)]

As unity-scope-click is under the CLA, and is distributed as GPLv3+OpenSSL exception, I wonder if we can include this, given the cmake module being copy/pasted here is GPLv2+ without such an exception.

Is there any reason we cannot package the bits being added under cmake/ as a separate project/package in Ubuntu, which all our projects depend on? It will make updating for any changes much easier, and will not introduce such copyright/license conflicts in the distribution of the source.

Also, where is the ParseArguments.cmake module from? It has no copyright information at all in it, so I wonder if we are even allowed to redistribute it? Is there a particular reason that the standard cmake argument parsing module can't be used?

I'm basically OK with the changes that enable the feature, but I think we need to ship the actual cmake module separately as its own package.

[Allan LeSage (allanlesage)]

I've long wanted to find the means to do this, but I'm wondering to where our CMake coverage bits would be shipped? I may need some instructio--meanwhile I'll try to collect some licensing information from Canonical committers for the files you've singled out.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:211
http://jenkins.qa.ubuntu.com/job/unity-scope-click-ci/436/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scope-click-trusty-amd64-ci/337
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scope-click-trusty-armhf-ci/334
        deb: http://jenkins.qa.ubuntu.com/job/unity-scope-click-trusty-armhf-ci/334/artifact/work/output/*zip*/output.zip

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scope-click-ci/436/rebuild

review: Approve (continuous-integration)

[Allan LeSage (allanlesage)]

Switched to the standard CMake argument parsing macro per dobey's suggestion.

Rodney, do lines 67-8 resolve your licensing concern?

Lastly can you provide some instruction on packaging these CMake files? I.e. does a standard archive exist already for such, and how are they invoked, etc.?

[dobey (dobey)]

> Rodney, do lines 67-8 resolve your licensing concern?

Not really. The license is still different from the rest of the code we ship. And we cannot add the OpenSSL exception clause to this license. With the other file, which had no copyright/license info at all, gone from the branch, the licensing is at least something we are legally allowed to redistribute (and change) now. It just means shipping things under different licenses, in the same source, and having to deal with managing any necessary changes to these files across several projects, versus a central source for it.

> Lastly can you provide some instruction on packaging these CMake files? I.e.
> does a standard archive exist already for such, and how are they invoked,
> etc.?

On Ubuntu at least, cmake-data and qtbase5-dev include the majority of cmake modules. The Qt ones are shipped by Qt, and not cmake. I'm not sure why these haven't been included in upstream cmake already, but getting them accepted there would be one way to get them packaged centrally. Another option would be inside the gcovr/lcov packages, if the licensing allows for it. And another option would be to simply place them in their own package, cmake-coverage or something, which only ships these files.

[dobey (dobey)]

17 +include(EnableCoverageReport)

Can you include a comment next to this, in the form of "# Using gcov cmake modules from https://wherever/they/came/from" so that we have a link to the true upstream of these cmake files?

Also, can you please update debian/copyright to add the necessary bits for the cmake/*.cmake files, which are not (C) Canonical, and which are GPLv2+ rather than GPLv3+OpenSSL?

review: Needs Fixing

[Allan LeSage (allanlesage)]

Resubmitted under a fresh branch based on devel per dobey's req, updating to the most recent upstream versions of these macros and registering the proper license in debian/copyright, with requested notes.

Note that our version of EnableCoverageReport.cmake differs from the upstream version in omitting the ParseArguments.cmake, which is unlicensed upstream--instead we're using a CMake-native macro (as in the original version of this MP).

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:238
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~allanlesage/unity-scope-click/enable-coverage-reporting-refreshed/+merge/218164/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/unity-scope-click-ci/460/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scope-click-trusty-amd64-ci/361/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scope-click-trusty-armhf-ci/358/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scope-click-ci/460/rebuild

review: Needs Fixing (continuous-integration)

[lp:~allanlesage/unity-scope-click/enable-coverage-reporting-refreshed]

239. By Allan LeSage on 2014-05-05

Correct Wienke copyright to 2010-2012, correct cor-lab.org URL.

240. By Allan LeSage on 2014-05-05

Coverage tooling origin comment http->https per dobey req.

Unmerged revisions

=== modified file 'CMakeLists.txt'
--- CMakeLists.txt	2014-04-30 16:00:10 +0000
+++ CMakeLists.txt	2014-05-02 22:12:55 +0000
@@ -10,6 +10,8 @@
 SET (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -O2 -g -Wall -Wextra -Werror -fPIC")
 SET (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11 -O2 -g -Wextra -Wall -Werror -fPIC")
 
+set(CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} ${CMAKE_CURRENT_SOURCE_DIR}/cmake)
+
 include(GNUInstallDirs)
 
 set(SCOPE_LIB_DIR ${CMAKE_INSTALL_FULL_LIBDIR}/unity-scopes/clickscope/)
@@ -38,6 +40,24 @@
 add_subdirectory(data)
 add_subdirectory(po)
 
+include(EnableCoverageReport) # Using gcov CMake modules from http://code.cor-lab/projects/rsc
+#####################################################################
+# Enable code coverage calculation with gcov/gcovr/lcov
+# Usage:
+#  * Switch build type to coverage (use ccmake or cmake-gui)
+#  * Invoke make, make test, make coverage (or ninja if you use that backend)
+#  * Find html report in subdir coveragereport
+#  * Find xml report feasible for jenkins in coverage.xml
+#####################################################################
+string(TOLOWER "${CMAKE_BUILD_TYPE}" cmake_build_type_lower)
+if(cmake_build_type_lower MATCHES coverage)
+  set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} --coverage" )
+  set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} --coverage" )
+  set(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS} --coverage" )
+  set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} --coverage" )
+  ENABLE_COVERAGE_REPORT(TARGETS ${SCOPE_LIB_UNVERSIONED} FILTER /usr/include ${CMAKE_SOURCE_DIR}/tests/* ${CMAKE_BINARY_DIR}/*)
+endif()
+
 # Custom targets for the tests
 add_custom_target (test
   DEPENDS test-click-scope

=== added directory 'cmake'
=== added file 'cmake/EnableCoverageReport.cmake'
--- cmake/EnableCoverageReport.cmake	1970-01-01 00:00:00 +0000
+++ cmake/EnableCoverageReport.cmake	2014-05-02 22:12:55 +0000
@@ -0,0 +1,158 @@
+# - Creates a special coverage build type and target on GCC.
+#
+# Defines a function ENABLE_COVERAGE_REPORT which generates the coverage target
+# for selected targets. Optional arguments to this function are used to filter
+# unwanted results using globbing expressions. Moreover targets with tests for
+# the source code can be specified to trigger regenerating the report if the
+# test has changed
+#
+# ENABLE_COVERAGE_REPORT(TARGETS target... [FILTER filter...] [TESTS test targets...])
+#
+# To generate a coverage report first build the project with
+# CMAKE_BUILD_TYPE=coverage, then call make test and afterwards make coverage.
+#
+# The coverage report is based on gcov. Depending on the availability of lcov
+# a HTML report will be generated and/or an XML report of gcovr is found.
+# The generated coverage target executes all found solutions. Special targets
+# exist to create e.g. only the xml report: coverage-xml. 
+#
+# Copyright (C) 2010 by Johannes Wienke <jwienke at techfak dot uni-bielefeld dot de>
+#
+# This file may be licensed under the terms of the
+# GNU Lesser General Public License Version 3 (the ``LGPL''),
+# or (at your option) any later version.
+#
+# Software distributed under the License is distributed
+# on an ``AS IS'' basis, WITHOUT WARRANTY OF ANY KIND, either
+# express or implied. See the LGPL for the specific language
+# governing rights and limitations.
+#
+# You should have received a copy of the LGPL along with this
+# program. If not, go to http://www.gnu.org/licenses/lgpl.html
+# or write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# The development of this software was supported by:
+#   CoR-Lab, Research Institute for Cognition and Robotics
+#     Bielefeld University
+
+FIND_PACKAGE(Lcov)
+FIND_PACKAGE(gcovr)
+
+FUNCTION(ENABLE_COVERAGE_REPORT)
+    
+    # argument parsing
+    SET(MULTI_VALUE_ARGS FILTER TARGETS TESTS)
+    CMAKE_PARSE_ARGUMENTS(ENABLE_COVERAGE_REPORT "" "" "${MULTI_VALUE_ARGS}" ${ARGN})
+
+    SET(COVERAGE_RAW_FILE "${CMAKE_BINARY_DIR}/coverage.raw.info")
+    SET(COVERAGE_FILTERED_FILE "${CMAKE_BINARY_DIR}/coverage.info")
+    SET(COVERAGE_REPORT_DIR "${CMAKE_BINARY_DIR}/coveragereport")
+    SET(COVERAGE_XML_FILE "${CMAKE_BINARY_DIR}/coverage.xml")
+    SET(COVERAGE_XML_COMMAND_FILE "${CMAKE_BINARY_DIR}/coverage-xml.cmake")
+    
+    # decide if there is any tool to create coverage data
+    SET(TOOL_FOUND FALSE)
+    IF(LCOV_FOUND OR GCOVR_FOUND)
+        SET(TOOL_FOUND TRUE)
+    ENDIF()
+    IF(NOT TOOL_FOUND)
+        MESSAGE(STATUS "Cannot enable coverage targets because neither lcov nor gcovr are found.")
+    ENDIF()
+    
+    STRING(TOLOWER "${CMAKE_BUILD_TYPE}" COVERAGE_BUILD_TYPE)
+    IF(CMAKE_COMPILER_IS_GNUCXX AND TOOL_FOUND AND "${COVERAGE_BUILD_TYPE}" MATCHES "coverage")
+    
+        MESSAGE(STATUS "Coverage support enabled for targets: ${ENABLE_COVERAGE_REPORT_TARGETS}")
+    
+        # create coverage build type
+        SET(CMAKE_CXX_FLAGS_COVERAGE ${CMAKE_CXX_FLAGS_DEBUG} PARENT_SCOPE)
+        SET(CMAKE_C_FLAGS_COVERAGE ${CMAKE_C_FLAGS_DEBUG} PARENT_SCOPE)
+        SET(CMAKE_CONFIGURATION_TYPES ${CMAKE_CONFIGURATION_TYPES} coverage PARENT_SCOPE)
+    
+        # instrument targets
+        SET_TARGET_PROPERTIES(${ENABLE_COVERAGE_REPORT_TARGETS} PROPERTIES COMPILE_FLAGS --coverage
+                                                        LINK_FLAGS --coverage)
+    
+        # html report
+        IF (LCOV_FOUND)
+        
+            MESSAGE(STATUS "Enabling HTML coverage report")
+    
+            # set up coverage target
+            
+            ADD_CUSTOM_COMMAND(OUTPUT ${COVERAGE_RAW_FILE}
+                               COMMAND ${LCOV_EXECUTABLE} -c -d ${CMAKE_BINARY_DIR} -o ${COVERAGE_RAW_FILE}
+                               WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
+                               COMMENT "Collecting coverage data"
+                               DEPENDS ${ENABLE_COVERAGE_REPORT_TARGETS} ${ENABLE_COVERAGE_REPORT_TESTS}
+                               VERBATIM)
+            
+            # filter unwanted stuff
+            LIST(LENGTH ENABLE_COVERAGE_REPORT_FILTER FILTER_LENGTH)
+            IF(${FILTER_LENGTH} GREATER 0)
+                SET(FILTER COMMAND ${LCOV_EXECUTABLE})
+                FOREACH(F ${ENABLE_COVERAGE_REPORT_FILTER})
+                    SET(FILTER ${FILTER} -r ${COVERAGE_FILTERED_FILE} ${F})
+                ENDFOREACH()
+                SET(FILTER ${FILTER} -o ${COVERAGE_FILTERED_FILE})
+            ELSE()
+                SET(FILTER "")
+            ENDIF()
+            
+            ADD_CUSTOM_COMMAND(OUTPUT ${COVERAGE_FILTERED_FILE}
+                               COMMAND ${LCOV_EXECUTABLE} -e ${COVERAGE_RAW_FILE} "${CMAKE_SOURCE_DIR}*"  -o ${COVERAGE_FILTERED_FILE}
+                               ${FILTER}
+                               DEPENDS ${COVERAGE_RAW_FILE}
+                               COMMENT "Filtering recorded coverage data for project-relevant entries"
+                               VERBATIM)
+            ADD_CUSTOM_COMMAND(OUTPUT ${COVERAGE_REPORT_DIR}
+                               COMMAND ${CMAKE_COMMAND} -E make_directory ${COVERAGE_REPORT_DIR}
+                               COMMAND ${GENHTML_EXECUTABLE} --legend --show-details -t "${PROJECT_NAME} test coverage" -o ${COVERAGE_REPORT_DIR} ${COVERAGE_FILTERED_FILE}
+                               DEPENDS ${COVERAGE_FILTERED_FILE}
+                               COMMENT "Generating HTML coverage report in ${COVERAGE_REPORT_DIR}"
+                               VERBATIM)
+                               
+            ADD_CUSTOM_TARGET(coverage-html
+                              DEPENDS ${COVERAGE_REPORT_DIR})
+                              
+        ENDIF()
+        
+        # xml coverage report
+        IF(GCOVR_FOUND)
+        
+            MESSAGE(STATUS "Enabling XML coverage report")
+        
+            # gcovr cannot write directly to a file so the execution needs to
+            # be wrapped in a cmake file that generates the file output
+            FILE(WRITE ${COVERAGE_XML_COMMAND_FILE}
+                 "SET(ENV{LANG} en)\n")
+            FILE(APPEND ${COVERAGE_XML_COMMAND_FILE}
+                 "EXECUTE_PROCESS(COMMAND \"${GCOVR_EXECUTABLE}\" -x -r \"${CMAKE_SOURCE_DIR}\" OUTPUT_FILE \"${COVERAGE_XML_FILE}\" WORKING_DIRECTORY \"${CMAKE_BINARY_DIR}\")\n")
+        
+            ADD_CUSTOM_COMMAND(OUTPUT ${COVERAGE_XML_FILE}
+                               COMMAND ${CMAKE_COMMAND} ARGS -P ${COVERAGE_XML_COMMAND_FILE}
+                               COMMENT "Generating coverage XML report"
+                               VERBATIM)
+                               
+            ADD_CUSTOM_TARGET(coverage-xml
+                              DEPENDS ${COVERAGE_XML_FILE})
+        
+        ENDIF()
+        
+        # provide a global coverage target executing both steps if available
+        SET(GLOBAL_DEPENDS "")
+        IF(LCOV_FOUND)
+            LIST(APPEND GLOBAL_DEPENDS ${COVERAGE_REPORT_DIR})
+        ENDIF()
+        IF(GCOVR_FOUND)
+            LIST(APPEND GLOBAL_DEPENDS ${COVERAGE_XML_FILE})
+        ENDIF()
+        IF(LCOV_FOUND OR GCOVR_FOUND)
+            ADD_CUSTOM_TARGET(coverage
+                              DEPENDS ${GLOBAL_DEPENDS})
+        ENDIF()
+                          
+    ENDIF()
+
+ENDFUNCTION()   

=== added file 'cmake/FindLcov.cmake'
--- cmake/FindLcov.cmake	1970-01-01 00:00:00 +0000
+++ cmake/FindLcov.cmake	2014-05-02 22:12:55 +0000
@@ -0,0 +1,35 @@
+# - Find lcov 
+# Will define:
+#
+# LCOV_EXECUTABLE - the lcov binary
+# GENHTML_EXECUTABLE - the genhtml executable
+#
+# Copyright (C) 2010 by Johannes Wienke <jwienke at techfak dot uni-bielefeld dot de>
+#
+# This file may be licensed under the terms of the
+# GNU Lesser General Public License Version 3 (the ``LGPL''),
+# or (at your option) any later version.
+#
+# Software distributed under the License is distributed
+# on an ``AS IS'' basis, WITHOUT WARRANTY OF ANY KIND, either
+# express or implied. See the LGPL for the specific language
+# governing rights and limitations.
+#
+# You should have received a copy of the LGPL along with this
+# program. If not, go to http://www.gnu.org/licenses/lgpl.html
+# or write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# The development of this software was supported by:
+#   CoR-Lab, Research Institute for Cognition and Robotics
+#     Bielefeld University
+
+INCLUDE(FindPackageHandleStandardArgs)
+
+FIND_PROGRAM(LCOV_EXECUTABLE lcov)
+FIND_PROGRAM(GENHTML_EXECUTABLE genhtml)
+
+FIND_PACKAGE_HANDLE_STANDARD_ARGS(Lcov DEFAULT_MSG LCOV_EXECUTABLE GENHTML_EXECUTABLE)
+
+# only visible in advanced view
+MARK_AS_ADVANCED(LCOV_EXECUTABLE GENHTML_EXECUTABLE)

=== added file 'cmake/Findgcovr.cmake'
--- cmake/Findgcovr.cmake	1970-01-01 00:00:00 +0000
+++ cmake/Findgcovr.cmake	2014-05-02 22:12:55 +0000
@@ -0,0 +1,43 @@
+# - Find gcovr scrip 
+# Will define:
+#
+# GCOVR_EXECUTABLE - the gcovr script
+#
+# Uses:
+#
+# GCOVR_ROOT - root to search for the script
+#
+# Copyright (C) 2011 by Johannes Wienke <jwienke at techfak dot uni-bielefeld dot de>
+#
+# This file may be licensed under the terms of the
+# GNU Lesser General Public License Version 3 (the ``LGPL''),
+# or (at your option) any later version.
+#
+# Software distributed under the License is distributed
+# on an ``AS IS'' basis, WITHOUT WARRANTY OF ANY KIND, either
+# express or implied. See the LGPL for the specific language
+# governing rights and limitations.
+#
+# You should have received a copy of the LGPL along with this
+# program. If not, go to http://www.gnu.org/licenses/lgpl.html
+# or write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+#
+# The development of this software was supported by:
+#   CoR-Lab, Research Institute for Cognition and Robotics
+#     Bielefeld University
+
+INCLUDE(FindPackageHandleStandardArgs)
+
+FIND_PROGRAM(GCOVR_EXECUTABLE gcovr
+             HINTS ${GCOVR_ROOT}
+                   "${GCOVR_ROOT}/bin"
+                   "${GCOVR_ROOT}/share/python"
+                   "/usr/share/python"
+                   "/usr/local/share/python"
+                   "${CMAKE_INSTALL_PREFIX}/share/python")
+
+FIND_PACKAGE_HANDLE_STANDARD_ARGS(gcovr DEFAULT_MSG GCOVR_EXECUTABLE)
+
+# only visible in advanced view
+MARK_AS_ADVANCED(GCOVR_EXECUTABLE)

=== modified file 'debian/copyright'
--- debian/copyright	2014-03-24 19:18:13 +0000
+++ debian/copyright	2014-05-02 22:12:55 +0000
@@ -18,3 +18,18 @@
  .
  On Debian/Ubuntu systems, the full text of the GPL v3 can be found in
  `/usr/share/common-licenses/GPL-3'
+
+Files: cmake/EnableCoverageReport.cmake cmake/Findgcovr.cmake cmake/FindLcov.cmake
+Copyright: 2010 Johannes Wienke
+License: LGPL-3
+ This file may be licensed under the terms of the
+ GNU Lesser General Public License Version 3 (the ``LGPL''),
+ or (at your option) any later version.
+ .
+ Software distributed under the License is distributed
+ on an ``AS IS'' basis, WITHOUT WARRANTY OF ANY KIND, either
+ express or implied. See the LGPL for the specific language
+ governing rights and limitations.
+ .
+ On Debian/Ubuntu systems, the full text of the LGPL v3 can be found in
+ `/usr/share/common-licenses/LGPL-3'