Merge lp:~mrazik/unity/coverage-support into lp:unity

[Martin Mrazik]

Adding -DCMAKE_BUILD_TYPE=coverage support as well as a custom coverage-xml target which is generated if gcovr is found on the system.

= Problem description =

This adds code coverage support to the build system by adding -DCMAKE_BUILD_TYPE=coverage and a custom coverage-xml target in case gcovr is installed.

= The fix =

CMakeLists.txt only change.

= Test coverage =

Build with -DCMAKE_BUILD_TYPE=coverage
https://jenkins.qa.ubuntu.com/job/unity-coverage-test/1/cobertura/?

Build log without -DCMAKE_BUILD_TYPE=coverage
https://jenkins.qa.ubuntu.com/job/unity-coverage-test/2/console

[Jussi Pakkanen (jpakkane)]

The if clause on line 18 is incorrect and always evaluates to true. It should be written like this:

if (NOT GCOVR_EXECUTABLE)

Also, trying to run coverity tests without gcovr is an error. It does not provide any useful data, and only wastes time. The build system should stop with an error. This is simple, just change line 19 to this:

message(FATAL_ERROR "Cannot enable coverage targets because gcovr was not found.")

review: Needs Fixing

[Jussi Pakkanen (jpakkane)]

Also, the comment says "gcovr cannot write directly to a file", but 'gcovr -h' prints the following:

-o OUTPUT, --output=OUTPUT Print output to this filename

Does this not work? Does it not do the correct thing? Or is there something else afoot?

[Martin Mrazik (mrazik)]

> Also, the comment says "gcovr cannot write directly to a file", but 'gcovr -h'
> prints the following:
>
> -o OUTPUT, --output=OUTPUT Print output to this filename
>
> Does this not work? Does it not do the correct thing? Or is there something
> else afoot?

Thomas, can you answer this? I've copy&pasted this part from your project.

[Martin Mrazik (mrazik)]

> The if clause on line 18 is incorrect and always evaluates to true. It should
> be written like this:
>
> if (NOT GCOVR_EXECUTABLE)
>
> Also, trying to run coverity tests without gcovr is an error. It does not
> provide any useful data, and only wastes time. The build system should stop
> with an error. This is simple, just change line 19 to this:
>
> message(FATAL_ERROR "Cannot enable coverage targets because gcovr was not
> found.")

Thanks. These are fixed.

[Jussi Pakkanen (jpakkane)]

I can't run make check because it errors out with "No rule to make target `../tests/test-gtest', needed by `tests/CMakeFiles/check'. Stop." However if I just run "make coverage-xml", I get an XML file with no real content in it.

It would be nice if this was done automatically (by e.g. having a dependency between coverity-xml and tests) but I assume that Jenkins will automatically run these in the correct order so adding the dependency would run the tests twice.

This is not a blocker in any case, just would-be-nice-to-have speculation.

[Martin Mrazik (mrazik)]

Setting to WIP as I forgot about the MP and I'm using this branch in the jenkins job for a test. I'll switch back to Needs Review once I'm finished with the testing.

[Martin Mrazik (mrazik)]

> Also, the comment says "gcovr cannot write directly to a file", but 'gcovr -h'
> prints the following:
>
> -o OUTPUT, --output=OUTPUT Print output to this filename
>
> Does this not work? Does it not do the correct thing? Or is there something
> else afoot?

Fixed and here is a sample run with it:
https://jenkins.qa.ubuntu.com/job/unity-coverage-test/4/cobertura/?

[Martin Mrazik (mrazik)]

> I can't run make check because it errors out with "No rule to make target
> `../tests/test-gtest', needed by `tests/CMakeFiles/check'. Stop." However if
> I just run "make coverage-xml", I get an XML file with no real content in it.
>
> It would be nice if this was done automatically (by e.g. having a dependency
> between coverity-xml and tests) but I assume that Jenkins will automatically
> run these in the correct order so adding the dependency would run the tests
> twice.
>
> This is not a blocker in any case, just would-be-nice-to-have speculation.

I would prefer to leave this is up to you guys. TBH a coverage-html target should be also created (which would be using lcov) so you can check the coverage locally as well. This is a minimal support for coverage for jenkins. Not much more than that.

[Jussi Pakkanen (jpakkane)]

You really should not do 'cd "${CMAKE_BINARY_DIR}"', because that should already be the working directory and cd'ing around is strongly discouraged. Remove that (works fine without on my machine with a quick test). If you must set the working directory, use the WORKING_DIRECTORY option of add_custom_command.

Once this is fixed, this is ready to merge.

review: Needs Fixing

[Martin Mrazik (mrazik)]

Done.

[Jussi Pakkanen (jpakkane)]

Nice.

review: Approve

=== modified file 'CMakeLists.txt'
--- CMakeLists.txt	2012-11-22 11:22:38 +0000
+++ CMakeLists.txt	2012-11-29 12:56:21 +0000
@@ -46,6 +46,25 @@
   set (UNITY_STANDALONE_LADD "-lunity-core-${UNITY_API_VERSION} -lm -lpthread -ldl -lGL -lGLU")
 endif ()
 
+if (CMAKE_BUILD_TYPE MATCHES coverage)
+  set (COVERAGE_XML_FILE "${CMAKE_BINARY_DIR}/coverage.xml")
+
+  set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} --coverage" )
+  set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} --coverage" )
+  set (CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS} --coverage" )
+  set (CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} --coverage" )
+
+  find_program(GCOVR_EXECUTABLE gcovr HINTS ${GCOVR_ROOT} "${GCOVR_ROOT}/bin")
+  if (NOT GCOVR_EXECUTABLE)
+      message(FATAL_ERROR "Cannot enable coverage targets because gcovr was not found.")
+  else ()
+      message (STATUS "Enabling XML coverage report")
+      add_custom_target (coverage-xml 
+          COMMAND  "${GCOVR_EXECUTABLE}" --exclude="tests.*" --exclude="obj-.*" -x -r "${CMAKE_SOURCE_DIR}" -o "${COVERAGE_XML_FILE}")
+  endif()
+endif (CMAKE_BUILD_TYPE MATCHES coverage)
+
+
 #
 # Niceties
 #