Merge lp:~pitti/autopilot-gtk/testsuite into lp:autopilot-gtk

[Martin Pitt]

Add make test/ctest integration, and run tests during package build

This branch makes it easier to run the tests.

There is no immediately obvious tests/run, "make check" or other command, and
it takes a nontrivial amount of time to figure out how to set $GTK_PATH and
which of the various test discovery alternatives to run for this.

With this branch, the standard cmake "make test" or "ctest" commands will work,
including with separate build trees.

[Martin Pitt (pitti)]

I tested this both in-tree:

  cmake . && make && ctest

and with a separate build tree:

  rm -rf build; mkdir build; cd build; cmake .. && make -j4 && ctest

both starting from a pristine/clean source tree.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:44
http://jenkins.qa.ubuntu.com/job/autopilot-gtk-ci/43/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-gtk-saucy-amd64-ci/5/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-gtk-saucy-armhf-ci/5/console

Click here to trigger a rebuild:
http://s-jenkins:8080/job/autopilot-gtk-ci/43/rebuild

review: Needs Fixing (continuous-integration)

[lp:~pitti/autopilot-gtk/testsuite]

45. By Martin Pitt on 2013-06-20

Run testsuite during package build

The test needs to run with a session bus and an X server, and uses gedit.
Create the necessary test environment and add corresponding build dependencies.

[Martin Pitt (pitti)]

Indeed, wit the previous commit tests run during package build automatically. That needs a couple of extra build dependencies and some tweaking of the test environment. Package build succeeds now under sbuild with r45 for me.

review: Needs Resubmitting

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:45
http://jenkins.qa.ubuntu.com/job/autopilot-gtk-ci/44/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-gtk-saucy-amd64-ci/6
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-gtk-saucy-armhf-ci/6

Click here to trigger a rebuild:
http://s-jenkins:8080/job/autopilot-gtk-ci/44/rebuild

review: Approve (continuous-integration)

[Martin Pitt (pitti)]

While developing some actual tests in lp:~pitti/autopilot-gtk/add-tests I noticed that running under xvfb-run takes very long, and also has some quirks (like, the "has_focus" property is false for a GtkEntry where it ought to be true). I think we should take the "make test"/"ctest" integration, but I'd appreciate an opinion whether you want the tests to run during package build (in debian/rules), or whether there is a more elegant way to let them run for merge proposals (i. e. CI testing) which runs them in an actual X session?

[Martin Pitt (pitti)]

https://code.launchpad.net/~pitti/autopilot-gtk/add-tests/+merge/171036 is the MP for landing some actual tests. It depends on this one.

[Francis Ginther (fginther)]

> While developing some actual tests in lp:~pitti/autopilot-gtk/add-tests I
> noticed that running under xvfb-run takes very long, and also has some quirks
> (like, the "has_focus" property is false for a GtkEntry where it ought to be
> true). I think we should take the "make test"/"ctest" integration, but I'd
> appreciate an opinion whether you want the tests to run during package build
> (in debian/rules), or whether there is a more elegant way to let them run for
> merge proposals (i. e. CI testing) which runs them in an actual X session?

Running these tests under a real X environment is a next step. There is actually a blueprint item for this month to make this possible for lp:autopilot which can then be extended here. However, until that is ready, I'm ok with adding these tests and running them with xvfb in order to get some testing enabled. Eventually we'll need to split out the autopilot tests and create a separate set of unittests that will be run during package build (and not require X).

The only thing I would address in this MP is the test runner used. I believe we should be using autopilot, but I can't figure out how to make it work. I'm currently trying this:

add_test(autopilot-gtk-functional autopilot run -f xml -o test-autopilot-gtk-functional.xml -v tests)

but I don't have the source directory right and it can't find the tests.

review: Needs Fixing

[Thomi Richards (thomir-deactivatedaccount)]

Hi,

This looks good, but, so far this is the only project to run autopilot tests as part of the package build using xvfb. I'm happy to trial this approach for this project, and we can see how it works and whether it's worth introducing elsewhere.

review: Approve

[Martin Pitt (pitti)]

> The only thing I would address in this MP is the test runner used. I believe we should be using autopilot,

I actually tried that first as it seems more natural. It doesn't seem to have a discovery mode, and if I naively try

$ autopilot run tests/autopilot/tests
ImportError: Import by filename is not supported.

$ autopilot run tests.autopilot.tests
ImportError: No module named tests

Apparently one would need some __init__.py there? NB that I didn't invent this rather deep tests/autopilot/tests/ hierarchy, that was already there. (I'd just put everything into tests/ directly, ap-gtk isn't going to have a terribly huge test suite).

What we could do is

$ cd tests/autopilot; GTK_PATH=../../lib/ autopilot run tests

That works, but would look a bit ugly in CMakeList (as that's just a program to run, not a shell command). I used what autopilot itself uses in debian/rules, but if you prefer calling autopilot run, I'm happy to move all files to tests/ directoy, and just call "autopilot run tests" instead?

[lp:~pitti/autopilot-gtk/testsuite]

46. By Martin Pitt on 2013-06-25

use "autopilot run" instead of testtools discover

[Martin Pitt (pitti)]

I updated the test runner invocation to use "autopilot run".

review: Needs Resubmitting

[Martin Pitt (pitti)]

> Eventually we'll need to split out the autopilot tests and create a separate set of unittests that will be run during package build (and not require X).

For the record, this likely isn't going to happen. The lowest sensible layer at which we can test is the D-BUS interface, but this already requires that some GTK app is running, which requires a $DISPLAY. The autopilot API bits on top of that don't need $DISPLAY, so we wouldn't gain anything by testing teh D-BUS layer instead of the autopilot API IMHO.

[Francis Ginther (fginther)]

It works!

review: Approve

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:46
http://jenkins.qa.ubuntu.com/job/autopilot-gtk-ci/51/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-gtk-saucy-amd64-ci/13
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-gtk-saucy-armhf-ci/13

Click here to trigger a rebuild:
http://s-jenkins:8080/job/autopilot-gtk-ci/51/rebuild

review: Approve (continuous-integration)

=== modified file 'CMakeLists.txt'
--- CMakeLists.txt	2012-11-28 16:57:09 +0000
+++ CMakeLists.txt	2013-06-25 14:44:27 +0000
@@ -9,3 +9,6 @@
 set (CMAKE_CXX_FLAGS "-DGNOME_DESKTOP_USE_UNSTABLE_API -std=c++11 -fno-permissive")
 INCLUDE(FindPkgConfig)
 add_subdirectory(lib)
+
+enable_testing()
+add_test(nose sh -c "cd ${CMAKE_CURRENT_SOURCE_DIR}/tests/autopilot; GTK_PATH=${CMAKE_CURRENT_BINARY_DIR}/lib autopilot run tests")

=== modified file 'debian/control'
--- debian/control	2013-04-18 05:15:53 +0000
+++ debian/control	2013-06-25 14:44:27 +0000
@@ -8,6 +8,10 @@
                libglib2.0-dev,
                libgtk-3-dev,
                libxpathselect-dev (>= 1.3),
+               xvfb,
+               dbus-x11,
+               python-autopilot,
+               gedit
 Standards-Version: 3.9.4
 Section: libs
 Homepage: https://launchpad.net/autopilot-gtk

=== modified file 'debian/rules'
--- debian/rules	2013-01-14 23:42:40 +0000
+++ debian/rules	2013-06-25 14:44:27 +0000
@@ -6,8 +6,13 @@
 %:
 	dh $@ 
 
+override_dh_auto_test:
+	mkdir -p debian/tmp/home
+	env HOME=debian/tmp/home xvfb-run dbus-launch dh_auto_test
+
 override_dh_install:
 	dh_install --fail-missing
 
 override_dh_makeshlibs:
 	dh_makeshlibs -V
+

=== modified file 'lib/CMakeLists.txt'
--- lib/CMakeLists.txt	2013-01-25 02:43:03 +0000
+++ lib/CMakeLists.txt	2013-06-25 14:44:27 +0000
@@ -31,5 +31,7 @@
   ${GTK_LIBRARIES}
   ${XPATHSELECT_LIBRARIES})
 
+execute_process(COMMAND ln -s . lib/modules)
+
 install(TARGETS autopilot
   LIBRARY DESTINATION ${CMAKE_INSTALL_LIBDIR}/gtk-3.0/modules)