Merge lp:~pitti/autopilot-gtk/add-tests into lp:autopilot-gtk
Status: | Merged | ||||
---|---|---|---|---|---|
Approved by: | Francis Ginther on 2013-06-26 | ||||
Approved revision: | 70 | ||||
Merged at revision: | 46 | ||||
Proposed branch: | lp:~pitti/autopilot-gtk/add-tests | ||||
Merge into: | lp:autopilot-gtk | ||||
Diff against target: |
1081 lines (+946/-61) 12 files modified
CMakeLists.txt (+1/-1) debian/control (+6/-1) debian/rules (+3/-2) tests/autopilot/tests/test_actions.py (+184/-0) tests/autopilot/tests/test_gnome_system_log.py (+61/-0) tests/autopilot/tests/test_matching_properties.py (+0/-32) tests/autopilot/tests/test_properties.py (+99/-0) tests/autopilot/tests/test_widget_tree.py (+186/-0) tests/autopilot/tests/test_xpath_query.py (+110/-0) tests/hello_color.py (+53/-0) tests/hello_color.ui (+243/-0) tests/test-matching.sh (+0/-25) |
||||
To merge this branch: | bzr merge lp:~pitti/autopilot-gtk/add-tests | ||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Francis Ginther | Approve on 2013-06-26 | ||
PS Jenkins bot | continuous-integration | Approve on 2013-06-26 | |
Martin Pitt (community) | Resubmit on 2013-06-25 | ||
Thomi Richards (community) | 2013-06-24 | Needs Fixing on 2013-06-24 | |
Autopilot Hackers | fixed license and test robustness | 2013-06-26 | Pending |
Review via email:
|
Commit message
Add integration test suite (LP: #1083612)
Description of the change
Add some integration tests, using a tiny tests/hello_
There is also a test case that exercises gnome-system-log, which is a lot
lighter than gedit. All these supersede the old test_matching_
Please see the individual commits for details.
Martin Pitt (pitti) wrote : | # |
PS Jenkins bot (ps-jenkins) wrote : | # |
PASSED: Continuous integration, rev:55
http://
Executed test runs:
SUCCESS: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
Hi,
It looks to me like your copyright header is malformed - you should have two paragraphs.
Other than that, looks good to me.
- 56. By Martin Pitt on 2013-06-25
-
add complete GPL copyright headers
PS Jenkins bot (ps-jenkins) wrote : | # |
PASSED: Continuous integration, rev:56
http://
Executed test runs:
SUCCESS: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
- 57. By Martin Pitt on 2013-06-25
-
Add test case for GtkMenuItem globalRect
This verifies https:/
/bugs.launchpad .net/autopilot- gtk/+bug/ 1133893
PS Jenkins bot (ps-jenkins) wrote : | # |
PASSED: Continuous integration, rev:57
http://
Executed test runs:
SUCCESS: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
- 58. By Martin Pitt on 2013-06-25
-
tests/hello_
color.py: Implement about dialog
PS Jenkins bot (ps-jenkins) wrote : | # |
FAILED: Continuous integration, rev:58
http://
Executed test runs:
FAILURE: http://
FAILURE: http://
Click here to trigger a rebuild:
http://
- 59. By Martin Pitt on 2013-06-25
-
adjust tests after previous commit
PS Jenkins bot (ps-jenkins) wrote : | # |
PASSED: Continuous integration, rev:59
http://
Executed test runs:
SUCCESS: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
- 60. By Martin Pitt on 2013-06-25
-
Add test case for XPath queries
This also reproduces https:/
/launchpad. net/bugs/ 1179806
PS Jenkins bot (ps-jenkins) wrote : | # |
PASSED: Continuous integration, rev:60
http://
Executed test runs:
SUCCESS: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
- 61. By Martin Pitt on 2013-06-25
-
refine test to work with the autopilot runner, too
- 62. By Martin Pitt on 2013-06-25
-
drop tests/test-
matching. sh; broken, and superseded by test_xpath_query.py - 63. By Martin Pitt on 2013-06-25
-
debian/control: Add trailing comma to build dependencies to avoid future patch noise
PS Jenkins bot (ps-jenkins) wrote : | # |
FAILED: Continuous integration, rev:63
http://
Executed test runs:
SUCCESS: http://
FAILURE: http://
Click here to trigger a rebuild:
http://
Francis Ginther (fginther) wrote : | # |
I've started another CI build to get another sample to see if the same error reproduces. The failure could be caused by a timing difference.
PS Jenkins bot (ps-jenkins) wrote : | # |
FAILED: Continuous integration, rev:63
http://
Executed test runs:
FAILURE: http://
FAILURE: http://
Click here to trigger a rebuild:
http://
Francis Ginther (fginther) wrote : | # |
More failures this time. I'm concerned that this is a product of running with xvfb. We may have to stop running the tests during package build.
I am working on a solution to running these types of tests during CI, but outside of the build itself. No ETA yet as I've only started.
Martin Pitt (pitti) wrote : | # |
I can reproduce this failure on my machine when I load it heavily (I have a VM with an autopkgtest running in the background). Looks like I need an Eventually matcher there, the test currently assumes that the GtkEntry has the typed text right after self.app.type(...).
BTW, I ran the tests with verbosity, and it seems it's that achingly slow not because of xvfb, but because of some timeouts due to the local D-BUS:
1: 06:44:22.605 INFO globals:49 - Starting test tests.test_
1: 06:44:23.077 INFO __init__:136 - Launching process: ['/home/
1: 06:44:23.082 INFO __init__:169 - Looking for autopilot interface for PID 11949 (and children)
1: 06:44:23.103 WARNING __init__:186 - Caught exception while searching for autopilot interface: 'DBusException(
1: 06:44:24.131 WARNING __init__:186 - Caught exception while searching for autopilot interface: 'DBusException(
1: 06:44:49.157 ERROR proxies:410 - Introspect error on :1.7:/com/
1: 06:44:49.157 DEBUG proxies:413 - Executing introspect queue due to error
1: 06:45:14.180 WARNING __init__:186 - Caught exception while searching for autopilot interface: 'DBusException('Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.',)'
1: 06:45:14.192 DEBUG dbus:354 - Selecting objects of type GtkEntry with attributes: {}
So it spends almost a minute(!) on each test case on these. I'll try to track that down, too.
- 64. By Martin Pitt on 2013-06-26
-
merge with trunk
- 65. By Martin Pitt on 2013-06-26
-
Run tests with verbosity, to see more detailled output in build logs on failures
- 66. By Martin Pitt on 2013-06-26
-
Fix d-bus startup during package build
Drop our own dbus-launch invocation in debian/rules; that instance fights with
at-spi's own D-BUS launcher, causing long timeouts in autopilot. Also, that
instance never got cleaned up.This drops the time to run the tests from 10.5 minutes to 47 seconds due to
avoiding the D-BUS timeouts. - 67. By Martin Pitt on 2013-06-26
-
Fix test_actions.
test_menu In xvfb, the menu might actually start at (0,0), cover this (literal) corner
case. - 68. By Martin Pitt on 2013-06-26
-
Work around default focus problem in Xvfb
When running under Xvfb, there is sometimes no default focus. If the entry_name
doesn't have focus, click it to work around this. - 69. By Martin Pitt on 2013-06-26
-
Use Eventually matchers for robustness
Martin Pitt (pitti) wrote : | # |
I made some progress:
* r66 fixes the abysmally long test run time due to the aforementioned D-BUS timeouts. It now runs in ~ 45 seconds instead of 10.5 minutes.
* r68 works around the test program in xvfb sometimes not having the default focus. That's what caused the above "Joe" != "" failures. Debugging this properly is a bit tricky, I wanted to find out which widget has the default focus (if at all), but I found that select_many() doesn't work for int/bool properties. I'm investigating this now, will file a bug, and add a test case. I marked these as "FIXME" for now, but the workaround is IMHO not too bad. Also, it will become more general and easier to spot after the fix for bug 1082391 lands.
* The other are various smaller improvements, see their changelogs.
Now, let's see how the CI run likes these :-) I ran this in sbuild about six times and it always passed.
PS Jenkins bot (ps-jenkins) wrote : | # |
PASSED: Continuous integration, rev:69
http://
Executed test runs:
SUCCESS: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
PS Jenkins bot (ps-jenkins) wrote : | # |
PASSED: Continuous integration, rev:70
http://
Executed test runs:
SUCCESS: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
Martin Pitt (pitti) wrote : | # |
OK, two passes in a row, and armhf tests only take 77.51 sec. Feel free to trigger a third build, but from my POV this is robust enough now (I ran this through sbuild another three times).
- 70. By Martin Pitt on 2013-06-26
-
Add (failing) test cases for selecting int/bool properties
This reproduces https:/
/launchpad. net/bugs/ 1194763
Francis Ginther (fginther) wrote : | # |
So I had some trouble running the tests on raring outside of the package build (i.e. "cmake .; make; make test"). But it all ran fine under saucy. The failures are in:
tests.test_
- The test attempts to open the menu where it would appear if it were embedded in the app window, but my menu is in the unity panel.
tests.test_
- This test fails because it can't find the 'GdRevealer'. Possibly this doesn't exist in the saucy version of gnome-system-log?
Given that the failures are not seen on saucy (where development is targeted). I don't see a reason not to approve.
Martin Pitt (pitti) wrote : | # |
Indeed in saucy some programs fell back to having integrated menus even under Unity, such as gtimelog or the hello_color.py test application. I'm not sure whether that's deliberate or a bug, I'll ask Didier later. But it should be possible to force a builtin menu with setting UBUNTU_MENUPROXY.
The test_search test should then just be skipped if gnome-system-log is too old.
I'll try this stuff in a raring VM (I don't currently have a raring installation), and do another MP to fix the package build on raring.
Martin Pitt (pitti) wrote : | # |
Francis, I fixed the tests for raring in https:/
This MP depends on (and contains) the bits for running the tests during package build: https:/ /code.launchpad .net/~pitti/ autopilot- gtk/testsuite/ +merge/ 170607 . So please review that one first.