Merge lp:~canonical-platform-qa/webbrowser-app/autopkgtest into lp:webbrowser-app
- autopkgtest
- Merge into trunk
Status: | Rejected |
---|---|
Rejected by: | Olivier Tilloy |
Proposed branch: | lp:~canonical-platform-qa/webbrowser-app/autopkgtest |
Merge into: | lp:webbrowser-app |
Diff against target: |
163 lines (+118/-3) 3 files modified
README (+57/-3) debian/tests/control (+26/-0) debian/tests/touch-session-autopilot (+35/-0) |
To merge this branch: | bzr merge lp:~canonical-platform-qa/webbrowser-app/autopkgtest |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Olivier Tilloy | Disapprove | ||
PS Jenkins bot | continuous-integration | Needs Fixing | |
Martin Pitt | Needs Fixing | ||
Review via email: mp+256858@code.launchpad.net |
Commit message
Added the dep8 tests and instructions on the README to run them in qemu or on the phone.
Description of the change
This introduces dep8 tests that can be run against a local qemu instance (optionally configured against -proposed) as well as a local phone.
It runs the two autopilot suites one dep8 test by autopilot suite (see README for adt-run runes).
Leo Arias (elopio) wrote : | # |
Brendan Donegan (brendan-donegan) wrote : | # |
Small comment about where the docs go
Olivier Tilloy (osomon) wrote : | # |
Also note that this source package contains two autopilot test suites: webbrowser_app and webapp_container. I guess both should be run.
Leo Arias (elopio) wrote : | # |
Thanks for the review Brendan. Reply inline. Please let me know what do you think.
Leo Arias (elopio) wrote : | # |
> Also note that this source package contains two autopilot test suites:
> webbrowser_app and webapp_container. I guess both should be run.
I'm giving that a try...
Vincent Ladeuil (vila) wrote : | # |
This now passes the tests against the current archive, -proposed and the phone (krillin devel-proposed 196).
@Leo: Feedback welcome, there may still be some rough edges and bits we'd like to be ready for reuse in other projects but I think that can wait.
Brendan Donegan (brendan-donegan) wrote : | # |
Small suggestion
Martin Pitt (pitti) wrote : | # |
Nice work! This should get some cleanup (see inline comments), but by and large LGTM.
This is a good prototype for seeing what we need to fix in our infrastructure to eventually not make this an eyebrow-riser, so once we can massage these tests to be simple again we can declare victory :-)
Martin Pitt (pitti) wrote : | # |
Also, I don't think we want to have all that README spread out/duplicated into all of our packages, but rather centrally in README.
PS Jenkins bot (ps-jenkins) wrote : | # |
PASSED: Continuous integration, rev:977
http://
Executed test runs:
SUCCESS: http://
SUCCESS: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
SUCCESS: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
- 978. By Vincent Ladeuil
-
From pitti's review: remove wrappers, use Test-Command instead. Import ubuntu-
touch-session env in a non-barbaric way. - 979. By Vincent Ladeuil
-
Revert the needs-recommends tweak, it's too greedy and on the phone it trigger importing python3-xlib which in turn triggers a bug in autopilot automatic platform detection.
Vincent Ladeuil (vila) wrote : | # |
Thanks a lot for the reviews pitti/brendand.
I've made a some modifications and pushed them.
Discussed on IRC:
- ADT_NORMAL_USER is not available for tests so keep using $USER
- ubuntu-
I tried needs-recommends instead of explicitly specifying python3-evdev but it broke on the phone so I reverted that part. My gut feeling is that python3-xlib is installed on the phone and confuses autopilot.
We may want to fix autopilot though.
Vincent Ladeuil (vila) wrote : | # |
Vincent Ladeuil (vila) wrote : | # |
> Also, I don't think we want to have all that README spread out/duplicated into
> all of our packages, but rather centrally in README.
> wiki?
Sorry, I missed that comment earlier.
I'm -0 here, until everybody knows how to use adt-run, I'm not against some doc duplication.
PS Jenkins bot (ps-jenkins) wrote : | # |
PASSED: Continuous integration, rev:979
http://
Executed test runs:
SUCCESS: http://
SUCCESS: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
SUCCESS: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
Vincent Ladeuil (vila) wrote : | # |
> > Also, I don't think we want to have all that README spread out/duplicated
> into
> > all of our packages, but rather centrally in README.
> > wiki?
>
> Sorry, I missed that comment earlier.
>
> I'm -0 here, until everybody knows how to use adt-run, I'm not against some
> doc duplication.
Meh, upon reflection I think we should indeed centralize the doc AND point to it from the README. What's the URL I should use ?
Federico Gimenez (fgimenez) wrote : | # |
Looks really great! One inline comment about test dependencies.
Don't know if it's relevant at this stage but I'm getting this [1] following the README for qemu and the current archive, there's an error for the first test:
bash: line 8: /tmp/tmp.
Thanks!
Federico Gimenez (fgimenez) wrote : | # |
Looks really great! One inline comment about test dependencies.
Don't know if it's relevant at this stage but I'm getting this [1] following the README, there's an error for the first test:
bash: line 8: /tmp/tmp.
Thanks!
Federico Gimenez (fgimenez) wrote : | # |
Looks really great! One inline comment about test dependencies.
Don't know if it's relevant at this stage but I'm getting this [1] following the README for qemu and the current archive, there's an error for the first test:
bash: line 8: /tmp/tmp.
Thanks!
Federico Gimenez (fgimenez) wrote : | # |
Looks really great! One inline comment about test dependencies.
Don't know if it's relevant at this stage but I'm getting this [1] following the README for qemu and the current archive, there's an error for the first test:
bash: line 8: /tmp/tmp.
Thanks!
Federico Gimenez (fgimenez) wrote : | # |
Looks really great! One inline comment about test dependencies.
Don't know if it's relevant at this stage but I'm getting this [1] following the README for qemu and the current archive, there's an error in the first test:
bash: line 8: /tmp/tmp.
Thanks!
- 980. By Vincent Ladeuil
-
Clarify why autopkgtest is a test dependency.
PS Jenkins bot (ps-jenkins) wrote : | # |
PASSED: Continuous integration, rev:980
http://
Executed test runs:
SUCCESS: http://
SUCCESS: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
SUCCESS: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
Martin Pitt (pitti) wrote : | # |
Thanks Vincent for the updates! One important bug (sudo -A) left, the others are nitpicks.
- 981. By Vincent Ladeuil
-
Mention the prefered targets for the documentation about running dep8 tests.
- 982. By Vincent Ladeuil
-
XS-Testsuite isn't necessary at all any more, dpkg-source adds this automatically now. (from pitti's review).
- 983. By Vincent Ladeuil
-
Remove the FIXME, explicitly Depend'ing on python3-evdev is fine. Added 'Classes: ubuntu-touch' to document the target test class (currently a no-op).
- 984. By Vincent Ladeuil
-
Use -A for sudo or the test can hang if sudo asks for a password.
Vincent Ladeuil (vila) wrote : | # |
Thanks for the review !
All fixed as mentioned in the inline comments.
PS Jenkins bot (ps-jenkins) wrote : | # |
FAILED: Continuous integration, rev:984
http://
Executed test runs:
FAILURE: http://
SUCCESS: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
FAILURE: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
PS Jenkins bot (ps-jenkins) wrote : | # |
FAILED: Continuous integration, rev:984
http://
Executed test runs:
FAILURE: http://
SUCCESS: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
FAILURE: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
PS Jenkins bot (ps-jenkins) wrote : | # |
FAILED: Continuous integration, rev:984
http://
Executed test runs:
FAILURE: http://
SUCCESS: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
FAILURE: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
Olivier Tilloy (osomon) wrote : | # |
Any update on:
+.. FIXME: This shouldn't be part of a specific package but instead be part
+ of the wiki
+ https:/
+ of autopkgtest README.
?
Olivier Tilloy (osomon) wrote : | # |
As far as the changes themselves, they look ok to me (I’m not really familiar with autopkgtests so I’ll trust Martin’s thorough review). I’ve tested the instructions and got errors while running the actual tests, but the runners seemed to be doing their job.
Leo Arias (elopio) wrote : | # |
Thanks for looking at this Olivier.
But it's really important that all the tests are passing, otherwise you won't be able to land an update to the archive.
We need to either fix them or select a subset of tests to begin with. Can you please paste the errors you are getting?
- 985. By Vincent Ladeuil
-
Remove the FIXME about doc by refering to the official autopkgtest documentation and dropping the proposed-migration example as it's not defined by the project.
Vincent Ladeuil (vila) wrote : | # |
> Any update on:
>
> +.. FIXME: This shouldn't be part of a specific package but instead be part
Sorry about the delay, I somehow never seen that comment :-/
Fixed now.
PS Jenkins bot (ps-jenkins) wrote : | # |
FAILED: Continuous integration, rev:985
http://
Executed test runs:
FAILURE: http://
SUCCESS: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
FAILURE: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
Vincent Ladeuil (vila) wrote : | # |
> Thanks for looking at this Olivier.
> But it's really important that all the tests are passing, otherwise you won't
> be able to land an update to the archive.
A bit of clarification here, proposed-migration will not block for tests that always failed in the past.
Since this is the first dep8 test, the landing won't be blocked.
Once the test is passing though, yes, new failures will block the landing.
>
> We need to either fix them or select a subset of tests to begin with. Can you
> please paste the errors you are getting?
From http://
The following packages have unmet dependencies:
webbrowser-app : Depends: liboxideqt-
E: Unable to correct problems, you have held broken packages.
ADB_RC=100
So,
1) it's failing on mako (which won't happen in proposed-migration (we're still far from running tests on phones in the proposed-migration)
2) I can use some help understanding the failure there as I don't quite get where liboxideqt-
Olivier Tilloy (osomon) wrote : | # |
> I can use some help understanding the failure there as I don't quite
> get where liboxideqt-
The release pocket of vivid has oxide 1.5.5-0ubuntu1.
The updates and security pockets have oxide 1.6.6-0ubuntu0.
The vivid overlay PPA has oxide 1.7.7-0ubuntu0.
Vincent Ladeuil (vila) wrote : | # |
> > I can use some help understanding the failure there as I don't quite
> > get where liboxideqt-
>
> The release pocket of vivid has oxide 1.5.5-0ubuntu1.
> The updates and security pockets have oxide 1.6.6-0ubuntu0.
> The vivid overlay PPA has oxide 1.7.7-0ubuntu0.
am I looking at the wrong place ?
Vincent Ladeuil (vila) wrote : | # |
> It's not in https:/
> /stable-
>
> am I looking at the wrong place ?
Yes and no ;) It's part of oxide-qt so available there, the -ci job mentions adding the overlay ppa too.
But apt-get doesn't find it...
PS Jenkins bot (ps-jenkins) wrote : | # |
FAILED: Continuous integration, rev:985
http://
Executed test runs:
FAILURE: http://
SUCCESS: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
FAILURE: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
Vincent Ladeuil (vila) wrote : | # |
Ok, so, the issue is that -ci use ubuntu/
PS Jenkins bot (ps-jenkins) wrote : | # |
FAILED: Continuous integration, rev:985
http://
Executed test runs:
FAILURE: http://
SUCCESS: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
FAILURE: http://
SUCCESS: http://
deb: http://
SUCCESS: http://
Click here to trigger a rebuild:
http://
Vincent Ladeuil (vila) wrote : | # |
.... and:
$ rmadison liboxideqt-
...
liboxideqt-
@Olivier: Do you know why 1.7 is only in wily-proposed ? What's the way out of this situation ?
Olivier Tilloy (osomon) wrote : | # |
@vila: migration of oxide 1.7.7 from proposed to release was blocked on some autopkgtest failure, it has now been resolved.
PS Jenkins bot (ps-jenkins) wrote : | # |
FAILED: Continuous integration, rev:985
http://
Executed test runs:
FAILURE: http://
FAILURE: http://
FAILURE: http://
FAILURE: http://
FAILURE: http://
Click here to trigger a rebuild:
http://
PS Jenkins bot (ps-jenkins) wrote : | # |
FAILED: Continuous integration, rev:985
http://
Executed test runs:
FAILURE: http://
FAILURE: http://
FAILURE: http://
FAILURE: http://
FAILURE: http://
Click here to trigger a rebuild:
http://
Olivier Tilloy (osomon) wrote : | # |
Unmerged revisions
Preview Diff
1 | === modified file 'README' |
2 | --- README 2015-04-01 05:57:42 +0000 |
3 | +++ README 2015-05-18 21:41:52 +0000 |
4 | @@ -40,11 +40,11 @@ |
5 | |
6 | $ ctest |
7 | |
8 | - |
9 | = Automated UI tests = |
10 | |
11 | webbrowser-app uses autopilot (https://launchpad.net/autopilot) to test its UI. |
12 | -To run the tests, you will need to install python3-autopilot and autopilot-qt5. |
13 | +To run the tests locally, you will need to install python3-autopilot and |
14 | +autopilot-qt5. |
15 | Then do the following: |
16 | |
17 | $ cd tests/autopilot/ |
18 | @@ -54,6 +54,8 @@ |
19 | |
20 | $ autopilot3 list webbrowser_app |
21 | |
22 | +In order to run the tests in a virtual machine with an environment closer to |
23 | +what a user will get in Ubuntu Touch, see the Dep8 tests section. |
24 | |
25 | = Code coverage = |
26 | |
27 | @@ -68,6 +70,59 @@ |
28 | This will generate a coverage report in XML format (coverage.xml) and an |
29 | interactive human-readable report in HTML format (coveragereport/index.html). |
30 | |
31 | += Dep8 tests = |
32 | + |
33 | +Dep8 tests excercise the package "as-installed". |
34 | + |
35 | +Currently, the webbrowser-app has one suite of dep8 tests that uses autopilot |
36 | +(https://launchpad.net/autopilot) to test from the point of view of the user. |
37 | + |
38 | +To run the tests you will need autopkgtest: |
39 | + |
40 | + $ sudo apt-get install autopkgtest |
41 | + |
42 | +You can use multiple test beds to execute the tests. Below you will find |
43 | +instructions to run them in a virtual machine |
44 | +You can find more information with: |
45 | + |
46 | + $ man adt-run |
47 | + |
48 | +== Run dep8 tests == |
49 | + |
50 | +To run the tests in a qemu virtual machine, you will first have to create it |
51 | +(see /usr/share/doc/autopkgtest/README.running-tests.rst.gz). We output the |
52 | +image to ~/ rather than the current directory, so it will be in a safer |
53 | +place to avoid rebuilding images every time. You can store it in any |
54 | +directory you wish. This image is better consumed "fresh", building it daily |
55 | +will avoid long updates/upgrades when running the tests. |
56 | + |
57 | + $ adt-buildvm-ubuntu-cloud -r vivid -a amd64 -o ~/ |
58 | + |
59 | +Then run the tests using adt-run with the qemu virtualization host against |
60 | +the current archive. |
61 | + |
62 | + $ adt-run -B -U --unbuilt-tree . \ |
63 | + -o ~/adt-browser-test/$(date +%Y-%m-%d-%H-%M) \ |
64 | + --- qemu ~/adt-vivid-amd64-cloud.img |
65 | + |
66 | +The tests can also be run on a local phone. |
67 | + |
68 | + $ adt-run -B -U --unbuilt-tree . \ |
69 | + -o ~/adt-browser-test/$(date +%Y-%m-%d-%H-%M) \ |
70 | + --- ssh -s adb -- -p <password> --serial <ADB_SERIAL> |
71 | + |
72 | +== Examine the dep8 autopilot results == |
73 | + |
74 | +To examine the test results, which are in subunit format, additional tools are |
75 | +required. |
76 | + |
77 | + $ sudo add-apt-repository ppa:thomir/trv |
78 | + $ sudo apt-get update |
79 | + $ sudo apt-get install trv |
80 | + $ trv |
81 | + |
82 | +You can find the results file in the directory |
83 | +~/adt-browser-test/$(date +%Y-%m-%d-%H-%M)/artifacts. |
84 | |
85 | = Settings = |
86 | |
87 | @@ -93,4 +148,3 @@ |
88 | |
89 | - restoreSession: whether to restore the previous browsing session at startup |
90 | (defaults to true) |
91 | - |
92 | |
93 | === added directory 'debian/tests' |
94 | === added file 'debian/tests/control' |
95 | --- debian/tests/control 1970-01-01 00:00:00 +0000 |
96 | +++ debian/tests/control 2015-05-18 21:41:52 +0000 |
97 | @@ -0,0 +1,26 @@ |
98 | +# Copyright 2015 Canonical |
99 | +# |
100 | +# This program is free software: you can redistribute it and/or modify it |
101 | +# under the terms of the GNU General Public License version 3, as published |
102 | +# by the Free Software Foundation. |
103 | +# |
104 | +# This program is distributed in the hope that it will be useful, |
105 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
106 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
107 | +# GNU General Public License for more details. |
108 | +# |
109 | +# You should have received a copy of the GNU General Public License |
110 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
111 | + |
112 | +# autopkgtest is a test dependency so we don't need to copy |
113 | +# ubuntu-touch-session to the testbed but can use it from autopkgtest instead |
114 | + |
115 | +Test-Command: debian/tests/touch-session-autopilot webbrowser_app |
116 | +Restrictions: allow-stderr |
117 | +Classes: ubuntu-touch |
118 | +Depends: webbrowser-app-autopilot, autopkgtest, python3-evdev |
119 | + |
120 | +Test-Command: debian/tests/touch-session-autopilot webapp_container |
121 | +Restrictions: allow-stderr |
122 | +Classes: ubuntu-touch |
123 | +Depends: webapp-container-autopilot, autopkgtest, python3-evdev |
124 | |
125 | === added file 'debian/tests/touch-session-autopilot' |
126 | --- debian/tests/touch-session-autopilot 1970-01-01 00:00:00 +0000 |
127 | +++ debian/tests/touch-session-autopilot 2015-05-18 21:41:52 +0000 |
128 | @@ -0,0 +1,35 @@ |
129 | +#!/bin/sh |
130 | + |
131 | +# Copyright 2015 Canonical |
132 | +# |
133 | +# This program is free software: you can redistribute it and/or modify it |
134 | +# under the terms of the GNU General Public License version 3, as published |
135 | +# by the Free Software Foundation. |
136 | +# |
137 | +# This program is distributed in the hope that it will be useful, |
138 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
139 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
140 | +# GNU General Public License for more details. |
141 | +# |
142 | +# You should have received a copy of the GNU General Public License |
143 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
144 | + |
145 | +# This runs the $1 autopilot suite ensuring that the testbed is configured |
146 | +# properly |
147 | + |
148 | +SUITE=$1 |
149 | + |
150 | +set -e |
151 | + |
152 | +if ! pgrep -f unity-system-compositor ; then |
153 | + # We're not using Mir, setup enough of an X stack |
154 | + |
155 | + # FIXME: We won't need an X stack once Mir provides mirvfb or any other |
156 | + # mean to test consistently on all platforms -- vila 2015-04-22 |
157 | + sudo -A ADT_NORMAL_USER=${USER} sh -e /usr/share/autopkgtest/setup-commands/ubuntu-touch-session |
158 | + # Import the environment produced above in the current shell |
159 | + set -a |
160 | + . /etc/environment |
161 | +fi |
162 | + |
163 | +autopilot3 run -v -f subunit -o ${ADT_ARTIFACTS} ${SUITE} |
Not yet ready for review because if we land this and the tests are run when the package is being upgraded in the archive, they will fail because the infrastructure is not ready yet to pass the --setup-command.