webbrowser-app

Merge lp:~canonical-platform-qa/webbrowser-app/autopkgtest into lp:webbrowser-app

autopkgtest
Merge into trunk

Proposed by Leo Arias on 2015-04-20

Status:	Rejected
Rejected by:	Olivier Tilloy on 2016-03-15
Proposed branch:	lp:~canonical-platform-qa/webbrowser-app/autopkgtest
Merge into:	lp:webbrowser-app
Diff against target:	163 lines (+118/-3) 3 files modified README (+57/-3) debian/tests/control (+26/-0) debian/tests/touch-session-autopilot (+35/-0)
To merge this branch:	bzr merge lp:~canonical-platform-qa/webbrowser-app/autopkgtest
Related bugs:	Link a bug report

Reviewer	Review Type	Date Requested	Status
Olivier Tilloy			Disapprove on 2016-03-15
PS Jenkins bot	continuous-integration		Needs Fixing on 2015-05-27
Martin Pitt		2015-04-20	Needs Fixing on 2015-05-05
Review via email: mp+256858@code.launchpad.net

Commit Message

Added the dep8 tests and instructions on the README to run them in qemu or on the phone.

Description of the Change

This introduces dep8 tests that can be run against a local qemu instance (optionally configured against -proposed) as well as a local phone.

It runs the two autopilot suites one dep8 test by autopilot suite (see README for adt-run runes).

Leo Arias (elopio) wrote on 2015-04-20:

Not yet ready for review because if we land this and the tests are run when the package is being upgraded in the archive, they will fail because the infrastructure is not ready yet to pass the --setup-command.

Brendan Donegan (brendan-donegan) wrote on 2015-04-20:

Small comment about where the docs go

Olivier Tilloy (osomon) wrote on 2015-04-21:

Also note that this source package contains two autopilot test suites: webbrowser_app and webapp_container. I guess both should be run.

Leo Arias (elopio) wrote on 2015-04-21:

Thanks for the review Brendan. Reply inline. Please let me know what do you think.

Leo Arias (elopio) wrote on 2015-04-21:

> Also note that this source package contains two autopilot test suites:
> webbrowser_app and webapp_container. I guess both should be run.

I'm giving that a try...

Vincent Ladeuil (vila) wrote on 2015-04-23:

This now passes the tests against the current archive, -proposed and the phone (krillin devel-proposed 196).

@Leo: Feedback welcome, there may still be some rough edges and bits we'd like to be ready for reuse in other projects but I think that can wait.

Brendan Donegan (brendan-donegan) wrote on 2015-04-23:

Small suggestion

Martin Pitt (pitti) wrote on 2015-04-23:

Nice work! This should get some cleanup (see inline comments), but by and large LGTM.

This is a good prototype for seeing what we need to fix in our infrastructure to eventually not make this an eyebrow-riser, so once we can massage these tests to be simple again we can declare victory :-)

Martin Pitt (pitti) wrote on 2015-04-23:

Also, I don't think we want to have all that README spread out/duplicated into all of our packages, but rather centrally in README.running-tests and/or the wiki?

PS Jenkins bot (ps-jenkins) wrote on 2015-04-23:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/webbrowser-app-ci/1680/rebuild

review: Approve (continuous-integration)

lp:~canonical-platform-qa/webbrowser-app/autopkgtest updated on 2015-04-23

978. By Vincent Ladeuil on 2015-04-23: From pitti's review: remove wrappers, use Test-Command instead. Import ubuntu-touch-session env in a non-barbaric way.
979. By Vincent Ladeuil on 2015-04-23: Revert the needs-recommends tweak, it's too greedy and on the phone it trigger importing python3-xlib which in turn triggers a bug in autopilot automatic platform detection.

Vincent Ladeuil (vila) wrote on 2015-04-23:

Thanks a lot for the reviews pitti/brendand.

I've made a some modifications and pushed them.

Discussed on IRC:
- ADT_NORMAL_USER is not available for tests so keep using $USER
- ubuntu-touch-session doesn't have the x bit so keep using sh -e

I tried needs-recommends instead of explicitly specifying python3-evdev but it broke on the phone so I reverted that part. My gut feeling is that python3-xlib is installed on the phone and confuses autopilot.
We may want to fix autopilot though.

Vincent Ladeuil (vila) wrote on 2015-04-23:

https://bugs.launchpad.net/autopilot/+bug/1447688 filed

Vincent Ladeuil (vila) wrote on 2015-04-23:

> Also, I don't think we want to have all that README spread out/duplicated into
> all of our packages, but rather centrally in README.running-tests and/or the
> wiki?

Sorry, I missed that comment earlier.

I'm -0 here, until everybody knows how to use adt-run, I'm not against some doc duplication.

PS Jenkins bot (ps-jenkins) wrote on 2015-04-23:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/webbrowser-app-ci/1682/rebuild

review: Approve (continuous-integration)

Vincent Ladeuil (vila) wrote on 2015-04-24:

> > Also, I don't think we want to have all that README spread out/duplicated
> into
> > all of our packages, but rather centrally in README.running-tests and/or the
> > wiki?
>
> Sorry, I missed that comment earlier.
>
> I'm -0 here, until everybody knows how to use adt-run, I'm not against some
> doc duplication.

Meh, upon reflection I think we should indeed centralize the doc AND point to it from the README. What's the URL I should use ?

Federico Gimenez (fgimenez) wrote on 2015-04-24:

Looks really great! One inline comment about test dependencies.

Don't know if it's relevant at this stage but I'm getting this [1] following the README for qemu and the current archive, there's an error for the first test:

bash: line 8: /tmp/tmp.bEKuLEBs9e/upstart/sessions/9325.session: No such file or directory

Thanks!

[1] http://paste.ubuntu.com/10877550/

Federico Gimenez (fgimenez) wrote on 2015-04-24:

Looks really great! One inline comment about test dependencies.

Don't know if it's relevant at this stage but I'm getting this [1] following the README, there's an error for the first test:

bash: line 8: /tmp/tmp.bEKuLEBs9e/upstart/sessions/9325.session: No such file or directory

Thanks!

[1] http://paste.ubuntu.com/10877550/

Federico Gimenez (fgimenez) wrote on 2015-04-24:

Looks really great! One inline comment about test dependencies.

Don't know if it's relevant at this stage but I'm getting this [1] following the README for qemu and the current archive, there's an error for the first test:

bash: line 8: /tmp/tmp.bEKuLEBs9e/upstart/sessions/9325.session: No such file or directory

Thanks!

[1] http://paste.ubuntu.com/10877550/

Federico Gimenez (fgimenez) wrote on 2015-04-24:

Looks really great! One inline comment about test dependencies.

Don't know if it's relevant at this stage but I'm getting this [1] following the README for qemu and the current archive, there's an error for the first test:

bash: line 8: /tmp/tmp.bEKuLEBs9e/upstart/sessions/9325.session: No such file or directory

Thanks!

[1] http://paste.ubuntu.com/10877550/

Federico Gimenez (fgimenez) wrote on 2015-04-24:

Looks really great! One inline comment about test dependencies.

Don't know if it's relevant at this stage but I'm getting this [1] following the README for qemu and the current archive, there's an error in the first test:

bash: line 8: /tmp/tmp.bEKuLEBs9e/upstart/sessions/9325.session: No such file or directory

Thanks!

[1] http://paste.ubuntu.com/10877550/

lp:~canonical-platform-qa/webbrowser-app/autopkgtest updated on 2015-04-24

980. By Vincent Ladeuil on 2015-04-24: Clarify why autopkgtest is a test dependency.

PS Jenkins bot (ps-jenkins) wrote on 2015-04-24:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/webbrowser-app-ci/1684/rebuild

review: Approve (continuous-integration)

Martin Pitt (pitti) wrote on 2015-05-05:

Thanks Vincent for the updates! One important bug (sudo -A) left, the others are nitpicks.

review: Needs Fixing

lp:~canonical-platform-qa/webbrowser-app/autopkgtest updated on 2015-05-05

981. By Vincent Ladeuil on 2015-05-05: Mention the prefered targets for the documentation about running dep8 tests.
982. By Vincent Ladeuil on 2015-05-05: XS-Testsuite isn't necessary at all any more, dpkg-source adds this automatically now. (from pitti's review).
983. By Vincent Ladeuil on 2015-05-05: Remove the FIXME, explicitly Depend'ing on python3-evdev is fine. Added 'Classes: ubuntu-touch' to document the target test class (currently a no-op).
984. By Vincent Ladeuil on 2015-05-05: Use -A for sudo or the test can hang if sudo asks for a password.

Vincent Ladeuil (vila) wrote on 2015-05-05:

Thanks for the review !

All fixed as mentioned in the inline comments.

PS Jenkins bot (ps-jenkins) wrote on 2015-05-05:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/webbrowser-app-ci/1724/rebuild

review: Needs Fixing (continuous-integration)

PS Jenkins bot (ps-jenkins) wrote on 2015-05-07:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/webbrowser-app-ci/1732/rebuild

review: Needs Fixing (continuous-integration)

PS Jenkins bot (ps-jenkins) wrote on 2015-05-07:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/webbrowser-app-ci/1734/rebuild

review: Needs Fixing (continuous-integration)

Olivier Tilloy (osomon) wrote on 2015-05-11:

Any update on:

+.. FIXME: This shouldn't be part of a specific package but instead be part
+ of the wiki
+ https://wiki.ubuntu.com/Touch/Testing#Running_tests_with_autopkgtest or
+ of autopkgtest README.running-tests -- vila 2015-05-05

Olivier Tilloy (osomon) wrote on 2015-05-11:

As far as the changes themselves, they look ok to me (I’m not really familiar with autopkgtests so I’ll trust Martin’s thorough review). I’ve tested the instructions and got errors while running the actual tests, but the runners seemed to be doing their job.

review: Approve

Leo Arias (elopio) wrote on 2015-05-11:

Thanks for looking at this Olivier.
But it's really important that all the tests are passing, otherwise you won't be able to land an update to the archive.

We need to either fix them or select a subset of tests to begin with. Can you please paste the errors you are getting?

lp:~canonical-platform-qa/webbrowser-app/autopkgtest updated on 2015-05-18

985. By Vincent Ladeuil on 2015-05-18: Remove the FIXME about doc by refering to the official autopkgtest documentation and dropping the proposed-migration example as it's not defined by the project.

Vincent Ladeuil (vila) wrote on 2015-05-18:

> Any update on:
>
> +.. FIXME: This shouldn't be part of a specific package but instead be part

Sorry about the delay, I somehow never seen that comment :-/

Fixed now.

PS Jenkins bot (ps-jenkins) wrote on 2015-05-18:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/webbrowser-app-ci/1785/rebuild

review: Needs Fixing (continuous-integration)

Vincent Ladeuil (vila) wrote on 2015-05-19:

> Thanks for looking at this Olivier.
> But it's really important that all the tests are passing, otherwise you won't
> be able to land an update to the archive.

A bit of clarification here, proposed-migration will not block for tests that always failed in the past.

Since this is the first dep8 test, the landing won't be blocked.

Once the test is passing though, yes, new failures will block the landing.

>
> We need to either fix them or select a subset of tests to begin with. Can you
> please paste the errors you are getting?

From http://s-jenkins.ubuntu-ci:8080/job/generic-deb-autopilot-runner-vivid-mako/2461/console

The following packages have unmet dependencies:
webbrowser-app : Depends: liboxideqt-qmlplugin (>= 1.7) but 1.5.5-0ubuntu1 is to be installed
E: Unable to correct problems, you have held broken packages.
ADB_RC=100

So,

1) it's failing on mako (which won't happen in proposed-migration (we're still far from running tests on phones in the proposed-migration)

2) I can use some help understanding the failure there as I don't quite get where liboxideqt-qmlplugin is coming from (or should come from).

Olivier Tilloy (osomon) wrote on 2015-05-19:

> I can use some help understanding the failure there as I don't quite
> get where liboxideqt-qmlplugin is coming from (or should come from).

The release pocket of vivid has oxide 1.5.5-0ubuntu1.
The updates and security pockets have oxide 1.6.6-0ubuntu0.15.04.1.
The vivid overlay PPA has oxide 1.7.7-0ubuntu0.15.04.1~ppa1.

Vincent Ladeuil (vila) wrote on 2015-05-19:

> > I can use some help understanding the failure there as I don't quite
> > get where liboxideqt-qmlplugin is coming from (or should come from).
>
> The release pocket of vivid has oxide 1.5.5-0ubuntu1.
> The updates and security pockets have oxide 1.6.6-0ubuntu0.15.04.1.
> The vivid overlay PPA has oxide 1.7.7-0ubuntu0.15.04.1~ppa1.

It's not in https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/stable-phone-overlay/+index?field.series_filter=vivid

am I looking at the wrong place ?

Vincent Ladeuil (vila) wrote on 2015-05-19:

> It's not in https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu
> /stable-phone-overlay/+index?field.series_filter=vivid
>
> am I looking at the wrong place ?

Yes and no ;) It's part of oxide-qt so available there, the -ci job mentions adding the overlay ppa too.

But apt-get doesn't find it...

PS Jenkins bot (ps-jenkins) wrote on 2015-05-19:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/webbrowser-app-ci/1793/rebuild

review: Needs Fixing (continuous-integration)

Vincent Ladeuil (vila) wrote on 2015-05-19:

Ok, so, the issue is that -ci use ubuntu/devel-proposed for mako which is wily and the overlay ppa does not have packages for wily.

PS Jenkins bot (ps-jenkins) wrote on 2015-05-19:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/webbrowser-app-ci/1794/rebuild

review: Needs Fixing (continuous-integration)

Vincent Ladeuil (vila) wrote on 2015-05-20:

.... and:

$ rmadison liboxideqt-qmlplugin

...

liboxideqt-qmlplugin | 1.7.7-0ubuntu0.15.04.1~ppa1 | wily-proposed | amd64, armhf, i386

@Olivier: Do you know why 1.7 is only in wily-proposed ? What's the way out of this situation ?

Olivier Tilloy (osomon) wrote on 2015-05-27:

@vila: migration of oxide 1.7.7 from proposed to release was blocked on some autopkgtest failure, it has now been resolved.

PS Jenkins bot (ps-jenkins) wrote on 2015-05-27:

FAILED: Continuous integration, rev:985
http://jenkins.qa.ubuntu.com/job/webbrowser-app-ci/1819/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-deb-autopilot-vivid-touch/2959/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/webbrowser-app-vivid-amd64-ci/576/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/webbrowser-app-vivid-armhf-ci/576/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/webbrowser-app-vivid-i386-ci/576/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-armhf/2957/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/webbrowser-app-ci/1819/rebuild

review: Needs Fixing (continuous-integration)

PS Jenkins bot (ps-jenkins) wrote on 2015-05-27:

FAILED: Continuous integration, rev:985
http://jenkins.qa.ubuntu.com/job/webbrowser-app-ci/1820/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-deb-autopilot-vivid-touch/2960/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/webbrowser-app-vivid-amd64-ci/577/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/webbrowser-app-vivid-armhf-ci/577/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/webbrowser-app-vivid-i386-ci/577/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-vivid-armhf/2958/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/webbrowser-app-ci/1820/rebuild

review: Needs Fixing (continuous-integration)

Olivier Tilloy (osomon) wrote on 2016-03-15:

Superseded by https://code.launchpad.net/~osomon/webbrowser-app/autopkgtests/+merge/289068.

review: Disapprove

Unmerged revisions

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk

Subscribers

People subscribed via source and target branches

to all changes:

Canonical Platform QA Team

Ubuntu Phablet Team

to status/vote changes:

Rachel Liu

 === modified file 'README'
 --- README	2015-04-01 05:57:42 +0000
 +++ README	2015-05-18 21:41:52 +0000
@@ -40,11 +40,11 @@
      $ ctest
--
  = Automated UI tests =
  webbrowser-app uses autopilot (https://launchpad.net/autopilot) to test its UI.
--To run the tests, you will need to install python3-autopilot and autopilot-qt5.
++To run the tests locally, you will need to install python3-autopilot and
++autopilot-qt5.
  Then do the following:
      $ cd tests/autopilot/
@@ -54,6 +54,8 @@
      $ autopilot3 list webbrowser_app
++In order to run the tests in a virtual machine with an environment closer to
++what a user will get in Ubuntu Touch, see the Dep8 tests section.
  = Code coverage =
@@ -68,6 +70,59 @@
  This will generate a coverage report in XML format (coverage.xml) and an
  interactive human-readable report in HTML format (coveragereport/index.html).
++= Dep8 tests =
++
++Dep8 tests excercise the package "as-installed".
++
++Currently, the webbrowser-app has one suite of dep8 tests that uses autopilot
++(https://launchpad.net/autopilot) to test from the point of view of the user.
++
++To run the tests you will need autopkgtest:
++
++    $ sudo apt-get install autopkgtest
++
++You can use multiple test beds to execute the tests. Below you will find
++instructions to run them in a virtual machine
++You can find more information with:
++
++    $ man adt-run
++
++== Run dep8 tests ==
++
++To run the tests in a qemu virtual machine, you will first have to create it
++(see /usr/share/doc/autopkgtest/README.running-tests.rst.gz). We output the
++image to ~/ rather than the current directory, so it will be in a safer
++place to avoid rebuilding images every time. You can store it in any
++directory you wish. This image is better consumed "fresh", building it daily
++will avoid long updates/upgrades when running the tests.
++
++    $ adt-buildvm-ubuntu-cloud -r vivid -a amd64 -o ~/
++
++Then run the tests using adt-run with the qemu virtualization host against
++the current archive.
++
++    $ adt-run -B -U --unbuilt-tree . \
++      -o ~/adt-browser-test/$(date +%Y-%m-%d-%H-%M) \
++      --- qemu ~/adt-vivid-amd64-cloud.img
++
++The tests can also be run on a local phone.
++
++    $ adt-run -B -U --unbuilt-tree . \
++      -o ~/adt-browser-test/$(date +%Y-%m-%d-%H-%M) \
++      --- ssh -s adb -- -p <password> --serial <ADB_SERIAL>
++
++== Examine the dep8 autopilot results ==
++
++To examine the test results, which are in subunit format, additional tools are
++required.
++
++    $ sudo add-apt-repository ppa:thomir/trv
++    $ sudo apt-get update
++    $ sudo apt-get install trv
++    $ trv
++
++You can find the results file in the directory
++~/adt-browser-test/$(date +%Y-%m-%d-%H-%M)/artifacts.
  = Settings =
@@ -93,4 +148,3 @@
   - restoreSession: whether to restore the previous browsing session at startup
     (defaults to true)
--
 === added directory 'debian/tests'
 === added file 'debian/tests/control'
 --- debian/tests/control	1970-01-01 00:00:00 +0000
 +++ debian/tests/control	2015-05-18 21:41:52 +0000
@@ -0,0 +1,26 @@
++# Copyright 2015 Canonical
++#
++# This program is free software: you can redistribute it and/or modify it
++# under the terms of the GNU General Public License version 3, as published
++# by the Free Software Foundation.
++#
++# This program is distributed in the hope that it will be useful,
++# but WITHOUT ANY WARRANTY; without even the implied warranty of
++# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
++# GNU General Public License for more details.
++#
++# You should have received a copy of the GNU General Public License
++# along with this program. If not, see <http://www.gnu.org/licenses/>.
++
++# autopkgtest is a test dependency so we don't need to copy
++# ubuntu-touch-session to the testbed but can use it from autopkgtest instead
++
++Test-Command: debian/tests/touch-session-autopilot webbrowser_app
++Restrictions: allow-stderr
++Classes: ubuntu-touch
++Depends: webbrowser-app-autopilot, autopkgtest, python3-evdev
++
++Test-Command: debian/tests/touch-session-autopilot webapp_container
++Restrictions: allow-stderr
++Classes: ubuntu-touch
++Depends: webapp-container-autopilot, autopkgtest, python3-evdev
 === added file 'debian/tests/touch-session-autopilot'
 --- debian/tests/touch-session-autopilot	1970-01-01 00:00:00 +0000
 +++ debian/tests/touch-session-autopilot	2015-05-18 21:41:52 +0000
@@ -0,0 +1,35 @@
++#!/bin/sh
++
++# Copyright 2015 Canonical
++#
++# This program is free software: you can redistribute it and/or modify it
++# under the terms of the GNU General Public License version 3, as published
++# by the Free Software Foundation.
++#
++# This program is distributed in the hope that it will be useful,
++# but WITHOUT ANY WARRANTY; without even the implied warranty of
++# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
++# GNU General Public License for more details.
++#
++# You should have received a copy of the GNU General Public License
++# along with this program. If not, see <http://www.gnu.org/licenses/>.
++
++# This runs the $1 autopilot suite ensuring that the testbed is configured
++# properly
++
++SUITE=$1
++
++set -e
++
++if ! pgrep -f unity-system-compositor ; then
++   # We're not using Mir, setup enough of an X stack
++
++   # FIXME: We won't need an X stack once Mir provides mirvfb or any other
++   # mean to test consistently on all platforms -- vila 2015-04-22
++   sudo -A ADT_NORMAL_USER=${USER} sh -e /usr/share/autopkgtest/setup-commands/ubuntu-touch-session
++   # Import the environment produced above in the current shell
++   set -a
++   . /etc/environment
++fi
++
++autopilot3 run -v -f subunit -o ${ADT_ARTIFACTS} ${SUITE}