Merge lp:~fginther/ubuntu-test-cases/add-reminders into lp:ubuntu-test-cases/touch

Proposed by Francis Ginther on 2014-07-10
Status: Needs review
Proposed branch: lp:~fginther/ubuntu-test-cases/add-reminders
Merge into: lp:ubuntu-test-cases/touch
Diff against target: 165 lines (+101/-5)
2 files modified
jenkins/testconfig.py (+39/-0)
scripts/run-autopilot-tests.sh (+62/-5)
To merge this branch: bzr merge lp:~fginther/ubuntu-test-cases/add-reminders
Reviewer Review Type Date Requested Status
Nicholas Skaggs (community) Needs Fixing on 2014-08-20
Paul Larson Needs Fixing on 2014-08-01
Francis Ginther Resubmit on 2014-07-31
Andy Doan (community) 2014-07-10 Approve on 2014-07-10
Martin Pitt Pending
Review via email: mp+226281@code.launchpad.net

Commit message

Add testing for reminders click app.

Description of the change

Add testing for reminders click app.

This requires the utopic version of autopkgtest or from ppa:canonical-ci-engineering/ci-airline-phase-0

To post a comment you must log in.
Andy Doan (doanac) wrote :

please merge by hand. we have no tarmac landing here

review: Approve
Paul Larson (pwlars) wrote :

Did you try running this locally or anything? Looks right, but last I had known reminders wasn't added yet. I always do a quick run-through by hand to make sure there are no surprises.

Paul Larson (pwlars) wrote :

I'm seeing 7 tests, 4 pass, 3 fail. Does this match with what they were expecting?

Leo Arias (elopio) wrote :

Paul, are you running the tests on the phone freshly flashed? They should all pass.
If you are running them on the desktop without your keyring unlocked, you will get errors on the ones that require credentials. They will fail also if you already have an evernote account.
Those details are on TODO.
I'm about to go to bed, but tomorrow I can take a look at your errors.

Paul Larson (pwlars) wrote :

No, they are on a freshly flashed phone with no other tests running

Paul Larson (pwlars) wrote :

Does reminders-app have any .deb dependencies that need to be installed?

Leo Arias (elopio) wrote :

Yes, they need evernote-sdk-python3 that's on the applications PPA.
I couldn't give them a try on friday, I'm sorry. Can you paste the output
of your failures somewhere?

Paul Larson (pwlars) wrote :

Ok, it turns out this is all due to a bug. phablet-test-run is trying to import reminders to decide if it's python2 or python3, and due to the deprecation warning from uitk, it decides to run python2, which is not even installed. xnox has a fix for this in phablet-tools and is going to get it landed. I suspect there are other tests like this too, and we were just getting lucky enough with them to run them after something that depended on python2 and installed it.

review: Approve
Paul Larson (pwlars) wrote :

Wait, do I understand correctly that for these tests to pass, we would need to install a package from some other ppa? That's a bit of a problem since clicks don't have dependencies. We could hack this in somehow, but it's *really* not ideal. Having it in the image, or at least in the source tree would probably be ideal. Otherwise, if we need to hack around it and force the install of a deb before running tests on a click package, we'll at least need it in the archives, not a ppa.

Leo Arias (elopio) wrote :

We need to involve balloons and dpm in this discussion. On option was to copy the evernote python module to the package, but I'm not sure if their license allows us to do it. And the alternative is to make our own python module to communicate with their API, less than ideal because we would need to maintain it.

Leo Arias (elopio) wrote :

Sorry, scratch all I said. The thrift and evernote modules now are copied to the same reminders python package.

David Planella (dpm) wrote :

We're already including the Python Evernote SDK in the sources, and they're available in the reminders-app-autopilot binary generated from the reminders-app source package [1] in the utopic archive.

If you need this to be included in the click package instead, I guess it shouldn't be much of a problem.

[1] https://launchpad.net/ubuntu/+source/reminders-app

Paul Larson (pwlars) wrote :

Perhaps there are other dependencies still needed then? Running it with phablet-test-run locally, I get.

http://paste.ubuntu.com/7799766/
...
File "/home/phablet/autopilot/evernote/api/client.py", line 5, in <module>
from requests_oauthlib import OAuth1Session
ImportError: No module named 'requests_oauthlib'

If I manually install python-requests-oauthlib, I can get a bit further, but I still get:
File "/home/phablet/autopilot/reminders/fake_services.py", line 22, in <module>
import dbusmock
ImportError: No module named 'dbusmock'

I can install python3-dbusmock, but it pulls in the notorious dbus-x11 that's known to cause problems in the past. Even if I install that manually and try again though, I still get:
File "/home/phablet/autopilot/reminders/credentials.py", line 133, in _process_session
session = Signon.AuthSession.new(identity, method)
TypeError: Argument 1 does not allow None as a value

Leo Arias (elopio) wrote :

Yes, that's python3-requests-oauthlib and python3-dbusmock.
The last error sounds familiar. Like when the oauth method was not available in online accounts. I'm now caught with the hack fest, so I can't try it atm.

Francis Ginther (fginther) wrote :

Given the need to install additional dependencies before running these tests with phablet-test-run or executing them with adt-run, this MP is clearly not sufficient.

review: Needs Fixing
Nicholas Skaggs (nskaggs) wrote :

I'll note I'm working on migrating reminders to adt run (it was the last one). With adt-run this will work fine as depends are properly handled. The old way of running didn't/doesn't allow for depends, thus the failures.

https://code.launchpad.net/~nskaggs/reminders-app/autopkgtest/+merge/226894

Francis Ginther (fginther) wrote :

I've added some very hackish support for running autopkgtests. There are a number of hacks in here that should be fixed before merging. But I wanted to push something now that it's actually usable.

review: Needs Fixing
Nicholas Skaggs (nskaggs) wrote :

@Francis, if you wish, try this branch lp:~nskaggs/reminders-app/workaround-1347905, and confirm if indeed the tests will now run in jenkins.

It disables the tests that fail on the device, and fixes the hanging and launching issues.

Nicholas Skaggs (nskaggs) wrote :

Just updating to say this should be good to go. Trunk should work properly on the device.

review: Approve
Francis Ginther (fginther) wrote :

I successfully tested with nskaggs's branch mentioned above. Got a fully passing reminders test.

Now I just need to clean this up.

Francis Ginther (fginther) wrote :

I've cleaned up based on my earlier comments or decided to see if it's good 'nuf.

Francis Ginther (fginther) wrote :

This is much reworked since the original, marking with a resubmit.

review: Resubmit
Paul Larson (pwlars) wrote :

I don't know too much about the adb backend for adt-run. Maybe one of you can shed some light on this. But I'm guessing maybe it needs to know which adb device it's connecting to? Does it let you specify the serial somehow?
+ adt-run -o /var/lib/jenkins/slaves/mako-10/workspace/plars-smoke-daily-test/clientlogs/reminders --click-source=/var/lib/jenkins/slaves/mako-10/workspace/plars-smoke-daily-test/clientlogs/source_dir --click=com.ubuntu.reminders --- ssh -s /usr/share/autopkgtest/ssh-setup/adb
adt-run [23:29:27]: version 3.2.3-0~795~ubuntu14.04.1
Waiting for device ADB to appear...

Configuring Ubuntu phone for testing...

Preparing Ubuntu phone for running tests...
Waiting for desktop to boot

<VirtSubproc>: failure: ['/usr/share/autopkgtest/ssh-setup/adb', 'open'] failed (exit status 255)
adt-run [23:30:32]: ERROR: testbed failed: cannot send to testbed: ['BrokenPipeError: [Errno 32] Broken pipe\n']
+ adt_ret=16
+ echo adt-run returned: 16
adt-run returned: 16
+ [ 16 -eq 0 ]
+ [ 16 -eq 2 ]
+ cat /var/lib/jenkins/slaves/mako-10/workspace/plars-smoke-daily-test/clientlogs/reminders/log
+ adt_log=adt-run [23:29:27]: version 3.2.3-0~795~ubuntu14.04.1
Waiting for device ADB to appear...
Configuring Ubuntu phone for testing...
Preparing Ubuntu phone for running tests...
Waiting for desktop to boot
<VirtSubproc>: failure: ['/usr/share/autopkgtest/ssh-setup/adb', 'open'] failed (exit status 255)
adt-run [23:30:32]: ERROR: testbed failed: cannot send to testbed: ['BrokenPipeError: [Errno 32] Broken pipe\n']
+ echo <testsuite errors="1" failures="0" tests="1"><testcase classname="reminders" name="adt-run"><failure type="adt-run.failure">adt-run encountered at least one failed test: <![CDATA[\nadt-run [23:29:27]: version 3.2.3-0~795~ubuntu14.04.1
Waiting for device ADB to appear...
Configuring Ubuntu phone for testing...
Preparing Ubuntu phone for running tests...
Waiting for desktop to boot
<VirtSubproc>: failure: ['/usr/share/autopkgtest/ssh-setup/adb', 'open'] failed (exit status 255)
adt-run [23:30:32]: ERROR: testbed failed: cannot send to testbed: ['BrokenPipeError: [Errno 32] Broken pipe\n']\n\n]]></failure></testcase></testsuite>

review: Needs Fixing
Nicholas Skaggs (nskaggs) wrote :

@plars, yes you can pass the serial with --serial or -s. Here's an example of me doing it with calendar:

adt-run calendar-app/ --click=com.ubuntu.calendar --- ssh -s adb -- -s 0720a62b

Note, a couple things. You can shorten "ssh -s /usr/share/autopkgtest/ssh-setup/adb" to simple "ssh -s adb". Secondly, you need the -- in order to pass the -s argument to adb and not ssh.

Nicholas Skaggs (nskaggs) wrote :

We discovered a bug with adt and how it handles --serial arguments improperly inside the adb script. This has been fixed in autopkgtest 3.3.2git1 which is in utopic. Sadly, errors still persist.

http://paste.ubuntu.com/7980971/

You can see the ssh connection fails:

+ cleanup
+ adb -s 00693fd555c9186a shell pkill powerd-cli
<VirtSubproc>: failure: ['/var/lib/jenkins/slaves/mako-12/workspace/plars-smoke-daily-test/add-reminders/scripts/adb-runner-debug', 'open', '-s', '00693fd555c9186a'] failed (exit status 255)
adt-run [12:14:39]: ERROR: testbed failed: cannot send to testbed: ['BrokenPipeError: [Errno 32] Broken pipe\n']\n\n]]></failure></testcase></testsuite>

Nicholas Skaggs (nskaggs) wrote :

@pitii, can you review the log above and give your thoughts on what is still broken here?

Martin Pitt (pitti) wrote :

The pastebin seems to use a local version of the setup script: /var/lib/jenkins/slaves/mako-12/workspace/plars-smoke-daily-test/add-reminders/scripts/adb-runner-debug . However, that looks recent (i. e. updated to the latest autopkgtest).

I think the setup script is fine now. My hunch is that the "exit status 255" comes from a failure of ssh itself. We noticed that in our current jenkins autopkgtest setup as well, it needs to get passed one or two extra -t to allocate a terminal, otherwise ssh fails with that error. But that would be easier to see with "--- ssh --debug -s ...", i. e. enable debugging output for adt-virt-ssh. I don't see that locally presumably because running it from a terminal always has a proper tty for ssh, but that's not the case when running it in jenkins. I'll try to reproduce this locally and report back here.

In the meantime I have some inline comments about the changes here. Also, in that log there are a lot of packages installed with "apt-get install" on the device. Many (python 2 bits) aren't needed at all, and this also requires the device to be read-write. adt-run already does all that (in r/o mode), so could these bits be skipped with adt-run?

Martin Pitt (pitti) wrote :

D'oh, I should have read the set -x more carefully. It's ssh-keygen that failed. Fixed in http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/commit/?id=ed8a8ff

But at least I checked adt-run in "nohup" and with "batch" now, and it seems to behave. Paul/Nick, can you please put that fix into your local adb-runner-debug and re-run? Thanks!

Martin Pitt (pitti) wrote :

Wrt. the summary, I'm happy to teach adt-run about producing an XML summary file as well. Please file a bug report, with a sketch of how such a file should look like.

Nicholas Skaggs (nskaggs) wrote :

Paul, I've updated lp:~nskaggs/ubuntu-test-cases/add-reminders, rev 264. It contains the cherrypicked fix from pitti. Can you re-run?

Nicholas Skaggs (nskaggs) wrote :

The new run is complete, similar errors.

http://paste.ubuntu.com/8098830/

Can you have a look pitti?

Martin Pitt (pitti) wrote :

No, it's a different issue now:

+ echo ~/.ssh/id_rsa.pub or ~/.ssh/id_rsa do not exist. Please generate a key

The current script assumes that you run this with an existing key. If there's none, it could certainly generate one, or generate a private one somewhere, I just didn't think of that. If that's a requirement I can look into that.

Otherwise, would it be possible to generate one there now, to see how much further it gets?

Nicholas Skaggs (nskaggs) wrote :

Looks like the run with a pre-generated key works:

http://paste.ubuntu.com/8099616/

Nicholas Skaggs (nskaggs) wrote :

Francis, I believe you need to add just one of the tweaks from the branch; the one passing the serial along:

http://bazaar.launchpad.net/~nskaggs/ubuntu-test-cases/add-reminders/revision/258

can you update the mp?

review: Needs Fixing
Martin Pitt (pitti) wrote :

For the record, Nick and I now figured out how to invoke autopilot usefully with creating a subunit stream: http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/commit/?id=f8ebaa

This will land in <output-dir>/artifacts/<testname>.subunit . This should then be published on Jenkins as an artifact?

I'll upload that as autopkgtest 3.3.3 now, will be in utopic tomorrow morning.

Unmerged revisions

257. By Francis Ginther on 2014-07-31

Revert ubuntu-calculator-app and filemanager back to ptr tests, make option handling more robust and do some cleanup.

256. By Francis Ginther on 2014-07-18

Add primitive support for running autopkgtests with run-autopilot-tests.sh. This includes modification of ubuntu-calculator-app and filemanager as examples.

255. By Francis Ginther on 2014-07-18

Create an AutopkgTest test classification and begin infrastructure to handle it.

254. By Francis Ginther on 2014-07-10

Add testing for reminders click app.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'jenkins/testconfig.py'
2--- jenkins/testconfig.py 2014-07-25 22:15:49 +0000
3+++ jenkins/testconfig.py 2014-07-31 21:33:54 +0000
4@@ -45,6 +45,18 @@
5 self.app = app
6
7
8+class AutopkgTest(Test):
9+ def __init__(self, name, branch, click):
10+ Test.__init__(self, name)
11+ # In all known cases, the app and name are equivalent
12+ self.app = name
13+ # Branch is needed until autpkgtest supports downloading
14+ # from the branch in the manifest file
15+ self.branch = branch
16+ # This is the click name as returned by 'click list'
17+ self.click = click
18+
19+
20 TESTSUITES = [
21 Test('default'),
22 ]
23@@ -77,6 +89,10 @@
24 ['ubuntu-system-settings-online-accounts-autopilot']),
25 ]
26 TESTSUITES += [
27+ AutopkgTest('reminders', 'lp:reminders-app',
28+ 'com.ubuntu.reminders'),
29+]
30+TESTSUITES += [
31 Test('click_image_tests'),
32 Test('sdk'),
33 Test('security'),
34@@ -135,6 +151,7 @@
35
36 def _handle_ap_apps(args):
37 tests = _get_tests(APTest, args.image_type)
38+ tests += _get_tests(AutopkgTest, args.image_type)
39 tests = _split_work(tests, args.total_workers, args.worker)
40 print(' '.join([t.app for t in tests]))
41
42@@ -149,6 +166,18 @@
43 print(' '.join(pkgs))
44
45
46+def _handle_autopkg_apps(args):
47+ tests = _get_tests(AutopkgTest, args.image_type)
48+ for test in tests:
49+ if test.app == args.branch:
50+ print(test.branch)
51+ return
52+ elif test.app == args.click:
53+ print(test.click)
54+ return
55+ print(' '.join([t.app for t in tests]))
56+
57+
58 def get_tests_mako(common_tests):
59 tests = common_tests
60 tests.extend([DevTest('suspend-blocker', 'mako-01')])
61@@ -188,6 +217,16 @@
62 p.add_argument('-w', '--worker', type=int, default=0,
63 help='The worker to allocate applications for testing to.')
64
65+ p = sub.add_parser('adt', help='List autopkgtest application names')
66+ p.set_defaults(func=_handle_autopkg_apps)
67+ g = p.add_mutually_exclusive_group()
68+ g.add_argument('-i', '--image-type',
69+ help='Return list of test configured for an image type.')
70+ g.add_argument('-b', '--branch',
71+ help='Return the branch needed for an autopkgtest app.')
72+ g.add_argument('-c', '--click',
73+ help='Return the click name for an autopkgtest app.')
74+
75 p = sub.add_parser('packages', help='List packages required for autopilot')
76 p.set_defaults(func=_handle_ap_packages)
77 g = p.add_mutually_exclusive_group()
78
79=== modified file 'scripts/run-autopilot-tests.sh'
80--- scripts/run-autopilot-tests.sh 2014-07-01 21:31:23 +0000
81+++ scripts/run-autopilot-tests.sh 2014-07-31 21:33:54 +0000
82@@ -76,7 +76,37 @@
83 [ -d $odir ] && rm -rf $odir
84 mkdir -p $odir || return 1
85
86+ # The source_dir is needed for adt-run and click packages
87+ source_dir=${RESDIR}/source_dir
88+ [ -d $source_dir ] && rm -rf $source_dir
89+
90 system_settle before $odir
91+
92+ adt_list=$(${BASEDIR}/jenkins/testconfig.py adt)
93+ if echo "$adt_list" | grep "$app" ; then
94+ echo "$app is executed with adt-run"
95+ test_app_adt $source_dir
96+ else
97+ echo "$app is executed with phablet-test-run"
98+ test_app_ptr
99+ fi
100+
101+ system_settle after $odir
102+
103+ if echo "$adt_list" | grep "$app" ; then
104+ echo "Clean up for adt tests"
105+ [ -d $source_dir ] && rm -rf $source_dir
106+ else
107+ echo "Clean up for ptr tests"
108+ setup_test $app teardown $odir
109+ fi
110+ if [ -f ${odir}/test_results.subunit ] ; then
111+ cat ${odir}/test_results.subunit | subunit2junitxml > ${odir}/test_results.xml
112+ fi
113+ ${BASEDIR}/scripts/combine_results ${odir}
114+}
115+
116+test_app_ptr() {
117 phablet-config autopilot --dbus-probe enable || \
118 (log_error "'autopilot dbus-probe enable' failed"; return 1)
119
120@@ -94,13 +124,40 @@
121 -a /var/crash -a /home/phablet/.cache/upstart \
122 -a /var/log/syslog -a /var/log/kern.log \
123 -v $app || true
124+}
125
126- system_settle after $odir
127- setup_test $app teardown $odir
128- if [ -f ${odir}/test_results.subunit ] ; then
129- cat ${odir}/test_results.subunit | subunit2junitxml > ${odir}/test_results.xml
130+test_app_adt() {
131+ # Here's the basic command for running adt tests for a click package.
132+ # Due to limitation in adt-run, the source dir has to be manually
133+ # checked out.
134+ # adt-run -o reminders.out --click-source=reminders-app \
135+ # --click=com.ubuntu.reminders --- \
136+ # ssh -s /usr/share/autopkgtest/ssh-setup/adb
137+ click_source_dir=$1
138+ click_name=$(${BASEDIR}/jenkins/testconfig.py adt --click $app)
139+ # TODO: check out the correct revno based on the click manifest
140+ click_source=$(${BASEDIR}/jenkins/testconfig.py adt --branch $app)
141+ bzr branch $click_source $click_source_dir
142+ adt-run -o ${odir} --click-source=$click_source_dir \
143+ --click=$click_name --- \
144+ ssh -s /usr/share/autopkgtest/ssh-setup/adb
145+ adt_ret=$?
146+ echo "adt-run returned: $adt_ret"
147+ # adt-run does not produce any pretty xml results.
148+ # As a workaround, construct an xml file that can at least be pulled
149+ # into the dashboard and contains data from any test failures.
150+ if [ $adt_ret -eq 0 ] || [ $adt_ret -eq 2 ] ; then
151+ # adt-run passed!
152+ # non-failure exit status:
153+ # 0 == all tests passed
154+ # 2 == at least one test skipped
155+ echo "<testsuite errors=\"0\" failures=\"0\" tests=\"1\"><testcase classname=\"$app\" name=\"adt-run\"></testcase></testsuite>" >> ${odir}/test_results.xml
156+ else
157+ # The adt run failed. The log file will contain the logging
158+ # from the test failures as well as any adt-run failures.
159+ adt_log=`cat ${odir}/log`
160+ echo "<testsuite errors=\"1\" failures=\"0\" tests=\"1\"><testcase classname=\"$app\" name=\"adt-run\"><failure type=\"adt-run.failure\">adt-run encountered at least one failed test: <![CDATA[\n$adt_log\n\n]]></failure></testcase></testsuite>" >> ${odir}/test_results.xml
161 fi
162- ${BASEDIR}/scripts/combine_results ${odir}
163 }
164
165 reboot_wait() {

Subscribers

People subscribed via source and target branches