Merge lp:~fginther/ubuntu-test-cases/add-reminders into lp:ubuntu-test-cases/touch

[Francis Ginther]

Add testing for reminders click app.

Add testing for reminders click app.

This requires the utopic version of autopkgtest or from ppa:canonical-ci-engineering/ci-airline-phase-0

[Andy Doan (doanac)]

please merge by hand. we have no tarmac landing here

review: Approve

[Paul Larson (pwlars)]

Did you try running this locally or anything? Looks right, but last I had known reminders wasn't added yet. I always do a quick run-through by hand to make sure there are no surprises.

[Paul Larson (pwlars)]

I'm seeing 7 tests, 4 pass, 3 fail. Does this match with what they were expecting?

[Leo Arias (elopio)]

Paul, are you running the tests on the phone freshly flashed? They should all pass.
If you are running them on the desktop without your keyring unlocked, you will get errors on the ones that require credentials. They will fail also if you already have an evernote account.
Those details are on TODO.
I'm about to go to bed, but tomorrow I can take a look at your errors.

[Paul Larson (pwlars)]

No, they are on a freshly flashed phone with no other tests running

[Paul Larson (pwlars)]

Does reminders-app have any .deb dependencies that need to be installed?

[Leo Arias (elopio)]

Yes, they need evernote-sdk-python3 that's on the applications PPA.
I couldn't give them a try on friday, I'm sorry. Can you paste the output
of your failures somewhere?

[Paul Larson (pwlars)]

Ok, it turns out this is all due to a bug. phablet-test-run is trying to import reminders to decide if it's python2 or python3, and due to the deprecation warning from uitk, it decides to run python2, which is not even installed. xnox has a fix for this in phablet-tools and is going to get it landed. I suspect there are other tests like this too, and we were just getting lucky enough with them to run them after something that depended on python2 and installed it.

review: Approve

[Paul Larson (pwlars)]

Wait, do I understand correctly that for these tests to pass, we would need to install a package from some other ppa? That's a bit of a problem since clicks don't have dependencies. We could hack this in somehow, but it's *really* not ideal. Having it in the image, or at least in the source tree would probably be ideal. Otherwise, if we need to hack around it and force the install of a deb before running tests on a click package, we'll at least need it in the archives, not a ppa.

[Leo Arias (elopio)]

We need to involve balloons and dpm in this discussion. On option was to copy the evernote python module to the package, but I'm not sure if their license allows us to do it. And the alternative is to make our own python module to communicate with their API, less than ideal because we would need to maintain it.

[Leo Arias (elopio)]

Sorry, scratch all I said. The thrift and evernote modules now are copied to the same reminders python package.

[David Planella (dpm)]

We're already including the Python Evernote SDK in the sources, and they're available in the reminders-app-autopilot binary generated from the reminders-app source package [1] in the utopic archive.

If you need this to be included in the click package instead, I guess it shouldn't be much of a problem.

[1] https://launchpad.net/ubuntu/+source/reminders-app

[Paul Larson (pwlars)]

Perhaps there are other dependencies still needed then? Running it with phablet-test-run locally, I get.

http://paste.ubuntu.com/7799766/
...
File "/home/phablet/autopilot/evernote/api/client.py", line 5, in <module>
from requests_oauthlib import OAuth1Session
ImportError: No module named 'requests_oauthlib'

If I manually install python-requests-oauthlib, I can get a bit further, but I still get:
File "/home/phablet/autopilot/reminders/fake_services.py", line 22, in <module>
import dbusmock
ImportError: No module named 'dbusmock'

I can install python3-dbusmock, but it pulls in the notorious dbus-x11 that's known to cause problems in the past. Even if I install that manually and try again though, I still get:
File "/home/phablet/autopilot/reminders/credentials.py", line 133, in _process_session
session = Signon.AuthSession.new(identity, method)
TypeError: Argument 1 does not allow None as a value

[Leo Arias (elopio)]

Yes, that's python3-requests-oauthlib and python3-dbusmock.
The last error sounds familiar. Like when the oauth method was not available in online accounts. I'm now caught with the hack fest, so I can't try it atm.

[Francis Ginther (fginther)]

Given the need to install additional dependencies before running these tests with phablet-test-run or executing them with adt-run, this MP is clearly not sufficient.

review: Needs Fixing

[Nicholas Skaggs (nskaggs)]

I'll note I'm working on migrating reminders to adt run (it was the last one). With adt-run this will work fine as depends are properly handled. The old way of running didn't/doesn't allow for depends, thus the failures.

https://code.launchpad.net/~nskaggs/reminders-app/autopkgtest/+merge/226894

[Francis Ginther (fginther)]

I've added some very hackish support for running autopkgtests. There are a number of hacks in here that should be fixed before merging. But I wanted to push something now that it's actually usable.

review: Needs Fixing

[Nicholas Skaggs (nskaggs)]

@Francis, if you wish, try this branch lp:~nskaggs/reminders-app/workaround-1347905, and confirm if indeed the tests will now run in jenkins.

It disables the tests that fail on the device, and fixes the hanging and launching issues.

[Nicholas Skaggs (nskaggs)]

Just updating to say this should be good to go. Trunk should work properly on the device.

review: Approve

[Francis Ginther (fginther)]

I successfully tested with nskaggs's branch mentioned above. Got a fully passing reminders test.

Now I just need to clean this up.

[Francis Ginther (fginther)]

I've cleaned up based on my earlier comments or decided to see if it's good 'nuf.

[Francis Ginther (fginther)]

This is much reworked since the original, marking with a resubmit.

review: Needs Resubmitting

[Paul Larson (pwlars)]

I don't know too much about the adb backend for adt-run. Maybe one of you can shed some light on this. But I'm guessing maybe it needs to know which adb device it's connecting to? Does it let you specify the serial somehow?
+ adt-run -o /var/lib/jenkins/slaves/mako-10/workspace/plars-smoke-daily-test/clientlogs/reminders --click-source=/var/lib/jenkins/slaves/mako-10/workspace/plars-smoke-daily-test/clientlogs/source_dir --click=com.ubuntu.reminders --- ssh -s /usr/share/autopkgtest/ssh-setup/adb
adt-run [23:29:27]: version 3.2.3-0~795~ubuntu14.04.1
Waiting for device ADB to appear...

Configuring Ubuntu phone for testing...

Preparing Ubuntu phone for running tests...
Waiting for desktop to boot

<VirtSubproc>: failure: ['/usr/share/autopkgtest/ssh-setup/adb', 'open'] failed (exit status 255)
adt-run [23:30:32]: ERROR: testbed failed: cannot send to testbed: ['BrokenPipeError: [Errno 32] Broken pipe\n']
+ adt_ret=16
+ echo adt-run returned: 16
adt-run returned: 16
+ [ 16 -eq 0 ]
+ [ 16 -eq 2 ]
+ cat /var/lib/jenkins/slaves/mako-10/workspace/plars-smoke-daily-test/clientlogs/reminders/log
+ adt_log=adt-run [23:29:27]: version 3.2.3-0~795~ubuntu14.04.1
Waiting for device ADB to appear...
Configuring Ubuntu phone for testing...
Preparing Ubuntu phone for running tests...
Waiting for desktop to boot
<VirtSubproc>: failure: ['/usr/share/autopkgtest/ssh-setup/adb', 'open'] failed (exit status 255)
adt-run [23:30:32]: ERROR: testbed failed: cannot send to testbed: ['BrokenPipeError: [Errno 32] Broken pipe\n']
+ echo <testsuite errors="1" failures="0" tests="1"><testcase classname="reminders" name="adt-run"><failure type="adt-run.failure">adt-run encountered at least one failed test: <![CDATA[\nadt-run [23:29:27]: version 3.2.3-0~795~ubuntu14.04.1
Waiting for device ADB to appear...
Configuring Ubuntu phone for testing...
Preparing Ubuntu phone for running tests...
Waiting for desktop to boot
<VirtSubproc>: failure: ['/usr/share/autopkgtest/ssh-setup/adb', 'open'] failed (exit status 255)
adt-run [23:30:32]: ERROR: testbed failed: cannot send to testbed: ['BrokenPipeError: [Errno 32] Broken pipe\n']\n\n]]></failure></testcase></testsuite>

review: Needs Fixing

[Nicholas Skaggs (nskaggs)]

@plars, yes you can pass the serial with --serial or -s. Here's an example of me doing it with calendar:

adt-run calendar-app/ --click=com.ubuntu.calendar --- ssh -s adb -- -s 0720a62b

Note, a couple things. You can shorten "ssh -s /usr/share/autopkgtest/ssh-setup/adb" to simple "ssh -s adb". Secondly, you need the -- in order to pass the -s argument to adb and not ssh.

[Nicholas Skaggs (nskaggs)]

We discovered a bug with adt and how it handles --serial arguments improperly inside the adb script. This has been fixed in autopkgtest 3.3.2git1 which is in utopic. Sadly, errors still persist.

http://paste.ubuntu.com/7980971/

You can see the ssh connection fails:

+ cleanup
+ adb -s 00693fd555c9186a shell pkill powerd-cli
<VirtSubproc>: failure: ['/var/lib/jenkins/slaves/mako-12/workspace/plars-smoke-daily-test/add-reminders/scripts/adb-runner-debug', 'open', '-s', '00693fd555c9186a'] failed (exit status 255)
adt-run [12:14:39]: ERROR: testbed failed: cannot send to testbed: ['BrokenPipeError: [Errno 32] Broken pipe\n']\n\n]]></failure></testcase></testsuite>

[Nicholas Skaggs (nskaggs)]

@pitii, can you review the log above and give your thoughts on what is still broken here?

[Martin Pitt (pitti)]

The pastebin seems to use a local version of the setup script: /var/lib/jenkins/slaves/mako-12/workspace/plars-smoke-daily-test/add-reminders/scripts/adb-runner-debug . However, that looks recent (i. e. updated to the latest autopkgtest).

I think the setup script is fine now. My hunch is that the "exit status 255" comes from a failure of ssh itself. We noticed that in our current jenkins autopkgtest setup as well, it needs to get passed one or two extra -t to allocate a terminal, otherwise ssh fails with that error. But that would be easier to see with "--- ssh --debug -s ...", i. e. enable debugging output for adt-virt-ssh. I don't see that locally presumably because running it from a terminal always has a proper tty for ssh, but that's not the case when running it in jenkins. I'll try to reproduce this locally and report back here.

In the meantime I have some inline comments about the changes here. Also, in that log there are a lot of packages installed with "apt-get install" on the device. Many (python 2 bits) aren't needed at all, and this also requires the device to be read-write. adt-run already does all that (in r/o mode), so could these bits be skipped with adt-run?

[Martin Pitt (pitti)]

D'oh, I should have read the set -x more carefully. It's ssh-keygen that failed. Fixed in http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/commit/?id=ed8a8ff

But at least I checked adt-run in "nohup" and with "batch" now, and it seems to behave. Paul/Nick, can you please put that fix into your local adb-runner-debug and re-run? Thanks!

[Martin Pitt (pitti)]

Wrt. the summary, I'm happy to teach adt-run about producing an XML summary file as well. Please file a bug report, with a sketch of how such a file should look like.

[Nicholas Skaggs (nskaggs)]

Paul, I've updated lp:~nskaggs/ubuntu-test-cases/add-reminders, rev 264. It contains the cherrypicked fix from pitti. Can you re-run?

[Nicholas Skaggs (nskaggs)]

The new run is complete, similar errors.

http://paste.ubuntu.com/8098830/

Can you have a look pitti?

[Martin Pitt (pitti)]

No, it's a different issue now:

+ echo ~/.ssh/id_rsa.pub or ~/.ssh/id_rsa do not exist. Please generate a key

The current script assumes that you run this with an existing key. If there's none, it could certainly generate one, or generate a private one somewhere, I just didn't think of that. If that's a requirement I can look into that.

Otherwise, would it be possible to generate one there now, to see how much further it gets?

[Martin Pitt (pitti)]

Automatic key generation is now done: http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/commit/?id=40981e13c

[Nicholas Skaggs (nskaggs)]

Looks like the run with a pre-generated key works:

http://paste.ubuntu.com/8099616/

[Nicholas Skaggs (nskaggs)]

Francis, I believe you need to add just one of the tweaks from the branch; the one passing the serial along:

http://bazaar.launchpad.net/~nskaggs/ubuntu-test-cases/add-reminders/revision/258

can you update the mp?

review: Needs Fixing

[Martin Pitt (pitti)]

For the record, Nick and I now figured out how to invoke autopilot usefully with creating a subunit stream: http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/commit/?id=f8ebaa

This will land in <output-dir>/artifacts/<testname>.subunit . This should then be published on Jenkins as an artifact?

I'll upload that as autopkgtest 3.3.3 now, will be in utopic tomorrow morning.

Unmerged revisions

257. By Francis Ginther on 2014-07-31

Revert ubuntu-calculator-app and filemanager back to ptr tests, make option handling more robust and do some cleanup.

256. By Francis Ginther on 2014-07-18

Add primitive support for running autopkgtests with run-autopilot-tests.sh. This includes modification of ubuntu-calculator-app and filemanager as examples.

255. By Francis Ginther on 2014-07-18

Create an AutopkgTest test classification and begin infrastructure to handle it.

254. By Francis Ginther on 2014-07-10

Add testing for reminders click app.

=== modified file 'jenkins/testconfig.py'
--- jenkins/testconfig.py	2014-07-25 22:15:49 +0000
+++ jenkins/testconfig.py	2014-07-31 21:33:54 +0000
@@ -45,6 +45,18 @@
         self.app = app
 
 
+class AutopkgTest(Test):
+    def __init__(self, name, branch, click):
+        Test.__init__(self, name)
+        # In all known cases, the app and name are equivalent
+        self.app = name
+        # Branch is needed until autpkgtest supports downloading
+        # from the branch in the manifest file
+        self.branch = branch
+        # This is the click name as returned by 'click list'
+        self.click = click
+
+
 TESTSUITES = [
     Test('default'),
 ]
@@ -77,6 +89,10 @@
            ['ubuntu-system-settings-online-accounts-autopilot']),
 ]
 TESTSUITES += [
+    AutopkgTest('reminders', 'lp:reminders-app',
+                'com.ubuntu.reminders'),
+]
+TESTSUITES += [
     Test('click_image_tests'),
     Test('sdk'),
     Test('security'),
@@ -135,6 +151,7 @@
 
 def _handle_ap_apps(args):
     tests = _get_tests(APTest, args.image_type)
+    tests += _get_tests(AutopkgTest, args.image_type)
     tests = _split_work(tests, args.total_workers, args.worker)
     print(' '.join([t.app for t in tests]))
 
@@ -149,6 +166,18 @@
     print(' '.join(pkgs))
 
 
+def _handle_autopkg_apps(args):
+    tests = _get_tests(AutopkgTest, args.image_type)
+    for test in tests:
+        if test.app == args.branch:
+            print(test.branch)
+            return
+        elif test.app == args.click:
+            print(test.click)
+            return
+    print(' '.join([t.app for t in tests]))
+
+
 def get_tests_mako(common_tests):
     tests = common_tests
     tests.extend([DevTest('suspend-blocker', 'mako-01')])
@@ -188,6 +217,16 @@
     p.add_argument('-w', '--worker', type=int, default=0,
                    help='The worker to allocate applications for testing to.')
 
+    p = sub.add_parser('adt', help='List autopkgtest application names')
+    p.set_defaults(func=_handle_autopkg_apps)
+    g = p.add_mutually_exclusive_group()
+    g.add_argument('-i', '--image-type',
+                   help='Return list of test configured for an image type.')
+    g.add_argument('-b', '--branch',
+                   help='Return the branch needed for an autopkgtest app.')
+    g.add_argument('-c', '--click',
+                   help='Return the click name for an autopkgtest app.')
+
     p = sub.add_parser('packages', help='List packages required for autopilot')
     p.set_defaults(func=_handle_ap_packages)
     g = p.add_mutually_exclusive_group()

=== modified file 'scripts/run-autopilot-tests.sh'
--- scripts/run-autopilot-tests.sh	2014-07-01 21:31:23 +0000
+++ scripts/run-autopilot-tests.sh	2014-07-31 21:33:54 +0000
@@ -76,7 +76,37 @@
 	[ -d $odir ] && rm -rf $odir
 	mkdir -p $odir || return 1
 
+	# The source_dir is needed for adt-run and click packages
+	source_dir=${RESDIR}/source_dir
+	[ -d $source_dir ] && rm -rf $source_dir
+
 	system_settle before $odir
+
+	adt_list=$(${BASEDIR}/jenkins/testconfig.py adt)
+	if echo "$adt_list" | grep "$app" ; then
+		echo "$app is executed with adt-run"
+		test_app_adt $source_dir
+	else
+		echo "$app is executed with phablet-test-run"
+		test_app_ptr
+	fi
+
+	system_settle after $odir
+
+	if echo "$adt_list" | grep "$app" ; then
+		echo "Clean up for adt tests"
+		[ -d $source_dir ] && rm -rf $source_dir
+	else
+		echo "Clean up for ptr tests"
+		setup_test $app teardown $odir
+	fi
+	if [ -f ${odir}/test_results.subunit ] ; then
+		cat ${odir}/test_results.subunit | subunit2junitxml > ${odir}/test_results.xml
+	fi
+	${BASEDIR}/scripts/combine_results ${odir}
+}
+
+test_app_ptr() {
 	phablet-config autopilot --dbus-probe enable || \
 		(log_error "'autopilot dbus-probe enable' failed"; return 1)
 
@@ -94,13 +124,40 @@
 		-a /var/crash -a /home/phablet/.cache/upstart \
                 -a /var/log/syslog -a /var/log/kern.log \
 		-v $app || true
+}
 
-	system_settle after $odir
-	setup_test $app teardown $odir
-	if [ -f ${odir}/test_results.subunit ] ; then
-		cat ${odir}/test_results.subunit | subunit2junitxml > ${odir}/test_results.xml
+test_app_adt() {
+	# Here's the basic command for running adt tests for a click package.
+	# Due to limitation in adt-run, the source dir has to be manually
+	# checked out.
+	# adt-run -o reminders.out --click-source=reminders-app \
+	#     --click=com.ubuntu.reminders --- \
+	#     ssh -s /usr/share/autopkgtest/ssh-setup/adb
+	click_source_dir=$1
+	click_name=$(${BASEDIR}/jenkins/testconfig.py adt --click $app)
+	# TODO: check out the correct revno based on the click manifest
+	click_source=$(${BASEDIR}/jenkins/testconfig.py adt --branch $app)
+	bzr branch $click_source $click_source_dir
+	adt-run -o ${odir} --click-source=$click_source_dir \
+		--click=$click_name --- \
+		ssh -s /usr/share/autopkgtest/ssh-setup/adb
+	adt_ret=$?
+	echo "adt-run returned: $adt_ret"
+	# adt-run does not produce any pretty xml results.
+	# As a workaround, construct an xml file that can at least be pulled
+	# into the dashboard and contains data from any test failures.
+	if [ $adt_ret -eq 0 ] || [ $adt_ret -eq 2 ] ; then
+		# adt-run passed!
+		# non-failure exit status:
+		#   0 == all tests passed
+		#   2 == at least one test skipped
+		echo "<testsuite errors=\"0\" failures=\"0\" tests=\"1\"><testcase classname=\"$app\" name=\"adt-run\"></testcase></testsuite>" >> ${odir}/test_results.xml
+	else
+		# The adt run failed. The log file will contain the logging
+		# from the test failures as well as any adt-run failures.
+		adt_log=`cat ${odir}/log`
+		echo "<testsuite errors=\"1\" failures=\"0\" tests=\"1\"><testcase classname=\"$app\" name=\"adt-run\"><failure type=\"adt-run.failure\">adt-run encountered at least one failed test: <![CDATA[\n$adt_log\n\n]]></failure></testcase></testsuite>" >> ${odir}/test_results.xml
 	fi
-	${BASEDIR}/scripts/combine_results ${odir}
 }
 
 reboot_wait() {