Merge lp:~canonical-platform-qa/lrt/lrt-to-dep8 into lp:lrt

[Allan LeSage]

Enable LRT as an autopkgtest.

Enable LRT as an autopkgtest.

[Allan LeSage (allanlesage)]

Performed a test run of this for kicks, and witnessed a failure. Note that I made a point of not changing any of ChrisGagnon's code as it's been working for us, however it could probably use a subsequent review or two.

[lp:~canonical-platform-qa/lrt/lrt-to-dep8]

96. By Allan LeSage on 2014-11-13

Restore test_switch.py, not quite ready for it.

[Christopher Lee (veebers)]

A couple of comments and questions.

review: Needs Fixing

[lp:~canonical-platform-qa/lrt/lrt-to-dep8]

97. By Allan LeSage on 2014-11-13

Per veebers' suggestions.

[Christopher Lee (veebers)]

1 remaining question regarding return code, otherwise it looks good. Once there is an answer to the retcode question I would be more than happy approving this.

review: Needs Information

[Martin Pitt (pitti)]

The whole sudo thing is really really ugly and bad. Please try to avoid it entirely, and the current implementation is definitively not working without special pre-arrangements to the testbed (like allowing sudo without password). I made some comments about the usage of sudo, and some alternatives.

[Martin Pitt (pitti)]

review: Needs Fixing

[Allan LeSage (allanlesage)]

Jeez I guess I didn't realize exactly how evil these sudos really were :) , in my defense we're just using what works from Chris' local setup--I'll do a test without and then ask for another review Martin, thanks.

[lp:~canonical-platform-qa/lrt/lrt-to-dep8]

98. By Allan LeSage on 2014-11-13

De-sudo-fication, having tested.

[Martin Pitt (pitti)]

/usr/share/polkit-1/actions/org.freedesktop.urfkill.policy limits the D-BUS calls to an active local session. Thus the call fails in phablet-shell:

$ /usr/share/urfkill/scripts/flight-mode 1
dbus.exceptions.DBusException: org.freedesktop.URfkill.Daemon.Error.General: not authorized

But it might behave differently in the actual test under adt-run, so if it works, perfect! Otherwise it's a little more tricky; we can't just "break into" the running session from outside and pretend the external ssh session would run in that; but we might be able to talk to the network indicator over (session) D-BUS to trigger flight mode, which then talks to urfkill?

[Allan LeSage (allanlesage)]

Martin's right, the flight mode setting doesn't work without root; will call the d-bus action instead.

=== added file 'PROVISIONING_IN_CI_README.sh'
--- PROVISIONING_IN_CI_README.sh	1970-01-01 00:00:00 +0000
+++ PROVISIONING_IN_CI_README.sh	2014-11-13 15:27:18 +0000
@@ -0,0 +1,10 @@
+#!/bin/sh
+
+# XXX 20141112 alesage
+# TO DISCUSS WITH CI: we need to report crashes via whoopsie; as noted
+# in dep-8-run-lrt.sh ChrisGagnon has secured special CRASH_IDs for
+# the defined tests.  ChrisGagnon modified the whoopsie.conf during
+# provisioning to reference this $CRASH_ID as follows:
+
+sed -i '/CRASH_DB_IDENTIFIER/d' /etc/init/whoopsie.conf
+sed -i '/env CRASH_DB_URL=https:\/\/daisy.ubuntu.com/a env CRASH_DB_IDENTIFIER=$CRASH_ID' /etc/init/whoopsie.conf

=== modified file 'README'
--- README	2014-08-06 21:52:32 +0000
+++ README	2014-11-13 15:27:18 +0000
@@ -39,3 +39,24 @@
 The jenkins url is stored in settings
 
 python -m helpers_py2.post_process_jenkins_jobs -p LRT -j generic-monkey-runner -t RandomGesture -i 20140805.3 -b 13000 |  python3 /insert_lrt_in_nfss.py LRT RandomGesture
+
+
+Running the tests on a Touch device under autopkgtest
+=====================================================
+
+This test should be run on an Ubuntu Touch device.  To run the test:
+
+1. Flash an Ubuntu Touch device.
+2. Enable adb access on the device.
+3. Get the test code:
+
+    $ bzr branch lp:lrt
+
+4. Connect the device to the runner machine with an USB cable.
+5. Run the tests with adt-run:
+
+    $ adt-run -B --built-tree=. --output-dir=output --- ssh -s adb
+
+The test executable exits with 0 if successful, non-zero otherwise.
+
+The NFSS json data file will be stored at the output/artifacts/ directory, along with a file that indicates that it should be uploaded only to the staging server.

=== added directory 'debian'
=== added file 'debian/changelog'
--- debian/changelog	1970-01-01 00:00:00 +0000
+++ debian/changelog	2014-11-13 15:27:18 +0000
@@ -0,0 +1,5 @@
+ubuntu-lrt-tests (0.1-1) unstable; urgency=low
+
+  * Initial release
+
+ -- Allan LeSage <[e-mail]>  Wed, 05 Nov 2014 09:44:55 -0600

=== added file 'debian/compat'
--- debian/compat	1970-01-01 00:00:00 +0000
+++ debian/compat	2014-11-13 15:27:18 +0000
@@ -0,0 +1,1 @@
+9

=== added file 'debian/control'
--- debian/control	1970-01-01 00:00:00 +0000
+++ debian/control	2014-11-13 15:27:18 +0000
@@ -0,0 +1,8 @@
+Source: ubuntu-lrt-tests
+Priority: optional
+Maintainer: Chris Gagnon <[e-mail]>
+Build-Depends: debhelper (>= 9), autotools-dev
+Standards-Version: 3.9.5
+Section: utils
+Vcs-bzr: lp:lrt
+XS-Testsuite: autopkgtest

=== added file 'debian/copyright'
--- debian/copyright	1970-01-01 00:00:00 +0000
+++ debian/copyright	2014-11-13 15:27:18 +0000
@@ -0,0 +1,27 @@
+Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+Upstream-Name: ubuntu-lrt-tests
+Source: https://code.launchpad.net/lrt
+
+Files: *
+Copyright: 2014 Ubuntu Quality Team <[e-mail]>
+License: GPL-3.0+
+
+Files: debian/*
+Copyright: 2014 Ubuntu Quality Team <[e-mail]>
+License: GPL-3.0+
+
+ This package is free software; you can redistribute it and/or modify
+ it under the terms of the GNU General Public License as published by
+ the Free Software Foundation; either version 3 of the License, or
+ (at your option) any later version.
+ .
+ This package is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ GNU General Public License for more details.
+ .
+ You should have received a copy of the GNU General Public License
+ along with this program. If not, see <http://www.gnu.org/licenses/>
+ .
+ On Debian systems, the complete text of the GNU General
+ Public License version 2 can be found in "/usr/share/common-licenses/GPL-3".

=== added directory 'debian/source'
=== added file 'debian/source/format'
--- debian/source/format	1970-01-01 00:00:00 +0000
+++ debian/source/format	2014-11-13 15:27:18 +0000
@@ -0,0 +1,1 @@
+3.0 (quilt)

=== added directory 'debian/tests'
=== added file 'debian/tests/control'
--- debian/tests/control	1970-01-01 00:00:00 +0000
+++ debian/tests/control	2014-11-13 15:27:18 +0000
@@ -0,0 +1,8 @@
+Tests: lrt-random-gestures
+Depends: android-tools-adb,
+	 autopilot-touch,
+	 bzr,
+	 imagemagick,
+	 python3-subunit,
+	 python3-autopilot,
+Restrictions: needs-root

=== added file 'debian/tests/lrt-random-gestures'
--- debian/tests/lrt-random-gestures	1970-01-01 00:00:00 +0000
+++ debian/tests/lrt-random-gestures	2014-11-13 15:27:18 +0000
@@ -0,0 +1,5 @@
+#!/bin/sh
+
+./dep-8-run-lrt.sh "lrt.test_random_gestures"
+
+exit $?

=== added file 'dep-8-run-lrt.sh'
--- dep-8-run-lrt.sh	1970-01-01 00:00:00 +0000
+++ dep-8-run-lrt.sh	2014-11-13 15:27:18 +0000
@@ -0,0 +1,72 @@
+#!/bin/bash
+set -e
+
+if [ -z $1 ]; then
+    echo 'you must specify the test to run, example: lrt.test_random_gestures'
+    exit 1
+fi
+TEST_NAME=$1
+
+# ChrisGagnon has secured special CRASH_IDs for the defined tests via
+# bdmurray (NOTE that each starts with 'deadbeef').
+# XXX alesage 20141112
+# NOTE that that $CRASH_ID needs to appear in whoopsie.conf;
+# suggestions for implementation in PROVISIONING_IN_CI_README.sh.
+if [ $TEST_NAME == "lrt.test_random_gestures" ]; then
+CRASH_ID=deadbeef2b73679bb94cc77874f81c4822482114dd8bf912741713eb15bf61533b5d8176f0d870f758cc6f241445ffd3359debe1616c832a8ab3b7626d9ed369
+fi
+# NOTE: re-enable these if and when each test below is brought online.
+#if [ $TEST_NAME == "lrt.test_switch" ]; then
+#CRASH_ID=deadbeef17372a2ac165df1ed82002d7cbeeb0aad6adaa39a7a65fc29cb6c724878e3009de7787c88de01911bf874932edb8d11d7debc7506331e8fb5c99a9e3
+#fi
+#if [ $test_to_run == &quot;lrt.test_random&quot; ]; then
+#CRASH_ID=deadbeef2b73679bb94cc77874f81c4822482114dd8bf912741713eb15bf61533b5d8176f0d870f758cc6f241445ffd3359debe1616c832a8ab3b7626d9ed369
+#fi
+#if [ $test_to_run == &quot;lrt.test_switch&quot; ]; then
+#CRASH_ID=deadbeef17372a2ac165df1ed82002d7cbeeb0aad6adaa39a7a65fc29cb6c724878e3009de7787c88de01911bf874932edb8d11d7debc7506331e8fb5c99a9e3
+#fi
+#if [ $test_to_run  == &quot;lrt.test_ap_core_apps&quot; ]; then
+#CRASH_ID=deadbeef6ad56ff7b1050f384439a7900acdd7dcce06e81cb051c3417320a19baa4f1769d9fe20e9b01ac5b970d7b1a339526d6edf572d9e2eb7fa3c3232d13c6
+#fi
+#if [ $test_to_run  == &quot;lrt.test_ap_system&quot; ]; then
+#CRASH_ID=deadbeef34567890123de0f384439a7900acdd7dcce06e81cb051c3417320a19baa4f1769d9fe20e9b01cd5b970d7b1e339897d6fdf572d9e2eb7fa3c3232d14
+#fi
+#if [ $test_to_run  == &quot;lrt.test_power&quot; ]; then
+#CRASH_ID=deadbeefad56ff7b1050f384439a7900acdd7dcce06e81cb051c3417320a19baa4f1769d9fe20e9b01ac5b970d7b1a339526d6edf572d9e2eb7fa3c3232d1376
+#fi
+
+python3 -m subunit.run $TEST_NAME > ~/result.subunit || true
+chmod a+rw ~/result.subunit
+
+# Prepare to whoopsie.
+# disable flight mode (a random gesture may have enabled)
+/usr/share/urfkill/scripts/flight-mode 0
+# wait for wifi a bit
+sleep 10
+# enable wifi, the test sometimes disables it
+ping -c2 launchpad.net || nmcli nm wifi on
+list_crashes=$(ls /var/crash)
+if [ -z "$list_crashes" ]; then
+    echo no crash files found
+    exit 0
+fi
+# wait for crash file to be ready
+ls /var/crash | grep crash | while read line; do
+    while [ $(stat -c "%a %n" /var/crash/$line | cut -d ' ' -f1) -le 0 ] ; do
+        sleep 5
+        echo "waiting for crash file to become ready" to upload
+    done
+done
+
+# gather our artifacts
+ls=$(ls /var/crash) 
+echo $ls > $ADT_ARTIFACTS/crash_files.log
+cp -R /var/crash/ $ADT_ARTIFACTS/crash
+cp -R /var/log/system-image $ADT_ARTIFACTS/system-image
+cp /home/phablet/result.subunit $ADT_ARTIFACTS/result.subunit
+cp /var/log/syslog $ADT_ARTIFACTS/syslog
+cp /var/log/kern.log $ADT_ARTIFACTS/kern.log || true
+cp -R /home/phablet/.cache/upstart $ADT_ARTIFACTS/upstart 
+echo "exiting"
+
+exit 1

=== modified file 'lrt/__init__.py'
--- lrt/__init__.py	2014-10-20 21:25:19 +0000
+++ lrt/__init__.py	2014-11-13 15:27:18 +0000
@@ -208,7 +208,6 @@
         self._start_time = time.clock_gettime(time.CLOCK_MONOTONIC_RAW)
         self._screenshot_clock = time.clock_gettime(time.CLOCK_MONOTONIC_RAW)
         self._desired_runtime = 43200  # run for 12 hours
-        unlock_ui()  # open greeter
         turn_off_sound()
 
     def tearDown(self):

=== modified file 'lrt/test_random_gestures.py'
--- lrt/test_random_gestures.py	2014-07-23 19:06:02 +0000
+++ lrt/test_random_gestures.py	2014-11-13 15:27:18 +0000
@@ -1,6 +1,8 @@
-
-"""Generate random gestures until something crashes, or something times out."""
+#!/usr/bin/python3
+"""Generate random gestures until something crashes, or something
+times out."""
 import random
+import unittest
 
 from testtools.content import text_content
 
@@ -26,3 +28,7 @@
             else:
                 swipe(SwipeDirection.get_random_direction())
             self.check_for_crash()
+
+
+if __name__ == '__main__':
+    unittest.main()