Merge lp:~canonical-platform-qa/qa-jenkins-jobs/ubuntu-system-tests-jobs-initial into lp:qa-jenkins-jobs

[Allan LeSage]

Initial set of jobs to run ubuntu-system-tests in 1SS Jenkins.

An initial set of jobs to run ubuntu-system-tests in our 1SS Jenkins.

Have been tested thoroughly (manually ;) , i.e. running in Jenkins).

One set of jobs is triggered by new image publication, the 'freestyle' jobs are left for tinkering: it's sometimes helpful to be able to run a failing test alone in a way that won't contribute to the pass/fail history, e.g.

[lp:~canonical-platform-qa/qa-jenkins-jobs/ubuntu-system-tests-jobs-initial]

126. By Allan LeSage on 2016-03-03

Merge trunk, resolving conflict.

127. By Allan LeSage on 2016-03-03

Remove obj: prefixes.

[Christopher Lee (veebers)]

LGTM, makes sense while reading though it. Well laid out and readable.

review: Approve

[Max Brustkern (nuclearbob)]

Having to use a venv and download the image still hurts whatever devops organ I've started to grow within me, but changing that is out of scope here. I'll want to come back later and wave the magic .deb wand over a few things, but for now, I've just got some diff comments.

[Max Brustkern (nuclearbob)]

Oh also: I think we should land and deploy this, but the krillin regression job is probably going to break until we run it in chunks. If you've have a better experience with this, I'd be delighted to not have to do that any more, but in brendand's and my experience, periodic reboots are necessary to keep things running smoothly.

[lp:~canonical-platform-qa/qa-jenkins-jobs/ubuntu-system-tests-jobs-initial]

128. By Allan LeSage on 2016-03-04

Max-requested changes: de-bashify, trust shell to interpret line-breaks correctly when passing lines of tests, add description, add license header.

129. By Allan LeSage on 2016-03-04

Set execution perms on flash-device.sh .

130. By Allan LeSage on 2016-03-04

Remove unsightly 'x' from case statement, add exit on no input.

131. By Allan LeSage on 2016-03-04

Redirect output of specify-tests complaint to stderr.

132. By Allan LeSage on 2016-03-04

Some shuffling of jobs.yaml script and remove default, instead check for null TESTS_TO_RUN.

133. By Allan LeSage on 2016-03-04

Last of the jobs.yaml shuffling.

134. By Allan LeSage on 2016-03-04

Set location of recovery image to /recovery.img.

135. By Allan LeSage on 2016-03-06

Correct single-quotes around recovery image term in flash-device.sh.

[Allan LeSage (allanlesage)]

Adjusted to your comments Max, ready for another review!

[Max Brustkern (nuclearbob)]

Ship it!

review: Approve

=== modified file 'scripts/flash-device.sh'
--- scripts/flash-device.sh	2016-02-10 15:58:04 +0000
+++ scripts/flash-device.sh	2016-03-07 16:07:31 +0000
@@ -50,13 +50,13 @@
   x"krillin")
     DEFAULT_CHANNEL="ubuntu-touch/rc-proposed/bq-aquaris.en"
     FASTBOOT=yes
-    RECOVERY='--recovery-image recovery.img'
+    RECOVERY="--recovery-image ${WORKSPACE}/recovery.img"
     ;;
   x"m75" | x"arale")
     DEVICE=arale
     DEFAULT_CHANNEL="ubuntu-touch/rc-proposed/meizu.en"
     FASTBOOT=yes
-    RECOVERY='--recovery-image recovery.img'
+    RECOVERY="--recovery-image ${WORKSPACE}/recovery.img"
     ;;
   *)
     echo "Unrecognized device: $DEVICE"

=== modified file 'scripts/get-recovery-image.sh'
--- scripts/get-recovery-image.sh	2016-02-09 20:17:59 +0000
+++ scripts/get-recovery-image.sh	2016-03-07 16:07:31 +0000
@@ -18,4 +18,4 @@
 export DEVICE=$(echo $NODE_NAME | sed 's/\(\w\)-.*/\1/')
 
 # from plars' lifeboat scripts, deposit the proper recovery image in ./recovery.img
-wget -O recovery.img http://people.canonical.com/~plars/touch/recovery-$DEVICE.img
+wget -O ${WORKSPACE}/recovery.img http://people.canonical.com/~plars/touch/recovery-$DEVICE.img

=== added directory 'scripts/ubuntu-system-tests'
=== added file 'scripts/ubuntu-system-tests/run-ubuntu-system-tests.sh'
--- scripts/ubuntu-system-tests/run-ubuntu-system-tests.sh	1970-01-01 00:00:00 +0000
+++ scripts/ubuntu-system-tests/run-ubuntu-system-tests.sh	2016-03-07 16:07:31 +0000
@@ -0,0 +1,28 @@
+#!/bin/sh
+set -x
+set +e
+
+export TESTS="$1"
+
+# let's work in a virtualenv
+virtualenv --python=python3 ve3
+. ve3/bin/activate
+
+# uci-config is broken in trusty ppa
+rm -rf uci-config
+bzr branch lp:uci-config
+CURDIR=`pwd`
+cd uci-config
+python3 setup.py install
+cd $CURDIR
+
+rm -rf ubuntu-system-tests
+bzr branch $UBUNTU_SYSTEM_TESTS_BRANCH ubuntu-system-tests
+cd ubuntu-system-tests
+python3 setup.py develop
+python3 -m ubuntu_system_tests.command_line -n -i -u --config=$CONFIG_PATH "$TESTS"
+
+deactivate # our virtualenv
+
+# TODO: run this as a post-build step
+subunit2junitxml $WORKSPACE/results/artifacts/test-results.subunit > $WORKSPACE/results/artifacts/test-results.xml || true

=== added file 'scripts/ubuntu-system-tests/write-config.sh'
--- scripts/ubuntu-system-tests/write-config.sh	1970-01-01 00:00:00 +0000
+++ scripts/ubuntu-system-tests/write-config.sh	2016-03-07 16:07:31 +0000
@@ -0,0 +1,4 @@
+#!/bin/sh
+set -x
+
+bzr cat ${UBUNTU_SYSTEM_TESTS_CONFIG_BRANCH}/config.ini > "${CONFIG_PATH}"

=== added directory 'ubuntu-system-tests'
=== added file 'ubuntu-system-tests/jobs.yaml'
--- ubuntu-system-tests/jobs.yaml	1970-01-01 00:00:00 +0000
+++ ubuntu-system-tests/jobs.yaml	2016-03-07 16:07:31 +0000
@@ -0,0 +1,175 @@
+# vim: sw=4 ts=4 et
+
+# QA Jenkins Jobs
+# Copyright 2016 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+- project:
+    name: ubuntu-system-tests
+    test_level:
+        - sanity
+        - regression
+    channel:
+        - 'rc'
+        - 'rc-proposed'
+        - 'devel-proposed'
+    device:
+        - krillin:
+            device_channel: 'bq-aquaris.en'
+        - arale:
+            device_channel: 'meizu.en'
+    release:
+        - vivid
+    jobs:
+        - 'ubuntu-system-tests_{release}_{channel}_{device}_{test_level}_triggered'
+        - 'ubuntu-system-tests_{release}_{channel}_{device}_{test_level}_freestyle'
+
+- job-template:
+    name: 'ubuntu-system-tests_{release}_{channel}_{device}_{test_level}_triggered'
+    description: "Triggered upon image update."
+    parameters:
+        - ubuntu-system-tests-parameters:
+            test_level: '{test_level}'
+            channel: '{channel}'
+            device_channel: '{device_channel}'
+    node: '{device}'
+    triggers:
+        - ubuntu-system-tests-triggers:
+            channel: '{channel}'
+            device: '{device}'
+            device_channel: '{device_channel}'
+    builders:
+        - ubuntu-system-tests-builder
+    publishers:
+        - archive:
+            artifacts: 'results/**/*'
+            allow-empty: 'true'
+        - junit:
+            results: 'results/artifacts/test-results.xml'
+
+- job-template:
+    name: 'ubuntu-system-tests_{release}_{channel}_{device}_{test_level}_freestyle'
+    description: "Provided for experimentation--e.g. re-running of failed tests--in way that won't contribute to the pass/fail record of the canonical job."
+    parameters:
+        - ubuntu-system-tests-parameters:
+            test_level: '{test_level}'
+            channel: '{channel}'
+            device_channel: '{device_channel}'
+    node: '{device}'
+    builders:
+        - ubuntu-system-tests-builder
+    publishers:
+        - archive:
+            artifacts: 'results/**/*'
+            allow-empty: 'true'
+        - junit:
+            results: 'results/artifacts/test-results.xml'
+
+- parameter:
+    name: ubuntu-system-tests-parameters
+    parameters:
+        - string:
+            name: UBUNTU_SYSTEM_TESTS_BRANCH
+            default: lp:ubuntu-system-tests
+            description: "Launchpad branch of ubuntu-system-tests from which to run tests."
+        - string:
+            name: TESTS_TO_RUN
+            default: '{test_level}'
+            description: "List of tests to run, use 'sanity' or 'regression' to run those suites."
+        - string:
+            name: LIFEBOAT_BRANCH
+            default: https://git.launchpad.net/~canonical-hw-cert/lifeboat
+            description: "Launchpad branch (may be git) of the lifeboat device descriptions."
+        # Begin flash-device params, TODO: !include these as YAML instead
+        - string:
+            name: CHANNEL
+            default: 'ubuntu-touch/{channel}/{device_channel}'
+            description: "The given alternate channel will be used if specified."
+        - string:
+            name: REVISION
+            default:
+            description:
+        - string:
+            name: WIZARD
+            default: "yes"
+            description: "Run the device setup wizard?  'no' disables."
+        - string:
+            name: EDGEINTRO
+            default: "yes"
+            description: "Run the device edges intro?  'no' disables."
+        - string:
+            name: QA_JENKINS_JOBS_BRANCH
+            default: lp:qa-jenkins-jobs
+            description: Launchpad branch of qa-jenkins-jobs scripts.
+
+- trigger:
+    name: ubuntu-system-tests-triggers
+    triggers:
+        - pollurl:
+            cron: 'H/15 * * * *'
+            polling-node: 'jenkins-slave-2'
+            urls:
+                - url: 'http://system-image.ubuntu.com/ubuntu-touch/{channel}/{device_channel}/{device}/index.json.asc'
+                  check-content:
+                    - simple: true
+
+- builder:
+    name: ubuntu-system-tests-builder
+    builders:
+        - shell: |
+
+            # use plars' lifeboat scripts to get our adb serial
+            rm -rf lifeboat
+            git clone $LIFEBOAT_BRANCH lifeboat
+            export ANDROID_SERIAL=$(./lifeboat/get-device-info serial $NODE_NAME)
+
+            rm -rf $WORKSPACE/results
+            mkdir $WORKSPACE/results
+            export OUTPUTDIR="$WORKSPACE/results"
+
+            rm -rf qa-jenkins-jobs
+            bzr branch $QA_JENKINS_JOBS_BRANCH qa-jenkins-jobs
+            cd qa-jenkins-jobs
+
+            export UBUNTU_SYSTEM_TESTS_CONFIG_BRANCH="lp:ubuntu-system-tests-config"
+            export CONFIG_PATH=${WORKSPACE}/ubuntu-system-tests.conf
+            ./scripts/ubuntu-system-tests/write-config.sh
+            cat >> $CONFIG_PATH <<EOF
+            device_serial = $ANDROID_SERIAL
+            output_dir = $OUTPUTDIR
+            EOF
+            cat $CONFIG_PATH
+
+            case "$TESTS_TO_RUN" in
+              "regression")
+                TESTS_TO_RUN=""
+                ;;
+              "sanity")
+                TESTS_TO_RUN=$(bzr cat "$UBUNTU_SYSTEM_TESTS_BRANCH/test_lists/sanity.tests")
+                ;;
+              "")
+                echo "Please specify tests to run!" >&2
+                exit 1
+                ;;
+            esac
+
+            ./scripts/get-recovery-image.sh
+
+            recover_device_flash_and_run_tests () {
+              # NOTE pass in a list of tests to run
+              ./scripts/recover-device.sh
+              ./scripts/flash-device.sh
+              ./scripts/ubuntu-system-tests/run-ubuntu-system-tests.sh "$1"
+            }
+            recover_device_flash_and_run_tests "$(echo "$TESTS_TO_RUN")"