Merge lp:~canonical-platform-qa/qa-jenkins-jobs/add-unity8-qemu-jobs into lp:qa-jenkins-jobs

[Richard Huddie]

Add jenkins jobs for ubuntu-system-tests app launch tests which run on qemu.

This adds jobs for building either a snap or deb based unity8 image from an iso file which is stored locally on venonat. Once the image builder job completes it triggers the app launch test jobs.

The test jobs use an overlay image on the original base image, so its quick and easy to re-run tests without having the re-build the image each time.

Jobs are built for xenial and zesty versions.

Note that these jobs require following branch to land in ubuntu-system-tests trunk: https://code.launchpad.net/~canonical-platform-qa/ubuntu-system-tests/detect-app-launch-tests/+merge/314993

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:205
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/73/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/235
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/235/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/233
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/233/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1547/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/73/rebuild

review: Approve (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:206
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/74/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/236
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/236/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/234
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/234/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1548/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/74/rebuild

review: Approve (continuous-integration)

[Santiago Baldassin (sbaldassin)]

This looks good. Let's get this landed so we can actually run them on venoat

review: Approve

[Richard Huddie (rhuddie)]

> This looks good. Let's get this landed so we can actually run them on venoat

Agree we need to get this landed. I'm just making some final updates to the corresponding ubuntu-system-tests branch which also needs to land to enable this. Once that is done we can land this one and deploy the jobs.

[lp:~canonical-platform-qa/qa-jenkins-jobs/add-unity8-qemu-jobs]

210. By Richard Huddie on 2017-02-03

Add lab requirements back.

211. By Richard Huddie on 2017-02-03

Use trunk version.

[Max Brustkern (nuclearbob)]

One small diff question, but nothing that should block the landing.

[lp:~canonical-platform-qa/qa-jenkins-jobs/add-unity8-qemu-jobs]

212. By Richard Huddie on 2017-02-03

Remove un-neccessary junit builder.

[Richard Huddie (rhuddie)]

Thanks for review. I've removed the junit builder as there are no subunit results for this job.

[Max Brustkern (nuclearbob)]

Ship it!

review: Approve

=== added file 'jobs/ubuntu-system-tests/applaunch-jobs.yaml'
--- jobs/ubuntu-system-tests/applaunch-jobs.yaml	1970-01-01 00:00:00 +0000
+++ jobs/ubuntu-system-tests/applaunch-jobs.yaml	2017-02-03 15:23:24 +0000
@@ -0,0 +1,110 @@
+# vim: sw=4 ts=4 et
+
+# QA Jenkins Jobs
+# Copyright 2017 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+- project:
+    name: ubuntu-system-tests-app-launch
+    automation_status:
+        - stable
+    release:
+        - xenial
+        - zesty
+    package:
+        - snap
+        - deb
+    target:
+        - qemu:
+            device_node: iso-testing||venonat-upgrade
+    jobs:
+        - 'ust_build_image_{release}_{target}_{package}'
+        - 'ust_test_launch_apps_{release}_{target}_{package}'
+
+- job-template: &UST_APP_LAUNCH_JOB_TEMPLATE
+    name: 'ust_app_launch_job_template'
+    concurrent: true
+    node: '{device_node}'
+    parameters:
+        - ubuntu-system-tests-app-launch-parameters:
+            release: '{release}'
+            package: '{package}'
+            target: '{target}'
+
+- job-template:
+    <<: *UST_APP_LAUNCH_JOB_TEMPLATE
+    name: 'ust_build_image_{release}_{target}_{package}'
+    description: 'Build image for target.'
+    builders:
+        - clear-artifacts
+        - ubuntu-system-tests-config-builder
+        - ubuntu-system-tests-image-builder
+    publishers:
+        - trigger:
+            project: 'ust_test_launch_apps_{release}_{target}_{package}'
+            threshold: 'SUCCESS'
+        - archive:
+            artifacts: 'results/**/*'
+            allow-empty: 'true'
+
+- job-template:
+    <<: *UST_APP_LAUNCH_JOB_TEMPLATE
+    name: 'ust_test_launch_apps_{release}_{target}_{package}'
+    description: 'Run app launch tests against image.'
+    builders:
+        - clear-artifacts
+        - ubuntu-system-tests-config-builder
+        - ubuntu-system-tests-run-tests-builder
+        - subunit-to-junit-results-builder
+    publishers:
+        - archive:
+            artifacts: 'results/**/*'
+            allow-empty: 'true'
+        - junit:
+            results: 'results/test-results.xml'
+            test-stability: 'true'
+
+- parameter:
+    name: ubuntu-system-tests-app-launch-parameters
+    parameters:
+        - string:
+            name: RELEASE
+            default: '{release}'
+            description: Release version tested.
+        - string:
+            name: PACKAGE
+            default: '{package}'
+            description: Package type for unity8 session, either deb or snap.
+        - string:
+            name: TARGET
+            default: '{target}'
+            description: Name of target, either qemu or adb.
+
+- builder:
+    name: ubuntu-system-tests-config-builder
+    builders:
+        - shell:
+            !include-raw: 'ubuntu-system-tests-config-builder.sh'
+
+- builder:
+    name: ubuntu-system-tests-image-builder
+    builders:
+        - shell:
+            !include-raw: 'ubuntu-system-tests-image-builder.sh'
+
+- builder:
+    name: ubuntu-system-tests-run-tests-builder
+    builders:
+        - shell:
+            !include-raw: 'ubuntu-system-tests-run-tests-builder.sh'

=== modified file 'jobs/ubuntu-system-tests/jobs.yaml'
--- jobs/ubuntu-system-tests/jobs.yaml	2016-09-27 18:10:05 +0000
+++ jobs/ubuntu-system-tests/jobs.yaml	2017-02-03 15:23:24 +0000
@@ -90,7 +90,7 @@
             artifacts: 'results/**/*'
             allow-empty: 'true'
         - junit:
-            results: 'results/artifacts/test-results.xml'
+            results: 'results/test-results.xml'
 
 - job-template:
     <<: *UST_DEFAULT_JOB_TEMPLATE
@@ -136,7 +136,7 @@
             artifacts: 'results/**/*'
             allow-empty: 'true'
         - junit:
-            results: 'results/artifacts/test-results.xml'
+            results: 'results/test-results.xml'
         - trigger-parameterized-builds:
             - project: ust_report-results-to-practitest
               predefined-parameters: |
@@ -172,7 +172,7 @@
             artifacts: 'results/**/*'
             allow-empty: 'true'
         - junit:
-            results: 'results/artifacts/test-results.xml'
+            results: 'results/test-results.xml'
 
 - parameter:
     name: ubuntu-system-tests-parameters
@@ -270,7 +270,7 @@
     builders:
         - copyartifact:
             project: "${{UPSTREAM_JOB_NAME}}"
-            filter: "results/artifacts/test-results.subunit"
+            filter: "results/test-results.subunit"
             which-build: upstream-build
         - shell:
             !include-raw: 'report-results-to-practitest-builder.sh'

=== modified file 'jobs/ubuntu-system-tests/report-results-to-grafana-builder.sh'
--- jobs/ubuntu-system-tests/report-results-to-grafana-builder.sh	2016-10-12 16:43:54 +0000
+++ jobs/ubuntu-system-tests/report-results-to-grafana-builder.sh	2017-02-03 15:23:24 +0000
@@ -19,4 +19,4 @@
 
 echo "Push to grafana"
 export OUTPUTDIR="$WORKSPACE/results"
-python3 qakit/qakit/prodenv/tests/sync_test_results.py -p $OUTPUTDIR/artifacts/test-results.pyunit -d {device} -c $CHANNEL -b $REVISION -s {setid}
+python3 qakit/qakit/prodenv/tests/sync_test_results.py -p $OUTPUTDIR/test-results.pyunit -d {device} -c $CHANNEL -b $REVISION -s {setid}

=== modified file 'jobs/ubuntu-system-tests/report-results-to-practitest-builder.sh'
--- jobs/ubuntu-system-tests/report-results-to-practitest-builder.sh	2016-05-26 22:22:08 +0000
+++ jobs/ubuntu-system-tests/report-results-to-practitest-builder.sh	2017-02-03 15:23:24 +0000
@@ -9,4 +9,4 @@
 bzr branch $QA_JENKINS_JOBS_BRANCH qa-jenkins-jobs
 
 # "1SS" is the name of the lab
-./qa-jenkins-jobs/scripts/ubuntu-system-tests/report-results-to-practitest.sh $WORKSPACE/results/artifacts/test-results.subunit $TEST_LEVEL 1SS
+./qa-jenkins-jobs/scripts/ubuntu-system-tests/report-results-to-practitest.sh $WORKSPACE/results/test-results.subunit $TEST_LEVEL 1SS

=== modified file 'jobs/ubuntu-system-tests/subunit-to-junit.sh'
--- jobs/ubuntu-system-tests/subunit-to-junit.sh	2016-08-18 23:58:31 +0000
+++ jobs/ubuntu-system-tests/subunit-to-junit.sh	2017-02-03 15:23:24 +0000
@@ -7,6 +7,6 @@
 
 # sometimes get unicode-encoding errors; -o option apparently broken in this env :/
 export PYTHONIOENCODING=utf-8
-subunit2junitxml $WORKSPACE/results/artifacts/test-results.subunit > $WORKSPACE/results/artifacts/test-results.xml || true
+subunit2junitxml $WORKSPACE/results/test-results.subunit > $WORKSPACE/results/test-results.xml || true
 
 deactivate # our virtualenv

=== added file 'jobs/ubuntu-system-tests/ubuntu-system-tests-config-builder.sh'
--- jobs/ubuntu-system-tests/ubuntu-system-tests-config-builder.sh	1970-01-01 00:00:00 +0000
+++ jobs/ubuntu-system-tests/ubuntu-system-tests-config-builder.sh	2017-02-03 15:23:24 +0000
@@ -0,0 +1,78 @@
+#!/bin/bash
+set -x
+
+export QA_JENKINS_JOBS_BRANCH=lp:qa-jenkins-jobs
+export UBUNTU_SYSTEM_TESTS_CONFIG_BRANCH="lp:ubuntu-system-tests-config"
+export CONFIG_PATH=${WORKSPACE}/ubuntu-system-tests.conf
+export CACHE_PATH=${WORKSPACE}/.ust-cache
+export OUTPUTDIR=$WORKSPACE/results
+
+# TODO: Use this once we can access the config project
+#rm -rf qa-jenkins-jobs
+#bzr branch $QA_JENKINS_JOBS_BRANCH qa-jenkins-jobs
+#./qa-jenkins-jobs/scripts/ubuntu-system-tests/write-ust-config.sh
+# TODO: Remove this once we can access the config project
+rm $CONFIG_PATH
+cat >> $CONFIG_PATH <<EOF
+[default]
+tests_to_run = FIXME
+ssh_ip = FIXME
+ssh_user = FIXME
+bluetooth_device_name = FIXME
+device_username = ubuntu
+device_password = ubuntu
+device_ip = localhost
+device_phone_number = FIXME
+device_security = Passcode
+max_unity8_retry_delay = 30000
+sim_0_pin = FIXME
+sim_1_pin = FIXME
+telephony_service_number1 = FIXME
+telephony_service_number2 = FIXME
+ssh_passphrase = FIXME
+ssh_private_key = FIXME
+device_phone_number2 = FIXME
+touch_visualization = false
+_twilio_account_sid = FIXME
+_twilio_auth_token = FIXME
+_practitest_project_id = FIXME
+_practitest_api_token = FIXME
+wifi_password = FIXME
+wifi_ssid = FIXME
+app_startup_cold_runs = 0
+app_startup_hot_runs = 0
+pictures_scalability_runs = 0
+videos_scalability_runs = 0
+music_scalability_runs = 0
+password_outlook = FIXME
+password_yahoo = FIXME
+password_imap = FIXME
+encryption_imap = FIXME
+hostname_imap = FIXME
+port_number_imap = FIXME
+username_imap = FIXME
+email_outlook = FIXME
+email_yahoo = FIXME
+email_smtp = FIXME
+encryption_smtp = FIXME
+hostname_smtp = FIXME
+name_smtp = FIXME
+password_smtp = FIXME
+port_number_smtp = FIXME
+username_smtp = FIXME
+country_sim = FIXME
+qemu_auto_iso_download = false
+qemu_iso_arch = amd64
+qemu_iso_url = http://cdimage.ubuntu.com/daily-live/current
+cache_data_path = $CACHE_PATH
+qemu_cpu_count = 2
+qemu_ram_alloc = 2G
+qemu_disk_size = 15G
+qemu_headless = true
+qemu_max_cache_file_count = 0
+device_serial = 1234567890
+output_dir = $OUTPUTDIR
+target_type = $TARGET
+package_type = $PACKAGE
+EOF
+cat $CONFIG_PATH

=== added file 'jobs/ubuntu-system-tests/ubuntu-system-tests-image-builder.sh'
--- jobs/ubuntu-system-tests/ubuntu-system-tests-image-builder.sh	1970-01-01 00:00:00 +0000
+++ jobs/ubuntu-system-tests/ubuntu-system-tests-image-builder.sh	2017-02-03 15:23:24 +0000
@@ -0,0 +1,36 @@
+#!/bin/bash
+set -x
+
+export UBUNTU_SYSTEM_TESTS_BRANCH=lp:ubuntu-system-tests
+export QA_JENKINS_JOBS_BRANCH=lp:qa-jenkins-jobs
+export ISO_PATH=/var/cache/utah/iso/$RELEASE-desktop-amd64.iso
+export SHARED_IMG_PATH=/var/cache/ust/img/$RELEASE-unity8-$PACKAGE.img
+export OUTPUTDIR=$WORKSPACE/results
+export CACHE_PATH=$WORKSPACE/.ust-cache
+export IMG_PATH=$CACHE_PATH/disk.img
+export CONFIG_PATH=$WORKSPACE/ubuntu-system-tests.conf
+export https_proxy=https://squid.internal:3128/
+
+rm -rf $OUTPUTDIR
+mkdir $OUTPUTDIR
+
+# let's work in a virtualenv
+rm -rf ve3
+virtualenv --system-site-packages --python=python3 ve3
+. ve3/bin/activate
+
+rm -rf ubuntu-system-tests
+bzr branch $UBUNTU_SYSTEM_TESTS_BRANCH ubuntu-system-tests
+cd ubuntu-system-tests
+python3 setup.py develop
+
+# build the image
+python3 -m ubuntu_system_tests.run build-qemu --iso $ISO_PATH --headless --silent --config $CONFIG_PATH --img $IMG_PATH
+
+# run setup on the image to install unity8-session as either deb or snap
+python3 -m ubuntu_system_tests.run setup --headless --silent --config $CONFIG_PATH --img $IMG_PATH
+
+# copy the resulting image file to shared location for use by tests
+sudo cp $IMG_PATH $SHARED_IMG_PATH
+
+deactivate # our virtualenv

=== added file 'jobs/ubuntu-system-tests/ubuntu-system-tests-run-tests-builder.sh'
--- jobs/ubuntu-system-tests/ubuntu-system-tests-run-tests-builder.sh	1970-01-01 00:00:00 +0000
+++ jobs/ubuntu-system-tests/ubuntu-system-tests-run-tests-builder.sh	2017-02-03 15:23:24 +0000
@@ -0,0 +1,31 @@
+#!/bin/bash
+set -x
+
+export UBUNTU_SYSTEM_TESTS_BRANCH=lp:ubuntu-system-tests
+export SHARED_IMG_PATH=/var/cache/ust/img/$RELEASE-unity8-$PACKAGE.img
+export OUTPUTDIR=$WORKSPACE/results
+export CONFIG_PATH=$WORKSPACE/ubuntu-system-tests.conf
+export https_proxy=https://squid.internal:3128/
+
+rm -rf $OUTPUTDIR
+mkdir $OUTPUTDIR
+
+# let's work in a virtualenv
+rm -rf ve3
+virtualenv --system-site-packages --python=python3 ve3
+. ve3/bin/activate
+
+rm -rf ubuntu-system-tests
+bzr branch $UBUNTU_SYSTEM_TESTS_BRANCH ubuntu-system-tests
+cd ubuntu-system-tests
+python3 setup.py develop
+
+# Run tests on the image.
+# For qemu this will use a temporary overlay image by default,
+# so we can use the shared image directly as base.
+python3 -m ubuntu_system_tests.run run --headless --silent --config $CONFIG_PATH --img $SHARED_IMG_PATH ubuntu_system_tests.tests.test_launch_apps.LaunchAppOnceTestCase
+
+# Run next set of tests, appending results to previous results
+python3 -m ubuntu_system_tests.run run --headless --silent --config $CONFIG_PATH --img $SHARED_IMG_PATH --append-results ubuntu_system_tests.tests.test_launch_apps.LaunchAppTwiceTestCase
+
+deactivate # our virtualenv