Merge lp:~canonical-platform-qa/qa-jenkins-jobs/daily-spread into lp:qa-jenkins-jobs

[Max Brustkern]

Add daily spread tests for ubuntu core

This branch adds daily jobs to run spread tests on virtual machines. Building daily images from candidate snaps and upgrading the ubuntu core 16 released images to current stable are currently supported. Only pc-amd64 is included in the initial release, but pc-i386 should be easy to support once a model assertion is available and the appropriate released image is setup. Triggering the tests based on updates is the next step, and grafana reporting will be added once spread can output junit.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:215
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/104/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/292
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/292/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/290
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/290/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1635/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/104/rebuild

review: Approve (continuous-integration)

[Max Brustkern (nuclearbob)]

pc-i386 is added now.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:216
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/105/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/293
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/293/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/291
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/291/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1637/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/105/rebuild

review: Approve (continuous-integration)

[Jean-Baptiste Lallement (jibel)]

LGTM, thanks!

review: Approve

[platform-qa-bot (platform-qa-bot)]

FAILED: Autolanding.
More details in the following jenkins job:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-autoland/2471/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/295/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/293/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-land-mp/2762/console

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:217
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/106/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/296/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/294/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1679/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/106/rebuild

review: Needs Fixing (continuous-integration)

[Max Brustkern (nuclearbob)]

Autolanding is still trying to run with the allanlesage user. I've deployed this manually so we can get daily tests while we fix that.

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:217
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/107/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/297/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/295/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1686/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/107/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:217
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/110/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/311/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/309/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1700/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/110/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:217
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/113/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/321/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/319/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1714/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/113/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:217
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/117/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/398/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/398/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1891/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/117/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:217
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/120/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/433/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/433/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1934/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/120/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:217
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/123/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/446/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/446/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1947/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/123/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:217
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/129/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/468/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/468/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1973/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/129/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:217
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/132/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/471/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/471/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1976/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/132/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:217
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/135/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/474/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/474/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1979/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/135/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:217
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/138/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/477/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/477/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1982/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/138/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:218
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/140/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/479/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/479/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/1984/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/140/rebuild

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

review: Approve (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

FAILED: Autolanding.
More details in the following jenkins job:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-autoland/2478/
Executed test runs:
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-amd64-package/495/console
    FAILURE: https://platform-qa-jenkins.ubuntu.com/job/build-xenial-i386-package/493/console
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-land-mp/2770/console

review: Needs Fixing (continuous-integration)

[platform-qa-bot (platform-qa-bot)]

review: Approve (continuous-integration)

=== modified file 'debian/control'
--- debian/control	2016-05-19 19:54:52 +0000
+++ debian/control	2017-02-23 16:39:06 +0000
@@ -2,7 +2,7 @@
 Section: devel
 Priority: optional
 Maintainer: Max Brustkern <[e-mail]>
-Build-Depends: debhelper (>= 9)
+Build-Depends: debhelper (>= 9), bzr
 Standards-Version: 3.9.7
 Homepage: https://launchpad.net/qa-jenkins-jobs
 

=== modified file 'jobs/macros.yaml'
--- jobs/macros.yaml	2017-02-14 11:57:56 +0000
+++ jobs/macros.yaml	2017-02-23 16:39:06 +0000
@@ -1,3 +1,20 @@
+# vim: sw=4 ts=4 et
+
+# Auto Upgrade Testing
+# Copyright 2016-2017 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 - builder: # This step removes artifacts from previous builds
     name: clear-artifacts
     builders:
@@ -77,3 +94,8 @@
         - archive:
             artifacts: '**/*'
             allow-empty: 'true'
+
+- trigger: # This will schedule a daily build at a staggered time
+    name: 'daily'
+    triggers:
+        - timed: 'H H * * *'

=== added directory 'jobs/snapcore-qa-tests'
=== added file 'jobs/snapcore-qa-tests/jobs.yaml'
--- jobs/snapcore-qa-tests/jobs.yaml	1970-01-01 00:00:00 +0000
+++ jobs/snapcore-qa-tests/jobs.yaml	2017-02-23 16:39:06 +0000
@@ -0,0 +1,114 @@
+# vim: sw=4 ts=4 et
+
+# QA Jenkins Jobs
+# Copyright 2017 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+- project:
+    name: snapcore-qa-tests-setup
+    node: iso-testing
+    jobs:
+        - snapcore-qa-tests-setup
+
+- project:
+    # Requires: Post Build Script plugin.
+    name: snapcore-qa-tests
+    model:
+        - pc-amd64
+        - pc-i386
+    jobs:
+        - snapcore-candidate-{model}-spread
+        - snapcore-stable-upgrade-{model}-spread
+
+- builder:
+    name: snap-vm
+    builders:
+        - shell: |
+            if [ -f ~/.proxy_info ] ; then
+                . ~/.proxy_info
+            fi
+
+            cd snapcore-qa-tests/autospread
+
+            export USER="{user}"
+            IMAGE="{image}"
+            if [ -z "$IMAGE" ] ; then
+                sudo ./create-image.sh -k ~/.ssh/id_rsa.pub -c {channel} -m {model}
+                IMAGE=ubuntu-core-16-{model}.img
+            fi
+
+            ./spread-vm.sh $IMAGE {upgrade}
+
+- job-template:
+    name: snapcore-candidate-{model}-spread
+    description: |
+        This job runs the spread tests on an image built from candidate snaps for the {model} model
+        Defined in lp:qa-jenkins-jobs
+    node: iso-testing
+    triggers:
+        - daily:
+    builders:
+        - clear-artifacts:
+        - branch-bzr-project:
+            project-url: 'lp:snapcore-qa-tests'
+            project-name: 'snapcore-qa-tests'
+        - snap-vm:
+            user: 'qa'
+            image: ''
+            channel: 'candidate'
+            model: '{model}'
+            upgrade: ''
+    publishers:
+        - archive:
+            artifacts: 'spread-output'
+            allow-empty: 'true'
+
+- job-template:
+    name: snapcore-stable-upgrade-{model}-spread
+    description: |
+        This job runs the spread tests on released ubuntu core 16 image upgrade to current stable for the {model} model
+        Defined in lp:qa-jenkins-jobs
+    node: iso-testing
+    triggers:
+        - daily:
+    builders:
+        - clear-artifacts:
+        - branch-bzr-project:
+            project-url: 'lp:snapcore-qa-tests'
+            project-name: 'snapcore-qa-tests'
+        - snap-vm:
+            user: 'nuclearbob'
+            image: '/var/cache/auto-upgrade-testing/ubuntu-core-16-{model}.img'
+            channel: ''
+            model: ''
+            upgrade: '-upgrade'
+    publishers:
+        - archive:
+            artifacts: 'spread-output'
+            allow-empty: 'true'
+
+- job:
+    name: 'snapcore-qa-tests-setup'
+    description: |
+        This job ensures we have everything needed to run snapcore-qa-tests
+        It can be run again to update to the latest versions of dependencies
+        Defined in lp:qa-jenkins-jobs
+    builders:
+        - clear-artifacts:
+        - branch-bzr-project:
+            project-url: 'lp:snapcore-qa-tests'
+            project-name: 'snapcore-qa-tests'
+        - shell: |
+            cd snapcore-qa-tests/autospread
+            ./setup.sh