Merge lp:~canonical-platform-qa/qa-jenkins-jobs/ust-mp-review-jobs into lp:qa-jenkins-jobs

[Allan LeSage]

Add Jenkins jobs for ust MP reviews.

Add the Jenkins jobs to do ubuntu-system-tests CI, using sbalda's script to glean tests from MPs.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:167
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/25/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/2962
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/2962/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/2962
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/2962/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/478/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/25/rebuild

review: Approve (continuous-integration)

[Santiago Baldassin (sbaldassin)]

Looks good to me but let's wait until Max takes a look

review: Approve

[Max Brustkern (nuclearbob)]

I'm slightly concerned that if we need to make changes to the main autolanding stuff in the future, this'll get missed. Can we maybe have a comment in the main ci-jobs.yaml noting that there exists a separate u-s-t ci-jobs.yaml, and changes here should be reflected there as well? I think it could be a very long time before that happens, but that's probably just long enough for us to forget we need it. :) One diff comment also.

[lp:~canonical-platform-qa/qa-jenkins-jobs/ust-mp-review-jobs]

168. By Allan LeSage on 2016-07-29

Merge trunk.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:168
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/26/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/2978
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/2978/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/2978
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/2978/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/491/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/26/rebuild

review: Approve (continuous-integration)

[lp:~canonical-platform-qa/qa-jenkins-jobs/ust-mp-review-jobs]

169. By Allan LeSage on 2016-07-29

Remove fake merge proposal.

170. By Allan LeSage on 2016-08-01

Adapt to sbalda's changes.

171. By Allan LeSage on 2016-08-01

Merge sbalda's changes, remove QA_JENKINS_JOBS hardcode.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:171
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/28/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/2993
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/2993/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/2993
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/2993/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/506/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/28/rebuild

review: Approve (continuous-integration)

[Max Brustkern (nuclearbob)]

I'd still like to see a note in the main ci-jobs.yaml file about how there's a separate file now and changes may need to be propagated to it as well.

[Allan LeSage (allanlesage)]

Haven't forgotten about you Max ;) , waiting for sbalda's changes to land and then will give another test.

[lp:~canonical-platform-qa/qa-jenkins-jobs/ust-mp-review-jobs]

172. By Allan LeSage on 2016-08-01

Revert a couple of necessary changes temporarily.

173. By Allan LeSage on 2016-08-01

Merge new sbalda changes, including working from MP instead of landing candidate.

174. By Allan LeSage on 2016-08-02

Restore changes, having landed sbalda's code.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:174
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/31/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/3012
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/3012/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/3012
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/3012/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/521/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/31/rebuild

review: Approve (continuous-integration)

[lp:~canonical-platform-qa/qa-jenkins-jobs/ust-mp-review-jobs]

175. By Allan LeSage on 2016-08-02

Add descriptive notes for tracking CI templates to ust, restore a QA_JENKINS_JOBS_BRANCH setting.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:175
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/32/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/3015
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/3015/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/3015
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/3015/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/523/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/32/rebuild

review: Approve (continuous-integration)

[Allan LeSage (allanlesage)]

100% ready for you Max, you can see that it's invoking the job properly here: https://platform-qa-jenkins.ubuntu.com/job/ust_rc-proposed_krillin_sanity_test/60/parameters/ .

[Max Brustkern (nuclearbob)]

Awesome! Should be good to go.

review: Approve

=== modified file 'jobs/mp-ci-autoland/ci-jobs.yaml'
--- jobs/mp-ci-autoland/ci-jobs.yaml	2016-06-16 20:25:29 +0000
+++ jobs/mp-ci-autoland/ci-jobs.yaml	2016-08-02 21:55:58 +0000
@@ -15,6 +15,11 @@
 # You should have received a copy of the GNU General Public License along
 # with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+
+# NOTE that a separate CI workflow and Jenkins jobs exist for
+# ubuntu-system-tests, using templates found here--changes made here
+# should be tested there as well, as noted below.
+
 - builder:
     name: create-chroot
     # This and the update-chroot builder share a lot of code. TODO: Dedupe/be
@@ -197,8 +202,6 @@
         # Must be projects lp:<name>
         - autopilot
         - auto-upgrade-testing
-        - ubuntu-system-tests:
-            hooks: 'H05set_package_version H10strip_native_depends D09add_ppa~canonical-platform-qa~ppa'
         - qa-jenkins-jobs
     dist:
         - wily
@@ -231,6 +234,7 @@
             name: ci_revision
             description: 'The revision of the processed branch'
 
+# NOTE that this template is employed in jobs/ubuntu-system-tests/ci-jobs.yaml
 - job-template:
     name: 'trigger-{target_project}-ci'
     description: |
@@ -358,8 +362,6 @@
       # Must be lp:<name>
       - autopilot
       - auto-upgrade-testing
-      - ubuntu-system-tests:
-          hooks: 'H05set_package_version H10strip_native_depends D09add_ppa~canonical-platform-qa~ppa'
       - qa-jenkins-jobs
 
 - job-template:
@@ -426,6 +428,7 @@
                 ci_merge_proposal=$merge_proposal
                 ci_revision=$candidate_revision
 
+# NOTE that this template is employed in jobs/ubuntu-system-tests/ci-jobs.yaml
 - job-template:
     name: 'trigger-{target_project}-autoland'
     description: |

=== added file 'jobs/ubuntu-system-tests/ci-jobs.yaml'
--- jobs/ubuntu-system-tests/ci-jobs.yaml	1970-01-01 00:00:00 +0000
+++ jobs/ubuntu-system-tests/ci-jobs.yaml	2016-08-02 21:55:58 +0000
@@ -0,0 +1,103 @@
+# vim: sw=4 ts=4 et
+
+# QA Jenkins Jobs
+# Copyright 2016 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+- project:
+    name: ubuntu-system-tests-ci
+    target_project:
+        - ubuntu-system-tests
+    jobs:
+        - 'ubuntu-system-tests-ci-test'
+        - 'ubuntu-system-tests-autoland'
+        # from jobs/mp-ci-autoland/ci-jobs.yaml
+        - 'trigger-{target_project}-ci'
+        - 'trigger-{target_project}-autoland'
+
+- job:
+    name: 'ubuntu-system-tests-ci-test'
+    description: |
+        Build packages and run tests for an ubuntu-system-tests merge proposal
+    node: 'jenkins-slave-2'
+    # This token is the string that was setup for the launchpad bot.
+    auth-token: BUILD_ME
+    builders:
+        - trigger-deb-builders:
+            landing_candidate: {landing_candidate}
+            target_branch: 'lp:ubuntu-system-tests'
+            hooks: 'H05set_package_version H10strip_native_depends D09add_ppa~canonical-platform-qa~ppa'
+            parallelism: 1
+        - trigger-ust-pre-builder
+        - trigger-ust-builder
+        - upcopy-i386-from-downstream
+        - upcopy-amd64-from-downstream
+        - trigger-generic-update-mp
+    publishers:
+        - collect-debs
+        - trigger-generic-update-mp-failed
+    parameters:
+        - string:
+            name: landing_candidate
+            description: The launchpad branch to process
+        - string:
+            name: candidate_revision
+            description: The revision of the branch
+        - string:
+            name: merge_proposal
+            description: The merge proposal to process
+
+- job-template:
+    name: 'ubuntu-system-tests-autoland'
+    description: |
+      Gets triggered when a MP exists that meets autoland criteria. Merges with
+      ubuntu-system-tests trunk.
+    node: 'jenkins-slave-2'
+    # This token is the string that was setup for the launchpad bot.
+    auth-token: BUILD_ME
+    parameters:
+      - string:
+          name: 'landing_candidate'
+          description: 'The launchpad branch to process'
+      - string:
+          name: 'candidate_revision'
+          description: 'The revision of the branch'
+      - string:
+          name: 'merge_proposal'
+          description: 'The merge proposal to process'
+    builders:
+      - trigger-deb-builders:
+            landing_candidate: {landing_candidate}
+            target_branch: 'lp:ubuntu-system-tests'
+            hooks: 'H05set_package_version H10strip_native_depends D09add_ppa~canonical-platform-qa~ppa'
+            parallelism: 1
+      - 'trigger-generic-land-mp'
+    publishers:
+      - 'trigger-generic-land-mp-failed'
+
+- builder:
+    name: trigger-ust-pre-builder
+    builders:
+        - shell:
+            !include-raw: trigger-ust-pre-builder.sh
+
+- builder:
+    name: trigger-ust-builder
+    builders:
+        - trigger-builds:
+            - project:
+                - 'ust_rc-proposed_krillin_sanity_test'
+              block: true
+              property-file: $WORKSPACE/ust-parameters.txt

=== added file 'jobs/ubuntu-system-tests/trigger-ust-pre-builder.sh'
--- jobs/ubuntu-system-tests/trigger-ust-pre-builder.sh	1970-01-01 00:00:00 +0000
+++ jobs/ubuntu-system-tests/trigger-ust-pre-builder.sh	2016-08-02 21:55:58 +0000
@@ -0,0 +1,14 @@
+
+QA_JENKINS_JOBS_BRANCH=lp:qa-jenkins-jobs
+rm -rf qa-jenkins-jobs
+bzr branch $QA_JENKINS_JOBS_BRANCH qa-jenkins-jobs
+
+TESTS_TO_RUN=$(python qa-jenkins-jobs/scripts/ubuntu-system-tests/get_tests_to_run.py -mpu "$merge_proposal" 2>&1)
+echo $TESTS_TO_RUN
+
+rm -f ${WORKSPACE}/ust-parameters.txt
+export PROPERTIES_FILE_PATH=${WORKSPACE}/ust-parameters.txt
+cat >> $PROPERTIES_FILE_PATH <<EOF
+TESTS_TO_RUN=$TESTS_TO_RUN
+EOF
+cat $PROPERTIES_FILE_PATH

=== added file 'scripts/ubuntu-system-tests/get_test_to_run.sh'
--- scripts/ubuntu-system-tests/get_test_to_run.sh	1970-01-01 00:00:00 +0000
+++ scripts/ubuntu-system-tests/get_test_to_run.sh	2016-08-02 21:55:58 +0000
@@ -0,0 +1,2 @@
+#!/bin/sh
+TESTS_TO_RUN=$(python qa-jenkins-jobs/scripts/ubuntu-system-tests/get_tests_to_run.py -mpu "$landing_candidate" 2>&1)

=== modified file 'scripts/ubuntu-system-tests/get_tests_to_run.py' (properties changed: +x to -x)