Merge lp:~canonical-platform-qa/qa-jenkins-jobs/auto_ci-land_upgrade_testing into lp:qa-jenkins-jobs

[Christopher Lee]

Template autolanding and ci jobs to add auto-upgrade-testing.

Template autolanding and ci jobs to add auto-upgrade-testing.

[Max Brustkern (nuclearbob)]

We've talked about creating separate macros files. Would this be a good time to start that, since we're directly reusing a lot of code from other jobs? It seems like otherwise, some of these job names will overlap as well. Even if we don't go as far as making a generic macros file, maybe making a generic autolanding job set would work.

[lp:~canonical-platform-qa/qa-jenkins-jobs/auto_ci-land_upgrade_testing]

28. By Christopher Lee on 2016-03-07

Merge parent

29. By Christopher Lee on 2016-03-07

Merge parent

Unmerged revisions

=== added directory 'mp-ci-autoland'
=== added file 'mp-ci-autoland/ci-jobs.yaml'
--- mp-ci-autoland/ci-jobs.yaml	1970-01-01 00:00:00 +0000
+++ mp-ci-autoland/ci-jobs.yaml	2016-03-04 03:55:40 +0000
@@ -0,0 +1,469 @@
+# vim: sw=4 ts=4 et
+
+# QA Jenkins Jobs
+# Copyright 2016 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+- builder:
+    name: create-chroot
+    # This and the update-chroot builder share a lot of code. TODO: Dedupe/be
+    # smarter about this.
+    builders:
+        - shell: |
+            # TODO: Use the production PPA
+            sudo add-apt-repository -y ppa:jenkaas-hackers/tools
+            sudo apt-get update
+            sudo apt-get install -y --assume-yes sphinx-common python3-setuptools pbuilder pbuilderjenkins bzr-builddeb devscripts dh-translations
+
+            # Only proceed if the config options are set for building chroots
+            if [ -z "$distributions" ] || [ -z "$arches" ]; then
+                echo "Missing vales for distributions and arches,"
+                echo "  skipping pbuilder chroot creation."
+                exit 1
+            fi
+
+            COMPONENTS="main restricted universe multiverse"
+            MIRROR="http://archive.ubuntu.com/ubuntu/"
+            EXTRA_PACKAGES="apt-transport-https ca-certificates devscripts eatmydata gnupg patchutils pbuilder vim-tiny"
+            for dist in $distributions; do
+                for arch in $arches; do
+                    if [ ! -f "/var/cache/pbuilder/$dist-$arch.tgz" ]; then
+                        OTHER_MIRROR="deb $MIRROR $dist-updates main restricted universe multiverse|deb-src $MIRROR $dist-updates main restricted universe multiverse|deb-src $MIRROR $dist main restricted universe multiverse"
+                        sudo /usr/sbin/pbuilder --create \
+                            --distribution "$dist" \
+                            --architecture "$arch" \
+                            --basetgz "/var/cache/pbuilder/$dist-$arch.tgz" \
+                            --debootstrapopts \
+                            --variant=buildd
+                        sudo /usr/sbin/pbuilder --update \
+                            --distribution "$dist" \
+                            --basetgz "/var/cache/pbuilder/$dist-$arch.tgz" \
+                            --components "$COMPONENTS" \
+                            --mirror "$MIRROR" \
+                            --othermirror "$OTHER_MIRROR" \
+                            --override-config \
+                            --extrapackages "$EXTRA_PACKAGES"
+                    else
+                        echo "Skipping creation of /var/cache/pbuilder/$dist-$arch.tgz, it already exists."
+                    fi
+                done
+            done
+
+
+- builder:
+    name: update-chroot
+    builders:
+        - shell: |
+            # Only proceed if the config options are set for building chroots
+            if [ -z "$distributions" ] || [ -z "$arches" ]; then
+                echo "Missing values for distributions and arches,"
+                echo "  skipping pbuilder chroot creation."
+                exit 1
+            fi
+
+            COMPONENTS="main restricted universe multiverse"
+            MIRROR="http://archive.ubuntu.com/ubuntu/"
+
+            for dist in "$distributions"; do
+                for arch in "$arches"; do
+                    if [ -f "/var/cache/pbuilder/$dist-$arch.tgz" ]; then
+                        OTHER_MIRROR="deb $MIRROR $dist-updates $COMPONENTS|deb-src $MIRROR $dist-updates $COMPONENTS|deb-src $MIRROR $dist $COMPONENTS"
+                        sudo /usr/sbin/pbuilder --update \
+                            --basetgz "/var/cache/pbuilder/$dist-$arch.tgz" \
+                            --distribution "$dist" \
+                            --components "$COMPONENTS" \
+                            --mirror "$MIRROR" \
+                            --othermirror "$OTHER_MIRROR" \
+                            --override-config
+                        if [ -n "$extra_packages" ]; then
+                            sudo /usr/sbin/pbuilder --update \
+                                --basetgz "/var/cache/pbuilder/$dist-$arch.tgz" \
+                                --extrapackages "$extra_packages"
+                        fi
+                    else
+                        echo "Chroot /var/cache/pbuilder/$dist-$arch.tgz not found, skipping update."
+                    fi
+                done
+            done
+
+- builder:
+    name: build-package
+    builders:
+        - shell: |
+            # Remove any prior work dir created by a prior build
+            rm -rf work
+
+            # local_archive_pocket is used by H05set_package_version and therefore necessary for most builds
+            export local_archive_pocket={dist}
+
+            pbuilderjenkins -c -j "$parallelism" -m "$landing_candidate" -t "$target_branch" --hook-list "$hooks" --base "{dist}-{arch}"
+
+- builder: # This step removes artifacts from previous builds
+    name: clear-artifacts
+    builders:
+        - shell: 'sudo rm -rf *'
+
+- builder:
+    name: trigger-deb-builders
+    builders:
+        - trigger-builds:
+            - project:
+                # I should be able to parameterize this, right?
+                - 'build-wily-amd64-package'
+                - 'build-wily-i386-package'
+              block: true
+              predefined-parameters: |
+                landing_candidate=$landing_candidate
+                target_branch={target_branch}
+                hooks={hooks}
+                parallelism={parallelism}
+
+- builder:
+    name: 'trigger-generic-update-mp'
+    builders:
+        - trigger-builds:
+            - project:
+                - 'generic-update-mp'
+              block: 'true'
+              predefined-parameters: |
+                ci_result=PASSED
+                ci_build=$BUILD_URL
+                ci_branch=$landing_candidate
+                ci_merge_proposal=$merge_proposal
+                ci_revision=$candidate_revision
+              block-thresholds:
+                build-step-failure-threshold: 'never'
+                unstable-threshold: 'never'
+                failure-threshold: 'never'
+
+- publisher:
+    name: 'trigger-generic-update-mp-failed'
+    publishers:
+        - trigger-parameterized-builds:
+          - project:
+                - 'generic-update-mp'
+            condition: 'UNSTABLE_OR_WORSE'
+            predefined-parameters: |
+                ci_result=FAILED
+                ci_build=$BUILD_URL
+                ci_branch=$landing_candidate
+                ci_merge_proposal=$merge_proposal
+                ci_revision=$candidate_revision
+
+# Can I parameterize a builder?
+- builder:
+    name: 'upcopy-amd64-from-downstream'
+    builders:
+        - copyartifact:
+            project: 'build-wily-amd64-package'
+            filter: '**/*.deb'
+            target: ./wily-amd64
+
+- builder:
+    name: 'upcopy-i386-from-downstream'
+    builders:
+        - copyartifact:
+            project: 'build-wily-i386-package'
+            filter: '**/*.deb'
+            target: ./wily-i386
+
+- publisher:
+    name: 'collect-debs'
+    publishers:
+        - archive:
+            artifacts: '**/*.deb'
+            only-if-success: 'true'
+
+- project:
+    name: 'ci-setup'
+    jobs:
+        - 'create-debian-chroot'
+        - 'update-debian-chroot'
+
+- project:
+    name: 'ci-details'
+    jobs:
+        - 'build-{dist}-{arch}-package'
+        - 'trigger-{target_project}-ci'
+        - '{target_project}-ci'
+        - 'generic-update-mp'
+    target_project:
+        # Must be projects lp:<name>
+        - 'autopilot'
+        - 'auto-upgrade-testing'
+    dist:
+        - wily
+    arch:
+        - amd64
+        - i386
+
+- job-template:
+    name: 'generic-update-mp'
+    description: |
+        Update a lanchpad merge proposal with the results of the CI run (Pass/Fail etc.)
+    node: 'jenkins-slave-2'
+    builders:
+        - shell: |
+            voteOnMergeProposal --status="$ci_result" --build-url="$ci_build" --branch="$ci_branch" --merge-proposal="$ci_merge_proposal" --revision="$ci_revision"
+    parameters:
+        - string:
+            name: ci_result
+            description: 'The result of the CI build'
+        - string:
+            name: ci_build
+            description: 'The jenkins URL of the build'
+        - string:
+            name: ci_branch
+            description: 'The launchpad branch that was processed'
+        - string:
+            name: ci_merge_proposal
+            description: 'The launchpad merge proposal that was processed'
+        - string:
+            name: ci_revision
+            description: 'The revision of the processed branch'
+
+- job-template:
+    name: 'trigger-{target_project}-ci'
+    description: |
+        Poll and trigger {target_project}-ci when there is a new MP that needs it.
+    node: 'jenkins-slave-2'
+    triggers:
+      - timed: 'H/5 * * * *'
+    builders:
+      - shell: |
+            BRANCH_NAME=lp:{target_project}
+            TARGET_JOB={target_project}-ci
+            # TARGET_JOB needs to already exist.
+
+            launchpadTrigger --lock-name=$JOB_NAME --job=$TARGET_JOB --branch=$BRANCH_NAME --trigger-ci
+
+            exit 0
+
+- job-template:
+    name: '{target_project}-ci'
+    description: |
+        Build packages and run tests for an {target_project} merge proposal
+    node: 'jenkins-slave-2'
+    # This token is the string that was setup for the launchpad bot.
+    auth-token: BUILD_ME
+    builders:
+        - trigger-deb-builders:
+            landing_candidate: {landing_candidate}
+            target_branch: 'lp:{target_project}'
+            hooks: 'H05set_package_version H10strip_native_depends'
+            parallelism: 1
+        - upcopy-i386-from-downstream
+        - upcopy-amd64-from-downstream
+        - trigger-generic-update-mp
+    publishers:
+        - collect-debs
+        - trigger-generic-update-mp-failed
+    parameters:
+        - string:
+            name: landing_candidate
+            description: The launchpad branch to process
+        - string:
+            name: candidate_revision
+            description: The revision of the branch
+        - string:
+            name: merge_proposal
+            description: The merge proposal to process
+
+- job-template:
+    name: 'create-debian-chroot'
+    description: |
+        Create the required chroots for pbuilder.
+    node: 'jenkins-slave-2'
+    parameters:
+        - string:
+            name: arches
+            default: 'amd64 i386'
+            description: 'Architectures to create machines of'
+        - string:
+            name: distributions
+            default: 'wily'
+            description: 'Distributions to create machines of'
+    builders:
+        - clear-artifacts
+        - create-chroot
+
+- job-template:
+    name: 'update-debian-chroot'
+    description: |
+        Create the required chroots for pbuilder.
+    node: 'jenkins-slave-2'
+    parameters:
+        - string:
+            name: arches
+            default: 'amd64 i386'
+            description: 'Architectures to create machines of'
+        - string:
+            name: distributions
+            default: 'wily'
+            description: 'Distributions to create machines of'
+    builders:
+        - clear-artifacts
+        - update-chroot
+
+- job-template:
+    name: 'build-{dist}-{arch}-package'
+    description: |
+        Build a debian package for {dist} - {arch}.
+    node: 'jenkins-slave-2'
+    # TODO: Explore if concurrent will work for us.
+    # concurrent: 'true' ?
+    parameters:
+        - string:
+            name: landing_candidate
+            description: 'LP branch to merge and build.'
+        - string:
+            name: target_branch
+            description: 'LP branch to merge into and build from.'
+        - string:
+            name: hooks
+            description: 'Build hooks to run during the build.'
+        - string:
+            name: parallelism
+            description: 'Level of parallelism to use.'
+    builders:
+        - clear-artifacts
+        - build-package:
+            dist: '{dist}'
+            arch: '{arch}'
+            landing_candidate: {landing_candidate}
+            target_branch: {target_branch}
+            hooks: {hooks}
+            parallelism: {parallelism}
+    publishers:
+        - collect-debs
+
+- project:
+    name: 'project-autolanding'
+    description: |
+      Autolanding jobs for projects in QA control
+    jobs:
+      - '{target_project}-autoland'
+      - 'trigger-{target_project}-autoland'
+      - 'generic-land-mp'
+    target_project:
+      # Must be lp:<name>
+      - 'autopilot'
+      - 'auto-upgrade-testing'
+
+- job-template:
+    name: '{target_project}-autoland'
+    description: |
+      Gets triggered when a MP exists that meets autoland criteria. Merges with
+      {target_project} trunk.
+    node: 'jenkins-slave-2'
+    auth-token: BUILD_ME
+    parameters:
+      - string:
+          name: 'landing_candidate'
+          description: 'The launchpad branch to process'
+      - string:
+          name: 'candidate_revision'
+          description: 'The revision of the branch'
+      - string:
+          name: 'merge_proposal'
+          description: 'The merge proposal to process'
+    builders:
+      - trigger-deb-builders:
+            landing_candidate: {landing_candidate}
+            target_branch: 'lp:{target_project}'
+            hooks: 'H05set_package_version H10strip_native_depends'
+            parallelism: 1
+      - 'trigger-generic-land-mp'
+    publishers:
+      - 'trigger-generic-land-mp-failed'
+
+- builder:
+    name: 'trigger-generic-land-mp'
+    description: |
+      Run when an autolanding job succeeds to update the MP with the success details.
+    builders:
+      - trigger-builds:
+          - project:
+              - 'generic-land-mp'
+            block: 'true'
+            predefined-parameters: |
+              ci_result=PASSED
+              ci_build=$BUILD_URL
+              ci_branch=$landing_candidate
+              ci_merge_proposal=$merge_proposal
+              ci_revision=$candidate_revision
+            block-thresholds:
+              build-step-failure-threshold: 'never'
+              unstable-threshold: 'never'
+              failure-threshold: 'never'
+
+- publisher:
+    name: 'trigger-generic-land-mp-failed'
+    description: |
+      Run when an autolanding job fails to update the MP with the failure details.
+    publishers:
+        - trigger-parameterized-builds:
+          - project:
+                - 'generic-land-mp'
+            condition: 'UNSTABLE_OR_WORSE'
+            predefined-parameters: |
+                ci_result=FAILED
+                ci_build=$BUILD_URL
+                ci_branch=$landing_candidate
+                ci_merge_proposal=$merge_proposal
+                ci_revision=$candidate_revision
+
+- job-template:
+    name: 'trigger-{target_project}-autoland'
+    description: |
+      Poll and trigger {target_project}-ci when there is a new MP that needs it.
+    node: 'jenkins-slave-2'
+    triggers:
+      - timed: 'H/5 * * * *'
+    builders:
+      - shell: |
+            #!/bin/bash
+            set -x
+
+            BRANCH_NAME=lp:{target_project}
+            TARGET_JOB={target_project}-autoland
+
+            launchpadTrigger --lock-name=$JOB_NAME --job=$TARGET_JOB --branch=$BRANCH_NAME --autoland
+
+            exit 0
+
+- job-template:
+    name: 'generic-land-mp'
+    description: |
+      Updates a MP with the autoland results (Success/Failure)
+    node: 'jenkins-slave-2'
+    parameters:
+      - string:
+          name: 'ci_result'
+          description: 'The result of the CI build'
+      - string:
+          name: 'ci_build'
+          description: 'The jenkins URL of the build'
+      - string:
+          name: 'ci_branch'
+          description: 'The launchpad branch that was processed'
+      - string:
+          name: 'ci_merge_proposal'
+          description: 'The launchpad merge proposal that was processed'
+      - string:
+          name: 'ci_revision'
+          description: 'The revision of the processed branch'
+    builders:
+      - shell: |
+            autoland --test-result="$ci_result" --build-job-url="$ci_build" --merge-proposal="$ci_merge_proposal" --revision="$ci_revision"