Merge lp:~canonical-platform-qa/qa-jenkins-jobs/deployment-job into lp:qa-jenkins-jobs

[Max Brustkern]

This branch adds:
- A script to deploy jobs from trunk (this will be expanded in the future)
- A job to automatically branch trunk and run that script
- Job configurations to the package
- A dep8 test that verifies the deployment script can run a test deployment of the job configurations

Once this is landed and deployed, we'll have auto-deployment of anything landed in qa-jenkins-jobs. Since we'll also have a test, my next update will be autolanding of qa-jenkins-jobs. From there, we can work on making the job configurations automatically generated (i.e., the auto-upgrade-testing jobs will be populated based on the auto-upgrade-testing-specifications branch instead of a list that has to be updated separately.

[lp:~canonical-platform-qa/qa-jenkins-jobs/deployment-job]

60. By Max Brustkern on 2016-05-27

Install pip before running it

61. By Max Brustkern on 2016-06-02

Added README

[Allan LeSage (allanlesage)]

Ok thanks I think this is clearer!

review: Approve

=== modified file 'Makefile'
--- Makefile	2016-03-08 19:52:05 +0000
+++ Makefile	2016-06-02 19:06:01 +0000
@@ -6,3 +6,5 @@
 		NEWNAME=$$(basename "$$FILE" | sed 's/\.sh//'); \
 	    cp "$$FILE" "$$DESTDIR/usr/bin/$$NEWNAME"; \
 	done
+	mkdir -p "$$DESTDIR/usr/share"
+	cp -r jobs "$$DESTDIR/usr/share/qa-jenkins-jobs"

=== added file 'README'
--- README	1970-01-01 00:00:00 +0000
+++ README	2016-06-02 19:06:01 +0000
@@ -0,0 +1,51 @@
+qa-jenkins-jobs
+
+This package provides automated testing jobs for deployment using 
+jenkins-job-builder. It is primarily intended for deployed on the Canonical QA 
+Team jenkins-as-a-service instance at https://platform-qa-jenkins.ubuntu.com 
+
+- jobs:
+  This directory contains yaml files that work with jenkins-job-builder.
+  At present, they require no preparation to de deployed from source, but this 
+  is likely to change in the future. As such, deploying them directly is not 
+  recommended.
+
+- scripts:
+  This directory contains scripts used by the jobs as well as the deployment 
+  script.
+  - deploy-jobs.sh:
+    This script is maintained as the supported way to install the jobs. It 
+    Requires the jenkins_jobs module to be installed for python2. Python3 
+    support is planned once unicode issues are resolved.
+    Run without arguments, it will attempt to recursively install jobs from a
+    "jobs" directory in the parent directory if the script's location (i.e., if
+    the script is in /path/to/scripts/deploy-jobs.sh, it will use /path/to/jobs 
+    as the jobs directory.) It can accept a JOB_PATH environment variable to
+    use an arbitrary directory. If the "test" environment is passed, it will 
+    use the test argument instead of the update argument. For actual 
+    deployment, it expects a config file in the default j-j-b location 
+    (/etc/jenkins_jobs/jenkins_jobs.ini) in the format:
+
+[jenkins]
+user=myuser
+password=mypassword
+url=http://myjenkins.url
+
+- Makefile:
+  The makefile currently only contains an install target. This will install all 
+  scripts ending with .sh under the scripts directory in /usr/bin without the 
+  .sh extension. It will install the jobs directory at 
+  /usr/share/qa-jenkins-jobs
+
+- debian:
+  This contains debian packaging files, including one dep8 test. The dep8 test
+  runs deploy-jobs test with JOB_PATH set to /usr/share/qa-jenkins-jobs
+  This should verify that the jobs are installed correctly and can be processed 
+  into valid XML buy the jenkins_jobs module. It is relatively safe to run on 
+  arbitrary targets (only package files are installed, and no remote servers 
+  are accessed.)
+
+- Basic testing:
+  Run:
+    scripts/deploy-jobs.sh test
+  XML should print to stdout, and the exit status should be 0

=== modified file 'debian/control'
--- debian/control	2016-03-08 19:52:05 +0000
+++ debian/control	2016-06-02 19:06:01 +0000
@@ -3,8 +3,8 @@
 Priority: optional
 Maintainer: Max Brustkern <[e-mail]>
 Build-Depends: debhelper (>= 9)
-Standards-Version: 3.9.6
-Homepage: https://launchpade.net/qa-jenkins-jobs
+Standards-Version: 3.9.7
+Homepage: https://launchpad.net/qa-jenkins-jobs
 
 Package: qa-jenkins-jobs
 Architecture: any

=== added directory 'debian/tests'
=== added file 'debian/tests/control'
--- debian/tests/control	1970-01-01 00:00:00 +0000
+++ debian/tests/control	2016-06-02 19:06:01 +0000
@@ -0,0 +1,3 @@
+Tests: test-deploy
+Restrictions: allow-stderr
+Depends: python-jenkins-job-builder, qa-jenkins-jobs

=== added file 'debian/tests/test-deploy'
--- debian/tests/test-deploy	1970-01-01 00:00:00 +0000
+++ debian/tests/test-deploy	2016-06-02 19:06:01 +0000
@@ -0,0 +1,2 @@
+export JOB_PATH=/usr/share/qa-jenkins-jobs
+deploy-jobs test

=== added directory 'jobs'
=== added file 'jobs.yaml'
--- jobs.yaml	1970-01-01 00:00:00 +0000
+++ jobs.yaml	2016-06-02 19:06:01 +0000
@@ -0,0 +1,51 @@
+# vim: sw=4 ts=4 et
+
+# QA Jenkins Jobs
+# Copyright 2016 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# The project stanza describes all jobs and parameters
+- project:
+    name: qa-jenkins-jobs
+    node: 'iso-testing'
+    jobs:
+        - 'setup-qa-jobs'
+        - 'deploy-qa-jobs'
+
+- job:
+    name: setup-qa-jobs
+    description: |
+        This job provides dependencies for qa jobs.
+        It can be rerun to update them.
+        Defined in lp:qa-jenkins-job
+    builder:
+        - shell: |
+            pip install --user jenkins-job-builder
+
+- job:
+    name: deploy-qa-jobs
+    description: |
+        This job automatically deploys qa-jenkins-job from trunk periodically.
+        Defined in lp:qa-jenkins-jobs
+    node: 'iso-testing'
+    triggers:
+        - timed: 'H/30 * * * *'
+    builders:
+        - clear-artifacts:
+        - shell: |
+            BRANCH="lp:qa-jenkins-jobs"
+            BRANCH_DIR=qa-jenkins-jobs
+            bzr export "$BRANCH" "$BRANCH_DIR"
+            cd "$BRANCH_DIR"
+            ./scripts/deploy-jobs.sh

=== renamed directory 'auto-upgrade-testing' => 'jobs/auto-upgrade-testing'
=== renamed file 'defaults.yaml' => 'jobs/defaults.yaml'
=== renamed directory 'iso-testing' => 'jobs/iso-testing'
=== added file 'jobs/jobs.yaml'
--- jobs/jobs.yaml	1970-01-01 00:00:00 +0000
+++ jobs/jobs.yaml	2016-06-02 19:06:01 +0000
@@ -0,0 +1,52 @@
+# vim: sw=4 ts=4 et
+
+# QA Jenkins Jobs
+# Copyright 2016 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# The project stanza describes all jobs and parameters
+- project:
+    name: qa-jenkins-jobs
+    node: 'iso-testing'
+    jobs:
+        - 'setup-qa-jobs'
+        - 'deploy-qa-jobs'
+
+- job:
+    name: setup-qa-jobs
+    description: |
+        This job provides dependencies for qa jobs.
+        It can be rerun to update them.
+        Defined in lp:qa-jenkins-job
+    builder:
+        - shell: |
+            sudo apt-get install python-pip
+            pip install --user jenkins-job-builder
+
+- job:
+    name: deploy-qa-jobs
+    description: |
+        This job automatically deploys qa-jenkins-job from trunk periodically.
+        Defined in lp:qa-jenkins-jobs
+    node: 'iso-testing'
+    triggers:
+        - timed: '*/30 * * * *'
+    builders:
+        - clear-artifacts:
+        - shell: |
+            BRANCH="lp:qa-jenkins-jobs"
+            BRANCH_DIR=qa-jenkins-jobs
+            bzr export "$BRANCH_DIR" "$BRANCH"
+            cd "$BRANCH_DIR"
+            ./scripts/deploy-jobs.sh

=== renamed file 'macros.yaml' => 'jobs/macros.yaml'
=== renamed directory 'mp-ci-autoland' => 'jobs/mp-ci-autoland'
=== renamed directory 'ubiquity' => 'jobs/ubiquity'
=== renamed directory 'ubuntu-system-tests' => 'jobs/ubuntu-system-tests'
=== added file 'scripts/deploy-jobs.sh'
--- scripts/deploy-jobs.sh	1970-01-01 00:00:00 +0000
+++ scripts/deploy-jobs.sh	2016-06-02 19:06:01 +0000
@@ -0,0 +1,13 @@
+#!/bin/sh
+
+set -ex
+
+JOB_PATH=${JOB_PATH-"$(cd "$(dirname "$0")"; pwd)"/../jobs}
+case $1 in
+    test)
+        ARGS="test";;
+    *)
+        ARGS="update";;
+esac
+
+python -m jenkins_jobs.cmd $ARGS -r "$JOB_PATH"