Merge ~morphis/snappy-hwe-snaps/+git/build-scripts:feature/ephemeral-lp-builds into ~snappy-hwe-team/snappy-hwe-snaps/+git/build-scripts:master

[Simon Fels]

Rework snap-build job to create ephemeral snap builds on launchpad

This makes our build much more reliable as we offload the actual build
to launchpad and can also build for multiple architectures instead of
just building the host architecture our jenkins agents runs as.

We use the jenkins-ci-builds repository on launchpad as staging area
for our builds.

Example build using this is at https://jenkins.canonical.com/system-enablement/job/morphis-generic-build-snap/1/

This got reworked a bit more. We will now trigger one generic-snap-build-worker job per architecture from the generic-snap-build job so that have individual builds per architecture instead of a big one for all.

See https://jenkins.canonical.com/system-enablement/job/generic-build-snap-worker/ and https://jenkins.canonical.com/system-enablement/job/morphis-generic-build-snap/ for the proposed setup. generic-build-snap-worker will be triggered with the ARCH= flag set. generic-build-snap can be tuned at a later point to use a architecture whitelist for each snap.

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:0e5ea052de69a5e5bfa89a6c6348e6a5cdbc0a85
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/957/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-run-snap-spread-tests/662
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/865/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/957/rebuild

review: Approve (continuous-integration)

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:d65825d349232d4bc238453f230382b5f0e333da
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/959/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-run-snap-spread-tests/664
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/867/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/959/rebuild

review: Approve (continuous-integration)

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:e02bfc1dcc1c5d998544b0002ed11bcd002be0b5
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/960/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-run-snap-spread-tests/665
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/868/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/960/rebuild

review: Approve (continuous-integration)

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:6d15d3720cca53d5e3d45228b238a2b11acb92b1
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/966/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-run-snap-spread-tests/669
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/874/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/966/rebuild

review: Approve (continuous-integration)

[System Enablement Bot (system-enablement-ci-bot)]

FAILED: Continuous integration, rev:9e4ca884ce5c0cccc30ea7cb827ad0faea3f1e98
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/972/
Executed test runs:
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/879/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/972/rebuild

review: Needs Fixing (continuous-integration)

[System Enablement Bot (system-enablement-ci-bot)]

FAILED: Continuous integration, rev:ffbe526031a6c20092031b0ec5c3c7676c33d8c0
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/973/
Executed test runs:
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/880/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/973/rebuild

review: Needs Fixing (continuous-integration)

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:f7fc8794f3349df6f7628e9de8501625394e6223
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/974/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/674
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/882/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/974/rebuild

review: Approve (continuous-integration)

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:6d17c2793fa91939f755823d22e9cea9ed9eff9e
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/976/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/675
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/883/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/976/rebuild

review: Approve (continuous-integration)

[Konrad Zapałowicz (kzapalowicz)]

LGTM

review: Approve (code)

[Jim Hodapp (jhodapp)]

LGTM

review: Approve

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:68696845ab9e3e07106d6747ac5a8416b9e339a6
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1005/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/693
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/913/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1005/rebuild

review: Approve (continuous-integration)

[Konrad Zapałowicz (kzapalowicz)]

ack

review: Approve (code)

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:ecda8713547a287d878a1abe05a4230ee227c243
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1006/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-test-snap/694
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/914/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/1006/rebuild

review: Approve (continuous-integration)

[Konrad Zapałowicz (kzapalowicz)]

approving on behalf of Jim who is not in the EU timezone and the other maintainer is the author of this MP

diff --git a/jobs/generic-build-snap b/jobs/generic-build-snap
deleted file mode 100755
index 82c27cf..0000000
--- a/jobs/generic-build-snap
+++ /dev/null
@@ -1,27 +0,0 @@
-#!/bin/sh
-#
-# Copyright (C) 2016 Canonical Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 3 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-set -x
-
-sudo $WORKSPACE/build-scripts/scripts/snap-build \
-	$WORKSPACE \
-	$SERIES \
-	$TARGET_GIT_REPO \
-	$TARGET_GIT_REPO_BRANCH \
-	$SOURCE_GIT_REPO \
-	$SOURCE_GIT_REPO_BRANCH \
-	$REVISION \
-	$FORCE
\ No newline at end of file
diff --git a/jobs/generic-build-snap-worker b/jobs/generic-build-snap-worker
new file mode 100755
index 0000000..0cfd16e
--- /dev/null
+++ b/jobs/generic-build-snap-worker
@@ -0,0 +1,63 @@
+#!/bin/sh
+#
+# Copyright (C) 2016 Canonical Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+set -x
+
+[ -e $WORKSPACE/src ] && rm -rf $WORKSPACE/src
+
+git clone -b $TARGET_GIT_REPO_BRANCH $TARGET_GIT_REPO $WORKSPACE/src
+cd $WORKSPACE/src
+
+if [ ! -e snapcraft.yaml ] ; then
+	echo "ERROR: Failed to build snap as no snapcraft.yaml exists!"
+	exit 1
+fi
+
+git config user.name "System Enablement CI Bot"
+git config user.email "[e-mail]"
+
+if [ -n "$SOURCE_GIT_REPO" ]; then
+	git remote add other $SOURCE_GIT_REPO
+	git fetch other
+	git merge \
+		--no-ff \
+		-m "Merge remote tracking branch other/$SOURCE_GIT_REPO_BRANCH" \
+		$REVISION
+fi
+
+SNAP_NAME=$(grep "name:" snapcraft.yaml | awk '{print $2}')
+SNAP_REV=$(git rev-parse --short HEAD)
+CI_BRANCH=build/$SNAP_NAME-$BUILD_ID-$SNAP_REV
+git checkout -b $CI_BRANCH
+git remote add jenkins-ci git+ssh://[e-mail]/~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-ci-builds
+git push jenkins-ci $CI_BRANCH
+
+EXTRA_ARGS=
+if [ -n "$ARCHITECTURES" ]; then
+    EXTRA_ARGS="$EXTRA_ARGS --architectures=$ARCHITECTURES"
+fi
+
+$WORKSPACE/build-scripts/scripts/trigger-lp-build.py \
+    -s $SNAP_NAME -n \
+    --git-repo=https://git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-ci-builds \
+    --git-repo-branch=$CI_BRANCH \
+    $EXTRA_ARGS
+# When build has finished (regardless of its result) we remove our CI branch
+# to keep the CI repository clean.
+git push jenkins-ci :$CI_BRANCH
+if [ $? -ne 0 ]; then
+    exit 1
+fi
diff --git a/scripts/snap-build b/scripts/snap-build
deleted file mode 100755
index abe40b4..0000000
--- a/scripts/snap-build
+++ /dev/null
@@ -1,111 +0,0 @@
-#!/bin/bash
-#
-# Copyright (C) 2016 Canonical Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 3 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-set -x
-set -e
-
-# Fix certain builds that require a locale set
-export LC_ALL=C
-
-WORKSPACE=$1
-SERIES=$2
-TARGET_GIT_REPO=$3
-TARGET_GIT_REPO_BRANCH=$4
-SOURCE_GIT_REPO=$5
-SOURCE_GIT_REPO_BRANCH=$6
-REVISION=$7
-FORCE=$8
-
-BUILD_ROOTFS_TARBALL=$WORKSPACE/snap-build-$SERIES.tar
-
-if [ ! -e $BUILD_ROOTFS_TARBALL ] || [ "$REBUILD_ROOTFS" -eq "1" ] ; then
-	sudo $WORKSPACE/build-scripts/scripts/build-rootfs-create $SERIES $BUILD_ROOTFS_TARBALL
-fi
-
-cd $WORKSPACE
-
-# Cleanup environment a bit
-[ -e build ] && rm -rf build
-[ -e results ] && rm -rf results
-
-mkdir build
-mkdir results
-
-tar xf $BUILD_ROOTFS_TARBALL -C build
-
-cat << EOF > build/rootfs/do-build.sh
-#!/bin/bash
-set -x
-set -e
-
-export TERM=linux
-export DEBIAN_FRONTEND=noninteractive
-
-# HACK: As we're not able to talk with parts.snapcraft.io yet
-# we need to fake the parts registry here as otherwise we can't
-# process our builds.
-export SNAPCRAFT_PARTS_URI=http://dev.null
-export XDG_DATA_HOME=/build/xdg-data
-mkdir -p /build/xdg-data/snapcraft
-touch /build/xdg-data/snapcraft/parts.yaml
-echo "{If-Modified-Since: '`date`'}" > /build/xdg-data/snapcraft/headers.yaml
-
-# Add missing package sources to the system to get necessary
-# updates for snapcraft & co
-echo "deb http://de.archive.ubuntu.com/ubuntu/ $SERIES-updates main" >> /etc/apt/sources.list
-echo "deb http://archive.ubuntu.com/ubuntu/ $SERIES-updates universe" >> /etc/apt/sources.list
-
-# Make sure all required things are installed
-apt-get update
-apt-get -y --force-yes upgrade
-apt-get -y --force-yes install snapcraft git
-
-mkdir -p /build
-git clone -b $TARGET_GIT_REPO_BRANCH $TARGET_GIT_REPO /build/src
-cd /build/src
-
-git config user.name "System Enablement CI Bot"
-git config user.email "[e-mail]"
-
-if [ -n "$SOURCE_GIT_REPO" ]; then
-	git remote add other $SOURCE_GIT_REPO
-	git fetch other
-	git merge \
-		--no-ff \
-		-m "Merge remote tracking branch other/$SOURCE_GIT_REPO_BRANCH" \
-		$REVISION
-fi
-
-# Only attempt to build projects where we have a valid snapcraft project.
-# For all others we just make sure we can merge source to target.
-if [ ! -e "/build/src/snapcraft.yaml" ] && [ ! -e "/build/src/.snapcraft.yaml" ] ; then
-	echo "WARNING: Project does not contain a snapcraft.yaml file!"
-	exit 0
-fi
-
-find /build/src
-snapcraft --version
-snapcraft
-EOF
-
-chmod +x build/rootfs/do-build.sh
-chroot build/rootfs /do-build.sh
-
-# Copy snaps to their target directory where jenkins will
-# pick them up and provide as buid artifacts
-cp build/rootfs/build/src/*.snap results/ || true
-
-rm -rf build
diff --git a/scripts/trigger-lp-build.py b/scripts/trigger-lp-build.py
index 7f0f544..ed3d3e8 100755
--- a/scripts/trigger-lp-build.py
+++ b/scripts/trigger-lp-build.py
@@ -3,7 +3,11 @@
 import os
 import sys
 import time
+import random
 import smtplib
+import string
+import urllib2
+import zlib
 
 from datetime import datetime
 from os.path import basename
@@ -18,9 +22,21 @@ parser.add_argument('-s', '--snap', required=True,
                     help="Name of the snap to build")
 parser.add_argument('-p', '--publish', action='store_true',
                     help="Trigger a publish build instead of a daily (default)")
+parser.add_argument('-n', '--new', action='store_true', help="Create a new ephemeral snap build on launchpad")
+parser.add_argument('--git-repo', help="Git repository to be used for new ephemeral snap build")
+parser.add_argument('--git-repo-branch', help="Git repository branch to be used for new ephemeral snap build")
+parser.add_argument('-a', '--architectures', help="Specify architectures to build for. Separate multiple architectures by ','")
 
 args = vars(parser.parse_args())
 
+ephemeral_build=False
+
+if args['new']:
+    ephemeral_build=True
+    if not 'git_repo' in args or not 'git_repo_branch' in args:
+        print("ERROR: No git repository or a branch supplied")
+        sys.exit(1)
+
 series = 'xenial'
 
 lp_app = se_utils.get_config_option("lp_app")
@@ -29,28 +45,70 @@ credential_store_path = se_utils.get_config_option('credential_store_path')
 launchpad = se_utils.get_launchpad(None, credential_store_path, lp_app, lp_env)
 
 team = launchpad.people['snappy-hwe-team']
-
-build_name = "%s-daily" % args["snap"]
-if args["publish"] == True:
-    build_name = "%s-publish" % args["snap"]
-
-snap = launchpad.snaps.getByName(name=build_name, owner=team)
 ubuntu = launchpad.distributions['ubuntu']
 release = ubuntu.getSeries(name_or_version=series)
 primary_archive = ubuntu.getArchive(name='primary')
 
+snap=None
+if ephemeral_build:
+    snap_arches=[]
+    if 'architectures' in args and len(args['architectures']) > 0:
+        snap_arches = args["architectures"].split(",")
+
+    if len(snap_arches) == 0:
+        print("WARNING: No architectures to build specified. Will only build for amd64.")
+        snap_arches=["amd64"]
+
+    processors=[]
+    for arch in snap_arches:
+        try:
+            p = launchpad.processors.getByName(name=arch)
+            processors.append(p.self_link)
+        except:
+            print("ERROR: Failed to find processor for '{}' architecture".format(arch))
+            sys.exit(1)
+
+    build_name = 'ci-%s-%s' % (args["snap"], ''.join(random.choice(string.ascii_lowercase + string.digits) for _ in range(16)))
+    snap = launchpad.snaps.new(name=build_name,
+                        processors=processors,
+                        auto_build=False, distro_series=release,
+                        git_repository_url=args['git_repo'],
+                        git_path='%s' % args["git_repo_branch"],
+                        owner=team)
+else:
+    build_name = "%s-daily" % args["snap"]
+    if args["publish"] == True:
+        build_name = "%s-publish" % args["snap"]
+    snap = launchpad.snaps.getByName(name=build_name, owner=team)
+
+if snap == None:
+    print("ERROR: Failed to create snap build on launchpad")
+
+# Not every snap is build agains all arches.
+arches = [processor.name for processor in snap.processors]
+if not ephemeral_build and "architectures" in args:
+    wanted_arches = args["architectures"].split(",")
+    possible_arches = []
+    for arch in wanted_arches:
+        if not arch in arches:
+            print("WARNING: Can't build snap for architecture {} as it is not enabled in the build job".format(args["snap"]))
+            continue
+        possible_arches.append(arch)
+    arches = possible_arches
+
+if len(arches) == 0:
+    print("ERROR: No architectures available to build for")
+    sys.exit(1)
+
 # Add a big fat warning that we don't really care about fixing things when
 # the job will be canceled after the following lines are printed out.
-print("!!!!!!! POINT OF NO RETURN !!!!!!!")
+print("!!!!!!! POINT OF NO RETURN !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
 print("DO NOT CANCEL THIS JOB AFTER THIS OR BAD THINGS WILL HAPPEN")
-print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
+print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
 
 stamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
 print("Trying to trigger builds at: {}".format(stamp))
 
-# Not every snap is build agains all arches.
-arches = [processor.name for processor in snap.processors]
-
 # We will now trigger a build for each whitelisted architecture, collect the
 # build job url and the wait for all builds to finish and collect their results
 # to vote for a successful or failed build.
@@ -60,7 +118,7 @@ for build_arch in arches:
     request = snap.requestBuild(archive=primary_archive, distro_arch_series=arch, pocket='Proposed')
     build_id = str(request).rsplit('/', 1)[-1]
     triggered_builds.append(build_id)
-    print("Arch: {} is building under: {}".format(build_arch, request))
+    print("Arch: {} is building under: {}".format(build_arch, request.web_link))
 
 failures = []
 while len(triggered_builds):
@@ -90,12 +148,21 @@ if len(failures):
         except:
             print ("Could not get failure data for {} (was there an LP timeout?)".format(build))
             continue
-        buildlog = response[build]['build_log_url']
+        buildlog = response[failure]['build_log_url']
         if buildlog != 'None':
-            print(buildlog)
             arch = str(buildlog).split('_')[4]
         print("{} snap {} build at {} failed for id: {} log: {}".format(args["snap"], arch, stamp, failure, buildlog))
+        # For ephermal builds we need to print out the log file as it will be gone after
+        # the launchpad build is removed.
+        if ephemeral_build:
+            response = urllib2.urlopen(buildlog)
+            log_data = zlib.decompress(response.read(), 16+zlib.MAX_WBITS)
+            print(log_data)
 
+if ephemeral_build:
+    snap.lp_delete()
+
+if len(failures):
     # Let the build fail as at least a single snap has failed to build
     sys.exit(1)