Merge ~morphis/snappy-hwe-snaps/+git/build-scripts:feature/run-tests-within-docker into ~snappy-hwe-team/snappy-hwe-snaps/+git/build-scripts:master

[Simon Fels]

Rework spread test job to use Docker instead of a custom chroot.

Running all our tests inside a Docker container solves different problems:

 * We don't have to care about a custom chroot
 * We can construct it when needed directly on the remote worker
 * It simplifies the job script a bit and we can clearly focus on running our tests

Test job for this is https://jenkins.canonical.com/system-enablement/job/morphis-generic-run-snap-spread-tests/11/

IMPORTANT: When merged we need to manually rename the generic-run-spread-tests job and all its reference on our jenkins server as the job is renamed with this MP to generic-test-snap.

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:3b4b36e97b6cd03d61f7af5298303d3537c25b7f
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/958/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-run-snap-spread-tests/663
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/866/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/958/rebuild

review: Approve (continuous-integration)

[Jim Hodapp (jhodapp)]

A few comments inline below.

review: Needs Fixing

[Simon Fels (morphis)]

Build testing last changes: https://jenkins.canonical.com/system-enablement/job/morphis-generic-test-snap/12/

[System Enablement Bot (system-enablement-ci-bot)]

PASSED: Continuous integration, rev:6035e9e72396bc753574a684acbf6258be140f7e
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/965/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-run-snap-spread-tests/670
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/873/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/965/rebuild

review: Approve (continuous-integration)

[Jim Hodapp (jhodapp)]

LGTM

review: Approve (code)

[Konrad Zapałowicz (kzapalowicz)]

top-approving on behalf of Jim as he has already Approved this MR anyway.

review: Approve (code)

diff --git a/jobs/generic-run-snap-spread-tests b/jobs/generic-run-snap-spread-tests
deleted file mode 100644
index ce3d49b..0000000
--- a/jobs/generic-run-snap-spread-tests
+++ /dev/null
@@ -1,129 +0,0 @@
-#!/bin/sh
-#
-# Copyright (C) 2016 Canonical Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 3 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-set -x
-set -e
-
-SSH_PATH="${JENKINS_HOME}/.ssh/"
-SSH_KEY_PATH="${SSH_PATH}/bazaar.launchpad.net/system-enablement-ci-bot"
-
-SSH="ssh -i $SSH_KEY_PATH/id_rsa $REMOTE_USER@$REMOTE_WORKER"
-SCP="scp -i $SSH_KEY_PATH/id_rsa"
-
-REMOTE_WORKSPACE=/home/$REMOTE_USER/$BUILD_TAG
-
-# Do a quick check if tests are maybe disabled for the repository
-# if not applicable.
-tmpdir=`mktemp -d`
-git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO $tmpdir/src
-if [ -e $tmpdir/src/.ci_tests_disabled ]; then
-    echo "Component has no CI tests so not running anything here"
-    exit 0
-fi
-
-$SSH mkdir -p $REMOTE_WORKSPACE/bin
-
-cat << EOF > $WORKSPACE/build-spread.sh
-#!/bin/bash
-set -x
-set -e
-
-cd $REMOTE_WORKSPACE
-
-export TERM=linux
-export DEBIAN_FRONTEND=noninteractive
-
-# At this time it's necessary to build spread manually because
-# the snapped version does not include the qemu/kvm backend.
-# Once the snapped version includes this backend, then we can
-# change the manual building of spread with making sure the snap
-# package is installed.
-export GOPATH=`mktemp -d`
-go get -d -v github.com/snapcore/spread/...
-go build github.com/snapcore/spread/cmd/spread
-cp spread $REMOTE_WORKSPACE/bin
-# Remove old executables of spread
-if [ -e /usr/local/bin/spread ]; then
-    rm /usr/local/bin/spread
-fi
-PATH=$REMOTE_WORKSPACE/bin:$PATH which spread
-rm -rf $GOPATH
-EOF
-
-$SCP $WORKSPACE/build-spread.sh $REMOTE_USER@$REMOTE_WORKER:$REMOTE_WORKSPACE/
-$SSH chmod u+x $REMOTE_WORKSPACE/build-spread.sh
-$SSH sudo systemd-run --scope $REMOTE_WORKSPACE/build-spread.sh
-
-if [ $($SSH snap list | grep ubuntu-image | wc -l) -eq 0 ] ; then
-    $SSH sudo snap install --edge --devmode ubuntu-image
-fi
-
-ensure_package_is_installed() {
-    status=$($SSH dpkg-query --show --showformat='${db:Status-Status}\n' "$1")
-    if [ "$status" != "installed" ] ; then
-        # Ignoring failures or for the tiny change of conflicting
-        # with another job running in parallel and trying the same.
-        $SSH sudo apt install -y --force-yes "$1" || true
-    fi
-}
-
-ensure_package_is_installed kpartx
-ensure_package_is_installed qemu-kvm
-
-cat << EOF > $WORKSPACE/run-spread-tests.sh
-#!/bin/bash
-set -x
-set -e
-
-export TERM=linux
-export DEBIAN_FRONTEND=noninteractive
-export PATH=$REMOTE_WORKSPACE/bin:$PATH
-
-cd $REMOTE_WORKSPACE
-
-if [ -e "src_branch" ] ; then
-    rm -rf src_branch
-fi
-git clone -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO src_branch
-cd src_branch
-# We require either a run-tests.sh interface script for the spread tests
-# or the basic spread.yaml spread test definition file, otherwise we fail
-# the Jenkins job. Spread tests are required for all MRs.
-if [ ! -e "run-tests.sh" ] && [ ! -e "spread.yaml" ]; then
-    echo "ERROR: missing spread test: you must provide a spread test"
-    exit 1
-fi
-if [ -e "run-tests.sh" ] ; then
-    if [ ! -z "$CHANNEL" ] ; then
-        ./run-tests.sh --channel=$CHANNEL --test-from-channel --debug --force-new-image
-    else
-        ./run-tests.sh --debug --force-new-image
-    fi
-else
-    if [ ! -z "$CHANNEL" ] ; then
-        SNAP_CHANNEL=$CHANNEL spread -debug
-    else
-        spread -debug
-    fi
-fi
-rm -rf src_branch
-EOF
-
-$SCP $WORKSPACE/run-spread-tests.sh $REMOTE_USER@$REMOTE_WORKER:$REMOTE_WORKSPACE/
-$SSH chmod u+x $REMOTE_WORKSPACE/run-spread-tests.sh
-$SSH sudo systemd-run --scope $REMOTE_WORKSPACE/run-spread-tests.sh
-
-$SSH sudo rm -rf $REMOTE_WORKSPACE
diff --git a/jobs/generic-test-snap b/jobs/generic-test-snap
new file mode 100644
index 0000000..840df2c
--- /dev/null
+++ b/jobs/generic-test-snap
@@ -0,0 +1,96 @@
+#!/bin/sh
+#
+# Copyright (C) 2016 Canonical Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+set -ex
+
+SSH_PATH="${JENKINS_HOME}/.ssh/"
+SSH_KEY_PATH="${SSH_PATH}/bazaar.launchpad.net/system-enablement-ci-bot"
+
+SSH="ssh -i $SSH_KEY_PATH/id_rsa $REMOTE_USER@$REMOTE_WORKER"
+SCP="scp -i $SSH_KEY_PATH/id_rsa"
+
+REMOTE_WORKSPACE=/home/$REMOTE_USER/$BUILD_TAG
+
+tmp_srcdir=`mktemp -d`
+git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO $tmp_srcdir/src
+# Components have the ability to disable CI tests if they can't provide any.
+# This only accepted in a few cases and should be generally avoided.
+if [ -e $tmp_srcdir/src/.ci_tests_disabled ]; then
+    echo "WARNING: Component has no CI tests so not running anything here"
+    exit 0
+fi
+# We require either a run-tests.sh interface script for the spread tests
+# or the basic spread.yaml spread test definition file, otherwise we fail
+# the Jenkins job. Spread tests are required for all MRs.
+if [ ! -e "$tmp_srcdir/src/run-tests.sh" ] && [ ! -e "$tmp_srcdir/src/spread.yaml" ]; then
+    echo "ERROR: missing spread test: you must provide a spread test"
+    exit 1
+fi
+rm -rf $tmp_srcdir
+
+$SSH sudo apt-get --yes --force-yes install docker.io
+
+cat << EOF > $WORKSPACE/run-tests.sh
+#!/bin/sh
+set -ex
+
+export TERM=linux
+export DEBIAN_FRONTEND=noninteractive
+export PATH=/build/bin:$PATH
+
+apt update
+apt install -y --force-yes kpartx qemu-kvm ubuntu-image snapd golang-go git sudo
+
+# At this time it's necessary to build spread manually because
+# the snapped version does not include the qemu/kvm backend.
+# Once the snapped version includes this backend, then we can
+# change the manual building of spread with making sure the snap
+# package is installed.
+export GOPATH=`mktemp -d`
+go get -d -v github.com/snapcore/spread/...
+go build github.com/snapcore/spread/cmd/spread
+mkdir /build/bin
+cp spread /build/bin
+
+git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO /build/src
+cd /build/src
+
+if [ -e "run-tests.sh" ] ; then
+    if [ ! -z "$CHANNEL" ] ; then
+        ./run-tests.sh --channel=$CHANNEL --test-from-channel --debug --force-new-image
+    else
+        ./run-tests.sh --debug --force-new-image
+    fi
+else
+    if [ ! -z "$CHANNEL" ] ; then
+        SNAP_CHANNEL=$CHANNEL spread -debug
+    else
+        spread -debug
+    fi
+fi
+EOF
+
+$SSH mkdir -p $REMOTE_WORKSPACE
+$SCP $WORKSPACE/run-tests.sh $REMOTE_USER@$REMOTE_WORKER:$REMOTE_WORKSPACE
+$SSH chmod u+x $REMOTE_WORKSPACE/run-tests.sh
+
+$SSH sudo docker run \
+	-v /dev:/dev \
+	-v $REMOTE_WORKSPACE:/build \
+	--privileged \
+	ubuntu:16.04 /build/run-tests.sh
+
+$SSH sudo rm -rf $REMOTE_WORKSPACE