snappy-hwe-snaps

Merge ~morphis/snappy-hwe-snaps/+git/build-scripts:feature/run-tests-within-docker into ~snappy-hwe-team/snappy-hwe-snaps/+git/build-scripts:master

  1. Git
  2. lp:~morphis/snappy-hwe-snaps/+git/build-scripts
  3. feature/run-tests-within-docker
  4. Merge into master
Proposed by Simon Fels
Status: Merged
Approved by: Konrad Zapałowicz
Approved revision: 6035e9e72396bc753574a684acbf6258be140f7e
Merged at revision: 52ee645d02884e285848db8844097846587f4471
Proposed branch: ~morphis/snappy-hwe-snaps/+git/build-scripts:feature/run-tests-within-docker
Merge into: ~snappy-hwe-team/snappy-hwe-snaps/+git/build-scripts:master
Diff against target: 237 lines (+96/-129)
2 files modified
dev/null (+0/-129)
jobs/generic-test-snap (+96/-0)
Reviewer Review Type Date Requested Status
Konrad Zapałowicz (community) code Approve
Jim Hodapp (community) code Approve
System Enablement Bot continuous-integration Approve
Review via email: mp+317595@code.launchpad.net

Description of the change

Rework spread test job to use Docker instead of a custom chroot.

Running all our tests inside a Docker container solves different problems:

 * We don't have to care about a custom chroot
 * We can construct it when needed directly on the remote worker
 * It simplifies the job script a bit and we can clearly focus on running our tests

Test job for this is https://jenkins.canonical.com/system-enablement/job/morphis-generic-run-snap-spread-tests/11/

IMPORTANT: When merged we need to manually rename the generic-run-spread-tests job and all its reference on our jenkins server as the job is renamed with this MP to generic-test-snap.

To post a comment you must log in.
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:3b4b36e97b6cd03d61f7af5298303d3537c25b7f
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/958/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-run-snap-spread-tests/663
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/866/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/958/rebuild

review: Approve (continuous-integration)
Revision history for this message
Jim Hodapp (jhodapp) wrote :

A few comments inline below.

review: Needs Fixing
Revision history for this message
Simon Fels (morphis) wrote :

Build testing last changes: https://jenkins.canonical.com/system-enablement/job/morphis-generic-test-snap/12/

Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :

PASSED: Continuous integration, rev:6035e9e72396bc753574a684acbf6258be140f7e
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/965/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/system-enablement/job/generic-run-snap-spread-tests/670
    None: https://jenkins.canonical.com/system-enablement/job/generic-update-snap-mp/873/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/system-enablement/job/generic-build-snap/965/rebuild

review: Approve (continuous-integration)
Revision history for this message
Jim Hodapp (jhodapp) wrote :

LGTM

review: Approve (code)
Revision history for this message
Konrad Zapałowicz (kzapalowicz) wrote :

top-approving on behalf of Jim as he has already Approved this MR anyway.

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/jobs/generic-run-snap-spread-tests b/jobs/generic-run-snap-spread-tests
0deleted file mode 1006440deleted file mode 100644
index ce3d49b..0000000
--- a/jobs/generic-run-snap-spread-tests
+++ /dev/null
@@ -1,129 +0,0 @@
1#!/bin/sh
2#
3# Copyright (C) 2016 Canonical Ltd
4#
5# This program is free software: you can redistribute it and/or modify
6# it under the terms of the GNU General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# This program is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU General Public License for more details.
13#
14# You should have received a copy of the GNU General Public License
15# along with this program. If not, see <http://www.gnu.org/licenses/>.
16
17set -x
18set -e
19
20SSH_PATH="${JENKINS_HOME}/.ssh/"
21SSH_KEY_PATH="${SSH_PATH}/bazaar.launchpad.net/system-enablement-ci-bot"
22
23SSH="ssh -i $SSH_KEY_PATH/id_rsa $REMOTE_USER@$REMOTE_WORKER"
24SCP="scp -i $SSH_KEY_PATH/id_rsa"
25
26REMOTE_WORKSPACE=/home/$REMOTE_USER/$BUILD_TAG
27
28# Do a quick check if tests are maybe disabled for the repository
29# if not applicable.
30tmpdir=`mktemp -d`
31git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO $tmpdir/src
32if [ -e $tmpdir/src/.ci_tests_disabled ]; then
33 echo "Component has no CI tests so not running anything here"
34 exit 0
35fi
36
37$SSH mkdir -p $REMOTE_WORKSPACE/bin
38
39cat << EOF > $WORKSPACE/build-spread.sh
40#!/bin/bash
41set -x
42set -e
43
44cd $REMOTE_WORKSPACE
45
46export TERM=linux
47export DEBIAN_FRONTEND=noninteractive
48
49# At this time it's necessary to build spread manually because
50# the snapped version does not include the qemu/kvm backend.
51# Once the snapped version includes this backend, then we can
52# change the manual building of spread with making sure the snap
53# package is installed.
54export GOPATH=`mktemp -d`
55go get -d -v github.com/snapcore/spread/...
56go build github.com/snapcore/spread/cmd/spread
57cp spread $REMOTE_WORKSPACE/bin
58# Remove old executables of spread
59if [ -e /usr/local/bin/spread ]; then
60 rm /usr/local/bin/spread
61fi
62PATH=$REMOTE_WORKSPACE/bin:$PATH which spread
63rm -rf $GOPATH
64EOF
65
66$SCP $WORKSPACE/build-spread.sh $REMOTE_USER@$REMOTE_WORKER:$REMOTE_WORKSPACE/
67$SSH chmod u+x $REMOTE_WORKSPACE/build-spread.sh
68$SSH sudo systemd-run --scope $REMOTE_WORKSPACE/build-spread.sh
69
70if [ $($SSH snap list | grep ubuntu-image | wc -l) -eq 0 ] ; then
71 $SSH sudo snap install --edge --devmode ubuntu-image
72fi
73
74ensure_package_is_installed() {
75 status=$($SSH dpkg-query --show --showformat='${db:Status-Status}\n' "$1")
76 if [ "$status" != "installed" ] ; then
77 # Ignoring failures or for the tiny change of conflicting
78 # with another job running in parallel and trying the same.
79 $SSH sudo apt install -y --force-yes "$1" || true
80 fi
81}
82
83ensure_package_is_installed kpartx
84ensure_package_is_installed qemu-kvm
85
86cat << EOF > $WORKSPACE/run-spread-tests.sh
87#!/bin/bash
88set -x
89set -e
90
91export TERM=linux
92export DEBIAN_FRONTEND=noninteractive
93export PATH=$REMOTE_WORKSPACE/bin:$PATH
94
95cd $REMOTE_WORKSPACE
96
97if [ -e "src_branch" ] ; then
98 rm -rf src_branch
99fi
100git clone -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO src_branch
101cd src_branch
102# We require either a run-tests.sh interface script for the spread tests
103# or the basic spread.yaml spread test definition file, otherwise we fail
104# the Jenkins job. Spread tests are required for all MRs.
105if [ ! -e "run-tests.sh" ] && [ ! -e "spread.yaml" ]; then
106 echo "ERROR: missing spread test: you must provide a spread test"
107 exit 1
108fi
109if [ -e "run-tests.sh" ] ; then
110 if [ ! -z "$CHANNEL" ] ; then
111 ./run-tests.sh --channel=$CHANNEL --test-from-channel --debug --force-new-image
112 else
113 ./run-tests.sh --debug --force-new-image
114 fi
115else
116 if [ ! -z "$CHANNEL" ] ; then
117 SNAP_CHANNEL=$CHANNEL spread -debug
118 else
119 spread -debug
120 fi
121fi
122rm -rf src_branch
123EOF
124
125$SCP $WORKSPACE/run-spread-tests.sh $REMOTE_USER@$REMOTE_WORKER:$REMOTE_WORKSPACE/
126$SSH chmod u+x $REMOTE_WORKSPACE/run-spread-tests.sh
127$SSH sudo systemd-run --scope $REMOTE_WORKSPACE/run-spread-tests.sh
128
129$SSH sudo rm -rf $REMOTE_WORKSPACE
diff --git a/jobs/generic-test-snap b/jobs/generic-test-snap
130new file mode 1006440new file mode 100644
index 0000000..840df2c
--- /dev/null
+++ b/jobs/generic-test-snap
@@ -0,0 +1,96 @@
1#!/bin/sh
2#
3# Copyright (C) 2016 Canonical Ltd
4#
5# This program is free software: you can redistribute it and/or modify
6# it under the terms of the GNU General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# This program is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU General Public License for more details.
13#
14# You should have received a copy of the GNU General Public License
15# along with this program. If not, see <http://www.gnu.org/licenses/>.
16
17set -ex
18
19SSH_PATH="${JENKINS_HOME}/.ssh/"
20SSH_KEY_PATH="${SSH_PATH}/bazaar.launchpad.net/system-enablement-ci-bot"
21
22SSH="ssh -i $SSH_KEY_PATH/id_rsa $REMOTE_USER@$REMOTE_WORKER"
23SCP="scp -i $SSH_KEY_PATH/id_rsa"
24
25REMOTE_WORKSPACE=/home/$REMOTE_USER/$BUILD_TAG
26
27tmp_srcdir=`mktemp -d`
28git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO $tmp_srcdir/src
29# Components have the ability to disable CI tests if they can't provide any.
30# This only accepted in a few cases and should be generally avoided.
31if [ -e $tmp_srcdir/src/.ci_tests_disabled ]; then
32 echo "WARNING: Component has no CI tests so not running anything here"
33 exit 0
34fi
35# We require either a run-tests.sh interface script for the spread tests
36# or the basic spread.yaml spread test definition file, otherwise we fail
37# the Jenkins job. Spread tests are required for all MRs.
38if [ ! -e "$tmp_srcdir/src/run-tests.sh" ] && [ ! -e "$tmp_srcdir/src/spread.yaml" ]; then
39 echo "ERROR: missing spread test: you must provide a spread test"
40 exit 1
41fi
42rm -rf $tmp_srcdir
43
44$SSH sudo apt-get --yes --force-yes install docker.io
45
46cat << EOF > $WORKSPACE/run-tests.sh
47#!/bin/sh
48set -ex
49
50export TERM=linux
51export DEBIAN_FRONTEND=noninteractive
52export PATH=/build/bin:$PATH
53
54apt update
55apt install -y --force-yes kpartx qemu-kvm ubuntu-image snapd golang-go git sudo
56
57# At this time it's necessary to build spread manually because
58# the snapped version does not include the qemu/kvm backend.
59# Once the snapped version includes this backend, then we can
60# change the manual building of spread with making sure the snap
61# package is installed.
62export GOPATH=`mktemp -d`
63go get -d -v github.com/snapcore/spread/...
64go build github.com/snapcore/spread/cmd/spread
65mkdir /build/bin
66cp spread /build/bin
67
68git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO /build/src
69cd /build/src
70
71if [ -e "run-tests.sh" ] ; then
72 if [ ! -z "$CHANNEL" ] ; then
73 ./run-tests.sh --channel=$CHANNEL --test-from-channel --debug --force-new-image
74 else
75 ./run-tests.sh --debug --force-new-image
76 fi
77else
78 if [ ! -z "$CHANNEL" ] ; then
79 SNAP_CHANNEL=$CHANNEL spread -debug
80 else
81 spread -debug
82 fi
83fi
84EOF
85
86$SSH mkdir -p $REMOTE_WORKSPACE
87$SCP $WORKSPACE/run-tests.sh $REMOTE_USER@$REMOTE_WORKER:$REMOTE_WORKSPACE
88$SSH chmod u+x $REMOTE_WORKSPACE/run-tests.sh
89
90$SSH sudo docker run \
91 -v /dev:/dev \
92 -v $REMOTE_WORKSPACE:/build \
93 --privileged \
94 ubuntu:16.04 /build/run-tests.sh
95
96$SSH sudo rm -rf $REMOTE_WORKSPACE

Subscribers

People subscribed via source and target branches

to all changes: