Merge ~morphis/snappy-hwe-snaps/+git/build-scripts:feature/run-tests-within-docker into ~snappy-hwe-team/snappy-hwe-snaps/+git/build-scripts:master

Proposed by Simon Fels on 2017-02-17
Status: Merged
Approved by: Konrad Zapałowicz on 2017-02-20
Approved revision: 6035e9e72396bc753574a684acbf6258be140f7e
Merged at revision: 52ee645d02884e285848db8844097846587f4471
Proposed branch: ~morphis/snappy-hwe-snaps/+git/build-scripts:feature/run-tests-within-docker
Merge into: ~snappy-hwe-team/snappy-hwe-snaps/+git/build-scripts:master
Diff against target: 237 lines (+96/-129)
2 files modified
dev/null (+0/-129)
jobs/generic-test-snap (+96/-0)
Reviewer Review Type Date Requested Status
Konrad Zapałowicz (community) code Approve on 2017-02-20
Jim Hodapp (community) code Approve on 2017-02-17
System Enablement Bot continuous-integration 2017-02-17 Approve on 2017-02-17
Review via email: mp+317595@code.launchpad.net

Description of the change

Rework spread test job to use Docker instead of a custom chroot.

Running all our tests inside a Docker container solves different problems:

 * We don't have to care about a custom chroot
 * We can construct it when needed directly on the remote worker
 * It simplifies the job script a bit and we can clearly focus on running our tests

Test job for this is https://jenkins.canonical.com/system-enablement/job/morphis-generic-run-snap-spread-tests/11/

IMPORTANT: When merged we need to manually rename the generic-run-spread-tests job and all its reference on our jenkins server as the job is renamed with this MP to generic-test-snap.

To post a comment you must log in.
Jim Hodapp (jhodapp) wrote :

A few comments inline below.

review: Needs Fixing
Jim Hodapp (jhodapp) wrote :

LGTM

review: Approve (code)
Konrad Zapałowicz (kzapalowicz) wrote :

top-approving on behalf of Jim as he has already Approved this MR anyway.

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/jobs/generic-run-snap-spread-tests b/jobs/generic-run-snap-spread-tests
2deleted file mode 100644
3index ce3d49b..0000000
4--- a/jobs/generic-run-snap-spread-tests
5+++ /dev/null
6@@ -1,129 +0,0 @@
7-#!/bin/sh
8-#
9-# Copyright (C) 2016 Canonical Ltd
10-#
11-# This program is free software: you can redistribute it and/or modify
12-# it under the terms of the GNU General Public License version 3 as
13-# published by the Free Software Foundation.
14-#
15-# This program is distributed in the hope that it will be useful,
16-# but WITHOUT ANY WARRANTY; without even the implied warranty of
17-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
18-# GNU General Public License for more details.
19-#
20-# You should have received a copy of the GNU General Public License
21-# along with this program. If not, see <http://www.gnu.org/licenses/>.
22-
23-set -x
24-set -e
25-
26-SSH_PATH="${JENKINS_HOME}/.ssh/"
27-SSH_KEY_PATH="${SSH_PATH}/bazaar.launchpad.net/system-enablement-ci-bot"
28-
29-SSH="ssh -i $SSH_KEY_PATH/id_rsa $REMOTE_USER@$REMOTE_WORKER"
30-SCP="scp -i $SSH_KEY_PATH/id_rsa"
31-
32-REMOTE_WORKSPACE=/home/$REMOTE_USER/$BUILD_TAG
33-
34-# Do a quick check if tests are maybe disabled for the repository
35-# if not applicable.
36-tmpdir=`mktemp -d`
37-git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO $tmpdir/src
38-if [ -e $tmpdir/src/.ci_tests_disabled ]; then
39- echo "Component has no CI tests so not running anything here"
40- exit 0
41-fi
42-
43-$SSH mkdir -p $REMOTE_WORKSPACE/bin
44-
45-cat << EOF > $WORKSPACE/build-spread.sh
46-#!/bin/bash
47-set -x
48-set -e
49-
50-cd $REMOTE_WORKSPACE
51-
52-export TERM=linux
53-export DEBIAN_FRONTEND=noninteractive
54-
55-# At this time it's necessary to build spread manually because
56-# the snapped version does not include the qemu/kvm backend.
57-# Once the snapped version includes this backend, then we can
58-# change the manual building of spread with making sure the snap
59-# package is installed.
60-export GOPATH=`mktemp -d`
61-go get -d -v github.com/snapcore/spread/...
62-go build github.com/snapcore/spread/cmd/spread
63-cp spread $REMOTE_WORKSPACE/bin
64-# Remove old executables of spread
65-if [ -e /usr/local/bin/spread ]; then
66- rm /usr/local/bin/spread
67-fi
68-PATH=$REMOTE_WORKSPACE/bin:$PATH which spread
69-rm -rf $GOPATH
70-EOF
71-
72-$SCP $WORKSPACE/build-spread.sh $REMOTE_USER@$REMOTE_WORKER:$REMOTE_WORKSPACE/
73-$SSH chmod u+x $REMOTE_WORKSPACE/build-spread.sh
74-$SSH sudo systemd-run --scope $REMOTE_WORKSPACE/build-spread.sh
75-
76-if [ $($SSH snap list | grep ubuntu-image | wc -l) -eq 0 ] ; then
77- $SSH sudo snap install --edge --devmode ubuntu-image
78-fi
79-
80-ensure_package_is_installed() {
81- status=$($SSH dpkg-query --show --showformat='${db:Status-Status}\n' "$1")
82- if [ "$status" != "installed" ] ; then
83- # Ignoring failures or for the tiny change of conflicting
84- # with another job running in parallel and trying the same.
85- $SSH sudo apt install -y --force-yes "$1" || true
86- fi
87-}
88-
89-ensure_package_is_installed kpartx
90-ensure_package_is_installed qemu-kvm
91-
92-cat << EOF > $WORKSPACE/run-spread-tests.sh
93-#!/bin/bash
94-set -x
95-set -e
96-
97-export TERM=linux
98-export DEBIAN_FRONTEND=noninteractive
99-export PATH=$REMOTE_WORKSPACE/bin:$PATH
100-
101-cd $REMOTE_WORKSPACE
102-
103-if [ -e "src_branch" ] ; then
104- rm -rf src_branch
105-fi
106-git clone -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO src_branch
107-cd src_branch
108-# We require either a run-tests.sh interface script for the spread tests
109-# or the basic spread.yaml spread test definition file, otherwise we fail
110-# the Jenkins job. Spread tests are required for all MRs.
111-if [ ! -e "run-tests.sh" ] && [ ! -e "spread.yaml" ]; then
112- echo "ERROR: missing spread test: you must provide a spread test"
113- exit 1
114-fi
115-if [ -e "run-tests.sh" ] ; then
116- if [ ! -z "$CHANNEL" ] ; then
117- ./run-tests.sh --channel=$CHANNEL --test-from-channel --debug --force-new-image
118- else
119- ./run-tests.sh --debug --force-new-image
120- fi
121-else
122- if [ ! -z "$CHANNEL" ] ; then
123- SNAP_CHANNEL=$CHANNEL spread -debug
124- else
125- spread -debug
126- fi
127-fi
128-rm -rf src_branch
129-EOF
130-
131-$SCP $WORKSPACE/run-spread-tests.sh $REMOTE_USER@$REMOTE_WORKER:$REMOTE_WORKSPACE/
132-$SSH chmod u+x $REMOTE_WORKSPACE/run-spread-tests.sh
133-$SSH sudo systemd-run --scope $REMOTE_WORKSPACE/run-spread-tests.sh
134-
135-$SSH sudo rm -rf $REMOTE_WORKSPACE
136diff --git a/jobs/generic-test-snap b/jobs/generic-test-snap
137new file mode 100644
138index 0000000..840df2c
139--- /dev/null
140+++ b/jobs/generic-test-snap
141@@ -0,0 +1,96 @@
142+#!/bin/sh
143+#
144+# Copyright (C) 2016 Canonical Ltd
145+#
146+# This program is free software: you can redistribute it and/or modify
147+# it under the terms of the GNU General Public License version 3 as
148+# published by the Free Software Foundation.
149+#
150+# This program is distributed in the hope that it will be useful,
151+# but WITHOUT ANY WARRANTY; without even the implied warranty of
152+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
153+# GNU General Public License for more details.
154+#
155+# You should have received a copy of the GNU General Public License
156+# along with this program. If not, see <http://www.gnu.org/licenses/>.
157+
158+set -ex
159+
160+SSH_PATH="${JENKINS_HOME}/.ssh/"
161+SSH_KEY_PATH="${SSH_PATH}/bazaar.launchpad.net/system-enablement-ci-bot"
162+
163+SSH="ssh -i $SSH_KEY_PATH/id_rsa $REMOTE_USER@$REMOTE_WORKER"
164+SCP="scp -i $SSH_KEY_PATH/id_rsa"
165+
166+REMOTE_WORKSPACE=/home/$REMOTE_USER/$BUILD_TAG
167+
168+tmp_srcdir=`mktemp -d`
169+git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO $tmp_srcdir/src
170+# Components have the ability to disable CI tests if they can't provide any.
171+# This only accepted in a few cases and should be generally avoided.
172+if [ -e $tmp_srcdir/src/.ci_tests_disabled ]; then
173+ echo "WARNING: Component has no CI tests so not running anything here"
174+ exit 0
175+fi
176+# We require either a run-tests.sh interface script for the spread tests
177+# or the basic spread.yaml spread test definition file, otherwise we fail
178+# the Jenkins job. Spread tests are required for all MRs.
179+if [ ! -e "$tmp_srcdir/src/run-tests.sh" ] && [ ! -e "$tmp_srcdir/src/spread.yaml" ]; then
180+ echo "ERROR: missing spread test: you must provide a spread test"
181+ exit 1
182+fi
183+rm -rf $tmp_srcdir
184+
185+$SSH sudo apt-get --yes --force-yes install docker.io
186+
187+cat << EOF > $WORKSPACE/run-tests.sh
188+#!/bin/sh
189+set -ex
190+
191+export TERM=linux
192+export DEBIAN_FRONTEND=noninteractive
193+export PATH=/build/bin:$PATH
194+
195+apt update
196+apt install -y --force-yes kpartx qemu-kvm ubuntu-image snapd golang-go git sudo
197+
198+# At this time it's necessary to build spread manually because
199+# the snapped version does not include the qemu/kvm backend.
200+# Once the snapped version includes this backend, then we can
201+# change the manual building of spread with making sure the snap
202+# package is installed.
203+export GOPATH=`mktemp -d`
204+go get -d -v github.com/snapcore/spread/...
205+go build github.com/snapcore/spread/cmd/spread
206+mkdir /build/bin
207+cp spread /build/bin
208+
209+git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO /build/src
210+cd /build/src
211+
212+if [ -e "run-tests.sh" ] ; then
213+ if [ ! -z "$CHANNEL" ] ; then
214+ ./run-tests.sh --channel=$CHANNEL --test-from-channel --debug --force-new-image
215+ else
216+ ./run-tests.sh --debug --force-new-image
217+ fi
218+else
219+ if [ ! -z "$CHANNEL" ] ; then
220+ SNAP_CHANNEL=$CHANNEL spread -debug
221+ else
222+ spread -debug
223+ fi
224+fi
225+EOF
226+
227+$SSH mkdir -p $REMOTE_WORKSPACE
228+$SCP $WORKSPACE/run-tests.sh $REMOTE_USER@$REMOTE_WORKER:$REMOTE_WORKSPACE
229+$SSH chmod u+x $REMOTE_WORKSPACE/run-tests.sh
230+
231+$SSH sudo docker run \
232+ -v /dev:/dev \
233+ -v $REMOTE_WORKSPACE:/build \
234+ --privileged \
235+ ubuntu:16.04 /build/run-tests.sh
236+
237+$SSH sudo rm -rf $REMOTE_WORKSPACE

Subscribers

People subscribed via source and target branches

to all changes: