Merge ~morphis/snappy-hwe-snaps/+git/jenkins-jobs:introduce-ci-id-var into ~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-jobs:master

Proposed by Simon Fels
Status: Merged
Approved by: Alfonso Sanchez-Beato
Approved revision: 7d0f98c13cbd8f4b213a19b0ff1729b9929617cc
Merged at revision: 93c6b4fc075c28c48a8acce069fd6ea01dc89b21
Proposed branch: ~morphis/snappy-hwe-snaps/+git/jenkins-jobs:introduce-ci-id-var
Merge into: ~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-jobs:master
Diff against target: 406 lines (+176/-82)
11 files modified
jobs/snap/common-job-prepare.sh (+2/-1)
jobs/snap/snap-build-prepare.sh (+88/-0)
jobs/snap/snap-build-prepare.yaml (+63/-0)
jobs/snap/snap-build-worker.sh (+2/-62)
jobs/snap/snap-build-worker.yaml (+4/-0)
jobs/snap/snap-build.yaml (+5/-5)
jobs/snap/snap-cleanup.sh (+1/-0)
jobs/snap/snap-cleanup.yaml (+4/-4)
jobs/snap/snap-project-jobs.yaml (+1/-0)
jobs/snap/snap-test.sh (+3/-7)
jobs/snap/snap-test.yaml (+3/-3)
Reviewer Review Type Date Requested Status
Alfonso Sanchez-Beato Approve
System Enablement Bot continuous-integration Approve
Konrad Zapałowicz (community) Approve
Jim Hodapp (community) Approve
Review via email: mp+331269@code.launchpad.net

Description of the change

Introduce new CI_ID variable which is handed from the top-level snap-build job to all subsequently triggered jobs and will identify the CI iteration. CI_ID is filled from BUILD_ID which is unique for each snap-build run.

This MP also changes the snap-test job to consume the same CI repository as the build job does so we have to merge target and source only once.

To post a comment you must log in.
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

Thanks, just one comment, see below.

Also, do we have a different BUILD_TAG for each architecture? We would need that too, or move the part that creates the repo from snap-build-worker (one per arch) to snap-build.

Revision history for this message
Simon Fels (morphis) wrote :

Yeah, I have implemented a snap-build-prepare job locally already which will be the origin of the BUILD_ID variable and will prepare the repository once and for all subsequent builds.

Revision history for this message
Jim Hodapp (jhodapp) wrote :

LGTM

review: Approve
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Konrad Zapałowicz (kzapalowicz) wrote :

ack

review: Approve
Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

Looks good, just some minor nits.

review: Needs Fixing
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

LGTM, thanks for the changes

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/jobs/snap/common-job-prepare.sh b/jobs/snap/common-job-prepare.sh
2index 8c622ae..9b5b8f4 100644
3--- a/jobs/snap/common-job-prepare.sh
4+++ b/jobs/snap/common-job-prepare.sh
5@@ -47,11 +47,12 @@ cat << EOF > $WORKSPACE/.build_env
6 BOT_USERNAME={bot_username}
7 LAUNCHPAD_PROJECT={launchpad_project}
8 LAUNCHPAD_TEAM={launchpad_team}
9-SNAP_BUILD_JOB={name}-snap-build
10+SNAP_BUILD_JOB={name}-snap-build-prepare
11 BUILD_SCRIPTS=$WORKSPACE/jenkins-jobs
12 BUILD_ON_LAUNCHPAD={build_on_launchpad}
13 AUTO_MERGE={auto_merge}
14 TRIGGER_CI={trigger_ci}
15 UPDATE_MPS={update_mps}
16 RUN_TESTS={run_tests}
17+CI_REPO=ci-build-${{CI_ID}}
18 EOF
19diff --git a/jobs/snap/snap-build-prepare.sh b/jobs/snap/snap-build-prepare.sh
20new file mode 100644
21index 0000000..77940d1
22--- /dev/null
23+++ b/jobs/snap/snap-build-prepare.sh
24@@ -0,0 +1,88 @@
25+#!/bin/bash
26+#
27+# Copyright (C) 2017 Canonical Ltd
28+#
29+# This program is free software: you can redistribute it and/or modify
30+# it under the terms of the GNU General Public License version 3 as
31+# published by the Free Software Foundation.
32+#
33+# This program is distributed in the hope that it will be useful,
34+# but WITHOUT ANY WARRANTY; without even the implied warranty of
35+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
36+# GNU General Public License for more details.
37+#
38+# You should have received a copy of the GNU General Public License
39+# along with this program. If not, see <http://www.gnu.org/licenses/>.
40+
41+set -ex
42+
43+. "$WORKSPACE/.build_env"
44+
45+# CI_ID isn't specified yet when we're being called so initialize
46+# it here with the same value as we do in snap-build-prepare.yaml
47+if [ -z "$CI_ID" ]; then
48+ CI_ID=$BUILD_ID
49+fi
50+
51+rm -rf $WORKSPACE/src
52+
53+git clone --no-checkout $TARGET_GIT_REPO $WORKSPACE/src
54+cd $WORKSPACE/src
55+for remote in $(git branch -r | grep -v origin/master); do
56+ git checkout --track $remote
57+done
58+
59+git checkout $TARGET_GIT_REPO_BRANCH
60+
61+git config user.name "System Enablement CI Bot"
62+git config user.email "ce-system-enablement@lists.canonical.com"
63+
64+if [ -n "$SOURCE_GIT_REPO" ]; then
65+ git remote add other $SOURCE_GIT_REPO
66+ git fetch other
67+ git merge \
68+ --no-ff \
69+ -m "Merge remote tracking branch other/$SOURCE_GIT_REPO_BRANCH" \
70+ $REVISION
71+fi
72+
73+# Try to find the correct branch we need to build from. In the case that
74+# $TARGET_GIT_REPO_BRANCH points us to an upstream component branch we
75+# will take master as the next suitable candidate.
76+CI_BRANCH=
77+SNAPCRAFT_YAML_PATH=
78+for branch in $TARGET_GIT_REPO_BRANCH master ; do
79+ git checkout $branch
80+ if [ -e snapcraft.yaml ]; then
81+ SNAPCRAFT_YAML_PATH=snapcraft.yaml
82+ elif [ -e snap/snapcraft.yaml ]; then
83+ SNAPCRAFT_YAML_PATH=snap/snapcraft.yaml
84+ fi
85+
86+ if [ -n "$SNAPCRAFT_YAML_PATH" ]; then
87+ CI_BRANCH=$branch
88+ break
89+ fi
90+done
91+
92+if [ -z "$CI_BRANCH" ]; then
93+ echo "WARNING: Can't build snap as no snapcraft.yaml exists!"
94+ exit 0
95+fi
96+
97+REPO_NAME=$(awk -v a="$TARGET_GIT_REPO" 'BEGIN{print substr(a, index(a, "+git/") + 5)}')
98+# We rely on the snapcraft.yaml to have the snap name in the first five lines
99+# which is the case for all our snaps. This is a bit lazy but the best way to
100+# ensure we don't fetch any other name: fields which might be present in the file.
101+SNAP_NAME=$(cat $SNAPCRAFT_YAML_PATH | grep -v ^\# | head -n 5 | grep "^name:" | awk '{print $2}')
102+SNAP_REV=$(git rev-parse --short HEAD)
103+CI_REPO=ci-build-$CI_ID
104+
105+sed -i "s/~$LAUNCHPAD_TEAM\/$LAUNCHPAD_PROJECT\/+git\/$REPO_NAME/~$LAUNCHPAD_TEAM\/$LAUNCHPAD_PROJECT\/+git\/$CI_REPO/g" \
106+ $SNAPCRAFT_YAML_PATH
107+git add snapcraft.yaml
108+git commit -m "Update snapcraft.yaml to build from right repository"
109+
110+git remote add jenkins-ci git+ssh://$BOT_USERNAME@git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO
111+git push jenkins-ci --all
112+git push jenkins-ci --tags
113diff --git a/jobs/snap/snap-build-prepare.yaml b/jobs/snap/snap-build-prepare.yaml
114new file mode 100644
115index 0000000..49c504e
116--- /dev/null
117+++ b/jobs/snap/snap-build-prepare.yaml
118@@ -0,0 +1,63 @@
119+- job-template:
120+ name: '{name}-snap-build-prepare'
121+ project-type: freestyle
122+ defaults: global
123+ description: ""
124+ display-name: "{name}-snap-build-prepare"
125+ concurrent: true
126+ node: snap && misc
127+ properties:
128+ - build-discarder:
129+ num-to-keep: 10
130+ - rebuild
131+ parameters:
132+ - string:
133+ name: TARGET_GIT_REPO
134+ default:
135+ description: "Target git repository"
136+ - string:
137+ name: TARGET_GIT_REPO_BRANCH
138+ default: master
139+ description: "Branch of the target git repository to build from"
140+ - string:
141+ name: SERIES
142+ default: xenial
143+ description: "Ubuntu archive series to build for"
144+ - string:
145+ name: FORCE
146+ default: "0"
147+ description: "Set to 1 to force the build"
148+ - string:
149+ name: SOURCE_GIT_REPO
150+ default:
151+ description: "Source git repository"
152+ - string:
153+ name: SOURCE_GIT_REPO_BRANCH
154+ default:
155+ description: "Branch of the source git repository to use"
156+ - string:
157+ name: MERGE_PROPOSAL
158+ default:
159+ description: "Link to the merge proposal this build relates to"
160+ - string:
161+ name: REVISION
162+ default:
163+ description: "Cleanup the whole workspace"
164+ - string:
165+ name: CLEANUP_WORKSPACE
166+ default: "0"
167+ description: "Cleanup the whole workspace"
168+ builders:
169+ - shell:
170+ !include-raw:
171+ - common-job-prepare.sh
172+ - shell:
173+ !include-raw-escape:
174+ - snap-build-prepare.sh
175+ - trigger-builds:
176+ - project: '{name}-snap-build'
177+ current-parameters: true
178+ predefined-parameters: |
179+ CI_ID=$BUILD_ID
180+ RESULTS_ID=$BUILD_TAG
181+ block: true
182diff --git a/jobs/snap/snap-build-worker.sh b/jobs/snap/snap-build-worker.sh
183index 02ee65c..99a1f75 100644
184--- a/jobs/snap/snap-build-worker.sh
185+++ b/jobs/snap/snap-build-worker.sh
186@@ -18,60 +18,9 @@ set -ex
187
188 . "$WORKSPACE/.build_env"
189
190-rm -rf $WORKSPACE/src $WORKSPACE/results $WORKSPACE/build-props
191+rm -rf $WORKSPACE/src $WORKSPACE/results
192
193-git clone --no-checkout $TARGET_GIT_REPO $WORKSPACE/src
194-cd $WORKSPACE/src
195-for remote in `git branch -r | grep -v origin/master`; do git checkout --track $remote ; done
196-
197-git checkout $TARGET_GIT_REPO_BRANCH
198-
199-git config user.name "System Enablement CI Bot"
200-git config user.email "ce-system-enablement@lists.canonical.com"
201-
202-if [ -n "$SOURCE_GIT_REPO" ]; then
203- git remote add other $SOURCE_GIT_REPO
204- git fetch other
205- git merge \
206- --no-ff \
207- -m "Merge remote tracking branch other/$SOURCE_GIT_REPO_BRANCH" \
208- $REVISION
209-fi
210-
211-# Try to find the correct branch we need to build from. In the case that
212-# $TARGET_GIT_REPO_BRANCH points us to an upstream component branch we
213-# will take master as the next suitable candidate.
214-CI_BRANCH=
215-SNAPCRAFT_YAML_PATH=
216-for branch in $TARGET_GIT_REPO_BRANCH master ; do
217- git checkout $branch
218- if [ -e snapcraft.yaml ]; then
219- SNAPCRAFT_YAML_PATH=snapcraft.yaml
220- elif [ -e snap/snapcraft.yaml ]; then
221- SNAPCRAFT_YAML_PATH=snap/snapcraft.yaml
222- fi
223-
224- if [ -n "$SNAPCRAFT_YAML_PATH" ]; then
225- CI_BRANCH=$branch
226- break
227- fi
228-done
229-
230-if [ -z "$CI_BRANCH" ]; then
231- echo "WARNING: Can't build snap as no snapcraft.yaml exists!"
232- exit 0
233-fi
234-
235-REPO_NAME=$(awk -v a="$TARGET_GIT_REPO" 'BEGIN{print substr(a, index(a, "+git/") + 5)}')
236-# We rely on the snapcraft.yaml to have the snap name in the first five lines
237-# which is the case for all our snaps. This is a bit lazy but the best way to
238-# ensure we don't fetch any other name: fields which might be present in the file.
239-SNAP_NAME=$(cat $SNAPCRAFT_YAML_PATH | grep -v ^\# | head -n 5 | grep "^name:" | awk '{print $2}')
240-SNAP_REV=$(git rev-parse --short HEAD)
241-CI_REPO=$REPO_NAME-$BUILD_ID-$SNAP_REV
242-
243-sed -i "s/~$LAUNCHPAD_TEAM\/$LAUNCHPAD_PROJECT\/+git\/$REPO_NAME/~$LAUNCHPAD_TEAM\/$LAUNCHPAD_PROJECT\/+git\/$CI_REPO/g" \
244- $SNAPCRAFT_YAML_PATH
245+git clone git+ssh://$BOT_USERNAME@git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO $WORKSPACE/src
246
247 # The project as two different options of how snaps can be build:
248 #
249@@ -118,15 +67,6 @@ if [ "$BUILD_ON_LAUNCHPAD" = False ]; then
250 --proxy=http://squid.internal:3128 \
251 $SNAPBUILD_EXTRA_ARGS
252 else
253- git remote add jenkins-ci git+ssh://$BOT_USERNAME@git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO
254- git push jenkins-ci --all
255- git push jenkins-ci --tags
256-
257- # Save repo name as soon as it gets created so it can be deleted by the cleanup
258- # job even if this job fails.
259- echo "CI_REPO=$CI_REPO" >> $WORKSPACE/build-props
260- echo "CI_BRANCH=$CI_BRANCH" >> $WORKSPACE/build-props
261-
262 EXTRA_ARGS=
263 if [ -n "$ARCHITECTURE" ]; then
264 EXTRA_ARGS="$EXTRA_ARGS --architectures=$ARCHITECTURE"
265diff --git a/jobs/snap/snap-build-worker.yaml b/jobs/snap/snap-build-worker.yaml
266index 6b45a89..6a306f5 100644
267--- a/jobs/snap/snap-build-worker.yaml
268+++ b/jobs/snap/snap-build-worker.yaml
269@@ -60,6 +60,10 @@
270 default: stable
271 description: "Channel of the core snap to use for testing the build snap"
272 - string:
273+ name: CI_ID
274+ default: ''
275+ description: "Numeric id which will identify the CI iteration through all downstream builds"
276+ - string:
277 name: RESULTS_ID
278 default: ''
279 description: "Alphanumeric identifier used to pass build artifacts through different jobs"
280diff --git a/jobs/snap/snap-build.yaml b/jobs/snap/snap-build.yaml
281index e22e8d9..3aa338b 100644
282--- a/jobs/snap/snap-build.yaml
283+++ b/jobs/snap/snap-build.yaml
284@@ -49,12 +49,15 @@
285 name: CLEANUP_WORKSPACE
286 default: "0"
287 description: "Cleanup the whole workspace"
288+ - string:
289+ name: CI_ID
290+ default: ''
291+ description: "Numeric id which will identify the CI iteration through all downstream builds"
292 builders:
293 - trigger-builds:
294 - project: '{name}-snap-build-worker'
295 current-parameters: true
296 predefined-parameters: |
297- RESULTS_ID=$BUILD_TAG
298 ARCHITECTURE=$ARCHITECTURE
299 block: true
300 - project: '{name}-snap-docs'
301@@ -63,13 +66,10 @@
302 - project: '{name}-snap-test'
303 current-parameters: true
304 predefined-parameters: |
305- RESULTS_ID=$BUILD_TAG
306+ ARCHITECTURE=$ARCHITECTURE
307 block: true
308 - project: '{name}-snap-cleanup'
309 current-parameters: true
310- predefined-parameters: |
311- RESULTS_ID=$BUILD_TAG
312- property-file: build-props
313 publishers:
314 - archive:
315 artifacts: '**/*.snap'
316diff --git a/jobs/snap/snap-cleanup.sh b/jobs/snap/snap-cleanup.sh
317index bf0e03c..e1b42d6 100644
318--- a/jobs/snap/snap-cleanup.sh
319+++ b/jobs/snap/snap-cleanup.sh
320@@ -19,6 +19,7 @@ set -x
321 . "$WORKSPACE/.build_env"
322
323 # Delete auxiliary repo used in the build
324+CI_REPO="ci-build-$CI_ID"
325 $BUILD_SCRIPTS/tools/delete-ci-repo.py \
326 --git-repo=https://git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO
327
328diff --git a/jobs/snap/snap-cleanup.yaml b/jobs/snap/snap-cleanup.yaml
329index 8aa4a03..1086c68 100644
330--- a/jobs/snap/snap-cleanup.yaml
331+++ b/jobs/snap/snap-cleanup.yaml
332@@ -8,11 +8,11 @@
333 node: snap && build
334 parameters:
335 - string:
336- name: CI_REPO
337- default: ""
338- description: "Auxiliary repo for the build, that we will remove"
339+ name: CI_ID
340+ default: ''
341+ description: "Numeric id which will identify the CI iteration through all downstream builds"
342 - string:
343- name: "RESULTS_ID"
344+ name: RESULTS_ID
345 default: ""
346 description: "Alphanumeric Id of the results being staged on the remote worker"
347 - string:
348diff --git a/jobs/snap/snap-project-jobs.yaml b/jobs/snap/snap-project-jobs.yaml
349index 05e4f3f..2f9b0d8 100644
350--- a/jobs/snap/snap-project-jobs.yaml
351+++ b/jobs/snap/snap-project-jobs.yaml
352@@ -5,6 +5,7 @@
353 - '{name}-snap-build-worker'
354 - '{name}-snap-build-update-chroot'
355 - '{name}-snap-build'
356+ - '{name}-snap-build-prepare'
357 - '{name}-snap-cleanup'
358 - '{name}-snap-release'
359 - '{name}-snap-test'
360diff --git a/jobs/snap/snap-test.sh b/jobs/snap/snap-test.sh
361index 752b375..66876ed 100644
362--- a/jobs/snap/snap-test.sh
363+++ b/jobs/snap/snap-test.sh
364@@ -38,12 +38,9 @@ REMOTE_WORKSPACE=/home/$REMOTE_USER/$BUILD_TAG
365 REMOTE_RESULTS_BASE_DIR=/home/$REMOTE_USER/results
366
367 tmp_srcdir=$(mktemp -d)
368-git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO $tmp_srcdir/src
369+CI_REPO="ci-build-$CI_ID"
370+git clone --depth 1 -b $CI_BRANCH https://git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO
371 cd $tmp_srcdir/src
372-# This will fail as we have set set -e above when the revision isn't part of
373-# of the repository we've cloned.
374-git branch --contains $SOURCE_GIT_REPO_REVISION | grep "$SOURCE_GIT_REPO_BRANCH"
375-git checkout -b ci-test $SOURCE_GIT_REPO_REVISION
376
377 # Components have the ability to disable CI tests if they can't provide any.
378 # This only accepted in a few cases and should be generally avoided.
379@@ -86,9 +83,8 @@ go build github.com/snapcore/spread/cmd/spread
380 mkdir /build/bin
381 cp spread /build/bin
382
383-git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO /build/src
384+git clone --depth 1 -b $CI_BRANCH https://git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO /build/src
385 cd /build/src
386-git checkout -b ci-tests $SOURCE_GIT_REPO_REVISION
387
388 # Copy any stage results from previous generic-build-snap-worker builds
389 cp -v /build/results/*.snap /build/src
390diff --git a/jobs/snap/snap-test.yaml b/jobs/snap/snap-test.yaml
391index 9cc5cea..d0f18d5 100644
392--- a/jobs/snap/snap-test.yaml
393+++ b/jobs/snap/snap-test.yaml
394@@ -56,9 +56,9 @@
395 default: ""
396 description: "Branch on which the tests should be executed"
397 - string:
398- name: CI_REPO
399- default: ""
400- description: "Git repository to use for testing (MUST contain $CI_BRANCH)"
401+ name: CI_ID
402+ default: ''
403+ description: "Numeric id which will identify the CI iteration through all downstream builds"
404 builders:
405 - shell:
406 !include-raw:

Subscribers

People subscribed via source and target branches

to all changes: