Merge ~morphis/snappy-hwe-snaps/+git/jenkins-jobs:introduce-ci-id-var into ~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-jobs:master

Proposed by Simon Fels
Status: Merged
Approved by: Alfonso Sanchez-Beato
Approved revision: 7d0f98c13cbd8f4b213a19b0ff1729b9929617cc
Merged at revision: 93c6b4fc075c28c48a8acce069fd6ea01dc89b21
Proposed branch: ~morphis/snappy-hwe-snaps/+git/jenkins-jobs:introduce-ci-id-var
Merge into: ~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-jobs:master
Diff against target: 406 lines (+176/-82)
11 files modified
jobs/snap/common-job-prepare.sh (+2/-1)
jobs/snap/snap-build-prepare.sh (+88/-0)
jobs/snap/snap-build-prepare.yaml (+63/-0)
jobs/snap/snap-build-worker.sh (+2/-62)
jobs/snap/snap-build-worker.yaml (+4/-0)
jobs/snap/snap-build.yaml (+5/-5)
jobs/snap/snap-cleanup.sh (+1/-0)
jobs/snap/snap-cleanup.yaml (+4/-4)
jobs/snap/snap-project-jobs.yaml (+1/-0)
jobs/snap/snap-test.sh (+3/-7)
jobs/snap/snap-test.yaml (+3/-3)
Reviewer Review Type Date Requested Status
Alfonso Sanchez-Beato Approve
System Enablement Bot continuous-integration Approve
Konrad Zapałowicz (community) Approve
Jim Hodapp (community) Approve
Review via email: mp+331269@code.launchpad.net

Description of the change

Introduce new CI_ID variable which is handed from the top-level snap-build job to all subsequently triggered jobs and will identify the CI iteration. CI_ID is filled from BUILD_ID which is unique for each snap-build run.

This MP also changes the snap-test job to consume the same CI repository as the build job does so we have to merge target and source only once.

To post a comment you must log in.
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

Thanks, just one comment, see below.

Also, do we have a different BUILD_TAG for each architecture? We would need that too, or move the part that creates the repo from snap-build-worker (one per arch) to snap-build.

Revision history for this message
Simon Fels (morphis) wrote :

Yeah, I have implemented a snap-build-prepare job locally already which will be the origin of the BUILD_ID variable and will prepare the repository once and for all subsequent builds.

Revision history for this message
Jim Hodapp (jhodapp) wrote :

LGTM

review: Approve
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Konrad Zapałowicz (kzapalowicz) wrote :

ack

review: Approve
Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

Looks good, just some minor nits.

review: Needs Fixing
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

LGTM, thanks for the changes

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/jobs/snap/common-job-prepare.sh b/jobs/snap/common-job-prepare.sh
index 8c622ae..9b5b8f4 100644
--- a/jobs/snap/common-job-prepare.sh
+++ b/jobs/snap/common-job-prepare.sh
@@ -47,11 +47,12 @@ cat << EOF > $WORKSPACE/.build_env
47BOT_USERNAME={bot_username}47BOT_USERNAME={bot_username}
48LAUNCHPAD_PROJECT={launchpad_project}48LAUNCHPAD_PROJECT={launchpad_project}
49LAUNCHPAD_TEAM={launchpad_team}49LAUNCHPAD_TEAM={launchpad_team}
50SNAP_BUILD_JOB={name}-snap-build50SNAP_BUILD_JOB={name}-snap-build-prepare
51BUILD_SCRIPTS=$WORKSPACE/jenkins-jobs51BUILD_SCRIPTS=$WORKSPACE/jenkins-jobs
52BUILD_ON_LAUNCHPAD={build_on_launchpad}52BUILD_ON_LAUNCHPAD={build_on_launchpad}
53AUTO_MERGE={auto_merge}53AUTO_MERGE={auto_merge}
54TRIGGER_CI={trigger_ci}54TRIGGER_CI={trigger_ci}
55UPDATE_MPS={update_mps}55UPDATE_MPS={update_mps}
56RUN_TESTS={run_tests}56RUN_TESTS={run_tests}
57CI_REPO=ci-build-${{CI_ID}}
57EOF58EOF
diff --git a/jobs/snap/snap-build-prepare.sh b/jobs/snap/snap-build-prepare.sh
58new file mode 10064459new file mode 100644
index 0000000..77940d1
--- /dev/null
+++ b/jobs/snap/snap-build-prepare.sh
@@ -0,0 +1,88 @@
1#!/bin/bash
2#
3# Copyright (C) 2017 Canonical Ltd
4#
5# This program is free software: you can redistribute it and/or modify
6# it under the terms of the GNU General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# This program is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU General Public License for more details.
13#
14# You should have received a copy of the GNU General Public License
15# along with this program. If not, see <http://www.gnu.org/licenses/>.
16
17set -ex
18
19. "$WORKSPACE/.build_env"
20
21# CI_ID isn't specified yet when we're being called so initialize
22# it here with the same value as we do in snap-build-prepare.yaml
23if [ -z "$CI_ID" ]; then
24 CI_ID=$BUILD_ID
25fi
26
27rm -rf $WORKSPACE/src
28
29git clone --no-checkout $TARGET_GIT_REPO $WORKSPACE/src
30cd $WORKSPACE/src
31for remote in $(git branch -r | grep -v origin/master); do
32 git checkout --track $remote
33done
34
35git checkout $TARGET_GIT_REPO_BRANCH
36
37git config user.name "System Enablement CI Bot"
38git config user.email "ce-system-enablement@lists.canonical.com"
39
40if [ -n "$SOURCE_GIT_REPO" ]; then
41 git remote add other $SOURCE_GIT_REPO
42 git fetch other
43 git merge \
44 --no-ff \
45 -m "Merge remote tracking branch other/$SOURCE_GIT_REPO_BRANCH" \
46 $REVISION
47fi
48
49# Try to find the correct branch we need to build from. In the case that
50# $TARGET_GIT_REPO_BRANCH points us to an upstream component branch we
51# will take master as the next suitable candidate.
52CI_BRANCH=
53SNAPCRAFT_YAML_PATH=
54for branch in $TARGET_GIT_REPO_BRANCH master ; do
55 git checkout $branch
56 if [ -e snapcraft.yaml ]; then
57 SNAPCRAFT_YAML_PATH=snapcraft.yaml
58 elif [ -e snap/snapcraft.yaml ]; then
59 SNAPCRAFT_YAML_PATH=snap/snapcraft.yaml
60 fi
61
62 if [ -n "$SNAPCRAFT_YAML_PATH" ]; then
63 CI_BRANCH=$branch
64 break
65 fi
66done
67
68if [ -z "$CI_BRANCH" ]; then
69 echo "WARNING: Can't build snap as no snapcraft.yaml exists!"
70 exit 0
71fi
72
73REPO_NAME=$(awk -v a="$TARGET_GIT_REPO" 'BEGIN{print substr(a, index(a, "+git/") + 5)}')
74# We rely on the snapcraft.yaml to have the snap name in the first five lines
75# which is the case for all our snaps. This is a bit lazy but the best way to
76# ensure we don't fetch any other name: fields which might be present in the file.
77SNAP_NAME=$(cat $SNAPCRAFT_YAML_PATH | grep -v ^\# | head -n 5 | grep "^name:" | awk '{print $2}')
78SNAP_REV=$(git rev-parse --short HEAD)
79CI_REPO=ci-build-$CI_ID
80
81sed -i "s/~$LAUNCHPAD_TEAM\/$LAUNCHPAD_PROJECT\/+git\/$REPO_NAME/~$LAUNCHPAD_TEAM\/$LAUNCHPAD_PROJECT\/+git\/$CI_REPO/g" \
82 $SNAPCRAFT_YAML_PATH
83git add snapcraft.yaml
84git commit -m "Update snapcraft.yaml to build from right repository"
85
86git remote add jenkins-ci git+ssh://$BOT_USERNAME@git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO
87git push jenkins-ci --all
88git push jenkins-ci --tags
diff --git a/jobs/snap/snap-build-prepare.yaml b/jobs/snap/snap-build-prepare.yaml
0new file mode 10064489new file mode 100644
index 0000000..49c504e
--- /dev/null
+++ b/jobs/snap/snap-build-prepare.yaml
@@ -0,0 +1,63 @@
1- job-template:
2 name: '{name}-snap-build-prepare'
3 project-type: freestyle
4 defaults: global
5 description: ""
6 display-name: "{name}-snap-build-prepare"
7 concurrent: true
8 node: snap && misc
9 properties:
10 - build-discarder:
11 num-to-keep: 10
12 - rebuild
13 parameters:
14 - string:
15 name: TARGET_GIT_REPO
16 default:
17 description: "Target git repository"
18 - string:
19 name: TARGET_GIT_REPO_BRANCH
20 default: master
21 description: "Branch of the target git repository to build from"
22 - string:
23 name: SERIES
24 default: xenial
25 description: "Ubuntu archive series to build for"
26 - string:
27 name: FORCE
28 default: "0"
29 description: "Set to 1 to force the build"
30 - string:
31 name: SOURCE_GIT_REPO
32 default:
33 description: "Source git repository"
34 - string:
35 name: SOURCE_GIT_REPO_BRANCH
36 default:
37 description: "Branch of the source git repository to use"
38 - string:
39 name: MERGE_PROPOSAL
40 default:
41 description: "Link to the merge proposal this build relates to"
42 - string:
43 name: REVISION
44 default:
45 description: "Cleanup the whole workspace"
46 - string:
47 name: CLEANUP_WORKSPACE
48 default: "0"
49 description: "Cleanup the whole workspace"
50 builders:
51 - shell:
52 !include-raw:
53 - common-job-prepare.sh
54 - shell:
55 !include-raw-escape:
56 - snap-build-prepare.sh
57 - trigger-builds:
58 - project: '{name}-snap-build'
59 current-parameters: true
60 predefined-parameters: |
61 CI_ID=$BUILD_ID
62 RESULTS_ID=$BUILD_TAG
63 block: true
diff --git a/jobs/snap/snap-build-worker.sh b/jobs/snap/snap-build-worker.sh
index 02ee65c..99a1f75 100644
--- a/jobs/snap/snap-build-worker.sh
+++ b/jobs/snap/snap-build-worker.sh
@@ -18,60 +18,9 @@ set -ex
1818
19. "$WORKSPACE/.build_env"19. "$WORKSPACE/.build_env"
2020
21rm -rf $WORKSPACE/src $WORKSPACE/results $WORKSPACE/build-props21rm -rf $WORKSPACE/src $WORKSPACE/results
2222
23git clone --no-checkout $TARGET_GIT_REPO $WORKSPACE/src23git clone git+ssh://$BOT_USERNAME@git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO $WORKSPACE/src
24cd $WORKSPACE/src
25for remote in `git branch -r | grep -v origin/master`; do git checkout --track $remote ; done
26
27git checkout $TARGET_GIT_REPO_BRANCH
28
29git config user.name "System Enablement CI Bot"
30git config user.email "ce-system-enablement@lists.canonical.com"
31
32if [ -n "$SOURCE_GIT_REPO" ]; then
33 git remote add other $SOURCE_GIT_REPO
34 git fetch other
35 git merge \
36 --no-ff \
37 -m "Merge remote tracking branch other/$SOURCE_GIT_REPO_BRANCH" \
38 $REVISION
39fi
40
41# Try to find the correct branch we need to build from. In the case that
42# $TARGET_GIT_REPO_BRANCH points us to an upstream component branch we
43# will take master as the next suitable candidate.
44CI_BRANCH=
45SNAPCRAFT_YAML_PATH=
46for branch in $TARGET_GIT_REPO_BRANCH master ; do
47 git checkout $branch
48 if [ -e snapcraft.yaml ]; then
49 SNAPCRAFT_YAML_PATH=snapcraft.yaml
50 elif [ -e snap/snapcraft.yaml ]; then
51 SNAPCRAFT_YAML_PATH=snap/snapcraft.yaml
52 fi
53
54 if [ -n "$SNAPCRAFT_YAML_PATH" ]; then
55 CI_BRANCH=$branch
56 break
57 fi
58done
59
60if [ -z "$CI_BRANCH" ]; then
61 echo "WARNING: Can't build snap as no snapcraft.yaml exists!"
62 exit 0
63fi
64
65REPO_NAME=$(awk -v a="$TARGET_GIT_REPO" 'BEGIN{print substr(a, index(a, "+git/") + 5)}')
66# We rely on the snapcraft.yaml to have the snap name in the first five lines
67# which is the case for all our snaps. This is a bit lazy but the best way to
68# ensure we don't fetch any other name: fields which might be present in the file.
69SNAP_NAME=$(cat $SNAPCRAFT_YAML_PATH | grep -v ^\# | head -n 5 | grep "^name:" | awk '{print $2}')
70SNAP_REV=$(git rev-parse --short HEAD)
71CI_REPO=$REPO_NAME-$BUILD_ID-$SNAP_REV
72
73sed -i "s/~$LAUNCHPAD_TEAM\/$LAUNCHPAD_PROJECT\/+git\/$REPO_NAME/~$LAUNCHPAD_TEAM\/$LAUNCHPAD_PROJECT\/+git\/$CI_REPO/g" \
74 $SNAPCRAFT_YAML_PATH
7524
76# The project as two different options of how snaps can be build:25# The project as two different options of how snaps can be build:
77#26#
@@ -118,15 +67,6 @@ if [ "$BUILD_ON_LAUNCHPAD" = False ]; then
118 --proxy=http://squid.internal:3128 \67 --proxy=http://squid.internal:3128 \
119 $SNAPBUILD_EXTRA_ARGS68 $SNAPBUILD_EXTRA_ARGS
120else69else
121 git remote add jenkins-ci git+ssh://$BOT_USERNAME@git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO
122 git push jenkins-ci --all
123 git push jenkins-ci --tags
124
125 # Save repo name as soon as it gets created so it can be deleted by the cleanup
126 # job even if this job fails.
127 echo "CI_REPO=$CI_REPO" >> $WORKSPACE/build-props
128 echo "CI_BRANCH=$CI_BRANCH" >> $WORKSPACE/build-props
129
130 EXTRA_ARGS=70 EXTRA_ARGS=
131 if [ -n "$ARCHITECTURE" ]; then71 if [ -n "$ARCHITECTURE" ]; then
132 EXTRA_ARGS="$EXTRA_ARGS --architectures=$ARCHITECTURE"72 EXTRA_ARGS="$EXTRA_ARGS --architectures=$ARCHITECTURE"
diff --git a/jobs/snap/snap-build-worker.yaml b/jobs/snap/snap-build-worker.yaml
index 6b45a89..6a306f5 100644
--- a/jobs/snap/snap-build-worker.yaml
+++ b/jobs/snap/snap-build-worker.yaml
@@ -60,6 +60,10 @@
60 default: stable60 default: stable
61 description: "Channel of the core snap to use for testing the build snap"61 description: "Channel of the core snap to use for testing the build snap"
62 - string:62 - string:
63 name: CI_ID
64 default: ''
65 description: "Numeric id which will identify the CI iteration through all downstream builds"
66 - string:
63 name: RESULTS_ID67 name: RESULTS_ID
64 default: ''68 default: ''
65 description: "Alphanumeric identifier used to pass build artifacts through different jobs"69 description: "Alphanumeric identifier used to pass build artifacts through different jobs"
diff --git a/jobs/snap/snap-build.yaml b/jobs/snap/snap-build.yaml
index e22e8d9..3aa338b 100644
--- a/jobs/snap/snap-build.yaml
+++ b/jobs/snap/snap-build.yaml
@@ -49,12 +49,15 @@
49 name: CLEANUP_WORKSPACE49 name: CLEANUP_WORKSPACE
50 default: "0"50 default: "0"
51 description: "Cleanup the whole workspace"51 description: "Cleanup the whole workspace"
52 - string:
53 name: CI_ID
54 default: ''
55 description: "Numeric id which will identify the CI iteration through all downstream builds"
52 builders:56 builders:
53 - trigger-builds:57 - trigger-builds:
54 - project: '{name}-snap-build-worker'58 - project: '{name}-snap-build-worker'
55 current-parameters: true59 current-parameters: true
56 predefined-parameters: |60 predefined-parameters: |
57 RESULTS_ID=$BUILD_TAG
58 ARCHITECTURE=$ARCHITECTURE61 ARCHITECTURE=$ARCHITECTURE
59 block: true62 block: true
60 - project: '{name}-snap-docs'63 - project: '{name}-snap-docs'
@@ -63,13 +66,10 @@
63 - project: '{name}-snap-test'66 - project: '{name}-snap-test'
64 current-parameters: true67 current-parameters: true
65 predefined-parameters: |68 predefined-parameters: |
66 RESULTS_ID=$BUILD_TAG69 ARCHITECTURE=$ARCHITECTURE
67 block: true70 block: true
68 - project: '{name}-snap-cleanup'71 - project: '{name}-snap-cleanup'
69 current-parameters: true72 current-parameters: true
70 predefined-parameters: |
71 RESULTS_ID=$BUILD_TAG
72 property-file: build-props
73 publishers:73 publishers:
74 - archive:74 - archive:
75 artifacts: '**/*.snap'75 artifacts: '**/*.snap'
diff --git a/jobs/snap/snap-cleanup.sh b/jobs/snap/snap-cleanup.sh
index bf0e03c..e1b42d6 100644
--- a/jobs/snap/snap-cleanup.sh
+++ b/jobs/snap/snap-cleanup.sh
@@ -19,6 +19,7 @@ set -x
19. "$WORKSPACE/.build_env"19. "$WORKSPACE/.build_env"
2020
21# Delete auxiliary repo used in the build21# Delete auxiliary repo used in the build
22CI_REPO="ci-build-$CI_ID"
22$BUILD_SCRIPTS/tools/delete-ci-repo.py \23$BUILD_SCRIPTS/tools/delete-ci-repo.py \
23 --git-repo=https://git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO24 --git-repo=https://git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO
2425
diff --git a/jobs/snap/snap-cleanup.yaml b/jobs/snap/snap-cleanup.yaml
index 8aa4a03..1086c68 100644
--- a/jobs/snap/snap-cleanup.yaml
+++ b/jobs/snap/snap-cleanup.yaml
@@ -8,11 +8,11 @@
8 node: snap && build8 node: snap && build
9 parameters:9 parameters:
10 - string:10 - string:
11 name: CI_REPO11 name: CI_ID
12 default: ""12 default: ''
13 description: "Auxiliary repo for the build, that we will remove"13 description: "Numeric id which will identify the CI iteration through all downstream builds"
14 - string:14 - string:
15 name: "RESULTS_ID"15 name: RESULTS_ID
16 default: ""16 default: ""
17 description: "Alphanumeric Id of the results being staged on the remote worker"17 description: "Alphanumeric Id of the results being staged on the remote worker"
18 - string:18 - string:
diff --git a/jobs/snap/snap-project-jobs.yaml b/jobs/snap/snap-project-jobs.yaml
index 05e4f3f..2f9b0d8 100644
--- a/jobs/snap/snap-project-jobs.yaml
+++ b/jobs/snap/snap-project-jobs.yaml
@@ -5,6 +5,7 @@
5 - '{name}-snap-build-worker'5 - '{name}-snap-build-worker'
6 - '{name}-snap-build-update-chroot'6 - '{name}-snap-build-update-chroot'
7 - '{name}-snap-build'7 - '{name}-snap-build'
8 - '{name}-snap-build-prepare'
8 - '{name}-snap-cleanup'9 - '{name}-snap-cleanup'
9 - '{name}-snap-release'10 - '{name}-snap-release'
10 - '{name}-snap-test'11 - '{name}-snap-test'
diff --git a/jobs/snap/snap-test.sh b/jobs/snap/snap-test.sh
index 752b375..66876ed 100644
--- a/jobs/snap/snap-test.sh
+++ b/jobs/snap/snap-test.sh
@@ -38,12 +38,9 @@ REMOTE_WORKSPACE=/home/$REMOTE_USER/$BUILD_TAG
38REMOTE_RESULTS_BASE_DIR=/home/$REMOTE_USER/results38REMOTE_RESULTS_BASE_DIR=/home/$REMOTE_USER/results
3939
40tmp_srcdir=$(mktemp -d)40tmp_srcdir=$(mktemp -d)
41git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO $tmp_srcdir/src41CI_REPO="ci-build-$CI_ID"
42git clone --depth 1 -b $CI_BRANCH https://git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO
42cd $tmp_srcdir/src43cd $tmp_srcdir/src
43# This will fail as we have set set -e above when the revision isn't part of
44# of the repository we've cloned.
45git branch --contains $SOURCE_GIT_REPO_REVISION | grep "$SOURCE_GIT_REPO_BRANCH"
46git checkout -b ci-test $SOURCE_GIT_REPO_REVISION
4744
48# Components have the ability to disable CI tests if they can't provide any.45# Components have the ability to disable CI tests if they can't provide any.
49# This only accepted in a few cases and should be generally avoided.46# This only accepted in a few cases and should be generally avoided.
@@ -86,9 +83,8 @@ go build github.com/snapcore/spread/cmd/spread
86mkdir /build/bin83mkdir /build/bin
87cp spread /build/bin84cp spread /build/bin
8885
89git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO /build/src86git clone --depth 1 -b $CI_BRANCH https://git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO /build/src
90cd /build/src87cd /build/src
91git checkout -b ci-tests $SOURCE_GIT_REPO_REVISION
9288
93# Copy any stage results from previous generic-build-snap-worker builds89# Copy any stage results from previous generic-build-snap-worker builds
94cp -v /build/results/*.snap /build/src90cp -v /build/results/*.snap /build/src
diff --git a/jobs/snap/snap-test.yaml b/jobs/snap/snap-test.yaml
index 9cc5cea..d0f18d5 100644
--- a/jobs/snap/snap-test.yaml
+++ b/jobs/snap/snap-test.yaml
@@ -56,9 +56,9 @@
56 default: ""56 default: ""
57 description: "Branch on which the tests should be executed"57 description: "Branch on which the tests should be executed"
58 - string:58 - string:
59 name: CI_REPO59 name: CI_ID
60 default: ""60 default: ''
61 description: "Git repository to use for testing (MUST contain $CI_BRANCH)"61 description: "Numeric id which will identify the CI iteration through all downstream builds"
62 builders:62 builders:
63 - shell:63 - shell:
64 !include-raw:64 !include-raw:

Subscribers

People subscribed via source and target branches

to all changes: