Merge ~morphis/snappy-hwe-snaps/+git/build-scripts:feature/snap-release-job into ~snappy-hwe-team/snappy-hwe-snaps/+git/build-scripts:master
- Git
- lp:~morphis/snappy-hwe-snaps/+git/build-scripts
- feature/snap-release-job
- Merge into master
Status: | Merged |
---|---|
Approved by: | Jim Hodapp |
Approved revision: | 235cb09d6d273e61dfc95f6a104aa949d1279a47 |
Merged at revision: | d94715fe5278700bbcdf52230dd71390118bd093 |
Proposed branch: | ~morphis/snappy-hwe-snaps/+git/build-scripts:feature/snap-release-job |
Merge into: | ~snappy-hwe-team/snappy-hwe-snaps/+git/build-scripts:master |
Diff against target: |
238 lines (+194/-6) 4 files modified
jobs/generic-release-snap (+63/-0) scripts/se_utils/__init__.py (+22/-0) scripts/snap-build (+9/-6) scripts/trigger-lp-build.py (+100/-0) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Jim Hodapp (community) | Approve | ||
System Enablement Bot | continuous-integration | Approve | |
Review via email: mp+316129@code.launchpad.net |
Commit message
Description of the change
Add snap release job
Follows the described process in our guidelines at https:/
Corresponding jenkins job is https:/
System Enablement Bot (system-enablement-ci-bot) wrote : | # |
System Enablement Bot (system-enablement-ci-bot) wrote : | # |
PASSED: Continuous integration, rev:397b72c1e43
https:/
Executed test runs:
SUCCESS: https:/
None: https:/
Click here to trigger a rebuild:
https:/
System Enablement Bot (system-enablement-ci-bot) wrote : | # |
PASSED: Continuous integration, rev:8bfa35fd224
https:/
Executed test runs:
SUCCESS: https:/
None: https:/
Click here to trigger a rebuild:
https:/
System Enablement Bot (system-enablement-ci-bot) wrote : | # |
PASSED: Continuous integration, rev:888533c5f1d
https:/
Executed test runs:
SUCCESS: https:/
None: https:/
Click here to trigger a rebuild:
https:/
System Enablement Bot (system-enablement-ci-bot) wrote : | # |
FAILED: Continuous integration, rev:888533c5f1d
https:/
Executed test runs:
None: https:/
Click here to trigger a rebuild:
https:/
System Enablement Bot (system-enablement-ci-bot) wrote : | # |
PASSED: Continuous integration, rev:8fe0b20174d
https:/
Executed test runs:
SUCCESS: https:/
None: https:/
Click here to trigger a rebuild:
https:/
System Enablement Bot (system-enablement-ci-bot) wrote : | # |
PASSED: Continuous integration, rev:e3859b8acc3
https:/
Executed test runs:
SUCCESS: https:/
None: https:/
Click here to trigger a rebuild:
https:/
System Enablement Bot (system-enablement-ci-bot) wrote : | # |
PASSED: Continuous integration, rev:07b35cdf7a5
https:/
Executed test runs:
SUCCESS: https:/
None: https:/
Click here to trigger a rebuild:
https:/
System Enablement Bot (system-enablement-ci-bot) wrote : | # |
PASSED: Continuous integration, rev:2f7e09d84ac
https:/
Executed test runs:
SUCCESS: https:/
None: https:/
Click here to trigger a rebuild:
https:/
Jim Hodapp (jhodapp) wrote : | # |
Just a few changes needed, see inline below.
System Enablement Bot (system-enablement-ci-bot) wrote : | # |
PASSED: Continuous integration, rev:2d0377908a5
https:/
Executed test runs:
SUCCESS: https:/
None: https:/
Click here to trigger a rebuild:
https:/
System Enablement Bot (system-enablement-ci-bot) wrote : | # |
PASSED: Continuous integration, rev:235cb09d6d2
https:/
Executed test runs:
SUCCESS: https:/
None: https:/
Click here to trigger a rebuild:
https:/
Preview Diff
1 | diff --git a/jobs/generic-release-snap b/jobs/generic-release-snap |
2 | new file mode 100755 |
3 | index 0000000..4c05499 |
4 | --- /dev/null |
5 | +++ b/jobs/generic-release-snap |
6 | @@ -0,0 +1,63 @@ |
7 | +#!/bin/sh |
8 | +# |
9 | +# Copyright (C) 2017 Canonical Ltd |
10 | +# |
11 | +# This program is free software: you can redistribute it and/or modify |
12 | +# it under the terms of the GNU General Public License version 3 as |
13 | +# published by the Free Software Foundation. |
14 | +# |
15 | +# This program is distributed in the hope that it will be useful, |
16 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
17 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
18 | +# GNU General Public License for more details. |
19 | +# |
20 | +# You should have received a copy of the GNU General Public License |
21 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
22 | + |
23 | +set -ex |
24 | + |
25 | +if [ -z "$VERSION" ]; then |
26 | + echo "ERROR: No version specified" |
27 | + exit 1 |
28 | +fi |
29 | + |
30 | +echo "Snap to be release: $SNAP_NAME" |
31 | +echo "Version to be released: $VERSION" |
32 | + |
33 | +GIT_USER=system-enablement-ci-bot |
34 | +REPOSITORY_URL="git+ssh://$GIT_USER@git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/$SNAP_NAME" |
35 | + |
36 | +if [ -e $SNAP_NAME ]; then |
37 | + rm -rf $SNAP_NAME |
38 | +fi |
39 | + |
40 | +git clone -b master $REPOSITORY_URL $SNAP_NAME |
41 | +cd $SNAP_NAME |
42 | + |
43 | +if [ ! -e snapcraft.yaml ]; then |
44 | + echo "ERROR: No top-level snapcraft.yaml file!" |
45 | + exit 1 |
46 | +fi |
47 | + |
48 | +git config user.name "System Enablement CI Bot" |
49 | +git config user.email "ce-system-enablement@lists.canonical.com" |
50 | + |
51 | +sed -i -e "s/version:.*/version: $VERSION/g" snapcraft.yaml |
52 | +git add snapcraft.yaml |
53 | +git commit -m "Bump version to $VERSION" |
54 | + |
55 | +git tag -a -m "$VERSION" $VERSION HEAD |
56 | +if ! git branch -r | grep origin/stable ; then |
57 | + git checkout -b stable origin/master |
58 | +else |
59 | + git checkout -b stable origin/stable |
60 | +fi |
61 | +git merge master |
62 | + |
63 | +git push origin stable |
64 | +git push origin master |
65 | +git push origin $VERSION |
66 | + |
67 | + |
68 | + |
69 | +exec $WORKSPACE/build-scripts/scripts/trigger-lp-build.py -s $SNAP_NAME -p |
70 | diff --git a/scripts/se_utils/__init__.py b/scripts/se_utils/__init__.py |
71 | index fcb95d7..280450c 100644 |
72 | --- a/scripts/se_utils/__init__.py |
73 | +++ b/scripts/se_utils/__init__.py |
74 | @@ -20,6 +20,7 @@ import sys |
75 | import time |
76 | import logging |
77 | import os |
78 | +import yaml |
79 | from shutil import rmtree |
80 | from launchpadlib.credentials import RequestTokenAuthorizationEngine |
81 | from lazr.restfulclient.errors import HTTPError |
82 | @@ -98,3 +99,24 @@ def get_launchpad(launchpadlib_dir=None, credential_store_path=None, lp_app=None |
83 | authorization_engine=authorization_engine, |
84 | launchpadlib_dir=lib_dir, |
85 | version='devel') |
86 | + |
87 | +# Load configuration for the current agent we're running on. All agents were |
88 | +# provisioned when they were setup with a proper configuration. See |
89 | +# https://wiki.canonical.com/InformationInfrastructure/Jenkaas/UserDocs for |
90 | +# more details. |
91 | +def load_config(): |
92 | + files = [os.path.expanduser('~/.jlp/jlp.config'), 'jlp.config'] |
93 | + for config_file in files: |
94 | + try: |
95 | + config = yaml.safe_load(open(config_file, 'r')) |
96 | + return config |
97 | + except IOError: |
98 | + pass |
99 | + print("ERROR: No config file found") |
100 | + sys.exit(1) |
101 | + |
102 | +# Return a configuration option from the agent configuration specified by the |
103 | +# name argument. |
104 | +def get_config_option(name): |
105 | + config = load_config() |
106 | + return config[name] |
107 | diff --git a/scripts/snap-build b/scripts/snap-build |
108 | index b597ab6..abe40b4 100755 |
109 | --- a/scripts/snap-build |
110 | +++ b/scripts/snap-build |
111 | @@ -79,12 +79,15 @@ cd /build/src |
112 | |
113 | git config user.name "System Enablement CI Bot" |
114 | git config user.email "ce-system-enablement@lists.canonical.com" |
115 | -git remote add other $SOURCE_GIT_REPO |
116 | -git fetch other |
117 | -git merge \ |
118 | - --no-ff \ |
119 | - -m "Merge remote tracking branch other/$SOURCE_GIT_REPO_BRANCH" \ |
120 | - $REVISION |
121 | + |
122 | +if [ -n "$SOURCE_GIT_REPO" ]; then |
123 | + git remote add other $SOURCE_GIT_REPO |
124 | + git fetch other |
125 | + git merge \ |
126 | + --no-ff \ |
127 | + -m "Merge remote tracking branch other/$SOURCE_GIT_REPO_BRANCH" \ |
128 | + $REVISION |
129 | +fi |
130 | |
131 | # Only attempt to build projects where we have a valid snapcraft project. |
132 | # For all others we just make sure we can merge source to target. |
133 | diff --git a/scripts/trigger-lp-build.py b/scripts/trigger-lp-build.py |
134 | new file mode 100755 |
135 | index 0000000..842acd5 |
136 | --- /dev/null |
137 | +++ b/scripts/trigger-lp-build.py |
138 | @@ -0,0 +1,100 @@ |
139 | +#! /usr/bin/python |
140 | + |
141 | +import os |
142 | +import sys |
143 | +import time |
144 | +import smtplib |
145 | + |
146 | +from datetime import datetime |
147 | +from os.path import basename |
148 | +from launchpadlib.launchpad import Launchpad |
149 | + |
150 | +from argparse import ArgumentParser |
151 | + |
152 | +import se_utils |
153 | + |
154 | +parser = ArgumentParser(description="Build a specific snap on launchpad") |
155 | +parser.add_argument('-s', '--snap', required=True, |
156 | + help="Name of the snap to build") |
157 | +parser.add_argument('-p', '--publish', action='store_true', |
158 | + help="Trigger a publish build instead of a daily (default)") |
159 | + |
160 | +args = vars(parser.parse_args()) |
161 | + |
162 | +arches = ['amd64', 'i386', 'armhf', 'arm64'] |
163 | +series = 'xenial' |
164 | + |
165 | +lp_app = se_utils.get_config_option("lp_app") |
166 | +lp_env = se_utils.get_config_option("lp_env") |
167 | +credential_store_path = se_utils.get_config_option('credential_store_path') |
168 | +launchpad = se_utils.get_launchpad(None, credential_store_path, lp_app, lp_env) |
169 | + |
170 | +team = launchpad.people['snappy-hwe-team'] |
171 | + |
172 | +build_name = "%s-daily" % args["snap"] |
173 | +if args["publish"] == True: |
174 | + build_name = "%s-publish" % args["snap"] |
175 | + |
176 | +snap = launchpad.snaps.getByName(name=build_name, owner=team) |
177 | +ubuntu = launchpad.distributions['ubuntu'] |
178 | +release = ubuntu.getSeries(name_or_version=series) |
179 | +primary_archive = ubuntu.getArchive(name='primary') |
180 | + |
181 | +# Add a big fat warning that we don't really care about fixing things when |
182 | +# the job will be canceled after the following lines are printed out. |
183 | +print("!!!!!!! POINT OF NO RETURN !!!!!!!") |
184 | +print("DO NOT CANCEL THIS JOB AFTER THIS OR BAD THINGS WILL HAPPEN") |
185 | +print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!") |
186 | + |
187 | +stamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S') |
188 | +print("Trying to trigger builds at: {}".format(stamp)) |
189 | + |
190 | +# We will now trigger a build for each whitelisted architecture, collect the |
191 | +# build job url and the wait for all builds to finish and collect their results |
192 | +# to vote for a successful or failed build. |
193 | +triggered_builds = [] |
194 | +for build_arch in arches: |
195 | + arch = release.getDistroArchSeries(archtag=build_arch) |
196 | + request = snap.requestBuild(archive=primary_archive, distro_arch_series=arch, pocket='Proposed') |
197 | + build_id = str(request).rsplit('/', 1)[-1] |
198 | + triggered_builds.append(build_id) |
199 | + print("Arch: {} is building under: {}".format(build_arch, request)) |
200 | + |
201 | +failures = [] |
202 | +while len(triggered_builds): |
203 | + for build in triggered_builds: |
204 | + try: |
205 | + response = snap.getBuildSummariesForSnapBuildIds(snap_build_ids=[build]) |
206 | + except: |
207 | + print("Could not get response for {} (was there an LP timeout?)".format(build)) |
208 | + continue |
209 | + status = response[build]['status'] |
210 | + if status == "FULLYBUILT": |
211 | + triggered_builds.remove(build) |
212 | + continue |
213 | + elif status == "FAILEDTOBUILD": |
214 | + failures.append(build) |
215 | + triggered_builds.remove(build) |
216 | + continue |
217 | + elif status == "CANCELLED": |
218 | + triggered_builds.remove(build) |
219 | + continue |
220 | + time.sleep(60) |
221 | + |
222 | +if len(failures): |
223 | + for failure in failures: |
224 | + try: |
225 | + response = snap.getBuildSummariesForSnapBuildIds(snap_build_ids=[failure]) |
226 | + except: |
227 | + print ("Could not get failure data for {} (was there an LP timeout?)".format(build)) |
228 | + continue |
229 | + buildlog = response[build]['build_log_url'] |
230 | + if buildlog != 'None': |
231 | + print(buildlog) |
232 | + arch = str(buildlog).split('_')[4] |
233 | + print("{} snap {} build at {} failed for id: {} log: {}".format(args["snap"], arch, stamp, failure, buildlog)) |
234 | + |
235 | + # Let the build fail as at least a single snap has failed to build |
236 | + sys.exit(1) |
237 | + |
238 | +print("Done!") |
PASSED: Continuous integration, rev:fd6c0ef79ff e8ff841e2a1c4bd 4b35da29ec1e93 /jenkins. canonical. com/system- enablement/ job/generic- build-snap/ 750/ /jenkins. canonical. com/system- enablement/ job/generic- run-snap- spread- tests/447 /jenkins. canonical. com/system- enablement/ job/generic- update- snap-mp/ 658/console
https:/
Executed test runs:
SUCCESS: https:/
None: https:/
Click here to trigger a rebuild: /jenkins. canonical. com/system- enablement/ job/generic- build-snap/ 750/rebuild
https:/