Merge ~kzapalowicz/snappy-hwe-snaps/+git/jenkins-jobs:fix/infrastructure-is-missing-file into ~snappy-hwe-team/snappy-hwe-snaps/+git/wifi-ap:master
- Git
- lp:~kzapalowicz/snappy-hwe-snaps/+git/jenkins-jobs
- fix/infrastructure-is-missing-file
- Merge into master
Status: | Superseded |
---|---|
Proposed branch: | ~kzapalowicz/snappy-hwe-snaps/+git/jenkins-jobs:fix/infrastructure-is-missing-file |
Merge into: | ~snappy-hwe-team/snappy-hwe-snaps/+git/wifi-ap:master |
Diff against target: |
4384 lines (+4018/-0) (has conflicts) 58 files modified
.ci_tests_disabled (+0/-0) README.md (+56/-0) docker/Dockerfile (+59/-0) docker/initial-setup.groovy (+34/-0) docker/jenkins.sh (+24/-0) docker/plugins.sh (+28/-0) docker/plugins.txt (+29/-0) jobs/image/common-job-prepare.sh (+7/-0) jobs/image/image-build-worker.sh (+27/-0) jobs/image/image-build-worker.yaml (+15/-0) jobs/image/image-project-jobs.yaml (+4/-0) jobs/infrastructure/common-job-prepare.sh (+14/-0) jobs/infrastructure/credentials-0-ssh.sh (+58/-0) jobs/infrastructure/credentials-0-ssh.yaml (+29/-0) jobs/infrastructure/credentials-1-launchpad.py (+86/-0) jobs/infrastructure/credentials-1-launchpad.yaml (+21/-0) jobs/infrastructure/credentials-2-launchpad-plugin.sh (+108/-0) jobs/infrastructure/credentials-2-launchpad-plugin.yaml (+24/-0) jobs/infrastructure/deploy-jenkins-jobs.sh (+58/-0) jobs/infrastructure/deploy-jenkins-jobs.yaml (+24/-0) jobs/infrastructure/infrastructure-jobs.yaml (+8/-0) jobs/infrastructure/prepare-0-install.sh (+47/-0) jobs/infrastructure/prepare-0-install.yaml (+27/-0) jobs/snap/common-job-prepare.sh (+24/-0) jobs/snap/snap-automerger.sh (+27/-0) jobs/snap/snap-automerger.yaml (+22/-0) jobs/snap/snap-build-update-chroot.sh (+28/-0) jobs/snap/snap-build-update-chroot.yaml (+29/-0) jobs/snap/snap-build-worker.sh (+147/-0) jobs/snap/snap-build-worker.yaml (+72/-0) jobs/snap/snap-build.yaml (+91/-0) jobs/snap/snap-cleanup.sh (+38/-0) jobs/snap/snap-cleanup.yaml (+32/-0) jobs/snap/snap-nightly.yaml (+41/-0) jobs/snap/snap-project-jobs.yaml (+13/-0) jobs/snap/snap-release.sh (+170/-0) jobs/snap/snap-release.yaml (+58/-0) jobs/snap/snap-test.sh (+121/-0) jobs/snap/snap-test.yaml (+64/-0) jobs/snap/snap-trigger-ci.sh (+29/-0) jobs/snap/snap-trigger-ci.yaml (+22/-0) jobs/snap/snap-update-mp.sh (+30/-0) jobs/snap/snap-update-mp.yaml (+31/-0) local.conf (+11/-0) local.yaml (+60/-0) run-tests.sh (+4/-0) tools/automerge-mps.py (+152/-0) tools/build-rootfs-create (+26/-0) tools/common.sh (+93/-0) tools/delete-ci-repo.py (+45/-0) tools/hardware-test.sh (+112/-0) tools/se_utils/__init__.py (+122/-0) tools/shyaml (+454/-0) tools/snapbuild.sh (+192/-0) tools/test-snap.sh (+100/-0) tools/trigger-ci.py (+270/-0) tools/trigger-lp-build.py (+230/-0) tools/vote-on-merge-proposal.py (+271/-0) Conflict in README.md Conflict in run-tests.sh |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Snappy HWE Team | Pending | ||
Review via email: mp+328106@code.launchpad.net |
This proposal has been superseded by a proposal from 2017-07-26.
Commit message
Description of the change
jobs/infrastruc
Unmerged commits
- 4fb056a... by =?utf-8?q?Konrad_Zapa=C5=82owicz?= <email address hidden>
-
jobs/infrastruc
ture: add missing file - a7a4e17... by System Enablement CI Bot <email address hidden>
-
Merge remote tracking branch debug-perm
Merge-Proposal: https:/
/code.launchpad .net/~alfonsosa nchezbeato/ snappy- hwe-snaps/ +git/jenkins- jobs/+merge/ 326212 Author: Alfonso Sanchez-Beato <email address hidden>
automerge: do not fail due to lack of e-mail
If the registrant of the MP had not defined an e-mail in the launchpad
account, we were failing to merge. - dd90414... by System Enablement CI Bot <email address hidden>
-
Merge remote tracking branch master
Merge-Proposal: https:/
/code.launchpad .net/~jasonlo/ snappy- hwe-snaps/ +git/jenkins- jobs/+merge/ 325971 Author: lo <email address hidden>
Create a system-
enablement- image-build- worker item on Jenkins to build the final image tarball. - 9d2a88b... by lo
-
Rebase to top of master.
- ff946e1... by lo
-
Remove output-dir from yaml and verify build worker script.
- de2c800... by lo
-
Verify description to be more general.
- 54b8a81... by lo
-
Use quote on variables.
- bede38b... by lo
-
Verify according to image build document.
- 7e81583... by lo
-
Add outdir parameter to indicate output file directory. To avoid frequently deploying after build.sh done some FIXME in the future, we keep one build.sh command in image-build-worker on Jenkins.
- 56b9a84... by lo
-
To expand build.sh from brando as an example of image build process.
Preview Diff
1 | diff --git a/.ci_tests_disabled b/.ci_tests_disabled |
2 | new file mode 100644 |
3 | index 0000000..e69de29 |
4 | --- /dev/null |
5 | +++ b/.ci_tests_disabled |
6 | diff --git a/README.md b/README.md |
7 | index 1a49a98..c4af3d3 100644 |
8 | --- a/README.md |
9 | +++ b/README.md |
10 | @@ -1,3 +1,4 @@ |
11 | +<<<<<<< README.md |
12 | # wifi-ap |
13 | |
14 | This snap provided WiFi AP functionality out of the box. |
15 | @@ -81,3 +82,58 @@ snap. |
16 | If you want to see more verbose debugging output of spread run |
17 | |
18 | $ ./run-tests --debug |
19 | +======= |
20 | +# Jenkins Jobs |
21 | + |
22 | +This is a repository collecting a set of jenkins job definitions |
23 | +used to build a contineous integrate pipeline for snap based |
24 | +components. |
25 | + |
26 | +## Required Jenkins Plugins |
27 | + |
28 | + * build-name-setter |
29 | + * conditional-buildstep |
30 | + * description-setter |
31 | + * dynamic-axis |
32 | + * matrix-combinations-parameter |
33 | + * matrix-project |
34 | + * nodelabelparameter |
35 | + * parameterized-trigger |
36 | + * rebuild |
37 | + * timestamper |
38 | + * ws-cleanup |
39 | + |
40 | +## Test locally |
41 | + |
42 | +Build the docker container which comes with this repository: |
43 | + |
44 | +``` |
45 | +$ cd docker |
46 | +$ sudo docker build -t se-jenkins . |
47 | +``` |
48 | + |
49 | +Spawn up a docker container with jenkins |
50 | + |
51 | +``` |
52 | +$ sudo docker run -p 8080:8080 -p 50000:50000 --name jenkins se-jenkins |
53 | +``` |
54 | + |
55 | +You can now login into jenkins on http://localhost:8080 with user 'system-enablement-ci-bot' |
56 | +and password 'jenkins'. |
57 | + |
58 | +Install jenkins-job-builder on your host via |
59 | + |
60 | +``` |
61 | +$ sudo apt install python3-pip |
62 | +$ pip3 install jenkins-job-builder |
63 | +``` |
64 | + |
65 | +Now you should have everything setup and ready for a first deployment: |
66 | + |
67 | +``` |
68 | +$ jenkins-jobs --conf local.conf update local.yaml:jobs/infrastructure:jobs/snap:jobs/image |
69 | +``` |
70 | + |
71 | +This will create all configured jobs on the jenkins instance or update |
72 | +them if already available. |
73 | +>>>>>>> README.md |
74 | diff --git a/docker/Dockerfile b/docker/Dockerfile |
75 | new file mode 100644 |
76 | index 0000000..a898d1e |
77 | --- /dev/null |
78 | +++ b/docker/Dockerfile |
79 | @@ -0,0 +1,59 @@ |
80 | +FROM ubuntu:latest |
81 | + |
82 | +ENV DEBIAN_FRONTEND noninteractive |
83 | +ENV INITRD No |
84 | +ENV LANG en_US.UTF-8 |
85 | + |
86 | +RUN apt-get update && \ |
87 | + apt-get upgrade -y && \ |
88 | + apt-get install -y --no-install-recommends \ |
89 | + vim.tiny wget curl sudo net-tools pwgen \ |
90 | + git-core logrotate software-properties-common locales openssh-client && \ |
91 | + locale-gen en_US en_US.UTF-8 && \ |
92 | + apt-get clean && \ |
93 | + rm -rf /var/lib/apt/lists/* |
94 | + |
95 | +RUN apt-get update && \ |
96 | + apt-get install --no-install-recommends -y openjdk-8-jre-headless && \ |
97 | + apt-get clean && \ |
98 | + rm -rf /var/lib/apt/lists/* |
99 | + |
100 | +RUN wget -qO - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add - && \ |
101 | + echo 'deb http://pkg.jenkins-ci.org/debian binary/' \ |
102 | + | tee /etc/apt/sources.list.d/jenkins.list && \ |
103 | + apt-get update && \ |
104 | + apt-get install --no-install-recommends -y jenkins && \ |
105 | + apt-get clean && \ |
106 | + rm -rf /var/lib/apt/lists/* && \ |
107 | + update-rc.d -f jenkins disable |
108 | + |
109 | +RUN usermod -d /var/jenkins jenkins |
110 | + |
111 | +# Passthrough all sudo requests without requiring a password for our jenkins user |
112 | +RUN echo "jenkins ALL = NOPASSWD: ALL" > /etc/sudoers.d/jenkins |
113 | + |
114 | +RUN apt-get update && apt-get install --yes unzip zip wget |
115 | + |
116 | +ENV JENKINS_UC https://updates.jenkins-ci.org |
117 | +ENV JENKINS_HOME /var/jenkins |
118 | +ENV COPY_REFERENCE_FILE_LOG $JENKINS_HOME/copy_reference_file.log |
119 | +ENV JENKINS_USER system-enablement-ci-bot |
120 | +ENV JENKINS_PASS jenkins |
121 | + |
122 | +COPY plugins.sh /usr/local/bin/plugins.sh |
123 | +RUN chmod +x /usr/local/bin/plugins.sh |
124 | +COPY plugins.txt /tmp/plugins.txt |
125 | +RUN /usr/local/bin/plugins.sh /tmp/plugins.txt |
126 | + |
127 | +COPY jenkins.sh /usr/local/bin/jenkins.sh |
128 | +RUN chmod +x /usr/local/bin/jenkins.sh |
129 | + |
130 | +COPY initial-setup.groovy /usr/share/jenkins/ref/init.groovy.d/ |
131 | + |
132 | +ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false |
133 | + |
134 | +EXPOSE 8080 50000 |
135 | + |
136 | +USER jenkins |
137 | + |
138 | +CMD ["/usr/local/bin/jenkins.sh"] |
139 | diff --git a/docker/initial-setup.groovy b/docker/initial-setup.groovy |
140 | new file mode 100644 |
141 | index 0000000..5a9598d |
142 | --- /dev/null |
143 | +++ b/docker/initial-setup.groovy |
144 | @@ -0,0 +1,34 @@ |
145 | +import jenkins.model.* |
146 | +import jenkins.security.* |
147 | +import hudson.security.* |
148 | + |
149 | +def env = System.getenv() |
150 | + |
151 | +def jenkins = Jenkins.getInstance() |
152 | + |
153 | +jenkins.setLabelString("snap build test release misc monitor") |
154 | + |
155 | +jenkins.setSecurityRealm(new HudsonPrivateSecurityRealm(false)) |
156 | +jenkins.setAuthorizationStrategy(new GlobalMatrixAuthorizationStrategy()) |
157 | + |
158 | +def user = jenkins.getSecurityRealm().createAccount(env.JENKINS_USER, env.JENKINS_PASS) |
159 | +ApiTokenProperty t = user.getProperty(ApiTokenProperty.class) |
160 | +def token = t.getApiTokenInsecure() |
161 | + |
162 | +println "" |
163 | +println "########################################################################" |
164 | +println "API token for user " + env.JENKINS_USER + " is " + token |
165 | +println "########################################################################" |
166 | +println "" |
167 | + |
168 | +user.save() |
169 | + |
170 | +jenkins.getAuthorizationStrategy().add(Jenkins.ADMINISTER, env.JENKINS_USER) |
171 | +jenkins.save() |
172 | + |
173 | +for (slave in jenkins.model.Jenkins.instance.slaves) { |
174 | + println "Slave: " + slave.getNodeName() + "\n"; |
175 | + println "Label: " + slave.getLabelString() + "\n\n"; |
176 | + slave.setLabelString("snap build test monitor release misc") |
177 | +} |
178 | +jenkins.save() |
179 | diff --git a/docker/jenkins.sh b/docker/jenkins.sh |
180 | new file mode 100644 |
181 | index 0000000..5d095be |
182 | --- /dev/null |
183 | +++ b/docker/jenkins.sh |
184 | @@ -0,0 +1,24 @@ |
185 | +#! /bin/bash |
186 | + |
187 | +set -e |
188 | + |
189 | +# Copy files from /usr/share/jenkins/ref into $JENKINS_HOME |
190 | +# So the initial JENKINS-HOME is set with expected content. |
191 | +# Don't override, as this is just a reference setup, and use from UI |
192 | +# can then change this, upgrade plugins, etc. |
193 | +copy_reference_file() { |
194 | + f=${1%/} |
195 | + rel=${f:23} |
196 | + dir=$(dirname ${f}) |
197 | + if [[ ! -e $JENKINS_HOME/${rel} ]] |
198 | + then |
199 | + mkdir -p $JENKINS_HOME/${dir:23} |
200 | + cp -r /usr/share/jenkins/ref/${rel} $JENKINS_HOME/${rel}; |
201 | + # pin plugins on initial copy |
202 | + [[ ${rel} == plugins/*.jpi ]] && touch $JENKINS_HOME/${rel}.pinned |
203 | + fi; |
204 | +} |
205 | +export -f copy_reference_file |
206 | +find /usr/share/jenkins/ref/ -type f -exec bash -c "copy_reference_file '{}'" \; |
207 | + |
208 | +exec /usr/bin/java $JAVA_OPTS -jar /usr/share/jenkins/jenkins.war |
209 | diff --git a/docker/plugins.sh b/docker/plugins.sh |
210 | new file mode 100755 |
211 | index 0000000..ec8e8e5 |
212 | --- /dev/null |
213 | +++ b/docker/plugins.sh |
214 | @@ -0,0 +1,28 @@ |
215 | +#! /bin/bash |
216 | + |
217 | +# Parse a support-core plugin -style txt file as specification for jenkins plugins to be installed |
218 | +# in the reference directory, so user can define a derived Docker image with just : |
219 | +# |
220 | +# FROM jenkins |
221 | +# COPY plugins.txt /plugins.txt |
222 | +# RUN /usr/local/bin/plugins.sh /plugins.txt |
223 | +# |
224 | + |
225 | +set -e |
226 | + |
227 | +REF=/usr/share/jenkins/ref/plugins |
228 | +mkdir -p $REF |
229 | + |
230 | +while read spec || [ -n "$spec" ]; do |
231 | + plugin=(${spec//:/ }); |
232 | + [[ ${plugin[0]} =~ ^# ]] && continue |
233 | + [[ ${plugin[0]} =~ ^\s*$ ]] && continue |
234 | + [[ -z ${plugin[1]} ]] && plugin[1]="latest" |
235 | + echo "Downloading ${plugin[0]}:${plugin[1]}" |
236 | + |
237 | + if [ -z "$JENKINS_UC_DOWNLOAD" ]; then |
238 | + JENKINS_UC_DOWNLOAD=$JENKINS_UC/download |
239 | + fi |
240 | + curl -sSL -f ${JENKINS_UC_DOWNLOAD}/plugins/${plugin[0]}/${plugin[1]}/${plugin[0]}.hpi -o $REF/${plugin[0]}.jpi |
241 | + unzip -qqt $REF/${plugin[0]}.jpi |
242 | +done < $1 |
243 | diff --git a/docker/plugins.txt b/docker/plugins.txt |
244 | new file mode 100644 |
245 | index 0000000..5020d33 |
246 | --- /dev/null |
247 | +++ b/docker/plugins.txt |
248 | @@ -0,0 +1,29 @@ |
249 | + build-name-setter |
250 | + conditional-buildstep |
251 | + description-setter |
252 | + dynamic-axis |
253 | + matrix-combinations-parameter |
254 | + matrix-project |
255 | + nodelabelparameter |
256 | + parameterized-trigger |
257 | + rebuild |
258 | + timestamper |
259 | + ws-cleanup |
260 | + maven-plugin |
261 | + token-macro |
262 | + junit |
263 | + script-security |
264 | + jquery |
265 | + workflow-basic-steps |
266 | + structs |
267 | + javadoc |
268 | + workflow-api |
269 | + workflow-step-api |
270 | + mailer |
271 | + matrix-auth |
272 | + scm-api |
273 | + display-url-api |
274 | + scm-api |
275 | + icon-shim |
276 | + resource-disposer |
277 | + run-condition |
278 | diff --git a/jobs/image/common-job-prepare.sh b/jobs/image/common-job-prepare.sh |
279 | new file mode 100644 |
280 | index 0000000..5feaa8d |
281 | --- /dev/null |
282 | +++ b/jobs/image/common-job-prepare.sh |
283 | @@ -0,0 +1,7 @@ |
284 | +cat << EOF > $WORKSPACE/.build_env |
285 | +BOT_USERNAME={bot_username} |
286 | +LAUNCHPAD_PROJECT={launchpad_project} |
287 | +LAUNCHPAD_TEAM={launchpad_team} |
288 | +BUILD_SCRIPTS=$WORKSPACE/jenkins-jobs |
289 | +BUILD_ID=$BUILD_ID |
290 | +EOF |
291 | diff --git a/jobs/image/image-build-worker.sh b/jobs/image/image-build-worker.sh |
292 | new file mode 100644 |
293 | index 0000000..161c5e4 |
294 | --- /dev/null |
295 | +++ b/jobs/image/image-build-worker.sh |
296 | @@ -0,0 +1,27 @@ |
297 | +#!/bin/bash |
298 | +# |
299 | +# Copyright (C) 2017 Canonical Ltd |
300 | +# |
301 | +# This program is free software: you can redistribute it and/or modify |
302 | +# it under the terms of the GNU General Public License version 3 as |
303 | +# published by the Free Software Foundation. |
304 | +# |
305 | +# This program is distributed in the hope that it will be useful, |
306 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
307 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
308 | +# GNU General Public License for more details. |
309 | +# |
310 | +# You should have received a copy of the GNU General Public License |
311 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
312 | + |
313 | +set -ex |
314 | + |
315 | +. "$WORKSPACE/.build_env" |
316 | + |
317 | +rm -rf "$WORKSPACE"/image-builds |
318 | + |
319 | +git clone git+ssh://git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/image-builds |
320 | +cd image-builds |
321 | + |
322 | +# Project specific image build process |
323 | +./build.sh |
324 | diff --git a/jobs/image/image-build-worker.yaml b/jobs/image/image-build-worker.yaml |
325 | new file mode 100644 |
326 | index 0000000..f34faf3 |
327 | --- /dev/null |
328 | +++ b/jobs/image/image-build-worker.yaml |
329 | @@ -0,0 +1,15 @@ |
330 | +- job-template: |
331 | + name: '{name}-image-build-worker' |
332 | + project-type: freestyle |
333 | + defaults: global |
334 | + description: "Build an image." |
335 | + display-name: "{name}-image-build-worker" |
336 | + concurrent: true |
337 | + node: snap && build |
338 | + builders: |
339 | + - shell: |
340 | + !include-raw: |
341 | + - common-job-prepare.sh |
342 | + - shell: |
343 | + !include-raw-escape: |
344 | + - image-build-worker.sh |
345 | diff --git a/jobs/image/image-project-jobs.yaml b/jobs/image/image-project-jobs.yaml |
346 | new file mode 100644 |
347 | index 0000000..0b517c7 |
348 | --- /dev/null |
349 | +++ b/jobs/image/image-project-jobs.yaml |
350 | @@ -0,0 +1,4 @@ |
351 | +- job-group: |
352 | + name: image-project-jobs |
353 | + jobs: |
354 | + - '{name}-image-build-worker' |
355 | diff --git a/jobs/infrastructure/common-job-prepare.sh b/jobs/infrastructure/common-job-prepare.sh |
356 | new file mode 100644 |
357 | index 0000000..3dbc9fd |
358 | --- /dev/null |
359 | +++ b/jobs/infrastructure/common-job-prepare.sh |
360 | @@ -0,0 +1,14 @@ |
361 | +JENKINS_JOBS_GIT_REPO="{jobs-git-repo}" |
362 | +JENKINS_JOBS_GIT_REPO_BRANCH="{jobs-git-repo-branch}" |
363 | + |
364 | +if [ -n "${{CLEANUP_WORKSPACE}}" ] && [ "${{CLEANUP_WORKSPACE}}" -eq 1 ]; then |
365 | + rm -rf ${{WORKSPACE}}/* |
366 | +fi |
367 | + |
368 | +cat << EOF > $WORKSPACE/.build_env |
369 | +JENKINS_JOBS_REPO="{jobs-git-repo}" |
370 | +JENKINS_JOBS_BRANCH="{jobs-git-repo-branch}" |
371 | +JENKINS_CONFIG_REPO="{config-git-repo}" |
372 | +JENKINS_CONFIG_BRANCH="{config-git-repo-branch}" |
373 | +JENKINS_INSTANCE="{jenkins-instance}" |
374 | +EOF |
375 | diff --git a/jobs/infrastructure/credentials-0-ssh.sh b/jobs/infrastructure/credentials-0-ssh.sh |
376 | new file mode 100644 |
377 | index 0000000..d72fd49 |
378 | --- /dev/null |
379 | +++ b/jobs/infrastructure/credentials-0-ssh.sh |
380 | @@ -0,0 +1,58 @@ |
381 | +#!/bin/sh -ex |
382 | +# |
383 | +# Copyright (C) 2016 Canonical Ltd |
384 | +# |
385 | +# This program is free software: you can redistribute it and/or modify |
386 | +# it under the terms of the GNU General Public License version 3 as |
387 | +# published by the Free Software Foundation. |
388 | +# |
389 | +# This program is distributed in the hope that it will be useful, |
390 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
391 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
392 | +# GNU General Public License for more details. |
393 | +# |
394 | +# You should have received a copy of the GNU General Public License |
395 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
396 | + |
397 | +# put it in $JENKINS_HOME so it doesn't get purged with the workspace |
398 | +SSH_PATH="${{JENKINS_HOME}}/.ssh/" |
399 | + |
400 | +# this is so that this key's only used with git |
401 | +SSH_KEY_PATH="${{SSH_PATH}}/git.launchpad.net/{bot_username}" |
402 | + |
403 | +if [ ! -d "${{SSH_KEY_PATH}}" ]; then |
404 | + mkdir -p "${{SSH_KEY_PATH}}" |
405 | +fi |
406 | + |
407 | +# don't care about host keys, but use the one id key for launchpad |
408 | +# TODO: use http://pad.lv/p/canonical-sshebang instead |
409 | +cat > "${{SSH_PATH}}/config" << EOF |
410 | +Host * |
411 | + StrictHostKeyChecking no |
412 | + |
413 | +Host git.launchpad.net |
414 | + User {bot_username} |
415 | + IdentityFile ~/.ssh/%h/%r/id_rsa |
416 | +EOF |
417 | + |
418 | +# only use keys if both were uploaded, otherwise fail |
419 | +if [ -f "keys/id_rsa" -o -f "keys/id_rsa.pub" ]; then |
420 | + [ -f "keys/id_rsa" -a -f "keys/id_rsa.pub" ] || \ |
421 | + (echo "ERROR: You need to upload both keys, or none of them"; exit 1) |
422 | + mv "keys/id_rsa" "keys/id_rsa.pub" "${{SSH_KEY_PATH}}/" |
423 | + chmod 600 ${{SSH_KEY_PATH}}/* |
424 | +fi |
425 | + |
426 | +# generate the keypair if none was uploaded or pre-existing |
427 | +if [ ! -f "${{SSH_KEY_PATH}}/id_rsa" ]; then |
428 | + ssh-keygen -f "${{SSH_KEY_PATH}}/id_rsa" |
429 | +fi |
430 | + |
431 | +# display the public key |
432 | +echo "The public key for this jenkins user is:" |
433 | +echo "----------" |
434 | +cat "${{SSH_KEY_PATH}}/id_rsa.pub" |
435 | +echo "----------" |
436 | + |
437 | +# Probe ssh public ssh key for relevant hosts and add it to our known_hosts file |
438 | +ssh-keyscan -t rsa,dsa git.launchpad.net >> ${{SSH_PATH}}/known_hosts |
439 | diff --git a/jobs/infrastructure/credentials-0-ssh.yaml b/jobs/infrastructure/credentials-0-ssh.yaml |
440 | new file mode 100644 |
441 | index 0000000..add42d4 |
442 | --- /dev/null |
443 | +++ b/jobs/infrastructure/credentials-0-ssh.yaml |
444 | @@ -0,0 +1,29 @@ |
445 | +- job-template: |
446 | + name: '{name}-credentials-0-ssh' |
447 | + project-type: matrix |
448 | + description: | |
449 | + This job will generate or store the supplied RSA keypair |
450 | + and display the public key for use with Launchpad. |
451 | + properties: |
452 | + - build-discarder: |
453 | + num-to-keep: 1 |
454 | + - rebuild |
455 | + parameters: |
456 | + - matrix-combinations: |
457 | + name: nodes |
458 | + - file: |
459 | + name: 'keys/id_rsa' |
460 | + description: Private RSA key |
461 | + - file: |
462 | + name: 'keys/id_rsa.pub' |
463 | + description: Public RSA key |
464 | + axes: |
465 | + - axis: |
466 | + type: slave |
467 | + name: node |
468 | + values: '{obj:build_slaves}' |
469 | + wrappers: |
470 | + - timestamps |
471 | + builders: |
472 | + - shell: |
473 | + !include-raw: credentials-0-ssh.sh |
474 | diff --git a/jobs/infrastructure/credentials-1-launchpad.py b/jobs/infrastructure/credentials-1-launchpad.py |
475 | new file mode 100644 |
476 | index 0000000..59dd706 |
477 | --- /dev/null |
478 | +++ b/jobs/infrastructure/credentials-1-launchpad.py |
479 | @@ -0,0 +1,86 @@ |
480 | +#!/usr/bin/env python |
481 | +# |
482 | +# Copyright (C) 2016 Canonical Ltd |
483 | +# |
484 | +# This program is free software: you can redistribute it and/or modify |
485 | +# it under the terms of the GNU General Public License version 3 as |
486 | +# published by the Free Software Foundation. |
487 | +# |
488 | +# This program is distributed in the hope that it will be useful, |
489 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
490 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
491 | +# GNU General Public License for more details. |
492 | +# |
493 | +# You should have received a copy of the GNU General Public License |
494 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
495 | + |
496 | +import sys |
497 | +import time |
498 | + |
499 | +from launchpadlib.credentials import RequestTokenAuthorizationEngine |
500 | +from launchpadlib.credentials import UnencryptedFileCredentialStore |
501 | +from launchpadlib.launchpad import Launchpad |
502 | +from lazr.restfulclient.errors import HTTPError |
503 | + |
504 | + |
505 | +ACCESS_TOKEN_POLL_TIME = 1 |
506 | +WAITING_FOR_USER = """Open this link: |
507 | +{{}} |
508 | +to authorize this program to access Launchpad on your behalf. |
509 | +Waiting to hear from Launchpad about your decision. . . .""" |
510 | + |
511 | + |
512 | +class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine): |
513 | + """Authorize a token in a server environment (with no browser). |
514 | + |
515 | + Print a link for the user to copy-and-paste into his/her browser |
516 | + for authentication. |
517 | + """ |
518 | + |
519 | + def __init__(self, *args, **kwargs): |
520 | + # as implemented in AuthorizeRequestTokenWithBrowser |
521 | + kwargs['consumer_name'] = None |
522 | + kwargs.pop('allow_access_levels', None) |
523 | + super(AuthorizeRequestTokenWithConsole, self).__init__(*args, **kwargs) |
524 | + |
525 | + def make_end_user_authorize_token(self, credentials, request_token): |
526 | + """Ask the end-user to authorize the token in their browser. |
527 | + |
528 | + """ |
529 | + authorization_url = self.authorization_url(request_token) |
530 | + print(WAITING_FOR_USER.format(authorization_url)) |
531 | + # if we don't flush we may not see the message |
532 | + sys.stdout.flush() |
533 | + while credentials.access_token is None: |
534 | + time.sleep(ACCESS_TOKEN_POLL_TIME) |
535 | + try: |
536 | + credentials.exchange_request_token_for_access_token( |
537 | + self.web_root) |
538 | + break |
539 | + except HTTPError as e: |
540 | + if e.response.status == 403: |
541 | + # The user decided not to authorize this |
542 | + # application. |
543 | + raise e |
544 | + elif e.response.status == 401: |
545 | + # The user has not made a decision yet. |
546 | + pass |
547 | + else: |
548 | + # There was an error accessing the server. |
549 | + raise e |
550 | + |
551 | + |
552 | +def get_launchpad(cred_path, launchpadlib_dir=None): |
553 | + """ return a launchpad API class. In case launchpadlib_dir is |
554 | + specified used that directory to store launchpadlib cache instead of |
555 | + the default """ |
556 | + store = UnencryptedFileCredentialStore(cred_path) |
557 | + authorization_engine = AuthorizeRequestTokenWithConsole( |
558 | + 'production', 'ci-jenkins-slave') |
559 | + return Launchpad.login_with('ci-jenkins-slave', 'production', |
560 | + credential_store=store, |
561 | + authorization_engine=authorization_engine, |
562 | + launchpadlib_dir=launchpadlib_dir) |
563 | + |
564 | +if __name__ == '__main__': |
565 | + get_launchpad("{credentials_path}") |
566 | diff --git a/jobs/infrastructure/credentials-1-launchpad.yaml b/jobs/infrastructure/credentials-1-launchpad.yaml |
567 | new file mode 100644 |
568 | index 0000000..d1c4968 |
569 | --- /dev/null |
570 | +++ b/jobs/infrastructure/credentials-1-launchpad.yaml |
571 | @@ -0,0 +1,21 @@ |
572 | +- job-template: |
573 | + name: '{name}-credentials-1-launchpad' |
574 | + project-type: matrix |
575 | + description: This job creates or validates OAuth tokens to allow launchpadlib to work. |
576 | + properties: |
577 | + - build-discarder: |
578 | + num-to-keep: 1 |
579 | + - rebuild |
580 | + parameters: |
581 | + - matrix-combinations: |
582 | + name: nodes |
583 | + axes: |
584 | + - axis: |
585 | + type: slave |
586 | + name: node |
587 | + values: '{obj:build_slaves}' |
588 | + wrappers: |
589 | + - timestamps |
590 | + builders: |
591 | + - shell: |
592 | + !include-raw: credentials-1-launchpad.py |
593 | diff --git a/jobs/infrastructure/credentials-2-launchpad-plugin.sh b/jobs/infrastructure/credentials-2-launchpad-plugin.sh |
594 | new file mode 100644 |
595 | index 0000000..d47164f |
596 | --- /dev/null |
597 | +++ b/jobs/infrastructure/credentials-2-launchpad-plugin.sh |
598 | @@ -0,0 +1,108 @@ |
599 | +#!/bin/bash -ex |
600 | +# |
601 | +# Copyright (C) 2016 Canonical Ltd |
602 | +# |
603 | +# This program is free software: you can redistribute it and/or modify |
604 | +# it under the terms of the GNU General Public License version 3 as |
605 | +# published by the Free Software Foundation. |
606 | +# |
607 | +# This program is distributed in the hope that it will be useful, |
608 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
609 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
610 | +# GNU General Public License for more details. |
611 | +# |
612 | +# You should have received a copy of the GNU General Public License |
613 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
614 | + |
615 | + # Apply the configuration file. |
616 | +CONFIG_DIR="${{JENKINS_HOME}}/.jlp" |
617 | +CONFIG_PATH="${{CONFIG_DIR}}/jlp.config" |
618 | + |
619 | +if [ ! -d "${{CONFIG_DIR}}" ]; then |
620 | + mkdir -p "${{CONFIG_DIR}}" |
621 | +fi |
622 | + |
623 | +cat > "${{CONFIG_PATH}}" << EOF |
624 | +#You must explicitely allow users to trigger the jobs on your jenkins |
625 | +#Otherwise anybody can run arbitrary code on your jenkins servers. |
626 | +allowed_users: [{allowed_users}] |
627 | + |
628 | +#path to your credentials file. The first time you run one of these scripts, |
629 | +#launchpad will ask you to authenticate (via a provided URL). Once you do so |
630 | +#(in launchpad) you won't need to do this again. |
631 | +#If your jenkins "lives" in /var/lib/jenkins you probably don't need to change |
632 | +#this |
633 | +credential_store_path: {credentials_path} |
634 | + |
635 | +# When doing a dput into ppa (in autoland.py) a new changelog entry is |
636 | +# generated. DEBEMAIL and DEBFULLNAME are used to generate the entry correctly. |
637 | +# Please note that the gpg keys of the user specified here must be available |
638 | +# on the host where autoland.py is running |
639 | +DEBEMAIL: |
640 | +DEBFULLNAME: |
641 | + |
642 | +#user and password for accessing jenkins. This is needed as we need to find |
643 | +#out if a job is being published to public jenkins or not. The user needs to be |
644 | +#able to see the job configuration |
645 | +jenkins_user: "{bot_username}" |
646 | +jenkins_password: "${{jenkins_api_token}}" |
647 | + |
648 | +#Actual URL of your jenkins (e.g. the jenkins backend URL) |
649 | +jenkins_url: "{backend_url}" |
650 | + |
651 | +#Proxy URL of your jenkins (e.g. the URL accessed by users) |
652 | +jenkins_proxy_url: "${{JENKINS_URL}}" |
653 | + |
654 | +#Token to pass when triggering a jenkins build (leave blank for none) |
655 | +jenkins_build_token: "BUILD_ME" |
656 | + |
657 | +# console output from the following jobs will not be printed to the |
658 | +# affected merge proposal (in the "Executed test runs:" section) |
659 | +jobs_blacklisted_from_messages: |
660 | +{blacklisted_jobs} |
661 | + |
662 | +#message that is used for "testing in progress" comment |
663 | +launchpad_build_in_progress_message: "Jenkins: testing in progress" |
664 | + |
665 | +#login of the launchpad user you will be using for this plugin |
666 | +#ideally this user is part of your project group |
667 | +launchpad_login: {bot_username} |
668 | + |
669 | +#Review type that is used for voting on merge proposals. |
670 | +#Usually you don't need to change this |
671 | +launchpad_review_type: continuous-integration |
672 | + |
673 | +# directory containing lockfiles for Launchpad merge proposals |
674 | +launchpadlocks_dir: /tmp/jenkins-launchpad-plugin/locks |
675 | + |
676 | +#lock file that is being used to limit the number of parallel launchpad |
677 | +#connections |
678 | +lock_name: launchpad-trigger-lock |
679 | + |
680 | +#you don't need to change this |
681 | +lp_app: launchpad-trigger |
682 | + |
683 | +#which launchpad are you using (production/staging) |
684 | +#you don't need to change this |
685 | +lp_env: production |
686 | + |
687 | +#URL of your public jenkins in case you are publishing your jobs to some |
688 | +#other jenkins |
689 | +public_jenkins_url: |
690 | + |
691 | +#in case you are running jenkins in a private infrastructure you probably don't |
692 | +#want to expose your private IPs in public merge proposals |
693 | +#the following defines (IP, replacement) pairs. Your URLs in merge proposals |
694 | +#are then replaced by the replacement (and you can e.g. edit your /etc/hosts |
695 | +#so the links still work for you). The form to specify a replacement is: |
696 | +#urls_to_hide: |
697 | +# - ['http://1.2.3.4:8080','http://jenkins:8080'] |
698 | +# |
699 | +#To specify no replacement: |
700 | +#urls_to_hide: [] |
701 | +urls_to_hide: [] |
702 | + |
703 | +# verbosity of the commands |
704 | +# one of: debug, info, warning, error, critical |
705 | +log_level: debug |
706 | +EOF |
707 | diff --git a/jobs/infrastructure/credentials-2-launchpad-plugin.yaml b/jobs/infrastructure/credentials-2-launchpad-plugin.yaml |
708 | new file mode 100644 |
709 | index 0000000..0d8ca61 |
710 | --- /dev/null |
711 | +++ b/jobs/infrastructure/credentials-2-launchpad-plugin.yaml |
712 | @@ -0,0 +1,24 @@ |
713 | +- job-template: |
714 | + name: '{name}-credentials-2-launchpad-plugin' |
715 | + project-type: matrix |
716 | + description: This job configures Launchpad integration. |
717 | + properties: |
718 | + - build-discarder: |
719 | + num-to-keep: 1 |
720 | + - rebuild |
721 | + parameters: |
722 | + - matrix-combinations: |
723 | + name: nodes |
724 | + - password: |
725 | + name: jenkins_api_token |
726 | + description: Jenkins API key of the "{bot_username}" account |
727 | + axes: |
728 | + - axis: |
729 | + type: slave |
730 | + name: node |
731 | + values: '{obj:build_slaves}' |
732 | + wrappers: |
733 | + - timestamps |
734 | + builders: |
735 | + - shell: |
736 | + !include-raw: credentials-2-launchpad-plugin.sh |
737 | diff --git a/jobs/infrastructure/deploy-jenkins-jobs.sh b/jobs/infrastructure/deploy-jenkins-jobs.sh |
738 | new file mode 100644 |
739 | index 0000000..7754fa9 |
740 | --- /dev/null |
741 | +++ b/jobs/infrastructure/deploy-jenkins-jobs.sh |
742 | @@ -0,0 +1,58 @@ |
743 | +#!/bin/sh |
744 | +# |
745 | +# Copyright (C) 2017 Canonical Ltd |
746 | +# |
747 | +# This program is free software: you can redistribute it and/or modify |
748 | +# it under the terms of the GNU General Public License version 3 as |
749 | +# published by the Free Software Foundation. |
750 | +# |
751 | +# This program is distributed in the hope that it will be useful, |
752 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
753 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPO. See the |
754 | +# GNU General Public License for more details. |
755 | +# |
756 | +# You should have received a copy of the GNU General Public License |
757 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
758 | + |
759 | +set -ex |
760 | + |
761 | +. "$WORKSPACE/.build_env" |
762 | + |
763 | +JENKINS_JOBS_FOLDER="jenkins-jobs" |
764 | +JENKINS_CONFIG_FOLDER="config" |
765 | + |
766 | +# Fetch Jenkaas configuration from config file |
767 | +CONFIG_DIR="$JENKINS_HOME/.jlp" |
768 | +CONFIG_PATH="$CONFIG_DIR/jlp.config" |
769 | + |
770 | +cat $CONFIG_PATH |
771 | + |
772 | +JENKINS_API_TOKEN=$(cat $CONFIG_PATH | grep 'jenkins_password' | sed 's/[a-z_]*:[ ]//g') |
773 | +JENKINS_DEPLOYMENT_SCOPE="jobs/infrastructure:jobs/snap" |
774 | + |
775 | +rm -rf $WORKSPACE/$JENKINS_JOBS_FOLDER |
776 | + |
777 | +# Clone the jenkins-jobs and jenkins-config repositories |
778 | +git clone -b $JENKINS_JOBS_BRANCH $JENKINS_JOBS_REPO $JENKINS_JOBS_FOLDER |
779 | +cd $JENKINS_JOBS_FOLDER |
780 | +git clone -b $JENKINS_CONFIG_BRANCH $JENKINS_CONFIG_REPO $JENKINS_CONFIG_FOLDER |
781 | + |
782 | +# Verify the configuration is present |
783 | +PROJECT_CONF="$JENKINS_CONFIG_FOLDER/$JENKINS_INSTANCE.conf" |
784 | +PROJECT_YAML="$JENKINS_CONFIG_FOLDER/$JENKINS_INSTANCE.yaml" |
785 | +if [ ! -f "$PROJECT_CONF" ] || [ ! -f "$PROJECT_YAML" ]; then |
786 | + echo "Cannot find .conf and .yaml pair for $JENKINS_INSTANCE" |
787 | + exit 1 |
788 | +fi |
789 | + |
790 | +# Update the config file with the API key for the Bot. |
791 | +sed -i "/<API TOKEN>/c\password=$JENKINS_API_TOKEN" "$PROJECT_CONF" |
792 | + |
793 | +# Test requested configuration |
794 | +/usr/local/bin/jenkins-jobs --conf $PROJECT_CONF test $PROJECT_YAML":"$JENKINS_DEPLOYMENT_SCOPE &>/dev/null |
795 | +if [ "$?" -ne 0 ]; then |
796 | + echo "Configuration failed verification" |
797 | + exit 1 |
798 | +fi |
799 | +# Deploy requested configuration |
800 | +/usr/local/bin/jenkins-jobs --conf $PROJECT_CONF update $PROJECT_YAML":"$JENKINS_DEPLOYMENT_SCOPE |
801 | diff --git a/jobs/infrastructure/deploy-jenkins-jobs.yaml b/jobs/infrastructure/deploy-jenkins-jobs.yaml |
802 | new file mode 100644 |
803 | index 0000000..24b24f0 |
804 | --- /dev/null |
805 | +++ b/jobs/infrastructure/deploy-jenkins-jobs.yaml |
806 | @@ -0,0 +1,24 @@ |
807 | +- job-template: |
808 | + name: '{name}-deploy-jenkins-jobs' |
809 | + project-type: matrix |
810 | + description: | |
811 | + This job will deploy jenkins jobs. |
812 | + properties: |
813 | + - build-discarder: |
814 | + num-to-keep: 1 |
815 | + - rebuild |
816 | + parameters: |
817 | + - matrix-combinations: |
818 | + name: nodes |
819 | + axes: |
820 | + - axis: |
821 | + type: slave |
822 | + name: node |
823 | + values: '{obj:build_slaves}' |
824 | + wrappers: |
825 | + - timestamps |
826 | + builders: |
827 | + - shell: |
828 | + !include-raw: common-job-prepare.sh |
829 | + - shell: |
830 | + !include-raw-escape: deploy-jenkins-jobs.sh |
831 | diff --git a/jobs/infrastructure/infrastructure-jobs.yaml b/jobs/infrastructure/infrastructure-jobs.yaml |
832 | new file mode 100644 |
833 | index 0000000..d440330 |
834 | --- /dev/null |
835 | +++ b/jobs/infrastructure/infrastructure-jobs.yaml |
836 | @@ -0,0 +1,8 @@ |
837 | +- job-group: |
838 | + name: infrastructure-jobs |
839 | + jobs: |
840 | + - '{name}-prepare-0-install' |
841 | + - '{name}-credentials-0-ssh' |
842 | + - '{name}-credentials-1-launchpad' |
843 | + - '{name}-credentials-2-launchpad-plugin' |
844 | + - '{name}-deploy-jenkins-jobs' |
845 | diff --git a/jobs/infrastructure/prepare-0-install.sh b/jobs/infrastructure/prepare-0-install.sh |
846 | new file mode 100644 |
847 | index 0000000..9fae262 |
848 | --- /dev/null |
849 | +++ b/jobs/infrastructure/prepare-0-install.sh |
850 | @@ -0,0 +1,47 @@ |
851 | +#!/bin/sh -ex |
852 | +# |
853 | +# Copyright (C) 2016 Canonical Ltd |
854 | +# |
855 | +# This program is free software: you can redistribute it and/or modify |
856 | +# it under the terms of the GNU General Public License version 3 as |
857 | +# published by the Free Software Foundation. |
858 | +# |
859 | +# This program is distributed in the hope that it will be useful, |
860 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
861 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
862 | +# GNU General Public License for more details. |
863 | +# |
864 | +# You should have received a copy of the GNU General Public License |
865 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
866 | + |
867 | +sudo apt-get install --yes software-properties-common |
868 | + |
869 | +# build tools as used in Launchpad |
870 | +sudo add-apt-repository --yes ppa:launchpad/buildd-staging |
871 | +sudo add-apt-repository --yes ppa:jenkaas-hackers/tools |
872 | +sudo add-apt-repository --yes ppa:snappy-hwe-team/ci-tools |
873 | + |
874 | +sudo apt-get update |
875 | + |
876 | +sudo apt-get install --yes \ |
877 | + git \ |
878 | + python \ |
879 | + python-launchpadlib \ |
880 | + python-bzrlib \ |
881 | + python-lockfile \ |
882 | + python-yaml \ |
883 | + python-jenkins \ |
884 | + python-git \ |
885 | + python-pip \ |
886 | + tarmac \ |
887 | + jenkins-launchpad-plugin \ |
888 | + openssh-client \ |
889 | + debootstrap \ |
890 | + qemu \ |
891 | + binfmt-support \ |
892 | + qemu-user-static \ |
893 | + {install_packages} |
894 | + |
895 | +# Install jenkins-jobs-builder |
896 | +sudo pip install --upgrade pip |
897 | +sudo pip install jenkins-job-builder |
898 | diff --git a/jobs/infrastructure/prepare-0-install.yaml b/jobs/infrastructure/prepare-0-install.yaml |
899 | new file mode 100644 |
900 | index 0000000..32e9612 |
901 | --- /dev/null |
902 | +++ b/jobs/infrastructure/prepare-0-install.yaml |
903 | @@ -0,0 +1,27 @@ |
904 | +- job-template: |
905 | + name: '{name}-prepare-0-install' |
906 | + install_packages: '' |
907 | + project-type: matrix |
908 | + description: | |
909 | + This job adds all the needed repositories and installs dependencies needed |
910 | + on build slaves. |
911 | + properties: |
912 | + - build-discarder: |
913 | + num-to-keep: 10 |
914 | + - rebuild |
915 | + parameters: |
916 | + - matrix-combinations: |
917 | + name: configurations |
918 | + description: Which slaves to install packages on |
919 | + - string: |
920 | + name: CLEANUP_WORKSPACE |
921 | + default: "0" |
922 | + axes: |
923 | + - axis: |
924 | + type: slave |
925 | + name: node |
926 | + values: '{obj:build_slaves}' |
927 | + builders: |
928 | + - shell: |
929 | + !include-raw: |
930 | + - prepare-0-install.sh |
931 | diff --git a/jobs/snap/common-job-prepare.sh b/jobs/snap/common-job-prepare.sh |
932 | new file mode 100644 |
933 | index 0000000..4be0408 |
934 | --- /dev/null |
935 | +++ b/jobs/snap/common-job-prepare.sh |
936 | @@ -0,0 +1,24 @@ |
937 | +JENKINS_JOBS_GIT_REPO="{jobs-git-repo}" |
938 | +JENKINS_JOBS_GIT_REPO_BRANCH="{jobs-git-repo-branch}" |
939 | + |
940 | +if [ -n "${{CLEANUP_WORKSPACE}}" ] && [ "${{CLEANUP_WORKSPACE}}" -eq 1 ]; then |
941 | + rm -rf ${{WORKSPACE}}/* |
942 | +fi |
943 | +if [ -e jenkins-jobs ] ; then |
944 | + (cd jenkins-jobs ; git clean -fdx . ; git fetch origin ; git reset --hard origin/${{JENKINS_JOBS_GIT_REPO_BRANCH}}) |
945 | +else |
946 | + git clone -b ${{JENKINS_JOBS_GIT_REPO_BRANCH}} ${{JENKINS_JOBS_GIT_REPO}} |
947 | +fi |
948 | + |
949 | +cat << EOF > $WORKSPACE/.build_env |
950 | +BOT_USERNAME={bot_username} |
951 | +LAUNCHPAD_PROJECT={launchpad_project} |
952 | +LAUNCHPAD_TEAM={launchpad_team} |
953 | +SNAP_BUILD_JOB={name}-snap-build |
954 | +BUILD_SCRIPTS=$WORKSPACE/jenkins-jobs |
955 | +BUILD_ON_LAUNCHPAD={build_on_launchpad} |
956 | +AUTO_MERGE={auto_merge} |
957 | +TRIGGER_CI={trigger_ci} |
958 | +UPDATE_MPS={update_mps} |
959 | +RUN_TESTS={run_tests} |
960 | +EOF |
961 | diff --git a/jobs/snap/snap-automerger.sh b/jobs/snap/snap-automerger.sh |
962 | new file mode 100644 |
963 | index 0000000..d955e52 |
964 | --- /dev/null |
965 | +++ b/jobs/snap/snap-automerger.sh |
966 | @@ -0,0 +1,27 @@ |
967 | +#!/bin/bash |
968 | +# |
969 | +# Copyright (C) 2017 Canonical Ltd |
970 | +# |
971 | +# This program is free software: you can redistribute it and/or modify |
972 | +# it under the terms of the GNU General Public License version 3 as |
973 | +# published by the Free Software Foundation. |
974 | +# |
975 | +# This program is distributed in the hope that it will be useful, |
976 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
977 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
978 | +# GNU General Public License for more details. |
979 | +# |
980 | +# You should have received a copy of the GNU General Public License |
981 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
982 | + |
983 | +set -ex |
984 | + |
985 | +. "$WORKSPACE/.build_env" |
986 | + |
987 | +if [ "$AUTO_MERGE" = False ]; then |
988 | + echo "WARNING: auto merge is disabled" |
989 | + exit 0 |
990 | +fi |
991 | + |
992 | +exec $BUILD_SCRIPTS/tools/automerge-mps.py \ |
993 | + -p $LAUNCHPAD_PROJECT |
994 | \ No newline at end of file |
995 | diff --git a/jobs/snap/snap-automerger.yaml b/jobs/snap/snap-automerger.yaml |
996 | new file mode 100644 |
997 | index 0000000..f55f252 |
998 | --- /dev/null |
999 | +++ b/jobs/snap/snap-automerger.yaml |
1000 | @@ -0,0 +1,22 @@ |
1001 | +- job-template: |
1002 | + name: '{name}-snap-automerger' |
1003 | + project-type: freestyle |
1004 | + defaults: global |
1005 | + description: "Monitor Launchpad for new merge proposals" |
1006 | + display-name: "{name}-snap-automerger" |
1007 | + concurrent: true |
1008 | + node: snap && misc |
1009 | + triggers: |
1010 | + - timed: # every five minutes |
1011 | + H/5 * * * * |
1012 | + properties: |
1013 | + - build-discarder: |
1014 | + num-to-keep: 10 |
1015 | + - rebuild |
1016 | + builders: |
1017 | + - shell: |
1018 | + !include-raw: |
1019 | + - common-job-prepare.sh |
1020 | + - shell: |
1021 | + !include-raw-escape: |
1022 | + - snap-automerger.sh |
1023 | diff --git a/jobs/snap/snap-build-update-chroot.sh b/jobs/snap/snap-build-update-chroot.sh |
1024 | new file mode 100644 |
1025 | index 0000000..912f687 |
1026 | --- /dev/null |
1027 | +++ b/jobs/snap/snap-build-update-chroot.sh |
1028 | @@ -0,0 +1,28 @@ |
1029 | +#!/bin/sh |
1030 | +# |
1031 | +# Copyright (C) 2016 Canonical Ltd |
1032 | +# |
1033 | +# This program is free software: you can redistribute it and/or modify |
1034 | +# it under the terms of the GNU General Public License version 3 as |
1035 | +# published by the Free Software Foundation. |
1036 | +# |
1037 | +# This program is distributed in the hope that it will be useful, |
1038 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
1039 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
1040 | +# GNU General Public License for more details. |
1041 | +# |
1042 | +# You should have received a copy of the GNU General Public License |
1043 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
1044 | + |
1045 | +set -ex |
1046 | + |
1047 | +. "$WORKSPACE/.build_env" |
1048 | + |
1049 | +if [ "$BUILD_ON_LAUNCHPAD" != False ]; then |
1050 | + exit 0 |
1051 | +fi |
1052 | + |
1053 | +sudo $BUILD_SCRIPTS/tools/snapbuild.sh \ |
1054 | + --update-chroot \ |
1055 | + --arch=$ARCHITECTURE \ |
1056 | + --series=$SERIES |
1057 | \ No newline at end of file |
1058 | diff --git a/jobs/snap/snap-build-update-chroot.yaml b/jobs/snap/snap-build-update-chroot.yaml |
1059 | new file mode 100644 |
1060 | index 0000000..e50b3c1 |
1061 | --- /dev/null |
1062 | +++ b/jobs/snap/snap-build-update-chroot.yaml |
1063 | @@ -0,0 +1,29 @@ |
1064 | +- job-template: |
1065 | + name: '{name}-snap-build-update-chroot' |
1066 | + project-type: matrix |
1067 | + defaults: global |
1068 | + description: "Build a snap on launchpad" |
1069 | + display-name: "{name}-snap-build-update-chroot" |
1070 | + concurrent: false |
1071 | + node: snap && build |
1072 | + axes: |
1073 | + - axis: |
1074 | + type: slave |
1075 | + name: node |
1076 | + values: '{obj:build_slaves}' |
1077 | + parameters: |
1078 | + - string: |
1079 | + name: ARCHITECTURE |
1080 | + default: amd64 |
1081 | + description: Architecture to build the snap for |
1082 | + - string: |
1083 | + name: SERIES |
1084 | + default: xenial |
1085 | + description: "Ubuntu archive series to build for" |
1086 | + builders: |
1087 | + - shell: |
1088 | + !include-raw: |
1089 | + - common-job-prepare.sh |
1090 | + - shell: |
1091 | + !include-raw-escape: |
1092 | + - snap-build-update-chroot.sh |
1093 | diff --git a/jobs/snap/snap-build-worker.sh b/jobs/snap/snap-build-worker.sh |
1094 | new file mode 100644 |
1095 | index 0000000..60775d7 |
1096 | --- /dev/null |
1097 | +++ b/jobs/snap/snap-build-worker.sh |
1098 | @@ -0,0 +1,147 @@ |
1099 | +#!/bin/sh |
1100 | +# |
1101 | +# Copyright (C) 2016 Canonical Ltd |
1102 | +# |
1103 | +# This program is free software: you can redistribute it and/or modify |
1104 | +# it under the terms of the GNU General Public License version 3 as |
1105 | +# published by the Free Software Foundation. |
1106 | +# |
1107 | +# This program is distributed in the hope that it will be useful, |
1108 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
1109 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
1110 | +# GNU General Public License for more details. |
1111 | +# |
1112 | +# You should have received a copy of the GNU General Public License |
1113 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
1114 | + |
1115 | +set -ex |
1116 | + |
1117 | +. "$WORKSPACE/.build_env" |
1118 | + |
1119 | +rm -rf $WORKSPACE/src $WORKSPACE/results $WORKSPACE/build-props |
1120 | + |
1121 | +git clone --no-checkout $TARGET_GIT_REPO $WORKSPACE/src |
1122 | +cd $WORKSPACE/src |
1123 | +for remote in `git branch -r | grep -v origin/master`; do git checkout --track $remote ; done |
1124 | + |
1125 | +git checkout $TARGET_GIT_REPO_BRANCH |
1126 | + |
1127 | +git config user.name "System Enablement CI Bot" |
1128 | +git config user.email "ce-system-enablement@lists.canonical.com" |
1129 | + |
1130 | +if [ -n "$SOURCE_GIT_REPO" ]; then |
1131 | + git remote add other $SOURCE_GIT_REPO |
1132 | + git fetch other |
1133 | + git merge \ |
1134 | + --no-ff \ |
1135 | + -m "Merge remote tracking branch other/$SOURCE_GIT_REPO_BRANCH" \ |
1136 | + $REVISION |
1137 | +fi |
1138 | + |
1139 | +# Try to find the correct branch we need to build from. In the case that |
1140 | +# $TARGET_GIT_REPO_BRANCH points us to an upstream component branch we |
1141 | +# will take master as the next suitable candidate. |
1142 | +CI_BRANCH= |
1143 | +SNAPCRAFT_YAML_PATH= |
1144 | +for branch in $TARGET_GIT_REPO_BRANCH master ; do |
1145 | + git checkout $branch |
1146 | + if [ -e snapcraft.yaml ]; then |
1147 | + SNAPCRAFT_YAML_PATH=snapcraft.yaml |
1148 | + elif [ -e snap/snapcraft.yaml ]; then |
1149 | + SNAPCRAFT_YAML_PATH=snap/snapcraft.yaml |
1150 | + fi |
1151 | + |
1152 | + if [ -n "$SNAPCRAFT_YAML_PATH" ]; then |
1153 | + CI_BRANCH=$branch |
1154 | + break |
1155 | + fi |
1156 | +done |
1157 | + |
1158 | +if [ -z "$CI_BRANCH" ]; then |
1159 | + echo "WARNING: Can't build snap as no snapcraft.yaml exists!" |
1160 | + exit 0 |
1161 | +fi |
1162 | + |
1163 | +REPO_NAME=$(awk -v a="$TARGET_GIT_REPO" 'BEGIN{print substr(a, index(a, "+git/") + 5)}') |
1164 | +# We rely on the snapcraft.yaml to have the snap name in the first five lines |
1165 | +# which is the case for all our snaps. This is a bit lazy but the best way to |
1166 | +# ensure we don't fetch any other name: fields which might be present in the file. |
1167 | +SNAP_NAME=$(cat $SNAPCRAFT_YAML_PATH | grep -v ^\# | head -n 5 | grep "^name:" | awk '{print $2}') |
1168 | +SNAP_REV=$(git rev-parse --short HEAD) |
1169 | +CI_REPO=$REPO_NAME-$BUILD_ID-$SNAP_REV |
1170 | + |
1171 | +sed -i "s/~$LAUNCHPAD_TEAM\/$LAUNCHPAD_PROJECT\/+git\/$REPO_NAME/~$LAUNCHPAD_TEAM\/$LAUNCHPAD_PROJECT\/+git\/$CI_REPO/g" \ |
1172 | + $SNAPCRAFT_YAML_PATH |
1173 | + |
1174 | +# The project as two different options of how snaps can be build: |
1175 | +# |
1176 | +# 1. Locally in a chroot but only for the host architecture |
1177 | +# 2. On launchpad for any architecture |
1178 | +# |
1179 | +# Which of both options will be used is configured in the |
1180 | +# $WORKSPACE/.build_env file. |
1181 | + |
1182 | +if [ "$BUILD_ON_LAUNCHPAD" = False ]; then |
1183 | + SNAPBUILD_EXTRA_ARGS= |
1184 | + SNAP_TYPE=$($BUILD_SCRIPTS/tools/shyaml get-value type < $SNAPCRAFT_YAML_PATH || echo app) |
1185 | + case "$SNAP_TYPE" in |
1186 | + kernel) |
1187 | + if [ "$ARCHITECTURE" != amd64 ]; then |
1188 | + # If we're building a kernel snap we have to cross-build it |
1189 | + # instead of building it in a native environment. |
1190 | + SNAPBUILD_EXTRA_ARGS="--cross-build" |
1191 | + fi |
1192 | + ;; |
1193 | + esac |
1194 | + |
1195 | + sudo $BUILD_SCRIPTS/tools/snapbuild.sh \ |
1196 | + --source-dir=$WORKSPACE/src \ |
1197 | + --results-dir=$WORKSPACE/results \ |
1198 | + --arch=$ARCHITECTURE \ |
1199 | + --series=$SERIES \ |
1200 | + --proxy=squid.internal:3128 \ |
1201 | + $SNAPBUILD_EXTRA_ARGS |
1202 | +else |
1203 | + git remote add jenkins-ci git+ssh://$BOT_USERNAME@git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO |
1204 | + git push jenkins-ci --all |
1205 | + git push jenkins-ci --tags |
1206 | + |
1207 | + # Save repo name as soon as it gets created so it can be deleted by the cleanup |
1208 | + # job even if this job fails. |
1209 | + echo "CI_REPO=$CI_REPO" >> $WORKSPACE/build-props |
1210 | + echo "CI_BRANCH=$CI_BRANCH" >> $WORKSPACE/build-props |
1211 | + |
1212 | + EXTRA_ARGS= |
1213 | + if [ -n "$ARCHITECTURE" ]; then |
1214 | + EXTRA_ARGS="$EXTRA_ARGS --architectures=$ARCHITECTURE" |
1215 | + fi |
1216 | + |
1217 | + $BUILD_SCRIPTS/tools/trigger-lp-build.py \ |
1218 | + -s $SNAP_NAME -n \ |
1219 | + --git-repo=https://git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO \ |
1220 | + --git-repo-branch=$CI_BRANCH \ |
1221 | + --results-dir=$WORKSPACE/results \ |
1222 | + $EXTRA_ARGS |
1223 | +fi |
1224 | + |
1225 | +if [ -z "$REMOTE_WORKER" ]; then |
1226 | + echo "INFO: No remote worker defined, not copying artifacts to it" |
1227 | + exit 0 |
1228 | +fi |
1229 | + |
1230 | +SSH_PATH="${JENKINS_HOME}/.ssh/" |
1231 | +SSH_KEY_PATH="${SSH_PATH}/git.launchpad.net/$BOT_USERNAME" |
1232 | +SSH="ssh -i $SSH_KEY_PATH/id_rsa $REMOTE_USER@$REMOTE_WORKER" |
1233 | +SCP="scp -i $SSH_KEY_PATH/id_rsa" |
1234 | + |
1235 | +RESULTS_ID=$(md5sum $(find $WORKSPACE/results/*.snap | tail -n1) | cut -d' ' -f 1) |
1236 | +REMOTE_RESULTS_BASE_DIR=/home/$REMOTE_USER/results |
1237 | + |
1238 | +$SSH mkdir -p $REMOTE_RESULTS_BASE_DIR/$RESULTS_ID |
1239 | +$SCP $WORKSPACE/results/*.snap $REMOTE_USER@$REMOTE_WORKER:$REMOTE_RESULTS_BASE_DIR/$RESULTS_ID/ |
1240 | + |
1241 | +# Save the id of our results so it can be used by a subsequent build |
1242 | +# in a properties file which is then being read from jenkins and its |
1243 | +# content passed as parameters to triggered builds. |
1244 | +echo "RESULTS_ID=$RESULTS_ID" >> $WORKSPACE/build-props |
1245 | +cat $WORKSPACE/build-props |
1246 | diff --git a/jobs/snap/snap-build-worker.yaml b/jobs/snap/snap-build-worker.yaml |
1247 | new file mode 100644 |
1248 | index 0000000..6b45a89 |
1249 | --- /dev/null |
1250 | +++ b/jobs/snap/snap-build-worker.yaml |
1251 | @@ -0,0 +1,72 @@ |
1252 | +- job-template: |
1253 | + name: '{name}-snap-build-worker' |
1254 | + project-type: freestyle |
1255 | + defaults: global |
1256 | + description: "Build a snap on launchpad" |
1257 | + display-name: "{name}-snap-build-worker" |
1258 | + concurrent: true |
1259 | + node: snap && build |
1260 | + parameters: |
1261 | + - string: |
1262 | + name: ARCHITECTURE |
1263 | + default: amd64 |
1264 | + description: Architecture to build the snap for |
1265 | + - string: |
1266 | + name: TARGET_GIT_REPO |
1267 | + default: |
1268 | + description: "Target git repository" |
1269 | + - string: |
1270 | + name: TARGET_GIT_REPO_BRANCH |
1271 | + default: master |
1272 | + description: "Branch of the target git repository to build from" |
1273 | + - string: |
1274 | + name: SERIES |
1275 | + default: xenial |
1276 | + description: "Ubuntu archive series to build for" |
1277 | + - string: |
1278 | + name: FORCE |
1279 | + default: "0" |
1280 | + description: "Set to 1 to force the build" |
1281 | + - string: |
1282 | + name: SOURCE_GIT_REPO |
1283 | + default: |
1284 | + description: "Source git repository" |
1285 | + - string: |
1286 | + name: SOURCE_GIT_REPO_BRANCH |
1287 | + default: |
1288 | + description: "Branch of the source git repository to use" |
1289 | + - string: |
1290 | + name: MERGE_PROPOSAL |
1291 | + default: |
1292 | + description: "Link to the merge proposal this build relates to" |
1293 | + - string: |
1294 | + name: REVISION |
1295 | + default: |
1296 | + description: "Cleanup the whole workspace" |
1297 | + - string: |
1298 | + name: CLEANUP_WORKSPACE |
1299 | + default: "0" |
1300 | + description: "Cleanup the whole workspace" |
1301 | + - string: |
1302 | + name: REMOTE_WORKER |
1303 | + default: "{obj:remote_worker}" |
1304 | + description: "The remote server to execute the spread jobs on. There's no need to change from the default value unless you know what you're doing." |
1305 | + - string: |
1306 | + name: REMOTE_USER |
1307 | + default: "{obj:remote_user}" |
1308 | + description: "The remote server username used to ssh to $REMOTE_WORKER." |
1309 | + - string: |
1310 | + name: CORE_CHANNEL |
1311 | + default: stable |
1312 | + description: "Channel of the core snap to use for testing the build snap" |
1313 | + - string: |
1314 | + name: RESULTS_ID |
1315 | + default: '' |
1316 | + description: "Alphanumeric identifier used to pass build artifacts through different jobs" |
1317 | + builders: |
1318 | + - shell: |
1319 | + !include-raw: |
1320 | + - common-job-prepare.sh |
1321 | + - shell: |
1322 | + !include-raw-escape: |
1323 | + - snap-build-worker.sh |
1324 | diff --git a/jobs/snap/snap-build.yaml b/jobs/snap/snap-build.yaml |
1325 | new file mode 100644 |
1326 | index 0000000..dec3b19 |
1327 | --- /dev/null |
1328 | +++ b/jobs/snap/snap-build.yaml |
1329 | @@ -0,0 +1,91 @@ |
1330 | +- job-template: |
1331 | + name: '{name}-snap-build' |
1332 | + project-type: matrix |
1333 | + defaults: global |
1334 | + description: "Build a snap with subsequent test execution" |
1335 | + display-name: "{name}-snap-build" |
1336 | + concurrent: true |
1337 | + sequential: false |
1338 | + node: monitor |
1339 | + axes: |
1340 | + - axis: |
1341 | + type: user-defined |
1342 | + name: ARCHITECTURE |
1343 | + values: '{obj:build_architectures}' |
1344 | + parameters: |
1345 | + - string: |
1346 | + name: TARGET_GIT_REPO |
1347 | + default: |
1348 | + description: "Target git repository" |
1349 | + - string: |
1350 | + name: TARGET_GIT_REPO_BRANCH |
1351 | + default: master |
1352 | + description: "Branch of the target git repository to build from" |
1353 | + - string: |
1354 | + name: SERIES |
1355 | + default: xenial |
1356 | + description: "Ubuntu archive series to build for" |
1357 | + - string: |
1358 | + name: FORCE |
1359 | + default: "0" |
1360 | + description: "Set to 1 to force the build" |
1361 | + - string: |
1362 | + name: SOURCE_GIT_REPO |
1363 | + default: |
1364 | + description: "Source git repository" |
1365 | + - string: |
1366 | + name: SOURCE_GIT_REPO_BRANCH |
1367 | + default: |
1368 | + description: "Branch of the source git repository to use" |
1369 | + - string: |
1370 | + name: MERGE_PROPOSAL |
1371 | + default: |
1372 | + description: "Link to the merge proposal this build relates to" |
1373 | + - string: |
1374 | + name: REVISION |
1375 | + default: |
1376 | + description: "Cleanup the whole workspace" |
1377 | + - string: |
1378 | + name: CLEANUP_WORKSPACE |
1379 | + default: "0" |
1380 | + description: "Cleanup the whole workspace" |
1381 | + builders: |
1382 | + - trigger-builds: |
1383 | + - project: '{name}-snap-build-worker' |
1384 | + current-parameters: true |
1385 | + predefined-parameters: | |
1386 | + RESULTS_ID=$BUILD_TAG |
1387 | + ARCHITECTURE=$ARCHITECTURE |
1388 | + block: true |
1389 | + - project: '{name}-snap-test' |
1390 | + current-parameters: true |
1391 | + predefined-parameters: | |
1392 | + RESULTS_ID=$BUILD_TAG |
1393 | + block: true |
1394 | + - project: '{name}-snap-cleanup' |
1395 | + current-parameters: true |
1396 | + predefined-parameters: | |
1397 | + RESULTS_ID=$BUILD_TAG |
1398 | + publishers: |
1399 | + - archive: |
1400 | + artifacts: '**/*.snap' |
1401 | + latest-only: false |
1402 | + allow-empty: true |
1403 | + fingerprint: false |
1404 | + - trigger-parameterized-builds: |
1405 | + - project: '{name}-snap-update-mp' |
1406 | + condition: "SUCCESS" |
1407 | + predefined-parameters: | |
1408 | + CI_RESULT=PASSED |
1409 | + CI_BUILD=${{BUILD_URL}} |
1410 | + CI_BRANCH="${{SOURCE_GIT_REPO_BRANCH}}@${{SOURCE_GIT_REPO}}" |
1411 | + CI_MERGE_PROPOSAL=${{MERGE_PROPOSAL}} |
1412 | + CI_REVISION=${{REVISION}} |
1413 | + - project: '{name}-snap-update-mp' |
1414 | + condition: "UNSTABLE_OR_WORSE" |
1415 | + predefined-parameters: | |
1416 | + CI_RESULT=FAILED |
1417 | + CI_BUILD=${{BUILD_URL}} |
1418 | + CI_BRANCH="${{SOURCE_GIT_REPO_BRANCH}}@${{SOURCE_GIT_REPO}}" |
1419 | + CI_MERGE_PROPOSAL=${{MERGE_PROPOSAL}} |
1420 | + CI_REVISION=${{REVISION}} |
1421 | diff --git a/jobs/snap/snap-cleanup.sh b/jobs/snap/snap-cleanup.sh |
1422 | new file mode 100644 |
1423 | index 0000000..bf0e03c |
1424 | --- /dev/null |
1425 | +++ b/jobs/snap/snap-cleanup.sh |
1426 | @@ -0,0 +1,38 @@ |
1427 | +#!/bin/sh |
1428 | +# |
1429 | +# Copyright (C) 2017 Canonical Ltd |
1430 | +# |
1431 | +# This program is free software: you can redistribute it and/or modify |
1432 | +# it under the terms of the GNU General Public License version 3 as |
1433 | +# published by the Free Software Foundation. |
1434 | +# |
1435 | +# This program is distributed in the hope that it will be useful, |
1436 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
1437 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
1438 | +# GNU General Public License for more details. |
1439 | +# |
1440 | +# You should have received a copy of the GNU General Public License |
1441 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
1442 | + |
1443 | +set -x |
1444 | + |
1445 | +. "$WORKSPACE/.build_env" |
1446 | + |
1447 | +# Delete auxiliary repo used in the build |
1448 | +$BUILD_SCRIPTS/tools/delete-ci-repo.py \ |
1449 | + --git-repo=https://git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$CI_REPO |
1450 | + |
1451 | +if [ -z "$REMOTE_WORKER" ]; then |
1452 | + echo "INFO: No remote system defined" |
1453 | + exit 0 |
1454 | +fi |
1455 | + |
1456 | +SSH_PATH="${JENKINS_HOME}/.ssh/" |
1457 | +SSH_KEY_PATH="${SSH_PATH}/git.launchpad.net/$BOT_USERNAME" |
1458 | +SSH="ssh -i $SSH_KEY_PATH/id_rsa $REMOTE_USER@$REMOTE_WORKER" |
1459 | +REMOTE_RESULTS_BASE_DIR=/home/$REMOTE_USER/results |
1460 | + |
1461 | +$SSH rm -rf $REMOTE_RESULTS_BASE_DIR/$RESULTS_ID |
1462 | + |
1463 | +# Now remove any container that might have been left behind... |
1464 | +$SSH sudo docker rm \$\(sudo docker ps -q --filter=status=exited --filter=ancestor=snap-spread-tests\) || true |
1465 | diff --git a/jobs/snap/snap-cleanup.yaml b/jobs/snap/snap-cleanup.yaml |
1466 | new file mode 100644 |
1467 | index 0000000..8aa4a03 |
1468 | --- /dev/null |
1469 | +++ b/jobs/snap/snap-cleanup.yaml |
1470 | @@ -0,0 +1,32 @@ |
1471 | +- job-template: |
1472 | + name: '{name}-snap-cleanup' |
1473 | + project-type: freestyle |
1474 | + defaults: global |
1475 | + description: "Cleanup artifacts left over from a snap build" |
1476 | + display-name: "{name}-snap-cleanup" |
1477 | + concurrent: true |
1478 | + node: snap && build |
1479 | + parameters: |
1480 | + - string: |
1481 | + name: CI_REPO |
1482 | + default: "" |
1483 | + description: "Auxiliary repo for the build, that we will remove" |
1484 | + - string: |
1485 | + name: "RESULTS_ID" |
1486 | + default: "" |
1487 | + description: "Alphanumeric Id of the results being staged on the remote worker" |
1488 | + - string: |
1489 | + name: REMOTE_WORKER |
1490 | + default: "{obj:remote_worker}" |
1491 | + description: "The remote server to execute the spread jobs on. There's no need to change from the default value unless you know what you're doing." |
1492 | + - string: |
1493 | + name: REMOTE_USER |
1494 | + default: "{obj:remote_user}" |
1495 | + description: "The remote server username used to ssh to $REMOTE_WORKER." |
1496 | + builders: |
1497 | + - shell: |
1498 | + !include-raw: |
1499 | + - common-job-prepare.sh |
1500 | + - shell: |
1501 | + !include-raw-escape: |
1502 | + - snap-cleanup.sh |
1503 | diff --git a/jobs/snap/snap-nightly.yaml b/jobs/snap/snap-nightly.yaml |
1504 | new file mode 100644 |
1505 | index 0000000..ae5a71f |
1506 | --- /dev/null |
1507 | +++ b/jobs/snap/snap-nightly.yaml |
1508 | @@ -0,0 +1,41 @@ |
1509 | +- job-template: |
1510 | + name: '{name}-snap-nightly' |
1511 | + project-type: matrix |
1512 | + concurrent: false |
1513 | + node: monitor |
1514 | + sequential: true |
1515 | + properties: |
1516 | + - build-discarder: |
1517 | + days-to-keep: 30 |
1518 | + - rebuild: |
1519 | + rebuild-disabled: true |
1520 | + axes: |
1521 | + - axis: |
1522 | + type: user-defined |
1523 | + name: CORE_CHANNEL |
1524 | + values: '{obj:nightly_core_channels}' |
1525 | + - axis: |
1526 | + type: user-defined |
1527 | + name: SNAP |
1528 | + values: '{obj:nightly_snaps}' |
1529 | + - axis: |
1530 | + type: user-defined |
1531 | + name: ARCHITECTURE |
1532 | + values: '{obj:nightly_architectures}' |
1533 | + wrappers: |
1534 | + - timeout: |
1535 | + timeout: 30 |
1536 | + abort: true |
1537 | + - timestamps |
1538 | + builders: |
1539 | + - trigger-builds: |
1540 | + - project: '{name}-snap-build-worker' |
1541 | + predefined-parameters: | |
1542 | + TARGET_GIT_REPO={base_snap_repo_url}/$SNAP |
1543 | + TARGET_GIT_REPO_BRANCH=master |
1544 | + SOURCE_GIT_REPO= |
1545 | + SOURCE_GIT_REPO_BRANCH= |
1546 | + REVISION= |
1547 | + ARCHITECTURES=$ARCHITECTURE |
1548 | + CORE_CHANNEL=$CORE_CHANNEL |
1549 | + block: true |
1550 | diff --git a/jobs/snap/snap-project-jobs.yaml b/jobs/snap/snap-project-jobs.yaml |
1551 | new file mode 100644 |
1552 | index 0000000..b693af3 |
1553 | --- /dev/null |
1554 | +++ b/jobs/snap/snap-project-jobs.yaml |
1555 | @@ -0,0 +1,13 @@ |
1556 | +- job-group: |
1557 | + name: snap-project-jobs |
1558 | + jobs: |
1559 | + - '{name}-snap-nightly' |
1560 | + - '{name}-snap-build-worker' |
1561 | + - '{name}-snap-build-update-chroot' |
1562 | + - '{name}-snap-build' |
1563 | + - '{name}-snap-cleanup' |
1564 | + - '{name}-snap-release' |
1565 | + - '{name}-snap-test' |
1566 | + - '{name}-snap-trigger-ci' |
1567 | + - '{name}-snap-update-mp' |
1568 | + - '{name}-snap-automerger' |
1569 | diff --git a/jobs/snap/snap-release.sh b/jobs/snap/snap-release.sh |
1570 | new file mode 100644 |
1571 | index 0000000..06fc6c4 |
1572 | --- /dev/null |
1573 | +++ b/jobs/snap/snap-release.sh |
1574 | @@ -0,0 +1,170 @@ |
1575 | +#!/bin/bash |
1576 | +# |
1577 | +# Copyright (C) 2017 Canonical Ltd |
1578 | +# |
1579 | +# This program is free software: you can redistribute it and/or modify |
1580 | +# it under the terms of the GNU General Public License version 3 as |
1581 | +# published by the Free Software Foundation. |
1582 | +# |
1583 | +# This program is distributed in the hope that it will be useful, |
1584 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
1585 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
1586 | +# GNU General Public License for more details. |
1587 | +# |
1588 | +# You should have received a copy of the GNU General Public License |
1589 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
1590 | + |
1591 | +set -ex |
1592 | + |
1593 | +. "$WORKSPACE/.build_env" |
1594 | + |
1595 | +if [ -z "$VERSION" ]; then |
1596 | + echo "ERROR: No version specified" |
1597 | + exit 1 |
1598 | +fi |
1599 | + |
1600 | +echo "Snap to be released: $SNAP_NAME" |
1601 | +echo "Version to be released: $VERSION" |
1602 | +echo "New development version: $NEXT_VERSION" |
1603 | + |
1604 | +REPOSITORY_URL="git+ssh://$BOT_USERNAME@git.launchpad.net/~$LAUNCHPAD_TEAM/$LAUNCHPAD_PROJECT/+git/$SNAP_NAME" |
1605 | + |
1606 | +if [ -e "$SNAP_NAME" ]; then |
1607 | + rm -rf "$SNAP_NAME" |
1608 | +fi |
1609 | + |
1610 | +set_git_identity() { |
1611 | + git config user.name "System Enablement CI Bot" |
1612 | + git config user.email "ce-system-enablement@lists.canonical.com" |
1613 | +} |
1614 | + |
1615 | +# Arguments are: |
1616 | +# $1 Snap name |
1617 | +# $2 Version to release |
1618 | +update_changelog() { |
1619 | + local latest_version commits i text range |
1620 | + local snap=$1 |
1621 | + local ver=$2 |
1622 | + local changelog_file=ChangeLog |
1623 | + local changes author full_text |
1624 | + |
1625 | + # latest tag is latest version |
1626 | + latest_version="$(git describe --abbrev=0)" || true |
1627 | + if [ -n "$latest_version" ]; then |
1628 | + range=$latest_version..HEAD |
1629 | + else |
1630 | + range=HEAD |
1631 | + fi |
1632 | + |
1633 | + commits=$(git rev-list --merges --reverse "$range") |
1634 | + declare -A changes |
1635 | + |
1636 | + for i in $commits; do |
1637 | + local body merge_proposal description text |
1638 | + |
1639 | + body=$(git log --format=%B -n1 "$i") |
1640 | + merge_proposal=$(echo "$body" | grep "^Merge-Proposal:") || true |
1641 | + author=$(echo "$body" | grep "^Author:") || true |
1642 | + author="${author#Author: *}" |
1643 | + if [ -z "$author" ]; then |
1644 | + if [ -z "$merge_proposal" ]; then |
1645 | + author="unknown" |
1646 | + else |
1647 | + author=${merge_proposal#*~} |
1648 | + author=${author%%/*} |
1649 | + fi |
1650 | + fi |
1651 | + # 'sed' removes leading blank lines first, then adds indentation |
1652 | + description=$(echo "$body" | grep -v "^Author:\|^Merge" | \ |
1653 | + sed '/./,$!d' | sed '2,$s/^/ /') || true |
1654 | + if [ -z "$description" ]; then |
1655 | + description="See more information in merge proposal" |
1656 | + fi |
1657 | + text=${changes[$author]} |
1658 | + printf -v text "%s\n * %s\n %s" \ |
1659 | + "$text" "$description" "$merge_proposal" |
1660 | + changes[$author]=$text |
1661 | + done |
1662 | + |
1663 | + printf -v full_text "%s\n" "$(date --rfc-3339=date --utc) $snap $ver" |
1664 | + for author in "${!changes[@]}"; do |
1665 | + printf -v full_text "%s\n [ %s ]%s\n" \ |
1666 | + "$full_text" "$author" "${changes[$author]}" |
1667 | + done |
1668 | + |
1669 | + if [ ! -e "$changelog_file" ]; then |
1670 | + touch "$changelog_file" |
1671 | + fi |
1672 | + echo "$full_text" | cat - "$changelog_file" > "$changelog_file".tmp |
1673 | + mv "$changelog_file".tmp "$changelog_file" |
1674 | + |
1675 | + git add "$changelog_file" |
1676 | + git commit -m "Update $changelog_file for $ver" |
1677 | +} |
1678 | + |
1679 | +# Arguments are: |
1680 | +# $1 Version to be set in the snapcraft.yaml file |
1681 | +# $2 Path to the snapcraft.yaml file |
1682 | +bump_version_and_tag() { |
1683 | + sed -i -e "s/^version:\ .*/version: $1/g" "$2" |
1684 | + git add "$2" |
1685 | + git commit -m "Bump version to $1" |
1686 | + git tag -a -m "$1" "$1" HEAD |
1687 | +} |
1688 | + |
1689 | +RELEASE_BASE_BRANCH=master |
1690 | +if [ "$RELEASE_FROM_STABLE" -eq 1 ]; then |
1691 | + RELEASE_BASE_BRANCH=stable |
1692 | +fi |
1693 | + |
1694 | +git clone -b "$RELEASE_BASE_BRANCH" "$REPOSITORY_URL" "$SNAP_NAME" |
1695 | +cd "$SNAP_NAME" |
1696 | + |
1697 | +SNAPCRAFT_YAML_PATH= |
1698 | +if [ -e snapcraft.yaml ]; then |
1699 | + SNAPCRAFT_YAML_PATH=snapcraft.yaml |
1700 | +elif [ -e snap/snapcraft.yaml ]; then |
1701 | + SNAPCRAFT_YAML_PATH=snap/snapcraft.yaml |
1702 | +fi |
1703 | + |
1704 | +if [ -z "$SNAPCRAFT_YAML_PATH" ]; then |
1705 | + echo "ERROR: No snapcraft.yaml or snap/snapcraft.yaml file!" |
1706 | + exit 1 |
1707 | +fi |
1708 | + |
1709 | +set_git_identity |
1710 | +update_changelog "$SNAP_NAME" "$VERSION" |
1711 | +bump_version_and_tag "$VERSION" "$SNAPCRAFT_YAML_PATH" |
1712 | + |
1713 | +if [ "$RELEASE_FROM_STABLE" -eq 1 ]; then |
1714 | + git push origin "$RELEASE_BASE_BRANCH" |
1715 | + git push origin "$VERSION" |
1716 | + |
1717 | + "$BUILD_SCRIPTS"/tools/trigger-lp-build.py -s "$SNAP_NAME" -p |
1718 | +else |
1719 | + if ! git branch -r | grep origin/stable ; then |
1720 | + git checkout -b stable origin/master |
1721 | + else |
1722 | + git checkout -b stable origin/stable |
1723 | + fi |
1724 | + |
1725 | + # We're using `-X theirs` here as master always takes priority over |
1726 | + # what is in the stable. If something was only submitted into stable |
1727 | + # the commiter needs to take care that the same change is submitted |
1728 | + # to master too or it is overriden the next time a release happens |
1729 | + # from master. |
1730 | + git merge --no-ff -X theirs "$RELEASE_BASE_BRANCH" |
1731 | + |
1732 | + git push origin stable |
1733 | + git push origin "$RELEASE_BASE_BRANCH" |
1734 | + git push origin "$VERSION" |
1735 | + |
1736 | + # Build before we change master branch |
1737 | + "$BUILD_SCRIPTS"/tools/trigger-lp-build.py -s "$SNAP_NAME" -p |
1738 | + |
1739 | + git checkout "$RELEASE_BASE_BRANCH" |
1740 | + sed -i -e "s/^version:\ .*/version: ${NEXT_VERSION}-dev/g" "$SNAPCRAFT_YAML_PATH" |
1741 | + git add "$SNAPCRAFT_YAML_PATH" |
1742 | + git commit -m "Open development for ${NEXT_VERSION}-dev" |
1743 | + git push origin "$RELEASE_BASE_BRANCH" |
1744 | +fi |
1745 | diff --git a/jobs/snap/snap-release.yaml b/jobs/snap/snap-release.yaml |
1746 | new file mode 100644 |
1747 | index 0000000..75e3b3a |
1748 | --- /dev/null |
1749 | +++ b/jobs/snap/snap-release.yaml |
1750 | @@ -0,0 +1,58 @@ |
1751 | +- job-template: |
1752 | + name: '{name}-snap-release' |
1753 | + project-type: freestyle |
1754 | + defaults: global |
1755 | + description: "A job implementing the release process used for snaps" |
1756 | + display-name: "{name}-snap-release" |
1757 | + concurrent: true |
1758 | + node: snap && release |
1759 | + parameters: |
1760 | + - string: |
1761 | + name: SNAP_NAME |
1762 | + default: "" |
1763 | + description: | |
1764 | + Name of the snap which should be released |
1765 | + |
1766 | + Normally the repositories we have on https://code.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/ |
1767 | + match with the snap name. In some cases this is not true, in those you have to set the repository |
1768 | + name here. For example for 'canonical-se-engineering-tests' it is 'engineering-tests' as this is |
1769 | + the repository name. |
1770 | + - string: |
1771 | + name: VERSION |
1772 | + default: "" |
1773 | + description: "New version of the snap" |
1774 | + - string: |
1775 | + name: NEXT_VERSION |
1776 | + default: "" |
1777 | + description: | |
1778 | + Version which will follow the version specified in the VERSION field. |
1779 | + For example if you specify VERSION = "1.1" next version is most likely |
1780 | + "1.2". The NEXT_VERSION parameter is used to write it into the component |
1781 | + snapcraft.yaml as "$NEXT_VERSION-dev" to clearly indicate that snaps |
1782 | + build from master are development versions. |
1783 | + |
1784 | + Please note that NEXT_VERSION is not set in stone and can be overriden |
1785 | + at any time by changes merged into master. |
1786 | + - string: |
1787 | + name: CLEANUP_WORKSPACE |
1788 | + default: "0" |
1789 | + description: "Cleanup the whole workspace" |
1790 | + - string: |
1791 | + name: SERIES |
1792 | + default: xenial |
1793 | + description: "Ubuntu archive series to build for" |
1794 | + - string: |
1795 | + name: RELEASE_FROM_STABLE |
1796 | + default: 0 |
1797 | + description: | |
1798 | + Set to '1' to force a release from stable branch without merging with |
1799 | + master. This can be used when single changes are picked into stable |
1800 | + manually and need to be released without pulling anything else from |
1801 | + master. PLEASE ENSURE THAT THOSE CHANGE GO INTO MASTER TOO!! |
1802 | + builders: |
1803 | + - shell: |
1804 | + !include-raw: |
1805 | + - common-job-prepare.sh |
1806 | + - shell: |
1807 | + !include-raw-escape: |
1808 | + - snap-release.sh |
1809 | diff --git a/jobs/snap/snap-test.sh b/jobs/snap/snap-test.sh |
1810 | new file mode 100644 |
1811 | index 0000000..bea5e3a |
1812 | --- /dev/null |
1813 | +++ b/jobs/snap/snap-test.sh |
1814 | @@ -0,0 +1,121 @@ |
1815 | +#!/bin/sh |
1816 | +# |
1817 | +# Copyright (C) 2016 Canonical Ltd |
1818 | +# |
1819 | +# This program is free software: you can redistribute it and/or modify |
1820 | +# it under the terms of the GNU General Public License version 3 as |
1821 | +# published by the Free Software Foundation. |
1822 | +# |
1823 | +# This program is distributed in the hope that it will be useful, |
1824 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
1825 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
1826 | +# GNU General Public License for more details. |
1827 | +# |
1828 | +# You should have received a copy of the GNU General Public License |
1829 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
1830 | + |
1831 | +set -ex |
1832 | + |
1833 | +. "$WORKSPACE/.build_env" |
1834 | + |
1835 | +if [ "$RUN_TESTS" = False ]; then |
1836 | + echo "WARNING: test execution is disabled" |
1837 | + exit 0 |
1838 | +fi |
1839 | + |
1840 | +SSH_PATH="${JENKINS_HOME}/.ssh/" |
1841 | +SSH_KEY_PATH="${SSH_PATH}/git.launchpad.net/system-enablement-ci-bot" |
1842 | + |
1843 | +SSH="ssh -i $SSH_KEY_PATH/id_rsa $REMOTE_USER@$REMOTE_WORKER" |
1844 | +SCP="scp -i $SSH_KEY_PATH/id_rsa" |
1845 | + |
1846 | +REMOTE_WORKSPACE=/home/$REMOTE_USER/$BUILD_TAG |
1847 | +REMOTE_RESULTS_BASE_DIR=/home/$REMOTE_USER/results |
1848 | + |
1849 | +tmp_srcdir=`mktemp -d` |
1850 | +git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO $tmp_srcdir/src |
1851 | +cd $tmp_srcdir/src |
1852 | +# This will fail as we have set set -e above when the revision isn't part of |
1853 | +# of the repository we've cloned. |
1854 | +git branch --contains $SOURCE_GIT_REPO_REVISION | grep "$SOURCE_GIT_REPO_BRANCH" |
1855 | +git checkout -b ci-test $SOURCE_GIT_REPO_REVISION |
1856 | + |
1857 | +# Components have the ability to disable CI tests if they can't provide any. |
1858 | +# This only accepted in a few cases and should be generally avoided. |
1859 | +if [ -e $tmp_srcdir/src/.ci_tests_disabled ]; then |
1860 | + echo "WARNING: Component has no CI tests so not running anything here" |
1861 | + exit 0 |
1862 | +fi |
1863 | +# We require either a run-tests.sh interface script for the spread tests |
1864 | +# or the basic spread.yaml spread test definition file, otherwise we fail |
1865 | +# the Jenkins job. Spread tests are required for all MRs. |
1866 | +if [ ! -e "$tmp_srcdir/src/run-tests.sh" ] && [ ! -e "$tmp_srcdir/src/spread.yaml" ]; then |
1867 | + echo "ERROR: missing spread test: you must provide a spread test" |
1868 | + exit 1 |
1869 | +fi |
1870 | +rm -rf $tmp_srcdir |
1871 | + |
1872 | +if [ -n "$RESULTS_ID" ]; then |
1873 | + $SSH mkdir -p $REMOTE_WORKSPACE/results |
1874 | + $SSH cp -v $REMOTE_RESULTS_BASE_DIR/$RESULTS_ID/*.snap $REMOTE_WORKSPACE/results |
1875 | +fi |
1876 | + |
1877 | +$SSH sudo apt-get --yes --force-yes install docker.io |
1878 | + |
1879 | +cat << EOF > $WORKSPACE/run-tests.sh |
1880 | +#!/bin/sh |
1881 | +set -ex |
1882 | + |
1883 | +export TERM=linux |
1884 | +export DEBIAN_FRONTEND=noninteractive |
1885 | +export PATH=/build/bin:$PATH |
1886 | + |
1887 | +# At this time it's necessary to build spread manually because |
1888 | +# the snapped version does not include the qemu/kvm backend. |
1889 | +# Once the snapped version includes this backend, then we can |
1890 | +# change the manual building of spread with making sure the snap |
1891 | +# package is installed. |
1892 | +export GOPATH=`mktemp -d` |
1893 | +go get -d -v github.com/snapcore/spread/... |
1894 | +go build github.com/snapcore/spread/cmd/spread |
1895 | +mkdir /build/bin |
1896 | +cp spread /build/bin |
1897 | + |
1898 | +git clone --depth 1 -b $SOURCE_GIT_REPO_BRANCH $SOURCE_GIT_REPO /build/src |
1899 | +cd /build/src |
1900 | +git checkout -b ci-tests $SOURCE_GIT_REPO_REVISION |
1901 | + |
1902 | +# Copy any stage results from previous generic-build-snap-worker builds |
1903 | +cp -v /build/results/*.snap /build/src |
1904 | + |
1905 | +if [ -e "run-tests.sh" ] ; then |
1906 | + if [ ! -z "$CHANNEL" ] ; then |
1907 | + ./run-tests.sh --channel=$CHANNEL --test-from-channel --debug --force-new-image |
1908 | + else |
1909 | + ./run-tests.sh --debug --force-new-image |
1910 | + fi |
1911 | +else |
1912 | + if [ ! -z "$CHANNEL" ] ; then |
1913 | + SNAP_CHANNEL=$CHANNEL spread -debug |
1914 | + else |
1915 | + spread -debug |
1916 | + fi |
1917 | +fi |
1918 | +EOF |
1919 | + |
1920 | +$SSH mkdir -p $REMOTE_WORKSPACE |
1921 | +$SCP $WORKSPACE/run-tests.sh $REMOTE_USER@$REMOTE_WORKER:$REMOTE_WORKSPACE |
1922 | +$SSH chmod u+x $REMOTE_WORKSPACE/run-tests.sh |
1923 | + |
1924 | +$SSH mkdir -p $REMOTE_WORKSPACE/docker |
1925 | +$SCP $WORKSPACE/build-scripts/docker/spread-tests/Dockerfile \ |
1926 | + $REMOTE_USER@$REMOTE_WORKER:$REMOTE_WORKSPACE/docker |
1927 | +$SSH time sudo docker build -t snap-spread-tests $REMOTE_WORKSPACE/docker |
1928 | + |
1929 | +$SSH time sudo docker run \ |
1930 | + -v /dev:/dev \ |
1931 | + -v $REMOTE_WORKSPACE:/build \ |
1932 | + --privileged \ |
1933 | + snap-spread-tests /build/run-tests.sh |
1934 | + |
1935 | +$SSH sudo rm -rf $REMOTE_WORKSPACE |
1936 | diff --git a/jobs/snap/snap-test.yaml b/jobs/snap/snap-test.yaml |
1937 | new file mode 100644 |
1938 | index 0000000..9c15699 |
1939 | --- /dev/null |
1940 | +++ b/jobs/snap/snap-test.yaml |
1941 | @@ -0,0 +1,64 @@ |
1942 | +- job-template: |
1943 | + name: '{name}-snap-test' |
1944 | + project-type: freestyle |
1945 | + defaults: global |
1946 | + description: "Run tests for a single snap on a remote agent which also allows spread execution inside KVM/QEMU" |
1947 | + display-name: "{name}-snap-test" |
1948 | + concurrent: true |
1949 | + node: snap && test |
1950 | + parameters: |
1951 | + - string: |
1952 | + name: SOURCE_GIT_REPO |
1953 | + default: "" |
1954 | + description: "Source git repository" |
1955 | + - string: |
1956 | + name: SOURCE_GIT_REPO_BRANCH |
1957 | + default: "" |
1958 | + description: "Branch of the source git repository to use" |
1959 | + - string: |
1960 | + name: CHANNEL |
1961 | + default: "" |
1962 | + description: "Run tests against an image build with a core snap from the specified channel" |
1963 | + - string: |
1964 | + name: REMOTE_WORKER |
1965 | + default: "{obj:remote_worker}" |
1966 | + description: "The remote server to execute the spread jobs on. There's no need to change from the default value unless you know what you're doing." |
1967 | + - string: |
1968 | + name: REMOTE_USER |
1969 | + default: "{obj:remote_user}" |
1970 | + description: "The remote server username used to ssh to $REMOTE_WORKER." |
1971 | + - string: |
1972 | + name: CLEANUP_WORKSPACE |
1973 | + default: "0" |
1974 | + description: "Cleanup the whole workspace" |
1975 | + - string: |
1976 | + name: SERIES |
1977 | + default: xenial |
1978 | + description: "Ubuntu archive series to build for" |
1979 | + - string: |
1980 | + name: REBUILD_ROOTFS |
1981 | + default: 0 |
1982 | + description: "Rebuild the chroot rootfs or not. Default is to not rebuild and use a previous job's rootfs." |
1983 | + - string: |
1984 | + name: RESULTS_ID |
1985 | + default: "" |
1986 | + description: "Alphanumeric Id of the results being staged on the remote worker" |
1987 | + - string: |
1988 | + name: CORE_CHANNEL |
1989 | + default: "stable" |
1990 | + description: "Channel used for the core snap inside the test environment. Defaults to 'stable'." |
1991 | + - string: |
1992 | + name: CI_BRANCH |
1993 | + default: "" |
1994 | + description: "Branch on which the tests should be executed" |
1995 | + - string: |
1996 | + name: CI_REPO |
1997 | + default: "" |
1998 | + description: "Git repository to use for testing (MUST contain $CI_BRANCH)" |
1999 | + builders: |
2000 | + - shell: |
2001 | + !include-raw: |
2002 | + - common-job-prepare.sh |
2003 | + - shell: |
2004 | + !include-raw-escape: |
2005 | + - snap-test.sh |
2006 | diff --git a/jobs/snap/snap-trigger-ci.sh b/jobs/snap/snap-trigger-ci.sh |
2007 | new file mode 100644 |
2008 | index 0000000..b86724d |
2009 | --- /dev/null |
2010 | +++ b/jobs/snap/snap-trigger-ci.sh |
2011 | @@ -0,0 +1,29 @@ |
2012 | +#!/bin/bash |
2013 | +# |
2014 | +# Copyright (C) 2016 Canonical Ltd |
2015 | +# |
2016 | +# This program is free software: you can redistribute it and/or modify |
2017 | +# it under the terms of the GNU General Public License version 3 as |
2018 | +# published by the Free Software Foundation. |
2019 | +# |
2020 | +# This program is distributed in the hope that it will be useful, |
2021 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
2022 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
2023 | +# GNU General Public License for more details. |
2024 | +# |
2025 | +# You should have received a copy of the GNU General Public License |
2026 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
2027 | + |
2028 | +set -ex |
2029 | + |
2030 | +. "$WORKSPACE/.build_env" |
2031 | + |
2032 | +if [ "$TRIGGER_CI" = False ]; then |
2033 | + echo "WARNING: CI is disabled" |
2034 | + exit 0 |
2035 | +fi |
2036 | + |
2037 | +exec "$BUILD_SCRIPTS"/tools/trigger-ci.py \ |
2038 | + -p "$LAUNCHPAD_PROJECT" \ |
2039 | + -j "$SNAP_BUILD_JOB" \ |
2040 | + -t "$LAUNCHPAD_TEAM" |
2041 | diff --git a/jobs/snap/snap-trigger-ci.yaml b/jobs/snap/snap-trigger-ci.yaml |
2042 | new file mode 100644 |
2043 | index 0000000..e412325 |
2044 | --- /dev/null |
2045 | +++ b/jobs/snap/snap-trigger-ci.yaml |
2046 | @@ -0,0 +1,22 @@ |
2047 | +- job-template: |
2048 | + name: '{name}-snap-trigger-ci' |
2049 | + project-type: freestyle |
2050 | + defaults: global |
2051 | + description: "Monitor Launchpad for new merge proposals" |
2052 | + display-name: "{name}-snap-trigger-ci" |
2053 | + concurrent: true |
2054 | + node: snap && misc |
2055 | + triggers: |
2056 | + - timed: # every five minutes |
2057 | + H/5 * * * * |
2058 | + properties: |
2059 | + - build-discarder: |
2060 | + num-to-keep: 10 |
2061 | + - rebuild |
2062 | + builders: |
2063 | + - shell: |
2064 | + !include-raw: |
2065 | + - common-job-prepare.sh |
2066 | + - shell: |
2067 | + !include-raw-escape: |
2068 | + - snap-trigger-ci.sh |
2069 | diff --git a/jobs/snap/snap-update-mp.sh b/jobs/snap/snap-update-mp.sh |
2070 | new file mode 100644 |
2071 | index 0000000..fd05d6e |
2072 | --- /dev/null |
2073 | +++ b/jobs/snap/snap-update-mp.sh |
2074 | @@ -0,0 +1,30 @@ |
2075 | +#!/bin/sh |
2076 | +# |
2077 | +# Copyright (C) 2016 Canonical Ltd |
2078 | +# |
2079 | +# This program is free software: you can redistribute it and/or modify |
2080 | +# it under the terms of the GNU General Public License version 3 as |
2081 | +# published by the Free Software Foundation. |
2082 | +# |
2083 | +# This program is distributed in the hope that it will be useful, |
2084 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
2085 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
2086 | +# GNU General Public License for more details. |
2087 | +# |
2088 | +# You should have received a copy of the GNU General Public License |
2089 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
2090 | + |
2091 | +set -ex |
2092 | + |
2093 | +. "$WORKSPACE/.build_env" |
2094 | + |
2095 | +if [ "$UPDATE_MPS" = False ]; then |
2096 | + echo "WARNING: MP updates are disabled" |
2097 | + exit 0 |
2098 | +fi |
2099 | + |
2100 | +exec $BUILD_SCRIPTS/tools/vote-on-merge-proposal.py \ |
2101 | + -s $CI_RESULT \ |
2102 | + -u $CI_BUILD \ |
2103 | + -r $CI_REVISION \ |
2104 | + -p $CI_MERGE_PROPOSAL |
2105 | diff --git a/jobs/snap/snap-update-mp.yaml b/jobs/snap/snap-update-mp.yaml |
2106 | new file mode 100644 |
2107 | index 0000000..967b891 |
2108 | --- /dev/null |
2109 | +++ b/jobs/snap/snap-update-mp.yaml |
2110 | @@ -0,0 +1,31 @@ |
2111 | +- job-template: |
2112 | + name: '{name}-snap-update-mp' |
2113 | + project-type: freestyle |
2114 | + defaults: global |
2115 | + description: "Update given merge-proposal with the result of the build" |
2116 | + display-name: "{name}-snap-update-mp" |
2117 | + concurrent: true |
2118 | + node: snap && misc |
2119 | + parameters: |
2120 | + - string: |
2121 | + name: CI_RESULT |
2122 | + description: Result of the CI build |
2123 | + - string: |
2124 | + name: CI_BUILD |
2125 | + description: Jenkins URL of the build |
2126 | + - string: |
2127 | + name: CI_BRANCH |
2128 | + description: Launchpad branch that was processed |
2129 | + - string: |
2130 | + name: CI_MERGE_PROPOSAL |
2131 | + description: Launchpad merge proposal that was processed |
2132 | + - string: |
2133 | + name: CI_REVISION |
2134 | + description: Revision of the processed branch |
2135 | + builders: |
2136 | + - shell: |
2137 | + !include-raw: |
2138 | + - common-job-prepare.sh |
2139 | + - shell: |
2140 | + !include-raw-escape: |
2141 | + - snap-update-mp.sh |
2142 | diff --git a/local.conf b/local.conf |
2143 | new file mode 100644 |
2144 | index 0000000..0170c70 |
2145 | --- /dev/null |
2146 | +++ b/local.conf |
2147 | @@ -0,0 +1,11 @@ |
2148 | +[job_builder] |
2149 | +ignore_cache=True |
2150 | +keep_descriptions=False |
2151 | +recursive=False |
2152 | +allow_duplicates=False |
2153 | + |
2154 | +[jenkins] |
2155 | +user=system-enablement-ci-bot |
2156 | +password=jenkins |
2157 | +url=http://localhost:8080 |
2158 | +query_plugins_info=False |
2159 | diff --git a/local.yaml b/local.yaml |
2160 | new file mode 100644 |
2161 | index 0000000..4de3a0c |
2162 | --- /dev/null |
2163 | +++ b/local.yaml |
2164 | @@ -0,0 +1,60 @@ |
2165 | +- project: |
2166 | + name: infrastructure |
2167 | + |
2168 | + jobs-git-repo: https://git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-jobs |
2169 | + jobs-git-repo-branch: master |
2170 | + config-git-repo: https://git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-config |
2171 | + config-git-repo-branch: master |
2172 | + |
2173 | + jenkins-instance: localhost |
2174 | + |
2175 | + bot_username: system-enablement-ci-bot |
2176 | + credentials_path: /var/lib/jenkins/.launchpad.credentials |
2177 | + allowed_users: "canonical-system-enablement" |
2178 | + backend_url: http://localhost:8080/ |
2179 | + blacklisted_jobs: "" |
2180 | + install_packages: "" |
2181 | + build_slaves: |
2182 | + - master |
2183 | + jobs: |
2184 | + - infrastructure-jobs |
2185 | + |
2186 | +- project: |
2187 | + name: system-enablement |
2188 | + |
2189 | + jobs-git-repo: https://git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-jobs |
2190 | + jobs-git-repo-branch: master |
2191 | + |
2192 | + bot_username: system-enablement-ci-bot |
2193 | + |
2194 | + allowed_users: "canonical-system-enablement" |
2195 | + launchpad_project: "snappy-hwe-snaps" |
2196 | + launchpad_team: "snappy-hwe-team" |
2197 | + |
2198 | + update_mps: false |
2199 | + run_tests: false |
2200 | + build_on_launchpad: false |
2201 | + trigger_ci: false |
2202 | + auto_merge: false |
2203 | + |
2204 | + build_architectures: [amd64] |
2205 | + |
2206 | + all_slaves: |
2207 | + - master |
2208 | + build_slaves: |
2209 | + - master |
2210 | + |
2211 | + blacklisted_jobs: "" |
2212 | + |
2213 | + remote_worker: "" |
2214 | + remote_user: "" |
2215 | + |
2216 | + nightly_architectures: [] |
2217 | + nightly_core_channels: [] |
2218 | + nightly_snaps: [] |
2219 | + |
2220 | + base_snap_repo_url: 'https://git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git' |
2221 | + |
2222 | + jobs: |
2223 | + - snap-project-jobs |
2224 | + - image-project-jobs |
2225 | diff --git a/run-tests.sh b/run-tests.sh |
2226 | index 3e02fc5..83cf437 100755 |
2227 | --- a/run-tests.sh |
2228 | +++ b/run-tests.sh |
2229 | @@ -1,4 +1,5 @@ |
2230 | #!/bin/sh |
2231 | +<<<<<<< run-tests.sh |
2232 | # |
2233 | # Copyright (C) 2016 Canonical Ltd |
2234 | # |
2235 | @@ -74,3 +75,6 @@ fi |
2236 | |
2237 | echo "INFO: Executing tests runner" |
2238 | cd $TESTS_EXTRAS_PATH && ./tests-runner.sh "$@" "$EXTRA_ARGS" |
2239 | +======= |
2240 | +echo "Nothing yet!" |
2241 | +>>>>>>> run-tests.sh |
2242 | diff --git a/tools/automerge-mps.py b/tools/automerge-mps.py |
2243 | new file mode 100755 |
2244 | index 0000000..8699dd8 |
2245 | --- /dev/null |
2246 | +++ b/tools/automerge-mps.py |
2247 | @@ -0,0 +1,152 @@ |
2248 | +#!/usr/bin/env python |
2249 | +# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*- |
2250 | +# |
2251 | +# Copyright (C) 2016 Canonical Ltd |
2252 | +# |
2253 | +# This program is free software: you can redistribute it and/or modify |
2254 | +# it under the terms of the GNU General Public License version 3 as |
2255 | +# published by the Free Software Foundation. |
2256 | +# |
2257 | +# This program is distributed in the hope that it will be useful, |
2258 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
2259 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
2260 | +# GNU General Public License for more details. |
2261 | +# |
2262 | +# You should have received a copy of the GNU General Public License |
2263 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
2264 | + |
2265 | +from launchpadlib.launchpad import Launchpad |
2266 | +import os |
2267 | +import sys |
2268 | +import yaml |
2269 | +import git |
2270 | +import shutil |
2271 | +import se_utils |
2272 | + |
2273 | +from argparse import ArgumentParser |
2274 | + |
2275 | +parser = ArgumentParser(description="Trigger snap builds for pending merge proposals") |
2276 | +parser.add_argument('-p', '--project', required=True, |
2277 | + help="Launchpad project to check for merge-proposals") |
2278 | + |
2279 | +args = vars(parser.parse_args()) |
2280 | + |
2281 | +class LaunchpadVote(): |
2282 | + APPROVE = 'Approve' |
2283 | + DISAPPROVE = 'Disapprove' |
2284 | + NEEDS_FIXING = 'Needs Fixing' |
2285 | + |
2286 | +def load_config(): |
2287 | + files = [os.path.expanduser('~/.jlp/jlp.config'), 'jlp.config'] |
2288 | + for config_file in files: |
2289 | + try: |
2290 | + config = yaml.safe_load(open(config_file, 'r')) |
2291 | + return config |
2292 | + except IOError: |
2293 | + pass |
2294 | + print("ERROR: No config file found") |
2295 | + sys.exit(1) |
2296 | + |
2297 | +def get_config_option(name): |
2298 | + config = load_config() |
2299 | + return config[name] |
2300 | + |
2301 | +def clean_branch_name(branch_name): |
2302 | + if branch_name.startswith("refs/heads/"): |
2303 | + return branch_name[11:] |
2304 | + return branch_name |
2305 | + |
2306 | +def correct_ssh_url(url): |
2307 | + if not url.startswith("git+ssh://"): |
2308 | + return url |
2309 | + new_url = "git+ssh://system-enablement-ci-bot@%s" % url[10:] |
2310 | + return new_url |
2311 | + |
2312 | + |
2313 | +def try_merge(proposal, target_repo, target_branch, source_repo, source_branch): |
2314 | + source_git_url=source_repo.git_https_url |
2315 | + # If we're operating with a private git repository we have to use the |
2316 | + # SSH url. Otherwise we try to avoid that to not cause any damage on |
2317 | + # the source repository. |
2318 | + if source_git_url == None: |
2319 | + source_git_url = source_repo.git_ssh_url |
2320 | + |
2321 | + print("Trying to merge %s:%s into %s:%s" % (source_git_url, source_branch, target_repo.git_ssh_url, target_branch)) |
2322 | + |
2323 | + repo_path = os.path.join(os.environ["WORKSPACE"], "repo") |
2324 | + if os.path.exists(repo_path): |
2325 | + shutil.rmtree(repo_path) |
2326 | + |
2327 | + repo = git.Repo.clone_from(correct_ssh_url(target_repo.git_ssh_url), repo_path, branch=target_branch) |
2328 | + source_remote = repo.create_remote("source", source_git_url) |
2329 | + source_remote.fetch() |
2330 | + |
2331 | + repo.git.config("user.name", "System Enablement CI Bot") |
2332 | + # FIXME: What is the real email address of the bot? |
2333 | + repo.git.config("user.email", "ce-system-enablement@lists.canonical.com") |
2334 | + |
2335 | + registrant_name = proposal.registrant.display_name |
2336 | + try: |
2337 | + registrant_mail = proposal.registrant.preferred_email_address.email |
2338 | + except Exception as e: |
2339 | + print("WARNING: cannot get e-mail for %s (%s)" % (registrant_name, e)) |
2340 | + registrant_mail="(unknown e-mail)" |
2341 | + |
2342 | + repo.git.merge("--no-ff", |
2343 | + "-m", "Merge remote tracking branch %s" % (source_branch), |
2344 | + "-m", "Merge-Proposal: %s" % proposal.web_link, |
2345 | + "-m", "Author: %s <%s>" % (registrant_name, registrant_mail), |
2346 | + "-m", "%s" % proposal.description, |
2347 | + "source/%s" % source_branch) |
2348 | + |
2349 | + repo.git.push("origin", target_branch) |
2350 | + |
2351 | +def get_last_mp_vote(mp): |
2352 | + for vote in mp.votes: |
2353 | + if not vote.comment: |
2354 | + continue |
2355 | + if vote.review_type == "continuous-integration" and vote.comment: |
2356 | + return vote.comment.vote |
2357 | + return None |
2358 | + |
2359 | +def mp_is_disapproved(mp): |
2360 | + for vote in mp.votes: |
2361 | + if vote.comment and vote.comment.vote == LaunchpadVote.DISAPPROVE: |
2362 | + return True |
2363 | + return False |
2364 | + |
2365 | +lp_app = get_config_option("lp_app") |
2366 | +lp_env = get_config_option("lp_env") |
2367 | +credential_store_path = get_config_option('credential_store_path') |
2368 | +launchpad = se_utils.get_launchpad(None, credential_store_path, lp_app, lp_env) |
2369 | + |
2370 | +project = launchpad.projects[args['project']] |
2371 | +proposals = project.getMergeProposals(status=['Approved']) |
2372 | + |
2373 | +failed_merges = 0 |
2374 | + |
2375 | +print("Found %d candidate merge proposals" % len(proposals)) |
2376 | + |
2377 | +for proposal in proposals: |
2378 | + if get_last_mp_vote(proposal) != LaunchpadVote.APPROVE: |
2379 | + print("Not merging %s as not approved by CI" % proposal.web_link) |
2380 | + continue |
2381 | + |
2382 | + if mp_is_disapproved(proposal): |
2383 | + print("Not merging %s as at least one reviewer has disapproved the change" % proposal.web_link) |
2384 | + continue |
2385 | + |
2386 | + print("Found proposal which is ready for merging: %s" % proposal.web_link) |
2387 | + |
2388 | + try: |
2389 | + try_merge(proposal, |
2390 | + launchpad.load(proposal.target_git_repository_link), |
2391 | + clean_branch_name(proposal.target_git_path), |
2392 | + launchpad.load(proposal.source_git_repository_link), |
2393 | + clean_branch_name(proposal.source_git_path)) |
2394 | + except Exception as e: |
2395 | + print("ERROR: Failed to merge %s (%s)" % (proposal.web_link, e)) |
2396 | + failed_merges += 1 |
2397 | + |
2398 | +if failed_merges > 0: |
2399 | + sys.exit(1) |
2400 | diff --git a/tools/build-rootfs-create b/tools/build-rootfs-create |
2401 | new file mode 100755 |
2402 | index 0000000..9a9f9ce |
2403 | --- /dev/null |
2404 | +++ b/tools/build-rootfs-create |
2405 | @@ -0,0 +1,26 @@ |
2406 | +#!/bin/bash |
2407 | +# |
2408 | +# Copyright (C) 2016 Canonical Ltd |
2409 | +# |
2410 | +# This program is free software: you can redistribute it and/or modify |
2411 | +# it under the terms of the GNU General Public License version 3 as |
2412 | +# published by the Free Software Foundation. |
2413 | +# |
2414 | +# This program is distributed in the hope that it will be useful, |
2415 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
2416 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
2417 | +# GNU General Public License for more details. |
2418 | +# |
2419 | +# You should have received a copy of the GNU General Public License |
2420 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
2421 | + |
2422 | +set -x |
2423 | +set -e |
2424 | + |
2425 | +SERIES=$1 |
2426 | +TARBALL=$2 |
2427 | + |
2428 | +mkdir -p rootfs |
2429 | +debootstrap --components=main,universe $SERIES rootfs |
2430 | +tar cf $TARBALL rootfs |
2431 | +rm -rf rootfs |
2432 | diff --git a/tools/common.sh b/tools/common.sh |
2433 | new file mode 100755 |
2434 | index 0000000..c30d2d7 |
2435 | --- /dev/null |
2436 | +++ b/tools/common.sh |
2437 | @@ -0,0 +1,93 @@ |
2438 | +#!/bin/sh -ex |
2439 | +# |
2440 | +# Copyright (C) 2017 Canonical Ltd |
2441 | +# |
2442 | +# This program is free software: you can redistribute it and/or modify |
2443 | +# it under the terms of the GNU General Public License version 3 as |
2444 | +# published by the Free Software Foundation. |
2445 | +# |
2446 | +# This program is distributed in the hope that it will be useful, |
2447 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
2448 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
2449 | +# GNU General Public License for more details. |
2450 | +# |
2451 | +# You should have received a copy of the GNU General Public License |
2452 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
2453 | + |
2454 | +# Set common variables used by the jenkins jobs |
2455 | +set_jenkins_env () |
2456 | +{ |
2457 | + SSH_PATH="${JENKINS_HOME}/.ssh/" |
2458 | + SSH_KEY_PATH="${SSH_PATH}/bazaar.launchpad.net/system-enablement-ci-bot" |
2459 | + |
2460 | + SSH="ssh -i $SSH_KEY_PATH/id_rsa $REMOTE_USER@$REMOTE_WORKER" |
2461 | + SCP="scp -i $SSH_KEY_PATH/id_rsa" |
2462 | + |
2463 | + REPO=https://git.launchpad.net/~snappy-hwe-team/snappy-hwe-snaps/+git/$CI_REPO |
2464 | + BRANCH=$CI_BRANCH |
2465 | + |
2466 | + # If no CI repo/branch is set fallback to the source repo/branch set |
2467 | + # which will be the case for those repositories which don't contain |
2468 | + # a snap. |
2469 | + if [ -z "$CI_REPO" ]; then |
2470 | + REPO=$SOURCE_GIT_REPO |
2471 | + BRANCH=$SOURCE_GIT_REPO_BRANCH |
2472 | + fi |
2473 | + |
2474 | + REMOTE_WORKSPACE=/home/$REMOTE_USER/$BUILD_TAG |
2475 | + REMOTE_RESULTS_BASE_DIR=/home/$REMOTE_USER/results |
2476 | +} |
2477 | + |
2478 | +# Sets variables |
2479 | +# TEST_TYPE={script, spread} |
2480 | +# HW_TESTS_RESULT={0, !=0} -> {has hw tests, does not have hw tests} |
2481 | +# FIXME Maybe depending on context the call to clone could be avoided |
2482 | +set_test_type () |
2483 | +{ |
2484 | + tmp_srcdir=$(mktemp -d) |
2485 | + |
2486 | + # We use FAIL to make sure we do not exit until we free tmp_srcdir |
2487 | + FAIL=no |
2488 | + git clone --depth 1 -b "$BRANCH" "$REPO" "$tmp_srcdir"/src || FAIL=yes |
2489 | + cd "$tmp_srcdir"/src || FAIL=yes |
2490 | + |
2491 | + TEST_TYPE=none |
2492 | + if [ -e "$tmp_srcdir/src/spread.yaml" ]; then |
2493 | + TEST_TYPE=spread |
2494 | + fi |
2495 | + # run-tests.sh gets priority over spread.yaml |
2496 | + if [ -e "$tmp_srcdir/src/run-tests.sh" ]; then |
2497 | + TEST_TYPE=script |
2498 | + fi |
2499 | + |
2500 | + # TODO: Use https://github.com/0k/shyaml in the future for this |
2501 | + if grep -q "type: adhoc" spread.yaml; then |
2502 | + HW_TESTS_RESULT=0 |
2503 | + else |
2504 | + HW_TESTS_RESULT=1 |
2505 | + fi |
2506 | + |
2507 | + # Components have the ability to disable CI tests if they can't provide any. |
2508 | + # This is only accepted in a few cases and should be generally avoided. |
2509 | + CI_TESTS_DISABLED=no |
2510 | + if [ -e "$tmp_srcdir"/src/.ci_tests_disabled ]; then |
2511 | + CI_TESTS_DISABLED=yes |
2512 | + fi |
2513 | + |
2514 | + rm -rf "$tmp_srcdir" |
2515 | + |
2516 | + if [ "$FAIL" = yes ]; then |
2517 | + echo "ERROR: critical in set_test_type()" |
2518 | + exit 1 |
2519 | + fi |
2520 | + |
2521 | + if [ "$CI_TESTS_DISABLED" = yes ]; then |
2522 | + echo "WARNING: Component has no CI tests so not running anything here" |
2523 | + exit 0 |
2524 | + fi |
2525 | + |
2526 | + if [ "$TEST_TYPE" = none ]; then |
2527 | + echo "ERROR: missing spread or script tests: you must provide one of them" |
2528 | + exit 1 |
2529 | + fi |
2530 | +} |
2531 | diff --git a/tools/delete-ci-repo.py b/tools/delete-ci-repo.py |
2532 | new file mode 100755 |
2533 | index 0000000..35c762d |
2534 | --- /dev/null |
2535 | +++ b/tools/delete-ci-repo.py |
2536 | @@ -0,0 +1,45 @@ |
2537 | +#!/usr/bin/env python |
2538 | +# |
2539 | +# Copyright (C) 2017 Canonical Ltd |
2540 | +# |
2541 | +# This program is free software: you can redistribute it and/or modify |
2542 | +# it under the terms of the GNU General Public License version 3 as |
2543 | +# published by the Free Software Foundation. |
2544 | +# |
2545 | +# This program is distributed in the hope that it will be useful, |
2546 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
2547 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
2548 | +# GNU General Public License for more details. |
2549 | +# |
2550 | +# You should have received a copy of the GNU General Public License |
2551 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
2552 | + |
2553 | +from launchpadlib.launchpad import Launchpad |
2554 | + |
2555 | +from argparse import ArgumentParser |
2556 | + |
2557 | +import se_utils |
2558 | + |
2559 | +print("Running delete-ci-repo") |
2560 | + |
2561 | +parser = ArgumentParser(description="Delete a git repository stored in launchpad") |
2562 | +parser.add_argument('--git-repo', help="Git repository to be deleted") |
2563 | + |
2564 | +args = vars(parser.parse_args()) |
2565 | + |
2566 | +git_repo = args['git_repo'] |
2567 | +ind = git_repo.find('~') |
2568 | +if ind == -1: |
2569 | + print("Bad git repo {}".format(git_repo)) |
2570 | + exit(1) |
2571 | + |
2572 | +lp_repo = git_repo[ind:] |
2573 | + |
2574 | +lp_app = se_utils.get_config_option("lp_app") |
2575 | +lp_env = se_utils.get_config_option("lp_env") |
2576 | +credential_store_path = se_utils.get_config_option('credential_store_path') |
2577 | +launchpad = se_utils.get_launchpad(None, credential_store_path, lp_app, lp_env) |
2578 | + |
2579 | +repo = launchpad.git_repositories.getByPath(path=lp_repo) |
2580 | +print("Removing {}".format(lp_repo)) |
2581 | +repo.lp_delete() |
2582 | diff --git a/tools/hardware-test.sh b/tools/hardware-test.sh |
2583 | new file mode 100755 |
2584 | index 0000000..41eee60 |
2585 | --- /dev/null |
2586 | +++ b/tools/hardware-test.sh |
2587 | @@ -0,0 +1,112 @@ |
2588 | +#!/bin/sh -ex |
2589 | +# |
2590 | +# Copyright (C) 2017 Canonical Ltd |
2591 | +# |
2592 | +# This program is free software: you can redistribute it and/or modify |
2593 | +# it under the terms of the GNU General Public License version 3 as |
2594 | +# published by the Free Software Foundation. |
2595 | +# |
2596 | +# This program is distributed in the hope that it will be useful, |
2597 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
2598 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
2599 | +# GNU General Public License for more details. |
2600 | +# |
2601 | +# You should have received a copy of the GNU General Public License |
2602 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
2603 | + |
2604 | +# Runs tests on real HW. Requires set_jenkins_env and set_test_type to have |
2605 | +# already run. |
2606 | +run_hardware_tests () |
2607 | +{ |
2608 | + TEST_RESULTS=test_results |
2609 | + |
2610 | + # Just dragonboard for the moment |
2611 | + DRAGONBOARD_TEST=testflinger-dragonboard.yaml |
2612 | + |
2613 | + # We use jq to process testflinger output |
2614 | + if ! which jq; then |
2615 | + sudo apt install --yes --allow-downgrades --allow-remove-essential \ |
2616 | + --allow-change-held-packages jq |
2617 | + fi |
2618 | + |
2619 | + # Initially the device has a password-less ubuntu user. But spread needs a |
2620 | + # user with a password, so we use the DEVICE_USER/DEVICE_PASSWORD pair to |
2621 | + # create it. Note: {device_ip} gets substituted by testflinger. |
2622 | + DEVICE_USER=test |
2623 | + DEVICE_PASSWORD=test |
2624 | + DEVICE_SSH="ssh -q -o UserKnownHostsFile=/dev/null |
2625 | + -o StrictHostKeyChecking=no -p 22 ubuntu@{device_ip}" |
2626 | + |
2627 | + if [ "$TEST_TYPE" = script ]; then |
2628 | + TEST_COMMAND="./run-tests.sh --spread-system=hw-ubuntu-core-16 |
2629 | + --external-address={device_ip}:22 --external-user=$DEVICE_USER |
2630 | + --external-password=$DEVICE_PASSWORD --debug" |
2631 | + else |
2632 | + TEST_COMMAND="export SPREAD_EXTERNAL_ADDRESS={device_ip}:22 && |
2633 | + export SPREAD_EXTERNAL_USER=$DEVICE_USER && |
2634 | + export SPREAD_EXTERNAL_PASSWORD=$DEVICE_PASSWORD && |
2635 | + ./spread -vv external:hw-ubuntu-core-16" |
2636 | + fi |
2637 | + |
2638 | + cd "$WORKSPACE" |
2639 | + |
2640 | + # Run testflinger from our bare metal server so we can install it |
2641 | + |
2642 | + $SSH mkdir -p "$REMOTE_WORKSPACE" |
2643 | + |
2644 | + # If the snap has been built, copy over |
2645 | + if [ -n "$RESULTS_ID" ]; then |
2646 | + set +x |
2647 | + # We need to flatten the key here to avoid issues with yaml parsing |
2648 | + SSH_KEY_DATA=$(tr '\n:' '?!' < "$SSH_KEY_PATH"/id_rsa) |
2649 | + set -x |
2650 | + fi |
2651 | + |
2652 | + cat << EOF > "$WORKSPACE"/"$DRAGONBOARD_TEST" |
2653 | +job_queue: dragonboard |
2654 | +provision_data: |
2655 | + channel: stable |
2656 | +test_data: |
2657 | + test_cmds: |
2658 | + - git clone --depth 1 -b $BRANCH $REPO src |
2659 | + - cd src && curl -s -O https://niemeyer.s3.amazonaws.com/spread-amd64.tar.gz && tar xzvf spread-amd64.tar.gz |
2660 | + - $DEVICE_SSH "sudo adduser --extrausers --quiet --disabled-password --gecos '' $DEVICE_USER" |
2661 | + - $DEVICE_SSH "echo $DEVICE_USER:$DEVICE_PASSWORD | sudo chpasswd" |
2662 | + - $DEVICE_SSH "echo '$DEVICE_USER ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/create-user-test" |
2663 | + - if [ -n $RESULTS_ID ]; then set +x; echo "$SSH_KEY_DATA" | tr '?!' '\n:' > ssh_key; set -x; chmod 600 ssh_key; scp -i ssh_key $REMOTE_USER@$REMOTE_WORKER:$REMOTE_RESULTS_BASE_DIR/$RESULTS_ID/*.snap src/; fi |
2664 | + - cd src && export PATH=$PATH:\$(pwd) && $TEST_COMMAND |
2665 | +EOF |
2666 | + |
2667 | + $SCP "$WORKSPACE"/"$DRAGONBOARD_TEST" "$REMOTE_USER"@"$REMOTE_WORKER":"$REMOTE_WORKSPACE"/ |
2668 | + |
2669 | + $SSH << EOF |
2670 | +#!/bin/sh |
2671 | +set -ex |
2672 | + |
2673 | +cd $REMOTE_WORKSPACE |
2674 | +if ! which virtualenv; then |
2675 | + sudo apt install --yes --allow-downgrades --allow-remove-essential --allow-change-held-packagesvirtualenv |
2676 | +fi |
2677 | + |
2678 | +git clone https://git.launchpad.net/testflinger-cli |
2679 | +cd testflinger-cli |
2680 | +virtualenv -p python3 env |
2681 | +. env/bin/activate |
2682 | +./setup.py install |
2683 | + |
2684 | +JOB_ID=\$(testflinger-cli submit -q $REMOTE_WORKSPACE/$DRAGONBOARD_TEST) |
2685 | +echo "JOB_ID: \${JOB_ID}" |
2686 | + |
2687 | +testflinger-cli poll \${JOB_ID} |
2688 | +testflinger-cli results \${JOB_ID} > $REMOTE_WORKSPACE/$TEST_RESULTS |
2689 | +EOF |
2690 | + |
2691 | + $SCP "$REMOTE_USER"@"$REMOTE_WORKER":"$REMOTE_WORKSPACE"/"$TEST_RESULTS" "$WORKSPACE"/ |
2692 | + |
2693 | + $SSH sudo rm -rf "$REMOTE_WORKSPACE" |
2694 | + |
2695 | + TEST_STATUS=$(jq -r .test_status "$WORKSPACE"/"$TEST_RESULTS") |
2696 | + echo "Test exit status: $TEST_STATUS" |
2697 | + |
2698 | + return "$TEST_STATUS" |
2699 | +} |
2700 | diff --git a/tools/se_utils/__init__.py b/tools/se_utils/__init__.py |
2701 | new file mode 100644 |
2702 | index 0000000..280450c |
2703 | --- /dev/null |
2704 | +++ b/tools/se_utils/__init__.py |
2705 | @@ -0,0 +1,122 @@ |
2706 | +#!/usr/bin/env python |
2707 | +# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*- |
2708 | +# |
2709 | +# Copyright (C) 2016 Canonical Ltd |
2710 | +# |
2711 | +# This program is free software: you can redistribute it and/or modify |
2712 | +# it under the terms of the GNU General Public License version 3 as |
2713 | +# published by the Free Software Foundation. |
2714 | +# |
2715 | +# This program is distributed in the hope that it will be useful, |
2716 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
2717 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
2718 | +# GNU General Public License for more details. |
2719 | +# |
2720 | +# You should have received a copy of the GNU General Public License |
2721 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
2722 | + |
2723 | +import atexit |
2724 | +import sys |
2725 | +import time |
2726 | +import logging |
2727 | +import os |
2728 | +import yaml |
2729 | +from shutil import rmtree |
2730 | +from launchpadlib.credentials import RequestTokenAuthorizationEngine |
2731 | +from lazr.restfulclient.errors import HTTPError |
2732 | +from launchpadlib.launchpad import Launchpad |
2733 | +from launchpadlib.credentials import UnencryptedFileCredentialStore |
2734 | + |
2735 | +ACCESS_TOKEN_POLL_TIME = 10 |
2736 | +WAITING_FOR_USER = """Open this link: |
2737 | +{} |
2738 | +to authorize this program to access Launchpad on your behalf. |
2739 | +Waiting to hear from Launchpad about your decision. . . .""" |
2740 | + |
2741 | + |
2742 | +class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine): |
2743 | + """Authorize a token in a server environment (with no browser). |
2744 | + |
2745 | + Print a link for the user to copy-and-paste into his/her browser |
2746 | + for authentication. |
2747 | + """ |
2748 | + |
2749 | + def __init__(self, *args, **kwargs): |
2750 | + # as implemented in AuthorizeRequestTokenWithBrowser |
2751 | + kwargs['consumer_name'] = None |
2752 | + kwargs.pop('allow_access_levels', None) |
2753 | + super(AuthorizeRequestTokenWithConsole, self).__init__(*args, **kwargs) |
2754 | + |
2755 | + def make_end_user_authorize_token(self, credentials, request_token): |
2756 | + """Ask the end-user to authorize the token in their browser. |
2757 | + |
2758 | + """ |
2759 | + authorization_url = self.authorization_url(request_token) |
2760 | + print WAITING_FOR_USER.format(authorization_url) |
2761 | + # if we don't flush we may not see the message |
2762 | + sys.stdout.flush() |
2763 | + while credentials.access_token is None: |
2764 | + time.sleep(ACCESS_TOKEN_POLL_TIME) |
2765 | + try: |
2766 | + credentials.exchange_request_token_for_access_token( |
2767 | + self.web_root) |
2768 | + break |
2769 | + except HTTPError, e: |
2770 | + if e.response.status == 403: |
2771 | + # The user decided not to authorize this |
2772 | + # application. |
2773 | + raise e |
2774 | + elif e.response.status == 401: |
2775 | + # The user has not made a decision yet. |
2776 | + pass |
2777 | + else: |
2778 | + # There was an error accessing the server. |
2779 | + raise e |
2780 | + |
2781 | +# launchpadlib is not thread/process safe so we are creating launchpadlib |
2782 | +# cache in /tmp per process which gets cleaned up at the end |
2783 | +# see also lp:459418 and lp:1025153 |
2784 | +launchpad_cachedir = os.path.join('/tmp', str(os.getpid()), '.launchpadlib') |
2785 | + |
2786 | +# `launchpad_cachedir` is leaked upon unexpected exits |
2787 | +# adding this cleanup to stop directories filling up `/tmp/` |
2788 | +atexit.register(rmtree, os.path.join('/tmp', |
2789 | + str(os.getpid())), |
2790 | + ignore_errors=True) |
2791 | + |
2792 | + |
2793 | +def get_launchpad(launchpadlib_dir=None, credential_store_path=None, lp_app=None, lp_env=None): |
2794 | + """ return a launchpad API class. In case launchpadlib_dir is |
2795 | + specified used that directory to store launchpadlib cache instead of |
2796 | + the default """ |
2797 | + store = UnencryptedFileCredentialStore(credential_store_path) |
2798 | + authorization_engine = AuthorizeRequestTokenWithConsole(lp_env, lp_app) |
2799 | + lib_dir=launchpad_cachedir |
2800 | + if launchpadlib_dir != None: |
2801 | + lib_dir = launchpadlib_dir |
2802 | + return Launchpad.login_with(lp_app, lp_env, |
2803 | + credential_store=store, |
2804 | + authorization_engine=authorization_engine, |
2805 | + launchpadlib_dir=lib_dir, |
2806 | + version='devel') |
2807 | + |
2808 | +# Load configuration for the current agent we're running on. All agents were |
2809 | +# provisioned when they were setup with a proper configuration. See |
2810 | +# https://wiki.canonical.com/InformationInfrastructure/Jenkaas/UserDocs for |
2811 | +# more details. |
2812 | +def load_config(): |
2813 | + files = [os.path.expanduser('~/.jlp/jlp.config'), 'jlp.config'] |
2814 | + for config_file in files: |
2815 | + try: |
2816 | + config = yaml.safe_load(open(config_file, 'r')) |
2817 | + return config |
2818 | + except IOError: |
2819 | + pass |
2820 | + print("ERROR: No config file found") |
2821 | + sys.exit(1) |
2822 | + |
2823 | +# Return a configuration option from the agent configuration specified by the |
2824 | +# name argument. |
2825 | +def get_config_option(name): |
2826 | + config = load_config() |
2827 | + return config[name] |
2828 | diff --git a/tools/se_utils/__init__.pyc b/tools/se_utils/__init__.pyc |
2829 | new file mode 100644 |
2830 | index 0000000..f1be9ca |
2831 | Binary files /dev/null and b/tools/se_utils/__init__.pyc differ |
2832 | diff --git a/tools/shyaml b/tools/shyaml |
2833 | new file mode 100755 |
2834 | index 0000000..e4618ec |
2835 | --- /dev/null |
2836 | +++ b/tools/shyaml |
2837 | @@ -0,0 +1,454 @@ |
2838 | +#!/usr/bin/env python |
2839 | + |
2840 | +# Taken from upstream git repository https://github.com/0k/shyaml |
2841 | +# at revision d77e30599a0971c51896ef97d21883550e7e9979 |
2842 | + |
2843 | +## Note: to launch test, you can use: |
2844 | +## python -m doctest -d shyaml.py |
2845 | +## or |
2846 | +## nosetests |
2847 | + |
2848 | +from __future__ import print_function |
2849 | + |
2850 | +import sys |
2851 | +import yaml |
2852 | +import os.path |
2853 | +import re |
2854 | + |
2855 | +PY3 = sys.version_info[0] >= 3 |
2856 | + |
2857 | +EXNAME = os.path.basename(sys.argv[0]) |
2858 | + |
2859 | +USAGE = """\ |
2860 | +Usage: |
2861 | + |
2862 | + %(exname)s (-h|--help) |
2863 | + %(exname)s [-y|--yaml] ACTION KEY [DEFAULT] |
2864 | +""" % {"exname": EXNAME} |
2865 | + |
2866 | +HELP = """ |
2867 | +Parses and output chosen subpart or values from YAML input. |
2868 | +It reads YAML in stdin and will output on stdout it's return value. |
2869 | + |
2870 | +%(usage)s |
2871 | + |
2872 | +Options: |
2873 | + |
2874 | + -y, --yaml |
2875 | + Output only YAML safe value, more precisely, even |
2876 | + literal values will be YAML quoted. This behavior |
2877 | + is required if you want to output YAML subparts and |
2878 | + further process it. If you know you have are dealing |
2879 | + with safe literal value, then you don't need this. |
2880 | + (Default: no safe YAML output) |
2881 | + |
2882 | + ACTION Depending on the type of data you've targetted |
2883 | + thanks to the KEY, ACTION can be: |
2884 | + |
2885 | + These ACTIONs applies to any YAML type: |
2886 | + |
2887 | + get-type ## returns a short string |
2888 | + get-value ## returns YAML |
2889 | + |
2890 | + This ACTION applies to 'sequence' and 'struct' YAML type: |
2891 | + |
2892 | + get-values{,-0} ## return list of YAML |
2893 | + |
2894 | + These ACTION applies to 'struct' YAML type: |
2895 | + |
2896 | + keys{,-0} ## return list of YAML |
2897 | + values{,-0} ## return list of YAML |
2898 | + key-values,{,-0} ## return list of YAML |
2899 | + |
2900 | + Note that any value returned is returned on stdout, and |
2901 | + when returning ``list of YAML``, it'll be separated by |
2902 | + ``\\n`` or ``NUL`` char depending of you've used the |
2903 | + ``-0`` suffixed ACTION. |
2904 | + |
2905 | + KEY Identifier to browse and target subvalues into YAML |
2906 | + structure. Use ``.`` to parse a subvalue. If you need |
2907 | + to use a literal ``.`` or ``\``, use ``\`` to quote it. |
2908 | + |
2909 | + Use struct keyword to browse ``struct`` YAML data and use |
2910 | + integers to browse ``sequence`` YAML data. |
2911 | + |
2912 | + DEFAULT if not provided and given KEY do not match any value in |
2913 | + the provided YAML, then DEFAULT will be returned. If no |
2914 | + default is provided and the KEY do not match any value |
2915 | + in the provided YAML, %(exname)s will fail with an error |
2916 | + message. |
2917 | + |
2918 | +Examples: |
2919 | + |
2920 | + ## get last grocery |
2921 | + cat recipe.yaml | %(exname)s get-value groceries.-1 |
2922 | + |
2923 | + ## get all words of my french dictionary |
2924 | + cat dictionaries.yaml | %(exname)s keys-0 french.dictionary |
2925 | + |
2926 | + ## get YAML config part of 'myhost' |
2927 | + cat hosts_config.yaml | %(exname)s get-value cfgs.myhost |
2928 | + |
2929 | +""" % {"exname": EXNAME, "usage": USAGE} |
2930 | + |
2931 | +## |
2932 | +## Keep previous order in YAML |
2933 | +## |
2934 | + |
2935 | +try: |
2936 | + # included in standard lib from Python 2.7 |
2937 | + from collections import OrderedDict |
2938 | +except ImportError: |
2939 | + # try importing the backported drop-in replacement |
2940 | + # it's available on PyPI |
2941 | + from ordereddict import OrderedDict |
2942 | + |
2943 | + |
2944 | +## Ensure that there are no collision with legacy OrderedDict |
2945 | +## that could be used for omap for instance. |
2946 | +class MyOrderedDict(OrderedDict): |
2947 | + pass |
2948 | + |
2949 | +yaml.add_representer( |
2950 | + MyOrderedDict, |
2951 | + lambda cls, data: cls.represent_dict(data.items())) |
2952 | + |
2953 | + |
2954 | +def construct_omap(cls, node): |
2955 | + ## Force unfolding reference and merges |
2956 | + ## otherwise it would fail on 'merge' |
2957 | + cls.flatten_mapping(node) |
2958 | + return MyOrderedDict(cls.construct_pairs(node)) |
2959 | + |
2960 | + |
2961 | +yaml.add_constructor( |
2962 | + yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, |
2963 | + construct_omap) |
2964 | + |
2965 | + |
2966 | +## |
2967 | +## Key specifier |
2968 | +## |
2969 | + |
2970 | +def tokenize(s): |
2971 | + r"""Returns an iterable through all subparts of string splitted by '.' |
2972 | + |
2973 | + So: |
2974 | + |
2975 | + >>> list(tokenize('foo.bar.wiz')) |
2976 | + ['foo', 'bar', 'wiz'] |
2977 | + |
2978 | + Contrary to traditional ``.split()`` method, this function has to |
2979 | + deal with any type of data in the string. So it actually |
2980 | + interprets the string. Characters with meaning are '.' and '\'. |
2981 | + Both of these can be included in a token by quoting them with '\'. |
2982 | + |
2983 | + So dot of slashes can be contained in token: |
2984 | + |
2985 | + >>> print('\n'.join(tokenize(r'foo.dot<\.>.slash<\\>'))) |
2986 | + foo |
2987 | + dot<.> |
2988 | + slash<\> |
2989 | + |
2990 | + Notice that empty keys are also supported: |
2991 | + |
2992 | + >>> list(tokenize(r'foo..bar')) |
2993 | + ['foo', '', 'bar'] |
2994 | + |
2995 | + Given an empty string: |
2996 | + |
2997 | + >>> list(tokenize(r'')) |
2998 | + [''] |
2999 | + |
3000 | + And a None value: |
3001 | + |
3002 | + >>> list(tokenize(None)) |
3003 | + [] |
3004 | + |
3005 | + """ |
3006 | + if s is None: |
3007 | + raise StopIteration |
3008 | + tokens = (re.sub(r'\\(\\|\.)', r'\1', m.group(0)) |
3009 | + for m in re.finditer(r'((\\.|[^.\\])*)', s)) |
3010 | + ## an empty string superfluous token is added after all non-empty token |
3011 | + for token in tokens: |
3012 | + if len(token) != 0: |
3013 | + next(tokens) |
3014 | + yield token |
3015 | + |
3016 | + |
3017 | +def mget(dct, key): |
3018 | + r"""Allow to get values deep in recursive dict with doted keys |
3019 | + |
3020 | + Accessing leaf values is quite straightforward: |
3021 | + |
3022 | + >>> dct = {'a': {'x': 1, 'b': {'c': 2}}} |
3023 | + >>> mget(dct, 'a.x') |
3024 | + 1 |
3025 | + >>> mget(dct, 'a.b.c') |
3026 | + 2 |
3027 | + |
3028 | + But you can also get subdict if your key is not targeting a |
3029 | + leaf value: |
3030 | + |
3031 | + >>> mget(dct, 'a.b') |
3032 | + {'c': 2} |
3033 | + |
3034 | + As a special feature, list access is also supported by providing a |
3035 | + (possibily signed) integer, it'll be interpreted as usual python |
3036 | + sequence access using bracket notation: |
3037 | + |
3038 | + >>> mget({'a': {'x': [1, 5], 'b': {'c': 2}}}, 'a.x.-1') |
3039 | + 5 |
3040 | + >>> mget({'a': {'x': 1, 'b': [{'c': 2}]}}, 'a.b.0.c') |
3041 | + 2 |
3042 | + |
3043 | + Keys that contains '.' can be accessed by escaping them: |
3044 | + |
3045 | + >>> dct = {'a': {'x': 1}, 'a.x': 3, 'a.y': 4} |
3046 | + >>> mget(dct, 'a.x') |
3047 | + 1 |
3048 | + >>> mget(dct, r'a\.x') |
3049 | + 3 |
3050 | + >>> mget(dct, r'a.y') ## doctest: +IGNORE_EXCEPTION_DETAIL |
3051 | + Traceback (most recent call last): |
3052 | + ... |
3053 | + MissingKeyError: missing key 'y' in dict. |
3054 | + >>> mget(dct, r'a\.y') |
3055 | + 4 |
3056 | + |
3057 | + As a consequence, if your key contains a '\', you should also escape it: |
3058 | + |
3059 | + >>> dct = {r'a\x': 3, r'a\.x': 4, 'a.x': 5, 'a\\': {'x': 6}} |
3060 | + >>> mget(dct, r'a\\x') |
3061 | + 3 |
3062 | + >>> mget(dct, r'a\\\.x') |
3063 | + 4 |
3064 | + >>> mget(dct, r'a\\.x') |
3065 | + 6 |
3066 | + >>> mget({'a\\': {'b': 1}}, r'a\\.b') |
3067 | + 1 |
3068 | + >>> mget({r'a.b\.c': 1}, r'a\.b\\\.c') |
3069 | + 1 |
3070 | + |
3071 | + And even empty strings key are supported: |
3072 | + |
3073 | + >>> dct = {r'a': {'': {'y': 3}, 'y': 4}, 'b': {'': {'': 1}}, '': 2} |
3074 | + >>> mget(dct, r'a..y') |
3075 | + 3 |
3076 | + >>> mget(dct, r'a.y') |
3077 | + 4 |
3078 | + >>> mget(dct, r'') |
3079 | + 2 |
3080 | + >>> mget(dct, r'b..') |
3081 | + 1 |
3082 | + |
3083 | + It will complain if you are trying to get into a leaf: |
3084 | + |
3085 | + >>> mget({'a': 1}, 'a.y') ## doctest: +IGNORE_EXCEPTION_DETAIL |
3086 | + Traceback (most recent call last): |
3087 | + ... |
3088 | + NonDictLikeTypeError: can't query subvalue 'y' of a leaf... |
3089 | + |
3090 | + if the key is None, the whole dct should be sent back: |
3091 | + |
3092 | + >>> mget({'a': 1}, None) |
3093 | + {'a': 1} |
3094 | + |
3095 | + """ |
3096 | + return aget(dct, tokenize(key)) |
3097 | + |
3098 | + |
3099 | +class MissingKeyError(KeyError): |
3100 | + """Raised when querying a dict-like structure on non-existing keys""" |
3101 | + |
3102 | + def __str__(self): |
3103 | + return self.message |
3104 | + |
3105 | + |
3106 | +class NonDictLikeTypeError(TypeError): |
3107 | + """Raised when attempting to traverse non-dict like structure""" |
3108 | + |
3109 | + |
3110 | +class IndexNotIntegerError(ValueError): |
3111 | + """Raised when attempting to traverse sequence without using an integer""" |
3112 | + |
3113 | + |
3114 | +class IndexOutOfRange(IndexError): |
3115 | + """Raised when attempting to traverse sequence without using an integer""" |
3116 | + |
3117 | + |
3118 | +def aget(dct, key): |
3119 | + r"""Allow to get values deep in a dict with iterable keys |
3120 | + |
3121 | + Accessing leaf values is quite straightforward: |
3122 | + |
3123 | + >>> dct = {'a': {'x': 1, 'b': {'c': 2}}} |
3124 | + >>> aget(dct, ('a', 'x')) |
3125 | + 1 |
3126 | + >>> aget(dct, ('a', 'b', 'c')) |
3127 | + 2 |
3128 | + |
3129 | + If key is empty, it returns unchanged the ``dct`` value. |
3130 | + |
3131 | + >>> aget({'x': 1}, ()) |
3132 | + {'x': 1} |
3133 | + |
3134 | + """ |
3135 | + key = iter(key) |
3136 | + try: |
3137 | + head = next(key) |
3138 | + except StopIteration: |
3139 | + return dct |
3140 | + |
3141 | + if isinstance(dct, list): |
3142 | + try: |
3143 | + idx = int(head) |
3144 | + except ValueError: |
3145 | + raise IndexNotIntegerError( |
3146 | + "non-integer index %r provided on a list." |
3147 | + % head) |
3148 | + try: |
3149 | + value = dct[idx] |
3150 | + except IndexError: |
3151 | + raise IndexOutOfRange( |
3152 | + "index %d is out of range (%d elements in list)." |
3153 | + % (idx, len(dct))) |
3154 | + else: |
3155 | + try: |
3156 | + value = dct[head] |
3157 | + except KeyError: |
3158 | + ## Replace with a more informative KeyError |
3159 | + raise MissingKeyError( |
3160 | + "missing key %r in dict." |
3161 | + % (head, )) |
3162 | + except: |
3163 | + raise NonDictLikeTypeError( |
3164 | + "can't query subvalue %r of a leaf%s." |
3165 | + % (head, |
3166 | + (" (leaf value is %r)" % dct) |
3167 | + if len(repr(dct)) < 15 else "")) |
3168 | + return aget(value, key) |
3169 | + |
3170 | + |
3171 | +def stderr(msg): |
3172 | + """Convenience function to write short message to stderr.""" |
3173 | + sys.stderr.write(msg) |
3174 | + |
3175 | + |
3176 | +def stdout(value): |
3177 | + """Convenience function to write short message to stdout.""" |
3178 | + sys.stdout.write(value) |
3179 | + |
3180 | + |
3181 | +def die(msg, errlvl=1, prefix="Error: "): |
3182 | + """Convenience function to write short message to stderr and quit.""" |
3183 | + stderr("%s%s\n" % (prefix, msg)) |
3184 | + sys.exit(errlvl) |
3185 | + |
3186 | +SIMPLE_TYPES = (str if PY3 else basestring, int, float, type(None)) |
3187 | +COMPLEX_TYPES = (list, dict) |
3188 | + |
3189 | + |
3190 | +def magic_dump(value): |
3191 | + """Returns a representation of values directly usable by bash. |
3192 | + |
3193 | + Literal types are printed as-is (avoiding quotes around string for |
3194 | + instance). But complex type are written in a YAML useable format. |
3195 | + |
3196 | + """ |
3197 | + return value if isinstance(value, SIMPLE_TYPES) \ |
3198 | + else yaml.dump(value, default_flow_style=False) |
3199 | + |
3200 | +def yaml_dump(value): |
3201 | + """Returns a representation of values directly usable by bash. |
3202 | + |
3203 | + Literal types are quoted and safe to use as YAML. |
3204 | + |
3205 | + """ |
3206 | + return yaml.dump(value, default_flow_style=False) |
3207 | + |
3208 | + |
3209 | +def type_name(value): |
3210 | + """Returns pseudo-YAML type name of given value.""" |
3211 | + return "struct" if isinstance(value, dict) else \ |
3212 | + "sequence" if isinstance(value, (tuple, list)) else \ |
3213 | + type(value).__name__ |
3214 | + |
3215 | + |
3216 | +def main(args): ## pylint: disable=too-many-branches |
3217 | + """Entrypoint of the whole application""" |
3218 | + |
3219 | + if len(args) == 0: |
3220 | + stderr("Error: Bad number of arguments.\n") |
3221 | + die(USAGE, errlvl=1, prefix="") |
3222 | + |
3223 | + if len(args) == 1 and args[0] in ("-h", "--help"): |
3224 | + stdout(HELP) |
3225 | + exit(0) |
3226 | + |
3227 | + dump = magic_dump |
3228 | + for arg in ["-y", "--yaml"]: |
3229 | + if arg in args: |
3230 | + args.remove(arg) |
3231 | + dump = yaml_dump |
3232 | + |
3233 | + action = args[0] |
3234 | + key_value = None if len(args) == 1 else args[1] |
3235 | + default = args[2] if len(args) > 2 else None |
3236 | + contents = yaml.load(sys.stdin) |
3237 | + |
3238 | + try: |
3239 | + try: |
3240 | + value = mget(contents, key_value) |
3241 | + except (IndexOutOfRange, MissingKeyError): |
3242 | + if default is None: |
3243 | + raise |
3244 | + value = default |
3245 | + except (IndexOutOfRange, MissingKeyError, |
3246 | + NonDictLikeTypeError, IndexNotIntegerError) as exc: |
3247 | + msg = str(exc.message) |
3248 | + die("invalid path %r, %s" |
3249 | + % (key_value, |
3250 | + msg.replace('list', 'sequence').replace('dict', 'struct'))) |
3251 | + |
3252 | + tvalue = type_name(value) |
3253 | + termination = "\0" if action.endswith("-0") else "\n" |
3254 | + |
3255 | + if action == "get-value": |
3256 | + print(dump(value), end='') |
3257 | + elif action in ("get-values", "get-values-0"): |
3258 | + if isinstance(value, dict): |
3259 | + for k, v in value.iteritems(): |
3260 | + stdout("%s%s%s%s" % (dump(k), termination, |
3261 | + dump(v), termination)) |
3262 | + elif isinstance(value, list): |
3263 | + for l in value: |
3264 | + stdout("%s%s" % (dump(l), termination)) |
3265 | + else: |
3266 | + die("%s does not support %r type. " |
3267 | + "Please provide or select a sequence or struct." |
3268 | + % (action, tvalue)) |
3269 | + elif action == "get-type": |
3270 | + print(tvalue) |
3271 | + elif action in ("keys", "keys-0", |
3272 | + "values", "values-0", |
3273 | + "key-values", "key-values-0"): |
3274 | + if isinstance(value, dict): |
3275 | + method = value.keys if action.startswith("keys") else \ |
3276 | + value.items if action.startswith("key-values") else \ |
3277 | + value.values |
3278 | + output = (lambda x: termination.join(str(dump(e)) for e in x)) \ |
3279 | + if action.startswith("key-values") else \ |
3280 | + dump |
3281 | + for k in method(): |
3282 | + stdout("%s%s" % (output(k), termination)) |
3283 | + else: |
3284 | + die("%s does not support %r type. " |
3285 | + "Please provide or select a struct." % (action, tvalue)) |
3286 | + else: |
3287 | + die("Invalid argument.\n%s" % USAGE) |
3288 | + |
3289 | + |
3290 | +if __name__ == "__main__": |
3291 | + sys.exit(main(sys.argv[1:])) |
3292 | diff --git a/tools/snapbuild.sh b/tools/snapbuild.sh |
3293 | new file mode 100755 |
3294 | index 0000000..53b4518 |
3295 | --- /dev/null |
3296 | +++ b/tools/snapbuild.sh |
3297 | @@ -0,0 +1,192 @@ |
3298 | +#!/bin/sh |
3299 | +# |
3300 | +# Copyright (C) 2017 Canonical Ltd |
3301 | +# |
3302 | +# This program is free software: you can redistribute it and/or modify |
3303 | +# it under the terms of the GNU General Public License version 3 as |
3304 | +# published by the Free Software Foundation. |
3305 | +# |
3306 | +# This program is distributed in the hope that it will be useful, |
3307 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
3308 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
3309 | +# GNU General Public License for more details. |
3310 | +# |
3311 | +# You should have received a copy of the GNU General Public License |
3312 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
3313 | + |
3314 | +set -ex |
3315 | + |
3316 | +if [ "$(id -u)" -ne 0 ]; then |
3317 | + echo "ERROR: You have to run this script as root!" |
3318 | + exit 1 |
3319 | +fi |
3320 | + |
3321 | +SERIES=xenial |
3322 | +SOURCE_DIR= |
3323 | +RESULTS_DIR= |
3324 | +# Whenever you change the chroot in a way which needs a regeneration |
3325 | +# on the build server bump the version here. This will tell the |
3326 | +# job which updates the chroots to generate a new one. |
3327 | +CHROOT_VERSION=1 |
3328 | +BUILD_ARCH=amd64 |
3329 | +TARGET_ARCH=amd64 |
3330 | +UPDATE_CHROOT=false |
3331 | +PROXY= |
3332 | +CROSS_BUILD=false |
3333 | +SNAPCRAFT_EXTRA_ARGS= |
3334 | + |
3335 | +while [ -n "$1" ]; do |
3336 | + case "$1" in |
3337 | + --series=*) |
3338 | + SERIES=${1#*=} |
3339 | + shift |
3340 | + ;; |
3341 | + --source-dir=*) |
3342 | + SOURCE_DIR=${1#*=} |
3343 | + shift |
3344 | + ;; |
3345 | + --results-dir=*) |
3346 | + RESULTS_DIR=${1#*=} |
3347 | + shift |
3348 | + ;; |
3349 | + --arch=*) |
3350 | + TARGET_ARCH=${1#*=} |
3351 | + BUILD_ARCH=$TARGET_ARCH |
3352 | + shift |
3353 | + ;; |
3354 | + --update-chroot) |
3355 | + UPDATE_CHROOT=true |
3356 | + shift |
3357 | + ;; |
3358 | + --proxy=*) |
3359 | + PROXY=${1#*=} |
3360 | + shift |
3361 | + ;; |
3362 | + --cross-build) |
3363 | + CROSS_BUILD=true |
3364 | + shift |
3365 | + ;; |
3366 | + *) |
3367 | + echo "ERROR: Unknown options $1" |
3368 | + exit 1 |
3369 | + esac |
3370 | +done |
3371 | + |
3372 | +if [ -z "$SERIES" ]; then |
3373 | + echo "ERROR: No series specified" |
3374 | + exit 1 |
3375 | +fi |
3376 | + |
3377 | +# If we're cross-building we have to switch the architecture we're using |
3378 | +# for our build environment to match our host arch. |
3379 | +if [ "$CROSS_BUILD" = true ]; then |
3380 | + BUILD_ARCH=$(dpkg --print-architecture) |
3381 | +fi |
3382 | + |
3383 | +CHROOT_STORE_PATH=/build/chroots |
3384 | +CHROOT_TARBALL=$SERIES-$BUILD_ARCH-$CHROOT_VERSION-rootfs.tar |
3385 | + |
3386 | +if [ "$UPDATE_CHROOT" = true ]; then |
3387 | + if [ ! -e $CHROOT_STORE_PATH/$CHROOT_TARBALL ] ; then |
3388 | + mkdir -p /build/chroots |
3389 | + WORKDIR=$(mktemp -d) |
3390 | + mkdir -p $WORKDIR/rootfs |
3391 | + |
3392 | + DEBOOTSTRAP=debootstrap |
3393 | + DEB_REPO_URL= |
3394 | + case "$BUILD_ARCH" in |
3395 | + amd64) |
3396 | + DEB_REPO_URL="http://archive.ubuntu.com/ubuntu/" |
3397 | + ;; |
3398 | + armhf) |
3399 | + DEBOOTSTRAP=qemu-debootstrap |
3400 | + DEB_REPO_URL="http://ports.ubuntu.com/ubuntu-ports" |
3401 | + ;; |
3402 | + *) |
3403 | + echo "ERROR: Unsupported architecture $BUILD_ARCH" |
3404 | + exit 1 |
3405 | + ;; |
3406 | + esac |
3407 | + |
3408 | + cleanup() { |
3409 | + rm -rf $WORKDIR |
3410 | + } |
3411 | + |
3412 | + trap cleanup INT EXIT |
3413 | + |
3414 | + $DEBOOTSTRAP --components=main,universe --arch $BUILD_ARCH $SERIES $WORKDIR/rootfs |
3415 | + cat << EOF > $WORKDIR/rootfs/etc/apt/sources.list.d/updates.list |
3416 | +deb $DEB_REPO_URL xenial universe |
3417 | +deb $DEB_REPO_URL xenial-updates main universe |
3418 | +EOF |
3419 | + cat << EOF > $WORKDIR/rootfs/setup.sh |
3420 | +#!/bin/sh |
3421 | +set -ex |
3422 | +apt update |
3423 | +apt upgrade -y |
3424 | +apt install -y snapcraft |
3425 | +EOF |
3426 | + chmod +x $WORKDIR/rootfs/setup.sh |
3427 | + sudo chroot $WORKDIR/rootfs /setup.sh |
3428 | + rm $WORKDIR/rootfs/setup.sh |
3429 | + |
3430 | + (cd $WORKDIR/rootfs; tar cf $CHROOT_STORE_PATH/$CHROOT_TARBALL .) |
3431 | + rm -rf $WORKDIR |
3432 | + fi |
3433 | + |
3434 | + exit 0 |
3435 | +fi |
3436 | + |
3437 | +if [ -z "$SOURCE_DIR" ]; then |
3438 | + echo "ERROR: No source dir specified" |
3439 | + exit 1 |
3440 | +fi |
3441 | + |
3442 | +if [ -z "$RESULTS_DIR" ]; then |
3443 | + echo "ERROR: No results dir specified" |
3444 | + exit 1 |
3445 | +fi |
3446 | + |
3447 | +BUILDDIR=$(mktemp -d) |
3448 | + |
3449 | +cleanup() { |
3450 | + rm -rf $BUILDDIR |
3451 | +} |
3452 | + |
3453 | +trap cleanup INT EXIT |
3454 | + |
3455 | +if [ ! -e $CHROOT_STORE_PATH/$CHROOT_TARBALL ] ; then |
3456 | + echo "ERROR: Up to date chroot tarball doesn't exist. Please run the snap-build-update-chroot job!" |
3457 | + exit 1 |
3458 | +fi |
3459 | + |
3460 | +tar xf $CHROOT_STORE_PATH/$CHROOT_TARBALL -C $BUILDDIR |
3461 | + |
3462 | +cp -ra $SOURCE_DIR $BUILDDIR/src |
3463 | + |
3464 | +if [ "$CROSS_BUILD" = true ]; then |
3465 | + SNAPCRAFT_EXTRA_ARGS="$SNAPCRAFT_EXTRA_ARGS --target-arch=$TARGET_ARCH" |
3466 | +fi |
3467 | + |
3468 | +cat << EOF > $BUILDDIR/do-build.sh |
3469 | +#!/bin/sh |
3470 | +set -ex |
3471 | +apt update |
3472 | +apt upgrade -y |
3473 | +cd /src |
3474 | + |
3475 | +# snapcraft is pretty unhappy when LC_ALL and LANG aren't set |
3476 | +export LC_ALL=C.UTF-8 |
3477 | +export LANG=C.UTF-8 |
3478 | + |
3479 | +# To access certain things like the snap store we need a proxy in place |
3480 | +# in some environments |
3481 | +export http_proxy=$PROXY |
3482 | +export https_proxy=$PROXY |
3483 | + |
3484 | +snapcraft clean |
3485 | +snapcraft $SNAPCRAFT_EXTRA_ARGS |
3486 | +EOF |
3487 | +chmod +x $BUILDDIR/do-build.sh |
3488 | + |
3489 | +sudo chroot $BUILDDIR /do-build.sh |
3490 | diff --git a/tools/test-snap.sh b/tools/test-snap.sh |
3491 | new file mode 100755 |
3492 | index 0000000..4851645 |
3493 | --- /dev/null |
3494 | +++ b/tools/test-snap.sh |
3495 | @@ -0,0 +1,100 @@ |
3496 | +#!/bin/sh -ex |
3497 | +# |
3498 | +# Copyright (C) 2017 Canonical Ltd |
3499 | +# |
3500 | +# This program is free software: you can redistribute it and/or modify |
3501 | +# it under the terms of the GNU General Public License version 3 as |
3502 | +# published by the Free Software Foundation. |
3503 | +# |
3504 | +# This program is distributed in the hope that it will be useful, |
3505 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
3506 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
3507 | +# GNU General Public License for more details. |
3508 | +# |
3509 | +# You should have received a copy of the GNU General Public License |
3510 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
3511 | + |
3512 | +# Import used functions |
3513 | +. "$WORKSPACE"/build-scripts/scripts/hardware-test.sh |
3514 | + |
3515 | +# Runs snap tests |
3516 | +run_snap_tests () |
3517 | +{ |
3518 | + if [ -n "$RESULTS_ID" ]; then |
3519 | + $SSH mkdir -p "$REMOTE_WORKSPACE"/results |
3520 | + $SSH cp -v "$REMOTE_RESULTS_BASE_DIR"/"$RESULTS_ID"/*.snap "$REMOTE_WORKSPACE"/results |
3521 | + fi |
3522 | + |
3523 | + $SSH sudo apt-get --yes --force-yes install docker.io |
3524 | + |
3525 | + cat << EOF > "$WORKSPACE"/run-tests.sh |
3526 | +#!/bin/sh |
3527 | +set -ex |
3528 | + |
3529 | +export TERM=linux |
3530 | +export DEBIAN_FRONTEND=noninteractive |
3531 | +export PATH=/build/bin:$PATH |
3532 | + |
3533 | +# At this time it's necessary to build spread manually because |
3534 | +# the snapped version does not include the qemu/kvm backend. |
3535 | +# Once the snapped version includes this backend, then we can |
3536 | +# change the manual building of spread with making sure the snap |
3537 | +# package is installed. |
3538 | +export GOPATH=$(mktemp -d) |
3539 | +go get -d -v github.com/snapcore/spread/... |
3540 | +go build github.com/snapcore/spread/cmd/spread |
3541 | +mkdir /build/bin |
3542 | +cp spread /build/bin |
3543 | + |
3544 | +git clone --depth 1 -b $BRANCH $REPO /build/src |
3545 | +cd /build/src |
3546 | + |
3547 | +# Copy any stage results from previous generic-build-snap-worker builds |
3548 | +if [ "\$(find /build/results -path ./misc -prune -o -name '*.snap' -print | wc -l)" -gt 0 ]; then |
3549 | + cp -v /build/results/*.snap /build/src |
3550 | +fi |
3551 | + |
3552 | +if [ -e "run-tests.sh" ] ; then |
3553 | + EXTRA_ARGS= |
3554 | + if [ -n "$CHANNEL" ] ; then |
3555 | + # If CHANNEL is specified we use that channel for image construction and |
3556 | + # also load the snap from that channel for testing. |
3557 | + EXTRA_ARGS="--test-from-channel=$CHANNEL" |
3558 | + fi |
3559 | + if [ -n "$CORE_CHANNEL" ]; then |
3560 | + EXTRA_ARGS="--channel=$CORE_CHANNEL" |
3561 | + fi |
3562 | + |
3563 | + ./run-tests.sh --force-new-image \$EXTRA_ARGS |
3564 | +else |
3565 | + if [ ! -z "$CHANNEL" ] ; then |
3566 | + SNAP_CHANNEL=$CHANNEL spread -v |
3567 | + else |
3568 | + spread -v |
3569 | + fi |
3570 | +fi |
3571 | +EOF |
3572 | + |
3573 | + $SSH mkdir -p "$REMOTE_WORKSPACE" |
3574 | + $SCP "$WORKSPACE"/run-tests.sh "$REMOTE_USER"@"$REMOTE_WORKER":"$REMOTE_WORKSPACE" |
3575 | + $SSH chmod u+x "$REMOTE_WORKSPACE"/run-tests.sh |
3576 | + |
3577 | + $SSH mkdir -p "$REMOTE_WORKSPACE"/docker |
3578 | + $SCP "$WORKSPACE"/build-scripts/docker/spread-tests/Dockerfile \ |
3579 | + "$REMOTE_USER"@"$REMOTE_WORKER":"$REMOTE_WORKSPACE"/docker |
3580 | + $SSH time sudo docker build -t snap-spread-tests "$REMOTE_WORKSPACE"/docker |
3581 | + |
3582 | + $SSH time sudo docker run \ |
3583 | + --rm \ |
3584 | + -v /dev:/dev \ |
3585 | + -v "$REMOTE_WORKSPACE":/build \ |
3586 | + --privileged \ |
3587 | + snap-spread-tests /build/run-tests.sh |
3588 | + |
3589 | + $SSH sudo rm -rf "$REMOTE_WORKSPACE" |
3590 | + |
3591 | + # Now run tests on real hardware if defined in the backend |
3592 | + if [ "$HW_TESTS_RESULT" -eq 0 ] ; then |
3593 | + run_hardware_tests |
3594 | + fi |
3595 | +} |
3596 | diff --git a/tools/trigger-ci.py b/tools/trigger-ci.py |
3597 | new file mode 100755 |
3598 | index 0000000..d50c021 |
3599 | --- /dev/null |
3600 | +++ b/tools/trigger-ci.py |
3601 | @@ -0,0 +1,270 @@ |
3602 | +#!/usr/bin/env python |
3603 | +# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*- |
3604 | +# |
3605 | +# Copyright (C) 2016 Canonical Ltd |
3606 | +# |
3607 | +# This program is free software: you can redistribute it and/or modify |
3608 | +# it under the terms of the GNU General Public License version 3 as |
3609 | +# published by the Free Software Foundation. |
3610 | +# |
3611 | +# This program is distributed in the hope that it will be useful, |
3612 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
3613 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
3614 | +# GNU General Public License for more details. |
3615 | +# |
3616 | +# You should have received a copy of the GNU General Public License |
3617 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
3618 | + |
3619 | +from launchpadlib.launchpad import Launchpad |
3620 | +import jenkins |
3621 | +import os |
3622 | +import sys |
3623 | +import yaml |
3624 | +import re |
3625 | +from jlp import launchpadutils |
3626 | +from jlp import jenkinsutils |
3627 | + |
3628 | +from argparse import ArgumentParser |
3629 | + |
3630 | +import se_utils |
3631 | + |
3632 | +parser = ArgumentParser(description="Trigger snap builds for pending merge proposals") |
3633 | +parser.add_argument('-p', '--project', required=True, |
3634 | + help="Launchpad project to check for new merge-proposals") |
3635 | +parser.add_argument('-j', '--job', required=True, |
3636 | + help="Jenkins job to trigger") |
3637 | +parser.add_argument('-t', '--team', required=True, |
3638 | + help="Launchpad team for the project") |
3639 | + |
3640 | +args = vars(parser.parse_args()) |
3641 | + |
3642 | +lp_app = se_utils.get_config_option("lp_app") |
3643 | +lp_env = se_utils.get_config_option("lp_env") |
3644 | +credential_store_path = se_utils.get_config_option('credential_store_path') |
3645 | +launchpad = se_utils.get_launchpad(None, credential_store_path, lp_app, lp_env) |
3646 | + |
3647 | +project = launchpad.projects[args['project']] |
3648 | +team = args['team'] |
3649 | + |
3650 | +proposals = project.getMergeProposals() |
3651 | + |
3652 | +def load_config(): |
3653 | + files = [os.path.expanduser('~/.jlp/jlp.config'), 'jlp.config'] |
3654 | + for config_file in files: |
3655 | + try: |
3656 | + config = yaml.safe_load(open(config_file, 'r')) |
3657 | + return config |
3658 | + except IOError: |
3659 | + pass |
3660 | + print("ERROR: No config file found") |
3661 | + sys.exit(1) |
3662 | + |
3663 | +def get_config_option(name): |
3664 | + config = load_config() |
3665 | + return config[name] |
3666 | + |
3667 | +j = jenkins.Jenkins(get_config_option('jenkins_url'), |
3668 | + get_config_option('jenkins_user'), |
3669 | + get_config_option('jenkins_password')) |
3670 | + |
3671 | +jenkins_job = args['job'] |
3672 | +jenkins_build_token = get_config_option('jenkins_build_token') |
3673 | + |
3674 | +job_info = j.get_job_info(jenkins_job) |
3675 | +if not job_info['buildable']: |
3676 | + print("ERROR: Job is not buildable (propably disabled)") |
3677 | + sys.exit(1) |
3678 | + |
3679 | + |
3680 | +def get_latest_revision(mp): |
3681 | + """Return the latest revision of the given merge proposal. |
3682 | + |
3683 | + :param mp: handle to merge proposal |
3684 | + """ |
3685 | + if '+git' in mp.web_link: |
3686 | + for ref in mp.source_git_repository.refs_collection: |
3687 | + if ref.path == mp.source_git_path: |
3688 | + return ref.commit_sha1 |
3689 | + return str(0) |
3690 | + else: |
3691 | + return mp.source_branch.revision_count |
3692 | + |
3693 | +def clean_branch_name(branch_name): |
3694 | + if branch_name.startswith("refs/heads/"): |
3695 | + return branch_name[11:] |
3696 | + return branch_name |
3697 | + |
3698 | +def series_from_branch_name(branch_name): |
3699 | + if branch_name.startswith("vivid/"): |
3700 | + return "vivid" |
3701 | + return "xenial" |
3702 | + |
3703 | +def get_latest_revision(mp): |
3704 | + """Return the latest revision of the given merge proposal. |
3705 | + |
3706 | + :param mp: handle to merge proposal |
3707 | + """ |
3708 | + if '+git' in mp.web_link: |
3709 | + for ref in mp.source_git_repository.refs_collection: |
3710 | + if ref.path == mp.source_git_path: |
3711 | + return ref.commit_sha1 |
3712 | + return str(0) |
3713 | + else: |
3714 | + return mp.source_branch.revision_count |
3715 | + |
3716 | +def get_review_revision_regex(mp): |
3717 | + if '+git' in mp.web_link: |
3718 | + return '^(PASSED|FAILED): Continuous integration, rev:([0-9a-f]+)' |
3719 | + else: |
3720 | + return '^(PASSED|FAILED): Continuous integration, rev:(\d+)' |
3721 | + |
3722 | +def get_latest_review(launchpad_user, mp): |
3723 | + """Return the latest revision reviewed by the given launchpad_user. |
3724 | + |
3725 | + This function expects a review comment in the following format: |
3726 | + '^(PASSED|FAILED): Continuous integration, rev:(\d+)' |
3727 | + |
3728 | + :param launchpad_user: handle to launchpad user |
3729 | + :param mp: handle to merge proposal |
3730 | + """ |
3731 | + revision = 0 |
3732 | + launchpad_review_type = get_config_option('launchpad_review_type') |
3733 | + for comment in mp.all_comments: |
3734 | + if comment.author.name == launchpad_user.name: |
3735 | + if comment.vote_tag == launchpad_review_type: |
3736 | + m = re.search( |
3737 | + get_review_revision_regex(mp), |
3738 | + comment.message_body) |
3739 | + if m: |
3740 | + revision = m.group(2) |
3741 | + return revision |
3742 | + |
3743 | +def latest_candidate_validated(launchpad_user, mp): |
3744 | + """Return if the latest candidate revision of the merge proposal is |
3745 | + validated. |
3746 | + |
3747 | + :param launchpad_user: handle to launchpad user used to validate the |
3748 | + merge proposals |
3749 | + :param mp: handle to merge proposal |
3750 | + """ |
3751 | + |
3752 | + latest_review = get_latest_review(launchpad_user, mp) |
3753 | + print 'Latest review is revision: ' + str(latest_review) |
3754 | + latest_revision = get_latest_revision(mp) |
3755 | + print 'Latest revision is: ' + str(latest_revision) |
3756 | + if latest_review == latest_revision: |
3757 | + print 'Skipping this MP. Current revision: ' + str(latest_revision) |
3758 | + return True |
3759 | + return False |
3760 | + |
3761 | +# NOTE: We can't use the testing_in_progress method from jlp as it checks for |
3762 | +# the lower case parameter 'merge_proposal' of the build job and ours is using |
3763 | +# the upper case variant 'MERGE_PROPOSAL'. |
3764 | +def testing_in_progress(mp, jenkins_job): |
3765 | + try: |
3766 | + if jenkinsutils.is_job_or_downstream_building( |
3767 | + jenkins_job, job_params={'MERGE_PROPOSAL': mp.web_link}): |
3768 | + print('Skipping this MP. It is currently being tested by Jenkins.') |
3769 | + return True |
3770 | + except: |
3771 | + print('Failed to check if MP is already building') |
3772 | + return False |
3773 | + |
3774 | +# Copied over from lp:jenkins-launchpad-plugin |
3775 | +def users_in_team(users, team): |
3776 | + """Determine whether any of these users are in the supplied team. |
3777 | + |
3778 | + :param users: The users which may be members of the supplied team. |
3779 | + :param team: The team which users may be part of. |
3780 | + :return: True if any of the users are members of the team, otherwise |
3781 | + False. |
3782 | + """ |
3783 | + for member in team.participants: |
3784 | + if member in users: |
3785 | + return True |
3786 | + else: |
3787 | + return False |
3788 | + |
3789 | +# Copied over from lp:jenkins-launchpad-plugin and slightly modified |
3790 | +def users_allowed_to_trigger_jobs(lp_users, allowed_people): |
3791 | + """Returns if an of the given users is allowed to run jobs on jenkins. |
3792 | + |
3793 | + This is to avoid random people to run jobs on our internal infrastructure. |
3794 | + A user is allowed if they are either directly in the ALLOWED_USERS list or |
3795 | + are member of a team in that list. |
3796 | + |
3797 | + :param lp_users: launchpad user handles |
3798 | + """ |
3799 | + if len(lp_users) == 0: |
3800 | + return False |
3801 | + for lp_user in lp_users: |
3802 | + if lp_user.name in allowed_people: |
3803 | + return True |
3804 | + lp = lp_users[0]._root |
3805 | + for allowed in allowed_people: |
3806 | + try: |
3807 | + allowed_person = lp.people[allowed] |
3808 | + except KeyError: |
3809 | + logger.warn('User {} from the allowed_users list is not in ' |
3810 | + 'launchpad!'.format(allowed)) |
3811 | + continue |
3812 | + if not allowed_person.is_team: |
3813 | + continue |
3814 | + if users_in_team(lp_users, allowed_person): |
3815 | + return True |
3816 | + logger.debug('Users "' + ', '.join(u.name for u in lp_users) + |
3817 | + '" not allowed to trigger jobs') |
3818 | + return False |
3819 | + |
3820 | +project_blacklist = [] |
3821 | + |
3822 | +for proposal in proposals: |
3823 | + launchpad_user = launchpad.people(get_config_option('launchpad_login')) |
3824 | + |
3825 | + print("Checking proposal %s .." % proposal.web_link) |
3826 | + |
3827 | + # Ignore certain projects which don't build here as they are |
3828 | + # fetching source from somewhere else. |
3829 | + ignore = False |
3830 | + for project in project_blacklist: |
3831 | + if project in proposal.web_link: |
3832 | + ignore = True |
3833 | + break |
3834 | + |
3835 | + if ignore: |
3836 | + print "Ignoring %s" % proposal.web_link |
3837 | + continue |
3838 | + |
3839 | + if not users_allowed_to_trigger_jobs([proposal.registrant], [team]): |
3840 | + continue |
3841 | + |
3842 | + if latest_candidate_validated(launchpad_user, proposal): |
3843 | + continue |
3844 | + |
3845 | + if testing_in_progress(proposal, jenkins_job): |
3846 | + continue |
3847 | + |
3848 | + target_repo = launchpad.load(proposal.target_git_repository_link) |
3849 | + source_repo = launchpad.load(proposal.source_git_repository_link) |
3850 | + |
3851 | + target_git_url = target_repo.git_https_url |
3852 | + if target_git_url == None: |
3853 | + target_git_url = target_repo.git_ssh_url |
3854 | + |
3855 | + source_git_url = source_repo.git_https_url |
3856 | + if source_git_url == None: |
3857 | + source_git_url = source_repo.git_ssh_url |
3858 | + |
3859 | + target_branch = clean_branch_name(proposal.target_git_path) |
3860 | + jenkins_params = { |
3861 | + 'TARGET_GIT_REPO': target_git_url, |
3862 | + 'TARGET_GIT_REPO_BRANCH': target_branch, |
3863 | + 'SOURCE_GIT_REPO': source_git_url, |
3864 | + 'SOURCE_GIT_REPO_BRANCH': clean_branch_name(proposal.source_git_path), |
3865 | + 'MERGE_PROPOSAL': proposal.web_link, |
3866 | + 'REVISION': get_latest_revision(proposal), |
3867 | + 'SERIES': series_from_branch_name(target_branch), |
3868 | + } |
3869 | + |
3870 | + print("Triggering build job for %s" % proposal.web_link) |
3871 | + j.build_job(jenkins_job, jenkins_params, jenkins_build_token) |
3872 | diff --git a/tools/trigger-lp-build.py b/tools/trigger-lp-build.py |
3873 | new file mode 100755 |
3874 | index 0000000..5434281 |
3875 | --- /dev/null |
3876 | +++ b/tools/trigger-lp-build.py |
3877 | @@ -0,0 +1,230 @@ |
3878 | +#! /usr/bin/python |
3879 | + |
3880 | +import os |
3881 | +import sys |
3882 | +import time |
3883 | +import random |
3884 | +import smtplib |
3885 | +import string |
3886 | +import urllib2 |
3887 | +import zlib |
3888 | + |
3889 | +from datetime import datetime |
3890 | +from os.path import basename |
3891 | +from launchpadlib.launchpad import Launchpad |
3892 | + |
3893 | +from argparse import ArgumentParser |
3894 | + |
3895 | +import se_utils |
3896 | + |
3897 | +parser = ArgumentParser(description="Build a specific snap on launchpad") |
3898 | +parser.add_argument('-s', '--snap', required=True, |
3899 | + help="Name of the snap to build") |
3900 | +parser.add_argument('-p', '--publish', action='store_true', |
3901 | + help="Trigger a publish build instead of a daily (default)") |
3902 | +parser.add_argument('-n', '--new', action='store_true', help="Create a new ephemeral snap build on launchpad") |
3903 | +parser.add_argument('--git-repo', help="Git repository to be used for new ephemeral snap build") |
3904 | +parser.add_argument('--git-repo-branch', help="Git repository branch to be used for new ephemeral snap build") |
3905 | +parser.add_argument('-a', '--architectures', help="Specify architectures to build for. Separate multiple architectures by ','") |
3906 | +parser.add_argument('-r', '--results-dir', help="Specify where results should be saved") |
3907 | + |
3908 | +args = vars(parser.parse_args()) |
3909 | + |
3910 | +ephemeral_build=False |
3911 | +results_dir=os.path.join(os.getcwd(), "results") |
3912 | + |
3913 | +if 'results_dir' in args: |
3914 | + results_dir=args['results_dir'] |
3915 | + |
3916 | +if args['new']: |
3917 | + ephemeral_build=True |
3918 | + if args['git_repo'] == None or args['git_repo_branch'] == None: |
3919 | + print("ERROR: No git repository or a branch supplied") |
3920 | + sys.exit(1) |
3921 | + |
3922 | +series = 'xenial' |
3923 | + |
3924 | +lp_app = se_utils.get_config_option("lp_app") |
3925 | +lp_env = se_utils.get_config_option("lp_env") |
3926 | +credential_store_path = se_utils.get_config_option('credential_store_path') |
3927 | +launchpad = se_utils.get_launchpad(None, credential_store_path, lp_app, lp_env) |
3928 | + |
3929 | +team = launchpad.people['snappy-hwe-team'] |
3930 | +ubuntu = launchpad.distributions['ubuntu'] |
3931 | +release = ubuntu.getSeries(name_or_version=series) |
3932 | +primary_archive = ubuntu.getArchive(name='primary') |
3933 | + |
3934 | +snap=None |
3935 | +if ephemeral_build: |
3936 | + snap_arches=[] |
3937 | + if 'architectures' in args and args['architectures'] != None: |
3938 | + snap_arches = args["architectures"].split(",") |
3939 | + |
3940 | + if len(snap_arches) == 0: |
3941 | + print("WARNING: No architectures to build specified. Will only build for amd64.") |
3942 | + snap_arches=["amd64"] |
3943 | + |
3944 | + processors=[] |
3945 | + for arch in snap_arches: |
3946 | + try: |
3947 | + p = launchpad.processors.getByName(name=arch) |
3948 | + processors.append(p.self_link) |
3949 | + except: |
3950 | + print("ERROR: Failed to find processor for '{}' architecture".format(arch)) |
3951 | + sys.exit(1) |
3952 | + |
3953 | + build_name = 'ci-%s-%s' % (args["snap"], ''.join(random.choice(string.ascii_lowercase + string.digits) for _ in range(16))) |
3954 | + snap = launchpad.snaps.new(name=build_name, |
3955 | + processors=processors, |
3956 | + auto_build=False, distro_series=release, |
3957 | + git_repository_url=args['git_repo'], |
3958 | + git_path='%s' % args["git_repo_branch"], |
3959 | + owner=team) |
3960 | +else: |
3961 | + build_name = "%s-daily" % args["snap"] |
3962 | + if args["publish"] == True: |
3963 | + build_name = "%s-publish" % args["snap"] |
3964 | + snap = launchpad.snaps.getByName(name=build_name, owner=team) |
3965 | + |
3966 | +if snap == None: |
3967 | + print("ERROR: Failed to create snap build on launchpad") |
3968 | + |
3969 | +# Not every snap is build agains all arches. |
3970 | +arches = [processor.name for processor in snap.processors] |
3971 | +if not ephemeral_build and args['architectures'] != None: |
3972 | + wanted_arches = args["architectures"].split(",") |
3973 | + possible_arches = [] |
3974 | + for arch in wanted_arches: |
3975 | + if not arch in arches: |
3976 | + print("WARNING: Can't build snap for architecture {} as it is not enabled in the build job".format(args["snap"])) |
3977 | + continue |
3978 | + possible_arches.append(arch) |
3979 | + arches = possible_arches |
3980 | + |
3981 | +if len(arches) == 0: |
3982 | + print("ERROR: No architectures available to build for") |
3983 | + sys.exit(1) |
3984 | + |
3985 | +# Add a big fat warning that we don't really care about fixing things when |
3986 | +# the job will be canceled after the following lines are printed out. |
3987 | +print("!!!!!!! POINT OF NO RETURN !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!") |
3988 | +print("DO NOT CANCEL THIS JOB AFTER THIS OR BAD THINGS WILL HAPPEN") |
3989 | +print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!") |
3990 | + |
3991 | +stamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S') |
3992 | +print("Trying to trigger builds at: {}".format(stamp)) |
3993 | + |
3994 | +# sometimes we see error such as "u'Unknown architecture lpia for ubuntu xenial'" |
3995 | +# and in order to workaround let's validate the arches agains set of valid |
3996 | +# architectures that the snap can choose from |
3997 | + |
3998 | + |
3999 | +# We will now trigger a build for each whitelisted architecture, collect the |
4000 | +# build job url and the wait for all builds to finish and collect their results |
4001 | +# to vote for a successful or failed build. |
4002 | +triggered_builds = [] |
4003 | +triggered_build_urls = {} |
4004 | +valid_arches = ['armhf', 'i386', 'amd64', 'arm64', 's390x', 'powerpc', 'ppc64el'] |
4005 | +for build_arch in arches: |
4006 | + # sometimes we see error such as "u'Unknown architecture lpia for |
4007 | + # ubuntu xenial'" and in order to workaround let's validate the arches |
4008 | + # agains set of valid architectures that the snap can choose from |
4009 | + if build_arch not in valid_arches: |
4010 | + print("WARNING: Can't build snap for architecture {} as it is not enabled in the build job".format(args["snap"])) |
4011 | + continue |
4012 | + |
4013 | + arch = release.getDistroArchSeries(archtag=build_arch) |
4014 | + request = snap.requestBuild(archive=primary_archive, distro_arch_series=arch, pocket='Proposed') |
4015 | + build_id = str(request).rsplit('/', 1)[-1] |
4016 | + triggered_builds.append(build_id) |
4017 | + triggered_build_urls[build_id] = request.self_link |
4018 | + print("Arch: {} is building under: {}".format(build_arch, request.self_link)) |
4019 | + |
4020 | +failures = [] |
4021 | +successful = [] |
4022 | +while len(triggered_builds): |
4023 | + for build in triggered_builds: |
4024 | + try: |
4025 | + response = snap.getBuildSummariesForSnapBuildIds(snap_build_ids=[build]) |
4026 | + except: |
4027 | + print("Could not get response for {} (was there an LP timeout?)".format(build)) |
4028 | + continue |
4029 | + status = response[build]['status'] |
4030 | + if status == "FULLYBUILT": |
4031 | + successful.append(build) |
4032 | + triggered_builds.remove(build) |
4033 | + continue |
4034 | + elif status == "FAILEDTOBUILD": |
4035 | + failures.append(build) |
4036 | + triggered_builds.remove(build) |
4037 | + continue |
4038 | + elif status == "CANCELLED": |
4039 | + print("INFO: {} snap build was canceled for id: {}".format(args["snap"], build)) |
4040 | + triggered_builds.remove(build) |
4041 | + continue |
4042 | + if len(triggered_builds) > 0: |
4043 | + time.sleep(60) |
4044 | + |
4045 | +if len(failures): |
4046 | + for failure in failures: |
4047 | + try: |
4048 | + response = snap.getBuildSummariesForSnapBuildIds(snap_build_ids=[failure]) |
4049 | + except: |
4050 | + print ("Could not get failure data for {} (was there an LP timeout?)".format(build)) |
4051 | + continue |
4052 | + |
4053 | + if not failure in response: |
4054 | + print("Launchpad didn't returned us the snap build summary we ask it for!?") |
4055 | + continue |
4056 | + |
4057 | + build_summary = response[failure] |
4058 | + arch = 'unknown' |
4059 | + buildlog = None |
4060 | + if 'build_log_url' in build_summary: |
4061 | + buildlog = build_summary['build_log_url'] |
4062 | + |
4063 | + if buildlog != None and len(buildlog) > 0: |
4064 | + parts = arch = str(buildlog).split('_') |
4065 | + if len(parts) >= 4: |
4066 | + arch = parts[4] |
4067 | + elif buildlog == None: |
4068 | + buildlog = 'not available' |
4069 | + |
4070 | + print("INFO: {} snap {} build at {} failed for id: {} log: {}".format(args["snap"], arch, stamp, failure, buildlog)) |
4071 | + |
4072 | + # For ephermal builds we need to print out the log file as it will be gone after |
4073 | + # the launchpad build is removed. |
4074 | + if ephemeral_build and buildlog != None: |
4075 | + response = urllib2.urlopen(buildlog) |
4076 | + log_data = zlib.decompress(response.read(), 16+zlib.MAX_WBITS) |
4077 | + print(log_data) |
4078 | + |
4079 | +# Fetch build results for successful builds and store those in the output |
4080 | +# directory so that the caller can reuse them. |
4081 | +if len(successful): |
4082 | + for success in successful: |
4083 | + try: |
4084 | + snap_build = launchpad.load(triggered_build_urls[success]) |
4085 | + urls = snap_build.getFileUrls() |
4086 | + if len(urls): |
4087 | + for u in urls: |
4088 | + print("Downloading snap from %s ..." % u) |
4089 | + response = urllib2.urlopen(u) |
4090 | + if not os.path.exists(results_dir): |
4091 | + os.makedirs(results_dir) |
4092 | + path = os.path.join(results_dir, os.path.basename(u)) |
4093 | + with open(path, "w") as out_file: |
4094 | + out_file.write(response.read()) |
4095 | + except: |
4096 | + print ("Could not retrieve snap build data for {} (was there an LP timeout?)".format(build)) |
4097 | + continue |
4098 | + |
4099 | + |
4100 | +if ephemeral_build: |
4101 | + snap.lp_delete() |
4102 | + |
4103 | +if len(failures): |
4104 | + # Let the build fail as at least a single snap has failed to build |
4105 | + sys.exit(1) |
4106 | + |
4107 | +print("Done!") |
4108 | diff --git a/tools/vote-on-merge-proposal.py b/tools/vote-on-merge-proposal.py |
4109 | new file mode 100755 |
4110 | index 0000000..bec6da5 |
4111 | --- /dev/null |
4112 | +++ b/tools/vote-on-merge-proposal.py |
4113 | @@ -0,0 +1,271 @@ |
4114 | +#!/usr/bin/env python |
4115 | +# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*- |
4116 | +# |
4117 | +# Copyright (C) 2016 Canonical Ltd |
4118 | +# |
4119 | +# This program is free software: you can redistribute it and/or modify |
4120 | +# it under the terms of the GNU General Public License version 3 as |
4121 | +# published by the Free Software Foundation. |
4122 | +# |
4123 | +# This program is distributed in the hope that it will be useful, |
4124 | +# but WITHOUT ANY WARRANTY; without even the implied warranty of |
4125 | +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
4126 | +# GNU General Public License for more details. |
4127 | +# |
4128 | +# You should have received a copy of the GNU General Public License |
4129 | +# along with this program. If not, see <http://www.gnu.org/licenses/>. |
4130 | + |
4131 | +import atexit |
4132 | +import sys |
4133 | +import time |
4134 | +import logging |
4135 | +import os |
4136 | +import yaml |
4137 | +import re |
4138 | +from shutil import rmtree |
4139 | +from argparse import ArgumentParser |
4140 | +from launchpadlib.credentials import RequestTokenAuthorizationEngine |
4141 | +from lazr.restfulclient.errors import HTTPError |
4142 | +from launchpadlib.launchpad import Launchpad |
4143 | +from launchpadlib.credentials import UnencryptedFileCredentialStore |
4144 | +from jlp import get_config_option |
4145 | +from jlp import launchpadutils, jenkinsutils, logger |
4146 | + |
4147 | +logger = logging.getLogger('jenkins-launchpad-plugin') |
4148 | +stdout_handler = logging.StreamHandler(stream=sys.stdout) |
4149 | +formatter = logging.Formatter('%(levelname)s: %(message)s') |
4150 | +stdout_handler.setFormatter(formatter) |
4151 | +logger.addHandler(stdout_handler) |
4152 | + |
4153 | +parser = ArgumentParser(description="Vote on a Launchpad merge proposal.") |
4154 | +parser.add_argument('-s', '--status') |
4155 | +parser.add_argument('-u', '--build-url', required=True, |
4156 | + help="URL of the Jenkins job") |
4157 | +parser.add_argument('-p', '--merge-proposal', required=True, |
4158 | + help="URL of the merge proposal to update") |
4159 | +parser.add_argument('-r', '--revision', required=True, |
4160 | + help="merge proposal candidate revision") |
4161 | + |
4162 | +args = vars(parser.parse_args()) |
4163 | + |
4164 | +ACCESS_TOKEN_POLL_TIME = 10 |
4165 | +WAITING_FOR_USER = """Open this link: |
4166 | +{} |
4167 | +to authorize this program to access Launchpad on your behalf. |
4168 | +Waiting to hear from Launchpad about your decision. . . .""" |
4169 | + |
4170 | + |
4171 | +class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine): |
4172 | + """Authorize a token in a server environment (with no browser). |
4173 | + |
4174 | + Print a link for the user to copy-and-paste into his/her browser |
4175 | + for authentication. |
4176 | + """ |
4177 | + |
4178 | + def __init__(self, *args, **kwargs): |
4179 | + # as implemented in AuthorizeRequestTokenWithBrowser |
4180 | + kwargs['consumer_name'] = None |
4181 | + kwargs.pop('allow_access_levels', None) |
4182 | + super(AuthorizeRequestTokenWithConsole, self).__init__(*args, **kwargs) |
4183 | + |
4184 | + def make_end_user_authorize_token(self, credentials, request_token): |
4185 | + """Ask the end-user to authorize the token in their browser. |
4186 | + |
4187 | + """ |
4188 | + authorization_url = self.authorization_url(request_token) |
4189 | + print WAITING_FOR_USER.format(authorization_url) |
4190 | + # if we don't flush we may not see the message |
4191 | + sys.stdout.flush() |
4192 | + while credentials.access_token is None: |
4193 | + time.sleep(ACCESS_TOKEN_POLL_TIME) |
4194 | + try: |
4195 | + credentials.exchange_request_token_for_access_token( |
4196 | + self.web_root) |
4197 | + break |
4198 | + except HTTPError, e: |
4199 | + if e.response.status == 403: |
4200 | + # The user decided not to authorize this |
4201 | + # application. |
4202 | + raise e |
4203 | + elif e.response.status == 401: |
4204 | + # The user has not made a decision yet. |
4205 | + pass |
4206 | + else: |
4207 | + # There was an error accessing the server. |
4208 | + raise e |
4209 | + |
4210 | + |
4211 | +def get_launchpad(launchpadlib_dir=None): |
4212 | + """ return a launchpad API class. In case launchpadlib_dir is |
4213 | + specified used that directory to store launchpadlib cache instead of |
4214 | + the default """ |
4215 | + store = UnencryptedFileCredentialStore( |
4216 | + get_config_option('credential_store_path')) |
4217 | + lp_app = get_config_option('lp_app') |
4218 | + lp_env = get_config_option('lp_env') |
4219 | + authorization_engine = AuthorizeRequestTokenWithConsole(lp_env, lp_app) |
4220 | + return Launchpad.login_with(lp_app, lp_env, |
4221 | + credential_store=store, |
4222 | + authorization_engine=authorization_engine, |
4223 | + launchpadlib_dir=launchpadlib_dir, |
4224 | + version='devel') |
4225 | + |
4226 | +def get_branch_handle_from_url(lp_handle, url): |
4227 | + """ Return a branch/repo handle for the given url. |
4228 | + Returns a launchpad branch or git repository handle for the given url. |
4229 | + :param lp_handle: launchpad API handle/instance |
4230 | + :param url: url of the branch or git repository |
4231 | + """ |
4232 | + if '+git' in url: |
4233 | + name = url.replace('https://code.launchpad.net/', '') |
4234 | + logger.debug('fetching repo: ' + name) |
4235 | + try: |
4236 | + return lp_handle.git_repositories.getByPath(path=name) |
4237 | + except AttributeError: |
4238 | + logger.debug('git_repositories.getByPath was not found. You may need to set lp_version=devel in the config') |
4239 | + return None |
4240 | + else: |
4241 | + name = url.replace('https://code.launchpad.net/', 'lp:') |
4242 | + name = name.replace('https://code.staging.launchpad.net/', 'lp://staging/') |
4243 | + logger.debug('fetching branch: ' + name) |
4244 | + return lp_handle.branches.getByUrl(url=name) |
4245 | + |
4246 | +def get_branch_from_mp(merge_proposal): |
4247 | + """Return a link to branch given a link to a merge proposal. |
4248 | + |
4249 | + If merge_proposal is: |
4250 | + https://copde.launchpad.net/~user/project/name/+merge/12345 |
4251 | + then the result will be: |
4252 | + https://copde.launchpad.net/~user/project/name/ |
4253 | + |
4254 | + :param merge_proposal: url of a launchpad merge proposal |
4255 | + """ |
4256 | + m = re.search('(.*)\+merge/[0-9]+$', merge_proposal) |
4257 | + if m: |
4258 | + return m.group(1) |
4259 | + return None |
4260 | + |
4261 | +def get_mp_handle_from_url(lp_handle, merge_proposal_link): |
4262 | + """ Get launchpad handle for merge proposal given a merge proposal URL. |
4263 | + |
4264 | + Returns None in case the merge proposal can't be found. |
4265 | + :param merge_proposal_link: URL of the merge proposal |
4266 | + """ |
4267 | + branch_link = get_branch_from_mp(merge_proposal_link) |
4268 | + if not branch_link: |
4269 | + logger.error('Unable to get branch link from merge proposal link.') |
4270 | + return None |
4271 | + |
4272 | + branch = get_branch_handle_from_url(lp_handle, branch_link) |
4273 | + if not branch: |
4274 | + logger.debug('Branch {} does not exist'.format(branch_link)) |
4275 | + return None |
4276 | + |
4277 | + logger.debug('mp_link: {}.'.format(merge_proposal_link)) |
4278 | + |
4279 | + for mp in branch.landing_targets: |
4280 | + logger.debug('mp.web_link: {}'.format(mp.web_link)) |
4281 | + if mp.web_link == merge_proposal_link: |
4282 | + return mp |
4283 | + |
4284 | + return None |
4285 | + |
4286 | +class LaunchpadVote(): |
4287 | + APPROVE = 'Approve' |
4288 | + DISAPPROVE = 'Disapprove' |
4289 | + NEEDS_FIXING = 'Needs Fixing' |
4290 | + |
4291 | +def get_vote_subject(mp): |
4292 | + """Given a mp handle return a subject for the vote message |
4293 | + |
4294 | + Unfortunately there is no method in the API that gives you the "standard" |
4295 | + subject that launchapd is using and some email clients (gmail) are |
4296 | + grouping conversations into threads based on subject. |
4297 | + |
4298 | + This returns what seems to be the launchpad way of doing subjects. |
4299 | + :param mp: launchpad merge proposal handle |
4300 | + """ |
4301 | + |
4302 | + if '+git' in mp.web_link: |
4303 | + source = mp.source_git_repository.display_name.replace('lp:', '') + \ |
4304 | + ':' + \ |
4305 | + mp.source_git_path.replace('refs/heads/', '') |
4306 | + target = mp.target_git_repository.display_name.replace('lp:', '') + \ |
4307 | + ':' + \ |
4308 | + mp.target_git_path.replace('refs/heads/', '') |
4309 | + return 'Re: [Merge] {} into {}'.format(source, target) |
4310 | + else: |
4311 | + return 'Re: [Merge] {} into {}'.format( |
4312 | + mp.source_branch.display_name, |
4313 | + mp.target_branch.display_name) |
4314 | + |
4315 | + |
4316 | +def approve_mp(mp, revision, build_url): |
4317 | + """Approve a given merge proposal a revision. |
4318 | + |
4319 | + :params mp: launchapd handle to the respective merge proposal |
4320 | + :params revision: revision that should be approved |
4321 | + :params build_url: jenkins build url with the details. This job is used to |
4322 | + generate the message with all the links to test runs as |
4323 | + well as artifacts (coverity, deb files, etc) |
4324 | + """ |
4325 | + state = 'PASSED: Continuous integration, rev:' + str(revision) |
4326 | + logger.debug(state) |
4327 | + content = jenkinsutils.format_message_for_mp_update(build_url, |
4328 | + state + "\n") |
4329 | + mp.createComment(review_type=get_config_option('launchpad_review_type'), |
4330 | + vote=LaunchpadVote.APPROVE, subject=get_vote_subject(mp), |
4331 | + content=content) |
4332 | + |
4333 | + |
4334 | +def disapprove_mp(mp, revision, build_url, reason=None): |
4335 | + """Disapprove a given merge proposal a revision (vote Needs Fixing). |
4336 | + |
4337 | + :params mp: launchapd handle to the respective merge proposal |
4338 | + :params revision: revision that should be fixed |
4339 | + :params build_url: jenkins build url with the details. This job is used to |
4340 | + generate the message with all the links to test runs as |
4341 | + well as artifacts (coverity, deb files, etc) |
4342 | + :params reason: optional string that is attached to the comment |
4343 | + """ |
4344 | + state = "FAILED: Continuous integration, rev:{revision}".format( |
4345 | + revision=revision) |
4346 | + if reason: |
4347 | + state = "{state}\n{reason}".format(state=state, reason=reason) |
4348 | + |
4349 | + logger.debug(state) |
4350 | + content = jenkinsutils.format_message_for_mp_update( |
4351 | + build_url, state + "\n") |
4352 | + mp.createComment(review_type=get_config_option('launchpad_review_type'), |
4353 | + vote=LaunchpadVote.NEEDS_FIXING, |
4354 | + subject=get_vote_subject(mp), |
4355 | + content=content) |
4356 | + |
4357 | +# launchpadlib is not thread/process safe so we are creating launchpadlib |
4358 | +# cache in /tmp per process which gets cleaned up at the end |
4359 | +# see also lp:459418 and lp:1025153 |
4360 | +launchpad_cachedir = os.path.join('/tmp', str(os.getpid()), '.launchpadlib') |
4361 | + |
4362 | +# `launchpad_cachedir` is leaked upon unexpected exits |
4363 | +# adding this cleanup to stop directories filling up `/tmp/` |
4364 | +atexit.register(rmtree, os.path.join('/tmp', |
4365 | + str(os.getpid())), |
4366 | + ignore_errors=True) |
4367 | + |
4368 | +lp_handle = get_launchpad(launchpadlib_dir=launchpad_cachedir) |
4369 | +mp = get_mp_handle_from_url(lp_handle, args["merge_proposal"]) |
4370 | +if not mp: |
4371 | + parser.error('merge proposal related to this branch was not found') |
4372 | + |
4373 | +# this is the status from tests |
4374 | +overal_status = args['status'] |
4375 | +# by default reason is empty as it is usually just a failed build |
4376 | +reason = '' |
4377 | + |
4378 | +if overal_status == 'PASSED': |
4379 | + approve_mp(mp, args['revision'], args['build_url']) |
4380 | +else: # status == False corresponds to NOT 'PASSED' |
4381 | + disapprove_mp(mp, |
4382 | + args['revision'], |
4383 | + args['build_url'], |
4384 | + reason) |