Merge ~the-meulengracht/snappy-hwe-snaps/+git/jenkins-jobs:master into ~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-jobs:master

Proposed by Philip Meulengracht
Status: Merged
Approved by: Alfonso Sanchez-Beato
Approved revision: d20dc2f3493ffbf7c1e1bbaf516dfe1689cfbf64
Merged at revision: 016fa6aca8bf2a9774f1fb3b629ece831ca2c6e1
Proposed branch: ~the-meulengracht/snappy-hwe-snaps/+git/jenkins-jobs:master
Merge into: ~snappy-hwe-team/snappy-hwe-snaps/+git/jenkins-jobs:master
Diff against target: 491 lines (+71/-69)
11 files modified
3rdparty/repo (+1/-1)
jobs/infrastructure/credentials-1-launchpad.py (+2/-1)
jobs/infrastructure/prepare-0-install.sh (+31/-29)
tools/automerge-mps.py (+1/-1)
tools/delete-ci-repo.py (+1/-3)
tools/se_utils/__init__.py (+17/-9)
tools/shyaml (+1/-1)
tools/trigger-ci.py (+1/-1)
tools/unstage-from-manifest.py (+1/-2)
tools/update-merge-proposal.py (+2/-10)
tools/vote-on-merge-proposal.py (+13/-11)
Reviewer Review Type Date Requested Status
Alfonso Sanchez-Beato Approve
System Enablement Bot continuous-integration Approve
Review via email: mp+422731@code.launchpad.net

Description of the change

This MP contains the following:
 - Support for focal
 - Migration to python3

To post a comment you must log in.
Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

Thanks for this! I've a couple of comments.

Also, I have run flake8 on the python scripts, that should help to find issues in the migration to python3. See https://paste.ubuntu.com/p/cBMsq89xvZ/ . I think most things are harmless, except maybe:

./tools/vote-on-merge-proposal.py:76:10: E999 SyntaxError: Missing parentheses in call to 'print'. Did you mean print(...)?

Anyway, it is a good opportunity to clean-up these warnings, although maybe you can ignore the "line too long" ones.

Revision history for this message
System Enablement Bot (system-enablement-ci-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Alfonso Sanchez-Beato (alfonsosanchezbeato) wrote :

LGTM, thanks for the changes!

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/3rdparty/repo b/3rdparty/repo
2index f9eb9e8..c069798 100644
3--- a/3rdparty/repo
4+++ b/3rdparty/repo
5@@ -1,4 +1,4 @@
6-#!/usr/bin/env python
7+#!/usr/bin/env python3
8
9 # repo default configuration
10 #
11diff --git a/jobs/infrastructure/credentials-1-launchpad.py b/jobs/infrastructure/credentials-1-launchpad.py
12index 59dd706..61e36de 100644
13--- a/jobs/infrastructure/credentials-1-launchpad.py
14+++ b/jobs/infrastructure/credentials-1-launchpad.py
15@@ -1,4 +1,4 @@
16-#!/usr/bin/env python
17+#!/usr/bin/env python3
18 #
19 # Copyright (C) 2016 Canonical Ltd
20 #
21@@ -82,5 +82,6 @@ def get_launchpad(cred_path, launchpadlib_dir=None):
22 authorization_engine=authorization_engine,
23 launchpadlib_dir=launchpadlib_dir)
24
25+
26 if __name__ == '__main__':
27 get_launchpad("{credentials_path}")
28diff --git a/jobs/infrastructure/prepare-0-install.sh b/jobs/infrastructure/prepare-0-install.sh
29index d50e5a9..afffe90 100644
30--- a/jobs/infrastructure/prepare-0-install.sh
31+++ b/jobs/infrastructure/prepare-0-install.sh
32@@ -16,6 +16,9 @@
33
34 set -ex
35
36+JENKINS_JOBS_GIT_REPO="{jobs-git-repo}"
37+JENKINS_JOBS_GIT_REPO_BRANCH="{jobs-git-repo-branch}"
38+
39 # Use correct home and preserve environment variables when calling `sudo ...`
40 SUDO="sudo -E -H"
41
42@@ -26,24 +29,27 @@ $SUDO apt-get install --yes software-properties-common
43 # build tools as used in Launchpad
44 $SUDO add-apt-repository --yes ppa:launchpad/buildd-staging
45 $SUDO add-apt-repository --yes ppa:jenkaas-hackers/tools
46-$SUDO add-apt-repository --yes ppa:snappy-hwe-team/ci-tools
47+
48+version=$(lsb_release -sc)
49+if [ "$version" = "xenial" ]; then
50+ $SUDO add-apt-repository --yes ppa:snappy-hwe-team/ci-tools
51+fi
52
53 $SUDO apt-get update
54
55 echo "postfix postfix/mailname string $(hostname)" | $SUDO debconf-set-selections
56 echo "postfix postfix/main_mailer_type string 'Internet Site'" | $SUDO debconf-set-selections
57
58+# install shared packages
59 $SUDO apt-get install --yes \
60 git \
61- python \
62- python-launchpadlib \
63- python-bzrlib \
64- python-lockfile \
65- python-yaml \
66- python-jenkins \
67- python-requests \
68- python-git \
69- python-pip \
70+ python3 \
71+ python3-launchpadlib \
72+ python3-lockfile \
73+ python3-yaml \
74+ python3-jenkins \
75+ python3-requests \
76+ python3-git \
77 python3-debian \
78 python3-pip \
79 tarmac \
80@@ -58,7 +64,6 @@ $SUDO apt-get install --yes \
81 jq \
82 postfix \
83 abootimg \
84- android-tools-fsutils \
85 golang-go \
86 dpkg-dev \
87 devscripts \
88@@ -66,31 +71,28 @@ $SUDO apt-get install --yes \
89 bzr \
90 dh-systemd \
91 expect \
92- tcl \
93- {install_packages}
94+ tcl {install_packages}
95+
96+# install additional packages based on distro
97+if [ "$version" != "xenial" ]; then
98+ $SUDO apt-get install --yes \
99+ android-sdk-libsparse-utils \
100+ android-sdk-ext4-utils
101+else
102+ $SUDO apt-get install --yes \
103+ android-tools-fsutils
104+fi
105
106 # Install snaps
107 $SUDO snap install snapcraft
108 $SUDO snap install yq
109
110-# Install additional sofware from the python package database
111-$SUDO pip2 install --upgrade pip
112-$SUDO pip2 install jenkins-job-builder pid
113+export http_proxy="{http_proxy}"
114+export https_proxy="{https_proxy}"
115
116 # documentation-builder is normally available as a snap but we can't
117 # use snaps yet on our agents.
118 $SUDO pip3 install --upgrade pip
119+$SUDO pip3 install --ignore-installed PyYAML
120+$SUDO pip3 install jenkins-job-builder pid
121 $SUDO pip3 install ubuntudesign.documentation-builder
122-
123-# Now http & https proxy is not needed, it will in fact interfere with curl
124-# below
125-
126-unset http_proxy
127-unset https_proxy
128-
129-# Fetch and install latest repo; we still can't use snaps reliably on our
130-# agents so we go with the upstream release.
131-$SUDO mkdir -p /usr/local/bin
132-curl $JENKINS_JOBS_GIT_REPO/plain/3rdparty/repo?h=$JENKINS_JOBS_GIT_REPO_BRANCH | $SUDO tee /usr/local/bin/repo
133-$SUDO mv $HOME/repo /usr/local/bin/repo
134-$SUDO chmod +x /usr/local/bin/repo
135diff --git a/tools/automerge-mps.py b/tools/automerge-mps.py
136index 2e1e0b6..150da07 100755
137--- a/tools/automerge-mps.py
138+++ b/tools/automerge-mps.py
139@@ -1,4 +1,4 @@
140-#!/usr/bin/env python
141+#!/usr/bin/env python3
142 # -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
143 #
144 # Copyright (C) 2016 Canonical Ltd
145diff --git a/tools/delete-ci-repo.py b/tools/delete-ci-repo.py
146index 35c762d..3a04613 100755
147--- a/tools/delete-ci-repo.py
148+++ b/tools/delete-ci-repo.py
149@@ -1,4 +1,4 @@
150-#!/usr/bin/env python
151+#!/usr/bin/env python3
152 #
153 # Copyright (C) 2017 Canonical Ltd
154 #
155@@ -14,8 +14,6 @@
156 # You should have received a copy of the GNU General Public License
157 # along with this program. If not, see <http://www.gnu.org/licenses/>.
158
159-from launchpadlib.launchpad import Launchpad
160-
161 from argparse import ArgumentParser
162
163 import se_utils
164diff --git a/tools/se_utils/__init__.py b/tools/se_utils/__init__.py
165index 64cc0cc..e78eb6b 100644
166--- a/tools/se_utils/__init__.py
167+++ b/tools/se_utils/__init__.py
168@@ -1,4 +1,4 @@
169-#!/usr/bin/env python
170+#!/usr/bin/env python3
171 # -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
172 #
173 # Copyright (C) 2016 Canonical Ltd
174@@ -18,7 +18,6 @@
175 import atexit
176 import sys
177 import time
178-import logging
179 import os
180 import re
181 import yaml
182@@ -28,11 +27,13 @@ from lazr.restfulclient.errors import HTTPError
183 from launchpadlib.launchpad import Launchpad
184 from launchpadlib.credentials import UnencryptedFileCredentialStore
185
186+
187 class LaunchpadVote():
188 APPROVE = 'Approve'
189 DISAPPROVE = 'Disapprove'
190 NEEDS_FIXING = 'Needs Fixing'
191
192+
193 ACCESS_TOKEN_POLL_TIME = 10
194 WAITING_FOR_USER = """Open this link:
195 {}
196@@ -58,7 +59,7 @@ class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine):
197
198 """
199 authorization_url = self.authorization_url(request_token)
200- print WAITING_FOR_USER.format(authorization_url)
201+ print(WAITING_FOR_USER.format(authorization_url))
202 # if we don't flush we may not see the message
203 sys.stdout.flush()
204 while credentials.access_token is None:
205@@ -67,7 +68,7 @@ class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine):
206 credentials.exchange_request_token_for_access_token(
207 self.web_root)
208 break
209- except HTTPError, e:
210+ except HTTPError as e:
211 if e.response.status == 403:
212 # The user decided not to authorize this
213 # application.
214@@ -79,6 +80,7 @@ class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine):
215 # There was an error accessing the server.
216 raise e
217
218+
219 # launchpadlib is not thread/process safe so we are creating launchpadlib
220 # cache in /tmp per process which gets cleaned up at the end
221 # see also lp:459418 and lp:1025153
222@@ -97,8 +99,8 @@ def get_launchpad(launchpadlib_dir=None, credential_store_path=None, lp_app=None
223 the default """
224 store = UnencryptedFileCredentialStore(credential_store_path)
225 authorization_engine = AuthorizeRequestTokenWithConsole(lp_env, lp_app)
226- lib_dir=launchpad_cachedir
227- if launchpadlib_dir != None:
228+ lib_dir = launchpad_cachedir
229+ if launchpadlib_dir is not None:
230 lib_dir = launchpadlib_dir
231 return Launchpad.login_with(lp_app, lp_env,
232 credential_store=store,
233@@ -106,6 +108,7 @@ def get_launchpad(launchpadlib_dir=None, credential_store_path=None, lp_app=None
234 launchpadlib_dir=lib_dir,
235 version='devel')
236
237+
238 # Load configuration for the current agent we're running on. All agents were
239 # provisioned when they were setup with a proper configuration. See
240 # https://wiki.canonical.com/InformationInfrastructure/Jenkaas/UserDocs for
241@@ -121,12 +124,14 @@ def load_config():
242 print("ERROR: No config file found")
243 sys.exit(1)
244
245+
246 # Return a configuration option from the agent configuration specified by the
247 # name argument.
248 def get_config_option(name):
249 config = load_config()
250 return config[name]
251
252+
253 def get_branch_handle_from_url(lp_handle, url):
254 """ Return a branch/repo handle for the given url.
255 Returns a launchpad branch or git repository handle for the given url.
256@@ -147,6 +152,7 @@ def get_branch_handle_from_url(lp_handle, url):
257 print('fetching branch: ' + name)
258 return lp_handle.branches.getByUrl(url=name)
259
260+
261 def get_branch_from_mp(merge_proposal):
262 """Return a link to branch given a link to a merge proposal.
263
264@@ -157,11 +163,12 @@ def get_branch_from_mp(merge_proposal):
265
266 :param merge_proposal: url of a launchpad merge proposal
267 """
268- m = re.search('(.*)\+merge/[0-9]+$', merge_proposal)
269+ m = re.search(r'(.*)\+merge/[0-9]+$', merge_proposal)
270 if m:
271 return m.group(1)
272 return None
273
274+
275 def get_mp_handle_from_url(lp_handle, merge_proposal_link):
276 """ Get launchpad handle for merge proposal given a merge proposal URL.
277
278@@ -173,7 +180,7 @@ def get_mp_handle_from_url(lp_handle, merge_proposal_link):
279 print('Unable to get branch link from merge proposal link.')
280 return None
281
282- branch = get_branch_handle_from_url(lp_handle, branch_link)
283+ branch = get_branch_handle_from_url(lp_handle, branch_link)
284 if not branch:
285 print('Branch {} does not exist'.format(branch_link))
286 return None
287@@ -187,6 +194,7 @@ def get_mp_handle_from_url(lp_handle, merge_proposal_link):
288
289 return None
290
291+
292 def get_vote_subject(mp):
293 """Given a mp handle return a subject for the vote message
294
295@@ -209,4 +217,4 @@ def get_vote_subject(mp):
296 else:
297 return 'Re: [Merge] {} into {}'.format(
298 mp.source_branch.display_name,
299- mp.target_branch.display_name)
300\ No newline at end of file
301+ mp.target_branch.display_name)
302diff --git a/tools/shyaml b/tools/shyaml
303index e4618ec..ea3db4f 100755
304--- a/tools/shyaml
305+++ b/tools/shyaml
306@@ -1,4 +1,4 @@
307-#!/usr/bin/env python
308+#!/usr/bin/env python3
309
310 # Taken from upstream git repository https://github.com/0k/shyaml
311 # at revision d77e30599a0971c51896ef97d21883550e7e9979
312diff --git a/tools/trigger-ci.py b/tools/trigger-ci.py
313index 65f3b3e..9267a16 100755
314--- a/tools/trigger-ci.py
315+++ b/tools/trigger-ci.py
316@@ -1,4 +1,4 @@
317-#!/usr/bin/env python
318+#!/usr/bin/env python3
319 # -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
320 #
321 # Copyright (C) 2016 Canonical Ltd
322diff --git a/tools/unstage-from-manifest.py b/tools/unstage-from-manifest.py
323index e189492..e3f79ec 100755
324--- a/tools/unstage-from-manifest.py
325+++ b/tools/unstage-from-manifest.py
326@@ -22,7 +22,6 @@
327 # store.
328
329 import sys
330-import yaml
331 from tools import yaml_utils
332
333
334@@ -34,7 +33,7 @@ def remove_from_staged(pkg_list_path, manifest_path, out_manifest_path):
335 try:
336 with open(pkg_list_path) as pkg_list_f:
337 pkg_list = pkg_list_f.read().splitlines()
338- except FileNotFoundError as e:
339+ except FileNotFoundError:
340 print(pkg_list_path, 'not found, just copying manifest')
341 pkg_list = []
342
343diff --git a/tools/update-merge-proposal.py b/tools/update-merge-proposal.py
344index 0054659..261fa6d 100755
345--- a/tools/update-merge-proposal.py
346+++ b/tools/update-merge-proposal.py
347@@ -1,4 +1,4 @@
348-#!/usr/bin/env python
349+#!/usr/bin/env python3
350 # -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
351 #
352 # Copyright (C) 2017 Canonical Ltd
353@@ -15,14 +15,6 @@
354 # You should have received a copy of the GNU General Public License
355 # along with this program. If not, see <http://www.gnu.org/licenses/>.
356
357-import atexit
358-import sys
359-import time
360-import logging
361-import os
362-import yaml
363-import re
364-from shutil import rmtree
365 from argparse import ArgumentParser
366
367 import se_utils
368@@ -54,7 +46,7 @@ content = content.decode('string_escape')
369 if not args['approve'] and not args['disapprove']:
370 parser.error('neither approved or disapproved the MP')
371
372-vote=""
373+vote = ""
374 if args["approve"]:
375 vote = se_utils.LaunchpadVote.APPROVE
376 elif args["disapprove"]:
377diff --git a/tools/vote-on-merge-proposal.py b/tools/vote-on-merge-proposal.py
378index bec6da5..96838a7 100755
379--- a/tools/vote-on-merge-proposal.py
380+++ b/tools/vote-on-merge-proposal.py
381@@ -1,4 +1,4 @@
382-#!/usr/bin/env python
383+#!/usr/bin/env python3
384 # -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
385 #
386 # Copyright (C) 2016 Canonical Ltd
387@@ -20,7 +20,6 @@ import sys
388 import time
389 import logging
390 import os
391-import yaml
392 import re
393 from shutil import rmtree
394 from argparse import ArgumentParser
395@@ -29,7 +28,7 @@ from lazr.restfulclient.errors import HTTPError
396 from launchpadlib.launchpad import Launchpad
397 from launchpadlib.credentials import UnencryptedFileCredentialStore
398 from jlp import get_config_option
399-from jlp import launchpadutils, jenkinsutils, logger
400+from jlp import jenkinsutils
401
402 logger = logging.getLogger('jenkins-launchpad-plugin')
403 stdout_handler = logging.StreamHandler(stream=sys.stdout)
404@@ -73,7 +72,7 @@ class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine):
405
406 """
407 authorization_url = self.authorization_url(request_token)
408- print WAITING_FOR_USER.format(authorization_url)
409+ print(WAITING_FOR_USER.format(authorization_url))
410 # if we don't flush we may not see the message
411 sys.stdout.flush()
412 while credentials.access_token is None:
413@@ -82,7 +81,7 @@ class AuthorizeRequestTokenWithConsole(RequestTokenAuthorizationEngine):
414 credentials.exchange_request_token_for_access_token(
415 self.web_root)
416 break
417- except HTTPError, e:
418+ except HTTPError as e:
419 if e.response.status == 403:
420 # The user decided not to authorize this
421 # application.
422@@ -110,6 +109,7 @@ def get_launchpad(launchpadlib_dir=None):
423 launchpadlib_dir=launchpadlib_dir,
424 version='devel')
425
426+
427 def get_branch_handle_from_url(lp_handle, url):
428 """ Return a branch/repo handle for the given url.
429 Returns a launchpad branch or git repository handle for the given url.
430@@ -130,6 +130,7 @@ def get_branch_handle_from_url(lp_handle, url):
431 logger.debug('fetching branch: ' + name)
432 return lp_handle.branches.getByUrl(url=name)
433
434+
435 def get_branch_from_mp(merge_proposal):
436 """Return a link to branch given a link to a merge proposal.
437
438@@ -140,11 +141,12 @@ def get_branch_from_mp(merge_proposal):
439
440 :param merge_proposal: url of a launchpad merge proposal
441 """
442- m = re.search('(.*)\+merge/[0-9]+$', merge_proposal)
443+ m = re.search(r'(.*)\+merge/[0-9]+$', merge_proposal)
444 if m:
445 return m.group(1)
446 return None
447
448+
449 def get_mp_handle_from_url(lp_handle, merge_proposal_link):
450 """ Get launchpad handle for merge proposal given a merge proposal URL.
451
452@@ -156,7 +158,7 @@ def get_mp_handle_from_url(lp_handle, merge_proposal_link):
453 logger.error('Unable to get branch link from merge proposal link.')
454 return None
455
456- branch = get_branch_handle_from_url(lp_handle, branch_link)
457+ branch = get_branch_handle_from_url(lp_handle, branch_link)
458 if not branch:
459 logger.debug('Branch {} does not exist'.format(branch_link))
460 return None
461@@ -170,11 +172,13 @@ def get_mp_handle_from_url(lp_handle, merge_proposal_link):
462
463 return None
464
465+
466 class LaunchpadVote():
467 APPROVE = 'Approve'
468 DISAPPROVE = 'Disapprove'
469 NEEDS_FIXING = 'Needs Fixing'
470
471+
472 def get_vote_subject(mp):
473 """Given a mp handle return a subject for the vote message
474
475@@ -241,6 +245,7 @@ def disapprove_mp(mp, revision, build_url, reason=None):
476 subject=get_vote_subject(mp),
477 content=content)
478
479+
480 # launchpadlib is not thread/process safe so we are creating launchpadlib
481 # cache in /tmp per process which gets cleaned up at the end
482 # see also lp:459418 and lp:1025153
483@@ -265,7 +270,4 @@ reason = ''
484 if overal_status == 'PASSED':
485 approve_mp(mp, args['revision'], args['build_url'])
486 else: # status == False corresponds to NOT 'PASSED'
487- disapprove_mp(mp,
488- args['revision'],
489- args['build_url'],
490- reason)
491+ disapprove_mp(mp, args['revision'], args['build_url'], reason)

Subscribers

People subscribed via source and target branches