Merge lp:~canonical-ci-engineering/britney/boottesting-support into lp:~ubuntu-release/britney/britney2-ubuntu
- boottesting-support
- Merge into britney2-ubuntu
Status: | Merged |
---|---|
Merged at revision: | 430 |
Proposed branch: | lp:~canonical-ci-engineering/britney/boottesting-support |
Merge into: | lp:~ubuntu-release/britney/britney2-ubuntu |
Prerequisite: | lp:~cprov/britney/test-refactoring |
Diff against target: |
971 lines (+814/-1) 7 files modified
boottest.py (+282/-0) britney.conf (+5/-0) britney.py (+99/-0) excuse.py (+2/-1) tests/__init__.py (+5/-0) tests/test_autopkgtest.py (+10/-0) tests/test_boottest.py (+411/-0) |
To merge this branch: | bzr merge lp:~canonical-ci-engineering/britney/boottesting-support |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Colin Watson (community) | Approve | ||
Francis Ginther (community) | Approve | ||
Ubuntu Release Team | Pending | ||
Review via email: mp+247642@code.launchpad.net |
Commit message
Add support for submitting and collecting results for a boottest to be performed on packages that contribute binaries to the Ubuntu Touch image.
Description of the change
(This MP supersedes https:/
Basic "boottest" testing criteria support for britney.
It extends britney to submit and consider "boottest" test results before promoting uploads producing binaries that are included in the phone image.
- 441. By Francis Ginther
-
Correct copy-n-paste of 'e.' for 'excuse.'.
Colin Watson (cjwatson) : | # |
- 442. By Celso Providelo
-
Merging manifest auto-fetching changes
- 443. By Celso Providelo
-
Implement 'run_boottest' flag, similarly to 'run_autopkgset'.
- 444. By Celso Providelo
-
Introduce BOOTTEST_FETCH configuration option for enabling/disabling TouchManifest auto-fetching feature, for faster and isolated tests.
- 445. By Celso Providelo
-
Checking BOOTTEST_ARCHES binaries before attempting to boottest.
- 446. By Celso Providelo
-
Basic debug info when fetching manifests.
- 447. By Celso Providelo
-
Wraps 'boottest-britney' script for checking status or submitting new jobs.
- 448. By Celso Providelo
-
Request boottests for sources, not binaries. Jenkins glue will drive adt properly.
Celso Providelo (cprov) wrote : | # |
Francis,
Thanks for the review.
Now we are dispatching boottest request per SP, instead of its BPs.
I will work on changes for BootTestJenkinsJob to mimic the API already established for autopkgtest (request/
- 449. By Celso Providelo
-
Merge the boottest-jenkins script wrapper into BootTest class for simplicity and extend the API to match what we already for auto-package-
testing (ADT).
Celso Providelo (cprov) wrote : | # |
Francis,
Now BootTest criteria uses the established `boottest-jenkins` API (request, submit, collect) very similarly to the existing adt-jenkins.
The only exception, a simplification in fact, is that boottests does not map source triggers as autopkgtest, since the trigger for boottests is always the source itself.
- 450. By Celso Providelo
-
Cleanup the boottest-britney testing script to reflect better its real behavior.
- 451. By Celso Providelo
-
Adjusting boottest-jenkins arguments to match reality.
- 452. By Joe Talbott
-
boottest - Add retry loop (3) for manifest fetching.
- 453. By Joe Talbott
-
boottest - Only make the manifest directory if it doesn't already exist.
Francis Ginther (fginther) wrote : | # |
Looks pretty good. My prior comments have been addressed and I don't see any issues with the proposed implementation.
- 454. By Vincent Ladeuil
-
According to cjwatson, cdimage deals with projects, not distribution, fix TouchManifest accordingly.
Fix boottest-britney location to match production.
Fix TestTouchManifest test failures, now that we retry on manifest download errors, the tests should inhibit the retries when testing the failures.5 out of the 8 TestBoottestEnd2End are still failing because the excuse says the tests are skipped instead of running/
failing/ being in progress, etc. - 455. By Celso Providelo
-
[test-fix] Adjusting test setup for the new manifest path (using project) and the auto-package-
testing project path. - 456. By Celso Providelo
-
[test-fix] Fixing test_autopkgtest.py configuration setup, so all tests pass now.
- 457. By Joe Talbott
-
boottest - Add exception handling for manifest fetching
- 458. By Celso Providelo
-
Boottest integration adjustments.
- 459. By Celso Providelo
-
Presenting links for the corresponding boottest jenkins job.
- 460. By Celso Providelo
-
Suppress boottest SKIPPED notice on excuses, it's unnecessary noise.
- 461. By Celso Providelo
-
Cope with missing/broken results due to outdated apt cache. Britney will not fail, but source promotion will be blocked. Also waiting for amd64 binaries (arch-indep deps) before running boottests.
- 462. By Vincent Ladeuil
-
Poperly initialize Excuse.
run_boottest.
Colin Watson (cjwatson) wrote : | # |
This looks pretty good to me now; just a few nits.
Steve Langasek (vorlon) wrote : | # |
Have merged this, but have updated to set BOOTTEST_ENABLE = no due to the following error:
Traceback (most recent call last):
File "/home/
Britney(
File "/home/
self.
File "/home/
upgrade_
ValueError: list.remove(x): x not in list
Independently of this bug, it looks like we need BOOTTEST_ENABLE = no because the only way we reached this code block was because p-m had decided there was a non-empty set of packages that needed to be blocked on boot test results, which we of course don't have yet.
Preview Diff
1 | === added file 'boottest.py' | |||
2 | --- boottest.py 1970-01-01 00:00:00 +0000 | |||
3 | +++ boottest.py 2015-02-18 13:50:50 +0000 | |||
4 | @@ -0,0 +1,282 @@ | |||
5 | 1 | # -*- coding: utf-8 -*- | ||
6 | 2 | |||
7 | 3 | # Copyright (C) 2015 Canonical Ltd. | ||
8 | 4 | |||
9 | 5 | # This program is free software; you can redistribute it and/or modify | ||
10 | 6 | # it under the terms of the GNU General Public License as published by | ||
11 | 7 | # the Free Software Foundation; either version 2 of the License, or | ||
12 | 8 | # (at your option) any later version. | ||
13 | 9 | |||
14 | 10 | # This program is distributed in the hope that it will be useful, | ||
15 | 11 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
16 | 12 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
17 | 13 | # GNU General Public License for more details. | ||
18 | 14 | from __future__ import print_function | ||
19 | 15 | |||
20 | 16 | from collections import defaultdict | ||
21 | 17 | from contextlib import closing | ||
22 | 18 | import os | ||
23 | 19 | import subprocess | ||
24 | 20 | import tempfile | ||
25 | 21 | from textwrap import dedent | ||
26 | 22 | import time | ||
27 | 23 | import urllib | ||
28 | 24 | |||
29 | 25 | import apt_pkg | ||
30 | 26 | |||
31 | 27 | from consts import BINARIES | ||
32 | 28 | |||
33 | 29 | |||
34 | 30 | FETCH_RETRIES = 3 | ||
35 | 31 | |||
36 | 32 | |||
37 | 33 | class TouchManifest(object): | ||
38 | 34 | """Parses a corresponding touch image manifest. | ||
39 | 35 | |||
40 | 36 | Based on http://cdimage.u.c/ubuntu-touch/daily-preinstalled/pending/vivid-preinstalled-touch-armhf.manifest | ||
41 | 37 | |||
42 | 38 | Assumes the deployment is arranged in a way the manifest is available | ||
43 | 39 | and fresh on: | ||
44 | 40 | |||
45 | 41 | '{britney_cwd}/boottest/images/{distribution}/{series}/manifest' | ||
46 | 42 | |||
47 | 43 | Only binary name matters, version is ignored, so callsites can: | ||
48 | 44 | |||
49 | 45 | >>> manifest = TouchManifest('ubuntu-touch', 'vivid') | ||
50 | 46 | >>> 'webbrowser-app' in manifest | ||
51 | 47 | True | ||
52 | 48 | >>> 'firefox' in manifest | ||
53 | 49 | False | ||
54 | 50 | |||
55 | 51 | """ | ||
56 | 52 | |||
57 | 53 | def __init__(self, project, series, verbose=False, fetch=True): | ||
58 | 54 | self.verbose = verbose | ||
59 | 55 | self.path = "boottest/images/{}/{}/manifest".format( | ||
60 | 56 | project, series) | ||
61 | 57 | success = False | ||
62 | 58 | if fetch: | ||
63 | 59 | retries = FETCH_RETRIES | ||
64 | 60 | success = self.__fetch_manifest(project, series) | ||
65 | 61 | |||
66 | 62 | while retries > 0 and not success: | ||
67 | 63 | success = self.__fetch_manifest(project, series) | ||
68 | 64 | retries -= 1 | ||
69 | 65 | if not success: | ||
70 | 66 | print("E: [%s] - Unable to fetch manifest: %s %s" % ( | ||
71 | 67 | time.asctime(), project, series)) | ||
72 | 68 | |||
73 | 69 | self._manifest = self._load() | ||
74 | 70 | |||
75 | 71 | def __fetch_manifest(self, project, series): | ||
76 | 72 | url = "http://cdimage.ubuntu.com/{}/daily-preinstalled/" \ | ||
77 | 73 | "pending/{}-preinstalled-touch-armhf.manifest".format( | ||
78 | 74 | project, series | ||
79 | 75 | ) | ||
80 | 76 | success = False | ||
81 | 77 | if self.verbose: | ||
82 | 78 | print( | ||
83 | 79 | "I: [%s] - Fetching manifest from %s" % ( | ||
84 | 80 | time.asctime(), url)) | ||
85 | 81 | print("I: [%s] - saving it to %s" % (time.asctime(), self.path)) | ||
86 | 82 | try: | ||
87 | 83 | response = urllib.urlopen(url) | ||
88 | 84 | except IOError as e: | ||
89 | 85 | print("W: [%s] - error connecting to %s: %s" % ( | ||
90 | 86 | time.asctime(), self.path, e)) | ||
91 | 87 | return success # failure | ||
92 | 88 | |||
93 | 89 | # Only [re]create the manifest file if one was successfully downloaded | ||
94 | 90 | # this allows for an existing image to be used if the download fails. | ||
95 | 91 | if response.code == 200: | ||
96 | 92 | path_dir = os.path.dirname(self.path) | ||
97 | 93 | if not os.path.exists(path_dir): | ||
98 | 94 | os.makedirs(path_dir) | ||
99 | 95 | with open(self.path, 'w') as fp: | ||
100 | 96 | fp.write(response.read()) | ||
101 | 97 | success = True | ||
102 | 98 | |||
103 | 99 | return success | ||
104 | 100 | |||
105 | 101 | def _load(self): | ||
106 | 102 | pkg_list = [] | ||
107 | 103 | |||
108 | 104 | if not os.path.exists(self.path): | ||
109 | 105 | return pkg_list | ||
110 | 106 | |||
111 | 107 | with open(self.path) as fd: | ||
112 | 108 | for line in fd.readlines(): | ||
113 | 109 | # skip headers and metadata | ||
114 | 110 | if 'DOCTYPE' in line: | ||
115 | 111 | continue | ||
116 | 112 | name, version = line.split() | ||
117 | 113 | name = name.split(':')[0] | ||
118 | 114 | if name == 'click': | ||
119 | 115 | continue | ||
120 | 116 | pkg_list.append(name) | ||
121 | 117 | |||
122 | 118 | return sorted(pkg_list) | ||
123 | 119 | |||
124 | 120 | def __contains__(self, key): | ||
125 | 121 | return key in self._manifest | ||
126 | 122 | |||
127 | 123 | |||
128 | 124 | class BootTest(object): | ||
129 | 125 | """Boottest criteria for Britney. | ||
130 | 126 | |||
131 | 127 | This class provides an API for handling the boottest-jenkins | ||
132 | 128 | integration layer (mostly derived from auto-package-testing/adt): | ||
133 | 129 | """ | ||
134 | 130 | VALID_STATUSES = ('PASS',) | ||
135 | 131 | |||
136 | 132 | EXCUSE_LABELS = { | ||
137 | 133 | "PASS": '<span style="background:#87d96c">Pass</span>', | ||
138 | 134 | "FAIL": '<span style="background:#ff6666">Regression</span>', | ||
139 | 135 | "RUNNING": '<span style="background:#99ddff">Test in progress</span>', | ||
140 | 136 | } | ||
141 | 137 | |||
142 | 138 | script_path = os.path.expanduser( | ||
143 | 139 | "~/auto-package-testing/jenkins/boottest-britney") | ||
144 | 140 | |||
145 | 141 | def __init__(self, britney, distribution, series, debug=False): | ||
146 | 142 | self.britney = britney | ||
147 | 143 | self.distribution = distribution | ||
148 | 144 | self.series = series | ||
149 | 145 | self.debug = debug | ||
150 | 146 | self.rc_path = None | ||
151 | 147 | self._read() | ||
152 | 148 | manifest_fetch = getattr( | ||
153 | 149 | self.britney.options, "boottest_fetch", "no") == "yes" | ||
154 | 150 | self.phone_manifest = TouchManifest( | ||
155 | 151 | 'ubuntu-touch', self.series, fetch=manifest_fetch, | ||
156 | 152 | verbose=self.britney.options.verbose) | ||
157 | 153 | |||
158 | 154 | @property | ||
159 | 155 | def _request_path(self): | ||
160 | 156 | return "boottest/work/adt.request.%s" % self.series | ||
161 | 157 | |||
162 | 158 | @property | ||
163 | 159 | def _result_path(self): | ||
164 | 160 | return "boottest/work/adt.result.%s" % self.series | ||
165 | 161 | |||
166 | 162 | def _ensure_rc_file(self): | ||
167 | 163 | if self.rc_path: | ||
168 | 164 | return | ||
169 | 165 | self.rc_path = os.path.abspath("boottest/rc.%s" % self.series) | ||
170 | 166 | with open(self.rc_path, "w") as rc_file: | ||
171 | 167 | home = os.path.expanduser("~") | ||
172 | 168 | print(dedent("""\ | ||
173 | 169 | release: %s | ||
174 | 170 | aptroot: ~/.chdist/%s-proposed-armhf/ | ||
175 | 171 | apturi: file:%s/mirror/%s | ||
176 | 172 | components: main restricted universe multiverse | ||
177 | 173 | rsync_host: rsync://tachash.ubuntu-ci/boottest/ | ||
178 | 174 | datadir: ~/proposed-migration/boottest/data""" % | ||
179 | 175 | (self.series, self.series, home, self.distribution)), | ||
180 | 176 | file=rc_file) | ||
181 | 177 | |||
182 | 178 | def _run(self, *args): | ||
183 | 179 | self._ensure_rc_file() | ||
184 | 180 | if not os.path.exists(self.script_path): | ||
185 | 181 | print("E: [%s] - Boottest/Jenking glue script missing: %s" % ( | ||
186 | 182 | time.asctime(), self.script_path)) | ||
187 | 183 | return '-' | ||
188 | 184 | command = [ | ||
189 | 185 | self.script_path, | ||
190 | 186 | "-c", self.rc_path, | ||
191 | 187 | "-r", self.series, | ||
192 | 188 | "-PU", | ||
193 | 189 | ] | ||
194 | 190 | if self.debug: | ||
195 | 191 | command.append("-d") | ||
196 | 192 | command.extend(args) | ||
197 | 193 | return subprocess.check_output(command).strip() | ||
198 | 194 | |||
199 | 195 | def _read(self): | ||
200 | 196 | """Loads a list of results (sources tests and their status). | ||
201 | 197 | |||
202 | 198 | Provides internal data for `get_status()`. | ||
203 | 199 | """ | ||
204 | 200 | self.pkglist = defaultdict(dict) | ||
205 | 201 | if not os.path.exists(self._result_path): | ||
206 | 202 | return | ||
207 | 203 | with open(self._result_path) as f: | ||
208 | 204 | for line in f: | ||
209 | 205 | line = line.strip() | ||
210 | 206 | if line.startswith("Suite:") or line.startswith("Date:"): | ||
211 | 207 | continue | ||
212 | 208 | linebits = line.split() | ||
213 | 209 | if len(linebits) < 2: | ||
214 | 210 | print("W: Invalid line format: '%s', skipped" % line) | ||
215 | 211 | continue | ||
216 | 212 | (src, ver, status) = linebits[:3] | ||
217 | 213 | if not (src in self.pkglist and ver in self.pkglist[src]): | ||
218 | 214 | self.pkglist[src][ver] = status | ||
219 | 215 | |||
220 | 216 | def get_status(self, name, version): | ||
221 | 217 | """Return test status for the given source name and version.""" | ||
222 | 218 | try: | ||
223 | 219 | return self.pkglist[name][version] | ||
224 | 220 | except KeyError: | ||
225 | 221 | # This error handling accounts for outdated apt caches, when | ||
226 | 222 | # `boottest-britney` erroneously reports results for the | ||
227 | 223 | # current source version, instead of the proposed. | ||
228 | 224 | # Returning None here will block source promotion with: | ||
229 | 225 | # 'UNKNOWN STATUS' excuse. If the jobs are retried and its | ||
230 | 226 | # results find an up-to-date cache, the problem is gone. | ||
231 | 227 | print("E: [%s] - Missing boottest results for %s_%s" % ( | ||
232 | 228 | time.asctime(), name, version)) | ||
233 | 229 | return None | ||
234 | 230 | |||
235 | 231 | def request(self, packages): | ||
236 | 232 | """Requests boottests for the given sources list ([(src, ver),]).""" | ||
237 | 233 | request_path = self._request_path | ||
238 | 234 | if os.path.exists(request_path): | ||
239 | 235 | os.unlink(request_path) | ||
240 | 236 | with closing(tempfile.NamedTemporaryFile(mode="w")) as request_file: | ||
241 | 237 | for src, ver in packages: | ||
242 | 238 | if src in self.pkglist and ver in self.pkglist[src]: | ||
243 | 239 | continue | ||
244 | 240 | print("%s %s" % (src, ver), file=request_file) | ||
245 | 241 | # Update 'pkglist' so even if submit/collect is not called | ||
246 | 242 | # (dry-run), britney has some results. | ||
247 | 243 | self.pkglist[src][ver] = 'RUNNING' | ||
248 | 244 | request_file.flush() | ||
249 | 245 | self._run("request", "-O", request_path, request_file.name) | ||
250 | 246 | |||
251 | 247 | def submit(self): | ||
252 | 248 | """Submits the current boottests requests for processing.""" | ||
253 | 249 | self._run("submit", self._request_path) | ||
254 | 250 | |||
255 | 251 | def collect(self): | ||
256 | 252 | """Collects boottests results and updates internal registry.""" | ||
257 | 253 | self._run("collect", "-O", self._result_path) | ||
258 | 254 | self._read() | ||
259 | 255 | if not self.britney.options.verbose: | ||
260 | 256 | return | ||
261 | 257 | for src in sorted(self.pkglist): | ||
262 | 258 | for ver in sorted(self.pkglist[src], cmp=apt_pkg.version_compare): | ||
263 | 259 | status = self.pkglist[src][ver] | ||
264 | 260 | print("I: [%s] - Collected boottest status for %s_%s: " | ||
265 | 261 | "%s" % (time.asctime(), src, ver, status)) | ||
266 | 262 | |||
267 | 263 | def needs_test(self, name, version): | ||
268 | 264 | """Whether or not the given source and version should be tested. | ||
269 | 265 | |||
270 | 266 | Sources are only considered for boottesting if they produce binaries | ||
271 | 267 | that are part of the phone image manifest. See `TouchManifest`. | ||
272 | 268 | """ | ||
273 | 269 | # Discover all binaries for the 'excused' source. | ||
274 | 270 | unstable_sources = self.britney.sources['unstable'] | ||
275 | 271 | # Dismiss if source is not yet recognized (??). | ||
276 | 272 | if name not in unstable_sources: | ||
277 | 273 | return False | ||
278 | 274 | # Binaries are a seq of "<binname>/<arch>" and, practically, boottest | ||
279 | 275 | # is only concerned about armhf binaries mentioned in the phone | ||
280 | 276 | # manifest. Anything else should be skipped. | ||
281 | 277 | phone_binaries = [ | ||
282 | 278 | b for b in unstable_sources[name][BINARIES] | ||
283 | 279 | if b.split('/')[1] in self.britney.options.boottest_arches.split() | ||
284 | 280 | and b.split('/')[0] in self.phone_manifest | ||
285 | 281 | ] | ||
286 | 282 | return bool(phone_binaries) | ||
287 | 0 | 283 | ||
288 | === modified file 'britney.conf' | |||
289 | --- britney.conf 2014-12-18 15:42:20 +0000 | |||
290 | +++ britney.conf 2015-02-18 13:50:50 +0000 | |||
291 | @@ -65,3 +65,8 @@ | |||
292 | 65 | ADT_ENABLE = yes | 65 | ADT_ENABLE = yes |
293 | 66 | ADT_DEBUG = no | 66 | ADT_DEBUG = no |
294 | 67 | ADT_ARCHES = amd64 i386 | 67 | ADT_ARCHES = amd64 i386 |
295 | 68 | |||
296 | 69 | BOOTTEST_ENABLE = yes | ||
297 | 70 | BOOTTEST_DEBUG = yes | ||
298 | 71 | BOOTTEST_ARCHES = armhf amd64 | ||
299 | 72 | BOOTTEST_FETCH = yes | ||
300 | 68 | 73 | ||
301 | === modified file 'britney.py' | |||
302 | --- britney.py 2015-01-09 11:57:33 +0000 | |||
303 | +++ britney.py 2015-02-18 13:50:50 +0000 | |||
304 | @@ -226,6 +226,8 @@ | |||
305 | 226 | SOURCE, SOURCEVER, ARCHITECTURE, DEPENDS, CONFLICTS, | 226 | SOURCE, SOURCEVER, ARCHITECTURE, DEPENDS, CONFLICTS, |
306 | 227 | PROVIDES, RDEPENDS, RCONFLICTS, MULTIARCH, ESSENTIAL) | 227 | PROVIDES, RDEPENDS, RCONFLICTS, MULTIARCH, ESSENTIAL) |
307 | 228 | from autopkgtest import AutoPackageTest, ADT_PASS, ADT_EXCUSES_LABELS | 228 | from autopkgtest import AutoPackageTest, ADT_PASS, ADT_EXCUSES_LABELS |
308 | 229 | from boottest import BootTest | ||
309 | 230 | |||
310 | 229 | 231 | ||
311 | 230 | __author__ = 'Fabio Tranchitella and the Debian Release Team' | 232 | __author__ = 'Fabio Tranchitella and the Debian Release Team' |
312 | 231 | __version__ = '2.0' | 233 | __version__ = '2.0' |
313 | @@ -1366,6 +1368,7 @@ | |||
314 | 1366 | # the starting point is that we will update the candidate and run autopkgtests | 1368 | # the starting point is that we will update the candidate and run autopkgtests |
315 | 1367 | update_candidate = True | 1369 | update_candidate = True |
316 | 1368 | run_autopkgtest = True | 1370 | run_autopkgtest = True |
317 | 1371 | run_boottest = True | ||
318 | 1369 | 1372 | ||
319 | 1370 | # if the version in unstable is older, then stop here with a warning in the excuse and return False | 1373 | # if the version in unstable is older, then stop here with a warning in the excuse and return False |
320 | 1371 | if source_t and apt_pkg.version_compare(source_u[VERSION], source_t[VERSION]) < 0: | 1374 | if source_t and apt_pkg.version_compare(source_u[VERSION], source_t[VERSION]) < 0: |
321 | @@ -1379,6 +1382,7 @@ | |||
322 | 1379 | excuse.addhtml("%s source package doesn't exist" % (src)) | 1382 | excuse.addhtml("%s source package doesn't exist" % (src)) |
323 | 1380 | update_candidate = False | 1383 | update_candidate = False |
324 | 1381 | run_autopkgtest = False | 1384 | run_autopkgtest = False |
325 | 1385 | run_boottest = False | ||
326 | 1382 | 1386 | ||
327 | 1383 | # retrieve the urgency for the upload, ignoring it if this is a NEW package (not present in testing) | 1387 | # retrieve the urgency for the upload, ignoring it if this is a NEW package (not present in testing) |
328 | 1384 | urgency = self.urgencies.get(src, self.options.default_urgency) | 1388 | urgency = self.urgencies.get(src, self.options.default_urgency) |
329 | @@ -1397,6 +1401,7 @@ | |||
330 | 1397 | excuse.addreason("remove") | 1401 | excuse.addreason("remove") |
331 | 1398 | update_candidate = False | 1402 | update_candidate = False |
332 | 1399 | run_autopkgtest = False | 1403 | run_autopkgtest = False |
333 | 1404 | run_boottest = False | ||
334 | 1400 | 1405 | ||
335 | 1401 | # check if there is a `block' or `block-udeb' hint for this package, or a `block-all source' hint | 1406 | # check if there is a `block' or `block-udeb' hint for this package, or a `block-all source' hint |
336 | 1402 | blocked = {} | 1407 | blocked = {} |
337 | @@ -1476,6 +1481,7 @@ | |||
338 | 1476 | else: | 1481 | else: |
339 | 1477 | update_candidate = False | 1482 | update_candidate = False |
340 | 1478 | run_autopkgtest = False | 1483 | run_autopkgtest = False |
341 | 1484 | run_boottest = False | ||
342 | 1479 | excuse.addreason("age") | 1485 | excuse.addreason("age") |
343 | 1480 | 1486 | ||
344 | 1481 | if suite in ['pu', 'tpu']: | 1487 | if suite in ['pu', 'tpu']: |
345 | @@ -1511,6 +1517,8 @@ | |||
346 | 1511 | update_candidate = False | 1517 | update_candidate = False |
347 | 1512 | if arch in self.options.adt_arches.split(): | 1518 | if arch in self.options.adt_arches.split(): |
348 | 1513 | run_autopkgtest = False | 1519 | run_autopkgtest = False |
349 | 1520 | if arch in self.options.boottest_arches.split(): | ||
350 | 1521 | run_boottest = False | ||
351 | 1514 | excuse.addreason("arch") | 1522 | excuse.addreason("arch") |
352 | 1515 | excuse.addreason("arch-%s" % arch) | 1523 | excuse.addreason("arch-%s" % arch) |
353 | 1516 | excuse.addreason("build-arch") | 1524 | excuse.addreason("build-arch") |
354 | @@ -1553,6 +1561,8 @@ | |||
355 | 1553 | update_candidate = False | 1561 | update_candidate = False |
356 | 1554 | if arch in self.options.adt_arches.split(): | 1562 | if arch in self.options.adt_arches.split(): |
357 | 1555 | run_autopkgtest = False | 1563 | run_autopkgtest = False |
358 | 1564 | if arch in self.options.boottest_arches.split(): | ||
359 | 1565 | run_boottest = False | ||
360 | 1556 | 1566 | ||
361 | 1557 | # if there are out-of-date packages, warn about them in the excuse and set update_candidate | 1567 | # if there are out-of-date packages, warn about them in the excuse and set update_candidate |
362 | 1558 | # to False to block the update; if the architecture where the package is out-of-date is | 1568 | # to False to block the update; if the architecture where the package is out-of-date is |
363 | @@ -1579,6 +1589,8 @@ | |||
364 | 1579 | update_candidate = False | 1589 | update_candidate = False |
365 | 1580 | if arch in self.options.adt_arches.split(): | 1590 | if arch in self.options.adt_arches.split(): |
366 | 1581 | run_autopkgtest = False | 1591 | run_autopkgtest = False |
367 | 1592 | if arch in self.options.boottest_arches.split(): | ||
368 | 1593 | run_boottest = False | ||
369 | 1582 | excuse.addreason("arch") | 1594 | excuse.addreason("arch") |
370 | 1583 | excuse.addreason("arch-%s" % arch) | 1595 | excuse.addreason("arch-%s" % arch) |
371 | 1584 | if uptodatebins: | 1596 | if uptodatebins: |
372 | @@ -1596,11 +1608,13 @@ | |||
373 | 1596 | excuse.addreason("no-binaries") | 1608 | excuse.addreason("no-binaries") |
374 | 1597 | update_candidate = False | 1609 | update_candidate = False |
375 | 1598 | run_autopkgtest = False | 1610 | run_autopkgtest = False |
376 | 1611 | run_boottest = False | ||
377 | 1599 | elif not built_anywhere: | 1612 | elif not built_anywhere: |
378 | 1600 | excuse.addhtml("%s has no up-to-date binaries on any arch" % src) | 1613 | excuse.addhtml("%s has no up-to-date binaries on any arch" % src) |
379 | 1601 | excuse.addreason("no-binaries") | 1614 | excuse.addreason("no-binaries") |
380 | 1602 | update_candidate = False | 1615 | update_candidate = False |
381 | 1603 | run_autopkgtest = False | 1616 | run_autopkgtest = False |
382 | 1617 | run_boottest = False | ||
383 | 1604 | 1618 | ||
384 | 1605 | # if the suite is unstable, then we have to check the release-critical bug lists before | 1619 | # if the suite is unstable, then we have to check the release-critical bug lists before |
385 | 1606 | # updating testing; if the unstable package has RC bugs that do not apply to the testing | 1620 | # updating testing; if the unstable package has RC bugs that do not apply to the testing |
386 | @@ -1633,6 +1647,7 @@ | |||
387 | 1633 | ["<a href=\"http://bugs.debian.org/%s\">#%s</a>" % (urllib.quote(a), a) for a in new_bugs]))) | 1647 | ["<a href=\"http://bugs.debian.org/%s\">#%s</a>" % (urllib.quote(a), a) for a in new_bugs]))) |
388 | 1634 | update_candidate = False | 1648 | update_candidate = False |
389 | 1635 | run_autopkgtest = False | 1649 | run_autopkgtest = False |
390 | 1650 | run_boottest = False | ||
391 | 1636 | excuse.addreason("buggy") | 1651 | excuse.addreason("buggy") |
392 | 1637 | 1652 | ||
393 | 1638 | if len(old_bugs) > 0: | 1653 | if len(old_bugs) > 0: |
394 | @@ -1651,6 +1666,7 @@ | |||
395 | 1651 | excuse.force() | 1666 | excuse.force() |
396 | 1652 | update_candidate = True | 1667 | update_candidate = True |
397 | 1653 | run_autopkgtest = True | 1668 | run_autopkgtest = True |
398 | 1669 | run_boottest = True | ||
399 | 1654 | 1670 | ||
400 | 1655 | # if the package can be updated, it is a valid candidate | 1671 | # if the package can be updated, it is a valid candidate |
401 | 1656 | if update_candidate: | 1672 | if update_candidate: |
402 | @@ -1660,6 +1676,7 @@ | |||
403 | 1660 | # TODO | 1676 | # TODO |
404 | 1661 | excuse.addhtml("Not considered") | 1677 | excuse.addhtml("Not considered") |
405 | 1662 | excuse.run_autopkgtest = run_autopkgtest | 1678 | excuse.run_autopkgtest = run_autopkgtest |
406 | 1679 | excuse.run_boottest = run_boottest | ||
407 | 1663 | 1680 | ||
408 | 1664 | self.excuses.append(excuse) | 1681 | self.excuses.append(excuse) |
409 | 1665 | return update_candidate | 1682 | return update_candidate |
410 | @@ -1883,6 +1900,88 @@ | |||
411 | 1883 | e.addreason("autopkgtest") | 1900 | e.addreason("autopkgtest") |
412 | 1884 | e.is_valid = False | 1901 | e.is_valid = False |
413 | 1885 | 1902 | ||
414 | 1903 | if (getattr(self.options, "boottest_enable", "no") == "yes" and | ||
415 | 1904 | self.options.series): | ||
416 | 1905 | # trigger 'boottest'ing for valid candidates. | ||
417 | 1906 | boottest_debug = getattr( | ||
418 | 1907 | self.options, "boottest_debug", "no") == "yes" | ||
419 | 1908 | boottest = BootTest( | ||
420 | 1909 | self, self.options.distribution, self.options.series, | ||
421 | 1910 | debug=boottest_debug) | ||
422 | 1911 | boottest_excuses = [] | ||
423 | 1912 | for excuse in self.excuses: | ||
424 | 1913 | # Skip already invalid excuses. | ||
425 | 1914 | if not excuse.run_boottest: | ||
426 | 1915 | continue | ||
427 | 1916 | # Also skip removals, binary-only candidates, proposed-updates | ||
428 | 1917 | # and unknown versions. | ||
429 | 1918 | if (excuse.name.startswith("-") or | ||
430 | 1919 | "/" in excuse.name or | ||
431 | 1920 | "_" in excuse.name or | ||
432 | 1921 | excuse.ver[1] == "-"): | ||
433 | 1922 | continue | ||
434 | 1923 | # Allows hints to skip boottest attempts | ||
435 | 1924 | hints = self.hints.search( | ||
436 | 1925 | 'force-skiptest', package=excuse.name) | ||
437 | 1926 | forces = [x for x in hints | ||
438 | 1927 | if same_source(excuse.ver[1], x.version)] | ||
439 | 1928 | if forces: | ||
440 | 1929 | excuse.addhtml( | ||
441 | 1930 | "boottest skipped from hints by %s" % forces[0].user) | ||
442 | 1931 | continue | ||
443 | 1932 | # Only sources whitelisted in the boottest context should | ||
444 | 1933 | # be tested (currently only sources building phone binaries). | ||
445 | 1934 | if not boottest.needs_test(excuse.name, excuse.ver[1]): | ||
446 | 1935 | # Silently skipping. | ||
447 | 1936 | continue | ||
448 | 1937 | # Okay, aggregate required boottests requests. | ||
449 | 1938 | boottest_excuses.append(excuse) | ||
450 | 1939 | boottest.request([(e.name, e.ver[1]) for e in boottest_excuses]) | ||
451 | 1940 | # Dry-run avoids data exchange with external systems. | ||
452 | 1941 | if not self.options.dry_run: | ||
453 | 1942 | boottest.submit() | ||
454 | 1943 | boottest.collect() | ||
455 | 1944 | # Boottest Jenkins views location. | ||
456 | 1945 | jenkins_public = "https://jenkins.qa.ubuntu.com/job" | ||
457 | 1946 | jenkins_private = ( | ||
458 | 1947 | "http://d-jenkins.ubuntu-ci:8080/view/%s/view/BootTest/job" % | ||
459 | 1948 | self.options.series.title()) | ||
460 | 1949 | # Update excuses from the boottest context. | ||
461 | 1950 | for excuse in boottest_excuses: | ||
462 | 1951 | status = boottest.get_status(excuse.name, excuse.ver[1]) | ||
463 | 1952 | label = BootTest.EXCUSE_LABELS.get(status, 'UNKNOWN STATUS') | ||
464 | 1953 | public_url = "%s/%s-boottest-%s/lastBuild" % ( | ||
465 | 1954 | jenkins_public, self.options.series, | ||
466 | 1955 | excuse.name.replace("+", "-")) | ||
467 | 1956 | private_url = "%s/%s-boottest-%s/lastBuild" % ( | ||
468 | 1957 | jenkins_private, self.options.series, | ||
469 | 1958 | excuse.name.replace("+", "-")) | ||
470 | 1959 | excuse.addhtml( | ||
471 | 1960 | "Boottest result: %s (Jenkins: <a href=\"%s\">public</a>" | ||
472 | 1961 | ", <a href=\"%s\">private</a>)" % ( | ||
473 | 1962 | label, public_url, private_url)) | ||
474 | 1963 | # Allows hints to force boottest failures/attempts | ||
475 | 1964 | # to be ignored. | ||
476 | 1965 | hints = self.hints.search('force', package=excuse.name) | ||
477 | 1966 | hints.extend( | ||
478 | 1967 | self.hints.search('force-badtest', package=excuse.name)) | ||
479 | 1968 | forces = [x for x in hints | ||
480 | 1969 | if same_source(excuse.ver[1], x.version)] | ||
481 | 1970 | if forces: | ||
482 | 1971 | excuse.addhtml( | ||
483 | 1972 | "Should wait for %s %s boottest, but forced by " | ||
484 | 1973 | "%s" % (excuse.name, excuse.ver[1], | ||
485 | 1974 | forces[0].user)) | ||
486 | 1975 | continue | ||
487 | 1976 | # Block promotion if any boottests attempt has failed or | ||
488 | 1977 | # still in progress. | ||
489 | 1978 | if status not in BootTest.VALID_STATUSES: | ||
490 | 1979 | excuse.addhtml("Not considered") | ||
491 | 1980 | excuse.addreason("boottest") | ||
492 | 1981 | excuse.is_valid = False | ||
493 | 1982 | upgrade_me.remove(excuse.name) | ||
494 | 1983 | unconsidered.append(excuse.name) | ||
495 | 1984 | |||
496 | 1886 | # invalidate impossible excuses | 1985 | # invalidate impossible excuses |
497 | 1887 | for e in self.excuses: | 1986 | for e in self.excuses: |
498 | 1888 | # parts[0] == package name | 1987 | # parts[0] == package name |
499 | 1889 | 1988 | ||
500 | === modified file 'excuse.py' | |||
501 | --- excuse.py 2014-12-10 10:36:01 +0000 | |||
502 | +++ excuse.py 2015-02-18 13:50:50 +0000 | |||
503 | @@ -1,6 +1,6 @@ | |||
504 | 1 | # -*- coding: utf-8 -*- | 1 | # -*- coding: utf-8 -*- |
505 | 2 | 2 | ||
507 | 3 | # Copyright (C) 2001-2004 Anthony Towns <ajt@debian.org> | 3 | # Copyright (C) 2006, 2011-2015 Anthony Towns <ajt@debian.org> |
508 | 4 | # Andreas Barth <aba@debian.org> | 4 | # Andreas Barth <aba@debian.org> |
509 | 5 | # Fabio Tranchitella <kobold@debian.org> | 5 | # Fabio Tranchitella <kobold@debian.org> |
510 | 6 | 6 | ||
511 | @@ -52,6 +52,7 @@ | |||
512 | 52 | self._dontinvalidate = False | 52 | self._dontinvalidate = False |
513 | 53 | self.forced = False | 53 | self.forced = False |
514 | 54 | self.run_autopkgtest = False | 54 | self.run_autopkgtest = False |
515 | 55 | self.run_boottest = False | ||
516 | 55 | self.distribution = "ubuntu" | 56 | self.distribution = "ubuntu" |
517 | 56 | 57 | ||
518 | 57 | self.invalid_deps = [] | 58 | self.invalid_deps = [] |
519 | 58 | 59 | ||
520 | === modified file 'tests/__init__.py' | |||
521 | --- tests/__init__.py 2015-02-18 13:50:50 +0000 | |||
522 | +++ tests/__init__.py 2015-02-18 13:50:50 +0000 | |||
523 | @@ -135,6 +135,11 @@ | |||
524 | 135 | def tearDown(self): | 135 | def tearDown(self): |
525 | 136 | del self.data | 136 | del self.data |
526 | 137 | 137 | ||
527 | 138 | def restore_config(self, content): | ||
528 | 139 | """Helper for restoring configuration contents on cleanup.""" | ||
529 | 140 | with open(self.britney_conf, 'w') as fp: | ||
530 | 141 | fp.write(content) | ||
531 | 142 | |||
532 | 138 | def run_britney(self, args=[]): | 143 | def run_britney(self, args=[]): |
533 | 139 | '''Run britney. | 144 | '''Run britney. |
534 | 140 | 145 | ||
535 | 141 | 146 | ||
536 | === modified file 'tests/test_autopkgtest.py' | |||
537 | --- tests/test_autopkgtest.py 2015-02-18 13:50:50 +0000 | |||
538 | +++ tests/test_autopkgtest.py 2015-02-18 13:50:50 +0000 | |||
539 | @@ -31,6 +31,16 @@ | |||
540 | 31 | def setUp(self): | 31 | def setUp(self): |
541 | 32 | super(TestAutoPkgTest, self).setUp() | 32 | super(TestAutoPkgTest, self).setUp() |
542 | 33 | 33 | ||
543 | 34 | # Mofify configuration according to the test context. | ||
544 | 35 | with open(self.britney_conf, 'r') as fp: | ||
545 | 36 | original_config = fp.read() | ||
546 | 37 | # Disable boottests. | ||
547 | 38 | new_config = original_config.replace( | ||
548 | 39 | 'BOOTTEST_ENABLE = yes', 'BOOTTEST_ENABLE = no') | ||
549 | 40 | with open(self.britney_conf, 'w') as fp: | ||
550 | 41 | fp.write(new_config) | ||
551 | 42 | self.addCleanup(self.restore_config, original_config) | ||
552 | 43 | |||
553 | 34 | # fake adt-britney script | 44 | # fake adt-britney script |
554 | 35 | self.adt_britney = os.path.join( | 45 | self.adt_britney = os.path.join( |
555 | 36 | self.data.home, 'auto-package-testing', 'jenkins', 'adt-britney') | 46 | self.data.home, 'auto-package-testing', 'jenkins', 'adt-britney') |
556 | 37 | 47 | ||
557 | === added file 'tests/test_boottest.py' | |||
558 | --- tests/test_boottest.py 1970-01-01 00:00:00 +0000 | |||
559 | +++ tests/test_boottest.py 2015-02-18 13:50:50 +0000 | |||
560 | @@ -0,0 +1,411 @@ | |||
561 | 1 | #!/usr/bin/python | ||
562 | 2 | # (C) 2014 Canonical Ltd. | ||
563 | 3 | # | ||
564 | 4 | # This program is free software; you can redistribute it and/or modify | ||
565 | 5 | # it under the terms of the GNU General Public License as published by | ||
566 | 6 | # the Free Software Foundation; either version 2 of the License, or | ||
567 | 7 | # (at your option) any later version. | ||
568 | 8 | |||
569 | 9 | import mock | ||
570 | 10 | import os | ||
571 | 11 | import shutil | ||
572 | 12 | import sys | ||
573 | 13 | import tempfile | ||
574 | 14 | import unittest | ||
575 | 15 | |||
576 | 16 | |||
577 | 17 | PROJECT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) | ||
578 | 18 | sys.path.insert(0, PROJECT_DIR) | ||
579 | 19 | |||
580 | 20 | import boottest | ||
581 | 21 | from tests import TestBase | ||
582 | 22 | |||
583 | 23 | |||
584 | 24 | def create_manifest(manifest_dir, lines): | ||
585 | 25 | """Helper function for writing touch image manifests.""" | ||
586 | 26 | os.makedirs(manifest_dir) | ||
587 | 27 | with open(os.path.join(manifest_dir, 'manifest'), 'w') as fd: | ||
588 | 28 | fd.write('\n'.join(lines)) | ||
589 | 29 | |||
590 | 30 | |||
591 | 31 | class FakeResponse(object): | ||
592 | 32 | |||
593 | 33 | def __init__(self, code=404, content=''): | ||
594 | 34 | self.code = code | ||
595 | 35 | self.content = content | ||
596 | 36 | |||
597 | 37 | def read(self): | ||
598 | 38 | return self.content | ||
599 | 39 | |||
600 | 40 | |||
601 | 41 | class TestTouchManifest(unittest.TestCase): | ||
602 | 42 | |||
603 | 43 | def setUp(self): | ||
604 | 44 | super(TestTouchManifest, self).setUp() | ||
605 | 45 | self.path = tempfile.mkdtemp(prefix='boottest') | ||
606 | 46 | os.chdir(self.path) | ||
607 | 47 | self.imagesdir = os.path.join(self.path, 'boottest/images') | ||
608 | 48 | os.makedirs(self.imagesdir) | ||
609 | 49 | self.addCleanup(shutil.rmtree, self.path) | ||
610 | 50 | _p = mock.patch('urllib.urlopen') | ||
611 | 51 | self.mocked_urlopen = _p.start() | ||
612 | 52 | self.mocked_urlopen.side_effect = [FakeResponse(code=404),] | ||
613 | 53 | self.addCleanup(_p.stop) | ||
614 | 54 | self.fetch_retries_orig = boottest.FETCH_RETRIES | ||
615 | 55 | def restore_fetch_retries(): | ||
616 | 56 | boottest.FETCH_RETRIES = self.fetch_retries_orig | ||
617 | 57 | boottest.FETCH_RETRIES = 0 | ||
618 | 58 | self.addCleanup(restore_fetch_retries) | ||
619 | 59 | |||
620 | 60 | def test_missing(self): | ||
621 | 61 | # Missing manifest file silently results in empty contents. | ||
622 | 62 | manifest = boottest.TouchManifest('I-dont-exist', 'vivid') | ||
623 | 63 | self.assertEqual([], manifest._manifest) | ||
624 | 64 | self.assertNotIn('foo', manifest) | ||
625 | 65 | |||
626 | 66 | def test_fetch(self): | ||
627 | 67 | # Missing manifest file is fetched dynamically | ||
628 | 68 | self.mocked_urlopen.side_effect = [ | ||
629 | 69 | FakeResponse(code=200, content='foo 1.0'), | ||
630 | 70 | ] | ||
631 | 71 | manifest = boottest.TouchManifest('ubuntu-touch', 'vivid') | ||
632 | 72 | self.assertNotEqual([], manifest._manifest) | ||
633 | 73 | |||
634 | 74 | def test_fetch_disabled(self): | ||
635 | 75 | # Manifest auto-fetching can be disabled. | ||
636 | 76 | manifest = boottest.TouchManifest('ubuntu-touch', 'vivid', fetch=False) | ||
637 | 77 | self.mocked_urlopen.assert_not_called() | ||
638 | 78 | self.assertEqual([], manifest._manifest) | ||
639 | 79 | |||
640 | 80 | def test_fetch_fails(self): | ||
641 | 81 | project = 'fake' | ||
642 | 82 | series = 'fake' | ||
643 | 83 | manifest_dir = os.path.join(self.imagesdir, project, series) | ||
644 | 84 | manifest_lines = [ | ||
645 | 85 | 'foo:armhf 1~beta1', | ||
646 | 86 | ] | ||
647 | 87 | create_manifest(manifest_dir, manifest_lines) | ||
648 | 88 | manifest = boottest.TouchManifest(project, series) | ||
649 | 89 | self.assertEqual(1, len(manifest._manifest)) | ||
650 | 90 | self.assertIn('foo', manifest) | ||
651 | 91 | |||
652 | 92 | def test_fetch_exception(self): | ||
653 | 93 | self.mocked_urlopen.side_effect = [IOError("connection refused")] | ||
654 | 94 | manifest = boottest.TouchManifest('not-real', 'not-real') | ||
655 | 95 | self.assertEqual(0, len(manifest._manifest)) | ||
656 | 96 | |||
657 | 97 | def test_simple(self): | ||
658 | 98 | # Existing manifest file allows callsites to properly check presence. | ||
659 | 99 | manifest_dir = os.path.join(self.imagesdir, 'ubuntu/vivid') | ||
660 | 100 | manifest_lines = [ | ||
661 | 101 | 'bar 1234', | ||
662 | 102 | 'foo:armhf 1~beta1', | ||
663 | 103 | 'boing1-1.2\t666', | ||
664 | 104 | 'click:com.ubuntu.shorts 0.2.346' | ||
665 | 105 | ] | ||
666 | 106 | create_manifest(manifest_dir, manifest_lines) | ||
667 | 107 | |||
668 | 108 | manifest = boottest.TouchManifest('ubuntu', 'vivid') | ||
669 | 109 | # We can dig deeper on the manifest package names list ... | ||
670 | 110 | self.assertEqual( | ||
671 | 111 | ['bar', 'boing1-1.2', 'foo'], manifest._manifest) | ||
672 | 112 | # but the '<name> in manifest' API reads better. | ||
673 | 113 | self.assertIn('foo', manifest) | ||
674 | 114 | self.assertIn('boing1-1.2', manifest) | ||
675 | 115 | self.assertNotIn('baz', manifest) | ||
676 | 116 | # 'click' name is blacklisted due to the click package syntax. | ||
677 | 117 | self.assertNotIn('click', manifest) | ||
678 | 118 | |||
679 | 119 | |||
680 | 120 | class TestBoottestEnd2End(TestBase): | ||
681 | 121 | """End2End tests (calling `britney`) for the BootTest criteria.""" | ||
682 | 122 | |||
683 | 123 | def setUp(self): | ||
684 | 124 | super(TestBoottestEnd2End, self).setUp() | ||
685 | 125 | |||
686 | 126 | # Modify shared configuration file. | ||
687 | 127 | with open(self.britney_conf, 'r') as fp: | ||
688 | 128 | original_config = fp.read() | ||
689 | 129 | # Disable autopkgtests. | ||
690 | 130 | new_config = original_config.replace( | ||
691 | 131 | 'ADT_ENABLE = yes', 'ADT_ENABLE = no') | ||
692 | 132 | # Disable TouchManifest auto-fetching. | ||
693 | 133 | new_config = new_config.replace( | ||
694 | 134 | 'BOOTTEST_FETCH = yes', 'BOOTTEST_FETCH = no') | ||
695 | 135 | with open(self.britney_conf, 'w') as fp: | ||
696 | 136 | fp.write(new_config) | ||
697 | 137 | self.addCleanup(self.restore_config, original_config) | ||
698 | 138 | |||
699 | 139 | self.data.add('libc6', False, {'Architecture': 'armhf'}), | ||
700 | 140 | |||
701 | 141 | self.data.add( | ||
702 | 142 | 'libgreen1', | ||
703 | 143 | False, | ||
704 | 144 | {'Source': 'green', 'Architecture': 'armhf', | ||
705 | 145 | 'Depends': 'libc6 (>= 0.9)'}) | ||
706 | 146 | self.data.add( | ||
707 | 147 | 'green', | ||
708 | 148 | False, | ||
709 | 149 | {'Source': 'green', 'Architecture': 'armhf', | ||
710 | 150 | 'Depends': 'libc6 (>= 0.9), libgreen1'}) | ||
711 | 151 | self.create_manifest([ | ||
712 | 152 | 'green 1.0', | ||
713 | 153 | 'pyqt5:armhf 1.0', | ||
714 | 154 | 'signon 1.0' | ||
715 | 155 | ]) | ||
716 | 156 | |||
717 | 157 | def create_manifest(self, lines): | ||
718 | 158 | """Create a manifest for this britney run context.""" | ||
719 | 159 | path = os.path.join( | ||
720 | 160 | self.data.path, | ||
721 | 161 | 'boottest/images/ubuntu-touch/{}'.format(self.data.series)) | ||
722 | 162 | create_manifest(path, lines) | ||
723 | 163 | |||
724 | 164 | def make_boottest(self): | ||
725 | 165 | """Create a stub version of boottest-britney script.""" | ||
726 | 166 | script_path = os.path.expanduser( | ||
727 | 167 | "~/auto-package-testing/jenkins/boottest-britney") | ||
728 | 168 | os.makedirs(os.path.dirname(script_path)) | ||
729 | 169 | with open(script_path, 'w') as f: | ||
730 | 170 | f.write('''#!%(py)s | ||
731 | 171 | import argparse | ||
732 | 172 | import os | ||
733 | 173 | import shutil | ||
734 | 174 | import sys | ||
735 | 175 | |||
736 | 176 | template = """ | ||
737 | 177 | green 1.1~beta RUNNING | ||
738 | 178 | pyqt5-src 1.1~beta PASS | ||
739 | 179 | pyqt5-src 1.1 FAIL | ||
740 | 180 | signon 1.1 PASS | ||
741 | 181 | """ | ||
742 | 182 | |||
743 | 183 | def request(): | ||
744 | 184 | work_path = os.path.dirname(args.output) | ||
745 | 185 | os.makedirs(work_path) | ||
746 | 186 | shutil.copy(args.input, os.path.join(work_path, 'test_input')) | ||
747 | 187 | with open(args.output, 'w') as f: | ||
748 | 188 | f.write(template) | ||
749 | 189 | |||
750 | 190 | def submit(): | ||
751 | 191 | pass | ||
752 | 192 | |||
753 | 193 | def collect(): | ||
754 | 194 | with open(args.output, 'w') as f: | ||
755 | 195 | f.write(template) | ||
756 | 196 | |||
757 | 197 | p = argparse.ArgumentParser() | ||
758 | 198 | p.add_argument('-r') | ||
759 | 199 | p.add_argument('-c') | ||
760 | 200 | p.add_argument('-d', default=False, action='store_true') | ||
761 | 201 | p.add_argument('-P', default=False, action='store_true') | ||
762 | 202 | p.add_argument('-U', default=False, action='store_true') | ||
763 | 203 | |||
764 | 204 | sp = p.add_subparsers() | ||
765 | 205 | |||
766 | 206 | psubmit = sp.add_parser('submit') | ||
767 | 207 | psubmit.add_argument('input') | ||
768 | 208 | psubmit.set_defaults(func=submit) | ||
769 | 209 | |||
770 | 210 | prequest = sp.add_parser('request') | ||
771 | 211 | prequest.add_argument('-O', dest='output') | ||
772 | 212 | prequest.add_argument('input') | ||
773 | 213 | prequest.set_defaults(func=request) | ||
774 | 214 | |||
775 | 215 | pcollect = sp.add_parser('collect') | ||
776 | 216 | pcollect.add_argument('-O', dest='output') | ||
777 | 217 | pcollect.set_defaults(func=collect) | ||
778 | 218 | |||
779 | 219 | args = p.parse_args() | ||
780 | 220 | args.func() | ||
781 | 221 | ''' % {'py': sys.executable}) | ||
782 | 222 | os.chmod(script_path, 0o755) | ||
783 | 223 | |||
784 | 224 | def do_test(self, context, expect=None, no_expect=None): | ||
785 | 225 | """Process the given package context and assert britney results.""" | ||
786 | 226 | for (pkg, fields) in context: | ||
787 | 227 | self.data.add(pkg, True, fields) | ||
788 | 228 | self.make_boottest() | ||
789 | 229 | (excuses, out) = self.run_britney() | ||
790 | 230 | #print('-------\nexcuses: %s\n-----' % excuses) | ||
791 | 231 | if expect: | ||
792 | 232 | for re in expect: | ||
793 | 233 | self.assertRegexpMatches(excuses, re) | ||
794 | 234 | if no_expect: | ||
795 | 235 | for re in no_expect: | ||
796 | 236 | self.assertNotRegexpMatches(excuses, re) | ||
797 | 237 | |||
798 | 238 | def test_runs(self): | ||
799 | 239 | # `Britney` runs and considers binary packages for boottesting | ||
800 | 240 | # when it is enabled in the configuration, only binaries needed | ||
801 | 241 | # in the phone image are considered for boottesting. | ||
802 | 242 | # The boottest status is presented along with its corresponding | ||
803 | 243 | # jenkins job urls for the public and the private servers. | ||
804 | 244 | # 'in progress' tests blocks package promotion. | ||
805 | 245 | context = [ | ||
806 | 246 | ('green', {'Source': 'green', 'Version': '1.1~beta', | ||
807 | 247 | 'Architecture': 'armhf', 'Depends': 'libc6 (>= 0.9)'}), | ||
808 | 248 | ('libgreen1', {'Source': 'green', 'Version': '1.1~beta', | ||
809 | 249 | 'Architecture': 'armhf', | ||
810 | 250 | 'Depends': 'libc6 (>= 0.9)'}), | ||
811 | 251 | ] | ||
812 | 252 | public_jenkins_url = ( | ||
813 | 253 | 'https://jenkins.qa.ubuntu.com/job/series-boottest-green/' | ||
814 | 254 | 'lastBuild') | ||
815 | 255 | private_jenkins_url = ( | ||
816 | 256 | 'http://d-jenkins.ubuntu-ci:8080/view/Series/view/BootTest/' | ||
817 | 257 | 'job/series-boottest-green/lastBuild') | ||
818 | 258 | self.do_test( | ||
819 | 259 | context, | ||
820 | 260 | [r'\bgreen\b.*>1</a> to .*>1.1~beta<', | ||
821 | 261 | r'<li>Boottest result: {} \(Jenkins: ' | ||
822 | 262 | r'<a href="{}">public</a>, <a href="{}">private</a>\)'.format( | ||
823 | 263 | boottest.BootTest.EXCUSE_LABELS['RUNNING'], | ||
824 | 264 | public_jenkins_url, private_jenkins_url), | ||
825 | 265 | '<li>Not considered']) | ||
826 | 266 | |||
827 | 267 | # The `boottest-britney` input (recorded for testing purposes), | ||
828 | 268 | # contains a line matching the requested boottest attempt. | ||
829 | 269 | # '<source> <version>\n' | ||
830 | 270 | test_input_path = os.path.join( | ||
831 | 271 | self.data.path, 'boottest/work/test_input') | ||
832 | 272 | self.assertEqual( | ||
833 | 273 | ['green 1.1~beta\n'], open(test_input_path).readlines()) | ||
834 | 274 | |||
835 | 275 | def test_pass(self): | ||
836 | 276 | # `Britney` updates boottesting information in excuses when the | ||
837 | 277 | # package test pass and marks the package as a valid candidate for | ||
838 | 278 | # promotion. | ||
839 | 279 | context = [] | ||
840 | 280 | context.append( | ||
841 | 281 | ('signon', {'Version': '1.1', 'Architecture': 'armhf'})) | ||
842 | 282 | self.do_test( | ||
843 | 283 | context, | ||
844 | 284 | [r'\bsignon\b.*\(- to .*>1.1<', | ||
845 | 285 | '<li>Boottest result: {}'.format( | ||
846 | 286 | boottest.BootTest.EXCUSE_LABELS['PASS']), | ||
847 | 287 | '<li>Valid candidate']) | ||
848 | 288 | |||
849 | 289 | def test_fail(self): | ||
850 | 290 | # `Britney` updates boottesting information in excuses when the | ||
851 | 291 | # package test fails and blocks the package promotion | ||
852 | 292 | # ('Not considered.') | ||
853 | 293 | context = [] | ||
854 | 294 | context.append( | ||
855 | 295 | ('pyqt5', {'Source': 'pyqt5-src', 'Version': '1.1', | ||
856 | 296 | 'Architecture': 'all'})) | ||
857 | 297 | self.do_test( | ||
858 | 298 | context, | ||
859 | 299 | [r'\bpyqt5-src\b.*\(- to .*>1.1<', | ||
860 | 300 | '<li>Boottest result: {}'.format( | ||
861 | 301 | boottest.BootTest.EXCUSE_LABELS['FAIL']), | ||
862 | 302 | '<li>Not considered']) | ||
863 | 303 | |||
864 | 304 | def test_unknown(self): | ||
865 | 305 | # `Britney` does not block on missing boottest results for a | ||
866 | 306 | # particular source/version, in this case pyqt5-src_1.2 (not | ||
867 | 307 | # listed in the testing result history). Instead it renders | ||
868 | 308 | # excuses with 'UNKNOWN STATUS' and links to the corresponding | ||
869 | 309 | # jenkins jobs for further investigation. Source promotion is | ||
870 | 310 | # blocked, though. | ||
871 | 311 | context = [ | ||
872 | 312 | ('pyqt5', {'Source': 'pyqt5-src', 'Version': '1.2', | ||
873 | 313 | 'Architecture': 'armhf'})] | ||
874 | 314 | self.do_test( | ||
875 | 315 | context, | ||
876 | 316 | [r'\bpyqt5-src\b.*\(- to .*>1.2<', | ||
877 | 317 | r'<li>Boottest result: UNKNOWN STATUS \(Jenkins: .*\)', | ||
878 | 318 | '<li>Not considered']) | ||
879 | 319 | |||
880 | 320 | def create_hint(self, username, content): | ||
881 | 321 | """Populates a hint file for the given 'username' with 'content'.""" | ||
882 | 322 | hints_path = os.path.join( | ||
883 | 323 | self.data.path, | ||
884 | 324 | 'data/{}-proposed/Hints/{}'.format(self.data.series, username)) | ||
885 | 325 | with open(hints_path, 'w') as fd: | ||
886 | 326 | fd.write(content) | ||
887 | 327 | |||
888 | 328 | def test_skipped_by_hints(self): | ||
889 | 329 | # `Britney` allows boottests to be skipped by hinting the | ||
890 | 330 | # corresponding source with 'force-skiptest'. The boottest | ||
891 | 331 | # attempt will not be requested. | ||
892 | 332 | context = [ | ||
893 | 333 | ('pyqt5', {'Source': 'pyqt5-src', 'Version': '1.1', | ||
894 | 334 | 'Architecture': 'all'}), | ||
895 | 335 | ] | ||
896 | 336 | self.create_hint('cjwatson', 'force-skiptest pyqt5-src/1.1') | ||
897 | 337 | self.do_test( | ||
898 | 338 | context, | ||
899 | 339 | [r'\bpyqt5-src\b.*\(- to .*>1.1<', | ||
900 | 340 | '<li>boottest skipped from hints by cjwatson', | ||
901 | 341 | '<li>Valid candidate']) | ||
902 | 342 | |||
903 | 343 | def test_fail_but_forced_by_hints(self): | ||
904 | 344 | # `Britney` allows boottests results to be ignored by hinting the | ||
905 | 345 | # corresponding source with 'force' or 'force-badtest'. The boottest | ||
906 | 346 | # attempt will still be requested and its results would be considered | ||
907 | 347 | # for other non-forced sources. | ||
908 | 348 | context = [ | ||
909 | 349 | ('pyqt5', {'Source': 'pyqt5-src', 'Version': '1.1', | ||
910 | 350 | 'Architecture': 'all'}), | ||
911 | 351 | ] | ||
912 | 352 | self.create_hint('cjwatson', 'force pyqt5-src/1.1') | ||
913 | 353 | self.do_test( | ||
914 | 354 | context, | ||
915 | 355 | [r'\bpyqt5-src\b.*\(- to .*>1.1<', | ||
916 | 356 | '<li>Boottest result: {}'.format( | ||
917 | 357 | boottest.BootTest.EXCUSE_LABELS['FAIL']), | ||
918 | 358 | '<li>Should wait for pyqt5-src 1.1 boottest, ' | ||
919 | 359 | 'but forced by cjwatson', | ||
920 | 360 | '<li>Valid candidate']) | ||
921 | 361 | |||
922 | 362 | def test_fail_but_ignored_by_hints(self): | ||
923 | 363 | # See `test_fail_but_forced_by_hints`. | ||
924 | 364 | context = [ | ||
925 | 365 | ('green', {'Source': 'green', 'Version': '1.1~beta', | ||
926 | 366 | 'Architecture': 'armhf', 'Depends': 'libc6 (>= 0.9)'}), | ||
927 | 367 | ] | ||
928 | 368 | self.create_hint('cjwatson', 'force-badtest green/1.1~beta') | ||
929 | 369 | self.do_test( | ||
930 | 370 | context, | ||
931 | 371 | [r'\bgreen\b.*>1</a> to .*>1.1~beta<', | ||
932 | 372 | '<li>Boottest result: {}'.format( | ||
933 | 373 | boottest.BootTest.EXCUSE_LABELS['RUNNING']), | ||
934 | 374 | '<li>Should wait for green 1.1~beta boottest, but forced ' | ||
935 | 375 | 'by cjwatson', | ||
936 | 376 | '<li>Valid candidate']) | ||
937 | 377 | |||
938 | 378 | def test_skipped_not_on_phone(self): | ||
939 | 379 | # `Britney` updates boottesting information in excuses when the | ||
940 | 380 | # package was skipped and marks the package as a valid candidate for | ||
941 | 381 | # promotion, but no notice about 'boottest' is added to the excuse. | ||
942 | 382 | context = [] | ||
943 | 383 | context.append( | ||
944 | 384 | ('apache2', {'Source': 'apache2-src', 'Architecture': 'all', | ||
945 | 385 | 'Version': '2.4.8-1ubuntu1'})) | ||
946 | 386 | self.do_test( | ||
947 | 387 | context, | ||
948 | 388 | [r'\bapache2-src\b.*\(- to .*>2.4.8-1ubuntu1<', | ||
949 | 389 | '<li>Valid candidate'], | ||
950 | 390 | ['<li>Boottest result:'], | ||
951 | 391 | ) | ||
952 | 392 | |||
953 | 393 | def test_skipped_architecture_not_allowed(self): | ||
954 | 394 | # `Britney` does not trigger boottests for source not yet built on | ||
955 | 395 | # the allowed architectures. | ||
956 | 396 | self.data.add( | ||
957 | 397 | 'pyqt5', False, {'Source': 'pyqt5-src', 'Architecture': 'armhf'}) | ||
958 | 398 | context = [ | ||
959 | 399 | ('pyqt5', {'Source': 'pyqt5-src', 'Version': '1.1', | ||
960 | 400 | 'Architecture': 'amd64'}), | ||
961 | 401 | ] | ||
962 | 402 | self.do_test( | ||
963 | 403 | context, | ||
964 | 404 | [r'\bpyqt5-src\b.*>1</a> to .*>1.1<', | ||
965 | 405 | r'<li>missing build on .*>armhf</a>: pyqt5 \(from .*>1</a>\)', | ||
966 | 406 | '<li>Not considered']) | ||
967 | 407 | |||
968 | 408 | |||
969 | 409 | |||
970 | 410 | if __name__ == '__main__': | ||
971 | 411 | unittest.main() |
I have a couple of comments for clarification around the 'check'/'submit' interface. The bigger question is about using source packages instead of binaries.