Merge lp:~twom/launchpad-buildd/initial-docker-build-support into lp:launchpad-buildd
- initial-docker-build-support
- Merge into trunk
Status: | Merged |
---|---|
Merged at revision: | 413 |
Proposed branch: | lp:~twom/launchpad-buildd/initial-docker-build-support |
Merge into: | lp:launchpad-buildd |
Diff against target: |
1478 lines (+1200/-54) 14 files modified
debian/changelog (+10/-4) lpbuildd/buildd-slave.tac (+2/-0) lpbuildd/oci.py (+222/-0) lpbuildd/snap.py (+32/-29) lpbuildd/target/build_oci.py (+126/-0) lpbuildd/target/build_snap.py (+7/-15) lpbuildd/target/cli.py (+2/-0) lpbuildd/target/lxd.py (+6/-2) lpbuildd/target/snapbuildproxy.py (+41/-0) lpbuildd/target/tests/test_build_oci.py (+407/-0) lpbuildd/target/tests/test_build_snap.py (+1/-1) lpbuildd/target/tests/test_lxd.py (+4/-3) lpbuildd/tests/oci_tarball.py (+60/-0) lpbuildd/tests/test_oci.py (+280/-0) |
To merge this branch: | bzr merge lp:~twom/launchpad-buildd/initial-docker-build-support |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Tom Wardill (community) | Approve | ||
Colin Watson (community) | Approve | ||
Review via email: mp+369775@code.launchpad.net |
Commit message
Add initial docker build support
Description of the change
Add a builder for docker, creating an image following a supplied Dockerfile.
Save the image, extract it and then tar each component layer individually for returning/caching in launchpad.
Colin Watson (cjwatson) : | # |
- 393. By Tom Wardill
-
Use shell_escape, compress output images
- 394. By Tom Wardill
-
Collect .tar.gz
- 395. By Tom Wardill
-
Use a python script to save docker image to layers
- 396. By Tom Wardill
-
Move image extraction to gatherResults
- 397. By Tom Wardill
-
Refactor joint proxy handling
- 398. By Tom Wardill
-
Fix changelog diff noise
- 399. By Tom Wardill
-
Do more preprocessing, create digests file
- 400. By Tom Wardill
-
Include all diffs in digests and waiting files
- 401. By Tom Wardill
-
Add layer_id to digests now we're using every layer
- 402. By <email address hidden>
-
Use docker from apt, rather than from snap
- 403. By <email address hidden>
-
Rename Docker to OCI
- 404. By <email address hidden>
-
Merge upstream
Colin Watson (cjwatson) : | # |
- 405. By <email address hidden>
-
Remove conflict marker
- 406. By <email address hidden>
-
Use proxy, extract sha from right location
- 407. By <email address hidden>
-
Refactor proxy handling into better named mixin
- 408. By <email address hidden>
-
Refactor out adding docker proxy
- 409. By <email address hidden>
-
Include snap build proxy code
- 410. By <email address hidden>
-
Move to an on-the-fly tarball creation
- 411. By <email address hidden>
-
Argument is build_file, not file
Colin Watson (cjwatson) : | # |
Tom Wardill (twom) wrote : | # |
Tested with latest buildbehaviour branch, should now be functional.
Preview Diff
1 | === modified file 'debian/changelog' |
2 | --- debian/changelog 2020-01-06 15:03:55 +0000 |
3 | +++ debian/changelog 2020-02-26 10:52:47 +0000 |
4 | @@ -1,3 +1,9 @@ |
5 | +launchpad-buildd (187) UNRELEASED; urgency=medium |
6 | + |
7 | + * Prototype Docker image building support. |
8 | + |
9 | + -- Colin Watson <cjwatson@ubuntu.com> Wed, 05 Jun 2019 15:06:54 +0100 |
10 | + |
11 | launchpad-buildd (186) xenial; urgency=medium |
12 | |
13 | * Fix sbuildrc compatibility with xenial's sbuild. |
14 | @@ -32,7 +38,7 @@ |
15 | |
16 | [ Michael Hudson-Doyle ] |
17 | * Do not make assumptions about what device major number the device mapper |
18 | - is using. (LP: #1852518) |
19 | + is using. (LP: #1852518) |
20 | |
21 | -- Colin Watson <cjwatson@ubuntu.com> Tue, 26 Nov 2019 12:22:37 +0000 |
22 | |
23 | @@ -801,7 +807,7 @@ |
24 | memory at once (LP: #1227086). |
25 | |
26 | [ Adam Conrad ] |
27 | - * Tidy up log formatting of the "Already reaped..." message. |
28 | + * Tidy up log formatting of the "Already reaped..." message. |
29 | |
30 | -- Colin Watson <cjwatson@ubuntu.com> Fri, 27 Sep 2013 13:08:59 +0100 |
31 | |
32 | @@ -986,7 +992,7 @@ |
33 | launchpad-buildd (98) hardy; urgency=low |
34 | |
35 | * Add launchpad-buildd dependency on python-apt, as an accomodation for it |
36 | - being only a Recommends but actually required by python-debian. |
37 | + being only a Recommends but actually required by python-debian. |
38 | LP: #890834 |
39 | |
40 | -- Martin Pool <mbp@canonical.com> Wed, 16 Nov 2011 10:28:48 +1100 |
41 | @@ -1040,7 +1046,7 @@ |
42 | |
43 | launchpad-buildd (90) hardy; urgency=low |
44 | |
45 | - * debhelper is a Build-Depends because it is needed to run 'clean'. |
46 | + * debhelper is a Build-Depends because it is needed to run 'clean'. |
47 | * python-lpbuildd conflicts with launchpad-buildd << 88. |
48 | * Add and adjust build-arch, binary-arch, build-indep to match policy. |
49 | * Complies with stardards version 3.9.2. |
50 | |
51 | === modified file 'lpbuildd/buildd-slave.tac' |
52 | --- lpbuildd/buildd-slave.tac 2019-02-12 10:35:12 +0000 |
53 | +++ lpbuildd/buildd-slave.tac 2020-02-26 10:52:47 +0000 |
54 | @@ -23,6 +23,7 @@ |
55 | |
56 | from lpbuildd.binarypackage import BinaryPackageBuildManager |
57 | from lpbuildd.builder import XMLRPCBuilder |
58 | +from lpbuildd.oci import OCIBuildManager |
59 | from lpbuildd.livefs import LiveFilesystemBuildManager |
60 | from lpbuildd.log import RotatableFileLogObserver |
61 | from lpbuildd.snap import SnapBuildManager |
62 | @@ -45,6 +46,7 @@ |
63 | TranslationTemplatesBuildManager, 'translation-templates') |
64 | builder.registerManager(LiveFilesystemBuildManager, "livefs") |
65 | builder.registerManager(SnapBuildManager, "snap") |
66 | +builder.registerManager(OCIBuildManager, "oci") |
67 | |
68 | application = service.Application('Builder') |
69 | application.addComponent( |
70 | |
71 | === added file 'lpbuildd/oci.py' |
72 | --- lpbuildd/oci.py 1970-01-01 00:00:00 +0000 |
73 | +++ lpbuildd/oci.py 2020-02-26 10:52:47 +0000 |
74 | @@ -0,0 +1,222 @@ |
75 | +# Copyright 2019 Canonical Ltd. This software is licensed under the |
76 | +# GNU Affero General Public License version 3 (see the file LICENSE). |
77 | + |
78 | +from __future__ import print_function |
79 | + |
80 | +__metaclass__ = type |
81 | + |
82 | +import hashlib |
83 | +import json |
84 | +import os |
85 | +import tarfile |
86 | +import tempfile |
87 | + |
88 | +from six.moves.configparser import ( |
89 | + NoOptionError, |
90 | + NoSectionError, |
91 | + ) |
92 | + |
93 | +from lpbuildd.debian import ( |
94 | + DebianBuildManager, |
95 | + DebianBuildState, |
96 | + ) |
97 | +from lpbuildd.snap import SnapBuildProxyMixin |
98 | + |
99 | + |
100 | +RETCODE_SUCCESS = 0 |
101 | +RETCODE_FAILURE_INSTALL = 200 |
102 | +RETCODE_FAILURE_BUILD = 201 |
103 | + |
104 | + |
105 | +class OCIBuildState(DebianBuildState): |
106 | + BUILD_OCI = "BUILD_OCI" |
107 | + |
108 | + |
109 | +class OCIBuildManager(SnapBuildProxyMixin, DebianBuildManager): |
110 | + """Build an OCI Image.""" |
111 | + |
112 | + backend_name = "lxd" |
113 | + initial_build_state = OCIBuildState.BUILD_OCI |
114 | + |
115 | + @property |
116 | + def needs_sanitized_logs(self): |
117 | + return True |
118 | + |
119 | + def initiate(self, files, chroot, extra_args): |
120 | + """Initiate a build with a given set of files and chroot.""" |
121 | + self.name = extra_args["name"] |
122 | + self.branch = extra_args.get("branch") |
123 | + self.git_repository = extra_args.get("git_repository") |
124 | + self.git_path = extra_args.get("git_path") |
125 | + self.build_file = extra_args.get("build_file") |
126 | + self.proxy_url = extra_args.get("proxy_url") |
127 | + self.revocation_endpoint = extra_args.get("revocation_endpoint") |
128 | + self.proxy_service = None |
129 | + |
130 | + super(OCIBuildManager, self).initiate(files, chroot, extra_args) |
131 | + |
132 | + def doRunBuild(self): |
133 | + """Run the process to build the snap.""" |
134 | + args = [] |
135 | + args.extend(self.startProxy()) |
136 | + if self.revocation_endpoint: |
137 | + args.extend(["--revocation-endpoint", self.revocation_endpoint]) |
138 | + if self.branch is not None: |
139 | + args.extend(["--branch", self.branch]) |
140 | + if self.git_repository is not None: |
141 | + args.extend(["--git-repository", self.git_repository]) |
142 | + if self.git_path is not None: |
143 | + args.extend(["--git-path", self.git_path]) |
144 | + if self.build_file is not None: |
145 | + args.extend(["--build-file", self.build_file]) |
146 | + try: |
147 | + snap_store_proxy_url = self._builder._config.get( |
148 | + "proxy", "snapstore") |
149 | + args.extend(["--snap-store-proxy-url", snap_store_proxy_url]) |
150 | + except (NoSectionError, NoOptionError): |
151 | + pass |
152 | + args.append(self.name) |
153 | + self.runTargetSubProcess("build-oci", *args) |
154 | + |
155 | + def iterate_BUILD_OCI(self, retcode): |
156 | + """Finished building the OCI image.""" |
157 | + self.stopProxy() |
158 | + self.revokeProxyToken() |
159 | + if retcode == RETCODE_SUCCESS: |
160 | + print("Returning build status: OK") |
161 | + return self.deferGatherResults() |
162 | + elif (retcode >= RETCODE_FAILURE_INSTALL and |
163 | + retcode <= RETCODE_FAILURE_BUILD): |
164 | + if not self.alreadyfailed: |
165 | + self._builder.buildFail() |
166 | + print("Returning build status: Build failed.") |
167 | + self.alreadyfailed = True |
168 | + else: |
169 | + if not self.alreadyfailed: |
170 | + self._builder.builderFail() |
171 | + print("Returning build status: Builder failed.") |
172 | + self.alreadyfailed = True |
173 | + self.doReapProcesses(self._state) |
174 | + |
175 | + def iterateReap_BUILD_OCI(self, retcode): |
176 | + """Finished reaping after building the OCI image.""" |
177 | + self._state = DebianBuildState.UMOUNT |
178 | + self.doUnmounting() |
179 | + |
180 | + def _calculateLayerSha(self, layer_path): |
181 | + with open(layer_path, 'rb') as layer_tar: |
182 | + sha256_hash = hashlib.sha256() |
183 | + for byte_block in iter(lambda: layer_tar.read(4096), b""): |
184 | + sha256_hash.update(byte_block) |
185 | + digest = sha256_hash.hexdigest() |
186 | + return digest |
187 | + |
188 | + def _gatherManifestSection(self, section, extract_path, sha_directory): |
189 | + config_file_path = os.path.join(extract_path, section["Config"]) |
190 | + self._builder.addWaitingFile(config_file_path) |
191 | + with open(config_file_path, 'r') as config_fp: |
192 | + config = json.load(config_fp) |
193 | + diff_ids = config["rootfs"]["diff_ids"] |
194 | + digest_diff_map = {} |
195 | + for diff_id, layer_id in zip(diff_ids, section['Layers']): |
196 | + layer_id = layer_id.split('/')[0] |
197 | + diff_file = os.path.join(sha_directory, diff_id.split(':')[1]) |
198 | + layer_path = os.path.join( |
199 | + extract_path, "{}.tar.gz".format(layer_id)) |
200 | + self._builder.addWaitingFile(layer_path) |
201 | + # If we have a mapping between diff and existing digest, |
202 | + # this means this layer has been pulled from a remote. |
203 | + # We should maintain the same digest to achieve layer reuse |
204 | + if os.path.exists(diff_file): |
205 | + with open(diff_file, 'r') as diff_fp: |
206 | + diff = json.load(diff_fp) |
207 | + # We should be able to just take the first occurence, |
208 | + # as that will be the 'most parent' image |
209 | + digest = diff[0]["Digest"] |
210 | + source = diff[0]["SourceRepository"] |
211 | + # If the layer has been build locally, we need to generate the |
212 | + # digest and then set the source to empty |
213 | + else: |
214 | + source = "" |
215 | + digest = self._calculateLayerSha(layer_path) |
216 | + digest_diff_map[diff_id] = { |
217 | + "digest": digest, |
218 | + "source": source, |
219 | + "layer_id": layer_id |
220 | + } |
221 | + |
222 | + return digest_diff_map |
223 | + |
224 | + def gatherResults(self): |
225 | + """Gather the results of the build and add them to the file cache.""" |
226 | + extract_path = tempfile.mkdtemp(prefix=self.name) |
227 | + proc = self.backend.run( |
228 | + ['docker', 'save', self.name], |
229 | + get_output=True, universal_newlines=False, return_process=True) |
230 | + try: |
231 | + tar = tarfile.open(fileobj=proc.stdout, mode="r|") |
232 | + except Exception as e: |
233 | + print(e) |
234 | + |
235 | + current_dir = '' |
236 | + directory_tar = None |
237 | + try: |
238 | + # The tarfile is a stream and must be processed in order |
239 | + for file in tar: |
240 | + # Directories are just nodes, you can't extract the children |
241 | + # directly, so keep track of what dir we're in. |
242 | + if file.isdir(): |
243 | + current_dir = file.name |
244 | + if directory_tar: |
245 | + # Close the old directory if we have one |
246 | + directory_tar.close() |
247 | + # We're going to add the layer.tar to a gzip |
248 | + directory_tar = tarfile.open( |
249 | + os.path.join( |
250 | + extract_path, '{}.tar.gz'.format(file.name)), |
251 | + 'w|gz') |
252 | + if current_dir and file.name.endswith('layer.tar'): |
253 | + # This is the actual layer data, we want to add it to |
254 | + # the directory gzip |
255 | + file.name = file.name.split('/')[1] |
256 | + directory_tar.addfile(file, tar.extractfile(file)) |
257 | + elif current_dir and file.name.startswith(current_dir): |
258 | + # Other files that are in the layer directories, |
259 | + # we don't care about |
260 | + continue |
261 | + else: |
262 | + # If it's not in a directory, we need that |
263 | + tar.extract(file, extract_path) |
264 | + except Exception as e: |
265 | + print(e) |
266 | + |
267 | + # We need these mapping files |
268 | + sha_directory = tempfile.mkdtemp() |
269 | + sha_path = ('/var/lib/docker/image/' |
270 | + 'vfs/distribution/v2metadata-by-diffid/sha256/') |
271 | + sha_files = [x for x in self.backend.listdir(sha_path) |
272 | + if not x.startswith('.')] |
273 | + for file in sha_files: |
274 | + self.backend.copy_out( |
275 | + os.path.join(sha_path, file), |
276 | + os.path.join(sha_directory, file) |
277 | + ) |
278 | + |
279 | + # Parse the manifest for the other files we need |
280 | + manifest_path = os.path.join(extract_path, 'manifest.json') |
281 | + self._builder.addWaitingFile(manifest_path) |
282 | + with open(manifest_path) as manifest_fp: |
283 | + manifest = json.load(manifest_fp) |
284 | + |
285 | + digest_maps = [] |
286 | + try: |
287 | + for section in manifest: |
288 | + digest_maps.append( |
289 | + self._gatherManifestSection(section, extract_path, |
290 | + sha_directory)) |
291 | + digest_map_file = os.path.join(extract_path, 'digests.json') |
292 | + with open(digest_map_file, 'w') as digest_map_fp: |
293 | + json.dump(digest_maps, digest_map_fp) |
294 | + self._builder.addWaitingFile(digest_map_file) |
295 | + except Exception as e: |
296 | + print(e) |
297 | |
298 | === modified file 'lpbuildd/snap.py' |
299 | --- lpbuildd/snap.py 2019-10-30 12:31:53 +0000 |
300 | +++ lpbuildd/snap.py 2020-02-26 10:52:47 +0000 |
301 | @@ -239,35 +239,7 @@ |
302 | BUILD_SNAP = "BUILD_SNAP" |
303 | |
304 | |
305 | -class SnapBuildManager(DebianBuildManager): |
306 | - """Build a snap.""" |
307 | - |
308 | - backend_name = "lxd" |
309 | - initial_build_state = SnapBuildState.BUILD_SNAP |
310 | - |
311 | - @property |
312 | - def needs_sanitized_logs(self): |
313 | - return True |
314 | - |
315 | - def initiate(self, files, chroot, extra_args): |
316 | - """Initiate a build with a given set of files and chroot.""" |
317 | - self.name = extra_args["name"] |
318 | - self.channels = extra_args.get("channels", {}) |
319 | - self.build_request_id = extra_args.get("build_request_id") |
320 | - self.build_request_timestamp = extra_args.get( |
321 | - "build_request_timestamp") |
322 | - self.build_url = extra_args.get("build_url") |
323 | - self.branch = extra_args.get("branch") |
324 | - self.git_repository = extra_args.get("git_repository") |
325 | - self.git_path = extra_args.get("git_path") |
326 | - self.proxy_url = extra_args.get("proxy_url") |
327 | - self.revocation_endpoint = extra_args.get("revocation_endpoint") |
328 | - self.build_source_tarball = extra_args.get( |
329 | - "build_source_tarball", False) |
330 | - self.private = extra_args.get("private", False) |
331 | - self.proxy_service = None |
332 | - |
333 | - super(SnapBuildManager, self).initiate(files, chroot, extra_args) |
334 | +class SnapBuildProxyMixin(): |
335 | |
336 | def startProxy(self): |
337 | """Start the local snap proxy, if necessary.""" |
338 | @@ -308,6 +280,37 @@ |
339 | self._builder.log( |
340 | "Unable to revoke token for %s: %s" % (url.username, e)) |
341 | |
342 | + |
343 | +class SnapBuildManager(SnapBuildProxyMixin, DebianBuildManager): |
344 | + """Build a snap.""" |
345 | + |
346 | + backend_name = "lxd" |
347 | + initial_build_state = SnapBuildState.BUILD_SNAP |
348 | + |
349 | + @property |
350 | + def needs_sanitized_logs(self): |
351 | + return True |
352 | + |
353 | + def initiate(self, files, chroot, extra_args): |
354 | + """Initiate a build with a given set of files and chroot.""" |
355 | + self.name = extra_args["name"] |
356 | + self.channels = extra_args.get("channels", {}) |
357 | + self.build_request_id = extra_args.get("build_request_id") |
358 | + self.build_request_timestamp = extra_args.get( |
359 | + "build_request_timestamp") |
360 | + self.build_url = extra_args.get("build_url") |
361 | + self.branch = extra_args.get("branch") |
362 | + self.git_repository = extra_args.get("git_repository") |
363 | + self.git_path = extra_args.get("git_path") |
364 | + self.proxy_url = extra_args.get("proxy_url") |
365 | + self.revocation_endpoint = extra_args.get("revocation_endpoint") |
366 | + self.build_source_tarball = extra_args.get( |
367 | + "build_source_tarball", False) |
368 | + self.private = extra_args.get("private", False) |
369 | + self.proxy_service = None |
370 | + |
371 | + super(SnapBuildManager, self).initiate(files, chroot, extra_args) |
372 | + |
373 | def status(self): |
374 | status_path = get_build_path(self.home, self._buildid, "status") |
375 | try: |
376 | |
377 | === added file 'lpbuildd/target/build_oci.py' |
378 | --- lpbuildd/target/build_oci.py 1970-01-01 00:00:00 +0000 |
379 | +++ lpbuildd/target/build_oci.py 2020-02-26 10:52:47 +0000 |
380 | @@ -0,0 +1,126 @@ |
381 | +# Copyright 2019 Canonical Ltd. This software is licensed under the |
382 | +# GNU Affero General Public License version 3 (see the file LICENSE). |
383 | + |
384 | +from __future__ import print_function |
385 | + |
386 | +__metaclass__ = type |
387 | + |
388 | +from collections import OrderedDict |
389 | +import logging |
390 | +import os.path |
391 | +import sys |
392 | +import tempfile |
393 | +from textwrap import dedent |
394 | + |
395 | +from lpbuildd.target.operation import Operation |
396 | +from lpbuildd.target.snapbuildproxy import SnapBuildProxyOperationMixin |
397 | +from lpbuildd.target.snapstore import SnapStoreOperationMixin |
398 | +from lpbuildd.target.vcs import VCSOperationMixin |
399 | + |
400 | + |
401 | +RETCODE_FAILURE_INSTALL = 200 |
402 | +RETCODE_FAILURE_BUILD = 201 |
403 | + |
404 | + |
405 | +logger = logging.getLogger(__name__) |
406 | + |
407 | + |
408 | +class BuildOCI(SnapBuildProxyOperationMixin, VCSOperationMixin, |
409 | + SnapStoreOperationMixin, Operation): |
410 | + |
411 | + description = "Build an OCI image." |
412 | + |
413 | + @classmethod |
414 | + def add_arguments(cls, parser): |
415 | + super(BuildOCI, cls).add_arguments(parser) |
416 | + parser.add_argument( |
417 | + "--build-file", help="path to Dockerfile in branch") |
418 | + parser.add_argument("name", help="name of snap to build") |
419 | + |
420 | + def __init__(self, args, parser): |
421 | + super(BuildOCI, self).__init__(args, parser) |
422 | + self.bin = os.path.dirname(sys.argv[0]) |
423 | + |
424 | + def _add_docker_engine_proxy_settings(self): |
425 | + """Add systemd file for docker proxy settings.""" |
426 | + # Create containing directory for systemd overrides |
427 | + self.backend.run( |
428 | + ["mkdir", "-p", "/etc/systemd/system/docker.service.d"]) |
429 | + # we need both http_proxy and https_proxy. The contents of the files |
430 | + # are otherwise identical |
431 | + for setting in ['http_proxy', 'https_proxy']: |
432 | + contents = dedent("""[Service] |
433 | + Environment="{}={}" |
434 | + """.format(setting.upper(), self.args.proxy_url)) |
435 | + file_path = "/etc/systemd/system/docker.service.d/{}.conf".format( |
436 | + setting) |
437 | + with tempfile.NamedTemporaryFile(mode="w+") as systemd_file: |
438 | + systemd_file.write(contents) |
439 | + systemd_file.flush() |
440 | + self.backend.copy_in(systemd_file.name, file_path) |
441 | + |
442 | + def run_build_command(self, args, env=None, **kwargs): |
443 | + """Run a build command in the target. |
444 | + |
445 | + :param args: the command and arguments to run. |
446 | + :param env: dictionary of additional environment variables to set. |
447 | + :param kwargs: any other keyword arguments to pass to Backend.run. |
448 | + """ |
449 | + full_env = OrderedDict() |
450 | + full_env["LANG"] = "C.UTF-8" |
451 | + full_env["SHELL"] = "/bin/sh" |
452 | + if env: |
453 | + full_env.update(env) |
454 | + return self.backend.run(args, env=full_env, **kwargs) |
455 | + |
456 | + def install(self): |
457 | + logger.info("Running install phase...") |
458 | + deps = [] |
459 | + if self.args.proxy_url: |
460 | + deps.extend(self.proxy_deps) |
461 | + self.install_git_proxy() |
462 | + # Add any proxy settings that are needed |
463 | + self._add_docker_engine_proxy_settings() |
464 | + deps.extend(self.vcs_deps) |
465 | + deps.extend(["docker.io"]) |
466 | + self.backend.run(["apt-get", "-y", "install"] + deps) |
467 | + if self.args.backend in ("lxd", "fake"): |
468 | + self.snap_store_set_proxy() |
469 | + self.backend.run(["systemctl", "restart", "docker"]) |
470 | + # The docker snap can't see /build, so we have to do our work under |
471 | + # /home/buildd instead. Make sure it exists. |
472 | + self.backend.run(["mkdir", "-p", "/home/buildd"]) |
473 | + |
474 | + def repo(self): |
475 | + """Collect git or bzr branch.""" |
476 | + logger.info("Running repo phase...") |
477 | + env = self.build_proxy_environment(proxy_url=self.args.proxy_url) |
478 | + self.vcs_fetch(self.args.name, cwd="/home/buildd", env=env) |
479 | + |
480 | + def build(self): |
481 | + logger.info("Running build phase...") |
482 | + args = ["docker", "build", "--no-cache"] |
483 | + if self.args.proxy_url: |
484 | + for var in ("http_proxy", "https_proxy"): |
485 | + args.extend( |
486 | + ["--build-arg", "{}={}".format(var, self.args.proxy_url)]) |
487 | + args.extend(["--tag", self.args.name]) |
488 | + if self.args.build_file is not None: |
489 | + args.extend(["--file", self.args.build_file]) |
490 | + buildd_path = os.path.join("/home/buildd", self.args.name) |
491 | + args.append(buildd_path) |
492 | + self.run_build_command(args) |
493 | + |
494 | + def run(self): |
495 | + try: |
496 | + self.install() |
497 | + except Exception: |
498 | + logger.exception('Install failed') |
499 | + return RETCODE_FAILURE_INSTALL |
500 | + try: |
501 | + self.repo() |
502 | + self.build() |
503 | + except Exception: |
504 | + logger.exception('Build failed') |
505 | + return RETCODE_FAILURE_BUILD |
506 | + return 0 |
507 | |
508 | === modified file 'lpbuildd/target/build_snap.py' |
509 | --- lpbuildd/target/build_snap.py 2019-10-30 12:31:53 +0000 |
510 | +++ lpbuildd/target/build_snap.py 2020-02-26 10:52:47 +0000 |
511 | @@ -17,6 +17,7 @@ |
512 | from six.moves.urllib.parse import urlparse |
513 | |
514 | from lpbuildd.target.operation import Operation |
515 | +from lpbuildd.target.snapbuildproxy import SnapBuildProxyOperationMixin |
516 | from lpbuildd.target.snapstore import SnapStoreOperationMixin |
517 | from lpbuildd.target.vcs import VCSOperationMixin |
518 | |
519 | @@ -46,7 +47,8 @@ |
520 | getattr(namespace, self.dest)[snap] = channel |
521 | |
522 | |
523 | -class BuildSnap(VCSOperationMixin, SnapStoreOperationMixin, Operation): |
524 | +class BuildSnap(SnapBuildProxyOperationMixin, VCSOperationMixin, |
525 | + SnapStoreOperationMixin, Operation): |
526 | |
527 | description = "Build a snap." |
528 | |
529 | @@ -69,10 +71,6 @@ |
530 | help="RFC3339 timestamp of the Launchpad build request") |
531 | parser.add_argument( |
532 | "--build-url", help="URL of this build on Launchpad") |
533 | - parser.add_argument("--proxy-url", help="builder proxy url") |
534 | - parser.add_argument( |
535 | - "--revocation-endpoint", |
536 | - help="builder proxy token revocation endpoint") |
537 | parser.add_argument( |
538 | "--build-source-tarball", default=False, action="store_true", |
539 | help=( |
540 | @@ -137,6 +135,9 @@ |
541 | def install(self): |
542 | logger.info("Running install phase...") |
543 | deps = [] |
544 | + if self.args.proxy_url: |
545 | + deps.extend(self.proxy_deps) |
546 | + self.install_git_proxy() |
547 | if self.args.backend == "lxd": |
548 | # udev is installed explicitly to work around |
549 | # https://bugs.launchpad.net/snapd/+bug/1731519. |
550 | @@ -144,8 +145,6 @@ |
551 | if self.backend.is_package_available(dep): |
552 | deps.append(dep) |
553 | deps.extend(self.vcs_deps) |
554 | - if self.args.proxy_url: |
555 | - deps.extend(["python3", "socat"]) |
556 | if "snapcraft" in self.args.channels: |
557 | # snapcraft requires sudo in lots of places, but can't depend on |
558 | # it when installed as a snap. |
559 | @@ -167,19 +166,12 @@ |
560 | "--channel=%s" % self.args.channels["snapcraft"], |
561 | "snapcraft"]) |
562 | if self.args.proxy_url: |
563 | - self.backend.copy_in( |
564 | - os.path.join(self.bin, "snap-git-proxy"), |
565 | - "/usr/local/bin/snap-git-proxy") |
566 | self.install_svn_servers() |
567 | |
568 | def repo(self): |
569 | """Collect git or bzr branch.""" |
570 | logger.info("Running repo phase...") |
571 | - env = OrderedDict() |
572 | - if self.args.proxy_url: |
573 | - env["http_proxy"] = self.args.proxy_url |
574 | - env["https_proxy"] = self.args.proxy_url |
575 | - env["GIT_PROXY_COMMAND"] = "/usr/local/bin/snap-git-proxy" |
576 | + env = self.build_proxy_environment(proxy_url=self.args.proxy_url) |
577 | self.vcs_fetch(self.args.name, cwd="/build", env=env) |
578 | status = {} |
579 | if self.args.branch is not None: |
580 | |
581 | === modified file 'lpbuildd/target/cli.py' |
582 | --- lpbuildd/target/cli.py 2017-09-08 15:57:18 +0000 |
583 | +++ lpbuildd/target/cli.py 2020-02-26 10:52:47 +0000 |
584 | @@ -14,6 +14,7 @@ |
585 | OverrideSourcesList, |
586 | Update, |
587 | ) |
588 | +from lpbuildd.target.build_oci import BuildOCI |
589 | from lpbuildd.target.build_livefs import BuildLiveFS |
590 | from lpbuildd.target.build_snap import BuildSnap |
591 | from lpbuildd.target.generate_translation_templates import ( |
592 | @@ -49,6 +50,7 @@ |
593 | |
594 | operations = { |
595 | "add-trusted-keys": AddTrustedKeys, |
596 | + "build-oci": BuildOCI, |
597 | "buildlivefs": BuildLiveFS, |
598 | "buildsnap": BuildSnap, |
599 | "generate-translation-templates": GenerateTranslationTemplates, |
600 | |
601 | === modified file 'lpbuildd/target/lxd.py' |
602 | --- lpbuildd/target/lxd.py 2019-12-09 11:17:14 +0000 |
603 | +++ lpbuildd/target/lxd.py 2020-02-26 10:52:47 +0000 |
604 | @@ -504,7 +504,8 @@ |
605 | "/etc/systemd/system/snapd.refresh.timer"]) |
606 | |
607 | def run(self, args, cwd=None, env=None, input_text=None, get_output=False, |
608 | - echo=False, **kwargs): |
609 | + echo=False, return_process=False, universal_newlines=True, |
610 | + **kwargs): |
611 | """See `Backend`.""" |
612 | env_params = [] |
613 | if env: |
614 | @@ -538,7 +539,10 @@ |
615 | if get_output: |
616 | kwargs["stdout"] = subprocess.PIPE |
617 | proc = subprocess.Popen( |
618 | - cmd, stdin=subprocess.PIPE, universal_newlines=True, **kwargs) |
619 | + cmd, stdin=subprocess.PIPE, |
620 | + universal_newlines=universal_newlines, **kwargs) |
621 | + if return_process: |
622 | + return proc |
623 | output, _ = proc.communicate(input_text) |
624 | if proc.returncode: |
625 | raise subprocess.CalledProcessError(proc.returncode, cmd) |
626 | |
627 | === added file 'lpbuildd/target/snapbuildproxy.py' |
628 | --- lpbuildd/target/snapbuildproxy.py 1970-01-01 00:00:00 +0000 |
629 | +++ lpbuildd/target/snapbuildproxy.py 2020-02-26 10:52:47 +0000 |
630 | @@ -0,0 +1,41 @@ |
631 | +# Copyright 2019-2020 Canonical Ltd. This software is licensed under the |
632 | +# GNU Affero General Public License version 3 (see the file LICENSE). |
633 | + |
634 | +from __future__ import print_function |
635 | + |
636 | +__metaclass__ = type |
637 | + |
638 | +from collections import OrderedDict |
639 | +import os |
640 | + |
641 | + |
642 | +class SnapBuildProxyOperationMixin: |
643 | + """Methods supporting the build time HTTP proxy for snap and OCI builds.""" |
644 | + |
645 | + @classmethod |
646 | + def add_arguments(cls, parser): |
647 | + super(SnapBuildProxyOperationMixin, cls).add_arguments(parser) |
648 | + parser.add_argument("--proxy-url", help="builder proxy url") |
649 | + parser.add_argument( |
650 | + "--revocation-endpoint", |
651 | + help="builder proxy token revocation endpoint") |
652 | + |
653 | + @property |
654 | + def proxy_deps(self): |
655 | + return ["python3", "socat"] |
656 | + |
657 | + def install_git_proxy(self): |
658 | + self.backend.copy_in( |
659 | + os.path.join(self.bin, "snap-git-proxy"), |
660 | + "/usr/local/bin/snap-git-proxy") |
661 | + |
662 | + def build_proxy_environment(self, proxy_url=None, env=None): |
663 | + """Extend a command environment to include http proxy variables.""" |
664 | + full_env = OrderedDict() |
665 | + if env: |
666 | + full_env.update(env) |
667 | + if proxy_url: |
668 | + full_env["http_proxy"] = self.args.proxy_url |
669 | + full_env["https_proxy"] = self.args.proxy_url |
670 | + full_env["GIT_PROXY_COMMAND"] = "/usr/local/bin/snap-git-proxy" |
671 | + return full_env |
672 | |
673 | === added file 'lpbuildd/target/tests/test_build_oci.py' |
674 | --- lpbuildd/target/tests/test_build_oci.py 1970-01-01 00:00:00 +0000 |
675 | +++ lpbuildd/target/tests/test_build_oci.py 2020-02-26 10:52:47 +0000 |
676 | @@ -0,0 +1,407 @@ |
677 | +# Copyright 2019 Canonical Ltd. This software is licensed under the |
678 | +# GNU Affero General Public License version 3 (see the file LICENSE). |
679 | + |
680 | +__metaclass__ = type |
681 | + |
682 | +import os.path |
683 | +import stat |
684 | +import subprocess |
685 | +from textwrap import dedent |
686 | + |
687 | +from fixtures import ( |
688 | + FakeLogger, |
689 | + TempDir, |
690 | + ) |
691 | +import responses |
692 | +from systemfixtures import FakeFilesystem |
693 | +from testtools import TestCase |
694 | +from testtools.matchers import ( |
695 | + AnyMatch, |
696 | + Equals, |
697 | + Is, |
698 | + MatchesAll, |
699 | + MatchesDict, |
700 | + MatchesListwise, |
701 | + ) |
702 | + |
703 | +from lpbuildd.target.build_oci import ( |
704 | + RETCODE_FAILURE_BUILD, |
705 | + RETCODE_FAILURE_INSTALL, |
706 | + ) |
707 | +from lpbuildd.target.cli import parse_args |
708 | +from lpbuildd.tests.fakebuilder import FakeMethod |
709 | + |
710 | + |
711 | +class RanCommand(MatchesListwise): |
712 | + |
713 | + def __init__(self, args, echo=None, cwd=None, input_text=None, |
714 | + get_output=None, **env): |
715 | + kwargs_matcher = {} |
716 | + if echo is not None: |
717 | + kwargs_matcher["echo"] = Is(echo) |
718 | + if cwd: |
719 | + kwargs_matcher["cwd"] = Equals(cwd) |
720 | + if input_text: |
721 | + kwargs_matcher["input_text"] = Equals(input_text) |
722 | + if get_output is not None: |
723 | + kwargs_matcher["get_output"] = Is(get_output) |
724 | + if env: |
725 | + kwargs_matcher["env"] = MatchesDict( |
726 | + {key: Equals(value) for key, value in env.items()}) |
727 | + super(RanCommand, self).__init__( |
728 | + [Equals((args,)), MatchesDict(kwargs_matcher)]) |
729 | + |
730 | + |
731 | +class RanAptGet(RanCommand): |
732 | + |
733 | + def __init__(self, *args): |
734 | + super(RanAptGet, self).__init__(["apt-get", "-y"] + list(args)) |
735 | + |
736 | + |
737 | +class RanSnap(RanCommand): |
738 | + |
739 | + def __init__(self, *args, **kwargs): |
740 | + super(RanSnap, self).__init__(["snap"] + list(args), **kwargs) |
741 | + |
742 | + |
743 | +class RanBuildCommand(RanCommand): |
744 | + |
745 | + def __init__(self, args, **kwargs): |
746 | + kwargs.setdefault("LANG", "C.UTF-8") |
747 | + kwargs.setdefault("SHELL", "/bin/sh") |
748 | + super(RanBuildCommand, self).__init__(args, **kwargs) |
749 | + |
750 | + |
751 | +class TestBuildOCI(TestCase): |
752 | + |
753 | + def test_run_build_command_no_env(self): |
754 | + args = [ |
755 | + "build-oci", |
756 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
757 | + "--branch", "lp:foo", "test-image", |
758 | + ] |
759 | + build_oci = parse_args(args=args).operation |
760 | + build_oci.run_build_command(["echo", "hello world"]) |
761 | + self.assertThat(build_oci.backend.run.calls, MatchesListwise([ |
762 | + RanBuildCommand(["echo", "hello world"]), |
763 | + ])) |
764 | + |
765 | + def test_run_build_command_env(self): |
766 | + args = [ |
767 | + "build-oci", |
768 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
769 | + "--branch", "lp:foo", "test-image", |
770 | + ] |
771 | + build_oci = parse_args(args=args).operation |
772 | + build_oci.run_build_command( |
773 | + ["echo", "hello world"], env={"FOO": "bar baz"}) |
774 | + self.assertThat(build_oci.backend.run.calls, MatchesListwise([ |
775 | + RanBuildCommand(["echo", "hello world"], FOO="bar baz"), |
776 | + ])) |
777 | + |
778 | + def test_install_bzr(self): |
779 | + args = [ |
780 | + "build-oci", |
781 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
782 | + "--branch", "lp:foo", "test-image" |
783 | + ] |
784 | + build_oci = parse_args(args=args).operation |
785 | + build_oci.install() |
786 | + self.assertThat(build_oci.backend.run.calls, MatchesListwise([ |
787 | + RanAptGet("install", "bzr", "docker.io"), |
788 | + RanCommand(["systemctl", "restart", "docker"]), |
789 | + RanCommand(["mkdir", "-p", "/home/buildd"]), |
790 | + ])) |
791 | + |
792 | + def test_install_git(self): |
793 | + args = [ |
794 | + "build-oci", |
795 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
796 | + "--git-repository", "lp:foo", "test-image" |
797 | + ] |
798 | + build_oci = parse_args(args=args).operation |
799 | + build_oci.install() |
800 | + self.assertThat(build_oci.backend.run.calls, MatchesListwise([ |
801 | + RanAptGet("install", "git", "docker.io"), |
802 | + RanCommand(["systemctl", "restart", "docker"]), |
803 | + RanCommand(["mkdir", "-p", "/home/buildd"]), |
804 | + ])) |
805 | + |
806 | + @responses.activate |
807 | + def test_install_snap_store_proxy(self): |
808 | + store_assertion = dedent("""\ |
809 | + type: store |
810 | + store: store-id |
811 | + url: http://snap-store-proxy.example |
812 | + |
813 | + body |
814 | + """) |
815 | + |
816 | + def respond(request): |
817 | + return 200, {"X-Assertion-Store-Id": "store-id"}, store_assertion |
818 | + |
819 | + responses.add_callback( |
820 | + "GET", "http://snap-store-proxy.example/v2/auth/store/assertions", |
821 | + callback=respond) |
822 | + args = [ |
823 | + "buildsnap", |
824 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
825 | + "--git-repository", "lp:foo", |
826 | + "--snap-store-proxy-url", "http://snap-store-proxy.example/", |
827 | + "test-snap", |
828 | + ] |
829 | + build_snap = parse_args(args=args).operation |
830 | + build_snap.install() |
831 | + self.assertThat(build_snap.backend.run.calls, MatchesListwise([ |
832 | + RanAptGet("install", "git", "snapcraft"), |
833 | + RanCommand( |
834 | + ["snap", "ack", "/dev/stdin"], input_text=store_assertion), |
835 | + RanCommand(["snap", "set", "core", "proxy.store=store-id"]), |
836 | + ])) |
837 | + |
838 | + def test_install_proxy(self): |
839 | + args = [ |
840 | + "build-oci", |
841 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
842 | + "--git-repository", "lp:foo", |
843 | + "--proxy-url", "http://proxy.example:3128/", |
844 | + "test-image", |
845 | + ] |
846 | + build_oci = parse_args(args=args).operation |
847 | + build_oci.bin = "/builderbin" |
848 | + self.useFixture(FakeFilesystem()).add("/builderbin") |
849 | + os.mkdir("/builderbin") |
850 | + with open("/builderbin/snap-git-proxy", "w") as proxy_script: |
851 | + proxy_script.write("proxy script\n") |
852 | + os.fchmod(proxy_script.fileno(), 0o755) |
853 | + build_oci.install() |
854 | + self.assertThat(build_oci.backend.run.calls, MatchesListwise([ |
855 | + RanCommand( |
856 | + ["mkdir", "-p", "/etc/systemd/system/docker.service.d"]), |
857 | + RanAptGet("install", "python3", "socat", "git", "docker.io"), |
858 | + RanCommand(["systemctl", "restart", "docker"]), |
859 | + RanCommand(["mkdir", "-p", "/home/buildd"]), |
860 | + ])) |
861 | + self.assertEqual( |
862 | + (b"proxy script\n", stat.S_IFREG | 0o755), |
863 | + build_oci.backend.backend_fs["/usr/local/bin/snap-git-proxy"]) |
864 | + |
865 | + def test_repo_bzr(self): |
866 | + args = [ |
867 | + "build-oci", |
868 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
869 | + "--branch", "lp:foo", "test-image", |
870 | + ] |
871 | + build_oci = parse_args(args=args).operation |
872 | + build_oci.backend.build_path = self.useFixture(TempDir()).path |
873 | + build_oci.backend.run = FakeMethod() |
874 | + build_oci.repo() |
875 | + self.assertThat(build_oci.backend.run.calls, MatchesListwise([ |
876 | + RanBuildCommand( |
877 | + ["bzr", "branch", "lp:foo", "test-image"], cwd="/home/buildd"), |
878 | + ])) |
879 | + |
880 | + def test_repo_git(self): |
881 | + args = [ |
882 | + "build-oci", |
883 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
884 | + "--git-repository", "lp:foo", "test-image", |
885 | + ] |
886 | + build_oci = parse_args(args=args).operation |
887 | + build_oci.backend.build_path = self.useFixture(TempDir()).path |
888 | + build_oci.backend.run = FakeMethod() |
889 | + build_oci.repo() |
890 | + self.assertThat(build_oci.backend.run.calls, MatchesListwise([ |
891 | + RanBuildCommand( |
892 | + ["git", "clone", "lp:foo", "test-image"], cwd="/home/buildd"), |
893 | + RanBuildCommand( |
894 | + ["git", "submodule", "update", "--init", "--recursive"], |
895 | + cwd="/home/buildd/test-image"), |
896 | + ])) |
897 | + |
898 | + def test_repo_git_with_path(self): |
899 | + args = [ |
900 | + "build-oci", |
901 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
902 | + "--git-repository", "lp:foo", "--git-path", "next", "test-image", |
903 | + ] |
904 | + build_oci = parse_args(args=args).operation |
905 | + build_oci.backend.build_path = self.useFixture(TempDir()).path |
906 | + build_oci.backend.run = FakeMethod() |
907 | + build_oci.repo() |
908 | + self.assertThat(build_oci.backend.run.calls, MatchesListwise([ |
909 | + RanBuildCommand( |
910 | + ["git", "clone", "-b", "next", "lp:foo", "test-image"], |
911 | + cwd="/home/buildd"), |
912 | + RanBuildCommand( |
913 | + ["git", "submodule", "update", "--init", "--recursive"], |
914 | + cwd="/home/buildd/test-image"), |
915 | + ])) |
916 | + |
917 | + def test_repo_git_with_tag_path(self): |
918 | + args = [ |
919 | + "build-oci", |
920 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
921 | + "--git-repository", "lp:foo", "--git-path", "refs/tags/1.0", |
922 | + "test-image", |
923 | + ] |
924 | + build_oci = parse_args(args=args).operation |
925 | + build_oci.backend.build_path = self.useFixture(TempDir()).path |
926 | + build_oci.backend.run = FakeMethod() |
927 | + build_oci.repo() |
928 | + self.assertThat(build_oci.backend.run.calls, MatchesListwise([ |
929 | + RanBuildCommand( |
930 | + ["git", "clone", "-b", "1.0", "lp:foo", "test-image"], |
931 | + cwd="/home/buildd"), |
932 | + RanBuildCommand( |
933 | + ["git", "submodule", "update", "--init", "--recursive"], |
934 | + cwd="/home/buildd/test-image"), |
935 | + ])) |
936 | + |
937 | + def test_repo_proxy(self): |
938 | + args = [ |
939 | + "build-oci", |
940 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
941 | + "--git-repository", "lp:foo", |
942 | + "--proxy-url", "http://proxy.example:3128/", |
943 | + "test-image", |
944 | + ] |
945 | + build_oci = parse_args(args=args).operation |
946 | + build_oci.backend.build_path = self.useFixture(TempDir()).path |
947 | + build_oci.backend.run = FakeMethod() |
948 | + build_oci.repo() |
949 | + env = { |
950 | + "http_proxy": "http://proxy.example:3128/", |
951 | + "https_proxy": "http://proxy.example:3128/", |
952 | + "GIT_PROXY_COMMAND": "/usr/local/bin/snap-git-proxy", |
953 | + } |
954 | + self.assertThat(build_oci.backend.run.calls, MatchesListwise([ |
955 | + RanBuildCommand( |
956 | + ["git", "clone", "lp:foo", "test-image"], |
957 | + cwd="/home/buildd", **env), |
958 | + RanBuildCommand( |
959 | + ["git", "submodule", "update", "--init", "--recursive"], |
960 | + cwd="/home/buildd/test-image", **env), |
961 | + ])) |
962 | + |
963 | + def test_build(self): |
964 | + args = [ |
965 | + "build-oci", |
966 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
967 | + "--branch", "lp:foo", "test-image", |
968 | + ] |
969 | + build_oci = parse_args(args=args).operation |
970 | + build_oci.backend.add_dir('/build/test-directory') |
971 | + build_oci.build() |
972 | + self.assertThat(build_oci.backend.run.calls, MatchesListwise([ |
973 | + RanBuildCommand( |
974 | + ["docker", "build", "--no-cache", "--tag", "test-image", |
975 | + "/home/buildd/test-image"]), |
976 | + ])) |
977 | + |
978 | + def test_build_with_file(self): |
979 | + args = [ |
980 | + "build-oci", |
981 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
982 | + "--branch", "lp:foo", "--build-file", "build-aux/Dockerfile", |
983 | + "test-image", |
984 | + ] |
985 | + build_oci = parse_args(args=args).operation |
986 | + build_oci.backend.add_dir('/build/test-directory') |
987 | + build_oci.build() |
988 | + self.assertThat(build_oci.backend.run.calls, MatchesListwise([ |
989 | + RanBuildCommand( |
990 | + ["docker", "build", "--no-cache", "--tag", "test-image", |
991 | + "--file", "build-aux/Dockerfile", |
992 | + "/home/buildd/test-image"]), |
993 | + ])) |
994 | + |
995 | + def test_build_proxy(self): |
996 | + args = [ |
997 | + "build-oci", |
998 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
999 | + "--branch", "lp:foo", "--proxy-url", "http://proxy.example:3128/", |
1000 | + "test-image", |
1001 | + ] |
1002 | + build_oci = parse_args(args=args).operation |
1003 | + build_oci.backend.add_dir('/build/test-directory') |
1004 | + build_oci.build() |
1005 | + self.assertThat(build_oci.backend.run.calls, MatchesListwise([ |
1006 | + RanBuildCommand( |
1007 | + ["docker", "build", "--no-cache", |
1008 | + "--build-arg", "http_proxy=http://proxy.example:3128/", |
1009 | + "--build-arg", "https_proxy=http://proxy.example:3128/", |
1010 | + "--tag", "test-image", "/home/buildd/test-image"]), |
1011 | + ])) |
1012 | + |
1013 | + def test_run_succeeds(self): |
1014 | + args = [ |
1015 | + "build-oci", |
1016 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
1017 | + "--branch", "lp:foo", "test-image", |
1018 | + ] |
1019 | + build_oci = parse_args(args=args).operation |
1020 | + build_oci.backend.build_path = self.useFixture(TempDir()).path |
1021 | + build_oci.backend.run = FakeMethod() |
1022 | + self.assertEqual(0, build_oci.run()) |
1023 | + self.assertThat(build_oci.backend.run.calls, MatchesAll( |
1024 | + AnyMatch(RanAptGet("install", "bzr", "docker.io")), |
1025 | + AnyMatch(RanBuildCommand( |
1026 | + ["bzr", "branch", "lp:foo", "test-image"], |
1027 | + cwd="/home/buildd")), |
1028 | + AnyMatch(RanBuildCommand( |
1029 | + ["docker", "build", "--no-cache", "--tag", "test-image", |
1030 | + "/home/buildd/test-image"])), |
1031 | + )) |
1032 | + |
1033 | + def test_run_install_fails(self): |
1034 | + class FailInstall(FakeMethod): |
1035 | + def __call__(self, run_args, *args, **kwargs): |
1036 | + super(FailInstall, self).__call__(run_args, *args, **kwargs) |
1037 | + if run_args[0] == "apt-get": |
1038 | + raise subprocess.CalledProcessError(1, run_args) |
1039 | + |
1040 | + self.useFixture(FakeLogger()) |
1041 | + args = [ |
1042 | + "build-oci", |
1043 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
1044 | + "--branch", "lp:foo", "test-image", |
1045 | + ] |
1046 | + build_oci = parse_args(args=args).operation |
1047 | + build_oci.backend.run = FailInstall() |
1048 | + self.assertEqual(RETCODE_FAILURE_INSTALL, build_oci.run()) |
1049 | + |
1050 | + def test_run_repo_fails(self): |
1051 | + class FailRepo(FakeMethod): |
1052 | + def __call__(self, run_args, *args, **kwargs): |
1053 | + super(FailRepo, self).__call__(run_args, *args, **kwargs) |
1054 | + if run_args[:2] == ["bzr", "branch"]: |
1055 | + raise subprocess.CalledProcessError(1, run_args) |
1056 | + |
1057 | + self.useFixture(FakeLogger()) |
1058 | + args = [ |
1059 | + "build-oci", |
1060 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
1061 | + "--branch", "lp:foo", "test-image", |
1062 | + ] |
1063 | + build_oci = parse_args(args=args).operation |
1064 | + build_oci.backend.run = FailRepo() |
1065 | + self.assertEqual(RETCODE_FAILURE_BUILD, build_oci.run()) |
1066 | + |
1067 | + def test_run_build_fails(self): |
1068 | + class FailBuild(FakeMethod): |
1069 | + def __call__(self, run_args, *args, **kwargs): |
1070 | + super(FailBuild, self).__call__(run_args, *args, **kwargs) |
1071 | + if run_args[0] == "docker": |
1072 | + raise subprocess.CalledProcessError(1, run_args) |
1073 | + |
1074 | + self.useFixture(FakeLogger()) |
1075 | + args = [ |
1076 | + "build-oci", |
1077 | + "--backend=fake", "--series=xenial", "--arch=amd64", "1", |
1078 | + "--branch", "lp:foo", "test-image", |
1079 | + ] |
1080 | + build_oci = parse_args(args=args).operation |
1081 | + build_oci.backend.build_path = self.useFixture(TempDir()).path |
1082 | + build_oci.backend.run = FailBuild() |
1083 | + self.assertEqual(RETCODE_FAILURE_BUILD, build_oci.run()) |
1084 | |
1085 | === modified file 'lpbuildd/target/tests/test_build_snap.py' |
1086 | --- lpbuildd/target/tests/test_build_snap.py 2019-06-05 14:10:20 +0000 |
1087 | +++ lpbuildd/target/tests/test_build_snap.py 2020-02-26 10:52:47 +0000 |
1088 | @@ -186,7 +186,7 @@ |
1089 | os.fchmod(proxy_script.fileno(), 0o755) |
1090 | build_snap.install() |
1091 | self.assertThat(build_snap.backend.run.calls, MatchesListwise([ |
1092 | - RanAptGet("install", "git", "python3", "socat", "snapcraft"), |
1093 | + RanAptGet("install", "python3", "socat", "git", "snapcraft"), |
1094 | RanCommand(["mkdir", "-p", "/root/.subversion"]), |
1095 | ])) |
1096 | self.assertEqual( |
1097 | |
1098 | === modified file 'lpbuildd/target/tests/test_lxd.py' |
1099 | --- lpbuildd/target/tests/test_lxd.py 2019-12-09 11:17:14 +0000 |
1100 | +++ lpbuildd/target/tests/test_lxd.py 2020-02-26 10:52:47 +0000 |
1101 | @@ -276,7 +276,7 @@ |
1102 | "lp-xenial-amd64", "lp-xenial-amd64") |
1103 | |
1104 | def assert_correct_profile(self, extra_raw_lxc_config=None, |
1105 | - driver_version="2.0"): |
1106 | + driver_version="2.0"): |
1107 | if extra_raw_lxc_config is None: |
1108 | extra_raw_lxc_config = [] |
1109 | |
1110 | @@ -356,7 +356,7 @@ |
1111 | } |
1112 | LXD("1", "xenial", "powerpc").create_profile() |
1113 | self.assert_correct_profile( |
1114 | - extra_raw_lxc_config=[("lxc.seccomp", ""),], |
1115 | + extra_raw_lxc_config=[("lxc.seccomp", ""), ], |
1116 | driver_version=driver_version or "3.0" |
1117 | ) |
1118 | |
1119 | @@ -463,7 +463,8 @@ |
1120 | "b", "7", str(minor)])) |
1121 | if not with_dm0: |
1122 | expected_args.extend([ |
1123 | - Equals(["sudo", "dmsetup", "create", "tmpdevice", "--notable"]), |
1124 | + Equals( |
1125 | + ["sudo", "dmsetup", "create", "tmpdevice", "--notable"]), |
1126 | Equals(["sudo", "dmsetup", "remove", "tmpdevice"]), |
1127 | ]) |
1128 | for minor in range(8): |
1129 | |
1130 | === added file 'lpbuildd/tests/oci_tarball.py' |
1131 | --- lpbuildd/tests/oci_tarball.py 1970-01-01 00:00:00 +0000 |
1132 | +++ lpbuildd/tests/oci_tarball.py 2020-02-26 10:52:47 +0000 |
1133 | @@ -0,0 +1,60 @@ |
1134 | +import json |
1135 | +import os |
1136 | +import StringIO |
1137 | +import tempfile |
1138 | +import tarfile |
1139 | + |
1140 | + |
1141 | +class OCITarball: |
1142 | + """Create a tarball for use in tests with OCI.""" |
1143 | + |
1144 | + def _makeFile(self, contents, name): |
1145 | + json_contents = json.dumps(contents) |
1146 | + tarinfo = tarfile.TarInfo(name) |
1147 | + tarinfo.size = len(json_contents) |
1148 | + return tarinfo, StringIO.StringIO(json_contents) |
1149 | + |
1150 | + @property |
1151 | + def config(self): |
1152 | + return self._makeFile( |
1153 | + {"rootfs": {"diff_ids": ["sha256:diff1", "sha256:diff2"]}}, |
1154 | + 'config.json') |
1155 | + |
1156 | + @property |
1157 | + def manifest(self): |
1158 | + return self._makeFile( |
1159 | + [{"Config": "config.json", |
1160 | + "Layers": ["layer-1/layer.tar", "layer-2/layer.tar"]}], |
1161 | + 'manifest.json') |
1162 | + |
1163 | + @property |
1164 | + def repositories(self): |
1165 | + return self._makeFile([], 'repositories') |
1166 | + |
1167 | + def layer_file(self, directory, layer_name): |
1168 | + contents = "{}-contents".format(layer_name) |
1169 | + tarinfo = tarfile.TarInfo(contents) |
1170 | + tarinfo.size = len(contents) |
1171 | + layer_contents = StringIO.StringIO(contents) |
1172 | + layer_tar_path = os.path.join( |
1173 | + directory, '{}.tar.gz'.format(layer_name)) |
1174 | + layer_tar = tarfile.open(layer_tar_path, 'w:gz') |
1175 | + layer_tar.addfile(tarinfo, layer_contents) |
1176 | + layer_tar.close() |
1177 | + return layer_tar_path |
1178 | + |
1179 | + def build_tar_file(self): |
1180 | + tar_directory = tempfile.mkdtemp() |
1181 | + tar_path = os.path.join(tar_directory, 'test-oci-image.tar') |
1182 | + tar = tarfile.open(tar_path, 'w') |
1183 | + tar.addfile(*self.config) |
1184 | + tar.addfile(*self.manifest) |
1185 | + tar.addfile(*self.repositories) |
1186 | + |
1187 | + for layer_name in ['layer-1', 'layer-2']: |
1188 | + layer = self.layer_file(tar_directory, layer_name) |
1189 | + tar.add(layer, arcname='{}.tar.gz'.format(layer_name)) |
1190 | + |
1191 | + tar.close() |
1192 | + |
1193 | + return tar_path |
1194 | |
1195 | === added file 'lpbuildd/tests/test_oci.py' |
1196 | --- lpbuildd/tests/test_oci.py 1970-01-01 00:00:00 +0000 |
1197 | +++ lpbuildd/tests/test_oci.py 2020-02-26 10:52:47 +0000 |
1198 | @@ -0,0 +1,280 @@ |
1199 | +# Copyright 2019 Canonical Ltd. This software is licensed under the |
1200 | +# GNU Affero General Public License version 3 (see the file LICENSE). |
1201 | + |
1202 | +__metaclass__ = type |
1203 | + |
1204 | +import io |
1205 | +import json |
1206 | +import os |
1207 | +try: |
1208 | + from unittest import mock |
1209 | +except ImportError: |
1210 | + import mock |
1211 | + |
1212 | +from fixtures import ( |
1213 | + EnvironmentVariable, |
1214 | + MockPatch, |
1215 | + TempDir, |
1216 | + ) |
1217 | +from testtools import TestCase |
1218 | +from testtools.matchers import Contains |
1219 | +from testtools.deferredruntest import AsynchronousDeferredRunTest |
1220 | +from twisted.internet import defer |
1221 | + |
1222 | +from lpbuildd.oci import ( |
1223 | + OCIBuildManager, |
1224 | + OCIBuildState, |
1225 | + ) |
1226 | +from lpbuildd.tests.fakebuilder import FakeBuilder |
1227 | +from lpbuildd.tests.oci_tarball import OCITarball |
1228 | + |
1229 | + |
1230 | +class MockBuildManager(OCIBuildManager): |
1231 | + def __init__(self, *args, **kwargs): |
1232 | + super(MockBuildManager, self).__init__(*args, **kwargs) |
1233 | + self.commands = [] |
1234 | + self.iterators = [] |
1235 | + |
1236 | + def runSubProcess(self, path, command, iterate=None, env=None): |
1237 | + self.commands.append([path] + command) |
1238 | + if iterate is None: |
1239 | + iterate = self.iterate |
1240 | + self.iterators.append(iterate) |
1241 | + return 0 |
1242 | + |
1243 | + |
1244 | +class MockOCITarSave(): |
1245 | + @property |
1246 | + def stdout(self): |
1247 | + return io.open(OCITarball().build_tar_file(), 'rb') |
1248 | + |
1249 | + |
1250 | +class TestOCIBuildManagerIteration(TestCase): |
1251 | + """Run OCIBuildManager through its iteration steps.""" |
1252 | + |
1253 | + run_tests_with = AsynchronousDeferredRunTest.make_factory(timeout=5) |
1254 | + |
1255 | + def setUp(self): |
1256 | + super(TestOCIBuildManagerIteration, self).setUp() |
1257 | + self.working_dir = self.useFixture(TempDir()).path |
1258 | + builder_dir = os.path.join(self.working_dir, "builder") |
1259 | + home_dir = os.path.join(self.working_dir, "home") |
1260 | + for dir in (builder_dir, home_dir): |
1261 | + os.mkdir(dir) |
1262 | + self.useFixture(EnvironmentVariable("HOME", home_dir)) |
1263 | + self.builder = FakeBuilder(builder_dir) |
1264 | + self.buildid = "123" |
1265 | + self.buildmanager = MockBuildManager(self.builder, self.buildid) |
1266 | + self.buildmanager._cachepath = self.builder._cachepath |
1267 | + |
1268 | + def getState(self): |
1269 | + """Retrieve build manager's state.""" |
1270 | + return self.buildmanager._state |
1271 | + |
1272 | + @defer.inlineCallbacks |
1273 | + def startBuild(self, args=None, options=None): |
1274 | + # The build manager's iterate() kicks off the consecutive states |
1275 | + # after INIT. |
1276 | + extra_args = { |
1277 | + "series": "xenial", |
1278 | + "arch_tag": "i386", |
1279 | + "name": "test-image", |
1280 | + } |
1281 | + if args is not None: |
1282 | + extra_args.update(args) |
1283 | + original_backend_name = self.buildmanager.backend_name |
1284 | + self.buildmanager.backend_name = "fake" |
1285 | + self.buildmanager.initiate({}, "chroot.tar.gz", extra_args) |
1286 | + self.buildmanager.backend_name = original_backend_name |
1287 | + |
1288 | + # Skip states that are done in DebianBuildManager to the state |
1289 | + # directly before BUILD_OCI. |
1290 | + self.buildmanager._state = OCIBuildState.UPDATE |
1291 | + |
1292 | + # BUILD_OCI: Run the builder's payload to build the snap package. |
1293 | + yield self.buildmanager.iterate(0) |
1294 | + self.assertEqual(OCIBuildState.BUILD_OCI, self.getState()) |
1295 | + expected_command = [ |
1296 | + "sharepath/bin/in-target", "in-target", "build-oci", |
1297 | + "--backend=lxd", "--series=xenial", "--arch=i386", self.buildid, |
1298 | + ] |
1299 | + if options is not None: |
1300 | + expected_command.extend(options) |
1301 | + expected_command.append("test-image") |
1302 | + self.assertEqual(expected_command, self.buildmanager.commands[-1]) |
1303 | + self.assertEqual( |
1304 | + self.buildmanager.iterate, self.buildmanager.iterators[-1]) |
1305 | + self.assertFalse(self.builder.wasCalled("chrootFail")) |
1306 | + |
1307 | + @defer.inlineCallbacks |
1308 | + def test_iterate(self): |
1309 | + # This sha would change as it includes file attributes in the |
1310 | + # tar file. Fix it so we can test against a known value. |
1311 | + sha_mock = self.useFixture( |
1312 | + MockPatch('lpbuildd.oci.OCIBuildManager._calculateLayerSha')) |
1313 | + sha_mock.mock.return_value = "testsha" |
1314 | + # The build manager iterates a normal build from start to finish. |
1315 | + args = { |
1316 | + "git_repository": "https://git.launchpad.dev/~example/+git/snap", |
1317 | + "git_path": "master", |
1318 | + } |
1319 | + expected_options = [ |
1320 | + "--git-repository", "https://git.launchpad.dev/~example/+git/snap", |
1321 | + "--git-path", "master", |
1322 | + ] |
1323 | + yield self.startBuild(args, expected_options) |
1324 | + |
1325 | + log_path = os.path.join(self.buildmanager._cachepath, "buildlog") |
1326 | + with open(log_path, "w") as log: |
1327 | + log.write("I am a build log.") |
1328 | + |
1329 | + self.buildmanager.backend.run.result = MockOCITarSave() |
1330 | + |
1331 | + self.buildmanager.backend.add_file( |
1332 | + '/var/lib/docker/image/' |
1333 | + 'vfs/distribution/v2metadata-by-diffid/sha256/diff1', |
1334 | + b"""[{"Digest": "test_digest", "SourceRepository": "test"}]""" |
1335 | + ) |
1336 | + |
1337 | + # After building the package, reap processes. |
1338 | + yield self.buildmanager.iterate(0) |
1339 | + expected_command = [ |
1340 | + "sharepath/bin/in-target", "in-target", "scan-for-processes", |
1341 | + "--backend=lxd", "--series=xenial", "--arch=i386", self.buildid, |
1342 | + ] |
1343 | + self.assertEqual(OCIBuildState.BUILD_OCI, self.getState()) |
1344 | + self.assertEqual(expected_command, self.buildmanager.commands[-1]) |
1345 | + self.assertNotEqual( |
1346 | + self.buildmanager.iterate, self.buildmanager.iterators[-1]) |
1347 | + self.assertFalse(self.builder.wasCalled("buildFail")) |
1348 | + expected_files = [ |
1349 | + 'manifest.json', |
1350 | + 'layer-1.tar.gz', |
1351 | + 'layer-2.tar.gz', |
1352 | + 'digests.json', |
1353 | + 'config.json', |
1354 | + ] |
1355 | + for expected in expected_files: |
1356 | + self.assertThat(self.builder.waitingfiles, Contains(expected)) |
1357 | + |
1358 | + cache_path = self.builder.cachePath( |
1359 | + self.builder.waitingfiles['digests.json']) |
1360 | + with open(cache_path, "rb") as f: |
1361 | + digests_contents = f.read() |
1362 | + digests_expected = [{ |
1363 | + "sha256:diff1": { |
1364 | + "source": "test", |
1365 | + "digest": "test_digest", |
1366 | + "layer_id": "layer-1" |
1367 | + }, |
1368 | + "sha256:diff2": { |
1369 | + "source": "", |
1370 | + "digest": "testsha", |
1371 | + "layer_id": "layer-2" |
1372 | + } |
1373 | + }] |
1374 | + self.assertEqual(digests_contents, json.dumps(digests_expected)) |
1375 | + # Control returns to the DebianBuildManager in the UMOUNT state. |
1376 | + self.buildmanager.iterateReap(self.getState(), 0) |
1377 | + expected_command = [ |
1378 | + "sharepath/bin/in-target", "in-target", "umount-chroot", |
1379 | + "--backend=lxd", "--series=xenial", "--arch=i386", self.buildid, |
1380 | + ] |
1381 | + self.assertEqual(OCIBuildState.UMOUNT, self.getState()) |
1382 | + self.assertEqual(expected_command, self.buildmanager.commands[-1]) |
1383 | + self.assertEqual( |
1384 | + self.buildmanager.iterate, self.buildmanager.iterators[-1]) |
1385 | + self.assertFalse(self.builder.wasCalled("buildFail")) |
1386 | + |
1387 | + @defer.inlineCallbacks |
1388 | + def test_iterate_with_file(self): |
1389 | + # This sha would change as it includes file attributes in the |
1390 | + # tar file. Fix it so we can test against a known value. |
1391 | + sha_mock = self.useFixture( |
1392 | + MockPatch('lpbuildd.oci.OCIBuildManager._calculateLayerSha')) |
1393 | + sha_mock.mock.return_value = "testsha" |
1394 | + # The build manager iterates a build that specifies a non-default |
1395 | + # Dockerfile location from start to finish. |
1396 | + args = { |
1397 | + "git_repository": "https://git.launchpad.dev/~example/+git/snap", |
1398 | + "git_path": "master", |
1399 | + "build_file": "build-aux/Dockerfile", |
1400 | + } |
1401 | + expected_options = [ |
1402 | + "--git-repository", "https://git.launchpad.dev/~example/+git/snap", |
1403 | + "--git-path", "master", |
1404 | + "--build-file", "build-aux/Dockerfile", |
1405 | + ] |
1406 | + yield self.startBuild(args, expected_options) |
1407 | + |
1408 | + log_path = os.path.join(self.buildmanager._cachepath, "buildlog") |
1409 | + with open(log_path, "w") as log: |
1410 | + log.write("I am a build log.") |
1411 | + |
1412 | + self.buildmanager.backend.run.result = MockOCITarSave() |
1413 | + |
1414 | + self.buildmanager.backend.add_file( |
1415 | + '/var/lib/docker/image/' |
1416 | + 'vfs/distribution/v2metadata-by-diffid/sha256/diff1', |
1417 | + b"""[{"Digest": "test_digest", "SourceRepository": "test"}]""" |
1418 | + ) |
1419 | + |
1420 | + # After building the package, reap processes. |
1421 | + yield self.buildmanager.iterate(0) |
1422 | + expected_command = [ |
1423 | + "sharepath/bin/in-target", "in-target", "scan-for-processes", |
1424 | + "--backend=lxd", "--series=xenial", "--arch=i386", self.buildid, |
1425 | + ] |
1426 | + self.assertEqual(OCIBuildState.BUILD_OCI, self.getState()) |
1427 | + self.assertEqual(expected_command, self.buildmanager.commands[-1]) |
1428 | + self.assertNotEqual( |
1429 | + self.buildmanager.iterate, self.buildmanager.iterators[-1]) |
1430 | + self.assertFalse(self.builder.wasCalled("buildFail")) |
1431 | + expected_files = [ |
1432 | + 'manifest.json', |
1433 | + 'layer-1.tar.gz', |
1434 | + 'layer-2.tar.gz', |
1435 | + 'digests.json', |
1436 | + 'config.json', |
1437 | + ] |
1438 | + for expected in expected_files: |
1439 | + self.assertThat(self.builder.waitingfiles, Contains(expected)) |
1440 | + |
1441 | + cache_path = self.builder.cachePath( |
1442 | + self.builder.waitingfiles['digests.json']) |
1443 | + with open(cache_path, "rb") as f: |
1444 | + digests_contents = f.read() |
1445 | + digests_expected = [{ |
1446 | + "sha256:diff1": { |
1447 | + "source": "test", |
1448 | + "digest": "test_digest", |
1449 | + "layer_id": "layer-1" |
1450 | + }, |
1451 | + "sha256:diff2": { |
1452 | + "source": "", |
1453 | + "digest": "testsha", |
1454 | + "layer_id": "layer-2" |
1455 | + } |
1456 | + }] |
1457 | + self.assertEqual(digests_contents, json.dumps(digests_expected)) |
1458 | + |
1459 | + # Control returns to the DebianBuildManager in the UMOUNT state. |
1460 | + self.buildmanager.iterateReap(self.getState(), 0) |
1461 | + expected_command = [ |
1462 | + "sharepath/bin/in-target", "in-target", "umount-chroot", |
1463 | + "--backend=lxd", "--series=xenial", "--arch=i386", self.buildid, |
1464 | + ] |
1465 | + self.assertEqual(OCIBuildState.UMOUNT, self.getState()) |
1466 | + self.assertEqual(expected_command, self.buildmanager.commands[-1]) |
1467 | + self.assertEqual( |
1468 | + self.buildmanager.iterate, self.buildmanager.iterators[-1]) |
1469 | + self.assertFalse(self.builder.wasCalled("buildFail")) |
1470 | + |
1471 | + @defer.inlineCallbacks |
1472 | + def test_iterate_snap_store_proxy(self): |
1473 | + # The build manager can be told to use a snap store proxy. |
1474 | + self.builder._config.set( |
1475 | + "proxy", "snapstore", "http://snap-store-proxy.example/") |
1476 | + expected_options = [ |
1477 | + "--snap-store-proxy-url", "http://snap-store-proxy.example/"] |
1478 | + yield self.startBuild(options=expected_options) |
While looking at the launchpad buildbehaviour side of this, realised MP doesn't support build file location.
It should do that.