Merge lp:~twom/launchpad-buildd/initial-docker-build-support into lp:launchpad-buildd

Proposed by Tom Wardill
Status: Merged
Merged at revision: 413
Proposed branch: lp:~twom/launchpad-buildd/initial-docker-build-support
Merge into: lp:launchpad-buildd
Diff against target: 1478 lines (+1200/-54)
14 files modified
debian/changelog (+10/-4)
lpbuildd/buildd-slave.tac (+2/-0)
lpbuildd/oci.py (+222/-0)
lpbuildd/snap.py (+32/-29)
lpbuildd/target/build_oci.py (+126/-0)
lpbuildd/target/build_snap.py (+7/-15)
lpbuildd/target/cli.py (+2/-0)
lpbuildd/target/lxd.py (+6/-2)
lpbuildd/target/snapbuildproxy.py (+41/-0)
lpbuildd/target/tests/test_build_oci.py (+407/-0)
lpbuildd/target/tests/test_build_snap.py (+1/-1)
lpbuildd/target/tests/test_lxd.py (+4/-3)
lpbuildd/tests/oci_tarball.py (+60/-0)
lpbuildd/tests/test_oci.py (+280/-0)
To merge this branch: bzr merge lp:~twom/launchpad-buildd/initial-docker-build-support
Reviewer Review Type Date Requested Status
Tom Wardill (community) Approve
Colin Watson Approve
Review via email: mp+369775@code.launchpad.net

Commit message

Add initial docker build support

Description of the change

Add a builder for docker, creating an image following a supplied Dockerfile.
Save the image, extract it and then tar each component layer individually for returning/caching in launchpad.

To post a comment you must log in.
Revision history for this message
Colin Watson (cjwatson) :
review: Needs Fixing
393. By Tom Wardill

Use shell_escape, compress output images

394. By Tom Wardill

Collect .tar.gz

395. By Tom Wardill

Use a python script to save docker image to layers

396. By Tom Wardill

Move image extraction to gatherResults

397. By Tom Wardill

Refactor joint proxy handling

398. By Tom Wardill

Fix changelog diff noise

399. By Tom Wardill

Do more preprocessing, create digests file

400. By Tom Wardill

Include all diffs in digests and waiting files

401. By Tom Wardill

Add layer_id to digests now we're using every layer

402. By <email address hidden>

Use docker from apt, rather than from snap

403. By <email address hidden>

Rename Docker to OCI

404. By <email address hidden>

Merge upstream

Revision history for this message
Colin Watson (cjwatson) :
review: Approve
405. By <email address hidden>

Remove conflict marker

406. By <email address hidden>

Use proxy, extract sha from right location

407. By <email address hidden>

Refactor proxy handling into better named mixin

408. By <email address hidden>

Refactor out adding docker proxy

409. By <email address hidden>

Include snap build proxy code

410. By <email address hidden>

Move to an on-the-fly tarball creation

Revision history for this message
Tom Wardill (twom) wrote :

While looking at the launchpad buildbehaviour side of this, realised MP doesn't support build file location.

It should do that.

review: Needs Fixing
411. By <email address hidden>

Argument is build_file, not file

Revision history for this message
Colin Watson (cjwatson) :
review: Approve
Revision history for this message
Tom Wardill (twom) wrote :

Tested with latest buildbehaviour branch, should now be functional.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'debian/changelog'
2--- debian/changelog 2020-01-06 15:03:55 +0000
3+++ debian/changelog 2020-02-26 10:52:47 +0000
4@@ -1,3 +1,9 @@
5+launchpad-buildd (187) UNRELEASED; urgency=medium
6+
7+ * Prototype Docker image building support.
8+
9+ -- Colin Watson <cjwatson@ubuntu.com> Wed, 05 Jun 2019 15:06:54 +0100
10+
11 launchpad-buildd (186) xenial; urgency=medium
12
13 * Fix sbuildrc compatibility with xenial's sbuild.
14@@ -32,7 +38,7 @@
15
16 [ Michael Hudson-Doyle ]
17 * Do not make assumptions about what device major number the device mapper
18- is using. (LP: #1852518)
19+ is using. (LP: #1852518)
20
21 -- Colin Watson <cjwatson@ubuntu.com> Tue, 26 Nov 2019 12:22:37 +0000
22
23@@ -801,7 +807,7 @@
24 memory at once (LP: #1227086).
25
26 [ Adam Conrad ]
27- * Tidy up log formatting of the "Already reaped..." message.
28+ * Tidy up log formatting of the "Already reaped..." message.
29
30 -- Colin Watson <cjwatson@ubuntu.com> Fri, 27 Sep 2013 13:08:59 +0100
31
32@@ -986,7 +992,7 @@
33 launchpad-buildd (98) hardy; urgency=low
34
35 * Add launchpad-buildd dependency on python-apt, as an accomodation for it
36- being only a Recommends but actually required by python-debian.
37+ being only a Recommends but actually required by python-debian.
38 LP: #890834
39
40 -- Martin Pool <mbp@canonical.com> Wed, 16 Nov 2011 10:28:48 +1100
41@@ -1040,7 +1046,7 @@
42
43 launchpad-buildd (90) hardy; urgency=low
44
45- * debhelper is a Build-Depends because it is needed to run 'clean'.
46+ * debhelper is a Build-Depends because it is needed to run 'clean'.
47 * python-lpbuildd conflicts with launchpad-buildd << 88.
48 * Add and adjust build-arch, binary-arch, build-indep to match policy.
49 * Complies with stardards version 3.9.2.
50
51=== modified file 'lpbuildd/buildd-slave.tac'
52--- lpbuildd/buildd-slave.tac 2019-02-12 10:35:12 +0000
53+++ lpbuildd/buildd-slave.tac 2020-02-26 10:52:47 +0000
54@@ -23,6 +23,7 @@
55
56 from lpbuildd.binarypackage import BinaryPackageBuildManager
57 from lpbuildd.builder import XMLRPCBuilder
58+from lpbuildd.oci import OCIBuildManager
59 from lpbuildd.livefs import LiveFilesystemBuildManager
60 from lpbuildd.log import RotatableFileLogObserver
61 from lpbuildd.snap import SnapBuildManager
62@@ -45,6 +46,7 @@
63 TranslationTemplatesBuildManager, 'translation-templates')
64 builder.registerManager(LiveFilesystemBuildManager, "livefs")
65 builder.registerManager(SnapBuildManager, "snap")
66+builder.registerManager(OCIBuildManager, "oci")
67
68 application = service.Application('Builder')
69 application.addComponent(
70
71=== added file 'lpbuildd/oci.py'
72--- lpbuildd/oci.py 1970-01-01 00:00:00 +0000
73+++ lpbuildd/oci.py 2020-02-26 10:52:47 +0000
74@@ -0,0 +1,222 @@
75+# Copyright 2019 Canonical Ltd. This software is licensed under the
76+# GNU Affero General Public License version 3 (see the file LICENSE).
77+
78+from __future__ import print_function
79+
80+__metaclass__ = type
81+
82+import hashlib
83+import json
84+import os
85+import tarfile
86+import tempfile
87+
88+from six.moves.configparser import (
89+ NoOptionError,
90+ NoSectionError,
91+ )
92+
93+from lpbuildd.debian import (
94+ DebianBuildManager,
95+ DebianBuildState,
96+ )
97+from lpbuildd.snap import SnapBuildProxyMixin
98+
99+
100+RETCODE_SUCCESS = 0
101+RETCODE_FAILURE_INSTALL = 200
102+RETCODE_FAILURE_BUILD = 201
103+
104+
105+class OCIBuildState(DebianBuildState):
106+ BUILD_OCI = "BUILD_OCI"
107+
108+
109+class OCIBuildManager(SnapBuildProxyMixin, DebianBuildManager):
110+ """Build an OCI Image."""
111+
112+ backend_name = "lxd"
113+ initial_build_state = OCIBuildState.BUILD_OCI
114+
115+ @property
116+ def needs_sanitized_logs(self):
117+ return True
118+
119+ def initiate(self, files, chroot, extra_args):
120+ """Initiate a build with a given set of files and chroot."""
121+ self.name = extra_args["name"]
122+ self.branch = extra_args.get("branch")
123+ self.git_repository = extra_args.get("git_repository")
124+ self.git_path = extra_args.get("git_path")
125+ self.build_file = extra_args.get("build_file")
126+ self.proxy_url = extra_args.get("proxy_url")
127+ self.revocation_endpoint = extra_args.get("revocation_endpoint")
128+ self.proxy_service = None
129+
130+ super(OCIBuildManager, self).initiate(files, chroot, extra_args)
131+
132+ def doRunBuild(self):
133+ """Run the process to build the snap."""
134+ args = []
135+ args.extend(self.startProxy())
136+ if self.revocation_endpoint:
137+ args.extend(["--revocation-endpoint", self.revocation_endpoint])
138+ if self.branch is not None:
139+ args.extend(["--branch", self.branch])
140+ if self.git_repository is not None:
141+ args.extend(["--git-repository", self.git_repository])
142+ if self.git_path is not None:
143+ args.extend(["--git-path", self.git_path])
144+ if self.build_file is not None:
145+ args.extend(["--build-file", self.build_file])
146+ try:
147+ snap_store_proxy_url = self._builder._config.get(
148+ "proxy", "snapstore")
149+ args.extend(["--snap-store-proxy-url", snap_store_proxy_url])
150+ except (NoSectionError, NoOptionError):
151+ pass
152+ args.append(self.name)
153+ self.runTargetSubProcess("build-oci", *args)
154+
155+ def iterate_BUILD_OCI(self, retcode):
156+ """Finished building the OCI image."""
157+ self.stopProxy()
158+ self.revokeProxyToken()
159+ if retcode == RETCODE_SUCCESS:
160+ print("Returning build status: OK")
161+ return self.deferGatherResults()
162+ elif (retcode >= RETCODE_FAILURE_INSTALL and
163+ retcode <= RETCODE_FAILURE_BUILD):
164+ if not self.alreadyfailed:
165+ self._builder.buildFail()
166+ print("Returning build status: Build failed.")
167+ self.alreadyfailed = True
168+ else:
169+ if not self.alreadyfailed:
170+ self._builder.builderFail()
171+ print("Returning build status: Builder failed.")
172+ self.alreadyfailed = True
173+ self.doReapProcesses(self._state)
174+
175+ def iterateReap_BUILD_OCI(self, retcode):
176+ """Finished reaping after building the OCI image."""
177+ self._state = DebianBuildState.UMOUNT
178+ self.doUnmounting()
179+
180+ def _calculateLayerSha(self, layer_path):
181+ with open(layer_path, 'rb') as layer_tar:
182+ sha256_hash = hashlib.sha256()
183+ for byte_block in iter(lambda: layer_tar.read(4096), b""):
184+ sha256_hash.update(byte_block)
185+ digest = sha256_hash.hexdigest()
186+ return digest
187+
188+ def _gatherManifestSection(self, section, extract_path, sha_directory):
189+ config_file_path = os.path.join(extract_path, section["Config"])
190+ self._builder.addWaitingFile(config_file_path)
191+ with open(config_file_path, 'r') as config_fp:
192+ config = json.load(config_fp)
193+ diff_ids = config["rootfs"]["diff_ids"]
194+ digest_diff_map = {}
195+ for diff_id, layer_id in zip(diff_ids, section['Layers']):
196+ layer_id = layer_id.split('/')[0]
197+ diff_file = os.path.join(sha_directory, diff_id.split(':')[1])
198+ layer_path = os.path.join(
199+ extract_path, "{}.tar.gz".format(layer_id))
200+ self._builder.addWaitingFile(layer_path)
201+ # If we have a mapping between diff and existing digest,
202+ # this means this layer has been pulled from a remote.
203+ # We should maintain the same digest to achieve layer reuse
204+ if os.path.exists(diff_file):
205+ with open(diff_file, 'r') as diff_fp:
206+ diff = json.load(diff_fp)
207+ # We should be able to just take the first occurence,
208+ # as that will be the 'most parent' image
209+ digest = diff[0]["Digest"]
210+ source = diff[0]["SourceRepository"]
211+ # If the layer has been build locally, we need to generate the
212+ # digest and then set the source to empty
213+ else:
214+ source = ""
215+ digest = self._calculateLayerSha(layer_path)
216+ digest_diff_map[diff_id] = {
217+ "digest": digest,
218+ "source": source,
219+ "layer_id": layer_id
220+ }
221+
222+ return digest_diff_map
223+
224+ def gatherResults(self):
225+ """Gather the results of the build and add them to the file cache."""
226+ extract_path = tempfile.mkdtemp(prefix=self.name)
227+ proc = self.backend.run(
228+ ['docker', 'save', self.name],
229+ get_output=True, universal_newlines=False, return_process=True)
230+ try:
231+ tar = tarfile.open(fileobj=proc.stdout, mode="r|")
232+ except Exception as e:
233+ print(e)
234+
235+ current_dir = ''
236+ directory_tar = None
237+ try:
238+ # The tarfile is a stream and must be processed in order
239+ for file in tar:
240+ # Directories are just nodes, you can't extract the children
241+ # directly, so keep track of what dir we're in.
242+ if file.isdir():
243+ current_dir = file.name
244+ if directory_tar:
245+ # Close the old directory if we have one
246+ directory_tar.close()
247+ # We're going to add the layer.tar to a gzip
248+ directory_tar = tarfile.open(
249+ os.path.join(
250+ extract_path, '{}.tar.gz'.format(file.name)),
251+ 'w|gz')
252+ if current_dir and file.name.endswith('layer.tar'):
253+ # This is the actual layer data, we want to add it to
254+ # the directory gzip
255+ file.name = file.name.split('/')[1]
256+ directory_tar.addfile(file, tar.extractfile(file))
257+ elif current_dir and file.name.startswith(current_dir):
258+ # Other files that are in the layer directories,
259+ # we don't care about
260+ continue
261+ else:
262+ # If it's not in a directory, we need that
263+ tar.extract(file, extract_path)
264+ except Exception as e:
265+ print(e)
266+
267+ # We need these mapping files
268+ sha_directory = tempfile.mkdtemp()
269+ sha_path = ('/var/lib/docker/image/'
270+ 'vfs/distribution/v2metadata-by-diffid/sha256/')
271+ sha_files = [x for x in self.backend.listdir(sha_path)
272+ if not x.startswith('.')]
273+ for file in sha_files:
274+ self.backend.copy_out(
275+ os.path.join(sha_path, file),
276+ os.path.join(sha_directory, file)
277+ )
278+
279+ # Parse the manifest for the other files we need
280+ manifest_path = os.path.join(extract_path, 'manifest.json')
281+ self._builder.addWaitingFile(manifest_path)
282+ with open(manifest_path) as manifest_fp:
283+ manifest = json.load(manifest_fp)
284+
285+ digest_maps = []
286+ try:
287+ for section in manifest:
288+ digest_maps.append(
289+ self._gatherManifestSection(section, extract_path,
290+ sha_directory))
291+ digest_map_file = os.path.join(extract_path, 'digests.json')
292+ with open(digest_map_file, 'w') as digest_map_fp:
293+ json.dump(digest_maps, digest_map_fp)
294+ self._builder.addWaitingFile(digest_map_file)
295+ except Exception as e:
296+ print(e)
297
298=== modified file 'lpbuildd/snap.py'
299--- lpbuildd/snap.py 2019-10-30 12:31:53 +0000
300+++ lpbuildd/snap.py 2020-02-26 10:52:47 +0000
301@@ -239,35 +239,7 @@
302 BUILD_SNAP = "BUILD_SNAP"
303
304
305-class SnapBuildManager(DebianBuildManager):
306- """Build a snap."""
307-
308- backend_name = "lxd"
309- initial_build_state = SnapBuildState.BUILD_SNAP
310-
311- @property
312- def needs_sanitized_logs(self):
313- return True
314-
315- def initiate(self, files, chroot, extra_args):
316- """Initiate a build with a given set of files and chroot."""
317- self.name = extra_args["name"]
318- self.channels = extra_args.get("channels", {})
319- self.build_request_id = extra_args.get("build_request_id")
320- self.build_request_timestamp = extra_args.get(
321- "build_request_timestamp")
322- self.build_url = extra_args.get("build_url")
323- self.branch = extra_args.get("branch")
324- self.git_repository = extra_args.get("git_repository")
325- self.git_path = extra_args.get("git_path")
326- self.proxy_url = extra_args.get("proxy_url")
327- self.revocation_endpoint = extra_args.get("revocation_endpoint")
328- self.build_source_tarball = extra_args.get(
329- "build_source_tarball", False)
330- self.private = extra_args.get("private", False)
331- self.proxy_service = None
332-
333- super(SnapBuildManager, self).initiate(files, chroot, extra_args)
334+class SnapBuildProxyMixin():
335
336 def startProxy(self):
337 """Start the local snap proxy, if necessary."""
338@@ -308,6 +280,37 @@
339 self._builder.log(
340 "Unable to revoke token for %s: %s" % (url.username, e))
341
342+
343+class SnapBuildManager(SnapBuildProxyMixin, DebianBuildManager):
344+ """Build a snap."""
345+
346+ backend_name = "lxd"
347+ initial_build_state = SnapBuildState.BUILD_SNAP
348+
349+ @property
350+ def needs_sanitized_logs(self):
351+ return True
352+
353+ def initiate(self, files, chroot, extra_args):
354+ """Initiate a build with a given set of files and chroot."""
355+ self.name = extra_args["name"]
356+ self.channels = extra_args.get("channels", {})
357+ self.build_request_id = extra_args.get("build_request_id")
358+ self.build_request_timestamp = extra_args.get(
359+ "build_request_timestamp")
360+ self.build_url = extra_args.get("build_url")
361+ self.branch = extra_args.get("branch")
362+ self.git_repository = extra_args.get("git_repository")
363+ self.git_path = extra_args.get("git_path")
364+ self.proxy_url = extra_args.get("proxy_url")
365+ self.revocation_endpoint = extra_args.get("revocation_endpoint")
366+ self.build_source_tarball = extra_args.get(
367+ "build_source_tarball", False)
368+ self.private = extra_args.get("private", False)
369+ self.proxy_service = None
370+
371+ super(SnapBuildManager, self).initiate(files, chroot, extra_args)
372+
373 def status(self):
374 status_path = get_build_path(self.home, self._buildid, "status")
375 try:
376
377=== added file 'lpbuildd/target/build_oci.py'
378--- lpbuildd/target/build_oci.py 1970-01-01 00:00:00 +0000
379+++ lpbuildd/target/build_oci.py 2020-02-26 10:52:47 +0000
380@@ -0,0 +1,126 @@
381+# Copyright 2019 Canonical Ltd. This software is licensed under the
382+# GNU Affero General Public License version 3 (see the file LICENSE).
383+
384+from __future__ import print_function
385+
386+__metaclass__ = type
387+
388+from collections import OrderedDict
389+import logging
390+import os.path
391+import sys
392+import tempfile
393+from textwrap import dedent
394+
395+from lpbuildd.target.operation import Operation
396+from lpbuildd.target.snapbuildproxy import SnapBuildProxyOperationMixin
397+from lpbuildd.target.snapstore import SnapStoreOperationMixin
398+from lpbuildd.target.vcs import VCSOperationMixin
399+
400+
401+RETCODE_FAILURE_INSTALL = 200
402+RETCODE_FAILURE_BUILD = 201
403+
404+
405+logger = logging.getLogger(__name__)
406+
407+
408+class BuildOCI(SnapBuildProxyOperationMixin, VCSOperationMixin,
409+ SnapStoreOperationMixin, Operation):
410+
411+ description = "Build an OCI image."
412+
413+ @classmethod
414+ def add_arguments(cls, parser):
415+ super(BuildOCI, cls).add_arguments(parser)
416+ parser.add_argument(
417+ "--build-file", help="path to Dockerfile in branch")
418+ parser.add_argument("name", help="name of snap to build")
419+
420+ def __init__(self, args, parser):
421+ super(BuildOCI, self).__init__(args, parser)
422+ self.bin = os.path.dirname(sys.argv[0])
423+
424+ def _add_docker_engine_proxy_settings(self):
425+ """Add systemd file for docker proxy settings."""
426+ # Create containing directory for systemd overrides
427+ self.backend.run(
428+ ["mkdir", "-p", "/etc/systemd/system/docker.service.d"])
429+ # we need both http_proxy and https_proxy. The contents of the files
430+ # are otherwise identical
431+ for setting in ['http_proxy', 'https_proxy']:
432+ contents = dedent("""[Service]
433+ Environment="{}={}"
434+ """.format(setting.upper(), self.args.proxy_url))
435+ file_path = "/etc/systemd/system/docker.service.d/{}.conf".format(
436+ setting)
437+ with tempfile.NamedTemporaryFile(mode="w+") as systemd_file:
438+ systemd_file.write(contents)
439+ systemd_file.flush()
440+ self.backend.copy_in(systemd_file.name, file_path)
441+
442+ def run_build_command(self, args, env=None, **kwargs):
443+ """Run a build command in the target.
444+
445+ :param args: the command and arguments to run.
446+ :param env: dictionary of additional environment variables to set.
447+ :param kwargs: any other keyword arguments to pass to Backend.run.
448+ """
449+ full_env = OrderedDict()
450+ full_env["LANG"] = "C.UTF-8"
451+ full_env["SHELL"] = "/bin/sh"
452+ if env:
453+ full_env.update(env)
454+ return self.backend.run(args, env=full_env, **kwargs)
455+
456+ def install(self):
457+ logger.info("Running install phase...")
458+ deps = []
459+ if self.args.proxy_url:
460+ deps.extend(self.proxy_deps)
461+ self.install_git_proxy()
462+ # Add any proxy settings that are needed
463+ self._add_docker_engine_proxy_settings()
464+ deps.extend(self.vcs_deps)
465+ deps.extend(["docker.io"])
466+ self.backend.run(["apt-get", "-y", "install"] + deps)
467+ if self.args.backend in ("lxd", "fake"):
468+ self.snap_store_set_proxy()
469+ self.backend.run(["systemctl", "restart", "docker"])
470+ # The docker snap can't see /build, so we have to do our work under
471+ # /home/buildd instead. Make sure it exists.
472+ self.backend.run(["mkdir", "-p", "/home/buildd"])
473+
474+ def repo(self):
475+ """Collect git or bzr branch."""
476+ logger.info("Running repo phase...")
477+ env = self.build_proxy_environment(proxy_url=self.args.proxy_url)
478+ self.vcs_fetch(self.args.name, cwd="/home/buildd", env=env)
479+
480+ def build(self):
481+ logger.info("Running build phase...")
482+ args = ["docker", "build", "--no-cache"]
483+ if self.args.proxy_url:
484+ for var in ("http_proxy", "https_proxy"):
485+ args.extend(
486+ ["--build-arg", "{}={}".format(var, self.args.proxy_url)])
487+ args.extend(["--tag", self.args.name])
488+ if self.args.build_file is not None:
489+ args.extend(["--file", self.args.build_file])
490+ buildd_path = os.path.join("/home/buildd", self.args.name)
491+ args.append(buildd_path)
492+ self.run_build_command(args)
493+
494+ def run(self):
495+ try:
496+ self.install()
497+ except Exception:
498+ logger.exception('Install failed')
499+ return RETCODE_FAILURE_INSTALL
500+ try:
501+ self.repo()
502+ self.build()
503+ except Exception:
504+ logger.exception('Build failed')
505+ return RETCODE_FAILURE_BUILD
506+ return 0
507
508=== modified file 'lpbuildd/target/build_snap.py'
509--- lpbuildd/target/build_snap.py 2019-10-30 12:31:53 +0000
510+++ lpbuildd/target/build_snap.py 2020-02-26 10:52:47 +0000
511@@ -17,6 +17,7 @@
512 from six.moves.urllib.parse import urlparse
513
514 from lpbuildd.target.operation import Operation
515+from lpbuildd.target.snapbuildproxy import SnapBuildProxyOperationMixin
516 from lpbuildd.target.snapstore import SnapStoreOperationMixin
517 from lpbuildd.target.vcs import VCSOperationMixin
518
519@@ -46,7 +47,8 @@
520 getattr(namespace, self.dest)[snap] = channel
521
522
523-class BuildSnap(VCSOperationMixin, SnapStoreOperationMixin, Operation):
524+class BuildSnap(SnapBuildProxyOperationMixin, VCSOperationMixin,
525+ SnapStoreOperationMixin, Operation):
526
527 description = "Build a snap."
528
529@@ -69,10 +71,6 @@
530 help="RFC3339 timestamp of the Launchpad build request")
531 parser.add_argument(
532 "--build-url", help="URL of this build on Launchpad")
533- parser.add_argument("--proxy-url", help="builder proxy url")
534- parser.add_argument(
535- "--revocation-endpoint",
536- help="builder proxy token revocation endpoint")
537 parser.add_argument(
538 "--build-source-tarball", default=False, action="store_true",
539 help=(
540@@ -137,6 +135,9 @@
541 def install(self):
542 logger.info("Running install phase...")
543 deps = []
544+ if self.args.proxy_url:
545+ deps.extend(self.proxy_deps)
546+ self.install_git_proxy()
547 if self.args.backend == "lxd":
548 # udev is installed explicitly to work around
549 # https://bugs.launchpad.net/snapd/+bug/1731519.
550@@ -144,8 +145,6 @@
551 if self.backend.is_package_available(dep):
552 deps.append(dep)
553 deps.extend(self.vcs_deps)
554- if self.args.proxy_url:
555- deps.extend(["python3", "socat"])
556 if "snapcraft" in self.args.channels:
557 # snapcraft requires sudo in lots of places, but can't depend on
558 # it when installed as a snap.
559@@ -167,19 +166,12 @@
560 "--channel=%s" % self.args.channels["snapcraft"],
561 "snapcraft"])
562 if self.args.proxy_url:
563- self.backend.copy_in(
564- os.path.join(self.bin, "snap-git-proxy"),
565- "/usr/local/bin/snap-git-proxy")
566 self.install_svn_servers()
567
568 def repo(self):
569 """Collect git or bzr branch."""
570 logger.info("Running repo phase...")
571- env = OrderedDict()
572- if self.args.proxy_url:
573- env["http_proxy"] = self.args.proxy_url
574- env["https_proxy"] = self.args.proxy_url
575- env["GIT_PROXY_COMMAND"] = "/usr/local/bin/snap-git-proxy"
576+ env = self.build_proxy_environment(proxy_url=self.args.proxy_url)
577 self.vcs_fetch(self.args.name, cwd="/build", env=env)
578 status = {}
579 if self.args.branch is not None:
580
581=== modified file 'lpbuildd/target/cli.py'
582--- lpbuildd/target/cli.py 2017-09-08 15:57:18 +0000
583+++ lpbuildd/target/cli.py 2020-02-26 10:52:47 +0000
584@@ -14,6 +14,7 @@
585 OverrideSourcesList,
586 Update,
587 )
588+from lpbuildd.target.build_oci import BuildOCI
589 from lpbuildd.target.build_livefs import BuildLiveFS
590 from lpbuildd.target.build_snap import BuildSnap
591 from lpbuildd.target.generate_translation_templates import (
592@@ -49,6 +50,7 @@
593
594 operations = {
595 "add-trusted-keys": AddTrustedKeys,
596+ "build-oci": BuildOCI,
597 "buildlivefs": BuildLiveFS,
598 "buildsnap": BuildSnap,
599 "generate-translation-templates": GenerateTranslationTemplates,
600
601=== modified file 'lpbuildd/target/lxd.py'
602--- lpbuildd/target/lxd.py 2019-12-09 11:17:14 +0000
603+++ lpbuildd/target/lxd.py 2020-02-26 10:52:47 +0000
604@@ -504,7 +504,8 @@
605 "/etc/systemd/system/snapd.refresh.timer"])
606
607 def run(self, args, cwd=None, env=None, input_text=None, get_output=False,
608- echo=False, **kwargs):
609+ echo=False, return_process=False, universal_newlines=True,
610+ **kwargs):
611 """See `Backend`."""
612 env_params = []
613 if env:
614@@ -538,7 +539,10 @@
615 if get_output:
616 kwargs["stdout"] = subprocess.PIPE
617 proc = subprocess.Popen(
618- cmd, stdin=subprocess.PIPE, universal_newlines=True, **kwargs)
619+ cmd, stdin=subprocess.PIPE,
620+ universal_newlines=universal_newlines, **kwargs)
621+ if return_process:
622+ return proc
623 output, _ = proc.communicate(input_text)
624 if proc.returncode:
625 raise subprocess.CalledProcessError(proc.returncode, cmd)
626
627=== added file 'lpbuildd/target/snapbuildproxy.py'
628--- lpbuildd/target/snapbuildproxy.py 1970-01-01 00:00:00 +0000
629+++ lpbuildd/target/snapbuildproxy.py 2020-02-26 10:52:47 +0000
630@@ -0,0 +1,41 @@
631+# Copyright 2019-2020 Canonical Ltd. This software is licensed under the
632+# GNU Affero General Public License version 3 (see the file LICENSE).
633+
634+from __future__ import print_function
635+
636+__metaclass__ = type
637+
638+from collections import OrderedDict
639+import os
640+
641+
642+class SnapBuildProxyOperationMixin:
643+ """Methods supporting the build time HTTP proxy for snap and OCI builds."""
644+
645+ @classmethod
646+ def add_arguments(cls, parser):
647+ super(SnapBuildProxyOperationMixin, cls).add_arguments(parser)
648+ parser.add_argument("--proxy-url", help="builder proxy url")
649+ parser.add_argument(
650+ "--revocation-endpoint",
651+ help="builder proxy token revocation endpoint")
652+
653+ @property
654+ def proxy_deps(self):
655+ return ["python3", "socat"]
656+
657+ def install_git_proxy(self):
658+ self.backend.copy_in(
659+ os.path.join(self.bin, "snap-git-proxy"),
660+ "/usr/local/bin/snap-git-proxy")
661+
662+ def build_proxy_environment(self, proxy_url=None, env=None):
663+ """Extend a command environment to include http proxy variables."""
664+ full_env = OrderedDict()
665+ if env:
666+ full_env.update(env)
667+ if proxy_url:
668+ full_env["http_proxy"] = self.args.proxy_url
669+ full_env["https_proxy"] = self.args.proxy_url
670+ full_env["GIT_PROXY_COMMAND"] = "/usr/local/bin/snap-git-proxy"
671+ return full_env
672
673=== added file 'lpbuildd/target/tests/test_build_oci.py'
674--- lpbuildd/target/tests/test_build_oci.py 1970-01-01 00:00:00 +0000
675+++ lpbuildd/target/tests/test_build_oci.py 2020-02-26 10:52:47 +0000
676@@ -0,0 +1,407 @@
677+# Copyright 2019 Canonical Ltd. This software is licensed under the
678+# GNU Affero General Public License version 3 (see the file LICENSE).
679+
680+__metaclass__ = type
681+
682+import os.path
683+import stat
684+import subprocess
685+from textwrap import dedent
686+
687+from fixtures import (
688+ FakeLogger,
689+ TempDir,
690+ )
691+import responses
692+from systemfixtures import FakeFilesystem
693+from testtools import TestCase
694+from testtools.matchers import (
695+ AnyMatch,
696+ Equals,
697+ Is,
698+ MatchesAll,
699+ MatchesDict,
700+ MatchesListwise,
701+ )
702+
703+from lpbuildd.target.build_oci import (
704+ RETCODE_FAILURE_BUILD,
705+ RETCODE_FAILURE_INSTALL,
706+ )
707+from lpbuildd.target.cli import parse_args
708+from lpbuildd.tests.fakebuilder import FakeMethod
709+
710+
711+class RanCommand(MatchesListwise):
712+
713+ def __init__(self, args, echo=None, cwd=None, input_text=None,
714+ get_output=None, **env):
715+ kwargs_matcher = {}
716+ if echo is not None:
717+ kwargs_matcher["echo"] = Is(echo)
718+ if cwd:
719+ kwargs_matcher["cwd"] = Equals(cwd)
720+ if input_text:
721+ kwargs_matcher["input_text"] = Equals(input_text)
722+ if get_output is not None:
723+ kwargs_matcher["get_output"] = Is(get_output)
724+ if env:
725+ kwargs_matcher["env"] = MatchesDict(
726+ {key: Equals(value) for key, value in env.items()})
727+ super(RanCommand, self).__init__(
728+ [Equals((args,)), MatchesDict(kwargs_matcher)])
729+
730+
731+class RanAptGet(RanCommand):
732+
733+ def __init__(self, *args):
734+ super(RanAptGet, self).__init__(["apt-get", "-y"] + list(args))
735+
736+
737+class RanSnap(RanCommand):
738+
739+ def __init__(self, *args, **kwargs):
740+ super(RanSnap, self).__init__(["snap"] + list(args), **kwargs)
741+
742+
743+class RanBuildCommand(RanCommand):
744+
745+ def __init__(self, args, **kwargs):
746+ kwargs.setdefault("LANG", "C.UTF-8")
747+ kwargs.setdefault("SHELL", "/bin/sh")
748+ super(RanBuildCommand, self).__init__(args, **kwargs)
749+
750+
751+class TestBuildOCI(TestCase):
752+
753+ def test_run_build_command_no_env(self):
754+ args = [
755+ "build-oci",
756+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
757+ "--branch", "lp:foo", "test-image",
758+ ]
759+ build_oci = parse_args(args=args).operation
760+ build_oci.run_build_command(["echo", "hello world"])
761+ self.assertThat(build_oci.backend.run.calls, MatchesListwise([
762+ RanBuildCommand(["echo", "hello world"]),
763+ ]))
764+
765+ def test_run_build_command_env(self):
766+ args = [
767+ "build-oci",
768+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
769+ "--branch", "lp:foo", "test-image",
770+ ]
771+ build_oci = parse_args(args=args).operation
772+ build_oci.run_build_command(
773+ ["echo", "hello world"], env={"FOO": "bar baz"})
774+ self.assertThat(build_oci.backend.run.calls, MatchesListwise([
775+ RanBuildCommand(["echo", "hello world"], FOO="bar baz"),
776+ ]))
777+
778+ def test_install_bzr(self):
779+ args = [
780+ "build-oci",
781+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
782+ "--branch", "lp:foo", "test-image"
783+ ]
784+ build_oci = parse_args(args=args).operation
785+ build_oci.install()
786+ self.assertThat(build_oci.backend.run.calls, MatchesListwise([
787+ RanAptGet("install", "bzr", "docker.io"),
788+ RanCommand(["systemctl", "restart", "docker"]),
789+ RanCommand(["mkdir", "-p", "/home/buildd"]),
790+ ]))
791+
792+ def test_install_git(self):
793+ args = [
794+ "build-oci",
795+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
796+ "--git-repository", "lp:foo", "test-image"
797+ ]
798+ build_oci = parse_args(args=args).operation
799+ build_oci.install()
800+ self.assertThat(build_oci.backend.run.calls, MatchesListwise([
801+ RanAptGet("install", "git", "docker.io"),
802+ RanCommand(["systemctl", "restart", "docker"]),
803+ RanCommand(["mkdir", "-p", "/home/buildd"]),
804+ ]))
805+
806+ @responses.activate
807+ def test_install_snap_store_proxy(self):
808+ store_assertion = dedent("""\
809+ type: store
810+ store: store-id
811+ url: http://snap-store-proxy.example
812+
813+ body
814+ """)
815+
816+ def respond(request):
817+ return 200, {"X-Assertion-Store-Id": "store-id"}, store_assertion
818+
819+ responses.add_callback(
820+ "GET", "http://snap-store-proxy.example/v2/auth/store/assertions",
821+ callback=respond)
822+ args = [
823+ "buildsnap",
824+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
825+ "--git-repository", "lp:foo",
826+ "--snap-store-proxy-url", "http://snap-store-proxy.example/",
827+ "test-snap",
828+ ]
829+ build_snap = parse_args(args=args).operation
830+ build_snap.install()
831+ self.assertThat(build_snap.backend.run.calls, MatchesListwise([
832+ RanAptGet("install", "git", "snapcraft"),
833+ RanCommand(
834+ ["snap", "ack", "/dev/stdin"], input_text=store_assertion),
835+ RanCommand(["snap", "set", "core", "proxy.store=store-id"]),
836+ ]))
837+
838+ def test_install_proxy(self):
839+ args = [
840+ "build-oci",
841+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
842+ "--git-repository", "lp:foo",
843+ "--proxy-url", "http://proxy.example:3128/",
844+ "test-image",
845+ ]
846+ build_oci = parse_args(args=args).operation
847+ build_oci.bin = "/builderbin"
848+ self.useFixture(FakeFilesystem()).add("/builderbin")
849+ os.mkdir("/builderbin")
850+ with open("/builderbin/snap-git-proxy", "w") as proxy_script:
851+ proxy_script.write("proxy script\n")
852+ os.fchmod(proxy_script.fileno(), 0o755)
853+ build_oci.install()
854+ self.assertThat(build_oci.backend.run.calls, MatchesListwise([
855+ RanCommand(
856+ ["mkdir", "-p", "/etc/systemd/system/docker.service.d"]),
857+ RanAptGet("install", "python3", "socat", "git", "docker.io"),
858+ RanCommand(["systemctl", "restart", "docker"]),
859+ RanCommand(["mkdir", "-p", "/home/buildd"]),
860+ ]))
861+ self.assertEqual(
862+ (b"proxy script\n", stat.S_IFREG | 0o755),
863+ build_oci.backend.backend_fs["/usr/local/bin/snap-git-proxy"])
864+
865+ def test_repo_bzr(self):
866+ args = [
867+ "build-oci",
868+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
869+ "--branch", "lp:foo", "test-image",
870+ ]
871+ build_oci = parse_args(args=args).operation
872+ build_oci.backend.build_path = self.useFixture(TempDir()).path
873+ build_oci.backend.run = FakeMethod()
874+ build_oci.repo()
875+ self.assertThat(build_oci.backend.run.calls, MatchesListwise([
876+ RanBuildCommand(
877+ ["bzr", "branch", "lp:foo", "test-image"], cwd="/home/buildd"),
878+ ]))
879+
880+ def test_repo_git(self):
881+ args = [
882+ "build-oci",
883+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
884+ "--git-repository", "lp:foo", "test-image",
885+ ]
886+ build_oci = parse_args(args=args).operation
887+ build_oci.backend.build_path = self.useFixture(TempDir()).path
888+ build_oci.backend.run = FakeMethod()
889+ build_oci.repo()
890+ self.assertThat(build_oci.backend.run.calls, MatchesListwise([
891+ RanBuildCommand(
892+ ["git", "clone", "lp:foo", "test-image"], cwd="/home/buildd"),
893+ RanBuildCommand(
894+ ["git", "submodule", "update", "--init", "--recursive"],
895+ cwd="/home/buildd/test-image"),
896+ ]))
897+
898+ def test_repo_git_with_path(self):
899+ args = [
900+ "build-oci",
901+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
902+ "--git-repository", "lp:foo", "--git-path", "next", "test-image",
903+ ]
904+ build_oci = parse_args(args=args).operation
905+ build_oci.backend.build_path = self.useFixture(TempDir()).path
906+ build_oci.backend.run = FakeMethod()
907+ build_oci.repo()
908+ self.assertThat(build_oci.backend.run.calls, MatchesListwise([
909+ RanBuildCommand(
910+ ["git", "clone", "-b", "next", "lp:foo", "test-image"],
911+ cwd="/home/buildd"),
912+ RanBuildCommand(
913+ ["git", "submodule", "update", "--init", "--recursive"],
914+ cwd="/home/buildd/test-image"),
915+ ]))
916+
917+ def test_repo_git_with_tag_path(self):
918+ args = [
919+ "build-oci",
920+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
921+ "--git-repository", "lp:foo", "--git-path", "refs/tags/1.0",
922+ "test-image",
923+ ]
924+ build_oci = parse_args(args=args).operation
925+ build_oci.backend.build_path = self.useFixture(TempDir()).path
926+ build_oci.backend.run = FakeMethod()
927+ build_oci.repo()
928+ self.assertThat(build_oci.backend.run.calls, MatchesListwise([
929+ RanBuildCommand(
930+ ["git", "clone", "-b", "1.0", "lp:foo", "test-image"],
931+ cwd="/home/buildd"),
932+ RanBuildCommand(
933+ ["git", "submodule", "update", "--init", "--recursive"],
934+ cwd="/home/buildd/test-image"),
935+ ]))
936+
937+ def test_repo_proxy(self):
938+ args = [
939+ "build-oci",
940+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
941+ "--git-repository", "lp:foo",
942+ "--proxy-url", "http://proxy.example:3128/",
943+ "test-image",
944+ ]
945+ build_oci = parse_args(args=args).operation
946+ build_oci.backend.build_path = self.useFixture(TempDir()).path
947+ build_oci.backend.run = FakeMethod()
948+ build_oci.repo()
949+ env = {
950+ "http_proxy": "http://proxy.example:3128/",
951+ "https_proxy": "http://proxy.example:3128/",
952+ "GIT_PROXY_COMMAND": "/usr/local/bin/snap-git-proxy",
953+ }
954+ self.assertThat(build_oci.backend.run.calls, MatchesListwise([
955+ RanBuildCommand(
956+ ["git", "clone", "lp:foo", "test-image"],
957+ cwd="/home/buildd", **env),
958+ RanBuildCommand(
959+ ["git", "submodule", "update", "--init", "--recursive"],
960+ cwd="/home/buildd/test-image", **env),
961+ ]))
962+
963+ def test_build(self):
964+ args = [
965+ "build-oci",
966+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
967+ "--branch", "lp:foo", "test-image",
968+ ]
969+ build_oci = parse_args(args=args).operation
970+ build_oci.backend.add_dir('/build/test-directory')
971+ build_oci.build()
972+ self.assertThat(build_oci.backend.run.calls, MatchesListwise([
973+ RanBuildCommand(
974+ ["docker", "build", "--no-cache", "--tag", "test-image",
975+ "/home/buildd/test-image"]),
976+ ]))
977+
978+ def test_build_with_file(self):
979+ args = [
980+ "build-oci",
981+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
982+ "--branch", "lp:foo", "--build-file", "build-aux/Dockerfile",
983+ "test-image",
984+ ]
985+ build_oci = parse_args(args=args).operation
986+ build_oci.backend.add_dir('/build/test-directory')
987+ build_oci.build()
988+ self.assertThat(build_oci.backend.run.calls, MatchesListwise([
989+ RanBuildCommand(
990+ ["docker", "build", "--no-cache", "--tag", "test-image",
991+ "--file", "build-aux/Dockerfile",
992+ "/home/buildd/test-image"]),
993+ ]))
994+
995+ def test_build_proxy(self):
996+ args = [
997+ "build-oci",
998+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
999+ "--branch", "lp:foo", "--proxy-url", "http://proxy.example:3128/",
1000+ "test-image",
1001+ ]
1002+ build_oci = parse_args(args=args).operation
1003+ build_oci.backend.add_dir('/build/test-directory')
1004+ build_oci.build()
1005+ self.assertThat(build_oci.backend.run.calls, MatchesListwise([
1006+ RanBuildCommand(
1007+ ["docker", "build", "--no-cache",
1008+ "--build-arg", "http_proxy=http://proxy.example:3128/",
1009+ "--build-arg", "https_proxy=http://proxy.example:3128/",
1010+ "--tag", "test-image", "/home/buildd/test-image"]),
1011+ ]))
1012+
1013+ def test_run_succeeds(self):
1014+ args = [
1015+ "build-oci",
1016+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
1017+ "--branch", "lp:foo", "test-image",
1018+ ]
1019+ build_oci = parse_args(args=args).operation
1020+ build_oci.backend.build_path = self.useFixture(TempDir()).path
1021+ build_oci.backend.run = FakeMethod()
1022+ self.assertEqual(0, build_oci.run())
1023+ self.assertThat(build_oci.backend.run.calls, MatchesAll(
1024+ AnyMatch(RanAptGet("install", "bzr", "docker.io")),
1025+ AnyMatch(RanBuildCommand(
1026+ ["bzr", "branch", "lp:foo", "test-image"],
1027+ cwd="/home/buildd")),
1028+ AnyMatch(RanBuildCommand(
1029+ ["docker", "build", "--no-cache", "--tag", "test-image",
1030+ "/home/buildd/test-image"])),
1031+ ))
1032+
1033+ def test_run_install_fails(self):
1034+ class FailInstall(FakeMethod):
1035+ def __call__(self, run_args, *args, **kwargs):
1036+ super(FailInstall, self).__call__(run_args, *args, **kwargs)
1037+ if run_args[0] == "apt-get":
1038+ raise subprocess.CalledProcessError(1, run_args)
1039+
1040+ self.useFixture(FakeLogger())
1041+ args = [
1042+ "build-oci",
1043+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
1044+ "--branch", "lp:foo", "test-image",
1045+ ]
1046+ build_oci = parse_args(args=args).operation
1047+ build_oci.backend.run = FailInstall()
1048+ self.assertEqual(RETCODE_FAILURE_INSTALL, build_oci.run())
1049+
1050+ def test_run_repo_fails(self):
1051+ class FailRepo(FakeMethod):
1052+ def __call__(self, run_args, *args, **kwargs):
1053+ super(FailRepo, self).__call__(run_args, *args, **kwargs)
1054+ if run_args[:2] == ["bzr", "branch"]:
1055+ raise subprocess.CalledProcessError(1, run_args)
1056+
1057+ self.useFixture(FakeLogger())
1058+ args = [
1059+ "build-oci",
1060+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
1061+ "--branch", "lp:foo", "test-image",
1062+ ]
1063+ build_oci = parse_args(args=args).operation
1064+ build_oci.backend.run = FailRepo()
1065+ self.assertEqual(RETCODE_FAILURE_BUILD, build_oci.run())
1066+
1067+ def test_run_build_fails(self):
1068+ class FailBuild(FakeMethod):
1069+ def __call__(self, run_args, *args, **kwargs):
1070+ super(FailBuild, self).__call__(run_args, *args, **kwargs)
1071+ if run_args[0] == "docker":
1072+ raise subprocess.CalledProcessError(1, run_args)
1073+
1074+ self.useFixture(FakeLogger())
1075+ args = [
1076+ "build-oci",
1077+ "--backend=fake", "--series=xenial", "--arch=amd64", "1",
1078+ "--branch", "lp:foo", "test-image",
1079+ ]
1080+ build_oci = parse_args(args=args).operation
1081+ build_oci.backend.build_path = self.useFixture(TempDir()).path
1082+ build_oci.backend.run = FailBuild()
1083+ self.assertEqual(RETCODE_FAILURE_BUILD, build_oci.run())
1084
1085=== modified file 'lpbuildd/target/tests/test_build_snap.py'
1086--- lpbuildd/target/tests/test_build_snap.py 2019-06-05 14:10:20 +0000
1087+++ lpbuildd/target/tests/test_build_snap.py 2020-02-26 10:52:47 +0000
1088@@ -186,7 +186,7 @@
1089 os.fchmod(proxy_script.fileno(), 0o755)
1090 build_snap.install()
1091 self.assertThat(build_snap.backend.run.calls, MatchesListwise([
1092- RanAptGet("install", "git", "python3", "socat", "snapcraft"),
1093+ RanAptGet("install", "python3", "socat", "git", "snapcraft"),
1094 RanCommand(["mkdir", "-p", "/root/.subversion"]),
1095 ]))
1096 self.assertEqual(
1097
1098=== modified file 'lpbuildd/target/tests/test_lxd.py'
1099--- lpbuildd/target/tests/test_lxd.py 2019-12-09 11:17:14 +0000
1100+++ lpbuildd/target/tests/test_lxd.py 2020-02-26 10:52:47 +0000
1101@@ -276,7 +276,7 @@
1102 "lp-xenial-amd64", "lp-xenial-amd64")
1103
1104 def assert_correct_profile(self, extra_raw_lxc_config=None,
1105- driver_version="2.0"):
1106+ driver_version="2.0"):
1107 if extra_raw_lxc_config is None:
1108 extra_raw_lxc_config = []
1109
1110@@ -356,7 +356,7 @@
1111 }
1112 LXD("1", "xenial", "powerpc").create_profile()
1113 self.assert_correct_profile(
1114- extra_raw_lxc_config=[("lxc.seccomp", ""),],
1115+ extra_raw_lxc_config=[("lxc.seccomp", ""), ],
1116 driver_version=driver_version or "3.0"
1117 )
1118
1119@@ -463,7 +463,8 @@
1120 "b", "7", str(minor)]))
1121 if not with_dm0:
1122 expected_args.extend([
1123- Equals(["sudo", "dmsetup", "create", "tmpdevice", "--notable"]),
1124+ Equals(
1125+ ["sudo", "dmsetup", "create", "tmpdevice", "--notable"]),
1126 Equals(["sudo", "dmsetup", "remove", "tmpdevice"]),
1127 ])
1128 for minor in range(8):
1129
1130=== added file 'lpbuildd/tests/oci_tarball.py'
1131--- lpbuildd/tests/oci_tarball.py 1970-01-01 00:00:00 +0000
1132+++ lpbuildd/tests/oci_tarball.py 2020-02-26 10:52:47 +0000
1133@@ -0,0 +1,60 @@
1134+import json
1135+import os
1136+import StringIO
1137+import tempfile
1138+import tarfile
1139+
1140+
1141+class OCITarball:
1142+ """Create a tarball for use in tests with OCI."""
1143+
1144+ def _makeFile(self, contents, name):
1145+ json_contents = json.dumps(contents)
1146+ tarinfo = tarfile.TarInfo(name)
1147+ tarinfo.size = len(json_contents)
1148+ return tarinfo, StringIO.StringIO(json_contents)
1149+
1150+ @property
1151+ def config(self):
1152+ return self._makeFile(
1153+ {"rootfs": {"diff_ids": ["sha256:diff1", "sha256:diff2"]}},
1154+ 'config.json')
1155+
1156+ @property
1157+ def manifest(self):
1158+ return self._makeFile(
1159+ [{"Config": "config.json",
1160+ "Layers": ["layer-1/layer.tar", "layer-2/layer.tar"]}],
1161+ 'manifest.json')
1162+
1163+ @property
1164+ def repositories(self):
1165+ return self._makeFile([], 'repositories')
1166+
1167+ def layer_file(self, directory, layer_name):
1168+ contents = "{}-contents".format(layer_name)
1169+ tarinfo = tarfile.TarInfo(contents)
1170+ tarinfo.size = len(contents)
1171+ layer_contents = StringIO.StringIO(contents)
1172+ layer_tar_path = os.path.join(
1173+ directory, '{}.tar.gz'.format(layer_name))
1174+ layer_tar = tarfile.open(layer_tar_path, 'w:gz')
1175+ layer_tar.addfile(tarinfo, layer_contents)
1176+ layer_tar.close()
1177+ return layer_tar_path
1178+
1179+ def build_tar_file(self):
1180+ tar_directory = tempfile.mkdtemp()
1181+ tar_path = os.path.join(tar_directory, 'test-oci-image.tar')
1182+ tar = tarfile.open(tar_path, 'w')
1183+ tar.addfile(*self.config)
1184+ tar.addfile(*self.manifest)
1185+ tar.addfile(*self.repositories)
1186+
1187+ for layer_name in ['layer-1', 'layer-2']:
1188+ layer = self.layer_file(tar_directory, layer_name)
1189+ tar.add(layer, arcname='{}.tar.gz'.format(layer_name))
1190+
1191+ tar.close()
1192+
1193+ return tar_path
1194
1195=== added file 'lpbuildd/tests/test_oci.py'
1196--- lpbuildd/tests/test_oci.py 1970-01-01 00:00:00 +0000
1197+++ lpbuildd/tests/test_oci.py 2020-02-26 10:52:47 +0000
1198@@ -0,0 +1,280 @@
1199+# Copyright 2019 Canonical Ltd. This software is licensed under the
1200+# GNU Affero General Public License version 3 (see the file LICENSE).
1201+
1202+__metaclass__ = type
1203+
1204+import io
1205+import json
1206+import os
1207+try:
1208+ from unittest import mock
1209+except ImportError:
1210+ import mock
1211+
1212+from fixtures import (
1213+ EnvironmentVariable,
1214+ MockPatch,
1215+ TempDir,
1216+ )
1217+from testtools import TestCase
1218+from testtools.matchers import Contains
1219+from testtools.deferredruntest import AsynchronousDeferredRunTest
1220+from twisted.internet import defer
1221+
1222+from lpbuildd.oci import (
1223+ OCIBuildManager,
1224+ OCIBuildState,
1225+ )
1226+from lpbuildd.tests.fakebuilder import FakeBuilder
1227+from lpbuildd.tests.oci_tarball import OCITarball
1228+
1229+
1230+class MockBuildManager(OCIBuildManager):
1231+ def __init__(self, *args, **kwargs):
1232+ super(MockBuildManager, self).__init__(*args, **kwargs)
1233+ self.commands = []
1234+ self.iterators = []
1235+
1236+ def runSubProcess(self, path, command, iterate=None, env=None):
1237+ self.commands.append([path] + command)
1238+ if iterate is None:
1239+ iterate = self.iterate
1240+ self.iterators.append(iterate)
1241+ return 0
1242+
1243+
1244+class MockOCITarSave():
1245+ @property
1246+ def stdout(self):
1247+ return io.open(OCITarball().build_tar_file(), 'rb')
1248+
1249+
1250+class TestOCIBuildManagerIteration(TestCase):
1251+ """Run OCIBuildManager through its iteration steps."""
1252+
1253+ run_tests_with = AsynchronousDeferredRunTest.make_factory(timeout=5)
1254+
1255+ def setUp(self):
1256+ super(TestOCIBuildManagerIteration, self).setUp()
1257+ self.working_dir = self.useFixture(TempDir()).path
1258+ builder_dir = os.path.join(self.working_dir, "builder")
1259+ home_dir = os.path.join(self.working_dir, "home")
1260+ for dir in (builder_dir, home_dir):
1261+ os.mkdir(dir)
1262+ self.useFixture(EnvironmentVariable("HOME", home_dir))
1263+ self.builder = FakeBuilder(builder_dir)
1264+ self.buildid = "123"
1265+ self.buildmanager = MockBuildManager(self.builder, self.buildid)
1266+ self.buildmanager._cachepath = self.builder._cachepath
1267+
1268+ def getState(self):
1269+ """Retrieve build manager's state."""
1270+ return self.buildmanager._state
1271+
1272+ @defer.inlineCallbacks
1273+ def startBuild(self, args=None, options=None):
1274+ # The build manager's iterate() kicks off the consecutive states
1275+ # after INIT.
1276+ extra_args = {
1277+ "series": "xenial",
1278+ "arch_tag": "i386",
1279+ "name": "test-image",
1280+ }
1281+ if args is not None:
1282+ extra_args.update(args)
1283+ original_backend_name = self.buildmanager.backend_name
1284+ self.buildmanager.backend_name = "fake"
1285+ self.buildmanager.initiate({}, "chroot.tar.gz", extra_args)
1286+ self.buildmanager.backend_name = original_backend_name
1287+
1288+ # Skip states that are done in DebianBuildManager to the state
1289+ # directly before BUILD_OCI.
1290+ self.buildmanager._state = OCIBuildState.UPDATE
1291+
1292+ # BUILD_OCI: Run the builder's payload to build the snap package.
1293+ yield self.buildmanager.iterate(0)
1294+ self.assertEqual(OCIBuildState.BUILD_OCI, self.getState())
1295+ expected_command = [
1296+ "sharepath/bin/in-target", "in-target", "build-oci",
1297+ "--backend=lxd", "--series=xenial", "--arch=i386", self.buildid,
1298+ ]
1299+ if options is not None:
1300+ expected_command.extend(options)
1301+ expected_command.append("test-image")
1302+ self.assertEqual(expected_command, self.buildmanager.commands[-1])
1303+ self.assertEqual(
1304+ self.buildmanager.iterate, self.buildmanager.iterators[-1])
1305+ self.assertFalse(self.builder.wasCalled("chrootFail"))
1306+
1307+ @defer.inlineCallbacks
1308+ def test_iterate(self):
1309+ # This sha would change as it includes file attributes in the
1310+ # tar file. Fix it so we can test against a known value.
1311+ sha_mock = self.useFixture(
1312+ MockPatch('lpbuildd.oci.OCIBuildManager._calculateLayerSha'))
1313+ sha_mock.mock.return_value = "testsha"
1314+ # The build manager iterates a normal build from start to finish.
1315+ args = {
1316+ "git_repository": "https://git.launchpad.dev/~example/+git/snap",
1317+ "git_path": "master",
1318+ }
1319+ expected_options = [
1320+ "--git-repository", "https://git.launchpad.dev/~example/+git/snap",
1321+ "--git-path", "master",
1322+ ]
1323+ yield self.startBuild(args, expected_options)
1324+
1325+ log_path = os.path.join(self.buildmanager._cachepath, "buildlog")
1326+ with open(log_path, "w") as log:
1327+ log.write("I am a build log.")
1328+
1329+ self.buildmanager.backend.run.result = MockOCITarSave()
1330+
1331+ self.buildmanager.backend.add_file(
1332+ '/var/lib/docker/image/'
1333+ 'vfs/distribution/v2metadata-by-diffid/sha256/diff1',
1334+ b"""[{"Digest": "test_digest", "SourceRepository": "test"}]"""
1335+ )
1336+
1337+ # After building the package, reap processes.
1338+ yield self.buildmanager.iterate(0)
1339+ expected_command = [
1340+ "sharepath/bin/in-target", "in-target", "scan-for-processes",
1341+ "--backend=lxd", "--series=xenial", "--arch=i386", self.buildid,
1342+ ]
1343+ self.assertEqual(OCIBuildState.BUILD_OCI, self.getState())
1344+ self.assertEqual(expected_command, self.buildmanager.commands[-1])
1345+ self.assertNotEqual(
1346+ self.buildmanager.iterate, self.buildmanager.iterators[-1])
1347+ self.assertFalse(self.builder.wasCalled("buildFail"))
1348+ expected_files = [
1349+ 'manifest.json',
1350+ 'layer-1.tar.gz',
1351+ 'layer-2.tar.gz',
1352+ 'digests.json',
1353+ 'config.json',
1354+ ]
1355+ for expected in expected_files:
1356+ self.assertThat(self.builder.waitingfiles, Contains(expected))
1357+
1358+ cache_path = self.builder.cachePath(
1359+ self.builder.waitingfiles['digests.json'])
1360+ with open(cache_path, "rb") as f:
1361+ digests_contents = f.read()
1362+ digests_expected = [{
1363+ "sha256:diff1": {
1364+ "source": "test",
1365+ "digest": "test_digest",
1366+ "layer_id": "layer-1"
1367+ },
1368+ "sha256:diff2": {
1369+ "source": "",
1370+ "digest": "testsha",
1371+ "layer_id": "layer-2"
1372+ }
1373+ }]
1374+ self.assertEqual(digests_contents, json.dumps(digests_expected))
1375+ # Control returns to the DebianBuildManager in the UMOUNT state.
1376+ self.buildmanager.iterateReap(self.getState(), 0)
1377+ expected_command = [
1378+ "sharepath/bin/in-target", "in-target", "umount-chroot",
1379+ "--backend=lxd", "--series=xenial", "--arch=i386", self.buildid,
1380+ ]
1381+ self.assertEqual(OCIBuildState.UMOUNT, self.getState())
1382+ self.assertEqual(expected_command, self.buildmanager.commands[-1])
1383+ self.assertEqual(
1384+ self.buildmanager.iterate, self.buildmanager.iterators[-1])
1385+ self.assertFalse(self.builder.wasCalled("buildFail"))
1386+
1387+ @defer.inlineCallbacks
1388+ def test_iterate_with_file(self):
1389+ # This sha would change as it includes file attributes in the
1390+ # tar file. Fix it so we can test against a known value.
1391+ sha_mock = self.useFixture(
1392+ MockPatch('lpbuildd.oci.OCIBuildManager._calculateLayerSha'))
1393+ sha_mock.mock.return_value = "testsha"
1394+ # The build manager iterates a build that specifies a non-default
1395+ # Dockerfile location from start to finish.
1396+ args = {
1397+ "git_repository": "https://git.launchpad.dev/~example/+git/snap",
1398+ "git_path": "master",
1399+ "build_file": "build-aux/Dockerfile",
1400+ }
1401+ expected_options = [
1402+ "--git-repository", "https://git.launchpad.dev/~example/+git/snap",
1403+ "--git-path", "master",
1404+ "--build-file", "build-aux/Dockerfile",
1405+ ]
1406+ yield self.startBuild(args, expected_options)
1407+
1408+ log_path = os.path.join(self.buildmanager._cachepath, "buildlog")
1409+ with open(log_path, "w") as log:
1410+ log.write("I am a build log.")
1411+
1412+ self.buildmanager.backend.run.result = MockOCITarSave()
1413+
1414+ self.buildmanager.backend.add_file(
1415+ '/var/lib/docker/image/'
1416+ 'vfs/distribution/v2metadata-by-diffid/sha256/diff1',
1417+ b"""[{"Digest": "test_digest", "SourceRepository": "test"}]"""
1418+ )
1419+
1420+ # After building the package, reap processes.
1421+ yield self.buildmanager.iterate(0)
1422+ expected_command = [
1423+ "sharepath/bin/in-target", "in-target", "scan-for-processes",
1424+ "--backend=lxd", "--series=xenial", "--arch=i386", self.buildid,
1425+ ]
1426+ self.assertEqual(OCIBuildState.BUILD_OCI, self.getState())
1427+ self.assertEqual(expected_command, self.buildmanager.commands[-1])
1428+ self.assertNotEqual(
1429+ self.buildmanager.iterate, self.buildmanager.iterators[-1])
1430+ self.assertFalse(self.builder.wasCalled("buildFail"))
1431+ expected_files = [
1432+ 'manifest.json',
1433+ 'layer-1.tar.gz',
1434+ 'layer-2.tar.gz',
1435+ 'digests.json',
1436+ 'config.json',
1437+ ]
1438+ for expected in expected_files:
1439+ self.assertThat(self.builder.waitingfiles, Contains(expected))
1440+
1441+ cache_path = self.builder.cachePath(
1442+ self.builder.waitingfiles['digests.json'])
1443+ with open(cache_path, "rb") as f:
1444+ digests_contents = f.read()
1445+ digests_expected = [{
1446+ "sha256:diff1": {
1447+ "source": "test",
1448+ "digest": "test_digest",
1449+ "layer_id": "layer-1"
1450+ },
1451+ "sha256:diff2": {
1452+ "source": "",
1453+ "digest": "testsha",
1454+ "layer_id": "layer-2"
1455+ }
1456+ }]
1457+ self.assertEqual(digests_contents, json.dumps(digests_expected))
1458+
1459+ # Control returns to the DebianBuildManager in the UMOUNT state.
1460+ self.buildmanager.iterateReap(self.getState(), 0)
1461+ expected_command = [
1462+ "sharepath/bin/in-target", "in-target", "umount-chroot",
1463+ "--backend=lxd", "--series=xenial", "--arch=i386", self.buildid,
1464+ ]
1465+ self.assertEqual(OCIBuildState.UMOUNT, self.getState())
1466+ self.assertEqual(expected_command, self.buildmanager.commands[-1])
1467+ self.assertEqual(
1468+ self.buildmanager.iterate, self.buildmanager.iterators[-1])
1469+ self.assertFalse(self.builder.wasCalled("buildFail"))
1470+
1471+ @defer.inlineCallbacks
1472+ def test_iterate_snap_store_proxy(self):
1473+ # The build manager can be told to use a snap store proxy.
1474+ self.builder._config.set(
1475+ "proxy", "snapstore", "http://snap-store-proxy.example/")
1476+ expected_options = [
1477+ "--snap-store-proxy-url", "http://snap-store-proxy.example/"]
1478+ yield self.startBuild(options=expected_options)

Subscribers

People subscribed via source and target branches