Merge lp:~twom/launchpad-buildd/initial-docker-build-support into lp:launchpad-buildd

[Tom Wardill]

Add initial docker build support

Add a builder for docker, creating an image following a supplied Dockerfile.
Save the image, extract it and then tar each component layer individually for returning/caching in launchpad.

[Colin Watson (cjwatson)]

review: Needs Fixing

[lp:~twom/launchpad-buildd/initial-docker-build-support]

393. By Tom Wardill on 2019-07-11

Use shell_escape, compress output images

394. By Tom Wardill on 2019-07-11

Collect .tar.gz

395. By Tom Wardill on 2019-07-12

Use a python script to save docker image to layers

396. By Tom Wardill on 2019-07-15

Move image extraction to gatherResults

397. By Tom Wardill on 2019-07-16

Refactor joint proxy handling

398. By Tom Wardill on 2019-07-16

Fix changelog diff noise

399. By Tom Wardill on 2019-07-24

Do more preprocessing, create digests file

400. By Tom Wardill on 2019-08-06

Include all diffs in digests and waiting files

401. By Tom Wardill on 2019-08-08

Add layer_id to digests now we're using every layer

402. By <email address hidden> on 2020-01-29

Use docker from apt, rather than from snap

403. By <email address hidden> on 2020-01-29

Rename Docker to OCI

404. By <email address hidden> on 2020-01-30

Merge upstream

[Colin Watson (cjwatson)]

review: Approve

[lp:~twom/launchpad-buildd/initial-docker-build-support]

405. By <email address hidden> on 2020-01-30

Remove conflict marker

406. By <email address hidden> on 2020-02-05

Use proxy, extract sha from right location

407. By <email address hidden> on 2020-02-05

Refactor proxy handling into better named mixin

408. By <email address hidden> on 2020-02-05

Refactor out adding docker proxy

409. By <email address hidden> on 2020-02-06

Include snap build proxy code

410. By <email address hidden> on 2020-02-06

Move to an on-the-fly tarball creation

[Tom Wardill (twom)]

While looking at the launchpad buildbehaviour side of this, realised MP doesn't support build file location.

It should do that.

review: Needs Fixing

[lp:~twom/launchpad-buildd/initial-docker-build-support]

411. By <email address hidden> on 2020-02-26

Argument is build_file, not file

[Colin Watson (cjwatson)]

review: Approve

[Tom Wardill (twom)]

Tested with latest buildbehaviour branch, should now be functional.

review: Approve

=== modified file 'debian/changelog'
--- debian/changelog	2020-01-06 15:03:55 +0000
+++ debian/changelog	2020-02-26 10:52:47 +0000
@@ -1,3 +1,9 @@
+launchpad-buildd (187) UNRELEASED; urgency=medium
+
+  * Prototype Docker image building support.
+
+  -- Colin Watson <[e-mail]>  Wed, 05 Jun 2019 15:06:54 +0100
+
 launchpad-buildd (186) xenial; urgency=medium
 
   * Fix sbuildrc compatibility with xenial's sbuild.
@@ -32,7 +38,7 @@
 
   [ Michael Hudson-Doyle ]
   * Do not make assumptions about what device major number the device mapper
-    is using. (LP: #1852518) 
+    is using. (LP: #1852518)
 
  -- Colin Watson <[e-mail]>  Tue, 26 Nov 2019 12:22:37 +0000
 
@@ -801,7 +807,7 @@
     memory at once (LP: #1227086).
 
   [ Adam Conrad ]
-  * Tidy up log formatting of the "Already reaped..." message. 
+  * Tidy up log formatting of the "Already reaped..." message.
 
  -- Colin Watson <[e-mail]>  Fri, 27 Sep 2013 13:08:59 +0100
 
@@ -986,7 +992,7 @@
 launchpad-buildd (98) hardy; urgency=low
 
   * Add launchpad-buildd dependency on python-apt, as an accomodation for it
-    being only a Recommends but actually required by python-debian.  
+    being only a Recommends but actually required by python-debian.
     LP: #890834
 
  -- Martin Pool <[e-mail]>  Wed, 16 Nov 2011 10:28:48 +1100
@@ -1040,7 +1046,7 @@
 
 launchpad-buildd (90) hardy; urgency=low
 
-  * debhelper is a Build-Depends because it is needed to run 'clean'. 
+  * debhelper is a Build-Depends because it is needed to run 'clean'.
   * python-lpbuildd conflicts with launchpad-buildd << 88.
   * Add and adjust build-arch, binary-arch, build-indep to match policy.
   * Complies with stardards version 3.9.2.
 
=== modified file 'lpbuildd/buildd-slave.tac'
--- lpbuildd/buildd-slave.tac	2019-02-12 10:35:12 +0000
+++ lpbuildd/buildd-slave.tac	2020-02-26 10:52:47 +0000
@@ -23,6 +23,7 @@
 
 from lpbuildd.binarypackage import BinaryPackageBuildManager
 from lpbuildd.builder import XMLRPCBuilder
+from lpbuildd.oci import OCIBuildManager
 from lpbuildd.livefs import LiveFilesystemBuildManager
 from lpbuildd.log import RotatableFileLogObserver
 from lpbuildd.snap import SnapBuildManager
@@ -45,6 +46,7 @@
     TranslationTemplatesBuildManager, 'translation-templates')
 builder.registerManager(LiveFilesystemBuildManager, "livefs")
 builder.registerManager(SnapBuildManager, "snap")
+builder.registerManager(OCIBuildManager, "oci")
 
 application = service.Application('Builder')
 application.addComponent(
 
=== added file 'lpbuildd/oci.py'
--- lpbuildd/oci.py	1970-01-01 00:00:00 +0000
+++ lpbuildd/oci.py	2020-02-26 10:52:47 +0000
@@ -0,0 +1,222 @@
+# Copyright 2019 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+from __future__ import print_function
+
+__metaclass__ = type
+
+import hashlib
+import json
+import os
+import tarfile
+import tempfile
+
+from six.moves.configparser import (
+    NoOptionError,
+    NoSectionError,
+    )
+
+from lpbuildd.debian import (
+    DebianBuildManager,
+    DebianBuildState,
+    )
+from lpbuildd.snap import SnapBuildProxyMixin
+
+
+RETCODE_SUCCESS = 0
+RETCODE_FAILURE_INSTALL = 200
+RETCODE_FAILURE_BUILD = 201
+
+
+class OCIBuildState(DebianBuildState):
+    BUILD_OCI = "BUILD_OCI"
+
+
+class OCIBuildManager(SnapBuildProxyMixin, DebianBuildManager):
+    """Build an OCI Image."""
+
+    backend_name = "lxd"
+    initial_build_state = OCIBuildState.BUILD_OCI
+
+    @property
+    def needs_sanitized_logs(self):
+        return True
+
+    def initiate(self, files, chroot, extra_args):
+        """Initiate a build with a given set of files and chroot."""
+        self.name = extra_args["name"]
+        self.branch = extra_args.get("branch")
+        self.git_repository = extra_args.get("git_repository")
+        self.git_path = extra_args.get("git_path")
+        self.build_file = extra_args.get("build_file")
+        self.proxy_url = extra_args.get("proxy_url")
+        self.revocation_endpoint = extra_args.get("revocation_endpoint")
+        self.proxy_service = None
+
+        super(OCIBuildManager, self).initiate(files, chroot, extra_args)
+
+    def doRunBuild(self):
+        """Run the process to build the snap."""
+        args = []
+        args.extend(self.startProxy())
+        if self.revocation_endpoint:
+            args.extend(["--revocation-endpoint", self.revocation_endpoint])
+        if self.branch is not None:
+            args.extend(["--branch", self.branch])
+        if self.git_repository is not None:
+            args.extend(["--git-repository", self.git_repository])
+        if self.git_path is not None:
+            args.extend(["--git-path", self.git_path])
+        if self.build_file is not None:
+            args.extend(["--build-file", self.build_file])
+        try:
+            snap_store_proxy_url = self._builder._config.get(
+                "proxy", "snapstore")
+            args.extend(["--snap-store-proxy-url", snap_store_proxy_url])
+        except (NoSectionError, NoOptionError):
+            pass
+        args.append(self.name)
+        self.runTargetSubProcess("build-oci", *args)
+
+    def iterate_BUILD_OCI(self, retcode):
+        """Finished building the OCI image."""
+        self.stopProxy()
+        self.revokeProxyToken()
+        if retcode == RETCODE_SUCCESS:
+            print("Returning build status: OK")
+            return self.deferGatherResults()
+        elif (retcode >= RETCODE_FAILURE_INSTALL and
+              retcode <= RETCODE_FAILURE_BUILD):
+            if not self.alreadyfailed:
+                self._builder.buildFail()
+                print("Returning build status: Build failed.")
+            self.alreadyfailed = True
+        else:
+            if not self.alreadyfailed:
+                self._builder.builderFail()
+                print("Returning build status: Builder failed.")
+            self.alreadyfailed = True
+        self.doReapProcesses(self._state)
+
+    def iterateReap_BUILD_OCI(self, retcode):
+        """Finished reaping after building the OCI image."""
+        self._state = DebianBuildState.UMOUNT
+        self.doUnmounting()
+
+    def _calculateLayerSha(self, layer_path):
+        with open(layer_path, 'rb') as layer_tar:
+            sha256_hash = hashlib.sha256()
+            for byte_block in iter(lambda: layer_tar.read(4096), b""):
+                sha256_hash.update(byte_block)
+            digest = sha256_hash.hexdigest()
+            return digest
+
+    def _gatherManifestSection(self, section, extract_path, sha_directory):
+        config_file_path = os.path.join(extract_path, section["Config"])
+        self._builder.addWaitingFile(config_file_path)
+        with open(config_file_path, 'r') as config_fp:
+            config = json.load(config_fp)
+        diff_ids = config["rootfs"]["diff_ids"]
+        digest_diff_map = {}
+        for diff_id, layer_id in zip(diff_ids, section['Layers']):
+            layer_id = layer_id.split('/')[0]
+            diff_file = os.path.join(sha_directory, diff_id.split(':')[1])
+            layer_path = os.path.join(
+                extract_path, "{}.tar.gz".format(layer_id))
+            self._builder.addWaitingFile(layer_path)
+            # If we have a mapping between diff and existing digest,
+            # this means this layer has been pulled from a remote.
+            # We should maintain the same digest to achieve layer reuse
+            if os.path.exists(diff_file):
+                with open(diff_file, 'r') as diff_fp:
+                    diff = json.load(diff_fp)
+                    # We should be able to just take the first occurence,
+                    # as that will be the 'most parent' image
+                    digest = diff[0]["Digest"]
+                    source = diff[0]["SourceRepository"]
+            # If the layer has been build locally, we need to generate the
+            # digest and then set the source to empty
+            else:
+                source = ""
+                digest = self._calculateLayerSha(layer_path)
+            digest_diff_map[diff_id] = {
+                "digest": digest,
+                "source": source,
+                "layer_id": layer_id
+            }
+
+        return digest_diff_map
+
+    def gatherResults(self):
+        """Gather the results of the build and add them to the file cache."""
+        extract_path = tempfile.mkdtemp(prefix=self.name)
+        proc = self.backend.run(
+            ['docker', 'save', self.name],
+            get_output=True, universal_newlines=False, return_process=True)
+        try:
+            tar = tarfile.open(fileobj=proc.stdout, mode="r|")
+        except Exception as e:
+            print(e)
+
+        current_dir = ''
+        directory_tar = None
+        try:
+            # The tarfile is a stream and must be processed in order
+            for file in tar:
+                # Directories are just nodes, you can't extract the children
+                # directly, so keep track of what dir we're in.
+                if file.isdir():
+                    current_dir = file.name
+                    if directory_tar:
+                        # Close the old directory if we have one
+                        directory_tar.close()
+                    # We're going to add the layer.tar to a gzip
+                    directory_tar = tarfile.open(
+                        os.path.join(
+                            extract_path, '{}.tar.gz'.format(file.name)),
+                        'w|gz')
+                if current_dir and file.name.endswith('layer.tar'):
+                    # This is the actual layer data, we want to add it to
+                    # the directory gzip
+                    file.name = file.name.split('/')[1]
+                    directory_tar.addfile(file, tar.extractfile(file))
+                elif current_dir and file.name.startswith(current_dir):
+                    # Other files that are in the layer directories,
+                    # we don't care about
+                    continue
+                else:
+                    # If it's not in a directory, we need that
+                    tar.extract(file, extract_path)
+        except Exception as e:
+            print(e)
+
+        # We need these mapping files
+        sha_directory = tempfile.mkdtemp()
+        sha_path = ('/var/lib/docker/image/'
+                    'vfs/distribution/v2metadata-by-diffid/sha256/')
+        sha_files = [x for x in self.backend.listdir(sha_path)
+                     if not x.startswith('.')]
+        for file in sha_files:
+            self.backend.copy_out(
+                os.path.join(sha_path, file),
+                os.path.join(sha_directory, file)
+            )
+
+        # Parse the manifest for the other files we need
+        manifest_path = os.path.join(extract_path, 'manifest.json')
+        self._builder.addWaitingFile(manifest_path)
+        with open(manifest_path) as manifest_fp:
+            manifest = json.load(manifest_fp)
+
+        digest_maps = []
+        try:
+            for section in manifest:
+                digest_maps.append(
+                    self._gatherManifestSection(section, extract_path,
+                                                sha_directory))
+            digest_map_file = os.path.join(extract_path, 'digests.json')
+            with open(digest_map_file, 'w') as digest_map_fp:
+                json.dump(digest_maps, digest_map_fp)
+            self._builder.addWaitingFile(digest_map_file)
+        except Exception as e:
+            print(e)
 
=== modified file 'lpbuildd/snap.py'
--- lpbuildd/snap.py	2019-10-30 12:31:53 +0000
+++ lpbuildd/snap.py	2020-02-26 10:52:47 +0000
@@ -239,35 +239,7 @@
     BUILD_SNAP = "BUILD_SNAP"
 
 
-class SnapBuildManager(DebianBuildManager):
-    """Build a snap."""
-
-    backend_name = "lxd"
-    initial_build_state = SnapBuildState.BUILD_SNAP
-
-    @property
-    def needs_sanitized_logs(self):
-        return True
-
-    def initiate(self, files, chroot, extra_args):
-        """Initiate a build with a given set of files and chroot."""
-        self.name = extra_args["name"]
-        self.channels = extra_args.get("channels", {})
-        self.build_request_id = extra_args.get("build_request_id")
-        self.build_request_timestamp = extra_args.get(
-            "build_request_timestamp")
-        self.build_url = extra_args.get("build_url")
-        self.branch = extra_args.get("branch")
-        self.git_repository = extra_args.get("git_repository")
-        self.git_path = extra_args.get("git_path")
-        self.proxy_url = extra_args.get("proxy_url")
-        self.revocation_endpoint = extra_args.get("revocation_endpoint")
-        self.build_source_tarball = extra_args.get(
-            "build_source_tarball", False)
-        self.private = extra_args.get("private", False)
-        self.proxy_service = None
-
-        super(SnapBuildManager, self).initiate(files, chroot, extra_args)
+class SnapBuildProxyMixin():
 
     def startProxy(self):
         """Start the local snap proxy, if necessary."""
@@ -308,6 +280,37 @@
             self._builder.log(
                 "Unable to revoke token for %s: %s" % (url.username, e))
 
+
+class SnapBuildManager(SnapBuildProxyMixin, DebianBuildManager):
+    """Build a snap."""
+
+    backend_name = "lxd"
+    initial_build_state = SnapBuildState.BUILD_SNAP
+
+    @property
+    def needs_sanitized_logs(self):
+        return True
+
+    def initiate(self, files, chroot, extra_args):
+        """Initiate a build with a given set of files and chroot."""
+        self.name = extra_args["name"]
+        self.channels = extra_args.get("channels", {})
+        self.build_request_id = extra_args.get("build_request_id")
+        self.build_request_timestamp = extra_args.get(
+            "build_request_timestamp")
+        self.build_url = extra_args.get("build_url")
+        self.branch = extra_args.get("branch")
+        self.git_repository = extra_args.get("git_repository")
+        self.git_path = extra_args.get("git_path")
+        self.proxy_url = extra_args.get("proxy_url")
+        self.revocation_endpoint = extra_args.get("revocation_endpoint")
+        self.build_source_tarball = extra_args.get(
+            "build_source_tarball", False)
+        self.private = extra_args.get("private", False)
+        self.proxy_service = None
+
+        super(SnapBuildManager, self).initiate(files, chroot, extra_args)
+
     def status(self):
         status_path = get_build_path(self.home, self._buildid, "status")
         try:
 
=== added file 'lpbuildd/target/build_oci.py'
--- lpbuildd/target/build_oci.py	1970-01-01 00:00:00 +0000
+++ lpbuildd/target/build_oci.py	2020-02-26 10:52:47 +0000
@@ -0,0 +1,126 @@
+# Copyright 2019 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+from __future__ import print_function
+
+__metaclass__ = type
+
+from collections import OrderedDict
+import logging
+import os.path
+import sys
+import tempfile
+from textwrap import dedent
+
+from lpbuildd.target.operation import Operation
+from lpbuildd.target.snapbuildproxy import SnapBuildProxyOperationMixin
+from lpbuildd.target.snapstore import SnapStoreOperationMixin
+from lpbuildd.target.vcs import VCSOperationMixin
+
+
+RETCODE_FAILURE_INSTALL = 200
+RETCODE_FAILURE_BUILD = 201
+
+
+logger = logging.getLogger(__name__)
+
+
+class BuildOCI(SnapBuildProxyOperationMixin, VCSOperationMixin,
+               SnapStoreOperationMixin, Operation):
+
+    description = "Build an OCI image."
+
+    @classmethod
+    def add_arguments(cls, parser):
+        super(BuildOCI, cls).add_arguments(parser)
+        parser.add_argument(
+            "--build-file", help="path to Dockerfile in branch")
+        parser.add_argument("name", help="name of snap to build")
+
+    def __init__(self, args, parser):
+        super(BuildOCI, self).__init__(args, parser)
+        self.bin = os.path.dirname(sys.argv[0])
+
+    def _add_docker_engine_proxy_settings(self):
+        """Add systemd file for docker proxy settings."""
+        # Create containing directory for systemd overrides
+        self.backend.run(
+            ["mkdir", "-p", "/etc/systemd/system/docker.service.d"])
+        # we need both http_proxy and https_proxy. The contents of the files
+        # are otherwise identical
+        for setting in ['http_proxy', 'https_proxy']:
+            contents = dedent("""[Service]
+                Environment="{}={}"
+                """.format(setting.upper(), self.args.proxy_url))
+            file_path = "/etc/systemd/system/docker.service.d/{}.conf".format(
+                setting)
+            with tempfile.NamedTemporaryFile(mode="w+") as systemd_file:
+                systemd_file.write(contents)
+                systemd_file.flush()
+                self.backend.copy_in(systemd_file.name, file_path)
+
+    def run_build_command(self, args, env=None, **kwargs):
+        """Run a build command in the target.
+
+        :param args: the command and arguments to run.
+        :param env: dictionary of additional environment variables to set.
+        :param kwargs: any other keyword arguments to pass to Backend.run.
+        """
+        full_env = OrderedDict()
+        full_env["LANG"] = "C.UTF-8"
+        full_env["SHELL"] = "/bin/sh"
+        if env:
+            full_env.update(env)
+        return self.backend.run(args, env=full_env, **kwargs)
+
+    def install(self):
+        logger.info("Running install phase...")
+        deps = []
+        if self.args.proxy_url:
+            deps.extend(self.proxy_deps)
+            self.install_git_proxy()
+            # Add any proxy settings that are needed
+            self._add_docker_engine_proxy_settings()
+        deps.extend(self.vcs_deps)
+        deps.extend(["docker.io"])
+        self.backend.run(["apt-get", "-y", "install"] + deps)
+        if self.args.backend in ("lxd", "fake"):
+            self.snap_store_set_proxy()
+        self.backend.run(["systemctl", "restart", "docker"])
+        # The docker snap can't see /build, so we have to do our work under
+        # /home/buildd instead.  Make sure it exists.
+        self.backend.run(["mkdir", "-p", "/home/buildd"])
+
+    def repo(self):
+        """Collect git or bzr branch."""
+        logger.info("Running repo phase...")
+        env = self.build_proxy_environment(proxy_url=self.args.proxy_url)
+        self.vcs_fetch(self.args.name, cwd="/home/buildd", env=env)
+
+    def build(self):
+        logger.info("Running build phase...")
+        args = ["docker", "build", "--no-cache"]
+        if self.args.proxy_url:
+            for var in ("http_proxy", "https_proxy"):
+                args.extend(
+                    ["--build-arg", "{}={}".format(var, self.args.proxy_url)])
+        args.extend(["--tag", self.args.name])
+        if self.args.build_file is not None:
+            args.extend(["--file", self.args.build_file])
+        buildd_path = os.path.join("/home/buildd", self.args.name)
+        args.append(buildd_path)
+        self.run_build_command(args)
+
+    def run(self):
+        try:
+            self.install()
+        except Exception:
+            logger.exception('Install failed')
+            return RETCODE_FAILURE_INSTALL
+        try:
+            self.repo()
+            self.build()
+        except Exception:
+            logger.exception('Build failed')
+            return RETCODE_FAILURE_BUILD
+        return 0
 
=== modified file 'lpbuildd/target/build_snap.py'
--- lpbuildd/target/build_snap.py	2019-10-30 12:31:53 +0000
+++ lpbuildd/target/build_snap.py	2020-02-26 10:52:47 +0000
@@ -17,6 +17,7 @@
 from six.moves.urllib.parse import urlparse
 
 from lpbuildd.target.operation import Operation
+from lpbuildd.target.snapbuildproxy import SnapBuildProxyOperationMixin
 from lpbuildd.target.snapstore import SnapStoreOperationMixin
 from lpbuildd.target.vcs import VCSOperationMixin
 
@@ -46,7 +47,8 @@
         getattr(namespace, self.dest)[snap] = channel
 
 
-class BuildSnap(VCSOperationMixin, SnapStoreOperationMixin, Operation):
+class BuildSnap(SnapBuildProxyOperationMixin, VCSOperationMixin,
+                SnapStoreOperationMixin, Operation):
 
     description = "Build a snap."
 
@@ -69,10 +71,6 @@
             help="RFC3339 timestamp of the Launchpad build request")
         parser.add_argument(
             "--build-url", help="URL of this build on Launchpad")
-        parser.add_argument("--proxy-url", help="builder proxy url")
-        parser.add_argument(
-            "--revocation-endpoint",
-            help="builder proxy token revocation endpoint")
         parser.add_argument(
             "--build-source-tarball", default=False, action="store_true",
             help=(
@@ -137,6 +135,9 @@
     def install(self):
         logger.info("Running install phase...")
         deps = []
+        if self.args.proxy_url:
+            deps.extend(self.proxy_deps)
+            self.install_git_proxy()
         if self.args.backend == "lxd":
             # udev is installed explicitly to work around
             # https://bugs.launchpad.net/snapd/+bug/1731519.
@@ -144,8 +145,6 @@
                 if self.backend.is_package_available(dep):
                     deps.append(dep)
         deps.extend(self.vcs_deps)
-        if self.args.proxy_url:
-            deps.extend(["python3", "socat"])
         if "snapcraft" in self.args.channels:
             # snapcraft requires sudo in lots of places, but can't depend on
             # it when installed as a snap.
@@ -167,19 +166,12 @@
                  "--channel=%s" % self.args.channels["snapcraft"],
                  "snapcraft"])
         if self.args.proxy_url:
-            self.backend.copy_in(
-                os.path.join(self.bin, "snap-git-proxy"),
-                "/usr/local/bin/snap-git-proxy")
             self.install_svn_servers()
 
     def repo(self):
         """Collect git or bzr branch."""
         logger.info("Running repo phase...")
-        env = OrderedDict()
-        if self.args.proxy_url:
-            env["http_proxy"] = self.args.proxy_url
-            env["https_proxy"] = self.args.proxy_url
-            env["GIT_PROXY_COMMAND"] = "/usr/local/bin/snap-git-proxy"
+        env = self.build_proxy_environment(proxy_url=self.args.proxy_url)
         self.vcs_fetch(self.args.name, cwd="/build", env=env)
         status = {}
         if self.args.branch is not None:
 
=== modified file 'lpbuildd/target/cli.py'
--- lpbuildd/target/cli.py	2017-09-08 15:57:18 +0000
+++ lpbuildd/target/cli.py	2020-02-26 10:52:47 +0000
@@ -14,6 +14,7 @@
     OverrideSourcesList,
     Update,
     )
+from lpbuildd.target.build_oci import BuildOCI
 from lpbuildd.target.build_livefs import BuildLiveFS
 from lpbuildd.target.build_snap import BuildSnap
 from lpbuildd.target.generate_translation_templates import (
@@ -49,6 +50,7 @@
 
 operations = {
     "add-trusted-keys": AddTrustedKeys,
+    "build-oci": BuildOCI,
     "buildlivefs": BuildLiveFS,
     "buildsnap": BuildSnap,
     "generate-translation-templates": GenerateTranslationTemplates,
 
=== modified file 'lpbuildd/target/lxd.py'
--- lpbuildd/target/lxd.py	2019-12-09 11:17:14 +0000
+++ lpbuildd/target/lxd.py	2020-02-26 10:52:47 +0000
@@ -504,7 +504,8 @@
              "/etc/systemd/system/snapd.refresh.timer"])
 
     def run(self, args, cwd=None, env=None, input_text=None, get_output=False,
-            echo=False, **kwargs):
+            echo=False, return_process=False, universal_newlines=True,
+            **kwargs):
         """See `Backend`."""
         env_params = []
         if env:
@@ -538,7 +539,10 @@
             if get_output:
                 kwargs["stdout"] = subprocess.PIPE
             proc = subprocess.Popen(
-                cmd, stdin=subprocess.PIPE, universal_newlines=True, **kwargs)
+                cmd, stdin=subprocess.PIPE,
+                universal_newlines=universal_newlines, **kwargs)
+            if return_process:
+                return proc
             output, _ = proc.communicate(input_text)
             if proc.returncode:
                 raise subprocess.CalledProcessError(proc.returncode, cmd)
 
=== added file 'lpbuildd/target/snapbuildproxy.py'
--- lpbuildd/target/snapbuildproxy.py	1970-01-01 00:00:00 +0000
+++ lpbuildd/target/snapbuildproxy.py	2020-02-26 10:52:47 +0000
@@ -0,0 +1,41 @@
+# Copyright 2019-2020 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+from __future__ import print_function
+
+__metaclass__ = type
+
+from collections import OrderedDict
+import os
+
+
+class SnapBuildProxyOperationMixin:
+    """Methods supporting the build time HTTP proxy for snap and OCI builds."""
+
+    @classmethod
+    def add_arguments(cls, parser):
+        super(SnapBuildProxyOperationMixin, cls).add_arguments(parser)
+        parser.add_argument("--proxy-url", help="builder proxy url")
+        parser.add_argument(
+            "--revocation-endpoint",
+            help="builder proxy token revocation endpoint")
+
+    @property
+    def proxy_deps(self):
+        return ["python3", "socat"]
+
+    def install_git_proxy(self):
+        self.backend.copy_in(
+            os.path.join(self.bin, "snap-git-proxy"),
+            "/usr/local/bin/snap-git-proxy")
+
+    def build_proxy_environment(self, proxy_url=None, env=None):
+        """Extend a command environment to include http proxy variables."""
+        full_env = OrderedDict()
+        if env:
+            full_env.update(env)
+        if proxy_url:
+            full_env["http_proxy"] = self.args.proxy_url
+            full_env["https_proxy"] = self.args.proxy_url
+            full_env["GIT_PROXY_COMMAND"] = "/usr/local/bin/snap-git-proxy"
+        return full_env
 
=== added file 'lpbuildd/target/tests/test_build_oci.py'
--- lpbuildd/target/tests/test_build_oci.py	1970-01-01 00:00:00 +0000
+++ lpbuildd/target/tests/test_build_oci.py	2020-02-26 10:52:47 +0000
@@ -0,0 +1,407 @@
+# Copyright 2019 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+__metaclass__ = type
+
+import os.path
+import stat
+import subprocess
+from textwrap import dedent
+
+from fixtures import (
+    FakeLogger,
+    TempDir,
+    )
+import responses
+from systemfixtures import FakeFilesystem
+from testtools import TestCase
+from testtools.matchers import (
+    AnyMatch,
+    Equals,
+    Is,
+    MatchesAll,
+    MatchesDict,
+    MatchesListwise,
+    )
+
+from lpbuildd.target.build_oci import (
+    RETCODE_FAILURE_BUILD,
+    RETCODE_FAILURE_INSTALL,
+    )
+from lpbuildd.target.cli import parse_args
+from lpbuildd.tests.fakebuilder import FakeMethod
+
+
+class RanCommand(MatchesListwise):
+
+    def __init__(self, args, echo=None, cwd=None, input_text=None,
+                 get_output=None, **env):
+        kwargs_matcher = {}
+        if echo is not None:
+            kwargs_matcher["echo"] = Is(echo)
+        if cwd:
+            kwargs_matcher["cwd"] = Equals(cwd)
+        if input_text:
+            kwargs_matcher["input_text"] = Equals(input_text)
+        if get_output is not None:
+            kwargs_matcher["get_output"] = Is(get_output)
+        if env:
+            kwargs_matcher["env"] = MatchesDict(
+                {key: Equals(value) for key, value in env.items()})
+        super(RanCommand, self).__init__(
+            [Equals((args,)), MatchesDict(kwargs_matcher)])
+
+
+class RanAptGet(RanCommand):
+
+    def __init__(self, *args):
+        super(RanAptGet, self).__init__(["apt-get", "-y"] + list(args))
+
+
+class RanSnap(RanCommand):
+
+    def __init__(self, *args, **kwargs):
+        super(RanSnap, self).__init__(["snap"] + list(args), **kwargs)
+
+
+class RanBuildCommand(RanCommand):
+
+    def __init__(self, args, **kwargs):
+        kwargs.setdefault("LANG", "C.UTF-8")
+        kwargs.setdefault("SHELL", "/bin/sh")
+        super(RanBuildCommand, self).__init__(args, **kwargs)
+
+
+class TestBuildOCI(TestCase):
+
+    def test_run_build_command_no_env(self):
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--branch", "lp:foo", "test-image",
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.run_build_command(["echo", "hello world"])
+        self.assertThat(build_oci.backend.run.calls, MatchesListwise([
+            RanBuildCommand(["echo", "hello world"]),
+            ]))
+
+    def test_run_build_command_env(self):
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--branch", "lp:foo", "test-image",
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.run_build_command(
+            ["echo", "hello world"], env={"FOO": "bar baz"})
+        self.assertThat(build_oci.backend.run.calls, MatchesListwise([
+            RanBuildCommand(["echo", "hello world"], FOO="bar baz"),
+            ]))
+
+    def test_install_bzr(self):
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--branch", "lp:foo", "test-image"
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.install()
+        self.assertThat(build_oci.backend.run.calls, MatchesListwise([
+            RanAptGet("install", "bzr", "docker.io"),
+            RanCommand(["systemctl", "restart", "docker"]),
+            RanCommand(["mkdir", "-p", "/home/buildd"]),
+            ]))
+
+    def test_install_git(self):
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--git-repository", "lp:foo", "test-image"
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.install()
+        self.assertThat(build_oci.backend.run.calls, MatchesListwise([
+            RanAptGet("install", "git", "docker.io"),
+            RanCommand(["systemctl", "restart", "docker"]),
+            RanCommand(["mkdir", "-p", "/home/buildd"]),
+            ]))
+
+    @responses.activate
+    def test_install_snap_store_proxy(self):
+        store_assertion = dedent("""\
+            type: store
+            store: store-id
+            url: http://snap-store-proxy.example
+
+            body
+            """)
+
+        def respond(request):
+            return 200, {"X-Assertion-Store-Id": "store-id"}, store_assertion
+
+        responses.add_callback(
+            "GET", "http://snap-store-proxy.example/v2/auth/store/assertions",
+            callback=respond)
+        args = [
+            "buildsnap",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--git-repository", "lp:foo",
+            "--snap-store-proxy-url", "http://snap-store-proxy.example/",
+            "test-snap",
+            ]
+        build_snap = parse_args(args=args).operation
+        build_snap.install()
+        self.assertThat(build_snap.backend.run.calls, MatchesListwise([
+            RanAptGet("install", "git", "snapcraft"),
+            RanCommand(
+                ["snap", "ack", "/dev/stdin"], input_text=store_assertion),
+            RanCommand(["snap", "set", "core", "proxy.store=store-id"]),
+            ]))
+
+    def test_install_proxy(self):
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--git-repository", "lp:foo",
+            "--proxy-url", "http://proxy.example:3128/",
+            "test-image",
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.bin = "/builderbin"
+        self.useFixture(FakeFilesystem()).add("/builderbin")
+        os.mkdir("/builderbin")
+        with open("/builderbin/snap-git-proxy", "w") as proxy_script:
+            proxy_script.write("proxy script\n")
+            os.fchmod(proxy_script.fileno(), 0o755)
+        build_oci.install()
+        self.assertThat(build_oci.backend.run.calls, MatchesListwise([
+            RanCommand(
+                ["mkdir", "-p", "/etc/systemd/system/docker.service.d"]),
+            RanAptGet("install", "python3", "socat", "git", "docker.io"),
+            RanCommand(["systemctl", "restart", "docker"]),
+            RanCommand(["mkdir", "-p", "/home/buildd"]),
+            ]))
+        self.assertEqual(
+            (b"proxy script\n", stat.S_IFREG | 0o755),
+            build_oci.backend.backend_fs["/usr/local/bin/snap-git-proxy"])
+
+    def test_repo_bzr(self):
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--branch", "lp:foo", "test-image",
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.backend.build_path = self.useFixture(TempDir()).path
+        build_oci.backend.run = FakeMethod()
+        build_oci.repo()
+        self.assertThat(build_oci.backend.run.calls, MatchesListwise([
+            RanBuildCommand(
+                ["bzr", "branch", "lp:foo", "test-image"], cwd="/home/buildd"),
+            ]))
+
+    def test_repo_git(self):
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--git-repository", "lp:foo", "test-image",
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.backend.build_path = self.useFixture(TempDir()).path
+        build_oci.backend.run = FakeMethod()
+        build_oci.repo()
+        self.assertThat(build_oci.backend.run.calls, MatchesListwise([
+            RanBuildCommand(
+                ["git", "clone", "lp:foo", "test-image"], cwd="/home/buildd"),
+            RanBuildCommand(
+                ["git", "submodule", "update", "--init", "--recursive"],
+                cwd="/home/buildd/test-image"),
+            ]))
+
+    def test_repo_git_with_path(self):
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--git-repository", "lp:foo", "--git-path", "next", "test-image",
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.backend.build_path = self.useFixture(TempDir()).path
+        build_oci.backend.run = FakeMethod()
+        build_oci.repo()
+        self.assertThat(build_oci.backend.run.calls, MatchesListwise([
+            RanBuildCommand(
+                ["git", "clone", "-b", "next", "lp:foo", "test-image"],
+                cwd="/home/buildd"),
+            RanBuildCommand(
+                ["git", "submodule", "update", "--init", "--recursive"],
+                cwd="/home/buildd/test-image"),
+            ]))
+
+    def test_repo_git_with_tag_path(self):
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--git-repository", "lp:foo", "--git-path", "refs/tags/1.0",
+            "test-image",
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.backend.build_path = self.useFixture(TempDir()).path
+        build_oci.backend.run = FakeMethod()
+        build_oci.repo()
+        self.assertThat(build_oci.backend.run.calls, MatchesListwise([
+            RanBuildCommand(
+                ["git", "clone", "-b", "1.0", "lp:foo", "test-image"],
+                cwd="/home/buildd"),
+            RanBuildCommand(
+                ["git", "submodule", "update", "--init", "--recursive"],
+                cwd="/home/buildd/test-image"),
+            ]))
+
+    def test_repo_proxy(self):
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--git-repository", "lp:foo",
+            "--proxy-url", "http://proxy.example:3128/",
+            "test-image",
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.backend.build_path = self.useFixture(TempDir()).path
+        build_oci.backend.run = FakeMethod()
+        build_oci.repo()
+        env = {
+            "http_proxy": "http://proxy.example:3128/",
+            "https_proxy": "http://proxy.example:3128/",
+            "GIT_PROXY_COMMAND": "/usr/local/bin/snap-git-proxy",
+            }
+        self.assertThat(build_oci.backend.run.calls, MatchesListwise([
+            RanBuildCommand(
+                ["git", "clone", "lp:foo", "test-image"],
+                cwd="/home/buildd", **env),
+            RanBuildCommand(
+                ["git", "submodule", "update", "--init", "--recursive"],
+                cwd="/home/buildd/test-image", **env),
+            ]))
+
+    def test_build(self):
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--branch", "lp:foo", "test-image",
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.backend.add_dir('/build/test-directory')
+        build_oci.build()
+        self.assertThat(build_oci.backend.run.calls, MatchesListwise([
+            RanBuildCommand(
+                ["docker", "build", "--no-cache", "--tag", "test-image",
+                 "/home/buildd/test-image"]),
+            ]))
+
+    def test_build_with_file(self):
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--branch", "lp:foo", "--build-file", "build-aux/Dockerfile",
+            "test-image",
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.backend.add_dir('/build/test-directory')
+        build_oci.build()
+        self.assertThat(build_oci.backend.run.calls, MatchesListwise([
+            RanBuildCommand(
+                ["docker", "build", "--no-cache", "--tag", "test-image",
+                 "--file", "build-aux/Dockerfile",
+                 "/home/buildd/test-image"]),
+            ]))
+
+    def test_build_proxy(self):
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--branch", "lp:foo", "--proxy-url", "http://proxy.example:3128/",
+            "test-image",
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.backend.add_dir('/build/test-directory')
+        build_oci.build()
+        self.assertThat(build_oci.backend.run.calls, MatchesListwise([
+            RanBuildCommand(
+                ["docker", "build", "--no-cache",
+                 "--build-arg", "http_proxy=http://proxy.example:3128/",
+                 "--build-arg", "https_proxy=http://proxy.example:3128/",
+                 "--tag", "test-image", "/home/buildd/test-image"]),
+            ]))
+
+    def test_run_succeeds(self):
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--branch", "lp:foo", "test-image",
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.backend.build_path = self.useFixture(TempDir()).path
+        build_oci.backend.run = FakeMethod()
+        self.assertEqual(0, build_oci.run())
+        self.assertThat(build_oci.backend.run.calls, MatchesAll(
+            AnyMatch(RanAptGet("install", "bzr", "docker.io")),
+            AnyMatch(RanBuildCommand(
+                ["bzr", "branch", "lp:foo", "test-image"],
+                cwd="/home/buildd")),
+            AnyMatch(RanBuildCommand(
+                ["docker", "build", "--no-cache", "--tag", "test-image",
+                 "/home/buildd/test-image"])),
+            ))
+
+    def test_run_install_fails(self):
+        class FailInstall(FakeMethod):
+            def __call__(self, run_args, *args, **kwargs):
+                super(FailInstall, self).__call__(run_args, *args, **kwargs)
+                if run_args[0] == "apt-get":
+                    raise subprocess.CalledProcessError(1, run_args)
+
+        self.useFixture(FakeLogger())
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--branch", "lp:foo", "test-image",
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.backend.run = FailInstall()
+        self.assertEqual(RETCODE_FAILURE_INSTALL, build_oci.run())
+
+    def test_run_repo_fails(self):
+        class FailRepo(FakeMethod):
+            def __call__(self, run_args, *args, **kwargs):
+                super(FailRepo, self).__call__(run_args, *args, **kwargs)
+                if run_args[:2] == ["bzr", "branch"]:
+                    raise subprocess.CalledProcessError(1, run_args)
+
+        self.useFixture(FakeLogger())
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--branch", "lp:foo", "test-image",
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.backend.run = FailRepo()
+        self.assertEqual(RETCODE_FAILURE_BUILD, build_oci.run())
+
+    def test_run_build_fails(self):
+        class FailBuild(FakeMethod):
+            def __call__(self, run_args, *args, **kwargs):
+                super(FailBuild, self).__call__(run_args, *args, **kwargs)
+                if run_args[0] == "docker":
+                    raise subprocess.CalledProcessError(1, run_args)
+
+        self.useFixture(FakeLogger())
+        args = [
+            "build-oci",
+            "--backend=fake", "--series=xenial", "--arch=amd64", "1",
+            "--branch", "lp:foo", "test-image",
+            ]
+        build_oci = parse_args(args=args).operation
+        build_oci.backend.build_path = self.useFixture(TempDir()).path
+        build_oci.backend.run = FailBuild()
+        self.assertEqual(RETCODE_FAILURE_BUILD, build_oci.run())
 
=== modified file 'lpbuildd/target/tests/test_build_snap.py'
--- lpbuildd/target/tests/test_build_snap.py	2019-06-05 14:10:20 +0000
+++ lpbuildd/target/tests/test_build_snap.py	2020-02-26 10:52:47 +0000
@@ -186,7 +186,7 @@
             os.fchmod(proxy_script.fileno(), 0o755)
         build_snap.install()
         self.assertThat(build_snap.backend.run.calls, MatchesListwise([
-            RanAptGet("install", "git", "python3", "socat", "snapcraft"),
+            RanAptGet("install", "python3", "socat", "git", "snapcraft"),
             RanCommand(["mkdir", "-p", "/root/.subversion"]),
             ]))
         self.assertEqual(
 
=== modified file 'lpbuildd/target/tests/test_lxd.py'
--- lpbuildd/target/tests/test_lxd.py	2019-12-09 11:17:14 +0000
+++ lpbuildd/target/tests/test_lxd.py	2020-02-26 10:52:47 +0000
@@ -276,7 +276,7 @@
             "lp-xenial-amd64", "lp-xenial-amd64")
 
     def assert_correct_profile(self, extra_raw_lxc_config=None,
-            driver_version="2.0"):
+                               driver_version="2.0"):
         if extra_raw_lxc_config is None:
             extra_raw_lxc_config = []
 
@@ -356,7 +356,7 @@
                     }
                 LXD("1", "xenial", "powerpc").create_profile()
                 self.assert_correct_profile(
-                        extra_raw_lxc_config=[("lxc.seccomp", ""),],
+                        extra_raw_lxc_config=[("lxc.seccomp", ""), ],
                         driver_version=driver_version or "3.0"
                         )
 
@@ -463,7 +463,8 @@
                      "b", "7", str(minor)]))
         if not with_dm0:
             expected_args.extend([
-                Equals(["sudo", "dmsetup", "create", "tmpdevice", "--notable"]),
+                Equals(
+                    ["sudo", "dmsetup", "create", "tmpdevice", "--notable"]),
                 Equals(["sudo", "dmsetup", "remove", "tmpdevice"]),
                 ])
         for minor in range(8):
 
=== added file 'lpbuildd/tests/oci_tarball.py'
--- lpbuildd/tests/oci_tarball.py	1970-01-01 00:00:00 +0000
+++ lpbuildd/tests/oci_tarball.py	2020-02-26 10:52:47 +0000
@@ -0,0 +1,60 @@
+import json
+import os
+import StringIO
+import tempfile
+import tarfile
+
+
+class OCITarball:
+    """Create a tarball for use in tests with OCI."""
+
+    def _makeFile(self, contents, name):
+        json_contents = json.dumps(contents)
+        tarinfo = tarfile.TarInfo(name)
+        tarinfo.size = len(json_contents)
+        return tarinfo, StringIO.StringIO(json_contents)
+
+    @property
+    def config(self):
+        return self._makeFile(
+            {"rootfs": {"diff_ids": ["sha256:diff1", "sha256:diff2"]}},
+            'config.json')
+
+    @property
+    def manifest(self):
+        return self._makeFile(
+            [{"Config": "config.json",
+              "Layers": ["layer-1/layer.tar", "layer-2/layer.tar"]}],
+            'manifest.json')
+
+    @property
+    def repositories(self):
+        return self._makeFile([], 'repositories')
+
+    def layer_file(self, directory, layer_name):
+        contents = "{}-contents".format(layer_name)
+        tarinfo = tarfile.TarInfo(contents)
+        tarinfo.size = len(contents)
+        layer_contents = StringIO.StringIO(contents)
+        layer_tar_path = os.path.join(
+            directory, '{}.tar.gz'.format(layer_name))
+        layer_tar = tarfile.open(layer_tar_path, 'w:gz')
+        layer_tar.addfile(tarinfo, layer_contents)
+        layer_tar.close()
+        return layer_tar_path
+
+    def build_tar_file(self):
+        tar_directory = tempfile.mkdtemp()
+        tar_path = os.path.join(tar_directory, 'test-oci-image.tar')
+        tar = tarfile.open(tar_path, 'w')
+        tar.addfile(*self.config)
+        tar.addfile(*self.manifest)
+        tar.addfile(*self.repositories)
+
+        for layer_name in ['layer-1', 'layer-2']:
+            layer = self.layer_file(tar_directory, layer_name)
+            tar.add(layer, arcname='{}.tar.gz'.format(layer_name))
+
+        tar.close()
+
+        return tar_path
 
=== added file 'lpbuildd/tests/test_oci.py'
--- lpbuildd/tests/test_oci.py	1970-01-01 00:00:00 +0000
+++ lpbuildd/tests/test_oci.py	2020-02-26 10:52:47 +0000
@@ -0,0 +1,280 @@
+# Copyright 2019 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+__metaclass__ = type
+
+import io
+import json
+import os
+try:
+    from unittest import mock
+except ImportError:
+    import mock
+
+from fixtures import (
+    EnvironmentVariable,
+    MockPatch,
+    TempDir,
+    )
+from testtools import TestCase
+from testtools.matchers import Contains
+from testtools.deferredruntest import AsynchronousDeferredRunTest
+from twisted.internet import defer
+
+from lpbuildd.oci import (
+    OCIBuildManager,
+    OCIBuildState,
+    )
+from lpbuildd.tests.fakebuilder import FakeBuilder
+from lpbuildd.tests.oci_tarball import OCITarball
+
+
+class MockBuildManager(OCIBuildManager):
+    def __init__(self, *args, **kwargs):
+        super(MockBuildManager, self).__init__(*args, **kwargs)
+        self.commands = []
+        self.iterators = []
+
+    def runSubProcess(self, path, command, iterate=None, env=None):
+        self.commands.append([path] + command)
+        if iterate is None:
+            iterate = self.iterate
+        self.iterators.append(iterate)
+        return 0
+
+
+class MockOCITarSave():
+    @property
+    def stdout(self):
+        return io.open(OCITarball().build_tar_file(), 'rb')
+
+
+class TestOCIBuildManagerIteration(TestCase):
+    """Run OCIBuildManager through its iteration steps."""
+
+    run_tests_with = AsynchronousDeferredRunTest.make_factory(timeout=5)
+
+    def setUp(self):
+        super(TestOCIBuildManagerIteration, self).setUp()
+        self.working_dir = self.useFixture(TempDir()).path
+        builder_dir = os.path.join(self.working_dir, "builder")
+        home_dir = os.path.join(self.working_dir, "home")
+        for dir in (builder_dir, home_dir):
+            os.mkdir(dir)
+        self.useFixture(EnvironmentVariable("HOME", home_dir))
+        self.builder = FakeBuilder(builder_dir)
+        self.buildid = "123"
+        self.buildmanager = MockBuildManager(self.builder, self.buildid)
+        self.buildmanager._cachepath = self.builder._cachepath
+
+    def getState(self):
+        """Retrieve build manager's state."""
+        return self.buildmanager._state
+
+    @defer.inlineCallbacks
+    def startBuild(self, args=None, options=None):
+        # The build manager's iterate() kicks off the consecutive states
+        # after INIT.
+        extra_args = {
+            "series": "xenial",
+            "arch_tag": "i386",
+            "name": "test-image",
+            }
+        if args is not None:
+            extra_args.update(args)
+        original_backend_name = self.buildmanager.backend_name
+        self.buildmanager.backend_name = "fake"
+        self.buildmanager.initiate({}, "chroot.tar.gz", extra_args)
+        self.buildmanager.backend_name = original_backend_name
+
+        # Skip states that are done in DebianBuildManager to the state
+        # directly before BUILD_OCI.
+        self.buildmanager._state = OCIBuildState.UPDATE
+
+        # BUILD_OCI: Run the builder's payload to build the snap package.
+        yield self.buildmanager.iterate(0)
+        self.assertEqual(OCIBuildState.BUILD_OCI, self.getState())
+        expected_command = [
+            "sharepath/bin/in-target", "in-target", "build-oci",
+            "--backend=lxd", "--series=xenial", "--arch=i386", self.buildid,
+            ]
+        if options is not None:
+            expected_command.extend(options)
+        expected_command.append("test-image")
+        self.assertEqual(expected_command, self.buildmanager.commands[-1])
+        self.assertEqual(
+            self.buildmanager.iterate, self.buildmanager.iterators[-1])
+        self.assertFalse(self.builder.wasCalled("chrootFail"))
+
+    @defer.inlineCallbacks
+    def test_iterate(self):
+        # This sha would change as it includes file attributes in the
+        # tar file. Fix it so we can test against a known value.
+        sha_mock = self.useFixture(
+            MockPatch('lpbuildd.oci.OCIBuildManager._calculateLayerSha'))
+        sha_mock.mock.return_value = "testsha"
+        # The build manager iterates a normal build from start to finish.
+        args = {
+            "git_repository": "https://git.launchpad.dev/~example/+git/snap",
+            "git_path": "master",
+            }
+        expected_options = [
+            "--git-repository", "https://git.launchpad.dev/~example/+git/snap",
+            "--git-path", "master",
+            ]
+        yield self.startBuild(args, expected_options)
+
+        log_path = os.path.join(self.buildmanager._cachepath, "buildlog")
+        with open(log_path, "w") as log:
+            log.write("I am a build log.")
+
+        self.buildmanager.backend.run.result = MockOCITarSave()
+
+        self.buildmanager.backend.add_file(
+            '/var/lib/docker/image/'
+            'vfs/distribution/v2metadata-by-diffid/sha256/diff1',
+            b"""[{"Digest": "test_digest", "SourceRepository": "test"}]"""
+        )
+
+        # After building the package, reap processes.
+        yield self.buildmanager.iterate(0)
+        expected_command = [
+            "sharepath/bin/in-target", "in-target", "scan-for-processes",
+            "--backend=lxd", "--series=xenial", "--arch=i386", self.buildid,
+            ]
+        self.assertEqual(OCIBuildState.BUILD_OCI, self.getState())
+        self.assertEqual(expected_command, self.buildmanager.commands[-1])
+        self.assertNotEqual(
+            self.buildmanager.iterate, self.buildmanager.iterators[-1])
+        self.assertFalse(self.builder.wasCalled("buildFail"))
+        expected_files = [
+            'manifest.json',
+            'layer-1.tar.gz',
+            'layer-2.tar.gz',
+            'digests.json',
+            'config.json',
+            ]
+        for expected in expected_files:
+            self.assertThat(self.builder.waitingfiles, Contains(expected))
+
+        cache_path = self.builder.cachePath(
+            self.builder.waitingfiles['digests.json'])
+        with open(cache_path, "rb") as f:
+            digests_contents = f.read()
+        digests_expected = [{
+            "sha256:diff1": {
+                "source": "test",
+                "digest": "test_digest",
+                "layer_id": "layer-1"
+            },
+            "sha256:diff2": {
+                "source": "",
+                "digest": "testsha",
+                "layer_id": "layer-2"
+            }
+        }]
+        self.assertEqual(digests_contents, json.dumps(digests_expected))
+        # Control returns to the DebianBuildManager in the UMOUNT state.
+        self.buildmanager.iterateReap(self.getState(), 0)
+        expected_command = [
+            "sharepath/bin/in-target", "in-target", "umount-chroot",
+            "--backend=lxd", "--series=xenial", "--arch=i386", self.buildid,
+            ]
+        self.assertEqual(OCIBuildState.UMOUNT, self.getState())
+        self.assertEqual(expected_command, self.buildmanager.commands[-1])
+        self.assertEqual(
+            self.buildmanager.iterate, self.buildmanager.iterators[-1])
+        self.assertFalse(self.builder.wasCalled("buildFail"))
+
+    @defer.inlineCallbacks
+    def test_iterate_with_file(self):
+        # This sha would change as it includes file attributes in the
+        # tar file. Fix it so we can test against a known value.
+        sha_mock = self.useFixture(
+            MockPatch('lpbuildd.oci.OCIBuildManager._calculateLayerSha'))
+        sha_mock.mock.return_value = "testsha"
+        # The build manager iterates a build that specifies a non-default
+        # Dockerfile location from start to finish.
+        args = {
+            "git_repository": "https://git.launchpad.dev/~example/+git/snap",
+            "git_path": "master",
+            "build_file": "build-aux/Dockerfile",
+            }
+        expected_options = [
+            "--git-repository", "https://git.launchpad.dev/~example/+git/snap",
+            "--git-path", "master",
+            "--build-file", "build-aux/Dockerfile",
+            ]
+        yield self.startBuild(args, expected_options)
+
+        log_path = os.path.join(self.buildmanager._cachepath, "buildlog")
+        with open(log_path, "w") as log:
+            log.write("I am a build log.")
+
+        self.buildmanager.backend.run.result = MockOCITarSave()
+
+        self.buildmanager.backend.add_file(
+            '/var/lib/docker/image/'
+            'vfs/distribution/v2metadata-by-diffid/sha256/diff1',
+            b"""[{"Digest": "test_digest", "SourceRepository": "test"}]"""
+        )
+
+        # After building the package, reap processes.
+        yield self.buildmanager.iterate(0)
+        expected_command = [
+            "sharepath/bin/in-target", "in-target", "scan-for-processes",
+            "--backend=lxd", "--series=xenial", "--arch=i386", self.buildid,
+            ]
+        self.assertEqual(OCIBuildState.BUILD_OCI, self.getState())
+        self.assertEqual(expected_command, self.buildmanager.commands[-1])
+        self.assertNotEqual(
+            self.buildmanager.iterate, self.buildmanager.iterators[-1])
+        self.assertFalse(self.builder.wasCalled("buildFail"))
+        expected_files = [
+            'manifest.json',
+            'layer-1.tar.gz',
+            'layer-2.tar.gz',
+            'digests.json',
+            'config.json',
+            ]
+        for expected in expected_files:
+            self.assertThat(self.builder.waitingfiles, Contains(expected))
+
+        cache_path = self.builder.cachePath(
+            self.builder.waitingfiles['digests.json'])
+        with open(cache_path, "rb") as f:
+            digests_contents = f.read()
+        digests_expected = [{
+            "sha256:diff1": {
+                "source": "test",
+                "digest": "test_digest",
+                "layer_id": "layer-1"
+            },
+            "sha256:diff2": {
+                "source": "",
+                "digest": "testsha",
+                "layer_id": "layer-2"
+            }
+        }]
+        self.assertEqual(digests_contents, json.dumps(digests_expected))
+
+        # Control returns to the DebianBuildManager in the UMOUNT state.
+        self.buildmanager.iterateReap(self.getState(), 0)
+        expected_command = [
+            "sharepath/bin/in-target", "in-target", "umount-chroot",
+            "--backend=lxd", "--series=xenial", "--arch=i386", self.buildid,
+            ]
+        self.assertEqual(OCIBuildState.UMOUNT, self.getState())
+        self.assertEqual(expected_command, self.buildmanager.commands[-1])
+        self.assertEqual(
+            self.buildmanager.iterate, self.buildmanager.iterators[-1])
+        self.assertFalse(self.builder.wasCalled("buildFail"))
+
+    @defer.inlineCallbacks
+    def test_iterate_snap_store_proxy(self):
+        # The build manager can be told to use a snap store proxy.
+        self.builder._config.set(
+            "proxy", "snapstore", "http://snap-store-proxy.example/")
+        expected_options = [
+            "--snap-store-proxy-url", "http://snap-store-proxy.example/"]
+        yield self.startBuild(options=expected_options)