Merge ~cjwatson/launchpad:livefs-build-pocket into launchpad:master

[Colin Watson]

Allow livefs build metadata to override the default build pocket

This makes it possible, for example, to distinguish the case of building a livefs using tools from -updates but with contents from the release pocket from the case of building a livefs using tools from -updates and contents from -updates.

The use case here is that at the moment buildd images are built with tools from -updates but with contents from the release pocket. I'd like it to be possible to build buildd images with contents from -updates as well, since we could use those for builds against -updates and it would save a good deal of time at the start of each build.

We've ended up in a bit of a compatibility corner due to some unfortunate past decisions (mostly by me), and this is the least bad solution I can think of. launchpad-buildd and livecd-rootfs will also need some work to support this, but this design allows it to be done in a somewhat backwards-compatible way (that is, it wouldn't break existing livecd-rootfs code). The one compatibility issue is that existing livefses that intend to continue building with tools from -updates and contents from the release pocket (or similar) will need to have "pocket": "release" explicitly added to their metadata before landing the corresponding launchpad-buildd change which starts paying attention to the "pocket" argument; we'll need to go through the current database to check for that kind of thing.

This is essentially the same as https://code.launchpad.net/~cjwatson/launchpad/livefs-build-pocket/+merge/372444, converted to git and rebased on master.

Unmerged commits

ed26842... by Colin Watson on 2019-09-06

Allow livefs build metadata to override the default build pocket.

This makes it possible, for example, to distinguish the case of building a
livefs using tools from -updates but with contents from the release pocket
from the case of building a livefs using tools from -updates and contents
from -updates.

diff --git a/lib/lp/soyuz/model/livefsbuildbehaviour.py b/lib/lp/soyuz/model/livefsbuildbehaviour.py
index bef72b0..ec5dbaf 100644
--- a/lib/lp/soyuz/model/livefsbuildbehaviour.py
+++ b/lib/lp/soyuz/model/livefsbuildbehaviour.py
@@ -86,15 +86,21 @@ class LiveFSBuildBehaviour(BuildFarmJobBehaviourBase):
         build = self.build
         base_args = yield super(LiveFSBuildBehaviour, self).extraBuildArgs(
             logger=logger)
+        args = {}
+        # Allow the metadata to override the default build pocket; this is
+        # useful e.g. to build a livefs using tools from -updates but with
+        # contents from the release pocket.  Note that the "pocket" argument
+        # here is only used for passing instructions to livecd-rootfs, not
+        # for constructing the sources.list used for the build.
+        args["pocket"] = build.pocket.name.lower()
         # Non-trivial metadata values may have been security-wrapped, which
         # is pointless here and just gets in the way of xmlrpclib
         # serialisation.
-        args = dict(removeSecurityProxy(build.livefs.metadata))
+        args.update(removeSecurityProxy(build.livefs.metadata))
         if build.metadata_override is not None:
             args.update(removeSecurityProxy(build.metadata_override))
         # Everything else overrides anything in the metadata.
         args.update(base_args)
-        args["pocket"] = build.pocket.name.lower()
         args["datestamp"] = build.version
         args["archives"], args["trusted_keys"] = (
             yield get_sources_list_for_building(
diff --git a/lib/lp/soyuz/tests/test_livefsbuildbehaviour.py b/lib/lp/soyuz/tests/test_livefsbuildbehaviour.py
index 29e697e..bfafa1f 100644
--- a/lib/lp/soyuz/tests/test_livefsbuildbehaviour.py
+++ b/lib/lp/soyuz/tests/test_livefsbuildbehaviour.py
@@ -261,7 +261,7 @@ class TestAsyncLiveFSBuildBehaviour(TestLiveFSBuildBehaviourBase):
 
     @defer.inlineCallbacks
     def test_extraBuildArgs_metadata_cannot_override_base(self):
-        # Items in the user-provided metadata cannot override the base
+        # Most items in the user-provided metadata cannot override the base
         # arguments.
         job = self.makeJob(
             metadata={"project": "distro", "arch_tag": "nonsense"},
@@ -271,6 +271,17 @@ class TestAsyncLiveFSBuildBehaviour(TestLiveFSBuildBehaviourBase):
         self.assertEqual("i386", args["arch_tag"])
 
     @defer.inlineCallbacks
+    def test_extraBuildArgs_metadata_pocket_overrides_base(self):
+        # The "pocket" item in the user-provided metadata overrides the base
+        # arguments, to allow for building a livefs with content whose
+        # source differs from the tools used to build it.
+        job = self.makeJob(
+            pocket=PackagePublishingPocket.UPDATES,
+            metadata={"pocket": "release"}, with_builder=True)
+        args = yield job.extraBuildArgs()
+        self.assertEqual("release", args["pocket"])
+
+    @defer.inlineCallbacks
     def test_composeBuildRequest(self):
         job = self.makeJob(with_builder=True)
         lfa = self.factory.makeLibraryFileAlias(db_only=True)