Merge lp:~tobijk/launchpad-buildd/launchpad-buildd-bionic into lp:launchpad-buildd

[Tobias Koch]

Update LXD backend to work with newer LXD versions.

This patch makes the buildd's LXD backend work on Bionic:

* Based on LXD version modify raw.lxc settings (some parameter names have changed)
* Fix a problem with header checks where uid and gid are set to 0, but need to be strings
* Explicitly specify a root device, which seems to be mandatory now

[Colin Watson (cjwatson)]

Thanks for working on this!

As well as the inline comments I've left, I'm pretty sure that this is going to need changes to the test suite.

review: Needs Fixing

[Tobias Koch (tobijk)]

Thanks a lot for the initial review! When you find the time, maybe you can skim over this and see if it is better now. I'll see what I can do about the tests.

[Tobias Koch (tobijk)]

@cjwatson, please review.

[Colin Watson (cjwatson)]

review: Needs Fixing

[lp:~tobijk/launchpad-buildd/launchpad-buildd-bionic]

348. By Colin Watson on 2018-10-09

[r=cjwatson] buildlivefs: support passing an IMAGE_TARGETS variable into the environment

This is needed for projects, such as ubuntu-cpc, that produce multiple
images as part of a single build, in order to be selective about which image
sets to produce.

349. By Colin Watson on 2018-10-11

[r=wgrant] Set SNAPCRAFT_BUILD_ENVIRONMENT=host when building snaps (LP: #1791201).

350. By Colin Watson on 2018-10-19

[r=wgrant] Call gatherResults in a different thread so that it doesn't block responses to XML-RPC requests.

351. By Colin Watson on 2018-10-19

releasing package launchpad-buildd version 165

[Tobias Koch (tobijk)]

@cjwatson, ready for re-review.

review: Needs Resubmitting

[Tobias Koch (tobijk)]

Is there a reason why only tests in lpbuildd/tests are run during package build?

[lp:~tobijk/launchpad-buildd/launchpad-buildd-bionic]

352. By Colin Watson on 2018-10-25

Run all tests at package build time, not just those in lpbuildd.tests.

[Colin Watson (cjwatson)]

(The "Resubmit" vote is something that your reviewer might use to indicate that the MP needs to be entirely withdrawn and resubmitted, perhaps against a different branch. It isn't something you do when you've made revisions in response to a review.)

Regarding tests at package build time, oops; I've fixed that in trunk. Thanks for pointing it out.

[lp:~tobijk/launchpad-buildd/launchpad-buildd-bionic]

353. By Tobias Koch on 2018-10-25

Update LXD backend to work with LXD 3.

[Tobias Koch (tobijk)]

@cjwatson, thanks for the clarification on the "resubmit". I'm not sure how you want the re-factoring on `raw_lxc_config` to look like, so maybe it's better if you just mold it to your liking. Thanks.

[Colin Watson (cjwatson)]

review: Approve

=== modified file 'debian/changelog'
--- debian/changelog	2018-10-25 09:04:12 +0000
+++ debian/changelog	2018-10-25 09:55:10 +0000
@@ -1,7 +1,11 @@
 launchpad-buildd (166) UNRELEASED; urgency=medium
 
+  [Colin Watson]
   * Run all tests at package build time, not just those in lpbuildd.tests.
 
+  [Tobias Koch]
+  * Update LXD backend to work with LXD 3.
+
  -- Colin Watson <[e-mail]>  Thu, 25 Oct 2018 10:03:14 +0100
 
 launchpad-buildd (165) xenial; urgency=medium

=== modified file 'lpbuildd/target/lxd.py'
--- lpbuildd/target/lxd.py	2018-06-12 23:23:13 +0000
+++ lpbuildd/target/lxd.py	2018-10-25 09:55:10 +0000
@@ -277,16 +277,30 @@
             old_profile.delete()
 
         raw_lxc_config = [
-            ("lxc.aa_profile", "unconfined"),
             ("lxc.cap.drop", ""),
             ("lxc.cap.drop", "sys_time sys_module"),
             ("lxc.cgroup.devices.deny", ""),
             ("lxc.cgroup.devices.allow", ""),
             ("lxc.mount.auto", ""),
             ("lxc.mount.auto", "proc:rw sys:rw"),
-            ("lxc.network.0.ipv4", ipv4_address),
-            ("lxc.network.0.ipv4.gateway", self.ipv4_network.ip),
             ]
+
+        lxc_version = self._client.host_info["environment"]["driver_version"]
+        major, minor = [int(v) for v in lxc_version.split(".")[0:2]]
+
+        if major >= 3:
+            raw_lxc_config.extend([
+                ("lxc.apparmor.profile", "unconfined"),
+                ("lxc.net.0.ipv4.address", ipv4_address),
+                ("lxc.net.0.ipv4.gateway", self.ipv4_network.ip),
+                ])
+        else:
+            raw_lxc_config.extend([
+                ("lxc.aa_profile", "unconfined"),
+                ("lxc.network.0.ipv4", ipv4_address),
+                ("lxc.network.0.ipv4.gateway", self.ipv4_network.ip),
+                ])
+
         # Linux 4.4 on powerpc doesn't support all the seccomp bits that LXD
         # needs.
         if self.arch == "powerpc":
@@ -296,7 +310,7 @@
             "security.nesting": "true",
             "raw.lxc": "".join(
                 "{key}={value}\n".format(key=key, value=value)
-                for key, value in raw_lxc_config),
+                for key, value in sorted(raw_lxc_config)),
             }
         devices = {
             "eth0": {
@@ -306,6 +320,12 @@
                 "type": "nic",
                 },
             }
+        if major >= 3:
+            devices["root"] = {
+                "path": "/",
+                "pool": "default",
+                "type": "disk",
+                }
         self.client.profiles.create(self.profile_name, config, devices)
 
     def start(self):
@@ -341,7 +361,17 @@
             hostname_file.flush()
             os.fchmod(hostname_file.fileno(), 0o644)
             self.copy_in(hostname_file.name, "/etc/hostname")
-        self.copy_in("/etc/resolv.conf", "/etc/resolv.conf")
+
+        resolv_conf = "/etc/resolv.conf"
+
+        if os.path.islink(resolv_conf):
+            resolv_conf = os.path.realpath(resolv_conf)
+            if (resolv_conf == "/run/systemd/resolve/stub-resolv.conf" and
+                os.path.isfile("/run/systemd/resolve/resolv.conf")):
+                    resolv_conf = "/run/systemd/resolve/resolv.conf"
+
+        self.copy_in(resolv_conf, "/etc/resolv.conf")
+
         with tempfile.NamedTemporaryFile(mode="w+") as policy_rc_d_file:
             policy_rc_d_file.write(policy_rc_d)
             policy_rc_d_file.flush()
@@ -481,8 +511,8 @@
             data = source_file.read()
             mode = stat.S_IMODE(os.fstat(source_file.fileno()).st_mode)
             headers = {
-                "X-LXD-uid": 0,
-                "X-LXD-gid": 0,
+                "X-LXD-uid": "0",
+                "X-LXD-gid": "0",
                 "X-LXD-mode": "%#o" % mode,
                 }
             try:

=== modified file 'lpbuildd/target/tests/test_lxd.py'
--- lpbuildd/target/tests/test_lxd.py	2018-05-08 09:37:16 +0000
+++ lpbuildd/target/tests/test_lxd.py	2018-10-25 09:55:10 +0000
@@ -155,23 +155,45 @@
         image.add_alias.assert_called_once_with(
             "lp-xenial-amd64", "lp-xenial-amd64")
 
-    def assert_correct_profile(self, extra_raw_lxc_config=""):
+    def assert_correct_profile(self, extra_raw_lxc_config=None,
+            driver_version="2.0"):
+        if extra_raw_lxc_config is None:
+            extra_raw_lxc_config = []
+
         client = pylxd.Client()
         client.profiles.get.assert_called_once_with("lpbuildd")
+
+        raw_lxc_config = [
+            ("lxc.cap.drop", ""),
+            ("lxc.cap.drop", "sys_time sys_module"),
+            ("lxc.cgroup.devices.deny", ""),
+            ("lxc.cgroup.devices.allow", ""),
+            ("lxc.mount.auto", ""),
+            ("lxc.mount.auto", "proc:rw sys:rw"),
+            ]
+
+        major, minor = [int(v) for v in driver_version.split(".")[0:2]]
+
+        if major >= 3:
+            raw_lxc_config.extend([
+                ("lxc.apparmor.profile", "unconfined"),
+                ("lxc.net.0.ipv4.address", "10.10.10.2/24"),
+                ("lxc.net.0.ipv4.gateway", "10.10.10.1"),
+                ])
+        else:
+            raw_lxc_config.extend([
+                ("lxc.aa_profile", "unconfined"),
+                ("lxc.network.0.ipv4", "10.10.10.2/24"),
+                ("lxc.network.0.ipv4.gateway", "10.10.10.1"),
+                ])
+
+        raw_lxc_config = "".join("{key}={val}\n".format(key=key, val=val)
+                for key, val in sorted(raw_lxc_config + extra_raw_lxc_config))
+
         expected_config = {
             "security.privileged": "true",
             "security.nesting": "true",
-            "raw.lxc": dedent("""\
-                lxc.aa_profile=unconfined
-                lxc.cap.drop=
-                lxc.cap.drop=sys_time sys_module
-                lxc.cgroup.devices.deny=
-                lxc.cgroup.devices.allow=
-                lxc.mount.auto=
-                lxc.mount.auto=proc:rw sys:rw
-                lxc.network.0.ipv4=10.10.10.2/24
-                lxc.network.0.ipv4.gateway=10.10.10.1
-                """) + extra_raw_lxc_config,
+            "raw.lxc": raw_lxc_config,
             }
         expected_devices = {
             "eth0": {
@@ -181,22 +203,42 @@
                 "type": "nic",
                 },
             }
+        if driver_version == "3.0":
+            expected_devices["root"] = {
+                "path": "/",
+                "pool": "default",
+                "type": "disk",
+                }
         client.profiles.create.assert_called_once_with(
             "lpbuildd", expected_config, expected_devices)
 
     def test_create_profile_amd64(self):
-        self.useFixture(MockPatch("pylxd.Client"))
-        client = pylxd.Client()
-        client.profiles.get.side_effect = FakeLXDAPIException
-        LXD("1", "xenial", "amd64").create_profile()
-        self.assert_correct_profile()
+        with MockPatch("pylxd.Client"):
+            for driver_version in ["2.0", "3.0"]:
+                client = pylxd.Client()
+                client.reset_mock()
+                client.profiles.get.side_effect = FakeLXDAPIException
+                client.host_info = {
+                    "environment": {"driver_version": driver_version}
+                    }
+                LXD("1", "xenial", "amd64").create_profile()
+                self.assert_correct_profile(
+                        driver_version=driver_version or "3.0")
 
     def test_create_profile_powerpc(self):
-        self.useFixture(MockPatch("pylxd.Client"))
-        client = pylxd.Client()
-        client.profiles.get.side_effect = FakeLXDAPIException
-        LXD("1", "xenial", "powerpc").create_profile()
-        self.assert_correct_profile("lxc.seccomp=\n")
+        with MockPatch("pylxd.Client"):
+            for driver_version in ["2.0", "3.0"]:
+                client = pylxd.Client()
+                client.reset_mock()
+                client.profiles.get.side_effect = FakeLXDAPIException
+                client.host_info = {
+                    "environment": {"driver_version": driver_version}
+                    }
+                LXD("1", "xenial", "powerpc").create_profile()
+                self.assert_correct_profile(
+                        extra_raw_lxc_config=[("lxc.seccomp", ""),],
+                        driver_version=driver_version or "3.0"
+                        )
 
     def test_start(self):
         fs_fixture = self.useFixture(FakeFilesystem())
@@ -213,6 +255,7 @@
         client.profiles.get.side_effect = FakeLXDAPIException
         container = client.containers.create.return_value
         client.containers.get.return_value = container
+        client.host_info = {"environment": {"driver_version": "2.0"}}
         container.start.side_effect = (
             lambda wait=False: setattr(container, "status_code", LXD_RUNNING))
         files_api = container.api.files
@@ -298,19 +341,19 @@
             data=(
                 b"127.0.0.1\tlocalhost\n\n"
                 b"127.0.1.1\texample.buildd example\n"),
-            headers={"X-LXD-uid": 0, "X-LXD-gid": 0, "X-LXD-mode": "0644"})
+            headers={"X-LXD-uid": "0", "X-LXD-gid": "0", "X-LXD-mode": "0644"})
         files_api.post.assert_any_call(
             params={"path": "/etc/hostname"},
             data=b"example\n",
-            headers={"X-LXD-uid": 0, "X-LXD-gid": 0, "X-LXD-mode": "0644"})
+            headers={"X-LXD-uid": "0", "X-LXD-gid": "0", "X-LXD-mode": "0644"})
         files_api.post.assert_any_call(
             params={"path": "/etc/resolv.conf"},
             data=b"host resolv.conf\n",
-            headers={"X-LXD-uid": 0, "X-LXD-gid": 0, "X-LXD-mode": "0644"})
+            headers={"X-LXD-uid": "0", "X-LXD-gid": "0", "X-LXD-mode": "0644"})
         files_api.post.assert_any_call(
             params={"path": "/usr/local/sbin/policy-rc.d"},
             data=policy_rc_d.encode("UTF-8"),
-            headers={"X-LXD-uid": 0, "X-LXD-gid": 0, "X-LXD-mode": "0755"})
+            headers={"X-LXD-uid": "0", "X-LXD-gid": "0", "X-LXD-mode": "0755"})
         files_api.session.get.assert_any_call(
             "/1.0/containers/lp-xenial-amd64/files",
             params={"path": "/etc/init/mounted-dev.conf"}, stream=True)
@@ -321,7 +364,7 @@
         files_api.post.assert_any_call(
             params={"path": "/etc/systemd/system/snapd.service.d/no-cdn.conf"},
             data=b"[Service]\nEnvironment=SNAPPY_STORE_NO_CDN=1\n",
-            headers={"X-LXD-uid": 0, "X-LXD-gid": 0, "X-LXD-mode": "0644"})
+            headers={"X-LXD-uid": "0", "X-LXD-gid": "0", "X-LXD-mode": "0644"})
         container.start.assert_called_once_with(wait=True)
         self.assertEqual(LXD_RUNNING, container.status_code)
 
@@ -340,6 +383,7 @@
         client.profiles.get.side_effect = FakeLXDAPIException
         container = client.containers.create.return_value
         client.containers.get.return_value = container
+        client.host_info = {"environment": {"driver_version": "2.0"}}
         container.start.side_effect = (
             lambda wait=False: setattr(container, "status_code", LXD_RUNNING))
         files_api = container.api.files
@@ -360,7 +404,7 @@
             data=(
                 fallback_hosts +
                 "\n127.0.1.1\texample.buildd example\n").encode("UTF-8"),
-            headers={"X-LXD-uid": 0, "X-LXD-gid": 0, "X-LXD-mode": "0644"})
+            headers={"X-LXD-uid": "0", "X-LXD-gid": "0", "X-LXD-mode": "0644"})
 
     def test_start_with_mounted_dev_conf(self):
         fs_fixture = self.useFixture(FakeFilesystem())
@@ -375,6 +419,7 @@
         self.useFixture(MockPatch("pylxd.Client"))
         client = pylxd.Client()
         client.profiles.get.side_effect = FakeLXDAPIException
+        client.host_info = {"environment": {"driver_version": "2.0"}}
         container = client.containers.create.return_value
         client.containers.get.return_value = container
         container.start.side_effect = (
@@ -406,7 +451,7 @@
                     : # /sbin/MAKEDEV std fd ppp tun
                 end script
                 """).encode("UTF-8"),
-            headers={"X-LXD-uid": 0, "X-LXD-gid": 0, "X-LXD-mode": "0644"})
+            headers={"X-LXD-uid": "0", "X-LXD-gid": "0", "X-LXD-mode": "0644"})
 
     def test_run(self):
         processes_fixture = self.useFixture(FakeProcesses())
@@ -456,7 +501,7 @@
         container.api.files.post.assert_called_once_with(
             params={"path": target_path},
             data=b"hello\n",
-            headers={"X-LXD-uid": 0, "X-LXD-gid": 0, "X-LXD-mode": "0644"})
+            headers={"X-LXD-uid": "0", "X-LXD-gid": "0", "X-LXD-mode": "0644"})
 
     def test_copy_in_error(self):
         source_dir = self.useFixture(TempDir()).path