Merge lp:~tribaal/launchpad-buildd/apt-proxy-injection into lp:launchpad-buildd

[Chris Glass]

This branch adds a "--apt-http-proxy" option to the buildlivefs command, injecting the http proxy it specifies into the chroot (by setting the appropriate env).

This allows speeding up local development of cloud images (for example) significantly by setting up a squid-deb-proxy or similar.

[Colin Watson (cjwatson)]

Please can you add a test for this? It should be possible to test the new behaviour in lpbuildd/target/tests/test_build_livefs.py.

Aside from the code review comments below, this is probably fine as far as it goes. However, I'd suggest some follow-up work (which could be in a separate branch).

One of my usual local tweaks to launchpad-buildd is to add this to the end of lpbuildd.target.apt.OverrideSourcesList.run:

        self.backend.copy_in(
            "/etc/apt/apt.conf.d/99proxy", "/etc/apt/apt.conf.d/99proxy")

This isn't totally satisfactory for various reasons, but it's been good enough as a local hack.

Now, if we were to polish this up, the most convenient approach would be if there were an item in the buildd's configuration file that gave a proxy URL. lpbuildd.debian.DebianBuildManager.doSourcesList could turn this into an option passed to override-sources-list, which could then inject an apt.conf.d file if that option is set. If we did that, it would make sense for lpbuildd.livefs to look at that same configuration item and pass the --apt-http-proxy option to buildlivefs too (which as it stands you're presumably having to do manually).

review: Needs Fixing

[lp:~tribaal/launchpad-buildd/apt-proxy-injection]

323. By Chris Glass on 2018-01-09

Addressed review comments:
* Added test
* Fixed style
* Fixed dch version
* Made --http-proxy a simple option (not a list)

[Chris Glass (tribaal)]

All comments should be addressed.

[Colin Watson (cjwatson)]

review: Approve

=== modified file 'debian/changelog'
--- debian/changelog	2017-11-29 14:19:20 +0000
+++ debian/changelog	2018-01-09 10:54:42 +0000
@@ -1,3 +1,9 @@
+launchpad-buildd (158) UNRELEASED; urgency=medium
+
+  * Add support for passing apt proxies to the debootstrap/chroot.
+
+ -- Christopher Glass (Ubuntu) <[e-mail]>  Fri, 05 Jan 2018 12:49:11 +0100
+
 launchpad-buildd (157) xenial; urgency=medium
 
   [ Colin Watson ]

=== modified file 'lpbuildd/target/build_livefs.py'
--- lpbuildd/target/build_livefs.py	2017-11-13 16:07:55 +0000
+++ lpbuildd/target/build_livefs.py	2018-01-09 10:54:42 +0000
@@ -58,6 +58,9 @@
             "--extra-ppa", dest="extra_ppas", default=[], action="append",
             help="use this additional PPA")
         parser.add_argument(
+            "--http-proxy", default="", action="store",
+            help="use this http proxy for apt.")
+        parser.add_argument(
             "--debug", default=False, action="store_true",
             help="enable detailed live-build debugging")
 
@@ -127,6 +130,13 @@
                 lb_env["PROPOSED"] = "1"
             if self.args.extra_ppas:
                 lb_env["EXTRA_PPAS"] = " ".join(self.args.extra_ppas)
+            if self.args.http_proxy:
+                proxy_dict = {
+                    "http_proxy": self.args.http_proxy,
+                    "LB_APT_HTTP_PROXY": self.args.http_proxy
+                    }
+                lb_env.update(proxy_dict)
+                base_lb_env.update(proxy_dict)
             self.run_build_command(["lb", "config"], env=lb_env)
             self.run_build_command(["lb", "build"], env=base_lb_env)
 

=== modified file 'lpbuildd/target/tests/test_build_livefs.py'
--- lpbuildd/target/tests/test_build_livefs.py	2017-11-10 20:55:33 +0000
+++ lpbuildd/target/tests/test_build_livefs.py	2018-01-09 10:54:42 +0000
@@ -184,6 +184,40 @@
             RanBuildCommand(["lb", "build"], PROJECT="ubuntu", ARCH="amd64"),
             ]))
 
+    def test_build_with_http_proxy(self):
+        proxy = "http://example.com:8000"
+        expected_env = {"PROJECT": "ubuntu-cpc",
+                        "ARCH": "amd64",
+                        "http_proxy": proxy,
+                        "LB_APT_HTTP_PROXY": proxy,
+                        }
+        args = [
+            "buildlivefs",
+            "--backend=fake", "--series=xenial", "--arch=amd64",
+            "--project=ubuntu-cpc",
+            "--http-proxy={}".format(proxy), "1"
+            ]
+        build_livefs = parse_args(args=args).operation
+        build_livefs.build()
+        self.assertThat(build_livefs.backend.run.calls, MatchesListwise([
+            RanBuildCommand(["rm", "-rf", "auto", "local"]),
+            RanBuildCommand(["mkdir", "-p", "auto"]),
+            RanBuildCommand(
+                ["ln", "-s",
+                 "/usr/share/livecd-rootfs/live-build/auto/config", "auto/"]),
+            RanBuildCommand(
+                ["ln", "-s",
+                 "/usr/share/livecd-rootfs/live-build/auto/build", "auto/"]),
+            RanBuildCommand(
+                ["ln", "-s",
+                 "/usr/share/livecd-rootfs/live-build/auto/clean", "auto/"]),
+            RanBuildCommand(["lb", "clean", "--purge"]),
+            RanBuildCommand(
+                ["lb", "config"],
+                SUITE="xenial", **expected_env),
+            RanBuildCommand(["lb", "build"], **expected_env),
+            ]))
+
     def test_run_succeeds(self):
         args = [
             "buildlivefs",