Merge lp:~sinzui/juju-release-tools/build-with-cloud-templates into lp:juju-release-tools

[Curtis Hovey]

Build arm64 binary packages.

This branch makes a few changes to support building arm64 packages, and as a consequence, allows us to stop pre-populating the lxc download template cache to machines on private networks. We discovered that the images for lxc "download" template are not maintained; the wily arm64 image is from June of the year! Even the ppc64el images are not updated as often as the amd64 images. Only lxc cloud images are genuinely current.

I previously tried and abandoned an effort to use the lxc ubuntu-cloud template because it was unreliable on ppc64el. It is also unreliable on arm64. We know understand that cloud init is changing the /etc/apt/sources.list, and the config is invalid for arm64 ad ppc64el. The lesser archs prefer ports.ubuntu.com/ubuntu-ports.

A. We now use the ubuntu-cloud template. This also means we don't need special egress rules for building on canonical networks.

B. We poll for cloud init to complete. This is done by tailing the cloud-init-output.log. The egrep call is case-insensitive because precise is "cloud-init", trusty and above is "Cloud-init"

C. For arm64 and ppc64el, we rewrite the archives that point to archives.ubuntu.com/ubuntu to ports.ubuntu.com/ubuntu-ports

This is tested on arm64 wily, ppc64el trusty, amd64 trusty.

We still cannot build on precise because the kernel does not support lxc-attach. We can switch from lxc-attach to ssh in the future if we wish.

[Martin Packman (gz)]

Looks good, one thing to fix, see inline comment.

review: Approve

=== modified file 'build_package.py'
--- build_package.py	2015-11-25 15:42:33 +0000
+++ build_package.py	2015-12-10 23:33:33 +0000
@@ -36,7 +36,7 @@
 
 CREATE_LXC_TEMPLATE = """\
 set -eu
-sudo lxc-create -t download -n {container} -- -d ubuntu -r {series} -a {arch}
+sudo lxc-create -t ubuntu-cloud -n {container} -- -r {series} -a {arch}
 sudo mkdir /var/lib/lxc/{container}/rootfs/workspace
 echo "lxc.mount.entry = {build_dir} workspace none bind 0 0" |
     sudo tee -a /var/lib/lxc/{container}/config
@@ -48,11 +48,20 @@
     set -eu
     echo "\nInstalling common build deps.\n"
     cd workspace
-    while ! ifconfig | grep -q "addr:10.0."; do
-        echo "Waiting for network"
-        sleep 1
+    # Wait for Cloud-init to complete to indicate the machine is in a ready
+    # state with network to do work,
+    while ! tail -1 /var/log/cloud-init-output.log | \
+            egrep -q 'Cloud-init .* finished'; do
+        echo "Waiting for Cloud-init to finish."
+        sleep 5
     done
     set +e
+    # The cloud-init breaks arm64 and ppc64el /etc/apt/sources.list.
+    if [[ $(dpkg --print-architecture) =~ ^(arm64|ppc64el)$ ]]; then
+        sed -i \
+            -e 's,archive.ubuntu.com/ubuntu,ports.ubuntu.com/ubuntu-ports,' \
+            /etc/apt/sources.list
+    fi
     # Adding the ppa directly to sources.list without the archive key
     # requires apt to be run with --force-yes
     echo "{ppa}" >> /etc/apt/sources.list