Merge ~xnox/maas-images:master into maas-images:master

[Dimitri John Ledkov]

I built a custom maas image using this code, and it does result in many new kernel modules included in the initrds.

https://paste.ubuntu.com/p/ZDJJTN6t62/

Hopefully this will finally fix the lack of cryptsetup modules, but we shall see.

[Blake Rouse (blake-rouse)]

Do you have a bug this is targeted to fix?

review: Needs Information

[Ryan Harper (raharper)]

Does this have any impact in previous releases or Disco only?

[Dimitri John Ledkov (xnox)]

I believe /etc/cryptsetup-initramfs/conf-hook is new, and we do want to tweak it, as stated if it exists. So it's ok to do the tweak unguarded for series tests.

This is still Bug #1818876: maas images lack enough of crypto for cryptsetup in the ephemeral images
Now linked to this PR... 3rd time lucky

[Blake Rouse (blake-rouse)]

That file exists on bionic

https://packages.ubuntu.com/search?searchon=contents&keywords=cryptsetup-initramfs%2Fconf-hook&mode=exactfilename&suite=bionic&arch=any

How does this change affect the previous releases?

review: Needs Information

[Dimitri John Ledkov (xnox)]

Yes it exists, shipped by a different package prior to the cryptsetup/cryptsetup-initramfs split. But the contents of it / interface is the same. Meaning if you want to force inclusion of cryptsetup in the initrd, one needs to set the CRYPTSETUP key to y - which we do want for maas images.

What is constantly changing is the logic as to what to do when that key is not set to anything and autodetection does or doesn't detect encrypted partitions. And we do want to stop update-initramfs from querieying disks as seen in the livefs builder and simply include the crypto tools in the initramfs.

c79018a... by Dimitri John Ledkov on 2019-03-14

Address review comments, replace CRYPTSETUP setting in full.

[Blake Rouse (blake-rouse)]

@ltrager - How can we do a run with this change in the CI? We should verify this doesn't break deployment before landing?

[Dimitri John Ledkov (xnox)]

I've built the maas-images with this change in, and lots of kernel modules are added to the initramfs now on disco. Meaning that one can use encryption.

I'm not sure what you mean by running this change in the CI though. As nothing uses encryption at the moment in production.

Do you not build test maas-images yourselves using livefs builds in launchpad to test things? Do you want me to provide a new maas-images build?

Also note, that it is odd how maas-images repository is structured, since all other images we build use per-series code to build them. Ie. livecd-rootfs (for ubuntu-desktop/server/stock-cloud-images/etc), CPC code (partner cloud-images) are all maintained in per-series branches. maas-images appears to be the only one that tries to build any series images, from the same master branch.

I find it disturbing that maas-images master branch is used for e.g. building bionic images. Rather than using bionic branch.... Cause it should be normal to land changes on master, to test it out with disco. And then later choose to backport that onto bionic branch to land a change in bionic maas-images.

[Lee Trager (ltrager)]

@blake-rouse - You can use proposed-images-manual in the CI. You can set the git repo although the images branch must be on master(we should fix that).

@xnox - lp:maas-images is taking the images that CPC generates on http://cloud-images.ubuntu.com/ and creates a stream for MAAS. The image itself(the SquashFS file) is exactly what CPC produces. All lp:maas-images is doing is chrooting into the image with a memfs overlay to get the kernel and initrd. Since the MAAS ephemeral environment needs additional resources in the initrd we cannot rely on what is in packaging. We try to keep the changes minimal from the image CPC produces. Typically a new series only requires modifying meph-v3.yaml.

When the images are being produced CPC does generate each image on the distro series that is being produced(e.g xenial are produced on xenial, bionic on bionc, etc). We keep everything in one branch to ease development and because its always been that way.

diff --git a/bin/maas-cloudimg2ephemeral b/bin/maas-cloudimg2ephemeral
index aedec14..e05c12d 100755
--- a/bin/maas-cloudimg2ephemeral
+++ b/bin/maas-cloudimg2ephemeral
@@ -487,6 +487,15 @@ apply_updates_chrooted() {
         log_change false "$change"
     fi
 
+    change="set CRYPTSETUP=y in /etc/cryptsetup-initramfs/conf-hook (LP: #1818876)"
+    if [ -e "/etc/cryptsetup-initramfs/conf-hook" ]; then
+        log_change true "$change"
+        sed -i 's/#CRYPTSETUP=.*/CRYPTSETUP=y/' /etc/cryptsetup-initramfs/conf-hook ||
+            { error "failed to $change"; return 1; }
+    else
+        log_change false "$change"
+    fi
+
     change="update-initramfs"
     if [ "$kpkg" != "none" ]; then
         local k="" i=""