Merge lp:~vorlon/livecd-rootfs/minimize-round-two into lp:livecd-rootfs

[Steve Langasek]

Round two of minimizing our minimized cloud images. The first round removed packages from the image that were part of the minimal seed but unnecessary in the minimized target env; now we look at packages that are part of the cloud/server seed and similarly unnecessary in a non-interactive VM.

Also includes some generic improvements to image size, by removing various backup files that don't need to be retained in a brand new image.

[Chris Glass (tribaal)]

One small comment inline.

Will run a full GKE cluster test on this.

[Chris Glass (tribaal)]

Ok, so testing this branch on GKE is in a bit of a catch-22 situation right now:

- I can't test this on xenial (the gke images are xenial-only for production): it fails to build because it needs a more recent live-build (https://pastebin.canonical.com/201353/). This is a heads up when backporting the changes to xenial, too.

- I can't test this on artful either (as a non-production image to kick the tires) because linux-gke (and linux-aws, even if not so relevant) don't exist there.

- I can't test a minimized *GCE* image with trunk sauce either, it fails with https://pastebin.canonical.com/201397/ . This one could probably be solved by running "mkdir -p <mountpoint>/etc/initramfs-tools/conf.d/" (initramfs-tools obviously didn't create that folder since it was not installed). I could also change the sauce to test for minimized, if that's a better solution.

All of those failures seem to be unrelated to your changes here (that look otherwise sensible). Will investigate/try to get a run with a (at least temporary) fixed sauce tomorrow.

[Steve Langasek (vorlon)]

I have an artful ppa set up for minimized image testing that includes copies of the xenial linux-gke and linux-kvm packages. You should be able to test an artful minimized build of this branch using EXTRA_PPAS=canonical-foundations/hyperv-staging.

[Steve Langasek (vorlon)]

Also, wrt backporting, this live-build versioned dependency is already fixed up on the xenial branch to reference the proper SRU version.

[Balint Reczey (rbalint)]

I tested this branch and the minimized image can still be used to derive a working Hyper-V image with OpenGCS.

LGTM.

[Balint Reczey (rbalint)]

review: Approve (lcow)

[Chris Glass (tribaal)]

Found a quirk in the xenial side here. Will fix in a branch locally and try to push through.

[Chris Glass (tribaal)]

Xenial FTB even when removing publicsuffix from the list of packages to remove:

https://pastebin.canonical.com/201731/

Please advise - I'm not sure I can test this branch on GCE/GKE as it is now.

[Steve Langasek (vorlon)]

On Thu, Oct 26, 2017 at 02:31:01PM -0000, Chris Glass wrote:
> Xenial FTB even when removing publicsuffix from the list of packages to remove:

> https://pastebin.canonical.com/201731/

This seems to be some sort of mis-merge.
lp:~ubuntu-core-dev/livecd-rootfs/xenial-proposed has the following in
live-build/ubuntu-cpc/hooks/032-disk-image.binary:

create_empty_partition_table() {
    apt-get install -qqy parted

    parted "$1" --script -- mklabel msdos
}

Your pastebin output clearly shows that the call to apt-get is missing:

+ create_empty_partition_table binary/boot/disk.ext4
+ parted binary/boot/disk.ext4 --script -- mklabel msdos

Some sort of local damage to your branch? Cosmic rays?

[Chris Glass (tribaal)]

Alright, I found where the problem was, and it wasn't this branch at all (or any branch for that matter): The update-grub postinst script noops inside containers (as reported by systemd-detect-virt), and therefore you would not see the problem running the build locally, but launchpad now isolates builds inside containers.

+1'ing this, provided publicsuffix is appropriately taken care of (see inline comments).

I will send a branch to fix the build separately.

review: Approve

[Chris Glass (tribaal)]

For the record:

This branch was "merged" onto the xenial-proposed livecd-rootfs branch, and with the publicsuffix package removed from the uninstalled list produced a GKE image that passed k8s cluster tests.

=== modified file 'debian/changelog'
--- debian/changelog	2017-10-19 06:20:19 +0000
+++ debian/changelog	2017-10-19 23:34:00 +0000
@@ -1,6 +1,20 @@
 livecd-rootfs (2.479) UNRELEASED; urgency=medium
 
   * Purge initramfs-tools from minimized images.
+  * Remove boot/grub leftovers from our root squashfs, left behind after
+    grub purge.
+  * Remove apt, debconf, dpkg cruft files from /var/cache and /var/lib in all
+    our livefses.
+  * Pass --cache false to lb config; otherwise we copy around caches of
+    .debs that are never used properly, and which prevent us from emptying
+    /var/cache/apt in images.
+  * When building minimized cloud images, remove various packages that we
+    don't want installed by default.  Some are tools that aren't needed for
+    non-interactive use; some are libraries whose reverse-dependencies
+    will have already been removed; and one, open-vm-tools, should only be
+    included in images that are targeted to VMWare (which is not the case
+    for any of the current minimal images), rather than being included
+    directly in the cloud-image seed.
 
  -- Steve Langasek <[e-mail]>  Fri, 13 Oct 2017 17:14:58 -0400
 

=== modified file 'live-build/auto/build'
--- live-build/auto/build	2017-10-14 03:27:07 +0000
+++ live-build/auto/build	2017-10-19 23:34:00 +0000
@@ -225,6 +225,10 @@
 			apt-get -y --purge autoremove"
 	fi
 
+	# remove crufty files that shouldn't be left in an image
+	rm -f chroot/var/cache/debconf/*-old chroot/var/lib/dpkg/*-old
+	Chroot chroot apt clean
+
 	if [ -f config/oem-config-preinstalled ]; then
 
 # This is cargo-culted almost verbatim (with some syntax changes for

=== modified file 'live-build/auto/config'
--- live-build/auto/config	2017-10-10 23:57:03 +0000
+++ live-build/auto/config	2017-10-19 23:34:00 +0000
@@ -694,6 +694,7 @@
 	--initsystem none \
 	--bootloader "$BOOTLOADER" \
 	--initramfs-compression lzma \
+	--cache false \
 	${BOOTAPPEND_LIVE:+--bootappend-live "$BOOTAPPEND_LIVE"} \
 	$OPTS \
 	"$@"

=== modified file 'live-build/ubuntu-cpc/hooks/032-root-squashfs.binary'
--- live-build/ubuntu-cpc/hooks/032-root-squashfs.binary	2017-09-13 18:31:12 +0000
+++ live-build/ubuntu-cpc/hooks/032-root-squashfs.binary	2017-10-19 23:34:00 +0000
@@ -31,6 +31,7 @@
 env DEBIAN_FRONTEND=noninteractive chroot binary/boot/squashfs.dir apt-get --purge remove --assume-yes '^linux-.*' 'linux-base+'
 env DEBIAN_FRONTEND=noninteractive chroot binary/boot/squashfs.dir apt-get --purge remove --assume-yes '^grub-.*'
 env DEBIAN_FRONTEND=noninteractive chroot binary/boot/squashfs.dir apt-get autoremove --purge --assume-yes
+rm -rf binary/boot/squashfs.dir/boot/grub
 chroot binary/boot/squashfs.dir mkdir /lib/modules
 
 chroot binary/boot/squashfs.dir rm /usr/sbin/grub-probe

=== modified file 'live-build/ubuntu-cpc/hooks/999-cpc-fixes.chroot'
--- live-build/ubuntu-cpc/hooks/999-cpc-fixes.chroot	2017-10-10 23:57:03 +0000
+++ live-build/ubuntu-cpc/hooks/999-cpc-fixes.chroot	2017-10-19 23:34:00 +0000
@@ -136,6 +136,27 @@
 	rm "${rootd}/boot/initrd.img"
 fi
 
+if [ "${SUBPROJECT:-}" = minimized ]; then
+	# Remove various packages that we don't want in the minimized images.
+	# Some of these are tools that don't make sense by default
+	# non-interactively; some are libraries whose reverse-dependencies
+	# will have already been removed; open-vm-tools, it's a bug that this
+	# is in the common cloud seed because this should only be included
+	# in VMWare guest images, and we know none of the minimized images
+	# are targeted at VMWare.
+	_xchroot "${rootd}" env DEBIAN_FRONTEND=noninteractive \
+		apt-mark auto '^lib.*' '^python*' vim-runtime 2>/dev/null
+	# FIXME: iso-codes is a dep of software-properties and shouldn't be
+	_xchroot "${rootd}" env DEBIAN_FRONTEND=noninteractive \
+		apt-get -y autoremove --purge iso-codes xauth pastebinit \
+		plymouth open-vm-tools git shared-mime-info vim vim-common \
+		console-setup ncurses-term tmux screen policykit-1 \
+		xdg-user-dirs less publicsuffix run-one apport-symptoms \
+		ubuntu-cloudimage-keyring file
+
+	_xchroot "${rootd}" apt clean
+fi
+
 #### END COMMON ARCH FUNCTIONS