Merge ~smoser/cloud-initramfs-tools:rooturl-fsimage into cloud-initramfs-tools:master

[Scott Moser]

mainly a work in progress.

[Dimitri John Ledkov (xnox)]

http://manpages.ubuntu.com/manpages/focal/en/man7/initramfs-tools.7.html

        loop= "<path to image>"
              path within the original root file system to loop-mount and use
              as the real root file system.

        loopfstype
              set the loop file system type, if applicable.

        loopflags
              set the loop file system mount option string, if applicable.

surely you simply want:

root=LABEL=my-images loop=my-squashfs.img

which should already work with standard initramfs-tools on ubuntu/debian?! for simple cases

In terms of layered images, casper already supports just randomly looking for a bunch of squashfs and assembling a multi-lowerdir squashfs mount as the root filesystem, either readonly or with an overlayfs as the top level. See all the layered images work that is available in livecd-rootfs / casper, but is not currently used by any production images. We do produce ubuntu desktop canary images that can boot to any language, minimal, or full desktop depending on which squashfs layers are used.

I'm not sure i want to reinvent the wheel and duplicate all of that work over again in cloud-initramfs-tools project when close enough stuff already exists in initramfs-tools & casper.

Casper has gained ability to setup vlans, static/dhcp networks, download images/isos over the network, and stack them. (used by subiquity). In many ways, there is a lot of feature parity in casper these days.

[Ryan Harper (raharper)]

If we're touching this, I'd like to see if we can retain a path to the original squashfs file we download. For curtin, in particular, we can specify a squash:/path/to/image.squashfs and curtin can use unsquashfs to unpack the squash to $target which runs in parallel; for some systems with fast storage/cpu this can be much faster than the rsync copy.

https://code.launchpad.net/~raharper/curtin/+git/curtin/+merge/383814

[Scott Moser (smoser)]

@xnox,
thanks for the pointers to loop= .
that likely would suffice for me.

I personally find the root= syntax cleaner, but having it "already done" is nice.

It'd be nice if casper's fsimage-layered support was added to initramfs-tools, or some other less "installer specific" path.

Maybe I'll look at moving it from casper to initramfs-tools.

Unmerged commits

c64be4d... by Scott Moser on 2020-05-12

rooturl support for "nested" mounts.

This adds support for "nested" mounts to rooturl.

A nested mount is in simplist for something like:

 root=LABEL=my-images:file=my-squashfs.img

Which instructs the initramfs to mount a filesystem with the
label 'my-images' and then loop mount a 'my-squashfs.img' inside it.
A live cd basically contains an image on a cdrom.

The goal of this code is to mimic that sort of behavior but more
generically.

 # Example:

  qemu-img create -f raw 10G disk.img
  mkfs.ext2 -L images disk.img
  mount -o loop disk.img /mnt
  wget -O /mnt/bionic.img \
    http://images.maas.io/ephemeral-v3/daily/bionic/amd64/20200507/squashfs
  umount /mnt

Then boot that with a kernel command line like:

   root=img:LABEL=images:bionic.img

This actually does work right now, booting with something like:

  cmdline="console=ttyS0 root=img:LABEL=images:file=squashfs overlayroot=tmpfs"
  qemu-system-x86_64 -enable-kvm \
    -device virtio-scsi-pci,id=virtio-scsi-xkvm \
    -object rng-random,filename=/dev/urandom,id=objrng0 \
    -device virtio-rng-pci,rng=objrng0,id=rng0 \
    -device virtio-net-pci,netdev=net00 \
    -netdev type=user,id=net00 \
    -drive file=disk.img,id=disk00,if=none,format=raw,index=0 \
    -device virtio-blk,drive=disk00,serial=disk.img \
    -m 768 -serial mon:stdio \
    -kernel boot-kernel \
    -initrd boot-initrd.new \
    -append "$cmdline"

 # TODO
 ## handle ROOT_FLAGS in the final mount

 ## copy to tmpfs and unmount
I'd like to have some syntax which says copy the file to a tmpfs, and
unmount under it.

So that something like:
   LABEL=images:tmpfile=my-squash.img

would do:
   mount images by label
   copy my-squash to /initramfs/tmp-images/my-squash.img
   mount /initramfs/tmp-images/my-squash.img
   umount /dev/disk/by-label/images

Then we'd be all in memory and a cdrom could be ejected.

 ## add support for 'fsimage-layered' support in curtin
I'd like to be able to use the un-modified squashfs images from
lxd (or maas) and overlay some changes on top, such as to
/etc/passwd or maybe /etc/cloud/ or something.

That would greatly ease the creation of derivative live-cds.
So that I could script something that would
 - pull kernel and initramfs
 - pull disk image / squashfs
 - use mkisofs (and grub) It would ease creation of live-cds

'fsimage-layered' as described at
https://curtin.readthedocs.io/en/latest/topics/config.html
is a mechanism that curtin uses.

diff --git a/rooturl/scripts/local-top/rooturl b/rooturl/scripts/local-top/rooturl
index 82025f8..27245fd 100755
--- a/rooturl/scripts/local-top/rooturl
+++ b/rooturl/scripts/local-top/rooturl
@@ -25,7 +25,7 @@ VERBOSITY="1"
 debug() {
 	local v=$1
 	shift
-	[ $v -lt $VERBOSITY ] && return 0
+	[ $v -gt $VERBOSITY ] && return 0
 	echo "::" "$@"
 }
 log_warn() { log_warning_msg "$*"; }
@@ -82,16 +82,72 @@ mount_url() {
 	"$helper" "$target" "$firsturl" "$@"
 }
 
+img_mount() {
+	local mntn=0 tmpd="/initramfs/mnt.d"
+	local targetmp="$1" cur="$2" fstype=""
+	local devdisk="/dev/disk" lastmntd=""
+	cur=${cur#img:}
+	[ -d "$tmpd" ] || mkdir -p "$tmpd" || return
+	[ -d "$targetmp" ] || mkdir -p "$targetmp" || return
+	while [ -n "$cur" ] && mntn=$((mntn+1)); do
+		debug 0 "mount $mntn: $cur"
+		mntd=$(printf "%s/mnt.%02d" "$tmpd" "$mntn")
+		mkdir "$mntd"
+		tok=${cur%%:*}
+		mopts=""
+		fstype=""
+		case "$tok" in
+			UUID=*|LABEL=*|PARTUUID=*|/dev/*)
+				case "$tok" in
+					LABEL=*) dev=$devdisk/by-label/${tok#*=};;
+					UUID=*) dev=$devdisk/by-uuid/${tok#*=};;
+					PARTUUID=*) dev=$devdisk/by-partuuid/${tok#*=};;
+					*) dev=$tok;;
+				esac
+				[ -b "$dev" ] || {
+					debug 0 "$dev did not exist yet (tok=$tok). waiting."
+				}
+				fstype=$( wait-for-root "$tok" 30 ) || {
+					log_warn "$tok did not appear"
+					return 1
+				}
+				;;
+			file=/*)
+				mopts="-o loop"
+				dev=${tok#*=};;
+			file=*)
+				mopts="-o loop"
+				dev=${lastmntd}/${tok#*=};;
+			*) log_warn "no idea on $tok"; return 1;;
+		esac
+		if ! [ -e "$dev" ]; then
+			udevadm settle
+		fi
+		debug 0 "want to mount $dev to $mntd mopts=$mopts fstype=$fstype"
+		mount ${fstype:+-t ${fstype}} ${mopts} "$dev" "$mntd" || {
+			log_warn "help, failed $dev -> $mntd"
+			return 1
+		}
+		lastmntd="$mntd"
+		cur=${cur#$tok}
+		cur=${cur#:}
+	done
+	mount -o move "$mntd" "$targetmp" || return 1
+}
+
 . /scripts/functions
 
 case "$ROOT" in
-	http://*|*:http://*) :;;
+	http://*|*:http://*)
+		configure_networking || exit;;
+	img:*) :;;
 	*) exit 0
 esac
-configure_networking || exit
 
-debug 1 "root=$ROOT"
-mount_url "${rootmnt}.tmp" "$ROOT" || exit
+case "$ROOT" in
+	http*|*:http://*) mount_url "${rootmnt}.tmp" "$ROOT" || exit;;
+	img:*) img_mount "${rootmnt}.tmp" "$ROOT" || exit;;
+esac
 
 {
 	echo 'ROOTFSTYPE="root_url"'