Merge lp:~laney/livecd-rootfs/snap-seeding into lp:livecd-rootfs

[Iain Lane]

This goes along with

  https://code.launchpad.net/~laney/germinate/+git/germinate/+merge/334039

and relies on its output format.

Please review, but I guess we should merge this after germinate.

Note that we run germinate for all builds now. It only takes a few seconds and it's easier to get its output into the logs if we do it this way, but if that's unacceptable we can make it happen on the first use of inheritance().

[Colin Watson (cjwatson)]

This basically seems OK, but as I mentioned in the germinate MP I think it's worth thinking about what we're going to do about classic snaps before landing any of this.

review: Approve

[lp:~laney/livecd-rootfs/snap-seeding]

1586. By Iain Lane on 2018-01-04

Support seeding of snaps

We run germinate during configure, and read from each resulting <seed>.snaps
file (expanded from STRUCTURE) to find out which snaps to seed. This is written
to config/seeded-snaps, which is read during build to download each one.

[Iain Lane (laney)]

Pushed an update to deal with "(classic)" and output this as "classic: true" in the corresponding seed.yaml stanza - please re-review.

[Colin Watson (cjwatson)]

Looks like it should do for now. Thanks.

review: Approve

=== modified file 'live-build/auto/build'
--- live-build/auto/build	2017-12-11 12:22:31 +0000
+++ live-build/auto/build	2018-01-04 12:13:16 +0000
@@ -42,7 +42,7 @@
 		chroot chroot sh -c "
 			set -x;
 			cd /var/lib/snapd/seed;
-			SNAPPY_STORE_NO_CDN=1 snap download \"$snap\""
+			SNAPPY_STORE_NO_CDN=1 snap download \"${snap%/*}\""
 	done
 	lb chroot_resolv remove
 
@@ -55,12 +55,14 @@
    file: ${CORE_SNAP}
 EOF
 	for snap in "$@"; do
+		snap_name=${snap%/*}
 		cat <<EOF >> chroot/var/lib/snapd/seed/seed.yaml
- - name: $snap
+ - name: ${snap_name}
    channel: stable
 EOF
+		case ${snap} in */classic) echo "   classic: true" >> chroot/var/lib/snapd/seed/seed.yaml;; esac
 		echo -n "   file: " >> chroot/var/lib/snapd/seed/seed.yaml
-		(cd chroot/var/lib/snapd/seed; ls -1 ${snap}_*.snap) \
+		(cd chroot/var/lib/snapd/seed; ls -1 ${snap_name}_*.snap) \
 			>> chroot/var/lib/snapd/seed/seed.yaml
 	done
 
@@ -367,40 +369,33 @@
 		rm chroot/etc/apt/sources.list.preinstall chroot/etc/apt/sources.list.orig
 	fi
 	case $PROJECT:$SUBPROJECT in
-		ubuntu-server:live|ubuntu-mate:*)
-			assertions_dir="chroot/var/lib/snapd/seed/assertions"
-			model_assertion="$assertions_dir/generic-classic.model"
-			account_key_assertion="$assertions_dir/generic.account-key"
-			account_assertion="$assertions_dir/generic.account"
-
-			mkdir -p "$assertions_dir"
-			snap known --remote model series=16 \
-				model=generic-classic brand-id=generic \
-				> "$model_assertion"
-			account_key=$(sed -n -e's/sign-key-sha3-384: //p' \
-				< "$model_assertion")
-
-			snap known --remote account-key \
-				public-key-sha3-384="$account_key" \
-				> "$account_key_assertion"
-			account=$(sed -n -e's/account-id: //p' \
-				< "$account_key_assertion")
-
-			snap known --remote account account-id=generic \
-				> "$account_assertion"
-			;;
-	esac
-
-	# FIXME: this should not be a hard-coded list in this script; this
-	# is an interim solution for 17.10.
-	case $PROJECT:$SUBPROJECT in
-		ubuntu-mate:*)
-			SNAP_LIST=pulsemixer
-			;;
-	esac
-	if [ -n "$SNAP_LIST" ]; then
-		preinstall_snaps $SNAP_LIST
-	fi
+		*)
+			if [ -e "config/seeded-snaps" ]; then
+				assertions_dir="chroot/var/lib/snapd/seed/assertions"
+				model_assertion="$assertions_dir/generic-classic.model"
+				account_key_assertion="$assertions_dir/generic.account-key"
+				account_assertion="$assertions_dir/generic.account"
+
+				mkdir -p "$assertions_dir"
+				snap known --remote model series=16 \
+					model=generic-classic brand-id=generic \
+					> "$model_assertion"
+				account_key=$(sed -n -e's/sign-key-sha3-384: //p' \
+					< "$model_assertion")
+
+				snap known --remote account-key \
+					public-key-sha3-384="$account_key" \
+					> "$account_key_assertion"
+				account=$(sed -n -e's/account-id: //p' \
+					< "$account_key_assertion")
+
+				snap known --remote account account-id=generic \
+					> "$account_assertion"
+				snap_list=$(cat config/seeded-snaps)
+				preinstall_snaps $snap_list
+			fi
+			;;
+	esac
 
 	if [ "$PROJECT" = "ubuntu-touch" ] || [ "$PROJECT" = "ubuntu-touch-custom" ]; then
 		if [ "$ARCH" = "armhf" ]; then
 
=== modified file 'live-build/auto/config'
--- live-build/auto/config	2017-11-23 19:27:16 +0000
+++ live-build/auto/config	2018-01-04 12:13:16 +0000
@@ -254,13 +254,76 @@
 	OPTS="${OPTS:+$OPTS }--bootstrap-flavour=minimal --linux-packages=linux-image"
 fi
 
+# cribbed from cdimage, perhaps this should be a small helper script in germinate?
+add_inheritance () {
+	case " $inherit " in
+		*" $1 "*)
+			;;
+		*)
+			inherit="${inherit:+$inherit }$1"
+			;;
+	esac
+}
+
+expand_inheritance () {
+	for seed in $(grep "^$1:" config/germinate-output/structure | cut -d: -f2); do
+		expand_inheritance "$seed"
+	done
+	add_inheritance "$1"
+}
+
+inheritance () {
+	inherit=
+	expand_inheritance "$1"
+	echo "$inherit"
+}
+
+mkdir -p config/germinate-output
+case $PROJECT in
+        kubuntu-active*)
+                SEED=kubuntu-active.$SUITE
+                ;;
+        kubuntu*)
+                SEED=kubuntu.$SUITE
+                ;;
+        xubuntu*)
+                SEED=xubuntu.$SUITE
+                ;;
+        *)
+                SEED=ubuntu.$SUITE
+                ;;
+esac
+
+if ! [ -e config/germinate-output/structure ]; then
+        echo "Running germinate..."
+        (cd config/germinate-output && germinate --no-rdepends --no-installer \
+                -S $SEEDMIRROR -m $MIRROR -d $SUITE -s $SEED \
+                ${COMPONENTS:+-c "$COMPONENTS"} -a $ARCH)
+fi
+
 case $PROJECT in
 	ubuntu|ubuntu-dvd)
 		add_task install minimal standard ubuntu-desktop
 		LIVE_TASK='ubuntu-live'
+		BASE_SEED='desktop'
+		SEEDS_EXPANDED=$(inheritance ${BASE_SEED})
 		case $ARCH in
 			amd64)	add_package live $SIGNED_KERNEL_PACKAGE ;;
 		esac
+		for seed in ${SEEDS_EXPANDED}; do
+			echo "snap: considering ${seed}"
+			file=config/germinate-output/${seed}.snaps
+			[ -e "${file}" ] || continue
+			# extract the first column (snap package name) from germinate's output
+			# translate the human-readable "foo (classic)" into a
+			# more machine readable "foo/classic"
+			seed_snaps=$(sed -rn '1,/-----/d;/-----/,$d; s/(.*) \|.*/\1/; s, \(classic\),/classic,; p' "${file}")
+			for snap in ${seed_snaps}; do
+				echo "snap: found ${snap}"
+				ALL_SNAPS="${ALL_SNAPS:+${ALL_SNAPS} }${snap}"
+			done
+		done
+		echo "${ALL_SNAPS}" > config/seeded-snaps
 		;;
 
 	ubuntu-desktop-next)
@@ -939,51 +1002,8 @@
 	;;
 esac
 
-# cribbed from cdimage, perhaps this should be a small helper script in germinate?
-add_inheritance () {
-	case " $inherit " in
-		*" $1 "*)
-			;;
-		*)
-			inherit="${inherit:+$inherit }$1"
-			;;
-	esac
-}
-
-expand_inheritance () {
-	for seed in $(grep "^$1:" config/germinate-output/structure | cut -d: -f2); do
-		expand_inheritance "$seed"
-	done
-	add_inheritance "$1"
-}
-
-inheritance () {
-	inherit=
-	expand_inheritance "$1"
-	echo "$inherit"
-}
-
 if [ "$PREINSTALLED" = "true" ]; then
 	if [ -n "$PREINSTALL_POOL_SEEDS" ]; then
-		mkdir -p config/germinate-output
-		case $PROJECT in
-			kubuntu-active*)
-				SEED=kubuntu-active.$SUITE
-				;;
-			kubuntu*)
-				SEED=kubuntu.$SUITE
-				;;
-			xubuntu*)
-				SEED=xubuntu.$SUITE
-				;;
-			*)
-				SEED=ubuntu.$SUITE
-				;;
-		esac
-		(cd config/germinate-output && germinate --no-rdepends --no-installer \
-			-S $SEEDMIRROR -m $MIRROR -d $SUITE -s $SEED \
-			${COMPONENTS:+-c "$COMPONENTS"} -a $ARCH)
-
 		UNWANTED_SEEDS="${LIVE_TASK:+$LIVE_TASK }boot installer required"
 		for i in $UNWANTED_SEEDS; do
 			UNWANTED_SEEDS="${UNWANTED_SEEDS:+$UNWANTED_SEEDS }$(inheritance $i)"