Merge lp:~jibel/livecd-rootfs/add_multi_layered_squashfses_support into lp:livecd-rootfs

[Jean-Baptiste Lallement]

Adds support for multi layer filesystem by generating one squashfs per layer and adding a new 'live-layered' image format.
squashfs are numbered so the order is preserved and mounted in order by casper.

Seeding of snap packages and seeds corresponding to the layers are not
available yet and will be proposed in the subsequent merge proposal.

[Dimitri John Ledkov (xnox)]

In passes, how does one specify multiple branches of stacks? Or is only a linear stack allowed by this code?

Cause the current subiquity images do not have `live` as the top of the stack.... They do this:

+--> Base +-----> Live
          |
          +-----> Rack +---> Region

Because I am expecting to have the ability to somehow specify what each pass depends on. But it looks like the base for a subsequent pass, is always the previous one?!

It would be nice to extend PASS syntax to optionally accept an arbitrary `base`, e.g.

PASSES="base rack region base:live" to encode the above graph, and such that `base:live` pass uses base as the lowerdir, instead of region (i.e. the previous pass result).

Or something...

not sure if : is acceptable pair delimiter here, or not.

Also not sure if we want to always enforce specifying `base` such that we can construct multiple root nodes in one go. E.g. `base base:rack rack:region base:live` or `:base rack region base:live` for the above graph.

Or like maybe always list all layers?! but that violates donot-repeat-yourself principle... E.g. `base base:rack base:rack:region base:live`

===

When calling includes/hooks are they aware which pass they are for? what is the source dir for each pass, for e.g. binary.includes? I'm guessing that PASS variable is set, but not sure.

===

filesystem.squashfs is somewhat is a special name, so it would be nice to keep that as the base one. And also possibly adjust logic in casper as to what it mounts by default.... cause e.g. i think i hide maas squashfes in a subdir, to prevent casper from mounting those, which is kind of a hack. I wonder if we do need to write out the valid stacks (passes?!), which casper can then use to boot to whichever stack is valid. With subiquity image this could then result in "Live Server, Live MAAS Rack, Live MAAS Region, Live Server with Installer" boot options. As example, for better or worse.

====

No idea if SUBPROJECT and IMAGE_FORMAT are the right things to extend for this..... and if they are easily extendable like this in launchpad livefs builders & ubuntu-cdimage codes.

Cause I can see the potential for using layers in SUBPROJECT=minimized, if for example, cpc builds are converted to layers they would have full and minimized layered builds.... and SUBPROJECT=minimized-layered sounds ugly =)

====

manifest diffs for layers is nice; cause in cpc we have struggled to consitently represent manifests / changelogs of "it's just like that image, but has this stuff on it"

====

packaging layers as actual static filesystems might be interesting, but i guess hooks will be able to do that anyway.

====

Overall, this looks ok, and shouldn't break any existing stuff - as long as we can clear the top level new extensions of

   SUBPROJECT=layered
   IMAGEFORMAT=live-layered

and that needs like an architect review.

[Jean-Baptiste Lallement (jibel)]

Thanks for the review

On 14/11/2018 00:35, Dimitri John Ledkov wrote:
>
> In passes, how does one specify multiple branches of stacks? Or is only a linear stack allowed by this code?

This code only allows linear stacks to avoid not make current code more
complex than it already is and diverging too much from existing logic.
Our approach is to define the stacking defining the live image in PASSES
and put the extra logic like “branched mounts” in hooks.

For instance with langpacks which is the use case for Desktop image, we
also have multiple branches of stacks. It is difficult to represent the
structure in a generic way and it adds extra complexity to the
maintenance of those definitions.

Following your proposal to represent the layers structure as a list, it
would be something like:
PASSES=“desktop-minimal:desktop:live desktop-minimal:lang-neg-min-fr
desktop-minimal:lang-neg-min-de desktop-minimal:desktop:lang-neg-fr
desktop-minimal:desktop:lang-neg-de …”

We thus have multiple “base” here, “desktop-minimal” or
“desktop-minimal:desktop”. Shortening the syntax doesn’t seem
appropriate here.

In a tree, it’s not better:
- desktop-min
   - desktop
     - Lang-neg-de
     - Lang-neg-fr
     - Lang-neg-es
     - …
     - Live
   - Lang-neg-min-de
   - Lang-neg-min-fr
   - Lang-neg-min-es
   - …

Addition or removal of any default language would be then error-prone.
This is to compare with a hook, where we just loop over “desktop-min”
and creates langpacks negative stacks, and then looping over “desktop”
as well to achieve the same. Besides, shell seems inappropriate to
implement this type of logic.

Note that the current implementation is similar than existing
ubuntu-server:live logic (just a little bit more generic) and we made
sure we didn’t break your use case. Note that though, you will be able
to remove your first hook, creating the “live” stack using the generic
code right now. However, the maas-* stacks will still be in hooks.

>
> Cause the current subiquity images do not have `live` as the top of the stack.... They do this:
>
> +--> Base +-----> Live
> |
> +-----> Rack +---> Region
>
> Because I am expecting to have the ability to somehow specify what each pass depends on. But it looks like the base for a subsequent pass, is always the previous one?!
>
>
> It would be nice to extend PASS syntax to optionally accept an arbitrary `base`, e.g.
>
> PASSES="base rack region base:live" to encode the above graph, and such that `base:live` pass uses base as the lowerdir, instead of region (i.e. the previous pass result).
>
> Or something...
>
> not sure if : is acceptable pair delimiter here, or not.
>
> Also not sure if we want to always enforce specifying `base` such that we can construct multiple root nodes in one go. E.g. `base base:rack rack:region base:live` or `:base rack region base:live` for the above graph.
>
> Or like maybe always list all layers?! but that violates donot-repeat-yourself principle... E.g. `base base:rack base:rack:region base:live`
>
> ===
>
> When calling includes/hooks are they aware which pass they are for? what is the source dir for each pass, for e.g. bina...

[Steve Langasek (vorlon)]

Thanks for working on this, it will be nice to have cleaner handling of the desktop-minimal stuff included here in livecd-rootfs.

I would like to see this converged with the ubuntu-server-live handling as part of landing of this branch (see also Dimitri's review comments). You've gone to some length to implement this in a way that's generalizable across projects, so we ought to make sure the implementation is actually reusable by the only other image that currently uses layered squashfs.

review: Needs Fixing

[Jean-Baptiste Lallement (jibel)]

We intended to propose several MP for snap and “sub-layers” to reduce the complexity of the review, but we’ll finally merge them into this one so you’ll have a good understanding of the full implementation.

Supporting the server-live use case adds makes the implementation more complex due to its specific requirements (hooks and includes). But we agree that a more generalizable implementation is better, so let’s spend some time to ensure those 2 additional requirements are supported, and once foundation/server team want to switch to it, that they will not encounter any major limitations.

[Jean-Baptiste Lallement (jibel)]

Following previous comments, this MP has been updated with these changes:
- Merged preseeding of snap packages and adds snap packages to the image manifest.
- Added lb_binary_layered to handle multi branches of squashfses with sublayers.
- Factorized several helpers called by lb_chroot_layered
- Removed obsolete chroot helpers from lb_chroot_layered and minor cleanup.
- Renamed subproject layered -> ubiquity-ng
- Use PASSES instead of IMAGEFORMAT to detect a multilayer project. Defining PASSES switches the image build into layered build.
- Adds includes by pass to customize chroot for specific passes.
- Moved back specific helpers and functions from functions to config.
- For Ubuntu Desktop:ubiquity-ng, build negative language packs and corresponding squashfs.

These changes have been tested against ubuntu-server live which still uses its current implementation (with maas-region and maas-rack squashfses built in hooks).
All the requirements of ubuntu-server:live can be ported from the current hooks to the new implementation of the layered images.
To do so:
- The project have to define PASSES.
- Snaps and packages for sublayers added to separated seeds and referenced in auto/config.
- Layers must be customized with chroot hooks (eg to modify existing files) and chroot includes (eg to add new files) depending on PASS.

Ubuntu Desktop (standard disco) has been modified to use layered images as requested. As a consequence the following MPs must be reviewed together with this one and released before uploading livecd-rootfs:
- debian-cd: https://code.launchpad.net/~jibel/debian-cd/support_for_multilayer_images/+merge/359228
- ubuntu-cdimage: https://code.launchpad.net/~jibel/ubuntu-cdimage/support_for_multilayer/+merge/359512

The subproject ubiquity-ng must be created for building the ubuntu-desktop:ubiquity-ng rootfs and match https://launchpad.net/~ubuntu-cdimage/+livefs/ubuntu/disco/ubuntu-desktop-ubiquity-ng/
This whole set of changes (livecd-rootfs, ubuntu-cdimage and debian-cd) have been tested with the following projects:
- Ubuntu Desktop layered (iso) (disco) -> test new set of layers
- Ubuntu Desktop unlayered (iso) (disco) -> test previous image format (one squashfs)
- Ubuntu Desktop ubiquity-ng (iso) -> test new set of layers + sublayers
- Ubuntu Server live (iso) (bionic and disco) -> test backward compatibility with current server image and hooks impacts.
- Lubuntu (rootfs only, no local archive with universe for building iso, similar build to ubuntu-desktop/unlayered) (disco) -> test traditional one squashfs image
- Ubuntu Mate (rootfs only, no local archive with universe for building iso, similar build to ubuntu-desktop/unlayered) (bionic) -> test traditional one squashfs image + snaps

[Adam Conrad (adconrad)]

I don't have time for a full review this morning, but unless we intend to keep these forked from the "real" ISOs for testing for a while, and then converge, I really don't like the SUBPROJECT use here. I'd expect the real 'ubuntu' project to be building these, not some ubuntu-random-subproject project.

As noted, you already can decide if you're building this type of image based on PASSES being defined, so it seems a bit odd to then also key on a subproject name.

[lp:~jibel/livecd-rootfs/add_multi_layered_squashfses_support]

1758. By Jean-Baptiste Lallement on 2018-11-28

configure network manager _after_ installing network-manager

[Jean-Baptiste Lallement (jibel)]

> I don't have time for a full review this morning, but unless we intend to keep
> these forked from the "real" ISOs for testing for a while, and then converge,
> I really don't like the SUBPROJECT use here. I'd expect the real 'ubuntu'
> project to be building these, not some ubuntu-random-subproject project.
>
> As noted, you already can decide if you're building this type of image based
> on PASSES being defined, so it seems a bit odd to then also key on a
> subproject name.
We need a subproject because the real 'ubuntu' and the new installer work will diverge quickly. Note that Ubuntu Desktop uses the layer system on Disco with this merge proposal without defining a subproject.

[Balint Reczey (rbalint)]

The target Bazaar branch is not active anymore.
Please resubmit the merge proposal against https://code.launchpad.net/~ubuntu-core-dev/livecd-rootfs/+git/livecd-rootfs/+ref/ubuntu/master .

[Steve Langasek (vorlon)]

This is done now in git.

=== modified file 'debian/tests/default-bootstraps'
--- debian/tests/default-bootstraps	2018-05-09 02:12:15 +0000
+++ debian/tests/default-bootstraps	2018-11-28 08:47:44 +0000
@@ -23,6 +23,7 @@
  lubuntu::
  mythbuntu::
  ubuntu::
+ ubuntu:ubiquity-ng:
  ubuntu-base::
  ubuntu-budgie::
  ubuntu-budgie-desktop::
 
=== modified file 'live-build/auto/build'
--- live-build/auto/build	2018-11-26 12:03:24 +0000
+++ live-build/auto/build	2018-11-28 08:47:44 +0000
@@ -252,107 +252,57 @@
 dpkg-divert --quiet --remove --rename /usr/sbin/update-initramfs
 EOF
 
-	lb chroot "$@"
-
-	if [ "${SUBPROJECT:-}" = minimized ]; then
-		# force removal of initramfs-tools, which we assert is not
-		# required for any minimized images but is still pulled in by
-		# default
-		# also remove landscape-common, which is heavyweight and
-		# in the server seed only to provide /etc/motd content which
-		# would only be seen by humans
-		Chroot chroot "env DEBIAN_FRONTEND=noninteractive \
-			apt-get -y purge initramfs-tools busybox-initramfs \
-				busybox-static landscape-common"
-		# and if initramfs-tools was configured before our kernel,
-		# /etc/kernel/postinst.d/initramfs-tools will have created
-		# an initramfs despite the generic dpkg-divert; so remove it
-		# here.
-		rm -f chroot/boot/initrd.img-*
-
-		# temporary workaround: don't remove linux-base which
-		# may have no other reverse-depends currently
-		Chroot chroot "env DEBIAN_FRONTEND=noninteractive \
-			apt-mark manual linux-base"
-		Chroot chroot "env DEBIAN_FRONTEND=noninteractive \
-			apt-get -y --purge autoremove"
-	fi
-
-	# remove crufty files that shouldn't be left in an image
-	rm -f chroot/var/cache/debconf/*-old chroot/var/lib/dpkg/*-old
-	Chroot chroot apt clean
-	if [ "${PROJECT}:${SUBPROJECT:-}" = "ubuntu-base:minimized" ]; then
-		# Save even more size by removing apt lists (that are currently removed
-		# downstream anyway)
-		rm -rf chroot/var/lib/apt/lists/*
-		# Having device notes in the docker image can cause problems
-		# (https://github.com/tianon/docker-brew-ubuntu-core/issues/62)
-		# so remove them.  We only do this for docker out of an
-		# abundance of caution.
-		rm -rf chroot/dev/*
-	fi
-
-	if [ -f config/universe-enabled ]; then
-
-# This is cargo-culted almost verbatim (with some syntax changes for
-# preinstalled being slightly different in what it doesn't ask) from
-# debian-installer's apt-setup:
-
-cat > chroot/etc/apt/sources.list << EOF
-# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
-# newer versions of the distribution.
-deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION main restricted
-# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION main restricted
-
-## Major bug fix updates produced after the final release of the
-## distribution.
-deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates main restricted
-# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates main restricted
-
-## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
-## team. Also, please note that software in universe WILL NOT receive any
-## review or updates from the Ubuntu security team.
-deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION universe
-# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION universe
-deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates universe
-# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates universe
-
-## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
-## team, and may not be under a free licence. Please satisfy yourself as to
-## your rights to use the software. Also, please note that software in
-## multiverse WILL NOT receive any review or updates from the Ubuntu
-## security team.
-deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION multiverse
-# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION multiverse
-deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates multiverse
-# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates multiverse
-
-## N.B. software from this repository may not have been tested as
-## extensively as that contained in the main release, although it includes
-## newer versions of some applications which may provide useful features.
-## Also, please note that software in backports WILL NOT receive any review
-## or updates from the Ubuntu security team.
-deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-backports main restricted universe multiverse
-# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-backports main restricted universe multiverse
-
-## Uncomment the following two lines to add software from Canonical's
-## 'partner' repository.
-## This software is not part of Ubuntu, but is offered by Canonical and the
-## respective vendors as a service to Ubuntu users.
-# deb http://archive.canonical.com/ubuntu $LB_DISTRIBUTION partner
-# deb-src http://archive.canonical.com/ubuntu $LB_DISTRIBUTION partner
-
-deb $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security main restricted
-# deb-src $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security main restricted
-deb $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security universe
-# deb-src $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security universe
-deb $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security multiverse
-# deb-src $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security multiverse
-EOF
-
-	fi
-	if [ -d chroot/var/lib/preinstalled-pool ]; then
-		cat > config/indices/apt.conf <<-EOF
+	if [ -n "${PASSES}" ]; then
+		PATH="config/:$PATH" lb chroot_layered "$@"
+	else
+		lb chroot "$@"
+	fi
+
+	# Let all configuration non multi-layered project here.
+	# If those are moving to a multi-layer layout, this needs to be
+	# done in chroot hooks.
+	if [ -z "$PASSES" ]; then
+		if [ "${SUBPROJECT:-}" = minimized ]; then
+			# force removal of initramfs-tools, which we assert is not
+			# required for any minimized images but is still pulled in by
+			# default
+			# also remove landscape-common, which is heavyweight and
+			# in the server seed only to provide /etc/motd content which
+			# would only be seen by humans
+			Chroot chroot "env DEBIAN_FRONTEND=noninteractive \
+				apt-get -y purge initramfs-tools busybox-initramfs \
+					busybox-static landscape-common"
+			# and if initramfs-tools was configured before our kernel,
+			# /etc/kernel/postinst.d/initramfs-tools will have created
+			# an initramfs despite the generic dpkg-divert; so remove it
+			# here.
+			rm -f chroot/boot/initrd.img-*
+
+			# temporary workaround: don't remove linux-base which
+			# may have no other reverse-depends currently
+			Chroot chroot "env DEBIAN_FRONTEND=noninteractive \
+				apt-mark manual linux-base"
+			Chroot chroot "env DEBIAN_FRONTEND=noninteractive \
+				apt-get -y --purge autoremove"
+		fi
+
+		clean_debian_chroot
+
+		if [ "${PROJECT}:${SUBPROJECT:-}" = "ubuntu-base:minimized" ]; then
+			# Save even more size by removing apt lists (that are currently removed
+			# downstream anyway)
+			rm -rf chroot/var/lib/apt/lists/*
+			# Having device notes in the docker image can cause problems
+			# (https://github.com/tianon/docker-brew-ubuntu-core/issues/62)
+			# so remove them.  We only do this for docker out of an
+			# abundance of caution.
+			rm -rf chroot/dev/*
+		fi
+
+		configure_universe
+
+		if [ -d chroot/var/lib/preinstalled-pool ]; then
+			cat > config/indices/apt.conf <<-EOF
 Dir {
  ArchiveDir "chroot/var/lib/preinstalled-pool";
  OverrideDir "config/indices";
@@ -369,11 +319,11 @@
  Contents " ";
 }
 EOF
-		for component in $LB_PARENT_ARCHIVE_AREAS; do
-			mkdir -p chroot/var/lib/preinstalled-pool/dists/$LB_DISTRIBUTION/$component/binary-$LB_ARCHITECTURES
-		done
-		apt-ftparchive generate config/indices/apt.conf
-		cat << @@EOF > chroot/etc/apt/sources.list.preinstall
+			for component in $LB_PARENT_ARCHIVE_AREAS; do
+				mkdir -p chroot/var/lib/preinstalled-pool/dists/$LB_DISTRIBUTION/$component/binary-$LB_ARCHITECTURES
+			done
+			apt-ftparchive generate config/indices/apt.conf
+			cat << @@EOF > chroot/etc/apt/sources.list.preinstall
 # This is a sources.list entry for a small pool of packages
 # provided on your preinstalled filesystem for your convenience.
 #
@@ -385,93 +335,89 @@
 #
 @@EOF
 
-		cp chroot/etc/apt/sources.list chroot/etc/apt/sources.list.orig
-		cp chroot/etc/apt/sources.list.preinstall chroot/etc/apt/sources.list
-
-		echo "Waiting on gnupg ("$GPG_PROCESS") to finish generating a key."
-		wait $GPG_PROCESS
-
-		R_ORIGIN=$(lsb_release -i -s)
-		R_CODENAME=$(lsb_release -c -s)
-		R_VERSION=$(lsb_release -r -s)
-		R_PRETTYNAME=$(echo $R_CODENAME | sed -e 's/^\(.\)/\U\1/')
-
-		apt-ftparchive -o APT::FTPArchive::Release::Origin=$R_ORIGIN \
-			-o APT::FTPArchive::Release::Label=$R_ORIGIN \
-			-o APT::FTPArchive::Release::Suite=$R_CODENAME-local \
-			-o APT::FTPArchive::Release::Version=$R_VERSION \
-			-o APT::FTPArchive::Release::Codename=$R_CODENAME \
-			-o APT::FTPArchive::Release::Description="$R_ORIGIN $R_PRETTYNAME Local" \
-			release chroot/var/lib/preinstalled-pool/dists/$R_CODENAME/ \
-				> config/gnupg/Release
-
-		gpg --home config/gnupg --detach-sign --armor config/gnupg/Release
-		mv config/gnupg/Release \
-			chroot/var/lib/preinstalled-pool/dists/$R_CODENAME/Release
-		mv config/gnupg/Release.asc \
-			chroot/var/lib/preinstalled-pool/dists/$R_CODENAME/Release.gpg
-		apt-key --keyring chroot/etc/apt/trusted.gpg add config/gnupg/pubring.gpg
-		find chroot/var/lib/preinstalled-pool/ -name Packages | xargs rm
-
-		Chroot chroot "apt-get update"
-		cat chroot/etc/apt/sources.list.preinstall chroot/etc/apt/sources.list.orig \
-			> chroot/etc/apt/sources.list
-		rm chroot/etc/apt/sources.list.preinstall chroot/etc/apt/sources.list.orig
-	fi
-	case $PROJECT:$SUBPROJECT in
-		*)
-			if [ -e "config/seeded-snaps" ]; then
-				snap_list=$(cat config/seeded-snaps)
-				preinstall_snaps $snap_list
-			fi
-			;;
-	esac
-
-	if [ "$PROJECT" = "ubuntu-touch" ] || [ "$PROJECT" = "ubuntu-touch-custom" ]; then
-		if [ "$ARCH" = "armhf" ]; then
-			INFO_DESC="$(lsb_release -d -s)"
-			echo "$INFO_DESC - $ARCH ($BUILDSTAMP)" >chroot/etc/media-info
-			mkdir -p chroot/var/log/installer
-			Chroot chroot "ln -s /etc/media-info /var/log/installer/media-info"
-		fi
-	fi
-	if [ "$PROJECT" = "ubuntu-cpc" ]; then
-		if [ "${SUBPROJECT:-}" = minimized ]; then
-			BUILD_NAME=minimal
-		else
-			BUILD_NAME=server
-		fi
-		cat > chroot/etc/cloud/build.info << EOF
+			cp chroot/etc/apt/sources.list chroot/etc/apt/sources.list.orig
+			cp chroot/etc/apt/sources.list.preinstall chroot/etc/apt/sources.list
+
+			echo "Waiting on gnupg ("$GPG_PROCESS") to finish generating a key."
+			wait $GPG_PROCESS
+
+			R_ORIGIN=$(lsb_release -i -s)
+			R_CODENAME=$(lsb_release -c -s)
+			R_VERSION=$(lsb_release -r -s)
+			R_PRETTYNAME=$(echo $R_CODENAME | sed -e 's/^\(.\)/\U\1/')
+
+			apt-ftparchive -o APT::FTPArchive::Release::Origin=$R_ORIGIN \
+				-o APT::FTPArchive::Release::Label=$R_ORIGIN \
+				-o APT::FTPArchive::Release::Suite=$R_CODENAME-local \
+				-o APT::FTPArchive::Release::Version=$R_VERSION \
+				-o APT::FTPArchive::Release::Codename=$R_CODENAME \
+				-o APT::FTPArchive::Release::Description="$R_ORIGIN $R_PRETTYNAME Local" \
+				release chroot/var/lib/preinstalled-pool/dists/$R_CODENAME/ \
+					> config/gnupg/Release
+
+			gpg --home config/gnupg --detach-sign --armor config/gnupg/Release
+			mv config/gnupg/Release \
+				chroot/var/lib/preinstalled-pool/dists/$R_CODENAME/Release
+			mv config/gnupg/Release.asc \
+				chroot/var/lib/preinstalled-pool/dists/$R_CODENAME/Release.gpg
+			apt-key --keyring chroot/etc/apt/trusted.gpg add config/gnupg/pubring.gpg
+			find chroot/var/lib/preinstalled-pool/ -name Packages | xargs rm
+
+			Chroot chroot "apt-get update"
+			cat chroot/etc/apt/sources.list.preinstall chroot/etc/apt/sources.list.orig \
+				> chroot/etc/apt/sources.list
+			rm chroot/etc/apt/sources.list.preinstall chroot/etc/apt/sources.list.orig
+		fi
+		case $PROJECT:$SUBPROJECT in
+			ubuntu-server:live)
+				lb chroot_resolv install
+				snap_prepare chroot
+				lb chroot_resolv remove
+				;;
+			*)
+				if [ -e "config/seeded-snaps" ]; then
+					snap_list=$(cat config/seeded-snaps)
+					preinstall_snaps $snap_list
+				fi
+				;;
+		esac
+
+		if [ "$PROJECT" = "ubuntu-touch" ] || [ "$PROJECT" = "ubuntu-touch-custom" ]; then
+			if [ "$ARCH" = "armhf" ]; then
+				INFO_DESC="$(lsb_release -d -s)"
+				echo "$INFO_DESC - $ARCH ($BUILDSTAMP)" >chroot/etc/media-info
+				mkdir -p chroot/var/log/installer
+				Chroot chroot "ln -s /etc/media-info /var/log/installer/media-info"
+			fi
+		fi
+		if [ "$PROJECT" = "ubuntu-cpc" ]; then
+			if [ "${SUBPROJECT:-}" = minimized ]; then
+				BUILD_NAME=minimal
+			else
+				BUILD_NAME=server
+			fi
+			cat > chroot/etc/cloud/build.info << EOF
 build_name: $BUILD_NAME
 serial: $BUILDSTAMP
 EOF
+		fi
+
+		configure_network_manager
+
+		echo "===== Checking size of /usr/share/doc ====="
+		echo BEGIN docdirs
+		(cd chroot && find usr/share/doc -maxdepth 1 -type d | xargs du -s | sort -nr)
+		echo END docdirs
+
+		/usr/share/livecd-rootfs/minimize-manual chroot
 	fi
 
-	# If the image pre-installs network-manager, let it manage all devices by
-	# default. Installing NM on an existing system only manages wifi and wwan via
-	# /usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf. When setting
-	# the global backend to NM, netplan overrides that file.
-	if [ -d chroot/usr/lib/NetworkManager ]; then
-		echo "===== Enabling all devices in NetworkManager ===="
-		mkdir -p chroot/etc/netplan
-		cat <<EOF > chroot/etc/netplan/01-network-manager-all.yaml
-# Let NetworkManager manage all devices on this system
-network:
-  version: 2
-  renderer: NetworkManager
-EOF
+	if [ -n "${PASSES}" ]; then
+		PATH="config/:$PATH" lb binary_layered "$@"
 	else
-		echo "==== NetworkManager not installed ===="
+		lb binary "$@"
 	fi
 
-	echo "===== Checking size of /usr/share/doc ====="
-	echo BEGIN docdirs
-	(cd chroot && find usr/share/doc -maxdepth 1 -type d | xargs du -s | sort -nr)
-	echo END docdirs
-
-	/usr/share/livecd-rootfs/minimize-manual chroot
-
-	lb binary "$@"
 	touch binary.success
 ) 2>&1 | tee binary.log
 
@@ -504,6 +450,12 @@
 	chmod 644 "$PREFIX.$OUTPUT"
 done
 
+# we don't need a manifest-remove for a layered-aware installer
+if [ "$SUBPROJECT" = "ubiquity-ng" ]; then
+	rm -f livecd.${PROJECT}-manifest-remove
+	rm -f config/manifest-minimal-remove
+fi
+
 if [ -e config/manifest-minimal-remove ]; then
         cp config/manifest-minimal-remove  "$PREFIX.manifest-minimal-remove"
 fi
 
=== modified file 'live-build/auto/clean'
--- live-build/auto/clean	2012-11-19 18:49:31 +0000
+++ live-build/auto/clean	2018-11-28 08:47:44 +0000
@@ -8,3 +8,5 @@
 rm -f binary.manifest binary.manifest-desktop binary.log
 rm -f livecd.*
 rm -rf userdata
+rm -rf chroot.*
+rm -rf *.manifest.full
 
=== modified file 'live-build/auto/config'
--- live-build/auto/config	2018-11-23 16:39:40 +0000
+++ live-build/auto/config	2018-11-28 08:47:44 +0000
@@ -33,15 +33,40 @@
 
 mkdir -p config
 cp -af /usr/share/livecd-rootfs/live-build/functions config/functions
+cp -af /usr/share/livecd-rootfs/live-build/lb_*_layered config/
 cp -af /usr/share/livecd-rootfs/live-build/snap-seed-parse.py config/snap-seed-parse
 
 mkdir -p config/package-lists
 
+. config/functions
+
+OPTS=
+COMPONENTS=
+BINARY_REMOVE_LINUX=:
+BINARY_IMAGES=none
+MEMTEST=none
+SOURCE='--source false'
+BOOTLOADER=none
+BOOTAPPEND_LIVE=
+LIVE_TASK=
+PREINSTALLED=false
+PREINSTALL_POOL=
+PREINSTALL_POOL_SEEDS=
+PREFIX="livecd.$PROJECT${SUBARCH:+-$SUBARCH}"
+
+CHROOT_HOOKS=
+BINARY_HOOKS=
+
+APT_OPTIONS=" --yes -oDebug::pkgDepCache::AutoInstall=yes "
+
 add_task ()
 {
 	local pass="$1"
 	shift
 	local task
+	local snap_list_file
+	local snap_list_files
+	local curseed
 
 	# The removal of direct task installation support from live-build
 	# poses some problems.  If the chroot has multiarch configured - for
@@ -58,12 +83,32 @@
 	# probably a lurking timebomb that we need to fix.  In the meantime,
 	# the Architecture restriction at least saves us from abject
 	# failure.
+	#
+	# We want as well to grab the snap list for each PASS. Resolve for all
+	# given task, and deduplicate them to generate snaps for the PASS.
 
 	for task; do
 		# We need a ridiculous number of backslashes to protect
 		# parentheses from eval.
 		echo "!chroot chroot apt-cache dumpavail | grep-dctrl -nsPackage \\\\\\( -XFArchitecture $ARCH -o -XFArchitecture all \\\\\\) -a -wFTask $task" >> "config/package-lists/livecd-rootfs.list.chroot_$pass"
+
+		curseed=$(seed_from_task ${task})
+		if [ -z "${curseed}" ]; then
+			echo "W: No seed matching task ${task}"
+			continue
+		fi
+		snap_list_file="config/package-lists/seed.${curseed}.snaplist.full"
+		snap_from_seed "${curseed}" $snap_list_file
+		if [ -e "$snap_list_file" ]; then
+			snap_list_files="${snap_list_files} $snap_list_file"
+		fi
 	done
+	# The snap list is one line, and could be duplicated between seeds via inheritance.
+	# Uniquely sort them and store them back in one line.
+	if [ -n "${snap_list_files}" ]; then
+		cat ${snap_list_files}|xargs -n1|sort -u > "config/package-lists/livecd-rootfs.snaplist.chroot_${pass}.full"
+		rm ${snap_list_files}
+	fi
 }
 
 add_package ()
@@ -77,24 +122,34 @@
 	done
 }
 
-OPTS=
-COMPONENTS=
-BINARY_REMOVE_LINUX=:
-BINARY_IMAGES=none
-MEMTEST=none
-SOURCE='--source false'
-BOOTLOADER=none
-BOOTAPPEND_LIVE=
-LIVE_TASK=
-PREINSTALLED=false
-PREINSTALL_POOL=
-PREINSTALL_POOL_SEEDS=
-PREFIX="livecd.$PROJECT${SUBARCH:+-$SUBARCH}"
-
-CHROOT_HOOKS=
-BINARY_HOOKS=
-
-APT_OPTIONS=" --yes -oDebug::pkgDepCache::AutoInstall=yes "
+add_layered_pass() {
+    # Add a layer to an existing pass based on seeds matching a regexp
+    # $1 base pass
+    # $2 seeds (regexp)
+
+    for seed in $(ls config/germinate-output/|grep -P "$2"); do
+       pass=${1}_${seed}
+       list_packages_from_seed ${seed} >> config/package-lists/livecd-rootfs.list.chroot_$pass
+    done
+}
+
+add_layered_pass_delta() {
+    # Add a layer to an existing pass based on delta between seeds matching a regexp and a base seed
+    # $1 base pass
+    # $2 base seed
+    # $3 seeds to remove from base seed (regexp). If empty, a no-<base-seed> sublayer is generated.
+
+    local seed_regexp="$3"
+    if [ -z "${seed_regexp}" ]; then
+        substract_package_lists ${2} "" >> config/package-lists/livecd-rootfs.removal-list.chroot_${1}_no-${2}
+        return
+    fi
+
+    for seed in $(ls config/germinate-output/|grep -P "$seed_regexp"); do
+       pass=${1}_${seed}
+       substract_package_lists ${2} ${seed} >> config/package-lists/livecd-rootfs.removal-list.chroot_$pass
+    done
+}
 
 add_chroot_hook ()
 {
@@ -288,30 +343,6 @@
 	OPTS="${OPTS:+$OPTS }--bootstrap-flavour=minimal --linux-packages=linux-image"
 fi
 
-# cribbed from cdimage, perhaps this should be a small helper script in germinate?
-add_inheritance () {
-	case " $inherit " in
-		*" $1 "*)
-			;;
-		*)
-			inherit="${inherit:+$inherit }$1"
-			;;
-	esac
-}
-
-expand_inheritance () {
-	for seed in $(grep "^$1:" config/germinate-output/structure | cut -d: -f2); do
-		expand_inheritance "$seed"
-	done
-	add_inheritance "$1"
-}
-
-inheritance () {
-	inherit=
-	expand_inheritance "$1"
-	echo "$inherit"
-}
-
 mkdir -p config/germinate-output
 case $PROJECT in
         kubuntu-active*)
@@ -358,8 +389,21 @@
 
 case $PROJECT in
 	ubuntu|ubuntu-dvd)
-		add_task install minimal standard ubuntu-desktop
+		PASSES="install-minimal install live"
+		add_task install-minimal minimal standard ubuntu-desktop-minimal ubuntu-desktop-minimal-default-languages
+		add_task install ubuntu-desktop ubuntu-desktop-default-languages
 		LIVE_TASK='ubuntu-live'
+
+		case ${SUBPROJECT:-} in
+			ubiquity-ng)
+				# LANG PASS for minimal and install
+				add_layered_pass_delta install-minimal desktop-minimal-default-languages '^desktop-minimal-(?!default-languages)[^.]+$'
+				add_layered_pass_delta install-minimal desktop-minimal-default-languages ''  # none (if no default langpack is selected)
+				add_layered_pass_delta install desktop-default-languages '^desktop-(?!default-languages|minimal|common)[^.]+$'
+				add_layered_pass_delta install desktop-default-languages ''  # none (if no default langpack is selected)
+				;;
+		esac
+
 		case $ARCH in
 			amd64)	add_package live $SIGNED_KERNEL_PACKAGE ;;
 		esac
@@ -728,30 +772,11 @@
 		;;
 	ubuntu-server:live)
 		BASE_SEED='server'
-		# subiquity is seeded but in a separate squashfs via hooks; set HOOK_SNAPS and ALL_SNAPS.
-		HOOK_SNAPS='subiquity'
-		ALL_SNAPS=''
 		;;
 esac
 
-if [ -n "${BASE_SEED}" ]; then
-	SEEDS_EXPANDED=$(inheritance ${BASE_SEED})
-	for seed in ${SEEDS_EXPANDED}; do
-		echo "snap: considering ${seed}"
-		file=config/germinate-output/${seed}.snaps
-		[ -e "${file}" ] || continue
-		# extract the first column (snap package name) from germinate's output
-		# translate the human-readable "foo (classic)" into a
-		# more machine readable "foo/classic"
-		seed_snaps=$(sed -rn '1,/-----/d;/-----/,$d; s/(.*) \|.*/\1/; s, \(classic\),/classic,; p' "${file}")
-		for snap in ${seed_snaps}; do
-			echo "snap: found ${snap}"
-			ALL_SNAPS="${ALL_SNAPS:+${ALL_SNAPS} }${snap}"
-		done
-	done
-	if [ -n "${ALL_SNAPS}" ] || [ -n "${HOOK_SNAPS}" ]; then
-		echo "${ALL_SNAPS}" > config/seeded-snaps
-	fi
+if [ -z "$PASSES" ] && [ -n "${BASE_SEED}" ]; then
+	snap_from_seed "${BASE_SEED}" config/seeded-snaps
 fi
 
 # grab a list of packags to remove for a "minimal" installation from the seed
@@ -890,6 +915,9 @@
 echo "LB_CHROOT_HOOKS=\"$CHROOT_HOOKS\"" >> config/chroot
 echo "SUBPROJECT=\"${SUBPROJECT:-}\"" >> config/chroot
 echo "LB_DISTRIBUTION=\"$SUITE\"" >> config/chroot
+if [ -n "$PASSES" ]; then
+	echo "PASSES=\"$PASSES\"" >> config/common
+fi
 echo "LB_BINARY_HOOKS=\"$BINARY_HOOKS\"" >> config/binary
 echo "BUILDSTAMP=\"$NOW\"" >> config/binary
 echo "SUBPROJECT=\"${SUBPROJECT:-}\"" >> config/binary
 
=== modified file 'live-build/functions'
--- live-build/functions	2018-11-13 19:21:19 +0000
+++ live-build/functions	2018-11-28 08:47:44 +0000
@@ -373,6 +373,30 @@
     distro-info --series="$LB_DISTRIBUTION" -r | awk '{ print $1 }'
 }
 
+# cribbed from cdimage, perhaps this should be a small helper script in germinate?
+add_inheritance () {
+	case " $inherit " in
+		*" $1 "*)
+			;;
+		*)
+			inherit="${inherit:+$inherit }$1"
+			;;
+	esac
+}
+
+expand_inheritance () {
+	for seed in $(grep "^$1:" config/germinate-output/structure | cut -d: -f2); do
+		expand_inheritance "$seed"
+	done
+	add_inheritance "$1"
+}
+
+inheritance () {
+	inherit=
+	expand_inheritance "$1"
+	echo "$inherit"
+}
+
 _snap_preseed() {
     # Download the snap/assertion and add to the preseed
     local CHROOT_ROOT=$1
@@ -387,6 +411,12 @@
 
     # Download the snap & assertion
     local snap_download_failed=0
+
+    # Preseed a snap only once
+    if [ -f ${snaps_dir}/${SNAP_NAME}_[0-9]*.snap ]; then
+        return
+    fi
+
     chroot $CHROOT_ROOT sh -c "
         set -x;
         cd /var/lib/snapd/seed;
@@ -432,6 +462,10 @@
     local account_key_assertion="$assertions_dir/account-key"
     local account_assertion="$assertions_dir/account"
 
+    if [ -d "$assertions_dir" ]; then
+        return
+    fi
+
     mkdir -p "$assertions_dir"
     mkdir -p "$snaps_dir"
 
@@ -476,15 +510,10 @@
     # used for the image's model assertion
     local CUSTOM_BRAND_MODEL=${2:-generic:generic-classic}
 
-    local seed_dir="$CHROOT_ROOT/var/lib/snapd/seed"
-    local snaps_dir="$seed_dir/snaps"
-
     snap_prepare_assertions "$CHROOT_ROOT" "$CUSTOM_BRAND_MODEL"
 
     # Download the core snap
-    if ! [ -f $snaps_dir/core_[0-9]*.snap ] ; then
-        _snap_preseed $CHROOT_ROOT core stable
-    fi
+    _snap_preseed $CHROOT_ROOT core stable
 }
 
 snap_preseed() {
@@ -501,3 +530,166 @@
     fi
     _snap_preseed $CHROOT_ROOT $SNAP $CHANNEL
 }
+
+snap_from_seed() {
+    local base_seed=$1
+    local out=$2
+    local all_snaps
+    local seeds_expanded
+
+    seeds_expanded=$(inheritance ${base_seed})
+    for seed in ${seeds_expanded}; do
+        echo "snap: considering ${seed}"
+        file=config/germinate-output/${seed}.snaps
+        [ -e "${file}" ] || continue
+        # extract the first column (snap package name) from germinate's output
+        # translate the human-readable "foo (classic)" into a
+        # more machine readable "foo/classic"
+        seed_snaps=$(sed -rn '1,/-----/d;/-----/,$d; s/(.*) \|.*/\1/; s, \(classic\),/classic,; p' "${file}")
+        for snap in ${seed_snaps}; do
+            echo "snap: found ${snap}"
+            all_snaps="${all_snaps:+${all_snaps} }${snap}"
+        done
+    done
+    if [ -n "${all_snaps}" ]; then
+        echo "${all_snaps}" > $out
+    fi
+}
+
+seed_from_task ()
+{
+    # Retrieve the name of the seed from a task name
+    local task=$1
+    local seed
+    local seedfile
+    local seedfiles
+
+    seedfile="$(grep -lE "^Task-Key: +${task}\$" config/germinate-output/*seedtext|head -1)"
+    if [ -n "$seedfile" ]; then
+        basename $seedfile .seedtext
+        return
+    fi
+
+    seedfiles="$(grep -lE "^Task-Per-Derivative: *1\$" config/germinate-output/*seedtext)"
+    if [ -n "$seedfiles" ]; then
+        for seed in $(echo $seedfiles | xargs basename -s .seedtext); do
+            if [ ${PROJECT}-${seed} = $task ]; then
+                echo ${seed}
+                return
+            fi
+        done
+    fi
+}
+
+list_packages_from_seed () {
+    # Store all packages for a given seed, including its seed dependency
+    # $1: Name of the seed to expand to a package list
+
+    local all_seeds="$(inheritance $1)"
+
+    for seed in $all_seeds; do
+        head -n-2 config/germinate-output/${seed}.seed|tail -n+3|awk '{print $1}'
+    done|sort -u
+}
+
+substract_package_lists() {
+    # Substract a package list from another
+    #
+    # $1 source package list
+    # $2 Package list to substract from source package list
+    local list1=$(mktemp)
+    local list2=$(mktemp)
+
+    list_packages_from_seed $1 > list1
+    list_packages_from_seed $2 > list2
+    comm -23 list1 list2
+
+    rm list1
+    rm list2
+}
+
+clean_debian_chroot() {
+    # remove crufty files that shouldn't be left in an image
+    rm -f chroot/var/cache/debconf/*-old chroot/var/lib/dpkg/*-old
+	Chroot chroot apt clean
+}
+
+configure_universe() {
+	if [ -f config/universe-enabled ]; then
+        # This is cargo-culted almost verbatim (with some syntax changes for
+        # preinstalled being slightly different in what it doesn't ask) from
+        # debian-installer's apt-setup:
+
+        cat > chroot/etc/apt/sources.list << EOF
+# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
+# newer versions of the distribution.
+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION main restricted
+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION main restricted
+
+## Major bug fix updates produced after the final release of the
+## distribution.
+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates main restricted
+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates main restricted
+
+## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
+## team. Also, please note that software in universe WILL NOT receive any
+## review or updates from the Ubuntu security team.
+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION universe
+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION universe
+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates universe
+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates universe
+
+## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
+## team, and may not be under a free licence. Please satisfy yourself as to
+## your rights to use the software. Also, please note that software in
+## multiverse WILL NOT receive any review or updates from the Ubuntu
+## security team.
+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION multiverse
+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION multiverse
+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates multiverse
+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates multiverse
+
+## N.B. software from this repository may not have been tested as
+## extensively as that contained in the main release, although it includes
+## newer versions of some applications which may provide useful features.
+## Also, please note that software in backports WILL NOT receive any review
+## or updates from the Ubuntu security team.
+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-backports main restricted universe multiverse
+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-backports main restricted universe multiverse
+
+## Uncomment the following two lines to add software from Canonical's
+## 'partner' repository.
+## This software is not part of Ubuntu, but is offered by Canonical and the
+## respective vendors as a service to Ubuntu users.
+# deb http://archive.canonical.com/ubuntu $LB_DISTRIBUTION partner
+# deb-src http://archive.canonical.com/ubuntu $LB_DISTRIBUTION partner
+
+deb $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security main restricted
+# deb-src $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security main restricted
+deb $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security universe
+# deb-src $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security universe
+deb $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security multiverse
+# deb-src $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security multiverse
+EOF
+
+fi
+}
+
+configure_network_manager() {
+    # If the image pre-installs network-manager, let it manage all devices by
+    # default. Installing NM on an existing system only manages wifi and wwan via
+    # /usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf. When setting
+    # the global backend to NM, netplan overrides that file.
+    if [ -d chroot/usr/lib/NetworkManager ]; then
+        echo "===== Enabling all devices in NetworkManager ===="
+        mkdir -p chroot/etc/netplan
+        cat <<EOF > chroot/etc/netplan/01-network-manager-all.yaml
+# Let NetworkManager manage all devices on this system
+network:
+  version: 2
+  renderer: NetworkManager
+EOF
+    else
+        echo "==== NetworkManager not installed ===="
+    fi
+}
 
=== added file 'live-build/lb_binary_layered'
--- live-build/lb_binary_layered	1970-01-01 00:00:00 +0000
+++ live-build/lb_binary_layered	2018-11-28 08:47:44 +0000
@@ -0,0 +1,132 @@
+#!/bin/sh
+
+## live-build(7) - System Build Scripts
+## Copyright (C) 2006-2012 Daniel Baumann <[e-mail]>
+##
+## This program comes with ABSOLUTELY NO WARRANTY; for details see COPYING.
+## This is free software, and you are welcome to redistribute it
+## under certain conditions; see COPYING for details.
+
+
+set -e
+
+# Including common functions
+( . "${LIVE_BUILD}/scripts/build.sh" > /dev/null 2>&1 || true ) || . /usr/lib/live/build.sh
+
+
+# Automatically populating config tree
+if [ -x auto/config ] && [ ! -e .build/config ]
+then
+	Echo_message "Automatically populating config tree."
+	lb config
+fi
+
+# Setting static variables
+DESCRIPTION="$(Echo 'build binary images')"
+HELP=""
+USAGE="${PROGRAM} [--force]"
+
+Arguments "${@}"
+
+# Reading configuration files
+Read_conffiles config/all config/common config/bootstrap config/chroot config/binary config/source
+Set_defaults
+
+# Setup cleanup function
+Setup_cleanup
+
+. config/functions
+
+build_layered_squashfs() {
+	local pass=$1   # install|install_subpass|install_subpass_subsubpass|…
+	local prevpass=$2  # install|install_subpass|…
+	local prefix=$3 # 01-|02-|…
+	local lowerlayers=$4
+	shift 4 # restore ${*}
+
+	# Cleanup root filesystem
+	lb binary_chroot ${*}
+
+	# Building squashfs filesystem & manifest
+	base="${PWD}/livecd.${PROJECT}.${prefix}${pass}"
+	squashfs_f="${base}.squashfs"
+
+	# We have already treated that pass
+	if [ -f "${squashfs_f}" ]; then
+		return
+	fi
+
+	if [ -n "${lowerlayers}" ]; then
+		mount_overlay ${lowerlayers} "chroot.${pass}/" chroot/
+	else
+		# first pass
+		rmdir chroot 2>/dev/null||true
+		ln -s "chroot.${pass}/" chroot
+	fi
+
+	# Full manifest until that PASS
+	squashfs_f_manifest="${base}.manifest"
+	create_manifest "chroot" "${squashfs_f_manifest}.full"
+
+	# Delta manifest
+	diff -NU0 ${PWD}/livecd.${PROJECT}.[0-9][0-9]-${prevpass}.manifest.full ${squashfs_f_manifest}.full|grep -v ^@ > $squashfs_f_manifest
+
+	squashfs_f_size="${base}.size"
+	du -B 1 -s "chroot.${pass}/" | cut -f1 > "${squashfs_f_size}"
+
+	(cd "chroot.${pass}/" &&
+		mksquashfs . ${squashfs_f} \
+		-no-progress -xattrs -comp xz )
+
+	if [ -n "${lowerlayers}" ]; then
+		umount chroot
+	else
+		rm chroot
+		mkdir chroot/
+	fi
+
+	# Handle direct sublayer of current one
+	# Extract the name of the pass corresponding to the sublayer
+	for subpass in $(ls -d chroot.${pass}_* 2>/dev/null | sed -e "s/chroot\.\(${pass}_[^_]\+\).*/\1/"); do
+		lowerlayers_for_subpass="chroot.${pass}:${lowerlayers}"
+		lowerlayers_for_subpass="${lowerlayers_for_subpass%:}"
+		build_layered_squashfs "${subpass}" "${pass}" "${prefix}" "${lowerlayers_for_subpass}" ${*}
+	done
+}
+
+CURPASS=1
+PREVPASS=""
+PASSPREFIX=""
+LOWER_LAYERS=""
+for _PASS in $PASSES
+do
+	PASSPREFIX="$(printf "%02g" $CURPASS)-"
+
+	build_layered_squashfs "${_PASS}" "${PREVPASS}" "$PASSPREFIX" "${LOWER_LAYERS}" ${*}
+
+	LOWER_LAYERS="chroot.${_PASS}:$LOWER_LAYERS"
+	LOWER_LAYERS="${LOWER_LAYERS%:}"
+	PREVPASS=${_PASS}
+	CURPASS=$(( CURPASS + 1 ))
+done
+
+# remount last "main" pass on chroot for lb binary
+mount_overlay "${LOWER_LAYERS}" "chroot.${_PASS}/" chroot/
+
+# Prepare initrd + kernel
+lb binary_linux-image ${*}
+
+umount chroot/
+
+# Full ISO manifest & size from last main PASS
+PREFIX="livecd.$PROJECT${SUBARCH:+-$SUBARCH}"
+cp "livecd.${PROJECT}.${PASSPREFIX}${_PASS}.size" "$PREFIX.size"
+cp "livecd.${PROJECT}.${PASSPREFIX}${_PASS}.manifest.full" "$PREFIX.manifest"
+
+# Ubiquity-compatible removal manifest for ISO not using a layered-aware installer
+if [ -n "$(ls livecd.${PROJECT}.[0-9][0-9]-live.manifest.full 2>/dev/null)" ] && \
+   [ -n "$(ls livecd.${PROJECT}.[0-9][0-9]-install.manifest.full 2>/dev/null)" ]; then
+	echo "$(diff livecd.${PROJECT}.[0-9][0-9]-live.manifest.full livecd.${PROJECT}.[0-9][0-9]-install.manifest.full | awk '/^< / { print $2 }')" > livecd.${PROJECT}-manifest-remove
+fi
+
+chmod 644 *.squashfs *.manifest* *.size
 
=== added file 'live-build/lb_chroot_layered'
--- live-build/lb_chroot_layered	1970-01-01 00:00:00 +0000
+++ live-build/lb_chroot_layered	2018-11-28 08:47:44 +0000
@@ -0,0 +1,246 @@
+#!/bin/sh
+
+## live-build(7) - System Build Scripts
+## Copyright (C) 2006-2012 Daniel Baumann <[e-mail]>
+##
+## This program comes with ABSOLUTELY NO WARRANTY; for details see COPYING.
+## This is free software, and you are welcome to redistribute it
+## under certain conditions; see COPYING for details.
+
+## This is a fork of lb_chroot for layered live system.
+## We don't want leaking host configuratino in each layer, and so,
+## we clean and setup the chroot each time.
+## In addition, we create the squashfs for each layer, but top one (live)
+## which still can be configured after lb chroot call.
+
+set -e
+
+# Including common functions
+( . "${LIVE_BUILD}/scripts/build.sh" > /dev/null 2>&1 || true ) || . /usr/lib/live/build.sh
+
+# Automatically populating config tree
+if [ -x auto/config ] && [ ! -e .build/config ]
+then
+	Echo_message "Automatically populating config tree."
+	lb config
+fi
+
+# Setting static variables
+DESCRIPTION="$(Echo 'customize the Debian system')"
+HELP=""
+USAGE="${PROGRAM} [--force]"
+
+Arguments "${@}"
+
+# Reading configuration files
+Read_conffiles config/all config/common config/bootstrap config/chroot config/binary config/source
+Set_defaults
+
+# Setup cleanup function
+Setup_cleanup
+
+. config/functions
+
+lb_chroot_remove_packages() {
+	# Remove packages from the chroot specific to this layer
+	#
+	# $1: Name of the pass*
+	local pass=$1
+
+	Expand_packagelist "$(basename config/package-lists/*.removal-list.chroot_${pass})" "config/package-lists" \
+			>> chroot/root/packages.chroot.removal
+	Chroot chroot "xargs --arg-file=/root/packages.chroot.removal apt-get ${APT_OPTIONS} autoremove --purge"
+	rm -f chroot/root/packages.chroot.removal
+}
+
+# Create the snap list specific to this layer
+lb_chroot_snap_lists() {
+	local pass=$1
+	local prevpass=$2
+
+	# This assumes that the prefix is unique for a given project
+	local snap_for_pass=$(ls config/package-lists/*.snaplist.chroot_${pass}.full 2>/dev/null || true)
+	local snap_for_prevpass=$(ls config/package-lists/*.snaplist.chroot_${prevpass}.full 2>/dev/null || true)
+
+	if [ -z "${snap_for_pass}" ]; then
+		return
+	fi
+
+	if [ -z "${snap_for_prevpass}" ]; then
+		cp ${snap_for_pass} ${snap_for_pass%.full}
+		return
+	fi
+
+	# Generate a list of snaps added to a layer.
+	diff -NU0 ${snap_for_prevpass} ${snap_for_pass}|grep -Ev '^(---|\+\+\+|@@)'|cut -c2- > ${snap_for_pass%.full}
+}
+
+lb_chroot_install_snaps() {
+	# Prepare the snap environment and install snaps into a chroot
+	#
+	# $1: Name of the pass
+
+	local snaplist_file=$(ls config/package-lists/*.snaplist.chroot_${1} 2>/dev/null || true)
+
+	if [ -z "${snaplist_file}" ]; then
+		return
+	fi
+
+	snap_prepare chroot
+
+	while read snap; do
+		snap_preseed chroot "${snap}"
+	done < $snaplist_file
+}
+
+lb_chroot_includes() {
+	# Copying includes from pass subdirectory
+	local pass="$1"
+
+	if [ ! -d config/includes.chroot.${pass} ]; then
+		return
+	fi
+
+	cd config/includes.chroot.${pass}
+	find . | cpio -dmpu --no-preserve-owner "${OLDPWD}"/chroot
+	cd "${OLDPWD}"
+}
+
+create_chroot_pass() {
+	local pass=$1
+	local prevpass=$2
+	local lowerlayers=$3
+	local passtype=$4  # "first"|"last"|"" empty string
+	shift 4 # restore ${*}
+
+	# We have already treated that pass
+	if [ -d "chroot.${pass}/" ]; then
+		return
+	fi
+
+	export PASS=${pass}
+
+	if [ "${passtype}" != "first" ]; then
+		mkdir chroot.${pass}
+		mount_overlay ${lowerlayers} "chroot.${pass}/" chroot/
+	fi
+
+	# Configuring chroot
+	lb chroot_cache restore ${*}
+	lb chroot_devpts install ${*}
+	lb chroot_proc install ${*}
+	lb chroot_sysfs install ${*}
+	lb chroot_debianchroot install ${*}
+	lb chroot_dpkg install ${*}
+	lb chroot_tmpfs install ${*}
+	lb chroot_hosts install ${*}
+	lb chroot_resolv install ${*}
+	lb chroot_hostname install ${*}
+	lb chroot_apt install ${*}
+	# Note: this triggers an upgrade + dist-ugprade; which may impact sublayers with more
+	# diff content than desired. However, we still need to setup the archive and teardown
+	# for each layer.
+	# We could modify livebuild if necessary to have conditional upgrade (first pass only).
+	lb chroot_archives chroot install ${*}
+
+	if [ "${passtype}" = "first" ]; then
+		configure_universe
+	fi
+
+	# Customizing chroot
+	lb chroot_linux-image ${*}
+	lb chroot_preseed ${*}
+	lb chroot_early_hooks ${*}
+
+	lb chroot_package-lists ${pass} ${*}
+	lb chroot_install-packages ${pass} ${*}
+	lb_chroot_remove_packages ${pass} ${*}
+
+	# Snap management
+	lb_chroot_snap_lists ${pass} ${prevpass}
+	lb_chroot_install_snaps ${pass} ${*}
+
+	# Kernel should be in first layer
+	if [ "${passtype}" = "first" ]; then
+		configure_network_manager
+		Chroot chroot "dpkg -l linux-headers-3* linux-headers-4*" 2>/dev/null \
+			| awk '/^i/ {print $2}' > chroot.headers
+		for i in $(cat chroot.headers); do
+			Chroot chroot "apt-mark auto $i"
+		done
+	fi
+
+	Chroot chroot "apt-get --purge -y autoremove"
+
+	# Add live packages to top layer
+	if [ "${passtype}" = "last" ]; then
+		lb chroot_live-packages ${*}
+	fi
+
+	# Run includes by pass
+	lb_chroot_includes ${pass} ${*}
+
+	lb chroot_hooks ${*}
+	lb chroot_hacks ${*}
+	lb chroot_interactive ${*}
+
+	# Misc ubuntu cleanup and post-layer configuration
+	clean_debian_chroot
+	/usr/share/livecd-rootfs/minimize-manual chroot
+
+	Chroot chroot "dpkg-query -W" > chroot.packages.${pass}
+
+	# Deconfiguring chroot
+	lb chroot_archives chroot remove ${*}
+	lb chroot_apt remove ${*}
+	lb chroot_hostname remove ${*}
+	lb chroot_resolv remove ${*}
+	lb chroot_hosts remove ${*}
+	lb chroot_tmpfs remove ${*}
+	lb chroot_dpkg remove ${*}
+	lb chroot_debianchroot remove ${*}
+	lb chroot_sysfs remove ${*}
+	lb chroot_proc remove ${*}
+	lb chroot_devpts remove ${*}
+	lb chroot_cache save ${*}
+
+	if [ "${passtype}" = "first" ]; then
+		mv chroot chroot.${pass}
+		mkdir chroot
+	else
+		umount chroot
+	fi
+
+	# Handle direct sublayer of current one
+	# Extract the name of the pass corresponding to the sublayer
+	for subpass in $(ls config/package-lists/*list.chroot_${pass}_* 2>/dev/null | sed -e "s/.*list\.chroot_\(${pass}_[^_]\+\).*/\1/"); do
+		lowerlayers_for_subpass="chroot.${pass}:${lowerlayers}"
+		lowerlayers_for_subpass="${lowerlayers_for_subpass%:}"
+		create_chroot_pass "${subpass}" "${pass}" "${lowerlayers_for_subpass}" "" ${*}
+	done
+}
+
+PASSES="${PASSES:-install live}"
+CURPASS=1
+PREVPASS=""
+LASTPASS=$(echo $PASSES|wc -w)
+LOWER_LAYERS=""
+for _PASS in $PASSES
+do
+	PASSTYPE=""
+	if [ $CURPASS -eq 1 ]; then
+		PASSTYPE="first"
+	elif [ $CURPASS -eq $LASTPASS ]; then
+		PASSTYPE="last"
+	fi
+
+	create_chroot_pass "$_PASS" "$PREVPASS" "$LOWER_LAYERS" "$PASSTYPE" ${*}
+	
+	LOWER_LAYERS="chroot.${_PASS}:$LOWER_LAYERS"
+	LOWER_LAYERS="${LOWER_LAYERS%:}"
+	PREVPASS=${_PASS}
+	
+	CURPASS=$(( CURPASS + 1 ))
+done
+
+rmdir chroot