Merge lp:~jibel/livecd-rootfs/add_multi_layered_squashfses_support into lp:livecd-rootfs

Proposed by Jean-Baptiste Lallement on 2018-11-08
Status: Needs review
Proposed branch: lp:~jibel/livecd-rootfs/add_multi_layered_squashfses_support
Merge into: lp:livecd-rootfs
Diff against target: 1242 lines (+808/-255)
7 files modified
debian/tests/default-bootstraps (+1/-0)
live-build/auto/build (+137/-185)
live-build/auto/clean (+2/-0)
live-build/auto/config (+92/-64)
live-build/functions (+198/-6)
live-build/lb_binary_layered (+132/-0)
live-build/lb_chroot_layered (+246/-0)
To merge this branch: bzr merge lp:~jibel/livecd-rootfs/add_multi_layered_squashfses_support
Reviewer Review Type Date Requested Status
Steve Langasek 2018-11-08 Needs Fixing on 2018-11-20
Review via email: mp+358490@code.launchpad.net

Description of the change

Adds support for multi layer filesystem by generating one squashfs per layer and adding a new 'live-layered' image format.
squashfs are numbered so the order is preserved and mounted in order by casper.

Seeding of snap packages and seeds corresponding to the layers are not
available yet and will be proposed in the subsequent merge proposal.

To post a comment you must log in.
Dimitri John Ledkov (xnox) wrote :

In passes, how does one specify multiple branches of stacks? Or is only a linear stack allowed by this code?

Cause the current subiquity images do not have `live` as the top of the stack.... They do this:

+--> Base +-----> Live
          |
          +-----> Rack +---> Region

Because I am expecting to have the ability to somehow specify what each pass depends on. But it looks like the base for a subsequent pass, is always the previous one?!

It would be nice to extend PASS syntax to optionally accept an arbitrary `base`, e.g.

PASSES="base rack region base:live" to encode the above graph, and such that `base:live` pass uses base as the lowerdir, instead of region (i.e. the previous pass result).

Or something...

not sure if : is acceptable pair delimiter here, or not.

Also not sure if we want to always enforce specifying `base` such that we can construct multiple root nodes in one go. E.g. `base base:rack rack:region base:live` or `:base rack region base:live` for the above graph.

Or like maybe always list all layers?! but that violates donot-repeat-yourself principle... E.g. `base base:rack base:rack:region base:live`

===

When calling includes/hooks are they aware which pass they are for? what is the source dir for each pass, for e.g. binary.includes? I'm guessing that PASS variable is set, but not sure.

===

filesystem.squashfs is somewhat is a special name, so it would be nice to keep that as the base one. And also possibly adjust logic in casper as to what it mounts by default.... cause e.g. i think i hide maas squashfes in a subdir, to prevent casper from mounting those, which is kind of a hack. I wonder if we do need to write out the valid stacks (passes?!), which casper can then use to boot to whichever stack is valid. With subiquity image this could then result in "Live Server, Live MAAS Rack, Live MAAS Region, Live Server with Installer" boot options. As example, for better or worse.

====

No idea if SUBPROJECT and IMAGE_FORMAT are the right things to extend for this..... and if they are easily extendable like this in launchpad livefs builders & ubuntu-cdimage codes.

Cause I can see the potential for using layers in SUBPROJECT=minimized, if for example, cpc builds are converted to layers they would have full and minimized layered builds.... and SUBPROJECT=minimized-layered sounds ugly =)

====

manifest diffs for layers is nice; cause in cpc we have struggled to consitently represent manifests / changelogs of "it's just like that image, but has this stuff on it"

====

packaging layers as actual static filesystems might be interesting, but i guess hooks will be able to do that anyway.

====

Overall, this looks ok, and shouldn't break any existing stuff - as long as we can clear the top level new extensions of

   SUBPROJECT=layered
   IMAGEFORMAT=live-layered

and that needs like an architect review.

Jean-Baptiste Lallement (jibel) wrote :
Download full text (5.9 KiB)

Thanks for the review

On 14/11/2018 00:35, Dimitri John Ledkov wrote:
>
> In passes, how does one specify multiple branches of stacks? Or is only a linear stack allowed by this code?

This code only allows linear stacks to avoid not make current code more
complex than it already is and diverging too much from existing logic.
Our approach is to define the stacking defining the live image in PASSES
and put the extra logic like “branched mounts” in hooks.

For instance with langpacks which is the use case for Desktop image, we
also have multiple branches of stacks. It is difficult to represent the
structure in a generic way and it adds extra complexity to the
maintenance of those definitions.

Following your proposal to represent the layers structure as a list, it
would be something like:
PASSES=“desktop-minimal:desktop:live desktop-minimal:lang-neg-min-fr
desktop-minimal:lang-neg-min-de desktop-minimal:desktop:lang-neg-fr
desktop-minimal:desktop:lang-neg-de …”

We thus have multiple “base” here, “desktop-minimal” or
“desktop-minimal:desktop”. Shortening the syntax doesn’t seem
appropriate here.

In a tree, it’s not better:
- desktop-min
   - desktop
     - Lang-neg-de
     - Lang-neg-fr
     - Lang-neg-es
     - …
     - Live
   - Lang-neg-min-de
   - Lang-neg-min-fr
   - Lang-neg-min-es
   - …

Addition or removal of any default language would be then error-prone.
This is to compare with a hook, where we just loop over “desktop-min”
and creates langpacks negative stacks, and then looping over “desktop”
as well to achieve the same. Besides, shell seems inappropriate to
implement this type of logic.

Note that the current implementation is similar than existing
ubuntu-server:live logic (just a little bit more generic) and we made
sure we didn’t break your use case. Note that though, you will be able
to remove your first hook, creating the “live” stack using the generic
code right now. However, the maas-* stacks will still be in hooks.

>
> Cause the current subiquity images do not have `live` as the top of the stack.... They do this:
>
> +--> Base +-----> Live
> |
> +-----> Rack +---> Region
>
> Because I am expecting to have the ability to somehow specify what each pass depends on. But it looks like the base for a subsequent pass, is always the previous one?!
>
>
> It would be nice to extend PASS syntax to optionally accept an arbitrary `base`, e.g.
>
> PASSES="base rack region base:live" to encode the above graph, and such that `base:live` pass uses base as the lowerdir, instead of region (i.e. the previous pass result).
>
> Or something...
>
> not sure if : is acceptable pair delimiter here, or not.
>
> Also not sure if we want to always enforce specifying `base` such that we can construct multiple root nodes in one go. E.g. `base base:rack rack:region base:live` or `:base rack region base:live` for the above graph.
>
> Or like maybe always list all layers?! but that violates donot-repeat-yourself principle... E.g. `base base:rack base:rack:region base:live`
>
> ===
>
> When calling includes/hooks are they aware which pass they are for? what is the source dir for each pass, for e.g. bina...

Read more...

Steve Langasek (vorlon) wrote :

Thanks for working on this, it will be nice to have cleaner handling of the desktop-minimal stuff included here in livecd-rootfs.

I would like to see this converged with the ubuntu-server-live handling as part of landing of this branch (see also Dimitri's review comments). You've gone to some length to implement this in a way that's generalizable across projects, so we ought to make sure the implementation is actually reusable by the only other image that currently uses layered squashfs.

review: Needs Fixing
Jean-Baptiste Lallement (jibel) wrote :

We intended to propose several MP for snap and “sub-layers” to reduce the complexity of the review, but we’ll finally merge them into this one so you’ll have a good understanding of the full implementation.

Supporting the server-live use case adds makes the implementation more complex due to its specific requirements (hooks and includes). But we agree that a more generalizable implementation is better, so let’s spend some time to ensure those 2 additional requirements are supported, and once foundation/server team want to switch to it, that they will not encounter any major limitations.

Jean-Baptiste Lallement (jibel) wrote :

Following previous comments, this MP has been updated with these changes:
- Merged preseeding of snap packages and adds snap packages to the image manifest.
- Added lb_binary_layered to handle multi branches of squashfses with sublayers.
- Factorized several helpers called by lb_chroot_layered
- Removed obsolete chroot helpers from lb_chroot_layered and minor cleanup.
- Renamed subproject layered -> ubiquity-ng
- Use PASSES instead of IMAGEFORMAT to detect a multilayer project. Defining PASSES switches the image build into layered build.
- Adds includes by pass to customize chroot for specific passes.
- Moved back specific helpers and functions from functions to config.
- For Ubuntu Desktop:ubiquity-ng, build negative language packs and corresponding squashfs.

These changes have been tested against ubuntu-server live which still uses its current implementation (with maas-region and maas-rack squashfses built in hooks).
All the requirements of ubuntu-server:live can be ported from the current hooks to the new implementation of the layered images.
To do so:
- The project have to define PASSES.
- Snaps and packages for sublayers added to separated seeds and referenced in auto/config.
- Layers must be customized with chroot hooks (eg to modify existing files) and chroot includes (eg to add new files) depending on PASS.

Ubuntu Desktop (standard disco) has been modified to use layered images as requested. As a consequence the following MPs must be reviewed together with this one and released before uploading livecd-rootfs:
- debian-cd: https://code.launchpad.net/~jibel/debian-cd/support_for_multilayer_images/+merge/359228
- ubuntu-cdimage: https://code.launchpad.net/~jibel/ubuntu-cdimage/support_for_multilayer/+merge/359512

The subproject ubiquity-ng must be created for building the ubuntu-desktop:ubiquity-ng rootfs and match https://launchpad.net/~ubuntu-cdimage/+livefs/ubuntu/disco/ubuntu-desktop-ubiquity-ng/
This whole set of changes (livecd-rootfs, ubuntu-cdimage and debian-cd) have been tested with the following projects:
- Ubuntu Desktop layered (iso) (disco) -> test new set of layers
- Ubuntu Desktop unlayered (iso) (disco) -> test previous image format (one squashfs)
- Ubuntu Desktop ubiquity-ng (iso) -> test new set of layers + sublayers
- Ubuntu Server live (iso) (bionic and disco) -> test backward compatibility with current server image and hooks impacts.
- Lubuntu (rootfs only, no local archive with universe for building iso, similar build to ubuntu-desktop/unlayered) (disco) -> test traditional one squashfs image
- Ubuntu Mate (rootfs only, no local archive with universe for building iso, similar build to ubuntu-desktop/unlayered) (bionic) -> test traditional one squashfs image + snaps

Adam Conrad (adconrad) wrote :

I don't have time for a full review this morning, but unless we intend to keep these forked from the "real" ISOs for testing for a while, and then converge, I really don't like the SUBPROJECT use here. I'd expect the real 'ubuntu' project to be building these, not some ubuntu-random-subproject project.

As noted, you already can decide if you're building this type of image based on PASSES being defined, so it seems a bit odd to then also key on a subproject name.

1758. By Jean-Baptiste Lallement on 2018-11-28

configure network manager _after_ installing network-manager

Jean-Baptiste Lallement (jibel) wrote :

> I don't have time for a full review this morning, but unless we intend to keep
> these forked from the "real" ISOs for testing for a while, and then converge,
> I really don't like the SUBPROJECT use here. I'd expect the real 'ubuntu'
> project to be building these, not some ubuntu-random-subproject project.
>
> As noted, you already can decide if you're building this type of image based
> on PASSES being defined, so it seems a bit odd to then also key on a
> subproject name.
We need a subproject because the real 'ubuntu' and the new installer work will diverge quickly. Note that Ubuntu Desktop uses the layer system on Disco with this merge proposal without defining a subproject.

Balint Reczey (rbalint) wrote :

The target Bazaar branch is not active anymore.
Please resubmit the merge proposal against https://code.launchpad.net/~ubuntu-core-dev/livecd-rootfs/+git/livecd-rootfs/+ref/ubuntu/master .

Unmerged revisions

1758. By Jean-Baptiste Lallement on 2018-11-28

configure network manager _after_ installing network-manager

1757. By Jean-Baptiste Lallement on 2018-11-26

Add ubiquity-ng image autopkgtest

1756. By Jean-Baptiste Lallement on 2018-11-23

Ensure snaps are available on ubuntu-server live

1755. By Jean-Baptiste Lallement on 2018-11-23

Fixed pattern for matching manifest

1754. By Jean-Baptiste Lallement on 2018-11-23

Cleanup and handle manifest-remove files

manifest-remove files are generated in all cases and removed for
projects/subprojects that don't need it like the new ubiquity.

1753. By Jean-Baptiste Lallement on 2018-11-23

Added lb_binary_layered

Created lb_binary_layered based on lb_binary
Factorized several helpers
These helpers are called from lb_chroot

1752. By Jean-Baptiste Lallement on 2018-11-23

Moved reusable functions to functions

Moved functions to reuse in lb_binary and lb_chroot to functions

1751. By Jean-Baptiste Lallement on 2018-11-23

Clean up chroot helpers

Removed obsolete chroot helpers on Disco+:
 - selinux
 - sysv-rc
 - upstart

Added a note about needlessly triggering chroot_archives on each pass.

1750. By Jean-Baptiste Lallement on 2018-11-23

Various fixes

Fixed prefix for passes to not conflict with global prefix
Set layer to none for layer without language packs
Copy size and manifest to the target directory
Make sure the diff always diffes existing manifests

1749. By Jean-Baptiste Lallement on 2018-11-23

Add includes by pass

lb_chroot_includes now takes the name of the pass as first argument and
will includes files for the corresponding pass from the directory
include.chroot.<pass>

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'debian/tests/default-bootstraps'
2--- debian/tests/default-bootstraps 2018-05-09 02:12:15 +0000
3+++ debian/tests/default-bootstraps 2018-11-28 08:47:44 +0000
4@@ -23,6 +23,7 @@
5 lubuntu::
6 mythbuntu::
7 ubuntu::
8+ ubuntu:ubiquity-ng:
9 ubuntu-base::
10 ubuntu-budgie::
11 ubuntu-budgie-desktop::
12
13=== modified file 'live-build/auto/build'
14--- live-build/auto/build 2018-11-26 12:03:24 +0000
15+++ live-build/auto/build 2018-11-28 08:47:44 +0000
16@@ -252,107 +252,57 @@
17 dpkg-divert --quiet --remove --rename /usr/sbin/update-initramfs
18 EOF
19
20- lb chroot "$@"
21-
22- if [ "${SUBPROJECT:-}" = minimized ]; then
23- # force removal of initramfs-tools, which we assert is not
24- # required for any minimized images but is still pulled in by
25- # default
26- # also remove landscape-common, which is heavyweight and
27- # in the server seed only to provide /etc/motd content which
28- # would only be seen by humans
29- Chroot chroot "env DEBIAN_FRONTEND=noninteractive \
30- apt-get -y purge initramfs-tools busybox-initramfs \
31- busybox-static landscape-common"
32- # and if initramfs-tools was configured before our kernel,
33- # /etc/kernel/postinst.d/initramfs-tools will have created
34- # an initramfs despite the generic dpkg-divert; so remove it
35- # here.
36- rm -f chroot/boot/initrd.img-*
37-
38- # temporary workaround: don't remove linux-base which
39- # may have no other reverse-depends currently
40- Chroot chroot "env DEBIAN_FRONTEND=noninteractive \
41- apt-mark manual linux-base"
42- Chroot chroot "env DEBIAN_FRONTEND=noninteractive \
43- apt-get -y --purge autoremove"
44- fi
45-
46- # remove crufty files that shouldn't be left in an image
47- rm -f chroot/var/cache/debconf/*-old chroot/var/lib/dpkg/*-old
48- Chroot chroot apt clean
49- if [ "${PROJECT}:${SUBPROJECT:-}" = "ubuntu-base:minimized" ]; then
50- # Save even more size by removing apt lists (that are currently removed
51- # downstream anyway)
52- rm -rf chroot/var/lib/apt/lists/*
53- # Having device notes in the docker image can cause problems
54- # (https://github.com/tianon/docker-brew-ubuntu-core/issues/62)
55- # so remove them. We only do this for docker out of an
56- # abundance of caution.
57- rm -rf chroot/dev/*
58- fi
59-
60- if [ -f config/universe-enabled ]; then
61-
62-# This is cargo-culted almost verbatim (with some syntax changes for
63-# preinstalled being slightly different in what it doesn't ask) from
64-# debian-installer's apt-setup:
65-
66-cat > chroot/etc/apt/sources.list << EOF
67-# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
68-# newer versions of the distribution.
69-deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION main restricted
70-# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION main restricted
71-
72-## Major bug fix updates produced after the final release of the
73-## distribution.
74-deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates main restricted
75-# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates main restricted
76-
77-## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
78-## team. Also, please note that software in universe WILL NOT receive any
79-## review or updates from the Ubuntu security team.
80-deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION universe
81-# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION universe
82-deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates universe
83-# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates universe
84-
85-## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
86-## team, and may not be under a free licence. Please satisfy yourself as to
87-## your rights to use the software. Also, please note that software in
88-## multiverse WILL NOT receive any review or updates from the Ubuntu
89-## security team.
90-deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION multiverse
91-# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION multiverse
92-deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates multiverse
93-# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates multiverse
94-
95-## N.B. software from this repository may not have been tested as
96-## extensively as that contained in the main release, although it includes
97-## newer versions of some applications which may provide useful features.
98-## Also, please note that software in backports WILL NOT receive any review
99-## or updates from the Ubuntu security team.
100-deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-backports main restricted universe multiverse
101-# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-backports main restricted universe multiverse
102-
103-## Uncomment the following two lines to add software from Canonical's
104-## 'partner' repository.
105-## This software is not part of Ubuntu, but is offered by Canonical and the
106-## respective vendors as a service to Ubuntu users.
107-# deb http://archive.canonical.com/ubuntu $LB_DISTRIBUTION partner
108-# deb-src http://archive.canonical.com/ubuntu $LB_DISTRIBUTION partner
109-
110-deb $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security main restricted
111-# deb-src $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security main restricted
112-deb $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security universe
113-# deb-src $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security universe
114-deb $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security multiverse
115-# deb-src $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security multiverse
116-EOF
117-
118- fi
119- if [ -d chroot/var/lib/preinstalled-pool ]; then
120- cat > config/indices/apt.conf <<-EOF
121+ if [ -n "${PASSES}" ]; then
122+ PATH="config/:$PATH" lb chroot_layered "$@"
123+ else
124+ lb chroot "$@"
125+ fi
126+
127+ # Let all configuration non multi-layered project here.
128+ # If those are moving to a multi-layer layout, this needs to be
129+ # done in chroot hooks.
130+ if [ -z "$PASSES" ]; then
131+ if [ "${SUBPROJECT:-}" = minimized ]; then
132+ # force removal of initramfs-tools, which we assert is not
133+ # required for any minimized images but is still pulled in by
134+ # default
135+ # also remove landscape-common, which is heavyweight and
136+ # in the server seed only to provide /etc/motd content which
137+ # would only be seen by humans
138+ Chroot chroot "env DEBIAN_FRONTEND=noninteractive \
139+ apt-get -y purge initramfs-tools busybox-initramfs \
140+ busybox-static landscape-common"
141+ # and if initramfs-tools was configured before our kernel,
142+ # /etc/kernel/postinst.d/initramfs-tools will have created
143+ # an initramfs despite the generic dpkg-divert; so remove it
144+ # here.
145+ rm -f chroot/boot/initrd.img-*
146+
147+ # temporary workaround: don't remove linux-base which
148+ # may have no other reverse-depends currently
149+ Chroot chroot "env DEBIAN_FRONTEND=noninteractive \
150+ apt-mark manual linux-base"
151+ Chroot chroot "env DEBIAN_FRONTEND=noninteractive \
152+ apt-get -y --purge autoremove"
153+ fi
154+
155+ clean_debian_chroot
156+
157+ if [ "${PROJECT}:${SUBPROJECT:-}" = "ubuntu-base:minimized" ]; then
158+ # Save even more size by removing apt lists (that are currently removed
159+ # downstream anyway)
160+ rm -rf chroot/var/lib/apt/lists/*
161+ # Having device notes in the docker image can cause problems
162+ # (https://github.com/tianon/docker-brew-ubuntu-core/issues/62)
163+ # so remove them. We only do this for docker out of an
164+ # abundance of caution.
165+ rm -rf chroot/dev/*
166+ fi
167+
168+ configure_universe
169+
170+ if [ -d chroot/var/lib/preinstalled-pool ]; then
171+ cat > config/indices/apt.conf <<-EOF
172 Dir {
173 ArchiveDir "chroot/var/lib/preinstalled-pool";
174 OverrideDir "config/indices";
175@@ -369,11 +319,11 @@
176 Contents " ";
177 }
178 EOF
179- for component in $LB_PARENT_ARCHIVE_AREAS; do
180- mkdir -p chroot/var/lib/preinstalled-pool/dists/$LB_DISTRIBUTION/$component/binary-$LB_ARCHITECTURES
181- done
182- apt-ftparchive generate config/indices/apt.conf
183- cat << @@EOF > chroot/etc/apt/sources.list.preinstall
184+ for component in $LB_PARENT_ARCHIVE_AREAS; do
185+ mkdir -p chroot/var/lib/preinstalled-pool/dists/$LB_DISTRIBUTION/$component/binary-$LB_ARCHITECTURES
186+ done
187+ apt-ftparchive generate config/indices/apt.conf
188+ cat << @@EOF > chroot/etc/apt/sources.list.preinstall
189 # This is a sources.list entry for a small pool of packages
190 # provided on your preinstalled filesystem for your convenience.
191 #
192@@ -385,93 +335,89 @@
193 #
194 @@EOF
195
196- cp chroot/etc/apt/sources.list chroot/etc/apt/sources.list.orig
197- cp chroot/etc/apt/sources.list.preinstall chroot/etc/apt/sources.list
198-
199- echo "Waiting on gnupg ("$GPG_PROCESS") to finish generating a key."
200- wait $GPG_PROCESS
201-
202- R_ORIGIN=$(lsb_release -i -s)
203- R_CODENAME=$(lsb_release -c -s)
204- R_VERSION=$(lsb_release -r -s)
205- R_PRETTYNAME=$(echo $R_CODENAME | sed -e 's/^\(.\)/\U\1/')
206-
207- apt-ftparchive -o APT::FTPArchive::Release::Origin=$R_ORIGIN \
208- -o APT::FTPArchive::Release::Label=$R_ORIGIN \
209- -o APT::FTPArchive::Release::Suite=$R_CODENAME-local \
210- -o APT::FTPArchive::Release::Version=$R_VERSION \
211- -o APT::FTPArchive::Release::Codename=$R_CODENAME \
212- -o APT::FTPArchive::Release::Description="$R_ORIGIN $R_PRETTYNAME Local" \
213- release chroot/var/lib/preinstalled-pool/dists/$R_CODENAME/ \
214- > config/gnupg/Release
215-
216- gpg --home config/gnupg --detach-sign --armor config/gnupg/Release
217- mv config/gnupg/Release \
218- chroot/var/lib/preinstalled-pool/dists/$R_CODENAME/Release
219- mv config/gnupg/Release.asc \
220- chroot/var/lib/preinstalled-pool/dists/$R_CODENAME/Release.gpg
221- apt-key --keyring chroot/etc/apt/trusted.gpg add config/gnupg/pubring.gpg
222- find chroot/var/lib/preinstalled-pool/ -name Packages | xargs rm
223-
224- Chroot chroot "apt-get update"
225- cat chroot/etc/apt/sources.list.preinstall chroot/etc/apt/sources.list.orig \
226- > chroot/etc/apt/sources.list
227- rm chroot/etc/apt/sources.list.preinstall chroot/etc/apt/sources.list.orig
228- fi
229- case $PROJECT:$SUBPROJECT in
230- *)
231- if [ -e "config/seeded-snaps" ]; then
232- snap_list=$(cat config/seeded-snaps)
233- preinstall_snaps $snap_list
234- fi
235- ;;
236- esac
237-
238- if [ "$PROJECT" = "ubuntu-touch" ] || [ "$PROJECT" = "ubuntu-touch-custom" ]; then
239- if [ "$ARCH" = "armhf" ]; then
240- INFO_DESC="$(lsb_release -d -s)"
241- echo "$INFO_DESC - $ARCH ($BUILDSTAMP)" >chroot/etc/media-info
242- mkdir -p chroot/var/log/installer
243- Chroot chroot "ln -s /etc/media-info /var/log/installer/media-info"
244- fi
245- fi
246- if [ "$PROJECT" = "ubuntu-cpc" ]; then
247- if [ "${SUBPROJECT:-}" = minimized ]; then
248- BUILD_NAME=minimal
249- else
250- BUILD_NAME=server
251- fi
252- cat > chroot/etc/cloud/build.info << EOF
253+ cp chroot/etc/apt/sources.list chroot/etc/apt/sources.list.orig
254+ cp chroot/etc/apt/sources.list.preinstall chroot/etc/apt/sources.list
255+
256+ echo "Waiting on gnupg ("$GPG_PROCESS") to finish generating a key."
257+ wait $GPG_PROCESS
258+
259+ R_ORIGIN=$(lsb_release -i -s)
260+ R_CODENAME=$(lsb_release -c -s)
261+ R_VERSION=$(lsb_release -r -s)
262+ R_PRETTYNAME=$(echo $R_CODENAME | sed -e 's/^\(.\)/\U\1/')
263+
264+ apt-ftparchive -o APT::FTPArchive::Release::Origin=$R_ORIGIN \
265+ -o APT::FTPArchive::Release::Label=$R_ORIGIN \
266+ -o APT::FTPArchive::Release::Suite=$R_CODENAME-local \
267+ -o APT::FTPArchive::Release::Version=$R_VERSION \
268+ -o APT::FTPArchive::Release::Codename=$R_CODENAME \
269+ -o APT::FTPArchive::Release::Description="$R_ORIGIN $R_PRETTYNAME Local" \
270+ release chroot/var/lib/preinstalled-pool/dists/$R_CODENAME/ \
271+ > config/gnupg/Release
272+
273+ gpg --home config/gnupg --detach-sign --armor config/gnupg/Release
274+ mv config/gnupg/Release \
275+ chroot/var/lib/preinstalled-pool/dists/$R_CODENAME/Release
276+ mv config/gnupg/Release.asc \
277+ chroot/var/lib/preinstalled-pool/dists/$R_CODENAME/Release.gpg
278+ apt-key --keyring chroot/etc/apt/trusted.gpg add config/gnupg/pubring.gpg
279+ find chroot/var/lib/preinstalled-pool/ -name Packages | xargs rm
280+
281+ Chroot chroot "apt-get update"
282+ cat chroot/etc/apt/sources.list.preinstall chroot/etc/apt/sources.list.orig \
283+ > chroot/etc/apt/sources.list
284+ rm chroot/etc/apt/sources.list.preinstall chroot/etc/apt/sources.list.orig
285+ fi
286+ case $PROJECT:$SUBPROJECT in
287+ ubuntu-server:live)
288+ lb chroot_resolv install
289+ snap_prepare chroot
290+ lb chroot_resolv remove
291+ ;;
292+ *)
293+ if [ -e "config/seeded-snaps" ]; then
294+ snap_list=$(cat config/seeded-snaps)
295+ preinstall_snaps $snap_list
296+ fi
297+ ;;
298+ esac
299+
300+ if [ "$PROJECT" = "ubuntu-touch" ] || [ "$PROJECT" = "ubuntu-touch-custom" ]; then
301+ if [ "$ARCH" = "armhf" ]; then
302+ INFO_DESC="$(lsb_release -d -s)"
303+ echo "$INFO_DESC - $ARCH ($BUILDSTAMP)" >chroot/etc/media-info
304+ mkdir -p chroot/var/log/installer
305+ Chroot chroot "ln -s /etc/media-info /var/log/installer/media-info"
306+ fi
307+ fi
308+ if [ "$PROJECT" = "ubuntu-cpc" ]; then
309+ if [ "${SUBPROJECT:-}" = minimized ]; then
310+ BUILD_NAME=minimal
311+ else
312+ BUILD_NAME=server
313+ fi
314+ cat > chroot/etc/cloud/build.info << EOF
315 build_name: $BUILD_NAME
316 serial: $BUILDSTAMP
317 EOF
318+ fi
319+
320+ configure_network_manager
321+
322+ echo "===== Checking size of /usr/share/doc ====="
323+ echo BEGIN docdirs
324+ (cd chroot && find usr/share/doc -maxdepth 1 -type d | xargs du -s | sort -nr)
325+ echo END docdirs
326+
327+ /usr/share/livecd-rootfs/minimize-manual chroot
328 fi
329
330- # If the image pre-installs network-manager, let it manage all devices by
331- # default. Installing NM on an existing system only manages wifi and wwan via
332- # /usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf. When setting
333- # the global backend to NM, netplan overrides that file.
334- if [ -d chroot/usr/lib/NetworkManager ]; then
335- echo "===== Enabling all devices in NetworkManager ===="
336- mkdir -p chroot/etc/netplan
337- cat <<EOF > chroot/etc/netplan/01-network-manager-all.yaml
338-# Let NetworkManager manage all devices on this system
339-network:
340- version: 2
341- renderer: NetworkManager
342-EOF
343+ if [ -n "${PASSES}" ]; then
344+ PATH="config/:$PATH" lb binary_layered "$@"
345 else
346- echo "==== NetworkManager not installed ===="
347+ lb binary "$@"
348 fi
349
350- echo "===== Checking size of /usr/share/doc ====="
351- echo BEGIN docdirs
352- (cd chroot && find usr/share/doc -maxdepth 1 -type d | xargs du -s | sort -nr)
353- echo END docdirs
354-
355- /usr/share/livecd-rootfs/minimize-manual chroot
356-
357- lb binary "$@"
358 touch binary.success
359 ) 2>&1 | tee binary.log
360
361@@ -504,6 +450,12 @@
362 chmod 644 "$PREFIX.$OUTPUT"
363 done
364
365+# we don't need a manifest-remove for a layered-aware installer
366+if [ "$SUBPROJECT" = "ubiquity-ng" ]; then
367+ rm -f livecd.${PROJECT}-manifest-remove
368+ rm -f config/manifest-minimal-remove
369+fi
370+
371 if [ -e config/manifest-minimal-remove ]; then
372 cp config/manifest-minimal-remove "$PREFIX.manifest-minimal-remove"
373 fi
374
375=== modified file 'live-build/auto/clean'
376--- live-build/auto/clean 2012-11-19 18:49:31 +0000
377+++ live-build/auto/clean 2018-11-28 08:47:44 +0000
378@@ -8,3 +8,5 @@
379 rm -f binary.manifest binary.manifest-desktop binary.log
380 rm -f livecd.*
381 rm -rf userdata
382+rm -rf chroot.*
383+rm -rf *.manifest.full
384
385=== modified file 'live-build/auto/config'
386--- live-build/auto/config 2018-11-23 16:39:40 +0000
387+++ live-build/auto/config 2018-11-28 08:47:44 +0000
388@@ -33,15 +33,40 @@
389
390 mkdir -p config
391 cp -af /usr/share/livecd-rootfs/live-build/functions config/functions
392+cp -af /usr/share/livecd-rootfs/live-build/lb_*_layered config/
393 cp -af /usr/share/livecd-rootfs/live-build/snap-seed-parse.py config/snap-seed-parse
394
395 mkdir -p config/package-lists
396
397+. config/functions
398+
399+OPTS=
400+COMPONENTS=
401+BINARY_REMOVE_LINUX=:
402+BINARY_IMAGES=none
403+MEMTEST=none
404+SOURCE='--source false'
405+BOOTLOADER=none
406+BOOTAPPEND_LIVE=
407+LIVE_TASK=
408+PREINSTALLED=false
409+PREINSTALL_POOL=
410+PREINSTALL_POOL_SEEDS=
411+PREFIX="livecd.$PROJECT${SUBARCH:+-$SUBARCH}"
412+
413+CHROOT_HOOKS=
414+BINARY_HOOKS=
415+
416+APT_OPTIONS=" --yes -oDebug::pkgDepCache::AutoInstall=yes "
417+
418 add_task ()
419 {
420 local pass="$1"
421 shift
422 local task
423+ local snap_list_file
424+ local snap_list_files
425+ local curseed
426
427 # The removal of direct task installation support from live-build
428 # poses some problems. If the chroot has multiarch configured - for
429@@ -58,12 +83,32 @@
430 # probably a lurking timebomb that we need to fix. In the meantime,
431 # the Architecture restriction at least saves us from abject
432 # failure.
433+ #
434+ # We want as well to grab the snap list for each PASS. Resolve for all
435+ # given task, and deduplicate them to generate snaps for the PASS.
436
437 for task; do
438 # We need a ridiculous number of backslashes to protect
439 # parentheses from eval.
440 echo "!chroot chroot apt-cache dumpavail | grep-dctrl -nsPackage \\\\\\( -XFArchitecture $ARCH -o -XFArchitecture all \\\\\\) -a -wFTask $task" >> "config/package-lists/livecd-rootfs.list.chroot_$pass"
441+
442+ curseed=$(seed_from_task ${task})
443+ if [ -z "${curseed}" ]; then
444+ echo "W: No seed matching task ${task}"
445+ continue
446+ fi
447+ snap_list_file="config/package-lists/seed.${curseed}.snaplist.full"
448+ snap_from_seed "${curseed}" $snap_list_file
449+ if [ -e "$snap_list_file" ]; then
450+ snap_list_files="${snap_list_files} $snap_list_file"
451+ fi
452 done
453+ # The snap list is one line, and could be duplicated between seeds via inheritance.
454+ # Uniquely sort them and store them back in one line.
455+ if [ -n "${snap_list_files}" ]; then
456+ cat ${snap_list_files}|xargs -n1|sort -u > "config/package-lists/livecd-rootfs.snaplist.chroot_${pass}.full"
457+ rm ${snap_list_files}
458+ fi
459 }
460
461 add_package ()
462@@ -77,24 +122,34 @@
463 done
464 }
465
466-OPTS=
467-COMPONENTS=
468-BINARY_REMOVE_LINUX=:
469-BINARY_IMAGES=none
470-MEMTEST=none
471-SOURCE='--source false'
472-BOOTLOADER=none
473-BOOTAPPEND_LIVE=
474-LIVE_TASK=
475-PREINSTALLED=false
476-PREINSTALL_POOL=
477-PREINSTALL_POOL_SEEDS=
478-PREFIX="livecd.$PROJECT${SUBARCH:+-$SUBARCH}"
479-
480-CHROOT_HOOKS=
481-BINARY_HOOKS=
482-
483-APT_OPTIONS=" --yes -oDebug::pkgDepCache::AutoInstall=yes "
484+add_layered_pass() {
485+ # Add a layer to an existing pass based on seeds matching a regexp
486+ # $1 base pass
487+ # $2 seeds (regexp)
488+
489+ for seed in $(ls config/germinate-output/|grep -P "$2"); do
490+ pass=${1}_${seed}
491+ list_packages_from_seed ${seed} >> config/package-lists/livecd-rootfs.list.chroot_$pass
492+ done
493+}
494+
495+add_layered_pass_delta() {
496+ # Add a layer to an existing pass based on delta between seeds matching a regexp and a base seed
497+ # $1 base pass
498+ # $2 base seed
499+ # $3 seeds to remove from base seed (regexp). If empty, a no-<base-seed> sublayer is generated.
500+
501+ local seed_regexp="$3"
502+ if [ -z "${seed_regexp}" ]; then
503+ substract_package_lists ${2} "" >> config/package-lists/livecd-rootfs.removal-list.chroot_${1}_no-${2}
504+ return
505+ fi
506+
507+ for seed in $(ls config/germinate-output/|grep -P "$seed_regexp"); do
508+ pass=${1}_${seed}
509+ substract_package_lists ${2} ${seed} >> config/package-lists/livecd-rootfs.removal-list.chroot_$pass
510+ done
511+}
512
513 add_chroot_hook ()
514 {
515@@ -288,30 +343,6 @@
516 OPTS="${OPTS:+$OPTS }--bootstrap-flavour=minimal --linux-packages=linux-image"
517 fi
518
519-# cribbed from cdimage, perhaps this should be a small helper script in germinate?
520-add_inheritance () {
521- case " $inherit " in
522- *" $1 "*)
523- ;;
524- *)
525- inherit="${inherit:+$inherit }$1"
526- ;;
527- esac
528-}
529-
530-expand_inheritance () {
531- for seed in $(grep "^$1:" config/germinate-output/structure | cut -d: -f2); do
532- expand_inheritance "$seed"
533- done
534- add_inheritance "$1"
535-}
536-
537-inheritance () {
538- inherit=
539- expand_inheritance "$1"
540- echo "$inherit"
541-}
542-
543 mkdir -p config/germinate-output
544 case $PROJECT in
545 kubuntu-active*)
546@@ -358,8 +389,21 @@
547
548 case $PROJECT in
549 ubuntu|ubuntu-dvd)
550- add_task install minimal standard ubuntu-desktop
551+ PASSES="install-minimal install live"
552+ add_task install-minimal minimal standard ubuntu-desktop-minimal ubuntu-desktop-minimal-default-languages
553+ add_task install ubuntu-desktop ubuntu-desktop-default-languages
554 LIVE_TASK='ubuntu-live'
555+
556+ case ${SUBPROJECT:-} in
557+ ubiquity-ng)
558+ # LANG PASS for minimal and install
559+ add_layered_pass_delta install-minimal desktop-minimal-default-languages '^desktop-minimal-(?!default-languages)[^.]+$'
560+ add_layered_pass_delta install-minimal desktop-minimal-default-languages '' # none (if no default langpack is selected)
561+ add_layered_pass_delta install desktop-default-languages '^desktop-(?!default-languages|minimal|common)[^.]+$'
562+ add_layered_pass_delta install desktop-default-languages '' # none (if no default langpack is selected)
563+ ;;
564+ esac
565+
566 case $ARCH in
567 amd64) add_package live $SIGNED_KERNEL_PACKAGE ;;
568 esac
569@@ -728,30 +772,11 @@
570 ;;
571 ubuntu-server:live)
572 BASE_SEED='server'
573- # subiquity is seeded but in a separate squashfs via hooks; set HOOK_SNAPS and ALL_SNAPS.
574- HOOK_SNAPS='subiquity'
575- ALL_SNAPS=''
576 ;;
577 esac
578
579-if [ -n "${BASE_SEED}" ]; then
580- SEEDS_EXPANDED=$(inheritance ${BASE_SEED})
581- for seed in ${SEEDS_EXPANDED}; do
582- echo "snap: considering ${seed}"
583- file=config/germinate-output/${seed}.snaps
584- [ -e "${file}" ] || continue
585- # extract the first column (snap package name) from germinate's output
586- # translate the human-readable "foo (classic)" into a
587- # more machine readable "foo/classic"
588- seed_snaps=$(sed -rn '1,/-----/d;/-----/,$d; s/(.*) \|.*/\1/; s, \(classic\),/classic,; p' "${file}")
589- for snap in ${seed_snaps}; do
590- echo "snap: found ${snap}"
591- ALL_SNAPS="${ALL_SNAPS:+${ALL_SNAPS} }${snap}"
592- done
593- done
594- if [ -n "${ALL_SNAPS}" ] || [ -n "${HOOK_SNAPS}" ]; then
595- echo "${ALL_SNAPS}" > config/seeded-snaps
596- fi
597+if [ -z "$PASSES" ] && [ -n "${BASE_SEED}" ]; then
598+ snap_from_seed "${BASE_SEED}" config/seeded-snaps
599 fi
600
601 # grab a list of packags to remove for a "minimal" installation from the seed
602@@ -890,6 +915,9 @@
603 echo "LB_CHROOT_HOOKS=\"$CHROOT_HOOKS\"" >> config/chroot
604 echo "SUBPROJECT=\"${SUBPROJECT:-}\"" >> config/chroot
605 echo "LB_DISTRIBUTION=\"$SUITE\"" >> config/chroot
606+if [ -n "$PASSES" ]; then
607+ echo "PASSES=\"$PASSES\"" >> config/common
608+fi
609 echo "LB_BINARY_HOOKS=\"$BINARY_HOOKS\"" >> config/binary
610 echo "BUILDSTAMP=\"$NOW\"" >> config/binary
611 echo "SUBPROJECT=\"${SUBPROJECT:-}\"" >> config/binary
612
613=== modified file 'live-build/functions'
614--- live-build/functions 2018-11-13 19:21:19 +0000
615+++ live-build/functions 2018-11-28 08:47:44 +0000
616@@ -373,6 +373,30 @@
617 distro-info --series="$LB_DISTRIBUTION" -r | awk '{ print $1 }'
618 }
619
620+# cribbed from cdimage, perhaps this should be a small helper script in germinate?
621+add_inheritance () {
622+ case " $inherit " in
623+ *" $1 "*)
624+ ;;
625+ *)
626+ inherit="${inherit:+$inherit }$1"
627+ ;;
628+ esac
629+}
630+
631+expand_inheritance () {
632+ for seed in $(grep "^$1:" config/germinate-output/structure | cut -d: -f2); do
633+ expand_inheritance "$seed"
634+ done
635+ add_inheritance "$1"
636+}
637+
638+inheritance () {
639+ inherit=
640+ expand_inheritance "$1"
641+ echo "$inherit"
642+}
643+
644 _snap_preseed() {
645 # Download the snap/assertion and add to the preseed
646 local CHROOT_ROOT=$1
647@@ -387,6 +411,12 @@
648
649 # Download the snap & assertion
650 local snap_download_failed=0
651+
652+ # Preseed a snap only once
653+ if [ -f ${snaps_dir}/${SNAP_NAME}_[0-9]*.snap ]; then
654+ return
655+ fi
656+
657 chroot $CHROOT_ROOT sh -c "
658 set -x;
659 cd /var/lib/snapd/seed;
660@@ -432,6 +462,10 @@
661 local account_key_assertion="$assertions_dir/account-key"
662 local account_assertion="$assertions_dir/account"
663
664+ if [ -d "$assertions_dir" ]; then
665+ return
666+ fi
667+
668 mkdir -p "$assertions_dir"
669 mkdir -p "$snaps_dir"
670
671@@ -476,15 +510,10 @@
672 # used for the image's model assertion
673 local CUSTOM_BRAND_MODEL=${2:-generic:generic-classic}
674
675- local seed_dir="$CHROOT_ROOT/var/lib/snapd/seed"
676- local snaps_dir="$seed_dir/snaps"
677-
678 snap_prepare_assertions "$CHROOT_ROOT" "$CUSTOM_BRAND_MODEL"
679
680 # Download the core snap
681- if ! [ -f $snaps_dir/core_[0-9]*.snap ] ; then
682- _snap_preseed $CHROOT_ROOT core stable
683- fi
684+ _snap_preseed $CHROOT_ROOT core stable
685 }
686
687 snap_preseed() {
688@@ -501,3 +530,166 @@
689 fi
690 _snap_preseed $CHROOT_ROOT $SNAP $CHANNEL
691 }
692+
693+snap_from_seed() {
694+ local base_seed=$1
695+ local out=$2
696+ local all_snaps
697+ local seeds_expanded
698+
699+ seeds_expanded=$(inheritance ${base_seed})
700+ for seed in ${seeds_expanded}; do
701+ echo "snap: considering ${seed}"
702+ file=config/germinate-output/${seed}.snaps
703+ [ -e "${file}" ] || continue
704+ # extract the first column (snap package name) from germinate's output
705+ # translate the human-readable "foo (classic)" into a
706+ # more machine readable "foo/classic"
707+ seed_snaps=$(sed -rn '1,/-----/d;/-----/,$d; s/(.*) \|.*/\1/; s, \(classic\),/classic,; p' "${file}")
708+ for snap in ${seed_snaps}; do
709+ echo "snap: found ${snap}"
710+ all_snaps="${all_snaps:+${all_snaps} }${snap}"
711+ done
712+ done
713+ if [ -n "${all_snaps}" ]; then
714+ echo "${all_snaps}" > $out
715+ fi
716+}
717+
718+seed_from_task ()
719+{
720+ # Retrieve the name of the seed from a task name
721+ local task=$1
722+ local seed
723+ local seedfile
724+ local seedfiles
725+
726+ seedfile="$(grep -lE "^Task-Key: +${task}\$" config/germinate-output/*seedtext|head -1)"
727+ if [ -n "$seedfile" ]; then
728+ basename $seedfile .seedtext
729+ return
730+ fi
731+
732+ seedfiles="$(grep -lE "^Task-Per-Derivative: *1\$" config/germinate-output/*seedtext)"
733+ if [ -n "$seedfiles" ]; then
734+ for seed in $(echo $seedfiles | xargs basename -s .seedtext); do
735+ if [ ${PROJECT}-${seed} = $task ]; then
736+ echo ${seed}
737+ return
738+ fi
739+ done
740+ fi
741+}
742+
743+list_packages_from_seed () {
744+ # Store all packages for a given seed, including its seed dependency
745+ # $1: Name of the seed to expand to a package list
746+
747+ local all_seeds="$(inheritance $1)"
748+
749+ for seed in $all_seeds; do
750+ head -n-2 config/germinate-output/${seed}.seed|tail -n+3|awk '{print $1}'
751+ done|sort -u
752+}
753+
754+substract_package_lists() {
755+ # Substract a package list from another
756+ #
757+ # $1 source package list
758+ # $2 Package list to substract from source package list
759+ local list1=$(mktemp)
760+ local list2=$(mktemp)
761+
762+ list_packages_from_seed $1 > list1
763+ list_packages_from_seed $2 > list2
764+ comm -23 list1 list2
765+
766+ rm list1
767+ rm list2
768+}
769+
770+clean_debian_chroot() {
771+ # remove crufty files that shouldn't be left in an image
772+ rm -f chroot/var/cache/debconf/*-old chroot/var/lib/dpkg/*-old
773+ Chroot chroot apt clean
774+}
775+
776+configure_universe() {
777+ if [ -f config/universe-enabled ]; then
778+ # This is cargo-culted almost verbatim (with some syntax changes for
779+ # preinstalled being slightly different in what it doesn't ask) from
780+ # debian-installer's apt-setup:
781+
782+ cat > chroot/etc/apt/sources.list << EOF
783+# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
784+# newer versions of the distribution.
785+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION main restricted
786+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION main restricted
787+
788+## Major bug fix updates produced after the final release of the
789+## distribution.
790+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates main restricted
791+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates main restricted
792+
793+## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
794+## team. Also, please note that software in universe WILL NOT receive any
795+## review or updates from the Ubuntu security team.
796+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION universe
797+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION universe
798+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates universe
799+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates universe
800+
801+## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
802+## team, and may not be under a free licence. Please satisfy yourself as to
803+## your rights to use the software. Also, please note that software in
804+## multiverse WILL NOT receive any review or updates from the Ubuntu
805+## security team.
806+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION multiverse
807+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION multiverse
808+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates multiverse
809+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-updates multiverse
810+
811+## N.B. software from this repository may not have been tested as
812+## extensively as that contained in the main release, although it includes
813+## newer versions of some applications which may provide useful features.
814+## Also, please note that software in backports WILL NOT receive any review
815+## or updates from the Ubuntu security team.
816+deb $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-backports main restricted universe multiverse
817+# deb-src $LB_PARENT_MIRROR_BINARY $LB_DISTRIBUTION-backports main restricted universe multiverse
818+
819+## Uncomment the following two lines to add software from Canonical's
820+## 'partner' repository.
821+## This software is not part of Ubuntu, but is offered by Canonical and the
822+## respective vendors as a service to Ubuntu users.
823+# deb http://archive.canonical.com/ubuntu $LB_DISTRIBUTION partner
824+# deb-src http://archive.canonical.com/ubuntu $LB_DISTRIBUTION partner
825+
826+deb $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security main restricted
827+# deb-src $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security main restricted
828+deb $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security universe
829+# deb-src $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security universe
830+deb $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security multiverse
831+# deb-src $LB_PARENT_MIRROR_BINARY_SECURITY $LB_DISTRIBUTION-security multiverse
832+EOF
833+
834+fi
835+}
836+
837+configure_network_manager() {
838+ # If the image pre-installs network-manager, let it manage all devices by
839+ # default. Installing NM on an existing system only manages wifi and wwan via
840+ # /usr/lib/NetworkManager/conf.d/10-globally-managed-devices.conf. When setting
841+ # the global backend to NM, netplan overrides that file.
842+ if [ -d chroot/usr/lib/NetworkManager ]; then
843+ echo "===== Enabling all devices in NetworkManager ===="
844+ mkdir -p chroot/etc/netplan
845+ cat <<EOF > chroot/etc/netplan/01-network-manager-all.yaml
846+# Let NetworkManager manage all devices on this system
847+network:
848+ version: 2
849+ renderer: NetworkManager
850+EOF
851+ else
852+ echo "==== NetworkManager not installed ===="
853+ fi
854+}
855
856=== added file 'live-build/lb_binary_layered'
857--- live-build/lb_binary_layered 1970-01-01 00:00:00 +0000
858+++ live-build/lb_binary_layered 2018-11-28 08:47:44 +0000
859@@ -0,0 +1,132 @@
860+#!/bin/sh
861+
862+## live-build(7) - System Build Scripts
863+## Copyright (C) 2006-2012 Daniel Baumann <daniel@debian.org>
864+##
865+## This program comes with ABSOLUTELY NO WARRANTY; for details see COPYING.
866+## This is free software, and you are welcome to redistribute it
867+## under certain conditions; see COPYING for details.
868+
869+
870+set -e
871+
872+# Including common functions
873+( . "${LIVE_BUILD}/scripts/build.sh" > /dev/null 2>&1 || true ) || . /usr/lib/live/build.sh
874+
875+
876+# Automatically populating config tree
877+if [ -x auto/config ] && [ ! -e .build/config ]
878+then
879+ Echo_message "Automatically populating config tree."
880+ lb config
881+fi
882+
883+# Setting static variables
884+DESCRIPTION="$(Echo 'build binary images')"
885+HELP=""
886+USAGE="${PROGRAM} [--force]"
887+
888+Arguments "${@}"
889+
890+# Reading configuration files
891+Read_conffiles config/all config/common config/bootstrap config/chroot config/binary config/source
892+Set_defaults
893+
894+# Setup cleanup function
895+Setup_cleanup
896+
897+. config/functions
898+
899+build_layered_squashfs() {
900+ local pass=$1 # install|install_subpass|install_subpass_subsubpass|…
901+ local prevpass=$2 # install|install_subpass|…
902+ local prefix=$3 # 01-|02-|…
903+ local lowerlayers=$4
904+ shift 4 # restore ${*}
905+
906+ # Cleanup root filesystem
907+ lb binary_chroot ${*}
908+
909+ # Building squashfs filesystem & manifest
910+ base="${PWD}/livecd.${PROJECT}.${prefix}${pass}"
911+ squashfs_f="${base}.squashfs"
912+
913+ # We have already treated that pass
914+ if [ -f "${squashfs_f}" ]; then
915+ return
916+ fi
917+
918+ if [ -n "${lowerlayers}" ]; then
919+ mount_overlay ${lowerlayers} "chroot.${pass}/" chroot/
920+ else
921+ # first pass
922+ rmdir chroot 2>/dev/null||true
923+ ln -s "chroot.${pass}/" chroot
924+ fi
925+
926+ # Full manifest until that PASS
927+ squashfs_f_manifest="${base}.manifest"
928+ create_manifest "chroot" "${squashfs_f_manifest}.full"
929+
930+ # Delta manifest
931+ diff -NU0 ${PWD}/livecd.${PROJECT}.[0-9][0-9]-${prevpass}.manifest.full ${squashfs_f_manifest}.full|grep -v ^@ > $squashfs_f_manifest
932+
933+ squashfs_f_size="${base}.size"
934+ du -B 1 -s "chroot.${pass}/" | cut -f1 > "${squashfs_f_size}"
935+
936+ (cd "chroot.${pass}/" &&
937+ mksquashfs . ${squashfs_f} \
938+ -no-progress -xattrs -comp xz )
939+
940+ if [ -n "${lowerlayers}" ]; then
941+ umount chroot
942+ else
943+ rm chroot
944+ mkdir chroot/
945+ fi
946+
947+ # Handle direct sublayer of current one
948+ # Extract the name of the pass corresponding to the sublayer
949+ for subpass in $(ls -d chroot.${pass}_* 2>/dev/null | sed -e "s/chroot\.\(${pass}_[^_]\+\).*/\1/"); do
950+ lowerlayers_for_subpass="chroot.${pass}:${lowerlayers}"
951+ lowerlayers_for_subpass="${lowerlayers_for_subpass%:}"
952+ build_layered_squashfs "${subpass}" "${pass}" "${prefix}" "${lowerlayers_for_subpass}" ${*}
953+ done
954+}
955+
956+CURPASS=1
957+PREVPASS=""
958+PASSPREFIX=""
959+LOWER_LAYERS=""
960+for _PASS in $PASSES
961+do
962+ PASSPREFIX="$(printf "%02g" $CURPASS)-"
963+
964+ build_layered_squashfs "${_PASS}" "${PREVPASS}" "$PASSPREFIX" "${LOWER_LAYERS}" ${*}
965+
966+ LOWER_LAYERS="chroot.${_PASS}:$LOWER_LAYERS"
967+ LOWER_LAYERS="${LOWER_LAYERS%:}"
968+ PREVPASS=${_PASS}
969+ CURPASS=$(( CURPASS + 1 ))
970+done
971+
972+# remount last "main" pass on chroot for lb binary
973+mount_overlay "${LOWER_LAYERS}" "chroot.${_PASS}/" chroot/
974+
975+# Prepare initrd + kernel
976+lb binary_linux-image ${*}
977+
978+umount chroot/
979+
980+# Full ISO manifest & size from last main PASS
981+PREFIX="livecd.$PROJECT${SUBARCH:+-$SUBARCH}"
982+cp "livecd.${PROJECT}.${PASSPREFIX}${_PASS}.size" "$PREFIX.size"
983+cp "livecd.${PROJECT}.${PASSPREFIX}${_PASS}.manifest.full" "$PREFIX.manifest"
984+
985+# Ubiquity-compatible removal manifest for ISO not using a layered-aware installer
986+if [ -n "$(ls livecd.${PROJECT}.[0-9][0-9]-live.manifest.full 2>/dev/null)" ] && \
987+ [ -n "$(ls livecd.${PROJECT}.[0-9][0-9]-install.manifest.full 2>/dev/null)" ]; then
988+ echo "$(diff livecd.${PROJECT}.[0-9][0-9]-live.manifest.full livecd.${PROJECT}.[0-9][0-9]-install.manifest.full | awk '/^< / { print $2 }')" > livecd.${PROJECT}-manifest-remove
989+fi
990+
991+chmod 644 *.squashfs *.manifest* *.size
992
993=== added file 'live-build/lb_chroot_layered'
994--- live-build/lb_chroot_layered 1970-01-01 00:00:00 +0000
995+++ live-build/lb_chroot_layered 2018-11-28 08:47:44 +0000
996@@ -0,0 +1,246 @@
997+#!/bin/sh
998+
999+## live-build(7) - System Build Scripts
1000+## Copyright (C) 2006-2012 Daniel Baumann <daniel@debian.org>
1001+##
1002+## This program comes with ABSOLUTELY NO WARRANTY; for details see COPYING.
1003+## This is free software, and you are welcome to redistribute it
1004+## under certain conditions; see COPYING for details.
1005+
1006+## This is a fork of lb_chroot for layered live system.
1007+## We don't want leaking host configuratino in each layer, and so,
1008+## we clean and setup the chroot each time.
1009+## In addition, we create the squashfs for each layer, but top one (live)
1010+## which still can be configured after lb chroot call.
1011+
1012+set -e
1013+
1014+# Including common functions
1015+( . "${LIVE_BUILD}/scripts/build.sh" > /dev/null 2>&1 || true ) || . /usr/lib/live/build.sh
1016+
1017+# Automatically populating config tree
1018+if [ -x auto/config ] && [ ! -e .build/config ]
1019+then
1020+ Echo_message "Automatically populating config tree."
1021+ lb config
1022+fi
1023+
1024+# Setting static variables
1025+DESCRIPTION="$(Echo 'customize the Debian system')"
1026+HELP=""
1027+USAGE="${PROGRAM} [--force]"
1028+
1029+Arguments "${@}"
1030+
1031+# Reading configuration files
1032+Read_conffiles config/all config/common config/bootstrap config/chroot config/binary config/source
1033+Set_defaults
1034+
1035+# Setup cleanup function
1036+Setup_cleanup
1037+
1038+. config/functions
1039+
1040+lb_chroot_remove_packages() {
1041+ # Remove packages from the chroot specific to this layer
1042+ #
1043+ # $1: Name of the pass*
1044+ local pass=$1
1045+
1046+ Expand_packagelist "$(basename config/package-lists/*.removal-list.chroot_${pass})" "config/package-lists" \
1047+ >> chroot/root/packages.chroot.removal
1048+ Chroot chroot "xargs --arg-file=/root/packages.chroot.removal apt-get ${APT_OPTIONS} autoremove --purge"
1049+ rm -f chroot/root/packages.chroot.removal
1050+}
1051+
1052+# Create the snap list specific to this layer
1053+lb_chroot_snap_lists() {
1054+ local pass=$1
1055+ local prevpass=$2
1056+
1057+ # This assumes that the prefix is unique for a given project
1058+ local snap_for_pass=$(ls config/package-lists/*.snaplist.chroot_${pass}.full 2>/dev/null || true)
1059+ local snap_for_prevpass=$(ls config/package-lists/*.snaplist.chroot_${prevpass}.full 2>/dev/null || true)
1060+
1061+ if [ -z "${snap_for_pass}" ]; then
1062+ return
1063+ fi
1064+
1065+ if [ -z "${snap_for_prevpass}" ]; then
1066+ cp ${snap_for_pass} ${snap_for_pass%.full}
1067+ return
1068+ fi
1069+
1070+ # Generate a list of snaps added to a layer.
1071+ diff -NU0 ${snap_for_prevpass} ${snap_for_pass}|grep -Ev '^(---|\+\+\+|@@)'|cut -c2- > ${snap_for_pass%.full}
1072+}
1073+
1074+lb_chroot_install_snaps() {
1075+ # Prepare the snap environment and install snaps into a chroot
1076+ #
1077+ # $1: Name of the pass
1078+
1079+ local snaplist_file=$(ls config/package-lists/*.snaplist.chroot_${1} 2>/dev/null || true)
1080+
1081+ if [ -z "${snaplist_file}" ]; then
1082+ return
1083+ fi
1084+
1085+ snap_prepare chroot
1086+
1087+ while read snap; do
1088+ snap_preseed chroot "${snap}"
1089+ done < $snaplist_file
1090+}
1091+
1092+lb_chroot_includes() {
1093+ # Copying includes from pass subdirectory
1094+ local pass="$1"
1095+
1096+ if [ ! -d config/includes.chroot.${pass} ]; then
1097+ return
1098+ fi
1099+
1100+ cd config/includes.chroot.${pass}
1101+ find . | cpio -dmpu --no-preserve-owner "${OLDPWD}"/chroot
1102+ cd "${OLDPWD}"
1103+}
1104+
1105+create_chroot_pass() {
1106+ local pass=$1
1107+ local prevpass=$2
1108+ local lowerlayers=$3
1109+ local passtype=$4 # "first"|"last"|"" empty string
1110+ shift 4 # restore ${*}
1111+
1112+ # We have already treated that pass
1113+ if [ -d "chroot.${pass}/" ]; then
1114+ return
1115+ fi
1116+
1117+ export PASS=${pass}
1118+
1119+ if [ "${passtype}" != "first" ]; then
1120+ mkdir chroot.${pass}
1121+ mount_overlay ${lowerlayers} "chroot.${pass}/" chroot/
1122+ fi
1123+
1124+ # Configuring chroot
1125+ lb chroot_cache restore ${*}
1126+ lb chroot_devpts install ${*}
1127+ lb chroot_proc install ${*}
1128+ lb chroot_sysfs install ${*}
1129+ lb chroot_debianchroot install ${*}
1130+ lb chroot_dpkg install ${*}
1131+ lb chroot_tmpfs install ${*}
1132+ lb chroot_hosts install ${*}
1133+ lb chroot_resolv install ${*}
1134+ lb chroot_hostname install ${*}
1135+ lb chroot_apt install ${*}
1136+ # Note: this triggers an upgrade + dist-ugprade; which may impact sublayers with more
1137+ # diff content than desired. However, we still need to setup the archive and teardown
1138+ # for each layer.
1139+ # We could modify livebuild if necessary to have conditional upgrade (first pass only).
1140+ lb chroot_archives chroot install ${*}
1141+
1142+ if [ "${passtype}" = "first" ]; then
1143+ configure_universe
1144+ fi
1145+
1146+ # Customizing chroot
1147+ lb chroot_linux-image ${*}
1148+ lb chroot_preseed ${*}
1149+ lb chroot_early_hooks ${*}
1150+
1151+ lb chroot_package-lists ${pass} ${*}
1152+ lb chroot_install-packages ${pass} ${*}
1153+ lb_chroot_remove_packages ${pass} ${*}
1154+
1155+ # Snap management
1156+ lb_chroot_snap_lists ${pass} ${prevpass}
1157+ lb_chroot_install_snaps ${pass} ${*}
1158+
1159+ # Kernel should be in first layer
1160+ if [ "${passtype}" = "first" ]; then
1161+ configure_network_manager
1162+ Chroot chroot "dpkg -l linux-headers-3* linux-headers-4*" 2>/dev/null \
1163+ | awk '/^i/ {print $2}' > chroot.headers
1164+ for i in $(cat chroot.headers); do
1165+ Chroot chroot "apt-mark auto $i"
1166+ done
1167+ fi
1168+
1169+ Chroot chroot "apt-get --purge -y autoremove"
1170+
1171+ # Add live packages to top layer
1172+ if [ "${passtype}" = "last" ]; then
1173+ lb chroot_live-packages ${*}
1174+ fi
1175+
1176+ # Run includes by pass
1177+ lb_chroot_includes ${pass} ${*}
1178+
1179+ lb chroot_hooks ${*}
1180+ lb chroot_hacks ${*}
1181+ lb chroot_interactive ${*}
1182+
1183+ # Misc ubuntu cleanup and post-layer configuration
1184+ clean_debian_chroot
1185+ /usr/share/livecd-rootfs/minimize-manual chroot
1186+
1187+ Chroot chroot "dpkg-query -W" > chroot.packages.${pass}
1188+
1189+ # Deconfiguring chroot
1190+ lb chroot_archives chroot remove ${*}
1191+ lb chroot_apt remove ${*}
1192+ lb chroot_hostname remove ${*}
1193+ lb chroot_resolv remove ${*}
1194+ lb chroot_hosts remove ${*}
1195+ lb chroot_tmpfs remove ${*}
1196+ lb chroot_dpkg remove ${*}
1197+ lb chroot_debianchroot remove ${*}
1198+ lb chroot_sysfs remove ${*}
1199+ lb chroot_proc remove ${*}
1200+ lb chroot_devpts remove ${*}
1201+ lb chroot_cache save ${*}
1202+
1203+ if [ "${passtype}" = "first" ]; then
1204+ mv chroot chroot.${pass}
1205+ mkdir chroot
1206+ else
1207+ umount chroot
1208+ fi
1209+
1210+ # Handle direct sublayer of current one
1211+ # Extract the name of the pass corresponding to the sublayer
1212+ for subpass in $(ls config/package-lists/*list.chroot_${pass}_* 2>/dev/null | sed -e "s/.*list\.chroot_\(${pass}_[^_]\+\).*/\1/"); do
1213+ lowerlayers_for_subpass="chroot.${pass}:${lowerlayers}"
1214+ lowerlayers_for_subpass="${lowerlayers_for_subpass%:}"
1215+ create_chroot_pass "${subpass}" "${pass}" "${lowerlayers_for_subpass}" "" ${*}
1216+ done
1217+}
1218+
1219+PASSES="${PASSES:-install live}"
1220+CURPASS=1
1221+PREVPASS=""
1222+LASTPASS=$(echo $PASSES|wc -w)
1223+LOWER_LAYERS=""
1224+for _PASS in $PASSES
1225+do
1226+ PASSTYPE=""
1227+ if [ $CURPASS -eq 1 ]; then
1228+ PASSTYPE="first"
1229+ elif [ $CURPASS -eq $LASTPASS ]; then
1230+ PASSTYPE="last"
1231+ fi
1232+
1233+ create_chroot_pass "$_PASS" "$PREVPASS" "$LOWER_LAYERS" "$PASSTYPE" ${*}
1234+
1235+ LOWER_LAYERS="chroot.${_PASS}:$LOWER_LAYERS"
1236+ LOWER_LAYERS="${LOWER_LAYERS%:}"
1237+ PREVPASS=${_PASS}
1238+
1239+ CURPASS=$(( CURPASS + 1 ))
1240+done
1241+
1242+rmdir chroot

Subscribers

People subscribed via source and target branches