Merge lp:~cjwatson/livecd-rootfs/buildd into lp:livecd-rootfs

[Colin Watson]

Add a buildd subproject.

This can be used for building buildd chroots. The necessary launchpad-buildd metadata dictionary is:

  {"project": "ubuntu-base", "subproject": "buildd", "image_format": "none"}

The differences from a baseline bionic amd64 chroot look sensible. I'm still working on testing this on more architectures, and we may want to backport it to earlier series eventually, but it's at least ready for initial review.

[lp:~cjwatson/livecd-rootfs/buildd]

1670. By Colin Watson on 2018-04-29

Use a more generic version number.

1671. By Colin Watson on 2018-04-29

Use a public-facing mirror URL in the final buildd /etc/apt/sources.list.

1672. By Colin Watson on 2018-04-29

Sort buildd tarballs by name and use numeric user and group IDs.

1673. By Colin Watson on 2018-04-29

Fix sedding of /etc/apt/sources.list.

[Michael Hudson-Doyle (mwhudson)]

This looks OK to me but I don't know if I'm qualified for a full review. A few comments mostly about stuff I think is redundant.

[lp:~cjwatson/livecd-rootfs/buildd]

1674. By Colin Watson on 2018-04-30

Remove some unnecessary buildd configuration.

1675. By Colin Watson on 2018-04-30

Move some buildd hooks to includes.chroot.

1676. By Colin Watson on 2018-04-30

Explicitly order remaining buildd hooks.

[Colin Watson (cjwatson)]

Thanks - your suggestions all seem reasonable, and I've applied them all.

[lp:~cjwatson/livecd-rootfs/buildd]

1677. By Colin Watson on 2018-04-30

Add init to buildd chroots, since it's needed for LXD-based builds.

1678. By Colin Watson on 2018-05-01

Add an empty /etc/resolv.conf to buildd chroots.

We don't want the symlink to /run/systemd/resolve/stub-resolv.conf, and
launchpad-buildd will inject its own version if it doesn't get confused by a
symlink.

1679. By Colin Watson on 2018-06-29

We know cosmic's name now.

1680. By Colin Watson on 2018-06-29

Merge trunk.

1681. By Colin Watson on 2018-09-19

Generate tar file with --xattrs.

1682. By Colin Watson on 2018-09-19

Merge trunk.

[Adam Conrad (adconrad)]

LGTM on the Nth pass of reviewing this, please make with the merging and uploading.

review: Approve

=== modified file 'debian/changelog'
--- debian/changelog	2018-09-18 10:15:23 +0000
+++ debian/changelog	2018-09-19 14:24:39 +0000
@@ -1,3 +1,9 @@
+livecd-rootfs (2.538) UNRELEASED; urgency=medium
+
+  * Add a buildd subproject.
+
+ -- Colin Watson <[e-mail]>  Wed, 19 Sep 2018 16:19:58 +0200
+
 livecd-rootfs (2.537) cosmic; urgency=medium
 
   * Minimize the number of manually installed packages in images by marking

=== modified file 'live-build/auto/build'
--- live-build/auto/build	2018-09-18 09:44:06 +0000
+++ live-build/auto/build	2018-09-19 14:24:39 +0000
@@ -497,6 +497,16 @@
 	chmod 644 "$PREFIX.rootfs.tar.gz"
 elif [ -e binary-tar.tar.gz ]; then
 	cp -a binary-tar.tar.gz "$PREFIX.rootfs.tar.gz"
+elif [ "$SUBPROJECT" = buildd ]; then
+	# A few things (launchpad-buildd, sbuild-launchpad-chroot) rely on
+	# the top-level directory being "chroot-autobuild", so we have to do
+	# this ourselves.
+	# gzip was chosen for fastest decompression speed: it decompresses
+	# buildd chroots about twice as fast as xz and about five times as
+	# fast as bzip2.
+	tar --transform='s,^binary,chroot-autobuild,' \
+		--sort=name --numeric-owner --xattrs \
+		-czf "$PREFIX.rootfs.tar.gz" binary
 fi
 
 if [ "$PROJECT:${SUBPROJECT:-}" = "ubuntu-core:system-image" ]; then

=== modified file 'live-build/auto/config'
--- live-build/auto/config	2018-08-17 00:01:27 +0000
+++ live-build/auto/config	2018-09-19 14:24:39 +0000
@@ -642,6 +642,54 @@
 		;;
 esac
 
+case $SUBPROJECT in
+	buildd)
+		OPTS="${OPTS:+$OPTS }--archive-areas main"
+		COMPONENTS='main restricted universe multiverse'
+		OPTS="${OPTS:+$OPTS }--apt-recommends false"
+		OPTS="${OPTS:+$OPTS }--apt-secure false"
+		OPTS="${OPTS:+$OPTS }--parent-mirror-binary ${MIRROR}"
+		# XXX cjwatson 2018-04-27: We need to work out how to make
+		# this conditional so that we can do things like building
+		# buildd chroots with -updates.  This probably involves
+		# either extending the PROPOSED hack or fixing the strange
+		# way that SUITE is in fact a series; in either case it's
+		# likely to involve work both here and in launchpad-buildd.
+		OPTS="${OPTS:+$OPTS }--security false --volatile false"
+
+		add_package install adduser
+		add_package install policyrcd-script-zg2
+		add_package install pkgbinarymangler
+		add_package install ca-certificates
+		add_package install gpg
+		add_package install gpg-agent
+		case $SUITE in
+			precise|trusty|xenial)
+				# no longer needed in >= artful
+				add_package install pkg-create-dbgsym
+				# no longer needed in >= bionic
+				add_package install apt-transport-https
+				# no longer needed in >= cosmic
+				add_package install tzdata
+				;;
+			artful)
+				# no longer needed in >= bionic
+				add_package install apt-transport-https
+				# no longer needed in >= cosmic
+				add_package install tzdata
+				;;
+			bionic)
+				# no longer needed in >= cosmic
+				add_package install tzdata
+				;;
+		esac
+		add_package install fakeroot
+		add_package install build-essential
+		# Needed for LXD-based builds.
+		add_package install init
+		;;
+esac
+
 # we'll expand the base seed given here according to the STRUCTURE file, and
 # then look in all of the seeds found to see which snaps are seeded
 case $PROJECT:${SUBPROJECT:-} in
@@ -946,6 +994,12 @@
 		;;
 esac
 
+case $SUBPROJECT in
+	buildd)
+		cp -af /usr/share/livecd-rootfs/live-build/buildd/* config/
+		;;
+esac
+
 if [ "$EXTRA_PPAS" ]; then
 	rm -f config/archives/extra-ppas.list.chroot \
 	      config/archives/extra-ppas.pref.chroot \

=== added directory 'live-build/buildd'
=== added directory 'live-build/buildd/hooks'
=== added file 'live-build/buildd/hooks/00-kernel-img.chroot'
--- live-build/buildd/hooks/00-kernel-img.chroot	1970-01-01 00:00:00 +0000
+++ live-build/buildd/hooks/00-kernel-img.chroot	2018-09-19 14:24:39 +0000
@@ -0,0 +1,5 @@
+#! /bin/sh
+set -e
+
+# At one point, kernel builds needed this.
+echo do_initrd = Yes >>/etc/kernel-img.conf

=== added file 'live-build/buildd/hooks/00-mirror.binary'
--- live-build/buildd/hooks/00-mirror.binary	1970-01-01 00:00:00 +0000
+++ live-build/buildd/hooks/00-mirror.binary	2018-09-19 14:24:39 +0000
@@ -0,0 +1,12 @@
+#! /bin/sh
+set -e
+
+. config/bootstrap
+
+# Use a public-facing mirror URL, for the benefit of
+# sbuild-launchpad-chroot.  We deliberately do this only after live-build
+# has run "apt-get update" for the last time, in order that
+# /var/lib/apt/lists/ has suitable cached Packages files; this speeds up
+# builds on buildds.
+sed -i "s,${LB_PARENT_MIRROR_BINARY},${LB_MIRROR_BINARY},g" \
+	binary/etc/apt/sources.list

=== added file 'live-build/buildd/hooks/01-pkgbinarymangler.chroot'
--- live-build/buildd/hooks/01-pkgbinarymangler.chroot	1970-01-01 00:00:00 +0000
+++ live-build/buildd/hooks/01-pkgbinarymangler.chroot	2018-09-19 14:24:39 +0000
@@ -0,0 +1,10 @@
+#! /bin/sh
+set -e
+
+# Configure pkgbinarymangler.
+sed -i /^enable/s/false/true/ \
+	/etc/pkgbinarymangler/maintainermangler.conf \
+	/etc/pkgbinarymangler/striptranslations.conf || true
+sed -i /^invalid_current/s/ignore/fail/ \
+	/etc/pkgbinarymangler/maintainermangler.conf \
+	/etc/pkgbinarymangler/striptranslations.conf || true

=== added file 'live-build/buildd/hooks/02-user.chroot'
--- live-build/buildd/hooks/02-user.chroot	1970-01-01 00:00:00 +0000
+++ live-build/buildd/hooks/02-user.chroot	2018-09-19 14:24:39 +0000
@@ -0,0 +1,9 @@
+#! /bin/sh
+set -e
+
+# Create the buildd user and group.
+addgroup --gid 2501 buildd
+adduser --system --disabled-password --gecos 'Build Daemon user' \
+	--ingroup buildd --uid 2001 --shell /bin/bash buildd
+mkdir -p /build/buildd
+chown buildd:buildd /build/buildd

=== added directory 'live-build/buildd/includes.chroot'
=== added directory 'live-build/buildd/includes.chroot/etc'
=== added directory 'live-build/buildd/includes.chroot/etc/apt'
=== added directory 'live-build/buildd/includes.chroot/etc/apt/apt.conf.d'
=== added file 'live-build/buildd/includes.chroot/etc/apt/apt.conf.d/99buildd'
--- live-build/buildd/includes.chroot/etc/apt/apt.conf.d/99buildd	1970-01-01 00:00:00 +0000
+++ live-build/buildd/includes.chroot/etc/apt/apt.conf.d/99buildd	2018-09-19 14:24:39 +0000
@@ -0,0 +1,2 @@
+DPkg::Options {"--force-unsafe-io";};
+DPkg::Use-Pty "false";

=== added directory 'live-build/buildd/includes.chroot/etc/apt/preferences.d'
=== added file 'live-build/buildd/includes.chroot/etc/apt/preferences.d/backports'
--- live-build/buildd/includes.chroot/etc/apt/preferences.d/backports	1970-01-01 00:00:00 +0000
+++ live-build/buildd/includes.chroot/etc/apt/preferences.d/backports	2018-09-19 14:24:39 +0000
@@ -0,0 +1,3 @@
+Package: *
+Pin: release a=*-backports
+Pin-Priority: 500

=== added file 'live-build/buildd/includes.chroot/etc/fstab'
--- live-build/buildd/includes.chroot/etc/fstab	1970-01-01 00:00:00 +0000
+++ live-build/buildd/includes.chroot/etc/fstab	2018-09-19 14:24:39 +0000
@@ -0,0 +1,1 @@
+/dev/root / ext2 noatime,errors=remount-ro 0 1

=== added file 'live-build/buildd/includes.chroot/etc/hostname'
--- live-build/buildd/includes.chroot/etc/hostname	1970-01-01 00:00:00 +0000
+++ live-build/buildd/includes.chroot/etc/hostname	2018-09-19 14:24:39 +0000
@@ -0,0 +1,1 @@
+INVALID

=== added file 'live-build/buildd/includes.chroot/etc/hosts'
--- live-build/buildd/includes.chroot/etc/hosts	1970-01-01 00:00:00 +0000
+++ live-build/buildd/includes.chroot/etc/hosts	2018-09-19 14:24:39 +0000
@@ -0,0 +1,9 @@
+127.0.0.1	localhost.localdomain	localhost
+
+# The following lines are desirable for IPv6 capable hosts
+::1     ip6-localhost ip6-loopback
+fe00::0 ip6-localnet
+ff00::0 ip6-mcastprefix
+ff02::1 ip6-allnodes
+ff02::2 ip6-allrouters
+ff02::3 ip6-allhosts

=== added file 'live-build/buildd/includes.chroot/etc/resolv.conf'
=== added directory 'live-build/buildd/includes.chroot/usr'
=== added directory 'live-build/buildd/includes.chroot/usr/local'
=== added directory 'live-build/buildd/includes.chroot/usr/local/sbin'
=== added file 'live-build/buildd/includes.chroot/usr/local/sbin/policy-rc.d'
--- live-build/buildd/includes.chroot/usr/local/sbin/policy-rc.d	1970-01-01 00:00:00 +0000
+++ live-build/buildd/includes.chroot/usr/local/sbin/policy-rc.d	2018-09-19 14:24:39 +0000
@@ -0,0 +1,13 @@
+#!/bin/sh
+
+# policy-rc.d script for chroots.
+# Copyright (c) 2007 Peter Palfrader <[e-mail]>
+# License: <weasel> MIT, if you want one.
+
+while true; do
+  case "$1" in
+    -*)         shift ;;
+    makedev)    exit 0;;
+    *) echo "Not running services in chroot."; exit 101 ;;
+  esac
+done

=== added directory 'live-build/buildd/preseed'
=== added file 'live-build/buildd/preseed/debconf.preseed'
--- live-build/buildd/preseed/debconf.preseed	1970-01-01 00:00:00 +0000
+++ live-build/buildd/preseed/debconf.preseed	2018-09-19 14:24:39 +0000
@@ -0,0 +1,2 @@
+# We never want debconf interaction.
+debconf debconf/frontend select Noninteractive

=== added file 'live-build/buildd/preseed/man-db.preseed'
--- live-build/buildd/preseed/man-db.preseed	1970-01-01 00:00:00 +0000
+++ live-build/buildd/preseed/man-db.preseed	2018-09-19 14:24:39 +0000
@@ -0,0 +1,3 @@
+# Avoid unnecessary manual page database builds (see
+# https://bugs.debian.org/554914).
+man-db man-db/auto-update boolean false

=== added file 'live-build/buildd/preseed/sun-java6.preseed'
--- live-build/buildd/preseed/sun-java6.preseed	1970-01-01 00:00:00 +0000
+++ live-build/buildd/preseed/sun-java6.preseed	2018-09-19 14:24:39 +0000
@@ -0,0 +1,3 @@
+# Pre-accept interactive EULA prompts.
+sun-java6-bin shared/accepted-sun-dlj-v1-1 boolean true
+sun-java6-jre shared/accepted-sun-dlj-v1-1 boolean true