Merge lp:~ubuntu-core-dev/livecd-rootfs/split-custom-tarball into lp:livecd-rootfs

[Steve Langasek]

First cut at separating a custom tarball out of the ubuntu-touch rootfs
generation. This is a bit ugly because it hard-codes the list of click
packages in the config, but it lets us ensure consistent handling of the
packages in the rootfs vs. the custom tarball with no duplication.

[Colin Watson (cjwatson)]

I think this looks OK with just one small inline comment, and ubuntu-cdimage and ubuntu-system-image/server should now be ready for it. What testing have you been able to do of this, if any?

We'll need to be pretty careful about landing this, though, and it's all-or-nothing right now. Stéphane had a suggestion to make this easier which I think is worth doing. Now that click-apparmor has been fixed, it should be safe to include these click packages in both rootfs and custom temporarily (obviously incurring a space penalty on devices in the interim). That would then give us the following landing procedure:

  1) land a modified version of this branch which includes packages in both rootfs and custom; don't change system-image configuration yet
  2) once step 1 is done, build an image; check that relevant packages are still usable on a device (i.e. in rootfs); inspect published custom.tar.gz by hand and make any necessary corrections
  3) modify all relevant custom tarballs built elsewhere to include the packages due to be removed from the rootfs (can be done in parallel with steps 1 and 2)
  4) once all of steps 1, 2, and 3 are done, switch system-image configuration to use cdimage-custom
  5) once step 4 is done, build an image; check that relevant packages are still usable on a device, and that they are present in /custom
  6) once step 5 is done, land a further livecd-rootfs change that drops the packages from rootfs
  7) once step 6 is done, build an image; check that relevant packages are still usable on a device, and that they are only present in /custom and not rootfs

review: Needs Fixing

[Chris Wayne (cwayne)]

As per Joe (product owner) we won't be pre-installing sudoku, dropping letters, filemanager, and terminal on commercial builds. My feeling then is that they should be removed from here.

[Colin Watson (cjwatson)]

The point of this work was to decouple things we aren't installing in
commercial builds from the set of things that go in the community
images. It's not clear why what we install in commercial builds should
continue to have a bearing here. We're removing them from the rootfs so
that the two sets of builds can have independent click package
selections.

[lp:~ubuntu-core-dev/livecd-rootfs/split-custom-tarball]

977. By Colin Watson on 2014-10-11

Update hardcoded libmir*-android installations for Mir 0.8.0.

978. By Colin Watson on 2014-10-11

releasing package livecd-rootfs version 2.253

979. By Oliver Grawert on 2014-10-11

ubuntu-touch: make sure we force the android alternative for libmir*- packages, in case their -mesa equivalent is installed by accident

980. By Oliver Grawert on 2014-10-11

releasing package livecd-rootfs version 2.254

[Colin Watson (cjwatson)]

I think this is working properly now with my modifications, and should be incrementally-landable.

review: Approve

[lp:~ubuntu-core-dev/livecd-rootfs/split-custom-tarball]

981. By Colin Watson on 2014-10-13

merge lp:~ubuntu-core-dev/livecd-rootfs/split-custom-tarball

=== modified file 'debian/changelog'
--- debian/changelog	2014-10-11 11:13:39 +0000
+++ debian/changelog	2014-10-13 19:11:47 +0000
@@ -1,3 +1,15 @@
+livecd-rootfs (2.255) UNRELEASED; urgency=medium
+
+  [ Steve Langasek ]
+  * Add support to the ubuntu-touch target to generate a separate custom
+    tarball as part of the build, moving most click packages into it.
+
+  [ Colin Watson ]
+  * Temporarily install the to-be-moved click packages in both core and
+    custom databases, to simplify the landing process.
+
+ -- Steve Langasek <[e-mail]>  Thu, 09 Oct 2014 20:46:46 -0700
+
 livecd-rootfs (2.254) utopic; urgency=medium
 
   * force the right alternative when the libmir*-android packages are

=== modified file 'live-build/auto/build'
--- live-build/auto/build	2014-06-26 11:31:07 +0000
+++ live-build/auto/build	2014-10-13 19:11:47 +0000
@@ -298,6 +298,12 @@
 	cp -a binary-tar.tar.gz "$PREFIX.rootfs.tar.gz"
 fi
 
+if [ "$PROJECT" = "ubuntu-touch" ]; then
+	(cd "binary/$INITFS/custom.dir/" && tar -c *) | \
+		gzip -9 --rsyncable > "$PREFIX.custom.tar.gz"
+	chmod 644 "$PREFIX.custom.tar.gz"
+fi
+
 # '--initramfs none' produces different manifest names.
 if [ -e "binary/$INITFS/filesystem.packages" ]; then
 	ln "binary/$INITFS/filesystem.packages" "$PREFIX.manifest"

=== added file 'live-build/ubuntu-touch/hooks/60-install-click.binary'
--- live-build/ubuntu-touch/hooks/60-install-click.binary	1970-01-01 00:00:00 +0000
+++ live-build/ubuntu-touch/hooks/60-install-click.binary	2014-10-13 19:11:47 +0000
@@ -0,0 +1,9 @@
+#!/bin/sh
+
+set -e
+
+echo "Splitting out custom tarball"
+
+rm -f binary/boot/custom.dir
+mkdir binary/boot/custom.dir
+mv binary/boot/filesystem.dir/custom binary/boot/custom.dir/custom

=== modified file 'live-build/ubuntu-touch/hooks/60-install-click.chroot'
--- live-build/ubuntu-touch/hooks/60-install-click.chroot	2013-11-26 15:43:06 +0000
+++ live-build/ubuntu-touch/hooks/60-install-click.chroot	2014-10-13 19:11:47 +0000
@@ -7,10 +7,15 @@
 click_uri=http://archive-team.internal/click_packages
 click_list=$click_uri/click_list
 click_db=/usr/share/click/preinstalled
+click_db_custom=/custom/click
 
 mkdir -p -m 755 "$click_db"
 chown clickpkg:clickpkg "$click_db"
 
+# some of these get installed to /custom/click
+mkdir -p -m 755 "$click_db_custom"
+chown clickpkg:clickpkg "$click_db_custom"
+
 tmpdir="$(mktemp -d)"
 cleanup () { rm -rf "$tmpdir"; }
 trap cleanup EXIT
@@ -23,7 +28,40 @@
     if echo $package | egrep -q "_$CLICKARCH.click|_all.click|_unknown.click"; then
         echo "Setting up $package"
         wget --no-verbose -O "$tmpdir/$package" "$click_uri/$package"
-        click install --force-missing-framework --root="$click_db" --all-users \
-            "$tmpdir/$package"
+	# FIXME: first attempt, a hard-coded list of the packages that go to
+        # the custom tarball
+        case $package in
+            com.ubuntu.developer.webapps.webapp-amazon_*|\
+            com.ubuntu.developer.webapps.webapp-ebay_*|\
+            com.ubuntu.developer.webapps.webapp-facebook_*|\
+            com.ubuntu.developer.webapps.webapp-gmail_*|\
+            com.ubuntu.developer.webapps.webapp-twitter_*|\
+            com.ubuntu.dropping-letters_*|\
+            com.ubuntu.filemanager_*|\
+            com.ubuntu.reminders_*|\
+            com.ubuntu.sudoku_*|\
+            com.ubuntu.terminal_*)
+                roots="$click_db $click_db_custom"
+                ;;
+            *)
+                roots="$click_db"
+                ;;
+        esac
+        for root in $roots; do
+            if [ "$root" = "$click_db_custom" ]; then
+                # FIXME: there is no good way to stop click from
+                # deduplicating things when installing the same package in
+                # multiple databases; the best we can do is to temporarily
+                # pretend that the core database does not exist
+                mv /etc/click/databases/10_core.conf \
+                   /etc/click/databases/10_core.conf.tmp
+            fi
+            click install --force-missing-framework --root="$root" --all-users \
+                "$tmpdir/$package"
+            if [ "$root" = "$click_db_custom" ]; then
+                mv /etc/click/databases/10_core.conf.tmp \
+                   /etc/click/databases/10_core.conf
+            fi
+        done
     fi
 done

=== modified file 'live-build/ubuntu-touch/hooks/90-precompile-apparmor-policies.chroot'
--- live-build/ubuntu-touch/hooks/90-precompile-apparmor-policies.chroot	2014-07-15 20:35:28 +0000
+++ live-build/ubuntu-touch/hooks/90-precompile-apparmor-policies.chroot	2014-10-13 19:11:47 +0000
@@ -7,3 +7,45 @@
 
 echo "I: precompiling deb apparmor policies"
 /sbin/apparmor_parser -M ${FEATURES} -Q --write-cache --cache-loc=/etc/apparmor.d/cache/ `find /etc/apparmor.d/ -maxdepth 1 -type f -not -path '*/\.*'`
+
+echo "I: precompiling custom click apparmor policies"
+mkdir -p /custom/cache/apparmor
+/sbin/apparmor_parser -M ${FEATURES} -Q --write-cache --cache-loc=/custom/cache/apparmor/ `find /var/lib/apparmor/profiles/ -maxdepth 1 -type f -not -path '*/\.*'`
+
+
+#get the apparmor manifests and profiles
+mkdir -p /custom/lib/apparmor/clicks
+mkdir -p /custom/lib/apparmor/profiles
+
+for manifest in /var/lib/apparmor/clicks/*; do
+	# FIXME: if this code survives for very long, it should probably be
+	# rewritten using click's Python bindings
+	pkgdir="$(click pkgdir "$manifest")"
+	manifest_real="$(readlink -f "$manifest")"
+	manifest_tail="${manifest_real#$pkgdir}"
+	# Does this package exist in the custom tarball?  If so, move its
+	# profiles there (if it only exists in custom) or copy them (if it
+	# also exists in core).
+	version="${pkgdir##*/}"
+	pkgdir_noversion="${pkgdir%/*}"
+	name="${pkgdir_noversion##*/}"
+	profile="$(basename "$manifest" .json)"
+	if [ -d "/custom/click/$name/$version" ]; then
+		# Clone into custom.
+		ln -nsf "/custom/click/$name/$version$manifest_tail" "/custom/lib/apparmor/clicks/${manifest##*/}"
+		cp -a "/var/lib/apparmor/profiles/click_$profile" /custom/lib/apparmor/profiles/
+		if [ -d "/usr/share/click/preinstalled/$name/$version" ]; then
+			# Ensure that the version in the rootfs points to
+			# the core database.
+			ln -nsf "/usr/share/click/preinstalled/$name/$version$manifest_tail" "$manifest"
+		else
+			# Remove from the rootfs.
+			rm -f "$manifest"
+			rm -f "/var/lib/apparmor/profiles/click_$profile"
+			rm -f "/var/cache/apparmor/click_$profile"
+		fi
+	else
+		# Remove from custom.
+		rm -f "/custom/cache/apparmor/click_$profile"
+	fi
+done