Merge lp:~mvo/ubuntu-archive-publishing/sync-cnf-metadata into lp:ubuntu-archive-publishing

[Michael Vogt]

This PR adds pulling the command-not-found metadata from the juju machine that extracts the data to the right place. It is modeled after the appstream data. This means that we can (finally!) get rid of the command-not-found-data package and use dynamic and up-to-date metadata in command not found. This will also enable c-n-f to display version information once this data is available.

There are two open questions:
- Do we need to open some firewall rules so that the machine that runs this script can rsync from cnf-extrctor.internal?
- We should limit the publishing initially to cosmic (or even cosmic+1). Bionic and later has a config in c-n-f to fetch and use the metadata once its available on the server and we need to double check this with cosmic first. I can change this PR so that it only fetches here or we can do it at the LP side (that is pending), not sure what is the best approach here (advise welcome!).

There is also some support in LP needed, I can look into this next.

[Colin Watson (cjwatson)]

Firewall rules: https://code.launchpad.net/~cjwatson/canonical-is-firewalls/+git/firewall-configs/+merge/363371

[Colin Watson (cjwatson)]

Regarding publishing, I think if we just make sure that cnf-extractor only extracts information for >= disco series then we're good. That seems to be the case today.

[Colin Watson (cjwatson)]

I think this is mostly OK, but the set of Commands files on cnf-extractor.internal seems a little excessive:

$ rsync -v cnf-extractor.internal::cnf-data/disco/main/cnf/
receiving file list ... done
drwxr-xr-x 4,096 2019/02/20 10:19:15 .
-rw-r--r-- 112,438 2019/02/20 08:18:42 Commands-amd64
-rw-r--r-- 32,576 2019/02/20 08:18:42 Commands-amd64.gz
-rw-r--r-- 29,836 2019/02/20 08:18:42 Commands-amd64.xz
-rw-r--r-- 105,141 2019/02/20 08:19:01 Commands-arm64
-rw-r--r-- 31,197 2019/02/20 08:19:01 Commands-arm64.gz
-rw-r--r-- 28,720 2019/02/20 08:19:01 Commands-arm64.xz
-rw-r--r-- 102,213 2019/02/20 08:19:10 Commands-armhf
-rw-r--r-- 30,712 2019/02/20 08:19:10 Commands-armhf.gz
-rw-r--r-- 28,300 2019/02/20 08:19:10 Commands-armhf.xz
-rw-r--r-- 112,758 2019/02/20 08:18:51 Commands-i386
-rw-r--r-- 32,564 2019/02/20 08:18:51 Commands-i386.gz
-rw-r--r-- 29,828 2019/02/20 08:18:51 Commands-i386.xz
-rw-r--r-- 108,720 2019/02/20 08:19:18 Commands-ppc64el
-rw-r--r-- 31,838 2019/02/20 08:19:18 Commands-ppc64el.gz
-rw-r--r-- 29,300 2019/02/20 08:19:18 Commands-ppc64el.xz
-rw-r--r-- 101,287 2019/02/20 08:19:27 Commands-s390x
-rw-r--r-- 30,675 2019/02/20 08:19:27 Commands-s390x.gz
-rw-r--r-- 28,316 2019/02/20 08:19:27 Commands-s390x.xz

sent 24 bytes received 429 bytes 302.00 bytes/sec
total size is 1,006,419 speedup is 2,221.68

This depends a little on how the client is going to deal with it, but I don't think we should need the uncompressed files (noting that Launchpad with your branch will already deal with putting the uncompressed checksum in InRelease given only a .gz or .bz2 file). With a fairly small extra tweak to Launchpad we might only need the .xz file. What does the client actually use?

review: Needs Information

[lp:~mvo/ubuntu-archive-publishing/sync-cnf-metadata]

106. By Michael Vogt on 2019-03-05

35-command-not-found: rsync only .xz files

[Michael Vogt (mvo)]

Thanks Colin!

I updated the rsync script to only pull the .xz files and updated the charm to not produce .gz files anymore. I also checked that apt will be fine with just having the .xz file (checked via "deb http://people.canonical.com/~mvo/c-n-f bionic main" and putting only a .xz there).

[Colin Watson (cjwatson)]

review: Approve

=== added file 'finalize.d/35-command-not-found'
--- finalize.d/35-command-not-found	1970-01-01 00:00:00 +0000
+++ finalize.d/35-command-not-found	2019-03-05 10:09:25 +0000
@@ -0,0 +1,36 @@
+#! /bin/sh
+set -e
+
+# Sync Command-not-found metadata, staging it for installation by the next
+# publisher run.
+
+# The finalize.d protocol passes the list of root directories for the
+# distribution's archives in the ARCHIVEROOTS environment variable, joined
+# by spaces.  We only care about the first archive.
+ARCHIVEROOT="${ARCHIVEROOTS%% *}"
+TEMPROOT="$ARCHIVEROOT/../ubuntu-temp"
+STAGINGROOT="$ARCHIVEROOT/../ubuntu-staging"
+
+echo "$(date -R): Syncing Command-not-found metadata..."
+mkdir -p "$TEMPROOT"
+rsync -aq --timeout=30 --delete --delete-after \
+    cnf-extractor.internal::cnf-data/ "$TEMPROOT/cnf/" || true
+for dir in "$TEMPROOT/cnf"/*; do
+    [ -d "$dir" ] || continue
+    suite="${dir##*/}"
+    for component in main restricted universe multiverse; do
+        if [ -d "$dir/$component" ]; then
+            mkdir -p "$STAGINGROOT/$suite/$component"
+            rsync -aq --delete --delete-after  \
+		--include='*/'  --include='*.xz' --exclude="*"  \
+                   "$dir/$component/" "$STAGINGROOT/$suite/$component/cnf/"
+        else # remove stale cnf data if the origin no longer exists:
+            if [ -d "$STAGINGROOT/$suite/$component/cnf/" ]; then
+                rm -r "$STAGINGROOT/$suite/$component/cnf/"
+            fi
+        fi
+    done
+done
+echo "$(date -R): Command-not-found metadata sync complete."
+
+exit 0