Merge ~paride/autopkgtest-cloud:clean-alien-packages into autopkgtest-cloud:master

[Paride Legovini]

[Paride Legovini (paride)]

To reviewers: note the usage of -o Dpkg::Options::="--force-confnew". My idea is that if we *downgrade* a package we want the "new" config files (i.e. those from the *older* package, the one being installed), as the already installed conf files may not be supported by the older package version.

[Julian Andres Klode (juliank)]

Some comment but overall this should work.

review: Approve

[Brian Murray (brian-murray)]

One comment.

[Paride Legovini (paride)]

Ready for another review.

[Paride Legovini (paride)]

I found a bug: we need DEBIAN_FRONTEND=noninteractive even when purging '?obsolete' packages, as debconf questions may also appear in that stage. Actually hit this in staging. The script was trying to remove a running kernel.

review: Needs Fixing

[Paride Legovini (paride)]

I updated the MP, the only change wrt the previous version is that I added

  DEBIAN_FRONTEND=noninteractive

to the line where we purge '?obsolete' packages. I tested this in staging by removing the mantic image and manually running the script, simulating what we do when opening a new series.

[Paride Legovini (paride)]

As discussed, added a comment on '?obsolete' removal possibly removing the running kernel.

diff --git a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker-config-production/setup-canonical.sh b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker-config-production/setup-canonical.sh
index c6d401c..92c2d70 100644
--- a/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker-config-production/setup-canonical.sh
+++ b/charms/focal/autopkgtest-cloud-worker/autopkgtest-cloud/worker-config-production/setup-canonical.sh
@@ -1,5 +1,55 @@
 # Canonical/Ubuntu specific testbed setup
 
+RELEASE=$(lsb_release --codename --short 2>/dev/null)
+
+# Make the installed package versions match what is available from the configured
+# repositories, even if a downgrade is required.
+# 
+# We may end up with images with packages newer than those available from the
+# repositories when opening a new release (images are created using the
+# previous release dailies as a base, which may include 0-day SRUs, or
+# Launchpad may fail to copy packages, see openssl 3.0.8-1ubuntu2), or when
+# package versions are pulled from the archive.
+# 
+# Note: 'release a=' matches the Suite field in the repository Release file.
+cat >/etc/apt/preferences.d/force-downgrade-to-release.pref <<EOF
+Package: *
+Pin: release a=$RELEASE
+Pin-Priority: 1001
+
+Package: *
+Pin: release a=$RELEASE-updates
+Pin-Priority: 1001
+
+Package: *
+Pin: release n=$RELEASE-security
+Pin-Priority: 1001
+EOF
+# This is going to be a downgrade-only (or no-op) dist-upgrade, as setup-testbed
+# already dist-upgraded the system. Force the usage of the conf files of the
+# package being installed (--force-confnew), as conf files "from the future"
+# may not be supported by the older package version we're downgrading to.
+DEBIAN_FRONTEND=noninteractive eatmydata apt-get -y --allow-downgrades -o Dpkg::Options::="--force-confnew" dist-upgrade
+rm /etc/apt/preferences.d/force-downgrade-to-release.pref
+
+# Remove packages not available from any configured repository, e.g.
+# packages fully removed from the archive. This requires the analyze-pattern
+# feature, which has been introduced in apt 2.0.1 (>= Focal).
+#
+# Note: autoremove/--autoremove always runs on the entire set of installed
+# packages. Together with --autopurge/purge each removal becomes a purge.
+#
+# Removing '?obsolete' packages may remove the running kernel. This is not
+# expected to be an issue.
+if /usr/lib/apt/apt-helper analyze-pattern '?true' >/dev/null 2>&1; then
+    DEBIAN_FRONTEND=noninteractive eatmydata apt-get -y purge --autoremove '?obsolete'
+else
+    # We're on an older release (analyze-pattern is not available).
+    # Still do an autopurge to get rid of packages that got orphaned
+    # by downgrades.
+    eatmydata apt-get -y autoremove --purge
+fi
+
 # compatibility with old Jenkins testbeds for trusty
 if grep -q trusty /etc/lsb-release; then apt-get update || (sleep 10; apt-get update); apt-get install -y build-essential; fi