Merge ~xnox/ubuntu-release-upgrader:ignore-base-tasks into ubuntu-release-upgrader:ubuntu/master

[Dimitri John Ledkov]

DistUpgradeCache: Ignore BaseMetaPkgs' & obsolete tasks
 during server upgrade

ubuntu-minimal and ubuntu-base are forced to stay installed via
BaseMetaPkgs / keepBaseMetaPkgs rule during either desktop or server
upgrades. During server upgrades, we try to establish which Tasks are
installed, we do this by iterating all packages and checking if all
packages of a given Task are installed. This is the case for
ubuntu-minimal & ubuntu-standard. But they are not manually-installed
tasks. Instead they are handled with a metapackage only, to allow
alternative implementations of certain components
(i.e. time-daemon). Thus during upgrade we should not force install
those, but instead rely on keeping BaseMetaPkg installed only to
esnure that alternative selections of time-daemons is permitted and is
not changed.

Also, this function might missdetect as obsolete tasks as
installed. For example Task: ubuntu-core should no longer exist, as
the metapackages for it got dropped, but we forgot to remove the
metadata from the archive. If it happens that ubuntu-core appears to
be fully installed, do not try to preserve it on upgrades.

LP: #1872902

DistUpgradeCache: Ignore BaseMetaPkgs' & obsolete tasks
 during server upgrade

ubuntu-minimal and ubuntu-base are forced to stay installed via
BaseMetaPkgs / keepBaseMetaPkgs rule during either desktop or server
upgrades. During server upgrades, we try to establish which Tasks are
installed, we do this by iterating all packages and checking if all
packages of a given Task are installed. This is the case for
ubuntu-minimal & ubuntu-standard. But they are not manually-installed
tasks. Instead they are handled with a metapackage only, to allow
alternative implementations of certain components
(i.e. time-daemon). Thus during upgrade we should not force install
those, but instead rely on keeping BaseMetaPkg installed only to
esnure that alternative selections of time-daemons is permitted and is
not changed.

Also, this function might missdetect as obsolete tasks as
installed. For example Task: ubuntu-core should no longer exist, as
the metapackages for it got dropped, but we forgot to remove the
metadata from the archive. If it happens that ubuntu-core appears to
be fully installed, do not try to preserve it on upgrades.

LP: #1872902

[Dimitri John Ledkov (xnox)]

To try upgrading using this upgrader:

git clone -b ignore-base-tasks https://git.launchpad.net/~xnox/ubuntu-release-upgrader

cd ubuntu-release-upgrader/DistUpgrade

./dist-upgrade.py --datadir=../data --disable-gnu-screen

With eoan lxd container + chrony installed, previously we tried to force install:
Installed tasks are: {'standard', 'server', 'openssh-server', 'minimal', 'cloud-image'}

But with this change, we try to force install:
Installed tasks are: {'openssh-server', 'cloud-image', 'server'}

Which imho are the correct tasks to force-upgrade for a lxd container.

[Dimitri John Ledkov (xnox)]

minimal, standard tasks are already handled separately via _keepBaseMetaPkgsInstalled which forces to keep them installed as one of the last actions before finishing calculating the upgrade, to make sure they are installed/upgraded.

[Balint Reczey (rbalint)]

I think this change is a bit too intrusive for an SRU from testing/assessing the potential regressions POV and also does not fully resolve the redundancy between tasks and meta packages, just special-cases two of them.

I think the minimal fix of skipping just systemd-timesyncd either in u-r-u or in the seed generation for focal only would be a more appropriate to limit the impact.

For groovy and later I think the task generation should be changed to not pull in dependencies to tasks and as a result u-r-u could keep handling tasks the way they used to be handled.

I'm OK with picking this change and SRU-ing it because the change is not wronk, but I think it is between a minimal fix and a general solution and this makes it not optimal in my eyes.

review: Abstain

[Dimitri John Ledkov (xnox)]

So old ubuntu server installer allowed to install tasks, and exposed them as a list. Hence the upgrader wanted to preserve tasks that users installed at the time.

However, tasks that were explicitely installed, vs accidentally all of a tasks fields are satisfied is arbitrary to me.

I'm not sure why we still publish Tasks in the archive =/

Or like for example, why we publish tasks for things that are metapackages.

[Brian Murray (brian-murray)]

I'm okay with this approach to resolving the issue, but this needs fixing as data/DistUpgrade.cfg.bionic will also need updating.

I tested desktop and server upgrades with and without chrony and did not notice any issues.

If and when we land this in -proposed I'd like to see the automatic upgrade testing use the version of the dist-upgrader with these changes.

review: Needs Fixing

[Brian Murray (brian-murray)]

If nobody objects I'm going to merge and upload this on 2020-05-05.

diff --git a/DistUpgrade/DistUpgradeCache.py b/DistUpgrade/DistUpgradeCache.py
index 3cd8504..8d9ba57 100644
--- a/DistUpgrade/DistUpgradeCache.py
+++ b/DistUpgrade/DistUpgradeCache.py
@@ -818,6 +818,9 @@ class MyCache(apt.Cache):
                         tasks[task].add(pkg.name)
         for task in tasks:
             installed = True
+            ignored_tasks = self.config.getlist("Distro", "IgnoredTasks")
+            if task in ignored_tasks:
+                installed = False
             for pkgname in tasks[task]:
                 if not self[pkgname].is_installed:
                     installed = False
diff --git a/data/DistUpgrade.cfg b/data/DistUpgrade.cfg
index a349f4a..2b26f82 100644
--- a/data/DistUpgrade.cfg
+++ b/data/DistUpgrade.cfg
@@ -16,6 +16,9 @@ MetaPkgs=ubuntu-desktop, kubuntu-desktop, xubuntu-core,
    ubuntu-mate-core, ubuntu-mate-desktop,
    ubuntukylin-desktop, ubuntu-budgie-desktop
 BaseMetaPkgs=ubuntu-minimal, ubuntu-standard
+# minimal & standard, are handed via BaseMetaPkgs rule
+# ubuntu-core is left-over task we forgot to remove
+IgnoredTasks=minimal, standard, ubuntu-core
 Demotions=demoted.cfg
 #RemoveEssentialOk=sysvinit, sysvutils, belocs-locales-bin
 RemovalBlacklistFile=removal_blacklist.cfg
diff --git a/debian/changelog b/debian/changelog
index fe72e60..798f180 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+ubuntu-release-upgrader (1:20.04.19) UNRELEASED; urgency=medium
+
+  * DistUpgradeCache: Ignore BaseMetaPkgs' & obsolete tasks during server upgrade
+    (LP: #1872902)
+
+ -- Dimitri John Ledkov <[e-mail]>  Sun, 26 Apr 2020 17:07:08 +0100
+
 ubuntu-release-upgrader (1:20.04.18) focal; urgency=medium
 
   [ Steve Langasek ]