Merge ~slyon/ubuntu-release-upgrader:slyon/focal-python2-fix into ubuntu-release-upgrader:ubuntu/focal

[Lukas Märdian]

Split the python_is_python2 quirk into a "prepare stage" (before calculating the dist-upgrade), where python-minimal is removed to avoid it producing any conflicts with packages depending on 'python' or 'python-minimal' and a "install stage" (after calculating the dist-upgrade), where the python-is-python2 package is installed to provide a clean upgrade path.

Fixes LP: #1875523

[Brian Murray (brian-murray)]

While this looks fine, I've tested the diff on a bionic virtual machine on which I've applied the apt-clone file in bug 1875523 and I wasn't able to upgrade. Is there something else wrong with the packages in that clone file? Is there another way I can test this?

review: Needs Information

[Lukas Märdian (slyon)]

Hmmm... I used the apt-clone file and traced it down to this minimal reproducer:

host:~$ multipass launch -n repr bionic && multipass shell repr

guest:~$ sudo apt update && sudo apt -y upgrade && sudo apt -y install python-pyside python-gi && sudo reboot

host:~$ multipass shell repr

guest:~$ sudo do-release-upgrade -d

The patch works with this minimal reproducer.

But let my try again with the full apt-clone state...

[Lukas Märdian (slyon)]

I pushed a commit, which fixes the upgrade for the full apt-clone state of LP #1875523 as well.

[Dimitri John Ledkov (xnox)]

review: Approve (lgtm)

[Lukas Märdian (slyon)]

The release notes for Focal have been updated accordingly:

https://wiki.ubuntu.com/FocalFossa/ReleaseNotes#Python3_by_default

[Brian Murray (brian-murray)]

This looks good to me, thanks for working on it. I think this should also be merged into the master branch for ubuntu-release-upgrader so that we have the "PreDistUpgradeCache" quirks available in the future.

review: Approve

[Lukas Märdian (slyon)]

Thanks for the review, Brian!

I applied the same patches to the master branch and prepared a MP for it here:

https://code.launchpad.net/~slyon/ubuntu-release-upgrader/+git/ubuntu-release-upgrader/+merge/387577

diff --git a/DistUpgrade/DistUpgradeCache.py b/DistUpgrade/DistUpgradeCache.py
index 8d9ba57..b2dd086 100644
--- a/DistUpgrade/DistUpgradeCache.py
+++ b/DistUpgrade/DistUpgradeCache.py
@@ -488,6 +488,12 @@ class MyCache(apt.Cache):
                                 self._keep_installed(pkg.name, "%s KeepInstalledSection rule: %s" % (key, section))
 
 
+    def pre_upgrade_rule(self):
+        " run before the upgrade was done in the cache "
+        # run the quirks handlers
+        if not self.partialUpgrade:
+            self.quirks.run("PreDistUpgradeCache")
+
     def post_upgrade_rule(self):
         " run after the upgrade was done in the cache "
         for (rule, action) in [("Install", self.mark_install),
@@ -614,6 +620,9 @@ class MyCache(apt.Cache):
             # mvo: disabled as it casues to many errornous installs
             #self._apply_dselect_upgrade()
 
+            # run PreDistUpgradeCache quirks
+            self.pre_upgrade_rule()
+
             # upgrade (and make sure this way that the cache is ok)
             self.upgrade(True)
 
diff --git a/DistUpgrade/DistUpgradeQuirks.py b/DistUpgrade/DistUpgradeQuirks.py
index 24bd563..4c4b545 100644
--- a/DistUpgrade/DistUpgradeQuirks.py
+++ b/DistUpgrade/DistUpgradeQuirks.py
@@ -67,6 +67,8 @@ class DistUpgradeQuirks(object):
                         to set options that affect the cache
         - PostInitialUpdate: run *before* the sources.list is rewritten but
                              after an initial apt-get update
+        - PreDistUpgradeCache: run *right before* the dist-upgrade is
+                               calculated in the cache
         - PostDistUpgradeCache: run *after* the dist-upgrade was calculated
                                 in the cache
         - StartUpgrade: before the first package gets installed (but the
@@ -157,13 +159,19 @@ class DistUpgradeQuirks(object):
         self._pokeScreensaver()
         self._stopDocvertConverter()
 
+    # individual quirks handler that run *right before* the dist-upgrade
+    # is calculated in the cache
+    def PreDistUpgradeCache(self):
+        """ run right before calculating the dist-upgrade """
+        logging.debug("running Quirks.PreDistUpgradeCache")
+        self._install_python_is_python2()
+
     # individual quirks handler that run *after* the dist-upgrade was
     # calculated in the cache
     def PostDistUpgradeCache(self):
         """ run after calculating the dist-upgrade """
         logging.debug("running Quirks.PostDistUpgradeCache")
         self._install_linux_metapackage()
-        self._install_python_is_python2()
 
     # helpers
     def _get_pci_ids(self):
@@ -836,8 +844,9 @@ class DistUpgradeQuirks(object):
 
     def _install_python_is_python2(self):
         """
-        Ensure the python-is-python2 is installed if python-minimal
-        was installed.
+        Ensure python is removed, before it can produce a conflict with any
+        other package and the python-is-python2 package is installed instead,
+        if python-minimal was installed.
         """
         old = 'python-minimal'
         new = 'python-is-python2'
@@ -847,6 +856,17 @@ class DistUpgradeQuirks(object):
             reason = "%s was installed on the system" % old
             if not cache.mark_install(new, reason):
                 logging.info("failed to install %s" % new)
+            logging.info("removing %s because %s is being installed" % (old, new))
+            reason = "%s is being installed on the system" % new
+            if not cache.mark_remove(old, reason):
+                logging.info("failed to remove %s", old)
+
+            # protect our decision to remove legacy 'python' (as a dependency
+            # of python-minimal, removed above)
+            py = 'python'
+            if py in cache and cache[py].marked_delete:
+                resolver = apt.cache.ProblemResolver(cache)
+                resolver.protect(cache[py])
 
     def ensure_recommends_are_installed_on_desktops(self):
         """ ensure that on a desktop install recommends are installed