gnome-software-plugin-snap consider an unwanted removal for 20.04 ubuntu-desktop

I tested an upgrade where I added gnome-software-plugin-snap to the ForcedObsoletes section of ubuntu-desktop and the following packages were marked for removal. (I did a diff of removed packages w/ and w/o that package in ForcedObsoletes.)

command-not-found-data
example-content
gcc-7-base
imagemagick-6-common
libalgorithm-diff-perl
libart-2.0-2
libasan4
libboost-system1.65.1
libboost-thread1.65.1
libclass-xsaccessor-perl
libcpanel-json-xs-perl
libexempi3
libexiv2-14
libexporter-tiny-perl
libfakeroot
libfuture-perl
libgtksourceview-3.0-common
libgweather-3-15
libhunspell-1.6-0
libhwloc5
libilmbase12
libilmbase24
libio-async-perl
libio-pty-perl
libip4tc0
liblist-compare-perl
liblqr-1-0
libmozjs-52-0
libnet-ip-perl
libnss-myhostname
libnumber-compare-perl
liboauth0
libpackage-stash-xs-perl
libplymouth4
libpython2.7
libraw16
libreoffice-avmedia-backend-gstreamer
libreoffice-style-galaxy
libstruct-dumb-perl
libtest-fatal-perl
libubsan0
libunicode-utf8-perl
libusbmuxd4
libxapian30
libxml-libxml-perl
libxml-namespacesupport-perl
libxml-sax-base-perl
libxml-sax-expat-perl
libxml-sax-perl
libxml-writer-perl
libzeitgeist-2.0-0
light-themes
linux-headers-5.4.0-42
linux-headers-5.4.0-42-generic
linux-hwe-5.4-headers-5.4.0-47
linux-image-5.4.0-42-generic
linux-modules-5.4.0-42-generic
linux-modules-extra-5.4.0-42-generic
python-talloc
python3-asn1crypto
python3-crypto
python3-keyrings.alt
python3-oauth
python3-simplegeneric
python3-zope.interface
python3.6
python3.6-minimal
qpdf
t1utils
ubuntu-sounds
ubuntu-system-service
xserver-xorg-hwe-18.04
xserver-xorg-input-all-hwe-18.04
xserver-xorg-input-libinput-hwe-18.04
xserver-xorg-video-ati-hwe-18.04
xserver-xorg-video-fbdev-hwe-18.04
xserver-xorg-video-intel-hwe-18.04

Here are the upgrade logs from the test upgrade.

To double check my logic I ran an upgrade of a stock Ubuntu desktop system from bionic to focal and immediately after the upgrade ran 'sudo apt-get autoremove'. The list of packages marked for removal are the same as the ones the release-upgrader suggested with my change.

bdmurray@clean-bionic-amd64:/tmp/ubuntu-release-upgrader-idl254bt$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.1 LTS"
bdmurray@clean-bionic-amd64:/tmp/ubuntu-release-upgrader-idl254bt$ sudo apt-get autoremove
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  command-not-found-data diffstat gnome-software-common gsfonts guile-2.0-libs libalgorithm-diff-perl libalgorithm-merge-perl
  libappstream-glib8 libarchive-zip-perl libboost-filesystem1.65.1 libboost-system1.65.1 libcgi-pm-perl libclass-method-modifiers-perl
  libclass-xsaccessor-perl libclone-perl libcommon-sense-perl libcrypt-dev libdigest-bubblebabble-perl libenchant1c2a libexempi3
  libexiv2-14 libfftw3-double3 libgmime-3.0-0 libgtksourceview-3.0-common libgweather-3-15 libindicator3-7 libip4tc0 libip6tc0 libitm1
  libjson-xs-perl liblouis14 liblouisutdml8 liblwres160 libnet-domain-tld-perl libnetpbm10 liboauth0 libprotobuf10 libpython2.7-minimal
  libpython2.7-stdlib libraw16 libref-util-perl libref-util-xs-perl libreoffice-avmedia-backend-gstreamer libreoffice-style-galaxy
  librole-tiny-perl libssl1.0.0 libsub-quote-perl libtest-refcount-perl libtypes-serialiser-perl libunicode-utf8-perl
  linux-hwe-5.4-headers-5.4.0-45 linux-hwe-5.4-headers-5.4.0-47 linux-libc-dev manpages-dev python3-asn1crypto python3-oauth
  python3-simplegeneric python3-zope.interface qpdf xserver-xorg-input-all-hwe-18.04 xserver-xorg-legacy-hwe-18.04
  xserver-xorg-video-ati-hwe-18.04 xserver-xorg-video-fbdev-hwe-18.04 xserver-xorg-video-qxl-hwe-18.04 xserver-xorg-video-radeon-hwe-18.04
  xserver-xorg-video-vesa-hwe-18.04
0 upgraded, 0 newly installed, 66 to remove and 0 not upgraded.
After this operation, 214 MB disk space will be freed.

Given that this only affects upgrades to Focal there is no need to add this change to Groovy.

Hello Brian, or anyone else affected,

Accepted ubuntu-release-upgrader into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:20.04.27 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

For the verification I've captured command outputs and log files with the version of the release upgrader in the filename so 20.04.25 is the current version of the release upgrader and 20.04.27 is the one from -proposed.

With regards to the autoremoval of packages not every package which was autoremoved after upgrading with release upgrader version 20.04.25 was removed during the upgrade process e.g.:

 $ for pkg in $(cat uru-20.04.25-autoremove-pkgs.txt); do grep " $pkg " uru-20.04.27-installed-pkgs.txt; done
ii gnome-software-common 3.36.1-0ubuntu0.20.04.0 all Software Center for GNOME (common files)

So after the upgrade with version 20.04.27 there were still some packages that were autoremovable.

 $ cat uru-20.04.27-autoremove.txt
Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be REMOVED:
  gnome-software-common libappstream-glib8
0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.
After this operation, 6,538 kB disk space will be freed.
Do you want to continue? [Y/n] Abort.

So while this is much better it doesn't seem to be a complete fix, so the questions is - is it worth releasing?

I tested an upgrade of an Ubuntu 18.04 system with xubuntu-desktop installed and gnome-software-plugin-snap remained installed.

bdmurray@xubuntu-bionic-server-amd64:/tmp/uru$ grep "upgrader version" /var/log/dist-upgrade/main.log
2020-09-21 13:08:57,559 INFO release-upgrader version '20.04.27' started
bdmurray@xubuntu-bionic-server-amd64:/tmp/uru$ apt-cache policy xubuntu-desktop
xubuntu-desktop:
  Installed: 2.233
  Candidate: 2.233
  Version table:
 *** 2.233 500
        500 http://192.168.10.7/ubuntu focal/universe amd64 Packages
        100 /var/lib/dpkg/status
bdmurray@xubuntu-bionic-server-amd64:/tmp/uru$ apt-cache policy gnome-software-plugin-snap
gnome-software-plugin-snap:
  Installed: 3.36.1-0ubuntu0.20.04.0
  Candidate: 3.36.1-0ubuntu0.20.04.0
  Version table:
 *** 3.36.1-0ubuntu0.20.04.0 500
        500 http://192.168.10.7/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     3.36.0-0ubuntu3 500
        500 http://192.168.10.7/ubuntu focal/main amd64 Packages

Taking the list of packages that were autoremoved when using 'sudo apt autoremove' after upgrading with dist-upgrader version 20.04.25 and checking to see if they were marked for removal with dist-upgrader version 20.04.27. There are some interesting discrepancies.

for pkg in $(cat uru-20.04.25-autoremove-pkgs.txt); do grep -q "^$pkg$" uru-20.04.27-marked-for-removal.txt; echo $pkg $?; done

...
libtypes-serialiser-perl 1
libjson-xs-perl 1
libcommon-sense-perl 1
libappstream-glib8 1
gnome-software-common 1
...

Two of those, libappstream-glib8 and gnome-software-common, ended up being autoremovable after the upgrade with 20.04.27. The other three were kept and I don't know why.

Looking at the list of packages which were installed on the system which upgraded with 20.04.25 and after using 'sudo apt autoremove' I checked to see if they are all installed on the system upgraded with 20.04.27 after using 'sudo apt autoremove'. There do end up being some discrepancies there too.

 $ for pkg in $(awk '{print $2}' uru-20.04.25-installed-pkgs-after-autoremove.txt); do grep -q $pkg uru-20.04.27-installed-pkgs-after-autoremove.txt; if [ $? -eq 1 ]; then echo $pkg; fi; done

fakeroot
libdigest-hmac-perl
libfakeroot:amd64
libsub-name-perl
libxml-libxml-perl
libxml-namespacesupport-perl
libxml-sax-base-perl
libxml-sax-expat-perl
libxml-sax-perl
perl-modules-5.26
ubuntu-sounds
ubuntu-wallpapers-bionic
ubuntu-web-launchers

All of these are were marked for removal by the dist-upgrader version 20.04.25 but were considered unwanted, so I think this is okay.

 $ for pkg in $(awk '{print $2}' uru-20.04.25-installed-pkgs-after-autoremove.txt); do grep -q $pkg uru-20.04.27-installed-pkgs-after-autoremove.txt; if [ $? -eq 1 ]; then grep "$pkg.*unwanted removal" uru-20.04.25-main.log; fi; done
2020-09-18 16:40:37,846 DEBUG package 'fakeroot' produces an unwanted removal 'gnome-software-plugin-snap', skipping
2020-09-18 16:41:40,666 DEBUG package 'libfakeroot' produces an unwanted removal 'gnome-software-plugin-snap', skipping
2020-09-18 16:38:55,556 DEBUG package 'libdigest-hmac-perl' produces an unwanted removal 'gnome-software-plugin-snap', skipping
2020-09-18 16:40:07,091 DEBUG package 'libsub-name-perl' produces an unwanted removal 'gnome-software-plugin-snap', skipping
2020-09-18 16:42:10,902 DEBUG package 'libxml-libxml-perl' produces an unwanted removal 'gnome-software-plugin-snap', skipping
2020-09-18 16:38:26,155 DEBUG package 'libxml-namespacesupport-perl' produces an unwanted removal 'gnome-software-plugin-snap', skipping
2020-09-18 16:42:12,856 DEBUG package 'libxml-sax-base-perl' produces an unwanted removal 'gnome-software-plugin-snap', skipping
2020-09-18 16:36:46,649 DEBUG package 'libxml-sax-expat-perl' produces an unwanted removal 'gnome-software-plugin-snap', skipping
2020-09-18 16:39:43,708 DEBUG package 'libxml-sax-perl' produces an unwanted removal 'gnome-software-plugin-snap', skipping
2020-09-18 16:36:48,563 DEBUG package 'perl-modules-5.26' produces an unwanted removal 'gnome-software-plugin-snap', skipping
2020-09-18 16:38:45,587 DEBUG package 'ubuntu-sounds' produces an unwanted removal 'gnome-software-plugin-snap', skipping
2020-09-18 16:38:59,222 DEBUG package 'ubuntu-wallpapers-bionic' produces an unwanted removal 'gnome-software-plugin-snap', skipping
2020-09-18 16:37:27,601 DEBUG package 'ubuntu-web-launchers' produces an unwanted removal 'gnome-software-plugin-snap', skipping

Here are all my log files for dist-upgrader version 20.04.25.

autoremove-pkgs.txt is one package name per line
marked-for-removal.txt is from main.log with one package name per line

Here are all my log files for dist-upgrader version 20.04.27.

autoremove-pkgs.txt is one package name per line
marked-for-removal.txt is from main.log with one package name per line

Based on the packages which were kept installed in comment #9 and the release-upgrader behaving strangely when removing packages we've decided to mark this as verification failed and drop the change for now.

The version of ubuntu-release-upgrader in the proposed pocket of Focal that was purported to fix this bug report has been removed because one or more bugs that were to be fixed by the upload have failed verification and been in this state for more than 10 days.

This bug was fixed in the package ubuntu-release-upgrader - 1:20.04.28

---------------
ubuntu-release-upgrader (1:20.04.28) focal; urgency=medium

  [ Brian Murray ]
  * DistUpgrade/DistUpgradeQuirks.py: In addition to quirking python-minimal
    we also need to quirk python-dbg, python-doc, and python-dev.
    (LP: #1887544)
  * drop changes to data/DistUpgrade.cfg.bionic regarding
    gnome-software-plugin-snap as it failed verification.

  [ Chad Smith ]
  * data/mirrors.cfg: add all ubuntu-advantage services as valid mirrors.
    This includes: fips, fips-updates, esm-infra, esm-apps and cc-eal and
    Ubuntu Pro stable public PPA. (LP: #1893717)
  * DistUpgrade/DistUpgradeController.py: release cache lock during
    runPostInstallScripts (LP: #1897778)

 -- Chad Smith <email address hidden> Wed, 07 Oct 2020 10:49:37 -0600