distribution upgrade failed due pip install of distro-info

Status changed to 'Confirmed' because the bug affects multiple users.

Upgrade from 19.04 to 19.10 failed for me

Fetched 586 B in 0s (0 B/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done

Restoring original system state

Aborting
Reading package lists... Done
Building dependency tree
Reading state information... Done

2019-10-19 13:05:38,522 INFO Using config files '['./DistUpgrade.cfg']'
2019-10-19 13:05:38,522 INFO uname information: 'Linux DreamServer 5.0.0-32-generic #34-Ubuntu SMP Wed Oct 2 02:06:48 UTC 2019 x86_64'
2019-10-19 13:05:38,912 INFO apt version: '1.8.3'
2019-10-19 13:05:38,912 INFO python version: '3.7.3 (default, Oct 7 2019, 12:56:13)
[GCC 8.3.0]'
2019-10-19 13:05:38,916 INFO release-upgrader version '19.10.12' started
2019-10-19 13:05:38,924 INFO locale: 'en_US' 'UTF-8'
2019-10-19 13:05:38,969 INFO screen could not be run
2019-10-19 13:05:39,003 DEBUG Using 'DistUpgradeViewText' view
2019-10-19 13:05:39,042 DEBUG enable dpkg --force-overwrite
2019-10-19 13:05:39,084 DEBUG creating statefile: '/var/log/dist-upgrade/apt-clone_system_state.tar.gz'
2019-10-19 13:05:43,722 DEBUG lsb-release: 'disco'
2019-10-19 13:05:43,723 DEBUG _pythonSymlinkCheck run
2019-10-19 13:05:43,724 DEBUG openCache()
2019-10-19 13:05:43,724 DEBUG quirks: running PreCacheOpen
2019-10-19 13:05:43,724 DEBUG running Quirks.PreCacheOpen
2019-10-19 13:05:44,336 DEBUG /openCache(), new cache size 95561
2019-10-19 13:05:44,336 DEBUG need_server_mode(): run in 'desktop' mode, (because of pkg 'ubuntu-desktop')
2019-10-19 13:05:44,336 DEBUG checkViewDepends()
2019-10-19 13:05:44,337 DEBUG running doUpdate() (showErrors=False)
2019-10-19 13:05:46,638 DEBUG openCache()
2019-10-19 13:05:47,275 DEBUG /openCache(), new cache size 95561
2019-10-19 13:05:47,275 DEBUG doPostInitialUpdate
2019-10-19 13:05:47,275 DEBUG quirks: running eoanPostInitialUpdate
2019-10-19 13:05:47,275 DEBUG running Quirks.eoanPostInitialUpdate
2019-10-19 13:05:52,295 DEBUG abort called
2019-10-19 13:05:52,295 DEBUG openCache()
2019-10-19 13:05:52,950 DEBUG /openCache(), new cache size 95561

Upgraded update-manager to 1:19.10.15
<https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:19.10.15>
That allowed update to occur.
I did have a problem with logging in after update. For some reason only
2 ids were available on the login screen (unity). One of which (Sophos
Anti-Virus) was disabled (it hadn't been listed before upgrade), and
another account that I was able to log in to (with admin privileges) .
My usual account was not available for login. Settings showed only the
same two users noted above. I was able to open a terminal and su to my
normal ID, however. I ended up doing a deluser on my user account, and
then go into settings and add my usual account back. ...

On 10/19/19 1:20 AM, Arul wrote:
> Upgrade from 19.04 to 19.10 failed for me
>
> Fetched 586 B in 0s (0 B/s)
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
>
> Restoring original system state
>
> Aborting
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
>

Upgraded update-manager to 1:19.10.15
<https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:19.10.15>
That allowed update to occur.
I did have a problem with logging in after update. For some reason only
2 ids were available on the login screen (unity). One of which (Sophos
Anti-Virus) was disabled (it hadn't been listed before upgrade), and
another account that I was able to log in to (with admin privileges) .
My usual account was not available for login. Settings showed only the
same two users noted above. I was able to open a terminal and su to my
normal ID, however. I ended up doing a deluser on my usual user account,
and then went into settings and add my usual account back. ...
On 10/19/19 1:15 AM, Launchpad Bug Tracker wrote:
> Status changed to 'Confirmed' because the bug affects multiple users.
>
> ** Changed in: ubuntu-release-upgrader (Ubuntu)
> Status: New => Confirmed
>

I reported bug <https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1849669> as I get completely different results from this Bug Reports description.

See here:
=====================
chippo@chippo-Aspire-V3-731:~$ sudo update-manager -d
[sudo] password for chippo:
Checking for a new Ubuntu release
/usr/lib/python3/dist-packages/DistUpgrade/DistUpgradeFetcher.py:23: PyGIWarning: Gtk was imported without specifying a version first. Use gi.require_version('Gtk', '3.0') before import to ensure that the right version gets loaded.
  from gi.repository import Gtk, Gdk
/usr/lib/python3/dist-packages/DistUpgrade/ReleaseNotesViewerWebkit.py:33: PyGIWarning: WebKit2 was imported without specifying a version first. Use gi.require_version('WebKit2', '4.0') before import to ensure that the right version gets loaded.
  from gi.repository import WebKit2 as WebKit

(do-release-upgrade:27476): IBUS-WARNING **: 15:35:46.567: The owner of /home/chippo/.config/ibus/bus is not root!
authenticate 'eoan.tar.gz' against 'eoan.tar.gz.gpg'
extracting 'eoan.tar.gz'

(eoan:27476): IBUS-WARNING **: 15:35:58.088: The owner of /home/chippo/.config/ibus/bus is not root!
=====================

And that's all. The GUI comes and everything looks fine until you read the Installation Notes and press upgrade. Then the GUI disappears, but no more errors/logs are printed into the terminal.

So, I added a new bug.

However, looking at Arul's posts above, I see that his output is the same as mine. So, I think that there are 2 different bugs, but Arul is confirming mine, not this one.

I have no idea what to do about it administratively, either. Maybe I'll add another comment in the other bug to also look here...

A possible workaround, which was successful for me:
1. Uninstall distro-info 0.0.0 Python package via pip.
2. Install/reinstall distro-info package from Ubuntu repos.

After these two steps, I was able to continue with my upgrade.

I still have no idea how the distro-info Python package got installed in my Ubuntu 19.04. Also, why is it's version 0.0.0.

@Kent - do you have distro-info installed from pip? You can check via pip list.

I have multiple versions of python installed... not sure of the path
that was used .
However of the versions that may have been used ...
the only "distro" I see is

python3.8 -m pip list | grep distro
WARNING: pip is configured with locations that require TLS/SSL, however
the ssl module in Python is not available.
 > pip3 list | grep distro
distro?????????????????????????????????????????????? 1.4.0
 > python3 -m pip list | grep distro
distro?????????????????????????????????????????????? 1.4.0
 >python3.7 -m pip list | grep distro
distro?????????????????????????????????????????????? 1.4.0
 > python -m pip list | grep distro
distro?????????????????????????????????????????????? 1.4.0

I am now up and running on 19.10
I upgraded update-manager to version 1.19.10.15 (19.04 had an older
version)
That allowed update to occur.
I did have a problem with logging in after update. For some reason only
2 ids were available on the login screen (unity). One of which (Sophos
Anti-Virus) was disabled (it hadn't been listed before upgrade) and was
id 9999 (one less than my usual id, so maybe a one-off problem?), and
another account that I was able to log in to (with admin privileges) .
My usual account was not available for login. Settings showed only the
same two users noted above. I was able to open a terminal and su to my
normal ID, however. I ended up doing a deluser on my usual user account,
and then went into settings and added my usual account back. ...I think
all is well now...

On 10/25/19 3:11 PM, Brian Murray wrote:
> @Kent - do you have distro-info installed from pip? You can check via
> pip list.
>
> ** Summary changed:
>
> - upgrade from 19.04 to 19.10 fails due pip install of distro-info
> + upgrade from 19.04 to 19.10 failed due pip install of distro-info
>
> ** Changed in: ubuntu-release-upgrader (Ubuntu)
> Status: Confirmed => Incomplete
>

[Expired for ubuntu-release-upgrader (Ubuntu) because there has been no activity for 60 days.]

Re-installation of python3-distro-info helped in my case:

1)python3 -m pip uninstall distro-info

2)sudo apt-get reinstall python3-distro-info

According to the Error Tracker there have been 49 crashes (12th highest) with this Traceback in the past week.

It might help to ensure that release upgrader is started with a _clean_ path; and then append anything it needs from the tarball.

probably ubuntu-release-upgrader should run by #!/usr/bin/python3 -E

and then append path of the tarball.

This (distro-info at least) doesn't seem to be an issue on Ubuntu 20.04 LTS which would affect upgrades to 20.10.

bdmurray@clean-focal-amd64:~$ pip3 install distro-info
Requirement already satisfied: distro-info in /usr/lib/python3/dist-packages (0.23ubuntu1)

Using -E didn't do the trick but using -I did which means we really want -s. From the manual:

-s Don't add user site directory to sys.path.

I ran a test upgrade from bionic to focal after appending "-s" to "#!/usr/bin/python3" in the focal script which is part of the dist-upgrader tarball and it worked well.

This bug was fixed in the package ubuntu-release-upgrader - 1:21.04.2

---------------
ubuntu-release-upgrader (1:21.04.2) hirsute; urgency=medium

  * DistUpgrade/DistUpgradeQuirks.py: ensure there are matches for both regexs
    before trying to use them.
  * DistUpgrade/dist-upgrade.py: When running the release upgrade do not add
    user site directory to sys.path thereby preventing user python packages
    from interfering with the upgrade process. (LP: #1848829)
  * Remove code relevant to peforming a distribution upgrade with a "cdrom" of
    the release to which you are upgrading as it has been hidden and broken
    for quite some time.

 -- Brian Murray <email address hidden> Mon, 09 Nov 2020 11:20:25 -0800

Hello Kent, or anyone else affected,

Accepted ubuntu-release-upgrader into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:20.10.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Kent, or anyone else affected,

Accepted ubuntu-release-upgrader into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:20.04.29 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

With the version of the dist-upgrader from -proposed I was able to upgrade a system with distro-info installed via pip3 to Ubuntu 20.10.

bdmurray@clean-focal-amd64:~$ pip3 list | grep distro-info
distro-info 0.0.0
bdmurray@clean-focal-amd64:~$ head -n10 /var/log/dist-upgrade/main.log
2020-11-19 10:25:53,100 INFO Using config files '['./DistUpgrade.cfg']'
2020-11-19 10:25:53,100 INFO uname information: 'Linux clean-focal-amd64 5.4.0-54-generic #60-Ubuntu SMP Fri Nov 6 10:37:59 UTC 2020 x86_64'
2020-11-19 10:25:53,366 INFO apt version: '2.0.2ubuntu0.1'
2020-11-19 10:25:53,366 INFO python version: '3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0]'
2020-11-19 10:25:53,368 INFO release-upgrader version '20.10.14' started
2020-11-19 10:25:53,371 INFO locale: 'en_US' 'UTF-8'
2020-11-19 10:25:53,434 DEBUG Using 'DistUpgradeViewText' view
2020-11-19 10:25:53,466 DEBUG enable dpkg --force-overwrite
2020-11-19 10:25:53,499 DEBUG creating statefile: '/var/log/dist-upgrade/apt-clone_system_state.tar.gz'
bdmurray@clean-focal-amd64:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.10
DISTRIB_CODENAME=groovy
DISTRIB_DESCRIPTION="Ubuntu 20.10"

With the version of the dist-upgrader from -proposed I was able to upgrade a system with distro-info installed via pip to Ubuntu 20.04 LTS. Unfortunately, as pip was uninstalled there isn't an easy to prove distro-info was installed.

bdmurray@clean-bionic-amd64:~$ head -n10 /var/log/dist-upgrade/main.log
2020-11-19 10:58:43,700 INFO Using config files '['./DistUpgrade.cfg.bionic']'
2020-11-19 10:58:43,700 INFO uname information: 'Linux clean-bionic-amd64 5.4.0-54-generic #60~18.04.1-Ubuntu SMP Fri Nov 6 17:25:16 UTC 2020
 x86_64'
2020-11-19 10:58:43,716 INFO apt version: '1.6.12ubuntu0.1'
2020-11-19 10:58:43,716 INFO python version: '3.6.9 (default, Oct 8 2020, 12:12:24)
[GCC 8.4.0]'
2020-11-19 10:58:43,717 INFO release-upgrader version '20.04.29' started
2020-11-19 10:58:43,720 INFO locale: 'en_US' 'UTF-8'
2020-11-19 10:58:43,787 DEBUG Using 'DistUpgradeViewText' view
2020-11-19 10:58:43,827 DEBUG enable dpkg --force-overwrite
2020-11-19 10:58:43,854 DEBUG creating statefile: '/var/log/dist-upgrade/apt-clone_system_state.tar.gz'
bdmurray@clean-bionic-amd64:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.1 LTS"

The distro-info version 0.0.0 is so weird...

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package ubuntu-release-upgrader - 1:20.04.29

---------------
ubuntu-release-upgrader (1:20.04.29) focal; urgency=medium

  * DistUpgrade/dist-upgrade.py: When running the release upgrade do not add
    user site directory to sys.path thereby preventing user python packages
    from interfering with the upgrade process. (LP: #1848829)

 -- Brian Murray <email address hidden> Tue, 17 Nov 2020 15:26:41 -0800

This bug was fixed in the package ubuntu-release-upgrader - 1:20.10.14

---------------
ubuntu-release-upgrader (1:20.10.14) groovy; urgency=medium

  * DistUpgrade/dist-upgrade.py: When running the release upgrade do not add
    user site directory to sys.path thereby preventing user python packages
    from interfering with the upgrade process. (LP: #1848829)

 -- Brian Murray <email address hidden> Tue, 17 Nov 2020 14:47:37 -0800

This just happened on an attempted do-release-upgrade from Kubuntu 20.04 to 22.04. I've attached a log of the upgrade.

How do I recover and continue the upgrade process at this point?

@Dan Parnham - the Traceback in your log file is different than the one which this bug report is about. Please open a new bug using 'ubuntu-bug ubuntu-release-upgrader' and let us know the number of the new report. Thanks in advance!

Apologies for posting to the wrong place - I've created a new report at https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1986387