13.10 to 14.04 upgrade: UnicodeEncodeError: 'ascii' codec can't encode character u'\xab' in position 13: ordinal not in range(128) Logged from file DistUpgradeCache.py, line 661 in apt.log

From apt.log:

Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
    record.message = record.getMessage()
  File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage
    msg = msg % self.args
UnicodeEncodeError: 'ascii' codec can't encode character u'\xab' in position 13: ordinal not in range(128)
Logged from file DistUpgradeCache.py, line 661

This is the problematic line:

            logging.error("Dist-upgrade failed: '%s'", e)

Maybe, the comma is causing the issue.

Some experiencing this bug could test the the theory by editing the file DistUpgradeCache.py in /tmp/ubuntu-release-upgrader-TMPNAME and then cd'ing into that directory and running sudo ./trusty and seeing if they get the same traceback.

Well I can confirm that line 661 was the one that tried to log error, but changing
logging.error("Dist-upgrade failed: '%s'", e)
to
logging.error("Dist-upgrade failed: '%s'", unicode(e))

only results in correctly printing 'Dist-upgrade' failed error to log file:

2014-04-22 20:06:46,539 DEBUG The package 'update-manager-kde' is marked for removal but it's in the removal blacklist
2014-04-22 20:06:49,343 ERROR Dist-upgrade failed: 'Pakiet "update-manager-kde" jest oznaczony do usunięcia, lecz znajduje się na liście pakietów, których nie należy usuwać.'
2014-04-22 20:06:49,345 DEBUG abort called

and does NOT fix the root cause. (Meaning that this change only adds one line to logfile).

I might add that u'xab' appears to be character « according to http://es.wikipedia.org/wiki/ISO_8859-1 (and my test with python2.7).

Thanks for uploading the fix for this bug report to -proposed. However, when reviewing the package in -proposed and the details of this bug report I noticed that the bug description is missing information required for the SRU process. You can find full details at http://wiki.ubuntu.com/StableReleaseUpdates#Procedure but essentially this bug is missing some of the following: a statement of impact, a test case and details regarding the regression potential. Thanks in advance!

Hello Patrizio, or anyone else affected,

Accepted ubuntu-release-upgrader into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:0.220.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Sorry for delay, but I cannot install 1:0.220.3 . I enabled proposed, run
apitude update
but all I see in aptitude versions ubuntu-release-upgrader
is
1:0.205.7 saucy-proposed 500
Is 64bit package is in repo already? I selected Ubuntu main repo instead of local mirror but it did not help.

To use the upgrader from Trusty proposed you need to use the following command:

"do-release-upgrade --proposed --frontend DistUpgradeViewText"

Notice the --proposed switch. This is because the release upgrade process downloads a new version of itself from the release you are upgrading to.

Ok I tested it on pl locale. Looks like in newer version it looks better, because I got

DEBUG The package 'update-manager-kde' is marked for removal but it's in the removal blacklist

line in main.log (which previoulsy was not there), but unfortunetlly my proposed fix for DistUpgradeCache was invalid
since error was still shown in apt.log :
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0119' in position 51: ordinal not in range(128)
Logged from file DistUpgradeCache.py, line 661

(\u0119 i ę character which probably appeared in polish version of blacklist line:
Dist-upgrade failed: 'Pakiet "update-manager-kde" jest oznaczony do usunięcia, lecz znajduje się na liście pakietów, których nie należy usuwać.'
I do not know why it try to encode this line when it shown english version in main.log)

I attach patch that fixes apt.log issue for me.

This bug was fixed in the package ubuntu-release-upgrader - 1:14.10.2

---------------
ubuntu-release-upgrader (1:14.10.2) utopic; urgency=low

  * debian/test/control:
    - add missing pep8 test dependency
  * tests/test_xorg_fix_intrepid.py:
    -
 -- Michael Vogt <email address hidden> Sat, 03 May 2014 08:37:51 +0200

I still get the mentioned error-messages.
What has to be done to benefit from the fix ?

I finaly succeeded upgrading to 14.04.
The simple solution was to deinstall 'update-manager-kde' manually before restarting the upgrade.

I've gone ahead and recreated this bug following the description by adding foomatic-filters to the blacklist. I received the Traceback indicated with the ubuntu-release-upgrader version 0.220.2, but not with 0.220.3. I was using French for my test. Additionally, I did see unicode characters in my /var/log/dist-upgrade/main.log file.

The message included "ERROR Dist-upgrade failed: 'Le paquet << foomatic-fitlers >> est marqué". The Dist-upgrade failure should not appear in /var/log/dist-upgrade/apt.log so I'm not certain what Sebastian was seeing, but I'm going to mark this as verification done.

Hello Patrizio, or anyone else affected,

Accepted ubuntu-release-upgrader into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:0.220.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug was fixed in the package ubuntu-release-upgrader - 1:0.220.4

---------------
ubuntu-release-upgrader (1:0.220.4) trusty-proposed; urgency=low

  * DistUpgrade/DistUpgradeCache.py:
    - fix crash in free space check for certain locales (LP: #1172740)
    - removal of essential packages that are not in the main arch
      is ok (LP: #1307904)
    - fix str/unicode crash in certain locales when a package is
      found in the removal blacklist (LP: #1310053)
  * DistUpgrade/DistUpgradeViewGtk3.py:
   - make details dialog resize when details are expanded/collapsed
     (LP: #1072460)
    - load webkit only when needed
  * DistUpgrade/DistUpgradeViewText.py:
    - fix py3 crash in text mode
  * DistUpgrade/DistUpgradeMain.py:
    - log python version
  * po/{eo,id,ug}.po:
    - fix translations of check-new-release-gtk which cause a crash
      (LP: #1311396)
  * data/demoted.cfg:
    - autmatic refresh of the demoted packages
  * po/*.po:
    - automatic refresh from launchpad
 -- Michael Vogt <email address hidden> Thu, 24 Apr 2014 14:57:39 +0200

The verification of the Stable Release Update for ubuntu-release-upgrader has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.