Quantal failed to install: ubiquity crashed in apt/progress/text.py in pulse: OverflowError: Python int too large to convert to C long

In UbiquitySyslog.txt:

Jul 28 11:40:09 ubuntu ubiquity: Error in function pulse
Jul 28 11:40:09 ubuntu plugininstall.py: Exception during installation:
Jul 28 11:40:09 ubuntu plugininstall.py: Traceback (most recent call last):
Jul 28 11:40:09 ubuntu plugininstall.py: File "/usr/lib/ubiquity/ubiquity/install_misc.py", line 276, in pulse
Jul 28 11:40:09 ubuntu plugininstall.py: AcquireProgress.pulse(self, owner)
Jul 28 11:40:09 ubuntu plugininstall.py: File "/usr/lib/python3/dist-packages/apt/progress/text.py", line 167, in pulse
Jul 28 11:40:09 ubuntu plugininstall.py: for worker in owner.workers:
Jul 28 11:40:09 ubuntu plugininstall.py: OverflowError: Python int too large to convert to C long
Jul 28 11:40:09 ubuntu plugininstall.py:
Jul 28 11:45:15 ubuntu ubiquity: Gtk-Message: Not loading module "atk-bridge": The functionality is provided by GTK natively. Please try to not load it.

Status changed to 'Confirmed' because the bug affects multiple users.

This is a regression from Python 2. Here's a minimal test case:

<cjwatson@sarantium ~>$ python -c 'import apt_pkg; print(apt_pkg.size_to_str(2147483648))'
2147 M
<cjwatson@sarantium ~>$ python3 -c 'import apt_pkg; print(apt_pkg.size_to_str(2147483648))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
OverflowError: Python int too large to convert to C long

Hmm:

% python -c 'import apt_pkg; print(apt_pkg.size_to_str(2147483648))'
2147 M
% python3 -c 'import apt_pkg; print(apt_pkg.size_to_str(2147483648))'
2147 M

But even if that was the problem, then it confirms my suspicion as discussed with James on IRC, that some exception state is not being checked. If you look at the traceback in UbiquitySyslog.txt, the error is getting raised during iteration over owner.workers. Which makes no sense <wink>. Most likely this OverflowError is getting set some place but not checked at the point of error, and the traceback is a red herring. That would mean there are two bugs here (although as you can see, I cannot reproduce Colin's test case).

$ python3 -c 'import apt_pkg; print(apt_pkg.size_to_str(2147483648000))'
2147 G
$ python3 -c 'import apt_pkg; print(apt_pkg.size_to_str(2147483648000000))'
2147 T
$ python3 -c 'import apt_pkg; print(apt_pkg.size_to_str(2147483648000000000))'
2147 P
$ python3 -c 'import apt_pkg; print(apt_pkg.size_to_str(2147483648000000000000))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
OverflowError: Python int too large to convert to C long

Barry, did you try to reproduce on 32bit or 64bit ? This bug and all the duplicates are on 32bit.

On Oct 01, 2012, at 08:08 PM, Jean-Baptiste Lallement wrote:

>Barry, did you try to reproduce on 32bit or 64bit ? This bug and all the
>duplicates are on 32bit.

D'oh. Nope, only tried on 64bit. Let me try again.

Okay, this is clearly a problem in StrSizeToStr() and I think both bugs are related to this. First, StrSizeToStr() never checks whether an exception occurs during conversion, so that's likely the problem of the bogus traceback. Second, StrSizeToStr() needs to PyLong_Check() *before* it PyInt_Check()s because in Python 3, the latter is aliased to the former, however, if the object is an *actual* PyLong, it will be convertable to a double (as per PyLong_AsDouble()) for more values than it will be convertable to a long (as per PyInt_AsLong(), a.k.a. in Python 3, PyLong_asLong()).

The attached diff should fix both problems. I've tested it with various values in an i386 chroot and it seems to work correctly now.

Just don't try to convert 21474836480000000000000000000. I guess SizeToStr() can't handle anything over thousands of yottabytes. Darn, I might have to return my new hard drive. This patch doesn't do much about that.

Here's a test case.

This bug was fixed in the package python-apt - 0.8.7ubuntu2

---------------
python-apt (0.8.7ubuntu2) quantal; urgency=low

  * python/string.cc, tests/test_lp1030278.py: Fix StrSizeToStr() so that
    1) it first checks for PyLong-ness so that in Python 3 on i386, it
    will be able to convert larger numbers (via doubles rather than ints);
    2) before doing the conversions through the apt API, check to see if a
    Python exception occurred, e.g. OverflowError, and return an error
    condition in that case instead of masking it. (LP: #1030278)
 -- Barry Warsaw <email address hidden> Mon, 01 Oct 2012 18:04:35 -0400

Hi Barry,

Thanks for this - we seem to have collided slightly on this one! :)

Now that we've got a fix in quantal, I've undone my changes, but have raised an MP for additional checks and a formal test for size_to_str(). See:

lp:~jamesodhunt/python-apt/test-for-size_to_str

You're right about the yottabytes issue - see bug 1059783 raised yesterday.