crash in indexer.py, AttributeError: 'NoneType' object has no attribute 'unlink'

(k)ubuntu version 17.04
kernel version: 4.9.0-11.12
apt-xapian-index version: 0.47ubuntu11

This morning i received this mail from the cron.weekly run:

/etc/cron.weekly/apt-xapian-index:
Exception ignored in: <bound method ServerProgress.__del__ of <axi.indexer.ServerProgress object at 0x7f373e677710>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 306, in __del__
AttributeError: 'NoneType' object has no attribute 'unlink'

Status changed to 'Confirmed' because the bug affects multiple users.

Sorry, but forgot to mention that i'm running Ubuntu yakkety on my machine.

I didn't see any instances of this Traceback at errors.ubuntu.com. It's worth noting that apt-xapian-index is only included in Task: kubuntu-desktop, kubuntu-full, lubuntu-qt-desktop, ubuntukylin-desktop.

It looks like the same bug reappearing https://bugs.launchpad.net/ubuntu/+source/apt-xapian-index/+bug/1530518/comments/2

Running a-x-i directly doesn't trigger this bug, but the combination of nice and ionice makes it show up much more often. (When the index is already up to date)

Steps to reproduce:
 * `sudo update-apt-xapian-index`
 * Now that the index is updated we trigger the bug by running the following command (which is the same used in the cron job)
 * `sudo nice -n 19 ionice -c 3 update-apt-xapian-index --quiet`
 * may need to run 4 times or more

The intention of the proposed patch is to work around the problem by keeping a reference to the os.unlink function in the ServerProgress instance.
See http://stackoverflow.com/questions/17084260/imported-modules-become-none-when-running-a-function for an explanation of the issue in python.
Also see http://pydev.blogspot.ch/2015/01/creating-safe-cyclic-reference.html for a possible implementation using weakrefs.

The attachment "keep a reference to os.unlink" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Hello Carlo,

Could you please clarify which release this should go to? Yakkety is EOL now.

Also, irt to your patch, DEP-3 headers don't need "##" in front of them, please remove that.

Thank you for your interest in fixing this bug!

(unsubscribing ~ubuntu-sponsors as there's nothing to sponsor)

Hello Simon,

version 0.47ubuntu13 in zesty-updates is still affected.
version 0.47ubuntu8.3 in xenial-updates is affected too.

Artful beta1 has 0.47ubuntu13 like Zesty does, but I could not reproduce the bug. The reason could be that Python changed from 3.5.3 (in Zesty) to 3.6.2 (in Artful). The way Python unloads modules might have changed, but I couldn't tell for sure from the docs https://docs.python.org/3/whatsnew/3.6.html

Patch updated, as you suggested.

Sorry for the noise, patch updated again. This time it uses a weakref.finalizer instead of __del__. This is a little a cleaner solution. Moreover the finalizer is guaranteed to be called (at the latest at exit), whereas __del__ is not.

Carlo, I subscribed ~ubuntu-sponsors for you to review the debdiff.

Apologies for the delay, sponsored to Zesty. Unsubscribing ~ubuntu-sponsors for now, but it would be great if you could create a patch for Xenial and resubscribe when you're ready. ;)

Thanks!

This is the patch for Xenial. Tested in a 16.04.2 VM.

Hello Carlo, or anyone else affected,

Accepted apt-xapian-index into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt-xapian-index/0.47ubuntu13.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

It works for me on zesty with apt-xapian-index version 0.47ubuntu13.1 (python3 3.5.3-1)

This bug was fixed in the package apt-xapian-index - 0.47ubuntu13.1

---------------
apt-xapian-index (0.47ubuntu13.1) zesty; urgency=medium

  * debian/patches/08_race_no_attribute_unlink.patch: use a finalizer
    instead of __del__ to close and remove the socket.
    When ServerProgress.__del__() is called the global os module might
    already have been set to None by the Python shutdown, causing an
    exception. (LP: #1652641)

 -- Carlo Vanini <email address hidden> Fri, 24 Nov 2017 21:23:12 -0600

The verification of the Stable Release Update for apt-xapian-index has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Ubuntu 16.04 (GNOME + KDE)
Linux 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
VERSION="16.04.3 LTS (Xenial Xerus)"

Anacron job 'cron.weekly' on ubuntu-t180

Every time I got email message:
/etc/cron.weekly/apt-xapian-index:
Exception ignored in: <bound method ServerProgress.__del__ of <axi.indexer.ServerProgress object at 0x7fdf1ad81b00>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 306, in __del__
AttributeError: 'NoneType' object has no attribute 'unlink'

Regards,
Jorma Hytonen
StMichel, Finland

Hello Carlo, or anyone else affected,

Accepted apt-xapian-index into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt-xapian-index/0.47ubuntu8.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Works for me on Xenial with apt-xapian-index version 0.47ubuntu8.4

Ref to #20
Does this mean that I need to update my Ubuntu 16.04 system?
Download and install apt-xapian-index_0.47ubuntu8.4.tar.gz
Or can I wait for a regular update?

Hi Jorma,
now that the verification is done it will be moved to xenial-updates and it will be part of the regular updates (and then the status will become "Fix Released").

If, instead, you want to have the fix before that, or you want to confirm that the problem is indeed fixed for you too (that's the goal of verification) you can download and install the new version by hand.

Hi Carlo
Thank you for the information.
I look forward to a regular update.

Thanks for your activity in solving the problem.

Regards,
Jorma

This bug was fixed in the package apt-xapian-index - 0.47ubuntu8.4

---------------
apt-xapian-index (0.47ubuntu8.4) xenial; urgency=medium

  * debian/patches/08_race_no_attribute_unlink.patch: use a finalizer
    instead of __del__ to close and remove the socket.
    When ServerProgress.__del__() is called the global os module might
    already have been set to None by the Python shutdown, causing an
    exception. (LP: #1652641)

 -- Carlo Vanini <email address hidden> Fri, 23 Feb 2018 08:44:53 -0800