/etc/cron.weekly/apt-xapian-index reports TypeError

Also got it on cold boot; join the crash file

Almost positive what's happening is that Python shutdown process is interacting badly with ServerProgress's __del__().

__del__() methods are not the best way to reclaim resources, especially when you leave it to Python's shutdown process to implicitly free objects, like what happens in update-apt-xapian-index. Here's what happens:

When Python shutdown, after it's run the atexit handlers, it then tries to free up all objects that are still implicitly alive because of reference counts. It does this first by emptying the __dict__s of all modules, which will decref any still referenced globals. It actually does this first by setting all keys in the module's globals to None. Because Python can't know what's a safe order to do this, it will do it randomly.

Notice a few things. In update-a-x-i, indexer is a module global. When its .lock() method is called, it sets self.progress to a ServerProgress instance and neither instance is explicitly freed, so they live until Python shutdown. Now, when Python gets around to clearing the module dicts as described above, it will set u-a-x-i's indexer instance to None, which will decref it to zero, and so freeing the indexer instance.

Because the ServerProgress instance is kept alive by the reference from the indexer's self.progress attribute, once the indexer is freed, the ServerProgress instance probably decrefs to zero and then *it* gets freed. This will call ServerProgress.__del__(). Line 306 is the os.unlink() call and you'll notice that you get an exception that None is not callable. Why is that?

Well, because if the os module's dictionary got cleared *before* the ServerProgress instance's __del__() got called, the os module is still valid, and it still has a name bound in it called 'unlink' but that name has been bound to None! If you're lucky, the ServerProgress instance gets decrefed away before os gets cleared, but as you've observed in IRC, about every third time, you're unlucky, and os gets cleared first (remember, the order is random). In those cases... boom!

I don't like the design here, since relying on __del__ methods to free resources isn't good Python style, especially when those resources can be freed explicitly, but fixing that is probably more work than we want to do for a package that is abandoned in Debian. A bandaid might be easy though: before the sys.exit() call in u-a-x-i, let's try to explicitly `del indexer`. That will cause all the cascading decrefs before Python's shutdown machinery clears out the os module. I'll attach a diff to try.

diff --git a/update-apt-xapian-index b/update-apt-xapian-index
index 5b38c5f..f6e434a 100755
--- a/update-apt-xapian-index
+++ b/update-apt-xapian-index
@@ -103,4 +103,6 @@ if opts.update:
 else:
     indexer.rebuild(opts.pkgfile)

+# Free the resources explicitly. See LP: #1530518
+del indexer
 sys.exit(0)

I've tested Barry's change multiple times and it works for me.

Hello glitsj16, or anyone else affected,

Accepted apt-xapian-index into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/apt-xapian-index/0.47ubuntu8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Note that this issue has not happened since early Feb on my system (was a transitional problem)

This bug was fixed in the package apt-xapian-index - 0.47ubuntu8

---------------
apt-xapian-index (0.47ubuntu8) xenial; urgency=medium

  * debian/patches/05_python3.patch: del the indexer instance explicitly
    before sys.exit(), so that it gets decrefed and freed before Python's
    implicit shutdown machinery can potentially leave an empty os module for
    ServerProgress.__del__() to find. (LP: #1530518)

 -- Barry Warsaw <email address hidden> Wed, 13 Apr 2016 17:36:22 -0400

Confirming apt-xapian-index 0.47ubuntu8 fixes the issue. Thanks to all involved.

I've been getting identical emails to the above from my system recently.

I have checked /usr/sbin/update-apt-xapian-index and it has the bug fix suggested above.

$ uname -a
Linux 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -rd
Description: Ubuntu 16.04.2 LTS
Release: 16.04

$ apt-cache policy apt-xapian-index
apt-xapian-index:
  Installed: 0.47ubuntu8.3
  Candidate: 0.47ubuntu8.3
  Version table:
 *** 0.47ubuntu8.3 500
        500 http://gb.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
        500 http://gb.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages
        100 /var/lib/dpkg/status
     0.47ubuntu8 500
        500 http://gb.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
        500 http://gb.archive.ubuntu.com/ubuntu xenial/universe i386 Packages

$ sudo /usr/sbin/update-apt-xapian-index
The index /var/lib/apt-xapian-index is up to date
Exception ignored in: <bound method ServerProgress.__del__ of <axi.indexer.ServerProgress object at 0x7fd7aeb14748>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 306, in __del__
TypeError: 'NoneType' object is not callable

I confirm the same behaviour and error output as ikp1 reports! This with Ubuntu 16.10:

:~$ uname -a
Linux rudix 4.8.0-41-generic #44-Ubuntu SMP Fri Mar 3 15:27:17 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
:~$ lsb_release -rd
Description: Ubuntu 16.10
Release: 16.10
:~$ sudo /usr/sbin/update-apt-xapian-index
The index /var/lib/apt-xapian-index is up to date
Exception ignored in: <bound method ServerProgress.__del__ of <axi.indexer.ServerProgress object at 0x7f3f25b46c88>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 306, in __del__
AttributeError: 'NoneType' object has no attribute 'unlink'
hofer@rudix:~$

I'm also still getting this error.

:~$ uname -a
Linux request 3.13.0-79-generic #123-Ubuntu SMP Fri Feb 19 14:27:58 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
:~$ lsb_release -d
Description: Ubuntu 16.04.2 LTS
:~$ apt-cache policy apt-xapian-index
apt-xapian-index:
  Installed: 0.47ubuntu8.3
  Candidate: 0.47ubuntu8.3
  Version table:
 *** 0.47ubuntu8.3 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages
        100 /var/lib/dpkg/status
     0.47ubuntu8 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe i386 Packages

Not sure if its meaningful but apt-xapian-index script is old and the server has been updated through several Ubuntu LTS versions

/etc/cron.weekly:
drwxr-xr-x 2 root root 4096 Apr 21 09:15 .
drwxr-xr-x 122 root root 12288 Jun 9 15:29 ..
-rwxr-xr-x 1 root root 730 Sep 24 2011 apt-xapian-index

I confirm here as well the Bug still occurs on Ubuntu 16.04 64-bit

System Package Version:

apt-xapian-index:
  Installed: 0.47ubuntu8.3
  Candidate: 0.47ubuntu8.3
  Version table:
 *** 0.47ubuntu8.3 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/universe i386 Packages
        100 /var/lib/dpkg/status
     0.47ubuntu8 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages
        500 http://us.archive.ubuntu.com/ubuntu xenial/universe i386 Packages

Running Command:

$ sudo /usr/sbin/update-apt-xapian-index
The index /var/lib/apt-xapian-index is up to date
Exception ignored in: <bound method ServerProgress.__del__ of <axi.indexer.ServerProgress object at 0x7f82bd7c5898>>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/axi/indexer.py", line 306, in __del__
AttributeError: 'NoneType' object has no attribute 'unlink'

There are 2 more sys.exit(0) calls in the code, and they need their del indexer as well. Indeed, we keep getting the error message despite the presence of del indexer before the sys.exit(0) at the end of the file. Adding del indexer to the other 2 fixes it.

... and possibly the two sys.exit's with non-zero exit-status need it as well.

I can confirm that I have 0.47ubuntu8.3 installed and I'm *STILL* getting this ... definitely not fixed