AttributeError: _factory - lazy imports are not threadsafe

Bug #396819 reported by Michael Hudson-Doyle
34
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Bazaar
Fix Released
High
Unassigned
loggerhead
Triaged
Critical
Unassigned
loggerhead-breezy
Triaged
High
Unassigned

Bug Description

I can reasonably reliably reproduce this with loggerhead by firing it up and hitting it with ab -c, and it's not hard to see what happens in the code.

If two threads concurrently execute ScopeReplacer.__getattribute__ on the same instance, it's easy to see that one can execute _cleanup before the other enters _replace, and so the IllegalUseOfScopeReplacer condition gets hit.

In my case, simply not raising an exception in this situation would probably work, though I don't know if that would have any serious drawbacks (it looks to me like executing _import() twice won't have any real drawbacks, but it's not obvious).

File "/var/launchpad/tmp/eggs/bzr-2.2.2_lp_2-py2.6-linux-x86_64.egg/bzrlib/osutils.py", line 2065, in send_all
   report_activity(sent, 'write')
 File "/var/launchpad/tmp/eggs/bzr-2.2.2_lp_2-py2.6-linux-x86_64.egg/bzrlib/smart/medium.py", line 176, in _report_activity
   ui.ui_factory.report_transport_activity(self, bytes, direction)
 File "/var/launchpad/tmp/eggs/bzr-2.2.2_lp_2-py2.6-linux-x86_64.egg/bzrlib/lazy_import.py", line 108, in __getattribute__
   _cleanup()
 File "/var/launchpad/tmp/eggs/bzr-2.2.2_lp_2-py2.6-linux-x86_64.egg/bzrlib/lazy_import.py", line 97, in _cleanup
   del self._factory
AttributeError: _factory

Related branches

Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 396819] [NEW] lazy imports are not threadsafe

 status triaged
 importance high

I suspect this is the cause for launchpad-code, and test runs of just a
couple of tests encountering scope errors in the smart server from time
to time.

-Rob

Changed in bzr:
importance: Undecided → High
status: New → Triaged
Martin Pool (mbp)
Changed in bzr:
status: Triaged → Confirmed
Jelmer Vernooij (jelmer)
tags: added: lazy-imports
Revision history for this message
Robert Collins (lifeless) wrote :

I just trigger this in an ec2 test of LP.

description: updated
summary: - lazy imports are not threadsafe
+ AttributeError: _factory - lazy imports are not threadsafe
Revision history for this message
Diogo Matsubara (matsubara) wrote :

Dupe of bug 716770?

Changed in loggerhead:
status: New → Triaged
importance: Undecided → Critical
tags: added: oops
Revision history for this message
Benji York (benji) wrote :

Bug 716770 is indeed a dupe of this. I have a branch that fixes this bug at lp:~benji/bzr/bug-702914.

Changed in loggerhead:
status: Triaged → In Progress
Changed in bzr:
status: Confirmed → In Progress
Changed in loggerhead:
status: In Progress → Triaged
Changed in bzr:
assignee: nobody → Benji York (benji)
Revision history for this message
Martin Packman (gz) wrote :

Branch landed on bzr.dev at revision 6101, but may also want backporting to earlier series?

Changed in bzr:
milestone: none → 2.5b1
status: In Progress → Fix Released
Revision history for this message
Martin von Gagern (gagern) wrote :

Problem appears to be only moved. See lp:~gagern/bzr/bug-702914 for more tests still exposing this.

Imho there are only two ways to achieve thread safety: either use locks or risk repeating the import.

In any case, you can no longer rely on exceptions to detect repeated imports. Threads can enter methods of the replacer object concurrently, and if they do so, then the only threadsafe solution is having them both succeed without any exception. You could still enable the current cleanup & exception mechanism for selftests only, but for normal production use, the object should either never clean up, thus allowing repeated replacements, or it should fall back to proxying if the factory has already been cleaned up. The former is easier, and as repeated imports should be a rare scenario, the performance gain from the latter should be negligible. Particularly as even the fallback to proxying won't always prevent repeated imports, as there is still a chance the other thread was interrupted after importing but before cleaning. And I believe that cleaning is not worth the effort, as the whole replacer object will be cleaned soon after the replacement occurred. Therefore cleaning the attributes should have little benefit.

Benji York (benji)
Changed in bzr:
assignee: Benji York (benji) → nobody
Jelmer Vernooij (jelmer)
Changed in loggerhead-breezy:
status: New → Triaged
importance: Undecided → High
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.