Librarian requests occasionally fail: 'NoneType' object has no attribute 'get_object'

Bug #1420046 reported by William Grant
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Critical
Colin Watson

Bug Description

Roughly 25 Swift Librarian requests fail each day like OOPS-f1c692cf8f319c390c1a8940f8187009:

Unhandled Error
Traceback (most recent call last):
Failure: exceptions.AttributeError: 'NoneType' object has no attribute 'get_object'

Possibly the Swift connection pool ends up with a None in it?

Related branches

Revision history for this message
Stuart Bishop (stub) wrote :

An assert in ConnectionPool.put will locate anywhere that does this. Or log the traceback and throw the None away.

Revision history for this message
William Grant (wgrant) wrote :

https://code.launchpad.net/~wgrant/launchpad/couple-of-oopses/+merge/249788 adds the None check. Hopefully we'll get some useful tracebacks, though it's possible that this theory is completely wrong.

Revision history for this message
William Grant (wgrant) wrote :

New assertion confirms the theory, eg. OOPS-0858ab81859c279c8b8d55322b57402c:

Traceback (most recent call last):
  File "/srv/launchpadlibrarian.net/production/launchpad3-rev-17355/eggs/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 421, in errback
    self._startRunCallbacks(fail)
  File "/srv/launchpadlibrarian.net/production/launchpad3-rev-17355/eggs/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 488, in _startRunCallbacks
    self._runCallbacks()
  File "/srv/launchpadlibrarian.net/production/launchpad3-rev-17355/eggs/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 575, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/srv/launchpadlibrarian.net/production/launchpad3-rev-17355/eggs/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1126, in gotResult
    _inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
  File "/srv/launchpadlibrarian.net/production/launchpad3-rev-17355/eggs/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1068, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/srv/launchpadlibrarian.net/production/launchpad3-rev-17355/eggs/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/srv/launchpadlibrarian.net/production/launchpad3-rev-17355/lib/lp/services/librarianserver/web.py", line 322, in resumeProducing
    data = yield self.fileObject.read(self.bufferSize)
  File "/srv/launchpadlibrarian.net/production/launchpad3-rev-17355/eggs/Twisted-13.0.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1070, in _inlineCallbacks
    result = g.send(result)
  File "/srv/launchpadlibrarian.net/production/launchpad3-rev-17355/lib/lp/services/librarianserver/storage.py", line 150, in read
    swift.connection_pool.put(self._swift_connection)
  File "/srv/launchpadlibrarian.net/production/launchpad3-rev-17355/lib/lp/services/librarianserver/swift.py", line 351, in put
    "%r is not a swiftclient Connection." % swift_connection)
exceptions.AssertionError: None is not a swiftclient Connection.

Colin Watson (cjwatson)
Changed in launchpad:
status: Triaged → In Progress
assignee: nobody → Colin Watson (cjwatson)
Revision history for this message
Launchpad QA Bot (lpqabot) wrote :
tags: added: qa-needstesting
Changed in launchpad:
status: In Progress → Fix Committed
Revision history for this message
Colin Watson (cjwatson) wrote :

We don't have a relevant staging setup here, so trusting the test suite is about all we can do in terms of QA.

tags: added: qa-untestable
removed: qa-needstesting
Revision history for this message
Colin Watson (cjwatson) wrote :

Oh, actually, the staging librarian is separate and I think it has Swift? It still seems to be working OK, anyway.

tags: added: qa-ok
removed: qa-untestable
Colin Watson (cjwatson)
Changed in launchpad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.