lp.services.job.tests.test_runner.TestJobRunner.test_runJobHandleErrors_oops_generated_user_notify_fails fails intermittently/rarely in parallel tests

Bug #993510 reported by Gary Poster
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Francesco Banconi

Bug Description

I can reliably reproduce this error in a Lucid LXC container with ./bin/test --load-list tests.txt if tests.txt contains these two tests:

lp.archivepublisher.tests.test_publish_ftpmaster.TestPublishFTPMasterScript.test_getDirtySecuritySuites_returns_security_suites
lp.services.job.tests.test_runner.TestJobRunner.test_runJobHandleErrors_oops_generated_user_notify_fails

(The test passes by itself.)

======================================================================
FAILURE: lp.services.job.tests.test_runner.TestJobRunner.test_runJobHandleErrors_oops_generated_user_notify_fails
worker ID: worker-2
----------------------------------------------------------------------
Traceback (most recent call last):
_StringException: librarian-log: {{{
2012-05-03 00:50:59+0530 [-] Log opened.
2012-05-03 00:50:59+0530 [-] twistd 11.1.0 (/usr/bin/python2.6 2.6.5) starting up.
2012-05-03 00:50:59+0530 [-] reactor class: twisted.internet.pollreactor.PollReactor.
2012-05-03 00:50:59+0530 [-] FileUploadFactory starting on 47202
2012-05-03 00:50:59+0530 [-] Starting factory <lp.services.librarianserver.libraryprotocol.FileUploadFactory instance at 0xb8e338c>
2012-05-03 00:50:59+0530 [-] Site starting on 39654
2012-05-03 00:50:59+0530 [-] Starting factory <twisted.web.server.Site instance at 0xb8e328c>
2012-05-03 00:50:59+0530 [-] FileUploadFactory starting on 36628
2012-05-03 00:50:59+0530 [-] Starting factory <lp.services.librarianserver.libraryprotocol.FileUploadFactory instance at 0xb8e38ec>
2012-05-03 00:50:59+0530 [-] Site starting on 51591
2012-05-03 00:50:59+0530 [-] Starting factory <twisted.web.server.Site instance at 0xb8e3d2c>
2012-05-03 00:50:59+0530 [-] Not using upstream librarian
2012-05-03 00:50:59+0530 [-] daemon ready!
}}}

oops-0: {{{
Oops-Id: OOPS-3aacc5868ee52ab4a348b0c20c017d86
Exception-Type: RaisingJobException
Exception-Value: oops notifying users
Date: 2012-05-02T19:27:56.540247+00:00
Branch: devel
Revision: 15189
URL: None
Duration: -1
Oops-Reporter: T

%3Coops-message-2%3E=%7B%27foo%27: %27bar%27%7D
%3Coops-message-3%3E=Failure in _doOops: oops notifying users
path=/var/lib/buildbot/slaves/slave/lucid-devel/build/bin/test
script_name=publishftpmaster

00000-00003@memcache-set MemcachedLayer__live_test
00079-00081@memcache-set MemcachedLayer__live_test
00181-00183@memcache-set MemcachedLayer__live_test
05340-05352@librarian-connection http://localhost:39654/93/filename-100098
05353-05354@librarian-read http://localhost:39654/93/filename-100098
05357-05362@librarian-connection http://localhost:39654/94/filename-100124
05362-05362@librarian-read http://localhost:39654/94/filename-100124
05727-05733@librarian-connection http://localhost:39654/95/filename-100150
05734-05734@librarian-read http://localhost:39654/95/filename-100150
05736-05741@librarian-connection http://localhost:39654/94/filename-100124
05741-05741@librarian-read http://localhost:39654/94/filename-100124
06131-06139@librarian-connection http://localhost:39654/96/filename-100176
06140-06140@librarian-read http://localhost:39654/96/filename-100176
06142-06147@librarian-connection http://localhost:39654/94/filename-100124
06147-06147@librarian-read http://localhost:39654/94/filename-100124
14629-14640@librarian-connection http://localhost:39654/94/copyme_2.8-1_source.changes
14641-14642@librarian-read http://localhost:39654/94/copyme_2.8-1_source.changes
41333-41342@librarian-connection http://localhost:39654/93/unique-from-factory-py-line3282-101297.pot
41344-41344@librarian-read http://localhost:39654/93/unique-from-factory-py-line3282-101297.pot
43637-43646@librarian-connection http://localhost:39654/93/testing.pot
43647-43647@librarian-read http://localhost:39654/93/testing.pot
43650-43654@librarian-connection http://localhost:39654/94/sr.po
43655-43655@librarian-read http://localhost:39654/94/sr.po
53779-53820@librarian-connection http://localhost:39654/1/netapplet-1.0.0.tar.gz
53821-53821@librarian-read http://localhost:39654/1/netapplet-1.0.0.tar.gz
117810-117822@librarian-connection http://localhost:39654/95/foo_666.dsc
117823-117823@librarian-read http://localhost:39654/95/foo_666.dsc
117824-117824@librarian-read http://localhost:39654/95/foo_666.dsc
163327-163336@librarian-connection http://localhost:39654/95/foo_666.dsc
163337-163337@librarian-read http://localhost:39654/95/foo_666.dsc
163337-163337@librarian-read http://localhost:39654/95/foo_666.dsc
168474-168487@librarian-connection http://localhost:39654/93/hwinfo.xml
168487-168487@librarian-read http://localhost:39654/93/hwinfo.xml
190640-190649@librarian-connection http://localhost:39654/94/boing_1.0_source.changes
190650-190650@librarian-read http://localhost:39654/94/boing_1.0_source.changes
196903-196912@librarian-connection http://localhost:39654/95/foo_1.dsc
196913-196913@librarian-read http://localhost:39654/95/foo_1.dsc
196914-196914@librarian-read http://localhost:39654/95/foo_1.dsc
262559-262596@librarian-connection http://localhost:39654/1/netapplet-1.0.0.tar.gz
262596-262596@librarian-read http://localhost:39654/1/netapplet-1.0.0.tar.gz
273754-273762@librarian-connection http://localhost:39654/93/hwinfo.xml
273763-273763@librarian-read http://localhost:39654/93/hwinfo.xml
285616-285626@librarian-connection http://localhost:39654/93/lazy-building_1.0_source.changes
285626-285626@librarian-read http://localhost:39654/93/lazy-building_1.0_source.changes
335127-335136@librarian-connection http://localhost:39654/93/filename-105666
335137-335137@librarian-read http://localhost:39654/93/filename-105666
358219-358230@librarian-connection http://localhost:39654/100/zeca_1.0.orig.tar.gz
358230-358230@librarian-read http://localhost:39654/100/zeca_1.0.orig.tar.gz
358231-358231@librarian-read http://localhost:39654/100/zeca_1.0.orig.tar.gz
358233-358238@librarian-connection http://localhost:39654/101/zeca_1.0-2.diff.gz
358239-358239@librarian-read http://localhost:39654/101/zeca_1.0-2.diff.gz
358240-358240@librarian-read http://localhost:39654/101/zeca_1.0-2.diff.gz
358243-358249@librarian-connection http://localhost:39654/102/zeca_1.0-2.dsc
358249-358249@librarian-read http://localhost:39654/102/zeca_1.0-2.dsc
358250-358250@librarian-read http://localhost:39654/102/zeca_1.0-2.dsc
358253-358258@librarian-connection http://localhost:39654/95/zeca_1.0.orig.tar.gz
358259-358259@librarian-read http://localhost:39654/95/zeca_1.0.orig.tar.gz
358260-358260@librarian-read http://localhost:39654/95/zeca_1.0.orig.tar.gz
358263-358268@librarian-connection http://localhost:39654/96/zeca_1.0-1.diff.gz
358269-358269@librarian-read http://localhost:39654/96/zeca_1.0-1.diff.gz
358270-358270@librarian-read http://localhost:39654/96/zeca_1.0-1.diff.gz
358272-358278@librarian-connection http://localhost:39654/97/zeca_1.0-1.dsc
358279-358279@librarian-read http://localhost:39654/97/zeca_1.0-1.dsc
358280-358280@librarian-read http://localhost:39654/97/zeca_1.0-1.dsc
381046-381056@librarian-connection http://localhost:39654/93/filename-106394
381057-381057@librarian-read http://localhost:39654/93/filename-106394
386709-386718@librarian-connection http://localhost:39654/93/foo_1.0_source.changes
386719-386719@librarian-read http://localhost:39654/93/foo_1.0_source.changes

Traceback (most recent call last):
  Module lazr.jobrunner.jobrunner, line 199, in runJobHandleError
    job.notifyUserError(e)
  Module lp.services.job.tests.test_runner, line 122, in notifyUserError
    raise RaisingJobException('oops notifying users')
RaisingJobException: oops notifying users
}}}

oops-1: {{{
Oops-Id: OOPS-be9faf3485a50ec7731f2d6dc78dc886
Exception-Type: RaisingJobException
Exception-Value: oops notifying users
Date: 2012-05-02T19:27:56.559661+00:00
Branch: devel
Revision: 15189
Duration: -1
Oops-Reporter: T

%3Coops-message-2%3E=%7B%27foo%27: %27bar%27%7D

Traceback (most recent call last):
  Module lazr.jobrunner.jobrunner, line 199, in runJobHandleError
    job.notifyUserError(e)
  Module lp.services.job.tests.test_runner, line 122, in notifyUserError
    raise RaisingJobException('oops notifying users')
RaisingJobException: oops notifying users
}}}

Traceback (most recent call last):
  File "/var/lib/buildbot/slaves/slave/lucid-devel/build/lib/lp/services/job/tests/test_runner.py", line 366, in test_runJobHandleErrors_oops_generated_user_notify_fails
    self.assertEqual(1, len(self.oopses))
MismatchError: 1 != 2

Related branches

Brad Crittenden (bac)
description: updated
Revision history for this message
Gary Poster (gary) wrote :

We though this was a duplicate of bug 992692 but since the fix for that bug landed, we've seen this error again. The failure looks identical. http://pastebin.ubuntu.com/1001180/ I'm seeing if I can identify a way to replicate this reliably.

Revision history for this message
Gary Poster (gary) wrote :

I can reliably dupe the failure using --load-list with a file containing only these two tests:

lp.archivepublisher.tests.test_publish_ftpmaster.TestPublishFTPMasterScript.test_script_is_happy_with_no_publications
lp.services.job.tests.test_runner.TestJobRunner.test_runJobHandleErrors_oops_generated_user_notify_fails

Changed in launchpad:
assignee: nobody → Francesco Banconi (frankban)
status: Triaged → In Progress
Revision history for this message
Launchpad QA Bot (lpqabot) wrote :

Fixed in stable r15318 (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15318) by a commit, but not testable.

tags: added: qa-untestable
Changed in launchpad:
status: In Progress → Fix Committed
William Grant (wgrant)
Changed in launchpad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.