Launchpad Buildbot Configuration

Bug #687951
Comment #3

Comment 3 for bug 687951

Revision history for this message

Gary Poster (gary) wrote on 2010-12-20:

Edited IRC log conversation with lifeless.

gary: lifeless, if you are still around, can you point me to the code that makes the launchpad_ftest_NNN (e.g., launchpad_ftest_10452) databases for tests? I've gone a-hunting and have not found it yet--or at least mot recognized it
lifeless: gary: its the PgTestSetup stuff, now DatabaseFixture IIRC
lifeless: gary: to find it, start with layers.py
lifeless: gary: look at DatabaseLayer - you can see the fixture class it uses, and track back from there
lifeless: there is an lp specific thing in the same module that subclasses the more generic one
lifeless: the cause of the leakage will be one of a couple of things
lifeless: (I think)
lifeless: a) We're shutting down test processes via a signal that we don't trap
lifeless: solution : add a handler for that that will cause a stack unwind; stack unwinds should pretty reliably clean up
lifeless: b) Something is preventing the drop db happening - e.g. an appserver test process is slow to shutdown, we're not waiting for it to really shutdown in layer cleanup, and dropdb is thus failing because the db is in use
lifeless: c) there is a remaining bug in zope.testing causing our layer teardowns to not consistently happen
lifeless: I don't have a strong feeling for probability
gary: a) solution: stack unwinding: e.g., just raising a normal exception?
lifeless: fixing the functionallayer to use zope.testing.functional or whatever it is which (now) has a teardown function, to make the layer teardownable, is probably all thats needed
lifeless: gary: yeah, like test.in does
gary: gotcha, -ish
lifeless: gary: it may be missing a signal that we can expect
lifeless: I filed a bug about stale librarian pids
lifeless: in buildbot
lifeless: similar set of scenarios
lifeless: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/annotate/head:/buildout-templates/bin/test.in#L75
lifeless: e.g. if we're also receiving HUP in the test environment
lifeless: then we need to add signal.signal(signal.SIGHUP, exit_with_atexit_handlers)

Edited IRC log conversation with lifeless.

gary: lifeless, if you are still around, can you point me to the code that makes the launchpad_ftest_NNN (e.g., launchpad_ftest_10452) databases for tests?  I've gone a-hunting and have not found it yet--or at least mot recognized it
lifeless: gary: its the PgTestSetup stuff, now DatabaseFixture IIRC
lifeless: gary: to find it, start with layers.py
lifeless: gary: look at DatabaseLayer - you can see the fixture class it uses, and track back from there
lifeless: there is an lp specific thing in the same module that subclasses the more generic one
lifeless: the cause of the leakage will be one of a couple of things
lifeless: (I think)
lifeless: a) We're shutting down test processes via a signal that we don't trap
lifeless:   solution : add a handler for that that will cause a stack unwind; stack unwinds should pretty reliably clean up
lifeless: b) Something is preventing the drop db happening - e.g. an appserver test process is slow to shutdown, we're not waiting for it to really shutdown in layer cleanup, and dropdb is thus failing because the db is in use
lifeless: c) there is a remaining bug in zope.testing causing our layer teardowns to not consistently happen
lifeless: I don't have a strong feeling for probability
gary: a) solution: stack unwinding: e.g., just raising a normal exception?
lifeless: fixing the functionallayer to use zope.testing.functional or whatever it is which (now) has a teardown function, to make the layer teardownable, is probably all thats needed
lifeless: gary: yeah, like test.in does
gary: gotcha, -ish
lifeless: gary: it may be missing a signal that we can expect
lifeless: I filed a bug about stale librarian pids
lifeless: in buildbot
lifeless: similar set of scenarios
lifeless: http://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel/annotate/head:/buildout-templates/bin/test.in#L75
lifeless: e.g. if we're also receiving HUP in the test environment
lifeless: then we need to add signal.signal(signal.SIGHUP, exit_with_atexit_handlers)