Memcache warnings in test run output

As of Fri May 4, we started seeing memcache complaints in the stdout of test runs, like this:

WARNING:root:Memcache set failed for pt:testrunner_:lp/registry/templates/karmacontext-portlet-top-contributors.pt,11607:p:1,0:1,http_//127.0.0.1?None6fX5F0LldaQAQteC7XdOk6

You can see a lot more in this pastebin, from a parallel test run in the data center: http://pastebin.ubuntu.com/990503/ . See line 171 and following in particular.

Moreover, you can also see these in the non-parallel buildbot output. For instance, search for "WARNING:root:Memcache set failed for pt:testrunner_:" in https://lpbuildbot.canonical.com/builders/lucid_lp/builds/2073/steps/shell_6/logs/stdio and you'll find 92 instances.

In parallel tests, these can completely hose the subunit processing. However, the fix for bug 996729 will hopefully make that problem go away. The bigger problem--the one for this bug, and one not specific to parallel testing--is that the warnings themselves are concerning. It would be nice to know why we are getting these warnings, and either quiet them or do something about them, as appropriate.

According to buildbot, these warnings first started showing up in build https://lpbuildbot.canonical.com/builders/lucid_lp/builds/2037/steps/shell_6/logs/stdio . https://lpbuildbot.canonical.com/builders/lucid_lp/builds/2036/steps/shell_6/logs/stdio does not have them.

The following are the new checkins for build 2037, according to buildbot. As you can see, the likelihood is that these new warnings are self-inflicted by the squad working on parallel testing. The change to logs is particularly suspicious.

Changed by: Ian Booth <email address hidden>
Changed at: Thu 03 May 2012 23:12:09
Branch: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel
Revision: 15197

Changed files:

lib/lp/bugs/model/structuralsubscription.py file MODIFIED
lib/lp/services/database/stormexpr.py file MODIFIED
Comments:
[r=sinzui][no-qa] Moce some storm classes from
 lp.bug.model.structuralsubscription.py to lp.services.database.stormexpr.py
Properties:
Changed by: Benji York <email address hidden>
Changed at: Fri 04 May 2012 00:32:08
Branch: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel
Revision: 15198

Changed files:

lib/lp/testing/__init__.py file MODIFIED
Comments:
[r=jcsackett][bug=992692][no-qa] clear out any existing log handlers
 at the start of each test case so tests that add handlers don't
 effect those that depend on no handlers being registered.
Properties:
Changed by: Brad Crittenden <email address hidden>
Changed at: Fri 04 May 2012 00:46:08
Branch: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel
Revision: 15199

Changed files:

lib/lp/app/browser/tests/test_stringformatter.py file MODIFIED
lib/lp/testing/_login.py file MODIFIED
Comments:
[r=gary][bug=987499] Set LaunchBag.developer to False in
 lp.testing.logout().
Properties:
Changed by: Gary Poster <email address hidden>
Changed at: Fri 04 May 2012 00:50:09
Branch: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel
Revision: 15200

Changed files:

lib/lp/soyuz/doc/package-cache-script.txt file MODIFIED
Comments:
[r=bac][bug=994158][no-qa] Mark the db as dirty after a cronscript
 test so our automatic processes clean the db up for the next test.

I think this bug can be closed when the cause of the warnings is understood and the warnings themselves are no longer emitted (to stdout or, after bug 996729, to stderr when using subunit).