Launchpad itself

memcache rarely does not die on buildbot

Bug #729062 reported by Gary Poster
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Unassigned

Bug Description

Rarely (no clear indication of frequency, but wgrant remembers one other occurrence than the one this week, and mthaddon says it sounds vaguely familiar) buildbot has a memcache running after a testrun. This leads to test failures such as the following:

Failure in test testMemcachedWorking (canonical.testing.ftests.test_layers.BaseTestCase)
Traceback (most recent call last):
_StringException: Text attachment: traceback
------------
Traceback (most recent call last):
  File "/srv/buildbot/slaves/launchpad/lucid-db-devel/build/orig_sourcecode/eggs/testtools-0.9.8-py2.6.egg/testtools/runtest.py", line 169, in _run_user
    return fn(*args, **kwargs)
  File "/srv/buildbot/slaves/launchpad/lucid-db-devel/build/orig_sourcecode/eggs/testtools-0.9.8-py2.6.egg/testtools/testcase.py", line 499, in _run_test_method
    return self._get_test_method()()
  File "/srv/buildbot/slaves/launchpad/lucid-db-devel/build/lib/canonical/testing/ftests/test_layers.py", line 260, in testMemcachedWorking
    is_live, False, "memcached is live but should not be.")
  File "/srv/buildbot/slaves/launchpad/lucid-db-devel/build/orig_sourcecode/eggs/testtools-0.9.8-py2.6.egg/testtools/testcase.py", line 268, in assertEqual
    self.assertThat(observed, matcher)
  File "/srv/buildbot/slaves/launchpad/lucid-db-devel/build/orig_sourcecode/eggs/testtools-0.9.8-py2.6.egg/testtools/testcase.py", line 345, in assertThat
    % (matchee, matcher, mismatch.describe()))
AssertionError: Match failed. Matchee: "False"
Matcher: Annotate('memcached is live but should not be.', Equals(True))
Difference: True != False: memcached is live but should not be.

------------

The full set of failures are

   testMemcachedWorking (canonical.testing.ftests.test_layers.BaseTestCase)
   testMemcachedWorking (canonical.testing.ftests.test_layers.DatabaseTestCase)
   testMemcachedWorking (canonical.testing.ftests.test_layers.LibrarianTestCase)
   testMemcachedWorking (canonical.testing.ftests.test_layers.ZopelessTestCase)
   testMemcachedWorking (canonical.testing.ftests.test_layers.FunctionalTestCase)

Revision history for this message
Diogo Matsubara (matsubara) wrote :

On 2011-03-29 22:24 UTC we had an issue with the memcached registered in the IncidentLog, which is likely related to this bug.

After talking with William, he suggested that the test suite could be changed to kill the memcached process on startup.

Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 729062] Re: memcache rarely does not die on buildbot

On Fri, Apr 1, 2011 at 1:09 AM, Diogo Matsubara <email address hidden> wrote:
> On 2011-03-29 22:24 UTC we had an issue with the memcached registered in
> the IncidentLog, which is likely related to this bug.
>
> After talking with William, he suggested that the test suite could be
> changed to kill the memcached process on startup.

That will conflict with parallel testing and result in each parallel
worker killing each other.

Lets solve the root cause not workaround it: the cure is worse than
the disease otherwise.

Revision history for this message
Curtis Hovey (sinzui) wrote :

The db-devel build consistently fails because of this. though
    ./bin/test -vvc canonical.testing.ftests.test_layers
passes. The change that proceeded the fault was a lint fix.

I think these changes relate to the moving of code and tests, but ec2 and devel buildbot passed. I do not see anything damning in the diff between db-devel and devel. Maybe the import reordering of imports or the removal of the canonical.launchpad.scripts.__init__.py shim.

The issue may relate to bug 908799 which I saw in an ec2 run and was able to instrument playing a specific order of tests. Like this issue, The code was merged into db devel because ec2 and buildbot devel passed.

Revision history for this message
William Grant (wgrant) wrote :

Curtis, that was just due to an old memcached lying around on pilinut:

12:53:07 < hloeung> buildbot 30543 0.1 0.5 78348 19052 ? Sl Dec26 4:14 memcached -m 1 -l 127.0.0.1 -p 11242 -U 11242

It's been killed, so we can probably undisable the test.

Revision history for this message
Robert Collins (lifeless) wrote :

Agreed, undisable and see what shows up.

tags: added: build-infrastructure disabledtest
William Grant (wgrant)
Changed in launchpad:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.