Merge lp:~benji/testrepository/add-worker-id-tagging into lp:~testrepository/testrepository/trunk

[Benji York]

tag each test with the ID of the worker that executed the test

This branch adds worker-ID tagging by taking advantage of the new testtools functionality.

When each test is run it is tagged with a "worker-#" string where "#" is a number identifying the worker process, starting with 1.

[Jonathan Lange (jml)]

 * I can't seem to get this to actually generate output that includes the tags.

 * I get this failing test:

======================================================================
ERROR: testrepository.tests.commands.test_load.TestCommandLoad.test_load__wrap_result_inserts_worker_id_tag
----------------------------------------------------------------------
Traceback (most recent call last):
  File "testrepository/tests/commands/test_load.py", line 201, in test_load__wrap_result_inserts_worker_id_tag
    result.addSuccess(self)
  File "/home/jml/src/testtools/testtools/testresult/real.py", line 696, in addSuccess
    return self.decorated.addSuccess(test, details=details)
TypeError: addSuccess() got an unexpected keyword argument 'details'

Fix is probably to change:
    return Tagger(result, tags_to_add, tags_to_remove)

To:
    return Tagger(ExtendedToOriginalDecorator(result), tags_to_add, tags_to_remove)

[Jonathan Lange (jml)]

When I use subunit trunk, I get the tags in output. Yay. Still get failure though.

[Benji York (benji)]

On Wed, Apr 18, 2012 at 8:27 PM, Jonathan Lange <email address hidden> wrote:
> When I use subunit trunk, I get the tags in output. Yay. Still get failure though.

I had to use some trunks of testrepository dependencies. I will
identify which are important and report back here. I assume the failure
you get is the same as in your first comment.
--
Benji York

[Gary Poster (gary)]

I am seeing test tags leakage.

I ran the LP tests with these changes (and Jono's suggestion). It seemed to work. Looking deeper into the testr subunit result (in the .testrepository directory) after I set up worker tags, I saw something that concerned me. This should not include the "zope:layer" tag.

test: lib/lp/registry/stories/gpg-coc/xx-ubuntu-codeofconduct-signer.txt
time: 2012-04-19 01:15:58.192592Z
tags: worker-0 zope:layer
successful: lib/lp/registry/stories/gpg-coc/xx-ubuntu-codeofconduct-signer.txt [ multipart
]

The processing in testr/subunit/testtools is in fact adding it. Here's an excerpt from a raw run of the same test, run in the lptests container with ./bin/test --subunit -t stories/gpg. The test is at the end, but you can see a layer "test" earlier with the tag.

test: lp.testing.layers.PageTestLayer:setUp
tags: zope:layer
time: 2012-04-19 01:49:13.039859Z
successful: lp.testing.layers.PageTestLayer:setUp
tags: zope:layer:lp.testing.layers.PageTestLayer
test: lib/lp/registry/stories/gpg-coc/xx-gpg-coc.txt
successful: lib/lp/registry/stories/gpg-coc/xx-gpg-coc.txt
test: lib/lp/registry/stories/gpg-coc/xx-ubuntu-codeofconduct-signer.txt
successful: lib/lp/registry/stories/gpg-coc/xx-ubuntu-codeofconduct-signer.txt

No "zope:layer" tag is in there. It looks like something is keeping test tag state across tests.

I believe that the problem is not in this branch, but this branch exposes the feature, so it would be nice if the problem were resolved before the changes were merged.

[Jonathan Lange (jml)]

Yes, same failure in my second comment as in my first.

If I understand the "tag leaking" problem, it's that the PageTestLayer:setUp ought to have the zope:layer tag (and you guys are ok with that), but that the following tests ought not, (and yet, they do!). I'll have a poke around.

Just to be crystal clear, I don't think this problem is at all related to bug 664171. Do you?

[Jonathan Lange (jml)]

Tag leakage reported as bug 985613.

[Gary Poster (gary)]

@jml No, I don't think this problem is at all related to bug 664171.

Thank you!

[lp:~benji/testrepository/add-worker-id-tagging]

151. By Benji York on 2012-04-19

merge from trunk

152. By Benji York on 2012-04-19

fix test failure when using trunk of testtools

[Benji York (benji)]

I've fixed the test failure. This branch should now work using released versions of all testr dependencies plus the trunk of testtools.

[Jonathan Lange (jml)]

 * LoggingResult is deprecated and uses an old API. Contrary to the advice I gave you on IRC, it is entirely appropriate to wrap that with ExtendedToOriginalDecorator. Using addUnexpectedSuccess is fine though.

 * I added some tests to automate my integration testing (http://paste.ubuntu.com/938182/) to my local branch. test_worker_tags_in_subunit_output fails with my packaged subunit (0.0.7-1build1) and with lp:subunit r152 (0.0.7 release) but succeeds when I use subunit trunk. I'd kind of like to know why, but not enough to figure it out myself or block this landing.

review: Approve

=== modified file 'testrepository/commands/load.py'
--- testrepository/commands/load.py	2012-04-18 15:31:35 +0000
+++ testrepository/commands/load.py	2012-04-19 15:07:19 +0000
@@ -17,12 +17,19 @@
 import optparse
 
 import subunit
-from testtools import ConcurrentTestSuite, MultiTestResult
+from testtools import ConcurrentTestSuite, MultiTestResult, Tagger
 
 from testrepository.commands import Command
 from testrepository.repository import RepositoryNotFound
 
 
+def _wrap_result(result, thread_number):
+    worker_id = 'worker-%s' % thread_number
+    tags_to_add = set([worker_id])
+    tags_to_remove = set()
+    return Tagger(result, tags_to_add, tags_to_remove)
+
+
 class load(Command):
     """Load a subunit stream into a repository.
 
@@ -66,7 +73,7 @@
             streams = list(suite)[0]
             for stream in streams():
                 yield subunit.ProtocolTestCase(stream)
-        case = ConcurrentTestSuite(cases, make_tests)
+        case = ConcurrentTestSuite(cases, make_tests, _wrap_result)
         inserter = repo.get_inserter(partial=self.ui.options.partial)
         try:
             previous_run = repo.get_latest_run()

=== modified file 'testrepository/tests/commands/test_load.py'
--- testrepository/tests/commands/test_load.py	2011-11-03 18:25:11 +0000
+++ testrepository/tests/commands/test_load.py	2012-04-19 15:07:19 +0000
@@ -21,11 +21,13 @@
 import testtools
 from testtools.content import text_content
 from testtools.matchers import MatchesException
+from testtools.tests.helpers import LoggingResult
 
 from testrepository.commands import load
 from testrepository.ui.model import UI
 from testrepository.tests import ResourcedTestCase, Wildcard
 from testrepository.tests.test_repository import RecordingRepositoryFactory
+from testrepository.tests.repository.test_file import HomeDirTempDir
 from testrepository.repository import memory, RepositoryNotFound
 
 
@@ -189,6 +191,20 @@
             [('summary', True, 1, None, 2.0, None, [('id', 0, None)])],
             ui.outputs[1:])
 
+    def test_load__wrap_result_inserts_worker_id_tag(self):
+        # The load command uses a result wrapper that tags each test with the
+        # ID of the worker that executed the test.
+        log = []
+        result = load._wrap_result(LoggingResult(log), 99)
+        result.startTestRun()
+        result.startTest(self)
+        result.addUnexpectedSuccess(self)
+        result.stopTest(self)
+        result.stopTestRun()
+        # Even though no tag data was provided above, the test has been tagged
+        # with the worker ID.
+        self.assertIn(('tags', set(['worker-99']), set([])), log)
+
     def test_load_second_run(self):
         # If there's a previous run in the database, then show information
         # about the high level differences in the test run: how many more