Merge lp:~bac/testrepository/bug-949950 into lp:~testrepository/testrepository/trunk

[Brad Crittenden]

As requested, a '--subunit' option was added to the run command to get results in subunit format.

The change to test_cli.py was required or 'make_results' was unhappy due to not having an 'options' attribute, so the tests were made more robust.

Apologies for some of the formatting changes. The trailing whitespace was automatically cleaned up by my emacs save hook (not a bad thing) and the others were me being OCD. Sorry for the somewhat bloated diff.

[Jonathan Lange (jml)]

Hey Brad,

Thanks for submitting this. First up, no worries about the cleanups -- I'm really grateful that you took the time to do them.

We had a bit of a chat about this on IRC. Here are the highlights:

 * --subunit here only shows timestamps and failing tests, rather than the full test run. This isn't the behaviour you want.

 * Probably the right thing to do is push the call to TestResultFilter that's in load.py down into the UI make_result() method. This allows the UI to have full control over which tests are shown in the output. Note that this will probably have a bunch of annoying test fallout. My hunch is that this will make progress display easier too.

 * I think that the default for --subunit should be as for the current output: only show failing tests. testr should then grow a new option for showing full results. --full-results is as good a spelling as any.

 * Both --subunit and --full-results should be applied to both 'testr run' and 'testr load'.

 * "Output collated results" is very parallel-run specific. Should probably say, "display results in subunit format" or similar.

 * I haven't seen vars(x).get(y) before. I'm guessing this is equivalent to getattr(x, y, None)?

Thanks,
jml

review: Disapprove

[Brad Crittenden (bac)]

Thanks for the review Jono. I've created a new branch and merge proposal that addresses your issues. Please mark this one 'rejected'.

Unmerged revisions

=== modified file 'testrepository/commands/run.py'
--- testrepository/commands/run.py	2010-12-19 09:17:29 +0000
+++ testrepository/commands/run.py	2012-04-17 14:20:23 +0000
@@ -1,11 +1,11 @@
 #
-# Copyright (c) 2010 Testrepository Contributors
-# 
+# Copyright (c) 2010-2012 Testrepository Contributors
+#
 # Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
 # license at the users choice. A copy of both licenses are available in the
 # project source as Apache-2.0 and BSD. You may not use this file except in
 # compliance with one of these two licences.
-# 
+#
 # Unless required by applicable law or agreed to in writing, software
 # distributed under these licenses is distributed on an "AS IS" BASIS, WITHOUT
 # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the
@@ -14,11 +14,7 @@
 
 """Run a projects tests and load them into testrepository."""
 
-import ConfigParser
-from cStringIO import StringIO
 import optparse
-import os.path
-import string
 
 from testtools import TestResult
 
@@ -39,7 +35,10 @@
         optparse.Option("--parallel", action="store_true",
             default=False, help="Run tests in parallel processes."),
         optparse.Option("--partial", action="store_true",
-            default=False, help="Only some tests will be run. Implied by --failing."),
+            default=False,
+            help="Only some tests will be run. Implied by --failing."),
+        optparse.Option("--subunit", action="store_true",
+            default=False, help="Output collated results."),
         ]
     args = [StringArgument('testargs', 0, None)]
     # Can be assigned to to inject a custom command factory.

=== modified file 'testrepository/tests/test_ui.py'
--- testrepository/tests/test_ui.py	2011-11-02 19:27:08 +0000
+++ testrepository/tests/test_ui.py	2012-04-17 14:20:23 +0000
@@ -1,11 +1,11 @@
 #
 # Copyright (c) 2009, 2010 Testrepository Contributors
-# 
+#
 # Licensed under either the Apache License, Version 2.0 or the BSD 3-clause
 # license at the users choice. A copy of both licenses are available in the
 # project source as Apache-2.0 and BSD. You may not use this file except in
 # compliance with one of these two licences.
-# 
+#
 # Unless required by applicable law or agreed to in writing, software
 # distributed under these licenses is distributed on an "AS IS" BASIS, WITHOUT
 # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the
@@ -17,12 +17,14 @@
 from cStringIO import StringIO
 import optparse
 import subprocess
+import subunit
 import sys
 
 from testtools.matchers import raises
 
 from testrepository import arguments, commands
 from testrepository.commands import load
+from testrepository.commands.run import run
 from testrepository.repository import memory
 from testrepository.ui import cli, decorator, model
 from testrepository.tests import ResourcedTestCase
@@ -119,7 +121,7 @@
         # output_table shows a table.
         ui = self.get_test_ui()
         ui.output_table([('col1', 'col2'), ('row1c1','row1c2')])
-        
+
     def test_output_tests(self):
         # output_tests can be called, and takes a list of tests to output.
         ui = self.get_test_ui()
@@ -228,7 +230,22 @@
         # make_result can take a previous run.
         ui = self.ui_factory()
         ui.set_command(commands.Command(ui))
-        result = ui.make_result(lambda: None, memory.Repository().get_failing())
+        result = ui.make_result(
+            lambda: None, memory.Repository().get_failing())
         result.startTestRun()
         result.stopTestRun()
         self.assertEqual(0, result.testsRun)
+
+
+class TestCLIUISpecific(ResourcedTestCase):
+
+    def test_run_subunit_option(self):
+        ui = cli_ui_factory(options=[('subunit', True)])
+        ui.set_command(run)
+        self.assertEqual(True, ui.options.subunit)
+
+    def test_check_result_output(self):
+        ui = cli_ui_factory(options=[('subunit', True)])
+        ui.set_command(run)
+        result = ui.make_result(lambda: None)
+        self.assertTrue(isinstance(result, subunit.TestProtocolClient))

=== modified file 'testrepository/tests/ui/test_cli.py'
--- testrepository/tests/ui/test_cli.py	2012-04-02 12:14:26 +0000
+++ testrepository/tests/ui/test_cli.py	2012-04-17 14:20:23 +0000
@@ -234,6 +234,7 @@
         if stream is None:
             stream = StringIO()
         ui = cli.UI([], None, stream, None)
+        ui.set_command(commands.Command(ui))
         return ui.make_result(lambda: None)
 
     def test_initial_stream(self):

=== modified file 'testrepository/ui/cli.py'
--- testrepository/ui/cli.py	2012-03-29 10:58:30 +0000
+++ testrepository/ui/cli.py	2012-04-17 14:20:23 +0000
@@ -17,6 +17,7 @@
 from optparse import OptionParser
 import os
 import signal
+import subunit
 import sys
 
 from testtools.compat import unicode_output_stream
@@ -71,7 +72,11 @@
         yield self._stdin
 
     def make_result(self, get_id, previous_run=None):
-        return CLITestResult(self, get_id, self._stdout, previous_run)
+        if vars(self.options).get('subunit'):
+            results = subunit.TestProtocolClient(self._stdout)
+        else:
+            results = CLITestResult(self, get_id, self._stdout, previous_run)
+        return results
 
     def output_error(self, error_tuple):
         self._stderr.write(str(error_tuple[1]) + '\n')