Merge lp:~james-w/subunit/discovery into lp:~subunit/subunit/trunk

[James Westby]

Support test discovery.

Hi,

Here's a hacky first pass at implementing discovery in the way you suggested.

The biggest thing is that there is no way to pass a testsuite to TestProgram,
so I implemented some of that class inline to get the job done.

I tested this with testr

mkdir foo
touch foo/__init__.py
create foo/test_foo.py with
from testtools import TestCase

class foo(TestCase):
    def test_pass(self): pass
    def test_fail(self): self.fail("foo")

and .testr.conf with
[DEFAULT]
test_command=python -m subunit.run $IDLIST
test_id_list_default=--discover .

as I said on IRC the --discover . was a little unexpected to me, and I
don't think it's ideal that it is required to be that.

This is clearly quite hacky, so I'm looking for suggestions on cleaning it
up.

Thanks,

James

[Robert Collins (lifeless)]

As discussed, this is best done in testtools.run and subunit.run changed to layer on that; the testtools part has landed.

[James Westby (james-w)]

How does it look now?

Thanks,

James

[Robert Collins (lifeless)]

Very nice except that the help is now the testtools help, and it really should mention that it will be outputting subunit ;).

review: Needs Fixing

[lp:~james-w/subunit/discovery]

129. By James Westby on 2010-06-22

Modify the usage message from subunit.run.

[James Westby (james-w)]

On Mon, 14 Jun 2010 08:58:22 -0000, Robert Collins <email address hidden> wrote:
> Review: Needs Fixing
> Very nice except that the help is now the testtools help, and it really should mention that it will be outputting subunit ;).

Have a look at it now.

python -m subunit.run --asdf
option --asdf not recognized
Usage: run.py [options] [tests]

Options:
  -h, --help Show this message
  -v, --verbose Verbose output
  -q, --quiet Minimal output
  -f, --failfast Stop on first failure
  -c, --catch Catch control-C and display results
  -b, --buffer Buffer stdout and stderr during test runs

Examples:
  run.py test_module - run tests from test_module
  run.py module.TestClass - run tests from module.TestClass
  run.py module.Class.test_method - run specified test method

[tests] can be a list of any number of test modules, classes and test
methods.

Alternative Usage: run.py discover [options]

Options:
  -v, --verbose Verbose output
  -f, --failfast Stop on first failure
  -c, --catch Catch control-C and display results
  -b, --buffer Buffer stdout and stderr during test runs
  -s directory Directory to start discovery ('.' default)
  -p pattern Pattern to match test files ('test*.py' default)
  -t directory Top level directory of project (default to
                   start directory)

For test discovery all test modules must be importable from the top
level directory of the project.

The output will be in subunit format.

Thanks,

James

[Robert Collins (lifeless)]

I think I'd like to see something like this:

python -m subunit.run --asdf
option --asdf not recognized

Execute a python test suite with subunit reporting.

Usage: python -m subunit.run [options] [tests]

.. the stock stuff here.

And in testtools it should say

python -m testtools.run --asdf
option --asdf not recognized

Execute a python test suite.

Usage: testtools.run [options] [tests]

...

----

I note that the thing to run is being misdetected for you; I thought I
fixed that, perhaps you didn't grab testtools trunk - or my fix was
incomplete.

[James Westby (james-w)]

On Tue, 22 Jun 2010 19:10:49 -0000, Robert Collins <email address hidden> wrote:
> I think I'd like to see something like this:
>
> python -m subunit.run --asdf
> option --asdf not recognized
>
> Execute a python test suite with subunit reporting.
>
> Usage: python -m subunit.run [options] [tests]
>
> .. the stock stuff here.
>
> And in testtools it should say
>
> python -m testtools.run --asdf
> option --asdf not recognized
>
> Execute a python test suite.
>
> Usage: testtools.run [options] [tests]
>
> ...
>
> ----
>
> I note that the thing to run is being misdetected for you; I thought I
> fixed that, perhaps you didn't grab testtools trunk - or my fix was
> incomplete.

I wasn't using trunk.

Doing what you ask is possible, but means we move further from 2.7's
TestProgram, and don't pick up extra options in the usage output
that easily. That at least would go away once 2.7 was the minimum.

I'm happy to do that if you like.

Thanks,

James

[Robert Collins (lifeless)]

On Wed, Jun 23, 2010 at 7:34 AM, James Westby <email address hidden> wrote:
> I wasn't using trunk.
>
>
> Doing what you ask is possible, but means we move further from 2.7's
> TestProgram, and don't pick up extra options in the usage output
> that easily. That at least would go away once 2.7 was the minimum.
>
> I'm happy to do that if you like

We should file patches for changes we make that are relevant to the
reusability of TestProgram in python upstream; simple porting changes
aren't relevant because - well that obvious :).

So far, I checked and our changes didn't need upstream patches; we may
at this point need to send one in, and thats fine IMO.

We don't need to block on sending such a patch in, we should get
something we're happy with, then send in a patch.

-Rob

=== modified file 'NEWS'
--- NEWS	2010-06-11 01:35:58 +0000
+++ NEWS	2010-06-22 09:04:24 +0000
@@ -23,6 +23,13 @@
   mangled by the default line ending translation on that platform.
   (Robert Collins, Martin [gz], #579296)
 
+IMPROVEMENTS
+~~~~~~~~~~~~
+
+* subunit now supports test discovery by building on the testtools support for
+  it. You can take advantage of it with "python -m subunit.run discover [path]"
+  and see "python -m subunit.run discover --help" for more options.
+
 0.0.5
 -----
 

=== modified file 'python/subunit/run.py'
--- python/subunit/run.py	2010-01-15 04:16:50 +0000
+++ python/subunit/run.py	2010-06-22 09:04:24 +0000
@@ -23,6 +23,13 @@
 import sys
 
 from subunit import TestProtocolClient, get_default_formatter
+from testtools.run import (
+    BUFFEROUTPUT,
+    CATCHBREAK,
+    FAILFAST,
+    TestProgram,
+    USAGE_AS_MAIN,
+    )
 
 
 class SubunitTestRunner(object):
@@ -36,12 +43,27 @@
         return result
 
 
+class SubunitTestProgram(TestProgram):
+
+    USAGE = USAGE_AS_MAIN
+
+    def usageExit(self, msg=None):
+        if msg:
+            print msg
+        usage = {'progName': self.progName, 'catchbreak': '', 'failfast': '',
+                 'buffer': ''}
+        if self.failfast != False:
+            usage['failfast'] = FAILFAST
+        if self.catchbreak != False:
+            usage['catchbreak'] = CATCHBREAK
+        if self.buffer != False:
+            usage['buffer'] = BUFFEROUTPUT
+        print self.USAGE % usage
+        print "The output will be in subunit format.\n"
+        sys.exit(2)
+
+
 if __name__ == '__main__':
-    import optparse
-    from unittest import TestProgram
-    parser = optparse.OptionParser(__doc__)
-    args = parser.parse_args()[1]
     stream = get_default_formatter()
     runner = SubunitTestRunner(stream)
-    program = TestProgram(module=None, argv=[sys.argv[0]] + args,
-                          testRunner=runner)
+    SubunitTestProgram(module=None, argv=sys.argv, testRunner=runner)