Merge lp:~duncf/pyjunitxml/cmdline-runner into lp:pyjunitxml

[Duncan Findlay]

Adds support for running unit tests under junitxml directly from the command line. Tests can be specified by name, or (if available) they can be discovered automatically.

This allows for existing unit tests to be run directly by a continuous integration system without needing to write custom wrapper scripts. It also allows for test discovery so that new unit tests can be run without needing to update a wrapper script.

To use, run "python -m junitxml", "python -m junitxml.main" or "python -c 'import junitxml.main;junitxml.main.main()'" (depending on Python version). Tests can be specified by name. If no tests are specified, discovery is used. Discovery is supported for Python >= 2.7 and >= 3.2 or with the unittest2 package installed. Output file can be specified with the -o option (defaults to junit.xml). Number of tests run and final results are also reported on stderr. -h/--help provides a summary of available options (help differs depending on whether discovery is possible).

New code is thoroughly unit tested, and has been tested on Python 2.4, 2.5, 2.6, 2.7, 3.1 and 3.2 with and without unittest2 installed.

[Robert Collins (lifeless)]

Thanks for this, sorry for being slack - RealLife was interfering (new baby). I should get to it soon now.

[Robert Collins (lifeless)]

Ok, I've reviewed this and it looks fine. Thanks for writing it. I'd like to check that your contribution is also under the LGPL-3 (which the rest of the package is).

If you can confirm that, I'll add appropriate headers to the files and merge it.

Cheers,
Rob

review: Approve

[lp:~duncf/pyjunitxml/cmdline-runner]

29. By Duncan Findlay on 2011-12-07

Add pyjunitxml script to avoid the tricky python-version-specific ways of starting junitxml.

[Duncan Findlay (duncf)]

Added a pending change I had lying around. Turns out when building using many versions of python, having to remember the different ways to start pyjunitxml is annoying, so I added a script to do it, similar to how unittest2 works.

[Duncan Findlay (duncf)]

On Dec 3, 2011, at 12:07 AM, Robert Collins wrote:
> I'd like to check that your contribution is also under the LGPL-3 (which the rest of the package is).
>
> If you can confirm that, I'll add appropriate headers to the files and merge it.

My contribution is under LGPL-3.

Duncan

[Robert Collins (lifeless)]

Merged and pushed. Thanks, and thanks for your patience while I got
accustomed to parenthood :)

=== modified file 'junitxml/__init__.py'
--- junitxml/__init__.py	2011-11-18 18:50:00 +0000
+++ junitxml/__init__.py	2011-12-07 01:45:28 +0000
@@ -9,8 +9,11 @@
 
 import datetime
 import re
-import time
-import unittest
+
+try:
+    import unittest2 as unittest
+except ImportError:
+    import unittest
 
 # same format as sys.version_info: "A tuple containing the five components of
 # the version number: major, minor, micro, releaselevel, and serial. All
@@ -89,7 +92,7 @@
 
 class JUnitXmlResult(unittest.TestResult):
     """A TestResult which outputs JUnit compatible XML."""
-    
+
     def __init__(self, stream):
         """Create a JUnitXmlResult.
 
@@ -219,3 +222,6 @@
         self._test_case_string(test)
         self._results.append('/>\n')
 
+if __name__ == '__main__':
+    import junitxml.main
+    junitxml.main.main()

=== added file 'junitxml/__main__.py'
--- junitxml/__main__.py	1970-01-01 00:00:00 +0000
+++ junitxml/__main__.py	2011-12-07 01:45:28 +0000
@@ -0,0 +1,12 @@
+"""Command line functionality for junitxml.
+
+:Author: Duncan Findlay <[e-mail]>
+"""
+import sys
+
+import junitxml.main
+
+if __name__ == '__main__':
+    if sys.argv[0].endswith('__main__.py'):
+        sys.argv[0] = 'python -m junitxml'
+    junitxml.main.main()

=== added file 'junitxml/main.py'
--- junitxml/main.py	1970-01-01 00:00:00 +0000
+++ junitxml/main.py	2011-12-07 01:45:28 +0000
@@ -0,0 +1,167 @@
+"""Command line functionality for junitxml.
+
+Runs specific tests or does automatic discovery with output in XML format.
+
+:Author: Duncan Findlay <[e-mail]>
+"""
+import optparse
+import sys
+
+# If we're using Python < 2.7, we want unittest2 if we can get it, otherwise
+# unittest will suffice.
+try:
+    import unittest2 as unittest
+except ImportError:
+    import unittest
+
+import junitxml
+import junitxml.runner
+
+ADDITIONAL_HELP_NO_DISCOVERY = """\
+
+<xml file> is the name of a file used for output.
+<tests> is a list of any number of test modules, classes and test
+methods. If no tests are specified, automatic discovery is performed.
+
+Example for executing specific tests:
+  %(prog_name)s -o junit.xml test_module
+  %(prog_name)s test_module.TestClass
+  %(prog_name)s test_module.TestClass.test_method
+"""
+
+ADDITIONAL_HELP_DISCOVERY = """\
+
+<xml file> is the name of a file used for output.
+[tests] can be a list of any number of test modules, classes and test
+methods. If no tests are specified, automatic discovery is performed.
+
+Example for executing specific tests:
+  %(prog_name)s -o junit.xml test_module
+  %(prog_name)s test_module.TestClass
+  %(prog_name)s test_module.TestClass.test_method
+
+Example for test discovery:
+  %(prog_name)s
+  %(prog_name)s -o junit.xml -s tests/ -p '*.py'
+  %(prog_name)s -t myproj/ -s myproj/tests/ -p '*.py'
+
+For test discovery all test modules must be importable from the top
+level directory of the project.
+
+It is an error to specify discovery options and specific tests.
+"""
+
+class XmlTestProgram(object):
+    """Command line program for running tests with XML output."""
+
+    loader = unittest.defaultTestLoader
+    runner_class = junitxml.runner.JUnitXmlTestRunner
+
+    def __init__(self, can_discover=None):
+        self.tests = None
+        self.output_filename = None
+
+        self._can_discover = can_discover
+        if self._can_discover is None:
+            self._can_discover = bool(hasattr(self.loader, 'discover'))
+
+        if self._can_discover:
+            self._usage = 'Usage: %prog [options] [tests]'
+            self._help = ADDITIONAL_HELP_DISCOVERY
+        else:
+            self._usage = 'Usage: %prog [options] <tests>'
+            self._help = ADDITIONAL_HELP_NO_DISCOVERY
+
+    def parse_args(self, argv=None):
+        """Parse command line arguments."""
+        parser = optparse.OptionParser(
+            usage=self._usage, add_help_option=False)
+
+        parser.add_option('-h', '--help', dest='help', action='store_true',
+                          help='Show option summary and exit')
+        parser.add_option(
+            '-o', '--output-file', dest='output',
+            help='Specify name of output XML file. (Default: %default)',
+            default='./junit.xml')
+
+        if self._can_discover:
+            discovery_group = optparse.OptionGroup(
+                parser, 'Discovery options',
+                'Used to control discovery (when no tests specified).')
+
+            discovery_group.add_option(
+                '-s', '--start-directory', dest='start',
+                default=None, help="Directory to start discovery "
+                "('.' default)")
+            discovery_group.add_option(
+                '-p', '--pattern', dest='pattern',  default=None,
+                help="Pattern to match tests ('test*.py' default)")
+            discovery_group.add_option(
+                '-t', '--top-level-directory', dest='top', default=None,
+                help='Top level directory of project (defaults to start '
+                'directory)')
+            parser.add_option_group(discovery_group)
+
+        if argv is None:
+            argv = sys.argv[1:]
+        options, args = parser.parse_args(argv)
+
+        if options.help:
+            parser.print_help()
+            sys.stdout.write(self._help % ({'prog_name': sys.argv[0]}))
+            sys.exit(1)
+
+        self.output_filename = options.output
+
+        if self._can_discover and args and \
+               (options.start or options.pattern or options.top):
+            parser.error(
+                'Cannot specify discovery options and specific tests.')
+        elif args:
+            self.tests = self._load_tests(args)
+        elif self._can_discover:
+            self.tests = self._do_discovery(options.start, options.pattern,
+                                            options.top)
+        else:
+            parser.error('Must specify tests to run.')
+
+    def _do_discovery(self, start_dir, pattern, top_level_dir):
+        assert self._can_discover
+
+        if start_dir is None:
+            start_dir = '.'
+        if pattern is None:
+            pattern = 'test*.py'
+        if top_level_dir is None:
+            top_level_dir = start_dir
+
+        return self.loader.discover(start_dir, pattern, top_level_dir)
+
+    def _load_tests(self, test_names):
+        return self.loader.loadTestsFromNames(test_names)
+
+    def run(self):
+        """Run the specified tests.
+
+        :Returns: TestResult object.
+        """
+        stream = open(self.output_filename, 'w')
+        try:
+            test_runner = self.runner_class(xml_stream=stream)
+            return test_runner.run(self.tests)
+        finally:
+            stream.close()
+
+
+def main(args=None, prog=None):
+    if args is None:
+        args = sys.argv[1:]
+    if prog is None:
+        prog = XmlTestProgram()
+
+    prog.parse_args(args)
+    result = prog.run()
+    sys.exit(int(not result.wasSuccessful()))
+
+if __name__ == '__main__':
+    main()

=== added file 'junitxml/runner.py'
--- junitxml/runner.py	1970-01-01 00:00:00 +0000
+++ junitxml/runner.py	2011-12-07 01:45:28 +0000
@@ -0,0 +1,71 @@
+"""Simple Test Runner for XML output.
+
+Writes XML output and reports test status to stderr.
+
+:Author: Duncan Findlay <[e-mail]>
+"""
+import sys
+import time
+
+import junitxml
+
+class JUnitXmlTestRunner(object):
+
+    """Simple Test Runner that writes XML output and reports status.
+
+    Provides high-level status suitable for command-line operation as well as
+    XML output.
+    """
+
+    resultclass = junitxml.JUnitXmlResult
+
+    def __init__(self, xml_stream, txt_stream=None, **kwargs):
+        if txt_stream is None:
+            txt_stream = sys.stderr
+        self._txt_stream = txt_stream
+        self._xml_stream = xml_stream
+
+    def _make_result(self):
+        return self.resultclass(self._xml_stream)
+
+    def run(self, test):
+        result = self._make_result()
+        result.startTestRun()
+
+        start_time = time.time()
+        test.run(result)
+        end_time = time.time()
+
+        result.stopTestRun()
+
+        self._write_summary(result, end_time - start_time)
+        return result
+
+    def _write_summary(self, result, time_elapsed):
+
+        plural = ''
+        if result.testsRun != 1:
+            plural = 's'
+
+        self._txt_stream.write('Ran %d test%s in %.3fs\n\n' %
+                               (result.testsRun, plural, time_elapsed))
+        test_info = []
+
+        for result_attr, desc in (
+            ('failures', 'failures'), ('errors', 'errors'),
+            ('skipped', 'skipped'), ('expectedFailures', 'expected failures'),
+            ('unexpectedSuccesses', 'unexpected successes')):
+
+            num = len(getattr(result, result_attr, []))
+            if num > 0:
+                test_info.append('%s=%s' % (desc, num))
+
+        test_info_str = ''
+        if test_info:
+            test_info_str = ' (%s)' % (', '.join(test_info),)
+
+        if result.wasSuccessful():
+            self._txt_stream.write('OK%s\n' % (test_info_str,))
+        else:
+            self._txt_stream.write('FAILED%s\n' % (test_info_str,))
+

=== modified file 'junitxml/tests/__init__.py'
--- junitxml/tests/__init__.py	2009-08-01 07:45:44 +0000
+++ junitxml/tests/__init__.py	2011-12-07 01:45:28 +0000
@@ -8,9 +8,17 @@
 
 from junitxml.tests import (
     test_junitxml,
+    test_main,
+    test_runner
     )
 
 def test_suite():
     return unittest.TestLoader().loadTestsFromNames([
         'junitxml.tests.test_junitxml',
+        'junitxml.tests.test_main',
+        'junitxml.tests.test_runner'
         ])
+
+if __name__ == '__main__':
+    suite = test_suite()
+    unittest.TextTestRunner(verbosity=2).run(suite)

=== modified file 'junitxml/tests/test_junitxml.py'
--- junitxml/tests/test_junitxml.py	2011-11-18 18:50:00 +0000
+++ junitxml/tests/test_junitxml.py	2011-12-07 01:45:28 +0000
@@ -12,9 +12,13 @@
 import datetime
 import re
 import sys
-import unittest
 import xml.dom.minidom
 
+try:
+    import unittest2 as unittest
+except ImportError:
+    import unittest
+
 import junitxml
 
 class TestImports(unittest.TestCase):

=== added file 'junitxml/tests/test_main.py'
--- junitxml/tests/test_main.py	1970-01-01 00:00:00 +0000
+++ junitxml/tests/test_main.py	2011-12-07 01:45:28 +0000
@@ -0,0 +1,198 @@
+"""Test "main" functionality for junitxml.
+
+:Author: Duncan Findlay <[e-mail]>
+"""
+import os
+import shutil
+import sys
+import tempfile
+import xml.dom.minidom
+try:
+    from cStringIO import StringIO
+except ImportError:
+    from io import StringIO
+
+try:
+    import unittest2 as unittest
+except ImportError:
+    import unittest
+
+import junitxml.main
+
+def _skip_if(condition):
+    """Decorator for skipping tests no matter what version of unittest."""
+    if condition:
+        def decorator(func):
+            if hasattr(unittest, 'skip'):
+                # If we don't have discovery, we probably don't skip, but we'll
+                # try anyways...
+                return unittest.skip('Discovery not supported.')(func)
+            else:
+                return None
+        return decorator
+    else:
+        # Condition is false, return the do-nothing decorator.
+        def decorator(func):
+            return func
+        return decorator
+
+
+class FakeLoader(object):
+    """Fake TestLoader to stub out test loading."""
+
+    def discover(self, start_dir, pattern=None, top_level_dir=None):
+        self._did_discovery = (start_dir, pattern, top_level_dir)
+        return unittest.TestSuite()
+
+    def loadTestsFromNames(self, names, module=None):
+        self._loaded_tests = (names, module)
+        return unittest.TestSuite()
+
+
+class RedirectedTestCase(unittest.TestCase):
+
+    """Redirects test output away from stdout/stderr."""
+
+    def setUp(self):
+        self._stderr = StringIO()
+        self._stdout = StringIO()
+        self._old_stderr = sys.stderr
+        self._old_stdout = sys.stdout
+        sys.stderr = self._stderr
+        sys.stdout = self._stdout
+
+    def tearDown(self):
+        sys.stderr = self._old_stderr
+        sys.stdout = self._old_stdout
+
+
+class TestArgs(RedirectedTestCase):
+
+    def _test_bad_args(self, args, can_discover=None):
+        # XmlTestPrograrm uses some magic to figure out whether it can do test
+        # discovery. We want to manually control that, sometimes.
+        try:
+            if can_discover is not None:
+                prog = junitxml.main.XmlTestProgram(can_discover=can_discover)
+                junitxml.main.main(args, prog=prog)
+            else:
+                junitxml.main.main(args)
+            self.fail('No exception thrown')
+        except SystemExit:
+            e = sys.exc_info()[1]
+            self.assertEqual(e.code, 2, self._stderr.getvalue())
+        except:
+            self.fail('No SystemExit exception thrown')
+
+    def test_bad_opts(self):
+        """Bad option combinations are rejected."""
+        # Mix of tests and discovery opts.
+        self._test_bad_args(['-s', '..', 'my_test'])
+        self._test_bad_args(['-p', 'foo*.py', 'my_test'])
+        self._test_bad_args(['-t', '..', 'my_test'])
+        self._test_bad_args(['--top', '..', 'my_test'])
+
+        # Incomplete options.
+        self._test_bad_args(['-o'])
+        self._test_bad_args(['-p'])
+        self._test_bad_args(['-s'])
+        self._test_bad_args(['-t'])
+
+    def test_help(self):
+        """Help is displayed with --help."""
+        try:
+            junitxml.main.main(['--help'])
+        except SystemExit:
+            e = sys.exc_info()[1]
+            self.assertEqual(e.code, 1)
+        except:
+            self.fail('No SystemExit exception thrown.')
+        self.assertTrue('Example ' in self._stdout.getvalue())
+
+    def test_no_discovery(self):
+        """Discovery options are rejected if discovery is not available."""
+        self._test_bad_args([], can_discover=False)
+        self._test_bad_args(['-s', '..'], can_discover=False)
+        self._test_bad_args(['-t', '..'], can_discover=False)
+        self._test_bad_args(['-p', '*.*'], can_discover=False)
+
+
+class TestLoad(RedirectedTestCase):
+
+    def setUp(self):
+        super(TestLoad, self).setUp()
+        self._tmpdir = tempfile.mkdtemp(prefix='unittest_junitxml_')
+        self._output_file = os.path.join(self._tmpdir, 'junit.xml')
+
+    def tearDown(self):
+        super(TestLoad, self).tearDown()
+        shutil.rmtree(self._tmpdir)
+
+    def test_loaded_tests(self):
+        """Verify named tests are properly loaded."""
+        prog = junitxml.main.XmlTestProgram()
+        prog.loader = FakeLoader()
+        prog.parse_args(['-o', self._output_file, 'my_test1', 'my_test2'])
+        result = prog.run()
+
+        document = xml.dom.minidom.parse(self._output_file)
+        self.assertEqual(document.documentElement.tagName, 'testsuite')
+        self.assertEqual(document.documentElement.getAttribute('tests'), '0')
+
+        self.assertEqual(result.wasSuccessful(), True)
+        self.assertTrue(hasattr(prog.loader, '_loaded_tests'))
+        self.assertEqual(prog.loader._loaded_tests,
+                         (['my_test1', 'my_test2'], None))
+
+    def test_discovery_all_args(self):
+        """Verify cmdline opts are used for discovery."""
+        prog = junitxml.main.XmlTestProgram(can_discover=True)
+        prog.loader = FakeLoader()
+        prog.parse_args(['-o', self._output_file, '-s', self._tmpdir,
+                         '--pattern', '*.py', '-t', '.'])
+        result = prog.run()
+
+        document = xml.dom.minidom.parse(self._output_file)
+        self.assertEqual(document.documentElement.tagName, 'testsuite')
+        self.assertEqual(document.documentElement.getAttribute('tests'), '0')
+
+        self.assertEqual(result.wasSuccessful(), True)
+        self.assertTrue(hasattr(prog.loader, '_did_discovery'))
+        self.assertEqual(prog.loader._did_discovery,
+                         (self._tmpdir, '*.py', '.'))
+
+    def test_discovery_top_dir(self):
+        """Verify top-level dir properly defaults to start directory."""
+        prog = junitxml.main.XmlTestProgram(can_discover=True)
+        prog.loader = FakeLoader()
+        prog.parse_args(['-o', self._output_file, '--start-dir', self._tmpdir])
+        result = prog.run()
+
+        document = xml.dom.minidom.parse(self._output_file)
+        self.assertEqual(document.documentElement.tagName, 'testsuite')
+        self.assertEqual(document.documentElement.getAttribute('tests'), '0')
+
+        self.assertEqual(result.wasSuccessful(), True)
+        self.assertTrue(hasattr(prog.loader, '_did_discovery'))
+        self.assertEqual(prog.loader._did_discovery,
+                         (self._tmpdir, 'test*.py', self._tmpdir))
+
+    def test_discovery_no_args(self):
+        """Verify good defaults are used for discovery when not specified."""
+        prog = junitxml.main.XmlTestProgram(can_discover=True)
+        prog.loader = FakeLoader()
+        prog.parse_args(['-o', self._output_file])
+        result = prog.run()
+
+        document = xml.dom.minidom.parse(self._output_file)
+        self.assertEqual(document.documentElement.tagName, 'testsuite')
+        self.assertEqual(document.documentElement.getAttribute('tests'), '0')
+
+        self.assertEqual(result.wasSuccessful(), True)
+        self.assertTrue(hasattr(prog.loader, '_did_discovery'))
+        self.assertEqual(prog.loader._did_discovery,
+                         ('.', 'test*.py', '.'))
+
+
+if __name__ == '__main__':
+    unittest.main()

=== added file 'junitxml/tests/test_runner.py'
--- junitxml/tests/test_runner.py	1970-01-01 00:00:00 +0000
+++ junitxml/tests/test_runner.py	2011-12-07 01:45:28 +0000
@@ -0,0 +1,119 @@
+"""Test XmlTestRunner functionality for junitxml.
+
+:Author: Duncan Findlay <[e-mail]>
+"""
+import xml.dom.minidom
+try:
+    from cStringIO import StringIO
+except ImportError:
+    from io import StringIO
+
+try:
+    import unittest2 as unittest
+except ImportError:
+    import unittest
+
+import junitxml.runner
+
+
+# Old versions of unittest don't have these "fancy" types of results.
+_FANCY_UNITTEST = (hasattr(unittest, 'skip') and
+                   hasattr(unittest, 'expectedFailure'))
+
+
+class TestXMLTestRunner(unittest.TestCase):
+
+    class DummyTestCase(unittest.TestCase):
+
+        def test_pass(self):
+            pass
+
+        def test_fail(self):
+            self.fail()
+
+        def test_error(self):
+            raise Exception()
+
+        if _FANCY_UNITTEST:
+
+            @unittest.skip('skipped')
+            def test_skip(self):
+                pass
+
+            @unittest.expectedFailure
+            def test_xfail(self):
+                self.fail('all is good')
+
+            @unittest.expectedFailure
+            def test_unexpected_success(self):
+                pass
+
+    def _run_runner(self, test_suite):
+        xml_out = StringIO()
+        console = StringIO()
+
+        runner = junitxml.runner.JUnitXmlTestRunner(
+            xml_stream=xml_out, txt_stream=console)
+        result = runner.run(test_suite)
+
+        return (result, xml_out, console)
+
+    def test_xml_output(self):
+        """Tests that runner properly gives XML output."""
+        test_suite = unittest.TestLoader().loadTestsFromTestCase(
+            self.DummyTestCase)
+
+        result, xml_out, console = self._run_runner(test_suite)
+
+        num_tests = test_suite.countTestCases()
+
+        # Make sure the XML output looks correct.
+        value = xml_out.getvalue()
+        document = xml.dom.minidom.parseString(value)
+
+        self.assertEqual(document.documentElement.tagName, 'testsuite')
+        self.assertEqual(document.documentElement.getAttribute('tests'),
+                         str(num_tests))
+
+    def test_console_output_fail(self):
+        """Tests that failure is reported properly on stderr."""
+        test_suite = unittest.TestLoader().loadTestsFromTestCase(
+            self.DummyTestCase)
+
+        result, xml_out, console = self._run_runner(test_suite)
+
+        num_tests = test_suite.countTestCases()
+
+        # Make sure the console output looks correct.
+        value = console.getvalue()
+        self.assertTrue('Ran %d tests in ' % (num_tests,) in value,
+                        'Output was:\n%s' % (value,))
+        self.assertTrue('FAILED (failures=1' in value,
+                        'Output was:\n%s' % (value,))
+        self.assertTrue('errors=1' in value,
+                        'Output was:\n%s' % (value,))
+
+        if _FANCY_UNITTEST:
+            self.assertTrue('expected failures=1' in value,
+                            'Output was:\n%s' % (value,))
+            self.assertTrue('skipped=1' in value,
+                            'Output was:\n%s' % (value,))
+            self.assertTrue('unexpected successes=1' in value,
+                            'Output was:\n%s' % (value,))
+
+    def test_console_output_ok(self):
+        """Tests that success is reported properly on stderr."""
+        test_suite = unittest.TestSuite()
+        test_suite.addTest(self.DummyTestCase('test_pass'))
+
+        result, xml_out, console = self._run_runner(test_suite)
+
+        value = console.getvalue()
+        self.assertTrue('Ran 1 test in ' in value,
+                        'Output was:\n%s' % (value,))
+        self.assertTrue('OK\n' in value,
+                        'Output was:\n%s' % (value,))
+
+
+if __name__ == '__main__':
+    unittest.main()

=== added file 'pyjunitxml'
--- pyjunitxml	1970-01-01 00:00:00 +0000
+++ pyjunitxml	2011-12-07 01:45:28 +0000
@@ -0,0 +1,7 @@
+#!/usr/bin/env python
+
+"""Script to run unit tests under junitxml."""
+
+if __name__ == '__main__':
+    import junitxml.main
+    junitxml.main.main()

=== modified file 'setup.py'
--- setup.py	2010-09-11 22:36:30 +0000
+++ setup.py	2011-12-07 01:45:28 +0000
@@ -9,5 +9,6 @@
       maintainer_email="[e-mail]",
       url="https://launchpad.net/pyjunitxml",
       packages=['junitxml', 'junitxml.tests'],
+      scripts=['pyjunitxml'],
       license="LGPL-3",
       )