Merge lp:~gmb/zope.testing/make-skips-work-bug-1019275 into lp:~launchpad/zope.testing/3.9.4-fork

[Graham Binns]

Add an addSkip() method to zope.testing's own version of TestResult. The lack of this method caused skips in python 2.7 to not be recorded in zope.testing-generated subunit streams.

This branch fixes bug 1019275 by adding an addSkip() method to zope.testing.testrunner.runner.TestResult, and by updating the OutputFormatter classes to support test_skip() as a means out outputting data about the skip where possible.

I've added tests to cover the changes; this should help us un-break parallel testing on Python 2.7.

[Jelmer Vernooij (jelmer)]

As discussed in person:
 * W00t!
 * TestLayer and MockOptions use old style classes
 * Needs empty line under class definitions per PEP8

review: Approve

[Launchpad QA Bot (lpqabot)]

The attempt to merge lp:~gmb/zope.testing/make-skips-work-bug-1019275 into lp:~launchpad/zope.testing/3.9.4-fork failed. Below is the output from the failed tests.

Creating directory '/home/tarmac/repos/zope.testing/3.9.4-fork/bin'.
Creating directory '/home/tarmac/repos/zope.testing/3.9.4-fork/parts'.
Creating directory '/home/tarmac/repos/zope.testing/3.9.4-fork/eggs'.
Creating directory '/home/tarmac/repos/zope.testing/3.9.4-fork/develop-eggs'.
Getting distribution for 'setuptools'.
Got setuptools 0.6c12dev-r88846.
Generated script '/home/tarmac/repos/zope.testing/3.9.4-fork/bin/buildout'.
Develop: '/home/tarmac/repos/zope.testing/3.9.4-fork/.'
Getting distribution for 'zc.recipe.testrunner'.
Got zc.recipe.testrunner 1.4.0.
Getting distribution for 'z3c.recipe.scripts>=1.0.0'.
Got z3c.recipe.scripts 1.0.1.
Getting distribution for 'zope.testrunner'.
Got zope.testrunner 4.0.4.
Getting distribution for 'zc.recipe.egg>=1.3.0'.
Got zc.recipe.egg 1.3.2.
Getting distribution for 'zope.interface'.
********************************************************************************
WARNING:

        An optional code optimization (C extension) could not be compiled.

        Optimizations for this package will not be available!
()
command 'gcc' failed with exit status 1
********************************************************************************
Got zope.interface 4.0.1.
Getting distribution for 'zope.exceptions'.
Got zope.exceptions 4.0.0.1.
Installing test.
Generated script '/home/tarmac/repos/zope.testing/3.9.4-fork/bin/test'.
Running zope.testing.testrunner.test_testresult.TestLayer tests:
  Set up zope.testing.testrunner.test_testresult.TestLayer in 0.000 seconds.
  Ran 1 tests with 0 failures and 0 errors in 0.000 seconds.
Running zope.testrunner.layer.UnitTests tests:
  Tear down zope.testing.testrunner.test_testresult.TestLayer in 0.000 seconds.
  Set up zope.testrunner.layer.UnitTests in 0.000 seconds.

Error in test test_SubunitOutputFormatter_adds_timestamp_to_skip_if_verbose (zope.testing.testrunner.test_testresult.TestOutputFormatters)
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 327, in run
    testMethod()
  File "/home/tarmac/repos/zope.testing/3.9.4-fork/src/zope/testing/testrunner/test_testresult.py", line 128, in test_SubunitOutputFormatter_adds_timestamp_to_skip_if_verbose
    options=MockOptions(verbose=3), stream=stream)
  File "/home/tarmac/repos/zope.testing/3.9.4-fork/src/zope/testing/testrunner/formatter.py", line 726, in __init__
    raise Exception("Requires subunit 0.0.5 or better")
Exception: Requires subunit 0.0.5 or better

Error in test test_SubunitOutputFormatter_outputs_skip (zope.testing.testrunner.test_testresult.TestOutputFormatters)
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 327, in run
    testMethod()
  File "/home/tarmac/repos/zope.testing/3.9.4-fork/src/zope/testing/testrunner/test_testresult.py", line 115, in test_SubunitOutputFormatter_outputs_skip
    options=MockOptions(), stream=stream)
  File "/home/tarmac/repos/zope.testing/3.9.4-fork/src/zope/testing/testrunner/formatter.py", line 726, in __init__
    raise Exce...

=== modified file 'setup.py'
--- setup.py	2012-06-21 16:14:54 +0000
+++ setup.py	2012-06-29 13:47:18 +0000
@@ -85,7 +85,7 @@
 
 setup(
     name='zope.testing',
-    version = '3.9.4-p15',
+    version = '3.9.4-p16',
     url='http://pypi.python.org/pypi/zope.testing',
     license='ZPL 2.1',
     description='Zope testing framework, including the testrunner script.',

=== modified file 'src/zope/testing/testrunner/formatter.py'
--- src/zope/testing/testrunner/formatter.py	2012-06-20 15:05:38 +0000
+++ src/zope/testing/testrunner/formatter.py	2012-06-29 13:47:18 +0000
@@ -285,6 +285,10 @@
 
         sys.stdout.flush()
 
+    def test_skip(self, test, reason):
+        """This is a no-op that exists for compatibility reasons."""
+        pass
+
     def test_success(self, test, seconds):
         """Report that a test was successful.
 
@@ -910,6 +914,11 @@
         """Clean up the output state after a collection of tests."""
         self._stream.flush()
 
+    def test_skip(self, test, reason):
+        if self._time_tests:
+            self._emit_timestamp()
+        self._subunit.addSkip(test, reason)
+
     def test_success(self, test, seconds):
         if self._time_tests:
             self._emit_timestamp()

=== modified file 'src/zope/testing/testrunner/runner.py'
--- src/zope/testing/testrunner/runner.py	2012-06-21 19:40:19 +0000
+++ src/zope/testing/testrunner/runner.py	2012-06-29 13:47:18 +0000
@@ -794,6 +794,10 @@
         self._threads = threading.enumerate()
         self._start_time = time.time()
 
+    def addSkip(self, test, reason):
+        unittest.TestResult.addSkip(self, test, reason)
+        self.options.output.test_skip(test, reason)
+
     def addSuccess(self, test):
         t = max(time.time() - self._start_time, 0.0)
         self.options.output.test_success(test, t)

=== added file 'src/zope/testing/testrunner/test_testresult.py'
--- src/zope/testing/testrunner/test_testresult.py	1970-01-01 00:00:00 +0000
+++ src/zope/testing/testrunner/test_testresult.py	2012-06-29 13:47:18 +0000
@@ -0,0 +1,135 @@
+##############################################################################
+#
+# Copyright (c) 2004-2012 Zope Foundation and Contributors.
+# All Rights Reserved.
+#
+# This software is subject to the provisions of the Zope Public License,
+# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
+# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
+# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
+# FOR A PARTICULAR PURPOSE.
+#
+##############################################################################
+"""Unit tests for the testrunner's subunit integration.
+"""
+
+__metaclass__ = type
+
+import unittest
+from StringIO import StringIO
+
+from zope.testing.testrunner.runner import TestResult
+from zope.testing.testrunner.formatter import (
+    OutputFormatter,
+    SubunitOutputFormatter,
+    )
+
+
+class MockOutput(StringIO):
+
+    def __init__(self):
+        self.test_skip_called = False
+
+    def test_skip(self, test, reason):
+        self.test_skip_called = True
+
+
+class MockOptions:
+
+    def __init__(self, output=None, verbose=None):
+        self.output = output or MockOutput()
+        self.verbose = verbose or 1
+
+
+class TestLayer:
+
+    def setUp(cls):
+        pass
+    setUp = classmethod(setUp)
+
+    def tearDown(cls):
+        pass
+    tearDown = classmethod(tearDown)
+
+    def testSetUp(cls):
+        pass
+    testSetUp = classmethod(testSetUp)
+
+    def testTearDown(cls):
+        pass
+    testTearDown = classmethod(testTearDown)
+
+
+class SampleTestCase(unittest.TestCase):
+
+    layer = TestLayer
+
+    def test_something(self):
+        return
+
+
+class TestTestResult(unittest.TestCase):
+    """Tests for zope.testing's own verison of TestResult."""
+
+    def setUp(self):
+        self.example_suite = (unittest.makeSuite(SampleTestCase))
+        options = MockOptions()
+        self.output = options.output
+        self.result = TestResult(
+            options, tests=[self.example_suite],
+            layer_name="zope.testing.testrunner.test_testresult.TestLayer")
+
+    def test_addSkip_calls_test_skip_on_output(self):
+        # TestResult.addSkip() calls the test_skip() method on whatever
+        # it has as an output formatter.
+        test = self.example_suite._tests[0]
+        self.result.addSkip(test, "Testing")
+        self.assertTrue(self.output.test_skip_called)
+
+    def test_addSkip_appends_skipped_test_to_skipped(self):
+        # TestResult.addSkip() appends the skipped test to the
+        # TestResult's skipped attribute.
+        test = self.example_suite._tests[0]
+        self.result.addSkip(test, "Testing")
+        self.assertIn((test, "Testing"), self.result.skipped)
+
+
+class TestOutputFormatters(unittest.TestCase):
+    """Tests for the handling of test_skip by different output formatters."""
+
+    def test_OutputFormatter_test_skip_exists(self):
+        # The test_skip() method of OutputFormatter exists, but it is a
+        # no-op; it prints nothing.
+        formatter = OutputFormatter(options=MockOptions())
+        # We just expect this not to fail.
+        formatter.test_skip(object(), "blah")
+
+    def test_SubunitOutputFormatter_outputs_skip(self):
+        # SubunitOutputFormatter.test_skip() outputs a skip: $testid
+        # line to the subunit stream.
+        example_suite = (unittest.makeSuite(SampleTestCase))
+        test = example_suite._tests[0]
+        stream = StringIO()
+        formatter = SubunitOutputFormatter(
+            options=MockOptions(), stream=stream)
+        formatter.test_skip(test, "Testing!")
+        stream.seek(0)
+        self.assertEqual(
+            "skip: %s [\nTesting!\n]\n" % test.id(), stream.read())
+
+    def test_SubunitOutputFormatter_adds_timestamp_to_skip_if_verbose(self):
+        # If the SubunitOutputFormatter is being verbose, it will emit a
+        # timestamp when it is emitting a skip.
+        example_suite = (unittest.makeSuite(SampleTestCase))
+        test = example_suite._tests[0]
+        stream = StringIO()
+        formatter = SubunitOutputFormatter(
+            options=MockOptions(verbose=3), stream=stream)
+        formatter.test_skip(test, "Testing!")
+        stream.seek(0)
+        self.assertTrue(
+            stream.readline().startswith("time:"),
+            "SubunitOutputFormatter did not emit a timestamp.")
+        self.assertEqual(
+            "skip: %s [\nTesting!\n]\n" % test.id(), stream.read())

=== modified file 'src/zope/testing/testrunner/tests.py'
--- src/zope/testing/testrunner/tests.py	2012-06-21 20:49:39 +0000
+++ src/zope/testing/testrunner/tests.py	2012-06-29 13:47:18 +0000
@@ -200,6 +200,8 @@
         doctest.DocTestSuite('zope.testing.testrunner.find'),
         unittest.defaultTestLoader.loadTestsFromName(
                 'zope.testing.testrunner.test_uniqueness'),
+        unittest.defaultTestLoader.loadTestsFromName(
+                'zope.testing.testrunner.test_testresult'),
         ]
 
     if sys.platform == 'win32':