Merge lp:~jml/testtools/deferred-support into lp:~testtools-committers/testtools/trunk

[Jonathan Lange]

This branch adds experimental support for tests that return Deferreds.

Most of the change in the diff is in the new RunTest objects and the tests I've added for those. It's my hope that the code is self-documenting, and that any questionable implementation decisions have explanatory comments. Let me know if it's otherwise.

The changes in the rest of testtools are just to make it a little more friendly to new RunTest objects. I've split off the RunTest code for handling user exceptions so I can re-use it in the async code, and I've made more methods on TestCase return things.

Since RunTest objects are actually a bit of a pain to use (see bug 657780 as an example), I haven't yet tried these new runners in anger. I might try them with some of the Launchpad tests and see what happens. When I do, I'll update the MP.

[Robert Collins (lifeless)]

Its awesome that you've done this.

I have one small suggestion; perhaps the twisted specific bits of this
should be in twisted.trial? There aren't, AFAIK, any third party
implementations of deferreds, yet.

Some code thoughts inline below:

On Mon, Oct 11, 2010 at 5:49 AM, Jonathan Lange <email address hidden> wrote:
> === modified file 'NEWS'
> --- NEWS        2010-09-18 02:10:58 +0000
> +++ NEWS        2010-10-10 16:49:39 +0000
> +class SynchronousDeferredRunTest(RunTest):
> +    """Runner for tests that return synchronous Deferreds."""
> +
> +    def _run_user(self, function, *args):
> +        d = defer.maybeDeferred(function, *args)
> +        def got_exception(failure):
> +            return self._got_user_exception(
> +                (failure.type, failure.value, failure.tb))
> +        d.addErrback(got_exception)
> +        result = extract_result(d)
> +        return result

There's no reason for got_exception to be an inner function here.

There was a bare except: I trimmed out, doh. Anyhow, except Exception:
would be better.

> +        # XXX: This can call addError on result multiple times. Not sure if
> +        # this is a good idea.

^ - definitely a bad idea, we avoid this in the normal sync code,
instead we trigger multiple exceptions which get accumulated via
details.

> +    def _run_user(self, function, *args):
> +        # XXX: I think this traps KeyboardInterrupt, and I think this is a bad
> +        # thing. Perhaps we should have a maybeDeferred-like thing that
> +        # re-raises KeyboardInterrupt. Or, we should have our own exception
> +        # handler that stops the test run in the case of KeyboardInterrupt. But
> +        # of course, the reactor installs a SIGINT handler anyway.

Squashing KeyboardInterrupt and SystemExit would be bad :). And if
we're catching one, we're probably catching the other.

It would be nice to have less of a parallel implementation feel, but
c'est la vie. I haven't read your tests in detail, but the conceptual
stuff seems fine.

It would be good, I think, to point voidspace at this patch as well,
because with this working, we're at point of being able to propose a
patch to core to let regular python run twisted test cases with less
of a complete-reimplementation, which is pretty cool. So I've CC'd him
:)

-Rob

[Jonathan Lange (jml)]

On Sun, Oct 10, 2010 at 9:49 PM, Robert Collins
<email address hidden> wrote:
> Its awesome that you've done this.
>
> I have one small suggestion; perhaps the twisted specific bits of this
> should be in twisted.trial? There aren't, AFAIK, any third party
> implementations of deferreds, yet.
>

The Twisted-specific bits (specifically _Spinner, DeferredNotFired,
extract_result, UnhandledErrorInDeferred, trap_unhandled_errors,
UncleanReactorError, TimeoutError) could well find a home in Twisted.
Some of them already have analogues.

However, I'm not in a rush to push those changes through:
  * _Spinner cannot go into Twisted as-is until Twisted stops using
setUpClass. It would have to become less strict. I don't want that for
my testtools-using code.
  * Putting extract_result into Twisted has been discussed in the
past, but it has been discouraged because it encourages newbies to
abuse it.
  * trap_unhandled_errors is kind of a hack. It only works here
because we're controlling the reactor and making assumptions about
test isolation.

> Some code thoughts inline below:
>
> On Mon, Oct 11, 2010 at 5:49 AM, Jonathan Lange <email address hidden> wrote:
>> === modified file 'NEWS'
>> --- NEWS        2010-09-18 02:10:58 +0000
>> +++ NEWS        2010-10-10 16:49:39 +0000
>> +class SynchronousDeferredRunTest(RunTest):
>> +    """Runner for tests that return synchronous Deferreds."""
>> +
>> +    def _run_user(self, function, *args):
>> +        d = defer.maybeDeferred(function, *args)
>> +        def got_exception(failure):
>> +            return self._got_user_exception(
>> +                (failure.type, failure.value, failure.tb))
>> +        d.addErrback(got_exception)
>> +        result = extract_result(d)
>> +        return result
>
> There's no reason for got_exception to be an inner function here.
>

You mean it should be a lambda or you mean it should be a method? It's
all the same to me, as long as it's not a part of the interface.

> There was a bare except: I trimmed out, doh. Anyhow, except Exception:
> would be better.
>

Changed. Note that KeyboardInterrupt will not be raised for Ctrl-C
while the reactor is running.

>> +        # XXX: This can call addError on result multiple times. Not sure if
>> +        # this is a good idea.
>
> ^ - definitely a bad idea, we avoid this in the normal sync code,
> instead we trigger multiple exceptions which get accumulated via
> details.
>

OK. Will fix this.

>> +    def _run_user(self, function, *args):
>> +        # XXX: I think this traps KeyboardInterrupt, and I think this is a bad
>> +        # thing. Perhaps we should have a maybeDeferred-like thing that
>> +        # re-raises KeyboardInterrupt. Or, we should have our own exception
>> +        # handler that stops the test run in the case of KeyboardInterrupt. But
>> +        # of course, the reactor installs a SIGINT handler anyway.
>
> Squashing KeyboardInterrupt and SystemExit would be bad :). And if
> we're catching one, we're probably catching the other.
>

We aren't squashing them, Twisted is. It installs a SIGINT handler
that overrides the default one. Without the default, Ctrl-C won't
raise KeyboardInterrupt.

> It would be nice to h...

[Robert Collins (lifeless)]

On Tue, Oct 12, 2010 at 1:46 AM, Jonathan Lange <email address hidden> wrote:
> However, I'm not in a rush to push those changes through:
>  * _Spinner cannot go into Twisted as-is until Twisted stops using
> setUpClass. It would have to become less strict. I don't want that for
> my testtools-using code.
>  * Putting extract_result into Twisted has been discussed in the
> past, but it has been discouraged because it encourages newbies to
> abuse it.
>  * trap_unhandled_errors is kind of a hack. It only works here
> because we're controlling the reactor and making assumptions about
> test isolation.

^ So, its up to you :).
>>> +    def _run_user(self, function, *args):
>>> +        d = defer.maybeDeferred(function, *args)
>>> +        def got_exception(failure):
>>> +            return self._got_user_exception(
>>> +                (failure.type, failure.value, failure.tb))
>>> +        d.addErrback(got_exception)
>>> +        result = extract_result(d)
>>> +        return result
>>
>> There's no reason for got_exception to be an inner function here.
>>
>
> You mean it should be a lambda or you mean it should be a method? It's
> all the same to me, as long as it's not a part of the interface.

def _got_errback(failure):
   ...

> I'd be very reluctant to put this code into the standard library as
> is. It's experimental, and the Twisted community frowns strongly on
> anything that turns async code to sync. However, if you just mean the
> RunTest mechanism, I'd be less reluctant, but still think it needs
> more usage out in the wild before standardizing. (e.g. there's no
> convenient syntax for using a different RunTest object)

What I mean is that once this is working we're in a position to have
an extensible core appropriate for trial to build on.

Not that its done :)

-Rob

[Robert Collins (lifeless)]

If this is ready to land please do so.

[Jonathan Lange (jml)]

On Thu, Oct 14, 2010 at 11:15 PM, Robert Collins
<email address hidden> wrote:
> If this is ready to land please do so.

Not at all. Still need to fix the multiple errors & SIGINT handling.
It's my next hacking task.

jml

[Jonathan Lange (jml)]

Now working in substance, but still a lot of things to do before it can actually be used in anger.

testtools/_spinner.py:219:
  XXX: Not tested. Not sure that the cost of testing this reliably
  outweighs the benefits.

testtools/_spinner.py:222:
  XXX: Also, we probably need to restore the threadpool the second
  time we run.

testtools/_spinner.py:280:
  XXX: It might be a better idea to either install custom signal
  handlers or to override the methods that are Twisted's signal
  handlers.

testtools/deferredruntest.py:28:
  TODO: Need a helper to replace Trial's assertFailure.

testtools/deferredruntest.py:30:
  TODO: Need a conversion guide for flushLoggedErrors

testtools/deferredruntest.py:59:
  TODO: 0.005s is probably too small a timeout for a default.

testtools/deferredruntest.py:60:
  TODO: docstring

testtools/deferredruntest.py:138:
  XXX: Right now, TimeoutErrors are re-raised, causing the test
  runner to crash. We probably just want to record them like test
  errors.

testtools/deferredruntest.py:149:
  TODO: Actually, rather than raising this with a special error,
  we could add a traceback for each unhandled Deferred, or
  something like that. Would be way more helpful than just a list
  of the reprs of the failures.

[Robert Collins (lifeless)]

You appear to have some conflicts in NEWS. Are there any particular incremental changes you want re-eyeballed?

[lp:~jml/testtools/deferred-support]

156. By Jonathan Lange on 2010-10-17

Merge trunk, get recent changes.

157. By Jonathan Lange on 2010-10-17

Merge branch that provides syntax for specifying test runner.

158. By Jonathan Lange on 2010-10-17

Don't use extract result, instead use custom test runner.

=== modified file 'MANUAL'
--- MANUAL	2010-09-18 02:10:58 +0000
+++ MANUAL	2010-10-17 21:32:44 +0000
@@ -11,11 +11,12 @@
 Extensions to TestCase
 ----------------------
 
-Controlling test execution
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+Custom exception handling
+~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Testtools supports two ways to control how tests are executed. The simplest
-is to add a new exception to self.exception_handlers::
+testtools provides a way to control how test exceptions are handled.  To do
+this, add a new exception to self.exception_handlers on a TestCase.  For
+example::
 
     >>> self.exception_handlers.insert(-1, (ExceptionClass, handler)).
 
@@ -23,12 +24,36 @@
 ExceptionClass, handler will be called with the test case, test result and the
 raised exception.
 
-Secondly, by overriding __init__ to pass in runTest=RunTestFactory the whole
-execution of the test can be altered. The default is testtools.runtest.RunTest
-and calls  case._run_setup, case._run_test_method and finally
-case._run_teardown. Other methods to control what RunTest is used may be
-added in future.
-
+Controlling test execution
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to control more than just how exceptions are raised, you can
+provide a custom `RunTest` to a TestCase.  The `RunTest` object can change
+everything about how the test executes.
+
+To work with `testtools.TestCase`, a `RunTest` must have a factory that takes
+a test and an optional list of exception handlers.  Instances returned by the
+factory must have a `run()` method that takes an optional `TestResult` object.
+
+The default is `testtools.runtest.RunTest` and calls 'setUp', the test method
+and 'tearDown' in the normal, vanilla way that Python's standard unittest
+does.
+
+To specify a `RunTest` for all the tests in a `TestCase` class, do something
+like this::
+
+  class SomeTests(TestCase):
+      run_tests_with = CustomRunTestFactory
+
+To specify a `RunTest` for a specific test in a `TestCase` class, do::
+
+  class SomeTests(TestCase):
+      @run_test_with(CustomRunTestFactory, extra_arg=42, foo='whatever')
+      def test_something(self):
+          pass
+
+In addition, either of these can be overridden by passing a factory in to the
+`TestCase` constructor with the optional 'runTest' argument.
 
 TestCase.addCleanup
 ~~~~~~~~~~~~~~~~~~~
@@ -249,3 +274,18 @@
 For more information see the Python 2.7 unittest documentation, or::
 
     python -m testtools.run --help
+
+
+Twisted support
+---------------
+
+Support for running Twisted tests is very experimental right now.  You
+shouldn't really do it.  However, if you are going to, here are some tips for
+converting your Trial tests into testtools tests.
+
+ * Use the AsynchronousDeferredRunTest runner
+ * Make sure to upcall to setUp and tearDown
+ * Don't use setUpClass or tearDownClass
+ * Don't expect setting .todo, .timeout or .skip attributes to do anything
+ * flushLoggedErrors is not there for you.  Sorry.
+ * assertFailure is not there for you.  Even more sorry.

=== modified file 'NEWS'
--- NEWS	2010-10-17 12:54:44 +0000
+++ NEWS	2010-10-17 21:32:44 +0000
@@ -4,6 +4,15 @@
 NEXT
 ~~~~
 
+* Experimental support for running tests that return Deferreds.
+  (Jonathan Lange)
+
+* Provide a per-test decoractor, run_test_with, to specify which RunTest
+  object to use for a given test.  (Jonathan Lange, #657780)
+
+* Fix the runTest parameter of TestCase to actually work, rather than raising
+  a TypeError.  (Jonathan Lange, #657760)
+
 
 0.9.7
 ~~~~~

=== modified file 'testtools/__init__.py'
--- testtools/__init__.py	2010-10-17 12:55:59 +0000
+++ testtools/__init__.py	2010-10-17 21:32:44 +0000
@@ -11,6 +11,7 @@
     'MultipleExceptions',
     'MultiTestResult',
     'PlaceHolder',
+    'run_test_with',
     'TestCase',
     'TestResult',
     'TextTestResult',
@@ -33,6 +34,7 @@
     PlaceHolder,
     TestCase,
     clone_test_with_new_id,
+    run_test_with,
     skip,
     skipIf,
     skipUnless,

=== added file 'testtools/_spinner.py'
--- testtools/_spinner.py	1970-01-01 00:00:00 +0000
+++ testtools/_spinner.py	2010-10-17 21:32:44 +0000
@@ -0,0 +1,280 @@
+# Copyright (c) 2010 Jonathan M. Lange. See LICENSE for details.
+
+"""Evil reactor-spinning logic for running Twisted tests.
+
+This code is highly experimental, liable to change and not to be trusted.  If
+you couldn't write this yourself, you should not be using it.
+"""
+
+__all__ = [
+    'DeferredNotFired',
+    'extract_result',
+    'NoResultError',
+    'not_reentrant',
+    'ReentryError',
+    'Spinner',
+    'StaleJunkError',
+    'TimeoutError',
+    'trap_unhandled_errors',
+    ]
+
+import signal
+
+from twisted.internet import defer
+from twisted.internet.interfaces import IReactorThreads
+from twisted.python.failure import Failure
+from twisted.python.util import mergeFunctionMetadata
+
+
+class ReentryError(Exception):
+    """Raised when we try to re-enter a function that forbids it."""
+
+    def __init__(self, function):
+        super(ReentryError, self).__init__(
+            "%r in not re-entrant but was called within a call to itself."
+            % (function,))
+
+
+def not_reentrant(function, _calls={}):
+    """Decorates a function as not being re-entrant.
+
+    The decorated function will raise an error if called from within itself.
+    """
+    def decorated(*args, **kwargs):
+        if _calls.get(function, False):
+            raise ReentryError(function)
+        _calls[function] = True
+        try:
+            return function(*args, **kwargs)
+        finally:
+            _calls[function] = False
+    return mergeFunctionMetadata(function, decorated)
+
+
+class DeferredNotFired(Exception):
+    """Raised when we extract a result from a Deferred that's not fired yet."""
+
+
+def extract_result(deferred):
+    """Extract the result from a fired deferred.
+
+    It can happen that you have an API that returns Deferreds for
+    compatibility with Twisted code, but is in fact synchronous, i.e. the
+    Deferreds it returns have always fired by the time it returns.  In this
+    case, you can use this function to convert the result back into the usual
+    form for a synchronous API, i.e. the result itself or a raised exception.
+
+    It would be very bad form to use this as some way of checking if a
+    Deferred has fired.
+    """
+    failures = []
+    successes = []
+    deferred.addCallbacks(successes.append, failures.append)
+    if len(failures) == 1:
+        failures[0].raiseException()
+    elif len(successes) == 1:
+        return successes[0]
+    else:
+        raise DeferredNotFired("%r has not fired yet." % (deferred,))
+
+
+def trap_unhandled_errors(function, *args, **kwargs):
+    """Run a function, trapping any unhandled errors in Deferreds.
+
+    Assumes that 'function' will have handled any errors in Deferreds by the
+    time it is complete.  This is almost never true of any Twisted code, since
+    you can never tell when someone has added an errback to a Deferred.
+
+    If 'function' raises, then don't bother doing any unhandled error
+    jiggery-pokery, since something horrible has probably happened anyway.
+
+    :return: A tuple of '(result, error)', where 'result' is the value returned
+        by 'function' and 'error' is a list of `defer.DebugInfo` objects that
+        have unhandled errors in Deferreds.
+    """
+    real_DebugInfo = defer.DebugInfo
+    debug_infos = []
+    def DebugInfo():
+        info = real_DebugInfo()
+        debug_infos.append(info)
+        return info
+    defer.DebugInfo = DebugInfo
+    try:
+        result = function(*args, **kwargs)
+    finally:
+        defer.DebugInfo = real_DebugInfo
+    errors = []
+    for info in debug_infos:
+        if info.failResult is not None:
+            errors.append(info)
+            # Disable the destructor that logs to error. We are already
+            # catching the error here.
+            info.__del__ = lambda: None
+    return result, errors
+
+
+class TimeoutError(Exception):
+    """Raised when run_in_reactor takes too long to run a function."""
+
+    def __init__(self, function, timeout):
+        super(TimeoutError, self).__init__(
+            "%r took longer than %s seconds" % (function, timeout))
+
+
+class NoResultError(Exception):
+    """Raised when the reactor has stopped but we don't have any result."""
+
+    def __init__(self):
+        super(NoResultError, self).__init__(
+            "Tried to get test's result from Deferred when no result is "
+            "available.  Probably means we received SIGINT or similar.")
+
+
+class StaleJunkError(Exception):
+    """Raised when there's junk in the spinner from a previous run."""
+
+    def __init__(self, junk):
+        super(StaleJunkError, self).__init__(
+            "There was junk in the spinner from a previous run. "
+            "Use clear_junk() to clear it out: %r" % (junk,))
+
+
+class Spinner(object):
+    """Spin the reactor until a function is done.
+
+    This class emulates the behaviour of twisted.trial in that it grotesquely
+    and horribly spins the Twisted reactor while a function is running, and
+    then kills the reactor when that function is complete and all the
+    callbacks in its chains are done.
+    """
+
+    _UNSET = object()
+
+    # Signals that we save and restore for each spin.
+    _PRESERVED_SIGNALS = [
+        signal.SIGINT,
+        signal.SIGTERM,
+        signal.SIGCHLD,
+        ]
+
+    def __init__(self, reactor):
+        self._reactor = reactor
+        self._timeout_call = None
+        self._success = self._UNSET
+        self._failure = self._UNSET
+        self._saved_signals = []
+        self._junk = []
+
+    def _cancel_timeout(self):
+        if self._timeout_call:
+            self._timeout_call.cancel()
+
+    def _get_result(self):
+        if self._failure is not self._UNSET:
+            self._failure.raiseException()
+        if self._success is not self._UNSET:
+            return self._success
+        raise NoResultError()
+
+    def _got_failure(self, result):
+        self._cancel_timeout()
+        self._failure = result
+
+    def _got_success(self, result):
+        self._cancel_timeout()
+        self._success = result
+
+    def _stop_reactor(self, ignored=None):
+        """Stop the reactor!"""
+        self._reactor.crash()
+
+    def _timed_out(self, function, timeout):
+        e = TimeoutError(function, timeout)
+        self._failure = Failure(e)
+        self._stop_reactor()
+
+    def _clean(self):
+        """Clean up any junk in the reactor."""
+        junk = []
+        for delayed_call in self._reactor.getDelayedCalls():
+            delayed_call.cancel()
+            junk.append(delayed_call)
+        for selectable in self._reactor.removeAll():
+            # Twisted sends a 'KILL' signal to selectables that provide
+            # IProcessTransport.  Since only _dumbwin32proc processes do this,
+            # we aren't going to bother.
+            junk.append(selectable)
+        if IReactorThreads.providedBy(self._reactor):
+            self._reactor.suggestThreadPoolSize(0)
+            if self._reactor.threadpool is not None:
+                self._reactor._stopThreadPool()
+        self._junk.extend(junk)
+        return junk
+
+    def clear_junk(self):
+        """Clear out our recorded junk.
+
+        :return: Whatever junk was there before.
+        """
+        junk = self._junk
+        self._junk = []
+        return junk
+
+    def get_junk(self):
+        """Return any junk that has been found on the reactor."""
+        return self._junk
+
+    def _save_signals(self):
+        self._saved_signals = [
+            (sig, signal.getsignal(sig)) for sig in self._PRESERVED_SIGNALS]
+
+    def _restore_signals(self):
+        for sig, hdlr in self._saved_signals:
+            signal.signal(sig, hdlr)
+        self._saved_signals = []
+
+    @not_reentrant
+    def run(self, timeout, function, *args, **kwargs):
+        """Run 'function' in a reactor.
+
+        If 'function' returns a Deferred, the reactor will keep spinning until
+        the Deferred fires and its chain completes or until the timeout is
+        reached -- whichever comes first.
+
+        :raise TimeoutError: If 'timeout' is reached before the `Deferred`
+            returned by 'function' has completed its callback chain.
+        :raise NoResultError: If the reactor is somehow interrupted before
+            the `Deferred` returned by 'function' has completed its callback
+            chain.
+        :raise StaleJunkError: If there's junk in the spinner from a previous
+            run.
+        :return: Whatever is at the end of the function's callback chain.  If
+            it's an error, then raise that.
+        """
+        junk = self.get_junk()
+        if junk:
+            raise StaleJunkError(junk)
+        self._save_signals()
+        self._timeout_call = self._reactor.callLater(
+            timeout, self._timed_out, function, timeout)
+        # Calling 'stop' on the reactor will make it impossible to re-start
+        # the reactor.  Since the default signal handlers for TERM, BREAK and
+        # INT all call reactor.stop(), we'll patch it over with crash.
+        # XXX: It might be a better idea to either install custom signal
+        # handlers or to override the methods that are Twisted's signal
+        # handlers.
+        stop, self._reactor.stop = self._reactor.stop, self._reactor.crash
+        def run_function():
+            d = defer.maybeDeferred(function, *args, **kwargs)
+            d.addCallbacks(self._got_success, self._got_failure)
+            d.addBoth(self._stop_reactor)
+        try:
+            self._reactor.callWhenRunning(run_function)
+            self._reactor.run()
+        finally:
+            self._reactor.stop = stop
+            self._restore_signals()
+        try:
+            return self._get_result()
+        finally:
+            self._clean()

=== added file 'testtools/deferredruntest.py'
--- testtools/deferredruntest.py	1970-01-01 00:00:00 +0000
+++ testtools/deferredruntest.py	2010-10-17 21:32:44 +0000
@@ -0,0 +1,234 @@
+# Copyright (c) 2010 Jonathan M. Lange. See LICENSE for details.
+
+"""Individual test case execution for tests that return Deferreds.
+
+This module is highly experimental and is liable to change in ways that cause
+subtle failures in tests.  Use at your own peril.
+"""
+
+__all__ = [
+    'assert_fails_with',
+    'AsynchronousDeferredRunTest',
+    'SynchronousDeferredRunTest',
+    ]
+
+import sys
+
+from testtools.runtest import RunTest
+from testtools._spinner import (
+    extract_result,
+    NoResultError,
+    Spinner,
+    TimeoutError,
+    trap_unhandled_errors,
+    )
+
+from twisted.internet import defer
+
+
+# TODO: Need a conversion guide for flushLoggedErrors
+
+class SynchronousDeferredRunTest(RunTest):
+    """Runner for tests that return synchronous Deferreds."""
+
+    def _run_user(self, function, *args):
+        d = defer.maybeDeferred(function, *args)
+        def got_exception(failure):
+            return self._got_user_exception(
+                (failure.type, failure.value, failure.tb))
+        d.addErrback(got_exception)
+        result = extract_result(d)
+        return result
+
+
+class AsynchronousDeferredRunTest(RunTest):
+    """Runner for tests that return Deferreds that fire asynchronously.
+
+    That is, this test runner assumes that the Deferreds will only fire if the
+    reactor is left to spin for a while.
+
+    Do not rely too heavily on the nuances of the behaviour of this class.
+    What it does to the reactor is black magic, and if we can find nicer ways
+    of doing it we will gladly break backwards compatibility.
+
+    This is highly experimental code.  Use at your own risk.
+    """
+
+    def __init__(self, case, handlers=None, reactor=None, timeout=0.005):
+        """Construct an `AsynchronousDeferredRunTest`.
+
+        :param case: The `testtools.TestCase` to run.
+        :param handlers: A list of exception handlers (ExceptionType, handler)
+            where 'handler' is a callable that takes a `TestCase`, a
+            `TestResult` and the exception raised.
+        :param reactor: The Twisted reactor to use.  If not given, we use the
+            default reactor.
+        :param timeout: The maximum time allowed for running a test.  The
+            default is 0.005s.
+        """
+        super(AsynchronousDeferredRunTest, self).__init__(case, handlers)
+        if reactor is None:
+            from twisted.internet import reactor
+        self._reactor = reactor
+        self._timeout = timeout
+
+    @classmethod
+    def make_factory(cls, reactor, timeout):
+        return lambda case, handlers=None: AsynchronousDeferredRunTest(
+            case, handlers, reactor, timeout)
+
+    @defer.inlineCallbacks
+    def _run_cleanups(self):
+        """Run the cleanups on the test case.
+
+        We expect that the cleanups on the test case can also return
+        asynchronous Deferreds.  As such, we take the responsibility for
+        running the cleanups, rather than letting TestCase do it.
+        """
+        while self.case._cleanups:
+            f, args, kwargs = self.case._cleanups.pop()
+            try:
+                yield defer.maybeDeferred(f, *args, **kwargs)
+            except Exception:
+                exc_info = sys.exc_info()
+                self.case._report_traceback(exc_info)
+                last_exception = exc_info[1]
+        defer.returnValue(last_exception)
+
+    def _run_deferred(self):
+        """Run the test, assuming everything in it is Deferred-returning.
+
+        This should return a Deferred that fires with True if the test was
+        successful and False if the test was not successful.  It should *not*
+        call addSuccess on the result, because there's reactor clean up that
+        we needs to be done afterwards.
+        """
+        fails = []
+
+        def fail_if_exception_caught(exception_caught):
+            if self.exception_caught == exception_caught:
+                fails.append(None)
+
+        def clean_up(ignored=None):
+            """Run the cleanups."""
+            d = self._run_cleanups()
+            def clean_up_done(result):
+                if result is not None:
+                    self._exceptions.append(result)
+                    fails.append(None)
+            return d.addCallback(clean_up_done)
+
+        def set_up_done(exception_caught):
+            """Set up is done, either clean up or run the test."""
+            if self.exception_caught == exception_caught:
+                fails.append(None)
+                return clean_up()
+            else:
+                d = self._run_user(self.case._run_test_method, self.result)
+                d.addCallback(fail_if_exception_caught)
+                d.addBoth(tear_down)
+                return d
+
+        def tear_down(ignored):
+            d = self._run_user(self.case._run_teardown, self.result)
+            d.addCallback(fail_if_exception_caught)
+            d.addBoth(clean_up)
+            return d
+
+        d = self._run_user(self.case._run_setup, self.result)
+        d.addCallback(set_up_done)
+        d.addBoth(lambda ignored: len(fails) == 0)
+        return d
+
+    def _log_user_exception(self, e):
+        """Raise 'e' and report it as a user exception."""
+        try:
+            raise e
+        except e.__class__:
+            self._got_user_exception(sys.exc_info())
+
+    def _run_core(self):
+        spinner = Spinner(self._reactor)
+        try:
+            successful, unhandled = trap_unhandled_errors(
+                spinner.run, self._timeout, self._run_deferred)
+        except NoResultError:
+            # We didn't get a result at all!  This could be for any number of
+            # reasons, but most likely someone hit Ctrl-C during the test.
+            raise KeyboardInterrupt
+        except TimeoutError:
+            # The function took too long to run.  No point reporting about
+            # junk and we don't have any information about unhandled errors in
+            # deferreds.  Report the timeout and skip to the end.
+            self._log_user_exception(TimeoutError(self.case, self._timeout))
+            return
+
+        if unhandled:
+            successful = False
+            # XXX: Maybe we could log creator & invoker here as well if
+            # present.
+            # XXX: Is there anything flagging these as unhandled errors?
+            for debug_info in unhandled:
+                f = debug_info.failResult
+                self._got_user_exception((f.type, f.value, f.tb))
+        junk = spinner.clear_junk()
+        if junk:
+            successful = False
+            self._log_user_exception(UncleanReactorError(junk))
+        if successful:
+            self.result.addSuccess(self.case, details=self.case.getDetails())
+
+    def _run_user(self, function, *args):
+        """Run a user-supplied function.
+
+        This just makes sure that it returns a Deferred, regardless of how the
+        user wrote it.
+        """
+        return defer.maybeDeferred(
+            super(AsynchronousDeferredRunTest, self)._run_user,
+            function, *args)
+
+
+def assert_fails_with(d, *exc_types, **kwargs):
+    """Assert that 'd' will fail with one of 'exc_types'.
+
+    The normal way to use this is to return the result of 'assert_fails_with'
+    from your unit test.
+
+    :param d: A Deferred that is expected to fail.
+    :param *exc_types: The exception types that the Deferred is expected to
+        fail with.
+    :param failureException: An optional keyword argument.  If provided, will
+        raise that exception instead of `testtools.TestCase.failureException`.
+    :return: A Deferred that will fail with an `AssertionError` if 'd' does
+        not fail with one of the exception types.
+    """
+    failureException = kwargs.pop('failureException', None)
+    if failureException is None:
+        # Avoid circular imports.
+        from testtools import TestCase
+        failureException = TestCase.failureException
+    expected_names = ", ".join(exc_type.__name__ for exc_type in exc_types)
+    def got_success(result):
+        raise failureException(
+            "%s not raised (%r returned)" % (expected_names, result))
+    def got_failure(failure):
+        if failure.check(*exc_types):
+            return failure.value
+        raise failureException("%s raised instead of %s:\n %s" % (
+            failure.type.__name__, expected_names, failure.getTraceback()))
+    return d.addCallbacks(got_success, got_failure)
+
+
+class UncleanReactorError(Exception):
+    """Raised when the reactor has junk in it."""
+
+    def __init__(self, junk):
+        super(UncleanReactorError, self).__init__(
+            "The reactor still thinks it needs to do things. Close all "
+            "connections, kill all processes and make sure all delayed "
+            "calls have either fired or been cancelled.  The management "
+            "thanks you: %s"
+            % map(repr, junk))
+
+

=== modified file 'testtools/runtest.py'
--- testtools/runtest.py	2010-08-15 23:18:59 +0000
+++ testtools/runtest.py	2010-10-17 21:32:44 +0000
@@ -2,7 +2,6 @@
 
 """Individual test case execution."""
 
-__metaclass__ = type
 __all__ = [
     'RunTest',
     ]
@@ -145,14 +144,17 @@
         except KeyboardInterrupt:
             raise
         except:
-            exc_info = sys.exc_info()
-            try:
-                e = exc_info[1]
-                self.case.onException(exc_info)
-            finally:
-                del exc_info
-            for exc_class, handler in self.handlers:
-                if isinstance(e, exc_class):
-                    self._exceptions.append(e)
-                    return self.exception_caught
-            raise e
+            return self._got_user_exception(sys.exc_info())
+
+    def _got_user_exception(self, exc_info):
+        """Called when user code raises an exception."""
+        try:
+            e = exc_info[1]
+            self.case.onException(exc_info)
+        finally:
+            del exc_info
+        for exc_class, handler in self.handlers:
+            if isinstance(e, exc_class):
+                self._exceptions.append(e)
+                return self.exception_caught
+        raise e

=== modified file 'testtools/testcase.py'
--- testtools/testcase.py	2010-10-14 22:58:49 +0000
+++ testtools/testcase.py	2010-10-17 21:32:44 +0000
@@ -6,10 +6,11 @@
 __all__ = [
     'clone_test_with_new_id',
     'MultipleExceptions',
-    'TestCase',
+    'run_test_with',
     'skip',
     'skipIf',
     'skipUnless',
+    'TestCase',
     ]
 
 import copy
@@ -61,6 +62,29 @@
         """
 
 
+def run_test_with(test_runner, **kwargs):
+    """Decorate a test as using a specific `RunTest`.
+
+    e.g.
+      @run_test_with(CustomRunner, timeout=42)
+      def test_foo(self):
+          self.assertTrue(True)
+
+    :param test_runner: A `RunTest` factory that takes a test case and an
+        optional list of exception handlers.  See `RunTest`.
+    :param **kwargs: Keyword arguments to pass on as extra arguments to
+        `test_runner`.
+    :return: A decorator to be used for marking a test as needing a special
+        runner.
+    """
+    def make_test_runner(case, handlers=None):
+        return test_runner(case, handlers=handlers, **kwargs)
+    def decorator(f):
+        f._run_test_with = make_test_runner
+        return f
+    return decorator
+
+
 class MultipleExceptions(Exception):
     """Represents many exceptions raised from some operation.
 
@@ -74,18 +98,25 @@
     :ivar exception_handlers: Exceptions to catch from setUp, runTest and
         tearDown. This list is able to be modified at any time and consists of
         (exception_class, handler(case, result, exception_value)) pairs.
+    :cvar run_tests_with: A factory to make the `RunTest` to run tests with.
+        Defaults to `RunTest`.  The factory is expected to take a test case
+        and an optional list of exception handlers.
     """
 
     skipException = TestSkipped
 
+    run_tests_with = RunTest
+
     def __init__(self, *args, **kwargs):
         """Construct a TestCase.
 
         :param testMethod: The name of the method to run.
         :param runTest: Optional class to use to execute the test. If not
-            supplied testtools.runtest.RunTest is used. The instance to be
+            supplied `testtools.runtest.RunTest` is used. The instance to be
             used is created when run() is invoked, so will be fresh each time.
+            Overrides `run_tests_with` if given.
         """
+        runTest = kwargs.pop('runTest', None)
         unittest.TestCase.__init__(self, *args, **kwargs)
         self._cleanups = []
         self._unique_id_gen = itertools.count(1)
@@ -95,7 +126,11 @@
         # __details is lazy-initialized so that a constructed-but-not-run
         # TestCase is safe to use with clone_test_with_new_id.
         self.__details = None
-        self.__RunTest = kwargs.get('runTest', RunTest)
+        test_method = self._get_test_method()
+        if runTest is None:
+            runTest = getattr(
+                test_method, '_run_test_with', self.run_tests_with)
+        self.__RunTest = runTest
         self.__exception_handlers = []
         self.exception_handlers = [
             (self.skipException, self._report_skip),
@@ -448,13 +483,14 @@
         :raises ValueError: If the base class setUp is not called, a
             ValueError is raised.
         """
-        self.setUp()
+        ret = self.setUp()
         if not self.__setup_called:
             raise ValueError(
                 "TestCase.setUp was not called. Have you upcalled all the "
                 "way up the hierarchy from your setUp? e.g. Call "
                 "super(%s, self).setUp() from your setUp()."
                 % self.__class__.__name__)
+        return ret
 
     def _run_teardown(self, result):
         """Run the tearDown function for this test.
@@ -463,28 +499,31 @@
         :raises ValueError: If the base class tearDown is not called, a
             ValueError is raised.
         """
-        self.tearDown()
+        ret = self.tearDown()
         if not self.__teardown_called:
             raise ValueError(
                 "TestCase.tearDown was not called. Have you upcalled all the "
                 "way up the hierarchy from your tearDown? e.g. Call "
                 "super(%s, self).tearDown() from your tearDown()."
                 % self.__class__.__name__)
-
-    def _run_test_method(self, result):
-        """Run the test method for this test.
-
-        :param result: A testtools.TestResult to report activity to.
-        :return: None.
-        """
+        return ret
+
+    def _get_test_method(self):
         absent_attr = object()
         # Python 2.5+
         method_name = getattr(self, '_testMethodName', absent_attr)
         if method_name is absent_attr:
             # Python 2.4
             method_name = getattr(self, '_TestCase__testMethodName')
-        testMethod = getattr(self, method_name)
-        testMethod()
+        return getattr(self, method_name)
+
+    def _run_test_method(self, result):
+        """Run the test method for this test.
+
+        :param result: A testtools.TestResult to report activity to.
+        :return: None.
+        """
+        return self._get_test_method()()
 
     def setUp(self):
         unittest.TestCase.setUp(self)

=== modified file 'testtools/tests/__init__.py'
--- testtools/tests/__init__.py	2010-08-04 12:45:22 +0000
+++ testtools/tests/__init__.py	2010-10-17 21:32:44 +0000
@@ -7,9 +7,11 @@
     test_compat,
     test_content,
     test_content_type,
+    test_deferredruntest,
     test_matchers,
     test_monkey,
     test_runtest,
+    test_spinner,
     test_testtools,
     test_testresult,
     test_testsuite,
@@ -22,9 +24,11 @@
         test_compat,
         test_content,
         test_content_type,
+        test_deferredruntest,
         test_matchers,
         test_monkey,
         test_runtest,
+        test_spinner,
         test_testresult,
         test_testsuite,
         test_testtools,

=== added file 'testtools/tests/test_deferredruntest.py'
--- testtools/tests/test_deferredruntest.py	1970-01-01 00:00:00 +0000
+++ testtools/tests/test_deferredruntest.py	2010-10-17 21:32:44 +0000
@@ -0,0 +1,531 @@
+# Copyright (c) 2010 Jonathan M. Lange. See LICENSE for details.
+
+"""Tests for the DeferredRunTest single test execution logic."""
+
+import os
+import signal
+
+from testtools import (
+    TestCase,
+    )
+from testtools.deferredruntest import (
+    assert_fails_with,
+    AsynchronousDeferredRunTest,
+    SynchronousDeferredRunTest,
+    )
+from testtools.tests.helpers import ExtendedTestResult
+from testtools.matchers import (
+    Equals,
+    )
+from testtools.runtest import RunTest
+
+from twisted.internet import defer
+from twisted.python import failure
+
+
+class X(object):
+    """Tests that we run as part of our tests, nested to avoid discovery."""
+
+    class Base(TestCase):
+        def setUp(self):
+            super(X.Base, self).setUp()
+            self.calls = ['setUp']
+            self.addCleanup(self.calls.append, 'clean-up')
+        def test_something(self):
+            self.calls.append('test')
+        def tearDown(self):
+            self.calls.append('tearDown')
+            super(X.Base, self).tearDown()
+
+    class Success(Base):
+        expected_calls = ['setUp', 'test', 'tearDown', 'clean-up']
+        expected_results = [['addSuccess']]
+
+    class ErrorInSetup(Base):
+        expected_calls = ['setUp', 'clean-up']
+        expected_results = [('addError', RuntimeError)]
+        def setUp(self):
+            super(X.ErrorInSetup, self).setUp()
+            raise RuntimeError("Error in setUp")
+
+    class ErrorInTest(Base):
+        expected_calls = ['setUp', 'tearDown', 'clean-up']
+        expected_results = [('addError', RuntimeError)]
+        def test_something(self):
+            raise RuntimeError("Error in test")
+
+    class FailureInTest(Base):
+        expected_calls = ['setUp', 'tearDown', 'clean-up']
+        expected_results = [('addFailure', AssertionError)]
+        def test_something(self):
+            self.fail("test failed")
+
+    class ErrorInTearDown(Base):
+        expected_calls = ['setUp', 'test', 'clean-up']
+        expected_results = [('addError', RuntimeError)]
+        def tearDown(self):
+            raise RuntimeError("Error in tearDown")
+
+    class ErrorInCleanup(Base):
+        expected_calls = ['setUp', 'test', 'tearDown', 'clean-up']
+        expected_results = [('addError', ZeroDivisionError)]
+        def test_something(self):
+            self.calls.append('test')
+            self.addCleanup(lambda: 1/0)
+
+    class TestIntegration(TestCase):
+
+        def assertResultsMatch(self, test, result):
+            events = list(result._events)
+            self.assertEqual(('startTest', test), events.pop(0))
+            for expected_result in test.expected_results:
+                result = events.pop(0)
+                if len(expected_result) == 1:
+                    self.assertEqual((expected_result[0], test), result)
+                else:
+                    self.assertEqual((expected_result[0], test), result[:2])
+                    error_type = expected_result[1]
+                    self.assertIn(error_type.__name__, str(result[2]))
+            self.assertEqual([('stopTest', test)], events)
+
+        def test_runner(self):
+            result = ExtendedTestResult()
+            test = self.test_factory('test_something', runTest=self.runner)
+            test.run(result)
+            self.assertEqual(test.calls, self.test_factory.expected_calls)
+            self.assertResultsMatch(test, result)
+
+
+def make_integration_tests():
+    from unittest import TestSuite
+    from testtools import clone_test_with_new_id
+    runners = [
+        RunTest,
+        SynchronousDeferredRunTest,
+        AsynchronousDeferredRunTest,
+        ]
+
+    tests = [
+        X.Success,
+        X.ErrorInSetup,
+        X.ErrorInTest,
+        X.ErrorInTearDown,
+        X.FailureInTest,
+        X.ErrorInCleanup,
+        ]
+    base_test = X.TestIntegration('test_runner')
+    integration_tests = []
+    for runner in runners:
+        for test in tests:
+            new_test = clone_test_with_new_id(
+                base_test, '%s(%s, %s)' % (
+                    base_test.id(),
+                    runner.__name__,
+                    test.__name__))
+            new_test.test_factory = test
+            new_test.runner = runner
+            integration_tests.append(new_test)
+    return TestSuite(integration_tests)
+
+
+class TestSynchronousDeferredRunTest(TestCase):
+
+    def make_result(self):
+        return ExtendedTestResult()
+
+    def make_runner(self, test):
+        return SynchronousDeferredRunTest(test, test.exception_handlers)
+
+    def test_success(self):
+        class SomeCase(TestCase):
+            def test_success(self):
+                return defer.succeed(None)
+        test = SomeCase('test_success')
+        runner = self.make_runner(test)
+        result = self.make_result()
+        runner.run(result)
+        self.assertThat(
+            result._events, Equals([
+                ('startTest', test),
+                ('addSuccess', test),
+                ('stopTest', test)]))
+
+    def test_failure(self):
+        class SomeCase(TestCase):
+            def test_failure(self):
+                return defer.maybeDeferred(self.fail, "Egads!")
+        test = SomeCase('test_failure')
+        runner = self.make_runner(test)
+        result = self.make_result()
+        runner.run(result)
+        self.assertThat(
+            [event[:2] for event in result._events], Equals([
+                ('startTest', test),
+                ('addFailure', test),
+                ('stopTest', test)]))
+
+    def test_setUp_followed_by_test(self):
+        class SomeCase(TestCase):
+            def setUp(self):
+                super(SomeCase, self).setUp()
+                return defer.succeed(None)
+            def test_failure(self):
+                return defer.maybeDeferred(self.fail, "Egads!")
+        test = SomeCase('test_failure')
+        runner = self.make_runner(test)
+        result = self.make_result()
+        runner.run(result)
+        self.assertThat(
+            [event[:2] for event in result._events], Equals([
+                ('startTest', test),
+                ('addFailure', test),
+                ('stopTest', test)]))
+
+
+class TestAsynchronousDeferredRunTest(TestCase):
+
+    def make_reactor(self):
+        from twisted.internet import reactor
+        return reactor
+
+    def make_result(self):
+        return ExtendedTestResult()
+
+    def make_runner(self, test, timeout=None):
+        if timeout is None:
+            timeout = self.make_timeout()
+        return AsynchronousDeferredRunTest(
+            test, test.exception_handlers, timeout=timeout)
+
+    def make_timeout(self):
+        return 0.005
+
+    def test_setUp_returns_deferred_that_fires_later(self):
+        # setUp can return a Deferred that might fire at any time.
+        # AsynchronousDeferredRunTest will not go on to running the test until
+        # the Deferred returned by setUp actually fires.
+        call_log = []
+        marker = object()
+        d = defer.Deferred().addCallback(call_log.append)
+        class SomeCase(TestCase):
+            def setUp(self):
+                super(SomeCase, self).setUp()
+                call_log.append('setUp')
+                return d
+            def test_something(self):
+                call_log.append('test')
+        def fire_deferred():
+            self.assertThat(call_log, Equals(['setUp']))
+            d.callback(marker)
+        test = SomeCase('test_something')
+        timeout = self.make_timeout()
+        runner = self.make_runner(test, timeout=timeout)
+        result = self.make_result()
+        reactor = self.make_reactor()
+        reactor.callLater(timeout, fire_deferred)
+        runner.run(result)
+        self.assertThat(call_log, Equals(['setUp', marker, 'test']))
+
+    def test_calls_setUp_test_tearDown_in_sequence(self):
+        # setUp, the test method and tearDown can all return
+        # Deferreds. AsynchronousDeferredRunTest will make sure that each of
+        # these are run in turn, only going on to the next stage once the
+        # Deferred from the previous stage has fired.
+        call_log = []
+        a = defer.Deferred()
+        a.addCallback(lambda x: call_log.append('a'))
+        b = defer.Deferred()
+        b.addCallback(lambda x: call_log.append('b'))
+        c = defer.Deferred()
+        c.addCallback(lambda x: call_log.append('c'))
+        class SomeCase(TestCase):
+            def setUp(self):
+                super(SomeCase, self).setUp()
+                call_log.append('setUp')
+                return a
+            def test_success(self):
+                call_log.append('test')
+                return b
+            def tearDown(self):
+                super(SomeCase, self).tearDown()
+                call_log.append('tearDown')
+                return c
+        test = SomeCase('test_success')
+        timeout = self.make_timeout()
+        runner = self.make_runner(test, timeout)
+        result = self.make_result()
+        reactor = self.make_reactor()
+        def fire_a():
+            self.assertThat(call_log, Equals(['setUp']))
+            a.callback(None)
+        def fire_b():
+            self.assertThat(call_log, Equals(['setUp', 'a', 'test']))
+            b.callback(None)
+        def fire_c():
+            self.assertThat(
+                call_log, Equals(['setUp', 'a', 'test', 'b', 'tearDown']))
+            c.callback(None)
+        reactor.callLater(timeout * 0.25, fire_a)
+        reactor.callLater(timeout * 0.5, fire_b)
+        reactor.callLater(timeout * 0.75, fire_c)
+        runner.run(result)
+        self.assertThat(
+            call_log, Equals(['setUp', 'a', 'test', 'b', 'tearDown', 'c']))
+
+    def test_async_cleanups(self):
+        # Cleanups added with addCleanup can return
+        # Deferreds. AsynchronousDeferredRunTest will run each of them in
+        # turn.
+        class SomeCase(TestCase):
+            def test_whatever(self):
+                pass
+        test = SomeCase('test_whatever')
+        log = []
+        a = defer.Deferred().addCallback(lambda x: log.append('a'))
+        b = defer.Deferred().addCallback(lambda x: log.append('b'))
+        c = defer.Deferred().addCallback(lambda x: log.append('c'))
+        test.addCleanup(lambda: a)
+        test.addCleanup(lambda: b)
+        test.addCleanup(lambda: c)
+        def fire_a():
+            self.assertThat(log, Equals([]))
+            a.callback(None)
+        def fire_b():
+            self.assertThat(log, Equals(['a']))
+            b.callback(None)
+        def fire_c():
+            self.assertThat(log, Equals(['a', 'b']))
+            c.callback(None)
+        timeout = self.make_timeout()
+        reactor = self.make_reactor()
+        reactor.callLater(timeout * 0.25, fire_a)
+        reactor.callLater(timeout * 0.5, fire_b)
+        reactor.callLater(timeout * 0.75, fire_c)
+        runner = self.make_runner(test, timeout)
+        result = self.make_result()
+        runner.run(result)
+        self.assertThat(log, Equals(['a', 'b', 'c']))
+
+    def test_clean_reactor(self):
+        # If there's cruft left over in the reactor, the test fails.
+        reactor = self.make_reactor()
+        timeout = self.make_timeout()
+        class SomeCase(TestCase):
+            def test_cruft(self):
+                reactor.callLater(timeout * 2.0, lambda: None)
+        test = SomeCase('test_cruft')
+        runner = self.make_runner(test, timeout)
+        result = self.make_result()
+        runner.run(result)
+        error = result._events[1][2]
+        result._events[1] = ('addError', test, None)
+        self.assertThat(result._events, Equals(
+            [('startTest', test),
+             ('addError', test, None),
+             ('stopTest', test)]))
+        self.assertThat(list(error.keys()), Equals(['traceback']))
+
+    def test_unhandled_error_from_deferred(self):
+        # If there's a Deferred with an unhandled error, the test fails.  Each
+        # unhandled error is reported with a separate traceback.
+        class SomeCase(TestCase):
+            def test_cruft(self):
+                # Note we aren't returning the Deferred so that the error will
+                # be unhandled.
+                defer.maybeDeferred(lambda: 1/0)
+                defer.maybeDeferred(lambda: 2/0)
+        test = SomeCase('test_cruft')
+        runner = self.make_runner(test)
+        result = self.make_result()
+        runner.run(result)
+        error = result._events[1][2]
+        result._events[1] = ('addError', test, None)
+        self.assertThat(result._events, Equals(
+            [('startTest', test),
+             ('addError', test, None),
+             ('stopTest', test)]))
+        self.assertThat(
+            list(error.keys()), Equals(['traceback', 'traceback-1']))
+
+    def test_keyboard_interrupt_stops_test_run(self):
+        # If we get a SIGINT during a test run, the test stops and no more
+        # tests run.
+        class SomeCase(TestCase):
+            def test_pause(self):
+                return defer.Deferred()
+        test = SomeCase('test_pause')
+        reactor = self.make_reactor()
+        timeout = self.make_timeout()
+        runner = self.make_runner(test, timeout * 5)
+        result = self.make_result()
+        reactor.callLater(timeout, os.kill, os.getpid(), signal.SIGINT)
+        self.assertRaises(KeyboardInterrupt, runner.run, result)
+
+    def test_fast_keyboard_interrupt_stops_test_run(self):
+        # If we get a SIGINT during a test run, the test stops and no more
+        # tests run.
+        class SomeCase(TestCase):
+            def test_pause(self):
+                return defer.Deferred()
+        test = SomeCase('test_pause')
+        reactor = self.make_reactor()
+        timeout = self.make_timeout()
+        runner = self.make_runner(test, timeout * 5)
+        result = self.make_result()
+        reactor.callWhenRunning(os.kill, os.getpid(), signal.SIGINT)
+        self.assertRaises(KeyboardInterrupt, runner.run, result)
+
+    def test_timeout_causes_test_error(self):
+        # If a test times out, it reports itself as having failed with a
+        # TimeoutError.
+        class SomeCase(TestCase):
+            def test_pause(self):
+                return defer.Deferred()
+        test = SomeCase('test_pause')
+        runner = self.make_runner(test)
+        result = self.make_result()
+        runner.run(result)
+        error = result._events[1][2]
+        self.assertThat(
+            [event[:2] for event in result._events], Equals(
+            [('startTest', test),
+             ('addError', test),
+             ('stopTest', test)]))
+        self.assertIn('TimeoutError', str(error['traceback']))
+
+    def test_convenient_construction(self):
+        # As a convenience method, AsynchronousDeferredRunTest has a
+        # classmethod that returns an AsynchronousDeferredRunTest
+        # factory. This factory has the same API as the RunTest constructor.
+        reactor = object()
+        timeout = object()
+        handler = object()
+        factory = AsynchronousDeferredRunTest.make_factory(reactor, timeout)
+        runner = factory(self, [handler])
+        self.assertIs(reactor, runner._reactor)
+        self.assertIs(timeout, runner._timeout)
+        self.assertIs(self, runner.case)
+        self.assertEqual([handler], runner.handlers)
+
+    def test_only_addError_once(self):
+        # Even if the reactor is unclean and the test raises an error and the
+        # cleanups raise errors, we only called addError once per test.
+        reactor = self.make_reactor()
+        class WhenItRains(TestCase):
+            def it_pours(self):
+                # Add a dirty cleanup.
+                self.addCleanup(lambda: 3 / 0)
+                # Dirty the reactor.
+                from twisted.internet.protocol import ServerFactory
+                reactor.listenTCP(0, ServerFactory())
+                # Unhandled error.
+                defer.maybeDeferred(lambda: 2 / 0)
+                # Actual error.
+                raise RuntimeError("Excess precipitation")
+        test = WhenItRains('it_pours')
+        runner = self.make_runner(test)
+        result = self.make_result()
+        runner.run(result)
+        self.assertThat(
+            [event[:2] for event in result._events],
+            Equals([
+                ('startTest', test),
+                ('addError', test),
+                ('stopTest', test)]))
+        error = result._events[1][2]
+        self.assertThat(
+            sorted(error.keys()), Equals([
+                'traceback',
+                'traceback-1',
+                'traceback-2',
+                'traceback-3',
+                ]))
+
+
+class TestAssertFailsWith(TestCase):
+    """Tests for `assert_fails_with`."""
+
+    run_tests_with = SynchronousDeferredRunTest
+
+    def test_assert_fails_with_success(self):
+        # assert_fails_with fails the test if it's given a Deferred that
+        # succeeds.
+        marker = object()
+        d = assert_fails_with(defer.succeed(marker), RuntimeError)
+        def check_result(failure):
+            failure.trap(self.failureException)
+            self.assertThat(
+                str(failure.value),
+                Equals("RuntimeError not raised (%r returned)" % (marker,)))
+        d.addCallbacks(
+            lambda x: self.fail("Should not have succeeded"), check_result)
+        return d
+
+    def test_assert_fails_with_success_multiple_types(self):
+        # assert_fails_with fails the test if it's given a Deferred that
+        # succeeds.
+        marker = object()
+        d = assert_fails_with(
+            defer.succeed(marker), RuntimeError, ZeroDivisionError)
+        def check_result(failure):
+            failure.trap(self.failureException)
+            self.assertThat(
+                str(failure.value),
+                Equals("RuntimeError, ZeroDivisionError not raised "
+                       "(%r returned)" % (marker,)))
+        d.addCallbacks(
+            lambda x: self.fail("Should not have succeeded"), check_result)
+        return d
+
+    def test_assert_fails_with_wrong_exception(self):
+        # assert_fails_with fails the test if it's given a Deferred that
+        # succeeds.
+        d = assert_fails_with(
+            defer.maybeDeferred(lambda: 1/0), RuntimeError, KeyboardInterrupt)
+        def check_result(failure):
+            failure.trap(self.failureException)
+            lines = str(failure.value).splitlines()
+            self.assertThat(
+                lines[:2],
+                Equals([
+                    ("ZeroDivisionError raised instead of RuntimeError, "
+                     "KeyboardInterrupt:"),
+                    " Traceback (most recent call last):",
+                    ]))
+        d.addCallbacks(
+            lambda x: self.fail("Should not have succeeded"), check_result)
+        return d
+
+    def test_assert_fails_with_expected_exception(self):
+        # assert_fails_with calls back with the value of the failure if it's
+        # one of the expected types of failures.
+        try:
+            1/0
+        except ZeroDivisionError:
+            f = failure.Failure()
+        d = assert_fails_with(defer.fail(f), ZeroDivisionError)
+        return d.addCallback(self.assertThat, Equals(f.value))
+
+    def test_custom_failure_exception(self):
+        # If assert_fails_with is passed a 'failureException' keyword
+        # argument, then it will raise that instead of `AssertionError`.
+        class CustomException(Exception):
+            pass
+        marker = object()
+        d = assert_fails_with(
+            defer.succeed(marker), RuntimeError,
+            failureException=CustomException)
+        def check_result(failure):
+            failure.trap(CustomException)
+            self.assertThat(
+                str(failure.value),
+                Equals("RuntimeError not raised (%r returned)" % (marker,)))
+        return d.addCallbacks(
+            lambda x: self.fail("Should not have succeeded"), check_result)
+
+
+def test_suite():
+    from unittest import TestLoader, TestSuite
+    return TestSuite(
+        [TestLoader().loadTestsFromName(__name__),
+         make_integration_tests()])

=== modified file 'testtools/tests/test_runtest.py'
--- testtools/tests/test_runtest.py	2010-05-13 12:15:12 +0000
+++ testtools/tests/test_runtest.py	2010-10-17 21:32:44 +0000
@@ -4,10 +4,12 @@
 
 from testtools import (
     ExtendedToOriginalDecorator,
+    run_test_with,
     RunTest,
     TestCase,
     TestResult,
     )
+from testtools.matchers import Is
 from testtools.tests.helpers import ExtendedTestResult
 
 
@@ -176,6 +178,99 @@
             ], result._events)
 
 
+class CustomRunTest(RunTest):
+
+    marker = object()
+
+    def run(self, result=None):
+        return self.marker
+
+
+class TestTestCaseSupportForRunTest(TestCase):
+
+    def test_pass_custom_run_test(self):
+        class SomeCase(TestCase):
+            def test_foo(self):
+                pass
+        result = TestResult()
+        case = SomeCase('test_foo', runTest=CustomRunTest)
+        from_run_test = case.run(result)
+        self.assertThat(from_run_test, Is(CustomRunTest.marker))
+
+    def test_default_is_runTest_class_variable(self):
+        class SomeCase(TestCase):
+            run_tests_with = CustomRunTest
+            def test_foo(self):
+                pass
+        result = TestResult()
+        case = SomeCase('test_foo')
+        from_run_test = case.run(result)
+        self.assertThat(from_run_test, Is(CustomRunTest.marker))
+
+    def test_constructor_argument_overrides_class_variable(self):
+        # If a 'runTest' argument is passed to the test's constructor, that
+        # overrides the class variable.
+        marker = object()
+        class DifferentRunTest(RunTest):
+            def run(self, result=None):
+                return marker
+        class SomeCase(TestCase):
+            run_tests_with = CustomRunTest
+            def test_foo(self):
+                pass
+        result = TestResult()
+        case = SomeCase('test_foo', runTest=DifferentRunTest)
+        from_run_test = case.run(result)
+        self.assertThat(from_run_test, Is(marker))
+
+    def test_decorator_for_run_test(self):
+        # Individual test methods can be marked as needing a special runner.
+        class SomeCase(TestCase):
+            @run_test_with(CustomRunTest)
+            def test_foo(self):
+                pass
+        result = TestResult()
+        case = SomeCase('test_foo')
+        from_run_test = case.run(result)
+        self.assertThat(from_run_test, Is(CustomRunTest.marker))
+
+    def test_extended_decorator_for_run_test(self):
+        # Individual test methods can be marked as needing a special runner.
+        # Extra arguments can be passed to the decorator which will then be
+        # passed on to the RunTest object.
+        marker = object()
+        class FooRunTest(RunTest):
+            def __init__(self, case, handlers=None, bar=None):
+                super(FooRunTest, self).__init__(case, handlers)
+                self.bar = bar
+            def run(self, result=None):
+                return self.bar
+        class SomeCase(TestCase):
+            @run_test_with(FooRunTest, bar=marker)
+            def test_foo(self):
+                pass
+        result = TestResult()
+        case = SomeCase('test_foo')
+        from_run_test = case.run(result)
+        self.assertThat(from_run_test, Is(marker))
+
+    def test_constructor_overrides_decorator(self):
+        # If a 'runTest' argument is passed to the test's constructor, that
+        # overrides the decorator.
+        marker = object()
+        class DifferentRunTest(RunTest):
+            def run(self, result=None):
+                return marker
+        class SomeCase(TestCase):
+            @run_test_with(CustomRunTest)
+            def test_foo(self):
+                pass
+        result = TestResult()
+        case = SomeCase('test_foo', runTest=DifferentRunTest)
+        from_run_test = case.run(result)
+        self.assertThat(from_run_test, Is(marker))
+
+
 def test_suite():
     from unittest import TestLoader
     return TestLoader().loadTestsFromName(__name__)

=== added file 'testtools/tests/test_spinner.py'
--- testtools/tests/test_spinner.py	1970-01-01 00:00:00 +0000
+++ testtools/tests/test_spinner.py	2010-10-17 21:32:44 +0000
@@ -0,0 +1,306 @@
+# Copyright (c) 2010 Jonathan M. Lange. See LICENSE for details.
+
+"""Tests for the evil Twisted reactor-spinning we do."""
+
+import os
+import signal
+
+from testtools import (
+    TestCase,
+    )
+from testtools.matchers import (
+    Equals,
+    Is,
+    )
+from testtools._spinner import (
+    DeferredNotFired,
+    extract_result,
+    NoResultError,
+    not_reentrant,
+    ReentryError,
+    Spinner,
+    StaleJunkError,
+    TimeoutError,
+    trap_unhandled_errors,
+    )
+
+from twisted.internet import defer
+from twisted.python.failure import Failure
+
+
+class TestNotReentrant(TestCase):
+
+    def test_not_reentrant(self):
+        # A function decorated as not being re-entrant will raise a
+        # ReentryError if it is called while it is running.
+        calls = []
+        @not_reentrant
+        def log_something():
+            calls.append(None)
+            if len(calls) < 5:
+                log_something()
+        self.assertRaises(ReentryError, log_something)
+        self.assertEqual(1, len(calls))
+
+    def test_deeper_stack(self):
+        calls = []
+        @not_reentrant
+        def g():
+            calls.append(None)
+            if len(calls) < 5:
+                f()
+        @not_reentrant
+        def f():
+            calls.append(None)
+            if len(calls) < 5:
+                g()
+        self.assertRaises(ReentryError, f)
+        self.assertEqual(2, len(calls))
+
+
+class TestExtractResult(TestCase):
+
+    def test_not_fired(self):
+        # extract_result raises DeferredNotFired if it's given a Deferred that
+        # has not fired.
+        self.assertRaises(DeferredNotFired, extract_result, defer.Deferred())
+
+    def test_success(self):
+        # extract_result returns the value of the Deferred if it has fired
+        # successfully.
+        marker = object()
+        d = defer.succeed(marker)
+        self.assertThat(extract_result(d), Equals(marker))
+
+    def test_failure(self):
+        # extract_result raises the failure's exception if it's given a
+        # Deferred that is failing.
+        try:
+            1/0
+        except ZeroDivisionError:
+            f = Failure()
+        d = defer.fail(f)
+        self.assertRaises(ZeroDivisionError, extract_result, d)
+
+
+class TestTrapUnhandledErrors(TestCase):
+
+    def test_no_deferreds(self):
+        marker = object()
+        result, errors = trap_unhandled_errors(lambda: marker)
+        self.assertEqual([], errors)
+        self.assertIs(marker, result)
+
+    def test_unhandled_error(self):
+        failures = []
+        def make_deferred_but_dont_handle():
+            try:
+                1/0
+            except ZeroDivisionError:
+                f = Failure()
+                failures.append(f)
+                defer.fail(f)
+        result, errors = trap_unhandled_errors(make_deferred_but_dont_handle)
+        self.assertIs(None, result)
+        self.assertEqual(failures, [error.failResult for error in errors])
+
+
+class TestRunInReactor(TestCase):
+
+    def make_reactor(self):
+        from twisted.internet import reactor
+        return reactor
+
+    def make_spinner(self, reactor=None):
+        if reactor is None:
+            reactor = self.make_reactor()
+        return Spinner(reactor)
+
+    def make_timeout(self):
+        return 0.01
+
+    def test_function_called(self):
+        # run_in_reactor actually calls the function given to it.
+        calls = []
+        marker = object()
+        self.make_spinner().run(self.make_timeout(), calls.append, marker)
+        self.assertThat(calls, Equals([marker]))
+
+    def test_return_value_returned(self):
+        # run_in_reactor returns the value returned by the function given to
+        # it.
+        marker = object()
+        result = self.make_spinner().run(self.make_timeout(), lambda: marker)
+        self.assertThat(result, Is(marker))
+
+    def test_exception_reraised(self):
+        # If the given function raises an error, run_in_reactor re-raises that
+        # error.
+        self.assertRaises(
+            ZeroDivisionError,
+            self.make_spinner().run, self.make_timeout(), lambda: 1 / 0)
+
+    def test_keyword_arguments(self):
+        # run_in_reactor passes keyword arguments on.
+        calls = []
+        function = lambda *a, **kw: calls.extend([a, kw])
+        self.make_spinner().run(self.make_timeout(), function, foo=42)
+        self.assertThat(calls, Equals([(), {'foo': 42}]))
+
+    def test_not_reentrant(self):
+        # run_in_reactor raises an error if it is called inside another call
+        # to run_in_reactor.
+        spinner = self.make_spinner()
+        self.assertRaises(
+            ReentryError,
+            spinner.run, self.make_timeout(),
+            spinner.run, self.make_timeout(), lambda: None)
+
+    def test_deferred_value_returned(self):
+        # If the given function returns a Deferred, run_in_reactor returns the
+        # value in the Deferred at the end of the callback chain.
+        marker = object()
+        result = self.make_spinner().run(
+            self.make_timeout(), lambda: defer.succeed(marker))
+        self.assertThat(result, Is(marker))
+
+    def test_preserve_signal_handler(self):
+        signals = [signal.SIGINT, signal.SIGTERM, signal.SIGCHLD]
+        for sig in signals:
+            self.addCleanup(signal.signal, sig, signal.getsignal(sig))
+        new_hdlrs = [lambda *a: None, lambda *a: None, lambda *a: None]
+        for sig, hdlr in zip(signals, new_hdlrs):
+            signal.signal(sig, hdlr)
+        spinner = self.make_spinner()
+        spinner.run(self.make_timeout(), lambda: None)
+        self.assertEqual(new_hdlrs, map(signal.getsignal, signals))
+
+    def test_timeout(self):
+        # If the function takes too long to run, we raise a TimeoutError.
+        timeout = self.make_timeout()
+        self.assertRaises(
+            TimeoutError,
+            self.make_spinner().run, timeout, lambda: defer.Deferred())
+
+    def test_no_junk_by_default(self):
+        # If the reactor hasn't spun yet, then there cannot be any junk.
+        spinner = self.make_spinner()
+        self.assertThat(spinner.get_junk(), Equals([]))
+
+    def test_clean_do_nothing(self):
+        # If there's nothing going on in the reactor, then clean does nothing
+        # and returns an empty list.
+        spinner = self.make_spinner()
+        result = spinner._clean()
+        self.assertThat(result, Equals([]))
+
+    def test_clean_delayed_call(self):
+        # If there's a delayed call in the reactor, then clean cancels it and
+        # returns an empty list.
+        reactor = self.make_reactor()
+        spinner = self.make_spinner(reactor)
+        call = reactor.callLater(10, lambda: None)
+        results = spinner._clean()
+        self.assertThat(results, Equals([call]))
+        self.assertThat(call.active(), Equals(False))
+
+    def test_clean_delayed_call_cancelled(self):
+        # If there's a delayed call that's just been cancelled, then it's no
+        # longer there.
+        reactor = self.make_reactor()
+        spinner = self.make_spinner(reactor)
+        call = reactor.callLater(10, lambda: None)
+        call.cancel()
+        results = spinner._clean()
+        self.assertThat(results, Equals([]))
+
+    def test_clean_selectables(self):
+        # If there's still a selectable (e.g. a listening socket), then
+        # clean() removes it from the reactor's registry.
+        #
+        # Note that the socket is left open. This emulates a bug in trial.
+        from twisted.internet.protocol import ServerFactory
+        reactor = self.make_reactor()
+        spinner = self.make_spinner(reactor)
+        port = reactor.listenTCP(0, ServerFactory())
+        spinner.run(self.make_timeout(), lambda: None)
+        results = spinner.get_junk()
+        self.assertThat(results, Equals([port]))
+
+    def test_clean_running_threads(self):
+        import threading
+        import time
+        current_threads = list(threading.enumerate())
+        reactor = self.make_reactor()
+        timeout = self.make_timeout()
+        spinner = self.make_spinner(reactor)
+        spinner.run(timeout, reactor.callInThread, time.sleep, timeout / 2.0)
+        self.assertThat(list(threading.enumerate()), Equals(current_threads))
+
+    def test_leftover_junk_available(self):
+        # If 'run' is given a function that leaves the reactor dirty in some
+        # way, 'run' will clean up the reactor and then store information
+        # about the junk. This information can be got using get_junk.
+        from twisted.internet.protocol import ServerFactory
+        reactor = self.make_reactor()
+        spinner = self.make_spinner(reactor)
+        port = spinner.run(
+            self.make_timeout(), reactor.listenTCP, 0, ServerFactory())
+        self.assertThat(spinner.get_junk(), Equals([port]))
+
+    def test_will_not_run_with_previous_junk(self):
+        # If 'run' is called and there's still junk in the spinner's junk
+        # list, then the spinner will refuse to run.
+        from twisted.internet.protocol import ServerFactory
+        reactor = self.make_reactor()
+        spinner = self.make_spinner(reactor)
+        timeout = self.make_timeout()
+        spinner.run(timeout, reactor.listenTCP, 0, ServerFactory())
+        self.assertRaises(
+            StaleJunkError, spinner.run, timeout, lambda: None)
+
+    def test_clear_junk_clears_previous_junk(self):
+        # If 'run' is called and there's still junk in the spinner's junk
+        # list, then the spinner will refuse to run.
+        from twisted.internet.protocol import ServerFactory
+        reactor = self.make_reactor()
+        spinner = self.make_spinner(reactor)
+        timeout = self.make_timeout()
+        port = spinner.run(timeout, reactor.listenTCP, 0, ServerFactory())
+        junk = spinner.clear_junk()
+        self.assertThat(junk, Equals([port]))
+        self.assertThat(spinner.get_junk(), Equals([]))
+
+    def test_sigint_raises_no_result_error(self):
+        # If we get a SIGINT during a run, we raise NoResultError.
+        reactor = self.make_reactor()
+        spinner = self.make_spinner(reactor)
+        timeout = self.make_timeout()
+        reactor.callLater(timeout, os.kill, os.getpid(), signal.SIGINT)
+        self.assertRaises(
+            NoResultError, spinner.run, timeout * 5, defer.Deferred)
+        self.assertEqual([], spinner._clean())
+
+    def test_sigint_raises_no_result_error_second_time(self):
+        # If we get a SIGINT during a run, we raise NoResultError.  This test
+        # is exactly the same as test_sigint_raises_no_result_error, and
+        # exists to make sure we haven't futzed with state.
+        self.test_sigint_raises_no_result_error()
+
+    def test_fast_sigint_raises_no_result_error(self):
+        # If we get a SIGINT during a run, we raise NoResultError.
+        reactor = self.make_reactor()
+        spinner = self.make_spinner(reactor)
+        timeout = self.make_timeout()
+        reactor.callWhenRunning(os.kill, os.getpid(), signal.SIGINT)
+        self.assertRaises(
+            NoResultError, spinner.run, timeout * 5, defer.Deferred)
+        self.assertEqual([], spinner._clean())
+
+    def test_fast_sigint_raises_no_result_error_second_time(self):
+        self.test_fast_sigint_raises_no_result_error()
+
+
+def test_suite():
+    from unittest import TestLoader
+    return TestLoader().loadTestsFromName(__name__)

=== modified file 'testtools/tests/test_testresult.py'
--- testtools/tests/test_testresult.py	2010-10-14 22:50:38 +0000
+++ testtools/tests/test_testresult.py	2010-10-17 21:32:44 +0000
@@ -406,7 +406,7 @@
   File "...testtools...runtest.py", line ..., in _run_user...
     return fn(*args)
   File "...testtools...testcase.py", line ..., in _run_test_method
-    testMethod()
+    return self._get_test_method()()
   File "...testtools...tests...test_testresult.py", line ..., in error
     1/0
 ZeroDivisionError:... divi... by zero...
@@ -420,7 +420,7 @@
   File "...testtools...runtest.py", line ..., in _run_user...
     return fn(*args)
   File "...testtools...testcase.py", line ..., in _run_test_method
-    testMethod()
+    return self._get_test_method()()
   File "...testtools...tests...test_testresult.py", line ..., in failed
     self.fail("yo!")
 AssertionError: yo!