Merge lp:~gz/testtools/unicode_tracebacks_501166 into lp:~testtools-committers/testtools/trunk

[Martin Packman]

Proposed just to get this on the radar and solicit some early feedback. Needs a fair bit more work (and testing, I haven't looked at Python 3 yet) before it could be merged. Am happy to be told to move around code, rewrite tests, whatever.

As described in the bug, testtools on Python 2 currently falls over if any part of a traceback is not ascii. 90% of the problem is just the stringification of exception instances, but to be properly robust against funny inputs other elements of the traceback need to be correctly decoded as well.

The aim of the branch is to swap out the TestResult traceback formatting code on Python 2 only, and make it behave pretty much as Python 3 already does. I did have some additional changes with added improvements to the presentation, but have removed most of them for the moment.

Problem code is scattered across unittest, traceback and lineache which leaves a choice of three unpalatable options:
* Duplicating a lot of code from the standard library to change a few lines
* Invasive monkey patching of core modules
* Crazy and not terribly succinct hackery
I have avoided monkey patching, but much of the added code is not very nice.

Brief overview of the diff:

<testtools/content.py>
Make sure unicode route is used (needing to create a dummy TestResult instance here feels like an abstraction leak)

<testtools/run.py>
Make stub runner use unicode route

<testtools/testresult/real.py>
On Python implementations where str is bytes make traceback formatting use a unicode returning alternative to the standard traceback functions, and a couple of other changes to stay on unicode route

<testtools/utils.py>
Substantive changes here. The various new functions essentially only matter for Python 2 so seemed next to _u and _b might be the right place.
* unicode_output_stream - helper to make it easy to treat a stream as writing unicode.
* _detect_encoding and _get_source_encoding - handling Python source encoding.
* _get_exception_encoding and _exception_to_text - stringifying exception instances.
* _format_exc_info - overall traceback and exception formatting as unicode.

<testtools/tests/__init__.py>
Add new test module to suite.

<testtools/tests/test_testresult.py>
End-to-end testing of encoding handling and formatting. Rather ad-hoc setup at the moment, but the intention in the test cases themselves is about right.

<testtools/tests/test_utils.py>
Unit testing of added functions in utils that don't depend too much on the environment. Currently just the decoding of lines from Python source files. The other functions could be tested here, but because the type and how it is handled throughout the package matters, they all need to be tested as a whole anyway.

[Robert Collins (lifeless)]

Thank you very much for doing this! I appreciate its early days, so I'll simply brain dump what I see and we can filter it in future passes.

I think

38 - result = TextTestResult(sys.stdout)
39 + result = TextTestResult(unicode_output_stream(sys.stdout))
40 result.startTestRun()

Needs a comment (such as 'the testtools TextTestResult outputs unicode, and sys.stdout is usually a bytestream not an encoded stream'). Writing this makes me think 'why doesn't TextTestResult take care of this for users of the API ?

I don't really like testtools.utils; I'd consider putting stuff in small dedicated modules rather than a catchall; not mandatory - what do you think? E.g. we could move _u and _b to testtools.sequencetypes. Or something.

+ if sys.version_info > (3, 0) or sys.platform == "cli":
This is repeated; helper function time.

+ # Python 3 and IronPython strings are unicode, use parent class method
Perhaps "traceback strings are unicode, ..."

The deep magic is a little mind bending at first encounter, perhaps a couple of hints in there to aid understanding in new readers. Also, if it was a helper you wouldn't need the manual del, which is otherwise pretty opaque.

Minor nit: PEP8 got changed strangely; please don't use leading empty lines in docstrings.
+class TestNonAsciiResults(TestCase):
+ """
+ Test all kinds of tracebacks are cleanly interpreted as unicode
+
->
+class TestNonAsciiResults(TestCase):
+ """Test all kinds of tracebacks are cleanly interpreted as unicode

The _run helper you have suggests the api for testtools.run isn't powerful enough/easy enough yet. Please consider improving that rather than reimplementing (even though its tiny code-wise, principle applies here)

_run_external_case wants to be two things:
 - create a module returning info on it with a cleanup method
 - use that info to run the test

whats different between rmtempdir and shutil.rmtree(ignore_errors=True) ?

rather than using a list of sample texts, consider subclasses that set _sample_text - this is closer to using multiply_tests, and gives cleaner info about things that go wrong as well as simpler test code (no loops needed). Or you could even just use testscenarios and multiply_tests.

I agree about better fidelity on the testing; consider an RE Matcher, or something along those lines.

Please file a bug about this - it seems unrelated to unicode:
# GZ 2010-05-25: Seems this breaks testtools internals.

Rather than assertIn, the DocTestMatches might be closer fidelity, or make an RE or similar one.

Only one line of VWS here please (PEP8 :P)
397 +import traceback
398 +
399 +
400 +import testtools

TestGetSourceEncoding would benefit from being able to create a module as per the split up run helper suggested above.

And to come back to utils - it really likes like we want an encodings or stringencodings or something module. If thats all that is in utils at the moment, just rename utils; otherwise put your new stuff in a different file. Or whatever feels right.

This looks pretty good; I'd be happy to land it with the above things done - its nearly *there*.

Thanks again,
Rob

review: Needs Fixing

[Martin Packman (gz)]

> I think
>
> 38 - result = TextTestResult(sys.stdout)
> 39 + result = TextTestResult(unicode_output_stream(sys.stdout))
> 40 result.startTestRun()
>
> Needs a comment (such as 'the testtools TextTestResult outputs unicode, and
> sys.stdout is usually a bytestream not an encoded stream'). Writing this makes
> me think 'why doesn't TextTestResult take care of this for users of the API ?

I think this is a weakness with the Runner code really, it's rather light on useful logic, and half the module is under `if __name__ == '__main__':` so none of it's reusable. And there's still bug 501174 too.

> I don't really like testtools.utils; I'd consider putting stuff in small
> dedicated modules rather than a catchall; not mandatory - what do you think?
> E.g. we could move _u and _b to testtools.sequencetypes. Or something.

Well, these are mostly compatibility hacks rather than useful tools, and a bunch of them should never run on some versions of Python. A better name than 'utils' would be good.

> + if sys.version_info > (3, 0) or sys.platform == "cli":
> This is repeated; helper function time.

I'm still not sure on the best spelling of this check, it might be more readable as `if str is unicode:` but I need to check all of these conditionals actually do the right thing with Python 3 and the 2to3 conversion script.

> + # Python 3 and IronPython strings are unicode, use parent class
> method
> Perhaps "traceback strings are unicode, ..."

Well, as in the str type is unicode, but yeah, this cares about what the traceback module returns.

> The deep magic is a little mind bending at first encounter, perhaps a couple
> of hints in there to aid understanding in new readers. Also, if it was a
> helper you wouldn't need the manual del, which is otherwise pretty opaque.

It's unfortunately not the kind of thing that's reusable, so a helper may not help much. The good news is if we do change the stack stripping logic we'll be ripping this out anyway.

Perhaps just copying the function straight from unittest would be less painful anyway. One of the things I thought might need checking is what is okay to copy things into the MIT-licensed testtools from unittest and the rest of the Python standard library. Writing this sort of patch means the code is inevitably some sort of derivative, even without verbatim copying of functions.

> The _run helper you have suggests the api for testtools.run isn't powerful
> enough/easy enough yet. Please consider improving that rather than
> reimplementing (even though its tiny code-wise, principle applies here)

Yep. Was trying to avoid rolling that issue into this one though, as it felt more rewrite to me than fix.

> _run_external_case wants to be two things:
> - create a module returning info on it with a cleanup method
> - use that info to run the test

Yeah, wants teasing out into a couple of smaller, more logical helpers.

> whats different between rmtempdir and shutil.rmtree(ignore_errors=True) ?

Nothing useful currently, I wrote it because I thought I'd have to work around a problem with unicode on byte based filesystems (as per posix) and bytes on unicode based f...

[Robert Collins (lifeless)]

briefly, and I know I'm missing bits.

grab me on IRC for the matcher

yes, split utils into the iter and the encoding stuff

For the rest, do what you think makes the most sense ;)

-Rob

[lp:~gz/testtools/unicode_tracebacks_501166]

67. By Martin Packman on 2010-06-04

PEP 8 style fixes raised in review

68. By Martin Packman on 2010-06-11

Fix a bunch of boneheaded errors in TestNonAsciiResults

69. By Martin Packman on 2010-06-11

Merge trunk to pick up fixes for IronPython and Python 3

70. By Martin Packman on 2010-06-14

Changes to make tests work on Python 3, and a short term hack to make output robust

71. By Martin Packman on 2010-06-14

Make final problem test for Python 3 run rather than skip

72. By Martin Packman on 2010-06-14

Rearrange some TestNonAsciiResults helpers

73. By Martin Packman on 2010-06-14

Use flag in utils to record whether the str type is unicode rather than checking version and platform in multiple places

74. By Martin Packman on 2010-06-14

Work around a couple of quirks in non-mainstream Python implementations

75. By Martin Packman on 2010-06-15

Revert work around for IronPython coding declaration as it breaks Python 2

76. By Martin Packman on 2010-06-15

Poke encoding specific code in TestNonAsciiResults some more

77. By Martin Packman on 2010-06-15

Fix test_traceback_rechecks_encoding which got broken when adding Python 3 support

78. By Martin Packman on 2010-06-15

Found `ascii` function in Python 3 which saves some reimplementation

79. By Martin Packman on 2010-06-15

Unbreak test_non_ascii_dirname from IronPython returning filesystem encoding as None

80. By Martin Packman on 2010-06-15

Revert no longer needed cosmetic changes to test_content

81. By Martin Packman on 2010-06-15

Sanitise unicode_output_stream a bit and add some tests, needs more work

82. By Martin Packman on 2010-06-18

Make Jython skip unicode filesystem tests correctly again

83. By Martin Packman on 2010-06-18

Fix and test a bug with encoding of SyntaxError lines

84. By Martin Packman on 2010-06-18

Move most of utils module to compat and put iterate_tests in testsuite module

85. By Martin Packman on 2010-06-20

Trivial issues seen when scanning the diff for review

86. By Martin Packman on 2010-06-20

getlocale can return None if no locale is set, default to ascii

87. By Martin Packman on 2010-06-20

Allow detect encoding tests to work when a Python implementation may object to compiling them

88. By Martin Packman on 2010-06-20

Allow for mangling in shift_jis test if the exception encoding is incompatible

89. By Martin Packman on 2010-06-20

Document in tests the problems caused by CPython patch for issue #1031213

90. By Martin Packman on 2010-06-22

Avoid problem introduced with fix to Python bug #1031213 by ignoring the SyntaxError line and rereading the file instead

91. By Martin Packman on 2010-06-22

Change a test from using iso-8859-7 to iso-8859-5 which doesn't happen to have the test character on the same codepoint as iso-8859-1

92. By Martin Packman on 2010-06-22

Don't encode output as UTF-8 for TestNonAsciiResults

93. By Martin Packman on 2010-06-22

Make StringException.__str__ return UTF-8 rather than ascii/replace to prepare for merge

94. By Martin Packman on 2010-06-22

Merge trunk to resolve conflict in _StringException stringifying methods

95. By Martin Packman on 2010-06-22

Add NEWS for unicode tracebacks on Python 2

96. By Martin Packman on 2010-06-22

Unbreak test_assertion_text_shift_jis for Pythons with a unicode str type

97. By Martin Packman on 2010-06-22

Move Python 2 _StringException argument type check to __init__ from stringify methods

Unmerged revisions

=== modified file 'testtools/__init__.py'
--- testtools/__init__.py	2010-05-13 12:39:18 +0000
+++ testtools/__init__.py	2010-06-18 21:35:41 +0000
@@ -40,8 +40,8 @@
     )
 from testtools.testsuite import (
     ConcurrentTestSuite,
+    iterate_tests,
     )
-from testtools.utils import iterate_tests
 
 # same format as sys.version_info: "A tuple containing the five components of
 # the version number: major, minor, micro, releaselevel, and serial. All

=== renamed file 'testtools/utils.py' => 'testtools/compat.py'
--- testtools/utils.py	2010-06-11 00:04:30 +0000
+++ testtools/compat.py	2010-06-18 21:35:41 +0000
@@ -1,13 +1,21 @@
-# Copyright (c) 2008 Jonathan M. Lange. See LICENSE for details.
+# Copyright (c) 2008-2010 testtools developers. See LICENSE for details.
 
 """Utilities for dealing with stuff in unittest."""
 
 
+import codecs
+import linecache
+import locale
+import os
+import re
 import sys
+import traceback
 
 __metaclass__ = type
 __all__ = [
-    'iterate_tests',
+    'advance_iterator',
+    'str_is_unicode',
+    'unicode_output_stream',
     ]
 
 
@@ -15,6 +23,7 @@
     def _u(s):
         """Replacement for u'some string' in Python 3."""
         return s
+    _r = ascii
     def _b(s):
         """A byte literal."""
         return s.encode("latin-1")
@@ -23,9 +32,13 @@
         return isinstance(x, str)
     def classtypes():
         return (type,)
+    str_is_unicode = True
 else:
     def _u(s):
-        return unicode(s, "latin-1")
+        """Use _u('\u1234') over u'\u1234' to avoid Python 3 syntax error"""        
+        return (s.replace("\\", "\\\\").replace("\\\\u", "\\u")
+            .replace("\\\\U", "\\U").decode("unicode-escape"))
+    _r = repr
     def _b(s):
         return s
     advance_iterator = lambda it: it.next()
@@ -34,15 +47,173 @@
     def classtypes():
         import types
         return (type, types.ClassType)
-
-
-def iterate_tests(test_suite_or_case):
-    """Iterate through all of the test cases in 'test_suite_or_case'."""
-    try:
-        suite = iter(test_suite_or_case)
-    except TypeError:
-        yield test_suite_or_case
-    else:
-        for test in suite:
-            for subtest in iterate_tests(test):
-                yield subtest
+    str_is_unicode = sys.platform == "cli"
+
+
+def unicode_output_stream(stream):
+    """Get wrapper for given stream that writes any unicode without exception
+
+    Characters that can't be coerced to the encoding of the stream, or 'ascii'
+    if valid encoding is not found, will be replaced. The original stream may
+    be returned in situations where a wrapper is determined unneeded.
+
+    The wrapper only allows unicode to be written, not non-ascii bytestrings,
+    which is a good thing to ensure sanity and sanitation.
+    """
+    if sys.platform == "cli":
+        # Best to never encode before writing in IronPython
+        return stream
+    try:
+        writer = codecs.getwriter(stream.encoding or "")
+    except (AttributeError, LookupError):
+        # GZ 2010-06-16: Python 3 StringIO ends up here, but probably needs
+        #                different handling as it doesn't want bytestrings
+        return codecs.getwriter("ascii")(stream, "replace")
+    if writer.__module__.rsplit(".", 1)[1].startswith("utf"):
+        # The current stream has a unicode encoding so no error handler is needed
+        return stream
+    if sys.version_info > (3, 0):
+        # Python 3 doesn't seem to make this easy, handle a common case
+        try:
+            return stream.__class__(stream.buffer, stream.encoding, "replace",
+                stream.newlines, stream.line_buffering)
+        except AttributeError:
+            pass
+    return writer(stream, "replace")    
+
+
+# The default source encoding is actually "iso-8859-1" until Python 2.5 but
+# using non-ascii causes a deprecation warning in 2.4 and it's cleaner to
+# treat all versions the same way
+_default_source_encoding = "ascii"
+
+# Pattern specified in <http://www.python.org/dev/peps/pep-0263/>
+_cookie_search=re.compile("coding[:=]\s*([-\w.]+)").search
+
+def _detect_encoding(lines):
+    """Get the encoding of a Python source file from a list of lines as bytes
+
+    This function does less than tokenize.detect_encoding added in Python 3 as
+    it does not attempt to raise a SyntaxError when the interpreter would, it
+    just wants the encoding of a source file Python has already compiled and
+    determined is valid.
+    """
+    if not lines:
+        return _default_source_encoding
+    if lines[0].startswith("\xef\xbb\xbf"):
+        # Source starting with UTF-8 BOM is either UTF-8 or a SyntaxError
+        return "utf-8"
+    # Only the first two lines of the source file are examined
+    magic = _cookie_search("".join(lines[:2]))
+    if magic is None:
+        return _default_source_encoding
+    encoding = magic.group(1)
+    try:
+        codecs.lookup(encoding)
+    except LookupError:
+        # Some codecs raise something other than LookupError if they don't
+        # support the given error handler, but not the text ones that could
+        # actually be used for Python source code
+        return _default_source_encoding
+    return encoding
+
+
+class _EncodingTuple(tuple):
+    """A tuple type that can have an encoding attribute smuggled on"""
+
+
+def _get_source_encoding(filename):
+    """Detect, cache and return the encoding of Python source at filename"""
+    try:
+        return linecache.cache[filename].encoding
+    except (AttributeError, KeyError):
+        encoding = _detect_encoding(linecache.getlines(filename))
+        if filename in linecache.cache:
+            newtuple = _EncodingTuple(linecache.cache[filename])
+            newtuple.encoding = encoding
+            linecache.cache[filename] = newtuple
+        return encoding
+
+def _get_exception_encoding():
+    """Return the encoding we expect messages from the OS to be encoded in"""
+    if os.name == "nt":
+        # GZ 2010-05-24: Really want the codepage number instead, the error
+        #                handling of standard codecs is more deterministic
+        return "mbcs"
+    # GZ 2010-05-23: We need this call to be after initialisation, but there's
+    #                no benefit in asking more than once as it's a global
+    #                setting that can change after the message is formatted.
+    return locale.getlocale(locale.LC_MESSAGES)[1]
+
+def _exception_to_text(evalue):
+    """Try hard to get a sensible text value out of an exception instance"""
+    try:
+        return unicode(evalue)
+    except KeyboardInterrupt:
+        raise
+    except:
+        pass
+    try:
+        return str(evalue).decode(_get_exception_encoding(), "replace")
+    except KeyboardInterrupt:
+        raise
+    except:
+        pass
+    # Okay, out of ideas, let higher level handle it
+    return None
+
+# GZ 2010-05-23: This function is huge and horrible and I welcome suggestions
+#                on the best way to break it up
+def _format_exc_info(eclass, evalue, tb, limit=None):
+    """Format a stack trace and the exception information as unicode
+
+    Compatibility function for Python 2 which ensures each component of a
+    traceback is correctly decoded according to its origins.
+
+    Based on traceback.format_exception and related functions.
+    """
+    fs_enc = sys.getfilesystemencoding()
+    if tb:
+        list = ['Traceback (most recent call last):\n']
+        extracted_list = []
+        for filename, lineno, name, line in traceback.extract_tb(tb, limit):
+            extracted_list.append((
+                filename.decode(fs_enc, "replace"),
+                lineno,
+                name.decode("ascii", "replace"),
+                line.decode(_get_source_encoding(filename), "replace")))
+        list.extend(traceback.format_list(extracted_list))
+    else:
+        list = []
+    if evalue is None:
+        # Is a (deprecated) string exception
+        list.append(sclass.decode("ascii", "replace"))
+    elif isinstance(evalue, SyntaxError) and len(evalue.args) > 1:
+        # Avoid duplicating the special formatting for SyntaxError here,
+        # instead create a new instance with unicode filename and line
+        # Potentially gives duff spacing, but that's a pre-existing issue
+        filename, lineno, offset, line = evalue.args[1]
+        if filename:
+            filename = filename.decode(fs_enc, "replace")
+        if line:
+            # Errors during parsing give the line from buffer encoded as
+            # latin-1 if the given coding or utf-8 for all other codings
+            # Can't know which was used, so just try utf-8 first
+            try:
+                line = line.decode("utf-8")
+            except UnicodeDecodeError:
+                line = line.decode("latin-1")
+        evalue = eclass(evalue.args[0], (filename, lineno, offset, line))
+        list.extend(traceback.format_exception_only(eclass, evalue))
+    else:
+        sclass = eclass.__name__
+        svalue = _exception_to_text(evalue)
+        if svalue:
+            list.append("%s: %s\n" % (sclass, svalue))
+        elif svalue is None:
+            # GZ 2010-05-24: Not a great fallback message, but keep for the
+            #                the same for compatibility for the moment
+            list.append("%s: <unprintable %s object>\n" % (sclass, sclass))
+        else:
+            list.append("%s\n" % sclass)
+    return list

=== modified file 'testtools/content.py'
--- testtools/content.py	2010-01-16 01:31:27 +0000
+++ testtools/content.py	2010-06-18 21:35:41 +0000
@@ -3,10 +3,10 @@
 """Content - a MIME-like Content object."""
 
 import codecs
-from unittest import TestResult
 
+from testtools.testresult import TestResult
+from testtools.compat import _b
 from testtools.content_type import ContentType
-from testtools.utils import _b
 
 
 class Content(object):
@@ -86,6 +86,6 @@
         content_type = ContentType('text', 'x-traceback',
             {"language": "python", "charset": "utf8"})
         self._result = TestResult()
-        value = self._result._exc_info_to_string(err, test)
+        value = self._result._exc_info_to_unicode(err, test)
         super(TracebackContent, self).__init__(
             content_type, lambda: [value.encode("utf8")])

=== modified file 'testtools/run.py'
--- testtools/run.py	2010-06-13 22:12:17 +0000
+++ testtools/run.py	2010-06-18 21:35:41 +0000
@@ -13,7 +13,7 @@
 import sys
 
 from testtools import TextTestResult
-from testtools.utils import classtypes, istext
+from testtools.compat import classtypes, istext, unicode_output_stream
 
 
 defaultTestLoader = unittest.defaultTestLoader
@@ -36,7 +36,7 @@
 
     def run(self, test):
         "Run the given test case or test suite."
-        result = TextTestResult(sys.stdout)
+        result = TextTestResult(unicode_output_stream(sys.stdout))
         result.startTestRun()
         try:
             return test.run(result)

=== modified file 'testtools/testcase.py'
--- testtools/testcase.py	2010-06-13 05:15:20 +0000
+++ testtools/testcase.py	2010-06-18 21:35:41 +0000
@@ -22,9 +22,9 @@
 import unittest
 
 from testtools import content
+from testtools.compat import advance_iterator
 from testtools.runtest import RunTest
 from testtools.testresult import TestResult
-from testtools.utils import advance_iterator
 
 
 try:

=== modified file 'testtools/testresult/real.py'
--- testtools/testresult/real.py	2010-06-11 01:08:23 +0000
+++ testtools/testresult/real.py	2010-06-18 21:35:41 +0000
@@ -11,8 +11,11 @@
     ]
 
 import datetime
+import sys
 import unittest
 
+from testtools.compat import _format_exc_info, str_is_unicode
+
 
 class TestResult(unittest.TestResult):
     """Subclass of unittest.TestResult extending the protocol for flexability.
@@ -105,10 +108,27 @@
         """Called when a test was expected to fail, but succeed."""
         self.unexpectedSuccesses.append(test)
 
+    if str_is_unicode:
+        # Python 3 and IronPython strings are unicode, use parent class method
+        _exc_info_to_unicode = unittest.TestResult._exc_info_to_string
+    else:
+        # For Python 2, need to decode components of traceback according to
+        # their source, so can't use traceback.format_exception
+        # Here follows a little deep magic to copy the existing method and
+        # replace the formatter with one that returns unicode instead
+        from types import FunctionType as __F, ModuleType as __M
+        __f = unittest.TestResult._exc_info_to_string.im_func
+        __g = dict(__f.func_globals)
+        __m = __M("__fake_traceback")
+        __m.format_exception = _format_exc_info
+        __g["traceback"] = __m
+        _exc_info_to_unicode = __F(__f.func_code, __g, "_exc_info_to_unicode")
+        del __F, __M, __f, __g, __m
+
     def _err_details_to_string(self, test, err=None, details=None):
         """Convert an error in exc_info form or a contents dict to a string."""
         if err is not None:
-            return self._exc_info_to_string(err, test)
+            return self._exc_info_to_unicode(err, test)
         return _details_to_str(details)
 
     def _now(self):
@@ -511,6 +531,7 @@
     def __hash__(self):
         return id(self)
 
+<<<<<<< TREE
     def __str__(self):
         """Stringify better than 2.x's default behaviour of ascii encoding."""
         if type(self.args[0]) is str:
@@ -521,6 +542,14 @@
         if type(self.args[0]) is str:
             return self.args[0].decode('utf8')
         return self.args[0]
+=======
+    if not str_is_unicode:
+        def __str__(self):
+            return self.args[0].encode("ascii", "replace")
+
+        def __unicode__(self):
+            return self.args[0]
+>>>>>>> MERGE-SOURCE
 
     def __eq__(self, other):
         try:

=== modified file 'testtools/tests/__init__.py'
--- testtools/tests/__init__.py	2010-01-16 01:31:27 +0000
+++ testtools/tests/__init__.py	2010-06-18 21:35:41 +0000
@@ -4,6 +4,7 @@
 
 import unittest
 from testtools.tests import (
+    test_compat,
     test_content,
     test_content_type,
     test_matchers,
@@ -17,6 +18,7 @@
 def test_suite():
     suites = []
     modules = [
+        test_compat,
         test_content,
         test_content_type,
         test_matchers,

=== added file 'testtools/tests/test_compat.py'
--- testtools/tests/test_compat.py	1970-01-01 00:00:00 +0000
+++ testtools/tests/test_compat.py	2010-06-18 21:35:41 +0000
@@ -0,0 +1,239 @@
+# Copyright (c) 2010 testtools developers. See LICENSE for details.
+
+"""Tests for miscellaneous compatibility functions"""
+
+import linecache
+import os
+import sys
+import tempfile
+import traceback
+
+import testtools
+
+from testtools.compat import (
+    _b,
+    _detect_encoding,
+    _get_source_encoding,
+    _u,
+    unicode_output_stream,
+    )
+
+
+class TestDetectEncoding(testtools.TestCase):
+    """Test detection of Python source encodings"""
+
+    def _check_encoding(self, expected, lines):
+        """Check lines are valid Python and encoding is as expected"""
+        compile(_b("".join(lines)), "<str>", "exec")
+        encoding = _detect_encoding(lines)
+        self.assertEqual(expected, encoding,
+            "Encoding %r expected but got %r from lines %r" %
+                (expected, encoding, lines))
+
+    def test_examples_from_pep(self):
+        """Check the examples given in PEP 263 all work as specified
+
+        See 'Examples' section of <http://www.python.org/dev/peps/pep-0263/>
+        """
+        # With interpreter binary and using Emacs style file encoding comment:
+        self._check_encoding("latin-1", (
+            "#!/usr/bin/python\n",
+            "# -*- coding: latin-1 -*-\n",
+            "import os, sys\n"))
+        self._check_encoding("iso-8859-15", (
+            "#!/usr/bin/python\n",
+            "# -*- coding: iso-8859-15 -*-\n",
+            "import os, sys\n"))
+        self._check_encoding("ascii", (
+            "#!/usr/bin/python\n",
+            "# -*- coding: ascii -*-\n",
+            "import os, sys\n"))
+        # Without interpreter line, using plain text:
+        self._check_encoding("utf-8", (
+            "# This Python file uses the following encoding: utf-8\n",
+            "import os, sys\n"))
+        # Text editors might have different ways of defining the file's
+        # encoding, e.g.
+        self._check_encoding("latin-1", (
+            "#!/usr/local/bin/python\n",
+            "# coding: latin-1\n",
+            "import os, sys\n"))
+        # Without encoding comment, Python's parser will assume ASCII text:
+        self._check_encoding("ascii", (
+            "#!/usr/local/bin/python\n",
+            "import os, sys\n"))
+        # Encoding comments which don't work:
+        #   Missing "coding:" prefix:
+        self._check_encoding("ascii", (
+            "#!/usr/local/bin/python\n",
+            "# latin-1\n",
+            "import os, sys\n"))
+        #   Encoding comment not on line 1 or 2:
+        self._check_encoding("ascii", (
+            "#!/usr/local/bin/python\n",
+            "#\n",
+            "# -*- coding: latin-1 -*-\n",
+            "import os, sys\n"))
+        #   Unsupported encoding:
+        #     Correct behaviour is a SyntaxError, which we aren't testing for
+        #     and fails instead with MemoryError on Python 2.4
+        # self._check_encoding("ascii", (
+        #     "#!/usr/local/bin/python\n",
+        #     "# -*- coding: utf-42 -*-\n",
+        #     "import os, sys\n"))
+
+    def test_bom(self):
+        """Test the UTF-8 BOM counts as an encoding declaration"""
+        self._check_encoding("utf-8", (
+            "\xef\xbb\xbfimport sys\n",
+            ))
+        self._check_encoding("utf-8", (
+            "\xef\xbb\xbf# File encoding: UTF-8\n",
+            ))
+        self._check_encoding("utf-8", (
+            '\xef\xbb\xbf"""Module docstring\n',
+            '\xef\xbb\xbfThat should just be a ZWNB"""\n'))
+        self._check_encoding("latin-1", (
+            '"""Is this coding: latin-1 or coding: utf-8 instead?\n',
+            '\xef\xbb\xbfThose should be latin-1 bytes"""\n'))
+        self._check_encoding("utf-8", (
+            "\xef\xbb\xbf# Is the coding: utf-8 or coding: euc-jp instead?\n",
+            '"""Module docstring say \xe2\x98\x86"""\n'))
+
+    def test_multiple_coding_comments(self):
+        """Test only the first of multiple coding declarations counts"""
+        self._check_encoding("iso-8859-1", (
+            "# Is the coding: iso-8859-1\n",
+            "# Or is it coding: iso-8859-2\n"))
+        self._check_encoding("iso-8859-1", (
+            "#!/usr/bin/python\n",
+            "# Is the coding: iso-8859-1\n",
+            "# Or is it coding: iso-8859-2\n"))
+        self._check_encoding("iso-8859-1", (
+            "# Is the coding: iso-8859-1 or coding: iso-8859-2\n",
+            "# Or coding: iso-8859-3 or coding: iso-8859-4\n"))
+        self._check_encoding("iso-8859-2", (
+            "# Is the coding iso-8859-1 or coding: iso-8859-2\n",
+            "# Spot the missing colon above\n"))
+
+
+class TestGetSourceEncoding(testtools.TestCase):
+    """Test reading and caching the encodings of source files"""
+
+    def setUp(self):
+        testtools.TestCase.setUp(self)
+        dir = tempfile.mkdtemp()
+        self.addCleanup(os.rmdir, dir)
+        self.filename = os.path.join(dir, self.id().rsplit(".", 1)[1] + ".py")
+        self._written = False
+
+    def put_source(self, text):
+        f = open(self.filename, "w")
+        try:
+            f.write(text)
+        finally:
+            f.close()
+            if not self._written:
+                self._written = True
+                self.addCleanup(os.remove, self.filename)
+                self.addCleanup(linecache.cache.pop, self.filename, None)
+
+    def test_nonexistant_file_as_ascii(self):
+        """When file can't be found, the encoding should default to ascii"""
+        self.assertEquals("ascii", _get_source_encoding(self.filename))
+
+    def test_encoding_is_cached(self):
+        """The encoding should stay the same if the cache isn't invalidated"""
+        self.put_source(
+            "# coding: iso-8859-13\n"
+            "import os\n")
+        self.assertEquals("iso-8859-13", _get_source_encoding(self.filename))
+        self.put_source(
+            "# coding: rot-13\n"
+            "vzcbeg bf\n")
+        self.assertEquals("iso-8859-13", _get_source_encoding(self.filename))
+
+    def test_traceback_rechecks_encoding(self):
+        """A traceback function checks the cache and resets the encoding"""
+        self.put_source(
+            "# coding: iso-8859-8\n"
+            "import os\n")
+        self.assertEquals("iso-8859-8", _get_source_encoding(self.filename))
+        self.put_source(
+            "# coding: utf-8\n"
+            "import os\n")
+        try:
+            exec (compile("raise RuntimeError\n", self.filename, "exec"))
+        except RuntimeError:
+            traceback.extract_tb(sys.exc_info()[2])
+        else:
+            self.fail("RuntimeError not raised")
+        self.assertEquals("utf-8", _get_source_encoding(self.filename))
+
+
+class _FakeOutputStream(object):
+    """A simple file-like object for testing"""
+    def __init__(self):
+        self.writelog = []
+    def write(self, obj):
+        self.writelog.append(obj)
+
+
+class TestUnicodeOutputStream(testtools.TestCase):
+    """Test wrapping output streams so they work with arbitrary unicode"""
+    uni = _u("pa\u026a\u03b8\u0259n")
+    def setUp(self):
+        super(TestUnicodeOutputStream, self).setUp()
+        if sys.platform == "cli":
+            self.skip("IronPython shouldn't wrap streams to do encoding")
+    def test_no_encoding_becomes_ascii(self):
+        """A stream with no encoding attribute gets ascii/replace strings"""
+        sout = _FakeOutputStream()
+        unicode_output_stream(sout).write(self.uni)
+        self.assertEqual([_b("pa???n")], sout.writelog)
+    def test_encoding_as_none_becomes_ascii(self):
+        """A stream with encoding value of None gets ascii/replace strings"""
+        sout = _FakeOutputStream()
+        sout.encoding = None
+        unicode_output_stream(sout).write(self.uni)
+        self.assertEqual([_b("pa???n")], sout.writelog)
+    def test_bogus_encoding_becomes_ascii(self):
+        """A stream with a bogus encoding gets ascii/replace strings"""
+        sout = _FakeOutputStream()
+        sout.encoding = "bogus"
+        unicode_output_stream(sout).write(self.uni)
+        self.assertEqual([_b("pa???n")], sout.writelog)
+    def test_partial_encoding_replace(self):
+        """A string which can be partly encoded correctly should be"""
+        sout = _FakeOutputStream()
+        sout.encoding = "iso-8859-7"
+        unicode_output_stream(sout).write(self.uni)
+        self.assertEqual([_b("pa?\xe8?n")], sout.writelog)
+    def test_unicode_encodings_not_wrapped(self):
+        """A unicode encoding is left unwrapped as needs no error handler"""
+        sout = _FakeOutputStream()
+        sout.encoding = "utf-8"
+        self.assertIs(unicode_output_stream(sout), sout)
+        sout = _FakeOutputStream()
+        sout.encoding = "utf-16-be"
+        self.assertIs(unicode_output_stream(sout), sout)
+    def test_stringio(self):
+        """A StringIO object should maybe get an ascii native str type"""
+        try:
+            from cStringIO import StringIO
+            newio = False
+        except ImportError:
+            from io import StringIO
+            newio = True
+        sout = StringIO()
+        soutwrapper = unicode_output_stream(sout)
+        if newio:
+            self.expectFailure("Python 3 StringIO expects bytes",
+                self.assertRaises, TypeError, soutwrapper.write, self.uni)
+        soutwrapper.write(self.uni)
+        self.assertEqual("pa???n", sout.getvalue())
+
+
+def test_suite():
+    from unittest import TestLoader
+    return TestLoader().loadTestsFromName(__name__)

=== modified file 'testtools/tests/test_content.py'
--- testtools/tests/test_content.py	2009-12-31 03:15:19 +0000
+++ testtools/tests/test_content.py	2010-06-18 21:35:41 +0000
@@ -1,9 +1,9 @@
 # Copyright (c) 2008 Jonathan M. Lange. See LICENSE for details.
 
 import unittest
+from testtools.compat import _u
 from testtools.content import Content, TracebackContent
 from testtools.content_type import ContentType
-from testtools.utils import _u
 from testtools.tests.helpers import an_exc_info
 
 

=== modified file 'testtools/tests/test_testresult.py'
--- testtools/tests/test_testresult.py	2010-06-10 19:00:59 +0000
+++ testtools/tests/test_testresult.py	2010-06-18 21:35:41 +0000
@@ -4,14 +4,19 @@
 
 __metaclass__ = type
 
+import codecs
 import datetime
 try:
     from cStringIO import StringIO
 except ImportError:
     from io import StringIO
 import doctest
+import os
+import shutil
 import sys
+import tempfile
 import threading
+import warnings
 
 from testtools import (
     ExtendedToOriginalDecorator,
@@ -22,9 +27,16 @@
     ThreadsafeForwardingResult,
     testresult,
     )
+from testtools.compat import (
+    _b,
+    _get_exception_encoding,
+    _r,
+    _u,
+    str_is_unicode,
+    unicode_output_stream,
+    )
 from testtools.content import Content, ContentType
 from testtools.matchers import DocTestMatches
-from testtools.utils import _u, _b
 from testtools.tests.helpers import (
     LoggingResult,
     Python26TestResult,
@@ -802,6 +814,278 @@
         self.assertEqual(2, self.converter.foo())
     
 
+class TestNonAsciiResults(TestCase):
+    """Test all kinds of tracebacks are cleanly interpreted as unicode
+
+    Currently only uses weak "contains" assertions, would be good to be much
+    stricter about the expected output. This would add a few failures for the
+    current release of IronPython for instance, which gets some traceback
+    lines muddled.
+    """
+
+    _sample_texts = (
+        _u("pa\u026a\u03b8\u0259n"), # Unicode encodings only
+        _u("\u5357\u7121"), # In ISO 2022 encodings
+        _u("\xa7\xa7\xa7"), # In ISO 8859 encodings
+        )
+    # Everything but Jython shows syntax errors on the current character
+    _error_on_character = os.name != "java"
+
+    def _run(self, stream, test):
+        """Run the test, the same as in testtools.run but not to stdout"""
+        if not str_is_unicode:
+            stream = codecs.getwriter("UTF-8")(stream)
+        result = TextTestResult(stream)
+        result.startTestRun()
+        try:
+            return test.run(result)
+        finally:
+            result.stopTestRun()
+
+    def _write_module(self, name, encoding, contents):
+        """Create Python module on disk with contents in given encoding"""
+        f = codecs.open(os.path.join(self.dir, name + ".py"), "w", encoding)
+        try:
+            f.write(contents)
+        finally:
+            f.close()
+
+    def _test_external_case(self, testline, coding="ascii", modulelevel="",
+            suffix=""):
+        """Create and run a test case in a seperate module"""
+        self._setup_external_case(testline, coding, modulelevel, suffix)
+        return self._run_external_case()
+
+    def _setup_external_case(self, testline, coding="ascii", modulelevel="",
+            suffix=""):
+        """Create a test case in a seperate module"""
+        _, prefix, self.modname = self.id().rsplit(".", 2)
+        self.dir = tempfile.mkdtemp(prefix=prefix, suffix=suffix)
+        self.addCleanup(shutil.rmtree, self.dir)
+        try:
+            # Need to pre-check that the coding is valid or codecs.open drops
+            # the file without closing it which breaks non-refcounted pythons
+            codecs.lookup(coding)
+        except LookupError:
+            self.skip("Encoding unsupported by implementation: %r" % coding)
+        self._write_module(self.modname, coding,
+            # Older Python 2 versions don't see a coding declaration in a
+            # docstring so it has to be in a comment, but then we can't
+            # workaround bug: <http://ironpython.codeplex.com/workitem/26940>
+            "# coding: %s\n"
+            "import testtools\n"
+            "%s\n"
+            "class Test(testtools.TestCase):\n"
+            "    def runTest(self):\n"
+            "        %s\n" % (coding, modulelevel, testline))
+
+    def _run_external_case(self):
+        """Run the prepared test case in a seperate module"""
+        sys.path.insert(0, self.dir)
+        self.addCleanup(sys.path.remove, self.dir)
+        module = __import__(self.modname)
+        self.addCleanup(sys.modules.pop, self.modname)
+        stream = StringIO()
+        self._run(stream, module.Test())
+        return stream.getvalue()
+
+    def _silence_deprecation_warnings(self):
+        """Shut up DeprecationWarning for this test only"""
+        warnings.simplefilter("ignore", DeprecationWarning)
+        self.addCleanup(warnings.filters.remove, warnings.filters[0])
+
+    def _get_sample_text(self, encoding="unicode_internal"):
+        if encoding is None and str_is_unicode:
+           encoding = "unicode_internal"
+        for u in self._sample_texts:
+            try:
+                b = u.encode(encoding)
+                if u == b.decode(encoding):
+                   if str_is_unicode:
+                       return u, u
+                   return u, b
+            except (LookupError, UnicodeError):
+                pass
+        self.skip("Could not find a sample text for encoding: %r" % encoding)
+
+    def _as_output(self, text):
+        if str_is_unicode:
+            return text
+        return text.encode("UTF-8")
+
+    def test_non_ascii_failure_string(self):
+        """Assertion contents can be non-ascii and should get decoded"""
+        text, raw = self._get_sample_text(_get_exception_encoding())
+        textoutput = self._test_external_case("self.fail(%s)" % _r(raw))
+        self.assertIn(self._as_output(text), textoutput)
+
+    def test_control_characters_in_failure_string(self):
+        """Control characters in assertions should be escaped"""
+        textoutput = self._test_external_case("self.fail('\\a\\a\\a')")
+        self.expectFailure("Defense against the beeping horror unimplemented",
+            self.assertNotIn, self._as_output("\a\a\a"), textoutput)
+        self.assertIn(self._as_output(_u("\uFFFD\uFFFD\uFFFD")), textoutput)
+
+    def test_os_error(self):
+        """Locale error messages from the OS shouldn't break anything"""
+        textoutput = self._test_external_case(
+            modulelevel="import os",
+            testline="os.mkdir('/')")
+        if os.name != "nt" or sys.version_info < (2, 5):
+            self.assertIn(self._as_output("OSError: "), textoutput)
+        else:
+            self.assertIn(self._as_output("WindowsError: "), textoutput)
+
+    def test_assertion_text_shift_jis(self):
+        """A terminal raw backslash in an encoded string is weird but fine"""
+        example_text = _u("\u5341")
+        textoutput = self._test_external_case(
+            coding="shift_jis",
+            testline="self.fail('%s')" % example_text)
+        self.assertIn(self._as_output("AssertionError: %s" % example_text),
+            textoutput)
+
+    def test_file_comment_iso2022_jp(self):
+        """Control character escapes must be preserved if valid encoding"""
+        example_text, _ = self._get_sample_text("iso2022_jp")
+        textoutput = self._test_external_case(
+            coding="iso2022_jp",
+            testline="self.fail('Simple') # %s" % example_text)
+        self.assertIn(self._as_output(example_text), textoutput)
+
+    def test_unicode_exception(self):
+        """Exceptions that can be formated losslessly as unicode should be"""
+        example_text, _ = self._get_sample_text()
+        exception_class = (
+            "class FancyError(Exception):\n"
+            # A __unicode__ method does nothing on py3k but the default works
+            "    def __unicode__(self):\n"
+            "        return self.args[0]\n")
+        textoutput = self._test_external_case(
+            modulelevel=exception_class,
+            testline="raise FancyError(%s)" % _r(example_text))
+        self.assertIn(self._as_output(example_text), textoutput)
+
+    def test_unprintable_exception(self):
+        """A totally useless exception instance still prints something"""
+        exception_class = (
+            "class UnprintableError(Exception):\n"
+            "    def __str__(self):\n"
+            "        raise RuntimeError\n"
+            "    def __repr__(self):\n"
+            "        raise RuntimeError\n")
+        textoutput = self._test_external_case(
+            modulelevel=exception_class,
+            testline="raise UnprintableError")
+        self.assertIn(self._as_output(
+            "UnprintableError: <unprintable UnprintableError object>\n"),
+            textoutput)
+
+    # GZ 2010-05-25: Seems this breaks testtools internals.
+    def _disabled_test_string_exception(self):
+        """Raise a string rather than an exception instance if supported"""
+        if sys.version_info > (2, 6):
+            self.skip("No string exceptions in Python 2.6 or later")
+        elif sys.version_info > (2, 5):
+            self._silence_deprecation_warnings()
+        textoutput = self._test_external_case(testline="raise 'plain str'")
+        self.assertIn(self._as_output("\nplain str\n"), textoutput)
+
+    def test_non_ascii_dirname(self):
+        """Script paths in the traceback can be non-ascii"""
+        text, raw = self._get_sample_text(sys.getfilesystemencoding())
+        textoutput = self._test_external_case(
+            # Avoid bug in Python 3 by giving a unicode source encoding rather
+            # than just ascii which raises a SyntaxError with no other details
+            coding="utf-8",
+            testline="self.fail('Simple')",
+            suffix=raw)
+        self.assertIn(self._as_output(text), textoutput)
+
+    def test_syntax_error(self):
+        """Syntax errors should still have fancy special-case formatting"""
+        textoutput = self._test_external_case("exec ('f(a, b c)')")
+        self.assertIn(self._as_output(
+            '  File "<string>", line 1\n'
+            '    f(a, b c)\n'
+            + ' ' * self._error_on_character +
+            '          ^\n'
+            'SyntaxError: '
+            ), textoutput)
+
+    def test_syntax_error_import_binary(self):
+        """Importing a binary file shouldn't break SyntaxError formatting"""
+        if sys.version_info < (2, 5):
+            # Python 2.4 assumes the file is latin-1 and tells you off
+            self._silence_deprecation_warnings()
+        self._setup_external_case("import bad")
+        f = open(os.path.join(self.dir, "bad.py"), "wb")
+        try:
+            f.write(_b("x\x9c\xcb*\xcd\xcb\x06\x00\x04R\x01\xb9"))
+        finally:
+            f.close()
+        textoutput = self._run_external_case()
+        self.assertIn(self._as_output("\nSyntaxError: "), textoutput)
+
+    def test_syntax_error_line_iso_8859_1(self):
+        """Syntax error on a latin-1 line shows the line decoded"""
+        text, raw = self._get_sample_text("iso-8859-1")
+        textoutput = self._setup_external_case("import bad")
+        self._write_module("bad", "iso-8859-1",
+            "# coding: iso-8859-1\n$ = 0 # %s\n" % text)
+        textoutput = self._run_external_case()
+        self.assertIn(self._as_output(_u(
+            #'bad.py", line 2\n'
+            '    $ = 0 # %s\n'
+            + ' ' * self._error_on_character +
+            '   ^\n'
+            'SyntaxError: ') %
+            (text,)), textoutput)
+
+    def test_syntax_error_line_iso_8859_7(self):
+        """Syntax error on a iso-8859-7 line shows the line decoded"""
+        text, raw = self._get_sample_text("iso-8859-7")
+        textoutput = self._setup_external_case("import bad")
+        self._write_module("bad", "iso-8859-7",
+            "# coding: iso-8859-7\n%% = 0 # %s\n" % text)
+        textoutput = self._run_external_case()
+        self.assertIn(self._as_output(_u(
+            #'bad.py", line 2\n'
+            '    %% = 0 # %s\n'
+            + ' ' * self._error_on_character +
+            '   ^\n'
+            'SyntaxError: ') %
+            (text,)), textoutput)
+
+    def test_syntax_error_line_utf_8(self):
+        """Syntax error on a utf-8 line shows the line decoded"""
+        text, raw = self._get_sample_text("utf-8")
+        textoutput = self._setup_external_case("import bad")
+        self._write_module("bad", "utf-8", _u("\ufeff^ = 0 # %s\n") % text)
+        textoutput = self._run_external_case()
+        self.assertIn(self._as_output(_u(
+            'bad.py", line 1\n'
+            '    ^ = 0 # %s\n'
+            + ' ' * self._error_on_character +
+            '   ^\n'
+            'SyntaxError: ') %
+            text), textoutput)
+
+
+class TestNonAsciiResultsWithUnittest(TestNonAsciiResults):
+    """Test that running under unittest produces clean ascii strings"""
+
+    from unittest import TextTestRunner as _Runner
+
+    def _run(self, stream, test):
+        return self._Runner(stream).run(test)
+
+    def _as_output(self, text):
+        if str_is_unicode:
+            return text
+        return text.encode("ascii", "replace")
+
+
 def test_suite():
     from unittest import TestLoader
     return TestLoader().loadTestsFromName(__name__)

=== modified file 'testtools/testsuite.py'
--- testtools/testsuite.py	2010-06-10 19:01:06 +0000
+++ testtools/testsuite.py	2010-06-18 21:35:41 +0000
@@ -5,6 +5,7 @@
 __metaclass__ = type
 __all__ = [
   'ConcurrentTestSuite',
+  'iterate_tests',
   ]
 
 try:
@@ -17,6 +18,18 @@
 import testtools
 
 
+def iterate_tests(test_suite_or_case):
+    """Iterate through all of the test cases in 'test_suite_or_case'."""
+    try:
+        suite = iter(test_suite_or_case)
+    except TypeError:
+        yield test_suite_or_case
+    else:
+        for test in suite:
+            for subtest in iterate_tests(test):
+                yield subtest
+
+
 class ConcurrentTestSuite(unittest.TestSuite):
     """A TestSuite whose run() calls out to a concurrency strategy."""