UnicodeEncodeError: 'ascii' codec can't encode characters in position 2217-2258: ordinal not in range(128)

Interestingly, setting LANG=C fixes this, but the unit test in question still fails. It turns out that the unit test failure was caused by a missing dependency, but it's odd that the missing unit test dependency would give this error.

I think this is some bad interaction between doctest and testtools. What output do you get if you run `python -m doctest wikkid.contrib.creole_1_1.creole2html` instead, with the same missing dependency?

See output below...

thomi@neil-desktop:~/code/wikkid/wikkid$ python -m doctest wikkid.contrib.creole_1_1.creole2html Traceback (most recent call last):
  File "/usr/lib/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.6/doctest.py", line 2686, in <module>
    sys.exit(_test())
  File "/usr/lib/python2.6/doctest.py", line 2677, in _test
    failures, _ = testfile(filename, module_relative=False)
  File "/usr/lib/python2.6/doctest.py", line 1929, in testfile
    text, filename = _load_testfile(filename, package, module_relative)
  File "/usr/lib/python2.6/doctest.py", line 219, in _load_testfile
    return open(filename).read(), filename
IOError: [Errno 2] No such file or directory: 'wikkid.contrib.creole_1_1.creole2html'
thomi@neil-desktop:~/code/wikkid/wikkid$ cd wikkid/contrib/creole_1_1/

thomi@neil-desktop:~/code/wikkid/wikkid/wikkid/contrib/creole_1_1$ python -m doctest creole2html.py

**********************************************************************
File "creole2html.py", line 15, in creole2html
Failed example:
    import lxml.html.usedoctest
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python2.6/doctest.py", line 1241, in __run
        compileflags, 1) in test.globs
      File "<doctest creole2html[0]>", line 1, in <module>
        import lxml.html.usedoctest
    ImportError: No module named lxml.html.usedoctest
**********************************************************************
File "creole2html.py", line 23, in creole2html
Failed example:
    parse(u'test')
Expected:
    <p>test</p>
Got:
    <p>test</p>
    <BLANKLINE>
**********************************************************************
File "creole2html.py", line 26, in creole2html
Failed example:
    parse(u'test\ntest')
Expected:
    <p>test test</p>
Got:
    <p>test test</p>
    <BLANKLINE>
**********************************************************************
File "creole2html.py", line 32, in creole2html
Failed example:
    parse(u'test\n\ntest')
Expected:
    <p>test</p><p>test</p>
Got:
    <p>test</p>
    <p>test</p>
    <BLANKLINE>
**********************************************************************
File "creole2html.py", line 35, in creole2html
Failed example:
    parse(u'test\\\\test')
Expected:
    <p>test<br>test</p>
Got:
    <p>test<br>test</p>
    <BLANKLINE>
Traceback (most recent call last):
  File "/usr/lib/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.6/doctest.py", line 2686, in <module>
    sys.exit(_test())
  File "/usr/lib/python2.6/doctest.py", line 2675, in _test
    failures, _ = testmod(m)
  File "/usr/lib/python2.6/doctest.py", line 1830, in testmod
    runner.run(test)
  File "/usr/lib/python2.6/doctest.py", line 1374, in run
    return self.__run(test, compileflags, out)
  File "/usr/lib/python2.6/doctest.py", line 1290, in __run
    self.repor...

Okay, I see that doctest bug, it's filed upstream as <http://bugs.python.org/issue1293741> and is actually fixed in Python 2.7 despite the grousing there.

However I don't the same behaviour with that kind of test using testtools, as the exception is trapped when formatting the output and displayed unhelpfully, but safely, as:

    Traceback (most recent call last):
      File "C:\Python24\lib\doctest.py", line 2157, in runTest
        raise self.failureException(self.format_failure(new.getvalue()))
    AssertionError: <unprintable AssertionError object>

So, what version of testtools are you running there?

This is our last unfixed critical bug. I guess the next step is to try trunk against wikkid, see if the error happens still. Thomi, would you be interested in doing that?

Hi,

On 8 August 2011 23:22, Jonathan Lange <email address hidden> wrote:
> This is our last unfixed critical bug. I guess the next step is to try
> trunk against wikkid, see if the error happens still. Thomi, would you
> be interested in doing that?
>

Certainly, I'll give it a go.

--
Thomi Richards

On Mon, Aug 8, 2011 at 8:02 PM, Thomi Richards
<email address hidden> wrote:
> Hi,
>
> On 8 August 2011 23:22, Jonathan Lange <email address hidden> wrote:
>> This is our last unfixed critical bug. I guess the next step is to try
>> trunk against wikkid, see if the error happens still. Thomi, would you
>> be interested in doing that?
>>
>
> Certainly, I'll give it a go.
>

Thanks!

jml

I just tried running this against wikkid trunk (r69), testtools trunk (r225), with python-lxml removed from my system, with Python 2.6.7 and with LANG=en_NZ.UTF-8. I could not reproduce the failure.

Full command:
$ LANG=en_NZ.UTF-8 PYTHONPATH=.:~/src/testtools/trunk/:~/src/bzr/trunk python2.6 -m testtools.run wikkid.tests.test_suite

If this is just a bug with Python 2.6.4, then we should say "wont fix". It's hard enough supporting the latest versions of each major release without also having to support earlier minor releases.

Hi,

Sorry it took me so long to get back to this. Here's my findings:

First, this is only reproducable on my machine if I don't have the python-lxml package installed. With that package installed then everything passes as expected.

My results are as follows:

Python 2.6
---------------

testtools 0.9.8-1: reproduce original bug
trunk revno 225: unit tests fail, but give correct output.

Python 2.7
---------------

testtools 0.9.8-1: unit tests fail, but give correct output.
trunk revno 225: unit tests fail, but give correct output.

So it seems that you've fixed this for python2.6, and it works in python2.7 regardless of your patch.

Cheers,

Thanks so much. Closing now.