Test descriptions with non-ascii characters cause tests to not be run, backend to crash

Sorry for the meta, but i got another automatic crash report to this same crash, is there a way to link the uploaded report to this bugreport?

Hi Paul,

Thank you for taking the time to report this bug and helping to make Ubuntu better.

I think I'm able to replicate this behavior, but I wanted to confirm first whether this file was generated on your system when the crash occurred:

/var/crash/_usr_share_checkbox_backend.0.crash

Or something to that effect indicating that the checkbox backend crashed.

Once you confirm this, which would strongly indicate we're getting the same crash, I'll be able to provide a more detailed analysis on this bug.

Thanks!

Of course, thanks for the prompt response...
I attached /var/crash/_usr_share_checkbox_backend.0.crash and cat /var/crash/_usr_share_checkbox_autotest_suite.0.crash from the same crash.

Paul,

Thanks to you for getting back to me so quickly!

Based on the fact that you also got the backend crash I was expecting, here are steps to reproduce on ubuntu Oneiric (this is mainly for the benefit of developers who are going to work on this bug).

1- install a language that may contain non-ascii characters (so far this has been observed in russian and german):
   sudo apt-get install language-pack-de language-pack-gnome-de
2- Launch checkbox using this installed language:
  LANG=de_AT.UTF-8 LANGUAGE=de_AT:de checkbox-gtk
3- Start a test run with all the tests. It's OK if you skip them, the problem gets triggered either way.

Expected result:
 - All tests run correctly and the run finishes.

Actual result:
- As seen in Paul's checkbox log file, any tests whose description contains non-ascii characters generate an exception (excerpt below).
- As soon as the first job specifying "user" gets dispatched to the backend, the backend crashes with the trace seen in bug 827859. Since it crashed in the middle of executing a job, the frontend doesn't notice this condition and will wait forever. As Paul observed, the window eventually becomes unresponsive.

Here's the description-related trace:

2011-08-25 14:32:53,782 DEBUG Calling checkbox.arguments JobsInfo.replacement({'description': 'Ihr(e) Soundger\xc3\xa4t(e) werden erkan
nt:\n\n$output\n\nIst dies korrekt?', 'plugin': 'manual', 'name': 'audio/list_devices', 'command': 'cat /proc/asound/cards', 'suite': '__audio__', 'requires': "device.c
ategory == 'AUDIO'\npackage.name == 'alsa-base'"}) for report-job with priority -100.
2011-08-25 14:32:53,782 ERROR Error running event handler checkbox.arguments JobsInfo.replacement({'description': 'Ihr(e) Soundger\xc3\xa4t(e) werden erkannt:\n\n$output\n\nIst dies korrekt?', 'plugin': 'manual', 'name': 'audio/list_devices', 'command': 'cat /proc/asound/cards', 'suite': '__audio__', 'requires': "device.category == 'AUDIO'\npackage.name == 'alsa-base'"}) for event type 'report-job'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/checkbox/reactor.py", line 74, in fire
    results.append(handler(*args, **kwargs))
  File "/usr/lib/python2.7/dist-packages/checkbox/arguments.py", line 90, in replacement
    kwargs)
  File "/usr/lib/python2.7/dist-packages/checkbox/arguments.py", line 66, in replace_arguments
    args[name_index] = self.argument_replacer(name, value)
  File "/usr/lib/python2.7/dist-packages/checkbox/arguments.py", line 83, in replacer
    return schemas[name].coerce(value)
  File "/usr/lib/python2.7/dist-packages/checkbox/properties.py", line 71, in coerce
    return self._variable_class(**self._variable_kwargs).coerce(value)
  File "/usr/lib/python2.7/dist-packages/checkbox/variables.py", line 267, in coerce
    % (attribute, value, e))
ValueError: Value of 'description' key of dict {'description': 'Ihr(e) Soundger\xc3\xa4t(e) werden erkannt:\n\n$output\n\nIst dies korrekt?', 'plugin': 'manual', 'name': 'audio/list_devices', 'command': 'cat /proc/asound/cards', 'suite': '__audio__', 'requires': "device.category == 'AUDIO'\npackage.name == 'alsa-base'"} could not be converted: 'a...

Also of note: the first job that runs with user: (and triggers the problem with the backend) is autotest which is NOT included in the default whitelist. Some investigation is needed as to why this job is being run.

There are two changes to be made here:

- First, checkbox/variable.py, the UnicodeVariable class now uses the 'ascii' encoder by default (decoder in this case I think), so if the string to be converted to unicode contains non-ascii characters, the call to unicode will throw an exception, which was the originally observed problem. Fix is to explicitly specify the encoding to use while reading the string. I used utf-8.

- Second, when loading strings into a GtkTextBuffer, it expects utf8 encoding, otherwise it "eats" characters and the string gets truncated. So we check to see if the passed text is a unicode instance and, if so, it gets explicitly re-encoded as utf8, thus keeping the TextBuffer happy.

I'll link a branch with these fixes in a couple of minutes.

This bug was fixed in the package checkbox - 0.12.7

---------------
checkbox (0.12.7) oneiric; urgency=low

  New upstream release (LP: #850395):

  [Brendan Donegan]
  * Redirecting stderr to pipe to fix the gconf_resource script (LP: #832321)
  * Clear jobs directory when user selects No to recover question (LP: #836623)

  [Daniel Manrique]
  * checkbox/job.py: Guard against bogus timeout values (LP: #827859)
  * More explicit handling of string decoding/encoding, avoids problems with
    non-ascii characters (LP: #833747)
  * Changed architecture from all to any for checkbox base, to build
    architecture-specific binaries (LP: #833696)

  [Jeff Lane]
  * Several corrections necessary due to test name changes or typos found in
    job files

  [Marc Tardif]
  * Connecting hyper text widgets only once (LP: #827904)
  * Detecting MMC readers as OTHER instead of DISK (LP: #822948)
  * Validating the hostname in the SSL certificate (LP: #625076)
  * Validating the submission.xml (LP: #838123)
 -- Daniel Manrique <email address hidden> Fri, 14 Sep 2011 17:15:26 -0400