UnicodeDecodeError when loading job files

I notice

encoding='ANSI_X3.4-1968'

which is weird, because we should read the files with encoding='UTF-8' I think.

This may be happening because the locale is not set and the default charset is used, see this:

charset 'C' (canonical name: ANSI_X3.4-1968) will be used.

So one solution would be to explicitly set the encoding for when a file is read.

This is just speculation though. BTW, this has happened only on server installations which are more prone to having weird language configurations: contrast values of LANG, on desktop it's for example "en_US.UTF-8", while on server it just has "en_US".

I'll keep looking into this.

I modified the /etc/init.d/checkbox-certification server script to output a list of exports as well as LANG, and it turns out it's not set:

export INSTANCE=''
export JOB='networking'
export PATH='/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin'
export PWD='/'
export TERM='linux'
export UPSTART_EVENTS='started'
export UPSTART_INSTANCE=''
export UPSTART_JOB='checkbox-certification-server'

echoing $LANG turns up nothing.

I'll next try to find a place to explicitly set the encoding so that checkbox is at least able to read the jobs. If this works, we can then determine whether it's sensible to always fall back to UTF-8 encoding, but honor the system's encoding first if present.

OK, steps to reproduce the underlying problem:

$ unset LANG #to fallback to C locale, ANSI-whatever encoding
$ python3
>>> from checkbox.lib.template_i18n import TemplateI18n
>>> t=TemplateI18n()
>>> t.load_file(open("/usr/share/checkbox/jobs/usb.txt","r"),"")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/checkbox/lib/template_i18n.py", line 132, in load_file
    elements = super(TemplateI18n, self).load_file(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/checkbox/lib/template.py", line 73, in load_file
    for string in self._reader(file):
  File "/usr/lib/python3/dist-packages/checkbox/lib/template.py", line 31, in _reader
    buffer_new = file.read(size)
  File "/usr/lib/python3.2/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 291: ordinal not in range(128)
>>>

And further, to run this from trunk:

LANG= PYTHONPATH=./ python3

and from within the interpreter:

>>> from checkbox.lib.template_i18n import TemplateI18n
>>> t=TemplateI18n()
>>> t.load_file(open("/usr/share/checkbox/jobs/usb.txt","r"),"")

Submissions are coming in freely now, so this fix seems to have worked.

Note that this bug does NOT occur on Python 2, meaning that it won't be candidate for SRU, as the Python 3 version is also not SRUable for obvious reasons :)