UnicodeDecodeError: 'utf8' codec can't decode byte 0xad in position 7: unexpected code byte on some bson reports

according to the backtrace we have a unicode string which when serialized by the serializer cannot be deserialized.

(decode_element reads the type code then dispatches to decode_string_element. I'm a bit confused why decode_cstring is doing a utf8 decode..)

Ah, the decode_cstring is for the key of the dict. So it looks like encode_cstring has perhaps gotten horribly confused.

One possibility is a failure to write the key correctly. According to the bson spec key names must be
cstring ::= (byte*) "\x00" CString
(where byte* are valid utf8) - its a utf8 bytestring without embedded \x00.

b'field\x85ries_filter' is the key that is giving grief. Which appears to be one of the keys in args.
>>> bson.loads(bson.dumps({'field\x85ries_filter':0}))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/robertc/source/launchpad/oops-datedir-repo/working/eggs/bson-0.3.2-py2.6.egg/bson/__init__.py", line 75, in loads
    return decode_document(data, 0)[1]
  File "/home/robertc/source/launchpad/oops-datedir-repo/working/eggs/bson-0.3.2-py2.6.egg/bson/codec.py", line 240, in decode_document
    base, name, value = decode_element(data, base)
  File "/home/robertc/source/launchpad/oops-datedir-repo/working/eggs/bson-0.3.2-py2.6.egg/bson/codec.py", line 232, in decode_element
    return decode_func(data, base)
  File "/home/robertc/source/launchpad/oops-datedir-repo/working/eggs/bson-0.3.2-py2.6.egg/bson/codec.py", line 313, in decode_int32_element
    base, name = decode_cstring(data, base + 1)
  File "/home/robertc/source/launchpad/oops-datedir-repo/working/eggs/bson-0.3.2-py2.6.egg/bson/codec.py", line 121, in decode_cstring
    return (base + length, buf.getvalue().decode("utf8"))
  File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x85 in position 5: unexpected code byte

So this is a case of a) bson accepting something it shouldn't and b) LP generating something invalid.

def encode_cstring(value):
    if isinstance(value, unicode):
        value = value.encode("utf8")
    return value + "\x00"

clearly unsafe: while most invalid keys will raise e.g. TypeError:
>>> bson.encode({1:0})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'encode'
>>> bson.dumps({1:0})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/robertc/source/launchpad/oops-datedir-repo/working/eggs/bson-0.3.2-py2.6.egg/bson/__init__.py", line 69, in dumps
    return encode_document(obj, [], generator_func = generator)
  File "/home/robertc/source/launchpad/oops-datedir-repo/working/eggs/bson-0.3.2-py2.6.egg/bson/codec.py", line 207, in encode_document
    encode_value(name, value, buf, traversal_stack, generator_func)
  File "/home/robertc/source/launchpad/oops-datedir-repo/working/eggs/bson-0.3.2-py2.6.egg/bson/codec.py", line 193, in encode_value
    buf.write(encode_int32_element(name, value))
  File "/home/robertc/source/launchpad/oops-datedir-repo/working/eggs/bson-0.3.2-py2.6.egg/bson/codec.py", line 310, in encode_int32_element
    return "\x10" + encode_cstring(name) + struct.pack("<i", value)
  File "/home/robertc/source/launchpad/oops-datedir-repo/working/eggs/bson-0.3.2-py2.6.egg/bson/codec.py", line 111, in encode_cstring
    return value + "\x00"
TypeError: unsupported operand type(s) for +: 'int' and 'str'

but a bytestring is assumed to be utf8 already, which is quite clearly not the case here.
def encode_cstring(value):
    if isinstance(value, unicode):
        value = value.encode("utf8")
    elif isinstance(value, str):
         # check value is utf8.
         value.decode('utf8')
    else:
        raise TypeError('Invalid type for cstring %r' % value)
    return value + "\x00"

https://github.com/martinkou/bson/issues/11 upstream issue.

datedir-repo isn't actually at fault here, nor the cause, so dropping its task to high, and moving onto LP itself.

Fixed in stable r14399 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/14399>.

Can't trigger the oops anymore because the bug with +login is also fixed.