Merge lp:~jkakar/txfluiddb/failing-tests into lp:txfluiddb

Proposed by Jamu Kakar
Status: Merged
Approved by: Tristan Seligmann
Approved revision: 11
Merged at revision: 10
Proposed branch: lp:~jkakar/txfluiddb/failing-tests
Merge into: lp:txfluiddb
Diff against target: 130 lines (+72/-5)
2 files modified
txfluiddb/client.py (+21/-4)
txfluiddb/test/test_client.py (+51/-1)
To merge this branch: bzr merge lp:~jkakar/txfluiddb/failing-tests
Reviewer Review Type Date Requested Status
Tristan Seligmann Approve
Review via email: mp+27431@code.launchpad.net

Description of the change

This branch introduces the following changes:

- Uses the 'json' package, if available, otherwise falls back to
  'simplejson'. I *think* this will solve the issues, but I've only
  tested on Ubuntu 10.04 with Python 2.6.5.

- Adds a test using a unicode value, to make sure encoding/decoding
  works as expected.

To post a comment you must log in.
Revision history for this message
Tristan Seligmann (mithrandi) wrote :

Using simplejson (on Python 2.5.5), it still fails like so:

===============================================================================
[FAIL]: txfluiddb.test.test_client.ObjectTests.test_getString

Traceback (most recent call last):
  File "/home/mithrandi/code/txFluidDB/trunk/txfluiddb/test/test_client.py", line 680, in _gotResponse
    self.assertIsInstance(response, unicode)
  File "/home/mithrandi/code/Twisted/trunk/twisted/trial/unittest.py", line 504, in failUnlessIsInstance
    self.fail("%r is not an instance of %s" % (instance, classOrTuple))
twisted.trial.unittest.FailTest: 'Data goes where?' is not an instance of <type 'unicode'>
===============================================================================
[ERROR]: txfluiddb.test.test_client.NamespaceTests.test_getNamespaces

Traceback (most recent call last):
  File "/home/mithrandi/code/txFluidDB/trunk/txfluiddb/client.py", line 230, in _parseResponse
    namespaces.append(self.child(name))
  File "/home/mithrandi/code/txFluidDB/trunk/txfluiddb/client.py", line 152, in child
    return type(self)(*path)
  File "/home/mithrandi/code/txFluidDB/trunk/txfluiddb/client.py", line 47, in __init__
    self.checkComponent(component)
  File "/home/mithrandi/code/txFluidDB/trunk/txfluiddb/client.py", line 60, in checkComponent
    'Name of type %r is not unicode: %r' % (type(name), name))
txfluiddb.errors.InvalidName: Name of type <type 'str'> is not unicode: 'ns1'
===============================================================================
[ERROR]: txfluiddb.test.test_client.NamespaceTests.test_getTags

Traceback (most recent call last):
  File "/home/mithrandi/code/txFluidDB/trunk/txfluiddb/client.py", line 251, in _parseResponse
    tags.append(self.tag(name))
  File "/home/mithrandi/code/txFluidDB/trunk/txfluiddb/client.py", line 167, in tag
    return Tag(*path)
  File "/home/mithrandi/code/txFluidDB/trunk/txfluiddb/client.py", line 47, in __init__
    self.checkComponent(component)
  File "/home/mithrandi/code/txFluidDB/trunk/txfluiddb/client.py", line 60, in checkComponent
    'Name of type %r is not unicode: %r' % (type(name), name))
txfluiddb.errors.InvalidName: Name of type <type 'str'> is not unicode: 'tag1'
-------------------------------------------------------------------------------

I think this is really just an issue with the use of json.dumps in the tests; when it gives us back a str, when we pass that to json.loads again, it'll also give us strs. Introducing a utility function that decodes the result of json.dumps if it is str should be sufficient to deal with the problem, I hope.

review: Needs Fixing
Revision history for this message
Jamu Kakar (jkakar) wrote :

Yeah, I thought the same thing... but it's not just about
str/unicode results. If a JSON object has a string, like {foo:
"bar"} for example, the embedded string "foo" will either be unicode
or str, depending on its contents. If its a str the expectation is
broken.

Revision history for this message
Jamu Kakar (jkakar) wrote :

Registering a custom decoder that always returns unicode for string
data will probably do the trick. I'll check it out and report back.

Revision history for this message
Tristan Seligmann (mithrandi) wrote :

As far as I can tell, if the argument to json.loads is unicode, then all strings embedded in the resulting return value will always be unicode. For example:

>>> simplejson.loads('[{"foo": "bar"}, "baz"]')
[{'foo': 'bar'}, 'baz']
>>> simplejson.loads(u'[{"foo": "bar"}, "baz"]')
[{u'foo': u'bar'}, u'baz']

lp:~jkakar/txfluiddb/failing-tests updated
10. By Jamu Kakar

- Implemented a loads function that converts JSON encoded data into
  unicode before decoding it into Python.

11. By Jamu Kakar

- Improved docstring.

Revision history for this message
Tristan Seligmann (mithrandi) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'txfluiddb/client.py'
2--- txfluiddb/client.py 2010-05-11 14:42:26 +0000
3+++ txfluiddb/client.py 2010-06-12 21:37:25 +0000
4@@ -1,9 +1,8 @@
5 from urllib import quote, urlencode
6 from re import compile, U
7 from base64 import b64encode
8-
9+import types
10 import simplejson as json
11-import types
12
13 from txfluiddb.errors import InvalidName, InvalidTagValueType
14 from txfluiddb.http import getPage
15@@ -490,7 +489,7 @@
16 def _parse((status, headers, page)):
17 if page:
18 # assert headers['content-type'] == 'application/json'
19- return json.loads(page)
20+ return loads(page)
21
22 d = self.getPage(url=url,
23 method=method,
24@@ -545,7 +544,7 @@
25 # Let this raise a KeyError if there's no content-type header?
26 ct = headers['content-type'][0]
27 if ct == PRIMITIVE_CONTENT_TYPE:
28- return json.loads(page)
29+ return loads(page)
30 else:
31 return Blob(ct, page)
32
33@@ -743,5 +742,23 @@
34
35
36
37+def loads(data):
38+ """
39+ Load JSON data and decode it into Python objects.
40+
41+ This function always converts JSON data to C{unicode} before decoding it
42+ into Python objects. This guarantees that strings are always represented
43+ as C{unicode} objects.
44+
45+ @type data: C{str} or C{unicode}
46+ @param data: The JSON data to load, assumed to be encoded in UTF-8.
47+ @return: A Python representation of C{data}.
48+ """
49+ if isinstance(data, str):
50+ data = unicode(data, 'utf-8')
51+ return json.loads(data)
52+
53+
54+
55 __all__ = ['Namespace', 'Tag', 'BasicCreds', 'FLUIDDB_ENDPOINT', 'Endpoint',
56 'Object', 'Blob']
57
58=== modified file 'txfluiddb/test/test_client.py'
59--- txfluiddb/test/test_client.py 2010-01-28 16:28:38 +0000
60+++ txfluiddb/test/test_client.py 2010-06-12 21:37:25 +0000
61@@ -5,7 +5,7 @@
62
63 from txfluiddb.errors import InvalidName
64 from txfluiddb.client import (
65- Namespace, Tag, Endpoint, _HasPath, Object, Blob, BasicCreds)
66+ Namespace, Tag, Endpoint, _HasPath, Object, Blob, BasicCreds, loads)
67
68
69
70@@ -682,6 +682,18 @@
71 return self.makeGetRequest(response).addCallback(_gotResponse)
72
73
74+ def test_getUnicode(self):
75+ """
76+ Retrieving a unicode value results in a C{unicode} object.
77+ """
78+ response = (json.dumps(u'\N{HIRAGANA LETTER A}'),
79+ 'application/vnd.fluiddb.value+json')
80+ def _gotResponse(response):
81+ self.assertIsInstance(response, unicode)
82+ self.assertEqual(response, u'\N{HIRAGANA LETTER A}')
83+ return self.makeGetRequest(response).addCallback(_gotResponse)
84+
85+
86 def test_getInteger(self):
87 """
88 Retrieving a integer value results in an integer object.
89@@ -857,3 +869,41 @@
90 self.assertEqual(self.endpoint.url, 'http://fluiddb.url/objects?query=has+fluiddb%2Fabout')
91 self.assertEqual(self.endpoint.data, None)
92 return d
93+
94+
95+
96+class LoadsTests(TestCase):
97+ """
98+ Tests for L{loads}.
99+ """
100+ def test_loadsString(self):
101+ """
102+ If JSON encoded data is passed to L{loads} as a C{str} it will be
103+ converted to C{unicode}.
104+ """
105+ encoded = '{"foo": "bar"}'
106+ decoded = loads(encoded)
107+ self.assertEqual(decoded, {u"foo": u"bar"})
108+ self.assertIsInstance(decoded.keys()[0], unicode)
109+ self.assertIsInstance(decoded[u"foo"], unicode)
110+
111+ def test_loadsUnicode(self):
112+ """
113+ If JSON encoded data is passed to L{loads} as a C{str} it will be
114+ converted to C{unicode}.
115+ """
116+ encoded = u'{"foo": "bar"}'
117+ decoded = loads(encoded)
118+ self.assertEqual(decoded, {u"foo": u"bar"})
119+ self.assertIsInstance(decoded.keys()[0], unicode)
120+ self.assertIsInstance(decoded[u"foo"], unicode)
121+
122+ def test_loadsUTF8(self):
123+ """
124+ JSON encoded data in C{str} form is assumed to be encoded in UTF-8.
125+ """
126+ encoded = '{"foo": "\\u3042"}'
127+ decoded = loads(encoded)
128+ self.assertEqual(decoded, {u"foo": u"\N{HIRAGANA LETTER A}"})
129+ self.assertIsInstance(decoded.keys()[0], unicode)
130+ self.assertIsInstance(decoded[u"foo"], unicode)

Subscribers

People subscribed via source and target branches

to all changes: