Merge lp:~wgrant/lazr.restful/bug-684430 into lp:lazr.restful

[William Grant]

The webservice:json TALES function now produces HTML-escaping-resilient JSON.

It is impossible to correctly insert the output of the webservice:json TALES function into a script tag in an HTML document. HTML (not XHTML) specifies <script>'s contents as CDATA, which doesn't expand entities but can be terminated by any closing tag. So you can't turn <, >, & into entities, as the JSON parser will see the undecoded entities. And if you don't escape them at all then malicious JSON can break out of the element. So we need to escape them some other way.

simplejson now provides a JSONEncoderForHTML, which replaces <, > and & with their corresponding Unicode escape sequences. Its output is resilient to escaping, allowing it to be safely inserted into a document.

I've changed ResourceJSONEncoder to inherit JSONEncoderForHTML instead of JSONEncoder. This is a little too broad: it will perform the extra escaping on content returned normally as application/json, not just text/html. But this is OK: the encodings are equivalent, decoding to identical strings. The only downsides are reduced readability and minor representation bloat.

JSONEncoderForHTML is new in simplejson 2.1.0, so I incremented the requirement.

[Robert Collins (lifeless)]

review: Approve

=== modified file 'setup.py'
--- setup.py	2011-03-21 19:54:33 +0000
+++ setup.py	2011-03-31 04:54:29 +0000
@@ -65,7 +65,7 @@
         'martian==0.11',
         'pytz',
         'setuptools',
-        'simplejson>=2.0.9',
+        'simplejson>=2.1.0',
         'van.testing',
         'wsgiref',
         'zope.app.pagetemplate',

=== modified file 'src/lazr/restful/NEWS.txt'
--- src/lazr/restful/NEWS.txt	2011-03-25 11:37:15 +0000
+++ src/lazr/restful/NEWS.txt	2011-03-31 04:54:29 +0000
@@ -10,6 +10,9 @@
 The object modification event will not be fired if a client sends an
 empty changeset via PATCH.
 
+The webservice:json TALES function now returns JSON that will survive
+HTML escaping.
+
 0.18.0 (2011-03-23)
 ===================
 

=== modified file 'src/lazr/restful/_resource.py'
--- src/lazr/restful/_resource.py	2011-03-25 11:40:57 +0000
+++ src/lazr/restful/_resource.py	2011-03-31 04:54:29 +0000
@@ -34,6 +34,7 @@
 import copy
 import os
 import simplejson
+import simplejson.encoder
 import time
 
 # Import SHA in a way compatible with both Python 2.4 and Python 2.6.
@@ -199,7 +200,7 @@
     pass
 
 
-class ResourceJSONEncoder(simplejson.JSONEncoder):
+class ResourceJSONEncoder(simplejson.encoder.JSONEncoderForHTML):
     """A JSON encoder for JSON-exposable resources like entry resources.
 
     This class works with simplejson to encode objects as JSON if they

=== modified file 'src/lazr/restful/docs/webservice.txt'
--- src/lazr/restful/docs/webservice.txt	2011-03-02 19:33:48 +0000
+++ src/lazr/restful/docs/webservice.txt	2011-03-31 04:54:29 +0000
@@ -2121,9 +2121,13 @@
     False
 
 The json() function converts generic Python data structures to JSON,
-as well as objects that can be adapted to IEntry.
+as well as objects that can be adapted to IEntry. It converts markup
+characters (<, >, &) into their respective Unicode escape sequences,
+since entities within <script> tags are not expanded.
 
     >>> test_tales("context/webservice:json", context="foobar")
     '"foobar"'
     >>> test_tales("context/webservice:json", context=A1)
     '{"name": ...}'
+    >>> test_tales("context/webservice:json", context="<foo&>")
+    '"\\u003cfoo\\u0026\\u003e"'

=== modified file 'versions.cfg'
--- versions.cfg	2011-03-21 19:54:33 +0000
+++ versions.cfg	2011-03-31 04:54:29 +0000
@@ -25,7 +25,7 @@
 martian = 0.11
 pytz = 2010h
 setuptools = 0.6c11
-simplejson = 2.0.9
+simplejson = 2.1.3
 transaction = 1.0.0
 van.testing = 2.0.1
 wsgi-intercept = 0.4