Merge lp:~leonardr/lazr.restful/integrate-optimized-len into lp:lazr.restful

[Leonard Richardson]

Thanks to an optimization in lazr.batchnavigator, it's now very cheap to determine the total size of a list if the current batch is the last one in the list. It's also cheap to determine whether the current batch is the last one or not. (The most interesting case for our purposes is when the entire list fits in one batch, in which case the first batch is also the last.)

This branch backs lazr.restful away a little bit from its absolute insistence on sending 'total_size_link' instead of 'total_size' in responses to named operations. If the total size of the list turns out to be cheap to calculate, lazr.restful calculates it and sends 'total_size' instead of 'total_size_link'.

Most of this branch consists of test changes: creating scenarios in which the optimization doesn't apply, and showing that total_size_link is sent; creating scenarios in which the optimization does apply, and showing that total_size is sent instead. In one case this required creating extra sample data to pad out a collection returned from a named operation.

[lp:~leonardr/lazr.restful/integrate-optimized-len]

141. By Leonard Richardson on 2010-08-18

updated NEWS.

142. By Leonard Richardson on 2010-08-18

Use total(), which is part of the public interface, instead of .listlength.

[Edwin Grubbs (edwin-grubbs)]

Hi Leonard,

This looks good.

-Edwin

review: Approve (code)

=== modified file 'src/lazr/restful/NEWS.txt'
--- src/lazr/restful/NEWS.txt	2010-08-13 14:31:39 +0000
+++ src/lazr/restful/NEWS.txt	2010-08-18 16:44:44 +0000
@@ -2,6 +2,13 @@
 NEWS for lazr.restful
 =====================
 
+0.11.2 (Unreleased)
+===================
+
+Optimized lazr.restful to send 'total_size' instead of
+'total_size_link' when 'total_size' is easy to calculate, possibly
+saving the client from sending another HTTP request.
+
 0.11.1 (2010-08-13)
 ===================
 

=== modified file 'src/lazr/restful/_resource.py'
--- src/lazr/restful/_resource.py	2010-08-09 20:36:57 +0000
+++ src/lazr/restful/_resource.py	2010-08-18 16:44:44 +0000
@@ -640,13 +640,22 @@
 
         view_permission = getUtility(IWebServiceConfiguration).view_permission
         batch = { 'start' : navigator.batch.start }
-        total_size_link = self.total_size_link(navigator)
+        # If this is the last batch, then total() is easy to
+        # calculate. Let's use it and save the client from having to
+        # ask for the total size of the collection.
+        if navigator.batch.nextBatch() is None:
+            total_size_link = None
+        else:
+            total_size_link = self.total_size_link(navigator)
         if total_size_link is None:
-            batch['total_size'] = navigator.batch.listlength
+            total_size = navigator.batch.total()
+            if total_size < 0:
+                # lazr.batchnavigator uses -1 for an empty list.
+                # We want to use 0.
+                total_size = 0
+            batch['total_size'] = total_size
         else:
             batch['total_size_link'] = total_size_link
-        if navigator.batch.start < 0:
-            batch['start'] = None
         next_url = navigator.nextBatchURL()
         if next_url != "":
             batch['next_collection_link'] = next_url

=== modified file 'src/lazr/restful/docs/multiversion.txt'
--- src/lazr/restful/docs/multiversion.txt	2010-08-13 14:29:52 +0000
+++ src/lazr/restful/docs/multiversion.txt	2010-08-18 16:44:44 +0000
@@ -857,11 +857,11 @@
 
     >>> request_10 = create_web_service_request(
     ...     '/1.0/contacts',
-    ...     environ={'QUERY_STRING' : 'ws.op=find&string=Cleo'})
+    ...     environ={'QUERY_STRING' : 'ws.op=find&string=e&ws.size=2'})
     >>> operation = request_10.traverse(None)
     >>> result = simplejson.loads(operation())
     >>> [contact['name'] for contact in result['entries']]
-    ['Cleo Python']
+    ['Cleo Python', 'Oliver Bluth']
 
     >>> result['total_size']
     Traceback (most recent call last):
@@ -869,15 +869,18 @@
     KeyError: 'total_size'
 
     >>> print result['total_size_link']
-    http://.../1.0/contacts?string=Cleo&ws.op=find&ws.show=total_size
+    http://.../1.0/contacts?string=e&ws.op=find&ws.show=total_size
     >>> size_request = create_web_service_request(
     ...     '/1.0/contacts',
     ...     environ={'QUERY_STRING' :
-    ...         'string=Cleo&ws.op=find&ws.show=total_size'})
+    ...         'string=e&ws.op=find&ws.show=total_size'})
     >>> operation = size_request.traverse(None)
     >>> result = simplejson.loads(operation())
     >>> print result
-    1
+    3
+
+If the resultset fits on a single page, total_size will be provided
+instead of total_size_link, as a convenience.
 
     >>> request_10 = create_web_service_request(
     ...     '/1.0/contacts',
@@ -886,6 +889,8 @@
     >>> result = simplejson.loads(operation())
     >>> [contact['fax_number'] for contact in result['entries']]
     ['111-2121']
+    >>> result['total_size']
+    1
 
 Attempting to invoke the operation using its 'beta' name won't work.
 

=== modified file 'src/lazr/restful/example/base/tests/collection.txt'
--- src/lazr/restful/example/base/tests/collection.txt	2010-08-10 11:38:06 +0000
+++ src/lazr/restful/example/base/tests/collection.txt	2010-08-18 16:44:44 +0000
@@ -209,6 +209,32 @@
    >>> print webservice.get(s_recipes['total_size_link']).jsonBody()
    3
 
+If the entire collection fits in a single 'page' of results, the
+'total_size_link' is not present; instead, lazr.restful provides the
+total size as a convenience to the client.
+
+   >>> full_list = search_recipes("chicken", size=100)
+   >>> len(full_list['entries'])
+   3
+   >>> full_list['total_size']
+   3
+   >>> full_list['total_size_link']
+   Traceback (most recent call last):
+   ...
+   KeyError: 'total_size_link'
+
+The same is true if the client requests the last page of a list.
+
+   >>> last_page = search_recipes("chicken", start=2, size=2)
+   >>> len(last_page['entries'])
+   1
+   >>> full_list['total_size']
+   3
+   >>> full_list['total_size_link']
+   Traceback (most recent call last):
+   ...
+   KeyError: 'total_size_link'
+
 Custom operations may have error handling. In this case, the error
 handling is in the validate() method of the 'search' field.
 
@@ -224,9 +250,8 @@
 
     >>> general_cookbooks = webservice.get(
     ...     "/cookbooks?ws.op=find_for_cuisine&cuisine=General")
-    >>> print general_cookbooks.jsonBody()['total_size_link']
-    http://cookbooks.dev/devel/cookbooks?cuisine=General&ws.op=find_for_cuisine&ws.show=total_size
-
+    >>> print general_cookbooks.jsonBody()['total_size']
+    3
 
 POST operations
 ===============

=== modified file 'src/lazr/restful/example/multiversion/root.py'
--- src/lazr/restful/example/multiversion/root.py	2010-08-13 14:29:52 +0000
+++ src/lazr/restful/example/multiversion/root.py	2010-08-18 16:44:44 +0000
@@ -47,6 +47,8 @@
         pairset = PairSet()
         pairset.pairs = [
             KeyValuePair(pairset, "foo", "bar"),
+            KeyValuePair(pairset, "foo2", "bar"),
+            KeyValuePair(pairset, "foo3", "bar"),
             KeyValuePair(pairset, "1", "2"),
             KeyValuePair(pairset, "Some", None),
             KeyValuePair(pairset, "Delete", "me"),

=== modified file 'src/lazr/restful/example/multiversion/tests/introduction.txt'
--- src/lazr/restful/example/multiversion/tests/introduction.txt	2010-08-13 14:29:52 +0000
+++ src/lazr/restful/example/multiversion/tests/introduction.txt	2010-08-18 16:44:44 +0000
@@ -86,12 +86,16 @@
     Also delete: me
     Delete: me
     foo: bar
+    foo2: bar
+    foo3: bar
 
     >>> show_pairs('1.0')
     1: 2
     Also delete: me
     Delete: me
     foo: bar
+    foo2: bar
+    foo3: bar
 
     >>> show_pairs('2.0')
     1: 2
@@ -99,6 +103,8 @@
     Delete: me
     Some: None
     foo: bar
+    foo2: bar
+    foo3: bar
 
     >>> show_pairs('3.0')
     1: 2
@@ -106,6 +112,8 @@
     Delete: me
     Some: None
     foo: bar
+    foo2: bar
+    foo3: bar
 
 Entries
 =======
@@ -199,6 +207,8 @@
     1: 2
     Also delete: me
     foo: bar
+    foo2: bar
+    foo3: bar
 
 In '3.0', deleting a key-value pair simply sets its 'deleted' field
 to True. (This is an abuse of the HTTP DELETE method, but it makes a
@@ -217,6 +227,8 @@
     1: 2
     Also delete: me
     foo: bar
+    foo2: bar
+    foo3: bar
 
 And in a version which publishes the 'delete' field, we can check the
 key-value pair's value for that field.

=== modified file 'src/lazr/restful/example/multiversion/tests/operation.txt'
--- src/lazr/restful/example/multiversion/tests/operation.txt	2010-08-13 14:29:52 +0000
+++ src/lazr/restful/example/multiversion/tests/operation.txt	2010-08-18 16:44:44 +0000
@@ -24,26 +24,36 @@
     >>> print config.first_version_with_total_size_link
     2.0
 
-When the 'byValue' operation is invoked in version 1.0, it returns a
+When the 'byValue' operation is invoked in version 1.0, it always returns a
 total_size.
 
-    >>> def get_collection(version, op='byValue'):
-    ...     url = '/pairs?ws.op=%s&value=bar' % op
+    >>> def get_collection(version, op='byValue', value="bar", size=2):
+    ...     url = '/pairs?ws.op=%s&value=%s&ws.size=%s' % (op, value, size)
     ...     return webservice.get(url, api_version=version).jsonBody()
 
     >>> print sorted(get_collection('1.0').keys())
-    [u'entries', u'start', u'total_size']
+    [u'entries', u'next_collection_link', u'start', u'total_size']
 
 The operation itself doesn't change between 1.0 and 2.0, but in
 version 2.0, the operation starts returning total_size_link.
 
     >>> print sorted(get_collection('2.0').keys())
-    [u'entries', u'start', u'total_size_link']
+    [u'entries', u'next_collection_link', u'start', u'total_size_link']
 
 The same happens in 3.0.
 
     >>> print sorted(get_collection('3.0', 'by_value').keys())
-    [u'entries', u'start', u'total_size_link']
+    [u'entries', u'next_collection_link', u'start', u'total_size_link']
+
+However, if the total size is easy to calculate (for instance, because
+all the results fit on one page), a total_size is returned instead of
+total_size_link.
+
+    >>> print sorted(get_collection('3.0', 'by_value', size=100).keys())
+    [u'entries', u'start', u'total_size']
+
+    >>> print get_collection('3.0', 'by_value', 'no-such-value')
+    {u'total_size': 0, u'start': 0, u'entries': []}
 
 Mutators as named operations
 ----------------------------

=== modified file 'src/lazr/restful/version.txt'
--- src/lazr/restful/version.txt	2010-08-13 14:31:39 +0000
+++ src/lazr/restful/version.txt	2010-08-18 16:44:44 +0000
@@ -1,1 +1,1 @@
-0.11.1
+0.11.2

=== modified file 'versions.cfg'
--- versions.cfg	2010-08-10 18:36:09 +0000
+++ versions.cfg	2010-08-18 16:44:44 +0000
@@ -16,7 +16,7 @@
 docutils = 0.5
 epydoc = 3.0.1
 grokcore.component = 1.6
-lazr.batchnavigator = 1.2.0
+lazr.batchnavigator = 1.2.2
 lazr.delegates = 1.2.0
 lazr.enum = 1.1.2
 lazr.lifecycle = 1.0