Merge lp:~leonardr/launchpad/new-lazr.restful-and-friends into lp:launchpad/db-devel

[Leonard Richardson]

This branch integrates new versions of many other libraries into Launchpad: launchpadlib, lazr.batchnavigator, lazr.delegates, lazr.restful, lazr.restfulclient, and wadllib. A couple of these (lazr.delegates and wadllib) are just due for a refresh. The other changes are necessary to have the "devel" version of the web service switch from serving total_size_link to serving total_size.

Note the changes made to the tests having to do with changes to lazr.batchnavigator. A SELECT COUNT query is no longer made unless you explicitly ask for the length of a queryset. And an empty collection has a 'start' of 0 instead of None.

[Paul Hummer (rockstar)]

review: Approve

[Leonard Richardson (leonardr)]

I need a follow-up review. I integrated a new version of lazr.restful into it and added an end-to-end test to prevent another bug like the one that almost slipped through.

http://paste.ubuntu.com/477538/

[Guilherme Salgado (salgado)]

Looks good to me

review: Approve (code)

=== modified file 'lib/canonical/launchpad/doc/batch_navigation.txt'
--- lib/canonical/launchpad/doc/batch_navigation.txt	2009-08-21 17:43:28 +0000
+++ lib/canonical/launchpad/doc/batch_navigation.txt	2010-08-18 13:47:45 +0000
@@ -69,27 +69,33 @@
   >>> email_batch = batch_nav.currentBatch()
   >>> batch_items = list(email_batch)
 
-Because we're only looking at the first batch, the database is only asked for
-the number of rows, and the first 10 rows:
+Because we're only looking at the first batch, the database is only
+asked for the first 11 rows. (lazr.batchnavigator asks for 11 instead
+of 10 so that it can reliably detect the end of the dataset).
 
   >>> len(CursorWrapper.last_executed_sql)
-  2
-  >>> print CursorWrapper.last_executed_sql[1]
-  SELECT ... FROM EmailAddress ... LIMIT 10...
+  1
   >>> print CursorWrapper.last_executed_sql[0]
-  SELECT COUNT(*) FROM EmailAddress
+  SELECT ... FROM EmailAddress ... LIMIT 11...
 
-Get the next 10.  The database is only asked for the next 10 rows -- no further
-COUNTs are issued:
+Get the next 10.  The database is only asked for the next 11 rows:
 
   >>> CursorWrapper.last_executed_sql = []
   >>> email_batch2 = email_batch.nextBatch()
   >>> batch_items = list(email_batch2)
   >>> len(CursorWrapper.last_executed_sql)
   1
-  >>> CursorWrapper.last_executed_sql[0].endswith('LIMIT 10 OFFSET 10')
+  >>> CursorWrapper.last_executed_sql[0].endswith('LIMIT 11 OFFSET 10')
   True
 
+As seen above, simply accessing the batch doesn't trigger a SQL query
+asking for the length. But explicitly asking for the length will
+trigger a SQL query.
+
+  >>> CursorWrapper.last_executed_sql = []
+  >>> ignored = email_batch.total()
+  >>> print CursorWrapper.last_executed_sql[0]
+  SELECT COUNT(*) FROM EmailAddress
 
 Multiple pages
 ==============

=== modified file 'lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt'
--- lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt	2010-07-07 20:42:22 +0000
+++ lib/canonical/launchpad/pagetests/webservice/launchpadlib.txt	2010-08-18 13:47:45 +0000
@@ -71,7 +71,8 @@
     >>> print lp_anon.me.name
     Traceback (most recent call last):
       ...
-    HTTPError: HTTP Error 401: Unauthorized...
+    Unauthorized: HTTP Error 401: Unauthorized...
+
 
 Caching
 =======

=== modified file 'lib/canonical/launchpad/pagetests/webservice/multiversion.txt'
--- lib/canonical/launchpad/pagetests/webservice/multiversion.txt	2010-03-05 19:03:12 +0000
+++ lib/canonical/launchpad/pagetests/webservice/multiversion.txt	2010-08-18 13:47:45 +0000
@@ -1,5 +1,36 @@
+****************************
 Differences between versions
-----------------------------
+****************************
+
+total_size_link
+===============
+
+In the 'devel' version of the web service, named operations that
+return collections will return a 'total_size_link' pointing to the
+total size of the collection.
+
+  >>> def get_collection(version):
+  ...     collection = webservice.get(
+  ...         "/people?ws.op=find&text=salgado", api_version=version)
+  ...     return collection.jsonBody()
+
+  >>> collection = get_collection("devel")
+  >>> print sorted(collection.keys())
+  [u'entries', u'start', u'total_size_link']
+  >>> print webservice.get(collection['total_size_link']).jsonBody()
+  1
+
+In previous versions, the same named operations will return a
+'total_size' containing the actual size of the collection.
+
+  >>> collection = get_collection("1.0")
+  >>> print sorted(collection.keys())
+  [u'entries', u'start', u'total_size']
+  >>> print collection['total_size']
+  1
+
+Mutator operations
+==================
 
 In the 'beta' version of the web service, mutator methods like
 IBugTask.transitionToStatus are published as named operations. In

=== modified file 'lib/canonical/launchpad/rest/configuration.py'
--- lib/canonical/launchpad/rest/configuration.py	2010-06-03 16:37:27 +0000
+++ lib/canonical/launchpad/rest/configuration.py	2010-08-18 13:47:45 +0000
@@ -25,6 +25,7 @@
     path_override = "api"
     active_versions = ["beta", "1.0", "devel"]
     last_version_with_mutator_named_operations = "beta"
+    first_version_with_total_size_link = "devel"
     view_permission = "launchpad.View"
     compensate_for_mod_compress_etag_modification = True
 

=== modified file 'lib/lp/bugs/stories/webservice/xx-bug.txt'
--- lib/lp/bugs/stories/webservice/xx-bug.txt	2010-06-30 21:19:36 +0000
+++ lib/lp/bugs/stories/webservice/xx-bug.txt	2010-08-18 13:47:45 +0000
@@ -572,7 +572,7 @@
   ...     firefox_bugtask['self_link'],
   ...     'findSimilarBugs')
   HTTP/1.1 200 Ok...
-  {"total_size": 0, "start": null, "entries": []}
+  {"total_size": 0, "start": 0, "entries": []}
 
 If we add a new bug that's quite similar to others, findSimilarBugs()
 will return something more useful.
@@ -626,7 +626,7 @@
 
     >>> pprint_collection(webservice.named_get(
     ...     '/bugs/%d' % bug.id, 'getNominations').jsonBody())
-    start: None
+    start: 0
     total_size: 0
     ---
 
@@ -1216,7 +1216,7 @@
   ...     bug_one['attachments_collection_link']).jsonBody()
   >>> pprint_collection(attachments)
   resource_type_link: u'http://.../#bug_attachment-page-resource'
-  start: None
+  start: 0
   total_size: 0
   ---
 
@@ -1348,7 +1348,7 @@
   ...     bug_one['attachments_collection_link']).jsonBody()
   >>> pprint_collection(attachments)
   resource_type_link: u'http://api.launchpad.dev/beta/#bug_attachment-page-resource'
-  start: None
+  start: 0
   total_size: 0
   ---
 
@@ -1433,7 +1433,7 @@
     ...     '/ubuntu', 'searchTasks',
     ...     tags=['crash', 'dataloss'],
     ...     tags_combinator='All').jsonBody())
-    start: None
+    start: 0
     total_size: 0
     ---
 
@@ -1442,7 +1442,7 @@
     >>> pprint_collection(webservice.named_get(
     ...     '/ubuntu', 'searchTasks',
     ...     modified_since=u'2011-01-01T00:00:00+00:00').jsonBody())
-    start: None
+    start: 0
     total_size: 0
     ---
 
@@ -1684,7 +1684,7 @@
     ...     '/~testuser3', 'searchTasks'
     ...     ).jsonBody()
     >>> pprint_collection(related)
-    start: None
+    start: 0
     total_size: 0
     ---
 
@@ -1841,7 +1841,7 @@
     HTTP/1.1 200 Ok...
     >>> pprint_collection(webservice.get(bug_one_cves_url).jsonBody())
     resource_type_link: u'http://.../#cve-page-resource'
-    start: None
+    start: 0
     total_size: 0
     ---
 

=== modified file 'lib/lp/registry/stories/webservice/xx-structuralsubscription.txt'
--- lib/lp/registry/stories/webservice/xx-structuralsubscription.txt	2009-12-24 01:41:54 +0000
+++ lib/lp/registry/stories/webservice/xx-structuralsubscription.txt	2010-08-18 13:47:45 +0000
@@ -22,7 +22,7 @@
     >>> subscriptions = webservice.named_get(
     ...     '/fooix', 'getSubscriptions').jsonBody()
     >>> pprint_collection(subscriptions)
-    start: None
+    start: 0
     total_size: 0
     ---
 
@@ -146,6 +146,6 @@
     >>> subscriptions = webservice.named_get(
     ...     '/fooix', 'getSubscriptions').jsonBody()
     >>> pprint_collection(subscriptions)
-    start: None
+    start: 0
     total_size: 0
     ---

=== modified file 'lib/lp/soyuz/stories/webservice/xx-packageset.txt'
--- lib/lp/soyuz/stories/webservice/xx-packageset.txt	2009-11-04 00:00:09 +0000
+++ lib/lp/soyuz/stories/webservice/xx-packageset.txt	2010-08-18 13:47:45 +0000
@@ -197,7 +197,7 @@
     >>> print response
     HTTP/1.1 200 Ok
     ...
-    {"total_size": 0, "start": null, "entries": []}
+    {"total_size": 0, "start": 0, "entries": []}
 
 Let's create a few more package sets and set up a package set hierarchy.
 

=== modified file 'versions.cfg'
--- versions.cfg	2010-08-12 21:02:08 +0000
+++ versions.cfg	2010-08-18 13:47:45 +0000
@@ -24,15 +24,15 @@
 grokcore.component = 1.6
 httplib2 = 0.6.0
 ipython = 0.9.1
-launchpadlib = 1.6.0
+launchpadlib = 1.6.4
 lazr.authentication = 0.1.1
-lazr.batchnavigator = 1.1.1
+lazr.batchnavigator = 1.2.1
 lazr.config = 1.1.3
-lazr.delegates = 1.1.0
+lazr.delegates = 1.2.0
 lazr.enum = 1.1.2
 lazr.lifecycle = 1.1
-lazr.restful = 0.9.29
-lazr.restfulclient = 0.9.14
+lazr.restful = 0.11.1
+lazr.restfulclient = 0.10.0
 lazr.smtptest = 1.1
 lazr.testing = 0.1.1
 lazr.uri = 1.0.2
@@ -71,7 +71,7 @@
 Twisted = 10.1.0
 uuid = 1.30
 van.testing = 2.0.1
-wadllib = 1.1.4
+wadllib = 1.1.5
 webunit = 1.3.8
 # r1440 of lp:~bjornt/windmill/1.3-lp. It includes our patches to make test
 # setup and tear down more robust, which didn't make it into the 1.3 release.