Merge lp:~leonardr/lazr.restful/cache-service-root into lp:lazr.restful

[Leonard Richardson]

This branch gets lazr.restful to set the Cache-Control header when serving a representation of the service root. This is a big win because the service root only changes when you deploy a new version of your lazr.restful application (eg. Launchpad), but every user makes 2 requests for it every time they start up lazr.restfulclient.

There are two 'max-age' times set in the Cache-Control header: one for the latest version of the web service (which changes more often) and one for all other versions. By default, the max-age for the latest version is one hour and the max-age for all other versions is one week.

After the max-age expires, the client will make a *conditional* request to see whether the service root has changed. If it hasn't, I believe the client will update the client-side max-age and not make any requests for another hour (or week). But I need to check to make sure httplib2 actually does this.

[Gavin Panella (allenap)]

Nice branch. Conversation from IRC:

<allenap> leonardr: Is there any way config.active_versions could be empty?
<leonardr> allenap: no, that would prevent lazr.restful from starting up
<allenap> leonardr: Instead of having [604800, 3600], could you have a module-level constant like `HOUR = 3600 # seconds`, then the default can be specified as [7 * 24 * HOUR, 1 * HOUR]?
<leonardr> sure

review: Approve

[lp:~leonardr/lazr.restful/cache-service-root]

127. By Leonard Richardson on 2010-04-07

Set the Date header as well, since httplib2 uses it to determine whether a cached representation is stale.

[Leonard Richardson (leonardr)]

In my experiments with lazr.restfulclient I discovered that httplib2 will ignore Cache-Control unless the Date header is also set to provide a starting point. I've updated the branch to set the Date header.

[Gavin Panella (allenap)]

The late addition, r127, looks good too.

review: Approve

[lp:~leonardr/lazr.restful/cache-service-root]

128. By Leonard Richardson on 2010-04-07

Set Cache-Control and Date even on a conditional request.

129. By Leonard Richardson on 2010-04-07

Set caching headers whether the request was conditional or not.

[Leonard Richardson (leonardr)]

UNfortunately, this branch breaks lazr.restfulclient due to what I believe is a bug in httplib2 (http://code.google.com/p/httplib2/issues/detail?id=97). We can still land this branch, but we'll only be able to serve the goodness to clients that signal they can work around the bug. This means changing clients to send custom values for User-Agent.

[lp:~leonardr/lazr.restful/cache-service-root]

130. By Leonard Richardson on 2010-04-13

Don't serve cache control headers to httplib2 clients to avoid triggering a bug in httplib2.

131. By Leonard Richardson on 2010-04-13

Removed temporary hack for testing purposes.

[Leonard Richardson (leonardr)]

All right, I've changed lazr.restful to avoid triggering the httplib2 bug on old versions of lazr.restfulclient. (I've tested this version of lazr.restful with an old and a new lazr.restfulclient, and they both worked, though obviously the old one didn't get the benefit of the Cache-Control headers.)

[lp:~leonardr/lazr.restful/cache-service-root]

132. By Leonard Richardson on 2010-04-13

Slight refactoring.

[Gavin Panella (allenap)]

Still looking good :)

review: Approve

=== modified file 'src/lazr/restful/NEWS.txt'
--- src/lazr/restful/NEWS.txt	2010-03-17 14:38:16 +0000
+++ src/lazr/restful/NEWS.txt	2010-04-13 15:37:34 +0000
@@ -2,6 +2,18 @@
 NEWS for lazr.restful
 =====================
 
+Development
+===========
+
+Special note: This version introduces a new configuration element,
+'caching_policy'. This element starts out simple but may become more
+complex in future versions. See the IWebServiceConfiguration interface
+for more details.
+
+Service root resources are now client-side cacheable for an amount of
+time that depends on the server configuration and the version of the
+web service requested.
+
 0.9.24 (2010-03-17)
 ====================
 

=== modified file 'src/lazr/restful/_resource.py'
--- src/lazr/restful/_resource.py	2010-03-16 15:07:00 +0000
+++ src/lazr/restful/_resource.py	2010-04-13 15:37:34 +0000
@@ -32,9 +32,11 @@
 import copy
 from cStringIO import StringIO
 from datetime import datetime, date
+from email.Utils import formatdate
 from gzip import GzipFile
 import os
 import simplejson
+import time
 import zlib
 
 # Import SHA in a way compatible with both Python 2.4 and Python 2.6.
@@ -1621,10 +1623,36 @@
             result = ""
         return self.applyTransferEncoding(result)
 
+    def setCachingHeaders(self):
+        "How long should the client cache this service root?"
+        user_agent = self.request.getHeader('User-Agent', '')
+        if user_agent.startswith('Python-httplib2'):
+            # XXX leonardr 20100412
+            # bug=http://code.google.com/p/httplib2/issues/detail?id=97
+            #
+            # A client with a User-Agent of "Python/httplib2" (such as
+            # old versions of lazr.restfulclient) gives inconsistent
+            # results when a resource is served with both ETag and
+            # Cache-Control. We check for that User-Agent and omit the
+            # Cache-Control headers if it makes a request.
+            return
+        config = getUtility(IWebServiceConfiguration)
+        caching_policy = config.caching_policy
+        if self.request.version == config.active_versions[-1]:
+            max_age = caching_policy[-1]
+        else:
+            max_age = caching_policy[0]
+        if max_age > 0:
+            self.request.response.setHeader(
+                'Cache-Control', 'max-age=%d' % max_age)
+            # Also set the Date header so that client-side caches will
+            # have something to work from.
+            self.request.response.setHeader('Date', formatdate(time.time()))
+
     def do_GET(self):
-        """Describe the capabilities of the web service in WADL."""
-
+        """Describe the capabilities of the web service."""
         media_type = self.handleConditionalGET()
+        self.setCachingHeaders()
         if media_type is None:
             # The conditional GET succeeded. Serve nothing.
             return ""

=== modified file 'src/lazr/restful/example/base/root.py'
--- src/lazr/restful/example/base/root.py	2010-03-10 18:45:04 +0000
+++ src/lazr/restful/example/base/root.py	2010-04-13 15:37:34 +0000
@@ -385,6 +385,7 @@
 
 class WebServiceConfiguration(BaseWebServiceConfiguration):
     directives.publication_class(WebServiceTestPublication)
+    caching_policy = [10000, 2]
     code_revision = 'test.revision'
     default_batch_size = 5
     hostname = 'cookbooks.dev'

=== modified file 'src/lazr/restful/example/base/tests/root.txt'
--- src/lazr/restful/example/base/tests/root.txt	2010-02-11 17:57:16 +0000
+++ src/lazr/restful/example/base/tests/root.txt	2010-04-13 15:37:34 +0000
@@ -173,3 +173,91 @@
 
     >>> print top_level_links['featured_cookbook_link']
     http://.../cookbooks/featured
+
+Caching policy
+==============
+
+The service root resource is served with the Cache-Control header
+giving a configurable value for "max-age". An old version of the
+service root can be cached for a long time:
+
+    >>> response = webservice.get('/', api_version='1.0')
+    >>> print response.getheader('Cache-Control')
+    max-age=10000
+
+The latest version of the service root should be cached for less time.
+
+    >>> response = webservice.get('/', api_version='devel')
+    >>> print response.getheader('Cache-Control')
+    max-age=2
+
+Both the WADL and JSON representations of the service root are
+cacheable.
+
+    >>> wadl_type = 'application/vnd.sun.wadl+xml'
+    >>> response = webservice.get('/', wadl_type)
+    >>> print response.getheader('Cache-Control')
+    max-age=2
+
+The Date header is set along with Cache-Control so that the client can
+easily determine when the cache is stale.
+
+    >>> response.getheader('Date') is None
+    False
+
+Date and Cache-Control are set even when the request is a conditional
+request where the condition failed.
+
+    >>> etag = response.getheader('ETag')
+    >>> conditional_response = webservice.get(
+    ...     '/', wadl_type, headers={'If-None-Match' : etag})
+    >>> conditional_response.status
+    304
+    >>> print conditional_response.getheader('Cache-Control')
+    max-age=2
+    >>> conditional_response.getheader('Date') is None
+    False
+
+To avoid triggering a bug in httplib2, lazr.restful does not send the
+Cache-Control or Date headers to clients that identify as
+Python-httplib2.
+
+    # XXX leonardr 20100412
+    # bug=http://code.google.com/p/httplib2/issues/detail?id=97
+    >>> agent = 'Python-httplib2/$Rev: 259$'
+    >>> response = webservice.get(
+    ...     '/', wadl_type, headers={'User-Agent' : agent})
+    >>> print response.getheader('Cache-Control')
+    None
+    >>> print response.getheader('Date')
+    None
+
+If the client identifies as an agent _based on_ httplib2, we take a
+chance and send the Cache-Control headers.
+
+    >>> agent = "Custom client (%s)" % agent
+    >>> response = webservice.get(
+    ...     '/', wadl_type, headers={'User-Agent' : agent})
+    >>> print response.getheader('Cache-Control')
+    max-age=2
+    >>> response.getheader('Date') is None
+    False
+
+If the caching policy says not to cache the service root resource at
+all, the Cache-Control and Date headers are not present.
+
+    >>> from zope.component import getUtility
+    >>> from lazr.restful.interfaces import IWebServiceConfiguration
+    >>> policy = getUtility(IWebServiceConfiguration).caching_policy
+    >>> old_value = policy[-1]
+    >>> policy[-1] = 0
+
+    >>> response = webservice.get('/')
+    >>> print response.getheader('Cache-Control')
+    None
+    >>> response.getheader('Date') is None
+    True
+
+Cleanup.
+
+    >>> policy[-1] = old_value

=== modified file 'src/lazr/restful/interfaces/_rest.py'
--- src/lazr/restful/interfaces/_rest.py	2010-03-10 19:03:11 +0000
+++ src/lazr/restful/interfaces/_rest.py	2010-04-13 15:37:34 +0000
@@ -63,6 +63,9 @@
     IBrowserRequest, IDefaultBrowserLayer)
 from lazr.batchnavigator.interfaces import InvalidBatchSizeError
 
+# Constants for periods of time
+HOUR = 3600 # seconds
+
 # The namespace prefix for LAZR web service-related tags.
 LAZR_WEBSERVICE_NS = 'lazr.restful'
 
@@ -394,6 +397,17 @@
     These are miscellaneous strings that may differ in different web
     services.
     """
+    caching_policy = List(
+        value_type=Int(),
+        default = [7 * 24 * HOUR, 1 * HOUR],
+        title=u"The web service caching policy.",
+        description = u"""A list of two numbers, each to be used in the
+        'max-age' field of the Cache-Control header. The first number is
+        used when serving the service root for any web service version
+        except the latest one. The second number is used when serving the
+        service root for the latest version (which probably changes more
+        often).""")
+
     service_description = TextLine(
         title=u"Service description",
         description=u"""A human-readable description of the web service.