Merge lp:~leonardr/lazr.restful/multiversion-rename-params into lp:lazr.restful

[Leonard Richardson]

[Leonard Richardson (leonardr)]

This branch continues my progress towards making all the named operation annotations work in a multi-versioned environment. This branch gets @rename_parameters_as to work by radically simplifying the BleedThroughDict data structure.

Since BleedThroughDict has only been around for a couple days, I'll explain how it works. It's a stack of dicts that acts like a single dict. It's used to gather annotations in a multi-versioned environment: the 1.0 annotations all go into a dict at the bottom of the stack, the 2.0 annotations go into a dict on top of that, the 3.0 annotations go into a dict on top of _that_, and so on.

Annotations for version n+1 are inherited from n unless redefined for version n. For instance, if the name of the method being published as a named operation is 'foo', but in the definition of version 2.0 there's an @export_operation_as("bar"), then the name of the operation is 'foo' (the default) in 1.0, but 'bar' in 2.0 and 3.0.

My original plan was that each dict would only contain the annotations specific to that version, and inheritance would be implemented by moving down the stack until I found the version that defined the appropriate annotation. In the example above, the stack would look like this:

[(3.0, {}),
 (2.0, {'as' : 'bar'}),
 (1.0, {}]

This is elegant, but it only works for annotations that add objects to the annotation dict. The @rename_parameters_as annotation takes an object already in the dict and modifies its __name__ attribute in place. This means that if you have @rename_parameters_as annotations anywhere in your definition, the last one takes precedence, and modifies the name of that parameter for every single version.

So I changedBleedThroughDict to get the same basic behavior in a much cruder way. When you push a new dict onto the stack, it's initialized with a deep copy of the dict that was originally the top of the stack. In the example above, the stack would look like this:

[(3.0, {'as' : 'bar'}),
 (2.0, {'as' : 'bar'}),
 (1.0, {}]

Now a 2.0 @rename_parameters_as will modify only the copy of the parameter objects found on the 2.0 level of the stack. It won't affect the objects on the 1.0 level. The 3.0 level will inherit the change made on the 2.0 level (thanks to the deep copy), but it it contains its own @rename_parameters_as, that declaration won't affect the 2.0 or the 1.0 levels.

The new BleedThroughDict is not nearly as flexible as the old, but we don't need that flexibility. We always define versions in ascending order and we never go back to modify an old version once it's defined.

I'm entertaining ideas about new names for BleedThroughDict, but I think the name is still appropriate (it's just that the bleeding through happens all at once, not whenever you do a lookup).

[lp:~leonardr/lazr.restful/multiversion-rename-params]

101. By Leonard Richardson on 2010-01-20

Renamed BleedThroughDict to VersionedDict in response to feedback.

[Gary Poster (gary)]

This looks good. I like the simplification.

On IRC I asked how you were going to use this data structure. You said that you would collect the data, then build the top-most version, pop the dict, build the next lower version, pop the dict, and so on. Therefore, this data structure will be very transient, and you don't need an API to access lower versions while the upper versions are still present. That's fine.

At this point, the whole thing could really be accomplished without a new class. Just the ``stack`` list of (name, dict) tuples, and direct calls to deepcopy for a new level in the stack would seem to add very little additional weight in the call sites. I'm tempted to suggest that you go that way, since it would remove code that might have little win, but on balance I'm fine with you seeing how this data structure will work in practice; maybe this class will actually be significantly nicer to work with for your use cases.

For the name, ``VersionedDict`` makes sense to me, particularly with new approach. OTOH, ``BleedThroughDict`` has the vampire-ish connotations that are so in vogue.

Gary

review: Approve

[lp:~leonardr/lazr.restful/multiversion-rename-params]

102. By Leonard Richardson on 2010-01-20

Made sure the cache_for directive works in multiversion.

103. By Leonard Richardson on 2010-01-20

Implemented operation_removed_in_version.

=== modified file 'src/lazr/restful/declarations.py'
--- src/lazr/restful/declarations.py	2010-01-20 15:21:38 +0000
+++ src/lazr/restful/declarations.py	2010-01-20 21:27:13 +0000
@@ -299,7 +299,7 @@
         # in the interface method specification.
         annotations = method.__dict__.get(LAZR_WEBSERVICE_EXPORTED, None)
         if annotations is None:
-            # Create a new bleed-through dict which associates
+            # Create a new versuibed dict which associates
             # annotation data with the earliest active version of the
             # web service. Future @webservice_version annotations will
             # push later versions onto the VersionedDict, allowing
@@ -475,12 +475,27 @@
     def annotate_method(self, method, annotations):
         """See `_method_annotator`."""
         # The annotations dict is a VersionedDict. Push a new dict
-        # onto its stack, labeled with the version number, so that
-        # future annotations can override old annotations without
-        # destroying them.
+        # onto its stack, labeled with the version number, and copy in
+        # the old version's annotations so that this version can
+        # modify those annotations without destroying them.
         annotations.push(self.version)
 
 
+class operation_removed_in_version(operation_for_version):
+    """Decoration removing this operation from the web service.
+
+    This operation will not be present in the given version of the web
+    service, or any subsequent version, unless it's re-published with
+    an export_*_operation method.
+    """
+    def annotate_method(self, method, annotations):
+        """See `_method_annotator`."""
+        # The annotations dict is a VersionedDict. Push a new dict
+        # onto its stack, labeled with the version number. Make sure the
+        # new dict is empty rather than copying the old annotations
+        annotations.push(self.version, True)
+
+
 class export_operation_as(_method_annotator):
     """Decorator specifying the name to export the method as."""
 

=== modified file 'src/lazr/restful/docs/webservice-declarations.txt'
--- src/lazr/restful/docs/webservice-declarations.txt	2010-01-20 13:43:44 +0000
+++ src/lazr/restful/docs/webservice-declarations.txt	2010-01-20 21:27:13 +0000
@@ -503,85 +503,6 @@
     >>> param_defs['optional2'].default
     u'Default2'
 
-Versioning
-----------
-
-Different versions of the webservice can publish the same interface
-method in totally different ways. Here's a simple example. This method
-appears differently in three versions of the web service: 2.0, 1.0,
-and in an unnamed pre-1.0 version.
-
-    >>> from lazr.restful.declarations import operation_for_version
-    >>> class MultiVersionMethod(Interface):
-    ...     export_as_webservice_entry()
-    ...
-    ...     @call_with(fixed='2.0 value')
-    ...     @operation_for_version('2.0')
-    ...
-    ...     @call_with(fixed='1.0 value')
-    ...     @export_operation_as('new_name')
-    ...     @rename_parameters_as(required="required_argument")
-    ...     @operation_for_version('1.0')
-    ...
-    ...     @call_with(fixed='pre-1.0 value')
-    ...     @operation_parameters(
-    ...         required=TextLine(),
-    ...         fixed=TextLine()
-    ...         )
-    ...     @export_read_operation()
-    ...     def a_method(required, fixed='Fixed value'):
-    ...         """Method demonstrating multiversion publication."""
-
-The tagged value containing the annotations looks like a dictionary,
-but it's actually a stack of dictionaries named after the versions.
-
-    >>> dictionary = MultiVersionMethod['a_method'].getTaggedValue(
-    ...     'lazr.restful.exported')
-    >>> dictionary.dict_names
-    [None, '1.0', '2.0']
-
-The dictionary on top of the stack is for the 2.0 version of the web
-service. In 2.0, the method is published as 'new_name' and its 'fixed'
-argument is fixed to the string '2.0 value'.
-
-    >>> print dictionary['as']
-    new_name
-    >>> dictionary['call_with']
-    {'fixed': '2.0 value'}
-
-The published name of the 'required' argument is 'required_argument',
-not 'required'.
-
-    >>> print dictionary['params']['required'].__name__
-    required_argument
-
-Let's pop the 2.0 version off the stack. Now we can see how the method
-looks in 1.0. It's still called 'new_name', and its 'required'
-argument is still called 'required_argument', but its 'fixed' argument
-is fixed to the string '1.0 value'.
-
-    >>> ignored = dictionary.pop()
-    >>> print dictionary['as']
-    new_name
-    >>> dictionary['call_with']
-    {'fixed': '1.0 value'}
-    >>> print dictionary['params']['required'].__name__
-    required_argument
-
-Let's pop one more time to see how the method looks in the pre-1.0
-version. It hasn't yet been renamed to 'new_name', its 'required'
-argument hasn't yet been renamed to 'required_argument', and its
-'fixed' argument is fixed to the string 'pre-1.0 value'.
-
-    >>> ignored = dictionary.pop()
-    >>> print dictionary.get('as')
-    None
-    >>> print dictionary['params']['required'].__name__
-    required
-    >>> dictionary['call_with']
-    {'fixed': 'pre-1.0 value'}
-
-
 Error handling
 --------------
 
@@ -1526,8 +1447,8 @@
 It is possible to cache a server response in the browser cache using
 the @cache_for decorator:
 
+    >>> from lazr.restful.testing.webservice import FakeRequest
     >>> from lazr.restful.declarations import cache_for
-    >>> from lazr.restful.testing.webservice import FakeRequest
     >>>
     >>> class ICachedBookSet(IBookSet):
     ...     """IBookSet supporting caching."""
@@ -1577,6 +1498,212 @@
     ...
     ValueError: Caching duration should be a positive number: -15
 
+Versioned services
+==================
+
+Different versions of the webservice can publish the same data model
+object in totally different ways.
+
+Named operations
+----------------
+
+It's easy to reflect the most common changes between versions:
+operations and arguments being renamed, changes in fixed values, etc.
+This method appears differently in three versions of the web service:
+2.0, 1.0, and in an unnamed pre-1.0 version.
+
+    >>> from lazr.restful.declarations import operation_for_version
+    >>> class MultiVersionMethod(Interface):
+    ...     export_as_webservice_entry()
+    ...
+    ...     @call_with(fixed='2.0 value')
+    ...     @cache_for(300)
+    ...     @operation_for_version('2.0')
+    ...
+    ...     @call_with(fixed='1.0 value')
+    ...     @export_operation_as('new_name')
+    ...     @rename_parameters_as(required="required_argument")
+    ...     @operation_for_version('1.0')
+    ...
+    ...     @call_with(fixed='pre-1.0 value')
+    ...     @cache_for(100)
+    ...     @operation_parameters(
+    ...         required=TextLine(),
+    ...         fixed=TextLine()
+    ...         )
+    ...     @export_read_operation()
+    ...     def a_method(required, fixed='Fixed value'):
+    ...         """Method demonstrating multiversion publication."""
+
+The tagged value containing the annotations looks like a dictionary,
+but it's actually a stack of dictionaries named after the versions.
+
+    >>> dictionary = MultiVersionMethod['a_method'].getTaggedValue(
+    ...     'lazr.restful.exported')
+    >>> dictionary.dict_names
+    [None, '1.0', '2.0']
+
+The dictionary on top of the stack is for the 2.0 version of the web
+service. In 2.0, the method is published as 'new_name' and its 'fixed'
+argument is fixed to the string '2.0 value'.
+
+    >>> print dictionary['as']
+    new_name
+    >>> dictionary['call_with']
+    {'fixed': '2.0 value'}
+    >>> dictionary['cache_for']
+    300
+
+The published name of the 'required' argument is 'required_argument',
+not 'required'.
+
+    >>> print dictionary['params']['required'].__name__
+    required_argument
+
+Let's pop the 2.0 version off the stack. Now we can see how the method
+looks in 1.0. It's still called 'new_name', and its 'required'
+argument is still called 'required_argument', but its 'fixed' argument
+is fixed to the string '1.0 value'.
+
+    >>> ignored = dictionary.pop()
+    >>> print dictionary['as']
+    new_name
+    >>> dictionary['call_with']
+    {'fixed': '1.0 value'}
+    >>> print dictionary['params']['required'].__name__
+    required_argument
+    >>> dictionary['cache_for']
+    100
+
+Let's pop one more time to see how the method looks in the pre-1.0
+version. It hasn't yet been renamed to 'new_name', its 'required'
+argument hasn't yet been renamed to 'required_argument', and its
+'fixed' argument is fixed to the string 'pre-1.0 value'.
+
+    >>> ignored = dictionary.pop()
+    >>> print dictionary.get('as')
+    None
+    >>> print dictionary['params']['required'].__name__
+    required
+    >>> dictionary['call_with']
+    {'fixed': 'pre-1.0 value'}
+    >>> dictionary['cache_for']
+    100
+
+@operation_removed_in_version
+*****************************
+
+Sometimes you want version n+1 to remove a named operation that was
+present in version n. The @operation_removed_in_version declaration
+does just this.
+
+Let's define an operation that's introduced in 1.0 and removed in 2.0.
+
+    >>> from lazr.restful.declarations import operation_removed_in_version
+    >>> class DisappearingMultiversionMethod(Interface):
+    ...     export_as_webservice_entry()
+    ...     @operation_removed_in_version(2.0)
+    ...     @operation_parameters(arg=Float())
+    ...     @export_read_operation()
+    ...     @operation_for_version(1.0)
+    ...     def method(arg):
+    ...         """A doomed method."""
+
+    >>> dictionary = DisappearingMultiversionMethod[
+    ...     'method'].getTaggedValue('lazr.restful.exported')
+
+The method is not present in 2.0:
+
+    >>> version, attrs = dictionary.pop()
+    >>> print version
+    2.0
+    >>> sorted(attrs.items())
+    []
+
+It is present in 1.0:
+
+    >>> version, attrs = dictionary.pop()
+    >>> print version
+    1.0
+    >>> print attrs['type']
+    read_operation
+    >>> print attrs['params']['arg']
+    <zope.schema._field.Float object...>
+
+But it's not present in the unnamed pre-1.0 version, since it hadn't
+been defined yet:
+
+    >>> print dictionary.pop()
+    (None, {})
+
+The @operation_removed_in_version declaration can also be used to
+reset a named operation's definition if you need to completely re-do
+it.
+
+For instance, ordinarily you can't change the type of an operation, or
+totally redefine its parameters--and you shouldn't really need
+to. It's usually easier to publish two different operations that have
+the same name in different versions. But you can do it with a single
+operation, by removing the operation with
+@operation_removed_in_version and defining it again--either in the
+same version or in some later version.
+
+In this example, the type of the operation, the type and number of the
+arguments, and the return value change in version 1.0.
+
+    >>> class ReadOrWriteMethod(Interface):
+    ...     export_as_webservice_entry()
+    ...
+    ...     @operation_parameters(arg=TextLine(), arg2=TextLine())
+    ...     @export_write_operation()
+    ...     @operation_removed_in_version(1.0)
+    ...
+    ...     @operation_parameters(arg=Float())
+    ...     @operation_returns_collection_of(Interface)
+    ...     @export_read_operation()
+    ...     def method(arg, arg2='default'):
+    ...         """A read *or* a write operation, depending on version."""
+
+    >>> dictionary = ReadOrWriteMethod[
+    ...     'method'].getTaggedValue('lazr.restful.exported')
+
+In version 1.0, the 'method' named operation is a write operation that
+takes two TextLine arguments and has no special return value.
+
+    >>> version, attrs = dictionary.pop()
+    >>> print version
+    1.0
+    >>> print attrs['type']
+    write_operation
+    >>> attrs['params']['arg']
+    <zope.schema._bootstrapfields.TextLine object...>
+    >>> attrs['params']['arg2']
+    <zope.schema._bootstrapfields.TextLine object...>
+    >>> print attrs.get('return_type')
+    None
+
+In the unnamed pre-1.0 version, the 'method' operation is a read
+operation that takes a single Float argument and returns a collection.
+
+    >>> version, attrs = dictionary.pop()
+    >>> print attrs['type']
+    read_operation
+
+    >>> attrs['params']['arg']
+    <zope.schema._field.Float object...>
+    >>> attrs['params'].keys()
+    ['arg']
+
+    >>> attrs['return_type']
+    <lazr.restful.fields.CollectionField object...>
+
+[XXX leonardr mutator_for modifies the field, not the method, so it
+won't work until I add multiversion support for fields. Also, it's
+possible to remove a named operation from a certain version and then
+uselessly annotate it some more. The best place to catch this error
+turns out to be when generating the adapter classes, not when
+collecting annotations.]
+
 Security
 ========