Merge lp:~juju/txzookeeper/basic-node-api into lp:txzookeeper

[Kapil Thangavelu]

a minimal node abstraction api, with get/set data/acl and node iteration.

[Gustavo Niemeyer (niemeyer)]

Just for the record, as we discussed over the phone:

[1]

The notification abstractions hide a bit the intention behind the way that the ZooKeeper API is designed: watchers are set up on data retrieval because it only makes sense to be told about changes once you know the existing value, and once changes happen the client must ask again about the value and reset the notification if it's still interested in knowing about changes.

As a consequence of this model, fast subsequent changes won't create a swarm effect, because all the clients are told about is that the value isn't the same they originally knew about anymore, but more than one change may have happened, and that's still alright.

With that in mind, I suggest having something like this, in addition to get_data() and get_children():

    def get_data_and_watch(self):
        (...)
        return data_deferred, watch_deferred

    def get_children_and_watch(self):
        (...)
        return children_deferred, watch_deferred

This uses twisted deferreds as well.

[2]

This isn't about a specific action required.. just a feeling I had.

I understand the intention behind the internal keeping of the node versioning, and I think it's a good model to start with. It's just not entirely clear to me how we'll want these things to behave in practice, but we'll certainly figure it out.

review: Needs Fixing

[lp:~juju/txzookeeper/basic-node-api]

19. By Kapil Thangavelu on 2010-05-21

node - remove subscriptions methods, add additional data accessors which also return on change deferreds

20. By Kapil Thangavelu on 2010-05-24

re-enable the subscription tests as data accessors with watches, add an additional test for bad version error handling.

21. By Kapil Thangavelu on 2010-05-24

rename exists_with_watch to exists_and_watch for consistency.

[Kapil Thangavelu (hazmat)]

On Wed, 19 May 2010 05:43:22 -0400, Gustavo Niemeyer
<email address hidden> wrote:

> Review: Needs Fixing
> Just for the record, as we discussed over the phone:
>
> [1]
>
> The notification abstractions hide a bit the intention behind the way
> that the ZooKeeper API is designed: watchers are set up on data
> retrieval because it only makes sense to be told about changes once you
> know the existing value, and once changes happen the client must ask
> again about the value and reset the notification if it's still
> interested in knowing about changes.
>
> As a consequence of this model, fast subsequent changes won't create a
> swarm effect, because all the clients are told about is that the value
> isn't the same they originally knew about anymore, but more than one
> change may have happened, and that's still alright.
>
> With that in mind, I suggest having something like this, in addition to
> get_data() and get_children():
>
> def get_data_and_watch(self):
> (...)
> return data_deferred, watch_deferred
>
> def get_children_and_watch(self):
> (...)
> return children_deferred, watch_deferred
>
> This uses twisted deferreds as well.

i've gone and ahead and switched the node implementation to utilize the
accesssors with explicit watches. i ended up adding an additional one for
exists_and_watch, as its nesc. to detect node creation.

>
> [2]
>
> This isn't about a specific action required.. just a feeling I had.
>
> I understand the intention behind the internal keeping of the node
> versioning, and I think it's a good model to start with. It's just not
> entirely clear to me how we'll want these things to behave in practice,
> but we'll certainly figure it out.
>

best usage is still tbd but there should be some value to the abstraction
layer else we're just using the client api. i think we'll end up
revisiting with some more experience.

cheers,

Kapil

[Kapil Thangavelu (hazmat)]

On Wed, 19 May 2010 05:43:22 -0400, Gustavo Niemeyer
<email address hidden> wrote:

> Review: Needs Fixing
> Just for the record, as we discussed over the phone:
>
> [1]
>
> The notification abstractions hide a bit the intention behind the way
> that the ZooKeeper API is designed: watchers are set up on data
> retrieval because it only makes sense to be told about changes once you
> know the existing value, and once changes happen the client must ask
> again about the value and reset the notification if it's still
> interested in knowing about changes.
>
> As a consequence of this model, fast subsequent changes won't create a
> swarm effect, because all the clients are told about is that the value
> isn't the same they originally knew about anymore, but more than one
> change may have happened, and that's still alright.
>
> With that in mind, I suggest having something like this, in addition to
> get_data() and get_children():
>
> def get_data_and_watch(self):
> (...)
> return data_deferred, watch_deferred
>
> def get_children_and_watch(self):
> (...)
> return children_deferred, watch_deferred
>
> This uses twisted deferreds as well.

I've gone ahead and implemented the data accessors with watches.
>
> [2]
>
> This isn't about a specific action required.. just a feeling I had.
>
> I understand the intention behind the internal keeping of the node
> versioning, and I think it's a good model to start with. It's just not
> entirely clear to me how we'll want these things to behave in practice,
> but we'll certainly figure it out.

noted. we'll revisit based on experience.

[Gustavo Niemeyer (niemeyer)]

[3]

Thanks for the watch implementation. It looks sensible.

As we discussed, one thing I wonder is if we should change the watch parameter into an Event object with "node" (the object, rather than the path), "kind", and "status". Might potentially make it more comfortable to consume the event.

[4]

+ def __cmp__(self, other):
+ return cmp(self.path, other.path)

This should be turned into __eq__ I believe, otherwise we'll get weird calls like node1 < node2 working in an arbitrary way, when they should error out.

[5]

+ if self._node_exists:
+ # if the node is deleted while we have this cached state we should
+ # covert the set to a create. XXX ??? Should we

On a second look, this starts to smell a bit strange. If we look at the algorithms created around ZooKeeper, they're generally very sensitive about when things are created, changed, deleted, etc. Having our Node API doing things behind the programmer's back feels like a hole we shouldn't get into. It reminds me of the sqlite Python wrapper, which does a lot of "smart" magic with the transaction, and as a side effect turns transactions into something unreliable (which is very ironic).

I understand and agree that having a create() method in a Node object which already exists may feel a bit weird from an OO perspective, but I really think we should go towards this way if we want to make the logic being created on top of this really easy to follow.

If we don't do this, we'll always to have to keep in mind "Ohh, this method call actually *creates* the node too!" when thinking of higher-level algorithms on top of ZooKeeper, which feels undesirable.

What do you think?

review: Needs Fixing

[lp:~juju/txzookeeper/basic-node-api]

22. By Kapil Thangavelu on 2010-06-10

use failure trap for excpetion handlers, properties for node name&path.

23. By Kapil Thangavelu on 2010-06-10

explicit create method, explicit errors on get data, and set data on non existant nodes.

24. By Kapil Thangavelu on 2010-06-11

node no longer tracks existance, just last node version stat

25. By Kapil Thangavelu on 2010-06-11

node event object to aggregate watch callback values

26. By Kapil Thangavelu on 2010-06-11

node event encapsulates node object that the watch was fired on

[Kapil Thangavelu (hazmat)]

[3] There's now an event object wrapping the watch callback values along with the node object.

[4] This is for sorting node collections, and is deterministic. It might be uintended in the node1 < node2 case. Alternative would be either providing a list compare function in the module, or letting users define one. Providing it on the class directly depends on how common the sorting nodes would be vs. confusion on other __cmp__ usages.

[5] All the implicit stuff has been yanked except the last version tracking. There is now an explicit create method on the node.

[lp:~juju/txzookeeper/basic-node-api]

27. By Kapil Thangavelu on 2010-06-11

test for nodeevent repr

[Gustavo Niemeyer (niemeyer)]

Nice improvements again, thank you!

Only one comment, and +1!

[6]

+ def __init__(self, event_type, connection_state, node):
+ self.type = event_type
+ self.connection_state = connection_state
+ self.node = node
+ self.path = node.path
+
+ @property
+ def kind(self):
+ return self.kind_map[self.type]

It'd be nice to have the naming convention consistent here. event.type returns an integer, and event.kind returns a string, but they both mean exactly the same thing with a different data type. As a side effect, a few weeks from now it will be hard to remember whether the string is the type or the kind. I suggest something like kind and kind_name, or type and type_name.

review: Approve

=== added file 'txzookeeper/node.py'
--- txzookeeper/node.py	1970-01-01 00:00:00 +0000
+++ txzookeeper/node.py	2010-06-11 16:18:25 +0000
@@ -0,0 +1,229 @@
+from zookeeper import NoNodeException, BadVersionException
+from twisted.internet.defer import Deferred
+from txzookeeper.client import ZOO_OPEN_ACL_UNSAFE
+
+
+class NodeEvent(object):
+    """
+    A node event is returned when a watch deferred fires. It denotes some event
+    on the zookeeper node that the watch was requested on.
+
+    @ivar path: Path to the node the event was about.
+
+    @ivar type: An integer corresponding to the event type. The symbolic
+    name mapping is available from the zookeeper module attributes. For
+    convience one is included on the C{NodeEvent} class.
+
+    @ivar kind: A symbolic name for the event's type.
+
+    @ivar connection_state: integer representing the state of the
+    zookeeper connection.
+    """
+
+    __slots__ = ('type', 'connection_state', 'path', 'node')
+
+    kind_map = {
+        1: 'created',
+        2: 'deleted',
+        3: 'changed',
+        4: 'child'}
+
+    def __init__(self, event_type, connection_state, node):
+        self.type = event_type
+        self.connection_state = connection_state
+        self.node = node
+        self.path = node.path
+
+    @property
+    def kind(self):
+        return self.kind_map[self.type]
+
+    def __repr__(self):
+        return  "<NodeEvent %s at %r>"%(self.kind, self.path)
+
+
+class ZNode(object):
+    """
+    A minimal object abstraction over a zookeeper node, utilizes no caching
+    outside of trying to keep track of node existance and the last read
+    version. It will attempt to utilize the last read version when modifying
+    the node. On a bad version exception this values are cleared and the
+    except reraised (errback chain continue.)
+    """
+
+    def __init__(self, path, context):
+        self._path = path
+        self._name = path.split("/")[-1]
+        self._context = context
+        self._node_stat = None
+
+    @property
+    def path(self):
+        """
+        The path to the node from the zookeeper root.
+        """
+        return self._path
+
+    @property
+    def name(self):
+        """
+        The name of the node in its container.
+        """
+        return self._name
+
+    def _get_version(self):
+        if not self._node_stat:
+            return -1
+        return self._node_stat["version"]
+
+    def _on_error_bad_version(self, failure):
+        failure.trap(BadVersionException)
+        self._node_stat = None
+        return failure
+
+    def create(self, data="", acl=None, flags=0):
+        """
+        Create the node with the given data, acl, and persistence flags. If no
+        acl is given the default public acl is used. The default creation flag
+        is persistent.
+        """
+        if acl is None:
+            acl = [ZOO_OPEN_ACL_UNSAFE]
+        d = self._context.create(self.path, data, acl, flags)
+        return d
+
+    def _on_exists_success(self, node_stat):
+        if node_stat is None:
+            return False
+        self._node_stat = node_stat
+        return True
+
+    def exists(self):
+        """
+        Does the node exist or not, returns a boolean.
+        """
+        d = self._context.exists(self.path)
+        d.addCallback(self._on_exists_success)
+        return d
+
+    def exists_and_watch(self):
+        """
+        Returns a boolean based on the node's existence. Also returns a
+        deferred that fires when the node is modified/created/added/deleted.
+        """
+        node_changed = Deferred()
+
+        def on_node_event(event, state, path):
+            return node_changed.callback(
+                NodeEvent(event, state, self))
+
+        d = self._context.exists(self.path, on_node_event)
+        d.addCallback(self._on_exists_success)
+        return d, node_changed
+
+    def _on_get_node_error(self, failure):
+        failure.trap(NoNodeException)
+        self._node_stat = None
+        return failure
+
+    def _on_get_node_success(self, data):
+        (node_data, node_stat) = data
+        self._node_stat = node_stat
+        return node_data
+
+    def get_data(self):
+        """Retrieve the node's data."""
+        d = self._context.get(self.path)
+        d.addCallback(self._on_get_node_success)
+        d.addErrback(self._on_get_node_error)
+        return d
+
+    def get_data_and_watch(self):
+        """
+        Retrieve the node's data and a deferred that fires when this data
+        changes.
+        """
+        node_changed = Deferred()
+
+        def on_node_change(event, status, path):
+            node_changed.callback(
+                NodeEvent(event, status, self))
+
+        d = self._context.get(self.path, watcher=on_node_change)
+        d.addCallback(self._on_get_node_success)
+        d.addErrback(self._on_get_node_error)
+        return d, node_changed
+
+    def set_data(self, data):
+        """Set the node's data."""
+
+        def on_success(value):
+            self._node_stat = None
+            return self
+
+        version = self._get_version()
+        d = self._context.set(self.path, data, version)
+        d.addErrback(self._on_error_bad_version)
+        d.addCallback(on_success)
+        return d
+
+    def get_acl(self):
+        """
+        Get the ACL for this node. An ACL is a list of access control
+        entity dictionaries.
+        """
+
+        def on_success((acl, stat)):
+            if stat is not None:
+                self._node_stat = stat
+            return acl
+        d = self._context.get_acl(self.path)
+        d.addCallback(on_success)
+        return d
+
+    def set_acl(self, acl):
+        """
+        Set the ACL for this node.
+        """
+        d = self._context.set_acl(
+            self.path, acl, self._get_version())
+        d.addErrback(self._on_error_bad_version)
+        return d
+
+    def _on_get_children_filter_results(self, children, prefix):
+        if prefix:
+            children = [
+                name for name in children if name.startswith(prefix)]
+        return [
+            self.__class__("/".join((self.path, name)), self._context)
+            for name in children]
+
+    def get_children(self, prefix=None):
+        """
+        Get the children of this node, as ZNode objects. Optionally
+        a name prefix may be passed which the child node must abide.
+        """
+        d = self._context.get_children(self.path)
+        d.addCallback(self._on_get_children_filter_results, prefix)
+        return d
+
+    def get_children_and_watch(self, prefix=None):
+        """
+        Get the children of this node, as ZNode objects, and also return
+        a deferred that fires if a child is added or deleted. Optionally
+        a name prefix may be passed which the child node must abide.
+        """
+        children_changed = Deferred()
+
+        def on_child_added_removed(event, status, path):
+            # path is the container not the child.
+            children_changed.callback(
+                NodeEvent(event, status, self))
+
+        d = self._context.get_children(
+            self.path, watcher=on_child_added_removed)
+        d.addCallback(self._on_get_children_filter_results, prefix)
+        return d, children_changed
+
+    def __cmp__(self, other):
+        return cmp(self.path, other.path)
 
=== added file 'txzookeeper/tests/test_node.py'
--- txzookeeper/tests/test_node.py	1970-01-01 00:00:00 +0000
+++ txzookeeper/tests/test_node.py	2010-06-11 16:18:25 +0000
@@ -0,0 +1,316 @@
+import zookeeper
+import hashlib
+import base64
+
+from twisted.internet.defer import Deferred, inlineCallbacks
+from twisted.python.failure import Failure
+from txzookeeper.node import ZNode
+from txzookeeper.tests import TestCase
+from txzookeeper.tests.utils import deleteTree
+from txzookeeper import ZookeeperClient
+
+
+class NodeTest(TestCase):
+
+    def setUp(self):
+        super(NodeTest, self).setUp()
+        zookeeper.set_debug_level(zookeeper.LOG_LEVEL_ERROR)
+        self.client = ZookeeperClient("127.0.0.1:2181", 2000)
+        d = self.client.connect()
+        self.client2 = None
+
+        def create_zoo(client):
+            client.create("/zoo")
+        d.addCallback(create_zoo)
+        return d
+
+    def tearDown(self):
+        super(NodeTest, self).tearDown()
+        deleteTree(handle=self.client.handle)
+        if self.client.connected:
+            self.client.close()
+        if self.client2 and self.client2.connected:
+            self.client2.close()
+        zookeeper.set_debug_level(zookeeper.LOG_LEVEL_DEBUG)
+
+    def _make_digest_identity(self, credentials):
+        user, password = credentials.split(":")
+        digest = hashlib.new("sha1", credentials).digest()
+        return "%s:%s"%(user, base64.b64encode(digest))
+
+    def test_node_name_and_path(self):
+        """
+        Each node has name and path.
+        """
+        node = ZNode("/zoo/rabbit", self.client)
+        self.assertEqual(node.name, "rabbit")
+        self.assertEqual(node.path, "/zoo/rabbit")
+
+    @inlineCallbacks
+    def test_node_exists_nonexistant(self):
+        """
+        A node knows whether it exists or not.
+        """
+        node = ZNode("/zoo/rabbit", self.client)
+        exists = yield node.exists()
+        self.assertFalse(exists)
+
+    @inlineCallbacks
+    def test_node_set_data_on_nonexistant(self):
+        """
+        Setting data on a non existant node raises a no node exception.
+        """
+        node = ZNode("/zoo/rabbit", self.client)
+        d = node.set_data("big furry ears")
+        self.failUnlessFailure(d, zookeeper.NoNodeException)
+        yield d
+
+    @inlineCallbacks
+    def test_node_create_set_data(self):
+        """
+        A node can be created and have its data set.
+        """
+        node = ZNode("/zoo/rabbit", self.client)
+        data = "big furry ears"
+        yield node.create(data)
+        exists = yield self.client.exists("/zoo/rabbit")
+        self.assertTrue(exists)
+        node_data = yield node.get_data()
+        self.assertEqual(data, node_data)
+        data = data*2
+
+        yield node.set_data(data)
+        node_data = yield node.get_data()
+        self.assertEqual(data, node_data)
+
+    @inlineCallbacks
+    def test_node_get_data(self):
+        """
+        Data can be fetched from a node.
+        """
+        yield self.client.create("/zoo/giraffe", "mouse")
+        data = yield ZNode("/zoo/giraffe", self.client).get_data()
+        self.assertEqual(data, "mouse")
+
+    @inlineCallbacks
+    def test_node_get_data_nonexistant(self):
+        """
+        Attempting to fetch data from a nonexistant node returns
+        a non existant error.
+        """
+        d = ZNode("/zoo/giraffe", self.client).get_data()
+        self.failUnlessFailure(d, zookeeper.NoNodeException)
+        yield d
+
+    @inlineCallbacks
+    def test_node_get_acl(self):
+        """
+        The ACL for a node can be retrieved.
+        """
+        yield self.client.create("/zoo/giraffe")
+        acl = yield ZNode("/zoo/giraffe", self.client).get_acl()
+        self.assertEqual(len(acl), 1)
+        self.assertEqual(acl[0]['scheme'], 'world')
+
+    def test_node_get_acl_nonexistant(self):
+        """
+        The fetching the ACL for a non-existant node results in an error.
+        """
+        node = ZNode("/zoo/giraffe", self.client)
+
+        def assert_failed(failed):
+            if not isinstance(failed, Failure):
+                self.fail("Should have failed")
+            self.assertTrue(
+                isinstance(failed.value, zookeeper.NoNodeException))
+        d = node.get_acl()
+        d.addBoth(assert_failed)
+        return d
+
+    @inlineCallbacks
+    def test_node_set_acl(self):
+        """
+        The ACL for a node can be modified.
+        """
+        path = yield self.client.create("/zoo/giraffe")
+        credentials = "zebra:moon"
+        acl = [{"id": self._make_digest_identity(credentials),
+                "scheme": "digest",
+                "perms":zookeeper.PERM_ALL}]
+        node = ZNode(path, self.client)
+        # little hack around slow auth issue 770 zookeeper
+        d = self.client.add_auth("digest", credentials)
+        yield node.set_acl(acl)
+        yield d
+        node_acl, stat = yield self.client.get_acl(path)
+        self.assertEqual(node_acl, acl)
+
+    @inlineCallbacks
+    def test_node_set_data_update_with_cached_exists(self):
+        """
+        Data can be set on an existing node, updating it
+        in place.
+        """
+        node = ZNode("/zoo/monkey", self.client)
+        yield self.client.create("/zoo/monkey", "stripes")
+        exists = yield node.exists()
+        self.assertTrue(exists)
+        yield node.set_data("banana")
+        data, stat = yield self.client.get("/zoo/monkey")
+        self.assertEqual(data, "banana")
+
+    @inlineCallbacks
+    def test_node_set_data_update_with_invalid_cached_exists(self):
+        """
+        If a node is deleted, attempting to set data on it
+        raises a no node exception.
+        """
+        node = ZNode("/zoo/monkey", self.client)
+        yield self.client.create("/zoo/monkey", "stripes")
+        exists = yield node.exists()
+        self.assertTrue(exists)
+        yield self.client.delete("/zoo/monkey")
+        d = node.set_data("banana")
+        self.failUnlessFailure(d, zookeeper.NoNodeException)
+        yield d
+
+    @inlineCallbacks
+    def test_node_set_data_update_with_exists(self):
+        """
+        Data can be set on an existing node, updating it
+        in place.
+        """
+        node = ZNode("/zoo/monkey", self.client)
+        yield self.client.create("/zoo/monkey", "stripes")
+        yield node.set_data("banana")
+        data, stat = yield self.client.get("/zoo/monkey")
+        self.assertEqual(data, "banana")
+
+    @inlineCallbacks
+    def test_node_exists_with_watch_nonexistant(self):
+        """
+        The node's existance can be checked with the exist_watch api
+        a deferred will be returned and any node level events,
+        created, deleted, modified invoke the callback. You can
+        get these create event callbacks for non existant nodes.
+        """
+        node = ZNode("/zoo/elephant", self.client)
+        exists, watch = yield node.exists_and_watch()
+
+        self.client.create("/zoo/elephant")
+        event = yield watch
+        self.assertEqual(event.type, zookeeper.CREATED_EVENT)
+        self.assertEqual(event.path, node.path)
+
+    @inlineCallbacks
+    def test_node_get_data_with_watch_on_update(self):
+        """
+        Subscribing to a node will get node update events.
+        """
+        yield self.client.create("/zoo/elephant")
+
+        node = ZNode("/zoo/elephant", self.client)
+        data, watch = yield node.get_data_and_watch()
+        yield self.client.set("/zoo/elephant")
+        event = yield watch
+        self.assertEqual(event.type, zookeeper.CHANGED_EVENT)
+        self.assertEqual(event.path, "/zoo/elephant")
+
+    @inlineCallbacks
+    def test_node_get_data_with_watch_on_delete(self):
+        """
+        Subscribing to a node will get node deletion events.
+        """
+        yield self.client.create("/zoo/elephant")
+
+        node = ZNode("/zoo/elephant", self.client)
+        data, watch = yield node.get_data_and_watch()
+
+        yield self.client.delete("/zoo/elephant")
+        event = yield watch
+        self.assertEqual(event.type, zookeeper.DELETED_EVENT)
+        self.assertEqual(event.path, "/zoo/elephant")
+
+    @inlineCallbacks
+    def test_node_children(self):
+        """
+        A node's children can be introspected.
+        """
+        node = ZNode("/zoo", self.client)
+        node_path_a = yield self.client.create("/zoo/lion")
+        node_path_b = yield self.client.create("/zoo/tiger")
+        children = yield node.get_children()
+        children.sort()
+        self.assertEqual(children[0].path, node_path_a)
+        self.assertEqual(children[1].path, node_path_b)
+
+    @inlineCallbacks
+    def test_node_children_by_prefix(self):
+        """
+        A node's children can be introspected optionally with a prefix.
+        """
+        node = ZNode("/zoo", self.client)
+        node_path_a = yield self.client.create("/zoo/lion")
+        yield self.client.create("/zoo/tiger")
+        children = yield node.get_children("lion")
+        children.sort()
+        self.assertEqual(children[0].path, node_path_a)
+        self.assertEqual(len(children), 1)
+
+    @inlineCallbacks
+    def test_node_get_children_with_watch_create(self):
+        """
+        A node's children can explicitly be watched to given existance
+        events for node creation and destruction.
+        """
+        node = ZNode("/zoo", self.client)
+        children, watch = yield node.get_children_and_watch()
+        yield self.client.create("/zoo/lion")
+        event = yield watch
+        self.assertEqual(event.path, "/zoo")
+        self.assertEqual(event.type, zookeeper.CHILD_EVENT)
+        self.assertEqual(event.kind, "child")
+
+    @inlineCallbacks
+    def test_node_get_children_with_watch_delete(self):
+        """
+        A node's children can explicitly be watched to given existance
+        events for node creation and destruction.
+        """
+        node = ZNode("/zoo", self.client)
+        yield self.client.create("/zoo/lion")
+        children, watch = yield node.get_children_and_watch()
+        yield self.client.delete("/zoo/lion")
+        event = yield watch
+        self.assertEqual(event.path, "/zoo")
+        self.assertEqual(event.type, zookeeper.CHILD_EVENT)
+        self.assertEqual(repr(event),
+                         "<NodeEvent child at '/zoo'>")
+
+    @inlineCallbacks
+    def test_bad_version_error(self):
+        """
+        The node captures the node version on any read operations, which
+        it utilizes for write operations. On a concurrent modification error
+        the node return a bad version error, this also clears the cached
+        state so subsequent modifications will be against the latest version,
+        unless the cache is seeded again by a read operation.
+        """
+        node = ZNode("/zoo/lion", self.client)
+
+        self.client2 = ZookeeperClient("127.0.0.1:2181")
+        yield self.client2.connect()
+
+        yield self.client.create("/zoo/lion", "mouse")
+        yield node.get_data()
+        yield self.client2.set("/zoo/lion", "den2")
+        data = yield self.client.exists("/zoo/lion")
+        self.assertEqual(data['version'], 1)
+        d = node.set_data("zebra")
+        self.failUnlessFailure(d, zookeeper.BadVersionException)
+        yield d
+
+        # after failure the cache is deleted, and a set proceeds
+        yield node.set_data("zebra")
+        data = yield node.get_data()
+        self.assertEqual(data, "zebra")