Merge lp:~free.ekanayaka/landscape-client/reconnect into lp:~landscape/landscape-client/amp-trunk

[Free Ekanayaka]

[lp:~free.ekanayaka/landscape-client/reconnect]

221. By Free Ekanayaka on 2010-01-29

Pass a callback/errback pair to add_notifier

[Jamu Kakar (jkakar)]

[]

+ for callback, _ in self._notifiers:

Been hacking Erlang lately? The _ for an unused variable reminds me
of that. :)

[2]

+ @cvar factor: The time factor by which the delay between two subsequent
+ connection retries will decrease.

I believe you need to s/decrease/increase/ here, no?

[3]

+ """Call the given function on connection, reconnection or giveup.

s/giveup/give up/

[4]

+ to perfom again requests that failed due to a lost connection, as

s/perfom/perform/

[5]

+ can't perform them again successfully within this amout of seconds,

s/amout/number/

[6]

+ @param failure: The L{Failure} raised by the requested L{MethodCall}
+ @param name: The method name associated with the failed L{MethodCall}

and

+ """Called when the first connection has been established"""

These sentences are missing periods.

Very nice branch. It hurt my brain a bit, but it looks good. :) +1!

review: Approve

[Thomas Herve (therve)]

[1] The tests in MethodCallClientFactoryTest are a bit unfortunate, because highly dependent of time. ReconnectionClientFactory has a "clock" attribute which makes it fairly easy to mock, using twisted.internet.task.Clock

[2]
+ ports.append(reactor.listenUNIX(self.socket, self.server_factory))
+
+ def assert_connection(protocol):
+ self.assertTrue(isinstance(protocol, WordsClientProtocol))
+ client_factory.stopTrying()
+ protocol.transport.loseConnection()
+ ports[0].stopListening()

You shouldn't rely on the test to succeed to close the port connection, otherwise it's going to give you spurious errors. You should use addCleanup just after the listenUNIX.

+1 with most of the callLater removed from tests.

review: Needs Fixing

[lp:~free.ekanayaka/landscape-client/reconnect]

222. By Free Ekanayaka on 2010-02-08

Make MethodCallClientFactoryTest tests time-independent (therve [1])

223. By Free Ekanayaka on 2010-02-08

Fix typos (Jamu [2] to [6])

[Free Ekanayaka (free.ekanayaka)]

Jamu, Thomas thanks for the reviews!

@Jamu: Hehe, I didn't hack on Erlang, but I looked at some code recently which was using that pattern and I found it nice, because it feels a good hint for the reader.

@Therve: I fixed all the tests in MethodCallClientFactoryTest as you suggest. There are still some callLater-full tests in RemoteObjectCreatorTest, but they should be fine.

[lp:~free.ekanayaka/landscape-client/reconnect]

224. By Free Ekanayaka on 2010-02-12

Reactor-less testing for MethodCallClientFactory

225. By Free Ekanayaka on 2010-02-12

Fix testing on hardy

[Free Ekanayaka (free.ekanayaka)]

I discussed this a little bit with Thomas on IRC and for now we're going to leave the tests time-dependent. Even though this makes them breakable in principle, the rational is that making them time-independent would require some deeper mocking making them less "real-world" and meaningful. In practice I don't expect them to break easily, however if we start to see problems in the buildbots we can think about changing them. Thomas +1'ed for merging on IRC.

review: Approve

=== modified file 'landscape/__init__.py'
--- landscape/__init__.py	2009-10-27 12:45:38 +0000
+++ landscape/__init__.py	2010-02-12 16:06:15 +0000
@@ -1,4 +1,4 @@
-DEBIAN_REVISION = ""
+DEBIAN_REVISION = "-0ubuntu0.9.10.0"
 UPSTREAM_VERSION = "1.4.0"
 VERSION = "%s%s" % (UPSTREAM_VERSION, DEBIAN_REVISION)
 API = "3.2"

=== modified file 'landscape/lib/amp.py'
--- landscape/lib/amp.py	2010-01-25 08:28:36 +0000
+++ landscape/lib/amp.py	2010-02-12 16:06:15 +0000
@@ -2,7 +2,7 @@
 
 from uuid import uuid4
 from twisted.internet.defer import Deferred, maybeDeferred
-from twisted.internet.protocol import ServerFactory, ClientFactory
+from twisted.internet.protocol import ServerFactory, ReconnectingClientFactory
 from twisted.protocols.amp import Argument, String, Command, AMP
 from twisted.python.failure import Failure
 
@@ -212,25 +212,64 @@
         self.object = object
 
 
-class MethodCallClientFactory(ClientFactory):
-    """Factory for building L{AMP} connections to L{MethodCall} servers."""
+class MethodCallClientFactory(ReconnectingClientFactory):
+    """Factory for building L{AMP} connections to L{MethodCall} servers.
+
+    If the connection fails or is lost the factory will keep retrying to
+    establish it.
+
+    @cvar protocol: The factory used to build protocol instances.
+    @cvar factor: The time factor by which the delay between two subsequent
+        connection retries will increase.
+    """
 
     protocol = MethodCallClientProtocol
+    factor = 1.6180339887498948
 
-    def __init__(self, reactor, notifier):
+    def __init__(self, reactor):
         """
-        @param reactor: The reactor used by the created protocols to schedule
-            notifications and timeouts.
-        @param notifier: A function that will be called when the connection is
-            established. It will be passed the protocol instance as argument.
+        @param reactor: The reactor used by the created protocols
+            to schedule notifications and timeouts.
         """
         self.reactor = reactor
-        self._notifier = notifier
+        self.clock = self.reactor
+        self._notifiers = []
+
+    def add_notifier(self, callback, errback=None):
+        """Call the given function on connection, reconnection or give up.
+
+        @param notifier: A function that will be called when the factory builds
+            a new connected protocol or gives up connecting.  It will be passed
+            the new protocol instance as argument, or the connectionf failure.
+        """
+        self._notifiers.append((callback, errback))
+
+    def remove_notifier(self, callback, errback=None):
+        """Remove a notifier."""
+        self._notifiers.remove((callback, errback))
+
+    def notify_success(self, *args, **kwargs):
+        """Notify all registered notifier callbacks."""
+        for callback, _ in self._notifiers:
+            self.reactor.callLater(0, callback, *args, **kwargs)
+
+    def notify_failure(self, failure):
+        """Notify all registered notifier errbacks."""
+        for _, errback in self._notifiers:
+            if errback is not None:
+                self.reactor.callLater(0, errback, failure)
+
+    def clientConnectionFailed(self, connector, reason):
+        ReconnectingClientFactory.clientConnectionFailed(self, connector,
+                                                         reason)
+        if self.maxRetries is not None and (self.retries > self.maxRetries):
+            self.notify_failure(reason) # Give up
 
     def buildProtocol(self, addr):
+        self.resetDelay()
         protocol = self.protocol()
         protocol.factory = self
-        self.reactor.callLater(0, self._notifier, protocol)
+        self.notify_success(protocol)
         return protocol
 
 
@@ -242,12 +281,24 @@
     the remote object exposed by the peer.
     """
 
-    def __init__(self, protocol):
+    def __init__(self, protocol, retry_on_reconnect=False, timeout=None):
         """
         @param protocol: A reference to a connected L{AMP} protocol instance,
             which will be used to send L{MethodCall} commands.
+        @param retry_on_reconnect: If C{True}, this L{RemoteObject} will retry
+            to perform again requests that failed due to a lost connection, as
+            soon as a new connection is available.
+        @param timeout: A timeout for failed requests, if the L{RemoteObject}
+            can't perform them again successfully within this number of
+            seconds, they will errback with a L{MethodCallError}.
         """
         self._protocol = protocol
+        self._factory = protocol.factory
+        self._reactor = protocol.factory.reactor
+        self._retry_on_reconnect = retry_on_reconnect
+        self._timeout = timeout
+        self._pending_requests = {}
+        self._factory.add_notifier(self._handle_reconnect)
 
     def __getattr__(self, method):
         """Return a function sending a L{MethodCall} for the given C{method}.
@@ -262,10 +313,97 @@
             result = self._protocol.send_method_call(method=method,
                                                      args=args,
                                                      kwargs=kwargs)
-            return result.addCallback(lambda response: response["result"])
+            deferred = Deferred()
+            result.addCallback(self._handle_response, deferred)
+            result.addErrback(self._handle_failure, method, args, kwargs,
+                              deferred)
+            return deferred
 
         return send_method_call
 
+    def _handle_response(self, response, deferred, call=None):
+        """Handles a successful L{MethodCall} response.
+
+        @param response: The L{MethodCall} response.
+        @param deferred: The deferred that was returned to the caller.
+        @param call: If not C{None}, the scheduled timeout call associated with
+            the given deferred.
+        """
+        result = response["result"]
+        if call is not None:
+            call.cancel() # This is a successful retry, cancel the timeout.
+        deferred.callback(result)
+
+    def _handle_failure(self, failure, method, args, kwargs, deferred,
+                        call=None):
+        """Called when a L{MethodCall} command fails.
+
+        If a failure is due to a connection error and if C{retry_on_reconnect}
+        is C{True}, we will try to perform the requested L{MethodCall} again
+        as soon as a new connection becomes available, giving up after the
+        specified C{timeout}, if any.
+
+        @param failure: The L{Failure} raised by the requested L{MethodCall}.
+        @param name: The method name associated with the failed L{MethodCall}.
+        @param args: The positional arguments of the failed L{MethodCall}.
+        @param kwargs: The keyword arguments of the failed L{MethodCall}.
+        @param deferred: The deferred that was returned to the caller.
+        @param call: If not C{None}, the scheduled timeout call associated with
+            the given deferred.
+        """
+        is_method_call_error = failure.type is MethodCallError
+        dont_retry = self._retry_on_reconnect == False
+
+        if is_method_call_error or dont_retry:
+            # This means either that the connection is working, and a
+            # MethodCall protocol error occured, or that we gave up
+            # trying and raised a timeout. In any case just propagate
+            # the error.
+            if deferred in self._pending_requests:
+                self._pending_requests.pop(deferred)
+            if call:
+                call.cancel()
+            deferred.errback(failure)
+            return
+
+        if self._timeout and call is None:
+            # This is the first failure for this request, let's schedule a
+            # timeout call.
+            timeout = Failure(MethodCallError("timeout"))
+            call = self._reactor.callLater(self._timeout,
+                                           self._handle_failure,
+                                           timeout, method, args,
+                                           kwargs, deferred=deferred)
+
+        self._pending_requests[deferred] = (method, args, kwargs, call)
+
+    def _handle_reconnect(self, protocol):
+        """Handles a reconnection.
+
+        @param protocol: The newly connected protocol instance.
+        """
+        self._protocol = protocol
+        if self._retry_on_reconnect:
+            self._retry()
+
+    def _retry(self):
+        """Try to perform again requests that failed."""
+
+        # We need to copy the requests list before iterating over it, because
+        # if we are actually still disconnected, callRemote will return a
+        # failed deferred and the _handle_failure errback will be executed
+        # synchronously during the loop, modifing the requests list itself.
+        requests = self._pending_requests.copy()
+        self._pending_requests.clear()
+
+        while requests:
+            deferred, (method, args, kwargs, call) = requests.popitem()
+            result = self._protocol.send_method_call(method, args, kwargs)
+            result.addCallback(self._handle_response,
+                               deferred=deferred, call=call)
+            result.addErrback(self._handle_failure, method, args, kwargs,
+                              deferred=deferred, call=call)
+
 
 class RemoteObjectCreator(object):
     """Connect to remote objects exposed by a L{MethodCallProtocol}."""
@@ -273,32 +411,48 @@
     factory = MethodCallClientFactory
     remote = RemoteObject
 
-    def __init__(self, reactor, socket_path):
+    def __init__(self, reactor, socket_path, *args, **kwargs):
         """
         @param reactor: A reactor able to connect to Unix sockets.
         @param socket: The path to the socket we want to connect to.
+        @param args: Arguments to passed to the created L{RemoteObject}.
+        @param kwargs: Keyword arguments for the created L{RemoteObject}.
         """
         self._socket_path = socket_path
         self._reactor = reactor
-        self._remote = None
+        self._args = args
+        self._kwargs = kwargs
 
-    def connect(self):
+    def connect(self, max_retries=None):
         """Connect to a remote object exposed by a L{MethodCallProtocol}.
 
         This method will connect to the socket provided in the constructor
         and return a L{Deferred} resulting in a connected L{RemoteObject}.
+
+        @param max_retries: If not C{None} give up try to connect after this
+            amount of times.
         """
-        deferred = Deferred()
-        factory = self.factory(self._reactor, deferred.callback)
-        self._reactor.connectUNIX(self._socket_path, factory)
-        deferred.addCallback(self._connection_made)
-        return deferred
-
-    def _connection_made(self, protocol):
-        """Called when the connection has been established"""
-        self._remote = self.remote(protocol)
-        return self._remote
+        self._connected = Deferred()
+        self._factory = self.factory(self._reactor)
+        self._factory.maxRetries = max_retries
+        self._factory.add_notifier(self._success, self._failure)
+        self._reactor.connectUNIX(self._socket_path, self._factory)
+        return self._connected
+
+    def _success(self, result):
+        """Called when the first connection has been established"""
+
+        # We did our job, remove our own notifier and let the remote object
+        # handle reconnections.
+        self._factory.remove_notifier(self._success, self._failure)
+        self._remote = self.remote(result, *self._args, **self._kwargs)
+        self._connected.callback(self._remote)
+
+    def _failure(self, failure):
+        """Called when the first connection has failed"""
+        self._connected.errback(failure)
 
     def disconnect(self):
         """Disconnect the L{RemoteObject} that we have created."""
+        self._factory.stopTrying()
         self._remote._protocol.transport.loseConnection()

=== modified file 'landscape/lib/tests/test_amp.py'
--- landscape/lib/tests/test_amp.py	2010-01-20 09:20:46 +0000
+++ landscape/lib/tests/test_amp.py	2010-02-12 16:06:15 +0000
@@ -1,12 +1,15 @@
 from twisted.internet import reactor
-from twisted.internet.defer import Deferred
+from twisted.internet.defer import Deferred, DeferredList
 from twisted.internet.protocol import ClientCreator
+from twisted.internet.error import ConnectionDone, ConnectError
+from twisted.internet.task import Clock
 
 from landscape.lib.amp import (
     MethodCallError, MethodCallServerProtocol,
     MethodCallClientProtocol, MethodCallServerFactory,
     MethodCallClientFactory, RemoteObject, RemoteObjectCreator)
 from landscape.tests.helpers import LandscapeTest
+from landscape.tests.mocker import KWARGS
 
 
 class WordsException(Exception):
@@ -81,7 +84,7 @@
         return deferred
 
 
-class WordsProtocol(MethodCallServerProtocol):
+class WordsServerProtocol(MethodCallServerProtocol):
 
     methods = ["empty",
                "motd",
@@ -96,19 +99,39 @@
                "google"]
 
 
+class WordsClientProtocol(MethodCallClientProtocol):
+
+    timeout = 0.1
+
+
+class WordsServerFactory(MethodCallServerFactory):
+
+    protocol = WordsServerProtocol
+
+
+class WordsClientFactory(MethodCallClientFactory):
+
+    protocol = WordsClientProtocol
+    factor = 0.19
+
+
+class RemoteWordsCreator(RemoteObjectCreator):
+
+    factory = WordsClientFactory
+
+
 class MethodCallProtocolTest(LandscapeTest):
 
     def setUp(self):
         super(MethodCallProtocolTest, self).setUp()
         socket = self.mktemp()
-        factory = MethodCallServerFactory(Words())
-        factory.protocol = WordsProtocol
+        factory = WordsServerFactory(Words())
         self.port = reactor.listenUNIX(socket, factory)
 
         def set_protocol(protocol):
             self.protocol = protocol
 
-        connector = ClientCreator(reactor, MethodCallClientProtocol)
+        connector = ClientCreator(reactor, WordsClientProtocol)
         connected = connector.connectUNIX(socket)
         return connected.addCallback(set_protocol)
 
@@ -240,17 +263,18 @@
     def setUp(self):
         super(RemoteObjectTest, self).setUp()
         socket = self.mktemp()
-        server_factory = MethodCallServerFactory(Words())
-        server_factory.protocol = WordsProtocol
+        server_factory = WordsServerFactory(Words())
         self.port = reactor.listenUNIX(socket, server_factory)
 
         def set_remote(protocol):
             self.protocol = protocol
             self.words = RemoteObject(protocol)
+            client_factory.stopTrying()
 
         connected = Deferred()
         connected.addCallback(set_remote)
-        client_factory = MethodCallClientFactory(reactor, connected.callback)
+        client_factory = WordsClientFactory(reactor)
+        client_factory.add_notifier(connected.callback)
         reactor.connectUNIX(socket, client_factory)
         return connected
 
@@ -385,7 +409,6 @@
         If the peer protocol doesn't send a response for a deferred within
         the given timeout, the method call fails.
         """
-        self.protocol.timeout = 0.1
         result = self.words.google("Long query")
         return self.assertFailure(result, MethodCallError)
 
@@ -407,19 +430,83 @@
         return result.addCallback(assert_late_response_is_handled)
 
 
+class MethodCallClientFactoryTest(LandscapeTest):
+
+    def setUp(self):
+        super(MethodCallClientFactoryTest, self).setUp()
+        self.clock = Clock()
+        self.factory = WordsClientFactory(self.clock)
+
+    def test_add_notifier(self):
+        """
+        The L{MethodCallClientFactory.add_notifier} method can be used to
+        add a callback function to be called when a connection is made and
+        a new protocol instance built.
+        """
+        protocol = self.factory.protocol()
+        self.factory.protocol = lambda: protocol
+        callback = self.mocker.mock()
+        callback(protocol)
+        self.mocker.replay()
+        self.factory.add_notifier(callback)
+        self.factory.buildProtocol(None)
+        self.clock.advance(0)
+
+    def test_remove_notifier(self):
+        """
+        The L{MethodCallClientFactory.remove_notifier} method can be used to
+        remove a previously added notifier callback.
+        """
+        callback = lambda protocol: 1 / 0
+        self.factory.add_notifier(callback)
+        self.factory.remove_notifier(callback)
+        self.factory.buildProtocol(None)
+        self.clock.advance(0)
+
+    def test_client_connection_failed(self):
+        """
+        The L{MethodCallClientFactory} keeps trying to connect if maxRetries
+        is not reached.
+        """
+        # This is sub-optimal but the ReconnectingFactory in Hardy's Twisted
+        # doesn't support task.Clock
+        self.factory.retry = self.mocker.mock()
+        self.factory.retry(KWARGS)
+        self.mocker.replay()
+        self.assertEquals(self.factory.retries, 0)
+        self.factory.clientConnectionFailed(None, None)
+
+    def test_client_connection_failed_with_max_retries_reached(self):
+        """
+        The L{MethodCallClientFactory} stops trying to connect if maxRetries
+        is reached.
+        """
+        callback = lambda protocol: 1 / 0
+        errback = self.mocker.mock()
+        errback("failure")
+        self.mocker.replay()
+
+        self.factory.add_notifier(callback, errback)
+        self.factory.maxRetries = 1
+        self.factory.retries = self.factory.maxRetries
+        self.factory.clientConnectionFailed(object(), "failure")
+        self.clock.advance(0)
+
+
 class RemoteObjectCreatorTest(LandscapeTest):
 
     def setUp(self):
         super(RemoteObjectCreatorTest, self).setUp()
-        socket = self.mktemp()
-        factory = MethodCallServerFactory(Words())
-        factory.protocol = WordsProtocol
-        self.port = reactor.listenUNIX(socket, factory)
-
-        def set_remote(remote):
-            self.words = remote
-
-        self.connector = RemoteObjectCreator(reactor, socket)
+        self.socket = self.mktemp()
+        self.server_factory = WordsServerFactory(Words())
+        self.port = reactor.listenUNIX(self.socket, self.server_factory)
+        self.connector = RemoteWordsCreator(reactor, self.socket,
+                                            retry_on_reconnect=True,
+                                            timeout=0.7)
+
+        def set_remote(words):
+            self.words = words
+
         connected = self.connector.connect()
         return connected.addCallback(set_remote)
 
@@ -430,7 +517,206 @@
 
     def test_connect(self):
         """
-        A L{RemoteObject} can send L{MethodCall}s without arguments and with
-        an empty response.
+        The L{RemoteObject} resulting form the deferred returned by
+        L{RemoteObjectCreator.connect} is properly connected to the
+        remote peer.
         """
         return self.assertSuccess(self.words.empty())
+
+    def test_connect_with_max_retries(self):
+        """
+        If C{max_retries} is passed to the L{RemoteObjectCreator} method,
+        then it will give up trying to connect after that amout of times.
+        """
+        self.connector.disconnect()
+        self.port.stopListening()
+
+        def reconnect(ignored):
+            self.port = reactor.listenUNIX(self.socket, self.server_factory)
+            return self.connector.connect()
+
+        result = self.connector.connect(max_retries=0)
+        self.assertFailure(result, ConnectError)
+        return result.addCallback(reconnect)
+
+    def test_reconnect(self):
+        """
+        If the connection is lost, the L{RemoteObject} created by the creator
+        will transparently handle the reconnection.
+        """
+        self.words._protocol.transport.loseConnection()
+        self.port.stopListening()
+
+        def restart_listening():
+            self.port = reactor.listenUNIX(self.socket, self.server_factory)
+            reactor.callLater(0.3, assert_remote)
+
+        def assert_remote():
+            result = self.words.empty()
+            result.addCallback(lambda x: reconnected.callback(None))
+            return result
+
+        reactor.callLater(0.01, restart_listening)
+        reconnected = Deferred()
+        return reconnected
+
+    def test_method_call_error(self):
+        """
+        If a L{MethodCall} fails due to a L{MethodCallError}, the
+        L{RemoteObject} won't try to perform it again.
+        """
+        return self.assertFailure(self.words.secret(), MethodCallError)
+
+    def test_retry(self):
+        """
+        If the connection is lost, the L{RemoteObject} created by the creator
+        will transparently retry to perform the L{MethodCall} requests that
+        failed due to the broken connection.
+        """
+        self.words._protocol.transport.loseConnection()
+        self.port.stopListening()
+
+        def restart_listening():
+            self.port = reactor.listenUNIX(self.socket, self.server_factory)
+
+        reactor.callLater(0.1, restart_listening)
+        return self.words.empty()
+
+    def test_retry_with_method_call_error(self):
+        """
+        If a retried L{MethodCall} request fails due to a L{MethodCallError},
+        the L{RemoteObject} will properly propagate the error to the original
+        caller.
+        """
+        self.words._protocol.transport.loseConnection()
+        self.port.stopListening()
+
+        def restart_listening():
+            self.port = reactor.listenUNIX(self.socket, self.server_factory)
+
+        def assert_failure(error):
+            self.assertEquals(str(error), "Forbidden method 'secret'")
+
+        reactor.callLater(0.5, restart_listening)
+        result = self.words.secret()
+        self.assertFailure(result, MethodCallError)
+        return result.addCallback(assert_failure)
+
+    def test_wb_retry_with_while_still_disconnected(self):
+        """
+        The L{RemoteObject._retry} method gets called as soon as a new
+        connection is ready. If for whatever reason the connection drops
+        again very quickly, the C{_retry} method will behave as expected.
+        """
+        self.words._protocol.transport.loseConnection()
+        self.port.stopListening()
+
+        def handle_reconnect(protocol):
+            # In this precise moment we have a newly connected protocol
+            self.words._protocol = protocol
+
+            # Pretend that the connection is lost again very quickly
+            protocol.transport.loseConnection()
+            self.port.stopListening()
+
+            # Force RemoteObject._retry to run using a disconnected protocol
+            reactor.callLater(0, self.words._retry)
+
+            # Restore the real handler and start listening again very soon
+            self.connector._factory.remove_notifier(handle_reconnect)
+            self.connector._factory.add_notifier(self.words._handle_reconnect)
+            reactor.callLater(0.2, restart_listening)
+
+        def restart_listening():
+            self.port = reactor.listenUNIX(self.socket, self.server_factory)
+
+        def assert_failure(error):
+            self.assertEquals(str(error), "Forbidden method 'secret'")
+
+        # Use our own reconnect handler
+        self.connector._factory.remove_notifier(self.words._handle_reconnect)
+        self.connector._factory.add_notifier(handle_reconnect)
+
+        reactor.callLater(0.2, restart_listening)
+        result = self.words.secret()
+        self.assertFailure(result, MethodCallError)
+        return result.addCallback(assert_failure)
+
+    def test_retry_with_many_method_calls(self):
+        """
+        If several L{MethodCall} requests were issued while disconnected, they
+        will be all eventually completed when the connection gets established
+        again.
+        """
+        self.words._protocol.transport.loseConnection()
+        self.port.stopListening()
+
+        def restart_listening():
+            self.port = reactor.listenUNIX(self.socket, self.server_factory)
+
+        def assert_guess(response):
+            self.assertEquals(response, "Guessed!")
+
+        def assert_secret(failure):
+            self.assertEquals(str(failure.value), "Forbidden method 'secret'")
+
+        def assert_motd(response):
+            self.assertEquals(response, "Words are cool")
+
+        reactor.callLater(0.1, restart_listening)
+
+        results = [self.words.guess("word", "cool", value=4),
+                   self.words.secret(),
+                   self.words.motd()]
+        results[0].addCallback(assert_guess)
+        results[1].addErrback(assert_secret)
+        results[2].addCallback(assert_motd)
+        return DeferredList(results)
+
+    def test_retry_without_retry_on_reconnect(self):
+        """
+        If C{retry_on_reconnect} is C{False}, the L{RemoteObject} object won't
+        retry to perform requests which failed because the connection was
+        lost, however requests made after a reconnection will still succeed.
+        """
+        self.words._protocol.transport.loseConnection()
+        self.port.stopListening()
+
+        def restart_listening():
+            self.port = reactor.listenUNIX(self.socket, self.server_factory)
+            reactor.callLater(0.3, assert_reconnected)
+
+        def assert_reconnected():
+            result = self.words.empty()
+            result.addCallback(lambda x: reconnected.callback(None))
+            return result
+
+        reactor.callLater(0.1, restart_listening)
+        self.words._retry_on_reconnect = False
+        result = self.words.empty()
+        self.assertFailure(result, ConnectionDone)
+        reconnected = Deferred()
+        return result.addCallback(lambda x: reconnected)
+
+    def test_retry_with_timeout(self):
+        """
+        If a C{timeout} is set, the L{RemoteObject} object will errback failed
+        L{MethodCall}s after that amount of seconds, without retrying them when
+        the connection established again.
+        """
+        self.words._protocol.transport.loseConnection()
+        self.port.stopListening()
+
+        def restart_listening():
+            self.port = reactor.listenUNIX(self.socket, self.server_factory)
+            reactor.callLater(0.1, reconnected.callback, None)
+
+        def assert_failure(error):
+            self.assertEquals(str(error), "timeout")
+            return reconnected
+
+        reactor.callLater(0.9, restart_listening)
+        result = self.words.empty()
+        self.assertFailure(result, MethodCallError)
+        reconnected = Deferred()
+        return result.addCallback(assert_failure)