Merge lp:~thisfred/u1db/fahrenheit-503 into lp:u1db

[Eric Casteleijn]

Added retry with backoff logic on 503s from a server.

Added retry with backoff logic on 503s from a server.

This is a request for feedback maybe more than a merge proposal: What I ended up doing is *extremely simple*. It may be too simple, so that's one question. the current tests for Unavailable exercise this code, and the other client tests exercise the success case.

I would like to add a test that succeeds on the second (or any but the first try) but I am unsure how to do it, which is where I may have to admit that I over simplified.

[Eric Casteleijn (thisfred)]

Also, I didn't do the C part :)

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 6/22/2012 10:28 PM, Eric Casteleijn wrote:
> Eric Casteleijn has proposed merging
> lp:~thisfred/u1db/fahrenheit-503 into lp:u1db.
>
> Requested reviews: Ubuntu One hackers (ubuntuone-hackers) Related
> bugs: Bug #999562 in U1DB: "retry logic on 503 "
> https://bugs.launchpad.net/u1db/+bug/999562
>
> For more details, see:
> https://code.launchpad.net/~thisfred/u1db/fahrenheit-503/+merge/111673
>
>
>
Added retry with backoff logic on 503s from a server.
>
> This is a request for feedback maybe more than a merge proposal:
> What I ended up doing is *extremely simple*. It may be too simple,
> so that's one question. the current tests for Unavailable exercise
> this code, and the other client tests exercise the success case.
>
> I would like to add a test that succeeds on the second (or any but
> the first try) but I am unsure how to do it, which is where I may
> have to admit that I over simplified.
>

We at least discussed at one point giving the server the ability to
tell clients to back off for a particular amount of time. (So if we
got a thundering horde, we could get them split up by random time, and
so we can slow down clients when load is high.)

That was more about when to check for updates, rather than during a
503, but something we might want to think about.

I will say, I don't think we want to hide a sleep + retry that is
10-40s long. You really don't want your app to think it is busy and
hang for that long when nothing is going on.

So I think the 2, 5 might be reasonable, but above that, the app
itself should be aware and retry the operation. At least, that's my 2c.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/lxZAACgkQJdeBCYSNAANsKgCdHi2AP+LHwujBk0CF2uSefq5A
EyoAniMCJKhL/gnqqC5R9oR8p5GRJaIW
=Be8c
-----END PGP SIGNATURE-----

[Samuele Pedroni (pedronis)]

> So I think the 2, 5 might be reasonable, but above that, the app
> itself should be aware and retry the operation. At least, that's my 2c.

I agree on this point

[Eric Casteleijn (thisfred)]

I changed the delays to be 1, 1, 2, and 4 seconds. I don't know if that's still too much of a maximum total wait.

I'm having a hard time adding tests for this: should the HTTPApp be made capable of sending 503s, and then made to do so in the sync tests?

[Eric Casteleijn (thisfred)]

I mean I have a hard time testing the C http sync implementation. I do have a test for the python client.

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/25/2012 08:12 PM, Eric Casteleijn wrote:
> I mean I have a hard time testing the C http sync implementation. I
> do have a test for the python client.
>

I think it would be reasonable to have an HTTP implementation that can
return 503s. We'd want it for integration testing as well. So some
tweaking there would be quite reasonable.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/pZekACgkQJdeBCYSNAANvDQCgrDP7bLBHPo3cx+oyFE0MZU3o
oxcAoIFsODdNuYmOHMxHRu1yqSJ0IRqJ
=9At7
-----END PGP SIGNATURE-----

[Eric Casteleijn (thisfred)]

Working c backend tests, which shook out some errors. Unfortunately, calling curl_easy_perform repeatedly requires setting the options again, or it will hang. Since the options are different for the different requests, a single retry function was no longer feasible.

Also: I would like to set the RETRY_DELAYS all to 0 for the tests (which now take a few seconds longer) but I am unsure how to do this from Cython

[lp:~thisfred/u1db/fahrenheit-503]

340. By Eric Casteleijn on 2012-06-27

resolved conflicts

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/27/2012 08:51 PM, Eric Casteleijn wrote:
> Eric Casteleijn has proposed merging
> lp:~thisfred/u1db/fahrenheit-503 into lp:u1db.
>
> Requested reviews: Ubuntu One hackers (ubuntuone-hackers) Related
> bugs: Bug #999562 in U1DB: "retry logic on 503 "
> https://bugs.launchpad.net/u1db/+bug/999562
>
> For more details, see:
> https://code.launchpad.net/~thisfred/u1db/fahrenheit-503/+merge/111673
>
> Added retry with backoff logic on 503s from a server.
>
> This is a request for feedback maybe more than a merge proposal:
> What I ended up doing is *extremely simple*. It may be too simple,
> so that's one question. the current tests for Unavailable exercise
> this code, and the other client tests exercise the success case.
>
> I would like to add a test that succeeds on the second (or any but
> the first try) but I am unsure how to do it, which is where I may
> have to admit that I over simplified.
>

If you want to have a test that succeeds on the second try, I think it
is just a matter of poking at your 'wrapper':

You have code like this:
+ def wrapper(instance, *args, **kwargs):
+ tries.append(None)
+ raise errors.Unavailable

You just change it to:

+ def wrapper(instance, *args, **kwargs):
+ if tries:
+ return orig(instance, *args, **kwargs)
+ tries.append(None)
+ raise errors.Unavailable

Or whatever the signature is. So on the *first* call, it raises
unavailable, and logs that it was called. And on subsequent calls it
just passes through to the original value.

It would be really neat if we had an actual HTTP service that could be
taught to do this, but in the short term, the above seems fine.

(I haven't looked deeply at the code yet.)

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/sYF8ACgkQJdeBCYSNAAM0AwCfQSd/tAkxpVPWfbZ1TkJ2HokI
kxEAoL4NEVUXstcY+jcxKT1diBMbqhZh
=pj+o
-----END PGP SIGNATURE-----

[Eric Casteleijn (thisfred)]

Yep, that's how I did it. :)

[lp:~thisfred/u1db/fahrenheit-503]

341. By Eric Casteleijn on 2012-06-29

resolved conflict

[Samuele Pedroni (pedronis)]

looks good

review: Approve

=== modified file 'include/u1db/u1db.h'
--- include/u1db/u1db.h	2012-06-22 00:03:55 +0000
+++ include/u1db/u1db.h	2012-06-29 17:03:18 +0000
@@ -72,6 +72,7 @@
 #define U1DB_INVALID_GLOBBING -19
 #define U1DB_INVALID_TRANSACTION_ID -20
 #define U1DB_INVALID_GENERATION -21
+#define U1DB_TARGET_UNAVAILABLE -22
 #define U1DB_INTERNAL_ERROR -999
 
 // Used by put_doc_if_newer

=== modified file 'src/u1db_http_sync_target.c'
--- src/u1db_http_sync_target.c	2012-06-22 00:03:55 +0000
+++ src/u1db_http_sync_target.c	2012-06-29 17:03:18 +0000
@@ -24,10 +24,18 @@
 #include <json/json.h>
 #include <curl/curl.h>
 #include <oauth.h>
+#include <sys/socket.h>
+#include <sys/time.h>
 
 #ifndef max
 #define max(a, b) ((a) > (b) ? (a) : (b))
 #endif // max
+#define TRIES 4
+#ifndef RETRY_DELAYS
+#define RETRY_DELAYS {1, 1, 2, 4}
+#endif
+
+int retry_delays[] = RETRY_DELAYS;
 
 struct _http_state;
 struct _http_request;
@@ -357,7 +365,6 @@
     return U1DB_OK;
 }
 
-
 static int
 st_http_get_sync_info(u1db_sync_target *st,
         const char *source_replica_uid,
@@ -368,9 +375,11 @@
     struct _http_request req = {0};
     char *url = NULL;
     const char *tmp = NULL;
-    int status;
+    int status = U1DB_OK;
     long http_code;
     struct curl_slist *headers = NULL;
+    int attempt = 0;
+    struct timeval timeout;
 
     json_object *json = NULL, *obj = NULL;
 
@@ -393,23 +402,42 @@
     req.state = state;
     status = u1db__format_sync_url(st, source_replica_uid, &url);
     if (status != U1DB_OK) { goto finish; }
-    status = curl_easy_setopt(state->curl, CURLOPT_HTTPGET, 1L);
-    if (status != CURLE_OK) { goto finish; }
-    // status = curl_easy_setopt(state->curl, CURLOPT_USERAGENT, "...");
-    status = curl_easy_setopt(state->curl, CURLOPT_URL, url);
-    if (status != CURLE_OK) { goto finish; }
-    req.body_buffer = req.header_buffer = NULL;
-    status = simple_set_curl_data(state->curl, &req, &req, NULL);
-    if (status != CURLE_OK) { goto finish; }
-    status = maybe_sign_url(st, "GET", url, &headers);
+    for (;;) {
+        status = curl_easy_setopt(state->curl, CURLOPT_HTTPGET, 1L);
+        if (status != CURLE_OK) { goto finish; }
+        // status = curl_easy_setopt(state->curl, CURLOPT_USERAGENT, "...");
+        status = curl_easy_setopt(state->curl, CURLOPT_URL, url);
+        if (status != CURLE_OK) { goto finish; }
+        req.body_buffer = req.header_buffer = NULL;
+        status = simple_set_curl_data(state->curl, &req, &req, NULL);
+        if (status != CURLE_OK) { goto finish; }
+        status = maybe_sign_url(st, "GET", url, &headers);
+        if (status != U1DB_OK) { goto finish; }
+        status = curl_easy_setopt(state->curl, CURLOPT_HTTPHEADER, headers);
+        if (status != CURLE_OK) { goto finish; }
+        // Now do the GET
+        status = curl_easy_perform(state->curl);
+        if (status != CURLE_OK) {
+            goto finish; }
+        status = curl_easy_getinfo(
+            state->curl, CURLINFO_RESPONSE_CODE, &http_code);
+        if (status != CURLE_OK) {
+            goto finish; }
+        status = U1DB_OK;
+        if (http_code == 503) {
+            status = U1DB_TARGET_UNAVAILABLE;
+            if (attempt < TRIES) {
+                timeout.tv_sec = retry_delays[attempt];
+                timeout.tv_usec = 0;
+                select(0, NULL, NULL, NULL, &timeout);
+                attempt++;
+                req.num_body_bytes = 0;
+                continue;
+            }
+        }
+        break;
+    }
     if (status != U1DB_OK) { goto finish; }
-    status = curl_easy_setopt(state->curl, CURLOPT_HTTPHEADER, headers);
-    if (status != CURLE_OK) { goto finish; }
-    // Now do the GET
-    status = curl_easy_perform(state->curl);
-    if (status != CURLE_OK) { goto finish; }
-    status = curl_easy_getinfo(state->curl, CURLINFO_RESPONSE_CODE, &http_code);
-    if (status != CURLE_OK) { goto finish; }
     if (http_code != 200) { // 201 for created? shouldn't happen on GET
         status = http_code;
         goto finish;
@@ -541,6 +569,8 @@
     const char *raw_body = NULL;
     int raw_len;
     struct curl_slist *headers = NULL;
+    int attempt = 0;
+    struct timeval timeout;
 
     if (st == NULL || source_replica_uid == NULL || st->implementation == NULL)
     {
@@ -572,27 +602,46 @@
     // confirmation of the post.
     headers = curl_slist_append(headers, "Expect:");
 
-    status = curl_easy_setopt(state->curl, CURLOPT_URL, url);
-    if (status != CURLE_OK) { goto finish; }
-    status = curl_easy_setopt(state->curl, CURLOPT_HTTPHEADER, headers);
-    if (status != CURLE_OK) { goto finish; }
-    status = curl_easy_setopt(state->curl, CURLOPT_UPLOAD, 1L);
-    if (status != CURLE_OK) { goto finish; }
-    status = curl_easy_setopt(state->curl, CURLOPT_PUT, 1L);
-    if (status != CURLE_OK) { goto finish; }
-    status = simple_set_curl_data(state->curl, &req, &req, &req);
-    if (status != CURLE_OK) { goto finish; }
-    status = curl_easy_setopt(state->curl, CURLOPT_INFILESIZE_LARGE,
-                              (curl_off_t)req.num_put_bytes);
-    if (status != CURLE_OK) { goto finish; }
-    status = maybe_sign_url(st, "PUT", url, &headers);
+    for (;;) {
+        status = curl_easy_setopt(state->curl, CURLOPT_URL, url);
+        if (status != CURLE_OK) { goto finish; }
+        status = curl_easy_setopt(state->curl, CURLOPT_HTTPHEADER, headers);
+        if (status != CURLE_OK) { goto finish; }
+        status = curl_easy_setopt(state->curl, CURLOPT_UPLOAD, 1L);
+        if (status != CURLE_OK) { goto finish; }
+        status = curl_easy_setopt(state->curl, CURLOPT_PUT, 1L);
+        if (status != CURLE_OK) { goto finish; }
+        status = simple_set_curl_data(state->curl, &req, &req, &req);
+        if (status != CURLE_OK) { goto finish; }
+        status = curl_easy_setopt(state->curl, CURLOPT_INFILESIZE_LARGE,
+                                (curl_off_t)req.num_put_bytes);
+        if (status != CURLE_OK) { goto finish; }
+        status = maybe_sign_url(st, "PUT", url, &headers);
+        if (status != U1DB_OK) { goto finish; }
+
+        // Now actually send the data
+        status = curl_easy_perform(state->curl);
+        if (status != CURLE_OK) {
+            goto finish; }
+        status = curl_easy_getinfo(
+            state->curl, CURLINFO_RESPONSE_CODE, &http_code);
+        if (status != CURLE_OK) {
+            goto finish; }
+        status = U1DB_OK;
+        if (http_code == 503) {
+            status = U1DB_TARGET_UNAVAILABLE;
+            if (attempt < TRIES) {
+                timeout.tv_sec = retry_delays[attempt];
+                timeout.tv_usec = 0;
+                select(0, NULL, NULL, NULL, &timeout);
+                attempt++;
+                req.num_body_bytes = 0;
+                continue;
+            }
+        }
+        break;
+    }
     if (status != U1DB_OK) { goto finish; }
-
-    // Now actually send the data
-    status = curl_easy_perform(state->curl);
-    if (status != CURLE_OK) { goto finish; }
-    status = curl_easy_getinfo(state->curl, CURLINFO_RESPONSE_CODE, &http_code);
-    if (status != CURLE_OK) { goto finish; }
     if (http_code != 200 && http_code != 201) {
         status = http_code;
         goto finish;
@@ -753,6 +802,8 @@
     char *url = NULL;
     struct _http_state *state;
     struct curl_slist *headers = NULL;
+    int attempt = 0;
+    struct timeval timeout;
 
     fputs("\r\n]", temp_fd);
     status = impl_as_http_state(st->implementation, &state);
@@ -761,18 +812,38 @@
     }
     status = u1db__format_sync_url(st, source_replica_uid, &url);
     if (status != U1DB_OK) { goto finish; }
-    status = curl_easy_setopt(state->curl, CURLOPT_URL, url);
-    if (status != CURLE_OK) { goto finish; }
-    status = setup_curl_for_sync(state->curl, &headers, req, temp_fd);
-    if (status != CURLE_OK) { goto finish; }
-    status = maybe_sign_url(st, "POST", url, &headers);
+    for (;;) {
+        status = curl_easy_setopt(state->curl, CURLOPT_URL, url);
+        if (status != CURLE_OK) { goto finish; }
+        status = setup_curl_for_sync(state->curl, &headers, req, temp_fd);
+        if (status != CURLE_OK) { goto finish; }
+        status = maybe_sign_url(st, "POST", url, &headers);
+        if (status != U1DB_OK) { goto finish; }
+        // Now send off the messages, and handle the returned content.
+        status = curl_easy_perform(state->curl);
+        if (status != CURLE_OK) {
+            goto finish; }
+        status = curl_easy_getinfo(
+                state->curl, CURLINFO_RESPONSE_CODE, &http_code);
+        if (status != CURLE_OK) {
+            goto finish; }
+        status = U1DB_OK;
+        if (http_code == 503) {
+            status = U1DB_TARGET_UNAVAILABLE;
+            if (attempt < TRIES) {
+                timeout.tv_sec = retry_delays[attempt];
+                timeout.tv_usec = 0;
+                select(0, NULL, NULL, NULL, &timeout);
+                attempt++;
+                req->num_body_bytes = 0;
+                continue;
+            }
+        }
+        break;
+    }
     if (status != U1DB_OK) { goto finish; }
-    // Now send off the messages, and handle the returned content.
-    status = curl_easy_perform(state->curl);
-    if (status != CURLE_OK) { goto finish; }
-    status = curl_easy_getinfo(state->curl, CURLINFO_RESPONSE_CODE, &http_code);
-    if (status != CURLE_OK) { goto finish; }
     if (http_code != 200 && http_code != 201) {
+        printf("broken 0\n");
         status = U1DB_BROKEN_SYNC_STREAM;
         goto finish;
     }
@@ -802,29 +873,34 @@
 
     json = json_tokener_parse(response);
     if (json == NULL || !json_object_is_type(json, json_type_array)) {
+        printf("broken 1, response: %s\n", response);
         status = U1DB_BROKEN_SYNC_STREAM;
         goto finish;
     }
     doc_count = json_object_array_length(json);
     if (doc_count < 1) {
         // the first response is the new_generation info, so it must exist
+        printf("broken 2\n");
         status = U1DB_BROKEN_SYNC_STREAM;
         goto finish;
     }
     obj = json_object_array_get_idx(json, 0);
     attr = json_object_object_get(obj, "new_generation");
     if (attr == NULL) {
+        printf("broken 3\n");
         status = U1DB_BROKEN_SYNC_STREAM;
         goto finish;
     }
     *target_gen = json_object_get_int(attr);
     attr = json_object_object_get(obj, "new_transaction_id");
     if (attr == NULL) {
+        printf("broken 4\n");
         status = U1DB_BROKEN_SYNC_STREAM;
         goto finish;
     }
     tmp = json_object_get_string(attr);
     if (tmp == NULL) {
+        printf("broken 5\n");
         status = U1DB_BROKEN_SYNC_STREAM;
         goto finish;
     }

=== modified file 'u1db/__init__.py'
--- u1db/__init__.py	2012-06-22 00:03:55 +0000
+++ u1db/__init__.py	2012-06-29 17:03:18 +0000
@@ -32,6 +32,8 @@
     :param path: The filesystem path for the database to open.
     :param create: True/False, should the database be created if it doesn't
         already exist?
+    :param document_factory: A function that will be called with the same
+        parameters as Document.__init__.
     :return: An instance of Database.
     """
     from u1db.backends import sqlite_backend

=== modified file 'u1db/errors.py'
--- u1db/errors.py	2012-06-22 00:03:55 +0000
+++ u1db/errors.py	2012-06-29 17:03:18 +0000
@@ -137,6 +137,7 @@
 
     wire_description = None
 
+
 # mapping wire (transimission) descriptions/tags for errors to the exceptions
 wire_description_to_exc = dict(
     (x.wire_description, x) for x in globals().values()

=== modified file 'u1db/remote/http_app.py'
--- u1db/remote/http_app.py	2012-06-14 23:28:58 +0000
+++ u1db/remote/http_app.py	2012-06-29 17:03:18 +0000
@@ -514,8 +514,7 @@
         except errors.U1DBError, e:
             self.request_u1db_error(environ, e)
             status = http_errors.wire_description_to_status.get(
-                                                            e.wire_description,
-                                                            500)
+                e.wire_description, 500)
             responder.send_response_json(status, error=e.wire_description)
         except BadRequest:
             self.request_bad_request(environ)

=== modified file 'u1db/remote/http_client.py'
--- u1db/remote/http_client.py	2012-05-25 19:00:46 +0000
+++ u1db/remote/http_client.py	2012-06-29 17:03:18 +0000
@@ -25,6 +25,7 @@
 import urlparse
 import urllib
 
+from time import sleep
 from u1db import (
     errors,
     )
@@ -33,8 +34,8 @@
     )
 
 from u1db.remote.ssl_match_hostname import (
+    CertificateError,
     match_hostname,
-    CertificateError,
     )
 
 # Ubuntu/debian
@@ -90,6 +91,10 @@
     # attacks for example) one would need HTTPS
     oauth_signature_method = oauth.OAuthSignatureMethod_HMAC_SHA1()
 
+    # Will use these delays to retry on 503 befor finally giving up. The final
+    # 0 is there to not wait after the final try fails.
+    _delays = (1, 1, 2, 4, 0)
+
     def __init__(self, url):
         self._url = urlparse.urlsplit(url)
         self._conn = None
@@ -186,8 +191,13 @@
             headers['content-type'] = content_type
         headers.update(
             self._sign_request(method, unquoted_url, encoded_params))
-        self._conn.request(method, url_query, body, headers)
-        return self._response()
+        for delay in self._delays:
+            try:
+                self._conn.request(method, url_query, body, headers)
+                return self._response()
+            except errors.Unavailable, e:
+                sleep(delay)
+        raise e
 
     def _request_json(self, method, url_parts, params=None, body=None,
                                                             content_type=None):

=== modified file 'u1db/tests/c_backend_wrapper.pyx'
--- u1db/tests/c_backend_wrapper.pyx	2012-06-22 00:03:55 +0000
+++ u1db/tests/c_backend_wrapper.pyx	2012-06-29 17:03:18 +0000
@@ -138,6 +138,7 @@
     int U1DB_INVALID_GENERATION
     int U1DB_INVALID_TRANSACTION_ID
     int U1DB_INTERNAL_ERROR
+    int U1DB_TARGET_UNAVAILABLE
 
     int U1DB_INSERTED
     int U1DB_SUPERSEDED
@@ -596,6 +597,8 @@
         raise errors.InvalidGeneration
     if status == U1DB_INVALID_TRANSACTION_ID:
         raise errors.InvalidTransactionId
+    if status == U1DB_TARGET_UNAVAILABLE:
+        raise errors.Unavailable
     if status == U1DB_INVALID_JSON:
         raise errors.InvalidJSON
     raise RuntimeError('%s (status: %s)' % (context, status))

=== modified file 'u1db/tests/test_c_backend.py'
--- u1db/tests/test_c_backend.py	2012-06-29 16:18:14 +0000
+++ u1db/tests/test_c_backend.py	2012-06-29 17:03:18 +0000
@@ -429,6 +429,48 @@
         self.assertGetDoc(db, mem_doc.doc_id, mem_doc.rev, mem_doc.get_json(),
                           False)
 
+    def test_unavailable(self):
+        mem_db = self.request_state._create_database('test.db')
+        mem_db.create_doc(tests.nested_doc)
+        tries = []
+
+        def wrapper(instance, *args, **kwargs):
+            tries.append(None)
+            raise errors.Unavailable
+
+        mem_db.whats_changed = wrapper
+        url = self.getURL('test.db')
+        target = c_backend_wrapper.create_http_sync_target(url)
+        db = c_backend_wrapper.CDatabase(':memory:')
+        db.create_doc(tests.simple_doc)
+        self.assertRaises(
+            errors.Unavailable, c_backend_wrapper.sync_db_to_target, db,
+            target)
+        self.assertEqual(5, len(tries))
+
+    def test_unavailable_then_available(self):
+        mem_db = self.request_state._create_database('test.db')
+        mem_doc = mem_db.create_doc(tests.nested_doc)
+        orig_whatschanged = mem_db.whats_changed
+        tries = []
+
+        def wrapper(instance, *args, **kwargs):
+            if len(tries) < 1:
+                tries.append(None)
+                raise errors.Unavailable
+            return orig_whatschanged(instance, *args, **kwargs)
+
+        mem_db.whats_changed = wrapper
+        url = self.getURL('test.db')
+        target = c_backend_wrapper.create_http_sync_target(url)
+        db = c_backend_wrapper.CDatabase(':memory:')
+        doc = db.create_doc(tests.simple_doc)
+        c_backend_wrapper.sync_db_to_target(db, target)
+        self.assertEqual(1, len(tries))
+        self.assertGetDoc(mem_db, doc.doc_id, doc.rev, doc.get_json(), False)
+        self.assertGetDoc(db, mem_doc.doc_id, mem_doc.rev, mem_doc.get_json(),
+                          False)
+
 
 class TestSyncCtoOAuthHTTPViaC(tests.TestCaseWithServer):
 

=== modified file 'u1db/tests/test_http_client.py'
--- u1db/tests/test_http_client.py	2012-05-25 19:00:46 +0000
+++ u1db/tests/test_http_client.py	2012-06-29 17:03:18 +0000
@@ -43,6 +43,10 @@
 
 class TestHTTPClientBase(tests.TestCaseWithServer):
 
+    def setUp(self):
+        super(TestHTTPClientBase, self).setUp()
+        self.errors = 0
+
     def app(self, environ, start_response):
         if environ['PATH_INFO'].endswith('echo'):
             start_response("200 OK", [('Content-Type', 'application/json')])
@@ -54,7 +58,37 @@
                 content_length = int(environ['CONTENT_LENGTH'])
                 ret['body'] = environ['wsgi.input'].read(content_length)
             return [simplejson.dumps(ret)]
+        elif environ['PATH_INFO'].endswith('error_then_accept'):
+            if self.errors >= 3:
+                start_response(
+                    "200 OK", [('Content-Type', 'application/json')])
+                ret = {}
+                for name in ('REQUEST_METHOD', 'PATH_INFO', 'QUERY_STRING'):
+                    ret[name] = environ[name]
+                if environ['REQUEST_METHOD'] in ('PUT', 'POST'):
+                    ret['CONTENT_TYPE'] = environ['CONTENT_TYPE']
+                    content_length = int(environ['CONTENT_LENGTH'])
+                    ret['body'] = '{"oki": "doki"}'
+                return [simplejson.dumps(ret)]
+            self.errors += 1
+            content_length = int(environ['CONTENT_LENGTH'])
+            error = simplejson.loads(
+                environ['wsgi.input'].read(content_length))
+            response = error['response']
+            # In debug mode, wsgiref has an assertion that the status parameter
+            # is a 'str' object. However error['status'] returns a unicode
+            # object.
+            status = str(error['status'])
+            if isinstance(response, unicode):
+                response = str(response)
+            if isinstance(response, str):
+                start_response(status, [('Content-Type', 'text/plain')])
+                return [str(response)]
+            else:
+                start_response(status, [('Content-Type', 'application/json')])
+                return [simplejson.dumps(response)]
         elif environ['PATH_INFO'].endswith('error'):
+            self.errors += 1
             content_length = int(environ['CONTENT_LENGTH'])
             error = simplejson.loads(
                 environ['wsgi.input'].read(content_length))
@@ -204,13 +238,31 @@
 
     def test_unavailable_proper(self):
         cli = self.getClient()
+        cli._delays = (0, 0, 0, 0, 0)
         self.assertRaises(errors.Unavailable,
                           cli._request_json, 'POST', ['error'], {},
                           {'status': "503 Service Unavailable",
                            'response': {"error": "unavailable"}})
+        self.assertEqual(5, self.errors)
+
+    def test_unavailable_then_available(self):
+        cli = self.getClient()
+        cli._delays = (0, 0, 0, 0, 0)
+        res, headers = cli._request_json(
+            'POST', ['error_then_accept'], {'b': 2},
+            {'status': "503 Service Unavailable",
+             'response': {"error": "unavailable"}})
+        self.assertEqual('application/json', headers['content-type'])
+        self.assertEqual({'CONTENT_TYPE': 'application/json',
+                          'PATH_INFO': '/dbase/error_then_accept',
+                          'QUERY_STRING': 'b=2',
+                          'body': '{"oki": "doki"}',
+                          'REQUEST_METHOD': 'POST'}, res)
+        self.assertEqual(3, self.errors)
 
     def test_unavailable_random_source(self):
         cli = self.getClient()
+        cli._delays = (0, 0, 0, 0, 0)
         try:
             cli._request_json('POST', ['error'], {},
                               {'status': "503 Service Unavailable",
@@ -221,6 +273,7 @@
         self.assertEqual(503, e.status)
         self.assertEqual("random unavailable.", e.message)
         self.assertTrue("content-type" in e.headers)
+        self.assertEqual(5, self.errors)
 
     def test_generic_u1db_error(self):
         cli = self.getClient()