Merge lp:~urbanape/bindwood/manifest into lp:bindwood

[Zachery Bir]

[Zachery Bir (urbanape)]

I apologize in advance for the huge diff this is going to generate.

This branch does a lot, but only for brand new users:

  * Completely sync the structure of the bookmarks tree(s): Bookmarks, folders, feeds, and separators

  * Properly handle moves

To test:

  This branch includes an environment variable that will designate the database name to use. If you don't want the database to be replicated to Couch, first do the following:

  * Open your desktop couch futon, and edit the pair record in your management database (not sure what the id will be) to include a field called "excluded_names" if it's not already there. Create the value as a list, including the name of a fictional database you'll be testing with (I called mine 'test_boomarks' (clever, no?)).

  * I've moved the debugging information setting to a preference, so launch Firefox and add this boolean key to about:config:

    * bindwood.debug true

  * If you've used Bindwood before (particularly the Bindwood from my bindwood-exp PPA), you'll also want to reset (duplicating first) the bindwood.last_seq and bindwood.latest_modified preferences as well.

  * With this branch in place, and all wired up (see: https://wiki.ubuntu.com/TestingBindwood), quit Firefox and re-launch it from the command line as so:

    * BINDWOOD_DB=fictional_database_name firefox &

When Bindwood starts up, and your initial lag is over, you should be able to visit your Futon, and see more records. In particular, you'll see a root folder record at root_default (assuming you're using your default Firefox profile). It will have a 'children' field with three entries corresponding to the toolbar folder, the bookmarks menu, and the unfiled bookmarks. These will be important for testing the moving code.

TESTING

  * Structure: Follow the children ids in the root_default folder record, and you should be able to suss out the entire tree (if you're using folders).

  * Moving records: If you go into the record for the toolbar folder, and edit its children to swap the first and second ids, and save it, you should see things shuffle around a bit when Bindwood performs its next pull sync, and the first and second entries in your toolbar should be swapped.

[Zachery Bir (urbanape)]

I should note that both the subsequent client sync and pre-existing user migration code paths are not accounted for in this branch, and will each be the target of a following (hopefully much smaller) branch.

[dobey (dobey)]

We don't need to include the COPYING.BSD file here. As a whole, bindwood is GPLv3. Also, The Regents of UC are not the copyright holders of the BSD licesned code that you're including. Mozilla is. That copyright needs to be retained in the bindwood.js file. Derivitve works are not required to be the same BSD license, and we're allowed to ship it as GPLv3 with our code, but the (C) Mozilla must be retained from the original sync.js in the license header, unless all of those parts have been removed. Otherwise, they still retain copyright on part of the code.

review: Needs Fixing

[lp:~urbanape/bindwood/manifest]

48. By Zachery Bir on 2010-01-22

Don't need this.

[Zachery Bir (urbanape)]

Well, the file in question (new sync.js) is no longer needed by the code, and has since been removed, along with COPYING.BSD. Diff should reflect that shortly.

[dobey (dobey)]

review: Approve

[Guillermo Gonzalez (verterok)]

review: Approve

=== renamed file 'content/sync.js' => 'content/bindwood.js'
--- content/sync.js	2009-10-26 19:33:27 +0000
+++ content/bindwood.js	2010-01-22 19:18:13 +0000
@@ -20,6 +20,8 @@
 var Bindwood = {
     bookmarksService: Cc["@mozilla.org/browser/nav-bookmarks-service;1"]
         .getService(Ci.nsINavBookmarksService),
+    livemarkService: Cc["@mozilla.org/browser/livemark-service;2"]
+        .getService(Ci.nsILivemarkService),
     uuidService: Cc["@mozilla.org/uuid-generator;1"]
         .getService(Ci.nsIUUIDGenerator),
     annotationService: Cc["@mozilla.org/browser/annotation-service;1"]
@@ -36,22 +38,54 @@
         .getService(Ci.nsIProperties),
     windowService: Cc["@mozilla.org/appshell/window-mediator;1"]
         .getService(Ci.nsIWindowMediator),
+    // Technically, a branch, rather than the actual service, but
+    // consistency wins, I think
+    prefsService: Cc["@mozilla.org/preferences-service;1"]
+        .getService(Ci.nsIPrefService).getBranch("bindwood."),
+
+    TYPE_BOOKMARK: "http://www.freedesktop.org/wiki/Specifications/desktopcouch/bookmark",
+    TYPE_FOLDER: "http://www.freedesktop.org/wiki/Specifications/desktopcouch/folder",
+    TYPE_FEED: "http://www.freedesktop.org/wiki/Specifications/desktopcouch/feed",
+    TYPE_SEPARATOR: "http://www.freedesktop.org/wiki/Specifications/desktopcouch/separator",
 
     push: 'ENABLED', // Start off enabled
     annotationKey: "bindwood/uuid",
     uuidItemIdMap: {},
+    records: [],
+    seen_revisions: {},
+    timestamps: {},
 
     // Debugging/Console Behavior
     writeMessage: function(aMessage) {
         // convenience method for logging. Way better than alert()s.
-        if (Bindwood.envService.exists("BINDWOOD_DEBUG")) {
-            Bindwood.consoleService.logStringMessage("Bindwood: " + aMessage);
+        if (Bindwood.prefsService.getBoolPref("debug")) {
+            Bindwood.consoleService.logStringMessage(
+                "Bindwood: " + aMessage);
         }
     },
 
     writeError: function(aMessage, e) {
         // This should fire whether we're in DEBUG or not
-        Bindwood.consoleService.logStringMessage("Bindwood: " + aMessage + " message: '" + e.message + "', reason: '" + e.reason + "', description: '" + e.description + "', error: '" + e.error + "'");
+        Bindwood.consoleService.logStringMessage(
+            "Bindwood: " + aMessage +
+                " message: '" + e.message +
+                "', reason: '" + e.reason +
+                "', description: '" + e.description +
+                "', error: '" + e.error +
+                "', raw error: '" + JSON.stringify(e) + "'");
+    },
+
+    noteStartTime: function(key) {
+        var now = new Date();
+        Bindwood.timestamps[key] = now.getTime();
+    },
+
+    noteEndTime: function(key) {
+        var now = new Date();
+        var diff = now.getTime() - Bindwood.timestamps[key];
+        Bindwood.writeMessage(
+            key + " took " + diff + " milliseconds");
+        delete Bindwood.timestamps[key];
     },
 
     extractProfileName: function(path) {
@@ -69,7 +103,8 @@
         var possible_profile = segments[segments.length - 1];
         var first_period = possible_profile.indexOf('.');
         if (first_period == -1) {
-            return possible_profile; // no periods in the last segment, return as is
+            // no periods in the last segment, return as is
+            return possible_profile;
         } else {
             return possible_profile.substr(first_period + 1);
         }
@@ -81,10 +116,11 @@
         // http://forums.mozillazine.org/viewtopic.php?f=19&t=657911&start=0
         // It ensures that we're only running that code on the first window.
 
-        Bindwood.currentProfile = Bindwood.extractProfileName(Bindwood.directoryService.get('ProfD', Ci.nsIFile).path);
+        Bindwood.currentProfile = Bindwood.extractProfileName(
+            Bindwood.directoryService.get('ProfD', Ci.nsIFile).path);
 
         if(Bindwood.windowService.getEnumerator("").getNext() == window) {
-            Bindwood.getCouchEnvironment(Bindwood.startProcess);
+            Bindwood.getCouchEnvironment(Bindwood.setUpEnvironment);
         }
     },
 
@@ -96,22 +132,22 @@
         // find OS temp dir to put the tempfile in
         // https://developer.mozilla.org/index.php?title=File_I%2F%2FO#Getting_special_files
         var tmpdir = Cc["@mozilla.org/file/directory_service;1"]
-                     .getService(Ci.nsIProperties)
-                     .get("TmpD", Ci.nsIFile);
+            .getService(Ci.nsIProperties).get("TmpD", Ci.nsIFile);
         // create a randomly named tempfile in the tempdir
         var tmpfile = Cc["@mozilla.org/file/local;1"]
-                     .createInstance(Ci.nsILocalFile);
+            .createInstance(Ci.nsILocalFile);
         tmpfile.initWithPath(tmpdir.path + "/desktopcouch." + Math.random());
         tmpfile.createUnique(tmpfile.NORMAL_FILE_TYPE, 0600);
 
         // find the D-Bus bash script, which is in our extension folder
         var MY_ID = "[e-mail]";
-        var em = Cc["@mozilla.org/extensions/manager;1"].
-            getService(Ci.nsIExtensionManager);
-        var couchdb_env_script = em.getInstallLocation(MY_ID).getItemFile(MY_ID, "couchdb_env.sh");
+        var em = Cc["@mozilla.org/extensions/manager;1"]
+            .getService(Ci.nsIExtensionManager);
+        var couchdb_env_script = em.getInstallLocation(MY_ID)
+            .getItemFile(MY_ID, "couchdb_env.sh");
         // create an nsILocalFile for the executable
         var nsifile = Cc["@mozilla.org/file/local;1"]
-                     .createInstance(Ci.nsILocalFile);
+            .createInstance(Ci.nsILocalFile);
         nsifile.initWithPath(couchdb_env_script.path);
 
         // create an nsIProcess2 to execute this bash script
@@ -126,25 +162,29 @@
                 // If the script exists cleanly, we should have a file
                 // containing the port couch is running on as well as
                 // the various OAuth tokens necessary to talk to it.
+                var shouldProceed = true;
                 if (finishState == "process-finished") {
                     // read temp file to find couch environment
                     // https://developer.mozilla.org/en/Code_snippets/File_I%2f%2fO#Reading_from_a_file
                     var environment;
-                    var fstream = Cc["@mozilla.org/network/file-input-stream;1"].
-                        createInstance(Ci.nsIFileInputStream);
-                    var cstream = Cc["@mozilla.org/intl/converter-input-stream;1"].
-                        createInstance(Ci.nsIConverterInputStream);
+                    var fstream = Cc["@mozilla.org/network/file-input-stream;1"]
+                        .createInstance(Ci.nsIFileInputStream);
+                    var cstream = Cc["@mozilla.org/intl/converter-input-stream;1"]
+                        .createInstance(Ci.nsIConverterInputStream);
                     fstream.init(tmpfile, -1, 0, 0);
                     cstream.init(fstream, "UTF-8", 0, 0);
                     let (str = {}) {
-                      cstream.readString(-1, str); // read the whole file and put it in str.value
-                      environment = str.value;
+                        // read the whole file and put it in str.value
+                        cstream.readString(-1, str);
+                        environment = str.value;
                     };
                     cstream.close(); // this closes fstream
-                    environment = environment.replace(/^\s\s*/, '').replace(/\s\s*$/, '');
+                    environment = environment.replace(/^\s\s*/, '')
+                        .replace(/\s\s*$/, '');
                 } else {
                     // If we fail, we should just return
                     Bindwood.writeMessage("D-Bus port find failed");
+                    shouldProceed = false;
                 }
                 tmpfile.remove(false);
 
@@ -152,19 +192,23 @@
                     // No Couch environment found. Just spit out a
                     // message and return, stopping Bindwood from
                     // doing anything further.
-                    Bindwood.writeError("No suitable Couch environment found. Not proceeding.", e);
-                } else if (environment) {
-                    // If we don't have a Couch environment, don't bother
-                    // trying to fall back on the system CouchDB.
+                    Bindwood.writeError(
+                        "No suitable Couch environment found." +
+                            " Not proceeding.", e);
+                    shouldProceed = false;
+                }
+
+                if (shouldProceed && environment) {
                     continueFunction(environment);
-                } else { // No environment found at all
-                  return;
+                } else {
+                    // Unregister our observer for bookmark events; we're done
+                    Bindwood.bookmarksService.removeObserver(Bindwood.Observer);
                 }
             }
         });
     },
 
-    startProcess: function(couchEnvironment) {
+    setUpEnvironment: function(couchEnvironment) {
         var env_array = couchEnvironment.split(':');
         var port = env_array[0];
         var consumer_key = env_array[1];
@@ -188,81 +232,229 @@
             }
         };
 
-        Bindwood.couch = new CouchDB('bookmarks');
-
-        try {
-            Bindwood.pushBookmarks();
-        } catch(e) {
-            Bindwood.writeError("Error when calling pushBookmarks: ", e);
-        }
-        Bindwood.createViews();
-        Bindwood.pullBookmarks();
-    },
-
-    createViews: function() {
-        var views = [{ id: "_design/all_bookmarks",
-                       map: "function(doc) { " +
-                           "var scheme = doc.uri.split(':',1)[0]; " +
-                           "var uri; " +
-                           "if (scheme == 'http' || scheme == 'https') {" +
-                               "uri = doc.uri.split('/')[2];" +
-                               "if (uri.length < 30) {" +
-                                   " uri += '/' + " +
-                                       "doc.uri.split('/',4)[3].substr(0,30-uri.length) + '...';" +
-                               "}" +
-                           "} else {" +
-                               "uri = scheme + ' URL';" +
-                           "}" +
-                           "if (!doc.deleted) {" +
-                               "emit(doc.title, uri);" +
-                           "}" +
-                        "}" },
-                     { id: "_design/deleted_bookmarks",
-                       map: "function(doc) { if (doc.deleted) { emit (doc.title, doc.uri); } }" },
-                     { id: "_design/" + Bindwood.currentProfile,
-                       map: "function(doc) { var scheme = doc.uri.split(':',1)[0]; var uri; if (scheme == 'http' || scheme == 'https') {uri = doc.uri.split('/')[2];if (uri.length < 30) { uri += '/' + doc.uri.split('/',4)[3].substr(0,30-uri.length) + '...';}} else {uri = scheme + ' URL';}if ((!doc.deleted) && (doc.application_annotations.Firefox.profile == '" + Bindwood.currentProfile + "')) {emit(doc.title, uri);}}" }];
-        for (var i = 0; i < views.length; i++) {
-            var view_info = views[i];
-            var dirty = false;
+        var db_name = 'bookmarks';
+        if (Bindwood.envService.exists('BINDWOOD_DB')) {
+            db_name = Bindwood.envService.get('BINDWOOD_DB');
+        }
+
+        Bindwood.couch = new CouchDB(db_name);
+
+        try {
+            Bindwood.startProcess();
+        } catch(e) {
+            Bindwood.writeError(
+                "Something wrong with the process, exiting.", e);
+            return;
+        }
+    },
+
+    getLastSequence: function() {
+        var seq;
+        try {
+            seq = Bindwood.prefsService.getIntPref('last_seq');
+        } catch(e) {
+            seq = 0;
+        }
+        return seq;
+    },
+
+    setLastSequence: function(seq) {
+        Bindwood.prefsService.setIntPref('last_seq', seq);
+        return seq;
+    },
+
+    getLatestModified: function() {
+        var mod;
+        try {
+            mod = Number(Bindwood.prefsService.getCharPref('latest_modified'));
+        } catch(e) {
+            mod = 0;
+        }
+        return mod;
+    },
+
+    setLatestModified: function(mod) {
+        Bindwood.prefsService.setCharPref(
+            'latest_modified', Number(mod).toString());
+        return mod;
+    },
+
+    startProcess: function() {
+        Bindwood.writeMessage("Starting process");
+        Bindwood.last_seq = Bindwood.getLastSequence();
+        Bindwood.writeMessage("Got our last known sequence number: " + Bindwood.last_seq);
+        Bindwood.latest_modified = Bindwood.getLatestModified();
+        Bindwood.writeMessage("Got our latest known last_modified: " + Bindwood.latest_modified);
+
+        Bindwood.writeMessage("Ensuring the database exisits");
+        Bindwood.ensureDatabase();
+        Bindwood.writeMessage("Ensuring the views exist");
+        Bindwood.ensureViews();
+        Bindwood.writeMessage("Ensuring our scratch folder exists");
+        Bindwood.scratch_folder = Bindwood.ensureLocalScratchFolder();
+
+        var HAVE_LAST_SEQ = Bindwood.last_seq ? 1 : 0; // 0 or 1
+        var HAVE_PROFILE_ROOT = Bindwood.profileExists(); // 0 or 2
+
+        switch(HAVE_LAST_SEQ | HAVE_PROFILE_ROOT) {
+        case 0:
+            // Neither the profile root exists, nor do we have a last_seq. Ergo, we are a first
+            // time user. Proceed normally.
+
+            // Bindwood.display_first_sync_page();
+            break;
+        case 1:
+            // We have a last_seq, but the profile root does not exist. Ergo, we are an old user
+            // and must migrate to the new way of doing things.
+
+            /* Migration strategy:
+
+               Pull all records from Couch, and for each:
+
+                 - get the uuid off the bookmark.
+
+                 - look up itemId by uuid
+
+                 - annotate the itemId with the document's _id
+
+                 - delete the uuid field off the record
+            
+             */
+
+            // Bindwood.display_migration_sync_page();
+            break;
+        case 2:
+            // We have no last_seq, but the profile root exists. Ergo, we are starting up a
+            // subsequent client. We must make our local bookmarks look like remote, and ensure
+            // that any unaccounted for local bookmarks are sent to CouchDB.
+
+            // Bindwood.display_subsequent_client_sync_page();
+            // Show a helpful page explaining what's happening, so the user doesn't feel
+            // something is broken.
+
+            // Bindwood.move_local_records_to_scratch();
+            // Move all local records (folders, bookmarks, &c) to a temporary folder
+
+            // Bindwood.situate_remote_records();
+            // Pull all remote records, putting them in place according to pullRecords()
+
+            // Bindwood.situate_local_records();
+            // Iterate through our tmp tree, moving back bookmarks that aren't present.
+
+            // Bindwood.remove_tmp_folder();
+            // Anything left in the tmp folder has already been accounted for.
+            break;
+        case 3: // Should this just be default?
+            // We have a last_seq, and the profile root exists. Ergo, we are a normally operating
+            // client. Proceed normally.
+            break;
+        default:
+            break;
+        }
+
+        Bindwood.noteStartTime('Generating the manifest');
+        Bindwood.generateManifest();
+        Bindwood.noteEndTime('Generating the manifest');
+        Bindwood.noteStartTime('Pushing records');
+        Bindwood.pushLatestRecords();
+        Bindwood.noteEndTime('Pushing records');
+        Bindwood.pullChanges();
+    },
+
+    ensureDatabase: function() {
+        // This function will create the database if it does not
+        // exist, but we return a boolean representing whether or not
+        // the database existed prior.
+        try {
+            Bindwood.couch.createDb();
+        } catch (e) {
+            if (e.error != 'file_exists') {
+                Bindwood.writeError("Error creating database: ", e);
+                throw(e);
+            }
+        }
+    },
+
+    ensureViews: function() {
+        var view = {
+            _id: "_design/bookmarks",
+            views: {
+                profile: {
+                    map: "function(doc) { var scheme = doc.uri.split(':',1)[0]; var uri; if (scheme == 'http' || scheme == 'https') {uri = doc.uri.split('/')[2];if (uri.length < 30) { uri += '/' + doc.uri.split('/',4)[3].substr(0,30-uri.length) + '...';}} else {uri = scheme + ' URL';}if ((!doc.deleted) && (doc.application_annotations.Firefox.profile)) {emit(doc.application_annotations.Firefox.profile, [doc.title, uri]);}}"
+                },
+                live_bookmarks: {
+                    map: "function(doc) { var scheme = doc.uri.split(':',1)[0]; var uri; if (scheme == 'http' || scheme == 'https') {uri = doc.uri.split('/')[2];if (uri.length < 30) { uri += '/' + doc.uri.split('/',4)[3].substr(0,30-uri.length) + '...';}} else {uri = scheme + ' URL';}if (!doc.deleted) {emit(doc.title, uri);}}"
+                },
+                deleted_bookmarks: {
+                    map: "function(doc) { if (doc.deleted) { emit (doc.title, doc.uri); } }"
+                }
+            }
+        };
+
+        try {
+            var doc = Bindwood.couch.open(view._id);
+            if (!doc) {
+                doc = view;
+            }
             try {
-                var new_doc;
-                var current_doc = Bindwood.couch.open(view_info.id);
-                if (current_doc !== null) {
-                    var old_map = current_doc.views.display.map;
-                    if (old_map != view_info.map) { // Ours is definitive
-                        new_doc = current_doc;
-                        new_doc.views.display.map = view_info.map;
-                        dirty = true;
-                    }
-                } else {
-                    new_doc = {
-                        _id: view_info.id,
-                        views: {
-                            display: {
-                                map: view_info.map
-                            }
-                        }
-                    };
-                    dirty = true;
-                }
-                if (dirty) {
-                    try {
-                        Bindwood.couch.save(new_doc);
-                    } catch(e) {
-                        Bindwood.writeError("Problem saving view: ", e);
-                    }
-                }
+                Bindwood.couch.save(doc);
             } catch(e) {
-                // some kind of error fetching the existing design doc
-                Bindwood.writeError("Problem checking for view: ", e);
-            }
-        }
+                Bindwood.writeError("Problem saving view: ", e);
+            }
+        } catch(e) {
+            // some kind of error fetching the existing design doc
+            Bindwood.writeError("Problem checking for view: ", e);
+        }
+    },
+
+    ensureLocalScratchFolder: function() {
+        // Because records come from CouchDB without a parent field, until
+        // we get an updated folder record with a record situated properly
+        // in its children field, we need a place to temporarily store
+        // records from Couch. This is that place.
+        var folder = Bindwood.bookmarksService.unfiledBookmarksFolder;
+        var rootNode = Bindwood.getFolderRoot(folder);
+        rootNode.containerOpen = true;
+        for (var i=0; i<rootNode.childCount; i++) {
+            var node = rootNode.getChild(i);
+            if (node.title == 'Desktop Couch Scratch') {
+                rootNode.containerOpen = false;
+                return node.itemId;
+            }
+        }
+        rootNode.containerOpen = false;
+
+        var folderId = Bindwood.bookmarksService.createFolder(
+            folder, 'Desktop Couch Scratch', -1);
+        return folderId;
+    },
+
+    profileExists: function() {
+        // Check to see if the current profile's manifest exists in
+        // the database.
+        try {
+            Bindwood.couch.open('root_' + Bindwood.currentProfile);
+        } catch(e) {
+            if (e.error == 'not_found') {
+                return 0;
+            }
+        }
+        return 2;
     },
 
     // Looking up records locally
     annotateItemWithUUID: function(itemId, seed_uuid) {
-        var uuid = seed_uuid ? seed_uuid : Bindwood.uuidService.generateUUID().toString();
-        Bindwood.annotationService.setItemAnnotation(itemId, Bindwood.annotationKey, uuid, 0, Bindwood.annotationService.EXPIRE_NEVER);
+        var uuid = (seed_uuid ?
+                    seed_uuid :
+                    Bindwood.uuidService.generateUUID().toString());
+        Bindwood.writeMessage("UUID We came up with: " + uuid);
+        Bindwood.writeMessage("Annotating the item now.");
+        Bindwood.annotationService.setItemAnnotation(
+            itemId,
+            Bindwood.annotationKey,
+            uuid,
+            0,
+            Bindwood.annotationService.EXPIRE_NEVER);
         // Whenever we create a new UUID, stash it and the itemId in
         // our local cache.
         Bindwood.uuidItemIdMap[uuid] = itemId;
@@ -275,16 +467,19 @@
         var itemId = Bindwood.uuidItemIdMap[uuid];
 
         if (!itemId) {
-            var items = Bindwood.annotationService.getItemsWithAnnotation(Bindwood.annotationKey, {});
+            var items = Bindwood.annotationService.getItemsWithAnnotation(
+                Bindwood.annotationKey, {});
             var num_items = items.length;
             for (var i = 0; i < items.length; i++) {
-                if (Bindwood.annotationService.getItemAnnotation(items[i], Bindwood.annotationKey) == uuid) {
+                if (Bindwood.annotationService.getItemAnnotation(
+                        items[i], Bindwood.annotationKey) == uuid) {
                     Bindwood.uuidItemIdMap[uuid] = itemId = items[i];
                     break;
                 }
             }
             if (!itemId) {
-                Bindwood.writeMessage("XXX: Still haven't found the right itemId!");
+                Bindwood.writeMessage(
+                    "XXX: Still haven't found the right itemId!");
             }
         }
         return itemId;
@@ -295,80 +490,65 @@
         // and return it.
         var uuid;
         try {
-            uuid = Bindwood.annotationService.getItemAnnotation(itemId, Bindwood.annotationKey);
+            uuid = Bindwood.annotationService.getItemAnnotation(
+                itemId, Bindwood.annotationKey);
+            Bindwood.uuidItemIdMap[uuid] = itemId;
         } catch(e) {
-            Bindwood.writeError("Couldn't find a UUID for itemId: " + itemId, e);
+            Bindwood.writeError(
+                "Couldn't find a UUID for itemId: " + itemId, e);
             uuid = Bindwood.makeLocalChangeOnly(
-                function() { return Bindwood.annotateItemWithUUID(itemId, null); } );
+                function() { return Bindwood.annotateItemWithUUID(
+                                 itemId, null); } );
         }
 
         return uuid;
     },
 
-    resolveLocalBookmark: function(bm) {
-        // This function is only used to resolve bookmarks from Couch with uuids
-        var couch_uuid = bm.application_annotations.Firefox.uuid;
-        var itemId = Bindwood.itemIdForUUID(couch_uuid);
-        if (itemId) {
-            return itemId;
-        } else {
-            // This bookmark has a uuid, but it's not one of ours.
-            // We need to work out whether (a) it's the same as one
-            // of ours but with a different uuid (so we need to
-            // make the uuids the same), or (b) it's a new one
-            // that happens to have been created on a different
-            // machine.
-            try {
-                var uri = Bindwood.ioService.newURI(bm.uri, null, null);
-            } catch(e) {
-                Bindwood.writeError("Problem creating URI (" + bm.uri + ") for bookmark, skipping: ", e);
-                throw e;
-            }
-            var ids = Bindwood.bookmarksService.getBookmarkIdsForURI(uri, {});
-            if (ids.length == 0) {
-                return;
-            }
-            Bindwood.writeMessage("Returning the first bookmark for that URI");
-            return ids[0]; // XXX: Just return the first one. Right now, we don't worry too much about duplicates.
-        }
-    },
-
-    couchReadyObjectForNodeInBookmarksFolder: function(node, bookmarksFolderName) {
-        var itemId = node.itemId;
-
-        var title = Bindwood.bookmarksService.getItemTitle(itemId);
+    couchRecordForItemId: function(itemId) {
+        var bs = Bindwood.bookmarksService;
+
         var uuid = Bindwood.uuidForItemId(itemId);
-        var folder = bookmarksFolderName;
         var profile = Bindwood.currentProfile;
+        var last_modified = Bindwood.bookmarksService.getItemLastModified(itemId);
 
-        var bookmark = {
-            title: title,
+        var record = {
+            "_id": uuid,
             application_annotations: {
                 Firefox: {
-                    uuid: uuid,
-                    folder: folder,
-                    profile: profile
+                    profile: profile,
+                    last_modified: last_modified
                 }
             }
         };
 
-        switch(Bindwood.bookmarksService.getItemType(itemId)) {
-        case Bindwood.bookmarksService.TYPE_BOOKMARK:
-            bookmark.record_type = "http://www.freedesktop.org/wiki/Specifications/desktopcouch/bookmark";
-            bookmark.uri = Bindwood.bookmarksService.getBookmarkURI(itemId).spec;
-            break;
-        case Bindwood.bookmarksService.TYPE_FOLDER:
-            bookmark.record_type = "http://www.freedesktop.org/wiki/Specifications/desktopcouch/folder";
-            break;
-        case Bindwood.bookmarksService.TYPE_SEPARATOR:
-            bookmark.record_type = "http://www.freedesktop.org/wiki/Specifications/desktopcouch/separator";
+        switch(bs.getItemType(itemId)) {
+        case bs.TYPE_BOOKMARK:
+            record.title = bs.getItemTitle(itemId);
+            record.record_type = Bindwood.TYPE_BOOKMARK;
+            record.uri = bs.getBookmarkURI(itemId).spec;
+            break;
+        case bs.TYPE_FOLDER:
+            record.title = bs.getItemTitle(itemId);
+
+            // Firefox doesn't differentiate between regular folders
+            // and livemark folders. *sigh* So, we override it here
+            if (Bindwood.livemarkService.isLivemark(itemId)) {
+                record.record_type = Bindwood.TYPE_FEED;
+                record.site_uri = Bindwood.livemarkService.getSiteURI(itemId).spec;
+                record.feed_uri = Bindwood.livemarkService.getFeedURI(itemId).spec;
+            } else {
+                record.record_type = Bindwood.TYPE_FOLDER;
+                record.children = [];
+            }
+            break;
+        case bs.TYPE_SEPARATOR:
+            record.record_type = Bindwood.TYPE_SEPARATOR;
             break;
         default:
             break;
         }
 
-        Bindwood.writeMessage("Prepared bookmark (" + uuid + ") for Couch: " + JSON.stringify(bookmark));
-        return bookmark;
+        return record;
     },
 
     makeLocalChangeOnly: function(func) {
@@ -379,334 +559,419 @@
     },
 
     // Back and forth
-    pushBookmarks: function() {
-        // Prime the pump, so to speak, by uploading all our local
-        // bookmarks to CouchDB (if they're not there already).
-        // Create the DB if it doesn't already exist
-        try {
-            Bindwood.couch.createDb();
-        } catch (e) {
-            if (e.error == 'file_exists') {
-                Bindwood.writeMessage("Database already exists. We're okay.");
-            } else {
-                Bindwood.writeError("Error when creating database in pushBookmarks: ", e);
-            }
-        }
-
-        try {
-            Bindwood.pushBookmarksFromFolder("toolbarFolder", Bindwood.bookmarksService.toolbarFolder);
-        } catch(e) {
-            Bindwood.writeError("Error pushing toolbarFolder bookmarks: ", e);
-        }
-
-        try {
-            Bindwood.pushBookmarksFromFolder("bookmarksMenuFolder", Bindwood.bookmarksService.bookmarksMenuFolder);
-        } catch(e) {
-            Bindwood.writeError("Error pushing bookmarksMenuFolder bookmarks: ", e);
-        }
-
-        try {
-            Bindwood.pushBookmarksFromFolder("unfiledBookmarksFolder", Bindwood.bookmarksService.unfiledBookmarksFolder);
-        } catch(e) {
-            Bindwood.writeError("Error pushing unfiledBookmarksFolder bookmarks: ", e);
-        }
-    },
-
-    getBookmarksFolder: function(bookmarksFolder) {
+    getFolderRoot: function(folder) {
         var options = Bindwood.historyService.getNewQueryOptions();
         var query = Bindwood.historyService.getNewQuery();
-        query.setFolders([bookmarksFolder], 1);
+        query.setFolders([folder], 1);
         var result = Bindwood.historyService.executeQuery(query, options);
         return result.root;
     },
 
-    getBookmarksFromFolder: function(bookmarksFolderName, bookmarksFolder, accum) {
-        Bindwood.writeMessage("Fetching bookmarks from: " + bookmarksFolderName);
-        var rootNode = Bindwood.getBookmarksFolder(bookmarksFolder);
-        rootNode.containerOpen = true;
-        for (var i=0; i<rootNode.childCount; i++) {
-            var node = rootNode.getChild(i);
-            // If node is a Livemark container or a Dynamic container, skip it
-            if (PlacesUtils.nodeIsDynamicContainer(node) ||
-                PlacesUtils.nodeIsLivemarkContainer(node)) {
+    getUUIDsFromFolder: function(folder) {
+        var folderRoot = Bindwood.getFolderRoot(folder);
+        folderRoot.containerOpen = true;
+        var uuids = [];
+
+        for (var i=0; i<folderRoot.childCount; i++) {
+            var node = folderRoot.getChild(i);
+            uuids.push(Bindwood.uuidForItemId(node.itemId));
+        }
+
+        return uuids;
+    },
+
+    getRecordsFromFolder: function(folder) {
+        // Make a record for us, populating a children field with the _ids of all our children
+        var folderRoot = Bindwood.getFolderRoot(folder);
+        folderRoot.containerOpen = true;
+
+        var folder_record = Bindwood.couchRecordForItemId(folder);
+
+        for (var i=0; i<folderRoot.childCount; i++) {
+            var node = folderRoot.getChild(i);
+
+            var record = Bindwood.couchRecordForItemId(node.itemId);
+            folder_record.children.push(record._id);
+
+            // If node is a folder (but not a Livemark or Dynamic container),
+            // descend into it, looking for its contents
+            if (record.record_type == Bindwood.TYPE_FOLDER) {
+                Bindwood.getRecordsFromFolder(node.itemId)
+            } else {
+                Bindwood.records.push(record);
+            }
+        }
+        folderRoot.containerOpen = false;
+        Bindwood.records.push(folder_record);
+        return folder_record;
+    },
+
+    generateManifest: function() {
+        // Fill up the Bindwood.manifest and initial push lists
+        var primaryFolders = [Bindwood.bookmarksService.toolbarFolder,
+                              Bindwood.bookmarksService.bookmarksMenuFolder,
+                              Bindwood.bookmarksService.unfiledBookmarksFolder];
+
+        var profile_root = {
+            "_id": "root_" + Bindwood.currentProfile,
+            children: [],
+            application_annotations: {
+                Firefox: {
+                    profile: Bindwood.currentProfile,
+                    last_modified: 1
+                }
+            }
+        };
+
+        for (var i=0; i<primaryFolders.length; i++) {
+            var folder = primaryFolders[i];
+            var folder_record = Bindwood.getRecordsFromFolder(folder);
+
+            profile_root.children.push(folder_record._id);
+        }
+
+        Bindwood.records.push(profile_root);
+    },
+
+    sortByLastModDesc: function(a, b) {
+        var a_mod = a.application_annotations.Firefox.last_modified;
+        var b_mod = b.application_annotations.Firefox.last_modified;
+        return b_mod - a_mod; // descending
+    },
+
+    pushLatestRecords: function() {
+        Bindwood.records.sort(Bindwood.sortByLastModDesc);
+        // Now that the record are all sorted descending by last
+        // mod time, we can check each in turn to see if its mod time
+        // is greater than our persisted mod time. Afterwards, we'll
+        // set our persisted latest mod time to be the first record's
+        // mod time.
+        var newest = Bindwood.records[0];
+        var newest_ff = newest.application_annotations.Firefox;
+        var new_latest_modified = newest_ff.last_modified;
+        for (var i = 0; i < Bindwood.records.length; i++) {
+            // find this record in CouchDB
+            var record = Bindwood.records[i];
+            var ff = record.application_annotations.Firefox;
+
+            if (ff.last_modified <= Bindwood.latest_modified) {
                 Bindwood.writeMessage(
-                    "Skipping Dynamic or Livemark container: " + node.title);
-                continue;
-            }
-            // If node is a folder, descend into it, looking for its contents
-            if (node.type == 6) { // RESULT_TYPE_FOLDER
-                Bindwood.writeMessage("Descending into folder: " + node.title);
-                accum = accum.concat(Bindwood.getBookmarksFromFolder(node.title, node.itemId, accum));
-            } else if (node.type == 0 || node.type == 7) { // RESULT_TYPE_URI or RESULT_TYPE_SEPARATOR
-                Bindwood.writeMessage("Adding current node: " + node.title);
-                accum.push([node.itemId, Bindwood.couchReadyObjectForNodeInBookmarksFolder(node, bookmarksFolderName)]);
-            }
-        }
-        rootNode.containerOpen = false;
-        return accum;
-    },
-
-    pushBookmarksFromFolder: function(bookmarksFolderName, bookmarksFolder) {
-        Bindwood.writeMessage("Pushing bookmarks from " + bookmarksFolderName);
-        var bookmarkData = Bindwood.getBookmarksFromFolder(bookmarksFolderName, bookmarksFolder, new Array());
-        for (var i = 0; i < bookmarkData.length; i++) {
-            // find this bookmark in CouchDB
-            var itemId = bookmarkData[i][0];
-            var bookmark = bookmarkData[i][1];
-            var uuid = bookmark.application_annotations.Firefox.uuid;
-            Bindwood.writeMessage("Bookmark so far: " + JSON.stringify(bookmark));
-
-            // Even though this happens in annotateItemWithUUID as well,
-            //  we're priming the cache here for first run.
-            Bindwood.uuidItemIdMap[uuid] = itemId;
-
-            try {
-                var results = Bindwood.couch.query(function(doc) {
-                                           if (doc.application_annotations &&
-                                               doc.application_annotations.Firefox &&
-                                               doc.application_annotations.Firefox.uuid) {
-                                               emit(doc.application_annotations.Firefox.uuid, doc);
-                                           }
-                                       }, null, {
-                                           startkey: uuid, endkey: uuid
-                                       });
-            } catch(e) {
-                Bindwood.writeError("Error querying couch: ", e);
-            }
-
-            if (results.rows.length === 0) {
-                // this bookmark is not in CouchDB, so write it
+                    "We've reached records we've already dealt with." +
+                        " Breaking out of the loop.");
+                break;
+            }
+
+            var doc = Bindwood.couch.open(record._id);
+            if (!doc) {
+                // this record is not in CouchDB, so write it
                 try {
-                    Bindwood.couch.save(bookmark);
-                    Bindwood.writeMessage("Saved bookmark (" + uuid + ") to Couch.");
+                    var response = Bindwood.couch.save(record);
+                    // We can avoid having to process this revision when we pull it later
+                    Bindwood.seen_revisions[response.rev] = true;
                 } catch(e) {
-                    Bindwood.writeError("Problem saving bookmark to CouchDB; bookmark is " + JSON.stringify(bookmark) + ": ", e);
+                    Bindwood.writeError(
+                        "Problem saving record to CouchDB; record is " +
+                            JSON.stringify(record) + ": ", e);
                 }
             } else {
-                Bindwood.writeMessage("This bookmark (" + uuid + ") is already in Couch, skipping");
-                // bookmark is already in CouchDB, so do nothing
-            }
-        }
-    },
-
-    pullBookmarks: function() {
-        // Fetch all bookmark documents from the database
-        // The query function is evaluated by Couch, which doesn't know
-        // what Bindwood.RECORD_TYPE is, so we string-encode it first to
-        // include the literal value
-        Bindwood.writeMessage("Fetching bookmarks for " + Bindwood.currentProfile);
-        var rows = {rows: []};
-        try {
-            rows = Bindwood.couch.query(
-                "function (doc) {" +
-                    " if (doc.record_type == \"http://www.freedesktop.org/wiki/Specifications/desktopcouch/bookmark\" &&" +
-                    " doc.application_annotations &&" +
-                    " doc.application_annotations.Firefox &&" +
-                    " doc.application_annotations.Firefox.profile &&" +
-                    " doc.application_annotations.Firefox.profile == \"" + Bindwood.currentProfile + "\") {" +
-                    " emit(doc._id,doc);}}");
-        } catch(e) {
-            Bindwood.writeError("Problem fetching all bookmarks from Couch: ", e);
-        }
-        Bindwood.writeMessage("We got back " + rows.rows.length + " rows");
-        for (var i = 0; i < rows.rows.length; i++) {
-            var recordid = rows.rows[i].id;
-            Bindwood.writeMessage("Pulling record: " + recordid);
-            var bm = rows.rows[i].value;
-
-            // First, check to see if the bookmark we've pulled down is flagged as deleted.
-            // If so, we should make sure any local copy we have of this bookmark has also been deleted.
-            if (bm.deleted) {
-                Bindwood.writeMessage("Bookmark in Couch marked as deleted; attempting to delete local copy.");
-                Bindwood.deleteLocalBookmark(bm);
-            } else if (bm.application_annotations &&
-                       bm.application_annotations.Firefox &&
-                       bm.application_annotations.Firefox.uuid) {
-                Bindwood.writeMessage("Bookmark in Couch has UUID, merging local copy.");
-                Bindwood.mergeLocalBookmark(bm, recordid);
-            } else {
-                Bindwood.writeMessage("Bookmark in Couch has no UUID, creating new local copy, annotating, and pushing back.");
-                Bindwood.addLocalBookmark(bm, recordid, null);
-            }
-        }
+                // record is already in CouchDB, so do nothing
+                Bindwood.writeMessage(
+                    "This record (" + record._id + ") is already in Couch, skipping");
+            }
+        }
+        Bindwood.latest_modified = Bindwood.setLatestModified(
+            new_latest_modified);
+    },
+
+    pushFolderChildren: function(folder, children) {
+        var doc = Bindwood.couch.open(Bindwood.uuidForItemId(folder));
+        var new_children = children;
+        if (!new_children) {
+            new_children = Bindwood.getUUIDsFromFolder(folder);
+        }
+        doc.children = new_children;
+        var response = Bindwood.couch.save(doc);
+        Bindwood.seen_revisions[response.rev] = true;
+    },
+
+    pullChanges: function() {
+        Bindwood.pullRecords();
+
         // reschedule ourself
         Bindwood.writeMessage("Successful run, rescheduling ourself");
-        setTimeout(Bindwood.pullBookmarks, 30000);
-    },
-
-    deleteLocalBookmark: function(bm) {
-        // If we can't resolve the itemId, even by looking up URI, assume it's already gone.
-        var itemId = Bindwood.resolveLocalBookmark(bm);
-        if (itemId) {
-            Bindwood.makeLocalChangeOnly(
-                function() { return Bindwood.bookmarksService.removeItem(itemId); });
-        }
-    },
-
-    mergeLocalBookmark: function(bm, recordid) {
-        // this bookmark has a uuid, so check its values haven't changed
-        // find the bookmark with this uuid
-        var itemId = Bindwood.resolveLocalBookmark(bm);
-        var couch_uuid = bm.application_annotations.Firefox.uuid;
-
-        if (itemId) {
-            // Found one local bookmark. Replace its uuid to
-            // be the one from Couch.
-            var old_uuid = Bindwood.uuidForItemId(itemId);
-            if (old_uuid != couch_uuid) {
-                delete Bindwood.uuidItemIdMap[old_uuid];
-                Bindwood.makeLocalChangeOnly(
-                    function() { return Bindwood.annotateItemWithUUID(itemId, couch_uuid); } );
-            }
-
-            // XXX: I'm still not positive about this. While I'm far more sure that we're seeding
-            // Couch properly from the beginning and properly making new bookmarks, I still wonder
-            // if we'll see cases where Couch has null for title and uri, and how best to manage
-            // them. I'm thinking timestamps. - urbanape
-            var title = Bindwood.bookmarksService.getItemTitle(itemId);
-            if (title != bm.title) {
-                Bindwood.writeMessage("Resetting local title to title from Couch");
-                Bindwood.makeLocalChangeOnly(
-                    function() { return Bindwood.bookmarksService.setItemTitle(itemId, bm.title); });
-            }
-
-            var metadata = Bindwood.bookmarksService.getBookmarkURI(itemId);
-            if (metadata.spec != bm.uri) {
-                Bindwood.writeMessage("The URI from Couch (" + bm.uri + ") is different from local (" + metadata.spec + ")");
-                try {
-                    var new_uri = Bindwood.ioService.newURI(bm.uri, null, null);
-                    Bindwood.writeMessage("Creating a new URI for our local bookmark");
-                    Bindwood.makeLocalChangeOnly(
-                        function() { return Bindwood.bookmarksService.changeBookmarkURI(itemId, new_uri); });
-                } catch(e) {
-                    Bindwood.writeError("Problem creating a new URI for bookmark: ", e);
-                }
-            }
-        } else {
-            /// No local bookmarks
-            Bindwood.writeMessage("No local bookmark found, must be a new entry in Couch. Creating locally.");
-            Bindwood.addLocalBookmark(bm, recordid, couch_uuid);
-        }
-    },
-
-    guaranteeDesktopCouchFolder: function() {
-        // If we're syncing from CouchDB and we have new bookmarks,
-        // until we have proper replication of hierarchy and folders,
-        // we should put all "new" bookmarks in a collected, sane
-        // place. Unfiled bookmarks are hard to get to (only available
-        // through "Organize bookmarks"), so we'll put them in a
-        // folder in the Boomarks Menu called "Desktop Couch",
-        // creating that folder first, if necessary.
-        var bookmarksFolder = Bindwood.bookmarksService.bookmarksMenuFolder;
-        var rootNode = Bindwood.getBookmarksFolder(bookmarksFolder);
-        rootNode.containerOpen = true;
-        for (var i=0; i<rootNode.childCount; i++) {
-            var node = rootNode.getChild(i);
-            if (node.title == 'Desktop Couch') {
-                rootNode.containerOpen = false;
-                return node.itemId;
-            }
-        }
-        rootNode.containerOpen = false;
-
-        var folderId = Bindwood.bookmarksService.createFolder(
-            bookmarksFolder, 'Desktop Couch', -1);
-        return folderId;
-    },
-
-    addLocalBookmark: function(bm, recordid, uuid) {
-        // If uuid is present, we only need create the bookmark locally,
-        // as it already exists (with uuid) in Couch.
-        //
-        // If uuid is null, then after we create it locally,
-        // we want to annotate it and push back to Couch.
-        //
-        // By default, store new bookmarks in the unfiledBookmarksFolder
-        var folder;
-        if (bm.application_annotations &&
-            bm.application_annotations.Firefox &&
-            bm.application_annotations.Firefox.folder) {
-            switch (bm.application_annotations.Firefox.folder) {
-            case "toolbarFolder":
-                folder = Bindwood.bookmarksService.toolbarFolder;
-                break;
-            case "bookmarksMenuFolder":
-                folder = Bindwood.bookmarksService.bookmarksMenuFolder;
-                break;
-            default:
-                folder = Bindwood.guaranteeDesktopCouchFolder();
-                break;
-            }
-        }
-
+        setTimeout(Bindwood.pullChanges, 30000);
+    },
+
+    pullRecords: function() {
+        // Check to see if our prefsService has a preference set (I
+        // know, bad form) for last_seq, which would designate the
+        // last Couch sequence we've seen (this might be 0 if we've
+        // never synced before, in which case, we'd get all
+        // changes). Afterwards, set the last known sequence in
+        // prefs. Then, future polls will use last_seq as the start
+        // for finding changes.
+
+        // XXX: currently, we do a single changes pull. Eventually, if
+        // we need to use threads, we can do long polling in a
+        // background thread.
+
+        Bindwood.noteStartTime('Pulling records');
+        var results = {results: [], last_seq: 0};
         try {
-            var new_uri = Bindwood.ioService.newURI(bm.uri, null, null);
+            results = Bindwood.couch.changes(
+                {since: Bindwood.last_seq},
+                null
+            );
         } catch(e) {
-            Bindwood.writeError("Problem creating a new URI for bookmark: ", e);
-        }
-
-        var itemId = Bindwood.makeLocalChangeOnly(
-            function() { return Bindwood.bookmarksService.insertBookmark(
-                             folder, new_uri, -1, bm.title); });
-
-        if (uuid) { // We were provided a uuid, so no need to send back to Couch, just annotate locally
-            Bindwood.makeLocalChangeOnly(
-                function() { return Bindwood.annotateItemWithUUID(itemId, uuid); } );
-        } else {  // This is an unadorned bookmark from Couch. Save it to Couch once we establish its uuid
-            var new_uuid = Bindwood.uuidForItemId(itemId);
-            var doc = Bindwood.couch.open(recordid);
-            if (!doc.application_annotations) {
-                doc.application_annotations = {};
-            }
-            if (!doc.application_annotations.Firefox) {
-                doc.application_annotations.Firefox = {};
-            }
-            doc.application_annotations.Firefox.uuid = new_uuid;
-            try {
-                Bindwood.couch.save(doc);
-                Bindwood.writeMessage("Saved the newly annotated doc back to Couch");
-            } catch(e) {
-                Bindwood.writeError("Problem writing record for new bookmark: ",e);
-            }
-        }
-    },
-
-    findDocumentByUUID: function(uuid) {
-        Bindwood.writeMessage("Looking up a document in Couch by uuid: " + uuid);
-        var results = Bindwood.couch.query(function(doc) {
-        if (doc.application_annotations &&
-                doc.application_annotations.Firefox &&
-                doc.application_annotations.Firefox.uuid) {
-                emit(doc.application_annotations.Firefox.uuid, doc);
-            }
-        }, null, {
-            startkey: uuid, endkey: uuid
-        });
-        if (results.rows.length === 0) {
-            Bindwood.writeMessage("Problem finding document");
-            return;
-        }
-        Bindwood.writeMessage("Found: " + JSON.stringify(results));
-        return results;
+            Bindwood.writeError(
+                "Problem long polling bookmarks from Couch: ", e);
+        }
+        var revisions = results.results;
+        for (var i = 0; i < revisions.length; i++) {
+            var rev = revisions[i];
+            var revno = rev.changes[0].rev;
+            var recordid = rev.id;
+
+            // Skip (for now) if we're dealing with a root folder or a
+            // design doc
+            if (recordid.indexOf('root_') === 0 ||
+                recordid.indexOf('_design') === 0) {
+                Bindwood.writeMessage("Root profile or design doc, skipping...");
+                continue;
+            }
+
+            // Skip any revisions we've already seen (because we just
+            // put them there)
+            if (Bindwood.seen_revisions[revno]) {
+                Bindwood.writeMessage("We've seen this revision (" + revno + ") before, when we created it.");
+                delete Bindwood.seen_revisions[revno];
+                continue;
+            }
+
+            var record = Bindwood.couch.open(recordid);
+
+            if (!Bindwood.recordInCurrentProfile(record)) {
+                Bindwood.writeMessage("Record isn't in our current profile. Skipping...");
+                continue;
+            }
+
+            // Next, check to see if the record we've pulled down
+            // is flagged as deleted.  If so, we should make sure any
+            // local copy we have of this record has also been
+            // deleted.
+            if (Bindwood.isDeleted(record)) {
+                Bindwood.makeLocalChangeOnly(
+                    function() {
+                        Bindwood.writeMessage(
+                            "Record in Couch marked as deleted;" +
+                                " attempting to delete local copy.");
+                        Bindwood.deleteLocalRecord(record);
+                    });
+                continue; // Don't bother continuing to process anything further in this revision
+            }
+
+            switch(record.record_type) {
+            case Bindwood.TYPE_BOOKMARK:
+                Bindwood.makeLocalChangeOnly(
+                    function() {
+                        Bindwood.processCouchBookmarkRevision(record);
+                    });
+                break;
+            case Bindwood.TYPE_FOLDER:
+                Bindwood.makeLocalChangeOnly(
+                    function() {
+                        Bindwood.processCouchFolderRevision(record);
+                    });
+                break;
+            case Bindwood.TYPE_FEED:
+                Bindwood.makeLocalChangeOnly(
+                    function() {
+                        Bindwood.processCouchFeedRevision(record);
+                    });
+                break;
+            case Bindwood.TYPE_SEPARATOR:
+                Bindwood.makeLocalChangeOnly(
+                    function() {
+                        Bindwood.processCouchSeparatorRevision(record);
+                    });
+                break;
+            default:
+                break;
+            }
+            // STOPPING HERE - Time to do per-type dispatch
+        }
+
+        Bindwood.last_seq = Bindwood.setLastSequence(results.last_seq);
+        Bindwood.noteEndTime('Pulling records');
+    },
+
+    recordInCurrentProfile: function(record) {
+        if (record.application_annotations &&
+            record.application_annotations.Firefox &&
+            record.application_annotations.Firefox.profile &&
+            record.application_annotations.Firefox.profile == Bindwood.currentProfile) {
+            return true;
+        }
+        return false;
+    },
+
+    isDeleted: function(record) {
+        if (record.application_annotations &&
+            record.application_annotations["Ubuntu One"] &&
+            record.application_annotations["Ubuntu One"].private_application_annotations &&
+            record.application_annotations["Ubuntu One"].private_application_annotations.deleted) {
+            return true;
+        }
+        return false;
+    },
+
+    deleteLocalRecord: function(record) {
+        // If we can't resolve the itemId, even by looking up URI,
+        // assume it's already gone.
+        var itemId = Bindwood.itemIdForUUID(record._id);
+        if (itemId) {
+            return Bindwood.bookmarksService.removeItem(itemId);
+        }
+    },
+
+    processCouchBookmarkRevision: function(record) {
+        // Could be an add or change revision. Delete was handled earlier.
+        // If it's an addition (we can't resolve its _id to be one of our itemIds),
+        // add it to the Desktop Couch folder in unfiled.
+        Bindwood.writeMessage("Processing bookmark record: " + JSON.stringify(record));
+        var itemId = Bindwood.itemIdForUUID(record._id);
+        if (itemId) {
+            // It's a change. Stamp everything remote on the local bookmark
+            Bindwood.bookmarksService.setItemTitle(itemId, record.title);
+            Bindwood.bookmarksService.changeBookmarkURI(itemId,
+                Bindwood.ioService.newURI(record.uri, null, null));
+        } else {
+            // It's an addition. Add a new bookmark to our scratch folder,
+            // annotate it, and we're done.
+            itemId = Bindwood.bookmarksService.insertBookmark(
+                Bindwood.scratch_folder,
+                Bindwood.ioService.newURI(record.uri, null, null),
+                -1,
+                record.title);
+            Bindwood.annotateItemWithUUID(itemId, record._id);
+        }
+    },
+
+    processCouchFolderRevision: function(record) {
+        // Could be an add or change revision. Delete was handled earlier.
+        // If it's an addition (we can't resolve its _id to be one of our itemIds),
+        // add it to the Desktop Couch folder in unfiled.
+        Bindwood.writeMessage("Processing folder record: " + JSON.stringify(record));
+        var itemId = Bindwood.itemIdForUUID(record._id);
+        if (itemId) {
+            // It's a change. Stamp remote title on the folder, and deal with any
+            // changed children.
+            Bindwood.noteStartTime('Shuffling folder children');
+            Bindwood.bookmarksService.setItemTitle(itemId, record.title);
+            // Iterate through our current folder children, and compare with remote.
+            // Move all local children to the scratch folder, then move them back
+            // in the order of the remote children.
+            var local_children = Bindwood.getUUIDsFromFolder(itemId);
+            Bindwood.writeMessage("Moving local children " + JSON.stringify(local_children) + " to scratch folder");
+            for (var i = 0; i<local_children.length; i++) {
+                var child = local_children[i];
+                var child_itemId = Bindwood.itemIdForUUID(child);
+                try {
+                    Bindwood.bookmarksService.moveItem(child_itemId, Bindwood.scratch_folder, -1);
+                } catch(e) {
+                    Bindwood.writeError("Problem moving item to scratch folder: " + JSON.stringify(e), e);
+                }
+            }
+            Bindwood.writeMessage("Moving children identified by record " + JSON.stringify(record.children) + " to this folder");
+            for (var j = 0; j<record.children.length; j++) {
+                var new_child = record.children[j];
+                var new_child_itemId = Bindwood.itemIdForUUID(new_child);
+                try {
+                    Bindwood.bookmarksService.moveItem(new_child_itemId, itemId, -1);
+                } catch(e) {
+                    Bindwood.writeError("Problem moving item from scratch folder: " + JSON.stringify(e), e);
+                }
+            }
+            Bindwood.noteEndTime('Shuffling folder children');
+        } else {
+            // It's an addition. Add a new bookmark to our scratch folder,
+            // annotate it, and we're done.
+            itemId = Bindwood.bookmarksService.createFolder(
+                Bindwood.scratch_folder,
+                record.title,
+                -1);
+            Bindwood.annotateItemWithUUID(itemId, record._id);
+        }
+    },
+
+    processCouchFeedRevision: function(record) {
+        // Could be an add or change revision. Delete was handled earlier.
+        // If it's an addition (we can't resolve its _id to be one of our itemIds),
+        // add it to the Desktop Couch folder in unfiled.
+        Bindwood.writeMessage("Processing feed record: " + JSON.stringify(record));
+        var itemId = Bindwood.itemIdForUUID(record._id);
+        if (itemId) {
+            // It's a change. Stamp everything remote on the local bookmark
+            Bindwood.bookmarksService.setItemTitle(itemId, record.title);
+            Bindwood.livemarkService.setSiteURI(itemId,
+                Bindwood.ioService.newURI(record.site_uri, null, null));
+            Bindwood.livemarkService.setFeedURI(itemId,
+                Bindwood.ioService.newURI(record.feed_uri, null, null));
+        } else {
+            // It's an addition. Add a new bookmark to our scratch folder,
+            // annotate it, and we're done.
+            var newItemId = Bindwood.livemarkService.createLivemarkFolderOnly(
+                Bindwood.bookmarksService,
+                Bindwood.scratch_folder,
+                record.title,
+                Bindwood.ioService.newURI(record.site_uri, null, null),
+                Bindwood.ioService.newURI(record.feed_uri, null, null),
+                -1);
+            Bindwood.annotateItemWithUUID(newItemId, record._id);
+        }
+    },
+
+    processCouchSeparatorRevision: function(record) {
+        // Should only be an add revision. There's nothing to change, and delete was
+        // handled earlier.
+        // If it's an addition (we can't resolve its _id to be one of our itemIds),
+        // add it to the Desktop Couch folder in unfiled.
+        Bindwood.writeMessage("Processing separator record: " + JSON.stringify(record));
+        var itemId = Bindwood.itemIdForUUID(record._id);
+        if (!itemId) {
+            // There's nothing to change about a separator, so...
+            // It's an addition. Add a new bookmark to our scratch folder,
+            // annotate it, and we're done.
+            var newItemId = Bindwood.bookmarksService.insertSeparator(
+                Bindwood.scratch_folder,
+                -1);
+            Bindwood.annotateItemWithUUID(newItemId, record._id);
+        }
     },
 
     updateDocAndSave: function(uuid, attribute, value, callback) {
-        Bindwood.writeMessage("Updating a document (" + uuid + ") setting (" + attribute + ") to (" + value + ")");
-        // Some attributes that we track should remain inside the application_annotations object
+        Bindwood.writeMessage(
+            "Updating a document (" +
+                uuid +
+                ") setting (" +
+                attribute +
+                ") to (" + value + ")");
+
+        // Some attributes that we track should remain inside the
+        // application_annotations object
         var attrMap = {
-             title: false,
-             uri: false,
-             deleted: false,
-             uuid: true,
-             folder: true,
-             favicon: true,
-             profle: true };
-        var results = Bindwood.findDocumentByUUID(uuid);
-        if (!results) {
-            throw {error: "Could not find document in Couch"};
-        }
-        var doc = Bindwood.couch.open(results.rows[0].id);
-        if (attrMap[attribute.toString()]) { // belongs on annotations
+             title: true,
+             uri: true,
+             feed_uri: true,
+             site_uri: true,
+             children: true,
+             favicon: false,
+             profile: false };
+
+        var doc = Bindwood.couch.open(uuid);
+        if (attrMap[attribute.toString()] || false) { // belongs at top-level
+            doc[attribute.toString()] = value.toString();
+        } else {
             if (!doc.application_annotations) {
                 doc.application_annotations = {};
             }
@@ -714,23 +979,20 @@
                 doc.application_annotations.Firefox = {};
             }
             doc.application_annotations.Firefox[attribute.toString()] = value.toString();
-        } else {
-            doc[attribute.toString()] = value.toString();
         }
         try {
-            var result = Bindwood.couch.save(doc);
-            Bindwood.writeMessage("Successfully save document (" + uuid + ") back to Couch.");
+            var response = Bindwood.couch.save(doc);
+            Bindwood.seen_revisions[response.rev] = true;
         } catch(e) {
             Bindwood.writeError("Problem saving document to Couch", e);
             throw e;
         }
 
         if (callback) {
-            Bindwood.writeMessage("We've got a callback to run: " + callback.toString());
             callback();
         }
 
-        return result;
+        return response;
     },
 
     itemWeCareAbout: function(itemId) {
@@ -740,6 +1002,7 @@
         var root = 0;
         var parent;
         while (parent != root) {
+            Bindwood.writeMessage("Looking for parent of " + itemId);
             parent = Bindwood.bookmarksService.getFolderIdForItem(itemId);
             if (parent != root &&
                 Bindwood.annotationService.itemHasAnnotation(
@@ -754,69 +1017,20 @@
     Observer: {
         // An nsINavBookmarkObserver
         onItemAdded: function(aItemId, aFolder, aIndex) {
-            // A bookmark has been added, so we create a blank entry
+            Bindwood.writeMessage("onItemAdded: called when push is " + Bindwood.push);
+            // An item has been added, so we create a blank entry
             // in Couch with our local itemId attached.
             if (!Bindwood.itemWeCareAbout(aItemId)) {
                 Bindwood.writeMessage("Ignoring this add event");
                 return;
             }
 
-            Bindwood.writeMessage("A new bookmark was created. Its id is: " + aItemId +
-                                  " at location: " + aIndex +
-                                  " in folder: " + aFolder );
-            netscape.security.PrivilegeManager.enablePrivilege("UniversalBrowserRead UniversalBrowserWrite");
-
-            var uuid = Bindwood.uuidForItemId(aItemId);
-            Bindwood.writeMessage("Determined uuid for new bookmark: " + uuid);
-
-            var folder;
-            switch(aFolder) {
-                case Bindwood.bookmarksService.toolbarFolder:
-                    folder = "toolbarFolder";
-                    break;
-                case Bindwood.bookmarksService.bookmarksMenuFolder:
-                    folder = "bookmarksMenuFolder";
-                    break;
-                case Bindwood.bookmarksService.unfiledBookmarksFolder:
-                    folder = "unfiledBookmarksFolder";
-                    break;
-                default:
-                    folder = Bindwood.bookmarksService.getItemTitle(aFolder);
-                    break;
-            }
-            var uri;
-            var title;
-
-            Bindwood.writeMessage("Going to look up bookmark URI");
-            try {
-                uri = Bindwood.bookmarksService.getBookmarkURI(aItemId).spec;
-            } catch(e) {
-                Bindwood.writeError("Problem with something: ", error);
-                uri = '';
-            }
-
-            Bindwood.writeMessage("Going to look up bookmark Title");
-            try {
-                title = Bindwood.bookmarksService.getItemTitle(aItemId);
-            } catch(e) {
-                Bindwood.writeError("Problem with something: ", error);
-                title = '';
-            }
-
-            var doc = {
-                record_type: "http://www.freedesktop.org/wiki/Specifications/desktopcouch/bookmark",
-                uri: uri,
-                title: title,
-                application_annotations: {
-                    Firefox: {
-                        uuid: uuid,
-                        folder: folder,
-                        profile: Bindwood.currentProfile
-                    }
-                }
-            };
-
-            Bindwood.writeMessage("Created a minimal record document with our uuid: " + JSON.stringify(doc));
+            Bindwood.writeMessage(
+                "A new item was created. Its id is: " + aItemId +
+                    " at location: " + aIndex +
+                    " in folder: " + aFolder );
+            netscape.security.PrivilegeManager.enablePrivilege(
+                "UniversalBrowserRead UniversalBrowserWrite");
 
             switch (Bindwood.push) {
             case 'DISABLED':
@@ -824,47 +1038,77 @@
                 break;
             case 'ENABLED':
                 try {
-                    var result = Bindwood.couch.save(doc);
+                    var doc = Bindwood.couchRecordForItemId(aItemId);
+                    var response = Bindwood.couch.save(doc);
                     Bindwood.writeMessage("Saved new, bare record to Couch.");
+                    Bindwood.seen_revisions[response.rev] = true;
+                    Bindwood.pushFolderChildren(aFolder);
                 } catch(e) {
-                    Bindwood.writeError("Problem saving new bookmark to Couch: ", e);
+                    Bindwood.writeError(
+                        "Problem saving new bookmark to Couch: ", e);
                 }
                 break;
             default:
                 break;
             }
+
+            Bindwood.setLatestModified(
+                Bindwood.bookmarksService.getItemLastModified(aItemId));
         },
         onBeforeItemRemoved: function(aItemId) {
+            Bindwood.writeMessage("onBeforeItemRemoved: called when push is " + Bindwood.push);
             // A bookmark has been removed. This is called before it's
             // been removed locally, though we're passed the itemId,
             // which we use to delete from Couch.
+            var folderId = Bindwood.bookmarksService.getFolderIdForItem(aItemId);
             if (!Bindwood.itemWeCareAbout(aItemId)) {
                 Bindwood.writeMessage("Ignoring this before remove event");
                 return;
             }
 
-            Bindwood.writeMessage("Record " + aItemId + " is about to be removed locally.");
+            Bindwood.writeMessage(
+                "Record " + aItemId + " is about to be removed locally.");
             var uuid = Bindwood.uuidForItemId(aItemId);
 
             switch (Bindwood.push) {
             case 'DISABLED':
                 delete Bindwood.uuidItemIdMap[uuid];
-                Bindwood.writeMessage("Deleted from local uuid map, but not saving back to Couch.");
+                Bindwood.writeMessage(
+                    "Deleted from local uuid map, but not saving back to Couch.");
                 break;
             case 'ENABLED':
-                netscape.security.PrivilegeManager.enablePrivilege("UniversalBrowserRead UniversalBrowserWrite");
+                netscape.security.PrivilegeManager.enablePrivilege(
+                    "UniversalBrowserRead UniversalBrowserWrite");
 
+                var doc = Bindwood.couch.open(uuid);
+                if (!doc.application_annotations) {
+                    doc.application_annotations = {};
+                }
+                if (!doc.application_annotations['Ubuntu One']) {
+                    doc.application_annotations['Ubuntu One'] = {};
+                }
+                if (!doc.application_annotations['Ubuntu One'].private_application_annotations) {
+                    doc.application_annotations['Ubuntu One'].private_application_annotations = {};
+                }
+                doc.application_annotations['Ubuntu One'].private_application_annotations.deleted = true;
+                
                 try {
                     // Also remove from our local cache and remove
                     // annotation from service.
-                    var result = Bindwood.updateDocAndSave(
-                        uuid, 'deleted', true,
-                        function() {
-                            delete Bindwood.uuidItemIdMap[uuid];
-                            Bindwood.writeMessage("Deleted local reference in the uuid-itemId mapping."); });
-                    Bindwood.writeMessage("Saved document back to Couch with deleted flag set.");
+                    var response = Bindwood.couch.save(doc);
+                    Bindwood.seen_revisions[response.rev] = true;
+                    delete Bindwood.uuidItemIdMap[uuid];
+                    Bindwood.writeMessage(
+                        "Deleted local reference in the" +
+                            " uuid-itemId mapping.");
+                    Bindwood.writeMessage(
+                        "Saved document back to Couch with deleted flag set.");
+                    var new_children = Bindwood.getUUIDsFromFolder(folderId);
+                    new_children.splice(new_children.indexOf(uuid), 1);
+                    Bindwood.pushFolderChildren(folderId, new_children);
                 } catch(e) {
-                    Bindwood.writeError("Problem pushing deleted record to Couch: ", e);
+                    Bindwood.writeError(
+                        "Problem pushing deleted record to Couch: ", e);
                 }
                 break;
             default:
@@ -872,6 +1116,7 @@
             }
         },
         onItemRemoved: function(aItemId, aFolder, aIndex) {
+            Bindwood.writeMessage("onItemRemoved: called when push is " + Bindwood.push);
             // This only happens locally, so there's never a need to push
             if (!Bindwood.itemWeCareAbout(aItemId)) {
                 Bindwood.writeMessage("Ignoring this remove event");
@@ -879,39 +1124,85 @@
             }
 
             Bindwood.makeLocalChangeOnly(
-                function() { return Bindwood.annotationService.removeItemAnnotation(aItemId, Bindwood.annotationKey); });
-            Bindwood.writeMessage("Removed annotations from bookmark identified by: " + aItemId);
+                function() {
+                    return Bindwood.annotationService.removeItemAnnotation(
+                        aItemId, Bindwood.annotationKey); });
+            Bindwood.writeMessage(
+                "Removed annotations from bookmark identified by: " + aItemId);
         },
-        onItemChanged: function(aBookmarkId, aProperty, aIsAnnotationProperty, aValue) {
+        onItemChanged: function(aItemId, aProperty, aIsAnnotationProperty, aValue) {
+            Bindwood.writeMessage("onItemChanged: called when push is " + Bindwood.push);
             // A property of a bookmark has changed. On multiple
             // property updates, this will be called multiple times,
             // once per property (i.e., for title and URI)
-            if (!Bindwood.itemWeCareAbout(aBookmarkId)) {
+            if (!Bindwood.itemWeCareAbout(aItemId)) {
                 Bindwood.writeMessage("Ignoring this change event");
                 return;
             }
 
-            if (!aIsAnnotationProperty) { // We only care if these are bookmark properties (for right now)
-                Bindwood.writeMessage("A property (" + aProperty + ") on bookmark id: " + aBookmarkId + " has been set to: " + aValue);
-                var uuid = Bindwood.uuidForItemId(aBookmarkId);
-
-                switch (Bindwood.push) {
-                case 'DISABLED':
-                    Bindwood.writeMessage("Updated, but not saving back to Couch.");
-                    break;
-                case 'ENABLED':
-                    Bindwood.writeMessage("We will push this change back to Couch.");
-                    netscape.security.PrivilegeManager.enablePrivilege("UniversalBrowserRead UniversalBrowserWrite");
-                    try {
-                        var result = Bindwood.updateDocAndSave(
-                            uuid, aProperty.toString(), aValue.toString(),
-                            function() { Bindwood.writeMessage("Saved the document back to Couch"); });
-                    } catch(e) {
-                        Bindwood.writeError("Problem saving updated bookmark to Couch: ", e);
-                    }
-                    break;
-                default:
-                    break;
+            Bindwood.writeMessage(
+                "A property (" +
+                    aProperty +
+                    ") on item id: " + aItemId +
+                    " has been set to: " + aValue);
+            var uuid = Bindwood.uuidForItemId(aItemId);
+
+            switch (Bindwood.push) {
+            case 'DISABLED':
+                Bindwood.writeMessage(
+                    "Updated, but not saving back to Couch.");
+                break;
+            case 'ENABLED':
+                Bindwood.writeMessage(
+                    "We will push this change back to Couch.");
+                netscape.security.PrivilegeManager.enablePrivilege(
+                    "UniversalBrowserRead UniversalBrowserWrite");
+                try {
+                    var result = Bindwood.updateDocAndSave(
+                        uuid, aProperty.toString(), aValue.toString(),
+                        function() {
+                            Bindwood.writeMessage(
+                                "Saved the document back to Couch"); });
+                } catch(e) {
+                    Bindwood.writeError(
+                        "Problem saving updated bookmark to Couch: ", e);
+                }
+                break;
+            default:
+                break;
+            }
+
+            Bindwood.setLatestModified(
+                Bindwood.bookmarksService.getItemLastModified(aItemId));
+        },
+
+        onItemMoved: function(aItemId, aOldParent, aOldIndex, aNewParent, aNewIndex) {
+            Bindwood.writeMessage("onItemMoved: called when push is " + Bindwood.push);
+            Bindwood.writeMessage(
+                "The item: " + aItemId + " was moved from (" + aOldParent + ", " + aOldIndex +
+                ") to (" + aNewParent + ", " + aNewIndex + ")"
+            );
+            var uuid = Bindwood.uuidForItemId(aItemId);
+            var old_parent_uuid = Bindwood.uuidForItemId(aOldParent);
+            var old_parent_doc = Bindwood.couch.open(old_parent_uuid);
+            old_parent_doc.children = Bindwood.getUUIDsFromFolder(aOldParent);
+            try {
+                var response = Bindwood.couch.save(old_parent_doc);
+                Bindwood.seen_revisions[response.rev] = true;
+            } catch(e) {
+                Bindwood.writeError(
+                    "Problem saving updated old parent doc to Couch: ", e);
+            }
+            if (aOldParent != aNewParent) {
+                var new_parent_uuid = Bindwood.uuidForItemId(aNewParent);
+                var new_parent_doc = Bindwood.couch.open(new_parent_uuid);
+                new_parent_doc.children = Bindwood.getUUIDsFromFolder(aNewParent);
+                try {
+                    var response = Bindwood.couch.save(new_parent_doc);
+                    Bindwood.seen_revisions[response.rev] = true;
+                } catch(e) {
+                    Bindwood.writeError(
+                        "Problem saving updated new parent doc to Couch: ", e);
                 }
             }
         },
@@ -919,7 +1210,6 @@
         // Currently unhandled
         onBeginUpdateBatch: function() {},
         onEndUpdateBatch: function() {},
-        onItemMoved: function(aItemId, aOldParent, aOldIndex, aNewParent, aNewIndex) {},
         onItemVisited: function(aBookmarkId, aVisitID, time) {},
 
         // Query Interface

=== modified file 'content/browserOverlay.xul'
--- content/browserOverlay.xul	2009-09-02 17:07:11 +0000
+++ content/browserOverlay.xul	2010-01-22 19:18:13 +0000
@@ -19,10 +19,9 @@
 <overlay id="bindwood"
          xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
 
-  <script type="application/x-javascript" src="chrome://bindwood/content/sha1.js" />
   <script type="application/x-javascript" src="chrome://bindwood/content/oauth.js" />
   <script type="application/x-javascript" src="chrome://bindwood/content/couch.js" />
-  <script type="application/x-javascript" src="chrome://bindwood/content/sync.js" />
+  <script type="application/x-javascript" src="chrome://bindwood/content/bindwood.js" />
 
   <window id="main-window">
     <script type="application/x-javascript">

=== modified file 'content/couch.js'
--- content/couch.js	2009-10-07 20:39:43 +0000
+++ content/couch.js	2010-01-22 19:18:13 +0000
@@ -208,6 +208,21 @@
     return this.allDocs({startkey:"_design", endkey:"_design0"});
   };
 
+  this.changes = function(options,keys) {
+    var req = null;
+    if(!keys) {
+      req = this.request("GET", this.uri + "_changes" + encodeOptions(options));
+    } else {
+      req = this.request("POST", this.uri + "_changes" + encodeOptions(options),
+                         {
+                             headers: {"Content-Type": "application/json"},
+                             body: JSON.stringify({keys:keys})
+                         });
+    }
+    CouchDB.maybeThrowError(req);
+    return JSON.parse(req.responseText);
+  };
+
   this.allDocsBySeq = function(options,keys) {
     var req = null;
     if(!keys) {

=== removed file 'content/sha1.js'
--- content/sha1.js	2009-09-02 17:07:11 +0000
+++ content/sha1.js	1970-01-01 00:00:00 +0000
@@ -1,202 +0,0 @@
-/*
- * A JavaScript implementation of the Secure Hash Algorithm, SHA-1, as defined
- * in FIPS PUB 180-1
- * Version 2.1a Copyright Paul Johnston 2000 - 2002.
- * Other contributors: Greg Holt, Andrew Kepert, Ydnar, Lostinet
- * Distributed under the BSD License
- * See http://pajhome.org.uk/crypt/md5 for details.
- */
-
-/*
- * Configurable variables. You may need to tweak these to be compatible with
- * the server-side, but the defaults work in most cases.
- */
-var hexcase = 0;  /* hex output format. 0 - lowercase; 1 - uppercase        */
-var b64pad  = ""; /* base-64 pad character. "=" for strict RFC compliance   */
-var chrsz   = 8;  /* bits per input character. 8 - ASCII; 16 - Unicode      */
-
-/*
- * These are the functions you'll usually want to call
- * They take string arguments and return either hex or base-64 encoded strings
- */
-function hex_sha1(s){return binb2hex(core_sha1(str2binb(s),s.length * chrsz));}
-function b64_sha1(s){return binb2b64(core_sha1(str2binb(s),s.length * chrsz));}
-function str_sha1(s){return binb2str(core_sha1(str2binb(s),s.length * chrsz));}
-function hex_hmac_sha1(key, data){ return binb2hex(core_hmac_sha1(key, data));}
-function b64_hmac_sha1(key, data){ return binb2b64(core_hmac_sha1(key, data));}
-function str_hmac_sha1(key, data){ return binb2str(core_hmac_sha1(key, data));}
-
-/*
- * Perform a simple self-test to see if the VM is working
- */
-function sha1_vm_test()
-{
-  return hex_sha1("abc") == "a9993e364706816aba3e25717850c26c9cd0d89d";
-}
-
-/*
- * Calculate the SHA-1 of an array of big-endian words, and a bit length
- */
-function core_sha1(x, len)
-{
-  /* append padding */
-  x[len >> 5] |= 0x80 << (24 - len % 32);
-  x[((len + 64 >> 9) << 4) + 15] = len;
-
-  var w = Array(80);
-  var a =  1732584193;
-  var b = -271733879;
-  var c = -1732584194;
-  var d =  271733878;
-  var e = -1009589776;
-
-  for(var i = 0; i < x.length; i += 16)
-  {
-    var olda = a;
-    var oldb = b;
-    var oldc = c;
-    var oldd = d;
-    var olde = e;
-
-    for(var j = 0; j < 80; j++)
-    {
-      if(j < 16) w[j] = x[i + j];
-      else w[j] = rol(w[j-3] ^ w[j-8] ^ w[j-14] ^ w[j-16], 1);
-      var t = safe_add(safe_add(rol(a, 5), sha1_ft(j, b, c, d)),
-                       safe_add(safe_add(e, w[j]), sha1_kt(j)));
-      e = d;
-      d = c;
-      c = rol(b, 30);
-      b = a;
-      a = t;
-    }
-
-    a = safe_add(a, olda);
-    b = safe_add(b, oldb);
-    c = safe_add(c, oldc);
-    d = safe_add(d, oldd);
-    e = safe_add(e, olde);
-  }
-  return Array(a, b, c, d, e);
-
-}
-
-/*
- * Perform the appropriate triplet combination function for the current
- * iteration
- */
-function sha1_ft(t, b, c, d)
-{
-  if(t < 20) return (b & c) | ((~b) & d);
-  if(t < 40) return b ^ c ^ d;
-  if(t < 60) return (b & c) | (b & d) | (c & d);
-  return b ^ c ^ d;
-}
-
-/*
- * Determine the appropriate additive constant for the current iteration
- */
-function sha1_kt(t)
-{
-  return (t < 20) ?  1518500249 : (t < 40) ?  1859775393 :
-         (t < 60) ? -1894007588 : -899497514;
-}
-
-/*
- * Calculate the HMAC-SHA1 of a key and some data
- */
-function core_hmac_sha1(key, data)
-{
-  var bkey = str2binb(key);
-  if(bkey.length > 16) bkey = core_sha1(bkey, key.length * chrsz);
-
-  var ipad = Array(16), opad = Array(16);
-  for(var i = 0; i < 16; i++)
-  {
-    ipad[i] = bkey[i] ^ 0x36363636;
-    opad[i] = bkey[i] ^ 0x5C5C5C5C;
-  }
-
-  var hash = core_sha1(ipad.concat(str2binb(data)), 512 + data.length * chrsz);
-  return core_sha1(opad.concat(hash), 512 + 160);
-}
-
-/*
- * Add integers, wrapping at 2^32. This uses 16-bit operations internally
- * to work around bugs in some JS interpreters.
- */
-function safe_add(x, y)
-{
-  var lsw = (x & 0xFFFF) + (y & 0xFFFF);
-  var msw = (x >> 16) + (y >> 16) + (lsw >> 16);
-  return (msw << 16) | (lsw & 0xFFFF);
-}
-
-/*
- * Bitwise rotate a 32-bit number to the left.
- */
-function rol(num, cnt)
-{
-  return (num << cnt) | (num >>> (32 - cnt));
-}
-
-/*
- * Convert an 8-bit or 16-bit string to an array of big-endian words
- * In 8-bit function, characters >255 have their hi-byte silently ignored.
- */
-function str2binb(str)
-{
-  var bin = Array();
-  var mask = (1 << chrsz) - 1;
-  for(var i = 0; i < str.length * chrsz; i += chrsz)
-    bin[i>>5] |= (str.charCodeAt(i / chrsz) & mask) << (32 - chrsz - i%32);
-  return bin;
-}
-
-/*
- * Convert an array of big-endian words to a string
- */
-function binb2str(bin)
-{
-  var str = "";
-  var mask = (1 << chrsz) - 1;
-  for(var i = 0; i < bin.length * 32; i += chrsz)
-    str += String.fromCharCode((bin[i>>5] >>> (32 - chrsz - i%32)) & mask);
-  return str;
-}
-
-/*
- * Convert an array of big-endian words to a hex string.
- */
-function binb2hex(binarray)
-{
-  var hex_tab = hexcase ? "0123456789ABCDEF" : "0123456789abcdef";
-  var str = "";
-  for(var i = 0; i < binarray.length * 4; i++)
-  {
-    str += hex_tab.charAt((binarray[i>>2] >> ((3 - i%4)*8+4)) & 0xF) +
-           hex_tab.charAt((binarray[i>>2] >> ((3 - i%4)*8  )) & 0xF);
-  }
-  return str;
-}
-
-/*
- * Convert an array of big-endian words to a base-64 string
- */
-function binb2b64(binarray)
-{
-  var tab = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
-  var str = "";
-  for(var i = 0; i < binarray.length * 4; i += 3)
-  {
-    var triplet = (((binarray[i   >> 2] >> 8 * (3 -  i   %4)) & 0xFF) << 16)
-                | (((binarray[i+1 >> 2] >> 8 * (3 - (i+1)%4)) & 0xFF) << 8 )
-                |  ((binarray[i+2 >> 2] >> 8 * (3 - (i+2)%4)) & 0xFF);
-    for(var j = 0; j < 4; j++)
-    {
-      if(i * 8 + j * 6 > binarray.length * 32) str += b64pad;
-      else str += tab.charAt((triplet >> 6*(3-j)) & 0x3F);
-    }
-  }
-  return str;
-}

=== modified file 'couchdb_env.sh'
--- couchdb_env.sh	2009-10-06 17:36:56 +0000
+++ couchdb_env.sh	2010-01-22 19:18:13 +0000
@@ -24,5 +24,3 @@
 else
   echo $PORT:$TOKENS > $OUT
 fi
-
-

=== modified file 'install.rdf'
--- install.rdf	2009-10-21 19:32:47 +0000
+++ install.rdf	2010-01-22 19:18:13 +0000
@@ -4,7 +4,7 @@
 
   <Description about="urn:mozilla:install-manifest">
     <em:id>[e-mail]</em:id>
-    <em:version>0.4.2</em:version>
+    <em:version>0.5</em:version>
     <em:type>2</em:type>
 
     <!-- Target Application this extension can install into,

=== added directory 'tests'
=== added file 'tests/run_test.sh'
--- tests/run_test.sh	1970-01-01 00:00:00 +0000
+++ tests/run_test.sh	2010-01-22 19:18:13 +0000
@@ -0,0 +1,20 @@
+#!/bin/bash
+
+CONSUMER_KEY=ck$RANDOM
+CONSUMER_SECRET=cs$RANDOM
+TOKEN=tk$RANDOM
+TOKEN_SECRET=ts$RANDOM
+METHOD=$1
+ACTION=$2
+SIGNATURE_METHOD=$3
+NONCE=$RANDOM
+TIMESTAMP=1251105028
+
+TESTDIR=$(dirname $0)
+js -f $TESTDIR/../content/oauth.js  -f $TESTDIR/../content/sha1.js \
+    $TESTDIR/test_oauth.js $CONSUMER_KEY $CONSUMER_SECRET $TOKEN $TOKEN_SECRET \
+    http://$ACTION $METHOD $NONCE $TIMESTAMP $SIGNATURE_METHOD
+
+python $TESTDIR/test_oauth.py $CONSUMER_KEY $CONSUMER_SECRET $TOKEN $TOKEN_SECRET \
+    http://$ACTION $METHOD $NONCE $TIMESTAMP $SIGNATURE_METHOD
+

=== added file 'tests/test_get_views.js'
--- tests/test_get_views.js	1970-01-01 00:00:00 +0000
+++ tests/test_get_views.js	2010-01-22 19:18:13 +0000
@@ -0,0 +1,135 @@
+var environment = arguments[0];
+var dbname = arguments[1];
+
+var env_array = environment.split(':');
+var port = env_array[0];
+var consumer_key = env_array[1];
+var consumer_secret = env_array[2];
+var token = env_array[3];
+var token_secret = env_array[4];
+
+if (XMLHttpRequest === undefined) {
+    var XMLHttpRequest = function () {
+        return Components.classes["@mozilla.org/xmlextras/xmlhttprequest;1"]
+            .getService(Components.interfaces.nsIXMLHttpRequest);
+    };
+}
+
+print("Assigning environment variables to CouchDB");
+CouchDB.port = port;
+CouchDB.accessor = {
+    consumerSecret: consumer_secret,
+    tokenSecret: token_secret
+};
+CouchDB.message = {
+    parameters: {
+        oauth_callback: "None",
+        oauth_consumer_key: consumer_key,
+        oauth_signature_method: "PLAINTEXT",
+        oauth_token: token,
+        oauth_verifier: "None",
+        oauth_version: "1.0"
+    }
+};
+
+print("Creating a CouchDB instance for " + dbname);
+var couch = new CouchDB(dbname);
+try {
+    couch.createDb();
+} catch (e) {
+    if (e.error == 'file_exists') {
+        print("Database already exists. We're okay.");
+    } else {
+        print("Error when creating database in pushBookmarks: " + JSON.stringify(e));
+    }
+}
+
+var views = [{ id: "_design/all_bookmarks",
+               map: "function(doc) { " +
+               "var scheme = doc.uri.split(':',1)[0]; " +
+               "var uri; " +
+               "if (scheme == 'http' || scheme == 'https') {" +
+               "uri = doc.uri.split('/')[2];" +
+               "if (uri.length < 30) {" +
+               " uri += '/' + " +
+               "doc.uri.split('/',4)[3].substr(0,30-uri.length) + '...';" +
+               "}" +
+               "} else {" +
+               "uri = scheme + ' URL';" +
+               "}" +
+               "if (!doc.deleted) {" +
+               "emit(doc.title, uri);" +
+               "}" +
+               "}" },
+             { id: "_design/deleted_bookmarks",
+               map: "function(doc) { if (doc.deleted) { emit (doc.title, doc.uri); } }" },
+             { id: "_design/default",
+               map: "function(doc) { var scheme = doc.uri.split(':',1)[0]; var uri; if (scheme == 'http' || scheme == 'https') {uri = doc.uri.split('/')[2];if (uri.length < 30) { uri += '/' + doc.uri.split('/',4)[3].substr(0,30-uri.length) + '...';}} else {uri = scheme + ' URL';}if ((!doc.deleted) && (doc.application_annotations.Firefox.profile == 'default')) {emit(doc.title, uri);}}" }];
+for (var i = 0; i < views.length; i++) {
+    var view_info = views[i];
+    var dirty = false;
+    try {
+        var new_doc;
+        var current_doc;
+        try {
+            print("Looking up " + view_info.id);
+            current_doc = couch.open(view_info.id);
+        } catch(e) {
+            print("Problem fetching " + view_info.id + " because of: " + JSON.stringify(e));
+            current_doc = null;
+        }
+        if (current_doc !== null) {
+            print("We've got an existing doc");
+            new_doc = current_doc;
+            var old_map = current_doc.views.display.map;
+            if (old_map != view_info.map) { // Ours is definitive
+                print("Existing map is different, clobbering.");
+                new_doc.views.display.map = view_info.map;
+                dirty = true;
+            }
+        } else {
+            print("Doc doesn't exist, making it from scratch");
+            new_doc = {
+                _id: view_info.id,
+                views: {
+                    display: {
+                        map: view_info.map
+                    }
+                }
+            };
+            dirty = true;
+        }
+        if (dirty) {
+            print("Something changed with the doc, we're saving it back");
+            try {
+                couch.save(new_doc);
+                print("Successfully save view " + new_doc._id);
+            } catch(e) {
+                print("Problem saving view: " + JSON.stringify(e));
+            }
+        }
+    } catch(e) {
+        // some kind of error fetching the existing design doc
+        print("Problem checking for view: " + JSON.stringify(e));
+    }
+}
+
+try {
+    var current_profile = 'default';
+    var rows = couch.query(function (doc) {
+                               if (doc.record_type == "http://www.freedesktop.org/wiki/Specifications/desktopcouch/bookmark" &&
+                                   doc.application_annotations &&
+                                   doc.application_annotations.Firefox &&
+                                   doc.application_annotations.Firefox.profile &&
+                                   doc.application_annotations.Firefox.profile == current_profile) {
+                                   emit(doc._id,doc);
+                               }
+                           });
+} catch(e) {
+    Bindwood.writeError("Problem fetching all bookmarks from Couch: ", e);
+}
+for (var i = 0; i < rows.rows.length; i++) {
+    var recordid = rows.rows[i].id;
+    print("Pulling record: " + recordid);
+}
+

=== added file 'tests/test_oauth.js'
--- tests/test_oauth.js	1970-01-01 00:00:00 +0000
+++ tests/test_oauth.js	2010-01-22 19:18:13 +0000
@@ -0,0 +1,40 @@
+var consumer_key = arguments[0];
+var consumer_secret = arguments[1];
+var token = arguments[2];
+var token_secret = arguments[3];
+var action = arguments[4];
+var method = arguments[5];
+var nonce = arguments[6];
+var timestamp = arguments[7];
+var signature_method = arguments[8];
+
+if (method != 'GET') {
+    action = encodeURIComponent(action);
+}
+
+var message = {
+    parameters: {
+        oauth_signature_method: signature_method,
+        oauth_consumer_key: consumer_key,
+        oauth_token: token,
+        oauth_token_secret: token_secret,
+        oauth_version: "1.0",
+        oauth_nonce: nonce,
+        oauth_timestamp: timestamp
+    },
+    action: action,
+    method: method
+};
+var accessor = {
+    consumerSecret: consumer_secret,
+    tokenSecret: token_secret
+};
+OAuth.completeRequest(message, accessor);
+var hdr = OAuth.getAuthorizationHeader('', message.parameters);
+var parts = hdr.split(",");
+for (var i=0; i<parts.length; i++) {
+    var smallparts = parts[i].split("=");
+    if (smallparts[0] == "oauth_signature") {
+        print("Signature from JavaScript: " + smallparts[1]);
+    }
+}

=== added file 'tests/test_oauth.py'
--- tests/test_oauth.py	1970-01-01 00:00:00 +0000
+++ tests/test_oauth.py	2010-01-22 19:18:13 +0000
@@ -0,0 +1,20 @@
+#!/usr/bin/python
+import sys
+from oauth import oauth
+
+(consumer_key, consumer_secret, actual_token, token_secret, url, 
+  method, nonce, timestamp, signature_method) = sys.argv[1:]
+
+signature_function = oauth.OAuthSignatureMethod_PLAINTEXT if signature_method == 'PLAINTEXT' else oauth.OAuthSignatureMethod_HMAC_SHA1
+
+consumer=oauth.OAuthConsumer(consumer_key, consumer_secret)
+token=oauth.OAuthToken(actual_token, token_secret); 
+rq=oauth.OAuthRequest.from_consumer_and_token(consumer, token=token, 
+  http_method=method, http_url=url,
+  parameters={'oauth_nonce':nonce,'oauth_timestamp':timestamp});
+rq.sign_request(signature_function(),consumer, token); 
+headers = rq.to_header()
+parts = [x.strip() for x in headers["Authorization"].split(",") if x.find("=") != -1]
+smallparts = dict([x.split("=") for x in parts])
+print "Signature from Python:    ", smallparts["oauth_signature"]
+

=== added file 'tests/view_test.sh'
--- tests/view_test.sh	1970-01-01 00:00:00 +0000
+++ tests/view_test.sh	2010-01-22 19:18:13 +0000
@@ -0,0 +1,16 @@
+#!/bin/bash
+
+
+
+PORT=$(dbus-send --session --dest=org.desktopcouch.CouchDB --print-reply --type=method_call / org.desktopcouch.CouchDB.getPort 2>/dev/null | grep int32 | awk '{print $2}')
+
+TOKENS=$(python -c "from desktopcouch import local_files; tokens = local_files.get_oauth_tokens(); print ':'.join([tokens['consumer_key'], tokens['consumer_secret'], tokens['token'], tokens['token_secret']])" 2>/dev/null)
+
+TESTDIR=$(dirname $0)
+
+LD_LIBRARY_PATH=/usr/lib/xulrunner-1.9.1.3 /usr/lib/xulrunner-1.9.1.3/xpcshell -f $TESTDIR/../content/oauth.js  -f $TESTDIR/../content/sha1.js -f $TESTDIR/../content/couch.js \
+     $TESTDIR/test_get_views.js $PORT:$TOKENS bookmarks
+
+#python $TESTDIR/test_oauth.py $CONSUMER_KEY $CONSUMER_SECRET $TOKEN $TOKEN_SECRET \
+#    http://$ACTION $METHOD $NONCE $TIMESTAMP $SIGNATURE_METHOD
+