Merge lp:~jelmer/launchpad/update-bzr-hg into lp:launchpad

[Jelmer Vernooij]

Use a newer version of bzr-hg.

This updates bzr-hg to a newer upstream revision, one that makes it use directories for caches rather than individual files. I've made matching changes to lp.codehosting to cope with the fact that the bzr-hg cache is now stored differently.

These changes make it possible to run the lp codehosting tests with success when python-tdb is installed. It should also make it easier to install python-tdb on the production machines later (there are some advantages to using python-tdb; it's quicker and allows multiple concurrent writers).

[Steve Kowalik (stevenk)]

Hi!

Your changes look good, but I have a few comments:

* Have we updated to the new bzr-hg that can take advantage of the tarball cache?
* You've removed the global db_file variable, but you have the magic string 'hg-cache.tar.gz' in the code twice.
* From my reading of the code, it looks like it is going to migrate new pushed branches to the tarball cache, is that the real intent?

review: Needs Information (code*)

[Jelmer Vernooij (jelmer)]

Hi Steve,

Thanks for the review. :-)

> Your changes look good, but I have a few comments:
>
> * Have we updated to the new bzr-hg that can take advantage of the tarball
> cache?
That's what this branch was doing (see 11347), but that change was reverted later when I remerged devel. I've now reintroduced it.

> * You've removed the global db_file variable, but you have the magic string
> 'hg-cache.tar.gz' in the code twice.
It's not really a magic filename like 'hg-v2.db', hg-cache.tar.gz is just the name of the temporary filename that's used to generate the tarball of the hg cache that's uploaded - similar to git-cache.tar.gz in the bzr-git worker class.

> * From my reading of the code, it looks like it is going to migrate new pushed
> branches to the tarball cache, is that the real intent?
Yes, that's correct.

[Tim Penhey (thumper)]

Jelmer, it still doesn't look like the bzr-hg is being updated. Should it be?

[Jelmer Vernooij (jelmer)]

Argh, Jelmer fail.

I fixed the branch but forgot to push it. Updated version should have bzr-hg updated to a newer version.

[Steve Kowalik (stevenk)]

This looks good to me.

review: Approve (code*)

[Tim Penhey (thumper)]

review: Approve

=== modified file 'lib/lp/codehosting/codeimport/tests/servers.py'
--- lib/lp/codehosting/codeimport/tests/servers.py	2010-08-20 20:31:18 +0000
+++ lib/lp/codehosting/codeimport/tests/servers.py	2010-09-02 12:12:50 +0000
@@ -233,5 +233,5 @@
                 f.write(contents)
             finally:
                 f.close()
-            repo.add([filename])
+            repo[None].add([filename])
         repo.commit(text='<The commit message>', user='jane Foo <[e-mail]>')

=== modified file 'lib/lp/codehosting/codeimport/worker.py'
--- lib/lp/codehosting/codeimport/worker.py	2010-08-20 20:31:18 +0000
+++ lib/lp/codehosting/codeimport/worker.py	2010-09-02 12:12:50 +0000
@@ -665,8 +665,6 @@
     The only behaviour we add is preserving the id-sha map between runs.
     """
 
-    db_file = 'hg-v2.db'
-
     @property
     def format_classes(self):
         """See `PullingImportWorker.opening_format`."""
@@ -677,26 +675,39 @@
     def getBazaarBranch(self):
         """See `ImportWorker.getBazaarBranch`.
 
-        In addition to the superclass' behaviour, we retrieve the 'hg-v2.db'
-        map from the import data store and put it where bzr-hg will find
-        it in the Bazaar tree, that is at '.bzr/repository/hg-v2.db'.
+        In addition to the superclass' behaviour, we retrieve the bzr-hg's
+        caches, both legacy and current and put them where bzr-hg will find
+        them in the Bazaar tree, that is at '.bzr/repository/hg-v2.db' and
+        '.bzr/repository/hg'.
         """
         branch = PullingImportWorker.getBazaarBranch(self)
+        # Fetch the legacy cache from the store, if present.
         self.import_data_store.fetch(
-            self.db_file, branch.repository._transport)
+            'hg-v2.db', branch.repository._transport)
+        # The cache dir from newer bzr-hgs is stored as a tarball.
+        local_name = 'hg-cache.tar.gz'
+        if self.import_data_store.fetch(local_name):
+            repo_transport = branch.repository._transport
+            repo_transport.mkdir('hg')
+            hg_db_dir = os.path.join(
+                local_path_from_url(repo_transport.base), 'hg')
+            extract_tarball(local_name, hg_db_dir)
         return branch
 
     def pushBazaarBranch(self, bazaar_branch):
         """See `ImportWorker.pushBazaarBranch`.
 
-        In addition to the superclass' behaviour, we store the 'hg-v2.db'
-        shamap that bzr-hg will have created at .bzr/repository/hg-v2.db into
+        In addition to the superclass' behaviour, we store the hg cache
+        that bzr-hg will have created at .bzr/repository/hg into
         the import data store.
         """
         non_trivial = PullingImportWorker.pushBazaarBranch(
             self, bazaar_branch)
-        self.import_data_store.put(
-            self.db_file, bazaar_branch.repository._transport)
+        repo_base = bazaar_branch.repository._transport.base
+        hg_db_dir = os.path.join(local_path_from_url(repo_base), 'hg')
+        local_name = 'hg-cache.tar.gz'
+        create_tarball(hg_db_dir, local_name)
+        self.import_data_store.put(local_name)
         return non_trivial
 
 

=== modified file 'utilities/sourcedeps.conf'
--- utilities/sourcedeps.conf	2010-08-30 00:51:55 +0000
+++ utilities/sourcedeps.conf	2010-09-02 12:12:50 +0000
@@ -1,6 +1,6 @@
 bzr-builder lp:~launchpad-pqm/bzr-builder/trunk;revno=65
 bzr-git lp:~launchpad-pqm/bzr-git/devel;revno=258
-bzr-hg lp:~launchpad-pqm/bzr-hg/devel;revno=282
+bzr-hg lp:~launchpad-pqm/bzr-hg/devel;revno=283
 bzr-loom lp:~launchpad-pqm/bzr-loom/trunk;revno=48
 bzr-svn lp:~launchpad-pqm/bzr-svn/devel;revno=2710
 cscvs lp:~launchpad-pqm/launchpad-cscvs/devel;revno=432

=== modified file 'versions.cfg'
--- versions.cfg	2010-08-30 16:01:56 +0000
+++ versions.cfg	2010-09-02 12:12:50 +0000
@@ -41,7 +41,7 @@
 martian = 0.11
 mechanize = 0.1.11
 meliae = 0.2.0.final.0
-mercurial = 1.3.1
+mercurial = 1.6.2
 mocker = 0.10.1
 mozrunner = 1.3.4
 numpy = 1.3.0