Merge lp:~jelmer/launchpad/bzr-code-imports into lp:launchpad

Proposed by Jelmer Vernooij
Status: Superseded
Proposed branch: lp:~jelmer/launchpad/bzr-code-imports
Merge into: lp:launchpad
Prerequisite: lp:~jelmer/launchpad/bzr-2.4b4
Diff against target: 2835 lines (+1213/-739)
26 files modified
lib/canonical/config/schema-lazr.conf (+5/-0)
lib/lp/code/bzr.py (+10/-6)
lib/lp/code/mail/codeimport.py (+3/-1)
lib/lp/code/model/codeimport.py (+8/-3)
lib/lp/code/model/codeimportevent.py (+2/-1)
lib/lp/code/model/tests/test_branchjob.py (+1/-1)
lib/lp/code/model/tests/test_codeimport.py (+44/-0)
lib/lp/codehosting/__init__.py (+10/-12)
lib/lp/codehosting/codeimport/tests/servers.py (+176/-23)
lib/lp/codehosting/codeimport/tests/test_worker.py (+75/-11)
lib/lp/codehosting/codeimport/tests/test_workermonitor.py (+26/-4)
lib/lp/codehosting/codeimport/worker.py (+97/-6)
lib/lp/codehosting/puller/tests/__init__.py (+6/-62)
lib/lp/codehosting/puller/tests/test_errors.py (+3/-5)
lib/lp/codehosting/puller/tests/test_worker.py (+18/-214)
lib/lp/codehosting/puller/tests/test_worker_formats.py (+7/-3)
lib/lp/codehosting/puller/worker.py (+239/-104)
lib/lp/codehosting/safe_open.py (+230/-0)
lib/lp/codehosting/tests/test_safe_open.py (+223/-0)
lib/lp/codehosting/vfs/__init__.py (+0/-2)
lib/lp/codehosting/vfs/branchfs.py (+0/-257)
lib/lp/registry/browser/productseries.py (+9/-17)
lib/lp/testing/factory.py (+9/-3)
lib/lp/translations/scripts/translations_to_branch.py (+4/-1)
scripts/code-import-worker.py (+7/-2)
versions.cfg (+1/-1)
To merge this branch: bzr merge lp:~jelmer/launchpad/bzr-code-imports
Reviewer Review Type Date Requested Status
Gavin Panella (community) Abstain
Robert Collins (community) Needs Information
Michael Hudson-Doyle Pending
Review via email: mp+68232@code.launchpad.net

This proposal supersedes a proposal from 2011-07-18.

This proposal has been superseded by a proposal from 2011-08-07.

Description of the change

Add support for importing code from Bazaar branches.

At the moment mirrors of remote Bazaar branches are created with completely different infrastructure as the code imports. This is confusing for users (bug 611837) and duplicates a lot of code. Several features are only available for code imports (bug 362622, bug 193607, bug 193607, bug 371484) and vice versa. Having shared infrastructure would also make it easier to fix several open bugs that affect both code imports and code mirrors (bug 519159, bug 136939)

Code imports are a bit heavier than mirrors at the moment, as they run on a separate machine and require an extra copy of the branch that is imported.

This branch only adds backend support for Bazaar branches, it does not yet add a UI which can add code imports of this kind nor does it migrate any of the existing code mirrors.

To post a comment you must log in.
Revision history for this message
Gavin Panella (allenap) wrote : Posted in a previous version of this proposal

I don't know the ins and outs of bzrlib or codehosting - for that I
guess that's why you've asked Michael to review - but the rest looks
good.

[1]

+ def test_partial(self):
+ # Skip tests of partial tests, as they are disabled for native imports
+ # at the moment.
+ return

You could use TestCase.skip() here:

    def test_partial(self):
        self.skip("Disabled for native imports at the moment.")

review: Approve
Revision history for this message
Jelmer Vernooij (jelmer) wrote : Posted in a previous version of this proposal

> I don't know the ins and outs of bzrlib or codehosting - for that I
> guess that's why you've asked Michael to review - but the rest looks
> good.
Thanks :)

I'm pretty confident about this code change (especially since nothing actually triggers the new code yet), but I'd like to get some feedback from more people (Michael, Jono?, Rob?) on to confirm this is the right direction to go in.

> + def test_partial(self):
> + # Skip tests of partial tests, as they are disabled for native
> imports
> + # at the moment.
> + return
>
> You could use TestCase.skip() here:
>
> def test_partial(self):
> self.skip("Disabled for native imports at the moment.")
I looked for things that raised SkipTest or TestSkipped and couldn't find any. I didn't know about TestCase.skip - thanks, fixed.

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote : Posted in a previous version of this proposal

Some random comments: it would have been nice to see the things you had to do wrt the bzr upgrade

The fact that you put this line in suggests that the tests aren't very well isolated:

        branch.get_config().set_user_option("create_signatures", "never")

Does something need to inherit from bzrlib's TestCase? It's all a complete tangle though.

Is simply prohibiting branch references the right thing to do? The branch puller goes to some lengths to support them safely -- being able to dump all that code would surely be very nice. Relatedly, and on thinking about it I think this is a bit more serious, I think you might need to be careful about stacking -- it's contrived but you might be able to construct a branch that was stacked on some DC-internal branch and have that be imported so you can grab it.

In terms of overall direction, I'm all for removing duplication. I think the UI will require some effort (e.g. do we want to count bzr mirrors as "imported branches" on the code.launchpad.net frontpage?) but engineering wise, this looks fine, modulo the above comments.

review: Approve
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote : Posted in a previous version of this proposal

"Some random comments: it would have been nice to see the things you had to do wrt the bzr upgrade " ... in a separate branch, I meant to say.

Revision history for this message
Jelmer Vernooij (jelmer) wrote : Posted in a previous version of this proposal

On 06/24/2011 05:35 AM, Michael Hudson-Doyle wrote:
> Review: Approve
> Some random comments: it would have been nice to see the things you had to do wrt the bzr upgrade
This branch has two prerequisite branches, but I can only set one in
Launchpad. Hopefully the diff will get smaller when the update to the
newer bzr lands on lp:launchpad...
>
> The fact that you put this line in suggests that the tests aren't very well isolated:
>
> branch.get_config().set_user_option("create_signatures", "never")
>
> Does something need to inherit from bzrlib's TestCase? It's all a complete tangle though.
Perhaps; bzr's TestCase does a lot though, and I'm kindof worried that
mixing it in will break other things. It should do more than just this
ad-hoc override though. Perhaps reset the global bzr configuration in setUp?

>
> Is simply prohibiting branch references the right thing to do? The branch puller goes to some lengths to support them safely -- being able to dump all that code would surely be very nice. Relatedly, and on thinking about it I think this is a bit more serious, I think you might need to be careful about stacking -- it's contrived but you might be able to construct a branch that was stacked on some DC-internal branch and have that be imported so you can grab it.
Stacking is a very good point, and one that I had not considered -
thanks. I should probably also have another, closer, look at the branch
puller to see what it does and why.

For branch references, the easiest thing to do for the moment seemed to
be to just refuse to mirror them. If code mirrors support branch
references at the moment, we should keep that support.

>
> In terms of overall direction, I'm all for removing duplication. I think the UI will require some effort (e.g. do we want to count bzr mirrors as "imported branches" on the code.launchpad.net frontpage?) but engineering wise, this looks fine, modulo the above comments.
Thanks for having a look at this. I'm only just getting into this code
and am not very familiar with it yet.

Cheers,

Jelmer

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote : Posted in a previous version of this proposal

On Fri, 24 Jun 2011 19:55:45 +0200, Jelmer Vernooij <email address hidden> wrote:
> On 06/24/2011 05:35 AM, Michael Hudson-Doyle wrote:
> > Review: Approve
> > Some random comments: it would have been nice to see the things you had to do wrt the bzr upgrade
> This branch has two prerequisite branches, but I can only set one in
> Launchpad. Hopefully the diff will get smaller when the update to the
> newer bzr lands on lp:launchpad...

Ah! I guess you could have created a branch with both of the
prerequisites merged in, but that's getting pretty tedious...

> >
> > The fact that you put this line in suggests that the tests aren't very well isolated:
> >
> > branch.get_config().set_user_option("create_signatures", "never")
> >
> > Does something need to inherit from bzrlib's TestCase? It's all a complete tangle though.
> Perhaps; bzr's TestCase does a lot though, and I'm kindof worried that
> mixing it in will break other things.

I'd be surprised if it broke other stuff on past experience, but you
might be right.

> It should do more than just this ad-hoc override though. Perhaps reset
> the global bzr configuration in setUp?

I guess what you want is a test fixture that just does the environment
isolation in bzrlib you can reuse here... but yes, doing something more
generic than just clearing create_signatures would be better, I think.

> >
> > Is simply prohibiting branch references the right thing to do? The branch puller goes to some lengths to support them safely -- being able to dump all that code would surely be very nice. Relatedly, and on thinking about it I think this is a bit more serious, I think you might need to be careful about stacking -- it's contrived but you might be able to construct a branch that was stacked on some DC-internal branch and have that be imported so you can grab it.
> Stacking is a very good point, and one that I had not considered -
> thanks. I should probably also have another, closer, look at the branch
> puller to see what it does and why.

For reasons I should probably be ashamed of, there seem to be two
implementations of 'safe opening' in Launchpad; one is around
lp.codehosting.bzrutils.safe_open and the other around
lp.codehosting.puller.worker.BranchMirrorer.

(looking at worker.py makes me realize how much code we can delete once
this branch is done, never mind how much can be deleted when the puller
is gone completely).

> For branch references, the easiest thing to do for the moment seemed to
> be to just refuse to mirror them. If code mirrors support branch
> references at the moment, we should keep that support.

Yeah, they do.

> >
> > In terms of overall direction, I'm all for removing duplication. I think the UI will require some effort (e.g. do we want to count bzr mirrors as "imported branches" on the code.launchpad.net frontpage?) but engineering wise, this looks fine, modulo the above comments.
> Thanks for having a look at this. I'm only just getting into this code
> and am not very familiar with it yet.

You seem to be doing fine :)

Cheers,
mwh

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

I don't see anything to do with stacking safety and/or supporting branch references in here yet -- is this really ready for review again?

Revision history for this message
Jelmer Vernooij (jelmer) wrote :

> I don't see anything to do with stacking safety and/or supporting branch
> references in here yet -- is this really ready for review again?
No, it indeed isn't - I was merely trying to generate a proper diff. Sorry.

Revision history for this message
Robert Collins (lifeless) wrote :

Does this permit imports from lp hosted branches? if so, thats likely to permit bypassing of private branch ACLs - can you check that that isn't the please?

review: Needs Information
Revision history for this message
Gavin Panella (allenap) :
review: Abstain
Revision history for this message
Jelmer Vernooij (jelmer) wrote :

On Sun, 2011-08-07 at 22:41 +0000, Robert Collins wrote:
> Review: Needs Information
> Does this permit imports from lp hosted branches? if so, thats likely to permit bypassing of private branch ACLs - can you check that that isn't the please?
It does not allow imports from lp hosted branches, importing of branch
references to launchpad branches or even importing of branches stacked
on Launchpad branches.

It also limits imports to a specific (whitelist) set of schemes, which
excludes bzr+ssh://, sftp:// and file://.

This is the same policy as is currently in place for code mirrors at the
moment.

Cheers,

Jelmer

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/canonical/config/schema-lazr.conf'
--- lib/canonical/config/schema-lazr.conf 2011-07-27 20:14:40 +0000
+++ lib/canonical/config/schema-lazr.conf 2011-08-07 20:42:25 +0000
@@ -488,6 +488,11 @@
488# datatype: integer488# datatype: integer
489default_interval_cvs: 43200489default_interval_cvs: 43200
490490
491# The default value of the update interval of a code import from
492# Bazaar, in seconds.
493# datatype: integer
494default_interval_bzr: 21600
495
491# Where the tarballs of foreign branches are uploaded for storage.496# Where the tarballs of foreign branches are uploaded for storage.
492# datatype: string497# datatype: string
493foreign_tree_store: sftp://hoover@escudero/srv/importd/sources/498foreign_tree_store: sftp://hoover@escudero/srv/importd/sources/
494499
=== modified file 'lib/lp/code/bzr.py'
--- lib/lp/code/bzr.py 2011-06-02 19:27:36 +0000
+++ lib/lp/code/bzr.py 2011-08-07 20:42:25 +0000
@@ -20,34 +20,38 @@
2020
21from bzrlib.branch import (21from bzrlib.branch import (
22 BranchReferenceFormat,22 BranchReferenceFormat,
23 BzrBranchFormat4,
24 BzrBranchFormat5,23 BzrBranchFormat5,
25 BzrBranchFormat6,24 BzrBranchFormat6,
26 BzrBranchFormat7,25 BzrBranchFormat7,
27 )26 )
28from bzrlib.bzrdir import (27from bzrlib.bzrdir import (
29 BzrDirFormat4,
30 BzrDirFormat5,
31 BzrDirFormat6,
32 BzrDirMetaFormat1,28 BzrDirMetaFormat1,
33 )29 )
34from bzrlib.plugins.loom.branch import (30from bzrlib.plugins.loom.branch import (
35 BzrBranchLoomFormat1,31 BzrBranchLoomFormat1,
36 BzrBranchLoomFormat6,32 BzrBranchLoomFormat6,
37 )33 )
34from bzrlib.plugins.weave_fmt.branch import (
35 BzrBranchFormat4,
36 )
37from bzrlib.plugins.weave_fmt.bzrdir import (
38 BzrDirFormat4,
39 BzrDirFormat5,
40 BzrDirFormat6,
41 )
38from bzrlib.repofmt.groupcompress_repo import RepositoryFormat2a42from bzrlib.repofmt.groupcompress_repo import RepositoryFormat2a
39from bzrlib.repofmt.knitrepo import (43from bzrlib.repofmt.knitrepo import (
40 RepositoryFormatKnit1,44 RepositoryFormatKnit1,
41 RepositoryFormatKnit3,45 RepositoryFormatKnit3,
42 RepositoryFormatKnit4,46 RepositoryFormatKnit4,
43 )47 )
44from bzrlib.repofmt.pack_repo import (48from bzrlib.repofmt.knitpack_repo import (
45 RepositoryFormatKnitPack1,49 RepositoryFormatKnitPack1,
46 RepositoryFormatKnitPack3,50 RepositoryFormatKnitPack3,
47 RepositoryFormatKnitPack4,51 RepositoryFormatKnitPack4,
48 RepositoryFormatKnitPack5,52 RepositoryFormatKnitPack5,
49 )53 )
50from bzrlib.repofmt.weaverepo import (54from bzrlib.plugins.weave_fmt.repository import (
51 RepositoryFormat4,55 RepositoryFormat4,
52 RepositoryFormat5,56 RepositoryFormat5,
53 RepositoryFormat6,57 RepositoryFormat6,
5458
=== modified file 'lib/lp/code/mail/codeimport.py'
--- lib/lp/code/mail/codeimport.py 2011-05-27 21:12:25 +0000
+++ lib/lp/code/mail/codeimport.py 2011-08-07 20:42:25 +0000
@@ -51,6 +51,7 @@
51 RevisionControlSystems.BZR_SVN: 'subversion',51 RevisionControlSystems.BZR_SVN: 'subversion',
52 RevisionControlSystems.GIT: 'git',52 RevisionControlSystems.GIT: 'git',
53 RevisionControlSystems.HG: 'mercurial',53 RevisionControlSystems.HG: 'mercurial',
54 RevisionControlSystems.BZR: 'bazaar',
54 }55 }
55 body = get_email_template('new-code-import.txt') % {56 body = get_email_template('new-code-import.txt') % {
56 'person': code_import.registrant.displayname,57 'person': code_import.registrant.displayname,
@@ -123,7 +124,8 @@
123 elif code_import.rcs_type in (RevisionControlSystems.SVN,124 elif code_import.rcs_type in (RevisionControlSystems.SVN,
124 RevisionControlSystems.BZR_SVN,125 RevisionControlSystems.BZR_SVN,
125 RevisionControlSystems.GIT,126 RevisionControlSystems.GIT,
126 RevisionControlSystems.HG):127 RevisionControlSystems.HG,
128 RevisionControlSystems.BZR):
127 if CodeImportEventDataType.OLD_URL in event_data:129 if CodeImportEventDataType.OLD_URL in event_data:
128 old_url = event_data[CodeImportEventDataType.OLD_URL]130 old_url = event_data[CodeImportEventDataType.OLD_URL]
129 body.append(131 body.append(
130132
=== modified file 'lib/lp/code/model/codeimport.py'
--- lib/lp/code/model/codeimport.py 2011-04-27 01:42:46 +0000
+++ lib/lp/code/model/codeimport.py 2011-08-07 20:42:25 +0000
@@ -116,6 +116,8 @@
116 config.codeimport.default_interval_git,116 config.codeimport.default_interval_git,
117 RevisionControlSystems.HG:117 RevisionControlSystems.HG:
118 config.codeimport.default_interval_hg,118 config.codeimport.default_interval_hg,
119 RevisionControlSystems.BZR:
120 config.codeimport.default_interval_bzr,
119 }121 }
120 seconds = default_interval_dict[self.rcs_type]122 seconds = default_interval_dict[self.rcs_type]
121 return timedelta(seconds=seconds)123 return timedelta(seconds=seconds)
@@ -133,7 +135,8 @@
133 RevisionControlSystems.SVN,135 RevisionControlSystems.SVN,
134 RevisionControlSystems.GIT,136 RevisionControlSystems.GIT,
135 RevisionControlSystems.BZR_SVN,137 RevisionControlSystems.BZR_SVN,
136 RevisionControlSystems.HG):138 RevisionControlSystems.HG,
139 RevisionControlSystems.BZR):
137 return self.url140 return self.url
138 else:141 else:
139 raise AssertionError(142 raise AssertionError(
@@ -252,7 +255,8 @@
252 elif rcs_type in (RevisionControlSystems.SVN,255 elif rcs_type in (RevisionControlSystems.SVN,
253 RevisionControlSystems.BZR_SVN,256 RevisionControlSystems.BZR_SVN,
254 RevisionControlSystems.GIT,257 RevisionControlSystems.GIT,
255 RevisionControlSystems.HG):258 RevisionControlSystems.HG,
259 RevisionControlSystems.BZR):
256 assert cvs_root is None and cvs_module is None260 assert cvs_root is None and cvs_module is None
257 assert url is not None261 assert url is not None
258 else:262 else:
@@ -262,7 +266,8 @@
262 if review_status is None:266 if review_status is None:
263 # Auto approve git and hg imports.267 # Auto approve git and hg imports.
264 if rcs_type in (268 if rcs_type in (
265 RevisionControlSystems.GIT, RevisionControlSystems.HG):269 RevisionControlSystems.GIT, RevisionControlSystems.HG,
270 RevisionControlSystems.BZR):
266 review_status = CodeImportReviewStatus.REVIEWED271 review_status = CodeImportReviewStatus.REVIEWED
267 else:272 else:
268 review_status = CodeImportReviewStatus.NEW273 review_status = CodeImportReviewStatus.NEW
269274
=== modified file 'lib/lp/code/model/codeimportevent.py'
--- lib/lp/code/model/codeimportevent.py 2010-10-17 22:51:50 +0000
+++ lib/lp/code/model/codeimportevent.py 2011-08-07 20:42:25 +0000
@@ -269,7 +269,8 @@
269 if code_import.rcs_type in (RevisionControlSystems.SVN,269 if code_import.rcs_type in (RevisionControlSystems.SVN,
270 RevisionControlSystems.BZR_SVN,270 RevisionControlSystems.BZR_SVN,
271 RevisionControlSystems.GIT,271 RevisionControlSystems.GIT,
272 RevisionControlSystems.HG):272 RevisionControlSystems.HG,
273 RevisionControlSystems.BZR):
273 yield 'URL', code_import.url274 yield 'URL', code_import.url
274 elif code_import.rcs_type == RevisionControlSystems.CVS:275 elif code_import.rcs_type == RevisionControlSystems.CVS:
275 yield 'CVS_ROOT', code_import.cvs_root276 yield 'CVS_ROOT', code_import.cvs_root
276277
=== modified file 'lib/lp/code/model/tests/test_branchjob.py'
--- lib/lp/code/model/tests/test_branchjob.py 2011-05-23 11:06:50 +0000
+++ lib/lp/code/model/tests/test_branchjob.py 2011-08-07 20:42:25 +0000
@@ -17,7 +17,7 @@
17 BzrBranchFormat7,17 BzrBranchFormat7,
18 )18 )
19from bzrlib.bzrdir import BzrDirMetaFormat119from bzrlib.bzrdir import BzrDirMetaFormat1
20from bzrlib.repofmt.pack_repo import RepositoryFormatKnitPack620from bzrlib.repofmt.knitpack_repo import RepositoryFormatKnitPack6
21from bzrlib.revision import NULL_REVISION21from bzrlib.revision import NULL_REVISION
22from bzrlib.transport import get_transport22from bzrlib.transport import get_transport
23import pytz23import pytz
2424
=== modified file 'lib/lp/code/model/tests/test_codeimport.py'
--- lib/lp/code/model/tests/test_codeimport.py 2011-05-27 21:12:25 +0000
+++ lib/lp/code/model/tests/test_codeimport.py 2011-08-07 20:42:25 +0000
@@ -71,6 +71,20 @@
71 # No job is created for the import.71 # No job is created for the import.
72 self.assertIs(None, code_import.import_job)72 self.assertIs(None, code_import.import_job)
7373
74 def test_new_svn_import_svn_scheme(self):
75 """A subversion import can use the svn:// scheme."""
76 code_import = CodeImportSet().new(
77 registrant=self.factory.makePerson(),
78 target=IBranchTarget(self.factory.makeProduct()),
79 branch_name='imported',
80 rcs_type=RevisionControlSystems.SVN,
81 url=self.factory.getUniqueURL(scheme="svn"))
82 self.assertEqual(
83 CodeImportReviewStatus.NEW,
84 code_import.review_status)
85 # No job is created for the import.
86 self.assertIs(None, code_import.import_job)
87
74 def test_reviewed_svn_import(self):88 def test_reviewed_svn_import(self):
75 """A specific review status can be set for a new import."""89 """A specific review status can be set for a new import."""
76 code_import = CodeImportSet().new(90 code_import = CodeImportSet().new(
@@ -117,6 +131,21 @@
117 # A job is created for the import.131 # A job is created for the import.
118 self.assertIsNot(None, code_import.import_job)132 self.assertIsNot(None, code_import.import_job)
119133
134 def test_git_import_git_scheme(self):
135 """A git import can have a git:// style URL."""
136 code_import = CodeImportSet().new(
137 registrant=self.factory.makePerson(),
138 target=IBranchTarget(self.factory.makeProduct()),
139 branch_name='imported',
140 rcs_type=RevisionControlSystems.GIT,
141 url=self.factory.getUniqueURL(scheme="git"),
142 review_status=None)
143 self.assertEqual(
144 CodeImportReviewStatus.REVIEWED,
145 code_import.review_status)
146 # A job is created for the import.
147 self.assertIsNot(None, code_import.import_job)
148
120 def test_git_import_reviewed(self):149 def test_git_import_reviewed(self):
121 """A new git import is always reviewed by default."""150 """A new git import is always reviewed by default."""
122 code_import = CodeImportSet().new(151 code_import = CodeImportSet().new(
@@ -147,6 +176,21 @@
147 # A job is created for the import.176 # A job is created for the import.
148 self.assertIsNot(None, code_import.import_job)177 self.assertIsNot(None, code_import.import_job)
149178
179 def test_bzr_import_reviewed(self):
180 """A new bzr import is always reviewed by default."""
181 code_import = CodeImportSet().new(
182 registrant=self.factory.makePerson(),
183 target=IBranchTarget(self.factory.makeProduct()),
184 branch_name='mirrored',
185 rcs_type=RevisionControlSystems.BZR,
186 url=self.factory.getUniqueURL(),
187 review_status=None)
188 self.assertEqual(
189 CodeImportReviewStatus.REVIEWED,
190 code_import.review_status)
191 # A job is created for the import.
192 self.assertIsNot(None, code_import.import_job)
193
150 def test_junk_code_import_rejected(self):194 def test_junk_code_import_rejected(self):
151 """You are not allowed to create code imports targetting +junk."""195 """You are not allowed to create code imports targetting +junk."""
152 registrant = self.factory.makePerson()196 registrant = self.factory.makePerson()
153197
=== modified file 'lib/lp/codehosting/__init__.py'
--- lib/lp/codehosting/__init__.py 2011-03-30 15:16:35 +0000
+++ lib/lp/codehosting/__init__.py 2011-08-07 20:42:25 +0000
@@ -70,15 +70,13 @@
70 __import__("bzrlib.plugins.%s" % plugin_name)70 __import__("bzrlib.plugins.%s" % plugin_name)
7171
7272
73def remove_hook(self, hook):73def load_bundled_plugin(plugin_name):
74 """Remove the hook from the HookPoint"""74 """Load a plugin bundled with Bazaar."""
75 self._callbacks.remove(hook)75 from bzrlib.plugin import get_core_plugin_path
76 for name, value in self._callback_names.iteritems():76 from bzrlib import plugins
77 if value is hook:77 if get_core_plugin_path() not in plugins.__path__:
78 del self._callback_names[name]78 plugins.__path__.append(get_core_plugin_path())
7979 __import__("bzrlib.plugins.%s" % plugin_name)
8080
81# XXX: JonathanLange 2011-03-30 bug=301472: Monkeypatch: Branch.hooks is a81
82# list in bzr 1.13, so it supports remove. It is a HookPoint in bzr 1.14, so82load_bundled_plugin("weave_fmt")
83# add HookPoint.remove.
84hooks.HookPoint.remove = remove_hook
8583
=== modified file 'lib/lp/codehosting/codeimport/tests/servers.py'
--- lib/lp/codehosting/codeimport/tests/servers.py 2011-06-02 10:48:54 +0000
+++ lib/lp/codehosting/codeimport/tests/servers.py 2011-08-07 20:42:25 +0000
@@ -4,6 +4,7 @@
4"""Server classes that know how to create various kinds of foreign archive."""4"""Server classes that know how to create various kinds of foreign archive."""
55
6__all__ = [6__all__ = [
7 'BzrServer',
7 'CVSServer',8 'CVSServer',
8 'GitServer',9 'GitServer',
9 'MercurialServer',10 'MercurialServer',
@@ -13,6 +14,7 @@
13__metaclass__ = type14__metaclass__ = type
1415
15from cStringIO import StringIO16from cStringIO import StringIO
17import errno
16import os18import os
17import shutil19import shutil
18import signal20import signal
@@ -20,8 +22,16 @@
20import subprocess22import subprocess
21import tempfile23import tempfile
22import time24import time
25import threading
2326
27from bzrlib.branch import Branch
28from bzrlib.branchbuilder import BranchBuilder
29from bzrlib.bzrdir import BzrDir
24from bzrlib.tests.treeshape import build_tree_contents30from bzrlib.tests.treeshape import build_tree_contents
31from bzrlib.tests.test_server import (
32 ReadonlySmartTCPServer_for_testing,
33 TestServer,
34 )
25from bzrlib.transport import Server35from bzrlib.transport import Server
26from bzrlib.urlutils import (36from bzrlib.urlutils import (
27 escape,37 escape,
@@ -31,6 +41,17 @@
31import dulwich.index41import dulwich.index
32from dulwich.objects import Blob42from dulwich.objects import Blob
33from dulwich.repo import Repo as GitRepo43from dulwich.repo import Repo as GitRepo
44from dulwich.server import (
45 DictBackend,
46 TCPGitServer,
47 )
48from mercurial.ui import (
49 ui as hg_ui,
50 )
51from mercurial.hgweb import (
52 hgweb,
53 server as hgweb_server,
54 )
34import subvertpy.ra55import subvertpy.ra
35import svn_oo56import svn_oo
3657
@@ -107,8 +128,8 @@
107 for i in range(10):128 for i in range(10):
108 try:129 try:
109 ra = self._get_ra(self.get_url())130 ra = self._get_ra(self.get_url())
110 except subvertpy.SubversionException, e:131 except OSError, e:
111 if 'Connection refused' in str(e):132 if e.errno == errno.ECONNREFUSED:
112 time.sleep(delay)133 time.sleep(delay)
113 delay *= 1.5134 delay *= 1.5
114 continue135 continue
@@ -204,45 +225,177 @@
204 self._repository = self.createRepository(self._repository_path)225 self._repository = self.createRepository(self._repository_path)
205226
206227
228class TCPGitServerThread(threading.Thread):
229 """TCP Git server that runs in a separate thread."""
230
231 def __init__(self, backend, address, port=None):
232 super(TCPGitServerThread, self).__init__()
233 self.setName("TCP Git server on %s:%s" % (address, port))
234 self.server = TCPGitServer(backend, address, port)
235
236 def run(self):
237 self.server.serve_forever()
238
239 def get_address(self):
240 return self.server.server_address
241
242 def stop(self):
243 self.server.shutdown()
244
245
207class GitServer(Server):246class GitServer(Server):
208247
209 def __init__(self, repo_url):248 def __init__(self, repository_path, use_server=False):
210 super(GitServer, self).__init__()249 super(GitServer, self).__init__()
211 self.repo_url = repo_url250 self.repository_path = repository_path
251 self._use_server = use_server
252
253 def get_url(self):
254 """Return a URL to the Git repository."""
255 if self._use_server:
256 return 'git://%s:%d/' % self._server.get_address()
257 else:
258 return local_path_to_url(self.repository_path)
259
260 def createRepository(self, path):
261 GitRepo.init(path)
262
263 def start_server(self):
264 super(GitServer, self).start_server()
265 self.createRepository(self.repository_path)
266 if self._use_server:
267 repo = GitRepo(self.repository_path)
268 self._server = TCPGitServerThread(
269 DictBackend({"/": repo}), "localhost", 0)
270 self._server.start()
271
272 def stop_server(self):
273 super(GitServer, self).stop_server()
274 if self._use_server:
275 self._server.stop()
212276
213 def makeRepo(self, tree_contents):277 def makeRepo(self, tree_contents):
214 wd = os.getcwd()278 repo = GitRepo(self.repository_path)
215 try:279 blobs = [
216 os.chdir(self.repo_url)280 (Blob.from_string(contents), filename) for (filename, contents)
217 repo = GitRepo.init(".")281 in tree_contents]
218 blobs = [282 repo.object_store.add_objects(blobs)
219 (Blob.from_string(contents), filename) for (filename, contents)283 root_id = dulwich.index.commit_tree(repo.object_store, [
220 in tree_contents]284 (filename, b.id, stat.S_IFREG | 0644)
221 repo.object_store.add_objects(blobs)285 for (b, filename) in blobs])
222 root_id = dulwich.index.commit_tree(repo.object_store, [286 repo.do_commit(committer='Joe Foo <joe@foo.com>',
223 (filename, b.id, stat.S_IFREG | 0644)287 message=u'<The commit message>', tree=root_id)
224 for (b, filename) in blobs])288
225 repo.do_commit(committer='Joe Foo <joe@foo.com>',289
226 message=u'<The commit message>', tree=root_id)290class MercurialServerThread(threading.Thread):
227 finally:291
228 os.chdir(wd)292 def __init__(self, path, address, port=0):
293 super(MercurialServerThread, self).__init__()
294 self.ui = hg_ui()
295 self.ui.setconfig("web", "address", address)
296 self.ui.setconfig("web", "port", 0)
297 self.app = hgweb(path, baseui=self.ui)
298 self.httpd = hgweb_server.create_server(self.ui, self.app)
299 # By default the Mercurial server output goes to stdout
300 self.httpd.errorlog = StringIO()
301 self.httpd.accesslog = StringIO()
302
303 def get_address(self):
304 return (self.httpd.addr, self.httpd.port)
305
306 def run(self):
307 self.httpd.serve_forever()
308
309 def stop(self):
310 self.httpd.shutdown()
229311
230312
231class MercurialServer(Server):313class MercurialServer(Server):
232314
233 def __init__(self, repo_url):315 def __init__(self, repository_path, use_server=False):
234 super(MercurialServer, self).__init__()316 super(MercurialServer, self).__init__()
235 self.repo_url = repo_url317 self.repository_path = repository_path
318 self._use_server = use_server
319
320 def get_url(self):
321 if self._use_server:
322 return "http://%s:%d/" % self._hgserver.get_address()
323 else:
324 return local_path_to_url(self.repository_path)
325
326 def start_server(self):
327 super(MercurialServer, self).start_server()
328 self.createRepository(self.repository_path)
329 if self._use_server:
330 self._hgserver = MercurialServerThread(self.repository_path, "localhost")
331 self._hgserver.start()
332
333 def stop_server(self):
334 super(MercurialServer, self).stop_server()
335 if self._use_server:
336 self._hgserver.stop()
337
338 def createRepository(self, path):
339 from mercurial.ui import ui
340 from mercurial.localrepo import localrepository
341 localrepository(ui(), self.repository_path, create=1)
236342
237 def makeRepo(self, tree_contents):343 def makeRepo(self, tree_contents):
238 from mercurial.ui import ui344 from mercurial.ui import ui
239 from mercurial.localrepo import localrepository345 from mercurial.localrepo import localrepository
240 repo = localrepository(ui(), self.repo_url, create=1)346 repo = localrepository(ui(), self.repository_path)
241 for filename, contents in tree_contents:347 for filename, contents in tree_contents:
242 f = open(os.path.join(self.repo_url, filename), 'w')348 f = open(os.path.join(self.repository_path, filename), 'w')
243 try:349 try:
244 f.write(contents)350 f.write(contents)
245 finally:351 finally:
246 f.close()352 f.close()
247 repo[None].add([filename])353 repo[None].add([filename])
248 repo.commit(text='<The commit message>', user='jane Foo <joe@foo.com>')354 repo.commit(text='<The commit message>', user='jane Foo <joe@foo.com>')
355
356
357class BzrServer(Server):
358
359 def __init__(self, repository_path, use_server=False):
360 super(BzrServer, self).__init__()
361 self.repository_path = repository_path
362 self._use_server = use_server
363
364 def createRepository(self, path):
365 BzrDir.create_branch_convenience(path)
366
367 def makeRepo(self, tree_contents):
368 branch = Branch.open(self.repository_path)
369 branch.get_config().set_user_option("create_signatures", "never")
370 builder = BranchBuilder(branch=branch)
371 actions = [('add', ('', 'tree-root', 'directory', None))]
372 actions += [('add', (path, path+'-id', 'file', content)) for (path,
373 content) in tree_contents]
374 builder.build_snapshot(None, None,
375 actions, committer='Joe Foo <joe@foo.com>',
376 message=u'<The commit message>')
377
378 def get_url(self):
379 if self._use_server:
380 return self._bzrserver.get_url()
381 else:
382 return local_path_to_url(self.repository_path)
383
384 def start_server(self):
385 super(BzrServer, self).start_server()
386 self.createRepository(self.repository_path)
387 class LocalURLServer(TestServer):
388 def __init__(self, repository_path):
389 self.repository_path = repository_path
390 def start_server(self): pass
391 def get_url(self):
392 return local_path_to_url(self.repository_path)
393 if self._use_server:
394 self._bzrserver = ReadonlySmartTCPServer_for_testing()
395 self._bzrserver.start_server(
396 LocalURLServer(self.repository_path))
397
398 def stop_server(self):
399 super(BzrServer, self).stop_server()
400 if self._use_server:
401 self._bzrserver.stop_server()
249402
=== modified file 'lib/lp/codehosting/codeimport/tests/test_worker.py'
--- lib/lp/codehosting/codeimport/tests/test_worker.py 2011-07-20 17:20:03 +0000
+++ lib/lp/codehosting/codeimport/tests/test_worker.py 2011-08-07 20:42:25 +0000
@@ -17,6 +17,9 @@
17 Branch,17 Branch,
18 BranchReferenceFormat,18 BranchReferenceFormat,
19 )19 )
20from bzrlib.branchbuilder import (
21 BranchBuilder,
22 )
20from bzrlib.bzrdir import (23from bzrlib.bzrdir import (
21 BzrDir,24 BzrDir,
22 BzrDirFormat,25 BzrDirFormat,
@@ -47,6 +50,7 @@
47 extract_tarball,50 extract_tarball,
48 )51 )
49from lp.codehosting.codeimport.tests.servers import (52from lp.codehosting.codeimport.tests.servers import (
53 BzrServer,
50 CVSServer,54 CVSServer,
51 GitServer,55 GitServer,
52 MercurialServer,56 MercurialServer,
@@ -54,6 +58,7 @@
54 )58 )
55from lp.codehosting.codeimport.worker import (59from lp.codehosting.codeimport.worker import (
56 BazaarBranchStore,60 BazaarBranchStore,
61 BzrImportWorker,
57 BzrSvnImportWorker,62 BzrSvnImportWorker,
58 CodeImportWorkerExitCode,63 CodeImportWorkerExitCode,
59 CSCVSImportWorker,64 CSCVSImportWorker,
@@ -959,7 +964,7 @@
959 def makeSourceDetails(self, branch_name, files):964 def makeSourceDetails(self, branch_name, files):
960 """Make a SVN `CodeImportSourceDetails` pointing at a real SVN repo.965 """Make a SVN `CodeImportSourceDetails` pointing at a real SVN repo.
961 """966 """
962 svn_server = SubversionServer(self.makeTemporaryDirectory())967 svn_server = SubversionServer(self.makeTemporaryDirectory(), True)
963 svn_server.start_server()968 svn_server.start_server()
964 self.addCleanup(svn_server.stop_server)969 self.addCleanup(svn_server.stop_server)
965970
@@ -999,10 +1004,10 @@
999 # import should be rejected.1004 # import should be rejected.
1000 args = {'rcstype': self.rcstype}1005 args = {'rcstype': self.rcstype}
1001 reference_url = self.createBranchReference()1006 reference_url = self.createBranchReference()
1002 if self.rcstype in ('git', 'bzr-svn', 'hg'):1007 if self.rcstype in ('git', 'bzr-svn', 'hg', 'bzr'):
1003 args['url'] = reference_url1008 args['url'] = reference_url
1004 else:1009 else:
1005 raise AssertionError("unexpected rcs_type %r" % self.rcs_type)1010 raise AssertionError("unexpected rcs_type %r" % self.rcstype)
1006 source_details = self.factory.makeCodeImportSourceDetails(**args)1011 source_details = self.factory.makeCodeImportSourceDetails(**args)
1007 worker = self.makeImportWorker(source_details)1012 worker = self.makeImportWorker(source_details)
1008 self.assertEqual(1013 self.assertEqual(
@@ -1011,7 +1016,21 @@
1011 def test_invalid(self):1016 def test_invalid(self):
1012 # If there is no branch in the target URL, exit with FAILURE_INVALID1017 # If there is no branch in the target URL, exit with FAILURE_INVALID
1013 worker = self.makeImportWorker(self.factory.makeCodeImportSourceDetails(1018 worker = self.makeImportWorker(self.factory.makeCodeImportSourceDetails(
1014 rcstype=self.rcstype, url="file:///path/non/existant"))1019 rcstype=self.rcstype, url="http://localhost/path/non/existant"))
1020 self.assertEqual(
1021 CodeImportWorkerExitCode.FAILURE_INVALID, worker.run())
1022
1023 def test_bad_url(self):
1024 # Local path URLs are not allowed
1025 worker = self.makeImportWorker(self.factory.makeCodeImportSourceDetails(
1026 rcstype=self.rcstype, url="file:///tmp/path/non/existant"))
1027 self.assertEqual(
1028 CodeImportWorkerExitCode.FAILURE_INVALID, worker.run())
1029
1030 def test_launchpad_url(self):
1031 # Launchpad URLs are not allowed
1032 worker = self.makeImportWorker(self.factory.makeCodeImportSourceDetails(
1033 rcstype=self.rcstype, url="https://code.launchpad.net/linux/"))
1015 self.assertEqual(1034 self.assertEqual(
1016 CodeImportWorkerExitCode.FAILURE_INVALID, worker.run())1035 CodeImportWorkerExitCode.FAILURE_INVALID, worker.run())
10171036
@@ -1071,7 +1090,7 @@
10711090
1072 def makeForeignCommit(self, source_details):1091 def makeForeignCommit(self, source_details):
1073 """Change the foreign tree, generating exactly one commit."""1092 """Change the foreign tree, generating exactly one commit."""
1074 repo = GitRepo(source_details.url)1093 repo = GitRepo(local_path_from_url(source_details.url))
1075 repo.do_commit(message=self.factory.getUniqueString(),1094 repo.do_commit(message=self.factory.getUniqueString(),
1076 committer="Joe Random Hacker <joe@example.com>")1095 committer="Joe Random Hacker <joe@example.com>")
1077 self.foreign_commit_count += 11096 self.foreign_commit_count += 1
@@ -1080,7 +1099,7 @@
1080 """Make a Git `CodeImportSourceDetails` pointing at a real Git repo.1099 """Make a Git `CodeImportSourceDetails` pointing at a real Git repo.
1081 """1100 """
1082 repository_path = self.makeTemporaryDirectory()1101 repository_path = self.makeTemporaryDirectory()
1083 git_server = GitServer(repository_path)1102 git_server = GitServer(repository_path, use_server=True)
1084 git_server.start_server()1103 git_server.start_server()
1085 self.addCleanup(git_server.stop_server)1104 self.addCleanup(git_server.stop_server)
10861105
@@ -1088,8 +1107,7 @@
1088 self.foreign_commit_count = 11107 self.foreign_commit_count = 1
10891108
1090 return self.factory.makeCodeImportSourceDetails(1109 return self.factory.makeCodeImportSourceDetails(
1091 rcstype='git', url=repository_path)1110 rcstype='git', url=git_server.get_url())
1092
10931111
10941112
1095class TestMercurialImport(WorkerTest, TestActualImportMixin,1113class TestMercurialImport(WorkerTest, TestActualImportMixin,
@@ -1124,7 +1142,7 @@
1124 """Change the foreign tree, generating exactly one commit."""1142 """Change the foreign tree, generating exactly one commit."""
1125 from mercurial.ui import ui1143 from mercurial.ui import ui
1126 from mercurial.localrepo import localrepository1144 from mercurial.localrepo import localrepository
1127 repo = localrepository(ui(), source_details.url)1145 repo = localrepository(ui(), local_path_from_url(source_details.url))
1128 repo.commit(text="hello world!", user="Jane Random Hacker", force=1)1146 repo.commit(text="hello world!", user="Jane Random Hacker", force=1)
1129 self.foreign_commit_count += 11147 self.foreign_commit_count += 1
11301148
@@ -1132,7 +1150,7 @@
1132 """Make a Mercurial `CodeImportSourceDetails` pointing at a real repo.1150 """Make a Mercurial `CodeImportSourceDetails` pointing at a real repo.
1133 """1151 """
1134 repository_path = self.makeTemporaryDirectory()1152 repository_path = self.makeTemporaryDirectory()
1135 hg_server = MercurialServer(repository_path)1153 hg_server = MercurialServer(repository_path, use_server=True)
1136 hg_server.start_server()1154 hg_server.start_server()
1137 self.addCleanup(hg_server.stop_server)1155 self.addCleanup(hg_server.stop_server)
11381156
@@ -1140,7 +1158,7 @@
1140 self.foreign_commit_count = 11158 self.foreign_commit_count = 1
11411159
1142 return self.factory.makeCodeImportSourceDetails(1160 return self.factory.makeCodeImportSourceDetails(
1143 rcstype='hg', url=repository_path)1161 rcstype='hg', url=hg_server.get_url())
11441162
11451163
1146class TestBzrSvnImport(WorkerTest, SubversionImportHelpers,1164class TestBzrSvnImport(WorkerTest, SubversionImportHelpers,
@@ -1160,5 +1178,51 @@
1160 self.bazaar_store, logging.getLogger())1178 self.bazaar_store, logging.getLogger())
11611179
11621180
1181class TestBzrImport(WorkerTest, TestActualImportMixin,
1182 PullingImportWorkerTests):
1183
1184 rcstype = 'bzr'
1185
1186 def setUp(self):
1187 super(TestBzrImport, self).setUp()
1188 self.setUpImport()
1189
1190 def makeImportWorker(self, source_details):
1191 """Make a new `ImportWorker`."""
1192 return BzrImportWorker(
1193 source_details, self.get_transport('import_data'),
1194 self.bazaar_store, logging.getLogger())
1195
1196 def makeForeignCommit(self, source_details):
1197 """Change the foreign tree, generating exactly one commit."""
1198 branch = Branch.open(source_details.url)
1199 builder = BranchBuilder(branch=branch)
1200 builder.build_commit(message=self.factory.getUniqueString(),
1201 committer="Joe Random Hacker <joe@example.com>")
1202 self.foreign_commit_count += 1
1203
1204 def makeSourceDetails(self, branch_name, files):
1205 """Make Bzr `CodeImportSourceDetails` pointing at a real Bzr repo.
1206 """
1207 repository_path = self.makeTemporaryDirectory()
1208 bzr_server = BzrServer(repository_path, use_server=True)
1209 bzr_server.start_server()
1210 self.addCleanup(bzr_server.stop_server)
1211
1212 bzr_server.makeRepo(files)
1213 self.foreign_commit_count = 1
1214
1215 return self.factory.makeCodeImportSourceDetails(
1216 rcstype='bzr', url=bzr_server.get_url())
1217
1218 def test_partial(self):
1219 self.skip(
1220 "Partial fetching is not supported for native bzr branches "
1221 "at the moment.")
1222
1223 def test_unsupported_feature(self):
1224 self.skip("All Bazaar features are supported by Bazaar.")
1225
1226
1163def test_suite():1227def test_suite():
1164 return unittest.TestLoader().loadTestsFromName(__name__)1228 return unittest.TestLoader().loadTestsFromName(__name__)
11651229
=== modified file 'lib/lp/codehosting/codeimport/tests/test_workermonitor.py'
--- lib/lp/codehosting/codeimport/tests/test_workermonitor.py 2011-06-16 23:43:04 +0000
+++ lib/lp/codehosting/codeimport/tests/test_workermonitor.py 2011-08-07 20:42:25 +0000
@@ -50,6 +50,7 @@
50from lp.code.model.codeimportjob import CodeImportJob50from lp.code.model.codeimportjob import CodeImportJob
51from lp.codehosting import load_optional_plugin51from lp.codehosting import load_optional_plugin
52from lp.codehosting.codeimport.tests.servers import (52from lp.codehosting.codeimport.tests.servers import (
53 BzrServer,
53 CVSServer,54 CVSServer,
54 GitServer,55 GitServer,
55 MercurialServer,56 MercurialServer,
@@ -661,26 +662,38 @@
661 def makeGitCodeImport(self):662 def makeGitCodeImport(self):
662 """Make a `CodeImport` that points to a real Git repository."""663 """Make a `CodeImport` that points to a real Git repository."""
663 load_optional_plugin('git')664 load_optional_plugin('git')
664 self.git_server = GitServer(self.repo_path)665 self.git_server = GitServer(self.repo_path, use_server=True)
665 self.git_server.start_server()666 self.git_server.start_server()
666 self.addCleanup(self.git_server.stop_server)667 self.addCleanup(self.git_server.stop_server)
667668
668 self.git_server.makeRepo([('README', 'contents')])669 self.git_server.makeRepo([('README', 'contents')])
669 self.foreign_commit_count = 1670 self.foreign_commit_count = 1
670671
671 return self.factory.makeCodeImport(git_repo_url=self.repo_path)672 return self.factory.makeCodeImport(
673 git_repo_url=self.git_server.get_url())
672674
673 def makeHgCodeImport(self):675 def makeHgCodeImport(self):
674 """Make a `CodeImport` that points to a real Mercurial repository."""676 """Make a `CodeImport` that points to a real Mercurial repository."""
675 load_optional_plugin('hg')677 load_optional_plugin('hg')
676 self.hg_server = MercurialServer(self.repo_path)678 self.hg_server = MercurialServer(self.repo_path, use_server=True)
677 self.hg_server.start_server()679 self.hg_server.start_server()
678 self.addCleanup(self.hg_server.stop_server)680 self.addCleanup(self.hg_server.stop_server)
679681
680 self.hg_server.makeRepo([('README', 'contents')])682 self.hg_server.makeRepo([('README', 'contents')])
681 self.foreign_commit_count = 1683 self.foreign_commit_count = 1
682684
683 return self.factory.makeCodeImport(hg_repo_url=self.repo_path)685 return self.factory.makeCodeImport(
686 hg_repo_url=self.hg_server.get_url())
687
688 def makeBzrCodeImport(self):
689 """Make a `CodeImport` that points to a real Bazaar branch."""
690 self.bzr_server = BzrServer(self.repo_path)
691 self.bzr_server.start_server()
692 self.addCleanup(self.bzr_server.stop_server)
693
694 self.bzr_server.makeRepo([('README', 'contents')])
695 self.foreign_commit_count = 1
696 return self.factory.makeCodeImport(bzr_branch_url=self.repo_path)
684697
685 def getStartedJobForImport(self, code_import):698 def getStartedJobForImport(self, code_import):
686 """Get a started `CodeImportJob` for `code_import`.699 """Get a started `CodeImportJob` for `code_import`.
@@ -782,6 +795,15 @@
782 result = self.performImport(job_id)795 result = self.performImport(job_id)
783 return result.addCallback(self.assertImported, code_import_id)796 return result.addCallback(self.assertImported, code_import_id)
784797
798 def test_import_bzr(self):
799 # Create a Bazaar CodeImport and import it.
800 job = self.getStartedJobForImport(self.makeBzrCodeImport())
801 code_import_id = job.code_import.id
802 job_id = job.id
803 self.layer.txn.commit()
804 result = self.performImport(job_id)
805 return result.addCallback(self.assertImported, code_import_id)
806
785 # XXX 2010-03-24 MichaelHudson, bug=541526: This test fails intermittently807 # XXX 2010-03-24 MichaelHudson, bug=541526: This test fails intermittently
786 # in EC2.808 # in EC2.
787 def DISABLED_test_import_bzrsvn(self):809 def DISABLED_test_import_bzrsvn(self):
788810
=== modified file 'lib/lp/codehosting/codeimport/worker.py'
--- lib/lp/codehosting/codeimport/worker.py 2011-08-02 11:28:46 +0000
+++ lib/lp/codehosting/codeimport/worker.py 2011-08-07 20:42:25 +0000
@@ -6,6 +6,7 @@
6__metaclass__ = type6__metaclass__ = type
7__all__ = [7__all__ = [
8 'BazaarBranchStore',8 'BazaarBranchStore',
9 'BzrImportWorker',
9 'BzrSvnImportWorker',10 'BzrSvnImportWorker',
10 'CSCVSImportWorker',11 'CSCVSImportWorker',
11 'CodeImportSourceDetails',12 'CodeImportSourceDetails',
@@ -21,6 +22,7 @@
21import os22import os
22import shutil23import shutil
2324
25from bzrlib.upgrade import upgrade
24from bzrlib.branch import (26from bzrlib.branch import (
25 Branch,27 Branch,
26 InterBranch,28 InterBranch,
@@ -38,7 +40,6 @@
38 )40 )
39from bzrlib.transport import get_transport41from bzrlib.transport import get_transport
40import bzrlib.ui42import bzrlib.ui
41from bzrlib.upgrade import upgrade
42from bzrlib.urlutils import (43from bzrlib.urlutils import (
43 join as urljoin,44 join as urljoin,
44 local_path_from_url,45 local_path_from_url,
@@ -49,11 +50,22 @@
49import SCM50import SCM
5051
51from canonical.config import config52from canonical.config import config
53
54from lazr.uri import (
55 URI,
56 )
57
52from lp.code.enums import RevisionControlSystems58from lp.code.enums import RevisionControlSystems
59from lp.code.interfaces.branch import get_blacklisted_hostnames
53from lp.codehosting.codeimport.foreigntree import (60from lp.codehosting.codeimport.foreigntree import (
54 CVSWorkingTree,61 CVSWorkingTree,
55 SubversionWorkingTree,62 SubversionWorkingTree,
56 )63 )
64from lp.codehosting.safe_open import (
65 BadUrl,
66 BranchOpenPolicy,
67 SafeBranchOpener,
68 )
57from lp.codehosting.codeimport.tarball import (69from lp.codehosting.codeimport.tarball import (
58 create_tarball,70 create_tarball,
59 extract_tarball,71 extract_tarball,
@@ -62,6 +74,52 @@
62from lp.services.propertycache import cachedproperty74from lp.services.propertycache import cachedproperty
6375
6476
77class CodeImportBranchOpenPolicy(BranchOpenPolicy):
78 """Branch open policy for code imports.
79
80 In summary:
81 - follow references,
82 - only open non-Launchpad URLs
83 - only open the allowed schemes
84 """
85
86 allowed_schemes = ['http', 'https', 'svn', 'git', 'ftp']
87
88 def shouldFollowReferences(self):
89 """See `BranchOpenPolicy.shouldFollowReferences`.
90
91 We traverse branch references for MIRRORED branches because they
92 provide a useful redirection mechanism and we want to be consistent
93 with the bzr command line.
94 """
95 return True
96
97 def transformFallbackLocation(self, branch, url):
98 """See `BranchOpenPolicy.transformFallbackLocation`.
99
100 For mirrored branches, we stack on whatever the remote branch claims
101 to stack on, but this URL still needs to be checked.
102 """
103 return urljoin(branch.base, url), True
104
105 def checkOneURL(self, url):
106 """See `BranchOpenPolicy.checkOneURL`.
107
108 We refuse to mirror from Launchpad or a ssh-like or file URL.
109 """
110 uri = URI(url)
111 launchpad_domain = config.vhost.mainsite.hostname
112 if uri.underDomain(launchpad_domain):
113 raise BadUrl(url)
114 for hostname in get_blacklisted_hostnames():
115 if uri.underDomain(hostname):
116 raise BadUrl(url)
117 if uri.scheme in ['sftp', 'bzr+ssh']:
118 raise BadUrl(url)
119 elif uri.scheme not in self.allowed_schemes:
120 raise BadUrl(url)
121
122
65class CodeImportWorkerExitCode:123class CodeImportWorkerExitCode:
66 """Exit codes used by the code import worker script."""124 """Exit codes used by the code import worker script."""
67125
@@ -187,9 +245,9 @@
187245
188 :ivar branch_id: The id of the branch associated to this code import, used246 :ivar branch_id: The id of the branch associated to this code import, used
189 for locating the existing import and the foreign tree.247 for locating the existing import and the foreign tree.
190 :ivar rcstype: 'svn' or 'cvs' as appropriate.248 :ivar rcstype: 'svn', 'cvs', 'hg', 'git', 'bzr-svn', 'bzr' as appropriate.
191 :ivar url: The branch URL if rcstype in ['svn', 'bzr-svn',249 :ivar url: The branch URL if rcstype in ['svn', 'bzr-svn',
192 'git'], None otherwise.250 'git', 'hg', 'bzr'], None otherwise.
193 :ivar cvs_root: The $CVSROOT if rcstype == 'cvs', None otherwise.251 :ivar cvs_root: The $CVSROOT if rcstype == 'cvs', None otherwise.
194 :ivar cvs_module: The CVS module if rcstype == 'cvs', None otherwise.252 :ivar cvs_module: The CVS module if rcstype == 'cvs', None otherwise.
195 """253 """
@@ -207,7 +265,7 @@
207 """Convert command line-style arguments to an instance."""265 """Convert command line-style arguments to an instance."""
208 branch_id = int(arguments.pop(0))266 branch_id = int(arguments.pop(0))
209 rcstype = arguments.pop(0)267 rcstype = arguments.pop(0)
210 if rcstype in ['svn', 'bzr-svn', 'git', 'hg']:268 if rcstype in ['svn', 'bzr-svn', 'git', 'hg', 'bzr']:
211 [url] = arguments269 [url] = arguments
212 cvs_root = cvs_module = None270 cvs_root = cvs_module = None
213 elif rcstype == 'cvs':271 elif rcstype == 'cvs':
@@ -234,6 +292,8 @@
234 return cls(branch_id, 'git', str(code_import.url))292 return cls(branch_id, 'git', str(code_import.url))
235 elif code_import.rcs_type == RevisionControlSystems.HG:293 elif code_import.rcs_type == RevisionControlSystems.HG:
236 return cls(branch_id, 'hg', str(code_import.url))294 return cls(branch_id, 'hg', str(code_import.url))
295 elif code_import.rcs_type == RevisionControlSystems.BZR:
296 return cls(branch_id, 'bzr', str(code_import.url))
237 else:297 else:
238 raise AssertionError("Unknown rcstype %r." % code_import.rcs_type)298 raise AssertionError("Unknown rcstype %r." % code_import.rcs_type)
239299
@@ -241,7 +301,7 @@
241 """Return a list of arguments suitable for passing to a child process.301 """Return a list of arguments suitable for passing to a child process.
242 """302 """
243 result = [str(self.branch_id), self.rcstype]303 result = [str(self.branch_id), self.rcstype]
244 if self.rcstype in ['svn', 'bzr-svn', 'git', 'hg']:304 if self.rcstype in ['svn', 'bzr-svn', 'git', 'hg', 'bzr']:
245 result.append(self.url)305 result.append(self.url)
246 elif self.rcstype == 'cvs':306 elif self.rcstype == 'cvs':
247 result.append(self.cvs_root)307 result.append(self.cvs_root)
@@ -601,11 +661,18 @@
601 def _doImport(self):661 def _doImport(self):
602 self._logger.info("Starting job.")662 self._logger.info("Starting job.")
603 saved_factory = bzrlib.ui.ui_factory663 saved_factory = bzrlib.ui.ui_factory
664 opener_policy = CodeImportBranchOpenPolicy()
665 opener = SafeBranchOpener(opener_policy)
604 bzrlib.ui.ui_factory = LoggingUIFactory(logger=self._logger)666 bzrlib.ui.ui_factory = LoggingUIFactory(logger=self._logger)
605 try:667 try:
606 self._logger.info(668 self._logger.info(
607 "Getting exising bzr branch from central store.")669 "Getting exising bzr branch from central store.")
608 bazaar_branch = self.getBazaarBranch()670 bazaar_branch = self.getBazaarBranch()
671 try:
672 opener_policy.checkOneURL(self.source_details.url)
673 except BadUrl, e:
674 self._logger.info("Invalid URL: %s" % e)
675 return CodeImportWorkerExitCode.FAILURE_INVALID
609 transport = get_transport(self.source_details.url)676 transport = get_transport(self.source_details.url)
610 for prober_kls in self.probers:677 for prober_kls in self.probers:
611 prober = prober_kls()678 prober = prober_kls()
@@ -617,7 +684,13 @@
617 else:684 else:
618 self._logger.info("No branch found at remote location.")685 self._logger.info("No branch found at remote location.")
619 return CodeImportWorkerExitCode.FAILURE_INVALID686 return CodeImportWorkerExitCode.FAILURE_INVALID
620 remote_branch = format.open(transport).open_branch()687 remote_dir = format.open(transport)
688 try:
689 remote_branch = opener.runWithTransformFallbackLocationHookInstalled(
690 remote_dir.open_branch)
691 except BadUrl, e:
692 self._logger.info("Invalid URL: %s" % e)
693 return CodeImportWorkerExitCode.FAILURE_INVALID
621 remote_branch_tip = remote_branch.last_revision()694 remote_branch_tip = remote_branch.last_revision()
622 inter_branch = InterBranch.get(remote_branch, bazaar_branch)695 inter_branch = InterBranch.get(remote_branch, bazaar_branch)
623 self._logger.info("Importing branch.")696 self._logger.info("Importing branch.")
@@ -820,3 +893,21 @@
820 """See `PullingImportWorker.probers`."""893 """See `PullingImportWorker.probers`."""
821 from bzrlib.plugins.svn import SvnRemoteProber894 from bzrlib.plugins.svn import SvnRemoteProber
822 return [SvnRemoteProber]895 return [SvnRemoteProber]
896
897
898class BzrImportWorker(PullingImportWorker):
899 """An import worker for importing Bazaar branches."""
900
901 invalid_branch_exceptions = []
902 unsupported_feature_exceptions = []
903
904 def getRevisionLimit(self):
905 """See `PullingImportWorker.getRevisionLimit`."""
906 # For now, just grab the whole branch at once
907 return None
908
909 @property
910 def probers(self):
911 """See `PullingImportWorker.probers`."""
912 from bzrlib.bzrdir import BzrProber, RemoteBzrProber
913 return [BzrProber, RemoteBzrProber]
823914
=== modified file 'lib/lp/codehosting/puller/tests/__init__.py'
--- lib/lp/codehosting/puller/tests/__init__.py 2011-06-02 11:16:47 +0000
+++ lib/lp/codehosting/puller/tests/__init__.py 2011-08-07 20:42:25 +0000
@@ -25,75 +25,19 @@
25from canonical.config import config25from canonical.config import config
26from lp.codehosting.puller.worker import (26from lp.codehosting.puller.worker import (
27 BranchMirrorer,27 BranchMirrorer,
28 BranchMirrorerPolicy,
28 PullerWorker,29 PullerWorker,
29 PullerWorkerProtocol,30 PullerWorkerProtocol,
30 )31 )
31from lp.codehosting.tests.helpers import LoomTestMixin32from lp.codehosting.tests.helpers import LoomTestMixin
33from lp.codehosting.safe_open import AcceptAnythingPolicy
32from lp.codehosting.vfs import branch_id_to_path34from lp.codehosting.vfs import branch_id_to_path
33from lp.codehosting.vfs.branchfs import (
34 BadUrl,
35 BranchPolicy,
36 )
37from lp.testing import TestCaseWithFactory35from lp.testing import TestCaseWithFactory
3836
3937
40class BlacklistPolicy(BranchPolicy):38class AcceptAnythingBranchMirrorerPolicy(AcceptAnythingPolicy,
41 """Branch policy that forbids certain URLs."""39 BranchMirrorerPolicy): pass
4240
43 def __init__(self, should_follow_references, unsafe_urls=None):
44 if unsafe_urls is None:
45 unsafe_urls = set()
46 self._unsafe_urls = unsafe_urls
47 self._should_follow_references = should_follow_references
48
49 def shouldFollowReferences(self):
50 return self._should_follow_references
51
52 def checkOneURL(self, url):
53 if url in self._unsafe_urls:
54 raise BadUrl(url)
55
56 def transformFallbackLocation(self, branch, url):
57 """See `BranchPolicy.transformFallbackLocation`.
58
59 This class is not used for testing our smarter stacking features so we
60 just do the simplest thing: return the URL that would be used anyway
61 and don't check it.
62 """
63 return urlutils.join(branch.base, url), False
64
65
66class AcceptAnythingPolicy(BlacklistPolicy):
67 """Accept anything, to make testing easier."""
68
69 def __init__(self):
70 super(AcceptAnythingPolicy, self).__init__(True, set())
71
72
73class WhitelistPolicy(BranchPolicy):
74 """Branch policy that only allows certain URLs."""
75
76 def __init__(self, should_follow_references, allowed_urls=None,
77 check=False):
78 if allowed_urls is None:
79 allowed_urls = []
80 self.allowed_urls = set(url.rstrip('/') for url in allowed_urls)
81 self.check = check
82
83 def shouldFollowReferences(self):
84 return self._should_follow_references
85
86 def checkOneURL(self, url):
87 if url.rstrip('/') not in self.allowed_urls:
88 raise BadUrl(url)
89
90 def transformFallbackLocation(self, branch, url):
91 """See `BranchPolicy.transformFallbackLocation`.
92
93 Here we return the URL that would be used anyway and optionally check
94 it.
95 """
96 return urlutils.join(branch.base, url), self.check
9741
9842
99class PullerWorkerMixin:43class PullerWorkerMixin:
@@ -114,7 +58,7 @@
114 oops_prefix = ''58 oops_prefix = ''
115 if branch_type is None:59 if branch_type is None:
116 if policy is None:60 if policy is None:
117 policy = AcceptAnythingPolicy()61 policy = AcceptAnythingBranchMirrorerPolicy()
118 opener = BranchMirrorer(policy, protocol)62 opener = BranchMirrorer(policy, protocol)
119 else:63 else:
120 opener = None64 opener = None
12165
=== modified file 'lib/lp/codehosting/puller/tests/test_errors.py'
--- lib/lp/codehosting/puller/tests/test_errors.py 2010-08-20 20:31:18 +0000
+++ lib/lp/codehosting/puller/tests/test_errors.py 2011-08-07 20:42:25 +0000
@@ -23,17 +23,15 @@
2323
24from lp.code.enums import BranchType24from lp.code.enums import BranchType
25from lp.codehosting.puller.worker import (25from lp.codehosting.puller.worker import (
26 BadUrlLaunchpad,
27 BadUrlScheme,
28 BadUrlSsh,
26 BranchLoopError,29 BranchLoopError,
27 BranchMirrorer,30 BranchMirrorer,
28 BranchReferenceForbidden,31 BranchReferenceForbidden,
29 PullerWorker,32 PullerWorker,
30 PullerWorkerProtocol,33 PullerWorkerProtocol,
31 )34 )
32from lp.codehosting.vfs.branchfs import (
33 BadUrlLaunchpad,
34 BadUrlScheme,
35 BadUrlSsh,
36 )
3735
3836
39class StubbedPullerWorkerProtocol(PullerWorkerProtocol):37class StubbedPullerWorkerProtocol(PullerWorkerProtocol):
4038
=== modified file 'lib/lp/codehosting/puller/tests/test_worker.py'
--- lib/lp/codehosting/puller/tests/test_worker.py 2010-10-15 08:47:20 +0000
+++ lib/lp/codehosting/puller/tests/test_worker.py 2011-08-07 20:42:25 +0000
@@ -14,18 +14,15 @@
14import bzrlib.branch14import bzrlib.branch
15from bzrlib.branch import (15from bzrlib.branch import (
16 BranchReferenceFormat,16 BranchReferenceFormat,
17 BzrBranchFormat7,
18 )17 )
19from bzrlib.bzrdir import (18from bzrlib.bzrdir import (
20 BzrDir,19 BzrDir,
21 BzrDirMetaFormat1,
22 )20 )
23from bzrlib.errors import (21from bzrlib.errors import (
24 IncompatibleRepositories,22 IncompatibleRepositories,
25 NotBranchError,23 NotBranchError,
26 NotStacked,24 NotStacked,
27 )25 )
28from bzrlib.repofmt.pack_repo import RepositoryFormatKnitPack1
29from bzrlib.revision import NULL_REVISION26from bzrlib.revision import NULL_REVISION
30from bzrlib.tests import (27from bzrlib.tests import (
31 TestCaseInTempDir,28 TestCaseInTempDir,
@@ -35,28 +32,25 @@
3532
36from lp.code.enums import BranchType33from lp.code.enums import BranchType
37from lp.codehosting.puller.tests import (34from lp.codehosting.puller.tests import (
38 AcceptAnythingPolicy,
39 BlacklistPolicy,
40 FixedHttpServer,35 FixedHttpServer,
41 PullerWorkerMixin,36 PullerWorkerMixin,
42 WhitelistPolicy,
43 )37 )
44from lp.codehosting.puller.worker import (38from lp.codehosting.puller.worker import (
45 BranchLoopError,
46 BranchMirrorer,
47 BranchReferenceForbidden,
48 install_worker_ui_factory,
49 PullerWorkerProtocol,
50 WORKER_ACTIVITY_NETWORK,
51 )
52from lp.codehosting.vfs.branchfs import (
53 BadUrl,
54 BadUrlLaunchpad,39 BadUrlLaunchpad,
55 BadUrlScheme,40 BadUrlScheme,
56 BadUrlSsh,41 BadUrlSsh,
57 BranchPolicy,42 BranchMirrorerPolicy,
58 ImportedBranchPolicy,43 ImportedBranchPolicy,
44 install_worker_ui_factory,
59 MirroredBranchPolicy,45 MirroredBranchPolicy,
46 PullerWorkerProtocol,
47 SafeBranchOpener,
48 WORKER_ACTIVITY_NETWORK,
49 )
50from lp.codehosting.safe_open import (
51 AcceptAnythingPolicy,
52 BadUrl,
53 BranchOpenPolicy,
60 )54 )
61from lp.testing import TestCase55from lp.testing import TestCase
62from lp.testing.factory import (56from lp.testing.factory import (
@@ -81,7 +75,8 @@
81 return strings75 return strings
8276
8377
84class PrearrangedStackedBranchPolicy(AcceptAnythingPolicy):78class PrearrangedStackedBranchPolicy(BranchMirrorerPolicy,
79 AcceptAnythingPolicy):
85 """A branch policy that returns a pre-configurable stack-on URL."""80 """A branch policy that returns a pre-configurable stack-on URL."""
8681
87 def __init__(self, stack_on_url):82 def __init__(self, stack_on_url):
@@ -280,199 +275,8 @@
280 self.assertEqual('', stacked_on_url)275 self.assertEqual('', stacked_on_url)
281276
282277
283class TestBranchMirrorerCheckAndFollowBranchReference(TestCase):278class TestReferenceOpener(TestCaseWithTransport):
284 """Unit tests for `BranchMirrorer.checkAndFollowBranchReference`."""279 """Feature tests for safe opening of branch references."""
285
286 class StubbedBranchMirrorer(BranchMirrorer):
287 """BranchMirrorer that provides canned answers.
288
289 We implement the methods we need to to be able to control all the
290 inputs to the `BranchMirrorer.checkSource` method, which is what is
291 being tested in this class.
292 """
293
294 def __init__(self, references, policy):
295 parent_cls = TestBranchMirrorerCheckAndFollowBranchReference
296 super(parent_cls.StubbedBranchMirrorer, self).__init__(policy)
297 self._reference_values = {}
298 for i in range(len(references) - 1):
299 self._reference_values[references[i]] = references[i+1]
300 self.follow_reference_calls = []
301
302 def followReference(self, url):
303 self.follow_reference_calls.append(url)
304 return self._reference_values[url]
305
306 def makeBranchMirrorer(self, should_follow_references, references,
307 unsafe_urls=None):
308 policy = BlacklistPolicy(should_follow_references, unsafe_urls)
309 opener = self.StubbedBranchMirrorer(references, policy)
310 return opener
311
312 def testCheckInitialURL(self):
313 # checkSource rejects all URLs that are not allowed.
314 opener = self.makeBranchMirrorer(None, [], set(['a']))
315 self.assertRaises(BadUrl, opener.checkAndFollowBranchReference, 'a')
316
317 def testNotReference(self):
318 # When branch references are forbidden, checkAndFollowBranchReference
319 # does not raise on non-references.
320 opener = self.makeBranchMirrorer(False, ['a', None])
321 self.assertEquals('a', opener.checkAndFollowBranchReference('a'))
322 self.assertEquals(['a'], opener.follow_reference_calls)
323
324 def testBranchReferenceForbidden(self):
325 # checkAndFollowBranchReference raises BranchReferenceForbidden if
326 # branch references are forbidden and the source URL points to a
327 # branch reference.
328 opener = self.makeBranchMirrorer(False, ['a', 'b'])
329 self.assertRaises(
330 BranchReferenceForbidden,
331 opener.checkAndFollowBranchReference, 'a')
332 self.assertEquals(['a'], opener.follow_reference_calls)
333
334 def testAllowedReference(self):
335 # checkAndFollowBranchReference does not raise if following references
336 # is allowed and the source URL points to a branch reference to a
337 # permitted location.
338 opener = self.makeBranchMirrorer(True, ['a', 'b', None])
339 self.assertEquals('b', opener.checkAndFollowBranchReference('a'))
340 self.assertEquals(['a', 'b'], opener.follow_reference_calls)
341
342 def testCheckReferencedURLs(self):
343 # checkAndFollowBranchReference checks if the URL a reference points
344 # to is safe.
345 opener = self.makeBranchMirrorer(
346 True, ['a', 'b', None], unsafe_urls=set('b'))
347 self.assertRaises(BadUrl, opener.checkAndFollowBranchReference, 'a')
348 self.assertEquals(['a'], opener.follow_reference_calls)
349
350 def testSelfReferencingBranch(self):
351 # checkAndFollowBranchReference raises BranchReferenceLoopError if
352 # following references is allowed and the source url points to a
353 # self-referencing branch reference.
354 opener = self.makeBranchMirrorer(True, ['a', 'a'])
355 self.assertRaises(
356 BranchLoopError, opener.checkAndFollowBranchReference, 'a')
357 self.assertEquals(['a'], opener.follow_reference_calls)
358
359 def testBranchReferenceLoop(self):
360 # checkAndFollowBranchReference raises BranchReferenceLoopError if
361 # following references is allowed and the source url points to a loop
362 # of branch references.
363 references = ['a', 'b', 'a']
364 opener = self.makeBranchMirrorer(True, references)
365 self.assertRaises(
366 BranchLoopError, opener.checkAndFollowBranchReference, 'a')
367 self.assertEquals(['a', 'b'], opener.follow_reference_calls)
368
369
370class TestBranchMirrorerStacking(TestCaseWithTransport):
371
372 def makeBranchMirrorer(self, allowed_urls):
373 policy = WhitelistPolicy(True, allowed_urls, True)
374 return BranchMirrorer(policy)
375
376 def makeBranch(self, path, branch_format, repository_format):
377 """Make a Bazaar branch at 'path' with the given formats."""
378 bzrdir_format = BzrDirMetaFormat1()
379 bzrdir_format.set_branch_format(branch_format)
380 bzrdir = self.make_bzrdir(path, format=bzrdir_format)
381 repository_format.initialize(bzrdir)
382 return bzrdir.create_branch()
383
384 def testAllowedURL(self):
385 # checkSource does not raise an exception for branches stacked on
386 # branches with allowed URLs.
387 stacked_on_branch = self.make_branch('base-branch', format='1.6')
388 stacked_branch = self.make_branch('stacked-branch', format='1.6')
389 stacked_branch.set_stacked_on_url(stacked_on_branch.base)
390 opener = self.makeBranchMirrorer(
391 [stacked_branch.base, stacked_on_branch.base])
392 # This doesn't raise an exception.
393 opener.open(stacked_branch.base)
394
395 def testUnstackableRepository(self):
396 # checkSource treats branches with UnstackableRepositoryFormats as
397 # being not stacked.
398 branch = self.makeBranch(
399 'unstacked', BzrBranchFormat7(), RepositoryFormatKnitPack1())
400 opener = self.makeBranchMirrorer([branch.base])
401 # This doesn't raise an exception.
402 opener.open(branch.base)
403
404 def testAllowedRelativeURL(self):
405 # checkSource passes on absolute urls to checkOneURL, even if the
406 # value of stacked_on_location in the config is set to a relative URL.
407 stacked_on_branch = self.make_branch('base-branch', format='1.6')
408 stacked_branch = self.make_branch('stacked-branch', format='1.6')
409 stacked_branch.set_stacked_on_url('../base-branch')
410 opener = self.makeBranchMirrorer(
411 [stacked_branch.base, stacked_on_branch.base])
412 # Note that stacked_on_branch.base is not '../base-branch', it's an
413 # absolute URL.
414 self.assertNotEqual('../base-branch', stacked_on_branch.base)
415 # This doesn't raise an exception.
416 opener.open(stacked_branch.base)
417
418 def testAllowedRelativeNested(self):
419 # Relative URLs are resolved relative to the stacked branch.
420 self.get_transport().mkdir('subdir')
421 a = self.make_branch('subdir/a', format='1.6')
422 b = self.make_branch('b', format='1.6')
423 b.set_stacked_on_url('../subdir/a')
424 c = self.make_branch('subdir/c', format='1.6')
425 c.set_stacked_on_url('../../b')
426 opener = self.makeBranchMirrorer([c.base, b.base, a.base])
427 # This doesn't raise an exception.
428 opener.open(c.base)
429
430 def testForbiddenURL(self):
431 # checkSource raises a BadUrl exception if a branch is stacked on a
432 # branch with a forbidden URL.
433 stacked_on_branch = self.make_branch('base-branch', format='1.6')
434 stacked_branch = self.make_branch('stacked-branch', format='1.6')
435 stacked_branch.set_stacked_on_url(stacked_on_branch.base)
436 opener = self.makeBranchMirrorer([stacked_branch.base])
437 self.assertRaises(BadUrl, opener.open, stacked_branch.base)
438
439 def testForbiddenURLNested(self):
440 # checkSource raises a BadUrl exception if a branch is stacked on a
441 # branch that is in turn stacked on a branch with a forbidden URL.
442 a = self.make_branch('a', format='1.6')
443 b = self.make_branch('b', format='1.6')
444 b.set_stacked_on_url(a.base)
445 c = self.make_branch('c', format='1.6')
446 c.set_stacked_on_url(b.base)
447 opener = self.makeBranchMirrorer([c.base, b.base])
448 self.assertRaises(BadUrl, opener.open, c.base)
449
450 def testSelfStackedBranch(self):
451 # checkSource raises StackingLoopError if a branch is stacked on
452 # itself. This avoids infinite recursion errors.
453 a = self.make_branch('a', format='1.6')
454 # Bazaar 1.17 and up make it harder to create branches like this.
455 # It's still worth testing that we don't blow up in the face of them,
456 # so we grovel around a bit to create one anyway.
457 a.get_config().set_user_option('stacked_on_location', a.base)
458 opener = self.makeBranchMirrorer([a.base])
459 self.assertRaises(BranchLoopError, opener.open, a.base)
460
461 def testLoopStackedBranch(self):
462 # checkSource raises StackingLoopError if a branch is stacked in such
463 # a way so that it is ultimately stacked on itself. e.g. a stacked on
464 # b stacked on a.
465 a = self.make_branch('a', format='1.6')
466 b = self.make_branch('b', format='1.6')
467 a.set_stacked_on_url(b.base)
468 b.set_stacked_on_url(a.base)
469 opener = self.makeBranchMirrorer([a.base, b.base])
470 self.assertRaises(BranchLoopError, opener.open, a.base)
471 self.assertRaises(BranchLoopError, opener.open, b.base)
472
473
474class TestReferenceMirroring(TestCaseWithTransport):
475 """Feature tests for mirroring of branch references."""
476280
477 def createBranchReference(self, url):281 def createBranchReference(self, url):
478 """Create a pure branch reference that points to the specified URL.282 """Create a pure branch reference that points to the specified URL.
@@ -513,19 +317,19 @@
513 self.assertEqual(opened_branch.base, target_branch.base)317 self.assertEqual(opened_branch.base, target_branch.base)
514318
515 def testFollowReferenceValue(self):319 def testFollowReferenceValue(self):
516 # BranchMirrorer.followReference gives the reference value for320 # SafeBranchOpener.followReference gives the reference value for
517 # a branch reference.321 # a branch reference.
518 opener = BranchMirrorer(BranchPolicy())322 opener = SafeBranchOpener(BranchOpenPolicy())
519 reference_value = 'http://example.com/branch'323 reference_value = 'http://example.com/branch'
520 reference_url = self.createBranchReference(reference_value)324 reference_url = self.createBranchReference(reference_value)
521 self.assertEqual(325 self.assertEqual(
522 reference_value, opener.followReference(reference_url))326 reference_value, opener.followReference(reference_url))
523327
524 def testFollowReferenceNone(self):328 def testFollowReferenceNone(self):
525 # BranchMirrorer.followReference gives None for a normal branch.329 # SafeBranchOpener.followReference gives None for a normal branch.
526 self.make_branch('repo')330 self.make_branch('repo')
527 branch_url = self.get_url('repo')331 branch_url = self.get_url('repo')
528 opener = BranchMirrorer(BranchPolicy())332 opener = SafeBranchOpener(BranchOpenPolicy())
529 self.assertIs(None, opener.followReference(branch_url))333 self.assertIs(None, opener.followReference(branch_url))
530334
531335
532336
=== modified file 'lib/lp/codehosting/puller/tests/test_worker_formats.py'
--- lib/lp/codehosting/puller/tests/test_worker_formats.py 2010-08-20 20:31:18 +0000
+++ lib/lp/codehosting/puller/tests/test_worker_formats.py 2011-08-07 20:42:25 +0000
@@ -7,14 +7,18 @@
77
8import unittest8import unittest
99
10import lp.codehosting # for bzr plugins
11
10from bzrlib.branch import Branch12from bzrlib.branch import Branch
11from bzrlib.bzrdir import (13from bzrlib.bzrdir import (
12 BzrDirFormat6,
13 BzrDirMetaFormat1,14 BzrDirMetaFormat1,
14 )15 )
15from bzrlib.repofmt.knitrepo import RepositoryFormatKnit116from bzrlib.repofmt.knitrepo import RepositoryFormatKnit1
16from bzrlib.repofmt.pack_repo import RepositoryFormatKnitPack517from bzrlib.repofmt.knitpack_repo import RepositoryFormatKnitPack5
17from bzrlib.repofmt.weaverepo import (18from bzrlib.plugins.weave_fmt.bzrdir import (
19 BzrDirFormat6,
20 )
21from bzrlib.plugins.weave_fmt.repository import (
18 RepositoryFormat6,22 RepositoryFormat6,
19 RepositoryFormat7,23 RepositoryFormat7,
20 )24 )
2125
=== modified file 'lib/lp/codehosting/puller/worker.py'
--- lib/lp/codehosting/puller/worker.py 2011-06-03 01:00:53 +0000
+++ lib/lp/codehosting/puller/worker.py 2011-08-07 20:42:25 +0000
@@ -8,20 +8,32 @@
8import sys8import sys
9import urllib29import urllib2
1010
11from bzrlib import errors11import lp.codehosting # to load bzr plugins
12
13from bzrlib import (
14 errors,
15 urlutils,
16 )
12from bzrlib.branch import (17from bzrlib.branch import (
13 Branch,18 Branch,
19 )
20from bzrlib.plugins.weave_fmt.branch import (
14 BzrBranchFormat4,21 BzrBranchFormat4,
15 )22 )
16from bzrlib.bzrdir import BzrDir23from bzrlib.plugins.weave_fmt.repository import (
17from bzrlib.repofmt.weaverepo import (
18 RepositoryFormat4,24 RepositoryFormat4,
19 RepositoryFormat5,25 RepositoryFormat5,
20 RepositoryFormat6,26 RepositoryFormat6,
21 )27 )
28
22import bzrlib.ui29import bzrlib.ui
30from bzrlib.plugins.loom.branch import LoomSupport
31from bzrlib.transport import get_transport
23from bzrlib.ui import SilentUIFactory32from bzrlib.ui import SilentUIFactory
24from lazr.uri import InvalidURIError33from lazr.uri import (
34 InvalidURIError,
35 URI,
36 )
2537
26from canonical.config import config38from canonical.config import config
27from canonical.launchpad.webapp import errorlog39from canonical.launchpad.webapp import errorlog
@@ -32,18 +44,21 @@
32from lp.code.enums import BranchType44from lp.code.enums import BranchType
33from lp.codehosting.bzrutils import identical_formats45from lp.codehosting.bzrutils import identical_formats
34from lp.codehosting.puller import get_lock_id_for_branch_id46from lp.codehosting.puller import get_lock_id_for_branch_id
35from lp.codehosting.vfs.branchfs import (47from lp.codehosting.safe_open import (
36 BadUrlLaunchpad,48 BadUrl,
37 BadUrlScheme,49 BranchLoopError,
38 BadUrlSsh,50 BranchOpenPolicy,
39 make_branch_mirrorer,51 BranchReferenceForbidden,
52 SafeBranchOpener,
40 )53 )
4154
4255
43__all__ = [56__all__ = [
57 'BadUrlLaunchpad',
58 'BadUrlScheme',
59 'BadUrlSsh',
44 'BranchMirrorer',60 'BranchMirrorer',
45 'BranchLoopError',61 'BranchMirrorerPolicy',
46 'BranchReferenceForbidden',
47 'get_canonical_url_for_branch_name',62 'get_canonical_url_for_branch_name',
48 'install_worker_ui_factory',63 'install_worker_ui_factory',
49 'PullerWorker',64 'PullerWorker',
@@ -51,19 +66,22 @@
51 ]66 ]
5267
5368
54class BranchReferenceForbidden(Exception):69class BadUrlSsh(BadUrl):
55 """Trying to mirror a branch reference and the branch type does not allow70 """Tried to access a branch from sftp or bzr+ssh."""
56 references.71
57 """72
5873class BadUrlLaunchpad(BadUrl):
5974 """Tried to access a branch from launchpad.net."""
60class BranchLoopError(Exception):75
61 """Encountered a branch cycle.76
6277class BadUrlScheme(BadUrl):
63 A URL may point to a branch reference or it may point to a stacked branch.78 """Found a URL with an untrusted scheme."""
64 In either case, it's possible for there to be a cycle in these references,79
65 and this exception is raised when we detect such a cycle.80 def __init__(self, scheme, url):
66 """81 BadUrl.__init__(self, scheme, url)
82 self.scheme = scheme
83
84
6785
6886
69def get_canonical_url_for_branch_name(unique_name):87def get_canonical_url_for_branch_name(unique_name):
@@ -120,12 +138,50 @@
120 self.sendEvent('log', fmt % args)138 self.sendEvent('log', fmt % args)
121139
122140
141class BranchMirrorerPolicy(BranchOpenPolicy):
142 """The policy for what branches to open and how to stack them."""
143
144 def createDestinationBranch(self, source_branch, destination_url):
145 """Create a destination branch for 'source_branch'.
146
147 Creates a branch at 'destination_url' that is has the same format as
148 'source_branch'. Any content already at 'destination_url' will be
149 deleted. Generally the new branch will have no revisions, but they
150 will be copied for import branches, because this can be done safely
151 and efficiently with a vfs-level copy (see `ImportedBranchPolicy`).
152
153 :param source_branch: The Bazaar branch that will be mirrored.
154 :param destination_url: The place to make the destination branch. This
155 URL must point to a writable location.
156 :return: The destination branch.
157 """
158 dest_transport = get_transport(destination_url)
159 if dest_transport.has('.'):
160 dest_transport.delete_tree('.')
161 if isinstance(source_branch, LoomSupport):
162 # Looms suck.
163 revision_id = None
164 else:
165 revision_id = 'null:'
166 source_branch.bzrdir.clone_on_transport(
167 dest_transport, revision_id=revision_id)
168 return Branch.open(destination_url)
169
170 def getStackedOnURLForDestinationBranch(self, source_branch,
171 destination_url):
172 """Get the stacked on URL for `source_branch`.
173
174 In particular, the URL it should be stacked on when it is mirrored to
175 `destination_url`.
176 """
177 return None
178
179
123class BranchMirrorer(object):180class BranchMirrorer(object):
124 """A `BranchMirrorer` safely makes mirrors of branches.181 """A `BranchMirrorer` safely makes mirrors of branches.
125182
126 A `BranchMirrorer` has a `BranchPolicy` to tell it which URLs are safe to183 A `BranchMirrorer` has a `BranchOpenPolicy` to tell it which URLs are safe
127 accesss, whether or not to follow branch references and how to stack184 to accesss and whether or not to follow branch references.
128 branches when they are mirrored.
129185
130 The mirrorer knows how to follow branch references, create new mirrors,186 The mirrorer knows how to follow branch references, create new mirrors,
131 update existing mirrors, determine stacked-on branches and the like.187 update existing mirrors, determine stacked-on branches and the like.
@@ -136,92 +192,20 @@
136 def __init__(self, policy, protocol=None, log=None):192 def __init__(self, policy, protocol=None, log=None):
137 """Construct a branch opener with 'policy'.193 """Construct a branch opener with 'policy'.
138194
139 :param policy: A `BranchPolicy` that tells us what URLs are valid and195 :param policy: A `BranchOpenPolicy` that tells us what URLs are valid
140 similar things.196 and similar things.
141 :param log: A callable which can be called with a format string and197 :param log: A callable which can be called with a format string and
142 arguments to log messages in the scheduler, or None, in which case198 arguments to log messages in the scheduler, or None, in which case
143 log messages are discarded.199 log messages are discarded.
144 """200 """
145 self._seen_urls = set()
146 self.policy = policy201 self.policy = policy
147 self.protocol = protocol202 self.protocol = protocol
203 self.opener = SafeBranchOpener(policy)
148 if log is not None:204 if log is not None:
149 self.log = log205 self.log = log
150 else:206 else:
151 self.log = lambda *args: None207 self.log = lambda *args: None
152208
153 def _runWithTransformFallbackLocationHookInstalled(
154 self, callable, *args, **kw):
155 Branch.hooks.install_named_hook(
156 'transform_fallback_location', self.transformFallbackLocationHook,
157 'BranchMirrorer.transformFallbackLocationHook')
158 try:
159 return callable(*args, **kw)
160 finally:
161 # XXX 2008-11-24 MichaelHudson, bug=301472: This is the hacky way
162 # to remove a hook. The linked bug report asks for an API to do
163 # it.
164 Branch.hooks['transform_fallback_location'].remove(
165 self.transformFallbackLocationHook)
166 # We reset _seen_urls here to avoid multiple calls to open giving
167 # spurious loop exceptions.
168 self._seen_urls = set()
169
170 def open(self, url):
171 """Open the Bazaar branch at url, first checking for safety.
172
173 What safety means is defined by a subclasses `followReference` and
174 `checkOneURL` methods.
175 """
176 url = self.checkAndFollowBranchReference(url)
177 return self._runWithTransformFallbackLocationHookInstalled(
178 Branch.open, url)
179
180 def transformFallbackLocationHook(self, branch, url):
181 """Installed as the 'transform_fallback_location' Branch hook.
182
183 This method calls `transformFallbackLocation` on the policy object and
184 either returns the url it provides or passes it back to
185 checkAndFollowBranchReference.
186 """
187 new_url, check = self.policy.transformFallbackLocation(branch, url)
188 if check:
189 return self.checkAndFollowBranchReference(new_url)
190 else:
191 return new_url
192
193 def followReference(self, url):
194 """Get the branch-reference value at the specified url.
195
196 This exists as a separate method only to be overriden in unit tests.
197 """
198 bzrdir = BzrDir.open(url)
199 return bzrdir.get_branch_reference()
200
201 def checkAndFollowBranchReference(self, url):
202 """Check URL (and possibly the referenced URL) for safety.
203
204 This method checks that `url` passes the policy's `checkOneURL`
205 method, and if `url` refers to a branch reference, it checks whether
206 references are allowed and whether the reference's URL passes muster
207 also -- recursively, until a real branch is found.
208
209 :raise BranchLoopError: If the branch references form a loop.
210 :raise BranchReferenceForbidden: If this opener forbids branch
211 references.
212 """
213 while True:
214 if url in self._seen_urls:
215 raise BranchLoopError()
216 self._seen_urls.add(url)
217 self.policy.checkOneURL(url)
218 next_url = self.followReference(url)
219 if next_url is None:
220 return url
221 url = next_url
222 if not self.policy.shouldFollowReferences():
223 raise BranchReferenceForbidden(url)
224
225 def createDestinationBranch(self, source_branch, destination_url):209 def createDestinationBranch(self, source_branch, destination_url):
226 """Create a destination branch for 'source_branch'.210 """Create a destination branch for 'source_branch'.
227211
@@ -234,7 +218,7 @@
234 URL must point to a writable location.218 URL must point to a writable location.
235 :return: The destination branch.219 :return: The destination branch.
236 """220 """
237 return self._runWithTransformFallbackLocationHookInstalled(221 return self.opener.runWithTransformFallbackLocationHookInstalled(
238 self.policy.createDestinationBranch, source_branch,222 self.policy.createDestinationBranch, source_branch,
239 destination_url)223 destination_url)
240224
@@ -296,6 +280,9 @@
296 stacked_on_url = self.updateBranch(source_branch, branch)280 stacked_on_url = self.updateBranch(source_branch, branch)
297 return branch, revid_before, stacked_on_url281 return branch, revid_before, stacked_on_url
298282
283 def open(self, url):
284 return self.opener.open(url)
285
299286
300class PullerWorker:287class PullerWorker:
301 """This class represents a single branch that needs mirroring.288 """This class represents a single branch that needs mirroring.
@@ -305,7 +292,7 @@
305 """292 """
306293
307 def _checkerForBranchType(self, branch_type):294 def _checkerForBranchType(self, branch_type):
308 """Return a `BranchMirrorer` with an appropriate `BranchPolicy`.295 """Return a `BranchMirrorer` with an appropriate policy.
309296
310 :param branch_type: A `BranchType`. The policy of the mirrorer will297 :param branch_type: A `BranchType`. The policy of the mirrorer will
311 be based on this.298 be based on this.
@@ -546,3 +533,151 @@
546 created by another puller worker process.533 created by another puller worker process.
547 """534 """
548 bzrlib.ui.ui_factory = PullerWorkerUIFactory(puller_worker_protocol)535 bzrlib.ui.ui_factory = PullerWorkerUIFactory(puller_worker_protocol)
536
537
538class MirroredBranchPolicy(BranchMirrorerPolicy):
539 """Mirroring policy for MIRRORED branches.
540
541 In summary:
542
543 - follow references,
544 - only open non-Launchpad http: and https: URLs.
545 """
546
547 def __init__(self, stacked_on_url=None):
548 self.stacked_on_url = stacked_on_url
549
550 def getStackedOnURLForDestinationBranch(self, source_branch,
551 destination_url):
552 """Return the stacked on URL for the destination branch.
553
554 Mirrored branches are stacked on the default stacked-on branch of
555 their product, except when we're mirroring the default stacked-on
556 branch itself.
557 """
558 if self.stacked_on_url is None:
559 return None
560 stacked_on_url = urlutils.join(destination_url, self.stacked_on_url)
561 if destination_url == stacked_on_url:
562 return None
563 return self.stacked_on_url
564
565 def shouldFollowReferences(self):
566 """See `BranchOpenPolicy.shouldFollowReferences`.
567
568 We traverse branch references for MIRRORED branches because they
569 provide a useful redirection mechanism and we want to be consistent
570 with the bzr command line.
571 """
572 return True
573
574 def transformFallbackLocation(self, branch, url):
575 """See `BranchOpenPolicy.transformFallbackLocation`.
576
577 For mirrored branches, we stack on whatever the remote branch claims
578 to stack on, but this URL still needs to be checked.
579 """
580 return urlutils.join(branch.base, url), True
581
582 def checkOneURL(self, url):
583 """See `BranchOpenPolicy.checkOneURL`.
584
585 We refuse to mirror from Launchpad or a ssh-like or file URL.
586 """
587 # Avoid circular import
588 from lp.code.interfaces.branch import get_blacklisted_hostnames
589 uri = URI(url)
590 launchpad_domain = config.vhost.mainsite.hostname
591 if uri.underDomain(launchpad_domain):
592 raise BadUrlLaunchpad(url)
593 for hostname in get_blacklisted_hostnames():
594 if uri.underDomain(hostname):
595 raise BadUrl(url)
596 if uri.scheme in ['sftp', 'bzr+ssh']:
597 raise BadUrlSsh(url)
598 elif uri.scheme not in ['http', 'https']:
599 raise BadUrlScheme(uri.scheme, url)
600
601
602class ImportedBranchPolicy(BranchMirrorerPolicy):
603 """Mirroring policy for IMPORTED branches.
604
605 In summary:
606
607 - don't follow references,
608 - assert the URLs start with the prefix we expect for imported branches.
609 """
610
611 def createDestinationBranch(self, source_branch, destination_url):
612 """See `BranchOpenPolicy.createDestinationBranch`.
613
614 Because we control the process that creates import branches, a
615 vfs-level copy is safe and more efficient than a bzr fetch.
616 """
617 source_transport = source_branch.bzrdir.root_transport
618 dest_transport = get_transport(destination_url)
619 while True:
620 # We loop until the remote file list before and after the copy is
621 # the same to catch the case where the remote side is being
622 # mutated as we copy it.
623 if dest_transport.has('.'):
624 dest_transport.delete_tree('.')
625 files_before = set(source_transport.iter_files_recursive())
626 source_transport.copy_tree_to_transport(dest_transport)
627 files_after = set(source_transport.iter_files_recursive())
628 if files_before == files_after:
629 break
630 return Branch.open_from_transport(dest_transport)
631
632 def shouldFollowReferences(self):
633 """See `BranchOpenerPolicy.shouldFollowReferences`.
634
635 We do not traverse references for IMPORTED branches because the
636 code-import system should never produce branch references.
637 """
638 return False
639
640 def transformFallbackLocation(self, branch, url):
641 """See `BranchOpenerPolicy.transformFallbackLocation`.
642
643 Import branches should not be stacked, ever.
644 """
645 raise AssertionError("Import branch unexpectedly stacked!")
646
647 def checkOneURL(self, url):
648 """See `BranchOpenerPolicy.checkOneURL`.
649
650 If the URL we are mirroring from does not start how we expect the pull
651 URLs of import branches to start, something has gone badly wrong, so
652 we raise AssertionError if that's happened.
653 """
654 if not url.startswith(config.launchpad.bzr_imports_root_url):
655 raise AssertionError(
656 "Bogus URL for imported branch: %r" % url)
657
658
659def make_branch_mirrorer(branch_type, protocol=None,
660 mirror_stacked_on_url=None):
661 """Create a `BranchMirrorer` with the appropriate `BranchOpenerPolicy`.
662
663 :param branch_type: A `BranchType` to select a policy by.
664 :param protocol: Optional protocol for the mirrorer to work with.
665 If given, its log will also be used.
666 :param mirror_stacked_on_url: For mirrored branches, the default URL
667 to stack on. Ignored for other branch types.
668 :return: A `BranchMirrorer`.
669 """
670 if branch_type == BranchType.MIRRORED:
671 policy = MirroredBranchPolicy(mirror_stacked_on_url)
672 elif branch_type == BranchType.IMPORTED:
673 policy = ImportedBranchPolicy()
674 else:
675 raise AssertionError(
676 "Unexpected branch type: %r" % branch_type)
677
678 if protocol is not None:
679 log_function = protocol.log
680 else:
681 log_function = None
682
683 return BranchMirrorer(policy, protocol, log_function)
549684
=== added file 'lib/lp/codehosting/safe_open.py'
--- lib/lp/codehosting/safe_open.py 1970-01-01 00:00:00 +0000
+++ lib/lp/codehosting/safe_open.py 2011-08-07 20:42:25 +0000
@@ -0,0 +1,230 @@
1# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Safe branch opening."""
5
6__metaclass__ = type
7
8from bzrlib import urlutils
9from bzrlib.branch import Branch
10from bzrlib.bzrdir import BzrDir
11
12__all__ = [
13 'AcceptAnythingPolicy',
14 'BadUrl',
15 'BlacklistPolicy',
16 'BranchLoopError',
17 'BranchOpenPolicy',
18 'BranchReferenceForbidden',
19 'SafeBranchOpener',
20 'WhitelistPolicy',
21 ]
22
23
24# TODO JelmerVernooij 2011-08-06: This module is generic enough to be
25# in bzrlib, and may be of use to others.
26
27
28class BadUrl(Exception):
29 """Tried to access a branch from a bad URL."""
30
31
32class BranchReferenceForbidden(Exception):
33 """Trying to mirror a branch reference and the branch type does not allow
34 references.
35 """
36
37
38class BranchLoopError(Exception):
39 """Encountered a branch cycle.
40
41 A URL may point to a branch reference or it may point to a stacked branch.
42 In either case, it's possible for there to be a cycle in these references,
43 and this exception is raised when we detect such a cycle.
44 """
45
46
47class BranchOpenPolicy:
48 """Policy on how to open branches.
49
50 In particular, a policy determines which branches are safe to open by
51 checking their URLs and deciding whether or not to follow branch
52 references.
53 """
54
55 def shouldFollowReferences(self):
56 """Whether we traverse references when mirroring.
57
58 Subclasses must override this method.
59
60 If we encounter a branch reference and this returns false, an error is
61 raised.
62
63 :returns: A boolean to indicate whether to follow a branch reference.
64 """
65 raise NotImplementedError(self.shouldFollowReferences)
66
67 def transformFallbackLocation(self, branch, url):
68 """Validate, maybe modify, 'url' to be used as a stacked-on location.
69
70 :param branch: The branch that is being opened.
71 :param url: The URL that the branch provides for its stacked-on
72 location.
73 :return: (new_url, check) where 'new_url' is the URL of the branch to
74 actually open and 'check' is true if 'new_url' needs to be
75 validated by checkAndFollowBranchReference.
76 """
77 raise NotImplementedError(self.transformFallbackLocation)
78
79 def checkOneURL(self, url):
80 """Check the safety of the source URL.
81
82 Subclasses must override this method.
83
84 :param url: The source URL to check.
85 :raise BadUrl: subclasses are expected to raise this or a subclass
86 when it finds a URL it deems to be unsafe.
87 """
88 raise NotImplementedError(self.checkOneURL)
89
90
91class BlacklistPolicy(BranchOpenPolicy):
92 """Branch policy that forbids certain URLs."""
93
94 def __init__(self, should_follow_references, unsafe_urls=None):
95 if unsafe_urls is None:
96 unsafe_urls = set()
97 self._unsafe_urls = unsafe_urls
98 self._should_follow_references = should_follow_references
99
100 def shouldFollowReferences(self):
101 return self._should_follow_references
102
103 def checkOneURL(self, url):
104 if url in self._unsafe_urls:
105 raise BadUrl(url)
106
107 def transformFallbackLocation(self, branch, url):
108 """See `BranchOpenPolicy.transformFallbackLocation`.
109
110 This class is not used for testing our smarter stacking features so we
111 just do the simplest thing: return the URL that would be used anyway
112 and don't check it.
113 """
114 return urlutils.join(branch.base, url), False
115
116
117class AcceptAnythingPolicy(BlacklistPolicy):
118 """Accept anything, to make testing easier."""
119
120 def __init__(self):
121 super(AcceptAnythingPolicy, self).__init__(True, set())
122
123
124class WhitelistPolicy(BranchOpenPolicy):
125 """Branch policy that only allows certain URLs."""
126
127 def __init__(self, should_follow_references, allowed_urls=None,
128 check=False):
129 if allowed_urls is None:
130 allowed_urls = []
131 self.allowed_urls = set(url.rstrip('/') for url in allowed_urls)
132 self.check = check
133
134 def shouldFollowReferences(self):
135 return self._should_follow_references
136
137 def checkOneURL(self, url):
138 if url.rstrip('/') not in self.allowed_urls:
139 raise BadUrl(url)
140
141 def transformFallbackLocation(self, branch, url):
142 """See `BranchOpenPolicy.transformFallbackLocation`.
143
144 Here we return the URL that would be used anyway and optionally check
145 it.
146 """
147 return urlutils.join(branch.base, url), self.check
148
149
150class SafeBranchOpener(object):
151 """Safe branch opener.
152
153 The policy object is expected to have the following methods:
154 * checkOneURL
155 * shouldFollowReferences
156 * transformFallbackLocation
157 """
158
159 def __init__(self, policy):
160 self.policy = policy
161 self._seen_urls = set()
162
163 def checkAndFollowBranchReference(self, url):
164 """Check URL (and possibly the referenced URL) for safety.
165
166 This method checks that `url` passes the policy's `checkOneURL`
167 method, and if `url` refers to a branch reference, it checks whether
168 references are allowed and whether the reference's URL passes muster
169 also -- recursively, until a real branch is found.
170
171 :raise BranchLoopError: If the branch references form a loop.
172 :raise BranchReferenceForbidden: If this opener forbids branch
173 references.
174 """
175 while True:
176 if url in self._seen_urls:
177 raise BranchLoopError()
178 self._seen_urls.add(url)
179 self.policy.checkOneURL(url)
180 next_url = self.followReference(url)
181 if next_url is None:
182 return url
183 url = next_url
184 if not self.policy.shouldFollowReferences():
185 raise BranchReferenceForbidden(url)
186
187 def transformFallbackLocationHook(self, branch, url):
188 """Installed as the 'transform_fallback_location' Branch hook.
189
190 This method calls `transformFallbackLocation` on the policy object and
191 either returns the url it provides or passes it back to
192 checkAndFollowBranchReference.
193 """
194 new_url, check = self.policy.transformFallbackLocation(branch, url)
195 if check:
196 return self.checkAndFollowBranchReference(new_url)
197 else:
198 return new_url
199
200 def runWithTransformFallbackLocationHookInstalled(
201 self, callable, *args, **kw):
202 Branch.hooks.install_named_hook(
203 'transform_fallback_location', self.transformFallbackLocationHook,
204 'SafeBranchOpener.transformFallbackLocationHook')
205 try:
206 return callable(*args, **kw)
207 finally:
208 Branch.hooks.uninstall_named_hook('transform_fallback_location',
209 'SafeBranchOpener.transformFallbackLocationHook')
210 # We reset _seen_urls here to avoid multiple calls to open giving
211 # spurious loop exceptions.
212 self._seen_urls = set()
213
214 def followReference(self, url):
215 """Get the branch-reference value at the specified url.
216
217 This exists as a separate method only to be overriden in unit tests.
218 """
219 bzrdir = BzrDir.open(url)
220 return bzrdir.get_branch_reference()
221
222 def open(self, url):
223 """Open the Bazaar branch at url, first checking for safety.
224
225 What safety means is defined by a subclasses `followReference` and
226 `checkOneURL` methods.
227 """
228 url = self.checkAndFollowBranchReference(url)
229 return self.runWithTransformFallbackLocationHookInstalled(
230 Branch.open, url)
0231
=== added file 'lib/lp/codehosting/tests/test_safe_open.py'
--- lib/lp/codehosting/tests/test_safe_open.py 1970-01-01 00:00:00 +0000
+++ lib/lp/codehosting/tests/test_safe_open.py 2011-08-07 20:42:25 +0000
@@ -0,0 +1,223 @@
1# Copyright 2009-2011 Canonical Ltd. This software is licensed under the
2# GNU Affero General Public License version 3 (see the file LICENSE).
3
4"""Tests for the safe branch open code."""
5
6
7__metaclass__ = type
8
9
10from lp.codehosting.puller.worker import (
11 BranchLoopError,
12 BranchReferenceForbidden,
13 SafeBranchOpener,
14 )
15from lp.codehosting.safe_open import (
16 BadUrl,
17 BlacklistPolicy,
18 WhitelistPolicy,
19 )
20
21from lp.testing import TestCase
22
23from bzrlib.branch import (
24 BzrBranchFormat7,
25 )
26from bzrlib.bzrdir import (
27 BzrDirMetaFormat1,
28 )
29from bzrlib.repofmt.pack_repo import RepositoryFormatKnitPack1
30from bzrlib.tests import (
31 TestCaseWithTransport,
32 )
33
34
35class TestSafeBranchOpenerCheckAndFollowBranchReference(TestCase):
36 """Unit tests for `SafeBranchOpener.checkAndFollowBranchReference`."""
37
38 class StubbedSafeBranchOpener(SafeBranchOpener):
39 """SafeBranchOpener that provides canned answers.
40
41 We implement the methods we need to to be able to control all the
42 inputs to the `BranchMirrorer.checkSource` method, which is what is
43 being tested in this class.
44 """
45
46 def __init__(self, references, policy):
47 parent_cls = TestSafeBranchOpenerCheckAndFollowBranchReference
48 super(parent_cls.StubbedSafeBranchOpener, self).__init__(policy)
49 self._reference_values = {}
50 for i in range(len(references) - 1):
51 self._reference_values[references[i]] = references[i+1]
52 self.follow_reference_calls = []
53
54 def followReference(self, url):
55 self.follow_reference_calls.append(url)
56 return self._reference_values[url]
57
58 def makeBranchOpener(self, should_follow_references, references,
59 unsafe_urls=None):
60 policy = BlacklistPolicy(should_follow_references, unsafe_urls)
61 opener = self.StubbedSafeBranchOpener(references, policy)
62 return opener
63
64 def testCheckInitialURL(self):
65 # checkSource rejects all URLs that are not allowed.
66 opener = self.makeBranchOpener(None, [], set(['a']))
67 self.assertRaises(BadUrl, opener.checkAndFollowBranchReference, 'a')
68
69 def testNotReference(self):
70 # When branch references are forbidden, checkAndFollowBranchReference
71 # does not raise on non-references.
72 opener = self.makeBranchOpener(False, ['a', None])
73 self.assertEquals('a', opener.checkAndFollowBranchReference('a'))
74 self.assertEquals(['a'], opener.follow_reference_calls)
75
76 def testBranchReferenceForbidden(self):
77 # checkAndFollowBranchReference raises BranchReferenceForbidden if
78 # branch references are forbidden and the source URL points to a
79 # branch reference.
80 opener = self.makeBranchOpener(False, ['a', 'b'])
81 self.assertRaises(
82 BranchReferenceForbidden,
83 opener.checkAndFollowBranchReference, 'a')
84 self.assertEquals(['a'], opener.follow_reference_calls)
85
86 def testAllowedReference(self):
87 # checkAndFollowBranchReference does not raise if following references
88 # is allowed and the source URL points to a branch reference to a
89 # permitted location.
90 opener = self.makeBranchOpener(True, ['a', 'b', None])
91 self.assertEquals('b', opener.checkAndFollowBranchReference('a'))
92 self.assertEquals(['a', 'b'], opener.follow_reference_calls)
93
94 def testCheckReferencedURLs(self):
95 # checkAndFollowBranchReference checks if the URL a reference points
96 # to is safe.
97 opener = self.makeBranchOpener(
98 True, ['a', 'b', None], unsafe_urls=set('b'))
99 self.assertRaises(BadUrl, opener.checkAndFollowBranchReference, 'a')
100 self.assertEquals(['a'], opener.follow_reference_calls)
101
102 def testSelfReferencingBranch(self):
103 # checkAndFollowBranchReference raises BranchReferenceLoopError if
104 # following references is allowed and the source url points to a
105 # self-referencing branch reference.
106 opener = self.makeBranchOpener(True, ['a', 'a'])
107 self.assertRaises(
108 BranchLoopError, opener.checkAndFollowBranchReference, 'a')
109 self.assertEquals(['a'], opener.follow_reference_calls)
110
111 def testBranchReferenceLoop(self):
112 # checkAndFollowBranchReference raises BranchReferenceLoopError if
113 # following references is allowed and the source url points to a loop
114 # of branch references.
115 references = ['a', 'b', 'a']
116 opener = self.makeBranchOpener(True, references)
117 self.assertRaises(
118 BranchLoopError, opener.checkAndFollowBranchReference, 'a')
119 self.assertEquals(['a', 'b'], opener.follow_reference_calls)
120
121
122class TestSafeBranchOpenerStacking(TestCaseWithTransport):
123
124 def makeBranchOpener(self, allowed_urls):
125 policy = WhitelistPolicy(True, allowed_urls, True)
126 return SafeBranchOpener(policy)
127
128 def makeBranch(self, path, branch_format, repository_format):
129 """Make a Bazaar branch at 'path' with the given formats."""
130 bzrdir_format = BzrDirMetaFormat1()
131 bzrdir_format.set_branch_format(branch_format)
132 bzrdir = self.make_bzrdir(path, format=bzrdir_format)
133 repository_format.initialize(bzrdir)
134 return bzrdir.create_branch()
135
136 def testAllowedURL(self):
137 # checkSource does not raise an exception for branches stacked on
138 # branches with allowed URLs.
139 stacked_on_branch = self.make_branch('base-branch', format='1.6')
140 stacked_branch = self.make_branch('stacked-branch', format='1.6')
141 stacked_branch.set_stacked_on_url(stacked_on_branch.base)
142 opener = self.makeBranchOpener(
143 [stacked_branch.base, stacked_on_branch.base])
144 # This doesn't raise an exception.
145 opener.open(stacked_branch.base)
146
147 def testUnstackableRepository(self):
148 # checkSource treats branches with UnstackableRepositoryFormats as
149 # being not stacked.
150 branch = self.makeBranch(
151 'unstacked', BzrBranchFormat7(), RepositoryFormatKnitPack1())
152 opener = self.makeBranchOpener([branch.base])
153 # This doesn't raise an exception.
154 opener.open(branch.base)
155
156 def testAllowedRelativeURL(self):
157 # checkSource passes on absolute urls to checkOneURL, even if the
158 # value of stacked_on_location in the config is set to a relative URL.
159 stacked_on_branch = self.make_branch('base-branch', format='1.6')
160 stacked_branch = self.make_branch('stacked-branch', format='1.6')
161 stacked_branch.set_stacked_on_url('../base-branch')
162 opener = self.makeBranchOpener(
163 [stacked_branch.base, stacked_on_branch.base])
164 # Note that stacked_on_branch.base is not '../base-branch', it's an
165 # absolute URL.
166 self.assertNotEqual('../base-branch', stacked_on_branch.base)
167 # This doesn't raise an exception.
168 opener.open(stacked_branch.base)
169
170 def testAllowedRelativeNested(self):
171 # Relative URLs are resolved relative to the stacked branch.
172 self.get_transport().mkdir('subdir')
173 a = self.make_branch('subdir/a', format='1.6')
174 b = self.make_branch('b', format='1.6')
175 b.set_stacked_on_url('../subdir/a')
176 c = self.make_branch('subdir/c', format='1.6')
177 c.set_stacked_on_url('../../b')
178 opener = self.makeBranchOpener([c.base, b.base, a.base])
179 # This doesn't raise an exception.
180 opener.open(c.base)
181
182 def testForbiddenURL(self):
183 # checkSource raises a BadUrl exception if a branch is stacked on a
184 # branch with a forbidden URL.
185 stacked_on_branch = self.make_branch('base-branch', format='1.6')
186 stacked_branch = self.make_branch('stacked-branch', format='1.6')
187 stacked_branch.set_stacked_on_url(stacked_on_branch.base)
188 opener = self.makeBranchOpener([stacked_branch.base])
189 self.assertRaises(BadUrl, opener.open, stacked_branch.base)
190
191 def testForbiddenURLNested(self):
192 # checkSource raises a BadUrl exception if a branch is stacked on a
193 # branch that is in turn stacked on a branch with a forbidden URL.
194 a = self.make_branch('a', format='1.6')
195 b = self.make_branch('b', format='1.6')
196 b.set_stacked_on_url(a.base)
197 c = self.make_branch('c', format='1.6')
198 c.set_stacked_on_url(b.base)
199 opener = self.makeBranchOpener([c.base, b.base])
200 self.assertRaises(BadUrl, opener.open, c.base)
201
202 def testSelfStackedBranch(self):
203 # checkSource raises StackingLoopError if a branch is stacked on
204 # itself. This avoids infinite recursion errors.
205 a = self.make_branch('a', format='1.6')
206 # Bazaar 1.17 and up make it harder to create branches like this.
207 # It's still worth testing that we don't blow up in the face of them,
208 # so we grovel around a bit to create one anyway.
209 a.get_config().set_user_option('stacked_on_location', a.base)
210 opener = self.makeBranchOpener([a.base])
211 self.assertRaises(BranchLoopError, opener.open, a.base)
212
213 def testLoopStackedBranch(self):
214 # checkSource raises StackingLoopError if a branch is stacked in such
215 # a way so that it is ultimately stacked on itself. e.g. a stacked on
216 # b stacked on a.
217 a = self.make_branch('a', format='1.6')
218 b = self.make_branch('b', format='1.6')
219 a.set_stacked_on_url(b.base)
220 b.set_stacked_on_url(a.base)
221 opener = self.makeBranchOpener([a.base, b.base])
222 self.assertRaises(BranchLoopError, opener.open, a.base)
223 self.assertRaises(BranchLoopError, opener.open, b.base)
0224
=== modified file 'lib/lp/codehosting/vfs/__init__.py'
--- lib/lp/codehosting/vfs/__init__.py 2010-09-22 17:16:19 +0000
+++ lib/lp/codehosting/vfs/__init__.py 2011-08-07 20:42:25 +0000
@@ -11,7 +11,6 @@
11 'get_ro_server',11 'get_ro_server',
12 'get_rw_server',12 'get_rw_server',
13 'LaunchpadServer',13 'LaunchpadServer',
14 'make_branch_mirrorer',
15 ]14 ]
1615
17from lp.codehosting.vfs.branchfs import (16from lp.codehosting.vfs.branchfs import (
@@ -21,7 +20,6 @@
21 get_ro_server,20 get_ro_server,
22 get_rw_server,21 get_rw_server,
23 LaunchpadServer,22 LaunchpadServer,
24 make_branch_mirrorer,
25 )23 )
26from lp.codehosting.vfs.branchfsclient import (24from lp.codehosting.vfs.branchfsclient import (
27 BranchFileSystemClient,25 BranchFileSystemClient,
2826
=== modified file 'lib/lp/codehosting/vfs/branchfs.py'
--- lib/lp/codehosting/vfs/branchfs.py 2011-05-25 19:57:07 +0000
+++ lib/lp/codehosting/vfs/branchfs.py 2011-08-07 20:42:25 +0000
@@ -46,17 +46,14 @@
46__metaclass__ = type46__metaclass__ = type
47__all__ = [47__all__ = [
48 'AsyncLaunchpadTransport',48 'AsyncLaunchpadTransport',
49 'BadUrl',
50 'BadUrlLaunchpad',49 'BadUrlLaunchpad',
51 'BadUrlScheme',50 'BadUrlScheme',
52 'BadUrlSsh',51 'BadUrlSsh',
53 'branch_id_to_path',52 'branch_id_to_path',
54 'BranchPolicy',
55 'DirectDatabaseLaunchpadServer',53 'DirectDatabaseLaunchpadServer',
56 'get_lp_server',54 'get_lp_server',
57 'get_ro_server',55 'get_ro_server',
58 'get_rw_server',56 'get_rw_server',
59 'make_branch_mirrorer',
60 'LaunchpadInternalServer',57 'LaunchpadInternalServer',
61 'LaunchpadServer',58 'LaunchpadServer',
62 ]59 ]
@@ -65,7 +62,6 @@
65import xmlrpclib62import xmlrpclib
6663
67from bzrlib import urlutils64from bzrlib import urlutils
68from bzrlib.branch import Branch
69from bzrlib.bzrdir import (65from bzrlib.bzrdir import (
70 BzrDir,66 BzrDir,
71 BzrDirFormat,67 BzrDirFormat,
@@ -76,7 +72,6 @@
76 PermissionDenied,72 PermissionDenied,
77 TransportNotPossible,73 TransportNotPossible,
78 )74 )
79from bzrlib.plugins.loom.branch import LoomSupport
80from bzrlib.smart.request import jail_info75from bzrlib.smart.request import jail_info
81from bzrlib.transport import get_transport76from bzrlib.transport import get_transport
82from bzrlib.transport.memory import MemoryServer77from bzrlib.transport.memory import MemoryServer
@@ -98,7 +93,6 @@
98from canonical.config import config93from canonical.config import config
99from canonical.launchpad.webapp import errorlog94from canonical.launchpad.webapp import errorlog
100from canonical.launchpad.xmlrpc import faults95from canonical.launchpad.xmlrpc import faults
101from lp.code.enums import BranchType
102from lp.code.interfaces.branchlookup import IBranchLookup96from lp.code.interfaces.branchlookup import IBranchLookup
103from lp.code.interfaces.codehosting import (97from lp.code.interfaces.codehosting import (
104 BRANCH_TRANSPORT,98 BRANCH_TRANSPORT,
@@ -126,26 +120,6 @@
126 )120 )
127121
128122
129class BadUrl(Exception):
130 """Tried to access a branch from a bad URL."""
131
132
133class BadUrlSsh(BadUrl):
134 """Tried to access a branch from sftp or bzr+ssh."""
135
136
137class BadUrlLaunchpad(BadUrl):
138 """Tried to access a branch from launchpad.net."""
139
140
141class BadUrlScheme(BadUrl):
142 """Found a URL with an untrusted scheme."""
143
144 def __init__(self, scheme, url):
145 BadUrl.__init__(self, scheme, url)
146 self.scheme = scheme
147
148
149# The directories allowed directly beneath a branch directory. These are the123# The directories allowed directly beneath a branch directory. These are the
150# directories that Bazaar creates as part of regular operation. We support124# directories that Bazaar creates as part of regular operation. We support
151# only two numbered backups to avoid indefinite space usage.125# only two numbered backups to avoid indefinite space usage.
@@ -774,234 +748,3 @@
774 seen_new_branch_hook)748 seen_new_branch_hook)
775 return lp_server749 return lp_server
776750
777
778class BranchPolicy:
779 """Policy on how to mirror branches.
780
781 In particular, a policy determines which branches are safe to mirror by
782 checking their URLs and deciding whether or not to follow branch
783 references. A policy also determines how the mirrors of branches should be
784 stacked.
785 """
786
787 def createDestinationBranch(self, source_branch, destination_url):
788 """Create a destination branch for 'source_branch'.
789
790 Creates a branch at 'destination_url' that is has the same format as
791 'source_branch'. Any content already at 'destination_url' will be
792 deleted. Generally the new branch will have no revisions, but they
793 will be copied for import branches, because this can be done safely
794 and efficiently with a vfs-level copy (see `ImportedBranchPolicy`,
795 below).
796
797 :param source_branch: The Bazaar branch that will be mirrored.
798 :param destination_url: The place to make the destination branch. This
799 URL must point to a writable location.
800 :return: The destination branch.
801 """
802 dest_transport = get_transport(destination_url)
803 if dest_transport.has('.'):
804 dest_transport.delete_tree('.')
805 if isinstance(source_branch, LoomSupport):
806 # Looms suck.
807 revision_id = None
808 else:
809 revision_id = 'null:'
810 source_branch.bzrdir.clone_on_transport(
811 dest_transport, revision_id=revision_id)
812 return Branch.open(destination_url)
813
814 def getStackedOnURLForDestinationBranch(self, source_branch,
815 destination_url):
816 """Get the stacked on URL for `source_branch`.
817
818 In particular, the URL it should be stacked on when it is mirrored to
819 `destination_url`.
820 """
821 return None
822
823 def shouldFollowReferences(self):
824 """Whether we traverse references when mirroring.
825
826 Subclasses must override this method.
827
828 If we encounter a branch reference and this returns false, an error is
829 raised.
830
831 :returns: A boolean to indicate whether to follow a branch reference.
832 """
833 raise NotImplementedError(self.shouldFollowReferences)
834
835 def transformFallbackLocation(self, branch, url):
836 """Validate, maybe modify, 'url' to be used as a stacked-on location.
837
838 :param branch: The branch that is being opened.
839 :param url: The URL that the branch provides for its stacked-on
840 location.
841 :return: (new_url, check) where 'new_url' is the URL of the branch to
842 actually open and 'check' is true if 'new_url' needs to be
843 validated by checkAndFollowBranchReference.
844 """
845 raise NotImplementedError(self.transformFallbackLocation)
846
847 def checkOneURL(self, url):
848 """Check the safety of the source URL.
849
850 Subclasses must override this method.
851
852 :param url: The source URL to check.
853 :raise BadUrl: subclasses are expected to raise this or a subclass
854 when it finds a URL it deems to be unsafe.
855 """
856 raise NotImplementedError(self.checkOneURL)
857
858
859class MirroredBranchPolicy(BranchPolicy):
860 """Mirroring policy for MIRRORED branches.
861
862 In summary:
863
864 - follow references,
865 - only open non-Launchpad http: and https: URLs.
866 """
867
868 def __init__(self, stacked_on_url=None):
869 self.stacked_on_url = stacked_on_url
870
871 def getStackedOnURLForDestinationBranch(self, source_branch,
872 destination_url):
873 """See `BranchPolicy.getStackedOnURLForDestinationBranch`.
874
875 Mirrored branches are stacked on the default stacked-on branch of
876 their product, except when we're mirroring the default stacked-on
877 branch itself.
878 """
879 if self.stacked_on_url is None:
880 return None
881 stacked_on_url = urlutils.join(destination_url, self.stacked_on_url)
882 if destination_url == stacked_on_url:
883 return None
884 return self.stacked_on_url
885
886 def shouldFollowReferences(self):
887 """See `BranchPolicy.shouldFollowReferences`.
888
889 We traverse branch references for MIRRORED branches because they
890 provide a useful redirection mechanism and we want to be consistent
891 with the bzr command line.
892 """
893 return True
894
895 def transformFallbackLocation(self, branch, url):
896 """See `BranchPolicy.transformFallbackLocation`.
897
898 For mirrored branches, we stack on whatever the remote branch claims
899 to stack on, but this URL still needs to be checked.
900 """
901 return urlutils.join(branch.base, url), True
902
903 def checkOneURL(self, url):
904 """See `BranchPolicy.checkOneURL`.
905
906 We refuse to mirror from Launchpad or a ssh-like or file URL.
907 """
908 # Avoid circular import
909 from lp.code.interfaces.branch import get_blacklisted_hostnames
910 uri = URI(url)
911 launchpad_domain = config.vhost.mainsite.hostname
912 if uri.underDomain(launchpad_domain):
913 raise BadUrlLaunchpad(url)
914 for hostname in get_blacklisted_hostnames():
915 if uri.underDomain(hostname):
916 raise BadUrl(url)
917 if uri.scheme in ['sftp', 'bzr+ssh']:
918 raise BadUrlSsh(url)
919 elif uri.scheme not in ['http', 'https']:
920 raise BadUrlScheme(uri.scheme, url)
921
922
923class ImportedBranchPolicy(BranchPolicy):
924 """Mirroring policy for IMPORTED branches.
925
926 In summary:
927
928 - don't follow references,
929 - assert the URLs start with the prefix we expect for imported branches.
930 """
931
932 def createDestinationBranch(self, source_branch, destination_url):
933 """See `BranchPolicy.createDestinationBranch`.
934
935 Because we control the process that creates import branches, a
936 vfs-level copy is safe and more efficient than a bzr fetch.
937 """
938 source_transport = source_branch.bzrdir.root_transport
939 dest_transport = get_transport(destination_url)
940 while True:
941 # We loop until the remote file list before and after the copy is
942 # the same to catch the case where the remote side is being
943 # mutated as we copy it.
944 if dest_transport.has('.'):
945 dest_transport.delete_tree('.')
946 files_before = set(source_transport.iter_files_recursive())
947 source_transport.copy_tree_to_transport(dest_transport)
948 files_after = set(source_transport.iter_files_recursive())
949 if files_before == files_after:
950 break
951 return Branch.open_from_transport(dest_transport)
952
953 def shouldFollowReferences(self):
954 """See `BranchPolicy.shouldFollowReferences`.
955
956 We do not traverse references for IMPORTED branches because the
957 code-import system should never produce branch references.
958 """
959 return False
960
961 def transformFallbackLocation(self, branch, url):
962 """See `BranchPolicy.transformFallbackLocation`.
963
964 Import branches should not be stacked, ever.
965 """
966 raise AssertionError("Import branch unexpectedly stacked!")
967
968 def checkOneURL(self, url):
969 """See `BranchPolicy.checkOneURL`.
970
971 If the URL we are mirroring from does not start how we expect the pull
972 URLs of import branches to start, something has gone badly wrong, so
973 we raise AssertionError if that's happened.
974 """
975 if not url.startswith(config.launchpad.bzr_imports_root_url):
976 raise AssertionError(
977 "Bogus URL for imported branch: %r" % url)
978
979
980def make_branch_mirrorer(branch_type, protocol=None,
981 mirror_stacked_on_url=None):
982 """Create a `BranchMirrorer` with the appropriate `BranchPolicy`.
983
984 :param branch_type: A `BranchType` to select a policy by.
985 :param protocol: Optional protocol for the mirrorer to work with.
986 If given, its log will also be used.
987 :param mirror_stacked_on_url: For mirrored branches, the default URL
988 to stack on. Ignored for other branch types.
989 :return: A `BranchMirrorer`.
990 """
991 # Avoid circular import
992 from lp.codehosting.puller.worker import BranchMirrorer
993
994 if branch_type == BranchType.MIRRORED:
995 policy = MirroredBranchPolicy(mirror_stacked_on_url)
996 elif branch_type == BranchType.IMPORTED:
997 policy = ImportedBranchPolicy()
998 else:
999 raise AssertionError(
1000 "Unexpected branch type: %r" % branch_type)
1001
1002 if protocol is not None:
1003 log_function = protocol.log
1004 else:
1005 log_function = None
1006
1007 return BranchMirrorer(policy, protocol, log_function)
1008751
=== modified file 'lib/lp/registry/browser/productseries.py'
--- lib/lp/registry/browser/productseries.py 2011-05-27 21:12:25 +0000
+++ lib/lp/registry/browser/productseries.py 2011-08-07 20:42:25 +0000
@@ -827,15 +827,6 @@
827 ))827 ))
828828
829829
830class RevisionControlSystemsExtended(RevisionControlSystems):
831 """External RCS plus Bazaar."""
832 BZR = DBItem(99, """
833 Bazaar
834
835 External Bazaar branch.
836 """)
837
838
839class SetBranchForm(Interface):830class SetBranchForm(Interface):
840 """The fields presented on the form for setting a branch."""831 """The fields presented on the form for setting a branch."""
841832
@@ -844,7 +835,7 @@
844 ['cvs_module'])835 ['cvs_module'])
845836
846 rcs_type = Choice(title=_("Type of RCS"),837 rcs_type = Choice(title=_("Type of RCS"),
847 required=False, vocabulary=RevisionControlSystemsExtended,838 required=False, vocabulary=RevisionControlSystems,
848 description=_(839 description=_(
849 "The version control system to import from. "))840 "The version control system to import from. "))
850841
@@ -908,7 +899,7 @@
908 @property899 @property
909 def initial_values(self):900 def initial_values(self):
910 return dict(901 return dict(
911 rcs_type=RevisionControlSystemsExtended.BZR,902 rcs_type=RevisionControlSystems.BZR,
912 branch_type=LINK_LP_BZR,903 branch_type=LINK_LP_BZR,
913 branch_location=self.context.branch)904 branch_location=self.context.branch)
914905
@@ -989,7 +980,7 @@
989 self.setFieldError(980 self.setFieldError(
990 'rcs_type',981 'rcs_type',
991 'You must specify the type of RCS for the remote host.')982 'You must specify the type of RCS for the remote host.')
992 elif rcs_type == RevisionControlSystemsExtended.CVS:983 elif rcs_type == RevisionControlSystems.CVS:
993 if 'cvs_module' not in data:984 if 'cvs_module' not in data:
994 self.setFieldError(985 self.setFieldError(
995 'cvs_module',986 'cvs_module',
@@ -1022,8 +1013,9 @@
1022 # Extend the allowed schemes for the repository URL based on1013 # Extend the allowed schemes for the repository URL based on
1023 # rcs_type.1014 # rcs_type.
1024 extra_schemes = {1015 extra_schemes = {
1025 RevisionControlSystemsExtended.BZR_SVN: ['svn'],1016 RevisionControlSystems.BZR_SVN: ['svn'],
1026 RevisionControlSystemsExtended.GIT: ['git'],1017 RevisionControlSystems.GIT: ['git'],
1018 RevisionControlSystems.BZR: ['bzr'],
1027 }1019 }
1028 schemes.update(extra_schemes.get(rcs_type, []))1020 schemes.update(extra_schemes.get(rcs_type, []))
1029 return schemes1021 return schemes
@@ -1050,7 +1042,7 @@
1050 # The branch location is not required for validation.1042 # The branch location is not required for validation.
1051 self._setRequired(['branch_location'], False)1043 self._setRequired(['branch_location'], False)
1052 # The cvs_module is required if it is a CVS import.1044 # The cvs_module is required if it is a CVS import.
1053 if rcs_type == RevisionControlSystemsExtended.CVS:1045 if rcs_type == RevisionControlSystems.CVS:
1054 self._setRequired(['cvs_module'], True)1046 self._setRequired(['cvs_module'], True)
1055 else:1047 else:
1056 raise AssertionError("Unknown branch type %s" % branch_type)1048 raise AssertionError("Unknown branch type %s" % branch_type)
@@ -1110,7 +1102,7 @@
1110 # Either create an externally hosted bzr branch1102 # Either create an externally hosted bzr branch
1111 # (a.k.a. 'mirrored') or create a new code import.1103 # (a.k.a. 'mirrored') or create a new code import.
1112 rcs_type = data.get('rcs_type')1104 rcs_type = data.get('rcs_type')
1113 if rcs_type == RevisionControlSystemsExtended.BZR:1105 if rcs_type == RevisionControlSystems.BZR:
1114 branch = self._createBzrBranch(1106 branch = self._createBzrBranch(
1115 BranchType.MIRRORED, branch_name, branch_owner,1107 BranchType.MIRRORED, branch_name, branch_owner,
1116 data['repo_url'])1108 data['repo_url'])
@@ -1123,7 +1115,7 @@
1123 'the series.')1115 'the series.')
1124 else:1116 else:
1125 # We need to create an import request.1117 # We need to create an import request.
1126 if rcs_type == RevisionControlSystemsExtended.CVS:1118 if rcs_type == RevisionControlSystems.CVS:
1127 cvs_root = data.get('repo_url')1119 cvs_root = data.get('repo_url')
1128 cvs_module = data.get('cvs_module')1120 cvs_module = data.get('cvs_module')
1129 url = None1121 url = None
11301122
=== modified file 'lib/lp/testing/factory.py'
--- lib/lp/testing/factory.py 2011-08-05 06:40:47 +0000
+++ lib/lp/testing/factory.py 2011-08-07 20:42:25 +0000
@@ -474,7 +474,7 @@
474 branch_id = self.getUniqueInteger()474 branch_id = self.getUniqueInteger()
475 if rcstype is None:475 if rcstype is None:
476 rcstype = 'svn'476 rcstype = 'svn'
477 if rcstype in ['svn', 'bzr-svn', 'hg']:477 if rcstype in ['svn', 'bzr-svn', 'hg', 'bzr']:
478 assert cvs_root is cvs_module is None478 assert cvs_root is cvs_module is None
479 if url is None:479 if url is None:
480 url = self.getUniqueURL()480 url = self.getUniqueURL()
@@ -2117,7 +2117,8 @@
21172117
2118 def makeCodeImport(self, svn_branch_url=None, cvs_root=None,2118 def makeCodeImport(self, svn_branch_url=None, cvs_root=None,
2119 cvs_module=None, target=None, branch_name=None,2119 cvs_module=None, target=None, branch_name=None,
2120 git_repo_url=None, hg_repo_url=None, registrant=None,2120 git_repo_url=None, hg_repo_url=None,
2121 bzr_branch_url=None, registrant=None,
2121 rcs_type=None, review_status=None):2122 rcs_type=None, review_status=None):
2122 """Create and return a new, arbitrary code import.2123 """Create and return a new, arbitrary code import.
21232124
@@ -2126,7 +2127,7 @@
2126 unique URL.2127 unique URL.
2127 """2128 """
2128 if (svn_branch_url is cvs_root is cvs_module is git_repo_url is2129 if (svn_branch_url is cvs_root is cvs_module is git_repo_url is
2129 hg_repo_url is None):2130 hg_repo_url is bzr_branch_url is None):
2130 svn_branch_url = self.getUniqueURL()2131 svn_branch_url = self.getUniqueURL()
21312132
2132 if target is None:2133 if target is None:
@@ -2157,6 +2158,11 @@
2157 registrant, target, branch_name,2158 registrant, target, branch_name,
2158 rcs_type=RevisionControlSystems.HG,2159 rcs_type=RevisionControlSystems.HG,
2159 url=hg_repo_url)2160 url=hg_repo_url)
2161 elif bzr_branch_url is not None:
2162 code_import = code_import_set.new(
2163 registrant, target, branch_name,
2164 rcs_type=RevisionControlSystems.BZR,
2165 url=bzr_branch_url)
2160 else:2166 else:
2161 assert rcs_type in (None, RevisionControlSystems.CVS)2167 assert rcs_type in (None, RevisionControlSystems.CVS)
2162 code_import = code_import_set.new(2168 code_import = code_import_set.new(
21632169
=== modified file 'lib/lp/translations/scripts/translations_to_branch.py'
--- lib/lp/translations/scripts/translations_to_branch.py 2011-05-27 13:36:02 +0000
+++ lib/lp/translations/scripts/translations_to_branch.py 2011-08-07 20:42:25 +0000
@@ -14,6 +14,7 @@
14import os.path14import os.path
1515
16from bzrlib.errors import NotBranchError16from bzrlib.errors import NotBranchError
17from bzrlib.revision import NULL_REVISION
17import pytz18import pytz
18from storm.expr import (19from storm.expr import (
19 And,20 And,
@@ -157,7 +158,9 @@
157158
158 revno, current_rev = branch.last_revision_info()159 revno, current_rev = branch.last_revision_info()
159 repository = branch.repository160 repository = branch.repository
160 for rev_id in repository.iter_reverse_revision_history(current_rev):161 graph = repository.get_graph()
162 for rev_id in graph.iter_lefthand_ancestry(
163 current_rev, (NULL_REVISION, )):
161 revision = repository.get_revision(rev_id)164 revision = repository.get_revision(rev_id)
162 revision_date = self._getRevisionTime(revision)165 revision_date = self._getRevisionTime(revision)
163 if self._isTranslationsCommit(revision):166 if self._isTranslationsCommit(revision):
164167
=== modified file 'scripts/code-import-worker.py'
--- scripts/code-import-worker.py 2011-06-16 23:43:04 +0000
+++ scripts/code-import-worker.py 2011-08-07 20:42:25 +0000
@@ -26,8 +26,9 @@
26from canonical.config import config26from canonical.config import config
27from lp.codehosting import load_optional_plugin27from lp.codehosting import load_optional_plugin
28from lp.codehosting.codeimport.worker import (28from lp.codehosting.codeimport.worker import (
29 BzrSvnImportWorker, CSCVSImportWorker, CodeImportSourceDetails,29 BzrImportWorker, BzrSvnImportWorker, CSCVSImportWorker,
30 GitImportWorker, HgImportWorker, get_default_bazaar_branch_store)30 CodeImportSourceDetails, GitImportWorker, HgImportWorker,
31 get_default_bazaar_branch_store)
31from canonical.launchpad import scripts32from canonical.launchpad import scripts
3233
3334
@@ -65,6 +66,10 @@
65 elif source_details.rcstype == 'hg':66 elif source_details.rcstype == 'hg':
66 load_optional_plugin('hg')67 load_optional_plugin('hg')
67 import_worker_cls = HgImportWorker68 import_worker_cls = HgImportWorker
69 elif source_details.rcstype == 'bzr':
70 load_optional_plugin('loom')
71 load_optional_plugin('weave_fmt')
72 import_worker_cls = BzrImportWorker
68 elif source_details.rcstype in ['cvs', 'svn']:73 elif source_details.rcstype in ['cvs', 'svn']:
69 import_worker_cls = CSCVSImportWorker74 import_worker_cls = CSCVSImportWorker
70 else:75 else:
7176
=== modified file 'versions.cfg'
--- versions.cfg 2011-08-03 13:05:12 +0000
+++ versions.cfg 2011-08-07 20:42:25 +0000
@@ -7,7 +7,7 @@
7ampoule = 0.2.07ampoule = 0.2.0
8amqplib = 0.6.18amqplib = 0.6.1
9BeautifulSoup = 3.1.0.19BeautifulSoup = 3.1.0.1
10bzr = 2.3.310bzr = 2.4b4-r6001
11chameleon.core = 1.0b3511chameleon.core = 1.0b35
12chameleon.zpt = 1.0b1712chameleon.zpt = 1.0b17
13ClientForm = 0.2.1013ClientForm = 0.2.10