Merge lp:~mwhudson/launchpad/incremental-code-imports-bug-512683 into lp:launchpad/db-devel
- incremental-code-imports-bug-512683
- Merge into db-devel
Status: | Merged |
---|---|
Approved by: | Michael Hudson-Doyle |
Approved revision: | not available |
Merged at revision: | not available |
Proposed branch: | lp:~mwhudson/launchpad/incremental-code-imports-bug-512683 |
Merge into: | lp:launchpad/db-devel |
Diff against target: |
504 lines (+174/-67) 13 files modified
lib/canonical/config/schema-lazr.conf (+4/-0) lib/canonical/launchpad/testing/codeimporthelpers.py (+2/-6) lib/lp/code/browser/branch.py (+2/-0) lib/lp/code/enums.py (+7/-0) lib/lp/code/model/codeimportjob.py (+4/-1) lib/lp/code/model/tests/test_codeimportjob.py (+15/-0) lib/lp/code/stories/codeimport/xx-codeimport-results.txt (+21/-16) lib/lp/codehosting/codeimport/tests/test_worker.py (+42/-8) lib/lp/codehosting/codeimport/tests/test_workermonitor.py (+15/-0) lib/lp/codehosting/codeimport/worker.py (+32/-9) lib/lp/codehosting/codeimport/workermonitor.py (+3/-0) lib/lp/testing/factory.py (+25/-25) utilities/sourcedeps.conf (+2/-2) |
To merge this branch: | bzr merge lp:~mwhudson/launchpad/incremental-code-imports-bug-512683 |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Tim Penhey (community) | Approve | ||
Review via email: mp+19674@code.launchpad.net |
Commit message
Support incremental imports for git -- repositories will be imported 1000 revisions at a time.
Description of the change
Michael Hudson-Doyle (mwhudson) wrote : | # |
Michael Hudson-Doyle (mwhudson) wrote : | # |
Oh, the new icon looks like this: http://
Do we need a UI review? I guess we probably should.
Tim Penhey (thumper) wrote : | # |
lib/lp/
> + if bazaar_
> + == foreign_
Our coding standard does suggest to use braces rather than line continuation
characters.
Or even:
last_revision = bazaar_
if (last_revision == foreign_
Everything else looks good!
merge approved
Michael Hudson-Doyle (mwhudson) wrote : | # |
Hi Tim,
This branch failed tests because some other tests (the distrobrancher ones, I think) failed when run with the bzr-git plugin loaded. So I added a layer that prevents any tests from being run in the same process after the code import worker tests. Can you review this approach please? The interdiff is here: http://
With this change, the branch passed all tests in ec2.
Cheers,
mwh
Tim Penhey (thumper) wrote : | # |
Add a comment about why this works, and land it. We can discuss the unplugability of the foreign branch bits at a later date.
Preview Diff
1 | === modified file 'lib/canonical/config/schema-lazr.conf' | |||
2 | --- lib/canonical/config/schema-lazr.conf 2010-02-20 13:16:38 +0000 | |||
3 | +++ lib/canonical/config/schema-lazr.conf 2010-02-22 05:40:42 +0000 | |||
4 | @@ -439,6 +439,10 @@ | |||
5 | 439 | # in a row. | 439 | # in a row. |
6 | 440 | consecutive_failure_limit: 5 | 440 | consecutive_failure_limit: 5 |
7 | 441 | 441 | ||
8 | 442 | # Import only this many revisions at once. | ||
9 | 443 | # Only applies to git imports for now. | ||
10 | 444 | revisions_import_limit: 1000 | ||
11 | 445 | |||
12 | 442 | 446 | ||
13 | 443 | [codeimportdispatcher] | 447 | [codeimportdispatcher] |
14 | 444 | # The directory where the code import worker should be directed to | 448 | # The directory where the code import worker should be directed to |
15 | 445 | 449 | ||
16 | === added file 'lib/canonical/launchpad/images/yes-gray.png' | |||
17 | 446 | Binary files lib/canonical/launchpad/images/yes-gray.png 1970-01-01 00:00:00 +0000 and lib/canonical/launchpad/images/yes-gray.png 2010-02-22 05:40:42 +0000 differ | 450 | Binary files lib/canonical/launchpad/images/yes-gray.png 1970-01-01 00:00:00 +0000 and lib/canonical/launchpad/images/yes-gray.png 2010-02-22 05:40:42 +0000 differ |
18 | === modified file 'lib/canonical/launchpad/testing/codeimporthelpers.py' | |||
19 | --- lib/canonical/launchpad/testing/codeimporthelpers.py 2009-06-25 05:30:52 +0000 | |||
20 | +++ lib/canonical/launchpad/testing/codeimporthelpers.py 2010-02-22 05:40:42 +0000 | |||
21 | @@ -119,17 +119,13 @@ | |||
22 | 119 | return code_import | 119 | return code_import |
23 | 120 | 120 | ||
24 | 121 | 121 | ||
26 | 122 | def make_all_result_types(code_import=None, factory=None, machine=None): | 122 | def make_all_result_types(code_import, factory, machine, start, count): |
27 | 123 | """Make a code import result of each possible type for the code import.""" | 123 | """Make a code import result of each possible type for the code import.""" |
28 | 124 | if factory is None: | ||
29 | 125 | factory = LaunchpadObjectFactory() | ||
30 | 126 | if code_import is None: | ||
31 | 127 | code_import = factory.makeCodeImport() | ||
32 | 128 | start_dates = time_counter( | 124 | start_dates = time_counter( |
33 | 129 | datetime(2007,12,1,12, tzinfo=UTC), timedelta(days=1)) | 125 | datetime(2007,12,1,12, tzinfo=UTC), timedelta(days=1)) |
34 | 130 | end_dates = time_counter( | 126 | end_dates = time_counter( |
35 | 131 | datetime(2007,12,1,13, tzinfo=UTC), timedelta(days=1, hours=1)) | 127 | datetime(2007,12,1,13, tzinfo=UTC), timedelta(days=1, hours=1)) |
37 | 132 | for result_status in CodeImportResultStatus.items: | 128 | for result_status in sorted(CodeImportResultStatus.items)[start:start+count]: |
38 | 133 | factory.makeCodeImportResult( | 129 | factory.makeCodeImportResult( |
39 | 134 | code_import, result_status, start_dates.next(), end_dates.next(), | 130 | code_import, result_status, start_dates.next(), end_dates.next(), |
40 | 135 | machine=machine) | 131 | machine=machine) |
41 | 136 | 132 | ||
42 | === modified file 'lib/lp/code/browser/branch.py' | |||
43 | --- lib/lp/code/browser/branch.py 2010-02-01 03:49:23 +0000 | |||
44 | +++ lib/lp/code/browser/branch.py 2010-02-22 05:40:42 +0000 | |||
45 | @@ -514,6 +514,8 @@ | |||
46 | 514 | """The icon to represent the `CodeImportResultStatus` `status`.""" | 514 | """The icon to represent the `CodeImportResultStatus` `status`.""" |
47 | 515 | if status in CodeImportResultStatus.successes: | 515 | if status in CodeImportResultStatus.successes: |
48 | 516 | return "/@@/yes" | 516 | return "/@@/yes" |
49 | 517 | elif status == CodeImportResultStatus.SUCCESS_PARTIAL: | ||
50 | 518 | return "/@@/yes-gray" | ||
51 | 517 | else: | 519 | else: |
52 | 518 | return "/@@/no" | 520 | return "/@@/no" |
53 | 519 | 521 | ||
54 | 520 | 522 | ||
55 | === modified file 'lib/lp/code/enums.py' | |||
56 | --- lib/lp/code/enums.py 2010-02-01 03:49:23 +0000 | |||
57 | +++ lib/lp/code/enums.py 2010-02-22 05:40:42 +0000 | |||
58 | @@ -806,6 +806,13 @@ | |||
59 | 806 | import. | 806 | import. |
60 | 807 | """) | 807 | """) |
61 | 808 | 808 | ||
62 | 809 | SUCCESS_PARTIAL = DBItem(120, """ | ||
63 | 810 | Partial Success | ||
64 | 811 | |||
65 | 812 | Import job successfully imported some but not all of the foreign | ||
66 | 813 | revisions. | ||
67 | 814 | """) | ||
68 | 815 | |||
69 | 809 | FAILURE = DBItem(200, """ | 816 | FAILURE = DBItem(200, """ |
70 | 810 | Failure | 817 | Failure |
71 | 811 | 818 | ||
72 | 812 | 819 | ||
73 | === modified file 'lib/lp/code/model/codeimportjob.py' | |||
74 | --- lib/lp/code/model/codeimportjob.py 2010-01-27 02:48:13 +0000 | |||
75 | +++ lib/lp/code/model/codeimportjob.py 2010-02-22 05:40:42 +0000 | |||
76 | @@ -286,7 +286,10 @@ | |||
77 | 286 | dict(review_status=CodeImportReviewStatus.FAILING), None) | 286 | dict(review_status=CodeImportReviewStatus.FAILING), None) |
78 | 287 | # Only start a new one if the import is still in the REVIEWED state. | 287 | # Only start a new one if the import is still in the REVIEWED state. |
79 | 288 | if code_import.review_status == CodeImportReviewStatus.REVIEWED: | 288 | if code_import.review_status == CodeImportReviewStatus.REVIEWED: |
81 | 289 | self.newJob(code_import) | 289 | extra = {} |
82 | 290 | if status == CodeImportResultStatus.SUCCESS_PARTIAL: | ||
83 | 291 | extra['date_due'] = UTC_NOW | ||
84 | 292 | self.newJob(code_import, **extra) | ||
85 | 290 | # If the status was successful, update date_last_successful. | 293 | # If the status was successful, update date_last_successful. |
86 | 291 | if status in [CodeImportResultStatus.SUCCESS, | 294 | if status in [CodeImportResultStatus.SUCCESS, |
87 | 292 | CodeImportResultStatus.SUCCESS_NOCHANGE]: | 295 | CodeImportResultStatus.SUCCESS_NOCHANGE]: |
88 | 293 | 296 | ||
89 | === modified file 'lib/lp/code/model/tests/test_codeimportjob.py' | |||
90 | --- lib/lp/code/model/tests/test_codeimportjob.py 2010-02-01 03:55:59 +0000 | |||
91 | +++ lib/lp/code/model/tests/test_codeimportjob.py 2010-02-22 05:40:42 +0000 | |||
92 | @@ -776,6 +776,21 @@ | |||
93 | 776 | new_job.date_due - running_job.date_due, | 776 | new_job.date_due - running_job.date_due, |
94 | 777 | code_import.effective_update_interval) | 777 | code_import.effective_update_interval) |
95 | 778 | 778 | ||
96 | 779 | def test_partialSuccessCreatesNewJobDueNow(self): | ||
97 | 780 | # If called with a status of SUCCESS_PARTIAL, finishJob() creates a | ||
98 | 781 | # new CodeImportJob for the given CodeImport that is due to run right | ||
99 | 782 | # now. | ||
100 | 783 | running_job = self.makeRunningJob() | ||
101 | 784 | code_import = running_job.code_import | ||
102 | 785 | self.switchDbUser() | ||
103 | 786 | getUtility(ICodeImportJobWorkflow).finishJob( | ||
104 | 787 | running_job, CodeImportResultStatus.SUCCESS_PARTIAL, None) | ||
105 | 788 | new_job = code_import.import_job | ||
106 | 789 | self.assert_(new_job is not None) | ||
107 | 790 | self.assertEqual(new_job.state, CodeImportJobState.PENDING) | ||
108 | 791 | self.assertEqual(new_job.machine, None) | ||
109 | 792 | self.assertSqlAttributeEqualsDate(new_job, 'date_due', UTC_NOW) | ||
110 | 793 | |||
111 | 779 | def test_doesntCreateNewJobIfCodeImportNotReviewed(self): | 794 | def test_doesntCreateNewJobIfCodeImportNotReviewed(self): |
112 | 780 | # finishJob() creates a new CodeImportJob for the given CodeImport, | 795 | # finishJob() creates a new CodeImportJob for the given CodeImport, |
113 | 781 | # unless the CodeImport has been suspended or marked invalid. | 796 | # unless the CodeImport has been suspended or marked invalid. |
114 | 782 | 797 | ||
115 | === modified file 'lib/lp/code/stories/codeimport/xx-codeimport-results.txt' | |||
116 | --- lib/lp/code/stories/codeimport/xx-codeimport-results.txt 2010-01-27 03:49:06 +0000 | |||
117 | +++ lib/lp/code/stories/codeimport/xx-codeimport-results.txt 2010-02-22 05:40:42 +0000 | |||
118 | @@ -6,29 +6,28 @@ | |||
119 | 6 | >>> login('test@canonical.com') | 6 | >>> login('test@canonical.com') |
120 | 7 | >>> from canonical.launchpad.testing.codeimporthelpers import ( | 7 | >>> from canonical.launchpad.testing.codeimporthelpers import ( |
121 | 8 | ... make_all_result_types) | 8 | ... make_all_result_types) |
123 | 9 | >>> code_import = factory.makeCodeImport() | 9 | >>> code_import_1 = factory.makeCodeImport() |
124 | 10 | >>> code_import_2 = factory.makeCodeImport() | ||
125 | 10 | 11 | ||
126 | 11 | The make_all_result_types helper method adds a code import result of | 12 | The make_all_result_types helper method adds a code import result of |
128 | 12 | each possible status value. | 13 | each possible status value. There are more status values than are |
129 | 14 | shown on the branch page, so we create two imports in order to test | ||
130 | 15 | how each result type is rendered. | ||
131 | 13 | 16 | ||
132 | 14 | >>> odin = factory.makeCodeImportMachine(hostname='odin') | 17 | >>> odin = factory.makeCodeImportMachine(hostname='odin') |
135 | 15 | >>> make_all_result_types(code_import, factory, machine=odin) | 18 | >>> make_all_result_types(code_import_1, factory, machine=odin, start=0, count=7) |
136 | 16 | >>> branch_url = canonical_url(code_import.branch) | 19 | >>> branch_url_1 = canonical_url(code_import_1.branch) |
137 | 20 | >>> make_all_result_types(code_import_2, factory, machine=odin, start=7, count=7) | ||
138 | 21 | >>> branch_url_2 = canonical_url(code_import_2.branch) | ||
139 | 17 | >>> logout() | 22 | >>> logout() |
140 | 18 | 23 | ||
141 | 19 | For each import result, the start date and finish date is shown along | 24 | For each import result, the start date and finish date is shown along |
142 | 20 | with the duration. A link to the log file is shown if there was a log | 25 | with the duration. A link to the log file is shown if there was a log |
143 | 21 | file stored with the result. | 26 | file stored with the result. |
144 | 22 | 27 | ||
146 | 23 | >>> browser.open(branch_url) | 28 | >>> browser.open(branch_url_1) |
147 | 24 | >>> import_results = find_tag_by_id(browser.contents, 'import-results') | 29 | >>> import_results = find_tag_by_id(browser.contents, 'import-results') |
148 | 25 | >>> print extract_text(import_results).replace('—', '--') | 30 | >>> print extract_text(import_results).replace('—', '--') |
149 | 26 | Import started on 2007-12-10 on odin and finished on 2007-12-10 | ||
150 | 27 | taking ten hours -- see the log | ||
151 | 28 | Import started on 2007-12-09 on odin and finished on 2007-12-09 | ||
152 | 29 | taking nine hours -- see the log | ||
153 | 30 | Import started on 2007-12-08 on odin and finished on 2007-12-08 | ||
154 | 31 | taking eight hours -- see the log | ||
155 | 32 | Import started on 2007-12-07 on odin and finished on 2007-12-07 | 31 | Import started on 2007-12-07 on odin and finished on 2007-12-07 |
156 | 33 | taking seven hours -- see the log | 32 | taking seven hours -- see the log |
157 | 34 | Import started on 2007-12-06 on odin and finished on 2007-12-06 | 33 | Import started on 2007-12-06 on odin and finished on 2007-12-06 |
158 | @@ -46,19 +45,25 @@ | |||
159 | 46 | 45 | ||
160 | 47 | Each of the lines is prefixed with a tick if the result status was | 46 | Each of the lines is prefixed with a tick if the result status was |
161 | 48 | success, or a cross if the status was a failure. The title of the image | 47 | success, or a cross if the status was a failure. The title of the image |
163 | 49 | is the text of the failure type. | 48 | is the text of the failure or success type. |
164 | 50 | 49 | ||
165 | 51 | >>> # The ordering here is dependant on the order the status values | 50 | >>> # The ordering here is dependant on the order the status values |
166 | 52 | >>> # are declared in the enumeration. | 51 | >>> # are declared in the enumeration. |
167 | 53 | >>> for img in import_results.fetch('img'): | 52 | >>> for img in import_results.fetch('img'): |
168 | 54 | ... print img | 53 | ... print img |
169 | 55 | <img src="/@@/no" title="Job killed" /> | ||
170 | 56 | <img src="/@@/no" title="Job reclaimed" /> | ||
171 | 57 | <img src="/@@/no" title="Bazaar Update Failed" /> | ||
172 | 58 | <img src="/@@/no" title="Source Update Failed" /> | ||
173 | 59 | <img src="/@@/no" title="Bazaar Import Failed" /> | 54 | <img src="/@@/no" title="Bazaar Import Failed" /> |
174 | 60 | <img src="/@@/no" title="Source Checkout Failed" /> | 55 | <img src="/@@/no" title="Source Checkout Failed" /> |
175 | 61 | <img src="/@@/no" title="Internal Failure" /> | 56 | <img src="/@@/no" title="Internal Failure" /> |
176 | 62 | <img src="/@@/no" title="Failure" /> | 57 | <img src="/@@/no" title="Failure" /> |
177 | 58 | <img src="/@@/yes-gray" title="Partial Success" /> | ||
178 | 63 | <img src="/@@/yes" title="Success with no changes" /> | 59 | <img src="/@@/yes" title="Success with no changes" /> |
179 | 64 | <img src="/@@/yes" title="Success" /> | 60 | <img src="/@@/yes" title="Success" /> |
180 | 61 | |||
181 | 62 | >>> browser.open(branch_url_2) | ||
182 | 63 | >>> import_results = find_tag_by_id(browser.contents, 'import-results') | ||
183 | 64 | >>> for img in import_results.fetch('img'): | ||
184 | 65 | ... print img | ||
185 | 66 | <img src="/@@/no" title="Job killed" /> | ||
186 | 67 | <img src="/@@/no" title="Job reclaimed" /> | ||
187 | 68 | <img src="/@@/no" title="Bazaar Update Failed" /> | ||
188 | 69 | <img src="/@@/no" title="Source Update Failed" /> | ||
189 | 65 | 70 | ||
190 | === modified file 'lib/lp/codehosting/codeimport/tests/test_worker.py' | |||
191 | --- lib/lp/codehosting/codeimport/tests/test_worker.py 2010-02-03 19:29:27 +0000 | |||
192 | +++ lib/lp/codehosting/codeimport/tests/test_worker.py 2010-02-22 05:40:42 +0000 | |||
193 | @@ -23,7 +23,6 @@ | |||
194 | 23 | 23 | ||
195 | 24 | from CVS import Repository, tree as CVSTree | 24 | from CVS import Repository, tree as CVSTree |
196 | 25 | 25 | ||
197 | 26 | from canonical.cachedproperty import cachedproperty | ||
198 | 27 | from canonical.config import config | 26 | from canonical.config import config |
199 | 28 | from canonical.launchpad.scripts.logger import QuietFakeLogger | 27 | from canonical.launchpad.scripts.logger import QuietFakeLogger |
200 | 29 | from canonical.testing import BaseLayer | 28 | from canonical.testing import BaseLayer |
201 | @@ -37,22 +36,46 @@ | |||
202 | 37 | CVSServer, GitServer, MercurialServer, SubversionServer) | 36 | CVSServer, GitServer, MercurialServer, SubversionServer) |
203 | 38 | from lp.codehosting.tests.helpers import ( | 37 | from lp.codehosting.tests.helpers import ( |
204 | 39 | create_branch_with_one_revision) | 38 | create_branch_with_one_revision) |
206 | 40 | from lp.testing.factory import LaunchpadObjectFactory | 39 | from lp.testing import TestCase |
207 | 41 | 40 | ||
208 | 42 | import pysvn | 41 | import pysvn |
209 | 43 | 42 | ||
210 | 44 | 43 | ||
211 | 44 | class ForeignBranchPluginLayer(BaseLayer): | ||
212 | 45 | """Ensure only specific tests are run with foreign branch plugins loaded. | ||
213 | 46 | """ | ||
214 | 47 | |||
215 | 48 | @classmethod | ||
216 | 49 | def setUp(cls): | ||
217 | 50 | pass | ||
218 | 51 | |||
219 | 52 | @classmethod | ||
220 | 53 | def tearDown(cls): | ||
221 | 54 | # Raise NotImplementedError to signal that this layer cannot be torn | ||
222 | 55 | # down. This means that the test runner will run subsequent tests in | ||
223 | 56 | # a different process. | ||
224 | 57 | raise NotImplementedError | ||
225 | 58 | |||
226 | 59 | @classmethod | ||
227 | 60 | def testSetUp(cls): | ||
228 | 61 | pass | ||
229 | 62 | |||
230 | 63 | @classmethod | ||
231 | 64 | def testTearDown(cls): | ||
232 | 65 | pass | ||
233 | 66 | |||
234 | 67 | |||
235 | 45 | default_format = BzrDirFormat.get_default_format() | 68 | default_format = BzrDirFormat.get_default_format() |
236 | 46 | 69 | ||
237 | 47 | 70 | ||
239 | 48 | class WorkerTest(TestCaseWithTransport): | 71 | class WorkerTest(TestCaseWithTransport, TestCase): |
240 | 49 | """Base test case for things that test the code import worker. | 72 | """Base test case for things that test the code import worker. |
241 | 50 | 73 | ||
242 | 51 | Provides Bazaar testing features, access to Launchpad objects and | 74 | Provides Bazaar testing features, access to Launchpad objects and |
243 | 52 | factories for some code import objects. | 75 | factories for some code import objects. |
244 | 53 | """ | 76 | """ |
245 | 54 | 77 | ||
247 | 55 | layer = BaseLayer | 78 | layer = ForeignBranchPluginLayer |
248 | 56 | 79 | ||
249 | 57 | def setUp(self): | 80 | def setUp(self): |
250 | 58 | TestCaseWithTransport.setUp(self) | 81 | TestCaseWithTransport.setUp(self) |
251 | @@ -70,10 +93,6 @@ | |||
252 | 70 | self.assertEqual( | 93 | self.assertEqual( |
253 | 71 | sorted(list_files(directory1)), sorted(list_files(directory2))) | 94 | sorted(list_files(directory1)), sorted(list_files(directory2))) |
254 | 72 | 95 | ||
255 | 73 | @cachedproperty | ||
256 | 74 | def factory(self): | ||
257 | 75 | return LaunchpadObjectFactory() | ||
258 | 76 | |||
259 | 77 | def makeTemporaryDirectory(self): | 96 | def makeTemporaryDirectory(self): |
260 | 78 | directory = tempfile.mkdtemp() | 97 | directory = tempfile.mkdtemp() |
261 | 79 | self.addCleanup(shutil.rmtree, directory) | 98 | self.addCleanup(shutil.rmtree, directory) |
262 | @@ -906,6 +925,21 @@ | |||
263 | 906 | return self.factory.makeCodeImportSourceDetails( | 925 | return self.factory.makeCodeImportSourceDetails( |
264 | 907 | rcstype='git', url=repository_path) | 926 | rcstype='git', url=repository_path) |
265 | 908 | 927 | ||
266 | 928 | def test_partial(self): | ||
267 | 929 | # Only config.codeimport.revisions_import_limit will be imported in a | ||
268 | 930 | # given run. When bzr-svn and bzr-hg support revision import limits, | ||
269 | 931 | # this test case can be moved up to PullingImportWorkerTests. | ||
270 | 932 | worker = self.makeImportWorker(self.makeSourceDetails( | ||
271 | 933 | 'trunk', [('README', 'Original contents')])) | ||
272 | 934 | self.makeForeignCommit(worker.source_details) | ||
273 | 935 | self.assertTrue(self.foreign_commit_count > 1) | ||
274 | 936 | self.pushConfig( | ||
275 | 937 | 'codeimport', revisions_import_limit=self.foreign_commit_count-1) | ||
276 | 938 | self.assertEqual( | ||
277 | 939 | CodeImportWorkerExitCode.SUCCESS_PARTIAL, worker.run()) | ||
278 | 940 | self.assertEqual( | ||
279 | 941 | CodeImportWorkerExitCode.SUCCESS, worker.run()) | ||
280 | 942 | |||
281 | 909 | 943 | ||
282 | 910 | class TestMercurialImport(WorkerTest, TestActualImportMixin, | 944 | class TestMercurialImport(WorkerTest, TestActualImportMixin, |
283 | 911 | PullingImportWorkerTests): | 945 | PullingImportWorkerTests): |
284 | 912 | 946 | ||
285 | === modified file 'lib/lp/codehosting/codeimport/tests/test_workermonitor.py' | |||
286 | --- lib/lp/codehosting/codeimport/tests/test_workermonitor.py 2010-02-03 19:29:27 +0000 | |||
287 | +++ lib/lp/codehosting/codeimport/tests/test_workermonitor.py 2010-02-22 05:40:42 +0000 | |||
288 | @@ -334,6 +334,21 @@ | |||
289 | 334 | # callFinishJob did not swallow the error, this will fail the test. | 334 | # callFinishJob did not swallow the error, this will fail the test. |
290 | 335 | return ret | 335 | return ret |
291 | 336 | 336 | ||
292 | 337 | def test_callFinishJobCallsFinishJobPartial(self): | ||
293 | 338 | # If the argument to callFinishJob indicates that the subprocess | ||
294 | 339 | # exited with a code of CodeImportWorkerExitCode.SUCCESS_PARTIAL, it | ||
295 | 340 | # calls finishJob with a status of SUCCESS_PARTIAL. | ||
296 | 341 | calls = self.patchOutFinishJob() | ||
297 | 342 | ret = self.worker_monitor.callFinishJob( | ||
298 | 343 | makeFailure( | ||
299 | 344 | error.ProcessTerminated, | ||
300 | 345 | exitCode=CodeImportWorkerExitCode.SUCCESS_PARTIAL)) | ||
301 | 346 | self.assertEqual(calls, [CodeImportResultStatus.SUCCESS_PARTIAL]) | ||
302 | 347 | self.assertOopsesLogged([]) | ||
303 | 348 | # We return the deferred that callFinishJob returns -- if | ||
304 | 349 | # callFinishJob did not swallow the error, this will fail the test. | ||
305 | 350 | return ret | ||
306 | 351 | |||
307 | 337 | def test_callFinishJobLogsTracebackOnFailure(self): | 352 | def test_callFinishJobLogsTracebackOnFailure(self): |
308 | 338 | # When callFinishJob is called with a failure, it dumps the traceback | 353 | # When callFinishJob is called with a failure, it dumps the traceback |
309 | 339 | # of the failure into the log file. | 354 | # of the failure into the log file. |
310 | 340 | 355 | ||
311 | === modified file 'lib/lp/codehosting/codeimport/worker.py' | |||
312 | --- lib/lp/codehosting/codeimport/worker.py 2010-02-01 04:26:12 +0000 | |||
313 | +++ lib/lp/codehosting/codeimport/worker.py 2010-02-22 05:40:42 +0000 | |||
314 | @@ -21,7 +21,7 @@ | |||
315 | 21 | import os | 21 | import os |
316 | 22 | import shutil | 22 | import shutil |
317 | 23 | 23 | ||
319 | 24 | from bzrlib.branch import Branch | 24 | from bzrlib.branch import Branch, InterBranch |
320 | 25 | from bzrlib.bzrdir import BzrDir, BzrDirFormat | 25 | from bzrlib.bzrdir import BzrDir, BzrDirFormat |
321 | 26 | from bzrlib.transport import get_transport | 26 | from bzrlib.transport import get_transport |
322 | 27 | from bzrlib.errors import NoSuchFile, NotBranchError | 27 | from bzrlib.errors import NoSuchFile, NotBranchError |
323 | @@ -50,6 +50,7 @@ | |||
324 | 50 | SUCCESS = 0 | 50 | SUCCESS = 0 |
325 | 51 | FAILURE = 1 | 51 | FAILURE = 1 |
326 | 52 | SUCCESS_NOCHANGE = 2 | 52 | SUCCESS_NOCHANGE = 2 |
327 | 53 | SUCCESS_PARTIAL = 3 | ||
328 | 53 | 54 | ||
329 | 54 | 55 | ||
330 | 55 | class BazaarBranchStore: | 56 | class BazaarBranchStore: |
331 | @@ -407,11 +408,7 @@ | |||
332 | 407 | saved_pwd = os.getcwd() | 408 | saved_pwd = os.getcwd() |
333 | 408 | os.chdir(working_directory) | 409 | os.chdir(working_directory) |
334 | 409 | try: | 410 | try: |
340 | 410 | non_trivial = self._doImport() | 411 | return self._doImport() |
336 | 411 | if non_trivial: | ||
337 | 412 | return CodeImportWorkerExitCode.SUCCESS | ||
338 | 413 | else: | ||
339 | 414 | return CodeImportWorkerExitCode.SUCCESS_NOCHANGE | ||
341 | 415 | finally: | 412 | finally: |
342 | 416 | shutil.rmtree(working_directory) | 413 | shutil.rmtree(working_directory) |
343 | 417 | os.chdir(saved_pwd) | 414 | os.chdir(saved_pwd) |
344 | @@ -497,7 +494,10 @@ | |||
345 | 497 | self.importToBazaar(foreign_tree, bazaar_tree) | 494 | self.importToBazaar(foreign_tree, bazaar_tree) |
346 | 498 | non_trivial = self.pushBazaarWorkingTree(bazaar_tree) | 495 | non_trivial = self.pushBazaarWorkingTree(bazaar_tree) |
347 | 499 | self.foreign_tree_store.archive(foreign_tree) | 496 | self.foreign_tree_store.archive(foreign_tree) |
349 | 500 | return non_trivial | 497 | if non_trivial: |
350 | 498 | return CodeImportWorkerExitCode.SUCCESS | ||
351 | 499 | else: | ||
352 | 500 | return CodeImportWorkerExitCode.SUCCESS_NOCHANGE | ||
353 | 501 | 501 | ||
354 | 502 | 502 | ||
355 | 503 | class PullingImportWorker(ImportWorker): | 503 | class PullingImportWorker(ImportWorker): |
356 | @@ -511,6 +511,15 @@ | |||
357 | 511 | """The format classes that should be tried for this import.""" | 511 | """The format classes that should be tried for this import.""" |
358 | 512 | raise NotImplementedError | 512 | raise NotImplementedError |
359 | 513 | 513 | ||
360 | 514 | def getExtraPullArgs(self): | ||
361 | 515 | """Return extra arguments to `InterBranch.pull`. | ||
362 | 516 | |||
363 | 517 | This method only really exists because only bzr-git supports the | ||
364 | 518 | 'limit' argument to this method. When bzr-svn and bzr-hg plugin do | ||
365 | 519 | too, this method can go away. | ||
366 | 520 | """ | ||
367 | 521 | return {} | ||
368 | 522 | |||
369 | 514 | def _doImport(self): | 523 | def _doImport(self): |
370 | 515 | bazaar_tree = self.getBazaarWorkingTree() | 524 | bazaar_tree = self.getBazaarWorkingTree() |
371 | 516 | self.bazaar_branch_store.push( | 525 | self.bazaar_branch_store.push( |
372 | @@ -529,10 +538,20 @@ | |||
373 | 529 | else: | 538 | else: |
374 | 530 | raise NotBranchError(self.source_details.url) | 539 | raise NotBranchError(self.source_details.url) |
375 | 531 | foreign_branch = format.open(transport).open_branch() | 540 | foreign_branch = format.open(transport).open_branch() |
377 | 532 | bazaar_tree.branch.pull(foreign_branch, overwrite=True) | 541 | inter_branch = InterBranch.get(foreign_branch, bazaar_tree.branch) |
378 | 542 | pull_result = inter_branch.pull( | ||
379 | 543 | overwrite=True, **self.getExtraPullArgs()) | ||
380 | 544 | self.pushBazaarWorkingTree(bazaar_tree) | ||
381 | 545 | last_imported_revison = bazaar_tree.branch.last_revision() | ||
382 | 546 | if last_imported_revison == foreign_branch.last_revision(): | ||
383 | 547 | if pull_result.old_revid != pull_result.new_revid: | ||
384 | 548 | return CodeImportWorkerExitCode.SUCCESS | ||
385 | 549 | else: | ||
386 | 550 | return CodeImportWorkerExitCode.SUCCESS_NOCHANGE | ||
387 | 551 | else: | ||
388 | 552 | return CodeImportWorkerExitCode.SUCCESS_PARTIAL | ||
389 | 533 | finally: | 553 | finally: |
390 | 534 | bzrlib.ui.ui_factory = saved_factory | 554 | bzrlib.ui.ui_factory = saved_factory |
391 | 535 | return self.pushBazaarWorkingTree(bazaar_tree) | ||
392 | 536 | 555 | ||
393 | 537 | 556 | ||
394 | 538 | class GitImportWorker(PullingImportWorker): | 557 | class GitImportWorker(PullingImportWorker): |
395 | @@ -549,6 +568,10 @@ | |||
396 | 549 | LocalGitBzrDirFormat, RemoteGitBzrDirFormat) | 568 | LocalGitBzrDirFormat, RemoteGitBzrDirFormat) |
397 | 550 | return [LocalGitBzrDirFormat, RemoteGitBzrDirFormat] | 569 | return [LocalGitBzrDirFormat, RemoteGitBzrDirFormat] |
398 | 551 | 570 | ||
399 | 571 | def getExtraPullArgs(self): | ||
400 | 572 | """See `PullingImportWorker.getExtraPullArgs`.""" | ||
401 | 573 | return {'limit': config.codeimport.revisions_import_limit} | ||
402 | 574 | |||
403 | 552 | def getBazaarWorkingTree(self): | 575 | def getBazaarWorkingTree(self): |
404 | 553 | """See `ImportWorker.getBazaarWorkingTree`. | 576 | """See `ImportWorker.getBazaarWorkingTree`. |
405 | 554 | 577 | ||
406 | 555 | 578 | ||
407 | === modified file 'lib/lp/codehosting/codeimport/workermonitor.py' | |||
408 | --- lib/lp/codehosting/codeimport/workermonitor.py 2010-02-01 03:49:23 +0000 | |||
409 | +++ lib/lp/codehosting/codeimport/workermonitor.py 2010-02-22 05:40:42 +0000 | |||
410 | @@ -302,6 +302,9 @@ | |||
411 | 302 | if reason.value.exitCode == \ | 302 | if reason.value.exitCode == \ |
412 | 303 | CodeImportWorkerExitCode.SUCCESS_NOCHANGE: | 303 | CodeImportWorkerExitCode.SUCCESS_NOCHANGE: |
413 | 304 | return CodeImportResultStatus.SUCCESS_NOCHANGE | 304 | return CodeImportResultStatus.SUCCESS_NOCHANGE |
414 | 305 | elif reason.value.exitCode == \ | ||
415 | 306 | CodeImportWorkerExitCode.SUCCESS_PARTIAL: | ||
416 | 307 | return CodeImportResultStatus.SUCCESS_PARTIAL | ||
417 | 305 | return CodeImportResultStatus.FAILURE | 308 | return CodeImportResultStatus.FAILURE |
418 | 306 | else: | 309 | else: |
419 | 307 | return CodeImportResultStatus.SUCCESS | 310 | return CodeImportResultStatus.SUCCESS |
420 | 308 | 311 | ||
421 | === modified file 'lib/lp/testing/factory.py' | |||
422 | --- lib/lp/testing/factory.py 2010-02-20 13:16:38 +0000 | |||
423 | +++ lib/lp/testing/factory.py 2010-02-22 05:40:42 +0000 | |||
424 | @@ -255,6 +255,31 @@ | |||
425 | 255 | host = "%s.domain.com" % self.getUniqueString('domain') | 255 | host = "%s.domain.com" % self.getUniqueString('domain') |
426 | 256 | return '%s://%s/%s' % (scheme, host, self.getUniqueString('path')) | 256 | return '%s://%s/%s' % (scheme, host, self.getUniqueString('path')) |
427 | 257 | 257 | ||
428 | 258 | def makeCodeImportSourceDetails(self, branch_id=None, rcstype=None, | ||
429 | 259 | url=None, cvs_root=None, cvs_module=None): | ||
430 | 260 | if branch_id is None: | ||
431 | 261 | branch_id = self.getUniqueInteger() | ||
432 | 262 | if rcstype is None: | ||
433 | 263 | rcstype = 'svn' | ||
434 | 264 | if rcstype in ['svn', 'bzr-svn', 'hg']: | ||
435 | 265 | assert cvs_root is cvs_module is None | ||
436 | 266 | if url is None: | ||
437 | 267 | url = self.getUniqueURL() | ||
438 | 268 | elif rcstype == 'cvs': | ||
439 | 269 | assert url is None | ||
440 | 270 | if cvs_root is None: | ||
441 | 271 | cvs_root = self.getUniqueString() | ||
442 | 272 | if cvs_module is None: | ||
443 | 273 | cvs_module = self.getUniqueString() | ||
444 | 274 | elif rcstype == 'git': | ||
445 | 275 | assert cvs_root is cvs_module is None | ||
446 | 276 | if url is None: | ||
447 | 277 | url = self.getUniqueURL(scheme='git') | ||
448 | 278 | else: | ||
449 | 279 | raise AssertionError("Unknown rcstype %r." % rcstype) | ||
450 | 280 | return CodeImportSourceDetails( | ||
451 | 281 | branch_id, rcstype, url, cvs_root, cvs_module) | ||
452 | 282 | |||
453 | 258 | 283 | ||
454 | 259 | class LaunchpadObjectFactory(ObjectFactory): | 284 | class LaunchpadObjectFactory(ObjectFactory): |
455 | 260 | """Factory methods for creating Launchpad objects. | 285 | """Factory methods for creating Launchpad objects. |
456 | @@ -1409,31 +1434,6 @@ | |||
457 | 1409 | code_import, machine, requesting_user, log_excerpt, log_alias, | 1434 | code_import, machine, requesting_user, log_excerpt, log_alias, |
458 | 1410 | result_status, date_started, date_finished) | 1435 | result_status, date_started, date_finished) |
459 | 1411 | 1436 | ||
460 | 1412 | def makeCodeImportSourceDetails(self, branch_id=None, rcstype=None, | ||
461 | 1413 | url=None, cvs_root=None, cvs_module=None): | ||
462 | 1414 | if branch_id is None: | ||
463 | 1415 | branch_id = self.getUniqueInteger() | ||
464 | 1416 | if rcstype is None: | ||
465 | 1417 | rcstype = 'svn' | ||
466 | 1418 | if rcstype in ['svn', 'bzr-svn', 'hg']: | ||
467 | 1419 | assert cvs_root is cvs_module is None | ||
468 | 1420 | if url is None: | ||
469 | 1421 | url = self.getUniqueURL() | ||
470 | 1422 | elif rcstype == 'cvs': | ||
471 | 1423 | assert url is None | ||
472 | 1424 | if cvs_root is None: | ||
473 | 1425 | cvs_root = self.getUniqueString() | ||
474 | 1426 | if cvs_module is None: | ||
475 | 1427 | cvs_module = self.getUniqueString() | ||
476 | 1428 | elif rcstype == 'git': | ||
477 | 1429 | assert cvs_root is cvs_module is None | ||
478 | 1430 | if url is None: | ||
479 | 1431 | url = self.getUniqueURL(scheme='git') | ||
480 | 1432 | else: | ||
481 | 1433 | raise AssertionError("Unknown rcstype %r." % rcstype) | ||
482 | 1434 | return CodeImportSourceDetails( | ||
483 | 1435 | branch_id, rcstype, url, cvs_root, cvs_module) | ||
484 | 1436 | |||
485 | 1437 | def makeCodeReviewComment(self, sender=None, subject=None, body=None, | 1437 | def makeCodeReviewComment(self, sender=None, subject=None, body=None, |
486 | 1438 | vote=None, vote_tag=None, parent=None, | 1438 | vote=None, vote_tag=None, parent=None, |
487 | 1439 | merge_proposal=None): | 1439 | merge_proposal=None): |
488 | 1440 | 1440 | ||
489 | === modified file 'utilities/sourcedeps.conf' | |||
490 | --- utilities/sourcedeps.conf 2010-02-18 13:53:20 +0000 | |||
491 | +++ utilities/sourcedeps.conf 2010-02-22 05:40:42 +0000 | |||
492 | @@ -1,10 +1,10 @@ | |||
493 | 1 | bzr-builder lp:~launchpad-pqm/bzr-builder/trunk;revno=63 | 1 | bzr-builder lp:~launchpad-pqm/bzr-builder/trunk;revno=63 |
495 | 2 | bzr-git lp:~launchpad-pqm/bzr-git/devel;revno=249 | 2 | bzr-git lp:~launchpad-pqm/bzr-git/devel;revno=250 |
496 | 3 | bzr-hg lp:~launchpad-pqm/bzr-hg/devel;revno=281 | 3 | bzr-hg lp:~launchpad-pqm/bzr-hg/devel;revno=281 |
497 | 4 | bzr-loom lp:~launchpad-pqm/bzr-loom/trunk;revno=47 | 4 | bzr-loom lp:~launchpad-pqm/bzr-loom/trunk;revno=47 |
498 | 5 | bzr-svn lp:~launchpad-pqm/bzr-svn/devel;revno=2707 | 5 | bzr-svn lp:~launchpad-pqm/bzr-svn/devel;revno=2707 |
499 | 6 | cscvs lp:~launchpad-pqm/launchpad-cscvs/devel;revno=430 | 6 | cscvs lp:~launchpad-pqm/launchpad-cscvs/devel;revno=430 |
501 | 7 | dulwich lp:~launchpad-pqm/dulwich/devel;revno=415 | 7 | dulwich lp:~launchpad-pqm/dulwich/devel;revno=416 |
502 | 8 | launchpad-loggerhead lp:~launchpad-pqm/launchpad-loggerhead/devel;revno=54 | 8 | launchpad-loggerhead lp:~launchpad-pqm/launchpad-loggerhead/devel;revno=54 |
503 | 9 | loggerhead lp:~launchpad-pqm/loggerhead/devel;revno=174 | 9 | loggerhead lp:~launchpad-pqm/loggerhead/devel;revno=174 |
504 | 10 | lpreview lp:~launchpad-pqm/bzr-lpreview/devel;revno=23 | 10 | lpreview lp:~launchpad-pqm/bzr-lpreview/devel;revno=23 |
Hi there,
This branch adds support to incrementally pull 5000 revisions at a time for git imports.
Not much more to say really, I hope the implementation mostly makes sense (at least if you know this area of the code).
Cheers,
mwh