Merge lp:~garyvdm/bzr/Bug427773 into lp:bzr/2.0
- Bug427773
- Merge into 2.0
Proposed by
Alexander Belchenko
Status: | Rejected | ||||
---|---|---|---|---|---|
Rejected by: | Gary van der Merwe | ||||
Proposed branch: | lp:~garyvdm/bzr/Bug427773 | ||||
Merge into: | lp:bzr/2.0 | ||||
Diff against target: | None lines | ||||
To merge this branch: | bzr merge lp:~garyvdm/bzr/Bug427773 | ||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
bzr-core | Pending | ||
Review via email: mp+11645@code.launchpad.net |
Commit message
Description of the change
To post a comment you must log in.
Revision history for this message
Gary van der Merwe (garyvdm) wrote : | # |
Preview Diff
[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1 | === modified file 'Makefile' | |||
2 | --- Makefile 2009-08-03 20:38:39 +0000 | |||
3 | +++ Makefile 2009-08-27 00:53:27 +0000 | |||
4 | @@ -1,4 +1,4 @@ | |||
6 | 1 | # Copyright (C) 2005, 2006, 2007, 2008 Canonical Ltd | 1 | # Copyright (C) 2005, 2006, 2007, 2008, 2009 Canonical Ltd |
7 | 2 | # | 2 | # |
8 | 3 | # This program is free software; you can redistribute it and/or modify | 3 | # This program is free software; you can redistribute it and/or modify |
9 | 4 | # it under the terms of the GNU General Public License as published by | 4 | # it under the terms of the GNU General Public License as published by |
10 | @@ -40,8 +40,6 @@ | |||
11 | 40 | 40 | ||
12 | 41 | check-nodocs: extensions | 41 | check-nodocs: extensions |
13 | 42 | $(PYTHON) -Werror -O ./bzr selftest -1v $(tests) | 42 | $(PYTHON) -Werror -O ./bzr selftest -1v $(tests) |
14 | 43 | @echo "Running all tests with no locale." | ||
15 | 44 | LC_CTYPE= LANG=C LC_ALL= ./bzr selftest -1v $(tests) 2>&1 | sed -e 's/^/[ascii] /' | ||
16 | 45 | 43 | ||
17 | 46 | # Run Python style checker (apt-get install pyflakes) | 44 | # Run Python style checker (apt-get install pyflakes) |
18 | 47 | # | 45 | # |
19 | 48 | 46 | ||
20 | === modified file 'NEWS' | |||
21 | --- NEWS 2009-08-25 05:32:07 +0000 | |||
22 | +++ NEWS 2009-09-11 14:23:16 +0000 | |||
23 | @@ -6,6 +6,309 @@ | |||
24 | 6 | .. contents:: List of Releases | 6 | .. contents:: List of Releases |
25 | 7 | :depth: 1 | 7 | :depth: 1 |
26 | 8 | 8 | ||
27 | 9 | In Development | ||
28 | 10 | ############## | ||
29 | 11 | |||
30 | 12 | Compatibility Breaks | ||
31 | 13 | ******************** | ||
32 | 14 | |||
33 | 15 | New Features | ||
34 | 16 | ************ | ||
35 | 17 | |||
36 | 18 | * Give more control on BZR_PLUGIN_PATH by providing a way to refer to or | ||
37 | 19 | disable the user, site and core plugin directories. | ||
38 | 20 | (Vincent Ladeuil, #412930, #316192, #145612) | ||
39 | 21 | |||
40 | 22 | Bug Fixes | ||
41 | 23 | ********* | ||
42 | 24 | |||
43 | 25 | * Bazaar's native protocol code now correctly handles EINTR, which most | ||
44 | 26 | noticeably occurs if you break in to the debugger while connected to a | ||
45 | 27 | bzr+ssh server. You can now can continue from the debugger (by typing | ||
46 | 28 | 'c') and the process continues. However, note that pressing C-\ in the | ||
47 | 29 | shell may still kill the SSH process, which is bug 162509, so you must | ||
48 | 30 | sent a signal to the bzr process specifically, for example by typing | ||
49 | 31 | ``kill -QUIT PID`` in another shell. (Martin Pool, #341535) | ||
50 | 32 | |||
51 | 33 | * ``bzr check`` in pack-0.92, 1.6 and 1.9 format repositories will no | ||
52 | 34 | longer report incorrect errors about ``Missing inventory ('TREE_ROOT', ...)`` | ||
53 | 35 | (Robert Collins, #416732) | ||
54 | 36 | |||
55 | 37 | * ``bzr info -v`` on a 2a format still claimed that it was a "Development | ||
56 | 38 | format" (John Arbash Meinel, #424392) | ||
57 | 39 | |||
58 | 40 | * ``bzr merge`` and ``bzr remove-tree`` now requires --force if pending | ||
59 | 41 | merges are present in the working tree. | ||
60 | 42 | (Vincent Ladeuil, #426344) | ||
61 | 43 | |||
62 | 44 | * Clearer message when Bazaar runs out of memory, instead of a ``MemoryError`` | ||
63 | 45 | traceback. (Martin Pool, #109115) | ||
64 | 46 | |||
65 | 47 | * Conversion to 2a will create a single pack for all the new revisions (as | ||
66 | 48 | long as it ran without interruption). This improves both ``bzr upgrade`` | ||
67 | 49 | and ``bzr pull`` or ``bzr merge`` from local branches in older formats. | ||
68 | 50 | The autopack logic that occurs every 100 revisions during local | ||
69 | 51 | conversions was not returning that pack's identifier, which resulted in | ||
70 | 52 | the partial packs created during the conversion not being consolidated | ||
71 | 53 | at the end of the conversion process. (Robert Collins, #423818) | ||
72 | 54 | |||
73 | 55 | * Don't restrict the command name used to run the test suite. | ||
74 | 56 | (Vincent Ladeuil, #419950) | ||
75 | 57 | |||
76 | 58 | * Fetches from 2a to 2a are now again requested in 'groupcompress' order. | ||
77 | 59 | Groups that are seen as 'underutilized' will be repacked on-the-fly. | ||
78 | 60 | This means that when the source is fully packed, there is minimal | ||
79 | 61 | overhead during the fetch, but if the source is poorly packed the result | ||
80 | 62 | is a fairly well packed repository (not as good as 'bzr pack' but | ||
81 | 63 | good-enough.) (Robert Collins, John Arbash Meinel, #402652) | ||
82 | 64 | |||
83 | 65 | * Network streams now decode adjacent records of the same type into a | ||
84 | 66 | single stream, reducing layering churn. (Robert Collins) | ||
85 | 67 | |||
86 | 68 | * Prevent some kinds of incomplete data from being committed to a 2a | ||
87 | 69 | repository, such as revisions without inventories or inventories without | ||
88 | 70 | chk_bytes root records. | ||
89 | 71 | (Andrew Bennetts, #423506) | ||
90 | 72 | |||
91 | 73 | * Make sure that we unlock the tree if we fail to create a TreeTransform | ||
92 | 74 | object when doing a merge, and there is limbo, or pending-deletions | ||
93 | 75 | directory. (Gary van der Merwe, #427773) | ||
94 | 76 | |||
95 | 77 | Improvements | ||
96 | 78 | ************ | ||
97 | 79 | |||
98 | 80 | Documentation | ||
99 | 81 | ************* | ||
100 | 82 | |||
101 | 83 | * Help on hooks no longer says 'Not deprecated' for hooks that are | ||
102 | 84 | currently supported. (Ian Clatworthy, #422415) | ||
103 | 85 | |||
104 | 86 | API Changes | ||
105 | 87 | *********** | ||
106 | 88 | |||
107 | 89 | * ``bzrlib.tests`` now uses ``stopTestRun`` for its ``TestResult`` | ||
108 | 90 | subclasses - the same as python's unittest module. (Robert Collins) | ||
109 | 91 | |||
110 | 92 | Internals | ||
111 | 93 | ********* | ||
112 | 94 | |||
113 | 95 | * ``BTreeLeafParser.extract_key`` has been tweaked slightly to reduce | ||
114 | 96 | mallocs while parsing the index (approx 3=>1 mallocs per key read). | ||
115 | 97 | This results in a 10% speedup while reading an index. | ||
116 | 98 | (John Arbash Meinel) | ||
117 | 99 | |||
118 | 100 | * The ``bzrlib.lsprof`` module has a new class ``BzrProfiler`` which makes | ||
119 | 101 | profiling in some situations like callbacks and generators easier. | ||
120 | 102 | (Robert Collins) | ||
121 | 103 | |||
122 | 104 | Testing | ||
123 | 105 | ******* | ||
124 | 106 | |||
125 | 107 | * Passing ``--lsprof-tests -v`` to bzr selftest will cause lsprof output to | ||
126 | 108 | be output for every test. Note that this is very verbose! (Robert Collins) | ||
127 | 109 | |||
128 | 110 | * Test parameterisation now does a shallow copy, not a deep copy of the test | ||
129 | 111 | to be parameterised. This is not expected to break external use of test | ||
130 | 112 | parameterisation, and is substantially faster. (Robert Collins) | ||
131 | 113 | |||
132 | 114 | |||
133 | 115 | bzr 2.0rc2 (not released yet) | ||
134 | 116 | ############################# | ||
135 | 117 | |||
136 | 118 | New Features | ||
137 | 119 | ************ | ||
138 | 120 | |||
139 | 121 | * Added post_commit hook for mutable trees. This allows the keywords | ||
140 | 122 | plugin to expand keywords on files changed by the commit. | ||
141 | 123 | (Ian Clatworthy, #408841) | ||
142 | 124 | |||
143 | 125 | Bug Fixes | ||
144 | 126 | ********* | ||
145 | 127 | |||
146 | 128 | * Bazaar's native protocol code now correctly handles EINTR, which most | ||
147 | 129 | noticeably occurs if you break in to the debugger while connected to a | ||
148 | 130 | bzr+ssh server. You can now can continue from the debugger (by typing | ||
149 | 131 | 'c') and the process continues. However, note that pressing C-\ in the | ||
150 | 132 | shell may still kill the SSH process, which is bug 162509, so you must | ||
151 | 133 | sent a signal to the bzr process specifically, for example by typing | ||
152 | 134 | ``kill -QUIT PID`` in another shell. (Martin Pool, #341535) | ||
153 | 135 | |||
154 | 136 | * ``bzr check`` in pack-0.92, 1.6 and 1.9 format repositories will no | ||
155 | 137 | longer report incorrect errors about ``Missing inventory ('TREE_ROOT', ...)`` | ||
156 | 138 | (Robert Collins, #416732) | ||
157 | 139 | |||
158 | 140 | * ``bzr info -v`` on a 2a format still claimed that it was a "Development | ||
159 | 141 | format" (John Arbash Meinel, #424392) | ||
160 | 142 | |||
161 | 143 | * ``bzr log stacked-branch`` shows the full log including | ||
162 | 144 | revisions that are in the fallback repository. (Regressed in 2.0rc1). | ||
163 | 145 | (John Arbash Meinel, #419241) | ||
164 | 146 | |||
165 | 147 | * Clearer message when Bazaar runs out of memory, instead of a ``MemoryError`` | ||
166 | 148 | traceback. (Martin Pool, #109115) | ||
167 | 149 | |||
168 | 150 | * Conversion to 2a will create a single pack for all the new revisions (as | ||
169 | 151 | long as it ran without interruption). This improves both ``bzr upgrade`` | ||
170 | 152 | and ``bzr pull`` or ``bzr merge`` from local branches in older formats. | ||
171 | 153 | The autopack logic that occurs every 100 revisions during local | ||
172 | 154 | conversions was not returning that pack's identifier, which resulted in | ||
173 | 155 | the partial packs created during the conversion not being consolidated | ||
174 | 156 | at the end of the conversion process. (Robert Collins, #423818) | ||
175 | 157 | |||
176 | 158 | * Fetches from 2a to 2a are now again requested in 'groupcompress' order. | ||
177 | 159 | Groups that are seen as 'underutilized' will be repacked on-the-fly. | ||
178 | 160 | This means that when the source is fully packed, there is minimal | ||
179 | 161 | overhead during the fetch, but if the source is poorly packed the result | ||
180 | 162 | is a fairly well packed repository (not as good as 'bzr pack' but | ||
181 | 163 | good-enough.) (Robert Collins, John Arbash Meinel, #402652) | ||
182 | 164 | |||
183 | 165 | * Fix a potential segmentation fault when doing 'log' of a branch that had | ||
184 | 166 | ghosts in its mainline. (Evaluating None as a tuple is bad.) | ||
185 | 167 | (John Arbash Meinel, #419241) | ||
186 | 168 | |||
187 | 169 | * ``groupcompress`` sort order is now more stable, rather than relying on | ||
188 | 170 | ``topo_sort`` ordering. The implementation is now | ||
189 | 171 | ``KnownGraph.gc_sort``. (John Arbash Meinel) | ||
190 | 172 | |||
191 | 173 | * Local data conversion will generate correct deltas. This is a critical | ||
192 | 174 | bugfix vs 2.0rc1, and all 2.0rc1 users should upgrade to 2.0rc2 before | ||
193 | 175 | converting repositories. (Robert Collins, #422849) | ||
194 | 176 | |||
195 | 177 | * Network streams now decode adjacent records of the same type into a | ||
196 | 178 | single stream, reducing layering churn. (Robert Collins) | ||
197 | 179 | |||
198 | 180 | * Prevent some kinds of incomplete data from being committed to a 2a | ||
199 | 181 | repository, such as revisions without inventories, a missing chk_bytes | ||
200 | 182 | record for an inventory, or a missing text referenced by an inventory. | ||
201 | 183 | (Andrew Bennetts, #423506, #406687) | ||
202 | 184 | |||
203 | 185 | Documentation | ||
204 | 186 | ************* | ||
205 | 187 | |||
206 | 188 | * Fix assertion error about "_remember_remote_is_before" when pushing to | ||
207 | 189 | older smart servers. | ||
208 | 190 | (Andrew Bennetts, #418931) | ||
209 | 191 | |||
210 | 192 | * Help on hooks no longer says 'Not deprecated' for hooks that are | ||
211 | 193 | currently supported. (Ian Clatworthy, #422415) | ||
212 | 194 | |||
213 | 195 | * The main table of contents now provides links to the new Migration Docs | ||
214 | 196 | and Plugins Guide. (Ian Clatworthy) | ||
215 | 197 | |||
216 | 198 | |||
217 | 199 | bzr 2.0rc1 | ||
218 | 200 | ########## | ||
219 | 201 | |||
220 | 202 | |||
221 | 203 | :Codename: no worries | ||
222 | 204 | :2.0rc1: 2009-08-26 | ||
223 | 205 | |||
224 | 206 | This release of Bazaar makes 2a 'brisbane-core' format the | ||
225 | 207 | default. Most of the work in this release now focuses on bug | ||
226 | 208 | fixes and stabilization, covering both 2a and previous formats. | ||
227 | 209 | |||
228 | 210 | The Bazaar team decided that 2.0 will be a long-term supported | ||
229 | 211 | release, with bugfix-only releases based on it continuing for at | ||
230 | 212 | least six months or until the following stable release (we said | ||
231 | 213 | that previously, but that's worth repeating). | ||
232 | 214 | |||
233 | 215 | Compatibility Breaks | ||
234 | 216 | ******************** | ||
235 | 217 | |||
236 | 218 | * The default format for bzr is now ``2a``. This format brings many | ||
237 | 219 | significant performance and size improvements. bzr can pull from | ||
238 | 220 | any existing repository into a ``2a`` one, but can only transfer | ||
239 | 221 | into ``rich-root`` repositories from ``2a``. The Upgrade guide | ||
240 | 222 | has more information about this change. (Robert Collins) | ||
241 | 223 | |||
242 | 224 | * On Windows auto-detection of Putty's plink.exe is disabled. | ||
243 | 225 | Default SSH client for Windows is paramiko. User still can force | ||
244 | 226 | usage of plink if explicitly set environment variable BZR_SSH=plink. | ||
245 | 227 | (#414743, Alexander Belchenko) | ||
246 | 228 | |||
247 | 229 | New Features | ||
248 | 230 | ************ | ||
249 | 231 | |||
250 | 232 | * ``bzr branch --switch`` can now switch the checkout in the current directory | ||
251 | 233 | to the newly created branch. (Lukáš Lalinský) | ||
252 | 234 | |||
253 | 235 | Bug Fixes | ||
254 | 236 | ********* | ||
255 | 237 | |||
256 | 238 | * Further tweaks to handling of ``bzr add`` messages about ignored files. | ||
257 | 239 | (Jason Spashett, #76616) | ||
258 | 240 | |||
259 | 241 | * Fetches were being requested in 'groupcompress' order, but weren't | ||
260 | 242 | recombining the groups. Thus they would 'fragment' to get the correct | ||
261 | 243 | order, but not 'recombine' to actually benefit from it. Until we get | ||
262 | 244 | recombining to work, switching to 'unordered' fetches avoids the | ||
263 | 245 | fragmentation. (John Arbash Meinel, #402645) | ||
264 | 246 | |||
265 | 247 | * Fix a pycurl related test failure on karmic by recognizing an error | ||
266 | 248 | raised by newer versions of pycurl. | ||
267 | 249 | (Vincent Ladeuil, #306264) | ||
268 | 250 | |||
269 | 251 | * Fix a test failure on karmic by making a locale test more robust. | ||
270 | 252 | (Vincent Ladeuil, #413514) | ||
271 | 253 | |||
272 | 254 | * Fix IndexError printing CannotBindAddress errors. | ||
273 | 255 | (Martin Pool, #286871) | ||
274 | 256 | |||
275 | 257 | * Fix "Revision ... not present" errors when upgrading stacked branches, | ||
276 | 258 | or when doing fetches from a stacked source to a stacked target. | ||
277 | 259 | (Andrew Bennetts, #399140) | ||
278 | 260 | |||
279 | 261 | * ``bzr branch`` of 2a repositories over HTTP is much faster. bzr now | ||
280 | 262 | batches together small fetches from 2a repositories, rather than | ||
281 | 263 | fetching only a few hundred bytes at a time. | ||
282 | 264 | (Andrew Bennetts, #402657) | ||
283 | 265 | |||
284 | 266 | Improvements | ||
285 | 267 | ************ | ||
286 | 268 | |||
287 | 269 | * A better description of the platform is shown in crash tracebacks, ``bzr | ||
288 | 270 | --version`` and ``bzr selftest``. | ||
289 | 271 | (Martin Pool, #409137) | ||
290 | 272 | |||
291 | 273 | * bzr can now (again) capture crash data through the apport library, | ||
292 | 274 | so that a single human-readable file can be attached to bug reports. | ||
293 | 275 | This can be disabled by using ``-Dno_apport`` on the command line, or by | ||
294 | 276 | putting ``no_apport`` into the ``debug_flags`` section of | ||
295 | 277 | ``bazaar.conf``. | ||
296 | 278 | (Martin Pool, Robert Collins, #389328) | ||
297 | 279 | |||
298 | 280 | * ``bzr push`` locally on windows will no longer give a locking error with | ||
299 | 281 | dirstate based formats. (Robert Collins) | ||
300 | 282 | |||
301 | 283 | * ``bzr shelve`` and ``bzr unshelve`` now work on windows. | ||
302 | 284 | (Robert Collins, #305006) | ||
303 | 285 | |||
304 | 286 | * Commit of specific files no longer prevents using the the iter_changes | ||
305 | 287 | codepath. On 2a repositories, commit of specific files should now be as | ||
306 | 288 | fast, or slightly faster, than a full commit. (Robert Collins) | ||
307 | 289 | |||
308 | 290 | * The internal core code that handles specific file operations like | ||
309 | 291 | ``bzr st FILENAME`` or ``bzr commit FILENAME`` has been changed to | ||
310 | 292 | include the parent directories if they have altered, and when a | ||
311 | 293 | directory stops being a directory its children are always included. This | ||
312 | 294 | fixes a number of causes for ``InconsistentDelta`` errors, and permits | ||
313 | 295 | faster commit of specific paths. (Robert Collins, #347649) | ||
314 | 296 | |||
315 | 297 | Documentation | ||
316 | 298 | ************* | ||
317 | 299 | |||
318 | 300 | * New developer documentation for content filtering. | ||
319 | 301 | (Martin Pool) | ||
320 | 302 | |||
321 | 303 | API Changes | ||
322 | 304 | *********** | ||
323 | 305 | |||
324 | 306 | * ``bzrlib.shelf_ui`` has had the ``from_args`` convenience methods of its | ||
325 | 307 | classes changed to manage lock lifetime of the trees they open in a way | ||
326 | 308 | consistent with reader-exclusive locks. (Robert Collins, #305006) | ||
327 | 309 | |||
328 | 310 | Testing | ||
329 | 311 | ******* | ||
330 | 9 | 312 | ||
331 | 10 | bzr 1.18.1 NOT RELEASED YET | 313 | bzr 1.18.1 NOT RELEASED YET |
332 | 11 | ########################### | 314 | ########################### |
333 | @@ -17,17 +320,144 @@ | |||
334 | 17 | conversion will commit too many copies a file. | 320 | conversion will commit too many copies a file. |
335 | 18 | (Martin Pool, #415508) | 321 | (Martin Pool, #415508) |
336 | 19 | 322 | ||
337 | 323 | Improvements | ||
338 | 324 | ************ | ||
339 | 325 | |||
340 | 326 | * ``bzr push`` locally on windows will no longer give a locking error with | ||
341 | 327 | dirstate based formats. (Robert Collins) | ||
342 | 328 | |||
343 | 329 | * ``bzr shelve`` and ``bzr unshelve`` now work on windows. | ||
344 | 330 | (Robert Collins, #305006) | ||
345 | 331 | |||
346 | 20 | API Changes | 332 | API Changes |
347 | 21 | *********** | 333 | *********** |
348 | 22 | 334 | ||
349 | 335 | * ``bzrlib.shelf_ui`` has had the ``from_args`` convenience methods of its | ||
350 | 336 | classes changed to manage lock lifetime of the trees they open in a way | ||
351 | 337 | consistent with reader-exclusive locks. (Robert Collins, #305006) | ||
352 | 338 | |||
353 | 23 | * ``Tree.path_content_summary`` may return a size of None, when called on | 339 | * ``Tree.path_content_summary`` may return a size of None, when called on |
354 | 24 | a tree with content filtering where the size of the canonical form | 340 | a tree with content filtering where the size of the canonical form |
355 | 25 | cannot be cheaply determined. (Martin Pool) | 341 | cannot be cheaply determined. (Martin Pool) |
356 | 26 | 342 | ||
357 | 343 | * When manually creating transport servers in test cases, a new helper | ||
358 | 344 | ``TestCase.start_server`` that registers a cleanup and starts the server | ||
359 | 345 | should be used. (Robert Collins) | ||
360 | 27 | 346 | ||
361 | 28 | bzr 1.18 | 347 | bzr 1.18 |
362 | 29 | ######## | 348 | ######## |
363 | 30 | 349 | ||
364 | 350 | Compatibility Breaks | ||
365 | 351 | ******************** | ||
366 | 352 | |||
367 | 353 | * Committing directly to a stacked branch from a lightweight checkout will | ||
368 | 354 | no longer work. In previous versions this would appear to work but would | ||
369 | 355 | generate repositories with insufficient data to create deltas, leading | ||
370 | 356 | to later errors when branching or reading from the repository. | ||
371 | 357 | (Robert Collins, bug #375013) | ||
372 | 358 | |||
373 | 359 | New Features | ||
374 | 360 | ************ | ||
375 | 361 | |||
376 | 362 | Bug Fixes | ||
377 | 363 | ********* | ||
378 | 364 | |||
379 | 365 | * Fetching from 2a branches from a version-2 bzr protocol would fail to | ||
380 | 366 | copy the internal inventory pages from the CHK store. This cannot happen | ||
381 | 367 | in normal use as all 2a compatible clients and servers support the | ||
382 | 368 | version-3 protocol, but it does cause test suite failures when testing | ||
383 | 369 | downlevel protocol behaviour. (Robert Collins) | ||
384 | 370 | |||
385 | 371 | * Fix a test failure on karmic by making a locale test more robust. | ||
386 | 372 | (Vincent Ladeuil, #413514) | ||
387 | 373 | |||
388 | 374 | * Fixed "Pack ... already exists" error when running ``bzr pack`` on a | ||
389 | 375 | fully packed 2a repository. (Andrew Bennetts, #382463) | ||
390 | 376 | |||
391 | 377 | * Further tweaks to handling of ``bzr add`` messages about ignored files. | ||
392 | 378 | (Jason Spashett, #76616) | ||
393 | 379 | |||
394 | 380 | * Properly handle fetching into a stacked branch while converting the | ||
395 | 381 | data, especially when there are also ghosts. The code was filling in | ||
396 | 382 | parent inventories incorrectly, and also not handling when one of the | ||
397 | 383 | parents was a ghost. (John Arbash Meinel, #402778, #412198) | ||
398 | 384 | |||
399 | 385 | * ``RemoteStreamSource.get_stream_for_missing_keys`` will fetch CHK | ||
400 | 386 | inventory pages when appropriate (by falling back to the vfs stream | ||
401 | 387 | source). (Andrew Bennetts, #406686) | ||
402 | 388 | |||
403 | 389 | * StreamSource generates rich roots from non-rich root sources correctly | ||
404 | 390 | now. (Andrew Bennetts, #368921) | ||
405 | 391 | |||
406 | 392 | * When deciding whether a repository was compatible for upgrading or | ||
407 | 393 | fetching, we previously incorrectly checked the default repository | ||
408 | 394 | format for the bzrdir format, rather than the format that was actually | ||
409 | 395 | present on disk. (Martin Pool, #408824) | ||
410 | 396 | |||
411 | 397 | Improvements | ||
412 | 398 | ************ | ||
413 | 399 | |||
414 | 400 | * A better description of the platform is shown in crash tracebacks, ``bzr | ||
415 | 401 | --version`` and ``bzr selftest``. | ||
416 | 402 | (Martin Pool, #409137) | ||
417 | 403 | |||
418 | 404 | * Cross-format fetches (such as between 1.9-rich-root and 2a) via the | ||
419 | 405 | smart server are more efficient now. They send inventory deltas rather | ||
420 | 406 | than full inventories. The smart server has two new requests, | ||
421 | 407 | ``Repository.get_stream_1.19`` and ``Repository.insert_stream_1.19`` to | ||
422 | 408 | support this. (Andrew Bennetts, #374738, #385826) | ||
423 | 409 | |||
424 | 410 | * Extracting the full ancestry and computing the ``merge_sort`` is now | ||
425 | 411 | significantly faster. This effects things like ``bzr log -n0``. (For | ||
426 | 412 | example, ``bzr log -r -10..-1 -n0 bzr.dev`` is 2.5s down to 1.0s. | ||
427 | 413 | (John Arbash Meinel) | ||
428 | 414 | |||
429 | 415 | Documentation | ||
430 | 416 | ************* | ||
431 | 417 | |||
432 | 418 | API Changes | ||
433 | 419 | *********** | ||
434 | 420 | |||
435 | 421 | Internals | ||
436 | 422 | ********* | ||
437 | 423 | |||
438 | 424 | * ``-Dstrict_locks`` can now be used to check that read and write locks | ||
439 | 425 | are treated properly w.r.t. exclusivity. (We don't try to take an OS | ||
440 | 426 | read lock on a file that we already have an OS write lock on.) This is | ||
441 | 427 | now set by default for all tests, if you have a test which cannot be | ||
442 | 428 | fixed, you can use ``self.thisFailsStrictLockCheck()`` as a | ||
443 | 429 | compatibility knob. (John Arbash Meinel) | ||
444 | 430 | |||
445 | 431 | * InterDifferingSerializer is now only used locally. Other fetches that | ||
446 | 432 | would have used InterDifferingSerializer now use the more network | ||
447 | 433 | friendly StreamSource, which now automatically does the same | ||
448 | 434 | transformations as InterDifferingSerializer. (Andrew Bennetts) | ||
449 | 435 | |||
450 | 436 | * ``KnownGraph`` now has a ``.topo_sort`` and ``.merge_sort`` member which | ||
451 | 437 | are implemented in pyrex and significantly faster. This is exposed along | ||
452 | 438 | with ``CombinedGraphIndex.find_ancestry()`` as | ||
453 | 439 | ``VersionedFiles.get_known_graph_ancestry(keys)``. | ||
454 | 440 | (John Arbash Meinel) | ||
455 | 441 | |||
456 | 442 | * RemoteBranch.open now honours ignore_fallbacks correctly on bzr-v2 | ||
457 | 443 | protocols. (Robert Collins) | ||
458 | 444 | |||
459 | 445 | * The index code now has some specialized routines to extract the full | ||
460 | 446 | ancestry of a key in a more efficient manner. | ||
461 | 447 | ``CombinedGraphIndex.find_ancestry()``. (Time to get ancestry for | ||
462 | 448 | bzr.dev drops from 1.5s down to 300ms. For OOo from 33s => 10.5s) (John | ||
463 | 449 | Arbash Meinel) | ||
464 | 450 | |||
465 | 451 | Testing | ||
466 | 452 | ******* | ||
467 | 453 | |||
468 | 454 | * Install the test ssl certificate and key so that installed bzr | ||
469 | 455 | can run the https tests. (Denys Duchier, #392401) | ||
470 | 456 | |||
471 | 457 | |||
472 | 458 | bzr 1.18rc1 | ||
473 | 459 | ########### | ||
474 | 460 | |||
475 | 31 | :Codename: little traveller | 461 | :Codename: little traveller |
476 | 32 | :1.18: 2009-08-20 | 462 | :1.18: 2009-08-20 |
477 | 33 | :1.18rc1: 2009-08-10 | 463 | :1.18rc1: 2009-08-10 |
478 | @@ -113,6 +543,9 @@ | |||
479 | 113 | running send and similar commands on 2a formats. | 543 | running send and similar commands on 2a formats. |
480 | 114 | (Martin Pool, #408201) | 544 | (Martin Pool, #408201) |
481 | 115 | 545 | ||
482 | 546 | * Fix crash in some invocations of ``bzr status`` in format 2a. | ||
483 | 547 | (Martin Pool, #403523) | ||
484 | 548 | |||
485 | 116 | * Fixed export to existing directory: if directory is empty then export | 549 | * Fixed export to existing directory: if directory is empty then export |
486 | 117 | will succeed, otherwise it fails with error. | 550 | will succeed, otherwise it fails with error. |
487 | 118 | (Alexander Belchenko, #406174) | 551 | (Alexander Belchenko, #406174) |
488 | @@ -133,7 +566,9 @@ | |||
489 | 133 | * Requests for unknown methods no longer cause the smart server to log | 566 | * Requests for unknown methods no longer cause the smart server to log |
490 | 134 | lots of backtraces about ``UnknownSmartMethod``, ``do_chunk`` or | 567 | lots of backtraces about ``UnknownSmartMethod``, ``do_chunk`` or |
491 | 135 | ``do_end``. (Andrew Bennetts, #338561) | 568 | ``do_end``. (Andrew Bennetts, #338561) |
493 | 136 | 569 | ||
494 | 570 | * Shelve will not shelve the initial add of the tree root. (Aaron Bentley) | ||
495 | 571 | |||
496 | 137 | * Streaming from bzr servers where there is a chain of stacked branches | 572 | * Streaming from bzr servers where there is a chain of stacked branches |
497 | 138 | (A stacked on B stacked on C) will now work. (Robert Collins, #406597) | 573 | (A stacked on B stacked on C) will now work. (Robert Collins, #406597) |
498 | 139 | 574 | ||
499 | @@ -245,6 +680,17 @@ | |||
500 | 245 | ``countTestsCases``. (Robert Collins) | 680 | ``countTestsCases``. (Robert Collins) |
501 | 246 | 681 | ||
502 | 247 | 682 | ||
503 | 683 | bzr 1.17.1 (unreleased) | ||
504 | 684 | ####################### | ||
505 | 685 | |||
506 | 686 | Bug Fixes | ||
507 | 687 | ********* | ||
508 | 688 | |||
509 | 689 | * The optional ``_knit_load_data_pyx`` C extension was never being | ||
510 | 690 | imported. This caused significant slowdowns when reading data from | ||
511 | 691 | knit format repositories. (Andrew Bennetts, #405653) | ||
512 | 692 | |||
513 | 693 | |||
514 | 248 | bzr 1.17 "So late it's brunch" 2009-07-20 | 694 | bzr 1.17 "So late it's brunch" 2009-07-20 |
515 | 249 | ######################################### | 695 | ######################################### |
516 | 250 | :Codename: so-late-its-brunch | 696 | :Codename: so-late-its-brunch |
517 | @@ -743,6 +1189,9 @@ | |||
518 | 743 | Testing | 1189 | Testing |
519 | 744 | ******* | 1190 | ******* |
520 | 745 | 1191 | ||
521 | 1192 | * ``make check`` no longer repeats the test run in ``LANG=C``. | ||
522 | 1193 | (Martin Pool, #386180) | ||
523 | 1194 | |||
524 | 746 | * The number of cores is now correctly detected on OSX. (John Szakmeister) | 1195 | * The number of cores is now correctly detected on OSX. (John Szakmeister) |
525 | 747 | 1196 | ||
526 | 748 | * The number of cores is also detected on Solaris and win32. (Vincent Ladeuil) | 1197 | * The number of cores is also detected on Solaris and win32. (Vincent Ladeuil) |
527 | @@ -4723,7 +5172,7 @@ | |||
528 | 4723 | checkouts. (Aaron Bentley, #182040) | 5172 | checkouts. (Aaron Bentley, #182040) |
529 | 4724 | 5173 | ||
530 | 4725 | * Stop polluting /tmp when running selftest. | 5174 | * Stop polluting /tmp when running selftest. |
532 | 4726 | (Vincent Ladeuil, #123623) | 5175 | (Vincent Ladeuil, #123363) |
533 | 4727 | 5176 | ||
534 | 4728 | * Switch from NFKC => NFC for normalization checks. NFC allows a few | 5177 | * Switch from NFKC => NFC for normalization checks. NFC allows a few |
535 | 4729 | more characters which should be considered valid. | 5178 | more characters which should be considered valid. |
536 | 4730 | 5179 | ||
537 | === modified file 'bzr' | |||
538 | --- bzr 2009-07-30 23:54:26 +0000 | |||
539 | +++ bzr 2009-08-28 05:11:10 +0000 | |||
540 | @@ -23,7 +23,7 @@ | |||
541 | 23 | import warnings | 23 | import warnings |
542 | 24 | 24 | ||
543 | 25 | # update this on each release | 25 | # update this on each release |
545 | 26 | _script_version = (1, 18, 0) | 26 | _script_version = (2, 1, 0) |
546 | 27 | 27 | ||
547 | 28 | if __doc__ is None: | 28 | if __doc__ is None: |
548 | 29 | print "bzr does not support python -OO." | 29 | print "bzr does not support python -OO." |
549 | 30 | 30 | ||
550 | === modified file 'bzrlib/__init__.py' | |||
551 | --- bzrlib/__init__.py 2009-08-20 08:38:09 +0000 | |||
552 | +++ bzrlib/__init__.py 2009-08-30 21:34:42 +0000 | |||
553 | @@ -50,7 +50,7 @@ | |||
554 | 50 | # Python version 2.0 is (2, 0, 0, 'final', 0)." Additionally we use a | 50 | # Python version 2.0 is (2, 0, 0, 'final', 0)." Additionally we use a |
555 | 51 | # releaselevel of 'dev' for unreleased under-development code. | 51 | # releaselevel of 'dev' for unreleased under-development code. |
556 | 52 | 52 | ||
558 | 53 | version_info = (1, 18, 0, 'final', 0) | 53 | version_info = (2, 1, 0, 'dev', 0) |
559 | 54 | 54 | ||
560 | 55 | # API compatibility version: bzrlib is currently API compatible with 1.15. | 55 | # API compatibility version: bzrlib is currently API compatible with 1.15. |
561 | 56 | api_minimum_version = (1, 17, 0) | 56 | api_minimum_version = (1, 17, 0) |
562 | @@ -70,6 +70,8 @@ | |||
563 | 70 | 1.2dev | 70 | 1.2dev |
564 | 71 | >>> print _format_version_tuple((1, 1, 1, 'candidate', 2)) | 71 | >>> print _format_version_tuple((1, 1, 1, 'candidate', 2)) |
565 | 72 | 1.1.1rc2 | 72 | 1.1.1rc2 |
566 | 73 | >>> print bzrlib._format_version_tuple((2, 1, 0, 'beta', 1)) | ||
567 | 74 | 2.1b1 | ||
568 | 73 | >>> print _format_version_tuple((1, 4, 0)) | 75 | >>> print _format_version_tuple((1, 4, 0)) |
569 | 74 | 1.4 | 76 | 1.4 |
570 | 75 | >>> print _format_version_tuple((1, 4)) | 77 | >>> print _format_version_tuple((1, 4)) |
571 | 76 | 78 | ||
572 | === modified file 'bzrlib/_btree_serializer_pyx.pyx' | |||
573 | --- bzrlib/_btree_serializer_pyx.pyx 2009-06-22 12:52:39 +0000 | |||
574 | +++ bzrlib/_btree_serializer_pyx.pyx 2009-09-04 21:16:14 +0000 | |||
575 | @@ -1,4 +1,4 @@ | |||
577 | 1 | # Copyright (C) 2008 Canonical Ltd | 1 | # Copyright (C) 2008, 2009 Canonical Ltd |
578 | 2 | # | 2 | # |
579 | 3 | # This program is free software; you can redistribute it and/or modify | 3 | # This program is free software; you can redistribute it and/or modify |
580 | 4 | # it under the terms of the GNU General Public License as published by | 4 | # it under the terms of the GNU General Public License as published by |
581 | @@ -41,8 +41,11 @@ | |||
582 | 41 | int PyString_AsStringAndSize_ptr(PyObject *, char **buf, Py_ssize_t *len) | 41 | int PyString_AsStringAndSize_ptr(PyObject *, char **buf, Py_ssize_t *len) |
583 | 42 | void PyString_InternInPlace(PyObject **) | 42 | void PyString_InternInPlace(PyObject **) |
584 | 43 | int PyTuple_CheckExact(object t) | 43 | int PyTuple_CheckExact(object t) |
585 | 44 | object PyTuple_New(Py_ssize_t n_entries) | ||
586 | 45 | void PyTuple_SET_ITEM(object, Py_ssize_t offset, object) # steals the ref | ||
587 | 44 | Py_ssize_t PyTuple_GET_SIZE(object t) | 46 | Py_ssize_t PyTuple_GET_SIZE(object t) |
588 | 45 | PyObject *PyTuple_GET_ITEM_ptr_object "PyTuple_GET_ITEM" (object tpl, int index) | 47 | PyObject *PyTuple_GET_ITEM_ptr_object "PyTuple_GET_ITEM" (object tpl, int index) |
589 | 48 | void Py_INCREF(object) | ||
590 | 46 | void Py_DECREF_ptr "Py_DECREF" (PyObject *) | 49 | void Py_DECREF_ptr "Py_DECREF" (PyObject *) |
591 | 47 | 50 | ||
592 | 48 | cdef extern from "string.h": | 51 | cdef extern from "string.h": |
593 | @@ -140,14 +143,12 @@ | |||
594 | 140 | cdef char *temp_ptr | 143 | cdef char *temp_ptr |
595 | 141 | cdef int loop_counter | 144 | cdef int loop_counter |
596 | 142 | # keys are tuples | 145 | # keys are tuples |
601 | 143 | loop_counter = 0 | 146 | key = PyTuple_New(self.key_length) |
602 | 144 | key_segments = [] | 147 | for loop_counter from 0 <= loop_counter < self.key_length: |
599 | 145 | while loop_counter < self.key_length: | ||
600 | 146 | loop_counter = loop_counter + 1 | ||
603 | 147 | # grab a key segment | 148 | # grab a key segment |
604 | 148 | temp_ptr = <char*>memchr(self._start, c'\0', last - self._start) | 149 | temp_ptr = <char*>memchr(self._start, c'\0', last - self._start) |
605 | 149 | if temp_ptr == NULL: | 150 | if temp_ptr == NULL: |
607 | 150 | if loop_counter == self.key_length: | 151 | if loop_counter + 1 == self.key_length: |
608 | 151 | # capture to last | 152 | # capture to last |
609 | 152 | temp_ptr = last | 153 | temp_ptr = last |
610 | 153 | else: | 154 | else: |
611 | @@ -164,8 +165,9 @@ | |||
612 | 164 | temp_ptr - self._start) | 165 | temp_ptr - self._start) |
613 | 165 | # advance our pointer | 166 | # advance our pointer |
614 | 166 | self._start = temp_ptr + 1 | 167 | self._start = temp_ptr + 1 |
617 | 167 | PyList_Append(key_segments, key_element) | 168 | Py_INCREF(key_element) |
618 | 168 | return tuple(key_segments) | 169 | PyTuple_SET_ITEM(key, loop_counter, key_element) |
619 | 170 | return key | ||
620 | 169 | 171 | ||
621 | 170 | cdef int process_line(self) except -1: | 172 | cdef int process_line(self) except -1: |
622 | 171 | """Process a line in the bytes.""" | 173 | """Process a line in the bytes.""" |
623 | 172 | 174 | ||
624 | === modified file 'bzrlib/_dirstate_helpers_pyx.pyx' | |||
625 | --- bzrlib/_dirstate_helpers_pyx.pyx 2009-07-27 05:44:19 +0000 | |||
626 | +++ bzrlib/_dirstate_helpers_pyx.pyx 2009-08-28 05:00:33 +0000 | |||
627 | @@ -28,7 +28,7 @@ | |||
628 | 28 | 28 | ||
629 | 29 | from bzrlib import cache_utf8, errors, osutils | 29 | from bzrlib import cache_utf8, errors, osutils |
630 | 30 | from bzrlib.dirstate import DirState | 30 | from bzrlib.dirstate import DirState |
632 | 31 | from bzrlib.osutils import pathjoin, splitpath | 31 | from bzrlib.osutils import parent_directories, pathjoin, splitpath |
633 | 32 | 32 | ||
634 | 33 | 33 | ||
635 | 34 | # This is the Windows equivalent of ENOTDIR | 34 | # This is the Windows equivalent of ENOTDIR |
636 | @@ -963,15 +963,21 @@ | |||
637 | 963 | 963 | ||
638 | 964 | cdef class ProcessEntryC: | 964 | cdef class ProcessEntryC: |
639 | 965 | 965 | ||
640 | 966 | cdef int doing_consistency_expansion | ||
641 | 966 | cdef object old_dirname_to_file_id # dict | 967 | cdef object old_dirname_to_file_id # dict |
642 | 967 | cdef object new_dirname_to_file_id # dict | 968 | cdef object new_dirname_to_file_id # dict |
643 | 968 | cdef object last_source_parent | 969 | cdef object last_source_parent |
644 | 969 | cdef object last_target_parent | 970 | cdef object last_target_parent |
646 | 970 | cdef object include_unchanged | 971 | cdef int include_unchanged |
647 | 972 | cdef int partial | ||
648 | 971 | cdef object use_filesystem_for_exec | 973 | cdef object use_filesystem_for_exec |
649 | 972 | cdef object utf8_decode | 974 | cdef object utf8_decode |
650 | 973 | cdef readonly object searched_specific_files | 975 | cdef readonly object searched_specific_files |
651 | 976 | cdef readonly object searched_exact_paths | ||
652 | 974 | cdef object search_specific_files | 977 | cdef object search_specific_files |
653 | 978 | # The parents up to the root of the paths we are searching. | ||
654 | 979 | # After all normal paths are returned, these specific items are returned. | ||
655 | 980 | cdef object search_specific_file_parents | ||
656 | 975 | cdef object state | 981 | cdef object state |
657 | 976 | # Current iteration variables: | 982 | # Current iteration variables: |
658 | 977 | cdef object current_root | 983 | cdef object current_root |
659 | @@ -989,31 +995,48 @@ | |||
660 | 989 | cdef object current_block_list | 995 | cdef object current_block_list |
661 | 990 | cdef object current_dir_info | 996 | cdef object current_dir_info |
662 | 991 | cdef object current_dir_list | 997 | cdef object current_dir_list |
663 | 998 | cdef object _pending_consistent_entries # list | ||
664 | 992 | cdef int path_index | 999 | cdef int path_index |
665 | 993 | cdef object root_dir_info | 1000 | cdef object root_dir_info |
666 | 994 | cdef object bisect_left | 1001 | cdef object bisect_left |
667 | 995 | cdef object pathjoin | 1002 | cdef object pathjoin |
668 | 996 | cdef object fstat | 1003 | cdef object fstat |
669 | 1004 | # A set of the ids we've output when doing partial output. | ||
670 | 1005 | cdef object seen_ids | ||
671 | 997 | cdef object sha_file | 1006 | cdef object sha_file |
672 | 998 | 1007 | ||
673 | 999 | def __init__(self, include_unchanged, use_filesystem_for_exec, | 1008 | def __init__(self, include_unchanged, use_filesystem_for_exec, |
674 | 1000 | search_specific_files, state, source_index, target_index, | 1009 | search_specific_files, state, source_index, target_index, |
675 | 1001 | want_unversioned, tree): | 1010 | want_unversioned, tree): |
676 | 1011 | self.doing_consistency_expansion = 0 | ||
677 | 1002 | self.old_dirname_to_file_id = {} | 1012 | self.old_dirname_to_file_id = {} |
678 | 1003 | self.new_dirname_to_file_id = {} | 1013 | self.new_dirname_to_file_id = {} |
679 | 1014 | # Are we doing a partial iter_changes? | ||
680 | 1015 | self.partial = set(['']).__ne__(search_specific_files) | ||
681 | 1004 | # Using a list so that we can access the values and change them in | 1016 | # Using a list so that we can access the values and change them in |
682 | 1005 | # nested scope. Each one is [path, file_id, entry] | 1017 | # nested scope. Each one is [path, file_id, entry] |
683 | 1006 | self.last_source_parent = [None, None] | 1018 | self.last_source_parent = [None, None] |
684 | 1007 | self.last_target_parent = [None, None] | 1019 | self.last_target_parent = [None, None] |
686 | 1008 | self.include_unchanged = include_unchanged | 1020 | if include_unchanged is None: |
687 | 1021 | self.include_unchanged = False | ||
688 | 1022 | else: | ||
689 | 1023 | self.include_unchanged = int(include_unchanged) | ||
690 | 1009 | self.use_filesystem_for_exec = use_filesystem_for_exec | 1024 | self.use_filesystem_for_exec = use_filesystem_for_exec |
691 | 1010 | self.utf8_decode = cache_utf8._utf8_decode | 1025 | self.utf8_decode = cache_utf8._utf8_decode |
692 | 1011 | # for all search_indexs in each path at or under each element of | 1026 | # for all search_indexs in each path at or under each element of |
696 | 1012 | # search_specific_files, if the detail is relocated: add the id, and add the | 1027 | # search_specific_files, if the detail is relocated: add the id, and |
697 | 1013 | # relocated path as one to search if its not searched already. If the | 1028 | # add the relocated path as one to search if its not searched already. |
698 | 1014 | # detail is not relocated, add the id. | 1029 | # If the detail is not relocated, add the id. |
699 | 1015 | self.searched_specific_files = set() | 1030 | self.searched_specific_files = set() |
700 | 1031 | # When we search exact paths without expanding downwards, we record | ||
701 | 1032 | # that here. | ||
702 | 1033 | self.searched_exact_paths = set() | ||
703 | 1016 | self.search_specific_files = search_specific_files | 1034 | self.search_specific_files = search_specific_files |
704 | 1035 | # The parents up to the root of the paths we are searching. | ||
705 | 1036 | # After all normal paths are returned, these specific items are returned. | ||
706 | 1037 | self.search_specific_file_parents = set() | ||
707 | 1038 | # The ids we've sent out in the delta. | ||
708 | 1039 | self.seen_ids = set() | ||
709 | 1017 | self.state = state | 1040 | self.state = state |
710 | 1018 | self.current_root = None | 1041 | self.current_root = None |
711 | 1019 | self.current_root_unicode = None | 1042 | self.current_root_unicode = None |
712 | @@ -1035,26 +1058,30 @@ | |||
713 | 1035 | self.current_block_pos = -1 | 1058 | self.current_block_pos = -1 |
714 | 1036 | self.current_dir_info = None | 1059 | self.current_dir_info = None |
715 | 1037 | self.current_dir_list = None | 1060 | self.current_dir_list = None |
716 | 1061 | self._pending_consistent_entries = [] | ||
717 | 1038 | self.path_index = 0 | 1062 | self.path_index = 0 |
718 | 1039 | self.root_dir_info = None | 1063 | self.root_dir_info = None |
719 | 1040 | self.bisect_left = bisect.bisect_left | 1064 | self.bisect_left = bisect.bisect_left |
720 | 1041 | self.pathjoin = osutils.pathjoin | 1065 | self.pathjoin = osutils.pathjoin |
721 | 1042 | self.fstat = os.fstat | 1066 | self.fstat = os.fstat |
722 | 1043 | self.sha_file = osutils.sha_file | 1067 | self.sha_file = osutils.sha_file |
723 | 1068 | if target_index != 0: | ||
724 | 1069 | # A lot of code in here depends on target_index == 0 | ||
725 | 1070 | raise errors.BzrError('unsupported target index') | ||
726 | 1044 | 1071 | ||
727 | 1045 | cdef _process_entry(self, entry, path_info): | 1072 | cdef _process_entry(self, entry, path_info): |
728 | 1046 | """Compare an entry and real disk to generate delta information. | 1073 | """Compare an entry and real disk to generate delta information. |
729 | 1047 | 1074 | ||
730 | 1048 | :param path_info: top_relpath, basename, kind, lstat, abspath for | 1075 | :param path_info: top_relpath, basename, kind, lstat, abspath for |
733 | 1049 | the path of entry. If None, then the path is considered absent. | 1076 | the path of entry. If None, then the path is considered absent in |
734 | 1050 | (Perhaps we should pass in a concrete entry for this ?) | 1077 | the target (Perhaps we should pass in a concrete entry for this ?) |
735 | 1051 | Basename is returned as a utf8 string because we expect this | 1078 | Basename is returned as a utf8 string because we expect this |
736 | 1052 | tuple will be ignored, and don't want to take the time to | 1079 | tuple will be ignored, and don't want to take the time to |
737 | 1053 | decode. | 1080 | decode. |
738 | 1054 | :return: (iter_changes_result, changed). If the entry has not been | 1081 | :return: (iter_changes_result, changed). If the entry has not been |
739 | 1055 | handled then changed is None. Otherwise it is False if no content | 1082 | handled then changed is None. Otherwise it is False if no content |
742 | 1056 | or metadata changes have occured, and None if any content or | 1083 | or metadata changes have occured, and True if any content or |
743 | 1057 | metadata change has occured. If self.include_unchanged is True then | 1084 | metadata change has occurred. If self.include_unchanged is True then |
744 | 1058 | if changed is not None, iter_changes_result will always be a result | 1085 | if changed is not None, iter_changes_result will always be a result |
745 | 1059 | tuple. Otherwise, iter_changes_result is None unless changed is | 1086 | tuple. Otherwise, iter_changes_result is None unless changed is |
746 | 1060 | True. | 1087 | True. |
747 | @@ -1099,9 +1126,12 @@ | |||
748 | 1099 | else: | 1126 | else: |
749 | 1100 | # add the source to the search path to find any children it | 1127 | # add the source to the search path to find any children it |
750 | 1101 | # has. TODO ? : only add if it is a container ? | 1128 | # has. TODO ? : only add if it is a container ? |
753 | 1102 | if not osutils.is_inside_any(self.searched_specific_files, | 1129 | if (not self.doing_consistency_expansion and |
754 | 1103 | source_details[1]): | 1130 | not osutils.is_inside_any(self.searched_specific_files, |
755 | 1131 | source_details[1])): | ||
756 | 1104 | self.search_specific_files.add(source_details[1]) | 1132 | self.search_specific_files.add(source_details[1]) |
757 | 1133 | # expanding from a user requested path, parent expansion | ||
758 | 1134 | # for delta consistency happens later. | ||
759 | 1105 | # generate the old path; this is needed for stating later | 1135 | # generate the old path; this is needed for stating later |
760 | 1106 | # as well. | 1136 | # as well. |
761 | 1107 | old_path = source_details[1] | 1137 | old_path = source_details[1] |
762 | @@ -1180,7 +1210,8 @@ | |||
763 | 1180 | file_id = entry[0][2] | 1210 | file_id = entry[0][2] |
764 | 1181 | self.old_dirname_to_file_id[old_path] = file_id | 1211 | self.old_dirname_to_file_id[old_path] = file_id |
765 | 1182 | # parent id is the entry for the path in the target tree | 1212 | # parent id is the entry for the path in the target tree |
767 | 1183 | if old_dirname == self.last_source_parent[0]: | 1213 | if old_basename and old_dirname == self.last_source_parent[0]: |
768 | 1214 | # use a cached hit for non-root source entries. | ||
769 | 1184 | source_parent_id = self.last_source_parent[1] | 1215 | source_parent_id = self.last_source_parent[1] |
770 | 1185 | else: | 1216 | else: |
771 | 1186 | try: | 1217 | try: |
772 | @@ -1196,7 +1227,8 @@ | |||
773 | 1196 | self.last_source_parent[0] = old_dirname | 1227 | self.last_source_parent[0] = old_dirname |
774 | 1197 | self.last_source_parent[1] = source_parent_id | 1228 | self.last_source_parent[1] = source_parent_id |
775 | 1198 | new_dirname = entry[0][0] | 1229 | new_dirname = entry[0][0] |
777 | 1199 | if new_dirname == self.last_target_parent[0]: | 1230 | if entry[0][1] and new_dirname == self.last_target_parent[0]: |
778 | 1231 | # use a cached hit for non-root target entries. | ||
779 | 1200 | target_parent_id = self.last_target_parent[1] | 1232 | target_parent_id = self.last_target_parent[1] |
780 | 1201 | else: | 1233 | else: |
781 | 1202 | try: | 1234 | try: |
782 | @@ -1313,8 +1345,13 @@ | |||
783 | 1313 | # a renamed parent. TODO: handle this efficiently. Its not | 1345 | # a renamed parent. TODO: handle this efficiently. Its not |
784 | 1314 | # common case to rename dirs though, so a correct but slow | 1346 | # common case to rename dirs though, so a correct but slow |
785 | 1315 | # implementation will do. | 1347 | # implementation will do. |
787 | 1316 | if not osutils.is_inside_any(self.searched_specific_files, target_details[1]): | 1348 | if (not self.doing_consistency_expansion and |
788 | 1349 | not osutils.is_inside_any(self.searched_specific_files, | ||
789 | 1350 | target_details[1])): | ||
790 | 1317 | self.search_specific_files.add(target_details[1]) | 1351 | self.search_specific_files.add(target_details[1]) |
791 | 1352 | # We don't expand the specific files parents list here as | ||
792 | 1353 | # the path is absent in target and won't create a delta with | ||
793 | 1354 | # missing parent. | ||
794 | 1318 | elif ((source_minikind == c'r' or source_minikind == c'a') and | 1355 | elif ((source_minikind == c'r' or source_minikind == c'a') and |
795 | 1319 | (target_minikind == c'r' or target_minikind == c'a')): | 1356 | (target_minikind == c'r' or target_minikind == c'a')): |
796 | 1320 | # neither of the selected trees contain this path, | 1357 | # neither of the selected trees contain this path, |
797 | @@ -1334,6 +1371,25 @@ | |||
798 | 1334 | def iter_changes(self): | 1371 | def iter_changes(self): |
799 | 1335 | return self | 1372 | return self |
800 | 1336 | 1373 | ||
801 | 1374 | cdef void _gather_result_for_consistency(self, result): | ||
802 | 1375 | """Check a result we will yield to make sure we are consistent later. | ||
803 | 1376 | |||
804 | 1377 | This gathers result's parents into a set to output later. | ||
805 | 1378 | |||
806 | 1379 | :param result: A result tuple. | ||
807 | 1380 | """ | ||
808 | 1381 | if not self.partial or not result[0]: | ||
809 | 1382 | return | ||
810 | 1383 | self.seen_ids.add(result[0]) | ||
811 | 1384 | new_path = result[1][1] | ||
812 | 1385 | if new_path: | ||
813 | 1386 | # Not the root and not a delete: queue up the parents of the path. | ||
814 | 1387 | self.search_specific_file_parents.update( | ||
815 | 1388 | osutils.parent_directories(new_path.encode('utf8'))) | ||
816 | 1389 | # Add the root directory which parent_directories does not | ||
817 | 1390 | # provide. | ||
818 | 1391 | self.search_specific_file_parents.add('') | ||
819 | 1392 | |||
820 | 1337 | cdef void _update_current_block(self): | 1393 | cdef void _update_current_block(self): |
821 | 1338 | if (self.block_index < len(self.state._dirblocks) and | 1394 | if (self.block_index < len(self.state._dirblocks) and |
822 | 1339 | osutils.is_inside(self.current_root, self.state._dirblocks[self.block_index][0])): | 1395 | osutils.is_inside(self.current_root, self.state._dirblocks[self.block_index][0])): |
823 | @@ -1406,8 +1462,11 @@ | |||
824 | 1406 | entry = self.root_entries[self.root_entries_pos] | 1462 | entry = self.root_entries[self.root_entries_pos] |
825 | 1407 | self.root_entries_pos = self.root_entries_pos + 1 | 1463 | self.root_entries_pos = self.root_entries_pos + 1 |
826 | 1408 | result, changed = self._process_entry(entry, self.root_dir_info) | 1464 | result, changed = self._process_entry(entry, self.root_dir_info) |
829 | 1409 | if changed is not None and changed or self.include_unchanged: | 1465 | if changed is not None: |
830 | 1410 | return result | 1466 | if changed: |
831 | 1467 | self._gather_result_for_consistency(result) | ||
832 | 1468 | if changed or self.include_unchanged: | ||
833 | 1469 | return result | ||
834 | 1411 | # Have we finished the prior root, or never started one ? | 1470 | # Have we finished the prior root, or never started one ? |
835 | 1412 | if self.current_root is None: | 1471 | if self.current_root is None: |
836 | 1413 | # TODO: the pending list should be lexically sorted? the | 1472 | # TODO: the pending list should be lexically sorted? the |
837 | @@ -1416,12 +1475,12 @@ | |||
838 | 1416 | self.current_root = self.search_specific_files.pop() | 1475 | self.current_root = self.search_specific_files.pop() |
839 | 1417 | except KeyError: | 1476 | except KeyError: |
840 | 1418 | raise StopIteration() | 1477 | raise StopIteration() |
841 | 1419 | self.current_root_unicode = self.current_root.decode('utf8') | ||
842 | 1420 | self.searched_specific_files.add(self.current_root) | 1478 | self.searched_specific_files.add(self.current_root) |
843 | 1421 | # process the entries for this containing directory: the rest will be | 1479 | # process the entries for this containing directory: the rest will be |
844 | 1422 | # found by their parents recursively. | 1480 | # found by their parents recursively. |
845 | 1423 | self.root_entries = self.state._entries_for_path(self.current_root) | 1481 | self.root_entries = self.state._entries_for_path(self.current_root) |
846 | 1424 | self.root_entries_len = len(self.root_entries) | 1482 | self.root_entries_len = len(self.root_entries) |
847 | 1483 | self.current_root_unicode = self.current_root.decode('utf8') | ||
848 | 1425 | self.root_abspath = self.tree.abspath(self.current_root_unicode) | 1484 | self.root_abspath = self.tree.abspath(self.current_root_unicode) |
849 | 1426 | try: | 1485 | try: |
850 | 1427 | root_stat = os.lstat(self.root_abspath) | 1486 | root_stat = os.lstat(self.root_abspath) |
851 | @@ -1458,6 +1517,8 @@ | |||
852 | 1458 | result, changed = self._process_entry(entry, self.root_dir_info) | 1517 | result, changed = self._process_entry(entry, self.root_dir_info) |
853 | 1459 | if changed is not None: | 1518 | if changed is not None: |
854 | 1460 | path_handled = -1 | 1519 | path_handled = -1 |
855 | 1520 | if changed: | ||
856 | 1521 | self._gather_result_for_consistency(result) | ||
857 | 1461 | if changed or self.include_unchanged: | 1522 | if changed or self.include_unchanged: |
858 | 1462 | return result | 1523 | return result |
859 | 1463 | # handle unversioned specified paths: | 1524 | # handle unversioned specified paths: |
860 | @@ -1476,7 +1537,8 @@ | |||
861 | 1476 | ) | 1537 | ) |
862 | 1477 | # If we reach here, the outer flow continues, which enters into the | 1538 | # If we reach here, the outer flow continues, which enters into the |
863 | 1478 | # per-root setup logic. | 1539 | # per-root setup logic. |
865 | 1479 | if self.current_dir_info is None and self.current_block is None: | 1540 | if (self.current_dir_info is None and self.current_block is None and not |
866 | 1541 | self.doing_consistency_expansion): | ||
867 | 1480 | # setup iteration of this root: | 1542 | # setup iteration of this root: |
868 | 1481 | self.current_dir_list = None | 1543 | self.current_dir_list = None |
869 | 1482 | if self.root_dir_info and self.root_dir_info[2] == 'tree-reference': | 1544 | if self.root_dir_info and self.root_dir_info[2] == 'tree-reference': |
870 | @@ -1606,6 +1668,8 @@ | |||
871 | 1606 | # advance the entry only, after processing. | 1668 | # advance the entry only, after processing. |
872 | 1607 | result, changed = self._process_entry(current_entry, None) | 1669 | result, changed = self._process_entry(current_entry, None) |
873 | 1608 | if changed is not None: | 1670 | if changed is not None: |
874 | 1671 | if changed: | ||
875 | 1672 | self._gather_result_for_consistency(result) | ||
876 | 1609 | if changed or self.include_unchanged: | 1673 | if changed or self.include_unchanged: |
877 | 1610 | return result | 1674 | return result |
878 | 1611 | self.block_index = self.block_index + 1 | 1675 | self.block_index = self.block_index + 1 |
879 | @@ -1618,6 +1682,15 @@ | |||
880 | 1618 | # More supplied paths to process | 1682 | # More supplied paths to process |
881 | 1619 | self.current_root = None | 1683 | self.current_root = None |
882 | 1620 | return self._iter_next() | 1684 | return self._iter_next() |
883 | 1685 | # Start expanding more conservatively, adding paths the user may not | ||
884 | 1686 | # have intended but required for consistent deltas. | ||
885 | 1687 | self.doing_consistency_expansion = 1 | ||
886 | 1688 | if not self._pending_consistent_entries: | ||
887 | 1689 | self._pending_consistent_entries = self._next_consistent_entries() | ||
888 | 1690 | while self._pending_consistent_entries: | ||
889 | 1691 | result, changed = self._pending_consistent_entries.pop() | ||
890 | 1692 | if changed is not None: | ||
891 | 1693 | return result | ||
892 | 1621 | raise StopIteration() | 1694 | raise StopIteration() |
893 | 1622 | 1695 | ||
894 | 1623 | cdef object _maybe_tree_ref(self, current_path_info): | 1696 | cdef object _maybe_tree_ref(self, current_path_info): |
895 | @@ -1705,6 +1778,8 @@ | |||
896 | 1705 | current_path_info) | 1778 | current_path_info) |
897 | 1706 | if changed is not None: | 1779 | if changed is not None: |
898 | 1707 | path_handled = -1 | 1780 | path_handled = -1 |
899 | 1781 | if not changed and not self.include_unchanged: | ||
900 | 1782 | changed = None | ||
901 | 1708 | # >- loop control starts here: | 1783 | # >- loop control starts here: |
902 | 1709 | # >- entry | 1784 | # >- entry |
903 | 1710 | if advance_entry and current_entry is not None: | 1785 | if advance_entry and current_entry is not None: |
904 | @@ -1726,7 +1801,7 @@ | |||
905 | 1726 | except UnicodeDecodeError: | 1801 | except UnicodeDecodeError: |
906 | 1727 | raise errors.BadFilenameEncoding( | 1802 | raise errors.BadFilenameEncoding( |
907 | 1728 | current_path_info[0], osutils._fs_enc) | 1803 | current_path_info[0], osutils._fs_enc) |
909 | 1729 | if result is not None: | 1804 | if changed is not None: |
910 | 1730 | raise AssertionError( | 1805 | raise AssertionError( |
911 | 1731 | "result is not None: %r" % result) | 1806 | "result is not None: %r" % result) |
912 | 1732 | result = (None, | 1807 | result = (None, |
913 | @@ -1737,6 +1812,7 @@ | |||
914 | 1737 | (None, self.utf8_decode(current_path_info[1])[0]), | 1812 | (None, self.utf8_decode(current_path_info[1])[0]), |
915 | 1738 | (None, current_path_info[2]), | 1813 | (None, current_path_info[2]), |
916 | 1739 | (None, new_executable)) | 1814 | (None, new_executable)) |
917 | 1815 | changed = True | ||
918 | 1740 | # dont descend into this unversioned path if it is | 1816 | # dont descend into this unversioned path if it is |
919 | 1741 | # a dir | 1817 | # a dir |
920 | 1742 | if current_path_info[2] in ('directory'): | 1818 | if current_path_info[2] in ('directory'): |
921 | @@ -1755,9 +1831,12 @@ | |||
922 | 1755 | current_path_info) | 1831 | current_path_info) |
923 | 1756 | else: | 1832 | else: |
924 | 1757 | current_path_info = None | 1833 | current_path_info = None |
926 | 1758 | if result is not None: | 1834 | if changed is not None: |
927 | 1759 | # Found a result on this pass, yield it | 1835 | # Found a result on this pass, yield it |
929 | 1760 | return result | 1836 | if changed: |
930 | 1837 | self._gather_result_for_consistency(result) | ||
931 | 1838 | if changed or self.include_unchanged: | ||
932 | 1839 | return result | ||
933 | 1761 | if self.current_block is not None: | 1840 | if self.current_block is not None: |
934 | 1762 | self.block_index = self.block_index + 1 | 1841 | self.block_index = self.block_index + 1 |
935 | 1763 | self._update_current_block() | 1842 | self._update_current_block() |
936 | @@ -1769,3 +1848,123 @@ | |||
937 | 1769 | self.current_dir_list = self.current_dir_info[1] | 1848 | self.current_dir_list = self.current_dir_info[1] |
938 | 1770 | except StopIteration: | 1849 | except StopIteration: |
939 | 1771 | self.current_dir_info = None | 1850 | self.current_dir_info = None |
940 | 1851 | |||
941 | 1852 | cdef object _next_consistent_entries(self): | ||
942 | 1853 | """Grabs the next specific file parent case to consider. | ||
943 | 1854 | |||
944 | 1855 | :return: A list of the results, each of which is as for _process_entry. | ||
945 | 1856 | """ | ||
946 | 1857 | results = [] | ||
947 | 1858 | while self.search_specific_file_parents: | ||
948 | 1859 | # Process the parent directories for the paths we were iterating. | ||
949 | 1860 | # Even in extremely large trees this should be modest, so currently | ||
950 | 1861 | # no attempt is made to optimise. | ||
951 | 1862 | path_utf8 = self.search_specific_file_parents.pop() | ||
952 | 1863 | if path_utf8 in self.searched_exact_paths: | ||
953 | 1864 | # We've examined this path. | ||
954 | 1865 | continue | ||
955 | 1866 | if osutils.is_inside_any(self.searched_specific_files, path_utf8): | ||
956 | 1867 | # We've examined this path. | ||
957 | 1868 | continue | ||
958 | 1869 | path_entries = self.state._entries_for_path(path_utf8) | ||
959 | 1870 | # We need either one or two entries. If the path in | ||
960 | 1871 | # self.target_index has moved (so the entry in source_index is in | ||
961 | 1872 | # 'ar') then we need to also look for the entry for this path in | ||
962 | 1873 | # self.source_index, to output the appropriate delete-or-rename. | ||
963 | 1874 | selected_entries = [] | ||
964 | 1875 | found_item = False | ||
965 | 1876 | for candidate_entry in path_entries: | ||
966 | 1877 | # Find entries present in target at this path: | ||
967 | 1878 | if candidate_entry[1][self.target_index][0] not in 'ar': | ||
968 | 1879 | found_item = True | ||
969 | 1880 | selected_entries.append(candidate_entry) | ||
970 | 1881 | # Find entries present in source at this path: | ||
971 | 1882 | elif (self.source_index is not None and | ||
972 | 1883 | candidate_entry[1][self.source_index][0] not in 'ar'): | ||
973 | 1884 | found_item = True | ||
974 | 1885 | if candidate_entry[1][self.target_index][0] == 'a': | ||
975 | 1886 | # Deleted, emit it here. | ||
976 | 1887 | selected_entries.append(candidate_entry) | ||
977 | 1888 | else: | ||
978 | 1889 | # renamed, emit it when we process the directory it | ||
979 | 1890 | # ended up at. | ||
980 | 1891 | self.search_specific_file_parents.add( | ||
981 | 1892 | candidate_entry[1][self.target_index][1]) | ||
982 | 1893 | if not found_item: | ||
983 | 1894 | raise AssertionError( | ||
984 | 1895 | "Missing entry for specific path parent %r, %r" % ( | ||
985 | 1896 | path_utf8, path_entries)) | ||
986 | 1897 | path_info = self._path_info(path_utf8, path_utf8.decode('utf8')) | ||
987 | 1898 | for entry in selected_entries: | ||
988 | 1899 | if entry[0][2] in self.seen_ids: | ||
989 | 1900 | continue | ||
990 | 1901 | result, changed = self._process_entry(entry, path_info) | ||
991 | 1902 | if changed is None: | ||
992 | 1903 | raise AssertionError( | ||
993 | 1904 | "Got entry<->path mismatch for specific path " | ||
994 | 1905 | "%r entry %r path_info %r " % ( | ||
995 | 1906 | path_utf8, entry, path_info)) | ||
996 | 1907 | # Only include changes - we're outside the users requested | ||
997 | 1908 | # expansion. | ||
998 | 1909 | if changed: | ||
999 | 1910 | self._gather_result_for_consistency(result) | ||
1000 | 1911 | if (result[6][0] == 'directory' and | ||
1001 | 1912 | result[6][1] != 'directory'): | ||
1002 | 1913 | # This stopped being a directory, the old children have | ||
1003 | 1914 | # to be included. | ||
1004 | 1915 | if entry[1][self.source_index][0] == 'r': | ||
1005 | 1916 | # renamed, take the source path | ||
1006 | 1917 | entry_path_utf8 = entry[1][self.source_index][1] | ||
1007 | 1918 | else: | ||
1008 | 1919 | entry_path_utf8 = path_utf8 | ||
1009 | 1920 | initial_key = (entry_path_utf8, '', '') | ||
1010 | 1921 | block_index, _ = self.state._find_block_index_from_key( | ||
1011 | 1922 | initial_key) | ||
1012 | 1923 | if block_index == 0: | ||
1013 | 1924 | # The children of the root are in block index 1. | ||
1014 | 1925 | block_index = block_index + 1 | ||
1015 | 1926 | current_block = None | ||
1016 | 1927 | if block_index < len(self.state._dirblocks): | ||
1017 | 1928 | current_block = self.state._dirblocks[block_index] | ||
1018 | 1929 | if not osutils.is_inside( | ||
1019 | 1930 | entry_path_utf8, current_block[0]): | ||
1020 | 1931 | # No entries for this directory at all. | ||
1021 | 1932 | current_block = None | ||
1022 | 1933 | if current_block is not None: | ||
1023 | 1934 | for entry in current_block[1]: | ||
1024 | 1935 | if entry[1][self.source_index][0] in 'ar': | ||
1025 | 1936 | # Not in the source tree, so doesn't have to be | ||
1026 | 1937 | # included. | ||
1027 | 1938 | continue | ||
1028 | 1939 | # Path of the entry itself. | ||
1029 | 1940 | self.search_specific_file_parents.add( | ||
1030 | 1941 | self.pathjoin(*entry[0][:2])) | ||
1031 | 1942 | if changed or self.include_unchanged: | ||
1032 | 1943 | results.append((result, changed)) | ||
1033 | 1944 | self.searched_exact_paths.add(path_utf8) | ||
1034 | 1945 | return results | ||
1035 | 1946 | |||
1036 | 1947 | cdef object _path_info(self, utf8_path, unicode_path): | ||
1037 | 1948 | """Generate path_info for unicode_path. | ||
1038 | 1949 | |||
1039 | 1950 | :return: None if unicode_path does not exist, or a path_info tuple. | ||
1040 | 1951 | """ | ||
1041 | 1952 | abspath = self.tree.abspath(unicode_path) | ||
1042 | 1953 | try: | ||
1043 | 1954 | stat = os.lstat(abspath) | ||
1044 | 1955 | except OSError, e: | ||
1045 | 1956 | if e.errno == errno.ENOENT: | ||
1046 | 1957 | # the path does not exist. | ||
1047 | 1958 | return None | ||
1048 | 1959 | else: | ||
1049 | 1960 | raise | ||
1050 | 1961 | utf8_basename = utf8_path.rsplit('/', 1)[-1] | ||
1051 | 1962 | dir_info = (utf8_path, utf8_basename, | ||
1052 | 1963 | osutils.file_kind_from_stat_mode(stat.st_mode), stat, | ||
1053 | 1964 | abspath) | ||
1054 | 1965 | if dir_info[2] == 'directory': | ||
1055 | 1966 | if self.tree._directory_is_tree_reference( | ||
1056 | 1967 | unicode_path): | ||
1057 | 1968 | self.root_dir_info = self.root_dir_info[:2] + \ | ||
1058 | 1969 | ('tree-reference',) + self.root_dir_info[3:] | ||
1059 | 1970 | return dir_info | ||
1060 | 1772 | 1971 | ||
1061 | === modified file 'bzrlib/_known_graph_py.py' | |||
1062 | --- bzrlib/_known_graph_py.py 2009-07-08 20:58:10 +0000 | |||
1063 | +++ bzrlib/_known_graph_py.py 2009-09-07 14:19:05 +0000 | |||
1064 | @@ -18,6 +18,7 @@ | |||
1065 | 18 | """ | 18 | """ |
1066 | 19 | 19 | ||
1067 | 20 | from bzrlib import ( | 20 | from bzrlib import ( |
1068 | 21 | errors, | ||
1069 | 21 | revision, | 22 | revision, |
1070 | 22 | ) | 23 | ) |
1071 | 23 | 24 | ||
1072 | @@ -40,6 +41,18 @@ | |||
1073 | 40 | self.parent_keys, self.child_keys) | 41 | self.parent_keys, self.child_keys) |
1074 | 41 | 42 | ||
1075 | 42 | 43 | ||
1076 | 44 | class _MergeSortNode(object): | ||
1077 | 45 | """Information about a specific node in the merge graph.""" | ||
1078 | 46 | |||
1079 | 47 | __slots__ = ('key', 'merge_depth', 'revno', 'end_of_merge') | ||
1080 | 48 | |||
1081 | 49 | def __init__(self, key, merge_depth, revno, end_of_merge): | ||
1082 | 50 | self.key = key | ||
1083 | 51 | self.merge_depth = merge_depth | ||
1084 | 52 | self.revno = revno | ||
1085 | 53 | self.end_of_merge = end_of_merge | ||
1086 | 54 | |||
1087 | 55 | |||
1088 | 43 | class KnownGraph(object): | 56 | class KnownGraph(object): |
1089 | 44 | """This is a class which assumes we already know the full graph.""" | 57 | """This is a class which assumes we already know the full graph.""" |
1090 | 45 | 58 | ||
1091 | @@ -84,6 +97,10 @@ | |||
1092 | 84 | return [node for node in self._nodes.itervalues() | 97 | return [node for node in self._nodes.itervalues() |
1093 | 85 | if not node.parent_keys] | 98 | if not node.parent_keys] |
1094 | 86 | 99 | ||
1095 | 100 | def _find_tips(self): | ||
1096 | 101 | return [node for node in self._nodes.itervalues() | ||
1097 | 102 | if not node.child_keys] | ||
1098 | 103 | |||
1099 | 87 | def _find_gdfo(self): | 104 | def _find_gdfo(self): |
1100 | 88 | nodes = self._nodes | 105 | nodes = self._nodes |
1101 | 89 | known_parent_gdfos = {} | 106 | known_parent_gdfos = {} |
1102 | @@ -171,3 +188,119 @@ | |||
1103 | 171 | self._known_heads[heads_key] = heads | 188 | self._known_heads[heads_key] = heads |
1104 | 172 | return heads | 189 | return heads |
1105 | 173 | 190 | ||
1106 | 191 | def topo_sort(self): | ||
1107 | 192 | """Return the nodes in topological order. | ||
1108 | 193 | |||
1109 | 194 | All parents must occur before all children. | ||
1110 | 195 | """ | ||
1111 | 196 | for node in self._nodes.itervalues(): | ||
1112 | 197 | if node.gdfo is None: | ||
1113 | 198 | raise errors.GraphCycleError(self._nodes) | ||
1114 | 199 | pending = self._find_tails() | ||
1115 | 200 | pending_pop = pending.pop | ||
1116 | 201 | pending_append = pending.append | ||
1117 | 202 | |||
1118 | 203 | topo_order = [] | ||
1119 | 204 | topo_order_append = topo_order.append | ||
1120 | 205 | |||
1121 | 206 | num_seen_parents = dict.fromkeys(self._nodes, 0) | ||
1122 | 207 | while pending: | ||
1123 | 208 | node = pending_pop() | ||
1124 | 209 | if node.parent_keys is not None: | ||
1125 | 210 | # We don't include ghost parents | ||
1126 | 211 | topo_order_append(node.key) | ||
1127 | 212 | for child_key in node.child_keys: | ||
1128 | 213 | child_node = self._nodes[child_key] | ||
1129 | 214 | seen_parents = num_seen_parents[child_key] + 1 | ||
1130 | 215 | if seen_parents == len(child_node.parent_keys): | ||
1131 | 216 | # All parents have been processed, enqueue this child | ||
1132 | 217 | pending_append(child_node) | ||
1133 | 218 | # This has been queued up, stop tracking it | ||
1134 | 219 | del num_seen_parents[child_key] | ||
1135 | 220 | else: | ||
1136 | 221 | num_seen_parents[child_key] = seen_parents | ||
1137 | 222 | # We started from the parents, so we don't need to do anymore work | ||
1138 | 223 | return topo_order | ||
1139 | 224 | |||
1140 | 225 | def gc_sort(self): | ||
1141 | 226 | """Return a reverse topological ordering which is 'stable'. | ||
1142 | 227 | |||
1143 | 228 | There are a few constraints: | ||
1144 | 229 | 1) Reverse topological (all children before all parents) | ||
1145 | 230 | 2) Grouped by prefix | ||
1146 | 231 | 3) 'stable' sorting, so that we get the same result, independent of | ||
1147 | 232 | machine, or extra data. | ||
1148 | 233 | To do this, we use the same basic algorithm as topo_sort, but when we | ||
1149 | 234 | aren't sure what node to access next, we sort them lexicographically. | ||
1150 | 235 | """ | ||
1151 | 236 | tips = self._find_tips() | ||
1152 | 237 | # Split the tips based on prefix | ||
1153 | 238 | prefix_tips = {} | ||
1154 | 239 | for node in tips: | ||
1155 | 240 | if node.key.__class__ is str or len(node.key) == 1: | ||
1156 | 241 | prefix = '' | ||
1157 | 242 | else: | ||
1158 | 243 | prefix = node.key[0] | ||
1159 | 244 | prefix_tips.setdefault(prefix, []).append(node) | ||
1160 | 245 | |||
1161 | 246 | num_seen_children = dict.fromkeys(self._nodes, 0) | ||
1162 | 247 | |||
1163 | 248 | result = [] | ||
1164 | 249 | for prefix in sorted(prefix_tips): | ||
1165 | 250 | pending = sorted(prefix_tips[prefix], key=lambda n:n.key, | ||
1166 | 251 | reverse=True) | ||
1167 | 252 | while pending: | ||
1168 | 253 | node = pending.pop() | ||
1169 | 254 | if node.parent_keys is None: | ||
1170 | 255 | # Ghost node, skip it | ||
1171 | 256 | continue | ||
1172 | 257 | result.append(node.key) | ||
1173 | 258 | for parent_key in sorted(node.parent_keys, reverse=True): | ||
1174 | 259 | parent_node = self._nodes[parent_key] | ||
1175 | 260 | seen_children = num_seen_children[parent_key] + 1 | ||
1176 | 261 | if seen_children == len(parent_node.child_keys): | ||
1177 | 262 | # All children have been processed, enqueue this parent | ||
1178 | 263 | pending.append(parent_node) | ||
1179 | 264 | # This has been queued up, stop tracking it | ||
1180 | 265 | del num_seen_children[parent_key] | ||
1181 | 266 | else: | ||
1182 | 267 | num_seen_children[parent_key] = seen_children | ||
1183 | 268 | return result | ||
1184 | 269 | |||
1185 | 270 | def merge_sort(self, tip_key): | ||
1186 | 271 | """Compute the merge sorted graph output.""" | ||
1187 | 272 | from bzrlib import tsort | ||
1188 | 273 | as_parent_map = dict((node.key, node.parent_keys) | ||
1189 | 274 | for node in self._nodes.itervalues() | ||
1190 | 275 | if node.parent_keys is not None) | ||
1191 | 276 | # We intentionally always generate revnos and never force the | ||
1192 | 277 | # mainline_revisions | ||
1193 | 278 | # Strip the sequence_number that merge_sort generates | ||
1194 | 279 | return [_MergeSortNode(key, merge_depth, revno, end_of_merge) | ||
1195 | 280 | for _, key, merge_depth, revno, end_of_merge | ||
1196 | 281 | in tsort.merge_sort(as_parent_map, tip_key, | ||
1197 | 282 | mainline_revisions=None, | ||
1198 | 283 | generate_revno=True)] | ||
1199 | 284 | |||
1200 | 285 | def get_parent_keys(self, key): | ||
1201 | 286 | """Get the parents for a key | ||
1202 | 287 | |||
1203 | 288 | Returns a list containg the parents keys. If the key is a ghost, | ||
1204 | 289 | None is returned. A KeyError will be raised if the key is not in | ||
1205 | 290 | the graph. | ||
1206 | 291 | |||
1207 | 292 | :param keys: Key to check (eg revision_id) | ||
1208 | 293 | :return: A list of parents | ||
1209 | 294 | """ | ||
1210 | 295 | return self._nodes[key].parent_keys | ||
1211 | 296 | |||
1212 | 297 | def get_child_keys(self, key): | ||
1213 | 298 | """Get the children for a key | ||
1214 | 299 | |||
1215 | 300 | Returns a list containg the children keys. A KeyError will be raised | ||
1216 | 301 | if the key is not in the graph. | ||
1217 | 302 | |||
1218 | 303 | :param keys: Key to check (eg revision_id) | ||
1219 | 304 | :return: A list of children | ||
1220 | 305 | """ | ||
1221 | 306 | return self._nodes[key].child_keys | ||
1222 | 174 | 307 | ||
1223 | === modified file 'bzrlib/_known_graph_pyx.pyx' | |||
1224 | --- bzrlib/_known_graph_pyx.pyx 2009-07-14 16:10:32 +0000 | |||
1225 | +++ bzrlib/_known_graph_pyx.pyx 2009-09-07 14:19:05 +0000 | |||
1226 | @@ -25,11 +25,18 @@ | |||
1227 | 25 | ctypedef struct PyObject: | 25 | ctypedef struct PyObject: |
1228 | 26 | pass | 26 | pass |
1229 | 27 | 27 | ||
1230 | 28 | int PyString_CheckExact(object) | ||
1231 | 29 | |||
1232 | 30 | int PyObject_RichCompareBool(object, object, int) | ||
1233 | 31 | int Py_LT | ||
1234 | 32 | |||
1235 | 33 | int PyTuple_CheckExact(object) | ||
1236 | 28 | object PyTuple_New(Py_ssize_t n) | 34 | object PyTuple_New(Py_ssize_t n) |
1237 | 29 | Py_ssize_t PyTuple_GET_SIZE(object t) | 35 | Py_ssize_t PyTuple_GET_SIZE(object t) |
1238 | 30 | PyObject * PyTuple_GET_ITEM(object t, Py_ssize_t o) | 36 | PyObject * PyTuple_GET_ITEM(object t, Py_ssize_t o) |
1239 | 31 | void PyTuple_SET_ITEM(object t, Py_ssize_t o, object v) | 37 | void PyTuple_SET_ITEM(object t, Py_ssize_t o, object v) |
1240 | 32 | 38 | ||
1241 | 39 | int PyList_CheckExact(object) | ||
1242 | 33 | Py_ssize_t PyList_GET_SIZE(object l) | 40 | Py_ssize_t PyList_GET_SIZE(object l) |
1243 | 34 | PyObject * PyList_GET_ITEM(object l, Py_ssize_t o) | 41 | PyObject * PyList_GET_ITEM(object l, Py_ssize_t o) |
1244 | 35 | int PyList_SetItem(object l, Py_ssize_t o, object l) except -1 | 42 | int PyList_SetItem(object l, Py_ssize_t o, object l) except -1 |
1245 | @@ -44,8 +51,9 @@ | |||
1246 | 44 | 51 | ||
1247 | 45 | void Py_INCREF(object) | 52 | void Py_INCREF(object) |
1248 | 46 | 53 | ||
1249 | 54 | import gc | ||
1250 | 47 | 55 | ||
1252 | 48 | from bzrlib import revision | 56 | from bzrlib import errors, revision |
1253 | 49 | 57 | ||
1254 | 50 | cdef object NULL_REVISION | 58 | cdef object NULL_REVISION |
1255 | 51 | NULL_REVISION = revision.NULL_REVISION | 59 | NULL_REVISION = revision.NULL_REVISION |
1256 | @@ -59,10 +67,9 @@ | |||
1257 | 59 | cdef object children | 67 | cdef object children |
1258 | 60 | cdef public long gdfo | 68 | cdef public long gdfo |
1259 | 61 | cdef int seen | 69 | cdef int seen |
1260 | 70 | cdef object extra | ||
1261 | 62 | 71 | ||
1262 | 63 | def __init__(self, key): | 72 | def __init__(self, key): |
1263 | 64 | cdef int i | ||
1264 | 65 | |||
1265 | 66 | self.key = key | 73 | self.key = key |
1266 | 67 | self.parents = None | 74 | self.parents = None |
1267 | 68 | 75 | ||
1268 | @@ -70,6 +77,7 @@ | |||
1269 | 70 | # Greatest distance from origin | 77 | # Greatest distance from origin |
1270 | 71 | self.gdfo = -1 | 78 | self.gdfo = -1 |
1271 | 72 | self.seen = 0 | 79 | self.seen = 0 |
1272 | 80 | self.extra = None | ||
1273 | 73 | 81 | ||
1274 | 74 | property child_keys: | 82 | property child_keys: |
1275 | 75 | def __get__(self): | 83 | def __get__(self): |
1276 | @@ -80,6 +88,18 @@ | |||
1277 | 80 | PyList_Append(keys, child.key) | 88 | PyList_Append(keys, child.key) |
1278 | 81 | return keys | 89 | return keys |
1279 | 82 | 90 | ||
1280 | 91 | property parent_keys: | ||
1281 | 92 | def __get__(self): | ||
1282 | 93 | if self.parents is None: | ||
1283 | 94 | return None | ||
1284 | 95 | |||
1285 | 96 | cdef _KnownGraphNode parent | ||
1286 | 97 | |||
1287 | 98 | keys = [] | ||
1288 | 99 | for parent in self.parents: | ||
1289 | 100 | PyList_Append(keys, parent.key) | ||
1290 | 101 | return keys | ||
1291 | 102 | |||
1292 | 83 | cdef clear_references(self): | 103 | cdef clear_references(self): |
1293 | 84 | self.parents = None | 104 | self.parents = None |
1294 | 85 | self.children = None | 105 | self.children = None |
1295 | @@ -107,17 +127,66 @@ | |||
1296 | 107 | return <_KnownGraphNode>temp_node | 127 | return <_KnownGraphNode>temp_node |
1297 | 108 | 128 | ||
1298 | 109 | 129 | ||
1300 | 110 | cdef _KnownGraphNode _get_parent(parents, Py_ssize_t pos): | 130 | cdef _KnownGraphNode _get_tuple_node(tpl, Py_ssize_t pos): |
1301 | 111 | cdef PyObject *temp_node | 131 | cdef PyObject *temp_node |
1302 | 112 | cdef _KnownGraphNode node | ||
1303 | 113 | 132 | ||
1305 | 114 | temp_node = PyTuple_GET_ITEM(parents, pos) | 133 | temp_node = PyTuple_GET_ITEM(tpl, pos) |
1306 | 115 | return <_KnownGraphNode>temp_node | 134 | return <_KnownGraphNode>temp_node |
1307 | 116 | 135 | ||
1308 | 117 | 136 | ||
1312 | 118 | # TODO: slab allocate all _KnownGraphNode objects. | 137 | def get_key(node): |
1313 | 119 | # We already know how many we are going to need, except for a couple of | 138 | cdef _KnownGraphNode real_node |
1314 | 120 | # ghosts that could be allocated on demand. | 139 | real_node = node |
1315 | 140 | return real_node.key | ||
1316 | 141 | |||
1317 | 142 | |||
1318 | 143 | cdef object _sort_list_nodes(object lst_or_tpl, int reverse): | ||
1319 | 144 | """Sort a list of _KnownGraphNode objects. | ||
1320 | 145 | |||
1321 | 146 | If lst_or_tpl is a list, it is allowed to mutate in place. It may also | ||
1322 | 147 | just return the input list if everything is already sorted. | ||
1323 | 148 | """ | ||
1324 | 149 | cdef _KnownGraphNode node1, node2 | ||
1325 | 150 | cdef int do_swap, is_tuple | ||
1326 | 151 | cdef Py_ssize_t length | ||
1327 | 152 | |||
1328 | 153 | is_tuple = PyTuple_CheckExact(lst_or_tpl) | ||
1329 | 154 | if not (is_tuple or PyList_CheckExact(lst_or_tpl)): | ||
1330 | 155 | raise TypeError('lst_or_tpl must be a list or tuple.') | ||
1331 | 156 | length = len(lst_or_tpl) | ||
1332 | 157 | if length == 0 or length == 1: | ||
1333 | 158 | return lst_or_tpl | ||
1334 | 159 | if length == 2: | ||
1335 | 160 | if is_tuple: | ||
1336 | 161 | node1 = _get_tuple_node(lst_or_tpl, 0) | ||
1337 | 162 | node2 = _get_tuple_node(lst_or_tpl, 1) | ||
1338 | 163 | else: | ||
1339 | 164 | node1 = _get_list_node(lst_or_tpl, 0) | ||
1340 | 165 | node2 = _get_list_node(lst_or_tpl, 1) | ||
1341 | 166 | if reverse: | ||
1342 | 167 | do_swap = PyObject_RichCompareBool(node1.key, node2.key, Py_LT) | ||
1343 | 168 | else: | ||
1344 | 169 | do_swap = PyObject_RichCompareBool(node2.key, node1.key, Py_LT) | ||
1345 | 170 | if not do_swap: | ||
1346 | 171 | return lst_or_tpl | ||
1347 | 172 | if is_tuple: | ||
1348 | 173 | return (node2, node1) | ||
1349 | 174 | else: | ||
1350 | 175 | # Swap 'in-place', since lists are mutable | ||
1351 | 176 | Py_INCREF(node1) | ||
1352 | 177 | PyList_SetItem(lst_or_tpl, 1, node1) | ||
1353 | 178 | Py_INCREF(node2) | ||
1354 | 179 | PyList_SetItem(lst_or_tpl, 0, node2) | ||
1355 | 180 | return lst_or_tpl | ||
1356 | 181 | # For all other sizes, we just use 'sorted()' | ||
1357 | 182 | if is_tuple: | ||
1358 | 183 | # Note that sorted() is just list(iterable).sort() | ||
1359 | 184 | lst_or_tpl = list(lst_or_tpl) | ||
1360 | 185 | lst_or_tpl.sort(key=get_key, reverse=reverse) | ||
1361 | 186 | return lst_or_tpl | ||
1362 | 187 | |||
1363 | 188 | |||
1364 | 189 | cdef class _MergeSorter | ||
1365 | 121 | 190 | ||
1366 | 122 | cdef class KnownGraph: | 191 | cdef class KnownGraph: |
1367 | 123 | """This is a class which assumes we already know the full graph.""" | 192 | """This is a class which assumes we already know the full graph.""" |
1368 | @@ -136,6 +205,9 @@ | |||
1369 | 136 | # Maps {sorted(revision_id, revision_id): heads} | 205 | # Maps {sorted(revision_id, revision_id): heads} |
1370 | 137 | self._known_heads = {} | 206 | self._known_heads = {} |
1371 | 138 | self.do_cache = int(do_cache) | 207 | self.do_cache = int(do_cache) |
1372 | 208 | # TODO: consider disabling gc since we are allocating a lot of nodes | ||
1373 | 209 | # that won't be collectable anyway. real world testing has not | ||
1374 | 210 | # shown a specific impact, yet. | ||
1375 | 139 | self._initialize_nodes(parent_map) | 211 | self._initialize_nodes(parent_map) |
1376 | 140 | self._find_gdfo() | 212 | self._find_gdfo() |
1377 | 141 | 213 | ||
1378 | @@ -183,11 +255,16 @@ | |||
1379 | 183 | parent_keys = <object>temp_parent_keys | 255 | parent_keys = <object>temp_parent_keys |
1380 | 184 | num_parent_keys = len(parent_keys) | 256 | num_parent_keys = len(parent_keys) |
1381 | 185 | node = self._get_or_create_node(key) | 257 | node = self._get_or_create_node(key) |
1384 | 186 | # We know how many parents, so we could pre allocate an exact sized | 258 | # We know how many parents, so we pre allocate the tuple |
1383 | 187 | # tuple here | ||
1385 | 188 | parent_nodes = PyTuple_New(num_parent_keys) | 259 | parent_nodes = PyTuple_New(num_parent_keys) |
1386 | 189 | # We use iter here, because parent_keys maybe be a list or tuple | ||
1387 | 190 | for pos2 from 0 <= pos2 < num_parent_keys: | 260 | for pos2 from 0 <= pos2 < num_parent_keys: |
1388 | 261 | # Note: it costs us 10ms out of 40ms to lookup all of these | ||
1389 | 262 | # parents, it doesn't seem to be an allocation overhead, | ||
1390 | 263 | # but rather a lookup overhead. There doesn't seem to be | ||
1391 | 264 | # a way around it, and that is one reason why | ||
1392 | 265 | # KnownGraphNode maintains a direct pointer to the parent | ||
1393 | 266 | # node. | ||
1394 | 267 | # We use [] because parent_keys may be a tuple or list | ||
1395 | 191 | parent_node = self._get_or_create_node(parent_keys[pos2]) | 268 | parent_node = self._get_or_create_node(parent_keys[pos2]) |
1396 | 192 | # PyTuple_SET_ITEM will steal a reference, so INCREF first | 269 | # PyTuple_SET_ITEM will steal a reference, so INCREF first |
1397 | 193 | Py_INCREF(parent_node) | 270 | Py_INCREF(parent_node) |
1398 | @@ -209,6 +286,19 @@ | |||
1399 | 209 | PyList_Append(tails, node) | 286 | PyList_Append(tails, node) |
1400 | 210 | return tails | 287 | return tails |
1401 | 211 | 288 | ||
1402 | 289 | def _find_tips(self): | ||
1403 | 290 | cdef PyObject *temp_node | ||
1404 | 291 | cdef _KnownGraphNode node | ||
1405 | 292 | cdef Py_ssize_t pos | ||
1406 | 293 | |||
1407 | 294 | tips = [] | ||
1408 | 295 | pos = 0 | ||
1409 | 296 | while PyDict_Next(self._nodes, &pos, NULL, &temp_node): | ||
1410 | 297 | node = <_KnownGraphNode>temp_node | ||
1411 | 298 | if PyList_GET_SIZE(node.children) == 0: | ||
1412 | 299 | PyList_Append(tips, node) | ||
1413 | 300 | return tips | ||
1414 | 301 | |||
1415 | 212 | def _find_gdfo(self): | 302 | def _find_gdfo(self): |
1416 | 213 | cdef _KnownGraphNode node | 303 | cdef _KnownGraphNode node |
1417 | 214 | cdef _KnownGraphNode child | 304 | cdef _KnownGraphNode child |
1418 | @@ -315,7 +405,7 @@ | |||
1419 | 315 | continue | 405 | continue |
1420 | 316 | if node.parents is not None and PyTuple_GET_SIZE(node.parents) > 0: | 406 | if node.parents is not None and PyTuple_GET_SIZE(node.parents) > 0: |
1421 | 317 | for pos from 0 <= pos < PyTuple_GET_SIZE(node.parents): | 407 | for pos from 0 <= pos < PyTuple_GET_SIZE(node.parents): |
1423 | 318 | parent_node = _get_parent(node.parents, pos) | 408 | parent_node = _get_tuple_node(node.parents, pos) |
1424 | 319 | last_item = last_item + 1 | 409 | last_item = last_item + 1 |
1425 | 320 | if last_item < PyList_GET_SIZE(pending): | 410 | if last_item < PyList_GET_SIZE(pending): |
1426 | 321 | Py_INCREF(parent_node) # SetItem steals a ref | 411 | Py_INCREF(parent_node) # SetItem steals a ref |
1427 | @@ -335,3 +425,454 @@ | |||
1428 | 335 | if self.do_cache: | 425 | if self.do_cache: |
1429 | 336 | PyDict_SetItem(self._known_heads, heads_key, heads) | 426 | PyDict_SetItem(self._known_heads, heads_key, heads) |
1430 | 337 | return heads | 427 | return heads |
1431 | 428 | |||
1432 | 429 | def topo_sort(self): | ||
1433 | 430 | """Return the nodes in topological order. | ||
1434 | 431 | |||
1435 | 432 | All parents must occur before all children. | ||
1436 | 433 | """ | ||
1437 | 434 | # This is, for the most part, the same iteration order that we used for | ||
1438 | 435 | # _find_gdfo, consider finding a way to remove the duplication | ||
1439 | 436 | # In general, we find the 'tails' (nodes with no parents), and then | ||
1440 | 437 | # walk to the children. For children that have all of their parents | ||
1441 | 438 | # yielded, we queue up the child to be yielded as well. | ||
1442 | 439 | cdef _KnownGraphNode node | ||
1443 | 440 | cdef _KnownGraphNode child | ||
1444 | 441 | cdef PyObject *temp | ||
1445 | 442 | cdef Py_ssize_t pos | ||
1446 | 443 | cdef int replace | ||
1447 | 444 | cdef Py_ssize_t last_item | ||
1448 | 445 | |||
1449 | 446 | pending = self._find_tails() | ||
1450 | 447 | if PyList_GET_SIZE(pending) == 0 and len(self._nodes) > 0: | ||
1451 | 448 | raise errors.GraphCycleError(self._nodes) | ||
1452 | 449 | |||
1453 | 450 | topo_order = [] | ||
1454 | 451 | |||
1455 | 452 | last_item = PyList_GET_SIZE(pending) - 1 | ||
1456 | 453 | while last_item >= 0: | ||
1457 | 454 | # Avoid pop followed by push, instead, peek, and replace | ||
1458 | 455 | # timing shows this is 930ms => 770ms for OOo | ||
1459 | 456 | node = _get_list_node(pending, last_item) | ||
1460 | 457 | last_item = last_item - 1 | ||
1461 | 458 | if node.parents is not None: | ||
1462 | 459 | # We don't include ghost parents | ||
1463 | 460 | PyList_Append(topo_order, node.key) | ||
1464 | 461 | for pos from 0 <= pos < PyList_GET_SIZE(node.children): | ||
1465 | 462 | child = _get_list_node(node.children, pos) | ||
1466 | 463 | if child.gdfo == -1: | ||
1467 | 464 | # We know we have a graph cycle because a node has a parent | ||
1468 | 465 | # which we couldn't find | ||
1469 | 466 | raise errors.GraphCycleError(self._nodes) | ||
1470 | 467 | child.seen = child.seen + 1 | ||
1471 | 468 | if child.seen == PyTuple_GET_SIZE(child.parents): | ||
1472 | 469 | # All parents of this child have been yielded, queue this | ||
1473 | 470 | # one to be yielded as well | ||
1474 | 471 | last_item = last_item + 1 | ||
1475 | 472 | if last_item < PyList_GET_SIZE(pending): | ||
1476 | 473 | Py_INCREF(child) # SetItem steals a ref | ||
1477 | 474 | PyList_SetItem(pending, last_item, child) | ||
1478 | 475 | else: | ||
1479 | 476 | PyList_Append(pending, child) | ||
1480 | 477 | # We have queued this node, we don't need to track it | ||
1481 | 478 | # anymore | ||
1482 | 479 | child.seen = 0 | ||
1483 | 480 | # We started from the parents, so we don't need to do anymore work | ||
1484 | 481 | return topo_order | ||
1485 | 482 | |||
1486 | 483 | def gc_sort(self): | ||
1487 | 484 | """Return a reverse topological ordering which is 'stable'. | ||
1488 | 485 | |||
1489 | 486 | There are a few constraints: | ||
1490 | 487 | 1) Reverse topological (all children before all parents) | ||
1491 | 488 | 2) Grouped by prefix | ||
1492 | 489 | 3) 'stable' sorting, so that we get the same result, independent of | ||
1493 | 490 | machine, or extra data. | ||
1494 | 491 | To do this, we use the same basic algorithm as topo_sort, but when we | ||
1495 | 492 | aren't sure what node to access next, we sort them lexicographically. | ||
1496 | 493 | """ | ||
1497 | 494 | cdef PyObject *temp | ||
1498 | 495 | cdef Py_ssize_t pos, last_item | ||
1499 | 496 | cdef _KnownGraphNode node, node2, parent_node | ||
1500 | 497 | |||
1501 | 498 | tips = self._find_tips() | ||
1502 | 499 | # Split the tips based on prefix | ||
1503 | 500 | prefix_tips = {} | ||
1504 | 501 | for pos from 0 <= pos < PyList_GET_SIZE(tips): | ||
1505 | 502 | node = _get_list_node(tips, pos) | ||
1506 | 503 | if PyString_CheckExact(node.key) or len(node.key) == 1: | ||
1507 | 504 | prefix = '' | ||
1508 | 505 | else: | ||
1509 | 506 | prefix = node.key[0] | ||
1510 | 507 | temp = PyDict_GetItem(prefix_tips, prefix) | ||
1511 | 508 | if temp == NULL: | ||
1512 | 509 | prefix_tips[prefix] = [node] | ||
1513 | 510 | else: | ||
1514 | 511 | tip_nodes = <object>temp | ||
1515 | 512 | PyList_Append(tip_nodes, node) | ||
1516 | 513 | |||
1517 | 514 | result = [] | ||
1518 | 515 | for prefix in sorted(prefix_tips): | ||
1519 | 516 | temp = PyDict_GetItem(prefix_tips, prefix) | ||
1520 | 517 | assert temp != NULL | ||
1521 | 518 | tip_nodes = <object>temp | ||
1522 | 519 | pending = _sort_list_nodes(tip_nodes, 1) | ||
1523 | 520 | last_item = PyList_GET_SIZE(pending) - 1 | ||
1524 | 521 | while last_item >= 0: | ||
1525 | 522 | node = _get_list_node(pending, last_item) | ||
1526 | 523 | last_item = last_item - 1 | ||
1527 | 524 | if node.parents is None: | ||
1528 | 525 | # Ghost | ||
1529 | 526 | continue | ||
1530 | 527 | PyList_Append(result, node.key) | ||
1531 | 528 | # Sorting the parent keys isn't strictly necessary for stable | ||
1532 | 529 | # sorting of a given graph. But it does help minimize the | ||
1533 | 530 | # differences between graphs | ||
1534 | 531 | # For bzr.dev ancestry: | ||
1535 | 532 | # 4.73ms no sort | ||
1536 | 533 | # 7.73ms RichCompareBool sort | ||
1537 | 534 | parents = _sort_list_nodes(node.parents, 1) | ||
1538 | 535 | for pos from 0 <= pos < len(parents): | ||
1539 | 536 | if PyTuple_CheckExact(parents): | ||
1540 | 537 | parent_node = _get_tuple_node(parents, pos) | ||
1541 | 538 | else: | ||
1542 | 539 | parent_node = _get_list_node(parents, pos) | ||
1543 | 540 | # TODO: GraphCycle detection | ||
1544 | 541 | parent_node.seen = parent_node.seen + 1 | ||
1545 | 542 | if (parent_node.seen | ||
1546 | 543 | == PyList_GET_SIZE(parent_node.children)): | ||
1547 | 544 | # All children have been processed, queue up this | ||
1548 | 545 | # parent | ||
1549 | 546 | last_item = last_item + 1 | ||
1550 | 547 | if last_item < PyList_GET_SIZE(pending): | ||
1551 | 548 | Py_INCREF(parent_node) # SetItem steals a ref | ||
1552 | 549 | PyList_SetItem(pending, last_item, parent_node) | ||
1553 | 550 | else: | ||
1554 | 551 | PyList_Append(pending, parent_node) | ||
1555 | 552 | parent_node.seen = 0 | ||
1556 | 553 | return result | ||
1557 | 554 | |||
1558 | 555 | def merge_sort(self, tip_key): | ||
1559 | 556 | """Compute the merge sorted graph output.""" | ||
1560 | 557 | cdef _MergeSorter sorter | ||
1561 | 558 | |||
1562 | 559 | # TODO: consider disabling gc since we are allocating a lot of nodes | ||
1563 | 560 | # that won't be collectable anyway. real world testing has not | ||
1564 | 561 | # shown a specific impact, yet. | ||
1565 | 562 | sorter = _MergeSorter(self, tip_key) | ||
1566 | 563 | return sorter.topo_order() | ||
1567 | 564 | |||
1568 | 565 | def get_parent_keys(self, key): | ||
1569 | 566 | """Get the parents for a key | ||
1570 | 567 | |||
1571 | 568 | Returns a list containg the parents keys. If the key is a ghost, | ||
1572 | 569 | None is returned. A KeyError will be raised if the key is not in | ||
1573 | 570 | the graph. | ||
1574 | 571 | |||
1575 | 572 | :param keys: Key to check (eg revision_id) | ||
1576 | 573 | :return: A list of parents | ||
1577 | 574 | """ | ||
1578 | 575 | return self._nodes[key].parent_keys | ||
1579 | 576 | |||
1580 | 577 | def get_child_keys(self, key): | ||
1581 | 578 | """Get the children for a key | ||
1582 | 579 | |||
1583 | 580 | Returns a list containg the children keys. A KeyError will be raised | ||
1584 | 581 | if the key is not in the graph. | ||
1585 | 582 | |||
1586 | 583 | :param keys: Key to check (eg revision_id) | ||
1587 | 584 | :return: A list of children | ||
1588 | 585 | """ | ||
1589 | 586 | return self._nodes[key].child_keys | ||
1590 | 587 | |||
1591 | 588 | |||
1592 | 589 | cdef class _MergeSortNode: | ||
1593 | 590 | """Tracks information about a node during the merge_sort operation.""" | ||
1594 | 591 | |||
1595 | 592 | # Public api | ||
1596 | 593 | cdef public object key | ||
1597 | 594 | cdef public long merge_depth | ||
1598 | 595 | cdef public object end_of_merge # True/False Is this the end of the current merge | ||
1599 | 596 | |||
1600 | 597 | # Private api, used while computing the information | ||
1601 | 598 | cdef _KnownGraphNode left_parent | ||
1602 | 599 | cdef _KnownGraphNode left_pending_parent | ||
1603 | 600 | cdef object pending_parents # list of _KnownGraphNode for non-left parents | ||
1604 | 601 | cdef long _revno_first | ||
1605 | 602 | cdef long _revno_second | ||
1606 | 603 | cdef long _revno_last | ||
1607 | 604 | # TODO: turn these into flag/bit fields rather than individual members | ||
1608 | 605 | cdef int is_first_child # Is this the first child? | ||
1609 | 606 | cdef int seen_by_child # A child node has seen this parent | ||
1610 | 607 | cdef int completed # Fully Processed | ||
1611 | 608 | |||
1612 | 609 | def __init__(self, key): | ||
1613 | 610 | self.key = key | ||
1614 | 611 | self.merge_depth = -1 | ||
1615 | 612 | self.left_parent = None | ||
1616 | 613 | self.left_pending_parent = None | ||
1617 | 614 | self.pending_parents = None | ||
1618 | 615 | self._revno_first = -1 | ||
1619 | 616 | self._revno_second = -1 | ||
1620 | 617 | self._revno_last = -1 | ||
1621 | 618 | self.is_first_child = 0 | ||
1622 | 619 | self.seen_by_child = 0 | ||
1623 | 620 | self.completed = 0 | ||
1624 | 621 | |||
1625 | 622 | def __repr__(self): | ||
1626 | 623 | return '%s(%s depth:%s rev:%s,%s,%s first:%s seen:%s)' % ( | ||
1627 | 624 | self.__class__.__name__, self.key, | ||
1628 | 625 | self.merge_depth, | ||
1629 | 626 | self._revno_first, self._revno_second, self._revno_last, | ||
1630 | 627 | self.is_first_child, self.seen_by_child) | ||
1631 | 628 | |||
1632 | 629 | cdef int has_pending_parents(self): | ||
1633 | 630 | if self.left_pending_parent is not None or self.pending_parents: | ||
1634 | 631 | return 1 | ||
1635 | 632 | return 0 | ||
1636 | 633 | |||
1637 | 634 | cdef object _revno(self): | ||
1638 | 635 | if self._revno_first == -1: | ||
1639 | 636 | if self._revno_second != -1: | ||
1640 | 637 | raise RuntimeError('Something wrong with: %s' % (self,)) | ||
1641 | 638 | return (self._revno_last,) | ||
1642 | 639 | else: | ||
1643 | 640 | return (self._revno_first, self._revno_second, self._revno_last) | ||
1644 | 641 | |||
1645 | 642 | property revno: | ||
1646 | 643 | def __get__(self): | ||
1647 | 644 | return self._revno() | ||
1648 | 645 | |||
1649 | 646 | |||
1650 | 647 | cdef class _MergeSorter: | ||
1651 | 648 | """This class does the work of computing the merge_sort ordering. | ||
1652 | 649 | |||
1653 | 650 | We have some small advantages, in that we get all the extra information | ||
1654 | 651 | that KnownGraph knows, like knowing the child lists, etc. | ||
1655 | 652 | """ | ||
1656 | 653 | |||
1657 | 654 | # Current performance numbers for merge_sort(bzr_dev_parent_map): | ||
1658 | 655 | # 302ms tsort.merge_sort() | ||
1659 | 656 | # 91ms graph.KnownGraph().merge_sort() | ||
1660 | 657 | # 40ms kg.merge_sort() | ||
1661 | 658 | |||
1662 | 659 | cdef KnownGraph graph | ||
1663 | 660 | cdef object _depth_first_stack # list | ||
1664 | 661 | cdef Py_ssize_t _last_stack_item # offset to last item on stack | ||
1665 | 662 | # cdef object _ms_nodes # dict of key => _MergeSortNode | ||
1666 | 663 | cdef object _revno_to_branch_count # {revno => num child branches} | ||
1667 | 664 | cdef object _scheduled_nodes # List of nodes ready to be yielded | ||
1668 | 665 | |||
1669 | 666 | def __init__(self, known_graph, tip_key): | ||
1670 | 667 | cdef _KnownGraphNode node | ||
1671 | 668 | |||
1672 | 669 | self.graph = known_graph | ||
1673 | 670 | # self._ms_nodes = {} | ||
1674 | 671 | self._revno_to_branch_count = {} | ||
1675 | 672 | self._depth_first_stack = [] | ||
1676 | 673 | self._last_stack_item = -1 | ||
1677 | 674 | self._scheduled_nodes = [] | ||
1678 | 675 | if (tip_key is not None and tip_key != NULL_REVISION | ||
1679 | 676 | and tip_key != (NULL_REVISION,)): | ||
1680 | 677 | node = self.graph._nodes[tip_key] | ||
1681 | 678 | self._push_node(node, 0) | ||
1682 | 679 | |||
1683 | 680 | cdef _MergeSortNode _get_ms_node(self, _KnownGraphNode node): | ||
1684 | 681 | cdef PyObject *temp_node | ||
1685 | 682 | cdef _MergeSortNode ms_node | ||
1686 | 683 | |||
1687 | 684 | if node.extra is None: | ||
1688 | 685 | ms_node = _MergeSortNode(node.key) | ||
1689 | 686 | node.extra = ms_node | ||
1690 | 687 | else: | ||
1691 | 688 | ms_node = <_MergeSortNode>node.extra | ||
1692 | 689 | return ms_node | ||
1693 | 690 | |||
1694 | 691 | cdef _push_node(self, _KnownGraphNode node, long merge_depth): | ||
1695 | 692 | cdef _KnownGraphNode parent_node | ||
1696 | 693 | cdef _MergeSortNode ms_node, ms_parent_node | ||
1697 | 694 | cdef Py_ssize_t pos | ||
1698 | 695 | |||
1699 | 696 | ms_node = self._get_ms_node(node) | ||
1700 | 697 | ms_node.merge_depth = merge_depth | ||
1701 | 698 | if node.parents is None: | ||
1702 | 699 | raise RuntimeError('ghost nodes should not be pushed' | ||
1703 | 700 | ' onto the stack: %s' % (node,)) | ||
1704 | 701 | if PyTuple_GET_SIZE(node.parents) > 0: | ||
1705 | 702 | parent_node = _get_tuple_node(node.parents, 0) | ||
1706 | 703 | ms_node.left_parent = parent_node | ||
1707 | 704 | if parent_node.parents is None: # left-hand ghost | ||
1708 | 705 | ms_node.left_pending_parent = None | ||
1709 | 706 | ms_node.left_parent = None | ||
1710 | 707 | else: | ||
1711 | 708 | ms_node.left_pending_parent = parent_node | ||
1712 | 709 | if PyTuple_GET_SIZE(node.parents) > 1: | ||
1713 | 710 | ms_node.pending_parents = [] | ||
1714 | 711 | for pos from 1 <= pos < PyTuple_GET_SIZE(node.parents): | ||
1715 | 712 | parent_node = _get_tuple_node(node.parents, pos) | ||
1716 | 713 | if parent_node.parents is None: # ghost | ||
1717 | 714 | continue | ||
1718 | 715 | PyList_Append(ms_node.pending_parents, parent_node) | ||
1719 | 716 | |||
1720 | 717 | ms_node.is_first_child = 1 | ||
1721 | 718 | if ms_node.left_parent is not None: | ||
1722 | 719 | ms_parent_node = self._get_ms_node(ms_node.left_parent) | ||
1723 | 720 | if ms_parent_node.seen_by_child: | ||
1724 | 721 | ms_node.is_first_child = 0 | ||
1725 | 722 | ms_parent_node.seen_by_child = 1 | ||
1726 | 723 | self._last_stack_item = self._last_stack_item + 1 | ||
1727 | 724 | if self._last_stack_item < PyList_GET_SIZE(self._depth_first_stack): | ||
1728 | 725 | Py_INCREF(node) # SetItem steals a ref | ||
1729 | 726 | PyList_SetItem(self._depth_first_stack, self._last_stack_item, | ||
1730 | 727 | node) | ||
1731 | 728 | else: | ||
1732 | 729 | PyList_Append(self._depth_first_stack, node) | ||
1733 | 730 | |||
1734 | 731 | cdef _pop_node(self): | ||
1735 | 732 | cdef PyObject *temp | ||
1736 | 733 | cdef _MergeSortNode ms_node, ms_parent_node, ms_prev_node | ||
1737 | 734 | cdef _KnownGraphNode node, parent_node, prev_node | ||
1738 | 735 | |||
1739 | 736 | node = _get_list_node(self._depth_first_stack, self._last_stack_item) | ||
1740 | 737 | ms_node = <_MergeSortNode>node.extra | ||
1741 | 738 | self._last_stack_item = self._last_stack_item - 1 | ||
1742 | 739 | if ms_node.left_parent is not None: | ||
1743 | 740 | # Assign the revision number from the left-hand parent | ||
1744 | 741 | ms_parent_node = <_MergeSortNode>ms_node.left_parent.extra | ||
1745 | 742 | if ms_node.is_first_child: | ||
1746 | 743 | # First child just increments the final digit | ||
1747 | 744 | ms_node._revno_first = ms_parent_node._revno_first | ||
1748 | 745 | ms_node._revno_second = ms_parent_node._revno_second | ||
1749 | 746 | ms_node._revno_last = ms_parent_node._revno_last + 1 | ||
1750 | 747 | else: | ||
1751 | 748 | # Not the first child, make a new branch | ||
1752 | 749 | # (mainline_revno, branch_count, 1) | ||
1753 | 750 | if ms_parent_node._revno_first == -1: | ||
1754 | 751 | # Mainline ancestor, the increment is on the last digit | ||
1755 | 752 | base_revno = ms_parent_node._revno_last | ||
1756 | 753 | else: | ||
1757 | 754 | base_revno = ms_parent_node._revno_first | ||
1758 | 755 | temp = PyDict_GetItem(self._revno_to_branch_count, | ||
1759 | 756 | base_revno) | ||
1760 | 757 | if temp == NULL: | ||
1761 | 758 | branch_count = 1 | ||
1762 | 759 | else: | ||
1763 | 760 | branch_count = (<object>temp) + 1 | ||
1764 | 761 | PyDict_SetItem(self._revno_to_branch_count, base_revno, | ||
1765 | 762 | branch_count) | ||
1766 | 763 | ms_node._revno_first = base_revno | ||
1767 | 764 | ms_node._revno_second = branch_count | ||
1768 | 765 | ms_node._revno_last = 1 | ||
1769 | 766 | else: | ||
1770 | 767 | temp = PyDict_GetItem(self._revno_to_branch_count, 0) | ||
1771 | 768 | if temp == NULL: | ||
1772 | 769 | # The first root node doesn't have a 3-digit revno | ||
1773 | 770 | root_count = 0 | ||
1774 | 771 | ms_node._revno_first = -1 | ||
1775 | 772 | ms_node._revno_second = -1 | ||
1776 | 773 | ms_node._revno_last = 1 | ||
1777 | 774 | else: | ||
1778 | 775 | root_count = (<object>temp) + 1 | ||
1779 | 776 | ms_node._revno_first = 0 | ||
1780 | 777 | ms_node._revno_second = root_count | ||
1781 | 778 | ms_node._revno_last = 1 | ||
1782 | 779 | PyDict_SetItem(self._revno_to_branch_count, 0, root_count) | ||
1783 | 780 | ms_node.completed = 1 | ||
1784 | 781 | if PyList_GET_SIZE(self._scheduled_nodes) == 0: | ||
1785 | 782 | # The first scheduled node is always the end of merge | ||
1786 | 783 | ms_node.end_of_merge = True | ||
1787 | 784 | else: | ||
1788 | 785 | prev_node = _get_list_node(self._scheduled_nodes, | ||
1789 | 786 | PyList_GET_SIZE(self._scheduled_nodes) - 1) | ||
1790 | 787 | ms_prev_node = <_MergeSortNode>prev_node.extra | ||
1791 | 788 | if ms_prev_node.merge_depth < ms_node.merge_depth: | ||
1792 | 789 | # The previously pushed node is to our left, so this is the end | ||
1793 | 790 | # of this right-hand chain | ||
1794 | 791 | ms_node.end_of_merge = True | ||
1795 | 792 | elif (ms_prev_node.merge_depth == ms_node.merge_depth | ||
1796 | 793 | and prev_node not in node.parents): | ||
1797 | 794 | # The next node is not a direct parent of this node | ||
1798 | 795 | ms_node.end_of_merge = True | ||
1799 | 796 | else: | ||
1800 | 797 | ms_node.end_of_merge = False | ||
1801 | 798 | PyList_Append(self._scheduled_nodes, node) | ||
1802 | 799 | |||
1803 | 800 | cdef _schedule_stack(self): | ||
1804 | 801 | cdef _KnownGraphNode last_node, next_node | ||
1805 | 802 | cdef _MergeSortNode ms_node, ms_last_node, ms_next_node | ||
1806 | 803 | cdef long next_merge_depth | ||
1807 | 804 | ordered = [] | ||
1808 | 805 | while self._last_stack_item >= 0: | ||
1809 | 806 | # Peek at the last item on the stack | ||
1810 | 807 | last_node = _get_list_node(self._depth_first_stack, | ||
1811 | 808 | self._last_stack_item) | ||
1812 | 809 | if last_node.gdfo == -1: | ||
1813 | 810 | # if _find_gdfo skipped a node, that means there is a graph | ||
1814 | 811 | # cycle, error out now | ||
1815 | 812 | raise errors.GraphCycleError(self.graph._nodes) | ||
1816 | 813 | ms_last_node = <_MergeSortNode>last_node.extra | ||
1817 | 814 | if not ms_last_node.has_pending_parents(): | ||
1818 | 815 | # Processed all parents, pop this node | ||
1819 | 816 | self._pop_node() | ||
1820 | 817 | continue | ||
1821 | 818 | while ms_last_node.has_pending_parents(): | ||
1822 | 819 | if ms_last_node.left_pending_parent is not None: | ||
1823 | 820 | # recurse depth first into the primary parent | ||
1824 | 821 | next_node = ms_last_node.left_pending_parent | ||
1825 | 822 | ms_last_node.left_pending_parent = None | ||
1826 | 823 | else: | ||
1827 | 824 | # place any merges in right-to-left order for scheduling | ||
1828 | 825 | # which gives us left-to-right order after we reverse | ||
1829 | 826 | # the scheduled queue. | ||
1830 | 827 | # Note: This has the effect of allocating common-new | ||
1831 | 828 | # revisions to the right-most subtree rather than the | ||
1832 | 829 | # left most, which will display nicely (you get | ||
1833 | 830 | # smaller trees at the top of the combined merge). | ||
1834 | 831 | next_node = ms_last_node.pending_parents.pop() | ||
1835 | 832 | ms_next_node = self._get_ms_node(next_node) | ||
1836 | 833 | if ms_next_node.completed: | ||
1837 | 834 | # this parent was completed by a child on the | ||
1838 | 835 | # call stack. skip it. | ||
1839 | 836 | continue | ||
1840 | 837 | # otherwise transfer it from the source graph into the | ||
1841 | 838 | # top of the current depth first search stack. | ||
1842 | 839 | |||
1843 | 840 | if next_node is ms_last_node.left_parent: | ||
1844 | 841 | next_merge_depth = ms_last_node.merge_depth | ||
1845 | 842 | else: | ||
1846 | 843 | next_merge_depth = ms_last_node.merge_depth + 1 | ||
1847 | 844 | self._push_node(next_node, next_merge_depth) | ||
1848 | 845 | # and do not continue processing parents until this 'call' | ||
1849 | 846 | # has recursed. | ||
1850 | 847 | break | ||
1851 | 848 | |||
1852 | 849 | cdef topo_order(self): | ||
1853 | 850 | cdef _MergeSortNode ms_node | ||
1854 | 851 | cdef _KnownGraphNode node | ||
1855 | 852 | cdef Py_ssize_t pos | ||
1856 | 853 | cdef PyObject *temp_key, *temp_node | ||
1857 | 854 | |||
1858 | 855 | # Note: allocating a _MergeSortNode and deallocating it for all nodes | ||
1859 | 856 | # costs approx 8.52ms (21%) of the total runtime | ||
1860 | 857 | # We might consider moving the attributes into the base | ||
1861 | 858 | # KnownGraph object. | ||
1862 | 859 | self._schedule_stack() | ||
1863 | 860 | |||
1864 | 861 | # We've set up the basic schedule, now we can continue processing the | ||
1865 | 862 | # output. | ||
1866 | 863 | # Note: This final loop costs us 40.0ms => 28.8ms (11ms, 25%) on | ||
1867 | 864 | # bzr.dev, to convert the internal Object representation into a | ||
1868 | 865 | # Tuple representation... | ||
1869 | 866 | # 2ms is walking the data and computing revno tuples | ||
1870 | 867 | # 7ms is computing the return tuple | ||
1871 | 868 | # 4ms is PyList_Append() | ||
1872 | 869 | ordered = [] | ||
1873 | 870 | # output the result in reverse order, and separate the generated info | ||
1874 | 871 | for pos from PyList_GET_SIZE(self._scheduled_nodes) > pos >= 0: | ||
1875 | 872 | node = _get_list_node(self._scheduled_nodes, pos) | ||
1876 | 873 | ms_node = <_MergeSortNode>node.extra | ||
1877 | 874 | PyList_Append(ordered, ms_node) | ||
1878 | 875 | node.extra = None | ||
1879 | 876 | # Clear out the scheduled nodes now that we're done | ||
1880 | 877 | self._scheduled_nodes = [] | ||
1881 | 878 | return ordered | ||
1882 | 338 | 879 | ||
1883 | === modified file 'bzrlib/annotate.py' | |||
1884 | --- bzrlib/annotate.py 2009-07-08 17:09:03 +0000 | |||
1885 | +++ bzrlib/annotate.py 2009-08-17 18:52:01 +0000 | |||
1886 | @@ -188,6 +188,10 @@ | |||
1887 | 188 | # or something. | 188 | # or something. |
1888 | 189 | last_revision = current_rev.revision_id | 189 | last_revision = current_rev.revision_id |
1889 | 190 | # XXX: Partially Cloned from branch, uses the old_get_graph, eep. | 190 | # XXX: Partially Cloned from branch, uses the old_get_graph, eep. |
1890 | 191 | # XXX: The main difficulty is that we need to inject a single new node | ||
1891 | 192 | # (current_rev) into the graph before it gets numbered, etc. | ||
1892 | 193 | # Once KnownGraph gets an 'add_node()' function, we can use | ||
1893 | 194 | # VF.get_known_graph_ancestry(). | ||
1894 | 191 | graph = repository.get_graph() | 195 | graph = repository.get_graph() |
1895 | 192 | revision_graph = dict(((key, value) for key, value in | 196 | revision_graph = dict(((key, value) for key, value in |
1896 | 193 | graph.iter_ancestry(current_rev.parent_ids) if value is not None)) | 197 | graph.iter_ancestry(current_rev.parent_ids) if value is not None)) |
1897 | 194 | 198 | ||
1898 | === modified file 'bzrlib/branch.py' | |||
1899 | --- bzrlib/branch.py 2009-08-04 04:36:34 +0000 | |||
1900 | +++ bzrlib/branch.py 2009-08-19 18:04:49 +0000 | |||
1901 | @@ -446,15 +446,11 @@ | |||
1902 | 446 | # start_revision_id. | 446 | # start_revision_id. |
1903 | 447 | if self._merge_sorted_revisions_cache is None: | 447 | if self._merge_sorted_revisions_cache is None: |
1904 | 448 | last_revision = self.last_revision() | 448 | last_revision = self.last_revision() |
1914 | 449 | graph = self.repository.get_graph() | 449 | last_key = (last_revision,) |
1915 | 450 | parent_map = dict(((key, value) for key, value in | 450 | known_graph = self.repository.revisions.get_known_graph_ancestry( |
1916 | 451 | graph.iter_ancestry([last_revision]) if value is not None)) | 451 | [last_key]) |
1917 | 452 | revision_graph = repository._strip_NULL_ghosts(parent_map) | 452 | self._merge_sorted_revisions_cache = known_graph.merge_sort( |
1918 | 453 | revs = tsort.merge_sort(revision_graph, last_revision, None, | 453 | last_key) |
1910 | 454 | generate_revno=True) | ||
1911 | 455 | # Drop the sequence # before caching | ||
1912 | 456 | self._merge_sorted_revisions_cache = [r[1:] for r in revs] | ||
1913 | 457 | |||
1919 | 458 | filtered = self._filter_merge_sorted_revisions( | 454 | filtered = self._filter_merge_sorted_revisions( |
1920 | 459 | self._merge_sorted_revisions_cache, start_revision_id, | 455 | self._merge_sorted_revisions_cache, start_revision_id, |
1921 | 460 | stop_revision_id, stop_rule) | 456 | stop_revision_id, stop_rule) |
1922 | @@ -470,27 +466,34 @@ | |||
1923 | 470 | """Iterate over an inclusive range of sorted revisions.""" | 466 | """Iterate over an inclusive range of sorted revisions.""" |
1924 | 471 | rev_iter = iter(merge_sorted_revisions) | 467 | rev_iter = iter(merge_sorted_revisions) |
1925 | 472 | if start_revision_id is not None: | 468 | if start_revision_id is not None: |
1927 | 473 | for rev_id, depth, revno, end_of_merge in rev_iter: | 469 | for node in rev_iter: |
1928 | 470 | rev_id = node.key[-1] | ||
1929 | 474 | if rev_id != start_revision_id: | 471 | if rev_id != start_revision_id: |
1930 | 475 | continue | 472 | continue |
1931 | 476 | else: | 473 | else: |
1932 | 477 | # The decision to include the start or not | 474 | # The decision to include the start or not |
1933 | 478 | # depends on the stop_rule if a stop is provided | 475 | # depends on the stop_rule if a stop is provided |
1937 | 479 | rev_iter = chain( | 476 | # so pop this node back into the iterator |
1938 | 480 | iter([(rev_id, depth, revno, end_of_merge)]), | 477 | rev_iter = chain(iter([node]), rev_iter) |
1936 | 481 | rev_iter) | ||
1939 | 482 | break | 478 | break |
1940 | 483 | if stop_revision_id is None: | 479 | if stop_revision_id is None: |
1943 | 484 | for rev_id, depth, revno, end_of_merge in rev_iter: | 480 | # Yield everything |
1944 | 485 | yield rev_id, depth, revno, end_of_merge | 481 | for node in rev_iter: |
1945 | 482 | rev_id = node.key[-1] | ||
1946 | 483 | yield (rev_id, node.merge_depth, node.revno, | ||
1947 | 484 | node.end_of_merge) | ||
1948 | 486 | elif stop_rule == 'exclude': | 485 | elif stop_rule == 'exclude': |
1950 | 487 | for rev_id, depth, revno, end_of_merge in rev_iter: | 486 | for node in rev_iter: |
1951 | 487 | rev_id = node.key[-1] | ||
1952 | 488 | if rev_id == stop_revision_id: | 488 | if rev_id == stop_revision_id: |
1953 | 489 | return | 489 | return |
1955 | 490 | yield rev_id, depth, revno, end_of_merge | 490 | yield (rev_id, node.merge_depth, node.revno, |
1956 | 491 | node.end_of_merge) | ||
1957 | 491 | elif stop_rule == 'include': | 492 | elif stop_rule == 'include': |
1960 | 492 | for rev_id, depth, revno, end_of_merge in rev_iter: | 493 | for node in rev_iter: |
1961 | 493 | yield rev_id, depth, revno, end_of_merge | 494 | rev_id = node.key[-1] |
1962 | 495 | yield (rev_id, node.merge_depth, node.revno, | ||
1963 | 496 | node.end_of_merge) | ||
1964 | 494 | if rev_id == stop_revision_id: | 497 | if rev_id == stop_revision_id: |
1965 | 495 | return | 498 | return |
1966 | 496 | elif stop_rule == 'with-merges': | 499 | elif stop_rule == 'with-merges': |
1967 | @@ -499,10 +502,12 @@ | |||
1968 | 499 | left_parent = stop_rev.parent_ids[0] | 502 | left_parent = stop_rev.parent_ids[0] |
1969 | 500 | else: | 503 | else: |
1970 | 501 | left_parent = _mod_revision.NULL_REVISION | 504 | left_parent = _mod_revision.NULL_REVISION |
1972 | 502 | for rev_id, depth, revno, end_of_merge in rev_iter: | 505 | for node in rev_iter: |
1973 | 506 | rev_id = node.key[-1] | ||
1974 | 503 | if rev_id == left_parent: | 507 | if rev_id == left_parent: |
1975 | 504 | return | 508 | return |
1977 | 505 | yield rev_id, depth, revno, end_of_merge | 509 | yield (rev_id, node.merge_depth, node.revno, |
1978 | 510 | node.end_of_merge) | ||
1979 | 506 | else: | 511 | else: |
1980 | 507 | raise ValueError('invalid stop_rule %r' % stop_rule) | 512 | raise ValueError('invalid stop_rule %r' % stop_rule) |
1981 | 508 | 513 | ||
1982 | @@ -1147,6 +1152,9 @@ | |||
1983 | 1147 | revision_id: if not None, the revision history in the new branch will | 1152 | revision_id: if not None, the revision history in the new branch will |
1984 | 1148 | be truncated to end with revision_id. | 1153 | be truncated to end with revision_id. |
1985 | 1149 | """ | 1154 | """ |
1986 | 1155 | if (repository_policy is not None and | ||
1987 | 1156 | repository_policy.requires_stacking()): | ||
1988 | 1157 | to_bzrdir._format.require_stacking(_skip_repo=True) | ||
1989 | 1150 | result = to_bzrdir.create_branch() | 1158 | result = to_bzrdir.create_branch() |
1990 | 1151 | result.lock_write() | 1159 | result.lock_write() |
1991 | 1152 | try: | 1160 | try: |
1992 | @@ -2064,7 +2072,7 @@ | |||
1993 | 2064 | BranchFormat.register_format(__format6) | 2072 | BranchFormat.register_format(__format6) |
1994 | 2065 | BranchFormat.register_format(__format7) | 2073 | BranchFormat.register_format(__format7) |
1995 | 2066 | BranchFormat.register_format(__format8) | 2074 | BranchFormat.register_format(__format8) |
1997 | 2067 | BranchFormat.set_default_format(__format6) | 2075 | BranchFormat.set_default_format(__format7) |
1998 | 2068 | _legacy_formats = [BzrBranchFormat4(), | 2076 | _legacy_formats = [BzrBranchFormat4(), |
1999 | 2069 | ] | 2077 | ] |
2000 | 2070 | network_format_registry.register( | 2078 | network_format_registry.register( |
2001 | 2071 | 2079 | ||
2002 | === modified file 'bzrlib/btree_index.py' | |||
2003 | --- bzrlib/btree_index.py 2009-07-01 10:51:47 +0000 | |||
2004 | +++ bzrlib/btree_index.py 2009-08-17 22:11:06 +0000 | |||
2005 | @@ -586,13 +586,19 @@ | |||
2006 | 586 | class _LeafNode(object): | 586 | class _LeafNode(object): |
2007 | 587 | """A leaf node for a serialised B+Tree index.""" | 587 | """A leaf node for a serialised B+Tree index.""" |
2008 | 588 | 588 | ||
2010 | 589 | __slots__ = ('keys',) | 589 | __slots__ = ('keys', 'min_key', 'max_key') |
2011 | 590 | 590 | ||
2012 | 591 | def __init__(self, bytes, key_length, ref_list_length): | 591 | def __init__(self, bytes, key_length, ref_list_length): |
2013 | 592 | """Parse bytes to create a leaf node object.""" | 592 | """Parse bytes to create a leaf node object.""" |
2014 | 593 | # splitlines mangles the \r delimiters.. don't use it. | 593 | # splitlines mangles the \r delimiters.. don't use it. |
2017 | 594 | self.keys = dict(_btree_serializer._parse_leaf_lines(bytes, | 594 | key_list = _btree_serializer._parse_leaf_lines(bytes, |
2018 | 595 | key_length, ref_list_length)) | 595 | key_length, ref_list_length) |
2019 | 596 | if key_list: | ||
2020 | 597 | self.min_key = key_list[0][0] | ||
2021 | 598 | self.max_key = key_list[-1][0] | ||
2022 | 599 | else: | ||
2023 | 600 | self.min_key = self.max_key = None | ||
2024 | 601 | self.keys = dict(key_list) | ||
2025 | 596 | 602 | ||
2026 | 597 | 603 | ||
2027 | 598 | class _InternalNode(object): | 604 | class _InternalNode(object): |
2028 | @@ -1039,6 +1045,39 @@ | |||
2029 | 1039 | output.append(cur_out) | 1045 | output.append(cur_out) |
2030 | 1040 | return output | 1046 | return output |
2031 | 1041 | 1047 | ||
2032 | 1048 | def _walk_through_internal_nodes(self, keys): | ||
2033 | 1049 | """Take the given set of keys, and find the corresponding LeafNodes. | ||
2034 | 1050 | |||
2035 | 1051 | :param keys: An unsorted iterable of keys to search for | ||
2036 | 1052 | :return: (nodes, index_and_keys) | ||
2037 | 1053 | nodes is a dict mapping {index: LeafNode} | ||
2038 | 1054 | keys_at_index is a list of tuples of [(index, [keys for Leaf])] | ||
2039 | 1055 | """ | ||
2040 | 1056 | # 6 seconds spent in miss_torture using the sorted() line. | ||
2041 | 1057 | # Even with out of order disk IO it seems faster not to sort it when | ||
2042 | 1058 | # large queries are being made. | ||
2043 | 1059 | keys_at_index = [(0, sorted(keys))] | ||
2044 | 1060 | |||
2045 | 1061 | for row_pos, next_row_start in enumerate(self._row_offsets[1:-1]): | ||
2046 | 1062 | node_indexes = [idx for idx, s_keys in keys_at_index] | ||
2047 | 1063 | nodes = self._get_internal_nodes(node_indexes) | ||
2048 | 1064 | |||
2049 | 1065 | next_nodes_and_keys = [] | ||
2050 | 1066 | for node_index, sub_keys in keys_at_index: | ||
2051 | 1067 | node = nodes[node_index] | ||
2052 | 1068 | positions = self._multi_bisect_right(sub_keys, node.keys) | ||
2053 | 1069 | node_offset = next_row_start + node.offset | ||
2054 | 1070 | next_nodes_and_keys.extend([(node_offset + pos, s_keys) | ||
2055 | 1071 | for pos, s_keys in positions]) | ||
2056 | 1072 | keys_at_index = next_nodes_and_keys | ||
2057 | 1073 | # We should now be at the _LeafNodes | ||
2058 | 1074 | node_indexes = [idx for idx, s_keys in keys_at_index] | ||
2059 | 1075 | |||
2060 | 1076 | # TODO: We may *not* want to always read all the nodes in one | ||
2061 | 1077 | # big go. Consider setting a max size on this. | ||
2062 | 1078 | nodes = self._get_leaf_nodes(node_indexes) | ||
2063 | 1079 | return nodes, keys_at_index | ||
2064 | 1080 | |||
2065 | 1042 | def iter_entries(self, keys): | 1081 | def iter_entries(self, keys): |
2066 | 1043 | """Iterate over keys within the index. | 1082 | """Iterate over keys within the index. |
2067 | 1044 | 1083 | ||
2068 | @@ -1082,32 +1121,7 @@ | |||
2069 | 1082 | needed_keys = keys | 1121 | needed_keys = keys |
2070 | 1083 | if not needed_keys: | 1122 | if not needed_keys: |
2071 | 1084 | return | 1123 | return |
2098 | 1085 | # 6 seconds spent in miss_torture using the sorted() line. | 1124 | nodes, nodes_and_keys = self._walk_through_internal_nodes(needed_keys) |
2073 | 1086 | # Even with out of order disk IO it seems faster not to sort it when | ||
2074 | 1087 | # large queries are being made. | ||
2075 | 1088 | needed_keys = sorted(needed_keys) | ||
2076 | 1089 | |||
2077 | 1090 | nodes_and_keys = [(0, needed_keys)] | ||
2078 | 1091 | |||
2079 | 1092 | for row_pos, next_row_start in enumerate(self._row_offsets[1:-1]): | ||
2080 | 1093 | node_indexes = [idx for idx, s_keys in nodes_and_keys] | ||
2081 | 1094 | nodes = self._get_internal_nodes(node_indexes) | ||
2082 | 1095 | |||
2083 | 1096 | next_nodes_and_keys = [] | ||
2084 | 1097 | for node_index, sub_keys in nodes_and_keys: | ||
2085 | 1098 | node = nodes[node_index] | ||
2086 | 1099 | positions = self._multi_bisect_right(sub_keys, node.keys) | ||
2087 | 1100 | node_offset = next_row_start + node.offset | ||
2088 | 1101 | next_nodes_and_keys.extend([(node_offset + pos, s_keys) | ||
2089 | 1102 | for pos, s_keys in positions]) | ||
2090 | 1103 | nodes_and_keys = next_nodes_and_keys | ||
2091 | 1104 | # We should now be at the _LeafNodes | ||
2092 | 1105 | node_indexes = [idx for idx, s_keys in nodes_and_keys] | ||
2093 | 1106 | |||
2094 | 1107 | # TODO: We may *not* want to always read all the nodes in one | ||
2095 | 1108 | # big go. Consider setting a max size on this. | ||
2096 | 1109 | |||
2097 | 1110 | nodes = self._get_leaf_nodes(node_indexes) | ||
2099 | 1111 | for node_index, sub_keys in nodes_and_keys: | 1125 | for node_index, sub_keys in nodes_and_keys: |
2100 | 1112 | if not sub_keys: | 1126 | if not sub_keys: |
2101 | 1113 | continue | 1127 | continue |
2102 | @@ -1120,6 +1134,133 @@ | |||
2103 | 1120 | else: | 1134 | else: |
2104 | 1121 | yield (self, next_sub_key, value) | 1135 | yield (self, next_sub_key, value) |
2105 | 1122 | 1136 | ||
2106 | 1137 | def _find_ancestors(self, keys, ref_list_num, parent_map, missing_keys): | ||
2107 | 1138 | """Find the parent_map information for the set of keys. | ||
2108 | 1139 | |||
2109 | 1140 | This populates the parent_map dict and missing_keys set based on the | ||
2110 | 1141 | queried keys. It also can fill out an arbitrary number of parents that | ||
2111 | 1142 | it finds while searching for the supplied keys. | ||
2112 | 1143 | |||
2113 | 1144 | It is unlikely that you want to call this directly. See | ||
2114 | 1145 | "CombinedGraphIndex.find_ancestry()" for a more appropriate API. | ||
2115 | 1146 | |||
2116 | 1147 | :param keys: A keys whose ancestry we want to return | ||
2117 | 1148 | Every key will either end up in 'parent_map' or 'missing_keys'. | ||
2118 | 1149 | :param ref_list_num: This index in the ref_lists is the parents we | ||
2119 | 1150 | care about. | ||
2120 | 1151 | :param parent_map: {key: parent_keys} for keys that are present in this | ||
2121 | 1152 | index. This may contain more entries than were in 'keys', that are | ||
2122 | 1153 | reachable ancestors of the keys requested. | ||
2123 | 1154 | :param missing_keys: keys which are known to be missing in this index. | ||
2124 | 1155 | This may include parents that were not directly requested, but we | ||
2125 | 1156 | were able to determine that they are not present in this index. | ||
2126 | 1157 | :return: search_keys parents that were found but not queried to know | ||
2127 | 1158 | if they are missing or present. Callers can re-query this index for | ||
2128 | 1159 | those keys, and they will be placed into parent_map or missing_keys | ||
2129 | 1160 | """ | ||
2130 | 1161 | if not self.key_count(): | ||
2131 | 1162 | # We use key_count() to trigger reading the root node and | ||
2132 | 1163 | # determining info about this BTreeGraphIndex | ||
2133 | 1164 | # If we don't have any keys, then everything is missing | ||
2134 | 1165 | missing_keys.update(keys) | ||
2135 | 1166 | return set() | ||
2136 | 1167 | if ref_list_num >= self.node_ref_lists: | ||
2137 | 1168 | raise ValueError('No ref list %d, index has %d ref lists' | ||
2138 | 1169 | % (ref_list_num, self.node_ref_lists)) | ||
2139 | 1170 | |||
2140 | 1171 | # The main trick we are trying to accomplish is that when we find a | ||
2141 | 1172 | # key listing its parents, we expect that the parent key is also likely | ||
2142 | 1173 | # to sit on the same page. Allowing us to expand parents quickly | ||
2143 | 1174 | # without suffering the full stack of bisecting, etc. | ||
2144 | 1175 | nodes, nodes_and_keys = self._walk_through_internal_nodes(keys) | ||
2145 | 1176 | |||
2146 | 1177 | # These are parent keys which could not be immediately resolved on the | ||
2147 | 1178 | # page where the child was present. Note that we may already be | ||
2148 | 1179 | # searching for that key, and it may actually be present [or known | ||
2149 | 1180 | # missing] on one of the other pages we are reading. | ||
2150 | 1181 | # TODO: | ||
2151 | 1182 | # We could try searching for them in the immediate previous or next | ||
2152 | 1183 | # page. If they occur "later" we could put them in a pending lookup | ||
2153 | 1184 | # set, and then for each node we read thereafter we could check to | ||
2154 | 1185 | # see if they are present. | ||
2155 | 1186 | # However, we don't know the impact of keeping this list of things | ||
2156 | 1187 | # that I'm going to search for every node I come across from here on | ||
2157 | 1188 | # out. | ||
2158 | 1189 | # It doesn't handle the case when the parent key is missing on a | ||
2159 | 1190 | # page that we *don't* read. So we already have to handle being | ||
2160 | 1191 | # re-entrant for that. | ||
2161 | 1192 | # Since most keys contain a date string, they are more likely to be | ||
2162 | 1193 | # found earlier in the file than later, but we would know that right | ||
2163 | 1194 | # away (key < min_key), and wouldn't keep searching it on every other | ||
2164 | 1195 | # page that we read. | ||
2165 | 1196 | # Mostly, it is an idea, one which should be benchmarked. | ||
2166 | 1197 | parents_not_on_page = set() | ||
2167 | 1198 | |||
2168 | 1199 | for node_index, sub_keys in nodes_and_keys: | ||
2169 | 1200 | if not sub_keys: | ||
2170 | 1201 | continue | ||
2171 | 1202 | # sub_keys is all of the keys we are looking for that should exist | ||
2172 | 1203 | # on this page, if they aren't here, then they won't be found | ||
2173 | 1204 | node = nodes[node_index] | ||
2174 | 1205 | node_keys = node.keys | ||
2175 | 1206 | parents_to_check = set() | ||
2176 | 1207 | for next_sub_key in sub_keys: | ||
2177 | 1208 | if next_sub_key not in node_keys: | ||
2178 | 1209 | # This one is just not present in the index at all | ||
2179 | 1210 | missing_keys.add(next_sub_key) | ||
2180 | 1211 | else: | ||
2181 | 1212 | value, refs = node_keys[next_sub_key] | ||
2182 | 1213 | parent_keys = refs[ref_list_num] | ||
2183 | 1214 | parent_map[next_sub_key] = parent_keys | ||
2184 | 1215 | parents_to_check.update(parent_keys) | ||
2185 | 1216 | # Don't look for things we've already found | ||
2186 | 1217 | parents_to_check = parents_to_check.difference(parent_map) | ||
2187 | 1218 | # this can be used to test the benefit of having the check loop | ||
2188 | 1219 | # inlined. | ||
2189 | 1220 | # parents_not_on_page.update(parents_to_check) | ||
2190 | 1221 | # continue | ||
2191 | 1222 | while parents_to_check: | ||
2192 | 1223 | next_parents_to_check = set() | ||
2193 | 1224 | for key in parents_to_check: | ||
2194 | 1225 | if key in node_keys: | ||
2195 | 1226 | value, refs = node_keys[key] | ||
2196 | 1227 | parent_keys = refs[ref_list_num] | ||
2197 | 1228 | parent_map[key] = parent_keys | ||
2198 | 1229 | next_parents_to_check.update(parent_keys) | ||
2199 | 1230 | else: | ||
2200 | 1231 | # This parent either is genuinely missing, or should be | ||
2201 | 1232 | # found on another page. Perf test whether it is better | ||
2202 | 1233 | # to check if this node should fit on this page or not. | ||
2203 | 1234 | # in the 'everything-in-one-pack' scenario, this *not* | ||
2204 | 1235 | # doing the check is 237ms vs 243ms. | ||
2205 | 1236 | # So slightly better, but I assume the standard 'lots | ||
2206 | 1237 | # of packs' is going to show a reasonable improvement | ||
2207 | 1238 | # from the check, because it avoids 'going around | ||
2208 | 1239 | # again' for everything that is in another index | ||
2209 | 1240 | # parents_not_on_page.add(key) | ||
2210 | 1241 | # Missing for some reason | ||
2211 | 1242 | if key < node.min_key: | ||
2212 | 1243 | # in the case of bzr.dev, 3.4k/5.3k misses are | ||
2213 | 1244 | # 'earlier' misses (65%) | ||
2214 | 1245 | parents_not_on_page.add(key) | ||
2215 | 1246 | elif key > node.max_key: | ||
2216 | 1247 | # This parent key would be present on a different | ||
2217 | 1248 | # LeafNode | ||
2218 | 1249 | parents_not_on_page.add(key) | ||
2219 | 1250 | else: | ||
2220 | 1251 | # assert key != node.min_key and key != node.max_key | ||
2221 | 1252 | # If it was going to be present, it would be on | ||
2222 | 1253 | # *this* page, so mark it missing. | ||
2223 | 1254 | missing_keys.add(key) | ||
2224 | 1255 | parents_to_check = next_parents_to_check.difference(parent_map) | ||
2225 | 1256 | # Might want to do another .difference() from missing_keys | ||
2226 | 1257 | # parents_not_on_page could have been found on a different page, or be | ||
2227 | 1258 | # known to be missing. So cull out everything that has already been | ||
2228 | 1259 | # found. | ||
2229 | 1260 | search_keys = parents_not_on_page.difference( | ||
2230 | 1261 | parent_map).difference(missing_keys) | ||
2231 | 1262 | return search_keys | ||
2232 | 1263 | |||
2233 | 1123 | def iter_entries_prefix(self, keys): | 1264 | def iter_entries_prefix(self, keys): |
2234 | 1124 | """Iterate over keys within the index using prefix matching. | 1265 | """Iterate over keys within the index using prefix matching. |
2235 | 1125 | 1266 | ||
2236 | 1126 | 1267 | ||
2237 | === modified file 'bzrlib/builtins.py' | |||
2238 | --- bzrlib/builtins.py 2009-07-27 06:22:57 +0000 | |||
2239 | +++ bzrlib/builtins.py 2009-09-08 16:45:11 +0000 | |||
2240 | @@ -120,6 +120,15 @@ | |||
2241 | 120 | 120 | ||
2242 | 121 | 121 | ||
2243 | 122 | def _get_one_revision_tree(command_name, revisions, branch=None, tree=None): | 122 | def _get_one_revision_tree(command_name, revisions, branch=None, tree=None): |
2244 | 123 | """Get a revision tree. Not suitable for commands that change the tree. | ||
2245 | 124 | |||
2246 | 125 | Specifically, the basis tree in dirstate trees is coupled to the dirstate | ||
2247 | 126 | and doing a commit/uncommit/pull will at best fail due to changing the | ||
2248 | 127 | basis revision data. | ||
2249 | 128 | |||
2250 | 129 | If tree is passed in, it should be already locked, for lifetime management | ||
2251 | 130 | of the trees internal cached state. | ||
2252 | 131 | """ | ||
2253 | 123 | if branch is None: | 132 | if branch is None: |
2254 | 124 | branch = tree.branch | 133 | branch = tree.branch |
2255 | 125 | if revisions is None: | 134 | if revisions is None: |
2256 | @@ -452,8 +461,8 @@ | |||
2257 | 452 | raise errors.BzrCommandError("You cannot remove the working tree" | 461 | raise errors.BzrCommandError("You cannot remove the working tree" |
2258 | 453 | " of a remote path") | 462 | " of a remote path") |
2259 | 454 | if not force: | 463 | if not force: |
2262 | 455 | # XXX: What about pending merges ? -- vila 20090629 | 464 | if (working.has_changes(working.basis_tree()) |
2263 | 456 | if working.has_changes(working.basis_tree()): | 465 | or len(working.get_parent_ids()) > 1): |
2264 | 457 | raise errors.UncommittedChanges(working) | 466 | raise errors.UncommittedChanges(working) |
2265 | 458 | 467 | ||
2266 | 459 | working_path = working.bzrdir.root_transport.base | 468 | working_path = working.bzrdir.root_transport.base |
2267 | @@ -603,6 +612,9 @@ | |||
2268 | 603 | branches that will be merged later (without showing the two different | 612 | branches that will be merged later (without showing the two different |
2269 | 604 | adds as a conflict). It is also useful when merging another project | 613 | adds as a conflict). It is also useful when merging another project |
2270 | 605 | into a subdirectory of this one. | 614 | into a subdirectory of this one. |
2271 | 615 | |||
2272 | 616 | Any files matching patterns in the ignore list will not be added | ||
2273 | 617 | unless they are explicitly mentioned. | ||
2274 | 606 | """ | 618 | """ |
2275 | 607 | takes_args = ['file*'] | 619 | takes_args = ['file*'] |
2276 | 608 | takes_options = [ | 620 | takes_options = [ |
2277 | @@ -616,7 +628,7 @@ | |||
2278 | 616 | help='Lookup file ids from this tree.'), | 628 | help='Lookup file ids from this tree.'), |
2279 | 617 | ] | 629 | ] |
2280 | 618 | encoding_type = 'replace' | 630 | encoding_type = 'replace' |
2282 | 619 | _see_also = ['remove'] | 631 | _see_also = ['remove', 'ignore'] |
2283 | 620 | 632 | ||
2284 | 621 | def run(self, file_list, no_recurse=False, dry_run=False, verbose=False, | 633 | def run(self, file_list, no_recurse=False, dry_run=False, verbose=False, |
2285 | 622 | file_ids_from=None): | 634 | file_ids_from=None): |
2286 | @@ -654,14 +666,6 @@ | |||
2287 | 654 | for path in ignored[glob]: | 666 | for path in ignored[glob]: |
2288 | 655 | self.outf.write("ignored %s matching \"%s\"\n" | 667 | self.outf.write("ignored %s matching \"%s\"\n" |
2289 | 656 | % (path, glob)) | 668 | % (path, glob)) |
2290 | 657 | else: | ||
2291 | 658 | match_len = 0 | ||
2292 | 659 | for glob, paths in ignored.items(): | ||
2293 | 660 | match_len += len(paths) | ||
2294 | 661 | self.outf.write("ignored %d file(s).\n" % match_len) | ||
2295 | 662 | self.outf.write("If you wish to add ignored files, " | ||
2296 | 663 | "please add them explicitly by name. " | ||
2297 | 664 | "(\"bzr ignored\" gives a list)\n") | ||
2298 | 665 | 669 | ||
2299 | 666 | 670 | ||
2300 | 667 | class cmd_mkdir(Command): | 671 | class cmd_mkdir(Command): |
2301 | @@ -1172,6 +1176,9 @@ | |||
2302 | 1172 | help='Hard-link working tree files where possible.'), | 1176 | help='Hard-link working tree files where possible.'), |
2303 | 1173 | Option('no-tree', | 1177 | Option('no-tree', |
2304 | 1174 | help="Create a branch without a working-tree."), | 1178 | help="Create a branch without a working-tree."), |
2305 | 1179 | Option('switch', | ||
2306 | 1180 | help="Switch the checkout in the current directory " | ||
2307 | 1181 | "to the new branch."), | ||
2308 | 1175 | Option('stacked', | 1182 | Option('stacked', |
2309 | 1176 | help='Create a stacked branch referring to the source branch. ' | 1183 | help='Create a stacked branch referring to the source branch. ' |
2310 | 1177 | 'The new branch will depend on the availability of the source ' | 1184 | 'The new branch will depend on the availability of the source ' |
2311 | @@ -1188,9 +1195,9 @@ | |||
2312 | 1188 | 1195 | ||
2313 | 1189 | def run(self, from_location, to_location=None, revision=None, | 1196 | def run(self, from_location, to_location=None, revision=None, |
2314 | 1190 | hardlink=False, stacked=False, standalone=False, no_tree=False, | 1197 | hardlink=False, stacked=False, standalone=False, no_tree=False, |
2316 | 1191 | use_existing_dir=False): | 1198 | use_existing_dir=False, switch=False): |
2317 | 1199 | from bzrlib import switch as _mod_switch | ||
2318 | 1192 | from bzrlib.tag import _merge_tags_if_possible | 1200 | from bzrlib.tag import _merge_tags_if_possible |
2319 | 1193 | |||
2320 | 1194 | accelerator_tree, br_from = bzrdir.BzrDir.open_tree_or_branch( | 1201 | accelerator_tree, br_from = bzrdir.BzrDir.open_tree_or_branch( |
2321 | 1195 | from_location) | 1202 | from_location) |
2322 | 1196 | if (accelerator_tree is not None and | 1203 | if (accelerator_tree is not None and |
2323 | @@ -1250,6 +1257,12 @@ | |||
2324 | 1250 | except (errors.NotStacked, errors.UnstackableBranchFormat, | 1257 | except (errors.NotStacked, errors.UnstackableBranchFormat, |
2325 | 1251 | errors.UnstackableRepositoryFormat), e: | 1258 | errors.UnstackableRepositoryFormat), e: |
2326 | 1252 | note('Branched %d revision(s).' % branch.revno()) | 1259 | note('Branched %d revision(s).' % branch.revno()) |
2327 | 1260 | if switch: | ||
2328 | 1261 | # Switch to the new branch | ||
2329 | 1262 | wt, _ = WorkingTree.open_containing('.') | ||
2330 | 1263 | _mod_switch.switch(wt.bzrdir, branch) | ||
2331 | 1264 | note('Switched to branch: %s', | ||
2332 | 1265 | urlutils.unescape_for_display(branch.base, 'utf-8')) | ||
2333 | 1253 | finally: | 1266 | finally: |
2334 | 1254 | br_from.unlock() | 1267 | br_from.unlock() |
2335 | 1255 | 1268 | ||
2336 | @@ -3025,6 +3038,10 @@ | |||
2337 | 3025 | raise errors.BzrCommandError("empty commit message specified") | 3038 | raise errors.BzrCommandError("empty commit message specified") |
2338 | 3026 | return my_message | 3039 | return my_message |
2339 | 3027 | 3040 | ||
2340 | 3041 | # The API permits a commit with a filter of [] to mean 'select nothing' | ||
2341 | 3042 | # but the command line should not do that. | ||
2342 | 3043 | if not selected_list: | ||
2343 | 3044 | selected_list = None | ||
2344 | 3028 | try: | 3045 | try: |
2345 | 3029 | tree.commit(message_callback=get_message, | 3046 | tree.commit(message_callback=get_message, |
2346 | 3030 | specific_files=selected_list, | 3047 | specific_files=selected_list, |
2347 | @@ -3365,6 +3382,8 @@ | |||
2348 | 3365 | Option('lsprof-timed', | 3382 | Option('lsprof-timed', |
2349 | 3366 | help='Generate lsprof output for benchmarked' | 3383 | help='Generate lsprof output for benchmarked' |
2350 | 3367 | ' sections of code.'), | 3384 | ' sections of code.'), |
2351 | 3385 | Option('lsprof-tests', | ||
2352 | 3386 | help='Generate lsprof output for each test.'), | ||
2353 | 3368 | Option('cache-dir', type=str, | 3387 | Option('cache-dir', type=str, |
2354 | 3369 | help='Cache intermediate benchmark output in this ' | 3388 | help='Cache intermediate benchmark output in this ' |
2355 | 3370 | 'directory.'), | 3389 | 'directory.'), |
2356 | @@ -3411,7 +3430,7 @@ | |||
2357 | 3411 | first=False, list_only=False, | 3430 | first=False, list_only=False, |
2358 | 3412 | randomize=None, exclude=None, strict=False, | 3431 | randomize=None, exclude=None, strict=False, |
2359 | 3413 | load_list=None, debugflag=None, starting_with=None, subunit=False, | 3432 | load_list=None, debugflag=None, starting_with=None, subunit=False, |
2361 | 3414 | parallel=None): | 3433 | parallel=None, lsprof_tests=False): |
2362 | 3415 | from bzrlib.tests import selftest | 3434 | from bzrlib.tests import selftest |
2363 | 3416 | import bzrlib.benchmarks as benchmarks | 3435 | import bzrlib.benchmarks as benchmarks |
2364 | 3417 | from bzrlib.benchmarks import tree_creator | 3436 | from bzrlib.benchmarks import tree_creator |
2365 | @@ -3451,6 +3470,7 @@ | |||
2366 | 3451 | "transport": transport, | 3470 | "transport": transport, |
2367 | 3452 | "test_suite_factory": test_suite_factory, | 3471 | "test_suite_factory": test_suite_factory, |
2368 | 3453 | "lsprof_timed": lsprof_timed, | 3472 | "lsprof_timed": lsprof_timed, |
2369 | 3473 | "lsprof_tests": lsprof_tests, | ||
2370 | 3454 | "bench_history": benchfile, | 3474 | "bench_history": benchfile, |
2371 | 3455 | "matching_tests_first": first, | 3475 | "matching_tests_first": first, |
2372 | 3456 | "list_only": list_only, | 3476 | "list_only": list_only, |
2373 | @@ -3633,13 +3653,14 @@ | |||
2374 | 3633 | verified = 'inapplicable' | 3653 | verified = 'inapplicable' |
2375 | 3634 | tree = WorkingTree.open_containing(directory)[0] | 3654 | tree = WorkingTree.open_containing(directory)[0] |
2376 | 3635 | 3655 | ||
2377 | 3636 | # die as quickly as possible if there are uncommitted changes | ||
2378 | 3637 | try: | 3656 | try: |
2379 | 3638 | basis_tree = tree.revision_tree(tree.last_revision()) | 3657 | basis_tree = tree.revision_tree(tree.last_revision()) |
2380 | 3639 | except errors.NoSuchRevision: | 3658 | except errors.NoSuchRevision: |
2381 | 3640 | basis_tree = tree.basis_tree() | 3659 | basis_tree = tree.basis_tree() |
2382 | 3660 | |||
2383 | 3661 | # die as quickly as possible if there are uncommitted changes | ||
2384 | 3641 | if not force: | 3662 | if not force: |
2386 | 3642 | if tree.has_changes(basis_tree): | 3663 | if tree.has_changes(basis_tree) or len(tree.get_parent_ids()) > 1: |
2387 | 3643 | raise errors.UncommittedChanges(tree) | 3664 | raise errors.UncommittedChanges(tree) |
2388 | 3644 | 3665 | ||
2389 | 3645 | view_info = _get_view_info_for_change_reporter(tree) | 3666 | view_info = _get_view_info_for_change_reporter(tree) |
2390 | @@ -5627,8 +5648,12 @@ | |||
2391 | 5627 | if writer is None: | 5648 | if writer is None: |
2392 | 5628 | writer = bzrlib.option.diff_writer_registry.get() | 5649 | writer = bzrlib.option.diff_writer_registry.get() |
2393 | 5629 | try: | 5650 | try: |
2396 | 5630 | Shelver.from_args(writer(sys.stdout), revision, all, file_list, | 5651 | shelver = Shelver.from_args(writer(sys.stdout), revision, all, |
2397 | 5631 | message, destroy=destroy).run() | 5652 | file_list, message, destroy=destroy) |
2398 | 5653 | try: | ||
2399 | 5654 | shelver.run() | ||
2400 | 5655 | finally: | ||
2401 | 5656 | shelver.work_tree.unlock() | ||
2402 | 5632 | except errors.UserAbort: | 5657 | except errors.UserAbort: |
2403 | 5633 | return 0 | 5658 | return 0 |
2404 | 5634 | 5659 | ||
2405 | @@ -5673,7 +5698,11 @@ | |||
2406 | 5673 | 5698 | ||
2407 | 5674 | def run(self, shelf_id=None, action='apply'): | 5699 | def run(self, shelf_id=None, action='apply'): |
2408 | 5675 | from bzrlib.shelf_ui import Unshelver | 5700 | from bzrlib.shelf_ui import Unshelver |
2410 | 5676 | Unshelver.from_args(shelf_id, action).run() | 5701 | unshelver = Unshelver.from_args(shelf_id, action) |
2411 | 5702 | try: | ||
2412 | 5703 | unshelver.run() | ||
2413 | 5704 | finally: | ||
2414 | 5705 | unshelver.tree.unlock() | ||
2415 | 5677 | 5706 | ||
2416 | 5678 | 5707 | ||
2417 | 5679 | class cmd_clean_tree(Command): | 5708 | class cmd_clean_tree(Command): |
2418 | 5680 | 5709 | ||
2419 | === modified file 'bzrlib/bzrdir.py' | |||
2420 | --- bzrlib/bzrdir.py 2009-08-04 14:48:59 +0000 | |||
2421 | +++ bzrlib/bzrdir.py 2009-08-21 02:10:06 +0000 | |||
2422 | @@ -129,9 +129,16 @@ | |||
2423 | 129 | return True | 129 | return True |
2424 | 130 | 130 | ||
2425 | 131 | def check_conversion_target(self, target_format): | 131 | def check_conversion_target(self, target_format): |
2426 | 132 | """Check that a bzrdir as a whole can be converted to a new format.""" | ||
2427 | 133 | # The only current restriction is that the repository content can be | ||
2428 | 134 | # fetched compatibly with the target. | ||
2429 | 132 | target_repo_format = target_format.repository_format | 135 | target_repo_format = target_format.repository_format |
2432 | 133 | source_repo_format = self._format.repository_format | 136 | try: |
2433 | 134 | source_repo_format.check_conversion_target(target_repo_format) | 137 | self.open_repository()._format.check_conversion_target( |
2434 | 138 | target_repo_format) | ||
2435 | 139 | except errors.NoRepositoryPresent: | ||
2436 | 140 | # No repo, no problem. | ||
2437 | 141 | pass | ||
2438 | 135 | 142 | ||
2439 | 136 | @staticmethod | 143 | @staticmethod |
2440 | 137 | def _check_supported(format, allow_unsupported, | 144 | def _check_supported(format, allow_unsupported, |
2441 | @@ -3039,7 +3046,8 @@ | |||
2442 | 3039 | new is _mod_branch.BzrBranchFormat8): | 3046 | new is _mod_branch.BzrBranchFormat8): |
2443 | 3040 | branch_converter = _mod_branch.Converter7to8() | 3047 | branch_converter = _mod_branch.Converter7to8() |
2444 | 3041 | else: | 3048 | else: |
2446 | 3042 | raise errors.BadConversionTarget("No converter", new) | 3049 | raise errors.BadConversionTarget("No converter", new, |
2447 | 3050 | branch._format) | ||
2448 | 3043 | branch_converter.convert(branch) | 3051 | branch_converter.convert(branch) |
2449 | 3044 | branch = self.bzrdir.open_branch() | 3052 | branch = self.bzrdir.open_branch() |
2450 | 3045 | old = branch._format.__class__ | 3053 | old = branch._format.__class__ |
2451 | @@ -3548,6 +3556,10 @@ | |||
2452 | 3548 | if self._require_stacking: | 3556 | if self._require_stacking: |
2453 | 3549 | raise | 3557 | raise |
2454 | 3550 | 3558 | ||
2455 | 3559 | def requires_stacking(self): | ||
2456 | 3560 | """Return True if this policy requires stacking.""" | ||
2457 | 3561 | return self._stack_on is not None and self._require_stacking | ||
2458 | 3562 | |||
2459 | 3551 | def _get_full_stack_on(self): | 3563 | def _get_full_stack_on(self): |
2460 | 3552 | """Get a fully-qualified URL for the stack_on location.""" | 3564 | """Get a fully-qualified URL for the stack_on location.""" |
2461 | 3553 | if self._stack_on is None: | 3565 | if self._stack_on is None: |
2462 | @@ -3860,11 +3872,11 @@ | |||
2463 | 3860 | # The following format should be an alias for the rich root equivalent | 3872 | # The following format should be an alias for the rich root equivalent |
2464 | 3861 | # of the default format | 3873 | # of the default format |
2465 | 3862 | format_registry.register_metadir('default-rich-root', | 3874 | format_registry.register_metadir('default-rich-root', |
2470 | 3863 | 'bzrlib.repofmt.pack_repo.RepositoryFormatKnitPack4', | 3875 | 'bzrlib.repofmt.groupcompress_repo.RepositoryFormat2a', |
2471 | 3864 | help='Default format, rich root variant. (needed for bzr-svn and bzr-git).', | 3876 | branch_format='bzrlib.branch.BzrBranchFormat7', |
2472 | 3865 | branch_format='bzrlib.branch.BzrBranchFormat6', | 3877 | tree_format='bzrlib.workingtree.WorkingTreeFormat6', |
2469 | 3866 | tree_format='bzrlib.workingtree.WorkingTreeFormat4', | ||
2473 | 3867 | alias=True, | 3878 | alias=True, |
2475 | 3868 | ) | 3879 | help='Same as 2a.') |
2476 | 3880 | |||
2477 | 3869 | # The current format that is made on 'bzr init'. | 3881 | # The current format that is made on 'bzr init'. |
2479 | 3870 | format_registry.set_default('pack-0.92') | 3882 | format_registry.set_default('2a') |
2480 | 3871 | 3883 | ||
2481 | === modified file 'bzrlib/commands.py' | |||
2482 | --- bzrlib/commands.py 2009-06-19 09:06:56 +0000 | |||
2483 | +++ bzrlib/commands.py 2009-09-09 17:51:19 +0000 | |||
2484 | @@ -1028,13 +1028,13 @@ | |||
2485 | 1028 | ret = apply_coveraged(opt_coverage_dir, run, *run_argv) | 1028 | ret = apply_coveraged(opt_coverage_dir, run, *run_argv) |
2486 | 1029 | else: | 1029 | else: |
2487 | 1030 | ret = run(*run_argv) | 1030 | ret = run(*run_argv) |
2488 | 1031 | if 'memory' in debug.debug_flags: | ||
2489 | 1032 | trace.debug_memory('Process status after command:', short=False) | ||
2490 | 1033 | return ret or 0 | 1031 | return ret or 0 |
2491 | 1034 | finally: | 1032 | finally: |
2492 | 1035 | # reset, in case we may do other commands later within the same | 1033 | # reset, in case we may do other commands later within the same |
2493 | 1036 | # process. Commands that want to execute sub-commands must propagate | 1034 | # process. Commands that want to execute sub-commands must propagate |
2494 | 1037 | # --verbose in their own way. | 1035 | # --verbose in their own way. |
2495 | 1036 | if 'memory' in debug.debug_flags: | ||
2496 | 1037 | trace.debug_memory('Process status after command:', short=False) | ||
2497 | 1038 | option._verbosity_level = saved_verbosity_level | 1038 | option._verbosity_level = saved_verbosity_level |
2498 | 1039 | 1039 | ||
2499 | 1040 | 1040 | ||
2500 | 1041 | 1041 | ||
2501 | === modified file 'bzrlib/commit.py' | |||
2502 | --- bzrlib/commit.py 2009-08-25 05:09:42 +0000 | |||
2503 | +++ bzrlib/commit.py 2009-08-28 05:00:33 +0000 | |||
2504 | @@ -209,7 +209,8 @@ | |||
2505 | 209 | :param timestamp: if not None, seconds-since-epoch for a | 209 | :param timestamp: if not None, seconds-since-epoch for a |
2506 | 210 | postdated/predated commit. | 210 | postdated/predated commit. |
2507 | 211 | 211 | ||
2509 | 212 | :param specific_files: If true, commit only those files. | 212 | :param specific_files: If not None, commit only those files. An empty |
2510 | 213 | list means 'commit no files'. | ||
2511 | 213 | 214 | ||
2512 | 214 | :param rev_id: If set, use this as the new revision id. | 215 | :param rev_id: If set, use this as the new revision id. |
2513 | 215 | Useful for test or import commands that need to tightly | 216 | Useful for test or import commands that need to tightly |
2514 | @@ -264,6 +265,8 @@ | |||
2515 | 264 | self.master_locked = False | 265 | self.master_locked = False |
2516 | 265 | self.recursive = recursive | 266 | self.recursive = recursive |
2517 | 266 | self.rev_id = None | 267 | self.rev_id = None |
2518 | 268 | # self.specific_files is None to indicate no filter, or any iterable to | ||
2519 | 269 | # indicate a filter - [] means no files at all, as per iter_changes. | ||
2520 | 267 | if specific_files is not None: | 270 | if specific_files is not None: |
2521 | 268 | self.specific_files = sorted( | 271 | self.specific_files = sorted( |
2522 | 269 | minimum_path_selection(specific_files)) | 272 | minimum_path_selection(specific_files)) |
2523 | @@ -285,7 +288,6 @@ | |||
2524 | 285 | # the command line parameters, and the repository has fast delta | 288 | # the command line parameters, and the repository has fast delta |
2525 | 286 | # generation. See bug 347649. | 289 | # generation. See bug 347649. |
2526 | 287 | self.use_record_iter_changes = ( | 290 | self.use_record_iter_changes = ( |
2527 | 288 | not self.specific_files and | ||
2528 | 289 | not self.exclude and | 291 | not self.exclude and |
2529 | 290 | not self.branch.repository._format.supports_tree_reference and | 292 | not self.branch.repository._format.supports_tree_reference and |
2530 | 291 | (self.branch.repository._format.fast_deltas or | 293 | (self.branch.repository._format.fast_deltas or |
2531 | @@ -333,7 +335,7 @@ | |||
2532 | 333 | self._gather_parents() | 335 | self._gather_parents() |
2533 | 334 | # After a merge, a selected file commit is not supported. | 336 | # After a merge, a selected file commit is not supported. |
2534 | 335 | # See 'bzr help merge' for an explanation as to why. | 337 | # See 'bzr help merge' for an explanation as to why. |
2536 | 336 | if len(self.parents) > 1 and self.specific_files: | 338 | if len(self.parents) > 1 and self.specific_files is not None: |
2537 | 337 | raise errors.CannotCommitSelectedFileMerge(self.specific_files) | 339 | raise errors.CannotCommitSelectedFileMerge(self.specific_files) |
2538 | 338 | # Excludes are a form of selected file commit. | 340 | # Excludes are a form of selected file commit. |
2539 | 339 | if len(self.parents) > 1 and self.exclude: | 341 | if len(self.parents) > 1 and self.exclude: |
2540 | @@ -619,12 +621,13 @@ | |||
2541 | 619 | """Update the commit builder with the data about what has changed. | 621 | """Update the commit builder with the data about what has changed. |
2542 | 620 | """ | 622 | """ |
2543 | 621 | exclude = self.exclude | 623 | exclude = self.exclude |
2545 | 622 | specific_files = self.specific_files or [] | 624 | specific_files = self.specific_files |
2546 | 623 | mutter("Selecting files for commit with filter %s", specific_files) | 625 | mutter("Selecting files for commit with filter %s", specific_files) |
2547 | 624 | 626 | ||
2548 | 625 | self._check_strict() | 627 | self._check_strict() |
2549 | 626 | if self.use_record_iter_changes: | 628 | if self.use_record_iter_changes: |
2551 | 627 | iter_changes = self.work_tree.iter_changes(self.basis_tree) | 629 | iter_changes = self.work_tree.iter_changes(self.basis_tree, |
2552 | 630 | specific_files=specific_files) | ||
2553 | 628 | iter_changes = self._filter_iter_changes(iter_changes) | 631 | iter_changes = self._filter_iter_changes(iter_changes) |
2554 | 629 | for file_id, path, fs_hash in self.builder.record_iter_changes( | 632 | for file_id, path, fs_hash in self.builder.record_iter_changes( |
2555 | 630 | self.work_tree, self.basis_revid, iter_changes): | 633 | self.work_tree, self.basis_revid, iter_changes): |
2556 | 631 | 634 | ||
2557 | === modified file 'bzrlib/config.py' | |||
2558 | --- bzrlib/config.py 2009-07-02 08:59:16 +0000 | |||
2559 | +++ bzrlib/config.py 2009-08-20 04:53:23 +0000 | |||
2560 | @@ -821,6 +821,29 @@ | |||
2561 | 821 | return osutils.pathjoin(config_dir(), 'ignore') | 821 | return osutils.pathjoin(config_dir(), 'ignore') |
2562 | 822 | 822 | ||
2563 | 823 | 823 | ||
2564 | 824 | def crash_dir(): | ||
2565 | 825 | """Return the directory name to store crash files. | ||
2566 | 826 | |||
2567 | 827 | This doesn't implicitly create it. | ||
2568 | 828 | |||
2569 | 829 | On Windows it's in the config directory; elsewhere in the XDG cache directory. | ||
2570 | 830 | """ | ||
2571 | 831 | if sys.platform == 'win32': | ||
2572 | 832 | return osutils.pathjoin(config_dir(), 'Crash') | ||
2573 | 833 | else: | ||
2574 | 834 | return osutils.pathjoin(xdg_cache_dir(), 'crash') | ||
2575 | 835 | |||
2576 | 836 | |||
2577 | 837 | def xdg_cache_dir(): | ||
2578 | 838 | # See http://standards.freedesktop.org/basedir-spec/latest/ar01s03.html | ||
2579 | 839 | # Possibly this should be different on Windows? | ||
2580 | 840 | e = os.environ.get('XDG_CACHE_DIR', None) | ||
2581 | 841 | if e: | ||
2582 | 842 | return e | ||
2583 | 843 | else: | ||
2584 | 844 | return os.path.expanduser('~/.cache') | ||
2585 | 845 | |||
2586 | 846 | |||
2587 | 824 | def _auto_user_id(): | 847 | def _auto_user_id(): |
2588 | 825 | """Calculate automatic user identification. | 848 | """Calculate automatic user identification. |
2589 | 826 | 849 | ||
2590 | 827 | 850 | ||
2591 | === added file 'bzrlib/crash.py' | |||
2592 | --- bzrlib/crash.py 1970-01-01 00:00:00 +0000 | |||
2593 | +++ bzrlib/crash.py 2009-08-20 05:47:53 +0000 | |||
2594 | @@ -0,0 +1,163 @@ | |||
2595 | 1 | # Copyright (C) 2009 Canonical Ltd | ||
2596 | 2 | # | ||
2597 | 3 | # This program is free software; you can redistribute it and/or modify | ||
2598 | 4 | # it under the terms of the GNU General Public License as published by | ||
2599 | 5 | # the Free Software Foundation; either version 2 of the License, or | ||
2600 | 6 | # (at your option) any later version. | ||
2601 | 7 | # | ||
2602 | 8 | # This program is distributed in the hope that it will be useful, | ||
2603 | 9 | # but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
2604 | 10 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
2605 | 11 | # GNU General Public License for more details. | ||
2606 | 12 | # | ||
2607 | 13 | # You should have received a copy of the GNU General Public License | ||
2608 | 14 | # along with this program; if not, write to the Free Software | ||
2609 | 15 | # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA | ||
2610 | 16 | |||
2611 | 17 | |||
2612 | 18 | """Handling and reporting crashes. | ||
2613 | 19 | """ | ||
2614 | 20 | |||
2615 | 21 | # for interactive testing, try the 'bzr assert-fail' command | ||
2616 | 22 | # or see http://code.launchpad.net/~mbp/bzr/bzr-fail | ||
2617 | 23 | |||
2618 | 24 | import os | ||
2619 | 25 | import platform | ||
2620 | 26 | import pprint | ||
2621 | 27 | import sys | ||
2622 | 28 | import time | ||
2623 | 29 | from StringIO import StringIO | ||
2624 | 30 | |||
2625 | 31 | import bzrlib | ||
2626 | 32 | from bzrlib import ( | ||
2627 | 33 | config, | ||
2628 | 34 | debug, | ||
2629 | 35 | osutils, | ||
2630 | 36 | plugin, | ||
2631 | 37 | trace, | ||
2632 | 38 | ) | ||
2633 | 39 | |||
2634 | 40 | |||
2635 | 41 | def report_bug(exc_info, stderr): | ||
2636 | 42 | if 'no_apport' not in debug.debug_flags: | ||
2637 | 43 | try: | ||
2638 | 44 | report_bug_to_apport(exc_info, stderr) | ||
2639 | 45 | return | ||
2640 | 46 | except ImportError, e: | ||
2641 | 47 | trace.mutter("couldn't find apport bug-reporting library: %s" % e) | ||
2642 | 48 | pass | ||
2643 | 49 | except Exception, e: | ||
2644 | 50 | # this should only happen if apport is installed but it didn't | ||
2645 | 51 | # work, eg because of an io error writing the crash file | ||
2646 | 52 | sys.stderr.write("bzr: failed to report crash using apport:\n " | ||
2647 | 53 | " %r\n" % e) | ||
2648 | 54 | pass | ||
2649 | 55 | report_bug_legacy(exc_info, stderr) | ||
2650 | 56 | |||
2651 | 57 | |||
2652 | 58 | def report_bug_legacy(exc_info, err_file): | ||
2653 | 59 | """Report a bug by just printing a message to the user.""" | ||
2654 | 60 | trace.print_exception(exc_info, err_file) | ||
2655 | 61 | err_file.write('\n') | ||
2656 | 62 | err_file.write('bzr %s on python %s (%s)\n' % \ | ||
2657 | 63 | (bzrlib.__version__, | ||
2658 | 64 | bzrlib._format_version_tuple(sys.version_info), | ||
2659 | 65 | platform.platform(aliased=1))) | ||
2660 | 66 | err_file.write('arguments: %r\n' % sys.argv) | ||
2661 | 67 | err_file.write( | ||
2662 | 68 | 'encoding: %r, fsenc: %r, lang: %r\n' % ( | ||
2663 | 69 | osutils.get_user_encoding(), sys.getfilesystemencoding(), | ||
2664 | 70 | os.environ.get('LANG'))) | ||
2665 | 71 | err_file.write("plugins:\n") | ||
2666 | 72 | err_file.write(_format_plugin_list()) | ||
2667 | 73 | err_file.write( | ||
2668 | 74 | "\n\n" | ||
2669 | 75 | "*** Bazaar has encountered an internal error. This probably indicates a\n" | ||
2670 | 76 | " bug in Bazaar. You can help us fix it by filing a bug report at\n" | ||
2671 | 77 | " https://bugs.launchpad.net/bzr/+filebug\n" | ||
2672 | 78 | " including this traceback and a description of the problem.\n" | ||
2673 | 79 | ) | ||
2674 | 80 | |||
2675 | 81 | |||
2676 | 82 | def report_bug_to_apport(exc_info, stderr): | ||
2677 | 83 | """Report a bug to apport for optional automatic filing. | ||
2678 | 84 | """ | ||
2679 | 85 | # this is based on apport_package_hook.py, but omitting some of the | ||
2680 | 86 | # Ubuntu-specific policy about what to report and when | ||
2681 | 87 | |||
2682 | 88 | # if this fails its caught at a higher level; we don't want to open the | ||
2683 | 89 | # crash file unless apport can be loaded. | ||
2684 | 90 | import apport | ||
2685 | 91 | |||
2686 | 92 | crash_file = _open_crash_file() | ||
2687 | 93 | try: | ||
2688 | 94 | _write_apport_report_to_file(exc_info, crash_file) | ||
2689 | 95 | finally: | ||
2690 | 96 | crash_file.close() | ||
2691 | 97 | |||
2692 | 98 | stderr.write("bzr: ERROR: %s.%s: %s\n" | ||
2693 | 99 | "\n" | ||
2694 | 100 | "*** Bazaar has encountered an internal error. This probably indicates a\n" | ||
2695 | 101 | " bug in Bazaar. You can help us fix it by filing a bug report at\n" | ||
2696 | 102 | " https://bugs.launchpad.net/bzr/+filebug\n" | ||
2697 | 103 | " attaching the crash file\n" | ||
2698 | 104 | " %s\n" | ||
2699 | 105 | " and including a description of the problem.\n" | ||
2700 | 106 | "\n" | ||
2701 | 107 | " The crash file is plain text and you can inspect or edit it to remove\n" | ||
2702 | 108 | " private information.\n" | ||
2703 | 109 | % (exc_info[0].__module__, exc_info[0].__name__, exc_info[1], | ||
2704 | 110 | crash_file.name)) | ||
2705 | 111 | |||
2706 | 112 | |||
2707 | 113 | def _write_apport_report_to_file(exc_info, crash_file): | ||
2708 | 114 | import traceback | ||
2709 | 115 | from apport.report import Report | ||
2710 | 116 | |||
2711 | 117 | exc_type, exc_object, exc_tb = exc_info | ||
2712 | 118 | |||
2713 | 119 | pr = Report() | ||
2714 | 120 | # add_proc_info gives you the memory map of the process: this seems rarely | ||
2715 | 121 | # useful for Bazaar and it does make the report harder to scan, though it | ||
2716 | 122 | # does tell you what binary modules are loaded. | ||
2717 | 123 | # pr.add_proc_info() | ||
2718 | 124 | pr.add_user_info() | ||
2719 | 125 | pr['CommandLine'] = pprint.pformat(sys.argv) | ||
2720 | 126 | pr['BzrVersion'] = bzrlib.__version__ | ||
2721 | 127 | pr['PythonVersion'] = bzrlib._format_version_tuple(sys.version_info) | ||
2722 | 128 | pr['Platform'] = platform.platform(aliased=1) | ||
2723 | 129 | pr['UserEncoding'] = osutils.get_user_encoding() | ||
2724 | 130 | pr['FileSystemEncoding'] = sys.getfilesystemencoding() | ||
2725 | 131 | pr['Locale'] = os.environ.get('LANG') | ||
2726 | 132 | pr['BzrPlugins'] = _format_plugin_list() | ||
2727 | 133 | pr['PythonLoadedModules'] = _format_module_list() | ||
2728 | 134 | pr['BzrDebugFlags'] = pprint.pformat(debug.debug_flags) | ||
2729 | 135 | |||
2730 | 136 | tb_file = StringIO() | ||
2731 | 137 | traceback.print_exception(exc_type, exc_object, exc_tb, file=tb_file) | ||
2732 | 138 | pr['Traceback'] = tb_file.getvalue() | ||
2733 | 139 | |||
2734 | 140 | pr.write(crash_file) | ||
2735 | 141 | |||
2736 | 142 | |||
2737 | 143 | def _open_crash_file(): | ||
2738 | 144 | crash_dir = config.crash_dir() | ||
2739 | 145 | # user-readable only, just in case the contents are sensitive. | ||
2740 | 146 | if not osutils.isdir(crash_dir): | ||
2741 | 147 | os.makedirs(crash_dir, mode=0700) | ||
2742 | 148 | filename = 'bzr-%s-%s.crash' % ( | ||
2743 | 149 | osutils.compact_date(time.time()), | ||
2744 | 150 | os.getpid(),) | ||
2745 | 151 | return open(osutils.pathjoin(crash_dir, filename), 'wt') | ||
2746 | 152 | |||
2747 | 153 | |||
2748 | 154 | def _format_plugin_list(): | ||
2749 | 155 | plugin_lines = [] | ||
2750 | 156 | for name, a_plugin in sorted(plugin.plugins().items()): | ||
2751 | 157 | plugin_lines.append(" %-20s %s [%s]" % | ||
2752 | 158 | (name, a_plugin.path(), a_plugin.__version__)) | ||
2753 | 159 | return '\n'.join(plugin_lines) | ||
2754 | 160 | |||
2755 | 161 | |||
2756 | 162 | def _format_module_list(): | ||
2757 | 163 | return pprint.pformat(sys.modules) | ||
2758 | 0 | 164 | ||
2759 | === modified file 'bzrlib/dirstate.py' | |||
2760 | --- bzrlib/dirstate.py 2009-07-27 05:44:19 +0000 | |||
2761 | +++ bzrlib/dirstate.py 2009-08-28 05:00:33 +0000 | |||
2762 | @@ -3166,15 +3166,18 @@ | |||
2763 | 3166 | 3166 | ||
2764 | 3167 | __slots__ = ["old_dirname_to_file_id", "new_dirname_to_file_id", | 3167 | __slots__ = ["old_dirname_to_file_id", "new_dirname_to_file_id", |
2765 | 3168 | "last_source_parent", "last_target_parent", "include_unchanged", | 3168 | "last_source_parent", "last_target_parent", "include_unchanged", |
2769 | 3169 | "use_filesystem_for_exec", "utf8_decode", "searched_specific_files", | 3169 | "partial", "use_filesystem_for_exec", "utf8_decode", |
2770 | 3170 | "search_specific_files", "state", "source_index", "target_index", | 3170 | "searched_specific_files", "search_specific_files", |
2771 | 3171 | "want_unversioned", "tree"] | 3171 | "searched_exact_paths", "search_specific_file_parents", "seen_ids", |
2772 | 3172 | "state", "source_index", "target_index", "want_unversioned", "tree"] | ||
2773 | 3172 | 3173 | ||
2774 | 3173 | def __init__(self, include_unchanged, use_filesystem_for_exec, | 3174 | def __init__(self, include_unchanged, use_filesystem_for_exec, |
2775 | 3174 | search_specific_files, state, source_index, target_index, | 3175 | search_specific_files, state, source_index, target_index, |
2776 | 3175 | want_unversioned, tree): | 3176 | want_unversioned, tree): |
2777 | 3176 | self.old_dirname_to_file_id = {} | 3177 | self.old_dirname_to_file_id = {} |
2778 | 3177 | self.new_dirname_to_file_id = {} | 3178 | self.new_dirname_to_file_id = {} |
2779 | 3179 | # Are we doing a partial iter_changes? | ||
2780 | 3180 | self.partial = search_specific_files != set(['']) | ||
2781 | 3178 | # Using a list so that we can access the values and change them in | 3181 | # Using a list so that we can access the values and change them in |
2782 | 3179 | # nested scope. Each one is [path, file_id, entry] | 3182 | # nested scope. Each one is [path, file_id, entry] |
2783 | 3180 | self.last_source_parent = [None, None] | 3183 | self.last_source_parent = [None, None] |
2784 | @@ -3183,14 +3186,25 @@ | |||
2785 | 3183 | self.use_filesystem_for_exec = use_filesystem_for_exec | 3186 | self.use_filesystem_for_exec = use_filesystem_for_exec |
2786 | 3184 | self.utf8_decode = cache_utf8._utf8_decode | 3187 | self.utf8_decode = cache_utf8._utf8_decode |
2787 | 3185 | # for all search_indexs in each path at or under each element of | 3188 | # for all search_indexs in each path at or under each element of |
2791 | 3186 | # search_specific_files, if the detail is relocated: add the id, and add the | 3189 | # search_specific_files, if the detail is relocated: add the id, and |
2792 | 3187 | # relocated path as one to search if its not searched already. If the | 3190 | # add the relocated path as one to search if its not searched already. |
2793 | 3188 | # detail is not relocated, add the id. | 3191 | # If the detail is not relocated, add the id. |
2794 | 3189 | self.searched_specific_files = set() | 3192 | self.searched_specific_files = set() |
2795 | 3193 | # When we search exact paths without expanding downwards, we record | ||
2796 | 3194 | # that here. | ||
2797 | 3195 | self.searched_exact_paths = set() | ||
2798 | 3190 | self.search_specific_files = search_specific_files | 3196 | self.search_specific_files = search_specific_files |
2799 | 3197 | # The parents up to the root of the paths we are searching. | ||
2800 | 3198 | # After all normal paths are returned, these specific items are returned. | ||
2801 | 3199 | self.search_specific_file_parents = set() | ||
2802 | 3200 | # The ids we've sent out in the delta. | ||
2803 | 3201 | self.seen_ids = set() | ||
2804 | 3191 | self.state = state | 3202 | self.state = state |
2805 | 3192 | self.source_index = source_index | 3203 | self.source_index = source_index |
2806 | 3193 | self.target_index = target_index | 3204 | self.target_index = target_index |
2807 | 3205 | if target_index != 0: | ||
2808 | 3206 | # A lot of code in here depends on target_index == 0 | ||
2809 | 3207 | raise errors.BzrError('unsupported target index') | ||
2810 | 3194 | self.want_unversioned = want_unversioned | 3208 | self.want_unversioned = want_unversioned |
2811 | 3195 | self.tree = tree | 3209 | self.tree = tree |
2812 | 3196 | 3210 | ||
2813 | @@ -3198,15 +3212,15 @@ | |||
2814 | 3198 | """Compare an entry and real disk to generate delta information. | 3212 | """Compare an entry and real disk to generate delta information. |
2815 | 3199 | 3213 | ||
2816 | 3200 | :param path_info: top_relpath, basename, kind, lstat, abspath for | 3214 | :param path_info: top_relpath, basename, kind, lstat, abspath for |
2819 | 3201 | the path of entry. If None, then the path is considered absent. | 3215 | the path of entry. If None, then the path is considered absent in |
2820 | 3202 | (Perhaps we should pass in a concrete entry for this ?) | 3216 | the target (Perhaps we should pass in a concrete entry for this ?) |
2821 | 3203 | Basename is returned as a utf8 string because we expect this | 3217 | Basename is returned as a utf8 string because we expect this |
2822 | 3204 | tuple will be ignored, and don't want to take the time to | 3218 | tuple will be ignored, and don't want to take the time to |
2823 | 3205 | decode. | 3219 | decode. |
2824 | 3206 | :return: (iter_changes_result, changed). If the entry has not been | 3220 | :return: (iter_changes_result, changed). If the entry has not been |
2825 | 3207 | handled then changed is None. Otherwise it is False if no content | 3221 | handled then changed is None. Otherwise it is False if no content |
2828 | 3208 | or metadata changes have occured, and None if any content or | 3222 | or metadata changes have occurred, and True if any content or |
2829 | 3209 | metadata change has occured. If self.include_unchanged is True then | 3223 | metadata change has occurred. If self.include_unchanged is True then |
2830 | 3210 | if changed is not None, iter_changes_result will always be a result | 3224 | if changed is not None, iter_changes_result will always be a result |
2831 | 3211 | tuple. Otherwise, iter_changes_result is None unless changed is | 3225 | tuple. Otherwise, iter_changes_result is None unless changed is |
2832 | 3212 | True. | 3226 | True. |
2833 | @@ -3463,6 +3477,25 @@ | |||
2834 | 3463 | def __iter__(self): | 3477 | def __iter__(self): |
2835 | 3464 | return self | 3478 | return self |
2836 | 3465 | 3479 | ||
2837 | 3480 | def _gather_result_for_consistency(self, result): | ||
2838 | 3481 | """Check a result we will yield to make sure we are consistent later. | ||
2839 | 3482 | |||
2840 | 3483 | This gathers result's parents into a set to output later. | ||
2841 | 3484 | |||
2842 | 3485 | :param result: A result tuple. | ||
2843 | 3486 | """ | ||
2844 | 3487 | if not self.partial or not result[0]: | ||
2845 | 3488 | return | ||
2846 | 3489 | self.seen_ids.add(result[0]) | ||
2847 | 3490 | new_path = result[1][1] | ||
2848 | 3491 | if new_path: | ||
2849 | 3492 | # Not the root and not a delete: queue up the parents of the path. | ||
2850 | 3493 | self.search_specific_file_parents.update( | ||
2851 | 3494 | osutils.parent_directories(new_path.encode('utf8'))) | ||
2852 | 3495 | # Add the root directory which parent_directories does not | ||
2853 | 3496 | # provide. | ||
2854 | 3497 | self.search_specific_file_parents.add('') | ||
2855 | 3498 | |||
2856 | 3466 | def iter_changes(self): | 3499 | def iter_changes(self): |
2857 | 3467 | """Iterate over the changes.""" | 3500 | """Iterate over the changes.""" |
2858 | 3468 | utf8_decode = cache_utf8._utf8_decode | 3501 | utf8_decode = cache_utf8._utf8_decode |
2859 | @@ -3546,6 +3579,8 @@ | |||
2860 | 3546 | result, changed = _process_entry(entry, root_dir_info) | 3579 | result, changed = _process_entry(entry, root_dir_info) |
2861 | 3547 | if changed is not None: | 3580 | if changed is not None: |
2862 | 3548 | path_handled = True | 3581 | path_handled = True |
2863 | 3582 | if changed: | ||
2864 | 3583 | self._gather_result_for_consistency(result) | ||
2865 | 3549 | if changed or self.include_unchanged: | 3584 | if changed or self.include_unchanged: |
2866 | 3550 | yield result | 3585 | yield result |
2867 | 3551 | if self.want_unversioned and not path_handled and root_dir_info: | 3586 | if self.want_unversioned and not path_handled and root_dir_info: |
2868 | @@ -3664,6 +3699,8 @@ | |||
2869 | 3664 | # advance the entry only, after processing. | 3699 | # advance the entry only, after processing. |
2870 | 3665 | result, changed = _process_entry(current_entry, None) | 3700 | result, changed = _process_entry(current_entry, None) |
2871 | 3666 | if changed is not None: | 3701 | if changed is not None: |
2872 | 3702 | if changed: | ||
2873 | 3703 | self._gather_result_for_consistency(result) | ||
2874 | 3667 | if changed or self.include_unchanged: | 3704 | if changed or self.include_unchanged: |
2875 | 3668 | yield result | 3705 | yield result |
2876 | 3669 | block_index +=1 | 3706 | block_index +=1 |
2877 | @@ -3702,6 +3739,8 @@ | |||
2878 | 3702 | # no path is fine: the per entry code will handle it. | 3739 | # no path is fine: the per entry code will handle it. |
2879 | 3703 | result, changed = _process_entry(current_entry, current_path_info) | 3740 | result, changed = _process_entry(current_entry, current_path_info) |
2880 | 3704 | if changed is not None: | 3741 | if changed is not None: |
2881 | 3742 | if changed: | ||
2882 | 3743 | self._gather_result_for_consistency(result) | ||
2883 | 3705 | if changed or self.include_unchanged: | 3744 | if changed or self.include_unchanged: |
2884 | 3706 | yield result | 3745 | yield result |
2885 | 3707 | elif (current_entry[0][1] != current_path_info[1] | 3746 | elif (current_entry[0][1] != current_path_info[1] |
2886 | @@ -3723,6 +3762,8 @@ | |||
2887 | 3723 | # advance the entry only, after processing. | 3762 | # advance the entry only, after processing. |
2888 | 3724 | result, changed = _process_entry(current_entry, None) | 3763 | result, changed = _process_entry(current_entry, None) |
2889 | 3725 | if changed is not None: | 3764 | if changed is not None: |
2890 | 3765 | if changed: | ||
2891 | 3766 | self._gather_result_for_consistency(result) | ||
2892 | 3726 | if changed or self.include_unchanged: | 3767 | if changed or self.include_unchanged: |
2893 | 3727 | yield result | 3768 | yield result |
2894 | 3728 | advance_path = False | 3769 | advance_path = False |
2895 | @@ -3730,6 +3771,8 @@ | |||
2896 | 3730 | result, changed = _process_entry(current_entry, current_path_info) | 3771 | result, changed = _process_entry(current_entry, current_path_info) |
2897 | 3731 | if changed is not None: | 3772 | if changed is not None: |
2898 | 3732 | path_handled = True | 3773 | path_handled = True |
2899 | 3774 | if changed: | ||
2900 | 3775 | self._gather_result_for_consistency(result) | ||
2901 | 3733 | if changed or self.include_unchanged: | 3776 | if changed or self.include_unchanged: |
2902 | 3734 | yield result | 3777 | yield result |
2903 | 3735 | if advance_entry and current_entry is not None: | 3778 | if advance_entry and current_entry is not None: |
2904 | @@ -3795,6 +3838,124 @@ | |||
2905 | 3795 | current_dir_info = dir_iterator.next() | 3838 | current_dir_info = dir_iterator.next() |
2906 | 3796 | except StopIteration: | 3839 | except StopIteration: |
2907 | 3797 | current_dir_info = None | 3840 | current_dir_info = None |
2908 | 3841 | for result in self._iter_specific_file_parents(): | ||
2909 | 3842 | yield result | ||
2910 | 3843 | |||
2911 | 3844 | def _iter_specific_file_parents(self): | ||
2912 | 3845 | """Iter over the specific file parents.""" | ||
2913 | 3846 | while self.search_specific_file_parents: | ||
2914 | 3847 | # Process the parent directories for the paths we were iterating. | ||
2915 | 3848 | # Even in extremely large trees this should be modest, so currently | ||
2916 | 3849 | # no attempt is made to optimise. | ||
2917 | 3850 | path_utf8 = self.search_specific_file_parents.pop() | ||
2918 | 3851 | if osutils.is_inside_any(self.searched_specific_files, path_utf8): | ||
2919 | 3852 | # We've examined this path. | ||
2920 | 3853 | continue | ||
2921 | 3854 | if path_utf8 in self.searched_exact_paths: | ||
2922 | 3855 | # We've examined this path. | ||
2923 | 3856 | continue | ||
2924 | 3857 | path_entries = self.state._entries_for_path(path_utf8) | ||
2925 | 3858 | # We need either one or two entries. If the path in | ||
2926 | 3859 | # self.target_index has moved (so the entry in source_index is in | ||
2927 | 3860 | # 'ar') then we need to also look for the entry for this path in | ||
2928 | 3861 | # self.source_index, to output the appropriate delete-or-rename. | ||
2929 | 3862 | selected_entries = [] | ||
2930 | 3863 | found_item = False | ||
2931 | 3864 | for candidate_entry in path_entries: | ||
2932 | 3865 | # Find entries present in target at this path: | ||
2933 | 3866 | if candidate_entry[1][self.target_index][0] not in 'ar': | ||
2934 | 3867 | found_item = True | ||
2935 | 3868 | selected_entries.append(candidate_entry) | ||
2936 | 3869 | # Find entries present in source at this path: | ||
2937 | 3870 | elif (self.source_index is not None and | ||
2938 | 3871 | candidate_entry[1][self.source_index][0] not in 'ar'): | ||
2939 | 3872 | found_item = True | ||
2940 | 3873 | if candidate_entry[1][self.target_index][0] == 'a': | ||
2941 | 3874 | # Deleted, emit it here. | ||
2942 | 3875 | selected_entries.append(candidate_entry) | ||
2943 | 3876 | else: | ||
2944 | 3877 | # renamed, emit it when we process the directory it | ||
2945 | 3878 | # ended up at. | ||
2946 | 3879 | self.search_specific_file_parents.add( | ||
2947 | 3880 | candidate_entry[1][self.target_index][1]) | ||
2948 | 3881 | if not found_item: | ||
2949 | 3882 | raise AssertionError( | ||
2950 | 3883 | "Missing entry for specific path parent %r, %r" % ( | ||
2951 | 3884 | path_utf8, path_entries)) | ||
2952 | 3885 | path_info = self._path_info(path_utf8, path_utf8.decode('utf8')) | ||
2953 | 3886 | for entry in selected_entries: | ||
2954 | 3887 | if entry[0][2] in self.seen_ids: | ||
2955 | 3888 | continue | ||
2956 | 3889 | result, changed = self._process_entry(entry, path_info) | ||
2957 | 3890 | if changed is None: | ||
2958 | 3891 | raise AssertionError( | ||
2959 | 3892 | "Got entry<->path mismatch for specific path " | ||
2960 | 3893 | "%r entry %r path_info %r " % ( | ||
2961 | 3894 | path_utf8, entry, path_info)) | ||
2962 | 3895 | # Only include changes - we're outside the users requested | ||
2963 | 3896 | # expansion. | ||
2964 | 3897 | if changed: | ||
2965 | 3898 | self._gather_result_for_consistency(result) | ||
2966 | 3899 | if (result[6][0] == 'directory' and | ||
2967 | 3900 | result[6][1] != 'directory'): | ||
2968 | 3901 | # This stopped being a directory, the old children have | ||
2969 | 3902 | # to be included. | ||
2970 | 3903 | if entry[1][self.source_index][0] == 'r': | ||
2971 | 3904 | # renamed, take the source path | ||
2972 | 3905 | entry_path_utf8 = entry[1][self.source_index][1] | ||
2973 | 3906 | else: | ||
2974 | 3907 | entry_path_utf8 = path_utf8 | ||
2975 | 3908 | initial_key = (entry_path_utf8, '', '') | ||
2976 | 3909 | block_index, _ = self.state._find_block_index_from_key( | ||
2977 | 3910 | initial_key) | ||
2978 | 3911 | if block_index == 0: | ||
2979 | 3912 | # The children of the root are in block index 1. | ||
2980 | 3913 | block_index +=1 | ||
2981 | 3914 | current_block = None | ||
2982 | 3915 | if block_index < len(self.state._dirblocks): | ||
2983 | 3916 | current_block = self.state._dirblocks[block_index] | ||
2984 | 3917 | if not osutils.is_inside( | ||
2985 | 3918 | entry_path_utf8, current_block[0]): | ||
2986 | 3919 | # No entries for this directory at all. | ||
2987 | 3920 | current_block = None | ||
2988 | 3921 | if current_block is not None: | ||
2989 | 3922 | for entry in current_block[1]: | ||
2990 | 3923 | if entry[1][self.source_index][0] in 'ar': | ||
2991 | 3924 | # Not in the source tree, so doesn't have to be | ||
2992 | 3925 | # included. | ||
2993 | 3926 | continue | ||
2994 | 3927 | # Path of the entry itself. | ||
2995 | 3928 | |||
2996 | 3929 | self.search_specific_file_parents.add( | ||
2997 | 3930 | osutils.pathjoin(*entry[0][:2])) | ||
2998 | 3931 | if changed or self.include_unchanged: | ||
2999 | 3932 | yield result | ||
3000 | 3933 | self.searched_exact_paths.add(path_utf8) | ||
3001 | 3934 | |||
3002 | 3935 | def _path_info(self, utf8_path, unicode_path): | ||
3003 | 3936 | """Generate path_info for unicode_path. | ||
3004 | 3937 | |||
3005 | 3938 | :return: None if unicode_path does not exist, or a path_info tuple. | ||
3006 | 3939 | """ | ||
3007 | 3940 | abspath = self.tree.abspath(unicode_path) | ||
3008 | 3941 | try: | ||
3009 | 3942 | stat = os.lstat(abspath) | ||
3010 | 3943 | except OSError, e: | ||
3011 | 3944 | if e.errno == errno.ENOENT: | ||
3012 | 3945 | # the path does not exist. | ||
3013 | 3946 | return None | ||
3014 | 3947 | else: | ||
3015 | 3948 | raise | ||
3016 | 3949 | utf8_basename = utf8_path.rsplit('/', 1)[-1] | ||
3017 | 3950 | dir_info = (utf8_path, utf8_basename, | ||
3018 | 3951 | osutils.file_kind_from_stat_mode(stat.st_mode), stat, | ||
3019 | 3952 | abspath) | ||
3020 | 3953 | if dir_info[2] == 'directory': | ||
3021 | 3954 | if self.tree._directory_is_tree_reference( | ||
3022 | 3955 | unicode_path): | ||
3023 | 3956 | self.root_dir_info = self.root_dir_info[:2] + \ | ||
3024 | 3957 | ('tree-reference',) + self.root_dir_info[3:] | ||
3025 | 3958 | return dir_info | ||
3026 | 3798 | 3959 | ||
3027 | 3799 | 3960 | ||
3028 | 3800 | # Try to load the compiled form if possible | 3961 | # Try to load the compiled form if possible |
3029 | 3801 | 3962 | ||
3030 | === modified file 'bzrlib/errors.py' | |||
3031 | --- bzrlib/errors.py 2009-07-14 21:07:36 +0000 | |||
3032 | +++ bzrlib/errors.py 2009-08-30 21:34:42 +0000 | |||
3033 | @@ -793,6 +793,12 @@ | |||
3034 | 793 | 793 | ||
3035 | 794 | 794 | ||
3036 | 795 | class IncompatibleRepositories(BzrError): | 795 | class IncompatibleRepositories(BzrError): |
3037 | 796 | """Report an error that two repositories are not compatible. | ||
3038 | 797 | |||
3039 | 798 | Note that the source and target repositories are permitted to be strings: | ||
3040 | 799 | this exception is thrown from the smart server and may refer to a | ||
3041 | 800 | repository the client hasn't opened. | ||
3042 | 801 | """ | ||
3043 | 796 | 802 | ||
3044 | 797 | _fmt = "%(target)s\n" \ | 803 | _fmt = "%(target)s\n" \ |
3045 | 798 | "is not compatible with\n" \ | 804 | "is not compatible with\n" \ |
3046 | @@ -2006,12 +2012,14 @@ | |||
3047 | 2006 | 2012 | ||
3048 | 2007 | class BadConversionTarget(BzrError): | 2013 | class BadConversionTarget(BzrError): |
3049 | 2008 | 2014 | ||
3051 | 2009 | _fmt = "Cannot convert to format %(format)s. %(problem)s" | 2015 | _fmt = "Cannot convert from format %(from_format)s to format %(format)s." \ |
3052 | 2016 | " %(problem)s" | ||
3053 | 2010 | 2017 | ||
3055 | 2011 | def __init__(self, problem, format): | 2018 | def __init__(self, problem, format, from_format=None): |
3056 | 2012 | BzrError.__init__(self) | 2019 | BzrError.__init__(self) |
3057 | 2013 | self.problem = problem | 2020 | self.problem = problem |
3058 | 2014 | self.format = format | 2021 | self.format = format |
3059 | 2022 | self.from_format = from_format or '(unspecified)' | ||
3060 | 2015 | 2023 | ||
3061 | 2016 | 2024 | ||
3062 | 2017 | class NoDiffFound(BzrError): | 2025 | class NoDiffFound(BzrError): |
3063 | @@ -2918,8 +2926,9 @@ | |||
3064 | 2918 | _fmt = 'Cannot bind address "%(host)s:%(port)i": %(orig_error)s.' | 2926 | _fmt = 'Cannot bind address "%(host)s:%(port)i": %(orig_error)s.' |
3065 | 2919 | 2927 | ||
3066 | 2920 | def __init__(self, host, port, orig_error): | 2928 | def __init__(self, host, port, orig_error): |
3067 | 2929 | # nb: in python2.4 socket.error doesn't have a useful repr | ||
3068 | 2921 | BzrError.__init__(self, host=host, port=port, | 2930 | BzrError.__init__(self, host=host, port=port, |
3070 | 2922 | orig_error=orig_error[1]) | 2931 | orig_error=repr(orig_error.args)) |
3071 | 2923 | 2932 | ||
3072 | 2924 | 2933 | ||
3073 | 2925 | class UnknownRules(BzrError): | 2934 | class UnknownRules(BzrError): |
3074 | 2926 | 2935 | ||
3075 | === modified file 'bzrlib/fetch.py' | |||
3076 | --- bzrlib/fetch.py 2009-07-09 08:59:51 +0000 | |||
3077 | +++ bzrlib/fetch.py 2009-08-07 04:29:36 +0000 | |||
3078 | @@ -25,16 +25,21 @@ | |||
3079 | 25 | 25 | ||
3080 | 26 | import operator | 26 | import operator |
3081 | 27 | 27 | ||
3082 | 28 | from bzrlib.lazy_import import lazy_import | ||
3083 | 29 | lazy_import(globals(), """ | ||
3084 | 30 | from bzrlib import ( | ||
3085 | 31 | tsort, | ||
3086 | 32 | versionedfile, | ||
3087 | 33 | ) | ||
3088 | 34 | """) | ||
3089 | 28 | import bzrlib | 35 | import bzrlib |
3090 | 29 | from bzrlib import ( | 36 | from bzrlib import ( |
3091 | 30 | errors, | 37 | errors, |
3092 | 31 | symbol_versioning, | 38 | symbol_versioning, |
3093 | 32 | ) | 39 | ) |
3094 | 33 | from bzrlib.revision import NULL_REVISION | 40 | from bzrlib.revision import NULL_REVISION |
3095 | 34 | from bzrlib.tsort import topo_sort | ||
3096 | 35 | from bzrlib.trace import mutter | 41 | from bzrlib.trace import mutter |
3097 | 36 | import bzrlib.ui | 42 | import bzrlib.ui |
3098 | 37 | from bzrlib.versionedfile import FulltextContentFactory | ||
3099 | 38 | 43 | ||
3100 | 39 | 44 | ||
3101 | 40 | class RepoFetcher(object): | 45 | class RepoFetcher(object): |
3102 | @@ -213,11 +218,9 @@ | |||
3103 | 213 | 218 | ||
3104 | 214 | def _find_root_ids(self, revs, parent_map, graph): | 219 | def _find_root_ids(self, revs, parent_map, graph): |
3105 | 215 | revision_root = {} | 220 | revision_root = {} |
3106 | 216 | planned_versions = {} | ||
3107 | 217 | for tree in self.iter_rev_trees(revs): | 221 | for tree in self.iter_rev_trees(revs): |
3108 | 218 | revision_id = tree.inventory.root.revision | 222 | revision_id = tree.inventory.root.revision |
3109 | 219 | root_id = tree.get_root_id() | 223 | root_id = tree.get_root_id() |
3110 | 220 | planned_versions.setdefault(root_id, []).append(revision_id) | ||
3111 | 221 | revision_root[revision_id] = root_id | 224 | revision_root[revision_id] = root_id |
3112 | 222 | # Find out which parents we don't already know root ids for | 225 | # Find out which parents we don't already know root ids for |
3113 | 223 | parents = set() | 226 | parents = set() |
3114 | @@ -229,7 +232,7 @@ | |||
3115 | 229 | for tree in self.iter_rev_trees(parents): | 232 | for tree in self.iter_rev_trees(parents): |
3116 | 230 | root_id = tree.get_root_id() | 233 | root_id = tree.get_root_id() |
3117 | 231 | revision_root[tree.get_revision_id()] = root_id | 234 | revision_root[tree.get_revision_id()] = root_id |
3119 | 232 | return revision_root, planned_versions | 235 | return revision_root |
3120 | 233 | 236 | ||
3121 | 234 | def generate_root_texts(self, revs): | 237 | def generate_root_texts(self, revs): |
3122 | 235 | """Generate VersionedFiles for all root ids. | 238 | """Generate VersionedFiles for all root ids. |
3123 | @@ -238,9 +241,8 @@ | |||
3124 | 238 | """ | 241 | """ |
3125 | 239 | graph = self.source.get_graph() | 242 | graph = self.source.get_graph() |
3126 | 240 | parent_map = graph.get_parent_map(revs) | 243 | parent_map = graph.get_parent_map(revs) |
3130 | 241 | rev_order = topo_sort(parent_map) | 244 | rev_order = tsort.topo_sort(parent_map) |
3131 | 242 | rev_id_to_root_id, root_id_to_rev_ids = self._find_root_ids( | 245 | rev_id_to_root_id = self._find_root_ids(revs, parent_map, graph) |
3129 | 243 | revs, parent_map, graph) | ||
3132 | 244 | root_id_order = [(rev_id_to_root_id[rev_id], rev_id) for rev_id in | 246 | root_id_order = [(rev_id_to_root_id[rev_id], rev_id) for rev_id in |
3133 | 245 | rev_order] | 247 | rev_order] |
3134 | 246 | # Guaranteed stable, this groups all the file id operations together | 248 | # Guaranteed stable, this groups all the file id operations together |
3135 | @@ -249,20 +251,93 @@ | |||
3136 | 249 | # yet, and are unlikely to in non-rich-root environments anyway. | 251 | # yet, and are unlikely to in non-rich-root environments anyway. |
3137 | 250 | root_id_order.sort(key=operator.itemgetter(0)) | 252 | root_id_order.sort(key=operator.itemgetter(0)) |
3138 | 251 | # Create a record stream containing the roots to create. | 253 | # Create a record stream containing the roots to create. |
3156 | 252 | def yield_roots(): | 254 | from bzrlib.graph import FrozenHeadsCache |
3157 | 253 | for key in root_id_order: | 255 | graph = FrozenHeadsCache(graph) |
3158 | 254 | root_id, rev_id = key | 256 | new_roots_stream = _new_root_data_stream( |
3159 | 255 | rev_parents = parent_map[rev_id] | 257 | root_id_order, rev_id_to_root_id, parent_map, self.source, graph) |
3160 | 256 | # We drop revision parents with different file-ids, because | 258 | return [('texts', new_roots_stream)] |
3161 | 257 | # that represents a rename of the root to a different location | 259 | |
3162 | 258 | # - its not actually a parent for us. (We could look for that | 260 | |
3163 | 259 | # file id in the revision tree at considerably more expense, | 261 | def _new_root_data_stream( |
3164 | 260 | # but for now this is sufficient (and reconcile will catch and | 262 | root_keys_to_create, rev_id_to_root_id_map, parent_map, repo, graph=None): |
3165 | 261 | # correct this anyway). | 263 | """Generate a texts substream of synthesised root entries. |
3166 | 262 | # When a parent revision is a ghost, we guess that its root id | 264 | |
3167 | 263 | # was unchanged (rather than trimming it from the parent list). | 265 | Used in fetches that do rich-root upgrades. |
3168 | 264 | parent_keys = tuple((root_id, parent) for parent in rev_parents | 266 | |
3169 | 265 | if parent != NULL_REVISION and | 267 | :param root_keys_to_create: iterable of (root_id, rev_id) pairs describing |
3170 | 266 | rev_id_to_root_id.get(parent, root_id) == root_id) | 268 | the root entries to create. |
3171 | 267 | yield FulltextContentFactory(key, parent_keys, None, '') | 269 | :param rev_id_to_root_id_map: dict of known rev_id -> root_id mappings for |
3172 | 268 | return [('texts', yield_roots())] | 270 | calculating the parents. If a parent rev_id is not found here then it |
3173 | 271 | will be recalculated. | ||
3174 | 272 | :param parent_map: a parent map for all the revisions in | ||
3175 | 273 | root_keys_to_create. | ||
3176 | 274 | :param graph: a graph to use instead of repo.get_graph(). | ||
3177 | 275 | """ | ||
3178 | 276 | for root_key in root_keys_to_create: | ||
3179 | 277 | root_id, rev_id = root_key | ||
3180 | 278 | parent_keys = _parent_keys_for_root_version( | ||
3181 | 279 | root_id, rev_id, rev_id_to_root_id_map, parent_map, repo, graph) | ||
3182 | 280 | yield versionedfile.FulltextContentFactory( | ||
3183 | 281 | root_key, parent_keys, None, '') | ||
3184 | 282 | |||
3185 | 283 | |||
3186 | 284 | def _parent_keys_for_root_version( | ||
3187 | 285 | root_id, rev_id, rev_id_to_root_id_map, parent_map, repo, graph=None): | ||
3188 | 286 | """Get the parent keys for a given root id. | ||
3189 | 287 | |||
3190 | 288 | A helper function for _new_root_data_stream. | ||
3191 | 289 | """ | ||
3192 | 290 | # Include direct parents of the revision, but only if they used the same | ||
3193 | 291 | # root_id and are heads. | ||
3194 | 292 | rev_parents = parent_map[rev_id] | ||
3195 | 293 | parent_ids = [] | ||
3196 | 294 | for parent_id in rev_parents: | ||
3197 | 295 | if parent_id == NULL_REVISION: | ||
3198 | 296 | continue | ||
3199 | 297 | if parent_id not in rev_id_to_root_id_map: | ||
3200 | 298 | # We probably didn't read this revision, go spend the extra effort | ||
3201 | 299 | # to actually check | ||
3202 | 300 | try: | ||
3203 | 301 | tree = repo.revision_tree(parent_id) | ||
3204 | 302 | except errors.NoSuchRevision: | ||
3205 | 303 | # Ghost, fill out rev_id_to_root_id in case we encounter this | ||
3206 | 304 | # again. | ||
3207 | 305 | # But set parent_root_id to None since we don't really know | ||
3208 | 306 | parent_root_id = None | ||
3209 | 307 | else: | ||
3210 | 308 | parent_root_id = tree.get_root_id() | ||
3211 | 309 | rev_id_to_root_id_map[parent_id] = None | ||
3212 | 310 | # XXX: why not: | ||
3213 | 311 | # rev_id_to_root_id_map[parent_id] = parent_root_id | ||
3214 | 312 | # memory consumption maybe? | ||
3215 | 313 | else: | ||
3216 | 314 | parent_root_id = rev_id_to_root_id_map[parent_id] | ||
3217 | 315 | if root_id == parent_root_id: | ||
3218 | 316 | # With stacking we _might_ want to refer to a non-local revision, | ||
3219 | 317 | # but this code path only applies when we have the full content | ||
3220 | 318 | # available, so ghosts really are ghosts, not just the edge of | ||
3221 | 319 | # local data. | ||
3222 | 320 | parent_ids.append(parent_id) | ||
3223 | 321 | else: | ||
3224 | 322 | # root_id may be in the parent anyway. | ||
3225 | 323 | try: | ||
3226 | 324 | tree = repo.revision_tree(parent_id) | ||
3227 | 325 | except errors.NoSuchRevision: | ||
3228 | 326 | # ghost, can't refer to it. | ||
3229 | 327 | pass | ||
3230 | 328 | else: | ||
3231 | 329 | try: | ||
3232 | 330 | parent_ids.append(tree.inventory[root_id].revision) | ||
3233 | 331 | except errors.NoSuchId: | ||
3234 | 332 | # not in the tree | ||
3235 | 333 | pass | ||
3236 | 334 | # Drop non-head parents | ||
3237 | 335 | if graph is None: | ||
3238 | 336 | graph = repo.get_graph() | ||
3239 | 337 | heads = graph.heads(parent_ids) | ||
3240 | 338 | selected_ids = [] | ||
3241 | 339 | for parent_id in parent_ids: | ||
3242 | 340 | if parent_id in heads and parent_id not in selected_ids: | ||
3243 | 341 | selected_ids.append(parent_id) | ||
3244 | 342 | parent_keys = [(root_id, parent_id) for parent_id in selected_ids] | ||
3245 | 343 | return parent_keys | ||
3246 | 269 | 344 | ||
3247 | === modified file 'bzrlib/graph.py' | |||
3248 | --- bzrlib/graph.py 2009-08-04 04:36:34 +0000 | |||
3249 | +++ bzrlib/graph.py 2009-08-17 18:36:14 +0000 | |||
3250 | @@ -21,7 +21,6 @@ | |||
3251 | 21 | errors, | 21 | errors, |
3252 | 22 | revision, | 22 | revision, |
3253 | 23 | trace, | 23 | trace, |
3254 | 24 | tsort, | ||
3255 | 25 | ) | 24 | ) |
3256 | 26 | from bzrlib.symbol_versioning import deprecated_function, deprecated_in | 25 | from bzrlib.symbol_versioning import deprecated_function, deprecated_in |
3257 | 27 | 26 | ||
3258 | @@ -926,6 +925,7 @@ | |||
3259 | 926 | An ancestor may sort after a descendant if the relationship is not | 925 | An ancestor may sort after a descendant if the relationship is not |
3260 | 927 | visible in the supplied list of revisions. | 926 | visible in the supplied list of revisions. |
3261 | 928 | """ | 927 | """ |
3262 | 928 | from bzrlib import tsort | ||
3263 | 929 | sorter = tsort.TopoSorter(self.get_parent_map(revisions)) | 929 | sorter = tsort.TopoSorter(self.get_parent_map(revisions)) |
3264 | 930 | return sorter.iter_topo_order() | 930 | return sorter.iter_topo_order() |
3265 | 931 | 931 | ||
3266 | 932 | 932 | ||
3267 | === modified file 'bzrlib/groupcompress.py' | |||
3268 | --- bzrlib/groupcompress.py 2009-08-04 04:36:34 +0000 | |||
3269 | +++ bzrlib/groupcompress.py 2009-09-09 13:05:33 +0000 | |||
3270 | @@ -33,7 +33,6 @@ | |||
3271 | 33 | pack, | 33 | pack, |
3272 | 34 | trace, | 34 | trace, |
3273 | 35 | ) | 35 | ) |
3274 | 36 | from bzrlib.graph import Graph | ||
3275 | 37 | from bzrlib.btree_index import BTreeBuilder | 36 | from bzrlib.btree_index import BTreeBuilder |
3276 | 38 | from bzrlib.lru_cache import LRUSizeCache | 37 | from bzrlib.lru_cache import LRUSizeCache |
3277 | 39 | from bzrlib.tsort import topo_sort | 38 | from bzrlib.tsort import topo_sort |
3278 | @@ -45,12 +44,15 @@ | |||
3279 | 45 | VersionedFiles, | 44 | VersionedFiles, |
3280 | 46 | ) | 45 | ) |
3281 | 47 | 46 | ||
3282 | 47 | # Minimum number of uncompressed bytes to try fetch at once when retrieving | ||
3283 | 48 | # groupcompress blocks. | ||
3284 | 49 | BATCH_SIZE = 2**16 | ||
3285 | 50 | |||
3286 | 48 | _USE_LZMA = False and (pylzma is not None) | 51 | _USE_LZMA = False and (pylzma is not None) |
3287 | 49 | 52 | ||
3288 | 50 | # osutils.sha_string('') | 53 | # osutils.sha_string('') |
3289 | 51 | _null_sha1 = 'da39a3ee5e6b4b0d3255bfef95601890afd80709' | 54 | _null_sha1 = 'da39a3ee5e6b4b0d3255bfef95601890afd80709' |
3290 | 52 | 55 | ||
3291 | 53 | |||
3292 | 54 | def sort_gc_optimal(parent_map): | 56 | def sort_gc_optimal(parent_map): |
3293 | 55 | """Sort and group the keys in parent_map into groupcompress order. | 57 | """Sort and group the keys in parent_map into groupcompress order. |
3294 | 56 | 58 | ||
3295 | @@ -62,16 +64,15 @@ | |||
3296 | 62 | # groupcompress ordering is approximately reverse topological, | 64 | # groupcompress ordering is approximately reverse topological, |
3297 | 63 | # properly grouped by file-id. | 65 | # properly grouped by file-id. |
3298 | 64 | per_prefix_map = {} | 66 | per_prefix_map = {} |
3301 | 65 | for item in parent_map.iteritems(): | 67 | for key, value in parent_map.iteritems(): |
3300 | 66 | key = item[0] | ||
3302 | 67 | if isinstance(key, str) or len(key) == 1: | 68 | if isinstance(key, str) or len(key) == 1: |
3303 | 68 | prefix = '' | 69 | prefix = '' |
3304 | 69 | else: | 70 | else: |
3305 | 70 | prefix = key[0] | 71 | prefix = key[0] |
3306 | 71 | try: | 72 | try: |
3308 | 72 | per_prefix_map[prefix].append(item) | 73 | per_prefix_map[prefix][key] = value |
3309 | 73 | except KeyError: | 74 | except KeyError: |
3311 | 74 | per_prefix_map[prefix] = [item] | 75 | per_prefix_map[prefix] = {key: value} |
3312 | 75 | 76 | ||
3313 | 76 | present_keys = [] | 77 | present_keys = [] |
3314 | 77 | for prefix in sorted(per_prefix_map): | 78 | for prefix in sorted(per_prefix_map): |
3315 | @@ -456,7 +457,6 @@ | |||
3316 | 456 | # There are code paths that first extract as fulltext, and then | 457 | # There are code paths that first extract as fulltext, and then |
3317 | 457 | # extract as storage_kind (smart fetch). So we don't break the | 458 | # extract as storage_kind (smart fetch). So we don't break the |
3318 | 458 | # refcycle here, but instead in manager.get_record_stream() | 459 | # refcycle here, but instead in manager.get_record_stream() |
3319 | 459 | # self._manager = None | ||
3320 | 460 | if storage_kind == 'fulltext': | 460 | if storage_kind == 'fulltext': |
3321 | 461 | return self._bytes | 461 | return self._bytes |
3322 | 462 | else: | 462 | else: |
3323 | @@ -468,6 +468,14 @@ | |||
3324 | 468 | class _LazyGroupContentManager(object): | 468 | class _LazyGroupContentManager(object): |
3325 | 469 | """This manages a group of _LazyGroupCompressFactory objects.""" | 469 | """This manages a group of _LazyGroupCompressFactory objects.""" |
3326 | 470 | 470 | ||
3327 | 471 | _max_cut_fraction = 0.75 # We allow a block to be trimmed to 75% of | ||
3328 | 472 | # current size, and still be considered | ||
3329 | 473 | # resuable | ||
3330 | 474 | _full_block_size = 4*1024*1024 | ||
3331 | 475 | _full_mixed_block_size = 2*1024*1024 | ||
3332 | 476 | _full_enough_block_size = 3*1024*1024 # size at which we won't repack | ||
3333 | 477 | _full_enough_mixed_block_size = 2*768*1024 # 1.5MB | ||
3334 | 478 | |||
3335 | 471 | def __init__(self, block): | 479 | def __init__(self, block): |
3336 | 472 | self._block = block | 480 | self._block = block |
3337 | 473 | # We need to preserve the ordering | 481 | # We need to preserve the ordering |
3338 | @@ -545,22 +553,23 @@ | |||
3339 | 545 | # time (self._block._content) is a little expensive. | 553 | # time (self._block._content) is a little expensive. |
3340 | 546 | self._block._ensure_content(self._last_byte) | 554 | self._block._ensure_content(self._last_byte) |
3341 | 547 | 555 | ||
3343 | 548 | def _check_rebuild_block(self): | 556 | def _check_rebuild_action(self): |
3344 | 549 | """Check to see if our block should be repacked.""" | 557 | """Check to see if our block should be repacked.""" |
3345 | 550 | total_bytes_used = 0 | 558 | total_bytes_used = 0 |
3346 | 551 | last_byte_used = 0 | 559 | last_byte_used = 0 |
3347 | 552 | for factory in self._factories: | 560 | for factory in self._factories: |
3348 | 553 | total_bytes_used += factory._end - factory._start | 561 | total_bytes_used += factory._end - factory._start |
3352 | 554 | last_byte_used = max(last_byte_used, factory._end) | 562 | if last_byte_used < factory._end: |
3353 | 555 | # If we are using most of the bytes from the block, we have nothing | 563 | last_byte_used = factory._end |
3354 | 556 | # else to check (currently more that 1/2) | 564 | # If we are using more than half of the bytes from the block, we have |
3355 | 565 | # nothing else to check | ||
3356 | 557 | if total_bytes_used * 2 >= self._block._content_length: | 566 | if total_bytes_used * 2 >= self._block._content_length: |
3360 | 558 | return | 567 | return None, last_byte_used, total_bytes_used |
3361 | 559 | # Can we just strip off the trailing bytes? If we are going to be | 568 | # We are using less than 50% of the content. Is the content we are |
3362 | 560 | # transmitting more than 50% of the front of the content, go ahead | 569 | # using at the beginning of the block? If so, we can just trim the |
3363 | 570 | # tail, rather than rebuilding from scratch. | ||
3364 | 561 | if total_bytes_used * 2 > last_byte_used: | 571 | if total_bytes_used * 2 > last_byte_used: |
3367 | 562 | self._trim_block(last_byte_used) | 572 | return 'trim', last_byte_used, total_bytes_used |
3366 | 563 | return | ||
3368 | 564 | 573 | ||
3369 | 565 | # We are using a small amount of the data, and it isn't just packed | 574 | # We are using a small amount of the data, and it isn't just packed |
3370 | 566 | # nicely at the front, so rebuild the content. | 575 | # nicely at the front, so rebuild the content. |
3371 | @@ -573,7 +582,80 @@ | |||
3372 | 573 | # expanding many deltas into fulltexts, as well. | 582 | # expanding many deltas into fulltexts, as well. |
3373 | 574 | # If we build a cheap enough 'strip', then we could try a strip, | 583 | # If we build a cheap enough 'strip', then we could try a strip, |
3374 | 575 | # if that expands the content, we then rebuild. | 584 | # if that expands the content, we then rebuild. |
3376 | 576 | self._rebuild_block() | 585 | return 'rebuild', last_byte_used, total_bytes_used |
3377 | 586 | |||
3378 | 587 | def check_is_well_utilized(self): | ||
3379 | 588 | """Is the current block considered 'well utilized'? | ||
3380 | 589 | |||
3381 | 590 | This heuristic asks if the current block considers itself to be a fully | ||
3382 | 591 | developed group, rather than just a loose collection of data. | ||
3383 | 592 | """ | ||
3384 | 593 | if len(self._factories) == 1: | ||
3385 | 594 | # A block of length 1 could be improved by combining with other | ||
3386 | 595 | # groups - don't look deeper. Even larger than max size groups | ||
3387 | 596 | # could compress well with adjacent versions of the same thing. | ||
3388 | 597 | return False | ||
3389 | 598 | action, last_byte_used, total_bytes_used = self._check_rebuild_action() | ||
3390 | 599 | block_size = self._block._content_length | ||
3391 | 600 | if total_bytes_used < block_size * self._max_cut_fraction: | ||
3392 | 601 | # This block wants to trim itself small enough that we want to | ||
3393 | 602 | # consider it under-utilized. | ||
3394 | 603 | return False | ||
3395 | 604 | # TODO: This code is meant to be the twin of _insert_record_stream's | ||
3396 | 605 | # 'start_new_block' logic. It would probably be better to factor | ||
3397 | 606 | # out that logic into a shared location, so that it stays | ||
3398 | 607 | # together better | ||
3399 | 608 | # We currently assume a block is properly utilized whenever it is >75% | ||
3400 | 609 | # of the size of a 'full' block. In normal operation, a block is | ||
3401 | 610 | # considered full when it hits 4MB of same-file content. So any block | ||
3402 | 611 | # >3MB is 'full enough'. | ||
3403 | 612 | # The only time this isn't true is when a given block has large-object | ||
3404 | 613 | # content. (a single file >4MB, etc.) | ||
3405 | 614 | # Under these circumstances, we allow a block to grow to | ||
3406 | 615 | # 2 x largest_content. Which means that if a given block had a large | ||
3407 | 616 | # object, it may actually be under-utilized. However, given that this | ||
3408 | 617 | # is 'pack-on-the-fly' it is probably reasonable to not repack large | ||
3409 | 618 | # content blobs on-the-fly. Note that because we return False for all | ||
3410 | 619 | # 1-item blobs, we will repack them; we may wish to reevaluate our | ||
3411 | 620 | # treatment of large object blobs in the future. | ||
3412 | 621 | if block_size >= self._full_enough_block_size: | ||
3413 | 622 | return True | ||
3414 | 623 | # If a block is <3MB, it still may be considered 'full' if it contains | ||
3415 | 624 | # mixed content. The current rule is 2MB of mixed content is considered | ||
3416 | 625 | # full. So check to see if this block contains mixed content, and | ||
3417 | 626 | # set the threshold appropriately. | ||
3418 | 627 | common_prefix = None | ||
3419 | 628 | for factory in self._factories: | ||
3420 | 629 | prefix = factory.key[:-1] | ||
3421 | 630 | if common_prefix is None: | ||
3422 | 631 | common_prefix = prefix | ||
3423 | 632 | elif prefix != common_prefix: | ||
3424 | 633 | # Mixed content, check the size appropriately | ||
3425 | 634 | if block_size >= self._full_enough_mixed_block_size: | ||
3426 | 635 | return True | ||
3427 | 636 | break | ||
3428 | 637 | # The content failed both the mixed check and the single-content check | ||
3429 | 638 | # so obviously it is not fully utilized | ||
3430 | 639 | # TODO: there is one other constraint that isn't being checked | ||
3431 | 640 | # namely, that the entries in the block are in the appropriate | ||
3432 | 641 | # order. For example, you could insert the entries in exactly | ||
3433 | 642 | # reverse groupcompress order, and we would think that is ok. | ||
3434 | 643 | # (all the right objects are in one group, and it is fully | ||
3435 | 644 | # utilized, etc.) For now, we assume that case is rare, | ||
3436 | 645 | # especially since we should always fetch in 'groupcompress' | ||
3437 | 646 | # order. | ||
3438 | 647 | return False | ||
3439 | 648 | |||
3440 | 649 | def _check_rebuild_block(self): | ||
3441 | 650 | action, last_byte_used, total_bytes_used = self._check_rebuild_action() | ||
3442 | 651 | if action is None: | ||
3443 | 652 | return | ||
3444 | 653 | if action == 'trim': | ||
3445 | 654 | self._trim_block(last_byte_used) | ||
3446 | 655 | elif action == 'rebuild': | ||
3447 | 656 | self._rebuild_block() | ||
3448 | 657 | else: | ||
3449 | 658 | raise ValueError('unknown rebuild action: %r' % (action,)) | ||
3450 | 577 | 659 | ||
3451 | 578 | def _wire_bytes(self): | 660 | def _wire_bytes(self): |
3452 | 579 | """Return a byte stream suitable for transmitting over the wire.""" | 661 | """Return a byte stream suitable for transmitting over the wire.""" |
3453 | @@ -975,23 +1057,139 @@ | |||
3454 | 975 | versioned_files.stream.close() | 1057 | versioned_files.stream.close() |
3455 | 976 | 1058 | ||
3456 | 977 | 1059 | ||
3457 | 1060 | class _BatchingBlockFetcher(object): | ||
3458 | 1061 | """Fetch group compress blocks in batches. | ||
3459 | 1062 | |||
3460 | 1063 | :ivar total_bytes: int of expected number of bytes needed to fetch the | ||
3461 | 1064 | currently pending batch. | ||
3462 | 1065 | """ | ||
3463 | 1066 | |||
3464 | 1067 | def __init__(self, gcvf, locations): | ||
3465 | 1068 | self.gcvf = gcvf | ||
3466 | 1069 | self.locations = locations | ||
3467 | 1070 | self.keys = [] | ||
3468 | 1071 | self.batch_memos = {} | ||
3469 | 1072 | self.memos_to_get = [] | ||
3470 | 1073 | self.total_bytes = 0 | ||
3471 | 1074 | self.last_read_memo = None | ||
3472 | 1075 | self.manager = None | ||
3473 | 1076 | |||
3474 | 1077 | def add_key(self, key): | ||
3475 | 1078 | """Add another to key to fetch. | ||
3476 | 1079 | |||
3477 | 1080 | :return: The estimated number of bytes needed to fetch the batch so | ||
3478 | 1081 | far. | ||
3479 | 1082 | """ | ||
3480 | 1083 | self.keys.append(key) | ||
3481 | 1084 | index_memo, _, _, _ = self.locations[key] | ||
3482 | 1085 | read_memo = index_memo[0:3] | ||
3483 | 1086 | # Three possibilities for this read_memo: | ||
3484 | 1087 | # - it's already part of this batch; or | ||
3485 | 1088 | # - it's not yet part of this batch, but is already cached; or | ||
3486 | 1089 | # - it's not yet part of this batch and will need to be fetched. | ||
3487 | 1090 | if read_memo in self.batch_memos: | ||
3488 | 1091 | # This read memo is already in this batch. | ||
3489 | 1092 | return self.total_bytes | ||
3490 | 1093 | try: | ||
3491 | 1094 | cached_block = self.gcvf._group_cache[read_memo] | ||
3492 | 1095 | except KeyError: | ||
3493 | 1096 | # This read memo is new to this batch, and the data isn't cached | ||
3494 | 1097 | # either. | ||
3495 | 1098 | self.batch_memos[read_memo] = None | ||
3496 | 1099 | self.memos_to_get.append(read_memo) | ||
3497 | 1100 | byte_length = read_memo[2] | ||
3498 | 1101 | self.total_bytes += byte_length | ||
3499 | 1102 | else: | ||
3500 | 1103 | # This read memo is new to this batch, but cached. | ||
3501 | 1104 | # Keep a reference to the cached block in batch_memos because it's | ||
3502 | 1105 | # certain that we'll use it when this batch is processed, but | ||
3503 | 1106 | # there's a risk that it would fall out of _group_cache between now | ||
3504 | 1107 | # and then. | ||
3505 | 1108 | self.batch_memos[read_memo] = cached_block | ||
3506 | 1109 | return self.total_bytes | ||
3507 | 1110 | |||
3508 | 1111 | def _flush_manager(self): | ||
3509 | 1112 | if self.manager is not None: | ||
3510 | 1113 | for factory in self.manager.get_record_stream(): | ||
3511 | 1114 | yield factory | ||
3512 | 1115 | self.manager = None | ||
3513 | 1116 | self.last_read_memo = None | ||
3514 | 1117 | |||
3515 | 1118 | def yield_factories(self, full_flush=False): | ||
3516 | 1119 | """Yield factories for keys added since the last yield. They will be | ||
3517 | 1120 | returned in the order they were added via add_key. | ||
3518 | 1121 | |||
3519 | 1122 | :param full_flush: by default, some results may not be returned in case | ||
3520 | 1123 | they can be part of the next batch. If full_flush is True, then | ||
3521 | 1124 | all results are returned. | ||
3522 | 1125 | """ | ||
3523 | 1126 | if self.manager is None and not self.keys: | ||
3524 | 1127 | return | ||
3525 | 1128 | # Fetch all memos in this batch. | ||
3526 | 1129 | blocks = self.gcvf._get_blocks(self.memos_to_get) | ||
3527 | 1130 | # Turn blocks into factories and yield them. | ||
3528 | 1131 | memos_to_get_stack = list(self.memos_to_get) | ||
3529 | 1132 | memos_to_get_stack.reverse() | ||
3530 | 1133 | for key in self.keys: | ||
3531 | 1134 | index_memo, _, parents, _ = self.locations[key] | ||
3532 | 1135 | read_memo = index_memo[:3] | ||
3533 | 1136 | if self.last_read_memo != read_memo: | ||
3534 | 1137 | # We are starting a new block. If we have a | ||
3535 | 1138 | # manager, we have found everything that fits for | ||
3536 | 1139 | # now, so yield records | ||
3537 | 1140 | for factory in self._flush_manager(): | ||
3538 | 1141 | yield factory | ||
3539 | 1142 | # Now start a new manager. | ||
3540 | 1143 | if memos_to_get_stack and memos_to_get_stack[-1] == read_memo: | ||
3541 | 1144 | # The next block from _get_blocks will be the block we | ||
3542 | 1145 | # need. | ||
3543 | 1146 | block_read_memo, block = blocks.next() | ||
3544 | 1147 | if block_read_memo != read_memo: | ||
3545 | 1148 | raise AssertionError( | ||
3546 | 1149 | "block_read_memo out of sync with read_memo" | ||
3547 | 1150 | "(%r != %r)" % (block_read_memo, read_memo)) | ||
3548 | 1151 | self.batch_memos[read_memo] = block | ||
3549 | 1152 | memos_to_get_stack.pop() | ||
3550 | 1153 | else: | ||
3551 | 1154 | block = self.batch_memos[read_memo] | ||
3552 | 1155 | self.manager = _LazyGroupContentManager(block) | ||
3553 | 1156 | self.last_read_memo = read_memo | ||
3554 | 1157 | start, end = index_memo[3:5] | ||
3555 | 1158 | self.manager.add_factory(key, parents, start, end) | ||
3556 | 1159 | if full_flush: | ||
3557 | 1160 | for factory in self._flush_manager(): | ||
3558 | 1161 | yield factory | ||
3559 | 1162 | del self.keys[:] | ||
3560 | 1163 | self.batch_memos.clear() | ||
3561 | 1164 | del self.memos_to_get[:] | ||
3562 | 1165 | self.total_bytes = 0 | ||
3563 | 1166 | |||
3564 | 1167 | |||
3565 | 978 | class GroupCompressVersionedFiles(VersionedFiles): | 1168 | class GroupCompressVersionedFiles(VersionedFiles): |
3566 | 979 | """A group-compress based VersionedFiles implementation.""" | 1169 | """A group-compress based VersionedFiles implementation.""" |
3567 | 980 | 1170 | ||
3569 | 981 | def __init__(self, index, access, delta=True): | 1171 | def __init__(self, index, access, delta=True, _unadded_refs=None): |
3570 | 982 | """Create a GroupCompressVersionedFiles object. | 1172 | """Create a GroupCompressVersionedFiles object. |
3571 | 983 | 1173 | ||
3572 | 984 | :param index: The index object storing access and graph data. | 1174 | :param index: The index object storing access and graph data. |
3573 | 985 | :param access: The access object storing raw data. | 1175 | :param access: The access object storing raw data. |
3574 | 986 | :param delta: Whether to delta compress or just entropy compress. | 1176 | :param delta: Whether to delta compress or just entropy compress. |
3575 | 1177 | :param _unadded_refs: private parameter, don't use. | ||
3576 | 987 | """ | 1178 | """ |
3577 | 988 | self._index = index | 1179 | self._index = index |
3578 | 989 | self._access = access | 1180 | self._access = access |
3579 | 990 | self._delta = delta | 1181 | self._delta = delta |
3581 | 991 | self._unadded_refs = {} | 1182 | if _unadded_refs is None: |
3582 | 1183 | _unadded_refs = {} | ||
3583 | 1184 | self._unadded_refs = _unadded_refs | ||
3584 | 992 | self._group_cache = LRUSizeCache(max_size=50*1024*1024) | 1185 | self._group_cache = LRUSizeCache(max_size=50*1024*1024) |
3585 | 993 | self._fallback_vfs = [] | 1186 | self._fallback_vfs = [] |
3586 | 994 | 1187 | ||
3587 | 1188 | def without_fallbacks(self): | ||
3588 | 1189 | """Return a clone of this object without any fallbacks configured.""" | ||
3589 | 1190 | return GroupCompressVersionedFiles(self._index, self._access, | ||
3590 | 1191 | self._delta, _unadded_refs=dict(self._unadded_refs)) | ||
3591 | 1192 | |||
3592 | 995 | def add_lines(self, key, parents, lines, parent_texts=None, | 1193 | def add_lines(self, key, parents, lines, parent_texts=None, |
3593 | 996 | left_matching_blocks=None, nostore_sha=None, random_id=False, | 1194 | left_matching_blocks=None, nostore_sha=None, random_id=False, |
3594 | 997 | check_content=True): | 1195 | check_content=True): |
3595 | @@ -1099,6 +1297,22 @@ | |||
3596 | 1099 | self._check_lines_not_unicode(lines) | 1297 | self._check_lines_not_unicode(lines) |
3597 | 1100 | self._check_lines_are_lines(lines) | 1298 | self._check_lines_are_lines(lines) |
3598 | 1101 | 1299 | ||
3599 | 1300 | def get_known_graph_ancestry(self, keys): | ||
3600 | 1301 | """Get a KnownGraph instance with the ancestry of keys.""" | ||
3601 | 1302 | # Note that this is identical to | ||
3602 | 1303 | # KnitVersionedFiles.get_known_graph_ancestry, but they don't share | ||
3603 | 1304 | # ancestry. | ||
3604 | 1305 | parent_map, missing_keys = self._index.find_ancestry(keys) | ||
3605 | 1306 | for fallback in self._fallback_vfs: | ||
3606 | 1307 | if not missing_keys: | ||
3607 | 1308 | break | ||
3608 | 1309 | (f_parent_map, f_missing_keys) = fallback._index.find_ancestry( | ||
3609 | 1310 | missing_keys) | ||
3610 | 1311 | parent_map.update(f_parent_map) | ||
3611 | 1312 | missing_keys = f_missing_keys | ||
3612 | 1313 | kg = _mod_graph.KnownGraph(parent_map) | ||
3613 | 1314 | return kg | ||
3614 | 1315 | |||
3615 | 1102 | def get_parent_map(self, keys): | 1316 | def get_parent_map(self, keys): |
3616 | 1103 | """Get a map of the graph parents of keys. | 1317 | """Get a map of the graph parents of keys. |
3617 | 1104 | 1318 | ||
3618 | @@ -1131,26 +1345,42 @@ | |||
3619 | 1131 | missing.difference_update(set(new_result)) | 1345 | missing.difference_update(set(new_result)) |
3620 | 1132 | return result, source_results | 1346 | return result, source_results |
3621 | 1133 | 1347 | ||
3642 | 1134 | def _get_block(self, index_memo): | 1348 | def _get_blocks(self, read_memos): |
3643 | 1135 | read_memo = index_memo[0:3] | 1349 | """Get GroupCompressBlocks for the given read_memos. |
3644 | 1136 | # get the group: | 1350 | |
3645 | 1137 | try: | 1351 | :returns: a series of (read_memo, block) pairs, in the order they were |
3646 | 1138 | block = self._group_cache[read_memo] | 1352 | originally passed. |
3647 | 1139 | except KeyError: | 1353 | """ |
3648 | 1140 | # read the group | 1354 | cached = {} |
3649 | 1141 | zdata = self._access.get_raw_records([read_memo]).next() | 1355 | for read_memo in read_memos: |
3650 | 1142 | # decompress - whole thing - this is not a bug, as it | 1356 | try: |
3651 | 1143 | # permits caching. We might want to store the partially | 1357 | block = self._group_cache[read_memo] |
3652 | 1144 | # decompresed group and decompress object, so that recent | 1358 | except KeyError: |
3653 | 1145 | # texts are not penalised by big groups. | 1359 | pass |
3654 | 1146 | block = GroupCompressBlock.from_bytes(zdata) | 1360 | else: |
3655 | 1147 | self._group_cache[read_memo] = block | 1361 | cached[read_memo] = block |
3656 | 1148 | # cheapo debugging: | 1362 | not_cached = [] |
3657 | 1149 | # print len(zdata), len(plain) | 1363 | not_cached_seen = set() |
3658 | 1150 | # parse - requires split_lines, better to have byte offsets | 1364 | for read_memo in read_memos: |
3659 | 1151 | # here (but not by much - we only split the region for the | 1365 | if read_memo in cached: |
3660 | 1152 | # recipe, and we often want to end up with lines anyway. | 1366 | # Don't fetch what we already have |
3661 | 1153 | return block | 1367 | continue |
3662 | 1368 | if read_memo in not_cached_seen: | ||
3663 | 1369 | # Don't try to fetch the same data twice | ||
3664 | 1370 | continue | ||
3665 | 1371 | not_cached.append(read_memo) | ||
3666 | 1372 | not_cached_seen.add(read_memo) | ||
3667 | 1373 | raw_records = self._access.get_raw_records(not_cached) | ||
3668 | 1374 | for read_memo in read_memos: | ||
3669 | 1375 | try: | ||
3670 | 1376 | yield read_memo, cached[read_memo] | ||
3671 | 1377 | except KeyError: | ||
3672 | 1378 | # Read the block, and cache it. | ||
3673 | 1379 | zdata = raw_records.next() | ||
3674 | 1380 | block = GroupCompressBlock.from_bytes(zdata) | ||
3675 | 1381 | self._group_cache[read_memo] = block | ||
3676 | 1382 | cached[read_memo] = block | ||
3677 | 1383 | yield read_memo, block | ||
3678 | 1154 | 1384 | ||
3679 | 1155 | def get_missing_compression_parent_keys(self): | 1385 | def get_missing_compression_parent_keys(self): |
3680 | 1156 | """Return the keys of missing compression parents. | 1386 | """Return the keys of missing compression parents. |
3681 | @@ -1322,55 +1552,35 @@ | |||
3682 | 1322 | unadded_keys, source_result) | 1552 | unadded_keys, source_result) |
3683 | 1323 | for key in missing: | 1553 | for key in missing: |
3684 | 1324 | yield AbsentContentFactory(key) | 1554 | yield AbsentContentFactory(key) |
3696 | 1325 | manager = None | 1555 | # Batch up as many keys as we can until either: |
3697 | 1326 | last_read_memo = None | 1556 | # - we encounter an unadded ref, or |
3698 | 1327 | # TODO: This works fairly well at batching up existing groups into a | 1557 | # - we run out of keys, or |
3699 | 1328 | # streamable format, and possibly allowing for taking one big | 1558 | # - the total bytes to retrieve for this batch > BATCH_SIZE |
3700 | 1329 | # group and splitting it when it isn't fully utilized. | 1559 | batcher = _BatchingBlockFetcher(self, locations) |
3690 | 1330 | # However, it doesn't allow us to find under-utilized groups and | ||
3691 | 1331 | # combine them into a bigger group on the fly. | ||
3692 | 1332 | # (Consider the issue with how chk_map inserts texts | ||
3693 | 1333 | # one-at-a-time.) This could be done at insert_record_stream() | ||
3694 | 1334 | # time, but it probably would decrease the number of | ||
3695 | 1335 | # bytes-on-the-wire for fetch. | ||
3701 | 1336 | for source, keys in source_keys: | 1560 | for source, keys in source_keys: |
3702 | 1337 | if source is self: | 1561 | if source is self: |
3703 | 1338 | for key in keys: | 1562 | for key in keys: |
3704 | 1339 | if key in self._unadded_refs: | 1563 | if key in self._unadded_refs: |
3709 | 1340 | if manager is not None: | 1564 | # Flush batch, then yield unadded ref from |
3710 | 1341 | for factory in manager.get_record_stream(): | 1565 | # self._compressor. |
3711 | 1342 | yield factory | 1566 | for factory in batcher.yield_factories(full_flush=True): |
3712 | 1343 | last_read_memo = manager = None | 1567 | yield factory |
3713 | 1344 | bytes, sha1 = self._compressor.extract(key) | 1568 | bytes, sha1 = self._compressor.extract(key) |
3714 | 1345 | parents = self._unadded_refs[key] | 1569 | parents = self._unadded_refs[key] |
3715 | 1346 | yield FulltextContentFactory(key, parents, sha1, bytes) | 1570 | yield FulltextContentFactory(key, parents, sha1, bytes) |
3732 | 1347 | else: | 1571 | continue |
3733 | 1348 | index_memo, _, parents, (method, _) = locations[key] | 1572 | if batcher.add_key(key) > BATCH_SIZE: |
3734 | 1349 | read_memo = index_memo[0:3] | 1573 | # Ok, this batch is big enough. Yield some results. |
3735 | 1350 | if last_read_memo != read_memo: | 1574 | for factory in batcher.yield_factories(): |
3736 | 1351 | # We are starting a new block. If we have a | 1575 | yield factory |
3721 | 1352 | # manager, we have found everything that fits for | ||
3722 | 1353 | # now, so yield records | ||
3723 | 1354 | if manager is not None: | ||
3724 | 1355 | for factory in manager.get_record_stream(): | ||
3725 | 1356 | yield factory | ||
3726 | 1357 | # Now start a new manager | ||
3727 | 1358 | block = self._get_block(index_memo) | ||
3728 | 1359 | manager = _LazyGroupContentManager(block) | ||
3729 | 1360 | last_read_memo = read_memo | ||
3730 | 1361 | start, end = index_memo[3:5] | ||
3731 | 1362 | manager.add_factory(key, parents, start, end) | ||
3737 | 1363 | else: | 1576 | else: |
3742 | 1364 | if manager is not None: | 1577 | for factory in batcher.yield_factories(full_flush=True): |
3743 | 1365 | for factory in manager.get_record_stream(): | 1578 | yield factory |
3740 | 1366 | yield factory | ||
3741 | 1367 | last_read_memo = manager = None | ||
3744 | 1368 | for record in source.get_record_stream(keys, ordering, | 1579 | for record in source.get_record_stream(keys, ordering, |
3745 | 1369 | include_delta_closure): | 1580 | include_delta_closure): |
3746 | 1370 | yield record | 1581 | yield record |
3750 | 1371 | if manager is not None: | 1582 | for factory in batcher.yield_factories(full_flush=True): |
3751 | 1372 | for factory in manager.get_record_stream(): | 1583 | yield factory |
3749 | 1373 | yield factory | ||
3752 | 1374 | 1584 | ||
3753 | 1375 | def get_sha1s(self, keys): | 1585 | def get_sha1s(self, keys): |
3754 | 1376 | """See VersionedFiles.get_sha1s().""" | 1586 | """See VersionedFiles.get_sha1s().""" |
3755 | @@ -1449,6 +1659,7 @@ | |||
3756 | 1449 | block_length = None | 1659 | block_length = None |
3757 | 1450 | # XXX: TODO: remove this, it is just for safety checking for now | 1660 | # XXX: TODO: remove this, it is just for safety checking for now |
3758 | 1451 | inserted_keys = set() | 1661 | inserted_keys = set() |
3759 | 1662 | reuse_this_block = reuse_blocks | ||
3760 | 1452 | for record in stream: | 1663 | for record in stream: |
3761 | 1453 | # Raise an error when a record is missing. | 1664 | # Raise an error when a record is missing. |
3762 | 1454 | if record.storage_kind == 'absent': | 1665 | if record.storage_kind == 'absent': |
3763 | @@ -1462,10 +1673,20 @@ | |||
3764 | 1462 | if reuse_blocks: | 1673 | if reuse_blocks: |
3765 | 1463 | # If the reuse_blocks flag is set, check to see if we can just | 1674 | # If the reuse_blocks flag is set, check to see if we can just |
3766 | 1464 | # copy a groupcompress block as-is. | 1675 | # copy a groupcompress block as-is. |
3767 | 1676 | # We only check on the first record (groupcompress-block) not | ||
3768 | 1677 | # on all of the (groupcompress-block-ref) entries. | ||
3769 | 1678 | # The reuse_this_block flag is then kept for as long as | ||
3770 | 1679 | if record.storage_kind == 'groupcompress-block': | ||
3771 | 1680 | # Check to see if we really want to re-use this block | ||
3772 | 1681 | insert_manager = record._manager | ||
3773 | 1682 | reuse_this_block = insert_manager.check_is_well_utilized() | ||
3774 | 1683 | else: | ||
3775 | 1684 | reuse_this_block = False | ||
3776 | 1685 | if reuse_this_block: | ||
3777 | 1686 | # We still want to reuse this block | ||
3778 | 1465 | if record.storage_kind == 'groupcompress-block': | 1687 | if record.storage_kind == 'groupcompress-block': |
3779 | 1466 | # Insert the raw block into the target repo | 1688 | # Insert the raw block into the target repo |
3780 | 1467 | insert_manager = record._manager | 1689 | insert_manager = record._manager |
3781 | 1468 | insert_manager._check_rebuild_block() | ||
3782 | 1469 | bytes = record._manager._block.to_bytes() | 1690 | bytes = record._manager._block.to_bytes() |
3783 | 1470 | _, start, length = self._access.add_raw_records( | 1691 | _, start, length = self._access.add_raw_records( |
3784 | 1471 | [(None, len(bytes))], bytes)[0] | 1692 | [(None, len(bytes))], bytes)[0] |
3785 | @@ -1476,6 +1697,11 @@ | |||
3786 | 1476 | 'groupcompress-block-ref'): | 1697 | 'groupcompress-block-ref'): |
3787 | 1477 | if insert_manager is None: | 1698 | if insert_manager is None: |
3788 | 1478 | raise AssertionError('No insert_manager set') | 1699 | raise AssertionError('No insert_manager set') |
3789 | 1700 | if insert_manager is not record._manager: | ||
3790 | 1701 | raise AssertionError('insert_manager does not match' | ||
3791 | 1702 | ' the current record, we cannot be positive' | ||
3792 | 1703 | ' that the appropriate content was inserted.' | ||
3793 | 1704 | ) | ||
3794 | 1479 | value = "%d %d %d %d" % (block_start, block_length, | 1705 | value = "%d %d %d %d" % (block_start, block_length, |
3795 | 1480 | record._start, record._end) | 1706 | record._start, record._end) |
3796 | 1481 | nodes = [(record.key, value, (record.parents,))] | 1707 | nodes = [(record.key, value, (record.parents,))] |
3797 | @@ -1593,7 +1819,7 @@ | |||
3798 | 1593 | 1819 | ||
3799 | 1594 | def __init__(self, graph_index, is_locked, parents=True, | 1820 | def __init__(self, graph_index, is_locked, parents=True, |
3800 | 1595 | add_callback=None, track_external_parent_refs=False, | 1821 | add_callback=None, track_external_parent_refs=False, |
3802 | 1596 | inconsistency_fatal=True): | 1822 | inconsistency_fatal=True, track_new_keys=False): |
3803 | 1597 | """Construct a _GCGraphIndex on a graph_index. | 1823 | """Construct a _GCGraphIndex on a graph_index. |
3804 | 1598 | 1824 | ||
3805 | 1599 | :param graph_index: An implementation of bzrlib.index.GraphIndex. | 1825 | :param graph_index: An implementation of bzrlib.index.GraphIndex. |
3806 | @@ -1619,7 +1845,8 @@ | |||
3807 | 1619 | self._is_locked = is_locked | 1845 | self._is_locked = is_locked |
3808 | 1620 | self._inconsistency_fatal = inconsistency_fatal | 1846 | self._inconsistency_fatal = inconsistency_fatal |
3809 | 1621 | if track_external_parent_refs: | 1847 | if track_external_parent_refs: |
3811 | 1622 | self._key_dependencies = knit._KeyRefs() | 1848 | self._key_dependencies = knit._KeyRefs( |
3812 | 1849 | track_new_keys=track_new_keys) | ||
3813 | 1623 | else: | 1850 | else: |
3814 | 1624 | self._key_dependencies = None | 1851 | self._key_dependencies = None |
3815 | 1625 | 1852 | ||
3816 | @@ -1679,10 +1906,14 @@ | |||
3817 | 1679 | result.append((key, value)) | 1906 | result.append((key, value)) |
3818 | 1680 | records = result | 1907 | records = result |
3819 | 1681 | key_dependencies = self._key_dependencies | 1908 | key_dependencies = self._key_dependencies |
3824 | 1682 | if key_dependencies is not None and self._parents: | 1909 | if key_dependencies is not None: |
3825 | 1683 | for key, value, refs in records: | 1910 | if self._parents: |
3826 | 1684 | parents = refs[0] | 1911 | for key, value, refs in records: |
3827 | 1685 | key_dependencies.add_references(key, parents) | 1912 | parents = refs[0] |
3828 | 1913 | key_dependencies.add_references(key, parents) | ||
3829 | 1914 | else: | ||
3830 | 1915 | for key, value, refs in records: | ||
3831 | 1916 | new_keys.add_key(key) | ||
3832 | 1686 | self._add_callback(records) | 1917 | self._add_callback(records) |
3833 | 1687 | 1918 | ||
3834 | 1688 | def _check_read(self): | 1919 | def _check_read(self): |
3835 | @@ -1719,6 +1950,10 @@ | |||
3836 | 1719 | if missing_keys: | 1950 | if missing_keys: |
3837 | 1720 | raise errors.RevisionNotPresent(missing_keys.pop(), self) | 1951 | raise errors.RevisionNotPresent(missing_keys.pop(), self) |
3838 | 1721 | 1952 | ||
3839 | 1953 | def find_ancestry(self, keys): | ||
3840 | 1954 | """See CombinedGraphIndex.find_ancestry""" | ||
3841 | 1955 | return self._graph_index.find_ancestry(keys, 0) | ||
3842 | 1956 | |||
3843 | 1722 | def get_parent_map(self, keys): | 1957 | def get_parent_map(self, keys): |
3844 | 1723 | """Get a map of the parents of keys. | 1958 | """Get a map of the parents of keys. |
3845 | 1724 | 1959 | ||
3846 | @@ -1741,7 +1976,7 @@ | |||
3847 | 1741 | """Return the keys of missing parents.""" | 1976 | """Return the keys of missing parents.""" |
3848 | 1742 | # Copied from _KnitGraphIndex.get_missing_parents | 1977 | # Copied from _KnitGraphIndex.get_missing_parents |
3849 | 1743 | # We may have false positives, so filter those out. | 1978 | # We may have false positives, so filter those out. |
3851 | 1744 | self._key_dependencies.add_keys( | 1979 | self._key_dependencies.satisfy_refs_for_keys( |
3852 | 1745 | self.get_parent_map(self._key_dependencies.get_unsatisfied_refs())) | 1980 | self.get_parent_map(self._key_dependencies.get_unsatisfied_refs())) |
3853 | 1746 | return frozenset(self._key_dependencies.get_unsatisfied_refs()) | 1981 | return frozenset(self._key_dependencies.get_unsatisfied_refs()) |
3854 | 1747 | 1982 | ||
3855 | @@ -1801,17 +2036,17 @@ | |||
3856 | 1801 | 2036 | ||
3857 | 1802 | This allows this _GCGraphIndex to keep track of any missing | 2037 | This allows this _GCGraphIndex to keep track of any missing |
3858 | 1803 | compression parents we may want to have filled in to make those | 2038 | compression parents we may want to have filled in to make those |
3860 | 1804 | indices valid. | 2039 | indices valid. It also allows _GCGraphIndex to track any new keys. |
3861 | 1805 | 2040 | ||
3862 | 1806 | :param graph_index: A GraphIndex | 2041 | :param graph_index: A GraphIndex |
3863 | 1807 | """ | 2042 | """ |
3871 | 1808 | if self._key_dependencies is not None: | 2043 | key_dependencies = self._key_dependencies |
3872 | 1809 | # Add parent refs from graph_index (and discard parent refs that | 2044 | if key_dependencies is None: |
3873 | 1810 | # the graph_index has). | 2045 | return |
3874 | 1811 | add_refs = self._key_dependencies.add_references | 2046 | for node in graph_index.iter_all_entries(): |
3875 | 1812 | for node in graph_index.iter_all_entries(): | 2047 | # Add parent refs from graph_index (and discard parent refs |
3876 | 1813 | add_refs(node[1], node[3][0]) | 2048 | # that the graph_index has). |
3877 | 1814 | 2049 | key_dependencies.add_references(node[1], node[3][0]) | |
3878 | 1815 | 2050 | ||
3879 | 1816 | 2051 | ||
3880 | 1817 | from bzrlib._groupcompress_py import ( | 2052 | from bzrlib._groupcompress_py import ( |
3881 | 1818 | 2053 | ||
3882 | === modified file 'bzrlib/help_topics/en/configuration.txt' | |||
3883 | --- bzrlib/help_topics/en/configuration.txt 2009-06-26 18:13:41 +0000 | |||
3884 | +++ bzrlib/help_topics/en/configuration.txt 2009-08-21 09:19:11 +0000 | |||
3885 | @@ -63,6 +63,60 @@ | |||
3886 | 63 | ~~~~~~~~~~~~~~~ | 63 | ~~~~~~~~~~~~~~~ |
3887 | 64 | 64 | ||
3888 | 65 | The path to the plugins directory that Bazaar should use. | 65 | The path to the plugins directory that Bazaar should use. |
3889 | 66 | If not set, Bazaar will search for plugins in: | ||
3890 | 67 | |||
3891 | 68 | * the user specific plugin directory (containing the ``user`` plugins), | ||
3892 | 69 | |||
3893 | 70 | * the bzrlib directory (containing the ``core`` plugins), | ||
3894 | 71 | |||
3895 | 72 | * the site specific plugin directory if applicable (containing | ||
3896 | 73 | the ``site`` plugins). | ||
3897 | 74 | |||
3898 | 75 | If ``BZR_PLUGIN_PATH`` is set in any fashion, it will change the | ||
3899 | 76 | the way the plugin are searched. | ||
3900 | 77 | |||
3901 | 78 | As for the ``PATH`` variables, if multiple directories are | ||
3902 | 79 | specified in ``BZR_PLUGIN_PATH`` they should be separated by the | ||
3903 | 80 | platform specific appropriate character (':' on Unix/Linux/etc, | ||
3904 | 81 | ';' on windows) | ||
3905 | 82 | |||
3906 | 83 | By default if ``BZR_PLUGIN_PATH`` is set, it replaces searching | ||
3907 | 84 | in ``user``. However it will continue to search in ``core`` and | ||
3908 | 85 | ``site`` unless they are explicitly removed. | ||
3909 | 86 | |||
3910 | 87 | If you need to change the order or remove one of these | ||
3911 | 88 | directories, you should use special values: | ||
3912 | 89 | |||
3913 | 90 | * ``-user``, ``-core``, ``-site`` will remove the corresponding | ||
3914 | 91 | path from the default values, | ||
3915 | 92 | |||
3916 | 93 | * ``+user``, ``+core``, ``+site`` will add the corresponding path | ||
3917 | 94 | before the remaining default values (and also remove it from | ||
3918 | 95 | the default values). | ||
3919 | 96 | |||
3920 | 97 | Note that the special values 'user', 'core' and 'site' should be | ||
3921 | 98 | used literally, they will be substituted by the corresponding, | ||
3922 | 99 | platform specific, values. | ||
3923 | 100 | |||
3924 | 101 | Examples: | ||
3925 | 102 | ^^^^^^^^^ | ||
3926 | 103 | |||
3927 | 104 | The examples below uses ':' as the separator, windows users | ||
3928 | 105 | should use ';'. | ||
3929 | 106 | |||
3930 | 107 | Overriding the default user plugin directory: | ||
3931 | 108 | ``BZR_PLUGIN_PATH='/path/to/my/other/plugins'`` | ||
3932 | 109 | |||
3933 | 110 | Disabling the site directory while retaining the user directory: | ||
3934 | 111 | ``BZR_PLUGIN_PATH='-site:+user'`` | ||
3935 | 112 | |||
3936 | 113 | Disabling all plugins (better achieved with --no-plugins): | ||
3937 | 114 | ``BZR_PLUGIN_PATH='-user:-core:-site'`` | ||
3938 | 115 | |||
3939 | 116 | Overriding the default site plugin directory: | ||
3940 | 117 | ``BZR_PLUGIN_PATH='/path/to/my/site/plugins:-site':+user`` | ||
3941 | 118 | |||
3942 | 119 | |||
3943 | 66 | 120 | ||
3944 | 67 | BZRPATH | 121 | BZRPATH |
3945 | 68 | ~~~~~~~ | 122 | ~~~~~~~ |
3946 | 69 | 123 | ||
3947 | === modified file 'bzrlib/help_topics/en/debug-flags.txt' | |||
3948 | --- bzrlib/help_topics/en/debug-flags.txt 2009-08-04 04:36:34 +0000 | |||
3949 | +++ bzrlib/help_topics/en/debug-flags.txt 2009-08-20 05:02:45 +0000 | |||
3950 | @@ -12,18 +12,26 @@ | |||
3951 | 12 | operations. | 12 | operations. |
3952 | 13 | -Dfetch Trace history copying between repositories. | 13 | -Dfetch Trace history copying between repositories. |
3953 | 14 | -Dfilters Emit information for debugging content filtering. | 14 | -Dfilters Emit information for debugging content filtering. |
3954 | 15 | -Dforceinvdeltas Force use of inventory deltas during generic streaming fetch. | ||
3955 | 15 | -Dgraph Trace graph traversal. | 16 | -Dgraph Trace graph traversal. |
3956 | 16 | -Dhashcache Log every time a working file is read to determine its hash. | 17 | -Dhashcache Log every time a working file is read to determine its hash. |
3957 | 17 | -Dhooks Trace hook execution. | 18 | -Dhooks Trace hook execution. |
3958 | 18 | -Dhpss Trace smart protocol requests and responses. | 19 | -Dhpss Trace smart protocol requests and responses. |
3959 | 19 | -Dhpssdetail More hpss details. | 20 | -Dhpssdetail More hpss details. |
3960 | 20 | -Dhpssvfs Traceback on vfs access to Remote objects. | 21 | -Dhpssvfs Traceback on vfs access to Remote objects. |
3962 | 21 | -Dhttp Trace http connections, requests and responses | 22 | -Dhttp Trace http connections, requests and responses. |
3963 | 22 | -Dindex Trace major index operations. | 23 | -Dindex Trace major index operations. |
3964 | 23 | -Dknit Trace knit operations. | 24 | -Dknit Trace knit operations. |
3965 | 25 | -Dstrict_locks Trace when OS locks are potentially used in a non-portable | ||
3966 | 26 | manner. | ||
3967 | 24 | -Dlock Trace when lockdir locks are taken or released. | 27 | -Dlock Trace when lockdir locks are taken or released. |
3968 | 25 | -Dprogress Trace progress bar operations. | 28 | -Dprogress Trace progress bar operations. |
3969 | 26 | -Dmerge Emit information for debugging merges. | 29 | -Dmerge Emit information for debugging merges. |
3970 | 30 | -Dno_apport Don't use apport to report crashes. | ||
3971 | 27 | -Dunlock Some errors during unlock are treated as warnings. | 31 | -Dunlock Some errors during unlock are treated as warnings. |
3972 | 28 | -Dpack Emit information about pack operations. | 32 | -Dpack Emit information about pack operations. |
3973 | 29 | -Dsftp Trace SFTP internals. | 33 | -Dsftp Trace SFTP internals. |
3974 | 34 | -Dstream Trace fetch streams. | ||
3975 | 35 | -DIDS_never Never use InterDifferingSerializer when fetching. | ||
3976 | 36 | -DIDS_always Always use InterDifferingSerializer to fetch if appropriate | ||
3977 | 37 | for the format, even for non-local fetches. | ||
3978 | 30 | 38 | ||
3979 | === modified file 'bzrlib/hooks.py' | |||
3980 | --- bzrlib/hooks.py 2009-06-10 03:31:01 +0000 | |||
3981 | +++ bzrlib/hooks.py 2009-09-01 12:29:54 +0000 | |||
3982 | @@ -219,9 +219,7 @@ | |||
3983 | 219 | strings.append('Introduced in: %s' % introduced_string) | 219 | strings.append('Introduced in: %s' % introduced_string) |
3984 | 220 | if self.deprecated: | 220 | if self.deprecated: |
3985 | 221 | deprecated_string = _format_version_tuple(self.deprecated) | 221 | deprecated_string = _format_version_tuple(self.deprecated) |
3989 | 222 | else: | 222 | strings.append('Deprecated in: %s' % deprecated_string) |
3987 | 223 | deprecated_string = 'Not deprecated' | ||
3988 | 224 | strings.append('Deprecated in: %s' % deprecated_string) | ||
3990 | 225 | strings.append('') | 223 | strings.append('') |
3991 | 226 | strings.extend(textwrap.wrap(self.__doc__, | 224 | strings.extend(textwrap.wrap(self.__doc__, |
3992 | 227 | break_long_words=False)) | 225 | break_long_words=False)) |
3993 | 228 | 226 | ||
3994 | === modified file 'bzrlib/index.py' | |||
3995 | --- bzrlib/index.py 2009-07-01 10:53:08 +0000 | |||
3996 | +++ bzrlib/index.py 2009-08-17 22:11:06 +0000 | |||
3997 | @@ -333,6 +333,22 @@ | |||
3998 | 333 | if combine_backing_indices is not None: | 333 | if combine_backing_indices is not None: |
3999 | 334 | self._combine_backing_indices = combine_backing_indices | 334 | self._combine_backing_indices = combine_backing_indices |
4000 | 335 | 335 | ||
4001 | 336 | def find_ancestry(self, keys, ref_list_num): | ||
4002 | 337 | """See CombinedGraphIndex.find_ancestry()""" | ||
4003 | 338 | pending = set(keys) | ||
4004 | 339 | parent_map = {} | ||
4005 | 340 | missing_keys = set() | ||
4006 | 341 | while pending: | ||
4007 | 342 | next_pending = set() | ||
4008 | 343 | for _, key, value, ref_lists in self.iter_entries(pending): | ||
4009 | 344 | parent_keys = ref_lists[ref_list_num] | ||
4010 | 345 | parent_map[key] = parent_keys | ||
4011 | 346 | next_pending.update([p for p in parent_keys if p not in | ||
4012 | 347 | parent_map]) | ||
4013 | 348 | missing_keys.update(pending.difference(parent_map)) | ||
4014 | 349 | pending = next_pending | ||
4015 | 350 | return parent_map, missing_keys | ||
4016 | 351 | |||
4017 | 336 | 352 | ||
4018 | 337 | class GraphIndex(object): | 353 | class GraphIndex(object): |
4019 | 338 | """An index for data with embedded graphs. | 354 | """An index for data with embedded graphs. |
4020 | @@ -702,6 +718,23 @@ | |||
4021 | 702 | # the last thing looked up was a terminal element | 718 | # the last thing looked up was a terminal element |
4022 | 703 | yield (self, ) + key_dict | 719 | yield (self, ) + key_dict |
4023 | 704 | 720 | ||
4024 | 721 | def _find_ancestors(self, keys, ref_list_num, parent_map, missing_keys): | ||
4025 | 722 | """See BTreeIndex._find_ancestors.""" | ||
4026 | 723 | # The api can be implemented as a trivial overlay on top of | ||
4027 | 724 | # iter_entries, it is not an efficient implementation, but it at least | ||
4028 | 725 | # gets the job done. | ||
4029 | 726 | found_keys = set() | ||
4030 | 727 | search_keys = set() | ||
4031 | 728 | for index, key, value, refs in self.iter_entries(keys): | ||
4032 | 729 | parent_keys = refs[ref_list_num] | ||
4033 | 730 | found_keys.add(key) | ||
4034 | 731 | parent_map[key] = parent_keys | ||
4035 | 732 | search_keys.update(parent_keys) | ||
4036 | 733 | # Figure out what, if anything, was missing | ||
4037 | 734 | missing_keys.update(set(keys).difference(found_keys)) | ||
4038 | 735 | search_keys = search_keys.difference(parent_map) | ||
4039 | 736 | return search_keys | ||
4040 | 737 | |||
4041 | 705 | def key_count(self): | 738 | def key_count(self): |
4042 | 706 | """Return an estimate of the number of keys in this index. | 739 | """Return an estimate of the number of keys in this index. |
4043 | 707 | 740 | ||
4044 | @@ -1297,6 +1330,69 @@ | |||
4045 | 1297 | except errors.NoSuchFile: | 1330 | except errors.NoSuchFile: |
4046 | 1298 | self._reload_or_raise() | 1331 | self._reload_or_raise() |
4047 | 1299 | 1332 | ||
4048 | 1333 | def find_ancestry(self, keys, ref_list_num): | ||
4049 | 1334 | """Find the complete ancestry for the given set of keys. | ||
4050 | 1335 | |||
4051 | 1336 | Note that this is a whole-ancestry request, so it should be used | ||
4052 | 1337 | sparingly. | ||
4053 | 1338 | |||
4054 | 1339 | :param keys: An iterable of keys to look for | ||
4055 | 1340 | :param ref_list_num: The reference list which references the parents | ||
4056 | 1341 | we care about. | ||
4057 | 1342 | :return: (parent_map, missing_keys) | ||
4058 | 1343 | """ | ||
4059 | 1344 | missing_keys = set() | ||
4060 | 1345 | parent_map = {} | ||
4061 | 1346 | keys_to_lookup = set(keys) | ||
4062 | 1347 | generation = 0 | ||
4063 | 1348 | while keys_to_lookup: | ||
4064 | 1349 | # keys that *all* indexes claim are missing, stop searching them | ||
4065 | 1350 | generation += 1 | ||
4066 | 1351 | all_index_missing = None | ||
4067 | 1352 | # print 'gen\tidx\tsub\tn_keys\tn_pmap\tn_miss' | ||
4068 | 1353 | # print '%4d\t\t\t%4d\t%5d\t%5d' % (generation, len(keys_to_lookup), | ||
4069 | 1354 | # len(parent_map), | ||
4070 | 1355 | # len(missing_keys)) | ||
4071 | 1356 | for index_idx, index in enumerate(self._indices): | ||
4072 | 1357 | # TODO: we should probably be doing something with | ||
4073 | 1358 | # 'missing_keys' since we've already determined that | ||
4074 | 1359 | # those revisions have not been found anywhere | ||
4075 | 1360 | index_missing_keys = set() | ||
4076 | 1361 | # Find all of the ancestry we can from this index | ||
4077 | 1362 | # keep looking until the search_keys set is empty, which means | ||
4078 | 1363 | # things we didn't find should be in index_missing_keys | ||
4079 | 1364 | search_keys = keys_to_lookup | ||
4080 | 1365 | sub_generation = 0 | ||
4081 | 1366 | # print ' \t%2d\t\t%4d\t%5d\t%5d' % ( | ||
4082 | 1367 | # index_idx, len(search_keys), | ||
4083 | 1368 | # len(parent_map), len(index_missing_keys)) | ||
4084 | 1369 | while search_keys: | ||
4085 | 1370 | sub_generation += 1 | ||
4086 | 1371 | # TODO: ref_list_num should really be a parameter, since | ||
4087 | 1372 | # CombinedGraphIndex does not know what the ref lists | ||
4088 | 1373 | # mean. | ||
4089 | 1374 | search_keys = index._find_ancestors(search_keys, | ||
4090 | 1375 | ref_list_num, parent_map, index_missing_keys) | ||
4091 | 1376 | # print ' \t \t%2d\t%4d\t%5d\t%5d' % ( | ||
4092 | 1377 | # sub_generation, len(search_keys), | ||
4093 | 1378 | # len(parent_map), len(index_missing_keys)) | ||
4094 | 1379 | # Now set whatever was missing to be searched in the next index | ||
4095 | 1380 | keys_to_lookup = index_missing_keys | ||
4096 | 1381 | if all_index_missing is None: | ||
4097 | 1382 | all_index_missing = set(index_missing_keys) | ||
4098 | 1383 | else: | ||
4099 | 1384 | all_index_missing.intersection_update(index_missing_keys) | ||
4100 | 1385 | if not keys_to_lookup: | ||
4101 | 1386 | break | ||
4102 | 1387 | if all_index_missing is None: | ||
4103 | 1388 | # There were no indexes, so all search keys are 'missing' | ||
4104 | 1389 | missing_keys.update(keys_to_lookup) | ||
4105 | 1390 | keys_to_lookup = None | ||
4106 | 1391 | else: | ||
4107 | 1392 | missing_keys.update(all_index_missing) | ||
4108 | 1393 | keys_to_lookup.difference_update(all_index_missing) | ||
4109 | 1394 | return parent_map, missing_keys | ||
4110 | 1395 | |||
4111 | 1300 | def key_count(self): | 1396 | def key_count(self): |
4112 | 1301 | """Return an estimate of the number of keys in this index. | 1397 | """Return an estimate of the number of keys in this index. |
4113 | 1302 | 1398 | ||
4114 | 1303 | 1399 | ||
4115 | === modified file 'bzrlib/inventory.py' | |||
4116 | --- bzrlib/inventory.py 2009-08-05 02:30:59 +0000 | |||
4117 | +++ bzrlib/inventory.py 2009-08-30 23:51:10 +0000 | |||
4118 | @@ -437,7 +437,13 @@ | |||
4119 | 437 | self.text_id is not None): | 437 | self.text_id is not None): |
4120 | 438 | checker._report_items.append('directory {%s} has text in revision {%s}' | 438 | checker._report_items.append('directory {%s} has text in revision {%s}' |
4121 | 439 | % (self.file_id, rev_id)) | 439 | % (self.file_id, rev_id)) |
4123 | 440 | # Directories are stored as ''. | 440 | # In non rich root repositories we do not expect a file graph for the |
4124 | 441 | # root. | ||
4125 | 442 | if self.name == '' and not checker.rich_roots: | ||
4126 | 443 | return | ||
4127 | 444 | # Directories are stored as an empty file, but the file should exist | ||
4128 | 445 | # to provide a per-fileid log. The hash of every directory content is | ||
4129 | 446 | # "da..." below (the sha1sum of ''). | ||
4130 | 441 | checker.add_pending_item(rev_id, | 447 | checker.add_pending_item(rev_id, |
4131 | 442 | ('texts', self.file_id, self.revision), 'text', | 448 | ('texts', self.file_id, self.revision), 'text', |
4132 | 443 | 'da39a3ee5e6b4b0d3255bfef95601890afd80709') | 449 | 'da39a3ee5e6b4b0d3255bfef95601890afd80709') |
4133 | @@ -743,6 +749,9 @@ | |||
4134 | 743 | """ | 749 | """ |
4135 | 744 | return self.has_id(file_id) | 750 | return self.has_id(file_id) |
4136 | 745 | 751 | ||
4137 | 752 | def has_filename(self, filename): | ||
4138 | 753 | return bool(self.path2id(filename)) | ||
4139 | 754 | |||
4140 | 746 | def id2path(self, file_id): | 755 | def id2path(self, file_id): |
4141 | 747 | """Return as a string the path to file_id. | 756 | """Return as a string the path to file_id. |
4142 | 748 | 757 | ||
4143 | @@ -751,6 +760,8 @@ | |||
4144 | 751 | >>> e = i.add(InventoryFile('foo-id', 'foo.c', parent_id='src-id')) | 760 | >>> e = i.add(InventoryFile('foo-id', 'foo.c', parent_id='src-id')) |
4145 | 752 | >>> print i.id2path('foo-id') | 761 | >>> print i.id2path('foo-id') |
4146 | 753 | src/foo.c | 762 | src/foo.c |
4147 | 763 | |||
4148 | 764 | :raises NoSuchId: If file_id is not present in the inventory. | ||
4149 | 754 | """ | 765 | """ |
4150 | 755 | # get all names, skipping root | 766 | # get all names, skipping root |
4151 | 756 | return '/'.join(reversed( | 767 | return '/'.join(reversed( |
4152 | @@ -1363,9 +1374,6 @@ | |||
4153 | 1363 | yield ie | 1374 | yield ie |
4154 | 1364 | file_id = ie.parent_id | 1375 | file_id = ie.parent_id |
4155 | 1365 | 1376 | ||
4156 | 1366 | def has_filename(self, filename): | ||
4157 | 1367 | return bool(self.path2id(filename)) | ||
4158 | 1368 | |||
4159 | 1369 | def has_id(self, file_id): | 1377 | def has_id(self, file_id): |
4160 | 1370 | return (file_id in self._byid) | 1378 | return (file_id in self._byid) |
4161 | 1371 | 1379 | ||
4162 | 1372 | 1380 | ||
4163 | === modified file 'bzrlib/inventory_delta.py' | |||
4164 | --- bzrlib/inventory_delta.py 2009-04-02 05:53:12 +0000 | |||
4165 | +++ bzrlib/inventory_delta.py 2009-08-13 00:20:29 +0000 | |||
4166 | @@ -29,6 +29,25 @@ | |||
4167 | 29 | from bzrlib import inventory | 29 | from bzrlib import inventory |
4168 | 30 | from bzrlib.revision import NULL_REVISION | 30 | from bzrlib.revision import NULL_REVISION |
4169 | 31 | 31 | ||
4170 | 32 | FORMAT_1 = 'bzr inventory delta v1 (bzr 1.14)' | ||
4171 | 33 | |||
4172 | 34 | |||
4173 | 35 | class InventoryDeltaError(errors.BzrError): | ||
4174 | 36 | """An error when serializing or deserializing an inventory delta.""" | ||
4175 | 37 | |||
4176 | 38 | # Most errors when serializing and deserializing are due to bugs, although | ||
4177 | 39 | # damaged input (i.e. a bug in a different process) could cause | ||
4178 | 40 | # deserialization errors too. | ||
4179 | 41 | internal_error = True | ||
4180 | 42 | |||
4181 | 43 | |||
4182 | 44 | class IncompatibleInventoryDelta(errors.BzrError): | ||
4183 | 45 | """The delta could not be deserialised because its contents conflict with | ||
4184 | 46 | the allow_versioned_root or allow_tree_references flags of the | ||
4185 | 47 | deserializer. | ||
4186 | 48 | """ | ||
4187 | 49 | internal_error = False | ||
4188 | 50 | |||
4189 | 32 | 51 | ||
4190 | 33 | def _directory_content(entry): | 52 | def _directory_content(entry): |
4191 | 34 | """Serialize the content component of entry which is a directory. | 53 | """Serialize the content component of entry which is a directory. |
4192 | @@ -49,7 +68,7 @@ | |||
4193 | 49 | exec_bytes = '' | 68 | exec_bytes = '' |
4194 | 50 | size_exec_sha = (entry.text_size, exec_bytes, entry.text_sha1) | 69 | size_exec_sha = (entry.text_size, exec_bytes, entry.text_sha1) |
4195 | 51 | if None in size_exec_sha: | 70 | if None in size_exec_sha: |
4197 | 52 | raise errors.BzrError('Missing size or sha for %s' % entry.file_id) | 71 | raise InventoryDeltaError('Missing size or sha for %s' % entry.file_id) |
4198 | 53 | return "file\x00%d\x00%s\x00%s" % size_exec_sha | 72 | return "file\x00%d\x00%s\x00%s" % size_exec_sha |
4199 | 54 | 73 | ||
4200 | 55 | 74 | ||
4201 | @@ -60,7 +79,7 @@ | |||
4202 | 60 | """ | 79 | """ |
4203 | 61 | target = entry.symlink_target | 80 | target = entry.symlink_target |
4204 | 62 | if target is None: | 81 | if target is None: |
4206 | 63 | raise errors.BzrError('Missing target for %s' % entry.file_id) | 82 | raise InventoryDeltaError('Missing target for %s' % entry.file_id) |
4207 | 64 | return "link\x00%s" % target.encode('utf8') | 83 | return "link\x00%s" % target.encode('utf8') |
4208 | 65 | 84 | ||
4209 | 66 | 85 | ||
4210 | @@ -71,7 +90,8 @@ | |||
4211 | 71 | """ | 90 | """ |
4212 | 72 | tree_revision = entry.reference_revision | 91 | tree_revision = entry.reference_revision |
4213 | 73 | if tree_revision is None: | 92 | if tree_revision is None: |
4215 | 74 | raise errors.BzrError('Missing reference revision for %s' % entry.file_id) | 93 | raise InventoryDeltaError( |
4216 | 94 | 'Missing reference revision for %s' % entry.file_id) | ||
4217 | 75 | return "tree\x00%s" % tree_revision | 95 | return "tree\x00%s" % tree_revision |
4218 | 76 | 96 | ||
4219 | 77 | 97 | ||
4220 | @@ -115,11 +135,8 @@ | |||
4221 | 115 | return result | 135 | return result |
4222 | 116 | 136 | ||
4223 | 117 | 137 | ||
4224 | 118 | |||
4225 | 119 | class InventoryDeltaSerializer(object): | 138 | class InventoryDeltaSerializer(object): |
4229 | 120 | """Serialize and deserialize inventory deltas.""" | 139 | """Serialize inventory deltas.""" |
4227 | 121 | |||
4228 | 122 | FORMAT_1 = 'bzr inventory delta v1 (bzr 1.14)' | ||
4230 | 123 | 140 | ||
4231 | 124 | def __init__(self, versioned_root, tree_references): | 141 | def __init__(self, versioned_root, tree_references): |
4232 | 125 | """Create an InventoryDeltaSerializer. | 142 | """Create an InventoryDeltaSerializer. |
4233 | @@ -141,6 +158,9 @@ | |||
4234 | 141 | def delta_to_lines(self, old_name, new_name, delta_to_new): | 158 | def delta_to_lines(self, old_name, new_name, delta_to_new): |
4235 | 142 | """Return a line sequence for delta_to_new. | 159 | """Return a line sequence for delta_to_new. |
4236 | 143 | 160 | ||
4237 | 161 | Both the versioned_root and tree_references flags must be set via | ||
4238 | 162 | require_flags before calling this. | ||
4239 | 163 | |||
4240 | 144 | :param old_name: A UTF8 revision id for the old inventory. May be | 164 | :param old_name: A UTF8 revision id for the old inventory. May be |
4241 | 145 | NULL_REVISION if there is no older inventory and delta_to_new | 165 | NULL_REVISION if there is no older inventory and delta_to_new |
4242 | 146 | includes the entire inventory contents. | 166 | includes the entire inventory contents. |
4243 | @@ -150,15 +170,20 @@ | |||
4244 | 150 | takes. | 170 | takes. |
4245 | 151 | :return: The serialized delta as lines. | 171 | :return: The serialized delta as lines. |
4246 | 152 | """ | 172 | """ |
4247 | 173 | if type(old_name) is not str: | ||
4248 | 174 | raise TypeError('old_name should be str, got %r' % (old_name,)) | ||
4249 | 175 | if type(new_name) is not str: | ||
4250 | 176 | raise TypeError('new_name should be str, got %r' % (new_name,)) | ||
4251 | 153 | lines = ['', '', '', '', ''] | 177 | lines = ['', '', '', '', ''] |
4252 | 154 | to_line = self._delta_item_to_line | 178 | to_line = self._delta_item_to_line |
4253 | 155 | for delta_item in delta_to_new: | 179 | for delta_item in delta_to_new: |
4257 | 156 | lines.append(to_line(delta_item)) | 180 | line = to_line(delta_item, new_name) |
4258 | 157 | if lines[-1].__class__ != str: | 181 | if line.__class__ != str: |
4259 | 158 | raise errors.BzrError( | 182 | raise InventoryDeltaError( |
4260 | 159 | 'to_line generated non-str output %r' % lines[-1]) | 183 | 'to_line generated non-str output %r' % lines[-1]) |
4261 | 184 | lines.append(line) | ||
4262 | 160 | lines.sort() | 185 | lines.sort() |
4264 | 161 | lines[0] = "format: %s\n" % InventoryDeltaSerializer.FORMAT_1 | 186 | lines[0] = "format: %s\n" % FORMAT_1 |
4265 | 162 | lines[1] = "parent: %s\n" % old_name | 187 | lines[1] = "parent: %s\n" % old_name |
4266 | 163 | lines[2] = "version: %s\n" % new_name | 188 | lines[2] = "version: %s\n" % new_name |
4267 | 164 | lines[3] = "versioned_root: %s\n" % self._serialize_bool( | 189 | lines[3] = "versioned_root: %s\n" % self._serialize_bool( |
4268 | @@ -173,7 +198,7 @@ | |||
4269 | 173 | else: | 198 | else: |
4270 | 174 | return "false" | 199 | return "false" |
4271 | 175 | 200 | ||
4273 | 176 | def _delta_item_to_line(self, delta_item): | 201 | def _delta_item_to_line(self, delta_item, new_version): |
4274 | 177 | """Convert delta_item to a line.""" | 202 | """Convert delta_item to a line.""" |
4275 | 178 | oldpath, newpath, file_id, entry = delta_item | 203 | oldpath, newpath, file_id, entry = delta_item |
4276 | 179 | if newpath is None: | 204 | if newpath is None: |
4277 | @@ -188,6 +213,10 @@ | |||
4278 | 188 | oldpath_utf8 = 'None' | 213 | oldpath_utf8 = 'None' |
4279 | 189 | else: | 214 | else: |
4280 | 190 | oldpath_utf8 = '/' + oldpath.encode('utf8') | 215 | oldpath_utf8 = '/' + oldpath.encode('utf8') |
4281 | 216 | if newpath == '/': | ||
4282 | 217 | raise AssertionError( | ||
4283 | 218 | "Bad inventory delta: '/' is not a valid newpath " | ||
4284 | 219 | "(should be '') in delta item %r" % (delta_item,)) | ||
4285 | 191 | # TODO: Test real-world utf8 cache hit rate. It may be a win. | 220 | # TODO: Test real-world utf8 cache hit rate. It may be a win. |
4286 | 192 | newpath_utf8 = '/' + newpath.encode('utf8') | 221 | newpath_utf8 = '/' + newpath.encode('utf8') |
4287 | 193 | # Serialize None as '' | 222 | # Serialize None as '' |
4288 | @@ -196,58 +225,78 @@ | |||
4289 | 196 | last_modified = entry.revision | 225 | last_modified = entry.revision |
4290 | 197 | # special cases for / | 226 | # special cases for / |
4291 | 198 | if newpath_utf8 == '/' and not self._versioned_root: | 227 | if newpath_utf8 == '/' and not self._versioned_root: |
4299 | 199 | if file_id != 'TREE_ROOT': | 228 | # This is an entry for the root, this inventory does not |
4300 | 200 | raise errors.BzrError( | 229 | # support versioned roots. So this must be an unversioned |
4301 | 201 | 'file_id %s is not TREE_ROOT for /' % file_id) | 230 | # root, i.e. last_modified == new revision. Otherwise, this |
4302 | 202 | if last_modified is not None: | 231 | # delta is invalid. |
4303 | 203 | raise errors.BzrError( | 232 | # Note: the non-rich-root repositories *can* have roots with |
4304 | 204 | 'Version present for / in %s' % file_id) | 233 | # file-ids other than TREE_ROOT, e.g. repo formats that use the |
4305 | 205 | last_modified = NULL_REVISION | 234 | # xml5 serializer. |
4306 | 235 | if last_modified != new_version: | ||
4307 | 236 | raise InventoryDeltaError( | ||
4308 | 237 | 'Version present for / in %s (%s != %s)' | ||
4309 | 238 | % (file_id, last_modified, new_version)) | ||
4310 | 206 | if last_modified is None: | 239 | if last_modified is None: |
4312 | 207 | raise errors.BzrError("no version for fileid %s" % file_id) | 240 | raise InventoryDeltaError("no version for fileid %s" % file_id) |
4313 | 208 | content = self._entry_to_content[entry.kind](entry) | 241 | content = self._entry_to_content[entry.kind](entry) |
4314 | 209 | return ("%s\x00%s\x00%s\x00%s\x00%s\x00%s\n" % | 242 | return ("%s\x00%s\x00%s\x00%s\x00%s\x00%s\n" % |
4315 | 210 | (oldpath_utf8, newpath_utf8, file_id, parent_id, last_modified, | 243 | (oldpath_utf8, newpath_utf8, file_id, parent_id, last_modified, |
4316 | 211 | content)) | 244 | content)) |
4317 | 212 | 245 | ||
4318 | 246 | |||
4319 | 247 | class InventoryDeltaDeserializer(object): | ||
4320 | 248 | """Deserialize inventory deltas.""" | ||
4321 | 249 | |||
4322 | 250 | def __init__(self, allow_versioned_root=True, allow_tree_references=True): | ||
4323 | 251 | """Create an InventoryDeltaDeserializer. | ||
4324 | 252 | |||
4325 | 253 | :param versioned_root: If True, any root entry that is seen is expected | ||
4326 | 254 | to be versioned, and root entries can have any fileid. | ||
4327 | 255 | :param tree_references: If True support tree-reference entries. | ||
4328 | 256 | """ | ||
4329 | 257 | self._allow_versioned_root = allow_versioned_root | ||
4330 | 258 | self._allow_tree_references = allow_tree_references | ||
4331 | 259 | |||
4332 | 213 | def _deserialize_bool(self, value): | 260 | def _deserialize_bool(self, value): |
4333 | 214 | if value == "true": | 261 | if value == "true": |
4334 | 215 | return True | 262 | return True |
4335 | 216 | elif value == "false": | 263 | elif value == "false": |
4336 | 217 | return False | 264 | return False |
4337 | 218 | else: | 265 | else: |
4339 | 219 | raise errors.BzrError("value %r is not a bool" % (value,)) | 266 | raise InventoryDeltaError("value %r is not a bool" % (value,)) |
4340 | 220 | 267 | ||
4341 | 221 | def parse_text_bytes(self, bytes): | 268 | def parse_text_bytes(self, bytes): |
4342 | 222 | """Parse the text bytes of a serialized inventory delta. | 269 | """Parse the text bytes of a serialized inventory delta. |
4343 | 223 | 270 | ||
4344 | 271 | If versioned_root and/or tree_references flags were set via | ||
4345 | 272 | require_flags, then the parsed flags must match or a BzrError will be | ||
4346 | 273 | raised. | ||
4347 | 274 | |||
4348 | 224 | :param bytes: The bytes to parse. This can be obtained by calling | 275 | :param bytes: The bytes to parse. This can be obtained by calling |
4349 | 225 | delta_to_lines and then doing ''.join(delta_lines). | 276 | delta_to_lines and then doing ''.join(delta_lines). |
4351 | 226 | :return: (parent_id, new_id, inventory_delta) | 277 | :return: (parent_id, new_id, versioned_root, tree_references, |
4352 | 278 | inventory_delta) | ||
4353 | 227 | """ | 279 | """ |
4354 | 280 | if bytes[-1:] != '\n': | ||
4355 | 281 | last_line = bytes.rsplit('\n', 1)[-1] | ||
4356 | 282 | raise InventoryDeltaError('last line not empty: %r' % (last_line,)) | ||
4357 | 228 | lines = bytes.split('\n')[:-1] # discard the last empty line | 283 | lines = bytes.split('\n')[:-1] # discard the last empty line |
4360 | 229 | if not lines or lines[0] != 'format: %s' % InventoryDeltaSerializer.FORMAT_1: | 284 | if not lines or lines[0] != 'format: %s' % FORMAT_1: |
4361 | 230 | raise errors.BzrError('unknown format %r' % lines[0:1]) | 285 | raise InventoryDeltaError('unknown format %r' % lines[0:1]) |
4362 | 231 | if len(lines) < 2 or not lines[1].startswith('parent: '): | 286 | if len(lines) < 2 or not lines[1].startswith('parent: '): |
4364 | 232 | raise errors.BzrError('missing parent: marker') | 287 | raise InventoryDeltaError('missing parent: marker') |
4365 | 233 | delta_parent_id = lines[1][8:] | 288 | delta_parent_id = lines[1][8:] |
4366 | 234 | if len(lines) < 3 or not lines[2].startswith('version: '): | 289 | if len(lines) < 3 or not lines[2].startswith('version: '): |
4368 | 235 | raise errors.BzrError('missing version: marker') | 290 | raise InventoryDeltaError('missing version: marker') |
4369 | 236 | delta_version_id = lines[2][9:] | 291 | delta_version_id = lines[2][9:] |
4370 | 237 | if len(lines) < 4 or not lines[3].startswith('versioned_root: '): | 292 | if len(lines) < 4 or not lines[3].startswith('versioned_root: '): |
4372 | 238 | raise errors.BzrError('missing versioned_root: marker') | 293 | raise InventoryDeltaError('missing versioned_root: marker') |
4373 | 239 | delta_versioned_root = self._deserialize_bool(lines[3][16:]) | 294 | delta_versioned_root = self._deserialize_bool(lines[3][16:]) |
4374 | 240 | if len(lines) < 5 or not lines[4].startswith('tree_references: '): | 295 | if len(lines) < 5 or not lines[4].startswith('tree_references: '): |
4376 | 241 | raise errors.BzrError('missing tree_references: marker') | 296 | raise InventoryDeltaError('missing tree_references: marker') |
4377 | 242 | delta_tree_references = self._deserialize_bool(lines[4][17:]) | 297 | delta_tree_references = self._deserialize_bool(lines[4][17:]) |
4386 | 243 | if delta_versioned_root != self._versioned_root: | 298 | if (not self._allow_versioned_root and delta_versioned_root): |
4387 | 244 | raise errors.BzrError( | 299 | raise IncompatibleInventoryDelta("versioned_root not allowed") |
4380 | 245 | "serialized versioned_root flag is wrong: %s" % | ||
4381 | 246 | (delta_versioned_root,)) | ||
4382 | 247 | if delta_tree_references != self._tree_references: | ||
4383 | 248 | raise errors.BzrError( | ||
4384 | 249 | "serialized tree_references flag is wrong: %s" % | ||
4385 | 250 | (delta_tree_references,)) | ||
4388 | 251 | result = [] | 300 | result = [] |
4389 | 252 | seen_ids = set() | 301 | seen_ids = set() |
4390 | 253 | line_iter = iter(lines) | 302 | line_iter = iter(lines) |
4391 | @@ -258,33 +307,58 @@ | |||
4392 | 258 | content) = line.split('\x00', 5) | 307 | content) = line.split('\x00', 5) |
4393 | 259 | parent_id = parent_id or None | 308 | parent_id = parent_id or None |
4394 | 260 | if file_id in seen_ids: | 309 | if file_id in seen_ids: |
4396 | 261 | raise errors.BzrError( | 310 | raise InventoryDeltaError( |
4397 | 262 | "duplicate file id in inventory delta %r" % lines) | 311 | "duplicate file id in inventory delta %r" % lines) |
4398 | 263 | seen_ids.add(file_id) | 312 | seen_ids.add(file_id) |
4409 | 264 | if newpath_utf8 == '/' and not delta_versioned_root and ( | 313 | if (newpath_utf8 == '/' and not delta_versioned_root and |
4410 | 265 | last_modified != 'null:' or file_id != 'TREE_ROOT'): | 314 | last_modified != delta_version_id): |
4411 | 266 | raise errors.BzrError("Versioned root found: %r" % line) | 315 | # Delta claims to be not have a versioned root, yet here's |
4412 | 267 | elif last_modified[-1] == ':': | 316 | # a root entry with a non-default version. |
4413 | 268 | raise errors.BzrError('special revisionid found: %r' % line) | 317 | raise InventoryDeltaError("Versioned root found: %r" % line) |
4414 | 269 | if not delta_tree_references and content.startswith('tree\x00'): | 318 | elif newpath_utf8 != 'None' and last_modified[-1] == ':': |
4415 | 270 | raise errors.BzrError("Tree reference found: %r" % line) | 319 | # Deletes have a last_modified of null:, but otherwise special |
4416 | 271 | content_tuple = tuple(content.split('\x00')) | 320 | # revision ids should not occur. |
4417 | 272 | entry = _parse_entry( | 321 | raise InventoryDeltaError('special revisionid found: %r' % line) |
4418 | 273 | newpath_utf8, file_id, parent_id, last_modified, content_tuple) | 322 | if content.startswith('tree\x00'): |
4419 | 323 | if delta_tree_references is False: | ||
4420 | 324 | raise InventoryDeltaError( | ||
4421 | 325 | "Tree reference found (but header said " | ||
4422 | 326 | "tree_references: false): %r" % line) | ||
4423 | 327 | elif not self._allow_tree_references: | ||
4424 | 328 | raise IncompatibleInventoryDelta( | ||
4425 | 329 | "Tree reference not allowed") | ||
4426 | 274 | if oldpath_utf8 == 'None': | 330 | if oldpath_utf8 == 'None': |
4427 | 275 | oldpath = None | 331 | oldpath = None |
4428 | 332 | elif oldpath_utf8[:1] != '/': | ||
4429 | 333 | raise InventoryDeltaError( | ||
4430 | 334 | "oldpath invalid (does not start with /): %r" | ||
4431 | 335 | % (oldpath_utf8,)) | ||
4432 | 276 | else: | 336 | else: |
4433 | 337 | oldpath_utf8 = oldpath_utf8[1:] | ||
4434 | 277 | oldpath = oldpath_utf8.decode('utf8') | 338 | oldpath = oldpath_utf8.decode('utf8') |
4435 | 278 | if newpath_utf8 == 'None': | 339 | if newpath_utf8 == 'None': |
4436 | 279 | newpath = None | 340 | newpath = None |
4437 | 341 | elif newpath_utf8[:1] != '/': | ||
4438 | 342 | raise InventoryDeltaError( | ||
4439 | 343 | "newpath invalid (does not start with /): %r" | ||
4440 | 344 | % (newpath_utf8,)) | ||
4441 | 280 | else: | 345 | else: |
4442 | 346 | # Trim leading slash | ||
4443 | 347 | newpath_utf8 = newpath_utf8[1:] | ||
4444 | 281 | newpath = newpath_utf8.decode('utf8') | 348 | newpath = newpath_utf8.decode('utf8') |
4445 | 349 | content_tuple = tuple(content.split('\x00')) | ||
4446 | 350 | if content_tuple[0] == 'deleted': | ||
4447 | 351 | entry = None | ||
4448 | 352 | else: | ||
4449 | 353 | entry = _parse_entry( | ||
4450 | 354 | newpath, file_id, parent_id, last_modified, content_tuple) | ||
4451 | 282 | delta_item = (oldpath, newpath, file_id, entry) | 355 | delta_item = (oldpath, newpath, file_id, entry) |
4452 | 283 | result.append(delta_item) | 356 | result.append(delta_item) |
4457 | 284 | return delta_parent_id, delta_version_id, result | 357 | return (delta_parent_id, delta_version_id, delta_versioned_root, |
4458 | 285 | 358 | delta_tree_references, result) | |
4459 | 286 | 359 | ||
4460 | 287 | def _parse_entry(utf8_path, file_id, parent_id, last_modified, content): | 360 | |
4461 | 361 | def _parse_entry(path, file_id, parent_id, last_modified, content): | ||
4462 | 288 | entry_factory = { | 362 | entry_factory = { |
4463 | 289 | 'dir': _dir_to_entry, | 363 | 'dir': _dir_to_entry, |
4464 | 290 | 'file': _file_to_entry, | 364 | 'file': _file_to_entry, |
4465 | @@ -292,8 +366,10 @@ | |||
4466 | 292 | 'tree': _tree_to_entry, | 366 | 'tree': _tree_to_entry, |
4467 | 293 | } | 367 | } |
4468 | 294 | kind = content[0] | 368 | kind = content[0] |
4470 | 295 | path = utf8_path[1:].decode('utf8') | 369 | if path.startswith('/'): |
4471 | 370 | raise AssertionError | ||
4472 | 296 | name = basename(path) | 371 | name = basename(path) |
4473 | 297 | return entry_factory[content[0]]( | 372 | return entry_factory[content[0]]( |
4474 | 298 | content, name, parent_id, file_id, last_modified) | 373 | content, name, parent_id, file_id, last_modified) |
4475 | 299 | 374 | ||
4476 | 375 | |||
4477 | 300 | 376 | ||
4478 | === modified file 'bzrlib/knit.py' | |||
4479 | --- bzrlib/knit.py 2009-08-04 04:36:34 +0000 | |||
4480 | +++ bzrlib/knit.py 2009-09-09 13:05:33 +0000 | |||
4481 | @@ -1190,6 +1190,19 @@ | |||
4482 | 1190 | generator = _VFContentMapGenerator(self, [key]) | 1190 | generator = _VFContentMapGenerator(self, [key]) |
4483 | 1191 | return generator._get_content(key) | 1191 | return generator._get_content(key) |
4484 | 1192 | 1192 | ||
4485 | 1193 | def get_known_graph_ancestry(self, keys): | ||
4486 | 1194 | """Get a KnownGraph instance with the ancestry of keys.""" | ||
4487 | 1195 | parent_map, missing_keys = self._index.find_ancestry(keys) | ||
4488 | 1196 | for fallback in self._fallback_vfs: | ||
4489 | 1197 | if not missing_keys: | ||
4490 | 1198 | break | ||
4491 | 1199 | (f_parent_map, f_missing_keys) = fallback._index.find_ancestry( | ||
4492 | 1200 | missing_keys) | ||
4493 | 1201 | parent_map.update(f_parent_map) | ||
4494 | 1202 | missing_keys = f_missing_keys | ||
4495 | 1203 | kg = _mod_graph.KnownGraph(parent_map) | ||
4496 | 1204 | return kg | ||
4497 | 1205 | |||
4498 | 1193 | def get_parent_map(self, keys): | 1206 | def get_parent_map(self, keys): |
4499 | 1194 | """Get a map of the graph parents of keys. | 1207 | """Get a map of the graph parents of keys. |
4500 | 1195 | 1208 | ||
4501 | @@ -1505,10 +1518,10 @@ | |||
4502 | 1505 | if source is parent_maps[0]: | 1518 | if source is parent_maps[0]: |
4503 | 1506 | # this KnitVersionedFiles | 1519 | # this KnitVersionedFiles |
4504 | 1507 | records = [(key, positions[key][1]) for key in keys] | 1520 | records = [(key, positions[key][1]) for key in keys] |
4506 | 1508 | for key, raw_data, sha1 in self._read_records_iter_raw(records): | 1521 | for key, raw_data in self._read_records_iter_unchecked(records): |
4507 | 1509 | (record_details, index_memo, _) = positions[key] | 1522 | (record_details, index_memo, _) = positions[key] |
4508 | 1510 | yield KnitContentFactory(key, global_map[key], | 1523 | yield KnitContentFactory(key, global_map[key], |
4510 | 1511 | record_details, sha1, raw_data, self._factory.annotated, None) | 1524 | record_details, None, raw_data, self._factory.annotated, None) |
4511 | 1512 | else: | 1525 | else: |
4512 | 1513 | vf = self._fallback_vfs[parent_maps.index(source) - 1] | 1526 | vf = self._fallback_vfs[parent_maps.index(source) - 1] |
4513 | 1514 | for record in vf.get_record_stream(keys, ordering, | 1527 | for record in vf.get_record_stream(keys, ordering, |
4514 | @@ -1583,6 +1596,13 @@ | |||
4515 | 1583 | # key = basis_parent, value = index entry to add | 1596 | # key = basis_parent, value = index entry to add |
4516 | 1584 | buffered_index_entries = {} | 1597 | buffered_index_entries = {} |
4517 | 1585 | for record in stream: | 1598 | for record in stream: |
4518 | 1599 | kind = record.storage_kind | ||
4519 | 1600 | if kind.startswith('knit-') and kind.endswith('-gz'): | ||
4520 | 1601 | # Check that the ID in the header of the raw knit bytes matches | ||
4521 | 1602 | # the record metadata. | ||
4522 | 1603 | raw_data = record._raw_record | ||
4523 | 1604 | df, rec = self._parse_record_header(record.key, raw_data) | ||
4524 | 1605 | df.close() | ||
4525 | 1586 | buffered = False | 1606 | buffered = False |
4526 | 1587 | parents = record.parents | 1607 | parents = record.parents |
4527 | 1588 | if record.storage_kind in delta_types: | 1608 | if record.storage_kind in delta_types: |
4528 | @@ -2560,6 +2580,33 @@ | |||
4529 | 2560 | except KeyError: | 2580 | except KeyError: |
4530 | 2561 | raise RevisionNotPresent(key, self) | 2581 | raise RevisionNotPresent(key, self) |
4531 | 2562 | 2582 | ||
4532 | 2583 | def find_ancestry(self, keys): | ||
4533 | 2584 | """See CombinedGraphIndex.find_ancestry()""" | ||
4534 | 2585 | prefixes = set(key[:-1] for key in keys) | ||
4535 | 2586 | self._load_prefixes(prefixes) | ||
4536 | 2587 | result = {} | ||
4537 | 2588 | parent_map = {} | ||
4538 | 2589 | missing_keys = set() | ||
4539 | 2590 | pending_keys = list(keys) | ||
4540 | 2591 | # This assumes that keys will not reference parents in a different | ||
4541 | 2592 | # prefix, which is accurate so far. | ||
4542 | 2593 | while pending_keys: | ||
4543 | 2594 | key = pending_keys.pop() | ||
4544 | 2595 | if key in parent_map: | ||
4545 | 2596 | continue | ||
4546 | 2597 | prefix = key[:-1] | ||
4547 | 2598 | try: | ||
4548 | 2599 | suffix_parents = self._kndx_cache[prefix][0][key[-1]][4] | ||
4549 | 2600 | except KeyError: | ||
4550 | 2601 | missing_keys.add(key) | ||
4551 | 2602 | else: | ||
4552 | 2603 | parent_keys = tuple([prefix + (suffix,) | ||
4553 | 2604 | for suffix in suffix_parents]) | ||
4554 | 2605 | parent_map[key] = parent_keys | ||
4555 | 2606 | pending_keys.extend([p for p in parent_keys | ||
4556 | 2607 | if p not in parent_map]) | ||
4557 | 2608 | return parent_map, missing_keys | ||
4558 | 2609 | |||
4559 | 2563 | def get_parent_map(self, keys): | 2610 | def get_parent_map(self, keys): |
4560 | 2564 | """Get a map of the parents of keys. | 2611 | """Get a map of the parents of keys. |
4561 | 2565 | 2612 | ||
4562 | @@ -2737,9 +2784,20 @@ | |||
4563 | 2737 | 2784 | ||
4564 | 2738 | class _KeyRefs(object): | 2785 | class _KeyRefs(object): |
4565 | 2739 | 2786 | ||
4567 | 2740 | def __init__(self): | 2787 | def __init__(self, track_new_keys=False): |
4568 | 2741 | # dict mapping 'key' to 'set of keys referring to that key' | 2788 | # dict mapping 'key' to 'set of keys referring to that key' |
4569 | 2742 | self.refs = {} | 2789 | self.refs = {} |
4570 | 2790 | if track_new_keys: | ||
4571 | 2791 | # set remembering all new keys | ||
4572 | 2792 | self.new_keys = set() | ||
4573 | 2793 | else: | ||
4574 | 2794 | self.new_keys = None | ||
4575 | 2795 | |||
4576 | 2796 | def clear(self): | ||
4577 | 2797 | if self.refs: | ||
4578 | 2798 | self.refs.clear() | ||
4579 | 2799 | if self.new_keys: | ||
4580 | 2800 | self.new_keys.clear() | ||
4581 | 2743 | 2801 | ||
4582 | 2744 | def add_references(self, key, refs): | 2802 | def add_references(self, key, refs): |
4583 | 2745 | # Record the new references | 2803 | # Record the new references |
4584 | @@ -2752,19 +2810,28 @@ | |||
4585 | 2752 | # Discard references satisfied by the new key | 2810 | # Discard references satisfied by the new key |
4586 | 2753 | self.add_key(key) | 2811 | self.add_key(key) |
4587 | 2754 | 2812 | ||
4588 | 2813 | def get_new_keys(self): | ||
4589 | 2814 | return self.new_keys | ||
4590 | 2815 | |||
4591 | 2755 | def get_unsatisfied_refs(self): | 2816 | def get_unsatisfied_refs(self): |
4592 | 2756 | return self.refs.iterkeys() | 2817 | return self.refs.iterkeys() |
4593 | 2757 | 2818 | ||
4595 | 2758 | def add_key(self, key): | 2819 | def _satisfy_refs_for_key(self, key): |
4596 | 2759 | try: | 2820 | try: |
4597 | 2760 | del self.refs[key] | 2821 | del self.refs[key] |
4598 | 2761 | except KeyError: | 2822 | except KeyError: |
4599 | 2762 | # No keys depended on this key. That's ok. | 2823 | # No keys depended on this key. That's ok. |
4600 | 2763 | pass | 2824 | pass |
4601 | 2764 | 2825 | ||
4603 | 2765 | def add_keys(self, keys): | 2826 | def add_key(self, key): |
4604 | 2827 | # satisfy refs for key, and remember that we've seen this key. | ||
4605 | 2828 | self._satisfy_refs_for_key(key) | ||
4606 | 2829 | if self.new_keys is not None: | ||
4607 | 2830 | self.new_keys.add(key) | ||
4608 | 2831 | |||
4609 | 2832 | def satisfy_refs_for_keys(self, keys): | ||
4610 | 2766 | for key in keys: | 2833 | for key in keys: |
4612 | 2767 | self.add_key(key) | 2834 | self._satisfy_refs_for_key(key) |
4613 | 2768 | 2835 | ||
4614 | 2769 | def get_referrers(self): | 2836 | def get_referrers(self): |
4615 | 2770 | result = set() | 2837 | result = set() |
4616 | @@ -2932,7 +2999,7 @@ | |||
4617 | 2932 | # If updating this, you should also update | 2999 | # If updating this, you should also update |
4618 | 2933 | # groupcompress._GCGraphIndex.get_missing_parents | 3000 | # groupcompress._GCGraphIndex.get_missing_parents |
4619 | 2934 | # We may have false positives, so filter those out. | 3001 | # We may have false positives, so filter those out. |
4621 | 2935 | self._key_dependencies.add_keys( | 3002 | self._key_dependencies.satisfy_refs_for_keys( |
4622 | 2936 | self.get_parent_map(self._key_dependencies.get_unsatisfied_refs())) | 3003 | self.get_parent_map(self._key_dependencies.get_unsatisfied_refs())) |
4623 | 2937 | return frozenset(self._key_dependencies.get_unsatisfied_refs()) | 3004 | return frozenset(self._key_dependencies.get_unsatisfied_refs()) |
4624 | 2938 | 3005 | ||
4625 | @@ -3049,6 +3116,10 @@ | |||
4626 | 3049 | options.append('no-eol') | 3116 | options.append('no-eol') |
4627 | 3050 | return options | 3117 | return options |
4628 | 3051 | 3118 | ||
4629 | 3119 | def find_ancestry(self, keys): | ||
4630 | 3120 | """See CombinedGraphIndex.find_ancestry()""" | ||
4631 | 3121 | return self._graph_index.find_ancestry(keys, 0) | ||
4632 | 3122 | |||
4633 | 3052 | def get_parent_map(self, keys): | 3123 | def get_parent_map(self, keys): |
4634 | 3053 | """Get a map of the parents of keys. | 3124 | """Get a map of the parents of keys. |
4635 | 3054 | 3125 | ||
4636 | 3055 | 3126 | ||
4637 | === modified file 'bzrlib/lock.py' | |||
4638 | --- bzrlib/lock.py 2009-07-20 08:47:58 +0000 | |||
4639 | +++ bzrlib/lock.py 2009-07-31 16:51:48 +0000 | |||
4640 | @@ -190,6 +190,13 @@ | |||
4641 | 190 | if self.filename in _fcntl_WriteLock._open_locks: | 190 | if self.filename in _fcntl_WriteLock._open_locks: |
4642 | 191 | self._clear_f() | 191 | self._clear_f() |
4643 | 192 | raise errors.LockContention(self.filename) | 192 | raise errors.LockContention(self.filename) |
4644 | 193 | if self.filename in _fcntl_ReadLock._open_locks: | ||
4645 | 194 | if 'strict_locks' in debug.debug_flags: | ||
4646 | 195 | self._clear_f() | ||
4647 | 196 | raise errors.LockContention(self.filename) | ||
4648 | 197 | else: | ||
4649 | 198 | trace.mutter('Write lock taken w/ an open read lock on: %s' | ||
4650 | 199 | % (self.filename,)) | ||
4651 | 193 | 200 | ||
4652 | 194 | self._open(self.filename, 'rb+') | 201 | self._open(self.filename, 'rb+') |
4653 | 195 | # reserve a slot for this lock - even if the lockf call fails, | 202 | # reserve a slot for this lock - even if the lockf call fails, |
4654 | @@ -220,6 +227,14 @@ | |||
4655 | 220 | def __init__(self, filename): | 227 | def __init__(self, filename): |
4656 | 221 | super(_fcntl_ReadLock, self).__init__() | 228 | super(_fcntl_ReadLock, self).__init__() |
4657 | 222 | self.filename = osutils.realpath(filename) | 229 | self.filename = osutils.realpath(filename) |
4658 | 230 | if self.filename in _fcntl_WriteLock._open_locks: | ||
4659 | 231 | if 'strict_locks' in debug.debug_flags: | ||
4660 | 232 | # We raise before calling _open so we don't need to | ||
4661 | 233 | # _clear_f | ||
4662 | 234 | raise errors.LockContention(self.filename) | ||
4663 | 235 | else: | ||
4664 | 236 | trace.mutter('Read lock taken w/ an open write lock on: %s' | ||
4665 | 237 | % (self.filename,)) | ||
4666 | 223 | _fcntl_ReadLock._open_locks.setdefault(self.filename, 0) | 238 | _fcntl_ReadLock._open_locks.setdefault(self.filename, 0) |
4667 | 224 | _fcntl_ReadLock._open_locks[self.filename] += 1 | 239 | _fcntl_ReadLock._open_locks[self.filename] += 1 |
4668 | 225 | self._open(filename, 'rb') | 240 | self._open(filename, 'rb') |
4669 | @@ -418,15 +433,15 @@ | |||
4670 | 418 | DWORD, # dwFlagsAndAttributes | 433 | DWORD, # dwFlagsAndAttributes |
4671 | 419 | HANDLE # hTemplateFile | 434 | HANDLE # hTemplateFile |
4672 | 420 | )((_function_name, ctypes.windll.kernel32)) | 435 | )((_function_name, ctypes.windll.kernel32)) |
4674 | 421 | 436 | ||
4675 | 422 | INVALID_HANDLE_VALUE = -1 | 437 | INVALID_HANDLE_VALUE = -1 |
4677 | 423 | 438 | ||
4678 | 424 | GENERIC_READ = 0x80000000 | 439 | GENERIC_READ = 0x80000000 |
4679 | 425 | GENERIC_WRITE = 0x40000000 | 440 | GENERIC_WRITE = 0x40000000 |
4680 | 426 | FILE_SHARE_READ = 1 | 441 | FILE_SHARE_READ = 1 |
4681 | 427 | OPEN_ALWAYS = 4 | 442 | OPEN_ALWAYS = 4 |
4682 | 428 | FILE_ATTRIBUTE_NORMAL = 128 | 443 | FILE_ATTRIBUTE_NORMAL = 128 |
4684 | 429 | 444 | ||
4685 | 430 | ERROR_ACCESS_DENIED = 5 | 445 | ERROR_ACCESS_DENIED = 5 |
4686 | 431 | ERROR_SHARING_VIOLATION = 32 | 446 | ERROR_SHARING_VIOLATION = 32 |
4687 | 432 | 447 | ||
4688 | 433 | 448 | ||
4689 | === modified file 'bzrlib/lsprof.py' | |||
4690 | --- bzrlib/lsprof.py 2009-03-08 06:18:06 +0000 | |||
4691 | +++ bzrlib/lsprof.py 2009-08-24 21:05:09 +0000 | |||
4692 | @@ -13,45 +13,74 @@ | |||
4693 | 13 | 13 | ||
4694 | 14 | __all__ = ['profile', 'Stats'] | 14 | __all__ = ['profile', 'Stats'] |
4695 | 15 | 15 | ||
4696 | 16 | _g_threadmap = {} | ||
4697 | 17 | |||
4698 | 18 | |||
4699 | 19 | def _thread_profile(f, *args, **kwds): | ||
4700 | 20 | # we lose the first profile point for a new thread in order to trampoline | ||
4701 | 21 | # a new Profile object into place | ||
4702 | 22 | global _g_threadmap | ||
4703 | 23 | thr = thread.get_ident() | ||
4704 | 24 | _g_threadmap[thr] = p = Profiler() | ||
4705 | 25 | # this overrides our sys.setprofile hook: | ||
4706 | 26 | p.enable(subcalls=True, builtins=True) | ||
4707 | 27 | |||
4708 | 28 | |||
4709 | 29 | def profile(f, *args, **kwds): | 16 | def profile(f, *args, **kwds): |
4710 | 30 | """Run a function profile. | 17 | """Run a function profile. |
4711 | 31 | 18 | ||
4712 | 32 | Exceptions are not caught: If you need stats even when exceptions are to be | 19 | Exceptions are not caught: If you need stats even when exceptions are to be |
4715 | 33 | raised, passing in a closure that will catch the exceptions and transform | 20 | raised, pass in a closure that will catch the exceptions and transform them |
4716 | 34 | them appropriately for your driver function. | 21 | appropriately for your driver function. |
4717 | 35 | 22 | ||
4718 | 36 | :return: The functions return value and a stats object. | 23 | :return: The functions return value and a stats object. |
4719 | 37 | """ | 24 | """ |
4724 | 38 | global _g_threadmap | 25 | profiler = BzrProfiler() |
4725 | 39 | p = Profiler() | 26 | profiler.start() |
4722 | 40 | p.enable(subcalls=True) | ||
4723 | 41 | threading.setprofile(_thread_profile) | ||
4726 | 42 | try: | 27 | try: |
4727 | 43 | ret = f(*args, **kwds) | 28 | ret = f(*args, **kwds) |
4728 | 44 | finally: | 29 | finally: |
4731 | 45 | p.disable() | 30 | stats = profiler.stop() |
4732 | 46 | for pp in _g_threadmap.values(): | 31 | return ret, stats |
4733 | 32 | |||
4734 | 33 | |||
4735 | 34 | class BzrProfiler(object): | ||
4736 | 35 | """Bzr utility wrapper around Profiler. | ||
4737 | 36 | |||
4738 | 37 | For most uses the module level 'profile()' function will be suitable. | ||
4739 | 38 | However profiling when a simple wrapped function isn't available may | ||
4740 | 39 | be easier to accomplish using this class. | ||
4741 | 40 | |||
4742 | 41 | To use it, create a BzrProfiler and call start() on it. Some arbitrary | ||
4743 | 42 | time later call stop() to stop profiling and retrieve the statistics | ||
4744 | 43 | from the code executed in the interim. | ||
4745 | 44 | """ | ||
4746 | 45 | |||
4747 | 46 | def start(self): | ||
4748 | 47 | """Start profiling. | ||
4749 | 48 | |||
4750 | 49 | This hooks into threading and will record all calls made until | ||
4751 | 50 | stop() is called. | ||
4752 | 51 | """ | ||
4753 | 52 | self._g_threadmap = {} | ||
4754 | 53 | self.p = Profiler() | ||
4755 | 54 | self.p.enable(subcalls=True) | ||
4756 | 55 | threading.setprofile(self._thread_profile) | ||
4757 | 56 | |||
4758 | 57 | def stop(self): | ||
4759 | 58 | """Stop profiling. | ||
4760 | 59 | |||
4761 | 60 | This unhooks from threading and cleans up the profiler, returning | ||
4762 | 61 | the gathered Stats object. | ||
4763 | 62 | |||
4764 | 63 | :return: A bzrlib.lsprof.Stats object. | ||
4765 | 64 | """ | ||
4766 | 65 | self.p.disable() | ||
4767 | 66 | for pp in self._g_threadmap.values(): | ||
4768 | 47 | pp.disable() | 67 | pp.disable() |
4769 | 48 | threading.setprofile(None) | 68 | threading.setprofile(None) |
4770 | 69 | p = self.p | ||
4771 | 70 | self.p = None | ||
4772 | 71 | threads = {} | ||
4773 | 72 | for tid, pp in self._g_threadmap.items(): | ||
4774 | 73 | threads[tid] = Stats(pp.getstats(), {}) | ||
4775 | 74 | self._g_threadmap = None | ||
4776 | 75 | return Stats(p.getstats(), threads) | ||
4777 | 49 | 76 | ||
4783 | 50 | threads = {} | 77 | def _thread_profile(self, f, *args, **kwds): |
4784 | 51 | for tid, pp in _g_threadmap.items(): | 78 | # we lose the first profile point for a new thread in order to |
4785 | 52 | threads[tid] = Stats(pp.getstats(), {}) | 79 | # trampoline a new Profile object into place |
4786 | 53 | _g_threadmap = {} | 80 | thr = thread.get_ident() |
4787 | 54 | return ret, Stats(p.getstats(), threads) | 81 | self._g_threadmap[thr] = p = Profiler() |
4788 | 82 | # this overrides our sys.setprofile hook: | ||
4789 | 83 | p.enable(subcalls=True, builtins=True) | ||
4790 | 55 | 84 | ||
4791 | 56 | 85 | ||
4792 | 57 | class Stats(object): | 86 | class Stats(object): |
4793 | 58 | 87 | ||
4794 | === modified file 'bzrlib/mail_client.py' | |||
4795 | --- bzrlib/mail_client.py 2009-06-10 03:56:49 +0000 | |||
4796 | +++ bzrlib/mail_client.py 2009-09-02 08:26:27 +0000 | |||
4797 | @@ -424,6 +424,10 @@ | |||
4798 | 424 | 424 | ||
4799 | 425 | _client_commands = ['emacsclient'] | 425 | _client_commands = ['emacsclient'] |
4800 | 426 | 426 | ||
4801 | 427 | def __init__(self, config): | ||
4802 | 428 | super(EmacsMail, self).__init__(config) | ||
4803 | 429 | self.elisp_tmp_file = None | ||
4804 | 430 | |||
4805 | 427 | def _prepare_send_function(self): | 431 | def _prepare_send_function(self): |
4806 | 428 | """Write our wrapper function into a temporary file. | 432 | """Write our wrapper function into a temporary file. |
4807 | 429 | 433 | ||
4808 | @@ -500,6 +504,7 @@ | |||
4809 | 500 | if attach_path is not None: | 504 | if attach_path is not None: |
4810 | 501 | # Do not create a file if there is no attachment | 505 | # Do not create a file if there is no attachment |
4811 | 502 | elisp = self._prepare_send_function() | 506 | elisp = self._prepare_send_function() |
4812 | 507 | self.elisp_tmp_file = elisp | ||
4813 | 503 | lmmform = '(load "%s")' % elisp | 508 | lmmform = '(load "%s")' % elisp |
4814 | 504 | mmform = '(bzr-add-mime-att "%s")' % \ | 509 | mmform = '(bzr-add-mime-att "%s")' % \ |
4815 | 505 | self._encode_path(attach_path, 'attachment') | 510 | self._encode_path(attach_path, 'attachment') |
4816 | 506 | 511 | ||
4817 | === modified file 'bzrlib/merge.py' | |||
4818 | --- bzrlib/merge.py 2009-07-02 13:07:14 +0000 | |||
4819 | +++ bzrlib/merge.py 2009-09-11 13:32:55 +0000 | |||
4820 | @@ -64,8 +64,12 @@ | |||
4821 | 64 | 64 | ||
4822 | 65 | 65 | ||
4823 | 66 | def transform_tree(from_tree, to_tree, interesting_ids=None): | 66 | def transform_tree(from_tree, to_tree, interesting_ids=None): |
4826 | 67 | merge_inner(from_tree.branch, to_tree, from_tree, ignore_zero=True, | 67 | from_tree.lock_tree_write() |
4827 | 68 | interesting_ids=interesting_ids, this_tree=from_tree) | 68 | try: |
4828 | 69 | merge_inner(from_tree.branch, to_tree, from_tree, ignore_zero=True, | ||
4829 | 70 | interesting_ids=interesting_ids, this_tree=from_tree) | ||
4830 | 71 | finally: | ||
4831 | 72 | from_tree.unlock() | ||
4832 | 69 | 73 | ||
4833 | 70 | 74 | ||
4834 | 71 | class Merger(object): | 75 | class Merger(object): |
4835 | @@ -102,6 +106,17 @@ | |||
4836 | 102 | self._is_criss_cross = None | 106 | self._is_criss_cross = None |
4837 | 103 | self._lca_trees = None | 107 | self._lca_trees = None |
4838 | 104 | 108 | ||
4839 | 109 | def cache_trees_with_revision_ids(self, trees): | ||
4840 | 110 | """Cache any tree in trees if it has a revision_id.""" | ||
4841 | 111 | for maybe_tree in trees: | ||
4842 | 112 | if maybe_tree is None: | ||
4843 | 113 | continue | ||
4844 | 114 | try: | ||
4845 | 115 | rev_id = maybe_tree.get_revision_id() | ||
4846 | 116 | except AttributeError: | ||
4847 | 117 | continue | ||
4848 | 118 | self._cached_trees[rev_id] = maybe_tree | ||
4849 | 119 | |||
4850 | 105 | @property | 120 | @property |
4851 | 106 | def revision_graph(self): | 121 | def revision_graph(self): |
4852 | 107 | if self._revision_graph is None: | 122 | if self._revision_graph is None: |
4853 | @@ -598,19 +613,21 @@ | |||
4854 | 598 | self.this_tree.lock_tree_write() | 613 | self.this_tree.lock_tree_write() |
4855 | 599 | self.base_tree.lock_read() | 614 | self.base_tree.lock_read() |
4856 | 600 | self.other_tree.lock_read() | 615 | self.other_tree.lock_read() |
4857 | 601 | self.tt = TreeTransform(self.this_tree, self.pb) | ||
4858 | 602 | try: | 616 | try: |
4864 | 603 | self.pp.next_phase() | 617 | self.tt = TreeTransform(self.this_tree, self.pb) |
4860 | 604 | self._compute_transform() | ||
4861 | 605 | self.pp.next_phase() | ||
4862 | 606 | results = self.tt.apply(no_conflicts=True) | ||
4863 | 607 | self.write_modified(results) | ||
4865 | 608 | try: | 618 | try: |
4869 | 609 | self.this_tree.add_conflicts(self.cooked_conflicts) | 619 | self.pp.next_phase() |
4870 | 610 | except UnsupportedOperation: | 620 | self._compute_transform() |
4871 | 611 | pass | 621 | self.pp.next_phase() |
4872 | 622 | results = self.tt.apply(no_conflicts=True) | ||
4873 | 623 | self.write_modified(results) | ||
4874 | 624 | try: | ||
4875 | 625 | self.this_tree.add_conflicts(self.cooked_conflicts) | ||
4876 | 626 | except UnsupportedOperation: | ||
4877 | 627 | pass | ||
4878 | 628 | finally: | ||
4879 | 629 | self.tt.finalize() | ||
4880 | 612 | finally: | 630 | finally: |
4881 | 613 | self.tt.finalize() | ||
4882 | 614 | self.other_tree.unlock() | 631 | self.other_tree.unlock() |
4883 | 615 | self.base_tree.unlock() | 632 | self.base_tree.unlock() |
4884 | 616 | self.this_tree.unlock() | 633 | self.this_tree.unlock() |
4885 | @@ -1516,6 +1533,7 @@ | |||
4886 | 1516 | get_revision_id = getattr(base_tree, 'get_revision_id', None) | 1533 | get_revision_id = getattr(base_tree, 'get_revision_id', None) |
4887 | 1517 | if get_revision_id is None: | 1534 | if get_revision_id is None: |
4888 | 1518 | get_revision_id = base_tree.last_revision | 1535 | get_revision_id = base_tree.last_revision |
4889 | 1536 | merger.cache_trees_with_revision_ids([other_tree, base_tree, this_tree]) | ||
4890 | 1519 | merger.set_base_revision(get_revision_id(), this_branch) | 1537 | merger.set_base_revision(get_revision_id(), this_branch) |
4891 | 1520 | return merger.do_merge() | 1538 | return merger.do_merge() |
4892 | 1521 | 1539 | ||
4893 | 1522 | 1540 | ||
4894 | === modified file 'bzrlib/missing.py' | |||
4895 | --- bzrlib/missing.py 2009-03-23 14:59:43 +0000 | |||
4896 | +++ bzrlib/missing.py 2009-08-17 18:52:01 +0000 | |||
4897 | @@ -138,31 +138,13 @@ | |||
4898 | 138 | if not ancestry: #Empty ancestry, no need to do any work | 138 | if not ancestry: #Empty ancestry, no need to do any work |
4899 | 139 | return [] | 139 | return [] |
4900 | 140 | 140 | ||
4922 | 141 | mainline_revs, rev_nos, start_rev_id, end_rev_id = log._get_mainline_revs( | 141 | merge_sorted_revisions = branch.iter_merge_sorted_revisions() |
4902 | 142 | branch, None, tip_revno) | ||
4903 | 143 | if not mainline_revs: | ||
4904 | 144 | return [] | ||
4905 | 145 | |||
4906 | 146 | # This asks for all mainline revisions, which is size-of-history and | ||
4907 | 147 | # should be addressed (but currently the only way to get correct | ||
4908 | 148 | # revnos). | ||
4909 | 149 | |||
4910 | 150 | # mainline_revisions always includes an extra revision at the | ||
4911 | 151 | # beginning, so don't request it. | ||
4912 | 152 | parent_map = dict(((key, value) for key, value | ||
4913 | 153 | in graph.iter_ancestry(mainline_revs[1:]) | ||
4914 | 154 | if value is not None)) | ||
4915 | 155 | # filter out ghosts; merge_sort errors on ghosts. | ||
4916 | 156 | # XXX: is this needed here ? -- vila080910 | ||
4917 | 157 | rev_graph = _mod_repository._strip_NULL_ghosts(parent_map) | ||
4918 | 158 | # XXX: what if rev_graph is empty now ? -- vila080910 | ||
4919 | 159 | merge_sorted_revisions = tsort.merge_sort(rev_graph, tip, | ||
4920 | 160 | mainline_revs, | ||
4921 | 161 | generate_revno=True) | ||
4923 | 162 | # Now that we got the correct revnos, keep only the relevant | 142 | # Now that we got the correct revnos, keep only the relevant |
4924 | 163 | # revisions. | 143 | # revisions. |
4925 | 164 | merge_sorted_revisions = [ | 144 | merge_sorted_revisions = [ |
4927 | 165 | (s, revid, n, d, e) for s, revid, n, d, e in merge_sorted_revisions | 145 | # log.reverse_by_depth expects seq_num to be present, but it is |
4928 | 146 | # stripped by iter_merge_sorted_revisions() | ||
4929 | 147 | (0, revid, n, d, e) for revid, n, d, e in merge_sorted_revisions | ||
4930 | 166 | if revid in ancestry] | 148 | if revid in ancestry] |
4931 | 167 | if not backward: | 149 | if not backward: |
4932 | 168 | merge_sorted_revisions = log.reverse_by_depth(merge_sorted_revisions) | 150 | merge_sorted_revisions = log.reverse_by_depth(merge_sorted_revisions) |
4933 | 169 | 151 | ||
4934 | === modified file 'bzrlib/mutabletree.py' | |||
4935 | --- bzrlib/mutabletree.py 2009-07-10 08:33:11 +0000 | |||
4936 | +++ bzrlib/mutabletree.py 2009-09-07 23:14:05 +0000 | |||
4937 | @@ -226,6 +226,9 @@ | |||
4938 | 226 | revprops=revprops, | 226 | revprops=revprops, |
4939 | 227 | possible_master_transports=possible_master_transports, | 227 | possible_master_transports=possible_master_transports, |
4940 | 228 | *args, **kwargs) | 228 | *args, **kwargs) |
4941 | 229 | post_hook_params = PostCommitHookParams(self) | ||
4942 | 230 | for hook in MutableTree.hooks['post_commit']: | ||
4943 | 231 | hook(post_hook_params) | ||
4944 | 229 | return committed_id | 232 | return committed_id |
4945 | 230 | 233 | ||
4946 | 231 | def _gather_kinds(self, files, kinds): | 234 | def _gather_kinds(self, files, kinds): |
4947 | @@ -581,14 +584,32 @@ | |||
4948 | 581 | self.create_hook(hooks.HookPoint('start_commit', | 584 | self.create_hook(hooks.HookPoint('start_commit', |
4949 | 582 | "Called before a commit is performed on a tree. The start commit " | 585 | "Called before a commit is performed on a tree. The start commit " |
4950 | 583 | "hook is able to change the tree before the commit takes place. " | 586 | "hook is able to change the tree before the commit takes place. " |
4953 | 584 | "start_commit is called with the bzrlib.tree.MutableTree that the " | 587 | "start_commit is called with the bzrlib.mutabletree.MutableTree " |
4954 | 585 | "commit is being performed on.", (1, 4), None)) | 588 | "that the commit is being performed on.", (1, 4), None)) |
4955 | 589 | self.create_hook(hooks.HookPoint('post_commit', | ||
4956 | 590 | "Called after a commit is performed on a tree. The hook is " | ||
4957 | 591 | "called with a bzrlib.mutabletree.PostCommitHookParams object. " | ||
4958 | 592 | "The mutable tree the commit was performed on is available via " | ||
4959 | 593 | "the mutable_tree attribute of that object.", (2, 0), None)) | ||
4960 | 586 | 594 | ||
4961 | 587 | 595 | ||
4962 | 588 | # install the default hooks into the MutableTree class. | 596 | # install the default hooks into the MutableTree class. |
4963 | 589 | MutableTree.hooks = MutableTreeHooks() | 597 | MutableTree.hooks = MutableTreeHooks() |
4964 | 590 | 598 | ||
4965 | 591 | 599 | ||
4966 | 600 | class PostCommitHookParams(object): | ||
4967 | 601 | """Parameters for the post_commit hook. | ||
4968 | 602 | |||
4969 | 603 | To access the parameters, use the following attributes: | ||
4970 | 604 | |||
4971 | 605 | * mutable_tree - the MutableTree object | ||
4972 | 606 | """ | ||
4973 | 607 | |||
4974 | 608 | def __init__(self, mutable_tree): | ||
4975 | 609 | """Create the parameters for the post_commit hook.""" | ||
4976 | 610 | self.mutable_tree = mutable_tree | ||
4977 | 611 | |||
4978 | 612 | |||
4979 | 592 | class _FastPath(object): | 613 | class _FastPath(object): |
4980 | 593 | """A path object with fast accessors for things like basename.""" | 614 | """A path object with fast accessors for things like basename.""" |
4981 | 594 | 615 | ||
4982 | 595 | 616 | ||
4983 | === modified file 'bzrlib/plugin.py' | |||
4984 | --- bzrlib/plugin.py 2009-03-24 01:53:42 +0000 | |||
4985 | +++ bzrlib/plugin.py 2009-09-04 15:36:48 +0000 | |||
4986 | @@ -52,12 +52,16 @@ | |||
4987 | 52 | from bzrlib import plugins as _mod_plugins | 52 | from bzrlib import plugins as _mod_plugins |
4988 | 53 | """) | 53 | """) |
4989 | 54 | 54 | ||
4991 | 55 | from bzrlib.symbol_versioning import deprecated_function | 55 | from bzrlib.symbol_versioning import ( |
4992 | 56 | deprecated_function, | ||
4993 | 57 | deprecated_in, | ||
4994 | 58 | ) | ||
4995 | 56 | 59 | ||
4996 | 57 | 60 | ||
4997 | 58 | DEFAULT_PLUGIN_PATH = None | 61 | DEFAULT_PLUGIN_PATH = None |
4998 | 59 | _loaded = False | 62 | _loaded = False |
4999 | 60 | 63 | ||
5000 | 64 | @deprecated_function(deprecated_in((2, 0, 0))) |
The diff has been truncated for viewing.
I've specially prepared a cherry-picked patch for the 2.0 branch. https:/ /code.launchpad .net/~garyvdm/ bzr/427773- 2.0-unlock- when-limbo/ +merge/ 11631