lp:~vila/bzr/fix277537

Created by Vincent Ladeuil and last modified
Get this branch:
bzr branch lp:~vila/bzr/fix277537
Only Vincent Ladeuil can upload to this branch. If you are Vincent Ladeuil please log in for upload directions.

Branch merges

Related bugs

Related blueprints

Branch information

Owner:
Vincent Ladeuil
Project:
Bazaar
Status:
Mature

Recent revisions

25. By Vincent Ladeuil

Fix regression in tests.

* tests/test_fix277537.py:
(BrokenRepoBuilderMixin.make_populated_repository): The tests has
been written with the hidden assumption that the default repo
format was non rich root.

24. By Vincent Ladeuil

Tweak batch sizes to have roughly the same memory consumption for both
storage kinds.

* reconcile.py:
(InventoryAncestryReconcilePacker._copy_fixed_nodes): Final tweak
for fulltext batch sizes.

23. By Vincent Ladeuil

Fix excessive memory consumption with fulltexts.

* reconcile.py:
(InventoryAncestryReconcilePacker._copy_fixed_nodes): Cache a
single parent when using fulltet as storage kind to reduce memory
consumption.

22. By Vincent Ladeuil

Ensure compatibility up to bzr-1.6.1 (at least).

* reconcile.py:
(InventoryAncestryReconcilePacker._copy_fixed_nodes): Ensure
old (as far as 1.6.1) bzr versions compatibility by using either
'chunked' or 'fulltext' storage kind. Fix another edge case in
batch calculation

21. By Vincent Ladeuil

Handle compression parent fix even if the plugin has already been
applied.

* tests/test_fix277537.py:
(TestCommand.assertFixedParents): Update to take compression
parents fixed into account.
(TestCommand.test_no_parents_fixed,
TestCommand.test_parents_fixed_but_not_twice): Update
assertFixedParents uses.

* reconcile.py:
(InventoryAncestryReconcilePacker.__init__): Add
compression_parent_fixed attribute.
(InventoryAncestryReconcilePacker._copy_text_texts): Separate th
fixes themselves from the part that decide which shold be
applied. Fix compression parent even if it isn't mentioned in the
inconsistent parents description. Change fixed_nodes description
to include only the desired parents (the wrong ones weren't
strictly needed). Delete surprising reference to knit._counter.
(InventoryAncestryReconcilePacker._copy_fixed_nodes): Fix bug when
there is a single remaining node to start a new batch. Update
fixed_nodes uses.
(InventoryAncestryReconciler.__init__): Add
compression_parent_fixed attribute.
(InventoryAncestryReconciler._reconcile_steps): Forward
compression_parent_fixed.

* __init__.py:
(cmd_fix277537.run): Also report compression parents fixed.

20. By John A Meinel

Change how we do the batch processing.
The old code used a fixed 100-node batch, the new code batches based on file-id,
so that no batch gets larger than 100, but we can still batch smaller groups
together.
We also now track the parent content cache, so less work needs to be done
for some of the text insertions.
Because we track parent texts, we don't seem to need to flush after every
record. This doesn't quite fit what I thought, but it seems to work just fine.
The hit rate isn't terrific, but it is reasonable. Memory still seems reasonable,
and the total time should be in the 10-minute range.
A little bit more could be gained by changing the bzr add_lines() code
to pass the parent_texts cache down into get_content_maps(), but it would
also require updating _get_record_map() to realize it doesn't need to
extract those specific entries.
(applying the delta in get_content_map is very fast versus extracting the
text in the first place in get_record_map)

19. By John A Meinel

properly import errno in case the file is missing

18. By John A Meinel

Use the right key layout

17. By John A Meinel

Change the sort order.

Group by file_id before sorting by revision topology. This should speed up text
extraction by getting multiple texts of the same file, rather than randomly.

16. By John A Meinel

Switch to using the topological sort order based on revision ids.

Branch metadata

Branch format:
Branch format 6
Repository format:
Bazaar pack repository format 1 (needs bzr 0.92)
This branch contains Public information 
Everyone can see this information.

Subscribers