Merge lp:~jameinel/bzr-fastimport/mooloolaba into lp:~bzr/bzr-fastimport/fastimport.dev
Proposed by
John A Meinel
on 2009-11-15
| Status: | Superseded |
|---|---|
| Proposed branch: | lp:~jameinel/bzr-fastimport/mooloolaba |
| Merge into: | lp:~bzr/bzr-fastimport/fastimport.dev |
| To merge this branch: | bzr merge lp:~jameinel/bzr-fastimport/mooloolaba |
| Related bugs: |
| Reviewer | Review Type | Date Requested | Status |
|---|---|---|---|
| Bazaar Developers | 2009-11-15 | Pending | |
|
Review via email:
|
|||
This proposal has been superseded by a proposal from 2009-11-29.
To post a comment you must log in.
| John A Meinel (jameinel) wrote : | # |
| Ian Clatworthy (ian-clatworthy) wrote : | # |
Can you push this branch again now that the trunk is in 2a format please?
This branch is in --2a format, but trunk isn't, so this request may fail.
Anyway, this is just a rollup of the stuff I worked on in Mooloolaba. Stuff like:
1) Memory reduction by referencing texts as strings rather than lines. This alone caused my peak qt import memory to go to 1.5GB from >2GB (I don't know how big, because I go OOM @2GB on my 32-bit machine).
2) Memory reduction by not caching all texts in memory. Instead, dump them into temp files. 'small' blob files are all collected into a single temp file, everything else gets a unique file that will be deleted when the last reference goes away. This got me down to between 500MB => 900MB during the import.
3) Switch the code that adds things to the repo to use CommitBuilder. It cheats a little bit here and there to get around deficiencies in the api. Namely, CommitBuilder wants to start_write_group and commit_write_group for every build, and we wanted to work around that.
I'm not sure what else is included, but that is the basics.