Merge lp:~mwilck/duplicity/duplicity into lp:~duplicity-team/duplicity/0.8-series
Status: | Merged |
---|---|
Merged at revision: | 1116 |
Proposed branch: | lp:~mwilck/duplicity/duplicity |
Merge into: | lp:~duplicity-team/duplicity/0.8-series |
Diff against target: |
74 lines (+19/-18) 2 files modified
duplicity/globmatch.py (+16/-12) duplicity/selection.py (+3/-6) |
To merge this branch: | bzr merge lp:~mwilck/duplicity/duplicity |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Kenneth Loafman | 2016-07-27 | Approve on 2016-07-28 | |
Review via email:
|
Description of the change
Glob matching in current duplicity uses a selector function that calls path_matches_
My proposed patches create a closure function instead that uses precalculated regular expressions; the regular expressions are thus constructed only once at initialization time.
This change speeds up duplicity a *lot* when complex include/exclude lists are in use: for my use case (dry-run, backup of an SSD filesystem), the speedup is a factor of 25 (runtime: 4s rather than 90s).
Martin Wilck (mwilck) wrote : | # |
Martin Wilck (mwilck) wrote : | # |
I added a merge request for the 0.7-series now, https:/
Kenneth Loafman (kenneth-loafman) wrote : | # |
'$ tox' yields the following:
=======
FAIL: test_dirx (testing.
Test cycle on testfiles/dirx
-------
Traceback (most recent call last):
File "/Users/
'testfiles/
File "/Users/
self.
File "/Users/
self.
File "/Users/
assert not os.system(command)
AssertionError
Kenneth Loafman (kenneth-loafman) wrote : | # |
Never mind... I reran with 'tox -r' and all is well.
- 1116. By ken on 2016-07-28
-
* Merged in lp:~mwilck/duplicity/duplicity
- Speedup of path_matches_glob() by about 8x. See
https://code.launchpad .net/~mwilck/ duplicity/ duplicity/ +merge/ 301268
for more details.
Aaron Whitehouse (aaron-whitehouse) wrote : | # |
For reference, while this talks about a x25 (or x8) speedup, for me there is no material speed difference for the test suite between r 1115 and r 1116, so this speedup must be in a scenario not exhibited in the tests.
edso (ed.so) wrote : | # |
On 10.02.2017 00:25, Aaron Whitehouse wrote:
> For reference, while this talks about a x25 (or x8) speedup, for me there is no material speed difference for the test suite between r 1115 and r 1116, so this speedup must be in a scenario not exhibited in the tests.
>
Aaron,
the slowdown only becomes apparent when dealing with many files. you may need to add a test with thousands of files to make it visible.
..ede/duply.net
The same change can be trivially merged into the 0.7 series, providing a similar speed-up. I did that in lp:~mwilck/duplicity/0.7-series. I'll postpone officially requesting this merge until someone has commented here.