Merge lp:~mwilck/duplicity/duplicity into lp:~duplicity-team/duplicity/0.8-series

[Martin Wilck]

Glob matching in current duplicity uses a selector function that calls path_matches_glob(). This means that whenever a filename is matched, path_matches_glob() goes through the process of transforming a glob expression into regular expressions for filename and directory matching.

My proposed patches create a closure function instead that uses precalculated regular expressions; the regular expressions are thus constructed only once at initialization time.

This change speeds up duplicity a *lot* when complex include/exclude lists are in use: for my use case (dry-run, backup of an SSD filesystem), the speedup is a factor of 25 (runtime: 4s rather than 90s).

[Martin Wilck (mwilck)]

The same change can be trivially merged into the 0.7 series, providing a similar speed-up. I did that in lp:~mwilck/duplicity/0.7-series. I'll postpone officially requesting this merge until someone has commented here.

[Martin Wilck (mwilck)]

I added a merge request for the 0.7-series now, https://code.launchpad.net/~mwilck/duplicity/0.7-series/+merge/301332.

[Kenneth Loafman (kenneth-loafman)]

'$ tox' yields the following:

======================================================================
FAIL: test_dirx (testing.functional.test_rdiffdir.RdiffdirTest)
Test cycle on testfiles/dirx
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/ken/workspace/duplicity-src8/testing/functional/test_rdiffdir.py", line 88, in test_dirx
    'testfiles/empty_dir'])
  File "/Users/ken/workspace/duplicity-src8/testing/functional/test_rdiffdir.py", line 50, in run_cycle
    self.run_rdiffdir("sig %s %s" % (seq_path.name, sig_path.name))
  File "/Users/ken/workspace/duplicity-src8/testing/functional/test_rdiffdir.py", line 37, in run_rdiffdir
    self.run_cmd("rdiffdir " + argstring)
  File "/Users/ken/workspace/duplicity-src8/testing/functional/test_rdiffdir.py", line 33, in run_cmd
    assert not os.system(command)
AssertionError

review: Needs Fixing

[Kenneth Loafman (kenneth-loafman)]

Never mind... I reran with 'tox -r' and all is well.

review: Approve

[lp:~mwilck/duplicity/duplicity]

1116. By ken on 2016-07-28

* Merged in lp:~mwilck/duplicity/duplicity
  - Speedup of path_matches_glob() by about 8x. See
    https://code.launchpad.net/~mwilck/duplicity/duplicity/+merge/301268
    for more details.

[Aaron Whitehouse (aaron-whitehouse)]

For reference, while this talks about a x25 (or x8) speedup, for me there is no material speed difference for the test suite between r 1115 and r 1116, so this speedup must be in a scenario not exhibited in the tests.

[edso (ed.so)]

On 10.02.2017 00:25, Aaron Whitehouse wrote:
> For reference, while this talks about a x25 (or x8) speedup, for me there is no material speed difference for the test suite between r 1115 and r 1116, so this speedup must be in a scenario not exhibited in the tests.
>

Aaron,

the slowdown only becomes apparent when dealing with many files. you may need to add a test with thousands of files to make it visible.

..ede/duply.net

=== modified file 'duplicity/globmatch.py'
--- duplicity/globmatch.py	2016-06-27 21:12:18 +0000
+++ duplicity/globmatch.py	2016-07-27 13:08:36 +0000
@@ -49,8 +49,9 @@
     return list(map(glob_to_regex, prefixes))
 
 
-def path_matches_glob(path, glob_str, include, ignore_case=False):
-    """Tests whether path matches glob, as per the Unix shell rules, taking as
+def path_matches_glob_fn(glob_str, include, ignore_case=False):
+    """Return a function test_fn(path) which
+    tests whether path matches glob, as per the Unix shell rules, taking as
     arguments a path, a glob string and include (0 indicating that the glob
     string is an exclude glob and 1 indicating that it is an include glob,
     returning:
@@ -83,16 +84,19 @@
     scan_comp_re = re_comp("^(%s)$" %
                            "|".join(_glob_get_prefix_regexs(glob_str)))
 
-    if match_only_dirs and not path.isdir():
-        # If the glob ended with a /, only match directories
-        return None
-    elif glob_comp_re.match(path.name):
-        return include
-    elif include == 1 and scan_comp_re.match(path.name):
-        return 2
-    else:
-        return None
-
+    def test_fn(path):
+
+        if match_only_dirs and not path.isdir():
+            # If the glob ended with a /, only match directories
+            return None
+        elif glob_comp_re.match(path.name):
+            return include
+        elif include == 1 and scan_comp_re.match(path.name):
+            return 2
+        else:
+            return None
+
+    return test_fn
 
 def glob_to_regex(pat):
     """Returned regular expression equivalent to shell glob pat

=== modified file 'duplicity/selection.py'
--- duplicity/selection.py	2016-07-24 12:30:45 +0000
+++ duplicity/selection.py	2016-07-27 13:08:36 +0000
@@ -33,7 +33,7 @@
 from duplicity import diffdir
 from duplicity import util  # @Reimport
 from duplicity.globmatch import GlobbingError, FilePrefixError, \
-    path_matches_glob
+    path_matches_glob_fn
 
 """Iterate exactly the requested files in a directory
 
@@ -544,13 +544,10 @@
             ignore_case = True
 
         # Check to make sure prefix is ok
-        if not path_matches_glob(self.rootpath, glob_str, include=1):
+        if not path_matches_glob_fn(glob_str, include=1)(self.rootpath):
             raise FilePrefixError(glob_str)
 
-        def sel_func(path):
-            return path_matches_glob(path, glob_str, include, ignore_case)
-
-        return sel_func
+        return path_matches_glob_fn(glob_str, include, ignore_case)
 
     def exclude_older_get_sf(self, date):
         """Return selection function based on files older than modification date """