Merge lp:~mwilck/duplicity/0.7-series into lp:duplicity/0.7-series

[Martin Wilck]

This merge request contains the same change as
https://code.launchpad.net/~mwilck/duplicity/duplicity/+merge/301268,
for the 0.7 series this time. I believe a factor-20 speedup could be counted as a "bug fix".

Moreover, it includes 2 fixes that were necessary on my system (OpenSUSE tumbleweed) to make
the test suite pass. See the commit logs for details. I needed this in order to verify that the
first change didn't introduce any regressions.

[lp:~mwilck/duplicity/0.7-series]

1232. By ken on 2016-07-28

* Merged in lp:~mwilck/duplicity/0.7-series
  - Speedup of path_matches_glob() by about 8x. See
    https://code.launchpad.net/~mwilck/duplicity/0.7-series/+merge/301332
    for more details.

[Kenneth Loafman (kenneth-loafman)]

I had to remove -w from setsid to get precise and trusty to build on Launchpad.

[Martin Wilck (mwilck)]

Hmm, that would mean that setsid behaves differently on different distributions. It might be a difference in how setsid behaves, or a difference in pexpect.

In which test did the problems with "setsid -w" occur on precise / trusty?

[Kenneth Loafman (kenneth-loafman)]

All of the tests in tests/functional, as far as I can tell.

On Thu, Jul 28, 2016 at 1:59 PM, Martin Wilck <email address hidden> wrote:

> Hmm, that would mean that setsid behaves differently on different
> distributions. It might be a difference in how setsid behaves, or a
> difference in pexpect.
>
> In which test did the problems with "setsid -w" occur on precise / trusty?
>
> --
> https://code.launchpad.net/~mwilck/duplicity/0.7-series/+merge/301332
> You are subscribed to branch lp:duplicity/0.7-series.
>

[Martin Wilck (mwilck)]

To make sure I understand correctly - my changes work on Xenial and Wily but not on older Ubuntu releases? Are you using Ubuntu native pexpect packages or pexpect from PyPi?

[Kenneth Loafman (kenneth-loafman)]

I use pip, Launchpad uses their repo, which is out of date.

On Thursday, July 28, 2016, Martin Wilck <email address hidden> wrote:

> To make sure I understand correctly - my changes work on Xenial and Wily
> but not on older Ubuntu releases? Are you using Ubuntu native pexpect
> packages or pexpect from PyPi?
>
>
> --
> https://code.launchpad.net/~mwilck/duplicity/0.7-series/+merge/301332
> You are subscribed to branch lp:duplicity/0.7-series.
>

[Martin Wilck (mwilck)]

That might be the reason, then? Maybe we need to put in a version check on the pexpect module?

[Martin Wilck (mwilck)]

The problem is caused by different syntax of setsid invocation. setsid on Trusty doesn't support -w.

I am pushing a patch that tests for -w support and uses the flag only if supported.

[Aaron Whitehouse (aaron-whitehouse)]

For reference, while this talks about a x20 speedup, for me there is no material speed difference for the test suite between r 1231 and r 1232, so this speedup must be in a scenario not exhibited in the tests.

=== modified file 'duplicity/globmatch.py'
--- duplicity/globmatch.py	2016-06-27 21:12:18 +0000
+++ duplicity/globmatch.py	2016-07-27 21:14:04 +0000
@@ -49,8 +49,9 @@
     return list(map(glob_to_regex, prefixes))
 
 
-def path_matches_glob(path, glob_str, include, ignore_case=False):
-    """Tests whether path matches glob, as per the Unix shell rules, taking as
+def path_matches_glob_fn(glob_str, include, ignore_case=False):
+    """Return a function test_fn(path) which
+    tests whether path matches glob, as per the Unix shell rules, taking as
     arguments a path, a glob string and include (0 indicating that the glob
     string is an exclude glob and 1 indicating that it is an include glob,
     returning:
@@ -83,16 +84,19 @@
     scan_comp_re = re_comp("^(%s)$" %
                            "|".join(_glob_get_prefix_regexs(glob_str)))
 
-    if match_only_dirs and not path.isdir():
-        # If the glob ended with a /, only match directories
-        return None
-    elif glob_comp_re.match(path.name):
-        return include
-    elif include == 1 and scan_comp_re.match(path.name):
-        return 2
-    else:
-        return None
-
+    def test_fn(path):
+
+        if match_only_dirs and not path.isdir():
+            # If the glob ended with a /, only match directories
+            return None
+        elif glob_comp_re.match(path.name):
+            return include
+        elif include == 1 and scan_comp_re.match(path.name):
+            return 2
+        else:
+            return None
+
+    return test_fn
 
 def glob_to_regex(pat):
     """Returned regular expression equivalent to shell glob pat

=== modified file 'duplicity/selection.py'
--- duplicity/selection.py	2016-07-24 00:07:50 +0000
+++ duplicity/selection.py	2016-07-27 21:14:04 +0000
@@ -33,7 +33,7 @@
 from duplicity import diffdir
 from duplicity import util  # @Reimport
 from duplicity.globmatch import GlobbingError, FilePrefixError, \
-    path_matches_glob
+    path_matches_glob_fn
 
 """Iterate exactly the requested files in a directory
 
@@ -544,13 +544,10 @@
             ignore_case = True
 
         # Check to make sure prefix is ok
-        if not path_matches_glob(self.rootpath, glob_str, include=1):
+        if not path_matches_glob_fn(glob_str, include=1)(self.rootpath):
             raise FilePrefixError(glob_str)
 
-        def sel_func(path):
-            return path_matches_glob(path, glob_str, include, ignore_case)
-
-        return sel_func
+        return path_matches_glob_fn(glob_str, include, ignore_case)
 
     def exclude_older_get_sf(self, date):
         """Return selection function based on files older than modification date """

=== modified file 'testing/functional/__init__.py'
--- testing/functional/__init__.py	2016-01-06 13:39:33 +0000
+++ testing/functional/__init__.py	2016-07-27 21:14:04 +0000
@@ -65,7 +65,7 @@
         # this way we force a failure if duplicity tries to read from the
         # console unexpectedly (like for gpg password or such).
         if platform.platform().startswith('Linux'):
-            cmd_list = ['setsid']
+            cmd_list = ['setsid', '-w']
         else:
             cmd_list = []
         cmd_list.extend(["duplicity"])

=== modified file 'testing/gnupg/README'
--- testing/gnupg/README	2011-11-04 12:48:04 +0000
+++ testing/gnupg/README	2016-07-27 21:14:04 +0000
@@ -11,3 +11,5 @@
 ID: 9B736B2A
 Name: Recipient Two <[e-mail]>
 No password
+
+See also the comments in gpg.conf.

=== added file 'testing/gnupg/gpg.conf'
--- testing/gnupg/gpg.conf	1970-01-01 00:00:00 +0000
+++ testing/gnupg/gpg.conf	2016-07-27 21:14:04 +0000
@@ -0,0 +1,9 @@
+# gpg2 doesn't try all secrets by default, so add this option
+# Otherwise the tests with hidden encryption key will fail
+try-all-secrets
+
+# gpg2 2.1.13 has a bug that prevents the line above from working
+# (https://bugs.gnupg.org/gnupg/issue1985)
+# Uncomment the line below if you have gnupg2 2.1.13
+# (but that line will break gpg 1.x, so we can't use it by default)
+#try-secret-key 96B629431B77DC52B1917B40839E6A2856538CCF