Merge lp:~mwilck/duplicity/0.7-series into lp:duplicity/0.7-series

Proposed by Martin Wilck on 2016-07-27
Status: Merged
Merged at revision: 1232
Proposed branch: lp:~mwilck/duplicity/0.7-series
Merge into: lp:duplicity/0.7-series
Diff against target: 111 lines (+31/-19)
5 files modified
duplicity/globmatch.py (+16/-12)
duplicity/selection.py (+3/-6)
testing/functional/__init__.py (+1/-1)
testing/gnupg/README (+2/-0)
testing/gnupg/gpg.conf (+9/-0)
To merge this branch: bzr merge lp:~mwilck/duplicity/0.7-series
Reviewer Review Type Date Requested Status
duplicity-team 2016-07-27 Pending
Review via email: mp+301332@code.launchpad.net

Description of the change

This merge request contains the same change as
https://code.launchpad.net/~mwilck/duplicity/duplicity/+merge/301268,
for the 0.7 series this time. I believe a factor-20 speedup could be counted as a "bug fix".

Moreover, it includes 2 fixes that were necessary on my system (OpenSUSE tumbleweed) to make
the test suite pass. See the commit logs for details. I needed this in order to verify that the
first change didn't introduce any regressions.

To post a comment you must log in.
lp:~mwilck/duplicity/0.7-series updated on 2016-07-28
1232. By ken on 2016-07-28

* Merged in lp:~mwilck/duplicity/0.7-series
  - Speedup of path_matches_glob() by about 8x. See
    https://code.launchpad.net/~mwilck/duplicity/0.7-series/+merge/301332
    for more details.

Kenneth Loafman (kenneth-loafman) wrote :

I had to remove -w from setsid to get precise and trusty to build on Launchpad.

Martin Wilck (mwilck) wrote :

Hmm, that would mean that setsid behaves differently on different distributions. It might be a difference in how setsid behaves, or a difference in pexpect.

In which test did the problems with "setsid -w" occur on precise / trusty?

Kenneth Loafman (kenneth-loafman) wrote :

All of the tests in tests/functional, as far as I can tell.

On Thu, Jul 28, 2016 at 1:59 PM, Martin Wilck <email address hidden> wrote:

> Hmm, that would mean that setsid behaves differently on different
> distributions. It might be a difference in how setsid behaves, or a
> difference in pexpect.
>
> In which test did the problems with "setsid -w" occur on precise / trusty?
>
> --
> https://code.launchpad.net/~mwilck/duplicity/0.7-series/+merge/301332
> You are subscribed to branch lp:duplicity/0.7-series.
>

Martin Wilck (mwilck) wrote :

To make sure I understand correctly - my changes work on Xenial and Wily but not on older Ubuntu releases? Are you using Ubuntu native pexpect packages or pexpect from PyPi?

Kenneth Loafman (kenneth-loafman) wrote :

I use pip, Launchpad uses their repo, which is out of date.

On Thursday, July 28, 2016, Martin Wilck <email address hidden> wrote:

> To make sure I understand correctly - my changes work on Xenial and Wily
> but not on older Ubuntu releases? Are you using Ubuntu native pexpect
> packages or pexpect from PyPi?
>
>
> --
> https://code.launchpad.net/~mwilck/duplicity/0.7-series/+merge/301332
> You are subscribed to branch lp:duplicity/0.7-series.
>

Martin Wilck (mwilck) wrote :

That might be the reason, then? Maybe we need to put in a version check on the pexpect module?

Martin Wilck (mwilck) wrote :

The problem is caused by different syntax of setsid invocation. setsid on Trusty doesn't support -w.

I am pushing a patch that tests for -w support and uses the flag only if supported.

Aaron Whitehouse (aaron-whitehouse) wrote :

For reference, while this talks about a x20 speedup, for me there is no material speed difference for the test suite between r 1231 and r 1232, so this speedup must be in a scenario not exhibited in the tests.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'duplicity/globmatch.py'
2--- duplicity/globmatch.py 2016-06-27 21:12:18 +0000
3+++ duplicity/globmatch.py 2016-07-27 21:14:04 +0000
4@@ -49,8 +49,9 @@
5 return list(map(glob_to_regex, prefixes))
6
7
8-def path_matches_glob(path, glob_str, include, ignore_case=False):
9- """Tests whether path matches glob, as per the Unix shell rules, taking as
10+def path_matches_glob_fn(glob_str, include, ignore_case=False):
11+ """Return a function test_fn(path) which
12+ tests whether path matches glob, as per the Unix shell rules, taking as
13 arguments a path, a glob string and include (0 indicating that the glob
14 string is an exclude glob and 1 indicating that it is an include glob,
15 returning:
16@@ -83,16 +84,19 @@
17 scan_comp_re = re_comp("^(%s)$" %
18 "|".join(_glob_get_prefix_regexs(glob_str)))
19
20- if match_only_dirs and not path.isdir():
21- # If the glob ended with a /, only match directories
22- return None
23- elif glob_comp_re.match(path.name):
24- return include
25- elif include == 1 and scan_comp_re.match(path.name):
26- return 2
27- else:
28- return None
29-
30+ def test_fn(path):
31+
32+ if match_only_dirs and not path.isdir():
33+ # If the glob ended with a /, only match directories
34+ return None
35+ elif glob_comp_re.match(path.name):
36+ return include
37+ elif include == 1 and scan_comp_re.match(path.name):
38+ return 2
39+ else:
40+ return None
41+
42+ return test_fn
43
44 def glob_to_regex(pat):
45 """Returned regular expression equivalent to shell glob pat
46
47=== modified file 'duplicity/selection.py'
48--- duplicity/selection.py 2016-07-24 00:07:50 +0000
49+++ duplicity/selection.py 2016-07-27 21:14:04 +0000
50@@ -33,7 +33,7 @@
51 from duplicity import diffdir
52 from duplicity import util # @Reimport
53 from duplicity.globmatch import GlobbingError, FilePrefixError, \
54- path_matches_glob
55+ path_matches_glob_fn
56
57 """Iterate exactly the requested files in a directory
58
59@@ -544,13 +544,10 @@
60 ignore_case = True
61
62 # Check to make sure prefix is ok
63- if not path_matches_glob(self.rootpath, glob_str, include=1):
64+ if not path_matches_glob_fn(glob_str, include=1)(self.rootpath):
65 raise FilePrefixError(glob_str)
66
67- def sel_func(path):
68- return path_matches_glob(path, glob_str, include, ignore_case)
69-
70- return sel_func
71+ return path_matches_glob_fn(glob_str, include, ignore_case)
72
73 def exclude_older_get_sf(self, date):
74 """Return selection function based on files older than modification date """
75
76=== modified file 'testing/functional/__init__.py'
77--- testing/functional/__init__.py 2016-01-06 13:39:33 +0000
78+++ testing/functional/__init__.py 2016-07-27 21:14:04 +0000
79@@ -65,7 +65,7 @@
80 # this way we force a failure if duplicity tries to read from the
81 # console unexpectedly (like for gpg password or such).
82 if platform.platform().startswith('Linux'):
83- cmd_list = ['setsid']
84+ cmd_list = ['setsid', '-w']
85 else:
86 cmd_list = []
87 cmd_list.extend(["duplicity"])
88
89=== modified file 'testing/gnupg/README'
90--- testing/gnupg/README 2011-11-04 12:48:04 +0000
91+++ testing/gnupg/README 2016-07-27 21:14:04 +0000
92@@ -11,3 +11,5 @@
93 ID: 9B736B2A
94 Name: Recipient Two <two@example.com>
95 No password
96+
97+See also the comments in gpg.conf.
98
99=== added file 'testing/gnupg/gpg.conf'
100--- testing/gnupg/gpg.conf 1970-01-01 00:00:00 +0000
101+++ testing/gnupg/gpg.conf 2016-07-27 21:14:04 +0000
102@@ -0,0 +1,9 @@
103+# gpg2 doesn't try all secrets by default, so add this option
104+# Otherwise the tests with hidden encryption key will fail
105+try-all-secrets
106+
107+# gpg2 2.1.13 has a bug that prevents the line above from working
108+# (https://bugs.gnupg.org/gnupg/issue1985)
109+# Uncomment the line below if you have gnupg2 2.1.13
110+# (but that line will break gpg 1.x, so we can't use it by default)
111+#try-secret-key 96B629431B77DC52B1917B40839E6A2856538CCF

Subscribers

People subscribed via source and target branches