Merge lp:~mwilck/duplicity/0.7-series into lp:~duplicity-team/duplicity/0.7-series

Proposed by Martin Wilck
Status: Merged
Merged at revision: 1232
Proposed branch: lp:~mwilck/duplicity/0.7-series
Merge into: lp:~duplicity-team/duplicity/0.7-series
Diff against target: 111 lines (+31/-19)
5 files modified
duplicity/globmatch.py (+16/-12)
duplicity/selection.py (+3/-6)
testing/functional/__init__.py (+1/-1)
testing/gnupg/README (+2/-0)
testing/gnupg/gpg.conf (+9/-0)
To merge this branch: bzr merge lp:~mwilck/duplicity/0.7-series
Reviewer Review Type Date Requested Status
duplicity-team Pending
Review via email: mp+301332@code.launchpad.net

Description of the change

This merge request contains the same change as
https://code.launchpad.net/~mwilck/duplicity/duplicity/+merge/301268,
for the 0.7 series this time. I believe a factor-20 speedup could be counted as a "bug fix".

Moreover, it includes 2 fixes that were necessary on my system (OpenSUSE tumbleweed) to make
the test suite pass. See the commit logs for details. I needed this in order to verify that the
first change didn't introduce any regressions.

To post a comment you must log in.
lp:~mwilck/duplicity/0.7-series updated
1232. By ken on 2016-07-28

* Merged in lp:~mwilck/duplicity/0.7-series
  - Speedup of path_matches_glob() by about 8x. See
    https://code.launchpad.net/~mwilck/duplicity/0.7-series/+merge/301332
    for more details.

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

I had to remove -w from setsid to get precise and trusty to build on Launchpad.

Revision history for this message
Martin Wilck (mwilck) wrote :

Hmm, that would mean that setsid behaves differently on different distributions. It might be a difference in how setsid behaves, or a difference in pexpect.

In which test did the problems with "setsid -w" occur on precise / trusty?

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

All of the tests in tests/functional, as far as I can tell.

On Thu, Jul 28, 2016 at 1:59 PM, Martin Wilck <email address hidden> wrote:

> Hmm, that would mean that setsid behaves differently on different
> distributions. It might be a difference in how setsid behaves, or a
> difference in pexpect.
>
> In which test did the problems with "setsid -w" occur on precise / trusty?
>
> --
> https://code.launchpad.net/~mwilck/duplicity/0.7-series/+merge/301332
> You are subscribed to branch lp:duplicity/0.7-series.
>

Revision history for this message
Martin Wilck (mwilck) wrote :

To make sure I understand correctly - my changes work on Xenial and Wily but not on older Ubuntu releases? Are you using Ubuntu native pexpect packages or pexpect from PyPi?

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

I use pip, Launchpad uses their repo, which is out of date.

On Thursday, July 28, 2016, Martin Wilck <email address hidden> wrote:

> To make sure I understand correctly - my changes work on Xenial and Wily
> but not on older Ubuntu releases? Are you using Ubuntu native pexpect
> packages or pexpect from PyPi?
>
>
> --
> https://code.launchpad.net/~mwilck/duplicity/0.7-series/+merge/301332
> You are subscribed to branch lp:duplicity/0.7-series.
>

Revision history for this message
Martin Wilck (mwilck) wrote :

That might be the reason, then? Maybe we need to put in a version check on the pexpect module?

Revision history for this message
Martin Wilck (mwilck) wrote :

The problem is caused by different syntax of setsid invocation. setsid on Trusty doesn't support -w.

I am pushing a patch that tests for -w support and uses the flag only if supported.

Revision history for this message
Aaron Whitehouse (aaron-whitehouse) wrote :

For reference, while this talks about a x20 speedup, for me there is no material speed difference for the test suite between r 1231 and r 1232, so this speedup must be in a scenario not exhibited in the tests.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'duplicity/globmatch.py'
2--- duplicity/globmatch.py 2016-06-27 21:12:18 +0000
3+++ duplicity/globmatch.py 2016-07-27 21:14:04 +0000
4@@ -49,8 +49,9 @@
5 return list(map(glob_to_regex, prefixes))
6
7
8-def path_matches_glob(path, glob_str, include, ignore_case=False):
9- """Tests whether path matches glob, as per the Unix shell rules, taking as
10+def path_matches_glob_fn(glob_str, include, ignore_case=False):
11+ """Return a function test_fn(path) which
12+ tests whether path matches glob, as per the Unix shell rules, taking as
13 arguments a path, a glob string and include (0 indicating that the glob
14 string is an exclude glob and 1 indicating that it is an include glob,
15 returning:
16@@ -83,16 +84,19 @@
17 scan_comp_re = re_comp("^(%s)$" %
18 "|".join(_glob_get_prefix_regexs(glob_str)))
19
20- if match_only_dirs and not path.isdir():
21- # If the glob ended with a /, only match directories
22- return None
23- elif glob_comp_re.match(path.name):
24- return include
25- elif include == 1 and scan_comp_re.match(path.name):
26- return 2
27- else:
28- return None
29-
30+ def test_fn(path):
31+
32+ if match_only_dirs and not path.isdir():
33+ # If the glob ended with a /, only match directories
34+ return None
35+ elif glob_comp_re.match(path.name):
36+ return include
37+ elif include == 1 and scan_comp_re.match(path.name):
38+ return 2
39+ else:
40+ return None
41+
42+ return test_fn
43
44 def glob_to_regex(pat):
45 """Returned regular expression equivalent to shell glob pat
46
47=== modified file 'duplicity/selection.py'
48--- duplicity/selection.py 2016-07-24 00:07:50 +0000
49+++ duplicity/selection.py 2016-07-27 21:14:04 +0000
50@@ -33,7 +33,7 @@
51 from duplicity import diffdir
52 from duplicity import util # @Reimport
53 from duplicity.globmatch import GlobbingError, FilePrefixError, \
54- path_matches_glob
55+ path_matches_glob_fn
56
57 """Iterate exactly the requested files in a directory
58
59@@ -544,13 +544,10 @@
60 ignore_case = True
61
62 # Check to make sure prefix is ok
63- if not path_matches_glob(self.rootpath, glob_str, include=1):
64+ if not path_matches_glob_fn(glob_str, include=1)(self.rootpath):
65 raise FilePrefixError(glob_str)
66
67- def sel_func(path):
68- return path_matches_glob(path, glob_str, include, ignore_case)
69-
70- return sel_func
71+ return path_matches_glob_fn(glob_str, include, ignore_case)
72
73 def exclude_older_get_sf(self, date):
74 """Return selection function based on files older than modification date """
75
76=== modified file 'testing/functional/__init__.py'
77--- testing/functional/__init__.py 2016-01-06 13:39:33 +0000
78+++ testing/functional/__init__.py 2016-07-27 21:14:04 +0000
79@@ -65,7 +65,7 @@
80 # this way we force a failure if duplicity tries to read from the
81 # console unexpectedly (like for gpg password or such).
82 if platform.platform().startswith('Linux'):
83- cmd_list = ['setsid']
84+ cmd_list = ['setsid', '-w']
85 else:
86 cmd_list = []
87 cmd_list.extend(["duplicity"])
88
89=== modified file 'testing/gnupg/README'
90--- testing/gnupg/README 2011-11-04 12:48:04 +0000
91+++ testing/gnupg/README 2016-07-27 21:14:04 +0000
92@@ -11,3 +11,5 @@
93 ID: 9B736B2A
94 Name: Recipient Two <two@example.com>
95 No password
96+
97+See also the comments in gpg.conf.
98
99=== added file 'testing/gnupg/gpg.conf'
100--- testing/gnupg/gpg.conf 1970-01-01 00:00:00 +0000
101+++ testing/gnupg/gpg.conf 2016-07-27 21:14:04 +0000
102@@ -0,0 +1,9 @@
103+# gpg2 doesn't try all secrets by default, so add this option
104+# Otherwise the tests with hidden encryption key will fail
105+try-all-secrets
106+
107+# gpg2 2.1.13 has a bug that prevents the line above from working
108+# (https://bugs.gnupg.org/gnupg/issue1985)
109+# Uncomment the line below if you have gnupg2 2.1.13
110+# (but that line will break gpg 1.x, so we can't use it by default)
111+#try-secret-key 96B629431B77DC52B1917B40839E6A2856538CCF

Subscribers

People subscribed via source and target branches