Merge lp:~yajo/duplicity/duplicity into lp:duplicity/0.7-series

Proposed by Yajo on 2019-04-22
Status: Merged
Merged at revision: 1373
Proposed branch: lp:~yajo/duplicity/duplicity
Merge into: lp:duplicity/0.7-series
Diff against target: 126 lines (+44/-7)
4 files modified
bin/duplicity (+28/-7)
bin/duplicity.1 (+8/-0)
duplicity/commandline.py (+5/-0)
duplicity/globals.py (+3/-0)
To merge this branch: bzr merge lp:~yajo/duplicity/duplicity
Reviewer Review Type Date Requested Status
edso 2019-04-22 Approve on 2019-04-24
Review via email: mp+366358@code.launchpad.net

Description of the change

Support partial metadata sync.

Fixes bug #1823858 by letting the user to choose partial syncing. Only the metadata for the target chain will be downloaded. If older (or newer) chains are encrypted with a different passphrase, the user will be able to restore to a given time by supplying only the passphrase for the chain selected by the `--restore-time` option when using this new option.

A side effect is that using this flag reduces dramatically the sync time when moving files from one to another location, in cases where big amounts of chains are found.

To post a comment you must log in.
Aaron Whitehouse (aaron-whitehouse) wrote :

Thank you for your work, Yajo.

Is there a reason that you have requested to merge this into the 0.7 series instead of the main duplicity (0.8 series)?

0.7 is not really suited for new features at this point.

edso (ed.so) wrote :

hi Yajo,

the approach seems sound. however here some small issues, apart from the merge request against the old stable branch.

1. leftover debugger import
+ import pudb; pudb.set_trace()

2. new default sync mode should be partial, as it is faster and transfers only needed files

3.
+from gettext import gettext as _
looks redundant. gettext should be imported by 'duplicity/__init__.py' already.

nice that you updated the manpage accordingly already!

..ede/duply.net

review: Needs Fixing
lp:~yajo/duplicity/duplicity updated on 2019-04-24
1373. By Yajo on 2019-04-24

Support partial metadata sync.

Fixes bug #1823858 by letting the user to choose partial syncing. Only the metadata for the target chain will be downloaded. If older (or newer) chains are encrypted with a different passphrase, the user will be able to restore to a given time by supplying only the passphrase for the chain selected by the `--restore-time` option when using this new option.

A side effect is that using this flag reduces dramatically the sync time when moving files from one to another location, in cases where big amounts of chains are found.

Yajo (yajo) wrote :

Hi there, concerns 1 and 3 fixed.

About number 2, the problem is that I really need this patch in 0.7 because I'm hitting bug #1823858 it in production, whereas 0.8 is not yet released. This is half feature half bugfix, but TBH what motivated me most to push the MR is the bugfix.

I thought that a good approach would be to add the feature but disable it by default (as I did) in 0.7, but in 0.8 we could default to the partial sync mode because, as you say, it is has most benefits.

Or can you think a better solution?

edso (ed.so) wrote :

> Hi there, concerns 1 and 3 fixed.

thx

> About number 2, the problem is that I really need this patch in 0.7 because I'm hitting bug #1823858 it in production, whereas 0.8 is not yet released. This is half feature half bugfix, but TBH what motivated me most to push the MR is the bugfix.

sounds reasonable. disabled by default even 0.7 could benefit.

> I thought that a good approach would be to add the feature but disable it by default (as I did) in 0.7, but in 0.8 we could default to the partial sync mode because, as you say, it is has most benefits.
>
> Or can you think a better solution?

nope. perfectly valid reasoning. in the end it's up to Ken the maintainer.
@Ken: what do you say?

..ede/duply.net

review: Approve
Yajo (yajo) wrote :

Thanks for the merge!

Could you push a new 0.7 release with this patch included?

Kenneth Loafman (kenneth-loafman) wrote :

Yes, I'll do that soon.

On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:

> Thanks for the merge!
>
> Could you push a new 0.7 release with this patch included?
> --
> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> You are subscribed to branch lp:duplicity/0.7-series.
>

edso (ed.so) wrote :

hey Ken,

it's not committed to 0.8 so far. please do not forget.

also i'd say we make 'partial' default from 0.8 on, as argumented in the merge ticket. ..ede

On 29.04.2019 14:06, Kenneth Loafman wrote:
> Yes, I'll do that soon.
>
> On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:
>
>> Thanks for the merge!
>>
>> Could you push a new 0.7 release with this patch included?
>> --
>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>> You are subscribed to branch lp:duplicity/0.7-series.
>>
>

Kenneth Loafman (kenneth-loafman) wrote :

Hi ede,

I'm working on it now. Have to do some of the changes manually for 0.8.
Lots of changes since 0.7!

...Ken

On Mon, Apr 29, 2019 at 7:19 AM edso <email address hidden> wrote:

> hey Ken,
>
> it's not committed to 0.8 so far. please do not forget.
>
> also i'd say we make 'partial' default from 0.8 on, as argumented in the
> merge ticket. ..ede
>
> On 29.04.2019 14:06, Kenneth Loafman wrote:
> > Yes, I'll do that soon.
> >
> > On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:
> >
> >> Thanks for the merge!
> >>
> >> Could you push a new 0.7 release with this patch included?
> >> --
> >> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> >> You are subscribed to branch lp:duplicity/0.7-series.
> >>
> >
>
>
> --
> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> You are subscribed to branch lp:duplicity/0.7-series.
>

edso (ed.so) wrote :

maybe Yajo wants to help wrt. this issue in 0.8? Yajo got the time? ..ede/duply.net

On 03.05.2019 21:39, Kenneth Loafman wrote:
> Hi ede,
>
> I'm working on it now. Have to do some of the changes manually for 0.8.
> Lots of changes since 0.7!
>
> ...Ken
>
> On Mon, Apr 29, 2019 at 7:19 AM edso <email address hidden> wrote:
>
>> hey Ken,
>>
>> it's not committed to 0.8 so far. please do not forget.
>>
>> also i'd say we make 'partial' default from 0.8 on, as argumented in the
>> merge ticket. ..ede
>>
>> On 29.04.2019 14:06, Kenneth Loafman wrote:
>>> Yes, I'll do that soon.
>>>
>>> On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:
>>>
>>>> Thanks for the merge!
>>>>
>>>> Could you push a new 0.7 release with this patch included?
>>>> --
>>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>>>> You are subscribed to branch lp:duplicity/0.7-series.
>>>>
>>>
>>
>>
>> --
>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>> You are subscribed to branch lp:duplicity/0.7-series.
>>
>

Kenneth Loafman (kenneth-loafman) wrote :

ede,

Manual merge is done. Should have it out in a couple of days.

...Ken

On Sat, May 4, 2019 at 5:06 AM edso <email address hidden> wrote:

> maybe Yajo wants to help wrt. this issue in 0.8? Yajo got the time? ..ede/
> duply.net
>
> On 03.05.2019 21:39, Kenneth Loafman wrote:
> > Hi ede,
> >
> > I'm working on it now. Have to do some of the changes manually for 0.8.
> > Lots of changes since 0.7!
> >
> > ...Ken
> >
> > On Mon, Apr 29, 2019 at 7:19 AM edso <email address hidden> wrote:
> >
> >> hey Ken,
> >>
> >> it's not committed to 0.8 so far. please do not forget.
> >>
> >> also i'd say we make 'partial' default from 0.8 on, as argumented in the
> >> merge ticket. ..ede
> >>
> >> On 29.04.2019 14:06, Kenneth Loafman wrote:
> >>> Yes, I'll do that soon.
> >>>
> >>> On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:
> >>>
> >>>> Thanks for the merge!
> >>>>
> >>>> Could you push a new 0.7 release with this patch included?
> >>>> --
> >>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> >>>> You are subscribed to branch lp:duplicity/0.7-series.
> >>>>
> >>>
> >>
> >>
> >> --
> >> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> >> You are subscribed to branch lp:duplicity/0.7-series.
> >>
> >
>
>
> --
> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> You are subscribed to branch lp:duplicity/0.7-series.
>

edso (ed.so) wrote :

Ken,

probably local at your box still? it's not merged afaics https://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/changes/

..ede

On 04.05.2019 18:30, Kenneth Loafman wrote:
> ede,
>
> Manual merge is done. Should have it out in a couple of days.
>
> ...Ken
>
>
> On Sat, May 4, 2019 at 5:06 AM edso <email address hidden> wrote:
>
>> maybe Yajo wants to help wrt. this issue in 0.8? Yajo got the time? ..ede/
>> duply.net
>>
>> On 03.05.2019 21:39, Kenneth Loafman wrote:
>>> Hi ede,
>>>
>>> I'm working on it now. Have to do some of the changes manually for 0.8.
>>> Lots of changes since 0.7!
>>>
>>> ...Ken
>>>
>>> On Mon, Apr 29, 2019 at 7:19 AM edso <email address hidden> wrote:
>>>
>>>> hey Ken,
>>>>
>>>> it's not committed to 0.8 so far. please do not forget.
>>>>
>>>> also i'd say we make 'partial' default from 0.8 on, as argumented in the
>>>> merge ticket. ..ede
>>>>
>>>> On 29.04.2019 14:06, Kenneth Loafman wrote:
>>>>> Yes, I'll do that soon.
>>>>>
>>>>> On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:
>>>>>
>>>>>> Thanks for the merge!
>>>>>>
>>>>>> Could you push a new 0.7 release with this patch included?
>>>>>> --
>>>>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>>>>>> You are subscribed to branch lp:duplicity/0.7-series.
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>>>> You are subscribed to branch lp:duplicity/0.7-series.
>>>>
>>>
>>
>>
>> --
>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>> You are subscribed to branch lp:duplicity/0.7-series.
>>
>

Kenneth Loafman (kenneth-loafman) wrote :

ede,

It's there now. Was unable to test. My python3 install needs to be fixed.

Please test and let me know.

...Ken

On Sun, May 5, 2019 at 6:33 AM edso <email address hidden> wrote:

> Ken,
>
> probably local at your box still? it's not merged afaics
> https://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/changes/
>
> ..ede
>
> On 04.05.2019 18:30, Kenneth Loafman wrote:
> > ede,
> >
> > Manual merge is done. Should have it out in a couple of days.
> >
> > ...Ken
> >
> >
> > On Sat, May 4, 2019 at 5:06 AM edso <email address hidden> wrote:
> >
> >> maybe Yajo wants to help wrt. this issue in 0.8? Yajo got the time?
> ..ede/
> >> duply.net
> >>
> >> On 03.05.2019 21:39, Kenneth Loafman wrote:
> >>> Hi ede,
> >>>
> >>> I'm working on it now. Have to do some of the changes manually for
> 0.8.
> >>> Lots of changes since 0.7!
> >>>
> >>> ...Ken
> >>>
> >>> On Mon, Apr 29, 2019 at 7:19 AM edso <email address hidden> wrote:
> >>>
> >>>> hey Ken,
> >>>>
> >>>> it's not committed to 0.8 so far. please do not forget.
> >>>>
> >>>> also i'd say we make 'partial' default from 0.8 on, as argumented in
> the
> >>>> merge ticket. ..ede
> >>>>
> >>>> On 29.04.2019 14:06, Kenneth Loafman wrote:
> >>>>> Yes, I'll do that soon.
> >>>>>
> >>>>> On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:
> >>>>>
> >>>>>> Thanks for the merge!
> >>>>>>
> >>>>>> Could you push a new 0.7 release with this patch included?
> >>>>>> --
> >>>>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> >>>>>> You are subscribed to branch lp:duplicity/0.7-series.
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> >>>> You are subscribed to branch lp:duplicity/0.7-series.
> >>>>
> >>>
> >>
> >>
> >> --
> >> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> >> You are subscribed to branch lp:duplicity/0.7-series.
> >>
> >
>
>
> --
> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> You are subscribed to branch lp:duplicity/0.7-series.
>

edso (ed.so) wrote :

unfortunately no time, no setup. Yajo, would you mind? ..ede

On 05.05.2019 14:21, Kenneth Loafman wrote:
> ede,
>
> It's there now. Was unable to test. My python3 install needs to be fixed.
>
> Please test and let me know.
>
> ...Ken
>
> On Sun, May 5, 2019 at 6:33 AM edso <email address hidden> wrote:
>
>> Ken,
>>
>> probably local at your box still? it's not merged afaics
>> https://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/changes/
>>
>> ..ede
>>
>> On 04.05.2019 18:30, Kenneth Loafman wrote:
>>> ede,
>>>
>>> Manual merge is done. Should have it out in a couple of days.
>>>
>>> ...Ken
>>>
>>>
>>> On Sat, May 4, 2019 at 5:06 AM edso <email address hidden> wrote:
>>>
>>>> maybe Yajo wants to help wrt. this issue in 0.8? Yajo got the time?
>> ..ede/
>>>> duply.net
>>>>
>>>> On 03.05.2019 21:39, Kenneth Loafman wrote:
>>>>> Hi ede,
>>>>>
>>>>> I'm working on it now. Have to do some of the changes manually for
>> 0.8.
>>>>> Lots of changes since 0.7!
>>>>>
>>>>> ...Ken
>>>>>
>>>>> On Mon, Apr 29, 2019 at 7:19 AM edso <email address hidden> wrote:
>>>>>
>>>>>> hey Ken,
>>>>>>
>>>>>> it's not committed to 0.8 so far. please do not forget.
>>>>>>
>>>>>> also i'd say we make 'partial' default from 0.8 on, as argumented in
>> the
>>>>>> merge ticket. ..ede
>>>>>>
>>>>>> On 29.04.2019 14:06, Kenneth Loafman wrote:
>>>>>>> Yes, I'll do that soon.
>>>>>>>
>>>>>>> On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:
>>>>>>>
>>>>>>>> Thanks for the merge!
>>>>>>>>
>>>>>>>> Could you push a new 0.7 release with this patch included?
>>>>>>>> --
>>>>>>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>>>>>>>> You are subscribed to branch lp:duplicity/0.7-series.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>>>>>> You are subscribed to branch lp:duplicity/0.7-series.
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>>>> You are subscribed to branch lp:duplicity/0.7-series.
>>>>
>>>
>>
>>
>> --
>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>> You are subscribed to branch lp:duplicity/0.7-series.
>>
>

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'bin/duplicity'
2--- bin/duplicity 2018-09-28 13:55:53 +0000
3+++ bin/duplicity 2019-04-24 07:53:33 +0000
4@@ -1020,7 +1020,7 @@
5 _("Rerun command with --force option to actually delete."))
6
7
8-def sync_archive():
9+def sync_archive(col_stats):
10 """
11 Synchronize local archive manifest file and sig chains to remote archives.
12 Copy missing files from remote to local as needed to make sure the local
13@@ -1031,6 +1031,27 @@
14 """
15 suffixes = [".g", ".gpg", ".z", ".gz", ".part"]
16
17+ def is_needed(filename):
18+ """Indicates if the metadata file should be synced.
19+
20+ In full sync mode, or if there's a collection misbehavior, all files
21+ are needed.
22+
23+ Otherwise, only the metadata for the target chain needs sync.
24+ """
25+ if globals.metadata_sync_mode == "full":
26+ return True
27+ assert globals.metadata_sync_mode == "partial"
28+ parsed = file_naming.parse(filename)
29+ try:
30+ target_chain = col_stats.get_backup_chain_at_time(
31+ globals.restore_time or dup_time.curtime)
32+ except collections.CollectionsError:
33+ # With zero or multiple chains at this time, do a full sync
34+ return True
35+ return parsed.end_time >= target_chain.start_time and \
36+ parsed.start_time <= target_chain.end_time
37+
38 def get_metafiles(filelist):
39 """
40 Return metafiles of interest from the file list.
41@@ -1180,7 +1201,7 @@
42 # already have a local partial. The local partial will already be
43 # complete in this case (seems we got interrupted before we could move
44 # it to its final location).
45- if key not in local_keys and key not in local_partials:
46+ if key not in local_keys and key not in local_partials and is_needed(key):
47 local_missing.append(remote_metafiles[key])
48
49 for key in local_keys:
50@@ -1414,15 +1435,15 @@
51 # check for disk space and available file handles
52 check_resources(action)
53
54- # check archive synch with remote, fix if needed
55- if action not in ["collection-status"]:
56- sync_archive()
57-
58 # get current collection status
59 col_stats = collections.CollectionsStatus(globals.backend,
60 globals.archive_dir,
61 action).set_values()
62
63+ # check archive synch with remote, fix if needed
64+ if action not in ["collection-status"]:
65+ sync_archive(col_stats)
66+
67 while True:
68 # if we have to clean up the last partial, then col_stats are invalidated
69 # and we have to start the process all over again until clean.
70@@ -1488,7 +1509,7 @@
71 elif action == "remove-all-but-n-full" or action == "remove-all-inc-of-but-n-full":
72 remove_all_but_n_full(col_stats)
73 elif action == "sync":
74- sync_archive()
75+ sync_archive(col_stats)
76 else:
77 assert action == "inc" or action == "full", action
78 # the passphrase for full and inc is used by --sign-key
79
80=== modified file 'bin/duplicity.1'
81--- bin/duplicity.1 2019-01-26 16:37:54 +0000
82+++ bin/duplicity.1 2019-04-24 07:53:33 +0000
83@@ -882,6 +882,14 @@
84 .BR "A NOTE ON SSL CERTIFICATE VERIFICATION" .
85
86 .TP
87+.BI "--metadata-sync-mode " mode
88+This option defaults to 'full', but you can set it to 'partial'
89+to avoid syncing metadata for backup chains that you are not going to use.
90+This saves time when restoring for the first time, and lets you restore an
91+old backup that was encrypted with a different passphrase by supplying only
92+the target passphrase.
93+
94+.TP
95 .BI "--tempdir " directory
96 Use this existing directory for duplicity temporary files instead of
97 the system default, which is usually the /tmp directory. This option
98
99=== modified file 'duplicity/commandline.py'
100--- duplicity/commandline.py 2018-02-01 21:17:08 +0000
101+++ duplicity/commandline.py 2019-04-24 07:53:33 +0000
102@@ -491,6 +491,11 @@
103 callback=lambda o, s, v, p: (setattr(p.values, o.dest, True),
104 old_fn_deprecation(s)))
105
106+ # Sync only required metadata
107+ parser.add_option("--metadata-sync-mode",
108+ default="full",
109+ choices=("full", "partial"))
110+
111 # Level of Redundancy in % for Par2 files
112 parser.add_option("--par2-redundancy", type="int", metavar=_("number"))
113
114
115=== modified file 'duplicity/globals.py'
116--- duplicity/globals.py 2018-02-01 21:17:08 +0000
117+++ duplicity/globals.py 2019-04-24 07:53:33 +0000
118@@ -225,6 +225,9 @@
119 # Can be changed with a command line argument.
120 imap_mailbox = "INBOX"
121
122+# Sync all metadata by default
123+metadata_sync_mode = "full"
124+
125 # Whether the old filename format is in effect.
126 old_filenames = False
127

Subscribers

People subscribed via source and target branches