Duplicity

Merge lp:~yajo/duplicity/duplicity into lp:~duplicity-team/duplicity/0.7-series

  1. duplicity
  2. Merge into 0.7-series
Proposed by Yajo
Status: Merged
Merged at revision: 1373
Proposed branch: lp:~yajo/duplicity/duplicity
Merge into: lp:~duplicity-team/duplicity/0.7-series
Diff against target: 126 lines (+44/-7)
4 files modified
bin/duplicity (+28/-7)
bin/duplicity.1 (+8/-0)
duplicity/commandline.py (+5/-0)
duplicity/globals.py (+3/-0)
To merge this branch: bzr merge lp:~yajo/duplicity/duplicity
Reviewer Review Type Date Requested Status
edso Approve
Review via email: mp+366358@code.launchpad.net

Description of the change

Support partial metadata sync.

Fixes bug #1823858 by letting the user to choose partial syncing. Only the metadata for the target chain will be downloaded. If older (or newer) chains are encrypted with a different passphrase, the user will be able to restore to a given time by supplying only the passphrase for the chain selected by the `--restore-time` option when using this new option.

A side effect is that using this flag reduces dramatically the sync time when moving files from one to another location, in cases where big amounts of chains are found.

To post a comment you must log in.
Revision history for this message
Aaron Whitehouse (aaron-whitehouse) wrote :

Thank you for your work, Yajo.

Is there a reason that you have requested to merge this into the 0.7 series instead of the main duplicity (0.8 series)?

0.7 is not really suited for new features at this point.

Revision history for this message
edso (ed.so) wrote :

hi Yajo,

the approach seems sound. however here some small issues, apart from the merge request against the old stable branch.

1. leftover debugger import
+ import pudb; pudb.set_trace()

2. new default sync mode should be partial, as it is faster and transfers only needed files

3.
+from gettext import gettext as _
looks redundant. gettext should be imported by 'duplicity/__init__.py' already.

nice that you updated the manpage accordingly already!

..ede/duply.net

review: Needs Fixing
lp:~yajo/duplicity/duplicity updated
1373. By Yajo

Support partial metadata sync.

Fixes bug #1823858 by letting the user to choose partial syncing. Only the metadata for the target chain will be downloaded. If older (or newer) chains are encrypted with a different passphrase, the user will be able to restore to a given time by supplying only the passphrase for the chain selected by the `--restore-time` option when using this new option.

A side effect is that using this flag reduces dramatically the sync time when moving files from one to another location, in cases where big amounts of chains are found.

Revision history for this message
Yajo (yajo) wrote :

Hi there, concerns 1 and 3 fixed.

About number 2, the problem is that I really need this patch in 0.7 because I'm hitting bug #1823858 it in production, whereas 0.8 is not yet released. This is half feature half bugfix, but TBH what motivated me most to push the MR is the bugfix.

I thought that a good approach would be to add the feature but disable it by default (as I did) in 0.7, but in 0.8 we could default to the partial sync mode because, as you say, it is has most benefits.

Or can you think a better solution?

Revision history for this message
edso (ed.so) wrote :

> Hi there, concerns 1 and 3 fixed.

thx

> About number 2, the problem is that I really need this patch in 0.7 because I'm hitting bug #1823858 it in production, whereas 0.8 is not yet released. This is half feature half bugfix, but TBH what motivated me most to push the MR is the bugfix.

sounds reasonable. disabled by default even 0.7 could benefit.

> I thought that a good approach would be to add the feature but disable it by default (as I did) in 0.7, but in 0.8 we could default to the partial sync mode because, as you say, it is has most benefits.
>
> Or can you think a better solution?

nope. perfectly valid reasoning. in the end it's up to Ken the maintainer.
@Ken: what do you say?

..ede/duply.net

review: Approve
Revision history for this message
Yajo (yajo) wrote :

Thanks for the merge!

Could you push a new 0.7 release with this patch included?

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

Yes, I'll do that soon.

On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:

> Thanks for the merge!
>
> Could you push a new 0.7 release with this patch included?
> --
> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> You are subscribed to branch lp:duplicity/0.7-series.
>

Revision history for this message
edso (ed.so) wrote :

hey Ken,

it's not committed to 0.8 so far. please do not forget.

also i'd say we make 'partial' default from 0.8 on, as argumented in the merge ticket. ..ede

On 29.04.2019 14:06, Kenneth Loafman wrote:
> Yes, I'll do that soon.
>
> On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:
>
>> Thanks for the merge!
>>
>> Could you push a new 0.7 release with this patch included?
>> --
>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>> You are subscribed to branch lp:duplicity/0.7-series.
>>
>

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

Hi ede,

I'm working on it now. Have to do some of the changes manually for 0.8.
Lots of changes since 0.7!

...Ken

On Mon, Apr 29, 2019 at 7:19 AM edso <email address hidden> wrote:

> hey Ken,
>
> it's not committed to 0.8 so far. please do not forget.
>
> also i'd say we make 'partial' default from 0.8 on, as argumented in the
> merge ticket. ..ede
>
> On 29.04.2019 14:06, Kenneth Loafman wrote:
> > Yes, I'll do that soon.
> >
> > On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:
> >
> >> Thanks for the merge!
> >>
> >> Could you push a new 0.7 release with this patch included?
> >> --
> >> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> >> You are subscribed to branch lp:duplicity/0.7-series.
> >>
> >
>
>
> --
> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> You are subscribed to branch lp:duplicity/0.7-series.
>

Revision history for this message
edso (ed.so) wrote :

maybe Yajo wants to help wrt. this issue in 0.8? Yajo got the time? ..ede/duply.net

On 03.05.2019 21:39, Kenneth Loafman wrote:
> Hi ede,
>
> I'm working on it now. Have to do some of the changes manually for 0.8.
> Lots of changes since 0.7!
>
> ...Ken
>
> On Mon, Apr 29, 2019 at 7:19 AM edso <email address hidden> wrote:
>
>> hey Ken,
>>
>> it's not committed to 0.8 so far. please do not forget.
>>
>> also i'd say we make 'partial' default from 0.8 on, as argumented in the
>> merge ticket. ..ede
>>
>> On 29.04.2019 14:06, Kenneth Loafman wrote:
>>> Yes, I'll do that soon.
>>>
>>> On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:
>>>
>>>> Thanks for the merge!
>>>>
>>>> Could you push a new 0.7 release with this patch included?
>>>> --
>>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>>>> You are subscribed to branch lp:duplicity/0.7-series.
>>>>
>>>
>>
>>
>> --
>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>> You are subscribed to branch lp:duplicity/0.7-series.
>>
>

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

ede,

Manual merge is done. Should have it out in a couple of days.

...Ken

On Sat, May 4, 2019 at 5:06 AM edso <email address hidden> wrote:

> maybe Yajo wants to help wrt. this issue in 0.8? Yajo got the time? ..ede/
> duply.net
>
> On 03.05.2019 21:39, Kenneth Loafman wrote:
> > Hi ede,
> >
> > I'm working on it now. Have to do some of the changes manually for 0.8.
> > Lots of changes since 0.7!
> >
> > ...Ken
> >
> > On Mon, Apr 29, 2019 at 7:19 AM edso <email address hidden> wrote:
> >
> >> hey Ken,
> >>
> >> it's not committed to 0.8 so far. please do not forget.
> >>
> >> also i'd say we make 'partial' default from 0.8 on, as argumented in the
> >> merge ticket. ..ede
> >>
> >> On 29.04.2019 14:06, Kenneth Loafman wrote:
> >>> Yes, I'll do that soon.
> >>>
> >>> On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:
> >>>
> >>>> Thanks for the merge!
> >>>>
> >>>> Could you push a new 0.7 release with this patch included?
> >>>> --
> >>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> >>>> You are subscribed to branch lp:duplicity/0.7-series.
> >>>>
> >>>
> >>
> >>
> >> --
> >> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> >> You are subscribed to branch lp:duplicity/0.7-series.
> >>
> >
>
>
> --
> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> You are subscribed to branch lp:duplicity/0.7-series.
>

Revision history for this message
edso (ed.so) wrote :

Ken,

probably local at your box still? it's not merged afaics https://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/changes/

..ede

On 04.05.2019 18:30, Kenneth Loafman wrote:
> ede,
>
> Manual merge is done. Should have it out in a couple of days.
>
> ...Ken
>
>
> On Sat, May 4, 2019 at 5:06 AM edso <email address hidden> wrote:
>
>> maybe Yajo wants to help wrt. this issue in 0.8? Yajo got the time? ..ede/
>> duply.net
>>
>> On 03.05.2019 21:39, Kenneth Loafman wrote:
>>> Hi ede,
>>>
>>> I'm working on it now. Have to do some of the changes manually for 0.8.
>>> Lots of changes since 0.7!
>>>
>>> ...Ken
>>>
>>> On Mon, Apr 29, 2019 at 7:19 AM edso <email address hidden> wrote:
>>>
>>>> hey Ken,
>>>>
>>>> it's not committed to 0.8 so far. please do not forget.
>>>>
>>>> also i'd say we make 'partial' default from 0.8 on, as argumented in the
>>>> merge ticket. ..ede
>>>>
>>>> On 29.04.2019 14:06, Kenneth Loafman wrote:
>>>>> Yes, I'll do that soon.
>>>>>
>>>>> On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:
>>>>>
>>>>>> Thanks for the merge!
>>>>>>
>>>>>> Could you push a new 0.7 release with this patch included?
>>>>>> --
>>>>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>>>>>> You are subscribed to branch lp:duplicity/0.7-series.
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>>>> You are subscribed to branch lp:duplicity/0.7-series.
>>>>
>>>
>>
>>
>> --
>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>> You are subscribed to branch lp:duplicity/0.7-series.
>>
>

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

ede,

It's there now. Was unable to test. My python3 install needs to be fixed.

Please test and let me know.

...Ken

On Sun, May 5, 2019 at 6:33 AM edso <email address hidden> wrote:

> Ken,
>
> probably local at your box still? it's not merged afaics
> https://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/changes/
>
> ..ede
>
> On 04.05.2019 18:30, Kenneth Loafman wrote:
> > ede,
> >
> > Manual merge is done. Should have it out in a couple of days.
> >
> > ...Ken
> >
> >
> > On Sat, May 4, 2019 at 5:06 AM edso <email address hidden> wrote:
> >
> >> maybe Yajo wants to help wrt. this issue in 0.8? Yajo got the time?
> ..ede/
> >> duply.net
> >>
> >> On 03.05.2019 21:39, Kenneth Loafman wrote:
> >>> Hi ede,
> >>>
> >>> I'm working on it now. Have to do some of the changes manually for
> 0.8.
> >>> Lots of changes since 0.7!
> >>>
> >>> ...Ken
> >>>
> >>> On Mon, Apr 29, 2019 at 7:19 AM edso <email address hidden> wrote:
> >>>
> >>>> hey Ken,
> >>>>
> >>>> it's not committed to 0.8 so far. please do not forget.
> >>>>
> >>>> also i'd say we make 'partial' default from 0.8 on, as argumented in
> the
> >>>> merge ticket. ..ede
> >>>>
> >>>> On 29.04.2019 14:06, Kenneth Loafman wrote:
> >>>>> Yes, I'll do that soon.
> >>>>>
> >>>>> On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:
> >>>>>
> >>>>>> Thanks for the merge!
> >>>>>>
> >>>>>> Could you push a new 0.7 release with this patch included?
> >>>>>> --
> >>>>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> >>>>>> You are subscribed to branch lp:duplicity/0.7-series.
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>> --
> >>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> >>>> You are subscribed to branch lp:duplicity/0.7-series.
> >>>>
> >>>
> >>
> >>
> >> --
> >> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> >> You are subscribed to branch lp:duplicity/0.7-series.
> >>
> >
>
>
> --
> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
> You are subscribed to branch lp:duplicity/0.7-series.
>

Revision history for this message
edso (ed.so) wrote :

unfortunately no time, no setup. Yajo, would you mind? ..ede

On 05.05.2019 14:21, Kenneth Loafman wrote:
> ede,
>
> It's there now. Was unable to test. My python3 install needs to be fixed.
>
> Please test and let me know.
>
> ...Ken
>
> On Sun, May 5, 2019 at 6:33 AM edso <email address hidden> wrote:
>
>> Ken,
>>
>> probably local at your box still? it's not merged afaics
>> https://bazaar.launchpad.net/~duplicity-team/duplicity/0.8-series/changes/
>>
>> ..ede
>>
>> On 04.05.2019 18:30, Kenneth Loafman wrote:
>>> ede,
>>>
>>> Manual merge is done. Should have it out in a couple of days.
>>>
>>> ...Ken
>>>
>>>
>>> On Sat, May 4, 2019 at 5:06 AM edso <email address hidden> wrote:
>>>
>>>> maybe Yajo wants to help wrt. this issue in 0.8? Yajo got the time?
>> ..ede/
>>>> duply.net
>>>>
>>>> On 03.05.2019 21:39, Kenneth Loafman wrote:
>>>>> Hi ede,
>>>>>
>>>>> I'm working on it now. Have to do some of the changes manually for
>> 0.8.
>>>>> Lots of changes since 0.7!
>>>>>
>>>>> ...Ken
>>>>>
>>>>> On Mon, Apr 29, 2019 at 7:19 AM edso <email address hidden> wrote:
>>>>>
>>>>>> hey Ken,
>>>>>>
>>>>>> it's not committed to 0.8 so far. please do not forget.
>>>>>>
>>>>>> also i'd say we make 'partial' default from 0.8 on, as argumented in
>> the
>>>>>> merge ticket. ..ede
>>>>>>
>>>>>> On 29.04.2019 14:06, Kenneth Loafman wrote:
>>>>>>> Yes, I'll do that soon.
>>>>>>>
>>>>>>> On Mon, Apr 29, 2019 at 2:44 AM Yajo <email address hidden> wrote:
>>>>>>>
>>>>>>>> Thanks for the merge!
>>>>>>>>
>>>>>>>> Could you push a new 0.7 release with this patch included?
>>>>>>>> --
>>>>>>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>>>>>>>> You are subscribed to branch lp:duplicity/0.7-series.
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>>>>>> You are subscribed to branch lp:duplicity/0.7-series.
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>>>> You are subscribed to branch lp:duplicity/0.7-series.
>>>>
>>>
>>
>>
>> --
>> https://code.launchpad.net/~yajo/duplicity/duplicity/+merge/366358
>> You are subscribed to branch lp:duplicity/0.7-series.
>>
>

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'bin/duplicity'
--- bin/duplicity 2018-09-28 13:55:53 +0000
+++ bin/duplicity 2019-04-24 07:53:33 +0000
@@ -1020,7 +1020,7 @@
1020 _("Rerun command with --force option to actually delete."))1020 _("Rerun command with --force option to actually delete."))
10211021
10221022
1023def sync_archive():1023def sync_archive(col_stats):
1024 """1024 """
1025 Synchronize local archive manifest file and sig chains to remote archives.1025 Synchronize local archive manifest file and sig chains to remote archives.
1026 Copy missing files from remote to local as needed to make sure the local1026 Copy missing files from remote to local as needed to make sure the local
@@ -1031,6 +1031,27 @@
1031 """1031 """
1032 suffixes = [".g", ".gpg", ".z", ".gz", ".part"]1032 suffixes = [".g", ".gpg", ".z", ".gz", ".part"]
10331033
1034 def is_needed(filename):
1035 """Indicates if the metadata file should be synced.
1036
1037 In full sync mode, or if there's a collection misbehavior, all files
1038 are needed.
1039
1040 Otherwise, only the metadata for the target chain needs sync.
1041 """
1042 if globals.metadata_sync_mode == "full":
1043 return True
1044 assert globals.metadata_sync_mode == "partial"
1045 parsed = file_naming.parse(filename)
1046 try:
1047 target_chain = col_stats.get_backup_chain_at_time(
1048 globals.restore_time or dup_time.curtime)
1049 except collections.CollectionsError:
1050 # With zero or multiple chains at this time, do a full sync
1051 return True
1052 return parsed.end_time >= target_chain.start_time and \
1053 parsed.start_time <= target_chain.end_time
1054
1034 def get_metafiles(filelist):1055 def get_metafiles(filelist):
1035 """1056 """
1036 Return metafiles of interest from the file list.1057 Return metafiles of interest from the file list.
@@ -1180,7 +1201,7 @@
1180 # already have a local partial. The local partial will already be1201 # already have a local partial. The local partial will already be
1181 # complete in this case (seems we got interrupted before we could move1202 # complete in this case (seems we got interrupted before we could move
1182 # it to its final location).1203 # it to its final location).
1183 if key not in local_keys and key not in local_partials:1204 if key not in local_keys and key not in local_partials and is_needed(key):
1184 local_missing.append(remote_metafiles[key])1205 local_missing.append(remote_metafiles[key])
11851206
1186 for key in local_keys:1207 for key in local_keys:
@@ -1414,15 +1435,15 @@
1414 # check for disk space and available file handles1435 # check for disk space and available file handles
1415 check_resources(action)1436 check_resources(action)
14161437
1417 # check archive synch with remote, fix if needed
1418 if action not in ["collection-status"]:
1419 sync_archive()
1420
1421 # get current collection status1438 # get current collection status
1422 col_stats = collections.CollectionsStatus(globals.backend,1439 col_stats = collections.CollectionsStatus(globals.backend,
1423 globals.archive_dir,1440 globals.archive_dir,
1424 action).set_values()1441 action).set_values()
14251442
1443 # check archive synch with remote, fix if needed
1444 if action not in ["collection-status"]:
1445 sync_archive(col_stats)
1446
1426 while True:1447 while True:
1427 # if we have to clean up the last partial, then col_stats are invalidated1448 # if we have to clean up the last partial, then col_stats are invalidated
1428 # and we have to start the process all over again until clean.1449 # and we have to start the process all over again until clean.
@@ -1488,7 +1509,7 @@
1488 elif action == "remove-all-but-n-full" or action == "remove-all-inc-of-but-n-full":1509 elif action == "remove-all-but-n-full" or action == "remove-all-inc-of-but-n-full":
1489 remove_all_but_n_full(col_stats)1510 remove_all_but_n_full(col_stats)
1490 elif action == "sync":1511 elif action == "sync":
1491 sync_archive()1512 sync_archive(col_stats)
1492 else:1513 else:
1493 assert action == "inc" or action == "full", action1514 assert action == "inc" or action == "full", action
1494 # the passphrase for full and inc is used by --sign-key1515 # the passphrase for full and inc is used by --sign-key
14951516
=== modified file 'bin/duplicity.1'
--- bin/duplicity.1 2019-01-26 16:37:54 +0000
+++ bin/duplicity.1 2019-04-24 07:53:33 +0000
@@ -882,6 +882,14 @@
882.BR "A NOTE ON SSL CERTIFICATE VERIFICATION" .882.BR "A NOTE ON SSL CERTIFICATE VERIFICATION" .
883883
884.TP884.TP
885.BI "--metadata-sync-mode " mode
886This option defaults to 'full', but you can set it to 'partial'
887to avoid syncing metadata for backup chains that you are not going to use.
888This saves time when restoring for the first time, and lets you restore an
889old backup that was encrypted with a different passphrase by supplying only
890the target passphrase.
891
892.TP
885.BI "--tempdir " directory893.BI "--tempdir " directory
886Use this existing directory for duplicity temporary files instead of894Use this existing directory for duplicity temporary files instead of
887the system default, which is usually the /tmp directory. This option895the system default, which is usually the /tmp directory. This option
888896
=== modified file 'duplicity/commandline.py'
--- duplicity/commandline.py 2018-02-01 21:17:08 +0000
+++ duplicity/commandline.py 2019-04-24 07:53:33 +0000
@@ -491,6 +491,11 @@
491 callback=lambda o, s, v, p: (setattr(p.values, o.dest, True),491 callback=lambda o, s, v, p: (setattr(p.values, o.dest, True),
492 old_fn_deprecation(s)))492 old_fn_deprecation(s)))
493493
494 # Sync only required metadata
495 parser.add_option("--metadata-sync-mode",
496 default="full",
497 choices=("full", "partial"))
498
494 # Level of Redundancy in % for Par2 files499 # Level of Redundancy in % for Par2 files
495 parser.add_option("--par2-redundancy", type="int", metavar=_("number"))500 parser.add_option("--par2-redundancy", type="int", metavar=_("number"))
496501
497502
=== modified file 'duplicity/globals.py'
--- duplicity/globals.py 2018-02-01 21:17:08 +0000
+++ duplicity/globals.py 2019-04-24 07:53:33 +0000
@@ -225,6 +225,9 @@
225# Can be changed with a command line argument.225# Can be changed with a command line argument.
226imap_mailbox = "INBOX"226imap_mailbox = "INBOX"
227227
228# Sync all metadata by default
229metadata_sync_mode = "full"
230
228# Whether the old filename format is in effect.231# Whether the old filename format is in effect.
229old_filenames = False232old_filenames = False
230233

Subscribers

People subscribed via source and target branches