Merge lp:~dawgfoto/duplicity/replicate into lp:~duplicity-team/duplicity/0.8-series

Proposed by Martin Nowak on 2017-04-20
Status: Merged
Merged at revision: 1209
Proposed branch: lp:~dawgfoto/duplicity/replicate
Merge into: lp:~duplicity-team/duplicity/0.8-series
Diff against target: 380 lines (+190/-24)
5 files modified
bin/duplicity (+116/-2)
bin/duplicity.1 (+17/-0)
duplicity/collections.py (+12/-3)
duplicity/commandline.py (+36/-19)
duplicity/file_naming.py (+9/-0)
To merge this branch: bzr merge lp:~dawgfoto/duplicity/replicate
Reviewer Review Type Date Requested Status
duplicity-team 2017-04-20 Pending
Review via email: mp+322836@code.launchpad.net

Description of the change

Initial request for feedback.

Add replicate command to replicate a backup (or backup sets older than a given time) to another backend, leveraging duplicity's backend and compression/encryption infrastructure.

To post a comment you must log in.
lp:~dawgfoto/duplicity/replicate updated on 2017-05-04
1191. By Kenneth Loafman on 2017-04-22

2017-04-20 Kenneth Loafman <email address hidden>

    * Fixed bug #1680682 with patch supplied from Dave Allan
      - Only specify --pinentry-mode=loopback when --use-agent is not specified
    * Fixed man page that had 'cancel' instead of 'loopback' for pinentry mode
    * Fixed bug #1684312 with suggestion from Wade Rossman
      - Use shutil.copyfile instead of os.system('cp ...')
      - Should reduce overhead of os.system() memory usage.

1192. By Kenneth Loafman on 2017-04-23

* Merged in lp:~dernils/duplicity/testing
  - Fixed minor stuff in requirements.txt.
  - Added a Dockerfile for testing.
  - Minor changes to README files.
  - Added README-TESTING with some information on testing.

1193. By Kenneth Loafman on 2017-04-23

* Merged in lp:~dernils/duplicity/documentation
  - Minor changes to README-REPO, README-TESTING
  - Also redo the changes to requirements.txt and Dockerfile

1194. By ken on 2017-04-25

* Add rdiff install and newline at end of file.

1195. By Kenneth Loafman on 2017-04-25

Move pep8 and pylint to requirements.

1196. By Kenneth Loafman on 2017-04-26

Whoops, deleted too much. Add rdiff again.

1197. By Kenneth Loafman on 2017-04-26

Merged in lp:~dernils/duplicity/Dockerfile
Fixed variable name change in commandline.py

1198. By Kenneth Loafman on 2017-04-27

More changes for testing:
- keep gpg1 version for future testing
- some changes for debugging functional tests
- add gpg-agent.conf with allow-loopback-pinentry

1199. By ken on 2017-04-27

A little reorg, just keeping pip things together.

1200. By ken on 2017-04-28

Quick fix for bug #1680682 and gnupg v1, add missing comma.

1201. By ken on 2017-04-28

- Simplify Dockerfile per https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/
- Add a .dockerignore file
- Uncomment some debug prints
- quick fix for bug #1680682 and gnupg v1, add missing comma

1202. By ken on 2017-04-28

Move branch duplicity up the food chain.

1203. By ken on 2017-04-28

Add test user and swap to non-priviledged.

1204. By ken on 2017-04-29

- Remove dependencies we did not need

1205. By ken on 2017-04-30

Merged in lp:~dernils/duplicity/Dockerfile
- separated requirements into requirements for duplicity (in requirements.txt) and for testing (in tox.ini)

1206. By ken on 2017-04-30

Add libffi-dev back. My bad.

1207. By ken on 2017-04-30

You need tox to run tox. Doh!

1208. By ken on 2017-05-03

We need tzdata (timezone data).

1209. By Kenneth Loafman on 2017-05-04

* Merged in lp:~dawgfoto/duplicity/replicate
  - Add replicate command to replicate a backup (or backup
    sets older than a given time) to another backend, leveraging
    duplicity's backend and compression/encryption infrastructure.
* Fixed some incoming PyLint and PEP-8 errors.

Aaron Whitehouse (aaron-whitehouse) wrote :

Many thanks Martin!

Could you please submit some tests that ensure your code keeps working as expected? There should be an example of most things you want to test and I would be happy to help navigate them to the extent that I can.

Just an end-to-end functional test would be a great start.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'bin/duplicity'
--- bin/duplicity 2017-03-02 22:38:47 +0000
+++ bin/duplicity 2017-04-24 16:35:20 +0000
@@ -28,6 +28,7 @@
28# any suggestions.28# any suggestions.
2929
30import duplicity.errors30import duplicity.errors
31import copy
31import gzip32import gzip
32import os33import os
33import platform34import platform
@@ -1006,6 +1007,116 @@
1006 "\n" + chain_times_str(chainlist) + "\n" +1007 "\n" + chain_times_str(chainlist) + "\n" +
1007 _("Rerun command with --force option to actually delete."))1008 _("Rerun command with --force option to actually delete."))
10081009
1010def replicate():
1011 """
1012 Replicate backup files from one remote to another, possibly encrypting or adding parity.
1013
1014 @rtype: void
1015 @return: void
1016 """
1017 time = globals.restore_time or dup_time.curtime
1018 src_stats = collections.CollectionsStatus(globals.src_backend, None).set_values(sig_chain_warning=None)
1019 tgt_stats = collections.CollectionsStatus(globals.backend, None).set_values(sig_chain_warning=None)
1020
1021 src_list = globals.src_backend.list()
1022 tgt_list = globals.backend.list()
1023
1024 src_chainlist = src_stats.get_signature_chains(local=False, filelist=src_list)[0]
1025 tgt_chainlist = tgt_stats.get_signature_chains(local=False, filelist=tgt_list)[0]
1026 sorted(src_chainlist, key=lambda chain: chain.start_time)
1027 sorted(tgt_chainlist, key=lambda chain: chain.start_time)
1028 if not src_chainlist:
1029 log.Notice(_("No old backup sets found."))
1030 return
1031 for src_chain in src_chainlist:
1032 try:
1033 tgt_chain = filter(lambda chain: chain.start_time == src_chain.start_time, tgt_chainlist)[0]
1034 except IndexError:
1035 tgt_chain = None
1036
1037 tgt_sigs = map(file_naming.parse, tgt_chain.get_filenames()) if tgt_chain else []
1038 for src_sig_filename in src_chain.get_filenames():
1039 src_sig = file_naming.parse(src_sig_filename)
1040 if not (src_sig.time or src_sig.end_time) < time:
1041 continue
1042 try:
1043 tgt_sigs.remove(src_sig)
1044 log.Info(_("Signature %s already replicated") % (src_sig_filename,))
1045 continue
1046 except ValueError:
1047 pass
1048 if src_sig.type == 'new-sig':
1049 dup_time.setprevtime(src_sig.start_time)
1050 dup_time.setcurtime(src_sig.time or src_sig.end_time)
1051 log.Notice(_("Replicating %s.") % (src_sig_filename,))
1052 fileobj = globals.src_backend.get_fileobj_read(src_sig_filename)
1053 filename = file_naming.get(src_sig.type, encrypted=globals.encryption, gzipped=globals.compression)
1054 tdp = dup_temp.new_tempduppath(file_naming.parse(filename))
1055 tmpobj = tdp.filtered_open(mode='wb')
1056 util.copyfileobj(fileobj, tmpobj) # decrypt, compress, (re)-encrypt
1057 fileobj.close()
1058 tmpobj.close()
1059 globals.backend.put(tdp, filename)
1060 tdp.delete()
1061
1062 src_chainlist = src_stats.get_backup_chains(filename_list = src_list)[0]
1063 tgt_chainlist = tgt_stats.get_backup_chains(filename_list = tgt_list)[0]
1064 sorted(src_chainlist, key=lambda chain: chain.start_time)
1065 sorted(tgt_chainlist, key=lambda chain: chain.start_time)
1066 for src_chain in src_chainlist:
1067 try:
1068 tgt_chain = filter(lambda chain: chain.start_time == src_chain.start_time, tgt_chainlist)[0]
1069 except IndexError:
1070 tgt_chain = None
1071
1072 tgt_sets = tgt_chain.get_all_sets() if tgt_chain else []
1073 for src_set in src_chain.get_all_sets():
1074 if not src_set.get_time() < time:
1075 continue
1076 try:
1077 tgt_sets.remove(src_set)
1078 log.Info(_("Backupset %s already replicated") % (src_set.remote_manifest_name,))
1079 continue
1080 except ValueError:
1081 pass
1082 if src_set.type == 'inc':
1083 dup_time.setprevtime(src_set.start_time)
1084 dup_time.setcurtime(src_set.get_time())
1085 rmf = src_set.get_remote_manifest()
1086 mf_filename = file_naming.get(src_set.type, manifest=True)
1087 mf_tdp = dup_temp.new_tempduppath(file_naming.parse(mf_filename))
1088 mf = manifest.Manifest(fh=mf_tdp.filtered_open(mode='wb'))
1089 for i, filename in src_set.volume_name_dict.iteritems():
1090 log.Notice(_("Replicating %s.") % (filename,))
1091 fileobj = restore_get_enc_fileobj(globals.src_backend, filename, rmf.volume_info_dict[i])
1092 filename = file_naming.get(src_set.type, i, encrypted=globals.encryption, gzipped=globals.compression)
1093 tdp = dup_temp.new_tempduppath(file_naming.parse(filename))
1094 tmpobj = tdp.filtered_open(mode='wb')
1095 util.copyfileobj(fileobj, tmpobj) # decrypt, compress, (re)-encrypt
1096 fileobj.close()
1097 tmpobj.close()
1098 globals.backend.put(tdp, filename)
1099
1100 vi = copy.copy(rmf.volume_info_dict[i])
1101 vi.set_hash("SHA1", gpg.get_hash("SHA1", tdp))
1102 mf.add_volume_info(vi)
1103
1104 tdp.delete()
1105
1106 mf.fh.close()
1107 # incremental GPG writes hang on close, so do any encryption here at once
1108 mf_fileobj = mf_tdp.filtered_open_with_delete(mode='rb')
1109 mf_final_filename = file_naming.get(src_set.type, manifest=True, encrypted=globals.encryption, gzipped=globals.compression)
1110 mf_final_tdp = dup_temp.new_tempduppath(file_naming.parse(mf_final_filename))
1111 mf_final_fileobj = mf_final_tdp.filtered_open(mode='wb')
1112 util.copyfileobj(mf_fileobj, mf_final_fileobj) # compress, encrypt
1113 mf_fileobj.close()
1114 mf_final_fileobj.close()
1115 globals.backend.put(mf_final_tdp, mf_final_filename)
1116 mf_final_tdp.delete()
1117
1118 globals.src_backend.close()
1119 globals.backend.close()
10091120
1010def sync_archive(decrypt):1121def sync_archive(decrypt):
1011 """1122 """
@@ -1408,8 +1519,9 @@
1408 check_resources(action)1519 check_resources(action)
14091520
1410 # check archive synch with remote, fix if needed1521 # check archive synch with remote, fix if needed
1411 decrypt = action not in ["collection-status"]1522 if not action == "replicate":
1412 sync_archive(decrypt)1523 decrypt = action not in ["collection-status"]
1524 sync_archive(decrypt)
14131525
1414 # get current collection status1526 # get current collection status
1415 col_stats = collections.CollectionsStatus(globals.backend,1527 col_stats = collections.CollectionsStatus(globals.backend,
@@ -1483,6 +1595,8 @@
1483 remove_all_but_n_full(col_stats)1595 remove_all_but_n_full(col_stats)
1484 elif action == "sync":1596 elif action == "sync":
1485 sync_archive(True)1597 sync_archive(True)
1598 elif action == "replicate":
1599 replicate()
1486 else:1600 else:
1487 assert action == "inc" or action == "full", action1601 assert action == "inc" or action == "full", action
1488 # the passphrase for full and inc is used by --sign-key1602 # the passphrase for full and inc is used by --sign-key
14891603
=== modified file 'bin/duplicity.1'
--- bin/duplicity.1 2017-04-22 19:30:28 +0000
+++ bin/duplicity.1 2017-04-24 16:35:20 +0000
@@ -48,6 +48,10 @@
48.I [options] [--force] [--extra-clean]48.I [options] [--force] [--extra-clean]
49target_url49target_url
5050
51.B duplicity replicate
52.I [options] [--time time]
53source_url target_url
54
51.SH DESCRIPTION55.SH DESCRIPTION
52Duplicity incrementally backs up files and folders into56Duplicity incrementally backs up files and folders into
53tar-format volumes encrypted with GnuPG and places them to a57tar-format volumes encrypted with GnuPG and places them to a
@@ -243,6 +247,19 @@
243.I --force247.I --force
244will be needed to delete the files instead of just listing them.248will be needed to delete the files instead of just listing them.
245249
250.TP
251.BI "replicate " "[--time time] <source_url> <target_url>"
252Replicate backup sets from source to target backend. Files will be
253(re)-encrypted and (re)-compressed depending on normal backend
254options. Signatures and volumes will not get recomputed, thus options like
255.BI --volsize
256or
257.BI --max-blocksize
258have no effect.
259When
260.I --time time
261is given, only backup sets older than time will be replicated.
262
246.SH OPTIONS263.SH OPTIONS
247264
248.TP265.TP
249266
=== modified file 'duplicity/collections.py'
--- duplicity/collections.py 2017-02-27 13:18:57 +0000
+++ duplicity/collections.py 2017-04-24 16:35:20 +0000
@@ -294,6 +294,15 @@
294 """294 """
295 return len(self.volume_name_dict.keys())295 return len(self.volume_name_dict.keys())
296296
297 def __eq__(self, other):
298 """
299 Return whether this backup set is equal to other
300 """
301 return self.type == other.type and \
302 self.time == other.time and \
303 self.start_time == other.start_time and \
304 self.end_time == other.end_time and \
305 len(self) == len(other)
297306
298class BackupChain:307class BackupChain:
299 """308 """
@@ -642,7 +651,7 @@
642 u"-----------------",651 u"-----------------",
643 _("Connecting with backend: %s") %652 _("Connecting with backend: %s") %
644 (self.backend.__class__.__name__,),653 (self.backend.__class__.__name__,),
645 _("Archive dir: %s") % (util.ufn(self.archive_dir_path.name),)]654 _("Archive dir: %s") % (util.ufn(self.archive_dir_path.name if self.archive_dir_path else 'None'),)]
646655
647 l.append("\n" +656 l.append("\n" +
648 ngettext("Found %d secondary backup chain.",657 ngettext("Found %d secondary backup chain.",
@@ -697,7 +706,7 @@
697 len(backend_filename_list))706 len(backend_filename_list))
698707
699 # get local filename list708 # get local filename list
700 local_filename_list = self.archive_dir_path.listdir()709 local_filename_list = self.archive_dir_path.listdir() if self.archive_dir_path else []
701 log.Debug(ngettext("%d file exists in cache",710 log.Debug(ngettext("%d file exists in cache",
702 "%d files exist in cache",711 "%d files exist in cache",
703 len(local_filename_list)) %712 len(local_filename_list)) %
@@ -894,7 +903,7 @@
894 if filelist is not None:903 if filelist is not None:
895 return filelist904 return filelist
896 elif local:905 elif local:
897 return self.archive_dir_path.listdir()906 return self.archive_dir_path.listdir() if self.archive_dir_path else []
898 else:907 else:
899 return self.backend.list()908 return self.backend.list()
900909
901910
=== modified file 'duplicity/commandline.py'
--- duplicity/commandline.py 2017-02-27 13:18:57 +0000
+++ duplicity/commandline.py 2017-04-24 16:35:20 +0000
@@ -54,6 +54,7 @@
54collection_status = None # Will be set to true if collection-status command given54collection_status = None # Will be set to true if collection-status command given
55cleanup = None # Set to true if cleanup command given55cleanup = None # Set to true if cleanup command given
56verify = None # Set to true if verify command given56verify = None # Set to true if verify command given
57replicate = None # Set to true if replicate command given
5758
58commands = ["cleanup",59commands = ["cleanup",
59 "collection-status",60 "collection-status",
@@ -65,6 +66,7 @@
65 "remove-all-inc-of-but-n-full",66 "remove-all-inc-of-but-n-full",
66 "restore",67 "restore",
67 "verify",68 "verify",
69 "replicate"
68 ]70 ]
6971
7072
@@ -236,7 +238,7 @@
236def parse_cmdline_options(arglist):238def parse_cmdline_options(arglist):
237 """Parse argument list"""239 """Parse argument list"""
238 global select_opts, select_files, full_backup240 global select_opts, select_files, full_backup
239 global list_current, collection_status, cleanup, remove_time, verify241 global list_current, collection_status, cleanup, remove_time, verify, replicate
240242
241 def set_log_fd(fd):243 def set_log_fd(fd):
242 if fd < 1:244 if fd < 1:
@@ -706,6 +708,9 @@
706 num_expect = 1708 num_expect = 1
707 elif cmd == "verify":709 elif cmd == "verify":
708 verify = True710 verify = True
711 elif cmd == "replicate":
712 replicate = True
713 num_expect = 2
709714
710 if len(args) != num_expect:715 if len(args) != num_expect:
711 command_line_error("Expected %d args, got %d" % (num_expect, len(args)))716 command_line_error("Expected %d args, got %d" % (num_expect, len(args)))
@@ -724,7 +729,12 @@
724 elif len(args) == 1:729 elif len(args) == 1:
725 backend_url = args[0]730 backend_url = args[0]
726 elif len(args) == 2:731 elif len(args) == 2:
727 lpath, backend_url = args_to_path_backend(args[0], args[1]) # @UnusedVariable732 if replicate:
733 if not backend.is_backend_url(args[0]) or not backend.is_backend_url(args[1]):
734 command_line_error("Two URLs expected for replicate.")
735 src_backend_url, backend_url= args[0], args[1]
736 else:
737 lpath, backend_url = args_to_path_backend(args[0], args[1]) # @UnusedVariable
728 else:738 else:
729 command_line_error("Too many arguments")739 command_line_error("Too many arguments")
730740
@@ -899,6 +909,7 @@
899 duplicity remove-older-than %(time)s [%(options)s] %(target_url)s909 duplicity remove-older-than %(time)s [%(options)s] %(target_url)s
900 duplicity remove-all-but-n-full %(count)s [%(options)s] %(target_url)s910 duplicity remove-all-but-n-full %(count)s [%(options)s] %(target_url)s
901 duplicity remove-all-inc-of-but-n-full %(count)s [%(options)s] %(target_url)s911 duplicity remove-all-inc-of-but-n-full %(count)s [%(options)s] %(target_url)s
912 duplicity replicate %(source_url)s %(target_url)s
902913
903""" % dict914""" % dict
904915
@@ -944,7 +955,8 @@
944 remove-older-than <%(time)s> <%(target_url)s>955 remove-older-than <%(time)s> <%(target_url)s>
945 remove-all-but-n-full <%(count)s> <%(target_url)s>956 remove-all-but-n-full <%(count)s> <%(target_url)s>
946 remove-all-inc-of-but-n-full <%(count)s> <%(target_url)s>957 remove-all-inc-of-but-n-full <%(count)s> <%(target_url)s>
947 verify <%(target_url)s> <%(source_dir)s>""" % dict958 verify <%(target_url)s> <%(source_dir)s>
959 replicate <%(source_url)s> <%(target_url)s>""" % dict
948960
949 return msg961 return msg
950962
@@ -1047,7 +1059,7 @@
10471059
1048def check_consistency(action):1060def check_consistency(action):
1049 """Final consistency check, see if something wrong with command line"""1061 """Final consistency check, see if something wrong with command line"""
1050 global full_backup, select_opts, list_current1062 global full_backup, select_opts, list_current, collection_status, cleanup, replicate
10511063
1052 def assert_only_one(arglist):1064 def assert_only_one(arglist):
1053 """Raises error if two or more of the elements of arglist are true"""1065 """Raises error if two or more of the elements of arglist are true"""
@@ -1058,8 +1070,8 @@
1058 assert n <= 1, "Invalid syntax, two conflicting modes specified"1070 assert n <= 1, "Invalid syntax, two conflicting modes specified"
10591071
1060 if action in ["list-current", "collection-status",1072 if action in ["list-current", "collection-status",
1061 "cleanup", "remove-old", "remove-all-but-n-full", "remove-all-inc-of-but-n-full"]:1073 "cleanup", "remove-old", "remove-all-but-n-full", "remove-all-inc-of-but-n-full", "replicate"]:
1062 assert_only_one([list_current, collection_status, cleanup,1074 assert_only_one([list_current, collection_status, cleanup, replicate,
1063 globals.remove_time is not None])1075 globals.remove_time is not None])
1064 elif action == "restore" or action == "verify":1076 elif action == "restore" or action == "verify":
1065 if full_backup:1077 if full_backup:
@@ -1137,22 +1149,27 @@
1137"file:///usr/local". See the man page for more information.""") % (args[0],),1149"file:///usr/local". See the man page for more information.""") % (args[0],),
1138 log.ErrorCode.bad_url)1150 log.ErrorCode.bad_url)
1139 elif len(args) == 2:1151 elif len(args) == 2:
1140 # Figure out whether backup or restore1152 if replicate:
1141 backup, local_pathname = set_backend(args[0], args[1])1153 globals.src_backend = backend.get_backend(args[0])
1142 if backup:1154 globals.backend = backend.get_backend(args[1])
1143 if full_backup:1155 action = "replicate"
1144 action = "full"
1145 else:
1146 action = "inc"
1147 else:1156 else:
1148 if verify:1157 # Figure out whether backup or restore
1149 action = "verify"1158 backup, local_pathname = set_backend(args[0], args[1])
1159 if backup:
1160 if full_backup:
1161 action = "full"
1162 else:
1163 action = "inc"
1150 else:1164 else:
1151 action = "restore"1165 if verify:
1166 action = "verify"
1167 else:
1168 action = "restore"
11521169
1153 process_local_dir(action, local_pathname)1170 process_local_dir(action, local_pathname)
1154 if action in ['full', 'inc', 'verify']:1171 if action in ['full', 'inc', 'verify']:
1155 set_selection()1172 set_selection()
1156 elif len(args) > 2:1173 elif len(args) > 2:
1157 raise AssertionError("this code should not be reachable")1174 raise AssertionError("this code should not be reachable")
11581175
11591176
=== modified file 'duplicity/file_naming.py'
--- duplicity/file_naming.py 2016-06-28 21:03:46 +0000
+++ duplicity/file_naming.py 2017-04-24 16:35:20 +0000
@@ -436,3 +436,12 @@
436 self.encrypted = encrypted # true if gpg encrypted436 self.encrypted = encrypted # true if gpg encrypted
437437
438 self.partial = partial438 self.partial = partial
439
440 def __eq__(self, other):
441 return self.type == other.type and \
442 self.manifest == other.manifest and \
443 self.time == other.time and \
444 self.start_time == other.start_time and \
445 self.end_time == other.end_time and \
446 self.partial == other.partial
447

Subscribers

People subscribed via source and target branches