Merge lp:~mterry/duplicity/manifest-oddities into lp:duplicity/0.6-series

Proposed by Michael Terry on 2013-11-16
Status: Merged
Merged at revision: 934
Proposed branch: lp:~mterry/duplicity/manifest-oddities
Merge into: lp:duplicity/0.6-series
Diff against target: 64 lines (+37/-1)
2 files modified
duplicity/manifest.py (+12/-1)
testing/tests/restarttest.py (+25/-0)
To merge this branch: bzr merge lp:~mterry/duplicity/manifest-oddities
Reviewer Review Type Date Requested Status
duplicity-team 2013-11-16 Pending
Review via email: mp+195460@code.launchpad.net

Description of the change

We may accidentally end up with an oddly inconsistent manifest like so:

Volume 1
Volume 2
Volume 3
Volume 2

As did get reported recently on the mailing list: http://lists.nongnu.org/archive/html/duplicity-talk/2013-11/msg00009.html

One way this can happen (the only way?) is if you back up, then duplicity gets interrupted between writing the manifest and uploading the volume. Then, when restarted, there is no longer enough data to create as many volumes as existed previously.

This situation can cause an exception when trying to restart the backup.

This branch fixes it by deleting any excess volume information encountered when loading in the manifest. We discard volume with higher numbers than the last one read.

To post a comment you must log in.
932. By Michael Terry on 2013-11-16

Fix test comments

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'duplicity/manifest.py'
2--- duplicity/manifest.py 2011-06-17 06:21:42 +0000
3+++ duplicity/manifest.py 2013-11-16 02:23:17 +0000
4@@ -177,12 +177,23 @@
5 next_vi_string_regexp = re.compile("(^|\\n)(volume\\s.*?)"
6 "(\\nvolume\\s|$)", re.I | re.S)
7 starting_s_index = 0
8+ highest_vol = 0
9+ latest_vol = 0
10 while 1:
11 match = next_vi_string_regexp.search(s[starting_s_index:])
12 if not match:
13 break
14- self.add_volume_info(VolumeInfo().from_string(match.group(2)))
15+ vi = VolumeInfo().from_string(match.group(2))
16+ self.add_volume_info(vi)
17+ highest_vol = max(highest_vol, vi.volume_number)
18+ latest_vol = vi.volume_number
19 starting_s_index += match.end(2)
20+ # If we restarted after losing some remote volumes, the highest volume
21+ # seen may be higher than the last volume recorded. That is, the
22+ # manifest could contain "vol1, vol2, vol3, vol2." If so, we don't
23+ # want to keep vol3's info.
24+ for i in range(latest_vol + 1, highest_vol + 1):
25+ self.del_volume_info(i)
26 return self
27
28 def __eq__(self, other):
29
30=== modified file 'testing/tests/restarttest.py'
31--- testing/tests/restarttest.py 2013-01-07 16:13:29 +0000
32+++ testing/tests/restarttest.py 2013-11-16 02:23:17 +0000
33@@ -446,6 +446,31 @@
34 assert not os.system("diff %s/file1 testfiles/restore_out/file1" % source)
35 assert not os.system("diff %s/z testfiles/restore_out/z" % source)
36
37+ def test_changed_source_dangling_manifest_volume(self):
38+ """
39+ If we restart but find remote volumes missing, we can easily end up
40+ with a manifest that lists "vol1, vol2, vol3, vol2", leaving a dangling
41+ vol3. Make sure we can gracefully handle that. This will only happen
42+ if the source data changes to be small enough to not create a vol3 on
43+ restart.
44+ """
45+ source = 'testfiles/largefiles'
46+ self.make_largefiles(count=5, size=1)
47+ # intentionally interrupt initial backup
48+ try:
49+ self.backup("full", source, options = ["--vol 1", "--fail 3"])
50+ self.fail()
51+ except CmdError, e:
52+ self.assertEqual(30, e.exit_status)
53+ # now delete the last volume on remote end and some source files
54+ assert not os.system("rm testfiles/output/duplicity-full*vol3.difftar*")
55+ assert not os.system("rm %s/file[2345]" % source)
56+ assert not os.system("echo hello > %s/z" % source)
57+ # finish backup
58+ self.backup("full", source)
59+ # and verify we can restore
60+ self.restore()
61+
62
63 # Note that this class duplicates all the tests in RestartTest
64 class RestartTestWithoutEncryption(RestartTest):

Subscribers

People subscribed via source and target branches

to all changes: