Merge lp:~mterry/duplicity/catch-seq-copy-error into lp:duplicity/0.6-series

Proposed by Michael Terry on 2013-09-17
Status: Merged
Merged at revision: 929
Proposed branch: lp:~mterry/duplicity/catch-seq-copy-error
Merge into: lp:duplicity/0.6-series
Diff against target: 40 lines (+15/-6)
1 file modified
duplicity/ (+15/-6)
To merge this branch: bzr merge lp:~mterry/duplicity/catch-seq-copy-error
Reviewer Review Type Date Requested Status
Peter Fenton (community) Needs Fixing on 2013-10-21
duplicity-team 2013-09-17 Pending
Review via email:

Description of the change

When converting a sequence of patches to a ROPath, sometimes duplicity wants to collapse the current set of patches into a temporary file for librsync's benefit, because it has encountered a non-true-file-object and librsync only works on true-file-objects.

When this happens, if an error occurs, the exception is not caught and will terminate duplicity.

If that same error happened lazily -- as it normally does if duplicity doesn't decide to collapse patches into a temporary file -- the exception is caught, like many others, by IterTreeReducer's handlers and is merely logged while the rest of duplicity chugs along.

Ideally these two cases would be handled more similarly. It makes bugs like bug 662442 much worse than they need to be because a problem with a single file prevents restoring the whole backup.

This branch captures common errors that occur during the patch-collapsing, logs them, and moves on. Behavior is still not *identical* between the two cases [1], but it's much better.

[1] For example, if you are restoring only one file, the fact that this branch treats files with collapsing problems as non-existent means that duplicity will say it didn't find any files to restore -- whereas the normal path will recognize that it at least found a file. This didn't seem like a huge problem and would have been a little awkward to fix (maybe have seq2ropath return a ticking-time-bomb ROPath that would raise its error next time it was used? Seemed hackier than this branch).

To post a comment you must log in.
930. By Michael Terry on 2013-09-18

Wrap whole seq2ropath function in a general try/except rather than a more focused approach -- any problem patching a file should be ignored but logged before moving on

Michael Terry (mterry) wrote :

I've updated this branch to use a simpler, more generic approach.

*Any* exception when running patch_seq2ropath should be ignored (though logged) and duplicity should move on. This covers the two asserts in that function (bug 1155345 and bug 720525) as well as errors that happen during file copying (bug 662442).

This doesn't solve any of those bugs, obviously. But it makes them less severe.

I'm getting the feeling that all three bugs have the same root cause: a missing sequence object. 662442 can be explained by a missing patch in the sequence, and the other two can be explained by a missing first snapshot...

931. By Michael Terry on 2013-09-19

Fix some spacing

Peter Fenton (ped-fenton) wrote :

Hi Michael
this is my first ever Linux Forum question. I have the (bug 662442) issue where Deja Dup crashes with the following comment.

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1411, in <module>
  File "/usr/bin/duplicity", line 1404, in with_tempdir
  File "/usr/bin/duplicity", line 1338, in main
  File "/usr/bin/duplicity", line 632, in restore
  File "/usr/lib/python2.7/dist-packages/duplicity/", line 526, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/", line 499, in integrate_patch_iters
    final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
  File "/usr/lib/python2.7/dist-packages/duplicity/", line 479, in patch_seq2ropath
    misc.copyfileobj( current_file, tempfp )
  File "/usr/lib/python2.7/dist-packages/duplicity/", line 166, in copyfileobj
    buf =
  File "/usr/lib/python2.7/dist-packages/duplicity/", line 80, in read
  File "/usr/lib/python2.7/dist-packages/duplicity/", line 94, in _add_to_outbuf_once
    raise librsyncError(str(e))
librsyncError: librsync error 103 while in patch cycle

I understand that there is a file that Deja Dup does not like, so is stopping the whole process, but I am not sure how to apply the .diff file so I can have Deja Dup ignore the error and move on.

Sorry if I am being daft about this, but I have no idea how to solve this and am very new to Ubuntu. This backup is my only copy of my data as the oridinal hard drive died:(

Any help would be appreciated as I feel I am going round in circles.


review: Needs Fixing

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'duplicity/'
2--- duplicity/ 2012-09-11 14:11:39 +0000
3+++ duplicity/ 2013-09-19 00:27:14 +0000
4@@ -460,10 +460,12 @@
5 def patch_seq2ropath( patch_seq ):
6 """Apply the patches in patch_seq, return single ropath"""
7 first = patch_seq[0]
8- assert first.difftype != "diff", patch_seq
9+ assert first.difftype != "diff", "First patch in sequence " \
10+ "%s was a diff" % patch_seq
11 if not first.isreg():
12 # No need to bother with data if not regular file
13- assert len( patch_seq ) == 1, len( patch_seq )
14+ assert len(patch_seq) == 1, "Patch sequence isn't regular, but " \
15+ "has %d entries" % len(patch_seq)
16 return first.get_ropath()
18 current_file = "rb" )
19@@ -496,10 +498,17 @@
20 """
21 collated = collate_iters( iter_list )
22 for patch_seq in collated:
23- final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
24- if final_ropath.exists():
25- # otherwise final patch was delete
26- yield final_ropath
27+ try:
28+ final_ropath = patch_seq2ropath(normalize_ps(patch_seq))
29+ if final_ropath.exists():
30+ # otherwise final patch was delete
31+ yield final_ropath
32+ except Exception, e:
33+ filename = patch_seq[-1].get_ropath().get_relative_path()
34+ log.Warn(_("Error '%s' patching %s") %
35+ (str(e), filename),
36+ log.WarningCode.cannot_process,
37+ util.escape(filename))
39 def tarfiles2rop_iter( tarfile_list, restrict_index=() ):
40 """Integrate tarfiles of diffs into single ROPath iter


People subscribed via source and target branches

to all changes: