Merge lp:~mterry/duplicity/catch-seq-copy-error into lp:duplicity/0.6

[Michael Terry]

When converting a sequence of patches to a ROPath, sometimes duplicity wants to collapse the current set of patches into a temporary file for librsync's benefit, because it has encountered a non-true-file-object and librsync only works on true-file-objects.

When this happens, if an error occurs, the exception is not caught and will terminate duplicity.

If that same error happened lazily -- as it normally does if duplicity doesn't decide to collapse patches into a temporary file -- the exception is caught, like many others, by IterTreeReducer's handlers and is merely logged while the rest of duplicity chugs along.

Ideally these two cases would be handled more similarly. It makes bugs like bug 662442 much worse than they need to be because a problem with a single file prevents restoring the whole backup.

This branch captures common errors that occur during the patch-collapsing, logs them, and moves on. Behavior is still not *identical* between the two cases [1], but it's much better.

[1] For example, if you are restoring only one file, the fact that this branch treats files with collapsing problems as non-existent means that duplicity will say it didn't find any files to restore -- whereas the normal path will recognize that it at least found a file. This didn't seem like a huge problem and would have been a little awkward to fix (maybe have seq2ropath return a ticking-time-bomb ROPath that would raise its error next time it was used? Seemed hackier than this branch).

[lp:~mterry/duplicity/catch-seq-copy-error]

930. By Michael Terry on 2013-09-18

Wrap whole seq2ropath function in a general try/except rather than a more focused approach -- any problem patching a file should be ignored but logged before moving on

[Michael Terry (mterry)]

I've updated this branch to use a simpler, more generic approach.

*Any* exception when running patch_seq2ropath should be ignored (though logged) and duplicity should move on. This covers the two asserts in that function (bug 1155345 and bug 720525) as well as errors that happen during file copying (bug 662442).

This doesn't solve any of those bugs, obviously. But it makes them less severe.

I'm getting the feeling that all three bugs have the same root cause: a missing sequence object. 662442 can be explained by a missing patch in the sequence, and the other two can be explained by a missing first snapshot...

[lp:~mterry/duplicity/catch-seq-copy-error]

931. By Michael Terry on 2013-09-19

Fix some spacing

[Peter Fenton (ped-fenton)]

Hi Michael
this is my first ever Linux Forum question. I have the (bug 662442) issue where Deja Dup crashes with the following comment.

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1411, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1404, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1338, in main
    restore(col_stats)
  File "/usr/bin/duplicity", line 632, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 526, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 499, in integrate_patch_iters
    final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 479, in patch_seq2ropath
    misc.copyfileobj( current_file, tempfp )
  File "/usr/lib/python2.7/dist-packages/duplicity/misc.py", line 166, in copyfileobj
    buf = infp.read(blocksize)
  File "/usr/lib/python2.7/dist-packages/duplicity/librsync.py", line 80, in read
    self._add_to_outbuf_once()
  File "/usr/lib/python2.7/dist-packages/duplicity/librsync.py", line 94, in _add_to_outbuf_once
    raise librsyncError(str(e))
librsyncError: librsync error 103 while in patch cycle

I understand that there is a file that Deja Dup does not like, so is stopping the whole process, but I am not sure how to apply the .diff file so I can have Deja Dup ignore the error and move on.

Sorry if I am being daft about this, but I have no idea how to solve this and am very new to Ubuntu. This backup is my only copy of my data as the oridinal hard drive died:(

Any help would be appreciated as I feel I am going round in circles.

Pete:)

review: Needs Fixing

=== modified file 'duplicity/patchdir.py'
--- duplicity/patchdir.py	2012-09-11 14:11:39 +0000
+++ duplicity/patchdir.py	2013-09-19 00:27:14 +0000
@@ -460,10 +460,12 @@
 def patch_seq2ropath( patch_seq ):
     """Apply the patches in patch_seq, return single ropath"""
     first = patch_seq[0]
-    assert first.difftype != "diff", patch_seq
+    assert first.difftype != "diff", "First patch in sequence " \
+                                     "%s was a diff" % patch_seq
     if not first.isreg():
         # No need to bother with data if not regular file
-        assert len( patch_seq ) == 1, len( patch_seq )
+        assert len(patch_seq) == 1, "Patch sequence isn't regular, but " \
+                                    "has %d entries" % len(patch_seq)
         return first.get_ropath()
 
     current_file = first.open( "rb" )
@@ -496,10 +498,17 @@
     """
     collated = collate_iters( iter_list )
     for patch_seq in collated:
-        final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
-        if final_ropath.exists():
-            # otherwise final patch was delete
-            yield final_ropath
+        try:
+            final_ropath = patch_seq2ropath(normalize_ps(patch_seq))
+            if final_ropath.exists():
+                # otherwise final patch was delete
+                yield final_ropath
+        except Exception, e:
+            filename = patch_seq[-1].get_ropath().get_relative_path()
+            log.Warn(_("Error '%s' patching %s") % 
+                     (str(e), filename),
+                     log.WarningCode.cannot_process,
+                     util.escape(filename))
 
 def tarfiles2rop_iter( tarfile_list, restrict_index=() ):
     """Integrate tarfiles of diffs into single ROPath iter