Possible data loss when restarting in the middle of a deleted file

This bug was fixed in the package duplicity - 0.6.22-1ubuntu3

---------------
duplicity (0.6.22-1ubuntu3) trusty; urgency=low

  * debian/patches/04-dont-skip-first-chunk-on-restart.patch:
    - When restarting a backup, if the file we were in the middle of
      backing up is now deleted, don't skip the first 65k chunk of the
      next file. Patch backported from upstream trunk. LP: #1252484
 -- Michael Terry <email address hidden> Mon, 18 Nov 2013 16:51:52 -0500

Hello Michael, or anyone else affected,

Accepted duplicity into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/duplicity/0.6.21-0ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

With the version in proposed the testcase in the description passed. Tested on Ubuntu Saucy (13.10) using duplicity 0.6.21-0ubuntu4.1

Thanks, Miklos!

Hello Michael, or anyone else affected,

Accepted duplicity into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/duplicity/0.6.21-0ubuntu1.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Michael, or anyone else affected,

Accepted duplicity into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/duplicity/0.6.19-0ubuntu2.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Michael, or anyone else affected,

Accepted duplicity into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/duplicity/0.6.18-0ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Michael, or anyone else affected,

Accepted duplicity into lucid-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/duplicity/0.6.08b-0ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug was fixed in the package duplicity - 0.6.21-0ubuntu4.1

---------------
duplicity (0.6.21-0ubuntu4.1) saucy; urgency=low

  * debian/patches/03-dont-skip-first-chunk-on-restart.dpatch:
    - When restarting a backup, if the file we were in the middle of
      backing up is now deleted, don't skip the first 65k chunk of the
      next file. Patch backported from upstream trunk. LP: #1252484
 -- Michael Terry <email address hidden> Mon, 18 Nov 2013 18:40:41 -0500

I should note for anyone experiencing any of the symptoms of this bug when restoring [1], this fix does not let you get back the 65k chunk of data that this bug lost. But duplicity trunk contains some additional fixes that should let you restore the rest of your data. That is, duplicity trunk no longer throws an exception when encountering the symptoms of this bug. It will continue to restore the data not affected by this bug.

So while you can't experience full relief, duplicity trunk will let you recover most things.

[1] It can cause various exceptions depending on how/when this bug happened; see the duplicate bugs.

I believe i'm still experiencing the problems.

I have a set of backups from a machine which started with Ubuntu 12.04 and was upgraded a few weeks ago to 13.10. The backups were created via deja-dup. I'm trying to restore on a Machine Running Linux Mint ( ~Ubuntu 13.10) with duplicity 0.6.21-0ubuntu4.1.

Here's what i get when i try to restore my backup manually (duplicity, not deja-dup):
python: ERROR: (rs_file_copy_cb) unexpected eof on fd122
python: ERROR: (rs_job_complete) patch job failed: unexpected end of input
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1414, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1407, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1341, in main
    restore(col_stats)
  File "/usr/bin/duplicity", line 635, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 526, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 499, in integrate_patch_iters
    final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 479, in patch_seq2ropath
    misc.copyfileobj( current_file, tempfp )
  File "/usr/lib/python2.7/dist-packages/duplicity/misc.py", line 166, in copyfileobj
    buf = infp.read(blocksize)
  File "/usr/lib/python2.7/dist-packages/duplicity/librsync.py", line 80, in read
    self._add_to_outbuf_once()
  File "/usr/lib/python2.7/dist-packages/duplicity/librsync.py", line 94, in _add_to_outbuf_once
    raise librsyncError(str(e))
librsyncError: librsync error 103 while in patch cycle

deja-dup detects that there's a problem and bails out. If i try to restore via duplicity, i get this message and the application hangs.
A ps -ef ef shows that there's the duplicity process, which deja-dup started, is still lingering around. My current process is idling.

Did i miss something?

Lennart, see my comment #11. This fix/SRU is for the root cause when making a backup, preventing the data loss in the first place.

But if you are trying to restore a backup affected by the bug (as you are), the traceback you see when restoring is worked-around by a separate fix in duplicity trunk. You still will not be able to correctly restore the file missing its 65k chunk, but duplicity trunk does prevent the restore from bailing out.

I just tried a backup and restore, encrypting the backup. Failed again
on the restore, in exactly the same way as far as I can tell.

At this point I simply use Grsync to run a complete backup to an
encrypted drive. It works well enough.

On 12/15/2013 04:56 PM, Michael Terry wrote:
> Lennart, see my comment #11. This fix/SRU is for the root cause when
> making a backup, preventing the data loss in the first place.
>
> But if you are trying to restore a backup affected by the bug (as you
> are), the traceback you see when restoring is worked-around by a
> separate fix in duplicity trunk. You still will not be able to
> correctly restore the file missing its 65k chunk, but duplicity trunk
> does prevent the restore from bailing out.
>

Dan, simply backing up with a fixed duplicity will not retroactively fix the bug. That is, if you have a backup chain that is already affected by the bug, adding more incremental backups to the chain won't be able to recover the previous missing 65k chunk. We're just ensuring that future backups won't be affected.

I tried this with a brand-new backup, not an old one.

Dan

On 12/17/2013 11:53 AM, Michael Terry wrote:
> Dan, simply backing up with a fixed duplicity will not retroactively fix
> the bug. That is, if you have a backup chain that is already affected
> by the bug, adding more incremental backups to the chain won't be able
> to recover the previous missing 65k chunk. We're just ensuring that
> future backups won't be affected.
>

Dan, if you are getting an error using the patched version of duplicity, can you please file a new bug with the logs and details the bug form asks for? It may be this bug, it may be something else. I'll triage separately.

OK, I'll do this when I get home from a trip.

Upside is I'm learning a lot about how rsync works!

Dan

On 12/18/2013 09:45 AM, Michael Terry wrote:
> Dan, if you are getting an error using the patched version of duplicity,
> can you please file a new bug with the logs and details the bug form
> asks for? It may be this bug, it may be something else. I'll triage
> separately.
>

It's been a while, so I've verified for 10.04 myself.

And same for 12.04.

This bug was fixed in the package duplicity - 0.6.08b-0ubuntu2.3

---------------
duplicity (0.6.08b-0ubuntu2.3) lucid; urgency=low

  * debian/patches/08-dont-skip-first-chunk-on-restart.dpatch:
    - When restarting a backup, if the file we were in the middle of
      backing up is now deleted, don't skip the first 65k chunk of the
      next file. Patch backported from upstream trunk. LP: #1252484
 -- Michael Terry <email address hidden> Tue, 19 Nov 2013 10:58:49 -0500

This bug was fixed in the package duplicity - 0.6.18-0ubuntu3.4

---------------
duplicity (0.6.18-0ubuntu3.4) precise; urgency=low

  * debian/patches/11-dont-skip-first-chunk-on-restart.dpatch:
    - When restarting a backup, if the file we were in the middle of
      backing up is now deleted, don't skip the first 65k chunk of the
      next file. Patch backported from upstream trunk. LP: #1252484
 -- Michael Terry <email address hidden> Tue, 19 Nov 2013 10:16:52 -0500

quantal has seen the end of its life and is no longer receiving any updates. Marking the quantal task for this ticket as "Won't Fix".

affected by this bug with duplicity 0.7.06

I'm not affected with duplicity 0.7.06 on Ubuntu 16.04.