restore fails with "librsync error 103 while in patch cycle"

I'm also getting this while doing a "duplicity restore --no-encryption". The specific error I see is "python: ERROR: (rs_file_copy_cb) unexpected eof on fd8". This is duplicity 0.6.13.

In my case, one of the incremental backups might have been truncated/interrupted as there were occasional read-errors on the source system in question. However, as this disk has now completely died and this backup is the only one, I'd like to hear if there is any way at all to make duplicity recover from this "unexpected end of input" and continue with the restore operation?

I have this problem during verify as well. I can work around this by specifying subdirectories, for example,

ubuntu@ubuntu:/media/1b[...snip...]a1/home/chris$ duplicity --verbosity info --file-to-restore "Documents/" --archive-dir /media/CHFOO_FAT32/backups/duplicity-cache/ verify file:///media/CHFOO_FAT32/backups/backups-laptop/ Documents/

It might be possible to list out the files (list-current-files) in the backups and iterate through them while specifying the filename to the --file-to-restore argument using a script for files that can't be restored in a specific directory without this error.

Well, I made and attached an emergency hax patch that skips the file—at least I think that's what it does—for those who want it. :)

Attached is an updated hax patch that works for restore as well.

How do you use the patch please?

In regards to https://bugs.launchpad.net/duplicity/+bug/662442/comments/5 you can use the command:

    patch -p0 < ../duplicity_hax_662442.diff

where you are in the source code directory and the patch is one level outside.

I'm attaching an updated hax patch for the latest version from the source code repo if the older patches don't work. It has not been tested as I don't have access to the faulty backup files any more.

You sir are a legend, thank you kindly. I am now restored.

Very smart, you saved my skin, it seems the issue was an uncaught python error, the double try except patch also did the trick for me, avoiding the software to colapse due to that issue, I was already accepting my doomed fate of losing all my software projects.

If it is an usefull information, I'm a Xubuntu 12.10 user; another person with the problem also used Xubuntu(12.04); I never had these backup corruption issues with plain ubuntu, may this be a Xubuntu specific issue? Or maybe it may be an issue elsewere during the backup process corrupted the backup, altought that only happened on Xubuntu as said above.

I wonder if Deja Dup is dangerous to use as backup tool on Xubuntu, version 24.0 and the current unstable presents backup issues sometimes, and what is a very bad, it cannot witstand a single backup file patch error without crumbling(without patch), I think the patch above should be applied to Deja dup permanenlty, together with an usefull gui log at the end of the backup restore showing which files couldn't be patched.

re: #8

That's a strange coincidence. I installed Xubuntu onto my regular Ubuntu install and I used Duplicity for backups.

As I mentioned earlier, I don't have physical access to my broken backups for testing. (I might have actually deleted them; can't remember.) It might be helpful if someone could post logs with and without my patches. I don't have time to actively debug this problem as I'm still in university, but if the devs arn't looking at this and more people have this issue, I'll dedicate some time to this bug.

Patch or no patch, I learned that it's best to use two backup systems at the expense of buying more external hard drives: one native and one cross-platform. (Currently I use Back In Time and Areca.)

I installed pure Xubuntu from the beggining(no Ubuntu first), amd64 on an imac 2011(no backup issues with ubuntu on it thought). I still have my bugged backups set saved. I wonder to where the patch additional except: error log messages are being sent to(currently still running backup restore), as I wanted to see them; maybe I must use duplicity on terminal to see them. Not sure if as the error log line is it would output to the log the faulty files(used to program python games only), knowing the faulty files I can by logic have a notion of what may had corrupted the backups at least on my case, which could be:

1 - One of Xubuntu's constant energy manager errors which sometimes freezes the system for a couple of seconds on imac 2011

2 - During my own software testing, as sometimes I find bugs which freezes the system for around 30 min or at least severly drain system memory for one minute.

3 - Maybe having the file open, and using it while it is being backed up may be a problem

4 - Issues between Deja Dup and Xubuntu(at least on my hardware)

4 - Something else

I also discovered, like yourself, I will need an additional external drive, upon which I will not backup with any software, but manually instead weekly. I will check Back in Time and Areca as I'm interested in alternatives to Deja Dup.

Allright, I saw some of the files which had to be ignored by the above patch; until now, all of them were open when Deja Dup was doing its backup process(a blender file and a python one), very probably their buggy backup was made after the november 15 full deja dup backup I did, which means only their changes compared to the source file were stored(not the full file). I could recover using the last option at Deja Dup 'Worst Case' web page the missing files, thought only the ones from november 15 full backup until now.

The attachment "Emergency hax patch that skips the unpatchable file" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-reviewers team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

full file can be found here if patch gets rejected:
http://www.binrand.com/post/4368177-path-index-dan-duplicity.html

Status changed to 'Confirmed' because the bug affects multiple users.

I'm having the same issue on gentoo

Here's a hint at what might be going wrong. While working on the Python 3 port of duplicity, I started getting this error. It turned out to be because I accidentally closed the patch basis file too soon.

Maybe some code path here is closing a file we need for rsync patching too early.

I have run into the same problem trying to restore my backups.

The error I get is -

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1411, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1404, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1338, in main
    restore(col_stats)
  File "/usr/bin/duplicity", line 632, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 526, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 499, in integrate_patch_iters
    final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 479, in patch_seq2ropath
    misc.copyfileobj( current_file, tempfp )
  File "/usr/lib/python2.7/dist-packages/duplicity/misc.py", line 166, in copyfileobj
    buf = infp.read(blocksize)
  File "/usr/lib/python2.7/dist-packages/duplicity/librsync.py", line 80, in read
    self._add_to_outbuf_once()
  File "/usr/lib/python2.7/dist-packages/duplicity/librsync.py", line 94, in _add_to_outbuf_once
    raise librsyncError(str(e))
librsyncError: librsync error 103 while in patch cycle

I have tried using a more up to date version, tried using an earlier version, but I get the same error every time.

Is there a way I can get the backups back ? the entire backup folder is about 165gb and most of it I'd rather no lose if possible.

I noticed there's a patch mentioned, but in all honesty I'm not sure how to apply it, so I'm willing to give that a go, if someone has an idiot proof guide on how to apply the patch.

And if there's anything else I can try I'd be happy to have a go, really don't want to lose the data.

I'm running linux mint 14 (have previously run mint and used deja dup with no issues) current deja dup version - 26.0~bzr1443.43~quantal1

I second the request on a quick how-to apply the patch for the less knowledgeable people (like my self) encountering the bug.

Sorry, I didn't see comment #17. This comment addresses #17 and #18 and any future readers.

First note, a disclaimer: I am *not* a developer nor maintainer of duplicity.

Ok, first open up a terminal. It should be in the Accessories menu.

Now type the following and hit enter. Do so for any commands listed from now on.

    sudo apt-get install build-essential bzr

Enter your password if prompted and hit yes to install the software. Once it's installed, let's switch the current directory to somewhere obvious like the Desktop folder. Use command:

    cd Desktop

Now download the software. Use command:

    bzr branch lp:duplicity

There should be a duplicty folder on your Desktop now.

Now grab the my hax file patch and put it on the Desktop.

Ok, now switch to the duplicity folder. Run command:

    cd duplicity

Before we apply to patch, we need to switch to older code. Run command:

    bzr revert -r884

Now run the command to patch. The double dot indicates it's saved one level outside the current folder, the Desktop folder. Run command:

    patch -p0 < ../duplicity_hax_662442_for_rev884.diff

It should say "patching file duplicity/patchdir.py".

Now switch to the folder duplicity/ (aka ~/Desktop/duplicity/duplicity). Run command:

    cd duplicity

Compile something. Run command:

    python compilec.py

Now switch back to the parent dir. Run command:

    cd ..

Now duplicity should be ready to run. You can run ./bin/duplicity.

This command will show the version. Run command:

    ./bin/duplicity -V

It will return "$version" which isn't set, but the main thing is that it successfully runs.

Hope this helps.

Thank you for the detailed instructions! I managed to build however I had to install python-dev and librsync-dev as well two get around the errors

Python.h: No such file or directory

and

librsync.h: No such file or directory

The commands are

sudo apt-get install python-dev

and

sudo apt-get install librsync-dev

However I now get this error

Traceback (most recent call last):
  File "./bin/duplicity", line 1403, in <module>
    with_tempdir(main)
  File "./bin/duplicity", line 1396, in with_tempdir
    fn()
  File "./bin/duplicity", line 1276, in main
    globals.archive_dir).set_values()
  File "/home/tantin/Desktop/duplicity/duplicity/collections.py", line 698, in set_values
    self.get_backup_chains(partials + backend_filename_list)
  File "/home/tantin/Desktop/duplicity/duplicity/collections.py", line 821, in get_backup_chains
    map(add_to_sets, filename_list)
  File "/home/tantin/Desktop/duplicity/duplicity/collections.py", line 811, in add_to_sets
    if set.add_filename(filename):
  File "/home/tantin/Desktop/duplicity/duplicity/collections.py", line 93, in add_filename
    self.set_manifest(filename)
  File "/home/tantin/Desktop/duplicity/duplicity/collections.py", line 124, in set_manifest
    remote_filename)
AssertionError: ('duplicity-inc.20130307T152041Z.to.20130404T084858Z.manifest.gpg', 'duplicity-inc.20130307T152041Z.to.20130404T084858Z.manifest')

So the quest to restore continues...

In which directory should i run "patch -p0 < ../duplicity_hax_662442_for_rev884.diff"

I also experience this bug. The backups I'm trying to restore are from my old Ubuntu 13.04 system - but actually one on which I installed the xubuntu-desktop package on top of - and trying to restore to a new computer with "plain" Ubuntu 13.04.

I saw a request for logs from before applying the patch, and afterwards. Below is my "before" log, whereafter I'll apply the patch mentioned above.

Log:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1411, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1404, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1338, in main
    restore(col_stats)
  File "/usr/bin/duplicity", line 632, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 526, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 499, in integrate_patch_iters
    final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 479, in patch_seq2ropath
    misc.copyfileobj( current_file, tempfp )
  File "/usr/lib/python2.7/dist-packages/duplicity/misc.py", line 166, in copyfileobj
    buf = infp.read(blocksize)
  File "/usr/lib/python2.7/dist-packages/duplicity/librsync.py", line 80, in read
    self._add_to_outbuf_once()
  File "/usr/lib/python2.7/dist-packages/duplicity/librsync.py", line 94, in _add_to_outbuf_once
    raise librsyncError(str(e))
librsyncError: librsync error 103 while in patch cycle

Just to give an update:

I used the patch Chris indicated / attached to manually patch patchdir.py. I wasn't able to compile Duplicity, but when I tried restoring my backup again it ran past the error - but I'm experiencing a different error now (below)!

In terms of Adriano's question asking in which directory to run the patch: From the errors above, I found the patchdir.py file in /usr/lib/python2.7/dist-packages/duplicity.

The error I now get is a different error, I think unrelated to this specific bug report, but here is the new error anyway:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1411, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1404, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1338, in main
    restore(col_stats)
  File "/usr/bin/duplicity", line 632, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 532, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 505, in integrate_patch_iters
    final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 463, in patch_seq2ropath
    assert first.difftype != "diff", patch_seq
AssertionError: [(('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg), (('home', 'stuart', '.config', 'chromium', 'Default', 'History Index 2013-04') reg)]

If anyone has a copy of a backup with this bug that they'd be willing to share, please let me know. We haven't been able to reproduce this error yet. :(

Michael, all,

I'm trying to restore one of my backups that DejaDup saved to my NAS. Until now I have been unsuccessfull in doing so.
My machine is Ubuntu 12.10 and I haven't tried the patching thing yet.

here is the CLI output.... but I'm happy to dig further upon advice.
$ duplicity --file-to-restore home/seb ftp://backup@10.1.1.6/backup/deja-dup/snoopy2 /media/seb/ext4_backup/recover/20121213
NcFTP version is 3.2.5
Synchronising remote metadata to local cache...
GnuPG passphrase:
Copying duplicity-full-signatures.20121213T154732Z.sigtar.gpg to local cache.
Copying duplicity-full.20121213T154732Z.manifest.gpg to local cache.
Copying duplicity-inc.20121213T154732Z.to.20121220T074758Z.manifest.gpg to local cache.
Copying duplicity-inc.20121220T074758Z.to.20121227T000634Z.manifest.gpg to local cache.
Copying duplicity-inc.20121227T000634Z.to.20130103T070657Z.manifest.gpg to local cache.
Copying duplicity-inc.20130103T070657Z.to.20130110T064523Z.manifest.gpg to local cache.
Copying duplicity-inc.20130110T064523Z.to.20130126T065714Z.manifest.gpg to local cache.
Copying duplicity-inc.20130126T065714Z.to.20130131T105745Z.manifest.gpg to local cache.
Copying duplicity-inc.20130131T105745Z.to.20130208T113225Z.manifest.gpg to local cache.
Copying duplicity-new-signatures.20121213T154732Z.to.20121220T074758Z.sigtar.gpg to local cache.
Copying duplicity-new-signatures.20121220T074758Z.to.20121227T000634Z.sigtar.gpg to local cache.
Copying duplicity-new-signatures.20121227T000634Z.to.20130103T070657Z.sigtar.gpg to local cache.
Copying duplicity-new-signatures.20130103T070657Z.to.20130110T064523Z.sigtar.gpg to local cache.
Copying duplicity-new-signatures.20130110T064523Z.to.20130126T065714Z.sigtar.gpg to local cache.
Copying duplicity-new-signatures.20130126T065714Z.to.20130131T105745Z.sigtar.gpg to local cache.
Copying duplicity-new-signatures.20130131T105745Z.to.20130208T113225Z.sigtar.gpg to local cache.
Warning, found incomplete backup sets, probably left from aborted session
Last full backup date: Thu Dec 13 16:47:32 2012
python: ERROR: (rs_file_copy_cb) unexpected eof on fd5
python: ERROR: (rs_job_complete) patch job failed: unexpected end of input
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1412, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1405, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1339, in main
    restore(col_stats)
  File "/usr/bin/duplicity", line 630, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 522, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 495, in integrate_patch_iters
    final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 475, in patch_seq2ropath
    misc.copyfileobj( current_file, tempfp )
  File "/usr/lib/python2.7/dist-packages/duplicity/misc.py", line 166, in copyfileobj
    buf = infp.read(blocksize)
  File "/usr/lib/python2.7/dist-packages/duplicity/librsync.py", line 80, in read
    self._add_to_outbuf_once()
  File "/usr/lib/py...

I managed to compile but it still doesn't work. This really unfortunate since this is my only backup of all documents (well partially that's in the cloud), photos and music. I updated it before formatting and now I'm pretty much fucked.

Below's the output.

tantin@Ubook:~/Desktop/duplicity$ sudo ./bin/duplicity restore --gio --force 'file:///media/tantin/8c9dba0f-0fcb-4d18-a0a8-4d22af8356e8/gpg' '/home/tantin/backup'
Local and Remote metadata are synchronized, no sync needed.
Traceback (most recent call last):
  File "./bin/duplicity", line 1403, in <module>
    with_tempdir(main)
  File "./bin/duplicity", line 1396, in with_tempdir
    fn()
  File "./bin/duplicity", line 1276, in main
    globals.archive_dir).set_values()
  File "/home/tantin/Desktop/duplicity/duplicity/collections.py", line 698, in set_values
    self.get_backup_chains(partials + backend_filename_list)
  File "/home/tantin/Desktop/duplicity/duplicity/collections.py", line 821, in get_backup_chains
    map(add_to_sets, filename_list)
  File "/home/tantin/Desktop/duplicity/duplicity/collections.py", line 811, in add_to_sets
    if set.add_filename(filename):
  File "/home/tantin/Desktop/duplicity/duplicity/collections.py", line 97, in add_filename
    (self.volume_name_dict, filename)
AssertionError: ({1: 'duplicity-full.20130307T152041Z.vol1.difftar', 2050: 'duplicity-full.20130307T152041Z.vol2050.difftar.gpg', 2056: 'duplicity-full.20130307T152041Z.vol2056.difftar.gpg', 2059: 'duplicity-full.20130307T152041Z.vol2059.difftar.gpg', 2061: 'duplicity-full.20130307T152041Z.vol2061.difftar.gpg', 14: 'duplicity-full.20130307T152041Z.vol14.difftar.gpg', 16: 'duplicity-full.20130307T152041Z.vol16.difftar.gpg', 2068: 'duplicity-full.20130307T152041Z.vol2068.difftar.gpg', 2075: 'duplicity-full.20130307T152041Z.vol2075.difftar.gpg', 28: 'duplicity-full.20130307T152041Z.vol28.difftar.gpg', 30: 'duplicity-full.20130307T152041Z.vol30.difftar.gpg', 2086: 'duplicity-full.20130307T152041Z.vol2086.difftar.gpg', 41: 'duplicity-full.20130307T152041Z.vol41.difftar.gpg', 2092: 'duplicity-full.20130307T152041Z.vol2092.difftar.gpg', 46: 'duplicity-full.20130307T152041Z.vol46.difftar.gpg', 47: 'duplicity-full.20130307T152041Z.vol47.difftar.gpg', 51: 'duplicity-full.20130307T152041Z.vol51.difftar.gpg', 52: 'duplicity-full.20130307T152041Z.vol52.difftar.gpg', 2105: 'duplicity-full.20130307T152041Z.vol2105.difftar.gpg', 62: 'duplicity-full.20130307T152041Z.vol62.difftar.gpg', 2111: 'duplicity-full.20130307T152041Z.vol2111.difftar.gpg', 65: 'duplicity-full.20130307T152041Z.vol65.difftar.gpg', 2: 'duplicity-full.20130307T152041Z.vol2.difftar', 67: 'duplicity-full.20130307T152041Z.vol67.difftar.gpg', 74: 'duplicity-full.20130307T152041Z.vol74.difftar.gpg', 77: 'duplicity-full.20130307T152041Z.vol77.difftar.gpg', 78: 'duplicity-full.20130307T152041Z.vol78.difftar.gpg', 79: 'duplicity-full.20130307T152041Z.vol79.difftar.gpg', 2062: 'duplicity-full.20130307T152041Z.vol2062.difftar.gpg', 86: 'duplicity-full.20130307T152041Z.vol86.difftar.gpg', 87: 'duplicity-full.20130307T152041Z.vol87.difftar.gpg', 2137: 'duplicity-full.20130307T152041Z.vol2137.difftar.gpg', 2140: 'duplicity-full.20130307T152041Z.vol2140....

Is there a completely manual way to restore? Removing the encryption shouldn't be to hard but it's an incremental backup and I worry about stitching the files together.

I tried and never could make it work. Lost a lot of work.

On 06/01/2013 09:41 AM, Constantin Gindele wrote:
> Is there a completely manual way to restore? Removing the encryption
> shouldn't be to hard but it's an incremental backup and I worry about
> stitching the files together.
>

Constantin, to fix the error you see in your comment #26, delete the file duplicity-full.20130307T152041Z.vol1001.difftar (note how there is no .gpg after it). To see instructions for manually recovering, see https://live.gnome.org/DejaDup/Help/Restore/WorstCase (especially near the bottom).

I have the same problem.

I really need to restore some files from my backup and it is ending with this error
librsyncError: librsync error 103 while in patch cycle

There is no file in my backup without .gpg
manuall recovery as suggested is causing the same error message.

I tried the given patch but it still crash.

Error 'librsync error 103 while in patch cycle' processing home/john/.eclipse/org.eclipse.platform_3.8_155965261/configuration/org.eclipse.equinox.simpleconfigurator/bundles.info
Traceback (most recent call last):
  File "./bin/duplicity", line 1403, in <module>
    with_tempdir(main)
  File "./bin/duplicity", line 1396, in with_tempdir
    fn()
  File "./bin/duplicity", line 1330, in main
    restore(col_stats)
  File "./bin/duplicity", line 624, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/tmp/duplicity/duplicity/patchdir.py", line 532, in Write_ROPaths
    for ropath in rop_iter:
  File "/tmp/duplicity/duplicity/patchdir.py", line 505, in integrate_patch_iters
    final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
  File "/tmp/duplicity/duplicity/patchdir.py", line 463, in patch_seq2ropath
    assert first.difftype != "diff", patch_seq
AssertionError: [(('home', 'john', '.eclipse', 'org.eclipse.platform_3.8_155965261', 'configuration', 'org.eclipse.osgi', 'bundles', '80', 'data', '-741665970', 'artifacts.xml') reg)]

I added a try/catch around the faulty lines (like the one in the patch) and get it going through the restore. just missing sevral files (logs, gnome config DB, etc.).

I just got a bad feeling about it since my backup is my 'last ressort' when something goes wrong (here a HD crash). I expect it to be bullet-proof.

I hope duplicity will fix it since it is still a sooo great software for backup.

Regards

Had this happen to me a year ago -- my mistake was failing to test the
restore immediately after the initial backup. I'm using grsync at this
point, a not-great but at least adequate substitute.

On 07/24/2013 12:53 AM, Frederic Dreier wrote:
> I added a try/catch around the faulty lines (like the one in the patch)
> and get it going through the restore. just missing sevral files (logs,
> gnome config DB, etc.).
>
> I just got a bad feeling about it since my backup is my 'last ressort'
> when something goes wrong (here a HD crash). I expect it to be bullet-
> proof.
>
> I hope duplicity will fix it since it is still a sooo great software for
> backup.
>
> Regards
>

I have the same problem. The restoration of the home gives the error "librsync error 103 while in patch cycle".
After applying the pacth, I can restore individual home folder, but not the entire home, it shows the error AssertionError: first.difftype! = "Diff"

I have the same problem. I used the patch Christopher Foo provided (many thanks for that!) - but unfortunately the outcome is the same as what gluca reports.
Would be very grateful for help

thx for the patch.. rescued my stuff successfully...
now going for a more robust backup tool :)

For those who still want (or have to) to restore their backups by hand, I found a solution for me.
The unpack-part is easy, the stich/join-part is not. So I wrote an application for that. More details here: http://blog.atoaudiovisual.com/2013/09/restore-broken-deja-dup-backup-hand/

The link to the source http://atoaudiovisual.com/x/duplicity_joiner.py seems
to be broken

On 6 September 2013 02:18, David Huss <email address hidden> wrote:

> For those who still want (or have to) to restore their backups by hand, I
> found a solution for me.
> The unpack-part is easy, the stich/join-part is not. So I wrote an
> application for that. More details here:
> http://blog.atoaudiovisual.com/2013/09/restore-broken-deja-dup-backup-hand/
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/662442
>
> Title:
> restore fails with "librsync error 103 while in patch cycle"
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/duplicity/+bug/662442/+subscriptions
>

--
Martin

A coworker (Steve Magoun) graciously lent me a backup data set of his that reproduces this problem.

What appears to have happened on his data set (and presumably others who are experiencing this) is that a file (say, test.txt) gets backed up in the initial full backup at time A. This initial backup seems fine.

Later, the file is included in an incremental backup at time B. But for some reason, the patch that duplicity creates is seemingly against a different basis file than the recorded at time A. So when restoring, the patch does not apply to the stored time A version of the file.

When restoring from time B, duplicity will complain but not error out. It will create a half-finished test.txt. If you restore from a later time point, duplicity will error out and create no file.

Here is the librsync output from my example data. (original basis file is an xml text file 68664 bytes large) You can see at the end, it tries to read from the basis file past its end. So the patch seems to be against a different (larger) basis file than we have. I'm not sure how this could happen yet, but I figured I'd dump these thoughts here.

---- original call to rs_job_iter here ----
python2.7: (rs_scoop_readahead) got 4 bytes from input buffer
python2.7: (rs_scoop_advance) advance over 4 bytes from input buffer
python2.7: (rs_patch_s_header) got patch magic 0x72730236
python2.7: (rs_scoop_readahead) got 1 bytes from input buffer
python2.7: (rs_scoop_advance) advance over 1 bytes from input buffer
python2.7: (rs_patch_s_cmdbyte) got command byte 0x42 (LITERAL), len_1=2
python2.7: (rs_scoop_readahead) got 2 bytes from input buffer
python2.7: (rs_scoop_readahead) got 2 bytes from input buffer
python2.7: (rs_scoop_advance) advance over 2 bytes from input buffer
python2.7: (rs_patch_s_run) running command 0x42, kind 1001
python2.7: (rs_patch_s_literal) LITERAL(len=512)
python2.7: (rs_tube_catchup_copy) copied 512 bytes from input buffer, 0 remain to be copied
python2.7: (rs_scoop_readahead) got 1 bytes from input buffer
python2.7: (rs_scoop_advance) advance over 1 bytes from input buffer
python2.7: (rs_patch_s_cmdbyte) got command byte 0x4a (COPY), len_1=2
python2.7: (rs_scoop_readahead) got 4 bytes from input buffer
python2.7: (rs_scoop_readahead) got 2 bytes from input buffer
python2.7: (rs_scoop_advance) advance over 2 bytes from input buffer
python2.7: (rs_scoop_readahead) got 2 bytes from input buffer
python2.7: (rs_scoop_advance) advance over 2 bytes from input buffer
python2.7: (rs_patch_s_run) running command 0x4a, kind 1003
python2.7: (rs_patch_s_copy) COPY(where=512, len=24064)
python2.7: (rs_patch_s_copying) copy 24064 bytes from basis at offset 512
python2.7: (rs_patch_s_copying) copy callback returned OK
python2.7: (rs_patch_s_copying) got 24064 bytes back from basis callback
python2.7: (rs_scoop_readahead) got 1 bytes from input buffer
python2.7: (rs_scoop_advance) advance over 1 bytes from input buffer
python2.7: (rs_patch_s_cmdbyte) got command byte 0x42 (LITERAL), len_1=2
python2.7: (rs_scoop_readahead) got 2 bytes from input buffer
python2.7: (rs_scoop_readahead) got 2 bytes from input buffer
python2.7: (rs_scoop_advance) advance over 2 by...

I still haven't found the source of this bug. But I have filed a branch [1] that is basically a glorified version of the patch in comment 6. This lets the rest of the restore continue when this error happens, though obviously without the problem file. (It will log an error about that file, so users will notice.)

Hopefully that will reduce the severity of this bug, but we should keep this bug open for the original problem.

[1] https://code.launchpad.net/~mterry/duplicity/catch-seq-copy-error/+merge/186106

Good news! I've confirmed this is the same underlying issue as bug 1252484 (which I will mark this bug a duplicate of). This is just one of its symptoms. I've attached the file 'test5.sh' which when run like 'sh test5.sh' should reliably reproduce this exception.

This bug has been fixed in duplicity trunk (and Ubuntu 13.10+). The fix will be in the upcoming duplicity 0.6.23 release.

I've prepared fixes for older Ubuntu releases, but the updates need testers before they can be released to everyone. If you are an Ubuntu user of an older release, please go to bug 1252484 and follow the instructions for testing the update. Thanks!

Hi,

I am not sure if this is related, but I am certainly hoping someone can assist

I'm struggling to restore backups from an older installation of Ubuntu to a new installation on Ubuntu 16.04. I get the following error when trying to restore:

Failed with an unknown error Followed by:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1532, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1526, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1380, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1461, in do_backup
    list_current(col_stats)
  File "/usr/bin/duplicity", line 698, in list_current
    for path in path_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 354, in combine_path_iters
    refresh_triple_list(triple_list)
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 341, in refresh_triple_list
    new_triple = get_triple(old_triple[1])
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 327, in get_triple
    path = path_iter_list[iter_index].next()
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 239, in sigtar2path_iter
    for tarinfo in tf:
  File "/usr/lib/python2.7/tarfile.py", line 2508, in next
    tarinfo = self.tarfile.next()
  File "/usr/lib/python2.7/tarfile.py", line 2350, in next
    raise ReadError("unexpected end of data")
ReadError: unexpected end of data

The backup completed successfully but I cannot get it to restore.

Thanks in advance!