AssertionError: first.difftype != "diff"

The first set of errors are due to the system being unable to 'chown' the files to their original owner. I don't think these are the main issue here.

I'm still familiarizing myself with the relevant parts of the code base, but here's my interpretation of the stack trace. At the time the exception is raised, duplicity is working on the following sequence of patches:

[(('.duplicity', 'filelist') reg), (('.duplicity', 'filelist') reg)]

When applying a set of patches, they must be applied in the order they were created, starting with the original file. In this case, the first patch in the sequence is a "diff", so an exception is raised.

This isn't due to the backup files being in a bad state since I was just able to restore everything on a different system with the current lp:duplicity branch.

There were a number of changes between 0.6.06 and 0.6.11 that may account
for this problem.

Would it be possible to upgrade the problem system to the newest version?

...Ken

On Fri, Feb 18, 2011 at 3:06 PM, Joe Snikeris <email address hidden> wrote:

> This isn't due to the backup files being in a bad state since I was just
> able to restore everything on a different system with the current
> lp:duplicity branch.
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/720525
>
> Title:
> Unable to restore backup; several errors then hang
>
> Status in Duplicity - Bandwidth Efficient Encrypted Backup:
> New
>
> Bug description:
> Duplicity version: 0.6.06
> Python version: 2.5.4
> OS: OpenBSD 4.8
> Remote filesystem: ext3
> Local filesystem: ffs
>
> I've attached a verbose log of the duplicity output and errors. I've
> copied the backup files from the remote server to my local machine to
> rule out transmission issues.
>
> The python process spins for a bit at 0.1% CPU usage, but then stops
> and top's TIME column doesn't increase. It continues to hang after
> that until I kill it.
>
> As you can see in the log, there are several 'Operation not permitted'
> errors; however, the processing continues. Then there is an
> AssertionError, and that's the last thing that is displayed before the
> process hangs.
>
> Also, here is a list of the files and directories that were restored:
> restore
> restore/.Xdefaults
> restore/.cache
> restore/.cache/duplicity
> restore/.cache/duplicity/4d69b8815c980ca928f8bf0170d40bfb
>
> restore/.cache/duplicity/4d69b8815c980ca928f8bf0170d40bfb/duplicity-full-signatures.20110110T003018Z.sigtar.gz
> restore/.duplicity
> restore/.duplicity/backup.log
>
> Also, if there is a workaround so that I can retrieve my files, I'd
> greatly appreciate being pointed in the right direction.
>

I just ran a restore on the problem system using the current lp:duplicity branch. Same issue.

The system that the restore worked on:
Openbsd 4.9-CURRENT i386; Python 2.6.6

The system that it didn't work on:
Openbsd 4.8 amd64; Python 2.6.5

I'm installing Python 2.6.6 on the problem system to eliminate that difference.

Also, I wasn't able to run duplicity-bin with python 2.5.4:

src/duplicity/duplicity-bin:868: Warning: 'as' will become a reserved keyword in Python 2.6
  File "src/duplicity/duplicity-bin", line 868
    except Exception as e:
                      ^
SyntaxError: invalid syntax

Is Python 2.6 required?

Eh, I'm not going to install 2.6.6 on this machine. 2.6.5 needed some OS-specific patches, and I'm not comfortable applying them. I doubt this is the issue anyway.

No, 2.6 is not required. Looks like the latest repo has a problem. Please
try the 0.6.11 tarball in downloads.

On Fri, Feb 18, 2011 at 3:49 PM, Joe Snikeris <email address hidden> wrote:

> Also, I wasn't able to run duplicity-bin with python 2.5.4:
>
> src/duplicity/duplicity-bin:868: Warning: 'as' will become a reserved
> keyword in Python 2.6
> File "src/duplicity/duplicity-bin", line 868
> except Exception as e:
> ^
> SyntaxError: invalid syntax
>
> Is Python 2.6 required?
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/720525
>
> Title:
> Unable to restore backup; several errors then hang
>
> Status in Duplicity - Bandwidth Efficient Encrypted Backup:
> New
>
> Bug description:
> Duplicity version: 0.6.06
> Python version: 2.5.4
> OS: OpenBSD 4.8
> Remote filesystem: ext3
> Local filesystem: ffs
>
> I've attached a verbose log of the duplicity output and errors. I've
> copied the backup files from the remote server to my local machine to
> rule out transmission issues.
>
> The python process spins for a bit at 0.1% CPU usage, but then stops
> and top's TIME column doesn't increase. It continues to hang after
> that until I kill it.
>
> As you can see in the log, there are several 'Operation not permitted'
> errors; however, the processing continues. Then there is an
> AssertionError, and that's the last thing that is displayed before the
> process hangs.
>
> Also, here is a list of the files and directories that were restored:
> restore
> restore/.Xdefaults
> restore/.cache
> restore/.cache/duplicity
> restore/.cache/duplicity/4d69b8815c980ca928f8bf0170d40bfb
>
> restore/.cache/duplicity/4d69b8815c980ca928f8bf0170d40bfb/duplicity-full-signatures.20110110T003018Z.sigtar.gz
> restore/.duplicity
> restore/.duplicity/backup.log
>
> Also, if there is a workaround so that I can retrieve my files, I'd
> greatly appreciate being pointed in the right direction.
>

Same issue with .11. Log attached.

This is strange...if I specify the file that I was bombing out on before (.duplicity/filelist), I can restore it successfully. See log.

When I specify the whole directory that contains the troublesome file, there is no problem either. I've added a few debug messages, and apparently there are three patches for this file:

[(('filelist',) reg), (('filelist',) reg), (('filelist',) reg)]
difftype: snapshot
difftype: diff
difftype: diff
Writing filelist of type reg

However, when I try to restore the whole backup, the snapshot is missing:
[(('.duplicity', 'filelist') reg), (('.duplicity', 'filelist') reg)]
difftype: diff
difftype: diff
Removing still remembered temporary file /tmp/duplicity-JP3frh-tempdir/mktemp-DbOWkN-18
Removing still remembered temporary file /tmp/duplicity-JP3frh-tempdir/mktemp-ZK_2dG-19
Removing still remembered temporary file /tmp/duplicity-JP3frh-tempdir/mkstemp-IBYs3s-1
Removing still remembered temporary file /tmp/duplicity-JP3frh-tempdir/mktemp-LdhqlW-17
Removing still remembered temporary file /tmp/duplicity-JP3frh-tempdir/mktemp-XQ30v6-20
Removing still remembered temporary file /tmp/duplicity-JP3frh-tempdir/mktemp-WLFdsw-15
Removing still remembered temporary file /tmp/duplicity-JP3frh-tempdir/mktemp-GRzVEs-16
Traceback (most recent call last):
  File "/home/joe/src/duplicity/duplicity-bin", line 1248, in <module>
    with_tempdir(main)
  File "/home/joe/src/duplicity/duplicity-bin", line 1241, in with_tempdir
    fn()
  File "/home/joe/src/duplicity/duplicity-bin", line 1195, in main
    restore(col_stats)
  File "/home/joe/src/duplicity/duplicity-bin", line 539, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/home/joe/src/duplicity/duplicity/patchdir.py", line 524, in Write_ROPaths
    for ropath in rop_iter:
  File "/home/joe/src/duplicity/duplicity/patchdir.py", line 497, in integrate_patch_iters
    final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
  File "/home/joe/src/duplicity/duplicity/patchdir.py", line 464, in patch_seq2ropath
    assert first.difftype != "diff", patch_seq
AssertionError: [(('.duplicity', 'filelist') reg), (('.duplicity', 'filelist') reg)]

I'll have to look into how these patch sequences are being generated to figure out why the snapshot is missing in the second case.

It would be super helpful if anyone that is experiencing this could share their backup files with me. Obviously that's unlikely, since most people's backup files hold personal information, but I ask on the off chance.

Closing deja-dup task, as this is a duplicity bug.

Still getting errors:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1404, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1397, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1277, in main
    globals.archive_dir).set_values()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 691, in set_values
    self.get_backup_chains(partials + backend_filename_list)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 814, in get_backup_chains
    map(add_to_sets, filename_list)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 804, in add_to_sets
    if set.add_filename(filename):
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 93, in add_filename
    self.set_manifest(filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 124, in set_manifest
    remote_filename)
AssertionError: ('duplicity-inc.20121111T021934Z.to.20121112T000242Z.manifest.gpg', 'duplicity-inc.20121111T021934Z.to.20121112T000242Z.manifest')

I made a backup because I needed to format my hard drive, then I installed Ubuntu again and tried to restore but it failed..
I really need to erstore my files..

I get the following error:

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1412, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1405, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1339, in main
    restore(col_stats)
  File "/usr/bin/duplicity", line 630, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 522, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 495, in integrate_patch_iters
    final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 462, in patch_seq2ropath
    assert first.difftype != "diff", patch_seq
AssertionError: [(('home', 'kapcom01', '.Skype', 'kapcom01', 'chatsync', '06', '064c14fdb16c70bc.dat') reg)]

I deleted ~/.cache/deja-dup and tried again, but failed again with the same error..

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1411, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1404, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1338, in main
    restore(col_stats)
  File "/usr/bin/duplicity", line 632, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 526, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 499, in integrate_patch_iters
    final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 463, in patch_seq2ropath
    assert first.difftype != "diff", patch_seq
AssertionError: [(('home', 'mfilipe', '.Skype', 'michel.filipe', 'chatsync', 'df', 'df23bc92967ee9a5.dat') reg)]

I am seeing this bug, too. All I can add is that it seems to happen on a file which is not present in the original full backup. The original full backup can be restored (or at least does not bump out when restoring the directory where the file listed in the assertion error lives).

For the record, I am using Fedora and there is a bug in their bugzilla, too:
https://bugzilla.redhat.com/show_bug.cgi?id=901420

There is also another entry on lauchpad which I found when googling for this problem:
https://bugs.launchpad.net/ubuntu/+source/duplicity/+bug/1222661

Unfortunately, no resolution in any of these, but the problem looks pretty serious.

I confirm this critical bug. With this bug, the ubuntu backup (deja-dup) is useless..

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1411, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1404, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1338, in main
    restore(col_stats)
  File "/usr/bin/duplicity", line 632, in restore
    restore_get_patched_rop_iter(col_stats)):
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 526, in Write_ROPaths
    for ropath in rop_iter:
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 499, in integrate_patch_iters
    final_ropath = patch_seq2ropath( normalize_ps( patch_seq ) )
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 466, in patch_seq2ropath
    assert len( patch_seq ) == 1, len( patch_seq )
AssertionError: 4

Good news! I've confirmed this is the same underlying issue as bug 1252484 (which I will mark this bug a duplicate of). This is just one of its symptoms. I've attached the file 'test2.sh' which when run like 'sh test2.sh' should reliably reproduce this exception.

This bug has been fixed in duplicity trunk (and Ubuntu 13.10+). The fix will be in the upcoming duplicity 0.6.23 release.

I've prepared fixes for older Ubuntu releases, but the updates need testers before they can be released to everyone. If you are an Ubuntu user of an older release, please go to bug 1252484 and follow the instructions for testing the update. Thanks!