AssertionError when Duplicity sees .par2 files in the remote file list

Bug #1406173 reported by Tyler Ham
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Duplicity
Fix Released
Medium
Unassigned

Bug Description

The assertion in collections.py add_filename is choking on .par2 files. I believe it is during a process that is looking at the destination file list. This list includes .par2 files, but the filename parser is not aware of them and ends up with the assertion error.

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1500, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1494, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1343, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1376, in do_backup
    globals.archive_dir).set_values()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 698, in set_values
    self.get_backup_chains(partials + backend_filename_list)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 821, in get_backup_chains
    add_to_sets(f)
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 810, in add_to_sets
    if set.add_filename(filename):
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 102, in add_filename
    (pr.volume_number, self.volume_name_dict, filename)
AssertionError: (1, {1: 'duplicity-full.20141229T025217Z.vol1.difftar.gpg'}, 'duplicity-full.20141229T025217Z.vol1.difftar.gpg.par2')

Duplicity version 0.7.0 (* see below), Python 2.7.3, Ubuntu 12.04.5 LTS
Target filesystem: Linux

* I am running Duplicity version 0.7.0 with a couple of patches applied in order to make par2+rsync work. Both of the patches appear to have already made it into the repository in 0.7.01 (2014/12/??):
1. I added 'rsync' to uses_netlock (see https://answers.launchpad.net/duplicity/+question/259173).
2. I added strip_prefix() in backend.py and replaced url_string.lstrip with the call to strip_prefix. This was necessary to make the par2+rsync:// schema parse properly. (see http://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/revision/1014)

My command line:
duplicity --archive-dir '/var/duply-cache' --name 'duply_officetest' --encrypt-key $ENCRYPTKEY --sign-key $SIGNKEY --verbosity 9 --full-if-older-than 1Y --volsize 100 --asynchronous-upload --exclude-globbing-filelist /etc/duply/officetest/exclude $SOURCE par2+rsync://$USERNAME@$HOST//path/to/destination

My solution:
I believe the par2 backend should be filtering out .par2 files before returning the list of remote files. The list() method in par2backend.py did this before the backends were reorganized by revision 981 (http://bazaar.launchpad.net/~duplicity-team/duplicity/0.7-series/revision/981). My AssertionError problem was resolved when I restored that filtering that was removed in revision 981. I have attached a patch.

Revision history for this message
Tyler Ham (tyler.ham) wrote :
Changed in duplicity:
importance: Undecided → Medium
milestone: none → 0.7.01
status: New → Fix Committed
Changed in duplicity:
status: Fix Committed → Fix Released
Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote : Re: [Bug 1406173] Re: AssertionError when Duplicity sees .par2 files in the remote file list
Download full text (4.8 KiB)

Looks like you had run with par2+?? before this run, then changed methods.
If that's the case, then you'll need to manually delete the .par2 files
from the remote and run again, or stick with the same method.

On Mon, Jan 12, 2015 at 3:22 PM, John Doe <email address hidden> wrote:

> Still getting this error, only tested when the cache
> (~/.cache/duplicity) doesn't exist. Version 0.7.1:-
>
> Local and Remote metadata are synchronized, no sync needed.
> Traceback (most recent call last):
> File "/usr/local/bin/duplicity", line 1500, in <module>
> with_tempdir(main)
> File "/usr/local/bin/duplicity", line 1494, in with_tempdir
> fn()
> File "/usr/local/bin/duplicity", line 1343, in main
> do_backup(action)
> File "/usr/local/bin/duplicity", line 1376, in do_backup
> globals.archive_dir).set_values()
> File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py",
> line 698, in set_values
> self.get_backup_chains(partials + backend_filename_list)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py",
> line 821, in get_backup_chains
> add_to_sets(f)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py",
> line 810, in add_to_sets
> if set.add_filename(filename):
> File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py",
> line 98, in add_filename
> self.set_manifest(filename)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/collections.py",
> line 129, in set_manifest
> remote_filename)
> AssertionError: ('duplicity-full.20150112T202904Z.manifest.gpg',
> 'duplicity-full.20150112T202904Z.manifest.gpg.par2')
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1406173
>
> Title:
> AssertionError when Duplicity sees .par2 files in the remote file list
>
> Status in Duplicity - Bandwidth Efficient Encrypted Backup:
> Fix Released
>
> Bug description:
> The assertion in collections.py add_filename is choking on .par2
> files. I believe it is during a process that is looking at the
> destination file list. This list includes .par2 files, but the
> filename parser is not aware of them and ends up with the assertion
> error.
>
> Traceback (most recent call last):
> File "/usr/bin/duplicity", line 1500, in <module>
> with_tempdir(main)
> File "/usr/bin/duplicity", line 1494, in with_tempdir
> fn()
> File "/usr/bin/duplicity", line 1343, in main
> do_backup(action)
> File "/usr/bin/duplicity", line 1376, in do_backup
> globals.archive_dir).set_values()
> File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line
> 698, in set_values
> self.get_backup_chains(partials + backend_filename_list)
> File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line
> 821, in get_backup_chains
> add_to_sets(f)
> File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line
> 810, in add_to_sets
> if set.add_filename(filename):
> File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line
> 102, in add_filename
> (pr.volume_number, self.vol...

Read more...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.