incremental backup does not work

I am also experiencing this. I am confirming the bug.

I'm experiencing this issue also.

I am experiencing this same incremental back up issue with duplicity on my Ubuntu Feisty servers. It appears that the files get backed up but --verify, restore, and the incremental backup does not “see” them. I do not have this problem on my Edgy servers (Version: 0.4.1-8ubuntu1).

Here is my output for a simple back up and --verify:

root@w2szswr601:~# duplicity --encrypt-key "91708B1B" /home/szswr6/vm_info file:///home/data/bur/backup
GnuPG passphrase:
Retype to confirm:
No signatures found, switching to full backup.
--------------[ Backup Statistics ]--------------
StartTime 1178639042.32 (Tue May 8 11:44:02 2007)
EndTime 1178639043.83 (Tue May 8 11:44:03 2007)
ElapsedTime 1.50 (1.50 seconds)
SourceFiles 4
SourceFileSize 3343424 (3.19 MB)
NewFiles 4
NewFileSize 3343424 (3.19 MB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 4
RawDeltaSize 3339328 (3.18 MB)
TotalDestinationSizeChange 2604284 (2.48 MB)
Errors 0
-------------------------------------------------

root@w2szswr601:~# duplicity --encrypt-key "91708B1B" --verbosity 4 --verify file:///home/data/bur/backup /home/szswr6/vm_info
GnuPG passphrase:
Retype to confirm:
Difference found: New file .
Difference found: New file VirtualBox_UserManual.pdf
Difference found: New file coe_build_into.txt~
Difference found: New file qemu_info_from_nick.txt
Verify complete: 4 files compared, 4 differences found.

Also filed at http://savannah.nongnu.org/support/?105849

The problem seems to have something to do with the python version, which is 2.5 on feisty, and 2.4 on edgy.

It works with python 2.4: if I copy the duplicity python modules into the python 2.4 tree and call duplicity with 'python2.4 /usr/bin/duplicity', everything works fine (despite a librsync version warning).

Using python2.4 to build and install duplicity from source also produces a working installation.

Hi,
I am also experiencing this. I wanted to use duplicity for incremental backups of our clients, but the missing incremental backup function makes it just about impossible.

I would really appreciate if this could be investigated! :)

Jens

Addendum: if you compile duplicity-0.4.2 from source using python2.4, detecing present backup sets fails, because of this (I think):

 Reading results of 'echo -e 'cd /users/benecke/KLAVIER
 ls -1' | sftp -b - benecke@orgel'

Full log:

jens@klavier:~$ duplicity --no-encryption --verbosity 9 --verify ssh://benecke@orgel//users/benecke/KLAVIER /
Reading globbing filelist /home/jens/.ftplicity/exclude
Main action: verify
Reading results of 'echo -e 'cd /users/benecke/KLAVIER
ls -1' | sftp -b - benecke@orgel'
Invalid command.
Ignoring file 'bin'
(... several other files ...)
Collection Status
-----------------
Connecting with backend: scpBackend
Archive dir: None
No backup chains with active signatures found

Found 0 backup chains without signatures.
No orphaned or incomplete backup sets found.
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 373, in ?
    if __name__ == "__main__": main()
  File "/usr/bin/duplicity", line 355, in main
    elif action == "verify": verify(col_stats)
  File "/usr/bin/duplicity", line 262, in verify
    collated = diffdir.collate2iters(restore_get_patched_rop_iter(col_stats),
  File "/usr/bin/duplicity", line 201, in restore_get_patched_rop_iter
    backup_chain = col_stats.get_backup_chain_at_time(time)
  File "/usr/lib/python2.4/site-packages/duplicity/collections.py", line 598, in get_backup_chain_at_time
    raise CollectionsError("No backup chains found")
duplicity.collections.CollectionsError: No backup chains found
Command exited with non-zero status 1

I am using SSH connections with the key stored in a key-agent (so passwordless connection is possible).

Duplicity 0.4.2 works fine on Debian Sarge, using python2.3.

Thanks,

Jens

There's something really evil going on with duplicity and python2.5. Unfortunately the duplicity codebase seems to be a bit messy, and it was easier to just fix the build scripts :)

Here's a fix debian/{rules,control} and friends which forces a build against python2.4. It also uses "$(PYTHON)" instead of "python" to call the build scripts, so it might be easier to step back and forth with different python versions.

Works for me.

This is caused by a ValueError exception in duplicity's tarfile.py, because it uses int() on strings containing \0 (NUL) characters.
This exception does not happen with Python 2.4, and also is not specific to Ubuntu's Python 2.5.

The attached debdiff fixes this.

Here's the upstream bug: https://savannah.nongnu.org/bugs/index.php?19998

The fix has been confirmed upstream.
It would be great if somebody could try my attached debdiff, which should probably work and fix the problem for you.
Please see https://wiki.ubuntu.com/UbuntuPackagingGuide/BuildFromDebdiff for creating a fixed package.

I am unable to patch using duplicity_0.4.2-13ubuntu2.dsc.debdiff
Here is my output:

root@w2szswr601:/usr/src/duplicity# cd duplicity-0.4.2 && patch -p1 < ../duplicity_0.4.2-13ubuntu2.dsc.debdiff
can't find file to patch at input line 4
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff -Nru /tmp/QRGPt4qMTY/duplicity-0.4.2/debian/changelog /tmp/Lq0Slwsx2n/duplicity-0.4.2/debian/changelog
|--- /tmp/QRGPt4qMTY/duplicity-0.4.2/debian/changelog 2007-06-01 00:57:07.000000000 +0200
|+++ /tmp/Lq0Slwsx2n/duplicity-0.4.2/debian/changelog 2007-06-01 00:57:07.000000000 +0200
--------------------------
File to patch: debian/changelog
patching file debian/changelog
Hunk #1 FAILED at 1.
1 out of 1 hunk FAILED -- saving rejects to file debian/changelog.rej
patching file tmp/QRGPt4qMTY/duplicity-0.4.2/debian/patches/07fix-tarfile-_buftoinfo-for-python2.5.dpatch

OK I now see my mistake (other than trying to patch too late at night when the brain has already shut down!) but.....
I can patch using duplicity_0.4.2-10.1ubuntu1.dsc.debdiff but not duplicity_0.4.2-13ubuntu2.dsc.debdiff

root@w2szswr601:/usr/src/duplicity/duplicity-0.4.2# patch -p4 < ../duplicity_0.4.2-10.1ubuntu1.dsc.debdiff
patching file debian/patches/06_fix-tarfile-_buftoinfo-for-python2.5

root@w2szswr601:/usr/src/duplicity/duplicity-0.4.2# patch -p4 < ../duplicity_0.4.2-13ubuntu2.dsc.debdiff
patching file debian/changelog
Hunk #1 FAILED at 1.
1 out of 1 hunk FAILED -- saving rejects to file debian/changelog.rej
patching file debian/patches/07fix-tarfile-_buftoinfo-for-python2.5.dpatch

Sorry Tom, the debdiff was against the gutsy package.

The one against feisty, with changelog entry is attached. You should apply this one with "patch -p1", as described in the wiki page linked above.

No problem dAniel.
Well...I have good news and bad news. It now patches ok, but I am getting the same result w/ the new package as I was w/ the old package.

- Added patch to 00list (used dpatch-edit-patch)
- Taken patch from upstream CVS, should make merging in the future easier

This time I have only good news to report!
The new debdiff worked perfectly. My first (and quick) tests show a perfectly working Duplicity. I will do some more thorough testing later toady, when I have more time.

debdiff duplicity_0.4.2-10.1ubuntu1.dsc duplicity_0.4.2-10.1ubuntu1.1.dsc Looks to be the fix!

I have been using the patched duplicity pretty heavily over the last few days and everything appears to be working fine.

Thanks!

Works for me too. Built and installed on feisty.

My apologies, but I'm a little confused as to which debdiff needs to be applied to each of the gutsy and feisty packages. Could you please either re-prepare or reupload debdiffs for new candidate revisions for each of feisty-proposed and gutsy, with a comment indicating which is correct. Alternately, please delete any previously attached debdiffs that do not apply. I have unsubscribed ubuntu-universe-sponsors pending this clarification: please resubscribe to again request uploads. Thank you.

I've removed all my previous debdiffs, only left the final one.

There's only a debdiff for feisty-proposed: there will be a new release of duplicity with a lot more fixes in the next days, so no need to get this particular fix into gutsy alone IMHO.

The attached debdiff is the one working for me on feisty.

Format: 1.7
Date: Sun, 3 Jun 2007 16:54:31 +0200
Source: duplicity
Binary: duplicity
Architecture: source
Version: 0.4.2-10.1ubuntu1.1
Distribution: feisty-proposed
Urgency: low
Maintainer: Ubuntu MOTU Developers <email address hidden>
Changed-By: dAniel hAhler <email address hidden>
Description:
 duplicity - encrypted bandwidth-efficient backup
Launchpad-Bugs-Fixed: 88617
Changes:
 duplicity (0.4.2-10.1ubuntu1.1) feisty-proposed; urgency=low
 .
   * debian/patches/06_fix-tarfile-_buftoinfo-for-python2.5:
     Fixed "incremental backup does not work" for Python 2.5 (LP: #88617)
   * Applied changes according to https://wiki.ubuntu.com/DebianMaintainerField
Files:
 af337311f168266c6608a160caae5622 758 utils optional duplicity_0.4.2-10.1ubuntu1
.1.dsc
 5784a495edd517fb4fc07792b4c0ab1d 7464 utils optional duplicity_0.4.2-10.1ubuntu
1.1.diff.gz
Original-Maintainer: Martin Wuertele <email address hidden>

Accepted into feisty-proposed. Please go ahead with QA testing and also fix this in gutsy ASAP. I will refuse to let this into -updates without having it tested and fixed in gutsy.

duplicity (0.4.2-10.1ubuntu1.1) feisty-proposed; urgency=low

  * debian/patches/06_fix-tarfile-_buftoinfo-for-python2.5:
    Fixed "incremental backup does not work" for Python 2.5 (LP: #88617)
  * Applied changes according to https://wiki.ubuntu.com/DebianMaintainerField

 -- dAniel hAhler <email address hidden> Sun, 3 Jun 2007 16:54:31 +0200

Please find attached the debdiff for Gutsy.

Format: 1.7
Date: Fri, 15 Jun 2007 15:29:08 +0200
Source: duplicity
Binary: duplicity
Architecture: source
Version: 0.4.2-14ubuntu2
Distribution: gutsy
Urgency: low
Maintainer: Ubuntu MOTU Developers <email address hidden>
Changed-By: dAniel hAhler <email address hidden>
Description:
 duplicity - encrypted bandwidth-efficient backup
Launchpad-Bugs-Fixed: 88617
Changes:
 duplicity (0.4.2-14ubuntu2) gutsy; urgency=low
 .
   * debian/patches/06_fix-tarfile-_buftoinfo-for-python2.5:
     Fixed "incremental backup does not work" for Python 2.5 (LP: #88617)
Files:
 a519da904c90d489459927cd7addf073 757 utils optional duplicity_0.4.2-14ubuntu2.dsc
 5c72964ab92f48257b591ef4306bc038 9137 utils optional duplicity_0.4.2-14ubuntu2.diff.gz
Original-Maintainer: Alexander Zangerl <email address hidden>

duplicity (0.4.2-14ubuntu2) gutsy; urgency=low

  * debian/patches/06_fix-tarfile-_buftoinfo-for-python2.5:
    Fixed "incremental backup does not work" for Python 2.5 (LP: #88617)

 -- dAniel hAhler <email address hidden> Fri, 15 Jun 2007 15:29:08 +0200

To get the fix into feisty-updates, I need two people who acknowledge that it works.
So, please test it by adding
deb http://archive.ubuntu.com/ubuntu/ feisty-proposed universe
to your /etc/apt/sources.list, upgrade duplicity from there and give feedback here.

Success!

"Verify complete: 3736 files compared, 16 differences found."

Earlier, with the original feisty version, (0.4.2-10-1ubuntu1,) that would have read something like "3736 files compared, 3736 differences found."

I can confirm that the 0.4.2-10.1ubuntu1.1 version does in fact fix the problem here. Thank you very much dAniel! This was a show-stopper bug on one of my systems.

Incremental backups now work.

Perhaps I should have included the following output in my last comment <a href="https://bugs.launchpad.net/ubuntu/feisty/+source/duplicity/+bug/88617/comments/33">a moment ago</a>:

"--------------[ Backup Statistics ]--------------
StartTime 1183342165.58 (Mon Jul 2 02:09:25 2007)
EndTime 1183342167.58 (Mon Jul 2 02:09:27 2007)
ElapsedTime 2.00 (2.00 seconds)
SourceFiles 3733
SourceFileSize 169660949 (162 MB)
NewFiles 9
NewFileSize 209738 (205 KB)
DeletedFiles 3
ChangedFiles 4
ChangedFileSize 26006 (25.4 KB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 16
RawDeltaSize 210716 (206 KB)
TotalDestinationSizeChange 124789 (122 KB)
Errors 0
-------------------------------------------------"

Thanks again, dAniel!

I have uninstalled the patched version that I have been running and installed duplicity from feisty-proposed. It is working perfectly for me.

Copied -proposed version into -updates. Thank you!