Duplicity

Cannot restore from no-encryption and no-compression backup

Bug #1511308 reported by Devrim Seral on 2015-10-29

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	Duplicity	Fix Released	High	Unassigned	Duplicity 0.7.06

Bug Description

Duplicity version: 0.7.05 or 0.6.23
Python: 2.7.6
OS : Ubuntu 14.04.3 LTS
Type of filesystem : Linux ext4 and ftp
Log;
Using archive dir: /root/.cache/duplicity/674bf12b26a94fde71894638beff7845
Using backup name: 674bf12b26a94fde71894638beff7845
Import of duplicity.backends.azurebackend Succeeded
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.copycombackend Succeeded
Import of duplicity.backends.dpbxbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.hubicbackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.lftpbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.multibackend Succeeded
Import of duplicity.backends.ncftpbackend Succeeded
Import of duplicity.backends.onedrivebackend Succeeded
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.pydrivebackend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.ssh_paramiko_backend Succeeded
Import of duplicity.backends.ssh_pexpect_backend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
LFTP version is 4.4.13
Using temporary directory /tmp/duplicity-U2b0wd-tempdir
Registering (mkstemp) temporary file /tmp/duplicity-U2b0wd-tempdir/mkstemp-qkmvhK-1
SETTINGS:
['set ssl:verify-certificate true\n', 'set ftp:ssl-allow false\n', 'set http:use-propfind true\n', 'set net:timeout 30\n', 'set net:max-retries 5\n', 'set ftp:passive-mode on\n', 'debug\n', "open -u 'bakuser,*****' ftp://10.0.0.199\n"]
Main action: restore
================================================================================
duplicity 0.7.05 ($reldate)
Args: /usr/bin/duplicity -v 9 --no-encryption --no-compression restore ftp://bakuser@10.0.0.199/backup_dir/db_backup/test restore/
Linux sisdb 3.19.0-31-generic #36~14.04.1-Ubuntu SMP Thu Oct 8 10:21:08 UTC 2015 x86_64 x86_64
/usr/bin/python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2]
================================================================================
Registering (mkstemp) temporary file /tmp/duplicity-U2b0wd-tempdir/mkstemp-Z8sYPS-2
Temp has 44162039808 available, backup will use approx 34078720.
CMD: lftp -c 'source '/tmp/duplicity-U2b0wd-tempdir/mkstemp-qkmvhK-1'; cd 'backup_dir/db_backup/test/' || exit 0; ls'
Reading results of 'lftp -c 'source '/tmp/duplicity-U2b0wd-tempdir/mkstemp-qkmvhK-1'; cd 'backup_dir/db_backup/test/' || exit 0; ls''
STDERR:
---- Resolving host address...
---- 1 address found: 10.0.0.199
---- Connecting to 10.0.0.199 (10.0.0.199) port 21
<--- 220 NASFTPD Turbo station 1.3.5a Server (ProFTPD) [10.0.0.199]
---> FEAT
<--- 211-Features:
<--- MFF modify;UNIX.group;UNIX.mode;
<--- REST STREAM
<--- MLST modify*;perm*;size*;type*;unique*;UNIX.group*;UNIX.mode*;UNIX.owner*;
<--- UTF8
<--- LANG en-US*
<--- EPRT
<--- EPSV
<--- MDTM
<--- TVFS
<--- MFMT
<--- SIZE
<--- 211 End
---> LANG
<--- 200 Using default language en_US
---> OPTS UTF8 ON
<--- 200 UTF8 set to on
---> OPTS MLST modify;perm;size;type;UNIX.group;UNIX.mode;UNIX.owner;
<--- 200 OPTS MLST modify;perm;size;type;UNIX.group;UNIX.mode;UNIX.owner;
---> USER bakuser
<--- 331 Password required for bakuser
---> PASS *****
<--- 230 User bakuser logged in
---> PWD
<--- 257 "/" is the current directory
---- CWD path to be sent is `/backup_dir/db_backup/test'
---> CWD /backup_dir/db_backup/test
<--- 250 CWD command successful
---> PASV
<--- 227 Entering Passive Mode (10,0,0,199,219,30).
---- Connecting data socket to (10.0.0.199) port 56094
---- Data connection established
---> LIST
<--- 150 Opening ASCII mode data connection for file list
---- Got EOF on data connection
---- Closing data socket
<--- 226 Transfer complete
---> QUIT
---- Closing control socket

STDOUT:
-rw-rw---- 1 bakuser everyone 20480 Oct 29 11:38 duplicity-full-signatures.20151029T093849Z.sigtar
-rw-rw---- 1 bakuser everyone 202 Oct 29 11:38 duplicity-full.20151029T093849Z.manifest
-rw-rw---- 1 bakuser everyone 615508 Oct 29 11:38 duplicity-full.20151029T093849Z.vol1.difftar

Local and Remote metadata are synchronized, no sync needed.
CMD: lftp -c 'source '/tmp/duplicity-U2b0wd-tempdir/mkstemp-qkmvhK-1'; cd 'backup_dir/db_backup/test/' || exit 0; ls'
Reading results of 'lftp -c 'source '/tmp/duplicity-U2b0wd-tempdir/mkstemp-qkmvhK-1'; cd 'backup_dir/db_backup/test/' || exit 0; ls''
STDERR:
---- Resolving host address...
---- 1 address found: 10.0.0.199
---- Connecting to 10.0.0.199 (10.0.0.199) port 21
<--- 220 NASFTPD Turbo station 1.3.5a Server (ProFTPD) [10.0.0.199]
---> FEAT
<--- 211-Features:
<--- MFF modify;UNIX.group;UNIX.mode;
<--- REST STREAM
<--- MLST modify*;perm*;size*;type*;unique*;UNIX.group*;UNIX.mode*;UNIX.owner*;
<--- UTF8
<--- LANG en-US*
<--- EPRT
<--- EPSV
<--- MDTM
<--- TVFS
<--- MFMT
<--- SIZE
<--- 211 End
---> LANG
<--- 200 Using default language en_US
---> OPTS UTF8 ON
<--- 200 UTF8 set to on
---> OPTS MLST modify;perm;size;type;UNIX.group;UNIX.mode;UNIX.owner;
<--- 200 OPTS MLST modify;perm;size;type;UNIX.group;UNIX.mode;UNIX.owner;
---> USER bakuser
<--- 331 Password required for bakuser
---> PASS *****
<--- 230 User bakuser logged in
---> PWD
<--- 257 "/" is the current directory
---- CWD path to be sent is `/backup_dir/db_backup/test'
---> CWD /backup_dir/db_backup/test
<--- 250 CWD command successful
---> PASV
<--- 227 Entering Passive Mode (10,0,0,199,220,59).
---- Connecting data socket to (10.0.0.199) port 56379
---- Data connection established
---> LIST
<--- 150 Opening ASCII mode data connection for file list
---- Got EOF on data connection
---- Closing data socket
<--- 226 Transfer complete
---> QUIT
---- Closing control socket

3 files exist on backend
4 files exist in cache
Extracting backup chains from list of files: [u'duplicity-full-signatures.20151029T093849Z.sigtar', u'duplicity-full.20151029T093849Z.manifest', u'duplicity-full.20151029T093849Z.vol1.difftar']
File duplicity-full-signatures.20151029T093849Z.sigtar is not part of a known set; creating new set
Ignoring file (rejected by backup set) 'duplicity-full-signatures.20151029T093849Z.sigtar'
File duplicity-full.20151029T093849Z.manifest is not part of a known set; creating new set
File duplicity-full.20151029T093849Z.vol1.difftar is part of known set
Found backup chain [Thu Oct 29 11:38:49 2015]-[Thu Oct 29 11:38:49 2015]
Last full backup date: Thu Oct 29 11:38:49 2015
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /root/.cache/duplicity/674bf12b26a94fde71894638beff7845

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Thu Oct 29 11:38:49 2015
Chain end time: Thu Oct 29 11:38:49 2015
Number of contained backup sets: 1
Total number of contained volumes: 1
Type of backup set: Time: Num volumes:
Full Thu Oct 29 11:38:49 2015 1
-------------------------
No orphaned or incomplete backup sets found.
Registering (mktemp) temporary file /tmp/duplicity-U2b0wd-tempdir/mktemp-1QZDXI-3
CMD: lftp -c 'source '/tmp/duplicity-U2b0wd-tempdir/mkstemp-qkmvhK-1'; get 'backup_dir/db_backup/test/duplicity-full.20151029T093849Z.vol1.difftar' -o '/tmp/duplicity-U2b0wd-tempdir/mktemp-1QZDXI-3''
Reading results of 'lftp -c 'source '/tmp/duplicity-U2b0wd-tempdir/mkstemp-qkmvhK-1'; get 'backup_dir/db_backup/test/duplicity-full.20151029T093849Z.vol1.difftar' -o '/tmp/duplicity-U2b0wd-tempdir/mktemp-1QZDXI-3'''
STDERR:
---- Resolving host address...
---- 1 address found: 10.0.0.199
---- Connecting to 10.0.0.199 (10.0.0.199) port 21
<--- 220 NASFTPD Turbo station 1.3.5a Server (ProFTPD) [10.0.0.199]
---> FEAT
<--- 211-Features:
<--- MFF modify;UNIX.group;UNIX.mode;
<--- REST STREAM
<--- MLST modify*;perm*;size*;type*;unique*;UNIX.group*;UNIX.mode*;UNIX.owner*;
<--- UTF8
<--- LANG en-US*
<--- EPRT
<--- EPSV
<--- MDTM
<--- TVFS
<--- MFMT
<--- SIZE
<--- 211 End
---> LANG
<--- 200 Using default language en_US
---> OPTS UTF8 ON
<--- 200 UTF8 set to on
---> OPTS MLST modify;perm;size;type;UNIX.group;UNIX.mode;UNIX.owner;
<--- 200 OPTS MLST modify;perm;size;type;UNIX.group;UNIX.mode;UNIX.owner;
---> USER bakuser
<--- 331 Password required for bakuser
---> PASS *****
<--- 230 User bakuser logged in
---> PWD
<--- 257 "/" is the current directory
---> TYPE I
<--- 200 Type set to I
---> SIZE backup_dir/db_backup/test/duplicity-full.20151029T093849Z.vol1.difftar
<--- 213 615508
---> MDTM backup_dir/db_backup/test/duplicity-full.20151029T093849Z.vol1.difftar
<--- 213 20151029093850
---> PASV
<--- 227 Entering Passive Mode (10,0,0,199,219,156).
---- Connecting data socket to (10.0.0.199) port 56220
---- Data connection established
---> RETR backup_dir/db_backup/test/duplicity-full.20151029T093849Z.vol1.difftar
<--- 150 Opening BINARY mode data connection for backup_dir/db_backup/test/duplicity-full.20151029T093849Z.vol1.difftar (615508 bytes)
---- Got EOF on data connection
---- Closing data socket
<--- 226 Transfer complete
---> QUIT
---- Closing control socket

STDOUT:

Deleting /tmp/duplicity-U2b0wd-tempdir/mktemp-1QZDXI-3
Forgetting temporary file /tmp/duplicity-U2b0wd-tempdir/mktemp-1QZDXI-3
Processed volume 1 of 1
No files found in archive - nothing restored.
Releasing lockfile <lockfile.LinkFileLock instance at 0x7ff4a5ae4368>
Removing still remembered temporary file /tmp/duplicity-U2b0wd-tempdir/mkstemp-qkmvhK-1
Removing still remembered temporary file /tmp/duplicity-U2b0wd-tempdir/mkstemp-Z8sYPS-2

Solution : While i was rename duplicity-full.20151029T093849Z.vol1.difftar as duplicity-full.20151029T093849Z.vol1.difftar.gz restore is working (Credit: http://serverfault.com/questions/726525/cannot-restore-duplicity-backup )

Kenneth Loafman (kenneth-loafman) on 2015-10-30

Changed in duplicity:
status:	New → In Progress
assignee:	nobody → Kenneth Loafman (kenneth-loafman)
importance:	Undecided → High
milestone:	none → 0.7.06

Kenneth Loafman (kenneth-loafman) on 2015-11-02

summary:

- duplicity 0.7.05 cannot restore from ftp backup
+ Cannot restore from no-encryption and no-compression backup

Kenneth Loafman (kenneth-loafman) on 2015-11-05

Changed in duplicity:
assignee:	Kenneth Loafman (kenneth-loafman) → nobody
status:	In Progress → Fix Committed

Kenneth Loafman (kenneth-loafman) on 2015-12-07

Changed in duplicity:
status:	Fix Committed → Fix Released

Revision history for this message

mirak (mirak-mirak) wrote on 2016-09-03:

I encountered this bug, and someone here found a solution :
http://serverfault.com/questions/726525/cannot-restore-duplicity-backup

The bug is that duplicity wasn't compressing at all despite the --no-compression option, so the files are compressed despite not having the .gz extension.

Renaming .sigtar to sigtar.gz and all .difftar to .difftar.gz fixes the damn problem.

$ for i in *.difftar;do mv -v $i $i.gz;done
$ for i in *.sigtar;do mv -v $i $i.gz;done

Revision history for this message

mirak (mirak-mirak) wrote on 2016-09-03:

I think duplicity should tell a message like "can't find .difftar.gz files", instead of saying "No files found in archive - nothing restored."

Also the bugfix should make duplicity check for a compressed .difftar anyway if it can't find a .difftar.gz

Revision history for this message

Alex N. (a-nox) wrote on 2016-12-06:

Still unsolved for me, same issue here with duplicity 0.7.06-2ubuntu2.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.