deja-dup fails to restore missing files, that contain "german umlaut" (ä,ö,ü,Ä,Ö,Ü).

Status changed to 'Confirmed' because the bug affects multiple users.

Nearly the same issue here, running on archlinux though with Version 32.0-1

I have the same problem, but with the letters "Ö" and "ü".

My short-term workaround to recover old backup files with umlauts after UnicodeDecodeError:

export LANG=en_US.utf8
export LANGUAGE=
export LC_CTYPE="en_US.utf8"
export LC_NUMERIC="en_US.utf8"
export LC_TIME="en_US.utf8"
export LC_COLLATE="en_US.utf8"
export LC_MONETARY="en_US.utf8"
export LC_MESSAGES="en_US.utf8"
export LC_PAPER=en_US.UTF-8
export LC_NAME="en_US.utf8"
export LC_ADDRESS="en_US.utf8"
export LC_TELEPHONE="en_US.utf8"
export LC_MEASUREMENT="en_US.utf8"
export LC_IDENTIFICATION="en_US.utf8"
export LC_ALL=

deja-dup --restore folderWithUmlauts

maybe someone should set the Importance. It seems that this bug will affect all German, Danish, Norwegian and Swedish user and can result in Data corruption?

..the same here under Ubuntu 14.04 (64.-bit).
Trying to restore some files including the german "Umlaut" (that means the letters ä;ü;ö....) ends with the error message from the first thread. Maybe, it should be checked if it's the same with some french letters like ê,ë,é,è, ç.

Otherwise, with this bug it is impossible to use this good software for german, french and northern users.
It sould be fixed asap...

Hello.

I want to bring to your knowledge, that we (in the german-commmunity) still see a lot of people having this problem.

If you want some reference, you can find a new theme here: https://forum.ubuntuusers.de/topic/datensicherung-fehlgeschlagen-was-tun (in german).

With this bug, I can't recommend usage of Déjà-Dup for anyone speaking german. This is very bad, because this problem is the preinstalled software in Ubuntu and used to be the most user-friendly software in the past.

Please set the priority and fix it or tell me, what to do against it.

Best Regards

Vej

(Supporter of the (unofficial) german supportforum for backups: https://forum.ubuntuusers.de/forum/backups/)

PS.: If you want to have some discussion, feel free to contact me via my profile.

Same with me as with Chris - so at the moment some of my backed-up files seem to be "lost".... :(

This Problem is since more as one year, incredible...

Same problem! Unuseable for me!

Please fix it.

Thanks and best Regards

This hasn't been on my radar, sorry. This looks more like a duplicity bug than a deja-dup one, will reassign. I might get some time to look at it later this week.

Thank you!

OK.... So this bug is a dup of 1356548. It's fixed upstream and in versions of Ubuntu past 14.04.

I'm going to mark this bug as a dup and consider backporting via SRU. Discussion can continue in that bug.

It seems this is not an exact duplicate of bug #1356548.

Even after that bug is fixed upstream it is not possible to/leads to errors when trying to do the following

1. go to the folder you backed up before (must contain a folder/file with umlauts e.g. "töstfolder")
2. launch: /usr/bin/deja-dup --restore-missing .
3. you will see malformed foldername/filename in the restore window which tells you that these entries are missing (which is wrong)
4. the restore of such a malformed file will also fail

Can anyone confirm this?

Greets

Hello Felix Haller,

did you used trusty-proposed for your test?

I tried your setup as well but was not able to see anything.

See bug #1356548 comment #16 for details.

Best Regards

Vej

Hello everyone!

I had a look at the diff from the fix of bug #1356548 and they only fixed the occurring of the Traceback in case the file was not found.

See this cite from comment #6 of this bug:

> You were experiencing the crash when duplicity was trying to say that the file you asked for didn't exist in the backup.
> When I tried restoring utf8 filenames that did exist in the backup, it worked fine.

So this is definitely not an duplicate of bug #1356548.

Best Regards

Vej

According to https://wiki.ubuntu.com/Bugs/Importance this bug must be marked as critcal!

Is bug #1406505 a duplicate of this bug?

@19 I agree, and marked bug #1406505 as a duplicate.

Thanks for pointing this out. This duplicate has some helpful additional informations.

I did some further tests with actual versions of deja-dup and found out, that the following restores does not work (I'm trying to restore a file named `dortö`, that is located in `Schreibtisch/tröstfolder` (''Schreibtisch'' means ''Desktop'', and ''tröstfolder'' is just some folder with an ö inside) :

Reproduction Steps:

0. Install your Ubuntu in german.
1. Create the folder Schreibtisch/tröstfolder and the file Schreibtisch/tröstfolder/dortö
2. Perform a fresh Backup to an empty folder (use encryption by setting a password).
3. Delete the file.
4. Try out the following commands:

{{{LC_ALL=C deja-dup --restore ~/Schreiptich/tröstfolder/dortö}} and {{{LC_ALL=C deja-dup --restore ~/Schreiptich/tröstfolder/dortö}}

work as expected.

{{{LC_ALL=C deja-dup --restore-missing tröstfolder/}} and
{{{deja-dup --restore-missing tröstfolder/da}}

result in empty lists of files.

I decrypted the backup using `gpg --multifile --decrypt duplicity-full.*.*.difftar.gpg` and unpacked the files manually using `for t in duplicity-full.20110127T131352Z.*.difftar; do tar xf $t; done` (see [[https://wiki.gnome.org/Apps/DejaDup/Help/Restore/WorstCase| "When Everything Goes Wrong" in Gnome Wiki ]]). `dortö` was inside of the folder `snapshot/home/test/Schreibtisch/tröstfolder`.

Versions: deja-dup `30.0-0ubuntu4` and duplicity `0.6.23-1ubuntu4.2`
OS.: Ubuntu Trusty Thar

Please Note: I installed the version `0.6.23-1ubuntu4.2` of duplicity from proposed, to avoid being hit by bug #1356548.

Vej, thanks for your clear instructions. And thanks to Martin Lex for poking me about this bug. I think I've fixed this in trunk. Hopefully will have a source release and an update to xenial shortly.

Once this hits xenial, I'd appreciate reports of whether that fixes it for people and they don't see any additional weirdness. Then I can prepare an SRU for trusty.

This bug was fixed in the package deja-dup - 34.2-0ubuntu1

---------------
deja-dup (34.2-0ubuntu1) xenial; urgency=medium

  * New bug-fix upstream release:
    - Fixes restoring some files with unicode characters (LP: #1377873)
  * debian/patch/series:
    - Disable appstream patch, upstream merged it

 -- Michael Terry <email address hidden> Thu, 07 Apr 2016 12:11:31 -0400

Hello Michael,

thanks for your efforts in fixing this bug.

I will try tomorrow, with today's daily build of Xenial and let you know about the results.

Hello everyone,

the version 34.2-0ubuntu1 performs good if you follow my scenario given in comment #21. Thanks to Michael Terry for that.

But if you do the following, you may encounter problems:

0. Install the dailybuild from http://cdimage.ubuntu.com/daily-live/current/ (I choosed the one created at 08-Apr-2016 08:03), choose German ("Deutsch") as your language.
1. Perform a fresh backup to a local folder (set a password for encryption).
2. Create the folder Schreibtisch/tröstfolder and the file Schreibtisch/tröstfolder/dortö.
3. Restart the Backup (perform an incremental Backup)
4. Delete the file.
5. Restart the Backup (perform an incremental Backup)
6. Try to restore the file using LC_ALL=C deja-dup --restore-missing tröstfolder/

Result: The list is empty.

However: If you leave out step five and try to restore the file after it was deleted, it works very well :).

So I'm wondering now if the "restore-missing"-function is expected to look at the last backup only. Can anyone explain this to me?

Thanking you in anticipation

Vej

Vej, thanks for testing! I will try testing that scenario. I *thought* we looked further back than just the latest backup. I will confirm.

Maybe it make sense to open a new Bug for #26?

I think this fix introduce a regression:

Running on my (debian) desktop the following command: deja-dup --restore-missing file:///home/bigon

With 34.1 I've more files/directories proposed than when using 34.2.

Most of the missing files are the one with accentuated character.

Locale: fr_BE.utf8

Hello Mr. Bigonville,

thanks for your contribution to this bug.

You wrote: "With 34.1 I've more files/directories proposed than when using 34.2."

This seems legit to me, if the accentuated files where proposed in 34.1 by mistake (because the comparison with the Unicode characters went wrong).

So could you please specify this? I would be interested in the following questions:
 - Are the files, that are not proposed in 34.2 actually missing in the folder/home/bigon ?
 - Were the missing files deleted before or after the last backup?
 - Was the last backup full or incremental?
 - Did you made a fresh clean backup with 34.2 or did you used an old one (initially created with 34.1 or older)?

I think I spoke too fast. Sorry for the noise :(

Hello Michael Terry,

did you managed to test #26?

Are you able to confirm (as mentioned in #27)?

Best Regards

Vej

Michael?

This but MUST be marked as critical...

The problem still exists on Ubuntu 16.04.

The problem stills exists on Ubuntu 16.04 when looking for --file-changed (version of files).