Uncached grp and pwd calls make duplicity slow with large group and passwd maps

Attached is a patch that backports this fix to precise.
A 64-bit build is available here for testing: http://people.canonical.com/~arges/lp1013446/

Chris, the debdiff looks good to me. I tested 0.6.18-0ubuntu4 and it fixes the performance issue I was seeing with 0.6.18-0ubuntu3. Thanks!

Thank for the work Chris, we need the bug fixing in raring first before having it backported to precise though, could you work on that?

Shouldn't this fix be proposed upstream to python?

Before this change, we had an exact copy of python 2.7's tarfile.py (with some changes to support python 2.4). Now we can't ever go back to a pure upstream tarfile.py.

And since the change seems to make sense for python itself too...

@mterry,
Seems reasonable, I can help out with this if needed.

For the time being it would still be good to have this in raring.
Is a debdiff for raring needed, or will the raring ubuntu package be synced anytime soon?

I can apply the same patch to raring and then upload this SRU

This bug was fixed in the package duplicity - 0.6.20-0ubuntu3

---------------
duplicity (0.6.20-0ubuntu3) raring; urgency=low

  * debian/patches/07caching.dpatch:
    - Backport caching work done by Steve Atwell, on behalf of Chris J Arges
    (LP: #1013446)
 -- Iain Lane <email address hidden> Tue, 11 Dec 2012 14:52:15 +0000

@laney
Thanks!

This SRU was uploaded to the Precise queue it seems:

https://launchpad.net/ubuntu/precise/+queue?queue_state=1

I'm unsubscribing sponsors - please resubscribe them if and when there is a quantal debdiff to sponsor.

Attached is a version for a quantal SRU.

Chris, it looks like there's a problem with the 0.6.18-0ubuntu4 package that was uploaded to the precise-proposed queue, as mentioned by Brian in comment #9.

It includes debian/patches/07caching.dpatch, but that patch wasn't added to debian/patches/00list, so it's not getting applied.

@satwell
Ok looks like the above quantal patch has this. I'll respin a patch for precise with this.

Here is a fixed debdiff that includes the proper addition to 00list for precise. Please SRU this patch instead.

I'll upload this again, bundling it up with an update I have for duplicity.

Hello Steve, or anyone else affected,

Accepted duplicity into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/duplicity/0.6.19-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Steve, or anyone else affected,

Accepted duplicity into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/duplicity/0.6.18-0ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I have tested the proposed package on my 12.04 (Precise) work desktop which has 93k files in my home. I measured the execution time of a backup without any change on the filesystem multiple times with both the current and the proposed duplicity. On average I experienced ~23% performance increase. I expect it would be even more if I had a larger group/passwd file. I also restored some files and folders, checked owners, permissions, checksums and no regression spotted.

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug was fixed in the package duplicity - 0.6.18-0ubuntu3.1

---------------
duplicity (0.6.18-0ubuntu3.1) precise-proposed; urgency=low

  [ Chris J Arges ]
  * debian/patches/07caching.dpatch:
    - Backport caching work done by Steve Atwell. (LP: #1013446)

  [ Michael Terry ]
  * debian/patches/08volcorruption.dpatch:
    - Fix some data-corruption issues when resuming an interrupted
      backup (LP: #1091269)
 -- Michael Terry <email address hidden> Fri, 11 Jan 2013 11:05:16 -0500

duplicity (0.6.19-0ubuntu2.2) quantal-proposed; urgency=low

  * debian/patches/09volcorruption.dpatch:
    - Backport a fix for a restart test that caused FTBFS on armhf

 -- Michael Terry <email address hidden> Fri, 11 Jan 2013 13:59:19 -0500

duplicity (0.6.19-0ubuntu2.1) quantal-proposed; urgency=low

  [ Michael Terry ]
  * debian/patches/07u1utf8.dpatch:
    - Update with a fix for a UnicodeDecodeError crash (LP: #1080423)
  * debian/patches/09volcorruption.dpatch:
    - Fix some data-corruption issues when resuming an interrupted
      backup (LP: #1091269)

  [ Chris J Arges ]
  * debian/patches/08caching.dpatch:
    - Backport caching work done by Steve Atwell. (LP: #1013446)

 -- Michael Terry <email address hidden> Wed, 05 Dec 2012 16:39:01 -0500