reduce unnecessary syscall with --exclude-if-present

Bug #1466582 reported by Kuang-che Wu
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Duplicity
Invalid
Undecided
Unassigned

Bug Description

I found "duplicity --exclude-if-present foo" have uncessary syscalls per file.
Even $path is a file, it still try to determine the existence of $path/foo.

I have a backup set with about 300k files. After a complete backup, I ran "duplicity inc --dry-run" and measured the time.
real 203.47
user 157.37
sys 27.35

With my patch, which checks $path/foo only if $path is a dir.
real 191.20 (-6%)
user 149.70 (-4.8%)
sys 22.78 (-16.7%)

Revision history for this message
Kuang-che Wu (kcwu) wrote :
Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

--exclude-if-present foo is not intended for use in that manner. It's intended to allow users to exclude directories by including files within the directories and not having to include the directory names on the command line. So, if I have a couple of directories with files like so, /home/ken/large-data1/.exclude and /home/ken/large-data2/.exclude, and use the option --exclude-if-present .exclude then those directories will be excluded during the backup.

Changed in duplicity:
status: New → Invalid
Revision history for this message
Kuang-che Wu (kcwu) wrote :

I feel you misunderstood this issue. I understand what --exclude-if-present intended to do but its implementation is inefficient.

By using truss on FreeBSD (like strace on Linux), "duplicity --exclude-if-present .no-backup" invokes syscalls like folowing:

lstat("/android/sdk/docs/reference/android/media/AudioAttributes.Builder.html",{ mode=-rw-r----- ,inode=93169,size=91164,blksize=91648 }) = 0 (0x0)
access("/android/sdk/docs/reference/android/media/AudioAttributes.Builder.html",4) = 0 (0x0)
lstat("/android/sdk/docs/reference/android/media/AudioAttributes.Builder.html/.no-backup",0x7fffffffc5d0) ERR#20 'Not a directory'
lstat("/android/sdk/docs/reference/android/media/AudioAttributes.html",{ mode=-rw-r----- ,inode=93170,size=126214,blksize=126464 }) = 0 (0x0)
access("/android/sdk/docs/reference/android/media/AudioAttributes.html",4) = 0 (0x0)
lstat("/android/sdk/docs/reference/android/media/AudioAttributes.html/.no-backup",0x7fffffffc5d0) ERR#20 'Not a directory'
lstat("/android/sdk/docs/reference/android/media/AudioFormat.Builder.html",{ mode=-rw-r----- ,inode=93171,size=87288,blksize=87552 }) = 0 (0x0)
access("/android/sdk/docs/reference/android/media/AudioFormat.Builder.html",4) = 0 (0x0)
lstat("/android/sdk/docs/reference/android/media/AudioFormat.Builder.html/.no-backup",0x7fffffffc5d0) ERR#20 'Not a directory'
lstat("/android/sdk/docs/reference/android/media/AudioFormat.html",{ mode=-rw-r----- ,inode=93172,size=134155,blksize=131072 }) = 0 (0x0)
access("/android/sdk/docs/reference/android/media/AudioFormat.html",4) = 0 (0x0)
lstat("/android/sdk/docs/reference/android/media/AudioFormat.html/.no-backup",0x7fffffffc5d0) ERR#20 'Not a directory'
lstat("/android/sdk/docs/reference/android/media/AudioManager.OnAudioFocusChangeListener.html",{ mode=-rw-r----- ,inode=93173,size=67505,blksize=67584 }) = 0 (0x0)
access("/android/sdk/docs/reference/android/media/AudioManager.OnAudioFocusChangeListener.html",4) = 0 (0x0)
lstat("/android/sdk/docs/reference/android/media/AudioManager.OnAudioFocusChangeListener.html/.no-backup",0x7fffffffc5d0) ERR#20 'Not a directory'

As you can see, duplicity already lstat(AudioAttributes.Builder.html) first, so it should already know it's a regular file. What's the point to lstat(AudioAttributes.Builder.html/.no-backup) ?

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote : Re: [Bug 1466582] Re: reduce unnecessary syscall with --exclude-if-present
Download full text (3.3 KiB)

You are right, I misunderstood. Will get it merged in.

On Fri, Jun 19, 2015 at 6:34 AM, Kuang-che Wu <email address hidden> wrote:

> I feel you misunderstood this issue. I understand what --exclude-if-
> present intended to do but its implementation is inefficient.
>
> By using truss on FreeBSD (like strace on Linux), "duplicity --exclude-
> if-present .no-backup" invokes syscalls like folowing:
>
> lstat("/android/sdk/docs/reference/android/media/AudioAttributes.Builder.html",{
> mode=-rw-r----- ,inode=93169,size=91164,blksize=91648 }) = 0 (0x0)
> access("/android/sdk/docs/reference/android/media/AudioAttributes.Builder.html",4)
> = 0 (0x0)
> lstat("/android/sdk/docs/reference/android/media/AudioAttributes.Builder.html/.no-backup",0x7fffffffc5d0)
> ERR#20 'Not a directory'
> lstat("/android/sdk/docs/reference/android/media/AudioAttributes.html",{
> mode=-rw-r----- ,inode=93170,size=126214,blksize=126464 }) = 0 (0x0)
> access("/android/sdk/docs/reference/android/media/AudioAttributes.html",4)
> = 0 (0x0)
> lstat("/android/sdk/docs/reference/android/media/AudioAttributes.html/.no-backup",0x7fffffffc5d0)
> ERR#20 'Not a directory'
> lstat("/android/sdk/docs/reference/android/media/AudioFormat.Builder.html",{
> mode=-rw-r----- ,inode=93171,size=87288,blksize=87552 }) = 0 (0x0)
> access("/android/sdk/docs/reference/android/media/AudioFormat.Builder.html",4)
> = 0 (0x0)
> lstat("/android/sdk/docs/reference/android/media/AudioFormat.Builder.html/.no-backup",0x7fffffffc5d0)
> ERR#20 'Not a directory'
> lstat("/android/sdk/docs/reference/android/media/AudioFormat.html",{
> mode=-rw-r----- ,inode=93172,size=134155,blksize=131072 }) = 0 (0x0)
> access("/android/sdk/docs/reference/android/media/AudioFormat.html",4) = 0
> (0x0)
> lstat("/android/sdk/docs/reference/android/media/AudioFormat.html/.no-backup",0x7fffffffc5d0)
> ERR#20 'Not a directory'
> lstat("/android/sdk/docs/reference/android/media/AudioManager.OnAudioFocusChangeListener.html",{
> mode=-rw-r----- ,inode=93173,size=67505,blksize=67584 }) = 0 (0x0)
> access("/android/sdk/docs/reference/android/media/AudioManager.OnAudioFocusChangeListener.html",4)
> = 0 (0x0)
> lstat("/android/sdk/docs/reference/android/media/AudioManager.OnAudioFocusChangeListener.html/.no-backup",0x7fffffffc5d0)
> ERR#20 'Not a directory'
>
> As you can see, duplicity already lstat(AudioAttributes.Builder.html)
> first, so it should already know it's a regular file. What's the point
> to lstat(AudioAttributes.Builder.html/.no-backup) ?
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/1466582
>
> Title:
> reduce unnecessary syscall with --exclude-if-present
>
> Status in Duplicity - Bandwidth Efficient Encrypted Backup:
> Invalid
>
> Bug description:
> I found "duplicity --exclude-if-present foo" have uncessary syscalls per
> file.
> Even $path is a file, it still try to determine the existence of
> $path/foo.
>
> I have a backup set with about 300k files. After a complete backup, I
> ran "duplicity inc --dry-run" and measured the time.
> real 203.47
> user 157.37
> sys 27.35
>
> With my patch, which checks $path/foo only if $path is a ...

Read more...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.