Globbing patterns fail to include some files if include contains "*" or "**"

Bug #884371 reported by Elifarley Cruz
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Duplicity
Fix Released
Medium
Unassigned

Bug Description

Duplicity doesn't include some files if "**" is used as prefix.

Duplicity 0.6.15
Python 2.7.2
ArchLinux x64
Linux filesystem

In the example below, I would expect to have all 3 files from folder-c included in the backup, but they are not:

--------------------------------------------------

BASE=/tmp
mkdir -p $BASE/test-dup/folder-{a,b,c}

touch $BASE/test-dup/file-{1.txt,2.jpg,3.tmp}
touch $BASE/test-dup/folder-a/file-{4.txt,5.jpg,6.tmp}
touch $BASE/test-dup/folder-b/file-{7.txt,8.jpg,9.tmp}
touch $BASE/test-dup/folder-c/file-{10.txt,11.jpg,12.tmp}

tree /tmp/test-dup/
# result:
/tmp/test-dup/
+-- file-1.txt
+-- file-2.jpg
+-- file-3.tmp
+-- folder-a
¦ +-- file-4.txt
¦ +-- file-5.jpg
¦ +-- file-6.tmp
+-- folder-b
¦ +-- file-7.txt
¦ +-- file-8.jpg
¦ +-- file-9.tmp
+-- folder-c
    +-- file-10.txt
    +-- file-11.jpg
    +-- file-12.tmp

cat my-file-list
# result:
- **/test-dup/folder-a
**/test-dup/folder-b/*.txt
- **/test-dup/folder-b

duplicity --name test-dup --include-globbing-filelist my-file-list
$BASE/test-dup/ file://$BASE/test-dup-backup

duplicity --name test-dup list-current-files file://$BASE/test-dup-backup
# result:
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon Oct 31 13:31:33 2011
Mon Oct 31 13:15:04 2011 .
Mon Oct 31 13:15:04 2011 folder-b
Mon Oct 31 13:15:04 2011 folder-b/file-7.txt

Tags: selection

Related branches

description: updated
Revision history for this message
Elifarley Cruz (elifarley) wrote :

Here's some actual test code to show the bug.
This bug only happens when I use '**' as prefix, instead of 'testfiles'. Please see below for details.

This method already exists in selectiontest.py, and it works just fine:
    def test_globbing_filelist(self):
        """Filelist glob test similar to above testParse2"""
        self.ParseTest([("--include-globbing-filelist", "file")],
                       [(), ('1',), ('1', '1'), ('1', '1', '2'),
                        ('1', '1', '3')],
                       ["""
- testfiles/select/1/1/1
testfiles/select/1/1
- testfiles/select/1
- **
"""])

I have added this method, which also works:

    def test_globbing_filelist_bug_884371_a(self):
        """Filelist glob test related to bug 884371"""
        self.ParseTest([("--include-globbing-filelist", "file")],
                       [(), ('1',), ('1', '1'), ('1', '1', '1'),
                        ('1', '1', '2'), ('1', '1', '3')],
                       ["""
**/select/1/1
- **
"""])

The one below doesn't work, and it's simply a modified version of 'test_globbing_filelist' (with the string 'testfiles' replaced by '**'):

    def test_globbing_filelist_bug_884371_b(self):
        """
  Modified version of 'test_globbing_filelist'
  (with the string 'testfiles' replaced by '**')
  """
        self.ParseTest([("--include-globbing-filelist", "file")],
                       [(), ('1',), ('1', '1'), ('1', '1', '2'),
                        ('1', '1', '3')],
                       ["""
- **/select/1/1/1
**/select/1/1
- **/select/1
- **
"""])

Here are the results of the tests:

========================================
test_globbing_filelist (__main__.ParseArgsTest)
Filelist glob test similar to above testParse2 ...
self.Select:
()
('1',)
('1', '1')
('1', '1', '2')
('1', '1', '3')
ok
test_globbing_filelist_bug_884371_a (__main__.ParseArgsTest)
Filelist glob test related to bug 884371 ...
self.Select:
()
('1',)
('1', '1')
('1', '1', '1')
('1', '1', '2')
('1', '1', '3')
ok
test_globbing_filelist_bug_884371_b (__main__.ParseArgsTest)
Filelist glob test related to bug 884371 ...
self.Select:
()
End when i2 = ('1',)
FAIL
========================================

summary: - Globbing patterns fail to include some files
+ Globbing patterns fail to include some files if prefix is "**"
description: updated
Revision history for this message
Elifarley Cruz (elifarley) wrote : Re: Globbing patterns fail to include some files if prefix is "**"
Changed in duplicity:
importance: Undecided → Medium
milestone: none → 0.6.18
status: New → Triaged
Changed in duplicity:
milestone: 0.6.18 → none
Revision history for this message
Aaron Whitehouse (aaron-whitehouse) wrote :

I've just added a bunch of test cases to the linked branch to dig into this in more detail.

It looks to me as though neither * nor ** are processed correctly if they are included anywhere in an include line, whether this is in a globbing-filelist or in an --include on the commandline.

* and ** appear to work as expected if they are in an exclude line of a filelist or in an --exclude on the commandline.

summary: - Globbing patterns fail to include some files if prefix is "**"
+ Globbing patterns fail to include some files if include contains "**"
summary: - Globbing patterns fail to include some files if include contains "**"
+ Globbing patterns fail to include some files if include contains "*" or
+ "**"
Changed in duplicity:
assignee: nobody → Aaron Whitehouse (aaron-whitehouse)
Changed in duplicity:
assignee: Aaron Whitehouse (aaron-whitehouse) → nobody
milestone: none → 0.7.04
status: Triaged → Fix Committed
Changed in duplicity:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.