exclude-globbing-filelist does not backup correct files with --progress option

Hi,
I think this problem is result from reading twice filelist file.

For example:
~/tmp/test-> tree -a in
in
├── .config
│   └── test1-in-config.txt
├── test2.txt
└── test3.txt

~/tmp/test-> cat filelist.txt
- /home/k/tmp/test/in/.config

duplicity -vi --progress --no-encryption --exclude-globbing-filelist filelist.txt /home/k/tmp/test/in/ file:///home/k/tmp/test/out/
Using archive dir: /home/k/.cache/duplicity/b5e2f4b829db69341672537d1ced3377
Using backup name: b5e2f4b829db69341672537d1ced3377
[....]
Reading globbing filelist filelist.txt

[******** First reading file list]

Main action: inc
================================================================================
duplicity 0.6.23 (January 24, 2014)
Args: /usr/bin/duplicity -vi --progress --no-encryption --exclude-globbing-filelist filelist.txt /home/k/tmp/test/in/ file:///home/k/tmp/test/out/
Linux kntbk 3.13-1-686-pae #1 SMP Debian 3.13.5-1 (2014-03-04) i686
/usr/bin/python 2.7.6 (default, Feb 26 2014, 06:42:04)
[GCC 4.8.2]
================================================================================
Using temporary directory /tmp/duplicity-iljo9e-tempdir
Temp has 47721136128 available, backup will use approx 34078720.
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
A .
A test2.txt
A test3.txt

[******** In this point file list is correct, below - second reading filelist ]

Reading globbing filelist filelist.txt
Using temporary directory /home/k/.cache/duplicity/b5e2f4b829db69341672537d1ced3377/duplicity-HXwYmn-tempdir
Using temporary directory /home/k/.cache/duplicity/b5e2f4b829db69341672537d1ced3377/duplicity-wbRfbF-tempdir
AsyncScheduler: instantiating at concurrency 0
A .
A .config
A .config/test1-in-config.txt
A test2.txt
A test3.txt

[******** in this point exclude list is empty]
[...]

IMHO - file with exclude list is simple not reopened.

Similar problem:
duplicity --progress --no-encryption --exclude-filelist filelist.txt /home/k/tmp/test/in/ file:///home/k/tmp/test/out/

Reading filelist filelist.txt
Sorting filelist filelist.txt
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Thu Mar 13 18:26:27 2014
Reading filelist filelist.txt
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1495, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1489, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1338, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1471, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 623, in incremental_backup
    commandline.set_selection()
  File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py", line 891, in set_selection
    sel.ParseArgs(select_opts, select_files)
  File "/usr/lib/python2.7/dist-packages/duplicity/selection.py", line 235, in ParseArgs
    filelists[filelists_index], 0, arg))
  File "/usr/lib/python2.7/dist-packages/duplicity/selection.py", line 313, in filelist_get_sf
    self.filelist_read(filelist_fp, inc_default, filelist_name)
  File "/usr/lib/python2.7/dist-packages/duplicity/selection.py", l...

Can you please check to see if there is a trailing space at the end of the line in your filelist?

I am experiencing the same problem and I've found it is triggered by the --progress flag.

The following demonstrates the problem:
location=~/duplicity-test
mkdir -p "$location"/{.cache,src,dst}
mkdir -p "$location"/src/{dir1,dir2}/{subdir1,subdir2}
find "$location/src" -type d -exec touch {}/testfile \;
echo "- $location/src/dir2/subdir2" > "$location/exclude"
duplicity --archive-dir "$location/.cache" --name test_backup --no-encryption --verbosity 5 --exclude-globbing-filelist "$location/exclude" "$location/src" file://"$location/dst"
Using archive dir: /home/scott/duplicity-test/.cache/test_backup
Using backup name: test_backup
Import of duplicity.backends.botobackend Succeeded
Import of duplicity.backends.cfbackend Succeeded
Import of duplicity.backends.copycombackend Succeeded
Import of duplicity.backends.dpbxbackend Failed: No module named dropbox
Import of duplicity.backends.ftpbackend Succeeded
Import of duplicity.backends.gdocsbackend Succeeded
Import of duplicity.backends.giobackend Succeeded
Import of duplicity.backends.hsibackend Succeeded
Import of duplicity.backends.imapbackend Succeeded
Import of duplicity.backends.localbackend Succeeded
Import of duplicity.backends.megabackend Succeeded
Import of duplicity.backends.par2backend Succeeded
Import of duplicity.backends.rsyncbackend Succeeded
Import of duplicity.backends.sshbackend Succeeded
Import of duplicity.backends.swiftbackend Succeeded
Import of duplicity.backends.sxbackend Succeeded
Import of duplicity.backends.tahoebackend Succeeded
Import of duplicity.backends.webdavbackend Succeeded
Reading globbing filelist /home/scott/duplicity-test/exclude
Main action: inc
================================================================================
duplicity 0.7.0 ($reldate)
Args: /usr/bin/duplicity --archive-dir /home/scott/duplicity-test/.cache --name test_backup --no-encryption --verbosity 5 --exclude-globbing-filelist /home/scott/duplicity-test/exclude /home/scott/duplicity-test/src file:///home/scott/duplicity-test/dst
Linux ddsmash 3.16.0-28-generic #38-Ubuntu SMP Sat Dec 13 16:13:28 UTC 2014 x86_64 x86_64
/usr/bin/python 2.7.6 (default, Mar 22 2014, 22:59:56)
[GCC 4.8.2]
================================================================================
Using temporary directory /tmp/duplicity-gzjWQ7-tempdir
Temp has 35840032768 available, backup will use approx 34078720.
Local and Remote metadata are synchronised, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
Using temporary directory /home/scott/duplicity-test/.cache/test_backup/duplicity-xqRenW-tempdir
Using temporary directory /home/scott/duplicity-test/.cache/test_backup/duplicity-y6w8cZ-tempdir
AsyncScheduler: instantiating at concurrency 0
A .
A dir1
A dir1/subdir1
A dir1/subdir1/testfile
A dir1/subdir2
A dir1/subdir2/testfile
A dir1/testfile
A dir2
A dir2/subdir1
A dir2/subdir1/testfile
A dir2/testfile
A testfile
AsyncScheduler: running task synchronously (asynchronicity disabled)
Writing duplicity-full.20150108T185804Z.vol1.difftar.gz
Deleting /tmp/duplicity-gzjWQ7-tempdir/mktemp-LZ1B2R-2
AsyncScheduler: task completed successfully
Processed volume 1
--------------[ Backup...

Thanks both for such a comprehensive report/comments. Much to my surprise, --progress does seem to cause the --exclude-globbing-filelist to backup the incorrect files. I have added an automated test showing this behaviour in the linked branch:
lp:~aaron-whitehouse/duplicity/progress_option_error

Jorman, can you please run your example again without the --progress option and see if this makes any difference?

This bug happens because in the first dry-run, Selection closes the filelist, so it can't read again in the second, real run.
I needed a quick fix for this, so I created a function that resets select_files in commandline.py and called it between each run in duplicity.