Merge lp:~ed.so/duplicity/lftp.ncftp.and.prefixes into lp:~duplicity-team/duplicity/0.7-series

[edso]

- retire --ssh-backend, --use-scp parameters
- introduce scheme prefixes for alternative backend selection e.g. ncftp+ftp://, see manpage
- scp is now selected via scheme e.g. scp://
- added lftp fish, webdav(s), sftp support

[Kenneth Loafman (kenneth-loafman)]

I like the general cleanup.

A couple of changes:

Please leave the file naming as it was. I had set these to be underscore-first to show they were called by the main ssh backend. See _boto*, _cf*.

I would like to see sftp: and scp: protocols go away. In reality they are just subsets of the OpenSSH (or other) package. You really can't run just scp: since we need 'ls' and 'del'. If we continue to allow these we need to emphasize that these are just aliases for ssh:.

review: Needs Fixing

[edso (ed.so)]

On 10.11.2014 17:00, Kenneth Loafman wrote:
> Review: Needs Fixing
>
> I like the general cleanup.
>
> A couple of changes:
>
> Please leave the file naming as it was. I had set these to be underscore-first to show they were called by the main ssh backend. See _boto*, _cf*.

with my mods there is no ssh backend anymore. backends can provide by registering scheme backend+sftp, backend+scp .. the default backends simply get assigned the prefixless scheme in addition.

as these backends are now equal i removed the underscore.. for the future i planned to remove the backend switching parameters for boto/cf as well and streamline it into the use of prefixes instead.

> I would like to see sftp: and scp: protocols go away.

why?

>In reality they are just subsets of the OpenSSH (or other) package. You really can't run just scp: since we need 'ls' and 'del'. If we continue to allow these we need to emphasize that these are just aliases for ssh:.
>

not true. paramiko has a proper scp support using _no_ (open)ssh binaries and listing via ssh remote shell, as far as i have seen.

i see no problem listing openssh as a requirement (like for the pexpect backend) for lftp+sftp then.

..ede

[Kenneth Loafman (kenneth-loafman)]

OK. My confusion. I was looking at the old sshbackend.py for reference.

I deleted the old sshbackend.py since it's no long valid and fixed a couple of format issues.

[edso (ed.so)]

On 10.11.2014 21:08, Kenneth Loafman wrote:
> OK. My confusion. I was looking at the old sshbackend.py for reference.
>
> I deleted the old sshbackend.py since it's no long valid and fixed a couple of format issues.
>

nP.. i actually meant to delete sshbackend.py in the branch but i seem to have forgotten to commit it..

what were the format issues you mention? ..ede

[Kenneth Loafman (kenneth-loafman)]

No line feed at end of file.

Python does not need backslash on list continuation.

That kind of thing.

On Mon, Nov 10, 2014 at 2:18 PM, edso <email address hidden> wrote:

> On 10.11.2014 21:08, Kenneth Loafman wrote:
> > OK. My confusion. I was looking at the old sshbackend.py for reference.
> >
> > I deleted the old sshbackend.py since it's no long valid and fixed a
> couple of format issues.
> >
>
> nP.. i actually meant to delete sshbackend.py in the branch but i seem to
> have forgotten to commit it..
>
> what were the format issues you mention? ..ede
>
> --
>
> https://code.launchpad.net/~ed.so/duplicity/lftp.ncftp.and.prefixes/+merge/240149
> You are reviewing the proposed merge of lp:~
> ed.so/duplicity/lftp.ncftp.and.prefixes into lp:duplicity.
>

[edso (ed.so)]

ok, good to know.. thanks ede

On 10.11.2014 21:57, Kenneth Loafman wrote:
> No line feed at end of file.
>
> Python does not need backslash on list continuation.
>
> That kind of thing.
>
>
> On Mon, Nov 10, 2014 at 2:18 PM, edso <email address hidden> wrote:
>
>> On 10.11.2014 21:08, Kenneth Loafman wrote:
>>> OK. My confusion. I was looking at the old sshbackend.py for reference.
>>>
>>> I deleted the old sshbackend.py since it's no long valid and fixed a
>> couple of format issues.
>>>
>>
>> nP.. i actually meant to delete sshbackend.py in the branch but i seem to
>> have forgotten to commit it..
>>
>> what were the format issues you mention? ..ede
>>
>> --
>>
>> https://code.launchpad.net/~ed.so/duplicity/lftp.ncftp.and.prefixes/+merge/240149
>> You are reviewing the proposed merge of lp:~
>> ed.so/duplicity/lftp.ncftp.and.prefixes into lp:duplicity.
>>
>

[Kenneth Loafman (kenneth-loafman)]

By no line feed at end of file, I mean there were a couple of missing line
feeds at the end of the file. Easy to get meanings reversed.

On Tue, Nov 11, 2014 at 3:45 AM, edso <email address hidden> wrote:

> ok, good to know.. thanks ede
>
> On 10.11.2014 21:57, Kenneth Loafman wrote:
> > No line feed at end of file.
> >
> > Python does not need backslash on list continuation.
> >
> > That kind of thing.
> >
> >
> > On Mon, Nov 10, 2014 at 2:18 PM, edso <email address hidden> wrote:
> >
> >> On 10.11.2014 21:08, Kenneth Loafman wrote:
> >>> OK. My confusion. I was looking at the old sshbackend.py for
> reference.
> >>>
> >>> I deleted the old sshbackend.py since it's no long valid and fixed a
> >> couple of format issues.
> >>>
> >>
> >> nP.. i actually meant to delete sshbackend.py in the branch but i seem
> to
> >> have forgotten to commit it..
> >>
> >> what were the format issues you mention? ..ede
> >>
> >> --
> >>
> >>
> https://code.launchpad.net/~ed.so/duplicity/lftp.ncftp.and.prefixes/+merge/240149
> >> You are reviewing the proposed merge of lp:~
> >> ed.so/duplicity/lftp.ncftp.and.prefixes into lp:duplicity.
> >>
> >
>
> --
>
> https://code.launchpad.net/~ed.so/duplicity/lftp.ncftp.and.prefixes/+merge/240149
> You are reviewing the proposed merge of lp:~
> ed.so/duplicity/lftp.ncftp.and.prefixes into lp:duplicity.
>

=== modified file 'bin/duplicity.1'
--- bin/duplicity.1	2014-10-23 11:56:13 +0000
+++ bin/duplicity.1	2014-10-30 18:01:04 +0000
@@ -68,22 +68,14 @@
 .B Rackspace CloudFiles Pyrax API
 - http://docs.rackspace.com/sdks/guide/content/python.html
 .TP
-.B "dpbx backend" (Dropbox)
+.BR "dpbx backend" " (Dropbox)"
 .B Dropbox Python SDK
 - https://www.dropbox.com/developers/reference/sdk
 .TP
-.B "copy backend" (Copy.com)
+.BR "copy backend" " (Copy.com)"
 .B python-urllib3
 - https://github.com/shazow/urllib3
 .TP
-.B "ftp backend"
-.B NcFTP Client
-- http://www.ncftp.com/
-.TP
-.B "ftps backend"
-.B LFTP Client
-- http://lftp.yar.ru/
-.TP
 .BR "gdocs backend" " (Google Docs)"
 .B Google Data APIs Python Client Library
 - http://code.google.com/p/gdata-python-client/
@@ -95,17 +87,25 @@
 .B D-Bus
 (dbus)- http://www.freedesktop.org/wiki/Software/dbus
 .TP
-.B "rsync backend"
-.B rsync client binary
-- http://rsync.samba.org/
+.BR "lftp backend" " (needed for ftp, ftps, fish [over ssh] - also supports sftp, webdav[s])"
+.B LFTP Client
+- http://lftp.yar.ru/
 .TP
 .BR "mega backend" " (mega.co.nz)"
 .B Python library for mega API
 - https://github.com/ckornacker/mega.py, ubuntu ppa - ppa:ckornacker/backup
 .TP
+.BR "ncftp backend" " (ftp, select via ncftp+ftp://)"
+.B NcFTP
+- http://www.ncftp.com/
+.TP
 .B "Par2 Wrapper Backend"
 .B par2cmdline
 - http://parchive.sourceforge.net/
+.TP
+.B "rsync backend"
+.B rsync client binary
+- http://rsync.samba.org/
 .PP
 There are two
 .B ssh backends
@@ -1067,87 +1067,169 @@
 or preceded by a double slash, '//path', to represent an absolute
 filesystem path.
 .PP
+.B Note:
+.RS
+Scheme (protocol) access may be provided by more than one backend.
+In case the default backend is buggy or simply not working in a specific case it might be worth trying an alternative implementation.
+Alternative backends can be selected by prefixing the scheme with the name of the alternative backend e.g.
+.B ncftp+ftp://
+and are mentioned below the scheme's syntax summary.
+.RE
+
+.PP
 Formats of each of the URL schemes follow:
+
+.PP
+.BR "Cloud Files" " (Rackspace)"
+.PP
 .RS
-.PP
-.BI "Rackspace Cloud Files"
-.br
 cf+http://container_name
-.br
+.PP
 See also
 .B "A NOTE ON CLOUD FILES ACCESS"
-.PP
-.BI Dropbox
-.br
+.RE
+.PP
+.B "Copy cloud storage"
+.PP
+.RS
+copy://user[:password]@copy.com/some_dir
+.RE
+.PP
+.B Dropbox
+.PP
+.RS
 dpbx:///some_dir
-.br
+.PP
 Make sure to read
 .BR "A NOTE ON DROPBOX ACCESS" " first!"
-.PP
-copy://user[:password]@copy.com/some_dir
-.PP
-.PP
+.RE
+.PP
+.B "Local file path"
+.PP
+.RS
 file://[relative|/absolute]/local/path
-.PP
+.RE
+.PP
+.BR "FISH" " (Files transferred over Shell protocol) over ssh"
+.PP
+.RS
+fish://user[:password]@other.host[:port]/[relative|/absolute]_path
+.RE
+.PP
+.B "FTP"
+.PP
+.RS
 ftp[s]://user[:password]@other.host[:port]/some_dir
 .PP
+.B NOTE:
+use lftp+, ncftp+ prefixes to enforce a specific backend, e.g. ncftp+ftp://...
+.RE
+.PP
+.B "Google Docs"
+.PP
+.RS
 gdocs://user[:password]@other.host/some_dir
-.PP
-.BI "Google Cloud Storage"
-.br
+.RE
+.PP
+.B "Google Cloud Storage"
+.PP
+.RS
 gs://bucket[/prefix]
-.PP
+.RE
+.PP
+.B "HSI"
+.PP
+.RS
 hsi://user[:password]@other.host/some_dir
-.PP
+.RE
+.PP
+.B "IMAP email storage"
+.PP
+.RS
 imap[s]://user[:password]@host.com[/from_address_prefix]
-.br
+.PP
 See also
 .B "A NOTE ON IMAP"
-.PP
+.RE
+.PP
+.B "Mega cloud storage"
+.PP
+.RS
 mega://user[:password]@mega.co.nz/some_dir
-.PP
-.BI "Par2 Wrapper Backend"
-.br
+.RE
+.PP
+.B "Par2 Wrapper Backend"
+.PP
+.RS
 par2+scheme://[user[:password]@]host[:port]/[/]path
-.br
+.PP
 See also
 .B "A NOTE ON PAR2 WRAPPER BACKEND"
-.PP
-.B "using rsync daemon"
-.br
+.RE
+.PP
+.B "Rsync via daemon"
+.PP
+.RS
 rsync://user[:password]@host.com[:port]::[/]module/some_dir
-.br
-.B "using rsync over ssh (only key auth)"
-.br
+.PP
+.RE
+.B "Rsync over ssh (only key auth)"
+.PP
+.RS
 rsync://[e-mail][:port]/[relative|/absolute]_path
-.PP
+.RE
+.PP
+.BR "S3 storage" " (Amazon)"
+.PP
+.RS
 s3://host/bucket_name[/prefix]
 .br
 s3+http://bucket_name[/prefix]
-.br
+.PP
 See also
 .B "A NOTE ON EUROPEAN S3 BUCKETS"
-.PP
-scp://.. or ssh://.. are synonymous with
-.br
-sftp://user[:password]@other.host[:port]/[/]some_dir
+.RE
+.PP
+.B "SCP/SFTP access"
+.PP
+.RS
+scp://.. or
+.br
+sftp://user[:password]@other.host[:port]/[relative|/absolute]_path
+.PP
+.BR "defaults" " are paramiko+scp:// and paramiko+sftp://"
+.br
+.BR "alternatively" " try pexpect+scp://, pexpect+sftp://, lftp+sftp://"
 .br
 See also
-.BR --ssh-backend ,
 .BR --ssh-askpass ,
-.BR --use-scp ,
 .B  --ssh-options
 and
 .BR "A NOTE ON SSH BACKENDS" .
-.PP
+.RE
+.PP
+.BR "Swift" " (Openstack)"
+.PP
+.RS
 swift://container_name
-.br
+.PP
 See also
 .B "A NOTE ON SWIFT (OPENSTACK OBJECT STORAGE) ACCESS"
-.PP
+.RE
+.PP
+.B "Tahoe-LAFS"
+.PP
+.RS
 tahoe://alias/directory
-.PP
+.RE
+.PP
+.B "WebDAV"
+.PP
+.RS
 webdav[s]://user[:password]@other.host[:port]/some_dir
+.PP
+.B alternatively
+try lftp+webdav[s]://
 .RE
 
 .SH TIME FORMATS

=== modified file 'duplicity/backend.py'
--- duplicity/backend.py	2014-10-27 02:27:36 +0000
+++ duplicity/backend.py	2014-10-30 18:01:04 +0000
@@ -32,6 +32,7 @@
 import re
 import getpass
 import gettext
+import re
 import types
 import urllib
 import urlparse
@@ -164,6 +165,11 @@
 
     _backend_prefixes[scheme] = backend_factory
 
+def strip_prefix(url_string, prefix_scheme):
+    """
+    strip the prefix from a string e.g. par2+ftp://... -> ftp://...
+    """
+    return re.sub('(?i)^'+re.escape(prefix_scheme)+'\+','',url_string)
 
 def is_backend_url(url_string):
     """
@@ -198,7 +204,7 @@
     for prefix in _backend_prefixes:
         if url_string.startswith(prefix + '+'):
             factory = _backend_prefixes[prefix]
-            pu = ParsedUrl(url_string.lstrip(prefix + '+'))
+            pu = ParsedUrl(strip_prefix(url_string,prefix))
             break
 
     if factory is None:
@@ -337,11 +343,8 @@
 def strip_auth_from_url(parsed_url):
     """Return a URL from a urlparse object without a username or password."""
 
-    # Get a copy of the network location without the username or password.
-    straight_netloc = parsed_url.netloc.split('@')[-1]
-
-    # Replace the full network location with the stripped copy.
-    return parsed_url.geturl().replace(parsed_url.netloc, straight_netloc, 1)
+    clean_url = re.sub('^([^:/]+://)(.*@)?(.*)',r'\1\3',parsed_url.geturl())
+    return clean_url
 
 def _get_code_from_exception(backend, operation, e):
     if isinstance(e, BackendException) and e.code != log.ErrorCode.backend_error:

=== renamed file 'duplicity/backends/ftpbackend.py' => 'duplicity/backends/lftpbackend.py'
--- duplicity/backends/ftpbackend.py	2014-10-01 20:35:16 +0000
+++ duplicity/backends/lftpbackend.py	2014-10-30 18:01:04 +0000
@@ -3,7 +3,10 @@
 # Copyright 2002 Ben Escoto <[e-mail]>
 # Copyright 2007 Kenneth Loafman <[e-mail]>
 # Copyright 2010 Marcel Pennewiss <[e-mail]>
-# Copyright 2014 Moritz Maisel <[e-mail]>
+# Copyright 2014 Edgar Soldin
+#                 - webdav, fish, sftp support
+#                 - https cert verification switches
+#                 - debug output
 #
 # This file is part of duplicity.
 #
@@ -23,15 +26,15 @@
 
 import os
 import os.path
+import re
 import urllib
-import re
 
 import duplicity.backend
 from duplicity import globals
 from duplicity import log
 from duplicity import tempdir
 
-class FTPBackend(duplicity.backend.Backend):
+class LFTPBackend(duplicity.backend.Backend):
     """Connect to remote store using File Transfer Protocol"""
     def __init__(self, parsed_url):
         duplicity.backend.Backend.__init__(self, parsed_url)
@@ -54,61 +57,141 @@
 
         self.parsed_url = parsed_url
 
-        self.url_string = duplicity.backend.strip_auth_from_url(self.parsed_url)
+#        self.url_string = duplicity.backend.strip_auth_from_url(self.parsed_url)
+#        # strip lftp+ prefix
+#        self.url_string = duplicity.backend.strip_prefix(self.url_string, 'lftp')
+
+        self.scheme = duplicity.backend.strip_prefix( parsed_url.scheme, 'lftp' ).lower()
+        self.scheme = re.sub('^webdav','http',self.scheme)
+        self.url_string = self.scheme + '://' + parsed_url.hostname
+        if parsed_url.port :
+            self.url_string += ":%s" % parsed_url.port
+
+        self.remote_path = re.sub('^/','',parsed_url.path)
 
         # Use an explicit directory name.
-        if self.url_string[-1] != '/':
-            self.url_string += '/'
+        if self.remote_path[-1] != '/':
+            self.remote_path += '/'
 
-        self.password = self.get_password()
+        self.authflag = ''
+        if self.parsed_url.username:
+            self.username = self.parsed_url.username
+            self.password = self.get_password()
+            self.authflag = "-u '%s,%s'" % (self.username,self.password)
 
         if globals.ftp_connection == 'regular':
             self.conn_opt = 'off'
         else:
             self.conn_opt = 'on'
 
-        if parsed_url.port != None and parsed_url.port != 21:
-            self.portflag = " -p '%s'" % (parsed_url.port)
-        else:
-            self.portflag = ""
+        # check for cacert file if https
+        self.cacert_file = globals.ssl_cacert_file
+        if self.scheme == 'https' and not globals.ssl_no_check_certificate:
+            cacert_candidates = [ "~/.duplicity/cacert.pem", \
+                             "~/duplicity_cacert.pem", \
+                             "/etc/duplicity/cacert.pem" ]
+            #
+            if not self.cacert_file:
+                for path in cacert_candidates :
+                    path = os.path.expanduser(path)
+                    if (os.path.isfile(path)):
+                        self.cacert_file = path
+                        break
+            # still no cacert file, inform user
+            if not self.cacert_file:
+                raise duplicity.errors.FatalBackendException("""For certificate verification a cacert database file is needed in one of these locations: %s
+Hints:
+  Consult the man page, chapter 'SSL Certificate Verification'.
+  Consider using the options --ssl-cacert-file, --ssl-no-check-certificate .""" % ", ".join(cacert_candidates) )
 
         self.tempfile, self.tempname = tempdir.default().mkstemp()
+        os.write(self.tempfile, "set ssl:verify-certificate " + ( "false" if globals.ssl_no_check_certificate else "true" ) + "\n")
+        if globals.ssl_cacert_file :
+            os.write(self.tempfile, "set ssl:ca-file '" + globals.ssl_cacert_file + "'\n")
         os.write(self.tempfile, "set ftp:ssl-allow true\n")
         os.write(self.tempfile, "set ftp:ssl-protect-data true\n")
         os.write(self.tempfile, "set ftp:ssl-protect-list true\n")
+        os.write(self.tempfile, "set http:use-propfind true\n")
         os.write(self.tempfile, "set net:timeout %s\n" % globals.timeout)
         os.write(self.tempfile, "set net:max-retries %s\n" % globals.num_retries)
         os.write(self.tempfile, "set ftp:passive-mode %s\n" % self.conn_opt)
-        os.write(self.tempfile, "open %s %s\n" % (self.portflag, self.parsed_url.hostname))
+        if log.getverbosity() >= log.DEBUG :
+            os.write(self.tempfile, "debug\n")
+        os.write(self.tempfile, "open %s %s\n" % (self.authflag, self.url_string) )
+#        os.write(self.tempfile, "open %s %s\n" % (self.portflag, self.parsed_url.hostname))
         # allow .netrc auth by only setting user/pass when user was actually given
-        if self.parsed_url.username:
-            os.write(self.tempfile, "user %s %s\n" % (self.parsed_url.username, self.password))
+#        if self.parsed_url.username:
+#            os.write(self.tempfile, "user %s %s\n" % (self.parsed_url.username, self.password))
         os.close(self.tempfile)
+        if log.getverbosity() >= log.DEBUG :
+            f = open(self.tempname, 'r')
+            log.Debug("SETTINGS: \n"
+                  "%s" % f.readlines() )
 
     def _put(self, source_path, remote_filename):
-        remote_path = os.path.join(urllib.unquote(self.parsed_url.path.lstrip('/')), remote_filename).rstrip()
-        commandline = "lftp -c 'source %s;put \'%s\' -o \'%s\''" % \
-            (self.tempname, source_path.name, remote_path)
-        self.subprocess_popen(commandline)
+        #remote_path = os.path.join(urllib.unquote(self.parsed_url.path.lstrip('/')), remote_filename).rstrip()
+        commandline = "lftp -c 'source \'%s\'; mkdir -p %s; put \'%s\' -o \'%s\''" % \
+            (self.tempname, self.remote_path, source_path.name, self.remote_path + remote_filename)
+        log.Debug("CMD: %s" % commandline)
+        s, l, e = self.subprocess_popen(commandline)
+        log.Debug("STATUS: %s" % s)
+        log.Debug("STDERR:\n"
+                  "%s" % (e))
+        log.Debug("STDOUT:\n"
+                  "%s" % (l))
 
     def _get(self, remote_filename, local_path):
-        remote_path = os.path.join(urllib.unquote(self.parsed_url.path), remote_filename).rstrip()
-        commandline = "lftp -c 'source %s;get %s -o %s'" % \
-            (self.tempname, remote_path.lstrip('/'), local_path.name)
-        self.subprocess_popen(commandline)
+        #remote_path = os.path.join(urllib.unquote(self.parsed_url.path), remote_filename).rstrip()
+        commandline = "lftp -c 'source \'%s\'; get \'%s\' -o \'%s\''" % \
+            (self.tempname, self.remote_path+remote_filename, local_path.name)
+        log.Debug("CMD: %s" % commandline)
+        _, l, e = self.subprocess_popen(commandline)
+        log.Debug("STDERR:\n"
+                  "%s" % (e))
+        log.Debug("STDOUT:\n"
+                  "%s" % (l))
 
     def _list(self):
         # Do a long listing to avoid connection reset
-        remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
-        commandline = "lftp -c 'source %s;ls \'%s\''" % (self.tempname, remote_dir)
-        _, l, _ = self.subprocess_popen(commandline)
+        #remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
+        remote_dir = urllib.unquote(self.parsed_url.path)
+        #print remote_dir
+        commandline = "lftp -c 'source \'%s\'; cd \'%s\' || exit 0; ls'" % (self.tempname, self.remote_path)
+        log.Debug("CMD: %s" % commandline)
+        _, l, e = self.subprocess_popen(commandline)
+        log.Debug("STDERR:\n"
+                  "%s" % (e))
+        log.Debug("STDOUT:\n"
+                  "%s" % (l))
+
         # Look for our files as the last element of a long list line
         return [x.split()[-1] for x in l.split('\n') if x]
 
     def _delete(self, filename):
-        remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
-        commandline = "lftp -c 'source %s;cd \'%s\';rm \'%s\''" % (self.tempname, remote_dir, filename)
-        self.subprocess_popen(commandline)
-
-duplicity.backend.register_backend("ftp", FTPBackend)
-duplicity.backend.register_backend("ftps", FTPBackend)
+        #remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
+        commandline = "lftp -c 'source \'%s\'; cd \'%s\'; rm \'%s\''" % (self.tempname, self.remote_path, filename)
+        log.Debug("CMD: %s" % commandline)
+        _, l, e = self.subprocess_popen(commandline)
+        log.Debug("STDERR:\n"
+                  "%s" % (e))
+        log.Debug("STDOUT:\n"
+                  "%s" % (l))
+
+duplicity.backend.register_backend("ftp", LFTPBackend)
+duplicity.backend.register_backend("ftps", LFTPBackend)
+duplicity.backend.register_backend("fish", LFTPBackend)
+
+duplicity.backend.register_backend("lftp+ftp", LFTPBackend)
+duplicity.backend.register_backend("lftp+ftps", LFTPBackend)
+duplicity.backend.register_backend("lftp+fish", LFTPBackend)
+duplicity.backend.register_backend("lftp+sftp", LFTPBackend)
+duplicity.backend.register_backend("lftp+webdav", LFTPBackend)
+duplicity.backend.register_backend("lftp+webdavs", LFTPBackend)
+duplicity.backend.register_backend("lftp+http", LFTPBackend)
+duplicity.backend.register_backend("lftp+https", LFTPBackend)
+
+duplicity.backend.uses_netloc.extend([ 'ftp', 'ftps', 'fish',\
+                                       'lftp+ftp', 'lftp+ftps',\
+                                       'lftp+fish', 'lftp+sftp',\
+                                       'lftp+webdav', 'lftp+webdavs',\
+                                       'lftp+http', 'lftp+https' ])
\ No newline at end of file

=== added file 'duplicity/backends/ncftpbackend.py'
--- duplicity/backends/ncftpbackend.py	1970-01-01 00:00:00 +0000
+++ duplicity/backends/ncftpbackend.py	2014-10-30 18:01:04 +0000
@@ -0,0 +1,118 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright 2002 Ben Escoto <[e-mail]>
+# Copyright 2007 Kenneth Loafman <[e-mail]>
+#
+# This file is part of duplicity.
+#
+# Duplicity is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# Duplicity is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with duplicity; if not, write to the Free Software Foundation,
+# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+import os.path
+import urllib
+
+import duplicity.backend
+from duplicity import globals
+from duplicity import log
+from duplicity import tempdir
+
+class NCFTPBackend(duplicity.backend.Backend):
+    """Connect to remote store using File Transfer Protocol"""
+    def __init__(self, parsed_url):
+        duplicity.backend.Backend.__init__(self, parsed_url)
+
+        # we expect an error return, so go low-level and ignore it
+        try:
+            p = os.popen("ncftpls -v")
+            fout = p.read()
+            ret = p.close()
+        except Exception:
+            pass
+        # the expected error is 8 in the high-byte and some output
+        if ret != 0x0800 or not fout:
+            log.FatalError("NcFTP not found:  Please install NcFTP version 3.1.9 or later",
+                           log.ErrorCode.ftp_ncftp_missing)
+
+        # version is the second word of the first line
+        version = fout.split('\n')[0].split()[1]
+        if version < "3.1.9":
+            log.FatalError("NcFTP too old:  Duplicity requires NcFTP version 3.1.9,"
+                           "3.2.1 or later.  Version 3.2.0 will not work properly.",
+                           log.ErrorCode.ftp_ncftp_too_old)
+        elif version == "3.2.0":
+            log.Warn("NcFTP (ncftpput) version 3.2.0 may fail with duplicity.\n"
+                     "see: http://www.ncftpd.com/ncftp/doc/changelog.html\n"
+                     "If you have trouble, please upgrade to 3.2.1 or later",
+                     log.WarningCode.ftp_ncftp_v320)
+        log.Notice("NcFTP version is %s" % version)
+
+        self.parsed_url = parsed_url
+
+        self.url_string = duplicity.backend.strip_auth_from_url(self.parsed_url)
+
+        # strip ncftp+ prefix
+        self.url_string = duplicity.backend.strip_prefix(self.url_string, 'ncftp')
+
+        # This squelches the "file not found" result from ncftpls when
+        # the ftp backend looks for a collection that does not exist.
+        # version 3.2.2 has error code 5, 1280 is some legacy value
+        self.popen_breaks[ 'ncftpls' ] = [ 5, 1280 ]
+
+        # Use an explicit directory name.
+        if self.url_string[-1] != '/':
+            self.url_string += '/'
+
+        self.password = self.get_password()
+
+        if globals.ftp_connection == 'regular':
+            self.conn_opt = '-E'
+        else:
+            self.conn_opt = '-F'
+
+        self.tempfile, self.tempname = tempdir.default().mkstemp()
+        os.write(self.tempfile, "host %s\n" % self.parsed_url.hostname)
+        os.write(self.tempfile, "user %s\n" % self.parsed_url.username)
+        os.write(self.tempfile, "pass %s\n" % self.password)
+        os.close(self.tempfile)
+        self.flags = "-f %s %s -t %s -o useCLNT=0,useHELP_SITE=0 " % \
+            (self.tempname, self.conn_opt, globals.timeout)
+        if parsed_url.port != None and parsed_url.port != 21:
+            self.flags += " -P '%s'" % (parsed_url.port)
+
+    def _put(self, source_path, remote_filename):
+        remote_path = os.path.join(urllib.unquote(self.parsed_url.path.lstrip('/')), remote_filename).rstrip()
+        commandline = "ncftpput %s -m -V -C '%s' '%s'" % \
+            (self.flags, source_path.name, remote_path)
+        self.subprocess_popen(commandline)
+
+    def _get(self, remote_filename, local_path):
+        remote_path = os.path.join(urllib.unquote(self.parsed_url.path), remote_filename).rstrip()
+        commandline = "ncftpget %s -V -C '%s' '%s' '%s'" % \
+            (self.flags, self.parsed_url.hostname, remote_path.lstrip('/'), local_path.name)
+        self.subprocess_popen(commandline)
+
+    def _list(self):
+        # Do a long listing to avoid connection reset
+        commandline = "ncftpls %s -l '%s'" % (self.flags, self.url_string)
+        _, l, _ = self.subprocess_popen(commandline)
+        # Look for our files as the last element of a long list line
+        return [x.split()[-1] for x in l.split('\n') if x and not x.startswith("total ")]
+
+    def _delete(self, filename):
+        commandline = "ncftpls %s -l -X 'DELE %s' '%s'" % \
+            (self.flags, filename, self.url_string)
+        self.subprocess_popen(commandline)
+
+duplicity.backend.register_backend("ncftp+ftp", NCFTPBackend)
+duplicity.backend.uses_netloc.extend([ 'ncftp+ftp' ])
\ No newline at end of file

=== renamed file 'duplicity/backends/_ssh_paramiko.py' => 'duplicity/backends/ssh_paramiko_backend.py'
--- duplicity/backends/_ssh_paramiko.py	2014-10-27 03:02:20 +0000
+++ duplicity/backends/ssh_paramiko_backend.py	2014-10-30 18:01:04 +0000
@@ -217,8 +217,11 @@
                                     self.config['port'],e))
         self.client.get_transport().set_keepalive((int)(globals.timeout / 2))
 
+        self.scheme = duplicity.backend.strip_prefix(parsed_url.scheme, 'paramiko')
+        self.use_scp = ( self.scheme == 'scp' )
+
         # scp or sftp?
-        if (globals.use_scp):
+        if (self.use_scp):
             # sanity-check the directory name
             if (re.search("'",self.remote_dir)):
                 raise BackendException("cannot handle directory names with single quotes with --use-scp!")
@@ -256,7 +259,7 @@
                         raise BackendException("sftp chdir to %s failed: %s" % (self.sftp.normalize(".")+"/"+d,e))
 
     def _put(self, source_path, remote_filename):
-        if globals.use_scp:
+        if self.use_scp:
             f=file(source_path.name,'rb')
             try:
                 chan=self.client.get_transport().open_session()
@@ -284,7 +287,7 @@
             self.sftp.put(source_path.name,remote_filename)
 
     def _get(self, remote_filename, local_path):
-        if globals.use_scp:
+        if self.use_scp:
             try:
                 chan=self.client.get_transport().open_session()
                 chan.settimeout(globals.timeout)
@@ -327,7 +330,7 @@
     def _list(self):
         # In scp mode unavoidable quoting issues will make this fail if the
         # directory name contains single quotes.
-        if globals.use_scp:
+        if self.use_scp:
             output = self.runremote("ls -1 '%s'" % self.remote_dir, False, "scp dir listing ")
             return output.splitlines()
         else:
@@ -336,7 +339,7 @@
     def _delete(self, filename):
         # In scp mode unavoidable quoting issues will cause failures if
         # filenames containing single quotes are encountered.
-        if globals.use_scp:
+        if self.use_scp:
             self.runremote("rm '%s/%s'" % (self.remote_dir, filename), False, "scp rm ")
         else:
             self.sftp.remove(filename)
@@ -370,3 +373,9 @@
             raise BackendException("could not load '%s', maybe corrupt?" % (file))
         
         return sshconfig.lookup(host)
+
+duplicity.backend.register_backend("sftp", SSHParamikoBackend)
+duplicity.backend.register_backend("scp", SSHParamikoBackend)
+duplicity.backend.register_backend("paramiko+sftp", SSHParamikoBackend)
+duplicity.backend.register_backend("paramiko+scp", SSHParamikoBackend)
+duplicity.backend.uses_netloc.extend([ 'sftp', 'scp', 'paramiko+sftp', 'paramiko+scp' ])

=== renamed file 'duplicity/backends/_ssh_pexpect.py' => 'duplicity/backends/ssh_pexpect_backend.py'
--- duplicity/backends/_ssh_pexpect.py	2014-04-28 02:49:39 +0000
+++ duplicity/backends/ssh_pexpect_backend.py	2014-10-30 18:01:04 +0000
@@ -49,6 +49,9 @@
 
         self.sftp_command = "sftp"
         if globals.sftp_command: self.sftp_command = globals.sftp_command
+        
+        self.scheme = duplicity.backend.strip_prefix(parsed_url.scheme, 'pexpect')
+        self.use_scp = ( self.scheme == 'scp' )
 
         # host string of form [user@]hostname
         if parsed_url.username:
@@ -212,7 +215,7 @@
             raise BackendException("Error running '%s': %s" % (commandline, msg))
 
     def _put(self, source_path, remote_filename):
-        if globals.use_scp:
+        if self.use_scp:
             self.put_scp(source_path, remote_filename)
         else:
             self.put_sftp(source_path, remote_filename)
@@ -234,7 +237,7 @@
         self.run_scp_command(commandline)
 
     def _get(self, remote_filename, local_path):
-        if globals.use_scp:
+        if self.use_scp:
             self.get_scp(remote_filename, local_path)
         else:
             self.get_sftp(remote_filename, local_path)
@@ -280,3 +283,8 @@
         commands.append("rm \"%s\"" % filename)
         commandline = ("%s %s %s" % (self.sftp_command, globals.ssh_options, self.host_string))
         self.run_sftp_command(commandline, commands)
+
+duplicity.backend.register_backend("pexpect+sftp", SSHPExpectBackend)
+duplicity.backend.register_backend("pexpect+scp", SSHPExpectBackend)
+duplicity.backend.uses_netloc.extend([ 'pexpect+sftp', 'pexpect+scp' ])
+

=== modified file 'duplicity/commandline.py'
--- duplicity/commandline.py	2014-10-27 02:27:36 +0000
+++ duplicity/commandline.py	2014-10-30 18:01:04 +0000
@@ -537,9 +537,6 @@
     # default to batch mode using public-key encryption
     parser.add_option("--ssh-askpass", action = "store_true")
 
-    # allow the user to switch ssh backend
-    parser.add_option("--ssh-backend", metavar = _("paramiko|pexpect"))
-
     # user added ssh options
     parser.add_option("--ssh-options", action = "extend", metavar = _("options"))
 
@@ -567,8 +564,6 @@
     # Whether to specify --use-agent in GnuPG options
     parser.add_option("--use-agent", action = "store_true")
 
-    parser.add_option("--use-scp", action = "store_true")
-
     parser.add_option("--verbosity", "-v", type = "verbosity", metavar = "[0-9]",
                       dest = "", action = "callback",
                       callback = lambda o, s, v, p: log.setverbosity(v))

=== modified file 'duplicity/file_naming.py'
--- duplicity/file_naming.py	2014-10-27 02:27:36 +0000
+++ duplicity/file_naming.py	2014-10-30 18:01:04 +0000
@@ -393,6 +393,20 @@
         else:
             pr.encrypted = None
 
+    def valid_extension():
+        """
+        plausibility check for duplicity file extension
+        before starting to extensively parse the filenames
+        """
+        res = re.match(r'.*\.(g|z|gpg|gz|tar|p|part|manifest|sigtar)$', filename )
+        #print filename, res
+        if res is None :
+            return False
+        return True
+
+    if not valid_extension() :
+        return None
+
     pr = check_full()
     if not pr:
         pr = check_inc()

=== modified file 'duplicity/globals.py'
--- duplicity/globals.py	2014-05-12 07:09:00 +0000
+++ duplicity/globals.py	2014-10-30 18:01:04 +0000
@@ -231,15 +231,9 @@
 # default to batch mode using public-key encryption
 ssh_askpass = False
 
-# default ssh backend is paramiko
-ssh_backend = "paramiko"
-
 # user added ssh options
 ssh_options = ""
 
-# whether to use scp for put/get, sftp is default
-use_scp = False
-
 # default cf backend is pyrax
 cf_backend = "pyrax"