Merge lp:~ed.so/duplicity/lftp.ncftp.and.prefixes into lp:duplicity/0.7-series

Proposed by edso on 2014-10-30
Status: Merged
Merged at revision: 1014
Proposed branch: lp:~ed.so/duplicity/lftp.ncftp.and.prefixes
Merge into: lp:duplicity/0.7-series
Diff against target: 816 lines (+416/-110)
9 files modified
bin/duplicity.1 (+137/-55)
duplicity/backend.py (+9/-6)
duplicity/backends/lftpbackend.py (+114/-31)
duplicity/backends/ncftpbackend.py (+118/-0)
duplicity/backends/ssh_paramiko_backend.py (+14/-5)
duplicity/backends/ssh_pexpect_backend.py (+10/-2)
duplicity/commandline.py (+0/-5)
duplicity/file_naming.py (+14/-0)
duplicity/globals.py (+0/-6)
To merge this branch: bzr merge lp:~ed.so/duplicity/lftp.ncftp.and.prefixes
Reviewer Review Type Date Requested Status
Kenneth Loafman 2014-10-30 Needs Fixing on 2014-11-10
Review via email: mp+240149@code.launchpad.net

Description of the change

- retire --ssh-backend, --use-scp parameters
- introduce scheme prefixes for alternative backend selection e.g. ncftp+ftp://, see manpage
- scp is now selected via scheme e.g. scp://
- added lftp fish, webdav(s), sftp support

To post a comment you must log in.
Kenneth Loafman (kenneth-loafman) wrote :

I like the general cleanup.

A couple of changes:

Please leave the file naming as it was. I had set these to be underscore-first to show they were called by the main ssh backend. See _boto*, _cf*.

I would like to see sftp: and scp: protocols go away. In reality they are just subsets of the OpenSSH (or other) package. You really can't run just scp: since we need 'ls' and 'del'. If we continue to allow these we need to emphasize that these are just aliases for ssh:.

review: Needs Fixing
edso (ed.so) wrote :

On 10.11.2014 17:00, Kenneth Loafman wrote:
> Review: Needs Fixing
>
> I like the general cleanup.
>
> A couple of changes:
>
> Please leave the file naming as it was. I had set these to be underscore-first to show they were called by the main ssh backend. See _boto*, _cf*.

with my mods there is no ssh backend anymore. backends can provide by registering scheme backend+sftp, backend+scp .. the default backends simply get assigned the prefixless scheme in addition.

as these backends are now equal i removed the underscore.. for the future i planned to remove the backend switching parameters for boto/cf as well and streamline it into the use of prefixes instead.

> I would like to see sftp: and scp: protocols go away.

why?

>In reality they are just subsets of the OpenSSH (or other) package. You really can't run just scp: since we need 'ls' and 'del'. If we continue to allow these we need to emphasize that these are just aliases for ssh:.
>

not true. paramiko has a proper scp support using _no_ (open)ssh binaries and listing via ssh remote shell, as far as i have seen.

i see no problem listing openssh as a requirement (like for the pexpect backend) for lftp+sftp then.

..ede

Kenneth Loafman (kenneth-loafman) wrote :

OK. My confusion. I was looking at the old sshbackend.py for reference.

I deleted the old sshbackend.py since it's no long valid and fixed a couple of format issues.

edso (ed.so) wrote :

On 10.11.2014 21:08, Kenneth Loafman wrote:
> OK. My confusion. I was looking at the old sshbackend.py for reference.
>
> I deleted the old sshbackend.py since it's no long valid and fixed a couple of format issues.
>

nP.. i actually meant to delete sshbackend.py in the branch but i seem to have forgotten to commit it..

what were the format issues you mention? ..ede

Kenneth Loafman (kenneth-loafman) wrote :

No line feed at end of file.

Python does not need backslash on list continuation.

That kind of thing.

On Mon, Nov 10, 2014 at 2:18 PM, edso <email address hidden> wrote:

> On 10.11.2014 21:08, Kenneth Loafman wrote:
> > OK. My confusion. I was looking at the old sshbackend.py for reference.
> >
> > I deleted the old sshbackend.py since it's no long valid and fixed a
> couple of format issues.
> >
>
> nP.. i actually meant to delete sshbackend.py in the branch but i seem to
> have forgotten to commit it..
>
> what were the format issues you mention? ..ede
>
> --
>
> https://code.launchpad.net/~ed.so/duplicity/lftp.ncftp.and.prefixes/+merge/240149
> You are reviewing the proposed merge of lp:~
> ed.so/duplicity/lftp.ncftp.and.prefixes into lp:duplicity.
>

edso (ed.so) wrote :

ok, good to know.. thanks ede

On 10.11.2014 21:57, Kenneth Loafman wrote:
> No line feed at end of file.
>
> Python does not need backslash on list continuation.
>
> That kind of thing.
>
>
> On Mon, Nov 10, 2014 at 2:18 PM, edso <email address hidden> wrote:
>
>> On 10.11.2014 21:08, Kenneth Loafman wrote:
>>> OK. My confusion. I was looking at the old sshbackend.py for reference.
>>>
>>> I deleted the old sshbackend.py since it's no long valid and fixed a
>> couple of format issues.
>>>
>>
>> nP.. i actually meant to delete sshbackend.py in the branch but i seem to
>> have forgotten to commit it..
>>
>> what were the format issues you mention? ..ede
>>
>> --
>>
>> https://code.launchpad.net/~ed.so/duplicity/lftp.ncftp.and.prefixes/+merge/240149
>> You are reviewing the proposed merge of lp:~
>> ed.so/duplicity/lftp.ncftp.and.prefixes into lp:duplicity.
>>
>

Kenneth Loafman (kenneth-loafman) wrote :

By no line feed at end of file, I mean there were a couple of missing line
feeds at the end of the file. Easy to get meanings reversed.

On Tue, Nov 11, 2014 at 3:45 AM, edso <email address hidden> wrote:

> ok, good to know.. thanks ede
>
> On 10.11.2014 21:57, Kenneth Loafman wrote:
> > No line feed at end of file.
> >
> > Python does not need backslash on list continuation.
> >
> > That kind of thing.
> >
> >
> > On Mon, Nov 10, 2014 at 2:18 PM, edso <email address hidden> wrote:
> >
> >> On 10.11.2014 21:08, Kenneth Loafman wrote:
> >>> OK. My confusion. I was looking at the old sshbackend.py for
> reference.
> >>>
> >>> I deleted the old sshbackend.py since it's no long valid and fixed a
> >> couple of format issues.
> >>>
> >>
> >> nP.. i actually meant to delete sshbackend.py in the branch but i seem
> to
> >> have forgotten to commit it..
> >>
> >> what were the format issues you mention? ..ede
> >>
> >> --
> >>
> >>
> https://code.launchpad.net/~ed.so/duplicity/lftp.ncftp.and.prefixes/+merge/240149
> >> You are reviewing the proposed merge of lp:~
> >> ed.so/duplicity/lftp.ncftp.and.prefixes into lp:duplicity.
> >>
> >
>
> --
>
> https://code.launchpad.net/~ed.so/duplicity/lftp.ncftp.and.prefixes/+merge/240149
> You are reviewing the proposed merge of lp:~
> ed.so/duplicity/lftp.ncftp.and.prefixes into lp:duplicity.
>

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'bin/duplicity.1'
--- bin/duplicity.1 2014-10-23 11:56:13 +0000
+++ bin/duplicity.1 2014-10-30 18:01:04 +0000
@@ -68,22 +68,14 @@
68.B Rackspace CloudFiles Pyrax API68.B Rackspace CloudFiles Pyrax API
69- http://docs.rackspace.com/sdks/guide/content/python.html69- http://docs.rackspace.com/sdks/guide/content/python.html
70.TP70.TP
71.B "dpbx backend" (Dropbox)71.BR "dpbx backend" " (Dropbox)"
72.B Dropbox Python SDK72.B Dropbox Python SDK
73- https://www.dropbox.com/developers/reference/sdk73- https://www.dropbox.com/developers/reference/sdk
74.TP74.TP
75.B "copy backend" (Copy.com)75.BR "copy backend" " (Copy.com)"
76.B python-urllib376.B python-urllib3
77- https://github.com/shazow/urllib377- https://github.com/shazow/urllib3
78.TP78.TP
79.B "ftp backend"
80.B NcFTP Client
81- http://www.ncftp.com/
82.TP
83.B "ftps backend"
84.B LFTP Client
85- http://lftp.yar.ru/
86.TP
87.BR "gdocs backend" " (Google Docs)"79.BR "gdocs backend" " (Google Docs)"
88.B Google Data APIs Python Client Library80.B Google Data APIs Python Client Library
89- http://code.google.com/p/gdata-python-client/81- http://code.google.com/p/gdata-python-client/
@@ -95,17 +87,25 @@
95.B D-Bus87.B D-Bus
96(dbus)- http://www.freedesktop.org/wiki/Software/dbus88(dbus)- http://www.freedesktop.org/wiki/Software/dbus
97.TP89.TP
98.B "rsync backend"90.BR "lftp backend" " (needed for ftp, ftps, fish [over ssh] - also supports sftp, webdav[s])"
99.B rsync client binary91.B LFTP Client
100- http://rsync.samba.org/92- http://lftp.yar.ru/
101.TP93.TP
102.BR "mega backend" " (mega.co.nz)"94.BR "mega backend" " (mega.co.nz)"
103.B Python library for mega API95.B Python library for mega API
104- https://github.com/ckornacker/mega.py, ubuntu ppa - ppa:ckornacker/backup96- https://github.com/ckornacker/mega.py, ubuntu ppa - ppa:ckornacker/backup
105.TP97.TP
98.BR "ncftp backend" " (ftp, select via ncftp+ftp://)"
99.B NcFTP
100- http://www.ncftp.com/
101.TP
106.B "Par2 Wrapper Backend"102.B "Par2 Wrapper Backend"
107.B par2cmdline103.B par2cmdline
108- http://parchive.sourceforge.net/104- http://parchive.sourceforge.net/
105.TP
106.B "rsync backend"
107.B rsync client binary
108- http://rsync.samba.org/
109.PP109.PP
110There are two110There are two
111.B ssh backends111.B ssh backends
@@ -1067,87 +1067,169 @@
1067or preceded by a double slash, '//path', to represent an absolute1067or preceded by a double slash, '//path', to represent an absolute
1068filesystem path.1068filesystem path.
1069.PP1069.PP
1070.B Note:
1071.RS
1072Scheme (protocol) access may be provided by more than one backend.
1073In case the default backend is buggy or simply not working in a specific case it might be worth trying an alternative implementation.
1074Alternative backends can be selected by prefixing the scheme with the name of the alternative backend e.g.
1075.B ncftp+ftp://
1076and are mentioned below the scheme's syntax summary.
1077.RE
1078
1079.PP
1070Formats of each of the URL schemes follow:1080Formats of each of the URL schemes follow:
1081
1082.PP
1083.BR "Cloud Files" " (Rackspace)"
1084.PP
1071.RS1085.RS
1072.PP
1073.BI "Rackspace Cloud Files"
1074.br
1075cf+http://container_name1086cf+http://container_name
1076.br1087.PP
1077See also1088See also
1078.B "A NOTE ON CLOUD FILES ACCESS"1089.B "A NOTE ON CLOUD FILES ACCESS"
1079.PP1090.RE
1080.BI Dropbox1091.PP
1081.br1092.B "Copy cloud storage"
1093.PP
1094.RS
1095copy://user[:password]@copy.com/some_dir
1096.RE
1097.PP
1098.B Dropbox
1099.PP
1100.RS
1082dpbx:///some_dir1101dpbx:///some_dir
1083.br1102.PP
1084Make sure to read1103Make sure to read
1085.BR "A NOTE ON DROPBOX ACCESS" " first!"1104.BR "A NOTE ON DROPBOX ACCESS" " first!"
1086.PP1105.RE
1087copy://user[:password]@copy.com/some_dir1106.PP
1088.PP1107.B "Local file path"
1089.PP1108.PP
1109.RS
1090file://[relative|/absolute]/local/path1110file://[relative|/absolute]/local/path
1091.PP1111.RE
1112.PP
1113.BR "FISH" " (Files transferred over Shell protocol) over ssh"
1114.PP
1115.RS
1116fish://user[:password]@other.host[:port]/[relative|/absolute]_path
1117.RE
1118.PP
1119.B "FTP"
1120.PP
1121.RS
1092ftp[s]://user[:password]@other.host[:port]/some_dir1122ftp[s]://user[:password]@other.host[:port]/some_dir
1093.PP1123.PP
1124.B NOTE:
1125use lftp+, ncftp+ prefixes to enforce a specific backend, e.g. ncftp+ftp://...
1126.RE
1127.PP
1128.B "Google Docs"
1129.PP
1130.RS
1094gdocs://user[:password]@other.host/some_dir1131gdocs://user[:password]@other.host/some_dir
1095.PP1132.RE
1096.BI "Google Cloud Storage"1133.PP
1097.br1134.B "Google Cloud Storage"
1135.PP
1136.RS
1098gs://bucket[/prefix]1137gs://bucket[/prefix]
1099.PP1138.RE
1139.PP
1140.B "HSI"
1141.PP
1142.RS
1100hsi://user[:password]@other.host/some_dir1143hsi://user[:password]@other.host/some_dir
1101.PP1144.RE
1145.PP
1146.B "IMAP email storage"
1147.PP
1148.RS
1102imap[s]://user[:password]@host.com[/from_address_prefix]1149imap[s]://user[:password]@host.com[/from_address_prefix]
1103.br1150.PP
1104See also1151See also
1105.B "A NOTE ON IMAP"1152.B "A NOTE ON IMAP"
1106.PP1153.RE
1154.PP
1155.B "Mega cloud storage"
1156.PP
1157.RS
1107mega://user[:password]@mega.co.nz/some_dir1158mega://user[:password]@mega.co.nz/some_dir
1108.PP1159.RE
1109.BI "Par2 Wrapper Backend"1160.PP
1110.br1161.B "Par2 Wrapper Backend"
1162.PP
1163.RS
1111par2+scheme://[user[:password]@]host[:port]/[/]path1164par2+scheme://[user[:password]@]host[:port]/[/]path
1112.br1165.PP
1113See also1166See also
1114.B "A NOTE ON PAR2 WRAPPER BACKEND"1167.B "A NOTE ON PAR2 WRAPPER BACKEND"
1115.PP1168.RE
1116.B "using rsync daemon"1169.PP
1117.br1170.B "Rsync via daemon"
1171.PP
1172.RS
1118rsync://user[:password]@host.com[:port]::[/]module/some_dir1173rsync://user[:password]@host.com[:port]::[/]module/some_dir
1119.br1174.PP
1120.B "using rsync over ssh (only key auth)"1175.RE
1121.br1176.B "Rsync over ssh (only key auth)"
1177.PP
1178.RS
1122rsync://user@host.com[:port]/[relative|/absolute]_path1179rsync://user@host.com[:port]/[relative|/absolute]_path
1123.PP1180.RE
1181.PP
1182.BR "S3 storage" " (Amazon)"
1183.PP
1184.RS
1124s3://host/bucket_name[/prefix]1185s3://host/bucket_name[/prefix]
1125.br1186.br
1126s3+http://bucket_name[/prefix]1187s3+http://bucket_name[/prefix]
1127.br1188.PP
1128See also1189See also
1129.B "A NOTE ON EUROPEAN S3 BUCKETS"1190.B "A NOTE ON EUROPEAN S3 BUCKETS"
1130.PP1191.RE
1131scp://.. or ssh://.. are synonymous with1192.PP
1132.br1193.B "SCP/SFTP access"
1133sftp://user[:password]@other.host[:port]/[/]some_dir1194.PP
1195.RS
1196scp://.. or
1197.br
1198sftp://user[:password]@other.host[:port]/[relative|/absolute]_path
1199.PP
1200.BR "defaults" " are paramiko+scp:// and paramiko+sftp://"
1201.br
1202.BR "alternatively" " try pexpect+scp://, pexpect+sftp://, lftp+sftp://"
1134.br1203.br
1135See also1204See also
1136.BR --ssh-backend ,
1137.BR --ssh-askpass ,1205.BR --ssh-askpass ,
1138.BR --use-scp ,
1139.B --ssh-options1206.B --ssh-options
1140and1207and
1141.BR "A NOTE ON SSH BACKENDS" .1208.BR "A NOTE ON SSH BACKENDS" .
1142.PP1209.RE
1210.PP
1211.BR "Swift" " (Openstack)"
1212.PP
1213.RS
1143swift://container_name1214swift://container_name
1144.br1215.PP
1145See also1216See also
1146.B "A NOTE ON SWIFT (OPENSTACK OBJECT STORAGE) ACCESS"1217.B "A NOTE ON SWIFT (OPENSTACK OBJECT STORAGE) ACCESS"
1147.PP1218.RE
1219.PP
1220.B "Tahoe-LAFS"
1221.PP
1222.RS
1148tahoe://alias/directory1223tahoe://alias/directory
1149.PP1224.RE
1225.PP
1226.B "WebDAV"
1227.PP
1228.RS
1150webdav[s]://user[:password]@other.host[:port]/some_dir1229webdav[s]://user[:password]@other.host[:port]/some_dir
1230.PP
1231.B alternatively
1232try lftp+webdav[s]://
1151.RE1233.RE
11521234
1153.SH TIME FORMATS1235.SH TIME FORMATS
11541236
=== modified file 'duplicity/backend.py'
--- duplicity/backend.py 2014-10-27 02:27:36 +0000
+++ duplicity/backend.py 2014-10-30 18:01:04 +0000
@@ -32,6 +32,7 @@
32import re32import re
33import getpass33import getpass
34import gettext34import gettext
35import re
35import types36import types
36import urllib37import urllib
37import urlparse38import urlparse
@@ -164,6 +165,11 @@
164165
165 _backend_prefixes[scheme] = backend_factory166 _backend_prefixes[scheme] = backend_factory
166167
168def strip_prefix(url_string, prefix_scheme):
169 """
170 strip the prefix from a string e.g. par2+ftp://... -> ftp://...
171 """
172 return re.sub('(?i)^'+re.escape(prefix_scheme)+'\+','',url_string)
167173
168def is_backend_url(url_string):174def is_backend_url(url_string):
169 """175 """
@@ -198,7 +204,7 @@
198 for prefix in _backend_prefixes:204 for prefix in _backend_prefixes:
199 if url_string.startswith(prefix + '+'):205 if url_string.startswith(prefix + '+'):
200 factory = _backend_prefixes[prefix]206 factory = _backend_prefixes[prefix]
201 pu = ParsedUrl(url_string.lstrip(prefix + '+'))207 pu = ParsedUrl(strip_prefix(url_string,prefix))
202 break208 break
203209
204 if factory is None:210 if factory is None:
@@ -337,11 +343,8 @@
337def strip_auth_from_url(parsed_url):343def strip_auth_from_url(parsed_url):
338 """Return a URL from a urlparse object without a username or password."""344 """Return a URL from a urlparse object without a username or password."""
339345
340 # Get a copy of the network location without the username or password.346 clean_url = re.sub('^([^:/]+://)(.*@)?(.*)',r'\1\3',parsed_url.geturl())
341 straight_netloc = parsed_url.netloc.split('@')[-1]347 return clean_url
342
343 # Replace the full network location with the stripped copy.
344 return parsed_url.geturl().replace(parsed_url.netloc, straight_netloc, 1)
345348
346def _get_code_from_exception(backend, operation, e):349def _get_code_from_exception(backend, operation, e):
347 if isinstance(e, BackendException) and e.code != log.ErrorCode.backend_error:350 if isinstance(e, BackendException) and e.code != log.ErrorCode.backend_error:
348351
=== renamed file 'duplicity/backends/ftpbackend.py' => 'duplicity/backends/lftpbackend.py'
--- duplicity/backends/ftpbackend.py 2014-10-01 20:35:16 +0000
+++ duplicity/backends/lftpbackend.py 2014-10-30 18:01:04 +0000
@@ -3,7 +3,10 @@
3# Copyright 2002 Ben Escoto <ben@emerose.org>3# Copyright 2002 Ben Escoto <ben@emerose.org>
4# Copyright 2007 Kenneth Loafman <kenneth@loafman.com>4# Copyright 2007 Kenneth Loafman <kenneth@loafman.com>
5# Copyright 2010 Marcel Pennewiss <opensource@pennewiss.de>5# Copyright 2010 Marcel Pennewiss <opensource@pennewiss.de>
6# Copyright 2014 Moritz Maisel <moritz@maisel.name>6# Copyright 2014 Edgar Soldin
7# - webdav, fish, sftp support
8# - https cert verification switches
9# - debug output
7#10#
8# This file is part of duplicity.11# This file is part of duplicity.
9#12#
@@ -23,15 +26,15 @@
2326
24import os27import os
25import os.path28import os.path
29import re
26import urllib30import urllib
27import re
2831
29import duplicity.backend32import duplicity.backend
30from duplicity import globals33from duplicity import globals
31from duplicity import log34from duplicity import log
32from duplicity import tempdir35from duplicity import tempdir
3336
34class FTPBackend(duplicity.backend.Backend):37class LFTPBackend(duplicity.backend.Backend):
35 """Connect to remote store using File Transfer Protocol"""38 """Connect to remote store using File Transfer Protocol"""
36 def __init__(self, parsed_url):39 def __init__(self, parsed_url):
37 duplicity.backend.Backend.__init__(self, parsed_url)40 duplicity.backend.Backend.__init__(self, parsed_url)
@@ -54,61 +57,141 @@
5457
55 self.parsed_url = parsed_url58 self.parsed_url = parsed_url
5659
57 self.url_string = duplicity.backend.strip_auth_from_url(self.parsed_url)60# self.url_string = duplicity.backend.strip_auth_from_url(self.parsed_url)
61# # strip lftp+ prefix
62# self.url_string = duplicity.backend.strip_prefix(self.url_string, 'lftp')
63
64 self.scheme = duplicity.backend.strip_prefix( parsed_url.scheme, 'lftp' ).lower()
65 self.scheme = re.sub('^webdav','http',self.scheme)
66 self.url_string = self.scheme + '://' + parsed_url.hostname
67 if parsed_url.port :
68 self.url_string += ":%s" % parsed_url.port
69
70 self.remote_path = re.sub('^/','',parsed_url.path)
5871
59 # Use an explicit directory name.72 # Use an explicit directory name.
60 if self.url_string[-1] != '/':73 if self.remote_path[-1] != '/':
61 self.url_string += '/'74 self.remote_path += '/'
6275
63 self.password = self.get_password()76 self.authflag = ''
77 if self.parsed_url.username:
78 self.username = self.parsed_url.username
79 self.password = self.get_password()
80 self.authflag = "-u '%s,%s'" % (self.username,self.password)
6481
65 if globals.ftp_connection == 'regular':82 if globals.ftp_connection == 'regular':
66 self.conn_opt = 'off'83 self.conn_opt = 'off'
67 else:84 else:
68 self.conn_opt = 'on'85 self.conn_opt = 'on'
6986
70 if parsed_url.port != None and parsed_url.port != 21:87 # check for cacert file if https
71 self.portflag = " -p '%s'" % (parsed_url.port)88 self.cacert_file = globals.ssl_cacert_file
72 else:89 if self.scheme == 'https' and not globals.ssl_no_check_certificate:
73 self.portflag = ""90 cacert_candidates = [ "~/.duplicity/cacert.pem", \
91 "~/duplicity_cacert.pem", \
92 "/etc/duplicity/cacert.pem" ]
93 #
94 if not self.cacert_file:
95 for path in cacert_candidates :
96 path = os.path.expanduser(path)
97 if (os.path.isfile(path)):
98 self.cacert_file = path
99 break
100 # still no cacert file, inform user
101 if not self.cacert_file:
102 raise duplicity.errors.FatalBackendException("""For certificate verification a cacert database file is needed in one of these locations: %s
103Hints:
104 Consult the man page, chapter 'SSL Certificate Verification'.
105 Consider using the options --ssl-cacert-file, --ssl-no-check-certificate .""" % ", ".join(cacert_candidates) )
74106
75 self.tempfile, self.tempname = tempdir.default().mkstemp()107 self.tempfile, self.tempname = tempdir.default().mkstemp()
108 os.write(self.tempfile, "set ssl:verify-certificate " + ( "false" if globals.ssl_no_check_certificate else "true" ) + "\n")
109 if globals.ssl_cacert_file :
110 os.write(self.tempfile, "set ssl:ca-file '" + globals.ssl_cacert_file + "'\n")
76 os.write(self.tempfile, "set ftp:ssl-allow true\n")111 os.write(self.tempfile, "set ftp:ssl-allow true\n")
77 os.write(self.tempfile, "set ftp:ssl-protect-data true\n")112 os.write(self.tempfile, "set ftp:ssl-protect-data true\n")
78 os.write(self.tempfile, "set ftp:ssl-protect-list true\n")113 os.write(self.tempfile, "set ftp:ssl-protect-list true\n")
114 os.write(self.tempfile, "set http:use-propfind true\n")
79 os.write(self.tempfile, "set net:timeout %s\n" % globals.timeout)115 os.write(self.tempfile, "set net:timeout %s\n" % globals.timeout)
80 os.write(self.tempfile, "set net:max-retries %s\n" % globals.num_retries)116 os.write(self.tempfile, "set net:max-retries %s\n" % globals.num_retries)
81 os.write(self.tempfile, "set ftp:passive-mode %s\n" % self.conn_opt)117 os.write(self.tempfile, "set ftp:passive-mode %s\n" % self.conn_opt)
82 os.write(self.tempfile, "open %s %s\n" % (self.portflag, self.parsed_url.hostname))118 if log.getverbosity() >= log.DEBUG :
119 os.write(self.tempfile, "debug\n")
120 os.write(self.tempfile, "open %s %s\n" % (self.authflag, self.url_string) )
121# os.write(self.tempfile, "open %s %s\n" % (self.portflag, self.parsed_url.hostname))
83 # allow .netrc auth by only setting user/pass when user was actually given122 # allow .netrc auth by only setting user/pass when user was actually given
84 if self.parsed_url.username:123# if self.parsed_url.username:
85 os.write(self.tempfile, "user %s %s\n" % (self.parsed_url.username, self.password))124# os.write(self.tempfile, "user %s %s\n" % (self.parsed_url.username, self.password))
86 os.close(self.tempfile)125 os.close(self.tempfile)
126 if log.getverbosity() >= log.DEBUG :
127 f = open(self.tempname, 'r')
128 log.Debug("SETTINGS: \n"
129 "%s" % f.readlines() )
87130
88 def _put(self, source_path, remote_filename):131 def _put(self, source_path, remote_filename):
89 remote_path = os.path.join(urllib.unquote(self.parsed_url.path.lstrip('/')), remote_filename).rstrip()132 #remote_path = os.path.join(urllib.unquote(self.parsed_url.path.lstrip('/')), remote_filename).rstrip()
90 commandline = "lftp -c 'source %s;put \'%s\' -o \'%s\''" % \133 commandline = "lftp -c 'source \'%s\'; mkdir -p %s; put \'%s\' -o \'%s\''" % \
91 (self.tempname, source_path.name, remote_path)134 (self.tempname, self.remote_path, source_path.name, self.remote_path + remote_filename)
92 self.subprocess_popen(commandline)135 log.Debug("CMD: %s" % commandline)
136 s, l, e = self.subprocess_popen(commandline)
137 log.Debug("STATUS: %s" % s)
138 log.Debug("STDERR:\n"
139 "%s" % (e))
140 log.Debug("STDOUT:\n"
141 "%s" % (l))
93142
94 def _get(self, remote_filename, local_path):143 def _get(self, remote_filename, local_path):
95 remote_path = os.path.join(urllib.unquote(self.parsed_url.path), remote_filename).rstrip()144 #remote_path = os.path.join(urllib.unquote(self.parsed_url.path), remote_filename).rstrip()
96 commandline = "lftp -c 'source %s;get %s -o %s'" % \145 commandline = "lftp -c 'source \'%s\'; get \'%s\' -o \'%s\''" % \
97 (self.tempname, remote_path.lstrip('/'), local_path.name)146 (self.tempname, self.remote_path+remote_filename, local_path.name)
98 self.subprocess_popen(commandline)147 log.Debug("CMD: %s" % commandline)
148 _, l, e = self.subprocess_popen(commandline)
149 log.Debug("STDERR:\n"
150 "%s" % (e))
151 log.Debug("STDOUT:\n"
152 "%s" % (l))
99153
100 def _list(self):154 def _list(self):
101 # Do a long listing to avoid connection reset155 # Do a long listing to avoid connection reset
102 remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()156 #remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
103 commandline = "lftp -c 'source %s;ls \'%s\''" % (self.tempname, remote_dir)157 remote_dir = urllib.unquote(self.parsed_url.path)
104 _, l, _ = self.subprocess_popen(commandline)158 #print remote_dir
159 commandline = "lftp -c 'source \'%s\'; cd \'%s\' || exit 0; ls'" % (self.tempname, self.remote_path)
160 log.Debug("CMD: %s" % commandline)
161 _, l, e = self.subprocess_popen(commandline)
162 log.Debug("STDERR:\n"
163 "%s" % (e))
164 log.Debug("STDOUT:\n"
165 "%s" % (l))
166
105 # Look for our files as the last element of a long list line167 # Look for our files as the last element of a long list line
106 return [x.split()[-1] for x in l.split('\n') if x]168 return [x.split()[-1] for x in l.split('\n') if x]
107169
108 def _delete(self, filename):170 def _delete(self, filename):
109 remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()171 #remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
110 commandline = "lftp -c 'source %s;cd \'%s\';rm \'%s\''" % (self.tempname, remote_dir, filename)172 commandline = "lftp -c 'source \'%s\'; cd \'%s\'; rm \'%s\''" % (self.tempname, self.remote_path, filename)
111 self.subprocess_popen(commandline)173 log.Debug("CMD: %s" % commandline)
112174 _, l, e = self.subprocess_popen(commandline)
113duplicity.backend.register_backend("ftp", FTPBackend)175 log.Debug("STDERR:\n"
114duplicity.backend.register_backend("ftps", FTPBackend)176 "%s" % (e))
177 log.Debug("STDOUT:\n"
178 "%s" % (l))
179
180duplicity.backend.register_backend("ftp", LFTPBackend)
181duplicity.backend.register_backend("ftps", LFTPBackend)
182duplicity.backend.register_backend("fish", LFTPBackend)
183
184duplicity.backend.register_backend("lftp+ftp", LFTPBackend)
185duplicity.backend.register_backend("lftp+ftps", LFTPBackend)
186duplicity.backend.register_backend("lftp+fish", LFTPBackend)
187duplicity.backend.register_backend("lftp+sftp", LFTPBackend)
188duplicity.backend.register_backend("lftp+webdav", LFTPBackend)
189duplicity.backend.register_backend("lftp+webdavs", LFTPBackend)
190duplicity.backend.register_backend("lftp+http", LFTPBackend)
191duplicity.backend.register_backend("lftp+https", LFTPBackend)
192
193duplicity.backend.uses_netloc.extend([ 'ftp', 'ftps', 'fish',\
194 'lftp+ftp', 'lftp+ftps',\
195 'lftp+fish', 'lftp+sftp',\
196 'lftp+webdav', 'lftp+webdavs',\
197 'lftp+http', 'lftp+https' ])
115\ No newline at end of file198\ No newline at end of file
116199
=== added file 'duplicity/backends/ncftpbackend.py'
--- duplicity/backends/ncftpbackend.py 1970-01-01 00:00:00 +0000
+++ duplicity/backends/ncftpbackend.py 2014-10-30 18:01:04 +0000
@@ -0,0 +1,118 @@
1# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
2#
3# Copyright 2002 Ben Escoto <ben@emerose.org>
4# Copyright 2007 Kenneth Loafman <kenneth@loafman.com>
5#
6# This file is part of duplicity.
7#
8# Duplicity is free software; you can redistribute it and/or modify it
9# under the terms of the GNU General Public License as published by the
10# Free Software Foundation; either version 2 of the License, or (at your
11# option) any later version.
12#
13# Duplicity is distributed in the hope that it will be useful, but
14# WITHOUT ANY WARRANTY; without even the implied warranty of
15# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
16# General Public License for more details.
17#
18# You should have received a copy of the GNU General Public License
19# along with duplicity; if not, write to the Free Software Foundation,
20# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
21
22import os.path
23import urllib
24
25import duplicity.backend
26from duplicity import globals
27from duplicity import log
28from duplicity import tempdir
29
30class NCFTPBackend(duplicity.backend.Backend):
31 """Connect to remote store using File Transfer Protocol"""
32 def __init__(self, parsed_url):
33 duplicity.backend.Backend.__init__(self, parsed_url)
34
35 # we expect an error return, so go low-level and ignore it
36 try:
37 p = os.popen("ncftpls -v")
38 fout = p.read()
39 ret = p.close()
40 except Exception:
41 pass
42 # the expected error is 8 in the high-byte and some output
43 if ret != 0x0800 or not fout:
44 log.FatalError("NcFTP not found: Please install NcFTP version 3.1.9 or later",
45 log.ErrorCode.ftp_ncftp_missing)
46
47 # version is the second word of the first line
48 version = fout.split('\n')[0].split()[1]
49 if version < "3.1.9":
50 log.FatalError("NcFTP too old: Duplicity requires NcFTP version 3.1.9,"
51 "3.2.1 or later. Version 3.2.0 will not work properly.",
52 log.ErrorCode.ftp_ncftp_too_old)
53 elif version == "3.2.0":
54 log.Warn("NcFTP (ncftpput) version 3.2.0 may fail with duplicity.\n"
55 "see: http://www.ncftpd.com/ncftp/doc/changelog.html\n"
56 "If you have trouble, please upgrade to 3.2.1 or later",
57 log.WarningCode.ftp_ncftp_v320)
58 log.Notice("NcFTP version is %s" % version)
59
60 self.parsed_url = parsed_url
61
62 self.url_string = duplicity.backend.strip_auth_from_url(self.parsed_url)
63
64 # strip ncftp+ prefix
65 self.url_string = duplicity.backend.strip_prefix(self.url_string, 'ncftp')
66
67 # This squelches the "file not found" result from ncftpls when
68 # the ftp backend looks for a collection that does not exist.
69 # version 3.2.2 has error code 5, 1280 is some legacy value
70 self.popen_breaks[ 'ncftpls' ] = [ 5, 1280 ]
71
72 # Use an explicit directory name.
73 if self.url_string[-1] != '/':
74 self.url_string += '/'
75
76 self.password = self.get_password()
77
78 if globals.ftp_connection == 'regular':
79 self.conn_opt = '-E'
80 else:
81 self.conn_opt = '-F'
82
83 self.tempfile, self.tempname = tempdir.default().mkstemp()
84 os.write(self.tempfile, "host %s\n" % self.parsed_url.hostname)
85 os.write(self.tempfile, "user %s\n" % self.parsed_url.username)
86 os.write(self.tempfile, "pass %s\n" % self.password)
87 os.close(self.tempfile)
88 self.flags = "-f %s %s -t %s -o useCLNT=0,useHELP_SITE=0 " % \
89 (self.tempname, self.conn_opt, globals.timeout)
90 if parsed_url.port != None and parsed_url.port != 21:
91 self.flags += " -P '%s'" % (parsed_url.port)
92
93 def _put(self, source_path, remote_filename):
94 remote_path = os.path.join(urllib.unquote(self.parsed_url.path.lstrip('/')), remote_filename).rstrip()
95 commandline = "ncftpput %s -m -V -C '%s' '%s'" % \
96 (self.flags, source_path.name, remote_path)
97 self.subprocess_popen(commandline)
98
99 def _get(self, remote_filename, local_path):
100 remote_path = os.path.join(urllib.unquote(self.parsed_url.path), remote_filename).rstrip()
101 commandline = "ncftpget %s -V -C '%s' '%s' '%s'" % \
102 (self.flags, self.parsed_url.hostname, remote_path.lstrip('/'), local_path.name)
103 self.subprocess_popen(commandline)
104
105 def _list(self):
106 # Do a long listing to avoid connection reset
107 commandline = "ncftpls %s -l '%s'" % (self.flags, self.url_string)
108 _, l, _ = self.subprocess_popen(commandline)
109 # Look for our files as the last element of a long list line
110 return [x.split()[-1] for x in l.split('\n') if x and not x.startswith("total ")]
111
112 def _delete(self, filename):
113 commandline = "ncftpls %s -l -X 'DELE %s' '%s'" % \
114 (self.flags, filename, self.url_string)
115 self.subprocess_popen(commandline)
116
117duplicity.backend.register_backend("ncftp+ftp", NCFTPBackend)
118duplicity.backend.uses_netloc.extend([ 'ncftp+ftp' ])
0\ No newline at end of file119\ No newline at end of file
1120
=== renamed file 'duplicity/backends/_ssh_paramiko.py' => 'duplicity/backends/ssh_paramiko_backend.py'
--- duplicity/backends/_ssh_paramiko.py 2014-10-27 03:02:20 +0000
+++ duplicity/backends/ssh_paramiko_backend.py 2014-10-30 18:01:04 +0000
@@ -217,8 +217,11 @@
217 self.config['port'],e))217 self.config['port'],e))
218 self.client.get_transport().set_keepalive((int)(globals.timeout / 2))218 self.client.get_transport().set_keepalive((int)(globals.timeout / 2))
219219
220 self.scheme = duplicity.backend.strip_prefix(parsed_url.scheme, 'paramiko')
221 self.use_scp = ( self.scheme == 'scp' )
222
220 # scp or sftp?223 # scp or sftp?
221 if (globals.use_scp):224 if (self.use_scp):
222 # sanity-check the directory name225 # sanity-check the directory name
223 if (re.search("'",self.remote_dir)):226 if (re.search("'",self.remote_dir)):
224 raise BackendException("cannot handle directory names with single quotes with --use-scp!")227 raise BackendException("cannot handle directory names with single quotes with --use-scp!")
@@ -256,7 +259,7 @@
256 raise BackendException("sftp chdir to %s failed: %s" % (self.sftp.normalize(".")+"/"+d,e))259 raise BackendException("sftp chdir to %s failed: %s" % (self.sftp.normalize(".")+"/"+d,e))
257260
258 def _put(self, source_path, remote_filename):261 def _put(self, source_path, remote_filename):
259 if globals.use_scp:262 if self.use_scp:
260 f=file(source_path.name,'rb')263 f=file(source_path.name,'rb')
261 try:264 try:
262 chan=self.client.get_transport().open_session()265 chan=self.client.get_transport().open_session()
@@ -284,7 +287,7 @@
284 self.sftp.put(source_path.name,remote_filename)287 self.sftp.put(source_path.name,remote_filename)
285288
286 def _get(self, remote_filename, local_path):289 def _get(self, remote_filename, local_path):
287 if globals.use_scp:290 if self.use_scp:
288 try:291 try:
289 chan=self.client.get_transport().open_session()292 chan=self.client.get_transport().open_session()
290 chan.settimeout(globals.timeout)293 chan.settimeout(globals.timeout)
@@ -327,7 +330,7 @@
327 def _list(self):330 def _list(self):
328 # In scp mode unavoidable quoting issues will make this fail if the331 # In scp mode unavoidable quoting issues will make this fail if the
329 # directory name contains single quotes.332 # directory name contains single quotes.
330 if globals.use_scp:333 if self.use_scp:
331 output = self.runremote("ls -1 '%s'" % self.remote_dir, False, "scp dir listing ")334 output = self.runremote("ls -1 '%s'" % self.remote_dir, False, "scp dir listing ")
332 return output.splitlines()335 return output.splitlines()
333 else:336 else:
@@ -336,7 +339,7 @@
336 def _delete(self, filename):339 def _delete(self, filename):
337 # In scp mode unavoidable quoting issues will cause failures if340 # In scp mode unavoidable quoting issues will cause failures if
338 # filenames containing single quotes are encountered.341 # filenames containing single quotes are encountered.
339 if globals.use_scp:342 if self.use_scp:
340 self.runremote("rm '%s/%s'" % (self.remote_dir, filename), False, "scp rm ")343 self.runremote("rm '%s/%s'" % (self.remote_dir, filename), False, "scp rm ")
341 else:344 else:
342 self.sftp.remove(filename)345 self.sftp.remove(filename)
@@ -370,3 +373,9 @@
370 raise BackendException("could not load '%s', maybe corrupt?" % (file))373 raise BackendException("could not load '%s', maybe corrupt?" % (file))
371 374
372 return sshconfig.lookup(host)375 return sshconfig.lookup(host)
376
377duplicity.backend.register_backend("sftp", SSHParamikoBackend)
378duplicity.backend.register_backend("scp", SSHParamikoBackend)
379duplicity.backend.register_backend("paramiko+sftp", SSHParamikoBackend)
380duplicity.backend.register_backend("paramiko+scp", SSHParamikoBackend)
381duplicity.backend.uses_netloc.extend([ 'sftp', 'scp', 'paramiko+sftp', 'paramiko+scp' ])
373382
=== renamed file 'duplicity/backends/_ssh_pexpect.py' => 'duplicity/backends/ssh_pexpect_backend.py'
--- duplicity/backends/_ssh_pexpect.py 2014-04-28 02:49:39 +0000
+++ duplicity/backends/ssh_pexpect_backend.py 2014-10-30 18:01:04 +0000
@@ -49,6 +49,9 @@
4949
50 self.sftp_command = "sftp"50 self.sftp_command = "sftp"
51 if globals.sftp_command: self.sftp_command = globals.sftp_command51 if globals.sftp_command: self.sftp_command = globals.sftp_command
52
53 self.scheme = duplicity.backend.strip_prefix(parsed_url.scheme, 'pexpect')
54 self.use_scp = ( self.scheme == 'scp' )
5255
53 # host string of form [user@]hostname56 # host string of form [user@]hostname
54 if parsed_url.username:57 if parsed_url.username:
@@ -212,7 +215,7 @@
212 raise BackendException("Error running '%s': %s" % (commandline, msg))215 raise BackendException("Error running '%s': %s" % (commandline, msg))
213216
214 def _put(self, source_path, remote_filename):217 def _put(self, source_path, remote_filename):
215 if globals.use_scp:218 if self.use_scp:
216 self.put_scp(source_path, remote_filename)219 self.put_scp(source_path, remote_filename)
217 else:220 else:
218 self.put_sftp(source_path, remote_filename)221 self.put_sftp(source_path, remote_filename)
@@ -234,7 +237,7 @@
234 self.run_scp_command(commandline)237 self.run_scp_command(commandline)
235238
236 def _get(self, remote_filename, local_path):239 def _get(self, remote_filename, local_path):
237 if globals.use_scp:240 if self.use_scp:
238 self.get_scp(remote_filename, local_path)241 self.get_scp(remote_filename, local_path)
239 else:242 else:
240 self.get_sftp(remote_filename, local_path)243 self.get_sftp(remote_filename, local_path)
@@ -280,3 +283,8 @@
280 commands.append("rm \"%s\"" % filename)283 commands.append("rm \"%s\"" % filename)
281 commandline = ("%s %s %s" % (self.sftp_command, globals.ssh_options, self.host_string))284 commandline = ("%s %s %s" % (self.sftp_command, globals.ssh_options, self.host_string))
282 self.run_sftp_command(commandline, commands)285 self.run_sftp_command(commandline, commands)
286
287duplicity.backend.register_backend("pexpect+sftp", SSHPExpectBackend)
288duplicity.backend.register_backend("pexpect+scp", SSHPExpectBackend)
289duplicity.backend.uses_netloc.extend([ 'pexpect+sftp', 'pexpect+scp' ])
290
283291
=== modified file 'duplicity/commandline.py'
--- duplicity/commandline.py 2014-10-27 02:27:36 +0000
+++ duplicity/commandline.py 2014-10-30 18:01:04 +0000
@@ -537,9 +537,6 @@
537 # default to batch mode using public-key encryption537 # default to batch mode using public-key encryption
538 parser.add_option("--ssh-askpass", action = "store_true")538 parser.add_option("--ssh-askpass", action = "store_true")
539539
540 # allow the user to switch ssh backend
541 parser.add_option("--ssh-backend", metavar = _("paramiko|pexpect"))
542
543 # user added ssh options540 # user added ssh options
544 parser.add_option("--ssh-options", action = "extend", metavar = _("options"))541 parser.add_option("--ssh-options", action = "extend", metavar = _("options"))
545542
@@ -567,8 +564,6 @@
567 # Whether to specify --use-agent in GnuPG options564 # Whether to specify --use-agent in GnuPG options
568 parser.add_option("--use-agent", action = "store_true")565 parser.add_option("--use-agent", action = "store_true")
569566
570 parser.add_option("--use-scp", action = "store_true")
571
572 parser.add_option("--verbosity", "-v", type = "verbosity", metavar = "[0-9]",567 parser.add_option("--verbosity", "-v", type = "verbosity", metavar = "[0-9]",
573 dest = "", action = "callback",568 dest = "", action = "callback",
574 callback = lambda o, s, v, p: log.setverbosity(v))569 callback = lambda o, s, v, p: log.setverbosity(v))
575570
=== modified file 'duplicity/file_naming.py'
--- duplicity/file_naming.py 2014-10-27 02:27:36 +0000
+++ duplicity/file_naming.py 2014-10-30 18:01:04 +0000
@@ -393,6 +393,20 @@
393 else:393 else:
394 pr.encrypted = None394 pr.encrypted = None
395395
396 def valid_extension():
397 """
398 plausibility check for duplicity file extension
399 before starting to extensively parse the filenames
400 """
401 res = re.match(r'.*\.(g|z|gpg|gz|tar|p|part|manifest|sigtar)$', filename )
402 #print filename, res
403 if res is None :
404 return False
405 return True
406
407 if not valid_extension() :
408 return None
409
396 pr = check_full()410 pr = check_full()
397 if not pr:411 if not pr:
398 pr = check_inc()412 pr = check_inc()
399413
=== modified file 'duplicity/globals.py'
--- duplicity/globals.py 2014-05-12 07:09:00 +0000
+++ duplicity/globals.py 2014-10-30 18:01:04 +0000
@@ -231,15 +231,9 @@
231# default to batch mode using public-key encryption231# default to batch mode using public-key encryption
232ssh_askpass = False232ssh_askpass = False
233233
234# default ssh backend is paramiko
235ssh_backend = "paramiko"
236
237# user added ssh options234# user added ssh options
238ssh_options = ""235ssh_options = ""
239236
240# whether to use scp for put/get, sftp is default
241use_scp = False
242
243# default cf backend is pyrax237# default cf backend is pyrax
244cf_backend = "pyrax"238cf_backend = "pyrax"
245239

Subscribers

People subscribed via source and target branches