Merge lp:~ed.so/duplicity/lftp.ncftp.and.prefixes into lp:duplicity/0.7-series

Proposed by edso on 2014-10-30
Status: Merged
Merged at revision: 1014
Proposed branch: lp:~ed.so/duplicity/lftp.ncftp.and.prefixes
Merge into: lp:duplicity/0.7-series
Diff against target: 816 lines (+416/-110)
9 files modified
bin/duplicity.1 (+137/-55)
duplicity/backend.py (+9/-6)
duplicity/backends/lftpbackend.py (+114/-31)
duplicity/backends/ncftpbackend.py (+118/-0)
duplicity/backends/ssh_paramiko_backend.py (+14/-5)
duplicity/backends/ssh_pexpect_backend.py (+10/-2)
duplicity/commandline.py (+0/-5)
duplicity/file_naming.py (+14/-0)
duplicity/globals.py (+0/-6)
To merge this branch: bzr merge lp:~ed.so/duplicity/lftp.ncftp.and.prefixes
Reviewer Review Type Date Requested Status
Kenneth Loafman 2014-10-30 Needs Fixing on 2014-11-10
Review via email: mp+240149@code.launchpad.net

Description of the change

- retire --ssh-backend, --use-scp parameters
- introduce scheme prefixes for alternative backend selection e.g. ncftp+ftp://, see manpage
- scp is now selected via scheme e.g. scp://
- added lftp fish, webdav(s), sftp support

To post a comment you must log in.
Kenneth Loafman (kenneth-loafman) wrote :

I like the general cleanup.

A couple of changes:

Please leave the file naming as it was. I had set these to be underscore-first to show they were called by the main ssh backend. See _boto*, _cf*.

I would like to see sftp: and scp: protocols go away. In reality they are just subsets of the OpenSSH (or other) package. You really can't run just scp: since we need 'ls' and 'del'. If we continue to allow these we need to emphasize that these are just aliases for ssh:.

review: Needs Fixing
edso (ed.so) wrote :

On 10.11.2014 17:00, Kenneth Loafman wrote:
> Review: Needs Fixing
>
> I like the general cleanup.
>
> A couple of changes:
>
> Please leave the file naming as it was. I had set these to be underscore-first to show they were called by the main ssh backend. See _boto*, _cf*.

with my mods there is no ssh backend anymore. backends can provide by registering scheme backend+sftp, backend+scp .. the default backends simply get assigned the prefixless scheme in addition.

as these backends are now equal i removed the underscore.. for the future i planned to remove the backend switching parameters for boto/cf as well and streamline it into the use of prefixes instead.

> I would like to see sftp: and scp: protocols go away.

why?

>In reality they are just subsets of the OpenSSH (or other) package. You really can't run just scp: since we need 'ls' and 'del'. If we continue to allow these we need to emphasize that these are just aliases for ssh:.
>

not true. paramiko has a proper scp support using _no_ (open)ssh binaries and listing via ssh remote shell, as far as i have seen.

i see no problem listing openssh as a requirement (like for the pexpect backend) for lftp+sftp then.

..ede

Kenneth Loafman (kenneth-loafman) wrote :

OK. My confusion. I was looking at the old sshbackend.py for reference.

I deleted the old sshbackend.py since it's no long valid and fixed a couple of format issues.

edso (ed.so) wrote :

On 10.11.2014 21:08, Kenneth Loafman wrote:
> OK. My confusion. I was looking at the old sshbackend.py for reference.
>
> I deleted the old sshbackend.py since it's no long valid and fixed a couple of format issues.
>

nP.. i actually meant to delete sshbackend.py in the branch but i seem to have forgotten to commit it..

what were the format issues you mention? ..ede

Kenneth Loafman (kenneth-loafman) wrote :

No line feed at end of file.

Python does not need backslash on list continuation.

That kind of thing.

On Mon, Nov 10, 2014 at 2:18 PM, edso <email address hidden> wrote:

> On 10.11.2014 21:08, Kenneth Loafman wrote:
> > OK. My confusion. I was looking at the old sshbackend.py for reference.
> >
> > I deleted the old sshbackend.py since it's no long valid and fixed a
> couple of format issues.
> >
>
> nP.. i actually meant to delete sshbackend.py in the branch but i seem to
> have forgotten to commit it..
>
> what were the format issues you mention? ..ede
>
> --
>
> https://code.launchpad.net/~ed.so/duplicity/lftp.ncftp.and.prefixes/+merge/240149
> You are reviewing the proposed merge of lp:~
> ed.so/duplicity/lftp.ncftp.and.prefixes into lp:duplicity.
>

edso (ed.so) wrote :

ok, good to know.. thanks ede

On 10.11.2014 21:57, Kenneth Loafman wrote:
> No line feed at end of file.
>
> Python does not need backslash on list continuation.
>
> That kind of thing.
>
>
> On Mon, Nov 10, 2014 at 2:18 PM, edso <email address hidden> wrote:
>
>> On 10.11.2014 21:08, Kenneth Loafman wrote:
>>> OK. My confusion. I was looking at the old sshbackend.py for reference.
>>>
>>> I deleted the old sshbackend.py since it's no long valid and fixed a
>> couple of format issues.
>>>
>>
>> nP.. i actually meant to delete sshbackend.py in the branch but i seem to
>> have forgotten to commit it..
>>
>> what were the format issues you mention? ..ede
>>
>> --
>>
>> https://code.launchpad.net/~ed.so/duplicity/lftp.ncftp.and.prefixes/+merge/240149
>> You are reviewing the proposed merge of lp:~
>> ed.so/duplicity/lftp.ncftp.and.prefixes into lp:duplicity.
>>
>

Kenneth Loafman (kenneth-loafman) wrote :

By no line feed at end of file, I mean there were a couple of missing line
feeds at the end of the file. Easy to get meanings reversed.

On Tue, Nov 11, 2014 at 3:45 AM, edso <email address hidden> wrote:

> ok, good to know.. thanks ede
>
> On 10.11.2014 21:57, Kenneth Loafman wrote:
> > No line feed at end of file.
> >
> > Python does not need backslash on list continuation.
> >
> > That kind of thing.
> >
> >
> > On Mon, Nov 10, 2014 at 2:18 PM, edso <email address hidden> wrote:
> >
> >> On 10.11.2014 21:08, Kenneth Loafman wrote:
> >>> OK. My confusion. I was looking at the old sshbackend.py for
> reference.
> >>>
> >>> I deleted the old sshbackend.py since it's no long valid and fixed a
> >> couple of format issues.
> >>>
> >>
> >> nP.. i actually meant to delete sshbackend.py in the branch but i seem
> to
> >> have forgotten to commit it..
> >>
> >> what were the format issues you mention? ..ede
> >>
> >> --
> >>
> >>
> https://code.launchpad.net/~ed.so/duplicity/lftp.ncftp.and.prefixes/+merge/240149
> >> You are reviewing the proposed merge of lp:~
> >> ed.so/duplicity/lftp.ncftp.and.prefixes into lp:duplicity.
> >>
> >
>
> --
>
> https://code.launchpad.net/~ed.so/duplicity/lftp.ncftp.and.prefixes/+merge/240149
> You are reviewing the proposed merge of lp:~
> ed.so/duplicity/lftp.ncftp.and.prefixes into lp:duplicity.
>

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'bin/duplicity.1'
2--- bin/duplicity.1 2014-10-23 11:56:13 +0000
3+++ bin/duplicity.1 2014-10-30 18:01:04 +0000
4@@ -68,22 +68,14 @@
5 .B Rackspace CloudFiles Pyrax API
6 - http://docs.rackspace.com/sdks/guide/content/python.html
7 .TP
8-.B "dpbx backend" (Dropbox)
9+.BR "dpbx backend" " (Dropbox)"
10 .B Dropbox Python SDK
11 - https://www.dropbox.com/developers/reference/sdk
12 .TP
13-.B "copy backend" (Copy.com)
14+.BR "copy backend" " (Copy.com)"
15 .B python-urllib3
16 - https://github.com/shazow/urllib3
17 .TP
18-.B "ftp backend"
19-.B NcFTP Client
20-- http://www.ncftp.com/
21-.TP
22-.B "ftps backend"
23-.B LFTP Client
24-- http://lftp.yar.ru/
25-.TP
26 .BR "gdocs backend" " (Google Docs)"
27 .B Google Data APIs Python Client Library
28 - http://code.google.com/p/gdata-python-client/
29@@ -95,17 +87,25 @@
30 .B D-Bus
31 (dbus)- http://www.freedesktop.org/wiki/Software/dbus
32 .TP
33-.B "rsync backend"
34-.B rsync client binary
35-- http://rsync.samba.org/
36+.BR "lftp backend" " (needed for ftp, ftps, fish [over ssh] - also supports sftp, webdav[s])"
37+.B LFTP Client
38+- http://lftp.yar.ru/
39 .TP
40 .BR "mega backend" " (mega.co.nz)"
41 .B Python library for mega API
42 - https://github.com/ckornacker/mega.py, ubuntu ppa - ppa:ckornacker/backup
43 .TP
44+.BR "ncftp backend" " (ftp, select via ncftp+ftp://)"
45+.B NcFTP
46+- http://www.ncftp.com/
47+.TP
48 .B "Par2 Wrapper Backend"
49 .B par2cmdline
50 - http://parchive.sourceforge.net/
51+.TP
52+.B "rsync backend"
53+.B rsync client binary
54+- http://rsync.samba.org/
55 .PP
56 There are two
57 .B ssh backends
58@@ -1067,87 +1067,169 @@
59 or preceded by a double slash, '//path', to represent an absolute
60 filesystem path.
61 .PP
62+.B Note:
63+.RS
64+Scheme (protocol) access may be provided by more than one backend.
65+In case the default backend is buggy or simply not working in a specific case it might be worth trying an alternative implementation.
66+Alternative backends can be selected by prefixing the scheme with the name of the alternative backend e.g.
67+.B ncftp+ftp://
68+and are mentioned below the scheme's syntax summary.
69+.RE
70+
71+.PP
72 Formats of each of the URL schemes follow:
73+
74+.PP
75+.BR "Cloud Files" " (Rackspace)"
76+.PP
77 .RS
78-.PP
79-.BI "Rackspace Cloud Files"
80-.br
81 cf+http://container_name
82-.br
83+.PP
84 See also
85 .B "A NOTE ON CLOUD FILES ACCESS"
86-.PP
87-.BI Dropbox
88-.br
89+.RE
90+.PP
91+.B "Copy cloud storage"
92+.PP
93+.RS
94+copy://user[:password]@copy.com/some_dir
95+.RE
96+.PP
97+.B Dropbox
98+.PP
99+.RS
100 dpbx:///some_dir
101-.br
102+.PP
103 Make sure to read
104 .BR "A NOTE ON DROPBOX ACCESS" " first!"
105-.PP
106-copy://user[:password]@copy.com/some_dir
107-.PP
108-.PP
109+.RE
110+.PP
111+.B "Local file path"
112+.PP
113+.RS
114 file://[relative|/absolute]/local/path
115-.PP
116+.RE
117+.PP
118+.BR "FISH" " (Files transferred over Shell protocol) over ssh"
119+.PP
120+.RS
121+fish://user[:password]@other.host[:port]/[relative|/absolute]_path
122+.RE
123+.PP
124+.B "FTP"
125+.PP
126+.RS
127 ftp[s]://user[:password]@other.host[:port]/some_dir
128 .PP
129+.B NOTE:
130+use lftp+, ncftp+ prefixes to enforce a specific backend, e.g. ncftp+ftp://...
131+.RE
132+.PP
133+.B "Google Docs"
134+.PP
135+.RS
136 gdocs://user[:password]@other.host/some_dir
137-.PP
138-.BI "Google Cloud Storage"
139-.br
140+.RE
141+.PP
142+.B "Google Cloud Storage"
143+.PP
144+.RS
145 gs://bucket[/prefix]
146-.PP
147+.RE
148+.PP
149+.B "HSI"
150+.PP
151+.RS
152 hsi://user[:password]@other.host/some_dir
153-.PP
154+.RE
155+.PP
156+.B "IMAP email storage"
157+.PP
158+.RS
159 imap[s]://user[:password]@host.com[/from_address_prefix]
160-.br
161+.PP
162 See also
163 .B "A NOTE ON IMAP"
164-.PP
165+.RE
166+.PP
167+.B "Mega cloud storage"
168+.PP
169+.RS
170 mega://user[:password]@mega.co.nz/some_dir
171-.PP
172-.BI "Par2 Wrapper Backend"
173-.br
174+.RE
175+.PP
176+.B "Par2 Wrapper Backend"
177+.PP
178+.RS
179 par2+scheme://[user[:password]@]host[:port]/[/]path
180-.br
181+.PP
182 See also
183 .B "A NOTE ON PAR2 WRAPPER BACKEND"
184-.PP
185-.B "using rsync daemon"
186-.br
187+.RE
188+.PP
189+.B "Rsync via daemon"
190+.PP
191+.RS
192 rsync://user[:password]@host.com[:port]::[/]module/some_dir
193-.br
194-.B "using rsync over ssh (only key auth)"
195-.br
196+.PP
197+.RE
198+.B "Rsync over ssh (only key auth)"
199+.PP
200+.RS
201 rsync://user@host.com[:port]/[relative|/absolute]_path
202-.PP
203+.RE
204+.PP
205+.BR "S3 storage" " (Amazon)"
206+.PP
207+.RS
208 s3://host/bucket_name[/prefix]
209 .br
210 s3+http://bucket_name[/prefix]
211-.br
212+.PP
213 See also
214 .B "A NOTE ON EUROPEAN S3 BUCKETS"
215-.PP
216-scp://.. or ssh://.. are synonymous with
217-.br
218-sftp://user[:password]@other.host[:port]/[/]some_dir
219+.RE
220+.PP
221+.B "SCP/SFTP access"
222+.PP
223+.RS
224+scp://.. or
225+.br
226+sftp://user[:password]@other.host[:port]/[relative|/absolute]_path
227+.PP
228+.BR "defaults" " are paramiko+scp:// and paramiko+sftp://"
229+.br
230+.BR "alternatively" " try pexpect+scp://, pexpect+sftp://, lftp+sftp://"
231 .br
232 See also
233-.BR --ssh-backend ,
234 .BR --ssh-askpass ,
235-.BR --use-scp ,
236 .B --ssh-options
237 and
238 .BR "A NOTE ON SSH BACKENDS" .
239-.PP
240+.RE
241+.PP
242+.BR "Swift" " (Openstack)"
243+.PP
244+.RS
245 swift://container_name
246-.br
247+.PP
248 See also
249 .B "A NOTE ON SWIFT (OPENSTACK OBJECT STORAGE) ACCESS"
250-.PP
251+.RE
252+.PP
253+.B "Tahoe-LAFS"
254+.PP
255+.RS
256 tahoe://alias/directory
257-.PP
258+.RE
259+.PP
260+.B "WebDAV"
261+.PP
262+.RS
263 webdav[s]://user[:password]@other.host[:port]/some_dir
264+.PP
265+.B alternatively
266+try lftp+webdav[s]://
267 .RE
268
269 .SH TIME FORMATS
270
271=== modified file 'duplicity/backend.py'
272--- duplicity/backend.py 2014-10-27 02:27:36 +0000
273+++ duplicity/backend.py 2014-10-30 18:01:04 +0000
274@@ -32,6 +32,7 @@
275 import re
276 import getpass
277 import gettext
278+import re
279 import types
280 import urllib
281 import urlparse
282@@ -164,6 +165,11 @@
283
284 _backend_prefixes[scheme] = backend_factory
285
286+def strip_prefix(url_string, prefix_scheme):
287+ """
288+ strip the prefix from a string e.g. par2+ftp://... -> ftp://...
289+ """
290+ return re.sub('(?i)^'+re.escape(prefix_scheme)+'\+','',url_string)
291
292 def is_backend_url(url_string):
293 """
294@@ -198,7 +204,7 @@
295 for prefix in _backend_prefixes:
296 if url_string.startswith(prefix + '+'):
297 factory = _backend_prefixes[prefix]
298- pu = ParsedUrl(url_string.lstrip(prefix + '+'))
299+ pu = ParsedUrl(strip_prefix(url_string,prefix))
300 break
301
302 if factory is None:
303@@ -337,11 +343,8 @@
304 def strip_auth_from_url(parsed_url):
305 """Return a URL from a urlparse object without a username or password."""
306
307- # Get a copy of the network location without the username or password.
308- straight_netloc = parsed_url.netloc.split('@')[-1]
309-
310- # Replace the full network location with the stripped copy.
311- return parsed_url.geturl().replace(parsed_url.netloc, straight_netloc, 1)
312+ clean_url = re.sub('^([^:/]+://)(.*@)?(.*)',r'\1\3',parsed_url.geturl())
313+ return clean_url
314
315 def _get_code_from_exception(backend, operation, e):
316 if isinstance(e, BackendException) and e.code != log.ErrorCode.backend_error:
317
318=== renamed file 'duplicity/backends/ftpbackend.py' => 'duplicity/backends/lftpbackend.py'
319--- duplicity/backends/ftpbackend.py 2014-10-01 20:35:16 +0000
320+++ duplicity/backends/lftpbackend.py 2014-10-30 18:01:04 +0000
321@@ -3,7 +3,10 @@
322 # Copyright 2002 Ben Escoto <ben@emerose.org>
323 # Copyright 2007 Kenneth Loafman <kenneth@loafman.com>
324 # Copyright 2010 Marcel Pennewiss <opensource@pennewiss.de>
325-# Copyright 2014 Moritz Maisel <moritz@maisel.name>
326+# Copyright 2014 Edgar Soldin
327+# - webdav, fish, sftp support
328+# - https cert verification switches
329+# - debug output
330 #
331 # This file is part of duplicity.
332 #
333@@ -23,15 +26,15 @@
334
335 import os
336 import os.path
337+import re
338 import urllib
339-import re
340
341 import duplicity.backend
342 from duplicity import globals
343 from duplicity import log
344 from duplicity import tempdir
345
346-class FTPBackend(duplicity.backend.Backend):
347+class LFTPBackend(duplicity.backend.Backend):
348 """Connect to remote store using File Transfer Protocol"""
349 def __init__(self, parsed_url):
350 duplicity.backend.Backend.__init__(self, parsed_url)
351@@ -54,61 +57,141 @@
352
353 self.parsed_url = parsed_url
354
355- self.url_string = duplicity.backend.strip_auth_from_url(self.parsed_url)
356+# self.url_string = duplicity.backend.strip_auth_from_url(self.parsed_url)
357+# # strip lftp+ prefix
358+# self.url_string = duplicity.backend.strip_prefix(self.url_string, 'lftp')
359+
360+ self.scheme = duplicity.backend.strip_prefix( parsed_url.scheme, 'lftp' ).lower()
361+ self.scheme = re.sub('^webdav','http',self.scheme)
362+ self.url_string = self.scheme + '://' + parsed_url.hostname
363+ if parsed_url.port :
364+ self.url_string += ":%s" % parsed_url.port
365+
366+ self.remote_path = re.sub('^/','',parsed_url.path)
367
368 # Use an explicit directory name.
369- if self.url_string[-1] != '/':
370- self.url_string += '/'
371+ if self.remote_path[-1] != '/':
372+ self.remote_path += '/'
373
374- self.password = self.get_password()
375+ self.authflag = ''
376+ if self.parsed_url.username:
377+ self.username = self.parsed_url.username
378+ self.password = self.get_password()
379+ self.authflag = "-u '%s,%s'" % (self.username,self.password)
380
381 if globals.ftp_connection == 'regular':
382 self.conn_opt = 'off'
383 else:
384 self.conn_opt = 'on'
385
386- if parsed_url.port != None and parsed_url.port != 21:
387- self.portflag = " -p '%s'" % (parsed_url.port)
388- else:
389- self.portflag = ""
390+ # check for cacert file if https
391+ self.cacert_file = globals.ssl_cacert_file
392+ if self.scheme == 'https' and not globals.ssl_no_check_certificate:
393+ cacert_candidates = [ "~/.duplicity/cacert.pem", \
394+ "~/duplicity_cacert.pem", \
395+ "/etc/duplicity/cacert.pem" ]
396+ #
397+ if not self.cacert_file:
398+ for path in cacert_candidates :
399+ path = os.path.expanduser(path)
400+ if (os.path.isfile(path)):
401+ self.cacert_file = path
402+ break
403+ # still no cacert file, inform user
404+ if not self.cacert_file:
405+ raise duplicity.errors.FatalBackendException("""For certificate verification a cacert database file is needed in one of these locations: %s
406+Hints:
407+ Consult the man page, chapter 'SSL Certificate Verification'.
408+ Consider using the options --ssl-cacert-file, --ssl-no-check-certificate .""" % ", ".join(cacert_candidates) )
409
410 self.tempfile, self.tempname = tempdir.default().mkstemp()
411+ os.write(self.tempfile, "set ssl:verify-certificate " + ( "false" if globals.ssl_no_check_certificate else "true" ) + "\n")
412+ if globals.ssl_cacert_file :
413+ os.write(self.tempfile, "set ssl:ca-file '" + globals.ssl_cacert_file + "'\n")
414 os.write(self.tempfile, "set ftp:ssl-allow true\n")
415 os.write(self.tempfile, "set ftp:ssl-protect-data true\n")
416 os.write(self.tempfile, "set ftp:ssl-protect-list true\n")
417+ os.write(self.tempfile, "set http:use-propfind true\n")
418 os.write(self.tempfile, "set net:timeout %s\n" % globals.timeout)
419 os.write(self.tempfile, "set net:max-retries %s\n" % globals.num_retries)
420 os.write(self.tempfile, "set ftp:passive-mode %s\n" % self.conn_opt)
421- os.write(self.tempfile, "open %s %s\n" % (self.portflag, self.parsed_url.hostname))
422+ if log.getverbosity() >= log.DEBUG :
423+ os.write(self.tempfile, "debug\n")
424+ os.write(self.tempfile, "open %s %s\n" % (self.authflag, self.url_string) )
425+# os.write(self.tempfile, "open %s %s\n" % (self.portflag, self.parsed_url.hostname))
426 # allow .netrc auth by only setting user/pass when user was actually given
427- if self.parsed_url.username:
428- os.write(self.tempfile, "user %s %s\n" % (self.parsed_url.username, self.password))
429+# if self.parsed_url.username:
430+# os.write(self.tempfile, "user %s %s\n" % (self.parsed_url.username, self.password))
431 os.close(self.tempfile)
432+ if log.getverbosity() >= log.DEBUG :
433+ f = open(self.tempname, 'r')
434+ log.Debug("SETTINGS: \n"
435+ "%s" % f.readlines() )
436
437 def _put(self, source_path, remote_filename):
438- remote_path = os.path.join(urllib.unquote(self.parsed_url.path.lstrip('/')), remote_filename).rstrip()
439- commandline = "lftp -c 'source %s;put \'%s\' -o \'%s\''" % \
440- (self.tempname, source_path.name, remote_path)
441- self.subprocess_popen(commandline)
442+ #remote_path = os.path.join(urllib.unquote(self.parsed_url.path.lstrip('/')), remote_filename).rstrip()
443+ commandline = "lftp -c 'source \'%s\'; mkdir -p %s; put \'%s\' -o \'%s\''" % \
444+ (self.tempname, self.remote_path, source_path.name, self.remote_path + remote_filename)
445+ log.Debug("CMD: %s" % commandline)
446+ s, l, e = self.subprocess_popen(commandline)
447+ log.Debug("STATUS: %s" % s)
448+ log.Debug("STDERR:\n"
449+ "%s" % (e))
450+ log.Debug("STDOUT:\n"
451+ "%s" % (l))
452
453 def _get(self, remote_filename, local_path):
454- remote_path = os.path.join(urllib.unquote(self.parsed_url.path), remote_filename).rstrip()
455- commandline = "lftp -c 'source %s;get %s -o %s'" % \
456- (self.tempname, remote_path.lstrip('/'), local_path.name)
457- self.subprocess_popen(commandline)
458+ #remote_path = os.path.join(urllib.unquote(self.parsed_url.path), remote_filename).rstrip()
459+ commandline = "lftp -c 'source \'%s\'; get \'%s\' -o \'%s\''" % \
460+ (self.tempname, self.remote_path+remote_filename, local_path.name)
461+ log.Debug("CMD: %s" % commandline)
462+ _, l, e = self.subprocess_popen(commandline)
463+ log.Debug("STDERR:\n"
464+ "%s" % (e))
465+ log.Debug("STDOUT:\n"
466+ "%s" % (l))
467
468 def _list(self):
469 # Do a long listing to avoid connection reset
470- remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
471- commandline = "lftp -c 'source %s;ls \'%s\''" % (self.tempname, remote_dir)
472- _, l, _ = self.subprocess_popen(commandline)
473+ #remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
474+ remote_dir = urllib.unquote(self.parsed_url.path)
475+ #print remote_dir
476+ commandline = "lftp -c 'source \'%s\'; cd \'%s\' || exit 0; ls'" % (self.tempname, self.remote_path)
477+ log.Debug("CMD: %s" % commandline)
478+ _, l, e = self.subprocess_popen(commandline)
479+ log.Debug("STDERR:\n"
480+ "%s" % (e))
481+ log.Debug("STDOUT:\n"
482+ "%s" % (l))
483+
484 # Look for our files as the last element of a long list line
485 return [x.split()[-1] for x in l.split('\n') if x]
486
487 def _delete(self, filename):
488- remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
489- commandline = "lftp -c 'source %s;cd \'%s\';rm \'%s\''" % (self.tempname, remote_dir, filename)
490- self.subprocess_popen(commandline)
491-
492-duplicity.backend.register_backend("ftp", FTPBackend)
493-duplicity.backend.register_backend("ftps", FTPBackend)
494+ #remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip()
495+ commandline = "lftp -c 'source \'%s\'; cd \'%s\'; rm \'%s\''" % (self.tempname, self.remote_path, filename)
496+ log.Debug("CMD: %s" % commandline)
497+ _, l, e = self.subprocess_popen(commandline)
498+ log.Debug("STDERR:\n"
499+ "%s" % (e))
500+ log.Debug("STDOUT:\n"
501+ "%s" % (l))
502+
503+duplicity.backend.register_backend("ftp", LFTPBackend)
504+duplicity.backend.register_backend("ftps", LFTPBackend)
505+duplicity.backend.register_backend("fish", LFTPBackend)
506+
507+duplicity.backend.register_backend("lftp+ftp", LFTPBackend)
508+duplicity.backend.register_backend("lftp+ftps", LFTPBackend)
509+duplicity.backend.register_backend("lftp+fish", LFTPBackend)
510+duplicity.backend.register_backend("lftp+sftp", LFTPBackend)
511+duplicity.backend.register_backend("lftp+webdav", LFTPBackend)
512+duplicity.backend.register_backend("lftp+webdavs", LFTPBackend)
513+duplicity.backend.register_backend("lftp+http", LFTPBackend)
514+duplicity.backend.register_backend("lftp+https", LFTPBackend)
515+
516+duplicity.backend.uses_netloc.extend([ 'ftp', 'ftps', 'fish',\
517+ 'lftp+ftp', 'lftp+ftps',\
518+ 'lftp+fish', 'lftp+sftp',\
519+ 'lftp+webdav', 'lftp+webdavs',\
520+ 'lftp+http', 'lftp+https' ])
521\ No newline at end of file
522
523=== added file 'duplicity/backends/ncftpbackend.py'
524--- duplicity/backends/ncftpbackend.py 1970-01-01 00:00:00 +0000
525+++ duplicity/backends/ncftpbackend.py 2014-10-30 18:01:04 +0000
526@@ -0,0 +1,118 @@
527+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
528+#
529+# Copyright 2002 Ben Escoto <ben@emerose.org>
530+# Copyright 2007 Kenneth Loafman <kenneth@loafman.com>
531+#
532+# This file is part of duplicity.
533+#
534+# Duplicity is free software; you can redistribute it and/or modify it
535+# under the terms of the GNU General Public License as published by the
536+# Free Software Foundation; either version 2 of the License, or (at your
537+# option) any later version.
538+#
539+# Duplicity is distributed in the hope that it will be useful, but
540+# WITHOUT ANY WARRANTY; without even the implied warranty of
541+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
542+# General Public License for more details.
543+#
544+# You should have received a copy of the GNU General Public License
545+# along with duplicity; if not, write to the Free Software Foundation,
546+# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
547+
548+import os.path
549+import urllib
550+
551+import duplicity.backend
552+from duplicity import globals
553+from duplicity import log
554+from duplicity import tempdir
555+
556+class NCFTPBackend(duplicity.backend.Backend):
557+ """Connect to remote store using File Transfer Protocol"""
558+ def __init__(self, parsed_url):
559+ duplicity.backend.Backend.__init__(self, parsed_url)
560+
561+ # we expect an error return, so go low-level and ignore it
562+ try:
563+ p = os.popen("ncftpls -v")
564+ fout = p.read()
565+ ret = p.close()
566+ except Exception:
567+ pass
568+ # the expected error is 8 in the high-byte and some output
569+ if ret != 0x0800 or not fout:
570+ log.FatalError("NcFTP not found: Please install NcFTP version 3.1.9 or later",
571+ log.ErrorCode.ftp_ncftp_missing)
572+
573+ # version is the second word of the first line
574+ version = fout.split('\n')[0].split()[1]
575+ if version < "3.1.9":
576+ log.FatalError("NcFTP too old: Duplicity requires NcFTP version 3.1.9,"
577+ "3.2.1 or later. Version 3.2.0 will not work properly.",
578+ log.ErrorCode.ftp_ncftp_too_old)
579+ elif version == "3.2.0":
580+ log.Warn("NcFTP (ncftpput) version 3.2.0 may fail with duplicity.\n"
581+ "see: http://www.ncftpd.com/ncftp/doc/changelog.html\n"
582+ "If you have trouble, please upgrade to 3.2.1 or later",
583+ log.WarningCode.ftp_ncftp_v320)
584+ log.Notice("NcFTP version is %s" % version)
585+
586+ self.parsed_url = parsed_url
587+
588+ self.url_string = duplicity.backend.strip_auth_from_url(self.parsed_url)
589+
590+ # strip ncftp+ prefix
591+ self.url_string = duplicity.backend.strip_prefix(self.url_string, 'ncftp')
592+
593+ # This squelches the "file not found" result from ncftpls when
594+ # the ftp backend looks for a collection that does not exist.
595+ # version 3.2.2 has error code 5, 1280 is some legacy value
596+ self.popen_breaks[ 'ncftpls' ] = [ 5, 1280 ]
597+
598+ # Use an explicit directory name.
599+ if self.url_string[-1] != '/':
600+ self.url_string += '/'
601+
602+ self.password = self.get_password()
603+
604+ if globals.ftp_connection == 'regular':
605+ self.conn_opt = '-E'
606+ else:
607+ self.conn_opt = '-F'
608+
609+ self.tempfile, self.tempname = tempdir.default().mkstemp()
610+ os.write(self.tempfile, "host %s\n" % self.parsed_url.hostname)
611+ os.write(self.tempfile, "user %s\n" % self.parsed_url.username)
612+ os.write(self.tempfile, "pass %s\n" % self.password)
613+ os.close(self.tempfile)
614+ self.flags = "-f %s %s -t %s -o useCLNT=0,useHELP_SITE=0 " % \
615+ (self.tempname, self.conn_opt, globals.timeout)
616+ if parsed_url.port != None and parsed_url.port != 21:
617+ self.flags += " -P '%s'" % (parsed_url.port)
618+
619+ def _put(self, source_path, remote_filename):
620+ remote_path = os.path.join(urllib.unquote(self.parsed_url.path.lstrip('/')), remote_filename).rstrip()
621+ commandline = "ncftpput %s -m -V -C '%s' '%s'" % \
622+ (self.flags, source_path.name, remote_path)
623+ self.subprocess_popen(commandline)
624+
625+ def _get(self, remote_filename, local_path):
626+ remote_path = os.path.join(urllib.unquote(self.parsed_url.path), remote_filename).rstrip()
627+ commandline = "ncftpget %s -V -C '%s' '%s' '%s'" % \
628+ (self.flags, self.parsed_url.hostname, remote_path.lstrip('/'), local_path.name)
629+ self.subprocess_popen(commandline)
630+
631+ def _list(self):
632+ # Do a long listing to avoid connection reset
633+ commandline = "ncftpls %s -l '%s'" % (self.flags, self.url_string)
634+ _, l, _ = self.subprocess_popen(commandline)
635+ # Look for our files as the last element of a long list line
636+ return [x.split()[-1] for x in l.split('\n') if x and not x.startswith("total ")]
637+
638+ def _delete(self, filename):
639+ commandline = "ncftpls %s -l -X 'DELE %s' '%s'" % \
640+ (self.flags, filename, self.url_string)
641+ self.subprocess_popen(commandline)
642+
643+duplicity.backend.register_backend("ncftp+ftp", NCFTPBackend)
644+duplicity.backend.uses_netloc.extend([ 'ncftp+ftp' ])
645\ No newline at end of file
646
647=== renamed file 'duplicity/backends/_ssh_paramiko.py' => 'duplicity/backends/ssh_paramiko_backend.py'
648--- duplicity/backends/_ssh_paramiko.py 2014-10-27 03:02:20 +0000
649+++ duplicity/backends/ssh_paramiko_backend.py 2014-10-30 18:01:04 +0000
650@@ -217,8 +217,11 @@
651 self.config['port'],e))
652 self.client.get_transport().set_keepalive((int)(globals.timeout / 2))
653
654+ self.scheme = duplicity.backend.strip_prefix(parsed_url.scheme, 'paramiko')
655+ self.use_scp = ( self.scheme == 'scp' )
656+
657 # scp or sftp?
658- if (globals.use_scp):
659+ if (self.use_scp):
660 # sanity-check the directory name
661 if (re.search("'",self.remote_dir)):
662 raise BackendException("cannot handle directory names with single quotes with --use-scp!")
663@@ -256,7 +259,7 @@
664 raise BackendException("sftp chdir to %s failed: %s" % (self.sftp.normalize(".")+"/"+d,e))
665
666 def _put(self, source_path, remote_filename):
667- if globals.use_scp:
668+ if self.use_scp:
669 f=file(source_path.name,'rb')
670 try:
671 chan=self.client.get_transport().open_session()
672@@ -284,7 +287,7 @@
673 self.sftp.put(source_path.name,remote_filename)
674
675 def _get(self, remote_filename, local_path):
676- if globals.use_scp:
677+ if self.use_scp:
678 try:
679 chan=self.client.get_transport().open_session()
680 chan.settimeout(globals.timeout)
681@@ -327,7 +330,7 @@
682 def _list(self):
683 # In scp mode unavoidable quoting issues will make this fail if the
684 # directory name contains single quotes.
685- if globals.use_scp:
686+ if self.use_scp:
687 output = self.runremote("ls -1 '%s'" % self.remote_dir, False, "scp dir listing ")
688 return output.splitlines()
689 else:
690@@ -336,7 +339,7 @@
691 def _delete(self, filename):
692 # In scp mode unavoidable quoting issues will cause failures if
693 # filenames containing single quotes are encountered.
694- if globals.use_scp:
695+ if self.use_scp:
696 self.runremote("rm '%s/%s'" % (self.remote_dir, filename), False, "scp rm ")
697 else:
698 self.sftp.remove(filename)
699@@ -370,3 +373,9 @@
700 raise BackendException("could not load '%s', maybe corrupt?" % (file))
701
702 return sshconfig.lookup(host)
703+
704+duplicity.backend.register_backend("sftp", SSHParamikoBackend)
705+duplicity.backend.register_backend("scp", SSHParamikoBackend)
706+duplicity.backend.register_backend("paramiko+sftp", SSHParamikoBackend)
707+duplicity.backend.register_backend("paramiko+scp", SSHParamikoBackend)
708+duplicity.backend.uses_netloc.extend([ 'sftp', 'scp', 'paramiko+sftp', 'paramiko+scp' ])
709
710=== renamed file 'duplicity/backends/_ssh_pexpect.py' => 'duplicity/backends/ssh_pexpect_backend.py'
711--- duplicity/backends/_ssh_pexpect.py 2014-04-28 02:49:39 +0000
712+++ duplicity/backends/ssh_pexpect_backend.py 2014-10-30 18:01:04 +0000
713@@ -49,6 +49,9 @@
714
715 self.sftp_command = "sftp"
716 if globals.sftp_command: self.sftp_command = globals.sftp_command
717+
718+ self.scheme = duplicity.backend.strip_prefix(parsed_url.scheme, 'pexpect')
719+ self.use_scp = ( self.scheme == 'scp' )
720
721 # host string of form [user@]hostname
722 if parsed_url.username:
723@@ -212,7 +215,7 @@
724 raise BackendException("Error running '%s': %s" % (commandline, msg))
725
726 def _put(self, source_path, remote_filename):
727- if globals.use_scp:
728+ if self.use_scp:
729 self.put_scp(source_path, remote_filename)
730 else:
731 self.put_sftp(source_path, remote_filename)
732@@ -234,7 +237,7 @@
733 self.run_scp_command(commandline)
734
735 def _get(self, remote_filename, local_path):
736- if globals.use_scp:
737+ if self.use_scp:
738 self.get_scp(remote_filename, local_path)
739 else:
740 self.get_sftp(remote_filename, local_path)
741@@ -280,3 +283,8 @@
742 commands.append("rm \"%s\"" % filename)
743 commandline = ("%s %s %s" % (self.sftp_command, globals.ssh_options, self.host_string))
744 self.run_sftp_command(commandline, commands)
745+
746+duplicity.backend.register_backend("pexpect+sftp", SSHPExpectBackend)
747+duplicity.backend.register_backend("pexpect+scp", SSHPExpectBackend)
748+duplicity.backend.uses_netloc.extend([ 'pexpect+sftp', 'pexpect+scp' ])
749+
750
751=== modified file 'duplicity/commandline.py'
752--- duplicity/commandline.py 2014-10-27 02:27:36 +0000
753+++ duplicity/commandline.py 2014-10-30 18:01:04 +0000
754@@ -537,9 +537,6 @@
755 # default to batch mode using public-key encryption
756 parser.add_option("--ssh-askpass", action = "store_true")
757
758- # allow the user to switch ssh backend
759- parser.add_option("--ssh-backend", metavar = _("paramiko|pexpect"))
760-
761 # user added ssh options
762 parser.add_option("--ssh-options", action = "extend", metavar = _("options"))
763
764@@ -567,8 +564,6 @@
765 # Whether to specify --use-agent in GnuPG options
766 parser.add_option("--use-agent", action = "store_true")
767
768- parser.add_option("--use-scp", action = "store_true")
769-
770 parser.add_option("--verbosity", "-v", type = "verbosity", metavar = "[0-9]",
771 dest = "", action = "callback",
772 callback = lambda o, s, v, p: log.setverbosity(v))
773
774=== modified file 'duplicity/file_naming.py'
775--- duplicity/file_naming.py 2014-10-27 02:27:36 +0000
776+++ duplicity/file_naming.py 2014-10-30 18:01:04 +0000
777@@ -393,6 +393,20 @@
778 else:
779 pr.encrypted = None
780
781+ def valid_extension():
782+ """
783+ plausibility check for duplicity file extension
784+ before starting to extensively parse the filenames
785+ """
786+ res = re.match(r'.*\.(g|z|gpg|gz|tar|p|part|manifest|sigtar)$', filename )
787+ #print filename, res
788+ if res is None :
789+ return False
790+ return True
791+
792+ if not valid_extension() :
793+ return None
794+
795 pr = check_full()
796 if not pr:
797 pr = check_inc()
798
799=== modified file 'duplicity/globals.py'
800--- duplicity/globals.py 2014-05-12 07:09:00 +0000
801+++ duplicity/globals.py 2014-10-30 18:01:04 +0000
802@@ -231,15 +231,9 @@
803 # default to batch mode using public-key encryption
804 ssh_askpass = False
805
806-# default ssh backend is paramiko
807-ssh_backend = "paramiko"
808-
809 # user added ssh options
810 ssh_options = ""
811
812-# whether to use scp for put/get, sftp is default
813-use_scp = False
814-
815 # default cf backend is pyrax
816 cf_backend = "pyrax"
817

Subscribers

People subscribed via source and target branches