Merge lp:~ed.so/duplicity/lftp.ncftp.and.prefixes into lp:~duplicity-team/duplicity/0.7-series
- lftp.ncftp.and.prefixes
- Merge into 0.7-series
Status: | Merged |
---|---|
Merged at revision: | 1014 |
Proposed branch: | lp:~ed.so/duplicity/lftp.ncftp.and.prefixes |
Merge into: | lp:~duplicity-team/duplicity/0.7-series |
Diff against target: |
816 lines (+416/-110) 9 files modified
bin/duplicity.1 (+137/-55) duplicity/backend.py (+9/-6) duplicity/backends/lftpbackend.py (+114/-31) duplicity/backends/ncftpbackend.py (+118/-0) duplicity/backends/ssh_paramiko_backend.py (+14/-5) duplicity/backends/ssh_pexpect_backend.py (+10/-2) duplicity/commandline.py (+0/-5) duplicity/file_naming.py (+14/-0) duplicity/globals.py (+0/-6) |
To merge this branch: | bzr merge lp:~ed.so/duplicity/lftp.ncftp.and.prefixes |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Kenneth Loafman | Needs Fixing | ||
Review via email: mp+240149@code.launchpad.net |
Commit message
Description of the change
- retire --ssh-backend, --use-scp parameters
- introduce scheme prefixes for alternative backend selection e.g. ncftp+ftp://, see manpage
- scp is now selected via scheme e.g. scp://
- added lftp fish, webdav(s), sftp support
edso (ed.so) wrote : | # |
On 10.11.2014 17:00, Kenneth Loafman wrote:
> Review: Needs Fixing
>
> I like the general cleanup.
>
> A couple of changes:
>
> Please leave the file naming as it was. I had set these to be underscore-first to show they were called by the main ssh backend. See _boto*, _cf*.
with my mods there is no ssh backend anymore. backends can provide by registering scheme backend+sftp, backend+scp .. the default backends simply get assigned the prefixless scheme in addition.
as these backends are now equal i removed the underscore.. for the future i planned to remove the backend switching parameters for boto/cf as well and streamline it into the use of prefixes instead.
> I would like to see sftp: and scp: protocols go away.
why?
>In reality they are just subsets of the OpenSSH (or other) package. You really can't run just scp: since we need 'ls' and 'del'. If we continue to allow these we need to emphasize that these are just aliases for ssh:.
>
not true. paramiko has a proper scp support using _no_ (open)ssh binaries and listing via ssh remote shell, as far as i have seen.
i see no problem listing openssh as a requirement (like for the pexpect backend) for lftp+sftp then.
..ede
Kenneth Loafman (kenneth-loafman) wrote : | # |
OK. My confusion. I was looking at the old sshbackend.py for reference.
I deleted the old sshbackend.py since it's no long valid and fixed a couple of format issues.
edso (ed.so) wrote : | # |
On 10.11.2014 21:08, Kenneth Loafman wrote:
> OK. My confusion. I was looking at the old sshbackend.py for reference.
>
> I deleted the old sshbackend.py since it's no long valid and fixed a couple of format issues.
>
nP.. i actually meant to delete sshbackend.py in the branch but i seem to have forgotten to commit it..
what were the format issues you mention? ..ede
Kenneth Loafman (kenneth-loafman) wrote : | # |
No line feed at end of file.
Python does not need backslash on list continuation.
That kind of thing.
On Mon, Nov 10, 2014 at 2:18 PM, edso <email address hidden> wrote:
> On 10.11.2014 21:08, Kenneth Loafman wrote:
> > OK. My confusion. I was looking at the old sshbackend.py for reference.
> >
> > I deleted the old sshbackend.py since it's no long valid and fixed a
> couple of format issues.
> >
>
> nP.. i actually meant to delete sshbackend.py in the branch but i seem to
> have forgotten to commit it..
>
> what were the format issues you mention? ..ede
>
> --
>
> https:/
> You are reviewing the proposed merge of lp:~
> ed.so/duplicity
>
edso (ed.so) wrote : | # |
ok, good to know.. thanks ede
On 10.11.2014 21:57, Kenneth Loafman wrote:
> No line feed at end of file.
>
> Python does not need backslash on list continuation.
>
> That kind of thing.
>
>
> On Mon, Nov 10, 2014 at 2:18 PM, edso <email address hidden> wrote:
>
>> On 10.11.2014 21:08, Kenneth Loafman wrote:
>>> OK. My confusion. I was looking at the old sshbackend.py for reference.
>>>
>>> I deleted the old sshbackend.py since it's no long valid and fixed a
>> couple of format issues.
>>>
>>
>> nP.. i actually meant to delete sshbackend.py in the branch but i seem to
>> have forgotten to commit it..
>>
>> what were the format issues you mention? ..ede
>>
>> --
>>
>> https:/
>> You are reviewing the proposed merge of lp:~
>> ed.so/duplicity
>>
>
Kenneth Loafman (kenneth-loafman) wrote : | # |
By no line feed at end of file, I mean there were a couple of missing line
feeds at the end of the file. Easy to get meanings reversed.
On Tue, Nov 11, 2014 at 3:45 AM, edso <email address hidden> wrote:
> ok, good to know.. thanks ede
>
> On 10.11.2014 21:57, Kenneth Loafman wrote:
> > No line feed at end of file.
> >
> > Python does not need backslash on list continuation.
> >
> > That kind of thing.
> >
> >
> > On Mon, Nov 10, 2014 at 2:18 PM, edso <email address hidden> wrote:
> >
> >> On 10.11.2014 21:08, Kenneth Loafman wrote:
> >>> OK. My confusion. I was looking at the old sshbackend.py for
> reference.
> >>>
> >>> I deleted the old sshbackend.py since it's no long valid and fixed a
> >> couple of format issues.
> >>>
> >>
> >> nP.. i actually meant to delete sshbackend.py in the branch but i seem
> to
> >> have forgotten to commit it..
> >>
> >> what were the format issues you mention? ..ede
> >>
> >> --
> >>
> >>
> https:/
> >> You are reviewing the proposed merge of lp:~
> >> ed.so/duplicity
> >>
> >
>
> --
>
> https:/
> You are reviewing the proposed merge of lp:~
> ed.so/duplicity
>
Preview Diff
1 | === modified file 'bin/duplicity.1' | |||
2 | --- bin/duplicity.1 2014-10-23 11:56:13 +0000 | |||
3 | +++ bin/duplicity.1 2014-10-30 18:01:04 +0000 | |||
4 | @@ -68,22 +68,14 @@ | |||
5 | 68 | .B Rackspace CloudFiles Pyrax API | 68 | .B Rackspace CloudFiles Pyrax API |
6 | 69 | - http://docs.rackspace.com/sdks/guide/content/python.html | 69 | - http://docs.rackspace.com/sdks/guide/content/python.html |
7 | 70 | .TP | 70 | .TP |
9 | 71 | .B "dpbx backend" (Dropbox) | 71 | .BR "dpbx backend" " (Dropbox)" |
10 | 72 | .B Dropbox Python SDK | 72 | .B Dropbox Python SDK |
11 | 73 | - https://www.dropbox.com/developers/reference/sdk | 73 | - https://www.dropbox.com/developers/reference/sdk |
12 | 74 | .TP | 74 | .TP |
14 | 75 | .B "copy backend" (Copy.com) | 75 | .BR "copy backend" " (Copy.com)" |
15 | 76 | .B python-urllib3 | 76 | .B python-urllib3 |
16 | 77 | - https://github.com/shazow/urllib3 | 77 | - https://github.com/shazow/urllib3 |
17 | 78 | .TP | 78 | .TP |
18 | 79 | .B "ftp backend" | ||
19 | 80 | .B NcFTP Client | ||
20 | 81 | - http://www.ncftp.com/ | ||
21 | 82 | .TP | ||
22 | 83 | .B "ftps backend" | ||
23 | 84 | .B LFTP Client | ||
24 | 85 | - http://lftp.yar.ru/ | ||
25 | 86 | .TP | ||
26 | 87 | .BR "gdocs backend" " (Google Docs)" | 79 | .BR "gdocs backend" " (Google Docs)" |
27 | 88 | .B Google Data APIs Python Client Library | 80 | .B Google Data APIs Python Client Library |
28 | 89 | - http://code.google.com/p/gdata-python-client/ | 81 | - http://code.google.com/p/gdata-python-client/ |
29 | @@ -95,17 +87,25 @@ | |||
30 | 95 | .B D-Bus | 87 | .B D-Bus |
31 | 96 | (dbus)- http://www.freedesktop.org/wiki/Software/dbus | 88 | (dbus)- http://www.freedesktop.org/wiki/Software/dbus |
32 | 97 | .TP | 89 | .TP |
36 | 98 | .B "rsync backend" | 90 | .BR "lftp backend" " (needed for ftp, ftps, fish [over ssh] - also supports sftp, webdav[s])" |
37 | 99 | .B rsync client binary | 91 | .B LFTP Client |
38 | 100 | - http://rsync.samba.org/ | 92 | - http://lftp.yar.ru/ |
39 | 101 | .TP | 93 | .TP |
40 | 102 | .BR "mega backend" " (mega.co.nz)" | 94 | .BR "mega backend" " (mega.co.nz)" |
41 | 103 | .B Python library for mega API | 95 | .B Python library for mega API |
42 | 104 | - https://github.com/ckornacker/mega.py, ubuntu ppa - ppa:ckornacker/backup | 96 | - https://github.com/ckornacker/mega.py, ubuntu ppa - ppa:ckornacker/backup |
43 | 105 | .TP | 97 | .TP |
44 | 98 | .BR "ncftp backend" " (ftp, select via ncftp+ftp://)" | ||
45 | 99 | .B NcFTP | ||
46 | 100 | - http://www.ncftp.com/ | ||
47 | 101 | .TP | ||
48 | 106 | .B "Par2 Wrapper Backend" | 102 | .B "Par2 Wrapper Backend" |
49 | 107 | .B par2cmdline | 103 | .B par2cmdline |
50 | 108 | - http://parchive.sourceforge.net/ | 104 | - http://parchive.sourceforge.net/ |
51 | 105 | .TP | ||
52 | 106 | .B "rsync backend" | ||
53 | 107 | .B rsync client binary | ||
54 | 108 | - http://rsync.samba.org/ | ||
55 | 109 | .PP | 109 | .PP |
56 | 110 | There are two | 110 | There are two |
57 | 111 | .B ssh backends | 111 | .B ssh backends |
58 | @@ -1067,87 +1067,169 @@ | |||
59 | 1067 | or preceded by a double slash, '//path', to represent an absolute | 1067 | or preceded by a double slash, '//path', to represent an absolute |
60 | 1068 | filesystem path. | 1068 | filesystem path. |
61 | 1069 | .PP | 1069 | .PP |
62 | 1070 | .B Note: | ||
63 | 1071 | .RS | ||
64 | 1072 | Scheme (protocol) access may be provided by more than one backend. | ||
65 | 1073 | In case the default backend is buggy or simply not working in a specific case it might be worth trying an alternative implementation. | ||
66 | 1074 | Alternative backends can be selected by prefixing the scheme with the name of the alternative backend e.g. | ||
67 | 1075 | .B ncftp+ftp:// | ||
68 | 1076 | and are mentioned below the scheme's syntax summary. | ||
69 | 1077 | .RE | ||
70 | 1078 | |||
71 | 1079 | .PP | ||
72 | 1070 | Formats of each of the URL schemes follow: | 1080 | Formats of each of the URL schemes follow: |
73 | 1081 | |||
74 | 1082 | .PP | ||
75 | 1083 | .BR "Cloud Files" " (Rackspace)" | ||
76 | 1084 | .PP | ||
77 | 1071 | .RS | 1085 | .RS |
78 | 1072 | .PP | ||
79 | 1073 | .BI "Rackspace Cloud Files" | ||
80 | 1074 | .br | ||
81 | 1075 | cf+http://container_name | 1086 | cf+http://container_name |
83 | 1076 | .br | 1087 | .PP |
84 | 1077 | See also | 1088 | See also |
85 | 1078 | .B "A NOTE ON CLOUD FILES ACCESS" | 1089 | .B "A NOTE ON CLOUD FILES ACCESS" |
89 | 1079 | .PP | 1090 | .RE |
90 | 1080 | .BI Dropbox | 1091 | .PP |
91 | 1081 | .br | 1092 | .B "Copy cloud storage" |
92 | 1093 | .PP | ||
93 | 1094 | .RS | ||
94 | 1095 | copy://user[:password]@copy.com/some_dir | ||
95 | 1096 | .RE | ||
96 | 1097 | .PP | ||
97 | 1098 | .B Dropbox | ||
98 | 1099 | .PP | ||
99 | 1100 | .RS | ||
100 | 1082 | dpbx:///some_dir | 1101 | dpbx:///some_dir |
102 | 1083 | .br | 1102 | .PP |
103 | 1084 | Make sure to read | 1103 | Make sure to read |
104 | 1085 | .BR "A NOTE ON DROPBOX ACCESS" " first!" | 1104 | .BR "A NOTE ON DROPBOX ACCESS" " first!" |
109 | 1086 | .PP | 1105 | .RE |
110 | 1087 | copy://user[:password]@copy.com/some_dir | 1106 | .PP |
111 | 1088 | .PP | 1107 | .B "Local file path" |
112 | 1089 | .PP | 1108 | .PP |
113 | 1109 | .RS | ||
114 | 1090 | file://[relative|/absolute]/local/path | 1110 | file://[relative|/absolute]/local/path |
116 | 1091 | .PP | 1111 | .RE |
117 | 1112 | .PP | ||
118 | 1113 | .BR "FISH" " (Files transferred over Shell protocol) over ssh" | ||
119 | 1114 | .PP | ||
120 | 1115 | .RS | ||
121 | 1116 | fish://user[:password]@other.host[:port]/[relative|/absolute]_path | ||
122 | 1117 | .RE | ||
123 | 1118 | .PP | ||
124 | 1119 | .B "FTP" | ||
125 | 1120 | .PP | ||
126 | 1121 | .RS | ||
127 | 1092 | ftp[s]://user[:password]@other.host[:port]/some_dir | 1122 | ftp[s]://user[:password]@other.host[:port]/some_dir |
128 | 1093 | .PP | 1123 | .PP |
129 | 1124 | .B NOTE: | ||
130 | 1125 | use lftp+, ncftp+ prefixes to enforce a specific backend, e.g. ncftp+ftp://... | ||
131 | 1126 | .RE | ||
132 | 1127 | .PP | ||
133 | 1128 | .B "Google Docs" | ||
134 | 1129 | .PP | ||
135 | 1130 | .RS | ||
136 | 1094 | gdocs://user[:password]@other.host/some_dir | 1131 | gdocs://user[:password]@other.host/some_dir |
140 | 1095 | .PP | 1132 | .RE |
141 | 1096 | .BI "Google Cloud Storage" | 1133 | .PP |
142 | 1097 | .br | 1134 | .B "Google Cloud Storage" |
143 | 1135 | .PP | ||
144 | 1136 | .RS | ||
145 | 1098 | gs://bucket[/prefix] | 1137 | gs://bucket[/prefix] |
147 | 1099 | .PP | 1138 | .RE |
148 | 1139 | .PP | ||
149 | 1140 | .B "HSI" | ||
150 | 1141 | .PP | ||
151 | 1142 | .RS | ||
152 | 1100 | hsi://user[:password]@other.host/some_dir | 1143 | hsi://user[:password]@other.host/some_dir |
154 | 1101 | .PP | 1144 | .RE |
155 | 1145 | .PP | ||
156 | 1146 | .B "IMAP email storage" | ||
157 | 1147 | .PP | ||
158 | 1148 | .RS | ||
159 | 1102 | imap[s]://user[:password]@host.com[/from_address_prefix] | 1149 | imap[s]://user[:password]@host.com[/from_address_prefix] |
161 | 1103 | .br | 1150 | .PP |
162 | 1104 | See also | 1151 | See also |
163 | 1105 | .B "A NOTE ON IMAP" | 1152 | .B "A NOTE ON IMAP" |
165 | 1106 | .PP | 1153 | .RE |
166 | 1154 | .PP | ||
167 | 1155 | .B "Mega cloud storage" | ||
168 | 1156 | .PP | ||
169 | 1157 | .RS | ||
170 | 1107 | mega://user[:password]@mega.co.nz/some_dir | 1158 | mega://user[:password]@mega.co.nz/some_dir |
174 | 1108 | .PP | 1159 | .RE |
175 | 1109 | .BI "Par2 Wrapper Backend" | 1160 | .PP |
176 | 1110 | .br | 1161 | .B "Par2 Wrapper Backend" |
177 | 1162 | .PP | ||
178 | 1163 | .RS | ||
179 | 1111 | par2+scheme://[user[:password]@]host[:port]/[/]path | 1164 | par2+scheme://[user[:password]@]host[:port]/[/]path |
181 | 1112 | .br | 1165 | .PP |
182 | 1113 | See also | 1166 | See also |
183 | 1114 | .B "A NOTE ON PAR2 WRAPPER BACKEND" | 1167 | .B "A NOTE ON PAR2 WRAPPER BACKEND" |
187 | 1115 | .PP | 1168 | .RE |
188 | 1116 | .B "using rsync daemon" | 1169 | .PP |
189 | 1117 | .br | 1170 | .B "Rsync via daemon" |
190 | 1171 | .PP | ||
191 | 1172 | .RS | ||
192 | 1118 | rsync://user[:password]@host.com[:port]::[/]module/some_dir | 1173 | rsync://user[:password]@host.com[:port]::[/]module/some_dir |
196 | 1119 | .br | 1174 | .PP |
197 | 1120 | .B "using rsync over ssh (only key auth)" | 1175 | .RE |
198 | 1121 | .br | 1176 | .B "Rsync over ssh (only key auth)" |
199 | 1177 | .PP | ||
200 | 1178 | .RS | ||
201 | 1122 | rsync://user@host.com[:port]/[relative|/absolute]_path | 1179 | rsync://user@host.com[:port]/[relative|/absolute]_path |
203 | 1123 | .PP | 1180 | .RE |
204 | 1181 | .PP | ||
205 | 1182 | .BR "S3 storage" " (Amazon)" | ||
206 | 1183 | .PP | ||
207 | 1184 | .RS | ||
208 | 1124 | s3://host/bucket_name[/prefix] | 1185 | s3://host/bucket_name[/prefix] |
209 | 1125 | .br | 1186 | .br |
210 | 1126 | s3+http://bucket_name[/prefix] | 1187 | s3+http://bucket_name[/prefix] |
212 | 1127 | .br | 1188 | .PP |
213 | 1128 | See also | 1189 | See also |
214 | 1129 | .B "A NOTE ON EUROPEAN S3 BUCKETS" | 1190 | .B "A NOTE ON EUROPEAN S3 BUCKETS" |
219 | 1130 | .PP | 1191 | .RE |
220 | 1131 | scp://.. or ssh://.. are synonymous with | 1192 | .PP |
221 | 1132 | .br | 1193 | .B "SCP/SFTP access" |
222 | 1133 | sftp://user[:password]@other.host[:port]/[/]some_dir | 1194 | .PP |
223 | 1195 | .RS | ||
224 | 1196 | scp://.. or | ||
225 | 1197 | .br | ||
226 | 1198 | sftp://user[:password]@other.host[:port]/[relative|/absolute]_path | ||
227 | 1199 | .PP | ||
228 | 1200 | .BR "defaults" " are paramiko+scp:// and paramiko+sftp://" | ||
229 | 1201 | .br | ||
230 | 1202 | .BR "alternatively" " try pexpect+scp://, pexpect+sftp://, lftp+sftp://" | ||
231 | 1134 | .br | 1203 | .br |
232 | 1135 | See also | 1204 | See also |
233 | 1136 | .BR --ssh-backend , | ||
234 | 1137 | .BR --ssh-askpass , | 1205 | .BR --ssh-askpass , |
235 | 1138 | .BR --use-scp , | ||
236 | 1139 | .B --ssh-options | 1206 | .B --ssh-options |
237 | 1140 | and | 1207 | and |
238 | 1141 | .BR "A NOTE ON SSH BACKENDS" . | 1208 | .BR "A NOTE ON SSH BACKENDS" . |
240 | 1142 | .PP | 1209 | .RE |
241 | 1210 | .PP | ||
242 | 1211 | .BR "Swift" " (Openstack)" | ||
243 | 1212 | .PP | ||
244 | 1213 | .RS | ||
245 | 1143 | swift://container_name | 1214 | swift://container_name |
247 | 1144 | .br | 1215 | .PP |
248 | 1145 | See also | 1216 | See also |
249 | 1146 | .B "A NOTE ON SWIFT (OPENSTACK OBJECT STORAGE) ACCESS" | 1217 | .B "A NOTE ON SWIFT (OPENSTACK OBJECT STORAGE) ACCESS" |
251 | 1147 | .PP | 1218 | .RE |
252 | 1219 | .PP | ||
253 | 1220 | .B "Tahoe-LAFS" | ||
254 | 1221 | .PP | ||
255 | 1222 | .RS | ||
256 | 1148 | tahoe://alias/directory | 1223 | tahoe://alias/directory |
258 | 1149 | .PP | 1224 | .RE |
259 | 1225 | .PP | ||
260 | 1226 | .B "WebDAV" | ||
261 | 1227 | .PP | ||
262 | 1228 | .RS | ||
263 | 1150 | webdav[s]://user[:password]@other.host[:port]/some_dir | 1229 | webdav[s]://user[:password]@other.host[:port]/some_dir |
264 | 1230 | .PP | ||
265 | 1231 | .B alternatively | ||
266 | 1232 | try lftp+webdav[s]:// | ||
267 | 1151 | .RE | 1233 | .RE |
268 | 1152 | 1234 | ||
269 | 1153 | .SH TIME FORMATS | 1235 | .SH TIME FORMATS |
270 | 1154 | 1236 | ||
271 | === modified file 'duplicity/backend.py' | |||
272 | --- duplicity/backend.py 2014-10-27 02:27:36 +0000 | |||
273 | +++ duplicity/backend.py 2014-10-30 18:01:04 +0000 | |||
274 | @@ -32,6 +32,7 @@ | |||
275 | 32 | import re | 32 | import re |
276 | 33 | import getpass | 33 | import getpass |
277 | 34 | import gettext | 34 | import gettext |
278 | 35 | import re | ||
279 | 35 | import types | 36 | import types |
280 | 36 | import urllib | 37 | import urllib |
281 | 37 | import urlparse | 38 | import urlparse |
282 | @@ -164,6 +165,11 @@ | |||
283 | 164 | 165 | ||
284 | 165 | _backend_prefixes[scheme] = backend_factory | 166 | _backend_prefixes[scheme] = backend_factory |
285 | 166 | 167 | ||
286 | 168 | def strip_prefix(url_string, prefix_scheme): | ||
287 | 169 | """ | ||
288 | 170 | strip the prefix from a string e.g. par2+ftp://... -> ftp://... | ||
289 | 171 | """ | ||
290 | 172 | return re.sub('(?i)^'+re.escape(prefix_scheme)+'\+','',url_string) | ||
291 | 167 | 173 | ||
292 | 168 | def is_backend_url(url_string): | 174 | def is_backend_url(url_string): |
293 | 169 | """ | 175 | """ |
294 | @@ -198,7 +204,7 @@ | |||
295 | 198 | for prefix in _backend_prefixes: | 204 | for prefix in _backend_prefixes: |
296 | 199 | if url_string.startswith(prefix + '+'): | 205 | if url_string.startswith(prefix + '+'): |
297 | 200 | factory = _backend_prefixes[prefix] | 206 | factory = _backend_prefixes[prefix] |
299 | 201 | pu = ParsedUrl(url_string.lstrip(prefix + '+')) | 207 | pu = ParsedUrl(strip_prefix(url_string,prefix)) |
300 | 202 | break | 208 | break |
301 | 203 | 209 | ||
302 | 204 | if factory is None: | 210 | if factory is None: |
303 | @@ -337,11 +343,8 @@ | |||
304 | 337 | def strip_auth_from_url(parsed_url): | 343 | def strip_auth_from_url(parsed_url): |
305 | 338 | """Return a URL from a urlparse object without a username or password.""" | 344 | """Return a URL from a urlparse object without a username or password.""" |
306 | 339 | 345 | ||
312 | 340 | # Get a copy of the network location without the username or password. | 346 | clean_url = re.sub('^([^:/]+://)(.*@)?(.*)',r'\1\3',parsed_url.geturl()) |
313 | 341 | straight_netloc = parsed_url.netloc.split('@')[-1] | 347 | return clean_url |
309 | 342 | |||
310 | 343 | # Replace the full network location with the stripped copy. | ||
311 | 344 | return parsed_url.geturl().replace(parsed_url.netloc, straight_netloc, 1) | ||
314 | 345 | 348 | ||
315 | 346 | def _get_code_from_exception(backend, operation, e): | 349 | def _get_code_from_exception(backend, operation, e): |
316 | 347 | if isinstance(e, BackendException) and e.code != log.ErrorCode.backend_error: | 350 | if isinstance(e, BackendException) and e.code != log.ErrorCode.backend_error: |
317 | 348 | 351 | ||
318 | === renamed file 'duplicity/backends/ftpbackend.py' => 'duplicity/backends/lftpbackend.py' | |||
319 | --- duplicity/backends/ftpbackend.py 2014-10-01 20:35:16 +0000 | |||
320 | +++ duplicity/backends/lftpbackend.py 2014-10-30 18:01:04 +0000 | |||
321 | @@ -3,7 +3,10 @@ | |||
322 | 3 | # Copyright 2002 Ben Escoto <ben@emerose.org> | 3 | # Copyright 2002 Ben Escoto <ben@emerose.org> |
323 | 4 | # Copyright 2007 Kenneth Loafman <kenneth@loafman.com> | 4 | # Copyright 2007 Kenneth Loafman <kenneth@loafman.com> |
324 | 5 | # Copyright 2010 Marcel Pennewiss <opensource@pennewiss.de> | 5 | # Copyright 2010 Marcel Pennewiss <opensource@pennewiss.de> |
326 | 6 | # Copyright 2014 Moritz Maisel <moritz@maisel.name> | 6 | # Copyright 2014 Edgar Soldin |
327 | 7 | # - webdav, fish, sftp support | ||
328 | 8 | # - https cert verification switches | ||
329 | 9 | # - debug output | ||
330 | 7 | # | 10 | # |
331 | 8 | # This file is part of duplicity. | 11 | # This file is part of duplicity. |
332 | 9 | # | 12 | # |
333 | @@ -23,15 +26,15 @@ | |||
334 | 23 | 26 | ||
335 | 24 | import os | 27 | import os |
336 | 25 | import os.path | 28 | import os.path |
337 | 29 | import re | ||
338 | 26 | import urllib | 30 | import urllib |
339 | 27 | import re | ||
340 | 28 | 31 | ||
341 | 29 | import duplicity.backend | 32 | import duplicity.backend |
342 | 30 | from duplicity import globals | 33 | from duplicity import globals |
343 | 31 | from duplicity import log | 34 | from duplicity import log |
344 | 32 | from duplicity import tempdir | 35 | from duplicity import tempdir |
345 | 33 | 36 | ||
347 | 34 | class FTPBackend(duplicity.backend.Backend): | 37 | class LFTPBackend(duplicity.backend.Backend): |
348 | 35 | """Connect to remote store using File Transfer Protocol""" | 38 | """Connect to remote store using File Transfer Protocol""" |
349 | 36 | def __init__(self, parsed_url): | 39 | def __init__(self, parsed_url): |
350 | 37 | duplicity.backend.Backend.__init__(self, parsed_url) | 40 | duplicity.backend.Backend.__init__(self, parsed_url) |
351 | @@ -54,61 +57,141 @@ | |||
352 | 54 | 57 | ||
353 | 55 | self.parsed_url = parsed_url | 58 | self.parsed_url = parsed_url |
354 | 56 | 59 | ||
356 | 57 | self.url_string = duplicity.backend.strip_auth_from_url(self.parsed_url) | 60 | # self.url_string = duplicity.backend.strip_auth_from_url(self.parsed_url) |
357 | 61 | # # strip lftp+ prefix | ||
358 | 62 | # self.url_string = duplicity.backend.strip_prefix(self.url_string, 'lftp') | ||
359 | 63 | |||
360 | 64 | self.scheme = duplicity.backend.strip_prefix( parsed_url.scheme, 'lftp' ).lower() | ||
361 | 65 | self.scheme = re.sub('^webdav','http',self.scheme) | ||
362 | 66 | self.url_string = self.scheme + '://' + parsed_url.hostname | ||
363 | 67 | if parsed_url.port : | ||
364 | 68 | self.url_string += ":%s" % parsed_url.port | ||
365 | 69 | |||
366 | 70 | self.remote_path = re.sub('^/','',parsed_url.path) | ||
367 | 58 | 71 | ||
368 | 59 | # Use an explicit directory name. | 72 | # Use an explicit directory name. |
371 | 60 | if self.url_string[-1] != '/': | 73 | if self.remote_path[-1] != '/': |
372 | 61 | self.url_string += '/' | 74 | self.remote_path += '/' |
373 | 62 | 75 | ||
375 | 63 | self.password = self.get_password() | 76 | self.authflag = '' |
376 | 77 | if self.parsed_url.username: | ||
377 | 78 | self.username = self.parsed_url.username | ||
378 | 79 | self.password = self.get_password() | ||
379 | 80 | self.authflag = "-u '%s,%s'" % (self.username,self.password) | ||
380 | 64 | 81 | ||
381 | 65 | if globals.ftp_connection == 'regular': | 82 | if globals.ftp_connection == 'regular': |
382 | 66 | self.conn_opt = 'off' | 83 | self.conn_opt = 'off' |
383 | 67 | else: | 84 | else: |
384 | 68 | self.conn_opt = 'on' | 85 | self.conn_opt = 'on' |
385 | 69 | 86 | ||
390 | 70 | if parsed_url.port != None and parsed_url.port != 21: | 87 | # check for cacert file if https |
391 | 71 | self.portflag = " -p '%s'" % (parsed_url.port) | 88 | self.cacert_file = globals.ssl_cacert_file |
392 | 72 | else: | 89 | if self.scheme == 'https' and not globals.ssl_no_check_certificate: |
393 | 73 | self.portflag = "" | 90 | cacert_candidates = [ "~/.duplicity/cacert.pem", \ |
394 | 91 | "~/duplicity_cacert.pem", \ | ||
395 | 92 | "/etc/duplicity/cacert.pem" ] | ||
396 | 93 | # | ||
397 | 94 | if not self.cacert_file: | ||
398 | 95 | for path in cacert_candidates : | ||
399 | 96 | path = os.path.expanduser(path) | ||
400 | 97 | if (os.path.isfile(path)): | ||
401 | 98 | self.cacert_file = path | ||
402 | 99 | break | ||
403 | 100 | # still no cacert file, inform user | ||
404 | 101 | if not self.cacert_file: | ||
405 | 102 | raise duplicity.errors.FatalBackendException("""For certificate verification a cacert database file is needed in one of these locations: %s | ||
406 | 103 | Hints: | ||
407 | 104 | Consult the man page, chapter 'SSL Certificate Verification'. | ||
408 | 105 | Consider using the options --ssl-cacert-file, --ssl-no-check-certificate .""" % ", ".join(cacert_candidates) ) | ||
409 | 74 | 106 | ||
410 | 75 | self.tempfile, self.tempname = tempdir.default().mkstemp() | 107 | self.tempfile, self.tempname = tempdir.default().mkstemp() |
411 | 108 | os.write(self.tempfile, "set ssl:verify-certificate " + ( "false" if globals.ssl_no_check_certificate else "true" ) + "\n") | ||
412 | 109 | if globals.ssl_cacert_file : | ||
413 | 110 | os.write(self.tempfile, "set ssl:ca-file '" + globals.ssl_cacert_file + "'\n") | ||
414 | 76 | os.write(self.tempfile, "set ftp:ssl-allow true\n") | 111 | os.write(self.tempfile, "set ftp:ssl-allow true\n") |
415 | 77 | os.write(self.tempfile, "set ftp:ssl-protect-data true\n") | 112 | os.write(self.tempfile, "set ftp:ssl-protect-data true\n") |
416 | 78 | os.write(self.tempfile, "set ftp:ssl-protect-list true\n") | 113 | os.write(self.tempfile, "set ftp:ssl-protect-list true\n") |
417 | 114 | os.write(self.tempfile, "set http:use-propfind true\n") | ||
418 | 79 | os.write(self.tempfile, "set net:timeout %s\n" % globals.timeout) | 115 | os.write(self.tempfile, "set net:timeout %s\n" % globals.timeout) |
419 | 80 | os.write(self.tempfile, "set net:max-retries %s\n" % globals.num_retries) | 116 | os.write(self.tempfile, "set net:max-retries %s\n" % globals.num_retries) |
420 | 81 | os.write(self.tempfile, "set ftp:passive-mode %s\n" % self.conn_opt) | 117 | os.write(self.tempfile, "set ftp:passive-mode %s\n" % self.conn_opt) |
422 | 82 | os.write(self.tempfile, "open %s %s\n" % (self.portflag, self.parsed_url.hostname)) | 118 | if log.getverbosity() >= log.DEBUG : |
423 | 119 | os.write(self.tempfile, "debug\n") | ||
424 | 120 | os.write(self.tempfile, "open %s %s\n" % (self.authflag, self.url_string) ) | ||
425 | 121 | # os.write(self.tempfile, "open %s %s\n" % (self.portflag, self.parsed_url.hostname)) | ||
426 | 83 | # allow .netrc auth by only setting user/pass when user was actually given | 122 | # allow .netrc auth by only setting user/pass when user was actually given |
429 | 84 | if self.parsed_url.username: | 123 | # if self.parsed_url.username: |
430 | 85 | os.write(self.tempfile, "user %s %s\n" % (self.parsed_url.username, self.password)) | 124 | # os.write(self.tempfile, "user %s %s\n" % (self.parsed_url.username, self.password)) |
431 | 86 | os.close(self.tempfile) | 125 | os.close(self.tempfile) |
432 | 126 | if log.getverbosity() >= log.DEBUG : | ||
433 | 127 | f = open(self.tempname, 'r') | ||
434 | 128 | log.Debug("SETTINGS: \n" | ||
435 | 129 | "%s" % f.readlines() ) | ||
436 | 87 | 130 | ||
437 | 88 | def _put(self, source_path, remote_filename): | 131 | def _put(self, source_path, remote_filename): |
442 | 89 | remote_path = os.path.join(urllib.unquote(self.parsed_url.path.lstrip('/')), remote_filename).rstrip() | 132 | #remote_path = os.path.join(urllib.unquote(self.parsed_url.path.lstrip('/')), remote_filename).rstrip() |
443 | 90 | commandline = "lftp -c 'source %s;put \'%s\' -o \'%s\''" % \ | 133 | commandline = "lftp -c 'source \'%s\'; mkdir -p %s; put \'%s\' -o \'%s\''" % \ |
444 | 91 | (self.tempname, source_path.name, remote_path) | 134 | (self.tempname, self.remote_path, source_path.name, self.remote_path + remote_filename) |
445 | 92 | self.subprocess_popen(commandline) | 135 | log.Debug("CMD: %s" % commandline) |
446 | 136 | s, l, e = self.subprocess_popen(commandline) | ||
447 | 137 | log.Debug("STATUS: %s" % s) | ||
448 | 138 | log.Debug("STDERR:\n" | ||
449 | 139 | "%s" % (e)) | ||
450 | 140 | log.Debug("STDOUT:\n" | ||
451 | 141 | "%s" % (l)) | ||
452 | 93 | 142 | ||
453 | 94 | def _get(self, remote_filename, local_path): | 143 | def _get(self, remote_filename, local_path): |
458 | 95 | remote_path = os.path.join(urllib.unquote(self.parsed_url.path), remote_filename).rstrip() | 144 | #remote_path = os.path.join(urllib.unquote(self.parsed_url.path), remote_filename).rstrip() |
459 | 96 | commandline = "lftp -c 'source %s;get %s -o %s'" % \ | 145 | commandline = "lftp -c 'source \'%s\'; get \'%s\' -o \'%s\''" % \ |
460 | 97 | (self.tempname, remote_path.lstrip('/'), local_path.name) | 146 | (self.tempname, self.remote_path+remote_filename, local_path.name) |
461 | 98 | self.subprocess_popen(commandline) | 147 | log.Debug("CMD: %s" % commandline) |
462 | 148 | _, l, e = self.subprocess_popen(commandline) | ||
463 | 149 | log.Debug("STDERR:\n" | ||
464 | 150 | "%s" % (e)) | ||
465 | 151 | log.Debug("STDOUT:\n" | ||
466 | 152 | "%s" % (l)) | ||
467 | 99 | 153 | ||
468 | 100 | def _list(self): | 154 | def _list(self): |
469 | 101 | # Do a long listing to avoid connection reset | 155 | # Do a long listing to avoid connection reset |
473 | 102 | remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip() | 156 | #remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip() |
474 | 103 | commandline = "lftp -c 'source %s;ls \'%s\''" % (self.tempname, remote_dir) | 157 | remote_dir = urllib.unquote(self.parsed_url.path) |
475 | 104 | _, l, _ = self.subprocess_popen(commandline) | 158 | #print remote_dir |
476 | 159 | commandline = "lftp -c 'source \'%s\'; cd \'%s\' || exit 0; ls'" % (self.tempname, self.remote_path) | ||
477 | 160 | log.Debug("CMD: %s" % commandline) | ||
478 | 161 | _, l, e = self.subprocess_popen(commandline) | ||
479 | 162 | log.Debug("STDERR:\n" | ||
480 | 163 | "%s" % (e)) | ||
481 | 164 | log.Debug("STDOUT:\n" | ||
482 | 165 | "%s" % (l)) | ||
483 | 166 | |||
484 | 105 | # Look for our files as the last element of a long list line | 167 | # Look for our files as the last element of a long list line |
485 | 106 | return [x.split()[-1] for x in l.split('\n') if x] | 168 | return [x.split()[-1] for x in l.split('\n') if x] |
486 | 107 | 169 | ||
487 | 108 | def _delete(self, filename): | 170 | def _delete(self, filename): |
494 | 109 | remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip() | 171 | #remote_dir = urllib.unquote(self.parsed_url.path.lstrip('/')).rstrip() |
495 | 110 | commandline = "lftp -c 'source %s;cd \'%s\';rm \'%s\''" % (self.tempname, remote_dir, filename) | 172 | commandline = "lftp -c 'source \'%s\'; cd \'%s\'; rm \'%s\''" % (self.tempname, self.remote_path, filename) |
496 | 111 | self.subprocess_popen(commandline) | 173 | log.Debug("CMD: %s" % commandline) |
497 | 112 | 174 | _, l, e = self.subprocess_popen(commandline) | |
498 | 113 | duplicity.backend.register_backend("ftp", FTPBackend) | 175 | log.Debug("STDERR:\n" |
499 | 114 | duplicity.backend.register_backend("ftps", FTPBackend) | 176 | "%s" % (e)) |
500 | 177 | log.Debug("STDOUT:\n" | ||
501 | 178 | "%s" % (l)) | ||
502 | 179 | |||
503 | 180 | duplicity.backend.register_backend("ftp", LFTPBackend) | ||
504 | 181 | duplicity.backend.register_backend("ftps", LFTPBackend) | ||
505 | 182 | duplicity.backend.register_backend("fish", LFTPBackend) | ||
506 | 183 | |||
507 | 184 | duplicity.backend.register_backend("lftp+ftp", LFTPBackend) | ||
508 | 185 | duplicity.backend.register_backend("lftp+ftps", LFTPBackend) | ||
509 | 186 | duplicity.backend.register_backend("lftp+fish", LFTPBackend) | ||
510 | 187 | duplicity.backend.register_backend("lftp+sftp", LFTPBackend) | ||
511 | 188 | duplicity.backend.register_backend("lftp+webdav", LFTPBackend) | ||
512 | 189 | duplicity.backend.register_backend("lftp+webdavs", LFTPBackend) | ||
513 | 190 | duplicity.backend.register_backend("lftp+http", LFTPBackend) | ||
514 | 191 | duplicity.backend.register_backend("lftp+https", LFTPBackend) | ||
515 | 192 | |||
516 | 193 | duplicity.backend.uses_netloc.extend([ 'ftp', 'ftps', 'fish',\ | ||
517 | 194 | 'lftp+ftp', 'lftp+ftps',\ | ||
518 | 195 | 'lftp+fish', 'lftp+sftp',\ | ||
519 | 196 | 'lftp+webdav', 'lftp+webdavs',\ | ||
520 | 197 | 'lftp+http', 'lftp+https' ]) | ||
521 | 115 | \ No newline at end of file | 198 | \ No newline at end of file |
522 | 116 | 199 | ||
523 | === added file 'duplicity/backends/ncftpbackend.py' | |||
524 | --- duplicity/backends/ncftpbackend.py 1970-01-01 00:00:00 +0000 | |||
525 | +++ duplicity/backends/ncftpbackend.py 2014-10-30 18:01:04 +0000 | |||
526 | @@ -0,0 +1,118 @@ | |||
527 | 1 | # -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*- | ||
528 | 2 | # | ||
529 | 3 | # Copyright 2002 Ben Escoto <ben@emerose.org> | ||
530 | 4 | # Copyright 2007 Kenneth Loafman <kenneth@loafman.com> | ||
531 | 5 | # | ||
532 | 6 | # This file is part of duplicity. | ||
533 | 7 | # | ||
534 | 8 | # Duplicity is free software; you can redistribute it and/or modify it | ||
535 | 9 | # under the terms of the GNU General Public License as published by the | ||
536 | 10 | # Free Software Foundation; either version 2 of the License, or (at your | ||
537 | 11 | # option) any later version. | ||
538 | 12 | # | ||
539 | 13 | # Duplicity is distributed in the hope that it will be useful, but | ||
540 | 14 | # WITHOUT ANY WARRANTY; without even the implied warranty of | ||
541 | 15 | # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU | ||
542 | 16 | # General Public License for more details. | ||
543 | 17 | # | ||
544 | 18 | # You should have received a copy of the GNU General Public License | ||
545 | 19 | # along with duplicity; if not, write to the Free Software Foundation, | ||
546 | 20 | # Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA | ||
547 | 21 | |||
548 | 22 | import os.path | ||
549 | 23 | import urllib | ||
550 | 24 | |||
551 | 25 | import duplicity.backend | ||
552 | 26 | from duplicity import globals | ||
553 | 27 | from duplicity import log | ||
554 | 28 | from duplicity import tempdir | ||
555 | 29 | |||
556 | 30 | class NCFTPBackend(duplicity.backend.Backend): | ||
557 | 31 | """Connect to remote store using File Transfer Protocol""" | ||
558 | 32 | def __init__(self, parsed_url): | ||
559 | 33 | duplicity.backend.Backend.__init__(self, parsed_url) | ||
560 | 34 | |||
561 | 35 | # we expect an error return, so go low-level and ignore it | ||
562 | 36 | try: | ||
563 | 37 | p = os.popen("ncftpls -v") | ||
564 | 38 | fout = p.read() | ||
565 | 39 | ret = p.close() | ||
566 | 40 | except Exception: | ||
567 | 41 | pass | ||
568 | 42 | # the expected error is 8 in the high-byte and some output | ||
569 | 43 | if ret != 0x0800 or not fout: | ||
570 | 44 | log.FatalError("NcFTP not found: Please install NcFTP version 3.1.9 or later", | ||
571 | 45 | log.ErrorCode.ftp_ncftp_missing) | ||
572 | 46 | |||
573 | 47 | # version is the second word of the first line | ||
574 | 48 | version = fout.split('\n')[0].split()[1] | ||
575 | 49 | if version < "3.1.9": | ||
576 | 50 | log.FatalError("NcFTP too old: Duplicity requires NcFTP version 3.1.9," | ||
577 | 51 | "3.2.1 or later. Version 3.2.0 will not work properly.", | ||
578 | 52 | log.ErrorCode.ftp_ncftp_too_old) | ||
579 | 53 | elif version == "3.2.0": | ||
580 | 54 | log.Warn("NcFTP (ncftpput) version 3.2.0 may fail with duplicity.\n" | ||
581 | 55 | "see: http://www.ncftpd.com/ncftp/doc/changelog.html\n" | ||
582 | 56 | "If you have trouble, please upgrade to 3.2.1 or later", | ||
583 | 57 | log.WarningCode.ftp_ncftp_v320) | ||
584 | 58 | log.Notice("NcFTP version is %s" % version) | ||
585 | 59 | |||
586 | 60 | self.parsed_url = parsed_url | ||
587 | 61 | |||
588 | 62 | self.url_string = duplicity.backend.strip_auth_from_url(self.parsed_url) | ||
589 | 63 | |||
590 | 64 | # strip ncftp+ prefix | ||
591 | 65 | self.url_string = duplicity.backend.strip_prefix(self.url_string, 'ncftp') | ||
592 | 66 | |||
593 | 67 | # This squelches the "file not found" result from ncftpls when | ||
594 | 68 | # the ftp backend looks for a collection that does not exist. | ||
595 | 69 | # version 3.2.2 has error code 5, 1280 is some legacy value | ||
596 | 70 | self.popen_breaks[ 'ncftpls' ] = [ 5, 1280 ] | ||
597 | 71 | |||
598 | 72 | # Use an explicit directory name. | ||
599 | 73 | if self.url_string[-1] != '/': | ||
600 | 74 | self.url_string += '/' | ||
601 | 75 | |||
602 | 76 | self.password = self.get_password() | ||
603 | 77 | |||
604 | 78 | if globals.ftp_connection == 'regular': | ||
605 | 79 | self.conn_opt = '-E' | ||
606 | 80 | else: | ||
607 | 81 | self.conn_opt = '-F' | ||
608 | 82 | |||
609 | 83 | self.tempfile, self.tempname = tempdir.default().mkstemp() | ||
610 | 84 | os.write(self.tempfile, "host %s\n" % self.parsed_url.hostname) | ||
611 | 85 | os.write(self.tempfile, "user %s\n" % self.parsed_url.username) | ||
612 | 86 | os.write(self.tempfile, "pass %s\n" % self.password) | ||
613 | 87 | os.close(self.tempfile) | ||
614 | 88 | self.flags = "-f %s %s -t %s -o useCLNT=0,useHELP_SITE=0 " % \ | ||
615 | 89 | (self.tempname, self.conn_opt, globals.timeout) | ||
616 | 90 | if parsed_url.port != None and parsed_url.port != 21: | ||
617 | 91 | self.flags += " -P '%s'" % (parsed_url.port) | ||
618 | 92 | |||
619 | 93 | def _put(self, source_path, remote_filename): | ||
620 | 94 | remote_path = os.path.join(urllib.unquote(self.parsed_url.path.lstrip('/')), remote_filename).rstrip() | ||
621 | 95 | commandline = "ncftpput %s -m -V -C '%s' '%s'" % \ | ||
622 | 96 | (self.flags, source_path.name, remote_path) | ||
623 | 97 | self.subprocess_popen(commandline) | ||
624 | 98 | |||
625 | 99 | def _get(self, remote_filename, local_path): | ||
626 | 100 | remote_path = os.path.join(urllib.unquote(self.parsed_url.path), remote_filename).rstrip() | ||
627 | 101 | commandline = "ncftpget %s -V -C '%s' '%s' '%s'" % \ | ||
628 | 102 | (self.flags, self.parsed_url.hostname, remote_path.lstrip('/'), local_path.name) | ||
629 | 103 | self.subprocess_popen(commandline) | ||
630 | 104 | |||
631 | 105 | def _list(self): | ||
632 | 106 | # Do a long listing to avoid connection reset | ||
633 | 107 | commandline = "ncftpls %s -l '%s'" % (self.flags, self.url_string) | ||
634 | 108 | _, l, _ = self.subprocess_popen(commandline) | ||
635 | 109 | # Look for our files as the last element of a long list line | ||
636 | 110 | return [x.split()[-1] for x in l.split('\n') if x and not x.startswith("total ")] | ||
637 | 111 | |||
638 | 112 | def _delete(self, filename): | ||
639 | 113 | commandline = "ncftpls %s -l -X 'DELE %s' '%s'" % \ | ||
640 | 114 | (self.flags, filename, self.url_string) | ||
641 | 115 | self.subprocess_popen(commandline) | ||
642 | 116 | |||
643 | 117 | duplicity.backend.register_backend("ncftp+ftp", NCFTPBackend) | ||
644 | 118 | duplicity.backend.uses_netloc.extend([ 'ncftp+ftp' ]) | ||
645 | 0 | \ No newline at end of file | 119 | \ No newline at end of file |
646 | 1 | 120 | ||
647 | === renamed file 'duplicity/backends/_ssh_paramiko.py' => 'duplicity/backends/ssh_paramiko_backend.py' | |||
648 | --- duplicity/backends/_ssh_paramiko.py 2014-10-27 03:02:20 +0000 | |||
649 | +++ duplicity/backends/ssh_paramiko_backend.py 2014-10-30 18:01:04 +0000 | |||
650 | @@ -217,8 +217,11 @@ | |||
651 | 217 | self.config['port'],e)) | 217 | self.config['port'],e)) |
652 | 218 | self.client.get_transport().set_keepalive((int)(globals.timeout / 2)) | 218 | self.client.get_transport().set_keepalive((int)(globals.timeout / 2)) |
653 | 219 | 219 | ||
654 | 220 | self.scheme = duplicity.backend.strip_prefix(parsed_url.scheme, 'paramiko') | ||
655 | 221 | self.use_scp = ( self.scheme == 'scp' ) | ||
656 | 222 | |||
657 | 220 | # scp or sftp? | 223 | # scp or sftp? |
659 | 221 | if (globals.use_scp): | 224 | if (self.use_scp): |
660 | 222 | # sanity-check the directory name | 225 | # sanity-check the directory name |
661 | 223 | if (re.search("'",self.remote_dir)): | 226 | if (re.search("'",self.remote_dir)): |
662 | 224 | raise BackendException("cannot handle directory names with single quotes with --use-scp!") | 227 | raise BackendException("cannot handle directory names with single quotes with --use-scp!") |
663 | @@ -256,7 +259,7 @@ | |||
664 | 256 | raise BackendException("sftp chdir to %s failed: %s" % (self.sftp.normalize(".")+"/"+d,e)) | 259 | raise BackendException("sftp chdir to %s failed: %s" % (self.sftp.normalize(".")+"/"+d,e)) |
665 | 257 | 260 | ||
666 | 258 | def _put(self, source_path, remote_filename): | 261 | def _put(self, source_path, remote_filename): |
668 | 259 | if globals.use_scp: | 262 | if self.use_scp: |
669 | 260 | f=file(source_path.name,'rb') | 263 | f=file(source_path.name,'rb') |
670 | 261 | try: | 264 | try: |
671 | 262 | chan=self.client.get_transport().open_session() | 265 | chan=self.client.get_transport().open_session() |
672 | @@ -284,7 +287,7 @@ | |||
673 | 284 | self.sftp.put(source_path.name,remote_filename) | 287 | self.sftp.put(source_path.name,remote_filename) |
674 | 285 | 288 | ||
675 | 286 | def _get(self, remote_filename, local_path): | 289 | def _get(self, remote_filename, local_path): |
677 | 287 | if globals.use_scp: | 290 | if self.use_scp: |
678 | 288 | try: | 291 | try: |
679 | 289 | chan=self.client.get_transport().open_session() | 292 | chan=self.client.get_transport().open_session() |
680 | 290 | chan.settimeout(globals.timeout) | 293 | chan.settimeout(globals.timeout) |
681 | @@ -327,7 +330,7 @@ | |||
682 | 327 | def _list(self): | 330 | def _list(self): |
683 | 328 | # In scp mode unavoidable quoting issues will make this fail if the | 331 | # In scp mode unavoidable quoting issues will make this fail if the |
684 | 329 | # directory name contains single quotes. | 332 | # directory name contains single quotes. |
686 | 330 | if globals.use_scp: | 333 | if self.use_scp: |
687 | 331 | output = self.runremote("ls -1 '%s'" % self.remote_dir, False, "scp dir listing ") | 334 | output = self.runremote("ls -1 '%s'" % self.remote_dir, False, "scp dir listing ") |
688 | 332 | return output.splitlines() | 335 | return output.splitlines() |
689 | 333 | else: | 336 | else: |
690 | @@ -336,7 +339,7 @@ | |||
691 | 336 | def _delete(self, filename): | 339 | def _delete(self, filename): |
692 | 337 | # In scp mode unavoidable quoting issues will cause failures if | 340 | # In scp mode unavoidable quoting issues will cause failures if |
693 | 338 | # filenames containing single quotes are encountered. | 341 | # filenames containing single quotes are encountered. |
695 | 339 | if globals.use_scp: | 342 | if self.use_scp: |
696 | 340 | self.runremote("rm '%s/%s'" % (self.remote_dir, filename), False, "scp rm ") | 343 | self.runremote("rm '%s/%s'" % (self.remote_dir, filename), False, "scp rm ") |
697 | 341 | else: | 344 | else: |
698 | 342 | self.sftp.remove(filename) | 345 | self.sftp.remove(filename) |
699 | @@ -370,3 +373,9 @@ | |||
700 | 370 | raise BackendException("could not load '%s', maybe corrupt?" % (file)) | 373 | raise BackendException("could not load '%s', maybe corrupt?" % (file)) |
701 | 371 | 374 | ||
702 | 372 | return sshconfig.lookup(host) | 375 | return sshconfig.lookup(host) |
703 | 376 | |||
704 | 377 | duplicity.backend.register_backend("sftp", SSHParamikoBackend) | ||
705 | 378 | duplicity.backend.register_backend("scp", SSHParamikoBackend) | ||
706 | 379 | duplicity.backend.register_backend("paramiko+sftp", SSHParamikoBackend) | ||
707 | 380 | duplicity.backend.register_backend("paramiko+scp", SSHParamikoBackend) | ||
708 | 381 | duplicity.backend.uses_netloc.extend([ 'sftp', 'scp', 'paramiko+sftp', 'paramiko+scp' ]) | ||
709 | 373 | 382 | ||
710 | === renamed file 'duplicity/backends/_ssh_pexpect.py' => 'duplicity/backends/ssh_pexpect_backend.py' | |||
711 | --- duplicity/backends/_ssh_pexpect.py 2014-04-28 02:49:39 +0000 | |||
712 | +++ duplicity/backends/ssh_pexpect_backend.py 2014-10-30 18:01:04 +0000 | |||
713 | @@ -49,6 +49,9 @@ | |||
714 | 49 | 49 | ||
715 | 50 | self.sftp_command = "sftp" | 50 | self.sftp_command = "sftp" |
716 | 51 | if globals.sftp_command: self.sftp_command = globals.sftp_command | 51 | if globals.sftp_command: self.sftp_command = globals.sftp_command |
717 | 52 | |||
718 | 53 | self.scheme = duplicity.backend.strip_prefix(parsed_url.scheme, 'pexpect') | ||
719 | 54 | self.use_scp = ( self.scheme == 'scp' ) | ||
720 | 52 | 55 | ||
721 | 53 | # host string of form [user@]hostname | 56 | # host string of form [user@]hostname |
722 | 54 | if parsed_url.username: | 57 | if parsed_url.username: |
723 | @@ -212,7 +215,7 @@ | |||
724 | 212 | raise BackendException("Error running '%s': %s" % (commandline, msg)) | 215 | raise BackendException("Error running '%s': %s" % (commandline, msg)) |
725 | 213 | 216 | ||
726 | 214 | def _put(self, source_path, remote_filename): | 217 | def _put(self, source_path, remote_filename): |
728 | 215 | if globals.use_scp: | 218 | if self.use_scp: |
729 | 216 | self.put_scp(source_path, remote_filename) | 219 | self.put_scp(source_path, remote_filename) |
730 | 217 | else: | 220 | else: |
731 | 218 | self.put_sftp(source_path, remote_filename) | 221 | self.put_sftp(source_path, remote_filename) |
732 | @@ -234,7 +237,7 @@ | |||
733 | 234 | self.run_scp_command(commandline) | 237 | self.run_scp_command(commandline) |
734 | 235 | 238 | ||
735 | 236 | def _get(self, remote_filename, local_path): | 239 | def _get(self, remote_filename, local_path): |
737 | 237 | if globals.use_scp: | 240 | if self.use_scp: |
738 | 238 | self.get_scp(remote_filename, local_path) | 241 | self.get_scp(remote_filename, local_path) |
739 | 239 | else: | 242 | else: |
740 | 240 | self.get_sftp(remote_filename, local_path) | 243 | self.get_sftp(remote_filename, local_path) |
741 | @@ -280,3 +283,8 @@ | |||
742 | 280 | commands.append("rm \"%s\"" % filename) | 283 | commands.append("rm \"%s\"" % filename) |
743 | 281 | commandline = ("%s %s %s" % (self.sftp_command, globals.ssh_options, self.host_string)) | 284 | commandline = ("%s %s %s" % (self.sftp_command, globals.ssh_options, self.host_string)) |
744 | 282 | self.run_sftp_command(commandline, commands) | 285 | self.run_sftp_command(commandline, commands) |
745 | 286 | |||
746 | 287 | duplicity.backend.register_backend("pexpect+sftp", SSHPExpectBackend) | ||
747 | 288 | duplicity.backend.register_backend("pexpect+scp", SSHPExpectBackend) | ||
748 | 289 | duplicity.backend.uses_netloc.extend([ 'pexpect+sftp', 'pexpect+scp' ]) | ||
749 | 290 | |||
750 | 283 | 291 | ||
751 | === modified file 'duplicity/commandline.py' | |||
752 | --- duplicity/commandline.py 2014-10-27 02:27:36 +0000 | |||
753 | +++ duplicity/commandline.py 2014-10-30 18:01:04 +0000 | |||
754 | @@ -537,9 +537,6 @@ | |||
755 | 537 | # default to batch mode using public-key encryption | 537 | # default to batch mode using public-key encryption |
756 | 538 | parser.add_option("--ssh-askpass", action = "store_true") | 538 | parser.add_option("--ssh-askpass", action = "store_true") |
757 | 539 | 539 | ||
758 | 540 | # allow the user to switch ssh backend | ||
759 | 541 | parser.add_option("--ssh-backend", metavar = _("paramiko|pexpect")) | ||
760 | 542 | |||
761 | 543 | # user added ssh options | 540 | # user added ssh options |
762 | 544 | parser.add_option("--ssh-options", action = "extend", metavar = _("options")) | 541 | parser.add_option("--ssh-options", action = "extend", metavar = _("options")) |
763 | 545 | 542 | ||
764 | @@ -567,8 +564,6 @@ | |||
765 | 567 | # Whether to specify --use-agent in GnuPG options | 564 | # Whether to specify --use-agent in GnuPG options |
766 | 568 | parser.add_option("--use-agent", action = "store_true") | 565 | parser.add_option("--use-agent", action = "store_true") |
767 | 569 | 566 | ||
768 | 570 | parser.add_option("--use-scp", action = "store_true") | ||
769 | 571 | |||
770 | 572 | parser.add_option("--verbosity", "-v", type = "verbosity", metavar = "[0-9]", | 567 | parser.add_option("--verbosity", "-v", type = "verbosity", metavar = "[0-9]", |
771 | 573 | dest = "", action = "callback", | 568 | dest = "", action = "callback", |
772 | 574 | callback = lambda o, s, v, p: log.setverbosity(v)) | 569 | callback = lambda o, s, v, p: log.setverbosity(v)) |
773 | 575 | 570 | ||
774 | === modified file 'duplicity/file_naming.py' | |||
775 | --- duplicity/file_naming.py 2014-10-27 02:27:36 +0000 | |||
776 | +++ duplicity/file_naming.py 2014-10-30 18:01:04 +0000 | |||
777 | @@ -393,6 +393,20 @@ | |||
778 | 393 | else: | 393 | else: |
779 | 394 | pr.encrypted = None | 394 | pr.encrypted = None |
780 | 395 | 395 | ||
781 | 396 | def valid_extension(): | ||
782 | 397 | """ | ||
783 | 398 | plausibility check for duplicity file extension | ||
784 | 399 | before starting to extensively parse the filenames | ||
785 | 400 | """ | ||
786 | 401 | res = re.match(r'.*\.(g|z|gpg|gz|tar|p|part|manifest|sigtar)$', filename ) | ||
787 | 402 | #print filename, res | ||
788 | 403 | if res is None : | ||
789 | 404 | return False | ||
790 | 405 | return True | ||
791 | 406 | |||
792 | 407 | if not valid_extension() : | ||
793 | 408 | return None | ||
794 | 409 | |||
795 | 396 | pr = check_full() | 410 | pr = check_full() |
796 | 397 | if not pr: | 411 | if not pr: |
797 | 398 | pr = check_inc() | 412 | pr = check_inc() |
798 | 399 | 413 | ||
799 | === modified file 'duplicity/globals.py' | |||
800 | --- duplicity/globals.py 2014-05-12 07:09:00 +0000 | |||
801 | +++ duplicity/globals.py 2014-10-30 18:01:04 +0000 | |||
802 | @@ -231,15 +231,9 @@ | |||
803 | 231 | # default to batch mode using public-key encryption | 231 | # default to batch mode using public-key encryption |
804 | 232 | ssh_askpass = False | 232 | ssh_askpass = False |
805 | 233 | 233 | ||
806 | 234 | # default ssh backend is paramiko | ||
807 | 235 | ssh_backend = "paramiko" | ||
808 | 236 | |||
809 | 237 | # user added ssh options | 234 | # user added ssh options |
810 | 238 | ssh_options = "" | 235 | ssh_options = "" |
811 | 239 | 236 | ||
812 | 240 | # whether to use scp for put/get, sftp is default | ||
813 | 241 | use_scp = False | ||
814 | 242 | |||
815 | 243 | # default cf backend is pyrax | 237 | # default cf backend is pyrax |
816 | 244 | cf_backend = "pyrax" | 238 | cf_backend = "pyrax" |
817 | 245 | 239 |
I like the general cleanup.
A couple of changes:
Please leave the file naming as it was. I had set these to be underscore-first to show they were called by the main ssh backend. See _boto*, _cf*.
I would like to see sftp: and scp: protocols go away. In reality they are just subsets of the OpenSSH (or other) package. You really can't run just scp: since we need 'ls' and 'del'. If we continue to allow these we need to emphasize that these are just aliases for ssh:.