Merge lp:~kevinoid/duplicity/windows-port into lp:duplicity/0.6-series

Proposed by Kevin Locke
Status: Rejected
Rejected by: Kenneth Loafman
Proposed branch: lp:~kevinoid/duplicity/windows-port
Merge into: lp:duplicity/0.6-series
Diff against target: 1552 lines (+668/-245)
18 files modified
dist/makedist (+58/-35)
dist/setup.py (+9/-10)
duplicity-bin (+96/-17)
duplicity.1 (+5/-3)
duplicity/GnuPGInterface.py (+215/-127)
duplicity/backend.py (+14/-1)
duplicity/backends/localbackend.py (+10/-1)
duplicity/commandline.py (+14/-1)
duplicity/compilec.py (+16/-3)
duplicity/dup_temp.py (+9/-5)
duplicity/globals.py (+48/-5)
duplicity/manifest.py (+9/-3)
duplicity/patchdir.py (+15/-0)
duplicity/path.py (+102/-17)
duplicity/selection.py (+21/-11)
duplicity/tarfile.py (+6/-2)
po/update-pot (+0/-4)
po/update-pot.py (+21/-0)
To merge this branch: bzr merge lp:~kevinoid/duplicity/windows-port
Reviewer Review Type Date Requested Status
duplicity-team Pending
Review via email: mp+39287@code.launchpad.net

Description of the change

This branch includes changes to support running Duplicity natively on Windows, as requested in bug 451582. I have done my best to separate out each change into logical units for commits and provide a detailed explanation and rationale for each change in the commit message.

The current work is only intended to port the main functionality of Duplicity and the local backend. The other backends have not been tested (and several, particularly ssh, are known not to work on Windows).

Most of the commits should not change any of the functionality of Duplicity. However, you may wish to take particular notice of revision 677 and 698 which do introduce functionality changes.

Note: Revision 682 added some portability improvements for the restart process and fixed bug 637556 in the process.

To post a comment you must log in.
Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

Rejecting for now. This needs to be reworked at this point to make it viable in the current codebase.

Unmerged revisions

703. By Kevin Locke <email address hidden>

Include updates to GnuPGInterface

These updates are from the current state of a branch of development that
I created to port GnuPGInterface to Windows, fix some bugs, and add
Python 3 support. The branch is available on github at
<http://github.com/kevinoid/py-gnupg>. These changes are taken from
commit 91667c.

I have assurances from the original author of GnuPGInterface that the
changes will be merged into his sources with minimal changes (if any)
once he has time and that it is safe to merge these changes into other
projects that need them without introducing a significant maintenance
burden.

Note: This version of GnuPGInterface now uses subprocess rather than
"raw" fork/exec. The threaded waitpid was removed due to threading
issues with subprocess (particularly Issue 1731717). It should be
largely unnecessary as any zombie child processes are reaped when a new
subprocess is started. If the more immediate reaping is required, the
threaded wait can easily be re-added (and less-easily be made thread
safe with subprocess).

Signed-off-by: Kevin Locke <email address hidden>

702. By Kevin Locke <email address hidden>

Unwrap temporary file instances

tempfile.TemporaryFile returns an instance of a wrapper class on
non-POSIX, non-Cygwin systems. Yet the librsync code requires a
standard file object. On Windows, the wrapper is unnecessary since the
file is opened with O_TEMPORARY and is deleted automatically on close.
On other systems the wrapper may be necessary to delete a file, so we
error out (in a way that future developers on those systems should be
able to find and understand...).

Note: This is a bit of a hack and relies on undocumented properties of
the object returned from tempfile.TemporaryFile. However, it avoids
the need to track and delete the temporary file ourselves.

Signed-off-by: Kevin Locke <email address hidden>

701. By Kevin Locke <email address hidden>

Make the manifest format more interoperable

At the cost of losing a bit of information, use the POSIX path
and EOL convention as part of the file format for manifest files.
This way backups created on one platform can be restored on another with
a different path and/or EOL convention.

To accomplish this, change the file mode to binary and convert native
paths to POSIX paths during manifest creation and back during load.

Note: During load the drive specifier for the target directory (if
there is one) is added to the POSIX-converted path. This allows
restoring cross-platform files more easily at the cost of losing a
warning about restoring files to a different drive than the original
backup. If this is unacceptable, the drive could be the first component
of the POSIX path or the cross-platform interoperability could be
dropped.

Signed-off-by: Kevin Locke <email address hidden>

700. By Kevin Locke <email address hidden>

Prevent Windows paths from being parsed as URLs

Windows paths which begin with a drive specifier are parsed as having a
scheme which is the drive letter. In order to avoid against these paths
being treated as URLs, check if the potential url_string looks like a
Windows path.

Note: The regex is not perfect, since it is possible that the "c" URL
scheme would not require slashes after the protocol. So limit the
checks to Windows platforms where paths are more likely to have this
form.

Signed-off-by: Kevin Locke <email address hidden>

699. By Kevin Locke <email address hidden>

Make file:// URL parsing more portable

Create path.from_url_path utility function which takes the path
component of a file URL and converts it into the native path
representation on the platform.

Note: RFC 1738 specifies that anything after file:// before the next
slash is a hostname (and therefore, implicitly, all paths are absolute).
However, to maintain backwards compatibility, allow for relative paths
to be specified in this manner.

Signed-off-by: Kevin Locke <email address hidden>

698. By Kevin Locke <email address hidden>

Ensure prefix ends with empty path component

WARNING: Functionality change

In order to prevent the given prefix from matching directories which
begin with the last component of prefix, ensure that prefix ends with an
empty path component (causing a tailing slash on POSIX systems).
Otherwise the prefix /foo/b would also include /foo/bar.

Note: If this functionality really was intended, wherever prefix is
removed from a path, the result should not start with a directory
separator (since it would result in a "/" first element in the return
value from path.split_all).

Signed-off-by: Kevin Locke <email address hidden>

697. By Kevin Locke <email address hidden>

Replace path manipulation based on "/"

Make use of path.split_all and os.path.join where paths were manipulated
based on "/"

In selection, change path import to keep path namespace, but alias
path.Path to Path to minimize the diff. The bare uses of split_all
seemed unnecessarily confusing and the other members of path are not
used directly so there is no real need for the * import.

Note: Removed unnecessary use of filter to remove blanks as this is
handled implicitly by os.path.split and therefore by path.split_all.

Note2: Need to be careful that os.path.join must have at least 1
argument, while "/".join() could work on a 0-length array.

Signed-off-by: Kevin Locke <email address hidden>

696. By Kevin Locke <email address hidden>

Create split_all utility function in path

This function is intended to replace the pathstr.split("/") idiom in
many places and provides a convenient way to break an arbitrary path
into its constituent components.

Note: Although it would be possible to split on os.altsep and os.sep
and it would work in many cases, and run a bit faster, it doesn't handle
more complex path systems and is likely to fail in corner cases even in
less complex path systems.

Signed-off-by: Kevin Locke <email address hidden>

695. By Kevin Locke <email address hidden>

Support systems without pwd and/or grp in tarfile

Inside tarfile this case is normally checked by the calling code, but
since uid2uname and gid2gname are called by Path without checking for
pwd and grp (and we want to support this case), remove the assertion
that these modules are defined. When they are not, throw KeyError
indicating the ID could not be mapped.

Signed-off-by: Kevin Locke <email address hidden>

694. By Kevin Locke <email address hidden>

Support more path systems in glob->regex conversion

 - Match against the system path separators, rather than "/"
 - Update duplicity.1 man page to indicate that the metacharacters match
   against the directory separator, rather than "/"

Note: This won't be portable to systems with more complex path systems
(like VMS), but that case is not trivial so it is worth waiting for a
need to arise.

Signed-off-by: Kevin Locke <email address hidden>

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'dist/makedist'
2--- dist/makedist 2010-07-22 19:15:11 +0000
3+++ dist/makedist 2010-10-25 15:49:45 +0000
4@@ -20,14 +20,14 @@
5 # along with duplicity; if not, write to the Free Software Foundation,
6 # Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
7
8-import os, re, shutil, time, sys
9+import os, re, shutil, subprocess, tarfile, traceback, time, sys, zipfile
10
11 SourceDir = "duplicity"
12 DistDir = "dist"
13
14 # Various details about the files must also be specified by the rpm
15 # spec template.
16-spec_template = "dist/duplicity.spec.template"
17+spec_template = os.path.join("dist", "duplicity.spec.template")
18
19 def VersionedCopy(source, dest):
20 """
21@@ -48,14 +48,13 @@
22 fout.write(outbuf)
23 assert not fout.close()
24
25-def MakeTar():
26+def MakeArchives():
27 """Create duplicity tar file"""
28 tardir = "duplicity-%s" % Version
29- tarfile = "duplicity-%s.tar.gz" % Version
30- try:
31- os.lstat(tardir)
32- os.system("rm -rf " + tardir)
33- except OSError: pass
34+ tarname = "duplicity-%s.tar.gz" % Version
35+ zipname = "duplicity-%s.zip" % Version
36+ if os.path.exists(tardir):
37+ shutil.rmtree(tardir)
38
39 os.mkdir(tardir)
40 for filename in [
41@@ -66,10 +65,10 @@
42 "LOG-README",
43 "README",
44 "tarfile-LICENSE",
45- SourceDir + "/_librsyncmodule.c",
46- DistDir + "/setup.py",
47+ os.path.join(SourceDir, "_librsyncmodule.c"),
48+ os.path.join(DistDir, "setup.py"),
49 ]:
50- assert not os.system("cp %s %s" % (filename, tardir)), filename
51+ shutil.copy(filename, tardir)
52
53 os.mkdir(os.path.join(tardir, "src"))
54 for filename in [
55@@ -104,39 +103,46 @@
56 "urlparse_2_5.py",
57 "util.py",
58 ]:
59- assert not os.system("cp %s/%s %s/src" % (SourceDir, filename, tardir)), filename
60+ shutil.copy(os.path.join(SourceDir, filename),
61+ os.path.join(tardir, "src"))
62
63 os.mkdir(os.path.join(tardir, "src", "backends"))
64 for filename in [
65- "backends/botobackend.py",
66- "backends/cloudfilesbackend.py",
67- "backends/ftpbackend.py",
68- "backends/giobackend.py",
69- "backends/hsibackend.py",
70- "backends/imapbackend.py",
71- "backends/__init__.py",
72- "backends/localbackend.py",
73- "backends/rsyncbackend.py",
74- "backends/sshbackend.py",
75- "backends/tahoebackend.py",
76- "backends/webdavbackend.py",
77+ "botobackend.py",
78+ "cloudfilesbackend.py",
79+ "ftpbackend.py",
80+ "giobackend.py",
81+ "hsibackend.py",
82+ "imapbackend.py",
83+ "__init__.py",
84+ "localbackend.py",
85+ "rsyncbackend.py",
86+ "sshbackend.py",
87+ "tahoebackend.py",
88+ "webdavbackend.py",
89 ]:
90- assert not os.system("cp %s/%s %s/src/backends" %
91- (SourceDir, filename, tardir)), filename
92+ shutil.copy(os.path.join(SourceDir, "backends", filename),
93+ os.path.join(tardir, "src", "backends"))
94+
95+ if subprocess.call([sys.executable, "update-pot.py"], cwd="po") != 0:
96+ sys.stderr.write("update-pot.py failed, translation files not updated!\n")
97
98 os.mkdir(os.path.join(tardir, "po"))
99- assert not os.system("cd po && ./update-pot")
100 for filename in [
101 "duplicity.pot",
102 ]:
103- assert not os.system("cp po/%s %s/po" % (filename, tardir)), filename
104- linguas = open('po/LINGUAS')
105+ shutil.copy(os.path.join("po", filename), os.path.join(tardir, "po"))
106+ linguas = open(os.path.join("po", "LINGUAS"))
107 for line in linguas:
108 langs = line.split()
109 for lang in langs:
110 assert not os.mkdir(os.path.join(tardir, "po", lang)), lang
111- assert not os.system("cp po/%s.po %s/po/%s" % (lang, tardir, lang)), lang
112- assert not os.system("msgfmt po/%s.po -o %s/po/%s/duplicity.mo" % (lang, tardir, lang)), lang
113+ shutil.copy(os.path.join("po", lang + ".po"),
114+ os.path.join(tardir, "po", lang))
115+ if os.system("msgfmt %s -o %s" %
116+ (os.path.join("po", lang + ".po"),
117+ os.path.join(tardir, "po", lang, "duplicity.mo"))) != 0:
118+ sys.stderr.write("Translation for " + lang + " NOT updated!\n")
119 linguas.close()
120
121 VersionedCopy(os.path.join(SourceDir, "globals.py"),
122@@ -154,9 +160,26 @@
123
124 os.chmod(os.path.join(tardir, "setup.py"), 0755)
125 os.chmod(os.path.join(tardir, "rdiffdir"), 0644)
126- os.system("tar -czf %s %s" % (tarfile, tardir))
127+
128+ with tarfile.open(tarname, "w:gz") as tar:
129+ tar.add(tardir)
130+
131+ def add_dir_to_zip(dir, zip, clen=None):
132+ if clen == None:
133+ clen = len(dir)
134+
135+ for entry in os.listdir(dir):
136+ entrypath = os.path.join(dir, entry)
137+ if os.path.isdir(entrypath):
138+ add_dir_to_zip(entrypath, zip, clen)
139+ else:
140+ zip.write(entrypath, entrypath[clen:])
141+
142+ with zipfile.ZipFile(zipname, "w") as zip:
143+ add_dir_to_zip(tardir, zip)
144+
145 shutil.rmtree(tardir)
146- return tarfile
147+ return (tarname, zipname)
148
149 def MakeSpecFile():
150 """Create spec file using spec template"""
151@@ -166,8 +189,8 @@
152
153 def Main():
154 print "Processing version " + Version
155- tarfile = MakeTar()
156- print "Made tar file " + tarfile
157+ archives = MakeArchives()
158+ print "Made archives: %s" % (archives,)
159 specfile = MakeSpecFile()
160 print "Made specfile " + specfile
161
162
163=== modified file 'dist/setup.py'
164--- dist/setup.py 2010-10-06 14:37:22 +0000
165+++ dist/setup.py 2010-10-25 15:49:45 +0000
166@@ -31,16 +31,15 @@
167
168 incdir_list = libdir_list = None
169
170-if os.name == 'posix':
171- LIBRSYNC_DIR = os.environ.get('LIBRSYNC_DIR', '')
172- args = sys.argv[:]
173- for arg in args:
174- if arg.startswith('--librsync-dir='):
175- LIBRSYNC_DIR = arg.split('=')[1]
176- sys.argv.remove(arg)
177- if LIBRSYNC_DIR:
178- incdir_list = [os.path.join(LIBRSYNC_DIR, 'include')]
179- libdir_list = [os.path.join(LIBRSYNC_DIR, 'lib')]
180+LIBRSYNC_DIR = os.environ.get('LIBRSYNC_DIR', '')
181+args = sys.argv[:]
182+for arg in args:
183+ if arg.startswith('--librsync-dir='):
184+ LIBRSYNC_DIR = arg.split('=')[1]
185+ sys.argv.remove(arg)
186+if LIBRSYNC_DIR:
187+ incdir_list = [os.path.join(LIBRSYNC_DIR, 'include')]
188+ libdir_list = [os.path.join(LIBRSYNC_DIR, 'lib')]
189
190 data_files = [('share/man/man1',
191 ['duplicity.1',
192
193=== modified file 'duplicity-bin'
194--- duplicity-bin 2010-08-26 13:01:10 +0000
195+++ duplicity-bin 2010-10-25 15:49:45 +0000
196@@ -28,11 +28,24 @@
197 # any suggestions.
198
199 import getpass, gzip, os, sys, time, types
200-import traceback, platform, statvfs, resource, re
201+import traceback, platform, statvfs, re
202
203 import gettext
204 gettext.install('duplicity')
205
206+try:
207+ import resource
208+ have_resource = True
209+except ImportError:
210+ have_resource = False
211+
212+if sys.platform == "win32":
213+ import ctypes
214+ import ctypes.util
215+ # Not to be confused with Python's msvcrt module which wraps part of msvcrt
216+ # Note: Load same msvcrt as Python to avoid cross-CRT problems
217+ ctmsvcrt = ctypes.cdll[ctypes.util.find_msvcrt()]
218+
219 from duplicity import log
220 log.setup()
221
222@@ -554,7 +567,10 @@
223 @param col_stats: collection status
224 """
225 if globals.restore_dir:
226- index = tuple(globals.restore_dir.split("/"))
227+ index = path.split_all(globals.restore_dir)
228+ if index[-1] == "":
229+ del index[-1]
230+ index = tuple(index)
231 else:
232 index = ()
233 time = globals.restore_time or dup_time.curtime
234@@ -994,16 +1010,13 @@
235 # First check disk space in temp area.
236 tempfile, tempname = tempdir.default().mkstemp()
237 os.close(tempfile)
238+
239 # strip off the temp dir and file
240- tempfs = os.path.sep.join(tempname.split(os.path.sep)[:-2])
241- try:
242- stats = os.statvfs(tempfs)
243- except:
244- log.FatalError(_("Unable to get free space on temp."),
245- log.ErrorCode.get_freespace_failed)
246+ tempfs = os.path.split(os.path.split(tempname)[0])[0]
247+
248 # Calculate space we need for at least 2 volumes of full or inc
249 # plus about 30% of one volume for the signature files.
250- freespace = stats[statvfs.F_FRSIZE] * stats[statvfs.F_BAVAIL]
251+ freespace = get_free_space(tempfs)
252 needspace = (((globals.async_concurrency + 1) * globals.volsize)
253 + int(0.30 * globals.volsize))
254 if freespace < needspace:
255@@ -1015,16 +1028,82 @@
256
257 # Some environments like Cygwin run with an artificially
258 # low value for max open files. Check for safe number.
259+ check_resource_limits()
260+
261+
262+def check_resource_limits():
263+ """
264+ Check for sufficient resource limits:
265+ - enough max open files
266+ Attempt to increase limits to sufficient values if insufficient
267+ Put out fatal error if not sufficient to run
268+
269+ Requires the resource module
270+
271+ @rtype: void
272+ @return: void
273+ """
274+ if have_resource:
275 try:
276 soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
277 except resource.error:
278 log.FatalError(_("Unable to get max open files."),
279 log.ErrorCode.get_ulimit_failed)
280- maxopen = min([l for l in (soft, hard) if l > -1])
281- if maxopen < 1024:
282- log.FatalError(_("Max open files of %s is too low, should be >= 1024.\n"
283- "Use 'ulimit -n 1024' or higher to correct.\n") % (maxopen,),
284- log.ErrorCode.maxopen_too_low)
285+
286+ if soft > -1 and soft < 1024 and soft < hard:
287+ try:
288+ newsoft = min(1024, hard)
289+ resource.setrlimit(resource.RLIMIT_NOFILE, (newsoft, hard))
290+ soft = newsoft
291+ except resource.error:
292+ pass
293+ elif sys.platform == "win32":
294+ # 2048 from http://msdn.microsoft.com/en-us/library/6e3b887c.aspx
295+ soft, hard = ctmsvcrt._getmaxstdio(), 2048
296+
297+ if soft < 1024:
298+ newsoft = ctmsvcrt._setmaxstdio(1024)
299+ if newsoft > -1:
300+ soft = newsoft
301+ else:
302+ log.FatalError(_("Unable to get max open files."),
303+ log.ErrorCode.get_ulimit_failed)
304+
305+ maxopen = min([l for l in (soft, hard) if l > -1])
306+ if maxopen < 1024:
307+ log.FatalError(_("Max open files of %s is too low, should be >= 1024.\n"
308+ "Use 'ulimit -n 1024' or higher to correct.\n") % (maxopen,),
309+ log.ErrorCode.maxopen_too_low)
310+
311+
312+def get_free_space(dir):
313+ """
314+ Get the free space available in a given directory
315+
316+ @type dir: string
317+ @param dir: directory in which to measure free space
318+
319+ @rtype: int
320+ @return: amount of free space on the filesystem containing dir (in bytes)
321+ """
322+ if hasattr(os, "statvfs"):
323+ try:
324+ stats = os.statvfs(dir)
325+ except:
326+ log.FatalError(_("Unable to get free space on temp."),
327+ log.ErrorCode.get_freespace_failed)
328+
329+ return stats[statvfs.F_FRSIZE] * stats[statvfs.F_BAVAIL]
330+ elif sys.platform == "win32":
331+ freespaceull = ctypes.c_ulonglong(0)
332+ ctypes.windll.kernel32.GetDiskFreeSpaceExW(ctypes.c_wchar_p(dir),
333+ None, None, ctypes.pointer(freespaceull))
334+
335+ return freespaceull.value
336+ else:
337+ log.FatalError(_("Unable to get free space on temp."),
338+ log.ErrorCode.get_freespace_failed)
339+
340
341 def log_startup_parms(verbosity=log.INFO):
342 """
343@@ -1071,7 +1150,7 @@
344 log.Notice(_("RESTART: The first volume failed to upload before termination.\n"
345 " Restart is impossible...starting backup from beginning."))
346 self.last_backup.delete()
347- os.execve(sys.argv[0], sys.argv[1:], os.environ)
348+ self.execv(sys.executable, sys.argv)
349 elif mf_len - self.start_vol > 0:
350 # upload of N vols failed, fix manifest and restart
351 log.Notice(_("RESTART: Volumes %d to %d failed to upload before termination.\n"
352@@ -1087,7 +1166,7 @@
353 " backup then restart the backup from the beginning.") %
354 (mf_len, self.start_vol))
355 self.last_backup.delete()
356- os.execve(sys.argv[0], sys.argv[1:], os.environ)
357+ self.execv(sys.executable, sys.argv)
358
359
360 def setLastSaved(self, mf):
361@@ -1112,7 +1191,7 @@
362
363 # if python is run setuid, it's only partway set,
364 # so make sure to run with euid/egid of root
365- if os.geteuid() == 0:
366+ if hasattr(os, "geteuid") and os.geteuid() == 0:
367 # make sure uid/gid match euid/egid
368 os.setuid(os.geteuid())
369 os.setgid(os.getegid())
370
371=== modified file 'duplicity.1'
372--- duplicity.1 2010-08-26 14:11:14 +0000
373+++ duplicity.1 2010-10-25 15:49:45 +0000
374@@ -871,14 +871,16 @@
375 .BR [...] .
376 As in a normal shell,
377 .B *
378-can be expanded to any string of characters not containing "/",
379+can be expanded to any string of characters not containing a directory
380+separator,
381 .B ?
382-expands to any character except "/", and
383+expands to any character except a directory separator, and
384 .B [...]
385 expands to a single character of those characters specified (ranges
386 are acceptable). The new special pattern,
387 .BR ** ,
388-expands to any string of characters whether or not it contains "/".
389+expands to any string of characters whether or not it contains a
390+directory separator.
391 Furthermore, if the pattern starts with "ignorecase:" (case
392 insensitive), then this prefix will be removed and any character in
393 the string can be replaced with an upper- or lowercase version of
394
395=== modified file 'duplicity/GnuPGInterface.py'
396--- duplicity/GnuPGInterface.py 2010-07-22 19:15:11 +0000
397+++ duplicity/GnuPGInterface.py 2010-10-25 15:49:45 +0000
398@@ -220,42 +220,55 @@
399 or see http://www.gnu.org/copyleft/lesser.html
400 """
401
402+import errno
403 import os
404+import subprocess
405 import sys
406-import fcntl
407-
408-from duplicity import log
409-
410-try:
411- import threading
412+
413+if sys.platform == "win32":
414+ # Required windows-only imports
415+ import msvcrt
416+ import _subprocess
417+
418+# Define next function for Python pre-2.6
419+try:
420+ next
421+except NameError:
422+ def next(itr):
423+ return itr.next()
424+
425+try:
426+ import fcntl
427 except ImportError:
428- import dummy_threading #@UnusedImport
429- log.Warn("Threading not available -- zombie processes may appear")
430+ # import success/failure is checked before use
431+ pass
432
433 __author__ = "Frank J. Tobin, ftobin@neverending.org"
434 __version__ = "0.3.2"
435-__revision__ = "$Id: GnuPGInterface.py,v 1.6 2009/06/06 17:35:19 loafman Exp $"
436+__revision__ = "$Id$"
437
438 # "standard" filehandles attached to processes
439 _stds = [ 'stdin', 'stdout', 'stderr' ]
440
441 # the permissions each type of fh needs to be opened with
442-_fd_modes = { 'stdin': 'w',
443- 'stdout': 'r',
444- 'stderr': 'r',
445- 'passphrase': 'w',
446- 'command': 'w',
447- 'logger': 'r',
448- 'status': 'r'
449+_fd_modes = { 'stdin': 'wb',
450+ 'stdout': 'rb',
451+ 'stderr': 'rb',
452+ 'passphrase': 'wb',
453+ 'attribute': 'rb',
454+ 'command': 'wb',
455+ 'logger': 'rb',
456+ 'status': 'rb'
457 }
458
459 # correlation between handle names and the arguments we'll pass
460 _fd_options = { 'passphrase': '--passphrase-fd',
461 'logger': '--logger-fd',
462 'status': '--status-fd',
463- 'command': '--command-fd' }
464+ 'command': '--command-fd',
465+ 'attribute': '--attribute-fd' }
466
467-class GnuPG:
468+class GnuPG(object):
469 """Class instances represent GnuPG.
470
471 Instance attributes of a GnuPG object are:
472@@ -276,6 +289,8 @@
473 the command-line options used when calling GnuPG.
474 """
475
476+ __slots__ = ['call', 'passphrase', 'options']
477+
478 def __init__(self):
479 self.call = 'gpg'
480 self.passphrase = None
481@@ -349,14 +364,14 @@
482 if attach_fhs == None: attach_fhs = {}
483
484 for std in _stds:
485- if not attach_fhs.has_key(std) \
486+ if std not in attach_fhs \
487 and std not in create_fhs:
488 attach_fhs.setdefault(std, getattr(sys, std))
489
490 handle_passphrase = 0
491
492 if self.passphrase != None \
493- and not attach_fhs.has_key('passphrase') \
494+ and 'passphrase' not in attach_fhs \
495 and 'passphrase' not in create_fhs:
496 handle_passphrase = 1
497 create_fhs.append('passphrase')
498@@ -366,7 +381,10 @@
499
500 if handle_passphrase:
501 passphrase_fh = process.handles['passphrase']
502- passphrase_fh.write( self.passphrase )
503+ if sys.version_info >= (3, 0) and isinstance(self.passphrase, str):
504+ passphrase_fh.write( self.passphrase.encode() )
505+ else:
506+ passphrase_fh.write( self.passphrase )
507 passphrase_fh.close()
508 del process.handles['passphrase']
509
510@@ -379,45 +397,40 @@
511
512 process = Process()
513
514- for fh_name in create_fhs + attach_fhs.keys():
515- if not _fd_modes.has_key(fh_name):
516- raise KeyError, \
517- "unrecognized filehandle name '%s'; must be one of %s" \
518- % (fh_name, _fd_modes.keys())
519+ for fh_name in create_fhs + list(attach_fhs.keys()):
520+ if fh_name not in _fd_modes:
521+ raise KeyError("unrecognized filehandle name '%s'; must be one of %s" \
522+ % (fh_name, list(_fd_modes.keys())))
523
524 for fh_name in create_fhs:
525 # make sure the user doesn't specify a filehandle
526 # to be created *and* attached
527- if attach_fhs.has_key(fh_name):
528- raise ValueError, \
529- "cannot have filehandle '%s' in both create_fhs and attach_fhs" \
530- % fh_name
531+ if fh_name in attach_fhs:
532+ raise ValueError("cannot have filehandle '%s' in both create_fhs and attach_fhs" \
533+ % fh_name)
534
535 pipe = os.pipe()
536 # fix by drt@un.bewaff.net noting
537 # that since pipes are unidirectional on some systems,
538 # so we have to 'turn the pipe around'
539 # if we are writing
540- if _fd_modes[fh_name] == 'w': pipe = (pipe[1], pipe[0])
541+ if _fd_modes[fh_name][0] == 'w': pipe = (pipe[1], pipe[0])
542+
543+ # Close the parent end in child to prevent deadlock
544+ if "fcntl" in globals():
545+ fcntl.fcntl(pipe[0], fcntl.F_SETFD, fcntl.FD_CLOEXEC)
546+
547 process._pipes[fh_name] = Pipe(pipe[0], pipe[1], 0)
548
549 for fh_name, fh in attach_fhs.items():
550 process._pipes[fh_name] = Pipe(fh.fileno(), fh.fileno(), 1)
551
552- process.pid = os.fork()
553- if process.pid != 0:
554- # start a threaded_waitpid on the child
555- process.thread = threading.Thread(target=threaded_waitpid,
556- name="wait%d" % process.pid,
557- args=(process,))
558- process.thread.start()
559-
560- if process.pid == 0: self._as_child(process, gnupg_commands, args)
561- return self._as_parent(process)
562-
563-
564- def _as_parent(self, process):
565- """Stuff run after forking in parent"""
566+ self._launch_process(process, gnupg_commands, args)
567+ return self._handle_pipes(process)
568+
569+
570+ def _handle_pipes(self, process):
571+ """Deal with pipes after the child process has been created"""
572 for k, p in process._pipes.items():
573 if not p.direct:
574 os.close(p.child)
575@@ -428,43 +441,137 @@
576
577 return process
578
579-
580- def _as_child(self, process, gnupg_commands, args):
581- """Stuff run after forking in child"""
582- # child
583- for std in _stds:
584- p = process._pipes[std]
585- os.dup2( p.child, getattr(sys, "__%s__" % std).fileno() )
586-
587- for k, p in process._pipes.items():
588- if p.direct and k not in _stds:
589- # we want the fh to stay open after execing
590- fcntl.fcntl( p.child, fcntl.F_SETFD, 0 )
591-
592+ def _create_preexec_fn(self, process):
593+ """Create and return a function to do cleanup before exec
594+
595+ The cleanup function will close all file descriptors which are not
596+ needed by the child process. This is required to prevent unnecessary
597+ blocking on the final read of pipes not set FD_CLOEXEC due to gpg
598+ inheriting an open copy of the input end of the pipe. This can cause
599+ delays in unrelated parts of the program or deadlocks in the case that
600+ one end of the pipe is passed to attach_fds.
601+
602+ FIXME: There is a race condition where a pipe can be created in
603+ another thread after this function runs before exec is called and it
604+ will not be closed. This race condition will remain until a better
605+ way to avoid closing the error pipe created by submodule is identified.
606+ """
607+ if sys.platform == "win32":
608+ return None # No cleanup necessary
609+
610+ try:
611+ MAXFD = os.sysconf("SC_OPEN_MAX")
612+ if MAXFD == -1:
613+ MAXFD = 256
614+ except:
615+ MAXFD = 256
616+
617+ # Get list of fds to close now, so we don't close the error pipe
618+ # created by submodule for reporting exec errors
619+ child_fds = [p.child for p in process._pipes.values()]
620+ child_fds.sort()
621+ child_fds.append(MAXFD) # Sentinel value, simplifies code greatly
622+
623+ child_fds_iter = iter(child_fds)
624+ child_fd = next(child_fds_iter)
625+ while child_fd < 3:
626+ child_fd = next(child_fds_iter)
627+
628+ extra_fds = []
629+ # FIXME: Is there a better (portable) way to list all open FDs?
630+ for fd in range(3, MAXFD):
631+ if fd > child_fd:
632+ child_fd = next(child_fds_iter)
633+
634+ if fd == child_fd:
635+ continue
636+
637+ try:
638+ # Note: Can't use lseek, can cause nul byte in pipes
639+ # where the position has not been set by read/write
640+ #os.lseek(fd, os.SEEK_CUR, 0)
641+ os.tcgetpgrp(fd)
642+ except OSError:
643+ # FIXME: When support for Python 2.5 is dropped, use 'as'
644+ oe = sys.exc_info()[1]
645+ if oe.errno == errno.EBADF:
646+ continue
647+
648+ extra_fds.append(fd)
649+
650+ def preexec_fn():
651+ # Note: This function runs after standard FDs have been renumbered
652+ # from their original values to 0, 1, 2
653+
654+ for fd in extra_fds:
655+ try:
656+ os.close(fd)
657+ except OSError:
658+ pass
659+
660+ # Ensure that all descriptors passed to the child will remain open
661+ # Arguably FD_CLOEXEC descriptors should be an argument error
662+ # But for backwards compatibility, we just fix it here (after fork)
663+ for fd in [0, 1, 2] + child_fds[:-1]:
664+ try:
665+ fcntl.fcntl(fd, fcntl.F_SETFD, 0)
666+ except OSError:
667+ # Will happen for renumbered FDs
668+ pass
669+
670+ return preexec_fn
671+
672+
673+ def _launch_process(self, process, gnupg_commands, args):
674+ """Run the child process"""
675 fd_args = []
676-
677 for k, p in process._pipes.items():
678 # set command-line options for non-standard fds
679- if k not in _stds:
680- fd_args.extend([ _fd_options[k], "%d" % p.child ])
681+ if k in _stds:
682+ continue
683
684- if not p.direct: os.close(p.parent)
685+ if sys.platform == "win32":
686+ # Must pass inheritable os file handle
687+ curproc = _subprocess.GetCurrentProcess()
688+ pchandle = msvcrt.get_osfhandle(p.child)
689+ pcihandle = _subprocess.DuplicateHandle(
690+ curproc, pchandle, curproc, 0, 1,
691+ _subprocess.DUPLICATE_SAME_ACCESS)
692+ fdarg = pcihandle.Detach()
693+ else:
694+ # Must pass file descriptor
695+ fdarg = p.child
696+ fd_args.extend([ _fd_options[k], str(fdarg) ])
697
698 command = [ self.call ] + fd_args + self.options.get_args() \
699 + gnupg_commands + args
700
701- os.execvp( command[0], command )
702-
703-
704-class Pipe:
705+ if len(fd_args) > 0:
706+ # Can't close all file descriptors
707+ # Create preexec function to close what we can
708+ preexec_fn = self._create_preexec_fn(process)
709+
710+ process._subproc = subprocess.Popen(command,
711+ stdin=process._pipes['stdin'].child,
712+ stdout=process._pipes['stdout'].child,
713+ stderr=process._pipes['stderr'].child,
714+ close_fds=not len(fd_args) > 0,
715+ preexec_fn=preexec_fn,
716+ shell=False)
717+ process.pid = process._subproc.pid
718+
719+
720+class Pipe(object):
721 """simple struct holding stuff about pipes we use"""
722+ __slots__ = ['parent', 'child', 'direct']
723+
724 def __init__(self, parent, child, direct):
725 self.parent = parent
726 self.child = child
727 self.direct = direct
728
729
730-class Options:
731+class Options(object):
732 """Objects of this class encompass options passed to GnuPG.
733 This class is responsible for determining command-line arguments
734 which are based on options. It can be said that a GnuPG
735@@ -493,6 +600,8 @@
736
737 * homedir
738 * default_key
739+ * keyring
740+ * secret_keyring
741 * comment
742 * compress_algo
743 * options
744@@ -536,38 +645,34 @@
745 ['--armor', '--recipient', 'Alice', '--recipient', 'Bob', '--no-secmem-warning']
746 """
747
748+ booleans = ('armor', 'no_greeting', 'verbose', 'no_verbose',
749+ 'batch', 'always_trust', 'rfc1991', 'openpgp',
750+ 'quiet', 'no_options', 'textmode', 'force_v3_sigs')
751+
752+ metas = ('meta_pgp_5_compatible', 'meta_pgp_2_compatible',
753+ 'meta_interactive')
754+
755+ strings = ('homedir', 'default_key', 'comment', 'compress_algo',
756+ 'options', 'keyring', 'secret_keyring')
757+
758+ lists = ('encrypt_to', 'recipients')
759+
760+ __slots__ = booleans + metas + strings + lists + ('extra_args',)
761+
762 def __init__(self):
763- # booleans
764- self.armor = 0
765- self.no_greeting = 0
766- self.verbose = 0
767- self.no_verbose = 0
768- self.quiet = 0
769- self.batch = 0
770- self.always_trust = 0
771- self.rfc1991 = 0
772- self.openpgp = 0
773- self.force_v3_sigs = 0
774- self.no_options = 0
775- self.textmode = 0
776+ for b in self.booleans:
777+ setattr(self, b, 0)
778
779- # meta-option booleans
780- self.meta_pgp_5_compatible = 0
781- self.meta_pgp_2_compatible = 0
782+ for m in self.metas:
783+ setattr(self, m, 0)
784 self.meta_interactive = 1
785
786- # strings
787- self.homedir = None
788- self.default_key = None
789- self.comment = None
790- self.compress_algo = None
791- self.options = None
792-
793- # lists
794- self.encrypt_to = []
795- self.recipients = []
796-
797- # miscellaneous arguments
798+ for s in self.strings:
799+ setattr(self, s, None)
800+
801+ for l in self.lists:
802+ setattr(self, l, [])
803+
804 self.extra_args = []
805
806 def get_args( self ):
807@@ -583,6 +688,8 @@
808 if self.comment != None: args.extend( [ '--comment', self.comment ] )
809 if self.compress_algo != None: args.extend( [ '--compress-algo', self.compress_algo ] )
810 if self.default_key != None: args.extend( [ '--default-key', self.default_key ] )
811+ if self.keyring != None: args.extend( [ '--keyring', self.keyring ] )
812+ if self.secret_keyring != None: args.extend( [ '--secret-keyring', self.secret_keyring ] )
813
814 if self.no_options: args.append( '--no-options' )
815 if self.armor: args.append( '--armor' )
816@@ -615,7 +722,7 @@
817 return args
818
819
820-class Process:
821+class Process(object):
822 """Objects of this class encompass properties of a GnuPG
823 process spawned by GnuPG.run().
824
825@@ -637,43 +744,24 @@
826 os.waitpid() to clean up the process, especially
827 if multiple calls are made to run().
828 """
829+ __slots__ = ['_pipes', 'handles', 'pid', '_subproc']
830
831 def __init__(self):
832- self._pipes = {}
833- self.handles = {}
834- self.pid = None
835- self._waited = None
836- self.thread = None
837- self.returned = None
838+ self._pipes = {}
839+ self.handles = {}
840+ self.pid = None
841+ self._subproc = None
842
843 def wait(self):
844- """
845- Wait on threaded_waitpid to exit and examine results.
846- Will raise an IOError if the process exits non-zero.
847- """
848- if self.returned == None:
849- self.thread.join()
850- if self.returned != 0:
851- raise IOError, "GnuPG exited non-zero, with code %d" % (self.returned >> 8)
852-
853-
854-def threaded_waitpid(process):
855- """
856- When started as a thread with the Process object, thread
857- will execute an immediate waitpid() against the process
858- pid and will collect the process termination info. This
859- will allow us to reap child processes as soon as possible,
860- thus freeing resources quickly.
861- """
862- try:
863- process.returned = os.waitpid(process.pid, 0)[1]
864- except:
865- log.Debug("GPG process %d terminated before wait()" % process.pid)
866- process.returned = 0
867-
868+ """Wait on the process to exit, allowing for child cleanup.
869+ Will raise an IOError if the process exits non-zero."""
870+
871+ e = self._subproc.wait()
872+ if e != 0:
873+ raise IOError("GnuPG exited non-zero, with code %d" % e)
874
875 def _run_doctests():
876- import doctest, GnuPGInterface #@UnresolvedImport
877+ import doctest, GnuPGInterface
878 return doctest.testmod(GnuPGInterface)
879
880 # deprecated
881
882=== modified file 'duplicity/backend.py'
883--- duplicity/backend.py 2010-08-09 18:56:03 +0000
884+++ duplicity/backend.py 2010-10-25 15:49:45 +0000
885@@ -64,7 +64,8 @@
886 @return: void
887 """
888 path = duplicity.backends.__path__[0]
889- assert path.endswith("duplicity/backends"), duplicity.backends.__path__
890+ assert os.path.normcase(path).endswith("duplicity" + os.path.sep + "backends"), \
891+ duplicity.backends.__path__
892
893 files = os.listdir(path)
894 for fn in files:
895@@ -201,6 +202,12 @@
896
897 Raise InvalidBackendURL on invalid URL's
898 """
899+
900+ if sys.platform == "win32":
901+ # Regex to match a path containing a Windows drive specifier
902+ # Valid paths include "C:" "C:/" "C:stuff" "C:/stuff", not "C://stuff"
903+ _drivespecre = re.compile("^[a-z]:(?![/\\\\]{2})", re.IGNORECASE)
904+
905 def __init__(self, url_string):
906 self.url_string = url_string
907 _ensure_urlparser_initialized()
908@@ -266,6 +273,12 @@
909 if not pu.scheme:
910 return
911
912+ # This happens with implicit local paths with a drive specifier
913+ if sys.platform == "win32" and \
914+ re.match(ParsedUrl._drivespecre, url_string):
915+ self.scheme = ""
916+ return
917+
918 # Our backends do not handle implicit hosts.
919 if pu.scheme in urlparser.uses_netloc and not pu.hostname:
920 raise InvalidBackendURL("Missing hostname in a backend URL which "
921
922=== modified file 'duplicity/backends/localbackend.py'
923--- duplicity/backends/localbackend.py 2010-07-22 19:15:11 +0000
924+++ duplicity/backends/localbackend.py 2010-10-25 15:49:45 +0000
925@@ -39,7 +39,16 @@
926 # The URL form "file:MyFile" is not a valid duplicity target.
927 if not parsed_url.path.startswith( '//' ):
928 raise BackendException( "Bad file:// path syntax." )
929- self.remote_pathdir = path.Path(parsed_url.path[2:])
930+
931+ # According to RFC 1738, file URLs take the form
932+ # file://<hostname>/<path> where <hostname> == "" is localhost
933+ # However, for backwards compatibility, interpret file://stuff/... as
934+ # being a relative path starting with directory stuff
935+ if parsed_url.path[2] == '/':
936+ pathstr = path.from_url_path(parsed_url.path[3:], is_abs=True)
937+ else:
938+ pathstr = path.from_url_path(parsed_url.path[2:], is_abs=False)
939+ self.remote_pathdir = path.Path(pathstr)
940
941 def put(self, source_path, remote_filename = None, rename = None):
942 """If rename is set, try that first, copying if doesn't work"""
943
944=== modified file 'duplicity/commandline.py'
945--- duplicity/commandline.py 2010-10-06 15:57:51 +0000
946+++ duplicity/commandline.py 2010-10-25 15:49:45 +0000
947@@ -184,8 +184,21 @@
948 def set_log_fd(fd):
949 if fd < 1:
950 raise optparse.OptionValueError("log-fd must be greater than zero.")
951+ if sys.platform == "win32":
952+ # Convert OS file handle to C file descriptor
953+ import msvcrt
954+ fd = msvcrt.open_osfhandle(fd, 1) # 1 = _O_WRONLY
955+ raise optparse.OptionValueError("Unable to open log-fd.")
956 log.add_fd(fd)
957
958+ def set_restore_dir(dir):
959+ # Remove empty tail component, if any
960+ head, tail = os.path.split(dir)
961+ if not tail:
962+ dir = head
963+
964+ globals.restore_dir = dir
965+
966 def set_time_sep(sep, opt):
967 if sep == '-':
968 raise optparse.OptionValueError("Dash ('-') not valid for time-separator.")
969@@ -291,7 +304,7 @@
970 # --archive-dir <path>
971 parser.add_option("--file-to-restore", "-r", action="callback", type="file",
972 metavar=_("path"), dest="restore_dir",
973- callback=lambda o, s, v, p: setattr(p.values, "restore_dir", v.rstrip('/')))
974+ callback=set_restore_dir)
975
976 # Used to confirm certain destructive operations like deleting old files.
977 parser.add_option("--force", action="store_true")
978
979=== modified file 'duplicity/compilec.py'
980--- duplicity/compilec.py 2009-04-01 15:07:45 +0000
981+++ duplicity/compilec.py 2010-10-25 15:49:45 +0000
982@@ -20,7 +20,9 @@
983 # along with duplicity; if not, write to the Free Software Foundation,
984 # Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
985
986-import sys, os
987+import os
988+import shutil
989+import sys
990 from distutils.core import setup, Extension
991
992 assert len(sys.argv) == 1
993@@ -33,5 +35,16 @@
994 ["_librsyncmodule.c"],
995 libraries=["rsync"])])
996
997-assert not os.system("mv `find build -name _librsync.so` .")
998-assert not os.system("rm -rf build")
999+def find_any_of(filenames, basedir="."):
1000+ for dirpath, dirnames, dirfilenames in os.walk(basedir):
1001+ for filename in filenames:
1002+ if filename in dirfilenames:
1003+ return os.path.join(dirpath, filename)
1004+
1005+extfile = find_any_of(("_librsync.pyd", "_librsync.so"), "build")
1006+if not extfile:
1007+ sys.stderr.write("Can't find _librsync extension binary, build failed?\n")
1008+ sys.exit(1)
1009+
1010+os.rename(extfile, os.path.basename(extfile))
1011+shutil.rmtree("build")
1012
1013=== modified file 'duplicity/dup_temp.py'
1014--- duplicity/dup_temp.py 2010-07-22 19:15:11 +0000
1015+++ duplicity/dup_temp.py 2010-10-25 15:49:45 +0000
1016@@ -161,9 +161,11 @@
1017 We have achieved the first checkpoint, make file visible and permanent.
1018 """
1019 assert not globals.restart
1020- self.tdp.rename(self.dirpath.append(self.partname))
1021+ # Can't rename files open for write on Windows. Wait for close hook
1022+ if sys.platform != "win32":
1023+ self.tdp.rename(self.dirpath.append(self.partname))
1024+ del self.hooklist[0]
1025 self.fileobj.flush()
1026- del self.hooklist[0]
1027
1028 def to_remote(self):
1029 """
1030@@ -173,13 +175,15 @@
1031 pr = file_naming.parse(self.remname)
1032 src = self.dirpath.append(self.partname)
1033 tgt = self.dirpath.append(self.remname)
1034- src_iter = SrcIter(src)
1035 if pr.compressed:
1036+ src_iter = SrcIter(src)
1037 gpg.GzipWriteFile(src_iter, tgt.name, size = sys.maxint)
1038 elif pr.encrypted:
1039+ src_iter = SrcIter(src)
1040 gpg.GPGWriteFile(src_iter, tgt.name, globals.gpg_profile, size = sys.maxint)
1041 else:
1042- os.system("cp -p %s %s" % (src.name, tgt.name))
1043+ src.copy(tgt)
1044+ src.copy_attribs(tgt)
1045 globals.backend.put(tgt) #@UndefinedVariable
1046 os.unlink(tgt.name)
1047
1048@@ -189,9 +193,9 @@
1049 """
1050 src = self.dirpath.append(self.partname)
1051 tgt = self.dirpath.append(self.permname)
1052- src_iter = SrcIter(src)
1053 pr = file_naming.parse(self.permname)
1054 if pr.compressed:
1055+ src_iter = SrcIter(src)
1056 gpg.GzipWriteFile(src_iter, tgt.name, size = sys.maxint)
1057 os.unlink(src.name)
1058 else:
1059
1060=== modified file 'duplicity/globals.py'
1061--- duplicity/globals.py 2010-08-26 13:01:10 +0000
1062+++ duplicity/globals.py 2010-10-25 15:49:45 +0000
1063@@ -21,7 +21,7 @@
1064
1065 """Store global configuration information"""
1066
1067-import socket, os
1068+import sys, socket, os
1069
1070 # The current version of duplicity
1071 version = "$version"
1072@@ -36,16 +36,59 @@
1073 # The symbolic name of the backup being operated upon.
1074 backup_name = None
1075
1076+# On Windows, use SHGetFolderPath for determining program directories
1077+if sys.platform == "win32":
1078+ import ctypes
1079+ import ctypes.wintypes as wintypes
1080+ windll = ctypes.windll
1081+
1082+ CSIDL_APPDATA = 0x001a
1083+ CSIDL_LOCAL_APPDATA = 0x001c
1084+ def get_csidl_folder_path(csidl):
1085+ SHGetFolderPath = windll.shell32.SHGetFolderPathW
1086+ SHGetFolderPath.argtypes = [
1087+ wintypes.HWND,
1088+ ctypes.c_int,
1089+ wintypes.HANDLE,
1090+ wintypes.DWORD,
1091+ wintypes.LPWSTR,
1092+ ]
1093+ folderpath = wintypes.create_unicode_buffer(wintypes.MAX_PATH)
1094+ result = SHGetFolderPath(0, csidl, 0, 0, folderpath)
1095+ if result != 0:
1096+ raise WindowsError(result, "Unable to get folder path")
1097+ return folderpath.value
1098+
1099+
1100 # Set to the Path of the archive directory (the directory which
1101 # contains the signatures and manifests of the relevent backup
1102 # collection), and for checkpoint state between volumes.
1103 # NOTE: this gets expanded in duplicity.commandline
1104-os.environ["XDG_CACHE_HOME"] = os.getenv("XDG_CACHE_HOME", os.path.expanduser("~/.cache"))
1105-archive_dir = os.path.expandvars("$XDG_CACHE_HOME/duplicity")
1106+if sys.platform == "win32":
1107+ try:
1108+ archive_dir = get_csidl_folder_path(CSIDL_LOCAL_APPDATA)
1109+ except WindowsError:
1110+ try:
1111+ archive_dir = get_csidl_folder_path(CSIDL_APPDATA)
1112+ except WindowsError:
1113+ archive_dir = os.getenv("LOCALAPPDATA") or \
1114+ os.getenv("APPDATA") or \
1115+ os.path.expanduser("~")
1116+ archive_dir = os.path.join(archive_dir, "Duplicity", "Archives")
1117+else:
1118+ os.environ["XDG_CACHE_HOME"] = os.getenv("XDG_CACHE_HOME", os.path.expanduser("~/.cache"))
1119+ archive_dir = os.path.expandvars("$XDG_CACHE_HOME/duplicity")
1120
1121 # config dir for future use
1122-os.environ["XDG_CONFIG_HOME"] = os.getenv("XDG_CONFIG_HOME", os.path.expanduser("~/.config"))
1123-config_dir = os.path.expandvars("$XDG_CONFIG_HOME/duplicity")
1124+if sys.platform == "win32":
1125+ try:
1126+ config_dir = get_csidl_folder_path(CSIDL_APPDATA)
1127+ except WindowsError:
1128+ config_dir = os.getenv("APPDATA") or os.path.expanduser("~")
1129+ config_dir = os.path.join(archive_dir, "Duplicity", "Config")
1130+else:
1131+ os.environ["XDG_CONFIG_HOME"] = os.getenv("XDG_CONFIG_HOME", os.path.expanduser("~/.config"))
1132+ config_dir = os.path.expandvars("$XDG_CONFIG_HOME/duplicity")
1133
1134 # Restores will try to bring back the state as of the following time.
1135 # If it is None, default to current time.
1136
1137=== modified file 'duplicity/manifest.py'
1138--- duplicity/manifest.py 2010-07-22 19:15:11 +0000
1139+++ duplicity/manifest.py 2010-10-25 15:49:45 +0000
1140@@ -25,6 +25,7 @@
1141
1142 from duplicity import log
1143 from duplicity import globals
1144+from duplicity import path
1145 from duplicity import util
1146
1147 class ManifestError(Exception):
1148@@ -67,7 +68,8 @@
1149 if self.hostname:
1150 self.fh.write("Hostname %s\n" % self.hostname)
1151 if self.local_dirname:
1152- self.fh.write("Localdir %s\n" % Quote(self.local_dirname))
1153+ self.fh.write("Localdir %s\n" % \
1154+ Quote(path.to_posix(self.local_dirname)))
1155 return self
1156
1157 def check_dirinfo(self):
1158@@ -146,7 +148,8 @@
1159 if self.hostname:
1160 result += "Hostname %s\n" % self.hostname
1161 if self.local_dirname:
1162- result += "Localdir %s\n" % Quote(self.local_dirname)
1163+ result += "Localdir %s\n" % \
1164+ Quote(path.to_posix(self.local_dirname))
1165
1166 vol_num_list = self.volume_info_dict.keys()
1167 vol_num_list.sort()
1168@@ -173,6 +176,9 @@
1169 return Unquote(m.group(2))
1170 self.hostname = get_field("hostname")
1171 self.local_dirname = get_field("localdir")
1172+ if self.local_dirname:
1173+ self.local_dirname = path.from_posix(self.local_dirname,
1174+ globals.local_path and globals.local_path.name)
1175
1176 next_vi_string_regexp = re.compile("(^|\\n)(volume\\s.*?)"
1177 "(\\nvolume\\s|$)", re.I | re.S)
1178@@ -221,7 +227,7 @@
1179 Write string version of manifest to given path
1180 """
1181 assert not path.exists()
1182- fout = path.open("w")
1183+ fout = path.open("wb")
1184 fout.write(self.to_string())
1185 assert not fout.close()
1186 path.setdata()
1187
1188=== modified file 'duplicity/patchdir.py'
1189--- duplicity/patchdir.py 2010-07-22 19:15:11 +0000
1190+++ duplicity/patchdir.py 2010-10-25 15:49:45 +0000
1191@@ -19,7 +19,9 @@
1192 # along with duplicity; if not, write to the Free Software Foundation,
1193 # Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
1194
1195+import os
1196 import re #@UnusedImport
1197+import sys
1198 import types
1199 import tempfile
1200
1201@@ -470,6 +472,19 @@
1202 if not isinstance( current_file, file ):
1203 # librsync needs true file
1204 tempfp = tempfile.TemporaryFile( dir=globals.temproot )
1205+ if os.name == "nt":
1206+ # Temp wrapper is unnecessary, file opened O_TEMPORARY
1207+ tempfp = tempfp.file
1208+ elif os.name != "posix" and sys.platform != "cygwin":
1209+ # Note to future developers:
1210+ # librsync needs direct access to the underlying file.
1211+ # On these systems temporary files are wrapper objects
1212+ # The wrapper must be retained until file access is finished
1213+ # so that when it is released the file can be deleted
1214+ raise NotImplementedError(
1215+ "No support for direct access of temporary files " +
1216+ "on this platform")
1217+
1218 misc.copyfileobj( current_file, tempfp )
1219 assert not current_file.close()
1220 tempfp.seek( 0 )
1221
1222=== modified file 'duplicity/path.py'
1223--- duplicity/path.py 2010-07-22 19:15:11 +0000
1224+++ duplicity/path.py 2010-10-25 15:49:45 +0000
1225@@ -41,6 +41,99 @@
1226 _copy_blocksize = 64 * 1024
1227 _tmp_path_counter = 1
1228
1229+def split_all(path):
1230+ """
1231+ Split path into components
1232+
1233+ Invariant: os.path.join(*split_all(path)) == path for a normalized path
1234+
1235+ @rtype list
1236+ @return List of components of path, beginning with "/" or a drive specifier
1237+ if absolute, ending with "" if path ended with a separator
1238+ """
1239+ parts = []
1240+ path, part = os.path.split(path)
1241+
1242+ # Special case for paths which end with a separator so rest is still split
1243+ if part == "":
1244+ parts.append(part)
1245+ path, part = os.path.split(path)
1246+
1247+ while path != "" and part != "":
1248+ parts.append(part)
1249+ path, part = os.path.split(path)
1250+
1251+ # Append root (path) for absolute path, or first relative component (part)
1252+ if path != "":
1253+ parts.append(path)
1254+ else:
1255+ parts.append(part)
1256+
1257+ parts.reverse()
1258+ return parts
1259+
1260+
1261+def from_posix(path, refpath=None):
1262+ """
1263+ Convert a POSIX-style path to the native path representation
1264+
1265+ Copy drive specification (if any) from refpath (if given)
1266+ """
1267+
1268+ # If native path representation is POSIX, no work needs to be done
1269+ if os.path.__name__ == "posixpath":
1270+ return path
1271+
1272+ parts = path.split("/")
1273+ if parts[0] == "":
1274+ parts[0] = os.path.sep
1275+
1276+ if refpath is not None:
1277+ drive = os.path.splitdrive(refpath)[0]
1278+ if drive:
1279+ parts.insert(0, drive)
1280+
1281+ return os.path.join(*parts)
1282+
1283+
1284+def to_posix(path):
1285+ """
1286+ Convert a path from the native path representation to a POSIX-style path
1287+
1288+ The path is broken into components according to split_all, then recombined
1289+ with "/" separating components. Any drive specifier is omitted.
1290+ """
1291+
1292+ # If native path representation is POSIX, no work needs to be done
1293+ if os.path.__name__ == "posixpath":
1294+ return path
1295+
1296+ parts = split_all(path)
1297+ if os.path.isabs(path):
1298+ return "/" + "/".join(parts[1:])
1299+ else:
1300+ return "/".join(parts)
1301+
1302+
1303+def from_url_path(url_path, is_abs=True):
1304+ """
1305+ Convert the <path> component of a file URL into a path in the native path
1306+ representation.
1307+ """
1308+
1309+ parts = url_path.split("/")
1310+ if is_abs:
1311+ if os.path.__name__ == "posixpath":
1312+ parts.insert(0, "/")
1313+ elif os.path.__name__ == "ntpath":
1314+ parts[0] += os.path.sep
1315+ else:
1316+ raise NotImplementedException(
1317+ "Method to create an absolute path not known")
1318+
1319+ return os.path.join(*parts)
1320+
1321+
1322 class StatResult:
1323 """Used to emulate the output of os.stat() and related"""
1324 # st_mode is required by the TarInfo class, but it's unclear how
1325@@ -142,7 +235,7 @@
1326 def get_relative_path(self):
1327 """Return relative path, created from index"""
1328 if self.index:
1329- return "/".join(self.index)
1330+ return os.path.join(*self.index)
1331 else:
1332 return "."
1333
1334@@ -435,7 +528,8 @@
1335 def copy_attribs(self, other):
1336 """Only copy attributes from self to other"""
1337 if isinstance(other, Path):
1338- util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))
1339+ if hasattr(os, "chown"):
1340+ util.maybe_ignore_errors(lambda: os.chown(other.name, self.stat.st_uid, self.stat.st_gid))
1341 util.maybe_ignore_errors(lambda: os.chmod(other.name, self.mode))
1342 util.maybe_ignore_errors(lambda: os.utime(other.name, (time.time(), self.stat.st_mtime)))
1343 other.setdata()
1344@@ -490,7 +584,7 @@
1345 try:
1346 self.stat = os.lstat(self.name)
1347 except OSError, e:
1348- err_string = errno.errorcode[e[0]]
1349+ err_string = errno.errorcode.get(e[0])
1350 if err_string == "ENOENT" or err_string == "ENOTDIR" or err_string == "ELOOP":
1351 self.stat, self.type = None, None # file doesn't exist
1352 self.mode = None
1353@@ -578,11 +672,7 @@
1354 if self.index:
1355 return Path(self.base, self.index[:-1])
1356 else:
1357- components = self.base.split("/")
1358- if len(components) == 2 and not components[0]:
1359- return Path("/") # already in root directory
1360- else:
1361- return Path("/".join(components[:-1]))
1362+ return Path(os.path.dirname(self.base))
1363
1364 def writefileobj(self, fin):
1365 """Copy file object fin to self. Close both when done."""
1366@@ -672,9 +762,7 @@
1367
1368 def get_filename(self):
1369 """Return filename of last component"""
1370- components = self.name.split("/")
1371- assert components and components[-1]
1372- return components[-1]
1373+ return os.path.basename(self.name)
1374
1375 def get_canonical(self):
1376 """
1377@@ -684,12 +772,9 @@
1378 it's harder to remove "..", as "foo/bar/.." is not necessarily
1379 "foo", so we can't use path.normpath()
1380 """
1381- newpath = "/".join(filter(lambda x: x and x != ".",
1382- self.name.split("/")))
1383- if self.name[0] == "/":
1384- return "/" + newpath
1385- elif newpath:
1386- return newpath
1387+ pathparts = filter(lambda x: x and x != ".", split_all(self.name))
1388+ if pathparts:
1389+ return os.path.join(*pathparts)
1390 else:
1391 return "."
1392
1393
1394=== modified file 'duplicity/selection.py'
1395--- duplicity/selection.py 2010-07-22 19:15:11 +0000
1396+++ duplicity/selection.py 2010-10-25 15:49:45 +0000
1397@@ -23,12 +23,15 @@
1398 import re #@UnusedImport
1399 import stat #@UnusedImport
1400
1401-from duplicity.path import * #@UnusedWildImport
1402+from duplicity import path
1403 from duplicity import log #@Reimport
1404 from duplicity import globals #@Reimport
1405 from duplicity import diffdir
1406 from duplicity import util #@Reimport
1407
1408+# For convenience
1409+Path = path.Path
1410+
1411 """Iterate exactly the requested files in a directory
1412
1413 Parses includes and excludes to yield correct files. More
1414@@ -93,6 +96,11 @@
1415 self.rootpath = path
1416 self.prefix = self.rootpath.name
1417
1418+ # Make sure prefix names a directory so prefix matching doesn't
1419+ # match partial directory names
1420+ if os.path.basename(self.prefix) != "":
1421+ self.prefix = os.path.join(self.prefix, "")
1422+
1423 def set_iter(self):
1424 """Initialize generator, prepare to iterate."""
1425 self.rootpath.setdata() # this may have changed since Select init
1426@@ -381,7 +389,7 @@
1427 if not line.startswith(self.prefix):
1428 raise FilePrefixError(line)
1429 line = line[len(self.prefix):] # Discard prefix
1430- index = tuple(filter(lambda x: x, line.split("/"))) # remove empties
1431+ index = tuple(path.split_all(line)) # remove empties
1432 return (index, include)
1433
1434 def filelist_pair_match(self, path, pair):
1435@@ -532,8 +540,7 @@
1436 """
1437 if not filename.startswith(self.prefix):
1438 raise FilePrefixError(filename)
1439- index = tuple(filter(lambda x: x,
1440- filename[len(self.prefix):].split("/")))
1441+ index = tuple(path.split_all(filename[len(self.prefix):]))
1442 return self.glob_get_tuple_sf(index, include)
1443
1444 def glob_get_tuple_sf(self, tuple, include):
1445@@ -614,17 +621,14 @@
1446
1447 def glob_get_prefix_res(self, glob_str):
1448 """Return list of regexps equivalent to prefixes of glob_str"""
1449- glob_parts = glob_str.split("/")
1450+ glob_parts = path.split_all(glob_str)
1451 if "" in glob_parts[1:-1]:
1452 # "" OK if comes first or last, as in /foo/
1453 raise GlobbingError("Consecutive '/'s found in globbing string "
1454 + glob_str)
1455
1456- prefixes = map(lambda i: "/".join(glob_parts[:i+1]),
1457+ prefixes = map(lambda i: os.path.join(*glob_parts[:i+1]),
1458 range(len(glob_parts)))
1459- # we must make exception for root "/", only dir to end in slash
1460- if prefixes[0] == "":
1461- prefixes[0] = "/"
1462 return map(self.glob_to_re, prefixes)
1463
1464 def glob_to_re(self, pat):
1465@@ -638,6 +642,12 @@
1466 by Donovan Baarda.
1467
1468 """
1469+ # Build regex for non-directory separator characters
1470+ notsep = os.path.sep
1471+ if os.path.altsep:
1472+ notsep += os.path.altsep
1473+ notsep = "[^" + notsep.replace("\\", "\\\\") + "]"
1474+
1475 i, n, res = 0, len(pat), ''
1476 while i < n:
1477 c, s = pat[i], pat[i:i+2]
1478@@ -646,9 +656,9 @@
1479 res = res + '.*'
1480 i = i + 1
1481 elif c == '*':
1482- res = res + '[^/]*'
1483+ res = res + notsep + '*'
1484 elif c == '?':
1485- res = res + '[^/]'
1486+ res = res + notsep
1487 elif c == '[':
1488 j = i
1489 if j < n and pat[j] in '!^':
1490
1491=== modified file 'duplicity/tarfile.py'
1492--- duplicity/tarfile.py 2010-07-22 19:15:11 +0000
1493+++ duplicity/tarfile.py 2010-10-25 15:49:45 +0000
1494@@ -1683,8 +1683,10 @@
1495 def set_pwd_dict():
1496 """Set global pwd caching dictionaries uid_dict and uname_dict"""
1497 global uid_dict, uname_dict
1498- assert uid_dict is None and uname_dict is None and pwd
1499+ assert uid_dict is None and uname_dict is None
1500 uid_dict = {}; uname_dict = {}
1501+ if pwd is None:
1502+ return
1503 for entry in pwd.getpwall():
1504 uname = entry[0]; uid = entry[2]
1505 uid_dict[uid] = uname
1506@@ -1702,8 +1704,10 @@
1507
1508 def set_grp_dict():
1509 global gid_dict, gname_dict
1510- assert gid_dict is None and gname_dict is None and grp
1511+ assert gid_dict is None and gname_dict is None
1512 gid_dict = {}; gname_dict = {}
1513+ if grp is None:
1514+ return
1515 for entry in grp.getgrall():
1516 gname = entry[0]; gid = entry[2]
1517 gid_dict[gid] = gname
1518
1519=== removed file 'po/update-pot'
1520--- po/update-pot 2009-09-15 02:13:01 +0000
1521+++ po/update-pot 1970-01-01 00:00:00 +0000
1522@@ -1,4 +0,0 @@
1523-#!/bin/sh
1524-
1525-intltool-update --pot -g duplicity
1526-sed -e 's/^#\. TRANSL:/#./' -i duplicity.pot
1527
1528=== added file 'po/update-pot.py'
1529--- po/update-pot.py 1970-01-01 00:00:00 +0000
1530+++ po/update-pot.py 2010-10-25 15:49:45 +0000
1531@@ -0,0 +1,21 @@
1532+#!/usr/bin/env python
1533+
1534+import os
1535+import re
1536+import sys
1537+import tempfile
1538+
1539+retval = os.system('intltool-update --pot -g duplicity')
1540+if retval != 0:
1541+ # intltool-update failed and already wrote errors, propagate failure
1542+ sys.exit(retval)
1543+
1544+replre = re.compile('^#\. TRANSL:')
1545+with open("duplicity.pot", "rb") as potfile:
1546+ with tempfile.NamedTemporaryFile(delete=False) as tmpfile:
1547+ tmpfilename = tmpfile.name
1548+ for line in potfile:
1549+ tmpfile.write(re.sub(replre, "", line))
1550+
1551+os.remove("duplicity.pot")
1552+os.rename(tmpfilename, "duplicity.pot")

Subscribers

People subscribed via source and target branches

to all changes: