Merge lp:~mandel/ubuntuone-client/illegal_windows_chars into lp:ubuntuone-client

Proposed by Manuel de la Peña
Status: Merged
Approved by: Manuel de la Peña
Approved revision: 1040
Merged at revision: 1007
Proposed branch: lp:~mandel/ubuntuone-client/illegal_windows_chars
Merge into: lp:ubuntuone-client
Diff against target: 763 lines (+377/-150)
3 files modified
tests/platform/windows/test_os_helper.py (+189/-100)
ubuntuone/platform/windows/ipc.py (+2/-3)
ubuntuone/platform/windows/os_helper.py (+186/-47)
To merge this branch: bzr merge lp:~mandel/ubuntuone-client/illegal_windows_chars
Reviewer Review Type Date Requested Status
Natalia Bidart (community) Approve
Alejandro J. Cura (community) Approve
Review via email: mp+65195@code.launchpad.net

Commit message

Fixes: lp:799707 and lp:799722

This branches fixes two different bugs:

1. The correct credentials management tool is used in the windows IPC.
2. Allowed to use illegal chars to be used in the os_helper so that they can be used in the local file system.

Description of the change

This branches fixes two different bugs:

1. The correct credentials management tool is used in the windows IPC.
2. Allowed to use illegal chars to be used in the os_helper so that they can be used in the local file system. In order to be able to get this branch working pywin32 has to be patch (https://sourceforge.net/tracker/index.php?func=detail&aid=3323058&group_id=78018&atid=551954#) with the following:

diff -r 7dce71d174a9 win32/src/win32security.i
--- a/win32/src/win32security.i Sat Jun 18 10:16:06 2011 -0400
+++ b/win32/src/win32security.i Mon Jun 20 14:15:27 2011 +0200
@@ -2108,7 +2108,7 @@
  if (!PyWinObject_AsTCHAR(obFname, &fname))
   goto done;

- if (GetFileSecurity(fname, info, psd, dwSize, &dwSize)) {
+ if (GetFileSecurityW(fname, info, psd, dwSize, &dwSize)) {
   PyErr_SetString(PyExc_RuntimeError, "Can't query for SECURITY_DESCRIPTOR size info?");
   goto done;
  }
@@ -2117,7 +2117,7 @@
   PyErr_SetString(PyExc_MemoryError, "allocating SECURITY_DESCRIPTOR");
   goto done;
  }
- if (!GetFileSecurity(fname, info, psd, dwSize, &dwSize)) {
+ if (!GetFileSecurityW(fname, info, psd, dwSize, &dwSize)) {
   PyWin_SetAPIError("GetFileSecurity");
   goto done;
  }
@@ -2153,7 +2153,7 @@
  PSECURITY_DESCRIPTOR psd;
  if (!PyWinObject_AsSECURITY_DESCRIPTOR(obsd, &psd))
   goto done;
- if (!SetFileSecurity(fname, info, psd)) {
+ if (!SetFileSecurityW(fname, info, psd)) {
   PyWin_SetAPIError("SetFileSecurity");
   goto done;
  }

Once pywin32 has been patched in your system please run the following test on windows:

python C:\Python27\Scripts u1trial tests\platform\test_os_helper.py
python C:\Python27\Scripts u1trial tests\platform\windows\test_os_helper.py
python C:\Python27\Scripts u1trial tests\syncdaemon\test_vm.py

The changes in the code should have not added too much overhead to the IO operations meaning that the test_vm tests should not timeout.

To post a comment you must log in.
Revision history for this message
Natalia Bidart (nataliabidart) wrote :

* Can you please replace:
        for char in WINDOWS_ILLEGAL_CHARS_MAP:
            self.testfile = self.testfile + char
with:
    self.testfile = ''.join(WINDOWS_ILLEGAL_CHARS_MAP)

* typos: "illechal", "illegla"

* can you please use os.path.splitdrive(path) (http://docs.python.org/library/os.path.html#os.path.splitdrive) instead of the custom logic in diff line 392 (and everywhere else where you split the volume letter)?

* can you replace:

    for illegal_char in WINDOWS_ILLEGAL_CHARS_MAP:
        if illegal_char in path:
            return True

with:

    any(c in WINDOWS_ILLEGAL_CHARS_MAP for c in path)

* can you please replace:

        for illegal_char in WINDOWS_ILLEGAL_CHARS_MAP:
            partial = partial.replace(illegal_char,
                                      WINDOWS_ILLEGAL_CHARS_MAP[illegal_char])

with:

       for illegal_char, replacement in WINDOWS_ILLEGAL_CHARS_MAP.iteritems():
            partial = partial.replace(illegal_char, replacement)

* can you please add a logger.exception statement inside the except OSError, e: within set_no_rights? it will be useful to diagnose potential issues from log reports from our users. Same for remove_file, remove_dir, make_dir, open_file, rename, stat_path.

* why are you doing this in abspath? path = path.replace('/', '\\') Seems unnecessary and may hide bugs in our code regarding multiplatform path management.

review: Needs Fixing
Revision history for this message
Alejandro J. Cura (alecu) wrote :

"if e.winerror and e.winerror == 123" should be formulated as """if getattr(e, "winerror", None) == 123""", in case the exception has no winerror attribute. And it should perhaps be moved into a function called let's say "checkWinError123(e)", since it appears at least 8 times through the branch.

calling set_no_rights from inside set_no_rights (in the case of errors) risks the possibility of entering an infinite recursion loop; I suggest moving the three lines inside the try to a small new function, and calling it twice, once from the try, and once from the "if e.winerror...".

typos:
 * "illechal" -> "illegal"
 * "there reason" -> "the reason"
 * spurious spaces have appeared at the end of lines 487 and 496 of the diff
 * "illegla" -> "illegal"

review: Needs Fixing
Revision history for this message
Manuel de la Peña (mandel) wrote :

> * Can you please replace:
> for char in WINDOWS_ILLEGAL_CHARS_MAP:
> self.testfile = self.testfile + char
> with:
> self.testfile = ''.join(WINDOWS_ILLEGAL_CHARS_MAP)
>
> * typos: "illechal", "illegla"
>
> * can you please use os.path.splitdrive(path)
> (http://docs.python.org/library/os.path.html#os.path.splitdrive) instead of
> the custom logic in diff line 392 (and everywhere else where you split the
> volume letter)?
>
> * can you replace:
>
> for illegal_char in WINDOWS_ILLEGAL_CHARS_MAP:
> if illegal_char in path:
> return True
>
> with:
>
> any(c in WINDOWS_ILLEGAL_CHARS_MAP for c in path)
>
> * can you please replace:
>
> for illegal_char in WINDOWS_ILLEGAL_CHARS_MAP:
> partial = partial.replace(illegal_char,
> WINDOWS_ILLEGAL_CHARS_MAP[illegal_char])
>
> with:
>
> for illegal_char, replacement in WINDOWS_ILLEGAL_CHARS_MAP.iteritems():
> partial = partial.replace(illegal_char, replacement)
>
> * can you please add a logger.exception statement inside the except OSError,
> e: within set_no_rights? it will be useful to diagnose potential issues from
> log reports from our users. Same for remove_file, remove_dir, make_dir,
> open_file, rename, stat_path.
>

All the above are fixed.

> * why are you doing this in abspath? path = path.replace('/', '\\') Seems
> unnecessary and may hide bugs in our code regarding multiplatform path
> management.

I have left it because if not there is a vm test that fails... the test uses / for the paths, and while we return the correct path, the assertion fails because on windows we use \\. Since verterok is not here I have left it there with a #TODO asking if it is needed.

Revision history for this message
Alejandro J. Cura (alecu) wrote :

there are still some instances of the old == 123 check, lines 459 and 638 in the diff

review: Needs Fixing
Revision history for this message
Natalia Bidart (nataliabidart) wrote :

* Unless I'm missing something, seems like the custom logic to separate the drive letter has not been replaced by os.path.splitdrive.

* All the logger.exception calls should not add the exception itself to the message string since is automatically added by the logger. So, a sentence like this:

        logger.exception('Goet exception %s when setting path %s to no rights',
                         e, path)

should be:

        logger.exception('Got exception when setting path %s to no rights:', path)

* As Alejandro mentioned, there are some new trailing spaces added by this branch, please remove that.

review: Needs Fixing
Revision history for this message
Alejandro J. Cura (alecu) wrote :

Just a few typos:

"Set the rigths" -> "Set the rights"
            ^

"Get exception when triying" -> "Got exception when trying"
  ^ ^

'Got exception when trun to remove path %s'
                      ^ ???

'Got exception when reyting to remove dir %s'
                      ^ retrying?

"TODO: o we really" -> "TODO: do we really"
       ^

review: Approve
Revision history for this message
Natalia Bidart (nataliabidart) wrote :

Looks good now!

review: Approve
Revision history for this message
Ubuntu One Auto Pilot (otto-pilot) wrote :
Download full text (35.1 KiB)

The attempt to merge lp:~mandel/ubuntuone-client/illegal_windows_chars into lp:ubuntuone-client failed. Below is the output from the failed tests.

/usr/bin/gnome-autogen.sh
checking for autoconf >= 2.53...
  testing autoconf2.50... not found.
  testing autoconf... found 2.67
checking for automake >= 1.10...
  testing automake-1.11... found 1.11.1
checking for libtool >= 1.5...
  testing libtoolize... found 2.2.6b
checking for intltool >= 0.30...
  testing intltoolize... found 0.41.1
checking for pkg-config >= 0.14.0...
  testing pkg-config... found 0.25
checking for gtk-doc >= 1.0...
  testing gtkdocize... found 1.17
Checking for required M4 macros...
Checking for forbidden M4 macros...
Processing ./configure.ac
Running libtoolize...
libtoolize: putting auxiliary files in `.'.
libtoolize: copying file `./ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
Running intltoolize...
Running gtkdocize...
Running aclocal-1.11...
Running autoconf...
Running autoheader...
Running automake-1.11...
Running ./configure --enable-gtk-doc --enable-debug ...
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking dependency style of gcc... gcc3
checking for library containing strerror... none required
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking dependency style of gcc... (cached) gcc3
checking build system type... i686-pc-linux-gnu
checking host system type... i686-pc-linux-gnu
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent...

Revision history for this message
Ubuntu One Auto Pilot (otto-pilot) wrote :

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'tests/platform/windows/test_os_helper.py'
--- tests/platform/windows/test_os_helper.py 2011-02-07 15:34:55 +0000
+++ tests/platform/windows/test_os_helper.py 2011-06-21 00:52:37 +0000
@@ -17,115 +17,204 @@
17"""Specific tests for the os_helper on Windows."""17"""Specific tests for the os_helper on Windows."""
1818
19import os19import os
2020import errno
21from win32api import GetUserName21
22
23from ntsecuritycon import (
24 FILE_ALL_ACCESS,
25 FILE_GENERIC_READ,
26 FILE_GENERIC_WRITE,
27)
28from ubuntuone.platform.windows.os_helper import (22from ubuntuone.platform.windows.os_helper import (
29 _set_file_attributes,23 _remove_windows_illegal_chars,
30 access,24 access,
31 set_no_rights25 listdir,
26 make_dir,
27 open_file,
28 path_exists,
29 remove_dir,
30 remove_file,
31 rename,
32 set_no_rights,
33 set_file_readwrite,
34 stat_path,
35 WINDOWS_ILLEGAL_CHARS_MAP
32)36)
33from contrib.testing.testcase import BaseTwistedTestCase37from contrib.testing.testcase import BaseTwistedTestCase
3438
35class OSWrapperTests(BaseTwistedTestCase):39
36 """Test specific windows implementation details."""40class TestIllegalPaths(BaseTwistedTestCase):
41 """Test all the operations using illegal paths."""
37 42
38 def setUp(self):43 def setUp(self):
39 """Setup for the tests."""44 """Setup for the tests."""
40 super(OSWrapperTests, self).setUp()45 super(TestIllegalPaths, self).setUp()
41 self.basedir = self.mktemp('test_root')46 self.basedir = self.mktemp('test_root')
42 self.testfile = os.path.join(self.basedir, "test_file")47 self.testfile = os.path.join(self.basedir, "test_file")
43 self.fd = open(self.testfile, 'w')48 self.testfile += ''.join(WINDOWS_ILLEGAL_CHARS_MAP)
4449 self.fd = open(_remove_windows_illegal_chars(self.testfile), 'w')
45 def tearDown(self):50
46 """Clean the env for the next test."""51 def test_path_file_exist_yes(self):
47 self.fd.close();52 """Test that the file exists."""
48 self.rmtree(self.basedir)53 self.assertTrue(path_exists(self.testfile))
49 super(OSWrapperTests, self).tearDown()54
5055 def test_path_file_exist_no(self):
51 def test_access_no_rights(self):56 """Test that the file doesn't exist."""
52 """Test when the sid is not present."""57 self.fd.close()
53 # remove all the rights from the test file so that58 os.remove(_remove_windows_illegal_chars(self.testfile))
54 # we cannot read or write59 self.assertFalse(path_exists(self.testfile))
60
61 def test_path_dir_exist_yes(self):
62 """Test that the dir exists."""
63 self.assertTrue(path_exists(self.basedir))
64
65 def test_path_dir_exist_no(self):
66 """Test that the dir doesn't exist."""
67 self.fd.close()
68 os.remove(_remove_windows_illegal_chars(self.testfile))
69 self.assertFalse(path_exists(os.path.join(self.basedir, 'nodir')))
70
71 def test_remove_file(self):
72 """Test the remove file."""
73 self.fd.close()
74 remove_file(self.testfile)
75 self.assertFalse(os.path.exists(
76 _remove_windows_illegal_chars(self.testfile)))
77
78 def test_remove_dir(self):
79 """Test the remove dir."""
80 testdir = os.path.join(self.basedir, 'foodir')
81 os.mkdir(testdir)
82 assert os.path.exists(testdir)
83 remove_dir(testdir)
84 self.assertFalse(path_exists(testdir))
85
86 def test_make_dir_one(self):
87 """Test the make dir with one dir."""
88 testdir = os.path.join(self.basedir, 'foodir')
89 assert not os.path.exists(testdir)
90 make_dir(testdir)
91 self.assertTrue(os.path.exists(testdir))
92
93 def test_make_dir_already_there(self):
94 """Test the make dir with one dir that exists."""
95 self.assertRaises(OSError, make_dir, self.basedir)
96
97 def test_make_dir_recursive_no(self):
98 """Test the make dir with some dirs, not recursive explicit."""
99 testdir = os.path.join(self.basedir, 'foo', 'bar')
100 assert not os.path.exists(testdir)
101 self.assertRaises(OSError, make_dir, testdir)
102
103 def test_make_dir_recursive_yes(self):
104 """Test the make dir with some dirs, recursive."""
105 testdir = os.path.join(self.basedir, 'foo', 'bar')
106 assert not os.path.exists(testdir)
107 make_dir(testdir, recursive=True)
108 self.assertTrue(os.path.exists(testdir))
109
110 def test_open_file_not_there(self):
111 """Open a file that does not exist."""
112 self.assertRaises(IOError, open_file, os.path.join(self.basedir, 'no'))
113
114 def test_open_file_gets_a_fileobject(self):
115 """Open a file, and get a file object."""
116 testfile = os.path.join(self.basedir, 'testfile')
117 open(testfile, 'w').close()
118 f = open_file(testfile)
119 self.assertTrue(isinstance(f, file))
120
121 def test_open_file_read(self):
122 """Open a file, and read."""
123 testfile = os.path.join(self.basedir, 'testfile')
124 with open(testfile, 'w') as fh:
125 fh.write("foo")
126 f = open_file(testfile, 'r')
127 self.assertTrue(f.read(), "foo")
128
129 def test_open_file_write(self):
130 """Open a file, and write."""
131 testfile = os.path.join(self.basedir, 'testfile')
132 with open_file(testfile, 'w') as fh:
133 fh.write("foo")
134 f = open(testfile)
135 self.assertTrue(f.read(), "foo")
136
137 def test_rename_not_there(self):
138 """Rename something that does not exist."""
139 self.assertRaises(OSError, rename,
140 os.path.join(self.basedir, 'no>'), 'foo')
141
142 def test_rename_file(self):
143 """Rename a file."""
144 testfile1 = os.path.join(self.basedir, 'testfile1>')
145 testfile2 = os.path.join(self.basedir, 'testfile2<')
146 open(_remove_windows_illegal_chars(testfile1), 'w').close()
147 rename(testfile1, testfile2)
148 self.assertFalse(os.path.exists(
149 _remove_windows_illegal_chars(testfile1)))
150 self.assertTrue(os.path.exists(
151 _remove_windows_illegal_chars(testfile2)))
152
153 def test_rename_dir(self):
154 """Rename a dir."""
155 testdir1 = os.path.join(self.basedir, 'testdir1>')
156 testdir2 = os.path.join(self.basedir, 'testdir2<')
157 os.mkdir(_remove_windows_illegal_chars(testdir1))
158 rename(testdir1, testdir2)
159 self.assertFalse(os.path.exists(
160 _remove_windows_illegal_chars(testdir1)))
161 self.assertTrue(os.path.exists(
162 _remove_windows_illegal_chars(testdir2)))
163
164 def test_listdir(self):
165 """Return a list of the files in a dir."""
166 open(os.path.join(self.basedir, 'foo'), 'w').close()
167 open(os.path.join(self.basedir, 'bar'), 'w').close()
168 l = listdir(self.basedir)
169 test_file = u'test_file\u200b\u2033\u200b\u200b\u203d\u200b\u200b' +\
170 u'\u26b9\u200b\u200b\u2223\u200b\u200b\u2236\u200b\u200b' +\
171 u'\u2039\u200b\u200b\u2044\u200b\u200b\u203a\u200b'
172 self.assertEqual(sorted(l), ['bar', 'foo', test_file])
173
174 def test_access_rw(self):
175 """Test access on a file with full permission."""
176 self.assertTrue(access(self.testfile))
177
178 def test_access_ro(self):
179 """Test access on a file with read only permission."""
180 fixed_path = _remove_windows_illegal_chars(self.testfile)
181 os.chmod(fixed_path, 0o444)
182 self.assertTrue(access(self.testfile))
183 os.chmod(fixed_path, 0o664)
184
185 def test_access_nothing(self):
186 """Test access on a file with no permission at all."""
55 set_no_rights(self.testfile)187 set_no_rights(self.testfile)
56 self.assertFalse(access(self.testfile))188 self.assertFalse(access(self.testfile))
57189 set_file_readwrite(self.testfile)
58 def test_access_read_write_user(self):190
59 """Test when the user sid has rw rights."""191 def test_stat_normal(self):
60 # set the file to be read and write just by the user192 """Test on a normal file."""
61 groups = {}193 self.assertEqual(os.stat(_remove_windows_illegal_chars(self.testfile)),
62 groups[GetUserName()] = FILE_GENERIC_READ | FILE_GENERIC_WRITE194 stat_path(self.testfile))
63 _set_file_attributes(self.testfile, groups)195
64 self.assertTrue(access(self.testfile))196 def test_stat_no_path(self):
65197 """Test that it raises proper error when no file is there."""
66 def test_access_read_write_everyone(self):198 try:
67 """Test when the everyone sid has rw rights."""199 return stat_path(os.path.join(self.basedir, 'nofile'))
68 groups = {}200 except OSError, e:
69 groups['Everyone'] = FILE_GENERIC_READ | FILE_GENERIC_WRITE201 self.assertEqual(e.errno, errno.ENOENT)
70 _set_file_attributes(self.testfile, groups)202
71 self.assertTrue(access(self.testfile))203 def test_path_exists_file_yes(self):
72 204 """The file is there."""
73 def test_access_write_user_everyone_read(self):205 self.assertTrue(path_exists(self.testfile))
74 """Test when the user sid has w rights."""206
75 groups = {}207 def test_path_exists_file_no(self):
76 groups[GetUserName()] = FILE_GENERIC_WRITE208 """The file is not there."""
77 groups['Everyone'] = FILE_GENERIC_READ209 self.fd.close()
78 _set_file_attributes(self.testfile, groups)210 os.remove(_remove_windows_illegal_chars(self.testfile))
79 self.assertTrue(access(self.testfile))211 self.assertFalse(path_exists(self.testfile))
80212
81 def test_access_write_everyone_user_read(self):213 def test_path_exists_dir_yes(self):
82 """Test when the everyone sid has w rights"""214 """The dir is there."""
83 groups = {}215 self.assertTrue(path_exists(self.basedir))
84 groups[GetUserName()] = FILE_GENERIC_READ 216
85 groups['Everyone'] = FILE_GENERIC_WRITE217 def test_path_exists_dir_no(self):
86 _set_file_attributes(self.testfile, groups)218 """The dir is not there."""
87 self.assertTrue(access(self.testfile))219 self.fd.close()
88220 self.assertFalse(path_exists(os.path.join(self.basedir, 'subdir')))
89 def test_access_write_user_everyone(self):
90 """Test when everyone and user have w rights."""
91 groups = {}
92 groups[GetUserName()] = FILE_GENERIC_WRITE
93 groups['Everyone'] = FILE_GENERIC_WRITE
94 _set_file_attributes(self.testfile, groups)
95 self.assertFalse(access(self.testfile))
96
97 def test_access_read_user(self):
98 """Test when the sid has r rights."""
99 groups = {}
100 groups[GetUserName()] = FILE_GENERIC_READ
101 _set_file_attributes(self.testfile, groups)
102 self.assertTrue(access(self.testfile))
103
104 def test_access_read_everyone(self):
105 """Test when everyone has r rights."""
106 groups = {}
107 groups['Everyone'] = FILE_GENERIC_READ
108 _set_file_attributes(self.testfile, groups)
109 self.assertTrue(access(self.testfile))
110
111 def test_access_read_user_everyone(self):
112 """Test when user and everyone have r rights."""
113 groups = {}
114 groups[GetUserName()] = FILE_GENERIC_READ
115 groups['Everyone'] = FILE_GENERIC_READ
116 _set_file_attributes(self.testfile, groups)
117 self.assertTrue(access(self.testfile))
118
119 def test_access_full_user(self):
120 """Test when the sid has full control."""
121 groups = {}
122 groups[GetUserName()] = FILE_ALL_ACCESS
123 _set_file_attributes(self.testfile, groups)
124 self.assertTrue(access(self.testfile))
125
126 def test_access_full_everyone(self):
127 """test when everyone has full control."""
128 groups = {}
129 groups['Everyone'] = FILE_ALL_ACCESS
130 _set_file_attributes(self.testfile, groups)
131 self.assertTrue(access(self.testfile))
132\ No newline at end of file221\ No newline at end of file
133222
=== modified file 'ubuntuone/platform/windows/ipc.py'
--- ubuntuone/platform/windows/ipc.py 2011-06-15 14:46:04 +0000
+++ ubuntuone/platform/windows/ipc.py 2011-06-21 00:52:37 +0000
@@ -58,8 +58,7 @@
58from twisted.spread.pb import Referenceable, Root, PBServerFactory58from twisted.spread.pb import Referenceable, Root, PBServerFactory
59from twisted.python.failure import Failure59from twisted.python.failure import Failure
60from ubuntuone.syncdaemon.interfaces import IMarker60from ubuntuone.syncdaemon.interfaces import IMarker
61from ubuntuone.credentials import CredentialsManagementTool61from ubuntuone.platform.windows import CredentialsManagementTool
62from ubuntuone.platform import get_creds_proxy
63from ubuntuone.platform.windows.network_manager import NetworkManager62from ubuntuone.platform.windows.network_manager import NetworkManager
64from ubuntuone.syncdaemon.interaction_interfaces import (63from ubuntuone.syncdaemon.interaction_interfaces import (
65 bool_str,64 bool_str,
@@ -1243,7 +1242,7 @@
12431242
1244 def _register_to_signals(self):1243 def _register_to_signals(self):
1245 """Register to the creds signals."""1244 """Register to the creds signals."""
1246 self.creds = get_creds_proxy()1245 self.creds = CredentialsManagementTool()
1247 self.creds.register_to_credentials_found(self.on_credentials_found_cb)1246 self.creds.register_to_credentials_found(self.on_credentials_found_cb)
1248 self.creds.register_to_credentials_not_found(1247 self.creds.register_to_credentials_not_found(
1249 self.on_credentials_error_cb)1248 self.on_credentials_error_cb)
12501249
=== modified file 'ubuntuone/platform/windows/os_helper.py'
--- ubuntuone/platform/windows/os_helper.py 2011-06-17 09:31:15 +0000
+++ ubuntuone/platform/windows/os_helper.py 2011-06-21 00:52:37 +0000
@@ -26,8 +26,8 @@
26from functools import wraps26from functools import wraps
27from contextlib import contextmanager27from contextlib import contextmanager
28from pywintypes import error as PyWinError28from pywintypes import error as PyWinError
29from win32api import MoveFileEx, GetUserName, GetFileAttributes29from win32api import GetUserName, GetFileAttributes
30from win32file import FindFilesW30from win32file import FindFilesW, MoveFileExW
31from win32com.client import Dispatch31from win32com.client import Dispatch
32from pywintypes import error32from pywintypes import error
33from win32con import FILE_ATTRIBUTE_READONLY33from win32con import FILE_ATTRIBUTE_READONLY
@@ -58,11 +58,36 @@
58if sys.platform != 'win32':58if sys.platform != 'win32':
59 WindowsError = None59 WindowsError = None
6060
61
62logger = logging.getLogger('ubuntuone.SyncDaemon.VM')
61platform = 'win32'63platform = 'win32'
6264
63LONG_PATH_PREFIX = '\\\\?\\'65LONG_PATH_PREFIX = '\\\\?\\'
64EVERYONE_GROUP = 'Everyone'66EVERYONE_GROUP = 'Everyone'
65ADMINISTRATORS_GROUP = 'Administrators'67ADMINISTRATORS_GROUP = 'Administrators'
68# a map that contains the illegal chars and their 'utf8' representation on
69# windows so that we can show something to the user
70WINDOWS_ILLEGAL_CHARS_MAP = {
71 '<':u'\N{ZERO WIDTH SPACE}\N{SINGLE LEFT-POINTING ANGLE QUOTATION MARK}\N{ZERO WIDTH SPACE}',
72 '>':u'\N{ZERO WIDTH SPACE}\N{SINGLE RIGHT-POINTING ANGLE QUOTATION MARK}\N{ZERO WIDTH SPACE}',
73 ':':u'\N{ZERO WIDTH SPACE}\N{RATIO}\N{ZERO WIDTH SPACE}',
74 '"':u'\N{ZERO WIDTH SPACE}\N{DOUBLE PRIME}\N{ZERO WIDTH SPACE}',
75 '/':u'\N{ZERO WIDTH SPACE}\N{FRACTION SLASH}\N{ZERO WIDTH SPACE}',
76 '|':u'\N{ZERO WIDTH SPACE}\N{DIVIDES}\N{ZERO WIDTH SPACE}',
77 '?':u'\N{ZERO WIDTH SPACE}\N{INTERROBANG}\N{ZERO WIDTH SPACE}',
78 '*':u'\N{ZERO WIDTH SPACE}\N{SEXTILE}\N{ZERO WIDTH SPACE}'
79}
80LINUX_ILLEGAL_CHARS_MAP = {
81 u'\N{ZERO WIDTH SPACE}\N{SINGLE LEFT-POINTING ANGLE QUOTATION MARK}\N{ZERO WIDTH SPACE}':'<',
82 u'\N{ZERO WIDTH SPACE}\N{SINGLE RIGHT-POINTING ANGLE QUOTATION MARK}\N{ZERO WIDTH SPACE}':'>',
83 u'\N{ZERO WIDTH SPACE}\N{RATIO}\N{ZERO WIDTH SPACE}':':',
84 u'\N{ZERO WIDTH SPACE}\N{DOUBLE PRIME}\N{ZERO WIDTH SPACE}':'"',
85 u'\N{ZERO WIDTH SPACE}\N{FRACTION SLASH}\N{ZERO WIDTH SPACE}':'/',
86 u'\N{ZERO WIDTH SPACE}\N{SET MINUS}\N{ZERO WIDTH SPACE}':'\\',
87 u'\N{ZERO WIDTH SPACE}\N{DIVIDES}\N{ZERO WIDTH SPACE}':'|',
88 u'\N{ZERO WIDTH SPACE}\N{INTERROBANG}\N{ZERO WIDTH SPACE}':'?',
89 u'\N{ZERO WIDTH SPACE}\N{SEXTILE}\N{ZERO WIDTH SPACE}':'*'
90}
6691
6792
68def _int_to_bin(n):93def _int_to_bin(n):
@@ -83,9 +108,79 @@
83 return LookupAccountName('', group_name)[0]108 return LookupAccountName('', group_name)[0]
84109
85110
111def _has_read_mask(number):
112 """Return if the read flag is present."""
113 # get the bin representation of the mask
114 binary = _int_to_bin(number)
115 # there is actual no documentation of this in MSDN but if bt 28 is set,
116 # the mask has full access, more info can be found here:
117 # http://www.iu.hio.no/cfengine/docs/cfengine-NT/node47.html
118 if binary[28] == '1':
119 return True
120 # there is no documentation in MSDN about this, but if bit 0 and 3 are true
121 # we have the read flag, more info can be found here:
122 # http://www.iu.hio.no/cfengine/docs/cfengine-NT/node47.html
123 return binary[0] == '1' and binary[3] == '1'
124
125
126def _check_winerror_123(exception, cb, path, *args, **kwargs):
127 """Checks if an exception is a 123 error and executes the cb."""
128 if getattr(exception, "winerror", None) == 123 :
129 path = _remove_windows_illegal_chars(path)
130 return cb(path, *args, **kwargs)
131 else:
132 raise exception
133
134
135def _is_illegal_path(path):
136 """Return if the path is illegal in the system."""
137 if LONG_PATH_PREFIX in path:
138 path = path.replace(LONG_PATH_PREFIX, '')
139 return any(c in WINDOWS_ILLEGAL_CHARS_MAP
140 for c in os.path.splitdrive(path)[1])
141
142
143def _remove_windows_illegal_chars(path):
144 """Remove illegal chars and return a valid windows path."""
145 if not _is_illegal_path(path):
146 return path
147 prefix = False
148 # do not split the path if you have a long path prefix since the
149 # partial paths will be wrong.
150 if LONG_PATH_PREFIX in path:
151 # remove it and remember to add it
152 path = path.replace(LONG_PATH_PREFIX, '')
153 prefix = True
154 volume, path = os.path.splitdrive(path)
155 partial_paths = path.split('\\')
156 result = []
157 for partial in partial_paths:
158 # ensure that the illegal chars are not there
159 for illegal_char, replacement in WINDOWS_ILLEGAL_CHARS_MAP.iteritems():
160 partial = partial.replace(illegal_char, replacement)
161 result.append(partial)
162 result = os.path.join(*result)
163 if volume:
164 result = volume + '\\' + result
165 if prefix:
166 result = LONG_PATH_PREFIX + result
167 return unicode(result)
168
86def _set_file_attributes(path, groups):169def _set_file_attributes(path, groups):
87 """Set file attributes using the wind32api."""170 """Set file attributes using the wind32api."""
88 security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)171 security_descriptor = None
172 try:
173 security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)
174 except PyWinError, e:
175 logger.exception('Got exception when setting file attributes %s ' +
176 'for path %s', path, groups)
177 #check if the issue was due to an illegal path
178 if getattr(e, "winerror", None) == 123:
179 path = _remove_windows_illegal_chars(path)
180 return _set_file_attributes(path, groups)
181 else:
182 raise
183
89 dacl = ACL()184 dacl = ACL()
90 for group_name in groups:185 for group_name in groups:
91 # set the attributes of the group only if not null186 # set the attributes of the group only if not null
@@ -94,30 +189,15 @@
94 dacl.AddAccessAllowedAceEx(ACL_REVISION,189 dacl.AddAccessAllowedAceEx(ACL_REVISION,
95 CONTAINER_INHERIT_ACE | OBJECT_INHERIT_ACE, groups[group_name],190 CONTAINER_INHERIT_ACE | OBJECT_INHERIT_ACE, groups[group_name],
96 group_sid)191 group_sid)
97 # the dacl has all the info of the dff groups passed in the parameters192 # the dacl has all the info of the diff groups passed in the parameters
98 security_descriptor.SetSecurityDescriptorDacl(1, dacl, 0)193 security_descriptor.SetSecurityDescriptorDacl(1, dacl, 0)
99 SetFileSecurity(path, DACL_SECURITY_INFORMATION, security_descriptor)194 SetFileSecurity(path, DACL_SECURITY_INFORMATION, security_descriptor)
100195
101196
102def _has_read_mask(number):
103 """Return if the read flag is present."""
104 # get the bin representation of the mask
105 binary = _int_to_bin(number)
106 # there is actual no documentation of this in MSDN but if bt 28 is set,
107 # the mask has full access, more info can be found here:
108 # http://www.iu.hio.no/cfengine/docs/cfengine-NT/node47.html
109 if binary[28] == '1':
110 return True
111 # there is no documentation in MSDN about this, but if bit 0 and 3 are true
112 # we have the read flag, more info can be found here:
113 # http://www.iu.hio.no/cfengine/docs/cfengine-NT/node47.html
114 return binary[0] == '1' and binary[3] == '1'
115
116
117def absparam(paths_indexes=[], paths_names=[]):197def absparam(paths_indexes=[], paths_names=[]):
118 """Ensure that the paths are absolute."""198 """Ensure that the paths are absolute."""
119 def decorator(function):199 def decorator(function):
120 """Decorate the funtion to make sure the paths are absolute."""200 """Decorate the function to make sure the paths are absolute."""
121 @wraps(function)201 @wraps(function)
122 def abspath_wrapper(*args, **kwargs):202 def abspath_wrapper(*args, **kwargs):
123 """Set the paths to be absolute."""203 """Set the paths to be absolute."""
@@ -131,7 +211,7 @@
131 kwargs[current_key] = \211 kwargs[current_key] = \
132 os.path.abspath(kwargs[current_key])212 os.path.abspath(kwargs[current_key])
133 except KeyError:213 except KeyError:
134 logging.warn('Patameter %s could not be made a long path',214 logger.warn('Parameter %s could not be made a long path',
135 current_key)215 current_key)
136 return function(*fixed_args, **kwargs)216 return function(*fixed_args, **kwargs)
137 return abspath_wrapper217 return abspath_wrapper
@@ -155,26 +235,36 @@
155 kwargs[current_key] = \235 kwargs[current_key] = \
156 _set_as_long_path(kwargs[current_key])236 _set_as_long_path(kwargs[current_key])
157 except KeyError:237 except KeyError:
158 logging.warn('Patameter %s could not be made a long path',238 logger.warn('Parameter %s could not be made a long path',
159 current_key)239 current_key)
160 return function(*fixed_args, **kwargs)240 return function(*fixed_args, **kwargs)
161 return long_path_wrapper241 return long_path_wrapper
162 return decorator242 return decorator
163243
164244
245def _set_no_rights_helper(path):
246 """Set the rights to be none."""
247 os.chmod(path, 0o000)
248 groups = {}
249 _set_file_attributes(path, groups)
250
251
165@longpath(paths_indexes=[0])252@longpath(paths_indexes=[0])
166def set_no_rights(path):253def set_no_rights(path):
167 # set the groups to be empty which will remove all the rights of the file254 # set the groups to be empty which will remove all the rights of the file
168 os.chmod(path, 0o000)255 try:
169 groups = {}256 _set_no_rights_helper(path)
170 _set_file_attributes(path, groups)257 except OSError, e:
258 logger.exception('Got exception when setting path %s to no rights',
259 path)
260 return _check_winerror_123(e, _set_no_rights_helper, path)
171261
172262
173@longpath(paths_indexes=[0])263@longpath(paths_indexes=[0])
174def set_file_readonly(path):264def set_file_readonly(path):
175 """Change path permissions to readonly in a file."""265 """Change path permissions to readonly in a file."""
176 # we use the win32 api because chmod just sets the readonly flag and266 # we use the win32 api because chmod just sets the readonly flag and
177 # we want to have imore control over the permissions267 # we want to have more control over the permissions
178 groups = {}268 groups = {}
179 groups[ADMINISTRATORS_GROUP] = FILE_GENERIC_READ | FILE_GENERIC_WRITE269 groups[ADMINISTRATORS_GROUP] = FILE_GENERIC_READ | FILE_GENERIC_WRITE
180 groups[GetUserName()] = FILE_GENERIC_READ270 groups[GetUserName()] = FILE_GENERIC_READ
@@ -191,7 +281,12 @@
191 groups[GetUserName()] = FILE_ALL_ACCESS281 groups[GetUserName()] = FILE_ALL_ACCESS
192 # the above equals more or less to 0774282 # the above equals more or less to 0774
193 _set_file_attributes(path, groups)283 _set_file_attributes(path, groups)
194 os.chmod(path, stat.S_IWRITE)284 try:
285 os.chmod(path, stat.S_IWRITE)
286 except OSError, e:
287 logger.exception('Get exception when trying to set readwrite ' +
288 'path %s', path)
289 return _check_winerror_123(e, os.chmod, path, stat.S_IWRITE)
195290
196291
197@longpath(paths_indexes=[0])292@longpath(paths_indexes=[0])
@@ -229,7 +324,7 @@
229@contextmanager324@contextmanager
230@longpath(paths_indexes=[0])325@longpath(paths_indexes=[0])
231def allow_writes(path):326def allow_writes(path):
232 """A very simple context manager to allow writting in RO dirs."""327 """A very simple context manager to allow witting in RO dirs."""
233 # get the old dacl of the file so that we can reset it back when done328 # get the old dacl of the file so that we can reset it back when done
234 security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)329 security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)
235 old_dacl = security_descriptor.GetSecurityDescriptorDacl()330 old_dacl = security_descriptor.GetSecurityDescriptorDacl()
@@ -243,18 +338,28 @@
243@longpath(paths_indexes=[0])338@longpath(paths_indexes=[0])
244def remove_file(path):339def remove_file(path):
245 """Remove a file."""340 """Remove a file."""
246 os.remove(path)341 try:
342 os.remove(path)
343 except OSError, e:
344 logger.exception('Got exception when trying to remove path %s', path)
345 return _check_winerror_123(e, os.remove, path)
247346
248347
249@longpath(paths_indexes=[0])348@longpath(paths_indexes=[0])
250def remove_dir(path):349def remove_dir(path):
251 """Remove a dir."""350 """Remove a dir."""
252 os.rmdir(path)351 try:
352 os.rmdir(path)
353 except OSError, e:
354 logger.exception('Got exception when trying to remove dir %s', path)
355 return _check_winerror_123(e, os.rmdir, path)
253356
254357
255@longpath(paths_indexes=[0])358@longpath(paths_indexes=[0])
256def path_exists(path):359def path_exists(path):
257 """Return if the path exists."""360 """Return if the path exists."""
361 if _is_illegal_path(path):
362 path = _remove_windows_illegal_chars(path)
258 return os.path.exists(path)363 return os.path.exists(path)
259364
260365
@@ -264,37 +369,60 @@
264 if recursive:369 if recursive:
265 try:370 try:
266 os.makedirs(path)371 os.makedirs(path)
267 except WindowsError, e:372 except OSError, e:
268 raise OSError(str(e))373 logger.exception('Got exception when trying to make dirs %s', path)
374 return _check_winerror_123(e, os.makedirs, path)
269 else:375 else:
270 try:376 try:
271 os.mkdir(path)377 os.mkdir(path)
272 except WindowsError, e:378 except OSError, e:
273 raise OSError(str(e))379 logger.exception('Got exception when trying to make dirs %s', path)
380 return _check_winerror_123(e, os.mkdir, path)
274381
275382
276@longpath(paths_indexes=[0])383@longpath(paths_indexes=[0])
277def open_file(path, mode='r'):384def open_file(path, mode='r'):
278 """Open a file."""385 """Open a file."""
279 return open(path, mode)386 try:
387 return open(path, mode)
388 except OSError, e:
389 logger.exception('Got exception when trying to open path %s', path)
390 return _check_winerror_123(e, open, path, mode)
280391
281392
282@absparam(paths_indexes=[0, 1])393@absparam(paths_indexes=[0, 1])
283@longpath(paths_indexes=[0, 1])394@longpath(paths_indexes=[0, 1])
284def rename(path_from, path_to):395def rename(path_from, path_to):
285 """Rename a file or directory."""396 """Rename a file or directory."""
286 # to ensure the same behavious as on linux, use the MoveEx function from397 # to ensure the same behaviors as on linux, use the MoveEx function from
287 # win32 which will allow to replace the destionation path if exists and398 # win32 which will allow to replace the destination path if exists and
288 # the user has the rights see:399 # the user has the rights see:
289 # http://msdn.microsoft.com/en-us/library/aa365240(v=vs.85).aspx400 # http://msdn.microsoft.com/en-us/library/aa365240(v=vs.85).aspx
290 try:401 try:
291 MoveFileEx(402 MoveFileExW(
292 path_from, path_to,403 path_from, path_to,
293 MOVEFILE_COPY_ALLOWED |404 MOVEFILE_COPY_ALLOWED |
294 MOVEFILE_REPLACE_EXISTING |405 MOVEFILE_REPLACE_EXISTING |
295 MOVEFILE_WRITE_THROUGH)406 MOVEFILE_WRITE_THROUGH)
296 except PyWinError, e:407 except PyWinError, e:
297 raise OSError(str(e))408 logger.exception('Got exception when trying to rename from ' +
409 '%s to %s', path_from, path_to)
410 #check if the issue was due to an illegal path
411 if getattr(e, "winerror", None) == 123:
412 path_from = _remove_windows_illegal_chars(path_from)
413 path_to = _remove_windows_illegal_chars(path_to)
414 try:
415 MoveFileExW(
416 path_from, path_to,
417 MOVEFILE_COPY_ALLOWED |
418 MOVEFILE_REPLACE_EXISTING |
419 MOVEFILE_WRITE_THROUGH)
420 except PyWinError, e:
421 logger.exception('Got exception when trying to rename' +
422 ' from %s to %s', path_from, path_to)
423 raise OSError(str(e))
424 else:
425 raise OSError(str(e))
298426
299427
300@absparam(paths_indexes=[0, 1])428@absparam(paths_indexes=[0, 1])
@@ -343,7 +471,7 @@
343@longpath(paths_indexes=[0])471@longpath(paths_indexes=[0])
344def listdir(directory):472def listdir(directory):
345 """List a directory."""473 """List a directory."""
346 # os.listdir combines / and \ in its search which breacks things on474 # os.listdir combines / and \ in its search which breaks things on
347 # windows, we use the win32 api instead.475 # windows, we use the win32 api instead.
348 result = []476 result = []
349 for file_info in FindFilesW(os.path.join(directory, '*.*')):477 for file_info in FindFilesW(os.path.join(directory, '*.*')):
@@ -357,8 +485,12 @@
357@longpath(paths_indexes=[0])485@longpath(paths_indexes=[0])
358def access(path):486def access(path):
359 """Return if the path is at least readable."""487 """Return if the path is at least readable."""
488 # lets consider the access on an illegal path to be a special case
489 # since that will only occur in the case where the user created the path
360 # for a file to be readable it has to be readable either by the user or490 # for a file to be readable it has to be readable either by the user or
361 # by the everyone group491 # by the everyone group
492 if _is_illegal_path(path):
493 path = _remove_windows_illegal_chars(path)
362 if not os.path.exists(path):494 if not os.path.exists(path):
363 return False495 return False
364 security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)496 security_descriptor = GetFileSecurity(path, DACL_SECURITY_INFORMATION)
@@ -380,13 +512,14 @@
380 # in our case we can ignore this since we are looking for visible512 # in our case we can ignore this since we are looking for visible
381 # users513 # users
382 continue514 continue
383 return (GetUserName() in accounts or EVERYONE_GROUP in accounts)515 return (GetUserName() in accounts or EVERYONE_GROUP in accounts) and\
516 os.access(path, os.R_OK)
384517
385518
386@longpath(paths_indexes=[0])519@longpath(paths_indexes=[0])
387def can_write(path):520def can_write(path):
388 """Return if the path can be written to."""521 """Return if the path can be written to."""
389 file_att = GetFileAttributes(path) 522 file_att = GetFileAttributes(path)
390 if file_att and FILE_ATTRIBUTE_READONLY:523 if file_att and FILE_ATTRIBUTE_READONLY:
391 return False524 return False
392 return True525 return True
@@ -395,14 +528,18 @@
395def stat_path(path, look_for_link=True):528def stat_path(path, look_for_link=True):
396 """Return stat info about a path."""529 """Return stat info about a path."""
397 # if the path end with .lnk, that means we are dealing with a link530 # if the path end with .lnk, that means we are dealing with a link
398 # and we should return the stat of th target path531 # and we should return the stat of the target path
399 if path.endswith('.lnk'):532 if path.endswith('.lnk'):
400 shell = Dispatch('WScript.Shell')533 shell = Dispatch('WScript.Shell')
401 shortcut = shell.CreateShortCut(path)534 shortcut = shell.CreateShortCut(path)
402 path = shortcut.Targetpath535 path = shortcut.Targetpath
403 if look_for_link and os.path.exists(path + '.lnk'):536 if look_for_link and os.path.exists(path + '.lnk'):
404 return stat_path(path + '.lnk')537 return stat_path(path + '.lnk')
405 return os.lstat(path)538 try:
539 return os.lstat(path)
540 except OSError, e:
541 logger.exception('Got exception when trying to stat path %s', path)
542 return _check_winerror_123(e, stat_path, path, look_for_link)
406543
407544
408def move_to_trash(path):545def move_to_trash(path):
@@ -415,7 +552,7 @@
415def set_application_name(app_name):552def set_application_name(app_name):
416 """Set the app name."""553 """Set the app name."""
417 # there is not way to do this on windows. The name will be correct when554 # there is not way to do this on windows. The name will be correct when
418 # executed from a bundle .exe otherwhise it will be python555 # executed from a bundle .exe otherwise it will be python
419556
420557
421def is_root():558def is_root():
@@ -432,7 +569,7 @@
432569
433570
434def validate_path_from_unix(path):571def validate_path_from_unix(path):
435 """Get a unix path and return it so that it cna be used."""572 """Get a unix path and return it so that it can be used."""
436 path = path.replace('/', '\\')573 path = path.replace('/', '\\')
437 return path574 return path
438575
@@ -449,7 +586,9 @@
449586
450def abspath(path):587def abspath(path):
451 """Return an abspath."""588 """Return an abspath."""
452 abs = os.path.abspath(path)589 # TODO: do we really need to do this, are the tests broken?
590 path = path.replace('/', '\\')
591 abs = os.path.abspath(_remove_windows_illegal_chars(path))
453 if not LONG_PATH_PREFIX in abs:592 if not LONG_PATH_PREFIX in abs:
454 abs = LONG_PATH_PREFIX + abs593 abs = LONG_PATH_PREFIX + abs
455 return abs594 return abs
456\ No newline at end of file595\ No newline at end of file

Subscribers

People subscribed via source and target branches