Merge lp:~diegosarmentero/ubuntuone-client/darwin2-fsevents into lp:ubuntuone-client
- darwin2-fsevents
- Merge into trunk
Status: | Merged | ||||
---|---|---|---|---|---|
Approved by: | Alejandro J. Cura | ||||
Approved revision: | 1268 | ||||
Merged at revision: | 1267 | ||||
Proposed branch: | lp:~diegosarmentero/ubuntuone-client/darwin2-fsevents | ||||
Merge into: | lp:ubuntuone-client | ||||
Prerequisite: | lp:~diegosarmentero/ubuntuone-client/darwin-fsevents-1 | ||||
Diff against target: |
912 lines (+376/-280) 4 files modified
tests/platform/filesystem_notifications/test_windows.py (+3/-3) ubuntuone/platform/filesystem_notifications/__init__.py (+1/-0) ubuntuone/platform/filesystem_notifications/common.py (+64/-277) ubuntuone/platform/filesystem_notifications/windows.py (+308/-0) |
||||
To merge this branch: | bzr merge lp:~diegosarmentero/ubuntuone-client/darwin2-fsevents | ||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Alejandro J. Cura (community) | Approve | ||
Manuel de la Peña (community) | Approve | ||
Review via email: mp+111427@code.launchpad.net |
Commit message
- Refactoring windows fsevents implementation, to be usable from the future darwin one (LP: #1013323).
Description of the change
- 1263. By Diego Sarmentero
-
Updating branch
Diego Sarmentero (diegosarmentero) wrote : | # |
> The implementation in add_watches_
> windows, so it needs to be moved to windows.py
>
Moved to windows, we will need to implement later for darwin something here, but not just returning True as in windows.
> ---
>
> FILESYSTEM_
> filesystem_
>
This is not broken, the mask is assigned in windows.py, and for darwin i'm using it as None, so i can share more common code in common.py.
> ---
>
> All of this would break on darwin:
> is_valid_
> is_valid_os_path = None
> os_path = windowspath = None
>
> So everything under:
> elif sys.platform == "darwin":
>
> should come on a following branch.
>
> ---
>
> Please rename "reference" to something less vague, and move each definition to
> the platform specific files.
Done
Alejandro J. Cura (alecu) wrote : | # |
> > FILESYSTEM_
> > filesystem_
> >
>
> This is not broken, the mask is assigned in windows.py, and for darwin i'm
> using it as None, so i can share more common code in common.py.
You are right! Sorry about that :P
Alejandro J. Cura (alecu) wrote : | # |
This comment is *Very* specific to windows:
# On windows there is no need to add the watches to the ancestors
# so we will always return true. The reason for this is that the
# handles that we open stop the user from renaming the ancestors of
# the UDF, for a user to do that he has to unsync the udf first
This too:
For Windows:
Also, they must not be literal paths, that is the \\?\ prefix should not be
in the path.
Both should be in windows.py.
- 1264. By Diego Sarmentero
-
merge
- 1265. By Diego Sarmentero
-
updates
- 1266. By Diego Sarmentero
-
adding not implemented error
- 1267. By Diego Sarmentero
-
moving event codes to filesystem notification init
Diego Sarmentero (diegosarmentero) wrote : | # |
Branch fixed.
I added the event codes in the filesystem_
Also i remove the if sys.platform == "win32": import decorators
on common, and in the future branch, when i add the darwin part, i'll make os_helper import the proper decorator based on the platform as it should be.
Manuel de la Peña (mandel) : | # |
Alejandro J. Cura (alecu) wrote : | # |
This will probably break on darwin, where fs paths are utf-8 bytes, so it should go from common.py to windows.py:
473 if not isinstance(path, unicode):
----
The comment that starts with:
# We are using this on windows and darwin.
# For windows the logic is as follow:
# ....
is still crap. The whole 9 lines of it, so it's not completely your fault, but it's crap.
Let's rewrite it with something simpler like this:
# We need to manually check if the path is a folder, because
# neither ReadDirectoryCh
Also the docstring in that function lies about "update the local subdir list".
---
- 1268. By Diego Sarmentero
-
fixing comment
Diego Sarmentero (diegosarmentero) wrote : | # |
> This will probably break on darwin, where fs paths are utf-8 bytes, so it
> should go from common.py to windows.py:
>
> 473 if not isinstance(path, unicode):
>
> ----
>
> The comment that starts with:
> # We are using this on windows and darwin.
> # For windows the logic is as follow:
> # ....
>
> is still crap. The whole 9 lines of it, so it's not completely your fault, but
> it's crap.
> Let's rewrite it with something simpler like this:
>
> # We need to manually check if the path is a folder, because
> # neither ReadDirectoryCh
>
> Also the docstring in that function lies about "update the local subdir list".
>
> ---
The comment has been fixed, and the isinstance thing is fixed in the next branch which involves actually the darwin things as we talk.
Alejandro J. Cura (alecu) wrote : | # |
Code looks fine. Ran tests on Precise and windows, and they all pass.
Preview Diff
1 | === modified file 'tests/platform/filesystem_notifications/test_windows.py' | |||
2 | --- tests/platform/filesystem_notifications/test_windows.py 2012-05-14 21:24:24 +0000 | |||
3 | +++ tests/platform/filesystem_notifications/test_windows.py 2012-06-26 19:23:19 +0000 | |||
4 | @@ -51,6 +51,7 @@ | |||
5 | 51 | IN_OPEN, | 51 | IN_OPEN, |
6 | 52 | ) | 52 | ) |
7 | 53 | from ubuntuone.platform.filesystem_notifications import ( | 53 | from ubuntuone.platform.filesystem_notifications import ( |
8 | 54 | common, | ||
9 | 54 | windows as filesystem_notifications, | 55 | windows as filesystem_notifications, |
10 | 55 | ) | 56 | ) |
11 | 56 | from ubuntuone.platform.filesystem_notifications.windows import ( | 57 | from ubuntuone.platform.filesystem_notifications.windows import ( |
12 | @@ -65,12 +66,11 @@ | |||
13 | 65 | FILE_NOTIFY_CHANGE_LAST_WRITE, | 66 | FILE_NOTIFY_CHANGE_LAST_WRITE, |
14 | 66 | FILE_NOTIFY_CHANGE_SECURITY, | 67 | FILE_NOTIFY_CHANGE_SECURITY, |
15 | 67 | FILE_NOTIFY_CHANGE_LAST_ACCESS, | 68 | FILE_NOTIFY_CHANGE_LAST_ACCESS, |
16 | 68 | WINDOWS_ACTIONS_NAMES, | ||
17 | 69 | ) | 69 | ) |
18 | 70 | 70 | ||
19 | 71 | #create a rever mapping to use it in the tests. | 71 | #create a rever mapping to use it in the tests. |
20 | 72 | REVERSE_WINDOWS_ACTIONS = {} | 72 | REVERSE_WINDOWS_ACTIONS = {} |
22 | 73 | for key, value in filesystem_notifications.WINDOWS_ACTIONS.iteritems(): | 73 | for key, value in common.COMMON_ACTIONS.iteritems(): |
23 | 74 | REVERSE_WINDOWS_ACTIONS[value] = key | 74 | REVERSE_WINDOWS_ACTIONS[value] = key |
24 | 75 | 75 | ||
25 | 76 | 76 | ||
26 | @@ -147,7 +147,7 @@ | |||
27 | 147 | # group all events in a single lists which is not what the COM API | 147 | # group all events in a single lists which is not what the COM API |
28 | 148 | # does. | 148 | # does. |
29 | 149 | str_events = [ | 149 | str_events = [ |
31 | 150 | (WINDOWS_ACTIONS_NAMES[action], path) for action, path in | 150 | (common.COMMON_ACTIONS_NAMES[action], path) for action, path in |
32 | 151 | events] | 151 | events] |
33 | 152 | self.raw_events.append(str_events) | 152 | self.raw_events.append(str_events) |
34 | 153 | return events | 153 | return events |
35 | 154 | 154 | ||
36 | === modified file 'ubuntuone/platform/filesystem_notifications/__init__.py' | |||
37 | --- ubuntuone/platform/filesystem_notifications/__init__.py 2012-05-16 16:24:23 +0000 | |||
38 | +++ ubuntuone/platform/filesystem_notifications/__init__.py 2012-06-26 19:23:19 +0000 | |||
39 | @@ -32,6 +32,7 @@ | |||
40 | 32 | 32 | ||
41 | 33 | 33 | ||
42 | 34 | if sys.platform == "win32": | 34 | if sys.platform == "win32": |
43 | 35 | EVENT_CODES = [1, 2, 3, 4, 5] | ||
44 | 35 | from ubuntuone.platform.filesystem_notifications import windows | 36 | from ubuntuone.platform.filesystem_notifications import windows |
45 | 36 | FilesystemMonitor = windows.FilesystemMonitor | 37 | FilesystemMonitor = windows.FilesystemMonitor |
46 | 37 | _GeneralINotifyProcessor = windows.NotifyProcessor | 38 | _GeneralINotifyProcessor = windows.NotifyProcessor |
47 | 38 | 39 | ||
48 | === renamed file 'ubuntuone/platform/filesystem_notifications/windows.py' => 'ubuntuone/platform/filesystem_notifications/common.py' | |||
49 | --- ubuntuone/platform/filesystem_notifications/windows.py 2012-05-14 21:24:24 +0000 | |||
50 | +++ ubuntuone/platform/filesystem_notifications/common.py 2012-06-26 19:23:19 +0000 | |||
51 | @@ -26,41 +26,14 @@ | |||
52 | 26 | # do not wish to do so, delete this exception statement from your | 26 | # do not wish to do so, delete this exception statement from your |
53 | 27 | # version. If you delete this exception statement from all source | 27 | # version. If you delete this exception statement from all source |
54 | 28 | # files in the program, then also delete it here. | 28 | # files in the program, then also delete it here. |
56 | 29 | """File notifications on windows.""" | 29 | """Generic File notifications.""" |
57 | 30 | 30 | ||
58 | 31 | import logging | 31 | import logging |
59 | 32 | import os | 32 | import os |
60 | 33 | 33 | ||
62 | 34 | from uuid import uuid4 | 34 | from twisted.internet import defer |
63 | 35 | 35 | ||
92 | 36 | from twisted.internet import defer, reactor | 36 | from ubuntuone.platform.filesystem_notifications import EVENT_CODES |
65 | 37 | from twisted.python.failure import Failure | ||
66 | 38 | from pywintypes import OVERLAPPED | ||
67 | 39 | from win32api import CloseHandle | ||
68 | 40 | from win32con import ( | ||
69 | 41 | FILE_SHARE_READ, | ||
70 | 42 | FILE_SHARE_WRITE, | ||
71 | 43 | FILE_FLAG_BACKUP_SEMANTICS, | ||
72 | 44 | FILE_NOTIFY_CHANGE_FILE_NAME, | ||
73 | 45 | FILE_NOTIFY_CHANGE_DIR_NAME, | ||
74 | 46 | FILE_NOTIFY_CHANGE_ATTRIBUTES, | ||
75 | 47 | FILE_NOTIFY_CHANGE_SIZE, | ||
76 | 48 | FILE_NOTIFY_CHANGE_LAST_WRITE, | ||
77 | 49 | FILE_NOTIFY_CHANGE_SECURITY, | ||
78 | 50 | OPEN_EXISTING) | ||
79 | 51 | from win32file import ( | ||
80 | 52 | AllocateReadBuffer, | ||
81 | 53 | CreateFileW, | ||
82 | 54 | GetOverlappedResult, | ||
83 | 55 | ReadDirectoryChangesW, | ||
84 | 56 | FILE_FLAG_OVERLAPPED, | ||
85 | 57 | FILE_NOTIFY_INFORMATION) | ||
86 | 58 | from win32event import ( | ||
87 | 59 | CreateEvent, | ||
88 | 60 | INFINITE, | ||
89 | 61 | SetEvent, | ||
90 | 62 | WaitForMultipleObjects, | ||
91 | 63 | WAIT_OBJECT_0) | ||
93 | 64 | from ubuntuone.platform.filesystem_notifications.pyinotify_agnostic import ( | 37 | from ubuntuone.platform.filesystem_notifications.pyinotify_agnostic import ( |
94 | 65 | Event, | 38 | Event, |
95 | 66 | WatchManagerError, | 39 | WatchManagerError, |
96 | @@ -75,40 +48,32 @@ | |||
97 | 75 | IN_MOVED_FROM, | 48 | IN_MOVED_FROM, |
98 | 76 | IN_MOVED_TO, | 49 | IN_MOVED_TO, |
99 | 77 | IN_MODIFY) | 50 | IN_MODIFY) |
100 | 51 | |||
101 | 52 | from ubuntuone import logger | ||
102 | 53 | |||
103 | 78 | from ubuntuone.platform.os_helper.windows import ( | 54 | from ubuntuone.platform.os_helper.windows import ( |
104 | 79 | is_valid_syncdaemon_path, | 55 | is_valid_syncdaemon_path, |
108 | 80 | is_valid_windows_path, | 56 | is_valid_windows_path as is_valid_os_path, |
109 | 81 | get_syncdaemon_valid_path, | 57 | windowspath as os_path, |
107 | 82 | windowspath, | ||
110 | 83 | ) | 58 | ) |
123 | 84 | from ubuntuone import logger | 59 | |
124 | 85 | 60 | # a map between the few events that we have on common platforms and those | |
113 | 86 | # our logging level | ||
114 | 87 | TRACE = logger.TRACE | ||
115 | 88 | |||
116 | 89 | # constant found in the msdn documentation: | ||
117 | 90 | # http://msdn.microsoft.com/en-us/library/ff538834(v=vs.85).aspx | ||
118 | 91 | FILE_LIST_DIRECTORY = 0x0001 | ||
119 | 92 | FILE_NOTIFY_CHANGE_LAST_ACCESS = 0x00000020 | ||
120 | 93 | FILE_NOTIFY_CHANGE_CREATION = 0x00000040 | ||
121 | 94 | |||
122 | 95 | # a map between the few events that we have on windows and those | ||
125 | 96 | # found in pyinotify | 61 | # found in pyinotify |
132 | 97 | WINDOWS_ACTIONS = { | 62 | COMMON_ACTIONS = { |
133 | 98 | 1: IN_CREATE, | 63 | EVENT_CODES[0]: IN_CREATE, |
134 | 99 | 2: IN_DELETE, | 64 | EVENT_CODES[1]: IN_DELETE, |
135 | 100 | 3: IN_MODIFY, | 65 | EVENT_CODES[2]: IN_MODIFY, |
136 | 101 | 4: IN_MOVED_FROM, | 66 | EVENT_CODES[3]: IN_MOVED_FROM, |
137 | 102 | 5: IN_MOVED_TO, | 67 | EVENT_CODES[4]: IN_MOVED_TO, |
138 | 103 | } | 68 | } |
139 | 104 | 69 | ||
140 | 105 | # a map of the actions to names so that we have better logs. | 70 | # a map of the actions to names so that we have better logs. |
147 | 106 | WINDOWS_ACTIONS_NAMES = { | 71 | COMMON_ACTIONS_NAMES = { |
148 | 107 | 1: 'IN_CREATE', | 72 | EVENT_CODES[0]: 'IN_CREATE', |
149 | 108 | 2: 'IN_DELETE', | 73 | EVENT_CODES[1]: 'IN_DELETE', |
150 | 109 | 3: 'IN_MODIFY', | 74 | EVENT_CODES[2]: 'IN_MODIFY', |
151 | 110 | 4: 'IN_MOVED_FROM', | 75 | EVENT_CODES[3]: 'IN_MOVED_FROM', |
152 | 111 | 5: 'IN_MOVED_TO', | 76 | EVENT_CODES[4]: 'IN_MOVED_TO', |
153 | 112 | } | 77 | } |
154 | 113 | 78 | ||
155 | 114 | # translates quickly the event and it's is_dir state to our standard events | 79 | # translates quickly the event and it's is_dir state to our standard events |
156 | @@ -125,17 +90,8 @@ | |||
157 | 125 | IN_MOVED_TO: 'FS_FILE_CREATE', | 90 | IN_MOVED_TO: 'FS_FILE_CREATE', |
158 | 126 | IN_MOVED_TO | IN_ISDIR: 'FS_DIR_CREATE'} | 91 | IN_MOVED_TO | IN_ISDIR: 'FS_DIR_CREATE'} |
159 | 127 | 92 | ||
171 | 128 | # the default mask to be used in the watches added by the FilesystemMonitor | 93 | # our logging level |
172 | 129 | # class | 94 | TRACE = logger.TRACE |
162 | 130 | FILESYSTEM_MONITOR_MASK = FILE_NOTIFY_CHANGE_FILE_NAME | \ | ||
163 | 131 | FILE_NOTIFY_CHANGE_DIR_NAME | \ | ||
164 | 132 | FILE_NOTIFY_CHANGE_ATTRIBUTES | \ | ||
165 | 133 | FILE_NOTIFY_CHANGE_SIZE | \ | ||
166 | 134 | FILE_NOTIFY_CHANGE_LAST_WRITE | \ | ||
167 | 135 | FILE_NOTIFY_CHANGE_SECURITY | \ | ||
168 | 136 | FILE_NOTIFY_CHANGE_LAST_ACCESS | ||
169 | 137 | |||
170 | 138 | THREADPOOL_MAX = 20 | ||
173 | 139 | 95 | ||
174 | 140 | 96 | ||
175 | 141 | # The implementation of the code that is provided as the pyinotify substitute | 97 | # The implementation of the code that is provided as the pyinotify substitute |
176 | @@ -144,15 +100,11 @@ | |||
177 | 144 | 100 | ||
178 | 145 | def __init__(self, watch_descriptor, path, mask, auto_add, processor, | 101 | def __init__(self, watch_descriptor, path, mask, auto_add, processor, |
179 | 146 | buf_size=8192): | 102 | buf_size=8192): |
182 | 147 | super(Watch, self).__init__() | 103 | self.log = logging.getLogger('ubuntuone.SyncDaemon.platform.common.' + |
181 | 148 | self.log = logging.getLogger('ubuntuone.SyncDaemon.platform.windows.' + | ||
183 | 149 | 'filesystem_notifications.Watch') | 104 | 'filesystem_notifications.Watch') |
184 | 150 | self.log.setLevel(TRACE) | 105 | self.log.setLevel(TRACE) |
185 | 151 | self._processor = processor | 106 | self._processor = processor |
186 | 152 | self._buf_size = buf_size | 107 | self._buf_size = buf_size |
187 | 153 | self._wait_stop = CreateEvent(None, 0, 0, None) | ||
188 | 154 | self._overlapped = OVERLAPPED() | ||
189 | 155 | self._overlapped.hEvent = CreateEvent(None, 0, 0, None) | ||
190 | 156 | self._watching = False | 108 | self._watching = False |
191 | 157 | self._descriptor = watch_descriptor | 109 | self._descriptor = watch_descriptor |
192 | 158 | self._auto_add = auto_add | 110 | self._auto_add = auto_add |
193 | @@ -170,36 +122,25 @@ | |||
194 | 170 | path += os.path.sep | 122 | path += os.path.sep |
195 | 171 | self._path = os.path.abspath(path) | 123 | self._path = os.path.abspath(path) |
196 | 172 | self._mask = mask | 124 | self._mask = mask |
197 | 173 | # this deferred is fired when the watch has started monitoring | ||
198 | 174 | # a directory from a thread | ||
199 | 175 | self._watch_started_deferred = defer.Deferred() | ||
200 | 176 | # and this one is fired when the watch has stopped | ||
201 | 177 | self._watch_stopped_deferred = defer.Deferred() | ||
202 | 178 | 125 | ||
204 | 179 | @is_valid_windows_path(path_indexes=[1]) | 126 | @is_valid_os_path(path_indexes=[1]) |
205 | 180 | def _update_subdirs(self, path, event): | 127 | def _update_subdirs(self, path, event): |
206 | 181 | """Adds the path to the internal subdirs. | 128 | """Adds the path to the internal subdirs. |
207 | 182 | 129 | ||
208 | 183 | The given path is considered to be a path and therefore this | 130 | The given path is considered to be a path and therefore this |
209 | 184 | will not be checked. | 131 | will not be checked. |
210 | 185 | """ | 132 | """ |
212 | 186 | if WINDOWS_ACTIONS[event] == IN_CREATE: | 133 | if COMMON_ACTIONS[event] == IN_CREATE: |
213 | 187 | self._subdirs.add(path) | 134 | self._subdirs.add(path) |
216 | 188 | elif WINDOWS_ACTIONS[event] == IN_DELETE and\ | 135 | elif COMMON_ACTIONS[event] == IN_DELETE and path in self._subdirs: |
215 | 189 | path in self._subdirs: | ||
217 | 190 | self._subdirs.remove(path) | 136 | self._subdirs.remove(path) |
218 | 191 | 137 | ||
220 | 192 | @is_valid_windows_path(path_indexes=[1]) | 138 | @is_valid_os_path(path_indexes=[1]) |
221 | 193 | def _path_is_dir(self, path): | 139 | def _path_is_dir(self, path): |
223 | 194 | """Check if the path is a dir and update the local subdir list.""" | 140 | """Check if the path is a dir.""" |
224 | 195 | 141 | ||
232 | 196 | # The logic of this function is the following: | 142 | # We need to manually check if the path is a folder, because |
233 | 197 | # 1. ReadDirectoryChangesW changes does not send if a path | 143 | # neither ReadDirectoryChangesW nor the FSEvents API tell us |
227 | 198 | # is a new dir or not. | ||
228 | 199 | # 2. We keep track of subdirs that in self._subdir. | ||
229 | 200 | # 3. We check if a path is a dir by: | ||
230 | 201 | # * Asking the os if the path exists. | ||
231 | 202 | # * Finding the path in self._subdirs | ||
234 | 203 | 144 | ||
235 | 204 | is_dir = False | 145 | is_dir = False |
236 | 205 | if os.path.exists(path): | 146 | if os.path.exists(path): |
237 | @@ -210,131 +151,7 @@ | |||
238 | 210 | self.log.debug('Is path %r a dir? %s', path, is_dir) | 151 | self.log.debug('Is path %r a dir? %s', path, is_dir) |
239 | 211 | return is_dir | 152 | return is_dir |
240 | 212 | 153 | ||
366 | 213 | def _process_events(self, events): | 154 | @is_valid_os_path(path_indexes=[1]) |
242 | 214 | """Process the events from the queue.""" | ||
243 | 215 | # do not do it if we stop watching and the events are empty | ||
244 | 216 | if not self._watching: | ||
245 | 217 | return | ||
246 | 218 | |||
247 | 219 | # we transform the events to be the same as the one in pyinotify | ||
248 | 220 | # and then use the proc_fun | ||
249 | 221 | for action, file_name in events: | ||
250 | 222 | if any([file_name.startswith(path) | ||
251 | 223 | for path in self._ignore_paths]): | ||
252 | 224 | continue | ||
253 | 225 | # map the windows events to the pyinotify ones, tis is dirty but | ||
254 | 226 | # makes the multiplatform better, linux was first :P | ||
255 | 227 | syncdaemon_path = get_syncdaemon_valid_path( | ||
256 | 228 | os.path.join(self._path, file_name)) | ||
257 | 229 | full_dir_path = os.path.join(self._path, file_name) | ||
258 | 230 | is_dir = self._path_is_dir(full_dir_path) | ||
259 | 231 | if is_dir: | ||
260 | 232 | # we need to update the list of subdirs that we have | ||
261 | 233 | self._update_subdirs(full_dir_path, action) | ||
262 | 234 | mask = WINDOWS_ACTIONS[action] | ||
263 | 235 | head, tail = os.path.split(file_name) | ||
264 | 236 | if is_dir: | ||
265 | 237 | mask |= IN_ISDIR | ||
266 | 238 | event_raw_data = { | ||
267 | 239 | 'wd': self._descriptor, | ||
268 | 240 | 'dir': is_dir, | ||
269 | 241 | 'mask': mask, | ||
270 | 242 | 'name': tail, | ||
271 | 243 | 'path': '.'} | ||
272 | 244 | # by the way in which the win api fires the events we know for | ||
273 | 245 | # sure that no move events will be added in the wrong order, this | ||
274 | 246 | # is kind of hacky, I dont like it too much | ||
275 | 247 | if WINDOWS_ACTIONS[action] == IN_MOVED_FROM: | ||
276 | 248 | self._cookie = str(uuid4()) | ||
277 | 249 | self._source_pathname = tail | ||
278 | 250 | event_raw_data['cookie'] = self._cookie | ||
279 | 251 | if WINDOWS_ACTIONS[action] == IN_MOVED_TO: | ||
280 | 252 | event_raw_data['src_pathname'] = self._source_pathname | ||
281 | 253 | event_raw_data['cookie'] = self._cookie | ||
282 | 254 | event = Event(event_raw_data) | ||
283 | 255 | # FIXME: event deduces the pathname wrong and we need to manually | ||
284 | 256 | # set it | ||
285 | 257 | event.pathname = syncdaemon_path | ||
286 | 258 | # add the event only if we do not have an exclude filter or | ||
287 | 259 | # the exclude filter returns False, that is, the event will not | ||
288 | 260 | # be excluded | ||
289 | 261 | self.log.debug('Pushing event %r to processor.', event) | ||
290 | 262 | self._processor(event) | ||
291 | 263 | |||
292 | 264 | def _call_deferred(self, f, *args): | ||
293 | 265 | """Executes the deferred call avoiding possible race conditions.""" | ||
294 | 266 | if not self._watch_started_deferred.called: | ||
295 | 267 | f(*args) | ||
296 | 268 | |||
297 | 269 | def _watch_wrapper(self): | ||
298 | 270 | """Wrap _watch, and errback on any unhandled error.""" | ||
299 | 271 | try: | ||
300 | 272 | self._watch() | ||
301 | 273 | except Exception: | ||
302 | 274 | reactor.callFromThread(self._call_deferred, | ||
303 | 275 | self._watch_started_deferred.errback, Failure()) | ||
304 | 276 | |||
305 | 277 | def _watch(self): | ||
306 | 278 | """Watch a path that is a directory.""" | ||
307 | 279 | self.log.debug('Adding watch for %r (exists? %r is dir? %r).', | ||
308 | 280 | self._path, | ||
309 | 281 | os.path.exists(self._path), os.path.isdir(self._path)) | ||
310 | 282 | # we are going to be using the ReadDirectoryChangesW whihc requires | ||
311 | 283 | # a directory handle and the mask to be used. | ||
312 | 284 | self._watch_handle = CreateFileW( | ||
313 | 285 | self._path, | ||
314 | 286 | FILE_LIST_DIRECTORY, | ||
315 | 287 | FILE_SHARE_READ | FILE_SHARE_WRITE, | ||
316 | 288 | None, | ||
317 | 289 | OPEN_EXISTING, | ||
318 | 290 | FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED, | ||
319 | 291 | None) | ||
320 | 292 | |||
321 | 293 | try: | ||
322 | 294 | self._watch_loop(self._watch_handle) | ||
323 | 295 | finally: | ||
324 | 296 | CloseHandle(self._watch_handle) | ||
325 | 297 | self._watch_handle = None | ||
326 | 298 | reactor.callFromThread(self.stopped.callback, True) | ||
327 | 299 | |||
328 | 300 | def _watch_loop(self, handle): | ||
329 | 301 | """The loop where we watch the directory.""" | ||
330 | 302 | while True: | ||
331 | 303 | # important information to know about the parameters: | ||
332 | 304 | # param 1: the handle to the dir | ||
333 | 305 | # param 2: the size to be used in the kernel to store events | ||
334 | 306 | # that might be lost while the call is being performed. This | ||
335 | 307 | # is complicated to fine tune since if you make lots of watcher | ||
336 | 308 | # you migh used too much memory and make your OS to BSOD | ||
337 | 309 | buf = AllocateReadBuffer(self._buf_size) | ||
338 | 310 | ReadDirectoryChangesW( | ||
339 | 311 | handle, | ||
340 | 312 | buf, | ||
341 | 313 | self._auto_add, | ||
342 | 314 | self._mask, | ||
343 | 315 | self._overlapped, | ||
344 | 316 | ) | ||
345 | 317 | if not self._watch_started_deferred.called: | ||
346 | 318 | reactor.callFromThread(self._call_deferred, | ||
347 | 319 | self._watch_started_deferred.callback, True) | ||
348 | 320 | # wait for an event and ensure that we either stop or read the | ||
349 | 321 | # data | ||
350 | 322 | rc = WaitForMultipleObjects((self._wait_stop, | ||
351 | 323 | self._overlapped.hEvent), | ||
352 | 324 | 0, INFINITE) | ||
353 | 325 | if rc == WAIT_OBJECT_0: | ||
354 | 326 | # Stop event | ||
355 | 327 | break | ||
356 | 328 | # if we continue, it means that we got some data, lets read it | ||
357 | 329 | data = GetOverlappedResult(handle, self._overlapped, True) | ||
358 | 330 | # lets ead the data and store it in the results | ||
359 | 331 | events = FILE_NOTIFY_INFORMATION(buf, data) | ||
360 | 332 | self.log.debug('Got from ReadDirectoryChangesW %r.', | ||
361 | 333 | [(WINDOWS_ACTIONS_NAMES[action], path) for action, path in | ||
362 | 334 | events]) | ||
363 | 335 | reactor.callFromThread(self._process_events, events) | ||
364 | 336 | |||
365 | 337 | @is_valid_windows_path(path_indexes=[1]) | ||
367 | 338 | def ignore_path(self, path): | 155 | def ignore_path(self, path): |
368 | 339 | """Add the path of the events to ignore.""" | 156 | """Add the path of the events to ignore.""" |
369 | 340 | if not path.endswith(os.path.sep): | 157 | if not path.endswith(os.path.sep): |
370 | @@ -343,7 +160,7 @@ | |||
371 | 343 | path = path[len(self._path):] | 160 | path = path[len(self._path):] |
372 | 344 | self._ignore_paths.append(path) | 161 | self._ignore_paths.append(path) |
373 | 345 | 162 | ||
375 | 346 | @is_valid_windows_path(path_indexes=[1]) | 163 | @is_valid_os_path(path_indexes=[1]) |
376 | 347 | def remove_ignored_path(self, path): | 164 | def remove_ignored_path(self, path): |
377 | 348 | """Reaccept path.""" | 165 | """Reaccept path.""" |
378 | 349 | if not path.endswith(os.path.sep): | 166 | if not path.endswith(os.path.sep): |
379 | @@ -363,16 +180,12 @@ | |||
380 | 363 | # process the events. | 180 | # process the events. |
381 | 364 | self.log.debug('Start watching path.') | 181 | self.log.debug('Start watching path.') |
382 | 365 | self._watching = True | 182 | self._watching = True |
383 | 366 | reactor.callInThread(self._watch_wrapper) | ||
384 | 367 | return self._watch_started_deferred | ||
385 | 368 | 183 | ||
386 | 369 | def stop_watching(self): | 184 | def stop_watching(self): |
387 | 370 | """Tell the watch to stop processing events.""" | 185 | """Tell the watch to stop processing events.""" |
388 | 371 | self.log.info('Stop watching %s', self._path) | 186 | self.log.info('Stop watching %s', self._path) |
389 | 372 | SetEvent(self._wait_stop) | ||
390 | 373 | self._watching = False | 187 | self._watching = False |
391 | 374 | self._subdirs = set() | 188 | self._subdirs = set() |
392 | 375 | return self.stopped | ||
393 | 376 | 189 | ||
394 | 377 | def update(self, mask, auto_add=False): | 190 | def update(self, mask, auto_add=False): |
395 | 378 | """Update the info used by the watcher.""" | 191 | """Update the info used by the watcher.""" |
396 | @@ -389,29 +202,18 @@ | |||
397 | 389 | def auto_add(self): | 202 | def auto_add(self): |
398 | 390 | return self._auto_add | 203 | return self._auto_add |
399 | 391 | 204 | ||
400 | 392 | @property | ||
401 | 393 | def started(self): | ||
402 | 394 | """A deferred that will be called when the watch is running.""" | ||
403 | 395 | return self._watch_started_deferred | ||
404 | 396 | |||
405 | 397 | @property | ||
406 | 398 | def stopped(self): | ||
407 | 399 | """A deferred fired when the watch thread has finished.""" | ||
408 | 400 | return self._watch_stopped_deferred | ||
409 | 401 | |||
410 | 402 | 205 | ||
411 | 403 | class WatchManager(object): | 206 | class WatchManager(object): |
412 | 404 | """Implement the same functions as pyinotify.WatchManager. | 207 | """Implement the same functions as pyinotify.WatchManager. |
413 | 405 | 208 | ||
415 | 406 | All paths passed to methods in this class should be windows paths. | 209 | All paths passed to methods in this class should be proper os paths. |
416 | 407 | 210 | ||
417 | 408 | """ | 211 | """ |
418 | 409 | 212 | ||
419 | 410 | def __init__(self, processor): | 213 | def __init__(self, processor): |
420 | 411 | """Init the manager to keep trak of the different watches.""" | 214 | """Init the manager to keep trak of the different watches.""" |
421 | 412 | super(WatchManager, self).__init__() | ||
422 | 413 | self._processor = processor | 215 | self._processor = processor |
424 | 414 | self.log = logging.getLogger('ubuntuone.SyncDaemon.platform.windows.' | 216 | self.log = logging.getLogger('ubuntuone.SyncDaemon.platform.common.' |
425 | 415 | + 'filesystem_notifications.WatchManager') | 217 | + 'filesystem_notifications.WatchManager') |
426 | 416 | self.log.setLevel(TRACE) | 218 | self.log.setLevel(TRACE) |
427 | 417 | self._wdm = {} | 219 | self._wdm = {} |
428 | @@ -420,12 +222,8 @@ | |||
429 | 420 | 222 | ||
430 | 421 | def stop(self): | 223 | def stop(self): |
431 | 422 | """Close the manager and stop all watches.""" | 224 | """Close the manager and stop all watches.""" |
438 | 423 | self.log.debug('Stopping watches.') | 225 | # Should be implemented for each platform |
439 | 424 | wait_list = [] | 226 | raise NotImplementedError("Not implemented on this platform.") |
434 | 425 | for current_wd in self._wdm: | ||
435 | 426 | wait_list.append(self._wdm[current_wd].stop_watching()) | ||
436 | 427 | self.log.debug('Stopping Watch on %r.', self._wdm[current_wd].path) | ||
437 | 428 | return defer.DeferredList(wait_list) | ||
440 | 429 | 227 | ||
441 | 430 | def get_watch(self, wd): | 228 | def get_watch(self, wd): |
442 | 431 | """Return the watch with the given descriptor.""" | 229 | """Return the watch with the given descriptor.""" |
443 | @@ -442,8 +240,7 @@ | |||
444 | 442 | except KeyError, e: | 240 | except KeyError, e: |
445 | 443 | logging.error(str(e)) | 241 | logging.error(str(e)) |
446 | 444 | 242 | ||
449 | 445 | def _add_single_watch(self, path, mask, auto_add=False, | 243 | def _add_single_watch(self, path, mask, auto_add=False, quiet=True): |
448 | 446 | quiet=True): | ||
450 | 447 | if path in self._ignored_paths: | 244 | if path in self._ignored_paths: |
451 | 448 | # simply removed it from the filter | 245 | # simply removed it from the filter |
452 | 449 | self._ignored_paths.remove(path) | 246 | self._ignored_paths.remove(path) |
453 | @@ -452,23 +249,21 @@ | |||
454 | 452 | self.log.debug('add_single_watch(%s, %s, %s, %s)', path, mask, | 249 | self.log.debug('add_single_watch(%s, %s, %s, %s)', path, mask, |
455 | 453 | auto_add, quiet) | 250 | auto_add, quiet) |
456 | 454 | 251 | ||
468 | 455 | # adjust the number of threads based on the UDFs watched | 252 | return self._adding_watch(path, mask, auto_add) |
469 | 456 | reactor.suggestThreadPoolSize(THREADPOOL_MAX + self._wd_count + 1) | 253 | |
470 | 457 | self._wdm[self._wd_count] = Watch(self._wd_count, path, | 254 | def _adding_watch(self, path, mask, auto_add): |
471 | 458 | mask, auto_add, self._processor) | 255 | """Add the watch to the dict and start it.""" |
472 | 459 | d = self._wdm[self._wd_count].start_watching() | 256 | # This should be implemented for each OS |
473 | 460 | self._wd_count += 1 | 257 | raise NotImplementedError("Not implemented on this platform.") |
474 | 461 | return d | 258 | |
475 | 462 | 259 | @is_valid_os_path(path_indexes=[1]) | |
476 | 463 | @is_valid_windows_path(path_indexes=[1]) | 260 | def add_watch(self, path, mask, auto_add=False, quiet=True): |
466 | 464 | def add_watch(self, path, mask, auto_add=False, | ||
467 | 465 | quiet=True): | ||
477 | 466 | """Add a new path tp be watch. | 261 | """Add a new path tp be watch. |
478 | 467 | 262 | ||
479 | 468 | The method will ensure that the path is not already present. | 263 | The method will ensure that the path is not already present. |
480 | 469 | """ | 264 | """ |
481 | 470 | if not isinstance(path, unicode): | 265 | if not isinstance(path, unicode): |
483 | 471 | e = NotImplementedError("No implementation on windows.") | 266 | e = NotImplementedError("No implementation on this platform.") |
484 | 472 | return defer.fail(e) | 267 | return defer.fail(e) |
485 | 473 | wd = self.get_wd(path) | 268 | wd = self.get_wd(path) |
486 | 474 | if wd is None: | 269 | if wd is None: |
487 | @@ -480,9 +275,9 @@ | |||
488 | 480 | 275 | ||
489 | 481 | def update_watch(self, wd, mask=None, rec=False, | 276 | def update_watch(self, wd, mask=None, rec=False, |
490 | 482 | auto_add=False, quiet=True): | 277 | auto_add=False, quiet=True): |
492 | 483 | raise NotImplementedError("Not implemented on windows.") | 278 | raise NotImplementedError("Not implemented on this platform.") |
493 | 484 | 279 | ||
495 | 485 | @is_valid_windows_path(path_indexes=[1]) | 280 | @is_valid_os_path(path_indexes=[1]) |
496 | 486 | def get_wd(self, path): | 281 | def get_wd(self, path): |
497 | 487 | """Return the watcher that is used to watch the given path.""" | 282 | """Return the watcher that is used to watch the given path.""" |
498 | 488 | if not path.endswith(os.path.sep): | 283 | if not path.endswith(os.path.sep): |
499 | @@ -512,7 +307,7 @@ | |||
500 | 512 | if not quiet: | 307 | if not quiet: |
501 | 513 | raise WatchManagerError('Watch %s was not found' % wd, {}) | 308 | raise WatchManagerError('Watch %s was not found' % wd, {}) |
502 | 514 | 309 | ||
504 | 515 | @is_valid_windows_path(path_indexes=[1]) | 310 | @is_valid_os_path(path_indexes=[1]) |
505 | 516 | def rm_path(self, path): | 311 | def rm_path(self, path): |
506 | 517 | """Remove a watch to the given path.""" | 312 | """Remove a watch to the given path.""" |
507 | 518 | wd = self.get_wd(path) | 313 | wd = self.get_wd(path) |
508 | @@ -527,9 +322,6 @@ | |||
509 | 527 | This interface will be exposed to syncdaemon, ergo all passed | 322 | This interface will be exposed to syncdaemon, ergo all passed |
510 | 528 | and returned paths must be a sequence of BYTES encoded with utf8. | 323 | and returned paths must be a sequence of BYTES encoded with utf8. |
511 | 529 | 324 | ||
512 | 530 | Also, they must not be literal paths, that is the \\?\ prefix should not be | ||
513 | 531 | in the path. | ||
514 | 532 | |||
515 | 533 | """ | 325 | """ |
516 | 534 | 326 | ||
517 | 535 | def __init__(self, monitor, ignore_config=None): | 327 | def __init__(self, monitor, ignore_config=None): |
518 | @@ -549,13 +341,10 @@ | |||
519 | 549 | """Add an event and path(s) to the mute filter.""" | 341 | """Add an event and path(s) to the mute filter.""" |
520 | 550 | self.general_processor.add_to_mute_filter(event, paths) | 342 | self.general_processor.add_to_mute_filter(event, paths) |
521 | 551 | 343 | ||
522 | 552 | @is_valid_syncdaemon_path(path_indexes=[1]) | ||
523 | 553 | def platform_is_ignored(self, path): | 344 | def platform_is_ignored(self, path): |
524 | 554 | """Should we ignore this path in the current platform.?""" | 345 | """Should we ignore this path in the current platform.?""" |
529 | 555 | # don't support links yet | 346 | # This should be implemented in each platform |
530 | 556 | if path.endswith('.lnk'): | 347 | raise NotImplementedError("Not implemented on this platform.") |
527 | 557 | return True | ||
528 | 558 | return False | ||
531 | 559 | 348 | ||
532 | 560 | @is_valid_syncdaemon_path(path_indexes=[1]) | 349 | @is_valid_syncdaemon_path(path_indexes=[1]) |
533 | 561 | def is_ignored(self, path): | 350 | def is_ignored(self, path): |
534 | @@ -572,7 +361,7 @@ | |||
535 | 572 | # lets ignore dir changes | 361 | # lets ignore dir changes |
536 | 573 | if event.dir: | 362 | if event.dir: |
537 | 574 | return | 363 | return |
539 | 575 | # on windows we just get IN_MODIFY, lets always fake | 364 | # on someplatforms we just get IN_MODIFY, lets always fake |
540 | 576 | # an OPEN & CLOSE_WRITE couple | 365 | # an OPEN & CLOSE_WRITE couple |
541 | 577 | raw_open = raw_close = { | 366 | raw_open = raw_close = { |
542 | 578 | 'wd': event.wd, | 367 | 'wd': event.wd, |
543 | @@ -665,7 +454,7 @@ | |||
544 | 665 | self.release_held_event() | 454 | self.release_held_event() |
545 | 666 | self.general_processor.push_event(event) | 455 | self.general_processor.push_event(event) |
546 | 667 | else: | 456 | else: |
548 | 668 | # We should never get here on windows, I really do not know how we | 457 | # We should never get here, I really do not know how we |
549 | 669 | # got here | 458 | # got here |
550 | 670 | self.general_processor.log.warn( | 459 | self.general_processor.log.warn( |
551 | 671 | 'Cookie does not match the previoues held event!') | 460 | 'Cookie does not match the previoues held event!') |
552 | @@ -739,11 +528,12 @@ | |||
553 | 739 | 528 | ||
554 | 740 | def __init__(self, eq, fs, ignore_config=None, timeout=1): | 529 | def __init__(self, eq, fs, ignore_config=None, timeout=1): |
555 | 741 | self.log = logging.getLogger('ubuntuone.SyncDaemon.FSMonitor') | 530 | self.log = logging.getLogger('ubuntuone.SyncDaemon.FSMonitor') |
556 | 531 | self.filesystem_monitor_mask = None | ||
557 | 742 | self.log.setLevel(TRACE) | 532 | self.log.setLevel(TRACE) |
558 | 743 | self.fs = fs | 533 | self.fs = fs |
559 | 744 | self.eq = eq | 534 | self.eq = eq |
562 | 745 | self._processor = NotifyProcessor(self, ignore_config) | 535 | # You will need to create the NotifyProcessor and WatchManager |
563 | 746 | self._watch_manager = WatchManager(self._processor) | 536 | # in each OS-specific implementation |
564 | 747 | 537 | ||
565 | 748 | def add_to_mute_filter(self, event, **info): | 538 | def add_to_mute_filter(self, event, **info): |
566 | 749 | """Add info to mute filter in the processor.""" | 539 | """Add info to mute filter in the processor.""" |
567 | @@ -757,27 +547,24 @@ | |||
568 | 757 | """Prepares the EQ to be closed.""" | 547 | """Prepares the EQ to be closed.""" |
569 | 758 | return self._watch_manager.stop() | 548 | return self._watch_manager.stop() |
570 | 759 | 549 | ||
572 | 760 | @windowspath(path_indexes=[1]) | 550 | @os_path(path_indexes=[1]) |
573 | 761 | def rm_watch(self, dirpath): | 551 | def rm_watch(self, dirpath): |
574 | 762 | """Remove watch from a dir.""" | 552 | """Remove watch from a dir.""" |
575 | 763 | # trust the implementation of the manager | 553 | # trust the implementation of the manager |
576 | 764 | self._watch_manager.rm_path(dirpath) | 554 | self._watch_manager.rm_path(dirpath) |
577 | 765 | 555 | ||
579 | 766 | @windowspath(path_indexes=[1]) | 556 | @os_path(path_indexes=[1]) |
580 | 767 | def add_watch(self, dirpath): | 557 | def add_watch(self, dirpath): |
581 | 768 | """Add watch to a dir.""" | 558 | """Add watch to a dir.""" |
582 | 769 | # the logic to check if the watch is already set | 559 | # the logic to check if the watch is already set |
583 | 770 | # is all in WatchManager.add_watch | 560 | # is all in WatchManager.add_watch |
584 | 771 | return self._watch_manager.add_watch(dirpath, | 561 | return self._watch_manager.add_watch(dirpath, |
586 | 772 | FILESYSTEM_MONITOR_MASK, auto_add=True) | 562 | self.filesystem_monitor_mask, auto_add=True) |
587 | 773 | 563 | ||
588 | 774 | def add_watches_to_udf_ancestors(self, volume): | 564 | def add_watches_to_udf_ancestors(self, volume): |
589 | 775 | """Add a inotify watch to volume's ancestors if it's an UDF.""" | 565 | """Add a inotify watch to volume's ancestors if it's an UDF.""" |
595 | 776 | # On windows there is no need to add the watches to the ancestors | 566 | |
596 | 777 | # so we will always return true. The reason for this is that the | 567 | # Should be implemented in each OS if necessary |
592 | 778 | # handles that we open stop the user from renaming the ancestors of | ||
593 | 779 | # the UDF, for a user to do that he has to unsync the udf first | ||
594 | 780 | return defer.succeed(True) | ||
597 | 781 | 568 | ||
598 | 782 | def is_frozen(self): | 569 | def is_frozen(self): |
599 | 783 | """Checks if there's something frozen.""" | 570 | """Checks if there's something frozen.""" |
600 | 784 | 571 | ||
601 | === added file 'ubuntuone/platform/filesystem_notifications/windows.py' | |||
602 | --- ubuntuone/platform/filesystem_notifications/windows.py 1970-01-01 00:00:00 +0000 | |||
603 | +++ ubuntuone/platform/filesystem_notifications/windows.py 2012-06-26 19:23:19 +0000 | |||
604 | @@ -0,0 +1,308 @@ | |||
605 | 1 | # -*- coding: utf-8 *-* | ||
606 | 2 | # | ||
607 | 3 | # Copyright 2011-2012 Canonical Ltd. | ||
608 | 4 | # | ||
609 | 5 | # This program is free software: you can redistribute it and/or modify it | ||
610 | 6 | # under the terms of the GNU General Public License version 3, as published | ||
611 | 7 | # by the Free Software Foundation. | ||
612 | 8 | # | ||
613 | 9 | # This program is distributed in the hope that it will be useful, but | ||
614 | 10 | # WITHOUT ANY WARRANTY; without even the implied warranties of | ||
615 | 11 | # MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR | ||
616 | 12 | # PURPOSE. See the GNU General Public License for more details. | ||
617 | 13 | # | ||
618 | 14 | # You should have received a copy of the GNU General Public License along | ||
619 | 15 | # with this program. If not, see <http://www.gnu.org/licenses/>. | ||
620 | 16 | # | ||
621 | 17 | # In addition, as a special exception, the copyright holders give | ||
622 | 18 | # permission to link the code of portions of this program with the | ||
623 | 19 | # OpenSSL library under certain conditions as described in each | ||
624 | 20 | # individual source file, and distribute linked combinations | ||
625 | 21 | # including the two. | ||
626 | 22 | # You must obey the GNU General Public License in all respects | ||
627 | 23 | # for all of the code used other than OpenSSL. If you modify | ||
628 | 24 | # file(s) with this exception, you may extend this exception to your | ||
629 | 25 | # version of the file(s), but you are not obligated to do so. If you | ||
630 | 26 | # do not wish to do so, delete this exception statement from your | ||
631 | 27 | # version. If you delete this exception statement from all source | ||
632 | 28 | # files in the program, then also delete it here. | ||
633 | 29 | """File notifications on windows.""" | ||
634 | 30 | |||
635 | 31 | import os | ||
636 | 32 | from uuid import uuid4 | ||
637 | 33 | |||
638 | 34 | from twisted.internet import defer, reactor | ||
639 | 35 | from twisted.python.failure import Failure | ||
640 | 36 | |||
641 | 37 | from pywintypes import OVERLAPPED | ||
642 | 38 | from win32api import CloseHandle | ||
643 | 39 | from win32con import ( | ||
644 | 40 | FILE_SHARE_READ, | ||
645 | 41 | FILE_SHARE_WRITE, | ||
646 | 42 | FILE_FLAG_BACKUP_SEMANTICS, | ||
647 | 43 | FILE_NOTIFY_CHANGE_FILE_NAME, | ||
648 | 44 | FILE_NOTIFY_CHANGE_DIR_NAME, | ||
649 | 45 | FILE_NOTIFY_CHANGE_ATTRIBUTES, | ||
650 | 46 | FILE_NOTIFY_CHANGE_SIZE, | ||
651 | 47 | FILE_NOTIFY_CHANGE_LAST_WRITE, | ||
652 | 48 | FILE_NOTIFY_CHANGE_SECURITY, | ||
653 | 49 | OPEN_EXISTING) | ||
654 | 50 | from win32file import ( | ||
655 | 51 | AllocateReadBuffer, | ||
656 | 52 | CreateFileW, | ||
657 | 53 | GetOverlappedResult, | ||
658 | 54 | ReadDirectoryChangesW, | ||
659 | 55 | FILE_FLAG_OVERLAPPED, | ||
660 | 56 | FILE_NOTIFY_INFORMATION) | ||
661 | 57 | from win32event import ( | ||
662 | 58 | CreateEvent, | ||
663 | 59 | INFINITE, | ||
664 | 60 | SetEvent, | ||
665 | 61 | WaitForMultipleObjects, | ||
666 | 62 | WAIT_OBJECT_0) | ||
667 | 63 | |||
668 | 64 | from ubuntuone.platform.os_helper.windows import ( | ||
669 | 65 | is_valid_syncdaemon_path, | ||
670 | 66 | get_syncdaemon_valid_path, | ||
671 | 67 | ) | ||
672 | 68 | |||
673 | 69 | from ubuntuone.platform.filesystem_notifications import common | ||
674 | 70 | |||
675 | 71 | |||
676 | 72 | # constant found in the msdn documentation: | ||
677 | 73 | # http://msdn.microsoft.com/en-us/library/ff538834(v=vs.85).aspx | ||
678 | 74 | FILE_LIST_DIRECTORY = 0x0001 | ||
679 | 75 | FILE_NOTIFY_CHANGE_LAST_ACCESS = 0x00000020 | ||
680 | 76 | FILE_NOTIFY_CHANGE_CREATION = 0x00000040 | ||
681 | 77 | |||
682 | 78 | THREADPOOL_MAX = 20 | ||
683 | 79 | |||
684 | 80 | |||
685 | 81 | class Watch(common.Watch): | ||
686 | 82 | """Implement the same functions as pyinotify.Watch.""" | ||
687 | 83 | |||
688 | 84 | def __init__(self, watch_descriptor, path, mask, auto_add, processor, | ||
689 | 85 | buf_size=8192): | ||
690 | 86 | super(Watch, self).__init__(watch_descriptor, path, mask, auto_add, | ||
691 | 87 | processor, buf_size) | ||
692 | 88 | self._wait_stop = CreateEvent(None, 0, 0, None) | ||
693 | 89 | self._overlapped = OVERLAPPED() | ||
694 | 90 | self._overlapped.hEvent = CreateEvent(None, 0, 0, None) | ||
695 | 91 | # remember the subdirs we have so that when we have a delete we can | ||
696 | 92 | # check if it was a remove | ||
697 | 93 | self._subdirs = set() | ||
698 | 94 | # this deferred is fired when the watch has started monitoring | ||
699 | 95 | # a directory from a thread | ||
700 | 96 | self._watch_started_deferred = defer.Deferred() | ||
701 | 97 | # and this one is fired when the watch has stopped | ||
702 | 98 | self._watch_stopped_deferred = defer.Deferred() | ||
703 | 99 | |||
704 | 100 | def _process_events(self, events): | ||
705 | 101 | """Process the events from the queue.""" | ||
706 | 102 | # do not do it if we stop watching and the events are empty | ||
707 | 103 | if not self._watching: | ||
708 | 104 | return | ||
709 | 105 | |||
710 | 106 | # we transform the events to be the same as the one in pyinotify | ||
711 | 107 | # and then use the proc_fun | ||
712 | 108 | for action, file_name in events: | ||
713 | 109 | if any([file_name.startswith(path) | ||
714 | 110 | for path in self._ignore_paths]): | ||
715 | 111 | continue | ||
716 | 112 | # map the windows events to the pyinotify ones, tis is dirty but | ||
717 | 113 | # makes the multiplatform better, linux was first :P | ||
718 | 114 | syncdaemon_path = get_syncdaemon_valid_path( | ||
719 | 115 | os.path.join(self._path, file_name)) | ||
720 | 116 | full_dir_path = os.path.join(self._path, file_name) | ||
721 | 117 | is_dir = self._path_is_dir(full_dir_path) | ||
722 | 118 | if is_dir: | ||
723 | 119 | # we need to update the list of subdirs that we have | ||
724 | 120 | self._update_subdirs(full_dir_path, action) | ||
725 | 121 | mask = common.COMMON_ACTIONS[action] | ||
726 | 122 | head, tail = os.path.split(file_name) | ||
727 | 123 | if is_dir: | ||
728 | 124 | mask |= common.IN_ISDIR | ||
729 | 125 | event_raw_data = { | ||
730 | 126 | 'wd': self._descriptor, | ||
731 | 127 | 'dir': is_dir, | ||
732 | 128 | 'mask': mask, | ||
733 | 129 | 'name': tail, | ||
734 | 130 | 'path': '.'} | ||
735 | 131 | # by the way in which the win api fires the events we know for | ||
736 | 132 | # sure that no move events will be added in the wrong order, this | ||
737 | 133 | # is kind of hacky, I dont like it too much | ||
738 | 134 | if common.COMMON_ACTIONS[action] == common.IN_MOVED_FROM: | ||
739 | 135 | self._cookie = str(uuid4()) | ||
740 | 136 | self._source_pathname = tail | ||
741 | 137 | event_raw_data['cookie'] = self._cookie | ||
742 | 138 | if common.COMMON_ACTIONS[action] == common.IN_MOVED_TO: | ||
743 | 139 | event_raw_data['src_pathname'] = self._source_pathname | ||
744 | 140 | event_raw_data['cookie'] = self._cookie | ||
745 | 141 | event = common.Event(event_raw_data) | ||
746 | 142 | # FIXME: event deduces the pathname wrong and we need to manually | ||
747 | 143 | # set it | ||
748 | 144 | event.pathname = syncdaemon_path | ||
749 | 145 | # add the event only if we do not have an exclude filter or | ||
750 | 146 | # the exclude filter returns False, that is, the event will not | ||
751 | 147 | # be excluded | ||
752 | 148 | self.log.debug('Pushing event %r to processor.', event) | ||
753 | 149 | self._processor(event) | ||
754 | 150 | |||
755 | 151 | def _call_deferred(self, f, *args): | ||
756 | 152 | """Executes the deferred call avoiding possible race conditions.""" | ||
757 | 153 | if not self._watch_started_deferred.called: | ||
758 | 154 | f(*args) | ||
759 | 155 | |||
760 | 156 | def _watch_wrapper(self): | ||
761 | 157 | """Wrap _watch, and errback on any unhandled error.""" | ||
762 | 158 | try: | ||
763 | 159 | self._watch() | ||
764 | 160 | except Exception: | ||
765 | 161 | reactor.callFromThread(self._call_deferred, | ||
766 | 162 | self._watch_started_deferred.errback, Failure()) | ||
767 | 163 | |||
768 | 164 | def _watch(self): | ||
769 | 165 | """Watch a path that is a directory.""" | ||
770 | 166 | self.log.debug('Adding watch for %r (exists? %r is dir? %r).', | ||
771 | 167 | self._path, | ||
772 | 168 | os.path.exists(self._path), os.path.isdir(self._path)) | ||
773 | 169 | # we are going to be using the ReadDirectoryChangesW whihc requires | ||
774 | 170 | # a directory handle and the mask to be used. | ||
775 | 171 | self._watch_handle = CreateFileW( | ||
776 | 172 | self._path, | ||
777 | 173 | FILE_LIST_DIRECTORY, | ||
778 | 174 | FILE_SHARE_READ | FILE_SHARE_WRITE, | ||
779 | 175 | None, | ||
780 | 176 | OPEN_EXISTING, | ||
781 | 177 | FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED, | ||
782 | 178 | None) | ||
783 | 179 | |||
784 | 180 | try: | ||
785 | 181 | self._watch_loop(self._watch_handle) | ||
786 | 182 | finally: | ||
787 | 183 | CloseHandle(self._watch_handle) | ||
788 | 184 | self._watch_handle = None | ||
789 | 185 | reactor.callFromThread(self.stopped.callback, True) | ||
790 | 186 | |||
791 | 187 | def _watch_loop(self, handle): | ||
792 | 188 | """The loop where we watch the directory.""" | ||
793 | 189 | while True: | ||
794 | 190 | # important information to know about the parameters: | ||
795 | 191 | # param 1: the handle to the dir | ||
796 | 192 | # param 2: the size to be used in the kernel to store events | ||
797 | 193 | # that might be lost while the call is being performed. This | ||
798 | 194 | # is complicated to fine tune since if you make lots of watcher | ||
799 | 195 | # you migh used too much memory and make your OS to BSOD | ||
800 | 196 | buf = AllocateReadBuffer(self._buf_size) | ||
801 | 197 | ReadDirectoryChangesW( | ||
802 | 198 | handle, | ||
803 | 199 | buf, | ||
804 | 200 | self._auto_add, | ||
805 | 201 | self._mask, | ||
806 | 202 | self._overlapped, | ||
807 | 203 | ) | ||
808 | 204 | if not self._watch_started_deferred.called: | ||
809 | 205 | reactor.callFromThread(self._call_deferred, | ||
810 | 206 | self._watch_started_deferred.callback, True) | ||
811 | 207 | # wait for an event and ensure that we either stop or read the | ||
812 | 208 | # data | ||
813 | 209 | rc = WaitForMultipleObjects((self._wait_stop, | ||
814 | 210 | self._overlapped.hEvent), | ||
815 | 211 | 0, INFINITE) | ||
816 | 212 | if rc == WAIT_OBJECT_0: | ||
817 | 213 | # Stop event | ||
818 | 214 | break | ||
819 | 215 | # if we continue, it means that we got some data, lets read it | ||
820 | 216 | data = GetOverlappedResult(handle, self._overlapped, True) | ||
821 | 217 | # lets ead the data and store it in the results | ||
822 | 218 | events = FILE_NOTIFY_INFORMATION(buf, data) | ||
823 | 219 | self.log.debug('Got from ReadDirectoryChangesW %r.', | ||
824 | 220 | [(common.COMMON_ACTIONS_NAMES[action], path) \ | ||
825 | 221 | for action, path in events]) | ||
826 | 222 | reactor.callFromThread(self._process_events, events) | ||
827 | 223 | |||
828 | 224 | def start_watching(self): | ||
829 | 225 | """Tell the watch to start processing events.""" | ||
830 | 226 | super(Watch, self).start_watching() | ||
831 | 227 | reactor.callInThread(self._watch_wrapper) | ||
832 | 228 | return self._watch_started_deferred | ||
833 | 229 | |||
834 | 230 | def stop_watching(self): | ||
835 | 231 | """Tell the watch to stop processing events.""" | ||
836 | 232 | super(Watch, self).stop_watching() | ||
837 | 233 | SetEvent(self._wait_stop) | ||
838 | 234 | return self.stopped | ||
839 | 235 | |||
840 | 236 | @property | ||
841 | 237 | def started(self): | ||
842 | 238 | """A deferred that will be called when the watch is running.""" | ||
843 | 239 | return self._watch_started_deferred | ||
844 | 240 | |||
845 | 241 | @property | ||
846 | 242 | def stopped(self): | ||
847 | 243 | """A deferred fired when the watch thread has finished.""" | ||
848 | 244 | return self._watch_stopped_deferred | ||
849 | 245 | |||
850 | 246 | |||
851 | 247 | class WatchManager(common.WatchManager): | ||
852 | 248 | |||
853 | 249 | def stop(self): | ||
854 | 250 | """Close the manager and stop all watches.""" | ||
855 | 251 | self.log.debug('Stopping watches.') | ||
856 | 252 | wait_list = [] | ||
857 | 253 | for current_wd in self._wdm: | ||
858 | 254 | wait_list.append(self._wdm[current_wd].stop_watching()) | ||
859 | 255 | self.log.debug('Stopping Watch on %r.', self._wdm[current_wd].path) | ||
860 | 256 | return defer.DeferredList(wait_list) | ||
861 | 257 | |||
862 | 258 | def _adding_watch(self, path, mask, auto_add): | ||
863 | 259 | # adjust the number of threads based on the UDFs watched | ||
864 | 260 | watch = Watch(self._wd_count, path, mask, auto_add, self._processor) | ||
865 | 261 | reactor.suggestThreadPoolSize(THREADPOOL_MAX + self._wd_count + 1) | ||
866 | 262 | self._wdm[self._wd_count] = watch | ||
867 | 263 | d = self._wdm[self._wd_count].start_watching() | ||
868 | 264 | self._wd_count += 1 | ||
869 | 265 | return d | ||
870 | 266 | |||
871 | 267 | |||
872 | 268 | class NotifyProcessor(common.NotifyProcessor): | ||
873 | 269 | """ | ||
874 | 270 | Processor that takes care of dealing with the events. | ||
875 | 271 | |||
876 | 272 | Also, they must not be literal paths, that is the \\?\ prefix should not be | ||
877 | 273 | in the path. | ||
878 | 274 | """ | ||
879 | 275 | |||
880 | 276 | @is_valid_syncdaemon_path(path_indexes=[1]) | ||
881 | 277 | def platform_is_ignored(self, path): | ||
882 | 278 | """Should we ignore this path in the current platform.?""" | ||
883 | 279 | # don't support links yet | ||
884 | 280 | if path.endswith('.lnk'): | ||
885 | 281 | return True | ||
886 | 282 | return False | ||
887 | 283 | |||
888 | 284 | |||
889 | 285 | class FilesystemMonitor(common.FilesystemMonitor): | ||
890 | 286 | """Manages the signals from filesystem.""" | ||
891 | 287 | |||
892 | 288 | def __init__(self, eq, fs, ignore_config=None, timeout=1): | ||
893 | 289 | super(FilesystemMonitor, self).__init__(eq, fs, ignore_config, timeout) | ||
894 | 290 | self._processor = NotifyProcessor(self, ignore_config) | ||
895 | 291 | self._watch_manager = WatchManager(self._processor) | ||
896 | 292 | # the default mask to be used in the watches | ||
897 | 293 | # added by the FilesystemMonitor class | ||
898 | 294 | self.filesystem_monitor_mask = FILE_NOTIFY_CHANGE_FILE_NAME | \ | ||
899 | 295 | FILE_NOTIFY_CHANGE_DIR_NAME | \ | ||
900 | 296 | FILE_NOTIFY_CHANGE_ATTRIBUTES | \ | ||
901 | 297 | FILE_NOTIFY_CHANGE_SIZE | \ | ||
902 | 298 | FILE_NOTIFY_CHANGE_LAST_WRITE | \ | ||
903 | 299 | FILE_NOTIFY_CHANGE_SECURITY | \ | ||
904 | 300 | FILE_NOTIFY_CHANGE_LAST_ACCESS | ||
905 | 301 | |||
906 | 302 | def add_watches_to_udf_ancestors(self, volume): | ||
907 | 303 | """Add a inotify watch to volume's ancestors if it's an UDF.""" | ||
908 | 304 | # There is no need to add the watches to the ancestors | ||
909 | 305 | # so we will always return true. The reason for this is that the | ||
910 | 306 | # handles that we open stop the user from renaming the ancestors of | ||
911 | 307 | # the UDF, for a user to do that he has to unsync the udf first | ||
912 | 308 | return defer.succeed(True) |
The implementation in add_watches_ to_udf_ ancestors is *very* specific to windows, so it needs to be moved to windows.py
---
FILESYSTEM_ MONITOR_ MASK gets broken in this branch because filesystem_ monitor_ mask seems to always be None.
---
All of this would break on darwin: valid_syncdaemo n_path = None valid_os_ path = None
is_
is_
os_path = windowspath = None
So everything under:
elif sys.platform == "darwin":
should come on a following branch.
---
Please rename "reference" to something less vague, and move each definition to the platform specific files.