Merge lp:~mandel/ubuntuone-client/add-virtual-watches into lp:ubuntuone-client

[Manuel de la Peña]

Fixes lp:907511

This branch allows the Windows Watch implementation to behave more like the one found on pyinotify. The way this is achieved is the following:

* Moved _subdirs var to be a dictionary that store a map between a child path and if it is watched or not. The reason is that child_paths are relative to self._path which means that we store less info. The boolean state if we are watching the path, if not, we ignore all the events from that path.
* When we start watching all child events are added to _subdirs are not being watch until sd explicitly asks it.
* When adding a child we add it children to the _subdirs. The are not thread or run issues because the sd will add the watches always in the main twisted thread which is the one used to process the event.
* Moved _ignored_paths to be a set for performance.
* Remove the auto-add parameters and var since this are not expose to the rest of sd and they do not work on linux.
* Modify the tests to use a MementoWatch. The MementoWatch memorizes all the events that have been triggered by the ReadDirecotryChangesW and later processes them, that way we can have tests and assert if the correct number of events was raised and that we filtered the expected ones.
* Added a new filter that ensures that IN_MODIFY|IS_DIR events are not raised when the child is not watched.

Fixes lp:907511

This branch allows the Windows Watch implementation to behave more like the one found on pyinotify. The way this is achieved is the following:

* Moved _subdirs var to be a dictionary that store a map between a child path and if it is watched or not. The reason is that child_paths are relative to self._path which means that we store less info. The boolean state if we are watching the path, if not, we ignore all the events from that path.
* When we start watching all child events are added to _subdirs are not being watch until sd explicitly asks it.
* When adding a child we add it children to the _subdirs. The are not thread or run issues because the sd will add the watches always in the main twisted thread which is the one used to process the event.
* Moved _ignored_paths to be a set for performance.
* Remove the auto-add parameters and var since this are not expose to the rest of sd and they do not work on linux.
* Modify the tests to use a MementoWatch. The MementoWatch memorizes all the events that have been triggered by the ReadDirecotryChangesW and later processes them, that way we can have tests and assert if the correct number of events was raised and that we filtered the expected ones.
* Added a new filter that ensures that IN_MODIFY|IS_DIR events are not raised when the child is not watched.

[Brian Curtin (brian.curtin)]

Here's a mini-review. I don't have my dev environment setup and don't know a lot of the context, plus I can't run it yet...but here's a few things I noticed.

######
test_filesystem_notifications.py

80 + self.mask = fs.FILE_NOTIFY_CHANGE_FILE_NAME | \
81 + fs.FILE_NOTIFY_CHANGE_DIR_NAME | \

Minor style thing, but I noticed in a big import (that was removed) that long lines were wrapped with parentheses instead of breaking with backslashes. Maybe do the same here? (Sorry, had to get my first nitpick cosmetic comment out of the way :)

===========================
filesystem_notifications.py

_get_partial_child_path_dir

It's about 2-3x faster to take a slice of the last character and compare it rather than using path.endswith. This may not be as clean as using endswith, but figured it may be worth mentioning. Something like "if not path[-1] == os.path.sep:"

Other than that, this method looks good.
******
+ self.log.warn('Adding watch to none existing path "%s"', path)

none existing > nonexistent
******
+ def add_child_watch(self, path):

This new method looks fine to me.
******
+ """Add a new path to be watch.

watch > watched
============
os_helper.py

native_listdir

What is the purpose of having listdir and native_listdir if they are the same?

[Manuel de la Peña (mandel)]

> Here's a mini-review. I don't have my dev environment setup and don't know a
> lot of the context, plus I can't run it yet...but here's a few things I
> noticed.
>
> ######
> test_filesystem_notifications.py
>
> 80 + self.mask = fs.FILE_NOTIFY_CHANGE_FILE_NAME | \
> 81 + fs.FILE_NOTIFY_CHANGE_DIR_NAME | \
>
> Minor style thing, but I noticed in a big import (that was removed) that long
> lines were wrapped with parentheses instead of breaking with backslashes.
> Maybe do the same here? (Sorry, had to get my first nitpick cosmetic comment
> out of the way :)

It makes sense, I actually prefer to use () than \ since for me it seems cleaner. Will get that fix.

>
> ===========================
> filesystem_notifications.py
>
> _get_partial_child_path_dir
>
> It's about 2-3x faster to take a slice of the last character and compare it
> rather than using path.endswith. This may not be as clean as using endswith,
> but figured it may be worth mentioning. Something like "if not path[-1] ==
> os.path.sep:"
>

On it! good catch!

> Other than that, this method looks good.
> ******
> + self.log.warn('Adding watch to none existing path "%s"', path)
>
> none existing > nonexistent

Fixing

> ******
> + def add_child_watch(self, path):
>
> This new method looks fine to me.
> ******
> + """Add a new path to be watch.
>
> watch > watched

Fixing.

> ============
> os_helper.py
>
> native_listdir
>
> What is the purpose of having listdir and native_listdir if they are the same?

Really good question, and it is indeed a bug. os.listdir does return certain values, such as system folders, that we want to ignore. I should be using the wrapped method from os_helper.

Thx for the review!

[lp:~mandel/ubuntuone-client/add-virtual-watches]

1199. By Manuel de la Peña on 2012-01-17

Fixed code according to review.

[Diego Sarmentero (diegosarmentero)]

+1
All green!

review: Approve

[Alejandro J. Cura (alecu)]

The branch looks interesting.

A few comments:
----
941 + if any([file_name.startswith(path) and not watched
942 + for path, watched in self._subdirs.iteritems()]):

The list comprehension used inside the "any" could be replaced by a generator expression just by dropping the [ and ]. With that change a new list won't have to be constructed, and the any will be able to finish faster by not checking the rest if any item matches the expression.

----

"we get no events, lets therefore ignore them" ->
"we get no events, let's therefore ignore them

----

"lets remove the path" ->
"let's remove the path"

----

There are many references to "manager._wdm[0].", and it looks a bit flaky.
I think that instead this line:

   self.patch(fs, 'Watch', MementoWatch)

should be replaced by something like:

   self.patch(fs, 'Watch', lambda *args, **kwargs: self.mementowatch_factory(all_events_count, *args, **kwargs))

and:

   def mementowatch_factory(self, all_events_count, *args, **kwargs):
       self.memento_watch = MementoWatch(...)
       return self.memento_watch

----
"suddirs" -> "subdirs"

[lp:~mandel/ubuntuone-client/add-virtual-watches]

1200. By Manuel de la Peña on 2012-01-23

Fixed code according to review comments.

1201. By Manuel de la Peña on 2012-01-23

Fixed some stupidly broken tests.

1202. By Manuel de la Peña on 2012-01-23

Merged with trunk.

[Facundo Batista (facundo)]

There's a race condition, as we talked by IRC, when getting subdirs for the _subdirs internal structure, and the real directory/files creation in disk.

review: Needs Fixing

[Alejandro J. Cura (alecu)]

Line 132 seems to be missing a leading space:

130 + def mementowatch_factory(all_events_count, *args, **kwargs):
131 + """Factory that creates a new Watch for the tests."""
132 + kwargs['all_events_count'] = all_events_count
133 + self.memento_watch = MementoWatch(*args, **kwargs)
134 + return self.memento_watch

[lp:~mandel/ubuntuone-client/add-virtual-watches]

1203. By Manuel de la Peña on 2012-01-26

Added a number of extra tests to ensure that we can work around the race condition.

1204. By Manuel de la Peña on 2012-01-30

Face palm.

1205. By Manuel de la Peña on 2012-01-30

Face palm.

1206. By Manuel de la Peña on 2012-01-31

Fixed the race condition to ensure that if a child is missing we add it to the _subdirs dict. Fixed old tests that relied in some implementation details.

1207. By Manuel de la Peña on 2012-01-31

Fixed broken listdir.

1208. By Manuel de la Peña on 2012-02-03

Simplified the number of if statements used to ensure that the child path is present.

1209. By Manuel de la Peña on 2012-02-03

Face palm.

1210. By Manuel de la Peña on 2012-02-03

Place the parent_dir definition closer to the use. Fix the deep test because a and A are the same on windows.

Unmerged revisions

1210. By Manuel de la Peña on 2012-02-03

Place the parent_dir definition closer to the use. Fix the deep test because a and A are the same on windows.

1209. By Manuel de la Peña on 2012-02-03

Face palm.

1208. By Manuel de la Peña on 2012-02-03

Simplified the number of if statements used to ensure that the child path is present.

1207. By Manuel de la Peña on 2012-01-31

Fixed broken listdir.

1206. By Manuel de la Peña on 2012-01-31

Fixed the race condition to ensure that if a child is missing we add it to the _subdirs dict. Fixed old tests that relied in some implementation details.

1205. By Manuel de la Peña on 2012-01-30

Face palm.

1204. By Manuel de la Peña on 2012-01-30

Face palm.

1203. By Manuel de la Peña on 2012-01-26

Added a number of extra tests to ensure that we can work around the race condition.

1202. By Manuel de la Peña on 2012-01-23

Merged with trunk.

1201. By Manuel de la Peña on 2012-01-23

Fixed some stupidly broken tests.

=== modified file 'tests/platform/windows/test_filesystem_notifications.py'
--- tests/platform/windows/test_filesystem_notifications.py	2011-10-28 18:43:23 +0000
+++ tests/platform/windows/test_filesystem_notifications.py	2012-02-03 11:33:18 +0000
@@ -21,8 +21,8 @@
 import os
 import tempfile
 import thread
-import time
 import itertools
+import string
 
 from twisted.internet import defer
 from win32file import FILE_NOTIFY_INFORMATION
@@ -37,25 +37,12 @@
     IN_DELETE,
     IN_OPEN,
 )
-from ubuntuone.platform.windows import filesystem_notifications
-from ubuntuone.platform.windows.filesystem_notifications import (
-    FilesystemMonitor,
-    NotifyProcessor,
-    Watch,
-    WatchManager,
-    FILE_NOTIFY_CHANGE_FILE_NAME,
-    FILE_NOTIFY_CHANGE_DIR_NAME,
-    FILE_NOTIFY_CHANGE_ATTRIBUTES,
-    FILE_NOTIFY_CHANGE_SIZE,
-    FILE_NOTIFY_CHANGE_LAST_WRITE,
-    FILE_NOTIFY_CHANGE_SECURITY,
-    FILE_NOTIFY_CHANGE_LAST_ACCESS,
-    WINDOWS_ACTIONS_NAMES,
-)
+from ubuntuone.platform.windows import filesystem_notifications as fs
+from ubuntuone.platform.windows.filesystem_notifications import Watch
 
 #create a rever mapping to use it in the tests.
 REVERSE_WINDOWS_ACTIONS = {}
-for key, value in filesystem_notifications.WINDOWS_ACTIONS.iteritems():
+for key, value in fs.WINDOWS_ACTIONS.iteritems():
     REVERSE_WINDOWS_ACTIONS[value] = key
 
 
@@ -101,6 +88,25 @@
         if self.main_thread_id:
             assert self.main_thread_id == thread.get_ident()
 
+class MementoWatch(Watch):
+    """A watch that will remember all the events raised."""
+
+    def __init__(self, watch_descriptor, path, mask, processor, buf_size=8192,
+                 all_events_count=0):
+        """Create a new instance."""
+        super(MementoWatch, self).__init__(watch_descriptor, path, mask,
+                                           processor, buf_size)
+        self.all_events_count = all_events_count
+        self._all_events = []
+        self.deferred = defer.Deferred()
+
+    def _process_events(self, events):
+        """Process evens from the queue."""
+        self._all_events.extend(events)
+        super(MementoWatch, self)._process_events(events)
+        if len(self._all_events) == self.all_events_count:
+            self.deferred.callback(self._all_events)
+
 
 class TestWatch(BaseTwistedTestCase):
     """Test the watch so that it returns the same events as pyinotify."""
@@ -111,18 +117,18 @@
     def setUp(self):
         yield super(TestWatch, self).setUp()
         self.basedir = self.mktemp('test_root')
-        self.mask = FILE_NOTIFY_CHANGE_FILE_NAME | \
-            FILE_NOTIFY_CHANGE_DIR_NAME | \
-            FILE_NOTIFY_CHANGE_ATTRIBUTES | \
-            FILE_NOTIFY_CHANGE_SIZE | \
-            FILE_NOTIFY_CHANGE_LAST_WRITE | \
-            FILE_NOTIFY_CHANGE_SECURITY | \
-            FILE_NOTIFY_CHANGE_LAST_ACCESS
+        self.mask = fs.FILE_NOTIFY_CHANGE_FILE_NAME | \
+            fs.FILE_NOTIFY_CHANGE_DIR_NAME | \
+            fs.FILE_NOTIFY_CHANGE_ATTRIBUTES | \
+            fs.FILE_NOTIFY_CHANGE_SIZE | \
+            fs.FILE_NOTIFY_CHANGE_LAST_WRITE | \
+            fs.FILE_NOTIFY_CHANGE_SECURITY | \
+            fs.FILE_NOTIFY_CHANGE_LAST_ACCESS
         self.memento = MementoHandler()
         self.memento.setLevel(logging.DEBUG)
         self.raw_events = []
         self.paths_checked = []
-        old_is_dir = Watch._path_is_dir
+        old_is_dir = fs.Watch._path_is_dir
 
         def file_notify_information_wrapper(buf, data):
             """Wrapper that gets the events and adds them to the list."""
@@ -131,7 +137,7 @@
             # If we use extend we wont have the same logging because it will
             # group all events in a single lists which is not what the COM API
             # does.
-            str_events = [(WINDOWS_ACTIONS_NAMES[action], path) for action, path in
+            str_events = [(fs.WINDOWS_ACTIONS_NAMES[action], path) for action, path in
                           events]
             self.raw_events.append(str_events)
             return events
@@ -142,46 +148,55 @@
             self.paths_checked.append((path, result))
             return result
 
-        self.patch(filesystem_notifications, 'FILE_NOTIFY_INFORMATION',
+        self.patch(fs, 'FILE_NOTIFY_INFORMATION',
                    file_notify_information_wrapper)
-        self.patch(filesystem_notifications.Watch, '_path_is_dir',
+        self.patch(fs.Watch, '_path_is_dir',
                    path_is_dir_wrapper)
 
     @defer.inlineCallbacks
-    def _perform_operations(self, path, mask, auto_add, actions,
-                            number_events):
+    def _perform_operations(self, path, mask, actions, filtered_events_count,
+                            children_paths=[], all_events_count=0, ignored_paths=[]):
         """Perform the file operations and returns the recorded events."""
-        handler = TestCaseHandler(number_events=number_events)
-        manager = WatchManager(handler)
-        yield manager.add_watch(os_helper.get_windows_valid_path(path), mask,
-                          auto_add=auto_add)
+        handler = TestCaseHandler(number_events=filtered_events_count)
+        manager = fs.WatchManager(handler)
+        self.memento_watch = None
+
+        def mementowatch_factory(all_events_count, *args, **kwargs):
+            """Factory that creates a new Watch for the tests."""
+            kwargs['all_events_count'] = all_events_count
+            self.memento_watch = MementoWatch(*args, **kwargs)
+            return self.memento_watch
+
+        # lets always use the memento watch, that way we know the number of events in total
+        self.patch(fs, 'Watch', lambda *args, **kwargs:
+                   mementowatch_factory(all_events_count, *args, **kwargs))
+
+        yield manager.add_watch(os_helper.get_windows_valid_path(path), mask)
+        self.memento_watch.all_events_count = all_events_count
+
+        # add a watch to the children paths
+        for child_path in children_paths:
+            yield manager.add_watch(os_helper.get_windows_valid_path(child_path),
+                                    mask)
+
+        # add that paths that will be ignored in the watch
+        for ignored in ignored_paths:
+            self.memento_watch.ignore_path(ignored)
+
         # change the logger so that we can check the logs if we wanted
-        manager._wdm[0].log.addHandler(self.memento)
+        self.memento_watch.log.addHandler(self.memento)
         # clean logger later
-        self.addCleanup(manager._wdm[0].log.removeHandler, self.memento)
+        self.addCleanup(self.memento_watch.log.removeHandler, self.memento)
         # execution the actions
         actions()
         # process the recorded events
-        ret = yield handler.deferred
+        events = yield self.memento_watch.deferred
+        if filtered_events_count != 0:
+            ret = yield handler.deferred
+        else:
+            ret = handler.processed_events
         self.addCleanup(manager.stop)
-        defer.returnValue(ret)
-
-    def _perform_timed_operations(self, path, mask, auto_add, actions,
-                                  time_out):
-        """Perform the file operations and returns the recorded events."""
-        manager = WatchManager()
-        manager.add_watch(os_helper.get_windows_valid_path(path), mask,
-                          auto_add=auto_add)
-        # change the logger so that we can check the logs if we wanted
-        manager._wdm[0].log.addHandler(self.memento)
-        # clean logger later
-        self.addCleanup(manager._wdm[0].log.removeHandler, self.memento)
-        # execution the actions
-        actions()
-        # process the recorded events
-        time.sleep(time_out)
-        events = self.handler.processed_events
-        return events
+        defer.returnValue((ret, events))
 
     def _assert_logs(self, events):
         """Assert the debug logs."""
@@ -208,8 +223,10 @@
             os.fsync(fd)
             fd.close()
 
-        events = yield self._perform_operations(self.basedir, self.mask, False,
-            create_file, 1)
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                            create_file,
+                                                            filtered_events_count=1,
+                                                            all_events_count=1)
         event = events[0]
         self.assertFalse(event.dir)
         self.assertEqual(0x100, event.mask)
@@ -230,8 +247,10 @@
             """Action for the test."""
             os.mkdir(dir_name)
 
-        events = yield self._perform_operations(self.basedir, self.mask, False,
-            create_dir, 1)
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                            create_dir,
+                                                            filtered_events_count=1,
+                                                            all_events_count=1)
         event = events[0]
         self.assertTrue(event.dir)
         self.assertEqual(0x40000100, event.mask)
@@ -254,8 +273,10 @@
             """Action for the test."""
             os.remove(file_name)
 
-        events = yield self._perform_operations(self.basedir, self.mask, False,
-                                                remove_file, 1)
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                            remove_file,
+                                                            filtered_events_count=1,
+                                                            all_events_count=1)
         event = events[0]
         self.assertFalse(event.dir)
         self.assertEqual(0x200, event.mask)
@@ -278,8 +299,10 @@
             """Action for the test."""
             os.rmdir(dir_name)
 
-        events = yield self._perform_operations(self.basedir, self.mask, False,
-                                                remove_dir, 1)
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                            remove_dir,
+                                                            filtered_events_count=1,
+                                                            all_events_count=1)
         event = events[0]
         self.assertTrue(event.dir)
         self.assertEqual(0x40000200, event.mask)
@@ -304,8 +327,10 @@
             fd.write('test')
             fd.close()
 
-        events = yield self._perform_operations(self.basedir, self.mask, False,
-                                                write_file, 1)
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                            write_file,
+                                                            filtered_events_count=1,
+                                                            all_events_count=1)
         event = events[0]
         self.assertFalse(event.dir)
         self.assertEqual(0x2, event.mask)
@@ -331,8 +356,10 @@
             """Action for the test."""
             os.rename(from_file_name, to_file_name)
 
-        events = yield self._perform_operations(self.basedir, self.mask,
-                                                False, move_file, 2)
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                            move_file,
+                                                            filtered_events_count=2,
+                                                            all_events_count=2)
         move_from_event = events[0]
         move_to_event = events[1]
         # first test the move from
@@ -376,8 +403,10 @@
         # while on linux we will have to do some sort of magic like facundo
         # did, on windows we will get a deleted event which is much more
         # cleaner, first time ever windows is nicer!
-        events = yield self._perform_operations(self.basedir, self.mask, False,
-                                                move_file, 1)
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                            move_file,
+                                                            filtered_events_count=1,
+                                                            all_events_count=1)
         event = events[0]
         self.assertFalse(event.dir)
         self.assertEqual(0x200, event.mask)
@@ -406,8 +435,10 @@
 
         # while on linux we have to do some magic operations like facundo did
         # on windows we have a created file, hurray!
-        events = yield self._perform_operations(self.basedir, self.mask, False,
-                                                move_files, 1)
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                            move_files,
+                                                            filtered_events_count=1,
+                                                            all_events_count=1)
         event = events[0]
         self.assertFalse(event.dir)
         self.assertEqual(0x100, event.mask)
@@ -433,8 +464,10 @@
             """Action for the test."""
             os.rename(from_dir_name, to_dir_name)
 
-        events = yield self._perform_operations(self.basedir,
-            self.mask, False, move_file, 2)
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                            move_file,
+                                                            filtered_events_count=2,
+                                                            all_events_count=2)
         move_from_event = events[0]
         move_to_event = events[1]
         # first test the move from
@@ -475,8 +508,10 @@
                 'test_dir_moved_to_not_watched_dir'))
 
         # on windows a move to outside a watched dir translates to a remove
-        events = yield self._perform_operations(self.basedir, self.mask, False,
-                                                move_dir, 1)
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                            move_dir,
+                                                            filtered_events_count=1,
+                                                            all_events_count=1)
         event = events[0]
         self.assertTrue(event.dir)
         self.assertEqual(0x40000200, event.mask)
@@ -501,8 +536,10 @@
             """Action for the test."""
             os.rename(from_dir_name, to_dir_name)
 
-        events = yield self._perform_operations(self.basedir, self.mask, False,
-                                                move_dir, 1)
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                            move_dir,
+                                                            filtered_events_count=1,
+                                                            all_events_count=1)
         event = events[0]
         self.assertTrue(event.dir)
         self.assertEqual(0x40000100, event.mask)
@@ -513,22 +550,6 @@
                          event.pathname)
         self.assertEqual(0, event.wd)
 
-    def test_exclude_filter(self):
-        """Test that the exclude filter works as expectd."""
-        handler = TestCaseHandler(number_events=0)
-        manager = WatchManager(handler)
-        # add a watch that will always exclude all actions
-        manager.add_watch(os_helper.get_windows_valid_path(self.basedir),
-                          self.mask, auto_add=True,
-                          exclude_filter=lambda x: True)
-        # execution the actions
-        file_name = os.path.join(self.basedir, 'test_file_create')
-        open(file_name, 'w').close()
-        # give some time for the system to get the events
-        time.sleep(1)
-        self.assertEqual(0, len(handler.processed_events))
-    test_exclude_filter.skip = "we must rethink this test."
-
     def test_open_dir_muted(self):
         """Test that the opening of dirs is ignored."""
         dir_name = os.path.join(tempfile.mkdtemp(), 'test_dir_open')
@@ -539,134 +560,192 @@
             """Action for the test."""
             os.startfile(dir_name)
 
-        events = self._perform_timed_operations(self.basedir, self.mask, False,
-            open_dir, 2)
+        events = self._perform_operations(self.basedir, self.mask,
+                                          open_dir,
+                                          filtered_events_count=2,
+                                          all_events_count=1)
         self.assertEqual(0, len(events))
     test_open_dir_muted.skip = "we must rethink this test."
 
+    @defer.inlineCallbacks
     def test_ignore_path(self):
         """Test that events from a path are ignored."""
-        events = []
-
-        def fake_processor(event):
-            """Memorize the processed events."""
-            events.append(event)
-
-        path = u'\\\\?\\C:\\path'  # a valid windows path
-        child = u'child'
-        watch = Watch(1, path, None, True, fake_processor)
-        watch.ignore_path(os.path.join(path, child))
-        paths_to_ignore = []
-        for file_name in 'abcdef':
-            paths_to_ignore.append((1, os.path.join(child, file_name)))
-        # ensure that the watch is watching
-        watch._watching = True
-        watch._process_events(paths_to_ignore)
-        self.assertEqual(0, len(events),
-                         'All events should have been ignored.')
-
+        child_path = os.path.join(self.basedir, 'child')
+        os.mkdir(child_path)
+        file_names = 'abcdef'
+
+        def create_files():
+            """Create files to get events."""
+            for file_name in file_names:
+                # create a file similar to the ignore, but not a dir to test if 
+                # we do not match them wrongly
+                fd = open(os.path.join(child_path, file_name), 'w')
+                fd.flush()
+                os.fsync(fd)
+                fd.close()
+
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                       create_files,
+                                       filtered_events_count=0,
+                                       all_events_count= len(file_names) * 2,
+                                       children_paths=[child_path],
+                                       ignored_paths=[u'\\\\?\\' + child_path])
+
+        not_processed_children = [file_name for event, file_name in all_events]
+        for file_name in file_names:
+            self.assertIn(os.path.join('child', file_name), not_processed_children)
+
+    @defer.inlineCallbacks
     def test_not_ignore_path(self):
         """Test that we do get the events when they do not match."""
-        events = []
-
-        def fake_processor(event):
-            """Memorize the processed events."""
-            events.append(event)
-
-        path = u'\\\\?\\C:\\path'  # a valid windows path
-        child = u'child'
-        watch = Watch(1, path, None, True, fake_processor)
-        watch.ignore_path(os.path.join(path, child))
-        paths_not_to_ignore = []
-        for file_name in 'abcdef':
-            paths_not_to_ignore.append((1, os.path.join(
-                                                child + file_name, file_name)))
-        # ensure that the watch is watching
-        watch._watching = True
-        watch._process_events(paths_not_to_ignore)
-        self.assertEqual(len(paths_not_to_ignore), len(events),
-                         'No events should have been ignored.')
-
+        child_path = os.path.join(self.basedir, 'child')
+        os.mkdir(child_path)
+        file_names = 'abcdef'
+
+        def create_files():
+            """Create files to get events."""
+            for file_name in file_names:
+                # create a file similar to the ignore, but not a dir to test if 
+                # we do not match them wrongly
+                fd = open(child_path + file_name, 'w')
+                fd.flush()
+                os.fsync(fd)
+                fd.close()
+
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                       create_files,
+                                       filtered_events_count=len(file_names),
+                                       all_events_count= len(file_names),
+                                       children_paths=[child_path],
+                                       ignored_paths=[u'\\\\?\\' + child_path])
+
+        processed_children = [e.pathname for e in events]
+        for file_name in file_names:
+            self.assertIn(child_path + file_name, processed_children)
+
+
+    @defer.inlineCallbacks
     def test_mixed_ignore_path(self):
         """Test that we do get the correct events."""
-        events = []
-
-        def fake_processor(event):
-            """Memorize the processed events."""
-            events.append(event.pathname)
-
-        child = u'child'
-        path = u'\\\\?\\C:\\path\\'  # a valid windows path
-        watch = Watch(1, path, None, True, fake_processor)
-        watch.ignore_path(os.path.join(path, child))
-        paths_not_to_ignore = []
-        paths_to_ignore = []
-        expected_events = []
-        for file_name in 'abcdef':
-            valid = os.path.join(child + file_name, file_name)
-            paths_to_ignore.append((1, os.path.join(child, file_name)))
-            paths_not_to_ignore.append((1, valid))
-            expected_events.append(os.path.join('C:\\path', valid))
-        # ensure that the watch is watching
-        watch._watching = True
-        watch._process_events(paths_not_to_ignore)
-        self.assertEqual(len(paths_not_to_ignore), len(events),
-                         'Wrong number of events ignored.')
-        self.assertTrue(all([event in expected_events for event in events]),
-                        'Paths ignored that should have not been ignored.')
-
+        child_path = os.path.join(self.basedir, 'child')
+        os.mkdir(child_path)
+        ignored_path = os.path.join(self.basedir, 'ignored') 
+        os.mkdir(ignored_path)
+        file_names = 'abcdef'
+
+        def create_files():
+            """Create files to get events."""
+            for file_name in file_names:
+                # create a file per child path
+                fd = open(os.path.join(child_path, file_name), 'w')
+                fd.flush()
+                os.fsync(fd)
+                fd.close()
+                fd = open(os.path.join(ignored_path, file_name), 'w')
+                fd.flush()
+                os.fsync(fd)
+                fd.close()
+        
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                       create_files,
+                                       filtered_events_count=len(file_names),
+                                       all_events_count= len(file_names * 2) * 2,
+                                       children_paths=[child_path, ignored_path],
+                                       ignored_paths=[u'\\\\?\\' + ignored_path])
+
+        processed_children = [e.pathname for e in events]
+
+        # assert that each expected child path is present
+        for file_name in file_names:
+            self.assertIn(os.path.join(child_path, file_name), processed_children)
+
+        # assert that the ignored are not present 
+        for file_name in file_names:
+            self.assertNotIn(os.path.join(ignored_path, file_name), processed_children)
+
+        # assert that the ignored path are indeed in all the events
+        not_processed_children = [file_name for event, file_name in all_events]
+
+        for file_name in file_names:
+            self.assertIn(os.path.join('ignored', file_name), not_processed_children)
+
+    @defer.inlineCallbacks
     def test_undo_ignore_path_ignored(self):
         """Test that we do deal with events from and old ignored path."""
-        events = []
-
-        def fake_processor(event):
-            """Memorize the processed events."""
-            events.append(event)
-
-        path = u'\\\\?\\C:\\path'  # a valid windows path
-        child = u'child'
-        watch = Watch(1, path, None, True, fake_processor)
-        watch.ignore_path(os.path.join(path, child))
-        watch.remove_ignored_path(os.path.join(path, child))
-        paths_not_to_ignore = []
-        for file_name in 'abcdef':
-            paths_not_to_ignore.append((1, os.path.join(child, file_name)))
-        # ensure that the watch is watching
-        watch._watching = True
-        watch._process_events(paths_not_to_ignore)
-        self.assertEqual(len(paths_not_to_ignore), len(events),
-                         'All events should have been accepted.')
-
+        child_path = os.path.join(self.basedir, 'child')
+        os.mkdir(child_path)
+        file_names = 'abcdef'
+
+        def create_files():
+            """Create files to get events."""
+            self.memento_watch.remove_ignored_path(u'\\\\?\\' + child_path)
+            for file_name in file_names:
+                # create a file similar to the ignore, but not a dir to test if 
+                # we do not match them wrongly
+                fd = open(os.path.join(child_path, file_name), 'w')
+                fd.flush()
+                os.fsync(fd)
+                fd.close()
+
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                       create_files,
+                                       filtered_events_count=len(file_names),
+                                       all_events_count= len(file_names) * 2,
+                                       children_paths=[child_path],
+                                       ignored_paths=[u'\\\\?\\' + child_path])
+
+        processed_children = [e.pathname for e in events]
+
+        # assert that each expected child path is present
+        for file_name in file_names:
+            self.assertIn(os.path.join(child_path, file_name), processed_children)
+
+    @defer.inlineCallbacks
     def test_undo_ignore_path_other_ignored(self):
         """Test that we can undo and the other path is ignored."""
-        events = []
-
-        def fake_processor(event):
-            """Memorize the processed events."""
-            events.append(event.pathname)
-
-        path = u'\\\\?\\C:\\path'  # a valid windows path
-        child_a = u'childa'
-        child_b = u'childb'
-        watch = Watch(1, path, None, True, fake_processor)
-        watch.ignore_path(os.path.join(path, child_a))
-        watch.ignore_path(os.path.join(path, child_b))
-        watch.remove_ignored_path(os.path.join(path, child_a))
-        paths_to_ignore = []
-        paths_not_to_ignore = []
-        expected_events = []
-        for file_name in 'abcdef':
-            paths_to_ignore.append((1, os.path.join(child_b, file_name)))
-            valid = os.path.join(child_a, file_name)
-            paths_not_to_ignore.append((1, valid))
-            expected_events.append(os.path.join('C:\\path', valid))
-        # ensure that the watch is watching
-        watch._watching = True
-        watch._process_events(paths_not_to_ignore)
-        self.assertEqual(len(paths_not_to_ignore), len(events),
-                         'All events should have been accepted.')
-        self.assertTrue(all([event in expected_events for event in events]),
-                        'Paths ignored that should have not been ignored.')
+        child_path = os.path.join(self.basedir, 'child')
+        os.mkdir(child_path)
+        ignored_path = os.path.join(self.basedir, 'ignored') 
+        os.mkdir(ignored_path)
+        file_names = 'abcdef'
+
+        def create_files():
+            """Create files to get events."""
+            self.memento_watch.remove_ignored_path(u'\\\\?\\' + child_path)
+            for file_name in file_names:
+                # create a file per child path
+                fd = open(os.path.join(child_path, file_name), 'w')
+                fd.flush()
+                os.fsync(fd)
+                fd.close()
+                fd = open(os.path.join(ignored_path, file_name), 'w')
+                fd.flush()
+                os.fsync(fd)
+                fd.close()
+        
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                       create_files,
+                                       filtered_events_count=len(file_names),
+                                       all_events_count= len(file_names * 2) * 2,
+                                       children_paths=[child_path, ignored_path],
+                                       ignored_paths=[u'\\\\?\\' + ignored_path,
+                                                      u'\\\\?\\' + child_path])
+
+        processed_children = [e.pathname for e in events]
+
+        # assert that each expected child path is present
+        for file_name in file_names:
+            self.assertIn(os.path.join(child_path, file_name), processed_children)
+
+        # assert that the ignored are not present 
+        for file_name in file_names:
+            self.assertNotIn(os.path.join(ignored_path, file_name), processed_children)
+
+        # assert that the ignored path are indeed in all the events
+        not_processed_children = [file_name for event, file_name in all_events]
+
+        for file_name in file_names:
+            self.assertIn(os.path.join('ignored', file_name), not_processed_children)
 
     @defer.inlineCallbacks
     def test_call_deferred_already_called(self):
@@ -678,7 +757,7 @@
             method_args.append((args, kwargs),)
 
         path = u'\\\\?\\C:\\path'  # a valid windows path
-        watch = Watch(1, path, None, True, None)
+        watch = fs.Watch(1, path, None, None)
         yield watch._watch_started_deferred.callback(True)
         watch._call_deferred(fake_call, None)
         self.assertEqual(0, len(method_args))
@@ -692,20 +771,20 @@
             method_args.append((args, kwargs),)
 
         path = u'\\\\?\\C:\\path'  # a valid windows path
-        watch = Watch(1, path, None, True, None)
+        watch = fs.Watch(1, path, None, None)
         watch._call_deferred(fake_call, None)
         self.assertEqual(1, len(method_args))
 
     def test_started_property(self):
         """Test that the started property returns the started deferred."""
         path = u'\\\\?\\C:\\path'  # a valid windows path
-        watch = Watch(1, path, None, True, None)
+        watch = fs.Watch(1, path, None, None)
         self.assertEqual(watch.started, watch._watch_started_deferred)
 
     def test_stopped_property(self):
         """Test that the stopped property returns the stopped deferred."""
         path = u'\\\\?\\C:\\path'  # a valid windows path
-        watch = Watch(1, path, None, True, None)
+        watch = fs.Watch(1, path, None, None)
         self.assertEqual(watch.stopped, watch._watch_stopped_deferred)
 
     def random_error(self, *args):
@@ -716,8 +795,8 @@
     def test_start_watching_fails_early_in_thread(self):
         """An early failure inside the thread should errback the deferred."""
         test_path = self.mktemp("test_directory")
-        self.patch(filesystem_notifications, "CreateFileW", self.random_error)
-        watch = Watch(1, test_path, None, True, None)
+        self.patch(fs, "CreateFileW", self.random_error)
+        watch = fs.Watch(1, test_path, None, None)
         d = watch.start_watching()
         yield self.assertFailure(d, FakeException)
 
@@ -725,9 +804,9 @@
     def test_start_watching_fails_late_in_thread(self):
         """A late failure inside the thread should errback the deferred."""
         test_path = self.mktemp("test_directory")
-        self.patch(filesystem_notifications, "ReadDirectoryChangesW",
+        self.patch(fs, "ReadDirectoryChangesW",
                    self.random_error)
-        watch = Watch(1, test_path, None, True, None)
+        watch = fs.Watch(1, test_path, None, None)
         d = watch.start_watching()
         yield self.assertFailure(d, FakeException)
 
@@ -736,12 +815,12 @@
         """CloseHandle is called when there's an error in the watch thread."""
         test_path = self.mktemp("test_directory")
         close_called = []
-        self.patch(filesystem_notifications, "CreateFileW", lambda *_: None)
-        self.patch(filesystem_notifications, "CloseHandle",
+        self.patch(fs, "CreateFileW", lambda *_: None)
+        self.patch(fs, "CloseHandle",
                    close_called.append)
-        self.patch(filesystem_notifications, "ReadDirectoryChangesW",
+        self.patch(fs, "ReadDirectoryChangesW",
                    self.random_error)
-        watch = Watch(1, test_path, self.mask, True, None)
+        watch = fs.Watch(1, test_path, self.mask, None)
         d = watch.start_watching()
         yield self.assertFailure(d, FakeException)
         self.assertEqual(len(close_called), 1)
@@ -751,7 +830,7 @@
     def test_stop_watching_fired_when_watch_thread_finishes(self):
         """The deferred returned is fired when the watch thread finishes."""
         test_path = self.mktemp("another_test_directory")
-        watch = Watch(1, test_path, self.mask, True, None)
+        watch = fs.Watch(1, test_path, self.mask, None)
         yield watch.start_watching()
         self.assertNotEqual(watch._watch_handle, None)
         yield watch.stop_watching()
@@ -762,7 +841,7 @@
         path = u'\\\\?\\C:\\path\\to\\no\\dir'
         test_path = self.mktemp("test_directory")
         self.patch(os.path, 'exists', lambda path: False)
-        watch = Watch(1, test_path, self.mask, True, None)
+        watch = fs.Watch(1, test_path, self.mask, None)
         self.assertFalse(watch._path_is_dir(path))
 
     def test_is_path_dir_missing_in_subdir(self):
@@ -770,8 +849,9 @@
         path = u'\\\\?\\C:\\path\\to\\no\\dir'
         test_path = self.mktemp("test_directory")
         self.patch(os.path, 'exists', lambda path: False)
-        watch = Watch(1, test_path, self.mask, True, None)
-        watch._subdirs.add(path)
+        watch = fs.Watch(1, test_path, self.mask, None)
+        child_path = watch._get_partial_child_path_dir(path)
+        watch._subdirs[child_path] = False
         self.assertTrue(watch._path_is_dir(path))
 
     def test_is_path_dir_present_is_dir(self):
@@ -780,8 +860,8 @@
         test_path = self.mktemp("test_directory")
         self.patch(os.path, 'exists', lambda path: True)
         self.patch(os.path, 'isdir', lambda path: True)
-        watch = Watch(1, test_path, self.mask, True, None)
-        watch._subdirs.add(path)
+        watch = fs.Watch(1, test_path, self.mask, None)
+        watch._subdirs[path] = False
         self.assertTrue(watch._path_is_dir(path))
 
     def test_is_path_dir_present_no_dir(self):
@@ -790,34 +870,36 @@
         test_path = self.mktemp("test_directory")
         self.patch(os.path, 'exists', lambda path: True)
         self.patch(os.path, 'isdir', lambda path: False)
-        watch = Watch(1, test_path, self.mask, True, None)
-        watch._subdirs.add(path)
+        watch = fs.Watch(1, test_path, self.mask, None)
+        watch._subdirs[path] = False
         self.assertFalse(watch._path_is_dir(path))
 
     def test_update_subdirs_create_not_present(self):
         """Test when we update on a create event and not present."""
         path = u'\\\\?\\C:\\path\\to\\no\\dir'
         test_path = self.mktemp("test_directory")
-        watch = Watch(1, test_path, self.mask, True, None)
-        watch._update_subdirs(path, REVERSE_WINDOWS_ACTIONS[IN_CREATE])
-        self.assertTrue(path in watch._subdirs)
+        watch = fs.Watch(1, test_path, self.mask, None)
+        child_path = watch._get_partial_child_path_dir(path)
+        watch._update_subdirs(child_path, REVERSE_WINDOWS_ACTIONS[IN_CREATE])
+        self.assertTrue(child_path in watch._subdirs)
 
     def test_update_subdirs_create_present(self):
         """Test when we update on a create event and is present."""
         path = u'\\\\?\\C:\\path\\to\\no\\dir'
         test_path = self.mktemp("test_directory")
-        watch = Watch(1, test_path, self.mask, True, None)
-        watch._subdirs.add(path)
+        watch = fs.Watch(1, test_path, self.mask, None)
+        child_path = watch._get_partial_child_path_dir(path)
+        watch._subdirs[child_path] = False
         old_length = len(watch._subdirs)
-        watch._update_subdirs(path, REVERSE_WINDOWS_ACTIONS[IN_CREATE])
-        self.assertTrue(path in watch._subdirs)
+        watch._update_subdirs(child_path, REVERSE_WINDOWS_ACTIONS[IN_CREATE])
+        self.assertTrue(child_path in watch._subdirs)
         self.assertEqual(old_length, len(watch._subdirs))
 
     def test_update_subdirs_delete_not_present(self):
         """Test when we delete and is not present."""
         path = u'\\\\?\\C:\\path\\to\\no\\dir'
         test_path = self.mktemp("test_directory")
-        watch = Watch(1, test_path, self.mask, True, None)
+        watch = fs.Watch(1, test_path, self.mask, None)
         watch._update_subdirs(path, REVERSE_WINDOWS_ACTIONS[IN_DELETE])
         self.assertTrue(path not in watch._subdirs)
 
@@ -825,11 +907,203 @@
         """Test when we delete and is present."""
         path = u'\\\\?\\C:\\path\\to\\no\\dir'
         test_path = self.mktemp("test_directory")
-        watch = Watch(1, test_path, self.mask, True, None)
-        watch._subdirs.add(path)
+        watch = fs.Watch(1, test_path, self.mask, None)
+        child_path = watch._get_partial_child_path_dir(path)
+        watch._subdirs[child_path] = False
         watch._update_subdirs(path, REVERSE_WINDOWS_ACTIONS[IN_DELETE])
         self.assertTrue(path not in watch._subdirs)
 
+    @defer.inlineCallbacks
+    def test_events_ignored_no_virtual_watch(self):
+        """Test when the events of a subdir are ignored."""
+        child_folder = os.path.join(self.basedir, 'child')
+        os.mkdir(child_folder)
+        file_name = os.path.join(child_folder, 'test_file_create')
+
+        def create_file():
+            """Action used for the test."""
+            # simply create a new file
+            fd = open(file_name, 'w')
+            fd.flush()
+            os.fsync(fd)
+            fd.close()
+
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                            create_file,
+                                                            filtered_events_count=0,
+                                                            all_events_count=2)
+        self.assertEqual(0, len(events), events)
+        # assert the logging
+        self._assert_logs(events)
+
+    @defer.inlineCallbacks
+    def test_file_events_raised_no_virtual_watch(self):
+        """Test when the events of files are fired but not suddirs."""
+        child_folder = os.path.join(self.basedir, 'child')
+        os.mkdir(child_folder)
+        file_name = os.path.join(self.basedir, 'test_file_create')
+        child_file_name = os.path.join(child_folder, 'test_file_create')
+
+        def create_file():
+            """Action used for the test."""
+            # simply create a new file
+            fd = open(file_name, 'w')
+            fd.flush()
+            os.fsync(fd)
+            fd.close()
+            # do the same with the file that should be ignored
+            fd = open(child_file_name, 'w')
+            fd.flush()
+            os.fsync(fd)
+            fd.close()
+
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                            create_file,
+                                                            filtered_events_count=1,
+                                                            all_events_count=2)
+        self.assertEqual(1, len(events))
+        event = events[0]
+        self.assertFalse(event.dir)
+        self.assertEqual(0x100, event.mask)
+        self.assertEqual('IN_CREATE', event.maskname)
+        self.assertEqual(os.path.split(file_name)[1], event.name)
+        self.assertEqual('.', event.path)
+        self.assertEqual(os.path.join(self.basedir, file_name), event.pathname)
+        self.assertEqual(0, event.wd)
+        # assert the logging
+        self._assert_logs(events)
+
+    @defer.inlineCallbacks
+    def test_events_ignored_deep_tree(self):
+        """Test that all the events if a deep tree are ignored."""
+        child_folder = os.path.join(self.basedir, *[c for c in string.letters])
+        os.makedirs(child_folder)
+
+        def create_files():
+            """Action used for the test."""
+            # create a file per level of the child folder
+            current_path = self.basedir
+            for c in string.letters[:24]:
+                current_path = os.path.join(current_path, c)
+                file_name = os.path.join(current_path, 'test_file_create')
+                fd = open(file_name, 'w')
+                fd.flush()
+                os.fsync(fd)
+                fd.close()
+
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                      create_files, filtered_events_count=0,
+                                      all_events_count=len(string.letters[:24]))
+        self.assertEqual(0, len(events), str(events))
+        # assert the logging
+        self._assert_logs(events)
+
+    @defer.inlineCallbacks
+    def test_events_raised_virtual_watch(self):
+        """Test that the events from a virtual watch are raised."""
+        child_folder = os.path.join(self.basedir, 'child')
+        os.mkdir(child_folder)
+        file_name = os.path.join(self.basedir, 'test_file_create')
+        child_file_name = os.path.join(child_folder, 'test_file_create')
+
+        def create_file():
+            """Action used for the test."""
+            # simply create a new file
+            fd = open(file_name, 'w')
+            fd.flush()
+            os.fsync(fd)
+            fd.close()
+            # do the same with the file that should be ignored
+            fd = open(child_file_name, 'w')
+            fd.flush()
+            os.fsync(fd)
+            fd.close()
+
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                    create_file,
+                                                    filtered_events_count=2,
+                                                    children_paths=[child_folder],
+                                                    all_events_count=2)
+        self.assertEqual(2, len(events))
+        # assert the logging
+        self._assert_logs(events)
+
+    @defer.inlineCallbacks
+    def test_start_watching_race_condition(self):
+        """Test that we can deal with a race condition in the start method."""
+        # create a dir with several child folders and add a watch. We will
+        # patch the native_listdir so that we do not see some of the child dirs
+        # and later will perfrom an event on one of those kids. The event
+        # should be filtered.
+        for c in string.letters[:24]:
+            os.mkdir(os.path.join(self.basedir, c))
+
+        forgotten_dirs = set(string.letters[15:22])
+
+        def slow_listdir(path):
+            """Similar to list dir but ignore some dirs."""
+            all_dirs = set(os.listdir(path))
+            return all_dirs - forgotten_dirs
+
+        self.patch(fs, 'native_listdir', slow_listdir)
+
+        # lets perform a number of events in the forgotten dirs
+        def create_files():
+            """Create a file on each forgotten dir."""
+            for c in forgotten_dirs:
+                file_name = os.path.join(self.basedir, c, 'test.txt')
+                fd = open(file_name, 'w')
+                fd.flush()
+                os.fsync(fd)
+                fd.close()
+
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                    create_files,
+                                                    filtered_events_count=0,
+                                                    all_events_count=len(forgotten_dirs) * 2)
+        # assert that the watch does have the correct subdirs
+        for c in string.letters[:24]:
+            self.assertIn(os.path.join(c, u''), self.memento_watch._subdirs) 
+        self.assertEqual(0, len(events))
+
+    def test_add_child_watch_race_condition(self):
+        """Test that we can deal with a race condition in the add child."""
+        # like with test_Start_watching race condition we are going to add a
+        # watch and a child watch. In the second addintion the data returned by
+        # the native_listdir will be wrong.
+        child_path = os.path.join(self.basedir, 'child_path')
+        os.mkdir(child_path)
+        for c in string.letters[:24]:
+            os.mkdir(os.path.join(child_path), c)
+
+        forgotten_dirs = set(string.letters[15:22])
+
+        def slow_listdir(path):
+            """Similar to list dir but ignore some dirs."""
+            # are we adding a child or stating?
+            if path == self.basedir:
+                return os.listdir(path)
+            else:
+                all_dirs = set(os.listdir(path))
+                return all_dirs - forgotten_dirs
+
+        def create_files():
+            """Create a file on each forgotten dir."""
+            for c in forgotten_dirs:
+                file_name = os.path.join(self.basedir, c, 'test.txt')
+                fd = open(file_name, 'w')
+                fd.flush()
+                os.fsync(fd)
+                fd.close()
+
+        events, all_events = yield self._perform_operations(self.basedir, self.mask,
+                                                    create_files,
+                                                    filtered_events_count=0,
+                                                    all_events_count=len(forgotten_dirs) * 2)
+        for c in string.letters[:24]:
+            self.assertIn(os.path.join(c, u''), self.memento_watch._subdirs) 
+        self.assertEqual(0, len(events))
+
 
 class TestWatchManager(BaseTwistedTestCase):
     """Test the watch manager."""
@@ -840,8 +1114,8 @@
         yield super(TestWatchManager, self).setUp()
         self.parent_path = u'\\\\?\\C:\\'  # a valid windows path
         self.path = self.parent_path + u'path'
-        self.watch = Watch(1, self.path, None, True, None)
-        self.manager = WatchManager(None)
+        self.watch = fs.Watch(1, self.path, None, None)
+        self.manager = fs.WatchManager(None)
         self.manager._wdm = {1: self.watch}
 
     @defer.inlineCallbacks
@@ -854,7 +1128,7 @@
             self.was_called = True
             return defer.succeed(True)
 
-        self.patch(Watch, "stop_watching", fake_stop_watching)
+        self.patch(fs.Watch, "stop_watching", fake_stop_watching)
         yield self.manager.stop()
         self.assertTrue(self.was_called, 'The watch stop should be called.')
 
@@ -866,9 +1140,9 @@
             """Another fake stop watch."""
             return watch.stopped
 
-        self.patch(Watch, "stop_watching", fake_stop_watching)
+        self.patch(fs.Watch, "stop_watching", fake_stop_watching)
         second_path = self.parent_path + u"second_path"
-        second_watch = Watch(2, second_path, None, True, None)
+        second_watch = fs.Watch(2, second_path, None, None)
         self.manager._wdm[2] = second_watch
         d = self.manager.stop()
         self.assertFalse(d.called, "Not fired before all watches end")
@@ -908,24 +1182,21 @@
             """Fake start watch."""
             self.was_called = True
 
-        self.patch(Watch, "start_watching", fake_start_watching)
+        self.patch(fs.Watch, "start_watching", fake_start_watching)
         self.manager._wdm = {}
 
         mask = 'bit_mask'
-        auto_add = True
-        self.manager.add_watch(self.path, mask, auto_add=auto_add)
+        self.manager.add_watch(self.path, mask)
         self.assertEqual(1, len(self.manager._wdm))
         self.assertTrue(self.was_called, 'The watch start was not called.')
         self.assertEqual(self.path + os.path.sep, self.manager._wdm[0]._path)
         self.assertEqual(mask, self.manager._wdm[0]._mask)
-        self.assertEqual(auto_add, self.manager._wdm[0]._auto_add)
 
     def test_update_present_watch(self):
         """Test the update of a present watch."""
         mask = 'mask'
         self.assertRaises(NotImplementedError, self.manager.update_watch,
                                                                  1, mask)
-
     def test_get_watch_present_wd(self):
         """Test that the correct path is returned."""
         self.assertEqual(self.path + os.path.sep, self.manager.get_path(1))
@@ -1002,11 +1273,12 @@
 
     def test_add_watch_twice(self):
         """Adding a watch twice succeeds when the watch is running."""
-        self.patch(Watch, "start_watching", lambda self: self.started)
-        manager = WatchManager(None)
+        self.patch(fs.Watch, "start_watching", lambda self: self.started)
+        manager = fs.WatchManager(None)
         # no need to stop watching because start_watching is fake
 
-        path = u'\\\\?\\C:\\test'  # a valid windows path
+        path = self.mktemp("test_directory")  # a valid windows path
+        path = u'\\\\?\\' + path
         mask = 'fake bit mask'
         d1 = manager.add_watch(path, mask)
         d2 = manager.add_watch(path, mask)
@@ -1114,7 +1386,7 @@
     def setUp(self):
         """set up the diffeent tests."""
         yield super(TestNotifyProcessor, self).setUp()
-        self.processor = NotifyProcessor(None)
+        self.processor = fs.NotifyProcessor(None)
         self.general = FakeGeneralProcessor()
         self.processor.general_processor = self.general
 
@@ -1466,8 +1738,8 @@
 
     def test_add_watch_twice(self):
         """Check the deferred returned by a second add_watch."""
-        self.patch(Watch, "start_watching", lambda self: self.started)
-        monitor = FilesystemMonitor(None, None)
+        self.patch(fs.Watch, "start_watching", lambda self: self.started)
+        monitor = fs.FilesystemMonitor(None, None)
         # no need to stop watching because start_watching is fake
 
         parent_path = 'C:\\test'  # a valid windows path in utf-8 bytes
@@ -1496,7 +1768,7 @@
 
         ancestors = ['~', '~\\Pictures', '~\\Pictures\\Home', ]
         volume = FakeVolume(ancestors)
-        monitor = FilesystemMonitor(None, None)
+        monitor = fs.FilesystemMonitor(None, None)
         added = yield monitor.add_watches_to_udf_ancestors(volume)
         self.assertTrue(added, 'We should always return true.')
         # lets ensure that we never added the watches

=== modified file 'ubuntuone/platform/windows/filesystem_notifications.py'
--- ubuntuone/platform/windows/filesystem_notifications.py	2011-10-18 13:44:10 +0000
+++ ubuntuone/platform/windows/filesystem_notifications.py	2012-02-03 11:33:18 +0000
@@ -1,9 +1,5 @@
 #
-# Authors: Manuel de la Pena <[e-mail]>
-#          Natalia B. Bidart <[e-mail]>
-#          Alejandro J. Cura <[e-mail]>
-#
-# Copyright 2011 Canonical Ltd.
+# Copyright 2011-2012 Canonical Ltd.
 #
 # This program is free software: you can redistribute it and/or modify it
 # under the terms of the GNU General Public License version 3, as published
@@ -70,6 +66,7 @@
     is_valid_syncdaemon_path,
     is_valid_windows_path,
     get_syncdaemon_valid_path,
+    native_listdir,
     windowspath,
 )
 from ubuntuone import logger
@@ -118,13 +115,13 @@
 
 # the default mask to be used in the watches added by the FilesystemMonitor
 # class
-FILESYSTEM_MONITOR_MASK = FILE_NOTIFY_CHANGE_FILE_NAME | \
-    FILE_NOTIFY_CHANGE_DIR_NAME | \
-    FILE_NOTIFY_CHANGE_ATTRIBUTES | \
-    FILE_NOTIFY_CHANGE_SIZE | \
-    FILE_NOTIFY_CHANGE_LAST_WRITE | \
-    FILE_NOTIFY_CHANGE_SECURITY | \
-    FILE_NOTIFY_CHANGE_LAST_ACCESS
+FILESYSTEM_MONITOR_MASK = (FILE_NOTIFY_CHANGE_FILE_NAME |
+    FILE_NOTIFY_CHANGE_DIR_NAME |
+    FILE_NOTIFY_CHANGE_ATTRIBUTES |
+    FILE_NOTIFY_CHANGE_SIZE |
+    FILE_NOTIFY_CHANGE_LAST_WRITE |
+    FILE_NOTIFY_CHANGE_SECURITY |
+    FILE_NOTIFY_CHANGE_LAST_ACCESS)
 
 THREADPOOL_MAX = 20
 
@@ -133,9 +130,7 @@
 class Watch(object):
     """Implement the same functions as pyinotify.Watch."""
 
-    def __init__(self, watch_descriptor, path, mask, auto_add, processor,
-        buf_size=8192):
-        super(Watch, self).__init__()
+    def __init__(self, watch_descriptor, path, mask, processor, buf_size=8192):
         self.log = logging.getLogger('ubuntuone.SyncDaemon.platform.windows.' +
             'filesystem_notifications.Watch')
         self.log.setLevel(TRACE)
@@ -146,18 +141,20 @@
         self._overlapped.hEvent = CreateEvent(None, 0, 0, None)
         self._watching = False
         self._descriptor = watch_descriptor
-        self._auto_add = auto_add
-        self._ignore_paths = []
+        # ignored paths are those that we actually do want to ignore while
+        # not watched paths are those whose events are not propagated to
+        # mimic the linux behaviour
+        self._ignore_paths = set()
         self._cookie = None
         self._source_pathname = None
         self._process_thread = None
         self._watch_handle = None
         # remember the subdirs we have so that when we have a delete we can
         # check if it was a remove
-        self._subdirs = set()
+        self._subdirs = {}
         # ensure that we work with an abspath and that we can deal with
         # long paths over 260 chars.
-        if not path.endswith(os.path.sep):
+        if path[-1] != os.path.sep:
             path += os.path.sep
         self._path = os.path.abspath(path)
         self._mask = mask
@@ -167,7 +164,18 @@
         # and this one is fired when the watch has stopped
         self._watch_stopped_deferred = defer.Deferred()
 
-    @is_valid_windows_path(path_indexes=[1])
+    def _get_partial_child_path_dir(self, path):
+        """Return a partial child path for a dir.
+
+        The path is used internally by the watcher to keep track of
+        child directories.
+        """
+        if path[-1] != os.path.sep:
+            path += os.path.sep
+        if path.startswith(self._path):
+            path = path[len(self._path):]
+        return path
+
     def _update_subdirs(self, path, event):
         """Adds the path to the internal subdirs.
 
@@ -175,13 +183,13 @@
         will not be checked.
         """
         if WINDOWS_ACTIONS[event] == IN_CREATE:
-            self._subdirs.add(path)
+            self._subdirs[path] = False
         elif WINDOWS_ACTIONS[event] == IN_DELETE and\
                 path in self._subdirs:
-            self._subdirs.remove(path)
+            del self._subdirs[path]
 
     @is_valid_windows_path(path_indexes=[1])
-    def _path_is_dir(self, path):
+    def _path_is_dir(self, full_path):
         """Check if the path is a dir and update the local subdir list."""
 
         # The logic of this function is the following:
@@ -191,14 +199,14 @@
         # 3. We check if a path is a dir by:
         #    * Asking the os if the path exists.
         #    * Finding the path in self._subdirs
-
         is_dir = False
-        if os.path.exists(path):
-            is_dir = os.path.isdir(path)
+        if os.path.exists(full_path):
+            is_dir = os.path.isdir(full_path)
         else:
             # path does not exists, was it in the internal list?
-            is_dir = path in self._subdirs
-        self.log.debug('Is path %r a dir? %s', path, is_dir)
+            child_path = self._get_partial_child_path_dir(full_path)
+            is_dir = child_path in self._subdirs
+        self.log.debug('Is path %r a dir? %s', full_path, is_dir)
         return is_dir
 
     def _process_events(self, events):
@@ -210,8 +218,20 @@
         # we transform the events to be the same as the one in pyinotify
         # and then use the proc_fun
         for action, file_name in events:
-            if any([file_name.startswith(path)
-                        for path in self._ignore_paths]):
+            # ignore those paths that are present in the _ignore_paths, this
+            # are those paths whose watch has been removed.
+            if any(file_name.startswith(path)
+                        for path in self._ignore_paths):
+                continue
+
+            # lets get the parent dir to be faster
+            parent_dir = os.path.dirname(file_name) + os.path.sep
+
+            if parent_dir != unicode(os.path.sep) and\
+                    not self._subdirs.get(parent_dir, False):
+                if not parent_dir in self._subdirs:
+                    self.log.warn('Child path %r is missing.', parent_dir)
+                    self._subdirs[parent_dir] = False
                 continue
             # map the windows events to the pyinotify ones, tis is dirty but
             # makes the multiplatform better, linux was first :P
@@ -221,7 +241,7 @@
             is_dir = self._path_is_dir(full_dir_path)
             if is_dir:
                 # we need to update the list of subdirs that we have
-                self._update_subdirs(full_dir_path, action)
+                self._update_subdirs(file_name + os.path.sep, action)
             mask = WINDOWS_ACTIONS[action]
             head, tail = os.path.split(file_name)
             if is_dir:
@@ -232,6 +252,11 @@
                 'mask': mask,
                 'name': tail,
                 'path': '.'}
+            # if we have a in_modify for a dir that is not watched on linux
+            # we get no events, let's therefore ignore them
+            if (WINDOWS_ACTIONS[action] == IN_MODIFY 
+                    and is_dir and file_name + os.path.sep in self._subdirs):
+                continue
             # by the way in which the win api fires the events we know for
             # sure that no move events will be added in the wrong order, this
             # is kind of hacky, I dont like it too much
@@ -246,9 +271,6 @@
             # FIXME: event deduces the pathname wrong and we need to manually
             # set it
             event.pathname = syncdaemon_path
-            # add the event only if we do not have an exclude filter or
-            # the exclude filter returns False, that is, the event will not
-            # be excluded
             self.log.debug('Pushing event %r to processor.', event)
             self._processor(event)
 
@@ -301,7 +323,7 @@
             ReadDirectoryChangesW(
                 handle,
                 buf,
-                self._auto_add,
+                True,
                 self._mask,
                 self._overlapped,
             )
@@ -316,9 +338,9 @@
             if rc == WAIT_OBJECT_0:
                 # Stop event
                 break
-            # if we continue, it means that we got some data, lets read it
+            # if we continue, it means that we got some data, let's read it
             data = GetOverlappedResult(handle, self._overlapped, True)
-            # lets ead the data and store it in the results
+            # let's read the data and store it in the results
             events = FILE_NOTIFY_INFORMATION(buf, data)
             self.log.debug('Got from ReadDirectoryChangesW %r.',
                     [(WINDOWS_ACTIONS_NAMES[action], path) for action, path in
@@ -328,28 +350,40 @@
     @is_valid_windows_path(path_indexes=[1])
     def ignore_path(self, path):
         """Add the path of the events to ignore."""
-        if not path.endswith(os.path.sep):
-            path += os.path.sep
-        if path.startswith(self._path):
-            path = path[len(self._path):]
-            self._ignore_paths.append(path)
+        path = self._get_partial_child_path_dir(path)
+        self._ignore_paths.add(path)
 
     @is_valid_windows_path(path_indexes=[1])
     def remove_ignored_path(self, path):
         """Reaccept path."""
-        if not path.endswith(os.path.sep):
-            path += os.path.sep
-        if path.startswith(self._path):
-            path = path[len(self._path):]
-            if path in self._ignore_paths:
-                self._ignore_paths.remove(path)
+        path = self._get_partial_child_path_dir(path)
+        if path in self._ignore_paths:
+            self._ignore_paths.remove(path)
+
+    @is_valid_windows_path(path_indexes=[1])
+    def add_child_watch(self, path):
+        """Adds a watch to a child path."""
+        if not os.path.exists(path):
+            self.log.warn('Adding watch to nonexisting path "%s"', path)
+            return
+        # we are adding a watch to a path, but not to the children of that
+        # path, ergo we have to add some extra ignored paths, and then remove
+        # the parent path from the ignored paths
+        for current_child in native_listdir(path):
+            full_child_path = os.path.join(self._path, current_child)
+            if os.path.isdir(full_child_path):
+                self._subdirs[current_child + os.path.sep] = False
+        # let's remove the path from the ignored paths since the children are
+        # the ones ignored.
+        sort_path = self._get_partial_child_path_dir(path)
+        self._subdirs[sort_path] = True
 
     def start_watching(self):
         """Tell the watch to start processing events."""
-        for current_child in os.listdir(self._path):
+        for current_child in native_listdir(self._path):
             full_child_path = os.path.join(self._path, current_child)
             if os.path.isdir(full_child_path):
-                self._subdirs.add(full_child_path)
+                self._subdirs[current_child + os.path.sep] = False
         # start to diff threads, one to watch the path, the other to
         # process the events.
         self.log.debug('Start watching path.')
@@ -362,14 +396,13 @@
         self.log.info('Stop watching %s', self._path)
         SetEvent(self._wait_stop)
         self._watching = False
-        self._subdirs = set()
+        self._subdirs = {}
         return self.stopped
 
-    def update(self, mask, auto_add=False):
+    def update(self, mask):
         """Update the info used by the watcher."""
-        self.log.debug('update(%s, %s)', mask, auto_add)
+        self.log.debug('update(%s, %s)', mask)
         self._mask = mask
-        self._auto_add = auto_add
 
     @property
     def path(self):
@@ -377,10 +410,6 @@
         return self._path
 
     @property
-    def auto_add(self):
-        return self._auto_add
-
-    @property
     def started(self):
         """A deferred that will be called when the watch is running."""
         return self._watch_started_deferred
@@ -433,28 +462,26 @@
         except KeyError, e:
             logging.error(str(e))
 
-    def _add_single_watch(self, path, mask, auto_add=False,
-        quiet=True):
+    def _add_single_watch(self, path, mask, quiet=True):
         if path in self._ignored_paths:
             # simply removed it from the filter
             self._ignored_paths.remove(path)
             return
         # we need to add a new watch
-        self.log.debug('add_single_watch(%s, %s, %s, %s)', path, mask,
-            auto_add, quiet)
+        self.log.debug('add_single_watch(%s, %s, %s)', path, mask,
+            quiet)
 
         # adjust the number of threads based on the UDFs watched
         reactor.suggestThreadPoolSize(THREADPOOL_MAX + self._wd_count + 1)
         self._wdm[self._wd_count] = Watch(self._wd_count, path,
-                                          mask, auto_add, self._processor)
+                                          mask, self._processor)
         d = self._wdm[self._wd_count].start_watching()
         self._wd_count += 1
         return d
 
     @is_valid_windows_path(path_indexes=[1])
-    def add_watch(self, path, mask, auto_add=False,
-        quiet=True):
-        """Add a new path tp be watch.
+    def add_watch(self, path, mask, quiet=True):
+        """Add a new path to be watched.
 
         The method will ensure that the path is not already present.
         """
@@ -464,24 +491,24 @@
         wd = self.get_wd(path)
         if wd is None:
             self.log.debug('Adding single watch on %r', path)
-            return self._add_single_watch(path, mask, auto_add, quiet)
+            return self._add_single_watch(path, mask, quiet)
         else:
             self.log.debug('Watch already exists on %r', path)
+            if path != self._wdm[wd].path:
+                self._wdm[wd].add_child_watch(path)
             return self._wdm[wd].started
 
-    def update_watch(self, wd, mask=None, rec=False,
-                     auto_add=False, quiet=True):
+    def update_watch(self, wd, mask=None, rec=False, quiet=True):
         raise NotImplementedError("Not implemented on windows.")
 
     @is_valid_windows_path(path_indexes=[1])
     def get_wd(self, path):
         """Return the watcher that is used to watch the given path."""
-        if not path.endswith(os.path.sep):
+        if path[-1] != os.path.sep:
             path = path + os.path.sep
         for current_wd in self._wdm:
             watch_path = self._wdm[current_wd].path
-            if ((watch_path == path or (
-                    watch_path in path and self._wdm[current_wd].auto_add))
+            if (watch_path in path
                     and path not in self._ignored_paths):
                 return current_wd
 
@@ -560,10 +587,10 @@
 
     def process_IN_MODIFY(self, event):
         """Capture a modify event and fake an open ^ close write events."""
-        # lets ignore dir changes
+        # let's ignore dir changes
         if event.dir:
             return
-        # on windows we just get IN_MODIFY, lets always fake
+        # on windows we just get IN_MODIFY, let's always fake
         # an OPEN & CLOSE_WRITE couple
         raw_open = raw_close = {
            'wd': event.wd,
@@ -760,7 +787,7 @@
         # the logic to check if the watch is already set
         # is all in WatchManager.add_watch
         return self._watch_manager.add_watch(dirpath,
-                             FILESYSTEM_MONITOR_MASK, auto_add=True)
+                             FILESYSTEM_MONITOR_MASK)
 
     def add_watches_to_udf_ancestors(self, volume):
         """Add a inotify watch to volume's ancestors if it's an UDF."""

=== modified file 'ubuntuone/platform/windows/os_helper.py'
--- ubuntuone/platform/windows/os_helper.py	2012-01-10 18:09:30 +0000
+++ ubuntuone/platform/windows/os_helper.py	2012-02-03 11:33:18 +0000
@@ -707,7 +707,11 @@
 @windowspath()
 def listdir(directory):
     """List a directory."""
-
+    return map(_unicode_to_bytes, native_listdir(directory))
+
+
+def native_listdir(directory):
+    """List a directory."""
     # The main reason why we have to append os.path.sep is the following:
     #
     # os.listdir implementation will append a unix path separator to the
@@ -732,8 +736,8 @@
     # return those paths that are system paths. Those paths are the ones that
     # we do not want to work with.
 
-    return map(_unicode_to_bytes, [p for p in os.listdir(directory) if not
-        native_is_system_path(os.path.join(directory, p))])
+    return [p for p in os.listdir(directory) if not
+                          native_is_system_path(os.path.join(directory, p))]
 
 
 @windowspath()