Merge lp:~dobey/ubuntuone-client/update-4-0 into lp:ubuntuone-client/stable-4-0

[dobey]

[Rodney Dawes]

    - Skip the test_being_hashed for now as it causes intermittent failures.

[Manuel de la Peña]

    - Fix stale broker problems by reconnecting to the server and requesting new valid instances for the reference objects in the client side (LP: #1040915).
    - Use the correct logger.
    - Used the new API (LP: #1040170).
    - Create events now have in_modify thx to the new API (LP: #1034127).

[Mike McCracken]

    - Fix typo in log format specifier.
    - Log to syncdaemon log which monitor implementation is used. (LP: #1042336)
    - Remove broken buildout env vars, and add lint to mac test script. (LP: #1037432)

[Alejandro J. Cura]

    - Log exceptions when running async_main (LP: #1041163).

[Eric Casteleijn (thisfred)]

review: Approve

[Ubuntu One Auto Pilot (otto-pilot)]

The attempt to merge lp:~dobey/ubuntuone-client/update-4-0 into lp:ubuntuone-client/stable-4-0 failed. Below is the output from the failed tests.

/usr/bin/gnome-autogen.sh
checking for autoconf >= 2.53...
  testing autoconf2.50... not found.
  testing autoconf... found 2.69
checking for automake >= 1.10...
  testing automake-1.11... found 1.11.5
checking for libtool >= 1.5...
  testing libtoolize... found 2.4.2
checking for intltool >= 0.30...
  testing intltoolize... found 0.50.2
checking for pkg-config >= 0.14.0...
  testing pkg-config... found 0.26
checking for gtk-doc >= 1.0...
  testing gtkdocize... found 1.18
Checking for required M4 macros...
Checking for forbidden M4 macros...
Processing ./configure.ac
Running libtoolize...
libtoolize: putting auxiliary files in `.'.
libtoolize: copying file `./ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
Running intltoolize...
Running gtkdocize...
Running aclocal-1.11...
Running autoconf...
Running autoheader...
Running automake-1.11...
Running ./configure --enable-gtk-doc --enable-debug ...
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking how to create a ustar tar archive... gnutar
checking whether make supports nested variables... yes
checking for style of include used by make... GNU
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking dependency style of gcc... gcc3
checking for library containing strerror... none required
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking dependency style of gcc... (cached) gcc3
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking how to print strings... printf
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking whether the shell understands some XSI constructs......

[lp:~dobey/ubuntuone-client/update-4-0]

1258. By dobey on 2012-08-29

[Rodney Dawes]

    - Skip the test_being_hashed for now as it causes intermittent failures.

[Manuel de la Peña]

    - Fix stale broker problems by reconnecting to the server and requesting new valid instances for the reference objects in the client side (LP: #1040915).
    - Use the correct logger.
    - Used the new API (LP: #1040170).
    - Create events now have in_modify thx to the new API (LP: #1034127).

[Mike McCracken]

    - Fix typo in log format specifier.
    - Log to syncdaemon log which monitor implementation is used. (LP: #1042336)
    - Remove broken buildout env vars, and add lint to mac test script. (LP: #1037432)

[Alejandro J. Cura]

    - Log exceptions when running async_main (LP: #1041163).

=== modified file 'bin/ubuntuone-syncdaemon'
--- bin/ubuntuone-syncdaemon	2012-07-26 22:00:58 +0000
+++ bin/ubuntuone-syncdaemon	2012-08-28 19:24:19 +0000
@@ -104,6 +104,7 @@
                                args=args, usage=usage)
     d = async_main(parser, options, argv)
     d.addErrback(check_death)
+    d.addErrback(logger.root_logger.exception)
 
     # check if we should start a twisted manhole
     if options.debug_manhole:

=== modified file 'run-mac-tests'
--- run-mac-tests	2012-08-01 14:25:58 +0000
+++ run-mac-tests	2012-08-28 19:24:19 +0000
@@ -42,8 +42,10 @@
 cp windows/clientdefs.py ubuntuone/clientdefs.py
 cp windows/logging.conf data/logging.conf
 # execute the tests with a number of ignored linux only modules
-python $u1trial --reactor=twisted -i "test_linux.py,test_windows.py" -p tests/platform/linux "$MODULE"
+u1trial --reactor=twisted -i "test_linux.py,test_windows.py" -p tests/platform/linux "$MODULE"
 rm -rf _trial_temp
 rm -rf build
 
 python tests/platform/check_reactor_import.py
+
+USE_PYFLAKES="true" u1lint

=== modified file 'tests/platform/tools/test_tools.py'
--- tests/platform/tools/test_tools.py	2012-05-24 15:09:34 +0000
+++ tests/platform/tools/test_tools.py	2012-08-28 19:24:19 +0000
@@ -32,6 +32,7 @@
 import operator
 
 from twisted.internet import defer
+from twisted.spread import pb
 from twisted.trial.unittest import TestCase
 
 from ubuntuone.platform.tools import perspective_broker
@@ -125,3 +126,66 @@
                 attr = getattr(self.sdtool, attr_name)
                 func_name = getattr(attr, "__name__", None)
                 self.assertNotEqual(func_name, "call_after_connection_inner")
+
+
+class FakeRemoteObject(object):
+    """Fake a remote object."""
+
+    def __init__(self):
+        """Create a new instance."""
+        self.number_calls = 0
+        self.called = []
+
+    def method_call(self, *args, **kwargs):
+        """Fake a remote method call."""
+        if self.number_calls == 0:
+           self.number_calls += 1
+           raise pb.DeadReferenceError()
+        else:
+           self.called.append((args, kwargs))
+           return defer.succeed(self.number_calls)
+
+
+class PerspectiveBrokerReconnect(TestCase):
+    """Test when the ipc is reconnected."""
+
+    @defer.inlineCallbacks
+    def setUp(self):
+        """Set the tests."""
+        yield super(PerspectiveBrokerReconnect, self).setUp()
+        self.sdtool = perspective_broker.SyncDaemonToolProxy()
+        self.sdtool.client.fake_remote = FakeRemoteObject()
+        self.connected_signals = []
+        self.reconnected = False
+
+        def connect_signal(my_self, *args, **kwargs):
+            """Fake connect_signal call."""
+            self.connected_signals.append(('connect_signal', args, kwargs))
+
+        self.patch(perspective_broker.SyncDaemonToolProxy, 'connect_signal',
+                connect_signal)
+
+        def fake_reconnect(_):
+            """Fake the reconnection of the client."""
+            self.reconnected = True
+
+        self.patch(perspective_broker.UbuntuOneClient, 'reconnect',
+                fake_reconnect)
+        self.patch(perspective_broker.UbuntuOneClient, 'connect',
+                lambda _: defer.succeed(True))
+
+    @defer.inlineCallbacks
+    def test_reconnect_no_signals(self):
+        """Test reconnection with no signals."""
+        yield self.sdtool.call_method('fake_remote', 'method_call')
+        self.assertTrue(self.reconnected)
+        self.assertEqual(0, len(self.connected_signals))
+
+    @defer.inlineCallbacks
+    def test_reconnect_signals(self):
+        """Test reconnection with signals."""
+        self.sdtool.connected_signals = dict(create_signal=lambda:None,
+                                             delete_signal=lambda:None)
+        yield self.sdtool.call_method('fake_remote', 'method_call')
+        self.assertTrue(self.reconnected)
+        self.assertEqual(2, len(self.connected_signals))

=== modified file 'tests/syncdaemon/test_hashqueue.py'
--- tests/syncdaemon/test_hashqueue.py	2012-04-09 20:07:05 +0000
+++ tests/syncdaemon/test_hashqueue.py	2012-08-28 19:24:19 +0000
@@ -42,6 +42,7 @@
 from twisted.trial.unittest import TestCase as TwistedTestCase
 from twisted.internet import defer, reactor
 from ubuntuone.devtools.handlers import MementoHandler
+from ubuntuone.devtools.testcases import skipTest
 
 from contrib.testing.testcase import BaseTwistedTestCase
 from ubuntuone.platform import open_file, stat_path
@@ -583,6 +584,7 @@
         self.assertEqual(kwargs['mdid'], "foo")
         self.assertTrue(call_later_called)
 
+    @skipTest('Causing intermittent failures. LP: #935568')
     def test_being_hashed(self):
         """Tell if something is being hashed."""
         tfile1 = os.path.join(self.test_dir, "tfile1")

=== modified file 'ubuntuone/platform/filesystem_notifications/monitor/__init__.py'
--- ubuntuone/platform/filesystem_notifications/monitor/__init__.py	2012-08-01 14:25:58 +0000
+++ ubuntuone/platform/filesystem_notifications/monitor/__init__.py	2012-08-28 19:24:19 +0000
@@ -28,11 +28,14 @@
 # files in the program, then also delete it here.
 """Filesystem monitors per platform."""
 
+import logging
 import sys
 
 from twisted.internet import defer
 
 DEFAULT_MONITOR = 'default'
+logger = logging.getLogger('ubuntuone.SyncDaemon.platform.' +
+            'filesystem_notifications.monitor')
 
 
 class NoAvailableMonitorError(Exception):
@@ -70,6 +73,7 @@
         DEFAULT_MONITOR: linux.FilesystemMonitor,
     }
 
+
 # mantain old API
 FilesystemMonitor = FILEMONITOR_IDS[DEFAULT_MONITOR]
 
@@ -77,21 +81,28 @@
 @defer.inlineCallbacks
 def get_filemonitor_class(monitor_id=None):
     """Return the class to be used."""
+    logger.debug('File monitor ids for platform "%s" are "%s"', sys.platform,
+                 FILEMONITOR_IDS)
+
     if monitor_id is None:
-        # use default
+        logger.debug('monitor_id is None, using default.')
         monitor_id = 'default'
 
     if monitor_id not in FILEMONITOR_IDS:
-        raise NoAvailableMonitorError(
-                'No available monitor with id "%s"could be found.' % monitor_id)
+        msg = 'No available monitor with id %r could be found.'
+        raise NoAvailableMonitorError(msg % monitor_id)
 
     # retrieve the correct class and assert it can be used
     cls = FILEMONITOR_IDS[monitor_id]
+    logger.debug('Checking availability of monitor class %s', cls)
     is_available = yield cls.is_available_monitor()
 
     if is_available:
+        logger.debug('Monitor is available, returning monitor with id "%s"',
+                     monitor_id)
         defer.returnValue(cls)
     elif not is_available and monitor_id != DEFAULT_MONITOR:
+        logger.debug('Monitor is NOT available, returning default monitor.')
         cls = yield get_filemonitor_class(DEFAULT_MONITOR)
         defer.returnValue(cls)
     else:

=== modified file 'ubuntuone/platform/filesystem_notifications/monitor/darwin/fsevents_client.py'
--- ubuntuone/platform/filesystem_notifications/monitor/darwin/fsevents_client.py	2012-07-13 15:08:15 +0000
+++ ubuntuone/platform/filesystem_notifications/monitor/darwin/fsevents_client.py	2012-08-28 19:24:19 +0000
@@ -124,7 +124,7 @@
     def __init__(self, log):
         """Init the manager to keep track of the different watches."""
         self.log = log
-        self.observer = fsevents.Observer()
+        self.observer = fsevents.Observer(latency=0, process_asap=True)
         self.observer.start()
 
     def stop_watch(self, watch):

=== modified file 'ubuntuone/platform/ipc/ipc_client.py'
--- ubuntuone/platform/ipc/ipc_client.py	2012-08-16 12:00:12 +0000
+++ ubuntuone/platform/ipc/ipc_client.py	2012-08-28 19:24:19 +0000
@@ -778,6 +778,18 @@
                             'Could not connect to the syncdaemon ipc.', e)
         # pylint: disable=W0702
 
+    @defer.inlineCallbacks
+    def reconnect(self):
+        """Reconnect and get the new remote objects."""
+        try:
+            root = yield self.factory.getRootObject()
+            yield self._request_remote_objects(root)
+            yield self.register_to_signals()
+            defer.returnValue(self)
+        except Exception, e:
+            raise SyncDaemonClientConnectionError(
+                            'Could not reconnect to the syncdaemon ipc.', e)
+
     def is_connected(self):
         """Return if the client is connected."""
         return (self.client is not None)

=== modified file 'ubuntuone/platform/tools/perspective_broker.py'
--- ubuntuone/platform/tools/perspective_broker.py	2012-07-13 16:06:27 +0000
+++ ubuntuone/platform/tools/perspective_broker.py	2012-08-28 19:24:19 +0000
@@ -28,9 +28,11 @@
 # files in the program, then also delete it here.
 """SyncDaemon Tools."""
 
+import logging
 import subprocess
 
 from twisted.internet import defer
+from twisted.spread.pb import DeadReferenceError
 
 from ubuntuone.platform.ipc.perspective_broker import is_already_running
 from ubuntuone.platform.ipc.ipc_client import UbuntuOneClient
@@ -99,6 +101,7 @@
     _DONT_VERIFY_CONNECTED = [
         "wait_connected",
         "client", "last_event", "delayed_call", "log", "connected",
+        "connected_signals"
     ]
 
     def _should_wrap(self, attr_name):
@@ -115,8 +118,11 @@
             return attr
 
     def __init__(self, bus=None):
+        self.log = logging.getLogger('ubuntuone.platform.tools.' +
+            'perspective_broker')
         self.client = UbuntuOneClient()
         self.connected = None
+        self.connected_signals = {}
 
     def _call_after_connection(self, method):
         """Make sure Perspective Broker is connected before calling."""
@@ -127,17 +133,41 @@
         def call_after_connection_inner(*args, **kwargs):
             """Call the given method after the connection to pb is made."""
             yield self.connected
-            retval = yield method(*args, **kwargs)
+            try:
+                retval = yield method(*args, **kwargs)
+            except DeadReferenceError:
+                self.log.debug('Got stale broker, atempting reconnect.')
+                # might be the case where we have a stale broker
+                yield self._reconnect_client()
+                retval = yield method(*args, **kwargs)
             defer.returnValue(retval)
 
         return call_after_connection_inner
 
+    @defer.inlineCallbacks
+    def _reconnect_client(self):
+        """Reconnect the client."""
+        self.connected = False
+        yield self.client.reconnect()
+        # do connect all the signals again
+        for signal_name, handler in self.connected_signals.items():
+            self.connect_signal(signal_name, handler)
+
+    @defer.inlineCallbacks
     def call_method(self, client_kind, method_name, *args, **kwargs):
         """Call the 'method_name' passing 'args' and 'kwargs'."""
         client = getattr(self.client, client_kind)
         method = getattr(client, method_name)
-        result = method(*args, **kwargs)
-        return result
+        try:
+            result = yield method(*args, **kwargs)
+        except DeadReferenceError:
+            self.log.debug('Got stale broker, atempting reconnect.')
+            # may happen in the case we reconnected and the server side objects
+            # for gc
+            yield self._reconnect_client()
+            result = yield self.call_method(client_kind, method_name,
+                    *args, **kwargs)
+        defer.returnValue(result)
 
     def shutdown(self):
         """Close connections."""
@@ -148,6 +178,8 @@
         client_kind, callback = self._SIGNAL_MAPPING[signal_name]
         client = getattr(self.client, client_kind)
         setattr(client, callback, handler)
+        # do remember the connected signal in case we need to reconnect
+        self.connected_signals[signal_name] = handler
         return handler
 
     def disconnect_signal(self, signal_name, handler_or_match):
@@ -155,6 +187,8 @@
         client_kind, callback = self._SIGNAL_MAPPING[signal_name]
         client = getattr(self.client, client_kind)
         setattr(client, callback, None)
+        # forget that the connection was made in case we need to reconnect
+        del self.connected_signals[signal_name]
         return handler_or_match
 
     def wait_connected(self):