Merge lp:~mandel/ubuntuone-dev-tools/allow_txnamedpipes_reactor into lp:ubuntuone-dev-tools

[Manuel de la Peña]

Provides yet an extra parameter so that the txnamedpipes reactor can be used to run the trial tests. The txnamedpipes-reactor provide an IOCP reactor implementation that allows to use named pipes to perform the IPC.

Provides yet an extra parameter so that the txnamedpipes reactor can be used to run the trial tests. The txnamedpipes-reactor provide an IOCP reactor implementation that allows to use named pipes to perform the IPC.

[Roberto Alsina (ralsina)]

+1 but really needs a review from dobey

review: Approve

[dobey (dobey)]

+NAMEDPIPES_REACTOR_PROJECT = 'lp:txnamedpipes'

This really should be a useful HTTP[S] URL instead. Something people can click on and get to a web page, as with the Qt reactor one.

Also, the ./run-tests now fails for me on Linux in this branch.

I think I would also prefer to have a --reactor={glib,qt,namedpipes} type of argument to specify the reactor, than constantly adding more --foo-reactor arguments for every one we add. And then also have some of them perhaps provide additional argument possibilities, such as --with-ui for the Qt reactor, so that proper things can be done there. Perhaps it is time to move the reactor handling into their own modules, so we can treat them like plug-ins and make this main code a little more generic.

review: Needs Fixing

[dobey (dobey)]

OK, the ./run-tests failed because the branch name is a bit long, it seems; so the path went over the filename length limit.

[Manuel de la Peña (mandel)]

> OK, the ./run-tests failed because the branch name is a bit long, it seems; so
> the path went over the filename length limit.

Cool, I was worried because in my machine was working.

[Manuel de la Peña (mandel)]

> +NAMEDPIPES_REACTOR_PROJECT = 'lp:txnamedpipes'
>
> This really should be a useful HTTP[S] URL instead. Something people can click
> on and get to a web page, as with the Qt reactor one.
>
Simple and doable. :)

> Also, the ./run-tests now fails for me on Linux in this branch.
>
> I think I would also prefer to have a --reactor={glib,qt,namedpipes} type of
> argument to specify the reactor, than constantly adding more --foo-reactor
> arguments for every one we add. And then also have some of them perhaps
> provide additional argument possibilities, such as --with-ui for the Qt
> reactor, so that proper things can be done there. Perhaps it is time to move
> the reactor handling into their own modules, so we can treat them like plug-
> ins and make this main code a little more generic.

That is a complete change in the idea. Can we add this as a bug.

[dobey (dobey)]

On Tue, 2011-05-24 at 07:41 +0000, Manuel de la Pena wrote:
> > I think I would also prefer to have a --reactor={glib,qt,namedpipes} type of
> > argument to specify the reactor, than constantly adding more --foo-reactor
> > arguments for every one we add. And then also have some of them perhaps
> > provide additional argument possibilities, such as --with-ui for the Qt
> > reactor, so that proper things can be done there. Perhaps it is time to move
> > the reactor handling into their own modules, so we can treat them like plug-
> > ins and make this main code a little more generic.
>
> That is a complete change in the idea. Can we add this as a bug.
>

I realize that. But I don't want to file a bug, land this as-is, and
then start using this everywhere as it is now, only to either break it
soon, or just end up having it never get changed. I think we should fix
it now, and avoid such possibilities.

[lp:~mandel/ubuntuone-dev-tools/allow_txnamedpipes_reactor]

31. By Manuel de la Peña on 2011-05-26

Added a way to allow the addition of new reactors by adding 'plugins'

32. By Manuel de la Peña on 2011-05-26

Merged with the previous state of the branch.

33. By Manuel de la Peña on 2011-05-26

Removed left behind code.

[dobey (dobey)]

I'm going to go ahead and reject this, due to the complexity here, and as I've made a simpler branch which accomplishes what we need:

https://code.launchpad.net/~dobey/ubuntuone-dev-tools/fukishima/+merge/62506

review: Disapprove

Unmerged revisions

35. By Manuel de la Peña on 2011-04-02

Added tests for the plugins.

34. By Manuel de la Peña on 2011-04-02

Fixed license. Merged with trunk.

33. By Manuel de la Peña on 2011-05-26

Removed left behind code.

32. By Manuel de la Peña on 2011-05-26

Merged with the previous state of the branch.

31. By Manuel de la Peña on 2011-05-26

Added a way to allow the addition of new reactors by adding 'plugins'

30. By Manuel de la Peña on 2011-05-19

Added support to the namedpipes reactor so that it can be used on windows.

=== modified file 'bin/u1trial'
--- bin/u1trial	2011-04-13 14:37:14 +0000
+++ bin/u1trial	2011-05-26 11:15:58 +0000
@@ -30,10 +30,9 @@
 
 from twisted.trial.runner import TrialRunner
 
-QTREACTOR_URL = 'https://github.com/ghtdak/qtreactor'
-
 sys.path.insert(0, os.path.abspath("."))
 
+
 def _is_in_ignored_path(testcase, paths):
     """Return if the testcase is in one of the ignored paths."""
     for ignored_path in paths:
@@ -41,47 +40,11 @@
             return True
     return False
 
-def _install_reactor(qt_reactor=None):
-    """Install the correct reactor according to the platform."""
-    # the select reactor seems to be the most stable one in darwin
-    # and win32, we will just install a reactor for windows and use
-    # the default one for the other platforms.
-    platform = sys.platform
-    if not qt_reactor and platform not in ['win32', 'darwin']:
-        # install the glib2reactor before any import of the reactor to avoid
-        # using the default SelectReactor and be able to run the dbus tests
-        from twisted.internet import glib2reactor
-        glib2reactor.install()
-    elif qt_reactor:
-        # install the qtreactor so that we can start a qapplication in the
-        # tests to work with
-        if qt_reactor == 'ui':
-            from PyQt4.QtGui import QApplication
-            # pylint: disable=W0612
-            app = QApplication(sys.argv)
-            # pylint: enable=W0612
-        elif qt_reactor != 'no-ui':
-            # gave the wrong value
-            print 'Invalid option "%s" passed to --qtreactor' % qt_reactor
-            exit(1)
-        try:
-            # pylint: disable=W0511, F0401
-            from qtreactor import qt4reactor
-            # pylint: enable=W0511
-            qt4reactor.install()
-        except ImportError:
-            print 'The qtreactor python module is ' +\
-                  'required. You can find it at: %s' % QTREACTOR_URL
-            exit(1)
 
 class TestRunner(TrialRunner):
     """The test runner implementation."""
 
-    def __init__(self, force_gc=False, qt_reactor=None):
-        # install the correct reactor according to the platform, do
-        # this to ensure that we can instantiate the best reactor per
-        # platform
-        _install_reactor(qt_reactor)
+    def __init__(self, force_gc=False):
         from twisted.trial.reporter import TreeReporter
 
         # set $HOME to the _trial_temp dir, to avoid breaking user files
@@ -249,6 +212,7 @@
 def main():
     """Do the deed."""
     from optparse import OptionParser
+    from ubuntuone.devtools.plugins import load_plugins, configure_plugins
     usage = '%prog [options] path'
     parser = OptionParser(usage=usage)
     parser.add_option("-t", "--test", dest="test",
@@ -268,12 +232,14 @@
     parser.add_option("--force-gc", action="store_true", dest="force_gc", 
                       default=False, help="Run gc.collect() before and after "
                       "each test case.")
-    parser.add_option("--qt-reactor", type="string", dest="qt_reactor",
-                      default=None, help="Run the tests using the qtreactor."
-                      + "There are two available modes. 'ui' for those tests "
-                      + "that will show UI and 'no-ui' for those that won't.",
-                      metavar="MODE")
+
+    # load the diff plugins installed in the system
+    plugins = load_plugins(parser, set())
     (options, args) = parser.parse_args()
+    # allow the plugins to be configured
+    configure_plugins(plugins, options, args)
+
+    # let the plugins reacto to the options
     if args:
         testpath = args[0]
         if not os.path.exists(testpath):
@@ -282,8 +248,8 @@
     else:
         parser.print_help()
         sys.exit(2)
-    TestRunner(force_gc=options.force_gc, qt_reactor=options.qt_reactor).run(
-                                                            testpath, options)
+
+    TestRunner(force_gc=options.force_gc).run(testpath, options)
 
 if __name__ == '__main__':
     main()

=== modified file 'man/u1trial.1'
--- man/u1trial.1	2011-03-31 16:42:40 +0000
+++ man/u1trial.1	2011-05-26 11:15:58 +0000
@@ -35,8 +35,9 @@
 .TP
 \fB\-\-qt-reactor\fR=MODE\fR
 Run the tests using the qtreactor. There are two available modes. 'ui'
-for those tests that will show UI and 'no-ui' for those that won't.",
-
+for those tests that will show UI and 'no-ui' for those that won't.
+\fB\-\-namedpipes-reactor\fR
+Run the tests using the txnamedpipes reactor.
 
 .SH AUTHOR
 This manual page was written by Rodney Dawes <[e-mail]>.

=== modified file 'setup.py'
--- setup.py	2011-02-11 20:42:28 +0000
+++ setup.py	2011-05-26 11:15:58 +0000
@@ -50,7 +50,9 @@
       url='http://launchpad.net/ubuntuone-dev-tools',
       packages=['ubuntuone',
                 'ubuntuone.devtools',
-                'ubuntuone.devtools.services'],
+                'ubuntuone.devtools.services',
+                'ubuntuone.devtools.plugins',
+                'ubuntuone.devtools.plugins.reactors'],
       extra_path='ubuntuone-dev-tools',
       scripts=['bin/u1lint',
                'bin/u1trial',

=== added directory 'ubuntuone/devtools/plugins'
=== added file 'ubuntuone/devtools/plugins/__init__.py'
--- ubuntuone/devtools/plugins/__init__.py	1970-01-01 00:00:00 +0000
+++ ubuntuone/devtools/plugins/__init__.py	2011-05-26 11:15:58 +0000
@@ -0,0 +1,72 @@
+# -*- coding: utf-8 -*-
+
+# Author: Manuel de la Pena [e-mail]
+#
+# Copyright 2010-2011 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>
+"""Basic plugins operations."""
+
+
+import os
+import logging
+
+DEVTOOLS_PLUGIN_PATHS = [os.path.join(os.path.dirname(__file__))]
+
+
+def load_plugins(options, blocking_semaphores):
+    """Load the desktopcouch application plug-ins.
+
+    The blocking_semaphores set is OPTIONALLY mutated by any plugin to signal
+    that the service is not ready until a plugin has finished its asynchronous
+    operations.  Plugins may add a distinguishing object to the set, and it
+    must remove what it adds when it is finished.
+
+    options -- The options set that is used to display extra arguments
+    according to the plugin.
+    blocking_semaphores -- the set() of semaphores, which we will mutate
+    """
+    plugin_names = set()
+    for path in DEVTOOLS_PLUGIN_PATHS:
+        try:
+            for _file in os.listdir(path):
+                file_abspath = os.path.abspath(os.path.join(path, _file))
+                if _file == '__init__':
+                    continue
+                elif os.path.isdir(file_abspath) and\
+                    os.path.exists(os.path.join(file_abspath, '__init__.py')):
+                    plugin_names.add(os.path.join(
+                            'ubuntuone', 'devtools', 'plugins', _file))
+                elif _file.endswith('.py') and _file != '__init__.py':
+                    plugin_names.add(os.path.join(
+                            'ubuntuone', 'devtools', 'plugins', _file))
+        except OSError:
+            continue
+    plugin_modules = set()
+    for name in plugin_names:
+        modpath = name.replace(os.path.sep, '.')
+        if name.endswith('.py'):
+            modpath = modpath[:-3]
+        try:
+            plugin = __import__(modpath, None, None, [''])
+            plugin_modules.add(plugin)
+            plugin.plugin_init(options, blocking_semaphores)
+        except (ImportError, AttributeError):
+            logging.warning('Failed to load plug-in: %s', modpath)
+    return plugin_modules
+
+
+def configure_plugins(loaded_plugins, options, args):
+    """Configure the different plugins."""
+    for plugin in loaded_plugins:
+        plugin.plugin_options(options, args)

=== added directory 'ubuntuone/devtools/plugins/reactors'
=== added file 'ubuntuone/devtools/plugins/reactors/__init__.py'
--- ubuntuone/devtools/plugins/reactors/__init__.py	1970-01-01 00:00:00 +0000
+++ ubuntuone/devtools/plugins/reactors/__init__.py	2011-05-26 11:15:58 +0000
@@ -0,0 +1,95 @@
+# -*- coding: utf-8 -*-
+
+# Author: Manuel de la Pena [e-mail]
+#
+# Copyright 2009-2010 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""Plugin that allows diff reactors to be used."""
+
+
+import logging
+import os
+
+REACTORS_PLUGIN_PATHS = [os.path.join(os.path.dirname(__file__))]
+PLUGIN_NAME = __name__
+
+
+def plugin_init(options, blocking_semaphores):
+    """Set up the signal handler for pairing with Ubuntu One."""
+    logging.info('Loading reactor plugins.')
+    # Signal that we are critical for desktopcouch usage, and the server
+    # must not begin until we are finished.  We are responsible for
+    # removing this item from the list.
+    logging.info('adding %s to to blocking semaphore list', PLUGIN_NAME)
+    blocking_semaphores.add(PLUGIN_NAME)
+    reactor_names = set()
+    reactor_names.add('glib')
+    load_reactors(options, reactor_names)
+    options.add_option('-r', '--reactor', type='string', dest='reactor',
+                      default='glib', help='Run the tests using the '
+                      + 'specified reactor. The installed reactors are %s' %
+                      reactor_names, metavar="REACTOR")
+    blocking_semaphores.discard(PLUGIN_NAME)
+
+
+def plugin_options(options, args):
+    """React to the options passed by the user."""
+    reactors_names = set()
+    for path in REACTORS_PLUGIN_PATHS:
+        try:
+            for _file in os.listdir(path):
+                if _file == '__init__':
+                    continue
+                elif _file.endswith('.py') and _file != '__init__.py':
+                    reactors_names.add(os.path.join('ubuntuone', 'devtools',
+                                     'plugins', 'reactors', _file))
+        except OSError:
+            continue
+
+    for name in reactors_names:
+        modpath = name.replace(os.path.sep, '.')[:-3]
+        try:
+            reactor = __import__(modpath, None, None, [''])
+            if options.reactor == reactor.REACTOR_NAME:
+                reactor.process_options(options)
+        except (ImportError, AttributeError):
+            logging.warning('Failed to load plug-in: %s', modpath)
+
+
+def load_reactors(options, reactor_names):
+    """Load the installed reactors in the system.
+
+    options -- The options set that is used to display extra arguments
+    according to the plugin.
+    reactor_names -- the set()  that contains all the diff reactors.
+    """
+    reactors_names = set()
+    for path in REACTORS_PLUGIN_PATHS:
+        try:
+            for _file in os.listdir(path):
+                if _file == '__init__':
+                    continue
+                elif _file.endswith('.py') and _file != '__init__.py':
+                    reactors_names.add(os.path.join('ubuntuone', 'devtools',
+                                     'plugins', 'reactors', _file))
+        except OSError:
+            continue
+
+    for name in reactors_names:
+        modpath = name.replace(os.path.sep, '.')[:-3]
+        try:
+            reactor = __import__(modpath, None, None, [''])
+            reactor.reactor_init(options, reactor_names)
+        except (ImportError, AttributeError):
+            logging.warning('Failed to load plug-in: %s', modpath)

=== added file 'ubuntuone/devtools/plugins/reactors/glib.py'
--- ubuntuone/devtools/plugins/reactors/glib.py	1970-01-01 00:00:00 +0000
+++ ubuntuone/devtools/plugins/reactors/glib.py	2011-05-26 11:15:58 +0000
@@ -0,0 +1,32 @@
+# -*- coding: utf-8 -*-
+
+# Author: Manuel de la Pena [e-mail]
+#
+# Copyright 2010-2011 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""Use the glib reactor to run the tests."""
+
+REACTOR_NAME = __name__[__name__.rfind('.') + 1:]
+
+
+def reactor_init(options, reactor_names):
+    """Init the required data so that the reactor can be choosen."""
+    # not much to be done, lets just add the name to the list
+    reactor_names.add(REACTOR_NAME)
+
+
+def process_options(options):
+    """Process the options provide by the user to perform the actions."""
+    from twisted.internet import glib2reactor
+    glib2reactor.install()

=== added file 'ubuntuone/devtools/plugins/reactors/qtreactor.py'
--- ubuntuone/devtools/plugins/reactors/qtreactor.py	1970-01-01 00:00:00 +0000
+++ ubuntuone/devtools/plugins/reactors/qtreactor.py	2011-05-26 11:15:58 +0000
@@ -0,0 +1,58 @@
+# -*- coding: utf-8 -*-
+
+# Author: Manuel de la Pena [e-mail]
+#
+# Copyright 2010-2011 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>
+"""Use the qt reactor to run the tests."""
+
+
+import logging
+import sys
+
+QTREACTOR_URL = 'https://github.com/ghtdak/qtreactor'
+REACTOR_NAME = __name__[__name__.rfind('.') + 1:]
+
+
+def reactor_init(options, reactor_names):
+    """Init the required data so that the reactor can be choosen."""
+    # only provide the reacto if we can import it, otherwhise ignore it
+    try:
+        # pylint: disable=W0511, F0401
+        # XXX: Don't know why but the reactor cannot be imported using
+        # from txnamedpipes import reactor
+        __import__('qtreactor.qt4reactor', None, None, [''])
+        # pylint: enable=W0511
+        reactor_names.add(REACTOR_NAME)
+        options.add_option('--qt-ui', action='store_true', dest='qt_ui',
+                           default=False, help='State if the Qt reactor '
+                           + 'should be ran with ui support.')
+    except ImportError:
+        logging.info('The qtreactor python module is required. You can find '
+                     + 'it at: %s' % QTREACTOR_URL)
+
+
+def process_options(options):
+    """Process the options provide by the user to perform the actions."""
+    # install the qtreactor so that we can start a qapplication in the
+    # tests to work with
+    if options.qt_ui:
+        from PyQt4.QtGui import QApplication
+        # pylint: disable=W0612
+        app = QApplication(sys.argv)
+        # pylint: enable=W0612
+    # pylint: disable=W0511, F0401
+    reactor = __import__('qtreactor.qt4reactor', None, None, [''])
+    # pylint: enable=W0511
+    reactor.install()

=== added file 'ubuntuone/devtools/plugins/reactors/txnamedpipes.py'
--- ubuntuone/devtools/plugins/reactors/txnamedpipes.py	1970-01-01 00:00:00 +0000
+++ ubuntuone/devtools/plugins/reactors/txnamedpipes.py	2011-05-26 11:15:58 +0000
@@ -0,0 +1,42 @@
+# -*- coding: utf-8 -*-
+
+# Author: Manuel de la Pena [e-mail]
+#
+# Copyright 2010-2011 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""Use the txnamedpipes reactor."""
+
+
+import logging
+
+NAMEDPIPES_REACTOR_URL = 'https://launchpad.net/txnamedpipes'
+REACTOR_NAME = __name__[__name__.rfind('.') + 1:]
+
+
+def reactor_init(options, reactor_names):
+    """Init the required data so that the reactor can be choosen."""
+    try:
+        # Don't know why but the reactor cannot be imported using
+        # from txnamedpipes import reactor
+        __import__('txnamedpipes.reactor', None, None, [''])
+        reactor_names.add(REACTOR_NAME)
+    except ImportError:
+        logging.info('The txnamedpipes python module is required. You can'
+                     + ' find it at: %s' % NAMEDPIPES_REACTOR_URL)
+
+
+def process_options(options):
+    """Process the options provide by the user to perform the actions."""
+    reactor = __import__('txnamedpipes.reactor', None, None, [''])
+    reactor.install()

=== added file 'ubuntuone/devtools/tests/test_plugins.py'
--- ubuntuone/devtools/tests/test_plugins.py	1970-01-01 00:00:00 +0000
+++ ubuntuone/devtools/tests/test_plugins.py	2011-05-26 11:15:58 +0000
@@ -0,0 +1,51 @@
+# -*- coding: utf-8 -*-
+
+# Author: Manuel de la Pena [e-mail]
+#
+# Copyright 2009-2010 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+"""Test the plugins."""
+
+import os
+
+from ubuntuone.devtools import plugins
+from unittest import TestCase
+
+
+class TestPlugins(TestCase):
+    """Test the loading of the plugins."""
+
+    def setUp(self):
+        """Set the diff tests."""
+        super(TestPlugins, self).setUp()
+        self.options = None
+        self.blockers = set()
+        self._imported = None
+
+    def test_load_plugins(self):
+        """Test that plugins are loaded."""
+        plugins.DEVTOOLS_PLUGIN_PATHS = [os.path.join(
+                os.path.dirname(__file__))]
+
+        def _fake_import(name, *args, **kwargs):
+            """Fake import method."""
+            modname = name.replace('.plugins.', '.tests.')
+            if modname == __name__:
+                self._imported = modname
+
+        old_import = __import__
+        __builtins__['__import__'] = _fake_import
+        plugins.load_plugins(self.options, self.blockers)
+        __builtins__['__import__'] = old_import
+        self.assertEqual(self._imported, __name__)