Merge lp:~zyga/lava-core/plugin-loader-respin into lp:lava-core

[Zygmunt Krynicki]

This is a respin of https://code.launchpad.net/~zkrynicki/lava-core/plugin-loader/+merge/102357
(it is a respin as I want to make lava-core standalone, co-installable and independent of any other lava.* packages)

To quote my commit message:

  Add plugins and errors modules

  The PluginLoader class is a simple thin wrapper around pkg_resources entry
  points. It de-couples LAVA from grotty pkg_resources details and allows us to
  migrate away from that technology later. It also allows us to explore frozen
  applications that would allow lava to run on windows.

  The PluginLoader class has basically two methods: to load all plugins that
  "reside" in a particular namespace and to load an explicitly named plugin (also
  from a particular namespace). Namespace names are translated 1-to-1 to
  pkg_resources namespaces but are sufficiently abstract that this is not
  important. All plugins must explicitly inherit from a base class designated by
  the caller. In lava-core this is typically an interface that has all the
  expected methods and properties that the calling side will later expect from
  the returned object.

  The errors module starts with two related exception classes. In subsequent
  patches it will be used as a bag of assorted exceptions used in all of
  lava-core.

This MP depends on the previous 'history and logging' proposal.

[Michael Hudson-Doyle (mwhudson)]

This is exactly the same as the code I reviewed before right?

review: Approve

[lp:~zyga/lava-core/plugin-loader-respin]

8. By Zygmunt Krynicki on 2012-05-16

Infrastructure for history and logging

10. By Zygmunt Krynicki on 2012-05-18

Make PluginLoader retain the order of entries from pkg_resources

Previously calling PluginLoader methods would trigger one-time initialization
via pkg_resources.iter_entry_points(). This method retains some of the ordering
of the declarations in setup.py files which can be useful, for example, for
listing commands in the expected order. This order was lost when entry points
were cached in a plain dictionary. This patch replaces all relevant uses of
plain dictionaries with OrderedDict (from the simplejson module)

11. By Zygmunt Krynicki on 2012-05-18

Add unit tests for lava.core.plugins

=== added file 'lava/core/errors.py'
--- lava/core/errors.py	1970-01-01 00:00:00 +0000
+++ lava/core/errors.py	2012-05-18 12:52:17 +0000
@@ -0,0 +1,38 @@
+# Copyright (C) 2011-2012 Linaro Limited
+#
+# Author: Zygmunt Krynicki <[e-mail]>
+#
+# This file is part of lava-core
+#
+# lava-core is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3
+# as published by the Free Software Foundation
+#
+# lava-core is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with lava-core.  If not, see <http://www.gnu.org/licenses/>.
+
+from __future__ import absolute_import
+
+"""
+lava.core.errors
+================
+
+Common error classes with fuzzy meaning
+"""
+
+
+class PluginLookupError(LookupError):
+    """
+    Error raised when a plugin cannot be found
+    """
+
+
+class PluginLoadError(ImportError):
+    """
+    Error raised when a plugin cannot be loaded
+    """

=== added file 'lava/core/plugins.py'
--- lava/core/plugins.py	1970-01-01 00:00:00 +0000
+++ lava/core/plugins.py	2012-05-18 12:52:17 +0000
@@ -0,0 +1,131 @@
+# Copyright (C) 2011-2012 Linaro Limited
+#
+# Author: Zygmunt Krynicki <[e-mail]>
+#
+# This file is part of lava-core
+#
+# lava-core is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3
+# as published by the Free Software Foundation
+#
+# lava-core is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with lava-core.  If not, see <http://www.gnu.org/licenses/>.
+
+from __future__ import absolute_import
+
+"""
+lava.core.plugins
+=================
+
+Formalized plugin system for LAVA.
+"""
+
+import pkg_resources
+from simplejson import OrderedDict
+
+from lava.core.errors import PluginLookupError, PluginLoadError
+from lava.core.logging import LoggingMixIn
+
+
+class PluginLoader(LoggingMixIn):
+    """
+    Loader for plugins that populate a specified namespace and conform to a
+    specified interface. It is expected that the objects pointed to by the
+    entry point system are _classes_. The plugin loader will call issubclass()
+    on them.
+
+    This plugin loader is based on pkg_resources machinery but it may be
+    expected to behave specially in the future. For example, when interacting
+    from a frozen binary on windows.
+
+    .. note::
+        Each instance has a cache of entry points that were discovered in the
+        past that is never purged. This should not be a problem under normal
+        circumstances.
+    """
+
+    def __init__(self, namespace, interface):
+        """
+        Create a plugin loader.
+        """
+        self.namespace = namespace
+        self.interface = interface
+        self._entry_points = None  # Cache of entry_point.name -> entry_point
+
+    def load_all_plugins(self):
+        """
+        Load all plugins, returns an dict of objects.
+        Plugins that fail to load are skipped.
+        """
+        self._iter_entry_points_if_needed()
+        plugins = OrderedDict()
+        for entry_point in self._entry_points.itervalues():
+            try:
+                plugin = self._load(entry_point)
+            except PluginLoadError:
+                pass
+            else:
+                plugins[entry_point.name] = plugin
+        return plugins
+
+    def load_plugin(self, name):
+        """
+        Load a plugin by name.
+
+        @raises PluginLookupError if the plugin cannot be found.
+        @raises PluginLoadError if the plugin cannot be loaded
+        """
+        self._iter_entry_points_if_needed()
+        try:
+            entry_point = self._entry_points[name]
+        except KeyError:
+            raise PluginLookupError(name)
+        else:
+            return self._load(entry_point)
+
+    def _load(self, entry_point):
+        """
+        Load an validate an entry point object.
+
+        ImportErrors and DistributionNotFound errors are caught and trigger a
+        diagnostic message, the return value is None if that happens.
+
+        The value loaded from the entry point must be a class that is a
+        subclass of the interface class passed to PluginLoader constructor.
+        Failure to conform triggers another diagnostic message and a None
+        return value.
+        """
+        try:
+            self.logger.debug("Loading entry point: %r", entry_point)
+            plugin = entry_point.load()
+        except (ImportError, pkg_resources.DistributionNotFound):
+            self.logger.exception(
+                "Unable to load plugin: %s (%r)",
+                entry_point.name, entry_point)
+            raise PluginLoadError(entry_point)
+        else:
+            if (not isinstance(plugin, type)
+                or not issubclass(plugin, self.interface)):
+                self.logger.exception(
+                    "Plugin %r does not implement interface %r",
+                    entry_point.name, self.interface)
+                raise PluginLoadError(entry_point)
+            return plugin
+
+    def _iter_entry_points_if_needed(self):
+        """
+        Iterate all the entry points in a specified namespace.
+
+        This method keeps the result in self._entry_points because
+        pkg_resources.iter_entry_points() is surprisingly slow, doing a lot of
+        IO on tiny files all over the disk.
+        """
+        if self._entry_points is None:
+            self._entry_points = OrderedDict()
+            for entry_point in pkg_resources.iter_entry_points(self.namespace):
+                self._entry_points[entry_point.name] = entry_point

=== added file 'lava/core/tests/test_plugins.py'
--- lava/core/tests/test_plugins.py	1970-01-01 00:00:00 +0000
+++ lava/core/tests/test_plugins.py	2012-05-18 12:52:17 +0000
@@ -0,0 +1,272 @@
+# Copyright (C) 2011-2012 Linaro Limited
+#
+# Author: Zygmunt Krynicki <[e-mail]>
+#
+# This file is part of lava-core
+#
+# lava-core is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3
+# as published by the Free Software Foundation
+#
+# lava-core is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with lava-core.  If not, see <http://www.gnu.org/licenses/>.
+
+"""
+lava.core.tests.test_plugins
+============================
+
+Unit tests for the lava.core.plugins module
+"""
+
+from mocker import Mocker, expect
+from unittest2 import TestCase
+import pkg_resources
+import simplejson
+
+from lava.core.plugins import PluginLoader
+from lava.core.errors import PluginLoadError, PluginLookupError
+
+
+class PluginLoaderTest(TestCase):
+
+    _namespace = "lava.unit_tests"
+    _interface = object
+    _name = "name"
+    _name2 = "name2"
+    _name3 = "name3"
+
+    def test__init__(self):
+        """
+        __init__() works
+        """
+        loader = PluginLoader(self._namespace, self._interface)
+        self.assertIs(loader._entry_points, None)
+        self.assertEqual(loader.namespace, self._namespace)
+        self.assertEqual(loader.interface, self._interface)
+
+    def test_load_all_plugins(self):
+        """
+        load_all_plugins() conceals PluginLoadError
+        """
+        mocker = Mocker()
+        entry_point = mocker.mock(pkg_resources.EntryPoint)
+        entry_point2 = mocker.mock(pkg_resources.EntryPoint)
+        entry_point3 = mocker.mock(pkg_resources.EntryPoint)
+        loader = PluginLoader(self._namespace, self._interface)
+        # Seed the cache
+        loader._entry_points = {
+            self._name: entry_point,
+            self._name2: entry_point2,
+            self._name3: entry_point3,
+        }
+        patched_loader = mocker.patch(loader)
+        expect(patched_loader._load(entry_point)).result(self._interface)
+        expect(entry_point.name).result(self._name)
+        # plugin 2 crashes
+        expect(patched_loader._load(entry_point2)).throw(PluginLoadError())
+        expect(patched_loader._load(entry_point3)).result(self._interface)
+        expect(entry_point3.name).result(self._name3)
+        with mocker:
+            plugins = loader.load_all_plugins()
+        # plugin 2 not returned, everything else, returned
+        self.assertEqual(plugins, {
+            self._name: self._interface,
+            self._name3: self._interface,
+        })
+
+    def test_load_all_plugins_keeps_order(self):
+        """
+        load_all_plugins() uses OrderedDict()
+        """
+        loader = PluginLoader(self._namespace, self._interface)
+        # Seed the cache
+        loader._entry_points = {}
+        plugins = loader.load_all_plugins()
+        self.assertIsInstance(plugins, simplejson.OrderedDict)
+
+    def test_load_plugin_missing(self):
+        """
+        load_plugin() raises PluginLookupError for missing plugins
+        """
+        loader = PluginLoader(self._namespace, self._interface)
+        # Seed the cache
+        loader._entry_points = {}
+        with self.assertRaises(PluginLookupError):
+            loader.load_plugin(self._name)
+
+    def test_load_plugin_failing_to_load(self):
+        """
+        load_plugin() propagates PluginLoadError
+        """
+        mocker = Mocker()
+        entry_point = mocker.mock(pkg_resources.EntryPoint)
+        loader = PluginLoader(self._namespace, self._interface)
+        loader._entry_points = {self._name: entry_point}
+        patched_loader = mocker.patch(loader)
+        expect(patched_loader._load(entry_point)).throw(PluginLoadError())
+        with mocker:
+            with self.assertRaises(PluginLoadError):
+                loader.load_plugin(self._name)
+
+    def test_load_plugin_all_good(self):
+        """
+        load_plugin() returns the plugin on success
+        """
+        mocker = Mocker()
+        entry_point = mocker.mock(pkg_resources.EntryPoint)
+        loader = PluginLoader(self._namespace, self._interface)
+        loader._entry_points = {self._name: entry_point}
+        patched_loader = mocker.patch(loader)
+        expect(patched_loader._load(entry_point)).result(self._interface)
+        with mocker:
+            plugin = loader.load_plugin(self._name)
+        self.assertEqual(plugin, self._interface)
+
+    def test__load_and_import_error(self):
+        """
+        _load() raises PluginLoadError when EntryPoint().load() raises
+        ImportError
+        """
+        mocker = Mocker()
+        loader = PluginLoader(self._namespace, self._interface)
+        patched_loader = mocker.patch(loader)
+        entry_point = mocker.mock(pkg_resources.EntryPoint)
+        patched_loader.logger.debug(
+            "Loading entry point: %r", entry_point)
+        expect(entry_point.load()).throw(ImportError())
+        expect(entry_point.name).result(self._name)
+        patched_loader.logger.exception(
+            "Unable to load plugin: %s (%r)",
+            self._name, entry_point)
+        with mocker:
+            with self.assertRaises(PluginLoadError):
+                loader._load(entry_point)
+
+    def test__load_and_the_other_error(self):
+        """
+        _load() raises PluginLoadError when EntryPoint.load() raises
+        pkg_resources.DistributionNotFound
+        """
+        mocker = Mocker()
+        loader = PluginLoader(self._namespace, self._interface)
+        patched_loader = mocker.patch(loader)
+        entry_point = mocker.mock(pkg_resources.EntryPoint)
+        patched_loader.logger.debug(
+            "Loading entry point: %r", entry_point)
+        expect(entry_point.load()).throw(pkg_resources.DistributionNotFound())
+        expect(entry_point.name).result(self._name)
+        patched_loader.logger.exception(
+            "Unable to load plugin: %s (%r)",
+            self._name, entry_point)
+        with mocker:
+            with self.assertRaises(PluginLoadError):
+                loader._load(entry_point)
+
+    def test__load_and_non_type_result(self):
+        """
+        _load() raises PluginLoadError when EntryPoint.load() returns a
+        non-type
+        """
+        mocker = Mocker()
+        loader = PluginLoader(self._namespace, self._interface)
+        patched_loader = mocker.patch(loader)
+        entry_point = mocker.mock(pkg_resources.EntryPoint)
+        patched_loader.logger.debug(
+            "Loading entry point: %r", entry_point)
+        expect(entry_point.load()).result(object())  # instance, not a type
+        expect(entry_point.name).result(self._name)
+        patched_loader.logger.exception(
+            "Plugin %r does not implement interface %r",
+            self._name, self._interface)
+        with mocker:
+            with self.assertRaises(PluginLoadError):
+                loader._load(entry_point)
+
+    def test__load_and_wrong_type_result(self):
+        """
+        _load() raises PluginLoadError when EntryPoint.load() returns a type
+        that does not match the interface
+        """
+        mocker = Mocker()
+        loader = PluginLoader(self._namespace, int)  # expecting int-subclass
+        patched_loader = mocker.patch(loader)
+        entry_point = mocker.mock(pkg_resources.EntryPoint)
+        patched_loader.logger.debug(
+            "Loading entry point: %r", entry_point)
+        expect(entry_point.load()).result(str)  # str, not int subclass
+        expect(entry_point.name).result(self._name)
+        patched_loader.logger.exception(
+            "Plugin %r does not implement interface %r",
+            self._name, int)
+        with mocker:
+            with self.assertRaises(PluginLoadError):
+                loader._load(entry_point)
+
+    def test__load_all_good(self):
+        """
+        _load() returns the plugin on success
+        """
+        mocker = Mocker()
+        loader = PluginLoader(self._namespace, self._interface)
+        patched_loader = mocker.patch(loader)
+        entry_point = mocker.mock(pkg_resources.EntryPoint)
+        patched_loader.logger.debug(
+            "Loading entry point: %r", entry_point)
+        expect(entry_point.load()).result(self._interface)
+        with mocker:
+            plugin = loader._load(entry_point)
+        # whatever we returned from load()
+        self.assertIs(plugin, self._interface)
+
+    def test__iter_entry_points_if_needed(self):
+        """
+        _iter_entry_points() fills the cache
+        """
+        mocker = Mocker()
+        entry = mocker.mock(pkg_resources.EntryPoint)
+        expect(entry.name).result(self._name)
+        entry2 = mocker.mock(pkg_resources.EntryPoint)
+        expect(entry2.name).result(self._name2)
+        # Need to use replace() as iter_entry_points is called indirectly
+        replaced_pkg_resources = mocker.replace(pkg_resources)
+        replaced_pkg_resources.iter_entry_points(self._namespace)
+        mocker.result(iter([entry, entry2]))
+        with mocker:
+            loader = PluginLoader(self._namespace, self._interface)
+            loader._iter_entry_points_if_needed()
+        # Mapping is correct
+        self.assertEqual(loader._entry_points,
+                         {self._name: entry, self._name2: entry2})
+
+    def test__iter_entry_points_keeps_order(self):
+        """
+        _iter_entry_points_if_needed() uses OrderedDict
+        """
+        mocker = Mocker()
+        # Need to use replace() as iter_entry_points is called indirectly
+        replaced_pkg_resources = mocker.replace(pkg_resources)
+        replaced_pkg_resources.iter_entry_points(self._namespace)
+        mocker.result(iter([]))
+        with mocker:
+            loader = PluginLoader(self._namespace, self._interface)
+            loader._iter_entry_points_if_needed()
+        self.assertIsInstance(loader._entry_points, simplejson.OrderedDict)
+
+    def test__iter_entry_points_if_needed__twice(self):
+        """
+        _iter_entry_points_if_needed() only runs once
+        """
+        mocker = Mocker()
+        # Need to use replace() as iter_entry_points is called indirectly
+        replaced_pkg_resources = mocker.replace(pkg_resources)
+        replaced_pkg_resources.iter_entry_points(self._namespace)
+        mocker.result(iter([]))
+        with mocker:
+            loader = PluginLoader(self._namespace, self._interface)
+            loader._iter_entry_points_if_needed()
+            loader._iter_entry_points_if_needed()