Merge lp:~wallyworld/launchpad/pillar-access-service-infrastructure into lp:launchpad

[Ian Booth]

[r=jcsackett,rharding][no-qa] Provides an easy way to define and use named services having methods returning json data and supporting method parameter marshalling using the lazr restful infrastructure.

== Implementation ==

This branch provides an easy way to define and use named services having methods returning json data and supporting method parameter marshalling using the lazr restful infrastructure.

This work is required for disclosure. I wanted to introduce this new infrastructure rather than perpetuating the broken design of attaching service interfaces to domain entities. The lazr restful stuff suits quite well, but we will likely need to extend it to support marshalling operation parameters of type dict with IEntry objects for keys or values, rather than just references or lists.

Services are located using a simple traversal mechanism: "/services/<service_name>"

To define a new named service, simply create an interface extending IService, provide an implementing class, and register the service as a named utility in zcml. The service is then accessible via the url mentioned above.

The infrastructure to provide the zope url and navigation glue is provided by ServicesLink (for the top level services entry), ServiceFactory (for the traversal off the services link), and the IService url rules defined in zcml.

As well as the infrastructure to make it work, a sample service has been implemented, AccessPolicyService. This will be fleshed out and used with the disclosure work. Note that the service infrastructure glue and bespoke service implementation logic is kept separate for ease of maintainability etc.

I had trouble getting this all glued together. Even though all the exported methods and launchpadlib instance etc use version 'devel', I had to export the service implementations themselves for version 'beta':

export_as_webservice_entry(publish_web_link=False, as_of='beta')

Until I did this, the wadl generation was broken. Thanks to wgrant for this bit.

== Demo and QA ==

Invoke a method on the access policy service from within a browser

https://launchpad.dev/api/devel/services/accesspolicy?ws.op=getAccessPolicies&ws.accept=application/json

or via launchpadlib:

from launchpadlib.launchpad import Launchpad
lp = Launchpad.login_with('testing', service_root='https://api.launchpad.dev', version='devel')
aps = lp.load('/services/accesspolicy')
aps.getAccessPolicies()

or from an XHR call using an lp.client.Launchpad instance

launchpad.named_get('/services/accesspolicy', 'getAccessPolicies')

== Tests ==

Add test for service traversal
Add test for service api invocation using a web services caller
Add test for service api invocation using launchpadlib

== Lint ==

Checking for conflicts and issues in changed files.

Linting changed files:
  lib/lp/app/services.py
  lib/lp/app/browser/launchpad.py
  lib/lp/app/interfaces/services.py
  lib/lp/app/tests/test_services.py
  lib/lp/registry/configure.zcml
  lib/lp/registry/services/
  lib/lp/registry/browser/configure.zcml
  lib/lp/registry/interfaces/accesspolicyservice.py
  lib/lp/registry/interfaces/webservice.py
  lib/lp/registry/services/__init__.py
  lib/lp/registry/services/accesspolicyservice.py
  lib/lp/registry/services/configure.zcml
  lib/lp/registry/services/tests/
  lib/lp/registry/services/tests/__init__.py
  lib/lp/registry/services/tests/test_accesspolicyservice.py
  lib/lp/services/webservice/services.py

[Richard Harding (rharding)]

review: Approve (code*)

[Curtis Hovey (sinzui)]

The services utilities in registry/configure.zcml can be moved into app/configure.zcml since none of the registrations use .registry domain objects. I think this is registering the objects used by LaunchpadRootNavigation which I think is your change in LaunchpadRootNavigation.

[j.c.sackett (jcsackett)]

I agree with Curtis, there's no good reason for the base service zcml stuff to be on registry; it makes a lot more sense for it to be in app.

As I don't want to create a block, I'm approving, but please make that change.

review: Approve

[Robert Collins (lifeless)]

Hi, this is neat (but I still want to talk with you and Curtis today about it).

Two nits:
 - don't accept broken stuff in our stack: that as_of beta should be
fixed at root.
 - /services/ will collide with a product called services won't it?
Please use /+services instead to mitigate this.

-Rob

[Ian Booth (wallyworld)]

Hi Rob

> Two nits:
> - don't accept broken stuff in our stack: that as_of beta should be
> fixed at root.

Ok. I wasn't 100% sure it was broken or me not appreciating a subtlety
of the lazr restful versioning. I'll file a bug and put in a XXX if we
think it is indeed broken.

> - /services/ will collide with a product called services won't it?
> Please use /+services instead to mitigate this.
>

I had planned on adding services to the names blacklist. My opinion is
that "services" should be a reserved word in this sense but if you fell
strongly that it should be +services I'll do that.

[Ian Booth (wallyworld)]

I've altered the zcml as requested.

On 24/02/12 01:58, Curtis Hovey wrote:
> The services utilities in registry/configure.zcml can be moved into app/configure.zcml since none of the registrations use .registry domain objects. I think this is registering the objects used by LaunchpadRootNavigation which I think is your change in LaunchpadRootNavigation.

[Robert Collins (lifeless)]

On Fri, Feb 24, 2012 at 10:01 AM, Ian Booth <email address hidden> wrote:
> I had planned on adding services to the names blacklist. My opinion is
> that "services" should be a reserved word in this sense but if you fell
> strongly that it should be +services I'll do that.

We've no reason to munge the two namespaces together - +foo is our
default choice for getting into a different namespace. So yes,
+services.

-Rob

=== modified file 'lib/lp/app/browser/launchpad.py'
--- lib/lp/app/browser/launchpad.py	2011-12-30 09:16:36 +0000
+++ lib/lp/app/browser/launchpad.py	2012-02-23 23:37:18 +0000
@@ -79,6 +79,7 @@
     )
 from lp.app.interfaces.headings import IMajorHeadingView
 from lp.app.interfaces.launchpad import ILaunchpadCelebrities
+from lp.app.interfaces.services import IServiceFactory
 from lp.app.widgets.project import ProjectScopeWidget
 from lp.blueprints.interfaces.specification import ISpecificationSet
 from lp.blueprints.interfaces.sprint import ISprintSet
@@ -652,6 +653,7 @@
     # hierarchical navigation model.
     stepto_utilities = {
         '+announcements': IAnnouncementSet,
+        '+services': IServiceFactory,
         'binarypackagenames': IBinaryPackageNameSet,
         'branches': IBranchSet,
         'bugs': IMaloneApplication,

=== modified file 'lib/lp/app/configure.zcml'
--- lib/lp/app/configure.zcml	2011-12-30 08:03:42 +0000
+++ lib/lp/app/configure.zcml	2012-02-23 23:37:18 +0000
@@ -55,4 +55,25 @@
         factory="lp.app.browser.badge.HasBadgeBase"
         permission="zope.Public"
         />
+
+    <!-- Services Infrastructure -->
+    <include
+        package="lp.registry.services"/>
+    <securedutility
+        class="lp.app.services.ServiceFactory"
+        provides="lp.app.interfaces.services.IServiceFactory">
+        <allow
+            interface="lp.app.interfaces.services.IServiceFactory"/>
+        <allow
+            interface="zope.publisher.interfaces.IPublishTraverse"/>
+    </securedutility>
+    <securedutility
+        class="lp.services.webservice.services.ServicesLink"
+        provides="lp.services.webservice.services.IServicesLink">
+        <allow
+            interface="lazr.restful.interfaces.ITopLevelEntryLink"/>
+        <allow
+            interface="lp.services.webapp.interfaces.ICanonicalUrlData"/>
+    </securedutility>
+
 </configure>

=== added file 'lib/lp/app/interfaces/services.py'
--- lib/lp/app/interfaces/services.py	1970-01-01 00:00:00 +0000
+++ lib/lp/app/interfaces/services.py	2012-02-23 23:37:18 +0000
@@ -0,0 +1,37 @@
+# Copyright 2012 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Interfaces used for named services."""
+
+
+__metaclass__ = type
+
+__all__ = [
+    'IService',
+    'IServiceFactory',
+    ]
+
+from zope.interface import Interface
+from zope.schema import TextLine
+from lazr.restful.declarations import (
+    export_as_webservice_entry,
+    exported,
+    )
+
+from lp import _
+
+
+class IService(Interface):
+    """Base interface for services."""
+
+    name = exported(
+        TextLine(
+            title=_('Name'),
+            description=_(
+                'The name of the service, used to generate the url.')))
+
+
+class IServiceFactory(Interface):
+    """Interface representing a factory used to access named services."""
+
+    export_as_webservice_entry(publish_web_link=False, as_of='beta')

=== added file 'lib/lp/app/services.py'
--- lib/lp/app/services.py	1970-01-01 00:00:00 +0000
+++ lib/lp/app/services.py	2012-02-23 23:37:18 +0000
@@ -0,0 +1,37 @@
+# Copyright 2012 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Factory used to get named services."""
+
+__metaclass__ = type
+__all__ = [
+    'ServiceFactory',
+    ]
+
+from zope.component import getUtility
+from zope.interface import implements
+
+from lp.app.interfaces.services import (
+    IService,
+    IServiceFactory,
+    )
+from lp.services.webapp.publisher import Navigation
+
+
+class ServiceFactory(Navigation):
+    """Creates a named service.
+
+    Services are traversed via urls of the form /services/<name>
+    Implementation classes are registered as named zope utilities.
+    """
+
+    implements(IServiceFactory)
+
+    def __init__(self):
+        super(ServiceFactory, self).__init__(None)
+
+    def traverse(self, name):
+        return self.getService(name)
+
+    def getService(self, service_name):
+        return getUtility(IService, service_name)

=== added file 'lib/lp/app/tests/test_services.py'
--- lib/lp/app/tests/test_services.py	1970-01-01 00:00:00 +0000
+++ lib/lp/app/tests/test_services.py	2012-02-23 23:37:18 +0000
@@ -0,0 +1,45 @@
+# Copyright 2012 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Tests for core services infrastructure."""
+
+from zope.component import getUtility
+from zope.interface.declarations import implements
+
+from lazr.restful.interfaces._rest import IHTTPResource
+
+from lp.app.interfaces.services import IService, IServiceFactory
+from lp.services.webapp.interaction import ANONYMOUS
+from lp.testing import (
+    FakeAdapterMixin,
+    TestCaseWithFactory,
+    )
+from lp.testing import login
+from lp.testing.layers import DatabaseFunctionalLayer
+from lp.testing.publication import test_traverse
+
+
+class IFakeService(IService):
+    """Fake service interface."""
+
+
+class FakeService:
+    implements(IFakeService, IHTTPResource)
+
+    name = 'fake_service'
+
+
+class TestServiceFactory(TestCaseWithFactory, FakeAdapterMixin):
+    """Tests for the ServiceFactory"""
+
+    layer = DatabaseFunctionalLayer
+
+    def test_service_traversal(self):
+        # Test that traversal to the named service works.
+        login(ANONYMOUS)
+        fake_service = FakeService()
+        self.registerUtility(fake_service, IService, "fake")
+        context, view, request = test_traverse(
+            'https://launchpad.dev/api/devel/+services/fake')
+        self.assertEqual(getUtility(IServiceFactory), context)
+        self.assertEqual(fake_service, view)

=== modified file 'lib/lp/registry/browser/configure.zcml'
--- lib/lp/registry/browser/configure.zcml	2012-02-20 05:12:41 +0000
+++ lib/lp/registry/browser/configure.zcml	2012-02-23 23:37:18 +0000
@@ -32,6 +32,15 @@
         module="lp.registry.feed.announcement"
         classes="LaunchpadAnnouncementsFeed TargetAnnouncementsFeed"
         />
+    <browser:url
+       for="lp.app.interfaces.services.IServiceFactory"
+       path_expression="string:+services"
+       parent_utility="lp.services.webapp.interfaces.ILaunchpadRoot"
+      />
+    <browser:url
+       for="lp.app.interfaces.services.IService"
+       path_expression="string:${name}"
+       parent_utility="lp.app.interfaces.services.IServiceFactory"/>
 
 <facet facet="overview">
     <browser:page

=== added file 'lib/lp/registry/interfaces/accesspolicyservice.py'
--- lib/lp/registry/interfaces/accesspolicyservice.py	1970-01-01 00:00:00 +0000
+++ lib/lp/registry/interfaces/accesspolicyservice.py	2012-02-23 23:37:18 +0000
@@ -0,0 +1,32 @@
+# Copyright 2011 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Interfaces for access policy service."""
+
+
+__metaclass__ = type
+
+__all__ = [
+    'IAccessPolicyService',
+    ]
+
+from lazr.restful.declarations import (
+    export_as_webservice_entry,
+    export_read_operation,
+    operation_for_version,
+    )
+
+from lp.app.interfaces.services import IService
+
+
+class IAccessPolicyService(IService):
+
+    # XXX 2012-02-24 wallyworld bug 939910
+    # Need to export for version 'beta' even though we only want to use it in
+    # version 'devel'
+    export_as_webservice_entry(publish_web_link=False, as_of='beta')
+
+    @export_read_operation()
+    @operation_for_version('devel')
+    def getAccessPolicies():
+        """Return the access policy types."""

=== modified file 'lib/lp/registry/interfaces/webservice.py'
--- lib/lp/registry/interfaces/webservice.py	2011-12-22 16:19:11 +0000
+++ lib/lp/registry/interfaces/webservice.py	2012-02-23 23:37:18 +0000
@@ -4,6 +4,8 @@
 """All the interfaces that are exposed through the webservice."""
 
 __all__ = [
+    'IAccessPolicyService',
+    'IServiceFactory',
     'DerivationError',
     'ICommercialSubscription',
     'IDistribution',
@@ -100,5 +102,9 @@
 from lp.registry.interfaces.teammembership import ITeamMembership
 from lp.registry.interfaces.wikiname import IWikiName
 
+# Services
+from lp.app.interfaces.services import IServiceFactory
+from lp.registry.interfaces.accesspolicyservice import IAccessPolicyService
+
 
 _schema_circular_imports

=== added directory 'lib/lp/registry/services'
=== added file 'lib/lp/registry/services/__init__.py'
--- lib/lp/registry/services/__init__.py	1970-01-01 00:00:00 +0000
+++ lib/lp/registry/services/__init__.py	2012-02-23 23:37:18 +0000
@@ -0,0 +1,1 @@
+"""The services namespace package."""

=== added file 'lib/lp/registry/services/accesspolicyservice.py'
--- lib/lp/registry/services/accesspolicyservice.py	1970-01-01 00:00:00 +0000
+++ lib/lp/registry/services/accesspolicyservice.py	2012-02-23 23:37:18 +0000
@@ -0,0 +1,45 @@
+# Copyright 2012 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Classes for pillar and artifact access policy services."""
+
+__metaclass__ = type
+__all__ = [
+    'AccessPolicyService',
+    ]
+
+import simplejson
+from lazr.restful import ResourceJSONEncoder
+from zope.interface import implements
+
+from lp.app.enums import InformationVisibilityPolicy
+from lp.registry.interfaces.accesspolicyservice import (
+    IAccessPolicyService,
+    )
+
+
+class AccessPolicyService:
+    """Service providing operations for access policies.
+
+    Service is accessed via a url of the form
+    '/services/accesspolicy?ws.op=...
+    """
+
+    implements(IAccessPolicyService)
+
+    @property
+    def name(self):
+        """See `IService`."""
+        return 'accesspolicy'
+
+    def getAccessPolicies(self):
+        policies = []
+        for x, policy in enumerate(InformationVisibilityPolicy):
+            item = dict(
+                index=x,
+                value=policy.token,
+                title=policy.title,
+                description=policy.value.description
+            )
+            policies.append(item)
+        return simplejson.dumps(policies, cls=ResourceJSONEncoder)

=== added file 'lib/lp/registry/services/configure.zcml'
--- lib/lp/registry/services/configure.zcml	1970-01-01 00:00:00 +0000
+++ lib/lp/registry/services/configure.zcml	2012-02-23 23:37:18 +0000
@@ -0,0 +1,18 @@
+<!-- Copyright 2012 Canonical Ltd.  This software is licensed under the
+     GNU Affero General Public License version 3 (see the file LICENSE).
+-->
+
+<configure
+    xmlns="http://namespaces.zope.org/zope"
+    xmlns:browser="http://namespaces.zope.org/browser"
+    i18n_domain="launchpad">
+
+    <!-- Named Services -->
+    <securedutility
+        name="accesspolicy"
+        class="lp.registry.services.accesspolicyservice.AccessPolicyService"
+        provides="lp.app.interfaces.services.IService">
+        <allow
+            interface="lp.registry.interfaces.accesspolicyservice.IAccessPolicyService"/>
+    </securedutility>
+</configure>

=== added directory 'lib/lp/registry/services/tests'
=== added file 'lib/lp/registry/services/tests/__init__.py'
=== added file 'lib/lp/registry/services/tests/test_accesspolicyservice.py'
--- lib/lp/registry/services/tests/test_accesspolicyservice.py	1970-01-01 00:00:00 +0000
+++ lib/lp/registry/services/tests/test_accesspolicyservice.py	2012-02-23 23:37:18 +0000
@@ -0,0 +1,78 @@
+# Copyright 2012 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+__metaclass__ = type
+
+
+import simplejson
+
+from zope.component import getUtility
+
+from lp.app.enums import InformationVisibilityPolicy
+from lp.registry.services.accesspolicyservice import AccessPolicyService
+from lp.services.webapp.interfaces import ILaunchpadRoot
+from lp.services.webapp.publisher import canonical_url
+from lp.testing import WebServiceTestCase, TestCaseWithFactory
+from lp.testing.layers import AppServerLayer
+from lp.testing.pages import LaunchpadWebServiceCaller
+
+
+class ApiTestMixin:
+    """Common tests for launchpadlib and webservice."""
+
+    def test_getAccessPolicies(self):
+        # Test the getAccessPolicies method.
+        json_policies = self._getAccessPolicies()
+        policies = simplejson.loads(json_policies)
+        expected_polices = []
+        for x, policy in enumerate(InformationVisibilityPolicy):
+            item = dict(
+                index=x,
+                value=policy.token,
+                title=policy.title,
+                description=policy.value.description
+            )
+            expected_polices.append(item)
+        self.assertContentEqual(expected_polices, policies)
+
+
+class TestWebService(WebServiceTestCase, ApiTestMixin):
+    """Test the web service interface for the Access Policy Service."""
+
+    def setUp(self):
+        super(TestWebService, self).setUp()
+        self.webservice = LaunchpadWebServiceCaller(
+            'launchpad-library', 'salgado-change-anything')
+
+    def test_url(self):
+        # Test that the url for the service is correct.
+        service = AccessPolicyService()
+        root_app = getUtility(ILaunchpadRoot)
+        self.assertEqual(
+            '%s+services/accesspolicy' % canonical_url(root_app),
+            canonical_url(service))
+
+    def _named_get(self, api_method, **kwargs):
+        return self.webservice.named_get(
+            '/+services/accesspolicy',
+            api_method, api_version='devel', **kwargs).jsonBody()
+
+    def _getAccessPolicies(self):
+        return self._named_get('getAccessPolicies')
+
+
+class TestLaunchpadlib(TestCaseWithFactory, ApiTestMixin):
+    """Test launchpadlib access for the Access Policy Service."""
+
+    layer = AppServerLayer
+
+    def setUp(self):
+        super(TestLaunchpadlib, self).setUp()
+        self.launchpad = self.factory.makeLaunchpadService()
+
+    def _getAccessPolicies(self):
+        # XXX 2012-02-23 wallyworld bug 681767
+        # Launchpadlib can't do relative url's
+        service = self.launchpad.load(
+            '%s/+services/accesspolicy' % self.launchpad._root_uri)
+        return service.getAccessPolicies()

=== added file 'lib/lp/services/webservice/services.py'
--- lib/lp/services/webservice/services.py	1970-01-01 00:00:00 +0000
+++ lib/lp/services/webservice/services.py	2012-02-23 23:37:18 +0000
@@ -0,0 +1,32 @@
+# Copyright 2009-2011 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""A class for the top-level link to the services factory."""
+
+__metaclass__ = type
+__all__ = [
+    'IServicesLink',
+    'ServicesLink',
+    ]
+
+from lazr.restful.interfaces import ITopLevelEntryLink
+from zope.interface import implements
+
+from lp.app.interfaces.services import IServiceFactory
+from lp.services.webapp.interfaces import ICanonicalUrlData
+
+
+class IServicesLink(ITopLevelEntryLink, ICanonicalUrlData):
+    """A marker interface."""
+
+
+class ServicesLink:
+    """The top-level link to the services factory."""
+    implements(IServicesLink)
+
+    link_name = 'services'
+    entry_type = IServiceFactory
+
+    inside = None
+    path = 'services'
+    rootsite = 'api'