Merge lp:~anso/nova/paste into lp:~hudson-openstack/nova/trunk

[Todd Willey]

Uses paste.deploy to make application running configurable. This includes the ability to swap out middlewares, define new endpoints, and generally move away from having code to build wsgi routers and middleware chains into a configurable, extensible method for running wsgi servers.

[Jay Pipes (jaypipes)]

Hi Todd!

Good direction and starting point :) Here are some notes...

1) pep8 will fail

I can see a number of things that will fail pep8 checks, so before going further, you may want to do:

pep8 --repeat bin/* nova

and fix all the warnings/errors reported

2) i18n'ing

You will need to do:

gettext.install('nova', unicode=1)

*before* importing wsgi/flags from nova. The reason is because gettext.install() places the _() method into the builtins namespace. If nova.wsgi or nova.flags has any text strings that are surrounded by _(), then this will result in an error.

In addition, please wrap all text strings that could be output to a user with the _() function.

For instance, here:

71 + LOG.debug("Paste config at %s has no secion for known apis", paste_config)
72 + print "Paste config at %s has no secion for any known apis" % paste_config

the strings should be wrapped in _()...

3)

This code could be simplified/made clearer:

86 + ip = config.get('host', None)
...
91 + host = config.get("%s_host" % api, None) or ip or '0.0.0.0'

to:

ip = config.get('host', '0.0.0.0')
...
host = config.get("%s_host" % api, ip)

Cheers!
jay

review: Needs Fixing

[lp:~anso/nova/paste]

475. By Todd Willey on 2010-12-29

Merge trunk.

476. By Todd Willey on 2010-12-29

Pep-8 cleanup.

477. By Todd Willey on 2010-12-29

i18n

478. By Todd Willey on 2010-12-30

Clean up how we determine IP to bind to.

479. By Todd Willey on 2010-12-30

remove cloudpipe from paste config

480. By Todd Willey on 2010-12-30

pep 8

[Todd Willey (xtoddx)]

Thanks! I started this branch before i18n landed, and apparently missed a few things. It should all be fixed up now, as well as the pep-8 line spacing.

[Jay Pipes (jaypipes)]

sweetness.

Suggestion: get with annegentle on IRC to assist her in documenting how to spin up nova-api-paste.

Nice work, Todd :)

review: Approve

[Devin Carlen (devcamcar)]

sweet

review: Approve

=== added file 'bin/nova-api-paste'
--- bin/nova-api-paste	1970-01-01 00:00:00 +0000
+++ bin/nova-api-paste	2010-12-30 07:23:08 +0000
@@ -0,0 +1,109 @@
+#!/usr/bin/env python
+# pylint: disable-msg=C0103
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright 2010 United States Government as represented by the
+# Administrator of the National Aeronautics and Space Administration.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+
+"""Starter script for Nova API."""
+
+import gettext
+import logging
+import os
+import sys
+
+from paste import deploy
+
+# If ../nova/__init__.py exists, add ../ to Python search path, so that
+# it will override what happens to be installed in /usr/(local/)lib/python...
+possible_topdir = os.path.normpath(os.path.join(os.path.abspath(sys.argv[0]),
+                                   os.pardir,
+                                   os.pardir))
+if os.path.exists(os.path.join(possible_topdir, 'nova', '__init__.py')):
+    sys.path.insert(0, possible_topdir)
+
+gettext.install('nova', unicode=1)
+
+from nova import flags
+from nova import wsgi
+
+LOG = logging.getLogger('nova.api')
+LOG.setLevel(logging.DEBUG)
+LOG.addHandler(logging.StreamHandler())
+
+FLAGS = flags.FLAGS
+
+API_ENDPOINTS = ['ec2', 'openstack']
+
+
+def load_configuration(paste_config):
+    """Load the paste configuration from the config file and return it."""
+    config = None
+    # Try each known name to get the global DEFAULTS, which will give ports
+    for name in API_ENDPOINTS:
+        try:
+            config = deploy.appconfig("config:%s" % paste_config, name=name)
+        except LookupError:
+            pass
+        if config:
+            verbose = config.get('verbose', None)
+            if verbose:
+                FLAGS.verbose = int(verbose) == 1
+                if FLAGS.verbose:
+                    logging.getLogger().setLevel(logging.DEBUG)
+            return config
+    LOG.debug(_("Paste config at %s has no secion for known apis"),
+              paste_config)
+    print _("Paste config at %s has no secion for any known apis") % \
+          paste_config
+    os.exit(1)
+
+
+def launch_api(paste_config_file, section, server, port, host):
+    """Launch an api server from the specified port and IP."""
+    LOG.debug(_("Launching %s api on %s:%s"), section, host, port)
+    app = deploy.loadapp('config:%s' % paste_config_file, name=section)
+    server.start(app, int(port), host)
+
+
+def run_app(paste_config_file):
+    LOG.debug(_("Using paste.deploy config at: %s"), configfile)
+    config = load_configuration(paste_config_file)
+    LOG.debug(_("Configuration: %r"), config)
+    server = wsgi.Server()
+    ip = config.get('host', '0.0.0.0')
+    for api in API_ENDPOINTS:
+        port = config.get("%s_port" % api, None)
+        if not port:
+            continue
+        host = config.get("%s_host" % api, ip)
+        launch_api(configfile, api, server, port, host)
+    LOG.debug(_("All api servers launched, now waiting"))
+    server.wait()
+
+
+if __name__ == '__main__':
+    FLAGS(sys.argv)
+    configfiles = ['/etc/nova/nova-api.conf']
+    if os.path.exists(os.path.join(possible_topdir, 'nova', '__init__.py')):
+        configfiles.insert(0,
+                       os.path.join(possible_topdir, 'etc', 'nova-api.conf'))
+    for configfile in configfiles:
+        if os.path.exists(configfile):
+            run_app(configfile)
+            break
+        else:
+            LOG.debug(_("Skipping missing configuration: %s"), configfile)

=== added directory 'etc'
=== added file 'etc/nova-api.conf'
--- etc/nova-api.conf	1970-01-01 00:00:00 +0000
+++ etc/nova-api.conf	2010-12-30 07:23:08 +0000
@@ -0,0 +1,63 @@
+[DEFAULT]
+verbose = 1
+ec2_port = 8773
+ec2_address = 0.0.0.0
+openstack_port = 8774
+openstack_address = 0.0.0.0
+
+#######
+# EC2 #
+#######
+
+[composite:ec2]
+use = egg:Paste#urlmap
+/: ec2versions
+/services: ec2api
+/latest: ec2metadata
+/200: ec2metadata
+/1.0: ec2metadata
+
+[pipeline:ec2api]
+pipeline = authenticate router authorizer ec2executor
+
+[filter:authenticate]
+paste.filter_factory = nova.api.ec2:authenticate_factory
+
+[filter:router]
+paste.filter_factory = nova.api.ec2:router_factory
+
+[filter:authorizer]
+paste.filter_factory = nova.api.ec2:authorizer_factory
+
+[app:ec2executor]
+paste.app_factory = nova.api.ec2:executor_factory
+
+[app:ec2versions]
+paste.app_factory = nova.api.ec2:versions_factory
+
+[app:ec2metadata]
+paste.app_factory = nova.api.ec2.metadatarequesthandler:metadata_factory
+
+#############
+# Openstack #
+#############
+
+[composite:openstack]
+use = egg:Paste#urlmap
+/: osversions
+/v1.0: openstackapi
+
+[pipeline:openstackapi]
+pipeline = auth ratelimit osapi
+
+[filter:auth]
+paste.filter_factory = nova.api.openstack.auth:auth_factory
+
+[filter:ratelimit]
+paste.filter_factory = nova.api.openstack.ratelimiting:ratelimit_factory
+
+[app:osapi]
+paste.app_factory = nova.api.openstack:router_factory
+
+[app:osversions]
+paste.app_factory = nova.api.openstack:versions_factory

=== modified file 'nova/api/ec2/__init__.py'
--- nova/api/ec2/__init__.py	2010-12-22 21:50:58 +0000
+++ nova/api/ec2/__init__.py	2010-12-30 07:23:08 +0000
@@ -294,10 +294,9 @@
         args = req.environ['ec2.action_args']
 
         api_request = apirequest.APIRequest(controller, action)
+        result = None
         try:
             result = api_request.send(context, **args)
-            req.headers['Content-Type'] = 'text/xml'
-            return result
         except exception.ApiError as ex:
 
             if ex.code:
@@ -307,6 +306,12 @@
         # TODO(vish): do something more useful with unknown exceptions
         except Exception as ex:
             return self._error(req, type(ex).__name__, str(ex))
+        else:
+            resp = webob.Response()
+            resp.status = 200
+            resp.headers['Content-Type'] = 'text/xml'
+            resp.body = str(result)
+            return resp
 
     def _error(self, req, code, message):
         logging.error("%s: %s", code, message)
@@ -318,3 +323,49 @@
                      '<Message>%s</Message></Error></Errors>'
                      '<RequestID>?</RequestID></Response>' % (code, message))
         return resp
+
+
+class Versions(wsgi.Application):
+
+    @webob.dec.wsgify
+    def __call__(self, req):
+        """Respond to a request for all EC2 versions."""
+        # available api versions
+        versions = [
+            '1.0',
+            '2007-01-19',
+            '2007-03-01',
+            '2007-08-29',
+            '2007-10-10',
+            '2007-12-15',
+            '2008-02-01',
+            '2008-09-01',
+            '2009-04-04',
+        ]
+        return ''.join('%s\n' % v for v in versions)
+
+
+def authenticate_factory(global_args, **local_args):
+    def authenticator(app):
+        return Authenticate(app)
+    return authenticator
+
+
+def router_factory(global_args, **local_args):
+    def router(app):
+        return Router(app)
+    return router
+
+
+def authorizer_factory(global_args, **local_args):
+    def authorizer(app):
+        return Authorizer(app)
+    return authorizer
+
+
+def executor_factory(global_args, **local_args):
+    return Executor()
+
+
+def versions_factory(global_args, **local_args):
+    return Versions()

=== modified file 'nova/api/ec2/metadatarequesthandler.py'
--- nova/api/ec2/metadatarequesthandler.py	2010-12-22 21:50:58 +0000
+++ nova/api/ec2/metadatarequesthandler.py	2010-12-30 07:23:08 +0000
@@ -79,3 +79,7 @@
         if data is None:
             raise webob.exc.HTTPNotFound()
         return self.print_data(data)
+
+
+def metadata_factory(global_args, **local_args):
+    return MetadataRequestHandler()

=== modified file 'nova/api/openstack/__init__.py'
--- nova/api/openstack/__init__.py	2010-12-28 21:49:36 +0000
+++ nova/api/openstack/__init__.py	2010-12-30 07:23:08 +0000
@@ -20,7 +20,6 @@
 WSGI middleware for OpenStack API controllers.
 """
 
-import json
 import time
 
 import logging
@@ -41,7 +40,6 @@
 from nova.api.openstack import ratelimiting
 from nova.api.openstack import servers
 from nova.api.openstack import sharedipgroups
-from nova.auth import manager
 
 
 FLAGS = flags.FLAGS
@@ -113,3 +111,24 @@
                         controller=sharedipgroups.Controller())
 
         super(APIRouter, self).__init__(mapper)
+
+
+class Versions(wsgi.Application):
+    @webob.dec.wsgify
+    def __call__(self, req):
+        """Respond to a request for all OpenStack API versions."""
+        response = {
+                "versions": [
+                    dict(status="CURRENT", id="v1.0")]}
+        metadata = {
+            "application/xml": {
+                "attributes": dict(version=["status", "id"])}}
+        return wsgi.Serializer(req.environ, metadata).to_content_type(response)
+
+
+def router_factory(global_cof, **local_conf):
+    return APIRouter()
+
+
+def versions_factory(global_conf, **local_conf):
+    return Versions()

=== modified file 'nova/api/openstack/auth.py'
--- nova/api/openstack/auth.py	2010-12-23 19:17:53 +0000
+++ nova/api/openstack/auth.py	2010-12-30 07:23:08 +0000
@@ -133,3 +133,9 @@
             token = self.db.auth_create_token(ctxt, token_dict)
             return token, user
         return None, None
+
+
+def auth_factory(global_conf, **local_conf):
+    def auth(app):
+        return AuthMiddleware(app)
+    return auth

=== modified file 'nova/api/openstack/ratelimiting/__init__.py'
--- nova/api/openstack/ratelimiting/__init__.py	2010-12-24 00:09:52 +0000
+++ nova/api/openstack/ratelimiting/__init__.py	2010-12-30 07:23:08 +0000
@@ -219,3 +219,9 @@
             # No delay
             return None
         return float(resp.getheader('X-Wait-Seconds'))
+
+
+def ratelimit_factory(global_conf, **local_conf):
+    def rl(app):
+        return RateLimitingMiddleware(app)
+    return rl

=== modified file 'tools/pip-requires'
--- tools/pip-requires	2010-12-22 20:59:53 +0000
+++ tools/pip-requires	2010-12-30 07:23:08 +0000
@@ -22,4 +22,6 @@
 greenlet==0.3.1
 nose
 bzr
-Twisted>=10.1.0
\ No newline at end of file
+Twisted>=10.1.0
+PasteDeploy
+paste