Merge lp:~vishvananda/nova/keystone-migration-ec2 into lp:~hudson-openstack/nova/trunk

[Vish Ishaya]

Next round of prep for keystone integration.

 * adds middleware for authenticating ec2 signature with keystone
 * adds middleware for converting keystone response into request context
 * gives examples of alternative pipelines for keystone integration

Next steps:
 * provide default config with no keystone integration (perhaps setting every context to admin?)
 * write authmanager to keystone conversion code
 * add api extension to create and destroy access/secret keys
 * deprecate authmanager
 * rename project to tenant

[lp:~vishvananda/nova/keystone-migration-ec2]

1420. By Vish Ishaya on 2011-08-12

remove extra log statements

[Brian Lamar (blamar)]

A few questions first:

Why include AdminContext middleware if it's not being used? Does it provide any benefit currently?

The KeystoneContext middleware is strikingly similar to the KeystoneAuthShim in keystone/middleware/nova_auth_token.py (in the Keystone project)... is this just another candidate for openstack-common?

Then some nits:

84 +"""
85 +Common Auth Middleware.
86 +
87 +"""
88 +
89 +from nova import context
90 +from nova import flags
91 +from nova import wsgi
92 +import webob.dec
93 +import webob.exc

"""Common Auth Middleware."""

import webob.dec
import webob.exc

from nova import context
from nova import flags
from nova import wsgi

104 + def __init__(self, context, *args, **kwargs):

Can you put a spacer before this?

139 + except:

except KeyError:

204 + except KeyError, e:

No need to specify e.

review: Needs Fixing

[Devin Carlen (devcamcar)]

Vish, this is looking good. I do have a couple of questions:

Is the intent here to return a dummy admin context, with no relationship to an auth system or real credentials? I ask because we have made efforts not to hardcode 'admin'/'admin' in other places.

123 + ctx = context.RequestContext('admin',
124 + 'admin',
125 + is_admin=True,
126 + remote_address=remote_address)

Also, you mentioned renaming project to tenant, but that work doesn't seem to have been done in this branch.

review: Needs Information

[Vish Ishaya (vishvananda)]

> Also, you mentioned renaming project to tenant, but that work doesn't seem to
> have been done in this branch.

The renaming was in next steps. Addressing the admin context point in my comment to Brian

[Vish Ishaya (vishvananda)]

> A few questions first:
>
> Why include AdminContext middleware if it's not being used? Does it provide
> any benefit currently?

I can remove this for now. This was more just an example of how we could offer a version with no authentication if we want to be able to run without keystone. The plan is to deprecate AuthManager, so if we want a way to run without keystone, we need to create a context without authentication. This could be just allowing a user to authenticate for any user and tenant or giving everyone an adimn context. I'm going to address this in a separate patch so I will remove it for now.
>
> The KeystoneContext middleware is strikingly similar to the KeystoneAuthShim
> in keystone/middleware/nova_auth_token.py (in the Keystone project)... is this
> just another candidate for openstack-common?

The auth-shim was a temporary shim until we could manage users and projects in keystone. The idea is that this middleware will replace the keystone shim. I think it belongs here instead of keystone.

>
>
> Then some nits:
>
> 84 +"""
> 85 +Common Auth Middleware.
> 86 +
> 87 +"""
> 88 +
> 89 +from nova import context
> 90 +from nova import flags
> 91 +from nova import wsgi
> 92 +import webob.dec
> 93 +import webob.exc
>
> """Common Auth Middleware."""
>
> import webob.dec
> import webob.exc
>
> from nova import context
> from nova import flags
> from nova import wsgi

done

>
>
> 104 + def __init__(self, context, *args, **kwargs):
>
> Can you put a spacer before this?

done

>
>
> 139 + except:
>
> except KeyError:

done
>
>
> 204 + except KeyError, e:
>
> No need to specify e.

done

[lp:~vishvananda/nova/keystone-migration-ec2]

1421. By Vish Ishaya on 2011-08-12

updates from review

1422. By Vish Ishaya on 2011-08-12

removed admincontext middleware

[Dan Prince (dan-prince)]

Removing the use_forwarded_for flag from nova/api/ec2/__init__.py will cause an AttributeError (because it is still being used in __init__.py).

review: Needs Fixing

[Vish Ishaya (vishvananda)]

> Removing the use_forwarded_for flag from nova/api/ec2/__init__.py will cause
> an AttributeError (because it is still being used in __init__.py).

Thanks Dan. Fixed the other files to declare the flag.

[lp:~vishvananda/nova/keystone-migration-ec2]

1423. By Vish Ishaya on 2011-08-18

declare the use_forwarded_for flag

[Dan Prince (dan-prince)]

Looks good.

review: Approve

[Brian Lamar (blamar)]

Good here.

review: Approve

=== modified file 'etc/nova/api-paste.ini'
--- etc/nova/api-paste.ini	2011-08-15 22:09:42 +0000
+++ etc/nova/api-paste.ini	2011-08-18 22:43:01 +0000
@@ -20,7 +20,8 @@
 
 [pipeline:ec2cloud]
 pipeline = logrequest authenticate cloudrequest authorizer ec2executor
-#pipeline = logrequest ec2lockout authenticate cloudrequest authorizer ec2executor
+# NOTE(vish): use the following pipeline for keystone
+# pipeline = logrequest totoken authtoken keystonecontext cloudrequest authorizer ec2executor
 
 [pipeline:ec2admin]
 pipeline = logrequest authenticate adminrequest authorizer ec2executor
@@ -37,6 +38,9 @@
 [filter:ec2lockout]
 paste.filter_factory = nova.api.ec2:Lockout.factory
 
+[filter:totoken]
+paste.filter_factory = nova.api.ec2:ToToken.factory
+
 [filter:authenticate]
 paste.filter_factory = nova.api.ec2:Authenticate.factory
 
@@ -72,9 +76,13 @@
 
 [pipeline:openstackapi10]
 pipeline = faultwrap auth ratelimit osapiapp10
+# NOTE(vish): use the following pipeline for keystone
+#pipeline = faultwrap authtoken keystonecontext ratelimit osapiapp10
 
 [pipeline:openstackapi11]
 pipeline = faultwrap auth ratelimit extensions osapiapp11
+# NOTE(vish): use the following pipeline for keystone
+# pipeline = faultwrap authtoken keystonecontext ratelimit extensions osapiapp11
 
 [filter:faultwrap]
 paste.filter_factory = nova.api.openstack:FaultWrapper.factory
@@ -99,3 +107,22 @@
 
 [app:osversionapp]
 paste.app_factory = nova.api.openstack.versions:Versions.factory
+
+##########
+# Shared #
+##########
+
+[filter:keystonecontext]
+paste.filter_factory = nova.api.auth:KeystoneContext.factory
+
+[filter:authtoken]
+paste.filter_factory = keystone.middleware.auth_token:filter_factory
+service_protocol = http
+service_host = 127.0.0.1
+service_port = 808
+auth_host = 127.0.0.1
+auth_port = 5001
+auth_protocol = http
+auth_uri = http://127.0.0.1:5000/
+admin_token = 999888777666
+

=== added file 'nova/api/auth.py'
--- nova/api/auth.py	1970-01-01 00:00:00 +0000
+++ nova/api/auth.py	2011-08-18 22:43:01 +0000
@@ -0,0 +1,75 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright (c) 2011 OpenStack, LLC
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+"""
+Common Auth Middleware.
+
+"""
+
+import webob.dec
+import webob.exc
+
+from nova import context
+from nova import flags
+from nova import wsgi
+
+
+FLAGS = flags.FLAGS
+flags.DEFINE_boolean('use_forwarded_for', False,
+                     'Treat X-Forwarded-For as the canonical remote address. '
+                     'Only enable this if you have a sanitizing proxy.')
+
+
+class InjectContext(wsgi.Middleware):
+    """Add a 'nova.context' to WSGI environ."""
+
+    def __init__(self, context, *args, **kwargs):
+        self.context = context
+        super(InjectContext, self).__init__(*args, **kwargs)
+
+    @webob.dec.wsgify(RequestClass=wsgi.Request)
+    def __call__(self, req):
+        req.environ['nova.context'] = self.context
+        return self.application
+
+
+class KeystoneContext(wsgi.Middleware):
+    """Make a request context from keystone headers"""
+
+    @webob.dec.wsgify(RequestClass=wsgi.Request)
+    def __call__(self, req):
+        try:
+            user_id = req.headers['X_USER']
+        except KeyError:
+            return webob.exc.HTTPUnauthorized()
+        # get the roles
+        roles = [r.strip() for r in req.headers.get('X_ROLE', '').split(',')]
+        project_id = req.headers['X_TENANT']
+        # Get the auth token
+        auth_token = req.headers.get('X_AUTH_TOKEN',
+                                     req.headers.get('X_STORAGE_TOKEN'))
+
+        # Build a context, including the auth_token...
+        remote_address = req.remote_addr
+        if FLAGS.use_forwarded_for:
+            remote_address = req.headers.get('X-Forwarded-For', remote_address)
+        ctx = context.RequestContext(user_id,
+                                     project_id,
+                                     roles=roles,
+                                     auth_token=auth_token,
+                                     remote_address=remote_address)
+
+        req.environ['nova.context'] = ctx
+        return self.application

=== modified file 'nova/api/ec2/__init__.py'
--- nova/api/ec2/__init__.py	2011-08-15 22:09:42 +0000
+++ nova/api/ec2/__init__.py	2011-08-18 22:43:01 +0000
@@ -20,6 +20,7 @@
 
 """
 
+import httplib2
 import webob
 import webob.dec
 import webob.exc
@@ -37,15 +38,16 @@
 
 FLAGS = flags.FLAGS
 LOG = logging.getLogger("nova.api")
-flags.DEFINE_boolean('use_forwarded_for', False,
-                     'Treat X-Forwarded-For as the canonical remote address. '
-                     'Only enable this if you have a sanitizing proxy.')
 flags.DEFINE_integer('lockout_attempts', 5,
                      'Number of failed auths before lockout.')
 flags.DEFINE_integer('lockout_minutes', 15,
                      'Number of minutes to lockout if triggered.')
 flags.DEFINE_integer('lockout_window', 15,
                      'Number of minutes for lockout window.')
+flags.DEFINE_string('keystone_ec2_url',
+                    'http://localhost:5000/v2.0/ec2tokens',
+                    'URL to get token from ec2 request.')
+flags.DECLARE('use_forwarded_for', 'nova.api.auth')
 
 
 class RequestLogging(wsgi.Middleware):
@@ -138,6 +140,49 @@
         return res
 
 
+class ToToken(wsgi.Middleware):
+    """Authenticate an EC2 request with keystone and convert to token."""
+
+    @webob.dec.wsgify(RequestClass=wsgi.Request)
+    def __call__(self, req):
+        # Read request signature and access id.
+        try:
+            signature = req.params['Signature']
+            access = req.params['AWSAccessKeyId']
+        except KeyError:
+            raise webob.exc.HTTPBadRequest()
+
+        # Make a copy of args for authentication and signature verification.
+        auth_params = dict(req.params)
+        # Not part of authentication args
+        auth_params.pop('Signature')
+
+        # Authenticate the request.
+        client = httplib2.Http()
+        creds = {'ec2Credentials': {'access': access,
+                                    'signature': signature,
+                                    'host': req.host,
+                                    'verb': req.method,
+                                    'path': req.path,
+                                    'params': auth_params,
+                                   }}
+        headers = {'Content-Type': 'application/json'},
+        resp, content = client.request(FLAGS.keystone_ec2_url,
+                                       'POST',
+                                       headers=headers,
+                                       body=utils.dumps(creds))
+        # NOTE(vish): We could save a call to keystone by
+        #             having keystone return token, tenant,
+        #             user, and roles from this call.
+        result = utils.loads(content)
+        # TODO(vish): check for errors
+        token_id = result['auth']['token']['id']
+
+        # Authenticated!
+        req.headers['X-Auth-Token'] = token_id
+        return self.application
+
+
 class Authenticate(wsgi.Middleware):
     """Authenticate an EC2 request and add 'nova.context' to WSGI environ."""
 
@@ -147,7 +192,7 @@
         try:
             signature = req.params['Signature']
             access = req.params['AWSAccessKeyId']
-        except KeyError, e:
+        except KeyError:
             raise webob.exc.HTTPBadRequest()
 
         # Make a copy of args for authentication and signature verification.

=== modified file 'nova/api/ec2/metadatarequesthandler.py'
--- nova/api/ec2/metadatarequesthandler.py	2011-06-26 00:29:14 +0000
+++ nova/api/ec2/metadatarequesthandler.py	2011-08-18 22:43:01 +0000
@@ -30,6 +30,7 @@
 
 LOG = logging.getLogger('nova.api.ec2.metadata')
 FLAGS = flags.FLAGS
+flags.DECLARE('use_forwarded_for', 'nova.api.auth')
 
 
 class MetadataRequestHandler(wsgi.Application):

=== modified file 'nova/tests/api/openstack/fakes.py'
--- nova/tests/api/openstack/fakes.py	2011-08-15 22:09:42 +0000
+++ nova/tests/api/openstack/fakes.py	2011-08-18 22:43:01 +0000
@@ -32,6 +32,7 @@
 from nova import wsgi
 import nova.api.openstack.auth
 from nova.api import openstack
+from nova.api import auth as api_auth
 from nova.api.openstack import auth
 from nova.api.openstack import extensions
 from nova.api.openstack import versions
@@ -83,9 +84,9 @@
             ctxt = fake_auth_context
         else:
             ctxt = context.RequestContext('fake', 'fake')
-        api10 = openstack.FaultWrapper(wsgi.InjectContext(ctxt,
+        api10 = openstack.FaultWrapper(api_auth.InjectContext(ctxt,
               limits.RateLimitingMiddleware(inner_app10)))
-        api11 = openstack.FaultWrapper(wsgi.InjectContext(ctxt,
+        api11 = openstack.FaultWrapper(api_auth.InjectContext(ctxt,
               limits.RateLimitingMiddleware(
                   extensions.ExtensionMiddleware(inner_app11))))
     else:

=== modified file 'nova/tests/test_api.py'
--- nova/tests/test_api.py	2011-08-09 12:47:47 +0000
+++ nova/tests/test_api.py	2011-08-18 22:43:01 +0000
@@ -32,6 +32,7 @@
 from nova import exception
 from nova import test
 from nova import wsgi
+from nova.api import auth
 from nova.api import ec2
 from nova.api.ec2 import apirequest
 from nova.api.ec2 import cloud
@@ -199,7 +200,7 @@
         # NOTE(vish): skipping the Authorizer
         roles = ['sysadmin', 'netadmin']
         ctxt = context.RequestContext('fake', 'fake', roles=roles)
-        self.app = wsgi.InjectContext(ctxt,
+        self.app = auth.InjectContext(ctxt,
                 ec2.Requestify(ec2.Authorizer(ec2.Executor()),
                                'nova.api.ec2.cloud.CloudController'))
 

=== modified file 'nova/wsgi.py'
--- nova/wsgi.py	2011-07-22 19:47:41 +0000
+++ nova/wsgi.py	2011-08-18 22:43:01 +0000
@@ -274,18 +274,6 @@
         return self.process_response(response)
 
 
-class InjectContext(Middleware):
-    """Add a 'nova.context' to WSGI environ."""
-    def __init__(self, context, *args, **kwargs):
-        self.context = context
-        super(InjectContext, self).__init__(*args, **kwargs)
-
-    @webob.dec.wsgify(RequestClass=Request)
-    def __call__(self, req):
-        req.environ['nova.context'] = self.context
-        return self.application
-
-
 class Debug(Middleware):
     """Helper class for debugging a WSGI application.