Merge lp:~usc-isi/nova/instance_type_extra_specs into lp:~hudson-openstack/nova/trunk

[Lorin Hochstein]

Adds support for "extra specs", additional capability requirements associated with instance types.

The instance_type dictionary now has a new extra_specs field.

Adds a new table to the database: InstanceTypeExtraSpecs. This is modeled on the existing InstanceMetadata table, except that it is associated with additional capability requirements of instance types rather than generic metadata for instances.

The InstanceTypeFilter has been modified to check for extra specs.

This will ultimately be needed for supporting heterogeneous instances: we'll annotate the instance types with info about whether GPUs are present, so users will be able to request something like a "cg1.4xlarge".

Includes api support as an extension for querying and modifying this info.

[Mark Washenberger (markwash)]

I'm unsure how this would show up in the api--and I would like to see that as part of this change

However, at minimum you need to bump your migration number.

[Lorin Hochstein (lorinh)]

Migration number has been bumped.

This code shouldn't affect the api (at least, not the user api). From a user's point of view, they are just requesting an instance type by name. For example, if you wanted a compute node with GPUs, the user would request something like a "cg1.4xlarge" instead of an "m1.large".

(Somebody needs to add the gpu-related metadata to cg1.4xlarge instance type, but this will be site-specific. Is there currently an admin api for adding/modifying instance types? I assumed that this was currently done by just modifying the database directly. We'd ultimately like to modify the dashboard to allow an admin to edit this type of content).

To use this data:

- The compute service needs to report its capabilities (this code is already there)
- The scheduler needs to check if an instance type's metadata matches a compute service's capabilities. (the distributed scheduler code has some generic facilities for filtering, we would just filter on the instance type metadata if it is present).

[Mark Washenberger (markwash)]

What you're saying is that a user finds out about the capabilities associated with each flavor/instance_type is by some mechanism outside of the api. (Or in any case that they _can_ find out about the capabilities without using the api for now.) Does that sound right?

[Lorin Hochstein (lorinh)]

Mark:

That's right. We expect that the local installation would have user documentation that describes which instance types were associated with which capabilities.

We don't expect the users to be able to query for this information directly via the api.

[Brian Waldon (bcwaldon)]

The code looks rather solid, however I do have a few higher-level questions:

1) The concept of 'metadata' doesn't fit in very well here. The data you are providing is integral to the provisioning process. Our usage of metadata in other places represents user-defined key/value pairs that our services will never care about. cpu_arch would probably fit better as a core instance type attribute, while we could look at the others as something closer to "optional attributes". An instance could use one of the standard instance types with a don't-care attitude when provisioning towards something like xpu_arch, while you could provide more specific instance types with those optional attributes being explicitly defined.

2) I'm not completely sold on the idea that a deployer should provide another method for communication of instance types. I would love to integrate this information into the existing flavors resource in the openstack api.

3) Am I correct in assuming another merge prop will be coming to fill in the scheduler integration? Otherwise, there is no way to respect whatever values are actually indicated in the metadata.

review: Needs Information

[Josh Kearney (jk0)]

Setting to WIP until above feedback is worked in.

[Lorin Hochstein (lorinh)]

On Jun 1, 2011, at 3:54 PM, Brian Waldon wrote:

> Review: Needs Information
> The code looks rather solid, however I do have a few higher-level questions:
>
> 1) The concept of 'metadata' doesn't fit in very well here. The data you are providing is integral to the provisioning process. Our usage of metadata in other places represents user-defined key/value pairs that our services will never care about. cpu_arch would probably fit better as a core instance type attribute, while we could look at the others as something closer to "optional attributes". An instance could use one of the standard instance types with a don't-care attitude when provisioning towards something like xpu_arch, while you could provide more specific instance types with those optional attributes being explicitly defined.
>

How about the name "InstanceTypeExtraSpecs"? This would keep terminology consistent with the distributed zone scheduler code, which uses the term "specs" to refer to requested attributes that need to match the capabilities of a compute node.

> 2) I'm not completely sold on the idea that a deployer should provide another method for communication of instance types. I would love to integrate this information into the existing flavors resource in the openstack api.
>

I must admit that I'm only familiar with the ec2 API. I'll take a look at the openstack api to try and figure out how to support querying this type of data.

> 3) Am I correct in assuming another merge prop will be coming to fill in the scheduler integration? Otherwise, there is no way to respect whatever values are actually indicated in the metadata.
>

Yes, this will come in a future merge prop, although I may end up adding it in to the nova/scheduler/HostFilterScheduler code in this merge proposal once I've made the additional changes mentioned above.

[Brian Waldon (bcwaldon)]

> > 1) The concept of 'metadata' doesn't fit in very well here. The data you are
> providing is integral to the provisioning process. Our usage of metadata in
> other places represents user-defined key/value pairs that our services will
> never care about. cpu_arch would probably fit better as a core instance type
> attribute, while we could look at the others as something closer to "optional
> attributes". An instance could use one of the standard instance types with a
> don't-care attitude when provisioning towards something like xpu_arch, while
> you could provide more specific instance types with those optional attributes
> being explicitly defined.
> >
> How about the name "InstanceTypeExtraSpecs"? This would keep terminology
> consistent with the distributed zone scheduler code, which uses the term
> "specs" to refer to requested attributes that need to match the capabilities
> of a compute node.

That name definitely feels better.

> > 2) I'm not completely sold on the idea that a deployer should provide
> another method for communication of instance types. I would love to integrate
> this information into the existing flavors resource in the openstack api.
> >
> I must admit that I'm only familiar with the ec2 API. I'll take a look at the
> openstack api to try and figure out how to support querying this type of data.

Something similar to the /servers/<id>/meta resource may be something to look at.

[Brian Waldon (bcwaldon)]

Can you implement the api-layer code as an extension? I don't think this is going to make it into the v1.1 spec.

[Lorin Hochstein (lorinh)]

> Can you implement the api-layer code as an extension? I don't think this is
> going to make it into the v1.1 spec.

Will do.

[Vish Ishaya (vishvananda)]

This seems good. Just a question really. Why is the:
inst_dict[i['name']] = _inst_type_query_to_dict(i)

necessary. It seems like the joined load should handle populating the subobjects. If there is some reason to convert to nested dictionaries, is utils.to_primitive() lacking something that requires a custom method to be written?

review: Needs Information

[Lorin Hochstein (lorinh)]

> This seems good. Just a question really. Why is the:
> inst_dict[i['name']] = _inst_type_query_to_dict(i)
>
> necessary. It seems like the joined load should handle populating the
> subobjects. If there is some reason to convert to nested dictionaries, is
> utils.to_primitive() lacking something that requires a custom method to be
> written?

Just ignorance, this is my first time plumbing the depths of sqlalchemy and I wasn't aware there was a utils.to_primitive function. I'll fix it in the code.

[Lorin Hochstein (lorinh)]

> This seems good. Just a question really. Why is the:
> inst_dict[i['name']] = _inst_type_query_to_dict(i)
>
> necessary. It seems like the joined load should handle populating the
> subobjects. If there is some reason to convert to nested dictionaries, is
> utils.to_primitive() lacking something that requires a custom method to be
> written?

Having gone over the code again, I now recall that a helper method is needed is to convert from:

'extra_specs': [{'key': 'k1', 'value': 'v1', ...}, {'key': 'k2', 'value': 'v2', ...}, ...]

to

'extra_specs': {'k1':'v1', 'k2':'v2', ...}

 I've renamed the helper method and modified the docstring to clarify this in the code.

[Vish Ishaya (vishvananda)]

Ok thanks. I understand now. LGTM.

review: Approve

[Brian Waldon (bcwaldon)]

Thanks for making an extension, Lorin. Here are the last few items:

110, 125: Can you please add a 'os-' prefix to your extension alias? See the volumes extension as an example.

529: This file should probably be moved into nova/tests/api/openstack/contrib. The other test files are fine where they are.

983: I think we might want version 0.6.1, not latest. Thoughts?

review: Needs Fixing

[Lorin Hochstein (lorinh)]

> Thanks for making an extension, Lorin. Here are the last few items:
>
> 110, 125: Can you please add a 'os-' prefix to your extension alias? See the
> volumes extension as an example.
>
> 529: This file should probably be moved into nova/tests/api/openstack/contrib.
> The other test files are fine where they are.
>
> 983: I think we might want version 0.6.1, not latest. Thoughts?

Sure, I think pegging it to 0.6.1 is reasonable.

[Brian Waldon (bcwaldon)]

Thanks, Lorin.

review: Approve

=== added file 'nova/api/openstack/contrib/flavorextraspecs.py'
--- nova/api/openstack/contrib/flavorextraspecs.py	1970-01-01 00:00:00 +0000
+++ nova/api/openstack/contrib/flavorextraspecs.py	2011-06-27 00:06:02 +0000
@@ -0,0 +1,126 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright 2011 University of Southern California
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+""" The instance type extra specs extension"""
+
+from webob import exc
+
+from nova import db
+from nova import quota
+from nova.api.openstack import extensions
+from nova.api.openstack import faults
+from nova.api.openstack import wsgi
+
+
+class FlavorExtraSpecsController(object):
+    """ The flavor extra specs API controller for the Openstack API """
+
+    def _get_extra_specs(self, context, flavor_id):
+        extra_specs = db.api.instance_type_extra_specs_get(context, flavor_id)
+        specs_dict = {}
+        for key, value in extra_specs.iteritems():
+            specs_dict[key] = value
+        return dict(extra_specs=specs_dict)
+
+    def _check_body(self, body):
+        if body == None or body == "":
+            expl = _('No Request Body')
+            raise exc.HTTPBadRequest(explanation=expl)
+
+    def index(self, req, flavor_id):
+        """ Returns the list of extra specs for a givenflavor """
+        context = req.environ['nova.context']
+        return self._get_extra_specs(context, flavor_id)
+
+    def create(self, req, flavor_id, body):
+        self._check_body(body)
+        context = req.environ['nova.context']
+        specs = body.get('extra_specs')
+        try:
+            db.api.instance_type_extra_specs_update_or_create(context,
+                                                              flavor_id,
+                                                              specs)
+        except quota.QuotaError as error:
+            self._handle_quota_error(error)
+        return body
+
+    def update(self, req, flavor_id, id, body):
+        self._check_body(body)
+        context = req.environ['nova.context']
+        if not id in body:
+            expl = _('Request body and URI mismatch')
+            raise exc.HTTPBadRequest(explanation=expl)
+        if len(body) > 1:
+            expl = _('Request body contains too many items')
+            raise exc.HTTPBadRequest(explanation=expl)
+        try:
+            db.api.instance_type_extra_specs_update_or_create(context,
+                                                               flavor_id,
+                                                               body)
+        except quota.QuotaError as error:
+            self._handle_quota_error(error)
+
+        return body
+
+    def show(self, req, flavor_id, id):
+        """ Return a single extra spec item """
+        context = req.environ['nova.context']
+        specs = self._get_extra_specs(context, flavor_id)
+        if id in specs['extra_specs']:
+            return {id: specs['extra_specs'][id]}
+        else:
+            return faults.Fault(exc.HTTPNotFound())
+
+    def delete(self, req, flavor_id, id):
+        """ Deletes an existing extra spec """
+        context = req.environ['nova.context']
+        db.api.instance_type_extra_specs_delete(context, flavor_id, id)
+
+    def _handle_quota_error(self, error):
+        """Reraise quota errors as api-specific http exceptions."""
+        if error.code == "MetadataLimitExceeded":
+            raise exc.HTTPBadRequest(explanation=error.message)
+        raise error
+
+
+class Flavorextraspecs(extensions.ExtensionDescriptor):
+
+    def get_name(self):
+        return "FlavorExtraSpecs"
+
+    def get_alias(self):
+        return "os-flavor-extra-specs"
+
+    def get_description(self):
+        return "Instance type (flavor) extra specs"
+
+    def get_namespace(self):
+        return \
+         "http://docs.openstack.org/ext/flavor_extra_specs/api/v1.1"
+
+    def get_updated(self):
+        return "2011-06-23T00:00:00+00:00"
+
+    def get_resources(self):
+        resources = []
+        res = extensions.ResourceExtension(
+                'os-extra_specs',
+                FlavorExtraSpecsController(),
+                parent=dict(member_name='flavor', collection_name='flavors'))
+
+        resources.append(res)
+        return resources
 
=== modified file 'nova/db/api.py'
--- nova/db/api.py	2011-06-25 19:38:07 +0000
+++ nova/db/api.py	2011-06-27 00:06:02 +0000
@@ -1339,3 +1339,24 @@
 def agent_build_update(context, agent_build_id, values):
     """Update agent build entry."""
     IMPL.agent_build_update(context, agent_build_id, values)
+
+
+####################
+
+
+def instance_type_extra_specs_get(context, instance_type_id):
+    """Get all extra specs for an instance type."""
+    return IMPL.instance_type_extra_specs_get(context, instance_type_id)
+
+
+def instance_type_extra_specs_delete(context, instance_type_id, key):
+    """Delete the given extra specs item."""
+    IMPL.instance_type_extra_specs_delete(context, instance_type_id, key)
+
+
+def instance_type_extra_specs_update_or_create(context, instance_type_id,
+                                               extra_specs):
+    """Create or update instance type extra specs. This adds or modifies the
+    key/value pairs specified in the extra specs dict argument"""
+    IMPL.instance_type_extra_specs_update_or_create(context, instance_type_id,
+                                                    extra_specs)
 
=== modified file 'nova/db/sqlalchemy/api.py'
--- nova/db/sqlalchemy/api.py	2011-06-25 19:38:07 +0000
+++ nova/db/sqlalchemy/api.py	2011-06-27 00:06:02 +0000
@@ -2597,7 +2597,22 @@
 
 @require_admin_context
 def instance_type_create(_context, values):
+    """Create a new instance type. In order to pass in extra specs,
+    the values dict should contain a 'extra_specs' key/value pair:
+
+    {'extra_specs' : {'k1': 'v1', 'k2': 'v2', ...}}
+
+    """
     try:
+        specs = values.get('extra_specs')
+        specs_refs = []
+        if specs:
+            for k, v in specs.iteritems():
+                specs_ref = models.InstanceTypeExtraSpecs()
+                specs_ref['key'] = k
+                specs_ref['value'] = v
+                specs_refs.append(specs_ref)
+        values['extra_specs'] = specs_refs
         instance_type_ref = models.InstanceTypes()
         instance_type_ref.update(values)
         instance_type_ref.save()
@@ -2606,6 +2621,25 @@
     return instance_type_ref
 
 
+def _dict_with_extra_specs(inst_type_query):
+    """Takes an instance type query returned by sqlalchemy
+    and returns it as a dictionary, converting the extra_specs
+    entry from a list of dicts:
+
+    'extra_specs' : [{'key': 'k1', 'value': 'v1', ...}, ...]
+
+    to a single dict:
+
+    'extra_specs' : {'k1': 'v1'}
+
+    """
+    inst_type_dict = dict(inst_type_query)
+    extra_specs = dict([(x['key'], x['value']) for x in \
+                        inst_type_query['extra_specs']])
+    inst_type_dict['extra_specs'] = extra_specs
+    return inst_type_dict
+
+
 @require_context
 def instance_type_get_all(context, inactive=False):
     """
@@ -2614,17 +2648,19 @@
     session = get_session()
     if inactive:
         inst_types = session.query(models.InstanceTypes).\
+                        options(joinedload('extra_specs')).\
                         order_by("name").\
                         all()
     else:
         inst_types = session.query(models.InstanceTypes).\
+                        options(joinedload('extra_specs')).\
                         filter_by(deleted=False).\
                         order_by("name").\
                         all()
     if inst_types:
         inst_dict = {}
         for i in inst_types:
-            inst_dict[i['name']] = dict(i)
+            inst_dict[i['name']] = _dict_with_extra_specs(i)
         return inst_dict
     else:
         raise exception.NoInstanceTypesFound()
@@ -2635,12 +2671,14 @@
     """Returns a dict describing specific instance_type"""
     session = get_session()
     inst_type = session.query(models.InstanceTypes).\
+                    options(joinedload('extra_specs')).\
                     filter_by(id=id).\
                     first()
+
     if not inst_type:
         raise exception.InstanceTypeNotFound(instance_type=id)
     else:
-        return dict(inst_type)
+        return _dict_with_extra_specs(inst_type)
 
 
 @require_context
@@ -2648,12 +2686,13 @@
     """Returns a dict describing specific instance_type"""
     session = get_session()
     inst_type = session.query(models.InstanceTypes).\
+                    options(joinedload('extra_specs')).\
                     filter_by(name=name).\
                     first()
     if not inst_type:
         raise exception.InstanceTypeNotFoundByName(instance_type_name=name)
     else:
-        return dict(inst_type)
+        return _dict_with_extra_specs(inst_type)
 
 
 @require_context
@@ -2661,12 +2700,13 @@
     """Returns a dict describing specific flavor_id"""
     session = get_session()
     inst_type = session.query(models.InstanceTypes).\
+                                    options(joinedload('extra_specs')).\
                                     filter_by(flavorid=int(id)).\
                                     first()
     if not inst_type:
         raise exception.FlavorNotFound(flavor_id=id)
     else:
-        return dict(inst_type)
+        return _dict_with_extra_specs(inst_type)
 
 
 @require_admin_context
@@ -2834,6 +2874,9 @@
     return metadata
 
 
+####################
+
+
 @require_admin_context
 def agent_build_create(context, values):
     agent_build_ref = models.AgentBuild()
@@ -2883,3 +2926,70 @@
                    first()
         agent_build_ref.update(values)
         agent_build_ref.save(session=session)
+
+
+####################
+
+
+@require_context
+def instance_type_extra_specs_get(context, instance_type_id):
+    session = get_session()
+
+    spec_results = session.query(models.InstanceTypeExtraSpecs).\
+                    filter_by(instance_type_id=instance_type_id).\
+                    filter_by(deleted=False).\
+                    all()
+
+    spec_dict = {}
+    for i in spec_results:
+        spec_dict[i['key']] = i['value']
+    return spec_dict
+
+
+@require_context
+def instance_type_extra_specs_delete(context, instance_type_id, key):
+    session = get_session()
+    session.query(models.InstanceTypeExtraSpecs).\
+        filter_by(instance_type_id=instance_type_id).\
+        filter_by(key=key).\
+        filter_by(deleted=False).\
+        update({'deleted': True,
+                'deleted_at': utils.utcnow(),
+                'updated_at': literal_column('updated_at')})
+
+
+@require_context
+def instance_type_extra_specs_get_item(context, instance_type_id, key):
+    session = get_session()
+
+    sppec_result = session.query(models.InstanceTypeExtraSpecs).\
+                    filter_by(instance_type_id=instance_type_id).\
+                    filter_by(key=key).\
+                    filter_by(deleted=False).\
+                    first()
+
+    if not spec_result:
+        raise exception.\
+           InstanceTypeExtraSpecsNotFound(extra_specs_key=key,
+                                        instance_type_id=instance_type_id)
+    return spec_result
+
+
+@require_context
+def instance_type_extra_specs_update_or_create(context, instance_type_id,
+                                               specs):
+    session = get_session()
+    spec_ref = None
+    for key, value in specs.iteritems():
+        try:
+            spec_ref = instance_type_extra_specs_get_item(context,
+                                                          instance_type_id,
+                                                          key,
+                                                          session)
+        except:
+            spec_ref = models.InstanceTypeExtraSpecs()
+        spec_ref.update({"key": key, "value": value,
+                         "instance_type_id": instance_type_id,
+                         "deleted": 0})
+        spec_ref.save(session=session)
+    return specs
 
=== added file 'nova/db/sqlalchemy/migrate_repo/versions/028_add_instance_type_extra_specs.py'
--- nova/db/sqlalchemy/migrate_repo/versions/028_add_instance_type_extra_specs.py	1970-01-01 00:00:00 +0000
+++ nova/db/sqlalchemy/migrate_repo/versions/028_add_instance_type_extra_specs.py	2011-06-27 00:06:02 +0000
@@ -0,0 +1,67 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright 2011 University of Southern California
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer
+from sqlalchemy import MetaData, String, Table
+from nova import log as logging
+
+meta = MetaData()
+
+# Just for the ForeignKey and column creation to succeed, these are not the
+# actual definitions of instances or services.
+instance_types = Table('instance_types', meta,
+        Column('id', Integer(), primary_key=True, nullable=False),
+        )
+
+#
+# New Tables
+#
+
+instance_type_extra_specs_table = Table('instance_type_extra_specs', meta,
+        Column('created_at', DateTime(timezone=False)),
+        Column('updated_at', DateTime(timezone=False)),
+        Column('deleted_at', DateTime(timezone=False)),
+        Column('deleted', Boolean(create_constraint=True, name=None)),
+        Column('id', Integer(), primary_key=True, nullable=False),
+        Column('instance_type_id',
+               Integer(),
+               ForeignKey('instance_types.id'),
+               nullable=False),
+        Column('key',
+               String(length=255, convert_unicode=False, assert_unicode=None,
+                      unicode_error=None, _warn_on_bytestring=False)),
+        Column('value',
+               String(length=255, convert_unicode=False, assert_unicode=None,
+                      unicode_error=None, _warn_on_bytestring=False)))
+
+
+def upgrade(migrate_engine):
+    # Upgrade operations go here. Don't create your own engine;
+    # bind migrate_engine to your metadata
+    meta.bind = migrate_engine
+    for table in (instance_type_extra_specs_table, ):
+        try:
+            table.create()
+        except Exception:
+            logging.info(repr(table))
+            logging.exception('Exception while creating table')
+            raise
+
+
+def downgrade(migrate_engine):
+    # Operations to reverse the above upgrade go here.
+    for table in (instance_type_extra_specs_table, ):
+        table.drop()
 
=== modified file 'nova/db/sqlalchemy/models.py'
--- nova/db/sqlalchemy/models.py	2011-06-24 12:01:51 +0000
+++ nova/db/sqlalchemy/models.py	2011-06-27 00:06:02 +0000
@@ -716,6 +716,21 @@
                                 'InstanceMetadata.deleted == False)')
 
 
+class InstanceTypeExtraSpecs(BASE, NovaBase):
+    """Represents additional specs as key/value pairs for an instance_type"""
+    __tablename__ = 'instance_type_extra_specs'
+    id = Column(Integer, primary_key=True)
+    key = Column(String(255))
+    value = Column(String(255))
+    instance_type_id = Column(Integer, ForeignKey('instance_types.id'),
+                              nullable=False)
+    instance_type = relationship(InstanceTypes, backref="extra_specs",
+                 foreign_keys=instance_type_id,
+                 primaryjoin='and_('
+                 'InstanceTypeExtraSpecs.instance_type_id == InstanceTypes.id,'
+                 'InstanceTypeExtraSpecs.deleted == False)')
+
+
 class Zone(BASE, NovaBase):
     """Represents a child zone of this zone."""
     __tablename__ = 'zones'
@@ -750,7 +765,7 @@
               Network, SecurityGroup, SecurityGroupIngressRule,
               SecurityGroupInstanceAssociation, AuthToken, User,
               Project, Certificate, ConsolePool, Console, Zone,
-              AgentBuild, InstanceMetadata, Migration)
+              AgentBuild, InstanceMetadata, InstanceTypeExtraSpecs, Migration)
     engine = create_engine(FLAGS.sql_connection, echo=False)
     for model in models:
         model.metadata.create_all(engine)
 
=== modified file 'nova/exception.py'
--- nova/exception.py	2011-06-24 12:01:51 +0000
+++ nova/exception.py	2011-06-27 00:06:02 +0000
@@ -504,6 +504,11 @@
                 "key %(metadata_key)s.")
 
 
+class InstanceTypeExtraSpecsNotFound(NotFound):
+    message = _("Instance Type %(instance_type_id)s has no extra specs with "
+                "key %(extra_specs_key)s.")
+
+
 class LDAPObjectNotFound(NotFound):
     message = _("LDAP object could not be found")
 
 
=== modified file 'nova/scheduler/host_filter.py'
--- nova/scheduler/host_filter.py	2011-06-03 20:32:42 +0000
+++ nova/scheduler/host_filter.py	2011-06-27 00:06:02 +0000
@@ -93,6 +93,26 @@
         """Use instance_type to filter hosts."""
         return (self._full_name(), instance_type)
 
+    def _satisfies_extra_specs(self, capabilities, instance_type):
+        """Check that the capabilities provided by the compute service
+        satisfy the extra specs associated with the instance type"""
+
+        if 'extra_specs' not in instance_type:
+            return True
+
+        # Note(lorinh): For now, we are just checking exact matching on the
+        # values. Later on, we  want to handle numerical
+        # values so we can represent things like number of GPU cards
+
+        try:
+            for key, value in instance_type['extra_specs'].iteritems():
+                if capabilities[key] != value:
+                    return False
+        except KeyError:
+            return False
+
+        return True
+
     def filter_hosts(self, zone_manager, query):
         """Return a list of hosts that can create instance_type."""
         instance_type = query
@@ -103,7 +123,11 @@
             disk_bytes = capabilities['disk_available']
             spec_ram = instance_type['memory_mb']
             spec_disk = instance_type['local_gb']
-            if host_ram_mb >= spec_ram and disk_bytes >= spec_disk:
+            extra_specs = instance_type['extra_specs']
+
+            if host_ram_mb >= spec_ram and \
+               disk_bytes >= spec_disk and \
+               self._satisfies_extra_specs(capabilities, instance_type):
                 selected_hosts.append((host, capabilities))
         return selected_hosts
 
 
=== added file 'nova/tests/api/openstack/extensions/test_flavors_extra_specs.py'
--- nova/tests/api/openstack/extensions/test_flavors_extra_specs.py	1970-01-01 00:00:00 +0000
+++ nova/tests/api/openstack/extensions/test_flavors_extra_specs.py	2011-06-27 00:06:02 +0000
@@ -0,0 +1,198 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright 2011 University of Southern California
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import json
+import stubout
+import unittest
+import webob
+import os.path
+
+
+from nova import flags
+from nova.api import openstack
+from nova.api.openstack import auth
+from nova.api.openstack import extensions
+from nova.tests.api.openstack import fakes
+import nova.wsgi
+
+FLAGS = flags.FLAGS
+
+
+def return_create_flavor_extra_specs(context, flavor_id, extra_specs):
+    return stub_flavor_extra_specs()
+
+
+def return_flavor_extra_specs(context, flavor_id):
+    return stub_flavor_extra_specs()
+
+
+def return_flavor_extra_specs(context, flavor_id):
+    return stub_flavor_extra_specs()
+
+
+def return_empty_flavor_extra_specs(context, flavor_id):
+    return {}
+
+
+def delete_flavor_extra_specs(context, flavor_id, key):
+    pass
+
+
+def stub_flavor_extra_specs():
+    specs = {
+            "key1": "value1",
+            "key2": "value2",
+            "key3": "value3",
+            "key4": "value4",
+            "key5": "value5"}
+    return specs
+
+
+class FlavorsExtraSpecsTest(unittest.TestCase):
+
+    def setUp(self):
+        super(FlavorsExtraSpecsTest, self).setUp()
+        FLAGS.osapi_extensions_path = os.path.join(os.path.dirname(__file__),
+                                                    "extensions")
+        self.stubs = stubout.StubOutForTesting()
+        fakes.FakeAuthManager.auth_data = {}
+        fakes.FakeAuthDatabase.data = {}
+        fakes.stub_out_auth(self.stubs)
+        fakes.stub_out_key_pair_funcs(self.stubs)
+        self.mware = auth.AuthMiddleware(
+                      extensions.ExtensionMiddleware(
+                      openstack.APIRouterV11()))
+
+    def tearDown(self):
+        self.stubs.UnsetAll()
+        super(FlavorsExtraSpecsTest, self).tearDown()
+
+    def test_index(self):
+        self.stubs.Set(nova.db.api, 'instance_type_extra_specs_get',
+                       return_flavor_extra_specs)
+        request = webob.Request.blank('/flavors/1/os-extra_specs')
+        res = request.get_response(self.mware)
+        self.assertEqual(200, res.status_int)
+        res_dict = json.loads(res.body)
+        self.assertEqual('application/json', res.headers['Content-Type'])
+        self.assertEqual('value1', res_dict['extra_specs']['key1'])
+
+    def test_index_no_data(self):
+        self.stubs.Set(nova.db.api, 'instance_type_extra_specs_get',
+                       return_empty_flavor_extra_specs)
+        req = webob.Request.blank('/flavors/1/os-extra_specs')
+        res = req.get_response(self.mware)
+        res_dict = json.loads(res.body)
+        self.assertEqual(200, res.status_int)
+        self.assertEqual('application/json', res.headers['Content-Type'])
+        self.assertEqual(0, len(res_dict['extra_specs']))
+
+    def test_show(self):
+        self.stubs.Set(nova.db.api, 'instance_type_extra_specs_get',
+                       return_flavor_extra_specs)
+        req = webob.Request.blank('/flavors/1/os-extra_specs/key5')
+        res = req.get_response(self.mware)
+        self.assertEqual(200, res.status_int)
+        res_dict = json.loads(res.body)
+        self.assertEqual('application/json', res.headers['Content-Type'])
+        self.assertEqual('value5', res_dict['key5'])
+
+    def test_show_spec_not_found(self):
+        self.stubs.Set(nova.db.api, 'instance_type_extra_specs_get',
+                       return_empty_flavor_extra_specs)
+        req = webob.Request.blank('/flavors/1/os-extra_specs/key6')
+        res = req.get_response(self.mware)
+        res_dict = json.loads(res.body)
+        self.assertEqual(404, res.status_int)
+
+    def test_delete(self):
+        self.stubs.Set(nova.db.api, 'instance_type_extra_specs_delete',
+                       delete_flavor_extra_specs)
+        req = webob.Request.blank('/flavors/1/os-extra_specs/key5')
+        req.method = 'DELETE'
+        res = req.get_response(self.mware)
+        self.assertEqual(200, res.status_int)
+
+    def test_create(self):
+        self.stubs.Set(nova.db.api,
+                       'instance_type_extra_specs_update_or_create',
+                       return_create_flavor_extra_specs)
+        req = webob.Request.blank('/flavors/1/os-extra_specs')
+        req.method = 'POST'
+        req.body = '{"extra_specs": {"key1": "value1"}}'
+        req.headers["content-type"] = "application/json"
+        res = req.get_response(self.mware)
+        res_dict = json.loads(res.body)
+        self.assertEqual(200, res.status_int)
+        self.assertEqual('application/json', res.headers['Content-Type'])
+        self.assertEqual('value1', res_dict['extra_specs']['key1'])
+
+    def test_create_empty_body(self):
+        self.stubs.Set(nova.db.api,
+                       'instance_type_extra_specs_update_or_create',
+                       return_create_flavor_extra_specs)
+        req = webob.Request.blank('/flavors/1/os-extra_specs')
+        req.method = 'POST'
+        req.headers["content-type"] = "application/json"
+        res = req.get_response(self.mware)
+        self.assertEqual(400, res.status_int)
+
+    def test_update_item(self):
+        self.stubs.Set(nova.db.api,
+                       'instance_type_extra_specs_update_or_create',
+                       return_create_flavor_extra_specs)
+        req = webob.Request.blank('/flavors/1/os-extra_specs/key1')
+        req.method = 'PUT'
+        req.body = '{"key1": "value1"}'
+        req.headers["content-type"] = "application/json"
+        res = req.get_response(self.mware)
+        self.assertEqual(200, res.status_int)
+        self.assertEqual('application/json', res.headers['Content-Type'])
+        res_dict = json.loads(res.body)
+        self.assertEqual('value1', res_dict['key1'])
+
+    def test_update_item_empty_body(self):
+        self.stubs.Set(nova.db.api,
+                       'instance_type_extra_specs_update_or_create',
+                       return_create_flavor_extra_specs)
+        req = webob.Request.blank('/flavors/1/os-extra_specs/key1')
+        req.method = 'PUT'
+        req.headers["content-type"] = "application/json"
+        res = req.get_response(self.mware)
+        self.assertEqual(400, res.status_int)
+
+    def test_update_item_too_many_keys(self):
+        self.stubs.Set(nova.db.api,
+                       'instance_type_extra_specs_update_or_create',
+                       return_create_flavor_extra_specs)
+        req = webob.Request.blank('/flavors/1/os-extra_specs/key1')
+        req.method = 'PUT'
+        req.body = '{"key1": "value1", "key2": "value2"}'
+        req.headers["content-type"] = "application/json"
+        res = req.get_response(self.mware)
+        self.assertEqual(400, res.status_int)
+
+    def test_update_item_body_uri_mismatch(self):
+        self.stubs.Set(nova.db.api,
+                       'instance_type_extra_specs_update_or_create',
+                       return_create_flavor_extra_specs)
+        req = webob.Request.blank('/flavors/1/os-extra_specs/bad')
+        req.method = 'PUT'
+        req.body = '{"key1": "value1"}'
+        req.headers["content-type"] = "application/json"
+        res = req.get_response(self.mware)
+        self.assertEqual(400, res.status_int)
 
=== modified file 'nova/tests/scheduler/test_host_filter.py'
--- nova/tests/scheduler/test_host_filter.py	2011-06-14 01:14:26 +0000
+++ nova/tests/scheduler/test_host_filter.py	2011-06-27 00:06:02 +0000
@@ -67,7 +67,18 @@
                 flavorid=1,
                 swap=500,
                 rxtx_quota=30000,
-                rxtx_cap=200)
+                rxtx_cap=200,
+                extra_specs={})
+        self.gpu_instance_type = dict(name='tiny.gpu',
+                memory_mb=50,
+                vcpus=10,
+                local_gb=500,
+                flavorid=2,
+                swap=500,
+                rxtx_quota=30000,
+                rxtx_cap=200,
+                extra_specs={'xpu_arch': 'fermi',
+                             'xpu_info': 'Tesla 2050'})
 
         self.zone_manager = FakeZoneManager()
         states = {}
@@ -75,6 +86,18 @@
             states['host%02d' % (x + 1)] = {'compute': self._host_caps(x)}
         self.zone_manager.service_states = states
 
+        # Add some extra capabilities to some hosts
+        host07 = self.zone_manager.service_states['host07']['compute']
+        host07['xpu_arch'] = 'fermi'
+        host07['xpu_info'] = 'Tesla 2050'
+
+        host08 = self.zone_manager.service_states['host08']['compute']
+        host08['xpu_arch'] = 'radeon'
+
+        host09 = self.zone_manager.service_states['host09']['compute']
+        host09['xpu_arch'] = 'fermi'
+        host09['xpu_info'] = 'Tesla 2150'
+
     def tearDown(self):
         FLAGS.default_host_filter = self.old_flag
 
@@ -116,6 +139,17 @@
         self.assertEquals('host05', just_hosts[0])
         self.assertEquals('host10', just_hosts[5])
 
+    def test_instance_type_filter_extra_specs(self):
+        hf = host_filter.InstanceTypeFilter()
+        # filter all hosts that can support 50 ram and 500 disk
+        name, cooked = hf.instance_type_to_filter(self.gpu_instance_type)
+        self.assertEquals('nova.scheduler.host_filter.InstanceTypeFilter',
+                          name)
+        hosts = hf.filter_hosts(self.zone_manager, cooked)
+        self.assertEquals(1, len(hosts))
+        just_hosts = [host for host, caps in hosts]
+        self.assertEquals('host07', just_hosts[0])
+
     def test_json_filter(self):
         hf = host_filter.JsonFilter()
         # filter all hosts that can support 50 ram and 500 disk
 
=== modified file 'nova/tests/test_host_filter.py'
--- nova/tests/test_host_filter.py	2011-06-02 19:08:19 +0000
+++ nova/tests/test_host_filter.py	2011-06-27 00:06:02 +0000
@@ -67,7 +67,8 @@
                 flavorid=1,
                 swap=500,
                 rxtx_quota=30000,
-                rxtx_cap=200)
+                rxtx_cap=200,
+                extra_specs={})
 
         self.zone_manager = FakeZoneManager()
         states = {}
 
=== added file 'nova/tests/test_instance_types_extra_specs.py'
--- nova/tests/test_instance_types_extra_specs.py	1970-01-01 00:00:00 +0000
+++ nova/tests/test_instance_types_extra_specs.py	2011-06-27 00:06:02 +0000
@@ -0,0 +1,165 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright 2011 University of Southern California
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+"""
+Unit Tests for instance types extra specs code
+"""
+
+from nova import context
+from nova import db
+from nova import test
+from nova.db.sqlalchemy.session import get_session
+from nova.db.sqlalchemy import models
+
+
+class InstanceTypeExtraSpecsTestCase(test.TestCase):
+
+    def setUp(self):
+        super(InstanceTypeExtraSpecsTestCase, self).setUp()
+        self.context = context.get_admin_context()
+        values = dict(name="cg1.4xlarge",
+                      memory_mb=22000,
+                      vcpus=8,
+                      local_gb=1690,
+                      flavorid=105)
+        specs = dict(cpu_arch="x86_64",
+                        cpu_model="Nehalem",
+                        xpu_arch="fermi",
+                        xpus=2,
+                        xpu_model="Tesla 2050")
+        values['extra_specs'] = specs
+        ref = db.api.instance_type_create(self.context,
+                                          values)
+        self.instance_type_id = ref.id
+
+    def tearDown(self):
+        # Remove the instance type from the database
+        db.api.instance_type_purge(context.get_admin_context(), "cg1.4xlarge")
+        super(InstanceTypeExtraSpecsTestCase, self).tearDown()
+
+    def test_instance_type_specs_get(self):
+        expected_specs = dict(cpu_arch="x86_64",
+                                 cpu_model="Nehalem",
+                                 xpu_arch="fermi",
+                                 xpus="2",
+                                 xpu_model="Tesla 2050")
+        actual_specs = db.api.instance_type_extra_specs_get(
+                              context.get_admin_context(),
+                              self.instance_type_id)
+        self.assertEquals(expected_specs, actual_specs)
+
+    def test_instance_type_extra_specs_delete(self):
+        expected_specs = dict(cpu_arch="x86_64",
+                                 cpu_model="Nehalem",
+                                 xpu_arch="fermi",
+                                 xpus="2")
+        db.api.instance_type_extra_specs_delete(context.get_admin_context(),
+                                      self.instance_type_id,
+                                      "xpu_model")
+        actual_specs = db.api.instance_type_extra_specs_get(
+                              context.get_admin_context(),
+                              self.instance_type_id)
+        self.assertEquals(expected_specs, actual_specs)
+
+    def test_instance_type_extra_specs_update(self):
+        expected_specs = dict(cpu_arch="x86_64",
+                                 cpu_model="Sandy Bridge",
+                                 xpu_arch="fermi",
+                                 xpus="2",
+                                 xpu_model="Tesla 2050")
+        db.api.instance_type_extra_specs_update_or_create(
+                              context.get_admin_context(),
+                              self.instance_type_id,
+                              dict(cpu_model="Sandy Bridge"))
+        actual_specs = db.api.instance_type_extra_specs_get(
+                              context.get_admin_context(),
+                              self.instance_type_id)
+        self.assertEquals(expected_specs, actual_specs)
+
+    def test_instance_type_extra_specs_create(self):
+        expected_specs = dict(cpu_arch="x86_64",
+                                 cpu_model="Nehalem",
+                                 xpu_arch="fermi",
+                                 xpus="2",
+                                 xpu_model="Tesla 2050",
+                                 net_arch="ethernet",
+                                 net_mbps="10000")
+        db.api.instance_type_extra_specs_update_or_create(
+                              context.get_admin_context(),
+                              self.instance_type_id,
+                              dict(net_arch="ethernet",
+                                   net_mbps=10000))
+        actual_specs = db.api.instance_type_extra_specs_get(
+                              context.get_admin_context(),
+                              self.instance_type_id)
+        self.assertEquals(expected_specs, actual_specs)
+
+    def test_instance_type_get_by_id_with_extra_specs(self):
+        instance_type = db.api.instance_type_get_by_id(
+                            context.get_admin_context(),
+                            self.instance_type_id)
+        self.assertEquals(instance_type['extra_specs'],
+                          dict(cpu_arch="x86_64",
+                               cpu_model="Nehalem",
+                               xpu_arch="fermi",
+                               xpus="2",
+                               xpu_model="Tesla 2050"))
+        instance_type = db.api.instance_type_get_by_id(
+                            context.get_admin_context(),
+                            5)
+        self.assertEquals(instance_type['extra_specs'], {})
+
+    def test_instance_type_get_by_name_with_extra_specs(self):
+        instance_type = db.api.instance_type_get_by_name(
+                            context.get_admin_context(),
+                            "cg1.4xlarge")
+        self.assertEquals(instance_type['extra_specs'],
+                          dict(cpu_arch="x86_64",
+                               cpu_model="Nehalem",
+                               xpu_arch="fermi",
+                               xpus="2",
+                               xpu_model="Tesla 2050"))
+
+        instance_type = db.api.instance_type_get_by_name(
+                            context.get_admin_context(),
+                            "m1.small")
+        self.assertEquals(instance_type['extra_specs'], {})
+
+    def test_instance_type_get_by_id_with_extra_specs(self):
+        instance_type = db.api.instance_type_get_by_flavor_id(
+                            context.get_admin_context(),
+                            105)
+        self.assertEquals(instance_type['extra_specs'],
+                          dict(cpu_arch="x86_64",
+                               cpu_model="Nehalem",
+                               xpu_arch="fermi",
+                               xpus="2",
+                               xpu_model="Tesla 2050"))
+
+        instance_type = db.api.instance_type_get_by_flavor_id(
+                            context.get_admin_context(),
+                            2)
+        self.assertEquals(instance_type['extra_specs'], {})
+
+    def test_instance_type_get_all(self):
+        specs = dict(cpu_arch="x86_64",
+                        cpu_model="Nehalem",
+                        xpu_arch="fermi",
+                        xpus='2',
+                        xpu_model="Tesla 2050")
+
+        types = db.api.instance_type_get_all(context.get_admin_context())
+
+        self.assertEquals(types['cg1.4xlarge']['extra_specs'], specs)
+        self.assertEquals(types['m1.small']['extra_specs'], {})
 
=== modified file 'tools/pip-requires'
--- tools/pip-requires	2011-06-25 19:38:07 +0000
+++ tools/pip-requires	2011-06-27 00:06:02 +0000
@@ -1,5 +1,5 @@
 SQLAlchemy==0.6.3
-pep8==0.5.0
+pep8==0.6.1
 pylint==0.19
 Cheetah==2.4.4
 M2Crypto==0.20.2