Merge lp:~0x44/nova/config-drive into lp:~hudson-openstack/nova/trunk

[Christopher MacGown]

Implements first-pass of config-drive that adds a vfat format drive to a vm when config_drive is True (or an image id).

[Dan Prince (dan-prince)]

Hi Chris,

Is there a reason that this feature is exposed via the OSAPI? Why not just enable or disable it via a flag via the config file? If we do need to expose it via the API then perhaps an extension would be better?

If guests are using this to pull in configuration metadata then it seems like you would just always want it on.

---

Also, You have a conflict in nova/api/openstack/create_instance_helper.py.

review: Needs Information

[Dan Prince (dan-prince)]

Your using FLAGS.default_local_format but it doesn't appear to be defined anywhere.

review: Needs Fixing

[Matt Dietz (cerberus)]

184 + instance_config_drive = None # pre-existing instances don't have one.
185 + migrate_engine.execute(instances.update()\
186 + .where(instances.c.id == row[0])\
187 + .values(config_drive=instance_config_drive))

It's likely I don't know something about sqlalchemy, but couldn't you just set the new column default value to be NULL and accomplish the same thing?

review: Needs Fixing

[Christopher MacGown (0x44)]

Dan,

This is exposed in the OSAPI to support one of the use cases requested in the blueprint discussion on the etherpad. Having the config-drive always be a VFAT formatted drive didn't work for the people at Canonical, so they requested being able to pass in a config-drive image reference on instance create that would populate a local volume with the contents of that image in lieu of the general VFAT formatted drive.

I'm ambivalent on the idea of exposing it with a FLAG, but if you really prefer it to be, it's a trivial change that wouldn't take long to do.

[Matt Dietz (cerberus)]

Thanks for the changes, Chris.

review: Approve

[Dan Prince (dan-prince)]

I guess my thought is that 'config_drive' isn't part of the core OSAPI. At the very least it should be documented as an extension in the code base. It seems like some service providers might like to make use of config drives "under the covers" but not allow end users to provide custom images for them.

---

Also, Can you bump the DB migration number so it doesn't conflict with trunk?

review: Needs Fixing

[lp:~0x44/nova/config-drive]

1466. By Ed Leafe on 2011-08-19

Removes the incorrect hard-coded filter path.

1467. By Vish Ishaya on 2011-08-19

Next round of prep for keystone integration.

 * adds middleware for authenticating ec2 signature with keystone
 * adds middleware for converting keystone response into request context
 * gives examples of alternative pipelines for keystone integration

Next steps:
 * provide default config with no keystone integration (perhaps setting every context to admin?)
 * write authmanager to keystone conversion code
 * add api extension to create and destroy access/secret keys
 * deprecate authmanager
 * rename project to tenant

1468. By Anthony Young on 2011-08-19

Assorted fixes to os-floating-ips to make it play nicely with an in-progress novaclient implementation, as well as some changes to make it more consistent with other os rest apis. Changes include:

* switch associate/disassociate to PUTs. Previously, it was doing create calls with one-off parameter resources.
* allow graceful handling when there are no floating ips for a tenant
* allow graceful handling when disassociating an already disassociated address

[Erica Windisch (ewindisch)]

This config_drive feature looks just like the OVF environment specification.

In this specification, you provide a transport over which an environment file is provided. The specification defines an 'iso' transport which is a virtual CDROM drive with an ISO9660 filesystem, but we could make this a FAT-family filesystem instead.

In the filesystem, for OVF compatibility, we would create an XML file containing details about the execution environment and could pass metadata.

Whatever is implemented here must be considered for interoperability and forward-compatibility. OVF may not be perfect and XML might not be the most popular container, but it has passed through a standardization body (DMTF) and already exists.

Reference: http://www.dmtf.org/sites/default/files/standards/documents/DSP0243_1.1.0.pdf

review: Needs Fixing

[Christopher MacGown (0x44)]

Eric: The original plan was that it would be an ISO9660 filesystem but the people who documented/requested the blueprint requested a read-write filesystem that you cannot obtain with ISO9660. You can follow the entire discussion in the blueprint linked in the description.

Dan: I'll move the osapi stuff into an extension and look at adding another flag to turn off the publicly exposed feature.

[Erica Windisch (ewindisch)]

I'm not opposed to FAT, OVF can work with FAT just as well as ISO9660.
Although, I must admit to prefer the simplicity of mkisofs to mkfs.vfat as
with FAT there is a need to mount & manipulate the filesystem. Also, there
is the option of UDF as well and the fact that FAT lacks access controls
that are available with, say, RockRidge.

Despite the filesystem, which I consider a lesser concern, there is the
matter of the format of the file itself. The proposed patch implements JSON
encapulation of metadata. This isn't necessarily bad, but I maintain it is
important to consider if implementing the OVF environment XML format might
not be the better choice here. I'm open to criticisms regarding this format,
if any, and suggestions of alternatives. On the surface, it seems to be the
right choice and a fairly minor change.

On Aug 20, 2011 10:34 AM, "Christopher MacGown" <email address hidden> wrote:
Eric: The original plan was that it would be an ISO9660 filesystem but the
people who documented/requested the blueprint requested a read-write
filesystem that you cannot obtain with ISO9660. You can follow the entire
discussion in the blueprint linked in the description.

Dan: I'll move the osapi stuff into an extension and look at adding another
flag to turn off the publicly exposed feature.

--
https://code.launchpad.net/~0x44/nova/config-drive/+merge/71429
You are reviewing the proposed m...

[lp:~0x44/nova/config-drive]

1469. By Tushar Patil on 2011-08-21

Added OS APIs to associate/disassociate security groups to/from instances.

I will add views to return list of security groups associated with the servers later after this branch is merged into trunk. The reason I will do this later is because my previous merge proposal (https://code.launchpad.net/~tpatil/nova/add-options-network-create-os-apis/+merge/68292) is dependent on this work. In this merge proposal I have added a new extension which still uses default OS v1.1 controllers and views, but I am going to override views in this extension to send extra information like security groups.

1470. By Sandy Walsh on 2011-08-22

Fixes utils.to_primitive (again) to handle modules, builtins and whatever other crap might be hiding in an object.

1471. By Alex Meade on 2011-08-22

Adds accessIPv4 and accessIPv6 to servers requests and responses as per the current spec.

[Vish Ishaya (vishvananda)]

I'm ok with adding this using the json format and putting in a bug for supporting the xml ovf format as well.

review: Approve

[Erica Windisch (ewindisch)]

Vish, while I'd honestly prefer one or the other, not both, it doesn't matter significantly. Overall, I'd prefer to establish a blueprint for something more flexible and revamping the entire injection mechanism.

Performing further code review, I think we also need the following:

=== modified file 'nova/virt/xenapi/vm_utils.py'
--- nova/virt/xenapi/vm_utils.py 2011-08-19 15:50:48 +0000
+++ nova/virt/xenapi/vm_utils.py 2011-08-22 19:43:08 +0000
@@ -742,7 +742,7 @@
         # everything
         mount_required = False
         key, net, metadata = _prepare_injectables(instance, network_info)
- mount_required = key or net
+ mount_required = key or net or metadata
         if not mount_required:
             return

[Erica Windisch (ewindisch)]

The metadata file should be written to, not appended to. Append makes more sense for SSH keys than it does for JSON.

=== modified file 'nova/virt/disk.py'
--- nova/virt/disk.py 2011-08-12 21:23:10 +0000
+++ nova/virt/disk.py 2011-08-22 19:56:34 +0000
@@ -228,7 +228,7 @@
     metadata_path = os.path.join(fs, "meta.js")
     metadata = dict([(m.key, m.value) for m in metadata])

- utils.execute('sudo', 'tee', '-a', metadata_path,
+ utils.execute('sudo', 'tee', metadata_path,
                   process_input=json.dumps(metadata))

[lp:~0x44/nova/config-drive]

1472. By William Wolf on 2011-08-22

implemented tenant ids to be included in request uris.

[Erica Windisch (ewindisch)]

Metadata wasn't written on XenServer unless using flat_injected networking.

=== modified file 'nova/virt/xenapi/vmops.py'
--- nova/virt/xenapi/vmops.py 2011-08-17 03:15:54 +0000
+++ nova/virt/xenapi/vmops.py 2011-08-22 20:13:04 +0000
@@ -240,7 +240,8 @@
             vdis)

         # Alter the image before VM start for, e.g. network injection
- if FLAGS.flat_injected:
+ # also alter if there is metadata
+ if FLAGS.flat_injected or instance['metadata']:
             VMHelper.preconfigure_instance(self._session, instance,
                                            first_vdi_ref, network_info)

[Christopher MacGown (0x44)]

Eric: There's already a blueprint for a way of facilitating two-way communication between the host and guest that would mostly end up superseding config-drive except as a place to write personality.

[Christopher MacGown (0x44)]

I'll add your fixes in my next push.

[lp:~0x44/nova/config-drive]

1473. By Alex Meade on 2011-08-22

The FixedIpCommandsTestCase in test_nova_manage previously accessed the database. This branch stubs out the database for these tests, lowering their run time from 104 secs -> .02 secs total.

I have verified the tested functionality is still being tested.

1474. By Soren Hansen on 2011-08-23

Move documentation from nova.virt.fake into nova.virt.driver.

1475. By Alex Meade on 2011-08-23

Fixes bug 831627 where nova-manage does not exit when given a non-existent network address

1476. By Tushar Patil on 2011-08-23

Our goal is to add optional parameter to the Create server OS 1.0 and 1.1 API to achieve following objectives:-

1) Specify number and order of networks to the create server API.

In the current implementation every instance you launch for a project having 3 networks assigned to it will always have 3 vnics. In this case it is not possible to have one instance with 2 vnics ,another with 1 vnic and so on. This is not flexible enough and the network resources are not used effectively.

2) Specify fixed IP address to the vnic at the boot time. When you launch a server, you can specify the fixed IP address you want to be assigned to the vnic from a particular network. If this fixed IP address is already in use, it will give exception.

Example of Server Create API request XML:
<?xml version="1.0" encoding="UTF-8"?>

<server xmlns="http://docs.nttpflab.com/servers/api/v1.0"
        name="new-server-test" imageId="1" flavorId="1">
  <metadata>
    <meta key="My Server Name">Apache1</meta>
  </metadata>
  <personality>
    <file path="/etc/banner.txt">
        ICAgICAgDQoiQSBjbG91ZCBkb2VzIG5vdCBrbm93IHdoeSBp
    </file>
  </personality>
  <networks>
      <network uuid="6622436e-5289-460f-8479-e4dcc63f16c5" fixed_ip="10.0.0.3">
      <network uuid="d97efefc-e071-4174-b6dd-b33af0a37706" fixed_ip="10.0.1.3">
  </networks>
</server>

3) Networks is an optional parameter, so if you don't provide any networks to the server Create API, it will behave exactly the same as of today.

This feature is supported to all of the network models.

1477. By Christopher MacGown <email address hidden> on 2011-08-23

Merged trunk

1478. By Christopher MacGown <email address hidden> on 2011-08-23

Moved migration and fixed tests from upstream

[Christopher MacGown (0x44)]

After discussion with RCB and the API discussion on the mailing list, I'm going to document config-drive as a core feature of the 1.1 API as it makes another core feature of the 1.1 API possible on non-XenServer hypervisors. I've added the other fixes suggested above.

[Vish Ishaya (vishvananda)]

relgtm

review: Approve

=== modified file 'Authors'
--- Authors	2011-08-17 07:41:17 +0000
+++ Authors	2011-08-23 05:22:32 +0000
@@ -18,6 +18,7 @@
 Chmouel Boudjnah <[e-mail]>
 Chris Behrens <[e-mail]>
 Christian Berendt <[e-mail]>
+Christopher MacGown <[e-mail]>
 Chuck Short <[e-mail]>
 Cory Wright <[e-mail]>
 Dan Prince <[e-mail]>

=== modified file 'nova/api/openstack/create_instance_helper.py'
--- nova/api/openstack/create_instance_helper.py	2011-08-22 23:35:09 +0000
+++ nova/api/openstack/create_instance_helper.py	2011-08-23 05:22:32 +0000
@@ -1,4 +1,5 @@
 # Copyright 2011 OpenStack LLC.
+# Copyright 2011 Piston Cloud Computing, Inc.
 # All Rights Reserved.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License"); you may
@@ -106,6 +107,7 @@
             raise exc.HTTPBadRequest(explanation=msg)
 
         personality = server_dict.get('personality')
+        config_drive = server_dict.get('config_drive')
 
         injected_files = []
         if personality:
@@ -159,6 +161,7 @@
             extra_values = {
                 'instance_type': inst_type,
                 'image_ref': image_href,
+                'config_drive': config_drive,
                 'password': password}
 
             return (extra_values,
@@ -183,7 +186,8 @@
                                   requested_networks=requested_networks,
                                   security_group=sg_names,
                                   user_data=user_data,
-                                  availability_zone=availability_zone))
+                                  availability_zone=availability_zone,
+                                  config_drive=config_drive,))
         except quota.QuotaError as error:
             self._handle_quota_error(error)
         except exception.ImageNotFound as error:

=== modified file 'nova/api/openstack/views/servers.py'
--- nova/api/openstack/views/servers.py	2011-08-22 12:28:12 +0000
+++ nova/api/openstack/views/servers.py	2011-08-23 05:22:32 +0000
@@ -1,6 +1,7 @@
 # vim: tabstop=4 shiftwidth=4 softtabstop=4
 
 # Copyright 2010-2011 OpenStack LLC.
+# Copyright 2011 Piston Cloud Computing, Inc.
 # All Rights Reserved.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License"); you may
@@ -187,6 +188,7 @@
     def _build_extra(self, response, inst):
         self._build_links(response, inst)
         response['uuid'] = inst['uuid']
+        self._build_config_drive(response, inst)
 
     def _build_links(self, response, inst):
         href = self.generate_href(inst["id"])
@@ -205,6 +207,9 @@
 
         response["links"] = links
 
+    def _build_config_drive(self, response, inst):
+        response['config_drive'] = inst.get('config_drive')
+
     def generate_href(self, server_id):
         """Create an url that refers to a specific server id."""
         return os.path.join(self.base_url, self.project_id,

=== modified file 'nova/compute/api.py'
--- nova/compute/api.py	2011-08-22 23:35:09 +0000
+++ nova/compute/api.py	2011-08-23 05:22:32 +0000
@@ -2,6 +2,7 @@
 
 # Copyright 2010 United States Government as represented by the
 # Administrator of the National Aeronautics and Space Administration.
+# Copyright 2011 Piston Cloud Computing, Inc.
 # All Rights Reserved.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License"); you may
@@ -164,7 +165,7 @@
                availability_zone=None, user_data=None, metadata=None,
                injected_files=None, admin_password=None, zone_blob=None,
                reservation_id=None, access_ip_v4=None, access_ip_v6=None,
-               requested_networks=None):
+               requested_networks=None, config_drive=None,):
         """Verify all the input parameters regardless of the provisioning
         strategy being performed."""
 
@@ -198,6 +199,11 @@
         (image_service, image_id) = nova.image.get_image_service(image_href)
         image = image_service.show(context, image_id)
 
+        config_drive_id = None
+        if config_drive and config_drive is not True:
+            # config_drive is volume id
+            config_drive, config_drive_id = None, config_drive
+
         os_type = None
         if 'properties' in image and 'os_type' in image['properties']:
             os_type = image['properties']['os_type']
@@ -225,6 +231,8 @@
             image_service.show(context, kernel_id)
         if ramdisk_id:
             image_service.show(context, ramdisk_id)
+        if config_drive_id:
+            image_service.show(context, config_drive_id)
 
         self.ensure_default_security_group(context)
 
@@ -243,6 +251,8 @@
             'image_ref': image_href,
             'kernel_id': kernel_id or '',
             'ramdisk_id': ramdisk_id or '',
+            'config_drive_id': config_drive_id or '',
+            'config_drive': config_drive or '',
             'state': 0,
             'state_description': 'scheduling',
             'user_id': context.user_id,
@@ -454,7 +464,7 @@
                injected_files=None, admin_password=None, zone_blob=None,
                reservation_id=None, block_device_mapping=None,
                access_ip_v4=None, access_ip_v6=None,
-               requested_networks=None):
+               requested_networks=None, config_drive=None):
         """Provision the instances by passing the whole request to
         the Scheduler for execution. Returns a Reservation ID
         related to the creation of all of these instances."""
@@ -471,7 +481,7 @@
                                availability_zone, user_data, metadata,
                                injected_files, admin_password, zone_blob,
                                reservation_id, access_ip_v4, access_ip_v6,
-                               requested_networks)
+                               requested_networks, config_drive)
 
         self._ask_scheduler_to_create_instance(context, base_options,
                                       instance_type, zone_blob,
@@ -491,7 +501,7 @@
                injected_files=None, admin_password=None, zone_blob=None,
                reservation_id=None, block_device_mapping=None,
                access_ip_v4=None, access_ip_v6=None,
-               requested_networks=None):
+               requested_networks=None, config_drive=None,):
         """
         Provision the instances by sending off a series of single
         instance requests to the Schedulers. This is fine for trival
@@ -516,7 +526,7 @@
                                availability_zone, user_data, metadata,
                                injected_files, admin_password, zone_blob,
                                reservation_id, access_ip_v4, access_ip_v6,
-                               requested_networks)
+                               requested_networks, config_drive)
 
         block_device_mapping = block_device_mapping or []
         instances = []

=== added file 'nova/db/sqlalchemy/migrate_repo/versions/041_add_config_drive_to_instances.py'
--- nova/db/sqlalchemy/migrate_repo/versions/041_add_config_drive_to_instances.py	1970-01-01 00:00:00 +0000
+++ nova/db/sqlalchemy/migrate_repo/versions/041_add_config_drive_to_instances.py	2011-08-23 05:22:32 +0000
@@ -0,0 +1,38 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+#
+# Copyright 2011 Piston Cloud Computing, Inc.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from sqlalchemy import Column, Integer, MetaData, String, Table
+
+from nova import utils
+
+
+meta = MetaData()
+
+instances = Table("instances", meta,
+        Column("id", Integer(), primary_key=True, nullable=False))
+
+# matches the size of an image_ref
+config_drive_column = Column("config_drive", String(255), nullable=True)
+
+
+def upgrade(migrate_engine):
+    meta.bind = migrate_engine
+    instances.create_column(config_drive_column)
+
+
+def downgrade(migrate_engine):
+    meta.bind = migrate_engine
+    instances.drop_column(config_drive_column)

=== modified file 'nova/db/sqlalchemy/models.py'
--- nova/db/sqlalchemy/models.py	2011-08-22 23:35:09 +0000
+++ nova/db/sqlalchemy/models.py	2011-08-23 05:22:32 +0000
@@ -2,6 +2,7 @@
 
 # Copyright 2010 United States Government as represented by the
 # Administrator of the National Aeronautics and Space Administration.
+# Copyright 2011 Piston Cloud Computing, Inc.
 # All Rights Reserved.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License"); you may
@@ -230,6 +231,7 @@
     uuid = Column(String(36))
 
     root_device_name = Column(String(255))
+    config_drive = Column(String(255))
 
     # User editable field meant to represent what ip should be used
     # to connect to the instance

=== modified file 'nova/tests/api/openstack/test_servers.py'
--- nova/tests/api/openstack/test_servers.py	2011-08-22 23:35:09 +0000
+++ nova/tests/api/openstack/test_servers.py	2011-08-23 05:22:32 +0000
@@ -1,6 +1,7 @@
 # vim: tabstop=4 shiftwidth=4 softtabstop=4
 
 # Copyright 2010-2011 OpenStack LLC.
+# Copyright 2011 Piston Cloud Computing, Inc.
 # All Rights Reserved.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License"); you may
@@ -233,7 +234,6 @@
 
 
 class ServersTest(test.TestCase):
-
     def setUp(self):
         self.maxDiff = None
         super(ServersTest, self).setUp()
@@ -265,6 +265,7 @@
         self.stubs.Set(nova.compute.API, "get_actions", fake_compute_api)
 
         self.webreq = common.webob_factory('/v1.0/servers')
+        self.config_drive = None
 
     def test_get_server_by_id(self):
         req = webob.Request.blank('/v1.0/servers/1')
@@ -379,6 +380,7 @@
                 "metadata": {
                     "seq": "1",
                 },
+                "config_drive": None,
                 "links": [
                     {
                         "rel": "self",
@@ -545,6 +547,7 @@
                 "metadata": {
                     "seq": "1",
                 },
+                "config_drive": None,
                 "links": [
                     {
                         "rel": "self",
@@ -638,6 +641,7 @@
                 "metadata": {
                     "seq": "1",
                 },
+                "config_drive": None,
                 "links": [
                     {
                         "rel": "self",
@@ -1399,6 +1403,7 @@
                     'image_ref': image_ref,
                     "created_at": datetime.datetime(2010, 10, 10, 12, 0, 0),
                     "updated_at": datetime.datetime(2010, 11, 11, 11, 0, 0),
+                    "config_drive": self.config_drive,
                    }
 
         def server_update(context, id, params):
@@ -1424,8 +1429,7 @@
         self.stubs.Set(nova.db.api, 'instance_create', instance_create)
         self.stubs.Set(nova.rpc, 'cast', fake_method)
         self.stubs.Set(nova.rpc, 'call', fake_method)
-        self.stubs.Set(nova.db.api, 'instance_update',
-            server_update)
+        self.stubs.Set(nova.db.api, 'instance_update', server_update)
         self.stubs.Set(nova.db.api, 'queue_get_for', queue_get_for)
         self.stubs.Set(nova.network.manager.VlanManager, 'allocate_fixed_ip',
             fake_method)
@@ -1768,6 +1772,129 @@
         res = req.get_response(fakes.wsgi_app())
         self.assertEqual(res.status_int, 400)
 
+    def test_create_instance_with_config_drive_v1_1(self):
+        self.config_drive = True
+        self._setup_for_create_instance()
+
+        image_href = 'http://localhost/v1.1/123/images/2'
+        flavor_ref = 'http://localhost/v1.1/123/flavors/3'
+        body = {
+            'server': {
+                'name': 'config_drive_test',
+                'imageRef': image_href,
+                'flavorRef': flavor_ref,
+                'metadata': {
+                    'hello': 'world',
+                    'open': 'stack',
+                },
+                'personality': {},
+                'config_drive': True,
+            },
+        }
+
+        req = webob.Request.blank('/v1.1/123/servers')
+        req.method = 'POST'
+        req.body = json.dumps(body)
+        req.headers["content-type"] = "application/json"
+
+        res = req.get_response(fakes.wsgi_app())
+        print res
+        self.assertEqual(res.status_int, 202)
+        server = json.loads(res.body)['server']
+        self.assertEqual(1, server['id'])
+        self.assertTrue(server['config_drive'])
+
+    def test_create_instance_with_config_drive_as_id_v1_1(self):
+        self.config_drive = 2
+        self._setup_for_create_instance()
+
+        image_href = 'http://localhost/v1.1/123/images/2'
+        flavor_ref = 'http://localhost/v1.1/123/flavors/3'
+        body = {
+            'server': {
+                'name': 'config_drive_test',
+                'imageRef': image_href,
+                'flavorRef': flavor_ref,
+                'metadata': {
+                    'hello': 'world',
+                    'open': 'stack',
+                },
+                'personality': {},
+                'config_drive': 2,
+            },
+        }
+
+        req = webob.Request.blank('/v1.1/123/servers')
+        req.method = 'POST'
+        req.body = json.dumps(body)
+        req.headers["content-type"] = "application/json"
+
+        res = req.get_response(fakes.wsgi_app())
+
+        self.assertEqual(res.status_int, 202)
+        server = json.loads(res.body)['server']
+        self.assertEqual(1, server['id'])
+        self.assertTrue(server['config_drive'])
+        self.assertEqual(2, server['config_drive'])
+
+    def test_create_instance_with_bad_config_drive_v1_1(self):
+        self.config_drive = "asdf"
+        self._setup_for_create_instance()
+
+        image_href = 'http://localhost/v1.1/123/images/2'
+        flavor_ref = 'http://localhost/v1.1/123/flavors/3'
+        body = {
+            'server': {
+                'name': 'config_drive_test',
+                'imageRef': image_href,
+                'flavorRef': flavor_ref,
+                'metadata': {
+                    'hello': 'world',
+                    'open': 'stack',
+                },
+                'personality': {},
+                'config_drive': 'asdf',
+            },
+        }
+
+        req = webob.Request.blank('/v1.1/123/servers')
+        req.method = 'POST'
+        req.body = json.dumps(body)
+        req.headers["content-type"] = "application/json"
+
+        res = req.get_response(fakes.wsgi_app())
+        self.assertEqual(res.status_int, 400)
+
+    def test_create_instance_without_config_drive_v1_1(self):
+        self._setup_for_create_instance()
+
+        image_href = 'http://localhost/v1.1/123/images/2'
+        flavor_ref = 'http://localhost/v1.1/123/flavors/3'
+        body = {
+            'server': {
+                'name': 'config_drive_test',
+                'imageRef': image_href,
+                'flavorRef': flavor_ref,
+                'metadata': {
+                    'hello': 'world',
+                    'open': 'stack',
+                },
+                'personality': {},
+                'config_drive': True,
+            },
+        }
+
+        req = webob.Request.blank('/v1.1/123/servers')
+        req.method = 'POST'
+        req.body = json.dumps(body)
+        req.headers["content-type"] = "application/json"
+
+        res = req.get_response(fakes.wsgi_app())
+        self.assertEqual(res.status_int, 202)
+        server = json.loads(res.body)['server']
+        self.assertEqual(1, server['id'])
+        self.assertFalse(server['config_drive'])
+
     def test_create_instance_v1_1_bad_href(self):
         self._setup_for_create_instance()
 
@@ -3449,6 +3576,7 @@
                         "href": "http://localhost/servers/1",
                     },
                 ],
+                "config_drive": None,
             }
         }
 
@@ -3461,6 +3589,7 @@
                 "id": 1,
                 "uuid": self.instance['uuid'],
                 "name": "test_server",
+                "config_drive": None,
                 "links": [
                     {
                         "rel": "self",
@@ -3513,6 +3642,7 @@
                 },
                 "addresses": {},
                 "metadata": {},
+                "config_drive": None,
                 "links": [
                     {
                         "rel": "self",
@@ -3566,6 +3696,7 @@
                 },
                 "addresses": {},
                 "metadata": {},
+                "config_drive": None,
                 "links": [
                     {
                         "rel": "self",
@@ -3618,6 +3749,7 @@
                 },
                 "addresses": {},
                 "metadata": {},
+                "config_drive": None,
                 "accessIPv4": "1.2.3.4",
                 "accessIPv6": "",
                 "links": [
@@ -3672,6 +3804,7 @@
                 },
                 "addresses": {},
                 "metadata": {},
+                "config_drive": None,
                 "accessIPv4": "",
                 "accessIPv6": "fead::1234",
                 "links": [
@@ -3734,6 +3867,7 @@
                     "Open": "Stack",
                     "Number": "1",
                 },
+                "config_drive": None,
                 "links": [
                     {
                         "rel": "self",

=== modified file 'nova/tests/test_compute.py'
--- nova/tests/test_compute.py	2011-08-16 19:12:42 +0000
+++ nova/tests/test_compute.py	2011-08-23 05:22:32 +0000
@@ -2,6 +2,7 @@
 
 # Copyright 2010 United States Government as represented by the
 # Administrator of the National Aeronautics and Space Administration.
+# Copyright 2011 Piston Cloud Computing, Inc.
 # All Rights Reserved.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License"); you may
@@ -159,6 +160,20 @@
             db.security_group_destroy(self.context, group['id'])
             db.instance_destroy(self.context, ref[0]['id'])
 
+    def test_create_instance_associates_config_drive(self):
+        """Make sure create associates a config drive."""
+
+        instance_id = self._create_instance(params={'config_drive': True, })
+
+        try:
+            self.compute.run_instance(self.context, instance_id)
+            instances = db.instance_get_all(context.get_admin_context())
+            instance = instances[0]
+
+            self.assertTrue(instance.config_drive)
+        finally:
+            db.instance_destroy(self.context, instance_id)
+
     def test_default_hostname_generator(self):
         cases = [(None, 'server_1'), ('Hello, Server!', 'hello_server'),
                  ('<}\x1fh\x10e\x08l\x02l\x05o\x12!{>', 'hello')]

=== modified file 'nova/virt/disk.py'
--- nova/virt/disk.py	2011-08-05 14:23:48 +0000
+++ nova/virt/disk.py	2011-08-23 05:22:32 +0000
@@ -2,6 +2,9 @@
 
 # Copyright 2010 United States Government as represented by the
 # Administrator of the National Aeronautics and Space Administration.
+#
+# Copyright 2011, Piston Cloud Computing, Inc.
+#
 # All Rights Reserved.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License"); you may
@@ -22,6 +25,7 @@
 
 """
 
+import json
 import os
 import tempfile
 import time
@@ -60,7 +64,8 @@
     utils.execute('resize2fs', image, check_exit_code=False)
 
 
-def inject_data(image, key=None, net=None, partition=None, nbd=False):
+def inject_data(image, key=None, net=None, metadata=None,
+                partition=None, nbd=False, tune2fs=True):
     """Injects a ssh key and optionally net data into a disk image.
 
     it will mount the image as a fully partitioned disk and attempt to inject
@@ -89,10 +94,10 @@
                                       ' only inject raw disk images): %s' %
                                       mapped_device)
 
-            # Configure ext2fs so that it doesn't auto-check every N boots
-            out, err = utils.execute('tune2fs', '-c', 0, '-i', 0,
-                                     mapped_device, run_as_root=True)
-
+            if tune2fs:
+                # Configure ext2fs so that it doesn't auto-check every N boots
+                out, err = utils.execute('tune2fs', '-c', 0, '-i', 0,
+                                         mapped_device, run_as_root=True)
             tmpdir = tempfile.mkdtemp()
             try:
                 # mount loopback to dir
@@ -103,7 +108,8 @@
                                           % err)
 
                 try:
-                    inject_data_into_fs(tmpdir, key, net, utils.execute)
+                    inject_data_into_fs(tmpdir, key, net, metadata,
+                                        utils.execute)
                 finally:
                     # unmount device
                     utils.execute('umount', mapped_device, run_as_root=True)
@@ -155,6 +161,7 @@
 
 def _link_device(image, nbd):
     """Link image to device using loopback or nbd"""
+
     if nbd:
         device = _allocate_device()
         utils.execute('qemu-nbd', '-c', device, image, run_as_root=True)
@@ -190,6 +197,7 @@
     # NOTE(vish): This assumes no other processes are allocating nbd devices.
     #             It may race cause a race condition if multiple
     #             workers are running on a given machine.
+
     while True:
         if not _DEVICES:
             raise exception.Error(_('No free nbd devices'))
@@ -203,7 +211,7 @@
     _DEVICES.append(device)
 
 
-def inject_data_into_fs(fs, key, net, execute):
+def inject_data_into_fs(fs, key, net, metadata, execute):
     """Injects data into a filesystem already mounted by the caller.
     Virt connections can call this directly if they mount their fs
     in a different way to inject_data
@@ -212,6 +220,16 @@
         _inject_key_into_fs(key, fs, execute=execute)
     if net:
         _inject_net_into_fs(net, fs, execute=execute)
+    if metadata:
+        _inject_metadata_into_fs(metadata, fs, execute=execute)
+
+
+def _inject_metadata_into_fs(metadata, fs, execute=None):
+    metadata_path = os.path.join(fs, "meta.js")
+    metadata = dict([(m.key, m.value) for m in metadata])
+
+    utils.execute('sudo', 'tee', metadata_path,
+                  process_input=json.dumps(metadata))
 
 
 def _inject_key_into_fs(key, fs, execute=None):

=== modified file 'nova/virt/libvirt.xml.template'
--- nova/virt/libvirt.xml.template	2011-08-02 02:36:58 +0000
+++ nova/virt/libvirt.xml.template	2011-08-23 05:22:32 +0000
@@ -106,6 +106,13 @@
             </disk>
         #end for
     #end if
+    #if $getVar('config_drive', False)
+        <disk type='file'>
+            <driver type='raw' />
+            <source file='${basepath}/disk.config' />
+            <target dev='${disk_prefix}z' bus='${disk_bus}' />
+        </disk>
+    #end if
 #end if
 
 #for $nic in $nics

=== modified file 'nova/virt/libvirt/connection.py'
--- nova/virt/libvirt/connection.py	2011-08-18 12:57:34 +0000
+++ nova/virt/libvirt/connection.py	2011-08-23 05:22:32 +0000
@@ -4,6 +4,7 @@
 # Administrator of the National Aeronautics and Space Administration.
 # All Rights Reserved.
 # Copyright (c) 2010 Citrix Systems, Inc.
+# Copyright (c) 2011 Piston Cloud Computing, Inc
 #
 #    Licensed under the Apache License, Version 2.0 (the "License"); you may
 #    not use this file except in compliance with the License. You may obtain
@@ -130,6 +131,10 @@
 flags.DEFINE_string('libvirt_vif_driver',
                     'nova.virt.libvirt.vif.LibvirtBridgeDriver',
                     'The libvirt VIF driver to configure the VIFs.')
+flags.DEFINE_string('default_local_format',
+                    None,
+                    'The default format a local_volume will be formatted with '
+                    'on creation.')
 
 
 def get_connection(read_only):
@@ -586,6 +591,7 @@
         self.firewall_driver.prepare_instance_filter(instance, network_info)
         self._create_image(context, instance, xml, network_info=network_info,
                            block_device_info=block_device_info)
+
         domain = self._create_new_domain(xml)
         LOG.debug(_("instance %s: is running"), instance['name'])
         self.firewall_driver.apply_instance_filter(instance, network_info)
@@ -759,10 +765,15 @@
         if size:
             disk.extend(target, size)
 
-    def _create_local(self, target, local_gb):
+    def _create_local(self, target, local_size, prefix='G', fs_format=None):
         """Create a blank image of specified size"""
-        utils.execute('truncate', target, '-s', "%dG" % local_gb)
-        # TODO(vish): should we format disk by default?
+
+        if not fs_format:
+            fs_format = FLAGS.default_local_format
+
+        utils.execute('truncate', target, '-s', "%d%c" % (local_size, prefix))
+        if fs_format:
+            utils.execute('mkfs', '-t', fs_format, target)
 
     def _create_swap(self, target, swap_gb):
         """Create a swap file of specified size"""
@@ -849,14 +860,14 @@
                               target=basepath('disk.local'),
                               fname="local_%s" % local_gb,
                               cow=FLAGS.use_cow_images,
-                              local_gb=local_gb)
+                              local_size=local_gb)
 
         for eph in driver.block_device_info_get_ephemerals(block_device_info):
             self._cache_image(fn=self._create_local,
                               target=basepath(_get_eph_disk(eph)),
                               fname="local_%s" % eph['size'],
                               cow=FLAGS.use_cow_images,
-                              local_gb=eph['size'])
+                              local_size=eph['size'])
 
         swap_gb = 0
 
@@ -882,9 +893,24 @@
         if not inst['kernel_id']:
             target_partition = "1"
 
-        if FLAGS.libvirt_type == 'lxc':
+        config_drive_id = inst.get('config_drive_id')
+        config_drive = inst.get('config_drive')
+
+        if any((FLAGS.libvirt_type == 'lxc', config_drive, config_drive_id)):
             target_partition = None
 
+        if config_drive_id:
+            fname = '%08x' % int(config_drive_id)
+            self._cache_image(fn=self._fetch_image,
+                              target=basepath('disk.config'),
+                              fname=fname,
+                              image_id=config_drive_id,
+                              user=user,
+                              project=project)
+        elif config_drive:
+            self._create_local(basepath('disk.config'), 64, prefix="M",
+                               fs_format='msdos')  # 64MB
+
         if inst['key_data']:
             key = str(inst['key_data'])
         else:
@@ -928,19 +954,29 @@
                                searchList=[{'interfaces': nets,
                                             'use_ipv6': FLAGS.use_ipv6}]))
 
-        if key or net:
+        metadata = inst.get('metadata')
+        if any((key, net, metadata)):
             inst_name = inst['name']
-            img_id = inst.image_ref
-            if key:
-                LOG.info(_('instance %(inst_name)s: injecting key into'
-                        ' image %(img_id)s') % locals())
-            if net:
-                LOG.info(_('instance %(inst_name)s: injecting net into'
-                        ' image %(img_id)s') % locals())
+
+            if config_drive:  # Should be True or None by now.
+                injection_path = basepath('disk.config')
+                img_id = 'config-drive'
+                tune2fs = False
+            else:
+                injection_path = basepath('disk')
+                img_id = inst.image_ref
+                tune2fs = True
+
+            for injection in ('metadata', 'key', 'net'):
+                if locals()[injection]:
+                    LOG.info(_('instance %(inst_name)s: injecting '
+                               '%(injection)s into image %(img_id)s'
+                               % locals()))
             try:
-                disk.inject_data(basepath('disk'), key, net,
+                disk.inject_data(injection_path, key, net, metadata,
                                  partition=target_partition,
-                                 nbd=FLAGS.use_cow_images)
+                                 nbd=FLAGS.use_cow_images,
+                                 tune2fs=tune2fs)
 
                 if FLAGS.libvirt_type == 'lxc':
                     disk.setup_container(basepath('disk'),
@@ -1070,6 +1106,10 @@
                                           block_device_info)):
             xml_info['swap_device'] = self.default_swap_device
 
+        config_drive = False
+        if instance.get('config_drive') or instance.get('config_drive_id'):
+            xml_info['config_drive'] = xml_info['basepath'] + "/disk.config"
+
         if FLAGS.vnc_enabled and FLAGS.libvirt_type not in ('lxc', 'uml'):
             xml_info['vncserver_host'] = FLAGS.vncserver_host
             xml_info['vnc_keymap'] = FLAGS.vnc_keymap

=== modified file 'nova/virt/xenapi/vm_utils.py'
--- nova/virt/xenapi/vm_utils.py	2011-08-15 18:35:53 +0000
+++ nova/virt/xenapi/vm_utils.py	2011-08-23 05:22:32 +0000
@@ -1,6 +1,7 @@
 # vim: tabstop=4 shiftwidth=4 softtabstop=4
 
 # Copyright (c) 2010 Citrix Systems, Inc.
+# Copyright 2011 Piston Cloud Computing, Inc.
 #
 #    Licensed under the Apache License, Version 2.0 (the "License"); you may
 #    not use this file except in compliance with the License. You may obtain
@@ -740,13 +741,14 @@
         # if at all, so determine whether it's required first, and then do
         # everything
         mount_required = False
-        key, net = _prepare_injectables(instance, network_info)
-        mount_required = key or net
+        key, net, metadata = _prepare_injectables(instance, network_info)
+        mount_required = key or net or metadata
         if not mount_required:
             return
 
         with_vdi_attached_here(session, vdi_ref, False,
-                               lambda dev: _mounted_processing(dev, key, net))
+                               lambda dev: _mounted_processing(dev, key, net,
+                                                               metadata))
 
     @classmethod
     def lookup_kernel_ramdisk(cls, session, vm):
@@ -1198,7 +1200,7 @@
     return False
 
 
-def _mounted_processing(device, key, net):
+def _mounted_processing(device, key, net, metadata):
     """Callback which runs with the image VDI attached"""
 
     dev_path = '/dev/' + device + '1'  # NB: Partition 1 hardcoded
@@ -1212,7 +1214,7 @@
                 if not _find_guest_agent(tmpdir, FLAGS.xenapi_agent_path):
                     LOG.info(_('Manipulating interface files '
                             'directly'))
-                    disk.inject_data_into_fs(tmpdir, key, net,
+                    disk.inject_data_into_fs(tmpdir, key, net, metadata,
                         utils.execute)
             finally:
                 utils.execute('umount', dev_path, run_as_root=True)
@@ -1235,6 +1237,7 @@
     template = t.Template
     template_data = open(FLAGS.injected_network_template).read()
 
+    metadata = inst['metadata']
     key = str(inst['key_data'])
     net = None
     if networks_info:
@@ -1272,4 +1275,4 @@
             net = str(template(template_data,
                                 searchList=[{'interfaces': interfaces_info,
                                             'use_ipv6': FLAGS.use_ipv6}]))
-    return key, net
+    return key, net, metadata

=== modified file 'nova/virt/xenapi/vmops.py'
--- nova/virt/xenapi/vmops.py	2011-08-17 03:15:54 +0000
+++ nova/virt/xenapi/vmops.py	2011-08-23 05:22:32 +0000
@@ -239,8 +239,9 @@
         self._attach_disks(instance, disk_image_type, vm_ref, first_vdi_ref,
             vdis)
 
-        # Alter the image before VM start for, e.g. network injection
-        if FLAGS.flat_injected:
+        # Alter the image before VM start for, e.g. network injection also
+        # alter the image if there's metadata.
+        if FLAGS.flat_injected or instance['metadata']:
             VMHelper.preconfigure_instance(self._session, instance,
                                            first_vdi_ref, network_info)
 

=== modified file 'run_tests.sh'
--- run_tests.sh	2011-07-29 14:54:20 +0000
+++ run_tests.sh	2011-08-23 05:22:32 +0000
@@ -67,7 +67,7 @@
     ERRSIZE=`wc -l run_tests.log | awk '{print \$1}'`
     if [ "$ERRSIZE" -lt "40" ];
     then
-      cat run_tests.log
+        cat run_tests.log
     fi
   fi
   return $RESULT