Merge lp:~yamahata/glance/lp802893 into lp:~hudson-openstack/glance/trunk

[Isaku Yamahata]

Glance doesn't allow structured metadata

boot-from-volume patch series needs to store the informations about
block-device-mapping in metadata.
It has structured data as following. That is, its value can be
a dictionary or a list. And its child value also can be...

local image service (which was deleted) and fakeimage service can
properly store such metadata as they just use json.dumps/loads.
However, glance image service doesn't so that block-device-mapping
informations can't be stored properly.

This patch teaches structured metadata to glance api/registry daemons.

metadata = {'name': 'fake public image',
           'properties': {
               'mappings': [
                   {'device_name': '/dev/sda1',
                    'snapshot_id': 0x12345678,
                    'delete_on_termination': False},
                   {'device_name': '/dev/sda2',
                    'no_device': True}],
               'block_device_mapping': [
   {'virtual': 'ami', 'device': 'sda1'},
                        {'virtual': 'root', 'device': '/dev/sda1'},

                        {'virtual': 'swap', 'device': 'sdb1'},
                        {'virtual': 'swap', 'device': 'sdb2'},

                        {'virtual': 'ephemeral0', 'device': 'sdc1'},
                        {'virtual': 'ephemeral1', 'device': 'sdc2'}]}}

boot-from-volume patch series needs to store the informations about
block-device-mapping in metadata.
It has structured data as following. That is, its value can be
a dictionary or a list. And its child value also can be...

local image service (which was deleted) and fakeimage service can
properly store such metadata as they just use json.dumps/loads.
However, glance image service doesn't so that block-device-mapping
informations can't be stored properly.

This patch teaches structured metadata to glance api/registry daemons.

metadata = {'name': 'fake public image',
           'properties': {
               'mappings': [
                   {'device_name': '/dev/sda1',
                    'snapshot_id': 0x12345678,
                    'delete_on_termination': False},
                   {'device_name': '/dev/sda2',
                    'no_device': True}],
               'block_device_mapping': [
   {'virtual': 'ami', 'device': 'sda1'},
                        {'virtual': 'root', 'device': '/dev/sda1'},

                        {'virtual': 'swap', 'device': 'sdb1'},
                        {'virtual': 'swap', 'device': 'sdb2'},

                        {'virtual': 'ephemeral0', 'device': 'sdc1'},
                        {'virtual': 'ephemeral1', 'device': 'sdc2'}]}}

[Brian Waldon (bcwaldon)]

This is an interesting idea, but I'm a little hesitant to call this a bug. Keeping metadata simple was an explicit design design. I think this should probably be filed as a blueprint targeted at diablo-3, assuming this is something we want. Thoughts, Jay?

I would also like to see a lot more testing around this. I would want to guarantee this will work at every level, not just the utils function that does the mapping.

[Christopher MacGown (0x44)]

I agree that this is probably something we should file as a blueprint and discuss. I think that if we need more complex metadata in Glance that it's more appropriate to make a specific effort for Glance to support querying against OVF manifests/metadata than rolling our own.

[Isaku Yamahata (yamahata)]

On Tue, Jun 28, 2011 at 02:48:38PM -0000, Brian Waldon wrote:
> This is an interesting idea, but I'm a little hesitant to call this a bug. Keeping metadata simple was an explicit design design. I think this should probably be filed as a blueprint targeted at diablo-3, assuming this is something we want. Thoughts, Jay?

Fair enough, I'll file a blueprint and let's start discussion.
I think the discussion involves both nova and glance. So I'll also start
a discussion thread on openstack devel ml in order to draw attention from
nova developer. So far I've thought it's Glance issue so that I created
this patch to Glance. Other option would be to change nova metadata handling.

Jay, any comments?

thanks,

> I would also like to see a lot more testing around this. I would want to guarantee this will work at every level, not just the utils function that does the mapping.
--
yamahata

[Jay Pipes (jaypipes)]

> On Tue, Jun 28, 2011 at 02:48:38PM -0000, Brian Waldon wrote:
> > This is an interesting idea, but I'm a little hesitant to call this a bug.
> Keeping metadata simple was an explicit design design. I think this should
> probably be filed as a blueprint targeted at diablo-3, assuming this is
> something we want. Thoughts, Jay?
>
> Fair enough, I'll file a blueprint and let's start discussion.

Cool. Agreed it's big enough to be a blueprint discussion.

> I think the discussion involves both nova and glance. So I'll also start
> a discussion thread on openstack devel ml in order to draw attention from
> nova developer. So far I've thought it's Glance issue so that I created
> this patch to Glance. Other option would be to change nova metadata handling.
>
> Jay, any comments?

I think it's a good feature to add, but I'd almost prefer to add it as new middleware on the registry, or even as an extension? That way you could do something like:

PUT /images/<ID>/container-format

with a body of something like:

'format': {
  'mappings': [
                   {'device_name': '/dev/sda1',
                    'snapshot_id': 0x12345678,
                    'delete_on_termination': False},
                   {'device_name': '/dev/sda2',
                    'no_device': True}
  ],
  'block_device_mapping': [
                   {'virtual': 'ami', 'device': 'sda1'},
                   {'virtual': 'root', 'device': '/dev/sda1'},
                   {'virtual': 'swap', 'device': 'sdb1'},
                   {'virtual': 'swap', 'device': 'sdb2'},
                   {'virtual': 'ephemeral0', 'device': 'sdc1'},
                   {'virtual': 'ephemeral1', 'device': 'sdc2'}
  ],
}

And then, the extension/middleware can store specific information about the container in separate tables in the database that can be queried using a more specific API than the very limited custom key/value properties we currently have.

Thoughts?
-jay

[Vish Ishaya (vishvananda)]

Is there any reason you can't just do a json.dumps and save it as a text blob, then use json.loads on the other end? I don't see why complex metadata needs to be split into a bunch of separate properties.

Vish

On Jun 29, 2011, at 2:08 PM, Jay Pipes wrote:

>> On Tue, Jun 28, 2011 at 02:48:38PM -0000, Brian Waldon wrote:
>>> This is an interesting idea, but I'm a little hesitant to call this a bug.
>> Keeping metadata simple was an explicit design design. I think this should
>> probably be filed as a blueprint targeted at diablo-3, assuming this is
>> something we want. Thoughts, Jay?
>>
>> Fair enough, I'll file a blueprint and let's start discussion.
>
> Cool. Agreed it's big enough to be a blueprint discussion.
>
>> I think the discussion involves both nova and glance. So I'll also start
>> a discussion thread on openstack devel ml in order to draw attention from
>> nova developer. So far I've thought it's Glance issue so that I created
>> this patch to Glance. Other option would be to change nova metadata handling.
>>
>> Jay, any comments?
>
> I think it's a good feature to add, but I'd almost prefer to add it as new middleware on the registry, or even as an extension? That way you could do something like:
>
> PUT /images/<ID>/container-format
>
> with a body of something like:
>
> 'format': {
> 'mappings': [
> {'device_name': '/dev/sda1',
> 'snapshot_id': 0x12345678,
> 'delete_on_termination': False},
> {'device_name': '/dev/sda2',
> 'no_device': True}
> ],
> 'block_device_mapping': [
> {'virtual': 'ami', 'device': 'sda1'},
> {'virtual': 'root', 'device': '/dev/sda1'},
> {'virtual': 'swap', 'device': 'sdb1'},
> {'virtual': 'swap', 'device': 'sdb2'},
> {'virtual': 'ephemeral0', 'device': 'sdc1'},
> {'virtual': 'ephemeral1', 'device': 'sdc2'}
> ],
> }
>
> And then, the extension/middleware can store specific information about the container in separate tables in the database that can be queried using a more specific API than the very limited custom key/value properties we currently have.
>
> Thoughts?
> -jay
>
> --
> https://code.launchpad.net/~yamahata/glance/lp802893/+merge/66096
> Your team Glance Core is requested to review the proposed merge of lp:~yamahata/glance/lp802893 into lp:glance.

[Jay Pipes (jaypipes)]

On Wed, Jun 29, 2011 at 4:43 PM, Vish Ishaya <email address hidden> wrote:
> Is there any reason you can't just do a json.dumps and save it as a text blob, then use json.loads on the other end?  I don't see why complex metadata needs to be split into a bunch of separate properties.

That's a perfectly reasonable solution.

Of course, it would preclude any ability to filter on such things. If
I wanted to find all images where delete_on_terminate was set, the
query would be a beast and issue a full table scan over
image_properties, since there would be no ability to do an equality
search.

So, long term, I think if it's information we want to query, we should
look to a more robust method of storing the pieces of information.

Cheers,
jay

[Isaku Yamahata (yamahata)]

On Wed, Jun 29, 2011 at 09:08:42PM -0000, Jay Pipes wrote:
> > On Tue, Jun 28, 2011 at 02:48:38PM -0000, Brian Waldon wrote:
> > > This is an interesting idea, but I'm a little hesitant to call this a bug.
> > Keeping metadata simple was an explicit design design. I think this should
> > probably be filed as a blueprint targeted at diablo-3, assuming this is
> > something we want. Thoughts, Jay?
> >
> > Fair enough, I'll file a blueprint and let's start discussion.
>
> Cool. Agreed it's big enough to be a blueprint discussion.

I created the blueprint and the spec page.

https://blueprints.launchpad.net/glance/+spec/structured-data-in-metadata
http://wiki.openstack.org/StructuredMetadata

I've listed ideas for implementation candidate for the discussion base.
--
yamahata

[Isaku Yamahata (yamahata)]

On Wed, Jun 29, 2011 at 09:49:27PM -0000, Jay Pipes wrote:
> On Wed, Jun 29, 2011 at 4:43 PM, Vish Ishaya <email address hidden> wrote:
> > Is there any reason you can't just do a json.dumps and save it as a text blob, then use json.loads on the other end? ??I don't see why complex metadata needs to be split into a bunch of separate properties.
>
> That's a perfectly reasonable solution.
>
> Of course, it would preclude any ability to filter on such things. If
> I wanted to find all images where delete_on_terminate was set, the
> query would be a beast and issue a full table scan over
> image_properties, since there would be no ability to do an equality
> search.
>
> So, long term, I think if it's information we want to query, we should
> look to a more robust method of storing the pieces of information.

If there is no major objection, I'd like to go for the middleware
idea that Jay suggested.

--
yamahata

[Jay Pipes (jaypipes)]

> If there is no major objection, I'd like to go for the middleware
> idea that Jay suggested.

Go for it! No need for permission :)

-jay

[Jay Pipes (jaypipes)]

Yamahata,

Feel free to hop onto IRC freenode.net #openstack-dev if you want to discuss this or need some pointers on where to start :)

-jay

Unmerged revisions

147. By Isaku Yamahata on 2011-06-28

test_utils: added unit tests for new meta data (de)serializer.

146. By Isaku Yamahata on 2011-06-28

utils, glance/registry: meta data serializer/deserialize allow dict/list value

This patch teaches registry sever structured json format.
So far glance-registry accepts only simple key value pair. It doesn't
accept python dict/list as value. On the other hand nova local image
service which was deleted/fake image service do.

On the other hand image meta data needs to have dict/list as value
for boot-from-volume. So this patch makes it possible.

=== modified file 'glance/registry/server.py'
--- glance/registry/server.py	2011-06-15 14:47:00 +0000
+++ glance/registry/server.py	2011-06-28 09:41:34 +0000
@@ -19,12 +19,12 @@
 Reference implementation registry server WSGI controller
 """
 
-import json
 import logging
 
 import routes
 from webob import exc
 
+from glance import utils
 from glance.common import wsgi
 from glance.common import exception
 from glance.registry.db import api as db_api
@@ -196,6 +196,8 @@
         # Ensure the image has a status set
         image_data.setdefault('status', 'active')
 
+        flatten_properties(image_data)
+
         context = None
         try:
             image_data = db_api.image_create(context, image_data)
@@ -220,6 +222,7 @@
 
         """
         image_data = body['image']
+        flatten_properties(image_data)
 
         purge_props = req.headers.get("X-Glance-Registry-Purge-Props", "false")
         context = None
@@ -262,6 +265,14 @@
         super(API, self).__init__(mapper)
 
 
+def flatten_properties(image_data):
+    """if necessary, convert json string into flatten key-value dict whose
+    values aren't dict nor list"""
+    properties = image_data.get('properties')
+    if properties:
+        image_data['properties'] = utils.properties_to_flatten_dict(properties)
+
+
 def make_image_dict(image):
     """
     Create a dict representation of an image which we can use to
@@ -275,8 +286,10 @@
     # TODO(sirp): should this be a dict, or a list of dicts?
     # A plain dict is more convenient, but list of dicts would provide
     # access to created_at, etc
-    properties = dict((p['name'], p['value'])
-                      for p in image['properties'] if not p['deleted'])
+    flatten_dict = dict((p['name'], p['value'])
+                              for p in image['properties'] if not p['deleted'])
+    properties = utils.flatten_dict_to_properties(flatten_dict)
+    utils.convert_properties_value(properties)
 
     image_dict = _fetch_attrs(image, db_api.IMAGE_ATTRS)
 

=== modified file 'glance/utils.py'
--- glance/utils.py	2011-06-11 18:18:03 +0000
+++ glance/utils.py	2011-06-28 09:41:34 +0000
@@ -19,6 +19,148 @@
 A few utility routines used throughout Glance
 """
 
+import functools
+import re
+
+
+def _list_to_enum_dict(image_meta):
+    """Convert list value to enumerated dict.
+    e.g.
+    {k: [v1, v2, v3]} => {k: {'1': v1, '2': v2, '3': v3}}
+    {a: x,
+     b: [{b: y},
+         {c: z}]}
+    =>
+    {a: x,
+     b: {'1': {b: y},
+         '2': {c: z}}}
+    """
+    result = {}
+    for k, v in image_meta.items():
+        if isinstance(v, list):
+            result[k] = dict(zip([str(x) for x in range(len(v))],
+                                 [_list_to_enum_dict(y) for y in v]))
+        elif isinstance(v, dict):
+            result[k] = _list_to_enum_dict(v)
+        else:
+            result[k] = v
+
+    return result
+
+
+def _enum_dict_to_list(enum_dict):
+    """convert enumerated dict elements in a given dict into lists
+    e.g.
+
+    {k: {'1': v1, '2': v2, '3': v3}} => {k: [v1, v2, v3]}
+    {a: x,
+     b: {'1': {b: y},
+         '2': {c: z}}}
+    =>
+    {a: x,
+     b: [{b: y},
+         {c: z}]}
+    """
+    result = {}
+    for k, v in enum_dict.items():
+        assert not isinstance(v, list)
+
+        if isinstance(v, dict):
+            if all([k1.isdigit() for k1 in v.keys()]):
+                result[k] = [_enum_dict_to_list(x) for x in v.values()]
+            else:
+                result[k] = _enum_dict_to_list(v)
+        else:
+            result[k] = v
+
+    return result
+
+
+_DELIMITOR = '--'
+
+
+def _flatten_dict(prefix, image_meta, result=None):
+    """ convert possibly nested dict into flatten dict by encoding
+    the child keys into the primary k
+    e.g.
+    {'k1': {'k2': {'k3': v}} => {prefix + '--k1--k2--k3': unicode(v)}
+    """
+    result = result or {}
+    for k, v in image_meta.items():
+        # list should be converted by _list_to_dict() beforehand
+        assert not isinstance(v, list)
+        if v is None:
+            v = ''
+
+        # k can be ''
+        tmp_prefix = prefix
+        if k:
+            tmp_prefix += _DELIMITOR + k.replace('-', '_').lower()
+
+        if isinstance(v, dict):
+            _flatten_dict(tmp_prefix, v, result)
+        else:
+            result[tmp_prefix] = unicode(v)
+
+    return result
+
+
+def _flatten_dict_to_dict(flatten_dict):
+    """ construct a nested dict from a flatten dict whose key is properly
+    encoded
+    e.g.
+    {'k1--k2--k3': v} => {'k1': {'k2': {'k3': v or None}}
+    """
+    def _descend(parent, k):
+        """descend down the potentially-nested dict.
+        If the value doesn't exist create new dict"""
+        if k in parent:
+            p = parent[k]
+            if not isinstance(p, dict):
+                parent[k] = {'': p}
+        else:
+            parent[k] = {}
+
+        return parent[k]
+
+    result = {}
+    for k, v in flatten_dict.items():
+        v = v or None
+
+        child = result
+        for k in k.split(_DELIMITOR):
+            k = k.replace('-', '_')
+            parent = child
+            child = _descend(parent, k)
+
+        p = parent[k]
+        assert isinstance(p, dict)
+        if p:
+            p[''] = v
+        else:
+            parent[k] = v
+
+    return result
+
+
+def properties_to_flatten_dict(properties):
+    """converted properties into flattened key-value dict."""
+    enum_dict = _list_to_enum_dict(properties)
+    flatten_dict = _flatten_dict('', enum_dict)
+    r = re.compile('^' + _DELIMITOR)
+    flatten_dict = dict([(r.sub('', item[0]), item[1]) for item in
+                         flatten_dict.items()])
+    return flatten_dict
+
+
+def flatten_dict_to_properties(flatten_dict):
+    """reverse conversion of the above function, properties_to_flatten_dict.
+    converted flattened key-value dict into properties.
+    """
+    enum_dict = _flatten_dict_to_dict(flatten_dict)
+    properties = _enum_dict_to_list(enum_dict)
+    return properties
+
 
 def image_meta_to_http_headers(image_meta):
     """
@@ -28,19 +170,83 @@
 
     :param image_meta: Mapping of image metadata
     """
-    headers = {}
-    for k, v in image_meta.items():
-        if v is None:
-            v = ''
-        if k == 'properties':
-            for pk, pv in v.items():
-                if pv is None:
-                    pv = ''
-                headers["x-image-meta-property-%s"
-                        % pk.lower()] = unicode(pv)
-        else:
-            headers["x-image-meta-%s" % k.lower()] = unicode(v)
-    return headers
+
+    # NOTE(yamahata):
+    # image_meta['properties'] is mapped to 'x-image-meta-property'.
+    # So the value of key, 'property', can't be dict or list
+    assert (('property' not in image_meta) or
+            (not (isinstance(image_meta['property'], dict) or
+                  isinstance(image_meta['property'], list))))
+
+    result = properties_to_flatten_dict(image_meta)
+
+    r = re.compile('^' + 'properties' + _DELIMITOR)
+    result = dict([('x-image-meta-' + r.sub('property-', item[0]), item[1])
+                   for item in result.items()])
+    return result
+
+
+def _conv(conv_list, result):
+    """convert value in metadata into python type"""
+    for (k, v) in result.items():
+        if isinstance(v, list):
+            tmp_list = []
+
+            for elem in v:
+                tmp_result = {k: elem}
+                _conv(conv_list, tmp_result)
+                tmp_list.append(tmp_result[k])
+
+            result[k] = tmp_list
+            continue
+
+        for conv in conv_list:
+            if k != conv[0]:
+                continue
+
+            conv = conv[1:]
+            if len(conv) > 1:
+                if isinstance(v, dict):
+                    _conv((conv,), v)
+                result[k] = v
+            else:
+                conv = conv[0]
+                assert isinstance(v, basestring)
+                assert callable(conv)
+                result[k] = conv(v)
+
+
+def convert_properties_value(properties):
+    """convert values in properties of unicode into python type"""
+    conversion = (('block_device_mapping', 'snapshot_id',
+                   functools.partial(int, base=0)),
+                  ('block_device_mapping', 'volume_id',
+                   functools.partial(int, base=0)),
+                  ('block_device_mapping', 'delete_on_termination',
+                   bool_from_header_value),
+                  ('block_device_mapping', 'no_device',
+                   bool_from_header_value))
+
+    _conv(conversion, properties)
+
+
+def _convert_value(result):
+    """Convert metadata values stored in http header as unicode into python
+    type where appropriate"""
+    conversion = (('id', int),
+                  ('size', int),
+                  ('is_public', bool_from_header_value),
+                  ('deleted', bool_from_header_value),
+                  ('properties', 'block_device_mapping', 'snapshot_id',
+                   functools.partial(int, base=0)),
+                  ('properties', 'block_device_mapping', 'volume_id',
+                   functools.partial(int, base=0)),
+                  ('properties', 'block_device_mapping',
+                   'delete_on_termination', bool_from_header_value),
+                  ('properties', 'block_device_mapping', 'no_device',
+                   bool_from_header_value))
+
+    _conv(conversion, result)
 
 
 def get_image_meta_from_headers(response):
@@ -52,30 +258,25 @@
     :param response: Response to process
     """
     result = {}
-    properties = {}
 
     if hasattr(response, 'getheaders'):  # httplib.HTTPResponse
         headers = response.getheaders()
     else:  # webob.Response
         headers = response.headers.items()
 
-    for key, value in headers:
-        key = str(key.lower())
-        if key.startswith('x-image-meta-property-'):
-            field_name = key[len('x-image-meta-property-'):].replace('-', '_')
-            properties[field_name] = value or None
-        elif key.startswith('x-image-meta-'):
-            field_name = key[len('x-image-meta-'):].replace('-', '_')
-            result[field_name] = value or None
-    result['properties'] = properties
-    if 'id' in result:
-        result['id'] = int(result['id'])
-    if 'size' in result:
-        result['size'] = int(result['size'])
-    if 'is_public' in result:
-        result['is_public'] = bool_from_header_value(result['is_public'])
-    if 'deleted' in result:
-        result['deleted'] = bool_from_header_value(result['deleted'])
+    r1 = re.compile('^x-image-meta-', flags=re.IGNORECASE)
+    r2 = re.compile('^property-', flags=re.IGNORECASE)
+    r3 = re.compile('([^-])-([^-])')
+    result = dict([(r3.sub('\\1_\\2',
+                           r2.sub('properties' + _DELIMITOR,
+                                  r1.sub('', str(item[0].lower())))),
+                    item[1])
+                   for item in headers if r1.match(item[0])])
+
+    result = flatten_dict_to_properties(result)
+
+    _convert_value(result)
+    result.setdefault('properties', {})
     return result
 
 

=== modified file 'tests/unit/test_utils.py'
--- tests/unit/test_utils.py	2011-04-12 21:45:16 +0000
+++ tests/unit/test_utils.py	2011-06-28 09:41:34 +0000
@@ -24,6 +24,18 @@
 
     """Test routines in glance.utils"""
 
+    def _assertDictMatch(self, D1, D2):
+        self.assertEqual(len(D1), len(D2))
+        for k in D1.keys():
+            self.assertTrue(k in D1)
+            self.assertEqual(D1[k], D2[k])
+
+    def _assertDictListMatch(self, L1_sorted, L2_sorted):
+        self.assertEqual(len(L1_sorted), len(L2_sorted))
+
+        for d1, d2 in zip(L1_sorted, L2_sorted):
+            self._assertDictMatch(d1, d2)
+
     def test_headers_are_unicode(self):
         """
         Verifies that the headers returned by conversion code are unicode.
@@ -63,9 +75,79 @@
         response = FakeResponse()
         response.headers = headers
         result = utils.get_image_meta_from_headers(response)
+        print "\nfixture %s\n" % fixture
         for k, v in fixture.iteritems():
+            print "k %s v %s result[k] %s" % (k, v, result[k])
             self.assertEqual(v, result[k])
 
+    def test_data_passed_properly_through_headers_mapping(self):
+        """
+        Verifies that data is the same after being passed through headers
+        """
+        mappings = [{'virtual': 'ami', 'device': 'sda1'},
+                    {'virtual': 'root', 'device': '/dev/sda1'},
+
+                    {'virtual': 'swap', 'device': 'sdb1'},
+                    {'virtual': 'swap', 'device': 'sdb2'},
+
+                    {'virtual': 'ephemeral0', 'device': 'sdc1'},
+                    {'virtual': 'ephemeral1', 'device': 'sdc2'}]
+        block_device_mapping = [
+                           # root
+                           {'device_name': '/dev/sda1',
+                            'snapshot_id': 0x12345678,
+                            'delete_on_termination': False},
+                           {'device_name': '/dev/sda2',
+                            'no_device': True},
+
+                           # overwrite swap
+                           {'device_name': '/dev/sdb2',
+                            'snapshot_id': 0x23456789,
+                            'delete_on_termination': False},
+                           {'device_name': '/dev/sdb3',
+                            'snapshot_id': 0x3456789A},
+
+                           # overwrite ephemeral
+                           {'device_name': '/dev/sdc2',
+                            'snapshot_id': 0x3456789A,
+                            'delete_on_termination': False},
+                           {'device_name': '/dev/sdc3',
+                            'snapshot_id': 0x456789AB},
+
+                           # volume
+                           {'device_name': '/dev/sdd1',
+                            'volume_id': 0x87654321,
+                            'delete_on_termination': False},
+                           {'device_name': '/dev/sdd2',
+                            'volume_id': 0x98765432}]
+
+        fixture = {'name': 'fake public image',
+                   'properties': {
+                       'mappings': mappings,
+                       'block_device_mapping':  block_device_mapping}}
+        headers = utils.image_meta_to_http_headers(fixture)
+
+        class FakeResponse():
+            pass
+
+        response = FakeResponse()
+        response.headers = headers
+        result = utils.get_image_meta_from_headers(response)
+        properties = result['properties']
+
+        def _mappings_key(bdm):
+            return bdm['device']
+        mappings.sort(key=_mappings_key)
+        properties['mappings'].sort(key=_mappings_key)
+        self._assertDictListMatch(properties['mappings'], mappings)
+
+        def _block_device_mappings_key(bdm):
+            return bdm['device_name']
+        block_device_mapping.sort(key=_block_device_mappings_key)
+        properties['block_device_mapping'].sort(key=_block_device_mappings_key)
+        self._assertDictListMatch(properties['block_device_mapping'],
+                                  block_device_mapping)
+
     def test_boolean_header_values(self):
         """
         Tests that boolean headers like is_public can be set