Merge lp:~justin-fathomdb/nova/ec2-api-with-glance into lp:~hudson-openstack/nova/trunk

[justinsb]

If the glance image type isn't one of the recognized ones, map it to axi- instead of just failing

[Jay Pipes (jaypipes)]

Is this just masking an error? Could you elaborate on what the code path is that is generating this issue? Thanks, Justin!

[Jay Pipes (jaypipes)]

review: Needs Information

[justinsb (justin-fathomdb)]

The properties collection from glance is empty; that looks to be the root problem (vs what the EC2 code is expecting).

I running glance from a relatively recent trunk - r86

---

I uploaded an image using this command:

bin/glance --verbose add name="lucid-8gpart-x64" container_format="bare" disk_format="raw" is_public="true" < /home/justinsb/images/ubuntu/createimage/ubuntu_x64_8g.raw.gz

Added new image with ID: 8
Returned the following metadata for the new image:
               container_format => bare
                     created_at => 2011-03-22T06:03:21.139434
                        deleted => False
                     deleted_at => None
                    disk_format => raw
                             id => 8
                      is_public => True
                       location => file:///var/lib/glance/images/8
                           name => lucid-8gpart-x64
                     properties => {}
                           size => 292789118
                         status => active
                     updated_at => None
Completed in 5.8610 sec.

---
Extract from /images/details:

 {"status": "active", "name": "lucid-8gpart-x64", "deleted": false, "container_format": "bare", "created_at": "2011-03-22T06:03:21.139434", "disk_format": "raw", "updated_at": "2011-03-22T06:03:26.675561", "id": 8, "location": "file:///var/lib/glance/images/8", "is_public": true, "deleted_at": null, "properties": {}, "size": 292789118}

---
Extract from euca-describe-images:

IMAGE axi-00000008 (lucid-8gpart-x64) private

(name is correct, axi hack is triggered, public/private is broken)

[Vish Ishaya (vishvananda)]

Ec2 needs some specific metadata. If you want images to work with ec2, you
should probably upload them with nova-manage image commands. If the image
doesn't have the proper metadata, maybe the ec2 api should just skip it.
On Mar 22, 2011 9:08 AM, "justinsb" <email address hidden> wrote:
> The properties collection from glance is empty; that looks to be the root
problem (vs what the EC2 code is expecting).
>
> I running glance from a relatively recent trunk - r86
>
> ---
>
> I uploaded an image using this command:
>
> bin/glance --verbose add name="lucid-8gpart-x64" container_format="bare"
disk_format="raw" is_public="true" <
/home/justinsb/images/ubuntu/createimage/ubuntu_x64_8g.raw.gz
>
> Added new image with ID: 8
> Returned the following metadata for the new image:
> container_format => bare
> created_at => 2011-03-22T06:03:21.139434
> deleted => False
> deleted_at => None
> disk_format => raw
> id => 8
> is_public => True
> location => file:///var/lib/glance/images/8
> name => lucid-8gpart-x64
> properties => {}
> size => 292789118
> status => active
> updated_at => None
> Completed in 5.8610 sec.
>
> ---
> Extract from /images/details:
>
> {"status": "active", "name": "lucid-8gpart-x64", "deleted": false,
"container_format": "bare", "created_at": "2011-03-22T06:03:21.139434",
"disk_format": "raw", "updated_at": "2011-03-22T06:03:26.675561", "id": 8,
"location": "file:///var/lib/glance/images/8", "is_public": true,
"deleted_at": null, "properties": {}, "size": 292789118}
>
> ---
> Extract from euca-describe-images:
>
> IMAGE axi-00000008 (lucid-8gpart-x64) private
>
>
> (name is correct, axi hack is triggered, public/private is broken)
>
> --
>
https://code.launchpad.net/~justin-fathomdb/nova/ec2-api-with-glance/+merge/54288
> You are subscribed to branch lp:nova.

[justinsb (justin-fathomdb)]

I think skipping over volumes would be pretty confusing, but it's better than failing the whole request. That's why I prefer "axi-", but I haven't checked the EC2 spec in detail here.

I think there's a deeper issue though, because public/private also seems broken. Which is "at fault" though - Glance or the EC2 API?

[Vish Ishaya (vishvananda)]

Glance doesn't actually return images if the is_public is false. Nor does it yet provide filtering options by owner, etc. Therefore the ec2 wrapper uses its own is_public and owner fields in the properties dictionary.

Vish

On Mar 22, 2011, at 12:55 PM, justinsb wrote:

> I think skipping over volumes would be pretty confusing, but it's better than failing the whole request. That's why I prefer "axi-", but I haven't checked the EC2 spec in detail here.
>
> I think there's a deeper issue though, because public/private also seems broken. Which is "at fault" though - Glance or the EC2 API?
>
>
> --
> https://code.launchpad.net/~justin-fathomdb/nova/ec2-api-with-glance/+merge/54288
> You are subscribed to branch lp:nova.

[Jay Pipes (jaypipes)]

> Glance doesn't actually return images if the is_public is false.

Yes it does. You need to call show(<ID>), not index(). index() and details() only return public images, as per the OpenStack images/ API, for good or bad.

> Nor does it
> yet provide filtering options by owner, etc. Therefore the ec2 wrapper uses
> its own is_public and owner fields in the properties dictionary.

Coming shortly.

https://blueprints.launchpad.net/glance/+spec/api-results-filtering

-jay

[Vish Ishaya (vishvananda)]

Ah good to know. Probably could do some refactoring knowing that. Is there a way we could add a parameter to the other calls allowing them to return public private? Perhaps as part of filtering?

Vish

On Mar 22, 2011, at 1:25 PM, Jay Pipes wrote:

>> Glance doesn't actually return images if the is_public is false.
>
> Yes it does. You need to call show(<ID>), not index(). index() and details() only return public images, as per the OpenStack images/ API, for good or bad.
>
>> Nor does it
>> yet provide filtering options by owner, etc. Therefore the ec2 wrapper uses
>> its own is_public and owner fields in the properties dictionary.
>
> Coming shortly.
>
> https://blueprints.launchpad.net/glance/+spec/api-results-filtering
>
> -jay
> --
> https://code.launchpad.net/~justin-fathomdb/nova/ec2-api-with-glance/+merge/54288
> You are subscribed to branch lp:nova.

[Jay Pipes (jaypipes)]

On Tue, Mar 22, 2011 at 4:31 PM, Vish Ishaya <email address hidden> wrote:
> Ah good to know.  Probably could do some refactoring knowing that. Is there a way we could add a parameter to the other calls allowing them to return public private?  Perhaps as part of filtering?

Definitely! I was actually planning on doing that for that blueprint
on filtering. :) Finally figured out migrations today, so hopefully
will be able to pound out that BP tomorrow.
-jay

[Rick Harris (rconradharris)]

The issue here is that the ImageServices (Glance vs S3) and APIs (EC2 vs OpenStack API) differ in their notion of image_type.

With Glance+OpenstackAPI we've moved to using `disk_format` and `container_format` which should be flexible enough to handle most cases. I think moving forward, we should expose this in the BaseImageService, then all of the APIs can rely on disk_format and container_format for doing different things.

So, this would involve:

  1. Adding `disk_format` + `container_format` as BASE_IMAGE_ATTRS for BaseImageService (see related_images patch for what this means)

  2. Modifying all of the ImageServices to populate this field

  3. Modifying the API to pull from this from the ImageService.disk_format+container_format fields instead of whatever ad-hoc way they're using now.

Thoughts?

-----

From IRC:

sirp_: justinsb: sorry for the delay, yeah, ultimately we need to have the BaseImageService abstract out the notion of image_types. I think going foward we want to use `disk_format` and `container_format`; so this would mean changing making changes to the S3ImageService to map image_types to the appropriate disk_format+container_format pair

[1:32pm] sirp_: justinsb: the related_images patch adds some code to make this translation easier, once that hits, i think we can work on unifiying the concepts of image_types amongst all of the ImageServices

review: Needs Information

[justinsb (justin-fathomdb)]

Thanks Rick. If the way forward is to fix all the image services, that's fine with me. I did try suggesting more of a 'design by contract' approach a few weeks ago:
https://code.launchpad.net/~justin-fathomdb/nova/strongly-typed-image-model

But we tabled that until the Design Summit.

It sounds like the approach you're suggesting could benefit from this approach though. If I won't get shot down again, I'm happy to recode the strong-typed image model approach...

Otherwise, I'm fine with any approach as long as we can get this working smoothly in Cactus!

[Vish Ishaya (vishvananda)]

Since this actually stops the image service from breaking, on bad images, we should probably just get this in and make another bug about cleaning image services.

[Jay Pipes (jaypipes)]

> The properties collection from glance is empty; that looks to be the root
> problem (vs what the EC2 code is expecting).

Still sure this is happening? I thought Rick's related_images patch, now in trunk I think, fixes this?

> Extract from euca-describe-images:
>
> IMAGE axi-00000008 (lucid-8gpart-x64) private
>
> (name is correct, axi hack is triggered, public/private is broken)

Sorry, how is the private/public bug related to this patch?

-jay

review: Needs Information

[justinsb (justin-fathomdb)]

> Still sure this is happening? I thought Rick's related_images patch, now in trunk I think, fixes this?

I can check...

> Sorry, how is the private/public bug related to this patch?

It was just commentary on things that were going wrong, to help understand the issue. I thought it was likely another property that also wasn't being mapped correctly.

[justinsb (justin-fathomdb)]

I don't see how this would have been fixed:
http://bazaar.launchpad.net/~hudson-openstack/nova/trunk/view/head:/nova/api/ec2/cloud.py

def _format_image(self, image):
        """Convert from format defined by BaseImageService to S3 format."""
        i = {}
        image_type = image['properties'].get('type')
        ec2_id = self._image_ec2_id(image.get('id'), image_type)

...

 _type_prefix_map = {'machine': 'ami',
                        'kernel': 'aki',
                        'ramdisk': 'ari'}

def _image_ec2_id(self, image_id, image_type='machine'):
        prefix = self._type_prefix_map[image_type]

---

No images['properties']['type'] => KAPOW!

[justinsb (justin-fathomdb)]

Presuming that somebody else has fixed this by now.

Moving to WIP

Unmerged revisions

843. By justinsb on 2011-03-22

Map unknown image_types to the EC2 prefix axi

=== modified file 'nova/api/ec2/cloud.py'
--- nova/api/ec2/cloud.py	2011-03-17 15:14:59 +0000
+++ nova/api/ec2/cloud.py	2011-03-22 00:52:27 +0000
@@ -856,7 +856,7 @@
                         'ramdisk': 'ari'}
 
     def _image_ec2_id(self, image_id, image_type='machine'):
-        prefix = self._type_prefix_map[image_type]
+        prefix = self._type_prefix_map.get(image_type, 'axi')
         template = prefix + '-%08x'
         return ec2utils.id_to_ec2_id(int(image_id), template=template)