Merge lp:~james-page/simplestreams/multihypervisor into lp:~smoser/simplestreams/trunk

[James Page]

Enable better support for multi-hypervisor regions in OpenStack

1) Determine correct disk format.
2) Set hypervisor_type on images based on ftype field.

[lp:~james-page/simplestreams/multihypervisor]

417. By James Page on 2015-11-20

Fixup syntax

418. By James Page on 2015-11-20

Fixup lint

419. By James Page on 2015-11-20

Fixup disk_format handling

420. By James Page on 2015-11-20

Set sane default

421. By James Page on 2015-11-20

Tidy

422. By James Page on 2015-11-20

Use constants for disk format mappings

[Scott Moser (smoser)]

<smoser> jamespage, is 'hypervisor_type' a openstack documented thing ?
<jamespage> smoser, it is
<smoser> can it be a list ?
<smoser> http://docs.openstack.org/cli-reference/content/chapter_cli-glance-property.html
<jamespage> smoser, http://docs.openstack.org/cli-reference/content/chapter_cli-glance-property.html
<smoser> right.
<smoser> it woudlseem limiting for us to say 'qemu' when in all likelyhood that image shoudl run in xen at least
<jamespage> smoser, I don't think so but I'll check
<jamespage> smoser, yeah that was actually my concern about turning this on by default

[lp:~james-page/simplestreams/multihypervisor]

423. By James Page on 2015-12-14

Add configuration option to enable hypervisor mapping in glance mirror

424. By James Page on 2015-12-14

Fixup hypervisor_mapping key use

425. By James Page on 2015-12-14

Also set hypervisor_type on stream data so that tools can index by hypervisor_type if need be

[James Page (james-page)]

I've added an option to enable this feature (it would be off by default) and also updated the Glance mirror to set the hypervisor_type in the stream data as well so that tools using the stream data can query by virt type as well.

review: Needs Resubmitting

=== modified file 'simplestreams/mirrors/glance.py'
--- simplestreams/mirrors/glance.py	2015-09-22 18:42:12 +0000
+++ simplestreams/mirrors/glance.py	2015-12-14 09:42:45 +0000
@@ -61,6 +61,37 @@
     return newarch
 
 
+LXC_FTYPES = [
+    'root.tar.gz',
+    'root.tar.xz',
+]
+
+QEMU_FTYPES = [
+    'disk.img',
+    'disk1.img',
+]
+
+
+def disk_format(ftype):
+    '''Canonicalize disk formats for use in OpenStack'''
+    newftype = ftype.lower()
+    if newftype in LXC_FTYPES:
+        return 'root-tar'
+    if newftype in QEMU_FTYPES:
+        return 'qcow2'
+    return None
+
+
+def hypervisor_type(ftype):
+    '''Determine hypervisor type based on image format'''
+    newftype = ftype.lower()
+    if newftype in LXC_FTYPES:
+        return 'lxc'
+    if newftype in QEMU_FTYPES:
+        return 'qemu'
+    return None
+
+
 # glance mirror 'image-downloads' content into glance
 # if provided an object store, it will produce a 'image-ids' mirror
 class GlanceMirror(mirrors.BasicMirrorWriter):
@@ -198,6 +229,12 @@
             t_item['arch'] = arch
             props['architecture'] = canonicalize_arch(arch)
 
+        if self.config['hypervisor_mapping'] and 'ftype' in flat:
+            _hypervisor_type = hypervisor_type(flat['ftype'])
+            if _hypervisor_type:
+                props['hypervisor_type'] = _hypervisor_type
+                t_item['hypervisor_type'] = _hypervisor_type
+
         if 'os' in flat:
             props['os_distro'] = flat['os']
 
@@ -208,7 +245,6 @@
         create_kwargs = {
             'name': fullname,
             'properties': props,
-            'disk_format': 'qcow2',
             'container_format': 'bare',
             'is_public': True,
         }
@@ -218,6 +254,13 @@
         if 'md5' in data:
             create_kwargs['checksum'] = data.get('md5')
 
+        if 'ftype' in flat:
+            create_kwargs['disk_format'] = (
+                disk_format(flat['ftype']) or 'qcow2'
+            )
+        else:
+            create_kwargs['disk_format'] = 'qcow2'
+
         if self.progress_callback:
             def progress_wrapper(written):
                 self.progress_callback(dict(status="Downloading",

=== modified file 'tools/sstream-mirror-glance'
--- tools/sstream-mirror-glance	2015-04-10 21:00:00 +0000
+++ tools/sstream-mirror-glance	2015-12-14 09:42:45 +0000
@@ -104,6 +104,11 @@
                         "Expressions are key[!]=literal_string "
                         "or key[!]~regexp.")
 
+    parser.add_argument('--hypervisor-mapping', action='store_true', default=False,
+                        help="Set hypervisor_type attribute on stored images. "
+                        "This is useful in OpenStack Clouds which use multiple "
+                        "hypervisor types with in a single region.")
+
     args = parser.parse_args()
 
     modify_hook = None
@@ -113,7 +118,8 @@
     mirror_config = {'max_items': args.max, 'keep_items': args.keep,
                      'cloud_name': args.cloud_name,
                      'modify_hook': modify_hook,
-                     'item_filters': args.item_filters}
+                     'item_filters': args.item_filters,
+                     'hypervisor_mapping': args.hypervisor_mapping}
 
     (mirror_url, args.path) = util.path_from_mirror_url(args.source_mirror,
                                                         args.path)