Merge ~smoser/ubuntu/+source/simplestreams:bug/xenial-1686437-keystone-v3 into ubuntu/+source/simplestreams:ubuntu/xenial-devel

[Scott Moser]

Openstack keystone v3 and nova-lxd squashfs SRU

Bugs:
 * bug 1578622: do not require that hypervisor_config be present.
 * bug 1583276: ignore inactive images
 * bug 1584938: glance fix race conditions in sync.
 * bug 1611987: glance-simplestreams-sync charm doesn't support keystone v3
 * bug 1686086: glance mirror and nova-lxd need support for squashfs images
 * bug 1686437: glance sync: need keystone v3 auth support
 * bug 1719879: [artful only] swift client needs to use v1 auth prior to ocata
 * bug 1728982: [artful only] openstack mirror with keystone v3 always imports new images

Merge proposals:
 - xenial: https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341214
 - artful: https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341215

PPA:
  https://launchpad.net/~smoser/+archive/ubuntu/sstream-ks3

[Felipe Reyes (freyes)]

I tested this patch using the procedure described in the test case of bug 1686437 . To make the testing easier I created a ppa ( https://launchpad.net/~freyes/+archive/ubuntu/lp1686437 ). After installing this version running the sync up script successfully finished (connecting to keystone using v3 as verified in the logs)

# /etc/cron.daily/glance_simplestreams_sync
created c99bb337-cb1e-4873-9420-11920d911eee: auto-sync/ubuntu-trusty-14.04-amd64-server-20180404-disk1.img
created 4073e5b1-6dde-4534-8f16-5c7ea126d637: auto-sync/ubuntu-xenial-16.04-amd64-server-20180405-disk1.img

/var/log/glance-simplestreams-sync.log -> http://paste.ubuntu.com/p/7VWYsWKx4y/

review: Approve

[David Ames (thedac)]

Also Noting here the released version on xenial does not currently support Keystone v3 and blocks Bug #1611987.

For the record, we have been running a bzr branch @455 on serverstack (a keystone v3 cloud) for months now. So the code in simplestreams works.

[Billy Olsen (billy-olsen)]

The patch looks good to me, but I have a question inlined below regarding the addition of the squashfs format and the implications for syncing multiple image types into a nova lxd cloud using the ftype filter.

[Scott Moser (smoser)]

Billy made a good point. If we're going to pull this back we should grab revno 455 also.

review: Needs Fixing

[Scott Moser (smoser)]

review: Needs Resubmitting

[Scott Moser (smoser)]

Billy and all,
Please re-review and ideally test the PPA upload.

Thanks.
Scott

[Billy Olsen (billy-olsen)]

Scott, thanks for pulling that back. Felipe is going to rebuild the package in the PPA with this latest patch pulled in. Code looks fine to me so I'll approve. I'll let Felipe comment on the testing of the PPA package.

review: Approve

a55c24b... by Scott Moser on 2018-04-12

Pull back glance related fixes from upstream.

This pulls back glance related changes up to upstream revision 455.

428-do-not-require-that-hypervisor_config-be-present.patch (LP: #1578622)
433-glance-ignore-inactive-images.patch (LP: #1583276)
436-glance-fix-race-conditions.patch (LP: #1584938)
450-453-454-keystone-v3-support.patch (LP: #1686437, #1728982, #1719879)
455-nova-lxd-support-squashfs-images.patch (LP: #1686086)

db8b81f... by Scott Moser on 2018-04-12

update changelog

7aa5026... by Scott Moser on 2018-04-12

Pull in revno 460 for keystone v2 session support.

  460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)

0ce3c41... by Scott Moser on 2018-04-12

update changelog

bb79379... by Scott Moser on 2018-04-24

mark 1.4 released

d197243... by Scott Moser on 2018-04-24

1.4~ppa0

[Scott Moser (smoser)]

an upload that I had done got into xenial for bug 1686437.
I've rebased this merge over the top of that.

Same content as before except for debian/changelog.
I've uploaded that to the ppa as 0.1.0~bzr426-0ubuntu1.4~ppa0 .
https://launchpad.net/~smoser/+archive/ubuntu/sstream-ks3/+packages

So from a code perspective, I'm pretty sure that all parties are on board with 0.1.0~bzr426-0ubuntu1.4 being uploaded... but we need to address SRU.

It now references 8 bugs, only 1 of which has an SRU template written.

[Rafael David Tinoco (rafaeldtinoco)]

I'm doing a -1 here according to:

https://bugs.launchpad.net/ubuntu/+source/simplestreams/+bug/1686437/comments/19

and

https://bugs.launchpad.net/ubuntu/+source/simplestreams/+bug/1686437/comments/21

And suggesting a new merge with same contents.

review: Disapprove

Unmerged commits

d197243... by Scott Moser on 2018-04-24

1.4~ppa0

bb79379... by Scott Moser on 2018-04-24

mark 1.4 released

0ce3c41... by Scott Moser on 2018-04-12

update changelog

7aa5026... by Scott Moser on 2018-04-12

Pull in revno 460 for keystone v2 session support.

  460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)

db8b81f... by Scott Moser on 2018-04-12

update changelog

a55c24b... by Scott Moser on 2018-04-12

Pull back glance related fixes from upstream.

This pulls back glance related changes up to upstream revision 455.

428-do-not-require-that-hypervisor_config-be-present.patch (LP: #1578622)
433-glance-ignore-inactive-images.patch (LP: #1583276)
436-glance-fix-race-conditions.patch (LP: #1584938)
450-453-454-keystone-v3-support.patch (LP: #1686437, #1728982, #1719879)
455-nova-lxd-support-squashfs-images.patch (LP: #1686086)

b21bc99... by Scott Moser on 2018-03-09

Import patches-unapplied version 0.1.0~bzr426-0ubuntu1.3 to ubuntu/xenial-proposed

Imported using git-ubuntu import.

Changelog parent: 42652aa520c7856e51d365c9e158124ace8d81a5

New changelog entries:
  * Openstack: Add keystone v3 auth support (LP: #1686437).

42652aa... by Andres Rodriguez on 2016-11-30

Import patches-unapplied version 0.1.0~bzr426-0ubuntu1.2 to ubuntu/xenial-proposed

Imported using git-ubuntu import.

Changelog parent: 353cf1dc2f62eb63936244e8b92c51c29b0d1e05

New changelog entries:
  * [SRU] Set custom user agent (LP: #1578624)

353cf1d... by Robie Basak on 2016-05-12

Import patches-unapplied version 0.1.0~bzr426-0ubuntu1.1 to ubuntu/xenial-proposed

Imported using git-ubuntu import.

Changelog parent: c2750bbdf3e9500a2da0c5f5503cb85d9a59fa00

New changelog entries:
  * Fix signature verification speed (LP: #1580534).

c2750bb... by Scott Moser on 2016-03-23

Import patches-unapplied version 0.1.0~bzr426-0ubuntu1 to ubuntu/xenial-proposed

Imported using git-ubuntu import.

Changelog parent: 69feb70fd3bbe7f44bc5c20a21d844a83e2d407f

New changelog entries:
  * debian/README.source, debian/new-upstream-snapshot: add
    script to do what README.source said to do before.
  * New upstream snapshot.
    - glance mirror: use 'virt' key for hypervisor type and use names
      'kvm' and 'lxd' rather than 'qemu' and 'lxc' (LP: #1560903)

diff --git a/debian/changelog b/debian/changelog
index 01c067e..8bffd8a 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,15 @@
+simplestreams (0.1.0~bzr426-0ubuntu1.4~ppa0) xenial; urgency=medium
+
+  * Pull back several upstream fixes to glance sync code:
+    - 428-do-not-require-that-hypervisor_config-be-present.patch (LP: #1578622)
+    - 433-glance-ignore-inactive-images.patch (LP: #1583276)
+    - 436-glance-fix-race-conditions.patch (LP: #1584938)
+    - 450-453-454-keystone-v3-support.patch (LP: #1686437, #1728982, #1719879)
+    - 455-nova-lxd-support-squashfs-images.patch (LP: #1686086)
+    - 460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)
+
+ -- Scott Moser <[e-mail]>  Thu, 12 Apr 2018 12:27:47 -0400
+
 simplestreams (0.1.0~bzr426-0ubuntu1.3) xenial-proposed; urgency=medium
 
   * Openstack: Add keystone v3 auth support (LP: #1686437).
diff --git a/debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch b/debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch
 new file mode 100644
index 0000000..f449dad
--- /dev/null
+++ b/debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch
@@ -0,0 +1,23 @@
+------------------------------------------------------------
+revno: 428
+fixes bug: https://launchpad.net/bugs/1578622
+committer: Scott Moser <[e-mail]>
+branch nick: trunk
+timestamp: Thu 2016-05-05 08:19:26 -0400
+message:
+  glance mirror: do not require that hypervisor_config be present
+  
+  This just allows for hypervisor_config to not be present.
+=== modified file 'simplestreams/mirrors/glance.py'
+--- a/simplestreams/mirrors/glance.py	2016-03-23 10:00:53 +0000
++++ b/simplestreams/mirrors/glance.py	2016-05-05 12:19:26 +0000
+@@ -239,7 +239,7 @@
+             t_item['arch'] = arch
+             props['architecture'] = canonicalize_arch(arch)
+ 
+-        if self.config['hypervisor_mapping'] and 'ftype' in flat:
++        if self.config.get('hypervisor_mapping', False) and 'ftype' in flat:
+             _hypervisor_type = hypervisor_type(flat['ftype'])
+             if _hypervisor_type:
+                 props['hypervisor_type'] = _hypervisor_type
+
diff --git a/debian/patches/433-glance-ignore-inactive-images.patch b/debian/patches/433-glance-ignore-inactive-images.patch
 new file mode 100644
index 0000000..ee704e0
--- /dev/null
+++ b/debian/patches/433-glance-ignore-inactive-images.patch
@@ -0,0 +1,42 @@
+------------------------------------------------------------
+revno: 433 [merge]
+fixes bug: https://launchpad.net/bugs/1583276
+committer: Scott Moser <[e-mail]>
+branch nick: trunk
+timestamp: Fri 2016-05-20 15:33:14 -0400
+message:
+  glance: ignore inactive images
+  
+  If connection to glance is broken during GlanceMirror.sync(), and image
+  would be left over in "status": "saving".
+  
+  Glance itself tries to protect against that, however when it is restarted
+  (eg. "service glance-api restart"), it does not clean the image up.
+  
+  This change makes load_product() ignore any images that do not have
+  "status": "active".  Images can only have deleted, active or saving
+  status.
+  
+  Note that this isn't perfect. The image in "saving" state is never cleaned
+  up, but at least the resulting cloud works. If we want to clean up broken
+  images, it'd be hard to do on the simplestreams side (glance is better
+  placed to do that: it knows if anything is going on with any of them, eg.
+  is it being currently updated or not).
+------------------------------------------------------------
+Use --include-merged or -n0 to see merged revisions.
+=== modified file 'simplestreams/mirrors/glance.py'
+--- a/simplestreams/mirrors/glance.py	2016-05-05 12:35:42 +0000
++++ b/simplestreams/mirrors/glance.py	2016-05-20 15:33:51 +0000
+@@ -179,6 +179,11 @@
+             if props.get('content_id') != my_cid:
+                 continue
+ 
++            if image.get('status') != "active":
++                LOG.warn("Ignoring inactive image %s with status '%s'" % (
++                    image['id'], image.get('status')))
++                continue
++
+             source_content_id = props.get('source_content_id')
+ 
+             product = props.get('product_name')
+
diff --git a/debian/patches/435-glance-refactor-for-testing.patch b/debian/patches/435-glance-refactor-for-testing.patch
 new file mode 100644
index 0000000..404b531
--- /dev/null
+++ b/debian/patches/435-glance-refactor-for-testing.patch
@@ -0,0 +1,853 @@
+------------------------------------------------------------
+revno: 435 [merge]
+committer: Scott Moser <[e-mail]>
+branch nick: trunk
+timestamp: Tue 2016-06-14 16:10:56 -0400
+message:
+  GlanceMirror: refactor insert_item for easier testing
+  
+  This change refactors GlanceMirror.insert_item() to allow for easier and
+  more contained testing. I needed to do this to understand everything that
+  was going on inside insert_item and other bits of code. There are now four
+  distinct things happening in it:
+  
+   1. Download image to a local file from a ContentSource
+   2. Construct extra properties to store in Glance along with image
+   3. Prepare arguments for GlanceClient.images.create() call
+   4. Adapt source simplestreams entry for an image for use in the target
+      simplestreams index
+  
+  It should be fully backwards compatible, and test coverage for all the
+  individual steps should be much better (I admit to it not being perfect,
+  but it's a step in the right direction, imho at least).
+------------------------------------------------------------
+Use --include-merged or -n0 to see merged revisions.
+=== modified file 'simplestreams/mirrors/glance.py'
+--- a/simplestreams/mirrors/glance.py	2016-05-20 15:33:51 +0000
++++ b/simplestreams/mirrors/glance.py	2016-06-10 18:47:52 +0000
+@@ -105,8 +105,15 @@
+ # glance mirror 'image-downloads' content into glance
+ # if provided an object store, it will produce a 'image-ids' mirror
+ class GlanceMirror(mirrors.BasicMirrorWriter):
++    """
++    GlanceMirror syncs external simplestreams index and images to Glance.
++
++    `client` argument is used for testing to override openstack module:
++    allows dependency injection of fake "openstack" module.
++    """
+     def __init__(self, config, objectstore=None, region=None,
+-                 name_prefix=None, progress_callback=None):
++                 name_prefix=None, progress_callback=None,
++                 client=None):
+         super(GlanceMirror, self).__init__(config=config)
+ 
+         self.item_filters = self.config.get('item_filters', [])
+@@ -123,7 +130,10 @@
+         self.loaded_content = {}
+         self.store = objectstore
+ 
+-        self.keystone_creds = openstack.load_keystone_creds()
++        if client is None:
++            client = openstack
++
++        self.keystone_creds = client.load_keystone_creds()
+ 
+         self.name_prefix = name_prefix or ""
+         if region is not None:
+@@ -131,8 +141,8 @@
+ 
+         self.progress_callback = progress_callback
+ 
+-        conn_info = openstack.get_service_conn_info('image',
+-                                                    **self.keystone_creds)
++        conn_info = client.get_service_conn_info(
++            'image', **self.keystone_creds)
+         self.gclient = get_glanceclient(**conn_info)
+         self.tenant_id = conn_info['tenant_id']
+ 
+@@ -151,6 +161,12 @@
+         return "streams/v1/%s.json" % content_id
+ 
+     def load_products(self, path=None, content_id=None):
++        """
++        Load metadata for all currently uploaded active images in Glance.
++
++        Uses glance as the definitive store, but loads metadata from existing
++        simplestreams indexes as well.
++        """
+         my_cid = self.content_id
+ 
+         # glance is the definitive store.  Any data loaded from the store
+@@ -219,105 +235,212 @@
+     def filter_item(self, data, src, target, pedigree):
+         return filters.filter_item(self.item_filters, data, src, pedigree)
+ 
+-    def insert_item(self, data, src, target, pedigree, contentsource):
+-        flat = util.products_exdata(src, pedigree, include_top=False)
+-
+-        tmp_path = None
+-        tmp_del = None
+-
+-        name = flat.get('pubname', flat.get('name'))
+-        if not name.endswith(flat['item_name']):
+-            name += "-%s" % (flat['item_name'])
+-
+-        t_item = flat.copy()
+-        if 'path' in t_item:
+-            del t_item['path']
+-
+-        props = {'content_id': target['content_id'],
+-                 'source_content_id': src['content_id']}
+-        for n in ('product_name', 'version_name', 'item_name'):
+-            props[n] = flat[n]
+-            del t_item[n]
+-
+-        arch = flat.get('arch')
+-        if arch:
+-            t_item['arch'] = arch
+-            props['architecture'] = canonicalize_arch(arch)
+-
+-        if self.config.get('hypervisor_mapping', False) and 'ftype' in flat:
+-            _hypervisor_type = hypervisor_type(flat['ftype'])
++    def create_glance_properties(self, content_id, source_content_id,
++                                 image_metadata, hypervisor_mapping):
++        """
++        Construct extra properties to store in Glance for an image.
++
++        Based on source image metadata.
++        """
++        properties = {
++            'content_id': content_id,
++            'source_content_id': source_content_id,
++        }
++        # An iterator of properties to carry over: if a property needs
++        # renaming, uses a tuple (old name, new name).
++        carry_over = (
++            'product_name', 'version_name', 'item_name',
++            ('os', 'os_distro'), ('version', 'os_version'),
++        )
++        for carry_over_property in carry_over:
++            if isinstance(carry_over_property, tuple):
++                name_old, name_new = carry_over_property
++            else:
++                name_old = name_new = carry_over_property
++            properties[name_new] = image_metadata[name_old]
++
++        if 'arch' in image_metadata:
++            properties['architecture'] = canonicalize_arch(
++                image_metadata['arch'])
++
++        if hypervisor_mapping and 'ftype' in image_metadata:
++            _hypervisor_type = hypervisor_type(image_metadata['ftype'])
+             if _hypervisor_type:
+-                props['hypervisor_type'] = _hypervisor_type
+-                _virt_type = virt_type(_hypervisor_type)
+-                if _virt_type:
+-                    t_item['virt'] = _virt_type
+-
+-        if 'os' in flat:
+-            props['os_distro'] = flat['os']
+-
+-        if 'version' in flat:
+-            props['os_version'] = flat['version']
+-
+-        fullname = self.name_prefix + name
++                properties['hypervisor_type'] = _hypervisor_type
++        return properties
++
++    def prepare_glance_arguments(self, full_image_name, image_metadata,
++                                 image_md5_hash, image_size, image_properties):
++        """
++        Prepare arguments to pass into Glance image creation method.
++
++        Uses `image_metadata` for source image to derive image size, md5 hash,
++        disk format (based on 'ftype' field, if defined, otherwise defaults to
++        'qcow2').
++
++        If `image_md5_hash` and `image_size` are defined, overrides the
++        values from image_metadata with their values.
++
++        Sets extra image properties to dict `image_properties`.
++
++        Returns a dict to use as keyword arguments passed directly to
++        GlanceClient.images.create().
++        """
+         create_kwargs = {
+-            'name': fullname,
+-            'properties': props,
++            'name': full_image_name,
+             'container_format': 'bare',
+             'is_public': True,
++            'properties': image_properties,
+         }
+-        if 'size' in data:
+-            create_kwargs['size'] = data.get('size')
+-
+-        if 'md5' in data:
+-            create_kwargs['checksum'] = data.get('md5')
+-
+-        if 'ftype' in flat:
++
++        if 'size' in image_metadata:
++            create_kwargs['size'] = image_metadata.get('size')
++        if 'md5' in image_metadata:
++            create_kwargs['checksum'] = image_metadata.get('md5')
++        if image_md5_hash and image_size:
++            create_kwargs.update({
++                'checksum': image_md5_hash,
++                'size': image_size,
++            })
++
++        if 'ftype' in image_metadata:
+             create_kwargs['disk_format'] = (
+-                disk_format(flat['ftype']) or 'qcow2'
++                disk_format(image_metadata['ftype']) or 'qcow2'
+             )
+         else:
+             create_kwargs['disk_format'] = 'qcow2'
+ 
++        return create_kwargs
++
++    def download_image(self, contentsource, image_stream_data):
++        """
++        Download an image from contentsource.
++
++        `image_stream_data` represents a flattened image metadata structure
++        to use for any logging messages.
++
++        Returns a tuple of (local-image-path, image-size, image-md5-hash).
++
++        If download fails, these values will all be None.
++        """
++        tmp_path = new_size = new_md5 = None
++
++        image_name = image_stream_data.get('pubname')
++        image_size = image_stream_data.get('size')
++
+         if self.progress_callback:
+             def progress_wrapper(written):
+                 self.progress_callback(dict(status="Downloading",
+-                                            name=flat.get('pubname'),
+-                                            size=data.get('size', 0),
++                                            name=image_name,
++                                            size=image_size,
+                                             written=written))
+         else:
+             def progress_wrapper(written):
+                 pass
+ 
+         try:
+-            try:
+-                (tmp_path, tmp_del) = util.get_local_copy(
+-                    contentsource, progress_callback=progress_wrapper)
+-
+-                if self.modify_hook:
+-                    (newsize, newmd5) = call_hook(item=t_item, path=tmp_path,
+-                                                  cmd=self.modify_hook)
+-                    create_kwargs['checksum'] = newmd5
+-                    create_kwargs['size'] = newsize
+-                    t_item['md5'] = newmd5
+-                    t_item['size'] = newsize
+-
+-            finally:
+-                contentsource.close()
+-
++            tmp_path, _ = util.get_local_copy(
++                contentsource, progress_callback=progress_wrapper)
++
++            if self.modify_hook:
++                (new_size, new_md5) = call_hook(
++                    item=image_stream_data, path=tmp_path,
++                    cmd=self.modify_hook)
++        finally:
++            contentsource.close()
++
++        return tmp_path, new_size, new_md5
++
++    def adapt_source_entry(self, source_entry, hypervisor_mapping, image_name,
++                           image_md5_hash, image_size):
++        """
++        Adapts the source simplestreams dict `source_entry` for use in the
++        generated local simplestreams index.
++        """
++        output_entry = source_entry.copy()
++
++        # Drop attributes not needed for the simplestreams index itself.
++        for property_name in ('path', 'product_name', 'version_name',
++                              'item_name'):
++            if property_name in output_entry:
++                del output_entry[property_name]
++
++        if hypervisor_mapping and 'ftype' in output_entry:
++            _hypervisor_type = hypervisor_type(output_entry['ftype'])
++            if _hypervisor_type:
++                _virt_type = virt_type(_hypervisor_type)
++                if _virt_type:
++                    output_entry['virt'] = _virt_type
++
++        output_entry['region'] = self.region
++        output_entry['endpoint'] = self.auth_url
++        output_entry['owner_id'] = self.tenant_id
++
++        output_entry['name'] = image_name
++        if image_md5_hash and image_size:
++            output_entry['md5'] = image_md5_hash
++            output_entry['size'] = image_size
++
++        return output_entry
++
++    def insert_item(self, data, src, target, pedigree, contentsource):
++        """
++        Upload image into glance and add image metadata to simplestreams index.
++
++        `data` is the metadata for a particular image file from the source:
++            unused since all that data is present in the `src` entry for
++            the corresponding image as well.
++        `src` contains the entire simplestreams index from the image syncing
++            source.
++        `target` is the simplestreams index for currently available images
++            in glance (generated by load_products()) to add this item to.
++        `pedigree` is a "path" to get to the `data` for the image we desire,
++            a tuple of (product_name, version_name, image_type).
++        `contentsource` is a ContentSource to download the actual image data
++            from.
++        """
++        # Extract and flatten metadata for a product image matching
++        #   (product-name, version-name, image-type)
++        # from the tuple `pedigree` in the source simplestreams index.
++        flattened_img_data = util.products_exdata(
++            src, pedigree, include_top=False)
++
++        tmp_path = None
++
++        full_image_name = "{}{}".format(
++            self.name_prefix,
++            flattened_img_data.get('pubname', flattened_img_data.get('name')))
++        if not full_image_name.endswith(flattened_img_data['item_name']):
++            full_image_name += "-{}".format(flattened_img_data['item_name'])
++
++        # Download images locally into a temporary file.
++        tmp_path, new_size, new_md5 = self.download_image(
++            contentsource, flattened_img_data)
++
++        hypervisor_mapping = self.config.get('hypervisor_mapping', False)
++
++        glance_props = self.create_glance_properties(
++            target['content_id'], src['content_id'], flattened_img_data,
++            hypervisor_mapping)
++        create_kwargs = self.prepare_glance_arguments(
++            full_image_name, flattened_img_data, new_md5, new_size,
++            glance_props)
++
++        target_sstream_item = self.adapt_source_entry(
++            flattened_img_data, hypervisor_mapping, full_image_name, new_md5,
++            new_size)
++
++        try:
+             create_kwargs['data'] = open(tmp_path, 'rb')
+-            ret = self.gclient.images.create(**create_kwargs)
+-            t_item['id'] = ret.id
+-            print("created %s: %s" % (ret.id, fullname))
++            glance_image = self.gclient.images.create(**create_kwargs)
++            target_sstream_item['id'] = glance_image.id
++            print("created %s: %s" % (glance_image.id, full_image_name))
+ 
+         finally:
+-            if tmp_del and os.path.exists(tmp_path):
++            if tmp_path and os.path.exists(tmp_path):
+                 os.unlink(tmp_path)
+ 
+-        t_item['region'] = self.region
+-        t_item['endpoint'] = self.auth_url
+-        t_item['owner_id'] = self.tenant_id
+-        t_item['name'] = fullname
+-        util.products_set(target, t_item, pedigree)
++        util.products_set(target, target_sstream_item, pedigree)
+ 
+     def remove_item(self, data, src, target, pedigree):
+         util.products_del(target, pedigree)
+
+=== added file 'tests/unittests/test_glancemirror.py'
+--- a/tests/unittests/test_glancemirror.py	1970-01-01 00:00:00 +0000
++++ b/tests/unittests/test_glancemirror.py	2016-06-10 18:47:52 +0000
+@@ -0,0 +1,479 @@
++from simplestreams.contentsource import MemoryContentSource
++from simplestreams.mirrors.glance import GlanceMirror
++import simplestreams.util
++
++import os
++from unittest import TestCase
++
++
++class FakeOpenstack(object):
++    """Fake 'openstack' module replacement for testing GlanceMirror."""
++    def load_keystone_creds(self):
++        return {"auth_url": "http://keystone/api/"}
++
++    def get_service_conn_info(self, url, region_name=None, auth_url=None):
++        return {"endpoint": "http://objectstore/api/",
++                "tenant_id": "bar456"}
++
++
++class FakeImage(object):
++    """Fake image objects returned by GlanceClient.images.create()."""
++    def __init__(self, identifier):
++        self.id = identifier
++
++
++class FakeImages(object):
++    """Fake GlanceClient.images implementation to track create() calls."""
++    def __init__(self):
++        self.create_calls = []
++
++    def create(self, **kwargs):
++        self.create_calls.append(kwargs)
++        return FakeImage('image-%d' % len(self.create_calls))
++
++
++class FakeGlanceClient(object):
++    """Fake GlanceClient implementation to track images.create() calls."""
++    def __init__(self, *args):
++        self.images = FakeImages()
++
++
++class TestGlanceMirror(TestCase):
++    """Tests for GlanceMirror methods."""
++
++    def setUp(self):
++        self.config = {"content_id": "foo123"}
++        self.mirror = GlanceMirror(
++            self.config, name_prefix="auto-sync/", region="region1",
++            client=FakeOpenstack())
++
++    def test_adapt_source_entry(self):
++        # Adapts source entry for use in a local simplestreams index.
++        source_entry = {"source-key": "source-value"}
++        output_entry = self.mirror.adapt_source_entry(
++            source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
++            image_md5_hash=None, image_size=None)
++
++        # Source and output entry are different objects.
++        self.assertNotEqual(source_entry, output_entry)
++
++        # Output entry gets a few new properties like the endpoint and
++        # owner_id taken from the GlanceMirror and  OpenStack configuration,
++        # region from the value passed into GlanceMirror constructor, and
++        # image name from the passed in value.
++        # It also contains the source entries as well.
++        self.assertEqual(
++            {"endpoint": "http://keystone/api/",
++             "name": "foobuntu-X",
++             "owner_id": "bar456",
++             "region": "region1",
++             "source-key": "source-value"},
++            output_entry)
++
++    def test_adapt_source_entry_ignored_properties(self):
++        # adapt_source_entry() drops some properties from the source entry.
++        source_entry = {"path": "foo",
++                        "product_name": "bar",
++                        "version_name": "baz",
++                        "item_name": "bah"}
++        output_entry = self.mirror.adapt_source_entry(
++            source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
++            image_md5_hash=None, image_size=None)
++
++        # None of the values in 'source_entry' are preserved.
++        for key in ("path", "product_name", "version_name", "item"):
++            self.assertNotIn("path", output_entry)
++
++    def test_adapt_source_entry_image_md5_and_size(self):
++        # adapt_source_entry() will use passed in values for md5 and size.
++        # Even old stale values will be overridden when image_md5_hash and
++        # image_size are passed in.
++        source_entry = {"md5": "stale-md5"}
++        output_entry = self.mirror.adapt_source_entry(
++            source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
++            image_md5_hash="new-md5", image_size=5)
++
++        self.assertEqual("new-md5", output_entry["md5"])
++        self.assertEqual(5, output_entry["size"])
++
++    def test_adapt_source_entry_image_md5_and_size_both_required(self):
++        # adapt_source_entry() requires both md5 and size to not ignore them.
++
++        source_entry = {"md5": "stale-md5"}
++
++        # image_size is not passed in, so md5 value is not used either.
++        output_entry1 = self.mirror.adapt_source_entry(
++            source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
++            image_md5_hash="new-md5", image_size=None)
++        self.assertEqual("stale-md5", output_entry1["md5"])
++        self.assertNotIn("size", output_entry1)
++
++        # image_md5_hash is not passed in, so image_size is not used either.
++        output_entry2 = self.mirror.adapt_source_entry(
++            source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
++            image_md5_hash=None, image_size=5)
++        self.assertEqual("stale-md5", output_entry2["md5"])
++        self.assertNotIn("size", output_entry2)
++
++    def test_adapt_source_entry_hypervisor_mapping(self):
++        # If hypervisor_mapping is set to True, 'virt' value is derived from
++        # the source entry 'ftype'.
++        source_entry = {"ftype": "disk1.img"}
++        output_entry = self.mirror.adapt_source_entry(
++            source_entry, hypervisor_mapping=True, image_name="foobuntu-X",
++            image_md5_hash=None, image_size=None)
++
++        self.assertEqual("kvm", output_entry["virt"])
++
++    def test_adapt_source_entry_hypervisor_mapping_ftype_required(self):
++        # If hypervisor_mapping is set to True, but 'ftype' is missing in the
++        # source entry, 'virt' value is not added to the returned entry.
++        source_entry = {}
++        output_entry = self.mirror.adapt_source_entry(
++            source_entry, hypervisor_mapping=True, image_name="foobuntu-X",
++            image_md5_hash=None, image_size=None)
++
++        self.assertNotIn("virt", output_entry)
++
++    def test_create_glance_properties(self):
++        # Constructs glance properties to set on image during upload
++        # based on source image metadata.
++        source_entry = {
++            # All of these are carried over and potentially re-named.
++            "product_name": "foobuntu",
++            "version_name": "X",
++            "item_name": "disk1.img",
++            "os": "ubuntu",
++            "version": "16.04",
++            # Other entries are ignored.
++            "something-else": "ignored",
++        }
++        properties = self.mirror.create_glance_properties(
++            "content-1", "source-1", source_entry, hypervisor_mapping=False)
++
++        # Output properties contain content-id and source-content-id based
++        # on the passed in parameters, and carry over (with changed keys
++        # for "os" and "version") product_name, version_name, item_name and
++        # os and version values from the source entry.
++        self.assertEqual(
++            {"content_id": "content-1",
++             "source_content_id": "source-1",
++             "product_name": "foobuntu",
++             "version_name": "X",
++             "item_name": "disk1.img",
++             "os_distro": "ubuntu",
++             "os_version": "16.04"},
++            properties)
++
++    def test_create_glance_properties_arch(self):
++        # When 'arch' is present in the source entry, it is adapted and
++        # returned inside 'architecture' field.
++        source_entry = {
++            "product_name": "foobuntu",
++            "version_name": "X",
++            "item_name": "disk1.img",
++            "os": "ubuntu",
++            "version": "16.04",
++            "arch": "amd64",
++        }
++        properties = self.mirror.create_glance_properties(
++            "content-1", "source-1", source_entry, hypervisor_mapping=False)
++        self.assertEqual("x86_64", properties["architecture"])
++
++    def test_create_glance_properties_hypervisor_mapping(self):
++        # When hypervisor_mapping is requested and 'ftype' is present in
++        # the image metadata, 'hypervisor_type' is added to returned
++        # properties.
++        source_entry = {
++            "product_name": "foobuntu",
++            "version_name": "X",
++            "item_name": "disk1.img",
++            "os": "ubuntu",
++            "version": "16.04",
++            "ftype": "root.tar.gz",
++        }
++        properties = self.mirror.create_glance_properties(
++            "content-1", "source-1", source_entry, hypervisor_mapping=True)
++        self.assertEqual("lxc", properties["hypervisor_type"])
++
++    def test_prepare_glance_arguments(self):
++        # Prepares arguments to pass to GlanceClient.images.create()
++        # based on image metadata from the simplestreams source.
++        source_entry = {}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
++            image_properties=None)
++
++        # Arguments to always pass in contain the image name, container format,
++        # disk format, whether image is public, and any passed-in properties.
++        self.assertEqual(
++            {"name": "foobuntu-X",
++             "container_format": 'bare',
++             "disk_format": "qcow2",
++             "is_public": True,
++             "properties": None},
++            create_arguments)
++
++    def test_prepare_glance_arguments_disk_format(self):
++        # Disk format is based on the image 'ftype' (if defined).
++        source_entry = {"ftype": "root.tar.gz"}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
++            image_properties=None)
++
++        self.assertEqual("root-tar", create_arguments["disk_format"])
++
++    def test_prepare_glance_arguments_size(self):
++        # Size is read from image metadata if defined.
++        source_entry = {"size": 5}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
++            image_properties=None)
++
++        self.assertEqual(5, create_arguments["size"])
++
++    def test_prepare_glance_arguments_checksum(self):
++        # Checksum is based on the source entry 'md5' value, if defined.
++        source_entry = {"md5": "foo123"}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
++            image_properties=None)
++
++        self.assertEqual("foo123", create_arguments["checksum"])
++
++    def test_prepare_glance_arguments_size_and_md5_override(self):
++        # Size and md5 hash are overridden from the passed-in values even if
++        # defined on the source entry.
++        source_entry = {"size": 5, "md5": "foo123"}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash="bar456", image_size=10,
++            image_properties=None)
++
++        self.assertEqual(10, create_arguments["size"])
++        self.assertEqual("bar456", create_arguments["checksum"])
++
++    def test_prepare_glance_arguments_size_and_md5_no_override_hash(self):
++        # If only one of image_md5_hash or image_size is passed directly in,
++        # the other value is not overridden either.
++        source_entry = {"size": 5, "md5": "foo123"}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash="bar456",
++            image_size=None, image_properties=None)
++
++        self.assertEqual(5, create_arguments["size"])
++        self.assertEqual("foo123", create_arguments["checksum"])
++
++    def test_prepare_glance_arguments_size_and_md5_no_override_size(self):
++        # If only one of image_md5_hash or image_size is passed directly in,
++        # the other value is not overridden either.
++        source_entry = {"size": 5, "md5": "foo123"}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash=None, image_size=10,
++            image_properties=None)
++
++        self.assertEqual(5, create_arguments["size"])
++        self.assertEqual("foo123", create_arguments["checksum"])
++
++    def test_download_image(self):
++        # Downloads image from a contentsource.
++        content = "foo bazes the bar"
++        content_source = MemoryContentSource(
++            url="http://image-store/fooubuntu-X-disk1.img", content=content)
++        image_metadata = {"pubname": "foobuntu-X", "size": 5}
++        path, size, md5_hash = self.mirror.download_image(
++            content_source, image_metadata)
++        self.addCleanup(os.unlink, path)
++        self.assertIsNotNone(path)
++        self.assertIsNone(size)
++        self.assertIsNone(md5_hash)
++
++    def test_download_image_progress_callback(self):
++        # Progress callback is called with image name, size, status and buffer
++        # size after every 10kb of data: 3 times for 25kb of data below.
++        content = "abcdefghij" * int(1024 * 2.5)
++        content_source = MemoryContentSource(
++            url="http://image-store/fooubuntu-X-disk1.img", content=content)
++        image_metadata = {"pubname": "foobuntu-X", "size": len(content)}
++
++        self.progress_calls = []
++
++        def log_progress_calls(message):
++            self.progress_calls.append(message)
++
++        self.addCleanup(
++            setattr, self.mirror, "progress_callback",
++            self.mirror.progress_callback)
++        self.mirror.progress_callback = log_progress_calls
++        path, size, md5_hash = self.mirror.download_image(
++            content_source, image_metadata)
++        self.addCleanup(os.unlink, path)
++
++        self.assertEqual(
++            [{"name": "foobuntu-X", "size": 25600, "status": "Downloading",
++              "written": 10240}] * 3,
++            self.progress_calls)
++
++    def test_download_image_error(self):
++        # When there's an error during download, contentsource is still closed
++        # and the error is propagated below.
++        content = "abcdefghij"
++        content_source = MemoryContentSource(
++            url="http://image-store/fooubuntu-X-disk1.img", content=content)
++        image_metadata = {"pubname": "foobuntu-X", "size": len(content)}
++
++        # MemoryContentSource has an internal file descriptor which indicates
++        # if close() method has been called on it.
++        self.assertFalse(content_source.fd.closed)
++
++        self.addCleanup(
++            setattr, self.mirror, "progress_callback",
++            self.mirror.progress_callback)
++        self.mirror.progress_callback = lambda message: 1/0
++
++        self.assertRaises(
++            ZeroDivisionError,
++            self.mirror.download_image, content_source, image_metadata)
++
++        # We rely on the MemoryContentSource.close() side-effect to ensure
++        # close() method has indeed been called on the passed-in ContentSource.
++        self.assertTrue(content_source.fd.closed)
++
++    def test_insert_item(self):
++        # Downloads an image from a contentsource, uploads it into Glance,
++        # adapting and munging as needed (it updates the keystone endpoint,
++        # image and owner ids).
++        # This test is basically an integration test to make sure all the
++        # methods used by insert_item() are tied together in one good
++        # fully functioning whole.
++
++        # This is a real snippet from the simplestreams index entry for
++        # Ubuntu 14.04 amd64 image from cloud-images.ubuntu.com as of
++        # 2016-06-05.
++        source_index = {
++            u'content_id': u'com.ubuntu.cloud:released:download',
++            u'datatype': u'image-downloads',
++            u'format': u'products:1.0',
++            u'license': (u'http://www.canonical.com/'
++                         u'intellectual-property-policy'),
++            u'products': {u'com.ubuntu.cloud:server:14.04:amd64': {
++                u'aliases': u'14.04,default,lts,t,trusty',
++                u'arch': u'amd64',
++                u'os': u'ubuntu',
++                u'release': u'trusty',
++                u'release_codename': u'Trusty Tahr',
++                u'release_title': u'14.04 LTS',
++                u'support_eol': u'2019-04-17',
++                u'supported': True,
++                u'version': u'14.04',
++                u'versions': {u'20160602': {
++                    u'items': {u'disk1.img': {
++                        u'ftype': u'disk1.img',
++                        u'md5': u'e5436cd36ae6cc298f081bf0f6b413f1',
++                        u'path': (
++                            u'server/releases/trusty/release-20160602/'
++                            u'ubuntu-14.04-server-cloudimg-amd64-disk1.img'),
++                        u'sha256': (u'5b982d7d4dd1a03e88ae5f35f02ed44f'
++                                    u'579e2711f3e0f27ea2bff20aef8c8d9e'),
++                        u'size': 259850752}},
++                    u'label': u'release',
++                    u'pubname': u'ubuntu-trusty-14.04-amd64-server-20160602',
++                }}}
++            }
++        }
++
++        # "Pedigree" is basically a "path" to get to the image data in
++        # simplestreams index, going through "products", their "versions",
++        # and nested "items".
++        pedigree = (
++            u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
++        product = source_index[u'products'][pedigree[0]]
++        image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
++
++        content_source = MemoryContentSource(
++            url="http://image-store/fooubuntu-X-disk1.img",
++            content="image-data")
++
++        # Use a fake GlanceClient to track arguments passed into
++        # GlanceClient.images.create().
++        self.addCleanup(setattr, self.mirror, "gclient", self.mirror.gclient)
++        self.mirror.gclient = FakeGlanceClient()
++
++        target = {
++            'content_id': 'auto.sync',
++            'datatype': 'image-ids',
++            'format': 'products:1.0',
++        }
++
++        self.mirror.insert_item(
++            image_data, source_index, target, pedigree, content_source)
++
++        passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
++
++        # There is a 'data' argument pointing to an open file descriptor
++        # for the locally downloaded image.
++        self.assertIn("data", passed_create_kwargs)
++        passed_create_kwargs.pop("data")
++
++        expected_create_kwargs = {
++            'name': ('auto-sync/'
++                     'ubuntu-trusty-14.04-amd64-server-20160602-disk1.img'),
++            'checksum': u'e5436cd36ae6cc298f081bf0f6b413f1',
++            'disk_format': 'qcow2',
++            'container_format': 'bare',
++            'is_public': True,
++            'properties': {
++                'os_distro': u'ubuntu',
++                'item_name': u'disk1.img',
++                'os_version': u'14.04',
++                'architecture': 'x86_64',
++                'version_name': u'20160602',
++                'content_id': 'auto.sync',
++                'product_name': u'com.ubuntu.cloud:server:14.04:amd64',
++                'source_content_id': u'com.ubuntu.cloud:released:download'},
++            'size': '259850752'}
++        self.assertEqual(
++            expected_create_kwargs, passed_create_kwargs)
++
++        # Almost real resulting data as produced by simplestreams before
++        # insert_item refactoring to allow for finer-grained testing.
++        expected_target_index = {
++            'content_id': 'auto.sync',
++            'datatype': 'image-ids',
++            'format': 'products:1.0',
++            'products': {
++                "com.ubuntu.cloud:server:14.04:amd64": {
++                    "aliases": "14.04,default,lts,t,trusty",
++                    "arch": "amd64",
++                    "label": "release",
++                    "os": "ubuntu",
++                    "owner_id": "bar456",
++                    "pubname": "ubuntu-trusty-14.04-amd64-server-20160602",
++                    "release": "trusty",
++                    "release_codename": "Trusty Tahr",
++                    "release_title": "14.04 LTS",
++                    "support_eol": "2019-04-17",
++                    "supported": "True",
++                    "version": "14.04",
++                    "versions": {"20160602": {"items": {"disk1.img": {
++                        "endpoint": "http://keystone/api/",
++                        "ftype": "disk1.img",
++                        "id": "image-1",
++                        "md5": "e5436cd36ae6cc298f081bf0f6b413f1",
++                        "name": ("auto-sync/ubuntu-trusty-14.04-amd64-"
++                                 "server-20160602-disk1.img"),
++                        "region": "region1",
++                        "sha256": ("5b982d7d4dd1a03e88ae5f35f02ed44f"
++                                   "579e2711f3e0f27ea2bff20aef8c8d9e"),
++                        "size": "259850752"
++                    }}}}
++                }
++            }
++        }
++
++        # Apply the condensing as done in GlanceMirror.insert_products()
++        # to ensure we compare with the desired resulting simplestreams data.
++        sticky = ['ftype', 'md5', 'sha256', 'size', 'name', 'id', 'endpoint',
++                  'region']
++        simplestreams.util.products_condense(target, sticky)
++
++        self.assertEqual(expected_target_index, target)
+
diff --git a/debian/patches/436-glance-fix-race-conditions.patch b/debian/patches/436-glance-fix-race-conditions.patch
 new file mode 100644
index 0000000..ae55dbe
--- /dev/null
+++ b/debian/patches/436-glance-fix-race-conditions.patch
@@ -0,0 +1,479 @@
+------------------------------------------------------------
+revno: 436 [merge]
+fixes bug: https://launchpad.net/bugs/1584938
+committer: Scott Moser <[e-mail]>
+branch nick: trunk
+timestamp: Tue 2016-06-14 16:21:05 -0400
+message:
+  GlanceMirror: fix a couple of race-related problems
+  
+  First, what can happen is that .sync() is interrupted for external reasons
+  (service restarts, network issues,...) after first image has been uploaded
+  to Glance when syncing multiple images. On re-run, the uploaded image is
+  considered "synced" already, but since simplestreams index is written out
+  only at the end of the sync, all the metadata is lost. We solve this by
+  adding "simplestreams_metadata" as one of the properties of image in
+  Glance where we put a JSON representation of all the metadata fields from
+  the original simplestreams index entry. We then load them in
+  load_products() if they are defined. This results in a slightly different
+  auto.sync.json entry: "endpoint" and "region" are on the product entry,
+  instead of on the disk1.img entry.
+  
+  Secondly, if sync is indeed interrupted after one or several images are
+  completely uploaded (but not all of them), there's no index file at all,
+  so any attempt to make use of them with juju would fail, even though it
+  should be fully functional. We solve this by calling "insert_products()"
+  at the end of the insert_item(). This means that the index is regenerated
+  after every single image is uploaded.
+  
+  Along the way, I slightly improve insert_item() tests from the pre-req
+  branch to add a test for minimal data required, rename the existing one
+  using real world data to test_insert_item_full, and re-use that data for a
+  test for a newly added call to insert_products().
+------------------------------------------------------------
+Use --include-merged or -n0 to see merged revisions.
+=== modified file 'simplestreams/mirrors/glance.py'
+--- a/simplestreams/mirrors/glance.py	2016-06-10 18:47:52 +0000
++++ b/simplestreams/mirrors/glance.py	2016-06-14 20:21:05 +0000
+@@ -25,6 +25,7 @@
+ import copy
+ import errno
+ import glanceclient
++import json
+ import os
+ import re
+ 
+@@ -219,6 +220,15 @@
+             except KeyError:
+                 item_data = {}
+ 
++            # If original simplestreams-metadata is stored on the image,
++            # use that as well.
++            if 'simplestreams_metadata' in props:
++                simplestreams_metadata = json.loads(
++                    props.get('simplestreams_metadata'))
++            else:
++                simplestreams_metadata = {}
++            item_data.update(simplestreams_metadata)
++
+             item_data.update({'name': image['name'], 'id': image['id']})
+             if 'owner_id' not in item_data:
+                 item_data['owner_id'] = self.tenant_id
+@@ -248,10 +258,10 @@
+         }
+         # An iterator of properties to carry over: if a property needs
+         # renaming, uses a tuple (old name, new name).
+-        carry_over = (
+-            'product_name', 'version_name', 'item_name',
+-            ('os', 'os_distro'), ('version', 'os_version'),
+-        )
++        carry_over_simple = (
++            'product_name', 'version_name', 'item_name')
++        carry_over = carry_over_simple + (
++            ('os', 'os_distro'), ('version', 'os_version'))
+         for carry_over_property in carry_over:
+             if isinstance(carry_over_property, tuple):
+                 name_old, name_new = carry_over_property
+@@ -267,6 +277,16 @@
+             _hypervisor_type = hypervisor_type(image_metadata['ftype'])
+             if _hypervisor_type:
+                 properties['hypervisor_type'] = _hypervisor_type
++
++        # Store flattened metadata for a source image along with the
++        # image in 'simplestreams_metadata' property.
++        simplestreams_metadata = image_metadata.copy()
++        drop_keys = carry_over_simple + ('path',)
++        for remove_key in drop_keys:
++            if remove_key in simplestreams_metadata:
++                del simplestreams_metadata[remove_key]
++        properties['simplestreams_metadata'] = json.dumps(
++            simplestreams_metadata, sort_keys=True)
+         return properties
+ 
+     def prepare_glance_arguments(self, full_image_name, image_metadata,
+@@ -441,6 +461,9 @@
+                 os.unlink(tmp_path)
+ 
+         util.products_set(target, target_sstream_item, pedigree)
++        # We can safely ignore path and content arguments since they are
++        # unused in insert_products below.
++        self.insert_products(None, target, None)
+ 
+     def remove_item(self, data, src, target, pedigree):
+         util.products_del(target, pedigree)
+
+=== modified file 'tests/unittests/test_glancemirror.py'
+--- a/tests/unittests/test_glancemirror.py	2016-06-10 18:47:52 +0000
++++ b/tests/unittests/test_glancemirror.py	2016-06-14 20:21:05 +0000
+@@ -1,11 +1,91 @@
+ from simplestreams.contentsource import MemoryContentSource
+ from simplestreams.mirrors.glance import GlanceMirror
++from simplestreams.objectstores import MemoryObjectStore
+ import simplestreams.util
+ 
++import copy
++import json
+ import os
+ from unittest import TestCase
+ 
+ 
++# This is a real snippet from the simplestreams index entry for
++# Ubuntu 14.04 amd64 image from cloud-images.ubuntu.com as of
++# 2016-06-05.
++TEST_SOURCE_INDEX_ENTRY = {
++    u'content_id': u'com.ubuntu.cloud:released:download',
++    u'datatype': u'image-downloads',
++    u'format': u'products:1.0',
++    u'license': (u'http://www.canonical.com/'
++                 u'intellectual-property-policy'),
++    u'products': {u'com.ubuntu.cloud:server:14.04:amd64': {
++        u'aliases': u'14.04,default,lts,t,trusty',
++        u'arch': u'amd64',
++        u'os': u'ubuntu',
++        u'release': u'trusty',
++        u'release_codename': u'Trusty Tahr',
++        u'release_title': u'14.04 LTS',
++        u'support_eol': u'2019-04-17',
++        u'supported': True,
++        u'version': u'14.04',
++        u'versions': {u'20160602': {
++            u'items': {u'disk1.img': {
++                u'ftype': u'disk1.img',
++                u'md5': u'e5436cd36ae6cc298f081bf0f6b413f1',
++                u'path': (
++                    u'server/releases/trusty/release-20160602/'
++                    u'ubuntu-14.04-server-cloudimg-amd64-disk1.img'),
++                u'sha256': (u'5b982d7d4dd1a03e88ae5f35f02ed44f'
++                            u'579e2711f3e0f27ea2bff20aef8c8d9e'),
++                u'size': 259850752}},
++            u'label': u'release',
++            u'pubname': u'ubuntu-trusty-14.04-amd64-server-20160602',
++        }}}
++    }
++}
++
++# "Pedigree" is basically a "path" to get to the image data in simplestreams
++# index, going through "products", their "versions", and nested "items".
++TEST_IMAGE_PEDIGREE = (
++    u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
++
++# Almost real resulting data as produced by simplestreams before
++# insert_item refactoring to allow for finer-grained testing.
++EXPECTED_OUTPUT_INDEX = {
++    u'content_id': u'auto.sync',
++    u'datatype': u'image-ids',
++    u'format': u'products:1.0',
++    u'products': {
++        u"com.ubuntu.cloud:server:14.04:amd64": {
++            u"aliases": u"14.04,default,lts,t,trusty",
++            u"arch": u"amd64",
++            u"label": u"release",
++            u"os": u"ubuntu",
++            u"owner_id": u"bar456",
++            u"pubname": u"ubuntu-trusty-14.04-amd64-server-20160602",
++            u"release": u"trusty",
++            u"release_codename": u"Trusty Tahr",
++            u"release_title": u"14.04 LTS",
++            u"support_eol": u"2019-04-17",
++            u"supported": u"True",
++            u"version": u"14.04",
++            u"versions": {u"20160602": {u"items": {u"disk1.img": {
++                u"endpoint": u"http://keystone/api/",
++                u"ftype": u"disk1.img",
++                u"id": u"image-1",
++                u"md5": u"e5436cd36ae6cc298f081bf0f6b413f1",
++                u"name": (u"auto-sync/ubuntu-trusty-14.04-amd64-"
++                          u"server-20160602-disk1.img"),
++                u"region": u"region1",
++                u"sha256": (u"5b982d7d4dd1a03e88ae5f35f02ed44f"
++                            u"579e2711f3e0f27ea2bff20aef8c8d9e"),
++                u"size": u"259850752"
++            }}}}
++        }
++    }
++}
++
++
+ class FakeOpenstack(object):
+     """Fake 'openstack' module replacement for testing GlanceMirror."""
+     def load_keystone_creds(self):
+@@ -145,8 +225,8 @@
+             "item_name": "disk1.img",
+             "os": "ubuntu",
+             "version": "16.04",
+-            # Other entries are ignored.
+-            "something-else": "ignored",
++            # Unknown entries are stored in 'simplestreams_metadata'.
++            "extra": "value",
+         }
+         properties = self.mirror.create_glance_properties(
+             "content-1", "source-1", source_entry, hypervisor_mapping=False)
+@@ -155,6 +235,8 @@
+         # on the passed in parameters, and carry over (with changed keys
+         # for "os" and "version") product_name, version_name, item_name and
+         # os and version values from the source entry.
++        # All the fields except product_name, version_name and item_name are
++        # also stored inside 'simplestreams_metadata' property as JSON data.
+         self.assertEqual(
+             {"content_id": "content-1",
+              "source_content_id": "source-1",
+@@ -162,7 +244,9 @@
+              "version_name": "X",
+              "item_name": "disk1.img",
+              "os_distro": "ubuntu",
+-             "os_version": "16.04"},
++             "os_version": "16.04",
++             "simplestreams_metadata": (
++                 '{"extra": "value", "os": "ubuntu", "version": "16.04"}')},
+             properties)
+ 
+     def test_create_glance_properties_arch(self):
+@@ -196,6 +280,26 @@
+             "content-1", "source-1", source_entry, hypervisor_mapping=True)
+         self.assertEqual("lxc", properties["hypervisor_type"])
+ 
++    def test_create_glance_properties_simplestreams_no_path(self):
++        # Other than 'product_name', 'version_name' and 'item_name', if 'path'
++        # is defined on the source entry, it is also not saved inside the
++        # 'simplestreams_metadata' property.
++        source_entry = {
++            "product_name": "foobuntu",
++            "version_name": "X",
++            "item_name": "disk1.img",
++            "os": "ubuntu",
++            "version": "16.04",
++            "path": "/path/to/foo",
++        }
++        properties = self.mirror.create_glance_properties(
++            "content-1", "source-1", source_entry, hypervisor_mapping=False)
++
++        # Path is omitted from the simplestreams_metadata property JSON.
++        self.assertEqual(
++            '{"os": "ubuntu", "version": "16.04"}',
++            properties["simplestreams_metadata"])
++
+     def test_prepare_glance_arguments(self):
+         # Prepares arguments to pass to GlanceClient.images.create()
+         # based on image metadata from the simplestreams source.
+@@ -342,45 +446,86 @@
+         # Downloads an image from a contentsource, uploads it into Glance,
+         # adapting and munging as needed (it updates the keystone endpoint,
+         # image and owner ids).
+-        # This test is basically an integration test to make sure all the
+-        # methods used by insert_item() are tied together in one good
+-        # fully functioning whole.
+ 
+-        # This is a real snippet from the simplestreams index entry for
+-        # Ubuntu 14.04 amd64 image from cloud-images.ubuntu.com as of
+-        # 2016-06-05.
++        # We use a minimal source simplestreams index, fake ContentSource and
++        # GlanceClient, and only test for side-effects of each of the
++        # subparts of the insert_item method.
+         source_index = {
+             u'content_id': u'com.ubuntu.cloud:released:download',
+-            u'datatype': u'image-downloads',
+-            u'format': u'products:1.0',
+-            u'license': (u'http://www.canonical.com/'
+-                         u'intellectual-property-policy'),
+             u'products': {u'com.ubuntu.cloud:server:14.04:amd64': {
+-                u'aliases': u'14.04,default,lts,t,trusty',
+                 u'arch': u'amd64',
+                 u'os': u'ubuntu',
+                 u'release': u'trusty',
+-                u'release_codename': u'Trusty Tahr',
+-                u'release_title': u'14.04 LTS',
+-                u'support_eol': u'2019-04-17',
+-                u'supported': True,
+                 u'version': u'14.04',
+                 u'versions': {u'20160602': {
+                     u'items': {u'disk1.img': {
+                         u'ftype': u'disk1.img',
+                         u'md5': u'e5436cd36ae6cc298f081bf0f6b413f1',
+-                        u'path': (
+-                            u'server/releases/trusty/release-20160602/'
+-                            u'ubuntu-14.04-server-cloudimg-amd64-disk1.img'),
+-                        u'sha256': (u'5b982d7d4dd1a03e88ae5f35f02ed44f'
+-                                    u'579e2711f3e0f27ea2bff20aef8c8d9e'),
+                         u'size': 259850752}},
+-                    u'label': u'release',
+                     u'pubname': u'ubuntu-trusty-14.04-amd64-server-20160602',
+                 }}}
+             }
+         }
+ 
++        pedigree = (
++            u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
++        product = source_index[u'products'][pedigree[0]]
++        image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
++
++        content_source = MemoryContentSource(
++            url="http://image-store/fooubuntu-X-disk1.img",
++            content="image-data")
++
++        # Use a fake GlanceClient to track calls and arguments passed to
++        # GlanceClient.images.create().
++        self.addCleanup(setattr, self.mirror, "gclient", self.mirror.gclient)
++        self.mirror.gclient = FakeGlanceClient()
++
++        target = {
++            'content_id': 'auto.sync',
++            'datatype': 'image-ids',
++            'format': 'products:1.0',
++        }
++
++        self.mirror.insert_item(
++            image_data, source_index, target, pedigree, content_source)
++
++        passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
++
++        # There is a 'data' argument pointing to an open file descriptor
++        # for the locally downloaded image.
++        image_content = passed_create_kwargs.pop("data").read()
++        self.assertEqual(u"image-data", image_content.decode('utf-8'))
++
++        # Value of "arch" from source entry is transformed into "architecture"
++        # image property in Glance: this ensures create_glance_properties()
++        # is called and result is properly passed.
++        self.assertEqual(
++            "x86_64", passed_create_kwargs["properties"]["architecture"])
++
++        # MD5 hash from source entry is put into 'checksum' field, and 'name'
++        # is based on full image name: this ensures prepare_glance_arguments()
++        # is called.
++        self.assertEqual(
++            u'e5436cd36ae6cc298f081bf0f6b413f1',
++            passed_create_kwargs["checksum"])
++        self.assertEqual(
++            u'auto-sync/ubuntu-trusty-14.04-amd64-server-20160602-disk1.img',
++            passed_create_kwargs["name"])
++
++        # Our local endpoint is set in the resulting entry, which ensures
++        # a call to adapt_source_entry() was indeed made.
++        target_product = target["products"][pedigree[0]]
++        target_image = target_product["versions"][pedigree[1]]["items"].get(
++            pedigree[2])
++        self.assertEqual(u"http://keystone/api/", target_image["endpoint"])
++
++    def test_insert_item_full(self):
++        # This test uses the full sample entries from the source simplestreams
++        # index from cloud-images.u.c and resulting local simplestreams index
++        # files.
++        source_index = copy.deepcopy(TEST_SOURCE_INDEX_ENTRY)
++
+         # "Pedigree" is basically a "path" to get to the image data in
+         # simplestreams index, going through "products", their "versions",
+         # and nested "items".
+@@ -409,9 +554,7 @@
+ 
+         passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
+ 
+-        # There is a 'data' argument pointing to an open file descriptor
+-        # for the locally downloaded image.
+-        self.assertIn("data", passed_create_kwargs)
++        # Drop the 'data' item pointing to an open temporary file.
+         passed_create_kwargs.pop("data")
+ 
+         expected_create_kwargs = {
+@@ -429,46 +572,21 @@
+                 'version_name': u'20160602',
+                 'content_id': 'auto.sync',
+                 'product_name': u'com.ubuntu.cloud:server:14.04:amd64',
++                'simplestreams_metadata': (
++                    '{"aliases": "14.04,default,lts,t,trusty", '
++                    '"arch": "amd64", "ftype": "disk1.img", '
++                    '"label": "release", "md5": '
++                    '"e5436cd36ae6cc298f081bf0f6b413f1", "os": "ubuntu", '
++                    '"pubname": "ubuntu-trusty-14.04-amd64-server-20160602", '
++                    '"release": "trusty", "release_codename": "Trusty Tahr", '
++                    '"release_title": "14.04 LTS", "sha256": '
++                    '"5b982d7d4dd1a03e88ae5f35f02ed44f'
++                    '579e2711f3e0f27ea2bff20aef8c8d9e", "size": "259850752", '
++                    '"support_eol": "2019-04-17", "supported": "True", '
++                    '"version": "14.04"}'),
+                 'source_content_id': u'com.ubuntu.cloud:released:download'},
+             'size': '259850752'}
+-        self.assertEqual(
+-            expected_create_kwargs, passed_create_kwargs)
+-
+-        # Almost real resulting data as produced by simplestreams before
+-        # insert_item refactoring to allow for finer-grained testing.
+-        expected_target_index = {
+-            'content_id': 'auto.sync',
+-            'datatype': 'image-ids',
+-            'format': 'products:1.0',
+-            'products': {
+-                "com.ubuntu.cloud:server:14.04:amd64": {
+-                    "aliases": "14.04,default,lts,t,trusty",
+-                    "arch": "amd64",
+-                    "label": "release",
+-                    "os": "ubuntu",
+-                    "owner_id": "bar456",
+-                    "pubname": "ubuntu-trusty-14.04-amd64-server-20160602",
+-                    "release": "trusty",
+-                    "release_codename": "Trusty Tahr",
+-                    "release_title": "14.04 LTS",
+-                    "support_eol": "2019-04-17",
+-                    "supported": "True",
+-                    "version": "14.04",
+-                    "versions": {"20160602": {"items": {"disk1.img": {
+-                        "endpoint": "http://keystone/api/",
+-                        "ftype": "disk1.img",
+-                        "id": "image-1",
+-                        "md5": "e5436cd36ae6cc298f081bf0f6b413f1",
+-                        "name": ("auto-sync/ubuntu-trusty-14.04-amd64-"
+-                                 "server-20160602-disk1.img"),
+-                        "region": "region1",
+-                        "sha256": ("5b982d7d4dd1a03e88ae5f35f02ed44f"
+-                                   "579e2711f3e0f27ea2bff20aef8c8d9e"),
+-                        "size": "259850752"
+-                    }}}}
+-                }
+-            }
+-        }
++        self.assertEqual(expected_create_kwargs, passed_create_kwargs)
+ 
+         # Apply the condensing as done in GlanceMirror.insert_products()
+         # to ensure we compare with the desired resulting simplestreams data.
+@@ -476,4 +594,40 @@
+                   'region']
+         simplestreams.util.products_condense(target, sticky)
+ 
+-        self.assertEqual(expected_target_index, target)
++        self.assertEqual(EXPECTED_OUTPUT_INDEX, target)
++
++    def test_insert_item_stores_the_index(self):
++        # Ensure insert_item calls insert_products() to generate the
++        # resulting simplestreams index file and insert it into store.
++
++        source_index = copy.deepcopy(TEST_SOURCE_INDEX_ENTRY)
++        pedigree = TEST_IMAGE_PEDIGREE
++        product = source_index[u'products'][pedigree[0]]
++        image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
++
++        content_source = MemoryContentSource(
++            url="http://image-store/fooubuntu-X-disk1.img",
++            content="image-data")
++        self.mirror.store = MemoryObjectStore()
++
++        self.addCleanup(setattr, self.mirror, "gclient", self.mirror.gclient)
++        self.mirror.gclient = FakeGlanceClient()
++
++        target = {
++            'content_id': 'auto.sync',
++            'datatype': 'image-ids',
++            'format': 'products:1.0',
++        }
++
++        self.mirror.insert_item(
++            image_data, source_index, target, pedigree, content_source)
++
++        stored_index_content = self.mirror.store.data[
++            'streams/v1/auto.sync.json']
++        stored_index = json.loads(stored_index_content.decode('utf-8'))
++
++        # Full index contains the 'updated' key with the date of last update.
++        self.assertIn(u"updated", stored_index)
++        del stored_index[u"updated"]
++
++        self.assertEqual(EXPECTED_OUTPUT_INDEX, stored_index)
+
diff --git a/debian/patches/keystone-v3-support.patch b/debian/patches/450-453-454-keystone-v3-support.patch
index 3b24355..e3a3adf 100644
--- a/debian/patches/keystone-v3-support.patch
+++ b/debian/patches/450-453-454-keystone-v3-support.patch
@@ -15,11 +15,9 @@ Bug: https://launchpad.net/bugs/1728982
 Bug: https://launchpad.net/bugs/1719879
 Author: Scott Moser <[e-mail]>
 
-diff --git a/simplestreams/mirrors/glance.py b/simplestreams/mirrors/glance.py
-index 807fe29..a13fede 100644
 --- a/simplestreams/mirrors/glance.py
 +++ b/simplestreams/mirrors/glance.py
-@@ -35,7 +35,11 @@ def get_glanceclient(version='1', **kwargs):
+@@ -36,7 +36,11 @@ def get_glanceclient(version='1', **kwar
      kwargs['endpoint'] = _strip_version(kwargs['endpoint'])
      pt = ('endpoint', 'token', 'insecure', 'cacert')
      kskw = {k: kwargs.get(k) for k in pt if k in kwargs}
@@ -32,7 +30,7 @@ index 807fe29..a13fede 100644
  
  
  def empty_iid_products(content_id):
-@@ -64,6 +68,7 @@ def canonicalize_arch(arch):
+@@ -65,6 +69,7 @@ def canonicalize_arch(arch):
  LXC_FTYPES = [
      'root.tar.gz',
      'root.tar.xz',
@@ -40,8 +38,15 @@ index 807fe29..a13fede 100644
  ]
  
  QEMU_FTYPES = [
-diff --git a/simplestreams/objectstores/swift.py b/simplestreams/objectstores/swift.py
-index d7598a7..9a6eb62 100644
+@@ -267,7 +272,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
+                 name_old, name_new = carry_over_property
+             else:
+                 name_old = name_new = carry_over_property
+-            properties[name_new] = image_metadata[name_old]
++            properties[name_new] = image_metadata.get(name_old)
+ 
+         if 'arch' in image_metadata:
+             properties['architecture'] = canonicalize_arch(
 --- a/simplestreams/objectstores/swift.py
 +++ b/simplestreams/objectstores/swift.py
 @@ -33,6 +33,15 @@ def get_swiftclient(**kwargs):
@@ -60,8 +65,6 @@ index d7598a7..9a6eb62 100644
      return Connection(**connargs)
  
  
-diff --git a/simplestreams/openstack.py b/simplestreams/openstack.py
-index 126dea5..d143926 100644
 --- a/simplestreams/openstack.py
 +++ b/simplestreams/openstack.py
 @@ -15,16 +15,47 @@
@@ -134,7 +137,7 @@ index 126dea5..d143926 100644
      if missing:
          raise ValueError("Need values for: %s" % missing)
  
-@@ -88,11 +128,49 @@ def get_regions(client=None, services=None, kscreds=None):
+@@ -88,11 +128,49 @@ def get_regions(client=None, services=No
      return list(regions)
  
  
@@ -188,7 +191,7 @@ index 126dea5..d143926 100644
  
  
  def get_service_conn_info(service='image', client=None, **kwargs):
-@@ -101,21 +179,27 @@ def get_service_conn_info(service='image', client=None, **kwargs):
+@@ -101,21 +179,27 @@ def get_service_conn_info(service='image
          client = get_ksclient(**kwargs)
  
      endpoint = _get_endpoint(client, service, **kwargs)
diff --git a/debian/patches/455-nova-lxd-support-squashfs-images.patch b/debian/patches/455-nova-lxd-support-squashfs-images.patch
 new file mode 100644
index 0000000..b4f0270
--- /dev/null
+++ b/debian/patches/455-nova-lxd-support-squashfs-images.patch
@@ -0,0 +1,230 @@
+------------------------------------------------------------
+revno: 455 [merge]
+fixes bug: https://launchpad.net/bugs/1686086
+committer: Scott Moser <[e-mail]>
+branch nick: trunk
+timestamp: Thu 2017-11-02 15:03:37 -0400
+message:
+  OpenStack: support uploading squash images for nova-lxd.
+  
+  Previously, populating a nova-lxd cloud was possible by using
+  root.tar.gz. A filter like:
+    ftype~(root.tar.gz|root.tar.xz)
+  would cause simplestreams to upload an image with 'disk-format' of
+  root-tar.
+  
+  However, Ubuntu 17.04 and newer do not have root.tar.gz or root.tar.xz
+  images available. Currently here is what is available:
+   14.04: root.tar.gz root.tar.xz
+   16.04: root.tar.gz root.tar.xz squashfs
+   17.10: squashfs
+  
+  If we simply expected the user to change their filter to include
+    root.tar.xz|squashfs
+  Then they would get two lxd images imported for 16.04 each version.
+  
+  The change here is to not do anything for an item insert, but instead
+  insert when the version's insert is called. Then, all the information
+  about what images there are is available, and it can "pick"
+  one or the other. Currently preference is given to the .tar.xz format.
+  
+  The end result is that now users can specify an ftype filter of:
+    ftype~(root.tar.gz|root.tar.xz|squashfs)
+  and the right thing will be done.
+  
+  Also here is simple knowledge that the squashfs type should be
+  uploaded to glance with a 'disk_format' of 'squashfs'.
+------------------------------------------------------------
+Use --include-merged or -n0 to see merged revisions.
+=== modified file 'simplestreams/mirrors/glance.py'
+--- a/simplestreams/mirrors/glance.py
++++ b/simplestreams/mirrors/glance.py
+@@ -66,25 +66,27 @@ def canonicalize_arch(arch):
+     return newarch
+ 
+ 
+-LXC_FTYPES = [
+-    'root.tar.gz',
+-    'root.tar.xz',
+-    'squashfs',
+-]
+-
+-QEMU_FTYPES = [
+-    'disk.img',
+-    'disk1.img',
+-]
++LXC_FTYPES = {
++    'root.tar.gz': 'root-tar',
++    'root.tar.xz': 'root-tar',
++    'squashfs': 'squashfs',
++}
++
++QEMU_FTYPES = {
++    'disk.img': 'qcow2',
++    'disk1.img': 'qcow2',
++}
+ 
+ 
+ def disk_format(ftype):
+-    '''Canonicalize disk formats for use in OpenStack'''
++    '''Canonicalize disk formats for use in OpenStack.
++    Input ftype is a 'ftype' from a simplestream feed.
++    Return value is the appropriate 'disk_format' for glance.'''
+     newftype = ftype.lower()
+     if newftype in LXC_FTYPES:
+-        return 'root-tar'
++        return LXC_FTYPES[newftype]
+     if newftype in QEMU_FTYPES:
+-        return 'qcow2'
++        return QEMU_FTYPES[newftype]
+     return None
+ 
+ 
+@@ -160,6 +162,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
+         self.content_id = config.get("content_id")
+         self.modify_hook = config.get("modify_hook")
+ 
++        self.inserts = {}
+         if not self.content_id:
+             raise TypeError("content_id is required")
+ 
+@@ -408,7 +411,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
+ 
+         return output_entry
+ 
+-    def insert_item(self, data, src, target, pedigree, contentsource):
++    def _insert_item(self, data, src, target, pedigree, contentsource):
+         """
+         Upload image into glance and add image metadata to simplestreams index.
+ 
+@@ -470,6 +473,55 @@ class GlanceMirror(mirrors.BasicMirrorWr
+         # unused in insert_products below.
+         self.insert_products(None, target, None)
+ 
++    def insert_item(self, data, src, target, pedigree, contentsource):
++        """Queue item to be inserted in subsequent call to insert_version
++
++        This adds the item to self.inserts which is then handled in
++        insert_version.  That allows the code to have context on
++        all the items for a given version, and "choose" one.  Ie,
++        if both root.tar.xz and squashfs are available, preference
++        can be given to the root.tar.gz.
++        """
++
++        product_name, version_name, item_name = pedigree
++        if product_name not in self.inserts:
++            self.inserts[product_name] = {}
++        if version_name not in self.inserts[product_name]:
++            self.inserts[product_name][version_name] = {}
++
++        if 'ftype' in data:
++            ftype = data['ftype']
++        else:
++            flat = util.products_exdata(src, pedigree, include_top=False)
++            ftype = flat.get('ftype')
++        self.inserts[product_name][version_name][item_name] = (
++            ftype, (data, src, target, pedigree, contentsource))
++
++    def insert_version(self, data, src, target, pedigree):
++        """Upload all images for this version into glance
++        and add image metadata to simplestreams index.
++
++        All the work actually happens in _insert_item.
++        """
++
++        product_name, version_name = pedigree
++        inserts = self.inserts.get(product_name, {}).get(version_name, [])
++
++        rtar_names = [f for f in inserts
++                      if inserts[f][0] in ('root.tar.gz', 'root.tar.xz')]
++
++        for _iname, (ftype, iargs) in inserts.items():
++            if ftype == "squashfs" and rtar_names:
++                LOG.info("[%s] Skipping ftype 'squashfs' image in preference"
++                         "for root tarball type in %s",
++                         '/'.join(pedigree), rtar_names)
++                continue
++            self._insert_item(*iargs)
++
++        # we do not specifically do anything for insert_version, but
++        # call parent.
++        super(GlanceMirror, self).insert_version(data, src, target, pedigree)
++
+     def remove_item(self, data, src, target, pedigree):
+         util.products_del(target, pedigree)
+         if 'id' in data:
+--- a/tests/unittests/test_glancemirror.py
++++ b/tests/unittests/test_glancemirror.py
+@@ -327,6 +327,15 @@ class TestGlanceMirror(TestCase):
+ 
+         self.assertEqual("root-tar", create_arguments["disk_format"])
+ 
++    def test_prepare_glance_arguments_disk_format_squashfs(self):
++        # squashfs images are acceptable for nova-lxd
++        source_entry = {"ftype": "squashfs"}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
++            image_properties=None)
++
++        self.assertEqual("squashfs", create_arguments["disk_format"])
++
+     def test_prepare_glance_arguments_size(self):
+         # Size is read from image metadata if defined.
+         source_entry = {"size": 5}
+@@ -470,7 +479,8 @@ class TestGlanceMirror(TestCase):
+         pedigree = (
+             u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
+         product = source_index[u'products'][pedigree[0]]
+-        image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
++        ver_data = product[u'versions'][pedigree[1]]
++        image_data = ver_data[u'items'][pedigree[2]]
+ 
+         content_source = MemoryContentSource(
+             url="http://image-store/fooubuntu-X-disk1.img",
+@@ -489,6 +499,8 @@ class TestGlanceMirror(TestCase):
+ 
+         self.mirror.insert_item(
+             image_data, source_index, target, pedigree, content_source)
++        self.mirror.insert_version(
++            ver_data, source_index, target, pedigree[0:2])
+ 
+         passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
+ 
+@@ -532,7 +544,8 @@ class TestGlanceMirror(TestCase):
+         pedigree = (
+             u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
+         product = source_index[u'products'][pedigree[0]]
+-        image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
++        ver_data = product[u'versions'][pedigree[1]]
++        image_data = ver_data[u'items'][pedigree[2]]
+ 
+         content_source = MemoryContentSource(
+             url="http://image-store/fooubuntu-X-disk1.img",
+@@ -551,6 +564,8 @@ class TestGlanceMirror(TestCase):
+ 
+         self.mirror.insert_item(
+             image_data, source_index, target, pedigree, content_source)
++        self.mirror.insert_version(
++            image_data, source_index, target, pedigree[0:2])
+ 
+         passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
+ 
+@@ -603,7 +618,8 @@ class TestGlanceMirror(TestCase):
+         source_index = copy.deepcopy(TEST_SOURCE_INDEX_ENTRY)
+         pedigree = TEST_IMAGE_PEDIGREE
+         product = source_index[u'products'][pedigree[0]]
+-        image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
++        ver_data = product[u'versions'][pedigree[1]]
++        image_data = ver_data[u'items'][pedigree[2]]
+ 
+         content_source = MemoryContentSource(
+             url="http://image-store/fooubuntu-X-disk1.img",
+@@ -621,6 +637,8 @@ class TestGlanceMirror(TestCase):
+ 
+         self.mirror.insert_item(
+             image_data, source_index, target, pedigree, content_source)
++        self.mirror.insert_version(
++            ver_data, source_index, target, pedigree[0:2])
+ 
+         stored_index_content = self.mirror.store.data[
+             'streams/v1/auto.sync.json']
diff --git a/debian/patches/460-glance-handle-v2-auth-with-sessions.patch b/debian/patches/460-glance-handle-v2-auth-with-sessions.patch
 new file mode 100644
index 0000000..5fca027
--- /dev/null
+++ b/debian/patches/460-glance-handle-v2-auth-with-sessions.patch
@@ -0,0 +1,33 @@
+------------------------------------------------------------
+revno: 460 [merge]
+fixes bug: https://launchpad.net/bugs/1611987
+author: David Ames <[e-mail]>
+committer: Scott Moser <[e-mail]>
+branch nick: trunk
+timestamp: Thu 2018-04-12 12:33:46 -0400
+message:
+  Glance: Handle Keystone v2 with session based authentication
+  
+  There are three cases we have to handle:
+   - keystone v2 without sessions
+   - keystone v2 with sessions
+   - keystone v3 with sessions
+  
+  We had the first and the last covered but not the middle. This change
+  addresses this.
+------------------------------------------------------------
+Use --include-merged or -n0 to see merged revisions.
+=== modified file 'simplestreams/openstack.py'
+--- a/simplestreams/openstack.py	2017-10-31 13:32:56 +0000
++++ b/simplestreams/openstack.py	2018-04-10 21:35:53 +0000
+@@ -181,7 +181,8 @@
+     endpoint = _get_endpoint(client, service, **kwargs)
+     # Session client does not have tenant_id set at client.tenant_id
+     # If client.tenant_id not set use method to get it
+-    tenant_id = client.tenant_id or client.auth.client.get_project_id()
++    tenant_id = (client.tenant_id or client.get_project_id(client.session) or
++                 client.auth.client.get_project_id())
+     info = {'token': client.auth_token, 'insecure': kwargs.get('insecure'),
+             'cacert': kwargs.get('cacert'), 'endpoint': endpoint,
+             'tenant_id': tenant_id}
+
diff --git a/debian/patches/series b/debian/patches/series
index 8eaeaf4..6b946bc 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -1,3 +1,10 @@
 read_signed-speed
 custom_user_agent_lp1578624.patch
-keystone-v3-support.patch
+428-do-not-require-that-hypervisor_config-be-present.patch
+433-glance-ignore-inactive-images.patch
+435-glance-refactor-for-testing.patch
+436-glance-fix-race-conditions.patch
+skip-openstack-tests-if-no-libs.patch
[phone]-keystone-v3-support.patch
+455-nova-lxd-support-squashfs-images.patch
+460-glance-handle-v2-auth-with-sessions.patch
diff --git a/debian/patches/skip-openstack-tests-if-no-libs.patch b/debian/patches/skip-openstack-tests-if-no-libs.patch
 new file mode 100644
index 0000000..43e3665
--- /dev/null
+++ b/debian/patches/skip-openstack-tests-if-no-libs.patch
@@ -0,0 +1,36 @@
+Description: Skip tests of openstack to avoid build-depends.
+ This takes a bit of upstream commit revno 440 to skip openstack
+ tests if the libraries are not present.
+Applied-Upstream: revno 440
+Author: Scott Moser <[e-mail]>
+--- a/tests/unittests/test_glancemirror.py
++++ b/tests/unittests/test_glancemirror.py
+@@ -1,12 +1,17 @@
+ from simplestreams.contentsource import MemoryContentSource
+-from simplestreams.mirrors.glance import GlanceMirror
+-from simplestreams.objectstores import MemoryObjectStore
++try:
++    from simplestreams.mirrors.glance import GlanceMirror
++    from simplestreams.objectstores import MemoryObjectStore
++    HAVE_OPENSTACK_LIBS = True
++except ImportError:
++    HAVE_OPENSTACK_LIBS = False
++
+ import simplestreams.util
+ 
+ import copy
+ import json
+ import os
+-from unittest import TestCase
++from unittest import TestCase, skipIf
+ 
+ 
+ # This is a real snippet from the simplestreams index entry for
+@@ -118,6 +123,7 @@ class FakeGlanceClient(object):
+         self.images = FakeImages()
+ 
+ 
++@skipIf(not HAVE_OPENSTACK_LIBS, "no python3 openstack available")
+ class TestGlanceMirror(TestCase):
+     """Tests for GlanceMirror methods."""
+