Merge ~rafaeldtinoco/ubuntu/+source/simplestreams:xenial-1686437-keystone-v3 into ubuntu/+source/simplestreams:ubuntu/xenial-devel

[Rafael David Tinoco]

SUMMARY FOR SIMPLESTREAMS SRU TO XENIAL

After analyzing the following bugs:

• LP: #1611987 - simplestreams - [SRU] glance-simplestreams-sync charm doesn't support keystone v3
• LP: #1686437 - simplestreams - can't sync images for keystone v3
• LP: #1719879 - simplestreams - [SRU] swift client needs to use v1 auth prior to ocata
• LP: #1728982 - simplestreams - [SRU] openstack mirror with keystone v3 always imports new images

For the keystone v3 fixes revno 454 is the minimum we need SRU'd back to xenial. Bionic 0.1.0~bzr460-0ubuntu1 has these changes. These two merges are the pertinent changes:

 1. Keystone v3 Support - https://is.gd/wq7r6g
 2. Fix KSv3 Bugs - https://is.gd/OOEo3G

0.1.0~bzr426-0ubuntu1.3 was uploaded. It contained a fix for LP: #1686437 (can't sync images for keystone v3). That change contained a regression (LP: #1728982 - openstack mirror with keystone v3 always imports new images - and LP: #1719879 - swift client needs to use v1 auth prior to ocata) and was marked as verification needed.

Work was done to fix that regression. A merge proposal is made for Xenial at https://is.gd/7ixQbO. We have a PPA at https://is.gd/Boda8J that contains a fix for the regression caused by 0.1.0~bzr426-0ubuntu1.3 and others.

Feedback from that PPA was asked and was given by Ed, Chris, Felipe and Billy. Billy found an issue about squashfs and that was fixed into 0.1.0~bzr426-0ubuntu1.4~ppa0, also uploaded to PPA at https://is.gd/Boda8J.

SRU template is needed in all referenced bugs:

 • 428-do-not-require-that-hypervisor_config-be-present.patch (LP: #1578622)
 • 433-glance-ignore-inactive-images.patch (LP: #1583276)
 • 436-glance-fix-race-conditions.patch (LP: #1584938)
 • 450-453-454-keystone-v3-support.patch (LP: #1686437, #1728982, #1719879)
 • 455-nova-lxd-support-squashfs-images.patch (LP: #1686086)

And version 0.1.0~bzr426-0ubuntu1.4 is good for a SRU and already tested by multiple people.

----

simplestreams (0.1.0~bzr426-0ubuntu1.4) xenial; urgency=medium

  * Pull back several upstream fixes to glance sync code
    - 428-do-not-require-that-hypervisor_config-be-present.patch (LP: #1578622)
    - 433-glance-ignore-inactive-images.patch (LP: #1583276)
    - 436-glance-fix-race-conditions.patch (LP: #1584938)
    - 450-453-454-keystone-v3-support.patch (LP: #1686437, #1728982, #1719879)
    - 455-nova-lxd-support-squashfs-images.patch (LP: #1686086)
    - 460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)

 -- Scott Moser <email address hidden> Thu, 12 Apr 2018 12:27:47 -0400

[Rafael David Tinoco (rafaeldtinoco)]

This merge request is wrapping up all the efforts made before at:

https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341214

I'm re-organizing all the Xenial SRU attempt in this one, as well as feedback from Ed and Felipe about this fix being good, together with a procedural SRU review from the Canonical Server Team.

[Rafael David Tinoco (rafaeldtinoco)]

Ed, Felipe,

Could you please review this merge request ? I'm uploading source package at the following PPA:

https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/simplestreams-xenial-sru

So you can grab simplestreams package from there.

Thanks a lot!

[Rafael David Tinoco (rafaeldtinoco)]

On 20/09/2019 07:20, Rafael David Tinoco wrote:
> Ed, Felipe,
>
> Could you please review this merge request ? I'm uploading source package at the following PPA:
>
> https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/simplestreams-xenial-sru
>
> So you can grab simplestreams package from there.
>
> Thanks a lot!
>
Spoke to Felipe today and he added this in his TODO list (to be done likely tomorrow. I also asked him to do the SRU verification in the public bug once package is uploaded to -proposed, as this bug is very cloudy and he is able to do review/verification quicker).

[Felipe Reyes (freyes)]

Hi Rafael,

Thanks for taking the time to prepare this patchset, I tested the PPA
and it works fine.

Without the ppa, the unit was reporting blocked state:

glance-simplestreams-sync/1* blocked idle 11 10.5.0.55 Image sync failed, retrying soon.

After the installation it could sync up properly:

glance-simplestreams-sync/1* active idle 11 10.5.0.55 Sync completed at 09/25/19 21:47:25

Images available in glance after the sync:
$ openstack image list
+--------------------------------------+---------------------------------------------------------------+--------+
| ID | Name | Status |
+--------------------------------------+---------------------------------------------------------------+--------+
| 205081a5-676d-4202-aa24-046b9a9038fc | auto-sync/ubuntu-bionic-18.04-amd64-server-20190918-disk1.img | active |
| f159ec29-3528-444d-9584-0c24c9aebd72 | auto-sync/ubuntu-trusty-14.04-amd64-server-20190514-disk1.img | active |
| 4a3b6833-7e27-4e88-8c3c-ede97d87f30b | auto-sync/ubuntu-xenial-16.04-amd64-server-20190918-disk1.img | active |
+--------------------------------------+---------------------------------------------------------------+--------+

Terminal output and logs:

glance-simplestreams-sync/1:/var/log/glance-simplestreams-sync.log
INFO * 09-25 21:32:05 [PID:25708] * root * configuring sync for url {'url': 'http://cloud-images.ubuntu.com/releases/', 'path': 'streams/v1/index.sjson', 'name_prefix': 'ubuntu:released', 'item_filters': ['release~(trusty|xenial|bionic)', 'arch~(x86_64|amd64)', 'ftype~(disk1.img|disk.img)'], 'max': 1}
ERROR * 09-25 21:32:05 [PID:25708] * root * Exception during syncing:
Traceback (most recent call last):
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 479, in main
    do_sync(charm_conf, status_exchange)
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 255, in do_sync
    store = SwiftObjectStore(SWIFT_DATA_DIR)
  File "/usr/lib/python2.7/dist-packages/simplestreams/objectstores/swift.py", line 58, in __init__
    self.keystone_creds = openstack.load_keystone_creds()
  File "/usr/lib/python2.7/dist-packages/simplestreams/openstack.py", line 61, in load_keystone_creds
    raise ValueError("(tenant_id or tenant_name)")
ValueError: (tenant_id or tenant_name)
WARNING * 09-25 21:32:05 [PID:25708] * root * no host info in configuration, can't set up rabbit.
WARNING * 09-25 21:32:05 [PID:25708] * root * No rabbitmq connection available for msg{'status': 'Error', 'message': 'Traceback (most recent call last):\n File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 479, in main\n do_sync(charm_conf, status_exchange)\n File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 255, in do_sync\n store = SwiftObjectStore(SWIFT_DATA_DIR)\n File "/usr/lib/python2.7/dist-packages/simplestreams/objectstores/swift.py", line 58, in __init__\n self.keystone_creds = openstack.load_keystone_creds()\n File "/usr/lib/python2.7/dist-pa...

[Felipe Reyes (freyes)]

review: Approve

[Bryce Harrington (bryce)]

The summary of this MP itemizes 6 patches, but the series file in the debdiff shows 8 patches added and 1 removed:

> -keystone-v3-support.patch
> +428-do-not-require-that-hypervisor_config-be-present.patch
> +433-glance-ignore-inactive-images.patch
> +435-glance-refactor-for-testing.patch
> +436-glance-fix-race-conditions.patch
> +skip-openstack-tests-if-no-libs.patch
> [phone]-keystone-v3-support.patch
> +455-nova-lxd-support-squashfs-images.patch
> +460-glance-handle-v2-auth-with-sessions.patch

I'd like to see the skip openstack tests and the 435 refactor mentioned in the changelog for completeness. 435 in particular adds/modifies a lot of code.

As others appear to have covered testing for this, I am mostly focusing on code review. I verified the various components of this patchset have gone through adequate levels of review. I've read through the debdiff itself looking for obvious flaws; I might implement a few things differently but am spotting nothing worth flagging as problematic.

review: Approve

[Rafael David Tinoco (rafaeldtinoco)]

> I'd like to see the skip openstack tests and the 435 refactor mentioned in the changelog for completeness. 435 in particular adds/modifies a lot of code.

Thanks a lot for the detailed review Bryce! Helped a lot!

Will work on this and ping you for uploading.

e3ea6a6... by Rafael David Tinoco on 2019-10-01

* Pull back several upstream fixes to glance sync code
  - 428-do-not-require-that-hypervisor_config-be-present.patch (LP: #1578622)
  - 433-glance-ignore-inactive-images.patch (LP: #1583276)
  - 435-glance-refactor-for-testing.patch
  - 436-glance-fix-race-conditions.patch (LP: #1584938)
  - 450-453-454-keystone-v3-support.patch (LP: #1686437, #1728982, #1719879)
  - 455-nova-lxd-support-squashfs-images.patch (LP: #1686086)
  - 460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)
  - skip-openstack-tests-if-no-libs.patch

Signed-off-by: Rafael David Tinoco <email address hidden>

f9c0d75... by Rafael David Tinoco on 2019-10-01

changelog

[Bryce Harrington (bryce)]

Branch uploaded to xenial proposed, and upload tag pushed:

trent:~/ubuntu/SimpleStreams/simplestreams-gu$ git push pkg upload/0.1.0_bzr426-0ubuntu1.4
Counting objects: 18, done.
Delta compression using up to 6 threads.
Compressing objects: 100% (18/18), done.
Writing objects: 100% (18/18), 23.76 KiB | 2.16 MiB/s, done.
Total 18 (delta 5), reused 0 (delta 0)
To ssh://git.launchpad.net/~usd-import-team/ubuntu/+source/simplestreams
 * [new tag] upload/0.1.0_bzr426-0ubuntu1.4 -> upload/0.1.0_bzr426-0ubuntu1.4
trent:~/ubuntu/SimpleStreams/review.mp373030$ dput ubuntu simplestreams_0.1.0~bzr426-0ubuntu1.4_source.changes
Checking signature on .changes
gpg: /home/bryce/ubuntu/SimpleStreams/review.mp373030/simplestreams_0.1.0~bzr426-0ubuntu1.4_source.changes: Valid signature from E603B2578FB8F0FB
Checking signature on .dsc
gpg: /home/bryce/ubuntu/SimpleStreams/review.mp373030/simplestreams_0.1.0~bzr426-0ubuntu1.4.dsc: Valid signature from E603B2578FB8F0FB
Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading simplestreams_0.1.0~bzr426-0ubuntu1.4.dsc: done.
  Uploading simplestreams_0.1.0~bzr426-0ubuntu1.4.debian.tar.xz: done.
  Uploading simplestreams_0.1.0~bzr426-0ubuntu1.4_source.buildinfo: done.
  Uploading simplestreams_0.1.0~bzr426-0ubuntu1.4_source.changes: done.
Successfully uploaded packages.

[Rafael David Tinoco (rafaeldtinoco)]

TL;DR version:

We are going to fix Bionic simplestreams package and provide this Bionic simplestreams package in Ubuntu Cloud Archive for Xenial. With that, we are fixing Xenial simplestreams behavior (to work with keystone v3) only if end-user enables Ubuntu Cloud Archive.

Conversation about this merge request SRU:

<rbasak> rafaeldtinoco: reviewing your simplestreams Xenial SRU, I'm not sure I follow why an SRU is necessary for about half of these bugs - relating to v3 support
<rbasak> I'm not aware that we usually backport new features to old LTSes due to new features/deprecations in future OpenStack releases - that's what I thought the cloud archive was for.
<rbasak> Am I wrong, or is this case exceptional somehow?
<rafaeldtinoco> rbasak: it was a seg request
<rafaeldtinoco> because of a charm
<rafaeldtinoco> let me open the lps
<rbasak> So a statement like "The OpenStack Keystone charm supports v3 only since Queens and later" doesn't help me - why doesn't the charm support the older version?
<rafaeldtinoco> ok.. so, with xenial, the keystone charm has to use simplestreams from a ppa in order to make it work whenever updating openstack
<rafaeldtinoco> the upgrade procedure needs v3 support for not to brake
<rafaeldtinoco> because of the ordering (services)
<rafaeldtinoco> but i must confess Im relying mostly on freyes feedback
<rafaeldtinoco> freyes: ^ do you have something else to add for rbasak ?
<freyes> rbasak, >=Queens OpenStack dropped support for Keystone v2
<rbasak> freyes: and the cloud archive provides Queens against Xenial as a backport, correct?
<freyes> rbasak, correct
<rbasak> OK, so shouldn't this simplestreams v3 support to into the cloud archive, and not the Ubuntu Xenial archive, to solve that problem?
<rbasak> go
<freyes> rbasak, the cloud archive doesn't carry the simplestreams package, and if there is an intention to do it for newer releases it won't help with this bug
<rbasak> I don't think that "doesn't carry the simplestreams package" automatically means that an SRU is justified, though the SRU team might conclude that it's OK if no other solution makes sense.
<rbasak> But I don't understand why the package couldn't just be added to the cloud archive
<rbasak> Just because it's not there right now doesn't mean it can't be added right now.
<rbasak> It'd bump users up, but that's what you'd be doing with the SRU anyway.
<freyes> another reason why we may want to fix the package in distro is because simplestreams being a client, it could be that someone just wants to talk to a cloud that runs keystone v3, so in the case you propose, they would need to add a cloud archive repo (pulling a lot of new packages)
<freyes> I see pros and cons on both ways
<rbasak> That's the same situation for any network protocol client in Xenial though, OpenStack or not.
<rafaeldtinoco> a question that triggers me is..
<rafaeldtinoco> instead of a SRU to cloud archive
<rbasak> We expect such clients to use a newer release, or a snap, or some backport PPA, etc.
<rafaeldtinoco> we would have to have the backport instead
<rafaeldtinoco> so there is a 1:1 relation with newer fixes
<rbasak> (or run in a container of a newer release; the list...

review: Disapprove

diff --git a/debian/changelog b/debian/changelog
index 01c067e..02010cd 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,18 @@
+simplestreams (0.1.0~bzr426-0ubuntu1.4) xenial; urgency=medium
+
+  [Scott Moser]
+  * Pull back several upstream fixes to glance sync code
+    - 428-do-not-require-that-hypervisor_config-be-present.patch (LP: #1578622)
+    - 433-glance-ignore-inactive-images.patch (LP: #1583276)
+    - 435-glance-refactor-for-testing.patch
+    - 436-glance-fix-race-conditions.patch (LP: #1584938)
+    - 450-453-454-keystone-v3-support.patch (LP: #1686437, #1728982, #1719879)
+    - 455-nova-lxd-support-squashfs-images.patch (LP: #1686086)
+    - 460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)
+    - skip-openstack-tests-if-no-libs.patch
+
+ -- Rafael David Tinoco <[e-mail]>  Tue, 01 Oct 2019 15:45:18 -0300
+
 simplestreams (0.1.0~bzr426-0ubuntu1.3) xenial-proposed; urgency=medium
 
   * Openstack: Add keystone v3 auth support (LP: #1686437).
diff --git a/debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch b/debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch
new file mode 100644
index 0000000..f449dad
--- /dev/null
+++ b/debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch
@@ -0,0 +1,23 @@
+------------------------------------------------------------
+revno: 428
+fixes bug: https://launchpad.net/bugs/1578622
+committer: Scott Moser <[e-mail]>
+branch nick: trunk
+timestamp: Thu 2016-05-05 08:19:26 -0400
+message:
+  glance mirror: do not require that hypervisor_config be present
+  
+  This just allows for hypervisor_config to not be present.
+=== modified file 'simplestreams/mirrors/glance.py'
+--- a/simplestreams/mirrors/glance.py	2016-03-23 10:00:53 +0000
++++ b/simplestreams/mirrors/glance.py	2016-05-05 12:19:26 +0000
+@@ -239,7 +239,7 @@
+             t_item['arch'] = arch
+             props['architecture'] = canonicalize_arch(arch)
+ 
+-        if self.config['hypervisor_mapping'] and 'ftype' in flat:
++        if self.config.get('hypervisor_mapping', False) and 'ftype' in flat:
+             _hypervisor_type = hypervisor_type(flat['ftype'])
+             if _hypervisor_type:
+                 props['hypervisor_type'] = _hypervisor_type
+
diff --git a/debian/patches/433-glance-ignore-inactive-images.patch b/debian/patches/433-glance-ignore-inactive-images.patch
new file mode 100644
index 0000000..ee704e0
--- /dev/null
+++ b/debian/patches/433-glance-ignore-inactive-images.patch
@@ -0,0 +1,42 @@
+------------------------------------------------------------
+revno: 433 [merge]
+fixes bug: https://launchpad.net/bugs/1583276
+committer: Scott Moser <[e-mail]>
+branch nick: trunk
+timestamp: Fri 2016-05-20 15:33:14 -0400
+message:
+  glance: ignore inactive images
+  
+  If connection to glance is broken during GlanceMirror.sync(), and image
+  would be left over in "status": "saving".
+  
+  Glance itself tries to protect against that, however when it is restarted
+  (eg. "service glance-api restart"), it does not clean the image up.
+  
+  This change makes load_product() ignore any images that do not have
+  "status": "active".  Images can only have deleted, active or saving
+  status.
+  
+  Note that this isn't perfect. The image in "saving" state is never cleaned
+  up, but at least the resulting cloud works. If we want to clean up broken
+  images, it'd be hard to do on the simplestreams side (glance is better
+  placed to do that: it knows if anything is going on with any of them, eg.
+  is it being currently updated or not).
+------------------------------------------------------------
+Use --include-merged or -n0 to see merged revisions.
+=== modified file 'simplestreams/mirrors/glance.py'
+--- a/simplestreams/mirrors/glance.py	2016-05-05 12:35:42 +0000
++++ b/simplestreams/mirrors/glance.py	2016-05-20 15:33:51 +0000
+@@ -179,6 +179,11 @@
+             if props.get('content_id') != my_cid:
+                 continue
+ 
++            if image.get('status') != "active":
++                LOG.warn("Ignoring inactive image %s with status '%s'" % (
++                    image['id'], image.get('status')))
++                continue
++
+             source_content_id = props.get('source_content_id')
+ 
+             product = props.get('product_name')
+
diff --git a/debian/patches/435-glance-refactor-for-testing.patch b/debian/patches/435-glance-refactor-for-testing.patch
new file mode 100644
index 0000000..404b531
--- /dev/null
+++ b/debian/patches/435-glance-refactor-for-testing.patch
@@ -0,0 +1,853 @@
+------------------------------------------------------------
+revno: 435 [merge]
+committer: Scott Moser <[e-mail]>
+branch nick: trunk
+timestamp: Tue 2016-06-14 16:10:56 -0400
+message:
+  GlanceMirror: refactor insert_item for easier testing
+  
+  This change refactors GlanceMirror.insert_item() to allow for easier and
+  more contained testing. I needed to do this to understand everything that
+  was going on inside insert_item and other bits of code. There are now four
+  distinct things happening in it:
+  
+   1. Download image to a local file from a ContentSource
+   2. Construct extra properties to store in Glance along with image
+   3. Prepare arguments for GlanceClient.images.create() call
+   4. Adapt source simplestreams entry for an image for use in the target
+      simplestreams index
+  
+  It should be fully backwards compatible, and test coverage for all the
+  individual steps should be much better (I admit to it not being perfect,
+  but it's a step in the right direction, imho at least).
+------------------------------------------------------------
+Use --include-merged or -n0 to see merged revisions.
+=== modified file 'simplestreams/mirrors/glance.py'
+--- a/simplestreams/mirrors/glance.py	2016-05-20 15:33:51 +0000
++++ b/simplestreams/mirrors/glance.py	2016-06-10 18:47:52 +0000
+@@ -105,8 +105,15 @@
+ # glance mirror 'image-downloads' content into glance
+ # if provided an object store, it will produce a 'image-ids' mirror
+ class GlanceMirror(mirrors.BasicMirrorWriter):
++    """
++    GlanceMirror syncs external simplestreams index and images to Glance.
++
++    `client` argument is used for testing to override openstack module:
++    allows dependency injection of fake "openstack" module.
++    """
+     def __init__(self, config, objectstore=None, region=None,
+-                 name_prefix=None, progress_callback=None):
++                 name_prefix=None, progress_callback=None,
++                 client=None):
+         super(GlanceMirror, self).__init__(config=config)
+ 
+         self.item_filters = self.config.get('item_filters', [])
+@@ -123,7 +130,10 @@
+         self.loaded_content = {}
+         self.store = objectstore
+ 
+-        self.keystone_creds = openstack.load_keystone_creds()
++        if client is None:
++            client = openstack
++
++        self.keystone_creds = client.load_keystone_creds()
+ 
+         self.name_prefix = name_prefix or ""
+         if region is not None:
+@@ -131,8 +141,8 @@
+ 
+         self.progress_callback = progress_callback
+ 
+-        conn_info = openstack.get_service_conn_info('image',
+-                                                    **self.keystone_creds)
++        conn_info = client.get_service_conn_info(
++            'image', **self.keystone_creds)
+         self.gclient = get_glanceclient(**conn_info)
+         self.tenant_id = conn_info['tenant_id']
+ 
+@@ -151,6 +161,12 @@
+         return "streams/v1/%s.json" % content_id
+ 
+     def load_products(self, path=None, content_id=None):
++        """
++        Load metadata for all currently uploaded active images in Glance.
++
++        Uses glance as the definitive store, but loads metadata from existing
++        simplestreams indexes as well.
++        """
+         my_cid = self.content_id
+ 
+         # glance is the definitive store.  Any data loaded from the store
+@@ -219,105 +235,212 @@
+     def filter_item(self, data, src, target, pedigree):
+         return filters.filter_item(self.item_filters, data, src, pedigree)
+ 
+-    def insert_item(self, data, src, target, pedigree, contentsource):
+-        flat = util.products_exdata(src, pedigree, include_top=False)
+-
+-        tmp_path = None
+-        tmp_del = None
+-
+-        name = flat.get('pubname', flat.get('name'))
+-        if not name.endswith(flat['item_name']):
+-            name += "-%s" % (flat['item_name'])
+-
+-        t_item = flat.copy()
+-        if 'path' in t_item:
+-            del t_item['path']
+-
+-        props = {'content_id': target['content_id'],
+-                 'source_content_id': src['content_id']}
+-        for n in ('product_name', 'version_name', 'item_name'):
+-            props[n] = flat[n]
+-            del t_item[n]
+-
+-        arch = flat.get('arch')
+-        if arch:
+-            t_item['arch'] = arch
+-            props['architecture'] = canonicalize_arch(arch)
+-
+-        if self.config.get('hypervisor_mapping', False) and 'ftype' in flat:
+-            _hypervisor_type = hypervisor_type(flat['ftype'])
++    def create_glance_properties(self, content_id, source_content_id,
++                                 image_metadata, hypervisor_mapping):
++        """
++        Construct extra properties to store in Glance for an image.
++
++        Based on source image metadata.
++        """
++        properties = {
++            'content_id': content_id,
++            'source_content_id': source_content_id,
++        }
++        # An iterator of properties to carry over: if a property needs
++        # renaming, uses a tuple (old name, new name).
++        carry_over = (
++            'product_name', 'version_name', 'item_name',
++            ('os', 'os_distro'), ('version', 'os_version'),
++        )
++        for carry_over_property in carry_over:
++            if isinstance(carry_over_property, tuple):
++                name_old, name_new = carry_over_property
++            else:
++                name_old = name_new = carry_over_property
++            properties[name_new] = image_metadata[name_old]
++
++        if 'arch' in image_metadata:
++            properties['architecture'] = canonicalize_arch(
++                image_metadata['arch'])
++
++        if hypervisor_mapping and 'ftype' in image_metadata:
++            _hypervisor_type = hypervisor_type(image_metadata['ftype'])
+             if _hypervisor_type:
+-                props['hypervisor_type'] = _hypervisor_type
+-                _virt_type = virt_type(_hypervisor_type)
+-                if _virt_type:
+-                    t_item['virt'] = _virt_type
+-
+-        if 'os' in flat:
+-            props['os_distro'] = flat['os']
+-
+-        if 'version' in flat:
+-            props['os_version'] = flat['version']
+-
+-        fullname = self.name_prefix + name
++                properties['hypervisor_type'] = _hypervisor_type
++        return properties
++
++    def prepare_glance_arguments(self, full_image_name, image_metadata,
++                                 image_md5_hash, image_size, image_properties):
++        """
++        Prepare arguments to pass into Glance image creation method.
++
++        Uses `image_metadata` for source image to derive image size, md5 hash,
++        disk format (based on 'ftype' field, if defined, otherwise defaults to
++        'qcow2').
++
++        If `image_md5_hash` and `image_size` are defined, overrides the
++        values from image_metadata with their values.
++
++        Sets extra image properties to dict `image_properties`.
++
++        Returns a dict to use as keyword arguments passed directly to
++        GlanceClient.images.create().
++        """
+         create_kwargs = {
+-            'name': fullname,
+-            'properties': props,
++            'name': full_image_name,
+             'container_format': 'bare',
+             'is_public': True,
++            'properties': image_properties,
+         }
+-        if 'size' in data:
+-            create_kwargs['size'] = data.get('size')
+-
+-        if 'md5' in data:
+-            create_kwargs['checksum'] = data.get('md5')
+-
+-        if 'ftype' in flat:
++
++        if 'size' in image_metadata:
++            create_kwargs['size'] = image_metadata.get('size')
++        if 'md5' in image_metadata:
++            create_kwargs['checksum'] = image_metadata.get('md5')
++        if image_md5_hash and image_size:
++            create_kwargs.update({
++                'checksum': image_md5_hash,
++                'size': image_size,
++            })
++
++        if 'ftype' in image_metadata:
+             create_kwargs['disk_format'] = (
+-                disk_format(flat['ftype']) or 'qcow2'
++                disk_format(image_metadata['ftype']) or 'qcow2'
+             )
+         else:
+             create_kwargs['disk_format'] = 'qcow2'
+ 
++        return create_kwargs
++
++    def download_image(self, contentsource, image_stream_data):
++        """
++        Download an image from contentsource.
++
++        `image_stream_data` represents a flattened image metadata structure
++        to use for any logging messages.
++
++        Returns a tuple of (local-image-path, image-size, image-md5-hash).
++
++        If download fails, these values will all be None.
++        """
++        tmp_path = new_size = new_md5 = None
++
++        image_name = image_stream_data.get('pubname')
++        image_size = image_stream_data.get('size')
++
+         if self.progress_callback:
+             def progress_wrapper(written):
+                 self.progress_callback(dict(status="Downloading",
+-                                            name=flat.get('pubname'),
+-                                            size=data.get('size', 0),
++                                            name=image_name,
++                                            size=image_size,
+                                             written=written))
+         else:
+             def progress_wrapper(written):
+                 pass
+ 
+         try:
+-            try:
+-                (tmp_path, tmp_del) = util.get_local_copy(
+-                    contentsource, progress_callback=progress_wrapper)
+-
+-                if self.modify_hook:
+-                    (newsize, newmd5) = call_hook(item=t_item, path=tmp_path,
+-                                                  cmd=self.modify_hook)
+-                    create_kwargs['checksum'] = newmd5
+-                    create_kwargs['size'] = newsize
+-                    t_item['md5'] = newmd5
+-                    t_item['size'] = newsize
+-
+-            finally:
+-                contentsource.close()
+-
++            tmp_path, _ = util.get_local_copy(
++                contentsource, progress_callback=progress_wrapper)
++
++            if self.modify_hook:
++                (new_size, new_md5) = call_hook(
++                    item=image_stream_data, path=tmp_path,
++                    cmd=self.modify_hook)
++        finally:
++            contentsource.close()
++
++        return tmp_path, new_size, new_md5
++
++    def adapt_source_entry(self, source_entry, hypervisor_mapping, image_name,
++                           image_md5_hash, image_size):
++        """
++        Adapts the source simplestreams dict `source_entry` for use in the
++        generated local simplestreams index.
++        """
++        output_entry = source_entry.copy()
++
++        # Drop attributes not needed for the simplestreams index itself.
++        for property_name in ('path', 'product_name', 'version_name',
++                              'item_name'):
++            if property_name in output_entry:
++                del output_entry[property_name]
++
++        if hypervisor_mapping and 'ftype' in output_entry:
++            _hypervisor_type = hypervisor_type(output_entry['ftype'])
++            if _hypervisor_type:
++                _virt_type = virt_type(_hypervisor_type)
++                if _virt_type:
++                    output_entry['virt'] = _virt_type
++
++        output_entry['region'] = self.region
++        output_entry['endpoint'] = self.auth_url
++        output_entry['owner_id'] = self.tenant_id
++
++        output_entry['name'] = image_name
++        if image_md5_hash and image_size:
++            output_entry['md5'] = image_md5_hash
++            output_entry['size'] = image_size
++
++        return output_entry
++
++    def insert_item(self, data, src, target, pedigree, contentsource):
++        """
++        Upload image into glance and add image metadata to simplestreams index.
++
++        `data` is the metadata for a particular image file from the source:
++            unused since all that data is present in the `src` entry for
++            the corresponding image as well.
++        `src` contains the entire simplestreams index from the image syncing
++            source.
++        `target` is the simplestreams index for currently available images
++            in glance (generated by load_products()) to add this item to.
++        `pedigree` is a "path" to get to the `data` for the image we desire,
++            a tuple of (product_name, version_name, image_type).
++        `contentsource` is a ContentSource to download the actual image data
++            from.
++        """
++        # Extract and flatten metadata for a product image matching
++        #   (product-name, version-name, image-type)
++        # from the tuple `pedigree` in the source simplestreams index.
++        flattened_img_data = util.products_exdata(
++            src, pedigree, include_top=False)
++
++        tmp_path = None
++
++        full_image_name = "{}{}".format(
++            self.name_prefix,
++            flattened_img_data.get('pubname', flattened_img_data.get('name')))
++        if not full_image_name.endswith(flattened_img_data['item_name']):
++            full_image_name += "-{}".format(flattened_img_data['item_name'])
++
++        # Download images locally into a temporary file.
++        tmp_path, new_size, new_md5 = self.download_image(
++            contentsource, flattened_img_data)
++
++        hypervisor_mapping = self.config.get('hypervisor_mapping', False)
++
++        glance_props = self.create_glance_properties(
++            target['content_id'], src['content_id'], flattened_img_data,
++            hypervisor_mapping)
++        create_kwargs = self.prepare_glance_arguments(
++            full_image_name, flattened_img_data, new_md5, new_size,
++            glance_props)
++
++        target_sstream_item = self.adapt_source_entry(
++            flattened_img_data, hypervisor_mapping, full_image_name, new_md5,
++            new_size)
++
++        try:
+             create_kwargs['data'] = open(tmp_path, 'rb')
+-            ret = self.gclient.images.create(**create_kwargs)
+-            t_item['id'] = ret.id
+-            print("created %s: %s" % (ret.id, fullname))
++            glance_image = self.gclient.images.create(**create_kwargs)
++            target_sstream_item['id'] = glance_image.id
++            print("created %s: %s" % (glance_image.id, full_image_name))
+ 
+         finally:
+-            if tmp_del and os.path.exists(tmp_path):
++            if tmp_path and os.path.exists(tmp_path):
+                 os.unlink(tmp_path)
+ 
+-        t_item['region'] = self.region
+-        t_item['endpoint'] = self.auth_url
+-        t_item['owner_id'] = self.tenant_id
+-        t_item['name'] = fullname
+-        util.products_set(target, t_item, pedigree)
++        util.products_set(target, target_sstream_item, pedigree)
+ 
+     def remove_item(self, data, src, target, pedigree):
+         util.products_del(target, pedigree)
+
+=== added file 'tests/unittests/test_glancemirror.py'
+--- a/tests/unittests/test_glancemirror.py	1970-01-01 00:00:00 +0000
++++ b/tests/unittests/test_glancemirror.py	2016-06-10 18:47:52 +0000
+@@ -0,0 +1,479 @@
++from simplestreams.contentsource import MemoryContentSource
++from simplestreams.mirrors.glance import GlanceMirror
++import simplestreams.util
++
++import os
++from unittest import TestCase
++
++
++class FakeOpenstack(object):
++    """Fake 'openstack' module replacement for testing GlanceMirror."""
++    def load_keystone_creds(self):
++        return {"auth_url": "http://keystone/api/"}
++
++    def get_service_conn_info(self, url, region_name=None, auth_url=None):
++        return {"endpoint": "http://objectstore/api/",
++                "tenant_id": "bar456"}
++
++
++class FakeImage(object):
++    """Fake image objects returned by GlanceClient.images.create()."""
++    def __init__(self, identifier):
++        self.id = identifier
++
++
++class FakeImages(object):
++    """Fake GlanceClient.images implementation to track create() calls."""
++    def __init__(self):
++        self.create_calls = []
++
++    def create(self, **kwargs):
++        self.create_calls.append(kwargs)
++        return FakeImage('image-%d' % len(self.create_calls))
++
++
++class FakeGlanceClient(object):
++    """Fake GlanceClient implementation to track images.create() calls."""
++    def __init__(self, *args):
++        self.images = FakeImages()
++
++
++class TestGlanceMirror(TestCase):
++    """Tests for GlanceMirror methods."""
++
++    def setUp(self):
++        self.config = {"content_id": "foo123"}
++        self.mirror = GlanceMirror(
++            self.config, name_prefix="auto-sync/", region="region1",
++            client=FakeOpenstack())
++
++    def test_adapt_source_entry(self):
++        # Adapts source entry for use in a local simplestreams index.
++        source_entry = {"source-key": "source-value"}
++        output_entry = self.mirror.adapt_source_entry(
++            source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
++            image_md5_hash=None, image_size=None)
++
++        # Source and output entry are different objects.
++        self.assertNotEqual(source_entry, output_entry)
++
++        # Output entry gets a few new properties like the endpoint and
++        # owner_id taken from the GlanceMirror and  OpenStack configuration,
++        # region from the value passed into GlanceMirror constructor, and
++        # image name from the passed in value.
++        # It also contains the source entries as well.
++        self.assertEqual(
++            {"endpoint": "http://keystone/api/",
++             "name": "foobuntu-X",
++             "owner_id": "bar456",
++             "region": "region1",
++             "source-key": "source-value"},
++            output_entry)
++
++    def test_adapt_source_entry_ignored_properties(self):
++        # adapt_source_entry() drops some properties from the source entry.
++        source_entry = {"path": "foo",
++                        "product_name": "bar",
++                        "version_name": "baz",
++                        "item_name": "bah"}
++        output_entry = self.mirror.adapt_source_entry(
++            source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
++            image_md5_hash=None, image_size=None)
++
++        # None of the values in 'source_entry' are preserved.
++        for key in ("path", "product_name", "version_name", "item"):
++            self.assertNotIn("path", output_entry)
++
++    def test_adapt_source_entry_image_md5_and_size(self):
++        # adapt_source_entry() will use passed in values for md5 and size.
++        # Even old stale values will be overridden when image_md5_hash and
++        # image_size are passed in.
++        source_entry = {"md5": "stale-md5"}
++        output_entry = self.mirror.adapt_source_entry(
++            source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
++            image_md5_hash="new-md5", image_size=5)
++
++        self.assertEqual("new-md5", output_entry["md5"])
++        self.assertEqual(5, output_entry["size"])
++
++    def test_adapt_source_entry_image_md5_and_size_both_required(self):
++        # adapt_source_entry() requires both md5 and size to not ignore them.
++
++        source_entry = {"md5": "stale-md5"}
++
++        # image_size is not passed in, so md5 value is not used either.
++        output_entry1 = self.mirror.adapt_source_entry(
++            source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
++            image_md5_hash="new-md5", image_size=None)
++        self.assertEqual("stale-md5", output_entry1["md5"])
++        self.assertNotIn("size", output_entry1)
++
++        # image_md5_hash is not passed in, so image_size is not used either.
++        output_entry2 = self.mirror.adapt_source_entry(
++            source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
++            image_md5_hash=None, image_size=5)
++        self.assertEqual("stale-md5", output_entry2["md5"])
++        self.assertNotIn("size", output_entry2)
++
++    def test_adapt_source_entry_hypervisor_mapping(self):
++        # If hypervisor_mapping is set to True, 'virt' value is derived from
++        # the source entry 'ftype'.
++        source_entry = {"ftype": "disk1.img"}
++        output_entry = self.mirror.adapt_source_entry(
++            source_entry, hypervisor_mapping=True, image_name="foobuntu-X",
++            image_md5_hash=None, image_size=None)
++
++        self.assertEqual("kvm", output_entry["virt"])
++
++    def test_adapt_source_entry_hypervisor_mapping_ftype_required(self):
++        # If hypervisor_mapping is set to True, but 'ftype' is missing in the
++        # source entry, 'virt' value is not added to the returned entry.
++        source_entry = {}
++        output_entry = self.mirror.adapt_source_entry(
++            source_entry, hypervisor_mapping=True, image_name="foobuntu-X",
++            image_md5_hash=None, image_size=None)
++
++        self.assertNotIn("virt", output_entry)
++
++    def test_create_glance_properties(self):
++        # Constructs glance properties to set on image during upload
++        # based on source image metadata.
++        source_entry = {
++            # All of these are carried over and potentially re-named.
++            "product_name": "foobuntu",
++            "version_name": "X",
++            "item_name": "disk1.img",
++            "os": "ubuntu",
++            "version": "16.04",
++            # Other entries are ignored.
++            "something-else": "ignored",
++        }
++        properties = self.mirror.create_glance_properties(
++            "content-1", "source-1", source_entry, hypervisor_mapping=False)
++
++        # Output properties contain content-id and source-content-id based
++        # on the passed in parameters, and carry over (with changed keys
++        # for "os" and "version") product_name, version_name, item_name and
++        # os and version values from the source entry.
++        self.assertEqual(
++            {"content_id": "content-1",
++             "source_content_id": "source-1",
++             "product_name": "foobuntu",
++             "version_name": "X",
++             "item_name": "disk1.img",
++             "os_distro": "ubuntu",
++             "os_version": "16.04"},
++            properties)
++
++    def test_create_glance_properties_arch(self):
++        # When 'arch' is present in the source entry, it is adapted and
++        # returned inside 'architecture' field.
++        source_entry = {
++            "product_name": "foobuntu",
++            "version_name": "X",
++            "item_name": "disk1.img",
++            "os": "ubuntu",
++            "version": "16.04",
++            "arch": "amd64",
++        }
++        properties = self.mirror.create_glance_properties(
++            "content-1", "source-1", source_entry, hypervisor_mapping=False)
++        self.assertEqual("x86_64", properties["architecture"])
++
++    def test_create_glance_properties_hypervisor_mapping(self):
++        # When hypervisor_mapping is requested and 'ftype' is present in
++        # the image metadata, 'hypervisor_type' is added to returned
++        # properties.
++        source_entry = {
++            "product_name": "foobuntu",
++            "version_name": "X",
++            "item_name": "disk1.img",
++            "os": "ubuntu",
++            "version": "16.04",
++            "ftype": "root.tar.gz",
++        }
++        properties = self.mirror.create_glance_properties(
++            "content-1", "source-1", source_entry, hypervisor_mapping=True)
++        self.assertEqual("lxc", properties["hypervisor_type"])
++
++    def test_prepare_glance_arguments(self):
++        # Prepares arguments to pass to GlanceClient.images.create()
++        # based on image metadata from the simplestreams source.
++        source_entry = {}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
++            image_properties=None)
++
++        # Arguments to always pass in contain the image name, container format,
++        # disk format, whether image is public, and any passed-in properties.
++        self.assertEqual(
++            {"name": "foobuntu-X",
++             "container_format": 'bare',
++             "disk_format": "qcow2",
++             "is_public": True,
++             "properties": None},
++            create_arguments)
++
++    def test_prepare_glance_arguments_disk_format(self):
++        # Disk format is based on the image 'ftype' (if defined).
++        source_entry = {"ftype": "root.tar.gz"}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
++            image_properties=None)
++
++        self.assertEqual("root-tar", create_arguments["disk_format"])
++
++    def test_prepare_glance_arguments_size(self):
++        # Size is read from image metadata if defined.
++        source_entry = {"size": 5}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
++            image_properties=None)
++
++        self.assertEqual(5, create_arguments["size"])
++
++    def test_prepare_glance_arguments_checksum(self):
++        # Checksum is based on the source entry 'md5' value, if defined.
++        source_entry = {"md5": "foo123"}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
++            image_properties=None)
++
++        self.assertEqual("foo123", create_arguments["checksum"])
++
++    def test_prepare_glance_arguments_size_and_md5_override(self):
++        # Size and md5 hash are overridden from the passed-in values even if
++        # defined on the source entry.
++        source_entry = {"size": 5, "md5": "foo123"}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash="bar456", image_size=10,
++            image_properties=None)
++
++        self.assertEqual(10, create_arguments["size"])
++        self.assertEqual("bar456", create_arguments["checksum"])
++
++    def test_prepare_glance_arguments_size_and_md5_no_override_hash(self):
++        # If only one of image_md5_hash or image_size is passed directly in,
++        # the other value is not overridden either.
++        source_entry = {"size": 5, "md5": "foo123"}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash="bar456",
++            image_size=None, image_properties=None)
++
++        self.assertEqual(5, create_arguments["size"])
++        self.assertEqual("foo123", create_arguments["checksum"])
++
++    def test_prepare_glance_arguments_size_and_md5_no_override_size(self):
++        # If only one of image_md5_hash or image_size is passed directly in,
++        # the other value is not overridden either.
++        source_entry = {"size": 5, "md5": "foo123"}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash=None, image_size=10,
++            image_properties=None)
++
++        self.assertEqual(5, create_arguments["size"])
++        self.assertEqual("foo123", create_arguments["checksum"])
++
++    def test_download_image(self):
++        # Downloads image from a contentsource.
++        content = "foo bazes the bar"
++        content_source = MemoryContentSource(
++            url="http://image-store/fooubuntu-X-disk1.img", content=content)
++        image_metadata = {"pubname": "foobuntu-X", "size": 5}
++        path, size, md5_hash = self.mirror.download_image(
++            content_source, image_metadata)
++        self.addCleanup(os.unlink, path)
++        self.assertIsNotNone(path)
++        self.assertIsNone(size)
++        self.assertIsNone(md5_hash)
++
++    def test_download_image_progress_callback(self):
++        # Progress callback is called with image name, size, status and buffer
++        # size after every 10kb of data: 3 times for 25kb of data below.
++        content = "abcdefghij" * int(1024 * 2.5)
++        content_source = MemoryContentSource(
++            url="http://image-store/fooubuntu-X-disk1.img", content=content)
++        image_metadata = {"pubname": "foobuntu-X", "size": len(content)}
++
++        self.progress_calls = []
++
++        def log_progress_calls(message):
++            self.progress_calls.append(message)
++
++        self.addCleanup(
++            setattr, self.mirror, "progress_callback",
++            self.mirror.progress_callback)
++        self.mirror.progress_callback = log_progress_calls
++        path, size, md5_hash = self.mirror.download_image(
++            content_source, image_metadata)
++        self.addCleanup(os.unlink, path)
++
++        self.assertEqual(
++            [{"name": "foobuntu-X", "size": 25600, "status": "Downloading",
++              "written": 10240}] * 3,
++            self.progress_calls)
++
++    def test_download_image_error(self):
++        # When there's an error during download, contentsource is still closed
++        # and the error is propagated below.
++        content = "abcdefghij"
++        content_source = MemoryContentSource(
++            url="http://image-store/fooubuntu-X-disk1.img", content=content)
++        image_metadata = {"pubname": "foobuntu-X", "size": len(content)}
++
++        # MemoryContentSource has an internal file descriptor which indicates
++        # if close() method has been called on it.
++        self.assertFalse(content_source.fd.closed)
++
++        self.addCleanup(
++            setattr, self.mirror, "progress_callback",
++            self.mirror.progress_callback)
++        self.mirror.progress_callback = lambda message: 1/0
++
++        self.assertRaises(
++            ZeroDivisionError,
++            self.mirror.download_image, content_source, image_metadata)
++
++        # We rely on the MemoryContentSource.close() side-effect to ensure
++        # close() method has indeed been called on the passed-in ContentSource.
++        self.assertTrue(content_source.fd.closed)
++
++    def test_insert_item(self):
++        # Downloads an image from a contentsource, uploads it into Glance,
++        # adapting and munging as needed (it updates the keystone endpoint,
++        # image and owner ids).
++        # This test is basically an integration test to make sure all the
++        # methods used by insert_item() are tied together in one good
++        # fully functioning whole.
++
++        # This is a real snippet from the simplestreams index entry for
++        # Ubuntu 14.04 amd64 image from cloud-images.ubuntu.com as of
++        # 2016-06-05.
++        source_index = {
++            u'content_id': u'com.ubuntu.cloud:released:download',
++            u'datatype': u'image-downloads',
++            u'format': u'products:1.0',
++            u'license': (u'http://www.canonical.com/'
++                         u'intellectual-property-policy'),
++            u'products': {u'com.ubuntu.cloud:server:14.04:amd64': {
++                u'aliases': u'14.04,default,lts,t,trusty',
++                u'arch': u'amd64',
++                u'os': u'ubuntu',
++                u'release': u'trusty',
++                u'release_codename': u'Trusty Tahr',
++                u'release_title': u'14.04 LTS',
++                u'support_eol': u'2019-04-17',
++                u'supported': True,
++                u'version': u'14.04',
++                u'versions': {u'20160602': {
++                    u'items': {u'disk1.img': {
++                        u'ftype': u'disk1.img',
++                        u'md5': u'e5436cd36ae6cc298f081bf0f6b413f1',
++                        u'path': (
++                            u'server/releases/trusty/release-20160602/'
++                            u'ubuntu-14.04-server-cloudimg-amd64-disk1.img'),
++                        u'sha256': (u'5b982d7d4dd1a03e88ae5f35f02ed44f'
++                                    u'579e2711f3e0f27ea2bff20aef8c8d9e'),
++                        u'size': 259850752}},
++                    u'label': u'release',
++                    u'pubname': u'ubuntu-trusty-14.04-amd64-server-20160602',
++                }}}
++            }
++        }
++
++        # "Pedigree" is basically a "path" to get to the image data in
++        # simplestreams index, going through "products", their "versions",
++        # and nested "items".
++        pedigree = (
++            u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
++        product = source_index[u'products'][pedigree[0]]
++        image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
++
++        content_source = MemoryContentSource(
++            url="http://image-store/fooubuntu-X-disk1.img",
++            content="image-data")
++
++        # Use a fake GlanceClient to track arguments passed into
++        # GlanceClient.images.create().
++        self.addCleanup(setattr, self.mirror, "gclient", self.mirror.gclient)
++        self.mirror.gclient = FakeGlanceClient()
++
++        target = {
++            'content_id': 'auto.sync',
++            'datatype': 'image-ids',
++            'format': 'products:1.0',
++        }
++
++        self.mirror.insert_item(
++            image_data, source_index, target, pedigree, content_source)
++
++        passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
++
++        # There is a 'data' argument pointing to an open file descriptor
++        # for the locally downloaded image.
++        self.assertIn("data", passed_create_kwargs)
++        passed_create_kwargs.pop("data")
++
++        expected_create_kwargs = {
++            'name': ('auto-sync/'
++                     'ubuntu-trusty-14.04-amd64-server-20160602-disk1.img'),
++            'checksum': u'e5436cd36ae6cc298f081bf0f6b413f1',
++            'disk_format': 'qcow2',
++            'container_format': 'bare',
++            'is_public': True,
++            'properties': {
++                'os_distro': u'ubuntu',
++                'item_name': u'disk1.img',
++                'os_version': u'14.04',
++                'architecture': 'x86_64',
++                'version_name': u'20160602',
++                'content_id': 'auto.sync',
++                'product_name': u'com.ubuntu.cloud:server:14.04:amd64',
++                'source_content_id': u'com.ubuntu.cloud:released:download'},
++            'size': '259850752'}
++        self.assertEqual(
++            expected_create_kwargs, passed_create_kwargs)
++
++        # Almost real resulting data as produced by simplestreams before
++        # insert_item refactoring to allow for finer-grained testing.
++        expected_target_index = {
++            'content_id': 'auto.sync',
++            'datatype': 'image-ids',
++            'format': 'products:1.0',
++            'products': {
++                "com.ubuntu.cloud:server:14.04:amd64": {
++                    "aliases": "14.04,default,lts,t,trusty",
++                    "arch": "amd64",
++                    "label": "release",
++                    "os": "ubuntu",
++                    "owner_id": "bar456",
++                    "pubname": "ubuntu-trusty-14.04-amd64-server-20160602",
++                    "release": "trusty",
++                    "release_codename": "Trusty Tahr",
++                    "release_title": "14.04 LTS",
++                    "support_eol": "2019-04-17",
++                    "supported": "True",
++                    "version": "14.04",
++                    "versions": {"20160602": {"items": {"disk1.img": {
++                        "endpoint": "http://keystone/api/",
++                        "ftype": "disk1.img",
++                        "id": "image-1",
++                        "md5": "e5436cd36ae6cc298f081bf0f6b413f1",
++                        "name": ("auto-sync/ubuntu-trusty-14.04-amd64-"
++                                 "server-20160602-disk1.img"),
++                        "region": "region1",
++                        "sha256": ("5b982d7d4dd1a03e88ae5f35f02ed44f"
++                                   "579e2711f3e0f27ea2bff20aef8c8d9e"),
++                        "size": "259850752"
++                    }}}}
++                }
++            }
++        }
++
++        # Apply the condensing as done in GlanceMirror.insert_products()
++        # to ensure we compare with the desired resulting simplestreams data.
++        sticky = ['ftype', 'md5', 'sha256', 'size', 'name', 'id', 'endpoint',
++                  'region']
++        simplestreams.util.products_condense(target, sticky)
++
++        self.assertEqual(expected_target_index, target)
+
diff --git a/debian/patches/436-glance-fix-race-conditions.patch b/debian/patches/436-glance-fix-race-conditions.patch
new file mode 100644
index 0000000..ae55dbe
--- /dev/null
+++ b/debian/patches/436-glance-fix-race-conditions.patch
@@ -0,0 +1,479 @@
+------------------------------------------------------------
+revno: 436 [merge]
+fixes bug: https://launchpad.net/bugs/1584938
+committer: Scott Moser <[e-mail]>
+branch nick: trunk
+timestamp: Tue 2016-06-14 16:21:05 -0400
+message:
+  GlanceMirror: fix a couple of race-related problems
+  
+  First, what can happen is that .sync() is interrupted for external reasons
+  (service restarts, network issues,...) after first image has been uploaded
+  to Glance when syncing multiple images. On re-run, the uploaded image is
+  considered "synced" already, but since simplestreams index is written out
+  only at the end of the sync, all the metadata is lost. We solve this by
+  adding "simplestreams_metadata" as one of the properties of image in
+  Glance where we put a JSON representation of all the metadata fields from
+  the original simplestreams index entry. We then load them in
+  load_products() if they are defined. This results in a slightly different
+  auto.sync.json entry: "endpoint" and "region" are on the product entry,
+  instead of on the disk1.img entry.
+  
+  Secondly, if sync is indeed interrupted after one or several images are
+  completely uploaded (but not all of them), there's no index file at all,
+  so any attempt to make use of them with juju would fail, even though it
+  should be fully functional. We solve this by calling "insert_products()"
+  at the end of the insert_item(). This means that the index is regenerated
+  after every single image is uploaded.
+  
+  Along the way, I slightly improve insert_item() tests from the pre-req
+  branch to add a test for minimal data required, rename the existing one
+  using real world data to test_insert_item_full, and re-use that data for a
+  test for a newly added call to insert_products().
+------------------------------------------------------------
+Use --include-merged or -n0 to see merged revisions.
+=== modified file 'simplestreams/mirrors/glance.py'
+--- a/simplestreams/mirrors/glance.py	2016-06-10 18:47:52 +0000
++++ b/simplestreams/mirrors/glance.py	2016-06-14 20:21:05 +0000
+@@ -25,6 +25,7 @@
+ import copy
+ import errno
+ import glanceclient
++import json
+ import os
+ import re
+ 
+@@ -219,6 +220,15 @@
+             except KeyError:
+                 item_data = {}
+ 
++            # If original simplestreams-metadata is stored on the image,
++            # use that as well.
++            if 'simplestreams_metadata' in props:
++                simplestreams_metadata = json.loads(
++                    props.get('simplestreams_metadata'))
++            else:
++                simplestreams_metadata = {}
++            item_data.update(simplestreams_metadata)
++
+             item_data.update({'name': image['name'], 'id': image['id']})
+             if 'owner_id' not in item_data:
+                 item_data['owner_id'] = self.tenant_id
+@@ -248,10 +258,10 @@
+         }
+         # An iterator of properties to carry over: if a property needs
+         # renaming, uses a tuple (old name, new name).
+-        carry_over = (
+-            'product_name', 'version_name', 'item_name',
+-            ('os', 'os_distro'), ('version', 'os_version'),
+-        )
++        carry_over_simple = (
++            'product_name', 'version_name', 'item_name')
++        carry_over = carry_over_simple + (
++            ('os', 'os_distro'), ('version', 'os_version'))
+         for carry_over_property in carry_over:
+             if isinstance(carry_over_property, tuple):
+                 name_old, name_new = carry_over_property
+@@ -267,6 +277,16 @@
+             _hypervisor_type = hypervisor_type(image_metadata['ftype'])
+             if _hypervisor_type:
+                 properties['hypervisor_type'] = _hypervisor_type
++
++        # Store flattened metadata for a source image along with the
++        # image in 'simplestreams_metadata' property.
++        simplestreams_metadata = image_metadata.copy()
++        drop_keys = carry_over_simple + ('path',)
++        for remove_key in drop_keys:
++            if remove_key in simplestreams_metadata:
++                del simplestreams_metadata[remove_key]
++        properties['simplestreams_metadata'] = json.dumps(
++            simplestreams_metadata, sort_keys=True)
+         return properties
+ 
+     def prepare_glance_arguments(self, full_image_name, image_metadata,
+@@ -441,6 +461,9 @@
+                 os.unlink(tmp_path)
+ 
+         util.products_set(target, target_sstream_item, pedigree)
++        # We can safely ignore path and content arguments since they are
++        # unused in insert_products below.
++        self.insert_products(None, target, None)
+ 
+     def remove_item(self, data, src, target, pedigree):
+         util.products_del(target, pedigree)
+
+=== modified file 'tests/unittests/test_glancemirror.py'
+--- a/tests/unittests/test_glancemirror.py	2016-06-10 18:47:52 +0000
++++ b/tests/unittests/test_glancemirror.py	2016-06-14 20:21:05 +0000
+@@ -1,11 +1,91 @@
+ from simplestreams.contentsource import MemoryContentSource
+ from simplestreams.mirrors.glance import GlanceMirror
++from simplestreams.objectstores import MemoryObjectStore
+ import simplestreams.util
+ 
++import copy
++import json
+ import os
+ from unittest import TestCase
+ 
+ 
++# This is a real snippet from the simplestreams index entry for
++# Ubuntu 14.04 amd64 image from cloud-images.ubuntu.com as of
++# 2016-06-05.
++TEST_SOURCE_INDEX_ENTRY = {
++    u'content_id': u'com.ubuntu.cloud:released:download',
++    u'datatype': u'image-downloads',
++    u'format': u'products:1.0',
++    u'license': (u'http://www.canonical.com/'
++                 u'intellectual-property-policy'),
++    u'products': {u'com.ubuntu.cloud:server:14.04:amd64': {
++        u'aliases': u'14.04,default,lts,t,trusty',
++        u'arch': u'amd64',
++        u'os': u'ubuntu',
++        u'release': u'trusty',
++        u'release_codename': u'Trusty Tahr',
++        u'release_title': u'14.04 LTS',
++        u'support_eol': u'2019-04-17',
++        u'supported': True,
++        u'version': u'14.04',
++        u'versions': {u'20160602': {
++            u'items': {u'disk1.img': {
++                u'ftype': u'disk1.img',
++                u'md5': u'e5436cd36ae6cc298f081bf0f6b413f1',
++                u'path': (
++                    u'server/releases/trusty/release-20160602/'
++                    u'ubuntu-14.04-server-cloudimg-amd64-disk1.img'),
++                u'sha256': (u'5b982d7d4dd1a03e88ae5f35f02ed44f'
++                            u'579e2711f3e0f27ea2bff20aef8c8d9e'),
++                u'size': 259850752}},
++            u'label': u'release',
++            u'pubname': u'ubuntu-trusty-14.04-amd64-server-20160602',
++        }}}
++    }
++}
++
++# "Pedigree" is basically a "path" to get to the image data in simplestreams
++# index, going through "products", their "versions", and nested "items".
++TEST_IMAGE_PEDIGREE = (
++    u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
++
++# Almost real resulting data as produced by simplestreams before
++# insert_item refactoring to allow for finer-grained testing.
++EXPECTED_OUTPUT_INDEX = {
++    u'content_id': u'auto.sync',
++    u'datatype': u'image-ids',
++    u'format': u'products:1.0',
++    u'products': {
++        u"com.ubuntu.cloud:server:14.04:amd64": {
++            u"aliases": u"14.04,default,lts,t,trusty",
++            u"arch": u"amd64",
++            u"label": u"release",
++            u"os": u"ubuntu",
++            u"owner_id": u"bar456",
++            u"pubname": u"ubuntu-trusty-14.04-amd64-server-20160602",
++            u"release": u"trusty",
++            u"release_codename": u"Trusty Tahr",
++            u"release_title": u"14.04 LTS",
++            u"support_eol": u"2019-04-17",
++            u"supported": u"True",
++            u"version": u"14.04",
++            u"versions": {u"20160602": {u"items": {u"disk1.img": {
++                u"endpoint": u"http://keystone/api/",
++                u"ftype": u"disk1.img",
++                u"id": u"image-1",
++                u"md5": u"e5436cd36ae6cc298f081bf0f6b413f1",
++                u"name": (u"auto-sync/ubuntu-trusty-14.04-amd64-"
++                          u"server-20160602-disk1.img"),
++                u"region": u"region1",
++                u"sha256": (u"5b982d7d4dd1a03e88ae5f35f02ed44f"
++                            u"579e2711f3e0f27ea2bff20aef8c8d9e"),
++                u"size": u"259850752"
++            }}}}
++        }
++    }
++}
++
++
+ class FakeOpenstack(object):
+     """Fake 'openstack' module replacement for testing GlanceMirror."""
+     def load_keystone_creds(self):
+@@ -145,8 +225,8 @@
+             "item_name": "disk1.img",
+             "os": "ubuntu",
+             "version": "16.04",
+-            # Other entries are ignored.
+-            "something-else": "ignored",
++            # Unknown entries are stored in 'simplestreams_metadata'.
++            "extra": "value",
+         }
+         properties = self.mirror.create_glance_properties(
+             "content-1", "source-1", source_entry, hypervisor_mapping=False)
+@@ -155,6 +235,8 @@
+         # on the passed in parameters, and carry over (with changed keys
+         # for "os" and "version") product_name, version_name, item_name and
+         # os and version values from the source entry.
++        # All the fields except product_name, version_name and item_name are
++        # also stored inside 'simplestreams_metadata' property as JSON data.
+         self.assertEqual(
+             {"content_id": "content-1",
+              "source_content_id": "source-1",
+@@ -162,7 +244,9 @@
+              "version_name": "X",
+              "item_name": "disk1.img",
+              "os_distro": "ubuntu",
+-             "os_version": "16.04"},
++             "os_version": "16.04",
++             "simplestreams_metadata": (
++                 '{"extra": "value", "os": "ubuntu", "version": "16.04"}')},
+             properties)
+ 
+     def test_create_glance_properties_arch(self):
+@@ -196,6 +280,26 @@
+             "content-1", "source-1", source_entry, hypervisor_mapping=True)
+         self.assertEqual("lxc", properties["hypervisor_type"])
+ 
++    def test_create_glance_properties_simplestreams_no_path(self):
++        # Other than 'product_name', 'version_name' and 'item_name', if 'path'
++        # is defined on the source entry, it is also not saved inside the
++        # 'simplestreams_metadata' property.
++        source_entry = {
++            "product_name": "foobuntu",
++            "version_name": "X",
++            "item_name": "disk1.img",
++            "os": "ubuntu",
++            "version": "16.04",
++            "path": "/path/to/foo",
++        }
++        properties = self.mirror.create_glance_properties(
++            "content-1", "source-1", source_entry, hypervisor_mapping=False)
++
++        # Path is omitted from the simplestreams_metadata property JSON.
++        self.assertEqual(
++            '{"os": "ubuntu", "version": "16.04"}',
++            properties["simplestreams_metadata"])
++
+     def test_prepare_glance_arguments(self):
+         # Prepares arguments to pass to GlanceClient.images.create()
+         # based on image metadata from the simplestreams source.
+@@ -342,45 +446,86 @@
+         # Downloads an image from a contentsource, uploads it into Glance,
+         # adapting and munging as needed (it updates the keystone endpoint,
+         # image and owner ids).
+-        # This test is basically an integration test to make sure all the
+-        # methods used by insert_item() are tied together in one good
+-        # fully functioning whole.
+ 
+-        # This is a real snippet from the simplestreams index entry for
+-        # Ubuntu 14.04 amd64 image from cloud-images.ubuntu.com as of
+-        # 2016-06-05.
++        # We use a minimal source simplestreams index, fake ContentSource and
++        # GlanceClient, and only test for side-effects of each of the
++        # subparts of the insert_item method.
+         source_index = {
+             u'content_id': u'com.ubuntu.cloud:released:download',
+-            u'datatype': u'image-downloads',
+-            u'format': u'products:1.0',
+-            u'license': (u'http://www.canonical.com/'
+-                         u'intellectual-property-policy'),
+             u'products': {u'com.ubuntu.cloud:server:14.04:amd64': {
+-                u'aliases': u'14.04,default,lts,t,trusty',
+                 u'arch': u'amd64',
+                 u'os': u'ubuntu',
+                 u'release': u'trusty',
+-                u'release_codename': u'Trusty Tahr',
+-                u'release_title': u'14.04 LTS',
+-                u'support_eol': u'2019-04-17',
+-                u'supported': True,
+                 u'version': u'14.04',
+                 u'versions': {u'20160602': {
+                     u'items': {u'disk1.img': {
+                         u'ftype': u'disk1.img',
+                         u'md5': u'e5436cd36ae6cc298f081bf0f6b413f1',
+-                        u'path': (
+-                            u'server/releases/trusty/release-20160602/'
+-                            u'ubuntu-14.04-server-cloudimg-amd64-disk1.img'),
+-                        u'sha256': (u'5b982d7d4dd1a03e88ae5f35f02ed44f'
+-                                    u'579e2711f3e0f27ea2bff20aef8c8d9e'),
+                         u'size': 259850752}},
+-                    u'label': u'release',
+                     u'pubname': u'ubuntu-trusty-14.04-amd64-server-20160602',
+                 }}}
+             }
+         }
+ 
++        pedigree = (
++            u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
++        product = source_index[u'products'][pedigree[0]]
++        image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
++
++        content_source = MemoryContentSource(
++            url="http://image-store/fooubuntu-X-disk1.img",
++            content="image-data")
++
++        # Use a fake GlanceClient to track calls and arguments passed to
++        # GlanceClient.images.create().
++        self.addCleanup(setattr, self.mirror, "gclient", self.mirror.gclient)
++        self.mirror.gclient = FakeGlanceClient()
++
++        target = {
++            'content_id': 'auto.sync',
++            'datatype': 'image-ids',
++            'format': 'products:1.0',
++        }
++
++        self.mirror.insert_item(
++            image_data, source_index, target, pedigree, content_source)
++
++        passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
++
++        # There is a 'data' argument pointing to an open file descriptor
++        # for the locally downloaded image.
++        image_content = passed_create_kwargs.pop("data").read()
++        self.assertEqual(u"image-data", image_content.decode('utf-8'))
++
++        # Value of "arch" from source entry is transformed into "architecture"
++        # image property in Glance: this ensures create_glance_properties()
++        # is called and result is properly passed.
++        self.assertEqual(
++            "x86_64", passed_create_kwargs["properties"]["architecture"])
++
++        # MD5 hash from source entry is put into 'checksum' field, and 'name'
++        # is based on full image name: this ensures prepare_glance_arguments()
++        # is called.
++        self.assertEqual(
++            u'e5436cd36ae6cc298f081bf0f6b413f1',
++            passed_create_kwargs["checksum"])
++        self.assertEqual(
++            u'auto-sync/ubuntu-trusty-14.04-amd64-server-20160602-disk1.img',
++            passed_create_kwargs["name"])
++
++        # Our local endpoint is set in the resulting entry, which ensures
++        # a call to adapt_source_entry() was indeed made.
++        target_product = target["products"][pedigree[0]]
++        target_image = target_product["versions"][pedigree[1]]["items"].get(
++            pedigree[2])
++        self.assertEqual(u"http://keystone/api/", target_image["endpoint"])
++
++    def test_insert_item_full(self):
++        # This test uses the full sample entries from the source simplestreams
++        # index from cloud-images.u.c and resulting local simplestreams index
++        # files.
++        source_index = copy.deepcopy(TEST_SOURCE_INDEX_ENTRY)
++
+         # "Pedigree" is basically a "path" to get to the image data in
+         # simplestreams index, going through "products", their "versions",
+         # and nested "items".
+@@ -409,9 +554,7 @@
+ 
+         passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
+ 
+-        # There is a 'data' argument pointing to an open file descriptor
+-        # for the locally downloaded image.
+-        self.assertIn("data", passed_create_kwargs)
++        # Drop the 'data' item pointing to an open temporary file.
+         passed_create_kwargs.pop("data")
+ 
+         expected_create_kwargs = {
+@@ -429,46 +572,21 @@
+                 'version_name': u'20160602',
+                 'content_id': 'auto.sync',
+                 'product_name': u'com.ubuntu.cloud:server:14.04:amd64',
++                'simplestreams_metadata': (
++                    '{"aliases": "14.04,default,lts,t,trusty", '
++                    '"arch": "amd64", "ftype": "disk1.img", '
++                    '"label": "release", "md5": '
++                    '"e5436cd36ae6cc298f081bf0f6b413f1", "os": "ubuntu", '
++                    '"pubname": "ubuntu-trusty-14.04-amd64-server-20160602", '
++                    '"release": "trusty", "release_codename": "Trusty Tahr", '
++                    '"release_title": "14.04 LTS", "sha256": '
++                    '"5b982d7d4dd1a03e88ae5f35f02ed44f'
++                    '579e2711f3e0f27ea2bff20aef8c8d9e", "size": "259850752", '
++                    '"support_eol": "2019-04-17", "supported": "True", '
++                    '"version": "14.04"}'),
+                 'source_content_id': u'com.ubuntu.cloud:released:download'},
+             'size': '259850752'}
+-        self.assertEqual(
+-            expected_create_kwargs, passed_create_kwargs)
+-
+-        # Almost real resulting data as produced by simplestreams before
+-        # insert_item refactoring to allow for finer-grained testing.
+-        expected_target_index = {
+-            'content_id': 'auto.sync',
+-            'datatype': 'image-ids',
+-            'format': 'products:1.0',
+-            'products': {
+-                "com.ubuntu.cloud:server:14.04:amd64": {
+-                    "aliases": "14.04,default,lts,t,trusty",
+-                    "arch": "amd64",
+-                    "label": "release",
+-                    "os": "ubuntu",
+-                    "owner_id": "bar456",
+-                    "pubname": "ubuntu-trusty-14.04-amd64-server-20160602",
+-                    "release": "trusty",
+-                    "release_codename": "Trusty Tahr",
+-                    "release_title": "14.04 LTS",
+-                    "support_eol": "2019-04-17",
+-                    "supported": "True",
+-                    "version": "14.04",
+-                    "versions": {"20160602": {"items": {"disk1.img": {
+-                        "endpoint": "http://keystone/api/",
+-                        "ftype": "disk1.img",
+-                        "id": "image-1",
+-                        "md5": "e5436cd36ae6cc298f081bf0f6b413f1",
+-                        "name": ("auto-sync/ubuntu-trusty-14.04-amd64-"
+-                                 "server-20160602-disk1.img"),
+-                        "region": "region1",
+-                        "sha256": ("5b982d7d4dd1a03e88ae5f35f02ed44f"
+-                                   "579e2711f3e0f27ea2bff20aef8c8d9e"),
+-                        "size": "259850752"
+-                    }}}}
+-                }
+-            }
+-        }
++        self.assertEqual(expected_create_kwargs, passed_create_kwargs)
+ 
+         # Apply the condensing as done in GlanceMirror.insert_products()
+         # to ensure we compare with the desired resulting simplestreams data.
+@@ -476,4 +594,40 @@
+                   'region']
+         simplestreams.util.products_condense(target, sticky)
+ 
+-        self.assertEqual(expected_target_index, target)
++        self.assertEqual(EXPECTED_OUTPUT_INDEX, target)
++
++    def test_insert_item_stores_the_index(self):
++        # Ensure insert_item calls insert_products() to generate the
++        # resulting simplestreams index file and insert it into store.
++
++        source_index = copy.deepcopy(TEST_SOURCE_INDEX_ENTRY)
++        pedigree = TEST_IMAGE_PEDIGREE
++        product = source_index[u'products'][pedigree[0]]
++        image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
++
++        content_source = MemoryContentSource(
++            url="http://image-store/fooubuntu-X-disk1.img",
++            content="image-data")
++        self.mirror.store = MemoryObjectStore()
++
++        self.addCleanup(setattr, self.mirror, "gclient", self.mirror.gclient)
++        self.mirror.gclient = FakeGlanceClient()
++
++        target = {
++            'content_id': 'auto.sync',
++            'datatype': 'image-ids',
++            'format': 'products:1.0',
++        }
++
++        self.mirror.insert_item(
++            image_data, source_index, target, pedigree, content_source)
++
++        stored_index_content = self.mirror.store.data[
++            'streams/v1/auto.sync.json']
++        stored_index = json.loads(stored_index_content.decode('utf-8'))
++
++        # Full index contains the 'updated' key with the date of last update.
++        self.assertIn(u"updated", stored_index)
++        del stored_index[u"updated"]
++
++        self.assertEqual(EXPECTED_OUTPUT_INDEX, stored_index)
+
diff --git a/debian/patches/keystone-v3-support.patch b/debian/patches/450-453-454-keystone-v3-support.patch
similarity index 84%
rename from debian/patches/keystone-v3-support.patch
rename to debian/patches/450-453-454-keystone-v3-support.patch
index 3b24355..e3a3adf 100644
--- a/debian/patches/keystone-v3-support.patch
+++ b/debian/patches/450-453-454-keystone-v3-support.patch
@@ -15,11 +15,9 @@ Bug: https://launchpad.net/bugs/1728982
 Bug: https://launchpad.net/bugs/1719879
 Author: Scott Moser <[e-mail]>
 
-diff --git a/simplestreams/mirrors/glance.py b/simplestreams/mirrors/glance.py
-index 807fe29..a13fede 100644
 --- a/simplestreams/mirrors/glance.py
 +++ b/simplestreams/mirrors/glance.py
-@@ -35,7 +35,11 @@ def get_glanceclient(version='1', **kwargs):
+@@ -36,7 +36,11 @@ def get_glanceclient(version='1', **kwar
      kwargs['endpoint'] = _strip_version(kwargs['endpoint'])
      pt = ('endpoint', 'token', 'insecure', 'cacert')
      kskw = {k: kwargs.get(k) for k in pt if k in kwargs}
@@ -32,7 +30,7 @@ index 807fe29..a13fede 100644
  
  
  def empty_iid_products(content_id):
-@@ -64,6 +68,7 @@ def canonicalize_arch(arch):
+@@ -65,6 +69,7 @@ def canonicalize_arch(arch):
  LXC_FTYPES = [
      'root.tar.gz',
      'root.tar.xz',
@@ -40,8 +38,15 @@ index 807fe29..a13fede 100644
  ]
  
  QEMU_FTYPES = [
-diff --git a/simplestreams/objectstores/swift.py b/simplestreams/objectstores/swift.py
-index d7598a7..9a6eb62 100644
+@@ -267,7 +272,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
+                 name_old, name_new = carry_over_property
+             else:
+                 name_old = name_new = carry_over_property
+-            properties[name_new] = image_metadata[name_old]
++            properties[name_new] = image_metadata.get(name_old)
+ 
+         if 'arch' in image_metadata:
+             properties['architecture'] = canonicalize_arch(
 --- a/simplestreams/objectstores/swift.py
 +++ b/simplestreams/objectstores/swift.py
 @@ -33,6 +33,15 @@ def get_swiftclient(**kwargs):
@@ -60,8 +65,6 @@ index d7598a7..9a6eb62 100644
      return Connection(**connargs)
  
  
-diff --git a/simplestreams/openstack.py b/simplestreams/openstack.py
-index 126dea5..d143926 100644
 --- a/simplestreams/openstack.py
 +++ b/simplestreams/openstack.py
 @@ -15,16 +15,47 @@
@@ -134,7 +137,7 @@ index 126dea5..d143926 100644
      if missing:
          raise ValueError("Need values for: %s" % missing)
  
-@@ -88,11 +128,49 @@ def get_regions(client=None, services=None, kscreds=None):
+@@ -88,11 +128,49 @@ def get_regions(client=None, services=No
      return list(regions)
  
  
@@ -188,7 +191,7 @@ index 126dea5..d143926 100644
  
  
  def get_service_conn_info(service='image', client=None, **kwargs):
-@@ -101,21 +179,27 @@ def get_service_conn_info(service='image', client=None, **kwargs):
+@@ -101,21 +179,27 @@ def get_service_conn_info(service='image
          client = get_ksclient(**kwargs)
  
      endpoint = _get_endpoint(client, service, **kwargs)
diff --git a/debian/patches/455-nova-lxd-support-squashfs-images.patch b/debian/patches/455-nova-lxd-support-squashfs-images.patch
new file mode 100644
index 0000000..b4f0270
--- /dev/null
+++ b/debian/patches/455-nova-lxd-support-squashfs-images.patch
@@ -0,0 +1,230 @@
+------------------------------------------------------------
+revno: 455 [merge]
+fixes bug: https://launchpad.net/bugs/1686086
+committer: Scott Moser <[e-mail]>
+branch nick: trunk
+timestamp: Thu 2017-11-02 15:03:37 -0400
+message:
+  OpenStack: support uploading squash images for nova-lxd.
+  
+  Previously, populating a nova-lxd cloud was possible by using
+  root.tar.gz. A filter like:
+    ftype~(root.tar.gz|root.tar.xz)
+  would cause simplestreams to upload an image with 'disk-format' of
+  root-tar.
+  
+  However, Ubuntu 17.04 and newer do not have root.tar.gz or root.tar.xz
+  images available. Currently here is what is available:
+   14.04: root.tar.gz root.tar.xz
+   16.04: root.tar.gz root.tar.xz squashfs
+   17.10: squashfs
+  
+  If we simply expected the user to change their filter to include
+    root.tar.xz|squashfs
+  Then they would get two lxd images imported for 16.04 each version.
+  
+  The change here is to not do anything for an item insert, but instead
+  insert when the version's insert is called. Then, all the information
+  about what images there are is available, and it can "pick"
+  one or the other. Currently preference is given to the .tar.xz format.
+  
+  The end result is that now users can specify an ftype filter of:
+    ftype~(root.tar.gz|root.tar.xz|squashfs)
+  and the right thing will be done.
+  
+  Also here is simple knowledge that the squashfs type should be
+  uploaded to glance with a 'disk_format' of 'squashfs'.
+------------------------------------------------------------
+Use --include-merged or -n0 to see merged revisions.
+=== modified file 'simplestreams/mirrors/glance.py'
+--- a/simplestreams/mirrors/glance.py
++++ b/simplestreams/mirrors/glance.py
+@@ -66,25 +66,27 @@ def canonicalize_arch(arch):
+     return newarch
+ 
+ 
+-LXC_FTYPES = [
+-    'root.tar.gz',
+-    'root.tar.xz',
+-    'squashfs',
+-]
+-
+-QEMU_FTYPES = [
+-    'disk.img',
+-    'disk1.img',
+-]
++LXC_FTYPES = {
++    'root.tar.gz': 'root-tar',
++    'root.tar.xz': 'root-tar',
++    'squashfs': 'squashfs',
++}
++
++QEMU_FTYPES = {
++    'disk.img': 'qcow2',
++    'disk1.img': 'qcow2',
++}
+ 
+ 
+ def disk_format(ftype):
+-    '''Canonicalize disk formats for use in OpenStack'''
++    '''Canonicalize disk formats for use in OpenStack.
++    Input ftype is a 'ftype' from a simplestream feed.
++    Return value is the appropriate 'disk_format' for glance.'''
+     newftype = ftype.lower()
+     if newftype in LXC_FTYPES:
+-        return 'root-tar'
++        return LXC_FTYPES[newftype]
+     if newftype in QEMU_FTYPES:
+-        return 'qcow2'
++        return QEMU_FTYPES[newftype]
+     return None
+ 
+ 
+@@ -160,6 +162,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
+         self.content_id = config.get("content_id")
+         self.modify_hook = config.get("modify_hook")
+ 
++        self.inserts = {}
+         if not self.content_id:
+             raise TypeError("content_id is required")
+ 
+@@ -408,7 +411,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
+ 
+         return output_entry
+ 
+-    def insert_item(self, data, src, target, pedigree, contentsource):
++    def _insert_item(self, data, src, target, pedigree, contentsource):
+         """
+         Upload image into glance and add image metadata to simplestreams index.
+ 
+@@ -470,6 +473,55 @@ class GlanceMirror(mirrors.BasicMirrorWr
+         # unused in insert_products below.
+         self.insert_products(None, target, None)
+ 
++    def insert_item(self, data, src, target, pedigree, contentsource):
++        """Queue item to be inserted in subsequent call to insert_version
++
++        This adds the item to self.inserts which is then handled in
++        insert_version.  That allows the code to have context on
++        all the items for a given version, and "choose" one.  Ie,
++        if both root.tar.xz and squashfs are available, preference
++        can be given to the root.tar.gz.
++        """
++
++        product_name, version_name, item_name = pedigree
++        if product_name not in self.inserts:
++            self.inserts[product_name] = {}
++        if version_name not in self.inserts[product_name]:
++            self.inserts[product_name][version_name] = {}
++
++        if 'ftype' in data:
++            ftype = data['ftype']
++        else:
++            flat = util.products_exdata(src, pedigree, include_top=False)
++            ftype = flat.get('ftype')
++        self.inserts[product_name][version_name][item_name] = (
++            ftype, (data, src, target, pedigree, contentsource))
++
++    def insert_version(self, data, src, target, pedigree):
++        """Upload all images for this version into glance
++        and add image metadata to simplestreams index.
++
++        All the work actually happens in _insert_item.
++        """
++
++        product_name, version_name = pedigree
++        inserts = self.inserts.get(product_name, {}).get(version_name, [])
++
++        rtar_names = [f for f in inserts
++                      if inserts[f][0] in ('root.tar.gz', 'root.tar.xz')]
++
++        for _iname, (ftype, iargs) in inserts.items():
++            if ftype == "squashfs" and rtar_names:
++                LOG.info("[%s] Skipping ftype 'squashfs' image in preference"
++                         "for root tarball type in %s",
++                         '/'.join(pedigree), rtar_names)
++                continue
++            self._insert_item(*iargs)
++
++        # we do not specifically do anything for insert_version, but
++        # call parent.
++        super(GlanceMirror, self).insert_version(data, src, target, pedigree)
++
+     def remove_item(self, data, src, target, pedigree):
+         util.products_del(target, pedigree)
+         if 'id' in data:
+--- a/tests/unittests/test_glancemirror.py
++++ b/tests/unittests/test_glancemirror.py
+@@ -327,6 +327,15 @@ class TestGlanceMirror(TestCase):
+ 
+         self.assertEqual("root-tar", create_arguments["disk_format"])
+ 
++    def test_prepare_glance_arguments_disk_format_squashfs(self):
++        # squashfs images are acceptable for nova-lxd
++        source_entry = {"ftype": "squashfs"}
++        create_arguments = self.mirror.prepare_glance_arguments(
++            "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
++            image_properties=None)
++
++        self.assertEqual("squashfs", create_arguments["disk_format"])
++
+     def test_prepare_glance_arguments_size(self):
+         # Size is read from image metadata if defined.
+         source_entry = {"size": 5}
+@@ -470,7 +479,8 @@ class TestGlanceMirror(TestCase):
+         pedigree = (
+             u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
+         product = source_index[u'products'][pedigree[0]]
+-        image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
++        ver_data = product[u'versions'][pedigree[1]]
++        image_data = ver_data[u'items'][pedigree[2]]
+ 
+         content_source = MemoryContentSource(
+             url="http://image-store/fooubuntu-X-disk1.img",
+@@ -489,6 +499,8 @@ class TestGlanceMirror(TestCase):
+ 
+         self.mirror.insert_item(
+             image_data, source_index, target, pedigree, content_source)
++        self.mirror.insert_version(
++            ver_data, source_index, target, pedigree[0:2])
+ 
+         passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
+ 
+@@ -532,7 +544,8 @@ class TestGlanceMirror(TestCase):
+         pedigree = (
+             u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
+         product = source_index[u'products'][pedigree[0]]
+-        image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
++        ver_data = product[u'versions'][pedigree[1]]
++        image_data = ver_data[u'items'][pedigree[2]]
+ 
+         content_source = MemoryContentSource(
+             url="http://image-store/fooubuntu-X-disk1.img",
+@@ -551,6 +564,8 @@ class TestGlanceMirror(TestCase):
+ 
+         self.mirror.insert_item(
+             image_data, source_index, target, pedigree, content_source)
++        self.mirror.insert_version(
++            image_data, source_index, target, pedigree[0:2])
+ 
+         passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
+ 
+@@ -603,7 +618,8 @@ class TestGlanceMirror(TestCase):
+         source_index = copy.deepcopy(TEST_SOURCE_INDEX_ENTRY)
+         pedigree = TEST_IMAGE_PEDIGREE
+         product = source_index[u'products'][pedigree[0]]
+-        image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
++        ver_data = product[u'versions'][pedigree[1]]
++        image_data = ver_data[u'items'][pedigree[2]]
+ 
+         content_source = MemoryContentSource(
+             url="http://image-store/fooubuntu-X-disk1.img",
+@@ -621,6 +637,8 @@ class TestGlanceMirror(TestCase):
+ 
+         self.mirror.insert_item(
+             image_data, source_index, target, pedigree, content_source)
++        self.mirror.insert_version(
++            ver_data, source_index, target, pedigree[0:2])
+ 
+         stored_index_content = self.mirror.store.data[
+             'streams/v1/auto.sync.json']
diff --git a/debian/patches/460-glance-handle-v2-auth-with-sessions.patch b/debian/patches/460-glance-handle-v2-auth-with-sessions.patch
new file mode 100644
index 0000000..5fca027
--- /dev/null
+++ b/debian/patches/460-glance-handle-v2-auth-with-sessions.patch
@@ -0,0 +1,33 @@
+------------------------------------------------------------
+revno: 460 [merge]
+fixes bug: https://launchpad.net/bugs/1611987
+author: David Ames <[e-mail]>
+committer: Scott Moser <[e-mail]>
+branch nick: trunk
+timestamp: Thu 2018-04-12 12:33:46 -0400
+message:
+  Glance: Handle Keystone v2 with session based authentication
+  
+  There are three cases we have to handle:
+   - keystone v2 without sessions
+   - keystone v2 with sessions
+   - keystone v3 with sessions
+  
+  We had the first and the last covered but not the middle. This change
+  addresses this.
+------------------------------------------------------------
+Use --include-merged or -n0 to see merged revisions.
+=== modified file 'simplestreams/openstack.py'
+--- a/simplestreams/openstack.py	2017-10-31 13:32:56 +0000
++++ b/simplestreams/openstack.py	2018-04-10 21:35:53 +0000
+@@ -181,7 +181,8 @@
+     endpoint = _get_endpoint(client, service, **kwargs)
+     # Session client does not have tenant_id set at client.tenant_id
+     # If client.tenant_id not set use method to get it
+-    tenant_id = client.tenant_id or client.auth.client.get_project_id()
++    tenant_id = (client.tenant_id or client.get_project_id(client.session) or
++                 client.auth.client.get_project_id())
+     info = {'token': client.auth_token, 'insecure': kwargs.get('insecure'),
+             'cacert': kwargs.get('cacert'), 'endpoint': endpoint,
+             'tenant_id': tenant_id}
+
diff --git a/debian/patches/series b/debian/patches/series
index 8eaeaf4..6b946bc 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -1,3 +1,10 @@
 read_signed-speed
 custom_user_agent_lp1578624.patch
-keystone-v3-support.patch
+428-do-not-require-that-hypervisor_config-be-present.patch
+433-glance-ignore-inactive-images.patch
+435-glance-refactor-for-testing.patch
+436-glance-fix-race-conditions.patch
+skip-openstack-tests-if-no-libs.patch
[phone]-keystone-v3-support.patch
+455-nova-lxd-support-squashfs-images.patch
+460-glance-handle-v2-auth-with-sessions.patch
diff --git a/debian/patches/skip-openstack-tests-if-no-libs.patch b/debian/patches/skip-openstack-tests-if-no-libs.patch
new file mode 100644
index 0000000..43e3665
--- /dev/null
+++ b/debian/patches/skip-openstack-tests-if-no-libs.patch
@@ -0,0 +1,36 @@
+Description: Skip tests of openstack to avoid build-depends.
+ This takes a bit of upstream commit revno 440 to skip openstack
+ tests if the libraries are not present.
+Applied-Upstream: revno 440
+Author: Scott Moser <[e-mail]>
+--- a/tests/unittests/test_glancemirror.py
++++ b/tests/unittests/test_glancemirror.py
+@@ -1,12 +1,17 @@
+ from simplestreams.contentsource import MemoryContentSource
+-from simplestreams.mirrors.glance import GlanceMirror
+-from simplestreams.objectstores import MemoryObjectStore
++try:
++    from simplestreams.mirrors.glance import GlanceMirror
++    from simplestreams.objectstores import MemoryObjectStore
++    HAVE_OPENSTACK_LIBS = True
++except ImportError:
++    HAVE_OPENSTACK_LIBS = False
++
+ import simplestreams.util
+ 
+ import copy
+ import json
+ import os
+-from unittest import TestCase
++from unittest import TestCase, skipIf
+ 
+ 
+ # This is a real snippet from the simplestreams index entry for
+@@ -118,6 +123,7 @@ class FakeGlanceClient(object):
+         self.images = FakeImages()
+ 
+ 
++@skipIf(not HAVE_OPENSTACK_LIBS, "no python3 openstack available")
+ class TestGlanceMirror(TestCase):
+     """Tests for GlanceMirror methods."""
+