Merge ~rafaeldtinoco/ubuntu/+source/simplestreams:xenial-1686437-keystone-v3 into ubuntu/+source/simplestreams:ubuntu/xenial-devel

Proposed by Rafael David Tinoco
Status: Merged
Merge reported by: Bryce Harrington
Merged at revision: f9c0d756f386732ab9dae15bf67df9520a92e380
Proposed branch: ~rafaeldtinoco/ubuntu/+source/simplestreams:xenial-1686437-keystone-v3
Merge into: ubuntu/+source/simplestreams:ubuntu/xenial-devel
Diff against target: 1851 lines (+1732/-11)
10 files modified
debian/changelog (+15/-0)
debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch (+23/-0)
debian/patches/433-glance-ignore-inactive-images.patch (+42/-0)
debian/patches/435-glance-refactor-for-testing.patch (+853/-0)
debian/patches/436-glance-fix-race-conditions.patch (+479/-0)
debian/patches/450-453-454-keystone-v3-support.patch (+13/-10)
debian/patches/455-nova-lxd-support-squashfs-images.patch (+230/-0)
debian/patches/460-glance-handle-v2-auth-with-sessions.patch (+33/-0)
debian/patches/series (+8/-1)
debian/patches/skip-openstack-tests-if-no-libs.patch (+36/-0)
Reviewer Review Type Date Requested Status
Rafael David Tinoco Disapprove
Bryce Harrington Approve
Felipe Reyes (community) Approve
Edward Hope-Morley Pending
Review via email: mp+373030@code.launchpad.net

Commit message

SUMMARY FOR SIMPLESTREAMS SRU TO XENIAL

After analyzing the following bugs:

• LP: #1611987 - simplestreams - [SRU] glance-simplestreams-sync charm doesn't support keystone v3
• LP: #1686437 - simplestreams - can't sync images for keystone v3
• LP: #1719879 - simplestreams - [SRU] swift client needs to use v1 auth prior to ocata
• LP: #1728982 - simplestreams - [SRU] openstack mirror with keystone v3 always imports new images

For the keystone v3 fixes revno 454 is the minimum we need SRU'd back to xenial. Bionic 0.1.0~bzr460-0ubuntu1 has these changes. These two merges are the pertinent changes:

 1. Keystone v3 Support - https://is.gd/wq7r6g
 2. Fix KSv3 Bugs - https://is.gd/OOEo3G

0.1.0~bzr426-0ubuntu1.3 was uploaded. It contained a fix for LP: #1686437 (can't sync images for keystone v3). That change contained a regression (LP: #1728982 - openstack mirror with keystone v3 always imports new images - and LP: #1719879 - swift client needs to use v1 auth prior to ocata) and was marked as verification needed.

Work was done to fix that regression. A merge proposal is made for Xenial at https://is.gd/7ixQbO. We have a PPA at https://is.gd/Boda8J that contains a fix for the regression caused by 0.1.0~bzr426-0ubuntu1.3 and others.

Feedback from that PPA was asked and was given by Ed, Chris, Felipe and Billy. Billy found an issue about squashfs and that was fixed into 0.1.0~bzr426-0ubuntu1.4~ppa0, also uploaded to PPA at https://is.gd/Boda8J.

SRU template is needed in all referenced bugs:

 • 428-do-not-require-that-hypervisor_config-be-present.patch (LP: #1578622)
 • 433-glance-ignore-inactive-images.patch (LP: #1583276)
 • 436-glance-fix-race-conditions.patch (LP: #1584938)
 • 450-453-454-keystone-v3-support.patch (LP: #1686437, #1728982, #1719879)
 • 455-nova-lxd-support-squashfs-images.patch (LP: #1686086)

And version 0.1.0~bzr426-0ubuntu1.4 is good for a SRU and already tested by multiple people.

----

simplestreams (0.1.0~bzr426-0ubuntu1.4) xenial; urgency=medium

  * Pull back several upstream fixes to glance sync code
    - 428-do-not-require-that-hypervisor_config-be-present.patch (LP: #1578622)
    - 433-glance-ignore-inactive-images.patch (LP: #1583276)
    - 436-glance-fix-race-conditions.patch (LP: #1584938)
    - 450-453-454-keystone-v3-support.patch (LP: #1686437, #1728982, #1719879)
    - 455-nova-lxd-support-squashfs-images.patch (LP: #1686086)
    - 460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)

 -- Scott Moser <email address hidden> Thu, 12 Apr 2018 12:27:47 -0400

To post a comment you must log in.
Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

This merge request is wrapping up all the efforts made before at:

https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341214

I'm re-organizing all the Xenial SRU attempt in this one, as well as feedback from Ed and Felipe about this fix being good, together with a procedural SRU review from the Canonical Server Team.

Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

Ed, Felipe,

Could you please review this merge request ? I'm uploading source package at the following PPA:

https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/simplestreams-xenial-sru

So you can grab simplestreams package from there.

Thanks a lot!

Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

On 20/09/2019 07:20, Rafael David Tinoco wrote:
> Ed, Felipe,
>
> Could you please review this merge request ? I'm uploading source package at the following PPA:
>
> https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/simplestreams-xenial-sru
>
> So you can grab simplestreams package from there.
>
> Thanks a lot!
>
Spoke to Felipe today and he added this in his TODO list (to be done likely tomorrow. I also asked him to do the SRU verification in the public bug once package is uploaded to -proposed, as this bug is very cloudy and he is able to do review/verification quicker).

Revision history for this message
Felipe Reyes (freyes) wrote :
Download full text (12.3 KiB)

Hi Rafael,

Thanks for taking the time to prepare this patchset, I tested the PPA
and it works fine.

Without the ppa, the unit was reporting blocked state:

glance-simplestreams-sync/1* blocked idle 11 10.5.0.55 Image sync failed, retrying soon.

After the installation it could sync up properly:

glance-simplestreams-sync/1* active idle 11 10.5.0.55 Sync completed at 09/25/19 21:47:25

Images available in glance after the sync:
$ openstack image list
+--------------------------------------+---------------------------------------------------------------+--------+
| ID | Name | Status |
+--------------------------------------+---------------------------------------------------------------+--------+
| 205081a5-676d-4202-aa24-046b9a9038fc | auto-sync/ubuntu-bionic-18.04-amd64-server-20190918-disk1.img | active |
| f159ec29-3528-444d-9584-0c24c9aebd72 | auto-sync/ubuntu-trusty-14.04-amd64-server-20190514-disk1.img | active |
| 4a3b6833-7e27-4e88-8c3c-ede97d87f30b | auto-sync/ubuntu-xenial-16.04-amd64-server-20190918-disk1.img | active |
+--------------------------------------+---------------------------------------------------------------+--------+

Terminal output and logs:

glance-simplestreams-sync/1:/var/log/glance-simplestreams-sync.log
INFO * 09-25 21:32:05 [PID:25708] * root * configuring sync for url {'url': 'http://cloud-images.ubuntu.com/releases/', 'path': 'streams/v1/index.sjson', 'name_prefix': 'ubuntu:released', 'item_filters': ['release~(trusty|xenial|bionic)', 'arch~(x86_64|amd64)', 'ftype~(disk1.img|disk.img)'], 'max': 1}
ERROR * 09-25 21:32:05 [PID:25708] * root * Exception during syncing:
Traceback (most recent call last):
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 479, in main
    do_sync(charm_conf, status_exchange)
  File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 255, in do_sync
    store = SwiftObjectStore(SWIFT_DATA_DIR)
  File "/usr/lib/python2.7/dist-packages/simplestreams/objectstores/swift.py", line 58, in __init__
    self.keystone_creds = openstack.load_keystone_creds()
  File "/usr/lib/python2.7/dist-packages/simplestreams/openstack.py", line 61, in load_keystone_creds
    raise ValueError("(tenant_id or tenant_name)")
ValueError: (tenant_id or tenant_name)
WARNING * 09-25 21:32:05 [PID:25708] * root * no host info in configuration, can't set up rabbit.
WARNING * 09-25 21:32:05 [PID:25708] * root * No rabbitmq connection available for msg{'status': 'Error', 'message': 'Traceback (most recent call last):\n File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 479, in main\n do_sync(charm_conf, status_exchange)\n File "/usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py", line 255, in do_sync\n store = SwiftObjectStore(SWIFT_DATA_DIR)\n File "/usr/lib/python2.7/dist-packages/simplestreams/objectstores/swift.py", line 58, in __init__\n self.keystone_creds = openstack.load_keystone_creds()\n File "/usr/lib/python2.7/dist-pa...

Revision history for this message
Felipe Reyes (freyes) :
review: Approve
Revision history for this message
Bryce Harrington (bryce) wrote :

The summary of this MP itemizes 6 patches, but the series file in the debdiff shows 8 patches added and 1 removed:

> -keystone-v3-support.patch
> +428-do-not-require-that-hypervisor_config-be-present.patch
> +433-glance-ignore-inactive-images.patch
> +435-glance-refactor-for-testing.patch
> +436-glance-fix-race-conditions.patch
> +skip-openstack-tests-if-no-libs.patch
> +450-453-454-keystone-v3-support.patch
> +455-nova-lxd-support-squashfs-images.patch
> +460-glance-handle-v2-auth-with-sessions.patch

I'd like to see the skip openstack tests and the 435 refactor mentioned in the changelog for completeness. 435 in particular adds/modifies a lot of code.

As others appear to have covered testing for this, I am mostly focusing on code review. I verified the various components of this patchset have gone through adequate levels of review. I've read through the debdiff itself looking for obvious flaws; I might implement a few things differently but am spotting nothing worth flagging as problematic.

review: Approve
Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

> I'd like to see the skip openstack tests and the 435 refactor mentioned in the changelog for completeness. 435 in particular adds/modifies a lot of code.

Thanks a lot for the detailed review Bryce! Helped a lot!

Will work on this and ping you for uploading.

e3ea6a6... by Rafael David Tinoco on 2019-10-01

* Pull back several upstream fixes to glance sync code
  - 428-do-not-require-that-hypervisor_config-be-present.patch (LP: #1578622)
  - 433-glance-ignore-inactive-images.patch (LP: #1583276)
  - 435-glance-refactor-for-testing.patch
  - 436-glance-fix-race-conditions.patch (LP: #1584938)
  - 450-453-454-keystone-v3-support.patch (LP: #1686437, #1728982, #1719879)
  - 455-nova-lxd-support-squashfs-images.patch (LP: #1686086)
  - 460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)
  - skip-openstack-tests-if-no-libs.patch

Signed-off-by: Rafael David Tinoco <email address hidden>

f9c0d75... by Rafael David Tinoco on 2019-10-01

changelog

Revision history for this message
Bryce Harrington (bryce) wrote :

Branch uploaded to xenial proposed, and upload tag pushed:

trent:~/ubuntu/SimpleStreams/simplestreams-gu$ git push pkg upload/0.1.0_bzr426-0ubuntu1.4
Counting objects: 18, done.
Delta compression using up to 6 threads.
Compressing objects: 100% (18/18), done.
Writing objects: 100% (18/18), 23.76 KiB | 2.16 MiB/s, done.
Total 18 (delta 5), reused 0 (delta 0)
To ssh://git.launchpad.net/~usd-import-team/ubuntu/+source/simplestreams
 * [new tag] upload/0.1.0_bzr426-0ubuntu1.4 -> upload/0.1.0_bzr426-0ubuntu1.4
trent:~/ubuntu/SimpleStreams/review.mp373030$ dput ubuntu simplestreams_0.1.0~bzr426-0ubuntu1.4_source.changes
Checking signature on .changes
gpg: /home/bryce/ubuntu/SimpleStreams/review.mp373030/simplestreams_0.1.0~bzr426-0ubuntu1.4_source.changes: Valid signature from E603B2578FB8F0FB
Checking signature on .dsc
gpg: /home/bryce/ubuntu/SimpleStreams/review.mp373030/simplestreams_0.1.0~bzr426-0ubuntu1.4.dsc: Valid signature from E603B2578FB8F0FB
Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading simplestreams_0.1.0~bzr426-0ubuntu1.4.dsc: done.
  Uploading simplestreams_0.1.0~bzr426-0ubuntu1.4.debian.tar.xz: done.
  Uploading simplestreams_0.1.0~bzr426-0ubuntu1.4_source.buildinfo: done.
  Uploading simplestreams_0.1.0~bzr426-0ubuntu1.4_source.changes: done.
Successfully uploaded packages.

Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :
Download full text (5.0 KiB)

TL;DR version:

We are going to fix Bionic simplestreams package and provide this Bionic simplestreams package in Ubuntu Cloud Archive for Xenial. With that, we are fixing Xenial simplestreams behavior (to work with keystone v3) only if end-user enables Ubuntu Cloud Archive.

Conversation about this merge request SRU:

<rbasak> rafaeldtinoco: reviewing your simplestreams Xenial SRU, I'm not sure I follow why an SRU is necessary for about half of these bugs - relating to v3 support
<rbasak> I'm not aware that we usually backport new features to old LTSes due to new features/deprecations in future OpenStack releases - that's what I thought the cloud archive was for.
<rbasak> Am I wrong, or is this case exceptional somehow?
<rafaeldtinoco> rbasak: it was a seg request
<rafaeldtinoco> because of a charm
<rafaeldtinoco> let me open the lps
<rbasak> So a statement like "The OpenStack Keystone charm supports v3 only since Queens and later" doesn't help me - why doesn't the charm support the older version?
<rafaeldtinoco> ok.. so, with xenial, the keystone charm has to use simplestreams from a ppa in order to make it work whenever updating openstack
<rafaeldtinoco> the upgrade procedure needs v3 support for not to brake
<rafaeldtinoco> because of the ordering (services)
<rafaeldtinoco> but i must confess Im relying mostly on freyes feedback
<rafaeldtinoco> freyes: ^ do you have something else to add for rbasak ?
<freyes> rbasak, >=Queens OpenStack dropped support for Keystone v2
<rbasak> freyes: and the cloud archive provides Queens against Xenial as a backport, correct?
<freyes> rbasak, correct
<rbasak> OK, so shouldn't this simplestreams v3 support to into the cloud archive, and not the Ubuntu Xenial archive, to solve that problem?
<rbasak> go
<freyes> rbasak, the cloud archive doesn't carry the simplestreams package, and if there is an intention to do it for newer releases it won't help with this bug
<rbasak> I don't think that "doesn't carry the simplestreams package" automatically means that an SRU is justified, though the SRU team might conclude that it's OK if no other solution makes sense.
<rbasak> But I don't understand why the package couldn't just be added to the cloud archive
<rbasak> Just because it's not there right now doesn't mean it can't be added right now.
<rbasak> It'd bump users up, but that's what you'd be doing with the SRU anyway.
<freyes> another reason why we may want to fix the package in distro is because simplestreams being a client, it could be that someone just wants to talk to a cloud that runs keystone v3, so in the case you propose, they would need to add a cloud archive repo (pulling a lot of new packages)
<freyes> I see pros and cons on both ways
<rbasak> That's the same situation for any network protocol client in Xenial though, OpenStack or not.
<rafaeldtinoco> a question that triggers me is..
<rafaeldtinoco> instead of a SRU to cloud archive
<rbasak> We expect such clients to use a newer release, or a snap, or some backport PPA, etc.
<rafaeldtinoco> we would have to have the backport instead
<rafaeldtinoco> so there is a 1:1 relation with newer fixes
<rbasak> (or run in a container of a newer release; the list...

Read more...

review: Disapprove

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/debian/changelog b/debian/changelog
2index 01c067e..02010cd 100644
3--- a/debian/changelog
4+++ b/debian/changelog
5@@ -1,3 +1,18 @@
6+simplestreams (0.1.0~bzr426-0ubuntu1.4) xenial; urgency=medium
7+
8+ [Scott Moser]
9+ * Pull back several upstream fixes to glance sync code
10+ - 428-do-not-require-that-hypervisor_config-be-present.patch (LP: #1578622)
11+ - 433-glance-ignore-inactive-images.patch (LP: #1583276)
12+ - 435-glance-refactor-for-testing.patch
13+ - 436-glance-fix-race-conditions.patch (LP: #1584938)
14+ - 450-453-454-keystone-v3-support.patch (LP: #1686437, #1728982, #1719879)
15+ - 455-nova-lxd-support-squashfs-images.patch (LP: #1686086)
16+ - 460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)
17+ - skip-openstack-tests-if-no-libs.patch
18+
19+ -- Rafael David Tinoco <rafaeldtinoco@ubuntu.com> Tue, 01 Oct 2019 15:45:18 -0300
20+
21 simplestreams (0.1.0~bzr426-0ubuntu1.3) xenial-proposed; urgency=medium
22
23 * Openstack: Add keystone v3 auth support (LP: #1686437).
24diff --git a/debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch b/debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch
25new file mode 100644
26index 0000000..f449dad
27--- /dev/null
28+++ b/debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch
29@@ -0,0 +1,23 @@
30+------------------------------------------------------------
31+revno: 428
32+fixes bug: https://launchpad.net/bugs/1578622
33+committer: Scott Moser <smoser@ubuntu.com>
34+branch nick: trunk
35+timestamp: Thu 2016-05-05 08:19:26 -0400
36+message:
37+ glance mirror: do not require that hypervisor_config be present
38+
39+ This just allows for hypervisor_config to not be present.
40+=== modified file 'simplestreams/mirrors/glance.py'
41+--- a/simplestreams/mirrors/glance.py 2016-03-23 10:00:53 +0000
42++++ b/simplestreams/mirrors/glance.py 2016-05-05 12:19:26 +0000
43+@@ -239,7 +239,7 @@
44+ t_item['arch'] = arch
45+ props['architecture'] = canonicalize_arch(arch)
46+
47+- if self.config['hypervisor_mapping'] and 'ftype' in flat:
48++ if self.config.get('hypervisor_mapping', False) and 'ftype' in flat:
49+ _hypervisor_type = hypervisor_type(flat['ftype'])
50+ if _hypervisor_type:
51+ props['hypervisor_type'] = _hypervisor_type
52+
53diff --git a/debian/patches/433-glance-ignore-inactive-images.patch b/debian/patches/433-glance-ignore-inactive-images.patch
54new file mode 100644
55index 0000000..ee704e0
56--- /dev/null
57+++ b/debian/patches/433-glance-ignore-inactive-images.patch
58@@ -0,0 +1,42 @@
59+------------------------------------------------------------
60+revno: 433 [merge]
61+fixes bug: https://launchpad.net/bugs/1583276
62+committer: Scott Moser <smoser@ubuntu.com>
63+branch nick: trunk
64+timestamp: Fri 2016-05-20 15:33:14 -0400
65+message:
66+ glance: ignore inactive images
67+
68+ If connection to glance is broken during GlanceMirror.sync(), and image
69+ would be left over in "status": "saving".
70+
71+ Glance itself tries to protect against that, however when it is restarted
72+ (eg. "service glance-api restart"), it does not clean the image up.
73+
74+ This change makes load_product() ignore any images that do not have
75+ "status": "active". Images can only have deleted, active or saving
76+ status.
77+
78+ Note that this isn't perfect. The image in "saving" state is never cleaned
79+ up, but at least the resulting cloud works. If we want to clean up broken
80+ images, it'd be hard to do on the simplestreams side (glance is better
81+ placed to do that: it knows if anything is going on with any of them, eg.
82+ is it being currently updated or not).
83+------------------------------------------------------------
84+Use --include-merged or -n0 to see merged revisions.
85+=== modified file 'simplestreams/mirrors/glance.py'
86+--- a/simplestreams/mirrors/glance.py 2016-05-05 12:35:42 +0000
87++++ b/simplestreams/mirrors/glance.py 2016-05-20 15:33:51 +0000
88+@@ -179,6 +179,11 @@
89+ if props.get('content_id') != my_cid:
90+ continue
91+
92++ if image.get('status') != "active":
93++ LOG.warn("Ignoring inactive image %s with status '%s'" % (
94++ image['id'], image.get('status')))
95++ continue
96++
97+ source_content_id = props.get('source_content_id')
98+
99+ product = props.get('product_name')
100+
101diff --git a/debian/patches/435-glance-refactor-for-testing.patch b/debian/patches/435-glance-refactor-for-testing.patch
102new file mode 100644
103index 0000000..404b531
104--- /dev/null
105+++ b/debian/patches/435-glance-refactor-for-testing.patch
106@@ -0,0 +1,853 @@
107+------------------------------------------------------------
108+revno: 435 [merge]
109+committer: Scott Moser <smoser@ubuntu.com>
110+branch nick: trunk
111+timestamp: Tue 2016-06-14 16:10:56 -0400
112+message:
113+ GlanceMirror: refactor insert_item for easier testing
114+
115+ This change refactors GlanceMirror.insert_item() to allow for easier and
116+ more contained testing. I needed to do this to understand everything that
117+ was going on inside insert_item and other bits of code. There are now four
118+ distinct things happening in it:
119+
120+ 1. Download image to a local file from a ContentSource
121+ 2. Construct extra properties to store in Glance along with image
122+ 3. Prepare arguments for GlanceClient.images.create() call
123+ 4. Adapt source simplestreams entry for an image for use in the target
124+ simplestreams index
125+
126+ It should be fully backwards compatible, and test coverage for all the
127+ individual steps should be much better (I admit to it not being perfect,
128+ but it's a step in the right direction, imho at least).
129+------------------------------------------------------------
130+Use --include-merged or -n0 to see merged revisions.
131+=== modified file 'simplestreams/mirrors/glance.py'
132+--- a/simplestreams/mirrors/glance.py 2016-05-20 15:33:51 +0000
133++++ b/simplestreams/mirrors/glance.py 2016-06-10 18:47:52 +0000
134+@@ -105,8 +105,15 @@
135+ # glance mirror 'image-downloads' content into glance
136+ # if provided an object store, it will produce a 'image-ids' mirror
137+ class GlanceMirror(mirrors.BasicMirrorWriter):
138++ """
139++ GlanceMirror syncs external simplestreams index and images to Glance.
140++
141++ `client` argument is used for testing to override openstack module:
142++ allows dependency injection of fake "openstack" module.
143++ """
144+ def __init__(self, config, objectstore=None, region=None,
145+- name_prefix=None, progress_callback=None):
146++ name_prefix=None, progress_callback=None,
147++ client=None):
148+ super(GlanceMirror, self).__init__(config=config)
149+
150+ self.item_filters = self.config.get('item_filters', [])
151+@@ -123,7 +130,10 @@
152+ self.loaded_content = {}
153+ self.store = objectstore
154+
155+- self.keystone_creds = openstack.load_keystone_creds()
156++ if client is None:
157++ client = openstack
158++
159++ self.keystone_creds = client.load_keystone_creds()
160+
161+ self.name_prefix = name_prefix or ""
162+ if region is not None:
163+@@ -131,8 +141,8 @@
164+
165+ self.progress_callback = progress_callback
166+
167+- conn_info = openstack.get_service_conn_info('image',
168+- **self.keystone_creds)
169++ conn_info = client.get_service_conn_info(
170++ 'image', **self.keystone_creds)
171+ self.gclient = get_glanceclient(**conn_info)
172+ self.tenant_id = conn_info['tenant_id']
173+
174+@@ -151,6 +161,12 @@
175+ return "streams/v1/%s.json" % content_id
176+
177+ def load_products(self, path=None, content_id=None):
178++ """
179++ Load metadata for all currently uploaded active images in Glance.
180++
181++ Uses glance as the definitive store, but loads metadata from existing
182++ simplestreams indexes as well.
183++ """
184+ my_cid = self.content_id
185+
186+ # glance is the definitive store. Any data loaded from the store
187+@@ -219,105 +235,212 @@
188+ def filter_item(self, data, src, target, pedigree):
189+ return filters.filter_item(self.item_filters, data, src, pedigree)
190+
191+- def insert_item(self, data, src, target, pedigree, contentsource):
192+- flat = util.products_exdata(src, pedigree, include_top=False)
193+-
194+- tmp_path = None
195+- tmp_del = None
196+-
197+- name = flat.get('pubname', flat.get('name'))
198+- if not name.endswith(flat['item_name']):
199+- name += "-%s" % (flat['item_name'])
200+-
201+- t_item = flat.copy()
202+- if 'path' in t_item:
203+- del t_item['path']
204+-
205+- props = {'content_id': target['content_id'],
206+- 'source_content_id': src['content_id']}
207+- for n in ('product_name', 'version_name', 'item_name'):
208+- props[n] = flat[n]
209+- del t_item[n]
210+-
211+- arch = flat.get('arch')
212+- if arch:
213+- t_item['arch'] = arch
214+- props['architecture'] = canonicalize_arch(arch)
215+-
216+- if self.config.get('hypervisor_mapping', False) and 'ftype' in flat:
217+- _hypervisor_type = hypervisor_type(flat['ftype'])
218++ def create_glance_properties(self, content_id, source_content_id,
219++ image_metadata, hypervisor_mapping):
220++ """
221++ Construct extra properties to store in Glance for an image.
222++
223++ Based on source image metadata.
224++ """
225++ properties = {
226++ 'content_id': content_id,
227++ 'source_content_id': source_content_id,
228++ }
229++ # An iterator of properties to carry over: if a property needs
230++ # renaming, uses a tuple (old name, new name).
231++ carry_over = (
232++ 'product_name', 'version_name', 'item_name',
233++ ('os', 'os_distro'), ('version', 'os_version'),
234++ )
235++ for carry_over_property in carry_over:
236++ if isinstance(carry_over_property, tuple):
237++ name_old, name_new = carry_over_property
238++ else:
239++ name_old = name_new = carry_over_property
240++ properties[name_new] = image_metadata[name_old]
241++
242++ if 'arch' in image_metadata:
243++ properties['architecture'] = canonicalize_arch(
244++ image_metadata['arch'])
245++
246++ if hypervisor_mapping and 'ftype' in image_metadata:
247++ _hypervisor_type = hypervisor_type(image_metadata['ftype'])
248+ if _hypervisor_type:
249+- props['hypervisor_type'] = _hypervisor_type
250+- _virt_type = virt_type(_hypervisor_type)
251+- if _virt_type:
252+- t_item['virt'] = _virt_type
253+-
254+- if 'os' in flat:
255+- props['os_distro'] = flat['os']
256+-
257+- if 'version' in flat:
258+- props['os_version'] = flat['version']
259+-
260+- fullname = self.name_prefix + name
261++ properties['hypervisor_type'] = _hypervisor_type
262++ return properties
263++
264++ def prepare_glance_arguments(self, full_image_name, image_metadata,
265++ image_md5_hash, image_size, image_properties):
266++ """
267++ Prepare arguments to pass into Glance image creation method.
268++
269++ Uses `image_metadata` for source image to derive image size, md5 hash,
270++ disk format (based on 'ftype' field, if defined, otherwise defaults to
271++ 'qcow2').
272++
273++ If `image_md5_hash` and `image_size` are defined, overrides the
274++ values from image_metadata with their values.
275++
276++ Sets extra image properties to dict `image_properties`.
277++
278++ Returns a dict to use as keyword arguments passed directly to
279++ GlanceClient.images.create().
280++ """
281+ create_kwargs = {
282+- 'name': fullname,
283+- 'properties': props,
284++ 'name': full_image_name,
285+ 'container_format': 'bare',
286+ 'is_public': True,
287++ 'properties': image_properties,
288+ }
289+- if 'size' in data:
290+- create_kwargs['size'] = data.get('size')
291+-
292+- if 'md5' in data:
293+- create_kwargs['checksum'] = data.get('md5')
294+-
295+- if 'ftype' in flat:
296++
297++ if 'size' in image_metadata:
298++ create_kwargs['size'] = image_metadata.get('size')
299++ if 'md5' in image_metadata:
300++ create_kwargs['checksum'] = image_metadata.get('md5')
301++ if image_md5_hash and image_size:
302++ create_kwargs.update({
303++ 'checksum': image_md5_hash,
304++ 'size': image_size,
305++ })
306++
307++ if 'ftype' in image_metadata:
308+ create_kwargs['disk_format'] = (
309+- disk_format(flat['ftype']) or 'qcow2'
310++ disk_format(image_metadata['ftype']) or 'qcow2'
311+ )
312+ else:
313+ create_kwargs['disk_format'] = 'qcow2'
314+
315++ return create_kwargs
316++
317++ def download_image(self, contentsource, image_stream_data):
318++ """
319++ Download an image from contentsource.
320++
321++ `image_stream_data` represents a flattened image metadata structure
322++ to use for any logging messages.
323++
324++ Returns a tuple of (local-image-path, image-size, image-md5-hash).
325++
326++ If download fails, these values will all be None.
327++ """
328++ tmp_path = new_size = new_md5 = None
329++
330++ image_name = image_stream_data.get('pubname')
331++ image_size = image_stream_data.get('size')
332++
333+ if self.progress_callback:
334+ def progress_wrapper(written):
335+ self.progress_callback(dict(status="Downloading",
336+- name=flat.get('pubname'),
337+- size=data.get('size', 0),
338++ name=image_name,
339++ size=image_size,
340+ written=written))
341+ else:
342+ def progress_wrapper(written):
343+ pass
344+
345+ try:
346+- try:
347+- (tmp_path, tmp_del) = util.get_local_copy(
348+- contentsource, progress_callback=progress_wrapper)
349+-
350+- if self.modify_hook:
351+- (newsize, newmd5) = call_hook(item=t_item, path=tmp_path,
352+- cmd=self.modify_hook)
353+- create_kwargs['checksum'] = newmd5
354+- create_kwargs['size'] = newsize
355+- t_item['md5'] = newmd5
356+- t_item['size'] = newsize
357+-
358+- finally:
359+- contentsource.close()
360+-
361++ tmp_path, _ = util.get_local_copy(
362++ contentsource, progress_callback=progress_wrapper)
363++
364++ if self.modify_hook:
365++ (new_size, new_md5) = call_hook(
366++ item=image_stream_data, path=tmp_path,
367++ cmd=self.modify_hook)
368++ finally:
369++ contentsource.close()
370++
371++ return tmp_path, new_size, new_md5
372++
373++ def adapt_source_entry(self, source_entry, hypervisor_mapping, image_name,
374++ image_md5_hash, image_size):
375++ """
376++ Adapts the source simplestreams dict `source_entry` for use in the
377++ generated local simplestreams index.
378++ """
379++ output_entry = source_entry.copy()
380++
381++ # Drop attributes not needed for the simplestreams index itself.
382++ for property_name in ('path', 'product_name', 'version_name',
383++ 'item_name'):
384++ if property_name in output_entry:
385++ del output_entry[property_name]
386++
387++ if hypervisor_mapping and 'ftype' in output_entry:
388++ _hypervisor_type = hypervisor_type(output_entry['ftype'])
389++ if _hypervisor_type:
390++ _virt_type = virt_type(_hypervisor_type)
391++ if _virt_type:
392++ output_entry['virt'] = _virt_type
393++
394++ output_entry['region'] = self.region
395++ output_entry['endpoint'] = self.auth_url
396++ output_entry['owner_id'] = self.tenant_id
397++
398++ output_entry['name'] = image_name
399++ if image_md5_hash and image_size:
400++ output_entry['md5'] = image_md5_hash
401++ output_entry['size'] = image_size
402++
403++ return output_entry
404++
405++ def insert_item(self, data, src, target, pedigree, contentsource):
406++ """
407++ Upload image into glance and add image metadata to simplestreams index.
408++
409++ `data` is the metadata for a particular image file from the source:
410++ unused since all that data is present in the `src` entry for
411++ the corresponding image as well.
412++ `src` contains the entire simplestreams index from the image syncing
413++ source.
414++ `target` is the simplestreams index for currently available images
415++ in glance (generated by load_products()) to add this item to.
416++ `pedigree` is a "path" to get to the `data` for the image we desire,
417++ a tuple of (product_name, version_name, image_type).
418++ `contentsource` is a ContentSource to download the actual image data
419++ from.
420++ """
421++ # Extract and flatten metadata for a product image matching
422++ # (product-name, version-name, image-type)
423++ # from the tuple `pedigree` in the source simplestreams index.
424++ flattened_img_data = util.products_exdata(
425++ src, pedigree, include_top=False)
426++
427++ tmp_path = None
428++
429++ full_image_name = "{}{}".format(
430++ self.name_prefix,
431++ flattened_img_data.get('pubname', flattened_img_data.get('name')))
432++ if not full_image_name.endswith(flattened_img_data['item_name']):
433++ full_image_name += "-{}".format(flattened_img_data['item_name'])
434++
435++ # Download images locally into a temporary file.
436++ tmp_path, new_size, new_md5 = self.download_image(
437++ contentsource, flattened_img_data)
438++
439++ hypervisor_mapping = self.config.get('hypervisor_mapping', False)
440++
441++ glance_props = self.create_glance_properties(
442++ target['content_id'], src['content_id'], flattened_img_data,
443++ hypervisor_mapping)
444++ create_kwargs = self.prepare_glance_arguments(
445++ full_image_name, flattened_img_data, new_md5, new_size,
446++ glance_props)
447++
448++ target_sstream_item = self.adapt_source_entry(
449++ flattened_img_data, hypervisor_mapping, full_image_name, new_md5,
450++ new_size)
451++
452++ try:
453+ create_kwargs['data'] = open(tmp_path, 'rb')
454+- ret = self.gclient.images.create(**create_kwargs)
455+- t_item['id'] = ret.id
456+- print("created %s: %s" % (ret.id, fullname))
457++ glance_image = self.gclient.images.create(**create_kwargs)
458++ target_sstream_item['id'] = glance_image.id
459++ print("created %s: %s" % (glance_image.id, full_image_name))
460+
461+ finally:
462+- if tmp_del and os.path.exists(tmp_path):
463++ if tmp_path and os.path.exists(tmp_path):
464+ os.unlink(tmp_path)
465+
466+- t_item['region'] = self.region
467+- t_item['endpoint'] = self.auth_url
468+- t_item['owner_id'] = self.tenant_id
469+- t_item['name'] = fullname
470+- util.products_set(target, t_item, pedigree)
471++ util.products_set(target, target_sstream_item, pedigree)
472+
473+ def remove_item(self, data, src, target, pedigree):
474+ util.products_del(target, pedigree)
475+
476+=== added file 'tests/unittests/test_glancemirror.py'
477+--- a/tests/unittests/test_glancemirror.py 1970-01-01 00:00:00 +0000
478++++ b/tests/unittests/test_glancemirror.py 2016-06-10 18:47:52 +0000
479+@@ -0,0 +1,479 @@
480++from simplestreams.contentsource import MemoryContentSource
481++from simplestreams.mirrors.glance import GlanceMirror
482++import simplestreams.util
483++
484++import os
485++from unittest import TestCase
486++
487++
488++class FakeOpenstack(object):
489++ """Fake 'openstack' module replacement for testing GlanceMirror."""
490++ def load_keystone_creds(self):
491++ return {"auth_url": "http://keystone/api/"}
492++
493++ def get_service_conn_info(self, url, region_name=None, auth_url=None):
494++ return {"endpoint": "http://objectstore/api/",
495++ "tenant_id": "bar456"}
496++
497++
498++class FakeImage(object):
499++ """Fake image objects returned by GlanceClient.images.create()."""
500++ def __init__(self, identifier):
501++ self.id = identifier
502++
503++
504++class FakeImages(object):
505++ """Fake GlanceClient.images implementation to track create() calls."""
506++ def __init__(self):
507++ self.create_calls = []
508++
509++ def create(self, **kwargs):
510++ self.create_calls.append(kwargs)
511++ return FakeImage('image-%d' % len(self.create_calls))
512++
513++
514++class FakeGlanceClient(object):
515++ """Fake GlanceClient implementation to track images.create() calls."""
516++ def __init__(self, *args):
517++ self.images = FakeImages()
518++
519++
520++class TestGlanceMirror(TestCase):
521++ """Tests for GlanceMirror methods."""
522++
523++ def setUp(self):
524++ self.config = {"content_id": "foo123"}
525++ self.mirror = GlanceMirror(
526++ self.config, name_prefix="auto-sync/", region="region1",
527++ client=FakeOpenstack())
528++
529++ def test_adapt_source_entry(self):
530++ # Adapts source entry for use in a local simplestreams index.
531++ source_entry = {"source-key": "source-value"}
532++ output_entry = self.mirror.adapt_source_entry(
533++ source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
534++ image_md5_hash=None, image_size=None)
535++
536++ # Source and output entry are different objects.
537++ self.assertNotEqual(source_entry, output_entry)
538++
539++ # Output entry gets a few new properties like the endpoint and
540++ # owner_id taken from the GlanceMirror and OpenStack configuration,
541++ # region from the value passed into GlanceMirror constructor, and
542++ # image name from the passed in value.
543++ # It also contains the source entries as well.
544++ self.assertEqual(
545++ {"endpoint": "http://keystone/api/",
546++ "name": "foobuntu-X",
547++ "owner_id": "bar456",
548++ "region": "region1",
549++ "source-key": "source-value"},
550++ output_entry)
551++
552++ def test_adapt_source_entry_ignored_properties(self):
553++ # adapt_source_entry() drops some properties from the source entry.
554++ source_entry = {"path": "foo",
555++ "product_name": "bar",
556++ "version_name": "baz",
557++ "item_name": "bah"}
558++ output_entry = self.mirror.adapt_source_entry(
559++ source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
560++ image_md5_hash=None, image_size=None)
561++
562++ # None of the values in 'source_entry' are preserved.
563++ for key in ("path", "product_name", "version_name", "item"):
564++ self.assertNotIn("path", output_entry)
565++
566++ def test_adapt_source_entry_image_md5_and_size(self):
567++ # adapt_source_entry() will use passed in values for md5 and size.
568++ # Even old stale values will be overridden when image_md5_hash and
569++ # image_size are passed in.
570++ source_entry = {"md5": "stale-md5"}
571++ output_entry = self.mirror.adapt_source_entry(
572++ source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
573++ image_md5_hash="new-md5", image_size=5)
574++
575++ self.assertEqual("new-md5", output_entry["md5"])
576++ self.assertEqual(5, output_entry["size"])
577++
578++ def test_adapt_source_entry_image_md5_and_size_both_required(self):
579++ # adapt_source_entry() requires both md5 and size to not ignore them.
580++
581++ source_entry = {"md5": "stale-md5"}
582++
583++ # image_size is not passed in, so md5 value is not used either.
584++ output_entry1 = self.mirror.adapt_source_entry(
585++ source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
586++ image_md5_hash="new-md5", image_size=None)
587++ self.assertEqual("stale-md5", output_entry1["md5"])
588++ self.assertNotIn("size", output_entry1)
589++
590++ # image_md5_hash is not passed in, so image_size is not used either.
591++ output_entry2 = self.mirror.adapt_source_entry(
592++ source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
593++ image_md5_hash=None, image_size=5)
594++ self.assertEqual("stale-md5", output_entry2["md5"])
595++ self.assertNotIn("size", output_entry2)
596++
597++ def test_adapt_source_entry_hypervisor_mapping(self):
598++ # If hypervisor_mapping is set to True, 'virt' value is derived from
599++ # the source entry 'ftype'.
600++ source_entry = {"ftype": "disk1.img"}
601++ output_entry = self.mirror.adapt_source_entry(
602++ source_entry, hypervisor_mapping=True, image_name="foobuntu-X",
603++ image_md5_hash=None, image_size=None)
604++
605++ self.assertEqual("kvm", output_entry["virt"])
606++
607++ def test_adapt_source_entry_hypervisor_mapping_ftype_required(self):
608++ # If hypervisor_mapping is set to True, but 'ftype' is missing in the
609++ # source entry, 'virt' value is not added to the returned entry.
610++ source_entry = {}
611++ output_entry = self.mirror.adapt_source_entry(
612++ source_entry, hypervisor_mapping=True, image_name="foobuntu-X",
613++ image_md5_hash=None, image_size=None)
614++
615++ self.assertNotIn("virt", output_entry)
616++
617++ def test_create_glance_properties(self):
618++ # Constructs glance properties to set on image during upload
619++ # based on source image metadata.
620++ source_entry = {
621++ # All of these are carried over and potentially re-named.
622++ "product_name": "foobuntu",
623++ "version_name": "X",
624++ "item_name": "disk1.img",
625++ "os": "ubuntu",
626++ "version": "16.04",
627++ # Other entries are ignored.
628++ "something-else": "ignored",
629++ }
630++ properties = self.mirror.create_glance_properties(
631++ "content-1", "source-1", source_entry, hypervisor_mapping=False)
632++
633++ # Output properties contain content-id and source-content-id based
634++ # on the passed in parameters, and carry over (with changed keys
635++ # for "os" and "version") product_name, version_name, item_name and
636++ # os and version values from the source entry.
637++ self.assertEqual(
638++ {"content_id": "content-1",
639++ "source_content_id": "source-1",
640++ "product_name": "foobuntu",
641++ "version_name": "X",
642++ "item_name": "disk1.img",
643++ "os_distro": "ubuntu",
644++ "os_version": "16.04"},
645++ properties)
646++
647++ def test_create_glance_properties_arch(self):
648++ # When 'arch' is present in the source entry, it is adapted and
649++ # returned inside 'architecture' field.
650++ source_entry = {
651++ "product_name": "foobuntu",
652++ "version_name": "X",
653++ "item_name": "disk1.img",
654++ "os": "ubuntu",
655++ "version": "16.04",
656++ "arch": "amd64",
657++ }
658++ properties = self.mirror.create_glance_properties(
659++ "content-1", "source-1", source_entry, hypervisor_mapping=False)
660++ self.assertEqual("x86_64", properties["architecture"])
661++
662++ def test_create_glance_properties_hypervisor_mapping(self):
663++ # When hypervisor_mapping is requested and 'ftype' is present in
664++ # the image metadata, 'hypervisor_type' is added to returned
665++ # properties.
666++ source_entry = {
667++ "product_name": "foobuntu",
668++ "version_name": "X",
669++ "item_name": "disk1.img",
670++ "os": "ubuntu",
671++ "version": "16.04",
672++ "ftype": "root.tar.gz",
673++ }
674++ properties = self.mirror.create_glance_properties(
675++ "content-1", "source-1", source_entry, hypervisor_mapping=True)
676++ self.assertEqual("lxc", properties["hypervisor_type"])
677++
678++ def test_prepare_glance_arguments(self):
679++ # Prepares arguments to pass to GlanceClient.images.create()
680++ # based on image metadata from the simplestreams source.
681++ source_entry = {}
682++ create_arguments = self.mirror.prepare_glance_arguments(
683++ "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
684++ image_properties=None)
685++
686++ # Arguments to always pass in contain the image name, container format,
687++ # disk format, whether image is public, and any passed-in properties.
688++ self.assertEqual(
689++ {"name": "foobuntu-X",
690++ "container_format": 'bare',
691++ "disk_format": "qcow2",
692++ "is_public": True,
693++ "properties": None},
694++ create_arguments)
695++
696++ def test_prepare_glance_arguments_disk_format(self):
697++ # Disk format is based on the image 'ftype' (if defined).
698++ source_entry = {"ftype": "root.tar.gz"}
699++ create_arguments = self.mirror.prepare_glance_arguments(
700++ "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
701++ image_properties=None)
702++
703++ self.assertEqual("root-tar", create_arguments["disk_format"])
704++
705++ def test_prepare_glance_arguments_size(self):
706++ # Size is read from image metadata if defined.
707++ source_entry = {"size": 5}
708++ create_arguments = self.mirror.prepare_glance_arguments(
709++ "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
710++ image_properties=None)
711++
712++ self.assertEqual(5, create_arguments["size"])
713++
714++ def test_prepare_glance_arguments_checksum(self):
715++ # Checksum is based on the source entry 'md5' value, if defined.
716++ source_entry = {"md5": "foo123"}
717++ create_arguments = self.mirror.prepare_glance_arguments(
718++ "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
719++ image_properties=None)
720++
721++ self.assertEqual("foo123", create_arguments["checksum"])
722++
723++ def test_prepare_glance_arguments_size_and_md5_override(self):
724++ # Size and md5 hash are overridden from the passed-in values even if
725++ # defined on the source entry.
726++ source_entry = {"size": 5, "md5": "foo123"}
727++ create_arguments = self.mirror.prepare_glance_arguments(
728++ "foobuntu-X", source_entry, image_md5_hash="bar456", image_size=10,
729++ image_properties=None)
730++
731++ self.assertEqual(10, create_arguments["size"])
732++ self.assertEqual("bar456", create_arguments["checksum"])
733++
734++ def test_prepare_glance_arguments_size_and_md5_no_override_hash(self):
735++ # If only one of image_md5_hash or image_size is passed directly in,
736++ # the other value is not overridden either.
737++ source_entry = {"size": 5, "md5": "foo123"}
738++ create_arguments = self.mirror.prepare_glance_arguments(
739++ "foobuntu-X", source_entry, image_md5_hash="bar456",
740++ image_size=None, image_properties=None)
741++
742++ self.assertEqual(5, create_arguments["size"])
743++ self.assertEqual("foo123", create_arguments["checksum"])
744++
745++ def test_prepare_glance_arguments_size_and_md5_no_override_size(self):
746++ # If only one of image_md5_hash or image_size is passed directly in,
747++ # the other value is not overridden either.
748++ source_entry = {"size": 5, "md5": "foo123"}
749++ create_arguments = self.mirror.prepare_glance_arguments(
750++ "foobuntu-X", source_entry, image_md5_hash=None, image_size=10,
751++ image_properties=None)
752++
753++ self.assertEqual(5, create_arguments["size"])
754++ self.assertEqual("foo123", create_arguments["checksum"])
755++
756++ def test_download_image(self):
757++ # Downloads image from a contentsource.
758++ content = "foo bazes the bar"
759++ content_source = MemoryContentSource(
760++ url="http://image-store/fooubuntu-X-disk1.img", content=content)
761++ image_metadata = {"pubname": "foobuntu-X", "size": 5}
762++ path, size, md5_hash = self.mirror.download_image(
763++ content_source, image_metadata)
764++ self.addCleanup(os.unlink, path)
765++ self.assertIsNotNone(path)
766++ self.assertIsNone(size)
767++ self.assertIsNone(md5_hash)
768++
769++ def test_download_image_progress_callback(self):
770++ # Progress callback is called with image name, size, status and buffer
771++ # size after every 10kb of data: 3 times for 25kb of data below.
772++ content = "abcdefghij" * int(1024 * 2.5)
773++ content_source = MemoryContentSource(
774++ url="http://image-store/fooubuntu-X-disk1.img", content=content)
775++ image_metadata = {"pubname": "foobuntu-X", "size": len(content)}
776++
777++ self.progress_calls = []
778++
779++ def log_progress_calls(message):
780++ self.progress_calls.append(message)
781++
782++ self.addCleanup(
783++ setattr, self.mirror, "progress_callback",
784++ self.mirror.progress_callback)
785++ self.mirror.progress_callback = log_progress_calls
786++ path, size, md5_hash = self.mirror.download_image(
787++ content_source, image_metadata)
788++ self.addCleanup(os.unlink, path)
789++
790++ self.assertEqual(
791++ [{"name": "foobuntu-X", "size": 25600, "status": "Downloading",
792++ "written": 10240}] * 3,
793++ self.progress_calls)
794++
795++ def test_download_image_error(self):
796++ # When there's an error during download, contentsource is still closed
797++ # and the error is propagated below.
798++ content = "abcdefghij"
799++ content_source = MemoryContentSource(
800++ url="http://image-store/fooubuntu-X-disk1.img", content=content)
801++ image_metadata = {"pubname": "foobuntu-X", "size": len(content)}
802++
803++ # MemoryContentSource has an internal file descriptor which indicates
804++ # if close() method has been called on it.
805++ self.assertFalse(content_source.fd.closed)
806++
807++ self.addCleanup(
808++ setattr, self.mirror, "progress_callback",
809++ self.mirror.progress_callback)
810++ self.mirror.progress_callback = lambda message: 1/0
811++
812++ self.assertRaises(
813++ ZeroDivisionError,
814++ self.mirror.download_image, content_source, image_metadata)
815++
816++ # We rely on the MemoryContentSource.close() side-effect to ensure
817++ # close() method has indeed been called on the passed-in ContentSource.
818++ self.assertTrue(content_source.fd.closed)
819++
820++ def test_insert_item(self):
821++ # Downloads an image from a contentsource, uploads it into Glance,
822++ # adapting and munging as needed (it updates the keystone endpoint,
823++ # image and owner ids).
824++ # This test is basically an integration test to make sure all the
825++ # methods used by insert_item() are tied together in one good
826++ # fully functioning whole.
827++
828++ # This is a real snippet from the simplestreams index entry for
829++ # Ubuntu 14.04 amd64 image from cloud-images.ubuntu.com as of
830++ # 2016-06-05.
831++ source_index = {
832++ u'content_id': u'com.ubuntu.cloud:released:download',
833++ u'datatype': u'image-downloads',
834++ u'format': u'products:1.0',
835++ u'license': (u'http://www.canonical.com/'
836++ u'intellectual-property-policy'),
837++ u'products': {u'com.ubuntu.cloud:server:14.04:amd64': {
838++ u'aliases': u'14.04,default,lts,t,trusty',
839++ u'arch': u'amd64',
840++ u'os': u'ubuntu',
841++ u'release': u'trusty',
842++ u'release_codename': u'Trusty Tahr',
843++ u'release_title': u'14.04 LTS',
844++ u'support_eol': u'2019-04-17',
845++ u'supported': True,
846++ u'version': u'14.04',
847++ u'versions': {u'20160602': {
848++ u'items': {u'disk1.img': {
849++ u'ftype': u'disk1.img',
850++ u'md5': u'e5436cd36ae6cc298f081bf0f6b413f1',
851++ u'path': (
852++ u'server/releases/trusty/release-20160602/'
853++ u'ubuntu-14.04-server-cloudimg-amd64-disk1.img'),
854++ u'sha256': (u'5b982d7d4dd1a03e88ae5f35f02ed44f'
855++ u'579e2711f3e0f27ea2bff20aef8c8d9e'),
856++ u'size': 259850752}},
857++ u'label': u'release',
858++ u'pubname': u'ubuntu-trusty-14.04-amd64-server-20160602',
859++ }}}
860++ }
861++ }
862++
863++ # "Pedigree" is basically a "path" to get to the image data in
864++ # simplestreams index, going through "products", their "versions",
865++ # and nested "items".
866++ pedigree = (
867++ u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
868++ product = source_index[u'products'][pedigree[0]]
869++ image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
870++
871++ content_source = MemoryContentSource(
872++ url="http://image-store/fooubuntu-X-disk1.img",
873++ content="image-data")
874++
875++ # Use a fake GlanceClient to track arguments passed into
876++ # GlanceClient.images.create().
877++ self.addCleanup(setattr, self.mirror, "gclient", self.mirror.gclient)
878++ self.mirror.gclient = FakeGlanceClient()
879++
880++ target = {
881++ 'content_id': 'auto.sync',
882++ 'datatype': 'image-ids',
883++ 'format': 'products:1.0',
884++ }
885++
886++ self.mirror.insert_item(
887++ image_data, source_index, target, pedigree, content_source)
888++
889++ passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
890++
891++ # There is a 'data' argument pointing to an open file descriptor
892++ # for the locally downloaded image.
893++ self.assertIn("data", passed_create_kwargs)
894++ passed_create_kwargs.pop("data")
895++
896++ expected_create_kwargs = {
897++ 'name': ('auto-sync/'
898++ 'ubuntu-trusty-14.04-amd64-server-20160602-disk1.img'),
899++ 'checksum': u'e5436cd36ae6cc298f081bf0f6b413f1',
900++ 'disk_format': 'qcow2',
901++ 'container_format': 'bare',
902++ 'is_public': True,
903++ 'properties': {
904++ 'os_distro': u'ubuntu',
905++ 'item_name': u'disk1.img',
906++ 'os_version': u'14.04',
907++ 'architecture': 'x86_64',
908++ 'version_name': u'20160602',
909++ 'content_id': 'auto.sync',
910++ 'product_name': u'com.ubuntu.cloud:server:14.04:amd64',
911++ 'source_content_id': u'com.ubuntu.cloud:released:download'},
912++ 'size': '259850752'}
913++ self.assertEqual(
914++ expected_create_kwargs, passed_create_kwargs)
915++
916++ # Almost real resulting data as produced by simplestreams before
917++ # insert_item refactoring to allow for finer-grained testing.
918++ expected_target_index = {
919++ 'content_id': 'auto.sync',
920++ 'datatype': 'image-ids',
921++ 'format': 'products:1.0',
922++ 'products': {
923++ "com.ubuntu.cloud:server:14.04:amd64": {
924++ "aliases": "14.04,default,lts,t,trusty",
925++ "arch": "amd64",
926++ "label": "release",
927++ "os": "ubuntu",
928++ "owner_id": "bar456",
929++ "pubname": "ubuntu-trusty-14.04-amd64-server-20160602",
930++ "release": "trusty",
931++ "release_codename": "Trusty Tahr",
932++ "release_title": "14.04 LTS",
933++ "support_eol": "2019-04-17",
934++ "supported": "True",
935++ "version": "14.04",
936++ "versions": {"20160602": {"items": {"disk1.img": {
937++ "endpoint": "http://keystone/api/",
938++ "ftype": "disk1.img",
939++ "id": "image-1",
940++ "md5": "e5436cd36ae6cc298f081bf0f6b413f1",
941++ "name": ("auto-sync/ubuntu-trusty-14.04-amd64-"
942++ "server-20160602-disk1.img"),
943++ "region": "region1",
944++ "sha256": ("5b982d7d4dd1a03e88ae5f35f02ed44f"
945++ "579e2711f3e0f27ea2bff20aef8c8d9e"),
946++ "size": "259850752"
947++ }}}}
948++ }
949++ }
950++ }
951++
952++ # Apply the condensing as done in GlanceMirror.insert_products()
953++ # to ensure we compare with the desired resulting simplestreams data.
954++ sticky = ['ftype', 'md5', 'sha256', 'size', 'name', 'id', 'endpoint',
955++ 'region']
956++ simplestreams.util.products_condense(target, sticky)
957++
958++ self.assertEqual(expected_target_index, target)
959+
960diff --git a/debian/patches/436-glance-fix-race-conditions.patch b/debian/patches/436-glance-fix-race-conditions.patch
961new file mode 100644
962index 0000000..ae55dbe
963--- /dev/null
964+++ b/debian/patches/436-glance-fix-race-conditions.patch
965@@ -0,0 +1,479 @@
966+------------------------------------------------------------
967+revno: 436 [merge]
968+fixes bug: https://launchpad.net/bugs/1584938
969+committer: Scott Moser <smoser@ubuntu.com>
970+branch nick: trunk
971+timestamp: Tue 2016-06-14 16:21:05 -0400
972+message:
973+ GlanceMirror: fix a couple of race-related problems
974+
975+ First, what can happen is that .sync() is interrupted for external reasons
976+ (service restarts, network issues,...) after first image has been uploaded
977+ to Glance when syncing multiple images. On re-run, the uploaded image is
978+ considered "synced" already, but since simplestreams index is written out
979+ only at the end of the sync, all the metadata is lost. We solve this by
980+ adding "simplestreams_metadata" as one of the properties of image in
981+ Glance where we put a JSON representation of all the metadata fields from
982+ the original simplestreams index entry. We then load them in
983+ load_products() if they are defined. This results in a slightly different
984+ auto.sync.json entry: "endpoint" and "region" are on the product entry,
985+ instead of on the disk1.img entry.
986+
987+ Secondly, if sync is indeed interrupted after one or several images are
988+ completely uploaded (but not all of them), there's no index file at all,
989+ so any attempt to make use of them with juju would fail, even though it
990+ should be fully functional. We solve this by calling "insert_products()"
991+ at the end of the insert_item(). This means that the index is regenerated
992+ after every single image is uploaded.
993+
994+ Along the way, I slightly improve insert_item() tests from the pre-req
995+ branch to add a test for minimal data required, rename the existing one
996+ using real world data to test_insert_item_full, and re-use that data for a
997+ test for a newly added call to insert_products().
998+------------------------------------------------------------
999+Use --include-merged or -n0 to see merged revisions.
1000+=== modified file 'simplestreams/mirrors/glance.py'
1001+--- a/simplestreams/mirrors/glance.py 2016-06-10 18:47:52 +0000
1002++++ b/simplestreams/mirrors/glance.py 2016-06-14 20:21:05 +0000
1003+@@ -25,6 +25,7 @@
1004+ import copy
1005+ import errno
1006+ import glanceclient
1007++import json
1008+ import os
1009+ import re
1010+
1011+@@ -219,6 +220,15 @@
1012+ except KeyError:
1013+ item_data = {}
1014+
1015++ # If original simplestreams-metadata is stored on the image,
1016++ # use that as well.
1017++ if 'simplestreams_metadata' in props:
1018++ simplestreams_metadata = json.loads(
1019++ props.get('simplestreams_metadata'))
1020++ else:
1021++ simplestreams_metadata = {}
1022++ item_data.update(simplestreams_metadata)
1023++
1024+ item_data.update({'name': image['name'], 'id': image['id']})
1025+ if 'owner_id' not in item_data:
1026+ item_data['owner_id'] = self.tenant_id
1027+@@ -248,10 +258,10 @@
1028+ }
1029+ # An iterator of properties to carry over: if a property needs
1030+ # renaming, uses a tuple (old name, new name).
1031+- carry_over = (
1032+- 'product_name', 'version_name', 'item_name',
1033+- ('os', 'os_distro'), ('version', 'os_version'),
1034+- )
1035++ carry_over_simple = (
1036++ 'product_name', 'version_name', 'item_name')
1037++ carry_over = carry_over_simple + (
1038++ ('os', 'os_distro'), ('version', 'os_version'))
1039+ for carry_over_property in carry_over:
1040+ if isinstance(carry_over_property, tuple):
1041+ name_old, name_new = carry_over_property
1042+@@ -267,6 +277,16 @@
1043+ _hypervisor_type = hypervisor_type(image_metadata['ftype'])
1044+ if _hypervisor_type:
1045+ properties['hypervisor_type'] = _hypervisor_type
1046++
1047++ # Store flattened metadata for a source image along with the
1048++ # image in 'simplestreams_metadata' property.
1049++ simplestreams_metadata = image_metadata.copy()
1050++ drop_keys = carry_over_simple + ('path',)
1051++ for remove_key in drop_keys:
1052++ if remove_key in simplestreams_metadata:
1053++ del simplestreams_metadata[remove_key]
1054++ properties['simplestreams_metadata'] = json.dumps(
1055++ simplestreams_metadata, sort_keys=True)
1056+ return properties
1057+
1058+ def prepare_glance_arguments(self, full_image_name, image_metadata,
1059+@@ -441,6 +461,9 @@
1060+ os.unlink(tmp_path)
1061+
1062+ util.products_set(target, target_sstream_item, pedigree)
1063++ # We can safely ignore path and content arguments since they are
1064++ # unused in insert_products below.
1065++ self.insert_products(None, target, None)
1066+
1067+ def remove_item(self, data, src, target, pedigree):
1068+ util.products_del(target, pedigree)
1069+
1070+=== modified file 'tests/unittests/test_glancemirror.py'
1071+--- a/tests/unittests/test_glancemirror.py 2016-06-10 18:47:52 +0000
1072++++ b/tests/unittests/test_glancemirror.py 2016-06-14 20:21:05 +0000
1073+@@ -1,11 +1,91 @@
1074+ from simplestreams.contentsource import MemoryContentSource
1075+ from simplestreams.mirrors.glance import GlanceMirror
1076++from simplestreams.objectstores import MemoryObjectStore
1077+ import simplestreams.util
1078+
1079++import copy
1080++import json
1081+ import os
1082+ from unittest import TestCase
1083+
1084+
1085++# This is a real snippet from the simplestreams index entry for
1086++# Ubuntu 14.04 amd64 image from cloud-images.ubuntu.com as of
1087++# 2016-06-05.
1088++TEST_SOURCE_INDEX_ENTRY = {
1089++ u'content_id': u'com.ubuntu.cloud:released:download',
1090++ u'datatype': u'image-downloads',
1091++ u'format': u'products:1.0',
1092++ u'license': (u'http://www.canonical.com/'
1093++ u'intellectual-property-policy'),
1094++ u'products': {u'com.ubuntu.cloud:server:14.04:amd64': {
1095++ u'aliases': u'14.04,default,lts,t,trusty',
1096++ u'arch': u'amd64',
1097++ u'os': u'ubuntu',
1098++ u'release': u'trusty',
1099++ u'release_codename': u'Trusty Tahr',
1100++ u'release_title': u'14.04 LTS',
1101++ u'support_eol': u'2019-04-17',
1102++ u'supported': True,
1103++ u'version': u'14.04',
1104++ u'versions': {u'20160602': {
1105++ u'items': {u'disk1.img': {
1106++ u'ftype': u'disk1.img',
1107++ u'md5': u'e5436cd36ae6cc298f081bf0f6b413f1',
1108++ u'path': (
1109++ u'server/releases/trusty/release-20160602/'
1110++ u'ubuntu-14.04-server-cloudimg-amd64-disk1.img'),
1111++ u'sha256': (u'5b982d7d4dd1a03e88ae5f35f02ed44f'
1112++ u'579e2711f3e0f27ea2bff20aef8c8d9e'),
1113++ u'size': 259850752}},
1114++ u'label': u'release',
1115++ u'pubname': u'ubuntu-trusty-14.04-amd64-server-20160602',
1116++ }}}
1117++ }
1118++}
1119++
1120++# "Pedigree" is basically a "path" to get to the image data in simplestreams
1121++# index, going through "products", their "versions", and nested "items".
1122++TEST_IMAGE_PEDIGREE = (
1123++ u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
1124++
1125++# Almost real resulting data as produced by simplestreams before
1126++# insert_item refactoring to allow for finer-grained testing.
1127++EXPECTED_OUTPUT_INDEX = {
1128++ u'content_id': u'auto.sync',
1129++ u'datatype': u'image-ids',
1130++ u'format': u'products:1.0',
1131++ u'products': {
1132++ u"com.ubuntu.cloud:server:14.04:amd64": {
1133++ u"aliases": u"14.04,default,lts,t,trusty",
1134++ u"arch": u"amd64",
1135++ u"label": u"release",
1136++ u"os": u"ubuntu",
1137++ u"owner_id": u"bar456",
1138++ u"pubname": u"ubuntu-trusty-14.04-amd64-server-20160602",
1139++ u"release": u"trusty",
1140++ u"release_codename": u"Trusty Tahr",
1141++ u"release_title": u"14.04 LTS",
1142++ u"support_eol": u"2019-04-17",
1143++ u"supported": u"True",
1144++ u"version": u"14.04",
1145++ u"versions": {u"20160602": {u"items": {u"disk1.img": {
1146++ u"endpoint": u"http://keystone/api/",
1147++ u"ftype": u"disk1.img",
1148++ u"id": u"image-1",
1149++ u"md5": u"e5436cd36ae6cc298f081bf0f6b413f1",
1150++ u"name": (u"auto-sync/ubuntu-trusty-14.04-amd64-"
1151++ u"server-20160602-disk1.img"),
1152++ u"region": u"region1",
1153++ u"sha256": (u"5b982d7d4dd1a03e88ae5f35f02ed44f"
1154++ u"579e2711f3e0f27ea2bff20aef8c8d9e"),
1155++ u"size": u"259850752"
1156++ }}}}
1157++ }
1158++ }
1159++}
1160++
1161++
1162+ class FakeOpenstack(object):
1163+ """Fake 'openstack' module replacement for testing GlanceMirror."""
1164+ def load_keystone_creds(self):
1165+@@ -145,8 +225,8 @@
1166+ "item_name": "disk1.img",
1167+ "os": "ubuntu",
1168+ "version": "16.04",
1169+- # Other entries are ignored.
1170+- "something-else": "ignored",
1171++ # Unknown entries are stored in 'simplestreams_metadata'.
1172++ "extra": "value",
1173+ }
1174+ properties = self.mirror.create_glance_properties(
1175+ "content-1", "source-1", source_entry, hypervisor_mapping=False)
1176+@@ -155,6 +235,8 @@
1177+ # on the passed in parameters, and carry over (with changed keys
1178+ # for "os" and "version") product_name, version_name, item_name and
1179+ # os and version values from the source entry.
1180++ # All the fields except product_name, version_name and item_name are
1181++ # also stored inside 'simplestreams_metadata' property as JSON data.
1182+ self.assertEqual(
1183+ {"content_id": "content-1",
1184+ "source_content_id": "source-1",
1185+@@ -162,7 +244,9 @@
1186+ "version_name": "X",
1187+ "item_name": "disk1.img",
1188+ "os_distro": "ubuntu",
1189+- "os_version": "16.04"},
1190++ "os_version": "16.04",
1191++ "simplestreams_metadata": (
1192++ '{"extra": "value", "os": "ubuntu", "version": "16.04"}')},
1193+ properties)
1194+
1195+ def test_create_glance_properties_arch(self):
1196+@@ -196,6 +280,26 @@
1197+ "content-1", "source-1", source_entry, hypervisor_mapping=True)
1198+ self.assertEqual("lxc", properties["hypervisor_type"])
1199+
1200++ def test_create_glance_properties_simplestreams_no_path(self):
1201++ # Other than 'product_name', 'version_name' and 'item_name', if 'path'
1202++ # is defined on the source entry, it is also not saved inside the
1203++ # 'simplestreams_metadata' property.
1204++ source_entry = {
1205++ "product_name": "foobuntu",
1206++ "version_name": "X",
1207++ "item_name": "disk1.img",
1208++ "os": "ubuntu",
1209++ "version": "16.04",
1210++ "path": "/path/to/foo",
1211++ }
1212++ properties = self.mirror.create_glance_properties(
1213++ "content-1", "source-1", source_entry, hypervisor_mapping=False)
1214++
1215++ # Path is omitted from the simplestreams_metadata property JSON.
1216++ self.assertEqual(
1217++ '{"os": "ubuntu", "version": "16.04"}',
1218++ properties["simplestreams_metadata"])
1219++
1220+ def test_prepare_glance_arguments(self):
1221+ # Prepares arguments to pass to GlanceClient.images.create()
1222+ # based on image metadata from the simplestreams source.
1223+@@ -342,45 +446,86 @@
1224+ # Downloads an image from a contentsource, uploads it into Glance,
1225+ # adapting and munging as needed (it updates the keystone endpoint,
1226+ # image and owner ids).
1227+- # This test is basically an integration test to make sure all the
1228+- # methods used by insert_item() are tied together in one good
1229+- # fully functioning whole.
1230+
1231+- # This is a real snippet from the simplestreams index entry for
1232+- # Ubuntu 14.04 amd64 image from cloud-images.ubuntu.com as of
1233+- # 2016-06-05.
1234++ # We use a minimal source simplestreams index, fake ContentSource and
1235++ # GlanceClient, and only test for side-effects of each of the
1236++ # subparts of the insert_item method.
1237+ source_index = {
1238+ u'content_id': u'com.ubuntu.cloud:released:download',
1239+- u'datatype': u'image-downloads',
1240+- u'format': u'products:1.0',
1241+- u'license': (u'http://www.canonical.com/'
1242+- u'intellectual-property-policy'),
1243+ u'products': {u'com.ubuntu.cloud:server:14.04:amd64': {
1244+- u'aliases': u'14.04,default,lts,t,trusty',
1245+ u'arch': u'amd64',
1246+ u'os': u'ubuntu',
1247+ u'release': u'trusty',
1248+- u'release_codename': u'Trusty Tahr',
1249+- u'release_title': u'14.04 LTS',
1250+- u'support_eol': u'2019-04-17',
1251+- u'supported': True,
1252+ u'version': u'14.04',
1253+ u'versions': {u'20160602': {
1254+ u'items': {u'disk1.img': {
1255+ u'ftype': u'disk1.img',
1256+ u'md5': u'e5436cd36ae6cc298f081bf0f6b413f1',
1257+- u'path': (
1258+- u'server/releases/trusty/release-20160602/'
1259+- u'ubuntu-14.04-server-cloudimg-amd64-disk1.img'),
1260+- u'sha256': (u'5b982d7d4dd1a03e88ae5f35f02ed44f'
1261+- u'579e2711f3e0f27ea2bff20aef8c8d9e'),
1262+ u'size': 259850752}},
1263+- u'label': u'release',
1264+ u'pubname': u'ubuntu-trusty-14.04-amd64-server-20160602',
1265+ }}}
1266+ }
1267+ }
1268+
1269++ pedigree = (
1270++ u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
1271++ product = source_index[u'products'][pedigree[0]]
1272++ image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
1273++
1274++ content_source = MemoryContentSource(
1275++ url="http://image-store/fooubuntu-X-disk1.img",
1276++ content="image-data")
1277++
1278++ # Use a fake GlanceClient to track calls and arguments passed to
1279++ # GlanceClient.images.create().
1280++ self.addCleanup(setattr, self.mirror, "gclient", self.mirror.gclient)
1281++ self.mirror.gclient = FakeGlanceClient()
1282++
1283++ target = {
1284++ 'content_id': 'auto.sync',
1285++ 'datatype': 'image-ids',
1286++ 'format': 'products:1.0',
1287++ }
1288++
1289++ self.mirror.insert_item(
1290++ image_data, source_index, target, pedigree, content_source)
1291++
1292++ passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
1293++
1294++ # There is a 'data' argument pointing to an open file descriptor
1295++ # for the locally downloaded image.
1296++ image_content = passed_create_kwargs.pop("data").read()
1297++ self.assertEqual(u"image-data", image_content.decode('utf-8'))
1298++
1299++ # Value of "arch" from source entry is transformed into "architecture"
1300++ # image property in Glance: this ensures create_glance_properties()
1301++ # is called and result is properly passed.
1302++ self.assertEqual(
1303++ "x86_64", passed_create_kwargs["properties"]["architecture"])
1304++
1305++ # MD5 hash from source entry is put into 'checksum' field, and 'name'
1306++ # is based on full image name: this ensures prepare_glance_arguments()
1307++ # is called.
1308++ self.assertEqual(
1309++ u'e5436cd36ae6cc298f081bf0f6b413f1',
1310++ passed_create_kwargs["checksum"])
1311++ self.assertEqual(
1312++ u'auto-sync/ubuntu-trusty-14.04-amd64-server-20160602-disk1.img',
1313++ passed_create_kwargs["name"])
1314++
1315++ # Our local endpoint is set in the resulting entry, which ensures
1316++ # a call to adapt_source_entry() was indeed made.
1317++ target_product = target["products"][pedigree[0]]
1318++ target_image = target_product["versions"][pedigree[1]]["items"].get(
1319++ pedigree[2])
1320++ self.assertEqual(u"http://keystone/api/", target_image["endpoint"])
1321++
1322++ def test_insert_item_full(self):
1323++ # This test uses the full sample entries from the source simplestreams
1324++ # index from cloud-images.u.c and resulting local simplestreams index
1325++ # files.
1326++ source_index = copy.deepcopy(TEST_SOURCE_INDEX_ENTRY)
1327++
1328+ # "Pedigree" is basically a "path" to get to the image data in
1329+ # simplestreams index, going through "products", their "versions",
1330+ # and nested "items".
1331+@@ -409,9 +554,7 @@
1332+
1333+ passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
1334+
1335+- # There is a 'data' argument pointing to an open file descriptor
1336+- # for the locally downloaded image.
1337+- self.assertIn("data", passed_create_kwargs)
1338++ # Drop the 'data' item pointing to an open temporary file.
1339+ passed_create_kwargs.pop("data")
1340+
1341+ expected_create_kwargs = {
1342+@@ -429,46 +572,21 @@
1343+ 'version_name': u'20160602',
1344+ 'content_id': 'auto.sync',
1345+ 'product_name': u'com.ubuntu.cloud:server:14.04:amd64',
1346++ 'simplestreams_metadata': (
1347++ '{"aliases": "14.04,default,lts,t,trusty", '
1348++ '"arch": "amd64", "ftype": "disk1.img", '
1349++ '"label": "release", "md5": '
1350++ '"e5436cd36ae6cc298f081bf0f6b413f1", "os": "ubuntu", '
1351++ '"pubname": "ubuntu-trusty-14.04-amd64-server-20160602", '
1352++ '"release": "trusty", "release_codename": "Trusty Tahr", '
1353++ '"release_title": "14.04 LTS", "sha256": '
1354++ '"5b982d7d4dd1a03e88ae5f35f02ed44f'
1355++ '579e2711f3e0f27ea2bff20aef8c8d9e", "size": "259850752", '
1356++ '"support_eol": "2019-04-17", "supported": "True", '
1357++ '"version": "14.04"}'),
1358+ 'source_content_id': u'com.ubuntu.cloud:released:download'},
1359+ 'size': '259850752'}
1360+- self.assertEqual(
1361+- expected_create_kwargs, passed_create_kwargs)
1362+-
1363+- # Almost real resulting data as produced by simplestreams before
1364+- # insert_item refactoring to allow for finer-grained testing.
1365+- expected_target_index = {
1366+- 'content_id': 'auto.sync',
1367+- 'datatype': 'image-ids',
1368+- 'format': 'products:1.0',
1369+- 'products': {
1370+- "com.ubuntu.cloud:server:14.04:amd64": {
1371+- "aliases": "14.04,default,lts,t,trusty",
1372+- "arch": "amd64",
1373+- "label": "release",
1374+- "os": "ubuntu",
1375+- "owner_id": "bar456",
1376+- "pubname": "ubuntu-trusty-14.04-amd64-server-20160602",
1377+- "release": "trusty",
1378+- "release_codename": "Trusty Tahr",
1379+- "release_title": "14.04 LTS",
1380+- "support_eol": "2019-04-17",
1381+- "supported": "True",
1382+- "version": "14.04",
1383+- "versions": {"20160602": {"items": {"disk1.img": {
1384+- "endpoint": "http://keystone/api/",
1385+- "ftype": "disk1.img",
1386+- "id": "image-1",
1387+- "md5": "e5436cd36ae6cc298f081bf0f6b413f1",
1388+- "name": ("auto-sync/ubuntu-trusty-14.04-amd64-"
1389+- "server-20160602-disk1.img"),
1390+- "region": "region1",
1391+- "sha256": ("5b982d7d4dd1a03e88ae5f35f02ed44f"
1392+- "579e2711f3e0f27ea2bff20aef8c8d9e"),
1393+- "size": "259850752"
1394+- }}}}
1395+- }
1396+- }
1397+- }
1398++ self.assertEqual(expected_create_kwargs, passed_create_kwargs)
1399+
1400+ # Apply the condensing as done in GlanceMirror.insert_products()
1401+ # to ensure we compare with the desired resulting simplestreams data.
1402+@@ -476,4 +594,40 @@
1403+ 'region']
1404+ simplestreams.util.products_condense(target, sticky)
1405+
1406+- self.assertEqual(expected_target_index, target)
1407++ self.assertEqual(EXPECTED_OUTPUT_INDEX, target)
1408++
1409++ def test_insert_item_stores_the_index(self):
1410++ # Ensure insert_item calls insert_products() to generate the
1411++ # resulting simplestreams index file and insert it into store.
1412++
1413++ source_index = copy.deepcopy(TEST_SOURCE_INDEX_ENTRY)
1414++ pedigree = TEST_IMAGE_PEDIGREE
1415++ product = source_index[u'products'][pedigree[0]]
1416++ image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
1417++
1418++ content_source = MemoryContentSource(
1419++ url="http://image-store/fooubuntu-X-disk1.img",
1420++ content="image-data")
1421++ self.mirror.store = MemoryObjectStore()
1422++
1423++ self.addCleanup(setattr, self.mirror, "gclient", self.mirror.gclient)
1424++ self.mirror.gclient = FakeGlanceClient()
1425++
1426++ target = {
1427++ 'content_id': 'auto.sync',
1428++ 'datatype': 'image-ids',
1429++ 'format': 'products:1.0',
1430++ }
1431++
1432++ self.mirror.insert_item(
1433++ image_data, source_index, target, pedigree, content_source)
1434++
1435++ stored_index_content = self.mirror.store.data[
1436++ 'streams/v1/auto.sync.json']
1437++ stored_index = json.loads(stored_index_content.decode('utf-8'))
1438++
1439++ # Full index contains the 'updated' key with the date of last update.
1440++ self.assertIn(u"updated", stored_index)
1441++ del stored_index[u"updated"]
1442++
1443++ self.assertEqual(EXPECTED_OUTPUT_INDEX, stored_index)
1444+
1445diff --git a/debian/patches/keystone-v3-support.patch b/debian/patches/450-453-454-keystone-v3-support.patch
1446similarity index 84%
1447rename from debian/patches/keystone-v3-support.patch
1448rename to debian/patches/450-453-454-keystone-v3-support.patch
1449index 3b24355..e3a3adf 100644
1450--- a/debian/patches/keystone-v3-support.patch
1451+++ b/debian/patches/450-453-454-keystone-v3-support.patch
1452@@ -15,11 +15,9 @@ Bug: https://launchpad.net/bugs/1728982
1453 Bug: https://launchpad.net/bugs/1719879
1454 Author: Scott Moser <smoser@ubuntu.com>
1455
1456-diff --git a/simplestreams/mirrors/glance.py b/simplestreams/mirrors/glance.py
1457-index 807fe29..a13fede 100644
1458 --- a/simplestreams/mirrors/glance.py
1459 +++ b/simplestreams/mirrors/glance.py
1460-@@ -35,7 +35,11 @@ def get_glanceclient(version='1', **kwargs):
1461+@@ -36,7 +36,11 @@ def get_glanceclient(version='1', **kwar
1462 kwargs['endpoint'] = _strip_version(kwargs['endpoint'])
1463 pt = ('endpoint', 'token', 'insecure', 'cacert')
1464 kskw = {k: kwargs.get(k) for k in pt if k in kwargs}
1465@@ -32,7 +30,7 @@ index 807fe29..a13fede 100644
1466
1467
1468 def empty_iid_products(content_id):
1469-@@ -64,6 +68,7 @@ def canonicalize_arch(arch):
1470+@@ -65,6 +69,7 @@ def canonicalize_arch(arch):
1471 LXC_FTYPES = [
1472 'root.tar.gz',
1473 'root.tar.xz',
1474@@ -40,8 +38,15 @@ index 807fe29..a13fede 100644
1475 ]
1476
1477 QEMU_FTYPES = [
1478-diff --git a/simplestreams/objectstores/swift.py b/simplestreams/objectstores/swift.py
1479-index d7598a7..9a6eb62 100644
1480+@@ -267,7 +272,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
1481+ name_old, name_new = carry_over_property
1482+ else:
1483+ name_old = name_new = carry_over_property
1484+- properties[name_new] = image_metadata[name_old]
1485++ properties[name_new] = image_metadata.get(name_old)
1486+
1487+ if 'arch' in image_metadata:
1488+ properties['architecture'] = canonicalize_arch(
1489 --- a/simplestreams/objectstores/swift.py
1490 +++ b/simplestreams/objectstores/swift.py
1491 @@ -33,6 +33,15 @@ def get_swiftclient(**kwargs):
1492@@ -60,8 +65,6 @@ index d7598a7..9a6eb62 100644
1493 return Connection(**connargs)
1494
1495
1496-diff --git a/simplestreams/openstack.py b/simplestreams/openstack.py
1497-index 126dea5..d143926 100644
1498 --- a/simplestreams/openstack.py
1499 +++ b/simplestreams/openstack.py
1500 @@ -15,16 +15,47 @@
1501@@ -134,7 +137,7 @@ index 126dea5..d143926 100644
1502 if missing:
1503 raise ValueError("Need values for: %s" % missing)
1504
1505-@@ -88,11 +128,49 @@ def get_regions(client=None, services=None, kscreds=None):
1506+@@ -88,11 +128,49 @@ def get_regions(client=None, services=No
1507 return list(regions)
1508
1509
1510@@ -188,7 +191,7 @@ index 126dea5..d143926 100644
1511
1512
1513 def get_service_conn_info(service='image', client=None, **kwargs):
1514-@@ -101,21 +179,27 @@ def get_service_conn_info(service='image', client=None, **kwargs):
1515+@@ -101,21 +179,27 @@ def get_service_conn_info(service='image
1516 client = get_ksclient(**kwargs)
1517
1518 endpoint = _get_endpoint(client, service, **kwargs)
1519diff --git a/debian/patches/455-nova-lxd-support-squashfs-images.patch b/debian/patches/455-nova-lxd-support-squashfs-images.patch
1520new file mode 100644
1521index 0000000..b4f0270
1522--- /dev/null
1523+++ b/debian/patches/455-nova-lxd-support-squashfs-images.patch
1524@@ -0,0 +1,230 @@
1525+------------------------------------------------------------
1526+revno: 455 [merge]
1527+fixes bug: https://launchpad.net/bugs/1686086
1528+committer: Scott Moser <smoser@ubuntu.com>
1529+branch nick: trunk
1530+timestamp: Thu 2017-11-02 15:03:37 -0400
1531+message:
1532+ OpenStack: support uploading squash images for nova-lxd.
1533+
1534+ Previously, populating a nova-lxd cloud was possible by using
1535+ root.tar.gz. A filter like:
1536+ ftype~(root.tar.gz|root.tar.xz)
1537+ would cause simplestreams to upload an image with 'disk-format' of
1538+ root-tar.
1539+
1540+ However, Ubuntu 17.04 and newer do not have root.tar.gz or root.tar.xz
1541+ images available. Currently here is what is available:
1542+ 14.04: root.tar.gz root.tar.xz
1543+ 16.04: root.tar.gz root.tar.xz squashfs
1544+ 17.10: squashfs
1545+
1546+ If we simply expected the user to change their filter to include
1547+ root.tar.xz|squashfs
1548+ Then they would get two lxd images imported for 16.04 each version.
1549+
1550+ The change here is to not do anything for an item insert, but instead
1551+ insert when the version's insert is called. Then, all the information
1552+ about what images there are is available, and it can "pick"
1553+ one or the other. Currently preference is given to the .tar.xz format.
1554+
1555+ The end result is that now users can specify an ftype filter of:
1556+ ftype~(root.tar.gz|root.tar.xz|squashfs)
1557+ and the right thing will be done.
1558+
1559+ Also here is simple knowledge that the squashfs type should be
1560+ uploaded to glance with a 'disk_format' of 'squashfs'.
1561+------------------------------------------------------------
1562+Use --include-merged or -n0 to see merged revisions.
1563+=== modified file 'simplestreams/mirrors/glance.py'
1564+--- a/simplestreams/mirrors/glance.py
1565++++ b/simplestreams/mirrors/glance.py
1566+@@ -66,25 +66,27 @@ def canonicalize_arch(arch):
1567+ return newarch
1568+
1569+
1570+-LXC_FTYPES = [
1571+- 'root.tar.gz',
1572+- 'root.tar.xz',
1573+- 'squashfs',
1574+-]
1575+-
1576+-QEMU_FTYPES = [
1577+- 'disk.img',
1578+- 'disk1.img',
1579+-]
1580++LXC_FTYPES = {
1581++ 'root.tar.gz': 'root-tar',
1582++ 'root.tar.xz': 'root-tar',
1583++ 'squashfs': 'squashfs',
1584++}
1585++
1586++QEMU_FTYPES = {
1587++ 'disk.img': 'qcow2',
1588++ 'disk1.img': 'qcow2',
1589++}
1590+
1591+
1592+ def disk_format(ftype):
1593+- '''Canonicalize disk formats for use in OpenStack'''
1594++ '''Canonicalize disk formats for use in OpenStack.
1595++ Input ftype is a 'ftype' from a simplestream feed.
1596++ Return value is the appropriate 'disk_format' for glance.'''
1597+ newftype = ftype.lower()
1598+ if newftype in LXC_FTYPES:
1599+- return 'root-tar'
1600++ return LXC_FTYPES[newftype]
1601+ if newftype in QEMU_FTYPES:
1602+- return 'qcow2'
1603++ return QEMU_FTYPES[newftype]
1604+ return None
1605+
1606+
1607+@@ -160,6 +162,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
1608+ self.content_id = config.get("content_id")
1609+ self.modify_hook = config.get("modify_hook")
1610+
1611++ self.inserts = {}
1612+ if not self.content_id:
1613+ raise TypeError("content_id is required")
1614+
1615+@@ -408,7 +411,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
1616+
1617+ return output_entry
1618+
1619+- def insert_item(self, data, src, target, pedigree, contentsource):
1620++ def _insert_item(self, data, src, target, pedigree, contentsource):
1621+ """
1622+ Upload image into glance and add image metadata to simplestreams index.
1623+
1624+@@ -470,6 +473,55 @@ class GlanceMirror(mirrors.BasicMirrorWr
1625+ # unused in insert_products below.
1626+ self.insert_products(None, target, None)
1627+
1628++ def insert_item(self, data, src, target, pedigree, contentsource):
1629++ """Queue item to be inserted in subsequent call to insert_version
1630++
1631++ This adds the item to self.inserts which is then handled in
1632++ insert_version. That allows the code to have context on
1633++ all the items for a given version, and "choose" one. Ie,
1634++ if both root.tar.xz and squashfs are available, preference
1635++ can be given to the root.tar.gz.
1636++ """
1637++
1638++ product_name, version_name, item_name = pedigree
1639++ if product_name not in self.inserts:
1640++ self.inserts[product_name] = {}
1641++ if version_name not in self.inserts[product_name]:
1642++ self.inserts[product_name][version_name] = {}
1643++
1644++ if 'ftype' in data:
1645++ ftype = data['ftype']
1646++ else:
1647++ flat = util.products_exdata(src, pedigree, include_top=False)
1648++ ftype = flat.get('ftype')
1649++ self.inserts[product_name][version_name][item_name] = (
1650++ ftype, (data, src, target, pedigree, contentsource))
1651++
1652++ def insert_version(self, data, src, target, pedigree):
1653++ """Upload all images for this version into glance
1654++ and add image metadata to simplestreams index.
1655++
1656++ All the work actually happens in _insert_item.
1657++ """
1658++
1659++ product_name, version_name = pedigree
1660++ inserts = self.inserts.get(product_name, {}).get(version_name, [])
1661++
1662++ rtar_names = [f for f in inserts
1663++ if inserts[f][0] in ('root.tar.gz', 'root.tar.xz')]
1664++
1665++ for _iname, (ftype, iargs) in inserts.items():
1666++ if ftype == "squashfs" and rtar_names:
1667++ LOG.info("[%s] Skipping ftype 'squashfs' image in preference"
1668++ "for root tarball type in %s",
1669++ '/'.join(pedigree), rtar_names)
1670++ continue
1671++ self._insert_item(*iargs)
1672++
1673++ # we do not specifically do anything for insert_version, but
1674++ # call parent.
1675++ super(GlanceMirror, self).insert_version(data, src, target, pedigree)
1676++
1677+ def remove_item(self, data, src, target, pedigree):
1678+ util.products_del(target, pedigree)
1679+ if 'id' in data:
1680+--- a/tests/unittests/test_glancemirror.py
1681++++ b/tests/unittests/test_glancemirror.py
1682+@@ -327,6 +327,15 @@ class TestGlanceMirror(TestCase):
1683+
1684+ self.assertEqual("root-tar", create_arguments["disk_format"])
1685+
1686++ def test_prepare_glance_arguments_disk_format_squashfs(self):
1687++ # squashfs images are acceptable for nova-lxd
1688++ source_entry = {"ftype": "squashfs"}
1689++ create_arguments = self.mirror.prepare_glance_arguments(
1690++ "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
1691++ image_properties=None)
1692++
1693++ self.assertEqual("squashfs", create_arguments["disk_format"])
1694++
1695+ def test_prepare_glance_arguments_size(self):
1696+ # Size is read from image metadata if defined.
1697+ source_entry = {"size": 5}
1698+@@ -470,7 +479,8 @@ class TestGlanceMirror(TestCase):
1699+ pedigree = (
1700+ u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
1701+ product = source_index[u'products'][pedigree[0]]
1702+- image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
1703++ ver_data = product[u'versions'][pedigree[1]]
1704++ image_data = ver_data[u'items'][pedigree[2]]
1705+
1706+ content_source = MemoryContentSource(
1707+ url="http://image-store/fooubuntu-X-disk1.img",
1708+@@ -489,6 +499,8 @@ class TestGlanceMirror(TestCase):
1709+
1710+ self.mirror.insert_item(
1711+ image_data, source_index, target, pedigree, content_source)
1712++ self.mirror.insert_version(
1713++ ver_data, source_index, target, pedigree[0:2])
1714+
1715+ passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
1716+
1717+@@ -532,7 +544,8 @@ class TestGlanceMirror(TestCase):
1718+ pedigree = (
1719+ u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
1720+ product = source_index[u'products'][pedigree[0]]
1721+- image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
1722++ ver_data = product[u'versions'][pedigree[1]]
1723++ image_data = ver_data[u'items'][pedigree[2]]
1724+
1725+ content_source = MemoryContentSource(
1726+ url="http://image-store/fooubuntu-X-disk1.img",
1727+@@ -551,6 +564,8 @@ class TestGlanceMirror(TestCase):
1728+
1729+ self.mirror.insert_item(
1730+ image_data, source_index, target, pedigree, content_source)
1731++ self.mirror.insert_version(
1732++ image_data, source_index, target, pedigree[0:2])
1733+
1734+ passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
1735+
1736+@@ -603,7 +618,8 @@ class TestGlanceMirror(TestCase):
1737+ source_index = copy.deepcopy(TEST_SOURCE_INDEX_ENTRY)
1738+ pedigree = TEST_IMAGE_PEDIGREE
1739+ product = source_index[u'products'][pedigree[0]]
1740+- image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
1741++ ver_data = product[u'versions'][pedigree[1]]
1742++ image_data = ver_data[u'items'][pedigree[2]]
1743+
1744+ content_source = MemoryContentSource(
1745+ url="http://image-store/fooubuntu-X-disk1.img",
1746+@@ -621,6 +637,8 @@ class TestGlanceMirror(TestCase):
1747+
1748+ self.mirror.insert_item(
1749+ image_data, source_index, target, pedigree, content_source)
1750++ self.mirror.insert_version(
1751++ ver_data, source_index, target, pedigree[0:2])
1752+
1753+ stored_index_content = self.mirror.store.data[
1754+ 'streams/v1/auto.sync.json']
1755diff --git a/debian/patches/460-glance-handle-v2-auth-with-sessions.patch b/debian/patches/460-glance-handle-v2-auth-with-sessions.patch
1756new file mode 100644
1757index 0000000..5fca027
1758--- /dev/null
1759+++ b/debian/patches/460-glance-handle-v2-auth-with-sessions.patch
1760@@ -0,0 +1,33 @@
1761+------------------------------------------------------------
1762+revno: 460 [merge]
1763+fixes bug: https://launchpad.net/bugs/1611987
1764+author: David Ames <david.ames@canonical.com>
1765+committer: Scott Moser <smoser@ubuntu.com>
1766+branch nick: trunk
1767+timestamp: Thu 2018-04-12 12:33:46 -0400
1768+message:
1769+ Glance: Handle Keystone v2 with session based authentication
1770+
1771+ There are three cases we have to handle:
1772+ - keystone v2 without sessions
1773+ - keystone v2 with sessions
1774+ - keystone v3 with sessions
1775+
1776+ We had the first and the last covered but not the middle. This change
1777+ addresses this.
1778+------------------------------------------------------------
1779+Use --include-merged or -n0 to see merged revisions.
1780+=== modified file 'simplestreams/openstack.py'
1781+--- a/simplestreams/openstack.py 2017-10-31 13:32:56 +0000
1782++++ b/simplestreams/openstack.py 2018-04-10 21:35:53 +0000
1783+@@ -181,7 +181,8 @@
1784+ endpoint = _get_endpoint(client, service, **kwargs)
1785+ # Session client does not have tenant_id set at client.tenant_id
1786+ # If client.tenant_id not set use method to get it
1787+- tenant_id = client.tenant_id or client.auth.client.get_project_id()
1788++ tenant_id = (client.tenant_id or client.get_project_id(client.session) or
1789++ client.auth.client.get_project_id())
1790+ info = {'token': client.auth_token, 'insecure': kwargs.get('insecure'),
1791+ 'cacert': kwargs.get('cacert'), 'endpoint': endpoint,
1792+ 'tenant_id': tenant_id}
1793+
1794diff --git a/debian/patches/series b/debian/patches/series
1795index 8eaeaf4..6b946bc 100644
1796--- a/debian/patches/series
1797+++ b/debian/patches/series
1798@@ -1,3 +1,10 @@
1799 read_signed-speed
1800 custom_user_agent_lp1578624.patch
1801-keystone-v3-support.patch
1802+428-do-not-require-that-hypervisor_config-be-present.patch
1803+433-glance-ignore-inactive-images.patch
1804+435-glance-refactor-for-testing.patch
1805+436-glance-fix-race-conditions.patch
1806+skip-openstack-tests-if-no-libs.patch
1807+450-453-454-keystone-v3-support.patch
1808+455-nova-lxd-support-squashfs-images.patch
1809+460-glance-handle-v2-auth-with-sessions.patch
1810diff --git a/debian/patches/skip-openstack-tests-if-no-libs.patch b/debian/patches/skip-openstack-tests-if-no-libs.patch
1811new file mode 100644
1812index 0000000..43e3665
1813--- /dev/null
1814+++ b/debian/patches/skip-openstack-tests-if-no-libs.patch
1815@@ -0,0 +1,36 @@
1816+Description: Skip tests of openstack to avoid build-depends.
1817+ This takes a bit of upstream commit revno 440 to skip openstack
1818+ tests if the libraries are not present.
1819+Applied-Upstream: revno 440
1820+Author: Scott Moser <smoser@ubuntu.com>
1821+--- a/tests/unittests/test_glancemirror.py
1822++++ b/tests/unittests/test_glancemirror.py
1823+@@ -1,12 +1,17 @@
1824+ from simplestreams.contentsource import MemoryContentSource
1825+-from simplestreams.mirrors.glance import GlanceMirror
1826+-from simplestreams.objectstores import MemoryObjectStore
1827++try:
1828++ from simplestreams.mirrors.glance import GlanceMirror
1829++ from simplestreams.objectstores import MemoryObjectStore
1830++ HAVE_OPENSTACK_LIBS = True
1831++except ImportError:
1832++ HAVE_OPENSTACK_LIBS = False
1833++
1834+ import simplestreams.util
1835+
1836+ import copy
1837+ import json
1838+ import os
1839+-from unittest import TestCase
1840++from unittest import TestCase, skipIf
1841+
1842+
1843+ # This is a real snippet from the simplestreams index entry for
1844+@@ -118,6 +123,7 @@ class FakeGlanceClient(object):
1845+ self.images = FakeImages()
1846+
1847+
1848++@skipIf(not HAVE_OPENSTACK_LIBS, "no python3 openstack available")
1849+ class TestGlanceMirror(TestCase):
1850+ """Tests for GlanceMirror methods."""
1851+

Subscribers

People subscribed via source and target branches