Merge ~smoser/ubuntu/+source/simplestreams:bug/xenial-1686437-keystone-v3 into ubuntu/+source/simplestreams:ubuntu/xenial-devel

Proposed by Scott Moser
Status: Needs review
Proposed branch: ~smoser/ubuntu/+source/simplestreams:bug/xenial-1686437-keystone-v3
Merge into: ubuntu/+source/simplestreams:ubuntu/xenial-devel
Diff against target: 1845 lines (+1729/-11)
10 files modified
debian/changelog (+12/-0)
debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch (+23/-0)
debian/patches/433-glance-ignore-inactive-images.patch (+42/-0)
debian/patches/435-glance-refactor-for-testing.patch (+853/-0)
debian/patches/436-glance-fix-race-conditions.patch (+479/-0)
debian/patches/450-453-454-keystone-v3-support.patch (+13/-10)
debian/patches/455-nova-lxd-support-squashfs-images.patch (+230/-0)
debian/patches/460-glance-handle-v2-auth-with-sessions.patch (+33/-0)
debian/patches/series (+8/-1)
debian/patches/skip-openstack-tests-if-no-libs.patch (+36/-0)
Reviewer Review Type Date Requested Status
Rafael David Tinoco Disapprove
Billy Olsen (community) Approve
Scott Moser (community) Needs Resubmitting
Felipe Reyes (community) Approve
Eric Desrochers Pending
Review via email: mp+341214@code.launchpad.net

Description of the change

Openstack keystone v3 and nova-lxd squashfs SRU

Bugs:
 * bug 1578622: do not require that hypervisor_config be present.
 * bug 1583276: ignore inactive images
 * bug 1584938: glance fix race conditions in sync.
 * bug 1611987: glance-simplestreams-sync charm doesn't support keystone v3
 * bug 1686086: glance mirror and nova-lxd need support for squashfs images
 * bug 1686437: glance sync: need keystone v3 auth support
 * bug 1719879: [artful only] swift client needs to use v1 auth prior to ocata
 * bug 1728982: [artful only] openstack mirror with keystone v3 always imports new images

Merge proposals:
 - xenial: https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341214
 - artful: https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341215

PPA:
  https://launchpad.net/~smoser/+archive/ubuntu/sstream-ks3

To post a comment you must log in.
Revision history for this message
Felipe Reyes (freyes) wrote :

I tested this patch using the procedure described in the test case of bug 1686437 . To make the testing easier I created a ppa ( https://launchpad.net/~freyes/+archive/ubuntu/lp1686437 ). After installing this version running the sync up script successfully finished (connecting to keystone using v3 as verified in the logs)

# /etc/cron.daily/glance_simplestreams_sync
created c99bb337-cb1e-4873-9420-11920d911eee: auto-sync/ubuntu-trusty-14.04-amd64-server-20180404-disk1.img
created 4073e5b1-6dde-4534-8f16-5c7ea126d637: auto-sync/ubuntu-xenial-16.04-amd64-server-20180405-disk1.img

/var/log/glance-simplestreams-sync.log -> http://paste.ubuntu.com/p/7VWYsWKx4y/

review: Approve
Revision history for this message
David Ames (thedac) wrote :

Also Noting here the released version on xenial does not currently support Keystone v3 and blocks Bug #1611987.

For the record, we have been running a bzr branch @455 on serverstack (a keystone v3 cloud) for months now. So the code in simplestreams works.

Revision history for this message
Billy Olsen (billy-olsen) wrote :

The patch looks good to me, but I have a question inlined below regarding the addition of the squashfs format and the implications for syncing multiple image types into a nova lxd cloud using the ftype filter.

Revision history for this message
Scott Moser (smoser) wrote :

Billy made a good point. If we're going to pull this back we should grab revno 455 also.

review: Needs Fixing
Revision history for this message
Scott Moser (smoser) :
review: Needs Resubmitting
Revision history for this message
Scott Moser (smoser) wrote :

Billy and all,
Please re-review and ideally test the PPA upload.

Thanks.
Scott

Revision history for this message
Billy Olsen (billy-olsen) wrote :

Scott, thanks for pulling that back. Felipe is going to rebuild the package in the PPA with this latest patch pulled in. Code looks fine to me so I'll approve. I'll let Felipe comment on the testing of the PPA package.

review: Approve
a55c24b... by Scott Moser on 2018-04-12

Pull back glance related fixes from upstream.

This pulls back glance related changes up to upstream revision 455.

428-do-not-require-that-hypervisor_config-be-present.patch (LP: #1578622)
433-glance-ignore-inactive-images.patch (LP: #1583276)
436-glance-fix-race-conditions.patch (LP: #1584938)
450-453-454-keystone-v3-support.patch (LP: #1686437, #1728982, #1719879)
455-nova-lxd-support-squashfs-images.patch (LP: #1686086)

db8b81f... by Scott Moser on 2018-04-12

update changelog

7aa5026... by Scott Moser on 2018-04-12

Pull in revno 460 for keystone v2 session support.

  460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)

0ce3c41... by Scott Moser on 2018-04-12

update changelog

bb79379... by Scott Moser on 2018-04-24

mark 1.4 released

d197243... by Scott Moser on 2018-04-24

1.4~ppa0

Revision history for this message
Scott Moser (smoser) wrote :

an upload that I had done got into xenial for bug 1686437.
I've rebased this merge over the top of that.

Same content as before except for debian/changelog.
I've uploaded that to the ppa as 0.1.0~bzr426-0ubuntu1.4~ppa0 .
https://launchpad.net/~smoser/+archive/ubuntu/sstream-ks3/+packages

So from a code perspective, I'm pretty sure that all parties are on board with 0.1.0~bzr426-0ubuntu1.4 being uploaded... but we need to address SRU.

It now references 8 bugs, only 1 of which has an SRU template written.

Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :
review: Disapprove

Unmerged commits

d197243... by Scott Moser on 2018-04-24

1.4~ppa0

bb79379... by Scott Moser on 2018-04-24

mark 1.4 released

0ce3c41... by Scott Moser on 2018-04-12

update changelog

7aa5026... by Scott Moser on 2018-04-12

Pull in revno 460 for keystone v2 session support.

  460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)

db8b81f... by Scott Moser on 2018-04-12

update changelog

a55c24b... by Scott Moser on 2018-04-12

Pull back glance related fixes from upstream.

This pulls back glance related changes up to upstream revision 455.

428-do-not-require-that-hypervisor_config-be-present.patch (LP: #1578622)
433-glance-ignore-inactive-images.patch (LP: #1583276)
436-glance-fix-race-conditions.patch (LP: #1584938)
450-453-454-keystone-v3-support.patch (LP: #1686437, #1728982, #1719879)
455-nova-lxd-support-squashfs-images.patch (LP: #1686086)

b21bc99... by Scott Moser on 2018-03-09

Import patches-unapplied version 0.1.0~bzr426-0ubuntu1.3 to ubuntu/xenial-proposed

Imported using git-ubuntu import.

Changelog parent: 42652aa520c7856e51d365c9e158124ace8d81a5

New changelog entries:
  * Openstack: Add keystone v3 auth support (LP: #1686437).

42652aa... by Andres Rodriguez on 2016-11-30

Import patches-unapplied version 0.1.0~bzr426-0ubuntu1.2 to ubuntu/xenial-proposed

Imported using git-ubuntu import.

Changelog parent: 353cf1dc2f62eb63936244e8b92c51c29b0d1e05

New changelog entries:
  * [SRU] Set custom user agent (LP: #1578624)

353cf1d... by Robie Basak on 2016-05-12

Import patches-unapplied version 0.1.0~bzr426-0ubuntu1.1 to ubuntu/xenial-proposed

Imported using git-ubuntu import.

Changelog parent: c2750bbdf3e9500a2da0c5f5503cb85d9a59fa00

New changelog entries:
  * Fix signature verification speed (LP: #1580534).

c2750bb... by Scott Moser on 2016-03-23

Import patches-unapplied version 0.1.0~bzr426-0ubuntu1 to ubuntu/xenial-proposed

Imported using git-ubuntu import.

Changelog parent: 69feb70fd3bbe7f44bc5c20a21d844a83e2d407f

New changelog entries:
  * debian/README.source, debian/new-upstream-snapshot: add
    script to do what README.source said to do before.
  * New upstream snapshot.
    - glance mirror: use 'virt' key for hypervisor type and use names
      'kvm' and 'lxd' rather than 'qemu' and 'lxc' (LP: #1560903)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/debian/changelog b/debian/changelog
2index 01c067e..8bffd8a 100644
3--- a/debian/changelog
4+++ b/debian/changelog
5@@ -1,3 +1,15 @@
6+simplestreams (0.1.0~bzr426-0ubuntu1.4~ppa0) xenial; urgency=medium
7+
8+ * Pull back several upstream fixes to glance sync code:
9+ - 428-do-not-require-that-hypervisor_config-be-present.patch (LP: #1578622)
10+ - 433-glance-ignore-inactive-images.patch (LP: #1583276)
11+ - 436-glance-fix-race-conditions.patch (LP: #1584938)
12+ - 450-453-454-keystone-v3-support.patch (LP: #1686437, #1728982, #1719879)
13+ - 455-nova-lxd-support-squashfs-images.patch (LP: #1686086)
14+ - 460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)
15+
16+ -- Scott Moser <smoser@ubuntu.com> Thu, 12 Apr 2018 12:27:47 -0400
17+
18 simplestreams (0.1.0~bzr426-0ubuntu1.3) xenial-proposed; urgency=medium
19
20 * Openstack: Add keystone v3 auth support (LP: #1686437).
21diff --git a/debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch b/debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch
22new file mode 100644
23index 0000000..f449dad
24--- /dev/null
25+++ b/debian/patches/428-do-not-require-that-hypervisor_config-be-present.patch
26@@ -0,0 +1,23 @@
27+------------------------------------------------------------
28+revno: 428
29+fixes bug: https://launchpad.net/bugs/1578622
30+committer: Scott Moser <smoser@ubuntu.com>
31+branch nick: trunk
32+timestamp: Thu 2016-05-05 08:19:26 -0400
33+message:
34+ glance mirror: do not require that hypervisor_config be present
35+
36+ This just allows for hypervisor_config to not be present.
37+=== modified file 'simplestreams/mirrors/glance.py'
38+--- a/simplestreams/mirrors/glance.py 2016-03-23 10:00:53 +0000
39++++ b/simplestreams/mirrors/glance.py 2016-05-05 12:19:26 +0000
40+@@ -239,7 +239,7 @@
41+ t_item['arch'] = arch
42+ props['architecture'] = canonicalize_arch(arch)
43+
44+- if self.config['hypervisor_mapping'] and 'ftype' in flat:
45++ if self.config.get('hypervisor_mapping', False) and 'ftype' in flat:
46+ _hypervisor_type = hypervisor_type(flat['ftype'])
47+ if _hypervisor_type:
48+ props['hypervisor_type'] = _hypervisor_type
49+
50diff --git a/debian/patches/433-glance-ignore-inactive-images.patch b/debian/patches/433-glance-ignore-inactive-images.patch
51new file mode 100644
52index 0000000..ee704e0
53--- /dev/null
54+++ b/debian/patches/433-glance-ignore-inactive-images.patch
55@@ -0,0 +1,42 @@
56+------------------------------------------------------------
57+revno: 433 [merge]
58+fixes bug: https://launchpad.net/bugs/1583276
59+committer: Scott Moser <smoser@ubuntu.com>
60+branch nick: trunk
61+timestamp: Fri 2016-05-20 15:33:14 -0400
62+message:
63+ glance: ignore inactive images
64+
65+ If connection to glance is broken during GlanceMirror.sync(), and image
66+ would be left over in "status": "saving".
67+
68+ Glance itself tries to protect against that, however when it is restarted
69+ (eg. "service glance-api restart"), it does not clean the image up.
70+
71+ This change makes load_product() ignore any images that do not have
72+ "status": "active". Images can only have deleted, active or saving
73+ status.
74+
75+ Note that this isn't perfect. The image in "saving" state is never cleaned
76+ up, but at least the resulting cloud works. If we want to clean up broken
77+ images, it'd be hard to do on the simplestreams side (glance is better
78+ placed to do that: it knows if anything is going on with any of them, eg.
79+ is it being currently updated or not).
80+------------------------------------------------------------
81+Use --include-merged or -n0 to see merged revisions.
82+=== modified file 'simplestreams/mirrors/glance.py'
83+--- a/simplestreams/mirrors/glance.py 2016-05-05 12:35:42 +0000
84++++ b/simplestreams/mirrors/glance.py 2016-05-20 15:33:51 +0000
85+@@ -179,6 +179,11 @@
86+ if props.get('content_id') != my_cid:
87+ continue
88+
89++ if image.get('status') != "active":
90++ LOG.warn("Ignoring inactive image %s with status '%s'" % (
91++ image['id'], image.get('status')))
92++ continue
93++
94+ source_content_id = props.get('source_content_id')
95+
96+ product = props.get('product_name')
97+
98diff --git a/debian/patches/435-glance-refactor-for-testing.patch b/debian/patches/435-glance-refactor-for-testing.patch
99new file mode 100644
100index 0000000..404b531
101--- /dev/null
102+++ b/debian/patches/435-glance-refactor-for-testing.patch
103@@ -0,0 +1,853 @@
104+------------------------------------------------------------
105+revno: 435 [merge]
106+committer: Scott Moser <smoser@ubuntu.com>
107+branch nick: trunk
108+timestamp: Tue 2016-06-14 16:10:56 -0400
109+message:
110+ GlanceMirror: refactor insert_item for easier testing
111+
112+ This change refactors GlanceMirror.insert_item() to allow for easier and
113+ more contained testing. I needed to do this to understand everything that
114+ was going on inside insert_item and other bits of code. There are now four
115+ distinct things happening in it:
116+
117+ 1. Download image to a local file from a ContentSource
118+ 2. Construct extra properties to store in Glance along with image
119+ 3. Prepare arguments for GlanceClient.images.create() call
120+ 4. Adapt source simplestreams entry for an image for use in the target
121+ simplestreams index
122+
123+ It should be fully backwards compatible, and test coverage for all the
124+ individual steps should be much better (I admit to it not being perfect,
125+ but it's a step in the right direction, imho at least).
126+------------------------------------------------------------
127+Use --include-merged or -n0 to see merged revisions.
128+=== modified file 'simplestreams/mirrors/glance.py'
129+--- a/simplestreams/mirrors/glance.py 2016-05-20 15:33:51 +0000
130++++ b/simplestreams/mirrors/glance.py 2016-06-10 18:47:52 +0000
131+@@ -105,8 +105,15 @@
132+ # glance mirror 'image-downloads' content into glance
133+ # if provided an object store, it will produce a 'image-ids' mirror
134+ class GlanceMirror(mirrors.BasicMirrorWriter):
135++ """
136++ GlanceMirror syncs external simplestreams index and images to Glance.
137++
138++ `client` argument is used for testing to override openstack module:
139++ allows dependency injection of fake "openstack" module.
140++ """
141+ def __init__(self, config, objectstore=None, region=None,
142+- name_prefix=None, progress_callback=None):
143++ name_prefix=None, progress_callback=None,
144++ client=None):
145+ super(GlanceMirror, self).__init__(config=config)
146+
147+ self.item_filters = self.config.get('item_filters', [])
148+@@ -123,7 +130,10 @@
149+ self.loaded_content = {}
150+ self.store = objectstore
151+
152+- self.keystone_creds = openstack.load_keystone_creds()
153++ if client is None:
154++ client = openstack
155++
156++ self.keystone_creds = client.load_keystone_creds()
157+
158+ self.name_prefix = name_prefix or ""
159+ if region is not None:
160+@@ -131,8 +141,8 @@
161+
162+ self.progress_callback = progress_callback
163+
164+- conn_info = openstack.get_service_conn_info('image',
165+- **self.keystone_creds)
166++ conn_info = client.get_service_conn_info(
167++ 'image', **self.keystone_creds)
168+ self.gclient = get_glanceclient(**conn_info)
169+ self.tenant_id = conn_info['tenant_id']
170+
171+@@ -151,6 +161,12 @@
172+ return "streams/v1/%s.json" % content_id
173+
174+ def load_products(self, path=None, content_id=None):
175++ """
176++ Load metadata for all currently uploaded active images in Glance.
177++
178++ Uses glance as the definitive store, but loads metadata from existing
179++ simplestreams indexes as well.
180++ """
181+ my_cid = self.content_id
182+
183+ # glance is the definitive store. Any data loaded from the store
184+@@ -219,105 +235,212 @@
185+ def filter_item(self, data, src, target, pedigree):
186+ return filters.filter_item(self.item_filters, data, src, pedigree)
187+
188+- def insert_item(self, data, src, target, pedigree, contentsource):
189+- flat = util.products_exdata(src, pedigree, include_top=False)
190+-
191+- tmp_path = None
192+- tmp_del = None
193+-
194+- name = flat.get('pubname', flat.get('name'))
195+- if not name.endswith(flat['item_name']):
196+- name += "-%s" % (flat['item_name'])
197+-
198+- t_item = flat.copy()
199+- if 'path' in t_item:
200+- del t_item['path']
201+-
202+- props = {'content_id': target['content_id'],
203+- 'source_content_id': src['content_id']}
204+- for n in ('product_name', 'version_name', 'item_name'):
205+- props[n] = flat[n]
206+- del t_item[n]
207+-
208+- arch = flat.get('arch')
209+- if arch:
210+- t_item['arch'] = arch
211+- props['architecture'] = canonicalize_arch(arch)
212+-
213+- if self.config.get('hypervisor_mapping', False) and 'ftype' in flat:
214+- _hypervisor_type = hypervisor_type(flat['ftype'])
215++ def create_glance_properties(self, content_id, source_content_id,
216++ image_metadata, hypervisor_mapping):
217++ """
218++ Construct extra properties to store in Glance for an image.
219++
220++ Based on source image metadata.
221++ """
222++ properties = {
223++ 'content_id': content_id,
224++ 'source_content_id': source_content_id,
225++ }
226++ # An iterator of properties to carry over: if a property needs
227++ # renaming, uses a tuple (old name, new name).
228++ carry_over = (
229++ 'product_name', 'version_name', 'item_name',
230++ ('os', 'os_distro'), ('version', 'os_version'),
231++ )
232++ for carry_over_property in carry_over:
233++ if isinstance(carry_over_property, tuple):
234++ name_old, name_new = carry_over_property
235++ else:
236++ name_old = name_new = carry_over_property
237++ properties[name_new] = image_metadata[name_old]
238++
239++ if 'arch' in image_metadata:
240++ properties['architecture'] = canonicalize_arch(
241++ image_metadata['arch'])
242++
243++ if hypervisor_mapping and 'ftype' in image_metadata:
244++ _hypervisor_type = hypervisor_type(image_metadata['ftype'])
245+ if _hypervisor_type:
246+- props['hypervisor_type'] = _hypervisor_type
247+- _virt_type = virt_type(_hypervisor_type)
248+- if _virt_type:
249+- t_item['virt'] = _virt_type
250+-
251+- if 'os' in flat:
252+- props['os_distro'] = flat['os']
253+-
254+- if 'version' in flat:
255+- props['os_version'] = flat['version']
256+-
257+- fullname = self.name_prefix + name
258++ properties['hypervisor_type'] = _hypervisor_type
259++ return properties
260++
261++ def prepare_glance_arguments(self, full_image_name, image_metadata,
262++ image_md5_hash, image_size, image_properties):
263++ """
264++ Prepare arguments to pass into Glance image creation method.
265++
266++ Uses `image_metadata` for source image to derive image size, md5 hash,
267++ disk format (based on 'ftype' field, if defined, otherwise defaults to
268++ 'qcow2').
269++
270++ If `image_md5_hash` and `image_size` are defined, overrides the
271++ values from image_metadata with their values.
272++
273++ Sets extra image properties to dict `image_properties`.
274++
275++ Returns a dict to use as keyword arguments passed directly to
276++ GlanceClient.images.create().
277++ """
278+ create_kwargs = {
279+- 'name': fullname,
280+- 'properties': props,
281++ 'name': full_image_name,
282+ 'container_format': 'bare',
283+ 'is_public': True,
284++ 'properties': image_properties,
285+ }
286+- if 'size' in data:
287+- create_kwargs['size'] = data.get('size')
288+-
289+- if 'md5' in data:
290+- create_kwargs['checksum'] = data.get('md5')
291+-
292+- if 'ftype' in flat:
293++
294++ if 'size' in image_metadata:
295++ create_kwargs['size'] = image_metadata.get('size')
296++ if 'md5' in image_metadata:
297++ create_kwargs['checksum'] = image_metadata.get('md5')
298++ if image_md5_hash and image_size:
299++ create_kwargs.update({
300++ 'checksum': image_md5_hash,
301++ 'size': image_size,
302++ })
303++
304++ if 'ftype' in image_metadata:
305+ create_kwargs['disk_format'] = (
306+- disk_format(flat['ftype']) or 'qcow2'
307++ disk_format(image_metadata['ftype']) or 'qcow2'
308+ )
309+ else:
310+ create_kwargs['disk_format'] = 'qcow2'
311+
312++ return create_kwargs
313++
314++ def download_image(self, contentsource, image_stream_data):
315++ """
316++ Download an image from contentsource.
317++
318++ `image_stream_data` represents a flattened image metadata structure
319++ to use for any logging messages.
320++
321++ Returns a tuple of (local-image-path, image-size, image-md5-hash).
322++
323++ If download fails, these values will all be None.
324++ """
325++ tmp_path = new_size = new_md5 = None
326++
327++ image_name = image_stream_data.get('pubname')
328++ image_size = image_stream_data.get('size')
329++
330+ if self.progress_callback:
331+ def progress_wrapper(written):
332+ self.progress_callback(dict(status="Downloading",
333+- name=flat.get('pubname'),
334+- size=data.get('size', 0),
335++ name=image_name,
336++ size=image_size,
337+ written=written))
338+ else:
339+ def progress_wrapper(written):
340+ pass
341+
342+ try:
343+- try:
344+- (tmp_path, tmp_del) = util.get_local_copy(
345+- contentsource, progress_callback=progress_wrapper)
346+-
347+- if self.modify_hook:
348+- (newsize, newmd5) = call_hook(item=t_item, path=tmp_path,
349+- cmd=self.modify_hook)
350+- create_kwargs['checksum'] = newmd5
351+- create_kwargs['size'] = newsize
352+- t_item['md5'] = newmd5
353+- t_item['size'] = newsize
354+-
355+- finally:
356+- contentsource.close()
357+-
358++ tmp_path, _ = util.get_local_copy(
359++ contentsource, progress_callback=progress_wrapper)
360++
361++ if self.modify_hook:
362++ (new_size, new_md5) = call_hook(
363++ item=image_stream_data, path=tmp_path,
364++ cmd=self.modify_hook)
365++ finally:
366++ contentsource.close()
367++
368++ return tmp_path, new_size, new_md5
369++
370++ def adapt_source_entry(self, source_entry, hypervisor_mapping, image_name,
371++ image_md5_hash, image_size):
372++ """
373++ Adapts the source simplestreams dict `source_entry` for use in the
374++ generated local simplestreams index.
375++ """
376++ output_entry = source_entry.copy()
377++
378++ # Drop attributes not needed for the simplestreams index itself.
379++ for property_name in ('path', 'product_name', 'version_name',
380++ 'item_name'):
381++ if property_name in output_entry:
382++ del output_entry[property_name]
383++
384++ if hypervisor_mapping and 'ftype' in output_entry:
385++ _hypervisor_type = hypervisor_type(output_entry['ftype'])
386++ if _hypervisor_type:
387++ _virt_type = virt_type(_hypervisor_type)
388++ if _virt_type:
389++ output_entry['virt'] = _virt_type
390++
391++ output_entry['region'] = self.region
392++ output_entry['endpoint'] = self.auth_url
393++ output_entry['owner_id'] = self.tenant_id
394++
395++ output_entry['name'] = image_name
396++ if image_md5_hash and image_size:
397++ output_entry['md5'] = image_md5_hash
398++ output_entry['size'] = image_size
399++
400++ return output_entry
401++
402++ def insert_item(self, data, src, target, pedigree, contentsource):
403++ """
404++ Upload image into glance and add image metadata to simplestreams index.
405++
406++ `data` is the metadata for a particular image file from the source:
407++ unused since all that data is present in the `src` entry for
408++ the corresponding image as well.
409++ `src` contains the entire simplestreams index from the image syncing
410++ source.
411++ `target` is the simplestreams index for currently available images
412++ in glance (generated by load_products()) to add this item to.
413++ `pedigree` is a "path" to get to the `data` for the image we desire,
414++ a tuple of (product_name, version_name, image_type).
415++ `contentsource` is a ContentSource to download the actual image data
416++ from.
417++ """
418++ # Extract and flatten metadata for a product image matching
419++ # (product-name, version-name, image-type)
420++ # from the tuple `pedigree` in the source simplestreams index.
421++ flattened_img_data = util.products_exdata(
422++ src, pedigree, include_top=False)
423++
424++ tmp_path = None
425++
426++ full_image_name = "{}{}".format(
427++ self.name_prefix,
428++ flattened_img_data.get('pubname', flattened_img_data.get('name')))
429++ if not full_image_name.endswith(flattened_img_data['item_name']):
430++ full_image_name += "-{}".format(flattened_img_data['item_name'])
431++
432++ # Download images locally into a temporary file.
433++ tmp_path, new_size, new_md5 = self.download_image(
434++ contentsource, flattened_img_data)
435++
436++ hypervisor_mapping = self.config.get('hypervisor_mapping', False)
437++
438++ glance_props = self.create_glance_properties(
439++ target['content_id'], src['content_id'], flattened_img_data,
440++ hypervisor_mapping)
441++ create_kwargs = self.prepare_glance_arguments(
442++ full_image_name, flattened_img_data, new_md5, new_size,
443++ glance_props)
444++
445++ target_sstream_item = self.adapt_source_entry(
446++ flattened_img_data, hypervisor_mapping, full_image_name, new_md5,
447++ new_size)
448++
449++ try:
450+ create_kwargs['data'] = open(tmp_path, 'rb')
451+- ret = self.gclient.images.create(**create_kwargs)
452+- t_item['id'] = ret.id
453+- print("created %s: %s" % (ret.id, fullname))
454++ glance_image = self.gclient.images.create(**create_kwargs)
455++ target_sstream_item['id'] = glance_image.id
456++ print("created %s: %s" % (glance_image.id, full_image_name))
457+
458+ finally:
459+- if tmp_del and os.path.exists(tmp_path):
460++ if tmp_path and os.path.exists(tmp_path):
461+ os.unlink(tmp_path)
462+
463+- t_item['region'] = self.region
464+- t_item['endpoint'] = self.auth_url
465+- t_item['owner_id'] = self.tenant_id
466+- t_item['name'] = fullname
467+- util.products_set(target, t_item, pedigree)
468++ util.products_set(target, target_sstream_item, pedigree)
469+
470+ def remove_item(self, data, src, target, pedigree):
471+ util.products_del(target, pedigree)
472+
473+=== added file 'tests/unittests/test_glancemirror.py'
474+--- a/tests/unittests/test_glancemirror.py 1970-01-01 00:00:00 +0000
475++++ b/tests/unittests/test_glancemirror.py 2016-06-10 18:47:52 +0000
476+@@ -0,0 +1,479 @@
477++from simplestreams.contentsource import MemoryContentSource
478++from simplestreams.mirrors.glance import GlanceMirror
479++import simplestreams.util
480++
481++import os
482++from unittest import TestCase
483++
484++
485++class FakeOpenstack(object):
486++ """Fake 'openstack' module replacement for testing GlanceMirror."""
487++ def load_keystone_creds(self):
488++ return {"auth_url": "http://keystone/api/"}
489++
490++ def get_service_conn_info(self, url, region_name=None, auth_url=None):
491++ return {"endpoint": "http://objectstore/api/",
492++ "tenant_id": "bar456"}
493++
494++
495++class FakeImage(object):
496++ """Fake image objects returned by GlanceClient.images.create()."""
497++ def __init__(self, identifier):
498++ self.id = identifier
499++
500++
501++class FakeImages(object):
502++ """Fake GlanceClient.images implementation to track create() calls."""
503++ def __init__(self):
504++ self.create_calls = []
505++
506++ def create(self, **kwargs):
507++ self.create_calls.append(kwargs)
508++ return FakeImage('image-%d' % len(self.create_calls))
509++
510++
511++class FakeGlanceClient(object):
512++ """Fake GlanceClient implementation to track images.create() calls."""
513++ def __init__(self, *args):
514++ self.images = FakeImages()
515++
516++
517++class TestGlanceMirror(TestCase):
518++ """Tests for GlanceMirror methods."""
519++
520++ def setUp(self):
521++ self.config = {"content_id": "foo123"}
522++ self.mirror = GlanceMirror(
523++ self.config, name_prefix="auto-sync/", region="region1",
524++ client=FakeOpenstack())
525++
526++ def test_adapt_source_entry(self):
527++ # Adapts source entry for use in a local simplestreams index.
528++ source_entry = {"source-key": "source-value"}
529++ output_entry = self.mirror.adapt_source_entry(
530++ source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
531++ image_md5_hash=None, image_size=None)
532++
533++ # Source and output entry are different objects.
534++ self.assertNotEqual(source_entry, output_entry)
535++
536++ # Output entry gets a few new properties like the endpoint and
537++ # owner_id taken from the GlanceMirror and OpenStack configuration,
538++ # region from the value passed into GlanceMirror constructor, and
539++ # image name from the passed in value.
540++ # It also contains the source entries as well.
541++ self.assertEqual(
542++ {"endpoint": "http://keystone/api/",
543++ "name": "foobuntu-X",
544++ "owner_id": "bar456",
545++ "region": "region1",
546++ "source-key": "source-value"},
547++ output_entry)
548++
549++ def test_adapt_source_entry_ignored_properties(self):
550++ # adapt_source_entry() drops some properties from the source entry.
551++ source_entry = {"path": "foo",
552++ "product_name": "bar",
553++ "version_name": "baz",
554++ "item_name": "bah"}
555++ output_entry = self.mirror.adapt_source_entry(
556++ source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
557++ image_md5_hash=None, image_size=None)
558++
559++ # None of the values in 'source_entry' are preserved.
560++ for key in ("path", "product_name", "version_name", "item"):
561++ self.assertNotIn("path", output_entry)
562++
563++ def test_adapt_source_entry_image_md5_and_size(self):
564++ # adapt_source_entry() will use passed in values for md5 and size.
565++ # Even old stale values will be overridden when image_md5_hash and
566++ # image_size are passed in.
567++ source_entry = {"md5": "stale-md5"}
568++ output_entry = self.mirror.adapt_source_entry(
569++ source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
570++ image_md5_hash="new-md5", image_size=5)
571++
572++ self.assertEqual("new-md5", output_entry["md5"])
573++ self.assertEqual(5, output_entry["size"])
574++
575++ def test_adapt_source_entry_image_md5_and_size_both_required(self):
576++ # adapt_source_entry() requires both md5 and size to not ignore them.
577++
578++ source_entry = {"md5": "stale-md5"}
579++
580++ # image_size is not passed in, so md5 value is not used either.
581++ output_entry1 = self.mirror.adapt_source_entry(
582++ source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
583++ image_md5_hash="new-md5", image_size=None)
584++ self.assertEqual("stale-md5", output_entry1["md5"])
585++ self.assertNotIn("size", output_entry1)
586++
587++ # image_md5_hash is not passed in, so image_size is not used either.
588++ output_entry2 = self.mirror.adapt_source_entry(
589++ source_entry, hypervisor_mapping=False, image_name="foobuntu-X",
590++ image_md5_hash=None, image_size=5)
591++ self.assertEqual("stale-md5", output_entry2["md5"])
592++ self.assertNotIn("size", output_entry2)
593++
594++ def test_adapt_source_entry_hypervisor_mapping(self):
595++ # If hypervisor_mapping is set to True, 'virt' value is derived from
596++ # the source entry 'ftype'.
597++ source_entry = {"ftype": "disk1.img"}
598++ output_entry = self.mirror.adapt_source_entry(
599++ source_entry, hypervisor_mapping=True, image_name="foobuntu-X",
600++ image_md5_hash=None, image_size=None)
601++
602++ self.assertEqual("kvm", output_entry["virt"])
603++
604++ def test_adapt_source_entry_hypervisor_mapping_ftype_required(self):
605++ # If hypervisor_mapping is set to True, but 'ftype' is missing in the
606++ # source entry, 'virt' value is not added to the returned entry.
607++ source_entry = {}
608++ output_entry = self.mirror.adapt_source_entry(
609++ source_entry, hypervisor_mapping=True, image_name="foobuntu-X",
610++ image_md5_hash=None, image_size=None)
611++
612++ self.assertNotIn("virt", output_entry)
613++
614++ def test_create_glance_properties(self):
615++ # Constructs glance properties to set on image during upload
616++ # based on source image metadata.
617++ source_entry = {
618++ # All of these are carried over and potentially re-named.
619++ "product_name": "foobuntu",
620++ "version_name": "X",
621++ "item_name": "disk1.img",
622++ "os": "ubuntu",
623++ "version": "16.04",
624++ # Other entries are ignored.
625++ "something-else": "ignored",
626++ }
627++ properties = self.mirror.create_glance_properties(
628++ "content-1", "source-1", source_entry, hypervisor_mapping=False)
629++
630++ # Output properties contain content-id and source-content-id based
631++ # on the passed in parameters, and carry over (with changed keys
632++ # for "os" and "version") product_name, version_name, item_name and
633++ # os and version values from the source entry.
634++ self.assertEqual(
635++ {"content_id": "content-1",
636++ "source_content_id": "source-1",
637++ "product_name": "foobuntu",
638++ "version_name": "X",
639++ "item_name": "disk1.img",
640++ "os_distro": "ubuntu",
641++ "os_version": "16.04"},
642++ properties)
643++
644++ def test_create_glance_properties_arch(self):
645++ # When 'arch' is present in the source entry, it is adapted and
646++ # returned inside 'architecture' field.
647++ source_entry = {
648++ "product_name": "foobuntu",
649++ "version_name": "X",
650++ "item_name": "disk1.img",
651++ "os": "ubuntu",
652++ "version": "16.04",
653++ "arch": "amd64",
654++ }
655++ properties = self.mirror.create_glance_properties(
656++ "content-1", "source-1", source_entry, hypervisor_mapping=False)
657++ self.assertEqual("x86_64", properties["architecture"])
658++
659++ def test_create_glance_properties_hypervisor_mapping(self):
660++ # When hypervisor_mapping is requested and 'ftype' is present in
661++ # the image metadata, 'hypervisor_type' is added to returned
662++ # properties.
663++ source_entry = {
664++ "product_name": "foobuntu",
665++ "version_name": "X",
666++ "item_name": "disk1.img",
667++ "os": "ubuntu",
668++ "version": "16.04",
669++ "ftype": "root.tar.gz",
670++ }
671++ properties = self.mirror.create_glance_properties(
672++ "content-1", "source-1", source_entry, hypervisor_mapping=True)
673++ self.assertEqual("lxc", properties["hypervisor_type"])
674++
675++ def test_prepare_glance_arguments(self):
676++ # Prepares arguments to pass to GlanceClient.images.create()
677++ # based on image metadata from the simplestreams source.
678++ source_entry = {}
679++ create_arguments = self.mirror.prepare_glance_arguments(
680++ "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
681++ image_properties=None)
682++
683++ # Arguments to always pass in contain the image name, container format,
684++ # disk format, whether image is public, and any passed-in properties.
685++ self.assertEqual(
686++ {"name": "foobuntu-X",
687++ "container_format": 'bare',
688++ "disk_format": "qcow2",
689++ "is_public": True,
690++ "properties": None},
691++ create_arguments)
692++
693++ def test_prepare_glance_arguments_disk_format(self):
694++ # Disk format is based on the image 'ftype' (if defined).
695++ source_entry = {"ftype": "root.tar.gz"}
696++ create_arguments = self.mirror.prepare_glance_arguments(
697++ "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
698++ image_properties=None)
699++
700++ self.assertEqual("root-tar", create_arguments["disk_format"])
701++
702++ def test_prepare_glance_arguments_size(self):
703++ # Size is read from image metadata if defined.
704++ source_entry = {"size": 5}
705++ create_arguments = self.mirror.prepare_glance_arguments(
706++ "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
707++ image_properties=None)
708++
709++ self.assertEqual(5, create_arguments["size"])
710++
711++ def test_prepare_glance_arguments_checksum(self):
712++ # Checksum is based on the source entry 'md5' value, if defined.
713++ source_entry = {"md5": "foo123"}
714++ create_arguments = self.mirror.prepare_glance_arguments(
715++ "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
716++ image_properties=None)
717++
718++ self.assertEqual("foo123", create_arguments["checksum"])
719++
720++ def test_prepare_glance_arguments_size_and_md5_override(self):
721++ # Size and md5 hash are overridden from the passed-in values even if
722++ # defined on the source entry.
723++ source_entry = {"size": 5, "md5": "foo123"}
724++ create_arguments = self.mirror.prepare_glance_arguments(
725++ "foobuntu-X", source_entry, image_md5_hash="bar456", image_size=10,
726++ image_properties=None)
727++
728++ self.assertEqual(10, create_arguments["size"])
729++ self.assertEqual("bar456", create_arguments["checksum"])
730++
731++ def test_prepare_glance_arguments_size_and_md5_no_override_hash(self):
732++ # If only one of image_md5_hash or image_size is passed directly in,
733++ # the other value is not overridden either.
734++ source_entry = {"size": 5, "md5": "foo123"}
735++ create_arguments = self.mirror.prepare_glance_arguments(
736++ "foobuntu-X", source_entry, image_md5_hash="bar456",
737++ image_size=None, image_properties=None)
738++
739++ self.assertEqual(5, create_arguments["size"])
740++ self.assertEqual("foo123", create_arguments["checksum"])
741++
742++ def test_prepare_glance_arguments_size_and_md5_no_override_size(self):
743++ # If only one of image_md5_hash or image_size is passed directly in,
744++ # the other value is not overridden either.
745++ source_entry = {"size": 5, "md5": "foo123"}
746++ create_arguments = self.mirror.prepare_glance_arguments(
747++ "foobuntu-X", source_entry, image_md5_hash=None, image_size=10,
748++ image_properties=None)
749++
750++ self.assertEqual(5, create_arguments["size"])
751++ self.assertEqual("foo123", create_arguments["checksum"])
752++
753++ def test_download_image(self):
754++ # Downloads image from a contentsource.
755++ content = "foo bazes the bar"
756++ content_source = MemoryContentSource(
757++ url="http://image-store/fooubuntu-X-disk1.img", content=content)
758++ image_metadata = {"pubname": "foobuntu-X", "size": 5}
759++ path, size, md5_hash = self.mirror.download_image(
760++ content_source, image_metadata)
761++ self.addCleanup(os.unlink, path)
762++ self.assertIsNotNone(path)
763++ self.assertIsNone(size)
764++ self.assertIsNone(md5_hash)
765++
766++ def test_download_image_progress_callback(self):
767++ # Progress callback is called with image name, size, status and buffer
768++ # size after every 10kb of data: 3 times for 25kb of data below.
769++ content = "abcdefghij" * int(1024 * 2.5)
770++ content_source = MemoryContentSource(
771++ url="http://image-store/fooubuntu-X-disk1.img", content=content)
772++ image_metadata = {"pubname": "foobuntu-X", "size": len(content)}
773++
774++ self.progress_calls = []
775++
776++ def log_progress_calls(message):
777++ self.progress_calls.append(message)
778++
779++ self.addCleanup(
780++ setattr, self.mirror, "progress_callback",
781++ self.mirror.progress_callback)
782++ self.mirror.progress_callback = log_progress_calls
783++ path, size, md5_hash = self.mirror.download_image(
784++ content_source, image_metadata)
785++ self.addCleanup(os.unlink, path)
786++
787++ self.assertEqual(
788++ [{"name": "foobuntu-X", "size": 25600, "status": "Downloading",
789++ "written": 10240}] * 3,
790++ self.progress_calls)
791++
792++ def test_download_image_error(self):
793++ # When there's an error during download, contentsource is still closed
794++ # and the error is propagated below.
795++ content = "abcdefghij"
796++ content_source = MemoryContentSource(
797++ url="http://image-store/fooubuntu-X-disk1.img", content=content)
798++ image_metadata = {"pubname": "foobuntu-X", "size": len(content)}
799++
800++ # MemoryContentSource has an internal file descriptor which indicates
801++ # if close() method has been called on it.
802++ self.assertFalse(content_source.fd.closed)
803++
804++ self.addCleanup(
805++ setattr, self.mirror, "progress_callback",
806++ self.mirror.progress_callback)
807++ self.mirror.progress_callback = lambda message: 1/0
808++
809++ self.assertRaises(
810++ ZeroDivisionError,
811++ self.mirror.download_image, content_source, image_metadata)
812++
813++ # We rely on the MemoryContentSource.close() side-effect to ensure
814++ # close() method has indeed been called on the passed-in ContentSource.
815++ self.assertTrue(content_source.fd.closed)
816++
817++ def test_insert_item(self):
818++ # Downloads an image from a contentsource, uploads it into Glance,
819++ # adapting and munging as needed (it updates the keystone endpoint,
820++ # image and owner ids).
821++ # This test is basically an integration test to make sure all the
822++ # methods used by insert_item() are tied together in one good
823++ # fully functioning whole.
824++
825++ # This is a real snippet from the simplestreams index entry for
826++ # Ubuntu 14.04 amd64 image from cloud-images.ubuntu.com as of
827++ # 2016-06-05.
828++ source_index = {
829++ u'content_id': u'com.ubuntu.cloud:released:download',
830++ u'datatype': u'image-downloads',
831++ u'format': u'products:1.0',
832++ u'license': (u'http://www.canonical.com/'
833++ u'intellectual-property-policy'),
834++ u'products': {u'com.ubuntu.cloud:server:14.04:amd64': {
835++ u'aliases': u'14.04,default,lts,t,trusty',
836++ u'arch': u'amd64',
837++ u'os': u'ubuntu',
838++ u'release': u'trusty',
839++ u'release_codename': u'Trusty Tahr',
840++ u'release_title': u'14.04 LTS',
841++ u'support_eol': u'2019-04-17',
842++ u'supported': True,
843++ u'version': u'14.04',
844++ u'versions': {u'20160602': {
845++ u'items': {u'disk1.img': {
846++ u'ftype': u'disk1.img',
847++ u'md5': u'e5436cd36ae6cc298f081bf0f6b413f1',
848++ u'path': (
849++ u'server/releases/trusty/release-20160602/'
850++ u'ubuntu-14.04-server-cloudimg-amd64-disk1.img'),
851++ u'sha256': (u'5b982d7d4dd1a03e88ae5f35f02ed44f'
852++ u'579e2711f3e0f27ea2bff20aef8c8d9e'),
853++ u'size': 259850752}},
854++ u'label': u'release',
855++ u'pubname': u'ubuntu-trusty-14.04-amd64-server-20160602',
856++ }}}
857++ }
858++ }
859++
860++ # "Pedigree" is basically a "path" to get to the image data in
861++ # simplestreams index, going through "products", their "versions",
862++ # and nested "items".
863++ pedigree = (
864++ u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
865++ product = source_index[u'products'][pedigree[0]]
866++ image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
867++
868++ content_source = MemoryContentSource(
869++ url="http://image-store/fooubuntu-X-disk1.img",
870++ content="image-data")
871++
872++ # Use a fake GlanceClient to track arguments passed into
873++ # GlanceClient.images.create().
874++ self.addCleanup(setattr, self.mirror, "gclient", self.mirror.gclient)
875++ self.mirror.gclient = FakeGlanceClient()
876++
877++ target = {
878++ 'content_id': 'auto.sync',
879++ 'datatype': 'image-ids',
880++ 'format': 'products:1.0',
881++ }
882++
883++ self.mirror.insert_item(
884++ image_data, source_index, target, pedigree, content_source)
885++
886++ passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
887++
888++ # There is a 'data' argument pointing to an open file descriptor
889++ # for the locally downloaded image.
890++ self.assertIn("data", passed_create_kwargs)
891++ passed_create_kwargs.pop("data")
892++
893++ expected_create_kwargs = {
894++ 'name': ('auto-sync/'
895++ 'ubuntu-trusty-14.04-amd64-server-20160602-disk1.img'),
896++ 'checksum': u'e5436cd36ae6cc298f081bf0f6b413f1',
897++ 'disk_format': 'qcow2',
898++ 'container_format': 'bare',
899++ 'is_public': True,
900++ 'properties': {
901++ 'os_distro': u'ubuntu',
902++ 'item_name': u'disk1.img',
903++ 'os_version': u'14.04',
904++ 'architecture': 'x86_64',
905++ 'version_name': u'20160602',
906++ 'content_id': 'auto.sync',
907++ 'product_name': u'com.ubuntu.cloud:server:14.04:amd64',
908++ 'source_content_id': u'com.ubuntu.cloud:released:download'},
909++ 'size': '259850752'}
910++ self.assertEqual(
911++ expected_create_kwargs, passed_create_kwargs)
912++
913++ # Almost real resulting data as produced by simplestreams before
914++ # insert_item refactoring to allow for finer-grained testing.
915++ expected_target_index = {
916++ 'content_id': 'auto.sync',
917++ 'datatype': 'image-ids',
918++ 'format': 'products:1.0',
919++ 'products': {
920++ "com.ubuntu.cloud:server:14.04:amd64": {
921++ "aliases": "14.04,default,lts,t,trusty",
922++ "arch": "amd64",
923++ "label": "release",
924++ "os": "ubuntu",
925++ "owner_id": "bar456",
926++ "pubname": "ubuntu-trusty-14.04-amd64-server-20160602",
927++ "release": "trusty",
928++ "release_codename": "Trusty Tahr",
929++ "release_title": "14.04 LTS",
930++ "support_eol": "2019-04-17",
931++ "supported": "True",
932++ "version": "14.04",
933++ "versions": {"20160602": {"items": {"disk1.img": {
934++ "endpoint": "http://keystone/api/",
935++ "ftype": "disk1.img",
936++ "id": "image-1",
937++ "md5": "e5436cd36ae6cc298f081bf0f6b413f1",
938++ "name": ("auto-sync/ubuntu-trusty-14.04-amd64-"
939++ "server-20160602-disk1.img"),
940++ "region": "region1",
941++ "sha256": ("5b982d7d4dd1a03e88ae5f35f02ed44f"
942++ "579e2711f3e0f27ea2bff20aef8c8d9e"),
943++ "size": "259850752"
944++ }}}}
945++ }
946++ }
947++ }
948++
949++ # Apply the condensing as done in GlanceMirror.insert_products()
950++ # to ensure we compare with the desired resulting simplestreams data.
951++ sticky = ['ftype', 'md5', 'sha256', 'size', 'name', 'id', 'endpoint',
952++ 'region']
953++ simplestreams.util.products_condense(target, sticky)
954++
955++ self.assertEqual(expected_target_index, target)
956+
957diff --git a/debian/patches/436-glance-fix-race-conditions.patch b/debian/patches/436-glance-fix-race-conditions.patch
958new file mode 100644
959index 0000000..ae55dbe
960--- /dev/null
961+++ b/debian/patches/436-glance-fix-race-conditions.patch
962@@ -0,0 +1,479 @@
963+------------------------------------------------------------
964+revno: 436 [merge]
965+fixes bug: https://launchpad.net/bugs/1584938
966+committer: Scott Moser <smoser@ubuntu.com>
967+branch nick: trunk
968+timestamp: Tue 2016-06-14 16:21:05 -0400
969+message:
970+ GlanceMirror: fix a couple of race-related problems
971+
972+ First, what can happen is that .sync() is interrupted for external reasons
973+ (service restarts, network issues,...) after first image has been uploaded
974+ to Glance when syncing multiple images. On re-run, the uploaded image is
975+ considered "synced" already, but since simplestreams index is written out
976+ only at the end of the sync, all the metadata is lost. We solve this by
977+ adding "simplestreams_metadata" as one of the properties of image in
978+ Glance where we put a JSON representation of all the metadata fields from
979+ the original simplestreams index entry. We then load them in
980+ load_products() if they are defined. This results in a slightly different
981+ auto.sync.json entry: "endpoint" and "region" are on the product entry,
982+ instead of on the disk1.img entry.
983+
984+ Secondly, if sync is indeed interrupted after one or several images are
985+ completely uploaded (but not all of them), there's no index file at all,
986+ so any attempt to make use of them with juju would fail, even though it
987+ should be fully functional. We solve this by calling "insert_products()"
988+ at the end of the insert_item(). This means that the index is regenerated
989+ after every single image is uploaded.
990+
991+ Along the way, I slightly improve insert_item() tests from the pre-req
992+ branch to add a test for minimal data required, rename the existing one
993+ using real world data to test_insert_item_full, and re-use that data for a
994+ test for a newly added call to insert_products().
995+------------------------------------------------------------
996+Use --include-merged or -n0 to see merged revisions.
997+=== modified file 'simplestreams/mirrors/glance.py'
998+--- a/simplestreams/mirrors/glance.py 2016-06-10 18:47:52 +0000
999++++ b/simplestreams/mirrors/glance.py 2016-06-14 20:21:05 +0000
1000+@@ -25,6 +25,7 @@
1001+ import copy
1002+ import errno
1003+ import glanceclient
1004++import json
1005+ import os
1006+ import re
1007+
1008+@@ -219,6 +220,15 @@
1009+ except KeyError:
1010+ item_data = {}
1011+
1012++ # If original simplestreams-metadata is stored on the image,
1013++ # use that as well.
1014++ if 'simplestreams_metadata' in props:
1015++ simplestreams_metadata = json.loads(
1016++ props.get('simplestreams_metadata'))
1017++ else:
1018++ simplestreams_metadata = {}
1019++ item_data.update(simplestreams_metadata)
1020++
1021+ item_data.update({'name': image['name'], 'id': image['id']})
1022+ if 'owner_id' not in item_data:
1023+ item_data['owner_id'] = self.tenant_id
1024+@@ -248,10 +258,10 @@
1025+ }
1026+ # An iterator of properties to carry over: if a property needs
1027+ # renaming, uses a tuple (old name, new name).
1028+- carry_over = (
1029+- 'product_name', 'version_name', 'item_name',
1030+- ('os', 'os_distro'), ('version', 'os_version'),
1031+- )
1032++ carry_over_simple = (
1033++ 'product_name', 'version_name', 'item_name')
1034++ carry_over = carry_over_simple + (
1035++ ('os', 'os_distro'), ('version', 'os_version'))
1036+ for carry_over_property in carry_over:
1037+ if isinstance(carry_over_property, tuple):
1038+ name_old, name_new = carry_over_property
1039+@@ -267,6 +277,16 @@
1040+ _hypervisor_type = hypervisor_type(image_metadata['ftype'])
1041+ if _hypervisor_type:
1042+ properties['hypervisor_type'] = _hypervisor_type
1043++
1044++ # Store flattened metadata for a source image along with the
1045++ # image in 'simplestreams_metadata' property.
1046++ simplestreams_metadata = image_metadata.copy()
1047++ drop_keys = carry_over_simple + ('path',)
1048++ for remove_key in drop_keys:
1049++ if remove_key in simplestreams_metadata:
1050++ del simplestreams_metadata[remove_key]
1051++ properties['simplestreams_metadata'] = json.dumps(
1052++ simplestreams_metadata, sort_keys=True)
1053+ return properties
1054+
1055+ def prepare_glance_arguments(self, full_image_name, image_metadata,
1056+@@ -441,6 +461,9 @@
1057+ os.unlink(tmp_path)
1058+
1059+ util.products_set(target, target_sstream_item, pedigree)
1060++ # We can safely ignore path and content arguments since they are
1061++ # unused in insert_products below.
1062++ self.insert_products(None, target, None)
1063+
1064+ def remove_item(self, data, src, target, pedigree):
1065+ util.products_del(target, pedigree)
1066+
1067+=== modified file 'tests/unittests/test_glancemirror.py'
1068+--- a/tests/unittests/test_glancemirror.py 2016-06-10 18:47:52 +0000
1069++++ b/tests/unittests/test_glancemirror.py 2016-06-14 20:21:05 +0000
1070+@@ -1,11 +1,91 @@
1071+ from simplestreams.contentsource import MemoryContentSource
1072+ from simplestreams.mirrors.glance import GlanceMirror
1073++from simplestreams.objectstores import MemoryObjectStore
1074+ import simplestreams.util
1075+
1076++import copy
1077++import json
1078+ import os
1079+ from unittest import TestCase
1080+
1081+
1082++# This is a real snippet from the simplestreams index entry for
1083++# Ubuntu 14.04 amd64 image from cloud-images.ubuntu.com as of
1084++# 2016-06-05.
1085++TEST_SOURCE_INDEX_ENTRY = {
1086++ u'content_id': u'com.ubuntu.cloud:released:download',
1087++ u'datatype': u'image-downloads',
1088++ u'format': u'products:1.0',
1089++ u'license': (u'http://www.canonical.com/'
1090++ u'intellectual-property-policy'),
1091++ u'products': {u'com.ubuntu.cloud:server:14.04:amd64': {
1092++ u'aliases': u'14.04,default,lts,t,trusty',
1093++ u'arch': u'amd64',
1094++ u'os': u'ubuntu',
1095++ u'release': u'trusty',
1096++ u'release_codename': u'Trusty Tahr',
1097++ u'release_title': u'14.04 LTS',
1098++ u'support_eol': u'2019-04-17',
1099++ u'supported': True,
1100++ u'version': u'14.04',
1101++ u'versions': {u'20160602': {
1102++ u'items': {u'disk1.img': {
1103++ u'ftype': u'disk1.img',
1104++ u'md5': u'e5436cd36ae6cc298f081bf0f6b413f1',
1105++ u'path': (
1106++ u'server/releases/trusty/release-20160602/'
1107++ u'ubuntu-14.04-server-cloudimg-amd64-disk1.img'),
1108++ u'sha256': (u'5b982d7d4dd1a03e88ae5f35f02ed44f'
1109++ u'579e2711f3e0f27ea2bff20aef8c8d9e'),
1110++ u'size': 259850752}},
1111++ u'label': u'release',
1112++ u'pubname': u'ubuntu-trusty-14.04-amd64-server-20160602',
1113++ }}}
1114++ }
1115++}
1116++
1117++# "Pedigree" is basically a "path" to get to the image data in simplestreams
1118++# index, going through "products", their "versions", and nested "items".
1119++TEST_IMAGE_PEDIGREE = (
1120++ u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
1121++
1122++# Almost real resulting data as produced by simplestreams before
1123++# insert_item refactoring to allow for finer-grained testing.
1124++EXPECTED_OUTPUT_INDEX = {
1125++ u'content_id': u'auto.sync',
1126++ u'datatype': u'image-ids',
1127++ u'format': u'products:1.0',
1128++ u'products': {
1129++ u"com.ubuntu.cloud:server:14.04:amd64": {
1130++ u"aliases": u"14.04,default,lts,t,trusty",
1131++ u"arch": u"amd64",
1132++ u"label": u"release",
1133++ u"os": u"ubuntu",
1134++ u"owner_id": u"bar456",
1135++ u"pubname": u"ubuntu-trusty-14.04-amd64-server-20160602",
1136++ u"release": u"trusty",
1137++ u"release_codename": u"Trusty Tahr",
1138++ u"release_title": u"14.04 LTS",
1139++ u"support_eol": u"2019-04-17",
1140++ u"supported": u"True",
1141++ u"version": u"14.04",
1142++ u"versions": {u"20160602": {u"items": {u"disk1.img": {
1143++ u"endpoint": u"http://keystone/api/",
1144++ u"ftype": u"disk1.img",
1145++ u"id": u"image-1",
1146++ u"md5": u"e5436cd36ae6cc298f081bf0f6b413f1",
1147++ u"name": (u"auto-sync/ubuntu-trusty-14.04-amd64-"
1148++ u"server-20160602-disk1.img"),
1149++ u"region": u"region1",
1150++ u"sha256": (u"5b982d7d4dd1a03e88ae5f35f02ed44f"
1151++ u"579e2711f3e0f27ea2bff20aef8c8d9e"),
1152++ u"size": u"259850752"
1153++ }}}}
1154++ }
1155++ }
1156++}
1157++
1158++
1159+ class FakeOpenstack(object):
1160+ """Fake 'openstack' module replacement for testing GlanceMirror."""
1161+ def load_keystone_creds(self):
1162+@@ -145,8 +225,8 @@
1163+ "item_name": "disk1.img",
1164+ "os": "ubuntu",
1165+ "version": "16.04",
1166+- # Other entries are ignored.
1167+- "something-else": "ignored",
1168++ # Unknown entries are stored in 'simplestreams_metadata'.
1169++ "extra": "value",
1170+ }
1171+ properties = self.mirror.create_glance_properties(
1172+ "content-1", "source-1", source_entry, hypervisor_mapping=False)
1173+@@ -155,6 +235,8 @@
1174+ # on the passed in parameters, and carry over (with changed keys
1175+ # for "os" and "version") product_name, version_name, item_name and
1176+ # os and version values from the source entry.
1177++ # All the fields except product_name, version_name and item_name are
1178++ # also stored inside 'simplestreams_metadata' property as JSON data.
1179+ self.assertEqual(
1180+ {"content_id": "content-1",
1181+ "source_content_id": "source-1",
1182+@@ -162,7 +244,9 @@
1183+ "version_name": "X",
1184+ "item_name": "disk1.img",
1185+ "os_distro": "ubuntu",
1186+- "os_version": "16.04"},
1187++ "os_version": "16.04",
1188++ "simplestreams_metadata": (
1189++ '{"extra": "value", "os": "ubuntu", "version": "16.04"}')},
1190+ properties)
1191+
1192+ def test_create_glance_properties_arch(self):
1193+@@ -196,6 +280,26 @@
1194+ "content-1", "source-1", source_entry, hypervisor_mapping=True)
1195+ self.assertEqual("lxc", properties["hypervisor_type"])
1196+
1197++ def test_create_glance_properties_simplestreams_no_path(self):
1198++ # Other than 'product_name', 'version_name' and 'item_name', if 'path'
1199++ # is defined on the source entry, it is also not saved inside the
1200++ # 'simplestreams_metadata' property.
1201++ source_entry = {
1202++ "product_name": "foobuntu",
1203++ "version_name": "X",
1204++ "item_name": "disk1.img",
1205++ "os": "ubuntu",
1206++ "version": "16.04",
1207++ "path": "/path/to/foo",
1208++ }
1209++ properties = self.mirror.create_glance_properties(
1210++ "content-1", "source-1", source_entry, hypervisor_mapping=False)
1211++
1212++ # Path is omitted from the simplestreams_metadata property JSON.
1213++ self.assertEqual(
1214++ '{"os": "ubuntu", "version": "16.04"}',
1215++ properties["simplestreams_metadata"])
1216++
1217+ def test_prepare_glance_arguments(self):
1218+ # Prepares arguments to pass to GlanceClient.images.create()
1219+ # based on image metadata from the simplestreams source.
1220+@@ -342,45 +446,86 @@
1221+ # Downloads an image from a contentsource, uploads it into Glance,
1222+ # adapting and munging as needed (it updates the keystone endpoint,
1223+ # image and owner ids).
1224+- # This test is basically an integration test to make sure all the
1225+- # methods used by insert_item() are tied together in one good
1226+- # fully functioning whole.
1227+
1228+- # This is a real snippet from the simplestreams index entry for
1229+- # Ubuntu 14.04 amd64 image from cloud-images.ubuntu.com as of
1230+- # 2016-06-05.
1231++ # We use a minimal source simplestreams index, fake ContentSource and
1232++ # GlanceClient, and only test for side-effects of each of the
1233++ # subparts of the insert_item method.
1234+ source_index = {
1235+ u'content_id': u'com.ubuntu.cloud:released:download',
1236+- u'datatype': u'image-downloads',
1237+- u'format': u'products:1.0',
1238+- u'license': (u'http://www.canonical.com/'
1239+- u'intellectual-property-policy'),
1240+ u'products': {u'com.ubuntu.cloud:server:14.04:amd64': {
1241+- u'aliases': u'14.04,default,lts,t,trusty',
1242+ u'arch': u'amd64',
1243+ u'os': u'ubuntu',
1244+ u'release': u'trusty',
1245+- u'release_codename': u'Trusty Tahr',
1246+- u'release_title': u'14.04 LTS',
1247+- u'support_eol': u'2019-04-17',
1248+- u'supported': True,
1249+ u'version': u'14.04',
1250+ u'versions': {u'20160602': {
1251+ u'items': {u'disk1.img': {
1252+ u'ftype': u'disk1.img',
1253+ u'md5': u'e5436cd36ae6cc298f081bf0f6b413f1',
1254+- u'path': (
1255+- u'server/releases/trusty/release-20160602/'
1256+- u'ubuntu-14.04-server-cloudimg-amd64-disk1.img'),
1257+- u'sha256': (u'5b982d7d4dd1a03e88ae5f35f02ed44f'
1258+- u'579e2711f3e0f27ea2bff20aef8c8d9e'),
1259+ u'size': 259850752}},
1260+- u'label': u'release',
1261+ u'pubname': u'ubuntu-trusty-14.04-amd64-server-20160602',
1262+ }}}
1263+ }
1264+ }
1265+
1266++ pedigree = (
1267++ u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
1268++ product = source_index[u'products'][pedigree[0]]
1269++ image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
1270++
1271++ content_source = MemoryContentSource(
1272++ url="http://image-store/fooubuntu-X-disk1.img",
1273++ content="image-data")
1274++
1275++ # Use a fake GlanceClient to track calls and arguments passed to
1276++ # GlanceClient.images.create().
1277++ self.addCleanup(setattr, self.mirror, "gclient", self.mirror.gclient)
1278++ self.mirror.gclient = FakeGlanceClient()
1279++
1280++ target = {
1281++ 'content_id': 'auto.sync',
1282++ 'datatype': 'image-ids',
1283++ 'format': 'products:1.0',
1284++ }
1285++
1286++ self.mirror.insert_item(
1287++ image_data, source_index, target, pedigree, content_source)
1288++
1289++ passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
1290++
1291++ # There is a 'data' argument pointing to an open file descriptor
1292++ # for the locally downloaded image.
1293++ image_content = passed_create_kwargs.pop("data").read()
1294++ self.assertEqual(u"image-data", image_content.decode('utf-8'))
1295++
1296++ # Value of "arch" from source entry is transformed into "architecture"
1297++ # image property in Glance: this ensures create_glance_properties()
1298++ # is called and result is properly passed.
1299++ self.assertEqual(
1300++ "x86_64", passed_create_kwargs["properties"]["architecture"])
1301++
1302++ # MD5 hash from source entry is put into 'checksum' field, and 'name'
1303++ # is based on full image name: this ensures prepare_glance_arguments()
1304++ # is called.
1305++ self.assertEqual(
1306++ u'e5436cd36ae6cc298f081bf0f6b413f1',
1307++ passed_create_kwargs["checksum"])
1308++ self.assertEqual(
1309++ u'auto-sync/ubuntu-trusty-14.04-amd64-server-20160602-disk1.img',
1310++ passed_create_kwargs["name"])
1311++
1312++ # Our local endpoint is set in the resulting entry, which ensures
1313++ # a call to adapt_source_entry() was indeed made.
1314++ target_product = target["products"][pedigree[0]]
1315++ target_image = target_product["versions"][pedigree[1]]["items"].get(
1316++ pedigree[2])
1317++ self.assertEqual(u"http://keystone/api/", target_image["endpoint"])
1318++
1319++ def test_insert_item_full(self):
1320++ # This test uses the full sample entries from the source simplestreams
1321++ # index from cloud-images.u.c and resulting local simplestreams index
1322++ # files.
1323++ source_index = copy.deepcopy(TEST_SOURCE_INDEX_ENTRY)
1324++
1325+ # "Pedigree" is basically a "path" to get to the image data in
1326+ # simplestreams index, going through "products", their "versions",
1327+ # and nested "items".
1328+@@ -409,9 +554,7 @@
1329+
1330+ passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
1331+
1332+- # There is a 'data' argument pointing to an open file descriptor
1333+- # for the locally downloaded image.
1334+- self.assertIn("data", passed_create_kwargs)
1335++ # Drop the 'data' item pointing to an open temporary file.
1336+ passed_create_kwargs.pop("data")
1337+
1338+ expected_create_kwargs = {
1339+@@ -429,46 +572,21 @@
1340+ 'version_name': u'20160602',
1341+ 'content_id': 'auto.sync',
1342+ 'product_name': u'com.ubuntu.cloud:server:14.04:amd64',
1343++ 'simplestreams_metadata': (
1344++ '{"aliases": "14.04,default,lts,t,trusty", '
1345++ '"arch": "amd64", "ftype": "disk1.img", '
1346++ '"label": "release", "md5": '
1347++ '"e5436cd36ae6cc298f081bf0f6b413f1", "os": "ubuntu", '
1348++ '"pubname": "ubuntu-trusty-14.04-amd64-server-20160602", '
1349++ '"release": "trusty", "release_codename": "Trusty Tahr", '
1350++ '"release_title": "14.04 LTS", "sha256": '
1351++ '"5b982d7d4dd1a03e88ae5f35f02ed44f'
1352++ '579e2711f3e0f27ea2bff20aef8c8d9e", "size": "259850752", '
1353++ '"support_eol": "2019-04-17", "supported": "True", '
1354++ '"version": "14.04"}'),
1355+ 'source_content_id': u'com.ubuntu.cloud:released:download'},
1356+ 'size': '259850752'}
1357+- self.assertEqual(
1358+- expected_create_kwargs, passed_create_kwargs)
1359+-
1360+- # Almost real resulting data as produced by simplestreams before
1361+- # insert_item refactoring to allow for finer-grained testing.
1362+- expected_target_index = {
1363+- 'content_id': 'auto.sync',
1364+- 'datatype': 'image-ids',
1365+- 'format': 'products:1.0',
1366+- 'products': {
1367+- "com.ubuntu.cloud:server:14.04:amd64": {
1368+- "aliases": "14.04,default,lts,t,trusty",
1369+- "arch": "amd64",
1370+- "label": "release",
1371+- "os": "ubuntu",
1372+- "owner_id": "bar456",
1373+- "pubname": "ubuntu-trusty-14.04-amd64-server-20160602",
1374+- "release": "trusty",
1375+- "release_codename": "Trusty Tahr",
1376+- "release_title": "14.04 LTS",
1377+- "support_eol": "2019-04-17",
1378+- "supported": "True",
1379+- "version": "14.04",
1380+- "versions": {"20160602": {"items": {"disk1.img": {
1381+- "endpoint": "http://keystone/api/",
1382+- "ftype": "disk1.img",
1383+- "id": "image-1",
1384+- "md5": "e5436cd36ae6cc298f081bf0f6b413f1",
1385+- "name": ("auto-sync/ubuntu-trusty-14.04-amd64-"
1386+- "server-20160602-disk1.img"),
1387+- "region": "region1",
1388+- "sha256": ("5b982d7d4dd1a03e88ae5f35f02ed44f"
1389+- "579e2711f3e0f27ea2bff20aef8c8d9e"),
1390+- "size": "259850752"
1391+- }}}}
1392+- }
1393+- }
1394+- }
1395++ self.assertEqual(expected_create_kwargs, passed_create_kwargs)
1396+
1397+ # Apply the condensing as done in GlanceMirror.insert_products()
1398+ # to ensure we compare with the desired resulting simplestreams data.
1399+@@ -476,4 +594,40 @@
1400+ 'region']
1401+ simplestreams.util.products_condense(target, sticky)
1402+
1403+- self.assertEqual(expected_target_index, target)
1404++ self.assertEqual(EXPECTED_OUTPUT_INDEX, target)
1405++
1406++ def test_insert_item_stores_the_index(self):
1407++ # Ensure insert_item calls insert_products() to generate the
1408++ # resulting simplestreams index file and insert it into store.
1409++
1410++ source_index = copy.deepcopy(TEST_SOURCE_INDEX_ENTRY)
1411++ pedigree = TEST_IMAGE_PEDIGREE
1412++ product = source_index[u'products'][pedigree[0]]
1413++ image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
1414++
1415++ content_source = MemoryContentSource(
1416++ url="http://image-store/fooubuntu-X-disk1.img",
1417++ content="image-data")
1418++ self.mirror.store = MemoryObjectStore()
1419++
1420++ self.addCleanup(setattr, self.mirror, "gclient", self.mirror.gclient)
1421++ self.mirror.gclient = FakeGlanceClient()
1422++
1423++ target = {
1424++ 'content_id': 'auto.sync',
1425++ 'datatype': 'image-ids',
1426++ 'format': 'products:1.0',
1427++ }
1428++
1429++ self.mirror.insert_item(
1430++ image_data, source_index, target, pedigree, content_source)
1431++
1432++ stored_index_content = self.mirror.store.data[
1433++ 'streams/v1/auto.sync.json']
1434++ stored_index = json.loads(stored_index_content.decode('utf-8'))
1435++
1436++ # Full index contains the 'updated' key with the date of last update.
1437++ self.assertIn(u"updated", stored_index)
1438++ del stored_index[u"updated"]
1439++
1440++ self.assertEqual(EXPECTED_OUTPUT_INDEX, stored_index)
1441+
1442diff --git a/debian/patches/keystone-v3-support.patch b/debian/patches/450-453-454-keystone-v3-support.patch
1443index 3b24355..e3a3adf 100644
1444--- a/debian/patches/keystone-v3-support.patch
1445+++ b/debian/patches/450-453-454-keystone-v3-support.patch
1446@@ -15,11 +15,9 @@ Bug: https://launchpad.net/bugs/1728982
1447 Bug: https://launchpad.net/bugs/1719879
1448 Author: Scott Moser <smoser@ubuntu.com>
1449
1450-diff --git a/simplestreams/mirrors/glance.py b/simplestreams/mirrors/glance.py
1451-index 807fe29..a13fede 100644
1452 --- a/simplestreams/mirrors/glance.py
1453 +++ b/simplestreams/mirrors/glance.py
1454-@@ -35,7 +35,11 @@ def get_glanceclient(version='1', **kwargs):
1455+@@ -36,7 +36,11 @@ def get_glanceclient(version='1', **kwar
1456 kwargs['endpoint'] = _strip_version(kwargs['endpoint'])
1457 pt = ('endpoint', 'token', 'insecure', 'cacert')
1458 kskw = {k: kwargs.get(k) for k in pt if k in kwargs}
1459@@ -32,7 +30,7 @@ index 807fe29..a13fede 100644
1460
1461
1462 def empty_iid_products(content_id):
1463-@@ -64,6 +68,7 @@ def canonicalize_arch(arch):
1464+@@ -65,6 +69,7 @@ def canonicalize_arch(arch):
1465 LXC_FTYPES = [
1466 'root.tar.gz',
1467 'root.tar.xz',
1468@@ -40,8 +38,15 @@ index 807fe29..a13fede 100644
1469 ]
1470
1471 QEMU_FTYPES = [
1472-diff --git a/simplestreams/objectstores/swift.py b/simplestreams/objectstores/swift.py
1473-index d7598a7..9a6eb62 100644
1474+@@ -267,7 +272,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
1475+ name_old, name_new = carry_over_property
1476+ else:
1477+ name_old = name_new = carry_over_property
1478+- properties[name_new] = image_metadata[name_old]
1479++ properties[name_new] = image_metadata.get(name_old)
1480+
1481+ if 'arch' in image_metadata:
1482+ properties['architecture'] = canonicalize_arch(
1483 --- a/simplestreams/objectstores/swift.py
1484 +++ b/simplestreams/objectstores/swift.py
1485 @@ -33,6 +33,15 @@ def get_swiftclient(**kwargs):
1486@@ -60,8 +65,6 @@ index d7598a7..9a6eb62 100644
1487 return Connection(**connargs)
1488
1489
1490-diff --git a/simplestreams/openstack.py b/simplestreams/openstack.py
1491-index 126dea5..d143926 100644
1492 --- a/simplestreams/openstack.py
1493 +++ b/simplestreams/openstack.py
1494 @@ -15,16 +15,47 @@
1495@@ -134,7 +137,7 @@ index 126dea5..d143926 100644
1496 if missing:
1497 raise ValueError("Need values for: %s" % missing)
1498
1499-@@ -88,11 +128,49 @@ def get_regions(client=None, services=None, kscreds=None):
1500+@@ -88,11 +128,49 @@ def get_regions(client=None, services=No
1501 return list(regions)
1502
1503
1504@@ -188,7 +191,7 @@ index 126dea5..d143926 100644
1505
1506
1507 def get_service_conn_info(service='image', client=None, **kwargs):
1508-@@ -101,21 +179,27 @@ def get_service_conn_info(service='image', client=None, **kwargs):
1509+@@ -101,21 +179,27 @@ def get_service_conn_info(service='image
1510 client = get_ksclient(**kwargs)
1511
1512 endpoint = _get_endpoint(client, service, **kwargs)
1513diff --git a/debian/patches/455-nova-lxd-support-squashfs-images.patch b/debian/patches/455-nova-lxd-support-squashfs-images.patch
1514new file mode 100644
1515index 0000000..b4f0270
1516--- /dev/null
1517+++ b/debian/patches/455-nova-lxd-support-squashfs-images.patch
1518@@ -0,0 +1,230 @@
1519+------------------------------------------------------------
1520+revno: 455 [merge]
1521+fixes bug: https://launchpad.net/bugs/1686086
1522+committer: Scott Moser <smoser@ubuntu.com>
1523+branch nick: trunk
1524+timestamp: Thu 2017-11-02 15:03:37 -0400
1525+message:
1526+ OpenStack: support uploading squash images for nova-lxd.
1527+
1528+ Previously, populating a nova-lxd cloud was possible by using
1529+ root.tar.gz. A filter like:
1530+ ftype~(root.tar.gz|root.tar.xz)
1531+ would cause simplestreams to upload an image with 'disk-format' of
1532+ root-tar.
1533+
1534+ However, Ubuntu 17.04 and newer do not have root.tar.gz or root.tar.xz
1535+ images available. Currently here is what is available:
1536+ 14.04: root.tar.gz root.tar.xz
1537+ 16.04: root.tar.gz root.tar.xz squashfs
1538+ 17.10: squashfs
1539+
1540+ If we simply expected the user to change their filter to include
1541+ root.tar.xz|squashfs
1542+ Then they would get two lxd images imported for 16.04 each version.
1543+
1544+ The change here is to not do anything for an item insert, but instead
1545+ insert when the version's insert is called. Then, all the information
1546+ about what images there are is available, and it can "pick"
1547+ one or the other. Currently preference is given to the .tar.xz format.
1548+
1549+ The end result is that now users can specify an ftype filter of:
1550+ ftype~(root.tar.gz|root.tar.xz|squashfs)
1551+ and the right thing will be done.
1552+
1553+ Also here is simple knowledge that the squashfs type should be
1554+ uploaded to glance with a 'disk_format' of 'squashfs'.
1555+------------------------------------------------------------
1556+Use --include-merged or -n0 to see merged revisions.
1557+=== modified file 'simplestreams/mirrors/glance.py'
1558+--- a/simplestreams/mirrors/glance.py
1559++++ b/simplestreams/mirrors/glance.py
1560+@@ -66,25 +66,27 @@ def canonicalize_arch(arch):
1561+ return newarch
1562+
1563+
1564+-LXC_FTYPES = [
1565+- 'root.tar.gz',
1566+- 'root.tar.xz',
1567+- 'squashfs',
1568+-]
1569+-
1570+-QEMU_FTYPES = [
1571+- 'disk.img',
1572+- 'disk1.img',
1573+-]
1574++LXC_FTYPES = {
1575++ 'root.tar.gz': 'root-tar',
1576++ 'root.tar.xz': 'root-tar',
1577++ 'squashfs': 'squashfs',
1578++}
1579++
1580++QEMU_FTYPES = {
1581++ 'disk.img': 'qcow2',
1582++ 'disk1.img': 'qcow2',
1583++}
1584+
1585+
1586+ def disk_format(ftype):
1587+- '''Canonicalize disk formats for use in OpenStack'''
1588++ '''Canonicalize disk formats for use in OpenStack.
1589++ Input ftype is a 'ftype' from a simplestream feed.
1590++ Return value is the appropriate 'disk_format' for glance.'''
1591+ newftype = ftype.lower()
1592+ if newftype in LXC_FTYPES:
1593+- return 'root-tar'
1594++ return LXC_FTYPES[newftype]
1595+ if newftype in QEMU_FTYPES:
1596+- return 'qcow2'
1597++ return QEMU_FTYPES[newftype]
1598+ return None
1599+
1600+
1601+@@ -160,6 +162,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
1602+ self.content_id = config.get("content_id")
1603+ self.modify_hook = config.get("modify_hook")
1604+
1605++ self.inserts = {}
1606+ if not self.content_id:
1607+ raise TypeError("content_id is required")
1608+
1609+@@ -408,7 +411,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
1610+
1611+ return output_entry
1612+
1613+- def insert_item(self, data, src, target, pedigree, contentsource):
1614++ def _insert_item(self, data, src, target, pedigree, contentsource):
1615+ """
1616+ Upload image into glance and add image metadata to simplestreams index.
1617+
1618+@@ -470,6 +473,55 @@ class GlanceMirror(mirrors.BasicMirrorWr
1619+ # unused in insert_products below.
1620+ self.insert_products(None, target, None)
1621+
1622++ def insert_item(self, data, src, target, pedigree, contentsource):
1623++ """Queue item to be inserted in subsequent call to insert_version
1624++
1625++ This adds the item to self.inserts which is then handled in
1626++ insert_version. That allows the code to have context on
1627++ all the items for a given version, and "choose" one. Ie,
1628++ if both root.tar.xz and squashfs are available, preference
1629++ can be given to the root.tar.gz.
1630++ """
1631++
1632++ product_name, version_name, item_name = pedigree
1633++ if product_name not in self.inserts:
1634++ self.inserts[product_name] = {}
1635++ if version_name not in self.inserts[product_name]:
1636++ self.inserts[product_name][version_name] = {}
1637++
1638++ if 'ftype' in data:
1639++ ftype = data['ftype']
1640++ else:
1641++ flat = util.products_exdata(src, pedigree, include_top=False)
1642++ ftype = flat.get('ftype')
1643++ self.inserts[product_name][version_name][item_name] = (
1644++ ftype, (data, src, target, pedigree, contentsource))
1645++
1646++ def insert_version(self, data, src, target, pedigree):
1647++ """Upload all images for this version into glance
1648++ and add image metadata to simplestreams index.
1649++
1650++ All the work actually happens in _insert_item.
1651++ """
1652++
1653++ product_name, version_name = pedigree
1654++ inserts = self.inserts.get(product_name, {}).get(version_name, [])
1655++
1656++ rtar_names = [f for f in inserts
1657++ if inserts[f][0] in ('root.tar.gz', 'root.tar.xz')]
1658++
1659++ for _iname, (ftype, iargs) in inserts.items():
1660++ if ftype == "squashfs" and rtar_names:
1661++ LOG.info("[%s] Skipping ftype 'squashfs' image in preference"
1662++ "for root tarball type in %s",
1663++ '/'.join(pedigree), rtar_names)
1664++ continue
1665++ self._insert_item(*iargs)
1666++
1667++ # we do not specifically do anything for insert_version, but
1668++ # call parent.
1669++ super(GlanceMirror, self).insert_version(data, src, target, pedigree)
1670++
1671+ def remove_item(self, data, src, target, pedigree):
1672+ util.products_del(target, pedigree)
1673+ if 'id' in data:
1674+--- a/tests/unittests/test_glancemirror.py
1675++++ b/tests/unittests/test_glancemirror.py
1676+@@ -327,6 +327,15 @@ class TestGlanceMirror(TestCase):
1677+
1678+ self.assertEqual("root-tar", create_arguments["disk_format"])
1679+
1680++ def test_prepare_glance_arguments_disk_format_squashfs(self):
1681++ # squashfs images are acceptable for nova-lxd
1682++ source_entry = {"ftype": "squashfs"}
1683++ create_arguments = self.mirror.prepare_glance_arguments(
1684++ "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
1685++ image_properties=None)
1686++
1687++ self.assertEqual("squashfs", create_arguments["disk_format"])
1688++
1689+ def test_prepare_glance_arguments_size(self):
1690+ # Size is read from image metadata if defined.
1691+ source_entry = {"size": 5}
1692+@@ -470,7 +479,8 @@ class TestGlanceMirror(TestCase):
1693+ pedigree = (
1694+ u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
1695+ product = source_index[u'products'][pedigree[0]]
1696+- image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
1697++ ver_data = product[u'versions'][pedigree[1]]
1698++ image_data = ver_data[u'items'][pedigree[2]]
1699+
1700+ content_source = MemoryContentSource(
1701+ url="http://image-store/fooubuntu-X-disk1.img",
1702+@@ -489,6 +499,8 @@ class TestGlanceMirror(TestCase):
1703+
1704+ self.mirror.insert_item(
1705+ image_data, source_index, target, pedigree, content_source)
1706++ self.mirror.insert_version(
1707++ ver_data, source_index, target, pedigree[0:2])
1708+
1709+ passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
1710+
1711+@@ -532,7 +544,8 @@ class TestGlanceMirror(TestCase):
1712+ pedigree = (
1713+ u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
1714+ product = source_index[u'products'][pedigree[0]]
1715+- image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
1716++ ver_data = product[u'versions'][pedigree[1]]
1717++ image_data = ver_data[u'items'][pedigree[2]]
1718+
1719+ content_source = MemoryContentSource(
1720+ url="http://image-store/fooubuntu-X-disk1.img",
1721+@@ -551,6 +564,8 @@ class TestGlanceMirror(TestCase):
1722+
1723+ self.mirror.insert_item(
1724+ image_data, source_index, target, pedigree, content_source)
1725++ self.mirror.insert_version(
1726++ image_data, source_index, target, pedigree[0:2])
1727+
1728+ passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
1729+
1730+@@ -603,7 +618,8 @@ class TestGlanceMirror(TestCase):
1731+ source_index = copy.deepcopy(TEST_SOURCE_INDEX_ENTRY)
1732+ pedigree = TEST_IMAGE_PEDIGREE
1733+ product = source_index[u'products'][pedigree[0]]
1734+- image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
1735++ ver_data = product[u'versions'][pedigree[1]]
1736++ image_data = ver_data[u'items'][pedigree[2]]
1737+
1738+ content_source = MemoryContentSource(
1739+ url="http://image-store/fooubuntu-X-disk1.img",
1740+@@ -621,6 +637,8 @@ class TestGlanceMirror(TestCase):
1741+
1742+ self.mirror.insert_item(
1743+ image_data, source_index, target, pedigree, content_source)
1744++ self.mirror.insert_version(
1745++ ver_data, source_index, target, pedigree[0:2])
1746+
1747+ stored_index_content = self.mirror.store.data[
1748+ 'streams/v1/auto.sync.json']
1749diff --git a/debian/patches/460-glance-handle-v2-auth-with-sessions.patch b/debian/patches/460-glance-handle-v2-auth-with-sessions.patch
1750new file mode 100644
1751index 0000000..5fca027
1752--- /dev/null
1753+++ b/debian/patches/460-glance-handle-v2-auth-with-sessions.patch
1754@@ -0,0 +1,33 @@
1755+------------------------------------------------------------
1756+revno: 460 [merge]
1757+fixes bug: https://launchpad.net/bugs/1611987
1758+author: David Ames <david.ames@canonical.com>
1759+committer: Scott Moser <smoser@ubuntu.com>
1760+branch nick: trunk
1761+timestamp: Thu 2018-04-12 12:33:46 -0400
1762+message:
1763+ Glance: Handle Keystone v2 with session based authentication
1764+
1765+ There are three cases we have to handle:
1766+ - keystone v2 without sessions
1767+ - keystone v2 with sessions
1768+ - keystone v3 with sessions
1769+
1770+ We had the first and the last covered but not the middle. This change
1771+ addresses this.
1772+------------------------------------------------------------
1773+Use --include-merged or -n0 to see merged revisions.
1774+=== modified file 'simplestreams/openstack.py'
1775+--- a/simplestreams/openstack.py 2017-10-31 13:32:56 +0000
1776++++ b/simplestreams/openstack.py 2018-04-10 21:35:53 +0000
1777+@@ -181,7 +181,8 @@
1778+ endpoint = _get_endpoint(client, service, **kwargs)
1779+ # Session client does not have tenant_id set at client.tenant_id
1780+ # If client.tenant_id not set use method to get it
1781+- tenant_id = client.tenant_id or client.auth.client.get_project_id()
1782++ tenant_id = (client.tenant_id or client.get_project_id(client.session) or
1783++ client.auth.client.get_project_id())
1784+ info = {'token': client.auth_token, 'insecure': kwargs.get('insecure'),
1785+ 'cacert': kwargs.get('cacert'), 'endpoint': endpoint,
1786+ 'tenant_id': tenant_id}
1787+
1788diff --git a/debian/patches/series b/debian/patches/series
1789index 8eaeaf4..6b946bc 100644
1790--- a/debian/patches/series
1791+++ b/debian/patches/series
1792@@ -1,3 +1,10 @@
1793 read_signed-speed
1794 custom_user_agent_lp1578624.patch
1795-keystone-v3-support.patch
1796+428-do-not-require-that-hypervisor_config-be-present.patch
1797+433-glance-ignore-inactive-images.patch
1798+435-glance-refactor-for-testing.patch
1799+436-glance-fix-race-conditions.patch
1800+skip-openstack-tests-if-no-libs.patch
1801+450-453-454-keystone-v3-support.patch
1802+455-nova-lxd-support-squashfs-images.patch
1803+460-glance-handle-v2-auth-with-sessions.patch
1804diff --git a/debian/patches/skip-openstack-tests-if-no-libs.patch b/debian/patches/skip-openstack-tests-if-no-libs.patch
1805new file mode 100644
1806index 0000000..43e3665
1807--- /dev/null
1808+++ b/debian/patches/skip-openstack-tests-if-no-libs.patch
1809@@ -0,0 +1,36 @@
1810+Description: Skip tests of openstack to avoid build-depends.
1811+ This takes a bit of upstream commit revno 440 to skip openstack
1812+ tests if the libraries are not present.
1813+Applied-Upstream: revno 440
1814+Author: Scott Moser <smoser@ubuntu.com>
1815+--- a/tests/unittests/test_glancemirror.py
1816++++ b/tests/unittests/test_glancemirror.py
1817+@@ -1,12 +1,17 @@
1818+ from simplestreams.contentsource import MemoryContentSource
1819+-from simplestreams.mirrors.glance import GlanceMirror
1820+-from simplestreams.objectstores import MemoryObjectStore
1821++try:
1822++ from simplestreams.mirrors.glance import GlanceMirror
1823++ from simplestreams.objectstores import MemoryObjectStore
1824++ HAVE_OPENSTACK_LIBS = True
1825++except ImportError:
1826++ HAVE_OPENSTACK_LIBS = False
1827++
1828+ import simplestreams.util
1829+
1830+ import copy
1831+ import json
1832+ import os
1833+-from unittest import TestCase
1834++from unittest import TestCase, skipIf
1835+
1836+
1837+ # This is a real snippet from the simplestreams index entry for
1838+@@ -118,6 +123,7 @@ class FakeGlanceClient(object):
1839+ self.images = FakeImages()
1840+
1841+
1842++@skipIf(not HAVE_OPENSTACK_LIBS, "no python3 openstack available")
1843+ class TestGlanceMirror(TestCase):
1844+ """Tests for GlanceMirror methods."""
1845+

Subscribers

People subscribed via source and target branches