Merge ~smoser/ubuntu/+source/simplestreams:bug/artful-1686437-keystone-v3 into ubuntu/+source/simplestreams:ubuntu/artful-devel

Proposed by Scott Moser
Status: Rejected
Rejected by: Scott Moser
Proposed branch: ~smoser/ubuntu/+source/simplestreams:bug/artful-1686437-keystone-v3
Merge into: ubuntu/+source/simplestreams:ubuntu/artful-devel
Diff against target: 380 lines (+342/-0)
6 files modified
debian/changelog (+8/-0)
debian/patches/460-glance-handle-v2-auth-with-sessions.patch (+33/-0)
debian/patches/keystone-v3-1719879.patch (+35/-0)
debian/patches/keystone-v3-1728982.patch (+32/-0)
debian/patches/nova-lxd-support-squashfs-images.patch (+230/-0)
debian/patches/series (+4/-0)
Reviewer Review Type Date Requested Status
Billy Olsen (community) Approve
Scott Moser (community) Needs Resubmitting
Felipe Reyes Pending
Eric Desrochers Pending
Review via email: mp+341215@code.launchpad.net

Description of the change

Openstack keystone v3 and nova-lxd squashfs SRU

* Openstack: Add keystone v3 auth support (LP: #1719879, #1728982).
* Openstack: support uploading squash images for nova-lxd (LP: #1686086)

Bugs:
 * bug 1686086: glance mirror and nova-lxd need support for squashfs images
 * bug 1686437: [SRU] glance sync: need keystone v3 auth support
 * bug 1719879: [artful only] swift client needs to use v1 auth prior to ocata
 * bug 1728982: [artful only] openstack mirror with keystone v3 always imports new images
 * bug 1611987: glance-simplestreams-sync charm doesn't support keystone v3

Merge proposals:
 - xenial: https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341214
 - artful: https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341215

To post a comment you must log in.
Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) :
review: Approve
Revision history for this message
Scott Moser (smoser) wrote :

we should pull revno 455 as Billy Olson pointed out in the xenial merge proposasl
https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341214

review: Needs Fixing
040d31b... by Scott Moser

OpenStack: support uploading squash images for nova-lxd

Ubuntu 17.04 and newer do not have root.tar.gz or root.tar.xz
images available. To add support for populating 17.10+ the mirror
needs to better support squashfs.

LP: #1686086

52c95f5... by Scott Moser

update changelog

Revision history for this message
Scott Moser (smoser) wrote :

I integrated Billy's suggestion, and uploaded to ppa at
 https://launchpad.net/~smoser/+archive/ubuntu/sstream-ks3

Revision history for this message
Scott Moser (smoser) :
review: Needs Resubmitting
Revision history for this message
Billy Olsen (billy-olsen) wrote :

Thanks for including it Scott - new changes look good to me.

review: Approve
Revision history for this message
Felipe Reyes (freyes) wrote :

Running this patch in an environment with the following characteristics:

* keystone -> preferred-api-version=3
* glance-simplestreams-sync -> use_swift=true
* swift is running xenial-mitaka (to test bug 1719879), the other components are running artful

Evidence: https://pastebin.ubuntu.com/p/NDXmXG9ZkK/
Bundles used:
 - swift: http://paste.ubuntu.com/p/RzZ2JMBjbg/
 - artful-pike: http://paste.ubuntu.com/p/27mzxxX9jC/

I have pending to do the tests for https://bugs.launchpad.net/simplestreams/+bug/1686086

9a83fc1... by Scott Moser

Pull in revno 460 for keystone v2 session support.

460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)

6029f26... by Scott Moser

update changelog

8faca4a... by Scott Moser

releasing package simplestreams version 0.1.0~bzr450-0ubuntu1.1

Revision history for this message
Scott Moser (smoser) wrote :

This (artful) merge proposal for SRU is in the same state as the xenial MP [1].
Interested parties have successfully tested the PPA [2].
It is ready for upload except for SRU templates on all the bugs fixed.
The bugs now fixed here, as seen in debian/changelog, are:

  * Openstack: keystone v3 auth fixes. (LP: #1719879, #1728982)
  * Openstack: support uploading squash images for nova-lxd (LP: #1686086)
  * Openstack: Handlel v2 authentication with sessions (LP: #1611987)

--
[1] https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341214

Revision history for this message
Scott Moser (smoser) wrote :

marked rejected, just due to artful EOL.

Unmerged commits

8faca4a... by Scott Moser

releasing package simplestreams version 0.1.0~bzr450-0ubuntu1.1

6029f26... by Scott Moser

update changelog

9a83fc1... by Scott Moser

Pull in revno 460 for keystone v2 session support.

460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)

52c95f5... by Scott Moser

update changelog

040d31b... by Scott Moser

OpenStack: support uploading squash images for nova-lxd

Ubuntu 17.04 and newer do not have root.tar.gz or root.tar.xz
images available. To add support for populating 17.10+ the mirror
needs to better support squashfs.

LP: #1686086

fa50428... by Scott Moser

update changelog

6105582... by Scott Moser

Openstack: keystone v3 auth fixes.

This brings the next two upstream commits back to Artful
(revno 453 and 454). They fix some keystone v3 releated issues.

LP: #1719879
LP: #1728982

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/debian/changelog b/debian/changelog
2index e9052a4..4dbbdee 100644
3--- a/debian/changelog
4+++ b/debian/changelog
5@@ -1,3 +1,11 @@
6+simplestreams (0.1.0~bzr450-0ubuntu1.1) artful; urgency=medium
7+
8+ * Openstack: keystone v3 auth fixes. (LP: #1719879, #1728982)
9+ * Openstack: support uploading squash images for nova-lxd (LP: #1686086)
10+ * Openstack: Handlel v2 authentication with sessions (LP: #1611987)
11+
12+ -- Scott Moser <smoser@ubuntu.com> Thu, 12 Apr 2018 12:59:49 -0400
13+
14 simplestreams (0.1.0~bzr450-0ubuntu1) artful; urgency=medium
15
16 * New upstream snapshot.
17diff --git a/debian/patches/460-glance-handle-v2-auth-with-sessions.patch b/debian/patches/460-glance-handle-v2-auth-with-sessions.patch
18new file mode 100644
19index 0000000..5fca027
20--- /dev/null
21+++ b/debian/patches/460-glance-handle-v2-auth-with-sessions.patch
22@@ -0,0 +1,33 @@
23+------------------------------------------------------------
24+revno: 460 [merge]
25+fixes bug: https://launchpad.net/bugs/1611987
26+author: David Ames <david.ames@canonical.com>
27+committer: Scott Moser <smoser@ubuntu.com>
28+branch nick: trunk
29+timestamp: Thu 2018-04-12 12:33:46 -0400
30+message:
31+ Glance: Handle Keystone v2 with session based authentication
32+
33+ There are three cases we have to handle:
34+ - keystone v2 without sessions
35+ - keystone v2 with sessions
36+ - keystone v3 with sessions
37+
38+ We had the first and the last covered but not the middle. This change
39+ addresses this.
40+------------------------------------------------------------
41+Use --include-merged or -n0 to see merged revisions.
42+=== modified file 'simplestreams/openstack.py'
43+--- a/simplestreams/openstack.py 2017-10-31 13:32:56 +0000
44++++ b/simplestreams/openstack.py 2018-04-10 21:35:53 +0000
45+@@ -181,7 +181,8 @@
46+ endpoint = _get_endpoint(client, service, **kwargs)
47+ # Session client does not have tenant_id set at client.tenant_id
48+ # If client.tenant_id not set use method to get it
49+- tenant_id = client.tenant_id or client.auth.client.get_project_id()
50++ tenant_id = (client.tenant_id or client.get_project_id(client.session) or
51++ client.auth.client.get_project_id())
52+ info = {'token': client.auth_token, 'insecure': kwargs.get('insecure'),
53+ 'cacert': kwargs.get('cacert'), 'endpoint': endpoint,
54+ 'tenant_id': tenant_id}
55+
56diff --git a/debian/patches/keystone-v3-1719879.patch b/debian/patches/keystone-v3-1719879.patch
57new file mode 100644
58index 0000000..cbc8575
59--- /dev/null
60+++ b/debian/patches/keystone-v3-1719879.patch
61@@ -0,0 +1,35 @@
62+------------------------------------------------------------
63+revno: 454
64+fixes bug: https://launchpad.net/bugs/1719879
65+author: David Ames <david.ames@canonical.com>
66+committer: Scott Moser <smoser@ubuntu.com>
67+branch nick: trunk
68+timestamp: Tue 2017-10-31 09:39:45 -0400
69+message:
70+ Support using older swift clients (older than Ocata).
71+
72+ The swift client lagged behind other openstack client libraries in
73+ gaining support for v3 auth. The fix here is to try the v3 auth, and
74+ catch a TypeError and fallback older.
75+diff:
76+=== modified file 'simplestreams/objectstores/swift.py'
77+--- a/simplestreams/objectstores/swift.py 2017-09-01 23:14:24 +0000
78++++ b/simplestreams/objectstores/swift.py 2017-10-31 13:39:45 +0000
79+@@ -35,9 +35,14 @@
80+ connargs.update({k: kwargs.get(k) for k in pt if k in kwargs})
81+ if kwargs.get('session'):
82+ sess = kwargs.get('session')
83+- return Connection(session=sess)
84+- else:
85+- return Connection(**connargs)
86++ try:
87++ # If session is available try it
88++ return Connection(session=sess)
89++ except TypeError:
90++ # The edge case where session is availble but swiftclient is
91++ # < 3.3.0. Use the old style method for Connection.
92++ pass
93++ return Connection(**connargs)
94+
95+
96+ class SwiftContentSource(cs.IteratorContentSource):
97diff --git a/debian/patches/keystone-v3-1728982.patch b/debian/patches/keystone-v3-1728982.patch
98new file mode 100644
99index 0000000..ec0cd2d
100--- /dev/null
101+++ b/debian/patches/keystone-v3-1728982.patch
102@@ -0,0 +1,32 @@
103+------------------------------------------------------------
104+revno: 453
105+fixes bug: https://launchpad.net/bugs/1728982
106+author: David Ames <david.ames@canonical.com>
107+committer: Scott Moser <smoser@ubuntu.com>
108+branch nick: trunk
109+timestamp: Tue 2017-10-31 09:32:56 -0400
110+message:
111+ Fix glance sync always uploading new image with keystone v3 auth.
112+
113+ When using keystone v3 auth, a 'sync' against a populated glance
114+ mirror would provide None as the tenant_id when querying glance for
115+ existing images.
116+
117+ The fix is just to get the tenant_id from the auth client in that case.
118+diff:
119+=== modified file 'simplestreams/openstack.py'
120+--- a/simplestreams/openstack.py 2017-09-11 16:00:49 +0000
121++++ b/simplestreams/openstack.py 2017-10-31 13:32:56 +0000
122+@@ -179,9 +179,12 @@
123+ client = get_ksclient(**kwargs)
124+
125+ endpoint = _get_endpoint(client, service, **kwargs)
126++ # Session client does not have tenant_id set at client.tenant_id
127++ # If client.tenant_id not set use method to get it
128++ tenant_id = client.tenant_id or client.auth.client.get_project_id()
129+ info = {'token': client.auth_token, 'insecure': kwargs.get('insecure'),
130+ 'cacert': kwargs.get('cacert'), 'endpoint': endpoint,
131+- 'tenant_id': client.tenant_id}
132++ 'tenant_id': tenant_id}
133+ if not _LEGACY_CLIENTS:
134+ info['session'] = client.session
135diff --git a/debian/patches/nova-lxd-support-squashfs-images.patch b/debian/patches/nova-lxd-support-squashfs-images.patch
136new file mode 100644
137index 0000000..071705c
138--- /dev/null
139+++ b/debian/patches/nova-lxd-support-squashfs-images.patch
140@@ -0,0 +1,230 @@
141+------------------------------------------------------------
142+revno: 455 [merge]
143+fixes bug: https://launchpad.net/bugs/1686086
144+committer: Scott Moser <smoser@ubuntu.com>
145+branch nick: trunk
146+timestamp: Thu 2017-11-02 15:03:37 -0400
147+message:
148+ OpenStack: support uploading squash images for nova-lxd.
149+
150+ Previously, populating a nova-lxd cloud was possible by using
151+ root.tar.gz. A filter like:
152+ ftype~(root.tar.gz|root.tar.xz)
153+ would cause simplestreams to upload an image with 'disk-format' of
154+ root-tar.
155+
156+ However, Ubuntu 17.04 and newer do not have root.tar.gz or root.tar.xz
157+ images available. Currently here is what is available:
158+ 14.04: root.tar.gz root.tar.xz
159+ 16.04: root.tar.gz root.tar.xz squashfs
160+ 17.10: squashfs
161+
162+ If we simply expected the user to change their filter to include
163+ root.tar.xz|squashfs
164+ Then they would get two lxd images imported for 16.04 each version.
165+
166+ The change here is to not do anything for an item insert, but instead
167+ insert when the version's insert is called. Then, all the information
168+ about what images there are is available, and it can "pick"
169+ one or the other. Currently preference is given to the .tar.xz format.
170+
171+ The end result is that now users can specify an ftype filter of:
172+ ftype~(root.tar.gz|root.tar.xz|squashfs)
173+ and the right thing will be done.
174+
175+ Also here is simple knowledge that the squashfs type should be
176+ uploaded to glance with a 'disk_format' of 'squashfs'.
177+------------------------------------------------------------
178+Use --include-merged or -n0 to see merged revisions.
179+=== modified file 'simplestreams/mirrors/glance.py'
180+--- a/simplestreams/mirrors/glance.py
181++++ b/simplestreams/mirrors/glance.py
182+@@ -66,25 +66,27 @@ def canonicalize_arch(arch):
183+ return newarch
184+
185+
186+-LXC_FTYPES = [
187+- 'root.tar.gz',
188+- 'root.tar.xz',
189+- 'squashfs',
190+-]
191+-
192+-QEMU_FTYPES = [
193+- 'disk.img',
194+- 'disk1.img',
195+-]
196++LXC_FTYPES = {
197++ 'root.tar.gz': 'root-tar',
198++ 'root.tar.xz': 'root-tar',
199++ 'squashfs': 'squashfs',
200++}
201++
202++QEMU_FTYPES = {
203++ 'disk.img': 'qcow2',
204++ 'disk1.img': 'qcow2',
205++}
206+
207+
208+ def disk_format(ftype):
209+- '''Canonicalize disk formats for use in OpenStack'''
210++ '''Canonicalize disk formats for use in OpenStack.
211++ Input ftype is a 'ftype' from a simplestream feed.
212++ Return value is the appropriate 'disk_format' for glance.'''
213+ newftype = ftype.lower()
214+ if newftype in LXC_FTYPES:
215+- return 'root-tar'
216++ return LXC_FTYPES[newftype]
217+ if newftype in QEMU_FTYPES:
218+- return 'qcow2'
219++ return QEMU_FTYPES[newftype]
220+ return None
221+
222+
223+@@ -160,6 +162,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
224+ self.content_id = config.get("content_id")
225+ self.modify_hook = config.get("modify_hook")
226+
227++ self.inserts = {}
228+ if not self.content_id:
229+ raise TypeError("content_id is required")
230+
231+@@ -408,7 +411,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
232+
233+ return output_entry
234+
235+- def insert_item(self, data, src, target, pedigree, contentsource):
236++ def _insert_item(self, data, src, target, pedigree, contentsource):
237+ """
238+ Upload image into glance and add image metadata to simplestreams index.
239+
240+@@ -470,6 +473,55 @@ class GlanceMirror(mirrors.BasicMirrorWr
241+ # unused in insert_products below.
242+ self.insert_products(None, target, None)
243+
244++ def insert_item(self, data, src, target, pedigree, contentsource):
245++ """Queue item to be inserted in subsequent call to insert_version
246++
247++ This adds the item to self.inserts which is then handled in
248++ insert_version. That allows the code to have context on
249++ all the items for a given version, and "choose" one. Ie,
250++ if both root.tar.xz and squashfs are available, preference
251++ can be given to the root.tar.gz.
252++ """
253++
254++ product_name, version_name, item_name = pedigree
255++ if product_name not in self.inserts:
256++ self.inserts[product_name] = {}
257++ if version_name not in self.inserts[product_name]:
258++ self.inserts[product_name][version_name] = {}
259++
260++ if 'ftype' in data:
261++ ftype = data['ftype']
262++ else:
263++ flat = util.products_exdata(src, pedigree, include_top=False)
264++ ftype = flat.get('ftype')
265++ self.inserts[product_name][version_name][item_name] = (
266++ ftype, (data, src, target, pedigree, contentsource))
267++
268++ def insert_version(self, data, src, target, pedigree):
269++ """Upload all images for this version into glance
270++ and add image metadata to simplestreams index.
271++
272++ All the work actually happens in _insert_item.
273++ """
274++
275++ product_name, version_name = pedigree
276++ inserts = self.inserts.get(product_name, {}).get(version_name, [])
277++
278++ rtar_names = [f for f in inserts
279++ if inserts[f][0] in ('root.tar.gz', 'root.tar.xz')]
280++
281++ for _iname, (ftype, iargs) in inserts.items():
282++ if ftype == "squashfs" and rtar_names:
283++ LOG.info("[%s] Skipping ftype 'squashfs' image in preference"
284++ "for root tarball type in %s",
285++ '/'.join(pedigree), rtar_names)
286++ continue
287++ self._insert_item(*iargs)
288++
289++ # we do not specifically do anything for insert_version, but
290++ # call parent.
291++ super(GlanceMirror, self).insert_version(data, src, target, pedigree)
292++
293+ def remove_item(self, data, src, target, pedigree):
294+ util.products_del(target, pedigree)
295+ if 'id' in data:
296+--- a/tests/unittests/test_glancemirror.py
297++++ b/tests/unittests/test_glancemirror.py
298+@@ -333,6 +333,15 @@ class TestGlanceMirror(TestCase):
299+
300+ self.assertEqual("root-tar", create_arguments["disk_format"])
301+
302++ def test_prepare_glance_arguments_disk_format_squashfs(self):
303++ # squashfs images are acceptable for nova-lxd
304++ source_entry = {"ftype": "squashfs"}
305++ create_arguments = self.mirror.prepare_glance_arguments(
306++ "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
307++ image_properties=None)
308++
309++ self.assertEqual("squashfs", create_arguments["disk_format"])
310++
311+ def test_prepare_glance_arguments_size(self):
312+ # Size is read from image metadata if defined.
313+ source_entry = {"size": 5}
314+@@ -476,7 +485,8 @@ class TestGlanceMirror(TestCase):
315+ pedigree = (
316+ u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
317+ product = source_index[u'products'][pedigree[0]]
318+- image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
319++ ver_data = product[u'versions'][pedigree[1]]
320++ image_data = ver_data[u'items'][pedigree[2]]
321+
322+ content_source = MemoryContentSource(
323+ url="http://image-store/fooubuntu-X-disk1.img",
324+@@ -495,6 +505,8 @@ class TestGlanceMirror(TestCase):
325+
326+ self.mirror.insert_item(
327+ image_data, source_index, target, pedigree, content_source)
328++ self.mirror.insert_version(
329++ ver_data, source_index, target, pedigree[0:2])
330+
331+ passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
332+
333+@@ -538,7 +550,8 @@ class TestGlanceMirror(TestCase):
334+ pedigree = (
335+ u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
336+ product = source_index[u'products'][pedigree[0]]
337+- image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
338++ ver_data = product[u'versions'][pedigree[1]]
339++ image_data = ver_data[u'items'][pedigree[2]]
340+
341+ content_source = MemoryContentSource(
342+ url="http://image-store/fooubuntu-X-disk1.img",
343+@@ -557,6 +570,8 @@ class TestGlanceMirror(TestCase):
344+
345+ self.mirror.insert_item(
346+ image_data, source_index, target, pedigree, content_source)
347++ self.mirror.insert_version(
348++ image_data, source_index, target, pedigree[0:2])
349+
350+ passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
351+
352+@@ -609,7 +624,8 @@ class TestGlanceMirror(TestCase):
353+ source_index = copy.deepcopy(TEST_SOURCE_INDEX_ENTRY)
354+ pedigree = TEST_IMAGE_PEDIGREE
355+ product = source_index[u'products'][pedigree[0]]
356+- image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
357++ ver_data = product[u'versions'][pedigree[1]]
358++ image_data = ver_data[u'items'][pedigree[2]]
359+
360+ content_source = MemoryContentSource(
361+ url="http://image-store/fooubuntu-X-disk1.img",
362+@@ -627,6 +643,8 @@ class TestGlanceMirror(TestCase):
363+
364+ self.mirror.insert_item(
365+ image_data, source_index, target, pedigree, content_source)
366++ self.mirror.insert_version(
367++ ver_data, source_index, target, pedigree[0:2])
368+
369+ stored_index_content = self.mirror.store.data[
370+ 'streams/v1/auto.sync.json']
371diff --git a/debian/patches/series b/debian/patches/series
372new file mode 100644
373index 0000000..ee81491
374--- /dev/null
375+++ b/debian/patches/series
376@@ -0,0 +1,4 @@
377+keystone-v3-1719879.patch
378+keystone-v3-1728982.patch
379+nova-lxd-support-squashfs-images.patch
380+460-glance-handle-v2-auth-with-sessions.patch

Subscribers

People subscribed via source and target branches