Ubuntu
simplestreams package

Merge ~smoser/ubuntu/+source/simplestreams:bug/artful-1686437-keystone-v3 into ubuntu/+source/simplestreams:ubuntu/artful-devel

  1. Git
  2. lp:~smoser/ubuntu/+source/simplestreams
  3. bug/artful-1686437-keystone-v3
  4. Merge into ubuntu/artful-devel
Proposed by Scott Moser
Status: Rejected
Rejected by: Scott Moser
Proposed branch: ~smoser/ubuntu/+source/simplestreams:bug/artful-1686437-keystone-v3
Merge into: ubuntu/+source/simplestreams:ubuntu/artful-devel
Diff against target: 380 lines (+342/-0)
6 files modified
debian/changelog (+8/-0)
debian/patches/460-glance-handle-v2-auth-with-sessions.patch (+33/-0)
debian/patches/keystone-v3-1719879.patch (+35/-0)
debian/patches/keystone-v3-1728982.patch (+32/-0)
debian/patches/nova-lxd-support-squashfs-images.patch (+230/-0)
debian/patches/series (+4/-0)
Reviewer Review Type Date Requested Status
Billy Olsen (community) Approve
Scott Moser (community) Needs Resubmitting
Felipe Reyes Pending
Eric Desrochers Pending
Review via email: mp+341215@code.launchpad.net

Description of the change

Openstack keystone v3 and nova-lxd squashfs SRU

* Openstack: Add keystone v3 auth support (LP: #1719879, #1728982).
* Openstack: support uploading squash images for nova-lxd (LP: #1686086)

Bugs:
 * bug 1686086: glance mirror and nova-lxd need support for squashfs images
 * bug 1686437: [SRU] glance sync: need keystone v3 auth support
 * bug 1719879: [artful only] swift client needs to use v1 auth prior to ocata
 * bug 1728982: [artful only] openstack mirror with keystone v3 always imports new images
 * bug 1611987: glance-simplestreams-sync charm doesn't support keystone v3

Merge proposals:
 - xenial: https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341214
 - artful: https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341215

To post a comment you must log in.
Revision history for this message
Scott Moser (smoser) wrote :

I've uploaded this just now. It is in the queue at
 https://launchpad.net/ubuntu/xenial/+upload/17909462/+files/simplestreams_0.1.0~bzr426-0ubuntu1.3.dsc

Revision history for this message
Scott Moser (smoser) wrote :

Sorry, bad link.

Uploaded to artful:
 sru queue: https://launchpad.net/ubuntu/artful/+queue?queue_state=1&queue_text=
 dsc file: https://launchpad.net/ubuntu/artful/+upload/17909445/+files/simplestreams_0.1.0~bzr450-0ubuntu1.1.dsc

Revision history for this message
Scott Moser (smoser) :
review: Approve
Revision history for this message
Scott Moser (smoser) wrote :

we should pull revno 455 as Billy Olson pointed out in the xenial merge proposasl
https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341214

review: Needs Fixing
~smoser/ubuntu/+source/simplestreams:bug/artful-1686437-keystone-v3 updated
040d31b... by Scott Moser

OpenStack: support uploading squash images for nova-lxd

Ubuntu 17.04 and newer do not have root.tar.gz or root.tar.xz
images available. To add support for populating 17.10+ the mirror
needs to better support squashfs.

LP: #1686086

52c95f5... by Scott Moser

update changelog

Revision history for this message
Scott Moser (smoser) wrote :

I integrated Billy's suggestion, and uploaded to ppa at
 https://launchpad.net/~smoser/+archive/ubuntu/sstream-ks3

Revision history for this message
Scott Moser (smoser) :
review: Needs Resubmitting
Revision history for this message
Billy Olsen (billy-olsen) wrote :

Thanks for including it Scott - new changes look good to me.

review: Approve
Revision history for this message
Felipe Reyes (freyes) wrote :

Running this patch in an environment with the following characteristics:

* keystone -> preferred-api-version=3
* glance-simplestreams-sync -> use_swift=true
* swift is running xenial-mitaka (to test bug 1719879), the other components are running artful

Evidence: https://pastebin.ubuntu.com/p/NDXmXG9ZkK/
Bundles used:
 - swift: http://paste.ubuntu.com/p/RzZ2JMBjbg/
 - artful-pike: http://paste.ubuntu.com/p/27mzxxX9jC/

I have pending to do the tests for https://bugs.launchpad.net/simplestreams/+bug/1686086

~smoser/ubuntu/+source/simplestreams:bug/artful-1686437-keystone-v3 updated
9a83fc1... by Scott Moser

Pull in revno 460 for keystone v2 session support.

460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)

6029f26... by Scott Moser

update changelog

8faca4a... by Scott Moser

releasing package simplestreams version 0.1.0~bzr450-0ubuntu1.1

Revision history for this message
Scott Moser (smoser) wrote :

This (artful) merge proposal for SRU is in the same state as the xenial MP [1].
Interested parties have successfully tested the PPA [2].
It is ready for upload except for SRU templates on all the bugs fixed.
The bugs now fixed here, as seen in debian/changelog, are:

  * Openstack: keystone v3 auth fixes. (LP: #1719879, #1728982)
  * Openstack: support uploading squash images for nova-lxd (LP: #1686086)
  * Openstack: Handlel v2 authentication with sessions (LP: #1611987)

--
[1] https://code.launchpad.net/~smoser/ubuntu/+source/simplestreams/+git/simplestreams/+merge/341214

Revision history for this message
Scott Moser (smoser) wrote :

marked rejected, just due to artful EOL.

Unmerged commits

8faca4a... by Scott Moser

releasing package simplestreams version 0.1.0~bzr450-0ubuntu1.1

6029f26... by Scott Moser

update changelog

9a83fc1... by Scott Moser

Pull in revno 460 for keystone v2 session support.

460-glance-handle-v2-auth-with-sessions.patch (LP: #1611987)

52c95f5... by Scott Moser

update changelog

040d31b... by Scott Moser

OpenStack: support uploading squash images for nova-lxd

Ubuntu 17.04 and newer do not have root.tar.gz or root.tar.xz
images available. To add support for populating 17.10+ the mirror
needs to better support squashfs.

LP: #1686086

fa50428... by Scott Moser

update changelog

6105582... by Scott Moser

Openstack: keystone v3 auth fixes.

This brings the next two upstream commits back to Artful
(revno 453 and 454). They fix some keystone v3 releated issues.

LP: #1719879
LP: #1728982

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/debian/changelog b/debian/changelog
index e9052a4..4dbbdee 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
1simplestreams (0.1.0~bzr450-0ubuntu1.1) artful; urgency=medium
2
3 * Openstack: keystone v3 auth fixes. (LP: #1719879, #1728982)
4 * Openstack: support uploading squash images for nova-lxd (LP: #1686086)
5 * Openstack: Handlel v2 authentication with sessions (LP: #1611987)
6
7 -- Scott Moser <smoser@ubuntu.com> Thu, 12 Apr 2018 12:59:49 -0400
8
1simplestreams (0.1.0~bzr450-0ubuntu1) artful; urgency=medium9simplestreams (0.1.0~bzr450-0ubuntu1) artful; urgency=medium
210
3 * New upstream snapshot.11 * New upstream snapshot.
diff --git a/debian/patches/460-glance-handle-v2-auth-with-sessions.patch b/debian/patches/460-glance-handle-v2-auth-with-sessions.patch
4new file mode 10064412new file mode 100644
index 0000000..5fca027
--- /dev/null
+++ b/debian/patches/460-glance-handle-v2-auth-with-sessions.patch
@@ -0,0 +1,33 @@
1------------------------------------------------------------
2revno: 460 [merge]
3fixes bug: https://launchpad.net/bugs/1611987
4author: David Ames <david.ames@canonical.com>
5committer: Scott Moser <smoser@ubuntu.com>
6branch nick: trunk
7timestamp: Thu 2018-04-12 12:33:46 -0400
8message:
9 Glance: Handle Keystone v2 with session based authentication
10
11 There are three cases we have to handle:
12 - keystone v2 without sessions
13 - keystone v2 with sessions
14 - keystone v3 with sessions
15
16 We had the first and the last covered but not the middle. This change
17 addresses this.
18------------------------------------------------------------
19Use --include-merged or -n0 to see merged revisions.
20=== modified file 'simplestreams/openstack.py'
21--- a/simplestreams/openstack.py 2017-10-31 13:32:56 +0000
22+++ b/simplestreams/openstack.py 2018-04-10 21:35:53 +0000
23@@ -181,7 +181,8 @@
24 endpoint = _get_endpoint(client, service, **kwargs)
25 # Session client does not have tenant_id set at client.tenant_id
26 # If client.tenant_id not set use method to get it
27- tenant_id = client.tenant_id or client.auth.client.get_project_id()
28+ tenant_id = (client.tenant_id or client.get_project_id(client.session) or
29+ client.auth.client.get_project_id())
30 info = {'token': client.auth_token, 'insecure': kwargs.get('insecure'),
31 'cacert': kwargs.get('cacert'), 'endpoint': endpoint,
32 'tenant_id': tenant_id}
33
diff --git a/debian/patches/keystone-v3-1719879.patch b/debian/patches/keystone-v3-1719879.patch
0new file mode 10064434new file mode 100644
index 0000000..cbc8575
--- /dev/null
+++ b/debian/patches/keystone-v3-1719879.patch
@@ -0,0 +1,35 @@
1------------------------------------------------------------
2revno: 454
3fixes bug: https://launchpad.net/bugs/1719879
4author: David Ames <david.ames@canonical.com>
5committer: Scott Moser <smoser@ubuntu.com>
6branch nick: trunk
7timestamp: Tue 2017-10-31 09:39:45 -0400
8message:
9 Support using older swift clients (older than Ocata).
10
11 The swift client lagged behind other openstack client libraries in
12 gaining support for v3 auth. The fix here is to try the v3 auth, and
13 catch a TypeError and fallback older.
14diff:
15=== modified file 'simplestreams/objectstores/swift.py'
16--- a/simplestreams/objectstores/swift.py 2017-09-01 23:14:24 +0000
17+++ b/simplestreams/objectstores/swift.py 2017-10-31 13:39:45 +0000
18@@ -35,9 +35,14 @@
19 connargs.update({k: kwargs.get(k) for k in pt if k in kwargs})
20 if kwargs.get('session'):
21 sess = kwargs.get('session')
22- return Connection(session=sess)
23- else:
24- return Connection(**connargs)
25+ try:
26+ # If session is available try it
27+ return Connection(session=sess)
28+ except TypeError:
29+ # The edge case where session is availble but swiftclient is
30+ # < 3.3.0. Use the old style method for Connection.
31+ pass
32+ return Connection(**connargs)
33
34
35 class SwiftContentSource(cs.IteratorContentSource):
diff --git a/debian/patches/keystone-v3-1728982.patch b/debian/patches/keystone-v3-1728982.patch
0new file mode 10064436new file mode 100644
index 0000000..ec0cd2d
--- /dev/null
+++ b/debian/patches/keystone-v3-1728982.patch
@@ -0,0 +1,32 @@
1------------------------------------------------------------
2revno: 453
3fixes bug: https://launchpad.net/bugs/1728982
4author: David Ames <david.ames@canonical.com>
5committer: Scott Moser <smoser@ubuntu.com>
6branch nick: trunk
7timestamp: Tue 2017-10-31 09:32:56 -0400
8message:
9 Fix glance sync always uploading new image with keystone v3 auth.
10
11 When using keystone v3 auth, a 'sync' against a populated glance
12 mirror would provide None as the tenant_id when querying glance for
13 existing images.
14
15 The fix is just to get the tenant_id from the auth client in that case.
16diff:
17=== modified file 'simplestreams/openstack.py'
18--- a/simplestreams/openstack.py 2017-09-11 16:00:49 +0000
19+++ b/simplestreams/openstack.py 2017-10-31 13:32:56 +0000
20@@ -179,9 +179,12 @@
21 client = get_ksclient(**kwargs)
22
23 endpoint = _get_endpoint(client, service, **kwargs)
24+ # Session client does not have tenant_id set at client.tenant_id
25+ # If client.tenant_id not set use method to get it
26+ tenant_id = client.tenant_id or client.auth.client.get_project_id()
27 info = {'token': client.auth_token, 'insecure': kwargs.get('insecure'),
28 'cacert': kwargs.get('cacert'), 'endpoint': endpoint,
29- 'tenant_id': client.tenant_id}
30+ 'tenant_id': tenant_id}
31 if not _LEGACY_CLIENTS:
32 info['session'] = client.session
diff --git a/debian/patches/nova-lxd-support-squashfs-images.patch b/debian/patches/nova-lxd-support-squashfs-images.patch
0new file mode 10064433new file mode 100644
index 0000000..071705c
--- /dev/null
+++ b/debian/patches/nova-lxd-support-squashfs-images.patch
@@ -0,0 +1,230 @@
1------------------------------------------------------------
2revno: 455 [merge]
3fixes bug: https://launchpad.net/bugs/1686086
4committer: Scott Moser <smoser@ubuntu.com>
5branch nick: trunk
6timestamp: Thu 2017-11-02 15:03:37 -0400
7message:
8 OpenStack: support uploading squash images for nova-lxd.
9
10 Previously, populating a nova-lxd cloud was possible by using
11 root.tar.gz. A filter like:
12 ftype~(root.tar.gz|root.tar.xz)
13 would cause simplestreams to upload an image with 'disk-format' of
14 root-tar.
15
16 However, Ubuntu 17.04 and newer do not have root.tar.gz or root.tar.xz
17 images available. Currently here is what is available:
18 14.04: root.tar.gz root.tar.xz
19 16.04: root.tar.gz root.tar.xz squashfs
20 17.10: squashfs
21
22 If we simply expected the user to change their filter to include
23 root.tar.xz|squashfs
24 Then they would get two lxd images imported for 16.04 each version.
25
26 The change here is to not do anything for an item insert, but instead
27 insert when the version's insert is called. Then, all the information
28 about what images there are is available, and it can "pick"
29 one or the other. Currently preference is given to the .tar.xz format.
30
31 The end result is that now users can specify an ftype filter of:
32 ftype~(root.tar.gz|root.tar.xz|squashfs)
33 and the right thing will be done.
34
35 Also here is simple knowledge that the squashfs type should be
36 uploaded to glance with a 'disk_format' of 'squashfs'.
37------------------------------------------------------------
38Use --include-merged or -n0 to see merged revisions.
39=== modified file 'simplestreams/mirrors/glance.py'
40--- a/simplestreams/mirrors/glance.py
41+++ b/simplestreams/mirrors/glance.py
42@@ -66,25 +66,27 @@ def canonicalize_arch(arch):
43 return newarch
44
45
46-LXC_FTYPES = [
47- 'root.tar.gz',
48- 'root.tar.xz',
49- 'squashfs',
50-]
51-
52-QEMU_FTYPES = [
53- 'disk.img',
54- 'disk1.img',
55-]
56+LXC_FTYPES = {
57+ 'root.tar.gz': 'root-tar',
58+ 'root.tar.xz': 'root-tar',
59+ 'squashfs': 'squashfs',
60+}
61+
62+QEMU_FTYPES = {
63+ 'disk.img': 'qcow2',
64+ 'disk1.img': 'qcow2',
65+}
66
67
68 def disk_format(ftype):
69- '''Canonicalize disk formats for use in OpenStack'''
70+ '''Canonicalize disk formats for use in OpenStack.
71+ Input ftype is a 'ftype' from a simplestream feed.
72+ Return value is the appropriate 'disk_format' for glance.'''
73 newftype = ftype.lower()
74 if newftype in LXC_FTYPES:
75- return 'root-tar'
76+ return LXC_FTYPES[newftype]
77 if newftype in QEMU_FTYPES:
78- return 'qcow2'
79+ return QEMU_FTYPES[newftype]
80 return None
81
82
83@@ -160,6 +162,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
84 self.content_id = config.get("content_id")
85 self.modify_hook = config.get("modify_hook")
86
87+ self.inserts = {}
88 if not self.content_id:
89 raise TypeError("content_id is required")
90
91@@ -408,7 +411,7 @@ class GlanceMirror(mirrors.BasicMirrorWr
92
93 return output_entry
94
95- def insert_item(self, data, src, target, pedigree, contentsource):
96+ def _insert_item(self, data, src, target, pedigree, contentsource):
97 """
98 Upload image into glance and add image metadata to simplestreams index.
99
100@@ -470,6 +473,55 @@ class GlanceMirror(mirrors.BasicMirrorWr
101 # unused in insert_products below.
102 self.insert_products(None, target, None)
103
104+ def insert_item(self, data, src, target, pedigree, contentsource):
105+ """Queue item to be inserted in subsequent call to insert_version
106+
107+ This adds the item to self.inserts which is then handled in
108+ insert_version. That allows the code to have context on
109+ all the items for a given version, and "choose" one. Ie,
110+ if both root.tar.xz and squashfs are available, preference
111+ can be given to the root.tar.gz.
112+ """
113+
114+ product_name, version_name, item_name = pedigree
115+ if product_name not in self.inserts:
116+ self.inserts[product_name] = {}
117+ if version_name not in self.inserts[product_name]:
118+ self.inserts[product_name][version_name] = {}
119+
120+ if 'ftype' in data:
121+ ftype = data['ftype']
122+ else:
123+ flat = util.products_exdata(src, pedigree, include_top=False)
124+ ftype = flat.get('ftype')
125+ self.inserts[product_name][version_name][item_name] = (
126+ ftype, (data, src, target, pedigree, contentsource))
127+
128+ def insert_version(self, data, src, target, pedigree):
129+ """Upload all images for this version into glance
130+ and add image metadata to simplestreams index.
131+
132+ All the work actually happens in _insert_item.
133+ """
134+
135+ product_name, version_name = pedigree
136+ inserts = self.inserts.get(product_name, {}).get(version_name, [])
137+
138+ rtar_names = [f for f in inserts
139+ if inserts[f][0] in ('root.tar.gz', 'root.tar.xz')]
140+
141+ for _iname, (ftype, iargs) in inserts.items():
142+ if ftype == "squashfs" and rtar_names:
143+ LOG.info("[%s] Skipping ftype 'squashfs' image in preference"
144+ "for root tarball type in %s",
145+ '/'.join(pedigree), rtar_names)
146+ continue
147+ self._insert_item(*iargs)
148+
149+ # we do not specifically do anything for insert_version, but
150+ # call parent.
151+ super(GlanceMirror, self).insert_version(data, src, target, pedigree)
152+
153 def remove_item(self, data, src, target, pedigree):
154 util.products_del(target, pedigree)
155 if 'id' in data:
156--- a/tests/unittests/test_glancemirror.py
157+++ b/tests/unittests/test_glancemirror.py
158@@ -333,6 +333,15 @@ class TestGlanceMirror(TestCase):
159
160 self.assertEqual("root-tar", create_arguments["disk_format"])
161
162+ def test_prepare_glance_arguments_disk_format_squashfs(self):
163+ # squashfs images are acceptable for nova-lxd
164+ source_entry = {"ftype": "squashfs"}
165+ create_arguments = self.mirror.prepare_glance_arguments(
166+ "foobuntu-X", source_entry, image_md5_hash=None, image_size=None,
167+ image_properties=None)
168+
169+ self.assertEqual("squashfs", create_arguments["disk_format"])
170+
171 def test_prepare_glance_arguments_size(self):
172 # Size is read from image metadata if defined.
173 source_entry = {"size": 5}
174@@ -476,7 +485,8 @@ class TestGlanceMirror(TestCase):
175 pedigree = (
176 u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
177 product = source_index[u'products'][pedigree[0]]
178- image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
179+ ver_data = product[u'versions'][pedigree[1]]
180+ image_data = ver_data[u'items'][pedigree[2]]
181
182 content_source = MemoryContentSource(
183 url="http://image-store/fooubuntu-X-disk1.img",
184@@ -495,6 +505,8 @@ class TestGlanceMirror(TestCase):
185
186 self.mirror.insert_item(
187 image_data, source_index, target, pedigree, content_source)
188+ self.mirror.insert_version(
189+ ver_data, source_index, target, pedigree[0:2])
190
191 passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
192
193@@ -538,7 +550,8 @@ class TestGlanceMirror(TestCase):
194 pedigree = (
195 u'com.ubuntu.cloud:server:14.04:amd64', u'20160602', u'disk1.img')
196 product = source_index[u'products'][pedigree[0]]
197- image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
198+ ver_data = product[u'versions'][pedigree[1]]
199+ image_data = ver_data[u'items'][pedigree[2]]
200
201 content_source = MemoryContentSource(
202 url="http://image-store/fooubuntu-X-disk1.img",
203@@ -557,6 +570,8 @@ class TestGlanceMirror(TestCase):
204
205 self.mirror.insert_item(
206 image_data, source_index, target, pedigree, content_source)
207+ self.mirror.insert_version(
208+ image_data, source_index, target, pedigree[0:2])
209
210 passed_create_kwargs = self.mirror.gclient.images.create_calls[0]
211
212@@ -609,7 +624,8 @@ class TestGlanceMirror(TestCase):
213 source_index = copy.deepcopy(TEST_SOURCE_INDEX_ENTRY)
214 pedigree = TEST_IMAGE_PEDIGREE
215 product = source_index[u'products'][pedigree[0]]
216- image_data = product[u'versions'][pedigree[1]][u'items'][pedigree[2]]
217+ ver_data = product[u'versions'][pedigree[1]]
218+ image_data = ver_data[u'items'][pedigree[2]]
219
220 content_source = MemoryContentSource(
221 url="http://image-store/fooubuntu-X-disk1.img",
222@@ -627,6 +643,8 @@ class TestGlanceMirror(TestCase):
223
224 self.mirror.insert_item(
225 image_data, source_index, target, pedigree, content_source)
226+ self.mirror.insert_version(
227+ ver_data, source_index, target, pedigree[0:2])
228
229 stored_index_content = self.mirror.store.data[
230 'streams/v1/auto.sync.json']
diff --git a/debian/patches/series b/debian/patches/series
0new file mode 100644231new file mode 100644
index 0000000..ee81491
--- /dev/null
+++ b/debian/patches/series
@@ -0,0 +1,4 @@
1keystone-v3-1719879.patch
2keystone-v3-1728982.patch
3nova-lxd-support-squashfs-images.patch
4460-glance-handle-v2-auth-with-sessions.patch

Subscribers

People subscribed via source and target branches