Merge lp:~lathiat/charms/trusty/glance-simplestreams-sync/trunk into lp:charms/trusty/glance-simplestreams-sync

[Trent Lloyd]

Base the product-streams endpoint on swift's endpoint using service_catalog.url_for() instead of generating it manually (LP #1552986)

ceph-radosgw uses a different URL scheme to swift-proxy without the AUTH_$(tenant_id)s part,
this fix will now base the URL on the configured endpoint so that it will work with either
object store and service_catalog.url_for() will replace the dynamic $(tenant_id) part for us

Once working, juju can now find images and tools in a private cloud automatically instead of needing to generate metadata manually.

[Данило Шеган (danilo)]

In my tests, this has worked with ceph/ceph deployment of Landscape Autopilot, but failed when swift/iscsi and swift/ceph were used due to Authorization errors.

A glance-simplestreams-sync.log is at http://paste.ubuntu.com/15806629/.

Btw, please respect the PEP-8 as well (80-character line length limit).

review: Needs Fixing

[Данило Шеган (danilo)]

To expand on the above, the problem seems to be that the keystone catalog has an "admin" tenant ID in the object_store URL (/v1/AUTH_<admin-tenant-id>), whereas the old code had the "services" tenant ID.

However, if I use the more recent API ("openstack endpoint list") and then "openstack endpoint show <object-store-service-id>", I get:

+--------------+-------------------------------------------------+
| Field | Value |
+--------------+-------------------------------------------------+
| adminurl | http://10.44.199.170:8080 |
| enabled | True |
| id | 69ff557520f24f27bf8d04761b5bf350 |
| internalurl | http://10.44.199.170:8080/v1/AUTH_$(tenant_id)s |
| publicurl | http://10.44.199.170:8080/v1/AUTH_$(tenant_id)s |
| region | region1 |
| service_id | e0d9101e28f24bb09cc101e8575c2df7 |
| service_name | swift |
| service_type | object-store |
+--------------+-------------------------------------------------+

What we need to do is to get those internalurl and publicurl values and replace $(tenant_id)s with "services" tenant ID in there. I am looking at how I get at this right now.

[Trent Lloyd (lathiat)]

According to your log that was the services tenant_id:

"tenant": {"description": "Created by Juju", "enabled": true, "id": "a953d9d28c0d4e9cb014c50d425b7671", "name": "services"}

INFO * 04-13 08:56:17 [PID:9560] * swiftclient * REQ: curl -i http://10.44.199.160:8080/v1/AUTH_a953d9d28c0d4e9cb014c50d425b7671/simplestreams -X PUT -H "Content-Length: 0" -H "X-Auth-Token: b2b261223cc349328a01403b27a2b3c4" -H "X-Container-Read: .r:*,.rlistings"

INFO * 04-13 08:56:17 [PID:9560] * swiftclient * RESP STATUS: 401 Unauthorized

The auth token in use was also for services. Suspect something else is at play. Will test.

[Данило Шеган (danilo)]

Ok, I was wrong above (too many things going on at once). This branch does work, but might hit a bit of a race with swift setup and thus hit the ClientException as above.

I've worked around that using https://pastebin.canonical.com/154167/ (mostly whitespace fixes, but also adding exception handling to keep the per-minute cronjob if swiftclient.ClientException is caught). In my last run, I haven't hit it, but as I said, it will sometimes happen with automated installs and sometimes not.

review: Approve

=== modified file 'scripts/glance-simplestreams-sync.py'
--- scripts/glance-simplestreams-sync.py	2015-11-05 11:28:15 +0000
+++ scripts/glance-simplestreams-sync.py	2016-04-05 06:18:42 +0000
@@ -216,28 +216,16 @@
     Updates URLs of product-streams endpoint to point to swift URLs.
     """
 
-    swift_services = [s for s in services
-                      if s['name'] == 'swift']
-    if len(swift_services) != 1:
-        log.error("found {} swift services. expecting one."
-                  " - not updating endpoint.".format(len(swift_services)))
-        return
-
-    swift_service_id = swift_services[0]['id']
+    try:
+      catalog = {endpoint_type: ksc.service_catalog.url_for(service_type='object-store', endpoint_type=endpoint_type) for endpoint_type in ['publicURL', 'internalURL', 'adminURL']}
+    except keystoneclient.exceptions.EndpointNotFound as e:
+      log.warning("could not retrieve swift endpoint, not updating product-streams endpoint: {}".format(e))
+    
+    for endpoint_type in ['publicURL', 'internalURL']:
+      catalog[endpoint_type] += "/{}".format(SWIFT_DATA_DIR)
 
     endpoints = [e._info for e in ksc.endpoints.list()
                  if e._info['region'] == region]
-
-    swift_endpoints = [e for e in endpoints
-                       if e['service_id'] == swift_service_id]
-    if len(swift_endpoints) != 1:
-        log.warning("found {} swift endpoints, expecting one - not"
-                    " updating product-streams"
-                    " endpoint.".format(len(swift_endpoints)))
-        return
-
-    swift_endpoint = swift_endpoints[0]
-
     ps_services = [s for s in services
                    if s['name'] == PRODUCT_STREAMS_SERVICE_NAME]
     if len(ps_services) != 1:
@@ -260,38 +248,17 @@
     log.info("Deleting existing product-streams endpoint: ")
     ksc.endpoints.delete(ps_endpoints[0]['id'])
 
-    services_tenant_ids = [t.id for t in ksc.tenants.list()
-                           if t.name == 'services']
-
-    if len(services_tenant_ids) != 1:
-        log.warning("found {} tenants named 'services',"
-                    " expecting one. Not updating"
-                    " endpoint".format(len(services_tenant_ids)))
-
-    services_tenant_id = services_tenant_ids[0]
-
-    path = "/v1/AUTH_{}/{}".format(services_tenant_id,
-                                   SWIFT_DATA_DIR)
-
-    swift_public_url = swift_endpoint['publicurl']
-    sr_p = urlsplit(swift_public_url)
-    ps_public_url = sr_p._replace(path=path).geturl()
-
-    swift_internal_url = swift_endpoint['internalurl']
-    sr_i = urlsplit(swift_internal_url)
-    ps_internal_url = sr_i._replace(path=path).geturl()
-
     create_args = dict(region=region,
                        service_id=ps_service_id,
-                       publicurl=ps_public_url,
-                       adminurl=swift_endpoint['adminurl'],
-                       internalurl=ps_internal_url)
+                       publicurl=catalog['publicURL'],
+                       adminurl=catalog['adminURL'],
+                       internalurl=catalog['internalURL'])
     status_set('maintenance', 'Updating product-streams endpoint')
     log.info("creating product-streams endpoint: {}".format(create_args))
     ksc.endpoints.create(**create_args)
-    update_endpoint_urls(region, ps_public_url,
-                         swift_endpoint['adminurl'],
-                         ps_internal_url)
+    update_endpoint_urls(region, catalog['publicURL'],
+                         catalog['adminURL'],
+                         catalog['internalURL'])
 
 
 def juju_run_cmd(cmd):