Merge lp:~danilo/charms/trusty/glance-simplestreams-sync/lathiats-endpoints-fix into lp:~landscape/charms/trusty/glance-simplestreams-sync/landscape

[Данило Шеган]

This is a resubmit of https://code.launchpad.net/~lathiat/charms/trusty/glance-simplestreams-sync/trunk/+merge/290950 against our branch with a few changes on top https://pastebin.canonical.com/154167/ (pep8 cleanups and a fix for an issue I've seen with ClientException).

This *has* been tested in ceph/ceph and swift/iscsi configurations (this requires only 4 nodes, whereas swift/ceph requires 6) and works properly and allows juju to bootstrap with the images.

This also supersedes my original proposal which only worked for ceph/ceph at https://code.launchpad.net/~danilo/charms/trusty/glance-simplestreams-sync/normalize-swift-endpoint/+merge/291612

This has been pushed to cs:~danilo/trusty/glance-simplestreams-sync-17 (see the old MP for testing instructions)

This in-turn is best tested with https://code.launchpad.net/~danilo/landscape/swift-endpoints-fix/+merge/291870 (otherwise, the code that branch removes might override the good value this sets: if that happens, re-run the /etc/cron.daily/glance-simplestreams-sync script on the glance-simplestreams-sync/0 unit).

[lp:~danilo/charms/trusty/glance-simplestreams-sync/lathiats-endpoints-fix]

66. By Данило Шеган on 2016-04-14

Merge "landscape" branch.

[Geoff Teale (tealeg)]

+1 Awesome.

review: Approve

[Adam Collard (adam-collard)]

Not in this diff but the warning log about finding the wrong number of endpoints is bogus.

    if len(ps_endpoints) != 1:
        log.warning("found {} product-streams endpoints in region {},"
                    " expecting one - not updating"
                    " endpoint".format(ps_endpoints, region,
                                       len(ps_endpoints)))

It has two placeholders and three values to format in there.

It's crying out for tests.

review: Needs Fixing

[Данило Шеган (danilo)]

Yes, it is crying out for tests, no disagreement there. Fixed the warning message about wrong number of endpoints, addressed your other comment inline.

[lp:~danilo/charms/trusty/glance-simplestreams-sync/lathiats-endpoints-fix]

67. By Данило Шеган on 2016-04-15

Address review comments regarding warning log messages.

[Adam Collard (adam-collard)]

Minor nit, let's test it properly with the corresponding branch for Landscape. +1

review: Approve

[lp:~danilo/charms/trusty/glance-simplestreams-sync/lathiats-endpoints-fix]

68. By Данило Шеган on 2016-04-15

Improve exceptions handling.

[Данило Шеган (danilo)]

Thanks, fixed the other exception cases as well to include a message, will clean this up in a follow-up branch.

=== modified file 'scripts/glance-simplestreams-sync.py'
--- scripts/glance-simplestreams-sync.py	2016-04-14 06:50:42 +0000
+++ scripts/glance-simplestreams-sync.py	2016-04-15 09:13:31 +0000
@@ -55,6 +55,7 @@
 from simplestreams.mirrors import glance, UrlMirrorReader
 from simplestreams.objectstores.swift import SwiftObjectStore
 from simplestreams.util import read_signed, path_from_mirror_url
+import swiftclient
 import sys
 import time
 import traceback
@@ -333,28 +334,21 @@
     Updates URLs of product-streams endpoint to point to swift URLs.
     """
 
-    swift_services = [s for s in services
-                      if s['name'] == 'swift']
-    if len(swift_services) != 1:
-        log.error("found {} swift services. expecting one."
-                  " - not updating endpoint.".format(len(swift_services)))
-        return
+    try:
+        catalog = {
+            endpoint_type: ksc.service_catalog.url_for(
+                service_type='object-store', endpoint_type=endpoint_type)
+            for endpoint_type in ['publicURL', 'internalURL', 'adminURL']}
+    except keystone_exceptions.EndpointNotFound as e:
+        log.warning("could not retrieve swift endpoint, not updating "
+                    "product-streams endpoint: {}".format(e))
+        raise
 
-    swift_service_id = swift_services[0]['id']
+    for endpoint_type in ['publicURL', 'internalURL']:
+        catalog[endpoint_type] += "/{}".format(SWIFT_DATA_DIR)
 
     endpoints = [e._info for e in ksc.endpoints.list()
                  if e._info['region'] == region]
-
-    swift_endpoints = [e for e in endpoints
-                       if e['service_id'] == swift_service_id]
-    if len(swift_endpoints) != 1:
-        log.warning("found {} swift endpoints, expecting one - not"
-                    " updating product-streams"
-                    " endpoint.".format(len(swift_endpoints)))
-        return
-
-    swift_endpoint = swift_endpoints[0]
-
     ps_services = [s for s in services
                    if s['name'] == PRODUCT_STREAMS_SERVICE_NAME]
     if len(ps_services) != 1:
@@ -370,45 +364,23 @@
     if len(ps_endpoints) != 1:
         log.warning("found {} product-streams endpoints in region {},"
                     " expecting one - not updating"
-                    " endpoint".format(ps_endpoints, region,
-                                       len(ps_endpoints)))
+                    " endpoint".format(len(ps_endpoints), region))
         return
 
     log.info("Deleting existing product-streams endpoint: ")
     ksc.endpoints.delete(ps_endpoints[0]['id'])
 
-    services_tenant_ids = [t.id for t in ksc.tenants.list()
-                           if t.name == 'services']
-
-    if len(services_tenant_ids) != 1:
-        log.warning("found {} tenants named 'services',"
-                    " expecting one. Not updating"
-                    " endpoint".format(len(services_tenant_ids)))
-
-    services_tenant_id = services_tenant_ids[0]
-
-    path = "/v1/AUTH_{}/{}".format(services_tenant_id,
-                                   SWIFT_DATA_DIR)
-
-    swift_public_url = swift_endpoint['publicurl']
-    sr_p = urlsplit(swift_public_url)
-    ps_public_url = sr_p._replace(path=path).geturl()
-
-    swift_internal_url = swift_endpoint['internalurl']
-    sr_i = urlsplit(swift_internal_url)
-    ps_internal_url = sr_i._replace(path=path).geturl()
-
     create_args = dict(region=region,
                        service_id=ps_service_id,
-                       publicurl=ps_public_url,
-                       adminurl=swift_endpoint['adminurl'],
-                       internalurl=ps_internal_url)
+                       publicurl=catalog['publicURL'],
+                       adminurl=catalog['adminURL'],
+                       internalurl=catalog['internalURL'])
     status_set('maintenance', 'Updating product-streams endpoint')
     log.info("creating product-streams endpoint: {}".format(create_args))
     ksc.endpoints.create(**create_args)
-    update_endpoint_urls(region, ps_public_url,
-                         swift_endpoint['adminurl'],
-                         ps_internal_url)
+    update_endpoint_urls(region, catalog['publicURL'],
+                         catalog['adminURL'],
+                         catalog['internalURL'])
 
 
 def juju_run_cmd(cmd):
@@ -570,6 +542,7 @@
                                            charm_conf['region'])
         except:
             log.exception("Exception during update_product_streams_service")
+            should_delete_cron_poll = False
 
     else:
         log.info("Not updating product streams service.")
@@ -597,10 +570,13 @@
             log.info("Glance endpoint not found, will continue polling.")
 
     except glanceclient.exc.ClientException as e:
-        log.exception("Glance Client exception during do_sync."
-                      " will continue polling.")
-        should_delete_cron_poll = False
-
+        log.exception("Glance Client exception during do_sync:\n%s\n"
+                      "Will continue polling." % e.message)
+        should_delete_cron_poll = False
+    except swiftclient.ClientException as e:
+        log.exception("Swift Client exception during do_sync:\n%s\n"
+                      "Will continue polling." % e.message)
+        should_delete_cron_poll = False
     except requests.exceptions.RequestException as e:
         log.exception("Requests exception during do_sync:\n%s\n"
                       "Will continue polling." % e.message)