Merge lp:~yolanda.robla/charms/precise/keystone/ha-support into lp:~charmers/charms/precise/keystone/trunk

[Yolanda Robla]

Added Ceilometer service

[lp:~yolanda.robla/charms/precise/keystone/ha-support]

49. By Adam Gandelman on 2013-02-01

Merge 'Added Ceilometer service'

50. By James Page on 2013-02-07

Adds better support for service leaders.

* The service leader is determined depending on how keystone is currently clustered. If there are multiple units, but no hacluster subordinate, the oldest service unit is elected leader (lowest unit number). If hacluster exists and the service is clustered, the CRM is consulted and the node hosting the resources is designated the leader.

* Only the leader may initialize or touch the database (create users, endpoints, etc)

* The leader is responsible for synchronizing a list of service credentials to all peers. The list is stored on disk and resolves the issue of the passwd dump files in /var/lib/keystone/ being out-of-sync among peers.

We can use the same approach in the rabbitmq-server charm if it works out here.

51. By James Page on 2013-02-14

Pass keystone information to services which just need to access keystone

52. By James Page on 2013-02-15

openstack-common: increase server/client timeout to 10seconds

53. By James Page on 2013-02-18

Use db_host, not private address to ensure VIP is used in HA scenarios

54. By Adam Gandelman on 2013-03-05

Sync openstack charm helpers.

55. By Adam Gandelman on 2013-03-06

Merge health_checks.d framework.

56. By James Page on 2013-03-08

Fixup keystone epoch version handling

57. By James Page on 2013-03-11

SSH based hook syncing

58. By Adam Gandelman on 2013-03-19

Big refactor and cleanup from James.

59. By Adam Gandelman on 2013-03-21

Sync openstack_common.py

60. By Adam Gandelman on 2013-03-21

Sync utils.py

61. By James Page on 2013-03-28

keystone health updates to use new cluster.determine_api_port method

62. By Adam Gandelman on 2013-04-04

Merge SSH_USER fix for non-clustered nodes.

63. By Yolanda Robla on 2013-04-08

grant requested role to user

64. By Yolanda Robla on 2013-04-08

upgrade revision

65. By Yolanda Robla on 2013-04-08

fixes in requested_roles

66. By Yolanda Robla on 2013-04-08

changes in requested roles

67. By Yolanda Robla on 2013-04-08

typo fix

68. By Yolanda Robla on 2013-04-08

fixes in requested roles

69. By Yolanda Robla on 2013-04-09

fix in create_role

70. By Yolanda Robla on 2013-04-09

fixes in grant requested roles

Unmerged revisions

=== modified file 'config.yaml'
--- config.yaml	2012-10-12 17:26:48 +0000
+++ config.yaml	2013-01-31 11:59:26 +0000
@@ -26,6 +26,10 @@
     default: "/etc/keystone/keystone.conf"
     type: string
     description: "Location of keystone configuration file"
+  log-level:
+    default: WARNING
+    type: string
+    description: Log level (WARNING, INFO, DEBUG, ERROR)
   service-port:
     default: 5000
     type: int
@@ -75,3 +79,36 @@
     default: "keystone"
     type: string
     description: "Database username"
+  region:
+    default: RegionOne
+    type: string
+    description: "OpenStack Region(s) - separate multiple regions with single space"
+  # HA configuration settings
+  vip:
+    type: string
+    description: "Virtual IP to use to front keystone in ha configuration"
+  vip_iface:
+    type: string
+    default: eth0
+    description: "Network Interface where to place the Virtual IP"
+  vip_cidr:
+    type: int
+    default: 24
+    description: "Netmask that will be used for the Virtual IP"
+  ha-bindiface:
+    type: string
+    default: eth0
+    description: |
+      Default network interface on which HA cluster will bind to communication
+      with the other members of the HA Cluster.
+  ha-mcastport:
+    type: int
+    default: 5403
+    description: |
+      Default multicast port number that will be used to communicate between
+      HA Cluster nodes.
+  # PKI enablement and configuration (Grizzly and beyond)
+  enable-pki:
+    default: "false"
+    type: string
+    description: "Enable PKI token signing (Grizzly and beyond)"

=== added symlink 'hooks/cluster-relation-changed'
=== target is u'keystone-hooks'
=== added symlink 'hooks/cluster-relation-departed'
=== target is u'keystone-hooks'
=== added symlink 'hooks/ha-relation-changed'
=== target is u'keystone-hooks'
=== added symlink 'hooks/ha-relation-joined'
=== target is u'keystone-hooks'
=== modified file 'hooks/keystone-hooks'
--- hooks/keystone-hooks	2012-12-12 03:52:01 +0000
+++ hooks/keystone-hooks	2013-01-31 11:59:26 +0000
@@ -8,7 +8,7 @@
 
 config = config_get()
 
-packages = "keystone python-mysqldb pwgen"
+packages = "keystone python-mysqldb pwgen haproxy python-jinja2"
 service = "keystone"
 
 # used to verify joined services are valid openstack components.
@@ -46,6 +46,14 @@
     "quantum": {
         "type": "network",
         "desc": "Quantum Networking Service"
+    },
+    "oxygen": {
+        "type": "oxygen",
+        "desc": "Oxygen Cloud Image Service"
+    },
+    "ceilometer": {
+        "type": "metering",
+        "desc": "Ceilometer Metering Service"
     }
 }
 
@@ -124,18 +132,30 @@
             realtion_set({ "admin_token": -1 })
             return
 
-    def add_endpoint(region, service, public_url, admin_url, internal_url):
+    def add_endpoint(region, service, publicurl, adminurl, internalurl):
         desc = valid_services[service]["desc"]
         service_type = valid_services[service]["type"]
         create_service_entry(service, service_type, desc)
         create_endpoint_template(region=region, service=service,
-                                 public_url=public_url,
-                                 admin_url=admin_url,
-                                 internal_url=internal_url)
+                                 publicurl=publicurl,
+                                 adminurl=adminurl,
+                                 internalurl=internalurl)
+
+    if is_clustered() and not is_leader():
+        # Only respond if service unit is the leader
+        return
 
     settings = relation_get_dict(relation_id=relation_id,
                                  remote_unit=remote_unit)
 
+    # Allow the remote service to request creation of any additional roles.
+    # Currently used by Swift.
+    if 'requested_roles' in settings and settings['requested_roles'] != 'None':
+        roles = settings['requested_roles'].split(',')
+        juju_log("Creating requested roles: %s" % roles)
+        for role in roles:
+            create_role(role, user=config['admin-user'], tenant='admin')
+
     # the minimum settings needed per endpoint
     single = set(['service', 'region', 'public_url', 'admin_url',
                   'internal_url'])
@@ -149,9 +169,9 @@
 
         ensure_valid_service(settings['service'])
         add_endpoint(region=settings['region'], service=settings['service'],
-                     public_url=settings['public_url'],
-                     admin_url=settings['admin_url'],
-                     internal_url=settings['internal_url'])
+                     publicurl=settings['public_url'],
+                     adminurl=settings['admin_url'],
+                     internalurl=settings['internal_url'])
         service_username = settings['service']
     else:
         # assemble multiple endpoints from relation data. service name
@@ -186,9 +206,9 @@
                 ep = endpoints[ep]
                 ensure_valid_service(ep['service'])
                 add_endpoint(region=ep['region'], service=ep['service'],
-                             public_url=ep['public_url'],
-                             admin_url=ep['admin_url'],
-                             internal_url=ep['internal_url'])
+                             publicurl=ep['public_url'],
+                             adminurl=ep['admin_url'],
+                             internalurl=ep['internal_url'])
                 services.append(ep['service'])
         service_username = '_'.join(services)
 
@@ -201,6 +221,7 @@
     token = get_admin_token()
     juju_log("Creating service credentials for '%s'" % service_username)
 
+    # TODO: This needs to be changed as it won't work with ha keystone
     stored_passwd = '/var/lib/keystone/%s.passwd' % service_username
     if os.path.isfile(stored_passwd):
         juju_log("Loading stored service passwd from %s" % stored_passwd)
@@ -213,14 +234,6 @@
     create_user(service_username, service_password, config['service-tenant'])
     grant_role(service_username, config['admin-role'], config['service-tenant'])
 
-    # Allow the remote service to request creation of any additional roles.
-    # Currently used by Swift.
-    if 'requested_roles' in settings:
-        roles = settings['requested_roles'].split(',')
-        juju_log("Creating requested roles: %s" % roles)
-        for role in roles:
-            create_role(role, user=config['admin-user'], tenant='admin')
-
     # As of https://review.openstack.org/#change,4675, all nodes hosting
     # an endpoint(s) needs a service username and password assigned to
     # the service tenant and granted admin role.
@@ -237,6 +250,13 @@
         "service_password": service_password,
         "service_tenant": config['service-tenant']
     }
+    # Check if clustered and use vip + haproxy ports if so
+    if is_clustered():
+        relation_data["auth_host"] = config['vip']
+        relation_data["auth_port"] = SERVICE_PORTS['keystone_admin']
+        relation_data["service_host"] = config['vip']
+        relation_data["service_port"] = SERVICE_PORTS['keystone_service']
+
     relation_set(relation_data)
 
 def config_changed():
@@ -246,11 +266,114 @@
     available = get_os_codename_install_source(config['openstack-origin'])
     installed = get_os_codename_package('keystone')
 
-    if get_os_version_codename(available) > get_os_version_codename(installed):
+    if (available and
+        get_os_version_codename(available) > get_os_version_codename(installed)):
         do_openstack_upgrade(config['openstack-origin'], packages)
 
     set_admin_token(config['admin-token'])
-    ensure_initial_admin(config)
+
+    if is_clustered() and is_leader():
+        juju_log('Cluster leader - ensuring endpoint configuration is up to date')
+        ensure_initial_admin(config)
+    elif not is_clustered():
+        ensure_initial_admin(config)
+
+    update_config_block('logger_root', level=config['log-level'],
+                        file='/etc/keystone/logging.conf')
+    if get_os_version_package('keystone') >= '2013.1':
+        # PKI introduced in Grizzly
+        configure_pki_tokens(config)
+
+    execute("service keystone restart", echo=True)
+    cluster_changed()
+
+
+def upgrade_charm():
+    cluster_changed()
+    if is_clustered() and is_leader():
+        juju_log('Cluster leader - ensuring endpoint configuration is up to date')
+        ensure_initial_admin(config)
+    elif not is_clustered():
+        ensure_initial_admin(config)
+
+
+SERVICE_PORTS = {
+    "keystone_admin": int(config['admin-port']) + 1,
+    "keystone_service": int(config['service-port']) + 1
+    }
+
+
+def cluster_changed():
+    cluster_hosts = {}
+    cluster_hosts['self'] = config['hostname']
+    for r_id in relation_ids('cluster'):
+        for unit in relation_list(r_id):
+            cluster_hosts[unit.replace('/','-')] = \
+                relation_get_dict(relation_id=r_id,
+                                  remote_unit=unit)['private-address']
+    configure_haproxy(cluster_hosts,
+                      SERVICE_PORTS)
+
+
+def ha_relation_changed():
+    relation_data = relation_get_dict()
+    if ('clustered' in relation_data and
+        is_leader()):
+        juju_log('Cluster configured, notifying other services and updating'
+                 'keystone endpoint configuration')
+        # Update keystone endpoint to point at VIP
+        ensure_initial_admin(config)
+        # Tell all related services to start using
+        # the VIP and haproxy ports instead
+        for r_id in relation_ids('identity-service'):
+            relation_set_2(rid=r_id,
+                           auth_host=config['vip'],
+                           service_host=config['vip'],
+                           service_port=SERVICE_PORTS['keystone_service'],
+                           auth_port=SERVICE_PORTS['keystone_admin'])
+
+
+def ha_relation_joined():
+    # Obtain the config values necessary for the cluster config. These
+    # include multicast port and interface to bind to.
+    corosync_bindiface = config['ha-bindiface']
+    corosync_mcastport = config['ha-mcastport']
+
+    # Obtain resources
+    resources = {
+            'res_ks_vip':'ocf:heartbeat:IPaddr2',
+            'res_ks_haproxy':'lsb:haproxy'
+        }
+    # TODO: Obtain netmask and nic where to place VIP.
+    resource_params = {
+            'res_ks_vip':'params ip="%s" cidr_netmask="%s" nic="%s"' % (config['vip'],
+                              config['vip_cidr'], config['vip_iface']),
+            'res_ks_haproxy':'op monitor interval="5s"'
+        }
+    init_services = {
+            'res_ks_haproxy':'haproxy'
+        }
+    groups = {
+            'grp_ks_haproxy':'res_ks_vip res_ks_haproxy'
+        }
+    #clones = {
+    #        'cln_ks_haproxy':'res_ks_haproxy meta globally-unique="false" interleave="true"'
+    #    }
+
+    #orders = {
+    #        'ord_vip_before_haproxy':'inf: res_ks_vip res_ks_haproxy'
+    #    }
+    #colocations = {
+    #        'col_vip_on_haproxy':'inf: res_ks_haproxy res_ks_vip'
+    #    }
+
+    relation_set_2(init_services=init_services,
+                   corosync_bindiface=corosync_bindiface,
+                   corosync_mcastport=corosync_mcastport,
+                   resources=resources,
+                   resource_params=resource_params,
+                   groups=groups)
+
 
 hooks = {
     "install": install_hook,
@@ -258,7 +381,12 @@
     "shared-db-relation-changed": db_changed,
     "identity-service-relation-joined": identity_joined,
     "identity-service-relation-changed": identity_changed,
-    "config-changed": config_changed
+    "config-changed": config_changed,
+    "cluster-relation-changed": cluster_changed,
+    "cluster-relation-departed": cluster_changed,
+    "ha-relation-joined": ha_relation_joined,
+    "ha-relation-changed": ha_relation_changed,
+    "upgrade-charm": upgrade_charm
 }
 
 # keystone-hooks gets called by symlink corresponding to the requested relation

=== modified file 'hooks/lib/openstack_common.py'
--- hooks/lib/openstack_common.py	2012-12-05 20:35:05 +0000
+++ hooks/lib/openstack_common.py	2013-01-31 11:59:26 +0000
@@ -3,6 +3,7 @@
 # Common python helper functions used for OpenStack charms.
 
 import subprocess
+import os
 
 CLOUD_ARCHIVE_URL = "http://ubuntu-cloud.archive.canonical.com/ubuntu"
 CLOUD_ARCHIVE_KEY_ID = '5EDB1B62EC4926EA'
@@ -22,6 +23,12 @@
     '2013.1': 'grizzly'
 }
 
+# The ugly duckling
+swift_codenames = {
+    '1.4.3': 'diablo',
+    '1.4.8': 'essex',
+    '1.7.4': 'folsom'
+}
 
 def juju_log(msg):
     subprocess.check_call(['juju-log', msg])
@@ -118,12 +125,32 @@
 
     vers = vers[:6]
     try:
-        return openstack_codenames[vers]
+        if 'swift' in pkg:
+            vers = vers[:5]
+            return swift_codenames[vers]
+        else:
+            vers = vers[:6]
+            return openstack_codenames[vers]
     except KeyError:
         e = 'Could not determine OpenStack codename for version %s' % vers
         error_out(e)
 
 
+def get_os_version_package(pkg):
+    '''Derive OpenStack version number from an installed package.'''
+    codename = get_os_codename_package(pkg)
+
+    if 'swift' in pkg:
+        vers_map = swift_codenames
+    else:
+        vers_map = openstack_codenames
+
+    for version, cname in vers_map.iteritems():
+        if cname == codename:
+            return version
+    e = "Could not determine OpenStack version for package: %s" % pkg
+    error_out(e)
+
 def configure_installation_source(rel):
     '''Configure apt installation source.'''
 
@@ -164,9 +191,11 @@
                 'version (%s)' % (ca_rel, ubuntu_rel)
             error_out(e)
 
-        if ca_rel == 'folsom/staging':
+        if 'staging' in ca_rel:
             # staging is just a regular PPA.
-            cmd = 'add-apt-repository -y ppa:ubuntu-cloud-archive/folsom-staging'
+            os_rel = ca_rel.split('/')[0]
+            ppa = 'ppa:ubuntu-cloud-archive/%s-staging' % os_rel
+            cmd = 'add-apt-repository -y %s' % ppa
             subprocess.check_call(cmd.split(' '))
             return
 
@@ -174,7 +203,10 @@
         pockets = {
             'folsom': 'precise-updates/folsom',
             'folsom/updates': 'precise-updates/folsom',
-            'folsom/proposed': 'precise-proposed/folsom'
+            'folsom/proposed': 'precise-proposed/folsom',
+            'grizzly': 'precise-updates/grizzly',
+            'grizzly/updates': 'precise-updates/grizzly',
+            'grizzly/proposed': 'precise-proposed/grizzly'
         }
 
         try:
@@ -191,3 +223,23 @@
     else:
         error_out("Invalid openstack-release specified: %s" % rel)
 
+HAPROXY_CONF = '/etc/haproxy/haproxy.cfg'
+HAPROXY_DEFAULT = '/etc/default/haproxy'
+
+def configure_haproxy(units, service_ports, template_dir=None):
+    template_dir = template_dir or 'templates'
+    import jinja2
+    context = {
+        'units': units,
+        'service_ports': service_ports
+        }
+    templates = jinja2.Environment(
+                    loader=jinja2.FileSystemLoader(template_dir)
+                    )
+    template = templates.get_template(
+                    os.path.basename(HAPROXY_CONF)
+                    )
+    with open(HAPROXY_CONF, 'w') as f:
+        f.write(template.render(context))
+    with open(HAPROXY_DEFAULT, 'w') as f:
+        f.write('ENABLED=1')

=== added symlink 'hooks/upgrade-charm'
=== target is u'keystone-hooks'
=== modified file 'hooks/utils.py'
--- hooks/utils.py	2012-12-12 03:52:41 +0000
+++ hooks/utils.py	2013-01-31 11:59:26 +0000
@@ -1,4 +1,5 @@
 #!/usr/bin/python
+import ConfigParser
 import subprocess
 import sys
 import json
@@ -11,6 +12,7 @@
 stored_passwd = "/var/lib/keystone/keystone.passwd"
 stored_token = "/var/lib/keystone/keystone.token"
 
+
 def execute(cmd, die=False, echo=False):
     """ Executes a command 
 
@@ -79,6 +81,20 @@
     for k in relation_data:
         execute("relation-set %s=%s" % (k, relation_data[k]), die=True)
 
+def relation_set_2(**kwargs):
+    cmd = [
+        'relation-set'
+        ]
+    args = []
+    for k, v in kwargs.items():
+        if k == 'rid':
+            cmd.append('-r')
+            cmd.append(v)
+        else:
+            args.append('{}={}'.format(k, v))
+    cmd += args
+    subprocess.check_call(cmd)
+
 def relation_get(relation_data):
     """ Obtain all current relation data
     relation_data is a list of options to query from the relation
@@ -149,106 +165,26 @@
                           keystone_conf)
     error_out('Could not find admin_token line in %s' % keystone_conf)
 
-def update_config_block(block, **kwargs):
+def update_config_block(section, **kwargs):
     """ Updates keystone.conf blocks given kwargs.
-    Can be used to update driver settings for a particular backend,
-    setting the sql connection, etc.
-
-    Parses block heading as '[block]'
-
-    If block does not exist, a new block will be created at end of file with
-    given kwargs
-    """
-    f = open(keystone_conf, "r+")
-    orig = f.readlines()
-    new = []
-    found_block = ""
-    heading = "[%s]\n" % block
-
-    lines = len(orig)
-    ln = 0
-
-    def update_block(block):
-        for k, v in kwargs.iteritems():
-            for l in block:
-                if l.strip().split(" ")[0] == k:
-                    block[block.index(l)] = "%s = %s\n" % (k, v)
-                    return
-            block.append('%s = %s\n' % (k, v))
-            block.append('\n')
-
-    try:
-        found = False
-        while ln < lines:
-            if orig[ln] != heading:
-                new.append(orig[ln])
-                ln += 1
-            else:
-                new.append(orig[ln])
-                ln += 1
-                block = []
-                while orig[ln].strip() != '':
-                    block.append(orig[ln])
-                    ln += 1
-                update_block(block)
-                new += block
-                found = True
-
-        if not found:
-            if new[(len(new) - 1)].strip() != '':
-                new.append('\n')
-            new.append('%s' % heading)
-            for k, v in kwargs.iteritems():
-                new.append('%s = %s\n' % (k, v))
-            new.append('\n')
-    except:
-        error_out('Error while attempting to update config block. '\
-                  'Refusing to overwite existing config.')
-
-        return
-
-    # backup original config
-    backup = open(keystone_conf + '.juju-back', 'w+')
-    for l in orig:
-        backup.write(l)
-    backup.close()
-
-    # update config
-    f.seek(0)
-    f.truncate()
-    for l in new:
-        f.write(l)
-
-
-def keystone_conf_update(opt, val):
-    """ Updates keystone.conf values 
-    If option exists, it is reset to new value
-    If it does not, it added to the top of the config file after the [DEFAULT]
-    heading to keep it out of the paste deploy config
-    """
-    f = open(keystone_conf, "r+")
-    orig = f.readlines()
-    new = ""
-    found = False
-    for l in orig:
-        if l.split(' ')[0] == opt:
-            juju_log("Updating %s, setting %s = %s" % (keystone_conf, opt, val))
-            new += "%s = %s\n" % (opt, val)
-            found  = True
-        else:
-            new += l
-    new = new.split('\n')
-    # insert a new value at the top of the file, after the 'DEFAULT' header so
-    # as not to muck up paste deploy configuration later in the file 
-    if not found:
-        juju_log("Adding new config option %s = %s" % (opt, val))
-        header = new.index("[DEFAULT]")
-        new.insert((header+1), "%s = %s" % (opt, val))
-    f.seek(0)
-    f.truncate()
-    for l in new:
-        f.write("%s\n" % l)
-    f.close
+    Update a config setting in a specific setting of a config
+    file (/etc/keystone/keystone.conf, by default)
+    """
+    if 'file' in kwargs:
+        conf_file = kwargs['file']
+        del kwargs['file']
+    else:
+        conf_file = keystone_conf
+    config = ConfigParser.RawConfigParser()
+    config.read(conf_file)
+
+    if section != 'DEFAULT' and not config.has_section(section):
+        config.add_section(section)
+
+    for k, v in kwargs.iteritems():
+        config.set(section, k, v)
+    with open(conf_file, 'wb') as out:
+        config.write(out)
 
 def create_service_entry(service_name, service_type, service_desc, owner=None):
     """ Add a new service entry to keystone if one does not already exist """
@@ -264,8 +200,8 @@
                                 description=service_desc)
     juju_log("Created new service entry '%s'" % service_name)
 
-def create_endpoint_template(region, service,  public_url, admin_url,
-                             internal_url):
+def create_endpoint_template(region, service,  publicurl, adminurl,
+                             internalurl):
     """ Create a new endpoint template for service if one does not already
         exist matching name *and* region """
     import manager
@@ -276,13 +212,24 @@
         if ep['service_id'] == service_id and ep['region'] == region:
             juju_log("Endpoint template already exists for '%s' in '%s'"
                       % (service, region))
-            return
+
+            up_to_date = True
+            for k in ['publicurl', 'adminurl', 'internalurl']:
+                if ep[k] != locals()[k]:
+                    up_to_date = False
+
+            if up_to_date:
+                return
+            else:
+                # delete endpoint and recreate if endpoint urls need updating.
+                juju_log("Updating endpoint template with new endpoint urls.")
+                manager.api.endpoints.delete(ep['id'])
 
     manager.api.endpoints.create(region=region,
                                  service_id=service_id,
-                                 publicurl=public_url,
-                                 adminurl=admin_url,
-                                 internalurl=internal_url)
+                                 publicurl=publicurl,
+                                 adminurl=adminurl,
+                                 internalurl=internalurl)
     juju_log("Created new endpoint template for '%s' in '%s'" %
                 (region, service))
 
@@ -411,12 +358,33 @@
     create_service_entry("keystone", "identity", "Keystone Identity Service")
     # following documentation here, perhaps we should be using juju
     # public/private addresses for public/internal urls.
-    public_url = "http://%s:%s/v2.0" % (config["hostname"], config["service-port"])
-    admin_url = "http://%s:%s/v2.0" % (config["hostname"], config["admin-port"])
-    internal_url = "http://%s:%s/v2.0" % (config["hostname"], config["service-port"])
-    create_endpoint_template("RegionOne", "keystone", public_url,
+    if is_clustered():
+        juju_log("Creating endpoint for clustered configuration")
+        for region in config['region'].split():
+            create_keystone_endpoint(service_host=config["vip"],
+                                     service_port=int(config["service-port"]) + 1,
+                                     auth_host=config["vip"],
+                                     auth_port=int(config["admin-port"]) + 1,
+                                     region=region)
+    else:
+        juju_log("Creating standard endpoint")
+        for region in config['region'].split():
+            create_keystone_endpoint(service_host=config["hostname"],
+                                     service_port=config["service-port"],
+                                     auth_host=config["hostname"],
+                                     auth_port=config["admin-port"],
+                                     region=region)
+
+
+def create_keystone_endpoint(service_host, service_port,
+                             auth_host, auth_port, region):
+    public_url = "http://%s:%s/v2.0" % (service_host, service_port)
+    admin_url = "http://%s:%s/v2.0" % (auth_host, auth_port)
+    internal_url = "http://%s:%s/v2.0" % (service_host, service_port)
+    create_endpoint_template(region, "keystone", public_url,
                              admin_url, internal_url)
 
+
 def update_user_password(username, password):
     import manager
     manager = manager.KeystoneManager(endpoint='http://localhost:35357/v2.0/',
@@ -431,6 +399,15 @@
     juju_log("Successfully updated password for user '%s'" % username)
 
 
+def configure_pki_tokens(config):
+    '''Configure PKI token signing, if enabled.'''
+    if config['enable-pki'] not in ['True', 'true']:
+        update_config_block('signing', token_format='UUID')
+    else:
+        juju_log('TODO: PKI Support, setting to UUID for now.')
+        update_config_block('signing', token_format='UUID')
+
+
 def do_openstack_upgrade(install_src, packages):
     '''Upgrade packages from a given install src.'''
 
@@ -474,9 +451,34 @@
                                          relation_data["private-address"],
                                          config["database"]))
 
-    juju_log('Running database migrations for %s' % new_vers)
     execute('service keystone stop', echo=True)
-    execute('keystone-manage db_sync', echo=True, die=True)
+    if ((is_clustered() and is_leader()) or
+        not is_clustered()):
+        juju_log('Running database migrations for %s' % new_vers)
+        execute('keystone-manage db_sync', echo=True, die=True)
+    else:
+        juju_log('Not cluster leader; snoozing whilst leader upgrades DB')
+        time.sleep(10)
     execute('service keystone start', echo=True)
     time.sleep(5)
     juju_log('Completed Keystone upgrade: %s -> %s' % (old_vers, new_vers))
+
+
+def is_clustered():
+    for r_id in (relation_ids('ha') or []):
+        for unit in (relation_list(r_id) or []):
+            relation_data = \
+                relation_get_dict(relation_id=r_id,
+                                  remote_unit=unit)
+            if 'clustered' in relation_data:
+                return True
+    return False
+
+
+def is_leader():
+    status = execute('crm resource show res_ks_vip', echo=True)[0].strip()
+    hostname = execute('hostname', echo=True)[0].strip()
+    if hostname in status:
+        return True
+    else:
+        return False

=== modified file 'metadata.yaml'
--- metadata.yaml	2012-06-07 17:43:42 +0000
+++ metadata.yaml	2013-01-31 11:59:26 +0000
@@ -11,3 +11,9 @@
 requires:
   shared-db:
     interface: mysql-shared
+  ha:
+    interface: hacluster
+    scope: container
+peers:
+  cluster:
+    interface: keystone-ha

=== modified file 'revision'
--- revision	2012-12-12 03:52:01 +0000
+++ revision	2013-01-31 11:59:26 +0000
@@ -1,1 +1,1 @@
-165
+195

=== added directory 'templates'
=== added file 'templates/haproxy.cfg'
--- templates/haproxy.cfg	1970-01-01 00:00:00 +0000
+++ templates/haproxy.cfg	2013-01-31 11:59:26 +0000
@@ -0,0 +1,35 @@
+global
+    log 127.0.0.1 local0
+    log 127.0.0.1 local1 notice
+    maxconn 4096
+    user haproxy
+    group haproxy
+    spread-checks 0
+
+defaults
+    log global
+    mode http
+    option httplog
+    option dontlognull
+    retries 3
+    timeout queue 1000
+    timeout connect 1000
+    timeout client 1000
+    timeout server 1000
+
+listen stats :8888
+    mode http
+    stats enable
+    stats hide-version
+    stats realm Haproxy\ Statistics
+    stats uri /
+    stats auth admin:password
+
+{% for service, port in service_ports.iteritems() -%}
+listen {{ service }} 0.0.0.0:{{ port }}
+    balance roundrobin
+    option tcplog
+    {% for unit, address in units.iteritems() -%}
+    server {{ unit }} {{ address }}:{{ port - 1 }} check
+    {% endfor %}
+{% endfor %}