Merge lp:~chad.smith/charms/precise/keystone/ha-support into lp:~charmers/charms/precise/keystone/trunk

[Chad Smith]

This is a merge proposal to establish a potential template for health_script.d, add_to_cluster and remove_from_cluster delivery within the ha-supported openstack charms.

This is intended as a talking point and guidance for what landscape-client will expect on openstack charms supporting an HA services. If there is anything that doesn't look acceptable, I'm game for a change but wanted to pull a template together to seed discussions. Any suggestions about better charm-related ways.

These script locations will be hard-coded within landscape-client and will be run during rolling Openstack upgrades. So, prior to ODS, we'd like to make sure this process makes sense for most haclustered services.

The example of landscape-client's interaction with charm scripts is our HAServiceManager plugin at https://code.launchpad.net/~chad.smith/landscape-client/ha-manager-skeleton/+merge/148593

1. Before any packages are upgraded, landscape-client will only call /var/lib/juju/units/<unit_name>/charm/remove_from_cluster.
    - remove_from_cluster should will place the node in crm standby state, thereby migrating any active HA resources off of the current node.
2. landscape-client will upgrade any packages and any additional package dependencies will be installed.
3. Upon successful upgrade (maybe involving a reboot), landscape-client will run all run-parts health scripts from /var/lib/juju/units/<unit_name>/charm/health_scripts.d. These health scripts will need to return success 0 or fail (non-zero) exit codes to indicate any issues.
4. If all health scripts succeed, landscape-client will run /var/lib/juju/units/<unit_name>/charm/add_to_cluster to bring the node online in the HA cluster again.

  As James mentioned in email, remove_from_cluster script might take more action to cleanly remove an API service from openstack schedulers or configs. If a charm's remove_from_cluster does more than just migrate HA resources away from the node, then we'll need to discuss how to potentially decompose add_to_cluster functionality into separate scripts. One script that allows us to ensure a local openstack API is running locally and can be "health checked" and a separate script add_to_cluster which only finalizes and activates configuration in the HA service.

[Adam Gandelman (gandelman-a)]

Hey Chad-

You targeted this merge into the upstream charm @ lp:charms/keystone, can you retarget toward our WIP HA branch @ lp:~openstack-charmers/charms/precise/keystone/ha-support ?

The general approach here LGTM. But I'd love for things to be made a bit more generic so we can just drop this work into other charms unmodified:

- Perhaps save_script_rc() can to be moved to lib/openstack_common.py without the hard-coded KEYSTONE_SERVICE_NAME, and the caller can add that to the kwargs instead?

- I imagine there are some tests that would be common across all charms (service_running, service_ports_live) It would be great if those can be synced to other charms unmodified as well. Eg, perhaps have the ports look for *_OPENSTACK_PORT in the environment and ensure each one? The services check can determine what it should check based on *_OPENSTACK_SERVICE_NAME? Not sure if these common tests should live in some directory other than the tests specific to each service. /var/lib/juju/units/$unit/healthchecks/common.d/ & /var/lib/juju/units/$unit/healthchecks/$unit.d/ ??

End goal for me is to have the common framework for the health checking live with the other common code we maintain, and we can just sync it to other charms when we get it right in one. Of course this is complicated by the combination of bash + python charms we currently maintain, but still doable I think.

[Chad Smith (chad.smith)]

Good points Adam. I'll make save_script_rc more generic and the health scripts will be a bit smarter to look for *OPENSTACK* vars where necessary.

I'm pulled onto something else at the moment, but I'll repost this review against the right source branch and get a change to you either today or tomorrow morning.

[lp:~chad.smith/charms/precise/keystone/ha-support]

56. By Chad Smith on 2013-02-22

move now-generic save_script_rc to lib/openstack_common.py. Update health scripts to be more flexible based on OPENSTACK_PORT* and OPENSTACK_SERVICE* environment variables

57. By Chad Smith on 2013-02-22

move add_cluster, remove_from_cluster and health_checks.d to a new scripts subdir

58. By Chad Smith on 2013-03-05

more strict netstat port matching

Unmerged revisions

=== added file 'add_to_cluster'
--- add_to_cluster	1970-01-01 00:00:00 +0000
+++ add_to_cluster	2013-02-22 19:26:19 +0000
@@ -0,0 +1,2 @@
+#!/bin/bash
+crm node online
 
=== modified file 'config.yaml'
--- config.yaml	2012-10-12 17:26:48 +0000
+++ config.yaml	2013-02-22 19:26:19 +0000
@@ -26,6 +26,10 @@
     default: "/etc/keystone/keystone.conf"
     type: string
     description: "Location of keystone configuration file"
+  log-level:
+    default: WARNING
+    type: string
+    description: Log level (WARNING, INFO, DEBUG, ERROR)
   service-port:
     default: 5000
     type: int
@@ -75,3 +79,36 @@
     default: "keystone"
     type: string
     description: "Database username"
+  region:
+    default: RegionOne
+    type: string
+    description: "OpenStack Region(s) - separate multiple regions with single space"
+  # HA configuration settings
+  vip:
+    type: string
+    description: "Virtual IP to use to front keystone in ha configuration"
+  vip_iface:
+    type: string
+    default: eth0
+    description: "Network Interface where to place the Virtual IP"
+  vip_cidr:
+    type: int
+    default: 24
+    description: "Netmask that will be used for the Virtual IP"
+  ha-bindiface:
+    type: string
+    default: eth0
+    description: |
+      Default network interface on which HA cluster will bind to communication
+      with the other members of the HA Cluster.
+  ha-mcastport:
+    type: int
+    default: 5403
+    description: |
+      Default multicast port number that will be used to communicate between
+      HA Cluster nodes.
+  # PKI enablement and configuration (Grizzly and beyond)
+  enable-pki:
+    default: "false"
+    type: string
+    description: "Enable PKI token signing (Grizzly and beyond)"
 
=== added directory 'health_checks.d'
=== added file 'health_checks.d/service_ports_live'
--- health_checks.d/service_ports_live	1970-01-01 00:00:00 +0000
+++ health_checks.d/service_ports_live	2013-02-22 19:26:19 +0000
@@ -0,0 +1,13 @@
+#!/bin/bash
+# Validate that service ports are active
+HEALTH_DIR=`dirname $0`
+UNIT_DIR=`dirname $HEALTH_DIR`
+. $UNIT_DIR/scriptrc
+set -e
+
+# Grab any OPENSTACK_PORT* environment variables
+openstack_ports=`env| awk -F '=' '(/OPENSTACK_PORT/){print $2}'`
+for port in $openstack_ports
+do
+    netstat -ln | grep -q $port
+done
 
=== added file 'health_checks.d/service_running'
--- health_checks.d/service_running	1970-01-01 00:00:00 +0000
+++ health_checks.d/service_running	2013-02-22 19:26:19 +0000
@@ -0,0 +1,13 @@
+#!/bin/bash
+# Validate that service is running
+HEALTH_DIR=`dirname $0`
+UNIT_DIR=`dirname $HEALTH_DIR`
+. $UNIT_DIR/scriptrc
+set -e
+
+# Grab any OPENSTACK_SERVICE* environment variables
+openstack_service_names=`env| awk -F '=' '(/OPENSTACK_SERVICE/){print $2}'`
+for service_name in $openstack_service_names
+do
+    service $service_name status | grep -q running
+done
 
=== added symlink 'hooks/cluster-relation-changed'
=== target is u'keystone-hooks'
=== added symlink 'hooks/cluster-relation-departed'
=== target is u'keystone-hooks'
=== added symlink 'hooks/ha-relation-changed'
=== target is u'keystone-hooks'
=== added symlink 'hooks/ha-relation-joined'
=== target is u'keystone-hooks'
=== modified file 'hooks/keystone-hooks'
--- hooks/keystone-hooks	2012-12-12 03:52:01 +0000
+++ hooks/keystone-hooks	2013-02-22 19:26:19 +0000
@@ -8,7 +8,7 @@
 
 config = config_get()
 
-packages = "keystone python-mysqldb pwgen"
+packages = "keystone python-mysqldb pwgen haproxy python-jinja2"
 service = "keystone"
 
 # used to verify joined services are valid openstack components.
@@ -46,6 +46,14 @@
     "quantum": {
         "type": "network",
         "desc": "Quantum Networking Service"
+    },
+    "oxygen": {
+        "type": "oxygen",
+        "desc": "Oxygen Cloud Image Service"
+    },
+    "ceilometer": {
+        "type": "metering",
+        "desc": "Ceilometer Metering Service"
     }
 }
 
@@ -69,12 +77,14 @@
                         driver='keystone.token.backends.sql.Token')
     update_config_block('ec2',
                         driver='keystone.contrib.ec2.backends.sql.Ec2')
+
     execute("service keystone stop", echo=True)
     execute("keystone-manage db_sync")
     execute("service keystone start", echo=True)
     time.sleep(5)
     ensure_initial_admin(config)
 
+
 def db_joined():
     relation_data = { "database": config["database"],
                       "username": config["database-user"],
@@ -84,15 +94,22 @@
 def db_changed():
     relation_data = relation_get_dict()
     if ('password' not in relation_data or
-        'private-address' not in relation_data):
-        juju_log("private-address or password not set. Peer not ready, exit 0")
+        'db_host' not in relation_data):
+        juju_log("db_host or password not set. Peer not ready, exit 0")
         exit(0)
     update_config_block('sql', connection="mysql://%s:%s@%s/%s" %
                             (config["database-user"],
                              relation_data["password"],
-                             relation_data["private-address"],
+                             relation_data["db_host"],
                              config["database"]))
+
     execute("service keystone stop", echo=True)
+
+    if not eligible_leader():
+        juju_log('Deferring DB initialization to service leader.')
+        execute("service keystone start")
+        return
+
     execute("keystone-manage db_sync", echo=True)
     execute("service keystone start")
     time.sleep(5)
@@ -124,18 +141,30 @@
             realtion_set({ "admin_token": -1 })
             return
 
-    def add_endpoint(region, service, public_url, admin_url, internal_url):
+    def add_endpoint(region, service, publicurl, adminurl, internalurl):
         desc = valid_services[service]["desc"]
         service_type = valid_services[service]["type"]
         create_service_entry(service, service_type, desc)
         create_endpoint_template(region=region, service=service,
-                                 public_url=public_url,
-                                 admin_url=admin_url,
-                                 internal_url=internal_url)
+                                 publicurl=publicurl,
+                                 adminurl=adminurl,
+                                 internalurl=internalurl)
+
+    if not eligible_leader():
+        juju_log('Deferring identity_changed() to service leader.')
+        return
 
     settings = relation_get_dict(relation_id=relation_id,
                                  remote_unit=remote_unit)
 
+    # Allow the remote service to request creation of any additional roles.
+    # Currently used by Swift.
+    if 'requested_roles' in settings and settings['requested_roles'] != 'None':
+        roles = settings['requested_roles'].split(',')
+        juju_log("Creating requested roles: %s" % roles)
+        for role in roles:
+            create_role(role, user=config['admin-user'], tenant='admin')
+
     # the minimum settings needed per endpoint
     single = set(['service', 'region', 'public_url', 'admin_url',
                   'internal_url'])
@@ -145,13 +174,27 @@
         if 'None' in [v for k,v in settings.iteritems()]:
             # Some backend services advertise no endpoint but require a
             # hook execution to update auth strategy.
+            relation_data = {}
+            # Check if clustered and use vip + haproxy ports if so
+            if is_clustered():
+                relation_data["auth_host"] = config['vip']
+                relation_data["auth_port"] = SERVICE_PORTS['keystone_admin']
+                relation_data["service_host"] = config['vip']
+                relation_data["service_port"] = SERVICE_PORTS['keystone_service']
+            else:
+                relation_data["auth_host"] = config['hostname']
+                relation_data["auth_port"] = config['auth-port']
+                relation_data["service_host"] = config['hostname']
+                relation_data["service_port"] = config['service-port']
+            relation_set(relation_data)
             return
 
+
         ensure_valid_service(settings['service'])
         add_endpoint(region=settings['region'], service=settings['service'],
-                     public_url=settings['public_url'],
-                     admin_url=settings['admin_url'],
-                     internal_url=settings['internal_url'])
+                     publicurl=settings['public_url'],
+                     adminurl=settings['admin_url'],
+                     internalurl=settings['internal_url'])
         service_username = settings['service']
     else:
         # assemble multiple endpoints from relation data. service name
@@ -186,9 +229,9 @@
                 ep = endpoints[ep]
                 ensure_valid_service(ep['service'])
                 add_endpoint(region=ep['region'], service=ep['service'],
-                             public_url=ep['public_url'],
-                             admin_url=ep['admin_url'],
-                             internal_url=ep['internal_url'])
+                             publicurl=ep['public_url'],
+                             adminurl=ep['admin_url'],
+                             internalurl=ep['internal_url'])
                 services.append(ep['service'])
         service_username = '_'.join(services)
 
@@ -201,26 +244,10 @@
     token = get_admin_token()
     juju_log("Creating service credentials for '%s'" % service_username)
 
-    stored_passwd = '/var/lib/keystone/%s.passwd' % service_username
-    if os.path.isfile(stored_passwd):
-        juju_log("Loading stored service passwd from %s" % stored_passwd)
-        service_password = open(stored_passwd, 'r').readline().strip('\n')
-    else:
-        juju_log("Generating a new service password for %s" % service_username)
-        service_password = execute('pwgen -c 32 1', die=True)[0].strip()
-        open(stored_passwd, 'w+').writelines("%s\n" % service_password)
-
+    service_password = get_service_password(service_username)
     create_user(service_username, service_password, config['service-tenant'])
     grant_role(service_username, config['admin-role'], config['service-tenant'])
 
-    # Allow the remote service to request creation of any additional roles.
-    # Currently used by Swift.
-    if 'requested_roles' in settings:
-        roles = settings['requested_roles'].split(',')
-        juju_log("Creating requested roles: %s" % roles)
-        for role in roles:
-            create_role(role, user=config['admin-user'], tenant='admin')
-
     # As of https://review.openstack.org/#change,4675, all nodes hosting
     # an endpoint(s) needs a service username and password assigned to
     # the service tenant and granted admin role.
@@ -237,7 +264,15 @@
         "service_password": service_password,
         "service_tenant": config['service-tenant']
     }
+    # Check if clustered and use vip + haproxy ports if so
+    if is_clustered():
+        relation_data["auth_host"] = config['vip']
+        relation_data["auth_port"] = SERVICE_PORTS['keystone_admin']
+        relation_data["service_host"] = config['vip']
+        relation_data["service_port"] = SERVICE_PORTS['keystone_service']
+
     relation_set(relation_data)
+    synchronize_service_credentials()
 
 def config_changed():
 
@@ -246,11 +281,117 @@
     available = get_os_codename_install_source(config['openstack-origin'])
     installed = get_os_codename_package('keystone')
 
-    if get_os_version_codename(available) > get_os_version_codename(installed):
+    if (available and
+        get_os_version_codename(available) > get_os_version_codename(installed)):
         do_openstack_upgrade(config['openstack-origin'], packages)
 
+    env_vars = {'OPENSTACK_SERVICE_KEYSTONE': 'keystone',
+                'OPENSTACK_PORT_ADMIN': config['admin-port'],
+                'OPENSTACK_PORT_PUBLIC': config['service-port']}
+    save_script_rc(**env_vars)
+
     set_admin_token(config['admin-token'])
-    ensure_initial_admin(config)
+
+    if eligible_leader():
+        juju_log('Cluster leader - ensuring endpoint configuration is up to date')
+        ensure_initial_admin(config)
+
+    update_config_block('logger_root', level=config['log-level'],
+                        file='/etc/keystone/logging.conf')
+    if get_os_version_package('keystone') >= '2013.1':
+        # PKI introduced in Grizzly
+        configure_pki_tokens(config)
+
+    execute("service keystone restart", echo=True)
+    cluster_changed()
+
+
+def upgrade_charm():
+    cluster_changed()
+    if eligible_leader():
+        juju_log('Cluster leader - ensuring endpoint configuration is up to date')
+        ensure_initial_admin(config)
+
+
+SERVICE_PORTS = {
+    "keystone_admin": int(config['admin-port']) + 1,
+    "keystone_service": int(config['service-port']) + 1
+    }
+
+
+def cluster_changed():
+    cluster_hosts = {}
+    cluster_hosts['self'] = config['hostname']
+    for r_id in relation_ids('cluster'):
+        for unit in relation_list(r_id):
+            cluster_hosts[unit.replace('/','-')] = \
+                relation_get_dict(relation_id=r_id,
+                                  remote_unit=unit)['private-address']
+    configure_haproxy(cluster_hosts,
+                      SERVICE_PORTS)
+
+    synchronize_service_credentials()
+
+
+def ha_relation_changed():
+    relation_data = relation_get_dict()
+    if ('clustered' in relation_data and
+        is_leader()):
+        juju_log('Cluster configured, notifying other services and updating'
+                 'keystone endpoint configuration')
+        # Update keystone endpoint to point at VIP
+        ensure_initial_admin(config)
+        # Tell all related services to start using
+        # the VIP and haproxy ports instead
+        for r_id in relation_ids('identity-service'):
+            relation_set_2(rid=r_id,
+                           auth_host=config['vip'],
+                           service_host=config['vip'],
+                           service_port=SERVICE_PORTS['keystone_service'],
+                           auth_port=SERVICE_PORTS['keystone_admin'])
+
+
+def ha_relation_joined():
+    # Obtain the config values necessary for the cluster config. These
+    # include multicast port and interface to bind to.
+    corosync_bindiface = config['ha-bindiface']
+    corosync_mcastport = config['ha-mcastport']
+
+    # Obtain resources
+    resources = {
+            'res_ks_vip':'ocf:heartbeat:IPaddr2',
+            'res_ks_haproxy':'lsb:haproxy'
+        }
+    # TODO: Obtain netmask and nic where to place VIP.
+    resource_params = {
+            'res_ks_vip':'params ip="%s" cidr_netmask="%s" nic="%s"' % (config['vip'],
+                              config['vip_cidr'], config['vip_iface']),
+            'res_ks_haproxy':'op monitor interval="5s"'
+        }
+    init_services = {
+            'res_ks_haproxy':'haproxy'
+        }
+    groups = {
+            'grp_ks_haproxy':'res_ks_vip res_ks_haproxy'
+        }
+    #clones = {
+    #        'cln_ks_haproxy':'res_ks_haproxy meta globally-unique="false" interleave="true"'
+    #    }
+
+    #orders = {
+    #        'ord_vip_before_haproxy':'inf: res_ks_vip res_ks_haproxy'
+    #    }
+    #colocations = {
+    #        'col_vip_on_haproxy':'inf: res_ks_haproxy res_ks_vip'
+    #    }
+
+    relation_set_2(init_services=init_services,
+                   corosync_bindiface=corosync_bindiface,
+                   corosync_mcastport=corosync_mcastport,
+                   resources=resources,
+                   resource_params=resource_params,
+                   groups=groups)
+
 
 hooks = {
     "install": install_hook,
@@ -258,12 +399,17 @@
     "shared-db-relation-changed": db_changed,
     "identity-service-relation-joined": identity_joined,
     "identity-service-relation-changed": identity_changed,
-    "config-changed": config_changed
+    "config-changed": config_changed,
+    "cluster-relation-changed": cluster_changed,
+    "cluster-relation-departed": cluster_changed,
+    "ha-relation-joined": ha_relation_joined,
+    "ha-relation-changed": ha_relation_changed,
+    "upgrade-charm": upgrade_charm
 }
 
 # keystone-hooks gets called by symlink corresponding to the requested relation
 # hook.
-arg0 = sys.argv[0].split("/").pop()
-if arg0 not in hooks.keys():
-    error_out("Unsupported hook: %s" % arg0)
-hooks[arg0]()
+hook = os.path.basename(sys.argv[0])
+if hook not in hooks.keys():
+    error_out("Unsupported hook: %s" % hook)
+hooks[hook]()
 
=== modified file 'hooks/lib/openstack_common.py'
--- hooks/lib/openstack_common.py	2012-12-05 20:35:05 +0000
+++ hooks/lib/openstack_common.py	2013-02-22 19:26:19 +0000
@@ -3,6 +3,7 @@
 # Common python helper functions used for OpenStack charms.
 
 import subprocess
+import os
 
 CLOUD_ARCHIVE_URL = "http://ubuntu-cloud.archive.canonical.com/ubuntu"
 CLOUD_ARCHIVE_KEY_ID = '5EDB1B62EC4926EA'
@@ -22,6 +23,12 @@
     '2013.1': 'grizzly'
 }
 
+# The ugly duckling
+swift_codenames = {
+    '1.4.3': 'diablo',
+    '1.4.8': 'essex',
+    '1.7.4': 'folsom'
+}
 
 def juju_log(msg):
     subprocess.check_call(['juju-log', msg])
@@ -118,12 +125,32 @@
 
     vers = vers[:6]
     try:
-        return openstack_codenames[vers]
+        if 'swift' in pkg:
+            vers = vers[:5]
+            return swift_codenames[vers]
+        else:
+            vers = vers[:6]
+            return openstack_codenames[vers]
     except KeyError:
         e = 'Could not determine OpenStack codename for version %s' % vers
         error_out(e)
 
 
+def get_os_version_package(pkg):
+    '''Derive OpenStack version number from an installed package.'''
+    codename = get_os_codename_package(pkg)
+
+    if 'swift' in pkg:
+        vers_map = swift_codenames
+    else:
+        vers_map = openstack_codenames
+
+    for version, cname in vers_map.iteritems():
+        if cname == codename:
+            return version
+    e = "Could not determine OpenStack version for package: %s" % pkg
+    error_out(e)
+
 def configure_installation_source(rel):
     '''Configure apt installation source.'''
 
@@ -164,9 +191,11 @@
                 'version (%s)' % (ca_rel, ubuntu_rel)
             error_out(e)
 
-        if ca_rel == 'folsom/staging':
+        if 'staging' in ca_rel:
             # staging is just a regular PPA.
-            cmd = 'add-apt-repository -y ppa:ubuntu-cloud-archive/folsom-staging'
+            os_rel = ca_rel.split('/')[0]
+            ppa = 'ppa:ubuntu-cloud-archive/%s-staging' % os_rel
+            cmd = 'add-apt-repository -y %s' % ppa
             subprocess.check_call(cmd.split(' '))
             return
 
@@ -174,7 +203,10 @@
         pockets = {
             'folsom': 'precise-updates/folsom',
             'folsom/updates': 'precise-updates/folsom',
-            'folsom/proposed': 'precise-proposed/folsom'
+            'folsom/proposed': 'precise-proposed/folsom',
+            'grizzly': 'precise-updates/grizzly',
+            'grizzly/updates': 'precise-updates/grizzly',
+            'grizzly/proposed': 'precise-proposed/grizzly'
         }
 
         try:
@@ -191,3 +223,39 @@
     else:
         error_out("Invalid openstack-release specified: %s" % rel)
 
+HAPROXY_CONF = '/etc/haproxy/haproxy.cfg'
+HAPROXY_DEFAULT = '/etc/default/haproxy'
+
+def configure_haproxy(units, service_ports, template_dir=None):
+    template_dir = template_dir or 'templates'
+    import jinja2
+    context = {
+        'units': units,
+        'service_ports': service_ports
+        }
+    templates = jinja2.Environment(
+                    loader=jinja2.FileSystemLoader(template_dir)
+                    )
+    template = templates.get_template(
+                    os.path.basename(HAPROXY_CONF)
+                    )
+    with open(HAPROXY_CONF, 'w') as f:
+        f.write(template.render(context))
+    with open(HAPROXY_DEFAULT, 'w') as f:
+        f.write('ENABLED=1')
+
+def save_script_rc(script_path="scriptrc", **env_vars):
+    """
+    Write an rc file in the charm-delivered directory containing
+    exported environment variables provided by env_vars. Any charm scripts run
+    outside the juju hook environment can source this scriptrc to obtain
+    updated config information necessary to perform health checks or
+    service changes.
+    """
+    unit_name = os.getenv('JUJU_UNIT_NAME').replace('/', '-')
+    juju_rc_path="/var/lib/juju/units/%s/charm/%s" % (unit_name, script_path)
+    with open(juju_rc_path, 'wb') as rc_script:
+        rc_script.write(
+            "#!/bin/bash\n")
+        [rc_script.write('export %s=%s\n' % (u, p))
+         for u, p in env_vars.iteritems() if u != "script_path"]
 
=== added symlink 'hooks/upgrade-charm'
=== target is u'keystone-hooks'
=== modified file 'hooks/utils.py'
--- hooks/utils.py	2012-12-12 03:52:41 +0000
+++ hooks/utils.py	2013-02-22 19:26:19 +0000
@@ -1,4 +1,5 @@
 #!/usr/bin/python
+import ConfigParser
 import subprocess
 import sys
 import json
@@ -10,9 +11,10 @@
 keystone_conf = "/etc/keystone/keystone.conf"
 stored_passwd = "/var/lib/keystone/keystone.passwd"
 stored_token = "/var/lib/keystone/keystone.token"
+SERVICE_PASSWD_PATH = '/var/lib/keystone/services.passwd'
 
 def execute(cmd, die=False, echo=False):
-    """ Executes a command 
+    """ Executes a command
 
     if die=True, script will exit(1) if command does not return 0
     if echo=True, output of command will be printed to stdout
@@ -79,6 +81,20 @@
     for k in relation_data:
         execute("relation-set %s=%s" % (k, relation_data[k]), die=True)
 
+def relation_set_2(**kwargs):
+    cmd = [
+        'relation-set'
+        ]
+    args = []
+    for k, v in kwargs.items():
+        if k == 'rid':
+            cmd.append('-r')
+            cmd.append(v)
+        else:
+            args.append('{}={}'.format(k, v))
+    cmd += args
+    subprocess.check_call(cmd)
+
 def relation_get(relation_data):
     """ Obtain all current relation data
     relation_data is a list of options to query from the relation
@@ -149,106 +165,26 @@
                           keystone_conf)
     error_out('Could not find admin_token line in %s' % keystone_conf)
 
-def update_config_block(block, **kwargs):
+def update_config_block(section, **kwargs):
     """ Updates keystone.conf blocks given kwargs.
-    Can be used to update driver settings for a particular backend,
-    setting the sql connection, etc.
-
-    Parses block heading as '[block]'
-
-    If block does not exist, a new block will be created at end of file with
-    given kwargs
-    """
-    f = open(keystone_conf, "r+")
-    orig = f.readlines()
-    new = []
-    found_block = ""
-    heading = "[%s]\n" % block
-
-    lines = len(orig)
-    ln = 0
-
-    def update_block(block):
-        for k, v in kwargs.iteritems():
-            for l in block:
-                if l.strip().split(" ")[0] == k:
-                    block[block.index(l)] = "%s = %s\n" % (k, v)
-                    return
-            block.append('%s = %s\n' % (k, v))
-            block.append('\n')
-
-    try:
-        found = False
-        while ln < lines:
-            if orig[ln] != heading:
-                new.append(orig[ln])
-                ln += 1
-            else:
-                new.append(orig[ln])
-                ln += 1
-                block = []
-                while orig[ln].strip() != '':
-                    block.append(orig[ln])
-                    ln += 1
-                update_block(block)
-                new += block
-                found = True
-
-        if not found:
-            if new[(len(new) - 1)].strip() != '':
-                new.append('\n')
-            new.append('%s' % heading)
-            for k, v in kwargs.iteritems():
-                new.append('%s = %s\n' % (k, v))
-            new.append('\n')
-    except:
-        error_out('Error while attempting to update config block. '\
-                  'Refusing to overwite existing config.')
-
-        return
-
-    # backup original config
-    backup = open(keystone_conf + '.juju-back', 'w+')
-    for l in orig:
-        backup.write(l)
-    backup.close()
-
-    # update config
-    f.seek(0)
-    f.truncate()
-    for l in new:
-        f.write(l)
-
-
-def keystone_conf_update(opt, val):
-    """ Updates keystone.conf values 
-    If option exists, it is reset to new value
-    If it does not, it added to the top of the config file after the [DEFAULT]
-    heading to keep it out of the paste deploy config
-    """
-    f = open(keystone_conf, "r+")
-    orig = f.readlines()
-    new = ""
-    found = False
-    for l in orig:
-        if l.split(' ')[0] == opt:
-            juju_log("Updating %s, setting %s = %s" % (keystone_conf, opt, val))
-            new += "%s = %s\n" % (opt, val)
-            found  = True
-        else:
-            new += l
-    new = new.split('\n')
-    # insert a new value at the top of the file, after the 'DEFAULT' header so
-    # as not to muck up paste deploy configuration later in the file 
-    if not found:
-        juju_log("Adding new config option %s = %s" % (opt, val))
-        header = new.index("[DEFAULT]")
-        new.insert((header+1), "%s = %s" % (opt, val))
-    f.seek(0)
-    f.truncate()
-    for l in new:
-        f.write("%s\n" % l)
-    f.close
+    Update a config setting in a specific setting of a config
+    file (/etc/keystone/keystone.conf, by default)
+    """
+    if 'file' in kwargs:
+        conf_file = kwargs['file']
+        del kwargs['file']
+    else:
+        conf_file = keystone_conf
+    config = ConfigParser.RawConfigParser()
+    config.read(conf_file)
+
+    if section != 'DEFAULT' and not config.has_section(section):
+        config.add_section(section)
+
+    for k, v in kwargs.iteritems():
+        config.set(section, k, v)
+    with open(conf_file, 'wb') as out:
+        config.write(out)
 
 def create_service_entry(service_name, service_type, service_desc, owner=None):
     """ Add a new service entry to keystone if one does not already exist """
@@ -264,8 +200,8 @@
                                 description=service_desc)
     juju_log("Created new service entry '%s'" % service_name)
 
-def create_endpoint_template(region, service,  public_url, admin_url,
-                             internal_url):
+def create_endpoint_template(region, service,  publicurl, adminurl,
+                             internalurl):
     """ Create a new endpoint template for service if one does not already
         exist matching name *and* region """
     import manager
@@ -276,13 +212,24 @@
         if ep['service_id'] == service_id and ep['region'] == region:
             juju_log("Endpoint template already exists for '%s' in '%s'"
                       % (service, region))
-            return
+
+            up_to_date = True
+            for k in ['publicurl', 'adminurl', 'internalurl']:
+                if ep[k] != locals()[k]:
+                    up_to_date = False
+
+            if up_to_date:
+                return
+            else:
+                # delete endpoint and recreate if endpoint urls need updating.
+                juju_log("Updating endpoint template with new endpoint urls.")
+                manager.api.endpoints.delete(ep['id'])
 
     manager.api.endpoints.create(region=region,
                                  service_id=service_id,
-                                 publicurl=public_url,
-                                 adminurl=admin_url,
-                                 internalurl=internal_url)
+                                 publicurl=publicurl,
+                                 adminurl=adminurl,
+                                 internalurl=internalurl)
     juju_log("Created new endpoint template for '%s' in '%s'" %
                 (region, service))
 
@@ -411,12 +358,33 @@
     create_service_entry("keystone", "identity", "Keystone Identity Service")
     # following documentation here, perhaps we should be using juju
     # public/private addresses for public/internal urls.
-    public_url = "http://%s:%s/v2.0" % (config["hostname"], config["service-port"])
-    admin_url = "http://%s:%s/v2.0" % (config["hostname"], config["admin-port"])
-    internal_url = "http://%s:%s/v2.0" % (config["hostname"], config["service-port"])
-    create_endpoint_template("RegionOne", "keystone", public_url,
+    if is_clustered():
+        juju_log("Creating endpoint for clustered configuration")
+        for region in config['region'].split():
+            create_keystone_endpoint(service_host=config["vip"],
+                                     service_port=int(config["service-port"]) + 1,
+                                     auth_host=config["vip"],
+                                     auth_port=int(config["admin-port"]) + 1,
+                                     region=region)
+    else:
+        juju_log("Creating standard endpoint")
+        for region in config['region'].split():
+            create_keystone_endpoint(service_host=config["hostname"],
+                                     service_port=config["service-port"],
+                                     auth_host=config["hostname"],
+                                     auth_port=config["admin-port"],
+                                     region=region)
+
+
+def create_keystone_endpoint(service_host, service_port,
+                             auth_host, auth_port, region):
+    public_url = "http://%s:%s/v2.0" % (service_host, service_port)
+    admin_url = "http://%s:%s/v2.0" % (auth_host, auth_port)
+    internal_url = "http://%s:%s/v2.0" % (service_host, service_port)
+    create_endpoint_template(region, "keystone", public_url,
                              admin_url, internal_url)
 
+
 def update_user_password(username, password):
     import manager
     manager = manager.KeystoneManager(endpoint='http://localhost:35357/v2.0/',
@@ -430,6 +398,40 @@
     manager.api.users.update_password(user=user_id, password=password)
     juju_log("Successfully updated password for user '%s'" % username)
 
+def load_stored_passwords(path=SERVICE_PASSWD_PATH):
+    creds = {}
+    if not os.path.isfile(path):
+        return creds
+
+    stored_passwd = open(path, 'r')
+    for l in stored_passwd.readlines():
+        user, passwd = l.strip().split(':')
+        creds[user] = passwd
+    return creds
+
+def save_stored_passwords(path=SERVICE_PASSWD_PATH, **creds):
+    with open(path, 'wb') as stored_passwd:
+        [stored_passwd.write('%s:%s\n' % (u, p)) for u, p in creds.iteritems()]
+
+def get_service_password(service_username):
+    creds = load_stored_passwords()
+    if service_username in creds:
+        return creds[service_username]
+
+    passwd = subprocess.check_output(['pwgen', '-c', '32', '1']).strip()
+    creds[service_username] = passwd
+    save_stored_passwords(**creds)
+
+    return passwd
+
+def configure_pki_tokens(config):
+    '''Configure PKI token signing, if enabled.'''
+    if config['enable-pki'] not in ['True', 'true']:
+        update_config_block('signing', token_format='UUID')
+    else:
+        juju_log('TODO: PKI Support, setting to UUID for now.')
+        update_config_block('signing', token_format='UUID')
+
 
 def do_openstack_upgrade(install_src, packages):
     '''Upgrade packages from a given install src.'''
@@ -474,9 +476,84 @@
                                          relation_data["private-address"],
                                          config["database"]))
 
-    juju_log('Running database migrations for %s' % new_vers)
     execute('service keystone stop', echo=True)
-    execute('keystone-manage db_sync', echo=True, die=True)
+    if ((is_clustered() and is_leader()) or
+        not is_clustered()):
+        juju_log('Running database migrations for %s' % new_vers)
+        execute('keystone-manage db_sync', echo=True, die=True)
+    else:
+        juju_log('Not cluster leader; snoozing whilst leader upgrades DB')
+        time.sleep(10)
     execute('service keystone start', echo=True)
     time.sleep(5)
     juju_log('Completed Keystone upgrade: %s -> %s' % (old_vers, new_vers))
+
+
+def is_clustered():
+    for r_id in (relation_ids('ha') or []):
+        for unit in (relation_list(r_id) or []):
+            relation_data = \
+                relation_get_dict(relation_id=r_id,
+                                  remote_unit=unit)
+            if 'clustered' in relation_data:
+                return True
+    return False
+
+
+def is_leader():
+    status = execute('crm resource show res_ks_vip', echo=True)[0].strip()
+    hostname = execute('hostname', echo=True)[0].strip()
+    if hostname in status:
+        return True
+    else:
+        return False
+
+
+def peer_units():
+    peers = []
+    for r_id in (relation_ids('cluster') or []):
+        for unit in (relation_list(r_id) or []):
+            peers.append(unit)
+    return peers
+
+def oldest_peer(peers):
+    local_unit_no = os.getenv('JUJU_UNIT_NAME').split('/')[1]
+    for peer in peers:
+        remote_unit_no = peer.split('/')[1]
+        if remote_unit_no < local_unit_no:
+            return False
+    return True
+
+
+def eligible_leader():
+    if is_clustered():
+        if not is_leader():
+            juju_log('Deferring action to CRM leader.')
+            return False
+    else:
+        peers = peer_units()
+        if peers and not oldest_peer(peers):
+            juju_log('Deferring action to oldest service unit.')
+            return False
+    return True
+
+
+def synchronize_service_credentials():
+    '''
+    Broadcast service credentials to peers or consume those that have been
+    broadcasted by peer, depending on hook context.
+    '''
+    if os.path.basename(sys.argv[0]) == 'cluster-relation-changed':
+        r_data = relation_get_dict()
+        if 'service_credentials' in r_data:
+            juju_log('Saving service passwords from peer.')
+            save_stored_passwords(**json.loads(r_data['service_credentials']))
+        return
+
+    creds = load_stored_passwords()
+    if not creds:
+        return
+    juju_log('Synchronizing service passwords to all peers.')
+    creds = json.dumps(creds)
+    for r_id in (relation_ids('cluster') or []):
+        relation_set_2(rid=r_id, service_credentials=creds)
 
=== modified file 'metadata.yaml'
--- metadata.yaml	2012-06-07 17:43:42 +0000
+++ metadata.yaml	2013-02-22 19:26:19 +0000
@@ -11,3 +11,9 @@
 requires:
   shared-db:
     interface: mysql-shared
+  ha:
+    interface: hacluster
+    scope: container
+peers:
+  cluster:
+    interface: keystone-ha
 
=== added file 'remove_from_cluster'
--- remove_from_cluster	1970-01-01 00:00:00 +0000
+++ remove_from_cluster	2013-02-22 19:26:19 +0000
@@ -0,0 +1,2 @@
+#!/bin/bash
+crm node standby
 
=== modified file 'revision'
--- revision	2012-12-12 03:52:01 +0000
+++ revision	2013-02-22 19:26:19 +0000
@@ -1,1 +1,1 @@
-165
+197
 
=== added directory 'templates'
=== added file 'templates/haproxy.cfg'
--- templates/haproxy.cfg	1970-01-01 00:00:00 +0000
+++ templates/haproxy.cfg	2013-02-22 19:26:19 +0000
@@ -0,0 +1,35 @@
+global
+    log 127.0.0.1 local0
+    log 127.0.0.1 local1 notice
+    maxconn 4096
+    user haproxy
+    group haproxy
+    spread-checks 0
+
+defaults
+    log global
+    mode http
+    option httplog
+    option dontlognull
+    retries 3
+    timeout queue 1000
+    timeout connect 1000
+    timeout client 10000
+    timeout server 10000
+
+listen stats :8888
+    mode http
+    stats enable
+    stats hide-version
+    stats realm Haproxy\ Statistics
+    stats uri /
+    stats auth admin:password
+
+{% for service, port in service_ports.iteritems() -%}
+listen {{ service }} 0.0.0.0:{{ port }}
+    balance roundrobin
+    option tcplog
+    {% for unit, address in units.iteritems() -%}
+    server {{ unit }} {{ address }}:{{ port - 1 }} check
+    {% endfor %}
+{% endfor %}