Merge lp:~gandelman-a/charms/precise/keystone/ssl_file_sync into lp:~openstack-charmers/charms/precise/keystone/ha-support

[Adam Gandelman]

Adds support for Keystone managed SSL certificates for service endpoints. Adds a keystone_ssl.py library that will initialize a local directory as what will be used as the certificate authority. In addition to service credentials, new services that join via the identity-service relation will also receive a signed SSL certificate, key and CA cert. The remote service charm is in charge of using this cert/key to configure its own HTTPS. When it has done so, it can update its endpoint in keystone catalog via its relation. Every service now has an optional ssl_cert + ssl_key config option in case admin wants to update certificates manually, override those installed by keystone, or use certificates signed by a real certificate authority.

The KS certificate authority needs to be synchronized among all peer keystone units. To acehive this, I've added a new unison.py library that takes care of setting up peer SSH access via an intermediate user, and allows for file and directory synchronization between peers using unison. It's assumed that only one node will be synchornizing files to peers (the eligible_leader). I've replaced the old file-sync-via-base64-relation-data for service passwords with this new method as well, and it is working well.

It is not required that the remote charms support HTTPS endpoint reconfiguration, they can simply choose not to use the generated certificate data. But the reviews for charms that support this are filed:

https://code.launchpad.net/~gandelman-a/charms/precise/nova-cloud-controller/https_endpoint/+merge/150387
https://code.launchpad.net/~gandelman-a/charms/precise/cinder/https_endpoint/+merge/150383
https://code.launchpad.net/~gandelman-a/charms/precise/glance/https_endpoint/+merge/150382
https://code.launchpad.net/~gandelman-a/charms/precise/nova-compute/https_endpoint/+merge/150381

[Adam Gandelman (gandelman-a)]

Also:

- This allows HTTPS to be turned on/off at will for the entire catalog via the config setting in keystone, if all services are using KS-managed SSL certs/keys.

- I should not that these changes do not yet take care of HTTPS for the actual keystone endpoint (OS_AUTH_URL). Only for the corresponding services in the catalog. If the approach proposed here works for other services, it should be easy to port to the keystone charm and have it manage its own endpoint similarly.

- After enabling HTTPs, many client tools will fail SSL verification and fail to interact with the API servers. Clients need to add the CA cert to their local system, eg:

$ curl http://$NOVA_CC_HOST/keystone_juju_ca_cert.crt | sudo tee /usr/local/share/ca-certificates/ks.crt && sudo update-ca-certificates

[James Page (james-page)]

Tested OK!

review: Approve

[lp:~gandelman-a/charms/precise/keystone/ssl_file_sync]

60. By Adam Gandelman on 2013-03-08

Rebase against current ha-support branch.

=== modified file 'config.yaml'
--- config.yaml	2013-01-22 17:49:36 +0000
+++ config.yaml	2013-03-08 21:36:24 +0000
@@ -112,3 +112,7 @@
     default: "false"
     type: string
     description: "Enable PKI token signing (Grizzly and beyond)"
+  https-service-endpoints:
+    default: "False"
+    type: string
+    description: "Manage SSL certificates for all service endpoints."
 
=== added symlink 'hooks/cluster-relation-joined'
=== target is u'keystone-hooks'
=== modified file 'hooks/keystone-hooks'
--- hooks/keystone-hooks	2013-02-22 19:20:54 +0000
+++ hooks/keystone-hooks	2013-03-08 21:36:24 +0000
@@ -2,13 +2,18 @@
 
 import sys
 import time
+import urlparse
+
+from base64 import b64encode
 
 from utils import *
+
 from lib.openstack_common import *
+import lib.unison as unison
 
 config = config_get()
 
-packages = "keystone python-mysqldb pwgen haproxy python-jinja2"
+packages = "keystone python-mysqldb pwgen haproxy python-jinja2 openssl unison"
 service = "keystone"
 
 # used to verify joined services are valid openstack components.
@@ -81,6 +86,11 @@
     execute("service keystone stop", echo=True)
     execute("keystone-manage db_sync")
     execute("service keystone start", echo=True)
+
+    # ensure /var/lib/keystone is g+wrx for peer relations that
+    # may be syncing data there via SSH_USER.
+    execute("chmod -R g+wrx /var/lib/keystone/")
+
     time.sleep(5)
     ensure_initial_admin(config)
 
@@ -97,6 +107,7 @@
         'db_host' not in relation_data):
         juju_log("db_host or password not set. Peer not ready, exit 0")
         exit(0)
+
     update_config_block('sql', connection="mysql://%s:%s@%s/%s" %
                             (config["database-user"],
                              relation_data["password"],
@@ -125,6 +136,21 @@
                      (id, unit))
             identity_changed(relation_id=id, remote_unit=unit)
 
+def ensure_valid_service(service):
+    if service not in valid_services.keys():
+        juju_log("WARN: Invalid service requested: '%s'" % service)
+        realtion_set({ "admin_token": -1 })
+        return
+
+def add_endpoint(region, service, public_url, admin_url, internal_url):
+    desc = valid_services[service]["desc"]
+    service_type = valid_services[service]["type"]
+    create_service_entry(service, service_type, desc)
+    create_endpoint_template(region=region, service=service,
+                             public_url=public_url,
+                             admin_url=admin_url,
+                             internal_url=internal_url)
+
 def identity_joined():
     """ Do nothing until we get information about requested service """
     pass
@@ -170,7 +196,6 @@
                   'internal_url'])
     if single.issubset(settings):
         # other end of relation advertised only one endpoint
-
         if 'None' in [v for k,v in settings.iteritems()]:
             # Some backend services advertise no endpoint but require a
             # hook execution to update auth strategy.
@@ -191,11 +216,14 @@
 
 
         ensure_valid_service(settings['service'])
+
         add_endpoint(region=settings['region'], service=settings['service'],
                      publicurl=settings['public_url'],
                      adminurl=settings['admin_url'],
                      internalurl=settings['internal_url'])
         service_username = settings['service']
+        https_cn = urlparse.urlparse(settings['internal_url'])
+        https_cn = https_cn.hostname
     else:
         # assemble multiple endpoints from relation data. service name
         # should be prepended to setting name, ie:
@@ -217,10 +245,11 @@
         for k,v in settings.iteritems():
             ep = k.split('_')[0]
             x = '_'.join(k.split('_')[1:])
-            if ep not in  endpoints:
+            if ep not in endpoints:
                 endpoints[ep] = {}
             endpoints[ep][x] = v
         services = []
+        https_cn = None
         for ep in endpoints:
             # weed out any unrelated relation stuff Juju might have added
             # by ensuring each possible endpiont has appropriate fields
@@ -233,6 +262,9 @@
                              adminurl=ep['admin_url'],
                              internalurl=ep['internal_url'])
                 services.append(ep['service'])
+                if not https_cn:
+                    https_cn = urlparse.urlparse(ep['internal_url'])
+                    https_cn = https_cn.hostname
         service_username = '_'.join(services)
 
     if 'None' in [v for k,v in settings.iteritems()]:
@@ -262,8 +294,16 @@
         "auth_port": config["admin-port"],
         "service_username": service_username,
         "service_password": service_password,
-        "service_tenant": config['service-tenant']
+        "service_tenant": config['service-tenant'],
+        "https_keystone": "False",
+        "ssl_cert": "",
+        "ssl_key": "",
+        "ca_cert": ""
     }
+
+    if relation_id:
+        relation_data['rid'] = relation_id
+
     # Check if clustered and use vip + haproxy ports if so
     if is_clustered():
         relation_data["auth_host"] = config['vip']
@@ -271,7 +311,19 @@
         relation_data["service_host"] = config['vip']
         relation_data["service_port"] = SERVICE_PORTS['keystone_service']
 
-    relation_set(relation_data)
+    # generate or get a new cert/key for service if set to manage certs.
+    if config['https-service-endpoints'] in ['True', 'true']:
+        ca = get_ca(user=SSH_USER)
+        service = os.getenv('JUJU_REMOTE_UNIT').split('/')[0]
+        cert, key = ca.get_cert_and_key(common_name=https_cn)
+        ca_bundle= ca.get_ca_bundle()
+        relation_data['ssl_cert'] = b64encode(cert)
+        relation_data['ssl_key'] = b64encode(key)
+        relation_data['ca_cert'] = b64encode(ca_bundle)
+        relation_data['https_keystone'] = 'True'
+        unison.sync_to_peers(peer_interface='cluster',
+                             paths=[SSL_DIR], user=SSH_USER, verbose=True)
+    relation_set_2(**relation_data)
     synchronize_service_credentials()
 
 def config_changed():
@@ -318,12 +370,24 @@
     "keystone_service": int(config['service-port']) + 1
     }
 
+def cluster_joined():
+    unison.ssh_authorized_peers(user=SSH_USER,
+                                group='keystone',
+                                peer_interface='cluster',
+                                ensure_user=True)
 
 def cluster_changed():
+    unison.ssh_authorized_peers(user=SSH_USER,
+                                   group='keystone',
+                                   peer_interface='cluster',
+                                   ensure_user=True)
     cluster_hosts = {}
     cluster_hosts['self'] = config['hostname']
     for r_id in relation_ids('cluster'):
         for unit in relation_list(r_id):
+            # trigger identity-changed to reconfigure HTTPS
+            # as necessary.
+            identity_changed(relation_id=r_id, remote_unit=unit)
             cluster_hosts[unit.replace('/','-')] = \
                 relation_get_dict(relation_id=r_id,
                                   remote_unit=unit)['private-address']
@@ -332,6 +396,10 @@
 
     synchronize_service_credentials()
 
+    for r_id in relation_ids('identity-service'):
+        for unit in relation_list(r_id):
+            # trigger identity-changed to reconfigure HTTPS as necessary
+            identity_changed(relation_id=r_id, remote_unit=unit)
 
 def ha_relation_changed():
     relation_data = relation_get_dict()
@@ -400,6 +468,7 @@
     "identity-service-relation-joined": identity_joined,
     "identity-service-relation-changed": identity_changed,
     "config-changed": config_changed,
+    "cluster-relation-joined": cluster_joined,
     "cluster-relation-changed": cluster_changed,
     "cluster-relation-departed": cluster_changed,
     "ha-relation-joined": ha_relation_joined,
 
=== added file 'hooks/keystone_ssl.py'
--- hooks/keystone_ssl.py	1970-01-01 00:00:00 +0000
+++ hooks/keystone_ssl.py	2013-03-08 21:36:24 +0000
@@ -0,0 +1,298 @@
+#!/usr/bin/python
+
+import base64
+import os
+import shutil
+import subprocess
+import tarfile
+import tempfile
+from utils import *
+
+CA_EXPIRY = '365'
+ORG_NAME = 'Ubuntu'
+ORG_UNIT = 'Ubuntu Cloud'
+CA_BUNDLE='/usr/local/share/ca-certificates/juju_ca_cert.crt'
+
+CA_CONFIG = """
+[ ca ]
+default_ca = CA_default
+
+[ CA_default ]
+dir                     = %(ca_dir)s
+policy                  = policy_match
+database                = $dir/index.txt
+serial                  = $dir/serial
+certs                   = $dir/certs
+crl_dir                 = $dir/crl
+new_certs_dir           = $dir/newcerts
+certificate             = $dir/cacert.pem
+private_key             = $dir/private/cacert.key
+RANDFILE                = $dir/private/.rand
+default_md              = default
+
+[ req ]
+default_bits            = 1024
+default_md              = sha1
+
+prompt                  = no
+distinguished_name      = ca_distinguished_name
+
+x509_extensions         = ca_extensions
+
+[ ca_distinguished_name ]
+organizationName        = %(org_name)s
+organizationalUnitName  = %(org_unit_name)s Certificate Authority
+commonName              = %(common_name)s
+
+[ policy_match ]
+countryName             = optional
+stateOrProvinceName     = optional
+organizationName        = match
+organizationalUnitName  = optional
+commonName              = supplied
+
+[ ca_extensions ]
+basicConstraints        = critical,CA:true
+subjectKeyIdentifier    = hash
+authorityKeyIdentifier  = keyid:always, issuer
+keyUsage                = cRLSign, keyCertSign
+"""
+
+SIGNING_CONFIG="""
+[ ca ]
+default_ca = CA_default
+
+[ CA_default ]
+dir                     = %(ca_dir)s
+policy                  = policy_match
+database                = $dir/index.txt
+serial                  = $dir/serial
+certs                   = $dir/certs
+crl_dir                 = $dir/crl
+new_certs_dir           = $dir/newcerts
+certificate             = $dir/cacert.pem
+private_key             = $dir/private/cacert.key
+RANDFILE                = $dir/private/.rand
+default_md              = default
+
+[ req ]
+default_bits            = 1024
+default_md              = sha1
+
+prompt                  = no
+distinguished_name      = req_distinguished_name
+
+x509_extensions         = req_extensions
+
+[ req_distinguished_name ]
+organizationName        = %(org_name)s
+organizationalUnitName  = %(org_unit_name)s Server Farm
+
+[ policy_match ]
+countryName             = optional
+stateOrProvinceName     = optional
+organizationName        = match
+organizationalUnitName  = optional
+commonName              = supplied
+
+[ req_extensions ]
+basicConstraints        = CA:false
+subjectKeyIdentifier    = hash
+authorityKeyIdentifier  = keyid:always, issuer
+keyUsage                = digitalSignature, keyEncipherment, keyAgreement
+extendedKeyUsage        = serverAuth, clientAuth
+"""
+
+
+def init_ca(ca_dir, common_name, org_name=ORG_NAME, org_unit_name=ORG_UNIT):
+    print 'Ensuring certificate authority exists at %s.' % ca_dir
+    if not os.path.exists(ca_dir):
+        print 'Initializing new certificate authority at %s' % ca_dir
+        os.mkdir(ca_dir)
+
+    for i in ['certs', 'crl', 'newcerts', 'private']:
+        d = os.path.join(ca_dir, i)
+        if not os.path.exists(d):
+            print 'Creating %s.' % d
+            os.mkdir(d)
+    os.chmod(os.path.join(ca_dir, 'private'), 0710)
+
+    if not os.path.isfile(os.path.join(ca_dir, 'serial')):
+        with open(os.path.join(ca_dir, 'serial'), 'wb') as out:
+            out.write('01\n')
+
+    if not os.path.isfile(os.path.join(ca_dir, 'index.txt')):
+        with open(os.path.join(ca_dir, 'index.txt'), 'wb') as out:
+            out.write('')
+    if not os.path.isfile(os.path.join(ca_dir, 'ca.cnf')):
+        print 'Creating new CA config in %s' % ca_dir
+        with open(os.path.join(ca_dir, 'ca.cnf'), 'wb') as out:
+            out.write(CA_CONFIG % locals())
+
+
+def root_ca_crt_key(ca_dir):
+    init = False
+    crt = os.path.join(ca_dir, 'cacert.pem')
+    key = os.path.join(ca_dir, 'private', 'cacert.key')
+    for f in [crt, key]:
+        if not os.path.isfile(f):
+            print 'Missing %s, will re-initialize cert+key.' % f
+            init = True
+        else:
+            print 'Found %s.' % f
+    if init:
+        cmd = ['openssl', 'req', '-config', os.path.join(ca_dir, 'ca.cnf'), '-x509',
+               '-nodes', '-newkey', 'rsa', '-days', '21360', '-keyout', key,
+               '-out', crt, '-outform', 'PEM']
+        subprocess.check_call(cmd)
+    return crt, key
+
+
+def intermediate_ca_csr_key(ca_dir):
+    print 'Creating new intermediate CSR.'
+    key = os.path.join(ca_dir, 'private', 'cacert.key')
+    csr = os.path.join(ca_dir, 'cacert.csr')
+    cmd = ['openssl', 'req', '-config', os.path.join(ca_dir, 'ca.cnf'), '-sha1',
+           '-newkey', 'rsa', '-nodes', '-keyout', key, '-out', csr, '-outform',
+           'PEM']
+    subprocess.check_call(cmd)
+    return csr, key
+
+
+def sign_int_csr(ca_dir, csr, common_name):
+    print 'Signing certificate request %s.' % csr
+    crt = os.path.join(ca_dir, 'certs',
+                        '%s.crt' % os.path.basename(csr).split('.')[0])
+    subj = '/O=%s/OU=%s/CN=%s' % (ORG_NAME, ORG_UNIT, common_name)
+    cmd = ['openssl', 'ca', '-batch', '-config', os.path.join(ca_dir, 'ca.cnf'),
+           '-extensions', 'ca_extensions', '-days', CA_EXPIRY, '-notext',
+           '-in', csr, '-out', crt, '-subj', subj, '-batch']
+    print ' '.join(cmd)
+    subprocess.check_call(cmd)
+    return crt
+
+
+def init_root_ca(ca_dir, common_name):
+    init_ca(ca_dir, common_name)
+    return root_ca_crt_key(ca_dir)
+
+
+def init_intermediate_ca(ca_dir, common_name, root_ca_dir,
+                         org_name=ORG_NAME, org_unit_name=ORG_UNIT):
+    init_ca(ca_dir, common_name)
+    if not os.path.isfile(os.path.join(ca_dir, 'cacert.pem')):
+        csr, key = intermediate_ca_csr_key(ca_dir)
+        crt = sign_int_csr(root_ca_dir, csr, common_name)
+        shutil.copy(crt, os.path.join(ca_dir, 'cacert.pem'))
+    else:
+        print 'Intermediate CA certificate already exists.'
+
+    if not os.path.isfile(os.path.join(ca_dir, 'signing.cnf')):
+        print 'Creating new signing config in %s' % ca_dir
+        with open(os.path.join(ca_dir, 'signing.cnf'), 'wb') as out:
+            out.write(SIGNING_CONFIG % locals())
+
+
+def create_certificate(ca_dir, service):
+    common_name = service
+    subj = '/O=%s/OU=%s/CN=%s' % (ORG_NAME, ORG_UNIT, common_name)
+    csr = os.path.join(ca_dir, 'certs', '%s.csr' % service)
+    key = os.path.join(ca_dir, 'certs', '%s.key' % service)
+    cmd = ['openssl', 'req', '-sha1', '-newkey', 'rsa', '-nodes', '-keyout',
+           key, '-out', csr, '-subj', subj]
+    subprocess.check_call(cmd)
+    crt = sign_csr(ca_dir, csr, common_name)
+    print 'Signed new CSR, crt @ %s' % crt
+    return
+
+def update_bundle(bundle_file, new_bundle):
+    return
+    if os.path.isfile(bundle_file):
+        current = open(bundle_file, 'r').read().strip()
+        if new_bundle == current:
+            print 'CA Bundle @ %s is up to date.' % bundle_file
+            return
+        else:
+            print 'Updating CA bundle @ %s.' % bundle_file
+
+    with open(bundle_file, 'wb') as out:
+        out.write(new_bundle)
+    subprocess.check_call(['update-ca-certificates'])
+
+def tar_directory(path):
+    cwd = os.getcwd()
+    parent=os.path.dirname(path)
+    directory=os.path.basename(path)
+    tmp = tempfile.TemporaryFile()
+    os.chdir(parent)
+    tarball = tarfile.TarFile(fileobj=tmp, mode='w')
+    tarball.add(directory)
+    tarball.close()
+    tmp.seek(0)
+    out = tmp.read()
+    tmp.close()
+    os.chdir(cwd)
+    return out
+
+class JujuCA(object):
+    def __init__(self, name, ca_dir, root_ca_dir, user, group):
+        root_crt, root_key = init_root_ca(root_ca_dir,
+                                          '%s Certificate Authority' % name)
+        init_intermediate_ca(ca_dir,
+                             '%s Intermediate Certificate Authority' % name,
+                             root_ca_dir)
+        cmd = ['chown', '-R', '%s.%s' % (user, group), ca_dir]
+        subprocess.check_call(cmd)
+        cmd = ['chown', '-R', '%s.%s' % (user, group), root_ca_dir]
+        subprocess.check_call(cmd)
+        self.ca_dir = ca_dir
+        self.root_ca_dir = root_ca_dir
+        self.user = user
+        self.group = group
+        update_bundle(CA_BUNDLE, self.get_ca_bundle())
+
+    def _sign_csr(self, csr, service, common_name):
+        subj = '/O=%s/OU=%s/CN=%s' % (ORG_NAME, ORG_UNIT, common_name)
+        crt = os.path.join(self.ca_dir, 'certs', '%s.crt' % common_name)
+        cmd = ['openssl', 'ca', '-config',
+               os.path.join(self.ca_dir, 'signing.cnf'), '-extensions',
+               'req_extensions', '-days', '365', '-notext', '-in', csr,
+               '-out', crt, '-batch', '-subj', subj]
+        subprocess.check_call(cmd)
+        return crt
+
+    def _create_certificate(self, service, common_name):
+        subj = '/O=%s/OU=%s/CN=%s' % (ORG_NAME, ORG_UNIT, common_name)
+        csr = os.path.join(self.ca_dir, 'certs', '%s.csr' % service)
+        key = os.path.join(self.ca_dir, 'certs', '%s.key' % service)
+        cmd = ['openssl', 'req', '-sha1', '-newkey', 'rsa', '-nodes', '-keyout',
+               key, '-out', csr, '-subj', subj]
+        subprocess.check_call(cmd)
+        crt = self._sign_csr(csr, service, common_name)
+        cmd = ['chown', '-R', '%s.%s' % (self.user, self.group), self.ca_dir]
+        subprocess.check_call(cmd)
+        print 'Signed new CSR, crt @ %s' % crt
+        return crt, key
+
+    def get_cert_and_key(self, common_name):
+        print 'Getting certificate and key for %s.' % common_name
+        key = os.path.join(self.ca_dir, 'certs', '%s.key' % common_name)
+        crt = os.path.join(self.ca_dir, 'certs', '%s.crt' % common_name)
+        if os.path.isfile(crt):
+            print 'Found existing certificate for %s.' % common_name
+            crt = open(crt, 'r').read()
+            try:
+                key = open(key, 'r').read()
+            except:
+                print 'Could not load ssl private key for %s from %s' %\
+                     (common_name, key)
+                exit(1)
+            return crt, key
+        crt, key = self._create_certificate(common_name, common_name)
+        return open(crt, 'r').read(), open(key, 'r').read()
+
+    def get_ca_bundle(self):
+        int_cert = open(os.path.join(self.ca_dir, 'cacert.pem')).read()
+        root_cert = open(os.path.join(self.root_ca_dir, 'cacert.pem')).read()
+        # NOTE: ordering of certs in bundle matters!
+        return int_cert + root_cert
 
=== added file 'hooks/lib/unison.py'
--- hooks/lib/unison.py	1970-01-01 00:00:00 +0000
+++ hooks/lib/unison.py	2013-03-08 21:36:24 +0000
@@ -0,0 +1,215 @@
+#!/usr/bin/python
+#
+# Easy file synchronization among peer units using ssh + unison.
+#
+# From *both* peer relation -joined and -changed, add a call to
+# ssh_authorized_peers() describing the peer relation and the desired
+# user + group.  After all peer relations have settled, all hosts should
+# be able to connect to on another via key auth'd ssh as the specified user.
+#
+# Other hooks are then free to synchronize files and directories using
+# sync_to_peers().
+#
+# For a peer relation named 'cluster', for example:
+#
+# cluster-relation-joined:
+# ...
+# ssh_authorized_peers(peer_interface='cluster',
+#                      user='juju_ssh', group='juju_ssh',
+#                      ensure_user=True)
+# ...
+#
+# cluster-relation-changed:
+# ...
+# ssh_authorized_peers(peer_interface='cluster',
+#                      user='juju_ssh', group='juju_ssh',
+#                      ensure_user=True)
+# ...
+#
+# Hooks are now free to sync files as easily as:
+#
+# files = ['/etc/fstab', '/etc/apt.conf.d/']
+# sync_to_peers(peer_interface='cluster',
+#                user='juju_ssh, paths=[files])
+#
+# It is assumed the charm itself has setup permissions on each unit
+# such that 'juju_ssh' has read + write permissions.  Also assumed
+# that the calling charm takes care of leader delegation.
+#
+# TODO: Currently depends on the utils.py shipped with the keystone charm.
+#       Either copy required functionality to this library or depend on
+#       something more generic.
+
+import json
+import os
+import sys
+import utils
+import subprocess
+import shutil
+import grp
+import pwd
+
+
+def get_homedir(user):
+    try:
+        user = pwd.getpwnam(user)
+        return user.pw_dir
+    except KeyError:
+        utils.juju_log('Could not get homedir for user %s: user exists?')
+        sys.exit(1)
+
+
+def get_keypair(user):
+    home_dir = get_homedir(user)
+    ssh_dir = os.path.join(home_dir, '.ssh')
+    if not os.path.isdir(ssh_dir):
+        os.mkdir(ssh_dir)
+
+    priv_key = os.path.join(ssh_dir, 'id_rsa')
+    if not os.path.isfile(priv_key):
+        utils.juju_log('Generating new ssh key for user %s.' % user)
+        cmd = ['ssh-keygen', '-q', '-N', '', '-t', 'rsa', '-b', '2048',
+               '-f', priv_key]
+        subprocess.check_call(cmd)
+
+    pub_key = '%s.pub' % priv_key
+    if not os.path.isfile(pub_key):
+        utils.juju_log('Generatring missing ssh public key @ %s.' % pub_key)
+        cmd = ['ssh-keygen', '-y', '-f', priv_key]
+        p = subprocess.check_output(cmd).strip()
+        with open(pub_key, 'wb') as out:
+            out.write(p)
+    subprocess.check_call(['chown', '-R', user, ssh_dir])
+    return open(priv_key, 'r').read().strip(), open(pub_key, 'r').read().strip()
+
+
+def write_authorized_keys(user, keys):
+    home_dir = get_homedir(user)
+    ssh_dir = os.path.join(home_dir, '.ssh')
+    auth_keys = os.path.join(ssh_dir, 'authorized_keys')
+    utils.juju_log('Syncing authorized_keys @ %s.' % auth_keys)
+    with open(auth_keys, 'wb') as out:
+        for k in keys:
+            out.write('%s\n' % k)
+
+
+def write_known_hosts(user, hosts):
+    home_dir = get_homedir(user)
+    ssh_dir = os.path.join(home_dir, '.ssh')
+    known_hosts  = os.path.join(ssh_dir, 'known_hosts')
+    khosts = []
+    for host in hosts:
+        cmd = ['ssh-keyscan', '-H', '-t', 'rsa', host]
+        remote_key = subprocess.check_output(cmd).strip()
+        khosts.append(remote_key)
+    utils.juju_log('Syncing known_hosts @ %s.' % known_hosts)
+    with open(known_hosts, 'wb') as out:
+        for host in khosts:
+            out.write('%s\n' % host)
+
+
+def _ensure_user(user, group=None):
+    # need to ensure a bash shell'd user exists.
+    try:
+        pwd.getpwnam(user)
+    except KeyError:
+        utils.juju_log('Creating new user %s.%s.' % (user, group))
+        cmd = ['adduser', '--system', '--shell', '/bin/bash', user]
+        if group:
+            try:
+                grp.getgrnam(group)
+            except KeyError:
+                subprocess.check_call(['addgroup', group])
+            cmd += ['--ingroup', group]
+        subprocess.check_call(cmd)
+
+
+def ssh_authorized_peers(peer_interface, user, group=None, ensure_user=False):
+    """
+    Main setup function, should be called from both peer -changed and -joined
+    hooks with the same parameters.
+    """
+    if ensure_user:
+        _ensure_user(user, group)
+    priv_key, pub_key = get_keypair(user)
+    hook = os.path.basename(sys.argv[0])
+    if hook == '%s-relation-joined' % peer_interface:
+        utils.relation_set_2(ssh_pub_key=pub_key)
+        print 'joined'
+    elif hook == '%s-relation-changed' % peer_interface:
+        hosts = []
+        keys = []
+        for r_id in utils.relation_ids(peer_interface):
+            for unit in utils.relation_list(r_id):
+                settings = utils.relation_get_dict(relation_id=r_id,
+                                                   remote_unit=unit)
+                if 'ssh_pub_key' in settings:
+                    keys.append(settings['ssh_pub_key'])
+                    hosts.append(settings['private-address'])
+                else:
+                    utils.juju_log('ssh_authorized_peers(): ssh_pub_key '\
+                                   'missing for unit %s, skipping.' % unit)
+        write_authorized_keys(user, keys)
+        write_known_hosts(user, hosts)
+        authed_hosts = ':'.join(hosts)
+        utils.relation_set_2(ssh_authorized_hosts=authed_hosts)
+
+
+def _run_as_user(user):
+    try:
+        user = pwd.getpwnam(user)
+    except KeyError:
+        utils.juju_log('Invalid user: %s' % user)
+        sys.exit(1)
+    uid, gid = user.pw_uid, user.pw_gid
+    os.environ['HOME'] = user.pw_dir
+    def _inner():
+        os.setgid(gid)
+        os.setuid(uid)
+    return _inner
+
+def run_as_user(user, cmd):
+    return subprocess.check_output(cmd, preexec_fn=_run_as_user(user))
+
+def sync_to_peers(peer_interface, user, paths=[], verbose=False):
+    base_cmd = ['unison', '-auto', '-batch=true', '-confirmbigdel=false',
+                '-fastcheck=true', '-group=false', '-owner=false', '-prefer=newer',
+                '-times=true']
+    if not verbose:
+        base_cmd.append('-silent')
+
+    hosts = []
+    for r_id in (utils.relation_ids(peer_interface) or []):
+        for unit in utils.relation_list(r_id):
+            settings = utils.relation_get_dict(relation_id=r_id,
+                                               remote_unit=unit)
+            try:
+                authed_hosts = settings['ssh_authorized_hosts'].split(':')
+            except KeyError:
+                print 'unison sync_to_peers: peer has not authorized *any* '\
+                      'hosts yet.'
+                return
+
+            unit_hostname = utils.unit_get('private-address')
+            add_host = None
+            for authed_host in authed_hosts:
+                if unit_hostname == authed_host:
+                    add_host = settings['private-address']
+            if add_host:
+                hosts.append(settings['private-address'])
+            else:
+                print 'unison sync_to_peers: peer (%s) has not authorized '\
+                      '*this* host yet, skipping.' %\
+                       settings['private-address']
+
+    for path in paths:
+        # removing trailing slash from directory paths, unison
+        # doesn't like these.
+        if path.endswith('/'):
+            path = path[:(len(path)-1)]
+        for host in hosts:
+            cmd = base_cmd + [path, 'ssh://%s@%s/%s' % (user, host, path)]
+            utils.juju_log('Syncing local path %s to %s@%s:%s' %\
+                            (path, user, host, path))
+            print ' '.join(cmd)
+            run_as_user(user, cmd)
 
=== modified file 'hooks/utils.py'
--- hooks/utils.py	2013-02-22 19:20:54 +0000
+++ hooks/utils.py	2013-03-08 21:36:24 +0000
@@ -4,15 +4,25 @@
 import sys
 import json
 import os
+import tarfile
+import tempfile
 import time
 
 from lib.openstack_common import *
 
+import keystone_ssl as ssl
+import lib.unison as unison
+
 keystone_conf = "/etc/keystone/keystone.conf"
 stored_passwd = "/var/lib/keystone/keystone.passwd"
 stored_token = "/var/lib/keystone/keystone.token"
 SERVICE_PASSWD_PATH = '/var/lib/keystone/services.passwd'
 
+SSL_DIR = '/var/lib/keystone/juju_ssl/'
+SSL_CA_NAME = 'Ubuntu Cloud'
+
+SSH_USER='juju_keystone'
+
 def execute(cmd, die=False, echo=False):
     """ Executes a command
 
@@ -95,6 +105,19 @@
     cmd += args
     subprocess.check_call(cmd)
 
+
+def unit_get(attribute):
+    cmd = [
+        'unit-get',
+        attribute
+        ]
+    value = subprocess.check_output(cmd).strip()  # IGNORE:E1103
+    if value == "":
+        return None
+    else:
+        return value
+
+
 def relation_get(relation_data):
     """ Obtain all current relation data
     relation_data is a list of options to query from the relation
@@ -356,8 +379,7 @@
     create_role("KeystoneAdmin", config["admin-user"], 'admin')
     create_role("KeystoneServiceAdmin", config["admin-user"], 'admin')
     create_service_entry("keystone", "identity", "Keystone Identity Service")
-    # following documentation here, perhaps we should be using juju
-    # public/private addresses for public/internal urls.
+
     if is_clustered():
         juju_log("Creating endpoint for clustered configuration")
         for region in config['region'].split():
@@ -543,17 +565,32 @@
     Broadcast service credentials to peers or consume those that have been
     broadcasted by peer, depending on hook context.
     '''
-    if os.path.basename(sys.argv[0]) == 'cluster-relation-changed':
-        r_data = relation_get_dict()
-        if 'service_credentials' in r_data:
-            juju_log('Saving service passwords from peer.')
-            save_stored_passwords(**json.loads(r_data['service_credentials']))
-        return
-
-    creds = load_stored_passwords()
-    if not creds:
+    if (not eligible_leader() or
+        not os.path.isfile(SERVICE_PASSWD_PATH)):
         return
     juju_log('Synchronizing service passwords to all peers.')
-    creds = json.dumps(creds)
-    for r_id in (relation_ids('cluster') or []):
-        relation_set_2(rid=r_id, service_credentials=creds)
+    unison.sync_to_peers(peer_interface='cluster',
+                         paths=[SERVICE_PASSWD_PATH], user=SSH_USER,
+                         verbose=True)
+
+CA = []
+def get_ca(user='keystone', group='keystone'):
+    """
+    Initialize a new CA object if one hasn't already been loaded.
+    This will create a new CA or load an existing one.
+    """
+    if not CA:
+        if not os.path.isdir(SSL_DIR):
+            os.mkdir(SSL_DIR)
+        d_name = '_'.join(SSL_CA_NAME.lower().split(' '))
+        ca = ssl.JujuCA(name=SSL_CA_NAME, user=user, group=group,
+                        ca_dir=os.path.join(SSL_DIR,
+                                            '%s_intermediate_ca' % d_name),
+                        root_ca_dir=os.path.join(SSL_DIR,
+                                            '%s_root_ca' % d_name))
+        # SSL_DIR is synchronized via all peers over unison+ssh, need
+        # to ensure permissions.
+        execute('chown -R %s.%s %s' % (user, group, SSL_DIR))
+        execute('chmod -R g+rwx %s' % SSL_DIR)
+        CA.append(ca)
+    return CA[0]
 
=== modified file 'revision'
--- revision	2013-03-06 21:50:30 +0000
+++ revision	2013-03-08 21:36:24 +0000
@@ -1,1 +1,1 @@
-198
+211