Merge lp:~james-page/charms/precise/hacluster/redux into lp:charms/hacluster

[James Page]

Redux ready for final submission (hopefully)

1) Refactored to use HA openstack-charm-helpers reducing charm specific code

2) 'ready' barrier on peer relation added

Each service-unit set a 'ready' flag on the peer relationship when its received enough information to configure corosync and pacemaker; service units are only counted as valid once they have set this.

This ensures that nodes that don't have enough config info don't get counted when assessing clusterable size

3) oldest peer configures services

This is to avoid a split brain when initially forming the cluster; the oldest peer will most likely be busy dealing with other service relations, whilst the other service units start getting ready to be clustered during early service archive

By ensuring that the oldest peer configures the actual services, we smooth the transition from peering->clustered and avoid one of the newer service units trying to take control before the cluster is fully formed.

4) configurable initial cluster size

Default to 2 units; but should be set to the target size for the initial cluster.

5) default to corosync/pacemake v1 integration

This is as recommended upstream and splits the control of the corosync and pacemaker daemons.

[Andres Rodriguez (andreserl)]

The branch looks good to me. However, I believe we need to make sure the pacemaker initscript is being set to defaults and that starts after corosync since it is disabled in packaging.

[Andres Rodriguez (andreserl)]

review: Approve

=== added file '.project'
--- .project	1970-01-01 00:00:00 +0000
+++ .project	2013-03-25 14:21:21 +0000
@@ -0,0 +1,17 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<projectDescription>
+	<name>hacluster</name>
+	<comment></comment>
+	<projects>
+	</projects>
+	<buildSpec>
+		<buildCommand>
+			<name>org.python.pydev.PyDevBuilder</name>
+			<arguments>
+			</arguments>
+		</buildCommand>
+	</buildSpec>
+	<natures>
+		<nature>org.python.pydev.pythonNature</nature>
+	</natures>
+</projectDescription>

=== added file '.pydevproject'
--- .pydevproject	1970-01-01 00:00:00 +0000
+++ .pydevproject	2013-03-25 14:21:21 +0000
@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<?eclipse-pydev version="1.0"?><pydev_project>
+<pydev_property name="org.python.pydev.PYTHON_PROJECT_VERSION">python 2.7</pydev_property>
+<pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">Default</pydev_property>
+<pydev_pathproperty name="org.python.pydev.PROJECT_SOURCE_PATH">
+<path>/hacluster/hooks</path>
+</pydev_pathproperty>
+</pydev_project>

=== modified file 'config.yaml'
--- config.yaml	2013-01-24 23:39:45 +0000
+++ config.yaml	2013-03-25 14:21:21 +0000
@@ -7,22 +7,23 @@
       If multiple clusters are on the same bindnetaddr network, this value
       can be changed.
   corosync_pcmk_ver:
-    default: 0
+    default: 1
     type: int
     description: |
       Service version for the Pacemaker service version. This will tell
       Corosync how to start pacemaker
   corosync_key:
     type: string
-    default: corosync-key
+    default: "64RxJNcCkwo8EJYBsaacitUvbQp5AW4YolJi5/2urYZYp2jfLxY+3IUCOaAUJHPle4Yqfy+WBXO0I/6ASSAjj9jaiHVNaxmVhhjcmyBqy2vtPf+m+0VxVjUXlkTyYsODwobeDdO3SIkbIABGfjLTu29yqPTsfbvSYr6skRb9ne0="
     description: |
       This value will become the Corosync authentication key. To generate
       a suitable value use:
       .
-        corosync-keygen
+        sudo corosync-keygen
+        sudo cat /etc/corosync/authkey | base64 -w 0
       .
       This configuration element is mandatory and the service will fail on
-      install if it is not provided.
+      install if it is not provided.  The value must be base64 encoded.
   stonith_enabled:
     type: string
     default: 'False'
@@ -36,3 +37,7 @@
   maas_credentials:
     type: string
     description: MAAS credentials (required for STONITH).
+  cluster_count:
+    type: int
+    default: 2
+    description: Number of peer units required to bootstrap cluster services.

=== modified file 'copyright'
--- copyright	2012-11-20 20:06:11 +0000
+++ copyright	2013-03-25 14:21:21 +0000
@@ -15,3 +15,8 @@
  .
  You should have received a copy of the GNU General Public License
  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ 
+ Files: ocf/ceph/*
+ Copyright: 2012 Florian Haas, hastexo 
+ License: LGPL-2.1
+  On Debian based systems, see /usr/share/common-licenses/LGPL-2.1.
\ No newline at end of file

=== removed symlink 'hooks/ha-relation-departed'
=== target was u'hooks.py'
=== renamed file 'hooks/utils.py' => 'hooks/hacluster.py'
--- hooks/utils.py	2013-02-20 10:10:42 +0000
+++ hooks/hacluster.py	2013-03-25 14:21:21 +0000
@@ -10,243 +10,16 @@
 import os
 import subprocess
 import socket
-import sys
 import fcntl
 import struct
-
-
-def do_hooks(hooks):
-    hook = os.path.basename(sys.argv[0])
-
-    try:
-        hooks[hook]()
-    except KeyError:
-        juju_log('INFO',
-                 "This charm doesn't know how to handle '{}'.".format(hook))
-
-
-def install(*pkgs):
-    cmd = [
-        'apt-get',
-        '-y',
-        'install'
-          ]
-    for pkg in pkgs:
-        cmd.append(pkg)
-    subprocess.check_call(cmd)
-
-TEMPLATES_DIR = 'templates'
-
-try:
-    import jinja2
-except ImportError:
-    install('python-jinja2')
-    import jinja2
-
-try:
-    from netaddr import IPNetwork
-except ImportError:
-    install('python-netaddr')
-    from netaddr import IPNetwork
-
-try:
-    import dns.resolver
-except ImportError:
-    install('python-dnspython')
-    import dns.resolver
-
-
-def render_template(template_name, context, template_dir=TEMPLATES_DIR):
-    templates = jinja2.Environment(
-                    loader=jinja2.FileSystemLoader(template_dir)
-                    )
-    template = templates.get_template(template_name)
-    return template.render(context)
-
-
-CLOUD_ARCHIVE = \
-""" # Ubuntu Cloud Archive
-deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main
-"""
-
-CLOUD_ARCHIVE_POCKETS = {
-    'precise-folsom': 'precise-updates/folsom',
-    'precise-folsom/updates': 'precise-updates/folsom',
-    'precise-folsom/proposed': 'precise-proposed/folsom',
-    'precise-grizzly': 'precise-updates/grizzly',
-    'precise-grizzly/updates': 'precise-updates/grizzly',
-    'precise-grizzly/proposed': 'precise-proposed/grizzly'
-    }
-
-
-def configure_source():
-    source = str(config_get('openstack-origin'))
-    if not source:
-        return
-    if source.startswith('ppa:'):
-        cmd = [
-            'add-apt-repository',
-            source
-            ]
-        subprocess.check_call(cmd)
-    if source.startswith('cloud:'):
-        install('ubuntu-cloud-keyring')
-        pocket = source.split(':')[1]
-        with open('/etc/apt/sources.list.d/cloud-archive.list', 'w') as apt:
-            apt.write(CLOUD_ARCHIVE.format(CLOUD_ARCHIVE_POCKETS[pocket]))
-    if source.startswith('deb'):
-        l = len(source.split('|'))
-        if l == 2:
-            (apt_line, key) = source.split('|')
-            cmd = [
-                'apt-key',
-                'adv', '--keyserver keyserver.ubuntu.com',
-                '--recv-keys', key
-                ]
-            subprocess.check_call(cmd)
-        elif l == 1:
-            apt_line = source
-
-        with open('/etc/apt/sources.list.d/quantum.list', 'w') as apt:
-            apt.write(apt_line + "\n")
-    cmd = [
-        'apt-get',
-        'update'
-        ]
-    subprocess.check_call(cmd)
-
-# Protocols
-TCP = 'TCP'
-UDP = 'UDP'
-
-
-def expose(port, protocol='TCP'):
-    cmd = [
-        'open-port',
-        '{}/{}'.format(port, protocol)
-        ]
-    subprocess.check_call(cmd)
-
-
-def juju_log(severity, message):
-    cmd = [
-        'juju-log',
-        '--log-level', severity,
-        message
-        ]
-    subprocess.check_call(cmd)
-
-
-def relation_ids(relation):
-    cmd = [
-        'relation-ids',
-        relation
-        ]
-    return subprocess.check_output(cmd).split()  # IGNORE:E1103
-
-
-def relation_list(rid):
-    cmd = [
-        'relation-list',
-        '-r', rid,
-        ]
-    return subprocess.check_output(cmd).split()  # IGNORE:E1103
-
-
-def relation_get(attribute, unit=None, rid=None):
-    cmd = [
-        'relation-get',
-        ]
-    if rid:
-        cmd.append('-r')
-        cmd.append(rid)
-    cmd.append(attribute)
-    if unit:
-        cmd.append(unit)
-    value = subprocess.check_output(cmd).strip()  # IGNORE:E1103
-    if value == "":
-        return None
-    else:
-        return value
-
-
-def relation_set(**kwargs):
-    cmd = [
-        'relation-set'
-        ]
-    args = []
-    for k, v in kwargs.items():
-        if k == 'rid':
-            cmd.append('-r')
-            cmd.append(v)
-        else:
-            args.append('{}={}'.format(k, v))
-    cmd += args
-    subprocess.check_call(cmd)
-
-
-def unit_get(attribute):
-    cmd = [
-        'unit-get',
-        attribute
-        ]
-    return subprocess.check_output(cmd).strip()  # IGNORE:E1103
-
-
-def config_get(attribute):
-    cmd = [
-        'config-get',
-        attribute
-        ]
-    return subprocess.check_output(cmd).strip()  # IGNORE:E1103
-
-
-def get_unit_hostname():
-    return socket.gethostname()
-
-
-def get_host_ip(hostname=unit_get('private-address')):
-    try:
-        # Test to see if already an IPv4 address
-        socket.inet_aton(hostname)
-        return hostname
-    except socket.error:
-        pass
-    try:
-        answers = dns.resolver.query(hostname, 'A')
-        if answers:
-            return answers[0].address
-    except dns.resolver.NXDOMAIN:
-        pass
-    return None
-
-
-def restart(*services):
-    for service in services:
-        subprocess.check_call(['service', service, 'restart'])
-
-
-def stop(*services):
-    for service in services:
-        subprocess.check_call(['service', service, 'stop'])
-
-
-def start(*services):
-    for service in services:
-        subprocess.check_call(['service', service, 'start'])
-
-
-def running(service):
-    try:
-        output = subprocess.check_output(['service', service, 'status'])
-    except subprocess.CalledProcessError:
-        return False
-    else:
-        if ("start/running" in output or
-            "is running" in output):
-            return True
-        else:
-            return False
+import lib.utils as utils
+
+
+try:
+    from netaddr import IPNetwork
+except ImportError:
+    utils.install('python-netaddr')
+    from netaddr import IPNetwork
 
 
 def disable_upstart_services(*services):

=== added symlink 'hooks/hanode-relation-changed'
=== target is u'hooks.py'
=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2013-03-07 09:37:28 +0000
+++ hooks/hooks.py	2013-03-25 14:21:21 +0000
@@ -11,10 +11,13 @@
 import sys
 import time
 import os
+from base64 import b64decode
 
 import maas as MAAS
-import utils
+import lib.utils as utils
+import lib.cluster_utils as cluster
 import pcmk
+import hacluster
 
 
 def install():
@@ -36,12 +39,12 @@
         for unit in utils.relation_list(relid):
             conf = {
                 'corosync_bindnetaddr':
-                    utils.get_network_address(
+                    hacluster.get_network_address(
                               utils.relation_get('corosync_bindiface',
                                                  unit, relid)
                               ),
                 'corosync_mcastport': utils.relation_get('corosync_mcastport',
-                                        unit, relid),
+                                                         unit, relid),
                 'corosync_mcastaddr': utils.config_get('corosync_mcastaddr'),
                 'corosync_pcmk_ver': utils.config_get('corosync_pcmk_ver'),
                 }
@@ -68,27 +71,27 @@
     with open('/etc/default/corosync', 'w') as corosync_default:
         corosync_default.write(utils.render_template('corosync',
                                                      corosync_default_context))
-
-    # write the authkey
     corosync_key = utils.config_get('corosync_key')
-    with open('/etc/corosync/authkey', 'w') as corosync_key_file:
-        corosync_key_file.write(corosync_key)
-    os.chmod = ('/etc/corosync/authkey', 0400)
+    if corosync_key:
+        # write the authkey
+        with open('/etc/corosync/authkey', 'w') as corosync_key_file:
+            corosync_key_file.write(b64decode(corosync_key))
+        os.chmod = ('/etc/corosync/authkey', 0400)
 
 
 def config_changed():
     utils.juju_log('INFO', 'Begin config-changed hook.')
 
     corosync_key = utils.config_get('corosync_key')
-    if corosync_key == '':
+    if not corosync_key:
         utils.juju_log('CRITICAL',
                        'No Corosync key supplied, cannot proceed')
         sys.exit(1)
 
     if int(utils.config_get('corosync_pcmk_ver')) == 1:
-        utils.enable_lsb_services('pacemaker')
+        hacluster.enable_lsb_services('pacemaker')
     else:
-        utils.disable_lsb_services('pacemaker')
+        hacluster.disable_lsb_services('pacemaker')
 
     # Create a new config file
     emit_base_conf()
@@ -109,14 +112,6 @@
     utils.juju_log('INFO', 'End upgrade-charm hook.')
 
 
-def start():
-    pass
-
-
-def stop():
-    pass
-
-
 def restart_corosync():
     if int(utils.config_get('corosync_pcmk_ver')) == 1:
         if utils.running("pacemaker"):
@@ -136,17 +131,23 @@
         utils.juju_log('INFO',
                        'HA already configured, not reconfiguring')
         return
-    # Check that there's enough nodes in order to perform the
-    # configuration of the HA cluster
-    if len(get_cluster_nodes()) < 2:
-        utils.juju_log('WARNING', 'Not enough nodes in cluster, bailing')
-        return
     # Check that we are related to a principle and that
     # it has already provided the required corosync configuration
     if not get_corosync_conf():
         utils.juju_log('WARNING',
                        'Unable to configure corosync right now, bailing')
         return
+    else:
+        utils.juju_log('INFO',
+                       'Ready to form cluster - informing peers')
+        utils.relation_set(ready=True,
+                           rid=utils.relation_ids('hanode')[0])
+    # Check that there's enough nodes in order to perform the
+    # configuration of the HA cluster
+    if (len(get_cluster_nodes()) <
+        int(utils.config_get('cluster_count'))):
+        utils.juju_log('WARNING', 'Not enough nodes in cluster, bailing')
+        return
 
     relids = utils.relation_ids('ha')
     if len(relids) == 1:  # Should only ever be one of these
@@ -225,98 +226,101 @@
           ' resource-stickiness="100"'
     pcmk.commit(cmd)
 
-    utils.juju_log('INFO', 'Configuring Resources')
-    utils.juju_log('INFO', str(resources))
-
-    for res_name, res_type in resources.iteritems():
-        # disable the service we are going to put in HA
-        if res_type.split(':')[0] == "lsb":
-            utils.disable_lsb_services(res_type.split(':')[1])
-            if utils.running(res_type.split(':')[1]):
-                utils.stop(res_type.split(':')[1])
-        elif (len(init_services) != 0 and
-              res_name in init_services and
-              init_services[res_name]):
-            utils.disable_upstart_services(init_services[res_name])
-            if utils.running(init_services[res_name]):
-                utils.stop(init_services[res_name])
-        # Put the services in HA, if not already done so
-        #if not pcmk.is_resource_present(res_name):
-        if not pcmk.crm_opt_exists(res_name):
-            if not res_name in resource_params:
-                cmd = 'crm -F configure primitive %s %s' % (res_name, res_type)
-            else:
-                cmd = 'crm -F configure primitive %s %s %s' % \
-                            (res_name,
-                             res_type,
-                             resource_params[res_name])
-            pcmk.commit(cmd)
-            utils.juju_log('INFO', '%s' % cmd)
-
-    utils.juju_log('INFO', 'Configuring Groups')
-    utils.juju_log('INFO', str(groups))
-    for grp_name, grp_params in groups.iteritems():
-        if not pcmk.crm_opt_exists(grp_name):
-            cmd = 'crm -F configure group %s %s' % (grp_name, grp_params)
-            pcmk.commit(cmd)
-            utils.juju_log('INFO', '%s' % cmd)
-
-    utils.juju_log('INFO', 'Configuring Master/Slave (ms)')
-    utils.juju_log('INFO', str(ms))
-    for ms_name, ms_params in ms.iteritems():
-        if not pcmk.crm_opt_exists(ms_name):
-            cmd = 'crm -F configure ms %s %s' % (ms_name, ms_params)
-            pcmk.commit(cmd)
-            utils.juju_log('INFO', '%s' % cmd)
-
-    utils.juju_log('INFO', 'Configuring Orders')
-    utils.juju_log('INFO', str(orders))
-    for ord_name, ord_params in orders.iteritems():
-        if not pcmk.crm_opt_exists(ord_name):
-            cmd = 'crm -F configure order %s %s' % (ord_name, ord_params)
-            pcmk.commit(cmd)
-            utils.juju_log('INFO', '%s' % cmd)
-
-    utils.juju_log('INFO', 'Configuring Colocations')
-    utils.juju_log('INFO', str(colocations))
-    for col_name, col_params in colocations.iteritems():
-        if not pcmk.crm_opt_exists(col_name):
-            cmd = 'crm -F configure colocation %s %s' % (col_name, col_params)
-            pcmk.commit(cmd)
-            utils.juju_log('INFO', '%s' % cmd)
-
-    utils.juju_log('INFO', 'Configuring Clones')
-    utils.juju_log('INFO', str(clones))
-    for cln_name, cln_params in clones.iteritems():
-        if not pcmk.crm_opt_exists(cln_name):
-            cmd = 'crm -F configure clone %s %s' % (cln_name, cln_params)
-            pcmk.commit(cmd)
-            utils.juju_log('INFO', '%s' % cmd)
-
-    for res_name, res_type in resources.iteritems():
-        if len(init_services) != 0 and res_name in init_services:
-            # Checks that the resources are running and started.
-            # Ensure that clones are excluded as the resource is
-            # not directly controllable (dealt with below)
-            # Ensure that groups are cleaned up as a whole rather
-            # than as individual resources.
-            if (res_name not in clones.values() and
-                res_name not in groups.values() and
-                not pcmk.crm_res_running(res_name)):
-                # Just in case, cleanup the resources to ensure they get
-                # started in case they failed for some unrelated reason.
-                cmd = 'crm resource cleanup %s' % res_name
-                pcmk.commit(cmd)
-
-    for cl_name in clones:
-        # Always cleanup clones
-        cmd = 'crm resource cleanup %s' % cl_name
-        pcmk.commit(cmd)
-
-    for grp_name in groups:
-        # Always cleanup groups
-        cmd = 'crm resource cleanup %s' % grp_name
-        pcmk.commit(cmd)
+    # Only configure the cluster resources
+    # from the oldest peer unit.
+    if cluster.oldest_peer(cluster.peer_units()):
+        utils.juju_log('INFO', 'Configuring Resources')
+        utils.juju_log('INFO', str(resources))
+        for res_name, res_type in resources.iteritems():
+            # disable the service we are going to put in HA
+            if res_type.split(':')[0] == "lsb":
+                hacluster.disable_lsb_services(res_type.split(':')[1])
+                if utils.running(res_type.split(':')[1]):
+                    utils.stop(res_type.split(':')[1])
+            elif (len(init_services) != 0 and
+                  res_name in init_services and
+                  init_services[res_name]):
+                hacluster.disable_upstart_services(init_services[res_name])
+                if utils.running(init_services[res_name]):
+                    utils.stop(init_services[res_name])
+            # Put the services in HA, if not already done so
+            #if not pcmk.is_resource_present(res_name):
+            if not pcmk.crm_opt_exists(res_name):
+                if not res_name in resource_params:
+                    cmd = 'crm -F configure primitive %s %s' % (res_name,
+                                                                res_type)
+                else:
+                    cmd = 'crm -F configure primitive %s %s %s' % \
+                                (res_name,
+                                 res_type,
+                                 resource_params[res_name])
+                pcmk.commit(cmd)
+                utils.juju_log('INFO', '%s' % cmd)
+
+        utils.juju_log('INFO', 'Configuring Groups')
+        utils.juju_log('INFO', str(groups))
+        for grp_name, grp_params in groups.iteritems():
+            if not pcmk.crm_opt_exists(grp_name):
+                cmd = 'crm -F configure group %s %s' % (grp_name, grp_params)
+                pcmk.commit(cmd)
+                utils.juju_log('INFO', '%s' % cmd)
+
+        utils.juju_log('INFO', 'Configuring Master/Slave (ms)')
+        utils.juju_log('INFO', str(ms))
+        for ms_name, ms_params in ms.iteritems():
+            if not pcmk.crm_opt_exists(ms_name):
+                cmd = 'crm -F configure ms %s %s' % (ms_name, ms_params)
+                pcmk.commit(cmd)
+                utils.juju_log('INFO', '%s' % cmd)
+
+        utils.juju_log('INFO', 'Configuring Orders')
+        utils.juju_log('INFO', str(orders))
+        for ord_name, ord_params in orders.iteritems():
+            if not pcmk.crm_opt_exists(ord_name):
+                cmd = 'crm -F configure order %s %s' % (ord_name, ord_params)
+                pcmk.commit(cmd)
+                utils.juju_log('INFO', '%s' % cmd)
+
+        utils.juju_log('INFO', 'Configuring Colocations')
+        utils.juju_log('INFO', str(colocations))
+        for col_name, col_params in colocations.iteritems():
+            if not pcmk.crm_opt_exists(col_name):
+                cmd = 'crm -F configure colocation %s %s' % (col_name, col_params)
+                pcmk.commit(cmd)
+                utils.juju_log('INFO', '%s' % cmd)
+
+        utils.juju_log('INFO', 'Configuring Clones')
+        utils.juju_log('INFO', str(clones))
+        for cln_name, cln_params in clones.iteritems():
+            if not pcmk.crm_opt_exists(cln_name):
+                cmd = 'crm -F configure clone %s %s' % (cln_name, cln_params)
+                pcmk.commit(cmd)
+                utils.juju_log('INFO', '%s' % cmd)
+
+        for res_name, res_type in resources.iteritems():
+            if len(init_services) != 0 and res_name in init_services:
+                # Checks that the resources are running and started.
+                # Ensure that clones are excluded as the resource is
+                # not directly controllable (dealt with below)
+                # Ensure that groups are cleaned up as a whole rather
+                # than as individual resources.
+                if (res_name not in clones.values() and
+                    res_name not in groups.values() and
+                    not pcmk.crm_res_running(res_name)):
+                    # Just in case, cleanup the resources to ensure they get
+                    # started in case they failed for some unrelated reason.
+                    cmd = 'crm resource cleanup %s' % res_name
+                    pcmk.commit(cmd)
+
+        for cl_name in clones:
+            # Always cleanup clones
+            cmd = 'crm resource cleanup %s' % cl_name
+            pcmk.commit(cmd)
+
+        for grp_name in groups:
+            # Always cleanup groups
+            cmd = 'crm resource cleanup %s' % grp_name
+            pcmk.commit(cmd)
 
     for rel_id in utils.relation_ids('ha'):
         utils.relation_set(rid=rel_id,
@@ -382,42 +386,28 @@
     pcmk.commit(cmd)
 
 
-def ha_relation_departed():
-    # TODO: Fin out which node is departing and put it in standby mode.
-    # If this happens, and a new relation is created in the same machine
-    # (which already has node), then check whether it is standby and put it
-    # in online mode. This should be done in ha_relation_joined.
-    pcmk.standby(utils.get_unit_hostname())
-
-
 def get_cluster_nodes():
     hosts = []
-    hosts.append('{}:6789'.format(utils.get_host_ip()))
-
+    hosts.append(utils.unit_get('private-address'))
     for relid in utils.relation_ids('hanode'):
         for unit in utils.relation_list(relid):
-            hosts.append(
-                '{}:6789'.format(utils.get_host_ip(
-                                    utils.relation_get('private-address',
-                                                       unit, relid)))
-                )
-
+            if utils.relation_get('ready',
+                                  rid=relid,
+                                  unit=unit):
+                hosts.append(utils.relation_get('private-address',
+                                                unit, relid))
     hosts.sort()
     return hosts
 
 
-utils.do_hooks({
-        'install': install,
-        'config-changed': config_changed,
-        'start': start,
-        'stop': stop,
-        'upgrade-charm': upgrade_charm,
-        'ha-relation-joined': configure_cluster,
-        'ha-relation-changed': configure_cluster,
-        'ha-relation-departed': ha_relation_departed,
-        'hanode-relation-joined': configure_cluster,
-        #'hanode-relation-departed': hanode_relation_departed,
-        # TODO: should probably remove nodes from the cluster
-        })
+hooks = {
+    'install': install,
+    'config-changed': config_changed,
+    'upgrade-charm': upgrade_charm,
+    'ha-relation-joined': configure_cluster,
+    'ha-relation-changed': configure_cluster,
+    'hanode-relation-joined': configure_cluster,
+    'hanode-relation-changed': configure_cluster,
+    }
 
-sys.exit(0)
+utils.do_hooks(hooks)

=== added directory 'hooks/lib'
=== added file 'hooks/lib/__init__.py'
=== added file 'hooks/lib/cluster_utils.py'
--- hooks/lib/cluster_utils.py	1970-01-01 00:00:00 +0000
+++ hooks/lib/cluster_utils.py	2013-03-25 14:21:21 +0000
@@ -0,0 +1,130 @@
+#
+# Copyright 2012 Canonical Ltd.
+#
+# This file is sourced from lp:openstack-charm-helpers
+#
+# Authors:
+#  James Page <[e-mail]>
+#  Adam Gandelman <[e-mail]>
+#
+
+from lib.utils import (
+    juju_log,
+    relation_ids,
+    relation_list,
+    relation_get,
+    get_unit_hostname,
+    config_get
+    )
+import subprocess
+import os
+
+
+def is_clustered():
+    for r_id in (relation_ids('ha') or []):
+        for unit in (relation_list(r_id) or []):
+            clustered = relation_get('clustered',
+                                     rid=r_id,
+                                     unit=unit)
+            if clustered:
+                return True
+    return False
+
+
+def is_leader(resource):
+    cmd = [
+        "crm", "resource",
+        "show", resource
+        ]
+    try:
+        status = subprocess.check_output(cmd)
+    except subprocess.CalledProcessError:
+        return False
+    else:
+        if get_unit_hostname() in status:
+            return True
+        else:
+            return False
+
+
+def peer_units():
+    peers = []
+    for r_id in (relation_ids('cluster') or []):
+        for unit in (relation_list(r_id) or []):
+            peers.append(unit)
+    return peers
+
+
+def oldest_peer(peers):
+    local_unit_no = int(os.getenv('JUJU_UNIT_NAME').split('/')[1])
+    for peer in peers:
+        remote_unit_no = int(peer.split('/')[1])
+        if remote_unit_no < local_unit_no:
+            return False
+    return True
+
+
+def eligible_leader(resource):
+    if is_clustered():
+        if not is_leader(resource):
+            juju_log('INFO', 'Deferring action to CRM leader.')
+            return False
+    else:
+        peers = peer_units()
+        if peers and not oldest_peer(peers):
+            juju_log('INFO', 'Deferring action to oldest service unit.')
+            return False
+    return True
+
+
+def https():
+    '''
+    Determines whether enough data has been provided in configuration
+    or relation data to configure HTTPS
+    .
+    returns: boolean
+    '''
+    if config_get('use-https') == "yes":
+        return True
+    if config_get('ssl_cert') and config_get('ssl_key'):
+        return True
+    for r_id in relation_ids('identity-service'):
+        for unit in relation_list(r_id):
+            if (relation_get('https_keystone', rid=r_id, unit=unit) and
+                relation_get('ssl_cert', rid=r_id, unit=unit) and
+                relation_get('ssl_key', rid=r_id, unit=unit) and
+                relation_get('ca_cert', rid=r_id, unit=unit)):
+                return True
+    return False
+
+
+def determine_api_port(public_port):
+    '''
+    Determine correct API server listening port based on
+    existence of HTTPS reverse proxy and/or haproxy.
+
+    public_port: int: standard public port for given service
+
+    returns: int: the correct listening port for the API service
+    '''
+    i = 0
+    if len(peer_units()) > 0 or is_clustered():
+        i += 1
+    if https():
+        i += 1
+    return public_port - (i * 10)
+
+
+def determine_haproxy_port(public_port):
+    '''
+    Description: Determine correct proxy listening port based on public IP +
+    existence of HTTPS reverse proxy.
+
+    public_port: int: standard public port for given service
+
+    returns: int: the correct listening port for the HAProxy service
+    '''
+    i = 0
+    if https():
+        i += 1
+    return public_port - (i * 10)

=== added file 'hooks/lib/utils.py'
--- hooks/lib/utils.py	1970-01-01 00:00:00 +0000
+++ hooks/lib/utils.py	2013-03-25 14:21:21 +0000
@@ -0,0 +1,332 @@
+#
+# Copyright 2012 Canonical Ltd.
+#
+# This file is sourced from lp:openstack-charm-helpers
+#
+# Authors:
+#  James Page <[e-mail]>
+#  Paul Collins <[e-mail]>
+#  Adam Gandelman <[e-mail]>
+#
+
+import json
+import os
+import subprocess
+import socket
+import sys
+
+
+def do_hooks(hooks):
+    hook = os.path.basename(sys.argv[0])
+
+    try:
+        hook_func = hooks[hook]
+    except KeyError:
+        juju_log('INFO',
+                 "This charm doesn't know how to handle '{}'.".format(hook))
+    else:
+        hook_func()
+
+
+def install(*pkgs):
+    cmd = [
+        'apt-get',
+        '-y',
+        'install'
+          ]
+    for pkg in pkgs:
+        cmd.append(pkg)
+    subprocess.check_call(cmd)
+
+TEMPLATES_DIR = 'templates'
+
+try:
+    import jinja2
+except ImportError:
+    install('python-jinja2')
+    import jinja2
+
+try:
+    import dns.resolver
+except ImportError:
+    install('python-dnspython')
+    import dns.resolver
+
+
+def render_template(template_name, context, template_dir=TEMPLATES_DIR):
+    templates = jinja2.Environment(
+                    loader=jinja2.FileSystemLoader(template_dir)
+                    )
+    template = templates.get_template(template_name)
+    return template.render(context)
+
+CLOUD_ARCHIVE = \
+""" # Ubuntu Cloud Archive
+deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main
+"""
+
+CLOUD_ARCHIVE_POCKETS = {
+    'folsom': 'precise-updates/folsom',
+    'folsom/updates': 'precise-updates/folsom',
+    'folsom/proposed': 'precise-proposed/folsom',
+    'grizzly': 'precise-updates/grizzly',
+    'grizzly/updates': 'precise-updates/grizzly',
+    'grizzly/proposed': 'precise-proposed/grizzly'
+    }
+
+
+def configure_source():
+    source = str(config_get('openstack-origin'))
+    if not source:
+        return
+    if source.startswith('ppa:'):
+        cmd = [
+            'add-apt-repository',
+            source
+            ]
+        subprocess.check_call(cmd)
+    if source.startswith('cloud:'):
+        # CA values should be formatted as cloud:ubuntu-openstack/pocket, eg:
+        #   cloud:precise-folsom/updates or cloud:precise-folsom/proposed
+        install('ubuntu-cloud-keyring')
+        pocket = source.split(':')[1]
+        pocket = pocket.split('-')[1]
+        with open('/etc/apt/sources.list.d/cloud-archive.list', 'w') as apt:
+            apt.write(CLOUD_ARCHIVE.format(CLOUD_ARCHIVE_POCKETS[pocket]))
+    if source.startswith('deb'):
+        l = len(source.split('|'))
+        if l == 2:
+            (apt_line, key) = source.split('|')
+            cmd = [
+                'apt-key',
+                'adv', '--keyserver keyserver.ubuntu.com',
+                '--recv-keys', key
+                ]
+            subprocess.check_call(cmd)
+        elif l == 1:
+            apt_line = source
+
+        with open('/etc/apt/sources.list.d/quantum.list', 'w') as apt:
+            apt.write(apt_line + "\n")
+    cmd = [
+        'apt-get',
+        'update'
+        ]
+    subprocess.check_call(cmd)
+
+# Protocols
+TCP = 'TCP'
+UDP = 'UDP'
+
+
+def expose(port, protocol='TCP'):
+    cmd = [
+        'open-port',
+        '{}/{}'.format(port, protocol)
+        ]
+    subprocess.check_call(cmd)
+
+
+def juju_log(severity, message):
+    cmd = [
+        'juju-log',
+        '--log-level', severity,
+        message
+        ]
+    subprocess.check_call(cmd)
+
+
+cache = {}
+
+
+def cached(func):
+    def wrapper(*args, **kwargs):
+        global cache
+        key = str((func, args, kwargs))
+        try:
+            return cache[key]
+        except KeyError:
+            res = func(*args, **kwargs)
+            cache[key] = res
+            return res
+    return wrapper
+
+
+@cached
+def relation_ids(relation):
+    cmd = [
+        'relation-ids',
+        relation
+        ]
+    result = str(subprocess.check_output(cmd)).split()
+    if result == "":
+        return None
+    else:
+        return result
+
+
+@cached
+def relation_list(rid):
+    cmd = [
+        'relation-list',
+        '-r', rid,
+        ]
+    result = str(subprocess.check_output(cmd)).split()
+    if result == "":
+        return None
+    else:
+        return result
+
+
+@cached
+def relation_get(attribute, unit=None, rid=None):
+    cmd = [
+        'relation-get',
+        ]
+    if rid:
+        cmd.append('-r')
+        cmd.append(rid)
+    cmd.append(attribute)
+    if unit:
+        cmd.append(unit)
+    value = subprocess.check_output(cmd).strip()  # IGNORE:E1103
+    if value == "":
+        return None
+    else:
+        return value
+
+
+@cached
+def relation_get_dict(relation_id=None, remote_unit=None):
+    """Obtain all relation data as dict by way of JSON"""
+    cmd = [
+        'relation-get', '--format=json'
+        ]
+    if relation_id:
+        cmd.append('-r')
+        cmd.append(relation_id)
+    if remote_unit:
+        remote_unit_orig = os.getenv('JUJU_REMOTE_UNIT', None)
+        os.environ['JUJU_REMOTE_UNIT'] = remote_unit
+    j = subprocess.check_output(cmd)
+    if remote_unit and remote_unit_orig:
+        os.environ['JUJU_REMOTE_UNIT'] = remote_unit_orig
+    d = json.loads(j)
+    settings = {}
+    # convert unicode to strings
+    for k, v in d.iteritems():
+        settings[str(k)] = str(v)
+    return settings
+
+
+def relation_set(**kwargs):
+    cmd = [
+        'relation-set'
+        ]
+    args = []
+    for k, v in kwargs.items():
+        if k == 'rid':
+            if v:
+                cmd.append('-r')
+                cmd.append(v)
+        else:
+            args.append('{}={}'.format(k, v))
+    cmd += args
+    subprocess.check_call(cmd)
+
+
+@cached
+def unit_get(attribute):
+    cmd = [
+        'unit-get',
+        attribute
+        ]
+    value = subprocess.check_output(cmd).strip()  # IGNORE:E1103
+    if value == "":
+        return None
+    else:
+        return value
+
+
+@cached
+def config_get(attribute):
+    cmd = [
+        'config-get',
+        '--format',
+        'json',
+        ]
+    out = subprocess.check_output(cmd).strip()  # IGNORE:E1103
+    cfg = json.loads(out)
+
+    try:
+        return cfg[attribute]
+    except KeyError:
+        return None
+
+
+@cached
+def get_unit_hostname():
+    return socket.gethostname()
+
+
+@cached
+def get_host_ip(hostname=unit_get('private-address')):
+    try:
+        # Test to see if already an IPv4 address
+        socket.inet_aton(hostname)
+        return hostname
+    except socket.error:
+        answers = dns.resolver.query(hostname, 'A')
+        if answers:
+            return answers[0].address
+    return None
+
+
+def _svc_control(service, action):
+    subprocess.check_call(['service', service, action])
+
+
+def restart(*services):
+    for service in services:
+        _svc_control(service, 'restart')
+
+
+def stop(*services):
+    for service in services:
+        _svc_control(service, 'stop')
+
+
+def start(*services):
+    for service in services:
+        _svc_control(service, 'start')
+
+
+def reload(*services):
+    for service in services:
+        try:
+            _svc_control(service, 'reload')
+        except subprocess.CalledProcessError:
+            # Reload failed - either service does not support reload
+            # or it was not running - restart will fixup most things
+            _svc_control(service, 'restart')
+
+
+def running(service):
+    try:
+        output = subprocess.check_output(['service', service, 'status'])
+    except subprocess.CalledProcessError:
+        return False
+    else:
+        if ("start/running" in output or
+            "is running" in output):
+            return True
+        else:
+            return False
+
+
+def is_relation_made(relation, key='private-address'):
+    for r_id in (relation_ids(relation) or []):
+        for unit in (relation_list(r_id) or []):
+            if relation_get(key, rid=r_id, unit=unit):
+                return True
+    return False

=== modified file 'hooks/maas.py'
--- hooks/maas.py	2013-02-20 10:10:42 +0000
+++ hooks/maas.py	2013-03-25 14:21:21 +0000
@@ -3,7 +3,7 @@
 import json
 import subprocess
 
-import utils
+import lib.utils as utils
 
 MAAS_STABLE_PPA = 'ppa:maas-maintainers/stable '
 MAAS_PROFILE_NAME = 'maas-juju-hacluster'

=== modified file 'hooks/pcmk.py'
--- hooks/pcmk.py	2013-02-20 10:10:42 +0000
+++ hooks/pcmk.py	2013-03-25 14:21:21 +0000
@@ -1,4 +1,4 @@
-import utils
+import lib.utils as utils
 import commands
 import subprocess
 

=== removed symlink 'hooks/start'
=== target was u'hooks.py'
=== removed symlink 'hooks/stop'
=== target was u'hooks.py'