Merge lp:~andreserl/charms/quantal/mysql/ceph-support into lp:~openstack-charmers/charms/quantal/mysql/hacluster-support

[Andres Rodriguez]

This branch removes support for DRBD and adds support for Ceph.

[James Page (james-page)]

1) ceph-relation-changed

Should figure out if mysql is already running on ceph rbd and not stop, start mysql.

Just checking for the device presence should be enough.

The rbd kernel module will pickup changes in /etc/ceph/ceph.conf automagically.

2) Upgrade

The charm needs to move /var/lib/juju/*.passwd /var/lib/mysql on upgrade.

3) drbd.py

Drop as no longer used.

4) /etc/mysql/debian.cnf

Also needs to be synched from master server as contains debian sys maintenance password for upgrades.

5) shared-db-relations

The is_clustered/eligible_leader check is to complicated - the eligible leader check should be dealing with the is_clustered bit anyway:

    if not utils.eligible_leader():
        utils.juju_log('INFO',
                       'MySQL service is peered, bailing shared-db relation')
        return

This should be enough - am I the leader whatever the context? no - OK bail!

6) config.yaml

    vip:
        type: string
        default: None
        description: "Virtual IP to use to front keystone in ha configuration"

This is inconsistent with the rest of the charms which define this; I'd prefer if you dropped the 'default: None' and just checked for the vip like this:

   if 'vip' in config:
      ...

[James Page (james-page)]

7) block-storage config option

Still don't like this - it can be inferred by looking at the relations a service has.

i.e. I kept forgetting to set it and ended up with a broken cluster!

[Adam Gandelman (gandelman-a)]

One suggestion I have is to add shared-db relation settings to signify the service's cluster status, similar to the hacluster's use of 'clustered=yes' in its relation to its principle. If the clustered status + VIP get passed to client services via the relation, the other side of the relation can do something similar to:

[[ "$(relation-get clustered)" == "True" ]] && db_host="$(relation-get vip)" || db_host="$(relation-get private-address)"
db_conn=mysql://$user:$passwd@$db_host/$database

Services should make use of private-address from the environment instead of hostnames / address provided via relation-settings wherever possible. We're totally bending the rules currently to support the floating IPs/hacluster scenario, and I imagine it is going to break in environments with split-horizon DNS or complex routing setups. I'd like to avoid using something like 'db_host' via relation unless its absolutely required (in a clustered deployment), and use the private-address from environment in all other instances.

[James Page (james-page)]

Adam

Not sure I agree; surely its the difference between doing the check on the mysql side (once) or the same check on the other end of the relation (many times).

[lp:~andreserl/charms/quantal/mysql/ceph-support]

118. By Andres Rodriguez on 2013-02-19

Address some of James' concerns

119. By Andres Rodriguez on 2013-02-19

Remove no longer supported drbd installation

120. By Andres Rodriguez on 2013-02-19

Do some cleanup and ensure rados image creation is done in the leader

121. By Andres Rodriguez on 2013-02-20

Move service_user2 and passwd files from /var/lib/juju to /var/lib/mysql on upgrade

[James Page (james-page)]

Still need to sort out the mysql restarts in ceph changed and the syncing of files.

Lets do that as a separate MP.

review: Approve

=== modified file 'config.yaml'
--- config.yaml	2013-01-16 21:46:58 +0000
+++ config.yaml	2013-02-20 16:05:25 +0000
@@ -50,18 +50,20 @@
           with the other members of the HA Cluster.
     ha-mcastport:
         type: int
-        default: 5410
+        default: 5411
         description: |
           Default multicast port number that will be used to communicate between
           HA Cluster nodes.
-    block-storage:
-        type: string
-        default: None
-        description: |
-          Default block storage type to use when setting up MySQL in HA. Options
-          are 'drbd' or 'ceph'.
-    block-device:
-        type: string
-        default: "/dev/sdb1"
-        description: |
-           "The *available* block device to use for DRBD."
+    block-size:
+        type: string
+        default: 5G
+        description: |
+          Default block storage size to create when setting up MySQL block storage.
+          This value should be specified in GB (e.g. 100G).
+    rbd-name:
+        type: string
+        default: mysql1
+        description: |
+          The name that will be used to create the Ceph's RBD image with. If the
+          image name exists in Ceph, it will be re-used and the data will be
+          overwritten.

=== added symlink 'hooks/ceph-relation-changed'
=== target is u'ha-relations'
=== added symlink 'hooks/ceph-relation-joined'
=== target is u'ha-relations'
=== added file 'hooks/ceph.py'
--- hooks/ceph.py	1970-01-01 00:00:00 +0000
+++ hooks/ceph.py	2013-02-20 16:05:25 +0000
@@ -0,0 +1,108 @@
+import utils
+import commands
+import re
+import subprocess
+import sys
+import os
+import shutil
+
+
+def execute(cmd):
+    subprocess.check_call(cmd)
+
+
+def execute_shell(cmd):
+    subprocess.check_call(cmd, shell=True)
+
+
+def create_image(service, image, sizemb):
+    cmd = [
+        'rbd',
+        'create',
+        image,
+        '--size',
+        sizemb,
+        '--id',
+        service,
+        '--pool',
+        'images'
+        ]
+    execute(cmd)
+
+
+def create_image_pool(service):
+    cmd = [
+        'rados',
+        '--id',
+        service,
+        'mkpool',
+        'images'
+        ]
+    execute(cmd)
+
+
+def create_keyring(service, keyring, key):
+    cmd = [
+        'ceph-authtool',
+        keyring,
+        '--create-keyring',
+        '--name=client.%s' % service,
+        '--add-key=%s' % key
+        ]
+    execute(cmd)
+
+
+def map_block_storage(service, image, keyfile):
+    cmd = [
+        'rbd',
+        'map',
+        'images/%s' % image,
+        '--user',
+        service,
+        '--secret',
+        keyfile,
+        ]
+    execute(cmd)
+
+
+def make_filesystem(service, blk_device, fstype='ext4'):
+    cmd = ['mkfs', '-t', fstype, blk_device]
+    execute(cmd)
+
+
+def place_data_on_ceph(service, blk_device, data_src_dst, fstype='ext4'):
+    # mount block device into /mnt
+    cmd = ['mount', '-t', fstype, blk_device, '/mnt']
+    execute(cmd)
+
+    # copy data to /mnt
+    try:
+        copy_files(data_src_dst, '/mnt')
+    except:
+        pass
+
+    # umount block device
+    cmd = ['umount', '/mnt']
+    execute(cmd)
+
+    # re-mount where the data should originally be
+    cmd = ['mount', '-t', fstype, blk_device, data_src_dst]
+    execute(cmd)
+
+
+# TODO: re-use
+def modprobe_kernel_module(module):
+    cmd = ['modprobe', module]
+    execute(cmd)
+    cmd = 'echo %s >> /etc/modules' % module
+    execute_shell(cmd)
+
+
+def copy_files(src, dst, symlinks=False, ignore=None):
+    for item in os.listdir(src):
+        s = os.path.join(src, item)
+        d = os.path.join(dst, item)
+        if os.path.isdir(s):
+            shutil.copytree(s, d, symlinks, ignore)
+        else:
+            shutil.copy2(s, d)

=== modified symlink 'hooks/cluster-relation-changed'
=== target changed u'ha-relation' => u'ha-relations'
=== modified file 'hooks/common.py'
--- hooks/common.py	2012-11-29 16:31:14 +0000
+++ hooks/common.py	2013-02-20 16:05:25 +0000
@@ -7,7 +7,7 @@
 import uuid
 
 def get_service_user_file(service):
-    return '/var/lib/juju/%s.service_user2' % service
+    return '/var/lib/mysql/%s.service_user2' % service
 
 
 def get_service_user(service):
@@ -58,7 +58,7 @@
 
 def get_db_cursor():
     # Connect to mysql
-    passwd = open("/var/lib/juju/mysql.passwd").read().strip()
+    passwd = open("/var/lib/mysql/mysql.passwd").read().strip()
     connection = MySQLdb.connect(user="root", host="localhost", passwd=passwd)
     return connection.cursor()
 

=== modified file 'hooks/config-changed'
--- hooks/config-changed	2013-01-17 04:37:31 +0000
+++ hooks/config-changed	2013-02-20 16:05:25 +0000
@@ -90,7 +90,7 @@
     check_call(['add-apt-repository','-y','deb http://%s %s main' % (source, series)])
     check_call(['apt-get','update'])
 
-with open('/var/lib/juju/mysql.passwd','r') as rpw:
+with open('/var/lib/mysql/mysql.passwd','r') as rpw:
     root_pass = rpw.read()
 
 dconf = Popen(['debconf-set-selections'], stdin=PIPE)
@@ -350,6 +350,3 @@
     except CalledProcessError:
         pass
     check_call(['service','mysql','start'])
-
-if configs['block-storage'] == "drbd":
-    utils.install('drbd8-utils')

=== removed file 'hooks/drbd.py'
--- hooks/drbd.py	2013-01-11 20:39:20 +0000
+++ hooks/drbd.py	1970-01-01 00:00:00 +0000
@@ -1,130 +0,0 @@
-import utils
-import commands
-import re
-import subprocess
-import sys
-import os
-import shutil
-
-
-def execute(cmd):
-    subprocess.check_call(cmd)
-
-
-def execute_shell(cmd):
-    subprocess.check_call(cmd, shell=True)
-
-
-def prepare_drbd_disk(block_device=None):
-    if block_device is None:
-        sys.exit(1)
-
-    cmd = 'dd if=/dev/zero of=%s bs=512 count=1 oflag=direct >/dev/null' % block_device
-    execute_shell(cmd)
-    cmd = '(echo n; echo p; echo 1; echo ; echo; echo w) | fdisk %s' % block_device
-    execute_shell(cmd)
-
-
-def modprobe_module():
-    cmd = ['modprobe', 'drbd']
-    execute(cmd)
-    cmd = 'echo drbd >> /etc/modules'
-    execute_shell(cmd)
-
-
-def create_md(resource):
-    cmd = ['drbdadm', '--', '--force', 'create-md', resource]
-    execute(cmd)
-
-
-def bring_resource_up(resource):
-    cmd = ['drbdadm', 'up', resource]
-    execute(cmd)
-
-
-def clear_bitmap(resource):
-    cmd = ['drbdadm', '--', '--clear-bitmap', 'new-current-uuid', resource]
-    execute(cmd)
-
-
-def make_primary(resource):
-    cmd = ['drbdadm', 'primary', resource]
-    execute(cmd)
-
-
-def make_secondary(resource):
-    cmd = ['drbdadm', 'secondary', resource]
-    execute(cmd)
-
-
-def format_drbd_device():
-    cmd = ['mkfs', '-t', 'ext3', '/dev/drbd0']
-    execute(cmd)
-
-
-def is_connected():
-    (status, output) = commands.getstatusoutput("drbd-overview")
-    show_re = re.compile("0:export  Connected")
-    quorum = show_re.search(output)
-    if quorum:
-        return True
-    return False
-
-
-def is_quorum_secondary():
-    (status, output) = commands.getstatusoutput("drbd-overview")
-    show_re = re.compile("Secondary/Secondary")
-    quorum = show_re.search(output)
-    if quorum:
-        return True
-    return False
-
-
-def is_quorum_primary():
-    (status, output) = commands.getstatusoutput("drbd-overview")
-    show_re = re.compile("Primary/Secondary")
-    quorum = show_re.search(output)
-    if quorum:
-        return True
-    return False
-
-
-def is_state_inconsistent():
-    (status, output) = commands.getstatusoutput("drbd-overview")
-    show_re = re.compile("Inconsistent/Inconsistent")
-    quorum = show_re.search(output)
-    if quorum:
-        return True
-    return False
-
-
-def is_state_uptodate():
-    (status, output) = commands.getstatusoutput("drbd-overview")
-    show_re = re.compile("UpToDate/UpToDate")
-    quorum = show_re.search(output)
-    if quorum:
-        return True
-    return False
-
-
-def copy_files(src, dst, symlinks=False, ignore=None):
-    for item in os.listdir(src):
-        s = os.path.join(src, item)
-        d = os.path.join(dst, item)
-        if os.path.isdir(s):
-            shutil.copytree(s, d, symlinks, ignore)
-        else:
-            shutil.copy2(s, d)
-
-
-def put_on_drbd():
-    cmd = ['mount', '-t', 'ext3', '/dev/drbd0', '/mnt']
-    execute(cmd)
-    # TODO: Before copying make sure it is mounted.
-    copy_files('/var/lib/mysql','/mnt')
-    cmd = ['chown', '-R', 'mysql:mysql', '/mnt']
-    execute(cmd)
-    cmd = ['umount', '/mnt']
-    execute(cmd)
-    cmd = ['mount', '-t', 'ext3', '/dev/drbd0', '/var/lib/mysql']
-    execute(cmd)

=== modified symlink 'hooks/ha-relation-changed'
=== target changed u'ha-relation' => u'ha-relations'
=== modified symlink 'hooks/ha-relation-joined'
=== target changed u'ha-relation' => u'ha-relations'
=== renamed file 'hooks/ha-relation' => 'hooks/ha-relations'
--- hooks/ha-relation	2013-01-17 04:37:31 +0000
+++ hooks/ha-relations	2013-02-20 16:05:25 +0000
@@ -4,235 +4,284 @@
 import sys
 import subprocess
 import os
+import time
+import commands
+
 import utils
-import drbd
-import time
+import ceph
 
 STORAGEMARKER = '/var/lib/juju/storageconfigured'
-DRBD_RESOURCE = 'mysql'
-DRBD_DEVICE = '/dev/drbd0'
-DRBD_MOUNTPOINT = '/var/lib/mysql'
+
+# CEPH
+DATA_SRC_DST = '/var/lib/mysql'
+SERVICE_NAME = utils.get_unit_name().replace('-','/').split('/')[0]
+KEYRING = "/etc/ceph/ceph.client.%s.keyring" % SERVICE_NAME
+KEYFILE = "/etc/ceph/ceph.client.%s.key" % SERVICE_NAME
+
 
 config=json.loads(subprocess.check_output(['config-get','--format=json']))
 
 
 def ha_relation_joined():
-    # obtain the block device
-    block_storage = config['block-storage']
-    block_device = config['block-device']
+
+    # Checking vip values
+    if not 'vip' in config:
+        utils.juju_log('WARNING', 'NO Virtual IP was defined, bailing')
+        sys.exit(1)
+
+    if config['vip_iface'] == "None" or not config['vip_iface']:
+        utils.juju_log('WARNING', 'NO Virtual IP interface was defined, bailing')
+        sys.exit(1)
+
+    if config['vip_cidr'] == "None" or not config['vip_cidr']:
+        utils.juju_log('WARNING', 'NO CIDR was defined for the Virtual IP, bailing')
+        sys.exit(1)
 
     # Obtain the config values necessary for the cluster config. These
     # include multicast port and interface to bind to.
     corosync_bindiface = config['ha-bindiface']
     corosync_mcastport = config['ha-mcastport']
 
-    if block_storage == "None":
-        utils.juju_log('WARNING',
-                       'NO block storage configured, not passing HA relation data')
+    # Starting configuring resources.
+    init_services = {
+            'res_mysqld':'mysql',
+        }
+
+
+    # If the 'ha' relation has been made *before* the 'ceph' relation,
+    # it doesn't make sense to make it until after the 'ceph' relation
+    # is made
+    if not utils.is_relation_made('ceph'):
+        utils.juju_log('INFO',
+                '*ceph* relation does not exist. Not sending *ha* relation data')
         return
-    elif block_storage == "drbd":
-        # Obtain resources
+    else:
+        utils.juju_log('INFO',
+                    '*ceph* relation exists. Sending *ha* relation data')
+
+        block_storage = 'ceph'
+
         resources = {
+                'res_mysql_rbd':'ocf:ceph:rbd',
+                'res_mysql_fs':'ocf:heartbeat:Filesystem',
                 'res_mysql_vip':'ocf:heartbeat:IPaddr2',
-                'res_mysql_fs':'ocf:heartbeat:Filesystem',
-                'res_mysql_drbd':'ocf:linbit:drbd',
                 'res_mysqld':'upstart:mysql',
             }
+
         resource_params = {
+                'res_mysql_rbd':'params name="%s" pool="images" user="%s" secret="%s"' % (
+                                config['rbd-name'], SERVICE_NAME, KEYFILE),
+                'res_mysql_fs':'params device="/dev/rbd/images/%s" directory="%s" fstype="ext4" op start start-delay="10s"' % (
+                                config['rbd-name'], DATA_SRC_DST),
                 'res_mysql_vip':'params ip="%s" cidr_netmask="%s" nic="%s"' % (config['vip'],
                                 config['vip_cidr'], config['vip_iface']),
-                'res_mysql_fs':'params device="%s" directory="%s" fstype="ext3"' % (DRBD_DEVICE, DRBD_MOUNTPOINT),
-                'res_mysql_drbd':'params drbd_resource="%s"' % DRBD_RESOURCE,
-                'res_mysqld':'op monitor interval=5s',
-            }
-
-        init_services = {
-                'res_mysqld':'mysql',
+                'res_mysqld':'op start start-delay="5s" op monitor interval="5s"',
             }
 
         groups = {
-                'grp_mysql':'res_mysql_fs res_mysql_vip res_mysqld',
-            }
-
-        ms = {
-                'ms_drbd_mysql':'res_mysql_drbd meta notify="true" master-max="1" master-node-max="1" clone-max="2" clone-node-max="1"'
-            }
-
-        orders = {
-                'ord_drbd_before_mysql':'inf: ms_drbd_mysql:promote grp_mysql:start'
-            }
-
-        colocations = {
-                'col_mysql_on_drbd':'inf: grp_mysql ms_drbd_mysql:Master'
-            }
-
-    utils.relation_set(block_storage=block_storage,
-                 block_device=block_device,
-                 corosync_bindiface=corosync_bindiface,
-                 corosync_mcastport=corosync_mcastport,
-                 resources=resources,
-                 resource_params=resource_params,
-                 init_services=init_services,
-                 colocations=colocations,
-                 orders=orders,
-                 groups=groups,
-                 ms=ms)
+                'grp_mysql':'res_mysql_rbd res_mysql_fs res_mysql_vip res_mysqld',
+            }
+
+        for rel_id in utils.relation_ids('ha'):
+            utils.relation_set(rid=rel_id,
+                               block_storage=block_storage,
+                               corosync_bindiface=corosync_bindiface,
+                               corosync_mcastport=corosync_mcastport,
+                               resources=resources,
+                               resource_params=resource_params,
+                               init_services=init_services,
+                               groups=groups)
 
 
 def ha_relation_changed():
-    pass
-
-
-def get_cluster_nodes():
+    relation_data = utils.relation_get_dict()
+    if ('clustered' in relation_data and
+        utils.is_leader()):
+        utils.juju_log('INFO', 'Cluster configured, notifying other services')
+        # Tell all related services to start using
+        # the VIP
+        for r_id in utils.relation_ids('shared-db'):
+            utils.relation_set(rid=r_id,
+                           db_host=config['vip'])
+
+
+def ceph_joined():
+    utils.juju_log('INFO', 'Start Ceph Relation Joined')
+
+    ceph_dir = "/etc/ceph"
+    if not os.path.isdir(ceph_dir):
+        os.mkdir(ceph_dir)
+    utils.install('ceph-common')
+
+    utils.juju_log('INFO', 'Finish Ceph Relation Joined')
+
+
+def ceph_changed():
+    utils.juju_log('INFO', 'Start Ceph Relation Changed')
+
+    # TODO: ask james: What happens if the relation data has changed?
+    # do we reconfigure ceph? What do we do with the data?
+    key = utils.relation_get('key')
+
+    if key:
+        # create KEYRING file
+        if not os.path.exists(KEYRING):
+            ceph.create_keyring(SERVICE_NAME, KEYRING, key)
+        # create a file containing the key
+        if not os.path.exists(KEYFILE):
+            fd = open(KEYFILE, 'w')
+            fd.write(key)
+            fd.close()
+    else:
+        sys.exit(0)
+
+    # emit ceph config
+    hosts = get_ceph_nodes()
+    mon_hosts = ",".join(map(str, hosts))
+    conf_context = {
+        'auth': utils.relation_get('auth'),
+        'keyring': KEYRING,
+        'mon_hosts': mon_hosts,
+        }
+    with open('/etc/ceph/ceph.conf', 'w') as ceph_conf:
+        ceph_conf.write(utils.render_template('ceph.conf',
+                                              conf_context))
+
+    # Create the images pool if it does not already exist
+    if utils.eligible_leader():
+        (status, output) = commands.getstatusoutput("rados --id %s lspools" % SERVICE_NAME)
+        pools = "images" in output
+        if not pools:
+            utils.juju_log('INFO','Creating image pool')
+            ceph.create_image_pool(SERVICE_NAME)
+
+    # Configure ceph()
+    configure_ceph()
+
+    # If 'ha' relation has been made before the 'ceph' relation
+    # it is important to make sure the ha-relation data is being
+    # sent.
+    if utils.is_relation_made('ha'):
+        utils.juju_log('INFO',
+                       '*ha* relation exists. Making sure the ha relation data is sent.')
+        ha_relation_joined()
+        return
+    else:
+        utils.juju_log('INFO',
+                       '*ha* relation does not exist.')
+
+    utils.juju_log('INFO', 'Finish Ceph Relation Changed')
+
+
+def configure_ceph():
+    utils.juju_log('INFO', 'Start Ceph Configuration')
+
+    block_sizemb = int(config['block-size'].split('G')[0]) * 1024
+    image_name = config['rbd-name']
+    fstype = 'ext4'
+    data_src = DATA_SRC_DST
+    blk_device = '/dev/rbd/images/%s' % image_name
+
+    # modprobe the kernel module
+    utils.juju_log('INFO','Loading kernel module')
+    ceph.modprobe_kernel_module('rbd')
+
+
+    # configure mysql for ceph storage options
+    if not utils.eligible_leader():
+        utils.juju_log('INFO','This is not the peer leader. Not configuring RBD.')
+        # Stopping MySQL
+        if utils.running('mysql'):
+            utils.juju_log('INFO','Stopping MySQL...')
+            utils.stop('mysql')
+        return
+
+    elif utils.eligible_leader():
+        # create an image/block device
+        (status, output) = commands.getstatusoutput('rbd list --id %s --pool images' % SERVICE_NAME)
+        rbd = image_name in output
+        if not rbd:
+            utils.juju_log('INFO', 'Creating RBD Image...')
+            ceph.create_image(SERVICE_NAME, image_name, str(block_sizemb))
+        else:
+            utils.juju_log('INFO',
+                       'Looks like RBD already exists. Not creating a new one.')
+
+        # map the image to a block device if not already mapped.
+        (status, output) = commands.getstatusoutput('rbd showmapped')
+        mapped = image_name in output
+        if not mapped:
+            # map block storage
+            utils.juju_log('INFO', 'Mapping RBD Image as a Block Device')
+            ceph.map_block_storage(SERVICE_NAME, image_name, KEYFILE)
+        else:
+            utils.juju_log('INFO',
+                       'Looks like RBD is already mapped. Not re-mapping.')
+
+        # make file system
+        # TODO: What happens if for whatever reason this is run again and
+        # the data is already in the rbd device and/or is mounted??
+        # When it is mounted already, it will fail to make the fs
+        utils.juju_log('INFO', 'Trying to move data over to RBD.')
+        if not filesystem_mounted(data_src):
+            utils.juju_log('INFO', 'Formating RBD.')
+            ceph.make_filesystem(SERVICE_NAME, blk_device, fstype)
+
+            # Stopping MySQL
+            if utils.running('mysql'):
+                utils.juju_log('INFO','Stopping MySQL before moving data to RBD.')
+                utils.stop('mysql')
+
+            # mount block device to temporary location and copy the data
+            utils.juju_log('INFO', 'Copying MySQL data to RBD.')
+            ceph.place_data_on_ceph(SERVICE_NAME, blk_device, data_src, fstype)
+
+            # Make files be owned by mysql user/pass
+            cmd = ['chown', '-R', 'mysql:mysql', data_src]
+            subprocess.check_call(cmd)
+        else:
+            utils.juju_log('INFO',
+                       'Looks like data is already on the RBD, skipping...')
+
+        if not utils.running('mysql'):
+            utils.start('mysql')
+
+    else:
+        return
+
+    utils.juju_log('INFO', 'Finish Ceph Configuration')
+
+
+def filesystem_mounted(fs):
+    return subprocess.call(['grep', '-wqs', fs, '/proc/mounts']) == 0
+
+
+def get_ceph_nodes():
     hosts = []
-    hosts.append('{}:6789'.format(utils.get_host_ip()))
-
-    for relid in utils.relation_ids('cluster'):
-        for unit in utils.relation_list(relid):
-            hosts.append(
-                '{}:6789'.format(utils.get_host_ip(
-                                    utils.relation_get('private-address',
-                                                       unit, relid)))
-                )
-
-    hosts.sort()
+    for r_id in utils.relation_ids('ceph'):
+        for unit in utils.relation_list(r_id):
+            #hosts.append(utils.relation_get_dict(relation_id=r_id,
+            #                                     remote_unit=unit)['private-address'])
+            hosts.append(utils.relation_get('private-address', unit=unit, rid=r_id))
     return hosts
 
 
-def get_cluster_leader():
-    # Obtains the unit name of the first service unit deploy.
-    # e.g. mysql-0
-    units = []
-    local = units.append(utils.get_unit_name())
-    for r_id in utils.relation_ids('cluster'):
-        for unit in utils.relation_list(r_id):
-            units.append(unit.replace('/','-'))
-
-    return min(unit for unit in units)
-
-
-def get_drbd_conf():
-    cluster_hosts = {}
-    # TODO: In MAAS private-address is the *hostname*. We need to set the
-    # private address in a relation.
-    cluster_hosts[utils.get_unit_hostname()] = utils.unit_get('private-address')
-    for r_id in utils.relation_ids('cluster'):
-        for unit in utils.relation_list(r_id):
-            cluster_hosts[unit.replace('/','-')] = \
-                utils.relation_get_dict(relation_id=r_id,
-                                  remote_unit=unit)['private-address']
-
-    conf = {
-            'block_device': config['block-device'],
-            'drbd_device': DRBD_DEVICE,
-            'units': cluster_hosts,
-                }
-    return conf
-
-
-def emit_drbd_conf():
-    # read config variables
-    drbd_conf_context = get_drbd_conf()
-    # write config file
-    with open('/etc/drbd.d/mysql.res', 'w') as drbd_conf:
-        drbd_conf.write(utils.render_template('mysql.res',
-                                              drbd_conf_context))
-
-
-def configure_drbd():
-    drbd.prepare_drbd_disk(config['block-device'])
-    drbd.modprobe_module()
-
-    emit_drbd_conf()
-    drbd.create_md(DRBD_RESOURCE)
-    drbd.bring_resource_up(DRBD_RESOURCE)
-
-    # Wait for quorum.
-    while not drbd.is_quorum_secondary():
-        time.sleep(1)
-    while not drbd.is_state_inconsistent():
-        time.sleep(1)
-
-    if utils.get_unit_name() == get_cluster_leader():
-        # clear bitmap
-        if drbd.is_quorum_secondary() and drbd.is_state_inconsistent():
-            drbd.clear_bitmap(DRBD_RESOURCE)
-        # wait for resources to be UpToDate
-        while not drbd.is_state_uptodate():
-            time.sleep(1)
-        # Make leader primary
-        if drbd.is_state_uptodate():
-            drbd.make_primary(DRBD_RESOURCE)
-        # Wait for node to become primary
-        while not drbd.is_quorum_primary():
-           time.sleep(1)
-        # Format DRBD resource
-        if drbd.is_quorum_primary():
-            drbd.format_drbd_device()
-
-    utils.stop("mysql")
-    if utils.get_unit_name() == get_cluster_leader() and drbd.is_quorum_primary():
-       drbd.put_on_drbd()
-       utils.start("mysql")
-
-    return True
-
-
 def cluster_changed():
-    # Check that we are not already configured
-    if os.path.exists(STORAGEMARKER):
-        utils.juju_log('INFO',
-                       'Block storage already configured, not reconfiguring')
-        return
-
     utils.juju_log('INFO', 'Begin cluster changed hook.')
-    if len(get_cluster_nodes()) != 2:
-        utils.juju_log('WARNING', 'Not enough nodes in cluster, bailing')
-        return
-
-    #TODO:
-    # 1. check if block-storage has been set
-    # 2. prepare block storage
-    # 3. emit DRBD conf??? -- maybe not because we need 2 nodes. Unless we call cluster_changed from cluster relationship.
-    if config['block-storage'] == "None":
-        utils.juju_log('WARNING', 'NO block storage configured, bailing')
-        return
-    elif config['block-storage'] == "drbd":
-        if config['block-device'] == "None":
-            utils.juju_log('WARNING',
-                           'NO block-device defined, cannot configure DRBD')
-            return
-        else:
-            storage_configured = configure_drbd()
-    elif config['block-storage'] == "ceph":
-        # TODO: Add support for ceph
-        pass
-
-    if not storage_configured:
-        utils.juju_log('WARNING', 'Unable to configure block storage, bailing')
-        return
-
-    # TODO: if leader fails, then marker should not be placed on secondary node, nor
-    # DRBD should be mounted on both.
-    # TODO: probably would be good idea to check that DRBD has been mounted in var/lib/mysql
-    # in primary and secondary, that should say it has been successful.
-    with open(STORAGEMARKER, 'w') as marker:
-        marker.write('done')
+
+    if config['block-size'] == "None":
+        utils.juju_log('WARNING', 'NO block storage size configured, bailing')
+        return
 
     utils.juju_log('INFO', 'End install hook.')
 
-def show_drbd():
-    import commands
-    (status, output) = commands.getstatusoutput("drbd-overview")
-    utils.juju_log('INFO', '############################################################3')
-    utils.juju_log('INFO', output)
-    utils.juju_log('INFO', '############################################################3')
 
 hooks = {
-    # TODO: cluster-relation-departed (remove drbdconfigured file)
     "cluster-relation-changed": cluster_changed,
     "ha-relation-joined": ha_relation_joined,
     "ha-relation-changed": ha_relation_changed,
+    "ceph-relation-joined": ceph_joined,
+    "ceph-relation-changed": ceph_changed,
 }
 
 # keystone-hooks gets called by symlink corresponding to the requested relation

=== modified file 'hooks/install'
--- hooks/install	2012-11-01 21:49:21 +0000
+++ hooks/install	2013-02-20 16:05:25 +0000
@@ -3,8 +3,9 @@
 apt-get update
 apt-get install -y debconf-utils python-mysqldb uuid pwgen dnsutils charm-helper-sh || exit 1
 
-PASSFILE=/var/lib/juju/mysql.passwd
+PASSFILE=/var/lib/mysql/mysql.passwd
 if ! [ -f $PASSFILE ] ; then
+  mkdir -p /var/lib/mysql
   touch $PASSFILE
 fi
 chmod 0600 $PASSFILE

=== modified file 'hooks/master-relation-changed'
--- hooks/master-relation-changed	2012-11-02 06:41:12 +0000
+++ hooks/master-relation-changed	2013-02-20 16:05:25 +0000
@@ -6,7 +6,7 @@
 
 . /usr/share/charm-helper/sh/net.sh
 
-ROOTARGS="-uroot -p`cat /var/lib/juju/mysql.passwd`"
+ROOTARGS="-uroot -p`cat /var/lib/mysql/mysql.passwd`"
 snapdir=/var/www/snaps
 mkdir -p $snapdir
 apt-get -y install apache2

=== modified file 'hooks/monitors.common.bash'
--- hooks/monitors.common.bash	2012-07-12 21:58:36 +0000
+++ hooks/monitors.common.bash	2013-02-20 16:05:25 +0000
@@ -1,4 +1,4 @@
-MYSQL="mysql -uroot -p`cat /var/lib/juju/mysql.passwd`"
+MYSQL="mysql -uroot -p`cat /var/lib/mysql/mysql.passwd`"
 monitor_user=monitors
 . /usr/share/charm-helper/sh/net.sh
 if [ -n "$JUJU_REMOTE_UNIT" ] ; then

=== modified file 'hooks/shared-db-relations'
--- hooks/shared-db-relations	2012-12-03 20:21:07 +0000
+++ hooks/shared-db-relations	2013-02-20 16:05:25 +0000
@@ -14,7 +14,9 @@
 import json
 import socket
 import os
+import utils
 
+config=json.loads(subprocess.check_output(['config-get','--format=json']))
 
 def pwgen():
     return subprocess.check_output(['pwgen', '-s', '16']).strip()
@@ -47,7 +49,7 @@
     def configure_db(hostname,
                      database,
                      username):
-        passwd_file = "/var/lib/juju/mysql-{}.passwd"\
+        passwd_file = "/var/lib/mysql/mysql-{}.passwd"\
                         .format(username)
         if hostname != local_hostname:
             remote_ip = socket.gethostbyname(hostname)
@@ -72,6 +74,11 @@
                          remote_ip, password)
         return password
 
+    if not utils.eligible_leader():
+        utils.juju_log('INFO',
+                       'MySQL service is peered, bailing shared-db relation')
+        return
+
     settings = relation_get()
     local_hostname = socket.getfqdn()
     singleset = set([ 
@@ -85,8 +92,13 @@
         password = configure_db(settings['hostname'],
                                 settings['database'],
                                 settings['username'])
-        relation_set(db_host=local_hostname,
-                     password=password)
+        if not utils.is_clustered():
+            relation_set(db_host=local_hostname,
+                         password=password)
+        else:
+            relation_set(db_host=config["vip"],
+                         password=password)
+
     else:
         # Process multiple database setup requests.
         # from incoming relation data:
@@ -121,7 +133,10 @@
                                  databases[db]['database'],
                                  databases[db]['username'])
         relation_set(**return_data)
-        relation_set(db_host=local_hostname)
+        if not utils.is_clustered():
+           relation_set(db_host=local_hostname)
+        else:
+            relation_set(db_host=config["vip"])
 
 hook = os.path.basename(sys.argv[0])
 hooks = {

=== modified file 'hooks/slave-relation-broken'
--- hooks/slave-relation-broken	2011-12-06 21:23:39 +0000
+++ hooks/slave-relation-broken	2013-02-20 16:05:25 +0000
@@ -1,8 +1,8 @@
 #!/bin/sh
 
 # Kill the replication
-mysql -uroot -p`cat /var/lib/juju/mysql.passwd` -e 'STOP SLAVE;'
-mysql -uroot -p`cat /var/lib/juju/mysql.passwd` -e 'RESET SLAVE;'
+mysql -uroot -p`cat /var/lib/mysql/mysql.passwd` -e 'STOP SLAVE;'
+mysql -uroot -p`cat /var/lib/mysql/mysql.passwd` -e 'RESET SLAVE;'
 # No longer a slave
 # XXX this may change the server-id .. not sure if thats what we
 # want!

=== modified file 'hooks/slave-relation-changed'
--- hooks/slave-relation-changed	2011-12-06 21:17:17 +0000
+++ hooks/slave-relation-changed	2013-02-20 16:05:25 +0000
@@ -2,7 +2,7 @@
 
 set -e
 
-ROOTARGS="-uroot -p`cat /var/lib/juju/mysql.passwd`"
+ROOTARGS="-uroot -p`cat /var/lib/mysql/mysql.passwd`"
 
 # Others can join that service but only the lowest will be the master
 # Note that we could be more automatic but for now we will wait for
@@ -39,7 +39,7 @@
 curl --silent --show-error $dumpurl |zcat| mysql $ROOTARGS
 # Root pw gets overwritten by import
 echo Re-setting Root Pasword -- can use ours because it hasnt been flushed
-myrootpw=`cat /var/lib/juju/mysql.passwd`
+myrootpw=`cat /var/lib/mysql/mysql.passwd`
 mysqladmin -uroot -p$myrootpw password $myrootpw
 # Debian packages expect debian-sys-maint@localhost to be root privileged and
 # configured in /etc/mysql/debian.cnf. we just broke that.. fix it

=== modified file 'hooks/upgrade-charm'
--- hooks/upgrade-charm	2012-11-02 06:41:12 +0000
+++ hooks/upgrade-charm	2013-02-20 16:05:25 +0000
@@ -2,10 +2,26 @@
 home=`dirname $0`
 # Remove any existing .service_user files, which will cause
 # new users/pw's to be generated, which is a good thing
-old_service_user_files=$(ls /var/lib/juju/$.service_user)
+old_service_user_files=$(ls /var/lib/juju/*.service_user)
 if [ -n "$old_service_user_files" ] ; then
     juju-log -l WARNING "Stale users left around, should be revoked: $(cat $old_service_user_files)"
     rm -f $old_service_user_files
 fi
+
+# Move service_user2 files to /var/lib/mysql as they are
+# now stored there to support HA clustering with ceph.
+new_service_user_files=$(ls /var/lib/juju/*.service_user2)
+if [ -n "$new_service_user_files" ]; then
+    juju-log -l INFO "Moving service_user files [$new_service_user_files] to [/var/lib/mysql]"
+    mv $new_service_user_files /var/lib/mysql/
+fi
+# Move passwd files to /var/lib/mysql as they are
+# now stored there to support HA clustering with ceph.
+password_files=$(ls /var/lib/juju/*.passwd)
+if [ -n "$password_files" ]; then
+    juju-log -l INFO "Moving passwd files [$password_files] to [/var/lib/mysql]"
+    mv $password_files /var/lib/mysql/
+fi
+
 $home/install
 exec $home/config-changed

=== modified file 'hooks/utils.py'
--- hooks/utils.py	2013-01-17 04:37:31 +0000
+++ hooks/utils.py	2013-02-20 16:05:25 +0000
@@ -383,3 +383,64 @@
         return str(ip.network)
     else:
         return None
+
+
+def is_clustered():
+    for r_id in (relation_ids('ha') or []):
+        for unit in (relation_list(r_id) or []):
+            relation_data = \
+                relation_get_dict(relation_id=r_id,
+                                  remote_unit=unit)
+            if 'clustered' in relation_data:
+                return True
+    return False
+
+
+def is_leader():
+    status = execute('crm resource show res_mysql_vip', echo=True)[0].strip()
+    hostname = execute('hostname', echo=True)[0].strip()
+    if hostname in status:
+        return True
+    else:
+        return False
+
+
+def peer_units():
+    peers = []
+    for r_id in (relation_ids('cluster') or []):
+        for unit in (relation_list(r_id) or []):
+            peers.append(unit)
+    return peers
+
+
+def oldest_peer(peers):
+    local_unit_no = os.getenv('JUJU_UNIT_NAME').split('/')[1]
+    for peer in peers:
+        remote_unit_no = peer.split('/')[1]
+        if remote_unit_no < local_unit_no:
+            return False
+    return True
+
+
+def eligible_leader():
+    if is_clustered():
+        if not is_leader():
+            juju_log('INFO', 'Deferring action to CRM leader.')
+            return False
+    else:
+        peers = peer_units()
+        if peers and not oldest_peer(peers):
+            juju_log('INFO', 'Deferring action to oldest service unit.')
+            return False
+    return True
+
+
+def is_relation_made(relation=None):
+    relation_data = []
+    for r_id in (relation_ids(relation) or []):
+        for unit in (relation_list(r_id) or []):
+            relation_data.append(relation_get_dict(relation_id=r_id,
+                                 remote_unit=unit))
+    if not relation_data:
+        return False
+    return True

=== modified file 'metadata.yaml'
--- metadata.yaml	2013-01-11 20:39:20 +0000
+++ metadata.yaml	2013-02-20 16:05:25 +0000
@@ -28,6 +28,8 @@
 requires:
   slave:
     interface: mysql-oneway-replication
+  ceph:
+    interface: ceph-client
   ha:
     interface: hacluster
     scope: container

=== modified file 'revision'
--- revision	2013-01-17 04:37:31 +0000
+++ revision	2013-02-20 16:05:25 +0000
@@ -1,1 +1,1 @@
-181
+293

=== added file 'templates/ceph.conf'
--- templates/ceph.conf	1970-01-01 00:00:00 +0000
+++ templates/ceph.conf	2013-02-20 16:05:25 +0000
@@ -0,0 +1,5 @@
+[global]
+ auth supported = {{ auth }}
+ keyring = {{ keyring }}
+ mon host = {{ mon_hosts }}
+

=== removed file 'templates/mysql.res'
--- templates/mysql.res	2013-01-17 00:56:56 +0000
+++ templates/mysql.res	1970-01-01 00:00:00 +0000
@@ -1,14 +0,0 @@
-resource mysql {
-        device {{ drbd_device }};
-        disk {{ block_device }}1;
-        meta-disk internal;
-        {% for unit, address in units.iteritems() -%}
-        on {{ unit }} {
-                address {{ address }}:7788;
-        }
-        {% endfor %}
-        syncer {
-                rate 10M;
-        }
-}
-