Merge lp:~hopem/charms/trusty/percona-cluster/min-cluster-size into lp:~openstack-charmers-archive/charms/trusty/percona-cluster/next

[Edward Hope-Morley]

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #6364 percona-cluster-next for hopem mp265108
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/6364/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #5996 percona-cluster-next for hopem mp265108
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/5996/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5175 percona-cluster-next for hopem mp265108
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/11893665/
Build: http://10.245.162.77:8080/job/charm_amulet_test/5175/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #6137 percona-cluster-next for hopem mp265108
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/6137/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #6505 percona-cluster-next for hopem mp265108
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/6505/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5227 percona-cluster-next for hopem mp265108
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/11908703/
Build: http://10.245.162.77:8080/job/charm_amulet_test/5227/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #6141 percona-cluster-next for hopem mp265108
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/6141/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #6509 percona-cluster-next for hopem mp265108
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/6509/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5231 percona-cluster-next for hopem mp265108
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/11909255/
Build: http://10.245.162.77:8080/job/charm_amulet_test/5231/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #6142 percona-cluster-next for hopem mp265108
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/6142/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #6510 percona-cluster-next for hopem mp265108
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/6510/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5232 percona-cluster-next for hopem mp265108
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/11909396/
Build: http://10.245.162.77:8080/job/charm_amulet_test/5232/

[David Ames (thedac)]

Ed,

This looks good and appears to solve bug 1475585.

I am new here but it seems we want to keep the *_hooks.py as clean as possible and have these helper functions in the *_utils.py. For example is_bootstrapped and get_wsrep_value and possibly all of the new functions could be moved over to the utils
functions into perconal_utils.py

This should really get its own amulet test. Setup with min-cluster-size, add min-cluster-size -1, verify percona has not started
, add another unit and verify it all comes up.

Lastly, is "boostrap-pxc mysql" idempotent? If any config change changes the config file this gets run. On a stable cluster would "boostrap-pxc mysql" being run break anything?

review: Needs Fixing

[Edward Hope-Morley (hopem)]

Thanks for the review David.

I totally agree that the helper functions should go into percona_utils.py and I will move them across.

I'll see what I can do amulet test-wise.

With regards to bootstrap-pxc, this should be safe since it will only be called once at bootstrap time (ideally once all nodes are configured but that is not a hard requirement). If it were called prior to more units being added to the cluster, on subsequent runs of config_changed() we should only ever be calling 'restart' although bootstrap-pxc is idempotent and, in fact, can be done before you have all units in the cluster. The charm will also only restart percona if the config file changes.

[Edward Hope-Morley (hopem)]

Oh and yes, bootstrap-pxc is idempotent.

[David Ames (thedac)]

> Oh and yes, bootstrap-pxc is idempotent.

Excellent.

Just to be clear bootstrap-pxc *will* be run more than once. After the cluster has been bootstrapped any config-changed run that changes the config file will "re-bootstrap" on the leader. This may not be your intent.

In config-changed bootstrapped is defaulted to False and therefore the leader will run render_config_restart_on_changed with bootstrap=True:

            elif clustered and is_leader():
                log("Leader unit - bootstrap required=%s" % (not bootstrapped),
                    DEBUG)
                render_config_restart_on_changed(clustered, hosts,
                                                 bootstrap=not bootstrapped)

And in render_config_restart_on_changed if the config file has been changed it will "re-bootstrap" rather than restart:

    if file_hash(MY_CNF) != pre_hash:
        if bootstrap:
            service('bootstrap-pxc', 'mysql')
            notify_bootstrapped()
            update_shared_db_rels()
        else:
            service_restart('mysql')

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #6248 percona-cluster-next for hopem mp265502
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/6248/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #6616 percona-cluster-next for hopem mp265502
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/6616/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5248 percona-cluster-next for hopem mp265502
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/11919293/
Build: http://10.245.162.77:8080/job/charm_amulet_test/5248/

[Edward Hope-Morley (hopem)]

I have done a full amulet test run with this charm (tests enabled in Makefile) and I get 100% success locally. I notice that the way amulet is responding to amulet.SKIP in osci is different to how it is handled locally i.e. I get:

juju-test.conductor.10-deploy_test.py DEBUG : Running 10-deploy_test.py (tests/10-deploy_test.py)
juju-test.conductor.10-deploy_test.py DEBUG : Please set the vip in local.yaml or env var AMULET_OS_VIP to run this test suite

juju-test.conductor.10-deploy_test.py DEBUG : Got exit code: 100
juju-test.conductor.10-deploy_test.py RESULT : ↷
juju-test.conductor DEBUG : Tearing down lxc juju environment
juju-test.conductor DEBUG : Calling "juju destroy-environment -y lxc"

yet OSCI says:

juju-test.conductor.10-deploy_test.py DEBUG : Running 10-deploy_test.py (tests/10-deploy_test.py)
juju-test.conductor.10-deploy_test.py DEBUG : Please set the vip in local.yaml or env var AMULET_OS_VIP to run this test suite

juju-test.conductor.10-deploy_test.py DEBUG : Got exit code: 100
juju-test.conductor.10-deploy_test.py RESULT : SKIP
juju-test.conductor INFO : Breaking here as requested by --set-e

[lp:~hopem/charms/trusty/percona-cluster/min-cluster-size]

66. By Edward Hope-Morley on 2015-07-22

[hopem,r=]

Add min-cluster-size config option. This allows the charm to wait
for a minimum number of peers to join before bootstrapping
percona and allowing relations to access the database.

Closes-Bug: 1475585

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #6249 percona-cluster-next for hopem mp265502
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/6249/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #6617 percona-cluster-next for hopem mp265502
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/6617/

[Edward Hope-Morley (hopem)]

Hmm on closer inspection it appears that SKIP is being treated as a failure for me too (if i remove the vip config). Unlike OSCI I am not using --set-e but I am also not using --fail-on-skip so they should not be getting treated as failures. I'm gonna re-disable amulet here for now until this gets sorted.

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5249 percona-cluster-next for hopem mp265502
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/11919482/
Build: http://10.245.162.77:8080/job/charm_amulet_test/5249/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #6619 percona-cluster-next for hopem mp265502
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/6619/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #6251 percona-cluster-next for hopem mp265502
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/6251/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5251 percona-cluster-next for hopem mp265502
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/11919598/
Build: http://10.245.162.77:8080/job/charm_amulet_test/5251/

[Ryan Beisner (1chb1n)]

FYI: in UOSCI, we use --set-e so that the runner will keep the juju environment after a failed test exits. We do that so that we can collect juju unit logs. Otherwise, you'd just get "I failed."

However, based on the juju test -h, I also don't believe the intended behavior is to have a SKIP invoke --set-e.

[Ryan Beisner (1chb1n)]

AMULET_OS_VIP (and other network environment variables) are now calculated and exported to dynamically represent the IP space of each job's arbitrary jenkins slave.

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5253 percona-cluster-next for hopem mp265502
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/5253/

[Ryan Beisner (1chb1n)]

Woomp there it is!

[Ryan Beisner (1chb1n)]

Amulet results from #5253 for those without private jenkins access: http://paste.ubuntu.com/11920594/

[Ryan Beisner (1chb1n)]

Also just fyi, an example of what now gets passed to all uosci amulet jobs:
http://paste.ubuntu.com/11920597/

[James Page (james-page)]

review: Approve

=== modified file 'Makefile'
--- Makefile	2015-04-20 10:53:43 +0000
+++ Makefile	2015-07-22 13:55:56 +0000
@@ -13,8 +13,8 @@
 	@echo Starting amulet tests...
 	#NOTE(beisner): can remove -v after bug 1320357 is fixed
 	#   https://bugs.launchpad.net/amulet/+bug/1320357
-	# @juju test -v -p AMULET_HTTP_PROXY,AMULET_OS_VIP --timeout 2700
-	echo "Tests disables; http://pad.lv/1446169"
+	@juju test -v -p AMULET_HTTP_PROXY,AMULET_OS_VIP --timeout 2700
+	#echo "Tests disables; http://pad.lv/1446169"
 
 bin/charm_helpers_sync.py:
 	@mkdir -p bin
 
=== modified file 'config.yaml'
--- config.yaml	2015-06-04 15:11:31 +0000
+++ config.yaml	2015-07-22 13:55:56 +0000
@@ -111,3 +111,9 @@
       but also can be set to any specific value for the system.
       Suffix this value with 'K','M','G', or 'T' to get the relevant kilo/mega/etc. bytes.
       If suffixed with %, one will get that percentage of system total memory devoted.
+  min-cluster-size:
+    type: int
+    default:
+    description: |
+      Minimum number of units expected to exist before charm will attempt to
+      bootstrap percona cluster. If no value is provided this setting is ignored.
 
=== modified file 'hooks/percona_hooks.py'
--- hooks/percona_hooks.py	2015-06-09 10:42:32 +0000
+++ hooks/percona_hooks.py	2015-07-22 13:55:56 +0000
@@ -1,17 +1,19 @@
 #!/usr/bin/python
 # TODO: Support changes to root and sstuser passwords
-
 import sys
 import json
 import os
 import socket
+import time
 
 from charmhelpers.core.hookenv import (
     Hooks, UnregisteredHookError,
     is_relation_made,
     log,
+    local_unit,
     relation_get,
     relation_set,
+    relation_id,
     relation_ids,
     related_units,
     unit_get,
@@ -20,10 +22,13 @@
     relation_type,
     DEBUG,
     INFO,
+    WARNING,
     is_leader,
 )
 from charmhelpers.core.host import (
+    service,
     service_restart,
+    service_start,
     file_hash,
     lsb_release,
 )
@@ -52,6 +57,9 @@
     get_db_helper,
     mark_seeded, seeded,
     install_mysql_ocf,
+    is_sufficient_peers,
+    notify_bootstrapped,
+    is_bootstrapped,
 )
 from charmhelpers.contrib.database.mysql import (
     PerconaClusterHelper,
@@ -131,6 +139,57 @@
     render(os.path.basename(MY_CNF), MY_CNF, context, perms=0o444)
 
 
+def render_config_restart_on_changed(clustered, hosts, bootstrap=False):
+    """Render mysql config and restart mysql service if file changes as a
+    result.
+
+    If bootstrap is True we do a bootstrap-pxc in order to bootstrap the
+    percona cluster. This should only be performed once at cluster creation
+    time.
+
+    If percona is already bootstrapped we can get away with just ensuring that
+    it is started so long as the new node to be added is guaranteed to have
+    been restarted so as to apply the new config.
+    """
+    pre_hash = file_hash(MY_CNF)
+    render_config(clustered, hosts)
+    if file_hash(MY_CNF) != pre_hash:
+        if bootstrap:
+            service('bootstrap-pxc', 'mysql')
+            notify_bootstrapped()
+            update_shared_db_rels()
+        else:
+            delay = 1
+            attempts = 0
+            max_retries = 5
+            # NOTE(dosaboy): avoid unnecessary restarts. Once mysql is started
+            # it needn't be restarted when new units join the cluster since the
+            # new units will join and apply their own config.
+            if not seeded():
+                action = service_restart
+            else:
+                action = service_start
+
+            while not action('mysql'):
+                if attempts == max_retries:
+                    raise Exception("Failed to start mysql (max retries "
+                                    "reached)")
+
+                log("Failed to start mysql - retrying in %ss" % (delay),
+                    WARNING)
+                time.sleep(delay)
+                delay += 2
+                attempts += 1
+            else:
+                mark_seeded()
+
+
+def update_shared_db_rels():
+    for r_id in relation_ids('shared-db'):
+        for unit in related_units(r_id):
+            shared_db_changed(r_id, unit)
+
+
 @hooks.hook('upgrade-charm')
 @hooks.hook('config-changed')
 def config_changed():
@@ -139,33 +198,48 @@
 
     hosts = get_cluster_hosts()
     clustered = len(hosts) > 1
-    pre_hash = file_hash(MY_CNF)
-    render_config(clustered, hosts)
-    if file_hash(MY_CNF) != pre_hash:
+    bootstrapped = is_bootstrapped()
+
+    # NOTE: only configure the cluster if we have sufficient peers. This only
+    # applies if min-cluster-size is provided and is used to avoid extraneous
+    # configuration changes and premature bootstrapping as the cluster is
+    # deployed.
+    if is_sufficient_peers():
         try:
             # NOTE(jamespage): try with leadership election
-            if clustered and not is_leader() and not seeded():
-                # Bootstrap node into seeded cluster
-                service_restart('mysql')
-                mark_seeded()
-            elif not clustered:
-                # Restart with new configuration
-                service_restart('mysql')
+            if not clustered:
+                render_config_restart_on_changed(clustered, hosts)
+            elif clustered and is_leader():
+                log("Leader unit - bootstrap required=%s" % (not bootstrapped),
+                    DEBUG)
+                render_config_restart_on_changed(clustered, hosts,
+                                                 bootstrap=not bootstrapped)
+            elif bootstrapped:
+                log("Cluster is bootstrapped - configuring mysql on this node",
+                    DEBUG)
+                render_config_restart_on_changed(clustered, hosts)
+            else:
+                log("Not configuring", DEBUG)
+
         except NotImplementedError:
             # NOTE(jamespage): fallback to legacy behaviour.
             oldest = oldest_peer(peer_units())
-            if clustered and not oldest and not seeded():
-                # Bootstrap node into seeded cluster
-                service_restart('mysql')
-                mark_seeded()
-            elif not clustered:
-                # Restart with new configuration
-                service_restart('mysql')
+            if not clustered:
+                render_config_restart_on_changed(clustered, hosts)
+            elif clustered and oldest:
+                log("Leader unit - bootstrap required=%s" % (not bootstrapped),
+                    DEBUG)
+                render_config_restart_on_changed(clustered, hosts,
+                                                 bootstrap=not bootstrapped)
+            elif bootstrapped:
+                log("Cluster is bootstrapped - configuring mysql on this node",
+                    DEBUG)
+                render_config_restart_on_changed(clustered, hosts)
+            else:
+                log("Not configuring", DEBUG)
 
     # Notify any changes to the access network
-    for r_id in relation_ids('shared-db'):
-        for unit in related_units(r_id):
-            shared_db_changed(r_id, unit)
+    update_shared_db_rels()
 
     # (re)install pcmkr agent
     install_mysql_ocf()
@@ -176,15 +250,20 @@
 
 
 @hooks.hook('cluster-relation-joined')
-def cluster_joined(relation_id=None):
+def cluster_joined():
     if config('prefer-ipv6'):
         addr = get_ipv6_addr(exc_list=[config('vip')])[0]
         relation_settings = {'private-address': addr,
                              'hostname': socket.gethostname()}
         log("Setting cluster relation: '%s'" % (relation_settings),
             level=INFO)
-        relation_set(relation_id=relation_id,
-                     relation_settings=relation_settings)
+        relation_set(relation_settings=relation_settings)
+
+    # Ensure all new peers are aware
+    cluster_state_uuid = relation_get('bootstrap-uuid', unit=local_unit())
+    if cluster_state_uuid:
+        notify_bootstrapped(cluster_rid=relation_id(),
+                            cluster_uuid=cluster_state_uuid)
 
 
 @hooks.hook('cluster-relation-departed')
@@ -282,10 +361,15 @@
 # TODO: This could be a hook common between mysql and percona-cluster
 @hooks.hook('shared-db-relation-changed')
 def shared_db_changed(relation_id=None, unit=None):
+    if not is_bootstrapped():
+        log("Percona cluster not yet bootstrapped - deferring shared-db rel "
+            "until bootstrapped", DEBUG)
+        return
+
     if not is_elected_leader(DC_RESOURCE_NAME):
         # NOTE(jamespage): relation level data candidate
-        log('Service is peered, clearing shared-db relation'
-            ' as this service unit is not the leader')
+        log('Service is peered, clearing shared-db relation '
+            'as this service unit is not the leader')
         relation_clear(relation_id)
         # Each unit needs to set the db information otherwise if the unit
         # with the info dies the settings die with it Bug# 1355848
@@ -419,7 +503,7 @@
 
     resources = {'res_mysql_vip': res_mysql_vip,
                  'res_mysql_monitor': 'ocf:percona:mysql_monitor'}
-    db_helper = get_db_helper()
+
     sstpsswd = config('sst-password')
     resource_params = {'res_mysql_vip': vip_params,
                        'res_mysql_monitor':
@@ -451,9 +535,7 @@
     if (clustered and is_elected_leader(DC_RESOURCE_NAME)):
         log('Cluster configured, notifying other services')
         # Tell all related services to start using the VIP
-        for r_id in relation_ids('shared-db'):
-            for unit in related_units(r_id):
-                shared_db_changed(r_id, unit)
+        update_shared_db_rels()
         for r_id in relation_ids('db'):
             for unit in related_units(r_id):
                 db_changed(r_id, unit, admin=False)
@@ -465,9 +547,7 @@
 @hooks.hook('leader-settings-changed')
 def leader_settings_changed():
     # Notify any changes to data in leader storage
-    for r_id in relation_ids('shared-db'):
-        for unit in related_units(r_id):
-            shared_db_changed(r_id, unit)
+    update_shared_db_rels()
 
 
 @hooks.hook('nrpe-external-master-relation-joined',
 
=== modified file 'hooks/percona_utils.py'
--- hooks/percona_utils.py	2015-05-13 10:21:30 +0000
+++ hooks/percona_utils.py	2015-07-22 13:55:56 +0000
@@ -5,6 +5,8 @@
 import tempfile
 import os
 import shutil
+import uuid
+
 from charmhelpers.core.host import (
     lsb_release
 )
@@ -20,6 +22,14 @@
     config,
     log,
     DEBUG,
+    INFO,
+    WARNING,
+    ERROR,
+    is_leader,
+)
+from charmhelpers.contrib.hahelpers.cluster import (
+    oldest_peer,
+    peer_units,
 )
 from charmhelpers.fetch import (
     apt_install,
@@ -32,6 +42,11 @@
     MySQLHelper,
 )
 
+# NOTE: python-mysqldb is installed by charmhelpers.contrib.database.mysql so
+# hence why we import here
+from MySQLdb import (
+    OperationalError
+)
 
 PACKAGES = [
     'percona-xtradb-cluster-server-5.5',
@@ -90,6 +105,29 @@
             return answers[0].address
 
 
+def is_sufficient_peers():
+    """If min-cluster-size has been provided, check that we have sufficient
+    number of peers to proceed with bootstrapping percona cluster.
+    """
+    min_size = config('min-cluster-size')
+    if min_size:
+        size = 0
+        for rid in relation_ids('cluster'):
+            size = len(related_units(rid))
+
+        # Include this unit
+        size += 1
+        if min_size > size:
+            log("Insufficient number of units to configure percona cluster "
+                "(expected=%s, got=%s)" % (min_size, size), level=INFO)
+            return False
+        else:
+            log("Sufficient units available to configure percona cluster "
+                "(>=%s)" % (min_size), level=DEBUG)
+
+    return True
+
+
 def get_cluster_hosts():
     hosts_map = {}
     hostname = get_host_ip()
@@ -246,3 +284,86 @@
             shutil.copy(src_file, dest_file)
         else:
             log("'%s' already exists, skipping" % dest_file, level='INFO')
+
+
+def get_wsrep_value(key):
+    m_helper = get_db_helper()
+    try:
+        m_helper.connect(password=m_helper.get_mysql_root_password())
+    except OperationalError:
+        log("Could not connect to db", DEBUG)
+        return None
+
+    cursor = m_helper.connection.cursor()
+    ret = None
+    try:
+        cursor.execute("show status like '%s'" % (key))
+        ret = cursor.fetchall()
+    except:
+        log("Failed to get '%s'", ERROR)
+        return None
+    finally:
+        cursor.close()
+
+    if ret:
+        return ret[0][1]
+
+    return None
+
+
+def is_bootstrapped():
+    if not is_sufficient_peers():
+        return False
+
+    uuids = []
+    rids = relation_ids('cluster') or []
+    for rid in rids:
+        units = related_units(rid)
+        units.append(local_unit())
+        for unit in units:
+            id = relation_get('bootstrap-uuid', unit=unit, rid=rid)
+            if id:
+                uuids.append(id)
+
+    if uuids:
+        if len(set(uuids)) > 1:
+            log("Found inconsistent bootstrap uuids - %s" % (uuids), WARNING)
+
+        return True
+
+    try:
+        if not is_leader():
+            return False
+    except:
+        oldest = oldest_peer(peer_units())
+        if not oldest:
+            return False
+
+    # If this is the leader but we have not yet broadcast the cluster uuid then
+    # do so now.
+    wsrep_ready = get_wsrep_value('wsrep_ready') or ""
+    if wsrep_ready.lower() in ['on', 'ready']:
+        cluster_state_uuid = get_wsrep_value('wsrep_cluster_state_uuid')
+        if cluster_state_uuid:
+            notify_bootstrapped(cluster_uuid=cluster_state_uuid)
+            return True
+
+    return False
+
+
+def notify_bootstrapped(cluster_rid=None, cluster_uuid=None):
+    if cluster_rid:
+        rids = [cluster_rid]
+    else:
+        rids = relation_ids('cluster')
+
+    log("Notifying peers that percona is bootstrapped", DEBUG)
+    if not cluster_uuid:
+        cluster_uuid = get_wsrep_value('wsrep_cluster_state_uuid')
+        if not cluster_uuid:
+            cluster_uuid = str(uuid.uuid4())
+            log("Could not determine cluster uuid so using '%s' instead" %
+                (cluster_uuid), INFO)
+
+    for rid in rids:
+        relation_set(relation_id=rid, **{'bootstrap-uuid': cluster_uuid})
 
=== modified file 'tests/10-deploy_test.py'
--- tests/10-deploy_test.py	2015-03-06 15:35:01 +0000
+++ tests/10-deploy_test.py	2015-07-22 13:55:56 +0000
@@ -19,7 +19,7 @@
         new_master = self.find_master()
         assert new_master is not None, "master unit not found"
         assert (new_master.info['public-address'] !=
-                    old_master.info['public-address'])
+                old_master.info['public-address'])
 
         assert self.is_port_open(address=self.vip), 'cannot connect to vip'
 
 
=== added file 'tests/40-test-bootstrap-single.py'
--- tests/40-test-bootstrap-single.py	1970-01-01 00:00:00 +0000
+++ tests/40-test-bootstrap-single.py	2015-07-22 13:55:56 +0000
@@ -0,0 +1,17 @@
+#!/usr/bin/env python
+# test percona-cluster (1 node)
+import basic_deployment
+
+
+class SingleNode(basic_deployment.BasicDeployment):
+    def __init__(self):
+        super(SingleNode, self).__init__(units=1)
+
+    def run(self):
+        super(SingleNode, self).run()
+        assert self.is_pxc_bootstrapped(), "Cluster not bootstrapped"
+
+
+if __name__ == "__main__":
+    t = SingleNode()
+    t.run()
 
=== added file 'tests/41-test-bootstrap-multi-notmin.py'
--- tests/41-test-bootstrap-multi-notmin.py	1970-01-01 00:00:00 +0000
+++ tests/41-test-bootstrap-multi-notmin.py	2015-07-22 13:55:56 +0000
@@ -0,0 +1,41 @@
+#!/usr/bin/env python
+# test percona-cluster (1 node)
+import basic_deployment
+
+
+class MultiNode(basic_deployment.BasicDeployment):
+    def __init__(self):
+        super(MultiNode, self).__init__(units=2)
+
+    def _get_configs(self):
+        """Configure all of the services."""
+        cfg_percona = {'sst-password': 'ubuntu',
+                       'root-password': 't00r',
+                       'dataset-size': '512M',
+                       'vip': self.vip,
+                       'min-cluster-size': 3}
+
+        cfg_ha = {'debug': True,
+                  'corosync_mcastaddr': '226.94.1.4',
+                  'corosync_key': ('xZP7GDWV0e8Qs0GxWThXirNNYlScgi3sRTdZk/IXKD'
+                                   'qkNFcwdCWfRQnqrHU/6mb6sz6OIoZzX2MtfMQIDcXu'
+                                   'PqQyvKuv7YbRyGHmQwAWDUA4ed759VWAO39kHkfWp9'
+                                   'y5RRk/wcHakTcWYMwm70upDGJEP00YT3xem3NQy27A'
+                                   'C1w=')}
+
+        configs = {'percona-cluster': cfg_percona}
+        if self.units > 1:
+            configs['hacluster'] = cfg_ha
+
+        return configs
+
+    def run(self):
+        super(MultiNode, self).run()
+        got = self.get_cluster_size()
+        msg = "Percona cluster unexpected size (wanted=%s, got=%s)" % (1, got)
+        assert got == '1', msg
+
+
+if __name__ == "__main__":
+    t = MultiNode()
+    t.run()
 
=== added file 'tests/42-test-bootstrap-multi-min.py'
--- tests/42-test-bootstrap-multi-min.py	1970-01-01 00:00:00 +0000
+++ tests/42-test-bootstrap-multi-min.py	2015-07-22 13:55:56 +0000
@@ -0,0 +1,43 @@
+#!/usr/bin/env python
+# test percona-cluster (1 node)
+import basic_deployment
+
+
+class MultiNode(basic_deployment.BasicDeployment):
+    def __init__(self):
+        super(MultiNode, self).__init__(units=3)
+
+    def _get_configs(self):
+        """Configure all of the services."""
+        cfg_percona = {'sst-password': 'ubuntu',
+                       'root-password': 't00r',
+                       'dataset-size': '512M',
+                       'vip': self.vip,
+                       'min-cluster-size': 3}
+
+        cfg_ha = {'debug': True,
+                  'corosync_mcastaddr': '226.94.1.4',
+                  'corosync_key': ('xZP7GDWV0e8Qs0GxWThXirNNYlScgi3sRTdZk/IXKD'
+                                   'qkNFcwdCWfRQnqrHU/6mb6sz6OIoZzX2MtfMQIDcXu'
+                                   'PqQyvKuv7YbRyGHmQwAWDUA4ed759VWAO39kHkfWp9'
+                                   'y5RRk/wcHakTcWYMwm70upDGJEP00YT3xem3NQy27A'
+                                   'C1w=')}
+
+        configs = {'percona-cluster': cfg_percona}
+        if self.units > 1:
+            configs['hacluster'] = cfg_ha
+
+        return configs
+
+    def run(self):
+        super(MultiNode, self).run()
+        msg = "Percona cluster failed to bootstrap"
+        assert self.is_pxc_bootstrapped(), msg
+        got = self.get_cluster_size()
+        msg = "Percona cluster unexpected size (wanted=%s, got=%s)" % (3, got)
+        assert got == '3', msg
+
+
+if __name__ == "__main__":
+    t = MultiNode()
+    t.run()
 
=== modified file 'tests/basic_deployment.py'
--- tests/basic_deployment.py	2015-04-17 10:05:16 +0000
+++ tests/basic_deployment.py	2015-07-22 13:55:56 +0000
@@ -1,8 +1,8 @@
 import amulet
+import re
 import os
 import time
 import telnetlib
-import unittest
 import yaml
 from charmhelpers.contrib.openstack.amulet.deployment import (
     OpenStackAmuletDeployment
@@ -17,19 +17,21 @@
         self.units = units
         self.master_unit = None
         self.vip = None
-        if vip:
-            self.vip = vip
-        elif 'AMULET_OS_VIP' in os.environ:
-            self.vip = os.environ.get('AMULET_OS_VIP')
-        elif os.path.isfile('local.yaml'):
-            with open('local.yaml', 'rb') as f:
-                self.cfg = yaml.safe_load(f.read())
+        if units > 1:
+            if vip:
+                self.vip = vip
+            elif 'AMULET_OS_VIP' in os.environ:
+                self.vip = os.environ.get('AMULET_OS_VIP')
+            elif os.path.isfile('local.yaml'):
+                with open('local.yaml', 'rb') as f:
+                    self.cfg = yaml.safe_load(f.read())
 
-            self.vip = self.cfg.get('vip')
-        else:
-            amulet.raise_status(amulet.SKIP,
-                                ("please set the vip in local.yaml or env var "
-                                 "AMULET_OS_VIP to run this test suite"))
+                self.vip = self.cfg.get('vip')
+            else:
+                amulet.raise_status(amulet.SKIP,
+                                    ("Please set the vip in local.yaml or "
+                                     "env var AMULET_OS_VIP to run this test "
+                                     "suite"))
 
     def _add_services(self):
         """Add services
@@ -40,16 +42,20 @@
            """
         this_service = {'name': 'percona-cluster',
                         'units': self.units}
-        other_services = [{'name': 'hacluster'}]
+        other_services = []
+        if self.units > 1:
+            other_services.append({'name': 'hacluster'})
+
         super(BasicDeployment, self)._add_services(this_service,
                                                    other_services)
 
     def _add_relations(self):
         """Add all of the relations for the services."""
-        relations = {'percona-cluster:ha': 'hacluster:ha'}
-        super(BasicDeployment, self)._add_relations(relations)
+        if self.units > 1:
+            relations = {'percona-cluster:ha': 'hacluster:ha'}
+            super(BasicDeployment, self)._add_relations(relations)
 
-    def _configure_services(self):
+    def _get_configs(self):
         """Configure all of the services."""
         cfg_percona = {'sst-password': 'ubuntu',
                        'root-password': 't00r',
@@ -64,45 +70,55 @@
                                    'y5RRk/wcHakTcWYMwm70upDGJEP00YT3xem3NQy27A'
                                    'C1w=')}
 
-        configs = {'percona-cluster': cfg_percona,
-                   'hacluster': cfg_ha}
-        super(BasicDeployment, self)._configure_services(configs)
+        configs = {'percona-cluster': cfg_percona}
+        if self.units > 1:
+            configs['hacluster'] = cfg_ha
+
+        return configs
+
+    def _configure_services(self):
+        super(BasicDeployment, self)._configure_services(self._get_configs())
 
     def run(self):
-        # The number of seconds to wait for the environment to setup.
-        seconds = 1200
-
         self._add_services()
         self._add_relations()
         self._configure_services()
         self._deploy()
 
-        i = 0
-        while i < 30 and not self.master_unit:
-            self.master_unit = self.find_master()
-            i += 1
-            time.sleep(10)
-
-        assert self.master_unit is not None, 'percona-cluster vip not found'
-
-        output, code = self.master_unit.run('sudo crm_verify --live-check')
-        assert code == 0, "'crm_verify --live-check' failed"
-
-        resources = ['res_mysql_vip']
-        resources += ['res_mysql_monitor:%d' % i for i in range(self.units)]
-
-        assert sorted(self.get_pcmkr_resources()) == sorted(resources)
+        if self.units > 1:
+            i = 0
+            while i < 30 and not self.master_unit:
+                self.master_unit = self.find_master()
+                i += 1
+                time.sleep(10)
+
+            msg = 'percona-cluster vip not found'
+            assert self.master_unit is not None, msg
+
+            _, code = self.master_unit.run('sudo crm_verify --live-check')
+            assert code == 0, "'crm_verify --live-check' failed"
+
+            resources = ['res_mysql_vip']
+            resources += ['res_mysql_monitor:%d' %
+                          i for i in range(self.units)]
+
+            assert sorted(self.get_pcmkr_resources()) == sorted(resources)
+        else:
+            self.master_unit = self.find_master(ha=False)
 
         for i in range(self.units):
             uid = 'percona-cluster/%d' % i
             unit = self.d.sentry.unit[uid]
             assert self.is_mysqld_running(unit), 'mysql not running: %s' % uid
 
-    def find_master(self):
+    def find_master(self, ha=True):
         for unit_id, unit in self.d.sentry.unit.items():
             if not unit_id.startswith('percona-cluster/'):
                 continue
 
+            if not ha:
+                return unit
+
             # is the vip running here?
             output, code = unit.run('sudo ip a | grep "inet %s/"' % self.vip)
             print('---')
@@ -130,13 +146,37 @@
         else:
             u = self.master_unit
 
-        output, code = u.run('pidof mysqld')
-
+        _, code = u.run('pidof mysqld')
         if code != 0:
+            print("ERROR: command returned non-zero '%s'" % (code))
             return False
 
         return self.is_port_open(u, '3306')
 
+    def get_wsrep_value(self, attr, unit=None):
+        if unit:
+            u = unit
+        else:
+            u = self.master_unit
+
+        cmd = ("mysql -uroot -pt00r -e\"show status like '%s';\"| "
+               "grep %s" % (attr, attr))
+        output, code = u.run(cmd)
+        if code != 0:
+            print("ERROR: command returned non-zero '%s'" % (code))
+            return ""
+
+        value = re.search(r"^.+?\s+(.+)", output).group(1)
+        print("%s = %s" % (attr, value))
+        return value
+
+    def is_pxc_bootstrapped(self, unit=None):
+        value = self.get_wsrep_value('wsrep_ready', unit)
+        return value.lower() in ['on', 'ready']
+
+    def get_cluster_size(self, unit=None):
+        return self.get_wsrep_value('wsrep_cluster_size', unit)
+
     def is_port_open(self, unit=None, port='3306', address=None):
         if unit:
             addr = unit.info['public-address']
@@ -144,8 +184,10 @@
             addr = address
         else:
             raise Exception('Please provide a unit or address')
+
         try:
             telnetlib.Telnet(addr, port)
             return True
         except TimeoutError:  # noqa this exception only available in py3
+            print("ERROR: could not connect to %s:%s" % (addr, port))
             return False
 
=== modified file 'unit_tests/test_percona_utils.py'
--- unit_tests/test_percona_utils.py	2014-10-13 12:38:14 +0000
+++ unit_tests/test_percona_utils.py	2015-07-22 13:55:56 +0000
@@ -128,3 +128,20 @@
                                                    '0.0.0.0': 'hostB'})
         mock_rel_get.assert_called_with(rid=2, unit=4)
         self.assertEqual(hosts, ['hostA', 'hostB'])
+
+    @mock.patch.object(percona_utils, 'related_units')
+    @mock.patch.object(percona_utils, 'relation_ids')
+    @mock.patch.object(percona_utils, 'config')
+    def test_is_sufficient_peers(self, mock_config, mock_relation_ids,
+                                 mock_related_units):
+        _config = {'min-cluster-size': None}
+        mock_config.side_effect = lambda key: _config.get(key)
+        self.assertTrue(percona_utils.is_sufficient_peers())
+
+        mock_relation_ids.return_value = ['cluster:0']
+        mock_related_units.return_value = ['test/0']
+        _config = {'min-cluster-size': 3}
+        self.assertFalse(percona_utils.is_sufficient_peers())
+
+        mock_related_units.return_value = ['test/0', 'test/1']
+        self.assertTrue(percona_utils.is_sufficient_peers())