Merge lp:~freyes/charms/trusty/percona-cluster/lp1426508 into lp:~openstack-charmers-archive/charms/trusty/percona-cluster/next

[Felipe Reyes]

Dear OpenStack Charmers,

This patch configures mysql_monitor[0] to keep updated two properties (readable and writable) on each node member of the cluster, these properties are used to define a location rule[1][2] that instructs pacemaker to run the vip only in nodes where the writable property is set to 1.

This fixes scenarios where mysql is out of sync, stopped (manually or because it crashed).

This MP also adds functional tests to check 2 scenarios: a standard 3 nodes deployment, another where mysql service is stopped in the node where the vip is running and it's checked that the vip was migrated to another node (and connectivity is OK after the migration). To run the functional tests a file called `local.yaml` has to be dropped in the charm's directory with the vip that has to be used, for instance if you're using lxc with the local provider you can run:

$ cat<<EOD > local.yaml
vip: "10.0.3.3"
EOD

Best,

Note0: This patch doesn't take care of starting mysql service if it's stopped, it just take care of monitor the service.
Note1: this patch requires hacluster MP available at [2] to support location rules definition
Note2: to know if the node is capable of receiving read/write requests the clustercheck[3] is used

[0] https://github.com/percona/percona-pacemaker-agents/blob/master/agents/mysql_monitor
[1] http://clusterlabs.org/doc/en-US/Pacemaker/1.1-crmsh/html/Clusters_from_Scratch/_specifying_a_preferred_location.html
[2] https://code.launchpad.net/~freyes/charms/trusty/hacluster/add-location/+merge/252127
[3] http://www.percona.com/doc/percona-xtradb-cluster/5.5/faq.html#q-how-can-i-check-the-galera-node-health

[James Page (james-page)]

Felipe

This looks like a really good start to resolving this challenge; generally you changes look fine (a few inline comments) but I really would like to see upgrades for existing deployments handled as well.

This would involve re-executing the ha_relation_joined function from the upgrade-charm/config-changed hook so that corosync can reconfigure its resources as required.

review: Needs Fixing

[Felipe Reyes (freyes)]

James, this new version of the patch addresses your feedback and adds a couple of unit tests for ha-relation-joined.

Thanks,

[Felipe Reyes (freyes)]

Mario was reviewing this patch and he found a problem when mysqld is killed and the pidfile is left the agent (mysql_monitor) doesn't properly detect that mysql is not running. I filed a pull request[0] to address this scenario.

[0] https://github.com/percona/percona-pacemaker-agents/pull/53

[lp:~freyes/charms/trusty/percona-cluster/lp1426508]

68. By Felipe Reyes on 2015-04-07

mysql_monitor: Apply patch available in upstream PR #52

https://github.com/percona/percona-pacemaker-agents/pull/53

69. By Felipe Reyes on 2015-04-15

Rename target to 'test' and use AMULET_OS_VIP to handoff the vip

70. By Felipe Reyes on 2015-04-15

Add tests/charmhelpers/

71. By Felipe Reyes on 2015-04-15

Add amulet test that runs 'killall -9 mysqld' in the master node

72. By Felipe Reyes on 2015-04-17

Resync charm helpers tests/

73. By Felipe Reyes on 2015-04-17

Pull hacluster from next using openstack charm-helpers base class

Unmerged revisions

=== modified file '.bzrignore'
--- .bzrignore	2015-02-06 07:28:54 +0000
+++ .bzrignore	2015-04-07 17:10:23 +0000
@@ -2,3 +2,6 @@
 .coverage
 .pydevproject
 .project
+*.pyc
+*.pyo
+__pycache__

=== modified file 'Makefile'
--- Makefile	2014-10-02 16:12:44 +0000
+++ Makefile	2015-04-07 17:10:23 +0000
@@ -9,6 +9,10 @@
 unit_test:
 	@$(PYTHON) /usr/bin/nosetests --nologcapture unit_tests
 
+functional_test:
+	@echo Starting amulet tests...
+	@juju test -v -p AMULET_HTTP_PROXY --timeout 900
+
 bin/charm_helpers_sync.py:
 	@mkdir -p bin
 	@bzr cat lp:charm-helpers/tools/charm_helpers_sync/charm_helpers_sync.py \

=== modified file 'copyright'
--- copyright	2013-09-19 15:40:50 +0000
+++ copyright	2015-04-07 17:10:23 +0000
@@ -15,3 +15,25 @@
  .
  You should have received a copy of the GNU General Public License
  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+Files: ocf/percona/mysql_monitor
+Copyright: Copyright (c) 2013, Percona inc., Yves Trudeau, Michael Coburn
+License: GPL-2
+ This program is free software; you can redistribute it and/or modify
+ it under the terms of version 2 of the GNU General Public License as
+ published by the Free Software Foundation.
+
+ This program is distributed in the hope that it would be useful, but
+ WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+
+ Further, this software is distributed without any warranty that it is
+ free of the rightful claim of any third person regarding infringement
+ or the like.  Any license provided herein, whether implied or
+ otherwise, applies only to this software file.  Patent licenses, if
+ any, provided herein do not apply to combinations of this program with
+ other software, or any other product whatsoever.
+
+ You should have received a copy of the GNU General Public License
+ along with this program; if not, write the Free Software Foundation,
+ Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA.

=== modified file 'hooks/percona_hooks.py'
--- hooks/percona_hooks.py	2015-02-16 14:12:42 +0000
+++ hooks/percona_hooks.py	2015-04-07 17:10:23 +0000
@@ -50,6 +50,7 @@
     assert_charm_supports_ipv6,
     unit_sorted,
     get_db_helper,
+    install_mysql_ocf,
 )
 from charmhelpers.contrib.database.mysql import (
     PerconaClusterHelper,
@@ -72,6 +73,13 @@
 hooks = Hooks()
 
 LEADER_RES = 'grp_percona_cluster'
+RES_MONITOR_PARAMS = ('params user="sstuser" password="%(sstpass)s" '
+                      'pid="/var/run/mysqld/mysqld.pid" '
+                      'socket="/var/run/mysqld/mysqld.sock" '
+                      'max_slave_lag="5" '
+                      'cluster_type="pxc" '
+                      'op monitor interval="1s" timeout="30s" '
+                      'OCF_CHECK_LEVEL="1"')
 
 
 @hooks.hook('install')
@@ -155,6 +163,13 @@
         for unit in related_units(r_id):
             shared_db_changed(r_id, unit)
 
+    # (re)install pcmkr agent
+    install_mysql_ocf()
+
+    if relation_ids('ha'):
+        # make sure all the HA resources are (re)created
+        ha_relation_joined()
+
 
 @hooks.hook('cluster-relation-joined')
 def cluster_joined(relation_id=None):
@@ -387,17 +402,34 @@
         vip_params = 'params ip="%s" cidr_netmask="%s" nic="%s"' % \
                      (vip, vip_cidr, vip_iface)
 
-    resources = {'res_mysql_vip': res_mysql_vip}
-    resource_params = {'res_mysql_vip': vip_params}
+    resources = {'res_mysql_vip': res_mysql_vip,
+                 'res_mysql_monitor': 'ocf:percona:mysql_monitor'}
+    db_helper = get_db_helper()
+    cfg_passwd = config('sst-password')
+    sstpsswd = db_helper.get_mysql_password(username='sstuser',
+                                            password=cfg_passwd)
+    resource_params = {'res_mysql_vip': vip_params,
+                       'res_mysql_monitor':
+                           RES_MONITOR_PARAMS % {'sstpass': sstpsswd}}
     groups = {'grp_percona_cluster': 'res_mysql_vip'}
 
+    clones = {'cl_mysql_monitor': 'res_mysql_monitor meta interleave=true'}
+
+    colocations = {'vip_mysqld': 'inf: grp_percona_cluster cl_mysql_monitor'}
+
+    locations = {'loc_percona_cluster':
+                 'grp_percona_cluster rule inf: writable eq 1'}
+
     for rel_id in relation_ids('ha'):
         relation_set(relation_id=rel_id,
                      corosync_bindiface=corosync_bindiface,
                      corosync_mcastport=corosync_mcastport,
                      resources=resources,
                      resource_params=resource_params,
-                     groups=groups)
+                     groups=groups,
+                     clones=clones,
+                     colocations=colocations,
+                     locations=locations)
 
 
 @hooks.hook('ha-relation-changed')

=== modified file 'hooks/percona_utils.py'
--- hooks/percona_utils.py	2015-02-05 09:59:36 +0000
+++ hooks/percona_utils.py	2015-04-07 17:10:23 +0000
@@ -4,10 +4,12 @@
 import socket
 import tempfile
 import os
+import shutil
 from charmhelpers.core.host import (
     lsb_release
 )
 from charmhelpers.core.hookenv import (
+    charm_dir,
     unit_get,
     relation_ids,
     related_units,
@@ -229,3 +231,18 @@
     """Return a sorted list of unit names."""
     return sorted(
         units, lambda a, b: cmp(int(a.split('/')[-1]), int(b.split('/')[-1])))
+
+
+def install_mysql_ocf():
+    dest_dir = '/usr/lib/ocf/resource.d/percona/'
+    for fname in ['ocf/percona/mysql_monitor']:
+        src_file = os.path.join(charm_dir(), fname)
+        if not os.path.isdir(dest_dir):
+            os.makedirs(dest_dir)
+
+        dest_file = os.path.join(dest_dir, os.path.basename(src_file))
+        if not os.path.exists(dest_file):
+            log('Installing %s' % dest_file, level='INFO')
+            shutil.copy(src_file, dest_file)
+        else:
+            log("'%s' already exists, skipping" % dest_file, level='INFO')

=== added directory 'ocf'
=== added directory 'ocf/percona'
=== added file 'ocf/percona/mysql_monitor'
--- ocf/percona/mysql_monitor	1970-01-01 00:00:00 +0000
+++ ocf/percona/mysql_monitor	2015-04-07 17:10:23 +0000
@@ -0,0 +1,636 @@
+#!/bin/bash
+#
+#
+#	MySQL_Monitor agent, set writeable and readable attributes based on the 
+#   state of the local MySQL, running and read_only or not.  The agent basis is
+#   the original "Dummy" agent written by Lars Marowsky-Brée and part of the
+#   Pacemaker distribution.  Many functions are from mysql_prm.
+#
+#
+# Copyright (c) 2013, Percona inc., Yves Trudeau, Michael Coburn
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of version 2 of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+#
+# Further, this software is distributed without any warranty that it is
+# free of the rightful claim of any third person regarding infringement
+# or the like.  Any license provided herein, whether implied or
+# otherwise, applies only to this software file.  Patent licenses, if
+# any, provided herein do not apply to combinations of this program with
+# other software, or any other product whatsoever.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 59 Temple Place - Suite 330, Boston MA 02111-1307, USA.
+#
+# Version: 20131119163921
+#
+# See usage() function below for more details...
+#
+# OCF instance parameters:
+#
+#  OCF_RESKEY_state
+#  OCF_RESKEY_user
+#  OCF_RESKEY_password
+#  OCF_RESKEY_client_binary
+#  OCF_RESKEY_pid
+#  OCF_RESKEY_socket
+#  OCF_RESKEY_reader_attribute
+#  OCF_RESKEY_reader_failcount
+#  OCF_RESKEY_writer_attribute
+#  OCF_RESKEY_max_slave_lag
+#  OCF_RESKEY_cluster_type
+#  
+#######################################################################
+# Initialization:
+
+: ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/lib/heartbeat}
+. ${OCF_FUNCTIONS_DIR}/ocf-shellfuncs
+
+#######################################################################
+
+HOSTOS=`uname`
+if [ "X${HOSTOS}" = "XOpenBSD" ];then
+OCF_RESKEY_client_binary_default="/usr/local/bin/mysql"
+OCF_RESKEY_pid_default="/var/mysql/mysqld.pid"
+OCF_RESKEY_socket_default="/var/run/mysql/mysql.sock"
+else
+OCF_RESKEY_client_binary_default="/usr/bin/mysql"
+OCF_RESKEY_pid_default="/var/run/mysql/mysqld.pid"
+OCF_RESKEY_socket_default="/var/lib/mysql/mysql.sock"
+fi
+OCF_RESKEY_reader_attribute_default="readable"
+OCF_RESKEY_writer_attribute_default="writable"
+OCF_RESKEY_reader_failcount_default="1"
+OCF_RESKEY_user_default="root"
+OCF_RESKEY_password_default=""
+OCF_RESKEY_max_slave_lag_default="3600"
+OCF_RESKEY_cluster_type_default="replication"
+
+: ${OCF_RESKEY_state=${HA_RSCTMP}/mysql-monitor-${OCF_RESOURCE_INSTANCE}.state}
+: ${OCF_RESKEY_client_binary=${OCF_RESKEY_client_binary_default}}
+: ${OCF_RESKEY_pid=${OCF_RESKEY_pid_default}}
+: ${OCF_RESKEY_socket=${OCF_RESKEY_socket_default}}
+: ${OCF_RESKEY_reader_attribute=${OCF_RESKEY_reader_attribute_default}}
+: ${OCF_RESKEY_reader_failcount=${OCF_RESKEY_reader_failcount_default}}
+: ${OCF_RESKEY_writer_attribute=${OCF_RESKEY_writer_attribute_default}}
+: ${OCF_RESKEY_user=${OCF_RESKEY_user_default}}
+: ${OCF_RESKEY_password=${OCF_RESKEY_password_default}}
+: ${OCF_RESKEY_max_slave_lag=${OCF_RESKEY_max_slave_lag_default}}
+: ${OCF_RESKEY_cluster_type=${OCF_RESKEY_cluster_type_default}}
+
+MYSQL="$OCF_RESKEY_client_binary -A -S $OCF_RESKEY_socket --connect_timeout=10 --user=$OCF_RESKEY_user --password=$OCF_RESKEY_password "
+HOSTNAME=`uname -n`
+CRM_ATTR="${HA_SBIN_DIR}/crm_attribute -N $HOSTNAME "
+
+meta_data() {
+	cat <<END
+<?xml version="1.0"?>
+<!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd">
+<resource-agent name="mysql_monitor" version="0.9">
+<version>1.0</version>
+
+<longdesc lang="en">
+This agent monitors the local MySQL instance and set the writable and readable
+attributes according to what it finds.  It checks if MySQL is running and if
+it is read-only or not.
+</longdesc>
+<shortdesc lang="en">Agent monitoring mysql</shortdesc>
+
+<parameters>
+<parameter name="state" unique="1">
+<longdesc lang="en">
+Location to store the resource state in.
+</longdesc>
+<shortdesc lang="en">State file</shortdesc>
+<content type="string" default="${HA_RSCTMP}/Mysql-monitor-${OCF_RESOURCE_INSTANCE}.state" />
+</parameter>
+
+<parameter name="user" unique="0">
+<longdesc lang="en">
+MySQL user to connect to the local MySQL instance to check the slave status and 
+if the read_only variable is set.  It requires the replication client priviledge.
+</longdesc>
+<shortdesc lang="en">MySQL user</shortdesc>
+<content type="string" default="${OCF_RESKEY_user_default}" />
+</parameter>
+
+<parameter name="password" unique="0">
+<longdesc lang="en">
+Password of the mysql user to connect to the local MySQL instance
+</longdesc>
+<shortdesc lang="en">MySQL password</shortdesc>
+<content type="string" default="${OCF_RESKEY_password_default}" />
+</parameter>
+
+<parameter name="client_binary" unique="0">
+<longdesc lang="en">
+MySQL Client Binary path.
+</longdesc>
+<shortdesc lang="en">MySQL client binary path</shortdesc>
+<content type="string" default="${OCF_RESKEY_client_binary_default}" />
+</parameter>
+
+<parameter name="socket" unique="0">
+<longdesc lang="en">
+Unix socket to use in order to connect to MySQL on the host
+</longdesc>
+<shortdesc lang="en">MySQL socket</shortdesc>
+<content type="string" default="${OCF_RESKEY_socket_default}" />
+</parameter>
+
+<parameter name="pid" unique="0">
+<longdesc lang="en">
+MySQL pid file, used to verify MySQL is running.
+</longdesc>
+<shortdesc lang="en">MySQL pid file</shortdesc>
+<content type="string" default="${OCF_RESKEY_pid_default}" />
+</parameter>
+
+<parameter name="reader_attribute" unique="0">
+<longdesc lang="en">
+The reader attribute in the cib that can be used by location rules to allow or not
+reader VIPs on a host.
+</longdesc>
+<shortdesc lang="en">Reader attribute</shortdesc>
+<content type="string" default="${OCF_RESKEY_reader_attribute_default}" />
+</parameter>
+
+<parameter name="writer_attribute" unique="0">
+<longdesc lang="en">
+The reader attribute in the cib that can be used by location rules to allow or not
+reader VIPs on a host.
+</longdesc>
+<shortdesc lang="en">Writer attribute</shortdesc>
+<content type="string" default="${OCF_RESKEY_writer_attribute_default}" />
+</parameter>
+
+<parameter name="max_slave_lag" unique="0" required="0">
+<longdesc lang="en">
+The maximum number of seconds a replication slave is allowed to lag
+behind its master in order to have a reader VIP on it.
+</longdesc>
+<shortdesc lang="en">Maximum time (seconds) a MySQL slave is allowed
+to lag behind a master</shortdesc>
+<content type="integer" default="${OCF_RESKEY_max_slave_lag_default}"/>
+</parameter>
+
+<parameter name="cluster_type" unique="0" required="0">
+<longdesc lang="en">
+Type of cluster, three possible values: pxc, replication, read-only.  "pxc" is 
+for Percona XtraDB cluster, it uses the clustercheck script and set the 
+reader_attribute and writer_attribute according to the return code.  
+"replication" checks the read-only state and the slave status, only writable 
+node(s) will get the writer_attribute (and the reader_attribute) and on the 
+read-only nodes, replication status will be checked and the reader_attribute set 
+according to the state.  "read-only" will just check if the read-only variable, 
+if read/write, it will get both the writer_attribute and reader_attribute set, if
+read-only it will get only the reader_attribute.
+</longdesc>
+<shortdesc lang="en">Type of cluster</shortdesc>
+<content type="string" default="${OCF_RESKEY_cluster_type_default}"/>
+</parameter>
+
+</parameters>
+
+<actions>
+<action name="start"        timeout="20" />
+<action name="stop"         timeout="20" />
+<action name="monitor"      timeout="20" interval="10" depth="0" />
+<action name="reload"       timeout="20" />
+<action name="migrate_to"   timeout="20" />
+<action name="migrate_from" timeout="20" />
+<action name="meta-data"    timeout="5" />
+<action name="validate-all"   timeout="20" />
+</actions>
+</resource-agent>
+END
+}
+
+#######################################################################
+# Non API functions
+
+# Extract fields from slave status
+parse_slave_info() {
+    # Extracts field $1 from result of "SHOW SLAVE STATUS\G" from file $2
+    sed -ne "s/^.* $1: \(.*\)$/\1/p" < $2
+}
+
+# Read the slave status and 
+get_slave_info() {
+
+    local mysql_options tmpfile
+            
+    if [ "$master_log_file" -a "$master_host" ]; then
+        # variables are already defined, get_slave_info has been run before
+        return $OCF_SUCCESS
+    else
+        tmpfile=`mktemp ${HA_RSCTMP}/check_slave.${OCF_RESOURCE_INSTANCE}.XXXXXX`
+
+        mysql_run -Q -sw -O $MYSQL $MYSQL_OPTIONS_REPL \
+        -e 'SHOW SLAVE STATUS\G' > $tmpfile
+
+        if [ -s $tmpfile ]; then
+            master_host=`parse_slave_info Master_Host $tmpfile`
+            slave_sql=`parse_slave_info Slave_SQL_Running $tmpfile`
+            slave_io=`parse_slave_info Slave_IO_Running $tmpfile`
+            slave_io_state=`parse_slave_info Slave_IO_State $tmpfile`
+            last_errno=`parse_slave_info Last_Errno $tmpfile`
+            secs_behind=`parse_slave_info Seconds_Behind_Master $tmpfile`
+            ocf_log debug "MySQL instance has a non empty slave status"
+        else
+            # Instance produced an empty "SHOW SLAVE STATUS" output --
+            # instance is not a slave
+            
+            ocf_log err "check_slave invoked on an instance that is not a replication slave."
+	        rm -f $tmpfile
+            return $OCF_ERR_GENERIC
+        fi
+        rm -f $tmpfile
+        return $OCF_SUCCESS
+    fi
+}
+
+get_read_only() {
+    # Check if read-only is set
+    local read_only_state
+
+    read_only_state=`mysql_run -Q -sw -O $MYSQL -N $MYSQL_OPTIONS_REPL \
+        -e "SHOW VARIABLES like 'read_only'" | awk '{print $2}'`
+
+    if [ "$read_only_state" = "ON" ]; then
+        return 0
+    else
+        return 1
+    fi
+}
+
+# get the attribute controlling the readers VIP
+get_reader_attr() {
+    local attr_value
+    local rc
+
+    attr_value=`$CRM_ATTR -l reboot --name ${OCF_RESKEY_reader_attribute} --query -q`
+    rc=$?
+    if [ "$rc" -eq "0" ]; then
+        echo $attr_value
+    else
+        echo -1
+    fi
+
+}
+
+# Set the attribute controlling the readers VIP
+set_reader_attr() {
+    local curr_attr_value
+
+    curr_attr_value=$(get_reader_attr)
+    
+    if [ "$1" -eq "0" ]; then
+        if [ "$curr_attr_value" -gt "0" ]; then
+            curr_attr_value=$((${curr_attr_value}-1))    
+            $CRM_ATTR -l reboot --name ${OCF_RESKEY_reader_attribute} -v $curr_attr_value
+        else
+            $CRM_ATTR -l reboot --name ${OCF_RESKEY_reader_attribute} -v 0
+        fi
+    else
+        if [ "$curr_attr_value" -ne "$OCF_RESKEY_reader_failcount" ]; then
+            $CRM_ATTR -l reboot --name ${OCF_RESKEY_reader_attribute} -v $OCF_RESKEY_reader_failcount
+        fi
+    fi
+
+}
+
+# get the attribute controlling the writer VIP
+get_writer_attr() {
+    local attr_value
+    local rc
+
+    attr_value=`$CRM_ATTR -l reboot --name ${OCF_RESKEY_writer_attribute} --query -q`
+    rc=$?
+    if [ "$rc" -eq "0" ]; then
+        echo $attr_value
+    else
+        echo -1
+    fi
+
+}
+
+# Set the attribute controlling the writer VIP
+set_writer_attr() {
+    local curr_attr_value
+
+    curr_attr_value=$(get_writer_attr)
+    
+    if [ "$1" -ne "$curr_attr_value" ]; then
+        if [ "$1" -eq "0" ]; then
+            $CRM_ATTR -l reboot --name ${OCF_RESKEY_writer_attribute} -v 0
+        else
+            $CRM_ATTR -l reboot --name ${OCF_RESKEY_writer_attribute} -v 1
+        fi
+    fi
+}
+
+#
+# mysql_run: Run a mysql command, log its output and return the proper error code.
+# Usage:   mysql_run [-Q] [-info|-warn|-err] [-O] [-sw] <command>
+#       -Q: don't log the output of the command if it succeeds
+#       -info|-warn|-err: log the output of the command at given
+#               severity if it fails (defaults to err)
+#       -O: echo the output of the command
+#       -sw: Suppress 5.6 client warning when password is used on the command line
+# Adapted from ocf_run.
+#
+mysql_run() {
+        local rc
+        local output outputfile
+        local verbose=1
+        local returnoutput
+        local loglevel=err
+        local suppress_56_password_warning
+        local var
+
+        for var in 1 2 3 4
+        do
+            case "$1" in
+                "-Q")
+                    verbose=""
+                    shift 1;;
+                "-info"|"-warn"|"-err")
+                    loglevel=`echo $1 | sed -e s/-//g`
+                    shift 1;;
+                "-O")
+                    returnoutput=1
+                    shift 1;;
+                "-sw")
+                    suppress_56_password_warning=1
+                    shift 1;;
+
+                *)
+                    ;;
+            esac
+        done
+
+        outputfile=`mktemp ${HA_RSCTMP}/mysql_run.${OCF_RESOURCE_INSTANCE}.XXXXXX`
+        error=`"$@" 2>&1 1>$outputfile`
+        rc=$?
+        if [ "$suppress_56_password_warning" -eq 1 ]; then
+                error=`echo "$error" | egrep -v '^Warning: Using a password on the command line'`
+        fi
+        output=`cat $outputfile`
+        rm -f $outputfile
+
+        if [ $rc -eq 0 ]; then
+            if [ "$verbose" -a ! -z "$output" ]; then
+                ocf_log info "$output"
+            fi
+
+            if [ "$returnoutput" -a ! -z "$output" ]; then
+                echo "$output"
+            fi
+
+            MYSQL_LAST_ERR=$OCF_SUCCESS
+            return $OCF_SUCCESS
+        else
+            if [ ! -z "$error" ]; then                 
+                ocf_log $loglevel "$error"
+                regex='^ERROR ([[:digit:]]{4}).*'
+                if [[ $error =~ $regex ]]; then
+                    mysql_code=${BASH_REMATCH[1]}
+                    if [ -n "$mysql_code" ]; then
+                        MYSQL_LAST_ERR=$mysql_code
+                        return $rc
+                    fi
+                fi
+            else
+                ocf_log $loglevel "command failed: $*"
+            fi
+            # No output to parse so return the standard exit code.
+            MYSQL_LAST_ERR=$rc
+            return $rc
+        fi
+}
+
+
+
+
+#######################################################################
+# API functions
+
+mysql_monitor_usage() {
+	cat <<END
+usage: $0 {start|stop|monitor|migrate_to|migrate_from|validate-all|meta-data}
+
+Expects to have a fully populated OCF RA-compliant environment set.
+END
+}
+
+mysql_monitor_start() {
+    
+    # Initialise the attribute in the cib if they are not already there.
+    if [ $(get_reader_attr) -eq -1 ]; then
+        set_reader_attr 0
+    fi
+    
+    if [ $(get_writer_attr) -eq -1 ]; then
+        set_writer_attr 0
+    fi
+    
+    mysql_monitor
+    mysql_monitor_monitor
+    if [ $? =  $OCF_SUCCESS ]; then
+	return $OCF_SUCCESS
+    fi
+    touch ${OCF_RESKEY_state}
+}
+
+mysql_monitor_stop() {
+    
+    set_reader_attr 0
+    set_writer_attr 0
+    
+    mysql_monitor_monitor
+    if [ $? =  $OCF_SUCCESS ]; then
+	rm ${OCF_RESKEY_state}
+    fi
+    return $OCF_SUCCESS
+
+}
+
+# Monitor MySQL, not the agent itself
+mysql_monitor() {
+    if [  -e $OCF_RESKEY_pid ]; then
+        pid=`cat $OCF_RESKEY_pid`;
+        if [ -d /proc -a -d /proc/1 ]; then
+            [ "u$pid" != "u" -a -d /proc/$pid ]
+        else
+            kill -s 0 $pid >/dev/null 2>&1
+        fi
+
+        if [ $? -eq 0 ]; then
+
+            case ${OCF_RESKEY_cluster_type} in
+            'replication'|'REPLICATION')
+                if get_read_only; then
+                    # a slave?
+                    
+                    set_writer_attr 0
+                    
+                    get_slave_info
+                    rc=$?
+
+                    if [ $rc -eq 0 ]; then
+                       # show slave status is not empty
+                       # Is there a master_log_file defined?  (master_log_file is deleted 
+                       # by reset slave
+                       if [ "$master_log_file" ]; then
+                          # is read_only but no slave config...
+                          
+                          set_reader_attr 0
+                          
+                       else
+                          # has a slave config
+                          
+                          if [ "$slave_sql" = 'Yes' -a "$slave_io" = 'Yes' ]; then
+                              # $secs_behind can be NULL so must be tested only
+                              # if replication is OK
+                              if [ $secs_behind -gt $OCF_RESKEY_max_slave_lag ]; then
+                                  set_reader_attr 0
+                              else
+                                  set_reader_attr 1
+                              fi
+                          else
+                              set_reader_attr 0
+                          fi
+                       fi
+                    else
+                       # "SHOW SLAVE STATUS" returns an empty set if instance is not a
+                       # replication slave
+                       
+                       set_reader_attr 0
+                       
+                    fi
+                else
+                    # host is RW
+                    set_reader_attr 1
+                    set_writer_attr 1
+                fi
+                ;;
+                
+            'pxc'|'PXC')
+                pxcstat=`/usr/bin/clustercheck $OCF_RESKEY_user $OCF_RESKEY_password `
+                if [ $? -eq 0 ]; then
+                    set_reader_attr 1
+                    set_writer_attr 1
+                else
+                    set_reader_attr 0
+                    set_writer_attr 0
+                fi
+                
+                ;;
+                
+            'read-only'|'READ-ONLY')
+                if get_read_only; then
+                    set_reader_attr 1
+                    set_writer_attr 0                    
+                else
+                    set_reader_attr 1
+                    set_writer_attr 1
+                fi
+                ;;
+                
+            esac
+        else
+            ocf_log $1 "MySQL is not running, but there is a pidfile"
+            set_reader_attr 0
+            set_writer_attr 0
+        fi
+    else
+        ocf_log $1 "MySQL is not running"
+        set_reader_attr 0
+        set_writer_attr 0
+    fi
+}
+    
+mysql_monitor_monitor() {
+	# Monitor _MUST!_ differentiate correctly between running
+	# (SUCCESS), failed (ERROR) or _cleanly_ stopped (NOT RUNNING).
+	# That is THREE states, not just yes/no.
+    
+	if [ -f ${OCF_RESKEY_state} ]; then
+	    return $OCF_SUCCESS
+	fi
+	if false ; then
+		return $OCF_ERR_GENERIC
+	fi
+	return $OCF_NOT_RUNNING
+}
+
+mysql_monitor_validate() {
+    
+    # Is the state directory writable? 
+    state_dir=`dirname "$OCF_RESKEY_state"`
+    touch "$state_dir/$$"
+    if [ $? != 0 ]; then
+	return $OCF_ERR_ARGS
+    fi
+    rm "$state_dir/$$"
+
+    return $OCF_SUCCESS
+}
+
+##########################################################################
+# If DEBUG_LOG is set, make this resource agent easy to debug: set up the
+# debug log and direct all output to it.  Otherwise, redirect to /dev/null.
+# The log directory must be a directory owned by root, with permissions 0700,
+# and the log must be writable and not a symlink.
+##########################################################################
+DEBUG_LOG="/tmp/mysql_monitor.ocf.ra.debug/log"
+if [ "${DEBUG_LOG}" -a -w "${DEBUG_LOG}" -a ! -L "${DEBUG_LOG}" ]; then
+    DEBUG_LOG_DIR="${DEBUG_LOG%/*}"
+    if [ -d "${DEBUG_LOG_DIR}" ]; then
+        exec 9>>"$DEBUG_LOG"
+        exec 2>&9
+        date >&9
+        echo "$*" >&9
+        env | grep OCF_ | sort >&9
+        set -x
+    else
+        exec 9>/dev/null
+    fi
+fi
+
+
+case $__OCF_ACTION in
+meta-data)	meta_data
+		exit $OCF_SUCCESS
+		;;
+start)		mysql_monitor_start;;
+stop)		mysql_monitor_stop;;
+monitor)	mysql_monitor
+            mysql_monitor_monitor;;
+migrate_to)	ocf_log info "Migrating ${OCF_RESOURCE_INSTANCE} to ${OCF_RESKEY_CRM_meta_migrate_target}."
+	        mysql_monitor_stop
+		;;
+migrate_from)	ocf_log info "Migrating ${OCF_RESOURCE_INSTANCE} from ${OCF_RESKEY_CRM_meta_migrate_source}."
+	        mysql_monitor_start
+		;;
+reload)		ocf_log info "Reloading ${OCF_RESOURCE_INSTANCE} ..."
+		;;
+validate-all)	mysql_monitor_validate;;
+usage|help)	mysql_monitor_usage
+		exit $OCF_SUCCESS
+		;;
+*)		mysql_monitor_usage
+		exit $OCF_ERR_UNIMPLEMENTED
+		;;
+esac
+rc=$?
+ocf_log debug "${OCF_RESOURCE_INSTANCE} $__OCF_ACTION : $rc"
+exit $rc
+

=== added file 'setup.cfg'
--- setup.cfg	1970-01-01 00:00:00 +0000
+++ setup.cfg	2015-04-07 17:10:23 +0000
@@ -0,0 +1,6 @@
+[nosetests]
+verbosity=2
+with-coverage=1
+cover-erase=1
+cover-package=hooks
+

=== modified file 'templates/my.cnf'
--- templates/my.cnf	2015-03-04 15:30:55 +0000
+++ templates/my.cnf	2015-04-07 17:10:23 +0000
@@ -11,6 +11,7 @@
 
 datadir=/var/lib/mysql
 user=mysql
+pid_file = /var/run/mysqld/mysqld.pid
 
 # Path to Galera library
 wsrep_provider=/usr/lib/libgalera_smm.so

=== added directory 'tests'
=== added file 'tests/00-setup.sh'
--- tests/00-setup.sh	1970-01-01 00:00:00 +0000
+++ tests/00-setup.sh	2015-04-07 17:10:23 +0000
@@ -0,0 +1,29 @@
+#!/bin/bash -x
+# The script installs amulet and other tools needed for the amulet tests.
+
+# Get the status of the amulet package, this returns 0 of package is installed.
+dpkg -s amulet
+if [ $? -ne 0 ]; then
+  # Install the Amulet testing harness.
+  sudo add-apt-repository -y ppa:juju/stable
+  sudo apt-get update 
+  sudo apt-get install -y -q amulet juju-core charm-tools
+fi
+
+
+PACKAGES="python3 python3-yaml"
+for pkg in $PACKAGES; do
+  dpkg -s python3
+  if [ $? -ne 0 ]; then
+    sudo apt-get install -y -q $pkg
+  fi
+done
+
+
+if [ ! -f "$(dirname $0)/../local.yaml" ]; then
+  echo "To run these amulet tests a vip is needed, create a file called \
+local.yaml in the charm dir, this file must contain a 'vip', if you're \
+using the local provider with lxc you could use a free IP from the range \
+10.0.3.0/24"
+  exit 1
+fi

=== added file 'tests/10-deploy_test.py'
--- tests/10-deploy_test.py	1970-01-01 00:00:00 +0000
+++ tests/10-deploy_test.py	2015-04-07 17:10:23 +0000
@@ -0,0 +1,29 @@
+#!/usr/bin/python3
+# test percona-cluster (3 nodes)
+
+import basic_deployment
+import time
+
+
+class ThreeNode(basic_deployment.BasicDeployment):
+    def __init__(self):
+        super(ThreeNode, self).__init__(units=3)
+
+    def run(self):
+        super(ThreeNode, self).run()
+        # we are going to kill the master
+        old_master = self.master_unit
+        self.master_unit.run('sudo poweroff')
+
+        time.sleep(10)  # give some time to pacemaker to react
+        new_master = self.find_master()
+        assert new_master is not None, "master unit not found"
+        assert (new_master.info['public-address'] !=
+                    old_master.info['public-address'])
+
+        assert self.is_port_open(address=self.vip), 'cannot connect to vip'
+
+
+if __name__ == "__main__":
+    t = ThreeNode()
+    t.run()

=== added file 'tests/20-broken-mysqld.py'
--- tests/20-broken-mysqld.py	1970-01-01 00:00:00 +0000
+++ tests/20-broken-mysqld.py	2015-04-07 17:10:23 +0000
@@ -0,0 +1,38 @@
+#!/usr/bin/python3
+# test percona-cluster (3 nodes)
+
+import basic_deployment
+import time
+
+
+class ThreeNode(basic_deployment.BasicDeployment):
+    def __init__(self):
+        super(ThreeNode, self).__init__(units=3)
+
+    def run(self):
+        super(ThreeNode, self).run()
+        # we are going to kill the master
+        old_master = self.master_unit
+        print('stopping mysql in %s' % str(self.master_unit.info))
+        self.master_unit.run('sudo service mysql stop')
+
+        print('looking for the new master')
+        i = 0
+        changed = False
+        while i < 10 and not changed:
+            i += 1
+            time.sleep(5)  # give some time to pacemaker to react
+            new_master = self.find_master()
+
+            if (new_master and new_master.info['unit_name'] !=
+                    old_master.info['unit_name']):
+                changed = True
+
+        assert changed, "The master didn't change"
+
+        assert self.is_port_open(address=self.vip), 'cannot connect to vip'
+
+
+if __name__ == "__main__":
+    t = ThreeNode()
+    t.run()

=== added file 'tests/basic_deployment.py'
--- tests/basic_deployment.py	1970-01-01 00:00:00 +0000
+++ tests/basic_deployment.py	2015-04-07 17:10:23 +0000
@@ -0,0 +1,126 @@
+import amulet
+import os
+import telnetlib
+import unittest
+import yaml
+
+
+class BasicDeployment(unittest.TestCase):
+    def __init__(self, vip=None, units=1):
+        self.units = units
+        self.master_unit = None
+        self.vip = None
+        if vip:
+            self.vip = vip
+        elif 'VIP' in os.environ:
+            self.vip = os.environ.get('VIP')
+        elif os.path.isfile('local.yaml'):
+            with open('local.yaml', 'rb') as f:
+                self.cfg = yaml.safe_load(f.read())
+
+            self.vip = self.cfg.get('vip')
+        else:
+            amulet.raise_status(amulet.SKIP,
+                                ("please set the vip in local.yaml "
+                                 "to run this test suite"))
+
+    def run(self):
+        # The number of seconds to wait for the environment to setup.
+        seconds = 1200
+
+        self.d = amulet.Deployment(series="trusty")
+        self.d.add('percona-cluster', units=self.units)
+        self.d.add('hacluster')
+        self.d.relate('percona-cluster:ha', 'hacluster:ha')
+
+        cfg_percona = {'sst-password': 'ubuntu',
+                       'root-password': 't00r',
+                       'dataset-size': '128M',
+                       'vip': self.vip}
+
+        cfg_ha = {'debug': True,
+                  'corosync_mcastaddr': '226.94.1.4',
+                  'corosync_key': ('xZP7GDWV0e8Qs0GxWThXirNNYlScgi3sRTdZk/IXKD'
+                                   'qkNFcwdCWfRQnqrHU/6mb6sz6OIoZzX2MtfMQIDcXu'
+                                   'PqQyvKuv7YbRyGHmQwAWDUA4ed759VWAO39kHkfWp9'
+                                   'y5RRk/wcHakTcWYMwm70upDGJEP00YT3xem3NQy27A'
+                                   'C1w=')}
+
+        self.d.configure('percona-cluster', cfg_percona)
+        self.d.configure('hacluster', cfg_ha)
+
+        try:
+            self.d.setup(timeout=seconds)
+            self.d.sentry.wait(seconds)
+        except amulet.helpers.TimeoutError:
+            message = 'The environment did not setup in %d seconds.' % seconds
+            amulet.raise_status(amulet.SKIP, msg=message)
+        except:
+            raise
+
+        self.master_unit = self.find_master()
+        assert self.master_unit is not None, 'percona-cluster vip not found'
+
+        output, code = self.master_unit.run('sudo crm_verify --live-check')
+        assert code == 0, "'crm_verify --live-check' failed"
+
+        resources = ['res_mysql_vip']
+        resources += ['res_mysql_monitor:%d' % i for i in range(self.units)]
+
+        assert sorted(self.get_pcmkr_resources()) == sorted(resources)
+
+        for i in range(self.units):
+            uid = 'percona-cluster/%d' % i
+            unit = self.d.sentry.unit[uid]
+            assert self.is_mysqld_running(unit), 'mysql not running: %s' % uid
+
+    def find_master(self):
+        for unit_id, unit in self.d.sentry.unit.items():
+            if not unit_id.startswith('percona-cluster/'):
+                continue
+
+            # is the vip running here?
+            output, code = unit.run('sudo ip a | grep %s' % self.vip)
+            print(unit_id)
+            print(output)
+            if code == 0:
+                print('vip(%s) running in %s' % (self.vip, unit_id))
+                return unit
+
+    def get_pcmkr_resources(self, unit=None):
+        if unit:
+            u = unit
+        else:
+            u = self.master_unit
+
+        output, code = u.run('sudo crm_resource -l')
+
+        assert code == 0, 'could not get "crm resource list"'
+
+        return output.split('\n')
+
+    def is_mysqld_running(self, unit=None):
+        if unit:
+            u = unit
+        else:
+            u = self.master_unit
+
+        output, code = u.run('pidof mysqld')
+
+        if code != 0:
+            return False
+
+        return self.is_port_open(u, '3306')
+
+    def is_port_open(self, unit=None, port='3306', address=None):
+        if unit:
+            addr = unit.info['public-address']
+        elif address:
+            addr = address
+        else:
+            raise Exception('Please provide a unit or address')
+        try:
+            telnetlib.Telnet(addr, port)
+            return True
+        except TimeoutError:  # noqa this exception only available in py3
+            return False

=== added file 'unit_tests/test_percona_hooks.py'
--- unit_tests/test_percona_hooks.py	1970-01-01 00:00:00 +0000
+++ unit_tests/test_percona_hooks.py	2015-04-07 17:10:23 +0000
@@ -0,0 +1,65 @@
+import mock
+import sys
+from test_utils import CharmTestCase
+
+sys.modules['MySQLdb'] = mock.Mock()
+import percona_hooks as hooks
+
+TO_PATCH = ['log', 'config',
+            'get_db_helper',
+            'relation_ids',
+            'relation_set']
+
+
+class TestHaRelation(CharmTestCase):
+    def setUp(self):
+        CharmTestCase.setUp(self, hooks, TO_PATCH)
+
+    @mock.patch('sys.exit')
+    def test_relation_not_configured(self, exit_):
+        self.config.return_value = None
+
+        class MyError(Exception):
+            pass
+
+        def f(x):
+            raise MyError(x)
+        exit_.side_effect = f
+        self.assertRaises(MyError, hooks.ha_relation_joined)
+
+    def test_resources(self):
+        self.relation_ids.return_value = ['ha:1']
+        password = 'ubuntu'
+        helper = mock.Mock()
+        attrs = {'get_mysql_password.return_value': password}
+        helper.configure_mock(**attrs)
+        self.get_db_helper.return_value = helper
+        self.test_config.set('vip', '10.0.3.3')
+        self.test_config.set('sst-password', password)
+        def f(k):
+            return self.test_config.get(k)
+
+        self.config.side_effect = f
+        hooks.ha_relation_joined()
+
+        resources = {'res_mysql_vip': 'ocf:heartbeat:IPaddr2',
+                     'res_mysql_monitor': 'ocf:percona:mysql_monitor'}
+        resource_params = {'res_mysql_vip': ('params ip="10.0.3.3" '
+                                             'cidr_netmask="24" '
+                                             'nic="eth0"'),
+                           'res_mysql_monitor':
+                               hooks.RES_MONITOR_PARAMS % {'sstpass': 'ubuntu'}}
+        groups = {'grp_percona_cluster': 'res_mysql_vip'}
+
+        clones = {'cl_mysql_monitor': 'res_mysql_monitor meta interleave=true'}
+
+        colocations = {'vip_mysqld': 'inf: grp_percona_cluster cl_mysql_monitor'}
+
+        locations = {'loc_percona_cluster':
+                     'grp_percona_cluster rule inf: writable eq 1'}
+
+        self.relation_set.assert_called_with(
+            relation_id='ha:1', corosync_bindiface=f('ha-bindiface'),
+            corosync_mcastport=f('ha-mcastport'), resources=resources,
+            resource_params=resource_params, groups=groups,
+            clones=clones, colocations=colocations, locations=locations)

=== added file 'unit_tests/test_utils.py'
--- unit_tests/test_utils.py	1970-01-01 00:00:00 +0000
+++ unit_tests/test_utils.py	2015-04-07 17:10:23 +0000
@@ -0,0 +1,121 @@
+import logging
+import unittest
+import os
+import yaml
+
+from contextlib import contextmanager
+from mock import patch, MagicMock
+
+
+def load_config():
+    '''
+    Walk backwords from __file__ looking for config.yaml, load and return the
+    'options' section'
+    '''
+    config = None
+    f = __file__
+    while config is None:
+        d = os.path.dirname(f)
+        if os.path.isfile(os.path.join(d, 'config.yaml')):
+            config = os.path.join(d, 'config.yaml')
+            break
+        f = d
+
+    if not config:
+        logging.error('Could not find config.yaml in any parent directory '
+                      'of %s. ' % file)
+        raise Exception
+
+    return yaml.safe_load(open(config).read())['options']
+
+
+def get_default_config():
+    '''
+    Load default charm config from config.yaml return as a dict.
+    If no default is set in config.yaml, its value is None.
+    '''
+    default_config = {}
+    config = load_config()
+    for k, v in config.iteritems():
+        if 'default' in v:
+            default_config[k] = v['default']
+        else:
+            default_config[k] = None
+    return default_config
+
+
+class CharmTestCase(unittest.TestCase):
+
+    def setUp(self, obj, patches):
+        super(CharmTestCase, self).setUp()
+        self.patches = patches
+        self.obj = obj
+        self.test_config = TestConfig()
+        self.test_relation = TestRelation()
+        self.patch_all()
+
+    def patch(self, method):
+        _m = patch.object(self.obj, method)
+        mock = _m.start()
+        self.addCleanup(_m.stop)
+        return mock
+
+    def patch_all(self):
+        for method in self.patches:
+            setattr(self, method, self.patch(method))
+
+
+class TestConfig(object):
+
+    def __init__(self):
+        self.config = get_default_config()
+
+    def get(self, attr=None):
+        if not attr:
+            return self.get_all()
+        try:
+            return self.config[attr]
+        except KeyError:
+            return None
+
+    def get_all(self):
+        return self.config
+
+    def set(self, attr, value):
+        if attr not in self.config:
+            raise KeyError
+        self.config[attr] = value
+
+
+class TestRelation(object):
+
+    def __init__(self, relation_data={}):
+        self.relation_data = relation_data
+
+    def set(self, relation_data):
+        self.relation_data = relation_data
+
+    def get(self, attr=None, unit=None, rid=None):
+        if attr is None:
+            return self.relation_data
+        elif attr in self.relation_data:
+            return self.relation_data[attr]
+        return None
+
+
+@contextmanager
+def patch_open():
+    '''Patch open() to allow mocking both open() itself and the file that is
+    yielded.
+
+    Yields the mock for "open" and "file", respectively.'''
+    mock_open = MagicMock(spec=open)
+    mock_file = MagicMock(spec=file)
+
+    @contextmanager
+    def stub_open(*args, **kwargs):
+        mock_open(*args, **kwargs)
+        yield mock_file
+
+    with patch('__builtin__.open', stub_open):
+        yield mock_open, mock_file