Merge lp:~niedbalski/charms/trusty/rabbitmq-server/fix-1442443 into lp:~openstack-charmers-archive/charms/trusty/rabbitmq-server/next

[Jorge Niedbalski]

- Added the cluster_partition_handling option fixes LP: #1442443
- Refactored the ssl handling code.
- Added a generic config loader

[Billy Olsen (billy-olsen)]

Jorge,

I think some of the refactor in here is a definite improvement and makes things cleaner, thanks! I've got a few inline comments included, but the biggest thing is that the rabbitmq-server charm now has a /next and /trunk branch to match the rest of the flow of the openstack charms (since this has recently come under ownership of the ~openstack-charmers team). Can you retarget for /next?

Also, I think the README file should be updated with some discussions regarding high availability.

I'll run through some tests on this but wanted to drop some feedback now.

review: Needs Fixing

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #3202 rabbitmq-server for niedbalski mp255770
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/3202/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #2990 rabbitmq-server for niedbalski mp255770
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/2990/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #3019 rabbitmq-server for niedbalski mp255770
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/10789138/
Build: http://10.245.162.77:8080/job/charm_amulet_test/3019/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #3210 rabbitmq-server-next for niedbalski mp255820
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/3210/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #2998 rabbitmq-server-next for niedbalski mp255820
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/2998/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #3027 rabbitmq-server-next for niedbalski mp255820
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/10791489/
Build: http://10.245.162.77:8080/job/charm_amulet_test/3027/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #3005 rabbitmq-server-next for niedbalski mp255820
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/3005/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #3217 rabbitmq-server-next for niedbalski mp255820
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/3217/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #3224 rabbitmq-server-next for niedbalski mp255872
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/3224/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #3012 rabbitmq-server-next for niedbalski mp255872
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/3012/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #3041 rabbitmq-server-next for niedbalski mp255872
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/10795233/
Build: http://10.245.162.77:8080/job/charm_amulet_test/3041/

[James Page (james-page)]

Please can you review the amulet test failures - they appear to be related to SSL.

review: Needs Fixing

[Jorge Niedbalski (niedbalski)]

@james-page:

Seems that this tests has been failing since Feb 23 (I tested rev 86), because the certificates being used in amulet tests expired:

verify error:num=10:certificate has expired
notAfter=Feb 23 20:08:18 2015 GMT

This is not directly implied to my proposed change, and seems that https://bugs.launchpad.net/charms/+source/rabbitmq-server/+bug/1436014 is already keeping track of this.

I would appreciate if you can review this proposal.

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #3086 rabbitmq-server-next for niedbalski mp255872
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/3086/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #3298 rabbitmq-server-next for niedbalski mp255872
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/3298/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #3115 rabbitmq-server-next for niedbalski mp255872
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/10817321/
Build: http://10.245.162.77:8080/job/charm_amulet_test/3115/

[Billy Olsen (billy-olsen)]

With these changes, this LGTM. I tested some of the partitioning logic here and things seem fine by me. It appears that an unknown value for cluster_partition_handling is treated as ignore which might be confusing for someone if they weren't expecting that, but its hard to predict what the future holds.

The fix for LP#1436014 is already in the queue and I don't think it should particularly hold this up, however osci bails as soon as the first test fails and I can't see the full output.

I'm running the amulet tests against your change + the fix for 1436014 and will mark it based on the findings of that.

As always, thanks for the submission Jorge!

[lp:~niedbalski/charms/trusty/rabbitmq-server/fix-1442443]

89. By Liam Young on 2015-04-16

[gnuoy,trivial] Pre-release charmhelper sync

90. By James Page on 2015-04-16

[wolsen,r=james-page] Fixup SSL tests to not fail when certs expire!

91. By Jorge Niedbalski on 2015-04-17

Rebase commit

92. By Jorge Niedbalski on 2015-09-23

Rebase

93. By Jorge Niedbalski on 2015-09-23

addressing @dosaboy comments

94. By Jorge Niedbalski on 2015-09-23

addressing @dosaboy comments

95. By Jorge Niedbalski on 2015-09-24

addressing @dosaboy comments

96. By Jorge Niedbalski on 2015-09-24

addressing comments

97. By Jorge Niedbalski on 2015-09-28

Addressed @beisner comments

98. By Jorge Niedbalski on 2015-09-28

Addressed @beisner comments

Unmerged revisions

=== modified file 'config.yaml'
--- config.yaml	2015-03-03 06:05:14 +0000
+++ config.yaml	2015-04-13 17:16:27 +0000
@@ -8,7 +8,7 @@
     type: string
     default: "off"
     description: |
-      Enable SSL connections on rabbitmq, valid values are 'off', 'on', 'only'. If ssl_key, 
+      Enable SSL connections on rabbitmq, valid values are 'off', 'on', 'only'. If ssl_key,
       ssl_cert, ssl_ca are provided then then those values will be used. Otherwise
       the service will act as its own certificate authority and pass its ca cert to clients.
       For HA or clustered rabbits ssl key/cert must be provided.
@@ -92,6 +92,28 @@
     description: |
       When set to true the 'ha-mode: all' policy is applied to all the exchages
       that match the expression '^(?!amq\.).*'
+  cluster-partition-handling:
+    type: string
+    default: ignore
+    description: |
+      RabbitMQ offers three ways to deal with network partitions automatically.
+      Available modes:
+
+      ignore - Your network is reliable. All your nodes are in a rack,
+      connected with a switch, and that switch is also the route to the outside world.
+      You don't want to run any risk of any of your cluster shutting down if any other part of it fails
+      (or you have a two node cluster).
+
+      pause_minority - Your network is maybe less reliable. You have clustered across 3 AZs in EC2,
+      and you assume that only one AZ will fail at once.
+      In that scenario you want the remaining two AZs to continue working and the nodes from the failed AZ
+      to rejoin automatically and without fuss when the AZ comes back.
+
+      autoheal - Your network may not be reliable.
+      You are more concerned with continuity of service than with data integrity.
+      You may have a two node cluster.
+
+      For more information see http://www.rabbitmq.com/partitions.html.
   rbd-size:
     type: string
     default: 5G
 
=== modified file 'hooks/rabbit_utils.py'
--- hooks/rabbit_utils.py	2015-03-26 13:07:43 +0000
+++ hooks/rabbit_utils.py	2015-04-13 17:16:27 +0000
@@ -1,6 +1,4 @@
 import os
-import pwd
-import grp
 import re
 import socket
 import sys
@@ -8,6 +6,13 @@
 import glob
 import tempfile
 
+from rabbitmq_context import (
+    RabbitMQSSLContext,
+    RabbitMQClusterContext
+)
+
+from charmhelpers.core.templating import render
+
 from charmhelpers.contrib.openstack.utils import (
     get_hostname,
 )
@@ -30,8 +35,6 @@
     cmp_pkgrevno
 )
 
-from charmhelpers.core.templating import render
-
 from charmhelpers.contrib.peerstorage import (
     peer_store,
     peer_retrieve
@@ -57,7 +60,10 @@
 # the charm doesn't concern itself with template specifics etc.
 CONFIG_FILES = OrderedDict([
     (RABBITMQ_CONF, {
-        'hook_contexts': None,
+        'hook_contexts': [
+            RabbitMQSSLContext(),
+            RabbitMQClusterContext(),
+        ],
         'services': ['rabbitmq-server']
     }),
     (ENV_CONF, {
@@ -71,6 +77,32 @@
 ])
 
 
+class ConfigRenderer():
+
+    def __init__(self, config):
+        self.config_data = {}
+
+        for config_path, data in config.items():
+            if 'hook_contexts' in data and data['hook_contexts']:
+                ctxt = {}
+                for svc_context in data['hook_contexts']:
+                    ctxt.update(svc_context())
+                self.config_data[config_path] = ctxt
+
+    def write(self, config_path):
+        data = self.config_data.get(config_path, None)
+        if data:
+            log("writing config file: %s , data: %s" % (config_path,
+                str(data)), level='DEBUG')
+
+            render(os.path.basename(config_path), config_path,
+                   data, perms=0o644)
+
+    def write_all(self):
+        for service in self.config_data.keys():
+            self.write(service)
+
+
 class RabbitmqError(Exception):
     pass
 
@@ -390,40 +422,6 @@
 def disable_plugin(plugin):
     _manage_plugin(plugin, 'disable')
 
-ssl_key_file = "/etc/rabbitmq/rabbit-server-privkey.pem"
-ssl_cert_file = "/etc/rabbitmq/rabbit-server-cert.pem"
-ssl_ca_file = "/etc/rabbitmq/rabbit-server-ca.pem"
-
-
-def enable_ssl(ssl_key, ssl_cert, ssl_port,
-               ssl_ca=None, ssl_only=False, ssl_client=None):
-    uid = pwd.getpwnam("root").pw_uid
-    gid = grp.getgrnam("rabbitmq").gr_gid
-
-    for contents, path in (
-            (ssl_key, ssl_key_file),
-            (ssl_cert, ssl_cert_file),
-            (ssl_ca, ssl_ca_file)):
-        if not contents:
-            continue
-        with open(path, 'w') as fh:
-            fh.write(contents)
-        os.chmod(path, 0o640)
-        os.chown(path, uid, gid)
-
-    data = {
-        "ssl_port": ssl_port,
-        "ssl_cert_file": ssl_cert_file,
-        "ssl_key_file": ssl_key_file,
-        "ssl_client": ssl_client,
-        "ssl_ca_file": "",
-        "ssl_only": ssl_only}
-
-    if ssl_ca:
-        data["ssl_ca_file"] = ssl_ca_file
-
-    render(os.path.basename(RABBITMQ_CONF), RABBITMQ_CONF, data, perms=0o644)
-
 
 def execute(cmd, die=False, echo=False):
     """ Executes a command
 
=== added file 'hooks/rabbitmq_context.py'
--- hooks/rabbitmq_context.py	1970-01-01 00:00:00 +0000
+++ hooks/rabbitmq_context.py	2015-04-13 17:16:27 +0000
@@ -0,0 +1,122 @@
+from charmhelpers.contrib.ssl.service import ServiceCA
+
+from charmhelpers.core.hookenv import (
+    open_port,
+    close_port,
+    config,
+    log,
+    ERROR,
+)
+
+import sys
+import pwd
+import grp
+import os
+import base64
+
+import ssl_utils
+
+ssl_key_file = "/etc/rabbitmq/rabbit-server-privkey.pem"
+ssl_cert_file = "/etc/rabbitmq/rabbit-server-cert.pem"
+ssl_ca_file = "/etc/rabbitmq/rabbit-server-ca.pem"
+
+
+def convert_from_base64(v):
+    # Rabbit originally supported pem encoded key/cert in config, play
+    # nice on upgrades as we now expect base64 encoded key/cert/ca.
+    if not v:
+        return v
+    if v.startswith('-----BEGIN'):
+        return v
+    try:
+        return base64.b64decode(v)
+    except TypeError:
+        return v
+
+
+class RabbitMQSSLContext(object):
+
+    def enable_ssl(self, ssl_key, ssl_cert, ssl_port,
+                   ssl_ca=None, ssl_only=False, ssl_client=None):
+
+        uid = pwd.getpwnam("root").pw_uid
+        gid = grp.getgrnam("rabbitmq").gr_gid
+
+        for contents, path in (
+                (ssl_key, ssl_key_file),
+                (ssl_cert, ssl_cert_file),
+                (ssl_ca, ssl_ca_file)):
+
+            if not contents:
+                continue
+
+            with open(path, 'w') as fh:
+                fh.write(contents)
+
+            os.chmod(path, 0o640)
+            os.chown(path, uid, gid)
+
+        data = {
+            "ssl_port": ssl_port,
+            "ssl_cert_file": ssl_cert_file,
+            "ssl_key_file": ssl_key_file,
+            "ssl_client": ssl_client,
+            "ssl_ca_file": "",
+            "ssl_only": ssl_only
+        }
+
+        if ssl_ca:
+            data["ssl_ca_file"] = ssl_ca_file
+
+        return data
+
+    def __call__(self):
+        """
+        The legacy config support adds some additional complications.
+
+        ssl_enabled = True, ssl = off -> ssl enabled
+        ssl_enabled = False, ssl = on -> ssl enabled
+        """
+        ssl_mode, external_ca = ssl_utils.get_ssl_mode()
+
+        ctxt = {
+            'ssl_mode': ssl_mode,
+        }
+
+        if ssl_mode == 'off':
+            close_port(config('ssl_port'))
+            ssl_utils.reconfigure_client_ssl()
+            return ctxt
+
+        ssl_key = convert_from_base64(config('ssl_key'))
+        ssl_cert = convert_from_base64(config('ssl_cert'))
+        ssl_ca = convert_from_base64(config('ssl_ca'))
+        ssl_port = config('ssl_port')
+
+        # If external managed certs then we need all the fields.
+        if (ssl_mode in ('on', 'only') and any((ssl_key, ssl_cert)) and
+                not all((ssl_key, ssl_cert))):
+            log('If ssl_key or ssl_cert are specified both are required.',
+                level=ERROR)
+            sys.exit(1)
+
+        if not external_ca:
+            ssl_cert, ssl_key, ssl_ca = ServiceCA.get_service_cert()
+
+        ctxt.update(self.enable_ssl(
+            ssl_key, ssl_cert, ssl_port, ssl_ca,
+            ssl_only=(ssl_mode == "only"), ssl_client=False
+        ))
+
+        ssl_utils.reconfigure_client_ssl(True)
+        open_port(ssl_port)
+
+        return ctxt
+
+
+class RabbitMQClusterContext(object):
+
+    def __call__(self):
+        return {
+            'cluster_partition_handling': config('cluster-partition-handling'),
+        }
 
=== modified file 'hooks/rabbitmq_server_relations.py'
--- hooks/rabbitmq_server_relations.py	2015-04-01 09:33:46 +0000
+++ hooks/rabbitmq_server_relations.py	2015-04-13 17:16:27 +0000
@@ -1,5 +1,5 @@
 #!/usr/bin/python
-import base64
+
 import os
 import shutil
 import sys
@@ -8,6 +8,8 @@
 import socket
 
 import rabbit_utils as rabbit
+import ssl_utils
+
 from lib.utils import (
     chown, chmod,
     is_newer,
@@ -60,7 +62,6 @@
     service_restart,
 )
 from charmhelpers.contrib.charmsupport import nrpe
-from charmhelpers.contrib.ssl.service import ServiceCA
 
 from charmhelpers.contrib.peerstorage import (
     peer_echo,
@@ -175,7 +176,7 @@
             get_address_in_network(config('access-network'),
                                    unit_get('private-address'))
 
-    configure_client_ssl(relation_settings)
+    ssl_utils.configure_client_ssl(relation_settings)
 
     if is_clustered():
         relation_settings['clustered'] = 'true'
@@ -512,103 +513,10 @@
 MAN_PLUGIN = 'rabbitmq_management'
 
 
-def configure_client_ssl(relation_data):
-    """Configure client with ssl
-    """
-    ssl_mode, external_ca = _get_ssl_mode()
-    if ssl_mode == 'off':
-        return
-    relation_data['ssl_port'] = config('ssl_port')
-    if external_ca:
-        if config('ssl_ca'):
-            relation_data['ssl_ca'] = base64.b64encode(
-                config('ssl_ca'))
-        return
-    ca = ServiceCA.get_ca()
-    relation_data['ssl_ca'] = base64.b64encode(ca.get_ca_bundle())
-
-
-def _get_ssl_mode():
-    ssl_mode = config('ssl')
-    external_ca = False
-    # Legacy config boolean option
-    ssl_on = config('ssl_enabled')
-    if ssl_mode == 'off' and ssl_on is False:
-        ssl_mode = 'off'
-    elif ssl_mode == 'off' and ssl_on:
-        ssl_mode = 'on'
-    ssl_key = config('ssl_key')
-    ssl_cert = config('ssl_cert')
-    if all((ssl_key, ssl_cert)):
-        external_ca = True
-    return ssl_mode, external_ca
-
-
-def _convert_from_base64(v):
-    # Rabbit originally supported pem encoded key/cert in config, play
-    # nice on upgrades as we now expect base64 encoded key/cert/ca.
-    if not v:
-        return v
-    if v.startswith('-----BEGIN'):
-        return v
-    try:
-        return base64.b64decode(v)
-    except TypeError:
-        return v
-
-
-def reconfigure_client_ssl(ssl_enabled=False):
-    ssl_config_keys = set(('ssl_key', 'ssl_cert', 'ssl_ca'))
-    for rid in relation_ids('amqp'):
-        rdata = relation_get(rid=rid, unit=os.environ['JUJU_UNIT_NAME'])
-        if not ssl_enabled and ssl_config_keys.intersection(rdata):
-            # No clean way to remove entirely, but blank them.
-            relation_set(relation_id=rid, ssl_key='', ssl_cert='', ssl_ca='')
-        elif ssl_enabled and not ssl_config_keys.intersection(rdata):
-            configure_client_ssl(rdata)
-            relation_set(relation_id=rid, **rdata)
-
-
-def configure_rabbit_ssl():
-    """
-    The legacy config support adds some additional complications.
-
-    ssl_enabled = True, ssl = off -> ssl enabled
-    ssl_enabled = False, ssl = on -> ssl enabled
-    """
-    ssl_mode, external_ca = _get_ssl_mode()
-
-    if ssl_mode == 'off':
-        if os.path.exists(rabbit.RABBITMQ_CONF):
-            os.remove(rabbit.RABBITMQ_CONF)
-        close_port(config('ssl_port'))
-        reconfigure_client_ssl()
-        return
-    ssl_key = _convert_from_base64(config('ssl_key'))
-    ssl_cert = _convert_from_base64(config('ssl_cert'))
-    ssl_ca = _convert_from_base64(config('ssl_ca'))
-    ssl_port = config('ssl_port')
-
-    # If external managed certs then we need all the fields.
-    if (ssl_mode in ('on', 'only') and any((ssl_key, ssl_cert)) and
-            not all((ssl_key, ssl_cert))):
-        log('If ssl_key or ssl_cert are specified both are required.',
-            level=ERROR)
-        sys.exit(1)
-
-    if not external_ca:
-        ssl_cert, ssl_key, ssl_ca = ServiceCA.get_service_cert()
-
-    rabbit.enable_ssl(
-        ssl_key, ssl_cert, ssl_port, ssl_ca,
-        ssl_only=(ssl_mode == "only"), ssl_client=False)
-    reconfigure_client_ssl(True)
-    open_port(ssl_port)
-
-
 @hooks.hook('config-changed')
 @restart_on_change(rabbit.restart_map())
 def config_changed():
+
     if config('prefer-ipv6'):
         rabbit.assert_charm_supports_ipv6()
 
@@ -638,9 +546,9 @@
         rabbit.disable_plugin(MAN_PLUGIN)
         close_port(55672)
 
-    configure_rabbit_ssl()
-
     rabbit.set_all_mirroring_queues(config('mirroring-queues'))
+    rabbit.ConfigRenderer(
+        rabbit.CONFIG_FILES).write_all()
 
     if is_relation_made("ha"):
         ha_is_active_active = config("ha-vip-only")
 
=== added file 'hooks/ssl_utils.py'
--- hooks/ssl_utils.py	1970-01-01 00:00:00 +0000
+++ hooks/ssl_utils.py	2015-04-13 17:16:27 +0000
@@ -0,0 +1,66 @@
+from charmhelpers.contrib.ssl.service import ServiceCA
+
+from charmhelpers.core.hookenv import (
+    config,
+    relation_ids,
+    relation_set,
+    relation_get,
+    log,
+    WARNING,
+    local_unit,
+)
+
+import base64
+
+
+def get_ssl_mode():
+    ssl_mode = config('ssl')
+    external_ca = False
+    # Legacy config boolean option
+    ssl_enabled = config('ssl_enabled')
+
+    if ssl_enabled:
+        log("Deprecated ssl_enabled config option is True -\
+        this should be set to False and 'ssl' config set to 'on' instead.",
+            level=WARNING)
+        ssl_mode = 'on'
+    elif ssl_mode == 'on':
+        ssl_mode = 'on'
+    else:
+        ssl_mode = 'off'
+
+    ssl_key = config('ssl_key')
+    ssl_cert = config('ssl_cert')
+
+    if all((ssl_key, ssl_cert)):
+        external_ca = True
+
+    return ssl_mode, external_ca
+
+
+def configure_client_ssl(relation_data):
+    """Configure client with ssl
+    """
+    ssl_mode, external_ca = get_ssl_mode()
+    if ssl_mode == 'off':
+        return
+    relation_data['ssl_port'] = config('ssl_port')
+    if external_ca:
+        if config('ssl_ca'):
+            relation_data['ssl_ca'] = base64.b64encode(
+                config('ssl_ca'))
+        return
+    ca = ServiceCA.get_ca()
+    relation_data['ssl_ca'] = base64.b64encode(ca.get_ca_bundle())
+
+
+def reconfigure_client_ssl(ssl_enabled=False):
+    ssl_config_keys = set(('ssl_key', 'ssl_cert', 'ssl_ca'))
+    for rid in relation_ids('amqp'):
+        rdata = relation_get(rid=rid, unit=local_unit())
+        if not ssl_enabled and ssl_config_keys.intersection(rdata):
+            # No clean way to remove entirely, but blank them.
+            relation_set(relation_id=rid, ssl_key='', ssl_cert='', ssl_ca='')
+        elif ssl_enabled and not ssl_config_keys.intersection(rdata):
+            configure_client_ssl(rdata)
+            relation_set(relation_id=rid, **rdata)
 
=== modified file 'templates/rabbitmq.config'
--- templates/rabbitmq.config	2014-05-23 08:13:05 +0000
+++ templates/rabbitmq.config	2015-04-13 17:16:27 +0000
@@ -1,21 +1,35 @@
 [
-    {rabbit, [    
-{% if ssl_only %} 
+    {rabbit, [
+{% if ssl_only %}
         {tcp_listeners, []},
 {% else %}
     	{tcp_listeners, [5672]},
 {% endif %}
-    	{ssl_listeners, [{{ ssl_port }}]},
+{% if ssl_port %}
+     	{ssl_listeners, [{{ ssl_port }}]},
+{% endif %}
+{% if ssl_mode == "on" %}
     	{ssl_options, [
  	        {verify, verify_peer},
-{% if ssl_client %} 
-                {fail_if_no_peer_cert, true}, 
+{% if ssl_client %}
+                {fail_if_no_peer_cert, true},
 {% else %}
                 {fail_if_no_peer_cert, false},
-{% endif %}{% if ssl_ca_file %} 
-                {cacertfile, "{{ ssl_ca_file }}"}, {% endif %}
-        	{certfile, "{{ ssl_cert_file }}"},
-        	{keyfile, "{{ ssl_key_file }}"}
-    	]}
+{% endif %}
+           {% if ssl_ca_file %}
+                {cacertfile, "{{ ssl_ca_file }}"},
+           {% endif %}
+           {% if ssl_cert_file %}
+                {certfile, "{{ ssl_cert_file }}"},
+           {% endif %}
+           {% if ssl_key_file %}
+                {keyfile, "{{ ssl_key_file }}"}
+           {% endif %}
+    	]},
+{% endif %}
+
+        {% if cluster_partition_handling %}
+        {cluster_partition_handling, {{ cluster_partition_handling }}}
+        {% endif %}
     ]}
-].    
 \ No newline at end of file
+].
 
=== added file 'tests/50_test_cluster_partition.py'
--- tests/50_test_cluster_partition.py	1970-01-01 00:00:00 +0000
+++ tests/50_test_cluster_partition.py	2015-04-13 17:16:27 +0000
@@ -0,0 +1,31 @@
+#!/usr/bin/python
+#
+# This Amulet test deploys rabbitmq-server
+#
+# Note: We use python2, because pika doesn't support python3
+import amulet
+
+# The number of seconds to wait for the environment to setup.
+seconds = 1200 
+d = amulet.Deployment(series="trusty")
+
+d.add('rabbitmq-server', units=1)
+# Create a configuration.
+configuration = {'cluster-partition-handling': "autoheal"} 
+d.configure('rabbitmq-server', configuration) 
+
+d.expose('rabbitmq-server') 
+try:
+    d.setup(timeout=seconds)
+    d.sentry.wait(seconds) 
+except amulet.helpers.TimeoutError:
+    message = 'The environment did not setup in %d seconds.' % seconds
+    amulet.raise_status(amulet.SKIP, msg=message)
+except:
+    raise 
+
+rabbit_unit = d.sentry.unit['rabbitmq-server/0'] 
+output, code = rabbit_unit.run("grep autoheal /etc/rabbitmq/rabbitmq.config") 
+
+if code != 0 or output == "":
+    amulet.raise_status(amulet.FAIL, msg="didn't find autoheal")
 
=== added file 'unit_tests/test_rabbitmq_context.py'
--- unit_tests/test_rabbitmq_context.py	1970-01-01 00:00:00 +0000
+++ unit_tests/test_rabbitmq_context.py	2015-04-13 17:16:27 +0000
@@ -0,0 +1,79 @@
+import rabbitmq_context
+
+import mock
+import unittest
+
+
+class TestRabbitMQSSLContext(unittest.TestCase):
+
+    @mock.patch("rabbitmq_context.config")
+    @mock.patch("rabbitmq_context.close_port")
+    @mock.patch("rabbitmq_context.ssl_utils.reconfigure_client_ssl")
+    @mock.patch("rabbitmq_context.ssl_utils.get_ssl_mode")
+    def test_context_ssl_off(self, get_ssl_mode, reconfig_ssl, close_port,
+                             config):
+        get_ssl_mode.return_value = ("off", "off")
+        self.assertEqual(rabbitmq_context.RabbitMQSSLContext().__call__(), {
+            "ssl_mode": "off"
+        })
+
+        close_port.assert_called_once()
+        reconfig_ssl.assert_called_once()
+
+    @mock.patch("rabbitmq_context.open_port")
+    @mock.patch("rabbitmq_context.os.chmod")
+    @mock.patch("rabbitmq_context.os.chown")
+    @mock.patch("rabbitmq_context.pwd.getpwnam")
+    @mock.patch("rabbitmq_context.grp.getgrnam")
+    @mock.patch("rabbitmq_context.config")
+    @mock.patch("rabbitmq_context.close_port")
+    @mock.patch("rabbitmq_context.ssl_utils.reconfigure_client_ssl")
+    @mock.patch("rabbitmq_context.ssl_utils.get_ssl_mode")
+    def test_context_ssl_on(self, get_ssl_mode, reconfig_ssl, close_port,
+                            config, gr, pw, chown, chmod, open_port):
+
+        get_ssl_mode.return_value = ("on", "on")
+
+        def config_get(n):
+            return None
+
+        config.side_effect = config_get
+
+        def pw(name):
+            class Uid(object):
+                pw_uid = 1
+                gr_gid = 100
+            return Uid()
+
+        pw.side_effect = pw
+        gr.side_effect = pw
+
+        m = mock.mock_open()
+        with mock.patch('rabbitmq_context.open', m, create=True):
+            self.assertEqual(
+                rabbitmq_context.RabbitMQSSLContext().__call__(), {
+                    "ssl_port": None,
+                    "ssl_cert_file": "/etc/rabbitmq/rabbit-server-cert.pem",
+                    "ssl_key_file": '/etc/rabbitmq/rabbit-server-privkey.pem',
+                    "ssl_client": False,
+                    "ssl_ca_file": "",
+                    "ssl_only": False,
+                    "ssl_mode": "on",
+                })
+
+        reconfig_ssl.assert_called_once()
+        open_port.assert_called_once()
+
+
+class TestRabbitMQClusterContext(unittest.TestCase):
+
+    @mock.patch("rabbitmq_context.config")
+    def test_context_ssl_off(self, config):
+        config.return_value = "ignore"
+
+        self.assertEqual(
+            rabbitmq_context.RabbitMQClusterContext().__call__(), {
+                'cluster_partition_handling': "ignore"
+            })
+
+        config.assert_called_once_with("cluster-partition-handling")
 
=== modified file 'unit_tests/test_rabbitmq_server_relations.py'
--- unit_tests/test_rabbitmq_server_relations.py	2015-03-03 22:59:25 +0000
+++ unit_tests/test_rabbitmq_server_relations.py	2015-04-13 17:16:27 +0000
@@ -37,7 +37,7 @@
     @patch('rabbitmq_server_relations.relation_set')
     @patch('apt_pkg.Cache')
     @patch('rabbitmq_server_relations.is_clustered')
-    @patch('rabbitmq_server_relations.configure_client_ssl')
+    @patch('rabbitmq_server_relations.ssl_utils.configure_client_ssl')
     @patch('rabbitmq_server_relations.unit_get')
     @patch('rabbitmq_server_relations.relation_get')
     @patch('rabbitmq_server_relations.is_elected_leader')
@@ -87,7 +87,7 @@
     @patch('rabbitmq_server_relations.relation_set')
     @patch('apt_pkg.Cache')
     @patch('rabbitmq_server_relations.is_clustered')
-    @patch('rabbitmq_server_relations.configure_client_ssl')
+    @patch('rabbitmq_server_relations.ssl_utils.configure_client_ssl')
     @patch('rabbitmq_server_relations.unit_get')
     @patch('rabbitmq_server_relations.relation_get')
     @patch('rabbitmq_server_relations.is_elected_leader')