Merge lp:~niedbalski/charms/trusty/rabbitmq-server/fix-1442443 into lp:~openstack-charmers-archive/charms/trusty/rabbitmq-server/next

[Jorge Niedbalski]

- Added the cluster_partition_handling option fixes LP: #1442443
- Refactored the ssl handling code.
- Added a generic config loader

[Billy Olsen (billy-olsen)]

Jorge,

I think some of the refactor in here is a definite improvement and makes things cleaner, thanks! I've got a few inline comments included, but the biggest thing is that the rabbitmq-server charm now has a /next and /trunk branch to match the rest of the flow of the openstack charms (since this has recently come under ownership of the ~openstack-charmers team). Can you retarget for /next?

Also, I think the README file should be updated with some discussions regarding high availability.

I'll run through some tests on this but wanted to drop some feedback now.

review: Needs Fixing

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #3202 rabbitmq-server for niedbalski mp255770
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/3202/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #2990 rabbitmq-server for niedbalski mp255770
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/2990/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #3019 rabbitmq-server for niedbalski mp255770
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/10789138/
Build: http://10.245.162.77:8080/job/charm_amulet_test/3019/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #3210 rabbitmq-server-next for niedbalski mp255820
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/3210/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #2998 rabbitmq-server-next for niedbalski mp255820
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/2998/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #3027 rabbitmq-server-next for niedbalski mp255820
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/10791489/
Build: http://10.245.162.77:8080/job/charm_amulet_test/3027/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #3005 rabbitmq-server-next for niedbalski mp255820
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/3005/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #3217 rabbitmq-server-next for niedbalski mp255820
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/3217/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #3224 rabbitmq-server-next for niedbalski mp255872
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/3224/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #3012 rabbitmq-server-next for niedbalski mp255872
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/3012/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #3041 rabbitmq-server-next for niedbalski mp255872
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/10795233/
Build: http://10.245.162.77:8080/job/charm_amulet_test/3041/

[James Page (james-page)]

Please can you review the amulet test failures - they appear to be related to SSL.

review: Needs Fixing

[Jorge Niedbalski (niedbalski)]

@james-page:

Seems that this tests has been failing since Feb 23 (I tested rev 86), because the certificates being used in amulet tests expired:

verify error:num=10:certificate has expired
notAfter=Feb 23 20:08:18 2015 GMT

This is not directly implied to my proposed change, and seems that https://bugs.launchpad.net/charms/+source/rabbitmq-server/+bug/1436014 is already keeping track of this.

I would appreciate if you can review this proposal.

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #3086 rabbitmq-server-next for niedbalski mp255872
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/3086/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #3298 rabbitmq-server-next for niedbalski mp255872
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/3298/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #3115 rabbitmq-server-next for niedbalski mp255872
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/10817321/
Build: http://10.245.162.77:8080/job/charm_amulet_test/3115/

[Billy Olsen (billy-olsen)]

With these changes, this LGTM. I tested some of the partitioning logic here and things seem fine by me. It appears that an unknown value for cluster_partition_handling is treated as ignore which might be confusing for someone if they weren't expecting that, but its hard to predict what the future holds.

The fix for LP#1436014 is already in the queue and I don't think it should particularly hold this up, however osci bails as soon as the first test fails and I can't see the full output.

I'm running the amulet tests against your change + the fix for 1436014 and will mark it based on the findings of that.

As always, thanks for the submission Jorge!

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #3235 rabbitmq-server-next for niedbalski mp256488
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/3235/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #3447 rabbitmq-server-next for niedbalski mp256488
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/3447/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #3245 rabbitmq-server-next for niedbalski mp256488
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/10833178/
Build: http://10.245.162.77:8080/job/charm_amulet_test/3245/

[Billy Olsen (billy-olsen)]

Ah the amulet tests look like they failed because the changes were likely based on the lp:charms/trusty/rabbitmq-server which is the trunk and not ~openstack-charmers/charms/trusty/rabbitmq-server/next. The lp:charms/trusty/rabbitmq-server branch doesn't contain the changes required in the amulet tests, however they do pass when I run them locally. I'll merge these in and push up a branch of mine just to get OSCI to vote on it (since I can't push to Jorge's branch) so I don't miss anything.

In local runs, the code changes look good and I'll be happy to merge them after OSCI does a + vote on them.

[Billy Olsen (billy-olsen)]

Jorge,

After the rebase and test with OSCI it was failing lint test (see Full lint test output: http://paste.ubuntu.com/10836031/).

I missed the lint issue so I rechecked what I was running and I made a mistake. The runs from my local amulet tests result in two failures (http://paste.ubuntu.com/10836873/) in tests 30-ssl-only and 40_test_mirroring_queues.py.

If this is based off of the ~openstack-charmers/charms/trusty/rabbitmq-server/next branch then the amulet tests should work for you locally.

review: Needs Fixing

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #3367 rabbitmq-server-next for niedbalski mp256675
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/3367/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #3579 rabbitmq-server-next for niedbalski mp256675
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/3579/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #3336 rabbitmq-server-next for niedbalski mp256675
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/10839461/
Build: http://10.245.162.77:8080/job/charm_amulet_test/3336/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #4238 rabbitmq-server-next for niedbalski mp256675
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [.venv] Error 127
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/10965101/
Build: http://10.245.162.77:8080/job/charm_lint_check/4238/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #3980 rabbitmq-server-next for niedbalski mp256675
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/3980/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #4254 rabbitmq-server-next for niedbalski mp256675
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/4254/

[Felipe Reyes (freyes)]

I did a round of tests exercising th default value (ignore), autoheal and pause_minority, so force a split I used lxc-freeze/lxc-unfreeze and I could recover the cluster without issues.

Thanks Jorge.

review: Approve

[Billy Olsen (billy-olsen)]

Jorge,

The amulet test continues to fail due to the SSL setup per - http://paste.ubuntu.com/11700673/. I believe this is due to a change that was possibly recommended by Ed (?). For RabbitMQ charm, SSL listens on one port where non-SSL listens on another port. You can disable ssl (only insecure), enable ssl (both insecure and secure), and ssl only (only secure socket is opened). The amulet test checks for these conditions and is failing because the insecure ssl socket is still opened.

If the behavior is going to be changed, we need to properly document the behavior change thoroughly and deprecate it appropriately. I think there's no need to revert this specific behavior though, I think the three modes of operation are perfectly viable, though I think in production ssl_only mode should be the only one used.

Please correct and resubmit.

Thanks,

Billy

review: Needs Fixing

[Edward Hope-Morley (hopem)]

Nice work for implementing this Jorge, i think we can definitely attribute some issues we are seeing in the field to this. I have one minor request for you to add some docstrings to a new class you created though.

[Ryan Beisner (1chb1n)]

Take note that the rabbitmq-server/next amulet tests are being refactored to be in line with other os-charms, to exercise more than just Trusty, and to fix misc issues in logic which are causing false passes.

Those new and refactored tests have are still in review, but have exposed two critical bugs. I would recommend holding on this proposal until the following bugs are resolved and the corresponding branches have landed in next. Then, I'd rebase your proposal and let the updated tests re-run to make sure everything still passes.

bug tldrs:

cluster race bug, affects stable and next
>50% fail to cluster with leadership election
>90% fail to cluster without leadership election
https://bugs.launchpad.net/charms/+source/rabbitmq-server/+bug/1486177

outright fail on vivid, affects stable and next
100% fail, rmq's pidfile location changed, cron job errors out
https://bugs.launchpad.net/charms/+source/rabbitmq-server/+bug/1485722

review: Needs Resubmitting

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #9751 rabbitmq-server-next for niedbalski mp272100
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/12530904/
Build: http://10.245.162.77:8080/job/charm_unit_test/9751/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #10563 rabbitmq-server-next for niedbalski mp272100
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/10563/

[Edward Hope-Morley (hopem)]

A few nits. The refactor of the ssl config rendering is a little heavy handed since it is not directly related to the addition of a new config option but you seem to have added some decent tests. Lets see what OSCI amulet has to say.

And fwiw I can't reproduce the unit test failure locally so not sure what happened there.

[lp:~niedbalski/charms/trusty/rabbitmq-server/fix-1442443]

93. By Jorge Niedbalski on 2015-09-23

addressing @dosaboy comments

94. By Jorge Niedbalski on 2015-09-23

addressing @dosaboy comments

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #9763 rabbitmq-server-next for niedbalski mp272100
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/12532213/
Build: http://10.245.162.77:8080/job/charm_unit_test/9763/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #10575 rabbitmq-server-next for niedbalski mp272100
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/10575/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #6702 rabbitmq-server-next for niedbalski mp272100
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12535959/
Build: http://10.245.162.77:8080/job/charm_amulet_test/6702/

[Edward Hope-Morley (hopem)]

Unit and amulet errors. See inline for unit test fixups. Try running your new amulet test in isolation to see if you can get it to run.

review: Needs Fixing

[lp:~niedbalski/charms/trusty/rabbitmq-server/fix-1442443]

95. By Jorge Niedbalski on 2015-09-24

addressing @dosaboy comments

[Jorge Niedbalski (niedbalski)]

@hopem: Addressed your comments, still looking to one of the failed amulet tests.

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #9886 rabbitmq-server-next for niedbalski mp272100
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/9886/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #10704 rabbitmq-server-next for niedbalski mp272100
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/10704/

[lp:~niedbalski/charms/trusty/rabbitmq-server/fix-1442443]

96. By Jorge Niedbalski on 2015-09-24

addressing comments

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #6755 rabbitmq-server-next for niedbalski mp272100
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12547772/
Build: http://10.245.162.77:8080/job/charm_amulet_test/6755/

[lp:~niedbalski/charms/trusty/rabbitmq-server/fix-1442443]

97. By Jorge Niedbalski on 2015-09-28

Addressed @beisner comments

98. By Jorge Niedbalski on 2015-09-28

Addressed @beisner comments

Unmerged revisions

=== modified file 'config.yaml'
--- config.yaml	2015-04-20 11:13:39 +0000
+++ config.yaml	2015-09-24 22:21:57 +0000
@@ -8,7 +8,7 @@
     type: string
     default: "off"
     description: |
-      Enable SSL connections on rabbitmq, valid values are 'off', 'on', 'only'. If ssl_key, 
+      Enable SSL connections on rabbitmq, valid values are 'off', 'on', 'only'. If ssl_key,
       ssl_cert, ssl_ca are provided then then those values will be used. Otherwise
       the service will act as its own certificate authority and pass its ca cert to clients.
       For HA or clustered rabbits ssl key/cert must be provided.
@@ -92,6 +92,28 @@
     description: |
       When set to true the 'ha-mode: all' policy is applied to all the exchages
       that match the expression '^(?!amq\.).*'
+  cluster-partition-handling:
+    type: string
+    default: ignore
+    description: |
+      RabbitMQ offers three ways to deal with network partitions automatically.
+      Available modes:
+
+      ignore - Your network is reliable. All your nodes are in a rack,
+      connected with a switch, and that switch is also the route to the outside world.
+      You don't want to run any risk of any of your cluster shutting down if any other part of it fails
+      (or you have a two node cluster).
+
+      pause_minority - Your network is maybe less reliable. You have clustered across 3 AZs in EC2,
+      and you assume that only one AZ will fail at once.
+      In that scenario you want the remaining two AZs to continue working and the nodes from the failed AZ
+      to rejoin automatically and without fuss when the AZ comes back.
+
+      autoheal - Your network may not be reliable.
+      You are more concerned with continuity of service than with data integrity.
+      You may have a two node cluster.
+
+      For more information see http://www.rabbitmq.com/partitions.html.
   rbd-size:
     type: string
     default: 5G

=== modified file 'hooks/rabbit_utils.py'
--- hooks/rabbit_utils.py	2015-09-16 17:56:59 +0000
+++ hooks/rabbit_utils.py	2015-09-24 22:21:57 +0000
@@ -1,12 +1,17 @@
 import os
-import pwd
-import grp
 import re
 import sys
 import subprocess
 import glob
 import tempfile
 
+from rabbitmq_context import (
+    RabbitMQSSLContext,
+    RabbitMQClusterContext,
+)
+
+from charmhelpers.core.templating import render
+
 from charmhelpers.contrib.openstack.utils import (
     get_hostname,
 )
@@ -29,8 +34,6 @@
     cmp_pkgrevno
 )
 
-from charmhelpers.core.templating import render
-
 from charmhelpers.contrib.peerstorage import (
     peer_store,
     peer_retrieve
@@ -58,7 +61,10 @@
 # the charm doesn't concern itself with template specifics etc.
 CONFIG_FILES = OrderedDict([
     (RABBITMQ_CONF, {
-        'hook_contexts': None,
+        'hook_contexts': [
+            RabbitMQSSLContext(),
+            RabbitMQClusterContext(),
+        ],
         'services': ['rabbitmq-server']
     }),
     (ENV_CONF, {
@@ -72,6 +78,42 @@
 ])
 
 
+class ConfigRenderer(object):
+    """
+    This class is a generic configuration renderer for
+    a given dict mapping configuration files and hook_contexts.
+    """
+    def __init__(self, config):
+        """
+        :param config: see CONFIG_FILES
+        :type config: dict
+        """
+        self.config_data = {}
+
+        for config_path, data in config.items():
+            hook_contexts = data.get('hook_contexts', None)
+            if hook_contexts:
+                ctxt = {}
+                for svc_context in hook_contexts:
+                    ctxt.update(svc_context())
+                self.config_data[config_path] = ctxt
+
+    def write(self, config_path):
+        data = self.config_data.get(config_path, None)
+        if data:
+            log("writing config file: %s , data: %s" % (config_path,
+                                                        str(data)),
+                level='DEBUG')
+
+            render(os.path.basename(config_path), config_path,
+                   data, perms=0o644)
+
+    def write_all(self):
+        """Write all the defined configuration files"""
+        for service in self.config_data.keys():
+            self.write(service)
+
+
 class RabbitmqError(Exception):
     pass
 
@@ -391,40 +433,6 @@
 def disable_plugin(plugin):
     _manage_plugin(plugin, 'disable')
 
-ssl_key_file = "/etc/rabbitmq/rabbit-server-privkey.pem"
-ssl_cert_file = "/etc/rabbitmq/rabbit-server-cert.pem"
-ssl_ca_file = "/etc/rabbitmq/rabbit-server-ca.pem"
-
-
-def enable_ssl(ssl_key, ssl_cert, ssl_port,
-               ssl_ca=None, ssl_only=False, ssl_client=None):
-    uid = pwd.getpwnam("root").pw_uid
-    gid = grp.getgrnam("rabbitmq").gr_gid
-
-    for contents, path in (
-            (ssl_key, ssl_key_file),
-            (ssl_cert, ssl_cert_file),
-            (ssl_ca, ssl_ca_file)):
-        if not contents:
-            continue
-        with open(path, 'w') as fh:
-            fh.write(contents)
-        os.chmod(path, 0o640)
-        os.chown(path, uid, gid)
-
-    data = {
-        "ssl_port": ssl_port,
-        "ssl_cert_file": ssl_cert_file,
-        "ssl_key_file": ssl_key_file,
-        "ssl_client": ssl_client,
-        "ssl_ca_file": "",
-        "ssl_only": ssl_only}
-
-    if ssl_ca:
-        data["ssl_ca_file"] = ssl_ca_file
-
-    render(os.path.basename(RABBITMQ_CONF), RABBITMQ_CONF, data, perms=0o644)
-
 
 def execute(cmd, die=False, echo=False):
     """ Executes a command

=== added file 'hooks/rabbitmq_context.py'
--- hooks/rabbitmq_context.py	1970-01-01 00:00:00 +0000
+++ hooks/rabbitmq_context.py	2015-09-24 22:21:57 +0000
@@ -0,0 +1,122 @@
+from charmhelpers.contrib.ssl.service import ServiceCA
+
+from charmhelpers.core.hookenv import (
+    open_port,
+    close_port,
+    config,
+    log,
+    ERROR,
+)
+
+import sys
+import pwd
+import grp
+import os
+import base64
+
+import ssl_utils
+
+ssl_key_file = "/etc/rabbitmq/rabbit-server-privkey.pem"
+ssl_cert_file = "/etc/rabbitmq/rabbit-server-cert.pem"
+ssl_ca_file = "/etc/rabbitmq/rabbit-server-ca.pem"
+
+
+def convert_from_base64(v):
+    # Rabbit originally supported pem encoded key/cert in config, play
+    # nice on upgrades as we now expect base64 encoded key/cert/ca.
+    if not v:
+        return v
+    if v.startswith('-----BEGIN'):
+        return v
+    try:
+        return base64.b64decode(v)
+    except TypeError:
+        return v
+
+
+class RabbitMQSSLContext(object):
+
+    def enable_ssl(self, ssl_key, ssl_cert, ssl_port,
+                   ssl_ca=None, ssl_only=False, ssl_client=None):
+
+        uid = pwd.getpwnam("root").pw_uid
+        gid = grp.getgrnam("rabbitmq").gr_gid
+
+        for contents, path in (
+                (ssl_key, ssl_key_file),
+                (ssl_cert, ssl_cert_file),
+                (ssl_ca, ssl_ca_file)):
+
+            if not contents:
+                continue
+
+            with open(path, 'w') as fh:
+                fh.write(contents)
+
+            os.chmod(path, 0o640)
+            os.chown(path, uid, gid)
+
+        data = {
+            "ssl_port": ssl_port,
+            "ssl_cert_file": ssl_cert_file,
+            "ssl_key_file": ssl_key_file,
+            "ssl_client": ssl_client,
+            "ssl_ca_file": "",
+            "ssl_only": ssl_only
+        }
+
+        if ssl_ca:
+            data["ssl_ca_file"] = ssl_ca_file
+
+        return data
+
+    def __call__(self):
+        """
+        The legacy config support adds some additional complications.
+
+        ssl_enabled = True, ssl = off -> ssl enabled
+        ssl_enabled = False, ssl = on -> ssl enabled
+        """
+        ssl_mode, external_ca = ssl_utils.get_ssl_mode()
+
+        ctxt = {
+            'ssl_mode': ssl_mode,
+        }
+
+        if ssl_mode == 'off':
+            close_port(config('ssl_port'))
+            ssl_utils.reconfigure_client_ssl()
+            return ctxt
+
+        ssl_key = convert_from_base64(config('ssl_key'))
+        ssl_cert = convert_from_base64(config('ssl_cert'))
+        ssl_ca = convert_from_base64(config('ssl_ca'))
+        ssl_port = config('ssl_port')
+
+        # If external managed certs then we need all the fields.
+        if (ssl_mode in ('on', 'only') and any((ssl_key, ssl_cert)) and
+                not all((ssl_key, ssl_cert))):
+            log('If ssl_key or ssl_cert are specified both are required.',
+                level=ERROR)
+            sys.exit(1)
+
+        if not external_ca:
+            ssl_cert, ssl_key, ssl_ca = ServiceCA.get_service_cert()
+
+        ctxt.update(self.enable_ssl(
+            ssl_key, ssl_cert, ssl_port, ssl_ca,
+            ssl_only=(ssl_mode == "only"), ssl_client=False
+        ))
+
+        ssl_utils.reconfigure_client_ssl(True)
+        open_port(ssl_port)
+
+        return ctxt
+
+
+class RabbitMQClusterContext(object):
+
+    def __call__(self):
+        return {
+            'cluster_partition_handling': config('cluster-partition-handling'),
+        }

=== modified file 'hooks/rabbitmq_server_relations.py'
--- hooks/rabbitmq_server_relations.py	2015-09-22 13:55:23 +0000
+++ hooks/rabbitmq_server_relations.py	2015-09-24 22:21:57 +0000
@@ -1,5 +1,5 @@
 #!/usr/bin/python
-import base64
+
 import os
 import shutil
 import sys
@@ -17,6 +17,8 @@
     import yaml  # flake8: noqa
 
 import rabbit_utils as rabbit
+import ssl_utils
+
 from lib.utils import (
     chown, chmod,
     is_newer,
@@ -73,7 +75,6 @@
     write_file,
 )
 from charmhelpers.contrib.charmsupport import nrpe
-from charmhelpers.contrib.ssl.service import ServiceCA
 
 from charmhelpers.contrib.peerstorage import (
     peer_echo,
@@ -232,7 +233,7 @@
             get_address_in_network(config('access-network'),
                                    unit_get('private-address'))
 
-    configure_client_ssl(relation_settings)
+    ssl_utils.configure_client_ssl(relation_settings)
 
     if is_clustered():
         relation_settings['clustered'] = 'true'
@@ -627,103 +628,10 @@
 MAN_PLUGIN = 'rabbitmq_management'
 
 
-def configure_client_ssl(relation_data):
-    """Configure client with ssl
-    """
-    ssl_mode, external_ca = _get_ssl_mode()
-    if ssl_mode == 'off':
-        return
-    relation_data['ssl_port'] = config('ssl_port')
-    if external_ca:
-        if config('ssl_ca'):
-            relation_data['ssl_ca'] = base64.b64encode(
-                config('ssl_ca'))
-        return
-    ca = ServiceCA.get_ca()
-    relation_data['ssl_ca'] = base64.b64encode(ca.get_ca_bundle())
-
-
-def _get_ssl_mode():
-    ssl_mode = config('ssl')
-    external_ca = False
-    # Legacy config boolean option
-    ssl_on = config('ssl_enabled')
-    if ssl_mode == 'off' and ssl_on is False:
-        ssl_mode = 'off'
-    elif ssl_mode == 'off' and ssl_on:
-        ssl_mode = 'on'
-    ssl_key = config('ssl_key')
-    ssl_cert = config('ssl_cert')
-    if all((ssl_key, ssl_cert)):
-        external_ca = True
-    return ssl_mode, external_ca
-
-
-def _convert_from_base64(v):
-    # Rabbit originally supported pem encoded key/cert in config, play
-    # nice on upgrades as we now expect base64 encoded key/cert/ca.
-    if not v:
-        return v
-    if v.startswith('-----BEGIN'):
-        return v
-    try:
-        return base64.b64decode(v)
-    except TypeError:
-        return v
-
-
-def reconfigure_client_ssl(ssl_enabled=False):
-    ssl_config_keys = set(('ssl_key', 'ssl_cert', 'ssl_ca'))
-    for rid in relation_ids('amqp'):
-        rdata = relation_get(rid=rid, unit=os.environ['JUJU_UNIT_NAME'])
-        if not ssl_enabled and ssl_config_keys.intersection(rdata):
-            # No clean way to remove entirely, but blank them.
-            relation_set(relation_id=rid, ssl_key='', ssl_cert='', ssl_ca='')
-        elif ssl_enabled and not ssl_config_keys.intersection(rdata):
-            configure_client_ssl(rdata)
-            relation_set(relation_id=rid, **rdata)
-
-
-def configure_rabbit_ssl():
-    """
-    The legacy config support adds some additional complications.
-
-    ssl_enabled = True, ssl = off -> ssl enabled
-    ssl_enabled = False, ssl = on -> ssl enabled
-    """
-    ssl_mode, external_ca = _get_ssl_mode()
-
-    if ssl_mode == 'off':
-        if os.path.exists(rabbit.RABBITMQ_CONF):
-            os.remove(rabbit.RABBITMQ_CONF)
-        close_port(config('ssl_port'))
-        reconfigure_client_ssl()
-        return
-    ssl_key = _convert_from_base64(config('ssl_key'))
-    ssl_cert = _convert_from_base64(config('ssl_cert'))
-    ssl_ca = _convert_from_base64(config('ssl_ca'))
-    ssl_port = config('ssl_port')
-
-    # If external managed certs then we need all the fields.
-    if (ssl_mode in ('on', 'only') and any((ssl_key, ssl_cert)) and
-            not all((ssl_key, ssl_cert))):
-        log('If ssl_key or ssl_cert are specified both are required.',
-            level=ERROR)
-        sys.exit(1)
-
-    if not external_ca:
-        ssl_cert, ssl_key, ssl_ca = ServiceCA.get_service_cert()
-
-    rabbit.enable_ssl(
-        ssl_key, ssl_cert, ssl_port, ssl_ca,
-        ssl_only=(ssl_mode == "only"), ssl_client=False)
-    reconfigure_client_ssl(True)
-    open_port(ssl_port)
-
-
 @hooks.hook('config-changed')
 @restart_on_change(rabbit.restart_map())
 def config_changed():
+
     if config('prefer-ipv6'):
         rabbit.assert_charm_supports_ipv6()
 
@@ -752,9 +660,9 @@
         rabbit.disable_plugin(MAN_PLUGIN)
         close_port(55672)
 
-    configure_rabbit_ssl()
-
     rabbit.set_all_mirroring_queues(config('mirroring-queues'))
+    rabbit.ConfigRenderer(
+        rabbit.CONFIG_FILES).write_all()
 
     if is_relation_made("ha"):
         ha_is_active_active = config("ha-vip-only")

=== added file 'hooks/ssl_utils.py'
--- hooks/ssl_utils.py	1970-01-01 00:00:00 +0000
+++ hooks/ssl_utils.py	2015-09-24 22:21:57 +0000
@@ -0,0 +1,58 @@
+from charmhelpers.contrib.ssl.service import ServiceCA
+
+from charmhelpers.core.hookenv import (
+    config,
+    relation_ids,
+    relation_set,
+    relation_get,
+    local_unit,
+)
+
+import base64
+
+
+def get_ssl_mode():
+    ssl_mode = config('ssl')
+    external_ca = False
+
+    # Legacy config boolean option
+    ssl_on = config('ssl_enabled')
+    if ssl_mode == 'off' and ssl_on is False:
+        ssl_mode = 'off'
+    elif ssl_mode == 'off' and ssl_on:
+        ssl_mode = 'on'
+
+    ssl_key = config('ssl_key')
+    ssl_cert = config('ssl_cert')
+
+    if all((ssl_key, ssl_cert)):
+        external_ca = True
+    return ssl_mode, external_ca
+
+
+def configure_client_ssl(relation_data):
+    """Configure client with ssl
+    """
+    ssl_mode, external_ca = get_ssl_mode()
+    if ssl_mode == 'off':
+        return
+    relation_data['ssl_port'] = config('ssl_port')
+    if external_ca:
+        if config('ssl_ca'):
+            relation_data['ssl_ca'] = base64.b64encode(
+                config('ssl_ca'))
+        return
+    ca = ServiceCA.get_ca()
+    relation_data['ssl_ca'] = base64.b64encode(ca.get_ca_bundle())
+
+
+def reconfigure_client_ssl(ssl_enabled=False):
+    ssl_config_keys = set(('ssl_key', 'ssl_cert', 'ssl_ca'))
+    for rid in relation_ids('amqp'):
+        rdata = relation_get(rid=rid, unit=local_unit())
+        if not ssl_enabled and ssl_config_keys.intersection(rdata):
+            # No clean way to remove entirely, but blank them.
+            relation_set(relation_id=rid, ssl_key='', ssl_cert='', ssl_ca='')
+        elif ssl_enabled and not ssl_config_keys.intersection(rdata):
+            configure_client_ssl(rdata)
+            relation_set(relation_id=rid, **rdata)

=== modified file 'templates/rabbitmq.config'
--- templates/rabbitmq.config	2014-05-23 08:13:05 +0000
+++ templates/rabbitmq.config	2015-09-24 22:21:57 +0000
@@ -1,21 +1,35 @@
 [
-    {rabbit, [    
-{% if ssl_only %} 
+    {rabbit, [
+{% if ssl_only %}
         {tcp_listeners, []},
 {% else %}
     	{tcp_listeners, [5672]},
 {% endif %}
-    	{ssl_listeners, [{{ ssl_port }}]},
+{% if ssl_port %}
+     	{ssl_listeners, [{{ ssl_port }}]},
+{% endif %}
+{% if ssl_mode == "on" %}
     	{ssl_options, [
  	        {verify, verify_peer},
-{% if ssl_client %} 
-                {fail_if_no_peer_cert, true}, 
+{% if ssl_client %}
+                {fail_if_no_peer_cert, true},
 {% else %}
                 {fail_if_no_peer_cert, false},
-{% endif %}{% if ssl_ca_file %} 
-                {cacertfile, "{{ ssl_ca_file }}"}, {% endif %}
-        	{certfile, "{{ ssl_cert_file }}"},
-        	{keyfile, "{{ ssl_key_file }}"}
-    	]}
+{% endif %}
+           {% if ssl_ca_file %}
+                {cacertfile, "{{ ssl_ca_file }}"},
+           {% endif %}
+           {% if ssl_cert_file %}
+                {certfile, "{{ ssl_cert_file }}"},
+           {% endif %}
+           {% if ssl_key_file %}
+                {keyfile, "{{ ssl_key_file }}"}
+           {% endif %}
+    	]},
+{% endif %}
+
+        {% if cluster_partition_handling %}
+        {cluster_partition_handling, {{ cluster_partition_handling }}}
+        {% endif %}
     ]}
-].    
\ No newline at end of file
+].

=== added file 'tests/50_test_cluster_partition.py'
--- tests/50_test_cluster_partition.py	1970-01-01 00:00:00 +0000
+++ tests/50_test_cluster_partition.py	2015-09-24 22:21:57 +0000
@@ -0,0 +1,31 @@
+#!/usr/bin/python
+#
+# This Amulet test deploys rabbitmq-server
+#
+# Note: We use python2, because pika doesn't support python3
+import amulet
+
+# The number of seconds to wait for the environment to setup.
+seconds = 1200
+d = amulet.Deployment(series="trusty")
+
+d.add('rabbitmq-server', units=1)
+# Create a configuration.
+configuration = {'cluster-partition-handling': "autoheal"}
+d.configure('rabbitmq-server', configuration)
+
+d.expose('rabbitmq-server')
+try:
+    d.setup(timeout=seconds)
+    d.sentry.wait(seconds)
+except amulet.helpers.TimeoutError:
+    message = 'The environment did not setup in %d seconds.' % seconds
+    amulet.raise_status(amulet.SKIP, msg=message)
+except:
+    raise
+
+rabbit_unit = d.sentry.unit['rabbitmq-server/0']
+output, code = rabbit_unit.run("grep autoheal /etc/rabbitmq/rabbitmq.config")
+
+if code != 0 or output == "":
+    amulet.raise_status(amulet.FAIL, msg="didn't find autoheal")

=== modified file 'unit_tests/test_rabbit_utils.py'
--- unit_tests/test_rabbit_utils.py	2015-04-13 15:42:57 +0000
+++ unit_tests/test_rabbit_utils.py	2015-09-24 22:21:57 +0000
@@ -3,11 +3,44 @@
 import unittest
 import tempfile
 import sys
+import collections
 
 import rabbit_utils
 sys.modules['MySQLdb'] = mock.Mock()
 
 
+class ConfigRendererTests(unittest.TestCase):
+
+    class FakeContext(object):
+        def __call__(self, *a, **k):
+            return {'foo': 'bar'}
+
+    config_map = collections.OrderedDict(
+        [('/this/is/a/config', {
+            'hook_contexts': [
+                FakeContext()
+            ]
+        })]
+    )
+
+    def setUp(self):
+        super(ConfigRendererTests, self).setUp()
+        self.renderer = rabbit_utils.ConfigRenderer(
+            self.config_map)
+
+    def test_has_config_data(self):
+        self.assertTrue(
+            '/this/is/a/config' in self.renderer.config_data.keys())
+
+    @mock.patch("rabbit_utils.log")
+    @mock.patch("rabbit_utils.render")
+    def test_write_all(self, log, render):
+        self.renderer.write_all()
+
+        self.assertTrue(render.called)
+        self.assertTrue(log.called)
+
+
 class UtilsTests(unittest.TestCase):
     def setUp(self):
         super(UtilsTests, self).setUp()

=== added file 'unit_tests/test_rabbitmq_context.py'
--- unit_tests/test_rabbitmq_context.py	1970-01-01 00:00:00 +0000
+++ unit_tests/test_rabbitmq_context.py	2015-09-24 22:21:57 +0000
@@ -0,0 +1,79 @@
+import rabbitmq_context
+
+import mock
+import unittest
+
+
+class TestRabbitMQSSLContext(unittest.TestCase):
+
+    @mock.patch("rabbitmq_context.config")
+    @mock.patch("rabbitmq_context.close_port")
+    @mock.patch("rabbitmq_context.ssl_utils.reconfigure_client_ssl")
+    @mock.patch("rabbitmq_context.ssl_utils.get_ssl_mode")
+    def test_context_ssl_off(self, get_ssl_mode, reconfig_ssl, close_port,
+                             config):
+        get_ssl_mode.return_value = ("off", "off")
+        self.assertEqual(rabbitmq_context.RabbitMQSSLContext().__call__(), {
+            "ssl_mode": "off"
+        })
+
+        self.assertTrue(close_port.called)
+        self.assertTrue(reconfig_ssl.called)
+
+    @mock.patch("rabbitmq_context.open_port")
+    @mock.patch("rabbitmq_context.os.chmod")
+    @mock.patch("rabbitmq_context.os.chown")
+    @mock.patch("rabbitmq_context.pwd.getpwnam")
+    @mock.patch("rabbitmq_context.grp.getgrnam")
+    @mock.patch("rabbitmq_context.config")
+    @mock.patch("rabbitmq_context.close_port")
+    @mock.patch("rabbitmq_context.ssl_utils.reconfigure_client_ssl")
+    @mock.patch("rabbitmq_context.ssl_utils.get_ssl_mode")
+    def test_context_ssl_on(self, get_ssl_mode, reconfig_ssl, close_port,
+                            config, gr, pw, chown, chmod, open_port):
+
+        get_ssl_mode.return_value = ("on", "on")
+
+        def config_get(n):
+            return None
+
+        config.side_effect = config_get
+
+        def pw(name):
+            class Uid(object):
+                pw_uid = 1
+                gr_gid = 100
+            return Uid()
+
+        pw.side_effect = pw
+        gr.side_effect = pw
+
+        m = mock.mock_open()
+        with mock.patch('rabbitmq_context.open', m, create=True):
+            self.assertEqual(
+                rabbitmq_context.RabbitMQSSLContext().__call__(), {
+                    "ssl_port": None,
+                    "ssl_cert_file": "/etc/rabbitmq/rabbit-server-cert.pem",
+                    "ssl_key_file": '/etc/rabbitmq/rabbit-server-privkey.pem',
+                    "ssl_client": False,
+                    "ssl_ca_file": "",
+                    "ssl_only": False,
+                    "ssl_mode": "on",
+                })
+
+        self.assertTrue(reconfig_ssl.called)
+        self.assertTrue(open_port.called)
+
+
+class TestRabbitMQClusterContext(unittest.TestCase):
+
+    @mock.patch("rabbitmq_context.config")
+    def test_context_ssl_off(self, config):
+        config.return_value = "ignore"
+
+        self.assertEqual(
+            rabbitmq_context.RabbitMQClusterContext().__call__(), {
+                'cluster_partition_handling': "ignore"
+            })
+
+        config.assert_called_once_with("cluster-partition-handling")

=== modified file 'unit_tests/test_rabbitmq_server_relations.py'
--- unit_tests/test_rabbitmq_server_relations.py	2015-04-20 11:13:39 +0000
+++ unit_tests/test_rabbitmq_server_relations.py	2015-09-24 22:21:57 +0000
@@ -37,7 +37,7 @@
     @patch('rabbitmq_server_relations.relation_set')
     @patch('apt_pkg.Cache')
     @patch('rabbitmq_server_relations.is_clustered')
-    @patch('rabbitmq_server_relations.configure_client_ssl')
+    @patch('rabbitmq_server_relations.ssl_utils.configure_client_ssl')
     @patch('rabbitmq_server_relations.unit_get')
     @patch('rabbitmq_server_relations.relation_get')
     @patch('rabbitmq_server_relations.is_elected_leader')
@@ -87,7 +87,7 @@
     @patch('rabbitmq_server_relations.relation_set')
     @patch('apt_pkg.Cache')
     @patch('rabbitmq_server_relations.is_clustered')
-    @patch('rabbitmq_server_relations.configure_client_ssl')
+    @patch('rabbitmq_server_relations.ssl_utils.configure_client_ssl')
     @patch('rabbitmq_server_relations.unit_get')
     @patch('rabbitmq_server_relations.relation_get')
     @patch('rabbitmq_server_relations.is_elected_leader')