We have observed that upgrading an existing 3 unit rabbitmq-server cluster to the latest stable charms can cause the cluster to break. This is because the charm will check that the rabbitmq-server (at least on the leader) is configured by ensuring that RABBITMQ_NODENAME=rabbit@<hostname> is set in /etc/rabbitmq/rabbitmq-env.conf. Trouble is, when the hook runs to perform this action, it will use whatever is returned by charmhelpers.contrib.openstack.utils.get_hostname(). Now, in some of the environments we recently upgraded, the MAAS dns entry for the rabbitmq-server host IP has multiple ptr entries e.g.
;; ANSWER SECTION:
1.5.27.172.in-addr.arpa. 300 IN PTR ctrlnode1.maas.
1.5.27.172.in-addr.arpa. 300 IN PTR 172-27-5-1.maas.
(see http://paste.ubuntu.com/13653431/ for full answer)
The result is that a rabbitmq node whose nodename was originally configured with 172-27-5-1 now switches to ctrlnode1 which, when the charm attempts to restart rabbitmq, causes it to fail since it no longer recognises the nodename e.g. https://pastebin.canonical.com/145396/
For the record this was with MAAS 1.5.4.