Comment 1 for bug 1807282

Based on the output produced during a clean redeploy, I can see that the charm logic is incorrect in handling config changes.

Only the first config-changed event with all necessary data present is handled
https://github.com/openstack-charmers/charm-interface-ceph-client/blob/0e191799fc9ee2d7662cd77f030790fa850c6dea/requires.py#L40-L48

at this time other units do not have ceph-public-address exposed via relation data yet, therefore, the following code simply ignores the lack of those:

https://github.com/openstack-charmers/charm-interface-ceph-client/blob/0e191799fc9ee2d7662cd77f030790fa850c6dea/requires.py#L108-L109

Then configure_exporter is executed and exporter.started is set which triggers ceph.conf rendering with only one monitor address present.

@when('ceph.available')
@when_not('exporter.started')
def configure_exporter(ceph_client):

This flag is never cleared and the reactive code of the exporter does not have any intelligent relation data change processing with regards to that.

This is confirmed by the timestamps:

# the first and the only rendering event happened at 02:46:45
root@juju-d85cbd-12-lxd-2:/var/lib/juju/agents/unit-prometheus-ceph-exporter-0/charm# grep ceph.conf /var/log/juju/unit-prometheus-ceph-exporter-0.log
2018-12-07 02:46:45 DEBUG juju-log ceph:295: Writing file /var/snap/prometheus-ceph-exporter/current/ceph.conf root:root 444

# joined events - nothing rendered after that as expected
root@juju-d85cbd-12-lxd-2:/var/lib/juju/agents/unit-prometheus-ceph-exporter-0/charm# grep ceph-relation-joined /var/log/juju/unit-prometheus-ceph-exporter-0.log
2018-12-07 02:28:11 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-joined
2018-12-07 02:28:12 DEBUG ceph-relation-joined lxc
2018-12-07 02:28:13 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-joined
2018-12-07 02:28:13 DEBUG ceph-relation-joined lxc
2018-12-07 02:29:29 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-joined
2018-12-07 02:29:29 DEBUG ceph-relation-joined lxc

root@juju-d85cbd-12-lxd-2:/var/lib/juju/agents/unit-prometheus-ceph-exporter-0/charm# grep ceph-relation-changed /var/log/juju/unit-prometheus-ceph-exporter-0.log
2018-12-07 02:28:12 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-changed
2018-12-07 02:28:12 DEBUG ceph-relation-changed lxc
2018-12-07 02:28:13 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-changed
2018-12-07 02:28:13 DEBUG ceph-relation-changed lxc
2018-12-07 02:30:10 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-changed
2018-12-07 02:30:10 DEBUG ceph-relation-changed lxc
2018-12-07 02:46:45 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-changed
2018-12-07 02:46:45 DEBUG ceph-relation-changed lxc
2018-12-07 02:46:55 DEBUG ceph-relation-changed active

^^^ this corresponds to the rendering event and no rendering attempts happen for other relation-changed events which results in the incorrect file content
2018-12-07 02:47:26 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-changed
2018-12-07 02:47:26 DEBUG ceph-relation-changed lxc
2018-12-07 02:47:58 INFO juju-log ceph:295: Reactive main running for hook ceph-relation-changed
2018-12-07 02:47:59 DEBUG ceph-relation-changed lxc