Merge lp:~roadmr/charms/trusty/elasticsearch/es2-charm-retry-harder into lp:~onlineservices-charmers/charms/trusty/elasticsearch/elasticsearch2

[Daniel Manrique]

Retry a few times while checking if unit joined the cluster.

Instead of just pausing for 30 seconds, which in practice sometimes
doesn't give enough time for the new unit to join the cluster,
poke the cluster health every 15 seconds, a maximum of 6 times (total
wait time: 90 seconds) but continue if we see a successful join at any
retry. Should be more responsive to early joins while more resilient to slow
joins.

Retry a few times while checking if unit joined the cluster.

Instead of just pausing for 30 seconds, which in practice sometimes
doesn't give enough time for the new unit to join the cluster,
poke the cluster health every 15 seconds, a maximum of 6 times (total
wait time: 90 seconds) but continue if we see a successful join at any
retry. Should be more responsive to early joins while more resilient to slow
joins.

Unmerged revisions

46. By Daniel Manrique on 2017-03-09

Retry a few times while checking if unit joined the cluster.

Instead of just pausing for 30 seconds, which in practice sometimes
doesn't give enough time for the new unit to join the cluster,
poke the cluster health every 15 seconds, a maximum of 6 times (total
wait time: 90 seconds) but continue if we see a successful join at any
retry. Should be more responsive to early joins while more resilient to slow
joins.

=== modified file 'tasks/peer-relations.yml'
--- tasks/peer-relations.yml	2015-10-27 22:30:30 +0000
+++ tasks/peer-relations.yml	2017-03-09 20:57:09 +0000
@@ -43,7 +43,11 @@
 - name: Pause to ensure that after restart unit has time to join.
   tags:
     - peer-relation-changed
-  pause: seconds=30
+  uri: url=http://localhost:9200/_cluster/health return_content=yes
+  register: cluster_health_wait
+  retries: 6
+  delay: 15
+  until: cluster_health_wait.json.number_of_nodes > 1
   when: cluster_health.json.number_of_nodes == 1
 
 - name: Record cluster health after restart