Merge ~hloeung/charm-graylog:master into ~graylog-charmers/charm-graylog:master

[Haw Loeung]

Add automatically determine no. of index shards

Update index_shards config option to allow the charm to automatically
work out how many index shards to use based on how many elasticsearch
units are available.

[Stuart Bishop (stub)]

Fine.

I worry about what happens if the shard count changes, and if users need to be warned. Or maybe graylog and/or elasticsearch doesn't suck and things get rebalanced magically.

Style comments inline.

review: Approve

[Haw Loeung (hloeung)]

> I worry about what happens if the shard count changes, and if users need to be
> warned. Or maybe graylog and/or elasticsearch doesn't suck and things get
> rebalanced magically.

Nothing happens to the existing data when shards are changed, even if you reduce the number of shards. It only applies to new index sets (either automatically rotated out as configured or manually via the UI/API).

So let's say you have 5 indices already and the index set was previously set to 4 shards. You add more elasticsearch units and bump the index set to 6, the 5 old indices will remain as is sharded across 4 (instead of 6). Elasticsearch will most likely relocate the shards or replicas to the new elasticsearch units but that's it.

Same goes for if you're removing and reducing down to say 3 elasticsearch units. Old indicies will still be sharded across 4 with Elasticsearch automatically relocating shards/replicas:

0 1 2 (unit)
1 2 3 (shards)
4 (shards)

With automatic index shards set, it can also take a while to apply because there's no way that I know of to detect additional elasticsearch unit changes unless something is done to the elasticsearch relation to pass that back.

diff --git a/config.yaml b/config.yaml
index 9614533..3245461 100644
--- a/config.yaml
+++ b/config.yaml
@@ -16,7 +16,7 @@ options:
     type: int
     default: 2
     description: |
-      Number of Elasticsearch shards used per index in this index set.
+      Number of Elasticsearch shards used per index in this index set. Set this to '0' to let the charm automatically calculate based on how many Elasticsearch units.
   index_rotation_strategy:
     type: string
     default: "time"
diff --git a/reactive/graylog.py b/reactive/graylog.py
index c465552..13b3881 100644
--- a/reactive/graylog.py
+++ b/reactive/graylog.py
@@ -100,11 +100,24 @@ def configure_index_sets(*discard):
     index_sets = g.index_set_get()
     if index_sets is None:
         return
+    index_shards = conf['index_shards']
+    if index_shards is 0:
+        # Automatically work out the index shards based on how
+        # many Elasticsearch units.
+        count = 0
+        for relid in hookenv.relation_ids('elasticsearch'):
+            count += len(hookenv.related_units(relid))
+        if count:
+            index_shards = count
+        else:
+            hookenv.log('Can\'t work out number of elasticsearch units')
+            return
+
     for iset in index_sets:
         log = ''
-        if iset['shards'] != conf['index_shards']:
-            log += ' shards from {} to {}'.format(iset['shards'], conf['index_shards'])
-        iset['shards'] = conf['index_shards']
+        if iset['shards'] != index_shards:
+            log += ' shards from {} to {}'.format(iset['shards'], index_shards)
+        iset['shards'] = index_shards
         if iset['replicas'] != conf['index_replicas']:
             log += ' replicas from {} to {}'.format(iset['replicas'], conf['index_replicas'])
         iset['replicas'] = conf['index_replicas']