Merge lp:~michael.nelson/charms/trusty/elasticsearch/heap-config into lp:~onlineservices-charmers/charms/trusty/elasticsearch/elasticsearch2

[Michael Nelson]

Add heap_size config option, defaulting to min(0.5*available_mem, 31g)

I've used http://stackoverflow.com/a/18152957/523729 as a basis. Testing it I can confirm locally that the heap size is updated in ES, using:

curl -s http://localhost:9200/_nodes/stats | python -m json.tool | grep heap_max_in_bytes

What I can't confirm is the memory lock - and that's either simply because I'm testing in an lxc container :P, or additionally because /etc/security/limits.d/ aren't applied to non-interactive users (ie. system jobs - I see this reported too).

I'll test on staging before landing. EDIT: Just tested on staging and confirm that setting the heap-size works as expected:

https://pastebin.canonical.com/144362/

Also, top shows a similar resident memory value too.

[Bret Barker (noise)]

minor comment - with the conditional below, you can't set a heap of e.g. 1.4g and can't specify it as 1400m either. I know the recommended heap is 1/2 total ram, but for a 2g system it fails pretty miserably with any kind of load unless you do 1300-1500m heap.

[Bret Barker (noise)]

+1

review: Approve

[Wes Mason (wesmason)]

lgtm

review: Approve

=== modified file 'config.yaml'
--- config.yaml	2015-10-01 19:00:31 +0000
+++ config.yaml	2015-11-18 20:40:41 +0000
@@ -27,3 +27,10 @@
       By default, the admin and peer ports (9200 and 9300) are only accessible
       to clients and peers respectively. Switch this to false to enable access
       from any machine.
+  heap_size:
+    type: string
+    default: ""
+    description: |
+      By default, the heap size will be set to the maximum of half of the
+      available memory and 30.5G. A number in gigabytes for the Elasticsearch
+      heap size.

=== modified file 'playbook.yaml'
--- playbook.yaml	2015-10-27 22:23:15 +0000
+++ playbook.yaml	2015-11-18 20:40:41 +0000
@@ -37,6 +37,22 @@
       notify:
         - Restart ElasticSearch
 
+    - name: Update defaults
+      tags:
+        - config-changed
+      template: src={{ charm_dir }}/templates/etc_default.j2
+                dest=/etc/default/elasticsearch
+      notify:
+        - Restart ElasticSearch
+
+    - name: Set Elasticsearch user limits
+      tags:
+        - upgrade-charm
+      template: src={{ charm_dir }}/templates/limits.conf
+                dest=/etc/security/limits.d/elasticsearch.conf
+      notify:
+        - Restart ElasticSearch
+
     - name: Open ES Port when exposed
       command: open-port 9200
       tags:

=== modified file 'templates/elasticsearch.yml'
--- templates/elasticsearch.yml	2015-10-30 03:32:29 +0000
+++ templates/elasticsearch.yml	2015-11-18 20:40:41 +0000
@@ -13,3 +13,6 @@
 {% for reln in relations.peer %}
   - {{ reln['private-address'] }}
 {% endfor %}
+
+# https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html
+bootstrap.mlockall: true

=== added file 'templates/etc_default.j2'
--- templates/etc_default.j2	1970-01-01 00:00:00 +0000
+++ templates/etc_default.j2	2015-11-18 20:40:41 +0000
@@ -0,0 +1,14 @@
+# See https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html
+{% set calculated_heap_size = ansible_memtotal_mb / 1024.0 / 2.0 %}
+{# min/max filters only available in ansible 1.8 #}
+{% if calculated_heap_size >= 31.5 %}{% set calculated_heap_size = 31 %}{% endif %}
+{% set heap_size = heap_size | default(calculated_heap_size, true) | float %}
+# Note: ES doesn't start if the heap size is not an int of either gigabytes (g)
+# or megabytes (m)
+{% if heap_size > 4 %}
+ES_HEAP_SIZE={{ heap_size | int }}g
+{% else %}
+ES_HEAP_SIZE={{ (heap_size * 1024) | int }}m
+{% endif %}
+MAX_OPEN_FILES=65535
+MAX_LOCKED_MEMORY=unlimited

=== added file 'templates/limits.conf'
--- templates/limits.conf	1970-01-01 00:00:00 +0000
+++ templates/limits.conf	2015-11-18 20:40:41 +0000
@@ -0,0 +1,3 @@
+# Ensure ES user can run with max open files and unlimited memlock.
+elasticsearch - nofile 65535
+elasticsearch - memlock unlimited