Merge lp:~michael.nelson/charms/trusty/elasticsearch/merge-elasticsearch2-branch into lp:charms/trusty/elasticsearch

[Michael Nelson]

Add heap_size option defaulting to half of available mem (and memlock for better performance).
Add nagios_health_color options so development deploys of 1 unit can be yellow without nagios failures.
Ensure ES is restarted on boot.

Since we're updating the defaults in the stable ES charm, here are the other changes we'd since added and tested with ES2. They don't seem to be ES2 specific, but I've not verified with ES1.x.

[Chris Glass (tribaal)]

Hi, thanks a lot for sending in your code!

It generally looks good, but I have small concerns with the parameters (see inline comments).

[Charles Butler (lazypower)]

Awesome! I like it. I'm going to +1 on tribaal's comments about the nagios context. Thats my only real nit here. otherwise LGTM as is.

Unmerged revisions

43. By Michael Nelson on 2016-03-10

Merge onlineservices elasticsearch2 branch.

=== modified file 'config.yaml'
--- config.yaml	2016-03-08 16:38:17 +0000
+++ config.yaml	2016-03-10 22:57:41 +0000
@@ -27,3 +27,17 @@
       By default, the admin and peer ports (9200 and 9300) are only accessible
       to clients and peers respectively. Switch this to false to enable access
       from any machine.
+  heap_size:
+    type: string
+    default: ""
+    description: |
+      By default, the heap size will be set to the maximum of half of the
+      available memory and 30.5G. A number in gigabytes for the Elasticsearch
+      heap size.
+  nagios_health_color:
+    type: string
+    default: green
+    description: |
+      By default we expect the cluster health to be green, but in some cases,
+      such as deploying a dev setup with a single instance, we can only expect
+      yellow as there is a primary shard but no replicas.

=== added directory 'files/nrpe'
=== added file 'files/nrpe/check_health'
--- files/nrpe/check_health	1970-01-01 00:00:00 +0000
+++ files/nrpe/check_health	2016-03-10 22:57:41 +0000
@@ -0,0 +1,41 @@
+#!/usr/bin/env python
+from __future__ import print_function
+
+import json
+import urllib
+import sys
+
+
+def main(url, override_green):
+    resp = urllib.urlopen(url)
+    if resp.code == 200:
+        info = json.loads(resp.read())
+        status = info['status']
+        if status == "green" or (override_green and override_green == status):
+            print(info)
+            sys.exit(0)
+        elif status == "yellow":
+            print(info)
+            sys.exit(1)
+        elif status == "red":
+            print(info)
+            sys.exit(2)
+        else:
+            print("UNKNOWN status: {} - {}".format(status, info))
+            sys.exit(3)
+    else:
+        print('HTTP CODE: {} - BODY: {}'.format(resp.code, resp.read()))
+        sys.exit(2)
+
+if __name__ == "__main__":
+    import argparse
+    parser = argparse.ArgumentParser()
+    parser.add_argument("health_url")
+    parser.add_argument("--override-green", dest="override_green",
+                        required=False, default="")
+    args = parser.parse_args()
+    try:
+        main(args.health_url, args.override_green)
+    except Exception as exc:
+        print(exc)
+        sys.exit(3)

=== modified file 'playbook.yaml'
--- playbook.yaml	2015-10-27 22:23:15 +0000
+++ playbook.yaml	2016-03-10 22:57:41 +0000
@@ -1,9 +1,9 @@
 - hosts: localhost
   roles:
     - role: nrpe
-      check_name: check_http
-      check_params: -H localhost -u /_cluster/health -p 9200 -w 2 -c 3 -s green
-      service_description: "Verify the cluster health is green."
+      check_name: check_health
+      check_params: http://localhost:9200/_cluster/health {{ '--override-green ' + nagios_health_color if nagios_health_color and nagios_health_color != 'green' else '' }}
+      service_description: "Verify the cluster health"
 
   handlers:
 
@@ -14,6 +14,20 @@
     - service_name: "{{ local_unit.split('/')[0] }}"
 
   tasks:
+    - name: Install nrpe plugins
+      tags:
+        - install
+        - upgrade-charm
+      copy: src="{{ charm_dir }}/files/nrpe/" dest="/usr/lib/nagios/plugins" mode=0755
+
+    - name: Remove old http nrpe check 
+      tags:
+        - upgrade-charm 
+      file: path={{ item.path }} state=absent 
+      with_items: 
+        - { path: "/etc/nagios/nrpe.d/check_http.cfg" }
+        - { path: "/var/lib/nagios/export/service__{{ relations['nrpe-external-master'][0]['nagios_hostname'] }}_check_http.cfg" }
+      when: '"nrpe-external-master" in relations and relations["nrpe-external-master"]|length > 0 and relations["nrpe-external-master"][0].nagios_hostname is defined'
 
     - include: tasks/install-elasticsearch.yml
     - include: tasks/peer-relations.yml
@@ -37,6 +51,14 @@
       notify:
         - Restart ElasticSearch
 
+    - name: Update defaults
+      tags:
+        - config-changed
+      template: src={{ charm_dir }}/templates/etc_default.j2
+                dest=/etc/default/elasticsearch
+      notify:
+        - Restart ElasticSearch
+
     - name: Open ES Port when exposed
       command: open-port 9200
       tags:

=== modified file 'tasks/install-elasticsearch.yml'
--- tasks/install-elasticsearch.yml	2015-10-01 19:00:31 +0000
+++ tasks/install-elasticsearch.yml	2016-03-10 22:57:41 +0000
@@ -45,3 +45,18 @@
     - install
     - upgrade-charm
   when: stat_elasticsearch_deb.stat.exists == true
+
+- name: Set Elasticsearch user limits
+  tags:
+    - upgrade-charm
+    - install
+  template: src={{ charm_dir }}/templates/limits.conf
+            dest=/etc/security/limits.d/elasticsearch.conf
+  notify:
+    - Restart ElasticSearch
+
+- name: Ensure Elasticsearch is started on boot
+  tags:
+    - upgrade-charm
+    - install
+  service: name=elasticsearch enabled=yes

=== modified file 'templates/elasticsearch.yml'
--- templates/elasticsearch.yml	2016-03-08 16:38:17 +0000
+++ templates/elasticsearch.yml	2016-03-10 22:57:41 +0000
@@ -11,3 +11,6 @@
 {% endfor %}
 # workaround for Kibana4 Export Everything bug https://github.com/elastic/kibana/issues/5524
 index.max_result_window: 2147483647
+
+# https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html
+bootstrap.mlockall: true

=== added file 'templates/etc_default.j2'
--- templates/etc_default.j2	1970-01-01 00:00:00 +0000
+++ templates/etc_default.j2	2016-03-10 22:57:41 +0000
@@ -0,0 +1,14 @@
+# See https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html
+{% set calculated_heap_size = ansible_memtotal_mb / 1024.0 / 2.0 %}
+{# min/max filters only available in ansible 1.8 #}
+{% if calculated_heap_size >= 31.5 %}{% set calculated_heap_size = 31 %}{% endif %}
+{% set heap_size = heap_size | default(calculated_heap_size, true) | float %}
+# Note: ES doesn't start if the heap size is not an int of either gigabytes (g)
+# or megabytes (m)
+{% if heap_size > 4 %}
+ES_HEAP_SIZE={{ heap_size | int }}g
+{% else %}
+ES_HEAP_SIZE={{ (heap_size * 1024) | int }}m
+{% endif %}
+MAX_OPEN_FILES=65535
+MAX_LOCKED_MEMORY=unlimited

=== added file 'templates/limits.conf'
--- templates/limits.conf	1970-01-01 00:00:00 +0000
+++ templates/limits.conf	2016-03-10 22:57:41 +0000
@@ -0,0 +1,3 @@
+# Ensure ES user can run with max open files and unlimited memlock.
+elasticsearch - nofile 65535
+elasticsearch - memlock unlimited