Merge lp:~verterok/charms/trusty/elasticsearch/elasticsearch2_more-details-in-health-check into lp:~onlineservices-charmers/charms/trusty/elasticsearch/elasticsearch2

[Guillermo Gonzalez]

Replace check_http based health nrpe check with a custom script that understands the status values.

Replace check_http based health check with a custom script that understand the status values.

[Michael Nelson (michael.nelson)]

21:41 < noodles> verterok: Is nagios_health_color currently a config option (defaulting to green)? From memory it was, because otherwise if you're developing and using only one unit, the health will always be yellow?
21:42 < noodles> If you think that's unnecessary, we can probably remove the option (as it should only be used in dev, and worst-case, anyone upgrading will see it's removed with a config error)
21:44 < caio1982> atomatt: nice! i will take a look!
21:44 < verterok> noodles: didn't know it is a config, assumed it was the way ES reported status
21:44 < verterok> let me check
21:45 < verterok> noodles: oh, I understand now
21:46 < verterok> noodles: I can tweak the script to accept a custom "OK" status if you think it's needed.
21:47 < noodles> verterok: If that's simple enough, it'll mean maintaining the ability to deploy just one unit in a devel spec without error, but it's not that important . Fine either way. The script is a great improvement :)
21:48 < verterok> noodles: will play with the script tomorrow and ping you back when it's done. thanks
21:48 < noodles> verterok: Ack - or if it's an issue, fine to just remove that config option. No need to check back - just land with either.

review: Approve

[lp:~verterok/charms/trusty/elasticsearch/elasticsearch2_more-details-in-health-check]

48. By Guillermo Gonzalez on 2016-02-16

add --override-green option to nrpe script and use nagios_health_color in nrpe check

49. By Guillermo Gonzalez on 2016-02-16

only use --override-green if nagios_health_color is != green

50. By Guillermo Gonzalez on 2016-02-16

fix conditional

[Guillermo Gonzalez (verterok)]

fixed and pushed.

I don't have perms to top-approve

=== added directory 'files/nrpe'
=== added file 'files/nrpe/check_health'
--- files/nrpe/check_health	1970-01-01 00:00:00 +0000
+++ files/nrpe/check_health	2016-02-16 16:43:37 +0000
@@ -0,0 +1,41 @@
+#!/usr/bin/env python
+from __future__ import print_function
+
+import json
+import urllib
+import sys
+
+
+def main(url, override_green):
+    resp = urllib.urlopen(url)
+    if resp.code == 200:
+        info = json.loads(resp.read())
+        status = info['status']
+        if status == "green" or (override_green and override_green == status):
+            print(info)
+            sys.exit(0)
+        elif status == "yellow":
+            print(info)
+            sys.exit(1)
+        elif status == "red":
+            print(info)
+            sys.exit(2)
+        else:
+            print("UNKNOWN status: {} - {}".format(status, info))
+            sys.exit(3)
+    else:
+        print('HTTP CODE: {} - BODY: {}'.format(resp.code, resp.read()))
+        sys.exit(2)
+
+if __name__ == "__main__":
+    import argparse
+    parser = argparse.ArgumentParser()
+    parser.add_argument("health_url")
+    parser.add_argument("--override-green", dest="override_green",
+                        required=False, default="")
+    args = parser.parse_args()
+    try:
+        main(args.health_url, args.override_green)
+    except Exception as exc:
+        print(exc)
+        sys.exit(3)

=== modified file 'playbook.yaml'
--- playbook.yaml	2015-11-26 04:10:47 +0000
+++ playbook.yaml	2016-02-16 16:43:37 +0000
@@ -1,9 +1,9 @@
 - hosts: localhost
   roles:
     - role: nrpe
-      check_name: check_http
-      check_params: -H localhost -u /_cluster/health -p 9200 -w 2 -c 3 -s {{ nagios_health_color }}
-      service_description: "Verify the cluster health is green."
+      check_name: check_health
+      check_params: http://localhost:9200/_cluster/health {{ '--override-green ' + nagios_health_color if nagios_health_color and nagios_health_color != 'green' else '' }}
+      service_description: "Verify the cluster health"
 
   handlers:
 
@@ -14,6 +14,20 @@
     - service_name: "{{ local_unit.split('/')[0] }}"
 
   tasks:
+    - name: Install nrpe plugins
+      tags:
+        - install
+        - upgrade-charm
+      copy: src="{{ charm_dir }}/files/nrpe/" dest="/usr/lib/nagios/plugins" mode=0755
+
+    - name: Remove old http nrpe check 
+      tags:
+        - upgrade-charm 
+      file: path={{ item.path }} state=absent 
+      with_items: 
+        - { path: "/etc/nagios/nrpe.d/check_http.cfg" }
+        - { path: "/var/lib/nagios/export/service__{{ relations['nrpe-external-master'][0]['nagios_hostname'] }}_check_http.cfg" }
+      when: '"nrpe-external-master" in relations and relations["nrpe-external-master"]|length > 0 and relations["nrpe-external-master"][0].nagios_hostname is defined'
 
     - include: tasks/install-elasticsearch.yml
     - include: tasks/peer-relations.yml