Merge lp:~michael.nelson/charms/trusty/logstash/trunk into lp:~canonical-is-sa/charms/trusty/logstash/trunk

[Michael Nelson]

Add a nagios cert check to avoid cert expiry.
Add extra_config to enable juju setting extra filters.

This branch just merges fixes that the spi (tanuki) team have added to their custom branch and adds two new features (extra-config and nagios_check_cert_params. I left out one spi-specific piece of functionality - opening 5959 for TCP/UDP using python-logstash.

Test still passes (below) but I've not tested with a deploy yet. EDIT: I've just tested this with a local deploy and it worked fine (with the new nagios check pointing out that my cert had only 29 days left of validity).

dev-trusty# ~/charms/logstash/trusty/logstash
$ ./tests/10-deploy
/usr/lib/python3/dist-packages/charmworldlib/api.py:18: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketType.SOCK_STREAM, proto=6, laddr=('10.55.32.45', 54709), raddr=('91.189.92.33', 443)>
  return self.fetch_json(endpoint, params, 'get')
2015-10-22 05:38:58 Starting deployment of local
2015-10-22 05:38:58 Deploying services...
2015-10-22 05:38:59 Deploying service elasticsearch using cs:trusty/elasticsearch-11
2015-10-22 05:39:03 Deploying service logstash using /home/michael/charms/logstash/trusty/logstash
2015-10-22 05:41:55 Adding relations...
2015-10-22 05:41:55 Adding relation elasticsearch:client <-> logstash:client
2015-10-22 05:42:56 Deployment complete in 238.55 seconds
/usr/lib/python3.4/unittest/case.py:574: ResourceWarning: unclosed <socket.socket fd=4, family=AddressFamily.AF_INET, type=SocketType.SOCK_STREAM, proto=6, laddr=('10.0.3.1', 37457), raddr=('10.0.3.73', 9200)>
  testMethod()
..
----------------------------------------------------------------------
Ran 2 tests in 339.381s

OK

[Michael Foley (foli)]

You mention specifically leaving out opening port 5959 from the spi/tanuki team merge but then the added "example 4" shows using port 5959. I think "example 4" should be left out for now or replaced with an example that actually works with this version of the charm.

review: Needs Fixing

[lp:~michael.nelson/charms/trusty/logstash/trunk]

56. By Michael Nelson on 2015-10-30

Remove example 4 with the python logstash handler.

[Michael Nelson (michael.nelson)]

> You mention specifically leaving out opening port 5959 from the spi/tanuki
> team merge but then the added "example 4" shows using port 5959. I think
> "example 4" should be left out for now or replaced with an example that
> actually works with this version of the charm.

Hrmm... I'd not noticed that was added in the tanuki branch - good spot. Removed with r56. Thanks.

[Michael Foley (foli)]

Looks good now.

review: Approve

=== modified file 'README.md'
--- README.md	2014-07-24 13:26:45 +0000
+++ README.md	2015-10-30 11:20:44 +0000
@@ -28,7 +28,7 @@
 http://ip-of-kibana
 
 example 2 - Indexer + 2 x ElasticSearch + Kibana
-============================================
+================================================
 
     juju deploy cs:trusty/elasticsearch
     juju add-unit elasticsearch
@@ -41,7 +41,7 @@
 http://ip-of-kibana
 
 example 3  - Agent + Indexer + 2 x ElasticSearch + Kibana
-=============================================
+=========================================================
 
     juju deploy cs:trusty/elasticsearch
     juju add-unit elasticsearch
@@ -54,7 +54,6 @@
     juju add-relation logstash-agent logstash-indexer:input
 
 
-
 ### Caveats
 
 The charm will fetch the logstash complete archive every time. 
@@ -63,13 +62,13 @@
 
 # Configuration
 
-The charm supports installation from anywhere that python requeusts can reach and understand. By default it will install a recent revision (1.4.2 as of this writing) from the elasticsearch.org site. this is configurable with 2 options
+The charm supports installation from anywhere that python requests can reach and understand. By default it will install a recent revision (1.4.2 as of this writing) from the elasticsearch.org site. this is configurable with 2 options
 
     juju set logstash logstash-source="https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz" logstash-sum="d59ef579c7614c5df9bd69cfdce20ed371f728ff"
 
 There is also a configuration option to add arbitrary packages pre-installation of logstash. The format is a space separated list.
 
-    juju set logstash extrapackages='vim byobu'
+    juju set logstash extra-packages='vim byobu'
 
 # Contact Information
 
 
=== modified file 'config.yaml'
--- config.yaml	2015-05-12 04:22:21 +0000
+++ config.yaml	2015-10-30 11:20:44 +0000
@@ -19,6 +19,10 @@
     default: ""
     type: string
     description: "Base64-encoded SSL key"
+  extra-config:
+    type: string
+    default: ''
+    description: "Base64-encoded custom configuration content."
   nagios_context:
     default: "juju"
     type: string
@@ -43,3 +47,7 @@
     default: "--ssl -H localhost -p 5043 -c 0.3"
     type: string
     description: The parameters to pass to the nrpe plugin check_tcp.
+  nagios_check_cert_params:
+    default: "-D 30,14 -H 127.0.0.1 -p 5043"
+    type: string
+    description: The parameters to pass to the nrpe plugin "check_tcp --ssl" to check certificate expiration date.
 
=== modified file 'hooks/client-relation-changed'
--- hooks/client-relation-changed	2015-04-21 04:25:51 +0000
+++ hooks/client-relation-changed	2015-10-30 11:20:44 +0000
@@ -29,6 +29,8 @@
 def write_config():
     with open('host_cache', 'r') as f:
         hosts = f.readlines()
+    if not hosts:
+        sys.exit(0)
 
     # Use last host in list as it will be the most recently added
     # and first host in list may not exist anymore! TODO fix that.
@@ -36,24 +38,26 @@
 
     out = os.path.join(BASEPATH, 'conf.d', 'output-elasticsearch.conf')
     with open(out, 'w') as p:
-      p.write(render(os.path.basename(out), opts))
-
+        p.write(render(os.path.basename(out), opts))
 
 
 def cache_hosts():
-    host = hookenv.relation_get('host')
-    if not host:
-      log('No host received. Assuming nothing to do.')
-      sys.exit(0)
-
+    rels = hookenv.relations_of_type("client")
+    if not rels:
+        log('No client relations. Assuming nothing to do.')
+        sys.exit(0)
     if not os.path.exists('host_cache'):
         open('host_cache', 'a').close()
-
-    with open('host_cache', 'r') as f:
-        hosts = f.readlines()
-    if not host in hosts:
-        with open('host_cache', 'a') as f:
-            f.write('{}\n'.format(host))
+    for rel in rels:
+        host = rel.get('host')
+        if not host:
+            log('No host received for relation: {}.'.format(rel))
+            continue
+        with open('host_cache', 'r') as f:
+            hosts = f.readlines()
+        if host not in hosts:
+            with open('host_cache', 'a') as f:
+                f.write('{}\n'.format(host))
 
 
 if __name__ == "__main__":
 
=== modified file 'hooks/config-changed'
--- hooks/config-changed	2015-05-05 23:38:46 +0000
+++ hooks/config-changed	2015-10-30 11:20:44 +0000
@@ -39,10 +39,10 @@
     # This only actually opens the port if we've exposed the service in juju
     hookenv.open_port(5043)
 
-    # The install hook is idempotent, so re-run it.
-    subprocess.check_output(shlex.split('hooks/install'))
+    # Restart the service when configuration has changed.
+    subprocess.check_output(shlex.split('hooks/start'))
 
-    # TODO: should call update_nrpe_checks() here.  See charmsupport/nrpe.py
+    subprocess.check_output(shlex.split('hooks/nrpe-external-master-relation-changed'))
 
 
 def copy_config():
@@ -54,11 +54,21 @@
     key_file = os.path.join(cert_dir, 'logstash.key')
 
     for f in files:
+        # skip output-elasticsearch.conf, is managed by
+        # hooks/client-relation-changed
+        if os.path.basename(f) == "output-elasticsearch.conf":
+            continue
         if os.path.basename(f) != lumberjack_template:
             with open(os.path.join(BASEPATH, 'conf.d', f), 'w') as p:
                 p.write(render(os.path.basename(f), opts))
 
     config_data = hookenv.config()
+
+    # Write custom configuration if set.
+    if config_data['extra-config']:
+        with open(os.path.join(BASEPATH, 'conf.d', 'extra.conf'), 'w') as f:
+            f.write(str(base64.b64decode(config_data['extra-config'])))
+
     # Only setup lumberjack protocol if ssl cert and key are configured
     if config_data['ssl_cert'] and config_data['ssl_key']:
         if not os.path.exists(cert_dir):
 
=== modified file 'hooks/nrpe-external-master-relation-changed'
--- hooks/nrpe-external-master-relation-changed	2015-05-05 23:38:46 +0000
+++ hooks/nrpe-external-master-relation-changed	2015-10-30 11:20:44 +0000
@@ -7,10 +7,46 @@
 
 from charmhelpers.core import hookenv
 from charmhelpers.contrib.charmsupport import nrpe
+from charmhelpers.contrib.charmsupport.nrpe import NRPE
 
 hooks = hookenv.Hooks()
 log = hookenv.log
 
+
+class CustomIntervalCheck(nrpe.Check):
+
+    service_template = ("""
+#---------------------------------------------------
+# This file is Juju managed
+#---------------------------------------------------
+define service {{
+    use                             active-service
+    host_name                       {nagios_hostname}
+    service_description             {nagios_hostname}[{shortname}] """
+                        """{description}
+    check_command                   check_nrpe!{command}
+    servicegroups                   {nagios_servicegroup}
+%s
+}}
+""")
+    intervals_template = "    {}                   {}\n"
+
+    def __init__(self, shortname, description, check_cmd, normal_check_interval=None,
+                 retry_check_interval=None, notification_interval=None):
+        super(CustomIntervalCheck, self).__init__(shortname, description, check_cmd)
+        intervals = {}
+        if normal_check_interval:
+            intervals['normal_check_interval'] = normal_check_interval
+        if retry_check_interval:
+            intervals['retry_check_interval'] = retry_check_interval
+        if notification_interval:
+            intervals['notification_interval'] = notification_interval
+        intervals_config = ""
+        for k, v in intervals.items():
+            intervals_config += self.intervals_template.format(k, v)
+        self.service_template = CustomIntervalCheck.service_template % intervals_config
+
+
 @hooks.hook('nrpe-external-master-relation-changed')
 def update_nrpe_checks():
     nrpe_compat = nrpe.NRPE()
@@ -23,6 +59,7 @@
             check_cmd='check_procs %s' % check_procs_params
         )
     check_tcp_params = conf.get('nagios_check_tcp_params')
+    check_cert_params = conf.get('nagios_check_cert_params')
     config_data = hookenv.config()
     # Only setup lumberjack protocol if ssl cert and key are configured
     if config_data['ssl_cert'] and config_data['ssl_key']:
@@ -32,6 +69,17 @@
                 description='Check logstash lumberjack input tcp port',
                 check_cmd='check_tcp %s' % check_tcp_params
             )
+        if check_cert_params:
+            # check certificate expiry date, daily and retry every 2 hs
+            cert_check = CustomIntervalCheck(
+                shortname='lumberjack_ssl_check',
+                description='Check logstash ssl certificate expiry date',
+                check_cmd='check_tcp --ssl {}'.format(check_cert_params),
+                normal_check_interval=1440,  # minutes
+                retry_check_interval=120,  # minutes
+            )
+            nrpe_compat.checks.append(cert_check)
+
     nrpe_compat.write()
 
 if __name__ == "__main__":