Merge lp:~michael.nelson/charms/trusty/logstash/enable-custom-patterns into lp:~canonical-is-sa/charms/trusty/logstash/logstash2

[Michael Nelson]

Add extra-patterns option.
Test config before restarting (for better error message in logs).

Using the logstash charm, I'm finding for any semi-useful filters I need to define my own grok patterns.

Also took the chance to improve the config descriptions.

[Michael Nelson (michael.nelson)]

Setting back to WIP after local spec run using this charm shows an issue.

[lp:~michael.nelson/charms/trusty/logstash/enable-custom-patterns]

62. By Michael Nelson on 2015-11-12

Un-shorten URI

63. By Michael Nelson on 2015-11-12

Reformat comment

[Michael Foley (foli)]

Great to see the ability to add custom grok patterns.

And great to see it catching config errors, which would otherwise stop it from processing logs.

review: Approve

=== modified file 'config.yaml'
--- config.yaml	2015-11-04 00:26:13 +0000
+++ config.yaml	2015-11-12 23:47:27 +0000
@@ -22,7 +22,24 @@
   extra-config:
     type: string
     default: ''
-    description: "Base64-encoded custom configuration content."
+    description: |
+      Base64-encoded custom configuration content. If you have a directory of
+      logstash filters, you can set this with:
+          juju set logstash extra-config="$(cat filters/* | base64)"
+  extra-patterns:
+    type: string
+    default: ''
+    description: |
+      Base64-encoded custom grok patterns for your logstash filters. If you
+      have a directory of patterns, you can set this with:
+          juju set logstash extra-patterns="$(cat patterns/* | base64)"
+      These will be placed into the /opt/logstash/patterns directory and so can be
+      referenced in a grok filter with:
+          grok {
+            patterns_dir => "./patterns"
+            match => { ... }
+          }
+      More info at https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#_custom_patterns
   nagios_context:
     default: "juju"
     type: string

=== modified file 'hooks/config-changed'
--- hooks/config-changed	2015-10-22 05:29:50 +0000
+++ hooks/config-changed	2015-11-12 23:47:27 +0000
@@ -39,7 +39,19 @@
     # This only actually opens the port if we've exposed the service in juju
     hookenv.open_port(5043)
 
-    # Restart the service when configuration has changed.
+    # Verify the config to get a decent error message for bad configuration.
+    try:
+        subprocess.check_output([
+            '/opt/logstash/bin/logstash', 'agent', '-f',
+            '/opt/logstash/conf.d/', '--configtest'
+        ])
+    except subprocess.CalledProcessError, e:
+        # The exception doesn't print the output, by default.
+        print(e.output)
+        raise
+
+    # Restart the service when configuration has changed
+    # and the config is valid.
     subprocess.check_output(shlex.split('hooks/start'))
 
     subprocess.check_output(shlex.split('hooks/nrpe-external-master-relation-changed'))
@@ -68,6 +80,12 @@
     if config_data['extra-config']:
         with open(os.path.join(BASEPATH, 'conf.d', 'extra.conf'), 'w') as f:
             f.write(str(base64.b64decode(config_data['extra-config'])))
+    if config_data['extra-patterns']:
+        patterns_dir = os.path.join(BASEPATH, 'patterns')
+        if not os.path.exists(patterns_dir):
+            os.mkdir(patterns_dir)
+        with open(os.path.join(patterns_dir, 'extra-patterns'), 'w') as f:
+            f.write(str(base64.b64decode(config_data['extra-patterns'])))
 
     # Only setup lumberjack protocol if ssl cert and key are configured
     if config_data['ssl_cert'] and config_data['ssl_key']: