Merge lp:~foli/charms/trusty/logstash/add-lumberjack-protocol into lp:~lazypower/charms/trusty/logstash/trunk

[Michael Foley]

Have added input-lumberjack.conf file for lumberjack protocol used by logstash-forwarder.

Added code to only install the conf file if necessary SSL certs are available, otherwise logstash fails to start.

Added a relation to haproxy of the lumberjack protocol so that multiple logstash units can be placed behind haproxy.

Tom added support for exec.d infrastructure so we can customise instances.

[lp:~foli/charms/trusty/logstash/add-lumberjack-protocol]

46. By Charles Butler on 2015-04-20

Removed excess comments from base of charm test

47. By Charles Butler on 2015-04-20

[r,a=lazypower] Michael Foley 2015-04-17 [foley] Add myself to maintainers field
Michael Foley 2015-04-17 [merge] [mthaddon,r=foli] Add support for exec.d infrastructure so we can customise instances
Tom Haddon 2015-04-16 Add support for exec.d infrastructure so we can customise instances
Michael Foley 2015-04-17 [merge] [foley] merge in upstream changes for fixing local tarball installation
Michael Foley 2015-04-15 [foli] changed lumberjack-relation-joined lumberjack-relation-changed with symlink for joined
Michael Foley 2015-04-15 [foli] add relation for haproxy to proxy lumberjack on port 5043
Michael Foley 2015-04-07 [foli] lumberjack conf file only enabled if ssl cert/key are configured
Michael Foley 2015-04-07 [merge] [mthaddon,r=foli] Open the port we need for the service
Tom Haddon 2015-04-02 Open the port we need for the service
Michael Foley 2015-04-01 [foli] Fix more copy-paste errors by adding necessary imports, fix file perms for ssl
Michael Foley 2015-04-01 [foli] Fix copy-paste error for config_data for ssl cert/key
Michael Foley 2015-03-31 [foli] Added lumberjack input conf file, added ssl_cert & ssl_key configuration

[Charles Butler (lazypower)]

All of this looks good to me. I just have a single comment - that we need to get a test built that leverages the lumberjack relationship and validate everything is working so we know if it breaks in the future.

I'll assign a bug for this and happily merge. Thanks for the contributions!

review: Approve

=== modified file 'config.yaml'
--- config.yaml	2014-08-18 20:06:27 +0000
+++ config.yaml	2015-04-17 08:58:06 +0000
@@ -11,3 +11,11 @@
     type: string
     default: ''
     description: Space separated list of extra apt packages to install.
+  ssl_cert:
+    default: ""
+    type: string
+    description: "Base64-encoded SSL certificate"
+  ssl_key:
+    default: ""
+    type: string
+    description: "Base64-encoded SSL key"

=== added directory 'exec.d'
=== modified file 'hooks/config-changed'
--- hooks/config-changed	2014-09-23 18:18:13 +0000
+++ hooks/config-changed	2015-04-17 08:58:06 +0000
@@ -4,6 +4,9 @@
 import shlex
 import subprocess
 import sys
+import pwd
+import grp
+import base64
 
 
 sys.path.insert(0, os.path.join(os.environ['CHARM_DIR'], 'lib'))
@@ -33,6 +36,9 @@
     copy_config()
     place_upstart_template()
 
+    # This only actually opens the port if we've exposed the service in juju
+    hookenv.open_port(5043)
+
     # The install hook is idempotent, so re-run it.
     subprocess.check_output(shlex.split('hooks/install'))
 
@@ -40,9 +46,36 @@
 def copy_config():
     files = os.listdir('templates')
     opts = {'BASEPATH': BASEPATH}
+    lumberjack_template = "input-lumberjack.conf"
+    cert_dir = os.path.join(BASEPATH, 'ssl')
+    cert_file = os.path.join(cert_dir, 'logstash.crt')
+    key_file = os.path.join(cert_dir, 'logstash.key')
+
     for f in files:
-      with open(os.path.join(BASEPATH, 'conf.d', f), 'w') as p:
-        p.write(render(os.path.basename(f), opts))
+        if os.path.basename(f) != lumberjack_template:
+            with open(os.path.join(BASEPATH, 'conf.d', f), 'w') as p:
+                p.write(render(os.path.basename(f), opts))
+
+    config_data = hookenv.config()
+    # Only setup lumberjack protocol if ssl cert and key are configured
+    if config_data['ssl_cert'] and config_data['ssl_key']:
+        if not os.path.exists(cert_dir):
+            os.mkdir(cert_dir, 0770)
+            os.chown(cert_dir, pwd.getpwnam('logstash').pw_uid, grp.getgrnam('logstash').gr_gid)
+
+        # Certificate provided as base64-encoded string.
+        if config_data['ssl_cert']:
+            log("Writing cert from config ssl_cert: %s" % cert_file)
+            with open(cert_file, 'w') as f:
+                f.write(str(base64.b64decode(config_data['ssl_cert'])))
+        # Private key provided as base64-encoded string.
+        if config_data['ssl_key']:
+            log("Writing key from config ssl_key: %s" % key_file)
+            with open(key_file, 'w') as f:
+                f.write(str(base64.b64decode(config_data['ssl_key'])))
+
+        with open(os.path.join(BASEPATH, 'conf.d', lumberjack_template), 'w') as p:
+            p.write(render(os.path.basename(lumberjack_template), opts))
 
 
 def place_upstart_template():
@@ -51,8 +84,7 @@
     opts = {'BASEPATH': BASEPATH}
 
     with open(out, 'w') as p:
-      p.write(render('{}.conf'.format(SERVICE), opts, template_dir=templ))
-
+        p.write(render('{}.conf'.format(SERVICE), opts, template_dir=templ))
 
 
 if __name__ == "__main__":

=== modified file 'hooks/install'
--- hooks/install	2015-03-30 18:36:01 +0000
+++ hooks/install	2015-04-17 08:58:06 +0000
@@ -22,6 +22,8 @@
     apt_install,
 )
 
+from charmhelpers.payload.execd import execd_preinstall
+
 hooks = Hooks()
 log = log
 
@@ -31,6 +33,7 @@
 
 @hooks.hook('install')
 def install():
+    execd_preinstall()
     log('Installing logstash')
     packages = ['openjdk-7-jre-headless', 'redis-server', 'python-jinja2']
     extra_packages = config('extra-packages')

=== added file 'hooks/lumberjack-relation-changed'
--- hooks/lumberjack-relation-changed	1970-01-01 00:00:00 +0000
+++ hooks/lumberjack-relation-changed	2015-04-17 08:58:06 +0000
@@ -0,0 +1,7 @@
+#!/bin/bash
+
+set -eux
+
+hostname=$(unit-get private-address)
+juju-log "Setting lumberjack URL to $hostname:5043"
+relation-set port=5043 hostname=$hostname

=== added symlink 'hooks/lumberjack-relation-joined'
=== target is u'lumberjack-relation-changed'
=== modified file 'metadata.yaml'
--- metadata.yaml	2014-10-29 16:10:22 +0000
+++ metadata.yaml	2015-04-17 08:58:06 +0000
@@ -1,6 +1,8 @@
 name: logstash
 summary: "Logstash indexer server"
-maintainer: charles <[e-mail]>
+maintainers:
+  - charles <[e-mail]> 
+  - Michael Foley <[e-mail]>
 description: |
   Installs the logstash indexer.   Initially stand alone version with a
   dependency on elasticsearch as the backend provider.
@@ -12,6 +14,8 @@
     interface: redis
   rest:
     interface: elasticsearch
+  lumberjack:
+    interface: http
 requires:
   client:
     interface: elasticsearch

=== added file 'templates/input-lumberjack.conf'
--- templates/input-lumberjack.conf	1970-01-01 00:00:00 +0000
+++ templates/input-lumberjack.conf	2015-04-17 08:58:06 +0000
@@ -0,0 +1,13 @@
+input {
+  lumberjack {
+    # The port to listen on
+    port => 5043
+
+    # The paths to your ssl cert and key
+    ssl_certificate => "{{BASEPATH}}/ssl/logstash.crt"
+    ssl_key => "{{BASEPATH}}/ssl/logstash.key"
+
+    # Set this to whatever you want.
+    type => "lumberjack"
+  }
+}