Merge lp:~michael.nelson/charms/trusty/logstash/trunk into lp:~canonical-is-sa/charms/trusty/logstash/trunk

Proposed by Michael Nelson on 2015-10-22
Status: Merged
Merged at revision: 54
Proposed branch: lp:~michael.nelson/charms/trusty/logstash/trunk
Merge into: lp:~canonical-is-sa/charms/trusty/logstash/trunk
Diff against target: 238 lines (+90/-21)
5 files modified
README.md (+4/-5)
config.yaml (+8/-0)
hooks/client-relation-changed (+17/-13)
hooks/config-changed (+13/-3)
hooks/nrpe-external-master-relation-changed (+48/-0)
To merge this branch: bzr merge lp:~michael.nelson/charms/trusty/logstash/trunk
Reviewer Review Type Date Requested Status
Michael Foley (community) 2015-10-22 Approve on 2015-11-01
Review via email: mp+275269@code.launchpad.net

Commit message

Add a nagios cert check to avoid cert expiry.
Add extra_config to enable juju setting extra filters.

Description of the change

This branch just merges fixes that the spi (tanuki) team have added to their custom branch and adds two new features (extra-config and nagios_check_cert_params. I left out one spi-specific piece of functionality - opening 5959 for TCP/UDP using python-logstash.

Test still passes (below) but I've not tested with a deploy yet. EDIT: I've just tested this with a local deploy and it worked fine (with the new nagios check pointing out that my cert had only 29 days left of validity).

dev-trusty# ~/charms/logstash/trusty/logstash
$ ./tests/10-deploy
/usr/lib/python3/dist-packages/charmworldlib/api.py:18: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketType.SOCK_STREAM, proto=6, laddr=('10.55.32.45', 54709), raddr=('91.189.92.33', 443)>
  return self.fetch_json(endpoint, params, 'get')
2015-10-22 05:38:58 Starting deployment of local
2015-10-22 05:38:58 Deploying services...
2015-10-22 05:38:59 Deploying service elasticsearch using cs:trusty/elasticsearch-11
2015-10-22 05:39:03 Deploying service logstash using /home/michael/charms/logstash/trusty/logstash
2015-10-22 05:41:55 Adding relations...
2015-10-22 05:41:55 Adding relation elasticsearch:client <-> logstash:client
2015-10-22 05:42:56 Deployment complete in 238.55 seconds
/usr/lib/python3.4/unittest/case.py:574: ResourceWarning: unclosed <socket.socket fd=4, family=AddressFamily.AF_INET, type=SocketType.SOCK_STREAM, proto=6, laddr=('10.0.3.1', 37457), raddr=('10.0.3.73', 9200)>
  testMethod()
..
----------------------------------------------------------------------
Ran 2 tests in 339.381s

OK

To post a comment you must log in.
Michael Foley (foli) wrote :

You mention specifically leaving out opening port 5959 from the spi/tanuki team merge but then the added "example 4" shows using port 5959. I think "example 4" should be left out for now or replaced with an example that actually works with this version of the charm.

review: Needs Fixing
56. By Michael Nelson on 2015-10-30

Remove example 4 with the python logstash handler.

Michael Nelson (michael.nelson) wrote :

> You mention specifically leaving out opening port 5959 from the spi/tanuki
> team merge but then the added "example 4" shows using port 5959. I think
> "example 4" should be left out for now or replaced with an example that
> actually works with this version of the charm.

Hrmm... I'd not noticed that was added in the tanuki branch - good spot. Removed with r56. Thanks.

Michael Foley (foli) wrote :

Looks good now.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'README.md'
2--- README.md 2014-07-24 13:26:45 +0000
3+++ README.md 2015-10-30 11:20:44 +0000
4@@ -28,7 +28,7 @@
5 http://ip-of-kibana
6
7 example 2 - Indexer + 2 x ElasticSearch + Kibana
8-============================================
9+================================================
10
11 juju deploy cs:trusty/elasticsearch
12 juju add-unit elasticsearch
13@@ -41,7 +41,7 @@
14 http://ip-of-kibana
15
16 example 3 - Agent + Indexer + 2 x ElasticSearch + Kibana
17-=============================================
18+=========================================================
19
20 juju deploy cs:trusty/elasticsearch
21 juju add-unit elasticsearch
22@@ -54,7 +54,6 @@
23 juju add-relation logstash-agent logstash-indexer:input
24
25
26-
27 ### Caveats
28
29 The charm will fetch the logstash complete archive every time.
30@@ -63,13 +62,13 @@
31
32 # Configuration
33
34-The charm supports installation from anywhere that python requeusts can reach and understand. By default it will install a recent revision (1.4.2 as of this writing) from the elasticsearch.org site. this is configurable with 2 options
35+The charm supports installation from anywhere that python requests can reach and understand. By default it will install a recent revision (1.4.2 as of this writing) from the elasticsearch.org site. this is configurable with 2 options
36
37 juju set logstash logstash-source="https://download.elasticsearch.org/logstash/logstash/logstash-1.4.2.tar.gz" logstash-sum="d59ef579c7614c5df9bd69cfdce20ed371f728ff"
38
39 There is also a configuration option to add arbitrary packages pre-installation of logstash. The format is a space separated list.
40
41- juju set logstash extrapackages='vim byobu'
42+ juju set logstash extra-packages='vim byobu'
43
44 # Contact Information
45
46
47=== modified file 'config.yaml'
48--- config.yaml 2015-05-12 04:22:21 +0000
49+++ config.yaml 2015-10-30 11:20:44 +0000
50@@ -19,6 +19,10 @@
51 default: ""
52 type: string
53 description: "Base64-encoded SSL key"
54+ extra-config:
55+ type: string
56+ default: ''
57+ description: "Base64-encoded custom configuration content."
58 nagios_context:
59 default: "juju"
60 type: string
61@@ -43,3 +47,7 @@
62 default: "--ssl -H localhost -p 5043 -c 0.3"
63 type: string
64 description: The parameters to pass to the nrpe plugin check_tcp.
65+ nagios_check_cert_params:
66+ default: "-D 30,14 -H 127.0.0.1 -p 5043"
67+ type: string
68+ description: The parameters to pass to the nrpe plugin "check_tcp --ssl" to check certificate expiration date.
69
70=== modified file 'hooks/client-relation-changed'
71--- hooks/client-relation-changed 2015-04-21 04:25:51 +0000
72+++ hooks/client-relation-changed 2015-10-30 11:20:44 +0000
73@@ -29,6 +29,8 @@
74 def write_config():
75 with open('host_cache', 'r') as f:
76 hosts = f.readlines()
77+ if not hosts:
78+ sys.exit(0)
79
80 # Use last host in list as it will be the most recently added
81 # and first host in list may not exist anymore! TODO fix that.
82@@ -36,24 +38,26 @@
83
84 out = os.path.join(BASEPATH, 'conf.d', 'output-elasticsearch.conf')
85 with open(out, 'w') as p:
86- p.write(render(os.path.basename(out), opts))
87-
88+ p.write(render(os.path.basename(out), opts))
89
90
91 def cache_hosts():
92- host = hookenv.relation_get('host')
93- if not host:
94- log('No host received. Assuming nothing to do.')
95- sys.exit(0)
96-
97+ rels = hookenv.relations_of_type("client")
98+ if not rels:
99+ log('No client relations. Assuming nothing to do.')
100+ sys.exit(0)
101 if not os.path.exists('host_cache'):
102 open('host_cache', 'a').close()
103-
104- with open('host_cache', 'r') as f:
105- hosts = f.readlines()
106- if not host in hosts:
107- with open('host_cache', 'a') as f:
108- f.write('{}\n'.format(host))
109+ for rel in rels:
110+ host = rel.get('host')
111+ if not host:
112+ log('No host received for relation: {}.'.format(rel))
113+ continue
114+ with open('host_cache', 'r') as f:
115+ hosts = f.readlines()
116+ if host not in hosts:
117+ with open('host_cache', 'a') as f:
118+ f.write('{}\n'.format(host))
119
120
121 if __name__ == "__main__":
122
123=== modified file 'hooks/config-changed'
124--- hooks/config-changed 2015-05-05 23:38:46 +0000
125+++ hooks/config-changed 2015-10-30 11:20:44 +0000
126@@ -39,10 +39,10 @@
127 # This only actually opens the port if we've exposed the service in juju
128 hookenv.open_port(5043)
129
130- # The install hook is idempotent, so re-run it.
131- subprocess.check_output(shlex.split('hooks/install'))
132+ # Restart the service when configuration has changed.
133+ subprocess.check_output(shlex.split('hooks/start'))
134
135- # TODO: should call update_nrpe_checks() here. See charmsupport/nrpe.py
136+ subprocess.check_output(shlex.split('hooks/nrpe-external-master-relation-changed'))
137
138
139 def copy_config():
140@@ -54,11 +54,21 @@
141 key_file = os.path.join(cert_dir, 'logstash.key')
142
143 for f in files:
144+ # skip output-elasticsearch.conf, is managed by
145+ # hooks/client-relation-changed
146+ if os.path.basename(f) == "output-elasticsearch.conf":
147+ continue
148 if os.path.basename(f) != lumberjack_template:
149 with open(os.path.join(BASEPATH, 'conf.d', f), 'w') as p:
150 p.write(render(os.path.basename(f), opts))
151
152 config_data = hookenv.config()
153+
154+ # Write custom configuration if set.
155+ if config_data['extra-config']:
156+ with open(os.path.join(BASEPATH, 'conf.d', 'extra.conf'), 'w') as f:
157+ f.write(str(base64.b64decode(config_data['extra-config'])))
158+
159 # Only setup lumberjack protocol if ssl cert and key are configured
160 if config_data['ssl_cert'] and config_data['ssl_key']:
161 if not os.path.exists(cert_dir):
162
163=== modified file 'hooks/nrpe-external-master-relation-changed'
164--- hooks/nrpe-external-master-relation-changed 2015-05-05 23:38:46 +0000
165+++ hooks/nrpe-external-master-relation-changed 2015-10-30 11:20:44 +0000
166@@ -7,10 +7,46 @@
167
168 from charmhelpers.core import hookenv
169 from charmhelpers.contrib.charmsupport import nrpe
170+from charmhelpers.contrib.charmsupport.nrpe import NRPE
171
172 hooks = hookenv.Hooks()
173 log = hookenv.log
174
175+
176+class CustomIntervalCheck(nrpe.Check):
177+
178+ service_template = ("""
179+#---------------------------------------------------
180+# This file is Juju managed
181+#---------------------------------------------------
182+define service {{
183+ use active-service
184+ host_name {nagios_hostname}
185+ service_description {nagios_hostname}[{shortname}] """
186+ """{description}
187+ check_command check_nrpe!{command}
188+ servicegroups {nagios_servicegroup}
189+%s
190+}}
191+""")
192+ intervals_template = " {} {}\n"
193+
194+ def __init__(self, shortname, description, check_cmd, normal_check_interval=None,
195+ retry_check_interval=None, notification_interval=None):
196+ super(CustomIntervalCheck, self).__init__(shortname, description, check_cmd)
197+ intervals = {}
198+ if normal_check_interval:
199+ intervals['normal_check_interval'] = normal_check_interval
200+ if retry_check_interval:
201+ intervals['retry_check_interval'] = retry_check_interval
202+ if notification_interval:
203+ intervals['notification_interval'] = notification_interval
204+ intervals_config = ""
205+ for k, v in intervals.items():
206+ intervals_config += self.intervals_template.format(k, v)
207+ self.service_template = CustomIntervalCheck.service_template % intervals_config
208+
209+
210 @hooks.hook('nrpe-external-master-relation-changed')
211 def update_nrpe_checks():
212 nrpe_compat = nrpe.NRPE()
213@@ -23,6 +59,7 @@
214 check_cmd='check_procs %s' % check_procs_params
215 )
216 check_tcp_params = conf.get('nagios_check_tcp_params')
217+ check_cert_params = conf.get('nagios_check_cert_params')
218 config_data = hookenv.config()
219 # Only setup lumberjack protocol if ssl cert and key are configured
220 if config_data['ssl_cert'] and config_data['ssl_key']:
221@@ -32,6 +69,17 @@
222 description='Check logstash lumberjack input tcp port',
223 check_cmd='check_tcp %s' % check_tcp_params
224 )
225+ if check_cert_params:
226+ # check certificate expiry date, daily and retry every 2 hs
227+ cert_check = CustomIntervalCheck(
228+ shortname='lumberjack_ssl_check',
229+ description='Check logstash ssl certificate expiry date',
230+ check_cmd='check_tcp --ssl {}'.format(check_cert_params),
231+ normal_check_interval=1440, # minutes
232+ retry_check_interval=120, # minutes
233+ )
234+ nrpe_compat.checks.append(cert_check)
235+
236 nrpe_compat.write()
237
238 if __name__ == "__main__":

Subscribers

People subscribed via source and target branches