Merge lp:~alexlist/charms/trusty/swift-storage/nrpe-rebased into lp:~openstack-charmers-archive/charms/trusty/swift-storage/next

Proposed by Alexander List
Status: Needs review
Proposed branch: lp:~alexlist/charms/trusty/swift-storage/nrpe-rebased
Merge into: lp:~openstack-charmers-archive/charms/trusty/swift-storage/next
Diff against target: 931 lines (+691/-32)
15 files modified
charm-helpers-hooks.yaml (+1/-0)
config.yaml (+14/-0)
files/nagios/check_swift_service (+25/-0)
files/nagios/check_swift_storage.py (+136/-0)
files/sudo/swift-storage (+1/-0)
hooks/charmhelpers/contrib/charmsupport/nrpe.py (+219/-0)
hooks/charmhelpers/contrib/charmsupport/volumes.py (+159/-0)
hooks/swift_storage_hooks.py (+58/-2)
hooks/swift_storage_utils.py (+33/-2)
metadata.yaml (+3/-0)
revision (+1/-1)
templates/050-swift-storage (+24/-0)
templates/rsyncd.conf (+5/-23)
unit_tests/test_swift_storage_relations.py (+10/-3)
unit_tests/test_swift_storage_utils.py (+2/-1)
To merge this branch: bzr merge lp:~alexlist/charms/trusty/swift-storage/nrpe-rebased
Reviewer Review Type Date Requested Status
James Page Needs Fixing
Review via email: mp+231406@code.launchpad.net

This proposal supersedes a proposal from 2014-07-31.

Description of the change

This change adds nrpe-external-master support to the charm.

Contains:

- re-sync of charmsupport
- renamed basic checks for swift ring hashes and replication
- status checks for swift storage services
- sudo permissions for above status checks

Rebase on -next as of 2014-08-19

To post a comment you must log in.
Revision history for this message
James Page (james-page) wrote : Posted in a previous version of this proposal

This merge proposal appears to have more than just the nrpe support - lots of changes around how rsync is managed as well?

Was this intentional? if so I would prefer that they where split out into two MP's to make it easier to test/review.

review: Needs Information
Revision history for this message
Jacek Nykis (jacekn) wrote : Posted in a previous version of this proposal

> This merge proposal appears to have more than just the nrpe support - lots of
> changes around how rsync is managed as well?
>
> Was this intentional? if so I would prefer that they where split out into two
> MP's to make it easier to test/review.

Hi James,

Yes it was intentional. The swift-storage charm assumed full control over rsync config which could lead to broken configuration when used with subordinate charms.

So the rsync management changes are prerequisite for n-e-m support because n-e-m needs to be able to add rsync stanzas to the config without destroying swift config (and the swift charm needs to be able to update its config without breaking n-e-m config)

Revision history for this message
Gareth Woolridge (moon127) wrote : Posted in a previous version of this proposal

James,

Rebase of Jacek's earlier work to apply cleanly to /next as we needed to pull in some other updates to PS4.

Can you respond to Jacek's earlier comment above? The nagios-external-master support and rsync changes go hand in hand.

Revision history for this message
James Page (james-page) wrote :

OK _ sorry it took me a while to get back to this; currently upgrades are broken and one of the unit tests fails due to incorrect ordering in the expected calls list - other than that LGTM.

review: Needs Fixing
Revision history for this message
Ryan Beisner (1chb1n) wrote :

UOSCI bot says:
This MP triggered a test on the Ubuntu OSCI system. Here is a summary of results.

#642 swift-storage-next for alexlist mp231406
charm_lint_check

This build:
http://10.98.191.181:8080/job/charm_lint_check/642/

MP URL:
https://code.launchpad.net/~alexlist/charms/trusty/swift-storage/nrpe-rebased/+merge/231406

Proposed branch:
lp:~alexlist/charms/trusty/swift-storage/nrpe-rebased

Results summary:
    LINT FAIL: lint-check failed

LINT Results (max last 25 lines) from
/var/lib/jenkins/workspace/charm_lint_check/make-lint.642:
hooks/swift_storage_hooks.py:45:1: E302 expected 2 blank lines, found 1
hooks/swift_storage_hooks.py:104:1: E302 expected 2 blank lines, found 1
hooks/swift_storage_hooks.py:135:22: E251 unexpected spaces around keyword / parameter equals
hooks/swift_storage_hooks.py:135:24: E251 unexpected spaces around keyword / parameter equals
hooks/swift_storage_hooks.py:139:1: E302 expected 2 blank lines, found 1
hooks/swift_storage_utils.py:96:1: E302 expected 2 blank lines, found 1
hooks/swift_storage_utils.py:240:1: E302 expected 2 blank lines, found 1
hooks/swift_storage_utils.py:249:1: W391 blank line at end of file
make: *** [lint] Error 1

Ubuntu OSCI Jenkins is currently in development on a Canonical private network, but we plan to publish results to a public instance soon. Tests are triggered if the proposed branch rev changes, or if the MP is placed into "Needs review" status after being otherwise for >= 1hr. Human review of results is still recommended.
http://10.98.191.181:8080/

Revision history for this message
Ryan Beisner (1chb1n) wrote :

UOSCI bot says:
This MP triggered a test on the Ubuntu OSCI system. Here is a summary of results.

#459 swift-storage-next for alexlist mp231406
charm_unit_test

This build:
http://10.98.191.181:8080/job/charm_unit_test/459/

MP URL:
https://code.launchpad.net/~alexlist/charms/trusty/swift-storage/nrpe-rebased/+merge/231406

Proposed branch:
lp:~alexlist/charms/trusty/swift-storage/nrpe-rebased

Results summary:
    UNIT FAIL: unit-test failed

UNIT Results (max last 25 lines) from
/var/lib/jenkins/workspace/charm_unit_test/unit-test.459:
 call('/srv/node', owner='swift', group='swift'),
 call('/etc/rsyncd.d', owner='root', group='root')]

======================================================================
FAIL: test_fetch_swift_rings (unit_tests.test_swift_storage_utils.SwiftStorageUtilsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/var/lib/jenkins/checkout/swift-storage/unit_tests/test_swift_storage_utils.py", line 99, in test_fetch_swift_rings
    self.assertEquals(wgets, self.check_call.call_args_list)
AssertionError: [call(['wget', 'http://someproxynode/rings/account.ring.gz', '-O', '/etc/swift/account.ring.gz']), call(['wget', 'http://someproxynode/rings/object.ring.gz', '-O', '/etc/swift/object.ring.gz']), call(['wget', 'http://someproxynode/rings/container.ring.gz', '-O', '/etc/swift/container.ring.gz'])] != [call(['wget', 'http://someproxynode/rings/account.ring.gz', '--retry-connrefused', '-t', '10', '-O', '/etc/swift/account.ring.gz']),
 call(['wget', 'http://someproxynode/rings/object.ring.gz', '--retry-connrefused', '-t', '10', '-O', '/etc/swift/object.ring.gz']),
 call(['wget', 'http://someproxynode/rings/container.ring.gz', '--retry-connrefused', '-t', '10', '-O', '/etc/swift/container.ring.gz'])]

Name Stmts Miss Cover Missing
-----------------------------------------------------
swift_storage_context 40 0 100%
swift_storage_hooks 70 16 77% 72, 77, 107-137
swift_storage_utils 111 8 93% 241-248
-----------------------------------------------------
TOTAL 221 24 89%
----------------------------------------------------------------------
Ran 31 tests in 1.384s

FAILED (failures=2)
make: *** [unit_test] Error 1

Ubuntu OSCI Jenkins is currently in development on a Canonical private network, but we plan to publish results to a public instance soon. Tests are triggered if the proposed branch rev changes, or if the MP is placed into "Needs review" status after being otherwise for >= 1hr. Human review of results is still recommended.
http://10.98.191.181:8080/

Revision history for this message
Ryan Beisner (1chb1n) wrote :

UOSCI bot says:
This MP triggered a test on the Ubuntu OSCI system. Here is a summary of results.

#653 swift-storage-next for alexlist mp231406
charm_lint_check

This build:
http://10.98.191.181:8080/job/charm_lint_check/653/

MP URL:
https://code.launchpad.net/~alexlist/charms/trusty/swift-storage/nrpe-rebased/+merge/231406

Proposed branch:
lp:~alexlist/charms/trusty/swift-storage/nrpe-rebased

Results summary:
    LINT FAIL: lint-check failed

LINT Results (max last 25 lines) from
/var/lib/jenkins/workspace/charm_lint_check/make-lint.653:
hooks/swift_storage_hooks.py:45:1: E302 expected 2 blank lines, found 1
hooks/swift_storage_hooks.py:104:1: E302 expected 2 blank lines, found 1
hooks/swift_storage_hooks.py:135:22: E251 unexpected spaces around keyword / parameter equals
hooks/swift_storage_hooks.py:135:24: E251 unexpected spaces around keyword / parameter equals
hooks/swift_storage_hooks.py:139:1: E302 expected 2 blank lines, found 1
hooks/swift_storage_utils.py:96:1: E302 expected 2 blank lines, found 1
hooks/swift_storage_utils.py:240:1: E302 expected 2 blank lines, found 1
hooks/swift_storage_utils.py:249:1: W391 blank line at end of file
make: *** [lint] Error 1

Ubuntu OSCI Jenkins is currently in development on a Canonical private network, but we plan to publish results to a public instance soon. Tests are triggered if the proposed branch rev changes, or if the MP is placed into "Needs review" status after being otherwise for >= 1hr. Human review of results is still recommended.
http://10.98.191.181:8080/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_lint_check #4234 swift-storage-next for alexlist mp231406
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/10965097/
Build: http://10.245.162.77:8080/job/charm_lint_check/4234/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_lint_check #4251 swift-storage-next for alexlist mp231406
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/10965294/
Build: http://10.245.162.77:8080/job/charm_lint_check/4251/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_unit_test #3977 swift-storage-next for alexlist mp231406
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [unit_test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/10965295/
Build: http://10.245.162.77:8080/job/charm_unit_test/3977/

Unmerged revisions

45. By Alexander List on 2014-08-19

[alexlist] move Nagios checks from scripts/ to files/, add service checks, add sudo permissions for service checks

44. By Alexander List on 2014-08-19

[alexlist] Initial support for nrpe-external-master, fix rsync configuration to support /etc/rsync.d

43. By Alexander List on 2014-08-19

[alexlist] re-sync charm-helpers, import charmsupport

42. By Alexander List on 2014-08-19

[alexlist] add config parameters for nrpe-external-master support

41. By Alexander List on 2014-08-19

[alexlist] add charmsupport dependency

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'charm-helpers-hooks.yaml'
2--- charm-helpers-hooks.yaml 2014-07-28 12:08:48 +0000
3+++ charm-helpers-hooks.yaml 2014-08-19 15:54:23 +0000
4@@ -10,3 +10,4 @@
5 - cluster
6 - payload.execd
7 - contrib.network.ip
8+ - contrib.charmsupport
9
10=== modified file 'config.yaml'
11--- config.yaml 2014-06-19 08:40:58 +0000
12+++ config.yaml 2014-08-19 15:54:23 +0000
13@@ -63,3 +63,17 @@
14 to not use a per-disk thread pool. It is recommended to keep this value
15 small, as large values can result in high read latencies due to large
16 queue depths. A good starting point is 4 threads per disk.
17+ nagios-check-params:
18+ default: "-m -r 60 180 10 20"
19+ type: string
20+ description: String appended to nagios check
21+ nagios_context:
22+ default: "juju"
23+ type: string
24+ description: |
25+ Used by the nrpe-external-master subordinate charm.
26+ A string that will be prepended to instance name to set the host name
27+ in nagios. So for instance the hostname would be something like:
28+ juju-myservice-0
29+ If you're running multiple environments with the same services in them
30+ this allows you to differentiate between them.
31
32=== added directory 'files'
33=== added directory 'files/nagios'
34=== added file 'files/nagios/check_swift_service'
35--- files/nagios/check_swift_service 1970-01-01 00:00:00 +0000
36+++ files/nagios/check_swift_service 2014-08-19 15:54:23 +0000
37@@ -0,0 +1,25 @@
38+#!/bin/sh
39+
40+#
41+# check_swift_service --- exactly what it says on the tin
42+#
43+# Copyright 2013 Canonical Ltd.
44+#
45+# Authors:
46+# Paul Collins <paul.collins@canonical.com>
47+#
48+
49+STATUS=$(sudo -u swift swift-init status "$1" 2>&1)
50+
51+case $? in
52+ 0)
53+ echo "OK: ${STATUS}"
54+ exit 0
55+ ;;
56+ *)
57+ echo "CRITICAL: ${STATUS}"
58+ exit 2
59+ ;;
60+esac
61+
62+exit 1
63
64=== added file 'files/nagios/check_swift_storage.py'
65--- files/nagios/check_swift_storage.py 1970-01-01 00:00:00 +0000
66+++ files/nagios/check_swift_storage.py 2014-08-19 15:54:23 +0000
67@@ -0,0 +1,136 @@
68+#!/usr/bin/env python
69+
70+# Copyright (C) 2014 Canonical
71+# All Rights Reserved
72+# Author: Jacek Nykis
73+
74+import sys
75+import json
76+import urllib2
77+import argparse
78+import hashlib
79+import datetime
80+
81+STATUS_OK = 0
82+STATUS_WARN = 1
83+STATUS_CRIT = 2
84+STATUS_UNKNOWN = 3
85+
86+
87+def generate_md5(filename):
88+ with open(filename, 'rb') as f:
89+ md5 = hashlib.md5()
90+ buffer = f.read(2 ** 20)
91+ while buffer:
92+ md5.update(buffer)
93+ buffer = f.read(2 ** 20)
94+ return md5.hexdigest()
95+
96+
97+def check_md5(base_url):
98+ url = base_url + "ringmd5"
99+ ringfiles = ["/etc/swift/object.ring.gz",
100+ "/etc/swift/account.ring.gz",
101+ "/etc/swift/container.ring.gz"]
102+ results = []
103+ try:
104+ data = urllib2.urlopen(url).read()
105+ j = json.loads(data)
106+ except urllib2.URLError:
107+ return [(STATUS_UNKNOWN, "Can't open url: {}".format(url))]
108+ except ValueError:
109+ return [(STATUS_UNKNOWN, "Can't parse status data")]
110+
111+ for ringfile in ringfiles:
112+ try:
113+ if generate_md5(ringfile) != j[ringfile]:
114+ results.append((STATUS_CRIT,
115+ "Ringfile {} MD5 sum mismatch".format(ringfile)))
116+ except IOError:
117+ results.append(
118+ (STATUS_UNKNOWN, "Can't open ringfile {}".format(ringfile)))
119+ if results:
120+ return results
121+ else:
122+ return [(STATUS_OK, "OK")]
123+
124+
125+def check_replication(base_url, limits):
126+ types = ["account", "object", "container"]
127+ results = []
128+ for repl in types:
129+ url = base_url + "replication/" + repl
130+ try:
131+ data = urllib2.urlopen(url).read()
132+ j = json.loads(data)
133+ except urllib2.URLError:
134+ results.append((STATUS_UNKNOWN, "Can't open url: {}".format(url)))
135+ continue
136+ except ValueError:
137+ results.append((STATUS_UNKNOWN, "Can't parse status data"))
138+ continue
139+
140+ if "object_replication_last" in j:
141+ repl_last = datetime.datetime.fromtimestamp(j["object_replication_last"])
142+ else:
143+ repl_last = datetime.datetime.fromtimestamp(j["replication_last"])
144+ delta = datetime.datetime.now() - repl_last
145+ if delta.seconds >= limits[1]:
146+ results.append((STATUS_CRIT,
147+ "'{}' replication lag is {} seconds".format(repl, delta.seconds)))
148+ elif delta.seconds >= limits[0]:
149+ results.append((STATUS_WARN,
150+ "'{}' replication lag is {} seconds".format(repl, delta.seconds)))
151+ if "replication_stats" in j:
152+ errors = j["replication_stats"]["failure"]
153+ if errors >= limits[3]:
154+ results.append(
155+ (STATUS_CRIT, "{} replication failures".format(errors)))
156+ elif errors >= limits[2]:
157+ results.append(
158+ (STATUS_WARN, "{} replication failures".format(errors)))
159+ if results:
160+ return results
161+ else:
162+ return [(STATUS_OK, "OK")]
163+
164+
165+if __name__ == '__main__':
166+ parser = argparse.ArgumentParser(description='Check swift-storage health')
167+ parser.add_argument('-H', '--host', dest='host', default='localhost',
168+ help='Hostname to query')
169+ parser.add_argument('-p', '--port', dest='port', default='6000',
170+ type=int, help='Port number')
171+ parser.add_argument('-r', '--replication', dest='check_replication',
172+ type=int, nargs=4, help='Check replication status',
173+ metavar=('lag_warn', 'lag_crit', 'failures_warn', 'failures_crit'))
174+ parser.add_argument('-m', '--md5', dest='check_md5', action='store_true',
175+ help='Compare server rings md5sum with local copy')
176+ args = parser.parse_args()
177+
178+ if not args.check_replication and not args.check_md5:
179+ print "You must use -r or -m switch"
180+ sys.exit(STATUS_UNKNOWN)
181+
182+ base_url = "http://{}:{}/recon/".format(args.host, args.port)
183+ results = []
184+ if args.check_replication:
185+ results.extend(check_replication(base_url, args.check_replication))
186+ if args.check_md5:
187+ results.extend(check_md5(base_url))
188+
189+ crits = ';'.join([i[1] for i in results if i[0] == STATUS_CRIT])
190+ warns = ';'.join([i[1] for i in results if i[0] == STATUS_WARN])
191+ unknowns = ';'.join([i[1] for i in results if i[0] == STATUS_UNKNOWN])
192+ if crits:
193+ print "CRITICAL: " + crits
194+ sys.exit(STATUS_CRIT)
195+ elif warns:
196+ print "WARNING: " + warns
197+ sys.exit(STATUS_WARN)
198+ elif unknowns:
199+ print "UNKNOWN: " + unknowns
200+ sys.exit(STATUS_UNKNOWN)
201+ else:
202+ print "OK"
203+ sys.exit(0)
204
205=== added directory 'files/sudo'
206=== added file 'files/sudo/swift-storage'
207--- files/sudo/swift-storage 1970-01-01 00:00:00 +0000
208+++ files/sudo/swift-storage 2014-08-19 15:54:23 +0000
209@@ -0,0 +1,1 @@
210+nagios ALL=(swift) NOPASSWD:/usr/bin/swift-init status *
211
212=== added directory 'hooks/charmhelpers/contrib/charmsupport'
213=== added file 'hooks/charmhelpers/contrib/charmsupport/__init__.py'
214=== added file 'hooks/charmhelpers/contrib/charmsupport/nrpe.py'
215--- hooks/charmhelpers/contrib/charmsupport/nrpe.py 1970-01-01 00:00:00 +0000
216+++ hooks/charmhelpers/contrib/charmsupport/nrpe.py 2014-08-19 15:54:23 +0000
217@@ -0,0 +1,219 @@
218+"""Compatibility with the nrpe-external-master charm"""
219+# Copyright 2012 Canonical Ltd.
220+#
221+# Authors:
222+# Matthew Wedgwood <matthew.wedgwood@canonical.com>
223+
224+import subprocess
225+import pwd
226+import grp
227+import os
228+import re
229+import shlex
230+import yaml
231+
232+from charmhelpers.core.hookenv import (
233+ config,
234+ local_unit,
235+ log,
236+ relation_ids,
237+ relation_set,
238+)
239+
240+from charmhelpers.core.host import service
241+
242+# This module adds compatibility with the nrpe-external-master and plain nrpe
243+# subordinate charms. To use it in your charm:
244+#
245+# 1. Update metadata.yaml
246+#
247+# provides:
248+# (...)
249+# nrpe-external-master:
250+# interface: nrpe-external-master
251+# scope: container
252+#
253+# and/or
254+#
255+# provides:
256+# (...)
257+# local-monitors:
258+# interface: local-monitors
259+# scope: container
260+
261+#
262+# 2. Add the following to config.yaml
263+#
264+# nagios_context:
265+# default: "juju"
266+# type: string
267+# description: |
268+# Used by the nrpe subordinate charms.
269+# A string that will be prepended to instance name to set the host name
270+# in nagios. So for instance the hostname would be something like:
271+# juju-myservice-0
272+# If you're running multiple environments with the same services in them
273+# this allows you to differentiate between them.
274+#
275+# 3. Add custom checks (Nagios plugins) to files/nrpe-external-master
276+#
277+# 4. Update your hooks.py with something like this:
278+#
279+# from charmsupport.nrpe import NRPE
280+# (...)
281+# def update_nrpe_config():
282+# nrpe_compat = NRPE()
283+# nrpe_compat.add_check(
284+# shortname = "myservice",
285+# description = "Check MyService",
286+# check_cmd = "check_http -w 2 -c 10 http://localhost"
287+# )
288+# nrpe_compat.add_check(
289+# "myservice_other",
290+# "Check for widget failures",
291+# check_cmd = "/srv/myapp/scripts/widget_check"
292+# )
293+# nrpe_compat.write()
294+#
295+# def config_changed():
296+# (...)
297+# update_nrpe_config()
298+#
299+# def nrpe_external_master_relation_changed():
300+# update_nrpe_config()
301+#
302+# def local_monitors_relation_changed():
303+# update_nrpe_config()
304+#
305+# 5. ln -s hooks.py nrpe-external-master-relation-changed
306+# ln -s hooks.py local-monitors-relation-changed
307+
308+
309+class CheckException(Exception):
310+ pass
311+
312+
313+class Check(object):
314+ shortname_re = '[A-Za-z0-9-_]+$'
315+ service_template = ("""
316+#---------------------------------------------------
317+# This file is Juju managed
318+#---------------------------------------------------
319+define service {{
320+ use active-service
321+ host_name {nagios_hostname}
322+ service_description {nagios_hostname}[{shortname}] """
323+ """{description}
324+ check_command check_nrpe!{command}
325+ servicegroups {nagios_servicegroup}
326+}}
327+""")
328+
329+ def __init__(self, shortname, description, check_cmd):
330+ super(Check, self).__init__()
331+ # XXX: could be better to calculate this from the service name
332+ if not re.match(self.shortname_re, shortname):
333+ raise CheckException("shortname must match {}".format(
334+ Check.shortname_re))
335+ self.shortname = shortname
336+ self.command = "check_{}".format(shortname)
337+ # Note: a set of invalid characters is defined by the
338+ # Nagios server config
339+ # The default is: illegal_object_name_chars=`~!$%^&*"|'<>?,()=
340+ self.description = description
341+ self.check_cmd = self._locate_cmd(check_cmd)
342+
343+ def _locate_cmd(self, check_cmd):
344+ search_path = (
345+ '/usr/lib/nagios/plugins',
346+ '/usr/local/lib/nagios/plugins',
347+ )
348+ parts = shlex.split(check_cmd)
349+ for path in search_path:
350+ if os.path.exists(os.path.join(path, parts[0])):
351+ command = os.path.join(path, parts[0])
352+ if len(parts) > 1:
353+ command += " " + " ".join(parts[1:])
354+ return command
355+ log('Check command not found: {}'.format(parts[0]))
356+ return ''
357+
358+ def write(self, nagios_context, hostname):
359+ nrpe_check_file = '/etc/nagios/nrpe.d/{}.cfg'.format(
360+ self.command)
361+ with open(nrpe_check_file, 'w') as nrpe_check_config:
362+ nrpe_check_config.write("# check {}\n".format(self.shortname))
363+ nrpe_check_config.write("command[{}]={}\n".format(
364+ self.command, self.check_cmd))
365+
366+ if not os.path.exists(NRPE.nagios_exportdir):
367+ log('Not writing service config as {} is not accessible'.format(
368+ NRPE.nagios_exportdir))
369+ else:
370+ self.write_service_config(nagios_context, hostname)
371+
372+ def write_service_config(self, nagios_context, hostname):
373+ for f in os.listdir(NRPE.nagios_exportdir):
374+ if re.search('.*{}.cfg'.format(self.command), f):
375+ os.remove(os.path.join(NRPE.nagios_exportdir, f))
376+
377+ templ_vars = {
378+ 'nagios_hostname': hostname,
379+ 'nagios_servicegroup': nagios_context,
380+ 'description': self.description,
381+ 'shortname': self.shortname,
382+ 'command': self.command,
383+ }
384+ nrpe_service_text = Check.service_template.format(**templ_vars)
385+ nrpe_service_file = '{}/service__{}_{}.cfg'.format(
386+ NRPE.nagios_exportdir, hostname, self.command)
387+ with open(nrpe_service_file, 'w') as nrpe_service_config:
388+ nrpe_service_config.write(str(nrpe_service_text))
389+
390+ def run(self):
391+ subprocess.call(self.check_cmd)
392+
393+
394+class NRPE(object):
395+ nagios_logdir = '/var/log/nagios'
396+ nagios_exportdir = '/var/lib/nagios/export'
397+ nrpe_confdir = '/etc/nagios/nrpe.d'
398+
399+ def __init__(self, hostname=None):
400+ super(NRPE, self).__init__()
401+ self.config = config()
402+ self.nagios_context = self.config['nagios_context']
403+ self.unit_name = local_unit().replace('/', '-')
404+ if hostname:
405+ self.hostname = hostname
406+ else:
407+ self.hostname = "{}-{}".format(self.nagios_context, self.unit_name)
408+ self.checks = []
409+
410+ def add_check(self, *args, **kwargs):
411+ self.checks.append(Check(*args, **kwargs))
412+
413+ def write(self):
414+ try:
415+ nagios_uid = pwd.getpwnam('nagios').pw_uid
416+ nagios_gid = grp.getgrnam('nagios').gr_gid
417+ except:
418+ log("Nagios user not set up, nrpe checks not updated")
419+ return
420+
421+ if not os.path.exists(NRPE.nagios_logdir):
422+ os.mkdir(NRPE.nagios_logdir)
423+ os.chown(NRPE.nagios_logdir, nagios_uid, nagios_gid)
424+
425+ nrpe_monitors = {}
426+ monitors = {"monitors": {"remote": {"nrpe": nrpe_monitors}}}
427+ for nrpecheck in self.checks:
428+ nrpecheck.write(self.nagios_context, self.hostname)
429+ nrpe_monitors[nrpecheck.shortname] = {
430+ "command": nrpecheck.command,
431+ }
432+
433+ service('restart', 'nagios-nrpe-server')
434+
435+ for rid in relation_ids("local-monitors"):
436+ relation_set(relation_id=rid, monitors=yaml.dump(monitors))
437
438=== added file 'hooks/charmhelpers/contrib/charmsupport/volumes.py'
439--- hooks/charmhelpers/contrib/charmsupport/volumes.py 1970-01-01 00:00:00 +0000
440+++ hooks/charmhelpers/contrib/charmsupport/volumes.py 2014-08-19 15:54:23 +0000
441@@ -0,0 +1,159 @@
442+'''
443+Functions for managing volumes in juju units. One volume is supported per unit.
444+Subordinates may have their own storage, provided it is on its own partition.
445+
446+Configuration stanzas::
447+
448+ volume-ephemeral:
449+ type: boolean
450+ default: true
451+ description: >
452+ If false, a volume is mounted as sepecified in "volume-map"
453+ If true, ephemeral storage will be used, meaning that log data
454+ will only exist as long as the machine. YOU HAVE BEEN WARNED.
455+ volume-map:
456+ type: string
457+ default: {}
458+ description: >
459+ YAML map of units to device names, e.g:
460+ "{ rsyslog/0: /dev/vdb, rsyslog/1: /dev/vdb }"
461+ Service units will raise a configure-error if volume-ephemeral
462+ is 'true' and no volume-map value is set. Use 'juju set' to set a
463+ value and 'juju resolved' to complete configuration.
464+
465+Usage::
466+
467+ from charmsupport.volumes import configure_volume, VolumeConfigurationError
468+ from charmsupport.hookenv import log, ERROR
469+ def post_mount_hook():
470+ stop_service('myservice')
471+ def post_mount_hook():
472+ start_service('myservice')
473+
474+ if __name__ == '__main__':
475+ try:
476+ configure_volume(before_change=pre_mount_hook,
477+ after_change=post_mount_hook)
478+ except VolumeConfigurationError:
479+ log('Storage could not be configured', ERROR)
480+
481+'''
482+
483+# XXX: Known limitations
484+# - fstab is neither consulted nor updated
485+
486+import os
487+from charmhelpers.core import hookenv
488+from charmhelpers.core import host
489+import yaml
490+
491+
492+MOUNT_BASE = '/srv/juju/volumes'
493+
494+
495+class VolumeConfigurationError(Exception):
496+ '''Volume configuration data is missing or invalid'''
497+ pass
498+
499+
500+def get_config():
501+ '''Gather and sanity-check volume configuration data'''
502+ volume_config = {}
503+ config = hookenv.config()
504+
505+ errors = False
506+
507+ if config.get('volume-ephemeral') in (True, 'True', 'true', 'Yes', 'yes'):
508+ volume_config['ephemeral'] = True
509+ else:
510+ volume_config['ephemeral'] = False
511+
512+ try:
513+ volume_map = yaml.safe_load(config.get('volume-map', '{}'))
514+ except yaml.YAMLError as e:
515+ hookenv.log("Error parsing YAML volume-map: {}".format(e),
516+ hookenv.ERROR)
517+ errors = True
518+ if volume_map is None:
519+ # probably an empty string
520+ volume_map = {}
521+ elif not isinstance(volume_map, dict):
522+ hookenv.log("Volume-map should be a dictionary, not {}".format(
523+ type(volume_map)))
524+ errors = True
525+
526+ volume_config['device'] = volume_map.get(os.environ['JUJU_UNIT_NAME'])
527+ if volume_config['device'] and volume_config['ephemeral']:
528+ # asked for ephemeral storage but also defined a volume ID
529+ hookenv.log('A volume is defined for this unit, but ephemeral '
530+ 'storage was requested', hookenv.ERROR)
531+ errors = True
532+ elif not volume_config['device'] and not volume_config['ephemeral']:
533+ # asked for permanent storage but did not define volume ID
534+ hookenv.log('Ephemeral storage was requested, but there is no volume '
535+ 'defined for this unit.', hookenv.ERROR)
536+ errors = True
537+
538+ unit_mount_name = hookenv.local_unit().replace('/', '-')
539+ volume_config['mountpoint'] = os.path.join(MOUNT_BASE, unit_mount_name)
540+
541+ if errors:
542+ return None
543+ return volume_config
544+
545+
546+def mount_volume(config):
547+ if os.path.exists(config['mountpoint']):
548+ if not os.path.isdir(config['mountpoint']):
549+ hookenv.log('Not a directory: {}'.format(config['mountpoint']))
550+ raise VolumeConfigurationError()
551+ else:
552+ host.mkdir(config['mountpoint'])
553+ if os.path.ismount(config['mountpoint']):
554+ unmount_volume(config)
555+ if not host.mount(config['device'], config['mountpoint'], persist=True):
556+ raise VolumeConfigurationError()
557+
558+
559+def unmount_volume(config):
560+ if os.path.ismount(config['mountpoint']):
561+ if not host.umount(config['mountpoint'], persist=True):
562+ raise VolumeConfigurationError()
563+
564+
565+def managed_mounts():
566+ '''List of all mounted managed volumes'''
567+ return filter(lambda mount: mount[0].startswith(MOUNT_BASE), host.mounts())
568+
569+
570+def configure_volume(before_change=lambda: None, after_change=lambda: None):
571+ '''Set up storage (or don't) according to the charm's volume configuration.
572+ Returns the mount point or "ephemeral". before_change and after_change
573+ are optional functions to be called if the volume configuration changes.
574+ '''
575+
576+ config = get_config()
577+ if not config:
578+ hookenv.log('Failed to read volume configuration', hookenv.CRITICAL)
579+ raise VolumeConfigurationError()
580+
581+ if config['ephemeral']:
582+ if os.path.ismount(config['mountpoint']):
583+ before_change()
584+ unmount_volume(config)
585+ after_change()
586+ return 'ephemeral'
587+ else:
588+ # persistent storage
589+ if os.path.ismount(config['mountpoint']):
590+ mounts = dict(managed_mounts())
591+ if mounts.get(config['mountpoint']) != config['device']:
592+ before_change()
593+ unmount_volume(config)
594+ mount_volume(config)
595+ after_change()
596+ else:
597+ before_change()
598+ mount_volume(config)
599+ after_change()
600+ return config['mountpoint']
601
602=== added symlink 'hooks/nrpe-external-master-relation-changed'
603=== target is u'swift_storage_hooks.py'
604=== added symlink 'hooks/nrpe-external-master-relation-joined'
605=== target is u'swift_storage_hooks.py'
606=== modified file 'hooks/swift_storage_hooks.py'
607--- hooks/swift_storage_hooks.py 2013-09-27 16:33:06 +0000
608+++ hooks/swift_storage_hooks.py 2014-08-19 15:54:23 +0000
609@@ -6,6 +6,7 @@
610 from swift_storage_utils import (
611 PACKAGES,
612 RESTART_MAP,
613+ SWIFT_SVCS,
614 determine_block_devices,
615 do_openstack_upgrade,
616 ensure_swift_directories,
617@@ -13,6 +14,7 @@
618 register_configs,
619 save_script_rc,
620 setup_storage,
621+ concat_rsync_fragments,
622 )
623
624 from charmhelpers.core.hookenv import (
625@@ -21,10 +23,11 @@
626 log,
627 relation_get,
628 relation_set,
629+ relations_of_type,
630 )
631
632 from charmhelpers.fetch import apt_install, apt_update
633-from charmhelpers.core.host import restart_on_change
634+from charmhelpers.core.host import restart_on_change, rsync
635 from charmhelpers.payload.execd import execd_preinstall
636
637 from charmhelpers.contrib.openstack.utils import (
638@@ -32,9 +35,12 @@
639 openstack_upgrade_available,
640 )
641
642+from charmhelpers.contrib.charmsupport.nrpe import NRPE
643+
644 hooks = Hooks()
645 CONFIGS = register_configs()
646-
647+NAGIOS_PLUGINS = '/usr/local/lib/nagios/plugins'
648+SUDOERS_D = '/etc/sudoers.d'
649
650 @hooks.hook()
651 def install():
652@@ -52,8 +58,24 @@
653 if openstack_upgrade_available('swift'):
654 do_openstack_upgrade(configs=CONFIGS)
655 CONFIGS.write_all()
656+
657+ # If basenode is not installed and managing rsyncd.conf, replicate
658+ # its core functionality. Otherwise concat files
659+ if not os.path.exists('/etc/rsyncd.d/001-basenode'):
660+ with open('templates/rsyncd.conf') as _in:
661+ rsync_header = _in.read()
662+ with open('/etc/rsyncd.d/050-swift-storage') as _in:
663+ rsync_fragment = _in.read()
664+ with open('/etc/rsyncd.conf', 'w') as out:
665+ out.write(rsync_header + rsync_fragment)
666+ else:
667+ concat_rsync_fragments()
668+
669 save_script_rc()
670
671+ if relations_of_type('nrpe-external-master'):
672+ nrpe_relation()
673+
674
675 @hooks.hook()
676 def swift_storage_relation_joined():
677@@ -79,6 +101,40 @@
678 CONFIGS.write('/etc/swift/swift.conf')
679 fetch_swift_rings(rings_url)
680
681+@hooks.hook('nrpe-external-master-relation-joined')
682+@hooks.hook('nrpe-external-master-relation-changed')
683+def nrpe_relation():
684+ log('Refreshing nrpe checks')
685+ rsync(os.path.join(os.getenv('CHARM_DIR'), 'files', 'nagios',
686+ 'check_swift_storage.py'),
687+ os.path.join(NAGIOS_PLUGINS, 'check_swift_storage.py'))
688+ rsync(os.path.join(os.getenv('CHARM_DIR'), 'files', 'nagios',
689+ 'check_swift_service'),
690+ os.path.join(NAGIOS_PLUGINS, 'check_swift_service'))
691+ rsync(os.path.join(os.getenv('CHARM_DIR'), 'files', 'sudo',
692+ 'swift-storage'),
693+ os.path.join(SUDOERS_D, 'swift-storage'))
694+ # Find out if nrpe set nagios_hostname
695+ hostname = None
696+ for rel in relations_of_type('nrpe-external-master'):
697+ if 'nagios_hostname' in rel:
698+ hostname = rel['nagios_hostname']
699+ break
700+ nrpe = NRPE(hostname=hostname)
701+ nrpe.add_check(
702+ shortname='swift_storage',
703+ description='Check swift storage ring hashes and replication',
704+ check_cmd='check_swift_storage.py {}'.format(
705+ config('nagios-check-params'))
706+ )
707+ # check services are running
708+ for service in SWIFT_SVCS:
709+ nrpe.add_check(
710+ shortname=service,
711+ description='swift-storage %s service' % service,
712+ check_cmd = 'check_swift_service %s' % service,
713+ )
714+ nrpe.write()
715
716 def main():
717 try:
718
719=== modified file 'hooks/swift_storage_utils.py'
720--- hooks/swift_storage_utils.py 2014-08-11 09:14:47 +0000
721+++ hooks/swift_storage_utils.py 2014-08-19 15:54:23 +0000
722@@ -70,13 +70,28 @@
723 ]
724
725 RESTART_MAP = {
726- '/etc/rsyncd.conf': ['rsync'],
727+ '/etc/rsyncd.d/050-swift-storage': ['rsync'],
728 '/etc/swift/account-server.conf': ACCOUNT_SVCS,
729 '/etc/swift/container-server.conf': CONTAINER_SVCS,
730 '/etc/swift/object-server.conf': OBJECT_SVCS,
731 '/etc/swift/swift.conf': ACCOUNT_SVCS + CONTAINER_SVCS + OBJECT_SVCS
732 }
733
734+SWIFT_SVCS = [
735+ 'account-auditor',
736+ 'account-reaper',
737+ 'account-replicator',
738+ 'account-server',
739+ 'container-auditor',
740+ 'container-replicator',
741+ 'container-server',
742+ 'container-sync',
743+ 'container-updater',
744+ 'object-auditor',
745+ 'object-replicator',
746+ 'object-server',
747+ 'object-updater',
748+ ]
749
750 def ensure_swift_directories():
751 '''
752@@ -90,6 +105,11 @@
753 ]
754 [mkdir(d, owner='swift', group='swift') for d in dirs
755 if not os.path.isdir(d)]
756+ root_dirs = [
757+ '/etc/rsyncd.d',
758+ ]
759+ [mkdir(d, owner='root', group='root') for d in root_dirs
760+ if not os.path.isdir(d)]
761
762
763 def register_configs():
764@@ -98,7 +118,7 @@
765 openstack_release=release)
766 configs.register('/etc/swift/swift.conf',
767 [SwiftStorageContext()])
768- configs.register('/etc/rsyncd.conf',
769+ configs.register('/etc/rsyncd.d/050-swift-storage',
770 [RsyncContext()])
771 for server in ['account', 'object', 'container']:
772 configs.register('/etc/swift/%s-server.conf' % server,
773@@ -216,3 +236,14 @@
774 'OPENSTACK_URL_%s' % svc: url,
775 })
776 _save_script_rc(**env_vars)
777+
778+def concat_rsync_fragments():
779+ log('Concatenating rsyncd.d fragments')
780+ rsyncd_dir = '/etc/rsyncd.d'
781+ rsyncd_conf = ""
782+ for filename in sorted(os.listdir(rsyncd_dir)):
783+ with open(os.path.join(rsyncd_dir, filename), 'r') as fragment:
784+ rsyncd_conf += fragment.read()
785+ with open('/etc/rsyncd.conf', 'w') as f:
786+ f.write(rsyncd_conf)
787+
788
789=== modified file 'metadata.yaml'
790--- metadata.yaml 2013-07-11 20:45:19 +0000
791+++ metadata.yaml 2014-08-19 15:54:23 +0000
792@@ -8,3 +8,6 @@
793 provides:
794 swift-storage:
795 interface: swift
796+ nrpe-external-master:
797+ interface: nrpe-external-master
798+ scope: container
799
800=== modified file 'revision'
801--- revision 2013-07-19 21:13:59 +0000
802+++ revision 2014-08-19 15:54:23 +0000
803@@ -1,1 +1,1 @@
804-90
805+101
806
807=== removed directory 'scripts'
808=== added file 'templates/050-swift-storage'
809--- templates/050-swift-storage 1970-01-01 00:00:00 +0000
810+++ templates/050-swift-storage 2014-08-19 15:54:23 +0000
811@@ -0,0 +1,24 @@
812+[account]
813+uid = swift
814+gid = swift
815+max connections = 2
816+path = /srv/node/
817+read only = false
818+lock file = /var/lock/account.lock
819+
820+[container]
821+uid = swift
822+gid = swift
823+max connections = 2
824+path = /srv/node/
825+read only = false
826+lock file = /var/lock/container.lock
827+
828+[object]
829+uid = swift
830+gid = swift
831+max connections = 2
832+path = /srv/node/
833+read only = false
834+lock file = /var/lock/object.lock
835+
836
837=== modified file 'templates/rsyncd.conf'
838--- templates/rsyncd.conf 2013-07-19 19:52:45 +0000
839+++ templates/rsyncd.conf 2014-08-19 15:54:23 +0000
840@@ -1,23 +1,5 @@
841-uid = swift
842-gid = swift
843-log file = /var/log/rsyncd.log
844-pid file = /var/run/rsyncd.pid
845-address = {{ local_ip }}
846-
847-[account]
848-max connections = 2
849-path = /srv/node/
850-read only = false
851-lock file = /var/lock/account.lock
852-
853-[container]
854-max connections = 2
855-path = /srv/node/
856-read only = false
857-lock file = /var/lock/container.lock
858-
859-[object]
860-max connections = 2
861-path = /srv/node/
862-read only = false
863-lock file = /var/lock/object.lock
864+uid = nobody
865+gid = nogroup
866+syslog facility = daemon
867+socket options = SO_KEEPALIVE
868+
869
870=== modified file 'unit_tests/test_swift_storage_relations.py'
871--- unit_tests/test_swift_storage_relations.py 2014-06-19 08:40:58 +0000
872+++ unit_tests/test_swift_storage_relations.py 2014-08-19 15:54:23 +0000
873@@ -1,6 +1,6 @@
874 from mock import patch, MagicMock
875
876-from test_utils import CharmTestCase
877+from test_utils import CharmTestCase, patch_open
878
879 import swift_storage_utils as utils
880
881@@ -21,6 +21,7 @@
882 'log',
883 'relation_set',
884 'relation_get',
885+ 'relations_of_type',
886 # charmhelpers.core.host
887 'apt_update',
888 'apt_install',
889@@ -61,13 +62,19 @@
890
891 def test_config_changed_no_upgrade_available(self):
892 self.openstack_upgrade_available.return_value = False
893- hooks.config_changed()
894+ self.relations_of_type.return_value = False
895+ with patch_open() as (_open, _file):
896+ _file.read.return_value = "foo"
897+ hooks.config_changed()
898 self.assertFalse(self.do_openstack_upgrade.called)
899 self.assertTrue(self.CONFIGS.write_all.called)
900
901 def test_config_changed_upgrade_available(self):
902 self.openstack_upgrade_available.return_value = True
903- hooks.config_changed()
904+ self.relations_of_type.return_value = False
905+ with patch_open() as (_open, _file):
906+ _file.read.return_value = "foo"
907+ hooks.config_changed()
908 self.assertTrue(self.do_openstack_upgrade.called)
909 self.assertTrue(self.CONFIGS.write_all.called)
910
911
912=== modified file 'unit_tests/test_swift_storage_utils.py'
913--- unit_tests/test_swift_storage_utils.py 2014-06-19 08:40:58 +0000
914+++ unit_tests/test_swift_storage_utils.py 2014-08-19 15:54:23 +0000
915@@ -75,6 +75,7 @@
916 ex_dirs = [
917 call('/etc/swift', owner='swift', group='swift'),
918 call('/var/cache/swift', owner='swift', group='swift'),
919+ call('/etc/rsyncd.d', owner='root', group='root'),
920 call('/srv/node', owner='swift', group='swift')
921 ]
922 self.assertEquals(ex_dirs, self.mkdir.call_args_list)
923@@ -211,7 +212,7 @@
924 openstack_release='grizzly')
925 ex = [
926 call('/etc/swift/swift.conf', ['swift_server_context']),
927- call('/etc/rsyncd.conf', ['rsync_context']),
928+ call('/etc/rsyncd.d/050-swift-storage', ['rsync_context']),
929 call('/etc/swift/account-server.conf', ['swift_context']),
930 call('/etc/swift/object-server.conf', ['swift_context']),
931 call('/etc/swift/container-server.conf', ['swift_context'])

Subscribers

People subscribed via source and target branches