Merge lp:~paulgear/charm-helpers/nrpe-service-immediate-check into lp:charm-helpers

[Paul Gear]

Add an option for immediate execution of nrpe service checks. This is necessary because (for sysvinit-based services) the cron file installed only runs the check every 5 minutes.

If a nagios service check is created but the check does not fire before the end of a CI run which does nagios checks at the end, the nagios check will return an error and the CI run will fail.

This change works around that by allowing a charm to specify that a check should be run immediately, to ensure the check file is created.

[Stuart Bishop (stub)]

Minor inline comments.

I think the default should be immediate_check=True. Charms need to explicitly call add_init_service_checks(), and any charm doing that for a bogus service that hangs on the status check is already broken. It is better to discover that now rather than later when there are a zillion hung /usr/local/lib/nagios/plugins/check_exit_status.pl processes. Or just add timeout(1) to the command line.

review: Abstain

[lp:~paulgear/charm-helpers/nrpe-service-immediate-check]

613. By Paul Gear on 2017-03-02

Implement Stuart's recommendations

614. By Paul Gear on 2017-03-02

Merge devel trunk

[Paul Gear (paulgear)]

Updated with Stuart's recommendations.

[Stuart Bishop (stub)]

======================================================================
ERROR: tests.contrib.charmsupport.test_nrpe.NRPEMiscTestCase.test_add_init_service_checks
----------------------------------------------------------------------
_StringException: Traceback (most recent call last):
  File "/home/stub/charms/charm-helpers/trunk/tests/contrib/charmsupport/test_nrpe.py", line 315, in test_add_init_service_checks
    nrpe.add_init_service_checks(bill, services, 'testunit')
  File "/home/stub/charms/charm-helpers/trunk/charmhelpers/contrib/charmsupport/nrpe.py", line 394, in add_init_service_checks
    os.chmod(checkpath, 0o644)
OSError: [Errno 2] No such file or directory: '/var/lib/nagios/service-check-haproxy.txt'

[Stuart Bishop (stub)]

review: Approve

[Stuart Bishop (stub)]

I've fixed the test

[Ryan Beisner (1chb1n)]

After syncing this into the Glance charm, we see failure:

2017-03-03 22:22:19 INFO worker.uniter.jujuc server.go:172 running hook tool "juju-log" ["Check command not found: check_status_file.py"]
2017-03-03 22:22:19 DEBUG worker.uniter.jujuc server.go:173 hook context id "glance/0-config-changed-2954441461654224842"; dir "/var/lib/juju/agents/unit-glance-0/charm"
2017-03-03 22:22:19 INFO juju-log Check command not found: check_status_file.py
2017-03-03 22:22:19 INFO config-changed Traceback (most recent call last):
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/config-changed", line 632, in <module>
2017-03-03 22:22:19 INFO config-changed hooks.execute(sys.argv)
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/hookenv.py", line 731, in execute
2017-03-03 22:22:19 INFO config-changed self._hooks[hook_name]()
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1864, in wrapped_f
2017-03-03 22:22:19 INFO config-changed restart_functions)
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/core/host.py", line 655, in restart_on_change_helper
2017-03-03 22:22:19 INFO config-changed r = lambda_f()
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 1863, in <lambda>
2017-03-03 22:22:19 INFO config-changed (lambda: f(*args, **kwargs)), restart_map, stopstart,
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/contrib/hardening/harden.py", line 79, in _harden_inner2
2017-03-03 22:22:19 INFO config-changed return f(*args, **kwargs)
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/config-changed", line 405, in config_changed
2017-03-03 22:22:19 INFO config-changed update_nrpe_config()
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/config-changed", line 593, in update_nrpe_config
2017-03-03 22:22:19 INFO config-changed nrpe.add_init_service_checks(nrpe_setup, services(), current_unit)
2017-03-03 22:22:19 INFO config-changed File "/var/lib/juju/agents/unit-glance-0/charm/hooks/charmhelpers/contrib/charmsupport/nrpe.py", line 387, in add_init_service_checks
2017-03-03 22:22:19 INFO config-changed f = open(checkpath, 'w')
2017-03-03 22:22:19 INFO config-changed IOError: [Errno 2] No such file or directory: '/var/lib/nagios/service-check-haproxy.txt'

[Ryan Beisner (1chb1n)]

Raise https://bugs.launchpad.net/charm-helpers/+bug/1670223

=== modified file 'charmhelpers/contrib/charmsupport/nrpe.py'
--- charmhelpers/contrib/charmsupport/nrpe.py	2016-09-06 06:25:17 +0000
+++ charmhelpers/contrib/charmsupport/nrpe.py	2017-03-02 07:53:57 +0000
@@ -338,13 +338,14 @@
     return unit
 
 
-def add_init_service_checks(nrpe, services, unit_name):
+def add_init_service_checks(nrpe, services, unit_name, immediate_check=True):
     """
     Add checks for each service in list
 
     :param NRPE nrpe: NRPE object to add check to
     :param list services: List of services to check
     :param str unit_name: Unit name to use in check description
+    :param bool immediate_check: For sysv init, run the service check immediately
     """
     for svc in services:
         # Don't add a check for these services from neutron-gateway
@@ -368,21 +369,29 @@
             )
         elif os.path.exists(sysv_init):
             cronpath = '/etc/cron.d/nagios-service-check-%s' % svc
-            cron_file = ('*/5 * * * * root '
-                         '/usr/local/lib/nagios/plugins/check_exit_status.pl '
-                         '-s /etc/init.d/%s status > '
-                         '/var/lib/nagios/service-check-%s.txt\n' % (svc,
-                                                                     svc)
-                         )
+            checkpath = '/var/lib/nagios/service-check-%s.txt' % svc
+            croncmd = (
+                '/usr/local/lib/nagios/plugins/check_exit_status.pl '
+                '-s /etc/init.d/%s status' % svc
+            )
+            cron_file = '*/5 * * * * root %s > %s\n' % (croncmd, checkpath)
             f = open(cronpath, 'w')
             f.write(cron_file)
             f.close()
             nrpe.add_check(
                 shortname=svc,
-                description='process check {%s}' % unit_name,
-                check_cmd='check_status_file.py -f '
-                          '/var/lib/nagios/service-check-%s.txt' % svc,
+                description='service check {%s}' % unit_name,
+                check_cmd='check_status_file.py -f %s' % checkpath,
             )
+            if immediate_check:
+                f = open(checkpath, 'w')
+                subprocess.call(
+                    croncmd.split(),
+                    stdout=f,
+                    stderr=subprocess.STDOUT
+                )
+                f.close()
+                os.chmod(checkpath, 0o644)
 
 
 def copy_nrpe_checks():

=== modified file 'tests/contrib/charmsupport/test_nrpe.py'
--- tests/contrib/charmsupport/test_nrpe.py	2016-09-06 06:25:17 +0000
+++ tests/contrib/charmsupport/test_nrpe.py	2017-03-02 07:53:57 +0000
@@ -300,6 +300,8 @@
                      '/usr/lib/nagios/plugins/check_upstart_job',
                      '/etc/init.d/haproxy',
                      '/usr/lib/nagios/plugins/check_status_file.py',
+                     '/etc/cron.d/nagios-service-check-haproxy',
+                     '/var/lib/nagios/service-check-haproxy.txt',
                      '/usr/lib/nagios/plugins/check_systemd.py'
                      ]
             return init_file in files