Merge lp:~sinzui/charms/precise/juju-gui/nagios into lp:charms/juju-gui

[Curtis Hovey]

Add nagios support.

RULES

    Pre-implementation: sidnei, moon127, wedgwood
    * We want to support the nagios subordinate charm to monitor production.
    * Webops prefer to use the nrpe module from charmsupport.
      * The project merged with charmhelpers (not related to the charm-helpers
        ppa that all PPA configured juju deploys get).
      * Webops write a custom script for each charm they deploy to embed the
        library in the charm; they alway deploy a forked charm. The charm
        runs with a modified python path to find that included libs.
      * Webops require the charm to ensure errors installing dependencies are
        are ignored since they have ensured the libs are already in the charm.
        * Or have they? Every change to the charm requires review of a webop
          to ensure the building of a fat charm really works.
    * I decided to include just the nrpe module and its dependencies in
      the charm's tree.
      * No action is needed to build out a fat charm.
      * No future coordination with webops to ensure the charm really
        installs its deps.
      * I chose an older version of nrpe because it has fewer deps than
        the current version. I chose the same version used by charmworld.
        I'd like to keep these two charms synced to simplify the parts used
        to deploy charmworld.com

QA

    DEPLOY JUJU-GUI AND NAGIOS
    juju bootstrap
    juju deploy --repo=~/Work/juju-charms/ local:precise/juju-gui
    juju deploy nagios
    juju add-relation nagios juju-gui
    juju expose nagios
    Verify you can see juju-gui in nagios with a single ssh check

    DEPLOY NRPE-EXTERNAL-MASTER
    juju deploy nrpe-external-master
    juju set nrpe-external-master nagios_master=http://10.0.3.45/<nagio-public-ip>
    juju add-relation nrpe-external-master:nrpe-external-master juju-gui:nrpe-external-master

    Verify you can see the nagios external cfg files in juju-gui's file system.
    ls /var/lib/nagios/export/
    Verify the unit config "host__juju-juju-gui-0.cfg" has the the nagio ip address.

IMPLEMENTATION

    * I added just the parts of the nrpe library needed.
      scripts/charmsupport/__init__.py
      scripts/charmsupport/hookenv.py
      scripts/charmsupport/nrpe.py

    * I added a hook that calls the update-nrpe.py script that in turn
      registers the check-front-page.sh script. nrpe requires the
      check scripts to be in files/nrpe-external-master/.
      hooks/nrpe-external-master-relation-changed
      scripts/update-nrpe.py
      files/nrpe-external-master/check-front-page.sh

    * I updated the metadata and config to enable the subordinate charm.
      config.yaml
      metadata.yaml
      revision

[lp:~sinzui/charms/precise/juju-gui/nagios]

78. By Curtis Hovey on 2013-07-31

Merged tip and resolved conflicts.

79. By Curtis Hovey on 2013-08-02

Merged tip.

[Benji York (benji)]

Thanks for wiring this up. Monitoring is essential.

Given that this MP doesn't have Reitveld links, I assume you didn't use
lbox to propose it. Despite it's many annoyances, lbox is the
prescribed way of proposing and landing GUI branches. The most
important thing it does is to run "make check" before proposing a
branch. Please manually run that command before landing. Thanks.

> I decided to include just the nrpe module and its dependencies in the
> charm's tree.

It's a shame that our policies push us toward doing things like copy
chunks of code from outdated dependencies, but I can't proffer anything
better.

In check-front-page.sh (line 31 of the diff) shouldn't it be "sites-enabled"
instead of "sites-available"?

Also, I suggest using "https://127.0.0.1:443/juju-ui/version.js" for ADDRESS
and "jujuGuiVersionInfo" for LIFE_SIGN. Being code, that is less likely
to change because of a UI/UX redesign and we can write a test for it
that ensures if it does change a test named "testMonitoringEndpoint"
will fail. Oh, and we need that test in this branch too.

Once the above are addressed, this will be ready to land.

review: Approve

[lp:~sinzui/charms/precise/juju-gui/nagios]

80. By Curtis Hovey on 2013-08-06

change script to check-app-access.sh because it now checks that the
GUI can be downloaded.

81. By Curtis Hovey on 2013-08-08

Added test to correlate check-app-access.sh to the path it checks.

[Gary Poster (gary)]

Hey Curtis. Thank you! Is it reasonable/easy to merge these revisions into the ~juju-gui charm branch as well, or are there conflicts?

=== modified file 'config.yaml'
--- config.yaml	2013-07-15 20:40:15 +0000
+++ config.yaml	2013-08-08 17:05:15 +0000
@@ -159,7 +159,7 @@
         canvas. This is also known as browse mode.
       - 'minimized': the charmbrowser will be minimized by default, and hidden.
     type: string
-    default: sidebar 
+    default: sidebar
   show-get-juju-button:
     description: |
       There are deployment modes for Juju GUI which are not intended as regular
@@ -167,3 +167,13 @@
       link to juju.ubuntu.com
     type: boolean
     default: false
+  nagios_context:
+    default: "juju"
+    type: string
+    description: |
+      Used by the nrpe-external-master subordinate charm.
+      A string that will be prepended to instance name to set the host name
+      in nagios. So for instance the hostname would be something like:
+          juju-myservice-0
+      If you're running multiple environments with the same services in them
+      this allows you to differentiate between them.
 
=== added directory 'files'
=== added directory 'files/nrpe-external-master'
=== added file 'files/nrpe-external-master/check-app-access.sh'
--- files/nrpe-external-master/check-app-access.sh	1970-01-01 00:00:00 +0000
+++ files/nrpe-external-master/check-app-access.sh	2013-08-08 17:05:15 +0000
@@ -0,0 +1,18 @@
+#!/bin/bash
+SITE_CONF='/etc/apache2/sites-enabled/juju-gui'
+ADDRESS='https://127.0.0.1:443/juju-ui/version.js'
+LIFE_SIGN='jujuGuiVersionInfo'
+
+if [[ ! -f $SITE_CONF ]]; then
+    echo Apache is not configured serve juju-gui.
+    exit 2
+fi
+
+match=$(curl -k $ADDRESS | grep "$LIFE_SIGN")
+
+if [[ -n "$match" ]]; then
+    exit 0
+else
+    echo juju-gui did not return content indicating it was loading.
+    exit 2
+fi
 
=== added symlink 'hooks/nrpe-external-master-relation-changed'
=== target is u'../scripts/update-nrpe.py'
=== modified file 'metadata.yaml'
--- metadata.yaml	2013-06-11 14:13:45 +0000
+++ metadata.yaml	2013-08-08 17:05:15 +0000
@@ -22,3 +22,6 @@
 provides:
   web:
     interface: http
+  nrpe-external-master:
+    interface: nrpe-external-master
+    scope: container
 
=== modified file 'revision'
--- revision	2013-08-02 14:19:45 +0000
+++ revision	2013-08-08 17:05:15 +0000
@@ -1,1 +1,1 @@
-65
+66
 
=== added directory 'scripts'
=== added directory 'scripts/charmsupport'
=== added file 'scripts/charmsupport/__init__.py'
=== added file 'scripts/charmsupport/hookenv.py'
--- scripts/charmsupport/hookenv.py	1970-01-01 00:00:00 +0000
+++ scripts/charmsupport/hookenv.py	2013-08-08 17:05:15 +0000
@@ -0,0 +1,150 @@
+"Interactions with the Juju environment"
+# source: 27:lp:charmsupport
+# Copyright 2012 Canonical Ltd.
+#
+# Authors:
+#  Matthew Wedgwood <[e-mail]>
+
+import os
+import json
+import yaml
+import subprocess
+
+CRITICAL = "CRITICAL"
+ERROR = "ERROR"
+WARNING = "WARNING"
+INFO = "INFO"
+DEBUG = "DEBUG"
+def log(message, level=DEBUG):
+    "Write a message to the juju log"
+    subprocess.call( [ 'juju-log', '-l', level, message ] )
+
+class Serializable(object):
+    "Wrapper, an object that can be serialized to yaml or json"
+    def __init__(self, obj):
+        # wrap the object
+        super(Serializable, self).__init__()
+        self._wrapped_obj = obj
+
+    def __getattr__(self, attr):
+        # see if this object has attr
+        if attr in self.__dict__:
+            return getattr(self, attr)
+        # proxy to the wrapped object
+        return self[attr]
+
+    def __getitem__(self, key):
+        return self._wrapped_obj[key]
+
+    def json(self):
+        "Serialize the object to json"
+        return json.dumps(self._wrapped_obj)
+
+    def yaml(self):
+        "Serialize the object to yaml"
+        return yaml.dump(self._wrapped_obj)
+
+def execution_environment():
+    """A convenient bundling of the current execution context"""
+    context = {}
+    context['conf'] = config()
+    context['unit'] = local_unit()
+    context['rel'] = relations_of_type()
+    context['env'] = os.environ
+    return context
+
+def in_relation_hook():
+    "Determine whether we're running in a relation hook"
+    return os.environ.has_key('JUJU_RELATION')
+
+def relation_type():
+    "The scope for the current relation hook"
+    return os.environ['JUJU_RELATION']
+def relation_id():
+    "The relation ID for the current relation hook"
+    return os.environ['JUJU_RELATION_ID']
+def local_unit():
+    "Local unit ID"
+    return os.environ['JUJU_UNIT_NAME']
+def remote_unit():
+    "The remote unit for the current relation hook"
+    return os.environ['JUJU_REMOTE_UNIT']
+
+def config(scope=None):
+    "Juju charm configuration"
+    config_cmd_line = ['config-get']
+    if scope is not None:
+        config_cmd_line.append(scope)
+    config_cmd_line.append('--format=json')
+    try:
+        config_data = json.loads(subprocess.check_output(config_cmd_line))
+    except (ValueError, OSError, subprocess.CalledProcessError) as err:
+        log(str(err), level=ERROR)
+        raise err
+    return Serializable(config_data)
+
+def relation_ids(reltype=None):
+    "A list of relation_ids"
+    reltype = reltype or relation_type()
+    relids = []
+    relid_cmd_line = ['relation-ids', '--format=json', reltype]
+    relids.extend(json.loads(subprocess.check_output(relid_cmd_line)))
+    return relids
+
+def related_units(relid=None):
+    "A list of related units"
+    relid = relid or relation_id()
+    units_cmd_line = ['relation-list', '--format=json', '-r', relid]
+    units = json.loads(subprocess.check_output(units_cmd_line))
+    return units
+
+def relation_for_unit(unit=None):
+    "Get the json represenation of a unit's relation"
+    unit = unit or remote_unit()
+    relation_cmd_line = ['relation-get', '--format=json', '-', unit]
+    try:
+        relation = json.loads(subprocess.check_output(relation_cmd_line))
+    except (ValueError, OSError, subprocess.CalledProcessError), err:
+        log(str(err), level=ERROR)
+        raise err
+    for key in relation:
+        if key.endswith('-list'):
+            relation[key] = relation[key].split()
+    relation['__unit__'] = unit
+    return Serializable(relation)
+
+def relations_for_id(relid=None):
+    "Get relations of a specific relation ID"
+    relation_data = []
+    relid = relid or relation_ids()
+    for unit in related_units(relid):
+        unit_data = relation_for_unit(unit)
+        unit_data['__relid__'] = relid
+        relation_data.append(unit_data)
+    return relation_data
+
+def relations_of_type(reltype=None):
+    "Get relations of a specific type"
+    relation_data = []
+    if in_relation_hook():
+        reltype = reltype or relation_type()
+        for relid in relation_ids(reltype):
+            for relation in relations_for_id(relid):
+                relation['__relid__'] = relid
+                relation_data.append(relation)
+    return relation_data
+
+class UnregisteredHookError(Exception): pass
+
+class Hooks(object):
+    def __init__(self):
+        super(Hooks, self).__init__()
+        self._hooks = {}
+    def register(self, name, function):
+        self._hooks[name] = function
+    def execute(self, args):
+        hook_name = os.path.basename(args[0])
+        if hook_name in self._hooks:
+            self._hooks[hook_name]()
+        else:
+            raise UnregisteredHookError(hook_name)
 
=== added file 'scripts/charmsupport/nrpe.py'
--- scripts/charmsupport/nrpe.py	1970-01-01 00:00:00 +0000
+++ scripts/charmsupport/nrpe.py	2013-08-08 17:05:15 +0000
@@ -0,0 +1,169 @@
+"""Compatibility with the nrpe-external-master charm"""
+# source: 27:lp:charmsupport
+# Copyright 2012 Canonical Ltd.
+#
+# Authors:
+#  Matthew Wedgwood <[e-mail]>
+
+import subprocess
+import pwd
+import grp
+import os
+import re
+import shlex
+
+from hookenv import config, local_unit
+
+# This module adds compatibility with the nrpe_external_master
+# subordinate charm. To use it in your charm:
+# 
+# 1. Update metadata.yaml
+#
+#   provides:
+#     (...)
+#     nrpe-external-master:
+#       interface: nrpe-external-master
+#           scope: container
+# 
+# 2. Add the following to config.yaml
+#
+#    nagios_context:
+#      default: "juju"
+#      type: string
+#      description: |
+#        Used by the nrpe-external-master subordinate charm.
+#        A string that will be prepended to instance name to set the host name
+#        in nagios. So for instance the hostname would be something like:
+#            juju-myservice-0
+#        If you're running multiple environments with the same services in them
+#        this allows you to differentiate between them.
+#
+# 3. Add custom checks (Nagios plugins) to files/nrpe-external-master
+#
+# 4. Update your hooks.py with something like this:
+#
+#    import nrpe
+#    (...)
+#    def update_nrpe_config():
+#        nrpe_compat = NRPE("myservice")
+#        nrpe_compat.add_check(
+#            shortname = "myservice",
+#            description = "Check MyService",
+#            check_cmd = "check_http -w 2 -c 10 http://localhost"
+#            )
+#        nrpe_compat.add_check(
+#            "myservice_other",
+#            "Check for widget failures",
+#            check_cmd = "/srv/myapp/scripts/widget_check"
+#            )
+#        nrpe_compat.write()
+#
+#    def config_changed():
+#        (...)
+#        update_nrpe_config()
+#    def nrpe_external_master_relation_changed():
+#        update_nrpe_config()
+#
+# 5. ln -s hooks.py nrpe-external-master-relation-changed
+
+class CheckException(Exception): pass
+class Check(object):
+    shortname_re = '[A-Za-z0-9-_]*'
+    service_template = """
+#---------------------------------------------------
+# This file is Juju managed
+#---------------------------------------------------
+define service {{
+    use                             active-service
+    host_name                       {nagios_hostname}
+    service_description             {nagios_hostname}[{shortname}] {description}
+    check_command                   check_nrpe!check_{shortname}
+    servicegroups                   {nagios_servicegroup}
+}}
+"""
+    def __init__(self, shortname, description, check_cmd):
+        super(Check, self).__init__()
+        # XXX: could be better to calculate this from the service name
+        if not re.match(self.shortname_re, shortname):
+            raise CheckException("shortname must match {}".format(Check.shortname_re))
+        self.shortname = shortname
+        # Note: a set of invalid characters is defined by the Nagios server config
+        # The default is: illegal_object_name_chars=`~!$%^&*"|'<>?,()=
+        self.description = description
+        self.check_cmd = self._locate_cmd(check_cmd)
+
+    def _locate_cmd(self, check_cmd):
+        search_path = (
+            '/',
+            os.path.join(os.environ['CHARM_DIR'], 'files/nrpe-external-master'),
+            '/usr/lib/nagios/plugins',
+        )
+        command = shlex.split(check_cmd)
+        for path in search_path:
+            if os.path.exists(os.path.join(path,command[0])):
+                return os.path.join(path, command[0]) + " " + " ".join(command[1:])
+        subprocess.call(['juju-log', 'Check command not found: {}'.format(command[0])])
+        return ''
+
+    def write(self, nagios_context, hostname):
+        for f in os.listdir(NRPE.nagios_exportdir):
+            if re.search('.*check_{}.cfg'.format(self.shortname), f):
+                os.remove(os.path.join(NRPE.nagios_exportdir, f))
+
+        templ_vars = {
+            'nagios_hostname': hostname,
+            'nagios_servicegroup': nagios_context,
+            'description': self.description,
+            'shortname': self.shortname,
+        }
+        nrpe_service_text = Check.service_template.format(**templ_vars)
+        nrpe_service_file = '{}/service__{}_check_{}.cfg'.format(
+            NRPE.nagios_exportdir, hostname, self.shortname)
+        with open(nrpe_service_file, 'w') as nrpe_service_config:
+            nrpe_service_config.write(str(nrpe_service_text))
+
+        nrpe_check_file = '/etc/nagios/nrpe.d/check_{}.cfg'.format(self.shortname)
+        with open(nrpe_check_file, 'w') as nrpe_check_config:
+            nrpe_check_config.write("# check {}\n".format(self.shortname))
+            nrpe_check_config.write("command[check_{}]={}\n".format(
+                self.shortname, self.check_cmd))
+
+    def run(self):
+        subprocess.call(self.check_cmd)
+
+class NRPE(object):
+    nagios_logdir = '/var/log/nagios'
+    nagios_exportdir = '/var/lib/nagios/export'
+    nrpe_confdir = '/etc/nagios/nrpe.d'
+    def __init__(self):
+        super(NRPE, self).__init__()
+        self.config = config()
+        self.nagios_context = self.config['nagios_context']
+        self.unit_name = local_unit().replace('/', '-')
+        self.hostname = "{}-{}".format(self.nagios_context, self.unit_name)
+        self.checks = []
+
+    def add_check(self, *args, **kwargs):
+        self.checks.append( Check(*args, **kwargs) )
+
+    def write(self):
+        try:
+            nagios_uid = pwd.getpwnam('nagios').pw_uid
+            nagios_gid = grp.getgrnam('nagios').gr_gid
+        except:
+            subprocess.call(['juju-log', "Nagios user not set up, nrpe checks not updated"])
+            return
+
+        if not os.path.exists(NRPE.nagios_exportdir):
+            subprocess.call(['juju-log', 'Exiting as {} is not accessible'.format(NRPE.nagios_exportdir)])
+            return
+
+        if not os.path.exists(NRPE.nagios_logdir):
+            os.mkdir(NRPE.nagios_logdir)
+            os.chown(NRPE.nagios_logdir, nagios_uid, nagios_gid)
+
+        for nrpecheck in self.checks:
+            nrpecheck.write(self.nagios_context, self.hostname)
+
+        if os.path.isfile('/etc/init.d/nagios-nrpe-server'):
+            subprocess.call(['service', 'nagios-nrpe-server', 'reload'])
 
=== added file 'scripts/update-nrpe.py'
--- scripts/update-nrpe.py	1970-01-01 00:00:00 +0000
+++ scripts/update-nrpe.py	2013-08-08 17:05:15 +0000
@@ -0,0 +1,14 @@
+#!/usr/bin/env python
+from charmsupport import nrpe
+
+
+def update_nrpe_config():
+    nrpe_compat = nrpe.NRPE()
+    nrpe_compat.add_check(
+        'App is accessible', 'Check the app can be downloaded',
+        'check-app-access.sh')
+    nrpe_compat.write()
+
+
+if __name__ == '__main__':
+    update_nrpe_config()
 
=== modified file 'tests/20-functional.test'
--- tests/20-functional.test	2013-08-01 14:33:00 +0000
+++ tests/20-functional.test	2013-08-08 17:05:15 +0000
@@ -202,7 +202,18 @@
         self.assertIn('max-age=0', cache_directives)
         self.assertIn('public', cache_directives)
         self.assertIn('must-revalidate', cache_directives)
-        
+
+    def test_nrpe_check_available(self):
+        # Make sure the check-app-access.sh script's ADDRESS is available.
+        unit_info = self.juju_deploy(
+            self.charm, options={'juju-gui-source': JUJU_GUI_TEST_BRANCH})
+        hostname = unit_info['public-address']
+        conn = httplib.HTTPSConnection(hostname)
+        # This request matches the ADDRESS var in the script.
+        conn.request('GET', '/juju-ui/version.js')
+        message = 'ADDRESS in check-app-access.sh is not accessible.'
+        self.assertEqual(200, conn.getresponse().status, message)
+
 
     @unittest.skipIf(is_legacy_juju, 'force-machine only works in juju-core')
     def test_force_machine(self):