Merge lp:~tcuthbert/charms/trusty/squid-reverseproxy/trunk into lp:charms/trusty/squid-reverseproxy

[Thomas Cuthbert]

This changes the way we monitor squid peers.

With the introduction of the nagios_servicegroups config option we are able to statically assign which nagios groups the checks will be in. This is useful in situation where the default group causes unnecessary critical alerts for minor events such as single peers going down.

The second change is a configurable weight option. This weight option influences the check_squidpeers script to alert when a percentage of the hosts are down (defaults at 50%). This can be optimised further by providing weights on individual peers (for example do we care if random peer in Eastern Europe goes down?).

[lp:~tcuthbert/charms/trusty/squid-reverseproxy/trunk]

58. By Thomas Cuthbert on 2016-12-29

[tcuthbert] Added calucalate percent up function and modified get_status to work off a configurable percentage down instead of all

[Barry Price (barryprice)]

The alert weighting option is a welcome addition, LGTM +1

review: Approve

[Stuart Bishop (stub)]

Yup. Looks good.

review: Approve

=== modified file 'config.yaml'
--- config.yaml	2015-07-14 02:52:30 +0000
+++ config.yaml	2016-12-29 03:20:43 +0000
@@ -91,6 +91,11 @@
     type: int
     default: 3401
     description: Port for snmp service
+  nagios_down_threshold:
+    default: 50
+    type: int
+    description: >
+      The percentage of downed squid peers that we care to alert on.
   nagios_context:
     default: "juju"
     type: string
@@ -116,6 +121,12 @@
     type: string
     description: >
       What service this component forms part of, e.g. supermassive-squid-cluster. Used by nrpe.
+  nagios_servicegroups:
+      default: ""
+      type: string
+      description: >
+          A comma-separated list of nagios servicegroups. If left empty,
+          the nagios_context will be used as the servicegroup.
   package_status:
      default: "install"
      type: "string"

=== modified file 'files/check_squidpeers'
--- files/check_squidpeers	2014-12-22 13:18:59 +0000
+++ files/check_squidpeers	2016-12-29 03:20:43 +0000
@@ -1,10 +1,10 @@
 #!/usr/bin/python
 
 import argparse
-from operator import itemgetter
 import re
 import subprocess
 import sys
+from operator import iadd
 
 
 parent_re = re.compile(r'^Parent\s+:\s*(?P<name>\S+)')
@@ -42,14 +42,19 @@
     return peers
 
 
-def get_status(peers):
-    ok = all(map(itemgetter('up'), peers))
+def calculate_perc_up(peers):
+    return ((reduce(iadd, [1. for i in peers if i['status'] == STATUS_UP]))
+            / len(peers)) * 100
+
+
+def get_status(peers, weight):
+    perc = calculate_perc_up(peers)
     if not peers:
         retcode = 1
         message = 'Squid has no configured peers.'
-    elif ok:
+    elif perc > float(weight):
         retcode = 0
-        message = 'All peers are UP according to squid.'
+        message = '{:.0f}% of peers are UP according to squid.'.format(perc)
     else:
         retcode = 2
         peer_info = ["{}: {}".format(p['name'], p['status']) for p in peers
@@ -63,6 +68,8 @@
     parser = argparse.ArgumentParser(description='check_squidpeers')
     parser.add_argument('-p', '--port', default=3128,
                         help='Port number squid listens on')
+    parser.add_argument('-w', '--weight', default=50,
+                        help='Percentage of down hosts to alert on')
     args = parser.parse_args()
     proc = subprocess.Popen(['squidclient', '-p', str(args.port), 'mgr:server_list'],
             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
@@ -71,7 +78,7 @@
         print("Error running squidclient: %s" % stderr)
         return 2
     peers = parse(stdout)
-    retcode, message = get_status(peers)
+    retcode, message = get_status(peers, args.weight)
     print(message)
     return retcode
 

=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2015-05-13 04:42:30 +0000
+++ hooks/hooks.py	2016-12-29 03:20:43 +0000
@@ -374,7 +374,7 @@
     nrpe_compat.add_check(
         shortname='squidpeers',
         description='Check Squid Peers',
-        check_cmd='check_squidpeers -p {port}'.format(**config_data)
+        check_cmd='check_squidpeers -p {port} -w {nagios_down_threshold}'.format(**config_data)
     )
     check_http_params = conf.get('nagios_check_http_params')
     if check_http_params: