Merge lp:~james-w/charms/precise/haproxy/metrics into lp:charms/haproxy

[James Westby]

Hi,

This is based on https://code.launchpad.net/~bloodearnest/charms/precise/squid-reverseproxy/metrics/+merge/217138, and adds support for converting the haproxy statistics to
carbon format and sending to graphite or a compatible server.

It relies on monitoring being enabled, rather than forcing a
localhost monitoring declaration.

Thanks,

James

[Simon Davy (bloodearnest)]

Shouldn't enabled_monitoring be added to config.yaml?

A thought: in the squid version, squid actually provides average aggregate measures for 1min/5min/60min windows, which is nice as your 5 min cron can use that for better accuracy.

I am assuming the we're sampling current unaggregated metrics here? So, it it worth thinking about a more frequent cron timing? Is 5min enough?

Another thing - I thing it would be worth trying to use statsd rather than carbon?

We'd need to do 2 things

1. change the "| nc ..." part to:

... | while read line; do echo $line | nc -u {{ statsd_host }} {{ statsd_port }}; done

which would send 1 metric per packet? Note: we could also update txstatsd to support multi-metric packets, which would then simplify this back to just "| nc -u ..."

2. add a function to output statsd format rather than carbonify (untested)

statsdify() {
   sed -r "s/([^ ]+) (.*)/${PREFIX}.\1.${PERIOD}:\2|g/"
}

(ps I like explicitly putting the period in there, rather than assuming it's in the metric_scheme, I may change the squid version to that, only reason I didn't was that I think it's custom to our statsd setup, wasn't sure it was valid to put upstream, but maybe its useful info even if your statsd setup doesn't do anything special with it)

[Simon Davy (bloodearnest)]

review: Approve

[David Ames (thedac)]

Confirmed with James the statsd format is correct and working

+1 looks good

Merging

review: Approve

[lp:~james-w/charms/precise/haproxy/metrics]

79. By David Ames on 2014-05-27

[james_w,r=dames] Allow sending haproxy metrics to external statsd service.

=== modified file 'README.md'
--- README.md	2014-01-07 19:03:41 +0000
+++ README.md	2014-05-27 21:55:55 +0000
@@ -118,6 +118,18 @@
 Many of the haproxy settings can be altered via the standard juju configuration
 settings.  Please see the config.yaml file as each is fairly clearly documented.
 
+## statsd
+
+This charm supports sending metrics to statsd.
+
+This is done by setting config values (metrics_target being the primary one)
+to a host/port of a (UDP) statsd server.
+
+This could instead be done using a relation, but it is common to have
+one statsd server that serves multiple environments. Once juju supports
+cross-environment relations then that will be the best way to handle 
+this configuration, as it will work in either scenario.
+
 ## HAProxy Project Information
 
 - [HAProxy Homepage](http://haproxy.1wt.eu/)

=== modified file 'config.yaml'
--- config.yaml	2014-04-30 21:48:17 +0000
+++ config.yaml	2014-05-27 21:55:55 +0000
@@ -137,3 +137,20 @@
             juju-postgresql-0
         If you're running multiple environments with the same services in them
         this allows you to differentiate between them.
+  metrics_target:
+    default: ""
+    type: string
+    description: |
+        Destination for statsd-format metrics, format "host:port". If
+        not present and valid, metrics disabled. Requires "enable_monitoring"
+        to be set to true to work.
+  metrics_prefix:
+    default: "dev.$UNIT.haproxy"
+    type: string
+    description: |
+        Prefix for metrics. Special value $UNIT can be used to include the
+        name of the unit in the prefix.
+  metrics_sample_interval:
+    default: 5
+    type: int
+    description: Period for metrics cron job to run in minutes

=== added directory 'files/metrics'
=== added file 'files/metrics/haproxy_to_statsd.sh'
--- files/metrics/haproxy_to_statsd.sh	1970-01-01 00:00:00 +0000
+++ files/metrics/haproxy_to_statsd.sh	2014-05-27 21:55:55 +0000
@@ -0,0 +1,27 @@
+#!/bin/bash
+# haproxy-to-stasd: query haproxy CSV status page, transform it
+# to stdout suitable for statsd
+#
+# Author: JuanJo Ciarlante <[e-mail]>
+# Copyright 2012, Canonical Ltd.
+# License: GPLv3
+set -u
+PREFIX=${1:?missing statsd node prefix, e.g.: production.host.${HOSTNAME}.haproxy.stats}
+PERIOD=${2:?missing period, e.g.: 10min}
+HOSTPORT=${3:?missing haproxy hostport, e.g.: localhost:10000}
+HTTPAUTH=${4:?missing httpauth, e.g.: user:pass}
+
+TSTAMP="$(date +%s)"
+
+# Filter only numeric metrics, cleanup to be only <var> <value>
+get_metrics() {
+    curl -s "http://${HTTPAUTH}@${HOSTPORT}/;csv;norefresh"| \
+        awk -v FS=, '/^#/{ for(i=1;i<NF;i++) fieldname[i]=$i;}
+/^[^#]/{ for(i=3;i<NF;i++) if(length($i) > 0) printf("%s.%s.%s %d\n", $1, $2, fieldname[i], $i);}'
+}
+
+# Add statsd-isms to <var> <value> lines
+statsdify() {
+    sed -r "s/([^ ]+) (.*)/${PREFIX}.\1.${PERIOD}:\2|g/"
+}
+get_metrics | statsdify

=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2014-04-30 21:48:17 +0000
+++ hooks/hooks.py	2014-05-27 21:55:55 +0000
@@ -16,6 +16,7 @@
 from charmhelpers.core.hookenv import (
     log,
     config as config_get,
+    local_unit,
     relation_set,
     relation_ids as get_relation_ids,
     relations_of_type,
@@ -36,6 +37,8 @@
 default_haproxy_config = "%s/haproxy.cfg" % default_haproxy_config_dir
 default_haproxy_service_config_dir = "/var/run/haproxy"
 default_haproxy_lib_dir = "/var/lib/haproxy"
+metrics_cronjob_path = "/etc/cron.d/haproxy_metrics"
+metrics_script_path = "/usr/local/bin/haproxy_to_statsd.sh"
 service_affecting_packages = ['haproxy']
 
 dupe_options = [
@@ -95,6 +98,15 @@
         dpkg.communicate(input=selections)
 
 
+def render_template(template_name, vars):
+    # deferred import so install hook can install jinja2
+    from jinja2 import Environment, FileSystemLoader
+    templates_dir = os.path.join(os.environ['CHARM_DIR'], 'templates')
+    template_env = Environment(loader=FileSystemLoader(templates_dir))
+    template = template_env.get_template(template_name)
+    return template.render(vars)
+
+
 #------------------------------------------------------------------------------
 # enable_haproxy:  Enabled haproxy at boot time
 #------------------------------------------------------------------------------
@@ -737,7 +749,7 @@
     if not os.path.exists(default_haproxy_service_config_dir):
         os.mkdir(default_haproxy_service_config_dir, 0600)
 
-    apt_install('haproxy', fatal=True)
+    apt_install(['haproxy', 'python-jinja2'], fatal=True)
     ensure_package_status(service_affecting_packages,
                           config_get('package_status'))
     enable_haproxy()
@@ -767,6 +779,9 @@
                              haproxy_monitoring,
                              haproxy_services)
 
+    write_metrics_cronjob(metrics_script_path,
+                          metrics_cronjob_path)
+
     if service_haproxy("check"):
         update_service_ports(old_service_ports, get_service_ports())
         service_haproxy("reload")
@@ -895,6 +910,61 @@
     nrpe_compat.write()
 
 
+def delete_metrics_cronjob(cron_path):
+    try:
+        os.unlink(cron_path)
+    except OSError:
+        pass
+
+
+def write_metrics_cronjob(script_path, cron_path):
+    config_data = config_get()
+
+    if config_data['enable_monitoring'] is False:
+        log("enable_monitoring must be set to true for metrics")
+        delete_metrics_cronjob(cron_path)
+        return
+
+    # need the following two configs to be valid
+    metrics_target = config_data['metrics_target'].strip()
+    metrics_sample_interval = config_data['metrics_sample_interval']
+    if (not metrics_target
+            or ':' not in metrics_target
+            or not metrics_sample_interval):
+        log("Required config not found or invalid "
+            "(metrics_target, metrics_sample_interval), "
+            "disabling metrics")
+        delete_metrics_cronjob(cron_path)
+        return
+
+    charm_dir = os.environ['CHARM_DIR']
+    statsd_host, statsd_port = metrics_target.split(':', 1)
+    metrics_prefix = config_data['metrics_prefix'].strip()
+    metrics_prefix = metrics_prefix.replace(
+        "$UNIT", local_unit().replace('.', '-').replace('/', '-'))
+    haproxy_hostport = ":".join(['localhost',
+                                str(config_data['monitoring_port'])])
+    haproxy_httpauth = ":".join([config_data['monitoring_username'].strip(),
+                                get_monitoring_password()])
+
+    # ensure script installed
+    shutil.copy2('%s/files/metrics/haproxy_to_statsd.sh' % charm_dir,
+                 metrics_script_path)
+
+    # write the crontab
+    with open(cron_path, 'w') as cronjob:
+        cronjob.write(render_template("metrics_cronjob.template", {
+            'interval': config_data['metrics_sample_interval'],
+            'script': script_path,
+            'metrics_prefix': metrics_prefix,
+            'metrics_sample_interval': metrics_sample_interval,
+            'haproxy_hostport': haproxy_hostport,
+            'haproxy_httpauth': haproxy_httpauth,
+            'statsd_host': statsd_host,
+            'statsd_port': statsd_port,
+        }))
+
+
 ###############################################################################
 # Main section
 ###############################################################################

=== modified file 'hooks/tests/test_config_changed_hooks.py'
--- hooks/tests/test_config_changed_hooks.py	2013-05-10 15:51:40 +0000
+++ hooks/tests/test_config_changed_hooks.py	2014-05-27 21:55:55 +0000
@@ -29,6 +29,7 @@
             "update_sysctl")
         self.notify_website = self.patch_hook("notify_website")
         self.notify_peer = self.patch_hook("notify_peer")
+        self.write_metrics_cronjob = self.patch_hook("write_metrics_cronjob")
         self.log = self.patch_hook("log")
         sys_exit = patch.object(sys, "exit")
         self.sys_exit = sys_exit.start()

=== added file 'hooks/tests/test_metrics.py'
--- hooks/tests/test_metrics.py	1970-01-01 00:00:00 +0000
+++ hooks/tests/test_metrics.py	2014-05-27 21:55:55 +0000
@@ -0,0 +1,75 @@
+import textwrap
+
+from testtools import TestCase
+from mock import patch, mock_open
+
+import hooks
+
+
+class MetricsTestCase(TestCase):
+
+    def add_patch(self, *args, **kwargs):
+        p = patch(*args, **kwargs)
+        self.addCleanup(p.stop)
+        return p.start()
+
+    def setUp(self):
+        super(MetricsTestCase, self).setUp()
+
+        self.open = mock_open()
+        self.add_patch("hooks.open", self.open, create=True)
+        self.copy2 = self.add_patch("shutil.copy2")
+        self.config_get = self.add_patch("hooks.config_get")
+        self.local_unit = self.add_patch("hooks.local_unit")
+        self.log = self.add_patch("hooks.log")
+        self.apt_install = self.add_patch("hooks.apt_install")
+        self.get_monitoring_password = self.add_patch(
+            "hooks.get_monitoring_password")
+
+        self.config_get.return_value = {
+            'metrics_sample_interval': 5,
+            'metrics_prefix': 'prefix.$UNIT',
+            'metrics_target': 'localhost:4321',
+            'enable_monitoring': True,
+            'monitoring_port': '1234',
+            'monitoring_username': 'monitor',
+            'monitoring_password': 'monitorpass',
+        }
+        self.local_unit.return_value = "unit/0"
+
+    @patch('hooks.os.unlink')
+    def test_write_metrics_cronjob_disabled_no_monitoring(self, mock_unlink):
+        self.config_get.return_value['enable_monitoring'] = False
+        hooks.write_metrics_cronjob('/script', '/cron')
+        self.assertEqual(mock_unlink.mock_calls[0][1][0], '/cron')
+
+    @patch('hooks.os.unlink')
+    def test_write_metrics_cronjob_disabled_no_target(self, mock_unlink):
+        self.config_get.return_value['metrics_target'] = ''
+        hooks.write_metrics_cronjob('/script', '/cron')
+        self.assertEqual(mock_unlink.mock_calls[0][1][0], '/cron')
+
+    @patch('hooks.os.unlink')
+    def test_write_metrics_cronjob_disabled_bad_target(self, mock_unlink):
+        self.config_get.return_value['metrics_target'] = 'sadfsadf'
+        hooks.write_metrics_cronjob('/script', '/cron')
+        self.assertEqual(mock_unlink.mock_calls[0][1][0], '/cron')
+
+    def test_write_metrics_cronjob_enabled(self):
+        self.get_monitoring_password.return_value = 'monitorpass'
+        self.config_get.return_value['metrics_target'] = 'localhost:4321'
+
+        hooks.write_metrics_cronjob('/script', '/cron')
+
+        cron_open_args = self.open.mock_calls[0][1]
+        self.assertEqual(cron_open_args, ('/cron', 'w'))
+
+        cron_write = self.open.mock_calls[2][1][0]
+        expected_cron = textwrap.dedent("""
+           # crontab for pushing haproxy metrics to statsd
+           */5 * * * * root bash /script prefix.unit-0 5min localhost:1234\
+ monitor:monitorpass | python -c "import socket, sys; sock =\
+ socket.socket(socket.AF_INET, socket.SOCK_DGRAM); map(lambda line:\
+ sock.sendto(line, ('localhost', 4321)), sys.stdin)"
+        """).lstrip()
+        self.assertEqual(expected_cron, cron_write)

=== added file 'templates/metrics_cronjob.template'
--- templates/metrics_cronjob.template	1970-01-01 00:00:00 +0000
+++ templates/metrics_cronjob.template	2014-05-27 21:55:55 +0000
@@ -0,0 +1,3 @@
+# crontab for pushing haproxy metrics to statsd
+*/{{ metrics_sample_interval }} * * * * root bash {{ script }} {{ metrics_prefix }} {{ metrics_sample_interval }}min {{ haproxy_hostport }} {{ haproxy_httpauth }} | python -c "import socket, sys; sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM); map(lambda line: sock.sendto(line, ('{{ statsd_host }}', {{ statsd_port }})), sys.stdin)"
+