Merge ~ilasc/turnip:add-statsd-env-label into turnip:master

[Ioana Lasc]

Add environment label for Statsd metrics

This branch changes the prefix from 'turnip_production_git'/'turnip_qastaging_git'/ 'turnip_test_git' to 'turnip' for all environments. This prefix together with the gauge name set to 'statsd' will give us the generic top label: 'turnip_statsd'.

Notes:
1: We already have an environment label in production, but not in QAS:
https://chat.canonical.com/canonical/pl/qgpj6kfuoifs7xqnin8pwqwfsa
https://chat.canonical.com/canonical/pl/4dagm948kb8btja3eenqk5kmny

For this reason I went with 'env' as the label name so it doesn't clash with the existent one in Production.

2: We need to pass in some name ('statsd') for the gauge so that the top label stays generic (currently in this branch: 'turnip_statsd' https://chat.canonical.com/canonical/pl/41jwxw9zu7nt8by4czr6qx61dh).
If we pass in only the prefix, the top label will not remain generic as the first gauge label (operation) gets attached to the prefix and results in as many top labels as there are git operations (git-receive-pack, git-upload-pack....): https://chat.canonical.com/canonical/pl/m3cpr1q98bfhxmso7w8anm4cqw

[Colin Watson (cjwatson)]

review: Approve

[Ioana Lasc (ilasc)]

Thanks Colin!
Addressed comments and indeed I changed from "git" to "statsd" in the gauge name as with "git" the top label would have been "turnip_git".

I figured "turnip_statsd" would be more indicative of the metrics origin in the dashboards, but I can definitely go back to "git" & "turnip_git" if you think that's better ?

[Colin Watson (cjwatson)]

statsd is the interface via which we post the metrics, but that doesn't serve to distinguish one metric from another within turnip. Once we're emitting metrics from more than one spot (for example, I can imagine that we might want to emit a similar metric for cgit subprocesses, or for various API operations, etc.), the metric name needs to help us distinguish between those.

"git" is possibly slightly vaguer than I would have chosen, but it's reasonable enough since these metrics describe the resource consumption of a git subprocess.

[Ioana Lasc (ilasc)]

That makes sense Colin, thank you, moved to "git" form "statsd".

diff --git a/charm/turnip-pack-backend/config.yaml b/charm/turnip-pack-backend/config.yaml
index 489c6ec..c3a26c1 100644
--- a/charm/turnip-pack-backend/config.yaml
+++ b/charm/turnip-pack-backend/config.yaml
@@ -21,5 +21,9 @@ options:
     description: Port for Telegraf configured with the statsd input.
   statsd_prefix:
     type: string
-    default: turnip.test
+    default: turnip
     description: Prefix for statsd metrics.
+  statsd_environment:
+    type: string
+    default: local
+    description: 'env' label to set for statsd metrics.
diff --git a/charm/turnip-pack-backend/templates/turnip-pack-backend.service.j2 b/charm/turnip-pack-backend/templates/turnip-pack-backend.service.j2
index cc92150..29d4cc6 100644
--- a/charm/turnip-pack-backend/templates/turnip-pack-backend.service.j2
+++ b/charm/turnip-pack-backend/templates/turnip-pack-backend.service.j2
@@ -19,6 +19,7 @@ Environment=VIRTINFO_TIMEOUT={{ virtinfo_timeout }}
 Environment=STATSD_HOST={{ statsd_host }}
 Environment=STATSD_PORT={{ statsd_port }}
 Environment=STATSD_PREFIX={{ statsd_prefix }}
+Environment=STATSD_ENVIRONMENT={{ statsd_environment }}
 ExecStart={{ venv_dir }}/bin/twistd --nodaemon --pidfile= --logfile={{ base_dir }}/logs/turnip-pack-backend.log --python=packbackendserver.tac
 ExecReload=/bin/kill -s HUP $MAINPID
 LimitNOFILE=1048576
diff --git a/config.yaml b/config.yaml
index af7daa5..0273840 100644
--- a/config.yaml
+++ b/config.yaml
@@ -23,4 +23,5 @@ main_site_root: https://launchpad.test/
 celery_broker: pyamqp://guest@localhost//
 statsd_host: 127.0.0.1
 statsd_port: 8125
-statsd_prefix: turnip.test
+statsd_prefix: turnip
+statsd_environment: local
diff --git a/turnip/pack/git.py b/turnip/pack/git.py
index 0d8292f..a38c4bb 100644
--- a/turnip/pack/git.py
+++ b/turnip/pack/git.py
@@ -327,24 +327,31 @@ class GitProcessProtocol(protocol.ProcessProtocol, object):
                 # can't be used in statsd
                 repository = re.sub('[^0-9a-zA-Z]+', '-',
                                     self.peer.raw_pathname)
-                gauge_name = ("git,operation={},repo={},metric=max_rss"
-                              .format(
-                                  self.peer.command,
-                                  repository))
+                environment = config.get("statsd_environment")
+                gauge_name = (
+                    "git,operation={},repo={},env={},metric=max_rss"
+                    .format(
+                        self.peer.command,
+                        repository,
+                        environment))
 
                 self.statsd_client.gauge(gauge_name, resource_usage['max_rss'])
 
-                gauge_name = ("git,operation={},repo={},metric=system_time"
-                              .format(
-                                  self.peer.command,
-                                  repository))
+                gauge_name = (
+                    "git,operation={},repo={},env={},metric=system_time"
+                    .format(
+                        self.peer.command,
+                        repository,
+                        environment))
                 self.statsd_client.gauge(gauge_name,
                                          resource_usage['system_time'])
 
-                gauge_name = ("git,operation={},repo={},metric=user_time"
-                              .format(
-                                  self.peer.command,
-                                  repository))
+                gauge_name = (
+                    "git,operation={},repo={},env={},metric=user_time"
+                    .format(
+                        self.peer.command,
+                        repository,
+                        environment))
                 self.statsd_client.gauge(gauge_name,
                                          resource_usage['user_time'])
 
diff --git a/turnip/pack/tests/test_functional.py b/turnip/pack/tests/test_functional.py
index fb168e7..4e5cbf2 100644
--- a/turnip/pack/tests/test_functional.py
+++ b/turnip/pack/tests/test_functional.py
@@ -106,6 +106,7 @@ class FunctionalTestMixin(WithScenarios):
         self.virtinfo.ref_permissions = {
             b'refs/heads/master': ['create', 'push']}
         config.defaults['virtinfo_endpoint'] = self.virtinfo_url
+        config.defaults['statsd_environment'] = 'local'
 
     def startHookRPC(self):
         self.hookrpc_handler = HookRPCHandler(self.virtinfo_url, 15)
@@ -189,8 +190,10 @@ class FunctionalTestMixin(WithScenarios):
         metrics = ['max_rss', 'system_time', 'user_time']
         repository = re.sub('[^0-9a-zA-Z]+', '-', repo)
         self.assertThat(self.statsd_client.vals, MatchesDict({
-            u'git,operation={},repo={},metric={}'.format(
-                command, repository, metric): Not(Is(None))
+            u'git,operation={},repo={},env={},metric={}'
+            .format(
+                command, repository,
+                config.get('statsd_environment'), metric): Not(Is(None))
             for metric in metrics}))
 
     @defer.inlineCallbacks