Merge lp:~tcuthbert/turku/turku-api into lp:turku/turku-api

[Thomas Cuthbert]

Refactor the metrics based on promtheus querying.

This change refactors the way we are exporting the health of backups metrics based on my findings with prometheus querying. Below outlines the required changes:

* Relying on the influx line protocol "timestamp" element is flimsy, influx expects nanosecond precision while telegraf defaults to 1s. So just abandon the whole idea and let telegraf handle the metric timestamp for us.
* Instead of exporting the metrics as either healthy/unhealthy (0/1), encode the health as a label which we can use to filter. The data is just the date last backed up as a unix timestamp. We can then use the prometheus time() function and subtract the date_last_backed value to work out how many days a back up is out of date.

So to figure out what hasn't been backed up in 90 days we can use an expression like `turku_sick_date_last_backed_up != 0 and (time () - turku_sick_date_last_backed_up{source_machine_name=~"hosts_i_care_about"}) / (60 * 60 * 24) > 90`.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[lp:~tcuthbert/turku/turku-api]

68. By Thomas Cuthbert on 2021-02-04

Refactor the metrics based on findings with prometheus querying.

[Junien F (axino)]

+1

review: Approve

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision 67

=== modified file 'scripts/turku_sick_sources'
--- scripts/turku_sick_sources	2021-02-04 05:08:58 +0000
+++ scripts/turku_sick_sources	2021-02-04 07:23:51 +0000
@@ -19,7 +19,7 @@
 import os
 import sys
 
-from time import gmtime, mktime, time
+from time import mktime
 
 BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
 sys.path.append(BASE_DIR)
@@ -52,6 +52,10 @@
     sick_sources.out
     """
 
+    # Telegraf doesn't like boolean metrics so use this primitive enum-like object
+    # for readability.
+    _health_enum = {0: "no", 1: "yes"}
+
     def run(self, options):
         sources_total = 0
         sources_sick = 0
@@ -63,45 +67,47 @@
         ):
             sources_total += 1
             if not source.healthy():
-                continue
-
-            sources_sick += 1
-            # Rather than mixing types, initialise timestamp as epoch.
-            timestamp = mktime(gmtime(0))
+                sources_sick += 1
+            # Rather than mixing types, initialise timestamp as epoch at default precision.
+            date_last_backed_up = 0.0
             if source.date_last_backed_up is not None:
-                timestamp = mktime(source.date_last_backed_up.timetuple())
-            source = (source.machine.unit_name, source.name, timestamp)
+                date_last_backed_up = mktime(source.date_last_backed_up.timetuple())
+            source = (
+                source.machine.unit_name,
+                source.name,
+                self._health_enum[int(source.healthy())],
+                date_last_backed_up,
+            )
             sick_objects.append(source)
 
         if sources_sick == 0:
             sys.stdout.write("Nothing to do, all turku sources are healthy\n")
             return
 
-        timestamp = int(time() * 1000)  # milliseconds
         totals = (
-            "{measurement} sources_unhealthy={sources_sick},sources_total={sources_total} "
-            "{timestamp}\n"
+            "{measurement} sources_unhealthy={sources_sick},sources_total={sources_total}\n"
         ).format(
             measurement=MEASUREMENT_NAME,
             sources_sick=sources_sick,
             sources_total=sources_total,
-            timestamp=timestamp,
         )
         data.append(totals)
 
         sys.stdout.write(totals)
 
         for source in sick_objects:
-            machine_unit_name, name, date_last_backed_up = source
+            machine_unit_name, name, health, date_last_backed_up = source
+            # The metrics we are exporting is the unix timestamp of date last backed up. Include a healthy label
+            # for concise query filtering.
             metric = (
-                "{measurement},source_machine_name={machine_unit_name},source_name={name},"
-                "date_last_backed_up={date_last_backed_up} unhealthy=1 {timestamp}\n"
+                "{measurement},source_machine_name={machine_unit_name},source_name={name},healthy={health} "
+                "date_last_backed_up={date_last_backed_up}\n"
             ).format(
                 measurement=MEASUREMENT_NAME,
-                date_last_backed_up=date_last_backed_up,
                 machine_unit_name=machine_unit_name,
                 name=name,
-                timestamp=timestamp,
+                health=health,
+                date_last_backed_up=date_last_backed_up,
             )
             data.append(metric)