Merge lp:~tcuthbert/turku/turku-api into lp:turku/turku-api

Proposed by Thomas Cuthbert
Status: Merged
Approved by: Thomas Cuthbert
Approved revision: 67
Merged at revision: 66
Proposed branch: lp:~tcuthbert/turku/turku-api
Merge into: lp:turku/turku-api
Diff against target: 125 lines (+121/-0)
1 file modified
scripts/turku_sick_sources (+121/-0)
To merge this branch: bzr merge lp:~tcuthbert/turku/turku-api
Reviewer Review Type Date Requested Status
Haw Loeung +1 Approve
Review via email: mp+397366@code.launchpad.net

Commit message

Add unhealthy turku backup sources monitoring script

To post a comment you must log in.
Revision history for this message
Canonical IS Mergebot (canonical-is-mergebot) wrote :

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

Revision history for this message
Haw Loeung (hloeung) wrote :

Overall, looks good to me.

Some comments inline.

review: Approve (+1)
lp:~tcuthbert/turku/turku-api updated
67. By Thomas Cuthbert

Reduce indentation level

Revision history for this message
Canonical IS Mergebot (canonical-is-mergebot) wrote :

Change successfully merged at revision 66

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file 'scripts/turku_sick_sources'
2--- scripts/turku_sick_sources 1970-01-01 00:00:00 +0000
3+++ scripts/turku_sick_sources 2021-02-04 05:09:24 +0000
4@@ -0,0 +1,121 @@
5+#!/usr/bin/env python2
6+
7+# Turku backups
8+# Copyright 2021 Canonical Ltd.
9+#
10+# This program is free software: you can redistribute it and/or modify it
11+# under the terms of the GNU General Public License version 3, as published by
12+# the Free Software Foundation.
13+#
14+# This program is distributed in the hope that it will be useful, but WITHOUT
15+# ANY WARRANTY; without even the implied warranties of MERCHANTABILITY,
16+# SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
17+# General Public License for more details.
18+#
19+# You should have received a copy of the GNU General Public License along with
20+# this program. If not, see <http://www.gnu.org/licenses/>.
21+
22+import argparse
23+import os
24+import sys
25+
26+from time import gmtime, mktime, time
27+
28+BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
29+sys.path.append(BASE_DIR)
30+os.environ.setdefault("DJANGO_SETTINGS_MODULE", "turku_api.settings")
31+
32+from turku_api.models import Source # NOQA: E402
33+
34+
35+MEASUREMENT_NAME = "turku_sick"
36+
37+
38+def options():
39+ parser = argparse.ArgumentParser(
40+ description=SickSources.__doc__, formatter_class=argparse.RawTextHelpFormatter
41+ )
42+ parser.add_argument(
43+ "--metrics_dir", help="directory where metrics will be dumped", type=str
44+ )
45+ return parser.parse_args()
46+
47+
48+class SickSources:
49+ """
50+ turku_sick_sources dumps metrics in the telegraf line format for consumption by telegraf.
51+
52+ The tool emits metrics for turku backup sources that are active and published, and failing their backups.
53+
54+ The tool has a --metrics_dir option that can be used for writing metrics to a file. This is useful when
55+ you want to use cron + the telegraf file input rather than exec. The metrics will be written to a file named
56+ sick_sources.out
57+ """
58+
59+ def run(self, options):
60+ sources_total = 0
61+ sources_sick = 0
62+ sick_objects = []
63+ data = []
64+
65+ for source in Source.objects.filter(
66+ machine__active=True, machine__published=True, active=True, published=True,
67+ ):
68+ sources_total += 1
69+ if not source.healthy():
70+ continue
71+
72+ sources_sick += 1
73+ # Rather than mixing types, initialise timestamp as epoch.
74+ timestamp = mktime(gmtime(0))
75+ if source.date_last_backed_up is not None:
76+ timestamp = mktime(source.date_last_backed_up.timetuple())
77+ source = (source.machine.unit_name, source.name, timestamp)
78+ sick_objects.append(source)
79+
80+ if sources_sick == 0:
81+ sys.stdout.write("Nothing to do, all turku sources are healthy\n")
82+ return
83+
84+ timestamp = int(time() * 1000) # milliseconds
85+ totals = (
86+ "{measurement} sources_unhealthy={sources_sick},sources_total={sources_total} "
87+ "{timestamp}\n"
88+ ).format(
89+ measurement=MEASUREMENT_NAME,
90+ sources_sick=sources_sick,
91+ sources_total=sources_total,
92+ timestamp=timestamp,
93+ )
94+ data.append(totals)
95+
96+ sys.stdout.write(totals)
97+
98+ for source in sick_objects:
99+ machine_unit_name, name, date_last_backed_up = source
100+ metric = (
101+ "{measurement},source_machine_name={machine_unit_name},source_name={name},"
102+ "date_last_backed_up={date_last_backed_up} unhealthy=1 {timestamp}\n"
103+ ).format(
104+ measurement=MEASUREMENT_NAME,
105+ date_last_backed_up=date_last_backed_up,
106+ machine_unit_name=machine_unit_name,
107+ name=name,
108+ timestamp=timestamp,
109+ )
110+ data.append(metric)
111+
112+ sys.stdout.write(metric)
113+
114+ if os.path.isdir(options.metrics_dir):
115+ file_p = os.path.join(options.metrics_dir, "sick_sources.out")
116+ with open(file_p, "w") as f:
117+ f.writelines(data)
118+
119+ sys.stdout.write(
120+ "Sick turku sources successfully dumped to: {}\n".format(file_p)
121+ )
122+
123+
124+if __name__ == "__main__":
125+ sys.exit(SickSources().run(options()))

Subscribers

People subscribed via source and target branches

to all changes: