Merge ~alexsander-souza/maas:lp1996204_fix_prometheus_labels into maas:master

Proposed by Alexsander de Souza
Status: Merged
Approved by: Alexsander de Souza
Approved revision: 986cef3e28fc281ee6aed5f6176310314ba784e7
Merge reported by: MAAS Lander
Merged at revision: not available
Proposed branch: ~alexsander-souza/maas:lp1996204_fix_prometheus_labels
Merge into: maas:master
Diff against target: 37 lines (+12/-3)
1 file modified
src/maasserver/prometheus/middleware.py (+12/-3)
Reviewer Review Type Date Requested Status
Christian Grabowski Approve
MAAS Lander Approve
Review via email: mp+443916@code.launchpad.net

Commit message

normalize request path used as prometheus label

using the original requested path effectively tracks metrics per database object, not per endpoint as intended. Additionally it leads to huge number of histograms in the prometheus database, exceeding the 2GB limit after some time.

fixes LP#1996204

To post a comment you must log in.
Revision history for this message
MAAS Lander (maas-lander) wrote :

UNIT TESTS
-b lp1996204_fix_prometheus_labels lp:~alexsander-souza/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas-tester/2639/console
COMMIT: ceceed1d1bcaa4e6fb9646a93c503658738d641f

review: Needs Fixing
Revision history for this message
MAAS Lander (maas-lander) wrote :

UNIT TESTS
-b lp1996204_fix_prometheus_labels lp:~alexsander-souza/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: SUCCESS
COMMIT: 986cef3e28fc281ee6aed5f6176310314ba784e7

review: Approve
Revision history for this message
Christian Grabowski (cgrabowski) wrote :

+1

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/src/maasserver/prometheus/middleware.py b/src/maasserver/prometheus/middleware.py
2index 02b4867..94cc270 100644
3--- a/src/maasserver/prometheus/middleware.py
4+++ b/src/maasserver/prometheus/middleware.py
5@@ -6,6 +6,7 @@ from time import time
6
7 from django.db import connections
8 from django.db.backends.utils import CursorWrapper
9+from django.urls import resolve, reverse
10
11 from provisioningserver.prometheus.metrics import PROMETHEUS_METRICS
12
13@@ -56,13 +57,21 @@ class PrometheusRequestMetricsMiddleware:
14 return response
15
16 def _process_metrics(self, request, response, latency, query_latencies):
17- op = request.POST.get("op", request.GET.get("op", ""))
18 labels = {
19 "method": request.method,
20- "path": request.path,
21 "status": response.status_code,
22- "op": op,
23+ "op": request.POST.get("op", request.GET.get("op", "")),
24+ "path": request.path,
25 }
26+ try:
27+ match = resolve(request.path.removeprefix("/MAAS"))
28+ args = [f":arg{i}" for i in range(len(match.args))]
29+ kwargs = {k: f":{k}" for k in match.kwargs.keys()}
30+ labels["path"] = reverse(match.url_name, None, args, kwargs)
31+ except Exception:
32+ # use the request path as-is
33+ pass
34+
35 self.prometheus_metrics.update(
36 "maas_http_request_latency",
37 "observe",

Subscribers

People subscribed via source and target branches