Merge lp:~blake-rouse/maas/maas-prometheus into lp:~maas-committers/maas/trunk

[Blake Rouse]

Add the ability to enable basic prometheus metric collections.

This allow very basic prometheus metric collection. To enable this on an installed system you have to install django-prometheus, its not something we will require in packaging. It is included in the snap so enabling there is much easier.

Each working runs on the ports:

5260 - worker 1
5261 - worker 2
5262 - worker 3
5263 - worker 4

Only very basic metrics are collected at the moment, but we can expand on this easily to get more information on the load of MAAS and its internals.

[Mike Pontillo (mpontillo)]

This is really cool. I tested the branch (with Blake's help) by doing the following:

 - sudo snap install prometheus
 - verified prometheus was running by hitting http://localhost:9090/
 - Installed the Prometheus Django plugin on the MAAS server
    sudo apt install python3-pip
    sudo pip3 install django-prometheus
 - Changed the configuration of the snap to be like the following[1]:
    http://paste.ubuntu.com/24375943/
    http://paste.ubuntu.com/24375947/
 - Changed /etc/maas/regiond.conf to add the following[2]:
    prometheus_export: true
    prometheus_export_address: 192.168.0.2

The good news is, I begun seeing metrics on the Prometheus dashboard. The bad news is, we found a couple of issues:

 - Due to what looks like a bug in the Prometheus Django plugin[3], a crash can occur which causes the dashboard not to appear.

 - Observed an "OSError: [Errno 98] Address already in use" error while running database migrations. Probably need a way to prevent Prometheus from running except on the "real" region (i.e., not in a script, etc.).

---

[1]: Except that I was running the MAAS region on a different machine than prometheus, so used the following for target config, where 192.168.0.2 is the MAAS IP address:
  - targets:
    - 192.168.0.2:5260
    - 192.168.0.2:5261
    - 192.168.0.2:5262
    - 192.168.0.2:5263

[2]: Blake and I agreed that these options should be renamed as follows:

    prometheus: true
    prometheus_listen_address: 172.16.42.2

[3]:
http://paste.ubuntu.com/24376097/

review: Needs Fixing

[Lee Trager (ltrager)]

You mentioned that django-prometheus will ship with the Snap. Does that mean prometheus will also be on in the Snap? I think even if django-prometheus is installed there should be some additional user action to get metrics.

Also, it looks like there is some additional configuration and setup to get metrics from this. It'd be helpful to have those steps documented in trunk.

review: Needs Information

[Mike Pontillo (mpontillo)]

@ltrager, prometheus must be installed separately.

[MAAS Lander (maas-lander)]

Transitioned to Git.

lp:maas has now moved from Bzr to Git.
Please propose your branches with Launchpad using Git.

git clone https://git.launchpad.net/maas

Unmerged revisions

5968. By Blake Rouse on 2017-04-12

Add the ability to enable prometheus metric collections.

=== modified file 'buildout.cfg'
--- buildout.cfg	2017-03-31 09:35:11 +0000
+++ buildout.cfg	2017-04-12 21:37:46 +0000
@@ -106,6 +106,7 @@
 recipe = zc.recipe.egg
 test-eggs =
   ${common:test-eggs}
+  django-prometheus
   selenium
 eggs =
   ${region:test-eggs}

=== modified file 'requirements.txt'
--- requirements.txt	2017-03-30 20:42:36 +0000
+++ requirements.txt	2017-04-12 21:37:46 +0000
@@ -2,6 +2,7 @@
 convoy==0.4.4
 crochet==1.5.0
 django==1.8
+django-prometheus==1.0.8
 formencode==1.3.1
 httplib2==0.9.2
 jsonschema==2.5.1

=== modified file 'src/maasserver/config.py'
--- src/maasserver/config.py	2017-03-30 19:40:18 +0000
+++ src/maasserver/config.py	2017-04-12 21:37:46 +0000
@@ -7,7 +7,10 @@
     "RegionConfiguration",
 ]
 
-from formencode.validators import Int
+from formencode.validators import (
+    Int,
+    StringBool,
+)
 from provisioningserver.config import (
     Configuration,
     ConfigurationFile,
@@ -51,3 +54,15 @@
     database_pass = ConfigurationOption(
         "database_pass", "The password for the PostgreSQL user.",
         UnicodeString(if_missing="", accept_python=False))
+
+    # Prometheus
+    prometheus_export = ConfigurationOption(
+        "prometheus_export",
+        "Enable exporting of prometheus metrics. A port will be selected "
+        "for each worker from 5260+.",
+        StringBool(if_missing=False, accept_python=False))
+    prometheus_export_address = ConfigurationOption(
+        "prometheus_export_address",
+        "Address that prometheus should export metrics on. "
+        "(Default: localhost)",
+        UnicodeString(if_missing="localhost", accept_python=False))

=== modified file 'src/maasserver/djangosettings/development.py'
--- src/maasserver/djangosettings/development.py	2017-03-30 18:47:16 +0000
+++ src/maasserver/djangosettings/development.py	2017-04-12 21:37:46 +0000
@@ -49,7 +49,10 @@
 DATABASES = {
     'default': {
         # 'postgresql_psycopg2', 'postgresql', 'mysql', 'sqlite3' etc.
-        'ENGINE': 'django.db.backends.postgresql_psycopg2',
+        'ENGINE': (
+            'django_prometheus.db.backends.postgresql'
+            if PROMETHEUS_EXPORT else  # noqa
+            'django.db.backends.postgresql_psycopg2'),
         'NAME': 'maas',
         # For PostgreSQL, a "hostname" starting with a slash indicates a
         # Unix socket directory.

=== modified file 'src/maasserver/djangosettings/settings.py'
--- src/maasserver/djangosettings/settings.py	2017-03-30 18:47:16 +0000
+++ src/maasserver/djangosettings/settings.py	2017-04-12 21:37:46 +0000
@@ -360,3 +360,22 @@
 
 # Fix crooked settings.
 fix_up_databases(DATABASES)
+
+# Enable prometheus metrics and exporting if enabled.
+with RegionConfiguration.open() as config:
+    PROMETHEUS_EXPORT = config.prometheus_export
+    PROMETHEUS_METRICS_EXPORT_ADDRESS = config.prometheus_export_address
+if PROMETHEUS_EXPORT:
+    INSTALLED_APPS += (
+        'django_prometheus',
+    )
+    MIDDLEWARE_CLASSES = (
+        'django_prometheus.middleware.PrometheusBeforeMiddleware',
+    ) + MIDDLEWARE_CLASSES + (
+        'django_prometheus.middleware.PrometheusAfterMiddleware',
+    )
+    if 'default' in DATABASES:
+        DATABASES['default']['ENGINE'] = (
+            'django_prometheus.db.backends.postgresql')
+    PROMETHEUS_METRICS_EXPORT_PORT = (
+        5259 + int(os.environ.get('MAAS_REGIOND_WORKER_ID', 1)))

=== modified file 'src/maasserver/models/event.py'
--- src/maasserver/models/event.py	2017-03-02 00:22:23 +0000
+++ src/maasserver/models/event.py	2017-04-12 21:37:46 +0000
@@ -19,6 +19,7 @@
 from maasserver.models.eventtype import EventType
 from maasserver.models.node import Node
 from maasserver.models.timestampedmodel import TimestampedModel
+from maasserver.utils.prometheus import ExportModelOperationsMixin
 from provisioningserver.events import EVENT_DETAILS
 from provisioningserver.logger import get_maas_logger
 from provisioningserver.utils.env import get_maas_id
@@ -65,7 +66,7 @@
             event_description=event_description)
 
 
-class Event(CleanSave, TimestampedModel):
+class Event(ExportModelOperationsMixin("event"), CleanSave, TimestampedModel):
     """An `Event` represents a MAAS event.
 
     :ivar type: The event's type.

=== modified file 'src/maasserver/models/node.py'
--- src/maasserver/models/node.py	2017-04-04 21:07:13 +0000
+++ src/maasserver/models/node.py	2017-04-12 21:37:46 +0000
@@ -152,6 +152,7 @@
     post_commit_do,
     transactional,
 )
+from maasserver.utils.prometheus import ExportModelOperationsMixin
 from maasserver.utils.threads import (
     callOutToDatabase,
     deferToDatabase,
@@ -789,7 +790,7 @@
     ]
 
 
-class Node(CleanSave, TimestampedModel):
+class Node(ExportModelOperationsMixin("node"), CleanSave, TimestampedModel):
     """A `Node` represents a physical machine used by the MAAS Server.
 
     :ivar system_id: The unique identifier for this `Node`.

=== modified file 'src/maasserver/models/regioncontrollerprocess.py'
--- src/maasserver/models/regioncontrollerprocess.py	2016-03-28 13:54:47 +0000
+++ src/maasserver/models/regioncontrollerprocess.py	2017-04-12 21:37:46 +0000
@@ -15,9 +15,12 @@
 from maasserver.models.cleansave import CleanSave
 from maasserver.models.node import Node
 from maasserver.models.timestampedmodel import TimestampedModel
-
-
-class RegionControllerProcess(CleanSave, TimestampedModel):
+from maasserver.utils.prometheus import ExportModelOperationsMixin
+
+
+class RegionControllerProcess(
+        ExportModelOperationsMixin("region_process"),
+        CleanSave, TimestampedModel):
     """A `RegionControllerProcess` that is running on a `RegionController` or
     `RegionRackController`.
 

=== modified file 'src/maasserver/models/regioncontrollerprocessendpoint.py'
--- src/maasserver/models/regioncontrollerprocessendpoint.py	2016-03-28 13:54:47 +0000
+++ src/maasserver/models/regioncontrollerprocessendpoint.py	2017-04-12 21:37:46 +0000
@@ -20,9 +20,12 @@
 from maasserver.models.cleansave import CleanSave
 from maasserver.models.regioncontrollerprocess import RegionControllerProcess
 from maasserver.models.timestampedmodel import TimestampedModel
-
-
-class RegionControllerProcessEndpoint(CleanSave, TimestampedModel):
+from maasserver.utils.prometheus import ExportModelOperationsMixin
+
+
+class RegionControllerProcessEndpoint(
+        ExportModelOperationsMixin("region_process_endpoint"),
+        CleanSave, TimestampedModel):
     """`RegionControllerProcessEndpoint` is a RPC endpoint on the
     `RegionControllerProcess` one endpoint is created per IP address on the
     `RegionControllerProcess`.

=== modified file 'src/maasserver/models/regionrackrpcconnection.py'
--- src/maasserver/models/regionrackrpcconnection.py	2016-03-28 13:54:47 +0000
+++ src/maasserver/models/regionrackrpcconnection.py	2017-04-12 21:37:46 +0000
@@ -15,9 +15,12 @@
     RegionControllerProcessEndpoint,
 )
 from maasserver.models.timestampedmodel import TimestampedModel
-
-
-class RegionRackRPCConnection(CleanSave, TimestampedModel):
+from maasserver.utils.prometheus import ExportModelOperationsMixin
+
+
+class RegionRackRPCConnection(
+        ExportModelOperationsMixin("rack_rpc_connection"),
+        CleanSave, TimestampedModel):
     """`RegionRackRPCConnection` records a connection between a region
     controller and rack controller within the MAAS communication strucutre.
 

=== modified file 'src/maasserver/models/staticipaddress.py'
--- src/maasserver/models/staticipaddress.py	2017-04-06 18:19:08 +0000
+++ src/maasserver/models/staticipaddress.py	2017-04-12 21:37:46 +0000
@@ -52,6 +52,7 @@
 from maasserver.models.timestampedmodel import TimestampedModel
 from maasserver.utils import orm
 from maasserver.utils.dns import get_ip_based_hostname
+from maasserver.utils.prometheus import ExportModelOperationsMixin
 from netaddr import IPAddress
 from provisioningserver.utils.enum import map_enum_reverse
 
@@ -663,7 +664,8 @@
             ], params=[family])
 
 
-class StaticIPAddress(CleanSave, TimestampedModel):
+class StaticIPAddress(
+        ExportModelOperationsMixin("ip_address"), CleanSave, TimestampedModel):
 
     class Meta(DefaultMeta):
         verbose_name = "Static IP Address"

=== added file 'src/maasserver/utils/prometheus.py'
--- src/maasserver/utils/prometheus.py	1970-01-01 00:00:00 +0000
+++ src/maasserver/utils/prometheus.py	2017-04-12 21:37:46 +0000
@@ -0,0 +1,14 @@
+# Copyright 2017 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Prometheus utilities."""
+
+__all__ = [
+    "ExportModelOperationsMixin"
+]
+
+
+try:
+    from django_prometheus.models import ExportModelOperationsMixin
+except ImportError:
+    ExportModelOperationsMixin = object

=== modified file 'utilities/check-imports'
--- utilities/check-imports	2017-02-17 14:23:04 +0000
+++ utilities/check-imports	2017-04-12 21:37:46 +0000
@@ -244,6 +244,7 @@
     Allow("curtin|curtin.**"),
     Allow("distro_info|distro_info.*"),
     Allow("django|django.**"),
+    Allow("django_prometheus|django_prometheus.**"),
     Allow("docutils.core"),
     Allow("formencode|formencode.**"),
     Allow("jsonschema|jsonschema.**"),

=== modified file 'versions.cfg'
--- versions.cfg	2017-02-17 14:23:04 +0000
+++ versions.cfg	2017-04-12 21:37:46 +0000
@@ -14,6 +14,7 @@
 coverage = 4.3.1
 decorator = 4.0.10
 django-nose = 1.4.2
+django-prometheus = 1.0.8
 extras = 1.0.0
 fixtures = 3.0.0
 hypothesis = 3.6.1
@@ -27,6 +28,7 @@
 pbr = 1.10.0
 pickleshare = 0.7.4
 postgresfixture = 0.4.0
+prometheus-client = 0.0.19
 prompt-toolkit = 1.0.9
 python-mimeparse = 1.6.0
 python-subunit = 1.2.0