Merge ~cgrabowski/maas:add_rpc_conn_pool_metric into maas:master

[Christian Grabowski]

add metric counting number of times RPC pool scales to max size

[MAAS Lander (maas-lander)]

UNIT TESTS
-b add_rpc_conn_pool_metric lp:~cgrabowski/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas-tester/634/consoleText
COMMIT: 49c5020e79abb31fe768ee724186659b7c0dcfe6

review: Needs Fixing

[Alexsander de Souza (alexsander-souza)]

+1

review: Approve

[MAAS Lander (maas-lander)]

UNIT TESTS
-b add_rpc_conn_pool_metric lp:~cgrabowski/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas-tester/636/consoleText
COMMIT: 72e51acb143aefc89da985a09083eefe5c37e0fb

review: Needs Fixing

[MAAS Lander (maas-lander)]

UNIT TESTS
-b add_rpc_conn_pool_metric lp:~cgrabowski/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas-tester/648/consoleText
COMMIT: ecff69b7c7dc52bb947ef85f40dc7e6f4c2f66d4

review: Needs Fixing

[Anton Troyanov (troyanov)]

+1

review: Approve

[Christian Grabowski (cgrabowski)]

jenkins: !test

[MAAS Lander (maas-lander)]

UNIT TESTS
-b add_rpc_conn_pool_metric lp:~cgrabowski/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: SUCCESS
COMMIT: ecff69b7c7dc52bb947ef85f40dc7e6f4c2f66d4

review: Approve

[MAAS Lander (maas-lander)]

LANDING
-b add_rpc_conn_pool_metric lp:~cgrabowski/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: FAILED BUILD
LOG: http://maas-ci.internal:8080/job/maas-tester/674/consoleText

diff --git a/src/provisioningserver/prometheus/metrics.py b/src/provisioningserver/prometheus/metrics.py
index 8f117ff..bc2e6bb 100644
--- a/src/provisioningserver/prometheus/metrics.py
+++ b/src/provisioningserver/prometheus/metrics.py
@@ -108,6 +108,15 @@ METRICS_DEFINITIONS = [
         "maas_lxd_fetch_machine_failure",
         "failures for fetching LXD machines",
     ),
+    MetricDefinition(
+        "Counter",
+        "maas_rpc_pool_exhaustion_count",
+        """
+        counts the number of occurances of the RPC
+        connection pool allocate its maxmimum number
+        of connections
+        """,
+    ),
     # Common metrics
     *node_metrics_definitions(),
 ]
diff --git a/src/provisioningserver/rpc/connectionpool.py b/src/provisioningserver/rpc/connectionpool.py
index 3acaf08..55b00cb 100644
--- a/src/provisioningserver/rpc/connectionpool.py
+++ b/src/provisioningserver/rpc/connectionpool.py
@@ -8,6 +8,7 @@ import random
 from twisted.internet.defer import inlineCallbacks
 from twisted.internet.endpoints import connectProtocol, TCP6ClientEndpoint
 
+from provisioningserver.prometheus.metrics import PROMETHEUS_METRICS
 from provisioningserver.rpc import exceptions
 
 
@@ -80,6 +81,7 @@ class ConnectionPool:
     def stage_connection(self, eventloop, connection):
         self.try_connections[eventloop] = connection
 
+    @PROMETHEUS_METRICS.failure_counter("maas_rpc_pool_exhaustion_count")
     @inlineCallbacks
     def scale_up_connections(self):
         for ev, ev_conns in self.connections.items():