Merge ~ack/maas:1871423-supervisord-backoff into maas:master

[Alberto Donato]

LP: #1871423 - handle BACKOFF status from supervisord
LP: #1871582 - add "startsecs" to supervisor program stanzas to avoid quick respawns

[MAAS Lander (maas-lander)]

UNIT TESTS
-b 1871423-supervisord-backoff lp:~ack/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: SUCCESS
COMMIT: 61f73ef7c776182f7d4176188be3df44649c90cc

review: Approve

[Björn Tillenius (bjornt)]

Discussed on IRC that mapping BACKOFF to DEAD seems a bit weird. But considering that the systemd states "activating" and "reloading" maps to DEAD as well, I guess it's not a problem.

So +1 for now, but we should rethink how we do service monitoring at some point.

review: Approve

diff --git a/snap/local/tree/usr/share/maas/supervisord.conf.template b/snap/local/tree/usr/share/maas/supervisord.conf.template
index d911830..78f5eac 100644
--- a/snap/local/tree/usr/share/maas/supervisord.conf.template
+++ b/snap/local/tree/usr/share/maas/supervisord.conf.template
@@ -24,6 +24,7 @@ stopasgroup=true
 killasgroup=true
 redirect_stderr=true
 stdout_logfile=%(ENV_SNAP_COMMON)s/log/postgresql.log
+startsecs=10
 {{endif}}
 
 
@@ -36,6 +37,7 @@ killasgroup=true
 redirect_stderr=true
 stdout_logfile=%(ENV_SNAP_COMMON)s/log/regiond.log
 serverurl=unix://%(ENV_SNAP_DATA)s/supervisord/sock
+startsecs=10
 {{endif}}
 
 
@@ -48,6 +50,7 @@ killasgroup=true
 redirect_stderr=true
 stdout_logfile=%(ENV_SNAP_COMMON)s/log/rackd.log
 serverurl=unix://%(ENV_SNAP_DATA)s/supervisord/sock
+startsecs=10
 
 [program:dhcpd]
 process_name=dhcpd
@@ -57,6 +60,7 @@ stopasgroup=true
 killasgroup=true
 redirect_stderr=true
 stdout_logfile=%(ENV_SNAP_COMMON)s/log/dhcpd.log
+startsecs=10
 
 [program:dhcpd6]
 process_name=dhcpd6
@@ -66,6 +70,7 @@ stopasgroup=true
 killasgroup=true
 redirect_stderr=true
 stdout_logfile=%(ENV_SNAP_COMMON)s/log/dhcpd6.log
+startsecs=10
 
 [program:http]
 process_name=http
@@ -74,6 +79,7 @@ stopasgroup=true
 killasgroup=true
 redirect_stderr=true
 stdout_logfile=%(ENV_SNAP_COMMON)s/log/nginx.log
+startsecs=10
 {{endif}}
 
 {{if rackd or regiond}}
@@ -84,6 +90,7 @@ stopasgroup=true
 killasgroup=true
 redirect_stderr=true
 stdout_logfile=%(ENV_SNAP_COMMON)s/log/named.log
+startsecs=10
 
 [program:ntp]
 process_name=ntp
@@ -92,6 +99,7 @@ stopasgroup=true
 killasgroup=true
 redirect_stderr=true
 stdout_logfile=%(ENV_SNAP_COMMON)s/log/chrony.log
+startsecs=10
 
 [program:proxy]
 process_name=proxy
@@ -101,6 +109,7 @@ stopasgroup=true
 killasgroup=true
 redirect_stderr=true
 stdout_logfile=%(ENV_SNAP_COMMON)s/log/proxy.log
+startsecs=10
 
 [program:syslog]
 process_name=syslog
@@ -110,4 +119,5 @@ stopasgroup=true
 killasgroup=true
 redirect_stderr=true
 stdout_logfile=%(ENV_SNAP_COMMON)s/log/rsyslog.log
+startsecs=10
 {{endif}}
diff --git a/src/provisioningserver/utils/service_monitor.py b/src/provisioningserver/utils/service_monitor.py
index 5770f01..5ce21ed 100644
--- a/src/provisioningserver/utils/service_monitor.py
+++ b/src/provisioningserver/utils/service_monitor.py
@@ -243,6 +243,7 @@ class ServiceMonitor:
     # Used to convert the supervisor state to the `SERVICE_STATE` enum.
     SUPERVISOR_TO_STATE = {
         "STARTING": SERVICE_STATE.ON,
+        "BACKOFF": SERVICE_STATE.DEAD,
         "RUNNING": SERVICE_STATE.ON,
         "STOPPED": SERVICE_STATE.OFF,
         "FATAL": SERVICE_STATE.DEAD,
diff --git a/src/provisioningserver/utils/tests/test_service_monitor.py b/src/provisioningserver/utils/tests/test_service_monitor.py
index 8dda950..c06f75e 100644
--- a/src/provisioningserver/utils/tests/test_service_monitor.py
+++ b/src/provisioningserver/utils/tests/test_service_monitor.py
@@ -1248,6 +1248,33 @@ class TestServiceMonitor(MAASTestCase):
         self.assertEqual("Result: exit-code", process_state)
 
     @inlineCallbacks
+    def test___loadSupervisorServiceState_backoff_returns_dead(self):
+        service = make_fake_service(SERVICE_STATE.ON)
+        service_monitor = self.make_service_monitor([service])
+        supervisor_status_output = (
+            dedent(
+                """\
+            %s              BACKOFF   Respawning too fast
+            """
+            )
+            % (service.snap_service_name)
+        )
+
+        mock_execSupervisorServiceAction = self.patch(
+            service_monitor, "_execSupervisorServiceAction"
+        )
+        mock_execSupervisorServiceAction.return_value = (
+            1,
+            supervisor_status_output,
+            "",
+        )
+        active_state, process_state = yield (
+            service_monitor._loadSupervisorServiceState(service)
+        )
+        self.assertEqual(SERVICE_STATE.DEAD, active_state)
+        self.assertEqual("Result: exit-code", process_state)
+
+    @inlineCallbacks
     def test___ensureService_logs_warning_in_mismatch_process_state(self):
         service = make_fake_service(SERVICE_STATE.ON)
         service_monitor = self.make_service_monitor([service])