Merge ~xavpaice/charm-hw-health:bug/1945151 into charm-hw-health:master

[Xav Paice]

    Retry check_ipmi_sensor on failure

    Alters the cron task which runs check_ipmi_sensor to sleep for a second
    then retry, to filter out intermittent issues where some BMCs show
    errors then immediately recover.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:ee25cd55b2082f585b561f7441161ae6b734bcef
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/35/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit/183/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/528/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/35//rebuild

review: Approve (continuous-integration)

[Xav Paice (xavpaice)]

FWIW, I believe the reason we use cron for this task rather than simply running the check from NRPE is that in some cases the task takes longer than NRPE's 10 seconds. This change therefore simply replicates a similar, though faster, retry that would normally be performed by Nagios if there's an error reported.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision 1835e401fc8723363e818e802972794a3c3d35ac

diff --git a/src/files/ipmi/cron_ipmi_sensors.py b/src/files/ipmi/cron_ipmi_sensors.py
index 5b53f7d..f270cbb 100644
--- a/src/files/ipmi/cron_ipmi_sensors.py
+++ b/src/files/ipmi/cron_ipmi_sensors.py
@@ -4,6 +4,7 @@
 import os
 import subprocess
 import sys
+from time import sleep
 
 CHECK_IPMI_PID = "/var/run/nagios/check_ipmi_sensors.pid"
 OUTPUT_FILE = "/var/lib/nagios/ipmi_sensors.out"
@@ -14,6 +15,7 @@ NAGIOS_ERRORS = {
     2: "CRITICAL",
     3: "UNKNOWN",
 }
+RETRIES = 3
 
 
 def write_output_file(output):
@@ -51,14 +53,18 @@ def gather_metrics():
     cmdline = [CMD]
     if len(sys.argv) > 1:
         cmdline.extend(sys.argv[1:])
-    try:
-        output = subprocess.check_output(cmdline).decode("utf8")
-        write_output_file(output)
-    except subprocess.CalledProcessError as error:
-        output = error.stdout.decode(errors="ignore")
-        write_output_file("{}: {}".format(NAGIOS_ERRORS[error.returncode], output))
-    except PermissionError as error:
-        write_output_file("UNKNOWN: {}".format(error))
+    for _ in range(RETRIES):
+        try:
+            output = subprocess.check_output(cmdline).decode("utf8")
+            break
+        except subprocess.CalledProcessError as error:
+            result = error.stdout.decode(errors="ignore")
+            output = "{}: {}".format(NAGIOS_ERRORS[error.returncode], result)
+            sleep(1)
+        except PermissionError as error:
+            output = "UNKNOWN: {}".format(error)
+            break
+    write_output_file(output)
 
     # remove pid reference
     os.remove(CHECK_IPMI_PID)