Merge charm-hw-health:xavpaice/bug/1924863 into charm-hw-health:master

[Adam Dyess]

Add two actions which will list the cron.d hwhealth tasks, and allow the juju user to run them.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:2066f776d875831e64aca6e65f8549bc82d61ef3
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/25/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit/170/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/498/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/25//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:3588efaddbe75a440f5f484222f472cb4e11abdc
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/26/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit/171/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/499/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/26//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:8adf0201c35fca527af2d9670e470063a94b2338
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/27/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit/172/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/500/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/27//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:8adf0201c35fca527af2d9670e470063a94b2338
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/28/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit/173/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/501/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/28//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[Xav Paice (xavpaice)]

7cda5c4918aad62c0811d210b779ae293df56bea got missed, the branch needed a rebase. I've done that, and re-pushed.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:c825549842ed5a4cdb054c4fccac54d80ed37a5c
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/29/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit/174/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/502/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/29//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:c50d75450c9d60d87e3539d3143a00319e2fc3ca
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/30/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit/175/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/503/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/30//rebuild

review: Approve (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:c50d75450c9d60d87e3539d3143a00319e2fc3ca
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/31/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit/176/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/504/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/31//rebuild

review: Approve (continuous-integration)

[Diko Parvanov (dparv)]

some comments below

review: Needs Fixing

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:9b2f45b115868c210b552bd746e666e14f0528ef
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/32/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit/180/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/510/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/32//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:687094c9871fc5433e7deaded75ff03f9c2c12ca
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/33/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit/181/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/511/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/33//rebuild

review: Approve (continuous-integration)

[James Troup (elmo)]

See comments inline. I consider the following to be blocking: 1) '_*' glob, 2) either undocument 'all' as an argument to run-cron-action or implement it. The other things are more subjective/not blocking.

review: Needs Fixing

[Adam Dyess (addyess)]

I incorporated @james' comments here to be more clear about how to invoke the actions.
I also ACTUALLY RAN the charm code within a real hw-health space to prove out the actions do as labeled on the box.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[James Troup (elmo)]

LGTM, thanks!

review: Approve

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision 00b40fba4d083ca0c7ae99afbdddef7eb27b05a9

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:359b372eedf8b561ee71639bcc86a798c68d8570
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/34/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit/182/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/522/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-hw-health-ci/34//rebuild

review: Approve (continuous-integration)

diff --git a/src/actions.yaml b/src/actions.yaml
index 888711d..ccd3be5 100644
--- a/src/actions.yaml
+++ b/src/actions.yaml
@@ -26,3 +26,14 @@ ack-sel:
 unack-sel:
   description: |
     Unset the date filter set by ack-sel action.
+"list-cron-actions":
+  "description": |
+    List all hw-health installed cron actions
+"run-cron-action":
+  "description": |
+    Run some or all of the the hw-health cron actions.
+  "params":
+    "entries":
+      "type": "string"
+      "description": "Optional comma separated list of cron actions to run."
+  "required": []
\ No newline at end of file
diff --git a/src/actions/actions.py b/src/actions/actions.py
index 4424545..62b2b50 100755
--- a/src/actions/actions.py
+++ b/src/actions/actions.py
@@ -16,6 +16,9 @@
 """Define hw-health charm actions."""
 
 import os
+import pathlib
+import re
+import shlex
 import subprocess
 import sys
 from datetime import datetime
@@ -27,6 +30,9 @@ from hwhealth import tools
 
 
 IPMI_SEL = "/usr/sbin/ipmi-sel"
+CRON_DIR = pathlib.Path("/etc") / "cron.d"
+RUN_PARTS_RE = re.compile("^[a-zA-Z0-9_-]+$")  # run-parts(8)
+CRON_RE = re.compile(r"^\S+\s+\S+\s+\S+\s+\S+\s+\S+\s+(\S+)\s+(.*)")
 
 
 def clear_sel():
@@ -126,6 +132,65 @@ def unack_sel():
         action_fail("Action failed with {}".format(e))
 
 
+def _get_cron_actions():
+    crond_files = CRON_DIR.glob("hwhealth_*")
+    return {
+        path.stem.split("_", 1)[-1]
+        for path in crond_files
+        if RUN_PARTS_RE.match(path.name)
+    }
+
+
+def list_cron_actions():
+    actions = _get_cron_actions()
+    message = "Cron Actions: {}".format(",".join(sorted(actions)))
+    log("Action list-cron-actions completed\n" + message)
+    action_set({"message": message})
+
+
+def _run_job_file(fname):
+    with fname.open() as f:
+        job_lines = f.readlines()
+        # match jobs according to crontab(5) man
+        # blank, space or tab are all ignorable
+        jobs = [m.groups() for m in (CRON_RE.match(line) for line in job_lines) if m]
+        outs = []
+        for who, cmd in jobs:
+            sudo_cmd = ["sudo", "-u", who] + shlex.split(cmd)
+            try:
+                out = subprocess.check_output(sudo_cmd, stderr=subprocess.STDOUT)
+            except subprocess.CalledProcessError as e:
+                action_fail("{}: {}\n\t{}".format(who, cmd, str(e)))
+                out = e.output + e.stderr
+            outs.append((sudo_cmd, out))
+        return outs
+
+
+def run_cron_action():
+    cron_entries = {entry for entry in action_get("entries").split(",") if entry}
+    cron_actions = _get_cron_actions()
+    to_run = cron_entries or cron_actions  # if nothing specified, run all
+    wont_run = to_run - cron_actions  # any entries not capable of being run?
+    to_run -= wont_run
+
+    results = [
+        "{}\n\t{}".format(" ".join(cmd), result)
+        for each in to_run
+        if each in cron_actions
+        for cmd, result in _run_job_file(CRON_DIR / "hwhealth_{}".format(each))
+    ]
+    response = {}
+    if results:
+        response["message"] = "\n".join(results)
+
+    if wont_run:
+        # skipping the won't runs b/c they were not listed as a cron action
+        response["skipped"] = "unavailable actions: {}".format(
+            ",".join(sorted(wont_run))
+        )
+    action_set(response)
+
+
 # A dictionary of all the defined actions to callables (which take
 # parsed arguments).
 ACTIONS = {
@@ -133,6 +198,8 @@ ACTIONS = {
     "show-sel": show_sel,
     "ack-sel": ack_sel,
     "unack-sel": unack_sel,
+    "list-cron-actions": list_cron_actions,
+    "run-cron-action": run_cron_action,
 }
 
 
diff --git a/src/actions/list-cron-actions b/src/actions/list-cron-actions
new file mode 120000
index 0000000..405a394
--- /dev/null
+++ b/src/actions/list-cron-actions
@@ -0,0 +1 @@
+actions.py
\ No newline at end of file
diff --git a/src/actions/run-cron-action b/src/actions/run-cron-action
new file mode 120000
index 0000000..405a394
--- /dev/null
+++ b/src/actions/run-cron-action
@@ -0,0 +1 @@
+actions.py
\ No newline at end of file
diff --git a/src/tests/unit/test_actions.py b/src/tests/unit/test_actions.py
index 3a3f4f2..15ee573 100644
--- a/src/tests/unit/test_actions.py
+++ b/src/tests/unit/test_actions.py
@@ -12,13 +12,21 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import pathlib
 import subprocess
 import sys
+import tempfile
 import unittest
 import unittest.mock as mock
 
 sys.path.append(".")
-from actions.actions import clear_sel, show_sel  # noqa:E402
+from actions.actions import (  # noqa:E402
+    clear_sel,
+    list_cron_actions,
+    _run_job_file,
+    run_cron_action,
+    show_sel,
+)
 
 
 class ClearSelTestCase(unittest.TestCase):
@@ -170,3 +178,120 @@ class ShowSelTestCase(unittest.TestCase):
             "Action failed with Command '['bogus-cmd']' "
             "returned non-zero exit status 1.",
         )
+
+
+class ListCronTestCase(unittest.TestCase):
+    @mock.patch("actions.actions.action_set")
+    @mock.patch("actions.actions.log")
+    @mock.patch("actions.actions.CRON_DIR")
+    def test_list_cron_jobs(self, mock_cron_dir, mock_log, mock_action_set):
+        mock_cron_dir.glob.return_value = [
+            pathlib.Path("/path/to/hwhealth_ipmi"),
+            pathlib.Path("/path/to/hwhealth_test"),
+        ]
+
+        list_cron_actions()
+        mock_cron_dir.glob.assert_called_once_with("hwhealth_*")
+        ui_response = "Cron Actions: ipmi,test"
+        mock_log.assert_called_once_with(
+            "Action list-cron-actions completed\n{}".format(ui_response)
+        )
+        mock_action_set.assert_called_once_with({"message": ui_response})
+
+    @mock.patch("subprocess.check_output")
+    def test_run_good_job_file(self, mock_check_output):
+        test_file = pathlib.Path(__file__).parent / "test_run_good_job_file.cron"
+        mock_check_output.side_effect = [
+            "result #1",
+            "result #2",
+        ]
+        job_output = _run_job_file(test_file)
+        assert job_output == [
+            (
+                [
+                    "sudo",
+                    "-u",
+                    "nagios",
+                    "/run/my/command",
+                    "with",
+                    "some",
+                    "--arguments",
+                ],
+                "result #1",
+            ),
+            (
+                [
+                    "sudo",
+                    "-u",
+                    "root",
+                    "/run/my/command",
+                    "with",
+                    "some",
+                    "--other-arguments",
+                ],
+                "result #2",
+            ),
+        ]
+
+    @mock.patch("actions.actions.action_fail")
+    @mock.patch("subprocess.check_output")
+    def test_run_job_file_failure(self, mock_check_output, mock_action_fail):
+        with tempfile.TemporaryDirectory() as tempdirname:
+            test_file = pathlib.Path(tempdirname) / "test"
+            with open(test_file, "w") as fp:
+                fp.write("* * * * * root /run/my/command with some --other-arguments")
+
+            mock_check_output.side_effect = [
+                subprocess.CalledProcessError(
+                    cmd="sudo",
+                    returncode=-1,
+                    output="No ",
+                    stderr="Way",
+                ),
+            ]
+            job_output = _run_job_file(test_file)
+            assert job_output == [
+                (
+                    [
+                        "sudo",
+                        "-u",
+                        "root",
+                        "/run/my/command",
+                        "with",
+                        "some",
+                        "--other-arguments",
+                    ],
+                    "No Way",
+                )
+            ]
+            mock_action_fail.assert_called_with(
+                "root: /run/my/command with some --other-arguments\n\t"
+                "Command 'sudo' died with <Signals.SIGHUP: 1>."
+            )
+
+    @mock.patch("actions.actions._run_job_file")
+    @mock.patch("actions.actions._get_cron_actions")
+    @mock.patch("actions.actions.action_get")
+    @mock.patch("actions.actions.action_set")
+    def test_run_cron_action(
+        self,
+        mock_action_set,
+        mock_action_get,
+        mock_get_cron_actions,
+        mock_run_job_file,
+    ):
+        mock_action_get.return_value = "test,garbage"
+        mock_get_cron_actions.return_value = {"test"}
+        mock_run_job_file.return_value = [
+            (["sudo", "-u", "root", "/run/my/cmd"], "works!")
+        ]
+        run_cron_action()
+        mock_action_set.assert_called_once_with(
+            {
+                "message": "sudo -u root /run/my/cmd\n\tworks!",
+                "skipped": "unavailable actions: garbage",
+            }
+        )
+        mock_run_job_file.assert_called_once_with(
+            pathlib.Path("/etc/cron.d/hwhealth_test")
+        )
diff --git a/src/tests/unit/test_run_good_job_file.cron b/src/tests/unit/test_run_good_job_file.cron
new file mode 100644
index 0000000..872e4a6
--- /dev/null
+++ b/src/tests/unit/test_run_good_job_file.cron
@@ -0,0 +1,6 @@
+# Cron Comments
+ lines starting with spaces are comments
+	so are lines starting with tab
+SHELL = /bin/sh
+* * * * * nagios /run/my/command with some --arguments
+*	*	*	*	*	root	/run/my/command with some --other-arguments