Merge ~exsdev/charm-kubernetes-service-checks:add_nodes_readiness_check_w_lint into charm-kubernetes-service-checks:master

[Edin S]

Added Kubernetes API Nagios plugin for node status

Diko's work with linting fixes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:5ed702a19431693f2a0d3e19a3379efdae513e55

No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want jenkins to rebuild you need to trigger it yourself):
https://code.launchpad.net/~dparv/charm-kubernetes-service-checks/+git/charm-kubernetes-service-checks/+merge/402981/+edit-commit-message

https://jenkins.canonical.com/bootstack/job/lp-charm-kubernetes-service-checks-ci/3/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit/74/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/216/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-kubernetes-service-checks-ci/3//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:5ed702a19431693f2a0d3e19a3379efdae513e55
https://jenkins.canonical.com/bootstack/job/lp-charm-kubernetes-service-checks-ci/4/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit/75/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/217/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-kubernetes-service-checks-ci/4//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:5ed702a19431693f2a0d3e19a3379efdae513e55
https://jenkins.canonical.com/bootstack/job/lp-charm-kubernetes-service-checks-ci/5/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit/99/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/240/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-kubernetes-service-checks-ci/5//rebuild

review: Needs Fixing (continuous-integration)

[Xav Paice (xavpaice)]

Please fix the lint error.

lint run-test: commands[1] | black --check --exclude '/(\.eggs|\.git|\.tox|\.venv|\.build|dist|charmhelpers|mod)/' .
would reformat files/plugins/check_kubernetes_api.py
Oh no! 💥 💔 💥
1 file would be reformatted, 9 files would be left unchanged.

review: Needs Fixing

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:98a7d24feb992840654535436f5ee9efc25fb60c
https://jenkins.canonical.com/bootstack/job/lp-charm-kubernetes-service-checks-ci/6/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit/111/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/263/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-kubernetes-service-checks-ci/6//rebuild

review: Approve (continuous-integration)

[James Troup (elmo)]

See comment inline

review: Needs Information

[Edin S (exsdev)]

See reply to comment inline.

[James Troup (elmo)]

Duh, thanks.

diff --git a/files/plugins/check_kubernetes_api.py b/files/plugins/check_kubernetes_api.py
index 6181699..6c31168 100755
--- a/files/plugins/check_kubernetes_api.py
+++ b/files/plugins/check_kubernetes_api.py
@@ -2,6 +2,7 @@
 """NRPE Plugin for checking Kubernetes API."""
 
 import argparse
+import json
 import sys
 
 import urllib3
@@ -67,6 +68,52 @@ def check_kubernetes_health(k8s_address, client_token, disable_ssl):
     return NAGIOS_STATUS_OK, "Kubernetes health 'ok'"
 
 
+def check_kubernetes_nodes(k8s_address, client_token, disable_ssl):
+    """Call <kubernetes-api>/api/v1/nodes endpoint and check each node status.
+
+    :param k8s_address: Address to kube-api-server formatted 'https://<IP>:<PORT>'
+    :param client_token: Token for authenticating with the kube-api
+    :param disable_ssl: Disables SSL Host Key verification
+    """
+    url = k8s_address + "/api/v1/nodes"
+    if disable_ssl:
+        # perform check without SSL verification
+        http = urllib3.PoolManager(cert_reqs="CERT_NONE", assert_hostname=False)
+    else:
+        http = urllib3.PoolManager()
+
+    try:
+        resp = http.request(
+            "GET", url, headers={"Authorization": "Bearer {}".format(client_token)}
+        )
+    except urllib3.exceptions.MaxRetryError as e:
+        return NAGIOS_STATUS_CRITICAL, e
+
+    if resp.status != 200:
+        return (
+            NAGIOS_STATUS_CRITICAL,
+            "Unexpected HTTP Response code ({})".format(resp.status),
+        )
+
+    response_body = json.loads(resp.data)
+    nodes_not_ready = []
+    for item in response_body["items"]:
+        for condition in item["status"]["conditions"]:
+            if condition["type"] == "Ready":
+                node_name = item["metadata"]["name"]
+                if condition["status"] != "True":
+                    nodes_not_ready.append(node_name)
+
+    if nodes_not_ready:
+        nodes = ", ".join(nodes_not_ready)
+        return (
+            NAGIOS_STATUS_CRITICAL,
+            f"Nodes NotReady: {nodes}",
+        )
+
+    return NAGIOS_STATUS_OK, "All Nodes Ready"
+
+
 if __name__ == "__main__":
     parser = argparse.ArgumentParser(
         description="Check Kubernetes API status",
@@ -98,7 +145,7 @@ if __name__ == "__main__":
         help="Client access token for authenticate with the Kubernetes API",
     )
 
-    check_choices = ["health"]
+    check_choices = ["health", "nodes"]
     parser.add_argument(
         "--check",
         dest="check",
@@ -119,7 +166,10 @@ if __name__ == "__main__":
     )
     args = parser.parse_args()
 
-    checks = {"health": check_kubernetes_health}
+    checks = {
+        "health": check_kubernetes_health,
+        "nodes": check_kubernetes_nodes,
+    }
 
     k8s_url = "https://{}:{}".format(args.host, args.port)
     nagios_exit(
@@ -134,9 +184,4 @@ Authorization: Bearer $TOKEN
 Accept: application/json
 Connection: close
 
-GET /api/va/nodes HTTP/1.1
-Authorization: Bearer $TOKEN
-Accept: application/json
-Connection: close
-
 """
diff --git a/lib/lib_kubernetes_service_checks.py b/lib/lib_kubernetes_service_checks.py
index 14db4e6..688f77c 100644
--- a/lib/lib_kubernetes_service_checks.py
+++ b/lib/lib_kubernetes_service_checks.py
@@ -108,9 +108,9 @@ class KSCHelper:
         if not os.path.exists(self.plugins_dir):
             os.makedirs(self.plugins_dir)
 
-        # register basic api health check
+        # register basic api health check and nodes readiness status
         check_k8s_plugin = os.path.join(self.plugins_dir, "check_kubernetes_api.py")
-        for check in ["health"]:
+        for check in ["health", "nodes"]:
             check_command = "{} -H {} -P {} -T {} --check {}".format(
                 check_k8s_plugin,
                 self.kubernetes_api_address,
diff --git a/src/charm.py b/src/charm.py
index 00923fb..066ebff 100755
--- a/src/charm.py
+++ b/src/charm.py
@@ -27,6 +27,7 @@ class KubernetesServiceChecksCharm(CharmBase):
         super().__init__(*args)
         # -- standard hook observation
         self.framework.observe(self.on.install, self.on_install)
+        self.framework.observe(self.on.upgrade_charm, self.on_upgrade_charm)
         self.framework.observe(self.on.start, self.on_start)
         self.framework.observe(self.on.config_changed, self.on_config_changed)
         self.framework.observe(