Merge ~whereisrysmind/charm-kubernetes-service-checks:master into charm-kubernetes-service-checks:master

[Ryan Farrell]

Feedback request for APAC and EMEA teams.

[Giuseppe Petralia (peppepetra)]

I have deployed the bundle xenial plus added the commented relations to get it unblocked. The plugin got created in /usr/local/lib/nagios/plugins/ but it is not executable while i think it should be to be used by the nrpe check. Also i can't see the nrpe check in /etc/nagios/nrpe.d and i can not test if it is working.

Other comments inline.

[Adam Dyess (addyess)]

I could work on this project. I think the style is clear for what's expected. Just a few suggestions below

review: Needs Fixing

[Ryan Farrell (whereisrysmind)]

Addressed each of the inline comments.

[Giuseppe Petralia (peppepetra)]

Fucntional tests were failing.
Once i commented
context=self.config["nagios_context"],
servicegroups=self.config["nagios_servicegroups"],

in nrpe.add_check in lib/lib_kubernetes_service_checks.py

they passed: https://pastebin.canonical.com/p/rG3QnNWY5f/

review: Needs Fixing

7f88877... by Ryan Farrell on 2020-07-17

Removed nagios context and servicegroup params

8205896... by Ryan Farrell on 2020-07-17

Added plugin functional tests

- Added tests confirming that nagios plugins and check configs exist
- Some code lint cleanup

2e6fb0f... by Ryan Farrell on 2020-07-17

Added nagios-nrpe-server restart

Added logic to restart the nrpe service at times when the charm goes
from an unconfigured state to configured.
Cleaned up some code lint.

8172b74... by Ryan Farrell on 2020-07-17

Lint Fixes

a17ae30... by Ryan Farrell on 2020-07-17

Fixed issues with SSL Host Key verification

- Added step to base64 decode of the SSL cert
- Changed plugin default to check SSL host key
- Updated unit tests
- Fixed fixed functional tests

Unmerged commits

a17ae30... by Ryan Farrell on 2020-07-17

Fixed issues with SSL Host Key verification

- Added step to base64 decode of the SSL cert
- Changed plugin default to check SSL host key
- Updated unit tests
- Fixed fixed functional tests

8172b74... by Ryan Farrell on 2020-07-17

Lint Fixes

2e6fb0f... by Ryan Farrell on 2020-07-17

Added nagios-nrpe-server restart

Added logic to restart the nrpe service at times when the charm goes
from an unconfigured state to configured.
Cleaned up some code lint.

8205896... by Ryan Farrell on 2020-07-17

Added plugin functional tests

- Added tests confirming that nagios plugins and check configs exist
- Some code lint cleanup

7f88877... by Ryan Farrell on 2020-07-17

Removed nagios context and servicegroup params

34a5d31... by Ryan Farrell on 2020-07-17

Set executable flag to the check k8s health plugin

7f4d465... by Ryan Farrell on 2020-07-17

Fixed functional tests

Found an issue with the functests where, although relation remove events
triggered by zaza would be honored by the charms quickly, the juju model
needed time to reflect the change. Relation add events would not tigger
events in the charm again until the model was done removing the
relation. Added waits in the functional tests to ensure this occurred
prior to moving to the next test.

650936d... by Ryan Farrell on 2020-07-16

Updated Charm

Addressed comments and concerts from the team including:
- Modified functional and unit tests
- Cleaned up some code around accessing and iterating over dicts
- Added missing nrpe.write() call

3257526... by Ryan Farrell on 2020-07-15

Fix for expected model state

1c0593a... by Ryan Farrell on 2020-07-15

Charm improvements and Bugfixes

- Added some functional tests, bundles and expected states
- Fixed some state issues when relations are departed

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..f04567f
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,25 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Log files
+*.log
+
+.tox/
+.coverage
+
+# vi
+.*.swp
+
+# pycharm
+.idea/
+
+# version data
+repo-info
+
+# reports
+report/*
+
+# coverage
+htmlcov/*
diff --git a/.gitmodules b/.gitmodules
index 0b10c26..2970f9f 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,4 +1,6 @@
 [submodule "mod/operator"]
 	path = mod/operator
 	url = https://github.com/canonical/operator.git
-	branch = 0.7.0
+[submodule "mod/charm-helpers"]
+	path = mod/charm-helpers
+	url = https://github.com/juju/charm-helpers.git
diff --git a/Makefile b/Makefile
new file mode 100644
index 0000000..4dad760
--- /dev/null
+++ b/Makefile
@@ -0,0 +1,78 @@
+PYTHON := /usr/bin/python3
+
+ifndef CHARM_BUILD_DIR
+	CHARM_BUILD_DIR=/tmp/charm-builds
+endif
+
+PROJECTPATH=$(dir $(realpath $(MAKEFILE_LIST)))
+METADATA_FILE="metadata.yaml"
+CHARM_NAME=$(shell cat ${PROJECTPATH}/${METADATA_FILE} | grep -E '^name:' | awk '{print $$2}')
+
+help:
+	@echo "This project supports the following targets"
+	@echo ""
+	@echo " make help - show this text"
+	@echo " make clean - remove unneeded files"
+	@echo " make submodules - make sure that the submodules are up-to-date"
+	@echo " make build - build the charm"
+	@echo " make release - run clean, submodules and build targets"
+	@echo " make lint - run flake8 and black"
+	@echo " make proof - run charm proof"
+	@echo " make unittests - run the tests defined in the unittest subdirectory"
+	@echo " make functional - run the tests defined in the functional subdirectory"
+	@echo " make test - run lint, proof, unittests and functional targets"
+	@echo ""
+
+clean:
+	@echo "Cleaning files"
+	@if [ -d .tox ] ; then rm -r .tox ; fi
+	@if [ -d .pytest_cache ] ; then rm -r .pytest_cache ; fi
+	@if [ -d .idea ] ; then rm -r .idea ; fi
+	@if [ -d .coverage ] ; then rm -r .coverage ; fi
+	@if [ -d report ] ; then rm -r report ; fi
+	@find . -iname __pycache__ -exec rm -r {} +
+	@find . -type f -name "*.py[cod]" -delete
+	@find . -type f -name "*$py.class" -delete
+	@find . -type f -name "*.log" -delete
+	@find . -type f -name "*.swp" -delete
+	@find . -type f -name ".unit-state.db" -delete
+	@echo "Cleaning existing build"
+	@rm -rf ${CHARM_BUILD_DIR}/${CHARM_NAME}
+
+submodules:
+	@echo "Cloning submodules"
+	@git submodule update --init --recursive
+
+build:
+	@echo "Building charm to base directory ${CHARM_BUILD_DIR}/${CHARM_NAME}"
+	@-git describe --tags > ./repo-info
+	@mkdir -p ${CHARM_BUILD_DIR}/${CHARM_NAME}
+	@cp -r ./* ${CHARM_BUILD_DIR}/${CHARM_NAME}
+	@echo "Installing/updating env from requirements.txt"
+	@mkdir -p ${CHARM_BUILD_DIR}/${CHARM_NAME}/env/
+	@if [ -f requirements.txt ] ; then @pip3 install --target=${CHARM_BUILD_DIR}/${CHARM_NAME}/env -r requirements.txt --upgrade ; fi
+
+release: clean submodules build
+	@echo "Charm is built at ${CHARM_BUILD_DIR}/${CHARM_NAME}"
+
+lint:
+	@echo "Running lint checks"
+	@tox -e lint
+
+proof:
+	@echo "Running charm proof"
+	@charm proof
+
+unittests:
+	@echo "Running unit tests"
+	@tox -e unit
+
+functional: build
+	@echo "Executing functional tests in ${CHARM_BUILD_DIR}"
+	@CHARM_BUILD_DIR=${CHARM_BUILD_DIR} tox -e func
+
+test: lint proof unittests functional
+	@echo "Tests completed for charm ${CHARM_NAME}."
+
+# The targets below don't depend on a file
+.PHONY: help submodules clean build release lint proof unittests functional test
\ No newline at end of file
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..1a50bd9
--- /dev/null
+++ b/README.md
@@ -0,0 +1,96 @@
+# kubernetes-service-checks Charm
+
+Overview
+--------
+
+This charm provides Kubernetes Service checks for Nagios
+
+Quickstart
+----------
+
+    juju deploy cs:kubernetes-service-checks
+    juju add-relation kubernetes-service-checks nrpe
+    juju add-relation kubernetes-service-checks:kube-api-endpoint kubernetes-master
+    juju add-relation kuberentes-service-checks:kube-control kuberentes-master
+    
+
+
+
+### Relations
+
+* **kubernetes-master:kube-api-endpoint** - Provides KSC with the kubernetes-api *hostname* and *port*
+
+* **kuberentes-master:kube-control** - Provides KSC with a kuberentes-api *client-token* for authentication
+
+* **nrpe:nrpe-external-master** - Required for nagios; provides additional plugins 
+
+
+**Note:** Future relations with kubernetes-master *may* be changed so that a 
+single relation can provide the K8S api hostname, port, client token and ssl ca
+cert.
+
+### Config Options
+
+**trusted_ssl_ca** *(Optional)* Setting this option enables SSL host 
+certificate authentication in the api checks
+    
+    juju config kubernetes-service-checks trusted_ssl_ca="${KUBERNETES_API_CA}"
+
+
+Service Checks
+--------------
+The plugin *check_kubernetes_api.py* ships with this charm and contains an array of checks for the k8s api health.
+
+```
+check_kubernetes_api.py --help
+usage: check_kubernetes_api.py [-h] [-H HOST] [-P PORT] [-T CLIENT_TOKEN]
+                               [--check health] [-C SSL_CA_PATH]
+
+Check Kubernetes API status
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -H HOST, --host HOST  Hostname or IP of the kube-api-server (default: None)
+  -P PORT, --port PORT  Port of the kube-api-server (default: 6443)
+  -T CLIENT_TOKEN, --token CLIENT_TOKEN
+                        Client access token for authenticate with the
+                        Kubernetes API (default: None)
+  --check health        which check to run (default: health)
+  -C SSL_CA_PATH, --trusted-ca-cert SSL_CA_PATH
+                        String containing path to the trusted CA certificate
+                        (default: None)
+
+```
+
+**health** - This polls the kubernetes-api */healthz* endpoint. Posting a GET to this URL endpoint is expected to 
+return 200 - 'ok' if the api is healthy, otherwise 500.  
+
+
+Other Checks
+------------
+
+**Certificate Expiration:** The *check_http* plugin is shipped with nrpe, and contains a built in cert expiration check. The warning and crit 
+thesholds are configurable:
+
+    juju config kubernetes-service-checks tls_warn_days=90
+    juju config kubernetes-service-checks tls_crit_days=30
+
+Testing
+-------
+
+Juju should be installed and bootstrapped on the system to run functional tests.
+
+
+```
+    export MODEL_SETTINGS=<semicolon-separated list of "juju model-config" settings>
+    make test
+```
+
+NOTE: If you are behind a proxy, be sure to export a MODEL_SETTINGS variable as
+described above. Note that you will need to use the juju-http-proxy, juju-https-proxy, juju-no-proxy 
+and similar settings.
+
+Contact
+-------
+ - Author: **Bootstack Charmers** *<[e-mail]>*
+ - Bug Tracker: [lp:charm-kubernetes-service-checks](https://launchpad.net/charm-kubernetes-service-checks)
diff --git a/config.yaml b/config.yaml
new file mode 100644
index 0000000..8560b6e
--- /dev/null
+++ b/config.yaml
@@ -0,0 +1,38 @@
+options:
+  channel:
+    type: string
+    default: 1.18/stable
+    description: |
+      Snap channel to install kubectl from
+  nagios_context:
+    default: "juju"
+    type: string
+    description: |
+      Used by the nrpe subordinate charms.
+      A string that will be prepended to instance name to set the host name
+      in nagios. So for instance the hostname would be something like:
+          juju-myservice-0
+      If you're running multiple environments with the same services in them
+      this allows you to differentiate between them.
+  nagios_servicegroups:
+    default: ""
+    type: string
+    description: |
+      A comma-separated list of nagios servicegroups.
+      If left empty, the nagios_context will be used as the servicegroup
+  tls_warn_days:
+    type: int
+    default: 60
+    description: |
+      Number of days left for the TLS certificate to expire before Warning.
+  tls_crit_days:
+    type: int
+    default: 30
+    description: |
+      Number of days left for the TLS certificate to expire before alerting Critical.
+  # temporary config setting for trusted SSL CA (see LP1886982)
+  trusted_ssl_ca:
+    type: string
+    default: ""
+    description: |
+      base64 encoded SSL ca cert to use for Kubernetes API client connections.
\ No newline at end of file
diff --git a/files/plugins/check_kubernetes_api.py b/files/plugins/check_kubernetes_api.py
new file mode 100755
index 0000000..7e77fb4
--- /dev/null
+++ b/files/plugins/check_kubernetes_api.py
@@ -0,0 +1,125 @@
+#!/usr/bin/python3
+"""NRPE Plugin for checking Kubernetes API."""
+
+import argparse
+import sys
+
+import urllib3
+
+NAGIOS_STATUS_OK = 0
+NAGIOS_STATUS_WARNING = 1
+NAGIOS_STATUS_CRITICAL = 2
+NAGIOS_STATUS_UNKNOWN = 3
+
+NAGIOS_STATUS = {
+    NAGIOS_STATUS_OK: "OK",
+    NAGIOS_STATUS_WARNING: "WARNING",
+    NAGIOS_STATUS_CRITICAL: "CRITICAL",
+    NAGIOS_STATUS_UNKNOWN: "UNKNOWN",
+}
+
+
+def nagios_exit(status, message):
+    """Return the check status in Nagios preferred format.
+
+    :param status: Nagios Check status code (in [0, 1, 2, 3])
+    :param message: Message describing the status
+    :return: sys.exit("{status_string}: {message}")
+    """
+    assert status in NAGIOS_STATUS, "Invalid Nagios status code"
+    # prefix status name to message
+    output = "{}: {}".format(NAGIOS_STATUS[status], message)
+    print(output)  # nagios requires print to stdout, no stderr
+    sys.exit(status)
+
+
+def check_kubernetes_health(k8s_address, client_token, disable_ssl):
+    """Call <kubernetes-api>/healthz endpoint and check return value is 'ok'.
+
+    :param k8s_address: Address to kube-api-server formatted 'https://<IP>:<PORT>'
+    :param client_token: Token for authenticating with the kube-api
+    :param disable_ssl: Disables SSL Host Key verification
+    """
+    url = k8s_address + "/healthz"
+    if disable_ssl:
+        # perform check without SSL verification
+        http = urllib3.PoolManager(
+            cert_reqs="CERT_NONE",
+            assert_hostname=False
+        )
+    else:
+        http = urllib3.PoolManager()
+
+    try:
+        req = http.request(
+            "GET",
+            url,
+            headers={"Authorization": "Bearer {}".format(client_token)}
+        )
+    except urllib3.exceptions.MaxRetryError as e:
+        return NAGIOS_STATUS_CRITICAL, e
+
+    if req.status != 200:
+        return NAGIOS_STATUS_CRITICAL, "Unexpected HTTP Response code ({})".format(req.status)
+    elif req.data != b"ok":
+        return NAGIOS_STATUS_WARNING, "Unexpected Kubernetes healthz status '{}'".format(req.data)
+    return NAGIOS_STATUS_OK, "Kubernetes health 'ok'"
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="Check Kubernetes API status",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+
+    parser.add_argument(
+        "-H", "--host", dest="host",
+        help="Hostname or IP of the kube-api-server"
+    )
+
+    parser.add_argument(
+        "-P", "--port", dest="port", type=int, default=6443,
+        help="Port of the kube-api-server"
+    )
+
+    parser.add_argument(
+        "-T", "--token", dest="client_token",
+        help="Client access token for authenticate with the Kubernetes API"
+    )
+
+    check_choices = ["health"]
+    parser.add_argument(
+        "--check", dest="check", metavar="|".join(check_choices),
+        type=str, choices=check_choices,
+        default=check_choices[0],
+        help="which check to run")
+
+    parser.add_argument(
+        "-d", "--disable-host-key-check", dest="disable_ssl", default=False,
+        action="store_true", help="Disables Host SSL Key Authentication"
+    )
+    args = parser.parse_args()
+
+    checks = {
+        "health": check_kubernetes_health,
+    }
+
+    k8s_url = "https://{}:{}".format(args.host, args.port)
+    nagios_exit(*checks[args.check](k8s_url,
+                                    args.client_token,
+                                    args.disable_ssl))
+
+"""
+TODO: Future Checks
+
+GET /api/v1/componentstatuses HTTP/1.1
+Authorization: Bearer $TOKEN
+Accept: application/json
+Connection: close
+
+GET /api/va/nodes HTTP/1.1
+Authorization: Bearer $TOKEN
+Accept: application/json
+Connection: close
+
+"""
diff --git a/hooks/install b/hooks/install
new file mode 120000
index 0000000..25b1f68
--- /dev/null
+++ b/hooks/install
@@ -0,0 +1 @@
+../src/charm.py
\ No newline at end of file
diff --git a/lib/.empty b/lib/.empty
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/lib/.empty
diff --git a/lib/charmhelpers b/lib/charmhelpers
new file mode 120000
index 0000000..cedc09e
--- /dev/null
+++ b/lib/charmhelpers
@@ -0,0 +1 @@
+../mod/charm-helpers/charmhelpers
\ No newline at end of file
diff --git a/lib/lib_kubernetes_service_checks.py b/lib/lib_kubernetes_service_checks.py
new file mode 100644
index 0000000..48f36b8
--- /dev/null
+++ b/lib/lib_kubernetes_service_checks.py
@@ -0,0 +1,161 @@
+"""Kubernetes Service Checks Helper Library."""
+import base64
+import logging
+import os
+import subprocess
+
+from charmhelpers.contrib.charmsupport.nrpe import NRPE
+from charmhelpers.core import hookenv, host
+from charmhelpers.fetch import snap
+
+CERT_FILE = "/usr/local/share/ca-certificates/kubernetes-service-checks.crt"
+NAGIOS_PLUGINS_DIR = "/usr/local/lib/nagios/plugins/"
+
+
+class KSCHelper():
+    """Kubernetes Service Checks Helper Class."""
+
+    def __init__(self, config, state):
+        """Initialize the Helper with the charm config and state."""
+        self.config = config
+        self.state = state
+
+    @property
+    def kubernetes_api_address(self):
+        """Get kubernetes api hostname."""
+        return self.state.kube_api_endpoint.get("hostname", None)
+
+    @property
+    def kubernetes_api_port(self):
+        """Get kubernetes api port."""
+        return self.state.kube_api_endpoint.get("port", None)
+
+    @property
+    def kubernetes_client_token(self):
+        """Get kubernetes client token."""
+        try:
+            data = eval(self.state.kube_control.get("creds", "{}"))
+        except SyntaxError:
+            data = {}
+        for creds in data.values():
+            token = creds.get("client_token", None)
+            if token:
+                return token
+        return None
+
+    @property
+    def use_tls_cert(self):
+        """Check if SSL cert is provided for use."""
+        return bool(self._ssl_certificate)
+
+    @property
+    def _ssl_certificate(self):
+        # TODO: Expand this later to take a cert from a relation or from the config.
+        # cert from the relation is to be prioritized
+        ssl_cert = self.config.get("trusted_ssl_ca", None)
+        if ssl_cert:
+            ssl_cert = ssl_cert.strip()
+        return ssl_cert
+
+    @property
+    def ssl_cert_path(self):
+        """Get cert file path."""
+        return CERT_FILE
+
+    @property
+    def plugins_dir(self):
+        """Get nagios plugins directory."""
+        return NAGIOS_PLUGINS_DIR
+
+    def restart_nrpe_service(self):
+        """Restart nagios-nrpe-server service."""
+        host.service_restart('nagios-nrpe-server')
+
+    def update_tls_certificates(self):
+        """Write the trusted ssl certificate to the CERT_FILE."""
+        if self._ssl_certificate:
+            cert_content = base64.b64decode(self._ssl_certificate).decode()
+            try:
+                logging.debug('Writing ssl ca cert to {}'.format(self.ssl_cert_path))
+                with open(self.ssl_cert_path, "w") as f:
+                    f.write(cert_content)
+                subprocess.call(['/usr/sbin/update-ca-certificates'])
+                return True
+            except subprocess.CalledProcessError as e:
+                logging.error(e)
+                return False
+            except PermissionError as e:
+                logging.error(e)
+                return False
+        else:
+            logging.error("Trusted SSL Certificate is not defined")
+            return False
+
+    def configure(self):
+        """Refresh configuration data."""
+        self.update_plugins()
+        self.render_checks()
+
+    def update_plugins(self):
+        """Rsync plugins to the plugin directory."""
+        charm_plugin_dir = os.path.join(hookenv.charm_dir(), "files", "plugins/")
+        host.rsync(charm_plugin_dir, self.plugins_dir, options=["--executability"])
+
+    def render_checks(self):
+        """Render nrpe checks."""
+        nrpe = NRPE()
+        if not os.path.exists(self.plugins_dir):
+            os.makedirs(self.plugins_dir)
+
+        # register basic api health check
+        check_k8s_plugin = os.path.join(self.plugins_dir, "check_kubernetes_api.py")
+        for check in ["health"]:
+            check_command = "{} -H {} -P {} -T {} --check {}".format(
+                check_k8s_plugin,
+                self.kubernetes_api_address,
+                self.kubernetes_api_port,
+                self.kubernetes_client_token,
+                check
+            ).strip()
+            if not self.use_tls_cert:
+                check_command += " -d"
+
+            nrpe.add_check(
+                shortname="k8s_api_{}".format(check),
+                description="Check Kubernetes API ({})".format(check),
+                check_cmd=check_command,
+            )
+
+        # register k8s host certificate expiration check
+        check_http_plugin = "/usr/lib/nagios/plugins/check_http"
+        check_command = "{} -I {} -p {} -C {},{}".format(
+            check_http_plugin,
+            self.kubernetes_api_address,
+            self.kubernetes_api_port,
+            self.config.get("tls_warn_days"),
+            self.config.get("tls_crit_days")
+        ).strip()
+        nrpe.add_check(
+            shortname="k8s_api_cert_expiration",
+            description="Check Kubernetes API ({})".format(check),
+            check_cmd=check_command,
+        )
+        nrpe.write()
+
+    def install_kubectl(self):
+        """Attempt to install kubectl.
+
+        :returns: bool, indicating whether or not successful
+        """
+        # snap retry is excessive
+        snap.SNAP_NO_LOCK_RETRY_DELAY = 0.5
+        snap.SNAP_NO_LOCK_RETRY_COUNT = 3
+        try:
+            channel = self.config.get('channel')
+            snap.snap_install("kubectl",
+                              "--classic",
+                              "--channel={}".format(channel)
+                              )
+            return True
+        except snap.CouldNotAcquireLockException:
+            return False
diff --git a/lib/ops b/lib/ops
new file mode 120000
index 0000000..d934193
--- /dev/null
+++ b/lib/ops
@@ -0,0 +1 @@
+../mod/operator/ops
\ No newline at end of file
diff --git a/metadata.yaml b/metadata.yaml
new file mode 100644
index 0000000..09b26a5
--- /dev/null
+++ b/metadata.yaml
@@ -0,0 +1,26 @@
+name: kubernetes-service-checks
+summary: Kubernetes Services NRPE Checks
+maintainers:
+    - Bootstack Charmers <[e-mail]>
+description: |
+    This charm provides NRPE Checks verifying Kubernetes API accessibility
+    and integrates with Nagios for timely alerting.
+tags:
+    - kubernetes
+    - ops
+    - monitoring
+series:
+    - focal
+    - bionic
+    - xenial
+requires:
+    kube-control:
+        interface: kube-control
+    kube-api-endpoint:
+        interface: http
+provides:
+    nrpe-external-master:
+        interface: nrpe-external-master
+        scope: container
+        optional: true
+subordinate: false
diff --git a/mod/.empty b/mod/.empty
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/mod/.empty
diff --git a/mod/charm-helpers b/mod/charm-helpers
new file mode 160000
index 0000000..4b3602e
--- /dev/null
+++ b/mod/charm-helpers
@@ -0,0 +1 @@
+Subproject commit 4b3602e2bdf101bf58cc808264ec0c8092a67cd0
diff --git a/src/charm.py b/src/charm.py
new file mode 100755
index 0000000..1e0a1ff
--- /dev/null
+++ b/src/charm.py
@@ -0,0 +1,193 @@
+#! /usr/bin/env python3
+# -*- coding: utf-8 -*-
+# vim:fenc=utf-8
+# Copyright © 2020 Bootstack Charmers  [e-mail]
+
+"""Operator Charm main library."""
+# Load modules from lib directory
+import logging
+
+import setuppath  # noqa:F401
+
+from lib_kubernetes_service_checks import KSCHelper  # noqa:I100
+from ops.charm import CharmBase
+from ops.framework import StoredState
+from ops.main import main
+from ops.model import ActiveStatus, BlockedStatus, MaintenanceStatus
+
+
+class Kubernetes_Service_ChecksCharm(CharmBase):  # noqa:N801
+    """Class representing this Operator charm."""
+
+    state = StoredState()
+
+    def __init__(self, *args):
+        """Initialize charm and configure states and events to observe."""
+        super().__init__(*args)
+        # -- standard hook observation
+        self.framework.observe(self.on.install, self.on_install)
+        self.framework.observe(self.on.start, self.on_start)
+        self.framework.observe(self.on.config_changed, self.on_config_changed)
+        self.framework.observe(
+            self.on.kube_api_endpoint_relation_changed,
+            self.on_kube_api_endpoint_relation_changed,
+        )
+        self.framework.observe(
+            self.on.kube_api_endpoint_relation_departed,
+            self.on_kube_api_endpoint_relation_departed
+        )
+        self.framework.observe(
+            self.on.kube_control_relation_changed,
+            self.on_kube_control_relation_changed,
+        )
+        self.framework.observe(
+            self.on.kube_control_relation_departed,
+            self.on_kube_control_relation_departed
+        )
+        self.framework.observe(
+            self.on.nrpe_external_master_relation_joined,
+            self.on_nrpe_external_master_relation_joined
+        )
+        self.framework.observe(
+            self.on.nrpe_external_master_relation_departed,
+            self.on_nrpe_external_master_relation_departed
+        )
+        # -- initialize states --
+        self.state.set_default(
+            installed=False,
+            configured=False,
+            started=False,
+            kube_control={},
+            kube_api_endpoint={},
+            nrpe_configured=False,
+        )
+        self.helper = KSCHelper(self.model.config, self.state)
+
+    def on_install(self, event):
+        """Handle install state."""
+        self.unit.status = MaintenanceStatus("Installing charm software")
+        self.unit.status = MaintenanceStatus("Install complete")
+        logging.info("Install of software complete")
+        self.state.installed = True
+
+    def on_upgrade_charm(self, event):
+        """Handle upgrade and resource updates."""
+        # Re-install for new snaps
+        logging.info("Reinstalling for upgrade-charm hook")
+        self.on_install(event)
+        self.check_charm_status()
+
+    def check_charm_status(self):
+        """Check that required data is available from relations."""
+        # check that relations are configured with expected data
+
+        if not self.helper.kubernetes_api_address or not self.helper.kubernetes_api_port:
+            logging.warning("kube-api-endpoint relation missing or misconfigured")
+            self.unit.status = BlockedStatus("missing kube-api-endpoint relation")
+            return
+        if not self.helper.kubernetes_client_token:
+            logging.warning("kube-control relation missing or misconfigured")
+            self.unit.status = BlockedStatus("missing kube-control relation")
+            return
+        if not self.state.nrpe_configured:
+            logging.warning("nrpe-external-master relation missing or misconfigured")
+            self.unit.status = BlockedStatus("missing nrpe-external-master relation")
+            return
+
+        # Check specific required config values
+        # Set up TLS Certificate
+        if self.helper.use_tls_cert:
+            logging.info("Updating tls certificates")
+            if self.helper.update_tls_certificates():
+                logging.info("TLS Certificates updated successfully")
+            else:
+                logging.error("Failed to update TLS Certificates")
+                self.unit.status = BlockedStatus("update-ca-certificates error. check logs")
+                return
+        else:
+            logging.warn("No trusted_ssl_ca provided, SSL Host Authentication disabled")
+
+        # configure nrpe checks
+        logging.info("Configuring Kubernetes Service Checks")
+        self.helper.configure()
+        if not self.state.configured:
+            logging.info('Reloading nagios-nrpe-server')
+            self.helper.restart_nrpe_service()
+        self.state.configured = True
+        self.unit.status = ActiveStatus("Unit is ready")
+
+    def on_config_changed(self, event):
+        """Handle config changed."""
+        if not self.state.installed:
+            logging.warning("Config changed called before install complete, deferring event: {}.".format(event.handle))
+            self._defer_once(event)
+            return
+        self.check_charm_status()
+
+    def on_start(self, event):
+        """Handle start state."""
+        if not self.state.configured:
+            logging.warning("Start called before configuration complete, deferring event: {}".format(event.handle))
+            event.defer()
+            return
+        self.unit.status = MaintenanceStatus("Starting charm software")
+        self.unit.status = ActiveStatus("Unit is ready")
+        self.state.started = True
+        logging.info("Started")
+
+    def _defer_once(self, event):
+        """Defer the given event, but only once."""
+        notice_count = 0
+        handle = str(event.handle)
+
+        for event_path, _, _ in self.framework._storage.notices(None):
+            if event_path.startswith(handle.split('[')[0]):
+                notice_count += 1
+                logging.debug("Found event: {} x {}".format(event_path, notice_count))
+
+        if notice_count > 1:
+            logging.debug("Not deferring {} notice count of {}".format(handle, notice_count))
+        else:
+            logging.debug("Deferring {} notice count of {}".format(handle, notice_count))
+            event.defer()
+
+    def on_kube_api_endpoint_relation_changed(self, event):
+        """Handle kube_api_endpoint relation changed."""
+        self.unit.status = MaintenanceStatus("Updating K8S Endpoint")
+        self.state.kube_api_endpoint.update(event.relation.data.get(event.unit, {}))
+        self.check_charm_status()
+
+    def on_kube_api_endpoint_relation_departed(self, event):
+        """Handle kube-api-endpoint relation departed."""
+        self.state.configured = False
+        for k in self.state.kube_api_endpoint.keys():
+            self.state.kube_api_endpoint[k] = ""
+        self.check_charm_status()
+
+    def on_kube_control_relation_changed(self, event):
+        """Handle kube-control relation changed."""
+        self.unit.status = MaintenanceStatus("Updating K8S Credentials")
+        self.state.kube_control.update(event.relation.data.get(event.unit, {}))
+        self.check_charm_status()
+
+    def on_kube_control_relation_departed(self, event):
+        """Handle kube-control relation departed."""
+        self.state.configured = False
+        for k in self.state.kube_control.keys():
+            self.state.kube_control[k] = ""
+        self.check_charm_status()
+
+    def on_nrpe_external_master_relation_joined(self, event):
+        """Handle nrpe-external-master relation joined."""
+        self.state.nrpe_configured = True
+        self.check_charm_status()
+
+    def on_nrpe_external_master_relation_departed(self, event):
+        """Handle nrpe-external-master relation departed."""
+        self.state.configured = False
+        self.state.nrpe_configured = False
+        self.check_charm_status()
+
+
+if __name__ == "__main__":
+    main(Kubernetes_Service_ChecksCharm)
diff --git a/src/setuppath.py b/src/setuppath.py
new file mode 100644
index 0000000..769d936
--- /dev/null
+++ b/src/setuppath.py
@@ -0,0 +1,3 @@
+"""Include ./lib in the charm's PATH."""
+import sys
+sys.path.append('lib')
diff --git a/tests/functional/requirements.txt b/tests/functional/requirements.txt
new file mode 100644
index 0000000..b7c9112
--- /dev/null
+++ b/tests/functional/requirements.txt
@@ -0,0 +1 @@
+git+https://github.com/openstack-charmers/zaza.git#egg=zaza
diff --git a/tests/functional/tests/bundles/bionic.yaml b/tests/functional/tests/bundles/bionic.yaml
new file mode 100644
index 0000000..c55ba41
--- /dev/null
+++ b/tests/functional/tests/bundles/bionic.yaml
@@ -0,0 +1,90 @@
+series: bionic
+applications:
+    kubernetes-service-checks:
+        charm: ../../../../
+        num_units: 1
+    containerd:
+        charm: cs:~containers/containerd
+        options:
+            gpu_driver: none
+        resources: {}
+    easyrsa:
+        charm: cs:~containers/easyrsa
+        num_units: 1
+        resources:
+          easyrsa: 5
+    etcd:
+        charm: cs:~containers/etcd
+        num_units: 1
+        options:
+          channel: 3.3/stable
+        resources:
+          core: 0
+          etcd: 3
+          snapshot: 0
+    flannel:
+        charm: cs:~containers/flannel
+        resources:
+          flannel-amd64: 625
+          flannel-arm64: 622
+          flannel-s390x: 609
+    kubernetes-master:
+        charm: cs:~containers/kubernetes-master
+        num_units: 1
+        constraints: cores=4 mem=4G root-disk=16G
+        options:
+          channel: 1.18/stable
+        resources:
+          cdk-addons: 0
+          core: 0
+          kube-apiserver: 0
+          kube-controller-manager: 0
+          kube-proxy: 0
+          kube-scheduler: 0
+          kubectl: 0
+    kubernetes-worker:
+        charm: cs:~containers/kubernetes-worker
+        expose: true
+        num_units: 1
+        constraints: cores=4 mem=4G root-disk=16G
+        options:
+            channel: 1.18/stable
+        resources:
+          cni-amd64: 645
+          cni-arm64: 636
+          cni-s390x: 648
+          core: 0
+          kube-proxy: 0
+          kubectl: 0
+          kubelet: 0
+    nrpe:
+        charm: cs:nrpe
+relations:
+    - - kubernetes-master:kube-api-endpoint
+      - kubernetes-worker:kube-api-endpoint
+    - - kubernetes-service-checks:nrpe-external-master
+      - nrpe:nrpe-external-master
+    - - kubernetes-master:kube-control
+      - kubernetes-worker:kube-control
+    - - kubernetes-master:certificates
+      - easyrsa:client
+    - - etcd:certificates
+      - easyrsa:client
+    - - kubernetes-master:etcd
+      - etcd:db
+    - - kubernetes-worker:certificates
+      - easyrsa:client
+    - - flannel:etcd
+      - etcd:db
+    - - flannel:cni
+      - kubernetes-master:cni
+    - - flannel:cni
+      - kubernetes-worker:cni
+    - - containerd:containerd
+      - kubernetes-worker:container-runtime
+    - - containerd:containerd
+      - kubernetes-master:container-runtime
+    - - kubernetes-service-checks:kube-control
+      - kubernetes-master:kube-control
+    - - kubernetes-service-checks:kube-api-endpoint
+      - kubernetes-master:kube-api-endpoint
\ No newline at end of file
diff --git a/tests/functional/tests/bundles/xenial.yaml b/tests/functional/tests/bundles/xenial.yaml
new file mode 100644
index 0000000..962bdfd
--- /dev/null
+++ b/tests/functional/tests/bundles/xenial.yaml
@@ -0,0 +1,90 @@
+series: xenial
+applications:
+    kubernetes-service-checks:
+        charm: ../../../../
+        num_units: 1
+    containerd:
+        charm: cs:~containers/containerd
+        options:
+            gpu_driver: none
+        resources: {}
+    easyrsa:
+        charm: cs:~containers/easyrsa
+        num_units: 1
+        resources:
+          easyrsa: 5
+    etcd:
+        charm: cs:~containers/etcd
+        num_units: 1
+        options:
+          channel: 3.3/stable
+        resources:
+          core: 0
+          etcd: 3
+          snapshot: 0
+    flannel:
+        charm: cs:~containers/flannel
+        resources:
+          flannel-amd64: 625
+          flannel-arm64: 622
+          flannel-s390x: 609
+    kubernetes-master:
+        charm: cs:~containers/kubernetes-master
+        num_units: 1
+        constraints: cores=4 mem=4G root-disk=16G
+        options:
+          channel: 1.18/stable
+        resources:
+          cdk-addons: 0
+          core: 0
+          kube-apiserver: 0
+          kube-controller-manager: 0
+          kube-proxy: 0
+          kube-scheduler: 0
+          kubectl: 0
+    kubernetes-worker:
+        charm: cs:~containers/kubernetes-worker
+        expose: true
+        num_units: 1
+        constraints: cores=4 mem=4G root-disk=16G
+        options:
+            channel: 1.18/stable
+        resources:
+          cni-amd64: 645
+          cni-arm64: 636
+          cni-s390x: 648
+          core: 0
+          kube-proxy: 0
+          kubectl: 0
+          kubelet: 0
+    nrpe:
+        charm: cs:nrpe
+relations:
+    - - kubernetes-master:kube-api-endpoint
+      - kubernetes-worker:kube-api-endpoint
+    - - kubernetes-service-checks:nrpe-external-master
+      - nrpe:nrpe-external-master
+    - - kubernetes-master:kube-control
+      - kubernetes-worker:kube-control
+    - - kubernetes-master:certificates
+      - easyrsa:client
+    - - etcd:certificates
+      - easyrsa:client
+    - - kubernetes-master:etcd
+      - etcd:db
+    - - kubernetes-worker:certificates
+      - easyrsa:client
+    - - flannel:etcd
+      - etcd:db
+    - - flannel:cni
+      - kubernetes-master:cni
+    - - flannel:cni
+      - kubernetes-worker:cni
+    - - containerd:containerd
+      - kubernetes-worker:container-runtime
+    - - containerd:containerd
+      - kubernetes-master:container-runtime
+    - - kubernetes-service-checks:kube-control
+      - kubernetes-master:kube-control
+    - - kubernetes-service-checks:kube-api-endpoint
+      - kubernetes-master:kube-api-endpoint
diff --git a/tests/functional/tests/kubernetes_service_checks.py b/tests/functional/tests/kubernetes_service_checks.py
new file mode 100644
index 0000000..39dc7ef
--- /dev/null
+++ b/tests/functional/tests/kubernetes_service_checks.py
@@ -0,0 +1,193 @@
+"""Charm Kubernetes Service Checks Functional Tests."""
+import concurrent.futures
+import logging
+import re
+import time
+import unittest
+
+from juju.errors import JujuAPIError
+import zaza.model
+
+
+class TestBase(unittest.TestCase):
+    """Base Class for charm functional tests."""
+
+    @classmethod
+    def setUpClass(cls):
+        """Run setup for tests."""
+        cls.model_name = zaza.model.get_juju_model()
+        cls.application_name = "kubernetes-service-checks"
+
+    def setUp(self):
+        """Set up  functional tests & ensure all relations added."""
+        for local_relation_name, remote_relation_unit in [
+                ("kube-api-endpoint", "kubernetes-master"),
+                ("kube-control", "kubernetes-master"),
+                ("nrpe-external-master", "nrpe")]:
+            logging.info("Adding relation {} with {}".format(local_relation_name,
+                                                             remote_relation_unit))
+            try:
+                zaza.model.add_relation(self.application_name,
+                                        local_relation_name,
+                                        remote_relation_unit,
+                                        self.model_name)
+            except JujuAPIError as e:
+                p = r"^.*cannot\ add\ relation.*already\ exists"
+                if re.search(p, e.message):
+                    pass
+                else:
+                    raise(e)
+        zaza.model.block_until_wl_status_info_starts_with(
+            self.application_name,
+            status="Unit is ready",
+            timeout=200
+        )
+
+
+class TestChecks(TestBase):
+    """Tests for availability and usefulness of nagios checks."""
+
+    expected_checks = ["check_k8s_api_health.cfg",
+                       "check_k8s_api_cert_expiration.cfg"]
+    checks_dir = "/etc/nagios/nrpe.d/"
+    expected_plugins = ["check_kubernetes_api.py"]
+    plugins_dir = "/usr/local/lib/nagios/plugins/"
+
+    # TODO: Need testing around setting the trusted_ssl_ca cert
+    #    - does it get written to /etc/ssl/certs/ca-certificates.crt?
+    #    - does the k8s check plugin see it and use it for verification?
+
+    def test_check_plugins_exist(self):
+        """Verify that kubernetes service checks plugins are found."""
+        fail_messages = []
+        for plugin in self.expected_plugins:
+            pluginpath = self.plugins_dir + plugin
+            response = zaza.model.run_on_unit(
+                "kubernetes-service-checks/0",
+                "[ -f \"{}\" ]".format(pluginpath),
+                model_name=self.model_name,
+                timeout=30
+            )
+            if response["Code"] != "0":
+                fail_messages.append("Missing plugin: {}".format(pluginpath))
+                continue
+
+            # check executable
+            response = zaza.model.run_on_unit(
+                "kubernetes-service-checks/0",
+                "[ -x \"{}\" ]".format(pluginpath),
+                model_name=self.model_name,
+                timeout=30
+            )
+
+            if response["Code"] != "0":
+                fail_messages.append("Plugin not executable: {}".format(pluginpath))
+
+        if fail_messages:
+            self.fail("\n".join(fail_messages))
+
+    def test_checks_exist(self):
+        """Verify that kubernetes service checks nrpe checks exist."""
+        fail_messages = []
+        for check in self.expected_checks:
+            checkpath = self.checks_dir + check
+            response = zaza.model.run_on_unit(
+                "kubernetes-service-checks/0",
+                "[ -f \"{}\" ]".format(checkpath),
+                model_name=self.model_name,
+                timeout=30
+            )
+            if response["Code"] != "0":
+                fail_messages.append("Missing check: {}".format(checkpath))
+        if fail_messages:
+            self.fail("\n".join(fail_messages))
+
+
+class TestRelations(TestBase):
+    """Tests for charm behavior adding and removing relations."""
+
+    def _get_relation_id(self, remote_application, interface_name):
+        return zaza.model.get_relation_id(self.application_name,
+                                          remote_application,
+                                          model_name=self.model_name,
+                                          remote_interface_name=interface_name)
+
+    def test_remove_kube_api_endpoint(self):
+        """Test removing kube-api-endpoint relation."""
+        rel_name = "kube-api-endpoint"
+        remote_app = "kubernetes-master"
+        logging.info("Removing kube-api-endpoint relation")
+
+        zaza.model.remove_relation(self.application_name,
+                                   rel_name,
+                                   remote_app,
+                                   self.model_name)
+        try:
+            zaza.model.block_until_wl_status_info_starts_with(
+                self.application_name,
+                status="missing kube-api-endpoint relation",
+                timeout=180
+            )
+        except concurrent.futures._base.TimeoutError:
+            self.fail("Timed out waiting for Unit to become blocked")
+
+        logging.info("Waiting for relation {} to be destroyed".format(rel_name))
+        timeout = time.time() + 600
+        while self._get_relation_id(remote_app, rel_name) is not None:
+            time.sleep(5)
+            if time.time() > timeout:
+                self.fail("Timed out waiting for the relation {} to be destroyed".format(rel_name))
+
+    def test_remove_kube_control(self):
+        """Test removing kube-control relation."""
+        rel_name = "kube-control"
+        remote_app = "kubernetes-master"
+        logging.info("Removing kube-control relation")
+
+        zaza.model.remove_relation(self.application_name,
+                                   rel_name,
+                                   remote_app,
+                                   self.model_name)
+
+        try:
+            zaza.model.block_until_wl_status_info_starts_with(
+                self.application_name,
+                status="missing kube-control relation",
+                timeout=180
+            )
+        except concurrent.futures._base.TimeoutError:
+            self.fail("Timed out waiting for Unit to become blocked")
+
+        logging.info("Waiting for relation {} to be destroyed".format(rel_name))
+        timeout = time.time() + 600
+        while self._get_relation_id(remote_app, rel_name) is not None:
+            time.sleep(5)
+            if time.time() > timeout:
+                self.fail("Timed out waiting for the relation {} to be destroyed".format(rel_name))
+
+    def test_remove_nrpe_external_master(self):
+        """Test removing nrpe-external-master relation."""
+        rel_name = "nrpe-external-master"
+        remote_app = "nrpe"
+        logging.info("Removing nrpe-external-master relation")
+
+        zaza.model.remove_relation(self.application_name,
+                                   rel_name,
+                                   remote_app,
+                                   self.model_name)
+
+        try:
+            zaza.model.block_until_wl_status_info_starts_with(
+                self.application_name,
+                status="missing nrpe-external-master relation",
+                timeout=180
+            )
+        except concurrent.futures._base.TimeoutError:
+            self.fail("Timed out waiting for Unit to become blocked")
+
+        logging.info("Waiting for relation {} to be destroyed".format(rel_name))
+        timeout = time.time() + 600
+        while self._get_relation_id(remote_app, rel_name) is not None:
+            time.sleep(5)
+            if time.time() > timeout:
+                self.fail("Timed out waiting for the relation {} to be destroyed".format(rel_name))
diff --git a/tests/functional/tests/tests.yaml b/tests/functional/tests/tests.yaml
new file mode 100644
index 0000000..75a9ce8
--- /dev/null
+++ b/tests/functional/tests/tests.yaml
@@ -0,0 +1,33 @@
+tests:
+    - tests.kubernetes_service_checks.TestChecks
+    - tests.kubernetes_service_checks.TestRelations
+target_deploy_status:
+    kubernetes-service-checks:
+        workload-status: blocked
+        workload-status-message: "missing kube-api-endpoint relation"
+    kubernetes-master:
+        workload-status: active
+        workload-status-message: "Kubernetes master running."
+    kubernetes-worker:
+        workload-status: active
+        workload-status-message: "Kubernetes worker running."
+    flannel:
+        workload-status: active
+        workload-status-message: "Flannel subnet"
+    easyrsa:
+        workload-status: active
+        workload-status-message: "Certificate Authority connected."
+    containerd:
+        workload-status: active
+        workload-status-message: "Container runtime available"
+    etcd:
+        workload-status: active
+        workload-status-message: "Healthy with 1 known peer"
+    nrpe:
+        workload-status: active
+        workload-status-message: "ready"
+gate_bundles:
+    - xenial
+    - bionic
+smoke_bundles:
+    - focal
diff --git a/tests/unit/requirements.txt b/tests/unit/requirements.txt
new file mode 100644
index 0000000..b5f9fbd
--- /dev/null
+++ b/tests/unit/requirements.txt
@@ -0,0 +1,5 @@
+mock
+pyyaml
+coverage
+six
+urllib3
diff --git a/tests/unit/setuppath.py b/tests/unit/setuppath.py
new file mode 100644
index 0000000..5e334a3
--- /dev/null
+++ b/tests/unit/setuppath.py
@@ -0,0 +1,5 @@
+"""Include ./lib ./src and ./file/plugins in the tests' PATH."""
+import sys
+sys.path.append('lib')
+sys.path.append('src')
+sys.path.append('files/plugins')
diff --git a/tests/unit/test_charm.py b/tests/unit/test_charm.py
new file mode 100644
index 0000000..93fd21d
--- /dev/null
+++ b/tests/unit/test_charm.py
@@ -0,0 +1,242 @@
+"""Charm unit tests."""
+import os
+from pathlib import Path
+import unittest
+
+
+import mock
+import yaml
+import setuppath  # noqa:F401
+
+from charm import Kubernetes_Service_ChecksCharm  # noqa:I100
+import ops.main
+from ops.testing import Harness
+
+TEST_KUBE_CONTOL_RELATION_DATA = {"creds":
+                                   """{"system:node:juju-62684f-0":
+                                      {"client_token": "DECAFBADBEEF",
+                                       "kubelet_token": "ABCDEF012345",
+                                       "proxy_token": "BADC0FFEEDAD",
+                                       "scope": "kubernetes-worker/0"}
+                                   }"""  # noqa:E127
+                                  }
+TEST_KUBE_API_ENDPOINT_RELATION_DATA = {"hostname": "1.1.1.1",
+                                        "port": "1111"}
+
+
+class TestKubernetes_Service_ChecksCharm(unittest.TestCase):  # noqa:N801
+    """Test Kubernetes Service Checks Charm Code."""
+
+    @classmethod
+    def setUpClass(cls):
+        """Prepare class fixture."""
+        # Stop unit test from calling fchown
+        fchown_patcher = mock.patch("os.fchown")
+        cls.mock_fchown = fchown_patcher.start()
+        chown_patcher = mock.patch("os.chown")
+        cls.mock_chown = chown_patcher.start()
+
+        # Stop charmhelpers host from logging via debug log
+        host_log_patcher = mock.patch("charmhelpers.core.host.log")
+        cls.mock_juju_log = host_log_patcher.start()
+
+        # Stop charmhelpers snap from logging via debug log
+        snap_log_patcher = mock.patch("charmhelpers.fetch.snap.log")
+        cls.mock_snap_log = snap_log_patcher.start()
+
+        charm_logger_patcher = mock.patch("charm.logging")
+        cls.mock_charm_log = charm_logger_patcher.start()
+
+        lib_logger_patcher = mock.patch("lib.lib_kubernetes_service_checks.logging")
+        cls.mock_lib_logger = lib_logger_patcher.start()
+
+        # Prevent charmhelpers from calling systemctl
+        host_service_patcher = mock.patch("charmhelpers.core.host.service_stop")
+        cls.mock_service_stop = host_service_patcher.start()
+        host_service_patcher = mock.patch("charmhelpers.core.host.service_start")
+        cls.mock_service_start = host_service_patcher.start()
+        host_service_patcher = mock.patch("charmhelpers.core.host.service_restart")
+        cls.mock_service_restart = host_service_patcher.start()
+
+        # Setup mock JUJU Environment variables
+        os.environ["JUJU_UNIT_NAME"] = "mock/0"
+        os.environ["JUJU_CHARM_DIR"] = "."
+
+    def setUp(self):
+        """Prepare tests."""
+        self.harness = Harness(Kubernetes_Service_ChecksCharm)
+        # Mock config_get to return default config
+        with open(ops.main._get_charm_dir() / Path("config.yaml"), "r") as config_file:
+            config = yaml.safe_load(config_file)
+        charm_config = {}
+
+        for key, _ in config["options"].items():
+            charm_config[key] = config["options"][key]["default"]
+
+        self.harness._backend._config = charm_config
+
+    def test_harness(self):
+        """Verify harness."""
+        self.harness.begin()
+        self.assertFalse(self.harness.charm.state.installed)
+
+    @mock.patch("charmhelpers.fetch.snap.subprocess.check_call")
+    def test_install(self, mock_snap_subprocess):
+        """Test response to an install event."""
+        mock_snap_subprocess.return_value = 0
+        mock_snap_subprocess.side_effect = None
+
+        self.harness.begin()
+        self.harness.charm.on.install.emit()
+
+        self.assertEqual(self.harness.charm.unit.status.name, "maintenance")
+        self.assertEqual(self.harness.charm.unit.status.message, "Install complete")
+        self.assertTrue(self.harness.charm.state.installed)
+
+    def test_config_changed(self):
+        """Test response to config changed event."""
+        self.harness.set_leader(True)
+        self.harness.populate_oci_resources()
+        self.harness.begin()
+        self.harness.charm.check_charm_status = mock.MagicMock()
+        self.harness.charm.state.installed = True
+        self.harness.charm.on.config_changed.emit()
+        self.harness.charm.check_charm_status.assert_called_once()
+
+    def test_start_not_installed(self):
+        """Test response to start event without install state."""
+        self.harness.begin()
+        self.harness.charm.on.start.emit()
+        self.assertFalse(self.harness.charm.state.started)
+
+    def test_start_not_configured(self):
+        """Test response to start event without configured state."""
+        self.harness.begin()
+        self.harness.charm.state.installed = True
+        self.harness.charm.on.start.emit()
+        self.assertFalse(self.harness.charm.state.started)
+
+    def test_start(self):
+        """Test response to start event."""
+        self.harness.begin()
+        self.harness.charm.state.installed = True
+        self.harness.charm.state.configured = True
+        self.harness.charm.on.start.emit()
+        self.assertTrue(self.harness.charm.state.started)
+        self.assertEqual(self.harness.charm.unit.status.name, "active")
+
+    def test_on_kube_api_endpoint_relation_changed(self):
+        """Check kube-api-endpoint relation changed handling."""
+        relation_id = self.harness.add_relation('kube-api-endpoint', 'kubernetes-master')
+        remote_unit = "kubernetes-master/0"
+        self.harness.begin()
+        self.harness.charm.check_charm_status = mock.MagicMock()
+        self.harness.add_relation_unit(relation_id, remote_unit)
+        self.harness.update_relation_data(relation_id, remote_unit, TEST_KUBE_API_ENDPOINT_RELATION_DATA)
+
+        self.harness.charm.check_charm_status.assert_called_once()
+        self.assertEqual(self.harness.charm.helper.kubernetes_api_address, "1.1.1.1")
+        self.assertEqual(self.harness.charm.helper.kubernetes_api_port, "1111")
+
+    def test_on_kube_control_relation_changed(self):
+        """Check kube-control relation changed handling."""
+        relation_id = self.harness.add_relation('kube-control', 'kubernetes-master')
+        remote_unit = "kubernetes-master/0"
+        self.harness.begin()
+        self.harness.charm.check_charm_status = mock.MagicMock()
+        self.harness.add_relation_unit(relation_id, remote_unit)
+        self.harness.update_relation_data(relation_id, remote_unit, TEST_KUBE_CONTOL_RELATION_DATA)
+
+        self.harness.charm.check_charm_status.assert_called_once()
+        assert self.harness.charm.helper.kubernetes_client_token == "DECAFBADBEEF"
+
+    def test_nrpe_external_master_relation_joined(self):
+        """Check that nrpe.configure is True after nrpe relation joined."""
+        relation_id = self.harness.add_relation('nrpe-external-master', 'nrpe')
+        remote_unit = "nrpe/0"
+        self.harness.begin()
+        self.assertFalse(self.harness.charm.state.nrpe_configured)
+        self.harness.charm.check_charm_status = mock.MagicMock()
+        self.harness.add_relation_unit(relation_id, remote_unit)
+
+        self.harness.charm.check_charm_status.assert_called_once()
+        self.assertTrue(self.harness.charm.state.nrpe_configured)
+
+    @mock.patch("ops.model.RelationData")
+    def test_nrpe_external_master_relation_departed(self, mock_relation_data):
+        """Check that nrpe.configure is False after nrpe relation departed."""
+        mock_relation_data.return_value.__getitem__.return_value = {}
+        self.harness.begin()
+        self.harness.charm.check_charm_status = mock.MagicMock()
+        self.emit("nrpe_external_master_relation_departed")
+        self.harness.charm.check_charm_status.assert_called_once()
+
+        self.assertFalse(self.harness.charm.state.nrpe_configured)
+
+    def test_check_charm_status_kube_api_endpoint_relation_missing(self):
+        """Check that the chatm blocks without kube-api-endpoint relation."""
+        self.harness.begin()
+        self.harness.charm.state.kube_control.update(TEST_KUBE_CONTOL_RELATION_DATA)
+        self.harness.charm.state.nrpe_configured = True
+        self.harness.charm.check_charm_status()
+
+        self.assertFalse(self.harness.charm.state.configured)
+        self.assertEqual(self.harness.charm.unit.status.name, "blocked")
+        self.assertEqual(self.harness.charm.unit.status.message, "missing kube-api-endpoint relation")
+
+    def test_check_charm_status_kube_control_relation_missing(self):
+        """Check that the charm blocks without kube-control relation."""
+        self.harness.begin()
+        self.harness.charm.state.kube_api_endpoint.update(TEST_KUBE_API_ENDPOINT_RELATION_DATA)
+        self.harness.charm.state.nrpe_configured = True
+        self.harness.charm.check_charm_status()
+
+        self.assertFalse(self.harness.charm.state.configured)
+        self.assertEqual(self.harness.charm.unit.status.name, "blocked")
+        self.assertEqual(self.harness.charm.unit.status.message, "missing kube-control relation")
+
+    def test_check_charm_status_nrpe_relation_missing(self):
+        """Check that the charm bloack without nrpe relation."""
+        self.harness.begin()
+        self.harness.charm.state.kube_control.update(TEST_KUBE_CONTOL_RELATION_DATA)
+        self.harness.charm.state.kube_api_endpoint.update(TEST_KUBE_API_ENDPOINT_RELATION_DATA)
+        self.harness.charm.check_charm_status()
+
+        self.assertFalse(self.harness.charm.state.configured)
+        self.assertEqual(self.harness.charm.unit.status.name, "blocked")
+        self.assertEqual(self.harness.charm.unit.status.message, "missing nrpe-external-master relation")
+
+    def test_check_charm_status_configured(self):
+        """Check the charm becomes configured."""
+        self.harness.begin()
+        self.harness.charm.helper.configure = mock.MagicMock()
+        self.harness.charm.state.kube_control.update(TEST_KUBE_CONTOL_RELATION_DATA)
+        self.harness.charm.state.kube_api_endpoint.update(TEST_KUBE_API_ENDPOINT_RELATION_DATA)
+        self.harness.charm.state.nrpe_configured = True
+        self.harness.charm.check_charm_status()
+
+        self.harness.charm.helper.configure.assert_called_once()
+        self.assertTrue(self.harness.charm.state.configured)
+
+    def emit(self, event):
+        """Emit the named hook on the charm."""
+        self.harness.charm.framework.reemit()
+
+        if "_relation_" in event:
+            relation_name = event.split("_relation")[0].replace("_", "-")
+            with mock.patch.dict(
+                "os.environ",
+                {
+                    "JUJU_RELATION": relation_name,
+                    "JUJU_RELATION_ID": "1",
+                    "JUJU_REMOTE_APP": "mock",
+                    "JUJU_REMOTE_UNIT": "mock/0",
+                },
+            ):
+                ops.main._emit_charm_event(self.harness.charm, event)
+        else:
+            ops.main._emit_charm_event(self.harness.charm, event)
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/tests/unit/test_lib_ksc.py b/tests/unit/test_lib_ksc.py
new file mode 100644
index 0000000..b8586bf
--- /dev/null
+++ b/tests/unit/test_lib_ksc.py
@@ -0,0 +1,171 @@
+"""Tests for Kubernetes Service Checks Helper."""
+import base64
+import os
+import subprocess
+from subprocess import CalledProcessError
+import tempfile
+import unittest
+
+from lib import lib_kubernetes_service_checks
+import mock
+import yaml
+
+
+TEST_CERTIFICATE = """-----BEGIN CERTIFICATE-----
+MIIDOzCCAiOgAwIBAgIJAPoOXrIwH+miMA0GCSqGSIb3DQEBCwUAMBgxFjAUBgNV
+BAMMDTEwLjEzMi4yNTEuNjAwHhcNMjAwNzE3MTMzMzI0WhcNMzAwNzE1MTMzMzI0
+WjAYMRYwFAYDVQQDDA0xMC4xMzIuMjUxLjYwMIIBIjANBgkqhkiG9w0BAQEFAAOC
+AQ8AMIIBCgKCAQEAqpYVlmT/eRBhCKHaqXjY6EAzvx5GZY0PhL/YGBl9uF8YQGEF
+F3k3Ec7pyJMIQblmWxdCPd1uNzHU8mwApiuPG9GtYOK+olqgslLsmOU9LTi6KJWX
+x956VxdefXDYvr0B6K/Hdgkb1x//XwvipSV1fZ1MCDIiP/hWKi4CmEq31sVpCBdp
+Uiz3qdCzsiGt0f4kbgIJSVtxhWlNJ5MaCOm7gXafkF8OIUTmWhmPp2gH7pfPzzl1
+glOX2Z41qwPuz7Jbcxx/z/yGjdPeJTQYoqJfpDpCrT2er5xyRf66HqKx9Ld/FiqM
+ZksRwmzF9WvqCBK8WoRmnvFxk1FZPGt6E5gotwIDAQABo4GHMIGEMB0GA1UdDgQW
+BBSUCCmRxb4tKD6w8jZ3hHs4ciFizDBIBgNVHSMEQTA/gBSUCCmRxb4tKD6w8jZ3
+hHs4ciFizKEcpBowGDEWMBQGA1UEAwwNMTAuMTMyLjI1MS42MIIJAPoOXrIwH+mi
+MAwGA1UdEwQFMAMBAf8wCwYDVR0PBAQDAgEGMA0GCSqGSIb3DQEBCwUAA4IBAQBv
+BYwILWI/4dGczqG0hcqt8tW04Oi+7y0HxzeI/oaUq/HKfvCz5a+WhpykMKRDJoaZ
+aejR2Oc7A0OUnenpvMeIiMcUIetM3Q1Gzx0aU+vqUNNaZlooSSbe3z1VK6bUsYDo
+qdKhs+mSyuEticK2SEWjT+ZWpV1rjSd5zRZ/UvC1ZhDNJGZotIIqryQWd3YfYl9l
+7JrdzUVCbxs4ywxNp9/I+MJEiBfMHQx8FWr1M2HvLDAm6NZLfM68y5FClzfGpopV
+0ARirz1AfbS6xUumyXHOH2qH527PUXFdfYGSn+juDG/dRTENYJ3OPAfWdj4ze1qQ
+n3ajLSYPvdyKaztdB1VL
+-----END CERTIFICATE-----"""
+
+
+class TestLibKSCHelper(unittest.TestCase):
+    """Unittest class for Kubernetes Service Checks Helper."""
+
+    @classmethod
+    def setUpClass(cls):
+        """Prepare Class Fixture."""
+        # Load default config
+        with open("./config.yaml") as default_config:
+            cls.config = yaml.safe_load(default_config)
+
+        # set defaults to the config object
+        for key in cls.config["options"]:
+            if "default" in cls.config["options"][key]:
+                cls.config[key] = cls.config["options"][key]["default"]
+
+        # Create test state object
+        class FakeStateObject(object):
+            kube_api_endpoint = {"hostname": "1.1.1.1",
+                                 "port": "1111"}
+            kube_control = {"creds": """{"kube-client": {"client_token": "abcdef0123456789"}}"""}
+            installed = False
+            configured = False
+            started = False
+            nrpe_configured = False
+
+        cls.state = FakeStateObject()
+
+        # Stop unit test from calling fchown
+        fchown_patcher = mock.patch("os.fchown")
+        cls.mock_fchown = fchown_patcher.start()
+        chown_patcher = mock.patch("os.chown")
+        cls.mock_chown = chown_patcher.start()
+
+        # Stop charmhelpers host from logging via debug log
+        host_log_patcher = mock.patch("charmhelpers.core.host.log")
+        cls.mock_juju_log = host_log_patcher.start()
+
+        host_logger_patcher = mock.patch("lib.lib_kubernetes_service_checks.logging")
+        cls.mock_logger = host_logger_patcher.start()
+
+        # Stop charmhelpers snap from logging via debug log
+        snap_log_patcher = mock.patch("charmhelpers.fetch.snap.log")
+        cls.mock_snap_log = snap_log_patcher.start()
+
+        # Setup a tmpdir
+        cls.tmpdir = tempfile.TemporaryDirectory()
+        cls.cert_path = os.path.join(
+                cls.tmpdir.name,
+                "kubernetes-service-checks.crt"
+            )
+
+        lib_kubernetes_service_checks.CERT_FILE = cls.cert_path
+        lib_kubernetes_service_checks.NAGIOS_PLUGINS_DIR = cls.tmpdir.name
+
+    @classmethod
+    def tearDownClass(cls):
+        """Tear down class fixture."""
+        mock.patch.stopall()
+        cls.tmpdir.cleanup()
+
+    def setUp(self):
+        """Prepare test fixture."""
+        self.helper = lib_kubernetes_service_checks.KSCHelper(self.config,
+                                                              self.state)
+
+    def tearDown(self):
+        """Clean up test fixture."""
+        try:
+            os.remove(self.cert_path)
+        except FileNotFoundError:
+            pass
+
+    def test_kube_api_endpoint_properties(self):
+        """Test that hostname and port properties get passed through."""
+        # kube_api_endpoint (relation) -> hostname & port
+        self.assertEqual(self.helper.kubernetes_api_address, "1.1.1.1")
+        self.assertEqual(self.helper.kubernetes_api_port, "1111")
+
+        self.helper.state.kube_api_endpoint = {}
+        self.assertEqual(self.helper.kubernetes_api_address, None)
+        self.assertEqual(self.helper.kubernetes_api_port, None)
+
+    def test_kube_control_endpoint_properties(self):
+        """Test KSCHelper client_token gets passed though."""
+        # kube-control (relation) -> kube client token
+        self.assertEqual(self.helper.kubernetes_client_token, "abcdef0123456789")
+
+        self.helper.state.kube_control = {}
+        self.assertEqual(self.helper.kubernetes_client_token, None)
+
+    @mock.patch("lib.lib_kubernetes_service_checks.subprocess.call")
+    def test_update_tls_certificates(self, mock_subprocess):
+        """Test that SSL certificates get updated."""
+        # returns False when no available trusted_ssl_cert
+        self.assertFalse(self.helper.update_tls_certificates())
+
+        # returns True when subprocess successful
+        self.helper.config["trusted_ssl_ca"] = base64.b64encode(str.encode(TEST_CERTIFICATE))
+        self.assertTrue(self.helper.update_tls_certificates())
+        with open(self.cert_path, "r") as f:
+            self.assertEqual(f.read(), TEST_CERTIFICATE)
+        mock_subprocess.assert_called_once_with(['/usr/sbin/update-ca-certificates'])
+        mock_subprocess.reset_mock()
+
+        # returns false when subprocess hits an exception
+        mock_subprocess.side_effect = CalledProcessError("Command", "Mock Subprocess Call Error")
+        self.assertFalse(self.helper.update_tls_certificates())
+
+    def test_render_checks(self):
+        """Test that NPRE is called to add KSC checks."""
+        # TODO
+        pass
+
+    @mock.patch("charmhelpers.fetch.snap.subprocess.check_call")
+    def test_install_kubectl(self, mock_snap_subprocess):
+        """Test install kubectl snap helper function."""
+        self.assertTrue(self.helper.install_kubectl())
+        channel = self.config.get("channel")
+        mock_snap_subprocess.assert_called_with(["snap",
+                                                 "install",
+                                                 "--classic",
+                                                 "--channel={}".format(channel),
+                                                 "kubectl"], env=os.environ)
+
+    @mock.patch("charmhelpers.fetch.snap.subprocess.check_call")
+    def test_install_snap_failure(self, mock_snap_subprocess):
+        """Test response to a failed install event."""
+        error = subprocess.CalledProcessError("cmd", "Install failed")
+        error.returncode = 1
+        mock_snap_subprocess.return_value = 1
+        mock_snap_subprocess.side_effect = error
+        self.assertFalse(self.helper.install_kubectl())
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/tests/unit/test_plugins.py b/tests/unit/test_plugins.py
new file mode 100644
index 0000000..d229902
--- /dev/null
+++ b/tests/unit/test_plugins.py
@@ -0,0 +1,95 @@
+"""Unit tests for Kubernetes Service Checks NRPE Plugins."""
+import unittest
+
+import check_kubernetes_api
+import mock
+
+
+class TestKSCPlugins(unittest.TestCase):
+    """Test cases for Kubernetes Service Checks NRPE plugins."""
+
+    @mock.patch("check_kubernetes_api.sys.exit")
+    @mock.patch("check_kubernetes_api.print")
+    def test_nagios_exit(self, mock_print, mock_sys_exit):
+        """Test the nagios_exit function."""
+        msg = "Test message"
+        for code, status in check_kubernetes_api.NAGIOS_STATUS.items():
+            expected_output = "{}: {}".format(status, msg)
+            check_kubernetes_api.nagios_exit(code, msg)
+
+            mock_print.assert_called_with(expected_output)
+            mock_sys_exit.assert_called_with(code)
+
+    @mock.patch("check_kubernetes_api.urllib3.PoolManager")
+    @mock.patch("check_kubernetes_api.os.path.exists")
+    def test_kubernetes_health_ssl(self,
+                                   mock_os_path_exists,
+                                   mock_http_pool_manager):
+        """Test the check k8s health function called with expected ssl params."""
+        host_address = "https://1.1.1.1:1111"
+        token = "0123456789abcdef"
+        disable_ssl = True
+
+        mock_http_pool_manager.return_value.status = 200
+        mock_http_pool_manager.return_value.data = b"ok"
+
+        check_kubernetes_api.check_kubernetes_health(
+            host_address,
+            token,
+            disable_ssl
+        )
+        mock_http_pool_manager.assert_called_with(cert_reqs='CERT_NONE', assert_hostname=False)
+
+        disable_ssl = False
+        mock_os_path_exists.return_value = True
+        check_kubernetes_api.check_kubernetes_health(
+            host_address,
+            token,
+            disable_ssl
+        )
+        mock_http_pool_manager.assert_called_with()
+
+    @mock.patch("check_kubernetes_api.urllib3.PoolManager")
+    @mock.patch("check_kubernetes_api.os.path.exists")
+    def test_kubernetes_health_status(self,
+                                      mock_os_path_exists,
+                                      mock_http_pool_manager):
+        """Test kubernetes health function."""
+        host_address = "https://1.1.1.1:1111"
+        token = "0123456789abcdef"
+        ssl_ca = "test/cert/path"
+
+        mock_os_path_exists.return_value = True
+        mock_http_pool_manager.return_value.request.return_value.status = 200
+        mock_http_pool_manager.return_value.request.return_value.data = b"ok"
+
+        # verify status OK
+        status, _ = check_kubernetes_api.check_kubernetes_health(
+            host_address,
+            token,
+            ssl_ca
+        )
+        self.assertEqual(status, check_kubernetes_api.NAGIOS_STATUS_OK)
+        mock_http_pool_manager.return_value.request.assert_called_once_with(
+            "GET",
+            "{}/healthz".format(host_address),
+            headers={"Authorization": "Bearer {}".format(token)}
+        )
+
+        mock_http_pool_manager.return_value.request.return_value.status = 500
+        mock_http_pool_manager.return_value.request.return_value.data = b"ok"
+        status, _ = check_kubernetes_api.check_kubernetes_health(
+            host_address,
+            token,
+            ssl_ca
+        )
+        self.assertEqual(status, check_kubernetes_api.NAGIOS_STATUS_CRITICAL)
+
+        mock_http_pool_manager.return_value.request.return_value.status = 200
+        mock_http_pool_manager.return_value.request.return_value.data = b"not ok"
+        status, _ = check_kubernetes_api.check_kubernetes_health(
+            host_address,
+            token,
+            ssl_ca
+        )
+        self.assertEqual(status, check_kubernetes_api.NAGIOS_STATUS_WARNING)
diff --git a/tox.ini b/tox.ini
new file mode 100644
index 0000000..2d5bb8f
--- /dev/null
+++ b/tox.ini
@@ -0,0 +1,49 @@
+[tox]
+skipsdist = True
+envlist = unit, func
+skip_missing_interpreters = True
+
+[testenv]
+basepython = python3
+setenv =
+  PYTHONPATH = {toxinidir}/lib/:{toxinidir}
+passenv =
+  HOME
+  MODEL_SETTINGS
+  CHARM_BUILD_DIR
+
+[testenv:unit]
+commands = 
+    coverage run -m unittest discover -s {toxinidir}/tests/unit -v
+    coverage report \
+	--omit tests/*,mod/*,.tox/*
+    coverage html \
+	--omit tests/*,mod/*,.tox/*
+deps = -r{toxinidir}/tests/unit/requirements.txt
+
+[testenv:func]
+changedir = {toxinidir}/tests/functional
+commands = functest-run-suite {posargs}
+deps = -r{toxinidir}/tests/functional/requirements.txt
+
+[testenv:lint]
+commands = flake8
+deps =
+    flake8
+    flake8-docstrings
+    flake8-import-order
+    pep8-naming
+    flake8-colors
+
+[flake8]
+exclude =
+    .git,
+    __pycache__,
+    .tox,
+    mod,
+max-line-length = 120
+max-complexity = 10
+import-order-style = google
+
+[isort]
+force_to_top=setuppath