Merge ~whereisrysmind/charm-kubernetes-service-checks:master into charm-kubernetes-service-checks:master

[Ryan Farrell]

Initial charm for Kubernetes Service Checks

Currently 2 NRPE checks:
  - check_http for cert expirations
  - check_k8s_api_health for api responding

3 Relations required:
  - nrpe:nrpe-external-master
  - kubernetes-master:kube-api-endpoint
  - kubernetes-master:kube-control

Charm will block if the relations are not configured or if required data
is missing.

+ Unit tests (passing)
+ Functional tests (passing)
+ lint & black (passing)

Initial charm for Kubernetes Service Checks

[Giuseppe Petralia (peppepetra)]

Generally code looks good.

The only comment I have is on the self.state.configured. It is not changed to False when a config changed is happening or a relation has joined, changed or departed (it is set to false only on departed), while it should be to reflect that this state is controlling whether check_charm_status should reconfigure or not the unit.

Also the self.helper.configure() is called in check_charm_status without checking if this state is true or false, while seems that this should be called only if configured is false.

Same for self.helper.update_tls_certificates(). This is needed only on initial deploy (and configured is False at the beginning) or later if a config_changed has occurred as the certificate can be updated only on a config changed at this stage.

Maybe i am not reading this state correctly. In this case more information are needed.

review: Needs Information

5e57fec... by Ryan Farrell on 2020-07-20

Edited when ssl configuration occurs

[Giuseppe Petralia (peppepetra)]

All tests passes. `make test` is blocked by charm proof that is testing things that are not valid for operator charm. I would suggest adding @-charm proof to ignore any error.

I am +1 to merge it.

Thanks.

review: Approve

Unmerged commits

5e57fec... by Ryan Farrell on 2020-07-20

Edited when ssl configuration occurs

003175a... by Ryan Farrell on 2020-07-20

Fixed charm configured flag handling

51c4121... by Ryan Farrell on 2020-07-20

Set required args in nrpe plugin

6f49451... by Ryan Farrell on 2020-07-19

Blackened code

05632f6... by Ryan Farrell on 2020-07-17

Replaced eval with json.loads

184e030... by Ryan Farrell on 2020-07-17

Added focal functional test bundle

296fbef... by Ryan Farrell on 2020-07-17

Some cleanup

- Merge some docstrings with neighboring comments
- Added Black section to Makefile
- Removed repeated set status calls in the charm

a17ae30... by Ryan Farrell on 2020-07-17

Fixed issues with SSL Host Key verification

- Added step to base64 decode of the SSL cert
- Changed plugin default to check SSL host key
- Updated unit tests
- Fixed fixed functional tests

8172b74... by Ryan Farrell on 2020-07-17

Lint Fixes

2e6fb0f... by Ryan Farrell on 2020-07-17

Added nagios-nrpe-server restart

Added logic to restart the nrpe service at times when the charm goes
from an unconfigured state to configured.
Cleaned up some code lint.

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..f04567f
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,25 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Log files
+*.log
+
+.tox/
+.coverage
+
+# vi
+.*.swp
+
+# pycharm
+.idea/
+
+# version data
+repo-info
+
+# reports
+report/*
+
+# coverage
+htmlcov/*
diff --git a/.gitmodules b/.gitmodules
index 0b10c26..2970f9f 100644
--- a/.gitmodules
+++ b/.gitmodules
@@ -1,4 +1,6 @@
 [submodule "mod/operator"]
 	path = mod/operator
 	url = https://github.com/canonical/operator.git
-	branch = 0.7.0
+[submodule "mod/charm-helpers"]
+	path = mod/charm-helpers
+	url = https://github.com/juju/charm-helpers.git
diff --git a/Makefile b/Makefile
new file mode 100644
index 0000000..4dad760
--- /dev/null
+++ b/Makefile
@@ -0,0 +1,78 @@
+PYTHON := /usr/bin/python3
+
+ifndef CHARM_BUILD_DIR
+	CHARM_BUILD_DIR=/tmp/charm-builds
+endif
+
+PROJECTPATH=$(dir $(realpath $(MAKEFILE_LIST)))
+METADATA_FILE="metadata.yaml"
+CHARM_NAME=$(shell cat ${PROJECTPATH}/${METADATA_FILE} | grep -E '^name:' | awk '{print $$2}')
+
+help:
+	@echo "This project supports the following targets"
+	@echo ""
+	@echo " make help - show this text"
+	@echo " make clean - remove unneeded files"
+	@echo " make submodules - make sure that the submodules are up-to-date"
+	@echo " make build - build the charm"
+	@echo " make release - run clean, submodules and build targets"
+	@echo " make lint - run flake8 and black"
+	@echo " make proof - run charm proof"
+	@echo " make unittests - run the tests defined in the unittest subdirectory"
+	@echo " make functional - run the tests defined in the functional subdirectory"
+	@echo " make test - run lint, proof, unittests and functional targets"
+	@echo ""
+
+clean:
+	@echo "Cleaning files"
+	@if [ -d .tox ] ; then rm -r .tox ; fi
+	@if [ -d .pytest_cache ] ; then rm -r .pytest_cache ; fi
+	@if [ -d .idea ] ; then rm -r .idea ; fi
+	@if [ -d .coverage ] ; then rm -r .coverage ; fi
+	@if [ -d report ] ; then rm -r report ; fi
+	@find . -iname __pycache__ -exec rm -r {} +
+	@find . -type f -name "*.py[cod]" -delete
+	@find . -type f -name "*$py.class" -delete
+	@find . -type f -name "*.log" -delete
+	@find . -type f -name "*.swp" -delete
+	@find . -type f -name ".unit-state.db" -delete
+	@echo "Cleaning existing build"
+	@rm -rf ${CHARM_BUILD_DIR}/${CHARM_NAME}
+
+submodules:
+	@echo "Cloning submodules"
+	@git submodule update --init --recursive
+
+build:
+	@echo "Building charm to base directory ${CHARM_BUILD_DIR}/${CHARM_NAME}"
+	@-git describe --tags > ./repo-info
+	@mkdir -p ${CHARM_BUILD_DIR}/${CHARM_NAME}
+	@cp -r ./* ${CHARM_BUILD_DIR}/${CHARM_NAME}
+	@echo "Installing/updating env from requirements.txt"
+	@mkdir -p ${CHARM_BUILD_DIR}/${CHARM_NAME}/env/
+	@if [ -f requirements.txt ] ; then @pip3 install --target=${CHARM_BUILD_DIR}/${CHARM_NAME}/env -r requirements.txt --upgrade ; fi
+
+release: clean submodules build
+	@echo "Charm is built at ${CHARM_BUILD_DIR}/${CHARM_NAME}"
+
+lint:
+	@echo "Running lint checks"
+	@tox -e lint
+
+proof:
+	@echo "Running charm proof"
+	@charm proof
+
+unittests:
+	@echo "Running unit tests"
+	@tox -e unit
+
+functional: build
+	@echo "Executing functional tests in ${CHARM_BUILD_DIR}"
+	@CHARM_BUILD_DIR=${CHARM_BUILD_DIR} tox -e func
+
+test: lint proof unittests functional
+	@echo "Tests completed for charm ${CHARM_NAME}."
+
+# The targets below don't depend on a file
+.PHONY: help submodules clean build release lint proof unittests functional test
\ No newline at end of file
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..1a50bd9
--- /dev/null
+++ b/README.md
@@ -0,0 +1,96 @@
+# kubernetes-service-checks Charm
+
+Overview
+--------
+
+This charm provides Kubernetes Service checks for Nagios
+
+Quickstart
+----------
+
+    juju deploy cs:kubernetes-service-checks
+    juju add-relation kubernetes-service-checks nrpe
+    juju add-relation kubernetes-service-checks:kube-api-endpoint kubernetes-master
+    juju add-relation kuberentes-service-checks:kube-control kuberentes-master
+    
+
+
+
+### Relations
+
+* **kubernetes-master:kube-api-endpoint** - Provides KSC with the kubernetes-api *hostname* and *port*
+
+* **kuberentes-master:kube-control** - Provides KSC with a kuberentes-api *client-token* for authentication
+
+* **nrpe:nrpe-external-master** - Required for nagios; provides additional plugins 
+
+
+**Note:** Future relations with kubernetes-master *may* be changed so that a 
+single relation can provide the K8S api hostname, port, client token and ssl ca
+cert.
+
+### Config Options
+
+**trusted_ssl_ca** *(Optional)* Setting this option enables SSL host 
+certificate authentication in the api checks
+    
+    juju config kubernetes-service-checks trusted_ssl_ca="${KUBERNETES_API_CA}"
+
+
+Service Checks
+--------------
+The plugin *check_kubernetes_api.py* ships with this charm and contains an array of checks for the k8s api health.
+
+```
+check_kubernetes_api.py --help
+usage: check_kubernetes_api.py [-h] [-H HOST] [-P PORT] [-T CLIENT_TOKEN]
+                               [--check health] [-C SSL_CA_PATH]
+
+Check Kubernetes API status
+
+optional arguments:
+  -h, --help            show this help message and exit
+  -H HOST, --host HOST  Hostname or IP of the kube-api-server (default: None)
+  -P PORT, --port PORT  Port of the kube-api-server (default: 6443)
+  -T CLIENT_TOKEN, --token CLIENT_TOKEN
+                        Client access token for authenticate with the
+                        Kubernetes API (default: None)
+  --check health        which check to run (default: health)
+  -C SSL_CA_PATH, --trusted-ca-cert SSL_CA_PATH
+                        String containing path to the trusted CA certificate
+                        (default: None)
+
+```
+
+**health** - This polls the kubernetes-api */healthz* endpoint. Posting a GET to this URL endpoint is expected to 
+return 200 - 'ok' if the api is healthy, otherwise 500.  
+
+
+Other Checks
+------------
+
+**Certificate Expiration:** The *check_http* plugin is shipped with nrpe, and contains a built in cert expiration check. The warning and crit 
+thesholds are configurable:
+
+    juju config kubernetes-service-checks tls_warn_days=90
+    juju config kubernetes-service-checks tls_crit_days=30
+
+Testing
+-------
+
+Juju should be installed and bootstrapped on the system to run functional tests.
+
+
+```
+    export MODEL_SETTINGS=<semicolon-separated list of "juju model-config" settings>
+    make test
+```
+
+NOTE: If you are behind a proxy, be sure to export a MODEL_SETTINGS variable as
+described above. Note that you will need to use the juju-http-proxy, juju-https-proxy, juju-no-proxy 
+and similar settings.
+
+Contact
+-------
+ - Author: **Bootstack Charmers** *<[e-mail]>*
+ - Bug Tracker: [lp:charm-kubernetes-service-checks](https://launchpad.net/charm-kubernetes-service-checks)
diff --git a/config.yaml b/config.yaml
new file mode 100644
index 0000000..8560b6e
--- /dev/null
+++ b/config.yaml
@@ -0,0 +1,38 @@
+options:
+  channel:
+    type: string
+    default: 1.18/stable
+    description: |
+      Snap channel to install kubectl from
+  nagios_context:
+    default: "juju"
+    type: string
+    description: |
+      Used by the nrpe subordinate charms.
+      A string that will be prepended to instance name to set the host name
+      in nagios. So for instance the hostname would be something like:
+          juju-myservice-0
+      If you're running multiple environments with the same services in them
+      this allows you to differentiate between them.
+  nagios_servicegroups:
+    default: ""
+    type: string
+    description: |
+      A comma-separated list of nagios servicegroups.
+      If left empty, the nagios_context will be used as the servicegroup
+  tls_warn_days:
+    type: int
+    default: 60
+    description: |
+      Number of days left for the TLS certificate to expire before Warning.
+  tls_crit_days:
+    type: int
+    default: 30
+    description: |
+      Number of days left for the TLS certificate to expire before alerting Critical.
+  # temporary config setting for trusted SSL CA (see LP1886982)
+  trusted_ssl_ca:
+    type: string
+    default: ""
+    description: |
+      base64 encoded SSL ca cert to use for Kubernetes API client connections.
\ No newline at end of file
diff --git a/files/plugins/check_kubernetes_api.py b/files/plugins/check_kubernetes_api.py
new file mode 100755
index 0000000..ad91f26
--- /dev/null
+++ b/files/plugins/check_kubernetes_api.py
@@ -0,0 +1,118 @@
+#!/usr/bin/python3
+"""NRPE Plugin for checking Kubernetes API."""
+
+import argparse
+import sys
+
+import urllib3
+
+NAGIOS_STATUS_OK = 0
+NAGIOS_STATUS_WARNING = 1
+NAGIOS_STATUS_CRITICAL = 2
+NAGIOS_STATUS_UNKNOWN = 3
+
+NAGIOS_STATUS = {
+    NAGIOS_STATUS_OK: "OK",
+    NAGIOS_STATUS_WARNING: "WARNING",
+    NAGIOS_STATUS_CRITICAL: "CRITICAL",
+    NAGIOS_STATUS_UNKNOWN: "UNKNOWN",
+}
+
+
+def nagios_exit(status, message):
+    """Return the check status in Nagios preferred format.
+
+    :param status: Nagios Check status code (in [0, 1, 2, 3])
+    :param message: Message describing the status
+    :return: sys.exit("{status_string}: {message}")
+    """
+    assert status in NAGIOS_STATUS, "Invalid Nagios status code"
+    # prefix status name to message
+    output = "{}: {}".format(NAGIOS_STATUS[status], message)
+    print(output)  # nagios requires print to stdout, no stderr
+    sys.exit(status)
+
+
+def check_kubernetes_health(k8s_address, client_token, disable_ssl):
+    """Call <kubernetes-api>/healthz endpoint and check return value is 'ok'.
+
+    :param k8s_address: Address to kube-api-server formatted 'https://<IP>:<PORT>'
+    :param client_token: Token for authenticating with the kube-api
+    :param disable_ssl: Disables SSL Host Key verification
+    """
+    url = k8s_address + "/healthz"
+    if disable_ssl:
+        # perform check without SSL verification
+        http = urllib3.PoolManager(cert_reqs="CERT_NONE", assert_hostname=False)
+    else:
+        http = urllib3.PoolManager()
+
+    try:
+        resp = http.request("GET", url, headers={"Authorization": "Bearer {}".format(client_token)})
+    except urllib3.exceptions.MaxRetryError as e:
+        return NAGIOS_STATUS_CRITICAL, e
+
+    if resp.status != 200:
+        return NAGIOS_STATUS_CRITICAL, "Unexpected HTTP Response code ({})".format(resp.status)
+    elif resp.data != b"ok":
+        return NAGIOS_STATUS_WARNING, "Unexpected Kubernetes healthz status '{}'".format(resp.data)
+    return NAGIOS_STATUS_OK, "Kubernetes health 'ok'"
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(
+        description="Check Kubernetes API status", formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+
+    parser.add_argument("-H", "--host", dest="host", help="Hostname or IP of the kube-api-server", required=True)
+
+    parser.add_argument(
+        "-P", "--port", dest="port", type=int, default=6443, help="Port of the kube-api-server", required=True
+    )
+
+    parser.add_argument(
+        "-T", "--token", dest="client_token", help="Client access token for authenticate with the Kubernetes API"
+    )
+
+    check_choices = ["health"]
+    parser.add_argument(
+        "--check",
+        dest="check",
+        metavar="|".join(check_choices),
+        type=str,
+        choices=check_choices,
+        default=check_choices[0],
+        help="which check to run",
+    )
+
+    parser.add_argument(
+        "-d",
+        "--disable-host-key-check",
+        dest="disable_host_key_check",
+        default=False,
+        action="store_true",
+        help="Disables Host SSL Key Authentication",
+    )
+    args = parser.parse_args()
+
+    checks = {
+        "health": check_kubernetes_health,
+    }
+
+    k8s_url = "https://{}:{}".format(args.host, args.port)
+    nagios_exit(*checks[args.check](k8s_url, args.client_token, args.disable_host_key_check))
+
+"""
+TODO: Future Checks
+
+GET /api/v1/componentstatuses HTTP/1.1
+Authorization: Bearer $TOKEN
+Accept: application/json
+Connection: close
+
+GET /api/va/nodes HTTP/1.1
+Authorization: Bearer $TOKEN
+Accept: application/json
+Connection: close
+
+"""
diff --git a/hooks/install b/hooks/install
new file mode 120000
index 0000000..25b1f68
--- /dev/null
+++ b/hooks/install
@@ -0,0 +1 @@
+../src/charm.py
\ No newline at end of file
diff --git a/lib/.empty b/lib/.empty
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/lib/.empty
diff --git a/lib/charmhelpers b/lib/charmhelpers
new file mode 120000
index 0000000..cedc09e
--- /dev/null
+++ b/lib/charmhelpers
@@ -0,0 +1 @@
+../mod/charm-helpers/charmhelpers
\ No newline at end of file
diff --git a/lib/lib_kubernetes_service_checks.py b/lib/lib_kubernetes_service_checks.py
new file mode 100644
index 0000000..14db4e6
--- /dev/null
+++ b/lib/lib_kubernetes_service_checks.py
@@ -0,0 +1,159 @@
+"""Kubernetes Service Checks Helper Library."""
+import base64
+import json
+import logging
+import os
+import subprocess
+
+from charmhelpers.contrib.charmsupport.nrpe import NRPE
+from charmhelpers.core import hookenv, host
+from charmhelpers.fetch import snap
+
+CERT_FILE = "/usr/local/share/ca-certificates/kubernetes-service-checks.crt"
+NAGIOS_PLUGINS_DIR = "/usr/local/lib/nagios/plugins/"
+
+
+class KSCHelper:
+    """Kubernetes Service Checks Helper Class."""
+
+    def __init__(self, config, state):
+        """Initialize the Helper with the charm config and state."""
+        self.config = config
+        self.state = state
+
+    @property
+    def kubernetes_api_address(self):
+        """Get kubernetes api hostname."""
+        return self.state.kube_api_endpoint.get("hostname", None)
+
+    @property
+    def kubernetes_api_port(self):
+        """Get kubernetes api port."""
+        return self.state.kube_api_endpoint.get("port", None)
+
+    @property
+    def kubernetes_client_token(self):
+        """Get kubernetes client token."""
+        try:
+            data = json.loads(self.state.kube_control.get("creds", "{}"))
+        except json.decoder.JSONDecodeError:
+            data = {}
+        for creds in data.values():
+            token = creds.get("client_token", None)
+            if token:
+                return token
+        return None
+
+    @property
+    def use_tls_cert(self):
+        """Check if SSL cert is provided for use."""
+        return bool(self._ssl_certificate)
+
+    @property
+    def _ssl_certificate(self):
+        # TODO: Expand this later to take a cert from a relation or from the config.
+        # cert from the relation is to be prioritized
+        ssl_cert = self.config.get("trusted_ssl_ca", None)
+        if ssl_cert:
+            ssl_cert = ssl_cert.strip()
+        return ssl_cert
+
+    @property
+    def ssl_cert_path(self):
+        """Get cert file path."""
+        return CERT_FILE
+
+    @property
+    def plugins_dir(self):
+        """Get nagios plugins directory."""
+        return NAGIOS_PLUGINS_DIR
+
+    def restart_nrpe_service(self):
+        """Restart nagios-nrpe-server service."""
+        host.service_restart("nagios-nrpe-server")
+
+    def update_tls_certificates(self):
+        """Write the trusted ssl certificate to the CERT_FILE."""
+        if self._ssl_certificate:
+            cert_content = base64.b64decode(self._ssl_certificate).decode()
+            try:
+                logging.debug("Writing ssl ca cert to {}".format(self.ssl_cert_path))
+                with open(self.ssl_cert_path, "w") as f:
+                    f.write(cert_content)
+                subprocess.call(["/usr/sbin/update-ca-certificates"])
+                return True
+            except subprocess.CalledProcessError as e:
+                logging.error(e)
+                return False
+            except PermissionError as e:
+                logging.error(e)
+                return False
+        else:
+            logging.error("Trusted SSL Certificate is not defined")
+            return False
+
+    def configure(self):
+        """Refresh configuration data."""
+        self.update_plugins()
+        self.render_checks()
+
+    def update_plugins(self):
+        """Rsync plugins to the plugin directory."""
+        charm_plugin_dir = os.path.join(hookenv.charm_dir(), "files", "plugins/")
+        host.rsync(charm_plugin_dir, self.plugins_dir, options=["--executability"])
+
+    def render_checks(self):
+        """Render nrpe checks."""
+        nrpe = NRPE()
+        if not os.path.exists(self.plugins_dir):
+            os.makedirs(self.plugins_dir)
+
+        # register basic api health check
+        check_k8s_plugin = os.path.join(self.plugins_dir, "check_kubernetes_api.py")
+        for check in ["health"]:
+            check_command = "{} -H {} -P {} -T {} --check {}".format(
+                check_k8s_plugin,
+                self.kubernetes_api_address,
+                self.kubernetes_api_port,
+                self.kubernetes_client_token,
+                check,
+            ).strip()
+            if not self.use_tls_cert:
+                check_command += " -d"
+
+            nrpe.add_check(
+                shortname="k8s_api_{}".format(check),
+                description="Check Kubernetes API ({})".format(check),
+                check_cmd=check_command,
+            )
+
+        # register k8s host certificate expiration check
+        check_http_plugin = "/usr/lib/nagios/plugins/check_http"
+        check_command = "{} -I {} -p {} -C {},{}".format(
+            check_http_plugin,
+            self.kubernetes_api_address,
+            self.kubernetes_api_port,
+            self.config.get("tls_warn_days"),
+            self.config.get("tls_crit_days"),
+        ).strip()
+        nrpe.add_check(
+            shortname="k8s_api_cert_expiration",
+            description="Check Kubernetes API ({})".format(check),
+            check_cmd=check_command,
+        )
+        nrpe.write()
+
+    def install_kubectl(self):
+        """Attempt to install kubectl.
+
+        :returns: bool, indicating whether or not successful
+        """
+        # snap retry is excessive
+        snap.SNAP_NO_LOCK_RETRY_DELAY = 0.5
+        snap.SNAP_NO_LOCK_RETRY_COUNT = 3
+        try:
+            channel = self.config.get("channel")
+            snap.snap_install("kubectl", "--classic", "--channel={}".format(channel))
+            return True
+        except snap.CouldNotAcquireLockException:
+            return False
diff --git a/lib/ops b/lib/ops
new file mode 120000
index 0000000..d934193
--- /dev/null
+++ b/lib/ops
@@ -0,0 +1 @@
+../mod/operator/ops
\ No newline at end of file
diff --git a/metadata.yaml b/metadata.yaml
new file mode 100644
index 0000000..09b26a5
--- /dev/null
+++ b/metadata.yaml
@@ -0,0 +1,26 @@
+name: kubernetes-service-checks
+summary: Kubernetes Services NRPE Checks
+maintainers:
+    - Bootstack Charmers <[e-mail]>
+description: |
+    This charm provides NRPE Checks verifying Kubernetes API accessibility
+    and integrates with Nagios for timely alerting.
+tags:
+    - kubernetes
+    - ops
+    - monitoring
+series:
+    - focal
+    - bionic
+    - xenial
+requires:
+    kube-control:
+        interface: kube-control
+    kube-api-endpoint:
+        interface: http
+provides:
+    nrpe-external-master:
+        interface: nrpe-external-master
+        scope: container
+        optional: true
+subordinate: false
diff --git a/mod/.empty b/mod/.empty
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/mod/.empty
diff --git a/mod/charm-helpers b/mod/charm-helpers
new file mode 160000
index 0000000..4b3602e
--- /dev/null
+++ b/mod/charm-helpers
@@ -0,0 +1 @@
+Subproject commit 4b3602e2bdf101bf58cc808264ec0c8092a67cd0
diff --git a/src/charm.py b/src/charm.py
new file mode 100755
index 0000000..b06381d
--- /dev/null
+++ b/src/charm.py
@@ -0,0 +1,191 @@
+#! /usr/bin/env python3
+# -*- coding: utf-8 -*-
+# vim:fenc=utf-8
+# Copyright © 2020 Bootstack Charmers  [e-mail]
+
+"""Operator Charm main library."""
+# Load modules from lib directory
+import logging
+
+import setuppath  # noqa:F401
+
+from lib_kubernetes_service_checks import KSCHelper  # noqa:I100
+from ops.charm import CharmBase
+from ops.framework import StoredState
+from ops.main import main
+from ops.model import ActiveStatus, BlockedStatus, MaintenanceStatus
+
+
+class KubernetesServiceChecksCharm(CharmBase):
+    """Class representing this Operator charm."""
+
+    state = StoredState()
+
+    def __init__(self, *args):
+        """Initialize charm and configure states and events to observe."""
+        super().__init__(*args)
+        # -- standard hook observation
+        self.framework.observe(self.on.install, self.on_install)
+        self.framework.observe(self.on.start, self.on_start)
+        self.framework.observe(self.on.config_changed, self.on_config_changed)
+        self.framework.observe(
+            self.on.kube_api_endpoint_relation_changed, self.on_kube_api_endpoint_relation_changed,
+        )
+        self.framework.observe(self.on.kube_api_endpoint_relation_departed, self.on_kube_api_endpoint_relation_departed)
+        self.framework.observe(
+            self.on.kube_control_relation_changed, self.on_kube_control_relation_changed,
+        )
+        self.framework.observe(self.on.kube_control_relation_departed, self.on_kube_control_relation_departed)
+        self.framework.observe(
+            self.on.nrpe_external_master_relation_joined, self.on_nrpe_external_master_relation_joined
+        )
+        self.framework.observe(
+            self.on.nrpe_external_master_relation_departed, self.on_nrpe_external_master_relation_departed
+        )
+        # -- initialize states --
+        self.state.set_default(
+            installed=False,
+            configured=False,
+            started=False,
+            kube_control={},
+            kube_api_endpoint={},
+            nrpe_configured=False,
+        )
+        self.helper = KSCHelper(self.model.config, self.state)
+
+    def on_install(self, event):
+        """Handle install state."""
+        self.unit.status = MaintenanceStatus("Install complete")
+        logging.info("Install of software complete")
+        self.state.installed = True
+
+    def on_upgrade_charm(self, event):
+        """Handle upgrade and resource updates."""
+        self.state.configured = False
+        logging.info("Reinstalling for upgrade-charm hook")
+        self.on_install(event)
+        self.check_charm_status()
+
+    def check_charm_status(self):
+        """
+        Check that required data is available from relations.
+
+        - Check kube-api-endpoint relation and data available
+        - Check kube-control relation and data available
+        - Check nrpe-external-master is configured
+        - Check any required config options
+        - Finally, configure the charms checks and set flags
+        """
+        if not self.helper.kubernetes_api_address or not self.helper.kubernetes_api_port:
+            logging.warning("kube-api-endpoint relation missing or misconfigured")
+            self.unit.status = BlockedStatus("missing kube-api-endpoint relation")
+            return
+        if not self.helper.kubernetes_client_token:
+            logging.warning("kube-control relation missing or misconfigured")
+            self.unit.status = BlockedStatus("missing kube-control relation")
+            return
+        if not self.state.nrpe_configured:
+            logging.warning("nrpe-external-master relation missing or misconfigured")
+            self.unit.status = BlockedStatus("missing nrpe-external-master relation")
+            return
+
+        # Check specific required config values
+        # Set up TLS Certificate
+        if self.helper.use_tls_cert:
+            logging.info("Updating tls certificates")
+            if self.helper.update_tls_certificates():
+                logging.info("TLS Certificates updated successfully")
+            else:
+                logging.error("Failed to update TLS Certificates")
+                self.unit.status = BlockedStatus("update-ca-certificates error. check logs")
+                return
+        else:
+            logging.warning("No trusted_ssl_ca provided, SSL Host Authentication disabled")
+
+        # configure nrpe checks
+        if not self.state.configured:
+            logging.info("Configuring Kubernetes Service Checks")
+            self.helper.configure()
+
+            logging.info("Reloading nagios-nrpe-server")
+            self.helper.restart_nrpe_service()
+            self.state.configured = True
+        self.unit.status = ActiveStatus("Unit is ready")
+
+    def on_config_changed(self, event):
+        """Handle config changed."""
+        self.state.configured = False
+        if not self.state.installed:
+            logging.warning("Config changed called before install complete, deferring event: {}.".format(event.handle))
+            self._defer_once(event)
+            return
+        self.check_charm_status()
+
+    def on_start(self, event):
+        """Handle start state."""
+        if not self.state.configured:
+            logging.warning("Start called before configuration complete, deferring event: {}".format(event.handle))
+            event.defer()
+            return
+        self.unit.status = ActiveStatus("Unit is ready")
+        self.state.started = True
+        logging.info("Started")
+
+    def _defer_once(self, event):
+        """Defer the given event, but only once."""
+        notice_count = 0
+        handle = str(event.handle)
+
+        for event_path, _, _ in self.framework._storage.notices(None):
+            if event_path.startswith(handle.split("[")[0]):
+                notice_count += 1
+                logging.debug("Found event: {} x {}".format(event_path, notice_count))
+
+        if notice_count > 1:
+            logging.debug("Not deferring {} notice count of {}".format(handle, notice_count))
+        else:
+            logging.debug("Deferring {} notice count of {}".format(handle, notice_count))
+            event.defer()
+
+    def on_kube_api_endpoint_relation_changed(self, event):
+        """Handle kube_api_endpoint relation changed."""
+        self.state.configured = False
+        self.unit.status = MaintenanceStatus("Updating K8S Endpoint")
+        self.state.kube_api_endpoint.update(event.relation.data.get(event.unit, {}))
+        self.check_charm_status()
+
+    def on_kube_api_endpoint_relation_departed(self, event):
+        """Handle kube-api-endpoint relation departed."""
+        self.state.configured = False
+        for k in self.state.kube_api_endpoint.keys():
+            self.state.kube_api_endpoint[k] = ""
+        self.check_charm_status()
+
+    def on_kube_control_relation_changed(self, event):
+        """Handle kube-control relation changed."""
+        self.state.configured = False
+        self.unit.status = MaintenanceStatus("Updating K8S Credentials")
+        self.state.kube_control.update(event.relation.data.get(event.unit, {}))
+        self.check_charm_status()
+
+    def on_kube_control_relation_departed(self, event):
+        """Handle kube-control relation departed."""
+        self.state.configured = False
+        for k in self.state.kube_control.keys():
+            self.state.kube_control[k] = ""
+        self.check_charm_status()
+
+    def on_nrpe_external_master_relation_joined(self, event):
+        """Handle nrpe-external-master relation joined."""
+        self.state.nrpe_configured = True
+        self.check_charm_status()
+
+    def on_nrpe_external_master_relation_departed(self, event):
+        """Handle nrpe-external-master relation departed."""
+        self.state.configured = False
+        self.state.nrpe_configured = False
+        self.check_charm_status()
+
+
+if __name__ == "__main__":
+    main(KubernetesServiceChecksCharm)
diff --git a/src/setuppath.py b/src/setuppath.py
new file mode 100644
index 0000000..768e049
--- /dev/null
+++ b/src/setuppath.py
@@ -0,0 +1,4 @@
+"""Include ./lib in the charm's PATH."""
+import sys
+
+sys.path.append("lib")
diff --git a/tests/functional/requirements.txt b/tests/functional/requirements.txt
new file mode 100644
index 0000000..b7c9112
--- /dev/null
+++ b/tests/functional/requirements.txt
@@ -0,0 +1 @@
+git+https://github.com/openstack-charmers/zaza.git#egg=zaza
diff --git a/tests/functional/tests/bundles/bionic.yaml b/tests/functional/tests/bundles/bionic.yaml
new file mode 100644
index 0000000..c55ba41
--- /dev/null
+++ b/tests/functional/tests/bundles/bionic.yaml
@@ -0,0 +1,90 @@
+series: bionic
+applications:
+    kubernetes-service-checks:
+        charm: ../../../../
+        num_units: 1
+    containerd:
+        charm: cs:~containers/containerd
+        options:
+            gpu_driver: none
+        resources: {}
+    easyrsa:
+        charm: cs:~containers/easyrsa
+        num_units: 1
+        resources:
+          easyrsa: 5
+    etcd:
+        charm: cs:~containers/etcd
+        num_units: 1
+        options:
+          channel: 3.3/stable
+        resources:
+          core: 0
+          etcd: 3
+          snapshot: 0
+    flannel:
+        charm: cs:~containers/flannel
+        resources:
+          flannel-amd64: 625
+          flannel-arm64: 622
+          flannel-s390x: 609
+    kubernetes-master:
+        charm: cs:~containers/kubernetes-master
+        num_units: 1
+        constraints: cores=4 mem=4G root-disk=16G
+        options:
+          channel: 1.18/stable
+        resources:
+          cdk-addons: 0
+          core: 0
+          kube-apiserver: 0
+          kube-controller-manager: 0
+          kube-proxy: 0
+          kube-scheduler: 0
+          kubectl: 0
+    kubernetes-worker:
+        charm: cs:~containers/kubernetes-worker
+        expose: true
+        num_units: 1
+        constraints: cores=4 mem=4G root-disk=16G
+        options:
+            channel: 1.18/stable
+        resources:
+          cni-amd64: 645
+          cni-arm64: 636
+          cni-s390x: 648
+          core: 0
+          kube-proxy: 0
+          kubectl: 0
+          kubelet: 0
+    nrpe:
+        charm: cs:nrpe
+relations:
+    - - kubernetes-master:kube-api-endpoint
+      - kubernetes-worker:kube-api-endpoint
+    - - kubernetes-service-checks:nrpe-external-master
+      - nrpe:nrpe-external-master
+    - - kubernetes-master:kube-control
+      - kubernetes-worker:kube-control
+    - - kubernetes-master:certificates
+      - easyrsa:client
+    - - etcd:certificates
+      - easyrsa:client
+    - - kubernetes-master:etcd
+      - etcd:db
+    - - kubernetes-worker:certificates
+      - easyrsa:client
+    - - flannel:etcd
+      - etcd:db
+    - - flannel:cni
+      - kubernetes-master:cni
+    - - flannel:cni
+      - kubernetes-worker:cni
+    - - containerd:containerd
+      - kubernetes-worker:container-runtime
+    - - containerd:containerd
+      - kubernetes-master:container-runtime
+    - - kubernetes-service-checks:kube-control
+      - kubernetes-master:kube-control
+    - - kubernetes-service-checks:kube-api-endpoint
+      - kubernetes-master:kube-api-endpoint
\ No newline at end of file
diff --git a/tests/functional/tests/bundles/focal.yaml b/tests/functional/tests/bundles/focal.yaml
new file mode 100644
index 0000000..31888a0
--- /dev/null
+++ b/tests/functional/tests/bundles/focal.yaml
@@ -0,0 +1,90 @@
+series: focal
+applications:
+    kubernetes-service-checks:
+        charm: ../../../../
+        num_units: 1
+    containerd:
+        charm: cs:~containers/containerd
+        options:
+            gpu_driver: none
+        resources: {}
+    easyrsa:
+        charm: cs:~containers/easyrsa
+        num_units: 1
+        resources:
+          easyrsa: 5
+    etcd:
+        charm: cs:~containers/etcd
+        num_units: 1
+        options:
+          channel: 3.3/stable
+        resources:
+          core: 0
+          etcd: 3
+          snapshot: 0
+    flannel:
+        charm: cs:~containers/flannel
+        resources:
+          flannel-amd64: 625
+          flannel-arm64: 622
+          flannel-s390x: 609
+    kubernetes-master:
+        charm: cs:~containers/kubernetes-master
+        num_units: 1
+        constraints: cores=4 mem=4G root-disk=16G
+        options:
+          channel: 1.18/stable
+        resources:
+          cdk-addons: 0
+          core: 0
+          kube-apiserver: 0
+          kube-controller-manager: 0
+          kube-proxy: 0
+          kube-scheduler: 0
+          kubectl: 0
+    kubernetes-worker:
+        charm: cs:~containers/kubernetes-worker
+        expose: true
+        num_units: 1
+        constraints: cores=4 mem=4G root-disk=16G
+        options:
+            channel: 1.18/stable
+        resources:
+          cni-amd64: 645
+          cni-arm64: 636
+          cni-s390x: 648
+          core: 0
+          kube-proxy: 0
+          kubectl: 0
+          kubelet: 0
+    nrpe:
+        charm: cs:nrpe
+relations:
+    - - kubernetes-master:kube-api-endpoint
+      - kubernetes-worker:kube-api-endpoint
+    - - kubernetes-service-checks:nrpe-external-master
+      - nrpe:nrpe-external-master
+    - - kubernetes-master:kube-control
+      - kubernetes-worker:kube-control
+    - - kubernetes-master:certificates
+      - easyrsa:client
+    - - etcd:certificates
+      - easyrsa:client
+    - - kubernetes-master:etcd
+      - etcd:db
+    - - kubernetes-worker:certificates
+      - easyrsa:client
+    - - flannel:etcd
+      - etcd:db
+    - - flannel:cni
+      - kubernetes-master:cni
+    - - flannel:cni
+      - kubernetes-worker:cni
+    - - containerd:containerd
+      - kubernetes-worker:container-runtime
+    - - containerd:containerd
+      - kubernetes-master:container-runtime
+    - - kubernetes-service-checks:kube-control
+      - kubernetes-master:kube-control
+    - - kubernetes-service-checks:kube-api-endpoint
+      - kubernetes-master:kube-api-endpoint
diff --git a/tests/functional/tests/bundles/xenial.yaml b/tests/functional/tests/bundles/xenial.yaml
new file mode 100644
index 0000000..962bdfd
--- /dev/null
+++ b/tests/functional/tests/bundles/xenial.yaml
@@ -0,0 +1,90 @@
+series: xenial
+applications:
+    kubernetes-service-checks:
+        charm: ../../../../
+        num_units: 1
+    containerd:
+        charm: cs:~containers/containerd
+        options:
+            gpu_driver: none
+        resources: {}
+    easyrsa:
+        charm: cs:~containers/easyrsa
+        num_units: 1
+        resources:
+          easyrsa: 5
+    etcd:
+        charm: cs:~containers/etcd
+        num_units: 1
+        options:
+          channel: 3.3/stable
+        resources:
+          core: 0
+          etcd: 3
+          snapshot: 0
+    flannel:
+        charm: cs:~containers/flannel
+        resources:
+          flannel-amd64: 625
+          flannel-arm64: 622
+          flannel-s390x: 609
+    kubernetes-master:
+        charm: cs:~containers/kubernetes-master
+        num_units: 1
+        constraints: cores=4 mem=4G root-disk=16G
+        options:
+          channel: 1.18/stable
+        resources:
+          cdk-addons: 0
+          core: 0
+          kube-apiserver: 0
+          kube-controller-manager: 0
+          kube-proxy: 0
+          kube-scheduler: 0
+          kubectl: 0
+    kubernetes-worker:
+        charm: cs:~containers/kubernetes-worker
+        expose: true
+        num_units: 1
+        constraints: cores=4 mem=4G root-disk=16G
+        options:
+            channel: 1.18/stable
+        resources:
+          cni-amd64: 645
+          cni-arm64: 636
+          cni-s390x: 648
+          core: 0
+          kube-proxy: 0
+          kubectl: 0
+          kubelet: 0
+    nrpe:
+        charm: cs:nrpe
+relations:
+    - - kubernetes-master:kube-api-endpoint
+      - kubernetes-worker:kube-api-endpoint
+    - - kubernetes-service-checks:nrpe-external-master
+      - nrpe:nrpe-external-master
+    - - kubernetes-master:kube-control
+      - kubernetes-worker:kube-control
+    - - kubernetes-master:certificates
+      - easyrsa:client
+    - - etcd:certificates
+      - easyrsa:client
+    - - kubernetes-master:etcd
+      - etcd:db
+    - - kubernetes-worker:certificates
+      - easyrsa:client
+    - - flannel:etcd
+      - etcd:db
+    - - flannel:cni
+      - kubernetes-master:cni
+    - - flannel:cni
+      - kubernetes-worker:cni
+    - - containerd:containerd
+      - kubernetes-worker:container-runtime
+    - - containerd:containerd
+      - kubernetes-master:container-runtime
+    - - kubernetes-service-checks:kube-control
+      - kubernetes-master:kube-control
+    - - kubernetes-service-checks:kube-api-endpoint
+      - kubernetes-master:kube-api-endpoint
diff --git a/tests/functional/tests/kubernetes_service_checks.py b/tests/functional/tests/kubernetes_service_checks.py
new file mode 100644
index 0000000..e1520f3
--- /dev/null
+++ b/tests/functional/tests/kubernetes_service_checks.py
@@ -0,0 +1,162 @@
+"""Charm Kubernetes Service Checks Functional Tests."""
+import concurrent.futures
+import logging
+import re
+import time
+import unittest
+
+from juju.errors import JujuAPIError
+import zaza.model
+
+
+class TestBase(unittest.TestCase):
+    """Base Class for charm functional tests."""
+
+    @classmethod
+    def setUpClass(cls):
+        """Run setup for tests."""
+        cls.model_name = zaza.model.get_juju_model()
+        cls.application_name = "kubernetes-service-checks"
+
+    def setUp(self):
+        """Set up  functional tests & ensure all relations added."""
+        for local_relation_name, remote_relation_unit in [
+            ("kube-api-endpoint", "kubernetes-master"),
+            ("kube-control", "kubernetes-master"),
+            ("nrpe-external-master", "nrpe"),
+        ]:
+            logging.info("Adding relation {} with {}".format(local_relation_name, remote_relation_unit))
+            try:
+                zaza.model.add_relation(
+                    self.application_name, local_relation_name, remote_relation_unit, self.model_name
+                )
+            except JujuAPIError as e:
+                p = r"^.*cannot\ add\ relation.*already\ exists"
+                if re.search(p, e.message):
+                    pass
+                else:
+                    raise (e)
+        zaza.model.block_until_wl_status_info_starts_with(self.application_name, status="Unit is ready", timeout=200)
+
+
+class TestChecks(TestBase):
+    """Tests for availability and usefulness of nagios checks."""
+
+    expected_checks = ["check_k8s_api_health.cfg", "check_k8s_api_cert_expiration.cfg"]
+    checks_dir = "/etc/nagios/nrpe.d/"
+    expected_plugins = ["check_kubernetes_api.py"]
+    plugins_dir = "/usr/local/lib/nagios/plugins/"
+
+    # TODO: Need testing around setting the trusted_ssl_ca cert
+    #    - does it get written to /etc/ssl/certs/ca-certificates.crt?
+    #    - does the k8s check plugin see it and use it for verification?
+
+    def test_check_plugins_exist(self):
+        """Verify that kubernetes service checks plugins are found."""
+        fail_messages = []
+        for plugin in self.expected_plugins:
+            pluginpath = self.plugins_dir + plugin
+            response = zaza.model.run_on_unit(
+                "kubernetes-service-checks/0", '[ -f "{}" ]'.format(pluginpath), model_name=self.model_name, timeout=30
+            )
+            if response["Code"] != "0":
+                fail_messages.append("Missing plugin: {}".format(pluginpath))
+                continue
+
+            # check executable
+            response = zaza.model.run_on_unit(
+                "kubernetes-service-checks/0", '[ -x "{}" ]'.format(pluginpath), model_name=self.model_name, timeout=30
+            )
+
+            if response["Code"] != "0":
+                fail_messages.append("Plugin not executable: {}".format(pluginpath))
+
+        if fail_messages:
+            self.fail("\n".join(fail_messages))
+
+    def test_checks_exist(self):
+        """Verify that kubernetes service checks nrpe checks exist."""
+        fail_messages = []
+        for check in self.expected_checks:
+            checkpath = self.checks_dir + check
+            response = zaza.model.run_on_unit(
+                "kubernetes-service-checks/0", '[ -f "{}" ]'.format(checkpath), model_name=self.model_name, timeout=30
+            )
+            if response["Code"] != "0":
+                fail_messages.append("Missing check: {}".format(checkpath))
+        if fail_messages:
+            self.fail("\n".join(fail_messages))
+
+
+class TestRelations(TestBase):
+    """Tests for charm behavior adding and removing relations."""
+
+    def _get_relation_id(self, remote_application, interface_name):
+        return zaza.model.get_relation_id(
+            self.application_name, remote_application, model_name=self.model_name, remote_interface_name=interface_name
+        )
+
+    def test_remove_kube_api_endpoint(self):
+        """Test removing kube-api-endpoint relation."""
+        rel_name = "kube-api-endpoint"
+        remote_app = "kubernetes-master"
+        logging.info("Removing kube-api-endpoint relation")
+
+        zaza.model.remove_relation(self.application_name, rel_name, remote_app, self.model_name)
+        try:
+            zaza.model.block_until_wl_status_info_starts_with(
+                self.application_name, status="missing kube-api-endpoint relation", timeout=180
+            )
+        except concurrent.futures._base.TimeoutError:
+            self.fail("Timed out waiting for Unit to become blocked")
+
+        logging.info("Waiting for relation {} to be destroyed".format(rel_name))
+        timeout = time.time() + 600
+        while self._get_relation_id(remote_app, rel_name) is not None:
+            time.sleep(5)
+            if time.time() > timeout:
+                self.fail("Timed out waiting for the relation {} to be destroyed".format(rel_name))
+
+    def test_remove_kube_control(self):
+        """Test removing kube-control relation."""
+        rel_name = "kube-control"
+        remote_app = "kubernetes-master"
+        logging.info("Removing kube-control relation")
+
+        zaza.model.remove_relation(self.application_name, rel_name, remote_app, self.model_name)
+
+        try:
+            zaza.model.block_until_wl_status_info_starts_with(
+                self.application_name, status="missing kube-control relation", timeout=180
+            )
+        except concurrent.futures._base.TimeoutError:
+            self.fail("Timed out waiting for Unit to become blocked")
+
+        logging.info("Waiting for relation {} to be destroyed".format(rel_name))
+        timeout = time.time() + 600
+        while self._get_relation_id(remote_app, rel_name) is not None:
+            time.sleep(5)
+            if time.time() > timeout:
+                self.fail("Timed out waiting for the relation {} to be destroyed".format(rel_name))
+
+    def test_remove_nrpe_external_master(self):
+        """Test removing nrpe-external-master relation."""
+        rel_name = "nrpe-external-master"
+        remote_app = "nrpe"
+        logging.info("Removing nrpe-external-master relation")
+
+        zaza.model.remove_relation(self.application_name, rel_name, remote_app, self.model_name)
+
+        try:
+            zaza.model.block_until_wl_status_info_starts_with(
+                self.application_name, status="missing nrpe-external-master relation", timeout=180
+            )
+        except concurrent.futures._base.TimeoutError:
+            self.fail("Timed out waiting for Unit to become blocked")
+
+        logging.info("Waiting for relation {} to be destroyed".format(rel_name))
+        timeout = time.time() + 600
+        while self._get_relation_id(remote_app, rel_name) is not None:
+            time.sleep(5)
+            if time.time() > timeout:
+                self.fail("Timed out waiting for the relation {} to be destroyed".format(rel_name))
diff --git a/tests/functional/tests/tests.yaml b/tests/functional/tests/tests.yaml
new file mode 100644
index 0000000..75a9ce8
--- /dev/null
+++ b/tests/functional/tests/tests.yaml
@@ -0,0 +1,33 @@
+tests:
+    - tests.kubernetes_service_checks.TestChecks
+    - tests.kubernetes_service_checks.TestRelations
+target_deploy_status:
+    kubernetes-service-checks:
+        workload-status: blocked
+        workload-status-message: "missing kube-api-endpoint relation"
+    kubernetes-master:
+        workload-status: active
+        workload-status-message: "Kubernetes master running."
+    kubernetes-worker:
+        workload-status: active
+        workload-status-message: "Kubernetes worker running."
+    flannel:
+        workload-status: active
+        workload-status-message: "Flannel subnet"
+    easyrsa:
+        workload-status: active
+        workload-status-message: "Certificate Authority connected."
+    containerd:
+        workload-status: active
+        workload-status-message: "Container runtime available"
+    etcd:
+        workload-status: active
+        workload-status-message: "Healthy with 1 known peer"
+    nrpe:
+        workload-status: active
+        workload-status-message: "ready"
+gate_bundles:
+    - xenial
+    - bionic
+smoke_bundles:
+    - focal
diff --git a/tests/unit/requirements.txt b/tests/unit/requirements.txt
new file mode 100644
index 0000000..b5f9fbd
--- /dev/null
+++ b/tests/unit/requirements.txt
@@ -0,0 +1,5 @@
+mock
+pyyaml
+coverage
+six
+urllib3
diff --git a/tests/unit/setuppath.py b/tests/unit/setuppath.py
new file mode 100644
index 0000000..c3404e5
--- /dev/null
+++ b/tests/unit/setuppath.py
@@ -0,0 +1,6 @@
+"""Include ./lib ./src and ./file/plugins in the tests' PATH."""
+import sys
+
+sys.path.append("lib")
+sys.path.append("src")
+sys.path.append("files/plugins")
diff --git a/tests/unit/test_charm.py b/tests/unit/test_charm.py
new file mode 100644
index 0000000..24f83f5
--- /dev/null
+++ b/tests/unit/test_charm.py
@@ -0,0 +1,243 @@
+"""Charm unit tests."""
+import os
+from pathlib import Path
+import unittest
+
+
+import mock
+import yaml
+
+# include ./lib in the charm's PATH
+import setuppath  # noqa:F401
+
+from charm import KubernetesServiceChecksCharm  # noqa:I100
+import ops.main
+from ops.testing import Harness
+
+TEST_KUBE_CONTOL_RELATION_DATA = {
+    "creds": """{"system:node:juju-62684f-0":
+                                      {"client_token": "DECAFBADBEEF",
+                                       "kubelet_token": "ABCDEF012345",
+                                       "proxy_token": "BADC0FFEEDAD",
+                                       "scope": "kubernetes-worker/0"}
+                                   }"""  # noqa:E127
+}
+TEST_KUBE_API_ENDPOINT_RELATION_DATA = {"hostname": "1.1.1.1", "port": "1111"}
+
+
+class TestKubernetesServiceChecksCharm(unittest.TestCase):
+    """Test Kubernetes Service Checks Charm Code."""
+
+    @classmethod
+    def setUpClass(cls):
+        """Prepare class fixture."""
+        # Stop unit test from calling fchown
+        fchown_patcher = mock.patch("os.fchown")
+        cls.mock_fchown = fchown_patcher.start()
+        chown_patcher = mock.patch("os.chown")
+        cls.mock_chown = chown_patcher.start()
+
+        # Stop charmhelpers host from logging via debug log
+        host_log_patcher = mock.patch("charmhelpers.core.host.log")
+        cls.mock_juju_log = host_log_patcher.start()
+
+        # Stop charmhelpers snap from logging via debug log
+        snap_log_patcher = mock.patch("charmhelpers.fetch.snap.log")
+        cls.mock_snap_log = snap_log_patcher.start()
+
+        charm_logger_patcher = mock.patch("charm.logging")
+        cls.mock_charm_log = charm_logger_patcher.start()
+
+        lib_logger_patcher = mock.patch("lib.lib_kubernetes_service_checks.logging")
+        cls.mock_lib_logger = lib_logger_patcher.start()
+
+        # Prevent charmhelpers from calling systemctl
+        host_service_patcher = mock.patch("charmhelpers.core.host.service_stop")
+        cls.mock_service_stop = host_service_patcher.start()
+        host_service_patcher = mock.patch("charmhelpers.core.host.service_start")
+        cls.mock_service_start = host_service_patcher.start()
+        host_service_patcher = mock.patch("charmhelpers.core.host.service_restart")
+        cls.mock_service_restart = host_service_patcher.start()
+
+        # Setup mock JUJU Environment variables
+        os.environ["JUJU_UNIT_NAME"] = "mock/0"
+        os.environ["JUJU_CHARM_DIR"] = "."
+
+    def setUp(self):
+        """Prepare tests."""
+        self.harness = Harness(KubernetesServiceChecksCharm)
+        # Mock config_get to return default config
+        with open(ops.main._get_charm_dir() / Path("config.yaml"), "r") as config_file:
+            config = yaml.safe_load(config_file)
+        charm_config = {}
+
+        for key, _ in config["options"].items():
+            charm_config[key] = config["options"][key]["default"]
+
+        self.harness._backend._config = charm_config
+
+    def test_harness(self):
+        """Verify harness."""
+        self.harness.begin()
+        self.assertFalse(self.harness.charm.state.installed)
+
+    @mock.patch("charmhelpers.fetch.snap.subprocess.check_call")
+    def test_install(self, mock_snap_subprocess):
+        """Test response to an install event."""
+        mock_snap_subprocess.return_value = 0
+        mock_snap_subprocess.side_effect = None
+
+        self.harness.begin()
+        self.harness.charm.on.install.emit()
+
+        self.assertEqual(self.harness.charm.unit.status.name, "maintenance")
+        self.assertEqual(self.harness.charm.unit.status.message, "Install complete")
+        self.assertTrue(self.harness.charm.state.installed)
+
+    def test_config_changed(self):
+        """Test response to config changed event."""
+        self.harness.set_leader(True)
+        self.harness.populate_oci_resources()
+        self.harness.begin()
+        self.harness.charm.check_charm_status = mock.MagicMock()
+        self.harness.charm.state.installed = True
+        self.harness.charm.on.config_changed.emit()
+        self.harness.charm.check_charm_status.assert_called_once()
+
+    def test_start_not_installed(self):
+        """Test response to start event without install state."""
+        self.harness.begin()
+        self.harness.charm.on.start.emit()
+        self.assertFalse(self.harness.charm.state.started)
+
+    def test_start_not_configured(self):
+        """Test response to start event without configured state."""
+        self.harness.begin()
+        self.harness.charm.state.installed = True
+        self.harness.charm.on.start.emit()
+        self.assertFalse(self.harness.charm.state.started)
+
+    def test_start(self):
+        """Test response to start event."""
+        self.harness.begin()
+        self.harness.charm.state.installed = True
+        self.harness.charm.state.configured = True
+        self.harness.charm.on.start.emit()
+        self.assertTrue(self.harness.charm.state.started)
+        self.assertEqual(self.harness.charm.unit.status.name, "active")
+
+    def test_on_kube_api_endpoint_relation_changed(self):
+        """Check kube-api-endpoint relation changed handling."""
+        relation_id = self.harness.add_relation("kube-api-endpoint", "kubernetes-master")
+        remote_unit = "kubernetes-master/0"
+        self.harness.begin()
+        self.harness.charm.check_charm_status = mock.MagicMock()
+        self.harness.add_relation_unit(relation_id, remote_unit)
+        self.harness.update_relation_data(relation_id, remote_unit, TEST_KUBE_API_ENDPOINT_RELATION_DATA)
+
+        self.harness.charm.check_charm_status.assert_called_once()
+        self.assertEqual(self.harness.charm.helper.kubernetes_api_address, "1.1.1.1")
+        self.assertEqual(self.harness.charm.helper.kubernetes_api_port, "1111")
+
+    def test_on_kube_control_relation_changed(self):
+        """Check kube-control relation changed handling."""
+        relation_id = self.harness.add_relation("kube-control", "kubernetes-master")
+        remote_unit = "kubernetes-master/0"
+        self.harness.begin()
+        self.harness.charm.check_charm_status = mock.MagicMock()
+        self.harness.add_relation_unit(relation_id, remote_unit)
+        self.harness.update_relation_data(relation_id, remote_unit, TEST_KUBE_CONTOL_RELATION_DATA)
+
+        self.harness.charm.check_charm_status.assert_called_once()
+        assert self.harness.charm.helper.kubernetes_client_token == "DECAFBADBEEF"
+
+    def test_nrpe_external_master_relation_joined(self):
+        """Check that nrpe.configure is True after nrpe relation joined."""
+        relation_id = self.harness.add_relation("nrpe-external-master", "nrpe")
+        remote_unit = "nrpe/0"
+        self.harness.begin()
+        self.assertFalse(self.harness.charm.state.nrpe_configured)
+        self.harness.charm.check_charm_status = mock.MagicMock()
+        self.harness.add_relation_unit(relation_id, remote_unit)
+
+        self.harness.charm.check_charm_status.assert_called_once()
+        self.assertTrue(self.harness.charm.state.nrpe_configured)
+
+    @mock.patch("ops.model.RelationData")
+    def test_nrpe_external_master_relation_departed(self, mock_relation_data):
+        """Check that nrpe.configure is False after nrpe relation departed."""
+        mock_relation_data.return_value.__getitem__.return_value = {}
+        self.harness.begin()
+        self.harness.charm.check_charm_status = mock.MagicMock()
+        self.emit("nrpe_external_master_relation_departed")
+        self.harness.charm.check_charm_status.assert_called_once()
+
+        self.assertFalse(self.harness.charm.state.nrpe_configured)
+
+    def test_check_charm_status_kube_api_endpoint_relation_missing(self):
+        """Check that the chatm blocks without kube-api-endpoint relation."""
+        self.harness.begin()
+        self.harness.charm.state.kube_control.update(TEST_KUBE_CONTOL_RELATION_DATA)
+        self.harness.charm.state.nrpe_configured = True
+        self.harness.charm.check_charm_status()
+
+        self.assertFalse(self.harness.charm.state.configured)
+        self.assertEqual(self.harness.charm.unit.status.name, "blocked")
+        self.assertEqual(self.harness.charm.unit.status.message, "missing kube-api-endpoint relation")
+
+    def test_check_charm_status_kube_control_relation_missing(self):
+        """Check that the charm blocks without kube-control relation."""
+        self.harness.begin()
+        self.harness.charm.state.kube_api_endpoint.update(TEST_KUBE_API_ENDPOINT_RELATION_DATA)
+        self.harness.charm.state.nrpe_configured = True
+        self.harness.charm.check_charm_status()
+
+        self.assertFalse(self.harness.charm.state.configured)
+        self.assertEqual(self.harness.charm.unit.status.name, "blocked")
+        self.assertEqual(self.harness.charm.unit.status.message, "missing kube-control relation")
+
+    def test_check_charm_status_nrpe_relation_missing(self):
+        """Check that the charm bloack without nrpe relation."""
+        self.harness.begin()
+        self.harness.charm.state.kube_control.update(TEST_KUBE_CONTOL_RELATION_DATA)
+        self.harness.charm.state.kube_api_endpoint.update(TEST_KUBE_API_ENDPOINT_RELATION_DATA)
+        self.harness.charm.check_charm_status()
+
+        self.assertFalse(self.harness.charm.state.configured)
+        self.assertEqual(self.harness.charm.unit.status.name, "blocked")
+        self.assertEqual(self.harness.charm.unit.status.message, "missing nrpe-external-master relation")
+
+    def test_check_charm_status_configured(self):
+        """Check the charm becomes configured."""
+        self.harness.begin()
+        self.harness.charm.helper.configure = mock.MagicMock()
+        self.harness.charm.state.kube_control.update(TEST_KUBE_CONTOL_RELATION_DATA)
+        self.harness.charm.state.kube_api_endpoint.update(TEST_KUBE_API_ENDPOINT_RELATION_DATA)
+        self.harness.charm.state.nrpe_configured = True
+        self.harness.charm.check_charm_status()
+
+        self.harness.charm.helper.configure.assert_called_once()
+        self.assertTrue(self.harness.charm.state.configured)
+
+    def emit(self, event):
+        """Emit the named hook on the charm."""
+        self.harness.charm.framework.reemit()
+
+        if "_relation_" in event:
+            relation_name = event.split("_relation")[0].replace("_", "-")
+            with mock.patch.dict(
+                "os.environ",
+                {
+                    "JUJU_RELATION": relation_name,
+                    "JUJU_RELATION_ID": "1",
+                    "JUJU_REMOTE_APP": "mock",
+                    "JUJU_REMOTE_UNIT": "mock/0",
+                },
+            ):
+                ops.main._emit_charm_event(self.harness.charm, event)
+        else:
+            ops.main._emit_charm_event(self.harness.charm, event)
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/tests/unit/test_lib_ksc.py b/tests/unit/test_lib_ksc.py
new file mode 100644
index 0000000..94f67bc
--- /dev/null
+++ b/tests/unit/test_lib_ksc.py
@@ -0,0 +1,164 @@
+"""Tests for Kubernetes Service Checks Helper."""
+import base64
+import os
+import subprocess
+from subprocess import CalledProcessError
+import tempfile
+import unittest
+
+from lib import lib_kubernetes_service_checks
+import mock
+import yaml
+
+
+TEST_CERTIFICATE = """-----BEGIN CERTIFICATE-----
+MIIDOzCCAiOgAwIBAgIJAPoOXrIwH+miMA0GCSqGSIb3DQEBCwUAMBgxFjAUBgNV
+BAMMDTEwLjEzMi4yNTEuNjAwHhcNMjAwNzE3MTMzMzI0WhcNMzAwNzE1MTMzMzI0
+WjAYMRYwFAYDVQQDDA0xMC4xMzIuMjUxLjYwMIIBIjANBgkqhkiG9w0BAQEFAAOC
+AQ8AMIIBCgKCAQEAqpYVlmT/eRBhCKHaqXjY6EAzvx5GZY0PhL/YGBl9uF8YQGEF
+F3k3Ec7pyJMIQblmWxdCPd1uNzHU8mwApiuPG9GtYOK+olqgslLsmOU9LTi6KJWX
+x956VxdefXDYvr0B6K/Hdgkb1x//XwvipSV1fZ1MCDIiP/hWKi4CmEq31sVpCBdp
+Uiz3qdCzsiGt0f4kbgIJSVtxhWlNJ5MaCOm7gXafkF8OIUTmWhmPp2gH7pfPzzl1
+glOX2Z41qwPuz7Jbcxx/z/yGjdPeJTQYoqJfpDpCrT2er5xyRf66HqKx9Ld/FiqM
+ZksRwmzF9WvqCBK8WoRmnvFxk1FZPGt6E5gotwIDAQABo4GHMIGEMB0GA1UdDgQW
+BBSUCCmRxb4tKD6w8jZ3hHs4ciFizDBIBgNVHSMEQTA/gBSUCCmRxb4tKD6w8jZ3
+hHs4ciFizKEcpBowGDEWMBQGA1UEAwwNMTAuMTMyLjI1MS42MIIJAPoOXrIwH+mi
+MAwGA1UdEwQFMAMBAf8wCwYDVR0PBAQDAgEGMA0GCSqGSIb3DQEBCwUAA4IBAQBv
+BYwILWI/4dGczqG0hcqt8tW04Oi+7y0HxzeI/oaUq/HKfvCz5a+WhpykMKRDJoaZ
+aejR2Oc7A0OUnenpvMeIiMcUIetM3Q1Gzx0aU+vqUNNaZlooSSbe3z1VK6bUsYDo
+qdKhs+mSyuEticK2SEWjT+ZWpV1rjSd5zRZ/UvC1ZhDNJGZotIIqryQWd3YfYl9l
+7JrdzUVCbxs4ywxNp9/I+MJEiBfMHQx8FWr1M2HvLDAm6NZLfM68y5FClzfGpopV
+0ARirz1AfbS6xUumyXHOH2qH527PUXFdfYGSn+juDG/dRTENYJ3OPAfWdj4ze1qQ
+n3ajLSYPvdyKaztdB1VL
+-----END CERTIFICATE-----"""
+
+
+class TestLibKSCHelper(unittest.TestCase):
+    """Unittest class for Kubernetes Service Checks Helper."""
+
+    @classmethod
+    def setUpClass(cls):
+        """Prepare Class Fixture."""
+        # Load default config
+        with open("./config.yaml") as default_config:
+            cls.config = yaml.safe_load(default_config)
+
+        # set defaults to the config object
+        for key in cls.config["options"]:
+            if "default" in cls.config["options"][key]:
+                cls.config[key] = cls.config["options"][key]["default"]
+
+        # Create test state object
+        class FakeStateObject(object):
+            kube_api_endpoint = {"hostname": "1.1.1.1", "port": "1111"}
+            kube_control = {"creds": """{"kube-client": {"client_token": "abcdef0123456789"}}"""}
+            installed = False
+            configured = False
+            started = False
+            nrpe_configured = False
+
+        cls.state = FakeStateObject()
+
+        # Stop unit test from calling fchown
+        fchown_patcher = mock.patch("os.fchown")
+        cls.mock_fchown = fchown_patcher.start()
+        chown_patcher = mock.patch("os.chown")
+        cls.mock_chown = chown_patcher.start()
+
+        # Stop charmhelpers host from logging via debug log
+        host_log_patcher = mock.patch("charmhelpers.core.host.log")
+        cls.mock_juju_log = host_log_patcher.start()
+
+        host_logger_patcher = mock.patch("lib.lib_kubernetes_service_checks.logging")
+        cls.mock_logger = host_logger_patcher.start()
+
+        # Stop charmhelpers snap from logging via debug log
+        snap_log_patcher = mock.patch("charmhelpers.fetch.snap.log")
+        cls.mock_snap_log = snap_log_patcher.start()
+
+        # Setup a tmpdir
+        cls.tmpdir = tempfile.TemporaryDirectory()
+        cls.cert_path = os.path.join(cls.tmpdir.name, "kubernetes-service-checks.crt")
+
+        lib_kubernetes_service_checks.CERT_FILE = cls.cert_path
+        lib_kubernetes_service_checks.NAGIOS_PLUGINS_DIR = cls.tmpdir.name
+
+    @classmethod
+    def tearDownClass(cls):
+        """Tear down class fixture."""
+        mock.patch.stopall()
+        cls.tmpdir.cleanup()
+
+    def setUp(self):
+        """Prepare test fixture."""
+        self.helper = lib_kubernetes_service_checks.KSCHelper(self.config, self.state)
+
+    def tearDown(self):
+        """Clean up test fixture."""
+        try:
+            os.remove(self.cert_path)
+        except FileNotFoundError:
+            pass
+
+    def test_kube_api_endpoint_properties(self):
+        """Test that hostname and port properties get passed through."""
+        # kube_api_endpoint (relation) -> hostname & port
+        self.assertEqual(self.helper.kubernetes_api_address, "1.1.1.1")
+        self.assertEqual(self.helper.kubernetes_api_port, "1111")
+
+        self.helper.state.kube_api_endpoint = {}
+        self.assertEqual(self.helper.kubernetes_api_address, None)
+        self.assertEqual(self.helper.kubernetes_api_port, None)
+
+    def test_kube_control_endpoint_properties(self):
+        """Test KSCHelper client_token gets passed though."""
+        # kube-control (relation) -> kube client token
+        self.assertEqual(self.helper.kubernetes_client_token, "abcdef0123456789")
+
+        self.helper.state.kube_control = {}
+        self.assertEqual(self.helper.kubernetes_client_token, None)
+
+    @mock.patch("lib.lib_kubernetes_service_checks.subprocess.call")
+    def test_update_tls_certificates(self, mock_subprocess):
+        """Test that SSL certificates get updated."""
+        # returns False when no available trusted_ssl_cert
+        self.assertFalse(self.helper.update_tls_certificates())
+
+        # returns True when subprocess successful
+        self.helper.config["trusted_ssl_ca"] = base64.b64encode(str.encode(TEST_CERTIFICATE))
+        self.assertTrue(self.helper.update_tls_certificates())
+        with open(self.cert_path, "r") as f:
+            self.assertEqual(f.read(), TEST_CERTIFICATE)
+        mock_subprocess.assert_called_once_with(["/usr/sbin/update-ca-certificates"])
+        mock_subprocess.reset_mock()
+
+        # returns false when subprocess hits an exception
+        mock_subprocess.side_effect = CalledProcessError("Command", "Mock Subprocess Call Error")
+        self.assertFalse(self.helper.update_tls_certificates())
+
+    def test_render_checks(self):
+        """Test that NPRE is called to add KSC checks."""
+        # TODO
+        pass
+
+    @mock.patch("charmhelpers.fetch.snap.subprocess.check_call")
+    def test_install_kubectl(self, mock_snap_subprocess):
+        """Test install kubectl snap helper function."""
+        self.assertTrue(self.helper.install_kubectl())
+        channel = self.config.get("channel")
+        mock_snap_subprocess.assert_called_with(
+            ["snap", "install", "--classic", "--channel={}".format(channel), "kubectl"], env=os.environ
+        )
+
+    @mock.patch("charmhelpers.fetch.snap.subprocess.check_call")
+    def test_install_snap_failure(self, mock_snap_subprocess):
+        """Test response to a failed install event."""
+        error = subprocess.CalledProcessError("cmd", "Install failed")
+        error.returncode = 1
+        mock_snap_subprocess.return_value = 1
+        mock_snap_subprocess.side_effect = error
+        self.assertFalse(self.helper.install_kubectl())
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/tests/unit/test_plugins.py b/tests/unit/test_plugins.py
new file mode 100644
index 0000000..42d1697
--- /dev/null
+++ b/tests/unit/test_plugins.py
@@ -0,0 +1,65 @@
+"""Unit tests for Kubernetes Service Checks NRPE Plugins."""
+import unittest
+
+import check_kubernetes_api
+import mock
+
+
+class TestKSCPlugins(unittest.TestCase):
+    """Test cases for Kubernetes Service Checks NRPE plugins."""
+
+    @mock.patch("check_kubernetes_api.sys.exit")
+    @mock.patch("check_kubernetes_api.print")
+    def test_nagios_exit(self, mock_print, mock_sys_exit):
+        """Test the nagios_exit function."""
+        msg = "Test message"
+        for code, status in check_kubernetes_api.NAGIOS_STATUS.items():
+            expected_output = "{}: {}".format(status, msg)
+            check_kubernetes_api.nagios_exit(code, msg)
+
+            mock_print.assert_called_with(expected_output)
+            mock_sys_exit.assert_called_with(code)
+
+    @mock.patch("check_kubernetes_api.urllib3.PoolManager")
+    def test_kubernetes_health_ssl(self, mock_http_pool_manager):
+        """Test the check k8s health function called with expected ssl params."""
+        host_address = "https://1.1.1.1:1111"
+        token = "0123456789abcdef"
+        disable_ssl = True
+
+        mock_http_pool_manager.return_value.status = 200
+        mock_http_pool_manager.return_value.data = b"ok"
+
+        check_kubernetes_api.check_kubernetes_health(host_address, token, disable_ssl)
+        mock_http_pool_manager.assert_called_with(cert_reqs="CERT_NONE", assert_hostname=False)
+
+        disable_ssl = False
+        check_kubernetes_api.check_kubernetes_health(host_address, token, disable_ssl)
+        mock_http_pool_manager.assert_called_with()
+
+    @mock.patch("check_kubernetes_api.urllib3.PoolManager")
+    def test_kubernetes_health_status(self, mock_http_pool_manager):
+        """Test kubernetes health function."""
+        host_address = "https://1.1.1.1:1111"
+        token = "0123456789abcdef"
+        ssl_ca = "test/cert/path"
+
+        mock_http_pool_manager.return_value.request.return_value.status = 200
+        mock_http_pool_manager.return_value.request.return_value.data = b"ok"
+
+        # verify status OK
+        status, _ = check_kubernetes_api.check_kubernetes_health(host_address, token, ssl_ca)
+        self.assertEqual(status, check_kubernetes_api.NAGIOS_STATUS_OK)
+        mock_http_pool_manager.return_value.request.assert_called_once_with(
+            "GET", "{}/healthz".format(host_address), headers={"Authorization": "Bearer {}".format(token)}
+        )
+
+        mock_http_pool_manager.return_value.request.return_value.status = 500
+        mock_http_pool_manager.return_value.request.return_value.data = b"ok"
+        status, _ = check_kubernetes_api.check_kubernetes_health(host_address, token, ssl_ca)
+        self.assertEqual(status, check_kubernetes_api.NAGIOS_STATUS_CRITICAL)
+
+        mock_http_pool_manager.return_value.request.return_value.status = 200
+        mock_http_pool_manager.return_value.request.return_value.data = b"not ok"
+        status, _ = check_kubernetes_api.check_kubernetes_health(host_address, token, ssl_ca)
+        self.assertEqual(status, check_kubernetes_api.NAGIOS_STATUS_WARNING)
diff --git a/tox.ini b/tox.ini
new file mode 100644
index 0000000..2f92371
--- /dev/null
+++ b/tox.ini
@@ -0,0 +1,52 @@
+[tox]
+skipsdist = True
+envlist = unit, func
+skip_missing_interpreters = True
+
+[testenv]
+basepython = python3
+setenv =
+  PYTHONPATH = {toxinidir}/lib/:{toxinidir}
+passenv =
+  HOME
+  MODEL_SETTINGS
+  CHARM_BUILD_DIR
+
+[testenv:unit]
+commands = 
+    coverage run -m unittest discover -s {toxinidir}/tests/unit -v
+    coverage report \
+	--omit tests/*,mod/*,.tox/*
+    coverage html \
+	--omit tests/*,mod/*,.tox/*
+deps = -r{toxinidir}/tests/unit/requirements.txt
+
+[testenv:func]
+changedir = {toxinidir}/tests/functional
+commands = functest-run-suite {posargs}
+deps = -r{toxinidir}/tests/functional/requirements.txt
+
+[testenv:lint]
+commands =
+    flake8
+    black --check --line-length 120 --exclude /(\.eggs|\.git|\.tox|\.venv|build|dist|charmhelpers|mod)/ .
+deps =
+    black
+    flake8
+    flake8-docstrings
+    flake8-import-order
+    pep8-naming
+    flake8-colors
+
+[flake8]
+exclude =
+    .git,
+    __pycache__,
+    .tox,
+    mod,
+max-line-length = 120
+max-complexity = 10
+import-order-style = google
+
+[isort]
+force_to_top=setuppath