Merge ~hloeung/charm-k8s-content-cache:master into charm-k8s-content-cache:master

[Haw Loeung]

Implement initial charm functionality and add unit tests

It works. Example is proxying / caching of archive.ubuntu.com.

| ubuntu@juju-87625f-hloeung-65:~/haw/docker$ jsft
| Model Controller Cloud/Region Version SLA Timestamp
| charm-k8s micro microk8s/localhost 2.8.1 unsupported 03:20:19Z
|
| App Version Status Scale Charm Store Rev OS Address Notes
| content-cache active 1 charm-k8s-content-cache local 22 kubernetes 10.152.183.163
|
| Unit Workload Agent Address Ports Message
| content-cache/42* active idle 10.1.75.118 80/TCP

Generated Nginx config - https://paste.ubuntu.com/p/yCg6W6pMw6/

Test against the unit itself (10.1.75.118) - https://paste.ubuntu.com/p/Wq3tp5JnTQ/

With the access logs:

| ubuntu@juju-87625f-hloeung-65:~/haw/docker$ kubectl -n charm-k8s logs content-cache-774767fd45-njkzs | grep -v 'kube-probe'
| - - - [08/Sep/2020:03:02:11 +0000] "GET / HTTP/1.0" 200 965 "-" "-" 0.281 MISS 0.128
| - - - [08/Sep/2020:03:02:21 +0000] "GET / HTTP/1.0" 200 964 "-" "-" 0.580 HIT -
| - - - [08/Sep/2020:03:04:36 +0000] "GET / HTTP/1.0" 200 964 "-" "-" 0.203 HIT -

kubectl describe ingress - https://paste.ubuntu.com/p/qJ3ZkqDWp5/

Test against the ingress address (10.152.183.163) - https://paste.ubuntu.com/p/qGT2cngrwJ/

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

f29809e... by Haw Loeung on 2020-09-08

Add additional logging

8ababba... by Haw Loeung on 2020-09-08

Update / set appropriate status messages when active

[Barry Price (barryprice)]

Looks good, left a few inline comments - you could also consider adding a lint target for the shell script(s) included under the docker/ dir - charm-k8s-bind uses shellcheck for this.

Also seems to be missing the standard COPYRIGHT/LICENSE files, plus the boilerplate copyright headers in indidivual files, but code itself looks good.

b11a529... by Haw Loeung on 2020-09-08

Fixed based on review

9946b92... by Haw Loeung on 2020-09-08

Work around lack of JUJU_UNIT - LP:1894782

4b3c7ec... by Haw Loeung on 2020-09-08

README, README now.

4e660a4... by Haw Loeung on 2020-09-08

shellcheck fixes per review

53a009b... by Haw Loeung on 2020-09-08

Fixed comment and reference right LICENSE file

d98e4e5... by Haw Loeung on 2020-09-08

Rename charm name removing references to charm and k8s so consistent with other K8s charms

[Tom Haddon (mthaddon)]

As discussed we'll want to break this up into two MPs, one with non-operator framework changes that we can review/approve/merge internally and then another with operator framework changes (src/charm.py + unit tests) that we can review internally and then put up for review with the charmcrafters LP team.

One comment inline about the name of the charm to deploy, which as discussed means we want to update metadata.yaml so the name of the charm is "content-cache".

Also, just a note that we'll want docstrings on all unit tests for when it comes to that MP.

Thanks!

[Haw Loeung (hloeung)]

Split up.

* charm and associated unit tests:

| https://code.launchpad.net/~hloeung/charm-k8s-content-cache/+git/charm-k8s-content-cache/+merge/390393

* The rest of the stuff (non-charm):

| https://code.launchpad.net/~hloeung/charm-k8s-content-cache/+git/charm-k8s-content-cache/+merge/390392

Unmerged commits

d98e4e5... by Haw Loeung on 2020-09-08

Rename charm name removing references to charm and k8s so consistent with other K8s charms

53a009b... by Haw Loeung on 2020-09-08

Fixed comment and reference right LICENSE file

4e660a4... by Haw Loeung on 2020-09-08

shellcheck fixes per review

4b3c7ec... by Haw Loeung on 2020-09-08

README, README now.

9946b92... by Haw Loeung on 2020-09-08

Work around lack of JUJU_UNIT - LP:1894782

b11a529... by Haw Loeung on 2020-09-08

Fixed based on review

8ababba... by Haw Loeung on 2020-09-08

Update / set appropriate status messages when active

f29809e... by Haw Loeung on 2020-09-08

Add additional logging

507feda... by Haw Loeung on 2020-09-08

Keep it simple, map back to Nginx' proxy_pass which takes a URL

baf5812... by Haw Loeung on 2020-09-08

Allow overriding client_max_body_size / nginx.ingress.kubernetes.io/proxy-body-size

diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..9759e36
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,8 @@
+*~
+*.charm
+*.py[cod]
+*.swp
+.coverage
+.tox/
+build/
+__pycache__/
diff --git a/.jujuignore b/.jujuignore
index 6ccd559..4365792 100644
--- a/.jujuignore
+++ b/.jujuignore
@@ -1,3 +1,13 @@
-/venv
-*.py[cod]
+/venv/
+*~
 *.charm
+*.py[cod]
+*.swp
+.coverage
+.gitignore
+.tox/
+Makefile
+build/
+docker/
+tests/
+__pycache__/
diff --git a/COPYRIGHT b/COPYRIGHT
new file mode 100644
index 0000000..1889423
--- /dev/null
+++ b/COPYRIGHT
@@ -0,0 +1,16 @@
+Format: http://dep.debian.net/deps/dep5/
+
+Files: *
+Copyright: Copyright 2020, Canonical Ltd.
+License: GPL-3
+ This program is free software: you can redistribute it and/or modify
+ it under the terms of the GNU General Public License version 3, as
+ published by the Free Software Foundation.
+ .
+ This program is distributed in the hope that it will be useful,
+ but WITHOUT ANY WARRANTY; without even the implied warranties of
+ MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+ PURPOSE.  See the GNU General Public License for more details.
+ .
+ You should have received a copy of the GNU General Public License
+ along with this program.  If not, see <http://www.gnu.org/licenses/>.
diff --git a/Makefile b/Makefile
new file mode 100644
index 0000000..af2c5d1
--- /dev/null
+++ b/Makefile
@@ -0,0 +1,24 @@
+# Copyright (C) 2020 Canonical Ltd.
+# See LICENSE file for licensing details.
+
+lint:
+	@echo "Normalising python layout with black."
+	@tox -e black
+	@echo "Running flake8"
+	@tox -e lint
+
+# We actually use the build directory created by charmcraft,
+# but the .charm file makes a much more convenient sentinel.
+unittest: content-cache.charm
+	@tox -e unit
+
+test: lint unittest
+
+clean:
+	@echo "Cleaning files"
+	@git clean -fXd
+
+content-cache.charm: src/*.py requirements.txt
+	charmcraft build
+
+.PHONY: lint test unittest clean
diff --git a/README.md b/README.md
index aabaf43..3c24b62 100644
--- a/README.md
+++ b/README.md
@@ -2,27 +2,33 @@
 
 ## Description
 
-TODO: fill out the description
+Deploy content caching layer into K8s.
 
 ## Usage
 
-TODO: explain how to use the charm
+Build the docker image:
+
+    `cd docker`
+    `docker build . -t myimage:v<revision>`
+    `docker tag myimage:v<revision> localhost:32000/myimage:v<revision>`
+    `docker push localhost:32000/myimage:v<revision>`
+
+Deploy:
+
+    `juju deploy content-cache.charm --config image_path=localhost:32000/myimage:v<revision> --config site=mysite.local --config backend=http://mybackend.local:80`
+
+### Test Deployment
+
+`curl --resolve mysite.local:80:<ingress IP> http://mysite.local`
 
 ### Scale Out Usage
 
-...
+Just run `juju add-unit <application name>`.
 
 ## Developing
 
-Create and activate a virtualenv,
-and install the development requirements,
-
-    virtualenv -p python3 venv
-    source venv/bin/activate
-    pip install -r requirements-dev.txt
+Just run `make lint`.
 
 ## Testing
 
-Just run `run_tests`:
-
-    ./run_tests
+Just run `make unittest`.
diff --git a/config.yaml b/config.yaml
index 1cf64d6..e25deb2 100644
--- a/config.yaml
+++ b/config.yaml
@@ -1,6 +1,61 @@
 options:
-  sites:
+  image_path:
     type: string
     description: >
-      YAML-formatted virtual hosts/sites. See the README.md for more details
-      and examples.
+      The location of the image to use, e.g. "localhost:32000/myimage:latest"
+
+      This setting is required.
+  image_username:
+    type: string
+    description: >
+      The username for accessing the registry specified in image_path.
+    default: ""
+  image_password:
+    type: string
+    description: >
+      The password associated with image_username for accessing the registry
+      specified in image_path.
+    default: ""
+  site:
+    type: string
+    description: >
+      The site name, e.g. "mysite.local"
+
+      This setting is required.
+  backend:
+    type: string
+    description: >
+      The backend to use for site, e.g. "http://mybackend.local:80"
+
+      This setting is required.
+  cache_inactive_time:
+    type: string
+    description: >
+      The maximum age/time inactive objects are stored in cache.
+    default: "10m"
+  cache_max_size:
+    type: string
+    description: >
+      The size of the Nginx storage cache.
+    default: "10G"
+  cache_use_stale:
+    type: string
+    description: >
+      Determines in which cases a stale cached response can be used
+      during communication with the proxied server.
+    default: "error timeout updating http_500 http_502 http_503 http_504"
+  cache_valid:
+    type: string
+    decription: >
+      Sets caching time for different response codes
+    default: "200 1h"
+  client_max_body_size:
+    type: string
+    description: >
+      Override max. request body size (default 1m).
+    default: ""
+  tls_secret_name:
+    type: string
+    description: >
+      The name of the K8s secret to be associated with the ingress resource.
+    default: ""
diff --git a/docker/Dockerfile b/docker/Dockerfile
new file mode 100644
index 0000000..7f89b21
--- /dev/null
+++ b/docker/Dockerfile
@@ -0,0 +1,27 @@
+# Copyright (C) 2020 Canonical Ltd.
+# See LICENSE file for licensing details.
+
+FROM ubuntu:latest
+
+ENV LANG C.UTF-8
+ENV DEBIAN_FRONTEND noninteractive
+
+# Don't install recommends to keep image small and avoid nasty surprises
+# e.g. rpcbind being pulled in by nrpe.
+RUN apt-get -qy update && \
+    apt-get -qy dist-upgrade --no-install-recommends && \
+    apt-get -qy install --no-install-recommends nginx-light && \
+    apt-get -qy clean && \
+    rm -f /var/lib/apt/lists/*_*
+
+RUN mkdir -p /srv/content-cache
+
+COPY entrypoint.sh /srv/content-cache
+COPY files /srv/content-cache/files/
+COPY templates /srv/content-cache/templates/
+
+ENTRYPOINT ["/srv/content-cache/entrypoint.sh"]
+
+CMD ["nginx", "-g", "daemon off;"]
+
+EXPOSE 80
diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
new file mode 100755
index 0000000..1f3fb10
--- /dev/null
+++ b/docker/entrypoint.sh
@@ -0,0 +1,22 @@
+#!/bin/sh
+
+# Copyright (C) 2020 Canonical Ltd.
+# See LICENSE file for licensing details.
+
+set -eu
+
+# https://pempek.net/articles/2013/07/08/bash-sh-as-template-engine/
+render_template() {
+    eval "echo \"$(cat "$1")\""
+}
+
+cp /srv/content-cache/files/nginx-logging-format.conf /etc/nginx/conf.d/nginx-logging-format.conf
+
+# https://bugs.launchpad.net/juju/+bug/1894782
+JUJU_UNIT=$(basename /var/lib/juju/tools/unit-* | sed -e 's/^unit-//' -e 's/-\([0-9]\+\)$/\/\1/')
+export JUJU_UNIT
+
+render_template /srv/content-cache/templates/nginx_cfg.tmpl > /etc/nginx/sites-available/default
+
+# Run the real command
+exec "$@"
diff --git a/docker/files/nginx-logging-format.conf b/docker/files/nginx-logging-format.conf
new file mode 100644
index 0000000..8591b3e
--- /dev/null
+++ b/docker/files/nginx-logging-format.conf
@@ -0,0 +1,4 @@
+log_format content_cache '$http_x_forwarded_for - $remote_user [$time_local] '
+                         '"$request" $status $bytes_sent '
+                         '"$http_referer" "$http_user_agent" $request_time '
+                         '$upstream_cache_status $upstream_response_time';
diff --git a/docker/templates/nginx_cfg.tmpl b/docker/templates/nginx_cfg.tmpl
new file mode 100644
index 0000000..02694a6
--- /dev/null
+++ b/docker/templates/nginx_cfg.tmpl
@@ -0,0 +1,32 @@
+proxy_cache_path ${NGINX_CACHE_PATH} use_temp_path=off levels=1:2 keys_zone=${NGINX_KEYS_ZONE}:10m inactive=${NGINX_CACHE_INACTIVE_TIME} max_size=${NGINX_CACHE_MAX_SIZE};
+
+server {
+    server_name ${NGINX_SITE_NAME};
+    listen *:80;
+
+    client_max_body_size ${NGINX_CLIENT_MAX_BODY_SIZE};
+
+    port_in_redirect off;
+    absolute_redirect off;
+
+    location / {
+        proxy_pass \"${NGINX_BACKEND}\";
+        proxy_set_header Host \"${NGINX_SITE_NAME}\";
+        # Removed the following headers to avoid cache poisoning.
+        proxy_set_header Forwarded \"\";
+        proxy_set_header X-Forwarded-Host \"\";
+        proxy_set_header X-Forwarded-Port \"\";
+        proxy_set_header X-Forwarded-Proto \"\";
+        proxy_set_header X-Forwarded-Scheme \"\";
+
+        add_header X-Cache-Status \"\$upstream_cache_status from ${JUJU_UNIT} (${HOSTNAME})\";
+
+        proxy_force_ranges on;
+        proxy_cache ${NGINX_KEYS_ZONE};
+        proxy_cache_use_stale ${NGINX_CACHE_USE_STALE};
+        proxy_cache_valid ${NGINX_CACHE_VALID};
+    }
+
+    access_log /dev/stdout content_cache;
+    error_log /dev/stdout info;
+}
diff --git a/metadata.yaml b/metadata.yaml
index 2874a21..dacd17e 100644
--- a/metadata.yaml
+++ b/metadata.yaml
@@ -1,4 +1,4 @@
-name: charm-k8s-content-cache
+name: content-cache
 description: |
   Useful for providing local mirrors of HTTP servers and building
   content delivery networks (CDN).
diff --git a/src/charm.py b/src/charm.py
index c5ab80b..1c27501 100755
--- a/src/charm.py
+++ b/src/charm.py
@@ -1,38 +1,210 @@
 #!/usr/bin/env python3
-# Copyright 2020 hloeung
+
+# Copyright (C) 2020 Canonical Ltd.
 # See LICENSE file for licensing details.
 
+import hashlib
 import logging
 
 from ops.charm import CharmBase
 from ops.main import main
 from ops.framework import StoredState
+from ops.model import (
+    ActiveStatus,
+    BlockedStatus,
+    MaintenanceStatus,
+)
 
 logger = logging.getLogger(__name__)
 
+CACHE_PATH = '/var/lib/nginx/proxy/cache'
+CONTAINER_PORT = 80
+REQUIRED_JUJU_CONFIGS = ['image_path', 'site', 'backend']
+
 
-class CharmK8SContentCacheCharm(CharmBase):
+class ContentCacheCharm(CharmBase):
     _stored = StoredState()
 
     def __init__(self, *args):
         super().__init__(*args)
+
+        self.framework.observe(self.on.start, self._on_start)
         self.framework.observe(self.on.config_changed, self._on_config_changed)
-        self.framework.observe(self.on.fortune_action, self._on_fortune_action)
-        self._stored.set_default(things=[])
-
-    def _on_config_changed(self, _):
-        current = self.model.config["thing"]
-        if current not in self._stored.things:
-            logger.debug("found a new thing: %r", current)
-            self._stored.things.append(current)
-
-    def _on_fortune_action(self, event):
-        fail = event.params["fail"]
-        if fail:
-            event.fail(fail)
+        self.framework.observe(self.on.leader_elected, self._on_leader_elected)
+        self.framework.observe(self.on.upgrade_charm, self._on_upgrade_charm)
+
+        self._stored.set_default()
+
+    def _on_start(self, event) -> None:
+        self.model.unit.status = ActiveStatus('Started')
+
+    def _on_config_changed(self, event) -> None:
+        if not self.model.unit.is_leader():
+            logger.info('Spec changes ignored by non-leader')
+            self.unit.status = ActiveStatus('Ready')
+            return
+        msg = 'Configuring pod (config-changed)'
+        logger.info(msg)
+        self.model.unit.status = MaintenanceStatus(msg)
+
+        self.configure_pod(event)
+
+    def _on_leader_elected(self, event) -> None:
+        msg = 'Configuring pod (leader-elected)'
+        logger.info(msg)
+        self.model.unit.status = MaintenanceStatus(msg)
+        self.configure_pod(event)
+
+    def _on_upgrade_charm(self, event) -> None:
+        if not self.model.unit.is_leader():
+            logger.info('Spec changes ignored by non-leader')
+            self.unit.status = ActiveStatus('Ready')
+            return
+        msg = 'Configuring pod (upgrade-charm)'
+        logger.info(msg)
+        self.model.unit.status = MaintenanceStatus(msg)
+        self.configure_pod(event)
+
+    def configure_pod(self, event) -> None:
+        missing = self._missing_charm_configs()
+        if missing:
+            msg = 'Required config(s) empty: {}'.format(', '.join(sorted(missing)))
+            logger.warning(msg)
+            self.unit.status = BlockedStatus(msg)
+            return
+
+        msg = 'Assembling K8s ingress spec'
+        logger.info(msg)
+        self.unit.status = MaintenanceStatus(msg)
+        ingress_spec = self._make_k8s_ingress_spec()
+        k8s_resources = {'kubernetesResources': {'ingressResources': ingress_spec}}
+
+        msg = 'Assembling pod spec'
+        logger.info(msg)
+        self.unit.status = MaintenanceStatus(msg)
+        pod_spec = self._make_pod_spec()
+
+        msg = 'Setting pod spec'
+        logger.info(msg)
+        self.unit.status = MaintenanceStatus(msg)
+        self.model.pod.set_spec(pod_spec, k8s_resources=k8s_resources)
+
+        msg = 'Done applying updated pod spec'
+        logger.info(msg)
+        self.unit.status = ActiveStatus('Ready')
+
+    def _generate_keys_zone(self, name):
+        return '{}-cache'.format(hashlib.md5(name.encode('UTF-8')).hexdigest()[0:12])
+
+    def _make_k8s_ingress_spec(self) -> list:
+        config = self.model.config
+
+        annotations = {}
+        ingress = {
+            'name': '{}-ingress'.format(self.app.name),
+            'spec': {
+                'rules': [
+                    {
+                        'host': config['site'],
+                        'http': {
+                            'paths': [
+                                {'path': '/', 'backend': {'serviceName': self.app.name, 'servicePort': CONTAINER_PORT}}
+                            ],
+                        },
+                    }
+                ],
+            },
+        }
+
+        client_max_body_size = config.get('client_max_body_size')
+        if client_max_body_size:
+            annotations['nginx.ingress.kubernetes.io/proxy-body-size'] = client_max_body_size
+
+        tls_secret_name = config.get('tls_secret_name')
+        if tls_secret_name:
+            ingress['spec']['tls'] = [{'hosts': config['site'], 'secretName': tls_secret_name}]
         else:
-            event.set_results({"fortune": "A bug in the code is worth two in the documentation."})
+            annotations['nginx.ingress.kubernetes.io/ssl-redirect'] = 'false'
+
+        if annotations:
+            ingress['annotations'] = annotations
+
+        return [ingress]
+
+    def _make_pod_spec(self) -> dict:
+        config = self.model.config
+
+        image_details = {
+            'imagePath': config['image_path'],
+        }
+        if config.get('image_username', None):
+            image_details.update({'username': config['image_username'], 'password': config['image_password']})
+
+        pod_config = self._make_pod_config()
+
+        pod_spec = {
+            'version': 3,  # otherwise resources are ignored
+            'containers': [
+                {
+                    'name': self.app.name,
+                    'envConfig': pod_config,
+                    'imageDetails': image_details,
+                    'imagePullPolicy': 'Always',
+                    'kubernetes': {
+                        'livenessProbe': {
+                            'httpGet': {'path': '/', 'port': CONTAINER_PORT},
+                            'initialDelaySeconds': 3,
+                            'periodSeconds': 3,
+                        },
+                        'readinessProbe': {
+                            'httpGet': {'path': '/', 'port': CONTAINER_PORT},
+                            'initialDelaySeconds': 3,
+                            'periodSeconds': 3,
+                        },
+                    },
+                    'ports': [{'containerPort': CONTAINER_PORT, 'protocol': 'TCP'}],
+                    'volumeConfig': [
+                        {
+                            'name': 'cache-volume',
+                            'mountPath': CACHE_PATH,
+                            'emptyDir': {'sizeLimit': config['cache_max_size']},
+                        }
+                    ],
+                }
+            ],
+        }
+
+        return pod_spec
+
+    def _make_pod_config(self) -> dict:
+        config = self.model.config
+
+        client_max_body_size = '1m'
+        if config.get('client_max_body_size', ''):
+            client_max_body_size = config.get('client_max_body_size')
+
+        pod_config = {
+            'NGINX_BACKEND': config['backend'],
+            'NGINX_CACHE_INACTIVE_TIME': config.get('cache_inactive_time', '10m'),
+            'NGINX_CACHE_MAX_SIZE': config.get('cache_max_size', '10G'),
+            'NGINX_CACHE_PATH': CACHE_PATH,
+            'NGINX_CACHE_USE_STALE': config['cache_use_stale'],
+            'NGINX_CACHE_VALID': config['cache_valid'],
+            'NGINX_CLIENT_MAX_BODY_SIZE': client_max_body_size,
+            'NGINX_KEYS_ZONE': self._generate_keys_zone(config['site']),
+            'NGINX_SITE_NAME': config['site'],
+        }
+
+        return pod_config
+
+    def _missing_charm_configs(self) -> list:
+        config = self.model.config
+        missing = []
+
+        missing.extend([setting for setting in REQUIRED_JUJU_CONFIGS if not config[setting]])
+
+        return sorted(list(set(missing)))
 
 
-if __name__ == "__main__":
-    main(CharmK8SContentCacheCharm)
+if __name__ == '__main__':  # pragma: no cover
+    main(ContentCacheCharm)
diff --git a/tests/requirements.txt b/tests/requirements.txt
new file mode 100644
index 0000000..f6c26b8
--- /dev/null
+++ b/tests/requirements.txt
@@ -0,0 +1,4 @@
+freezegun
+mock
+pytest
+pytest-cov
diff --git a/tests/test_charm.py b/tests/test_charm.py
new file mode 100644
index 0000000..5d962a5
--- /dev/null
+++ b/tests/test_charm.py
@@ -0,0 +1,385 @@
+# Copyright (C) 2020 Canonical Ltd.
+# See LICENSE file for licensing details.
+
+import copy
+import unittest
+from unittest import mock
+
+from ops.model import (
+    ActiveStatus,
+    BlockedStatus,
+    MaintenanceStatus,
+)
+from ops.testing import Harness
+from charm import ContentCacheCharm
+
+BASE_CONFIG = {
+    'image_path': 'localhost:32000/myimage:latest',
+    'site': 'mysite.local',
+    'backend': 'http://mybackend.local:80',
+    'cache_max_size': '10G',
+    'cache_use_stale': 'error timeout updating http_500 http_502 http_503 http_504',
+    'cache_valid': '200 1h',
+}
+CACHE_PATH = '/var/lib/nginx/proxy/cache'
+CONTAINER_PORT = 80
+POD_SPEC_TMPL = {
+    'version': 3,
+    'containers': [
+        {
+            'name': 'content-cache',
+            'envConfig': None,
+            'imageDetails': None,
+            'imagePullPolicy': 'Always',
+            'kubernetes': {
+                'livenessProbe': {
+                    'httpGet': {'path': '/', 'port': CONTAINER_PORT},
+                    'initialDelaySeconds': 3,
+                    'periodSeconds': 3,
+                },
+                'readinessProbe': {
+                    'httpGet': {'path': '/', 'port': CONTAINER_PORT},
+                    'initialDelaySeconds': 3,
+                    'periodSeconds': 3,
+                },
+            },
+            'ports': [{'containerPort': CONTAINER_PORT, 'protocol': 'TCP'}],
+            'volumeConfig': None,
+        }
+    ],
+}
+K8S_RESOURCES_TMPL = {
+    'kubernetesResources': {
+        'ingressResources': [
+            {
+                'annotations': {'nginx.ingress.kubernetes.io/ssl-redirect': 'false'},
+                'name': 'content-cache-ingress',
+                'spec': {
+                    'rules': [
+                        {
+                            'host': 'mysite.local',
+                            'http': {
+                                'paths': [
+                                    {
+                                        'backend': {'serviceName': 'content-cache', 'servicePort': 80},
+                                        'path': '/',
+                                    }
+                                ]
+                            },
+                        }
+                    ]
+                },
+            }
+        ]
+    }
+}
+
+
+class TestCharm(unittest.TestCase):
+    def setUp(self):
+        self.maxDiff = None
+        self.harness = Harness(ContentCacheCharm)
+
+    def tearDown(self):
+        # starting from ops 0.8, we also need to do:
+        self.addCleanup(self.harness.cleanup)
+
+    def test_on_start(self):
+        harness = self.harness
+        action_event = mock.Mock()
+
+        harness.begin()
+        harness.charm._on_start(action_event)
+        self.assertEqual(harness.charm.unit.status, ActiveStatus('Started'))
+
+    @mock.patch('charm.ContentCacheCharm.configure_pod')
+    def test_on_config_changed(self, configure_pod):
+        harness = self.harness
+
+        # Intentionally before harness.begin() to avoid firing leadership events.
+        harness.set_leader(True)
+        harness.begin()
+
+        config = copy.deepcopy(BASE_CONFIG)
+        harness.update_config(config)
+        self.assertEqual(harness.charm.unit.status, MaintenanceStatus('Configuring pod (config-changed)'))
+        configure_pod.assert_called_once()
+
+    @mock.patch('charm.ContentCacheCharm.configure_pod')
+    def test_on_config_changed_not_leader(self, configure_pod):
+        harness = self.harness
+
+        # Intentionally before harness.begin() to avoid firing leadership events.
+        harness.set_leader(False)
+        harness.begin()
+
+        config = copy.deepcopy(BASE_CONFIG)
+        harness.update_config(config)
+        self.assertNotEqual(harness.charm.unit.status, MaintenanceStatus('Configuring pod (config-changed)'))
+        configure_pod.assert_not_called()
+
+    @mock.patch('charm.ContentCacheCharm.configure_pod')
+    def test_on_leader_elected(self, configure_pod):
+        harness = self.harness
+
+        harness.begin()
+        # Intentionally after harness.begin() to trigger leadership events.
+        harness.set_leader(True)
+        self.assertEqual(harness.charm.unit.status, MaintenanceStatus('Configuring pod (leader-elected)'))
+        configure_pod.assert_called_once()
+
+    @mock.patch('charm.ContentCacheCharm.configure_pod')
+    def test_on_leader_elected_not_leader(self, configure_pod):
+        harness = self.harness
+
+        harness.begin()
+        # Intentionally after harness.begin() to trigger leadership events.
+        harness.set_leader(False)
+        self.assertNotEqual(harness.charm.unit.status, MaintenanceStatus('Configuring pod (leader-elected)'))
+        configure_pod.assert_not_called()
+
+    @mock.patch('charm.ContentCacheCharm.configure_pod')
+    def test_on_upgrade_charm(self, configure_pod):
+        harness = self.harness
+        action_event = mock.Mock()
+
+        # Disable hooks and fire them manually as that seems to be the
+        # only way to test upgrade-charm.
+        harness.disable_hooks()
+        harness.set_leader(True)
+        harness.begin()
+
+        harness.charm._on_upgrade_charm(action_event)
+        self.assertEqual(harness.charm.unit.status, MaintenanceStatus('Configuring pod (upgrade-charm)'))
+        configure_pod.assert_called_once()
+
+    @mock.patch('charm.ContentCacheCharm.configure_pod')
+    def test_on_upgrade_charm_not_leader(self, configure_pod):
+        harness = self.harness
+        action_event = mock.Mock()
+
+        # Disable hooks and fire them manually as that seems to be the
+        # only way to test upgrade-charm.
+        harness.disable_hooks()
+        harness.set_leader(False)
+        harness.begin()
+
+        harness.charm._on_upgrade_charm(action_event)
+        self.assertNotEqual(harness.charm.unit.status, MaintenanceStatus('Configuring pod (upgrade-charm)'))
+        configure_pod.assert_not_called()
+
+    @mock.patch('charm.ContentCacheCharm._make_pod_spec')
+    def test_configure_pod(self, make_pod_spec):
+        harness = self.harness
+
+        harness.set_leader(True)
+        harness.begin()
+
+        config = copy.deepcopy(BASE_CONFIG)
+        harness.update_config(config)
+        make_pod_spec.assert_called_once()
+        self.assertEqual(harness.charm.unit.status, ActiveStatus('Ready'))
+        pod_spec = harness.charm._make_pod_spec()
+        k8s_resources = copy.deepcopy(K8S_RESOURCES_TMPL)
+        self.assertEqual(harness.get_pod_spec(), (pod_spec, k8s_resources))
+
+    @mock.patch('charm.ContentCacheCharm._make_pod_spec')
+    def test_configure_pod_missing_configs(self, make_pod_spec):
+        harness = self.harness
+
+        harness.set_leader(True)
+        harness.begin()
+
+        config = copy.deepcopy(BASE_CONFIG)
+        config['site'] = None
+        harness.update_config(config)
+        make_pod_spec.assert_not_called()
+        self.assertEqual(harness.charm.unit.status, BlockedStatus('Required config(s) empty: site'))
+        self.assertEqual(harness.get_pod_spec(), None)
+
+    def test_generate_keys_zone(self):
+        harness = self.harness
+
+        harness.disable_hooks()
+        harness.begin()
+
+        expected = '39c631ffb52d-cache'
+        self.assertEqual(harness.charm._generate_keys_zone('mysite.local'), expected)
+        expected = '8b79f9e4b3e8-cache'
+        self.assertEqual(harness.charm._generate_keys_zone('my-really-really-really-long-site-name.local'), expected)
+        expected = 'd41d8cd98f00-cache'
+        self.assertEqual(harness.charm._generate_keys_zone(''), expected)
+
+    def test_make_k8s_ingress_spec(self):
+        harness = self.harness
+
+        harness.disable_hooks()
+        harness.begin()
+
+        config = copy.deepcopy(BASE_CONFIG)
+        harness.update_config(config)
+        k8s_resources = copy.deepcopy(K8S_RESOURCES_TMPL)
+        expected = k8s_resources['kubernetesResources']['ingressResources']
+        self.assertEqual(harness.charm._make_k8s_ingress_spec(), expected)
+
+    def test_make_k8s_ingress_spec_client_max_body_size(self):
+        harness = self.harness
+
+        harness.disable_hooks()
+        harness.begin()
+
+        config = copy.deepcopy(BASE_CONFIG)
+        config['client_max_body_size'] = '32m'
+        harness.update_config(config)
+        k8s_resources = copy.deepcopy(K8S_RESOURCES_TMPL)
+        t = k8s_resources['kubernetesResources']['ingressResources'][0]['annotations']
+        t['nginx.ingress.kubernetes.io/proxy-body-size'] = '32m'
+        expected = k8s_resources['kubernetesResources']['ingressResources']
+        self.assertEqual(harness.charm._make_k8s_ingress_spec(), expected)
+
+    def test_make_k8s_ingress_spec_tls_secrets(self):
+        harness = self.harness
+
+        harness.disable_hooks()
+        harness.begin()
+
+        config = copy.deepcopy(BASE_CONFIG)
+        config['tls_secret_name'] = '{}-tls'.format(config['site'])
+        harness.update_config(config)
+        k8s_resources = copy.deepcopy(K8S_RESOURCES_TMPL)
+        t = k8s_resources['kubernetesResources']['ingressResources'][0]
+        t.pop('annotations')
+        t['spec']['tls'] = [{'hosts': 'mysite.local', 'secretName': 'mysite.local-tls'}]
+        expected = k8s_resources['kubernetesResources']['ingressResources']
+        self.assertEqual(harness.charm._make_k8s_ingress_spec(), expected)
+
+    def test_make_pod_spec(self):
+        harness = self.harness
+
+        harness.set_leader(True)
+        harness.begin()
+
+        config = copy.deepcopy(BASE_CONFIG)
+        harness.update_config(config)
+        spec = copy.deepcopy(POD_SPEC_TMPL)
+        t = spec['containers'][0]
+        t['envConfig'] = harness.charm._make_pod_config()
+        t['imageDetails'] = {'imagePath': 'localhost:32000/myimage:latest'}
+        t['volumeConfig'] = [
+            {'name': 'cache-volume', 'mountPath': '/var/lib/nginx/proxy/cache', 'emptyDir': {'sizeLimit': '10G'}}
+        ]
+        k8s_resources = copy.deepcopy(K8S_RESOURCES_TMPL)
+        expected = (spec, k8s_resources)
+        self.assertEqual(harness.get_pod_spec(), expected)
+
+    def test_make_pod_spec_image_username(self):
+        harness = self.harness
+
+        harness.set_leader(True)
+        harness.begin()
+
+        config = copy.deepcopy(BASE_CONFIG)
+        config['image_username'] = 'myuser'
+        config['image_password'] = 'mypassword'
+        harness.update_config(config)
+        spec = copy.deepcopy(POD_SPEC_TMPL)
+        t = spec['containers'][0]
+        t['envConfig'] = harness.charm._make_pod_config()
+        t['imageDetails'] = {
+            'imagePath': 'localhost:32000/myimage:latest',
+            'username': 'myuser',
+            'password': 'mypassword',
+        }
+        t['volumeConfig'] = [
+            {'name': 'cache-volume', 'mountPath': '/var/lib/nginx/proxy/cache', 'emptyDir': {'sizeLimit': '10G'}}
+        ]
+        k8s_resources = copy.deepcopy(K8S_RESOURCES_TMPL)
+        expected = (spec, k8s_resources)
+        self.assertEqual(harness.get_pod_spec(), expected)
+
+    def test_make_pod_spec_cache_max_size(self):
+        harness = self.harness
+
+        harness.set_leader(True)
+        harness.begin()
+
+        config = copy.deepcopy(BASE_CONFIG)
+        config['cache_max_size'] = '201G'
+        harness.update_config(config)
+        spec = copy.deepcopy(POD_SPEC_TMPL)
+        t = spec['containers'][0]
+        t['envConfig'] = harness.charm._make_pod_config()
+        t['imageDetails'] = {'imagePath': 'localhost:32000/myimage:latest'}
+        t['volumeConfig'] = [
+            {'name': 'cache-volume', 'mountPath': '/var/lib/nginx/proxy/cache', 'emptyDir': {'sizeLimit': '201G'}}
+        ]
+        k8s_resources = copy.deepcopy(K8S_RESOURCES_TMPL)
+        expected = (spec, k8s_resources)
+        self.assertEqual(harness.get_pod_spec(), expected)
+
+    def test_make_pod_config(self):
+        harness = self.harness
+
+        harness.disable_hooks()
+        harness.begin()
+
+        config = copy.deepcopy(BASE_CONFIG)
+        harness.update_config(config)
+        expected = {
+            'NGINX_BACKEND': 'http://mybackend.local:80',
+            'NGINX_CACHE_INACTIVE_TIME': '10m',
+            'NGINX_CACHE_MAX_SIZE': '10G',
+            'NGINX_CACHE_PATH': CACHE_PATH,
+            'NGINX_CACHE_USE_STALE': 'error timeout updating http_500 http_502 http_503 http_504',
+            'NGINX_CACHE_VALID': '200 1h',
+            'NGINX_CLIENT_MAX_BODY_SIZE': '1m',
+            'NGINX_KEYS_ZONE': '39c631ffb52d-cache',
+            'NGINX_SITE_NAME': 'mysite.local',
+        }
+        self.assertEqual(harness.charm._make_pod_config(), expected)
+
+    def test_make_pod_config_client_max_body_size(self):
+        harness = self.harness
+
+        harness.disable_hooks()
+        harness.begin()
+
+        config = copy.deepcopy(BASE_CONFIG)
+        config['client_max_body_size'] = '50m'
+        harness.update_config(config)
+        expected = {
+            'NGINX_BACKEND': 'http://mybackend.local:80',
+            'NGINX_CACHE_INACTIVE_TIME': '10m',
+            'NGINX_CACHE_MAX_SIZE': '10G',
+            'NGINX_CACHE_PATH': CACHE_PATH,
+            'NGINX_CACHE_USE_STALE': 'error timeout updating http_500 http_502 http_503 http_504',
+            'NGINX_CACHE_VALID': '200 1h',
+            'NGINX_CLIENT_MAX_BODY_SIZE': '50m',
+            'NGINX_KEYS_ZONE': '39c631ffb52d-cache',
+            'NGINX_SITE_NAME': 'mysite.local',
+        }
+        self.assertEqual(harness.charm._make_pod_config(), expected)
+
+    def test_missing_charm_configs(self):
+        harness = self.harness
+
+        harness.disable_hooks()
+        harness.begin()
+
+        config = copy.deepcopy(BASE_CONFIG)
+        harness.update_config(config)
+        expected = []
+        self.assertEqual(harness.charm._missing_charm_configs(), expected)
+
+        config = copy.deepcopy(BASE_CONFIG)
+        config['site'] = None
+        harness.update_config(config)
+        expected = ['site']
+        self.assertEqual(harness.charm._missing_charm_configs(), expected)
+
+        config = copy.deepcopy(BASE_CONFIG)
+        config['image_path'] = None
+        config['site'] = None
+        harness.update_config(config)
+        expected = ['image_path', 'site']
+        self.assertEqual(harness.charm._missing_charm_configs(), expected)
diff --git a/tests/unit/test_charm.py b/tests/unit/test_charm.py
deleted file mode 100644
index 2f3db20..0000000
--- a/tests/unit/test_charm.py
+++ /dev/null
@@ -1,36 +0,0 @@
-# Copyright 2020 hloeung
-# See LICENSE file for licensing details.
-
-import unittest
-from unittest.mock import Mock
-
-from ops.testing import Harness
-from charm import CharmK8SContentCacheCharm
-
-
-class TestCharm(unittest.TestCase):
-    def test_config_changed(self):
-        harness = Harness(CharmK8SContentCacheCharm)
-        # from 0.8 you should also do:
-        # self.addCleanup(harness.cleanup)
-        harness.begin()
-        self.assertEqual(list(harness.charm._stored.things), [])
-        harness.update_config({"thing": "foo"})
-        self.assertEqual(list(harness.charm._stored.things), ["foo"])
-
-    def test_action(self):
-        harness = Harness(CharmK8SContentCacheCharm)
-        harness.begin()
-        # the harness doesn't (yet!) help much with actions themselves
-        action_event = Mock(params={"fail": ""})
-        harness.charm._on_fortune_action(action_event)
-
-        self.assertTrue(action_event.set_results.called)
-
-    def test_action_fail(self):
-        harness = Harness(CharmK8SContentCacheCharm)
-        harness.begin()
-        action_event = Mock(params={"fail": "fail this"})
-        harness.charm._on_fortune_action(action_event)
-
-        self.assertEqual(action_event.fail.call_args, [("fail this",)])
diff --git a/tox.ini b/tox.ini
new file mode 100644
index 0000000..2daecb5
--- /dev/null
+++ b/tox.ini
@@ -0,0 +1,37 @@
+[tox]
+skipsdist=True
+envlist = unit
+skip_missing_interpreters = True
+
+[testenv]
+basepython = python3
+setenv =
+  PYTHONPATH = {toxinidir}/src:{toxinidir}/build/lib:{toxinidir}/build/venv
+
+[testenv:unit]
+commands =
+    pytest \
+      {posargs:-v  --cov=src --cov-report=term-missing --cov-branch}
+deps = -r{toxinidir}/tests/requirements.txt
+       -r{toxinidir}/requirements.txt
+setenv =
+  PYTHONPATH={toxinidir}/src:{toxinidir}/build/lib:{toxinidir}/build/venv
+  TZ=UTC
+
+[testenv:black]
+commands = black --skip-string-normalization --line-length=120 src/ tests/
+deps = black
+
+[testenv:lint]
+commands = flake8 src/ tests/
+# Pin flake8 to 3.7.9 to match focal
+deps =
+    flake8==3.7.9
+
+[flake8]
+exclude =
+    .git,
+    __pycache__,
+    .tox,
+max-line-length = 120
+max-complexity = 10