Merge ~axino/charm-k8s-gunicorn/+git/charm-k8s-gunicorn:relations into charm-k8s-gunicorn:master

[Junien F]

add postgresql and influxdb relations, fix tests, improve documentation and other misc fixes

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Unable to determine commit message from repository - please click "Set commit message" and enter the commit message manually.

[Tom Haddon (mthaddon)]

Two small comments inline. Once those are addressed I think this is ready for review from charmcrafters.

[Tom Haddon (mthaddon)]

Looks good to me, will add charmcrafters for review, but leave my review as is for now. Once we're ready to merge I can add an approving review if needed.

[Facundo Batista (facundo)]

Looks awesome!!

I annotated some inline comments, thanks!!

review: Needs Fixing

[Junien F (axino)]

Thanks ! Fixes incoming

[Junien F (axino)]

Forgot to resubmit

review: Needs Resubmitting

[Facundo Batista (facundo)]

Approved, but please check the inline comments for two minor details. Thanks!!

review: Approve

[Tom Haddon (mthaddon)]

Adding an approving comment so this can be merged.

review: Approve

[Junien F (axino)]

Let's go mergebot !

review: Approve

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision 297c99c68838433f5b8d1c497835b7c19bea3d1d

diff --git a/.jujuignore b/.jujuignore
index 423f33d..ee9772c 100644
--- a/.jujuignore
+++ b/.jujuignore
@@ -1,6 +1,8 @@
 /env
 *.py[cod]
 *.charm
+.pytest_cache
 docker
 *swp
 venv
+*__pycache__*
diff --git a/Makefile b/Makefile
index 3e1801a..965b540 100644
--- a/Makefile
+++ b/Makefile
@@ -1,23 +1,24 @@
-blacken:		
+gunicorn.charm: src/*.py requirements.txt metadata.yaml config.yaml test
+	charmcraft build
+
+blacken:
 	@echo "Normalising python layout with black."
 	@tox -e black
-										
-lint: blacken	
+
+lint: blacken
 	@echo "Running flake8"
-	@tox -e lint	
+	@tox -e lint
 
 # We actually use the build directory created by charmcraft,
 # but the .charm file makes a much more convenient sentinel.
-unittest: gunicorn.charm							
+unittest:
 	@tox -e unit
 
 test: lint unittest
-					
-clean:	
+
+clean:
 	@echo "Cleaning files"
-	@git clean -fXd							
-										
-gunicorn.charm: src/*.py requirements.txt
-	charmcraft build
-					
-.PHONY: lint test unittest clean
+	@git clean -fXd
+
+
+.PHONY: lint test unittest blacken clean
diff --git a/README.md b/README.md
index 646cbc9..b5a44d2 100644
--- a/README.md
+++ b/README.md
@@ -6,9 +6,69 @@ A charm that allows you to deploy your gunicorn application in kubernetes.
 
 ## Usage
 
-juju deploy cs:gunicorn my-awesome-app
-juju config my-awesome-app image\_path=localhost:32000/myapp
+```
+juju deploy cs:gunicorn my-awesome-app --config image_path=localhost:32000/myapp --config external_hostname=my-awesome-app.com
+```
 
 ### Scale Out Usage
 
 juju add-unit my-awesome-app
+
+## OCI image
+
+### Using your own image
+
+You can, of course, supply our own OCI image. gunicorn is expected to listen on
+port 80.
+
+### Using gunicorn-base to build an image
+
+If you have a gunicorn app that's not available via a Docker image, you can use
+the provided `gunicorn-base` image. First, build the base image that's
+available in the `docker/gunicorn-base` directory. You can then use it as a
+base for other images, and there's an example of that in the `docker/app`
+directory.
+
+This example app will simply display all the environment variables given to
+your pods. It can be helpful to see what's available and to debug problems
+related to environment variables.
+
+## Environment variables and relations
+
+This charm has been designed to easily allow you to pass information coming
+from relation data to your pods. This is done by using the `environment` config
+option. This config option is a Jinja2 template for a YAML dict that will be
+added to the environment of your pods.
+
+The context used to render the Jinja2 template is constructed from relation
+data. For example, if you're relating with influxdb, you could do the following :
+```
+juju deploy cs:gunicorn my-awesome-app --config image_path=localhost:32000/myapp --config external_hostname=my-awesome-app.com
+juju config my-awesome-app environment="INFLUXDB_HOST: {{influxdb.hostname}}"
+```
+
+The charm will notice that you're trying to use data from the `influxdb` relation,
+and will block until such a relation is added. Once the relation is added, the
+charm will get the `hostname` from the relation, and will make it available to
+your pod as the `INFLUXDB_HOST` environment variable.
+
+If you want the charm to handle more "basic" relations such as the `influxdb`
+one described above, all you have to do is add the relation to metadata.yaml
+and rebuild the charm (see below).
+
+Some relations, such as the `postgresql` relation, are a bit more complex, in
+that they're managed by a library. Instead of using raw relation data, you use
+the library to get useful and usable information out of the relation. If you
+want to use such a relation, you will need to add a bit more code to make the
+information provided by the library availeble to the Jinja2 context. An example
+is provided in the charm with the `postgresql` relation implementation.
+
+## Building the charm
+
+It's as easy as running :
+```
+make
+```
+
+This will lint and format your code, then run unit tests, and then build the
+charm.
diff --git a/config.yaml b/config.yaml
index 3a87de3..0754ab4 100644
--- a/config.yaml
+++ b/config.yaml
@@ -1,28 +1,35 @@
 options:
   image_path:
     type: string
-    description: |
+    description: >
       The location of the image to use, e.g. "registry.example.com/my_gunicorn_app:v1".
 
       This setting is required.
     default: ''
   image_username:
     type: string
-    description: |
+    description: >
       The username for accessing the registry specified in image_path.
     default: ''
   image_password:
     type: string
-    description: |
+    description: >
       The password associated with image_username for accessing the registry specified in image_path.
     default: ''
   environment:
     type: string
-    description: |
-      This YAML-formatted associative array will be added to the environment of the container.
+    description: >
+      Jinja2 template of a YAML-formatted associative array which will be added
+      to the environment of the container. The context used to render the template
+      is filled with relation data. For example, if you use :
+        DEBUG_LEVEL: 2
+        DB_URI: {{pg.uri}}
+      your pods will get the "DEBUG_LEVEL" environment variable set to 2, and
+      the "DB_URI" environment variable will be set to the "uri" relation data
+      item provided by the "pg" relation.
     default: ''
   external_hostname:
     type: string
-    description: |
-      External hostname this gunicorn app should respond to
+    description: >
+      External hostname this gunicorn app should respond to.
     default: ''
diff --git a/docker/app/app/app.py b/docker/app/app/app.py
index 9d75da5..fbb9fc5 100644
--- a/docker/app/app/app.py
+++ b/docker/app/app/app.py
@@ -1,11 +1,12 @@
 import os
 
+
 def app(environ, start_response):
     status = '200 OK'
-    response_headers = [('Content-type','text/plain')]
+    response_headers = [('Content-type', 'text/plain')]
     start_response(status, response_headers)
     ret = [b'One of the nice things about the new operator framework is how easy it is to get started.\n']
-    for i,x in os.environ.items():
-      ret.append("{}: {}\n".format(i,x).encode('utf-8'))
+    for i, x in os.environ.items():
+        ret.append("{}: {}\n".format(i, x).encode('utf-8'))
 
     return ret
diff --git a/metadata.yaml b/metadata.yaml
index 3cbca9b..4e6d497 100644
--- a/metadata.yaml
+++ b/metadata.yaml
@@ -7,3 +7,10 @@ summary: |
   Gunicorn charm
 series: [kubernetes]
 min-juju-version: 2.8.0  # charm storage in state
+requires:
+  pg:
+    interface: pgsql
+    limit: 1
+  influxdb:
+    interface: influxdb-api
+    limit: 1
diff --git a/pyproject.toml b/pyproject.toml
new file mode 100644
index 0000000..d2f23b9
--- /dev/null
+++ b/pyproject.toml
@@ -0,0 +1,3 @@
+[tool.black]
+skip-string-normalization = true
+line-length = 120
diff --git a/requirements.txt b/requirements.txt
index 2d81d3b..fd6adcd 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1 +1,2 @@
 ops
+ops-lib-pgsql
diff --git a/src/charm.py b/src/charm.py
index 1bbece3..3772ef0 100755
--- a/src/charm.py
+++ b/src/charm.py
@@ -2,6 +2,7 @@
 # Copyright 2020 Canonical Ltd.
 # See LICENSE file for licensing details.
 
+from jinja2 import Environment, BaseLoader, meta
 import logging
 import yaml
 
@@ -14,6 +15,7 @@ from ops.model import (
     BlockedStatus,
     MaintenanceStatus,
 )
+import pgsql
 
 
 logger = logging.getLogger(__name__)
@@ -28,18 +30,72 @@ class GunicornK8sCharmJujuConfigError(Exception):
     pass
 
 
+class GunicornK8sCharmYAMLError(Exception):
+    """Exception raised when parsing YAML fails"""
+
+    pass
+
+
 class GunicornK8sCharm(CharmBase):
     _stored = StoredState()
 
     def __init__(self, *args):
         super().__init__(*args)
 
-        self.framework.observe(self.on.start, self.configure_pod)
-        self.framework.observe(self.on.config_changed, self.configure_pod)
-        self.framework.observe(self.on.leader_elected, self.configure_pod)
-        self.framework.observe(self.on.upgrade_charm, self.configure_pod)
+        self.framework.observe(self.on.start, self._configure_pod)
+        self.framework.observe(self.on.config_changed, self._configure_pod)
+        self.framework.observe(self.on.leader_elected, self._configure_pod)
+        self.framework.observe(self.on.upgrade_charm, self._configure_pod)
+
+        # For special-cased relations
+        self._stored.set_default(reldata={})
+
+        self._init_postgresql_relation()
+
+    def _init_postgresql_relation(self) -> None:
+        """Initialization related to the postgresql relation"""
+        if 'pg' not in self._stored.reldata:
+            self._stored.reldata['pg'] = {}
+        self.pg = pgsql.PostgreSQLClient(self, 'pg')
+        self.framework.observe(self.pg.on.database_relation_joined, self._on_database_relation_joined)
+        self.framework.observe(self.pg.on.master_changed, self._on_master_changed)
+        self.framework.observe(self.pg.on.standby_changed, self._on_standby_changed)
+
+    def _on_database_relation_joined(self, event: pgsql.DatabaseRelationJoinedEvent) -> None:
+        """Handle db-relation-joined."""
+        if self.model.unit.is_leader():
+            # Provide requirements to the PostgreSQL server.
+            event.database = self.app.name  # Request database named like the Juju app
+        elif event.database != self.app.name:
+            # Leader has not yet set requirements. Defer, in case this unit
+            # becomes leader and needs to perform that operation.
+            event.defer()
+
+    def _on_master_changed(self, event: pgsql.MasterChangedEvent) -> None:
+        """Handle changes in the primary database unit."""
+        if event.database != self.app.name:
+            # Leader has not yet set requirements. Wait until next
+            # event, or risk connecting to an incorrect database.
+            return
+
+        self._stored.reldata['pg']['conn_str'] = None if event.master is None else event.master.conn_str
+        self._stored.reldata['pg']['db_uri'] = None if event.master is None else event.master.uri
+
+        if event.master is None:
+            return
+
+        self._configure_pod(event)
+
+    def _on_standby_changed(self, event: pgsql.StandbyChangedEvent) -> None:
+        """Handle changes in the secondary database unit(s)."""
+        if event.database != self.app.name:
+            # Leader has not yet set requirements. Wait until next
+            # event, or risk connecting to an incorrect database.
+            return
 
-        self._stored.set_default(things=[])
+        self._stored.reldata['pg']['ro_uris'] = [c.uri for c in event.standbys]
+
+        # TODO: Emit event when we add support for read replicas
 
     def _check_juju_config(self) -> None:
         """Check if all the required Juju config options are set,
@@ -56,35 +112,13 @@ class GunicornK8sCharm(CharmBase):
                 errors.append(required)
         if errors:
             raise GunicornK8sCharmJujuConfigError(
-                "Required Juju config item not set : {0}".format(", ".join(sorted(errors)))
-            )
-
-        # Verify YAML formatting
-        errors = []
-        for item in JUJU_CONFIG_YAML_DICT_ITEMS:
-            supposed_yaml = self.model.config[item]
-
-            parsed = None
-
-            try:
-                parsed = yaml.safe_load(supposed_yaml)
-            except yaml.scanner.ScannerError as e:
-                errors.append(item)
-                logger.error("Juju config item '%s' is not YAML : %s", item, str(e))
-
-            if parsed and not isinstance(parsed, dict):
-                errors.append(item)
-                logger.error("Juju config item '%s' is not a YAML dict", item)
-
-        if errors:
-            raise GunicornK8sCharmJujuConfigError(
-                "YAML parsing failed on the Juju config item(s) : {0} - check \"juju debug-log -l ERROR\"".format(
-                    ", ".join(sorted(errors))
-                )
+                "Required Juju config item(s) not set : {}".format(", ".join(sorted(errors)))
             )
 
     def _make_k8s_ingress(self) -> list:
         """Return an ingress that you can use in k8s_resources
+
+        :returns: A list to be used as k8s ingress
         """
 
         hostname = self.model.config['external_hostname']
@@ -96,28 +130,134 @@ class GunicornK8sCharm(CharmBase):
                     {
                         "host": hostname,
                         "http": {
-                            "paths": [{"path": "/", "backend": {"serviceName": self.app.name, "servicePort": 80},}]
+                            "paths": [
+                                {
+                                    "path": "/",
+                                    "backend": {"serviceName": self.app.name, "servicePort": 80},
+                                }
+                            ]
                         },
                     }
                 ]
             },
-            "annotations": {'nginx.ingress.kubernetes.io/ssl-redirect': 'false',},
+            "annotations": {
+                'nginx.ingress.kubernetes.io/ssl-redirect': 'false',
+            },
         }
 
         return [ingress]
 
+    def _render_template(self, tmpl: str, ctx: dict) -> str:
+        """Render a Jinja2 template
+
+        :returns: A rendered Jinja2 template
+        """
+        j2env = Environment(loader=BaseLoader())
+        j2template = j2env.from_string(tmpl)
+
+        return j2template.render(**ctx)
+
+    def _get_context_from_relations(self) -> dict:
+        """Build a template context from relation data - to be used for Jinja2
+        template rendering
+
+        :returns: A dict with relation data that can be used as context for Jinja2 template rendering
+        """
+        ctx = {}
+
+        # Add variables from "special" relations
+        for rel in self._stored.reldata:
+            if self._stored.reldata[rel]:
+                ctx[str(rel)] = self._stored.reldata[rel]
+
+        # Add variables from raw relation data
+        for rels in self.model.relations.values():
+            if len(rels) > 0:
+                rel = rels[0]
+
+                if len(rels) > 1:
+                    logger.warning(
+                        'Multiple relations of type "%s" detected,'
+                        ' using only the first one (id: %s) for relation data.',
+                        rel.name,
+                        rel.id,
+                    )
+
+                if len(rel.units) > 0:
+                    # We want to always pick the same unit, so sort the set
+                    # before picking the first one.
+                    u = sorted(rel.units, key=lambda x: x.name)[0]
+
+                    if len(rel.units) > 1:
+                        logger.warning(
+                            'Multiple units detected in the relation "%s:%s", '
+                            'using only the first one (id: %s) for relation data.',
+                            rel.name,
+                            rel.id,
+                            u.name,
+                        )
+                    if rel.name not in ctx:  # can be present from the "special" relations above
+                        ctx[rel.name] = {}
+                    for k, v in rel.data[u].items():
+                        ctx[rel.name][k] = v
+
+        return ctx
+
+    def _validate_yaml(self, supposed_yaml: str, expected_type: type) -> None:
+        """Validate that the supplied YAML is parsed into the supplied type.
+
+        :raises GunicornK8sCharmYAMLError: if the YAML is incorrect, or if it's not parsed into the expected type
+        """
+        err = False
+        parsed = None
+
+        try:
+            parsed = yaml.safe_load(supposed_yaml)
+        except yaml.scanner.ScannerError as e:
+            logger.error("Error when parsing the following YAML : %s : %s", supposed_yaml, e)
+            err = True
+        else:
+            if not isinstance(parsed, expected_type):
+                err = True
+                logger.error(
+                    "Expected type '%s' but got '%s' when parsing YAML : %s",
+                    expected_type,
+                    parsed.__class__,
+                    supposed_yaml,
+                )
+
+        if err:
+            raise GunicornK8sCharmYAMLError('YAML parsing failed, please check "juju debug-log -l ERROR"')
+
     def _make_pod_env(self) -> dict:
         """Return an envConfig with some core configuration.
 
         :returns: A dictionary used for envConfig in podspec
-        :rtype: dict
         """
-        env = yaml.safe_load(self.model.config['environment'])
+        env = self.model.config['environment']
+
+        if not env:
+            return {}
+
+        ctx = self._get_context_from_relations()
+        rendered_env = self._render_template(env, ctx)
+
+        try:
+            self._validate_yaml(rendered_env, dict)
+        except GunicornK8sCharmYAMLError:
+            raise GunicornK8sCharmJujuConfigError(
+                "Could not parse Juju config 'environment' as a YAML dict - check \"juju debug-log -l ERROR\""
+            )
 
-        return env or {}
+        env = yaml.safe_load(rendered_env)
+
+        return env
 
     def _make_pod_spec(self) -> dict:
-        """Return a pod spec with some core configuration."""
+        """Return a pod spec with some core configuration.
+
+        :returns: A pod spec
+        """
 
         config = self.model.config
         image_details = {
@@ -139,17 +279,41 @@ class GunicornK8sCharm(CharmBase):
                     'imagePullPolicy': 'Always',
                     'ports': [{'containerPort': 80, 'protocol': 'TCP'}],
                     'envConfig': pod_env,
-                    'kubernetes': {'readinessProbe': {'httpGet': {'path': '/', 'port': 80}},},
+                    'kubernetes': {
+                        'readinessProbe': {'httpGet': {'path': '/', 'port': 80}},
+                    },
                 }
             ],
         }
 
-    def configure_pod(self, event: ops.framework.EventBase) -> None:
+    def _configure_pod(self, event: ops.framework.EventBase) -> None:
         """Assemble the pod spec and apply it, if possible.
 
-        :param ops.framework.EventBase event: Event that triggered the method.
+        :param event: Event that triggered the method.
         """
 
+        env = self.model.config['environment']
+        ctx = self._get_context_from_relations()
+
+        if env:
+            j2env = Environment(loader=BaseLoader)
+            j2template = j2env.parse(env)
+            missing_vars = set()
+
+            for req_var in meta.find_undeclared_variables(j2template):
+                if not ctx.get(req_var):
+                    missing_vars.add(req_var)
+
+            if missing_vars:
+                logger.info(
+                    "Missing YAML vars to interpolate the 'environment' config option, "
+                    "setting status to 'waiting' : %s",
+                    ", ".join(sorted(missing_vars)),
+                )
+                self.unit.status = BlockedStatus('Waiting for {} relation(s)'.format(", ".join(sorted(missing_vars))))
+                event.defer()
+                return
+
         if not self.unit.is_leader():
             self.unit.status = ActiveStatus()
             return
@@ -161,13 +325,19 @@ class GunicornK8sCharm(CharmBase):
             return
 
         self.unit.status = MaintenanceStatus('Assembling pod spec')
-        pod_spec = self._make_pod_spec()
+
+        try:
+            pod_spec = self._make_pod_spec()
+        except GunicornK8sCharmJujuConfigError as e:
+            self.unit.status = BlockedStatus(str(e))
+            return
 
         resources = pod_spec.get('kubernetesResources', {})
         resources['ingressResources'] = self._make_k8s_ingress()
 
         self.unit.status = MaintenanceStatus('Setting pod spec')
         self.model.pod.set_spec(pod_spec, k8s_resources={'kubernetesResources': resources})
+        logger.info("Setting active status")
         self.unit.status = ActiveStatus()
 
 
diff --git a/tests/unit/requirements.txt b/tests/unit/requirements.txt
index 7466bfe..8612ca6 100644
--- a/tests/unit/requirements.txt
+++ b/tests/unit/requirements.txt
@@ -2,3 +2,4 @@ PyYAML
 pytest
 pytest-cov
 pytest-subtests
+jinja2
diff --git a/tests/unit/scenario.py b/tests/unit/scenario.py
index 0886672..f71494f 100644
--- a/tests/unit/scenario.py
+++ b/tests/unit/scenario.py
@@ -29,45 +29,31 @@ def get_juju_default_config() -> dict:
 
 JUJU_DEFAULT_CONFIG = get_juju_default_config()
 
+TEST_PG_URI = 'postgresql://usr:pwd@1.2.3.4:5432/gunicorn'
+TEST_PG_CONNSTR = 'dbname=gunicorn host=1.2.3.4 password=pwd port=5432 user=usr'
+
 TEST_JUJU_CONFIG = {
     'defaults': {
         'config': {},
-        'logger': ["ERROR:charm:Required Juju config item not set : image_path", 'ERROR:charm:Required Juju config item not set : external_hostname'],
-        'expected': 'Required Juju config item not set : external_hostname, image_path',
+        'logger': [
+            "ERROR:charm:Required Juju config item not set : image_path",
+            'ERROR:charm:Required Juju config item not set : external_hostname',
+        ],
+        'expected': 'Required Juju config item(s) not set : external_hostname, image_path',
     },
     'missing_image_path': {
-        'config': {'external_hostname': 'example.com',},
-        'logger': ["ERROR:charm:Required Juju config item not set : image_path"],
-        'expected': 'Required Juju config item not set : image_path',
-    },
-    'missing_external_hostname': {
-        'config': {'image_path': 'my_gunicorn_app:devel',},
-        'logger': ["ERROR:charm:Required Juju config item not set : external_hostname"],
-        'expected': 'Required Juju config item not set : external_hostname',
-    },
-    'env_not_yaml': {
         'config': {
-            'image_path': 'my_gunicorn_app:devel',
-            'environment': 'badyaml: :',
             'external_hostname': 'example.com',
         },
-        'logger': [
-            "ERROR:charm:Juju config item 'environment' is not YAML : mapping values are "
-            'not allowed here\n'
-            '  in "<unicode string>", line 1, column 10:\n'
-            '    badyaml: :\n'
-            '             ^'
-        ],
-        'expected': 'YAML parsing failed on the Juju config item(s) : environment - check "juju debug-log -l ERROR"',
+        'logger': ["ERROR:charm:Required Juju config item not set : image_path"],
+        'expected': 'Required Juju config item(s) not set : image_path',
     },
-    'env_yaml_not_dict': {
+    'missing_external_hostname': {
         'config': {
             'image_path': 'my_gunicorn_app:devel',
-            'environment': 'not_a_dict',
-            'external_hostname': 'example.com',
         },
-        'logger': ["ERROR:charm:Juju config item 'environment' is not a YAML dict"],
-        'expected': 'YAML parsing failed on the Juju config item(s) : environment - check "juju debug-log -l ERROR"',
+        'logger': ["ERROR:charm:Required Juju config item not set : external_hostname"],
+        'expected': 'Required Juju config item(s) not set : external_hostname',
     },
     'good_config_no_env': {
         'config': {'image_path': 'my_gunicorn_app:devel', 'external_hostname': 'example.com'},
@@ -87,11 +73,18 @@ TEST_JUJU_CONFIG = {
 
 TEST_CONFIGURE_POD = {
     'bad_config': {
-        'config': {'external_hostname': 'example.com',},
-        'expected': 'Required Juju config item not set : image_path',
+        'config': {
+            'external_hostname': 'example.com',
+        },
+        '_leader_get': "5:\n  database: gunicorn\n  extensions: ''\n  roles: ''",
+        'expected': 'Required Juju config item(s) not set : image_path',
     },
     'good_config_no_env': {
-        'config': {'image_path': 'my_gunicorn_app:devel', 'external_hostname': 'example.com',},
+        'config': {
+            'image_path': 'my_gunicorn_app:devel',
+            'external_hostname': 'example.com',
+        },
+        '_leader_get': "5:\n  database: gunicorn\n  extensions: ''\n  roles: ''",
         'expected': False,
     },
     'good_config_with_env': {
@@ -100,19 +93,25 @@ TEST_CONFIGURE_POD = {
             'external_hostname': 'example.com',
             'environment': 'MYENV: foo',
         },
+        '_leader_get': "5:\n  database: gunicorn\n  extensions: ''\n  roles: ''",
         'expected': False,
     },
 }
 
 TEST_MAKE_POD_SPEC = {
     'basic_no_env': {
-        'config': {'image_path': 'my_gunicorn_app:devel', 'external_hostname': 'example.com',},
+        'config': {
+            'image_path': 'my_gunicorn_app:devel',
+            'external_hostname': 'example.com',
+        },
         'pod_spec': {
             'version': 3,  # otherwise resources are ignored
             'containers': [
                 {
                     'name': 'gunicorn',
-                    'imageDetails': {'imagePath': 'my_gunicorn_app:devel',},
+                    'imageDetails': {
+                        'imagePath': 'my_gunicorn_app:devel',
+                    },
                     'imagePullPolicy': 'Always',
                     'ports': [{'containerPort': 80, 'protocol': 'TCP'}],
                     'envConfig': {},
@@ -132,7 +131,9 @@ TEST_MAKE_POD_SPEC = {
             'containers': [
                 {
                     'name': 'gunicorn',
-                    'imageDetails': {'imagePath': 'my_gunicorn_app:devel',},
+                    'imageDetails': {
+                        'imagePath': 'my_gunicorn_app:devel',
+                    },
                     'imagePullPolicy': 'Always',
                     'ports': [{'containerPort': 80, 'protocol': 'TCP'}],
                     'envConfig': {'MYENV': 'foo'},
@@ -153,7 +154,11 @@ TEST_MAKE_POD_SPEC = {
             'containers': [
                 {
                     'name': 'gunicorn',
-                    'imageDetails': {'imagePath': 'my_gunicorn_app:devel', 'username': 'foo', 'password': 'bar',},
+                    'imageDetails': {
+                        'imagePath': 'my_gunicorn_app:devel',
+                        'username': 'foo',
+                        'password': 'bar',
+                    },
                     'imagePullPolicy': 'Always',
                     'ports': [{'containerPort': 80, 'protocol': 'TCP'}],
                     'envConfig': {},
@@ -167,7 +172,10 @@ TEST_MAKE_POD_SPEC = {
 
 TEST_MAKE_K8S_INGRESS = {
     'basic': {
-        'config': {'image_path': 'my_gunicorn_app:devel', 'external_hostname': 'example.com',},
+        'config': {
+            'image_path': 'my_gunicorn_app:devel',
+            'external_hostname': 'example.com',
+        },
         'expected': [
             {
                 'name': 'gunicorn-ingress',
@@ -176,13 +184,28 @@ TEST_MAKE_K8S_INGRESS = {
                         {
                             'host': 'example.com',
                             'http': {
-                                'paths': [{'path': '/', 'backend': {'serviceName': 'gunicorn', 'servicePort': 80},},],
+                                'paths': [
+                                    {
+                                        'path': '/',
+                                        'backend': {'serviceName': 'gunicorn', 'servicePort': 80},
+                                    },
+                                ],
                             },
                         },
                     ],
                 },
-                'annotations': {'nginx.ingress.kubernetes.io/ssl-redirect': 'false',},
+                'annotations': {
+                    'nginx.ingress.kubernetes.io/ssl-redirect': 'false',
+                },
             },
         ],
     },
 }
+
+TEST_RENDER_TEMPLATE = {
+    'working': {
+        'tmpl': "test {{db.x}}",
+        'ctx': {'db': {'x': 'foo'}},
+        'expected': "test foo",
+    }
+}
diff --git a/tests/unit/test_charm.py b/tests/unit/test_charm.py
index 827fd57..56ea550 100755
--- a/tests/unit/test_charm.py
+++ b/tests/unit/test_charm.py
@@ -4,12 +4,9 @@
 
 import unittest
 
-from unittest.mock import MagicMock
+from unittest.mock import MagicMock, patch
 
-from charm import (
-    GunicornK8sCharm,
-    GunicornK8sCharmJujuConfigError,
-)
+from charm import GunicornK8sCharm, GunicornK8sCharmJujuConfigError, GunicornK8sCharmYAMLError
 
 from ops import testing
 from ops.model import (
@@ -23,6 +20,9 @@ from scenario import (
     TEST_CONFIGURE_POD,
     TEST_MAKE_POD_SPEC,
     TEST_MAKE_K8S_INGRESS,
+    TEST_RENDER_TEMPLATE,
+    TEST_PG_URI,
+    TEST_PG_CONNSTR,
 )
 
 
@@ -39,6 +39,110 @@ class TestGunicornK8sCharm(unittest.TestCase):
         """Cleanup the harness."""
         self.harness.cleanup()
 
+    def test_init_postgresql_relation(self):
+        """Test the _init_postgresql_relation function."""
+
+        # We'll only test the case where _stored already
+        # has content. _stored being empty is basically tested
+        # by all the other functions
+
+        mock_stored = MagicMock()
+        mock_stored.reldata = {'pg': 'foo'}
+        mock_framework = MagicMock()
+        mock_pgsql = MagicMock()
+
+        with patch('test_charm.GunicornK8sCharm._stored', mock_stored):
+            with patch('pgsql.PostgreSQLClient', mock_pgsql):
+                c = GunicornK8sCharm(mock_framework)
+                self.assertEqual(c._stored, mock_stored)
+
+    def test_on_database_relation_joined(self):
+        """Test the _on_database_relation_joined function."""
+
+        # Unit is leader
+        mock_event = MagicMock()
+        self.harness.disable_hooks()  # we don't want leader-set to fire
+        self.harness.set_leader(True)
+
+        self.harness.charm._on_database_relation_joined(mock_event)
+
+        self.assertEqual(mock_event.database, self.harness.charm.app.name)
+
+        # Unit is not leader, DB not ready
+        mock_event = MagicMock()
+        self.harness.disable_hooks()  # we don't want leader-set to fire
+        self.harness.set_leader(False)
+
+        self.harness.charm._on_database_relation_joined(mock_event)
+
+        mock_event.defer.assert_called_once()
+
+        # Unit is leader, DB ready
+        mock_event = MagicMock()
+        self.harness.disable_hooks()  # we don't want leader-set to fire
+        self.harness.set_leader(False)
+        mock_event.database = self.harness.charm.app.name
+
+        r = self.harness.charm._on_database_relation_joined(mock_event)
+        self.assertEqual(r, None)
+
+    def test_on_master_changed(self):
+        """Test the _on_master_changed function."""
+
+        # No database
+        mock_event = MagicMock()
+        mock_event.database = None
+
+        r = self.harness.charm._on_master_changed(mock_event)
+        self.assertEqual(r, None)
+
+        # Database but no master
+        mock_event = MagicMock()
+        mock_event.database = self.harness.charm.app.name
+        mock_event.master = None
+
+        r = self.harness.charm._on_master_changed(mock_event)
+        reldata = self.harness.charm._stored.reldata
+        self.assertEqual(reldata['pg']['conn_str'], None)
+        self.assertEqual(reldata['pg']['db_uri'], None)
+        self.assertEqual(r, None)
+
+        # Database with master
+        mock_event = MagicMock()
+        mock_event.database = self.harness.charm.app.name
+        mock_event.master.conn_str = TEST_PG_CONNSTR
+        mock_event.master.uri = TEST_PG_URI
+        with patch('charm.GunicornK8sCharm._configure_pod') as configure_pod:
+            r = self.harness.charm._on_master_changed(mock_event)
+
+            reldata = self.harness.charm._stored.reldata
+            self.assertEqual(reldata['pg']['conn_str'], mock_event.master.conn_str)
+            self.assertEqual(reldata['pg']['db_uri'], mock_event.master.uri)
+            self.assertEqual(r, None)
+            configure_pod.assert_called_with(mock_event)
+
+    def test_on_standby_changed(self):
+        """Test the _on_standby_changed function."""
+
+        # Database not ready
+        mock_event = MagicMock()
+        mock_event.database = None
+
+        r = self.harness.charm._on_standby_changed(mock_event)
+        self.assertEqual(r, None)
+
+        # Database ready
+        mock_event = MagicMock()
+        mock_event.database = self.harness.charm.app.name
+
+        mock_event.standbys = [MagicMock()]
+        mock_event.standbys[0].uri = TEST_PG_URI
+
+        r = self.harness.charm._on_standby_changed(mock_event)
+
+        reldata = self.harness.charm._stored.reldata
+        self.assertEqual(reldata['pg']['ro_uris'], [TEST_PG_URI])
+
     def test_check_juju_config(self):
         """Check the required juju settings."""
         self.harness.update_config(JUJU_DEFAULT_CONFIG)
@@ -60,22 +164,187 @@ class TestGunicornK8sCharm(unittest.TestCase):
                 # The second argument is the list of key to reset
                 self.harness.update_config(JUJU_DEFAULT_CONFIG)
 
-    def test_configure_pod(self):
+    def test_make_k8s_ingress(self):
+        """Check the crafting of the ingress part of the pod spec."""
+        self.harness.update_config(JUJU_DEFAULT_CONFIG)
+
+        for scenario, values in TEST_MAKE_K8S_INGRESS.items():
+            with self.subTest(scenario=scenario):
+                self.harness.update_config(values['config'])
+                self.assertEqual(self.harness.charm._make_k8s_ingress(), values['expected'])
+                self.harness.update_config(JUJU_DEFAULT_CONFIG)  # You need to clean the config after each run
+
+    def test_render_template(self):
+        """Test template rendering."""
+
+        for scenario, values in TEST_RENDER_TEMPLATE.items():
+            with self.subTest(scenario=scenario):
+                r = self.harness.charm._render_template(values['tmpl'], values['ctx'])
+                self.assertEqual(r, values['expected'])
+
+    def test_get_context_from_relations(self):
+        """Test the _get_context_from_relations function."""
+
+        self.harness.disable_hooks()  # no need for hooks to fire for this test
+
+        # Set up PG "special case" relation data
+        reldata = self.harness.charm._stored.reldata
+        reldata['pg'] = {'conn_str': TEST_PG_CONNSTR, 'db_uri': TEST_PG_URI}
+
+        # Set up PG "raw" relation data
+        relation_id = self.harness.add_relation('pg', 'postgresql')
+        self.harness.add_relation_unit(relation_id, 'postgresql/0')
+        self.harness.update_relation_data(relation_id, 'postgresql/0', {'version': '10'})
+
+        # Set up random relation, with 2 units
+        relation_id = self.harness.add_relation('myrel', 'myapp')
+        self.harness.add_relation_unit(relation_id, 'myapp/0')
+        self.harness.add_relation_unit(relation_id, 'myapp/1')
+        self.harness.update_relation_data(relation_id, 'myapp/0', {'thing': 'bli'})
+        self.harness.update_relation_data(relation_id, 'myapp/1', {'thing': 'blo'})
+
+        # Set up same relation but with a different app
+        relation_id = self.harness.add_relation('myrel', 'myapp2')
+        self.harness.add_relation_unit(relation_id, 'myapp2/0')
+        self.harness.update_relation_data(relation_id, 'myapp2/0', {'thing': 'blu'})
+
+        # Set up random relation, no unit (can happen during relation init)
+        relation_id = self.harness.add_relation('myrel2', 'myapp2')
+
+        expected_ret = {
+            'pg': {'conn_str': TEST_PG_CONNSTR, 'db_uri': TEST_PG_URI, 'version': '10'},
+            'myrel': {'thing': 'bli'},
+        }
+        expected_logger = [
+            'WARNING:charm:Multiple relations of type "myrel" detected, '
+            'using only the first one (id: 1) for relation data.',
+            'WARNING:charm:Multiple units detected in the relation "myrel:1", '
+            'using only the first one (id: myapp/0) for relation data.',
+        ]
+
+        with self.assertLogs(level="WARNING") as logger:
+            r = self.harness.charm._get_context_from_relations()
+
+        self.assertEqual(sorted(logger.output), sorted(expected_logger))
+        self.assertEqual(r, expected_ret)
+
+    def test_validate_yaml(self):
+        """Test the _validate_yaml function."""
+
+        # Proper YAML and type
+        test_str = "a: b\n1: 2"
+        expected_type = dict
+
+        r = self.harness.charm._validate_yaml(test_str, expected_type)
+
+        self.assertEqual(r, None)
+
+        # Incorrect YAML
+        test_str = "a: :"
+        expected_type = dict
+        expected_output = [
+            'ERROR:charm:Error when parsing the following YAML : a: : : mapping values '
+            'are not allowed here\n'
+            '  in "<unicode string>", line 1, column 4:\n'
+            '    a: :\n'
+            '       ^'
+        ]
+        expected_exception = 'YAML parsing failed, please check "juju debug-log -l ERROR"'
+
+        with self.assertLogs(level='ERROR') as logger:
+            with self.assertRaises(GunicornK8sCharmYAMLError) as exc:
+                self.harness.charm._validate_yaml(test_str, expected_type)
+
+        self.assertEqual(sorted(logger.output), expected_output)
+        self.assertEqual(str(exc.exception), expected_exception)
+
+        # Proper YAML, incorrect type
+        test_str = "a: b"
+        expected_type = str
+        expected_output = [
+            "ERROR:charm:Expected type '<class 'str'>' but got '<class 'dict'>' when " 'parsing YAML : a: b'
+        ]
+
+        expected_exception = 'YAML parsing failed, please check "juju debug-log -l ERROR"'
+
+        with self.assertLogs(level='ERROR') as logger:
+            with self.assertRaises(GunicornK8sCharmYAMLError) as exc:
+                self.harness.charm._validate_yaml(test_str, expected_type)
+
+        self.assertEqual(sorted(logger.output), expected_output)
+        self.assertEqual(str(exc.exception), expected_exception)
+
+    def test_make_pod_env(self):
+        """Test the _make_pod_env function."""
+
+        # No env
+        self.harness.update_config(JUJU_DEFAULT_CONFIG)
+        self.harness.update_config({'environment': ''})
+        expected_ret = {}
+
+        r = self.harness.charm._make_pod_env()
+        self.assertEqual(r, expected_ret)
+
+        # Proper env, no templating/relation
+        self.harness.update_config(JUJU_DEFAULT_CONFIG)
+        self.harness.update_config({'environment': 'a: b'})
+        expected_ret = {'a': 'b'}
+
+        r = self.harness.charm._make_pod_env()
+        self.assertEqual(r, expected_ret)
+
+        # Proper env with templating/relations
+        self.harness.update_config(JUJU_DEFAULT_CONFIG)
+        self.harness.update_config({'environment': "DB: {{pg.db_uri}}\nTHING: {{myrel.thing}}}"})
+        expected_ret = {'a': 'b'}
+
+        # Set up PG relation
+        reldata = self.harness.charm._stored.reldata
+        reldata['pg'] = {'conn_str': TEST_PG_CONNSTR, 'db_uri': TEST_PG_URI}
+
+        # Set up random relation
+        self.harness.disable_hooks()  # no need for hooks to fire for this test
+        relation_id = self.harness.add_relation('myrel', 'myapp')
+        self.harness.add_relation_unit(relation_id, 'myapp/0')
+        self.harness.update_relation_data(relation_id, 'myapp/0', {'thing': 'bli'})
+
+        expected_ret = {'DB': TEST_PG_URI, 'THING': 'bli}'}
+
+        r = self.harness.charm._make_pod_env()
+        self.assertEqual(r, expected_ret)
+
+        # Improper env
+        self.harness.update_config(JUJU_DEFAULT_CONFIG)
+        self.harness.update_config({'environment': 'a: :'})
+        expected_ret = None
+        expected_exception = (
+            'Could not parse Juju config \'environment\' as a YAML dict - check "juju debug-log -l ERROR"'
+        )
+
+        with self.assertRaises(GunicornK8sCharmJujuConfigError) as exc:
+            self.harness.charm._make_pod_env()
+
+        self.assertEqual(str(exc.exception), expected_exception)
+
+    @patch('pgsql.pgsql._leader_get')
+    def test_configure_pod(self, mock_leader_get):
         """Test the pod configuration."""
+
         mock_event = MagicMock()
         self.harness.update_config(JUJU_DEFAULT_CONFIG)
 
         self.harness.set_leader(False)
         self.harness.charm.unit.status = BlockedStatus("Testing")
-        self.harness.charm.configure_pod(mock_event)
+        self.harness.charm._configure_pod(mock_event)
         self.assertEqual(self.harness.charm.unit.status, ActiveStatus())
         self.harness.update_config(JUJU_DEFAULT_CONFIG)  # You need to clean the config after each run
 
         for scenario, values in TEST_CONFIGURE_POD.items():
             with self.subTest(scenario=scenario):
+                mock_leader_get.return_value = values['_leader_get']
                 self.harness.update_config(values['config'])
                 self.harness.set_leader(True)
-                self.harness.charm.configure_pod(mock_event)
+                self.harness.charm._configure_pod(mock_event)
                 if values['expected']:
                     self.assertEqual(self.harness.charm.unit.status, BlockedStatus(values['expected']))
                 else:
@@ -83,6 +352,63 @@ class TestGunicornK8sCharm(unittest.TestCase):
 
                 self.harness.update_config(JUJU_DEFAULT_CONFIG)  # You need to clean the config after each run
 
+        # Test missing vars
+        self.harness.update_config(JUJU_DEFAULT_CONFIG)
+        self.harness.update_config(
+            {
+                'image_path': 'my_gunicorn_app:devel',
+                'external_hostname': 'example.com',
+                'environment': 'DB_URI: {{pg.uri}}',
+            }
+        )
+        self.harness.set_leader(True)
+        expected_status = 'Waiting for pg relation(s)'
+
+        self.harness.charm._configure_pod(mock_event)
+
+        mock_event.defer.assert_called_once()
+        self.assertEqual(self.harness.charm.unit.status, BlockedStatus(expected_status))
+
+        # Test no missing vars
+        self.harness.update_config(JUJU_DEFAULT_CONFIG)
+        self.harness.update_config(
+            {
+                'image_path': 'my_gunicorn_app:devel',
+                'external_hostname': 'example.com',
+                'environment': 'DB_URI: {{pg.uri}}',
+            }
+        )
+
+        reldata = self.harness.charm._stored.reldata
+        reldata['pg'] = {'conn_str': TEST_PG_CONNSTR, 'db_uri': TEST_PG_URI}
+        self.harness.set_leader(True)
+        # Set up random relation
+        self.harness.disable_hooks()  # no need for hooks to fire for this test
+        relation_id = self.harness.add_relation('myrel', 'myapp')
+        self.harness.add_relation_unit(relation_id, 'myapp/0')
+        self.harness.update_relation_data(relation_id, 'myapp/0', {'thing': 'bli'})
+        expected_status = 'Waiting for pg relation(s)'
+
+        self.harness.charm._configure_pod(mock_event)
+
+        self.assertEqual(self.harness.charm.unit.status, ActiveStatus())
+
+        # Test incorrect YAML
+        self.harness.update_config(JUJU_DEFAULT_CONFIG)
+        self.harness.update_config(
+            {
+                'image_path': 'my_gunicorn_app:devel',
+                'external_hostname': 'example.com',
+                'environment': 'a: :',
+            }
+        )
+        self.harness.set_leader(True)
+        expected_status = 'Could not parse Juju config \'environment\' as a YAML dict - check "juju debug-log -l ERROR"'
+
+        self.harness.charm._configure_pod(mock_event)
+
+        self.assertEqual(self.harness.charm.unit.status, BlockedStatus(expected_status))
+
     def test_make_pod_spec(self):
         """Check the crafting of the pod spec."""
         self.harness.update_config(JUJU_DEFAULT_CONFIG)
@@ -93,16 +419,6 @@ class TestGunicornK8sCharm(unittest.TestCase):
                 self.assertEqual(self.harness.charm._make_pod_spec(), values['pod_spec'])
                 self.harness.update_config(JUJU_DEFAULT_CONFIG)  # You need to clean the config after each run
 
-    def test_make_k8s_ingress(self):
-        """Check the crafting of the ingress part of the pod spec."""
-        self.harness.update_config(JUJU_DEFAULT_CONFIG)
-
-        for scenario, values in TEST_MAKE_K8S_INGRESS.items():
-            with self.subTest(scenario=scenario):
-                self.harness.update_config(values['config'])
-                self.assertEqual(self.harness.charm._make_k8s_ingress(), values['expected'])
-                self.harness.update_config(JUJU_DEFAULT_CONFIG)  # You need to clean the config after each run
-
 
 if __name__ == '__main__':
     unittest.main()
diff --git a/tox.ini b/tox.ini
index 665fad6..8dedaea 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,12 +1,10 @@
 [tox]
 skipsdist=True
-envlist = unit, functional
+envlist = unit
 skip_missing_interpreters = True
 
 [testenv]
 basepython = python3
-setenv =
-  PYTHONPATH = {toxinidir}/src:{toxinidir}/build/lib:{toxinidir}/build/venv
 
 [testenv:unit]
 commands =
@@ -18,22 +16,12 @@ setenv =
   PYTHONPATH={toxinidir}/src:{toxinidir}/build/lib:{toxinidir}/build/venv
   TZ=UTC
 
-[testenv:functional]
-passenv =
-  HOME
-  JUJU_REPOSITORY
-  PATH
-commands =
-        pytest -v --ignore mod --ignore {toxinidir}/tests/unit {posargs}
-deps = -r{toxinidir}/tests/functional/requirements.txt
-       -r{toxinidir}/requirements.txt
-
 [testenv:black]
-commands = black --skip-string-normalization --line-length=120 src/ tests/
+commands = black src/ tests/ docker/app/app/
 deps = black
 
 [testenv:lint]
-commands = flake8 src/ tests/
+commands = flake8 src/ tests/ docker/app/app/
 # Pin flake8 to 3.7.9 to match focal
 deps =
     flake8==3.7.9