Merge ~peter-sabaini/charm-graylog:lp1758175 into charm-graylog:master

[Drew Freiberger]

[Paul Goins (vultaire)]

This review is large, but I actually found the overall review to be good. I have no real critical feedback.

Re: the zaza bundles, something we've been adopting in other charms to reduce duplication is to have a single base bundle with all the common apps/relations ("base.yaml", not listed in tests.yaml), replace the existing bundles with symlinks to that base bundle, and then have all the per-bundle customizations moved to overlays. I don't think that's critical for this MR, but I'm mentioning it as it seems a good practice to reduce duplication and the chance of accidental divergence between bundles.

Re: code changes: the only thing I would suggest is that "make black" is run; src/tests/functional/tests/test_graylog_upgrade.py is not passing "make lint" but will simply by running "make black" and committing the changes.

I'll get this running a full test suite on my charmlab box... I'm +1 once the "make black" changes are in, assuming no failures arise during tests.

review: Needs Fixing

[Paul Goins (vultaire)]

Unit tests are failing; seems more mocking is needed. "make unittests" partial output showing the failure: https://pastebin.ubuntu.com/p/M94v64H8GS/
(It's trying to do an apt-get install ca-certificates-java on my test box, which fails.)

Running functional tests now.

review: Needs Fixing

[Paul Goins (vultaire)]

Functional tests pass.

If you can run "make black" and correct the unit test failure, I think this is +1 from me.

[Peter Sabaini (peter-sabaini)]

I've mocked out the java CA install and blackened, please take a look?

[Paul Goins (vultaire)]

LGTM.

review: Approve

[Drew Freiberger (afreiberger)]

Expected status blocked on bionic-graylog3-tls.

Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0 10.0.8.81 Certificate Authority connected.
elastic/0* active idle 1 10.0.8.91 9200/tcp Unit is ready
graylog/0* waiting idle 2 10.0.8.8 9000/tcp Waiting for: filebeat, REST API
  nrpe/0* active idle 10.0.8.8 icmp,5666/tcp ready
mongo/0* active idle 3 10.0.8.92 27017/tcp,27019/tcp,27021/tcp,28017/tcp Unit is ready
nagios/0* active idle 4 10.0.8.126 80/tcp ready
ubuntu/0* active idle 5 10.0.8.135 ready
  filebeat/0* active idle 10.0.8.135 Filebeat ready.

review: Needs Fixing

[Drew Freiberger (afreiberger)]

Graylog service not starting snap version is 3/stable with tls enabled:

2021-01-22T21:13:21Z systemd[1]: Started Service for snap application graylog.graylog.
2021-01-22T21:13:36Z systemd[1]: snap.graylog.graylog.service: Main process exited, code=exited, status=255/n/a
2021-01-22T21:13:36Z systemd[1]: snap.graylog.graylog.service: Failed with result 'exit-code'.

graylog traceback has this as the cause:

        Caused by: org.glassfish.grizzly.ssl.SSLContextConfigurator$GenericStoreException: java.io.FileNotFoundException: /var/snap/graylog/common/cacerts (No such file or directory)

Snap is running graylog with the following cli args: (most specifically: -Djavax.net.ssl.trustStore=/var/snap/graylog/common/cacerts)

root 7680 1 0 21:24 ? 00:00:03 /snap/graylog/17/usr/lib/jvm/java-8-openjdk-amd64/bin/java -Djavax.net.ssl.trustStore=/var/snap/graylog/common/cacerts -jar -Dlog4j.configurationFile=file:///snap/graylog/17/etc/graylog/server/log4j2.xml -Djava.library.path=/snap/graylog/17/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=snap /snap/graylog/17/usr/share/graylog-server/graylog.jar server -f /var/snap/graylog/common/server.conf -np

[Drew Freiberger (afreiberger)]

After adding the sync-graylog-snap file into /etc/ca-certificates/update.d, we must run update-ca-certificates to trigger this script.

review: Needs Fixing

[Drew Freiberger (afreiberger)]

Getting random snap core install errors (this was on bionic-graylog3-tls model.

https://pastebin.canonical.com/p/V4Y7wTytbV/

I feel like there weren't random functest errors in 20.10, I'm running functest against that release to see results for stable/20.10 with current upstream bits and bobs to see if it's in the upstream or in our charm/functests that need repairing.

[Drew Freiberger (afreiberger)]

We have uncovered a number of issues with our testing infrastructure (juju not updating daily images, zaza not upgrading packages, systemd bug affecting ES patched a couple weeks ago) that are causing random failures of the functional tests, but 30% pass w/out issue. I suggest merging this and opening a bug report for racy tests.

review: Approve

[Xav Paice (xavpaice)]

LGTM

review: Approve

diff --git a/src/layer.yaml b/src/layer.yaml
index 04e6d2a..5173d7e 100644
--- a/src/layer.yaml
+++ b/src/layer.yaml
@@ -3,6 +3,7 @@ includes:
   - 'layer:basic'
   - 'layer:snap'
   - 'layer:leadership'
+  - 'layer:tls-client'
   - 'interface:elasticsearch'
   - 'interface:elastic-beats'
   - 'interface:http'
diff --git a/src/lib/charms/layer/graylog/api.py b/src/lib/charms/layer/graylog/api.py
index 98e52cf..3d9bf93 100644
--- a/src/lib/charms/layer/graylog/api.py
+++ b/src/lib/charms/layer/graylog/api.py
@@ -4,6 +4,12 @@ import os
 
 import requests
 
+# When using 'certifi' from the virtualenv, the system-wide certificates store
+# is not used, so installed certificates won't be used to validate hosts.
+# Adding the system CA bundle
+# https://git.launchpad.net/ubuntu/+source/python-certifi/tree/debian/patches/0001-Use-Debian-provided-etc-ssl-certs-ca-certificates.cr.patch
+SYSTEM_CA_BUNDLE = "/etc/ssl/certs/ca-certificates.crt"
+
 # We are in a charm environment
 charm = False
 if os.environ.get("JUJU_UNIT_NAME"):
@@ -20,7 +26,9 @@ def get_ignore_indexer_failures_file():  # noqa: D103
 class GraylogApi:
     """Manage Graylog via its API."""
 
-    def __init__(self, base_url, username, password, token_name="graylog-api"):
+    def __init__(
+        self, base_url, username, password, token_name="graylog-api", verify=None
+    ):
         """Initialize HTTP values."""
         self.base_url = base_url
         if not base_url.endswith("/"):
@@ -33,6 +41,7 @@ class GraylogApi:
         self.input_types = None
         self.req_timeout = 3
         self.req_retries = 4
+        self.verify = verify
 
     def request(self, path, method="GET", data={}, params=None):  # noqa: C901
         """Retrieve data by URL."""
@@ -59,6 +68,7 @@ class GraylogApi:
                     params=params,
                     headers=headers,
                     timeout=self.req_timeout,
+                    verify=SYSTEM_CA_BUNDLE if self.verify is None else self.verify,
                 )
                 if resp.ok:
                     if method == "DELETE":
diff --git a/src/lib/charms/layer/graylog/constants.py b/src/lib/charms/layer/graylog/constants.py
index db38aa0..ce2bc59 100644
--- a/src/lib/charms/layer/graylog/constants.py
+++ b/src/lib/charms/layer/graylog/constants.py
@@ -1,5 +1,9 @@
 """Options used by the Graylog charm."""
+import os
+
+
 SNAP_NAME = "graylog"
+SNAP_COMMON_DIR = "/var/snap/graylog/common/"
 CONF_FILE = "/var/snap/graylog/common/server.conf"
 SNAP_CONF_FILE = "/snap/graylog/current/etc/graylog/server/server.conf"
 # /snap/graylog is a read-only squashfs so we use the initial settings
@@ -12,3 +16,5 @@ ELASTICSEARCH_DISCOVERY_PORT = "9300"
 SERVICE_NAME = "snap.graylog.graylog"
 DEFAULT_REST_API_TIMEOUT = 120
 NAGIOS_USERNAME = "nagios"
+CERT_PATH = os.path.join(SNAP_COMMON_DIR, "server.crt")
+CERT_KEY_PATH = os.path.join(SNAP_COMMON_DIR, "server.key")
diff --git a/src/lib/charms/layer/graylog/utils.py b/src/lib/charms/layer/graylog/utils.py
index 115e672..6b020c5 100644
--- a/src/lib/charms/layer/graylog/utils.py
+++ b/src/lib/charms/layer/graylog/utils.py
@@ -4,10 +4,10 @@ from urllib.parse import urlparse
 from charmhelpers.core import hookenv
 
 from charms.layer.graylog.constants import SNAP_NAME
+from charms.reactive import get_flags
 
 import netifaces
 
-
 # Allow mocking of other layers during unit tests
 try:
     from charms.layer import snap
@@ -20,7 +20,14 @@ def get_api_port():  # noqa: D103
 
 
 def get_api_url():  # noqa: D103
-    return "http://127.0.0.1:{}/api/".format(get_api_port())
+    return "{}://127.0.0.1:{}/api/".format(get_api_protocol(), get_api_port())
+
+
+def get_api_protocol():  # noqa: D103
+    if "graylog.certificates.configured" in get_flags():
+        return "https"
+    else:
+        return "http"
 
 
 def is_v2():  # noqa: D103
diff --git a/src/reactive/graylog.py b/src/reactive/graylog.py
index 9cc3a93..4f4a9ee 100644
--- a/src/reactive/graylog.py
+++ b/src/reactive/graylog.py
@@ -2,23 +2,29 @@
 import hashlib
 import os
 import re
+import socket
+import subprocess
 import time
 from urllib.parse import urlparse
 
 from charmhelpers.contrib.charmsupport import nrpe
-from charmhelpers.core import hookenv, host, unitdata
+from charmhelpers.core import hookenv, host, templating, unitdata
+from charmhelpers.fetch import filter_installed_packages, install
 
 import charms.leadership
-from charms.layer import snap
+from charms.layer import snap, tls_client
 from charms.layer.graylog import (
     GraylogApi,
     LogExtractPipeline,
     get_api_port,
+    get_api_protocol,
     get_api_url,
     is_v2,
     validate_api_uri,
 )
 from charms.layer.graylog.constants import (
+    CERT_KEY_PATH,
+    CERT_PATH,
     CONF_FILE,
     DEFAULT_REST_API_TIMEOUT,
     ELASTICSEARCH_DISCOVERY_PORT,
@@ -200,7 +206,8 @@ def configure_graylog():  # noqa: C901
         # intead of guessing the best IP.
         set_conf("http_bind_address", "0.0.0.0:9000")
         set_conf(
-            "http_publish_uri", validate_api_uri("http://0.0.0.0:{}/".format(api_port))
+            "http_publish_uri",
+            validate_api_uri("{}://0.0.0.0:{}/".format(get_api_protocol(), api_port)),
         )
 
         if conf["web_endpoint_uri"]:
@@ -1045,6 +1052,49 @@ def configure_nagios(nagios):
     set_state("graylog_nagios.configured")
 
 
+@when("certificates.available")
+@when_not("graylog.certificates.configured")
+def tls_request_certificate(tls):
+    """Create a server certificate for this server."""
+    # ca-certificates-java generates the ca-certificates in a jvm compatible
+    # keystore
+    if filter_installed_packages(["ca-certificates-java"]):
+        install(["ca-certificates-java"], fatal=True)
+
+    _maybe_install_ca_certificates_hook()
+    _maybe_configure_graylog_jvm_keystore()
+
+    # Use the public ip of this unit as the Common Name for the certificate.
+    common_name = hookenv.unit_public_ip()
+    # Get a list of Subject Alt Names for the certificate.
+    sans = []
+    sans.append(hookenv.unit_public_ip())
+    sans.append(hookenv.unit_private_ip())
+    sans.append(socket.gethostname())
+    sans.append("127.0.0.1")
+    sans.append("localhost")
+    tls_client.request_server_cert(
+        common_name, sans, crt_path=CERT_PATH, key_path=CERT_KEY_PATH
+    )
+
+    set_conf("http_enable_tls", "true")
+    set_conf("http_tls_cert_file", CERT_PATH)
+    set_conf("http_tls_key_file", CERT_KEY_PATH)
+
+    set_conf("rest_enable_tls", "true")
+    set_conf("rest_tls_cert_file", CERT_PATH)
+    set_conf("rest_tls_key_file", CERT_KEY_PATH)
+
+    set_conf("web_enable_tls", "true")
+    set_conf("web_tls_cert_file", CERT_PATH)
+    set_conf("web_tls_key_file", CERT_KEY_PATH)
+
+    set_conf("http_publish_uri", "https://0.0.0.0:{}/".format(get_api_port()))
+
+    set_state("graylog.certificates.configured")
+    set_state("graylog.needs_restart")
+
+
 def get_default_graylog_client():  # noqa: D103
     db = unitdata.kv()
     return GraylogApi(
@@ -1068,3 +1118,22 @@ def _verify_rest_api_is_alive(timeout=DEFAULT_REST_API_TIMEOUT):
         if time.time() - start_ts > timeout:
             raise ApiTimeout()
     hookenv.log("REST API is up")
+
+
+def _maybe_install_ca_certificates_hook():
+    templating.render(
+        "sync-graylog-snap",
+        "/etc/ca-certificates/update.d/sync-graylog-snap",
+        context={},
+        perms=0o755,
+    )
+    subprocess.check_call(["update-ca-certificates"])
+
+
+def _maybe_configure_graylog_jvm_keystore():
+    templating.render(
+        "default-graylog-server",
+        "/var/snap/graylog/current/default-graylog-server",
+        context={},
+        perms=0o644,
+    )
diff --git a/src/templates/default-graylog-server b/src/templates/default-graylog-server
new file mode 100644
index 0000000..ace24cb
--- /dev/null
+++ b/src/templates/default-graylog-server
@@ -0,0 +1,4 @@
+#
+# This file is managed by juju.
+#
+GRAYLOG_SERVER_JAVA_OPTS="-Djavax.net.ssl.trustStore=/var/snap/graylog/common/cacerts"
diff --git a/src/templates/sync-graylog-snap b/src/templates/sync-graylog-snap
new file mode 100755
index 0000000..6d08b7d
--- /dev/null
+++ b/src/templates/sync-graylog-snap
@@ -0,0 +1,7 @@
+#!/bin/sh
+#
+# This file is managed by juju.
+#
+
+set -e
+rsync /etc/ssl/certs/java/cacerts /var/snap/graylog/common/cacerts
diff --git a/src/tests/functional/requirements.txt b/src/tests/functional/requirements.txt
index 7345526..ecf87be 100644
--- a/src/tests/functional/requirements.txt
+++ b/src/tests/functional/requirements.txt
@@ -1,2 +1,4 @@
 tenacity
 git+https://github.com/openstack-charmers/zaza.git#egg=zaza
+netifaces
+charmhelpers
diff --git a/src/tests/functional/tests/base.py b/src/tests/functional/tests/base.py
new file mode 100644
index 0000000..46eb75f
--- /dev/null
+++ b/src/tests/functional/tests/base.py
@@ -0,0 +1,48 @@
+"""Base class for Testing."""
+import logging
+import os
+import tempfile
+import unittest
+
+from zaza import model
+
+
+class BaseTestCase(unittest.TestCase):
+    """Base class for graylog functional testing."""
+
+    @classmethod
+    def setUpClass(cls):
+        """Configure the test's environment.
+
+        - Get the CA from easyrsa (when available) and write it on disk.
+        """
+        if cls.get_api_protocol() == "https":
+            rel_id = model.get_relation_id(
+                "graylog", "easyrsa", remote_interface_name="client"
+            )
+            easyrsa_unit = model.get_units("easyrsa")[0]
+            cmd = "relation-get -r client:{} ca {}".format(
+                rel_id, easyrsa_unit.entity_id
+            )
+            logging.info(cmd)
+            result = model.run_on_unit(easyrsa_unit.entity_id, cmd)
+
+            with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
+                f.write(result["Stdout"])
+                f.flush()
+                cls.ca_path = f.name
+        else:
+            cls.ca_path = None
+
+    @classmethod
+    def tearDownClass(cls):
+        """Remove the CA file that was written to disk during the setUp."""
+        if cls.ca_path:
+            os.remove(cls.ca_path)
+
+    @classmethod
+    def get_api_protocol(cls):  # noqa: D102
+        if model.get_relation_id("graylog", "easyrsa"):
+            return "https"
+        else:
+            return "http"
diff --git a/src/tests/functional/tests/bundles/bionic-graylog2.yaml b/src/tests/functional/tests/bundles/bionic-graylog2.yaml
index bd256db..27703fd 100644
--- a/src/tests/functional/tests/bundles/bionic-graylog2.yaml
+++ b/src/tests/functional/tests/bundles/bionic-graylog2.yaml
@@ -30,6 +30,10 @@ applications:
   nrpe:
     charm: cs:nrpe
 
+  nagios:
+    charm: cs:nagios
+    num_units: 1
+
 relations:
   - - ubuntu
     - filebeat
@@ -43,3 +47,5 @@ relations:
     - haproxy
   - - graylog
     - nrpe
+  - - nagios
+    - nrpe
diff --git a/src/tests/functional/tests/bundles/bionic-graylog3-tls.yaml b/src/tests/functional/tests/bundles/bionic-graylog3-tls.yaml
new file mode 100644
index 0000000..954f9c5
--- /dev/null
+++ b/src/tests/functional/tests/bundles/bionic-graylog3-tls.yaml
@@ -0,0 +1,51 @@
+series: bionic
+
+applications:
+  ubuntu:
+    charm: cs:ubuntu
+    num_units: 1
+  
+  filebeat:
+    charm: cs:filebeat
+    num_units: 0
+  
+  graylog:
+    num_units: 1
+    series: bionic
+    options:
+      channel: 3/stable
+  
+  elastic:
+    charm: cs:elasticsearch
+    num_units: 1
+  
+  mongo:
+    charm: cs:mongodb
+    num_units: 1
+
+  nrpe:
+    charm: cs:nrpe
+
+  nagios:
+    charm: cs:nagios
+    num_units: 1
+
+  easyrsa:
+    charm: cs:~containers/easyrsa
+    num_units: 1
+
+relations:
+  - - ubuntu
+    - filebeat
+  - - graylog:beats
+    - filebeat:logstash
+  - - graylog
+    - mongo
+  - - graylog
+    - elastic
+  - - graylog
+    - nrpe
+  - - nagios
+    - nrpe
+  - - easyrsa:client
+    - graylog:certificates
diff --git a/src/tests/functional/tests/bundles/bionic-graylog3.yaml b/src/tests/functional/tests/bundles/bionic-graylog3.yaml
index 596cf7d..33ac7e5 100644
--- a/src/tests/functional/tests/bundles/bionic-graylog3.yaml
+++ b/src/tests/functional/tests/bundles/bionic-graylog3.yaml
@@ -30,6 +30,10 @@ applications:
   nrpe:
     charm: cs:nrpe
 
+  nagios:
+    charm: cs:nagios
+    num_units: 1
+
 relations:
   - - ubuntu
     - filebeat
@@ -43,3 +47,5 @@ relations:
     - haproxy
   - - graylog
     - nrpe
+  - - nagios
+    - nrpe
diff --git a/src/tests/functional/tests/bundles/focal-graylog2.yaml b/src/tests/functional/tests/bundles/focal-graylog2.yaml
index 368432e..9cc6016 100644
--- a/src/tests/functional/tests/bundles/focal-graylog2.yaml
+++ b/src/tests/functional/tests/bundles/focal-graylog2.yaml
@@ -30,6 +30,10 @@ applications:
   nrpe:
     charm: cs:nrpe
 
+  nagios:
+    charm: cs:nagios
+    num_units: 1
+
 relations:
   - - ubuntu
     - filebeat
@@ -43,3 +47,5 @@ relations:
     - haproxy
   - - graylog
     - nrpe
+  - - nagios
+    - nrpe
diff --git a/src/tests/functional/tests/bundles/focal-graylog3-tls.yaml b/src/tests/functional/tests/bundles/focal-graylog3-tls.yaml
new file mode 100644
index 0000000..100ccb4
--- /dev/null
+++ b/src/tests/functional/tests/bundles/focal-graylog3-tls.yaml
@@ -0,0 +1,51 @@
+series: bionic
+
+applications:
+  ubuntu:
+    charm: cs:ubuntu
+    num_units: 1
+
+  filebeat:
+    charm: cs:filebeat
+    num_units: 0
+
+  graylog:
+    num_units: 1
+    series: focal
+    options:
+      channel: 3/stable
+
+  elastic:
+    charm: cs:elasticsearch
+    num_units: 1
+
+  mongo:
+    charm: cs:mongodb
+    num_units: 1
+
+  nrpe:
+    charm: cs:nrpe
+
+  nagios:
+    charm: cs:nagios
+    num_units: 1
+
+  easyrsa:
+    charm: cs:~containers/easyrsa
+    num_units: 1
+
+relations:
+  - - ubuntu
+    - filebeat
+  - - graylog:beats
+    - filebeat:logstash
+  - - graylog
+    - mongo
+  - - graylog
+    - elastic
+  - - graylog
+    - nrpe
+  - - nagios
+    - nrpe
+  - - easyrsa:client
+    - graylog:certificates
diff --git a/src/tests/functional/tests/bundles/focal-graylog3.yaml b/src/tests/functional/tests/bundles/focal-graylog3.yaml
index cb26f52..e2e2a41 100644
--- a/src/tests/functional/tests/bundles/focal-graylog3.yaml
+++ b/src/tests/functional/tests/bundles/focal-graylog3.yaml
@@ -30,6 +30,10 @@ applications:
   nrpe:
     charm: cs:nrpe
 
+  nagios:
+    charm: cs:nagios
+    num_units: 1
+
 relations:
   - - ubuntu
     - filebeat
@@ -43,3 +47,5 @@ relations:
     - haproxy
   - - graylog
     - nrpe
+  - - nagios
+    - nrpe
diff --git a/src/tests/functional/tests/test_graylog_charm.py b/src/tests/functional/tests/test_graylog_charm.py
index df763b6..83d58e7 100644
--- a/src/tests/functional/tests/test_graylog_charm.py
+++ b/src/tests/functional/tests/test_graylog_charm.py
@@ -2,12 +2,13 @@
 
 import logging
 import time
-import unittest
 
 from api import GraylogApi
 
 import tenacity
 
+from tests.base import BaseTestCase
+
 import zaza.model as model
 
 
@@ -17,7 +18,7 @@ DEFAULT_API_PORT = "9001"
 DEFAULT_WEB_PORT = "9000"
 
 
-class BaseGraylogTest(unittest.TestCase):
+class BaseGraylogTest(BaseTestCase):
     """Base for Graylog charm tests."""
 
     @classmethod
@@ -56,10 +57,14 @@ class BaseGraylogTest(unittest.TestCase):
                 "org.graylog.plugins.threatintel.ThreatIntelPlugin",
             }
 
-        api_url = "http://{}:{}/api".format(cls.graylog_ip, cls.api_port)
+        api_url = "{}://{}:{}/api".format(
+            cls.get_api_protocol(), cls.graylog_ip, cls.api_port
+        )
         action = model.run_action_on_leader(cls.application_name, "show-admin-password")
         res = action.data["results"]
-        cls.api = GraylogApi(api_url, "admin", res["admin-password"])
+        cls.api = GraylogApi(
+            api_url, "admin", res["admin-password"], verify=cls.ca_path
+        )
         # try harder to get an api connection to cater for overloaded testsystems
         cls.api.req_timeout = REQ_TIMEOUT
         logging.debug("API at {}".format(api_url))
@@ -75,7 +80,9 @@ class CharmOperationTest(BaseGraylogTest):
         until the CURL_TIMEOUT because it may take a few seconds for the
         graylog systemd service to start.
         """
-        curl_command = "curl http://localhost:{}".format(self.api_port)
+        curl_command = "curl {}://localhost:{}".format(
+            self.get_api_protocol(), self.api_port
+        )
         timeout = time.time() + CURL_TIMEOUT
         while time.time() < timeout:
             response = model.run_on_unit(self.lead_unit_name, curl_command)
diff --git a/src/tests/functional/tests/test_legacy.py b/src/tests/functional/tests/test_legacy.py
index 5ee2eea..fcea119 100644
--- a/src/tests/functional/tests/test_legacy.py
+++ b/src/tests/functional/tests/test_legacy.py
@@ -1,9 +1,14 @@
 """Graylog v2 legacy tests."""
+import logging
 import re
 import unittest
 
 from api import GraylogApi
 
+from charmhelpers.core.decorators import retry_on_exception
+
+from tests.base import BaseTestCase
+
 import yaml
 
 from zaza import model
@@ -14,7 +19,7 @@ DEFAULT_API_PORT = "9001"  # Graylog 2 only
 DEFAULT_WEB_PORT = "9000"
 
 
-class LegacyTests(unittest.TestCase):
+class LegacyTests(BaseTestCase):
     """These tests were ported from tests/test_10_basic.py in changeset a3b54c2.
 
     They were temporarily deleted during the conversion from Amulet to Zaza.
@@ -24,8 +29,12 @@ class LegacyTests(unittest.TestCase):
 
     def test_api_ready(self):
         """Curl the api endpoint on the graylog unit."""
+        protocol = self.get_api_protocol()
         port = self.get_api_port()
-        curl_command = "curl http://localhost:{} --connect-timeout 30".format(port)
+        curl_command = ("curl {}://localhost:{} " "--connect-timeout 30").format(
+            protocol, port
+        )
+        logging.info(curl_command)
         self.run_command("graylog/0", curl_command)
 
     def test_elasticsearch_active(self):
@@ -60,10 +69,16 @@ class LegacyTests(unittest.TestCase):
         """Return the list of API clients to Graylog units."""
         graylog_units = juju.get_full_juju_status().applications["graylog"]["units"]
         graylog_addrs = [unit["public-address"] for unit in graylog_units.values()]
+        protocol = self.get_api_protocol()
         port = self.get_api_port()
         password = self.get_graylog_admin_password()
         return [
-            GraylogApi("http://{}:{}/api".format(graylog_addr, port), "admin", password)
+            GraylogApi(
+                "{}://{}:{}/api".format(protocol, graylog_addr, port),
+                "admin",
+                password,
+                verify=self.ca_path,
+            )
             for graylog_addr in graylog_addrs
         ]
 
@@ -76,8 +91,17 @@ class LegacyTests(unittest.TestCase):
         password = result.data["results"]["admin-password"]
         return password
 
-    def test_website_active(self):
+    def test_website_active_with_haproxy(self):
         """Verify if the Graylog endpoints are correctly configured."""
+        try:
+            model.get_application("haproxy")
+        except KeyError:
+            reason = (
+                "haproxy is not in the model, assuming there is "
+                "no front loadbalancer"
+            )
+            raise unittest.SkipTest(reason)
+
         data = juju.get_relation_from_unit("haproxy/0", "graylog/0", "website")
         self.assertEqual(data["port"], DEFAULT_WEB_PORT)
 
@@ -117,6 +141,7 @@ class LegacyTests(unittest.TestCase):
         )
         self.assertTrue(re.search(r"CRITICAL", cfg))
 
+    @retry_on_exception(num_retries=5, base_delay=2, exc_type=AssertionError)
     def get_file_contents(self, unit, filename):
         """Retrieve content of a file in a remote unit."""
         result = self.run_command(unit, "cat '{}'".format(filename))
diff --git a/src/tests/functional/tests/tests.yaml b/src/tests/functional/tests/tests.yaml
index c52c200..47db931 100644
--- a/src/tests/functional/tests/tests.yaml
+++ b/src/tests/functional/tests/tests.yaml
@@ -1,9 +1,11 @@
 charm_name: graylog
 gate_bundles:
   - gl2: bionic-graylog2
-  - gl3: bionic-graylog3
   - gl2: focal-graylog2
+  - gl3: bionic-graylog3
+  - gl3: bionic-graylog3-tls
   - gl3: focal-graylog3
+  - gl3: focal-graylog3-tls
 #  - gl2: bionic-graylog2-ha
 #  - gl3: bionic-graylog3-ha
 smoke_bundles:
@@ -28,3 +30,15 @@ target_deploy_status:
   nrpe:
     workload-status: blocked
     workload-status-message: Nagios server not configured or related
+  mongo:
+    workload-status: active
+    workload-status-message: 'Unit is ready'
+  graylog:
+    workload-status: active
+    workload-status-message: 'Ready with: filebeat, elasticsearch, mongodb'
+  elastic:
+    workload-status: active
+    workload-status-message: 'Unit is ready'
+  easyrsa:
+    workload-status-message: 'Certificate Authority connected.'
+
diff --git a/src/tests/unit/test_graylog.py b/src/tests/unit/test_graylog.py
index c723ce1..a9b0ae8 100644
--- a/src/tests/unit/test_graylog.py
+++ b/src/tests/unit/test_graylog.py
@@ -1,10 +1,15 @@
 """Tests around the graylog reactive script."""
 import os
+import socket
 import sys
 import tempfile
 import unittest
 from unittest import mock
 
+from charms.layer.graylog.constants import (
+    CERT_KEY_PATH,
+    CERT_PATH,
+)
 from charms.layer.graylog.snap_change import (
     ChannelChangeStatus,
     _is_channel_valid,
@@ -17,6 +22,8 @@ leader_mock = mock.Mock()
 sys.modules["charms.leadership"] = leader_mock
 snap_mock = mock.Mock()
 sys.modules["charms.layer.snap"] = snap_mock
+tls_client_mock = mock.Mock()
+sys.modules["charms.layer.tls_client"] = tls_client_mock
 
 from files import check_graylog_health  # noqa: E402
 
@@ -28,6 +35,7 @@ from reactive.graylog import (  # noqa: E402
     refresh_graylog,
     set_conf,
     set_jvm_heap_size,
+    tls_request_certificate,
     update_config,
     upgrade_charm,
 )
@@ -555,3 +563,81 @@ class TestIsChannelValid(unittest.TestCase):
             "2",  # track w/o risk
         ):
             self.assertFalse(_is_channel_valid(channel))
+
+
+class TestTlsClient(unittest.TestCase):
+    """Test TLS Client integration."""
+
+    @mock.patch("charmhelpers.core.host.mkdir")
+    @mock.patch("charmhelpers.core.host.write_file")
+    @mock.patch("charmhelpers.core.hookenv.charm_dir")
+    @mock.patch("charms.layer.graylog.utils.is_v2")
+    @mock.patch("reactive.graylog.set_conf")
+    @mock.patch("reactive.graylog.hookenv.unit_public_ip")
+    @mock.patch("reactive.graylog.hookenv.unit_private_ip")
+    @mock.patch("reactive.graylog.install")
+    @mock.patch("subprocess.check_call")
+    def test_request_certificate(
+        self,
+        mock_check_call,
+        mock_install,
+        mock_private_ip,
+        mock_public_ip,
+        mock_set_conf,
+        mock_is_v2,
+        mock_charm_dir,
+        mock_write_file,
+        mock_mkdir,
+    ):  # noqa: D102
+        charm_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../")
+        mock_charm_dir.return_value = charm_dir
+        mock_is_v2.return_value = False
+        mock_public_ip.return_value = "1.2.3.4"
+        mock_private_ip.return_value = "5.6.7.8"
+        mock_tls = mock.Mock()
+        tls_request_certificate(mock_tls)
+
+        tls_client_mock.request_server_cert.assert_called_with(
+            "1.2.3.4",
+            ["1.2.3.4", "5.6.7.8", socket.gethostname(), "127.0.0.1", "localhost"],
+            crt_path=CERT_PATH,
+            key_path=CERT_KEY_PATH,
+        )
+        mock_set_conf.assert_has_calls(
+            [
+                mock.call("rest_enable_tls", "true"),
+                mock.call("rest_tls_cert_file", CERT_PATH),
+                mock.call("rest_tls_key_file", CERT_KEY_PATH),
+                mock.call("web_enable_tls", "true"),
+                mock.call("web_tls_cert_file", CERT_PATH),
+                mock.call("web_tls_key_file", CERT_KEY_PATH),
+            ]
+        )
+
+        with open(os.path.join(charm_dir, "templates/sync-graylog-snap"), "rb") as f:
+            sync_script = f.read().strip()
+
+        with open(
+            os.path.join(charm_dir, "templates/default-graylog-server"), "rb"
+        ) as f:
+            default_graylog_server = f.read().strip()
+
+        mock_write_file.assert_has_calls(
+            [
+                mock.call(
+                    "/etc/ca-certificates/update.d/sync-graylog-snap",
+                    sync_script,
+                    "root",
+                    "root",
+                    0o755,
+                ),
+                mock.call(
+                    "/var/snap/graylog/current/default-graylog-server",
+                    default_graylog_server,
+                    "root",
+                    "root",
+                    0o644,
+                ),
+            ]
+        )
+        mock_check_call.assert_called_with(["update-ca-certificates"])
diff --git a/src/tests/unit/test_lib.py b/src/tests/unit/test_lib.py
index 2f7a7c3..5849ac5 100644
--- a/src/tests/unit/test_lib.py
+++ b/src/tests/unit/test_lib.py
@@ -12,17 +12,29 @@ from charms.layer.graylog import (
 class TestLibraryUtils(unittest.TestCase):
     """Test lib.charms.layer.graylog utils."""
 
+    @mock.patch("charms.layer.graylog.utils.get_flags")
     @mock.patch("charms.layer.graylog.utils.is_v2")
-    def test_api(self, mock_v2):
+    def test_api(self, mock_v2, mock_get_flags):
         """Validate the api port/url match what we expect for v2 and v3."""
         mock_v2.return_value = True
         self.assertEqual(get_api_port(), "9001")
+        mock_get_flags.return_value = []
         self.assertEqual(get_api_url(), "http://127.0.0.1:9001/api/")
 
+        mock_get_flags.reset_mock()
+        mock_get_flags.return_value = ["graylog.certificates.configured"]
+        self.assertEqual(get_api_url(), "https://127.0.0.1:9001/api/")
+
         mock_v2.return_value = False
         self.assertEqual(get_api_port(), "9000")
+        mock_get_flags.reset_mock()
+        mock_get_flags.return_value = []
         self.assertEqual(get_api_url(), "http://127.0.0.1:9000/api/")
 
+        mock_get_flags.reset_mock()
+        mock_get_flags.return_value = ["graylog.certificates.configured"]
+        self.assertEqual(get_api_url(), "https://127.0.0.1:9000/api/")
+
     @mock.patch("charms.layer.graylog.utils.is_v2")
     def test_validate_api_url(self, mock_v2):
         """Validate the URI suffix is valid for v2 and v3."""