Merge ~pjds/charm-telegraf:tls/prometheus-tls into charm-telegraf:master

[Peter Jose De Sousa]

TLS improvements for telegraf

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:6afaad62058749460dc6d6b8a9762941ed654dec
https://jenkins.canonical.com/bootstack/job/lp-charm-telegraf-ci/157/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/1326/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1655/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-telegraf-ci/157//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:abac31528b94190ef6c80e5df095e79be01cf94f
https://jenkins.canonical.com/bootstack/job/lp-charm-telegraf-ci/158/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/1337/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1667/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-telegraf-ci/158//rebuild

review: Needs Fixing (continuous-integration)

[Erhan Sunar (esunar)]

Can you fix this to pass unit tests
./tests/unit/test_telegraf.py:914:29: W291 trailing whitespace

review: Needs Fixing

[Tianqi Xiao (txiao)]

Thanks for your contribution. Aside from Erhan's suggestion of fixing the lint test, I have put a few inline comments and would like you to address them.

review: Needs Fixing

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:bf6e38c40ea8bc73a093f80e2a91967c5faf3acd
https://jenkins.canonical.com/bootstack/job/lp-charm-telegraf-ci/159/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/1348/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1677/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-telegraf-ci/159//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:6067b96cfbd14abaf3f4296fb37c02cdc1a6d9c1
https://jenkins.canonical.com/bootstack/job/lp-charm-telegraf-ci/160/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/1353/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1682/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-telegraf-ci/160//rebuild

review: Needs Fixing (continuous-integration)

[Peter Jose De Sousa (pjds)]

hey @txiao and @esunar thanks both for your prompt reviews, yes realised the work was below requirements so in the process of reworking it. Will ping you both when it's ready :-)

[Tianqi Xiao (txiao)]

No problem. I'm marking the MP as "Work in progress". When you are ready, you can change it back to "Needs review" and we will get notified.

[Peter Jose De Sousa (pjds)]

@esunar @txiao I believe these changes are ready now

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:d7de5ebbad0b4fc716c296ae6f2e64f8ac07fc9f
https://jenkins.canonical.com/bootstack/job/lp-charm-telegraf-ci/161/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/1452/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1704/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-telegraf-ci/161//rebuild

review: Needs Fixing (continuous-integration)

[Peter Jose De Sousa (pjds)]

fixed formatting again, I noticed there are 4 failing tests but they're also failing on main branch. Do these tests need fixing or is this expected?

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:c68b80207e0720af0ae6e4d1de423f12dfd370c2
https://jenkins.canonical.com/bootstack/job/lp-charm-telegraf-ci/162/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/1454/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/1706/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-telegraf-ci/162//rebuild

review: Approve (continuous-integration)

[Erhan Sunar (esunar)]

review: Approve

[Tianqi Xiao (txiao)]

LGTM

review: Approve

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision 1ea02f25cd000868c84d4056710c026d92914fed

diff --git a/src/layer.yaml b/src/layer.yaml
index c1937a1..347ac89 100644
--- a/src/layer.yaml
+++ b/src/layer.yaml
@@ -20,6 +20,7 @@ includes:
     - layer:leadership
     - layer:promreg-client
     - layer:snap
+    - layer:tls-client
     - interface:elasticsearch
     - interface:grafana-dashboard
     - interface:http
@@ -42,4 +43,10 @@ options:
             - sysstat
             - python3-yaml
             - python3-netifaces
+    tls-client:
+        ca_certificate_path: /etc/telegraf/tls/ca.crt
+        server_certificate_path: /etc/telegraf/tls/server.crt
+        server_key_path: /etc/telegraf/tls/server.key
+        client_certificate_path: /etc/telegraf/tls/client.crt
+        client_key_path: /etc/telegraf/tls/client.key
 repo: https://git.launchpad.net/telegraf-charm
diff --git a/src/reactive/telegraf.py b/src/reactive/telegraf.py
index 7e10d77..898c8e9 100644
--- a/src/reactive/telegraf.py
+++ b/src/reactive/telegraf.py
@@ -22,6 +22,7 @@ import io
 import ipaddress
 import json
 import os
+import pathlib
 import platform
 import re
 import socket
@@ -49,6 +50,7 @@ from charms.reactive import (
     set_flag,
     toggle_flag,
     when,
+    when_any,
     when_not,
 )
 from charms.reactive.bus import get_states
@@ -110,6 +112,8 @@ DEB_USER = "telegraf"
 RDT_MINIMUM_KERNEL_VERSION = (5, 4)
 RDT_MINIMUM_PKG_VERSION = "4.1-1"
 RDT_KERNEL_MODULE_NAME = "msr"
+TLS_CERT = pathlib.Path("/etc/telegraf/tls/server.crt")
+TLS_KEY = pathlib.Path("/etc/telegraf/tls/server.key")
 
 
 class InvalidInstallMethodError(Exception):
@@ -983,21 +987,30 @@ def install_telegraf():
         set_flag("telegraf.installed")
 
 
-@when("telegraf.installed")
-@when("apt.installed.telegraf")
-@when("plugins.prometheus-client.configured")
+@when_any(
+    "plugins.prometheus-client.configured",
+    "apt.installed.telegraf",
+    "telegraf.installed",
+    "tls_client.certs.saved",
+)
 @when_not("telegraf.configured")
 @when_not("telegraf.apt.configured")
 def configure_telegraf_deb():
+    # import pdb; pdb.set_trace()
     configure_telegraf()
 
 
-@when("telegraf.installed")
-@when("snap.installed.telegraf")
-@when("plugins.prometheus-client.configured")
+@when_any(
+    "plugins.prometheus-client.configured",
+    "apt.installed.telegraf",
+    "telegraf.installed",
+    "tls_client.certs.saved",
+)
+# @when_any("tls_client.certs.saved")
 @when_not("telegraf.configured")
 @when_not("telegraf.snap.configured")
 def configure_telegraf_snap():
+    # import pdb; pdb.set_trace()
     configure_telegraf()
 
 
@@ -1669,6 +1682,7 @@ def generate_prometheus_output_config(
     extra_options = get_extra_options()
     options = extra_options["outputs"].get("prometheus_client", {})
     listen = options.pop("listen", None)
+
     if not listen:
         if prometheus_output_address == "":
             listen = ":{}".format(prometheus_output_port)
@@ -1684,16 +1698,19 @@ def generate_prometheus_output_config(
             listen = get_prometheus_config_interface_ip(
                 prometheus_output_address, prometheus_output_port
             )
-
+        tls_config = configure_tls()
         options_ip_range = options.pop("ip_range", [])
         ip_range = options_ip_range + prometheus_ip_range
         if listen is not None:
+
             return {
                 "listen": listen,
                 "ip_range": ip_range,
                 "extra_options": render_extra_options(
                     "outputs", "prometheus_client", extra_options=extra_options
                 ),
+                "tls_cert": tls_config["tls_cert"],
+                "tls_key": tls_config["tls_key"],
             }
 
 
@@ -1771,6 +1788,7 @@ def render_prometheus_client_config(port, address, ip_range):
         ),
         level=hookenv.INFO,
     )
+
     context = generate_prometheus_output_config(port, address, ip_range)
     try:
         render(
@@ -1785,6 +1803,11 @@ def render_prometheus_client_config(port, address, ip_range):
         return
 
 
+@when_any(
+    "certificates.available",
+    "tls_client.certs.changed",
+    "tls_client.server.certs.changed",
+)
 @when("endpoint.prometheus-client.changed")
 def configure_prometheus_client_with_relation(prometheus):
     hookenv.log(
@@ -1835,7 +1858,13 @@ def configure_prometheus_client_with_relation(prometheus):
     clear_flag("endpoint.prometheus-client.changed")  # not automatic
 
 
-@when("telegraf.installed")
+@when_any(
+    "certificates.available",
+    "tls_client.certs.changed",
+    "tls_client.server.certs.changed",
+    "telegraf.installed",
+)
+# @when("telegraf.installed")
 @when_not("plugins.prometheus-client.configured")
 def configure_prometheus_client():
     hookenv.log("Configuring prometheus_client output plugin", level=hookenv.DEBUG)
@@ -2142,3 +2171,56 @@ def install_smart_metrics_packages():
 @when_not("apt.installed.intel-cmt-cat")
 def install_intel_rdt_packages():
     apt.queue_install(["intel-cmt-cat"])
+
+
+@when("certificates.available")
+def prepare_tls_certificates(tls):
+    from subprocess import CalledProcessError
+
+    try:
+        common_name = hookenv.unit_public_ip()
+    except CalledProcessError as e:
+        msg = "Public address not available yet"
+        hookenv.log(msg, hookenv.WARNING)
+        hookenv.log(e, hookenv.WARNING)
+        return
+
+    sans = set()
+    sans.add(common_name)
+    sans.update(hookenv.network_get("certificates"))
+    sans.add(socket.gethostname())
+
+    sans = sorted(sans)
+    certificate_name = hookenv.local_unit().replace("/", "_")
+    tls.request_server_cert(common_name, sans, certificate_name)
+
+    clear_flag("plugins.prometheus-client.configured")
+
+
+def configure_tls():
+    tls_config = {}
+    if TLS_CERT.exists() and TLS_KEY.exists():
+        tls_config["tls_cert"] = TLS_CERT.resolve()
+        tls_config["tls_key"] = TLS_KEY.resolve()
+        cmd = ["chown", "-R", "root:telegraf", "/etc/telegraf/tls"]
+        subprocess.check_call(cmd)
+        clear_flag("tls_client.certs.changed")
+        clear_flag("tls_client.server.certs.changed")
+
+    else:
+        tls_config["tls_cert"] = None
+        tls_config["tls_key"] = None
+
+    return tls_config
+
+
+@when("endpoint.certificates.departed")
+@when_not("certificates.available")
+def cleanup_tls():
+
+    if TLS_CERT.exists() and TLS_KEY.exists():
+        TLS_CERT.unlink()
+        TLS_KEY.unlink()
+    configure_prometheus_client()
+    clear_flag("plugins.prometheus-client.configured")
+    clear_flag("endpoint.certificates.departed")
diff --git a/src/templates/prometheus_client.tmpl b/src/templates/prometheus_client.tmpl
index a6e3c06..c65c911 100644
--- a/src/templates/prometheus_client.tmpl
+++ b/src/templates/prometheus_client.tmpl
@@ -1,4 +1,8 @@
 [[outputs.prometheus_client]]
   listen = "{{ listen }}"
   ip_range = {{ ip_range }}
+{%- if tls_cert and tls_key %}
+  tls_cert = "{{ tls_cert }}"
+  tls_key = "{{ tls_key }}"
+{%- endif %}
 {%- if extra_options %}{{ extra_options }}{%- endif %}
diff --git a/src/tests/unit/test_telegraf.py b/src/tests/unit/test_telegraf.py
index 6b453af..37a98ca 100644
--- a/src/tests/unit/test_telegraf.py
+++ b/src/tests/unit/test_telegraf.py
@@ -20,6 +20,7 @@ import getpass
 import grp
 import json
 import os
+import pathlib
 import platform
 import shutil
 import subprocess
@@ -829,7 +830,13 @@ def test_generate_prometheus_output_config_network(monkeypatch):
     )
     monkeypatch.setattr(telegraf.host, "list_nics", lambda: ["eth0"])
     result = telegraf.generate_prometheus_output_config(output_port, output_address, [])
-    expected = {"listen": "10.0.0.2:1234", "ip_range": [], "extra_options": []}
+    expected = {
+        "listen": "10.0.0.2:1234",
+        "ip_range": [],
+        "extra_options": [],
+        "tls_cert": None,
+        "tls_key": None,
+    }
     assert result == expected
 
 
@@ -841,7 +848,13 @@ def test_generate_prometheus_output_config_ip(monkeypatch):
         telegraf, "render_extra_options", lambda A, B, extra_options: []
     )
     result = telegraf.generate_prometheus_output_config(output_port, output_address, [])
-    expected = {"listen": "10.0.0.10:1234", "ip_range": [], "extra_options": []}
+    expected = {
+        "listen": "10.0.0.10:1234",
+        "ip_range": [],
+        "extra_options": [],
+        "tls_cert": None,
+        "tls_key": None,
+    }
     assert result == expected
 
 
@@ -854,7 +867,13 @@ def test_generate_prometheus_output_config_interface(monkeypatch):
     )
     monkeypatch.setattr(telegraf.host, "list_nics", lambda: ["eth5"])
     result = telegraf.generate_prometheus_output_config(output_port, output_address, [])
-    expected = {"listen": "10.0.0.20:1234", "ip_range": [], "extra_options": []}
+    expected = {
+        "listen": "10.0.0.20:1234",
+        "ip_range": [],
+        "extra_options": [],
+        "tls_cert": None,
+        "tls_key": None,
+    }
     assert result == expected
 
 
@@ -867,7 +886,13 @@ def test_generate_prometheus_output_config_interface_ipv4(monkeypatch):
     )
     monkeypatch.setattr(telegraf.host, "list_nics", lambda: ["eth5"])
     result = telegraf.generate_prometheus_output_config(output_port, output_address, [])
-    expected = {"listen": "10.0.0.20:1234", "ip_range": [], "extra_options": []}
+    expected = {
+        "listen": "10.0.0.20:1234",
+        "ip_range": [],
+        "extra_options": [],
+        "tls_cert": None,
+        "tls_key": None,
+    }
     assert result == expected
 
 
@@ -888,6 +913,8 @@ def test_generate_prometheus_output_config_interface_ipv6(monkeypatch):
         "listen": "2001:0db8:85a3:0000:0000:8a2e:0370:7334:1234",
         "ip_range": [],
         "extra_options": [],
+        "tls_cert": None,
+        "tls_key": None,
     }
     assert result == expected
 
@@ -1539,7 +1566,6 @@ def test_restart_on_output_plugin_relation_departed(mocker, monkeypatch, config)
     config["prometheus_output_port"] = ""
     config["prometheus_output_bindaddress"] = ""
     bus.discover()
-    set_flag("telegraf.installed")
     set_flag("snap.telegraf.installed")
     set_flag("telegraf.configured")
     interface = mocker.Mock(spec=RelationBase)
@@ -1983,3 +2009,72 @@ def test_check_valid_intel_rdt_configuration_pqos(monkeypatch):
         telegraf.InvalidIntelRDTConfigurationError, match="pqos -d failed"
     ):
         telegraf.check_valid_intel_rdt_configuration()
+
+
+def test_configure_tls(monkeypatch):
+    tnp_key_location = pathlib.Path("/tmp/fake-key")
+    tmp_cert_location = pathlib.Path("/tmp/fake-crt")
+    telegraf.TLS_KEY = tnp_key_location
+    telegraf.TLS_CERT = tmp_cert_location
+    monkeypatch.setattr(telegraf.subprocess, "check_call", lambda _: None)
+
+    files = [tmp_cert_location, tnp_key_location]
+
+    def operate_on_file(operation):
+        for file in files:
+            operation(file)
+
+    operate_on_file(lambda f: open(f, "w").write("testing"))
+
+    result = telegraf.configure_tls()
+
+    expected = {
+        "tls_cert": tmp_cert_location.resolve(),
+        "tls_key": tnp_key_location.resolve(),
+    }
+
+    assert result == expected
+
+    operate_on_file(lambda f: f.unlink())
+
+    result = telegraf.configure_tls()
+
+    expected = {"tls_cert": None, "tls_key": None}
+
+    assert result == expected
+
+
+def test_prepare_tls_certificates(mocker, monkeypatch):
+    monkeypatch.setattr(telegraf.host, "service_running", lambda n: True)
+
+    monkeypatch.setattr(telegraf.hookenv, "unit_public_ip", lambda: "0.0.0.0")
+    monkeypatch.setattr(telegraf.socket, "gethostname", lambda: "localhost")
+    open_ports = set()
+    monkeypatch.setattr(telegraf.hookenv, "open_port", lambda p: open_ports.add(p))
+    monkeypatch.setattr(telegraf.hookenv, "close_port", lambda p: open_ports.remove(p))
+    mocker.patch.object(
+        telegraf.hookenv, "network_get", return_value={"ingress_addresses": ["1.2.3.4"]}
+    )
+
+    set_flag("certificates.available")
+
+    class TLSObject:
+        def __init__(self) -> None:
+            pass
+
+        def request_server_cert(self, cn, sans, cert_name):
+            return None
+
+    telegraf.prepare_tls_certificates(TLSObject())
+
+
+def test_cleanup_tls(mocker, monkeypatch):
+    service_restart = mocker.patch("reactive.telegraf.host.service_restart")
+    monkeypatch.setattr(telegraf.host, "service_running", lambda n: True)
+    open_ports = set()
+    monkeypatch.setattr(telegraf.hookenv, "open_port", lambda p: open_ports.add(p))
+    monkeypatch.setattr(telegraf.hookenv, "close_port", lambda p: open_ports.remove(p))
+    set_flag("endpoint.certificates.departed")
+    bus.dispatch()
+    assert "plugins.prometheus-client.configured" in bus.get_states().keys()
+    assert service_restart.called_once()