Merge ~verterok/charm-graylog:beats-lb into charm-graylog:master

[Guillermo Gonzalez]

Add support to loadbalance beats protocol using a haproxy in front of a cluster via beats-lb relation

This adds a beats-lb relation, which allows to use haproxy as a reverseproxy for the beats protocol

No tests added, as I was not sure which kind of test a new relation is required for this change: unit or functional? Also, I couldn't find any testing related to haproxy relations, please point me out if I missed something.

e.g:
# deploy a graylog cluster

juju deploy haproxy

juju add-relation haproxy:reverseproxy graylog:beats-lb

juju config haproxy services=$(cat <<-END
- service_name: beats
  service_host: "0.0.0.0"
  service_port: 5044
  service_options:
      - mode tcp
      - balance leastconn
      - timeout server 30000
      - timeout client 30000
  server_options: check inter 2000 rise 2 fall 5 maxconn 200
END
)

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[James Troup (elmo)]

I'd personally be fine with a unit test for this, but given the typo, I do think it needs at least that. I don't object to a functional test, if you want, but given the code change, it seems like overkill to me. I'm curious about the use case?

(See comment inline.)

review: Needs Fixing

[Guillermo Gonzalez (verterok)]

Indeed, this was intended as a PoC of the idea.
As disucssed in MM:
[17:02] │ verterok: I have no proofs, but we have a huge log volume, and some units see a lot more traffic than others. so, if a couple of those "busy" units land in the same graylog node...ka-boom?
[17:02] elmo: ah, I see

[Guillermo Gonzalez (verterok)]

fixed and pushed

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:1cc3f08c7d862b71257931d3a5b0428424eac7dd
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/53/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/197/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/645/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/53//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:1cc3f08c7d862b71257931d3a5b0428424eac7dd
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/54/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/198/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/646/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/54//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:1cc3f08c7d862b71257931d3a5b0428424eac7dd
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/55/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/199/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/647/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/55//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:1cc3f08c7d862b71257931d3a5b0428424eac7dd
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/56/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/200/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/648/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/56//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:1cc3f08c7d862b71257931d3a5b0428424eac7dd
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/57/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/201/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/649/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/57//rebuild

review: Needs Fixing (continuous-integration)

[James Troup (elmo)]

CI still fails, but it fails on trunk too. Approving this for now.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision ab29d166b1eb89596a7c74e4b7499f8f746c3fc6

diff --git a/src/metadata.yaml b/src/metadata.yaml
index c5bdd50..4a88639 100644
--- a/src/metadata.yaml
+++ b/src/metadata.yaml
@@ -19,6 +19,8 @@ provides:
     interface: http
   beats:
     interface: elastic-beats
+  beats-lb:
+    interface: http
 requires:
   elasticsearch:
     interface: elasticsearch
diff --git a/src/reactive/graylog.py b/src/reactive/graylog.py
index b554506..2bb524d 100644
--- a/src/reactive/graylog.py
+++ b/src/reactive/graylog.py
@@ -838,6 +838,19 @@ def update_reverseproxy_config(website):
 
 
 @when("graylog.configured")
+@when("beats-lb.available")
+def update_beats_reverseproxy_config(beats_lb):
+    """Share Graylog beats endpoint details for a reverse proxy to use."""
+    conf = hookenv.config()
+    beats_port = conf["beats_port"]
+    if not data_changed("beats_port", beats_port):
+        # nothing changed
+        return
+    if beats_port:
+        beats_lb.configure(port=beats_port)
+
+
+@when("graylog.configured")
 @when("graylog.needs_restart")
 def restart_service(service=SERVICE_NAME):
     """Restart the service and on failure wait 15 seconds and try again.
diff --git a/src/tests/functional/tests/bundles/bionic-graylog2-ha.yaml b/src/tests/functional/tests/bundles/bionic-graylog2-ha.yaml
index 0f34a96..f1376ab 100644
--- a/src/tests/functional/tests/bundles/bionic-graylog2-ha.yaml
+++ b/src/tests/functional/tests/bundles/bionic-graylog2-ha.yaml
@@ -41,7 +41,7 @@ relations:
     - mongo
   - - graylog
     - elastic
-  - - graylog
-    - haproxy
+  - - graylog:website
+    - haproxy:reverseproxy
   - - graylog
     - nrpe
diff --git a/src/tests/functional/tests/bundles/bionic-graylog2.yaml b/src/tests/functional/tests/bundles/bionic-graylog2.yaml
index f97b308..09d2597 100644
--- a/src/tests/functional/tests/bundles/bionic-graylog2.yaml
+++ b/src/tests/functional/tests/bundles/bionic-graylog2.yaml
@@ -45,8 +45,8 @@ relations:
     - mongo
   - - graylog
     - elastic
-  - - graylog
-    - haproxy
+  - - graylog:website
+    - haproxy:reverseproxy
   - - graylog
     - nrpe
   - - nagios
diff --git a/src/tests/functional/tests/bundles/bionic-graylog3-ha.yaml b/src/tests/functional/tests/bundles/bionic-graylog3-ha.yaml
index ab2238a..8717729 100644
--- a/src/tests/functional/tests/bundles/bionic-graylog3-ha.yaml
+++ b/src/tests/functional/tests/bundles/bionic-graylog3-ha.yaml
@@ -41,7 +41,7 @@ relations:
     - mongo
   - - graylog
     - elastic
-  - - graylog
-    - haproxy
+  - - graylog:website
+    - haproxy:reverseproxy
   - - graylog
     - nrpe
diff --git a/src/tests/functional/tests/bundles/bionic-graylog3.yaml b/src/tests/functional/tests/bundles/bionic-graylog3.yaml
index a4509c3..2c6d6da 100644
--- a/src/tests/functional/tests/bundles/bionic-graylog3.yaml
+++ b/src/tests/functional/tests/bundles/bionic-graylog3.yaml
@@ -45,8 +45,8 @@ relations:
     - mongo
   - - graylog
     - elastic
-  - - graylog
-    - haproxy
+  - - graylog:website
+    - haproxy:reverseproxy
   - - graylog
     - nrpe
   - - nagios
diff --git a/src/tests/functional/tests/bundles/bionic-ha.yaml b/src/tests/functional/tests/bundles/bionic-ha.yaml
index 8e19581..711b68b 100644
--- a/src/tests/functional/tests/bundles/bionic-ha.yaml
+++ b/src/tests/functional/tests/bundles/bionic-ha.yaml
@@ -42,7 +42,7 @@ relations:
     - mongo
   - - graylog
     - elastic
-  - - graylog
-    - haproxy
+  - - graylog:website
+    - haproxy:reverseproxy
   - - graylog
     - nrpe
diff --git a/src/tests/functional/tests/bundles/focal-graylog2.yaml b/src/tests/functional/tests/bundles/focal-graylog2.yaml
index bb513e6..634fea7 100644
--- a/src/tests/functional/tests/bundles/focal-graylog2.yaml
+++ b/src/tests/functional/tests/bundles/focal-graylog2.yaml
@@ -45,8 +45,8 @@ relations:
     - mongo
   - - graylog
     - elastic
-  - - graylog
-    - haproxy
+  - - graylog:website
+    - haproxy:reverseproxy
   - - graylog
     - nrpe
   - - nagios
diff --git a/src/tests/functional/tests/bundles/focal-graylog3.yaml b/src/tests/functional/tests/bundles/focal-graylog3.yaml
index ae0991d..1ada366 100644
--- a/src/tests/functional/tests/bundles/focal-graylog3.yaml
+++ b/src/tests/functional/tests/bundles/focal-graylog3.yaml
@@ -45,8 +45,8 @@ relations:
     - mongo
   - - graylog
     - elastic
-  - - graylog
-    - haproxy
+  - - graylog:website
+    - haproxy:reverseproxy
   - - graylog
     - nrpe
   - - nagios
diff --git a/src/tests/unit/test_graylog.py b/src/tests/unit/test_graylog.py
index dc53ed7..1f9a57a 100644
--- a/src/tests/unit/test_graylog.py
+++ b/src/tests/unit/test_graylog.py
@@ -32,6 +32,7 @@ from reactive.graylog import (  # noqa: E402
     set_conf,
     set_jvm_heap_size,
     tls_request_certificate,
+    update_beats_reverseproxy_config,
     update_config,
     upgrade_charm,
     validate_jvm_heap_size,
@@ -894,3 +895,39 @@ class TestEmailConfig(unittest.TestCase):
 class Config(dict):
     def __init__(self, *args, **kw):
         super(Config, self).__init__(*args, **kw)
+
+
+class BeatsLBTestCase(unittest.TestCase):
+    @mock.patch("charmhelpers.core.hookenv.config")
+    @mock.patch("reactive.graylog.data_changed")
+    def test_update_beats_reverseproxy_config(
+        self,
+        mock_data_changed,
+        mock_hookenv_config,
+    ):
+        mock_hookenv_config.return_value = {
+            "beats_port": 5041,
+        }
+        mock_data_changed.return_value = True
+        mock_interface = mock.Mock()
+
+        update_beats_reverseproxy_config(mock_interface)
+
+        mock_interface.configure.assert_called_once_with(port=5041)
+
+    @mock.patch("charmhelpers.core.hookenv.config")
+    @mock.patch("reactive.graylog.data_changed")
+    def test_update_beats_reverseproxy_config_no_config_change(
+        self,
+        mock_data_changed,
+        mock_hookenv_config,
+    ):
+        mock_hookenv_config.return_value = {
+            "beats_port": 5041,
+        }
+        mock_data_changed.return_value = False
+        mock_interface = mock.Mock()
+
+        update_beats_reverseproxy_config(mock_interface)
+
+        mock_interface.configure.assert_not_called()