Merge ~raychan96/charm-graylog:fix_temporarily_es_down into charm-graylog:master

[Chi Wai CHAN]

Add elasticsearch.configured state to better handle the corner case that graylog tries to configure ES when ES might be temporarily down due to internal services changed.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:b16b271a1c016bf67abd0d665bd8d07f0ff42377
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/5/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/57/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/24/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/5//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:b16b271a1c016bf67abd0d665bd8d07f0ff42377
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/6/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/70/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/26/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/6//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:b16b271a1c016bf67abd0d665bd8d07f0ff42377
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/7/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/96/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/27/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/7//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:f1f0dab42aceebf76c4930b6323ac391ea3b4baa
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/8/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/97/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/28/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/8//rebuild

review: Needs Fixing (continuous-integration)

[Chi Wai CHAN (raychan96)]

The functional test keeps failing for different reasons. The recent test failed because of apparmor is blocking the open of graylog snap:

```
kernel: [ 268.578819] audit: type=1400 audit(1676865526.269:41): apparmor="DENIED" operation="open" profile="snap.graylog.graylog" name="/proc/20073/mountinfo" pid=20073 comm="java" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
...
systemd[1]: snap.graylog.graylog.service: Failed with result 'exit-code'.
```

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:f1f0dab42aceebf76c4930b6323ac391ea3b4baa
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/9/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/98/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/29/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/9//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:4e366bd556da12a25388c7703d1e55f82a0f3884
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/10/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/101/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/30/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/10//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:4e366bd556da12a25388c7703d1e55f82a0f3884
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/11/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/106/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/35/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/11//rebuild

review: Needs Fixing (continuous-integration)

[JamesLin (jneo8)]

Functional test keep failing.

review: Needs Fixing

[Chi Wai CHAN (raychan96)]

> Functional test keep failing.

It's failing for different reasons if you look at the previous runs (e.g graylog api failed because of unauthorized / kernel security module block the snap installation ...). I don't really think this is caused by this patch, but I maybe wrong.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:4e366bd556da12a25388c7703d1e55f82a0f3884
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/12/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/113/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/42/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/12//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:f1f0dab42aceebf76c4930b6323ac391ea3b4baa
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/13/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/120/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/49/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/13//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:3d0fe91f16baaddc07dfec5485ff8c335e5ab944
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/14/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/130/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/58/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/14//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:2e812fdfd057e0ff1a8634d714244e857ce34752
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/15/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-functest/133/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/61/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-graylog-ci/15//rebuild

review: Approve (continuous-integration)

[Chi Wai CHAN (raychan96)]

It finally passed the CI!

[Erhan Sunar (esunar)]

LGTM

review: Approve

[Ramesh Sattaru (rameshcan)]

review: Approve

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision 06202c3a18baf796676ac309994f1291b3cf4e0e

diff --git a/src/reactive/graylog.py b/src/reactive/graylog.py
index b90d1f4..282374d 100644
--- a/src/reactive/graylog.py
+++ b/src/reactive/graylog.py
@@ -778,8 +778,10 @@ def configure_elasticsearch(elasticsearch):
     try:
         es.disable_auto_create_index()
     except Exception as e:
-        hookenv.log("Error configuring ES: {}".format(e))
-        raise ConnectionError
+        hookenv.log(
+            "Error configuring ES: {}. Will retry after update status.".format(e)
+        )
+        return
 
     if not cluster:
         conf = hookenv.config()
diff --git a/src/tests/functional/tests/test_graylog_charm.py b/src/tests/functional/tests/test_graylog_charm.py
index 83d58e7..1f2c8fd 100644
--- a/src/tests/functional/tests/test_graylog_charm.py
+++ b/src/tests/functional/tests/test_graylog_charm.py
@@ -136,7 +136,7 @@ class CharmOperationTest(BaseGraylogTest):
         }
         for attempt in tenacity.Retrying(
             wait=tenacity.wait_fixed(10),  # seconds
-            stop=tenacity.stop_after_attempt(3),
+            stop=tenacity.stop_after_attempt(70),
             reraise=True,
         ):
             with attempt: