Merge ~rmescandon/influxdb-charm/+git/influxdb-charm:fix-connection-refused-on-install into influxdb-charm:master

[Roberto Mier Escandon]

From version 16 of the charm (the last not happening this problem) we have seen this error sometimes when installing from a bundle:

```
unit-influxdb-9: 17:48:10 ERROR unit.influxdb/9.juju-log Hook error:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 137, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 91, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 81, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 560, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 162, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 146, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
requests.packages.urllib3.exceptions.NewConnectionError: <requests.packages.urllib3.connection.HTTPConnection object at 0x7ff58777b9e8>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 610, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 273, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
requests.packages.urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8086): Max retries exceeded with url: /query?q=CREATE+USER+admin+WITH+PASSWORD+%27ce345c1b-3c99-4cf5-8885-f240723eb4e7%27+WITH+ALL+PRIVILEGES (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7ff58777b9e8>: Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-influxdb-9/.venv/lib/python3.5/site-packages/charms/reactive/__init__.py", line 73, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-influxdb-9/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 382, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-influxdb-9/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 358, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-influxdb-9/.venv/lib/python3.5/site-packages/charms/reactive/bus.py", line 180, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-influxdb-9/charm/reactive/influxdb.py", line 54, in install_influx
    configure_influxdb_auth()
  File "/var/lib/juju/agents/unit-influxdb-9/charm/reactive/influxdb.py", line 62, in configure_influxdb_auth
    create_influxdb_user(username, admin=True, local_user='root')
  File "lib/influxdbcharmlib.py", line 33, in create_influxdb_user
    influx.create_user(username, password, admin=admin)
  File "/usr/lib/python3/dist-packages/influxdb/client.py", line 649, in create_user
    self.query(text)
  File "/usr/lib/python3/dist-packages/influxdb/client.py", line 325, in query
    expected_response_code=expected_response_code
  File "/usr/lib/python3/dist-packages/influxdb/client.py", line 241, in request
    raise e
  File "/usr/lib/python3/dist-packages/influxdb/client.py", line 234, in request
    timeout=self._timeout
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 437, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8086): Max retries exceeded with url: /query?q=CREATE+USER+admin+WITH+PASSWORD+%27ce345c1b-3c99-4cf5-8885-f240723eb4e7%27+WITH+ALL+PRIVILEGES (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7ff58777b9e8>: Failed to establish a new connection: [Errno 111] Connection refused',))
```

I've realized that after starting influxdb.service the service is not fully ready (even when systemctl status says so) to be used by influxDBClient. Repeating the auth creation up to 3 times max solves the problem. In all my tests only first time fails sometimes

NOTE: Looks like the version 16 hadn't this problem because it executed several instructions of diverse complexity between service startup and auth configuration

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision eb3d41a350215d7b3312ea341e4ff03c48284845

diff --git a/lib/influxdbcharmlib.py b/lib/influxdbcharmlib.py
index 6f5cdfc..da6246b 100644
--- a/lib/influxdbcharmlib.py
+++ b/lib/influxdbcharmlib.py
@@ -30,7 +30,20 @@ def create_influxdb_user(username, admin=False, local_user=None):
     adminuser, adminpass = get_influxdb_credentials()
     influx = influxdb.InfluxDBClient(username=adminuser, password=adminpass)
     password = str(uuid.uuid4())
-    influx.create_user(username, password, admin=admin)
+
+    # Repeat the process of creating teh user until completed or max times
+    # reached because sometimes happens a race condition between the influxdb 
+    # service startup and the client use. In those cases the service is still 
+    # not fully started when the client is sending the first request
+    n = 3
+    while True:
+        try:
+            influx.create_user(username, password, admin=admin)
+            break
+        except Exception as ex:
+            n = n - 1
+            if n <= 0:
+                raise ex
 
     if local_user:
         passwd_entry = pwd.getpwnam(local_user)