Merge ~guoqiao/charm-openstack-service-checks:lp1924816-requests-ca-bundle into charm-openstack-service-checks:master

[Joe Guo]

ensure requests to use system ca bundle for ssl verify

`keystoneclient` will use `requests` to access api endpoints.
When https/ssl is enabled, `requests` will rely on package `certifi` to find ca certs for ssl verify.

However, `certifi` has different behavior:

- in python package, it will return builtin `cacert.pem` which is Mozilla Root Certificates.
- in deb package, it's modified to return `/etc/ssl/certs/ca-certificates.crt` as expected.

When we use vault, keystone endpoints will be https and ssl verify is needed.
The ca cert configured via `trusted_ssl_ca` will be merged into `/etc/ssl/certs/ca-certificates.crt`.

This is ok if charm is running globally without venv (certifi deb package is used).
But when charm is running in venv(certifi python package is used),
above cert will be ignored by requests and cause [SSL: CERTIFICATE_VERIFY_FAILED] error.

This patch set envvar REQUESTS_CA_BUNDLE to system ca bundle, so
requests will use it as ca cert, instead of `.venv/.../certifi/cacert.pem`.

Related bugs:

LP: #1924816
LP: #1926670

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:6ce0b38515e20147929fb7cdedc6343d3ad13e46
https://jenkins.canonical.com/bootstack/job/lp-charm-openstack-service-checks-ci/13/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit-withcompiler/22/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/191/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-openstack-service-checks-ci/13//rebuild

review: Approve (continuous-integration)

[Linda Guo (lihuiguo)]

approved

review: Approve

[Xav Paice (xavpaice)]

one comment inline. Might be good to rethink a little, or get another opinion.

[Xav Paice (xavpaice)]

it's been pointed out to me that /etc/ssl/certs/ca-certificates.crt includes all the certs available to the machine, and therefore is a more useful store than the default one that comes with requests.

In which case, let's remove the verify=REQUESTS_CA_BUNDLE and make use of the env var for this, and all the other, requests sessions.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:f65a209d5db724050b04a387a4a7d51c22e481c8
https://jenkins.canonical.com/bootstack/job/lp-charm-openstack-service-checks-ci/14/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit-withcompiler/24/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/196/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-openstack-service-checks-ci/14//rebuild

review: Needs Fixing (continuous-integration)

[Joe Guo (guoqiao)]

Hi Xav,

Thanks for your review.

I have verified that with only the envvar, SSL verify is also fixed.

A bugfix rev is released to `cs:~llama-charmers-next/openstack-service-checks-11` which can be used as test or work around.

Modified code re-pushed, new review appreciated.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:0be89e2704ce636fa9f9d1df54d264971a3ba26e
https://jenkins.canonical.com/bootstack/job/lp-charm-openstack-service-checks-ci/15/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/lp-charm-test-unit-withcompiler/25/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/197/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/lp-charm-openstack-service-checks-ci/15//rebuild

review: Approve (continuous-integration)

[Xav Paice (xavpaice)]

LGTM

review: Approve

diff --git a/src/lib/lib_openstack_service_checks.py b/src/lib/lib_openstack_service_checks.py
index aecb859..e30dd19 100644
--- a/src/lib/lib_openstack_service_checks.py
+++ b/src/lib/lib_openstack_service_checks.py
@@ -22,6 +22,19 @@ import keystoneauth1
 
 from keystoneclient import session
 
+# `requests` relies on package `certifi` to find ca certs.
+# certifi deb package will return `/etc/ssl/certs/ca-certificates.crt` as expected,
+# while its python package will return its builtin `cacert.pem`, which is
+# Mozilla Root Certificates.
+# when charm runs in venv, cetifi python package will be installed.
+# So when we use self-signed certs (e.g: form vault) to verify, even if
+# we included it in `/etc/ssl/certs/ca-certificates.crt`, it will be ignored
+# by requests and cause [SSL: CERTIFICATE_VERIFY_FAILED] error.
+# ref: LP#1924816
+# this envvar will ensure requests to use system bundle for ssl verify
+# instead of `certifi/cacert.pem`
+os.environ["REQUESTS_CA_BUNDLE"] = "/etc/ssl/certs/ca-certificates.crt"
+
 
 class OSCCredentialsError(Exception):
     """Define OSCCredentialError exception."""