Comment 13 for bug 1924816

Revision history for this message
Joe Guo (guoqiao) wrote :

From what I can see, the issue is caused by another python package `certifi`.

Here is how this package get used:

python-keystoneclient -> requests/certs.py -> certifi:where()

What does it do:

It's supposed to return where the single all-in-one cert file is.
You can try with: python3 -m certifi -c
And find out where is it: python3 -c "import certifi; print(certifi.__file__)"

With the pre-installed system version on ubuntu, certifi will return `/etc/ssl/certs/ca-certificates.crt`, which is correct.

The code is something like this (I checked both 18.04 and 20.04):

    cat /usr/lib/python3/dist-packages/certifi/core.py
    ...
    def where():
        f = os.path.dirname(__file__)
        return '/etc/ssl/certs/ca-certificates.crt'

However, above code is likely modified when packaging for distro releases.

The original content of core.py is here: https://github.com/certifi/python-certifi/blob/master/certifi/core.py

It will return a "cacert.pem" file inside the package, which includes Mozilla's Root Certificates.
(This behavior is same in old versions.)

So, when you are only using public certs, requests will work fine with https verify.
Because either /etc/ssl/certs/ca-certificates.crt or cacert.pem will have them included.

However, when charm is running in venv, the original version of certifi will be installed.
And it will return the content of this file:

/var/lib/juju/agents/unit-openstack-service-checks-0/.venv/lib/python3.6/site-packages/certifi/cacert.pem

The cert generated by vault, even we added it into /etc/ssl/certs/ca-certificates.crt, it's ignored.
That's how we get the [SSL: CERTIFICATE_VERIFY_FAILED] error.

If above theory is correct, then only cs:openstack-service-check-3 will work, since it doesn't use certifi.
You can check here: https://jaas.ai/openstack-service-checks/3, by click the "+" on "/wheelhouse".

For revision 4+, requests and certifi packages are in wheelhouse/: https://jaas.ai/openstack-service-checks/4
which will introduce the problem.