trusted_ssl_ca does not configure /etc/ca-certificates.conf or land file in proper directory for system CA trust

Workarounds:

1. If you have multiple certs in the bundle, login to openstack-service-checks/0 and separate each certificate in /usr/local/share/ca-certificates/openstack-service-checks.crt into it's own separate file in /usr/share/ca-certificates (such as openstack-service-checks-1.crt and openstack-service-checks-2.crt).

2. login to the openstack-service-checks/0 unit and run the following as root
cp /usr/local/share/ca-certificates/openstack-service-checks.crt /usr/share/ca-certificates/openstack-service-checks.crt
chmod 644 /usr/share/ca-certificates/openstack-service-checks.crt
echo openstack-service-checks.crt >> /etc/ca-certificates.conf
update-ca-certificates

Additional note, this may be due to CIS standards hardening on the site where this occurred. Ping me if you need details on the hardening applied.

I'm seeing this on a non-CIS environment.

This charm needs to grow support for the vault certificates interface to connect to keystone that is ssl managed by vault's certificates interface.

I can reproduce this issue using ssl_* options as well. This is not limited to keystone with vault certificates.

Thanks for the info in dupe-bug https://bugs.launchpad.net/charm-openstack-service-checks/+bug/1926670, @pguimaraes!

I'm working on testing a rebuild with requests=2.18.4 specified in wheelhouse, as the bug is in requests 2.25.

There's a quick fix released in the edge channel for this charm to hard-code the requests==2.18.4

cs:openstack-service-checks-5

https://jaas.ai/openstack-service-checks/5

1) From update-ca-certificates man page:

    Furthermore all certificates with a .crt extension found below /usr/local/share/ca-certificates are also included as implicitly trusted.

This is same on all supported Ubuntu (18.04+): http://manpages.ubuntu.com/manpages/bionic/man8/update-ca-certificates.8.html

So the charm is putting cert into right location, no change needed.

2) The root issue is introduced by urllib3 1.25.4:

https://github.com/urllib3/urllib3/issues/1682

This is already fixed in 1.25.5, released in 2019-09:

https://github.com/urllib3/urllib3/compare/1.25.4...1.25.5

`requests` starts to use it since 2.22.0:

https://github.com/psf/requests/commit/aeda65bbe57ac5edbcc2d80db85d010befb7d419
'urllib3>=1.21.1,<1.26,!=1.25.0,!=1.25.1'

So as long as we can avoid urllib3 1.25.4, either higher or lower, either via itself or requests, issue can be fixed.

A patch created to avoid urllib3 1.25.4 with minimal impact:

https://code.launchpad.net/~guoqiao/charm-openstack-service-checks/+git/charm-openstack-service-checks/+merge/402239

Hi @guoqiao,

Very nice digging! Thanks for narrowing down this issue. These are the tests that have been done on for this problem:

1) Bionic + o-s-c latest stable (v4): stays in BLOCKED: urllib 1.26 + requests 2.25
2) Bionic + o-s-c stable v3: moves to READY: urllib 1.22 + requests 2.18
3) Focal + o-s-c latest stable (v4): stays in BLOCKED: urllib 1.26 + requests 2.25
4) Focal + o-s-c stable v3: stays in BLOCKED: urllib 1.25.8 + requests 2.22.0

What differs here: essentially if we use v3, we won't have urllib or requests libs in the wheelhouse/ folder of the charm. The charm will use whatever we ship on the OS.

In this case, Focal uses:
urllib: 1.25.8, requests: 2.22.0 (checked on one server, packages installed via our repos)

Bionic uses:
urllib: 1.22, requests: 2.18 (got it from packages.ubuntu.com)

If we compare the test scenarios above: 2 vs. 4, we did run a test with 1.25.8 and it failed.

The bug on urllib also has comments saying that 1.25.5+ versions did not entirely fix it.
In special, this comment: https://github.com/urllib3/urllib3/issues/1682#issuecomment-533311857

Hi @pguimaraes:

I did some more tests on Ubuntu 16.04/18.04/20.04, according to the pkg versions you mentioned.

Here is my result:

https://docs.google.com/spreadsheets/d/1I_SYJCWZgdOU5eriQ3Uh7UxsRzIf27cK0EvxRYkCE7w/edit?usp=sharing

The only failure is on 18.04, which has OpenSSL 1.1.1, with urllib3 1.25.4.

The test script I am using is from this comment:

https://github.com/urllib3/urllib3/issues/1682#issuecomment-668664791

In your tests, O-S-C stays in BLOCKED may not be a accurate indicator of the issue.
I do noticed sometimes juju status takes longer than expected to settle, and may be affected by other components.

In one of your blocked env, could you help to also run the above test script?
You may need to adjust the url to an openstack endpoint for networking limits.

Also, about the comment you linked:

    It's a bug in old versions of Python that only occurs when Python's ssl module is compiled with OpenSSL 1.1.1. The problem is fixed in recent versions:

    Python 2.7 and 3.5 are not affected because they don't support PHA
    Python 3.6 is affected and cannot be fixed, because 3.6 in security bug-only mode.
    Python 3.7.4 is fixed. 3.7.3 and earlier are affected.
    3.8-beta is fine.

there could be another understanding:

this issue could be fixed form the python binary, but can also fix/workaround from the urllib3 package.
Another comment from sethmlarson also says: "This is closed in v1.25.5."

After more digging, I discovered that this issue is not caused by urllib3 1.25.4 or OpenSSL 1.1.1.

The urllib3 bug: https://github.com/urllib3/urllib3/issues/1682

It was really about:

When you access a self-signed https url and turned off https verify,
you still get CERTIFICATE_VERIFY_FAILED error, since urllib3 still verified it by mistake.

That is fixed in 1.25.5: https://github.com/urllib3/urllib3/compare/1.25.4...1.25.5

However, in our case, we are not turning off https verify.
Instead, we expect our cert is installed on system and requests/urllib3 can find it and verify ok.
But we actually get this error:

    unit.openstack-service-checks/0.juju-log Failed to create endpoint checks due issue communicating with Keystone. Error:
    Keystone ssl error when listing SSL exception connecting to https://10.0.8.91:35357/v3/auth/tokens:
    HTTPSConnectionPool(host='10.0.8.91', port=35357): Max retries exceeded with url:
    /v3/auth/tokens (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED]
    certificate verify failed (_ssl.c:852)'),)): endpoints

From what I can see, the issue is caused by another python package `certifi`.

Here is how this package get used:

python-keystoneclient -> requests/certs.py -> certifi:where()

What does it do:

It's supposed to return where the single all-in-one cert file is.
You can try with: python3 -m certifi -c
And find out where is it: python3 -c "import certifi; print(certifi.__file__)"

With the pre-installed system version on ubuntu, certifi will return `/etc/ssl/certs/ca-certificates.crt`, which is correct.

The code is something like this (I checked both 18.04 and 20.04):

    cat /usr/lib/python3/dist-packages/certifi/core.py
    ...
    def where():
        f = os.path.dirname(__file__)
        return '/etc/ssl/certs/ca-certificates.crt'

However, above code is likely modified when packaging for distro releases.

The original content of core.py is here: https://github.com/certifi/python-certifi/blob/master/certifi/core.py

It will return a "cacert.pem" file inside the package, which includes Mozilla's Root Certificates.
(This behavior is same in old versions.)

So, when you are only using public certs, requests will work fine with https verify.
Because either /etc/ssl/certs/ca-certificates.crt or cacert.pem will have them included.

However, when charm is running in venv, the original version of certifi will be installed.
And it will return the content of this file:

/var/lib/juju/agents/unit-openstack-service-checks-0/.venv/lib/python3.6/site-packages/certifi/cacert.pem

The cert generated by vault, even we added it into /etc/ssl/certs/ca-certificates.crt, it's ignored.
That's how we get the [SSL: CERTIFICATE_VERIFY_FAILED] error.

If above theory is correct, then only cs:openstack-service-check-3 will work, since it doesn't use certifi.
You can check here: https://jaas.ai/openstack-service-checks/3, by click the "+" on "/wheelhouse".

For revision 4+, requests and certifi packages are in wheelhouse/: https://jaas.ai/openstack-service-checks/4
which will introduce the problem.

Back to Pedro's tests in comment #10:
1) 2) 3) make sense, but 4) doesn't.
It's also v3 (rev3), which should work.
But since rev3 doesn't need urllib3 and requests, where are their versions from?
Are they system/global versions or in venv ?

There are a few ways to let requests use correct system certs:

1) do not use venv

so certifi deb package will be used. NA to our case and not ideal.

2) pass ca path to requests session:

s = requests.Session()
s.verify = False
s.request("GET", URL, verify="/etc/ssl/certs/ca-certificates.crt")

this verify arg can be boolean or path.

NA to our case as well, since we are not using request directly.

3) envvar
export REQUESTS_CA_BUNDLE="/etc/ssl/certs/ca-certificates.crt"

this will ensure anywhere when requests is used in this charm, it will use system certs.
Unless envvar is not respected which is possible.

4) use verify arg on keystoneclient session

sess = session.Session(auth=auth, verify="/etc/ssl/certs/ca-certificates.crt")

according to code comment, this verify arg will be pass down to requests as it is.

In our case, we can use both 3) and 4).
So not only keystone, all other usage of requests will be fixed.

(sorry, ignore the `s.verify = False` line above.)

I have been able to reproduce this issue with openstack-on-lxd/bundle-bionic-train.yaml + vault.

A new patch/MP with above fix created:

https://code.launchpad.net/~guoqiao/charm-openstack-service-checks/+git/charm-openstack-service-checks/+merge/402381

In my test env, once patch applied, the SSL error will resolve in a few minutes.

Review/test/comment appreciated.

Before charm release and upgrade, the easiest workaround I can think of so far:

On openstack-service-checks unit, modify the `certifi/core.py:where()` in venv to return system bundle and restart charm.

Example:

N=0
juju ssh openstack-service-checks/$N
sudo bash
N=0
cd /var/lib/juju/agents/unit-openstack-service-checks-$N/.venv/lib/python3.6/site-packages/certifi/
cp core.py core.py.orig
vim core.py

Just put this in:

    #!/usr/bin/env python3
    def where():
        return '/etc/ssl/certs/ca-certificates.crt'

restart:

    systemctl restart jujud-unit-openstack-service-checks-$N.service

Issue should self resolve in a few mins.

Just had a try on focal, it turns out the new version of requests also need the contents method.
So a full working version of core.py will be like this:

cat /var/lib/juju/agents/unit-openstack-service-checks-2/.venv/lib/python3.8/site-packages/certifi/core.py

#!/usr/bin/env python3

def where():
    return "/etc/ssl/certs/ca-certificates.crt"

def contents():
    with open(where()) as f:
        return f.read()

I have confirmd with above core.py + service restart, o-s-c SSL error will resolve on focal.

Applying the workaround above worked for me. Focal/Ussuri deployment.

Workaround :

juju ssh openstack-service-checks/0

sudo vi /var/lib/juju/agents/unit-openstack-service-checks-2/.venv/lib/python3.8/site-packages/certifi/core.py

#!/usr/bin/env python3

def where():
    return "/etc/ssl/certs/ca-certificates.crt"

def contents():
    with open(where()) as f:
        return f.read()

systemctl restart jujud-unit-openstack-service-checks-$N.service

I hit this today as well.

I independently found the same issue as Joe - that things work with system packages because the Ubuntu-packaged lib has patches applied. (Specifically: https://pastebin.ubuntu.com/p/tdf5kXc26Z/)

I agree that either modifying the where() would work, or modifying the get() or the Session() to take a verify flag.

Probably the easiest one to implement is adding the verify flag at Session instantiation, so as to avoid the need to modify pip-installed packages - but frankly, I think any of these approaches would work for how the charm is presently implemented.

Hi Paul,

To fix the current issue, there are 2 ways:

1) add "/etc/ssl/certs/ca-certificates.crt" as `verify` arg to keystoneclient.Session
2) set env var export REQUESTS_CA_BUNDLE="/etc/ssl/certs/ca-certificates.crt"

For 1), it will only affect the requests usage in keystoneclient.
For 2), it will change the requests behavior all over the charm.

Considering we may also being using/will use requests in other places in this charm, I have chosen 2) as the solution. The patch is merged by xav into master, but not released to mainline yet:

https://code.launchpad.net/~guoqiao/charm-openstack-service-checks/+git/charm-openstack-service-checks/+merge/402381

Also, I have released it as cs:~llama-charmers-next/openstack-service-checks-12.
So, in addition to modifying `venv/.../certifi/core.py`, the other workaround is to switch charm temporarily to this uri.

We hit this issue in test runs recently, switching to cs:~llama-charmers-next/openstack-service-checks-12 helped, thank you

Can this be backported to the stable version ? It's blocking UA handovers

Can you not just use the -next version of the charm to unblock handovers?