During update-status hook: "Could not find a suitable TLS CA certificate bundle, invalid path: /etc/ssl/certs/ca-certificates.crt" can happen

The API service is affected every 5 minutes.

$ grep 'Clearing symlinks in /etc/ssl/certs' /var/log/juju/unit-neutron-api-0.log
2018-04-09 11:57:11 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 11:57:49 DEBUG config-changed Clearing symlinks in /etc/ssl/certs...
2018-04-09 11:57:58 DEBUG config-changed Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:01:33 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:06:50 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:11:48 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:15:54 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:19:58 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:26:06 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:31:14 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:35:53 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:41:54 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:46:50 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:51:18 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 12:56:25 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:00:25 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:05:47 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:11:18 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:17:03 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:21:22 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:27:18 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:32:47 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:37:43 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:43:23 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:47:21 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:52:34 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 13:56:42 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 14:00:52 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 14:05:54 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 14:10:37 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 14:14:39 DEBUG update-status Clearing symlinks in /etc/ssl/certs...
2018-04-09 14:20:28 DEBUG update-status Clearing symlinks in /etc/ssl/certs...

Subscribing ~field-high since the API disruption in every 5 minutes is hard to be justified.

https://github.com/openstack/charm-neutron-api/blob/stable/18.02/hooks/charmhelpers/contrib/hahelpers/apache.py#L84-L95

$ sha256sum /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt
f2d19eb104885f6911cae938a34a10f5466df4f6ffef64b0871800ef1f65b0ff /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt

$ head -n1 /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt
-----BEGIN CERTIFICATE-----

^^^ plain text.

https://github.com/openstack/charm-neutron-api/blob/stable/18.02/hooks/charmhelpers/contrib/openstack/context.py#L803-L806
https://github.com/openstack/charm-neutron-api/blob/stable/18.02/hooks/charmhelpers/contrib/hahelpers/apache.py#L63-L73

$ juju run --unit neutron-api/0 'config-get ssl_ca | base64 -d | sha256sum'
f2d19eb104885f6911cae938a34a10f5466df4f6ffef64b0871800ef1f65b0ff -

^^^ with base64 decode, hashsum matches.

> if old_cert and old_cert == ca_cert:
> log("CA cert is the same as installed version", level=INFO)

Wondering why those do not match for 2. below.

> 1. update-status hook shouldn't change the files. It should be read-only.
> 2. Even with other hooks, the charm shouldn't touch /etc/ssl/certs or /etc/ssl/certs/ca-certificates.crt if SSL related configurations are unchanged. Touching /etc/ssl/certs causes the service disruption as you see above.

1. is still valid though.

The reason behind 2. seems as follows.

(Pdb) l
 85 import pdb; pdb.set_trace()
 86 if ca_cert:
 87 cert_file = ('/usr/local/share/ca-certificates/'
 88 'keystone_juju_ca_cert.crt')
 89 old_cert = retrieve_ca_cert(cert_file)
 90 -> if old_cert and old_cert == ca_cert:
 91 log("CA cert is the same as installed version", level=INFO)
 92 else:
 93 log("Installing new CA cert", level=INFO)
 94 with open(cert_file, 'wb') as crt:
 95 crt.write(ca_cert)
(Pdb) old_cert
'-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n'
(Pdb) ca_cert
b'-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----\n'

^^^ certs are identical, but the diff if plain text vs binary.

Ok, according to Alex Kavanagh, it's a known issue in charm-helpers.
https://github.com/juju/charm-helpers/issues/86

I think this bug is responsible for the issue: https://github.com/juju/charm-helpers/issues/86

The charm-helpers change is near trivial as suggested by Tilman:
https://github.com/juju/charm-helpers/pull/146

I will also add the other charms potentially affected. They will all require at minimum a charm-helpers sync.

We should look at keystone's handling of certificates as well. Although, the keystone charm is currently python 2 it is meant to go to python 3 soon.

Fix proposed to branch: master
Review: https://review.openstack.org/559814

@Nobutu Can you please test cs:~thedac/neutron-api-4 and see if that fixes things or if we have other problems lurking?

Reviewed: https://review.openstack.org/559814
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=522951870272f6b9036b2549f189851de11b2fb4
Submitter: Zuul
Branch: master

commit 522951870272f6b9036b2549f189851de11b2fb4
Author: David Ames <email address hidden>
Date: Mon Apr 9 11:43:24 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Change-Id: Icaf99f6a069c152f901210c38b26e71622560693
    Closes-Bug: #1762431

With ~openstack-charmers-next/neutron-api-337, deployment succeeds and the CA is considered the same as existing one properly.
https://api.jujucharms.com/charmstore/v5/~openstack-charmers-next/neutron-api-337/archive/repo-info

# grep 'CA cert' /var/log/juju/unit-neutron-api-0.log
2018-04-11 05:53:14 INFO juju-log Installing new CA cert
...
2018-04-11 13:10:42 INFO juju-log CA cert is the same as installed version
2018-04-11 13:16:18 INFO juju-log CA cert is the same as installed version
2018-04-11 13:21:10 INFO juju-log CA cert is the same as installed version
2018-04-11 13:25:46 INFO juju-log CA cert is the same as installed version
2018-04-11 13:30:16 INFO juju-log CA cert is the same as installed version
2018-04-11 13:36:07 INFO juju-log CA cert is the same as installed version

Fix proposed to branch: stable/18.02
Review: https://review.openstack.org/560421

Reviewed: https://review.openstack.org/560655
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=92f5248a072505315546a9d28f0c0739adb03e00
Submitter: Zuul
Branch: master

commit 92f5248a072505315546a9d28f0c0739adb03e00
Author: David Ames <email address hidden>
Date: Wed Apr 11 14:19:29 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Closes-Bug: #1762431

    Change-Id: Ic226149cc124ac5b84ab30d95a590f08489c67f2

Reviewed: https://review.openstack.org/560657
Committed: https://git.openstack.org/cgit/openstack/charm-glance/commit/?id=e581cbafc95670653cc8102edd2bf38a0d9b4822
Submitter: Zuul
Branch: master

commit e581cbafc95670653cc8102edd2bf38a0d9b4822
Author: David Ames <email address hidden>
Date: Wed Apr 11 14:22:15 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Closes-Bug: #1762431

    Change-Id: I65f57573f0b24094b5db8b6854e26146e1b78455

Reviewed: https://review.openstack.org/560660
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=794b8ed0bb9f908ca7c12362cdcba6c3b24f5dc8
Submitter: Zuul
Branch: master

commit 794b8ed0bb9f908ca7c12362cdcba6c3b24f5dc8
Author: David Ames <email address hidden>
Date: Wed Apr 11 14:26:49 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Closes-Bug: #1762431

    Change-Id: Ie2348a83671b9636bd94227e903b1a50bff7aecc

Reviewed: https://review.openstack.org/560656
Committed: https://git.openstack.org/cgit/openstack/charm-cinder/commit/?id=af394fc9e5560bc91201ba1bd0fbb1a74c7d214b
Submitter: Zuul
Branch: master

commit af394fc9e5560bc91201ba1bd0fbb1a74c7d214b
Author: David Ames <email address hidden>
Date: Wed Apr 11 14:21:04 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Change-Id: Ie591ab8c958b59ef190a2f255eee8db0338d5ee3
    Closes-Bug: #1762431

Reviewed: https://review.openstack.org/560661
Committed: https://git.openstack.org/cgit/openstack/charm-openstack-dashboard/commit/?id=0707590487376b8c45d3a9f056b427081f2b138a
Submitter: Zuul
Branch: master

commit 0707590487376b8c45d3a9f056b427081f2b138a
Author: David Ames <email address hidden>
Date: Wed Apr 11 14:28:26 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Change-Id: I3b233a79689858db0962bafe71eba0a0667c2bf0
    Closes-Bug: #1762431

Reviewed: https://review.openstack.org/560659
Committed: https://git.openstack.org/cgit/openstack/charm-heat/commit/?id=5f8d3317e6dd9b67861bab10258f0290281339fc
Submitter: Zuul
Branch: master

commit 5f8d3317e6dd9b67861bab10258f0290281339fc
Author: David Ames <email address hidden>
Date: Wed Apr 11 14:23:17 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Change-Id: I1271ff7dc7fd2ebc0327801cd97d62971ff82e94
    Closes-Bug: #1762431

Reviewed: https://review.openstack.org/560421
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-api/commit/?id=6620eeb340a66bec842813add8e31ce9a33527d7
Submitter: Zuul
Branch: stable/18.02

commit 6620eeb340a66bec842813add8e31ce9a33527d7
Author: David Ames <email address hidden>
Date: Mon Apr 9 11:43:24 2018 -0700

    Charm-helpers sync to fix CA cert comparison

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Change-Id: Icaf99f6a069c152f901210c38b26e71622560693
    Closes-Bug: #1762431
    (cherry picked from commit 522951870272f6b9036b2549f189851de11b2fb4)

This is field-critical for production live bootstack customers. Valid L2 issues on heat stacks as heat is constantly trying to manage stacks and will hit this bug with worse consequences than other apis that are user or program retry-able.

Fix proposed to branch: stable/18.02
Review: https://review.openstack.org/566734

Fix proposed to branch: stable/18.02
Review: https://review.openstack.org/566763

Reviewed: https://review.openstack.org/566734
Committed: https://git.openstack.org/cgit/openstack/charm-heat/commit/?id=c3fcab00c93f9f180b0859a590e9d73f82bf66e0
Submitter: Zuul
Branch: stable/18.02

commit c3fcab00c93f9f180b0859a590e9d73f82bf66e0
Author: David Ames <email address hidden>
Date: Mon May 7 17:42:20 2018 -0300

    Read in ca certificate as binary for PY3

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services. The writing of the file was changed to bytes
    in https://github.com/juju/charm-helpers/pull/58/.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Change-Id: Iedcf0b40004c387384a071c5ad4116e6eb33fd90
    Closes-Bug: #1762431

Reviewed: https://review.openstack.org/566763
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=9053e73d209387f6f90603694246df5d6d3dee76
Submitter: Zuul
Branch: stable/18.02

commit 9053e73d209387f6f90603694246df5d6d3dee76
Author: David Ames <email address hidden>
Date: Mon May 7 20:59:46 2018 -0300

    Read in ca certificate as binary for PY3

    The comparison of bytes vs string of the CA certificate produces a
    false negative. This leads to rewriting certificates and affecting
    connectivity to services. The writing of the file was changed to bytes
    in https://github.com/juju/charm-helpers/pull/58/.

    Read in the certificate as bytes as well for a bytes vs bytes
    comparison.

    Change-Id: I64eb2b90776bb1e37de009208f016f176bd6a82a
    Closes-Bug: #1762431

What is the plan for releasing this? Some of the charms are targetted for 18.05 and some are for 18.02.