Cannot put SSLCertificateChainFile in apache config to support SSL intermediate certificate

Apache supports grouping chain and certificate file into SSLCertificateFile. It's not elegant separation of chains and certificates, but it's perfectly fine to do it.

@ante,

Do you mean the part below?
http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslcertificatefile
"The files may also include intermediate CA certificates, sorted from leaf to root. This is supported with version 2.4.8 and later, and obsoletes SSLCertificateChainFile."

If I understand it correctly, trusty has 2.4.7 so we still need "SSLCertificateChainFile" for trusty (I didn't check if ubuntu's apache 2.4.7 is specially patched though).

I am doing it with 2.4.7 in Trusty and with whatever is in Precise. I took it for granted as a normal thing long time ago.

Most OpenStack charms, including openstack-dashboard that this was originally filed against, are affected by this issue because they use the apache2 template from charm-helpers (charmhelpers/contrib/openstack/templates/openstack_https_frontend). Targeting the bug to charm-helpers but other charms would then need to sync charmhelpers and re-release to fix the issue.

Ante's comment is incorrect, this feature was introduced in Apache2 in 2.4.8, as documented here:
http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslcertificatechainfile

This has become more of a problem in the last few years, as all public CAs started issuing certificates with chained roots instead of directly from a root certificate. Only a few years ago it was still common to get SSLs issue direct from the root where these chains weren't necessary which is why it is more of a problem now.

Trusty ships with 2.4.7 and I have confirmed both with the charm setup and with a manual setup that it does not send the certificate chain even if all the chain certificates are in the system SSL library (/etc/ssl) and even if all the chain certificates are in the SSLCertificateFile. The only way to make it work on trusty is to setup SSLCertificateChainFile. If you are testing this yourself, note that Google Chrome ships a copy of most intermediate certificates and generally will not complain but most other tools including firefox, wget, python https libraries, etc all require the full trust chain to be sent in order to validate the certificate.

On Xenial, it works fine if you simply chain all certificates into the single certificate file as described by Ante.

For this reason I am proposing a patch which sets SSLCertificateChainFile to the same certificate file as SSLCertificateFile. Though the ChainFile is not required to have the actual SSL Certificate in it, it does not seem to hurt or impact it in any way and this allows the same charm configuration to work on both Trusty and Xenial. The certificate file SHOULD be constructed from leaf to root in that order (simply concatenated into the same file) however in practice it seems to work in any order.

We could have instead used ssl_ca setting and done it separately to ssl_cert, but there are two problems with that
 (a) It would require extra charm code in all of the charms to support passing in the CA certificate, rendering/syncing out the /etc/apache2/ssl/ca_* files on all nodes, etc. This logic is already quite complicated.
 (b) The ssl_ca setting is currently used in some charms (e.g. keystone) to load this CA into the system SSL store in /etc/ssl on all related charms which is probably OK but may not actually be desired or expected. i.e. this lets an otherwise non-trusted and likely self-signed CA become trusted if you are using a private CA, but is overkill for a simple certificate chain.

Thus the simple approach to configure SSLCertificateChainFile as the same file is a single one-line patch, requires no extra logic, is equally compatible on Trusty and Xenial and has the least possible effect on the general SSL configuration of the system.

It should then fix all charms using the charm-helpers code once synced, a non-e...

@Trent my configuration of Apache SSL on Trusty:

        SSLEngine on
        SSLCertificateFile /etc/ssl/certs/my.bundle.crt
        SSLCertificateKeyFile /etc/ssl/private/my.key
        SSLCompression off
        SSLProtocol All -SSLv2 -SSLv3
        SSLCipherSuite AES128+EECDH:AES128+EDH

Content of /etc/ssl/certs/my.bundle.crt:

-----BEGIN CERTIFICATE-----
MIIFnzCCBIegAwIBAgIDAYQyMA0GCSqGSIb3DQEBCwUAMEcxCzAJBgNVBAYTAlVT
MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMSAwHgYDVQQDExdSYXBpZFNTTCBTSEEy
NTYgQ0EgLSBHMzAeFw0xNTAxMDcxOTU0MjlaFw0xNzAyMDgyMjU2NTVaMIGQMRMw
EQYDVQQLEwpHVDQxODI4MzEyMTEwLwYDVQQLEyhTZWUgd3d3LnJhcGlkc3NsLmNv
bS9yZXNvdXJjZXMvY3BzIChjKTE1MS8wLQYDVQQLEyZEb21haW4gQ29udHJvbCBW
YWxpZGF0ZWQgLSBSYXBpZFNTTChSKTEVMBMGA1UEAxMMbWFpbC5pbml0LmhyMIIC
IjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEArNXqEOh60UPlgCC/LvM3Hhje
WlOVyOs2qE1QWSzhhLh2CRBC9QWToFoFkLwMr9hHLZPZYah5Rtwwzz1vfinUo1NP
wul2MR6eKUsYxuH2/ktyR0i460N4xxBmbGmkK7Jsp+aevXFJtOAOz/j7cG4OBK8C
arbLPwqP/EQdekYmf1SbTdLwNwuLsCBVZO7nXrLf53/4++I6gZB4X/aNvRn+g29G
65H4AHhLkLdLK3X7P8tGWu6+1V2Vx9Yb9ixfFQDzMY33a2JXhrbctYNHSsS+GNAD
qUv6hv/xTvH3dFdX6tYwFFJ3B4baUh4+sW3ml2HHvUOgCXHw+4shnM/t1JThHSAG
C49LKb8DFqP5SsxSzdXl96/B0EPVQAt8lhD+ORuQzO5fGHjXs1zdCpuUL2z20pEF
GDB0XIqyRsmAjqKiuddPaNTyZ22izFoDDqY3WH9CB4lsAMuELZ6Dcijh5uwIsAwo
zzglkOBsX5jW8MdRRrvPQqGM0G5dyQ+EEd2I1golshGx7n6ZWe172Mq+YD7H7BF/
jNBUyVEtayKaVklpI5LI0PkG1dfZiYd45E8jqronGDi6EcHdns7H39noXBkKDV1R
f2UhiB7Bq3Xy80bHlgQ236KizPFDmj0DrNEpjeWCNYp42jLPOvx9T/PJtoajOEcy
hb0ChrqNM1SpHxMkM/8CAwEAAaOCAUgwggFEMB8GA1UdIwQYMBaAFMOc8/zTRgg0
u85Gf6B8W/PiCMtZMFcGCCsGAQUFBwEBBEswSTAfBggrBgEFBQcwAYYTaHR0cDov
L2d2LnN5bWNkLmNvbTAmBggrBgEFBQcwAoYaaHR0cDovL2d2LnN5bWNiLmNvbS9n
di5jcnQwDgYDVR0PAQH/BAQDAgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEF
BQcDAjAXBgNVHREEEDAOggxtYWlsLmluaXQuaHIwKwYDVR0fBCQwIjAgoB6gHIYa
aHR0cDovL2d2LnN5bWNiLmNvbS9ndi5jcmwwDAYDVR0TAQH/BAIwADBFBgNVHSAE
PjA8MDoGCmCGSAGG+EUBBzYwLDAqBggrBgEFBQcCARYeaHR0cHM6Ly93d3cucmFw
aWRzc2wuY29tL2xlZ2FsMA0GCSqGSIb3DQEBCwUAA4IBAQBQaECZ8YO9dYSomWSQ
4N+KbcrAYWCHo/4KZnjBPL34r04coLRiye2dueYJdP1ciN1nuPNrN8Z7RTyvP7Hr
Vspvbeq1opUcUW7upkb71fw/pD1/YDBo466D/T4jak5allui2/nGHSyaiMFSXgHd
d/QmbyUWJfOBFm1qbNVbyI6OaCj03piWyH1yMsSpoRWgzfBTmFk1wqgNohfjNjID
mQZdq7W5zOZwk5mikNoImZKfY5Pb83iOz0NiryXnDWXURE4xH667nHkHIkx1LfCX
Dcf+l6ZZsQeDFMmWg5/zQWrWHs7QOF2AIdpx/KNrDr3+3gEnzNNie+Ou5RRN9Jvl
ffnZ
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEJTCCAw2gAwIBAgIDAjp3MA0GCSqGSIb3DQEBCwUAMEIxCzAJBgNVBAYTAlVT
MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i
YWwgQ0EwHhcNMTQwODI5MjEzOTMyWhcNMjIwNTIwMjEzOTMyWjBHMQswCQYDVQQG
EwJVUzEWMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjEgMB4GA1UEAxMXUmFwaWRTU0wg
U0hBMjU2IENBIC0gRzMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCv
VJvZWF0eLFbG1eh/9H0WA//Qi1rkjqfdVC7UBMBdmJyNkA+8EGVf2prWRHzAn7Xp
SowLBkMEu/SW4ib2YQGRZjEiwzQ0Xz8/kS9EX9zHFLYDn4ZLDqP/oIACg8PTH2lS
1p1kD8mD5xvEcKyU58Okaiy9uJ5p2L4KjxZjWmhxgHsw3hUEv8zTvz5IBVV6s9cQ
DAP8m/0Ip4yM26eO8R5j3LMBL3+vV8M8SKeDaCGnL+enP/C1DPz1hNFTvA5yT2AM
QriYrRmIV9cE7Ie/fodOoyH5U/02mEiN1vi7SPIpyGTRzFRIU4uvt2UevykzKdkp
YEj4/5G8V1jlNS67abZZAgMBAAGjggEdMIIBGTAfBgNVHSMEGDAWgBTAephojYn7
qwVkDBF9qn1luMrMTjAdBgNVHQ4EFgQUw5zz/NNGCDS7zkZ/oHxb8+IIy1kwEgYD
VR0TAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAQYw...

Fix proposed to branch: master
Review: https://review.openstack.org/393722

Change abandoned by Edward Hope-Morley (<email address hidden>) on branch: master
Review: https://review.openstack.org/393722
Reason: this is now void since apparently somebody did a c-h sync since.

Change abandoned by Billy Olsen (<email address hidden>) on branch: master
Review: https://review.openstack.org/393005

Marking as Fix Committed as all charms have had a resync since the landing of the charm-helpers changes.

I'm adding the charm-guide project to this bug to track the need to document that Intermediate CAs must be part of the ssl_cert bundle rather than the ssl_ca bundle. This is counter-intuitive and should be called out here:

https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-certificate-management.html

Or perhaps added to each charm's ssl_cert documentation in config.yaml that Intermediate CAs must be part of the certificate bundle.

As a user story, the issue we have run into is the following:

We have a certificate chain with Root CA -> Policy CA -> Intermediate CA -> Wildcard Server Cert.

The endpoint browsers have Root CA and Policy CA within their trust, but the Intermediate CA is not in the local browser's cache. Visiting the openstack-dashboard URL with all of the CA certificates bundled properly in ssl_ca does not provide the intermediate CA to the browser to trust the middle-man.

Resolution seems to be to add the Intermediate CA after the Wildcard Server Cert in the ssl_cert bundle.

It's not clear from the documentation that ssl_cert can be multiple certificates/or an entire chain, nor that the ssl_ca is not provided as the chain served from Apache.

Reviewed: https://review.opendev.org/c/openstack/charm-guide/+/856360
Committed: https://opendev.org/openstack/charm-guide/commit/0b901f6935526ac12ed295c1c6339c98229b30b1
Submitter: "Zuul (22348)"
Branch: master

commit 0b901f6935526ac12ed295c1c6339c98229b30b1
Author: Tianqi Xiao <email address hidden>
Date: Wed Sep 7 23:21:20 2022 +0000

    Add guide for configuring SSL certificates via charm options

    Many OpenStack charms has config options which allows user to
    configure charm certificates directly (without using Vault).
    This guide describes the steps to accomplish it.

    Closes-Bug: #1484489
    Change-Id: I26cb0dbded691e9cdaaec6f56506576245d0bca6