Merge ~xavpaice/bootstack-specs:ssl-nagios-checks into bootstack-specs:master

[Xav Paice]

Unmerged commits

72edf30... by Xav Paice on 2018-12-17

ssl-nagios-checks

diff --git a/backlog/ssl-nagios-checks.rst b/backlog/ssl-nagios-checks.rst
new file mode 100644
index 0000000..dc768df
--- /dev/null
+++ b/backlog/ssl-nagios-checks.rst
@@ -0,0 +1,186 @@
+..
+  Copyright 2018 Canonical
+
+  This work is licensed under a Creative Commons Attribution 3.0
+  Unported License.
+  http://creativecommons.org/licenses/by/3.0/legalcode
+
+============================================
+Nagios monitoring for SSL certificate expiry
+============================================
+
+SSL certificates, either self-signed, corporate CA, or signed by a public
+authority, are used by various charms to provide https encryption for API
+endpoints.  These certs are added to the charm configuration, along with a CA
+certificate if the cert is signed by a private authority.  At, preferably
+before, the expiry date of either the SSL certificate itself, or the CA cert,
+we need to replace those certificates.
+
+Problem Description
+===================
+
+Currently there is no monitoring or alerting to warn operators when the
+certificates provided are nearing expiry.  This leaves an opportunity for
+certificates to expire without notice, causing API services to become
+unavailable on expiry, and no useful alerting message to explain why.
+
+Proposed Change
+===============
+
+The goal is to add an nrpe monitor that will check the expiry date of OpenStack
+API certificates and alert when the expiry is a set number of days away or
+less.
+
+First stage of change
+---------------------
+
+Currently the Foundation Cloud build includes the openstack-service-checks
+charm, which monitors a small number of items in Nova and Neutron.  The
+proposed change is to extend that charm to Loop through the Keystone catalog as
+provided by the output of `openstack endpoint list`, and add a check_http check
+for each entry.  The exact url should be determined by the endpoint provided,
+alongside a translation dictionary so that the 'healthcheck' url can be added
+if one exists for that service.  If the url is https, then add -C 30,14 to the
+check_http check so that we are warned at 30 days, and critical at 14 days till
+certificate expiry.
+
+The format of the new Nagios service provided for each service catalog entry
+would be::
+
+  nagios_context-unitname-$service-$type
+
+E.g. to check the keystone public URL::
+
+  bootstack-customer-os-service-checks-keystone-public
+
+Limitations of the proposed change:
+
+- Only certificates for OpenStack API endpoints are checked, if they're in the
+  Keystone catalog.  That means non-OpenStack services such as Kubernetes, or
+  other Apache services, are not monitored by this additional feature.
+- Only the endpoint URL is checked, for a cluster of multiple units behind the
+  hacluster charm we will only check one unit per nrpe check run.  This may
+  miss a unit should a certificate update fail at some stage.
+
+Second stage of the change
+--------------------------
+
+Create a Nagios plugin to take a certificate file and check it's expiry date.
+
+Add the new plugin to charmhelpers, similar to how check_haproxy.sh is added to
+./charm-helpers/charmhelpers/contrib/openstack/files/check_haproxy.sh.
+
+Add a method to charmhelpers.contrib.ssl which takes arguments for a
+certificate file path, plus the actual certificate, and stores the cert where
+asked.  If this method is called plus there's an nrpe relation, the method
+should also trigger a call to add an nrpe check for certificate expiry.
+
+Add a new method to charmhelpers.contrib.nrpe which adds checks for TLS
+certificates defined by ssl_cert and ssl_ca.
+
+For each non-reactive charm that uses charmhelpers, if ssl_cert and/or ssl_ca
+is set, plus there's an nrpe relation, use the method added above to add nrpe
+checks for certificate expiry.  It may be preferable to move the code for
+storing the certificate out of the individual charms.
+
+For reactive charms, update the charm-layer-openstack-api code to run the new
+charmhelpers methods to store ssl_cert and/or ssl_ca.
+
+Charm Relation changes
+----------------------
+
+The number of nrpe checks exported by openstack-service-checks will increase.
+No changes to the relation itself are required.
+
+Charm config changes
+--------------------
+
+New config options for openstack-service-checks:
+
+* Option for warning and critical number of days for certificate expiry
+* Option to disable endpoint types - public, internal and admin, in case the
+  network access to those endpoints is not available from the
+  openstack-service-checks unit.  The default should be to enable all, this
+  option is really just to allow existing clouds with limited networks to
+  function until such time as network access is provided.
+
+
+Charm upgrade risks
+-------------------
+
+None.
+
+
+Alternatives
+------------
+
+- Add a specific check to each openstack charm, to check the certificate
+  validity by URL or by file.  This will mean that the checks are available
+  without needing openstack-service-checks at all.  However, this will also
+  mean all charms need updating, and in the case of checking by URL we will
+  only see a result from the host which receives the request via haproxy.
+- Provide URLs to nagios to check ssl expiry dates.  This requires network
+  access from the Nagios host to all the URLs which we cannot guarantee will be
+  the case at every site.
+- Create a subordinate charm to add an nrpe check to test certificate validity
+  on each unit of an application.  This would allow fast deployment across
+  multiple applications, but at the cost of an extra Juju agent per unit.
+- Extend the NRPE charm to provide a new plugin to check a certificate file or
+  base64 string for expiry date.  Add logic to each charm that has the
+  nrpe-external-master interface to add nrpe checks for the cert if it's
+  configured.  Again, this means changing each charm in turn.  If certs are
+  provided by Easyrsa or some other mechanism no checks would be provided.
+
+Implementation
+==============
+
+Assignee(s)
+-----------
+
+Primary assignee:
+  xavpaice
+
+
+Gerrit Topic
+------------
+
+N/A, this is not an OpenStack project and does not use Gerrit.
+
+Work Items
+----------
+
+1. Evaluate the healthcheck urls available for each project, and if they're
+   different from release to release (Mitaka to Rocky).
+2. Add logic to the openstack-service-checks charm to add a check for each
+   endpoint url provided by Keystone.  If there's a healthcheck URL, use it,
+   else just check for a valid response from the base url.
+3. Add unit testing plus functional testing to the openstack-service-checks
+   charm (currently entirely missing).
+
+Repositories
+------------
+
+No new repositories, we will perform the work entirely within
+https://launchpad.net/charm-openstack-service-checks.
+
+Documentation
+-------------
+
+Documentation for the charm is located in README.md.  The new functionality
+will need to be described in that document.
+
+Security
+--------
+
+This change does not alter the security profile of the charm.
+
+Testing
+-------
+
+The charm currently has no unit testing or functional testing at all.  This gap
+should be addressed while working on this change.
+
+Dependencies
+============
+
+None