Merge ~whereisrysmind/charm-kubernetes-service-checks:master into charm-kubernetes-service-checks:master
- Git
- lp:~whereisrysmind/charm-kubernetes-service-checks
- master
- Merge into master
Status: | Superseded |
---|---|
Proposed branch: | ~whereisrysmind/charm-kubernetes-service-checks:master |
Merge into: | charm-kubernetes-service-checks:master |
Diff against target: |
1917 lines (+1742/-1) 28 files modified
.gitignore (+25/-0) .gitmodules (+3/-1) Makefile (+78/-0) README.md (+96/-0) config.yaml (+38/-0) files/plugins/check_kubernetes_api.py (+118/-0) hooks/install (+1/-0) lib/.empty (+0/-0) lib/charmhelpers (+1/-0) lib/lib_kubernetes_service_checks.py (+159/-0) lib/ops (+1/-0) metadata.yaml (+26/-0) mod/.empty (+0/-0) mod/charm-helpers (+1/-0) src/charm.py (+190/-0) src/setuppath.py (+4/-0) tests/functional/requirements.txt (+1/-0) tests/functional/tests/bundles/bionic.yaml (+90/-0) tests/functional/tests/bundles/focal.yaml (+90/-0) tests/functional/tests/bundles/xenial.yaml (+90/-0) tests/functional/tests/kubernetes_service_checks.py (+162/-0) tests/functional/tests/tests.yaml (+33/-0) tests/unit/requirements.txt (+5/-0) tests/unit/setuppath.py (+6/-0) tests/unit/test_charm.py (+243/-0) tests/unit/test_lib_ksc.py (+164/-0) tests/unit/test_plugins.py (+65/-0) tox.ini (+52/-0) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Ryan Farrell | Pending | ||
Joe Guo | Pending | ||
Adam Dyess | Pending | ||
Paul Goins | Pending | ||
Giuseppe Petralia | Pending | ||
Review via email: mp+388351@code.launchpad.net |
This proposal supersedes a proposal from 2020-07-22.
This proposal has been superseded by a proposal from 2020-07-30.
Commit message
Initial charm for Kubernetes Service Checks
Currently 2 NRPE checks:
- check_http for cert expirations
- check_k8s_
3 Relations required:
- nrpe:nrpe-
- kubernetes-
- kubernetes-
Charm will block if the relations are not configured or if required data
is missing.
+ Unit tests (passing)
+ Functional tests (passing)
+ lint & black (passing)
Description of the change
Initial charm for Kubernetes Service Checks
Giuseppe Petralia (peppepetra) wrote : Posted in a previous version of this proposal | # |
Ryan Farrell (whereisrysmind) wrote : Posted in a previous version of this proposal | # |
Giuseppe,
I addressed your comment , I agree it needed more consistent handling of the configured flag, and thusly for when to perform the configuration. Unit tests and functional tests are still passing.
Giuseppe Petralia (peppepetra) wrote : Posted in a previous version of this proposal | # |
All tests passes. `make test` is blocked by charm proof that is testing things that are not valid for operator charm. I would suggest adding @-charm proof to ignore any error.
I am +1 to merge it.
Thanks.
Ryan Farrell (whereisrysmind) wrote : Posted in a previous version of this proposal | # |
All comments have been address. Ready for review.
Paul Goins (vultaire) wrote : Posted in a previous version of this proposal | # |
No major concerns. A few typos which should be fixed though (since copy/pasting would give errors).
There's also a suggested rework regarding the zaza bundles; this methodology has been used in a few other charms to reduce duplication. I think it would be a really good fit here, too.
Unmerged commits
- 3100e6a... by Ryan Farrell
-
Ignoring make errors on 'charm proof'
- ac15205... by Ryan Farrell
-
Moved SSL coniguration block
- 5e57fec... by Ryan Farrell
-
Edited when ssl configuration occurs
- 003175a... by Ryan Farrell
-
Fixed charm configured flag handling
- 51c4121... by Ryan Farrell
-
Set required args in nrpe plugin
- 6f49451... by Ryan Farrell
-
Blackened code
- 05632f6... by Ryan Farrell
-
Replaced eval with json.loads
- 184e030... by Ryan Farrell
-
Added focal functional test bundle
- 296fbef... by Ryan Farrell
-
Some cleanup
- Merge some docstrings with neighboring comments
- Added Black section to Makefile
- Removed repeated set status calls in the charm - a17ae30... by Ryan Farrell
-
Fixed issues with SSL Host Key verification
- Added step to base64 decode of the SSL cert
- Changed plugin default to check SSL host key
- Updated unit tests
- Fixed fixed functional tests
Preview Diff
1 | diff --git a/.gitignore b/.gitignore |
2 | new file mode 100644 |
3 | index 0000000..f04567f |
4 | --- /dev/null |
5 | +++ b/.gitignore |
6 | @@ -0,0 +1,25 @@ |
7 | +# Byte-compiled / optimized / DLL files |
8 | +__pycache__/ |
9 | +*.py[cod] |
10 | +*$py.class |
11 | + |
12 | +# Log files |
13 | +*.log |
14 | + |
15 | +.tox/ |
16 | +.coverage |
17 | + |
18 | +# vi |
19 | +.*.swp |
20 | + |
21 | +# pycharm |
22 | +.idea/ |
23 | + |
24 | +# version data |
25 | +repo-info |
26 | + |
27 | +# reports |
28 | +report/* |
29 | + |
30 | +# coverage |
31 | +htmlcov/* |
32 | diff --git a/.gitmodules b/.gitmodules |
33 | index 0b10c26..2970f9f 100644 |
34 | --- a/.gitmodules |
35 | +++ b/.gitmodules |
36 | @@ -1,4 +1,6 @@ |
37 | [submodule "mod/operator"] |
38 | path = mod/operator |
39 | url = https://github.com/canonical/operator.git |
40 | - branch = 0.7.0 |
41 | +[submodule "mod/charm-helpers"] |
42 | + path = mod/charm-helpers |
43 | + url = https://github.com/juju/charm-helpers.git |
44 | diff --git a/Makefile b/Makefile |
45 | new file mode 100644 |
46 | index 0000000..d1435c4 |
47 | --- /dev/null |
48 | +++ b/Makefile |
49 | @@ -0,0 +1,78 @@ |
50 | +PYTHON := /usr/bin/python3 |
51 | + |
52 | +ifndef CHARM_BUILD_DIR |
53 | + CHARM_BUILD_DIR=/tmp/charm-builds |
54 | +endif |
55 | + |
56 | +PROJECTPATH=$(dir $(realpath $(MAKEFILE_LIST))) |
57 | +METADATA_FILE="metadata.yaml" |
58 | +CHARM_NAME=$(shell cat ${PROJECTPATH}/${METADATA_FILE} | grep -E '^name:' | awk '{print $$2}') |
59 | + |
60 | +help: |
61 | + @echo "This project supports the following targets" |
62 | + @echo "" |
63 | + @echo " make help - show this text" |
64 | + @echo " make clean - remove unneeded files" |
65 | + @echo " make submodules - make sure that the submodules are up-to-date" |
66 | + @echo " make build - build the charm" |
67 | + @echo " make release - run clean, submodules and build targets" |
68 | + @echo " make lint - run flake8 and black" |
69 | + @echo " make proof - run charm proof" |
70 | + @echo " make unittests - run the tests defined in the unittest subdirectory" |
71 | + @echo " make functional - run the tests defined in the functional subdirectory" |
72 | + @echo " make test - run lint, proof, unittests and functional targets" |
73 | + @echo "" |
74 | + |
75 | +clean: |
76 | + @echo "Cleaning files" |
77 | + @if [ -d .tox ] ; then rm -r .tox ; fi |
78 | + @if [ -d .pytest_cache ] ; then rm -r .pytest_cache ; fi |
79 | + @if [ -d .idea ] ; then rm -r .idea ; fi |
80 | + @if [ -d .coverage ] ; then rm -r .coverage ; fi |
81 | + @if [ -d report ] ; then rm -r report ; fi |
82 | + @find . -iname __pycache__ -exec rm -r {} + |
83 | + @find . -type f -name "*.py[cod]" -delete |
84 | + @find . -type f -name "*$py.class" -delete |
85 | + @find . -type f -name "*.log" -delete |
86 | + @find . -type f -name "*.swp" -delete |
87 | + @find . -type f -name ".unit-state.db" -delete |
88 | + @echo "Cleaning existing build" |
89 | + @rm -rf ${CHARM_BUILD_DIR}/${CHARM_NAME} |
90 | + |
91 | +submodules: |
92 | + @echo "Cloning submodules" |
93 | + @git submodule update --init --recursive |
94 | + |
95 | +build: |
96 | + @echo "Building charm to base directory ${CHARM_BUILD_DIR}/${CHARM_NAME}" |
97 | + @-git describe --tags > ./repo-info |
98 | + @mkdir -p ${CHARM_BUILD_DIR}/${CHARM_NAME} |
99 | + @cp -r ./* ${CHARM_BUILD_DIR}/${CHARM_NAME} |
100 | + @echo "Installing/updating env from requirements.txt" |
101 | + @mkdir -p ${CHARM_BUILD_DIR}/${CHARM_NAME}/env/ |
102 | + @if [ -f requirements.txt ] ; then @pip3 install --target=${CHARM_BUILD_DIR}/${CHARM_NAME}/env -r requirements.txt --upgrade ; fi |
103 | + |
104 | +release: clean submodules build |
105 | + @echo "Charm is built at ${CHARM_BUILD_DIR}/${CHARM_NAME}" |
106 | + |
107 | +lint: |
108 | + @echo "Running lint checks" |
109 | + @tox -e lint |
110 | + |
111 | +proof: |
112 | + @echo "Running charm proof" |
113 | + @-charm proof |
114 | + |
115 | +unittests: |
116 | + @echo "Running unit tests" |
117 | + @tox -e unit |
118 | + |
119 | +functional: build |
120 | + @echo "Executing functional tests in ${CHARM_BUILD_DIR}" |
121 | + @CHARM_BUILD_DIR=${CHARM_BUILD_DIR} tox -e func |
122 | + |
123 | +test: lint proof unittests functional |
124 | + @echo "Tests completed for charm ${CHARM_NAME}." |
125 | + |
126 | +# The targets below don't depend on a file |
127 | +.PHONY: help submodules clean build release lint proof unittests functional test |
128 | \ No newline at end of file |
129 | diff --git a/README.md b/README.md |
130 | new file mode 100644 |
131 | index 0000000..1a50bd9 |
132 | --- /dev/null |
133 | +++ b/README.md |
134 | @@ -0,0 +1,96 @@ |
135 | +# kubernetes-service-checks Charm |
136 | + |
137 | +Overview |
138 | +-------- |
139 | + |
140 | +This charm provides Kubernetes Service checks for Nagios |
141 | + |
142 | +Quickstart |
143 | +---------- |
144 | + |
145 | + juju deploy cs:kubernetes-service-checks |
146 | + juju add-relation kubernetes-service-checks nrpe |
147 | + juju add-relation kubernetes-service-checks:kube-api-endpoint kubernetes-master |
148 | + juju add-relation kuberentes-service-checks:kube-control kuberentes-master |
149 | + |
150 | + |
151 | + |
152 | + |
153 | +### Relations |
154 | + |
155 | +* **kubernetes-master:kube-api-endpoint** - Provides KSC with the kubernetes-api *hostname* and *port* |
156 | + |
157 | +* **kuberentes-master:kube-control** - Provides KSC with a kuberentes-api *client-token* for authentication |
158 | + |
159 | +* **nrpe:nrpe-external-master** - Required for nagios; provides additional plugins |
160 | + |
161 | + |
162 | +**Note:** Future relations with kubernetes-master *may* be changed so that a |
163 | +single relation can provide the K8S api hostname, port, client token and ssl ca |
164 | +cert. |
165 | + |
166 | +### Config Options |
167 | + |
168 | +**trusted_ssl_ca** *(Optional)* Setting this option enables SSL host |
169 | +certificate authentication in the api checks |
170 | + |
171 | + juju config kubernetes-service-checks trusted_ssl_ca="${KUBERNETES_API_CA}" |
172 | + |
173 | + |
174 | +Service Checks |
175 | +-------------- |
176 | +The plugin *check_kubernetes_api.py* ships with this charm and contains an array of checks for the k8s api health. |
177 | + |
178 | +``` |
179 | +check_kubernetes_api.py --help |
180 | +usage: check_kubernetes_api.py [-h] [-H HOST] [-P PORT] [-T CLIENT_TOKEN] |
181 | + [--check health] [-C SSL_CA_PATH] |
182 | + |
183 | +Check Kubernetes API status |
184 | + |
185 | +optional arguments: |
186 | + -h, --help show this help message and exit |
187 | + -H HOST, --host HOST Hostname or IP of the kube-api-server (default: None) |
188 | + -P PORT, --port PORT Port of the kube-api-server (default: 6443) |
189 | + -T CLIENT_TOKEN, --token CLIENT_TOKEN |
190 | + Client access token for authenticate with the |
191 | + Kubernetes API (default: None) |
192 | + --check health which check to run (default: health) |
193 | + -C SSL_CA_PATH, --trusted-ca-cert SSL_CA_PATH |
194 | + String containing path to the trusted CA certificate |
195 | + (default: None) |
196 | + |
197 | +``` |
198 | + |
199 | +**health** - This polls the kubernetes-api */healthz* endpoint. Posting a GET to this URL endpoint is expected to |
200 | +return 200 - 'ok' if the api is healthy, otherwise 500. |
201 | + |
202 | + |
203 | +Other Checks |
204 | +------------ |
205 | + |
206 | +**Certificate Expiration:** The *check_http* plugin is shipped with nrpe, and contains a built in cert expiration check. The warning and crit |
207 | +thesholds are configurable: |
208 | + |
209 | + juju config kubernetes-service-checks tls_warn_days=90 |
210 | + juju config kubernetes-service-checks tls_crit_days=30 |
211 | + |
212 | +Testing |
213 | +------- |
214 | + |
215 | +Juju should be installed and bootstrapped on the system to run functional tests. |
216 | + |
217 | + |
218 | +``` |
219 | + export MODEL_SETTINGS=<semicolon-separated list of "juju model-config" settings> |
220 | + make test |
221 | +``` |
222 | + |
223 | +NOTE: If you are behind a proxy, be sure to export a MODEL_SETTINGS variable as |
224 | +described above. Note that you will need to use the juju-http-proxy, juju-https-proxy, juju-no-proxy |
225 | +and similar settings. |
226 | + |
227 | +Contact |
228 | +------- |
229 | + - Author: **Bootstack Charmers** *<bootstack-charmers@lists.canonical.com>* |
230 | + - Bug Tracker: [lp:charm-kubernetes-service-checks](https://launchpad.net/charm-kubernetes-service-checks) |
231 | diff --git a/config.yaml b/config.yaml |
232 | new file mode 100644 |
233 | index 0000000..8560b6e |
234 | --- /dev/null |
235 | +++ b/config.yaml |
236 | @@ -0,0 +1,38 @@ |
237 | +options: |
238 | + channel: |
239 | + type: string |
240 | + default: 1.18/stable |
241 | + description: | |
242 | + Snap channel to install kubectl from |
243 | + nagios_context: |
244 | + default: "juju" |
245 | + type: string |
246 | + description: | |
247 | + Used by the nrpe subordinate charms. |
248 | + A string that will be prepended to instance name to set the host name |
249 | + in nagios. So for instance the hostname would be something like: |
250 | + juju-myservice-0 |
251 | + If you're running multiple environments with the same services in them |
252 | + this allows you to differentiate between them. |
253 | + nagios_servicegroups: |
254 | + default: "" |
255 | + type: string |
256 | + description: | |
257 | + A comma-separated list of nagios servicegroups. |
258 | + If left empty, the nagios_context will be used as the servicegroup |
259 | + tls_warn_days: |
260 | + type: int |
261 | + default: 60 |
262 | + description: | |
263 | + Number of days left for the TLS certificate to expire before Warning. |
264 | + tls_crit_days: |
265 | + type: int |
266 | + default: 30 |
267 | + description: | |
268 | + Number of days left for the TLS certificate to expire before alerting Critical. |
269 | + # temporary config setting for trusted SSL CA (see LP1886982) |
270 | + trusted_ssl_ca: |
271 | + type: string |
272 | + default: "" |
273 | + description: | |
274 | + base64 encoded SSL ca cert to use for Kubernetes API client connections. |
275 | \ No newline at end of file |
276 | diff --git a/files/plugins/check_kubernetes_api.py b/files/plugins/check_kubernetes_api.py |
277 | new file mode 100755 |
278 | index 0000000..ad91f26 |
279 | --- /dev/null |
280 | +++ b/files/plugins/check_kubernetes_api.py |
281 | @@ -0,0 +1,118 @@ |
282 | +#!/usr/bin/python3 |
283 | +"""NRPE Plugin for checking Kubernetes API.""" |
284 | + |
285 | +import argparse |
286 | +import sys |
287 | + |
288 | +import urllib3 |
289 | + |
290 | +NAGIOS_STATUS_OK = 0 |
291 | +NAGIOS_STATUS_WARNING = 1 |
292 | +NAGIOS_STATUS_CRITICAL = 2 |
293 | +NAGIOS_STATUS_UNKNOWN = 3 |
294 | + |
295 | +NAGIOS_STATUS = { |
296 | + NAGIOS_STATUS_OK: "OK", |
297 | + NAGIOS_STATUS_WARNING: "WARNING", |
298 | + NAGIOS_STATUS_CRITICAL: "CRITICAL", |
299 | + NAGIOS_STATUS_UNKNOWN: "UNKNOWN", |
300 | +} |
301 | + |
302 | + |
303 | +def nagios_exit(status, message): |
304 | + """Return the check status in Nagios preferred format. |
305 | + |
306 | + :param status: Nagios Check status code (in [0, 1, 2, 3]) |
307 | + :param message: Message describing the status |
308 | + :return: sys.exit("{status_string}: {message}") |
309 | + """ |
310 | + assert status in NAGIOS_STATUS, "Invalid Nagios status code" |
311 | + # prefix status name to message |
312 | + output = "{}: {}".format(NAGIOS_STATUS[status], message) |
313 | + print(output) # nagios requires print to stdout, no stderr |
314 | + sys.exit(status) |
315 | + |
316 | + |
317 | +def check_kubernetes_health(k8s_address, client_token, disable_ssl): |
318 | + """Call <kubernetes-api>/healthz endpoint and check return value is 'ok'. |
319 | + |
320 | + :param k8s_address: Address to kube-api-server formatted 'https://<IP>:<PORT>' |
321 | + :param client_token: Token for authenticating with the kube-api |
322 | + :param disable_ssl: Disables SSL Host Key verification |
323 | + """ |
324 | + url = k8s_address + "/healthz" |
325 | + if disable_ssl: |
326 | + # perform check without SSL verification |
327 | + http = urllib3.PoolManager(cert_reqs="CERT_NONE", assert_hostname=False) |
328 | + else: |
329 | + http = urllib3.PoolManager() |
330 | + |
331 | + try: |
332 | + resp = http.request("GET", url, headers={"Authorization": "Bearer {}".format(client_token)}) |
333 | + except urllib3.exceptions.MaxRetryError as e: |
334 | + return NAGIOS_STATUS_CRITICAL, e |
335 | + |
336 | + if resp.status != 200: |
337 | + return NAGIOS_STATUS_CRITICAL, "Unexpected HTTP Response code ({})".format(resp.status) |
338 | + elif resp.data != b"ok": |
339 | + return NAGIOS_STATUS_WARNING, "Unexpected Kubernetes healthz status '{}'".format(resp.data) |
340 | + return NAGIOS_STATUS_OK, "Kubernetes health 'ok'" |
341 | + |
342 | + |
343 | +if __name__ == "__main__": |
344 | + parser = argparse.ArgumentParser( |
345 | + description="Check Kubernetes API status", formatter_class=argparse.ArgumentDefaultsHelpFormatter, |
346 | + ) |
347 | + |
348 | + parser.add_argument("-H", "--host", dest="host", help="Hostname or IP of the kube-api-server", required=True) |
349 | + |
350 | + parser.add_argument( |
351 | + "-P", "--port", dest="port", type=int, default=6443, help="Port of the kube-api-server", required=True |
352 | + ) |
353 | + |
354 | + parser.add_argument( |
355 | + "-T", "--token", dest="client_token", help="Client access token for authenticate with the Kubernetes API" |
356 | + ) |
357 | + |
358 | + check_choices = ["health"] |
359 | + parser.add_argument( |
360 | + "--check", |
361 | + dest="check", |
362 | + metavar="|".join(check_choices), |
363 | + type=str, |
364 | + choices=check_choices, |
365 | + default=check_choices[0], |
366 | + help="which check to run", |
367 | + ) |
368 | + |
369 | + parser.add_argument( |
370 | + "-d", |
371 | + "--disable-host-key-check", |
372 | + dest="disable_host_key_check", |
373 | + default=False, |
374 | + action="store_true", |
375 | + help="Disables Host SSL Key Authentication", |
376 | + ) |
377 | + args = parser.parse_args() |
378 | + |
379 | + checks = { |
380 | + "health": check_kubernetes_health, |
381 | + } |
382 | + |
383 | + k8s_url = "https://{}:{}".format(args.host, args.port) |
384 | + nagios_exit(*checks[args.check](k8s_url, args.client_token, args.disable_host_key_check)) |
385 | + |
386 | +""" |
387 | +TODO: Future Checks |
388 | + |
389 | +GET /api/v1/componentstatuses HTTP/1.1 |
390 | +Authorization: Bearer $TOKEN |
391 | +Accept: application/json |
392 | +Connection: close |
393 | + |
394 | +GET /api/va/nodes HTTP/1.1 |
395 | +Authorization: Bearer $TOKEN |
396 | +Accept: application/json |
397 | +Connection: close |
398 | + |
399 | +""" |
400 | diff --git a/hooks/install b/hooks/install |
401 | new file mode 120000 |
402 | index 0000000..25b1f68 |
403 | --- /dev/null |
404 | +++ b/hooks/install |
405 | @@ -0,0 +1 @@ |
406 | +../src/charm.py |
407 | \ No newline at end of file |
408 | diff --git a/lib/.empty b/lib/.empty |
409 | new file mode 100644 |
410 | index 0000000..e69de29 |
411 | --- /dev/null |
412 | +++ b/lib/.empty |
413 | diff --git a/lib/charmhelpers b/lib/charmhelpers |
414 | new file mode 120000 |
415 | index 0000000..cedc09e |
416 | --- /dev/null |
417 | +++ b/lib/charmhelpers |
418 | @@ -0,0 +1 @@ |
419 | +../mod/charm-helpers/charmhelpers |
420 | \ No newline at end of file |
421 | diff --git a/lib/lib_kubernetes_service_checks.py b/lib/lib_kubernetes_service_checks.py |
422 | new file mode 100644 |
423 | index 0000000..14db4e6 |
424 | --- /dev/null |
425 | +++ b/lib/lib_kubernetes_service_checks.py |
426 | @@ -0,0 +1,159 @@ |
427 | +"""Kubernetes Service Checks Helper Library.""" |
428 | +import base64 |
429 | +import json |
430 | +import logging |
431 | +import os |
432 | +import subprocess |
433 | + |
434 | +from charmhelpers.contrib.charmsupport.nrpe import NRPE |
435 | +from charmhelpers.core import hookenv, host |
436 | +from charmhelpers.fetch import snap |
437 | + |
438 | +CERT_FILE = "/usr/local/share/ca-certificates/kubernetes-service-checks.crt" |
439 | +NAGIOS_PLUGINS_DIR = "/usr/local/lib/nagios/plugins/" |
440 | + |
441 | + |
442 | +class KSCHelper: |
443 | + """Kubernetes Service Checks Helper Class.""" |
444 | + |
445 | + def __init__(self, config, state): |
446 | + """Initialize the Helper with the charm config and state.""" |
447 | + self.config = config |
448 | + self.state = state |
449 | + |
450 | + @property |
451 | + def kubernetes_api_address(self): |
452 | + """Get kubernetes api hostname.""" |
453 | + return self.state.kube_api_endpoint.get("hostname", None) |
454 | + |
455 | + @property |
456 | + def kubernetes_api_port(self): |
457 | + """Get kubernetes api port.""" |
458 | + return self.state.kube_api_endpoint.get("port", None) |
459 | + |
460 | + @property |
461 | + def kubernetes_client_token(self): |
462 | + """Get kubernetes client token.""" |
463 | + try: |
464 | + data = json.loads(self.state.kube_control.get("creds", "{}")) |
465 | + except json.decoder.JSONDecodeError: |
466 | + data = {} |
467 | + for creds in data.values(): |
468 | + token = creds.get("client_token", None) |
469 | + if token: |
470 | + return token |
471 | + return None |
472 | + |
473 | + @property |
474 | + def use_tls_cert(self): |
475 | + """Check if SSL cert is provided for use.""" |
476 | + return bool(self._ssl_certificate) |
477 | + |
478 | + @property |
479 | + def _ssl_certificate(self): |
480 | + # TODO: Expand this later to take a cert from a relation or from the config. |
481 | + # cert from the relation is to be prioritized |
482 | + ssl_cert = self.config.get("trusted_ssl_ca", None) |
483 | + if ssl_cert: |
484 | + ssl_cert = ssl_cert.strip() |
485 | + return ssl_cert |
486 | + |
487 | + @property |
488 | + def ssl_cert_path(self): |
489 | + """Get cert file path.""" |
490 | + return CERT_FILE |
491 | + |
492 | + @property |
493 | + def plugins_dir(self): |
494 | + """Get nagios plugins directory.""" |
495 | + return NAGIOS_PLUGINS_DIR |
496 | + |
497 | + def restart_nrpe_service(self): |
498 | + """Restart nagios-nrpe-server service.""" |
499 | + host.service_restart("nagios-nrpe-server") |
500 | + |
501 | + def update_tls_certificates(self): |
502 | + """Write the trusted ssl certificate to the CERT_FILE.""" |
503 | + if self._ssl_certificate: |
504 | + cert_content = base64.b64decode(self._ssl_certificate).decode() |
505 | + try: |
506 | + logging.debug("Writing ssl ca cert to {}".format(self.ssl_cert_path)) |
507 | + with open(self.ssl_cert_path, "w") as f: |
508 | + f.write(cert_content) |
509 | + subprocess.call(["/usr/sbin/update-ca-certificates"]) |
510 | + return True |
511 | + except subprocess.CalledProcessError as e: |
512 | + logging.error(e) |
513 | + return False |
514 | + except PermissionError as e: |
515 | + logging.error(e) |
516 | + return False |
517 | + else: |
518 | + logging.error("Trusted SSL Certificate is not defined") |
519 | + return False |
520 | + |
521 | + def configure(self): |
522 | + """Refresh configuration data.""" |
523 | + self.update_plugins() |
524 | + self.render_checks() |
525 | + |
526 | + def update_plugins(self): |
527 | + """Rsync plugins to the plugin directory.""" |
528 | + charm_plugin_dir = os.path.join(hookenv.charm_dir(), "files", "plugins/") |
529 | + host.rsync(charm_plugin_dir, self.plugins_dir, options=["--executability"]) |
530 | + |
531 | + def render_checks(self): |
532 | + """Render nrpe checks.""" |
533 | + nrpe = NRPE() |
534 | + if not os.path.exists(self.plugins_dir): |
535 | + os.makedirs(self.plugins_dir) |
536 | + |
537 | + # register basic api health check |
538 | + check_k8s_plugin = os.path.join(self.plugins_dir, "check_kubernetes_api.py") |
539 | + for check in ["health"]: |
540 | + check_command = "{} -H {} -P {} -T {} --check {}".format( |
541 | + check_k8s_plugin, |
542 | + self.kubernetes_api_address, |
543 | + self.kubernetes_api_port, |
544 | + self.kubernetes_client_token, |
545 | + check, |
546 | + ).strip() |
547 | + if not self.use_tls_cert: |
548 | + check_command += " -d" |
549 | + |
550 | + nrpe.add_check( |
551 | + shortname="k8s_api_{}".format(check), |
552 | + description="Check Kubernetes API ({})".format(check), |
553 | + check_cmd=check_command, |
554 | + ) |
555 | + |
556 | + # register k8s host certificate expiration check |
557 | + check_http_plugin = "/usr/lib/nagios/plugins/check_http" |
558 | + check_command = "{} -I {} -p {} -C {},{}".format( |
559 | + check_http_plugin, |
560 | + self.kubernetes_api_address, |
561 | + self.kubernetes_api_port, |
562 | + self.config.get("tls_warn_days"), |
563 | + self.config.get("tls_crit_days"), |
564 | + ).strip() |
565 | + nrpe.add_check( |
566 | + shortname="k8s_api_cert_expiration", |
567 | + description="Check Kubernetes API ({})".format(check), |
568 | + check_cmd=check_command, |
569 | + ) |
570 | + nrpe.write() |
571 | + |
572 | + def install_kubectl(self): |
573 | + """Attempt to install kubectl. |
574 | + |
575 | + :returns: bool, indicating whether or not successful |
576 | + """ |
577 | + # snap retry is excessive |
578 | + snap.SNAP_NO_LOCK_RETRY_DELAY = 0.5 |
579 | + snap.SNAP_NO_LOCK_RETRY_COUNT = 3 |
580 | + try: |
581 | + channel = self.config.get("channel") |
582 | + snap.snap_install("kubectl", "--classic", "--channel={}".format(channel)) |
583 | + return True |
584 | + except snap.CouldNotAcquireLockException: |
585 | + return False |
586 | diff --git a/lib/ops b/lib/ops |
587 | new file mode 120000 |
588 | index 0000000..d934193 |
589 | --- /dev/null |
590 | +++ b/lib/ops |
591 | @@ -0,0 +1 @@ |
592 | +../mod/operator/ops |
593 | \ No newline at end of file |
594 | diff --git a/metadata.yaml b/metadata.yaml |
595 | new file mode 100644 |
596 | index 0000000..09b26a5 |
597 | --- /dev/null |
598 | +++ b/metadata.yaml |
599 | @@ -0,0 +1,26 @@ |
600 | +name: kubernetes-service-checks |
601 | +summary: Kubernetes Services NRPE Checks |
602 | +maintainers: |
603 | + - Bootstack Charmers <bootstack-charmers@lists.canonical.com> |
604 | +description: | |
605 | + This charm provides NRPE Checks verifying Kubernetes API accessibility |
606 | + and integrates with Nagios for timely alerting. |
607 | +tags: |
608 | + - kubernetes |
609 | + - ops |
610 | + - monitoring |
611 | +series: |
612 | + - focal |
613 | + - bionic |
614 | + - xenial |
615 | +requires: |
616 | + kube-control: |
617 | + interface: kube-control |
618 | + kube-api-endpoint: |
619 | + interface: http |
620 | +provides: |
621 | + nrpe-external-master: |
622 | + interface: nrpe-external-master |
623 | + scope: container |
624 | + optional: true |
625 | +subordinate: false |
626 | diff --git a/mod/.empty b/mod/.empty |
627 | new file mode 100644 |
628 | index 0000000..e69de29 |
629 | --- /dev/null |
630 | +++ b/mod/.empty |
631 | diff --git a/mod/charm-helpers b/mod/charm-helpers |
632 | new file mode 160000 |
633 | index 0000000..4b3602e |
634 | --- /dev/null |
635 | +++ b/mod/charm-helpers |
636 | @@ -0,0 +1 @@ |
637 | +Subproject commit 4b3602e2bdf101bf58cc808264ec0c8092a67cd0 |
638 | diff --git a/src/charm.py b/src/charm.py |
639 | new file mode 100755 |
640 | index 0000000..1047fc4 |
641 | --- /dev/null |
642 | +++ b/src/charm.py |
643 | @@ -0,0 +1,190 @@ |
644 | +#! /usr/bin/env python3 |
645 | +# -*- coding: utf-8 -*- |
646 | +# vim:fenc=utf-8 |
647 | +# Copyright © 2020 Bootstack Charmers bootstack-charmers@lists.canonical.com |
648 | + |
649 | +"""Operator Charm main library.""" |
650 | +# Load modules from lib directory |
651 | +import logging |
652 | + |
653 | +import setuppath # noqa:F401 |
654 | + |
655 | +from lib_kubernetes_service_checks import KSCHelper # noqa:I100 |
656 | +from ops.charm import CharmBase |
657 | +from ops.framework import StoredState |
658 | +from ops.main import main |
659 | +from ops.model import ActiveStatus, BlockedStatus, MaintenanceStatus |
660 | + |
661 | + |
662 | +class KubernetesServiceChecksCharm(CharmBase): |
663 | + """Class representing this Operator charm.""" |
664 | + |
665 | + state = StoredState() |
666 | + |
667 | + def __init__(self, *args): |
668 | + """Initialize charm and configure states and events to observe.""" |
669 | + super().__init__(*args) |
670 | + # -- standard hook observation |
671 | + self.framework.observe(self.on.install, self.on_install) |
672 | + self.framework.observe(self.on.start, self.on_start) |
673 | + self.framework.observe(self.on.config_changed, self.on_config_changed) |
674 | + self.framework.observe( |
675 | + self.on.kube_api_endpoint_relation_changed, self.on_kube_api_endpoint_relation_changed, |
676 | + ) |
677 | + self.framework.observe(self.on.kube_api_endpoint_relation_departed, self.on_kube_api_endpoint_relation_departed) |
678 | + self.framework.observe( |
679 | + self.on.kube_control_relation_changed, self.on_kube_control_relation_changed, |
680 | + ) |
681 | + self.framework.observe(self.on.kube_control_relation_departed, self.on_kube_control_relation_departed) |
682 | + self.framework.observe( |
683 | + self.on.nrpe_external_master_relation_joined, self.on_nrpe_external_master_relation_joined |
684 | + ) |
685 | + self.framework.observe( |
686 | + self.on.nrpe_external_master_relation_departed, self.on_nrpe_external_master_relation_departed |
687 | + ) |
688 | + # -- initialize states -- |
689 | + self.state.set_default( |
690 | + installed=False, |
691 | + configured=False, |
692 | + started=False, |
693 | + kube_control={}, |
694 | + kube_api_endpoint={}, |
695 | + nrpe_configured=False, |
696 | + ) |
697 | + self.helper = KSCHelper(self.model.config, self.state) |
698 | + |
699 | + def on_install(self, event): |
700 | + """Handle install state.""" |
701 | + self.unit.status = MaintenanceStatus("Install complete") |
702 | + logging.info("Install of software complete") |
703 | + self.state.installed = True |
704 | + |
705 | + def on_upgrade_charm(self, event): |
706 | + """Handle upgrade and resource updates.""" |
707 | + self.state.configured = False |
708 | + logging.info("Reinstalling for upgrade-charm hook") |
709 | + self.on_install(event) |
710 | + self.check_charm_status() |
711 | + |
712 | + def check_charm_status(self): |
713 | + """ |
714 | + Check that required data is available from relations. |
715 | + |
716 | + - Check kube-api-endpoint relation and data available |
717 | + - Check kube-control relation and data available |
718 | + - Check nrpe-external-master is configured |
719 | + - Check any required config options |
720 | + - Finally, configure the charms checks and set flags |
721 | + """ |
722 | + if not self.helper.kubernetes_api_address or not self.helper.kubernetes_api_port: |
723 | + logging.warning("kube-api-endpoint relation missing or misconfigured") |
724 | + self.unit.status = BlockedStatus("missing kube-api-endpoint relation") |
725 | + return |
726 | + if not self.helper.kubernetes_client_token: |
727 | + logging.warning("kube-control relation missing or misconfigured") |
728 | + self.unit.status = BlockedStatus("missing kube-control relation") |
729 | + return |
730 | + if not self.state.nrpe_configured: |
731 | + logging.warning("nrpe-external-master relation missing or misconfigured") |
732 | + self.unit.status = BlockedStatus("missing nrpe-external-master relation") |
733 | + return |
734 | + |
735 | + if not self.state.configured: |
736 | + # Check specific required config values |
737 | + # Set up TLS Certificate |
738 | + if self.helper.use_tls_cert: |
739 | + logging.info("Updating tls certificates") |
740 | + if self.helper.update_tls_certificates(): |
741 | + logging.info("TLS Certificates updated successfully") |
742 | + else: |
743 | + logging.error("Failed to update TLS Certificates") |
744 | + self.unit.status = BlockedStatus("update-ca-certificates error. check logs") |
745 | + return |
746 | + else: |
747 | + logging.warning("No trusted_ssl_ca provided, SSL Host Authentication disabled") |
748 | + |
749 | + logging.info("Configuring Kubernetes Service Checks") |
750 | + self.helper.configure() |
751 | + |
752 | + logging.info("Reloading nagios-nrpe-server") |
753 | + self.helper.restart_nrpe_service() |
754 | + self.state.configured = True |
755 | + self.unit.status = ActiveStatus("Unit is ready") |
756 | + |
757 | + def on_config_changed(self, event): |
758 | + """Handle config changed.""" |
759 | + self.state.configured = False |
760 | + if not self.state.installed: |
761 | + logging.warning("Config changed called before install complete, deferring event: {}.".format(event.handle)) |
762 | + self._defer_once(event) |
763 | + return |
764 | + self.check_charm_status() |
765 | + |
766 | + def on_start(self, event): |
767 | + """Handle start state.""" |
768 | + if not self.state.configured: |
769 | + logging.warning("Start called before configuration complete, deferring event: {}".format(event.handle)) |
770 | + event.defer() |
771 | + return |
772 | + self.unit.status = ActiveStatus("Unit is ready") |
773 | + self.state.started = True |
774 | + logging.info("Started") |
775 | + |
776 | + def _defer_once(self, event): |
777 | + """Defer the given event, but only once.""" |
778 | + notice_count = 0 |
779 | + handle = str(event.handle) |
780 | + |
781 | + for event_path, _, _ in self.framework._storage.notices(None): |
782 | + if event_path.startswith(handle.split("[")[0]): |
783 | + notice_count += 1 |
784 | + logging.debug("Found event: {} x {}".format(event_path, notice_count)) |
785 | + |
786 | + if notice_count > 1: |
787 | + logging.debug("Not deferring {} notice count of {}".format(handle, notice_count)) |
788 | + else: |
789 | + logging.debug("Deferring {} notice count of {}".format(handle, notice_count)) |
790 | + event.defer() |
791 | + |
792 | + def on_kube_api_endpoint_relation_changed(self, event): |
793 | + """Handle kube_api_endpoint relation changed.""" |
794 | + self.state.configured = False |
795 | + self.unit.status = MaintenanceStatus("Updating K8S Endpoint") |
796 | + self.state.kube_api_endpoint.update(event.relation.data.get(event.unit, {})) |
797 | + self.check_charm_status() |
798 | + |
799 | + def on_kube_api_endpoint_relation_departed(self, event): |
800 | + """Handle kube-api-endpoint relation departed.""" |
801 | + self.state.configured = False |
802 | + for k in self.state.kube_api_endpoint.keys(): |
803 | + self.state.kube_api_endpoint[k] = "" |
804 | + self.check_charm_status() |
805 | + |
806 | + def on_kube_control_relation_changed(self, event): |
807 | + """Handle kube-control relation changed.""" |
808 | + self.state.configured = False |
809 | + self.unit.status = MaintenanceStatus("Updating K8S Credentials") |
810 | + self.state.kube_control.update(event.relation.data.get(event.unit, {})) |
811 | + self.check_charm_status() |
812 | + |
813 | + def on_kube_control_relation_departed(self, event): |
814 | + """Handle kube-control relation departed.""" |
815 | + self.state.configured = False |
816 | + for k in self.state.kube_control.keys(): |
817 | + self.state.kube_control[k] = "" |
818 | + self.check_charm_status() |
819 | + |
820 | + def on_nrpe_external_master_relation_joined(self, event): |
821 | + """Handle nrpe-external-master relation joined.""" |
822 | + self.state.nrpe_configured = True |
823 | + self.check_charm_status() |
824 | + |
825 | + def on_nrpe_external_master_relation_departed(self, event): |
826 | + """Handle nrpe-external-master relation departed.""" |
827 | + self.state.configured = False |
828 | + self.state.nrpe_configured = False |
829 | + self.check_charm_status() |
830 | + |
831 | + |
832 | +if __name__ == "__main__": |
833 | + main(KubernetesServiceChecksCharm) |
834 | diff --git a/src/setuppath.py b/src/setuppath.py |
835 | new file mode 100644 |
836 | index 0000000..768e049 |
837 | --- /dev/null |
838 | +++ b/src/setuppath.py |
839 | @@ -0,0 +1,4 @@ |
840 | +"""Include ./lib in the charm's PATH.""" |
841 | +import sys |
842 | + |
843 | +sys.path.append("lib") |
844 | diff --git a/tests/functional/requirements.txt b/tests/functional/requirements.txt |
845 | new file mode 100644 |
846 | index 0000000..b7c9112 |
847 | --- /dev/null |
848 | +++ b/tests/functional/requirements.txt |
849 | @@ -0,0 +1 @@ |
850 | +git+https://github.com/openstack-charmers/zaza.git#egg=zaza |
851 | diff --git a/tests/functional/tests/bundles/bionic.yaml b/tests/functional/tests/bundles/bionic.yaml |
852 | new file mode 100644 |
853 | index 0000000..c55ba41 |
854 | --- /dev/null |
855 | +++ b/tests/functional/tests/bundles/bionic.yaml |
856 | @@ -0,0 +1,90 @@ |
857 | +series: bionic |
858 | +applications: |
859 | + kubernetes-service-checks: |
860 | + charm: ../../../../ |
861 | + num_units: 1 |
862 | + containerd: |
863 | + charm: cs:~containers/containerd |
864 | + options: |
865 | + gpu_driver: none |
866 | + resources: {} |
867 | + easyrsa: |
868 | + charm: cs:~containers/easyrsa |
869 | + num_units: 1 |
870 | + resources: |
871 | + easyrsa: 5 |
872 | + etcd: |
873 | + charm: cs:~containers/etcd |
874 | + num_units: 1 |
875 | + options: |
876 | + channel: 3.3/stable |
877 | + resources: |
878 | + core: 0 |
879 | + etcd: 3 |
880 | + snapshot: 0 |
881 | + flannel: |
882 | + charm: cs:~containers/flannel |
883 | + resources: |
884 | + flannel-amd64: 625 |
885 | + flannel-arm64: 622 |
886 | + flannel-s390x: 609 |
887 | + kubernetes-master: |
888 | + charm: cs:~containers/kubernetes-master |
889 | + num_units: 1 |
890 | + constraints: cores=4 mem=4G root-disk=16G |
891 | + options: |
892 | + channel: 1.18/stable |
893 | + resources: |
894 | + cdk-addons: 0 |
895 | + core: 0 |
896 | + kube-apiserver: 0 |
897 | + kube-controller-manager: 0 |
898 | + kube-proxy: 0 |
899 | + kube-scheduler: 0 |
900 | + kubectl: 0 |
901 | + kubernetes-worker: |
902 | + charm: cs:~containers/kubernetes-worker |
903 | + expose: true |
904 | + num_units: 1 |
905 | + constraints: cores=4 mem=4G root-disk=16G |
906 | + options: |
907 | + channel: 1.18/stable |
908 | + resources: |
909 | + cni-amd64: 645 |
910 | + cni-arm64: 636 |
911 | + cni-s390x: 648 |
912 | + core: 0 |
913 | + kube-proxy: 0 |
914 | + kubectl: 0 |
915 | + kubelet: 0 |
916 | + nrpe: |
917 | + charm: cs:nrpe |
918 | +relations: |
919 | + - - kubernetes-master:kube-api-endpoint |
920 | + - kubernetes-worker:kube-api-endpoint |
921 | + - - kubernetes-service-checks:nrpe-external-master |
922 | + - nrpe:nrpe-external-master |
923 | + - - kubernetes-master:kube-control |
924 | + - kubernetes-worker:kube-control |
925 | + - - kubernetes-master:certificates |
926 | + - easyrsa:client |
927 | + - - etcd:certificates |
928 | + - easyrsa:client |
929 | + - - kubernetes-master:etcd |
930 | + - etcd:db |
931 | + - - kubernetes-worker:certificates |
932 | + - easyrsa:client |
933 | + - - flannel:etcd |
934 | + - etcd:db |
935 | + - - flannel:cni |
936 | + - kubernetes-master:cni |
937 | + - - flannel:cni |
938 | + - kubernetes-worker:cni |
939 | + - - containerd:containerd |
940 | + - kubernetes-worker:container-runtime |
941 | + - - containerd:containerd |
942 | + - kubernetes-master:container-runtime |
943 | + - - kubernetes-service-checks:kube-control |
944 | + - kubernetes-master:kube-control |
945 | + - - kubernetes-service-checks:kube-api-endpoint |
946 | + - kubernetes-master:kube-api-endpoint |
947 | \ No newline at end of file |
948 | diff --git a/tests/functional/tests/bundles/focal.yaml b/tests/functional/tests/bundles/focal.yaml |
949 | new file mode 100644 |
950 | index 0000000..31888a0 |
951 | --- /dev/null |
952 | +++ b/tests/functional/tests/bundles/focal.yaml |
953 | @@ -0,0 +1,90 @@ |
954 | +series: focal |
955 | +applications: |
956 | + kubernetes-service-checks: |
957 | + charm: ../../../../ |
958 | + num_units: 1 |
959 | + containerd: |
960 | + charm: cs:~containers/containerd |
961 | + options: |
962 | + gpu_driver: none |
963 | + resources: {} |
964 | + easyrsa: |
965 | + charm: cs:~containers/easyrsa |
966 | + num_units: 1 |
967 | + resources: |
968 | + easyrsa: 5 |
969 | + etcd: |
970 | + charm: cs:~containers/etcd |
971 | + num_units: 1 |
972 | + options: |
973 | + channel: 3.3/stable |
974 | + resources: |
975 | + core: 0 |
976 | + etcd: 3 |
977 | + snapshot: 0 |
978 | + flannel: |
979 | + charm: cs:~containers/flannel |
980 | + resources: |
981 | + flannel-amd64: 625 |
982 | + flannel-arm64: 622 |
983 | + flannel-s390x: 609 |
984 | + kubernetes-master: |
985 | + charm: cs:~containers/kubernetes-master |
986 | + num_units: 1 |
987 | + constraints: cores=4 mem=4G root-disk=16G |
988 | + options: |
989 | + channel: 1.18/stable |
990 | + resources: |
991 | + cdk-addons: 0 |
992 | + core: 0 |
993 | + kube-apiserver: 0 |
994 | + kube-controller-manager: 0 |
995 | + kube-proxy: 0 |
996 | + kube-scheduler: 0 |
997 | + kubectl: 0 |
998 | + kubernetes-worker: |
999 | + charm: cs:~containers/kubernetes-worker |
1000 | + expose: true |
1001 | + num_units: 1 |
1002 | + constraints: cores=4 mem=4G root-disk=16G |
1003 | + options: |
1004 | + channel: 1.18/stable |
1005 | + resources: |
1006 | + cni-amd64: 645 |
1007 | + cni-arm64: 636 |
1008 | + cni-s390x: 648 |
1009 | + core: 0 |
1010 | + kube-proxy: 0 |
1011 | + kubectl: 0 |
1012 | + kubelet: 0 |
1013 | + nrpe: |
1014 | + charm: cs:nrpe |
1015 | +relations: |
1016 | + - - kubernetes-master:kube-api-endpoint |
1017 | + - kubernetes-worker:kube-api-endpoint |
1018 | + - - kubernetes-service-checks:nrpe-external-master |
1019 | + - nrpe:nrpe-external-master |
1020 | + - - kubernetes-master:kube-control |
1021 | + - kubernetes-worker:kube-control |
1022 | + - - kubernetes-master:certificates |
1023 | + - easyrsa:client |
1024 | + - - etcd:certificates |
1025 | + - easyrsa:client |
1026 | + - - kubernetes-master:etcd |
1027 | + - etcd:db |
1028 | + - - kubernetes-worker:certificates |
1029 | + - easyrsa:client |
1030 | + - - flannel:etcd |
1031 | + - etcd:db |
1032 | + - - flannel:cni |
1033 | + - kubernetes-master:cni |
1034 | + - - flannel:cni |
1035 | + - kubernetes-worker:cni |
1036 | + - - containerd:containerd |
1037 | + - kubernetes-worker:container-runtime |
1038 | + - - containerd:containerd |
1039 | + - kubernetes-master:container-runtime |
1040 | + - - kubernetes-service-checks:kube-control |
1041 | + - kubernetes-master:kube-control |
1042 | + - - kubernetes-service-checks:kube-api-endpoint |
1043 | + - kubernetes-master:kube-api-endpoint |
1044 | diff --git a/tests/functional/tests/bundles/xenial.yaml b/tests/functional/tests/bundles/xenial.yaml |
1045 | new file mode 100644 |
1046 | index 0000000..962bdfd |
1047 | --- /dev/null |
1048 | +++ b/tests/functional/tests/bundles/xenial.yaml |
1049 | @@ -0,0 +1,90 @@ |
1050 | +series: xenial |
1051 | +applications: |
1052 | + kubernetes-service-checks: |
1053 | + charm: ../../../../ |
1054 | + num_units: 1 |
1055 | + containerd: |
1056 | + charm: cs:~containers/containerd |
1057 | + options: |
1058 | + gpu_driver: none |
1059 | + resources: {} |
1060 | + easyrsa: |
1061 | + charm: cs:~containers/easyrsa |
1062 | + num_units: 1 |
1063 | + resources: |
1064 | + easyrsa: 5 |
1065 | + etcd: |
1066 | + charm: cs:~containers/etcd |
1067 | + num_units: 1 |
1068 | + options: |
1069 | + channel: 3.3/stable |
1070 | + resources: |
1071 | + core: 0 |
1072 | + etcd: 3 |
1073 | + snapshot: 0 |
1074 | + flannel: |
1075 | + charm: cs:~containers/flannel |
1076 | + resources: |
1077 | + flannel-amd64: 625 |
1078 | + flannel-arm64: 622 |
1079 | + flannel-s390x: 609 |
1080 | + kubernetes-master: |
1081 | + charm: cs:~containers/kubernetes-master |
1082 | + num_units: 1 |
1083 | + constraints: cores=4 mem=4G root-disk=16G |
1084 | + options: |
1085 | + channel: 1.18/stable |
1086 | + resources: |
1087 | + cdk-addons: 0 |
1088 | + core: 0 |
1089 | + kube-apiserver: 0 |
1090 | + kube-controller-manager: 0 |
1091 | + kube-proxy: 0 |
1092 | + kube-scheduler: 0 |
1093 | + kubectl: 0 |
1094 | + kubernetes-worker: |
1095 | + charm: cs:~containers/kubernetes-worker |
1096 | + expose: true |
1097 | + num_units: 1 |
1098 | + constraints: cores=4 mem=4G root-disk=16G |
1099 | + options: |
1100 | + channel: 1.18/stable |
1101 | + resources: |
1102 | + cni-amd64: 645 |
1103 | + cni-arm64: 636 |
1104 | + cni-s390x: 648 |
1105 | + core: 0 |
1106 | + kube-proxy: 0 |
1107 | + kubectl: 0 |
1108 | + kubelet: 0 |
1109 | + nrpe: |
1110 | + charm: cs:nrpe |
1111 | +relations: |
1112 | + - - kubernetes-master:kube-api-endpoint |
1113 | + - kubernetes-worker:kube-api-endpoint |
1114 | + - - kubernetes-service-checks:nrpe-external-master |
1115 | + - nrpe:nrpe-external-master |
1116 | + - - kubernetes-master:kube-control |
1117 | + - kubernetes-worker:kube-control |
1118 | + - - kubernetes-master:certificates |
1119 | + - easyrsa:client |
1120 | + - - etcd:certificates |
1121 | + - easyrsa:client |
1122 | + - - kubernetes-master:etcd |
1123 | + - etcd:db |
1124 | + - - kubernetes-worker:certificates |
1125 | + - easyrsa:client |
1126 | + - - flannel:etcd |
1127 | + - etcd:db |
1128 | + - - flannel:cni |
1129 | + - kubernetes-master:cni |
1130 | + - - flannel:cni |
1131 | + - kubernetes-worker:cni |
1132 | + - - containerd:containerd |
1133 | + - kubernetes-worker:container-runtime |
1134 | + - - containerd:containerd |
1135 | + - kubernetes-master:container-runtime |
1136 | + - - kubernetes-service-checks:kube-control |
1137 | + - kubernetes-master:kube-control |
1138 | + - - kubernetes-service-checks:kube-api-endpoint |
1139 | + - kubernetes-master:kube-api-endpoint |
1140 | diff --git a/tests/functional/tests/kubernetes_service_checks.py b/tests/functional/tests/kubernetes_service_checks.py |
1141 | new file mode 100644 |
1142 | index 0000000..e1520f3 |
1143 | --- /dev/null |
1144 | +++ b/tests/functional/tests/kubernetes_service_checks.py |
1145 | @@ -0,0 +1,162 @@ |
1146 | +"""Charm Kubernetes Service Checks Functional Tests.""" |
1147 | +import concurrent.futures |
1148 | +import logging |
1149 | +import re |
1150 | +import time |
1151 | +import unittest |
1152 | + |
1153 | +from juju.errors import JujuAPIError |
1154 | +import zaza.model |
1155 | + |
1156 | + |
1157 | +class TestBase(unittest.TestCase): |
1158 | + """Base Class for charm functional tests.""" |
1159 | + |
1160 | + @classmethod |
1161 | + def setUpClass(cls): |
1162 | + """Run setup for tests.""" |
1163 | + cls.model_name = zaza.model.get_juju_model() |
1164 | + cls.application_name = "kubernetes-service-checks" |
1165 | + |
1166 | + def setUp(self): |
1167 | + """Set up functional tests & ensure all relations added.""" |
1168 | + for local_relation_name, remote_relation_unit in [ |
1169 | + ("kube-api-endpoint", "kubernetes-master"), |
1170 | + ("kube-control", "kubernetes-master"), |
1171 | + ("nrpe-external-master", "nrpe"), |
1172 | + ]: |
1173 | + logging.info("Adding relation {} with {}".format(local_relation_name, remote_relation_unit)) |
1174 | + try: |
1175 | + zaza.model.add_relation( |
1176 | + self.application_name, local_relation_name, remote_relation_unit, self.model_name |
1177 | + ) |
1178 | + except JujuAPIError as e: |
1179 | + p = r"^.*cannot\ add\ relation.*already\ exists" |
1180 | + if re.search(p, e.message): |
1181 | + pass |
1182 | + else: |
1183 | + raise (e) |
1184 | + zaza.model.block_until_wl_status_info_starts_with(self.application_name, status="Unit is ready", timeout=200) |
1185 | + |
1186 | + |
1187 | +class TestChecks(TestBase): |
1188 | + """Tests for availability and usefulness of nagios checks.""" |
1189 | + |
1190 | + expected_checks = ["check_k8s_api_health.cfg", "check_k8s_api_cert_expiration.cfg"] |
1191 | + checks_dir = "/etc/nagios/nrpe.d/" |
1192 | + expected_plugins = ["check_kubernetes_api.py"] |
1193 | + plugins_dir = "/usr/local/lib/nagios/plugins/" |
1194 | + |
1195 | + # TODO: Need testing around setting the trusted_ssl_ca cert |
1196 | + # - does it get written to /etc/ssl/certs/ca-certificates.crt? |
1197 | + # - does the k8s check plugin see it and use it for verification? |
1198 | + |
1199 | + def test_check_plugins_exist(self): |
1200 | + """Verify that kubernetes service checks plugins are found.""" |
1201 | + fail_messages = [] |
1202 | + for plugin in self.expected_plugins: |
1203 | + pluginpath = self.plugins_dir + plugin |
1204 | + response = zaza.model.run_on_unit( |
1205 | + "kubernetes-service-checks/0", '[ -f "{}" ]'.format(pluginpath), model_name=self.model_name, timeout=30 |
1206 | + ) |
1207 | + if response["Code"] != "0": |
1208 | + fail_messages.append("Missing plugin: {}".format(pluginpath)) |
1209 | + continue |
1210 | + |
1211 | + # check executable |
1212 | + response = zaza.model.run_on_unit( |
1213 | + "kubernetes-service-checks/0", '[ -x "{}" ]'.format(pluginpath), model_name=self.model_name, timeout=30 |
1214 | + ) |
1215 | + |
1216 | + if response["Code"] != "0": |
1217 | + fail_messages.append("Plugin not executable: {}".format(pluginpath)) |
1218 | + |
1219 | + if fail_messages: |
1220 | + self.fail("\n".join(fail_messages)) |
1221 | + |
1222 | + def test_checks_exist(self): |
1223 | + """Verify that kubernetes service checks nrpe checks exist.""" |
1224 | + fail_messages = [] |
1225 | + for check in self.expected_checks: |
1226 | + checkpath = self.checks_dir + check |
1227 | + response = zaza.model.run_on_unit( |
1228 | + "kubernetes-service-checks/0", '[ -f "{}" ]'.format(checkpath), model_name=self.model_name, timeout=30 |
1229 | + ) |
1230 | + if response["Code"] != "0": |
1231 | + fail_messages.append("Missing check: {}".format(checkpath)) |
1232 | + if fail_messages: |
1233 | + self.fail("\n".join(fail_messages)) |
1234 | + |
1235 | + |
1236 | +class TestRelations(TestBase): |
1237 | + """Tests for charm behavior adding and removing relations.""" |
1238 | + |
1239 | + def _get_relation_id(self, remote_application, interface_name): |
1240 | + return zaza.model.get_relation_id( |
1241 | + self.application_name, remote_application, model_name=self.model_name, remote_interface_name=interface_name |
1242 | + ) |
1243 | + |
1244 | + def test_remove_kube_api_endpoint(self): |
1245 | + """Test removing kube-api-endpoint relation.""" |
1246 | + rel_name = "kube-api-endpoint" |
1247 | + remote_app = "kubernetes-master" |
1248 | + logging.info("Removing kube-api-endpoint relation") |
1249 | + |
1250 | + zaza.model.remove_relation(self.application_name, rel_name, remote_app, self.model_name) |
1251 | + try: |
1252 | + zaza.model.block_until_wl_status_info_starts_with( |
1253 | + self.application_name, status="missing kube-api-endpoint relation", timeout=180 |
1254 | + ) |
1255 | + except concurrent.futures._base.TimeoutError: |
1256 | + self.fail("Timed out waiting for Unit to become blocked") |
1257 | + |
1258 | + logging.info("Waiting for relation {} to be destroyed".format(rel_name)) |
1259 | + timeout = time.time() + 600 |
1260 | + while self._get_relation_id(remote_app, rel_name) is not None: |
1261 | + time.sleep(5) |
1262 | + if time.time() > timeout: |
1263 | + self.fail("Timed out waiting for the relation {} to be destroyed".format(rel_name)) |
1264 | + |
1265 | + def test_remove_kube_control(self): |
1266 | + """Test removing kube-control relation.""" |
1267 | + rel_name = "kube-control" |
1268 | + remote_app = "kubernetes-master" |
1269 | + logging.info("Removing kube-control relation") |
1270 | + |
1271 | + zaza.model.remove_relation(self.application_name, rel_name, remote_app, self.model_name) |
1272 | + |
1273 | + try: |
1274 | + zaza.model.block_until_wl_status_info_starts_with( |
1275 | + self.application_name, status="missing kube-control relation", timeout=180 |
1276 | + ) |
1277 | + except concurrent.futures._base.TimeoutError: |
1278 | + self.fail("Timed out waiting for Unit to become blocked") |
1279 | + |
1280 | + logging.info("Waiting for relation {} to be destroyed".format(rel_name)) |
1281 | + timeout = time.time() + 600 |
1282 | + while self._get_relation_id(remote_app, rel_name) is not None: |
1283 | + time.sleep(5) |
1284 | + if time.time() > timeout: |
1285 | + self.fail("Timed out waiting for the relation {} to be destroyed".format(rel_name)) |
1286 | + |
1287 | + def test_remove_nrpe_external_master(self): |
1288 | + """Test removing nrpe-external-master relation.""" |
1289 | + rel_name = "nrpe-external-master" |
1290 | + remote_app = "nrpe" |
1291 | + logging.info("Removing nrpe-external-master relation") |
1292 | + |
1293 | + zaza.model.remove_relation(self.application_name, rel_name, remote_app, self.model_name) |
1294 | + |
1295 | + try: |
1296 | + zaza.model.block_until_wl_status_info_starts_with( |
1297 | + self.application_name, status="missing nrpe-external-master relation", timeout=180 |
1298 | + ) |
1299 | + except concurrent.futures._base.TimeoutError: |
1300 | + self.fail("Timed out waiting for Unit to become blocked") |
1301 | + |
1302 | + logging.info("Waiting for relation {} to be destroyed".format(rel_name)) |
1303 | + timeout = time.time() + 600 |
1304 | + while self._get_relation_id(remote_app, rel_name) is not None: |
1305 | + time.sleep(5) |
1306 | + if time.time() > timeout: |
1307 | + self.fail("Timed out waiting for the relation {} to be destroyed".format(rel_name)) |
1308 | diff --git a/tests/functional/tests/tests.yaml b/tests/functional/tests/tests.yaml |
1309 | new file mode 100644 |
1310 | index 0000000..75a9ce8 |
1311 | --- /dev/null |
1312 | +++ b/tests/functional/tests/tests.yaml |
1313 | @@ -0,0 +1,33 @@ |
1314 | +tests: |
1315 | + - tests.kubernetes_service_checks.TestChecks |
1316 | + - tests.kubernetes_service_checks.TestRelations |
1317 | +target_deploy_status: |
1318 | + kubernetes-service-checks: |
1319 | + workload-status: blocked |
1320 | + workload-status-message: "missing kube-api-endpoint relation" |
1321 | + kubernetes-master: |
1322 | + workload-status: active |
1323 | + workload-status-message: "Kubernetes master running." |
1324 | + kubernetes-worker: |
1325 | + workload-status: active |
1326 | + workload-status-message: "Kubernetes worker running." |
1327 | + flannel: |
1328 | + workload-status: active |
1329 | + workload-status-message: "Flannel subnet" |
1330 | + easyrsa: |
1331 | + workload-status: active |
1332 | + workload-status-message: "Certificate Authority connected." |
1333 | + containerd: |
1334 | + workload-status: active |
1335 | + workload-status-message: "Container runtime available" |
1336 | + etcd: |
1337 | + workload-status: active |
1338 | + workload-status-message: "Healthy with 1 known peer" |
1339 | + nrpe: |
1340 | + workload-status: active |
1341 | + workload-status-message: "ready" |
1342 | +gate_bundles: |
1343 | + - xenial |
1344 | + - bionic |
1345 | +smoke_bundles: |
1346 | + - focal |
1347 | diff --git a/tests/unit/requirements.txt b/tests/unit/requirements.txt |
1348 | new file mode 100644 |
1349 | index 0000000..b5f9fbd |
1350 | --- /dev/null |
1351 | +++ b/tests/unit/requirements.txt |
1352 | @@ -0,0 +1,5 @@ |
1353 | +mock |
1354 | +pyyaml |
1355 | +coverage |
1356 | +six |
1357 | +urllib3 |
1358 | diff --git a/tests/unit/setuppath.py b/tests/unit/setuppath.py |
1359 | new file mode 100644 |
1360 | index 0000000..c3404e5 |
1361 | --- /dev/null |
1362 | +++ b/tests/unit/setuppath.py |
1363 | @@ -0,0 +1,6 @@ |
1364 | +"""Include ./lib ./src and ./file/plugins in the tests' PATH.""" |
1365 | +import sys |
1366 | + |
1367 | +sys.path.append("lib") |
1368 | +sys.path.append("src") |
1369 | +sys.path.append("files/plugins") |
1370 | diff --git a/tests/unit/test_charm.py b/tests/unit/test_charm.py |
1371 | new file mode 100644 |
1372 | index 0000000..24f83f5 |
1373 | --- /dev/null |
1374 | +++ b/tests/unit/test_charm.py |
1375 | @@ -0,0 +1,243 @@ |
1376 | +"""Charm unit tests.""" |
1377 | +import os |
1378 | +from pathlib import Path |
1379 | +import unittest |
1380 | + |
1381 | + |
1382 | +import mock |
1383 | +import yaml |
1384 | + |
1385 | +# include ./lib in the charm's PATH |
1386 | +import setuppath # noqa:F401 |
1387 | + |
1388 | +from charm import KubernetesServiceChecksCharm # noqa:I100 |
1389 | +import ops.main |
1390 | +from ops.testing import Harness |
1391 | + |
1392 | +TEST_KUBE_CONTOL_RELATION_DATA = { |
1393 | + "creds": """{"system:node:juju-62684f-0": |
1394 | + {"client_token": "DECAFBADBEEF", |
1395 | + "kubelet_token": "ABCDEF012345", |
1396 | + "proxy_token": "BADC0FFEEDAD", |
1397 | + "scope": "kubernetes-worker/0"} |
1398 | + }""" # noqa:E127 |
1399 | +} |
1400 | +TEST_KUBE_API_ENDPOINT_RELATION_DATA = {"hostname": "1.1.1.1", "port": "1111"} |
1401 | + |
1402 | + |
1403 | +class TestKubernetesServiceChecksCharm(unittest.TestCase): |
1404 | + """Test Kubernetes Service Checks Charm Code.""" |
1405 | + |
1406 | + @classmethod |
1407 | + def setUpClass(cls): |
1408 | + """Prepare class fixture.""" |
1409 | + # Stop unit test from calling fchown |
1410 | + fchown_patcher = mock.patch("os.fchown") |
1411 | + cls.mock_fchown = fchown_patcher.start() |
1412 | + chown_patcher = mock.patch("os.chown") |
1413 | + cls.mock_chown = chown_patcher.start() |
1414 | + |
1415 | + # Stop charmhelpers host from logging via debug log |
1416 | + host_log_patcher = mock.patch("charmhelpers.core.host.log") |
1417 | + cls.mock_juju_log = host_log_patcher.start() |
1418 | + |
1419 | + # Stop charmhelpers snap from logging via debug log |
1420 | + snap_log_patcher = mock.patch("charmhelpers.fetch.snap.log") |
1421 | + cls.mock_snap_log = snap_log_patcher.start() |
1422 | + |
1423 | + charm_logger_patcher = mock.patch("charm.logging") |
1424 | + cls.mock_charm_log = charm_logger_patcher.start() |
1425 | + |
1426 | + lib_logger_patcher = mock.patch("lib.lib_kubernetes_service_checks.logging") |
1427 | + cls.mock_lib_logger = lib_logger_patcher.start() |
1428 | + |
1429 | + # Prevent charmhelpers from calling systemctl |
1430 | + host_service_patcher = mock.patch("charmhelpers.core.host.service_stop") |
1431 | + cls.mock_service_stop = host_service_patcher.start() |
1432 | + host_service_patcher = mock.patch("charmhelpers.core.host.service_start") |
1433 | + cls.mock_service_start = host_service_patcher.start() |
1434 | + host_service_patcher = mock.patch("charmhelpers.core.host.service_restart") |
1435 | + cls.mock_service_restart = host_service_patcher.start() |
1436 | + |
1437 | + # Setup mock JUJU Environment variables |
1438 | + os.environ["JUJU_UNIT_NAME"] = "mock/0" |
1439 | + os.environ["JUJU_CHARM_DIR"] = "." |
1440 | + |
1441 | + def setUp(self): |
1442 | + """Prepare tests.""" |
1443 | + self.harness = Harness(KubernetesServiceChecksCharm) |
1444 | + # Mock config_get to return default config |
1445 | + with open(ops.main._get_charm_dir() / Path("config.yaml"), "r") as config_file: |
1446 | + config = yaml.safe_load(config_file) |
1447 | + charm_config = {} |
1448 | + |
1449 | + for key, _ in config["options"].items(): |
1450 | + charm_config[key] = config["options"][key]["default"] |
1451 | + |
1452 | + self.harness._backend._config = charm_config |
1453 | + |
1454 | + def test_harness(self): |
1455 | + """Verify harness.""" |
1456 | + self.harness.begin() |
1457 | + self.assertFalse(self.harness.charm.state.installed) |
1458 | + |
1459 | + @mock.patch("charmhelpers.fetch.snap.subprocess.check_call") |
1460 | + def test_install(self, mock_snap_subprocess): |
1461 | + """Test response to an install event.""" |
1462 | + mock_snap_subprocess.return_value = 0 |
1463 | + mock_snap_subprocess.side_effect = None |
1464 | + |
1465 | + self.harness.begin() |
1466 | + self.harness.charm.on.install.emit() |
1467 | + |
1468 | + self.assertEqual(self.harness.charm.unit.status.name, "maintenance") |
1469 | + self.assertEqual(self.harness.charm.unit.status.message, "Install complete") |
1470 | + self.assertTrue(self.harness.charm.state.installed) |
1471 | + |
1472 | + def test_config_changed(self): |
1473 | + """Test response to config changed event.""" |
1474 | + self.harness.set_leader(True) |
1475 | + self.harness.populate_oci_resources() |
1476 | + self.harness.begin() |
1477 | + self.harness.charm.check_charm_status = mock.MagicMock() |
1478 | + self.harness.charm.state.installed = True |
1479 | + self.harness.charm.on.config_changed.emit() |
1480 | + self.harness.charm.check_charm_status.assert_called_once() |
1481 | + |
1482 | + def test_start_not_installed(self): |
1483 | + """Test response to start event without install state.""" |
1484 | + self.harness.begin() |
1485 | + self.harness.charm.on.start.emit() |
1486 | + self.assertFalse(self.harness.charm.state.started) |
1487 | + |
1488 | + def test_start_not_configured(self): |
1489 | + """Test response to start event without configured state.""" |
1490 | + self.harness.begin() |
1491 | + self.harness.charm.state.installed = True |
1492 | + self.harness.charm.on.start.emit() |
1493 | + self.assertFalse(self.harness.charm.state.started) |
1494 | + |
1495 | + def test_start(self): |
1496 | + """Test response to start event.""" |
1497 | + self.harness.begin() |
1498 | + self.harness.charm.state.installed = True |
1499 | + self.harness.charm.state.configured = True |
1500 | + self.harness.charm.on.start.emit() |
1501 | + self.assertTrue(self.harness.charm.state.started) |
1502 | + self.assertEqual(self.harness.charm.unit.status.name, "active") |
1503 | + |
1504 | + def test_on_kube_api_endpoint_relation_changed(self): |
1505 | + """Check kube-api-endpoint relation changed handling.""" |
1506 | + relation_id = self.harness.add_relation("kube-api-endpoint", "kubernetes-master") |
1507 | + remote_unit = "kubernetes-master/0" |
1508 | + self.harness.begin() |
1509 | + self.harness.charm.check_charm_status = mock.MagicMock() |
1510 | + self.harness.add_relation_unit(relation_id, remote_unit) |
1511 | + self.harness.update_relation_data(relation_id, remote_unit, TEST_KUBE_API_ENDPOINT_RELATION_DATA) |
1512 | + |
1513 | + self.harness.charm.check_charm_status.assert_called_once() |
1514 | + self.assertEqual(self.harness.charm.helper.kubernetes_api_address, "1.1.1.1") |
1515 | + self.assertEqual(self.harness.charm.helper.kubernetes_api_port, "1111") |
1516 | + |
1517 | + def test_on_kube_control_relation_changed(self): |
1518 | + """Check kube-control relation changed handling.""" |
1519 | + relation_id = self.harness.add_relation("kube-control", "kubernetes-master") |
1520 | + remote_unit = "kubernetes-master/0" |
1521 | + self.harness.begin() |
1522 | + self.harness.charm.check_charm_status = mock.MagicMock() |
1523 | + self.harness.add_relation_unit(relation_id, remote_unit) |
1524 | + self.harness.update_relation_data(relation_id, remote_unit, TEST_KUBE_CONTOL_RELATION_DATA) |
1525 | + |
1526 | + self.harness.charm.check_charm_status.assert_called_once() |
1527 | + assert self.harness.charm.helper.kubernetes_client_token == "DECAFBADBEEF" |
1528 | + |
1529 | + def test_nrpe_external_master_relation_joined(self): |
1530 | + """Check that nrpe.configure is True after nrpe relation joined.""" |
1531 | + relation_id = self.harness.add_relation("nrpe-external-master", "nrpe") |
1532 | + remote_unit = "nrpe/0" |
1533 | + self.harness.begin() |
1534 | + self.assertFalse(self.harness.charm.state.nrpe_configured) |
1535 | + self.harness.charm.check_charm_status = mock.MagicMock() |
1536 | + self.harness.add_relation_unit(relation_id, remote_unit) |
1537 | + |
1538 | + self.harness.charm.check_charm_status.assert_called_once() |
1539 | + self.assertTrue(self.harness.charm.state.nrpe_configured) |
1540 | + |
1541 | + @mock.patch("ops.model.RelationData") |
1542 | + def test_nrpe_external_master_relation_departed(self, mock_relation_data): |
1543 | + """Check that nrpe.configure is False after nrpe relation departed.""" |
1544 | + mock_relation_data.return_value.__getitem__.return_value = {} |
1545 | + self.harness.begin() |
1546 | + self.harness.charm.check_charm_status = mock.MagicMock() |
1547 | + self.emit("nrpe_external_master_relation_departed") |
1548 | + self.harness.charm.check_charm_status.assert_called_once() |
1549 | + |
1550 | + self.assertFalse(self.harness.charm.state.nrpe_configured) |
1551 | + |
1552 | + def test_check_charm_status_kube_api_endpoint_relation_missing(self): |
1553 | + """Check that the chatm blocks without kube-api-endpoint relation.""" |
1554 | + self.harness.begin() |
1555 | + self.harness.charm.state.kube_control.update(TEST_KUBE_CONTOL_RELATION_DATA) |
1556 | + self.harness.charm.state.nrpe_configured = True |
1557 | + self.harness.charm.check_charm_status() |
1558 | + |
1559 | + self.assertFalse(self.harness.charm.state.configured) |
1560 | + self.assertEqual(self.harness.charm.unit.status.name, "blocked") |
1561 | + self.assertEqual(self.harness.charm.unit.status.message, "missing kube-api-endpoint relation") |
1562 | + |
1563 | + def test_check_charm_status_kube_control_relation_missing(self): |
1564 | + """Check that the charm blocks without kube-control relation.""" |
1565 | + self.harness.begin() |
1566 | + self.harness.charm.state.kube_api_endpoint.update(TEST_KUBE_API_ENDPOINT_RELATION_DATA) |
1567 | + self.harness.charm.state.nrpe_configured = True |
1568 | + self.harness.charm.check_charm_status() |
1569 | + |
1570 | + self.assertFalse(self.harness.charm.state.configured) |
1571 | + self.assertEqual(self.harness.charm.unit.status.name, "blocked") |
1572 | + self.assertEqual(self.harness.charm.unit.status.message, "missing kube-control relation") |
1573 | + |
1574 | + def test_check_charm_status_nrpe_relation_missing(self): |
1575 | + """Check that the charm bloack without nrpe relation.""" |
1576 | + self.harness.begin() |
1577 | + self.harness.charm.state.kube_control.update(TEST_KUBE_CONTOL_RELATION_DATA) |
1578 | + self.harness.charm.state.kube_api_endpoint.update(TEST_KUBE_API_ENDPOINT_RELATION_DATA) |
1579 | + self.harness.charm.check_charm_status() |
1580 | + |
1581 | + self.assertFalse(self.harness.charm.state.configured) |
1582 | + self.assertEqual(self.harness.charm.unit.status.name, "blocked") |
1583 | + self.assertEqual(self.harness.charm.unit.status.message, "missing nrpe-external-master relation") |
1584 | + |
1585 | + def test_check_charm_status_configured(self): |
1586 | + """Check the charm becomes configured.""" |
1587 | + self.harness.begin() |
1588 | + self.harness.charm.helper.configure = mock.MagicMock() |
1589 | + self.harness.charm.state.kube_control.update(TEST_KUBE_CONTOL_RELATION_DATA) |
1590 | + self.harness.charm.state.kube_api_endpoint.update(TEST_KUBE_API_ENDPOINT_RELATION_DATA) |
1591 | + self.harness.charm.state.nrpe_configured = True |
1592 | + self.harness.charm.check_charm_status() |
1593 | + |
1594 | + self.harness.charm.helper.configure.assert_called_once() |
1595 | + self.assertTrue(self.harness.charm.state.configured) |
1596 | + |
1597 | + def emit(self, event): |
1598 | + """Emit the named hook on the charm.""" |
1599 | + self.harness.charm.framework.reemit() |
1600 | + |
1601 | + if "_relation_" in event: |
1602 | + relation_name = event.split("_relation")[0].replace("_", "-") |
1603 | + with mock.patch.dict( |
1604 | + "os.environ", |
1605 | + { |
1606 | + "JUJU_RELATION": relation_name, |
1607 | + "JUJU_RELATION_ID": "1", |
1608 | + "JUJU_REMOTE_APP": "mock", |
1609 | + "JUJU_REMOTE_UNIT": "mock/0", |
1610 | + }, |
1611 | + ): |
1612 | + ops.main._emit_charm_event(self.harness.charm, event) |
1613 | + else: |
1614 | + ops.main._emit_charm_event(self.harness.charm, event) |
1615 | + |
1616 | + |
1617 | +if __name__ == "__main__": |
1618 | + unittest.main() |
1619 | diff --git a/tests/unit/test_lib_ksc.py b/tests/unit/test_lib_ksc.py |
1620 | new file mode 100644 |
1621 | index 0000000..94f67bc |
1622 | --- /dev/null |
1623 | +++ b/tests/unit/test_lib_ksc.py |
1624 | @@ -0,0 +1,164 @@ |
1625 | +"""Tests for Kubernetes Service Checks Helper.""" |
1626 | +import base64 |
1627 | +import os |
1628 | +import subprocess |
1629 | +from subprocess import CalledProcessError |
1630 | +import tempfile |
1631 | +import unittest |
1632 | + |
1633 | +from lib import lib_kubernetes_service_checks |
1634 | +import mock |
1635 | +import yaml |
1636 | + |
1637 | + |
1638 | +TEST_CERTIFICATE = """-----BEGIN CERTIFICATE----- |
1639 | +MIIDOzCCAiOgAwIBAgIJAPoOXrIwH+miMA0GCSqGSIb3DQEBCwUAMBgxFjAUBgNV |
1640 | +BAMMDTEwLjEzMi4yNTEuNjAwHhcNMjAwNzE3MTMzMzI0WhcNMzAwNzE1MTMzMzI0 |
1641 | +WjAYMRYwFAYDVQQDDA0xMC4xMzIuMjUxLjYwMIIBIjANBgkqhkiG9w0BAQEFAAOC |
1642 | +AQ8AMIIBCgKCAQEAqpYVlmT/eRBhCKHaqXjY6EAzvx5GZY0PhL/YGBl9uF8YQGEF |
1643 | +F3k3Ec7pyJMIQblmWxdCPd1uNzHU8mwApiuPG9GtYOK+olqgslLsmOU9LTi6KJWX |
1644 | +x956VxdefXDYvr0B6K/Hdgkb1x//XwvipSV1fZ1MCDIiP/hWKi4CmEq31sVpCBdp |
1645 | +Uiz3qdCzsiGt0f4kbgIJSVtxhWlNJ5MaCOm7gXafkF8OIUTmWhmPp2gH7pfPzzl1 |
1646 | +glOX2Z41qwPuz7Jbcxx/z/yGjdPeJTQYoqJfpDpCrT2er5xyRf66HqKx9Ld/FiqM |
1647 | +ZksRwmzF9WvqCBK8WoRmnvFxk1FZPGt6E5gotwIDAQABo4GHMIGEMB0GA1UdDgQW |
1648 | +BBSUCCmRxb4tKD6w8jZ3hHs4ciFizDBIBgNVHSMEQTA/gBSUCCmRxb4tKD6w8jZ3 |
1649 | +hHs4ciFizKEcpBowGDEWMBQGA1UEAwwNMTAuMTMyLjI1MS42MIIJAPoOXrIwH+mi |
1650 | +MAwGA1UdEwQFMAMBAf8wCwYDVR0PBAQDAgEGMA0GCSqGSIb3DQEBCwUAA4IBAQBv |
1651 | +BYwILWI/4dGczqG0hcqt8tW04Oi+7y0HxzeI/oaUq/HKfvCz5a+WhpykMKRDJoaZ |
1652 | +aejR2Oc7A0OUnenpvMeIiMcUIetM3Q1Gzx0aU+vqUNNaZlooSSbe3z1VK6bUsYDo |
1653 | +qdKhs+mSyuEticK2SEWjT+ZWpV1rjSd5zRZ/UvC1ZhDNJGZotIIqryQWd3YfYl9l |
1654 | +7JrdzUVCbxs4ywxNp9/I+MJEiBfMHQx8FWr1M2HvLDAm6NZLfM68y5FClzfGpopV |
1655 | +0ARirz1AfbS6xUumyXHOH2qH527PUXFdfYGSn+juDG/dRTENYJ3OPAfWdj4ze1qQ |
1656 | +n3ajLSYPvdyKaztdB1VL |
1657 | +-----END CERTIFICATE-----""" |
1658 | + |
1659 | + |
1660 | +class TestLibKSCHelper(unittest.TestCase): |
1661 | + """Unittest class for Kubernetes Service Checks Helper.""" |
1662 | + |
1663 | + @classmethod |
1664 | + def setUpClass(cls): |
1665 | + """Prepare Class Fixture.""" |
1666 | + # Load default config |
1667 | + with open("./config.yaml") as default_config: |
1668 | + cls.config = yaml.safe_load(default_config) |
1669 | + |
1670 | + # set defaults to the config object |
1671 | + for key in cls.config["options"]: |
1672 | + if "default" in cls.config["options"][key]: |
1673 | + cls.config[key] = cls.config["options"][key]["default"] |
1674 | + |
1675 | + # Create test state object |
1676 | + class FakeStateObject(object): |
1677 | + kube_api_endpoint = {"hostname": "1.1.1.1", "port": "1111"} |
1678 | + kube_control = {"creds": """{"kube-client": {"client_token": "abcdef0123456789"}}"""} |
1679 | + installed = False |
1680 | + configured = False |
1681 | + started = False |
1682 | + nrpe_configured = False |
1683 | + |
1684 | + cls.state = FakeStateObject() |
1685 | + |
1686 | + # Stop unit test from calling fchown |
1687 | + fchown_patcher = mock.patch("os.fchown") |
1688 | + cls.mock_fchown = fchown_patcher.start() |
1689 | + chown_patcher = mock.patch("os.chown") |
1690 | + cls.mock_chown = chown_patcher.start() |
1691 | + |
1692 | + # Stop charmhelpers host from logging via debug log |
1693 | + host_log_patcher = mock.patch("charmhelpers.core.host.log") |
1694 | + cls.mock_juju_log = host_log_patcher.start() |
1695 | + |
1696 | + host_logger_patcher = mock.patch("lib.lib_kubernetes_service_checks.logging") |
1697 | + cls.mock_logger = host_logger_patcher.start() |
1698 | + |
1699 | + # Stop charmhelpers snap from logging via debug log |
1700 | + snap_log_patcher = mock.patch("charmhelpers.fetch.snap.log") |
1701 | + cls.mock_snap_log = snap_log_patcher.start() |
1702 | + |
1703 | + # Setup a tmpdir |
1704 | + cls.tmpdir = tempfile.TemporaryDirectory() |
1705 | + cls.cert_path = os.path.join(cls.tmpdir.name, "kubernetes-service-checks.crt") |
1706 | + |
1707 | + lib_kubernetes_service_checks.CERT_FILE = cls.cert_path |
1708 | + lib_kubernetes_service_checks.NAGIOS_PLUGINS_DIR = cls.tmpdir.name |
1709 | + |
1710 | + @classmethod |
1711 | + def tearDownClass(cls): |
1712 | + """Tear down class fixture.""" |
1713 | + mock.patch.stopall() |
1714 | + cls.tmpdir.cleanup() |
1715 | + |
1716 | + def setUp(self): |
1717 | + """Prepare test fixture.""" |
1718 | + self.helper = lib_kubernetes_service_checks.KSCHelper(self.config, self.state) |
1719 | + |
1720 | + def tearDown(self): |
1721 | + """Clean up test fixture.""" |
1722 | + try: |
1723 | + os.remove(self.cert_path) |
1724 | + except FileNotFoundError: |
1725 | + pass |
1726 | + |
1727 | + def test_kube_api_endpoint_properties(self): |
1728 | + """Test that hostname and port properties get passed through.""" |
1729 | + # kube_api_endpoint (relation) -> hostname & port |
1730 | + self.assertEqual(self.helper.kubernetes_api_address, "1.1.1.1") |
1731 | + self.assertEqual(self.helper.kubernetes_api_port, "1111") |
1732 | + |
1733 | + self.helper.state.kube_api_endpoint = {} |
1734 | + self.assertEqual(self.helper.kubernetes_api_address, None) |
1735 | + self.assertEqual(self.helper.kubernetes_api_port, None) |
1736 | + |
1737 | + def test_kube_control_endpoint_properties(self): |
1738 | + """Test KSCHelper client_token gets passed though.""" |
1739 | + # kube-control (relation) -> kube client token |
1740 | + self.assertEqual(self.helper.kubernetes_client_token, "abcdef0123456789") |
1741 | + |
1742 | + self.helper.state.kube_control = {} |
1743 | + self.assertEqual(self.helper.kubernetes_client_token, None) |
1744 | + |
1745 | + @mock.patch("lib.lib_kubernetes_service_checks.subprocess.call") |
1746 | + def test_update_tls_certificates(self, mock_subprocess): |
1747 | + """Test that SSL certificates get updated.""" |
1748 | + # returns False when no available trusted_ssl_cert |
1749 | + self.assertFalse(self.helper.update_tls_certificates()) |
1750 | + |
1751 | + # returns True when subprocess successful |
1752 | + self.helper.config["trusted_ssl_ca"] = base64.b64encode(str.encode(TEST_CERTIFICATE)) |
1753 | + self.assertTrue(self.helper.update_tls_certificates()) |
1754 | + with open(self.cert_path, "r") as f: |
1755 | + self.assertEqual(f.read(), TEST_CERTIFICATE) |
1756 | + mock_subprocess.assert_called_once_with(["/usr/sbin/update-ca-certificates"]) |
1757 | + mock_subprocess.reset_mock() |
1758 | + |
1759 | + # returns false when subprocess hits an exception |
1760 | + mock_subprocess.side_effect = CalledProcessError("Command", "Mock Subprocess Call Error") |
1761 | + self.assertFalse(self.helper.update_tls_certificates()) |
1762 | + |
1763 | + def test_render_checks(self): |
1764 | + """Test that NPRE is called to add KSC checks.""" |
1765 | + # TODO |
1766 | + pass |
1767 | + |
1768 | + @mock.patch("charmhelpers.fetch.snap.subprocess.check_call") |
1769 | + def test_install_kubectl(self, mock_snap_subprocess): |
1770 | + """Test install kubectl snap helper function.""" |
1771 | + self.assertTrue(self.helper.install_kubectl()) |
1772 | + channel = self.config.get("channel") |
1773 | + mock_snap_subprocess.assert_called_with( |
1774 | + ["snap", "install", "--classic", "--channel={}".format(channel), "kubectl"], env=os.environ |
1775 | + ) |
1776 | + |
1777 | + @mock.patch("charmhelpers.fetch.snap.subprocess.check_call") |
1778 | + def test_install_snap_failure(self, mock_snap_subprocess): |
1779 | + """Test response to a failed install event.""" |
1780 | + error = subprocess.CalledProcessError("cmd", "Install failed") |
1781 | + error.returncode = 1 |
1782 | + mock_snap_subprocess.return_value = 1 |
1783 | + mock_snap_subprocess.side_effect = error |
1784 | + self.assertFalse(self.helper.install_kubectl()) |
1785 | + |
1786 | + |
1787 | +if __name__ == "__main__": |
1788 | + unittest.main() |
1789 | diff --git a/tests/unit/test_plugins.py b/tests/unit/test_plugins.py |
1790 | new file mode 100644 |
1791 | index 0000000..42d1697 |
1792 | --- /dev/null |
1793 | +++ b/tests/unit/test_plugins.py |
1794 | @@ -0,0 +1,65 @@ |
1795 | +"""Unit tests for Kubernetes Service Checks NRPE Plugins.""" |
1796 | +import unittest |
1797 | + |
1798 | +import check_kubernetes_api |
1799 | +import mock |
1800 | + |
1801 | + |
1802 | +class TestKSCPlugins(unittest.TestCase): |
1803 | + """Test cases for Kubernetes Service Checks NRPE plugins.""" |
1804 | + |
1805 | + @mock.patch("check_kubernetes_api.sys.exit") |
1806 | + @mock.patch("check_kubernetes_api.print") |
1807 | + def test_nagios_exit(self, mock_print, mock_sys_exit): |
1808 | + """Test the nagios_exit function.""" |
1809 | + msg = "Test message" |
1810 | + for code, status in check_kubernetes_api.NAGIOS_STATUS.items(): |
1811 | + expected_output = "{}: {}".format(status, msg) |
1812 | + check_kubernetes_api.nagios_exit(code, msg) |
1813 | + |
1814 | + mock_print.assert_called_with(expected_output) |
1815 | + mock_sys_exit.assert_called_with(code) |
1816 | + |
1817 | + @mock.patch("check_kubernetes_api.urllib3.PoolManager") |
1818 | + def test_kubernetes_health_ssl(self, mock_http_pool_manager): |
1819 | + """Test the check k8s health function called with expected ssl params.""" |
1820 | + host_address = "https://1.1.1.1:1111" |
1821 | + token = "0123456789abcdef" |
1822 | + disable_ssl = True |
1823 | + |
1824 | + mock_http_pool_manager.return_value.status = 200 |
1825 | + mock_http_pool_manager.return_value.data = b"ok" |
1826 | + |
1827 | + check_kubernetes_api.check_kubernetes_health(host_address, token, disable_ssl) |
1828 | + mock_http_pool_manager.assert_called_with(cert_reqs="CERT_NONE", assert_hostname=False) |
1829 | + |
1830 | + disable_ssl = False |
1831 | + check_kubernetes_api.check_kubernetes_health(host_address, token, disable_ssl) |
1832 | + mock_http_pool_manager.assert_called_with() |
1833 | + |
1834 | + @mock.patch("check_kubernetes_api.urllib3.PoolManager") |
1835 | + def test_kubernetes_health_status(self, mock_http_pool_manager): |
1836 | + """Test kubernetes health function.""" |
1837 | + host_address = "https://1.1.1.1:1111" |
1838 | + token = "0123456789abcdef" |
1839 | + ssl_ca = "test/cert/path" |
1840 | + |
1841 | + mock_http_pool_manager.return_value.request.return_value.status = 200 |
1842 | + mock_http_pool_manager.return_value.request.return_value.data = b"ok" |
1843 | + |
1844 | + # verify status OK |
1845 | + status, _ = check_kubernetes_api.check_kubernetes_health(host_address, token, ssl_ca) |
1846 | + self.assertEqual(status, check_kubernetes_api.NAGIOS_STATUS_OK) |
1847 | + mock_http_pool_manager.return_value.request.assert_called_once_with( |
1848 | + "GET", "{}/healthz".format(host_address), headers={"Authorization": "Bearer {}".format(token)} |
1849 | + ) |
1850 | + |
1851 | + mock_http_pool_manager.return_value.request.return_value.status = 500 |
1852 | + mock_http_pool_manager.return_value.request.return_value.data = b"ok" |
1853 | + status, _ = check_kubernetes_api.check_kubernetes_health(host_address, token, ssl_ca) |
1854 | + self.assertEqual(status, check_kubernetes_api.NAGIOS_STATUS_CRITICAL) |
1855 | + |
1856 | + mock_http_pool_manager.return_value.request.return_value.status = 200 |
1857 | + mock_http_pool_manager.return_value.request.return_value.data = b"not ok" |
1858 | + status, _ = check_kubernetes_api.check_kubernetes_health(host_address, token, ssl_ca) |
1859 | + self.assertEqual(status, check_kubernetes_api.NAGIOS_STATUS_WARNING) |
1860 | diff --git a/tox.ini b/tox.ini |
1861 | new file mode 100644 |
1862 | index 0000000..2f92371 |
1863 | --- /dev/null |
1864 | +++ b/tox.ini |
1865 | @@ -0,0 +1,52 @@ |
1866 | +[tox] |
1867 | +skipsdist = True |
1868 | +envlist = unit, func |
1869 | +skip_missing_interpreters = True |
1870 | + |
1871 | +[testenv] |
1872 | +basepython = python3 |
1873 | +setenv = |
1874 | + PYTHONPATH = {toxinidir}/lib/:{toxinidir} |
1875 | +passenv = |
1876 | + HOME |
1877 | + MODEL_SETTINGS |
1878 | + CHARM_BUILD_DIR |
1879 | + |
1880 | +[testenv:unit] |
1881 | +commands = |
1882 | + coverage run -m unittest discover -s {toxinidir}/tests/unit -v |
1883 | + coverage report \ |
1884 | + --omit tests/*,mod/*,.tox/* |
1885 | + coverage html \ |
1886 | + --omit tests/*,mod/*,.tox/* |
1887 | +deps = -r{toxinidir}/tests/unit/requirements.txt |
1888 | + |
1889 | +[testenv:func] |
1890 | +changedir = {toxinidir}/tests/functional |
1891 | +commands = functest-run-suite {posargs} |
1892 | +deps = -r{toxinidir}/tests/functional/requirements.txt |
1893 | + |
1894 | +[testenv:lint] |
1895 | +commands = |
1896 | + flake8 |
1897 | + black --check --line-length 120 --exclude /(\.eggs|\.git|\.tox|\.venv|build|dist|charmhelpers|mod)/ . |
1898 | +deps = |
1899 | + black |
1900 | + flake8 |
1901 | + flake8-docstrings |
1902 | + flake8-import-order |
1903 | + pep8-naming |
1904 | + flake8-colors |
1905 | + |
1906 | +[flake8] |
1907 | +exclude = |
1908 | + .git, |
1909 | + __pycache__, |
1910 | + .tox, |
1911 | + mod, |
1912 | +max-line-length = 120 |
1913 | +max-complexity = 10 |
1914 | +import-order-style = google |
1915 | + |
1916 | +[isort] |
1917 | +force_to_top=setuppath |
Generally code looks good.
The only comment I have is on the self.state. configured. It is not changed to False when a config changed is happening or a relation has joined, changed or departed (it is set to false only on departed), while it should be to reflect that this state is controlling whether check_charm_status should reconfigure or not the unit.
Also the self.helper. configure( ) is called in check_charm_status without checking if this state is true or false, while seems that this should be called only if configured is false.
Same for self.helper. update_ tls_certificate s(). This is needed only on initial deploy (and configured is False at the beginning) or later if a config_changed has occurred as the certificate can be updated only on a config changed at this stage.
Maybe i am not reading this state correctly. In this case more information are needed.