Merge ~addyess/charm-openstack-service-checks:bugs/lp1887561-check_octavia_filtering into ~llama-charmers/charm-openstack-service-checks:master

Proposed by Adam Dyess on 2020-07-14
Status: Merged
Approved by: Chris Sanders on 2020-07-15
Approved revision: dcf043ac09c1dea8c1fa18b7f85fa0c559911fa4
Merged at revision: b4a24bff8ea1f9d38d70b1297ef64c63b49327e0
Proposed branch: ~addyess/charm-openstack-service-checks:bugs/lp1887561-check_octavia_filtering
Merge into: ~llama-charmers/charm-openstack-service-checks:master
Diff against target: 802 lines (+370/-138)
9 files modified
README.md (+41/-0)
config.yaml (+20/-0)
files/plugins/check_octavia.py (+137/-89)
lib/lib_openstack_service_checks.py (+37/-15)
tests/unit/conftest.py (+4/-11)
tests/unit/test_check_cinder_services.py (+1/-5)
tests/unit/test_check_contrail_analytics_alarms.py (+12/-14)
tests/unit/test_check_nova_services.py (+1/-4)
tests/unit/test_check_octavia.py (+117/-0)
Reviewer Review Type Date Requested Status
Chris Sanders 2020-07-14 Approve on 2020-07-15
Review via email: mp+387394@code.launchpad.net

Commit message

Support for filtered alarming of octavia checks

To post a comment you must log in.
Adam Dyess (addyess) wrote :

This change added an ignore-list of keywords for each of the 4 octavia checks: loadbalancers, amphora, pools, and images.

each keyword in the ignore list will be blocked when it appears in the output of the check_octavia. Presume that you have a test or non-production loadbalancer you do not want alert checks from with the ID=deadbeef-1234-56789012-dead-beef

You can use this config

juju config <openstack-service-checks-app> octavia-loadbalancer-ignored='deadbeef-1234-56789012-dead-beef,'

to ignore any checks associated with the loadbalancer such as it being inactive or degraded.

Alternatively, you could silence all degraded loadbalancer alerts with

juju config <openstack-service-checks-app> octavia-loadbalancer-ignored='DEGRADED,'

Chris Sanders (chris.sanders) wrote :

The commit message here is a very good example, can you add something to the Readme about this as well?

A few in-line comments/questions as well.

review: Needs Information
review: Approve
Chris Sanders (chris.sanders) wrote :

Let's see those tests ;)

review: Needs Information
Chris Sanders (chris.sanders) wrote :

Alright thanks, +1

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/README.md b/README.md
2index 54dceb0..f589fb9 100644
3--- a/README.md
4+++ b/README.md
5@@ -38,6 +38,47 @@ If such API endpoints use TLS, new checks will monitor the certificates expirati
6
7 Alternatively, instead of the above relation, there is also an action "refresh-endpoint-checks" available. Running this action will update the service checks with the current endpoints.
8
9+## Octavia Checks
10+
11+Knowning when an openstack load-balancer is having an issue is an important
12+operational situation which this charm helps manage. There is both course
13+grain control over octavia checks, as well as more fine-grained control by
14+use of the following config items.
15+
16+### Course Grain
17+
18+ * `check-octavia`: `true` or `false` can enable or disable checks
19+
20+### Fine Grain
21+
22+ * `octavia-loadbalancers-ignored`
23+ * `octavia-amphorae-ignored`
24+ * `octavia-pools-ignored`
25+ * `octavia-image-ignored`
26+
27+Each of these config items adds an ignore-list of keywords. Each keyword in
28+the ignore list will be blocked when it appears in the output of the check.
29+
30+#### Examples
31+
32+---------------
33+Ignoring a test or non-production loadbalancer with the ID=`deadbeef-1234
34+-56789012-dead-beef` which is __INACTIVE__ or __DEGRADED__.
35+```bash
36+juju config my-openstack-service-checks octavia-loadbalancer-ignored='deadbeef-1234-56789012-dead-beef,'
37+```
38+
39+Ignoring all loadbalancers which happen to be __DEGRADED__.
40+```bash
41+juju config my-openstack-service-checks octavia-loadbalancer-ignored='DEGRADED,'
42+```
43+
44+Ignoring amphorae that are stuck in __BOOTING__ state
45+```bash
46+juju config my-openstack-service-checks octavia-amphorae-ignored='BOOTING,'
47+```
48+
49+
50 ## Compute services monitoring
51
52 Compute services are monitored via the 'os-services' interface. Several thresholds can
53diff --git a/config.yaml b/config.yaml
54index 81428e5..d36f4a9 100644
55--- a/config.yaml
56+++ b/config.yaml
57@@ -15,6 +15,26 @@ options:
58 type: boolean
59 description: |
60 Switch to turn on or off check for octavia services.
61+ octavia-loadbalancers-ignored:
62+ type: string
63+ default: ""
64+ description: |
65+ Comma separated list of octavia load balancer alerts to ignore
66+ octavia-amphorae-ignored:
67+ type: string
68+ default: ""
69+ description: |
70+ Comma separated list of octavia amphorae alerts to ignore
71+ octavia-pools-ignored:
72+ type: string
73+ default: ""
74+ description: |
75+ Comma separated list of octavia pool alerts to ignore
76+ octavia-image-ignored:
77+ type: string
78+ default: ""
79+ description: |
80+ Comma separated list of octavia image alerts to ignore
81 octavia-amp-image-tag:
82 default: "octavia-amphora"
83 type: string
84diff --git a/files/plugins/check_octavia.py b/files/plugins/check_octavia.py
85index 09fb6b1..bb6debc 100755
86--- a/files/plugins/check_octavia.py
87+++ b/files/plugins/check_octavia.py
88@@ -1,13 +1,19 @@
89 #!/usr/bin/env python3
90
91-import os
92-import sys
93-import json
94 import argparse
95-import subprocess
96+import collections
97 from datetime import datetime, timedelta
98+import json
99+import os
100+import re
101+import subprocess
102+import sys
103+
104 import openstack
105
106+
107+Alarm = collections.namedtuple('Alarm', 'lvl, desc')
108+DEFAULT_IGNORED = r''
109 NAGIOS_STATUS_OK = 0
110 NAGIOS_STATUS_WARNING = 1
111 NAGIOS_STATUS_CRITICAL = 2
112@@ -21,12 +27,48 @@ NAGIOS_STATUS = {
113 }
114
115
116-def nagios_exit(status, message):
117+def filter_checks(alarms, ignored=DEFAULT_IGNORED):
118+ """
119+ Reduce all checks down to an overall check based on the highest level
120+ not ignored
121+
122+ :param List[Tuple] alarms: list of alarms (lvl, message)
123+ :param str ignored: regular expression of messages to ignore
124+ :return:
125+ """
126+ search_re = re.compile(ignored)
127+ full = [Alarm(lvl, msg) for lvl, msg in alarms]
128+ ignoring = list(filter(lambda m: search_re.search(m.desc), full)) if ignored else []
129+ important = set(full) - set(ignoring)
130+
131+ total_crit = len([a for a in full if a.lvl == NAGIOS_STATUS_CRITICAL])
132+ important_crit = len([a for a in important if a.lvl == NAGIOS_STATUS_CRITICAL])
133+ important_count = len(important)
134+ if important_crit > 0:
135+ status = NAGIOS_STATUS_CRITICAL
136+ elif important_count > 0:
137+ status = NAGIOS_STATUS_WARNING
138+ else:
139+ status = NAGIOS_STATUS_OK
140+ msg = (
141+ "total_alarms[{}], total_crit[{}], total_ignored[{}], "
142+ "ignoring r'{}'\n"
143+ .format(len(full), total_crit, len(ignoring), ignored)
144+ )
145+ msg += '\n'.join(_.desc for _ in sorted(important))
146+ return status, msg
147+
148+
149+def nagios_exit(args, results):
150+ # parse ignored list
151+ unique = sorted(filter(None, set(args.ignored.split(","))))
152+ ignored_re = r'|'.join('(?:{})'.format(_) for _ in unique)
153+
154+ status, message = filter_checks(results, ignored=ignored_re)
155 assert status in NAGIOS_STATUS, "Invalid Nagios status code"
156 # prefix status name to message
157 output = '{}: {}'.format(NAGIOS_STATUS[status], message)
158- print(output) # nagios requires print to stdout, no stderr
159- sys.exit(status)
160+ return status, output
161
162
163 def check_loadbalancers(connection):
164@@ -39,77 +81,72 @@ def check_loadbalancers(connection):
165 lb_enabled = [lb for lb in lb_all if lb.is_admin_state_up]
166
167 # check provisioning_status is ACTIVE for each lb
168- bad_lbs = [lb for lb in lb_enabled if lb.provisioning_status != 'ACTIVE']
169- if bad_lbs:
170- parts = ['loadbalancer {} provisioning_status is {}'.format(
171- lb.id, lb.provisioning_status) for lb in bad_lbs]
172- message = ', '.join(parts)
173- return NAGIOS_STATUS_CRITICAL, message
174+ bad_lbs = [(
175+ NAGIOS_STATUS_CRITICAL,
176+ 'loadbalancer {} provisioning_status is {}'.format(
177+ lb.id, lb.provisioning_status)
178+ ) for lb in lb_enabled if lb.provisioning_status != 'ACTIVE']
179
180 # raise WARNING if operating_status is not ONLINE
181- bad_lbs = [lb for lb in lb_enabled if lb.operating_status != 'ONLINE']
182- if bad_lbs:
183- parts = ['loadbalancer {} operating_status is {}'.format(
184- lb.id, lb.operating_status) for lb in bad_lbs]
185- message = ', '.join(parts)
186- return NAGIOS_STATUS_CRITICAL, message
187+ bad_lbs += [(
188+ NAGIOS_STATUS_CRITICAL,
189+ 'loadbalancer {} operating_status is {}'.format(
190+ lb.id, lb.operating_status)
191+ ) for lb in lb_enabled if lb.operating_status != 'ONLINE']
192
193- net_mgr = connection.network
194 # check vip port exists for each lb
195- bad_lbs = []
196+ net_mgr = connection.network
197+ vip_lbs = []
198 for lb in lb_enabled:
199 try:
200 net_mgr.get_port(lb.vip_port_id)
201 except openstack.exceptions.NotFoundException:
202- bad_lbs.append(lb)
203- if bad_lbs:
204- parts = ['vip port {} for loadbalancer {} not found'.format(
205- lb.vip_port_id, lb.id) for lb in bad_lbs]
206- message = ', '.join(parts)
207- return NAGIOS_STATUS_CRITICAL, message
208+ vip_lbs.append(lb)
209+ bad_lbs += [(
210+ NAGIOS_STATUS_CRITICAL,
211+ 'vip port {} for loadbalancer {} not found'.format(
212+ lb.vip_port_id, lb.id)
213+ ) for lb in vip_lbs]
214
215 # warn about disabled lbs if no other error found
216- lb_disabled = [lb for lb in lb_all if not lb.is_admin_state_up]
217- if lb_disabled:
218- parts = ['loadbalancer {} admin_state_up is False'.format(lb.id)
219- for lb in lb_disabled]
220- message = ', '.join(parts)
221- return NAGIOS_STATUS_WARNING, message
222+ bad_lbs += [(
223+ NAGIOS_STATUS_WARNING,
224+ 'loadbalancer {} admin_state_up is False'.format(lb.id)
225+ ) for lb in lb_all if not lb.is_admin_state_up]
226
227- return NAGIOS_STATUS_OK, 'loadbalancers are happy'
228+ return bad_lbs
229
230
231 def check_pools(connection):
232 """check pools status."""
233 lb_mgr = connection.load_balancer
234 pools_all = lb_mgr.pools()
235+
236+ # only check enabled pools
237 pools_enabled = [pool for pool in pools_all if pool.is_admin_state_up]
238
239 # check provisioning_status is ACTIVE for each pool
240- bad_pools = [pool for pool in pools_enabled if pool.provisioning_status != 'ACTIVE']
241- if bad_pools:
242- parts = ['pool {} provisioning_status is {}'.format(
243- pool.id, pool.provisioning_status) for pool in bad_pools]
244- message = ', '.join(parts)
245- return NAGIOS_STATUS_CRITICAL, message
246+ bad_pools = [(
247+ NAGIOS_STATUS_CRITICAL,
248+ 'pool {} provisioning_status is {}'.format(
249+ pool.id, pool.provisioning_status)
250+ ) for pool in pools_enabled if pool.provisioning_status != 'ACTIVE']
251
252 # raise CRITICAL if operating_status is ERROR
253- bad_pools = [pool for pool in pools_enabled if pool.operating_status == 'ERROR']
254- if bad_pools:
255- parts = ['pool {} operating_status is {}'.format(
256- pool.id, pool.operating_status) for pool in bad_pools]
257- message = ', '.join(parts)
258- return NAGIOS_STATUS_CRITICAL, message
259+ bad_pools += [(
260+ NAGIOS_STATUS_CRITICAL,
261+ 'pool {} operating_status is {}'.format(
262+ pool.id, pool.operating_status)
263+ ) for pool in pools_enabled if pool.operating_status == 'ERROR']
264
265 # raise WARNING if operating_status is NO_MONITOR
266- bad_pools = [pool for pool in pools_enabled if pool.operating_status == 'NO_MONITOR']
267- if bad_pools:
268- parts = ['pool {} operating_status is {}'.format(
269- pool.id, pool.operating_status) for pool in bad_pools]
270- message = ', '.join(parts)
271- return NAGIOS_STATUS_WARNING, message
272+ bad_pools += [(
273+ NAGIOS_STATUS_WARNING,
274+ 'pool {} operating_status is {}'.format(
275+ pool.id, pool.operating_status)
276+ ) for pool in pools_enabled if pool.operating_status == 'NO_MONITOR']
277
278- return NAGIOS_STATUS_OK, 'pools are happy'
279+ return bad_pools
280
281
282 def check_amphorae(connection):
283@@ -120,7 +157,7 @@ def check_amphorae(connection):
284 resp = lb_mgr.get('/v2/octavia/amphorae')
285 # python api is not available yet, use url
286 if resp.status_code != 200:
287- return NAGIOS_STATUS_WARNING, 'amphorae api not working'
288+ return [(NAGIOS_STATUS_WARNING, 'amphorae api not working')]
289
290 data = json.loads(resp.content)
291 # ouput is like {"amphorae": [{...}, {...}, ...]}
292@@ -128,26 +165,20 @@ def check_amphorae(connection):
293
294 # raise CRITICAL for ERROR status
295 bad_status_list = ('ERROR',)
296- bad_items = [item for item in items if item['status'] in bad_status_list]
297- if bad_items:
298- parts = [
299- 'amphora {} status is {}'.format(item['id'], item['status'])
300- for item in bad_items]
301- message = ', '.join(parts)
302- return NAGIOS_STATUS_CRITICAL, message
303+ bad_amp = [(
304+ NAGIOS_STATUS_CRITICAL,
305+ 'amphora {} status is {}'.format(item['id'], item['status'])
306+ ) for item in items if item['status'] in bad_status_list]
307
308 # raise WARNING for these status
309 bad_status_list = (
310 'PENDING_CREATE', 'PENDING_UPDATE', 'PENDING_DELETE', 'BOOTING')
311- bad_items = [item for item in items if item['status'] in bad_status_list]
312- if bad_items:
313- parts = [
314- 'amphora {} status is {}'.format(item['id'], item['status'])
315- for item in bad_items]
316- message = ', '.join(parts)
317- return NAGIOS_STATUS_WARNING, message
318+ bad_amp += [(
319+ NAGIOS_STATUS_WARNING,
320+ 'amphora {} status is {}'.format(item['id'], item['status'])
321+ ) for item in items if item['status'] in bad_status_list]
322
323- return NAGIOS_STATUS_OK, 'amphorae are happy'
324+ return bad_amp
325
326
327 def check_image(connection, tag, days):
328@@ -157,28 +188,47 @@ def check_image(connection, tag, days):
329 if not images:
330 message = ('Octavia requires image with tag {} to create amphora, '
331 'but none exist').format(tag)
332- return NAGIOS_STATUS_CRITICAL, message
333+ return [(NAGIOS_STATUS_CRITICAL, message)]
334
335 active_images = [image for image in images if image.status == 'active']
336 if not active_images:
337- parts = ['{}({})'.format(image.name, image.id) for image in images]
338+ details = ['{}({})'.format(image.name, image.id) for image in images]
339 message = ('Octavia requires image with tag {} to create amphora, '
340- 'but none is active: {}').format(tag, ', '.join(parts))
341- return NAGIOS_STATUS_CRITICAL, message
342+ 'but none are active: {}').format(tag, ', '.join(details))
343+ return [(NAGIOS_STATUS_CRITICAL, message)]
344
345 # raise WARNING if image is too old
346 when = (datetime.now() - timedelta(days=days)).isoformat()
347 # updated_at str format: '2019-12-05T18:21:25Z'
348 fresh_images = [image for image in active_images if image.updated_at > when]
349 if not fresh_images:
350+ details = ['{}({})'.format(image.name, image.id) for image in images]
351 message = ('Octavia requires image with tag {} to create amphora, '
352- 'but it is older than {} days').format(tag, days)
353- return NAGIOS_STATUS_WARNING, message
354+ 'but all images are older than {} day(s): {}'
355+ '').format(tag, days, ', '.join(details))
356+ return [(NAGIOS_STATUS_WARNING, message)]
357
358- return NAGIOS_STATUS_OK, 'image is ready'
359+ return []
360
361
362-if __name__ == '__main__':
363+def process_checks(args):
364+ # use closure to make all checks have same signature
365+ # so we can handle them in same way
366+ def _check_image(_connection):
367+ return check_image(_connection, args.amp_image_tag, args.amp_image_days)
368+
369+ checks = {
370+ 'loadbalancers': check_loadbalancers,
371+ 'amphorae': check_amphorae,
372+ 'pools': check_pools,
373+ 'image': _check_image,
374+ }
375+
376+ connection = openstack.connect(cloud='envvars')
377+ return nagios_exit(args, checks[args.check](connection))
378+
379+
380+def main():
381 parser = argparse.ArgumentParser(
382 description='Check Octavia status',
383 formatter_class=argparse.ArgumentDefaultsHelpFormatter,
384@@ -195,6 +245,11 @@ if __name__ == '__main__':
385 help='which check to run')
386
387 parser.add_argument(
388+ '--ignored', dest="ignored", type=str,
389+ default=DEFAULT_IGNORED,
390+ help='Comma separated list of alerts to ignore')
391+
392+ parser.add_argument(
393 '--amp-image-tag', dest='amp_image_tag', default='octavia-amphora',
394 help='amphora image tag for image check')
395
396@@ -211,17 +266,10 @@ if __name__ == '__main__':
397 os.environ[key.decode('utf-8')] = value.rstrip().decode('utf-8')
398 proc.communicate()
399
400- # use closure to make all checks have same signature
401- # so we can handle them in same way
402- def _check_image(connection):
403- return check_image(connection, args.amp_image_tag, args.amp_image_days)
404+ status, message = process_checks(args)
405+ print(message)
406+ sys.exit(status)
407
408- checks = {
409- 'loadbalancers': check_loadbalancers,
410- 'amphorae': check_amphorae,
411- 'pools': check_pools,
412- 'image': _check_image,
413- }
414
415- connection = openstack.connect(cloud='envvars')
416- nagios_exit(*checks[args.check](connection))
417+if __name__ == '__main__':
418+ main()
419diff --git a/lib/lib_openstack_service_checks.py b/lib/lib_openstack_service_checks.py
420index 5e82395..c74d61d 100644
421--- a/lib/lib_openstack_service_checks.py
422+++ b/lib/lib_openstack_service_checks.py
423@@ -202,16 +202,8 @@ class OSCHelper():
424 charm_plugin_dir = os.path.join(hookenv.charm_dir(), 'files', 'plugins/')
425 host.rsync(charm_plugin_dir, self.plugins_dir, options=['--executability'])
426
427- def render_checks(self, creds):
428- render(source='nagios.novarc', target=self.novarc, context=creds,
429- owner='nagios', group='nagios')
430-
431- nrpe = NRPE()
432- if not os.path.exists(self.plugins_dir):
433- os.makedirs(self.plugins_dir)
434-
435- self.update_plugins()
436- # Nova services health
437+ def _render_nova_checks(self, nrpe):
438+ """Nova services health."""
439 nova_check_command = os.path.join(self.plugins_dir, 'check_nova_services.py')
440 check_command = '{} --warn {} --crit {} --skip-aggregates {} {}'.format(
441 nova_check_command, self.nova_warn, self.nova_crit, self.nova_skip_aggregates,
442@@ -221,7 +213,8 @@ class OSCHelper():
443 check_cmd=check_command,
444 )
445
446- # Neutron agents health
447+ def _render_neutron_checks(self, nrpe):
448+ """Neutron agents health."""
449 if self.is_neutron_agents_check_enabled:
450 nrpe.add_check(shortname='neutron_agents',
451 description='Check that enabled Neutron agents are up',
452@@ -231,6 +224,7 @@ class OSCHelper():
453 else:
454 nrpe.remove_check(shortname='neutron_agents')
455
456+ def _render_cinder_checks(self, nrpe):
457 # Cinder services health
458 cinder_check_command = os.path.join(self.plugins_dir, 'check_cinder_services.py')
459 check_command = '{} {}'.format(cinder_check_command, self.skip_disabled)
460@@ -239,6 +233,7 @@ class OSCHelper():
461 check_cmd=check_command,
462 )
463
464+ def _render_octavia_checks(self, nrpe):
465 # only care about octavia after 18.04
466 if host.lsb_release()['DISTRIB_RELEASE'] >= '18.04':
467 if self.is_octavia_check_enabled:
468@@ -246,24 +241,34 @@ class OSCHelper():
469 script = os.path.join(self.plugins_dir, 'check_octavia.py')
470
471 for check in ('loadbalancers', 'amphorae', 'pools'):
472+ check_cmd = '{} --check {}'.format(script, check)
473+ ignore = self.charm_config.get('octavia-%s-ignored' % check)
474+ if ignore:
475+ check_cmd += ' --ignored {}'.format(ignore)
476 nrpe.add_check(
477 shortname='octavia_{}'.format(check),
478 description='Check octavia {} status'.format(check),
479- check_cmd='{} --check {}'.format(script, check),
480+ check_cmd=check_cmd,
481 )
482
483 # image check has extra args, add it separately
484 check = 'image'
485+ check_cmd = "{} --check {}".format(script, check)
486+ check_cmd += " --amp-image-tag {}".format(self.octavia_amp_image_tag)
487+ check_cmd += " --amp-image-days {}".format(self.octavia_amp_image_days)
488+ ignore = self.charm_config.get('octavia-%s-ignored' % check)
489+ if ignore:
490+ check_cmd += " --ignored {}".format(ignore)
491 nrpe.add_check(
492 shortname='octavia_{}'.format(check),
493 description='Check octavia {} status'.format(check),
494- check_cmd='{} --check {} --amp-image-tag {} --amp-image-days {}'.format(
495- script, check, self.octavia_amp_image_tag, self.octavia_amp_image_days),
496+ check_cmd=check_cmd,
497 )
498 else:
499 for check in ('loadbalancers', 'amphorae', 'pools', 'image'):
500 nrpe.remove_check(shortname='octavia_{}'.format(check))
501
502+ def _render_contrail_checks(self, nrpe):
503 if self.contrail_analytics_vip:
504 contrail_check_command = '{} --host {}'.format(
505 os.path.join(self.plugins_dir, 'check_contrail_analytics_alarms.py'),
506@@ -279,6 +284,7 @@ class OSCHelper():
507 else:
508 nrpe.remove_check(shortname='contrail_analytics_alarms')
509
510+ def _render_dns_checks(self, nrpe):
511 if len(self.check_dns):
512 nrpe.add_check(shortname='dns_multi',
513 description='Check DNS names are resolvable',
514@@ -289,8 +295,24 @@ class OSCHelper():
515 )
516 else:
517 nrpe.remove_check(shortname='dns_multi')
518- nrpe.write()
519
520+ def render_checks(self, creds):
521+ render(source='nagios.novarc', target=self.novarc, context=creds,
522+ owner='nagios', group='nagios')
523+
524+ nrpe = NRPE()
525+ if not os.path.exists(self.plugins_dir):
526+ os.makedirs(self.plugins_dir)
527+
528+ self.update_plugins()
529+ self._render_nova_checks(nrpe)
530+ self._render_neutron_checks(nrpe)
531+ self._render_cinder_checks(nrpe)
532+ self._render_octavia_checks(nrpe)
533+ self._render_contrail_checks(nrpe)
534+ self._render_dns_checks(nrpe)
535+
536+ nrpe.write()
537 self.create_endpoint_checks(creds)
538
539 def _split_url(self, netloc, scheme):
540diff --git a/tests/unit/conftest.py b/tests/unit/conftest.py
541index 6797e85..639b91a 100644
542--- a/tests/unit/conftest.py
543+++ b/tests/unit/conftest.py
544@@ -4,6 +4,10 @@ import sys
545
546 import pytest
547
548+TEST_DIR = os.path.dirname(__file__)
549+CHECKS_DIR = os.path.join(TEST_DIR, '..', '..', 'files', 'plugins')
550+sys.path.append(CHECKS_DIR)
551+
552
553 # If layer options are used, add this to openstackservicechecks
554 # and import layer in lib_openstack_service_checks
555@@ -77,14 +81,3 @@ def openstackservicechecks(tmpdir, mock_hookenv_config, mock_charm_dir, monkeypa
556 monkeypatch.setattr('lib_openstack_service_checks.OSCHelper', lambda: helper)
557
558 return helper
559-
560-
561-@pytest.fixture(scope='module')
562-def check_contrail_analytics():
563- pre = sys.path
564- TEST_DIR = os.path.dirname(__file__)
565- tests_dir = os.path.join(TEST_DIR, '..', '..', 'files', 'plugins')
566- sys.path.append(tests_dir)
567- import check_contrail_analytics_alarms as checks # noqa
568- yield checks
569- sys.path = pre
570diff --git a/tests/unit/test_check_cinder_services.py b/tests/unit/test_check_cinder_services.py
571index 709b4dc..3428dd6 100644
572--- a/tests/unit/test_check_cinder_services.py
573+++ b/tests/unit/test_check_cinder_services.py
574@@ -1,11 +1,7 @@
575 import pytest
576 import nagios_plugin3
577
578-import sys
579-
580-sys.path.append("files/plugins")
581-
582-import check_cinder_services # noqa: E402
583+import check_cinder_services
584
585
586 @pytest.mark.parametrize(
587diff --git a/tests/unit/test_check_contrail_analytics_alarms.py b/tests/unit/test_check_contrail_analytics_alarms.py
588index c7fd6e9..886a396 100644
589--- a/tests/unit/test_check_contrail_analytics_alarms.py
590+++ b/tests/unit/test_check_contrail_analytics_alarms.py
591@@ -1,14 +1,15 @@
592 import json
593 import os
594
595+import check_contrail_analytics_alarms
596+
597 TEST_DIR = os.path.dirname(__file__)
598
599
600-def test_parse_contrail_alarms(check_contrail_analytics):
601+def test_parse_contrail_alarms():
602 with open(os.path.join(TEST_DIR, 'contrail_alert_data.json')) as f:
603 data = json.load(f)
604- assert hasattr(check_contrail_analytics, 'parse_contrail_alarms')
605- parsed = check_contrail_analytics.parse_contrail_alarms(data)
606+ parsed = check_contrail_analytics_alarms.parse_contrail_alarms(data)
607 assert parsed in """
608 CRITICAL: total_alarms[11], unacked_or_sev_gt_0[10], total_ignored[0], ignoring r''
609 CRITICAL: vrouter{compute-10.maas, sev=1, ts[2020-06-25 18:29:23.149146]} Vrouter interface(s) down.
610@@ -25,12 +26,11 @@ CRITICAL: vrouter{compute-7.maas, sev=1, ts[2020-07-03 18:30:32.481386]} Vrouter
611 """ # noqa: ignore=F501
612
613
614-def test_parse_contrail_alarms_filter_vrouter_control_9(check_contrail_analytics):
615+def test_parse_contrail_alarms_filter_vrouter_control_9():
616 with open(os.path.join(TEST_DIR, 'contrail_alert_data.json')) as f:
617 data = json.load(f)
618- assert hasattr(check_contrail_analytics, 'parse_contrail_alarms')
619 ignored_re = r'(?:vrouter)|(?:control-9)'
620- parsed = check_contrail_analytics.parse_contrail_alarms(data, ignored=ignored_re)
621+ parsed = check_contrail_analytics_alarms.parse_contrail_alarms(data, ignored=ignored_re)
622 assert parsed in """
623 CRITICAL: total_alarms[11], unacked_or_sev_gt_0[10], total_ignored[8], ignoring r'(?:vrouter)|(?:control-9)'
624 WARNING: control-node{control-8-contrail-rmq, sev=0, ts[2020-06-25 18:29:23.684803]} Node Failure. NodeStatus UVE not present.
625@@ -39,32 +39,30 @@ CRITICAL: control-node{control-7-contrail-rmq, sev=1, ts[2020-06-25 18:29:24.377
626 """ # noqa: ignore=F501
627
628
629-def test_parse_contrail_alarms_filter_critical(check_contrail_analytics):
630+def test_parse_contrail_alarms_filter_critical():
631 with open(os.path.join(TEST_DIR, 'contrail_alert_data.json')) as f:
632 data = json.load(f)
633- assert hasattr(check_contrail_analytics, 'parse_contrail_alarms')
634 ignored_re = r'(?:CRITICAL)'
635- parsed = check_contrail_analytics.parse_contrail_alarms(data, ignored=ignored_re)
636+ parsed = check_contrail_analytics_alarms.parse_contrail_alarms(data, ignored=ignored_re)
637 assert parsed in """
638 WARNING: total_alarms[11], unacked_or_sev_gt_0[10], total_ignored[10], ignoring r'(?:CRITICAL)'
639 WARNING: control-node{control-8-contrail-rmq, sev=0, ts[2020-06-25 18:29:23.684803]} Node Failure. NodeStatus UVE not present.
640 """ # noqa: ignore=F501
641
642
643-def test_parse_contrail_alarms_all_ignored(check_contrail_analytics):
644+def test_parse_contrail_alarms_all_ignored():
645 with open(os.path.join(TEST_DIR, 'contrail_alert_data.json')) as f:
646 data = json.load(f)
647- assert hasattr(check_contrail_analytics, 'parse_contrail_alarms')
648 ignored_re = r'(?:CRITICAL)|(?:WARNING)'
649- parsed = check_contrail_analytics.parse_contrail_alarms(data, ignored=ignored_re)
650+ parsed = check_contrail_analytics_alarms.parse_contrail_alarms(data, ignored=ignored_re)
651 assert parsed in """
652 OK: total_alarms[11], unacked_or_sev_gt_0[10], total_ignored[11], ignoring r'(?:CRITICAL)|(?:WARNING)'
653 """ # noqa: ignore=F501
654
655
656-def test_parse_contrail_alarms_no_alarms(check_contrail_analytics):
657+def test_parse_contrail_alarms_no_alarms():
658 ignored_re = r''
659- parsed = check_contrail_analytics.parse_contrail_alarms({}, ignored=ignored_re)
660+ parsed = check_contrail_analytics_alarms.parse_contrail_alarms({}, ignored=ignored_re)
661 assert parsed in """
662 OK: total_alarms[0], unacked_or_sev_gt_0[0], total_ignored[0], ignoring r''
663 """
664diff --git a/tests/unit/test_check_nova_services.py b/tests/unit/test_check_nova_services.py
665index 10c13ed..dad32a6 100644
666--- a/tests/unit/test_check_nova_services.py
667+++ b/tests/unit/test_check_nova_services.py
668@@ -1,10 +1,7 @@
669 import pytest
670 import nagios_plugin3
671
672-import sys
673-sys.path.append('files/plugins')
674-
675-import check_nova_services # noqa: E402
676+import check_nova_services
677
678
679 @pytest.mark.parametrize('is_skip_disabled,num_nodes',
680diff --git a/tests/unit/test_check_octavia.py b/tests/unit/test_check_octavia.py
681new file mode 100644
682index 0000000..08c9357
683--- /dev/null
684+++ b/tests/unit/test_check_octavia.py
685@@ -0,0 +1,117 @@
686+from datetime import datetime, timedelta
687+import json
688+import unittest.mock as mock
689+from uuid import uuid4
690+
691+import check_octavia
692+import pytest
693+
694+
695+@mock.patch('check_octavia.openstack.connect')
696+@pytest.mark.parametrize('check', [
697+ 'loadbalancers', 'pools', "amphorae", "image"
698+])
699+def test_stable_alarms(connect, check):
700+ args = mock.MagicMock()
701+ args.ignored = r''
702+ args.check = check
703+ if check == "amphorae":
704+ # Present 0 Amphora instances
705+ resp = connect().load_balancer.get()
706+ resp.status_code = 200
707+ resp.content = json.dumps({'amphora': []})
708+ elif check == "image":
709+ # Present 1 Active Fresh Amphora image
710+ args.amp_image_tag = 'octavia'
711+ args.amp_image_days = 1
712+ amp_image = mock.MagicMock()
713+ amp_image.status = 'active'
714+ amp_image.updated_at = datetime.now().isoformat()
715+ connect().image.images.return_value = [amp_image]
716+
717+ status, message = check_octavia.process_checks(args)
718+ assert message in """
719+OK: total_alarms[0], total_crit[0], total_ignored[0], ignoring r''
720+"""
721+ assert status == check_octavia.NAGIOS_STATUS_OK
722+
723+
724+@mock.patch('check_octavia.openstack.connect')
725+def test_no_images_is_ignorable(connect):
726+ args = mock.MagicMock()
727+ args.ignored = 'none exist'
728+ args.check = "image"
729+ # Present 1 Active Fresh Amphora image
730+ args.amp_image_tag = 'octavia'
731+ args.amp_image_days = 1
732+ connect().image.images.return_value = []
733+
734+ status, message = check_octavia.process_checks(args)
735+ assert message in """
736+OK: total_alarms[1], total_crit[1], total_ignored[1], ignoring r'(?:none exist)'
737+"""
738+ assert status == check_octavia.NAGIOS_STATUS_OK
739+
740+
741+@mock.patch('check_octavia.openstack.connect')
742+def test_no_images(connect):
743+ args = mock.MagicMock()
744+ args.ignored = r''
745+ args.check = "image"
746+ # Present 1 Active Fresh Amphora image
747+ args.amp_image_tag = 'octavia'
748+ args.amp_image_days = 1
749+ connect().image.images.return_value = []
750+
751+ status, message = check_octavia.process_checks(args)
752+ assert message in """
753+CRITICAL: total_alarms[1], total_crit[1], total_ignored[0], ignoring r''
754+Octavia requires image with tag octavia to create amphora, but none exist
755+"""
756+ assert status == check_octavia.NAGIOS_STATUS_CRITICAL
757+
758+
759+@mock.patch('check_octavia.openstack.connect')
760+def test_no_active_images(connect):
761+ args = mock.MagicMock()
762+ args.ignored = r''
763+ args.check = "image"
764+ # Present 1 Active Fresh Amphora image
765+ args.amp_image_tag = 'octavia'
766+ args.amp_image_days = 1
767+ amp_image = mock.MagicMock()
768+ amp_image.name = "bob-the-image"
769+ amp_image.id = str(uuid4())
770+ amp_image.status = 'inactive'
771+ amp_image.updated_at = datetime.now().isoformat()
772+ connect().image.images.return_value = [amp_image]
773+
774+ status, message = check_octavia.process_checks(args)
775+ assert message in """
776+CRITICAL: total_alarms[1], total_crit[1], total_ignored[0], ignoring r''
777+Octavia requires image with tag octavia to create amphora, but none are active: bob-the-image({})
778+""".format(amp_image.id)
779+ assert status == check_octavia.NAGIOS_STATUS_CRITICAL
780+
781+
782+@mock.patch('check_octavia.openstack.connect')
783+def test_no_fresh_images(connect):
784+ args = mock.MagicMock()
785+ args.ignored = r''
786+ args.check = "image"
787+ # Present 1 Active Fresh Amphora image
788+ args.amp_image_tag = 'octavia'
789+ args.amp_image_days = 1
790+ amp_image = mock.MagicMock()
791+ amp_image.name = "bob-the-image"
792+ amp_image.id = str(uuid4())
793+ amp_image.status = 'active'
794+ amp_image.updated_at = (datetime.now() - timedelta(days=2)).isoformat()
795+ connect().image.images.return_value = [amp_image]
796+
797+ status, message = check_octavia.process_checks(args)
798+ assert message in """
799+WARNING: total_alarms[1], total_crit[0], total_ignored[0], ignoring r''
800+Octavia requires image with tag octavia to create amphora, but all images are older than 1 day(s): bob-the-image({})
801+""".format(amp_image.id)
802+ assert status == check_octavia.NAGIOS_STATUS_WARNING

Subscribers

People subscribed via source and target branches