Merge ~addyess/charm-openstack-service-checks:bugs/lp1887561-check_octavia_filtering into ~llama-charmers/charm-openstack-service-checks:master

[Adam Dyess]

Support for filtered alarming of octavia checks

[Adam Dyess (addyess)]

This change added an ignore-list of keywords for each of the 4 octavia checks: loadbalancers, amphora, pools, and images.

each keyword in the ignore list will be blocked when it appears in the output of the check_octavia. Presume that you have a test or non-production loadbalancer you do not want alert checks from with the ID=deadbeef-1234-56789012-dead-beef

You can use this config

juju config <openstack-service-checks-app> octavia-loadbalancer-ignored='deadbeef-1234-56789012-dead-beef,'

to ignore any checks associated with the loadbalancer such as it being inactive or degraded.

Alternatively, you could silence all degraded loadbalancer alerts with

juju config <openstack-service-checks-app> octavia-loadbalancer-ignored='DEGRADED,'

[Chris Sanders (chris.sanders)]

The commit message here is a very good example, can you add something to the Readme about this as well?

A few in-line comments/questions as well.

review: Needs Information

[Chris Sanders (chris.sanders)]

review: Approve

[Chris Sanders (chris.sanders)]

Let's see those tests ;)

review: Needs Information

[Adam Dyess (addyess)]

CI Results: https://paste.ubuntu.com/p/btR4bqchSG/

[Chris Sanders (chris.sanders)]

Alright thanks, +1

review: Approve

diff --git a/README.md b/README.md
index 54dceb0..f589fb9 100644
--- a/README.md
+++ b/README.md
@@ -38,6 +38,47 @@ If such API endpoints use TLS, new checks will monitor the certificates expirati
 
 Alternatively, instead of the above relation, there is also an action "refresh-endpoint-checks" available. Running this action will update the service checks with the current endpoints.
 
+## Octavia Checks
+
+Knowning when an openstack load-balancer is having an issue is an important
+operational situation which this charm helps manage.  There is both course
+grain control over octavia checks, as well as more fine-grained control by
+use of the following config items.
+
+### Course Grain
+
+  * `check-octavia`: `true` or `false` can enable or disable checks
+  
+### Fine Grain
+
+  * `octavia-loadbalancers-ignored`
+  * `octavia-amphorae-ignored`
+  * `octavia-pools-ignored`
+  * `octavia-image-ignored`
+
+Each of these config items adds an ignore-list of keywords. Each keyword in
+the ignore list will be blocked when it appears in the output of the check. 
+
+#### Examples
+
+---------------
+Ignoring a test or non-production loadbalancer with the ID=`deadbeef-1234
+-56789012-dead-beef` which is __INACTIVE__ or __DEGRADED__.
+```bash
+juju config my-openstack-service-checks octavia-loadbalancer-ignored='deadbeef-1234-56789012-dead-beef,'
+```
+
+Ignoring all loadbalancers which happen to be __DEGRADED__.
+```bash
+juju config my-openstack-service-checks octavia-loadbalancer-ignored='DEGRADED,'
+```
+
+Ignoring amphorae that are stuck in __BOOTING__ state
+```bash
+juju config my-openstack-service-checks octavia-amphorae-ignored='BOOTING,'
+```
+
+
 ## Compute services monitoring
 
 Compute services are monitored via the 'os-services' interface. Several thresholds can
diff --git a/config.yaml b/config.yaml
index 81428e5..d36f4a9 100644
--- a/config.yaml
+++ b/config.yaml
@@ -15,6 +15,26 @@ options:
     type: boolean
     description: |
       Switch to turn on or off check for octavia services.
+  octavia-loadbalancers-ignored:
+    type: string
+    default: ""
+    description: |
+      Comma separated list of octavia load balancer alerts to ignore
+  octavia-amphorae-ignored:
+    type: string
+    default: ""
+    description: |
+      Comma separated list of octavia amphorae alerts to ignore
+  octavia-pools-ignored:
+    type: string
+    default: ""
+    description: |
+      Comma separated list of octavia pool alerts to ignore
+  octavia-image-ignored:
+    type: string
+    default: ""
+    description: |
+      Comma separated list of octavia image alerts to ignore
   octavia-amp-image-tag:
     default: "octavia-amphora"
     type: string
diff --git a/files/plugins/check_octavia.py b/files/plugins/check_octavia.py
index 09fb6b1..bb6debc 100755
--- a/files/plugins/check_octavia.py
+++ b/files/plugins/check_octavia.py
@@ -1,13 +1,19 @@
 #!/usr/bin/env python3
 
-import os
-import sys
-import json
 import argparse
-import subprocess
+import collections
 from datetime import datetime, timedelta
+import json
+import os
+import re
+import subprocess
+import sys
+
 import openstack
 
+
+Alarm = collections.namedtuple('Alarm', 'lvl, desc')
+DEFAULT_IGNORED = r''
 NAGIOS_STATUS_OK = 0
 NAGIOS_STATUS_WARNING = 1
 NAGIOS_STATUS_CRITICAL = 2
@@ -21,12 +27,48 @@ NAGIOS_STATUS = {
 }
 
 
-def nagios_exit(status, message):
+def filter_checks(alarms, ignored=DEFAULT_IGNORED):
+    """
+    Reduce all checks down to an overall check based on the highest level
+    not ignored
+
+    :param List[Tuple] alarms: list of alarms (lvl, message)
+    :param str ignored:        regular expression of messages to ignore
+    :return:
+    """
+    search_re = re.compile(ignored)
+    full = [Alarm(lvl, msg) for lvl, msg in alarms]
+    ignoring = list(filter(lambda m: search_re.search(m.desc), full)) if ignored else []
+    important = set(full) - set(ignoring)
+
+    total_crit = len([a for a in full if a.lvl == NAGIOS_STATUS_CRITICAL])
+    important_crit = len([a for a in important if a.lvl == NAGIOS_STATUS_CRITICAL])
+    important_count = len(important)
+    if important_crit > 0:
+        status = NAGIOS_STATUS_CRITICAL
+    elif important_count > 0:
+        status = NAGIOS_STATUS_WARNING
+    else:
+        status = NAGIOS_STATUS_OK
+    msg = (
+        "total_alarms[{}], total_crit[{}], total_ignored[{}], "
+        "ignoring r'{}'\n"
+        .format(len(full), total_crit, len(ignoring), ignored)
+    )
+    msg += '\n'.join(_.desc for _ in sorted(important))
+    return status, msg
+
+
+def nagios_exit(args, results):
+    # parse ignored list
+    unique = sorted(filter(None, set(args.ignored.split(","))))
+    ignored_re = r'|'.join('(?:{})'.format(_) for _ in unique)
+
+    status, message = filter_checks(results, ignored=ignored_re)
     assert status in NAGIOS_STATUS, "Invalid Nagios status code"
     # prefix status name to message
     output = '{}: {}'.format(NAGIOS_STATUS[status], message)
-    print(output)  # nagios requires print to stdout, no stderr
-    sys.exit(status)
+    return status, output
 
 
 def check_loadbalancers(connection):
@@ -39,77 +81,72 @@ def check_loadbalancers(connection):
     lb_enabled = [lb for lb in lb_all if lb.is_admin_state_up]
 
     # check provisioning_status is ACTIVE for each lb
-    bad_lbs = [lb for lb in lb_enabled if lb.provisioning_status != 'ACTIVE']
-    if bad_lbs:
-        parts = ['loadbalancer {} provisioning_status is {}'.format(
-            lb.id, lb.provisioning_status) for lb in bad_lbs]
-        message = ', '.join(parts)
-        return NAGIOS_STATUS_CRITICAL, message
+    bad_lbs = [(
+        NAGIOS_STATUS_CRITICAL,
+        'loadbalancer {} provisioning_status is {}'.format(
+            lb.id, lb.provisioning_status)
+        ) for lb in lb_enabled if lb.provisioning_status != 'ACTIVE']
 
     # raise WARNING if operating_status is not ONLINE
-    bad_lbs = [lb for lb in lb_enabled if lb.operating_status != 'ONLINE']
-    if bad_lbs:
-        parts = ['loadbalancer {} operating_status is {}'.format(
-            lb.id, lb.operating_status) for lb in bad_lbs]
-        message = ', '.join(parts)
-        return NAGIOS_STATUS_CRITICAL, message
+    bad_lbs += [(
+        NAGIOS_STATUS_CRITICAL,
+        'loadbalancer {} operating_status is {}'.format(
+            lb.id, lb.operating_status)
+        ) for lb in lb_enabled if lb.operating_status != 'ONLINE']
 
-    net_mgr = connection.network
     # check vip port exists for each lb
-    bad_lbs = []
+    net_mgr = connection.network
+    vip_lbs = []
     for lb in lb_enabled:
         try:
             net_mgr.get_port(lb.vip_port_id)
         except openstack.exceptions.NotFoundException:
-            bad_lbs.append(lb)
-    if bad_lbs:
-        parts = ['vip port {} for loadbalancer {} not found'.format(
-            lb.vip_port_id, lb.id) for lb in bad_lbs]
-        message = ', '.join(parts)
-        return NAGIOS_STATUS_CRITICAL, message
+            vip_lbs.append(lb)
+    bad_lbs += [(
+        NAGIOS_STATUS_CRITICAL,
+        'vip port {} for loadbalancer {} not found'.format(
+            lb.vip_port_id, lb.id)
+        ) for lb in vip_lbs]
 
     # warn about disabled lbs if no other error found
-    lb_disabled = [lb for lb in lb_all if not lb.is_admin_state_up]
-    if lb_disabled:
-        parts = ['loadbalancer {} admin_state_up is False'.format(lb.id)
-                 for lb in lb_disabled]
-        message = ', '.join(parts)
-        return NAGIOS_STATUS_WARNING, message
+    bad_lbs += [(
+        NAGIOS_STATUS_WARNING,
+        'loadbalancer {} admin_state_up is False'.format(lb.id)
+        ) for lb in lb_all if not lb.is_admin_state_up]
 
-    return NAGIOS_STATUS_OK, 'loadbalancers are happy'
+    return bad_lbs
 
 
 def check_pools(connection):
     """check pools status."""
     lb_mgr = connection.load_balancer
     pools_all = lb_mgr.pools()
+
+    # only check enabled pools
     pools_enabled = [pool for pool in pools_all if pool.is_admin_state_up]
 
     # check provisioning_status is ACTIVE for each pool
-    bad_pools = [pool for pool in pools_enabled if pool.provisioning_status != 'ACTIVE']
-    if bad_pools:
-        parts = ['pool {} provisioning_status is {}'.format(
-            pool.id, pool.provisioning_status) for pool in bad_pools]
-        message = ', '.join(parts)
-        return NAGIOS_STATUS_CRITICAL, message
+    bad_pools = [(
+        NAGIOS_STATUS_CRITICAL,
+        'pool {} provisioning_status is {}'.format(
+            pool.id, pool.provisioning_status)
+        ) for pool in pools_enabled if pool.provisioning_status != 'ACTIVE']
 
     # raise CRITICAL if operating_status is ERROR
-    bad_pools = [pool for pool in pools_enabled if pool.operating_status == 'ERROR']
-    if bad_pools:
-        parts = ['pool {} operating_status is {}'.format(
-            pool.id, pool.operating_status) for pool in bad_pools]
-        message = ', '.join(parts)
-        return NAGIOS_STATUS_CRITICAL, message
+    bad_pools += [(
+        NAGIOS_STATUS_CRITICAL,
+        'pool {} operating_status is {}'.format(
+            pool.id, pool.operating_status)
+        ) for pool in pools_enabled if pool.operating_status == 'ERROR']
 
     # raise WARNING if operating_status is NO_MONITOR
-    bad_pools = [pool for pool in pools_enabled if pool.operating_status == 'NO_MONITOR']
-    if bad_pools:
-        parts = ['pool {} operating_status is {}'.format(
-            pool.id, pool.operating_status) for pool in bad_pools]
-        message = ', '.join(parts)
-        return NAGIOS_STATUS_WARNING, message
+    bad_pools += [(
+        NAGIOS_STATUS_WARNING,
+        'pool {} operating_status is {}'.format(
+            pool.id, pool.operating_status)
+        ) for pool in pools_enabled if pool.operating_status == 'NO_MONITOR']
 
-    return NAGIOS_STATUS_OK, 'pools are happy'
+    return bad_pools
 
 
 def check_amphorae(connection):
@@ -120,7 +157,7 @@ def check_amphorae(connection):
     resp = lb_mgr.get('/v2/octavia/amphorae')
     # python api is not available yet, use url
     if resp.status_code != 200:
-        return NAGIOS_STATUS_WARNING, 'amphorae api not working'
+        return [(NAGIOS_STATUS_WARNING, 'amphorae api not working')]
 
     data = json.loads(resp.content)
     # ouput is like {"amphorae": [{...}, {...}, ...]}
@@ -128,26 +165,20 @@ def check_amphorae(connection):
 
     # raise CRITICAL for ERROR status
     bad_status_list = ('ERROR',)
-    bad_items = [item for item in items if item['status'] in bad_status_list]
-    if bad_items:
-        parts = [
-            'amphora {} status is {}'.format(item['id'], item['status'])
-            for item in bad_items]
-        message = ', '.join(parts)
-        return NAGIOS_STATUS_CRITICAL, message
+    bad_amp = [(
+        NAGIOS_STATUS_CRITICAL,
+        'amphora {} status is {}'.format(item['id'], item['status'])
+        ) for item in items if item['status'] in bad_status_list]
 
     # raise WARNING for these status
     bad_status_list = (
         'PENDING_CREATE', 'PENDING_UPDATE', 'PENDING_DELETE', 'BOOTING')
-    bad_items = [item for item in items if item['status'] in bad_status_list]
-    if bad_items:
-        parts = [
-            'amphora {} status is {}'.format(item['id'], item['status'])
-            for item in bad_items]
-        message = ', '.join(parts)
-        return NAGIOS_STATUS_WARNING, message
+    bad_amp += [(
+        NAGIOS_STATUS_WARNING,
+        'amphora {} status is {}'.format(item['id'], item['status'])
+        ) for item in items if item['status'] in bad_status_list]
 
-    return NAGIOS_STATUS_OK, 'amphorae are happy'
+    return bad_amp
 
 
 def check_image(connection, tag, days):
@@ -157,28 +188,47 @@ def check_image(connection, tag, days):
     if not images:
         message = ('Octavia requires image with tag {} to create amphora, '
                    'but none exist').format(tag)
-        return NAGIOS_STATUS_CRITICAL, message
+        return [(NAGIOS_STATUS_CRITICAL, message)]
 
     active_images = [image for image in images if image.status == 'active']
     if not active_images:
-        parts = ['{}({})'.format(image.name, image.id) for image in images]
+        details = ['{}({})'.format(image.name, image.id) for image in images]
         message = ('Octavia requires image with tag {} to create amphora, '
-                   'but none is active: {}').format(tag, ', '.join(parts))
-        return NAGIOS_STATUS_CRITICAL, message
+                   'but none are active: {}').format(tag, ', '.join(details))
+        return [(NAGIOS_STATUS_CRITICAL, message)]
 
     # raise WARNING if image is too old
     when = (datetime.now() - timedelta(days=days)).isoformat()
     # updated_at str format: '2019-12-05T18:21:25Z'
     fresh_images = [image for image in active_images if image.updated_at > when]
     if not fresh_images:
+        details = ['{}({})'.format(image.name, image.id) for image in images]
         message = ('Octavia requires image with tag {} to create amphora, '
-                   'but it is older than {} days').format(tag, days)
-        return NAGIOS_STATUS_WARNING, message
+                   'but all images are older than {} day(s): {}'
+                   '').format(tag, days, ', '.join(details))
+        return [(NAGIOS_STATUS_WARNING, message)]
 
-    return NAGIOS_STATUS_OK, 'image is ready'
+    return []
 
 
-if __name__ == '__main__':
+def process_checks(args):
+    # use closure to make all checks have same signature
+    # so we can handle them in same way
+    def _check_image(_connection):
+        return check_image(_connection, args.amp_image_tag, args.amp_image_days)
+
+    checks = {
+        'loadbalancers': check_loadbalancers,
+        'amphorae': check_amphorae,
+        'pools': check_pools,
+        'image': _check_image,
+    }
+
+    connection = openstack.connect(cloud='envvars')
+    return nagios_exit(args, checks[args.check](connection))
+
+
+def main():
     parser = argparse.ArgumentParser(
         description='Check Octavia status',
         formatter_class=argparse.ArgumentDefaultsHelpFormatter,
@@ -195,6 +245,11 @@ if __name__ == '__main__':
         help='which check to run')
 
     parser.add_argument(
+        '--ignored', dest="ignored", type=str,
+        default=DEFAULT_IGNORED,
+        help='Comma separated list of alerts to ignore')
+
+    parser.add_argument(
         '--amp-image-tag', dest='amp_image_tag', default='octavia-amphora',
         help='amphora image tag for image check')
 
@@ -211,17 +266,10 @@ if __name__ == '__main__':
         os.environ[key.decode('utf-8')] = value.rstrip().decode('utf-8')
     proc.communicate()
 
-    # use closure to make all checks have same signature
-    # so we can handle them in same way
-    def _check_image(connection):
-        return check_image(connection, args.amp_image_tag, args.amp_image_days)
+    status, message = process_checks(args)
+    print(message)
+    sys.exit(status)
 
-    checks = {
-        'loadbalancers': check_loadbalancers,
-        'amphorae': check_amphorae,
-        'pools': check_pools,
-        'image': _check_image,
-    }
 
-    connection = openstack.connect(cloud='envvars')
-    nagios_exit(*checks[args.check](connection))
+if __name__ == '__main__':
+    main()
diff --git a/lib/lib_openstack_service_checks.py b/lib/lib_openstack_service_checks.py
index 5e82395..c74d61d 100644
--- a/lib/lib_openstack_service_checks.py
+++ b/lib/lib_openstack_service_checks.py
@@ -202,16 +202,8 @@ class OSCHelper():
         charm_plugin_dir = os.path.join(hookenv.charm_dir(), 'files', 'plugins/')
         host.rsync(charm_plugin_dir, self.plugins_dir, options=['--executability'])
 
-    def render_checks(self, creds):
-        render(source='nagios.novarc', target=self.novarc, context=creds,
-               owner='nagios', group='nagios')
-
-        nrpe = NRPE()
-        if not os.path.exists(self.plugins_dir):
-            os.makedirs(self.plugins_dir)
-
-        self.update_plugins()
-        # Nova services health
+    def _render_nova_checks(self, nrpe):
+        """Nova services health."""
         nova_check_command = os.path.join(self.plugins_dir, 'check_nova_services.py')
         check_command = '{} --warn {} --crit {} --skip-aggregates {} {}'.format(
             nova_check_command, self.nova_warn, self.nova_crit, self.nova_skip_aggregates,
@@ -221,7 +213,8 @@ class OSCHelper():
                        check_cmd=check_command,
                        )
 
-        # Neutron agents health
+    def _render_neutron_checks(self, nrpe):
+        """Neutron agents health."""
         if self.is_neutron_agents_check_enabled:
             nrpe.add_check(shortname='neutron_agents',
                            description='Check that enabled Neutron agents are up',
@@ -231,6 +224,7 @@ class OSCHelper():
         else:
             nrpe.remove_check(shortname='neutron_agents')
 
+    def _render_cinder_checks(self, nrpe):
         # Cinder services health
         cinder_check_command = os.path.join(self.plugins_dir, 'check_cinder_services.py')
         check_command = '{} {}'.format(cinder_check_command, self.skip_disabled)
@@ -239,6 +233,7 @@ class OSCHelper():
                        check_cmd=check_command,
                        )
 
+    def _render_octavia_checks(self, nrpe):
         # only care about octavia after 18.04
         if host.lsb_release()['DISTRIB_RELEASE'] >= '18.04':
             if self.is_octavia_check_enabled:
@@ -246,24 +241,34 @@ class OSCHelper():
                 script = os.path.join(self.plugins_dir, 'check_octavia.py')
 
                 for check in ('loadbalancers', 'amphorae', 'pools'):
+                    check_cmd = '{} --check {}'.format(script, check)
+                    ignore = self.charm_config.get('octavia-%s-ignored' % check)
+                    if ignore:
+                        check_cmd += ' --ignored {}'.format(ignore)
                     nrpe.add_check(
                         shortname='octavia_{}'.format(check),
                         description='Check octavia {} status'.format(check),
-                        check_cmd='{} --check {}'.format(script, check),
+                        check_cmd=check_cmd,
                     )
 
                 # image check has extra args, add it separately
                 check = 'image'
+                check_cmd = "{} --check {}".format(script, check)
+                check_cmd += " --amp-image-tag {}".format(self.octavia_amp_image_tag)
+                check_cmd += " --amp-image-days {}".format(self.octavia_amp_image_days)
+                ignore = self.charm_config.get('octavia-%s-ignored' % check)
+                if ignore:
+                    check_cmd += " --ignored {}".format(ignore)
                 nrpe.add_check(
                     shortname='octavia_{}'.format(check),
                     description='Check octavia {} status'.format(check),
-                    check_cmd='{} --check {} --amp-image-tag {} --amp-image-days {}'.format(
-                        script, check, self.octavia_amp_image_tag, self.octavia_amp_image_days),
+                    check_cmd=check_cmd,
                 )
             else:
                 for check in ('loadbalancers', 'amphorae', 'pools', 'image'):
                     nrpe.remove_check(shortname='octavia_{}'.format(check))
 
+    def _render_contrail_checks(self, nrpe):
         if self.contrail_analytics_vip:
             contrail_check_command = '{} --host {}'.format(
                 os.path.join(self.plugins_dir, 'check_contrail_analytics_alarms.py'),
@@ -279,6 +284,7 @@ class OSCHelper():
         else:
             nrpe.remove_check(shortname='contrail_analytics_alarms')
 
+    def _render_dns_checks(self, nrpe):
         if len(self.check_dns):
             nrpe.add_check(shortname='dns_multi',
                            description='Check DNS names are resolvable',
@@ -289,8 +295,24 @@ class OSCHelper():
                            )
         else:
             nrpe.remove_check(shortname='dns_multi')
-        nrpe.write()
 
+    def render_checks(self, creds):
+        render(source='nagios.novarc', target=self.novarc, context=creds,
+               owner='nagios', group='nagios')
+
+        nrpe = NRPE()
+        if not os.path.exists(self.plugins_dir):
+            os.makedirs(self.plugins_dir)
+
+        self.update_plugins()
+        self._render_nova_checks(nrpe)
+        self._render_neutron_checks(nrpe)
+        self._render_cinder_checks(nrpe)
+        self._render_octavia_checks(nrpe)
+        self._render_contrail_checks(nrpe)
+        self._render_dns_checks(nrpe)
+
+        nrpe.write()
         self.create_endpoint_checks(creds)
 
     def _split_url(self, netloc, scheme):
diff --git a/tests/unit/conftest.py b/tests/unit/conftest.py
index 6797e85..639b91a 100644
--- a/tests/unit/conftest.py
+++ b/tests/unit/conftest.py
@@ -4,6 +4,10 @@ import sys
 
 import pytest
 
+TEST_DIR = os.path.dirname(__file__)
+CHECKS_DIR = os.path.join(TEST_DIR, '..', '..', 'files', 'plugins')
+sys.path.append(CHECKS_DIR)
+
 
 # If layer options are used, add this to openstackservicechecks
 # and import layer in lib_openstack_service_checks
@@ -77,14 +81,3 @@ def openstackservicechecks(tmpdir, mock_hookenv_config, mock_charm_dir, monkeypa
     monkeypatch.setattr('lib_openstack_service_checks.OSCHelper', lambda: helper)
 
     return helper
-
-
[e-mail](scope='module')
-def check_contrail_analytics():
-    pre = sys.path
-    TEST_DIR = os.path.dirname(__file__)
-    tests_dir = os.path.join(TEST_DIR, '..', '..', 'files', 'plugins')
-    sys.path.append(tests_dir)
-    import check_contrail_analytics_alarms as checks  # noqa
-    yield checks
-    sys.path = pre
diff --git a/tests/unit/test_check_cinder_services.py b/tests/unit/test_check_cinder_services.py
index 709b4dc..3428dd6 100644
--- a/tests/unit/test_check_cinder_services.py
+++ b/tests/unit/test_check_cinder_services.py
@@ -1,11 +1,7 @@
 import pytest
 import nagios_plugin3
 
-import sys
-
-sys.path.append("files/plugins")
-
-import check_cinder_services  # noqa: E402
+import check_cinder_services
 
 
 @pytest.mark.parametrize(
diff --git a/tests/unit/test_check_contrail_analytics_alarms.py b/tests/unit/test_check_contrail_analytics_alarms.py
index c7fd6e9..886a396 100644
--- a/tests/unit/test_check_contrail_analytics_alarms.py
+++ b/tests/unit/test_check_contrail_analytics_alarms.py
@@ -1,14 +1,15 @@
 import json
 import os
 
+import check_contrail_analytics_alarms
+
 TEST_DIR = os.path.dirname(__file__)
 
 
-def test_parse_contrail_alarms(check_contrail_analytics):
+def test_parse_contrail_alarms():
     with open(os.path.join(TEST_DIR, 'contrail_alert_data.json')) as f:
         data = json.load(f)
-    assert hasattr(check_contrail_analytics, 'parse_contrail_alarms')
-    parsed = check_contrail_analytics.parse_contrail_alarms(data)
+    parsed = check_contrail_analytics_alarms.parse_contrail_alarms(data)
     assert parsed in """
 CRITICAL: total_alarms[11], unacked_or_sev_gt_0[10], total_ignored[0], ignoring r''
 CRITICAL: vrouter{compute-10.maas, sev=1, ts[2020-06-25 18:29:23.149146]} Vrouter interface(s) down.
@@ -25,12 +26,11 @@ CRITICAL: vrouter{compute-7.maas, sev=1, ts[2020-07-03 18:30:32.481386]} Vrouter
 """ # noqa: ignore=F501
 
 
-def test_parse_contrail_alarms_filter_vrouter_control_9(check_contrail_analytics):
+def test_parse_contrail_alarms_filter_vrouter_control_9():
     with open(os.path.join(TEST_DIR, 'contrail_alert_data.json')) as f:
         data = json.load(f)
-    assert hasattr(check_contrail_analytics, 'parse_contrail_alarms')
     ignored_re = r'(?:vrouter)|(?:control-9)'
-    parsed = check_contrail_analytics.parse_contrail_alarms(data, ignored=ignored_re)
+    parsed = check_contrail_analytics_alarms.parse_contrail_alarms(data, ignored=ignored_re)
     assert parsed in """ 
 CRITICAL: total_alarms[11], unacked_or_sev_gt_0[10], total_ignored[8], ignoring r'(?:vrouter)|(?:control-9)'
 WARNING: control-node{control-8-contrail-rmq, sev=0, ts[2020-06-25 18:29:23.684803]} Node Failure. NodeStatus UVE not present.
@@ -39,32 +39,30 @@ CRITICAL: control-node{control-7-contrail-rmq, sev=1, ts[2020-06-25 18:29:24.377
 """ # noqa: ignore=F501
 
 
-def test_parse_contrail_alarms_filter_critical(check_contrail_analytics):
+def test_parse_contrail_alarms_filter_critical():
     with open(os.path.join(TEST_DIR, 'contrail_alert_data.json')) as f:
         data = json.load(f)
-    assert hasattr(check_contrail_analytics, 'parse_contrail_alarms')
     ignored_re = r'(?:CRITICAL)'
-    parsed = check_contrail_analytics.parse_contrail_alarms(data, ignored=ignored_re)
+    parsed = check_contrail_analytics_alarms.parse_contrail_alarms(data, ignored=ignored_re)
     assert parsed in """
 WARNING: total_alarms[11], unacked_or_sev_gt_0[10], total_ignored[10], ignoring r'(?:CRITICAL)'
 WARNING: control-node{control-8-contrail-rmq, sev=0, ts[2020-06-25 18:29:23.684803]} Node Failure. NodeStatus UVE not present.
 """ # noqa: ignore=F501
 
 
-def test_parse_contrail_alarms_all_ignored(check_contrail_analytics):
+def test_parse_contrail_alarms_all_ignored():
     with open(os.path.join(TEST_DIR, 'contrail_alert_data.json')) as f:
         data = json.load(f)
-    assert hasattr(check_contrail_analytics, 'parse_contrail_alarms')
     ignored_re = r'(?:CRITICAL)|(?:WARNING)'
-    parsed = check_contrail_analytics.parse_contrail_alarms(data, ignored=ignored_re)
+    parsed = check_contrail_analytics_alarms.parse_contrail_alarms(data, ignored=ignored_re)
     assert parsed in """
 OK: total_alarms[11], unacked_or_sev_gt_0[10], total_ignored[11], ignoring r'(?:CRITICAL)|(?:WARNING)'
 """ # noqa: ignore=F501
 
 
-def test_parse_contrail_alarms_no_alarms(check_contrail_analytics):
+def test_parse_contrail_alarms_no_alarms():
     ignored_re = r''
-    parsed = check_contrail_analytics.parse_contrail_alarms({}, ignored=ignored_re)
+    parsed = check_contrail_analytics_alarms.parse_contrail_alarms({}, ignored=ignored_re)
     assert parsed in """
 OK: total_alarms[0], unacked_or_sev_gt_0[0], total_ignored[0], ignoring r''
 """
diff --git a/tests/unit/test_check_nova_services.py b/tests/unit/test_check_nova_services.py
index 10c13ed..dad32a6 100644
--- a/tests/unit/test_check_nova_services.py
+++ b/tests/unit/test_check_nova_services.py
@@ -1,10 +1,7 @@
 import pytest
 import nagios_plugin3
 
-import sys
-sys.path.append('files/plugins')
-
-import check_nova_services  # noqa: E402
+import check_nova_services
 
 
 @pytest.mark.parametrize('is_skip_disabled,num_nodes',
diff --git a/tests/unit/test_check_octavia.py b/tests/unit/test_check_octavia.py
 new file mode 100644
index 0000000..08c9357
--- /dev/null
+++ b/tests/unit/test_check_octavia.py
@@ -0,0 +1,117 @@
+from datetime import datetime, timedelta
+import json
+import unittest.mock as mock
+from uuid import uuid4
+
+import check_octavia
+import pytest
+
+
[e-mail]('check_octavia.openstack.connect')
[e-mail]('check', [
+    'loadbalancers', 'pools', "amphorae", "image"
+])
+def test_stable_alarms(connect, check):
+    args = mock.MagicMock()
+    args.ignored = r''
+    args.check = check
+    if check == "amphorae":
+        # Present 0 Amphora instances
+        resp = connect().load_balancer.get()
+        resp.status_code = 200
+        resp.content = json.dumps({'amphora': []})
+    elif check == "image":
+        # Present 1 Active Fresh Amphora image
+        args.amp_image_tag = 'octavia'
+        args.amp_image_days = 1
+        amp_image = mock.MagicMock()
+        amp_image.status = 'active'
+        amp_image.updated_at = datetime.now().isoformat()
+        connect().image.images.return_value = [amp_image]
+
+    status, message = check_octavia.process_checks(args)
+    assert message in """
+OK: total_alarms[0], total_crit[0], total_ignored[0], ignoring r''
+"""
+    assert status == check_octavia.NAGIOS_STATUS_OK
+
+
[e-mail]('check_octavia.openstack.connect')
+def test_no_images_is_ignorable(connect):
+    args = mock.MagicMock()
+    args.ignored = 'none exist'
+    args.check = "image"
+    # Present 1 Active Fresh Amphora image
+    args.amp_image_tag = 'octavia'
+    args.amp_image_days = 1
+    connect().image.images.return_value = []
+
+    status, message = check_octavia.process_checks(args)
+    assert message in """
+OK: total_alarms[1], total_crit[1], total_ignored[1], ignoring r'(?:none exist)'
+"""
+    assert status == check_octavia.NAGIOS_STATUS_OK
+
+
[e-mail]('check_octavia.openstack.connect')
+def test_no_images(connect):
+    args = mock.MagicMock()
+    args.ignored = r''
+    args.check = "image"
+    # Present 1 Active Fresh Amphora image
+    args.amp_image_tag = 'octavia'
+    args.amp_image_days = 1
+    connect().image.images.return_value = []
+
+    status, message = check_octavia.process_checks(args)
+    assert message in """
+CRITICAL: total_alarms[1], total_crit[1], total_ignored[0], ignoring r''
+Octavia requires image with tag octavia to create amphora, but none exist
+"""
+    assert status == check_octavia.NAGIOS_STATUS_CRITICAL
+
+
[e-mail]('check_octavia.openstack.connect')
+def test_no_active_images(connect):
+    args = mock.MagicMock()
+    args.ignored = r''
+    args.check = "image"
+    # Present 1 Active Fresh Amphora image
+    args.amp_image_tag = 'octavia'
+    args.amp_image_days = 1
+    amp_image = mock.MagicMock()
+    amp_image.name = "bob-the-image"
+    amp_image.id = str(uuid4())
+    amp_image.status = 'inactive'
+    amp_image.updated_at = datetime.now().isoformat()
+    connect().image.images.return_value = [amp_image]
+
+    status, message = check_octavia.process_checks(args)
+    assert message in """
+CRITICAL: total_alarms[1], total_crit[1], total_ignored[0], ignoring r''
+Octavia requires image with tag octavia to create amphora, but none are active: bob-the-image({})
+""".format(amp_image.id)
+    assert status == check_octavia.NAGIOS_STATUS_CRITICAL
+
+
[e-mail]('check_octavia.openstack.connect')
+def test_no_fresh_images(connect):
+    args = mock.MagicMock()
+    args.ignored = r''
+    args.check = "image"
+    # Present 1 Active Fresh Amphora image
+    args.amp_image_tag = 'octavia'
+    args.amp_image_days = 1
+    amp_image = mock.MagicMock()
+    amp_image.name = "bob-the-image"
+    amp_image.id = str(uuid4())
+    amp_image.status = 'active'
+    amp_image.updated_at = (datetime.now() - timedelta(days=2)).isoformat()
+    connect().image.images.return_value = [amp_image]
+
+    status, message = check_octavia.process_checks(args)
+    assert message in """
+WARNING: total_alarms[1], total_crit[0], total_ignored[0], ignoring r''
+Octavia requires image with tag octavia to create amphora, but all images are older than 1 day(s): bob-the-image({})
+""".format(amp_image.id)
+    assert status == check_octavia.NAGIOS_STATUS_WARNING