Merge ~exsdev/juju-verify:lp1916231-neutron-gateway-juju-verifier-plugin into ~canonical-solutions-engineering/juju-verify:master

[Edin S]

Add Juju verify plugin for Neutron Gateway

Fixes LP#1916231

The other half of the required code (the required Charm actions) can be found here: https://review.opendev.org/c/openstack/charm-neutron-gateway/+/782896

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Unable to determine commit message from repository - please click "Set commit message" and enter the commit message manually.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:ddf29dc6f8628857a266a8223d32915d657e12b8

No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want jenkins to rebuild you need to trigger it yourself):
https://code.launchpad.net/~exsdev/juju-verify/+git/juju-verify/+merge/399961/+edit-commit-message

https://jenkins.canonical.com/bootstack/job/juju-verify-test/50/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/generic-make-test/71/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/50/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/juju-verify-test/50//rebuild

review: Needs Fixing (continuous-integration)

f244ee1... by Edin S on 2021-03-22

NGW: fix lint issues including typing information

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:f244ee183c33ef55492ec895a2bb0aeefb8c5850
https://jenkins.canonical.com/bootstack/job/juju-verify-test/51/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/generic-make-test/72/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/51/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/juju-verify-test/51//rebuild

review: Needs Fixing (continuous-integration)

8c7ff94... by Edin S on 2021-03-22

NGW: fix issues raised by pylint

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:8c7ff9469260e8f222802753de4caa544013e915
https://jenkins.canonical.com/bootstack/job/juju-verify-test/52/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/generic-make-test/73/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/52/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/juju-verify-test/52//rebuild

review: Needs Fixing (continuous-integration)

b3f7b69... by Edin S on 2021-03-26

NGW: Add test for verifying reboot/shutdown

bc56476... by Edin S on 2021-03-26

NGW: Add test for getting unit's hostname

83d72cb... by Edin S on 2021-03-26

NGW: Add test for getting a unit's agent resource list

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:83d72cb74ee77f219c50a757fe4e8e015f5b0cbc
https://jenkins.canonical.com/bootstack/job/juju-verify-test/65/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/generic-make-test/86/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/65/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/juju-verify-test/65//rebuild

review: Needs Fixing (continuous-integration)

[Edin S (exsdev)]

NeutronGateway.get_all_ngw_units() currently doesn't have a unit test for it, but is well-suited for functional tests.

[Robert Gildein (rgildein)]

review: Needs Fixing

[Martin Kalcok (martin-kalcok)]

I have few inline suggestions here, otherwise it looks good, thanks.

Regarding the code coverage, I think it's important that we keep 100% coverage in unit tests unless there's a really good reason why something is not unit-testable. Anyway, that's not high priority and can be done by someone else if you won't manage to finish it by the end of your rotation.

review: Needs Fixing

[Martin Kalcok (martin-kalcok)]

juju-verify development was moved to https://github.com/canonical/juju-verify

Unmerged commits

83d72cb... by Edin S on 2021-03-26

NGW: Add test for getting a unit's agent resource list

bc56476... by Edin S on 2021-03-26

NGW: Add test for getting unit's hostname

b3f7b69... by Edin S on 2021-03-26

NGW: Add test for verifying reboot/shutdown

8c7ff94... by Edin S on 2021-03-22

NGW: fix issues raised by pylint

f244ee1... by Edin S on 2021-03-22

NGW: fix lint issues including typing information

ddf29dc... by Edin S on 2021-03-22

Add NGW tests for warning about manual failover of HA routers

ff315e1... by Edin S on 2021-03-22

Add NGW tests for determining redundancy of resources

20f6288... by Edin S on 2021-03-22

Add NGW tests for generating list of resources that will remain online

ebd233b... by Edin S on 2021-03-22

Add NGW tests for generating list of resources to be shutdown

e1e6433... by Edin S on 2021-03-22

Add NGW tests for getting resource list

diff --git a/juju_verify/utils/unit.py b/juju_verify/utils/unit.py
index 05a9785..f672b20 100644
--- a/juju_verify/utils/unit.py
+++ b/juju_verify/utils/unit.py
@@ -64,6 +64,12 @@ def run_action_on_units(
     return result_map
 
 
+def run_action_on_unit(unit: Unit, action: str, **params: str) -> Any:
+    """Run Juju action on single unit, returning the result."""
+    results = run_action_on_units([unit], action=action, **params)
+    return results[unit.entity_id]
+
+
 def parse_charm_name(charm_url: str) -> str:
     """Parse charm name from full charm url.
 
diff --git a/juju_verify/verifiers/__init__.py b/juju_verify/verifiers/__init__.py
index 4400e0f..3908db2 100644
--- a/juju_verify/verifiers/__init__.py
+++ b/juju_verify/verifiers/__init__.py
@@ -28,6 +28,7 @@ from juju_verify.verifiers.base import BaseVerifier
 from juju_verify.verifiers.ceph import CephOsd
 from juju_verify.verifiers.nova_compute import NovaCompute
 from juju_verify.verifiers.result import Result
+from juju_verify.verifiers.neutron_gateway import NeutronGateway
 
 logger = logging.getLogger(__name__)
 
@@ -35,6 +36,7 @@ logger = logging.getLogger(__name__)
 SUPPORTED_CHARMS = {
     'nova-compute': NovaCompute,
     'ceph-osd': CephOsd,
+    'neutron-gateway': NeutronGateway,
 }
 
 
diff --git a/juju_verify/verifiers/neutron_gateway.py b/juju_verify/verifiers/neutron_gateway.py
new file mode 100644
index 0000000..d818ef0
--- /dev/null
+++ b/juju_verify/verifiers/neutron_gateway.py
@@ -0,0 +1,159 @@
+# Copyright 2021 Canonical Limited.
+#
+# This file is part of juju-verify.
+#
+# juju-verify is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation, either version 3 of the License, or (at your option) any later
+# version.
+#
+# juju-verify is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program. If not, see https://www.gnu.org/licenses/.
+"""neutron-gateway verification."""
+import json
+from typing import Dict, List
+
+from juju.unit import Unit
+
+from juju_verify.verifiers.base import BaseVerifier, Result
+from juju_verify.verifiers.result import aggregate_results
+from juju_verify.utils.action import data_from_action
+from juju_verify.utils.unit import run_action_on_unit
+
+
+def get_unit_hostname(unit: Unit) -> str:
+    """Return name of the host on which the unit is running."""
+    get_hostname_action = run_action_on_unit(unit, "node-name")
+    hostname = data_from_action(get_hostname_action, "node-name")
+    return hostname
+
+
+def get_unit_resource_list(unit: Unit, get_resource_action_name: str) -> List[dict]:
+    """Given a get resource action, return the relevant resources on the unit."""
+    get_resource_action = run_action_on_unit(unit, get_resource_action_name)
+    action_name_res = NeutronGateway.action_name_result_map[get_resource_action_name]
+    resource_list_json = data_from_action(get_resource_action, action_name_res)
+    resource_list = json.loads(resource_list_json)
+    return resource_list
+
+
+class NeutronGateway(BaseVerifier):
+    """Implementation of verification checks for the neutron-gateway charm."""
+
+    NAME = 'neutron-gateway'
+    action_name_result_map = {"get-status-routers": "router-list",
+                              "get-status-dhcp": "dhcp-networks",
+                              "get-status-lb": "load-balancers"}
+    action_name_failure_string_map = {"get-status-routers": ("The following routers are "
+                                                             "non-redundant: {}."),
+                                      "get-status-dhcp": ("The following DHCP networks "
+                                                          "are non-redundant: {}"),
+                                      "get-status-lb": ("The following LBaasV2 LBs were "
+                                                        "found: {}. LBaasV2 does not "
+                                                        "offer HA.")}
+
+    def __init__(self, units: List[Unit]) -> None:
+        """Neutron Gateway verifier constructor."""
+        super().__init__(units)
+        self.cache_action_resource_list_map: Dict[str, List] = {}
+
+    def get_all_ngw_units(self) -> List[Unit]:
+        """Get all neutron-gateway units, including those not being shutdown."""
+        application_name = NeutronGateway.NAME
+        for rel in self.model.relations:
+            if rel.matches("{}:cluster".format(application_name)):
+                application = rel.applications.pop()
+                all_ngw_units = application.units
+        return all_ngw_units
+
+    def get_resource_list(self, get_resource_action_name: str) -> List[dict]:
+        """Given a get resource action, return matching resources from all units."""
+        try:
+            return self.cache_action_resource_list_map[get_resource_action_name]
+        except KeyError:
+            pass
+
+        resource_list = []
+        shutdown_hostname_list = [get_unit_hostname(unit) for unit in self.units]
+
+        for unit in self.get_all_ngw_units():
+            hostname = get_unit_hostname(unit)
+            host_resource_list = get_unit_resource_list(unit,
+                                                        get_resource_action_name)
+
+            # add host metadata to resource
+            for resource in host_resource_list:
+                resource["host"] = hostname
+                resource["juju-entity-id"] = unit.entity_id
+                resource["shutdown"] = False
+
+                if hostname in shutdown_hostname_list:
+                    resource["shutdown"] = True
+
+                resource_list.append(resource)
+
+        self.cache_action_resource_list_map[get_resource_action_name] = resource_list
+        return resource_list
+
+    def get_shutdown_resource_list(self, get_resource_action_name: str) -> List[dict]:
+        """Return a list of resources matching action that are going to be shutdown."""
+        res_list = self.get_resource_list(get_resource_action_name)
+        return [r for r in res_list if r["shutdown"] and r["status"] == "ACTIVE"]
+
+    def get_online_resource_list(self, get_resource_action_name: str) -> List[dict]:
+        """Return a list of resources matching action, that will remain online."""
+        res_list = self.get_resource_list(get_resource_action_name)
+        return [r for r in res_list if not r["shutdown"] and r["status"] == "ACTIVE"]
+
+    def check_non_redundant_resource(self, action_name: str) -> Result:
+        """Check that there are no non-redundant resources matching the resource type."""
+        result = Result(True)
+        shutdown_resource_list = self.get_shutdown_resource_list(action_name)
+        redundant_resource_list = self.get_online_resource_list(action_name)
+
+        shutdown_resource_set = set(r["id"] for r in shutdown_resource_list)
+        redundant_resource_set = set(r["id"] for r in redundant_resource_list)
+        non_redundant_list = shutdown_resource_set - redundant_resource_set
+        if non_redundant_list:
+            result.success = False
+            failure_string = NeutronGateway.action_name_failure_string_map[action_name]
+            result.reason = failure_string.format(", ".join(non_redundant_list))
+        return result
+
+    def warn_router_ha(self) -> Result:
+        """Warn that HA routers should be manually failed over."""
+        action_name = "get-status-routers"
+        result = Result(True)
+        shutdown_resource_list = self.get_shutdown_resource_list(action_name)
+
+        router_failover_err_list = []
+        for router in shutdown_resource_list:
+            if router["ha"]:
+                _id = router["id"]
+                entity_id = router["juju-entity-id"]
+                host = router["host"]
+                error_string = "{} (on {}, hostname: {})".format(_id, entity_id, host)
+                router_failover_err_list.append(error_string)
+
+        if router_failover_err_list:
+            error_string = ("It's recommended that you manually failover the following "
+                            "routers: {}")
+            result.reason = error_string.format(", ".join(router_failover_err_list))
+
+        return result
+
+    def verify_reboot(self) -> Result:
+        """Verify that it's safe to reboot selected neutron-gateway units."""
+        return self.verify_shutdown()
+
+    def verify_shutdown(self) -> Result:
+        """Verify that it's safe to shutdown selected neutron-gateway units."""
+        return aggregate_results(self.warn_router_ha(),
+                                 self.check_non_redundant_resource("get-status-routers"),
+                                 self.check_non_redundant_resource("get-status-dhcp"),
+                                 self.check_non_redundant_resource("get-status-lb"))
diff --git a/tests/unit/verifiers/test_neutron_gateway.py b/tests/unit/verifiers/test_neutron_gateway.py
new file mode 100644
index 0000000..4aeb48d
--- /dev/null
+++ b/tests/unit/verifiers/test_neutron_gateway.py
@@ -0,0 +1,310 @@
+# Copyright 2021 Canonical Limited.
+#
+# This file is part of juju-verify.
+#
+# juju-verify is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation, either version 3 of the License, or (at your option) any later
+# version.
+#
+# juju-verify is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program. If not, see https://www.gnu.org/licenses/.
+"""NeutronGateway verifier class test suite."""
+import json
+from unittest import mock
+from unittest.mock import MagicMock
+
+from juju.unit import Unit
+
+from juju_verify.verifiers import NeutronGateway
+from juju_verify.verifiers.neutron_gateway import get_unit_hostname
+from juju_verify.verifiers.neutron_gateway import get_unit_resource_list
+
+all_ngw_units = []
+for i in range(3):
+    ngw = MagicMock()
+    ngw.entity_id = "neutron-gateway/{}".format(i)
+    all_ngw_units.append(ngw)
+
+mock_data = [
+    {
+        "host": "host0",
+        "shutdown": True,
+        "unit": all_ngw_units[0],
+        "routers": [{"id": "router0", "ha": False, "status": "ACTIVE"},
+                    {"id": "router1", "ha": False, "status": "ACTIVE"}]
+    },
+    {
+        "host": "host1",
+        "shutdown": False,
+        "unit": all_ngw_units[1],
+        "routers": [{"id": "router2", "ha": False, "status": "ACTIVE"}],
+    },
+    {
+        "host": "host2",
+        "shutdown": False,
+        "unit": all_ngw_units[2],
+        "routers": [{"id": "router3", "ha": False, "status": "ACTIVE"},
+                    {"id": "router4", "ha": False, "status": "ACTIVE"}],
+    },
+]
+
+all_ngw_host_names = [h["host"] for h in mock_data]
+
+model = MagicMock()
+
+
+def get_ngw_verifier():
+    """Get new NeutronGateway verifier (used for applying changes in shutdown list)."""
+    return NeutronGateway([Unit(h["unit"].entity_id, model)
+                           for h in mock_data if h["shutdown"]])
+
+
+def get_resource_lists():
+    """Get all routers in mock data."""
+    return [h["routers"] for h in mock_data]
+
+
+def get_shutdown_host_name_list():
+    """Get all hostnames of all hosts being shutdown."""
+    return [h["host"] for h in mock_data if h["shutdown"]]
+
+
+def set_router_status(routerid, status):
+    """Set status of given router id in mock data."""
+    for host in mock_data:
+        for router in host["routers"]:
+            if router["id"] == routerid:
+                router["status"] = status
+
+
[e-mail]("juju_verify.verifiers.neutron_gateway.get_unit_resource_list")
[e-mail]("juju_verify.verifiers.neutron_gateway.NeutronGateway.get_all_ngw_units")
[e-mail]("juju_verify.verifiers.neutron_gateway.get_unit_hostname")
+def test_get_resource_list(mock_get_unit_hostname,
+                           mock_get_all_ngw_units,
+                           mock_get_unit_resource_list):
+    """Test list of resources returned by get_resource_list."""
+    mock_get_unit_hostname.side_effect = (get_shutdown_host_name_list() +
+                                          all_ngw_host_names)
+    mock_get_all_ngw_units.return_value = all_ngw_units
+    mock_get_unit_resource_list.side_effect = get_resource_lists()
+
+    ngw_verifier = get_ngw_verifier()
+    router_list = ngw_verifier.get_resource_list("get-status-routers")
+
+    router_count = 0
+    for host in mock_data:
+        router_count += len(host["routers"])
+    assert len(router_list) == router_count
+
+
[e-mail]("juju_verify.verifiers.neutron_gateway.get_unit_resource_list")
[e-mail]("juju_verify.verifiers.neutron_gateway.NeutronGateway.get_all_ngw_units")
[e-mail]("juju_verify.verifiers.neutron_gateway.get_unit_hostname")
+def test_get_shutdown_resource_list(mock_get_unit_hostname,
+                                    mock_get_all_ngw_units,
+                                    mock_get_unit_resource_list):
+    """Test validity of list of resources to be shutdown."""
+    mock_get_unit_hostname.side_effect = (get_shutdown_host_name_list() +
+                                          all_ngw_host_names)
+    mock_get_all_ngw_units.return_value = all_ngw_units
+    mock_get_unit_resource_list.side_effect = get_resource_lists()
+
+    ngw_verifier = get_ngw_verifier()
+
+    router_shutdown_count = 0
+    for host in mock_data:
+        if host["shutdown"]:
+            router_shutdown_count += len(host["routers"])
+
+    shutdown_routers = ngw_verifier.get_shutdown_resource_list("get-status-routers")
+    assert len(shutdown_routers) == router_shutdown_count
+
+    # test that inactive resources are not being listed as being shutdown
+    set_router_status("router0", "NOTACTIVE")
+
+    mock_get_unit_hostname.side_effect = (get_shutdown_host_name_list() +
+                                          all_ngw_host_names)
+    mock_get_all_ngw_units.return_value = all_ngw_units
+    mock_get_unit_resource_list.side_effect = get_resource_lists()
+
+    shutdown_routers = ngw_verifier.get_shutdown_resource_list("get-status-routers")
+    assert len(shutdown_routers) == router_shutdown_count - 1
+
+    # set router0 back to active
+    set_router_status("router0", "ACTIVE")
+
+
[e-mail]("juju_verify.verifiers.neutron_gateway.get_unit_resource_list")
[e-mail]("juju_verify.verifiers.neutron_gateway.NeutronGateway.get_all_ngw_units")
[e-mail]("juju_verify.verifiers.neutron_gateway.get_unit_hostname")
+def test_get_online_resource_list(mock_get_unit_hostname,
+                                  mock_get_all_ngw_units,
+                                  mock_get_unit_resource_list):
+    """Test validity of resources that will remain online."""
+    mock_get_unit_hostname.side_effect = (get_shutdown_host_name_list() +
+                                          all_ngw_host_names)
+    mock_get_all_ngw_units.return_value = all_ngw_units
+    mock_get_unit_resource_list.side_effect = get_resource_lists()
+
+    ngw_verifier = get_ngw_verifier()
+
+    router_online_count = 0
+    for host in mock_data:
+        if not host["shutdown"]:
+            router_online_count += len(host["routers"])
+
+    online_routers = ngw_verifier.get_online_resource_list("get-status-routers")
+    assert len(online_routers) == router_online_count
+
+    # test that NOT ACTIVE resources are not being listed as online/available
+    set_router_status("router2", "NOTACTIVE")
+
+    mock_get_unit_hostname.side_effect = (get_shutdown_host_name_list() +
+                                          all_ngw_host_names)
+    mock_get_all_ngw_units.return_value = all_ngw_units
+    mock_get_unit_resource_list.side_effect = get_resource_lists()
+
+    online_routers = ngw_verifier.get_online_resource_list("get-status-routers")
+    assert len(online_routers) == router_online_count - 1
+
+    # set router2 back to active
+    set_router_status("router2", "ACTIVE")
+
+
[e-mail]("juju_verify.verifiers.neutron_gateway.get_unit_resource_list")
[e-mail]("juju_verify.verifiers.neutron_gateway.NeutronGateway.get_all_ngw_units")
[e-mail]("juju_verify.verifiers.neutron_gateway.get_unit_hostname")
+def test_check_non_redundant_resource(mock_get_unit_hostname,
+                                      mock_get_all_ngw_units,
+                                      mock_get_unit_resource_list):
+    """Test validity of list of resources determined to not be redundant."""
+    mock_get_unit_hostname.side_effect = (get_shutdown_host_name_list() +
+                                          all_ngw_host_names)
+    mock_get_all_ngw_units.return_value = all_ngw_units
+    mock_get_unit_resource_list.side_effect = get_resource_lists()
+
+    ngw_verifier = get_ngw_verifier()
+
+    # host0 being shutdown, with no redundancy for its routers (router0, router1)
+    result = ngw_verifier.check_non_redundant_resource("get-status-routers")
+    assert result.success is False
+
+    mock_get_unit_hostname.side_effect = (get_shutdown_host_name_list() +
+                                          all_ngw_host_names)
+    mock_get_all_ngw_units.return_value = all_ngw_units
+    mock_get_unit_resource_list.side_effect = get_resource_lists()
+
+    ngw_verifier = get_ngw_verifier()
+
+    # add redundancy (but not HA) for router0, router1 onto non-shutdown hosts
+    mock_data[1]["routers"].append({"id": "router0", "ha": False, "status": "ACTIVE"})
+    mock_data[2]["routers"].append({"id": "router1", "ha": False, "status": "ACTIVE"})
+    mock_get_unit_resource_list.side_effect = get_resource_lists()
+    result = ngw_verifier.check_non_redundant_resource("get-status-routers")
+    assert result.success
+
+    # test setting redundant redundant router0 to NOTACTIVE will result in failure
+    mock_data[1]["routers"][-1]["status"] = "NOTACTIVE"
+    mock_get_unit_resource_list.side_effect = get_resource_lists()
+    result = ngw_verifier.check_non_redundant_resource("get-status-routers")
+    assert result.success is False
+
+    # test shutdown host1, which will take down the redundant router0
+    mock_data[1]["shutdown"] = True
+    mock_get_unit_hostname.side_effect = (get_shutdown_host_name_list() +
+                                          all_ngw_host_names)
+    mock_get_all_ngw_units.return_value = all_ngw_units
+    mock_get_unit_resource_list.side_effect = get_resource_lists()
+
+    ngw_verifier = get_ngw_verifier()
+    result = ngw_verifier.check_non_redundant_resource("get-status-routers")
+    assert result.success is False
+
+    # reset shutdown for next tests
+    mock_data[1]["shutdown"] = False
+
+
[e-mail]("juju_verify.verifiers.neutron_gateway.get_unit_resource_list")
[e-mail]("juju_verify.verifiers.neutron_gateway.NeutronGateway.get_all_ngw_units")
[e-mail]("juju_verify.verifiers.neutron_gateway.get_unit_hostname")
+def test_warn_router_ha(mock_get_unit_hostname,
+                        mock_get_all_ngw_units,
+                        mock_get_unit_resource_list):
+    """Test existence of warning messages to manually failover HA routers when found."""
+    mock_get_unit_hostname.side_effect = (get_shutdown_host_name_list() +
+                                          all_ngw_host_names)
+    mock_get_all_ngw_units.return_value = all_ngw_units
+    mock_get_unit_resource_list.side_effect = get_resource_lists()
+
+    ngw_verifier = get_ngw_verifier()
+
+    result = ngw_verifier.warn_router_ha()
+    # no HA to failover, lack of redundancy is detected by check_non_redundant_resource
+    assert result.reason == ""
+
+    # Find router0 set it to HA
+    for host in mock_data:
+        for router in host["routers"]:
+            if router["id"] == "router0":
+                router["ha"] = True
+
+    mock_get_unit_hostname.side_effect = (get_shutdown_host_name_list() +
+                                          all_ngw_host_names)
+    mock_get_all_ngw_units.return_value = all_ngw_units
+    mock_get_unit_resource_list.side_effect = get_resource_lists()
+
+    ngw_verifier = get_ngw_verifier()
+
+    result = ngw_verifier.warn_router_ha()
+    # router is in HA, given instructions to failover
+    assert result.reason
+
+
[e-mail]("juju_verify.verifiers.neutron_gateway.NeutronGateway.check_non_redundant_resource")  # noqa: E501 pylint: disable=C0301
[e-mail]("juju_verify.verifiers.neutron_gateway.NeutronGateway.warn_router_ha")
[e-mail]("juju_verify.verifiers.neutron_gateway.aggregate_results")
+def test_verify_reboot_shutdown(mock_aggregate_results,
+                                mock_warn_router_ha,
+                                mock_check_non_redundant_resource):
+    """Test that reboot/shutdown call appropriate checks."""
+    ngw_verifier = get_ngw_verifier()
+    ngw_verifier.verify_reboot()
+    assert mock_check_non_redundant_resource.call_count == 3
+    mock_warn_router_ha.assert_called_once()
+    mock_aggregate_results.assert_called_once()
+
+    mock_check_non_redundant_resource.reset_mock()
+    mock_warn_router_ha.reset_mock()
+    mock_aggregate_results.reset_mock()
+
+    ngw_verifier.verify_shutdown()
+    mock_warn_router_ha.assert_called_once()
+    assert mock_check_non_redundant_resource.call_count == 3
+    mock_aggregate_results.assert_called_once()
+
+
[e-mail]("juju_verify.verifiers.neutron_gateway.run_action_on_unit")
[e-mail]("juju_verify.verifiers.neutron_gateway.data_from_action")
+def test_get_unit_hostname(mock_data_from_action, mock_run_action_on_unit):
+    """Test getting remote Unit's hostname."""
+    get_unit_hostname(all_ngw_units[0])
+    mock_run_action_on_unit.assert_called_once()
+    mock_data_from_action.assert_called_once()
+
+
[e-mail]("juju_verify.verifiers.neutron_gateway.run_action_on_unit")
[e-mail]("juju_verify.verifiers.neutron_gateway.data_from_action")
+def test_get_unit_resource_list(mock_data_from_action, mock_run_action_on_unit):
+    """Test Neutron agent resources are retrieved via Juju actions."""
+    resource = {"routers": [{"id": "r1"}]}
+    mock_data_from_action.return_value = json.dumps(resource)
+    resource_list = get_unit_resource_list(all_ngw_units[0], "get-status-routers")
+    mock_run_action_on_unit.assert_called_once()
+    assert resource == resource_list