Merge ~xavpaice/juju-verify:add_functest into ~canonical-solutions-engineering/juju-verify:master

[Xav Paice]

Add functional testing

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:f5e33f9e4548c7c1ed3e686ef59f2339bb1f4f31
https://jenkins.canonical.com/bootstack/job/juju-verify-test/46/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/generic-make-test/67/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/46/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/juju-verify-test/46//rebuild

review: Needs Fixing (continuous-integration)

[Xav Paice (xavpaice)]

Test failure was caused by proxy/firewall issues:

fatal: unable to access 'https://github.com/openstack-charmers/zaza.git/': Received HTTP code 403 from proxy after CONNECT

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:4e5addc7784471452e3b05ac11c3e8a4d4bacea8
https://jenkins.canonical.com/bootstack/job/juju-verify-test/53/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/generic-make-test/74/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/53/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/juju-verify-test/53//rebuild

review: Needs Fixing (continuous-integration)

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

FAILED: Continuous integration, rev:4e5addc7784471452e3b05ac11c3e8a4d4bacea8
https://jenkins.canonical.com/bootstack/job/juju-verify-test/54/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/bootstack/job/generic-make-test/75/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/54/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/juju-verify-test/54//rebuild

review: Needs Fixing (continuous-integration)

f472b29... by Xav Paice on 2021-03-23

Update functests for some errors

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:4def417ab7332867d81e1b42d6e0db385d1d1de7
https://jenkins.canonical.com/bootstack/job/juju-verify-test/55/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/generic-make-test/76/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/55/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/juju-verify-test/55//rebuild

review: Approve (continuous-integration)

[Martin Kalcok (martin-kalcok)]

Thanks for laying foundation for the func tests. Overall LGTM, I just had 2 small inline comments (only one is really blocking).

review: Needs Fixing

[Xav Paice (xavpaice)]

Added a few comments of my own, need to update the commit before merge.

cf71af9... by Xav Paice on 2021-03-26

fix hacluster functest

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

A CI job is currently in progress. A follow up comment will be added when it completes.

[🤖 prod-jenkaas-bootstack (prod-jenkaas-bootstack)]

PASSED: Continuous integration, rev:cf71af911c4000c522662eebb9442fe9b7c223a0
https://jenkins.canonical.com/bootstack/job/juju-verify-test/64/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/bootstack/job/generic-make-test/85/
    None: https://jenkins.canonical.com/bootstack/job/lp-update-mp/64/

Click here to trigger a rebuild:
https://jenkins.canonical.com/bootstack/job/juju-verify-test/64//rebuild

review: Approve (continuous-integration)

[Alvaro Uria (aluria)]

Follow-up work at https://code.launchpad.net/~martin-kalcok/juju-verify/+git/juju-verify/+merge/401271

Unmerged commits

cf71af9... by Xav Paice on 2021-03-26

fix hacluster functest

f472b29... by Xav Paice on 2021-03-23

Update functests for some errors

4e5addc... by Xav Paice on 2021-03-07

Add functional testing

This adds a basic test for Ceph OSD verification.

There is also an outline for Nova Compute, that relies on an action
which is not currently avaialable, therefore the test is not enabled.

A bundle for starting testing openstack-dashboard/hacluster is included,
but no tests.

diff --git a/Makefile b/Makefile
index 19c8c1f..703e60f 100644
--- a/Makefile
+++ b/Makefile
@@ -34,5 +34,9 @@ clean:
 	@echo "Cleaning files"
 	@git clean -fxd -e '!.idea'
 
+functional: build
+	@echo "Executing functional tests"
+	@tox -e func
+
 # The targets below don't depend on a file
 .PHONY: lint test unittest release clean help build build-verify
diff --git a/README.md b/README.md
index 592db45..29c3e8e 100644
--- a/README.md
+++ b/README.md
@@ -55,6 +55,10 @@ Don't forget to add unit and functional tests, and run:
 make test
 ```
 
+For functional tests some applications require the use of a VIP.  Please ensure
+you set the environment variable `OS_VIP00` to a suitable VIP address before
+running functional tests.
+
 ## Submit a bug
 
 If you prefer, file a bug or feature request at:
diff --git a/tests/functional/requirements.txt b/tests/functional/requirements.txt
new file mode 100644
index 0000000..e6b19ec
--- /dev/null
+++ b/tests/functional/requirements.txt
@@ -0,0 +1,2 @@
+python-openstackclient
+git+https://github.com/openstack-charmers/zaza.git#egg=zaza
diff --git a/tests/functional/tests/base.py b/tests/functional/tests/base.py
new file mode 100644
index 0000000..b95ba53
--- /dev/null
+++ b/tests/functional/tests/base.py
@@ -0,0 +1,21 @@
+"""Generic setup for functional tests."""
+
+import unittest
+
+from juju import loop
+import zaza.model
+
+from juju_verify import juju_verify
+
+
+class BaseTestCase(unittest.TestCase):
+    """Base class for functional testing of verifiers."""
+
+    @classmethod
+    def setUpClass(cls):
+        """Run class setup for running tests."""
+        super(BaseTestCase, cls).setUpClass()
+        cls.log_level = 'info'
+        cls.check = 'shutdown'
+        juju_verify.config_logger(cls.log_level)
+        cls.model = loop.run(juju_verify.connect_model(zaza.model.CURRENT_MODEL))
diff --git a/tests/functional/tests/bundles/ceph.yaml b/tests/functional/tests/bundles/ceph.yaml
new file mode 100644
index 0000000..b13e046
--- /dev/null
+++ b/tests/functional/tests/bundles/ceph.yaml
@@ -0,0 +1,16 @@
+series: xenial
+applications:
+  ceph-mon:
+    charm: cs:~xavpaice/ceph-mon  # https://review.opendev.org/c/openstack/charm-ceph-mon/+/778837
+    num_units: 3
+    options:
+      expected-osd-count: 3
+      monitor-count: 3
+  ceph-osd:
+    charm: cs:ceph-osd
+    num_units: 3
+    options:
+      osd-devices: /srv/osd
+relations:
+- - ceph-osd:mon
+  - ceph-mon:osd
diff --git a/tests/functional/tests/bundles/hacluster.yaml b/tests/functional/tests/bundles/hacluster.yaml
new file mode 100644
index 0000000..dd166b7
--- /dev/null
+++ b/tests/functional/tests/bundles/hacluster.yaml
@@ -0,0 +1,26 @@
+series: xenial
+applications:
+  dashboard-hacluster:
+    charm: cs:~xavpaice/hacluster-0
+  keystone:
+    charm: cs:keystone
+    num_units: 1
+  mysql:
+    charm: cs:percona-cluster
+    num_units: 1
+  nrpe:
+    charm: cs:nrpe
+  openstack-dashboard:
+    charm: cs:openstack-dashboard
+    num_units: 3
+    options:
+      vip: 10.5.0.250
+relations:
+- - openstack-dashboard:ha
+  - dashboard-hacluster:ha
+- - mysql:shared-db
+  - keystone:shared-db
+- - openstack-dashboard:identity-service
+  - keystone:identity-service
+- - nrpe:nrpe-external-master
+  - openstack-dashboard:nrpe-external-master
diff --git a/tests/functional/tests/bundles/nova-compute.yaml b/tests/functional/tests/bundles/nova-compute.yaml
new file mode 100644
index 0000000..865ae8b
--- /dev/null
+++ b/tests/functional/tests/bundles/nova-compute.yaml
@@ -0,0 +1,106 @@
+series: bionic
+relations:
+- - nova-compute:image-service
+  - glance:image-service
+- - nova-compute:amqp
+  - rabbitmq-server:amqp
+- - nova-cloud-controller:shared-db
+  - percona-cluster:shared-db
+- - nova-cloud-controller:identity-service
+  - keystone:identity-service
+- - nova-cloud-controller:amqp
+  - rabbitmq-server:amqp
+- - nova-cloud-controller:cloud-compute
+  - nova-compute:cloud-compute
+- - nova-cloud-controller:image-service
+  - glance:image-service
+- - keystone:shared-db
+  - percona-cluster:shared-db
+- - glance:identity-service
+  - keystone:identity-service
+- - glance:shared-db
+  - percona-cluster:shared-db
+- - glance:amqp
+  - rabbitmq-server:amqp
+- - neutron-gateway:amqp
+  - rabbitmq-server:amqp
+- - nova-cloud-controller:quantum-network-service
+  - neutron-gateway:quantum-network-service
+- - neutron-api:shared-db
+  - percona-cluster:shared-db
+- - neutron-api:amqp
+  - rabbitmq-server:amqp
+- - neutron-api:neutron-api
+  - nova-cloud-controller:neutron-api
+- - neutron-api:identity-service
+  - keystone:identity-service
+- - nova-compute:neutron-plugin
+  - neutron-openvswitch:neutron-plugin
+- - rabbitmq-server:amqp
+  - neutron-openvswitch:amqp
+- - placement:shared-db
+  - percona-cluster:shared-db
+- - placement:identity-service
+  - keystone:identity-service
+- - placement:placement
+  - nova-cloud-controller:placement
+applications:
+  rabbitmq-server:
+    charm: cs:~openstack-charmers-next/rabbitmq-server
+    num_units: 1
+  percona-cluster:
+    charm: cs:~openstack-charmers-next/percona-cluster
+    num_units: 1
+    options:
+      max-connections: 1000
+      innodb-buffer-pool-size: 256M
+  nova-cloud-controller:
+    charm: cs:~openstack-charmers-next/nova-cloud-controller
+    num_units: 1
+    options:
+      openstack-origin: cloud:bionic-ussuri
+      network-manager: Neutron
+      debug: true
+  neutron-api:
+    charm: cs:~openstack-charmers-next/neutron-api
+    num_units: 1
+    options:
+      manage-neutron-plugin-legacy-mode: true
+      openstack-origin: cloud:bionic-ussuri
+      flat-network-providers: physnet1
+      neutron-security-groups: true
+  keystone:
+    charm: cs:~openstack-charmers-next/keystone
+    num_units: 1
+    options:
+      openstack-origin: cloud:bionic-ussuri
+  neutron-gateway:
+    charm: cs:~openstack-charmers-next/neutron-gateway
+    num_units: 1
+    options:
+      openstack-origin: cloud:bionic-ussuri
+      bridge-mappings: physnet1:br-ex
+  glance:
+    charm: cs:~openstack-charmers-next/glance
+    num_units: 1
+    options:
+      openstack-origin: cloud:bionic-ussuri
+  neutron-openvswitch:
+    charm: cs:~openstack-charmers-next/neutron-openvswitch
+  placement:
+    charm: cs:~openstack-charmers-next/placement
+    num_units: 1
+    options:
+      openstack-origin: cloud:bionic-ussuri
+  nova-compute:
+    charm: cs:~martin-kalcok/nova-compute-0
+    num_units: 1
+    constraints: mem=4G cores=4
+    options:
+      openstack-origin: cloud:bionic-ussuri
+      config-flags: auto_assign_floating_ip=False
+      enable-live-migration: false
+      aa-profile-mode: enforce
+      ephemeral-device: /dev/vdb
+      ephemeral-unmount: /mnt
+      debug: true
diff --git a/tests/functional/tests/bundles/overlays/hacluster.yaml.j2 b/tests/functional/tests/bundles/overlays/hacluster.yaml.j2
new file mode 100644
index 0000000..829e57a
--- /dev/null
+++ b/tests/functional/tests/bundles/overlays/hacluster.yaml.j2
@@ -0,0 +1,4 @@
+applications:
+  openstack-dashboard:
+    options:
+      vip: '{{ OS_VIP00 }}'
diff --git a/tests/functional/tests/test_ceph.py b/tests/functional/tests/test_ceph.py
new file mode 100644
index 0000000..4fce49e
--- /dev/null
+++ b/tests/functional/tests/test_ceph.py
@@ -0,0 +1,26 @@
+"""Test deployment and functionality of juju-verify."""
+
+from juju import loop
+from tests.base import BaseTestCase
+
+from juju_verify import juju_verify
+from juju_verify.verifiers import get_verifier
+
+
+class CephTests(BaseTestCase):
+    """Functional testing for Ceph verifier."""
+
+    # To add:
+    # test not OK for two ceph-osd unit stops
+    # test we get a warning when a machine also hosts another unit
+
+    def test_single_osd_unit(self):
+        """Test that shutdown of a single unit returns OK."""
+        # juju-verify shutdown --units ceph-osd/1
+
+        units = ['ceph-osd/0']
+        check = 'shutdown'
+        unit_objects = loop.run(juju_verify.find_units(self.model, units))
+        verifier = get_verifier(unit_objects)
+        result = verifier.verify(check)
+        self.assertTrue(result.success is True)
diff --git a/tests/functional/tests/test_novacompute.py b/tests/functional/tests/test_novacompute.py
new file mode 100644
index 0000000..4bd8a6b
--- /dev/null
+++ b/tests/functional/tests/test_novacompute.py
@@ -0,0 +1,20 @@
+"""Test deployment and functionality of juju-verify."""
+
+from juju import loop
+from tests.base import BaseTestCase
+
+from juju_verify import juju_verify
+from juju_verify.verifiers import get_verifier
+
+
+class NovaCompute(BaseTestCase):
+    """Base class for functional testing of nova-compute verifier."""
+
+    def test_single_unit(self):
+        """Test that shutdown of a single unit returns OK."""
+        units = ['nova-compute/0']
+        check = 'shutdown'
+        unit_objects = loop.run(juju_verify.find_units(self.model, units))
+        verifier = get_verifier(unit_objects)
+        result = verifier.verify(check)
+        print(str(result))
diff --git a/tests/functional/tests/tests.yaml b/tests/functional/tests/tests.yaml
new file mode 100644
index 0000000..c4355fa
--- /dev/null
+++ b/tests/functional/tests/tests.yaml
@@ -0,0 +1,16 @@
+tests:
+  - nova-compute:
+    - tests.test_novacompute.NovaCompute
+  - ceph:
+    - tests.test_ceph.CephTests
+gate_bundles:
+  #- nova-compute: nova-compute
+  - ceph: ceph
+configure:
+  nova-comppute:
+    - zaza.openstack.charm_tests.glance.setup.add_cirros_image
+    - zaza.openstack.charm_tests.glance.setup.add_lts_image
+    - zaza.openstack.charm_tests.keystone.setup.add_demo_user
+    - zaza.openstack.charm_tests.neutron.setup.basic_overcloud_network
+    - zaza.openstack.charm_tests.nova.setup.create_flavors
+    - zaza.openstack.charm_tests.nova.setup.manage_ssh_key
diff --git a/tox.ini b/tox.ini
index db91995..0cce781 100644
--- a/tox.ini
+++ b/tox.ini
@@ -1,12 +1,26 @@
 [tox]
 skipsdist=True
-envlist = lint, unit
+envlist = lint, unit, func
 skip_missing_interpreters = True
 
 [testenv]
 basepython = python3
 setenv =
-  PYTHONPATH = .
+  PYTHONPATH = .:tests/functional
+passenv =
+  HOME
+  PATH
+  PYTEST_KEEP_MODEL
+  PYTEST_CLOUD_NAME
+  PYTEST_CLOUD_REGION
+  PYTEST_MODEL
+  MODEL_SETTINGS
+  HTTP_PROXY
+  HTTPS_PROXY
+  NO_PROXY
+  SNAP_HTTP_PROXY
+  SNAP_HTTPS_PROXY
+  OS_*
 
 [testenv:unit]
 deps = .[dev]
@@ -16,23 +30,12 @@ commands = pytest -v \
     --cov-report=term \
     --cov-report=annotate:report/annotated \
     --cov-report=html:report/html \
-    tests
-
-[testenv:functional]
-passenv =
-  HOME
-  JUJU_REPOSITORY
-  PATH
-deps = .[dev]
-commands = pytest -v --ignore {toxinidir}/tests/unit \
-    --cov=juju_verify \
-    --cov-report=term \
-    --cov-report=annotate:report/annotated \
-    --cov-report=html:report/html \
-    tests
+    tests/unit
 
 [testenv:lint]
-deps = .[dev]
+deps =
+  .[dev]
+  -r{toxinidir}/tests/functional/requirements.txt
 commands =
     flake8 {toxinidir}/juju_verify/ {toxinidir}/tests/
     mypy {toxinidir}/juju_verify/
@@ -67,3 +70,23 @@ application_import_names = juju_verify
 import-order-style = pep8
 max-line-length = 89
 max-complexity = 10
+
+
+[testenv:func]
+changedir = {toxinidir}/tests/functional
+commands =
+    pip install --editable {toxinidir}/
+    functest-run-suite {posargs}
+deps =
+  .[dev]
+  -r{toxinidir}/tests/functional/requirements.txt
+
+
+[testenv:func-debug]
+changedir = {toxinidir}/tests/functional
+commands =
+    pip install --editable {toxinidir}/
+    functest-run-suite {posargs} --keep-model --log DEBUG
+deps =
+  .[dev]
+  -r{toxinidir}/tests/functional/requirements.txt