Merge ~sajoupa/charm-k8s-telegraf:func_tests into charm-k8s-telegraf:master

[Laurent Sesquès]

Add functional tests

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

This merge proposal is being monitored by mergebot. Change the status to Approved to merge.

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Unable to determine commit message from repository - please click "Set commit message" and enter the commit message manually.

[John Lenton (chipaca)]

Lovely, thank you!

review: Approve

[Benjamin Allot (ballot)]

Few comments, more for clarity than anything else !

[Laurent Sesquès (sajoupa)]

> Few comments, more for clarity than anything else !

Updated, except for the regex part, will reply inline.

[Laurent Sesquès (sajoupa)]

Replied to inline comments.

[Benjamin Allot (ballot)]

Small one I missed on Makefile, you can merge without it IMO, given our Makefile usage, but if it's a copy/paste from another, better to have the best approach.

Ok for the regexp if it takes milliseconds.

Bottom approved, feel free to fix the makefile or merge as is.

review: Approve

[Benjamin Allot (ballot)]

In fact, I have the following error when I try to run the functional tests:

Executing functional tests in /home/ballot/charms/builds
functional create: /home/ballot/repos/charms/charm-k8s-telegraf/.tox/functional
functional installdeps: -r/home/ballot/repos/charms/charm-k8s-telegraf/tests/functional/requirements.txt
functional installed: appdirs==1.4.3,async-generator==1.10,bcrypt==3.2.0,CacheControl==0.12.6,certifi==2019.11.28,cffi==1.14.3,chardet==3.0.4,colorama==0.4.3,contextlib2==0.6.0,cryptography==3.2.1,debtcollector==2.2.0,distlib==0.3.0,distro==1.4.0,html5lib==1.0.1,hvac==0.6.4,idna==2.8,ipaddr==2.2.0,Jinja2==2.11.2,juju==2.8.4,juju-wait==2.8.4,jujubundlelib==0.5.6,lockfile==0.12.2,macaroonbakery==1.3.1,MarkupSafe==1.1.1,msgpack==0.6.2,mypy-extensions==0.4.3,netaddr==0.8.0,ops==1.0.1,oslo.config==6.11.2,oslo.i18n==5.0.1,packaging==20.3,paramiko==2.7.2,pbr==5.5.1,pep517==0.8.2,pkg-resources==0.0.0,progress==1.5,protobuf==3.13.0,pyasn1==0.4.8,pycparser==2.20,pymacaroons==0.13.0,PyNaCl==1.4.0,pyparsing==2.4.6,pyRFC3339==1.1,pytoml==0.1.21,pytz==2020.4,PyYAML==5.3.1,requests==2.22.0,retrying==1.3.3,rfc3986==1.4.0,six==1.14.0,stevedore==3.2.2,tenacity==6.2.0,theblues==0.5.2,toposort==1.5,typing-extensions==3.7.4.3,typing-inspect==0.6.0,urllib3==1.25.8,webencodings==0.5.1,websockets==7.0,wrapt==1.12.1,zaza==0.0.2.dev1
functional run-test-pre: PYTHONHASHSEED='3098041088'
functional run-test: commands[0] | functest-run-suite
2020-11-11 23:14:29 [WARNING] unknown delta type: id
2020-11-11 23:14:29 [INFO] Deploying bundle './tests/bundles/base.yaml' on to 'zaza-5567d93822fa' model
2020-11-11 23:14:29 [INFO] Rendered template '<Template 'local-charm-overlay.yaml.j2'>' to file '/tmp/tmp5z33c9e1/local-charm-overlay.yaml'
2020-11-11 23:14:29 [INFO] Deploying overlay '/tmp/tmp5z33c9e1/local-charm-overlay.yaml' on to 'zaza-5567d93822fa' model
2020-11-11 23:14:31 [INFO] ERROR cannot deploy bundle: the provided bundle has the following errors:
2020-11-11 23:14:31 [INFO] charm path in application "telegraf" does not exist: /home/ballot/charms/buildstelegraf.charm
2020-11-11 23:14:31 [ERROR] {'default_alias': 'zaza-5567d93822fa'}
2020-11-11 23:14:31 [ERROR] Model default_alias (zaza-5567d93822fa)
2020-11-11 23:14:31 [WARNING] unknown delta type: id
Traceback (most recent call last):
  File "/home/ballot/repos/charms/charm-k8s-telegraf/.tox/functional/bin/functest-run-suite", line 8, in <module>
    sys.exit(main())
  File "/home/ballot/repos/charms/charm-k8s-telegraf/.tox/functional/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 259, in main
    func_test_runner(
  File "/home/ballot/repos/charms/charm-k8s-telegraf/.tox/functional/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 200, in func_test_runner
    run_env_deployment(env_deployment, keep_model=preserve_model,
  File "/home/ballot/repos/charms/charm-k8s-telegraf/.tox/functional/lib/python3.8/site-packages/zaza/charm_lifecycle/func_test_runner.py", line 116, in run_env_deployment
    deploy.deploy(
  File "/home/ballot/repos/charms/charm-k8s-telegraf/.tox/functional/lib/python3.8/site-packages/zaza/charm_lifecycle/deploy.py", line 364, in de...

review: Needs Fixing

[Benjamin Allot (ballot)]

LGTM, tested and working on my microk8s !

review: Approve

[🤖 Canonical IS Merge Bot (canonical-is-mergebot)]

Change successfully merged at revision f45cd5af2ded313a2eefd6543fa39dd5c04fce23

diff --git a/.gitignore b/.gitignore
index 490cc43..ae614f7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -4,3 +4,6 @@
 .coverage
 __pycache__
 build
+
+# version data
+repo-info
diff --git a/Makefile b/Makefile
index f24682f..7a71e0f 100644
--- a/Makefile
+++ b/Makefile
@@ -1,5 +1,10 @@
 VERSION_TO_BUILD ?= latest
 
+PROJECTPATH=$(dir $(realpath $(lastword $(MAKEFILE_LIST))))
+CHARM_BUILD_DIR ?= ${PROJECTPATH:%/=%}
+METADATA_FILE="metadata.yaml"
+CHARM_NAME=$(shell cat ${PROJECTPATH}/${METADATA_FILE} | awk '/^name:/ {print $$2".charm"}')
+
 blacken:
 	@echo "Normalising python layout with black."
 	@tox -e black
@@ -11,17 +16,26 @@ lint: blacken
 
 # We actually use the build directory created by charmcraft,
 # but the .charm file makes a much more convenient sentinel.
-unittest: telegraf.charm
+unittests:
 	@tox -e unit
 
-test: lint unittest
+build:
+	@echo "Building charm to ${CHARM_BUILD_DIR}/${CHARM_NAME}"
+	@-git rev-parse HEAD > ./repo-info
+	@cd ${CHARM_BUILD_DIR} && TERM=linux charmcraft build -f ${PROJECTPATH}
+
+test: lint unittests functional
+	@echo "Tests completed for charm ${CHARM_NAME}."
+
+functional: build
+	@echo "Executing functional tests in ${CHARM_BUILD_DIR}"
+	@CHARM_BUILD_DIR=${CHARM_BUILD_DIR} tox -e functional
 
 clean:
 	@echo "Cleaning files"
-	@git clean -fXd
-
-telegraf.charm: src/*.py requirements.txt
-	charmcraft build
+	@git clean -fXd -e '!.idea'
+	@echo "Cleaning existing build"
+	@rm -rf ${CHARM_BUILD_DIR}/${CHARM_NAME}
 
 image-build:
 	@echo "Building the image."
@@ -31,4 +45,4 @@ image-build:
 		-t telegraf:$(VERSION_TO_BUILD) \
 		.
 
-.PHONY: blacken lint unittest test clean image-build
+.PHONY: blacken lint unittests test clean image-build build
diff --git a/tests/functional/requirements.txt b/tests/functional/requirements.txt
new file mode 100644
index 0000000..b7c9112
--- /dev/null
+++ b/tests/functional/requirements.txt
@@ -0,0 +1 @@
+git+https://github.com/openstack-charmers/zaza.git#egg=zaza
diff --git a/tests/functional/tests/bundles/base.yaml b/tests/functional/tests/bundles/base.yaml
new file mode 100644
index 0000000..f12ac73
--- /dev/null
+++ b/tests/functional/tests/bundles/base.yaml
@@ -0,0 +1,4 @@
+bundle: kubernetes
+applications:
+  telegraf:
+    scale: 1
diff --git a/tests/functional/tests/bundles/overlays/local-charm-overlay.yaml.j2 b/tests/functional/tests/bundles/overlays/local-charm-overlay.yaml.j2
new file mode 100644
index 0000000..d19c84a
--- /dev/null
+++ b/tests/functional/tests/bundles/overlays/local-charm-overlay.yaml.j2
@@ -0,0 +1,3 @@
+applications:
+  {{ charm_name }}:
+    charm: "{{ CHARM_BUILD_DIR }}/{{ charm_name }}.charm"
diff --git a/tests/functional/tests/test_telegraf.py b/tests/functional/tests/test_telegraf.py
new file mode 100644
index 0000000..523067e
--- /dev/null
+++ b/tests/functional/tests/test_telegraf.py
@@ -0,0 +1,63 @@
+import logging
+import requests
+import re
+import time
+import unittest
+
+from zaza import model
+
+DEFAULT_HTTPGET_TIMEOUT = 10
+DEFAULT_RETRIES = 12
+DEFAULT_TELEGRAF_EXPORTER_PORT = "9103"
+
+
+class BaseTelegrafTest(unittest.TestCase):
+    """Base for Telegraf charm tests."""
+
+    @classmethod
+    def setUpClass(cls):
+        super(BaseTelegrafTest, cls).setUpClass()
+        cls.model_name = model.get_juju_model()
+        cls.application_name = "telegraf"
+        cls.lead_unit_name = model.get_lead_unit_name(cls.application_name, model_name=cls.model_name)
+        logging.debug("Leader unit is {}".format(cls.lead_unit_name))
+
+
+class TestTelegraf(BaseTelegrafTest):
+    def check_re_pattern(self, re_pattern, text):
+        logging.info("checking metrics %s", re_pattern)
+        # findall returns a list, [] when no match
+        self.assertTrue(re.findall(re_pattern, text, flags=re.M))
+
+    def test_01_service(self):
+        status = model.get_status()
+        ip = status['applications']['telegraf']['public-address']
+        url = "http://{}:{}/metrics".format(ip, DEFAULT_TELEGRAF_EXPORTER_PORT)
+        # zaza.model.block_until_all_units_idle doesn't wait long enough in k8s models
+        # https://github.com/openstack-charmers/zaza/issues/396
+        time.sleep(15)
+
+        re_patterns = [
+            r"^internal_agent_",
+            r"^internal_memstats_",
+        ]
+
+        for retry in range(DEFAULT_RETRIES):
+            resp = requests.get(url, timeout=DEFAULT_HTTPGET_TIMEOUT)
+            self.assertEqual(resp.status_code, 200)
+
+            logging.info("test_service: endpoint {}:{} is reachable".format(ip, DEFAULT_TELEGRAF_EXPORTER_PORT))
+
+            resp = requests.get(url)
+            self.assertTrue(resp.ok)
+            text = resp.text
+
+            # check that metrics are available
+            for re_pattern in re_patterns:
+                self.check_re_pattern(re_pattern, text)
+
+            break
+        else:
+            msg = "test_service: endpoint {}:{} is not reachable".format(ip, DEFAULT_TELEGRAF_EXPORTER_PORT)
+            logging.error(msg)
+            self.fail(msg)
diff --git a/tests/functional/tests/tests.yaml b/tests/functional/tests/tests.yaml
new file mode 100644
index 0000000..5b7d050
--- /dev/null
+++ b/tests/functional/tests/tests.yaml
@@ -0,0 +1,8 @@
+charm_name: telegraf
+gate_bundles:
+  - base
+target_deploy_status:
+  telegraf:
+    workload-status-message: ''
+tests:
+  - tests.test_telegraf.TestTelegraf
diff --git a/tox.ini b/tox.ini
index 48bf1ab..8935a0b 100644
--- a/tox.ini
+++ b/tox.ini
@@ -7,6 +7,20 @@ skip_missing_interpreters = True
 basepython = python3
 setenv =
   PYTHONPATH = {toxinidir}/build/lib:{toxinidir}/build/venv
+passenv =
+  HOME
+  PATH
+  CHARM_BUILD_DIR
+  PYTEST_KEEP_MODEL
+  PYTEST_CLOUD_NAME
+  PYTEST_CLOUD_REGION
+  PYTEST_MODEL
+  MODEL_SETTINGS
+  HTTP_PROXY
+  HTTPS_PROXY
+  NO_PROXY
+  SNAP_HTTP_PROXY
+  SNAP_HTTPS_PROXY
 
 [testenv:unit]
 commands =
@@ -19,14 +33,9 @@ setenv =
   TZ=UTC
 
 [testenv:functional]
-passenv =
-  HOME
-  JUJU_REPOSITORY
-  PATH
-commands =
-	pytest -v --ignore mod --ignore {toxinidir}/tests/unit {posargs}
+changedir = {toxinidir}/tests/functional
+commands = functest-run-suite {posargs}
 deps = -r{toxinidir}/tests/functional/requirements.txt
-       -r{toxinidir}/requirements.txt
 
 [testenv:black]
 commands = black --skip-string-normalization --line-length=120 src/ tests/