Merge lp:~stub/charm-helpers/test-harness into lp:charm-helpers

[Stuart Bishop]

This is a Python test fixture for driving Juju, simplifying the task of writing charm tests.

charm-helpers seems a suitable home for this. contrib for now, as the API will likely change when Juju features like containers land.

[Matthew Wedgwood (mew)]

I think this is great, and and I agree that charmhelpers is the right place for this. Here's what I think it needs before merge:

* Combine the two modules into one. run() looks lonely.
* make lint, fix the issues
* Install python-fixtures for the user if they don't already have it installed. See charmhelpers.fetch.bzrurl.
* Move this to charmhelpers/testing/__init__.py (out of contrib)
* Add some docs, even if it's simply a description and example in the module docstring.

Thanks!

review: Needs Fixing

[lp:~stub/charm-helpers/test-harness]

44. By Stuart Bishop on 2013-09-24

Remove run.py

45. By Stuart Bishop on 2013-09-24

Move out of contrib

46. By Stuart Bishop on 2013-09-24

Merged bug-1214793-service-wrappers into test-harness.

47. By Stuart Bishop on 2013-10-09

delint

48. By Stuart Bishop on 2013-10-09

Some basic docs for the testing infrastructure

49. By Stuart Bishop on 2013-10-09

update docs

[Stuart Bishop (stub)]

All done, apart from installing python-fixtures automatically as it doesn't make sense for this use case.

[Stuart Bishop (stub)]

I'm thinking that a better home would be in amulet. It is already duplicating some of its functionality.

[lp:~stub/charm-helpers/test-harness]

50. By Stuart Bishop on 2013-10-10

Merged bug-1214793-service-wrappers into test-harness.

Unmerged revisions

52. By Stuart Bishop on 2014-06-05

Merged bug-1214793-service-wrappers into test-harness.

51. By Stuart Bishop on 2014-04-02

Merged bug-1214793-service-wrappers into test-harness.

50. By Stuart Bishop on 2013-10-10

Merged bug-1214793-service-wrappers into test-harness.

49. By Stuart Bishop on 2013-10-09

update docs

48. By Stuart Bishop on 2013-10-09

Some basic docs for the testing infrastructure

47. By Stuart Bishop on 2013-10-09

delint

46. By Stuart Bishop on 2013-09-24

Merged bug-1214793-service-wrappers into test-harness.

45. By Stuart Bishop on 2013-09-24

Move out of contrib

44. By Stuart Bishop on 2013-09-24

Remove run.py

43. By Stuart Bishop on 2013-08-23

Extract JujuFixture from postgresql charm

=== added directory 'charmhelpers/testing'
=== added file 'charmhelpers/testing/README'
--- charmhelpers/testing/README	1970-01-01 00:00:00 +0000
+++ charmhelpers/testing/README	2013-10-09 09:59:44 +0000
@@ -0,0 +1,36 @@
+This package provides a Juju test fixture to help you write tests for
+your charms as a Python test suite. It requires you to already have a
+bootstrapped environment.
+
+Below is a simple example using the fixture. You can find a more extensive
+example in the PostgreSQL charm.
+
+#!/usr/bin/python
+import os
+import fixtures
+import testtools
+import unittest
+from charmhelpers.testing.jujufixture import JujuFixture
+
+
+class CharmTestCase(testtools.TestCase, fixtures.TestWithFixtures):
+
+    def setUp(self):
+        super(CharmTestCase, self).setUp()
+
+        self.juju = self.useFixture(JujuFixture(reuse_machines=True))
+
+        # If the charms fail, we don't want tests to hang indefinitely.
+        timeout = int(os.environ.get('TEST_TIMEOUT', 900))
+        self.useFixture(fixtures.Timeout(timeout, gentle=True))
+
+    def test_basic(self):
+        self.juju.deploy('local:mycharm', 'test-mycharm', num_units=1)
+        self.juju.add_unit('local:mycharm', 'test-mycharm', num_units=2)
+        self.juju.deploy('local:othercharm')
+        self.juju.do(['add-relation', 'test-mycharm:rel', 'othercharm:rel'])
+        self.juju.wait_until_ready()
+        [ ... test the deployment ... ]
+
+if __name__ == '__main__':
+    unittest.main()

=== added file 'charmhelpers/testing/__init__.py'
--- charmhelpers/testing/__init__.py	1970-01-01 00:00:00 +0000
+++ charmhelpers/testing/__init__.py	2013-10-09 09:59:44 +0000
@@ -0,0 +1,3 @@
+from jujufixture import JujuFixture, run
+_ = JujuFixture, run
+del _

=== added file 'charmhelpers/testing/jujufixture.py'
--- charmhelpers/testing/jujufixture.py	1970-01-01 00:00:00 +0000
+++ charmhelpers/testing/jujufixture.py	2013-10-09 09:59:44 +0000
@@ -0,0 +1,193 @@
+import json
+import subprocess
+import time
+
+import fixtures
+from testtools.content import text_content
+
+
+__all__ = ['JujuFixture', 'run']
+
+
+class JujuFixture(fixtures.Fixture):
+    """Interact with juju.
+
+    Assumes juju environment is bootstrapped.
+    """
+
+    def __init__(self, reuse_machines=False, do_teardown=True):
+        super(JujuFixture, self).__init__()
+
+        self._deployed_charms = set()
+
+        self.reuse_machines = reuse_machines
+
+        # Optionally, don't teardown services and machines after running
+        # a test. If a subsequent test is run, they will be torn down at
+        # that point. This option is only useful when running a single
+        # test, or when the test harness is set to abort after the first
+        # failed test.
+        self.do_teardown = do_teardown
+
+        self._deployed_services = set()
+
+    def do(self, cmd):
+        cmd = ['juju'] + cmd
+        run(self, cmd)
+
+    def get_result(self, cmd):
+        cmd = ['juju'] + cmd + ['--format=json']
+        out = run(self, cmd)
+        if out:
+            return json.loads(out)
+        return None
+
+    def deploy(self, charm, name=None, num_units=1):
+        # The first time we deploy a local: charm in the test run, it
+        # needs to deploy with --update to ensure we are testing the
+        # desired revision of the charm. Subsequent deploys we do not
+        # use --update to avoid overhead and needless incrementing of the
+        # revision number.
+        if not charm.startswith('local:') or charm in self._deployed_charms:
+            cmd = ['deploy']
+        else:
+            cmd = ['deploy', '-u']
+            self._deployed_charms.add(charm)
+
+        cmd.append(charm)
+
+        if name is None:
+            name = charm.split(':', 1)[-1]
+
+        cmd.append(name)
+        self._deployed_services.add(name)
+
+        if self.reuse_machines and self._free_machines:
+            cmd.extend(['--to', str(self._free_machines.pop())])
+            self.do(cmd)
+            if num_units > 1:
+                self.add_unit(charm, name, num_units - 1)
+        else:
+            cmd.extend(['-n', str(num_units)])
+            self.do(cmd)
+
+    def add_unit(self, charm, name=None, num_units=1):
+        if name is None:
+            name = charm.split(':', 1)[-1]
+
+        num_units_spawned = 0
+        while self.reuse_machines and self._free_machines:
+            cmd = ['add-unit', '--to', str(self._free_machines.pop()), name]
+            self.do(cmd)
+            num_units_spawned += 1
+            if num_units_spawned == num_units:
+                return
+
+        cmd = ['add-unit', '-n', str(num_units - num_units_spawned), name]
+        self.do(cmd)
+
+    # The most recent environment status, updated by refresh_status()
+    status = None
+
+    def refresh_status(self):
+        self.status = self.get_result(['status'])
+
+        self._free_machines = set(
+            int(k) for k in self.status['machines'].keys()
+            if k != '0')
+        for service in self.status.get('services', {}).values():
+            for unit in service.get('units', []):
+                if 'machine' in unit:
+                    self._free_machines.remove(int(unit['machine']))
+
+        return self.status
+
+    def wait_until_ready(self, extra=45):
+        ready = False
+        while not ready:
+            self.refresh_status()
+            ready = True
+            for service in self.status['services']:
+                if self.status['services'][service].get('life', '') == 'dying':
+                    ready = False
+                units = self.status['services'][service].get('units', {})
+                for unit in units.keys():
+                    agent_state = units[unit].get('agent-state', '')
+                    if agent_state == 'error':
+                        raise RuntimeError('{} error: {}'.format(
+                            unit, units[unit].get('agent-state-info', '')))
+                    if agent_state != 'started':
+                        ready = False
+            time.sleep(1)
+        # Unfortunately, there is no way to tell when a system is
+        # actually ready for us to test. Juju only tells us that a
+        # relation has started being setup, and that no errors have been
+        # encountered yet. It utterly fails to inform us when the
+        # cascade of hooks this triggers has finished and the
+        # environment is in a stable and actually testable state.
+        # So as a work around for Bug #1200267, we need to sleep long
+        # enough that our system is probably stable. This means we have
+        # extremely slow and flaky tests, but that is possibly better
+        # than no tests.
+        time.sleep(extra)
+
+    def setUp(self):
+        super(JujuFixture, self).setUp()
+        self.reset()
+        if self.do_teardown:
+            self.addCleanup(self.reset)
+
+    def reset(self):
+        # Tear down any services left running that we know we spawned.
+        while True:
+            found_services = False
+            self.refresh_status()
+
+            # Kill any services started by the deploy() method.
+            for service_name, service in self.status.get(
+                    'services', {}).items():
+                if service_name in self._deployed_services:
+                    found_services = True
+                    if service.get('life', '') != 'dying':
+                        self.do(['destroy-service', service_name])
+                    # If any units have failed hooks, unstick them.
+                    for unit_name, unit in service.get('units', {}).items():
+                        if unit.get('agent-state', None) == 'error':
+                            self.do(['resolved', unit_name])
+            if not found_services:
+                break
+            time.sleep(1)
+
+        self._deployed_services = set()
+
+        # We need to wait for dying services
+        # to die before we can continue.
+        if found_services:
+            self.wait_until_ready(0)
+
+        # We shouldn't reuse machines, as we have no guarantee they are
+        # still in a usable state, so tear them down too. Per
+        # Bug #1190492 (INVALID), in the future this will be much nicer
+        # when we can use containers for isolation and can happily reuse
+        # machines.
+        if not self.reuse_machines:
+            self.do(['terminate-machine'] + list(self._free_machines))
+
+
+def run(detail_collector, cmd, input=''):
+    try:
+        proc = subprocess.Popen(
+            cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE)
+    except subprocess.CalledProcessError:
+        raise
+
+    (out, err) = proc.communicate(input)
+    if out:
+        detail_collector.addDetail('stdout', text_content(out))
+    if err:
+        detail_collector.addDetail('stderr', text_content(err))
+    if proc.returncode != 0:
+        raise subprocess.CalledProcessError(
+            proc.returncode, cmd, err)
+    return out