Merge lp:~hduran-8/juju-ci-tools/add_status_ci_tests into lp:juju-ci-tools

[Horacio Durán]

Added Status CI tests.

CI tests where added to ensure that the format of various
formats of status output remain the same through changes.
The formats covered are Tabular, yaml and json and the
testing is done by getting the same status in all formats
and ensuring all expected data is there and that there is
no unexpected data not covered by testing.

[lp:~hduran-8/juju-ci-tools/add_status_ci_tests]

981. By Horacio Durán on 2015-06-16

Merge with master and resolve conflict in jujupy.py

[Curtis Hovey (sinzui)]

Thank you Horatio. This was a long branch to review. I like the actual testing done in this branch, but I believe there are easier ways to do it. My comments are inline. I also like the idea of a template to write new tests, and I think there is a context manager that makes writing test easier than what you choose.

Please run make lint to address style issues. you may want to try autopep8 which claims to reformat python to pep8 and pep257. In my experience, there will be some reformatting that is not specified by pep8

review: Needs Fixing (code)

[lp:~hduran-8/juju-ci-tools/add_status_ci_tests]

982. By Horacio Durán on 2015-06-18

Proper python formatting.

983. By Horacio Durán on 2015-06-22

Fixed a couple of bugs on parse args.

[Curtis Hovey (sinzui)]

Hi Horatio. Sorry for taking so long to review this branch. We normally keep branches to 400 lines.

I think there is too much duplication with other feature in juju-ci-tools and Python itself. My comments are inline, but in general, there is already a helper to bring up and tear down an env, you should use it because CI needs more control over what is under test then your main() provides. We do not want to write assertions when Python provides them for free. I think all the calls to assert can be written in fewer lines using unittest.
     https://docs.python.org/2/library/unittest.html#unittest.FunctionTestCase

review: Needs Fixing (code)

[lp:~hduran-8/juju-ci-tools/add_status_ci_tests]

984. By Horacio Durán on 2015-07-10

Addressed curtis observations.

I refactored most of my tests to suit CI team
style and requirements as per Curtis comments.

[Horacio Durán (hduran-8)]

Hey the only thing I dont agree with is the issue about the literals for status output, I added a comment there, I dont really think we should use equivalent since the literal status output is what we are testing, I added more detail in the comment.

[lp:~hduran-8/juju-ci-tools/add_status_ci_tests]

985. By Horacio Durán on 2015-07-10

Addressed extra comments from Curtis.

[Curtis Hovey (sinzui)]

Thank you Horatio.

I will merge your branch. in a few minutes. I will then start a new branch to reconcile the duplication of of the argument parser with two other parsers. I want to update the template you provide to provide the same maintained infrastructure as other jobs so that test authors only need to add args that the test will need. The env, and the collection of logs, and clean up or resources will be done by the infrastructure.

review: Approve (code)

[lp:~hduran-8/juju-ci-tools/add_status_ci_tests]

986. By Horacio Durán on 2015-07-13

Merge master

987. By Horacio Durán on 2015-07-13

Added missing return

=== added file 'assess_status_output.py'
--- assess_status_output.py	1970-01-01 00:00:00 +0000
+++ assess_status_output.py	2015-07-13 18:48:12 +0000
@@ -0,0 +1,154 @@
+#!/usr/bin/python
+from __future__ import print_function
+
+__metaclass__ = type
+
+from base_asses import get_base_parser
+from status import StatusTester
+from jujupy import (
+    make_client,
+)
+from deploy_stack import (
+    boot_context,
+    prepare_environment,
+)
+
+
+def run_complete_status(client, status):
+    """run the complete set of tests possible for any StatusParser.
+
+    :param client: python juju client.
+    :type client: jujupy.EnvJujuClient
+    :param status: a BaseStatusParser
+    :type status: BaseStatusParser
+    """
+    status.s.assert_machines_len(2)
+    status.s.assert_machines_ids(("0", "1"))
+    juju_status = client.get_status()
+    for name, machine in juju_status.iter_machines(False):
+        status.s.assert_machine_agent_state(name, "started")
+        status.s.assert_machine_agent_version(name,
+                                              machine.get("agent-version"))
+        status.s.assert_machine_dns_name(name, machine.get("dns-name"))
+        status.s.assert_machine_instance_id(name,
+                                            machine.get("instance-id"))
+        status.s.assert_machine_series(name, machine.get("series"))
+        status.s.assert_machine_hardware(name, machine.get("hardware"))
+        state_server = machine.get("state-server-member-status", None)
+        if state_server:
+            status.s.assert_machine_member_status(name, "has-vote")
+
+    status.s.assert_service_charm("statusstresser",
+                                  "local:trusty/statusstresser-1")
+    status.s.assert_service_exposed("statusstresser", False)
+    status.s.assert_service_service_status("statusstresser",
+                                           {"current": "active",
+                                            "message": "called in "
+                                            "config-changed hook"})
+    status.s.assert_unit_workload_status("statusstresser/0",
+                                         {"current": "active",
+                                          "message": "called in "
+                                          "config-changed hook"})
+    status.s.assert_unit_agent_status("statusstresser/0",
+                                      {"current": "idle", "message": ""})
+    status.s.assert_unit_agent_state("statusstresser/0", "started")
+    agent_versions = juju_status.get_agent_versions()
+    for version in agent_versions:
+        for item in agent_versions[version]:
+            if not item.isdigit():
+                status.s.assert_unit_agent_version(item, version)
+    status.s.assert_unit_machine("statusstresser/0", "1")
+
+
+def run_reduced_status(client, status):
+    """run a subset of the status asserts.
+
+    run a reduced set of tests for a StatusParser, this is useful for
+    status outputs such as Tabular that hold less information.
+
+    :param client: python juju client.
+    :type client: jujupy.EnvJujuClient
+    :param status: a BaseStatusParser
+    :type status: BaseStatusParser
+    """
+    status.s.assert_machines_len(2)
+    status.s.assert_machines_ids(("0", "1"))
+    juju_status = client.get_status()
+    for name, machine in juju_status.iter_machines(False):
+        status.s.assert_machine_agent_state(name, "started")
+        status.s.assert_machine_agent_version(name,
+                                              machine.get("agent-version"))
+        status.s.assert_machine_dns_name(name, machine.get("dns-name"))
+        status.s.assert_machine_instance_id(name, machine.get("instance-id"))
+        status.s.assert_machine_series(name, machine.get("series"))
+        status.s.assert_machine_hardware(name, machine.get("hardware"))
+
+    status.s.assert_service_charm("statusstresser",
+                                  "local:trusty/statusstresser-1")
+    status.s.assert_service_exposed("statusstresser", False)
+    status.s.assert_service_service_status("statusstresser",
+                                           {"current": "active",
+                                            "message": ""})
+    status.s.assert_unit_workload_status("statusstresser/0",
+                                         {"current": "active",
+                                          "message": "called in "
+                                          "config-changed hook"})
+    status.s.assert_unit_agent_status("statusstresser/0",
+                                      {"current": "idle", "message": ""})
+    status.s.assert_unit_machine("statusstresser/0", "1")
+
+
+def test_status_set_on_install(client):
+    """Test the status after install.
+
+    Test that status set is proplerly called during install and
+    that all formats are returning proper information.
+
+    :param client: python juju client.
+    :type client: jujupy.EnvJujuClient
+    """
+    status = StatusTester.from_text(client.get_status(60, True,
+                                                      "--format=yaml"),
+                                    "yaml")
+    run_complete_status(client, status)
+    status = StatusTester.from_text(client.get_status(60, True,
+                                                      "--format=json"),
+                                    "json")
+    run_complete_status(client, status)
+    status = StatusTester.from_text(client.get_status(60, True,
+                                                      "--format=tabular"),
+                                    "tabular")
+    run_reduced_status(client, status)
+
+
+def parse_args():
+    """Parse all arguments."""
+    parser = get_base_parser('Test status outputs')
+    return parser.parse_args()
+
+
+def main():
+    args = parse_args()
+    log_dir = args.logs
+
+    client = make_client(
+        args.juju_path, args.debug, args.env, args.temp_env_name)
+    # client.destroy_environment()
+    series = args.series
+    if series is None:
+        series = 'precise'
+    with boot_context(args.job_name, client, args.bootstrap_host,
+                      args.machine, series, args.agent_url, args.agent_stream,
+                      log_dir, args.keep_env, args.upload_tools):
+        prepare_environment(
+            client, already_bootstrapped=True, machines=args.machine)
+
+        client.get_status(60)
+        client.juju("deploy", ('local:trusty/statusstresser',))
+        client.wait_for_started()
+
+        test_status_set_on_install(client)
+
+
+if __name__ == '__main__':
+    main()
 
=== added file 'base_asses.py'
--- base_asses.py	1970-01-01 00:00:00 +0000
+++ base_asses.py	2015-07-13 18:48:12 +0000
@@ -0,0 +1,58 @@
+from argparse import ArgumentParser
+from deploy_stack import add_juju_args, add_output_args
+from unittest import FunctionTestCase
+
+
+def get_base_parser(description=""):
+    """Return a parser with the base arguments required
+    for any asses_ test.
+
+    :param description: description for argument parser.
+    :type description: str
+    :returns: argument parser with base fields
+    :rtype: ArgumentParser
+    """
+    parser = ArgumentParser(description)
+    positional_args = [
+        ('env', 'The juju environment to base the temp test environment on.'),
+        ('juju_bin', 'Full path to the Juju binary.'),
+        ('logs', 'A directory in which to store logs.'),
+        ('temp_env_name', 'A temporary test environment name.'),
+    ]
+    for p_arg in positional_args:
+        name, help_txt = p_arg
+        parser.add_argument(name, help=help_txt)
+
+    parser.add_argument(
+        'juju_path', help='Directory your juju binary lives in.')
+    parser.add_argument(
+        'job_name', help='An arbitrary name for this job.')
+
+    add_juju_args(parser)
+    add_output_args(parser)
+
+    parser.add_argument('--upgrade', action="store_true", default=False,
+                        help='Perform an upgrade test.')
+    parser.add_argument('--bootstrap-host',
+                        help='The host to use for bootstrap.', default=None)
+    parser.add_argument('--machine', help='A machine to add or when used with '
+                        'KVM based MaaS, a KVM image to start.',
+                        action='append', default=[])
+    parser.add_argument('--keep-env', action='store_true', default=False,
+                        help='Keep the Juju environment after the test'
+                        ' completes.')
+    parser.add_argument(
+        '--upload-tools', action='store_true', default=False,
+        help='upload local version of tools before bootstrapping')
+    return parser
+
+
+def toUnitTest(testCase):
+    def unitTestify(*args, **kwargs):
+        tc = FunctionTestCase(testCase)
+        largs = list(args)
+        largs.insert(1, tc)
+        args = tuple(largs)
+        return testCase(*args, **kwargs)
+
+    return unitTestify
 
=== modified file 'jujupy.py'
--- jujupy.py	2015-06-16 20:18:37 +0000
+++ jujupy.py	2015-07-13 18:48:12 +0000
@@ -124,9 +124,9 @@
         return self.get_env_client(environment).get_juju_output(
             command, *args, **kwargs)
 
-    def get_status(self, environment, timeout=60):
+    def get_status(self, environment, timeout=60, raw=False, *args):
         """Get the current status as a dict."""
-        return self.get_env_client(environment).get_status(timeout)
+        return self.get_env_client(environment).get_status(timeout, raw, *args)
 
     def get_env_option(self, environment, option):
         """Return the value of the environment's configured option."""
@@ -297,10 +297,12 @@
                 print('!!! ' + e.stderr)
                 raise
 
-    def get_status(self, timeout=60):
+    def get_status(self, timeout=60, raw=False, *args):
         """Get the current status as a dict."""
         for ignored in until_timeout(timeout):
             try:
+                if raw:
+                    return self.get_juju_output('status', *args)
                 return Status.from_text(self.get_juju_output('status'))
             except subprocess.CalledProcessError as e:
                 pass
@@ -929,8 +931,8 @@
     def juju(self, command, *args):
         return self.client.juju(self, command, args)
 
-    def get_status(self, timeout=60):
-        return self.client.get_status(self, timeout)
+    def get_status(self, timeout=60, raw=False, *args):
+        return self.client.get_status(self, timeout, raw, *args)
 
     def wait_for_deploy_started(self, service_count=1, timeout=1200):
         """Wait until service_count services are 'started'.
 
=== added file 'status.py'
--- status.py	1970-01-01 00:00:00 +0000
+++ status.py	2015-07-13 18:48:12 +0000
@@ -0,0 +1,385 @@
+from __future__ import print_function
+
+__metaclass__ = type
+
+import json
+import re
+from base_asses import toUnitTest
+from jujupy import yaml_loads
+
+# Machine and unit, deprecated in unit
+AGENT_STATE_KEY = "agent-state"
+AGENT_VERSION_KEY = "agent-version"
+DNS_NAME_KEY = "dns-name"
+INSTANCE_ID_KEY = "instance-id"
+HARDWARE_KEY = "hardware"
+SERIES_KEY = "series"
+STATE_SERVER_MEMBER_STATUS_KEY = "state-server-member-status"
+# service
+CHARM_KEY = "charm"
+EXPOSED_KEY = "exposed"
+SERVICE_STATUS_KEY = "service-status"
+# unit
+WORKLOAD_STATUS_KEY = "workload-status"
+AGENT_STATUS_KEY = "agent-status"
+MACHINE_KEY = "machine"
+PUBLIC_ADDRESS_KEY = "public-address"
+
+MACHINE_TAB_HEADERS = ['ID', 'STATE', 'VERSION', 'DNS', 'INS-ID', 'SERIES',
+                       'HARDWARE']
+UNIT_TAB_HEADERS = ['ID', 'WORKLOAD-STATE', 'AGENT-STATE', 'VERSION',
+                    'MACHINE', 'PORTS', 'PUBLIC-ADDRESS', 'MESSAGE']
+SERVICE_TAB_HEADERS = ['NAME', 'STATUS', 'EXPOSED', 'CHARM']
+
+
+class StatusTester:
+    def __init__(self, text="", status_format="yaml"):
+        self._text = text
+        self._format = status_format
+        self.s = globals()["Status%sParser" % status_format.capitalize()](text)
+
+    @classmethod
+    def from_text(cls, text, status_format):
+        return cls(text, status_format)
+
+    def __unicode__(self):
+        return self._text
+
+    def __str__(self):
+        return self._text
+
+
+class ErrNoStatus(Exception):
+    """An exception for missing juju status."""
+
+
+class ErrMalformedStatus(Exception):
+    """An exception for unexpected formats of status."""
+
+
+class ErrUntestedStatusOutput(Exception):
+    """An exception for results returned by status.
+
+    Status that are known not to be covered by the tests should raise
+    this exception. """
+
+
+class BaseStatusParser:
+
+    _expected = set(["environment", "services", "machines"])
+
+    def __init__(self):
+        # expected entity storage
+        self._machines = dict()
+        self._services = dict()
+        self._units = dict()
+
+        self.parse()
+
+    def parse(self):
+        return self._parse()
+
+    @toUnitTest
+    def store(self, tc, parsed):
+        # Either there are less items than expected and therefore status
+        # is returning a subset of what it should or there are more and
+        # this means there are untested keys in status.
+        tc.assertItemsEqual(parsed.keys(), self._expected,
+                            "untested items or incomplete status output")
+
+        # status of machines.
+        for machine_id, machine in parsed.get("machines", {}).iteritems():
+            tc.assertNotIn(machine_id, self._machines,
+                           "Machine %s is repeated in yaml"
+                           " status " % machine_id)
+            self._machines[machine_id] = machine
+
+        # status of services
+        for service_name, service in parsed.get("services", {}).iteritems():
+            tc.assertNotIn(service_name, self._services,
+                           "Service %s is repeated in yaml "
+                           "status " % service_name)
+            self._services[service_name] = service
+
+        # status of units
+        for service_name, service in self._services.iteritems():
+            for unit_name, unit in service.get("units", {}).iteritems():
+                tc.assertNotIn(unit_name, self._units,
+                               "Unit %s is repeated in yaml "
+                               "status " % unit_name)
+                self._units[unit_name] = unit
+
+    @toUnitTest
+    def assert_machines_len(self, tc, expected_len):
+        """Assert that we got as many machines as we where expecting.
+
+        :param expected_len: expected quantity of machines.
+        :type expected_len: int
+        """
+        tc.assertEqual(len(self._machines), expected_len)
+
+    @toUnitTest
+    def assert_machines_ids(self, tc, expected_ids):
+        """Assert that we got the machines we where expecting.
+
+        :param expected_ids: expected ids of machines.
+        :type expected_ids: tuple
+        """
+        tc.assertItemsEqual(self._machines, expected_ids)
+
+    def _machine_key_get(self, tc, machine_id, key):
+        tc.assertIn(machine_id, self._machines,
+                    "Machine \"%s\" not present in machines" % machine_id)
+        tc.assertIn(key, self._machines[machine_id],
+                    "Key \"%s\" not present in Machine \"%s\"" %
+                    (key, machine_id))
+        return self._machines[machine_id][key]
+
+    @toUnitTest
+    def assert_machine_agent_state(self, tc, machine_id, state):
+        value = self._machine_key_get(tc, machine_id, AGENT_STATE_KEY)
+        tc.assertEqual(value, state)
+
+    @toUnitTest
+    def assert_machine_agent_version(self, tc, machine_id, version):
+        value = self._machine_key_get(tc, machine_id, AGENT_VERSION_KEY)
+        tc.assertEqual(value, version)
+
+    @toUnitTest
+    def assert_machine_dns_name(self, tc, machine_id, dns_name):
+        value = self._machine_key_get(tc, machine_id, DNS_NAME_KEY)
+        tc.assertEqual(value, dns_name)
+
+    @toUnitTest
+    def assert_machine_instance_id(self, tc, machine_id, instance_id):
+        value = self._machine_key_get(tc, machine_id, INSTANCE_ID_KEY)
+        tc.assertEqual(value, instance_id)
+
+    @toUnitTest
+    def assert_machine_series(self, tc, machine_id, series):
+        value = self._machine_key_get(tc, machine_id, SERIES_KEY)
+        tc.assertEqual(value, series)
+
+    @toUnitTest
+    def assert_machine_hardware(self, tc, machine_id, hardware):
+        value = self._machine_key_get(tc, machine_id, HARDWARE_KEY)
+        tc.assertEqual(value, hardware)
+
+    @toUnitTest
+    def assert_machine_member_status(self, tc, machine_id, member_status):
+        value = self._machine_key_get(tc, machine_id,
+                                      STATE_SERVER_MEMBER_STATUS_KEY)
+        tc.assertEqual(value, member_status)
+
+    def _service_key_get(self, tc, service_name, key):
+        tc.assertIn(service_name, self._services,
+                    "Service \"%s\" not present in services." % service_name)
+        tc.assertIn(key, self._services[service_name],
+                    "Key \"%s\" not present in Service \"%s\"" %
+                    (key, service_name))
+        return self._services[service_name][key]
+
+    # Service status
+    @toUnitTest
+    def assert_service_charm(self, tc, service_name, charm):
+        value = self._service_key_get(tc, service_name, CHARM_KEY)
+        tc.assertEqual(value, charm)
+
+    @toUnitTest
+    def assert_service_exposed(self, tc, service_name, exposed):
+        value = self._service_key_get(tc, service_name, EXPOSED_KEY)
+        tc.assertEqual(value, exposed)
+
+    @toUnitTest
+    def assert_service_service_status(self, tc, service_name,
+                                      status={"current": "", "message": ""}):
+        value = self._service_key_get(tc, service_name, SERVICE_STATUS_KEY)
+        tc.assertEqual(value["current"], status["current"])
+        tc.assertEqual(value["message"], status["message"])
+
+    def _unit_key_get(self, tc, unit_name, key):
+        tc.assertIn(unit_name, self._units,
+                    "Unit \"%s\" not present in units" % unit_name)
+        tc.assertIn(key, self._units[unit_name],
+                    "Key \"%s\" not present in Unit \"%s\"" %
+                    (key, unit_name))
+        return self._units[unit_name][key]
+
+    # Units status
+    @toUnitTest
+    def assert_unit_workload_status(self, tc, unit_name,
+                                    status={"current": "", "message": ""}):
+        value = self._unit_key_get(tc, unit_name, WORKLOAD_STATUS_KEY)
+        tc.assertEqual(value["current"], status["current"])
+        tc.assertEqual(value["message"], status["message"])
+
+    @toUnitTest
+    def assert_unit_agent_status(self, tc, unit_name,
+                                 status={"current": "", "message": ""}):
+        value = self._unit_key_get(tc, unit_name, AGENT_STATUS_KEY)
+        tc.assertEqual(value["current"], status["current"])
+        # Message is optional for unit agents.
+        tc.assertEqual(value.get("message", ""), status["message"])
+
+    @toUnitTest
+    def assert_unit_agent_state(self, tc, unit_name, state):
+        value = self._unit_key_get(tc, unit_name, AGENT_STATE_KEY)
+        tc.assertEqual(value, state)
+
+    @toUnitTest
+    def assert_unit_agent_version(self, tc, unit_name, version):
+        value = self._unit_key_get(tc, unit_name, AGENT_VERSION_KEY)
+        tc.assertEqual(value, version)
+
+    @toUnitTest
+    def assert_unit_machine(self, tc, unit_name, machine):
+        value = self._unit_key_get(tc, unit_name, MACHINE_KEY)
+        tc.assertEqual(value, machine)
+
+    @toUnitTest
+    def assert_unit_public_address(self, tc, unit_name, address):
+        value = self._unit_key_get(tc, unit_name, PUBLIC_ADDRESS_KEY)
+        tc.assertEqual(value, address)
+
+
+class StatusYamlParser(BaseStatusParser):
+    """StatusYamlParser handles parsing of status output in yaml format.
+
+    To be used by status tester.
+    """
+
+    def __init__(self, yaml=""):
+        self._yaml = yaml
+        if yaml == "":
+            raise ErrNoStatus("Yaml status was empty")
+        super(StatusYamlParser, self).__init__()
+
+    def _parse(self):
+        parsed = yaml_loads(self._yaml)
+        self.store(parsed)
+
+
+class StatusJsonParser(BaseStatusParser):
+    """StatusJSONParser handles parsing of status output in JSON format.
+
+    To be used by status tester.
+    """
+
+    def __init__(self, json_text=""):
+        self._json = json_text
+        if json_text == "":
+            raise ErrNoStatus("JSON status was empty")
+        super(StatusJsonParser, self).__init__()
+
+    def _parse(self):
+        parsed = json.loads(self._json)
+        self.store(parsed)
+
+
+class StatusTabularParser(BaseStatusParser):
+    """StatusTabularParser handles parsing of status output in Tabular format.
+
+    To be used by status tester.
+    """
+
+    def __init__(self, tabular_text=""):
+        self._tabular = tabular_text
+        if tabular_text == "":
+            raise ErrNoStatus("tabular status was empty")
+        super(StatusTabularParser, self).__init__()
+
+    @toUnitTest
+    def _normalize_machines(self, tc, header, items):
+        nitems = items[:6]
+        nitems.append(" ".join(items[6:]))
+        tc.assertEqual(header, MACHINE_TAB_HEADERS,
+                       "Unexpected headers for machine:\n"
+                       "wanted: %s"
+                       "got: %s" % (MACHINE_TAB_HEADERS, header))
+        normalized = dict(zip((AGENT_STATE_KEY, AGENT_VERSION_KEY,
+                                    DNS_NAME_KEY, INSTANCE_ID_KEY,
+                                    SERIES_KEY, HARDWARE_KEY),
+                               nitems[1:]))
+        return nitems[0], normalized
+
+    @toUnitTest
+    def _normalize_units(self, tc, header, items):
+        eid, wlstate, astate, version, machine, paddress = items[:6]
+        message = " ".join(items[6:])
+        wlstatus = {"current": wlstate, "message": message,
+                    "since": "bogus date"}
+        astatus = {"current": astate, "message": "", "since": "bogus date"}
+        tc.assertEqual(header, UNIT_TAB_HEADERS,
+                       "Unexpected headers for unit.\n"
+                       "wanted: %s"
+                       "got: %s" % (UNIT_TAB_HEADERS, header))
+        normalized = dict(zip((WORKLOAD_STATUS_KEY, AGENT_STATUS_KEY,
+                              AGENT_VERSION_KEY, MACHINE_KEY,
+                              PUBLIC_ADDRESS_KEY),
+                             (wlstatus, astatus, version, machine, paddress)))
+
+        return eid, normalized
+
+    @toUnitTest
+    def _normalize_services(self, tc, header, items):
+        name, status, exposed, charm = items
+        tc.assertEqual(header, SERVICE_TAB_HEADERS,
+                       "Unexpected headers for service.\n"
+                       "wanted: %s"
+                       "got: %s" % (SERVICE_TAB_HEADERS, header))
+        normalized = dict(zip((CHARM_KEY, EXPOSED_KEY, SERVICE_STATUS_KEY),
+                              (charm, exposed == "true", {"current": status,
+                               "message": ""})))
+        return name, normalized
+
+    def _parse(self):
+        section = re.compile("^\[(\w*)\]")
+        base = {"environment": "not provided"}
+        current_parent = ""
+        current_headers = []
+        prev_was_section = False
+        for line in self._tabular.splitlines():
+            # parse section
+            is_section = section.findall(line)
+            if len(is_section) == 1:
+                current_parent = is_section[0].lower()
+                if current_parent != "units":
+                    base[current_parent] = {}
+                prev_was_section = True
+                continue
+            # parse headers
+            if prev_was_section:
+                prev_was_section = False
+                current_headers = line.split()
+                continue
+
+            # parse content
+            if current_parent == "" or current_headers == []:
+                raise ErrMalformedStatus("Tabular status is malformed")
+            items = line.split()
+
+            # separation line
+            if len(items) == 0:
+                continue
+
+            normalize = None
+            if current_parent == "services":
+                normalize = self._normalize_services
+            elif current_parent == "units":
+                normalize = self._normalize_units
+            elif current_parent == "machines":
+                normalize = self._normalize_machines
+
+            if not normalize:
+                raise ErrUntestedStatusOutput("%s is not an expected tabular"
+                                              " status section" %
+                                              current_parent)
+            k, v = normalize(current_headers, items)
+            if current_parent == "units":
+                base.setdefault("services", dict())
+                service = k.split("/")[0]
+                base["services"][service].setdefault("units", dict())
+                base["services"][service]["units"][k] = v
+            else:
+                base[current_parent][k] = v
+        self.store(base)
 
=== added file 'template_assess.py.tmpl'
--- template_assess.py.tmpl	1970-01-01 00:00:00 +0000
+++ template_assess.py.tmpl	2015-07-13 18:48:12 +0000
@@ -0,0 +1,54 @@
+#!/usr/bin/python
+from __future__ import print_function
+
+__metaclass__ = type
+
+from base_asses import get_base_parser
+from status import StatusTester
+from jujupy import (
+    make_client,
+)
+from deploy_stack import (
+    boot_context,
+    prepare_environment,
+)
+
+TEST_HUMAN_NAME="<YOUR TEST HUMAN READABLE HERE>"
+
+
+def parse_args():
+    """Parse all arguments."""
+    parser = get_base_parser(TEST_HUMAN_NAME)
+    return parser.parse_args()
+
+
+def main():
+    args = parse_args()
+    log_dir = args.logs
+
+    client = make_client(
+        args.juju_path, args.debug, args.env_name, args.temp_env_name)
+    client.destroy_environment()
+    series = args.series
+    if series is None:
+        series = 'precise'
+    with boot_context(args.job_name, client, args.bootstrap_host,
+                      args.machine, series, args.agent_url, args.agent_stream,
+                      log_dir, args.keep_env, args.upload_tools):
+        prepare_environment(
+            client, already_bootstrapped=True, machines=args.machine)
+
+        client.get_status(60)
+        # Deploy charms, there are several under ./repository
+        client.juju("deploy", ('local:trusty/wordpress',))
+        # Wait for the deployment to finish.
+        client.wait_for_started()
+        #----------------- CALL YOUR TESTS HERE
+        # At this point you have a juju bootstraped with a wordpress charm
+        # deployed and active with the agent idle.
+
+        #----------------- TESTS END HERE
+
+
+if __name__ == '__main__':
+    main()
 
=== added file 'test_assess_status_output.py'
--- test_assess_status_output.py	1970-01-01 00:00:00 +0000
+++ test_assess_status_output.py	2015-07-13 18:48:12 +0000
@@ -0,0 +1,426 @@
+__metaclass__ = type
+
+from status import (
+    ErrNoStatus,
+    StatusYamlParser,
+    StatusJsonParser,
+    StatusTabularParser
+)
+
+from unittest import TestCase
+
+SAMPLE_YAML_OUTPUT = """environment: bogusec2
+machines:
+  "0":
+    agent-state: started
+    agent-version: 1.25-alpha1
+    dns-name: 54.82.51.4
+    instance-id: i-c0dadd10
+    instance-state: running
+    series: trusty
+    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M """ \
+    """root-disk=8192M availability-zone=us-east-1c
+    state-server-member-status: has-vote
+  "1":
+    agent-state: started
+    agent-version: 1.25-alpha1
+    dns-name: 54.162.95.230
+    instance-id: i-39280ac6
+    instance-state: running
+    series: trusty
+    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M """\
+    """root-disk=8192M availability-zone=us-east-1d
+services:
+  statusstresser:
+    charm: local:trusty/statusstresser-1
+    exposed: false
+    service-status:
+      current: active
+      message: called in config-changed hook
+      since: 12 Jun 2015 13:15:25-03:00
+    units:
+      statusstresser/0:
+        workload-status:
+          current: active
+          message: called in config-changed hook
+          since: 12 Jun 2015 13:15:25-03:00
+        agent-status:
+          current: idle
+          since: 12 Jun 2015 14:33:08-03:00
+          version: 1.25-alpha1
+        agent-state: started
+        agent-version: 1.25-alpha1
+        machine: "1"
+        public-address: 54.162.95.230
+"""
+# This could be replaced by a json.dumps({}) but the text is kept to
+# make this test about status output as accurate as possible.
+SAMPLE_JSON_OUTPUT = \
+    """{"environment":"perritoec2","machines":{"0":{"agent-state":"started","""\
+    """"agent-version":"1.25-alpha1","dns-name":"54.82.51.4","""\
+    """"instance-id":"i-c0dadd10","instance-state":"running","""\
+    """"series":"trusty","hardware":"arch=amd64 cpu-cores=1 cpu-power=100 """\
+    """mem=1740M root-disk=8192M availability-zone=us-east-1c","""\
+    """"state-server-member-status":"has-vote"},"1":{"agent-state":"""\
+    """"started","agent-version":"1.25-alpha1","dns-name":"54.162.95.230","""\
+    """"instance-id":"i-a7a2b377","instance-state":"running","""\
+    """"series":"trusty","hardware":"arch=amd64 cpu-cores=1 cpu-power=300 """\
+    """mem=3840M root-disk=8192M availability-zone=us-east-1c"}},"""\
+    """"services":{"statusstresser":{"charm": """\
+    """"local:trusty/statusstresser-1","exposed":false,"service-status":"""\
+    """{"current":"active","message":"called in config-changed hook","""\
+    """"since":"15 Jun 2015 20:56:29-03:00"},"""\
+    """"units":{"statusstresser/0":{"workload-status":"""\
+    """{"current":"active","message":"called in config-changed hook","""\
+    """"since":"15 Jun 2015 20:56:29-03:00"},"agent-status":"""\
+    """{"current":"idle","since":"15 Jun 2015 20:56:41-03:00","""\
+    """"version":"1.25-alpha1"},"agent-state":"started","agent-version":"""\
+    """"1.25-alpha1","machine":"1","public-address":"54.162.95.230"}}}}}"""
+
+SAMPLE_TABULAR_OUTPUT = """[Services]
+NAME           STATUS EXPOSED CHARM
+statusstresser active false   local:trusty/statusstresser-1
+
+[Units]
+ID               WORKLOAD-STATE AGENT-STATE VERSION     MACHINE PORTS """\
+"""PUBLIC-ADDRESS MESSAGE
+statusstresser/0 active         idle        1.25-alpha1 1             """\
+"""54.162.95.230  called in config-changed hook
+
+[Machines]
+ID         STATE   VERSION     DNS            INS-ID     SERIES HARDWARE
+0          started 1.25-alpha1 54.82.51.4     i-c0dadd10 trusty arch=amd64 """\
+    """cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M """\
+    """availability-zone=us-east-1c
+1          started 1.25-alpha1 54.162.95.230  i-39280ac6 trusty arch=amd64 """\
+    """cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M """\
+    """availability-zone=us-east-1d"""
+
+
+class ReducedTestStatus:
+
+    def test_assert_machine_ids(self):
+        self.parser.assert_machines_ids(["0", "1"])
+
+    def test_assert_machine_ids_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machines_ids(["0", "1", "2"])
+
+    def test_assert_machine_len(self):
+        self.parser.assert_machines_len(2)
+
+    def test_assert_machine_len_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machines_len(3)
+
+    def test_machine_agent_state_valid(self):
+        self.parser.assert_machine_agent_state("0", "started")
+
+    def test_machine_agent_state_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machine_agent_state("0", "stopped")
+
+    def test_machine_agent_state_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machine_agent_state("3", "stopped")
+
+    def test_assert_machine_agent_version(self):
+        self.parser.assert_machine_agent_version("0", "1.25-alpha1")
+
+    def test_assert_machine_agent_version_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machine_agent_version("0", "1.25-alpha2")
+
+    def test_assert_machine_agent_version_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machine_agent_version("5", "1.25-alpha1")
+
+    def test_assert_machine_dns_name(self):
+        self.parser.assert_machine_dns_name("0", "54.82.51.4")
+
+    def test_assert_machine_dns_name_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machine_dns_name("0", "54.82.51.5")
+
+    def test_assert_machine_dns_name_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machine_dns_name("3", "54.82.51.4")
+
+    def test_assert_machine_instance_id(self):
+        self.parser.assert_machine_instance_id("0", "i-c0dadd10")
+
+    def test_assert_machine_instance_id_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machine_instance_id("0", "i-c0dadd11")
+
+    def test_assert_machine_instance_id_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machine_instance_id("3", "i-c0dadd10")
+
+    def test_assert_machine_series(self):
+        self.parser.assert_machine_series("0", "trusty")
+
+    def test_assert_machine_series_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machine_series("0", "utopic")
+
+    def test_assert_machine_series_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machine_series("3", "trusty")
+
+    def test_assert_machine_hardware(self):
+        self.parser.assert_machine_hardware("0", "arch=amd64 cpu-cores=1 "
+                                                 "cpu-power=100 mem=1740M "
+                                                 "root-disk=8192M "
+                                                 "availability-zone="
+                                                 "us-east-1c")
+
+    def test_assert_machine_hardware_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machine_hardware("0", "arch=arm cpu-cores=1 "
+                                                     "cpu-power=100 mem=1740M "
+                                                     "root-disk=8192M "
+                                                     "availability-zone="
+                                                     "us-east-1c")
+
+    def test_assert_machine_hardware_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machine_hardware("3", "arch=amd64 cpu-cores=1 "
+                                                     "cpu-power=100 mem=1740M "
+                                                     "root-disk=8192M "
+                                                     "availability-zone="
+                                                     "us-east-1c")
+
+    def test_assert_service_charm(self):
+        self.parser.assert_service_charm("statusstresser",
+                                         "local:trusty/statusstresser-1")
+
+    def test_assert_service_charm_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_service_charm("statusstresser",
+                                             "local:trusty/statusstresser-2")
+
+    def test_assert_service_charm_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_service_charm("statusrelaxer",
+                                             "local:trusty/statusstresser-1")
+
+    def test_assert_service_exposed(self):
+        self.parser.assert_service_exposed("statusstresser", False)
+
+    def test_assert_service_exposed_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_service_exposed("statusstresser", True)
+
+    def test_assert_service_exposed_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_service_exposed("statusrelaxer", False)
+
+    def test_assert_unit_public_address(self):
+        self.parser.assert_unit_public_address("statusstresser/0",
+                                               "54.162.95.230")
+
+    def test_assert_unit_public_address_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_public_address("statusstresser/0",
+                                                   "54.162.95.231")
+
+    def test_assert_unit_public_address_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_public_address("statusrelaxer/0",
+                                                   "54.162.95.230")
+
+
+class BaseTestStatus(ReducedTestStatus):
+
+    def test_assert_machine_member_status(self):
+        self.parser.assert_machine_member_status("0", "has-vote")
+
+    def test_assert_machine_member_status_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machine_member_status("0", "not-voting")
+
+    def test_assert_machine_member_status_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_machine_member_status("3", "has-vote")
+
+    def test_assert_service_service_status(self):
+        self.parser.assert_service_service_status("statusstresser",
+                                                  {"current": "active",
+                                                   "message": "called in "
+                                                   "config-changed hook"})
+
+    def test_assert_service_service_status_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_service_service_status("statusstresser",
+                                                      {"current": "active",
+                                                       "message": "another "
+                                                       "message"})
+
+    def test_assert_service_service_status_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_service_service_status("statusrelaxer",
+                                                      {"current": "active",
+                                                       "message": "called in "
+                                                       "config-changed hook"})
+
+    def test_assert_unit_workload_status(self):
+        self.parser.assert_unit_workload_status("statusstresser/0",
+                                                {"current": "active",
+                                                 "message": "called in "
+                                                 "config-changed hook"})
+
+    def test_assert_unit_workload_status_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_workload_status("statusstresser/0",
+                                                    {"current": "active",
+                                                     "message": "another "
+                                                     "message"})
+
+    def test_assert_unit_workload_status_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_workload_status("statusrelaxer/0",
+                                                    {"current": "active",
+                                                     "message": "called in "
+                                                     "config-changed hook"})
+
+    def test_assert_unit_agent_status(self):
+        self.parser.assert_unit_agent_status("statusstresser/0",
+                                             {"current": "idle",
+                                              "message": ""})
+
+    def test_assert_unit_agent_status_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_agent_status("statusstresser/0",
+                                                 {"current": "idle",
+                                                  "message": "an unexpected "
+                                                  "message"})
+
+    def test_assert_unit_agent_status_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_agent_status("statusrelaxer/0",
+                                                 {"current": "idle",
+                                                  "message": ""})
+
+    def test_assert_unit_agent_state(self):
+        self.parser.assert_unit_agent_state("statusstresser/0", "started")
+
+    def test_assert_unit_agent_state_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_agent_state("statusstresser/0", "stopped")
+
+    def test_assert_unit_agent_state_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_agent_state("statusrelaxer/0", "started")
+
+    def test_assert_unit_agent_version(self):
+        self.parser.assert_unit_agent_version("statusstresser/0",
+                                              "1.25-alpha1")
+
+    def test_assert_unit_agent_version_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_agent_version("statusstresser/0",
+                                                  "1.25-alpha2")
+
+    def test_assert_unit_agent_version_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_agent_version("statusrelaxer/0",
+                                                  "1.25-alpha1")
+
+    def test_assert_unit_machine(self):
+        self.parser.assert_unit_machine("statusstresser/0", "1")
+
+    def test_assert_unit_machine_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_machine("statusstresser/0", "2")
+
+    def test_assert_unit_machine_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_machine("statusrelaxer/0", "1")
+
+
+class TestStatusForYaml(TestCase, BaseTestStatus):
+
+    def test_empty_yaml_fails(self):
+        with self.assertRaises(ErrNoStatus):
+            StatusYamlParser(yaml="")
+
+    def setUp(self):
+        self.parser = StatusYamlParser(yaml=SAMPLE_YAML_OUTPUT)
+
+
+class TestStatusForJson(TestCase, BaseTestStatus):
+
+    def test_empty_json_fails(self):
+        with self.assertRaises(ErrNoStatus):
+            StatusJsonParser(json_text="")
+
+    def setUp(self):
+        self.parser = StatusJsonParser(json_text=SAMPLE_JSON_OUTPUT)
+
+
+class TestStatusTabular(TestCase, ReducedTestStatus):
+
+    def test_empty_tabular_fails(self):
+        with self.assertRaises(ErrNoStatus):
+            StatusTabularParser("")
+
+    def setUp(self):
+        self.parser = StatusTabularParser(tabular_text=SAMPLE_TABULAR_OUTPUT)
+
+    def test_assert_service_service_status(self):
+        self.parser.assert_service_service_status("statusstresser",
+                                                  {"current": "active",
+                                                   "message": ""})
+
+    def test_assert_service_service_status_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_service_service_status("statusstresser",
+                                                      {"current": "active",
+                                                       "message": "another "
+                                                       "message"})
+
+    def test_assert_service_service_status_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_service_service_status("statusrelaxer",
+                                                      {"current": "active",
+                                                       "message": "called in "
+                                                       "config-changed hook"})
+
+    def test_assert_unit_workload_status(self):
+        self.parser.assert_unit_workload_status("statusstresser/0",
+                                                {"current": "active",
+                                                 "message": "called in "
+                                                 "config-changed hook"})
+
+    def test_assert_unit_workload_status_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_workload_status("statusstresser/0",
+                                                    {"current": "active",
+                                                     "message": "another "
+                                                     "message"})
+
+    def test_assert_unit_workload_status_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_workload_status("statusrelaxer/0",
+                                                    {"current": "active",
+                                                     "message": "called in "
+                                                     "config-changed hook"})
+
+    def test_assert_unit_agent_status(self):
+        self.parser.assert_unit_agent_status("statusstresser/0",
+                                             {"current": "idle",
+                                              "message": ""})
+
+    def test_assert_unit_agent_status_failed(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_agent_status("statusstresser/0",
+                                                 {"current": "idle",
+                                                  "message": "an unexpected "
+                                                  "message"})
+
+    def test_assert_unit_agent_status_error(self):
+        with self.assertRaises(AssertionError):
+            self.parser.assert_unit_agent_status("statusrelaxer/0",
+                                                 {"current": "idle",
+                                                  "message": ""})