Merge lp:~sseman/juju-chaos-monkey/chaos-monkey into lp:juju-chaos-monkey

[Seman]

Initial version of Chaos Monkey. It supports blocking a network and killing jujud process.

[John George (jog)]

I added a few initial comments and will save this now but need to continue later.

[John George (jog)]

On Wed, Apr 22, 2015 at 8:21 PM, Seman <email address hidden> wrote:

> Seman has proposed merging lp:~sseman/juju-chaos-monkey/chaos-moneky into
> lp:juju-chaos-monkey.
>
> Requested reviews:
> Juju QA Engineering (juju-qa)
>
> For more details, see:
>
> https://code.launchpad.net/~sseman/juju-chaos-monkey/chaos-moneky/+merge/257212
>
> Initial version of Chaos Monkey. It supports blocking a network and
> killing jujud process.
> --
> Your team Juju QA Engineering is requested to review the proposed merge of
> lp:~sseman/juju-chaos-monkey/chaos-moneky into lp:juju-chaos-monkey.
>
> === added file '.bzrignore'
> --- .bzrignore 1970-01-01 00:00:00 +0000
> +++ .bzrignore 2015-04-23 03:18:54 +0000
> @@ -0,0 +1,1 @@
> +/.idea
>
> === added file 'Makefile'
> --- Makefile 1970-01-01 00:00:00 +0000
> +++ Makefile 2015-04-23 03:18:54 +0000
> @@ -0,0 +1,6 @@
> +test:
> + python -m unittest discover -vv tests -p '*.py'
> +lint:
> + flake8 *.py tests/*.py
> +clean:
> + find . -name '*.pyc' -delete
>
> === added file '__init__.py'
> === added directory 'chaos'
> === added file 'chaos/__init__.py'
> === added file 'chaos/kill.py'
> --- chaos/kill.py 1970-01-01 00:00:00 +0000
> +++ chaos/kill.py 2015-04-23 03:18:54 +0000
> @@ -0,0 +1,62 @@
> +from chaos_monkey_base import (
> + Chaos,
> + ChaosMonkeyBase,
> +)
> +from utility import (
> + log,
> + NotFound,
> + run_shell_command,
> +)
> +
> +__metaclass__ = type
> +
> +
> +class Kill(ChaosMonkeyBase):
> + """
> + Kills a process
> + """
> + def __init__(self):
> + self.group = 'kill'
> + super(Kill, self).__init__()
> +
> + @classmethod
> + def factory(cls):
> + return cls()
> +
> + def get_pids(self, process):
> + pids = run_shell_command('pidof ' + process)
> + if not pids:
> + return None
> + return pids.split(' ')
> +
> + def kill_jujud(self):
> + log("Kill.kill_jujud")
> + pids = self.get_pids('jujud')
>
Do you really want to raise an exception here? Won't that terminate the
chaos monkey. If so I think you might want to just log that there were no
processes to kill, so the monkey can move on to it's not bit of chaos.

> + if not pids:
> + raise NotFound('Process id not found')
>
See my comment about explicit signal selection below.

> + run_shell_command('kill ' + pids[0])
> +
> + def kill_mongodb(self):
> + log("Kill.kill_mongod")
> + pids = self.get_pids('mongod')
> + if pids:
>
Should you be explicit about the signal you are sending with kill? By
default SIGTERM is sent but you may really want SIGKILL to ensure the
process is really terminated.

> + run_shell_command('kill ' + pids[0])
> +
> + def get_chaos(self):
> + chaos = list()
> + chaos.append(
> + Chaos(
> + enable=self.kill_jujud,
> + disable=None,
> + group=self.group,
> + command_str='jujud'))
> + chaos.append(
> + Chaos(
> + enable=self.kill_jujud,
> + disable=None,
> + group=self.gr...

[John George (jog)]

These are my comments related to runner.py. I'm posting these now so you can respond but will continue reviewing the rest of the merge proposal.

Please update all docstring formatting to follow:
https://www.python.org/dev/peps/pep-0257/

runner.py random()
    Please follow import formatting described in pep8 (required for merge)

    argparse and sys should be grouped together.

    chaos_monkey and utility should be grouped together.

    https://www.python.org/dev/peps/pep-0008/
    Imports should be grouped in the following order:

         1. standard library imports
         2. related third party imports
         3. local application/library specific imports

         You should put a blank line between each group of imports.

    For the merge is_juju_ok and report_juju_error don't need to be included.
    They are stubs that currently have not use. I'm not convinced this code
    is responsible for checking if juju is OK. how about just putting a TODO
    comment in rather than the stub code? (not required for merge but I think
    it would keep the code cleaner).

    I don't think the random() function is unit tested well enough, please see my
    comments under tests/test_runner.py (better unit testing required for merge)

    Main:
    action='append' with integers results in parse_args raising an error.
    Please make sure your do some basic testing before proposing.
    "action='append'" should be "type=int" (required for merge; fails to run otherwise)

    You should not need to pass 'sys.argv[1:]' to parser.parse_args()
    https://docs.python.org/2/library/argparse.html
    In a script, parse_args() will typically be called with no arguments,
    and the ArgumentParser will automatically determine the command-line
    arguments from sys.argv.

tests/test_runner.py TestRunner()
    Typo: autospect --> autospec

    Needs additional test coverage to cover the logic:
        1. assertRaises for pause_timeout > run_timeout
        2. Test pause_timeout assignement
            pause_timeout <= 0 test
                pause_timeout = 1 or pause_timeout
        3. Test run_random_chaos is called expected number of times
           (First see comments about if a pre-determind call count is really
            the right approach for total_timeout.)
        4. Test report_juju_error is call when is_juju_ok() is False.
        5. Test cm.shutdown is called

[John George (jog)]

Here is what I was referring to, in my last comment, when I said see my comments about a pre-determind call count. I'd made that comment in email but apparently Launchpad did not include it in the above comment grrrr

> + pause_timeout = 1 if pause_timeout <= 0 else pause_timeout
>
Total_run is a count used in the for loop to call run_random_chaos a
predetermined number of times?
If the random chaos functions take varying time to run, especially as
additional functions are added, it might make sense to check if the
run_timeout has been reached before making the next run_random_chaos call.
(this is not required for merge, just a suggestion)

> + total_run = int(run_timeout/pause_timeout)
> + cm = ChaosMonkey.factory()
> + for _ in xrange(total_run):
> + cm.run_random_chaos(pause_timeout)
> + if not is_juju_ok():
> + report_juju_error()
> + break
> + cm.shutdown()

[John George (jog)]

More comments and questions in-line.

[John George (jog)]

review: Needs Fixing

[lp:~sseman/juju-chaos-monkey/chaos-monkey]

3. By Seman on 2015-04-23

Updated the make file to include all subdirectories.

4. By Seman on 2015-04-29

Updated after a code review

[Seman (sseman)]

Thanks for you review and suggestions. Please see inline comments and code updates.

[John George (jog)]

I left a couple comments in-line but I think you can go ahead and land this branch.
I'm still a bit skeptical about your unit tests for run_random and how they can predict which chaos functions will be selected. I suspect these will need to change as additional chaos classes are added.

review: Approve

Unmerged revisions

4. By Seman on 2015-04-29

Updated after a code review

3. By Seman on 2015-04-23

Updated the make file to include all subdirectories.

2. By Seman on 2015-04-23

Initial commit

1. By Seman on 2015-04-16

CMonkey test

=== added file '.bzrignore'
--- .bzrignore	1970-01-01 00:00:00 +0000
+++ .bzrignore	2015-04-29 03:47:51 +0000
@@ -0,0 +1,1 @@
+/.idea

=== added file 'Makefile'
--- Makefile	1970-01-01 00:00:00 +0000
+++ Makefile	2015-04-29 03:47:51 +0000
@@ -0,0 +1,6 @@
+test:
+	python -m unittest discover -vv tests -p '*.py'
+lint:
+	flake8 .
+clean:
+	find . -name '*.pyc' -delete

=== added file '__init__.py'
=== added directory 'chaos'
=== added file 'chaos/__init__.py'
=== added file 'chaos/kill.py'
--- chaos/kill.py	1970-01-01 00:00:00 +0000
+++ chaos/kill.py	2015-04-29 03:47:51 +0000
@@ -0,0 +1,68 @@
+from chaos_monkey_base import (
+    Chaos,
+    ChaosMonkeyBase,
+)
+from utility import (
+    log,
+    NotFound,
+    run_shell_command,
+)
+
+__metaclass__ = type
+
+
+class Kill(ChaosMonkeyBase):
+    """Kills a process"""
+
+    def __init__(self):
+        self.group = 'kill'
+        super(Kill, self).__init__()
+
+    @classmethod
+    def factory(cls):
+        return cls()
+
+    def get_pids(self, process):
+        pids = run_shell_command('pidof ' + process)
+        if not pids:
+            return None
+        return pids.split(' ')
+
+    def kill_jujud(self, quiet_mode=True):
+        log("Kill.kill_jujud")
+        pids = self.get_pids('jujud')
+        if not pids:
+            log("Jujud process ID not found")
+            if not quiet_mode:
+                raise NotFound('Process id not found')
+            return
+        run_shell_command('kill -s SIGKILL ' + pids[0])
+
+    def kill_mongodb(self, quiet_mode=True):
+        log("Kill.kill_mongod")
+        pids = self.get_pids('mongod')
+        if not pids:
+            log("MongoDB process ID not found")
+            if not quiet_mode:
+                raise NotFound('Process id not found')
+            return
+        run_shell_command('kill -s SIGKILL ' + pids[0])
+
+    def get_chaos(self):
+        chaos = list()
+        chaos.append(
+            Chaos(
+                enable=self.kill_jujud,
+                disable=None,
+                group=self.group,
+                command_str='jujud'))
+        chaos.append(
+            Chaos(
+                enable=self.kill_jujud,
+                disable=None,
+                group=self.group,
+                command_str='mongod'))
+        return chaos
+
+    def shutdown(self):
+        pass

=== added file 'chaos/net.py'
--- chaos/net.py	1970-01-01 00:00:00 +0000
+++ chaos/net.py	2015-04-29 03:47:51 +0000
@@ -0,0 +1,119 @@
+from chaos_monkey_base import (
+    Chaos,
+    ChaosMonkeyBase,
+)
+from utility import (
+    log,
+    run_shell_command,
+)
+
+__metaclass__ = type
+
+
+class Net(ChaosMonkeyBase):
+    """Creates networking chaos."""
+
+    def __init__(self):
+        self.group = 'net'
+        super(Net, self).__init__()
+
+    @classmethod
+    def factory(cls):
+        return cls()
+
+    def reset(self):
+        log("Net.reset ")
+        cmd = 'ufw reset'
+        run_shell_command(cmd)
+
+    def default_deny(self):
+        log("Net.default_deny")
+        cmd = "ufw default deny"
+        run_shell_command(cmd)
+
+    def default_allow(self):
+        log("Net.default_allow")
+        cmd = "ufw default allow"
+        run_shell_command(cmd)
+
+    def allow_ssh(self):
+        log("Net.allow_ssh")
+        cmd = 'ufw allow ssh'
+        run_shell_command(cmd)
+
+    def deny_ssh(self):
+        log("Net.deny_ssh")
+        cmd = 'ufw deny ssh'
+        run_shell_command(cmd)
+
+    def deny_all_incoming_and_outgoing_except_ssh(self):
+        log("Net.deny_all_incoming_and_outgoing_except_ssh")
+        self.deny_all_incoming_except_ssh()
+        self.deny_all_outgoing_except_ssh()
+
+    def allow_all_incoming_and_outgoing(self):
+        log("Net.allow_all_incoming_and_outgoing")
+        self.allow_all_incoming()
+        self.allow_all_outgoing()
+
+    def deny_all_incoming_except_ssh(self):
+        log("Net.deny_all_incoming_except_ssh")
+        self.allow_ssh()
+        self.default_deny()
+
+    def allow_all_incoming(self):
+        log("Net.allow_all_incoming")
+        self.default_allow()
+
+    def deny_all_outgoing_except_ssh(self):
+        log("Net.deny_all_outgoing_except_ssh")
+        self.allow_ssh()
+        cmd = 'ufw deny out to any'
+        run_shell_command(cmd)
+
+    def allow_all_outgoing(self):
+        log("Net.allow_all_outgoing")
+        cmd = 'ufw delete deny out to any'
+        run_shell_command(cmd)
+
+    def deny_port(self, port=8080):
+        log("Net.deny_port port=%s" % port)
+        cmd = 'ufw deny ' + str(port)
+        run_shell_command(cmd)
+
+    def get_chaos(self):
+        chaos = list()
+        chaos.append(
+            Chaos(
+                enable=self.deny_all_incoming_and_outgoing_except_ssh,
+                disable=self.allow_all_incoming_and_outgoing,
+                group=self.group,
+                command_str='deny-all'))
+        chaos.append(
+            Chaos(
+                enable=self.deny_all_incoming_except_ssh,
+                disable=self.allow_all_incoming,
+                group=self.group,
+                command_str='deny-incoming'))
+        chaos.append(
+            Chaos(
+                enable=self.deny_all_outgoing_except_ssh,
+                disable=self.allow_all_outgoing,
+                group=self.group,
+                command_str='deny-outgoing'))
+        chaos.append(
+            Chaos(
+                enable=self.allow_ssh,
+                disable=None,
+                group=self.group,
+                command_str='allow-ssh'))
+        chaos.append(
+            Chaos(
+                enable=self.deny_ssh,
+                disable=self.allow_ssh,
+                group=self.group,
+                command_str='deny-ssh'))
+        return chaos
+
+    def shutdown(self):
+        self.reset()

=== added file 'chaos_monkey.py'
--- chaos_monkey.py	1970-01-01 00:00:00 +0000
+++ chaos_monkey.py	2015-04-29 03:47:51 +0000
@@ -0,0 +1,58 @@
+import random
+from time import sleep
+
+from chaos import (
+    kill,
+    net
+)
+from utility import NotFound
+
+__metaclass__ = type
+
+
+class ChaosMonkey:
+    """Runs chaos monkey commands."""
+
+    def __init__(self, chaos, factory_obj):
+        self.chaos = chaos
+        self.factory_obj = factory_obj
+
+    @classmethod
+    def factory(cls):
+        all_chaos, factory_obj = ChaosMonkey.get_all_chaos()
+        return cls(all_chaos, factory_obj)
+
+    @staticmethod
+    def get_all_chaos():
+        all_chaos = []
+        all_factory_obj = []
+        factories = [net.Net.factory, kill.Kill.factory]
+        for factory in factories:
+            factory_obj = factory()
+            all_factory_obj.append(factory_obj)
+            all_chaos.extend(factory_obj.get_chaos())
+        return all_chaos, all_factory_obj
+
+    def run_random_chaos(self, timeout=2):
+        random_chaos = random.choice(self.chaos)
+        self._run_command(random_chaos, timeout=timeout)
+
+    def run_chaos(self, group, command_str, timeout=2):
+        command_found = False
+        for chaos in self.chaos:
+            if chaos.group == group and chaos.command_str == command_str:
+                command_found = True
+                self._run_command(chaos, timeout=timeout)
+        if not command_found:
+            raise NotFound("Command not found: group: %s command_str:%s" % (
+                group, command_str))
+
+    def _run_command(self, chaos, timeout=2):
+        chaos.enable()
+        sleep(timeout)
+        if chaos.disable:
+            chaos.disable()
+
+    def shutdown(self):
+        for obj in self.factory_obj:
+            obj.shutdown()

=== added file 'chaos_monkey_base.py'
--- chaos_monkey_base.py	1970-01-01 00:00:00 +0000
+++ chaos_monkey_base.py	2015-04-29 03:47:51 +0000
@@ -0,0 +1,27 @@
+import abc
+
+__metaclass__ = type
+
+
+class ChaosMonkeyBase:
+    __metaclass__ = abc.ABCMeta
+
+    def __init__(self):
+        pass
+
+    @abc.abstractmethod
+    def get_chaos(self):
+        raise NotImplemented
+
+    @abc.abstractmethod
+    def shutdown(self):
+        raise NotImplemented
+
+
+class Chaos:
+
+    def __init__(self, enable, disable, group, command_str):
+        self.enable = enable
+        self.disable = disable
+        self.group = group
+        self.command_str = command_str

=== added file 'runner.py'
--- runner.py	1970-01-01 00:00:00 +0000
+++ runner.py	2015-04-29 03:47:51 +0000
@@ -0,0 +1,40 @@
+from argparse import ArgumentParser
+from time import time
+
+from chaos_monkey import ChaosMonkey
+from utility import BadRequest
+
+
+def random(run_timeout, enablement_timeout):
+    """
+    Runs a random chaos
+    :param run_timeout: Total time to run the chaos
+    :param enablement_timeout: Timeout between enabling and disabling a chaos.
+    Example: disable all the network, wait for timeout and enable it back
+    :rtype none
+    """
+    if enablement_timeout > run_timeout:
+        raise BadRequest(
+            "Total run timeout can't be less than enablement timeout")
+    if run_timeout <= 0:
+        raise BadRequest("Invalid value for run timeout")
+    if enablement_timeout < 0:
+        raise BadRequest("Invalid value for enablement timeout")
+    cm = ChaosMonkey.factory()
+    expire_time = time() + run_timeout
+    while time() < expire_time:
+        cm.run_random_chaos(enablement_timeout)
+    cm.shutdown()
+
+
+if __name__ == '__main__':
+    parser = ArgumentParser()
+    parser.add_argument(
+        '-pt', '--enablement-timeout', default=10, type=int,
+        help="Enablement timeout in seconds")
+    parser.add_argument(
+        '-tt', '--total-timeout', default=60, type=int,
+        help="Total timeout in seconds")
+    args = parser.parse_args()
+    random(run_timeout=args.total_timeout,
+           enablement_timeout=args.enablement_timeout)

=== added directory 'tests'
=== added file 'tests/test_chaos_monkey.py'
--- tests/test_chaos_monkey.py	1970-01-01 00:00:00 +0000
+++ tests/test_chaos_monkey.py	2015-04-29 03:47:51 +0000
@@ -0,0 +1,85 @@
+from unittest import TestCase
+
+from mock import patch, call
+
+from chaos_monkey import ChaosMonkey, NotFound
+from chaos_monkey_base import Chaos
+from chaos.net import Net
+
+
+class TestChaosMonkey(TestCase):
+
+    def test_factory(self):
+        cm = ChaosMonkey.factory()
+        self.assertIs(type(cm), ChaosMonkey)
+
+    def test_get_all_chaos(self):
+        cm = ChaosMonkey.factory()
+        all_chaos, all_factory_obj = cm.get_all_chaos()
+        self.assertItemsEqual(
+            self._get_all_command_str(all_chaos),  self._command_strings())
+
+    def test_run_random_chaos(self):
+        cm = ChaosMonkey.factory()
+        with patch('utility.check_output', autospec=True) as mock:
+            cm.run_random_chaos(timeout=0)
+        self.assertEqual(mock.called, True)
+
+    def test_run_random_chaos_passes_timeout(self):
+        cm = ChaosMonkey.factory()
+        with patch('chaos_monkey.ChaosMonkey._run_command',
+                   autospec=True) as mock:
+            cm.run_random_chaos(timeout=1)
+        self.assertEqual(1, mock.call_args_list[0][1]['timeout'])
+
+    def test_run_chaos(self):
+        cm = ChaosMonkey.factory()
+        with patch('utility.check_output', autospec=True) as mock:
+            cm.run_chaos('net', 'allow-ssh', timeout=0)
+        mock.assert_called_once_with(['ufw', 'allow', 'ssh'])
+
+    def test_run_chaos_passes_timeout(self):
+        cm = ChaosMonkey.factory()
+        with patch('chaos_monkey.ChaosMonkey._run_command',
+                   autospec=True) as mock:
+            cm.run_chaos('net', 'allow-ssh', timeout=0)
+        self.assertEqual(0, mock.call_args_list[0][1]['timeout'])
+
+    def test_run_chaos_raises_for_command_str(self):
+        cm = ChaosMonkey.factory()
+        with patch('utility.check_output', autospec=True):
+            with self.assertRaisesRegexp(
+                    NotFound,
+                    "Command not found: group: net command_str:foo"):
+                cm.run_chaos('net', 'foo', timeout=0)
+
+    def test_run_chaos_raises_for_group(self):
+        cm = ChaosMonkey.factory()
+        with patch('utility.check_output', autospec=True):
+            with self.assertRaisesRegexp(
+                    NotFound,
+                    "Command not found: group: bar command_str:allow-ssh"):
+                cm.run_chaos('bar', 'allow-ssh', timeout=0)
+
+    def test_run_command(self):
+        cm = ChaosMonkey.factory()
+        net = Net()
+        chaos = Chaos(enable=net.deny_ssh, disable=net.allow_ssh,
+                      group='net', command_str='deny-ssh')
+        with patch('utility.check_output', autospec=True) as mock:
+            cm._run_command(chaos, timeout=0)
+        self.assertEqual(mock.mock_calls, [
+            call(['ufw', 'deny', 'ssh']), call(['ufw', 'allow', 'ssh'])])
+
+    def test_shutdown(self):
+        cm = ChaosMonkey.factory()
+        with patch('utility.check_output', autospec=True) as mock:
+            cm.shutdown()
+        mock.assert_any_call(['ufw', 'reset'])
+
+    def _get_all_command_str(self, chaos):
+        return [c.command_str for c in chaos]
+
+    def _command_strings(self):
+        return ['deny-all', 'deny-incoming', 'deny-outgoing', 'allow-ssh',
+                'deny-ssh', 'jujud', 'mongod']

=== added file 'tests/test_kill.py'
--- tests/test_kill.py	1970-01-01 00:00:00 +0000
+++ tests/test_kill.py	2015-04-29 03:47:51 +0000
@@ -0,0 +1,41 @@
+from unittest import TestCase
+
+from mock import patch, call
+
+from chaos.kill import Kill
+
+
+class TestKill(TestCase):
+
+    def test_get_pids(self):
+        kill = Kill()
+        with patch('utility.check_output', autospec=True,
+                   return_value='1234 2345') as mock:
+            pids = kill.get_pids('jujud')
+        self.assertEqual(pids, ['1234', '2345'])
+        mock.assert_called_once_with(['pidof', 'jujud'])
+
+    def test_kill_jujud(self):
+        kill = Kill()
+        with patch('utility.check_output', autospec=True,
+                   return_value='1234 2345') as mock:
+            kill.kill_jujud()
+        self.assertEqual(mock.mock_calls, [
+            call(['pidof', 'jujud']),
+            call(['kill', '-s', 'SIGKILL', '1234'])
+        ])
+
+    def test_kill_mongodb(self):
+        kill = Kill()
+        with patch('utility.check_output', autospec=True,
+                   return_value='1234 2345') as mock:
+            kill.kill_mongodb()
+        self.assertEqual(mock.mock_calls, [
+            call(['pidof', 'mongod']),
+            call(['kill', '-s', 'SIGKILL', '1234'])
+        ])
+
+    def test_get_chaos(self):
+        kill = Kill()
+        chaos = kill.get_chaos()
+        self.assertEqual(len(chaos), 2)

=== added file 'tests/test_net.py'
--- tests/test_net.py	1970-01-01 00:00:00 +0000
+++ tests/test_net.py	2015-04-29 03:47:51 +0000
@@ -0,0 +1,99 @@
+from unittest import TestCase
+
+from mock import patch, call
+
+from chaos.net import Net
+
+
+class TestNet(TestCase):
+
+    def _run_test(self, method_call, assert_args):
+        net = Net()
+        with patch('utility.check_output', autospec=True) as mock:
+            getattr(net, method_call)()
+        mock.assert_called_once_with(assert_args)
+
+    def _run_mock_calls(self, method_call, call_list):
+        net = Net()
+        with patch('utility.check_output', autospec=True) as mock:
+            getattr(net, method_call)()
+        self.assertEqual(mock.mock_calls, call_list)
+
+    def test_reset(self):
+        self._run_test('reset', ['ufw', 'reset'])
+
+    def test_default_deny(self):
+        self._run_test('default_deny', ['ufw', 'default', 'deny'])
+
+    def test_default_allow(self):
+        self._run_test('default_allow', ['ufw', 'default', 'allow'])
+
+    def test_allow_ssh(self):
+        self._run_test('allow_ssh', ['ufw', 'allow', 'ssh'])
+
+    def test_deny_ssh(self):
+        self._run_test('deny_ssh', ['ufw', 'deny', 'ssh'])
+
+    def test_deny_all_incoming_and_outgoing_except_ssh(self):
+        self._run_mock_calls(
+            'deny_all_incoming_and_outgoing_except_ssh',
+            [self._allow_ssh_call(), self._default_deny_call(),
+             self._allow_ssh_call(),
+             call(['ufw', 'deny', 'out', 'to', 'any'])])
+
+    def test_allow_all_incoming_and_outgoing(self):
+        self._run_mock_calls(
+            'allow_all_incoming_and_outgoing',
+            [self._default_allow_call(),
+             call(['ufw', 'delete', 'deny', 'out', 'to', 'any'])])
+
+    def test_deny_all_incoming_except_ssh(self):
+        self._run_mock_calls(
+            'deny_all_incoming_except_ssh',
+            [self._allow_ssh_call(), self._default_deny_call()])
+
+    def test_allow_all_incoming(self):
+        self._run_test('allow_all_incoming', ['ufw', 'default', 'allow'])
+
+    def test_deny_all_outgoing_except_ssh(self):
+        self._run_mock_calls(
+            'deny_all_outgoing_except_ssh',
+            [self._allow_ssh_call(),
+             call(['ufw', 'deny', 'out', 'to', 'any'])])
+
+    def test_allow_all_outgoing(self):
+        self._run_test(
+            'allow_all_outgoing',
+            ['ufw', 'delete', 'deny', 'out', 'to', 'any'])
+
+    def test_deny_port(self):
+        self._run_test(
+            'deny_port', ['ufw', 'deny', '8080'])
+
+    def test_get_chaos(self):
+        net = Net()
+        chaos = net.get_chaos()
+        self.assertEqual(len(chaos),  5)
+        self.assertItemsEqual(
+            self._get_all_command_str(chaos), self._command_strings())
+        for c in chaos:
+            self.assertEqual('net', c.group)
+
+    def _get_all_command_str(self, chaos):
+        return [c.command_str for c in chaos]
+
+    def _command_strings(self):
+        return ['deny-all', 'deny-incoming', 'deny-outgoing', 'allow-ssh',
+                'deny-ssh']
+
+    def test_shutdown(self):
+        self._run_test('reset', ['ufw', 'reset'])
+
+    def _allow_ssh_call(self):
+        return call(['ufw', 'allow', 'ssh'])
+
+    def _default_deny_call(self):
+        return call(['ufw', 'default', 'deny'])
+
+    def _default_allow_call(self):
+        return call(['ufw', 'default', 'allow'])

=== added file 'tests/test_runner.py'
--- tests/test_runner.py	1970-01-01 00:00:00 +0000
+++ tests/test_runner.py	2015-04-29 03:47:51 +0000
@@ -0,0 +1,81 @@
+from time import time
+from unittest import TestCase
+
+from mock import patch
+
+from utility import BadRequest
+from runner import random
+
+
+class TestRunner(TestCase):
+
+    def test_random(self):
+        with patch('utility.check_output', autospec=True) as mock:
+            random(run_timeout=1, enablement_timeout=1)
+        self.assertEqual(mock.called, True)
+
+    def test_random_enablement_zero(self):
+        with patch('utility.check_output', autospec=True) as mock:
+            random(run_timeout=1, enablement_timeout=0)
+        self.assertEqual(mock.called, True)
+
+    def test_random_raises_exception(self):
+        with patch('utility.check_output', autospec=True):
+            with self.assertRaisesRegexp(
+                    BadRequest,
+                    "Total run timeout can't be less than enablement timeout"):
+                random(run_timeout=1, enablement_timeout=2)
+
+    def test_random_run_timeout_raises_exception_for_zero(self):
+        with patch('utility.check_output', autospec=True):
+            with self.assertRaisesRegexp(
+                    BadRequest, "Invalid value for run timeout"):
+                random(run_timeout=0, enablement_timeout=-1)
+
+    def test_random_run_timeout_raises_exception_for_less_than_zero(self):
+        with patch('utility.check_output', autospec=True):
+            with self.assertRaisesRegexp(
+                    BadRequest, "Invalid value for run timeout"):
+                random(run_timeout=-1, enablement_timeout=-2)
+
+    def test_random_run_enablement_raises_exception_for_less_than_zero(self):
+        with patch('utility.check_output', autospec=True):
+            with self.assertRaisesRegexp(
+                    BadRequest, "Invalid value for enablement timeout"):
+                random(run_timeout=2, enablement_timeout=-1)
+
+    def test_random_verify_timeout(self):
+        run_timeout = 6
+        with patch('utility.check_output', autospec=True) as mock:
+            current_time = time()
+            random(run_timeout=run_timeout, enablement_timeout=2)
+            end_time = time()
+        self.assertEqual(run_timeout, int(end_time-current_time))
+        self.assertEqual(mock.called, True)
+
+    def test_random_assert_chaos_monkey_methods_called(self):
+        with patch('runner.ChaosMonkey', autospec=True) as cm_mock:
+            random(run_timeout=1, enablement_timeout=1)
+        cm_mock.factory.return_value.run_random_chaos.assert_called_with(1)
+        cm_mock.factory.return_value.shutdown.assert_called_with()
+
+    def test_random_assert_chaos_methods_called(self):
+        net_ctx = patch('chaos.net.Net', autospec=True)
+        kill_ctx = patch('chaos.kill.Kill', autospec=True)
+        with patch('utility.check_output', autospec=True):
+            with patch('chaos_monkey.ChaosMonkey.run_random_chaos',
+                       autospec=True):
+                with net_ctx as net_mock:
+                    with kill_ctx as kill_mock:
+                        random(run_timeout=1, enablement_timeout=1)
+        net_mock.factory.return_value.shutdown.assert_called_with()
+        kill_mock.factory.return_value.shutdown.assert_called_with()
+        net_mock.factory.return_value.get_chaos.assert_called_with()
+        kill_mock.factory.return_value.get_chaos.assert_called_with()
+
+    def test_random_passes_timeout(self):
+        with patch('utility.check_output', autospec=True):
+            with patch('chaos_monkey.ChaosMonkey._run_command',
+                       autospec=True) as mock:
+                random(run_timeout=3, enablement_timeout=2)
+        self.assertEqual(mock.call_args_list[0][1]['timeout'], 2)

=== added file 'tests/test_utility.py'
--- tests/test_utility.py	1970-01-01 00:00:00 +0000
+++ tests/test_utility.py	2015-04-29 03:47:51 +0000
@@ -0,0 +1,22 @@
+from subprocess import CalledProcessError
+from unittest import TestCase
+
+from mock import patch
+
+from utility import run_shell_command
+
+
+class TestUtility(TestCase):
+
+    def test_run_shell_command(self):
+        with patch('utility.check_output', autospec=True) as mock:
+            run_shell_command('foo')
+        mock.assert_called_once_with(['foo'])
+
+    def test_run_shell_command_error(self):
+        with self.assertRaisesRegexp(CalledProcessError, ""):
+            run_shell_command('ls -W', quiet_mode=False)
+
+    def test_run_shell_command_output(self):
+        output = run_shell_command('echo "hello"')
+        self.assertEqual(output, '"hello"\n')

=== added file 'utility.py'
--- utility.py	1970-01-01 00:00:00 +0000
+++ utility.py	2015-04-29 03:47:51 +0000
@@ -0,0 +1,36 @@
+from __future__ import print_function
+
+from subprocess import (
+    CalledProcessError,
+    check_output,
+)
+
+
+def run_shell_command(cmd, quiet_mode=True):
+    shell_cmd = cmd.split(' ') if type(cmd) is str else cmd
+    output = None
+    try:
+        output = check_output(shell_cmd)
+    except CalledProcessError:
+        log("Command generated error: %s " % cmd)
+        if not quiet_mode:
+            raise
+    return output
+
+
+def log(log_str):
+    print(log_str)
+
+
+class NotFound(Exception):
+    """
+    Requested resource not found
+    """
+    error_code = 404
+
+
+class BadRequest(Exception):
+    """
+    Incorrectly formatted  request
+    """
+    error_code = 400