Merge lp:~vila/uci-engine/integration into lp:uci-engine

[Vincent Ladeuil]

Complete run-tests to support all tests defined in the project (still requires a venv).

This refactor deploy.py and run.py to share more code so run-tests can enjoy it too.

Some renaming had to be done for the integration test files so python can import them properly (and use a cleaner namespace).

At that point, run-tests still requires a venv but that MP is big enough, I'll address that in a followup.

[Evan (ev)]

=== modified file 'branch-source-builder/bsbuilder/run_worker.py' (properties changed: +x to -x)
=== modified file 'image-builder/imagebuilder/run_worker.py' (properties changed: +x to -x)
=== modified file 'test_runner/tstrun/run_worker.py' (properties changed: +x to -x)

:)

review: Needs Fixing

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:291
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/490/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/490/rebuild

review: Needs Fixing (continuous-integration)

[Evan (ev)]

  % time ./juju-deployer/deploy.py test-runner
Traceback (most recent call last):
  File "./juju-deployer/deploy.py", line 799, in <module>
    main()
  File "./juju-deployer/deploy.py", line 770, in main
    check_and_set_ppa_privacy(args.private_ppas_only)
  File "./juju-deployer/deploy.py", line 51, in check_and_set_ppa_privacy
    current_value = os.environ['CI_PRIVATE_PPAS_ONLY']
  File "/usr/lib/python2.7/UserDict.py", line 23, in __getitem__
    raise KeyError(key)
KeyError: 'CI_PRIVATE_PPAS_ONLY'

This is because check_and_set_ppa_privacy() is called before setup() and thus check_environment() where the environment variables are set to their defaults.

review: Needs Fixing

[Evan (ev)]

 % time ./juju-deployer/deploy.py test-runner
Private PPAs only is DISABLED (CI_PRIVATE_PPAS_ONLY: False)
Checking juju status
Traceback (most recent call last):
  File "./juju-deployer/deploy.py", line 799, in <module>
    main()
  File "./juju-deployer/deploy.py", line 773, in main
    check_dependencies()
  File "./juju-deployer/deploy.py", line 378, in check_dependencies
    if not distutils.spawn.find_executable(cmd):
AttributeError: 'module' object has no attribute 'spawn'

92 +import distutils
...
110 -from distutils import spawn

This should be import distutils.spawn

review: Needs Fixing

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:292
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/491/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/491/rebuild

review: Needs Fixing (continuous-integration)

[Evan (ev)]

  % time ./juju-deployer/deploy.py test-runner
Private PPAs only is DISABLED (CI_PRIVATE_PPAS_ONLY: False)
Checking juju status
Preparing local branch upload...
No handlers could be found for logger "keystoneclient.httpclient"
Uploading local branch, fingerprint 20b63b0bb3e58ec84ca6fb6348d163179b25f812
No revisions or tags to pull.
No revisions or tags to pull.
No revisions or tags to pull.
Traceback (most recent call last):
  File "./juju-deployer/deploy.py", line 799, in <module>
    main()
  File "./juju-deployer/deploy.py", line 780, in main
    setup()
  File "./juju-deployer/deploy.py", line 752, in setup
    os.environ['JUJU_DEPLOYER_DIR'] = working_dir
UnboundLocalError: local variable 'working_dir' referenced before assignment

Looks like you're missing an indent on:
os.environ['JUJU_DEPLOYER_DIR'] = working_dir

review: Needs Fixing

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:298
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/503/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/503/rebuild

review: Approve (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:299
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/505/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/505/rebuild

review: Approve (continuous-integration)

[Evan (ev)]

This works a treat now. +1

review: Approve

[Ubuntu CI Bot (uci-bot)]

The attempt to merge lp:~vila/uci-engine/integration into lp:uci-engine failed. Below is the output from the failed tests.

New python executable in /tmp/tmp.DVyq0XABre/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
== Testing ci-utils ....
Unpacking /tmp/tarmac/branch.Y5rJ_H/.deps/Babel-1.3.tar.gz
  Running setup.py egg_info for package from file:///tmp/tarmac/branch.Y5rJ_H/.deps/Babel-1.3.tar.gz

    warning: no previously-included files matching '*' found under directory 'docs/_build'
    warning: no previously-included files matching '*.pyc' found under directory 'tests'
    warning: no previously-included files matching '*.pyo' found under directory 'tests'
Installing collected packages: Babel
  Running setup.py install for Babel

    warning: no previously-included files matching '*' found under directory 'docs/_build'
    warning: no previously-included files matching '*.pyc' found under directory 'tests'
    warning: no previously-included files matching '*.pyo' found under directory 'tests'
    Installing pybabel script to /tmp/tmp.DVyq0XABre/bin
Successfully installed Babel
Cleaning up...
Using saved parent location: bzr+ssh://bazaar.launchpad.net/~canonical-ci-engineering/uci-engine/deps/
No revisions or tags to pull.
Unpacking /tmp/tarmac/branch.Y5rJ_H/.deps/pbr-0.6.tar.gz
  Running setup.py egg_info for package from file:///tmp/tarmac/branch.Y5rJ_H/.deps/pbr-0.6.tar.gz
    [pbr] Processing SOURCES.txt
    warning: LocalManifestMaker: standard file '-c' not found

    warning: no previously-included files found matching '.gitignore'
    warning: no previously-included files found matching '.gitreview'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files found matching '.gitignore'
    warning: no previously-included files found matching '.gitreview'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
Installing collected packages: pbr
  Running setup.py install for pbr
    [pbr] Reusing existing SOURCES.txt
Successfully installed pbr
Cleaning up...
Using saved parent location: bzr+ssh://bazaar.launchpad.net/~canonical-ci-engineering/uci-engine/deps/
No revisions or tags to pull.
Unpacking /tmp/tarmac/branch.Y5rJ_H/.deps/iso8601-0.1.8.tar.gz
  Running setup.py egg_info for package from file:///tmp/tarmac/branch.Y5rJ_H/.deps/iso8601-0.1.8.tar.gz

Installing collected packages: iso8601
  Running setup.py install for iso8601

Successfully installed iso8601
Cleaning up...
Using saved parent location: bzr+ssh://bazaar.launchpad.net/~canonical-ci-engineering/uci-engine/deps/
No revisions or tags to pull.
Unpacking /tmp/tarmac/branch.Y5rJ_H/.deps/prettytable-0.7.2.zip
  Running setup.py egg_info for package from file:///tmp/tarmac/branch.Y5rJ_H/.deps/prettytable-0.7.2.zip

Installing collected packages: prettytable
  Running setup.py install for prettytable

Successfully installed pret...

[lp:~vila/uci-engine/integration]

430. By Vincent Ladeuil on 2014-04-15

[r=Evan Dandrea, PS Jenkins bot] Complete run-tests to support all tests defined in the project (still requires a venv). from Vincent Ladeuil

=== modified file 'cli/tests/test_image.py'
--- cli/tests/test_image.py	2014-03-17 17:17:25 +0000
+++ cli/tests/test_image.py	2014-04-15 12:18:41 +0000
@@ -28,7 +28,6 @@
 from tests import capture_stdout
 
 import urllib2
-urllib2.urlopen = mock.Mock()
 
 
 class FakeUrlOpenReturn:
@@ -94,9 +93,10 @@
         if os.path.exists(self.image_path):
             os.remove(self.image_path)
 
+    @mock.patch('urllib2.urlopen')
     @mock.patch('ci_utils.data_store.create_for_ticket')
-    def test_get_ticket_image(self, mock_create_datastore):
-        urllib2.urlopen.side_effect = [
+    def test_get_ticket_image(self, mock_create_datastore, mock_urlopen):
+        mock_urlopen.side_effect = [
             FakeUrlOpenReturn(200, json.dumps(artifact_data))]
         mock_datastore = mock.Mock()
         mock_datastore.get_file.return_value = buffer(" ")
@@ -111,9 +111,11 @@
                 "Image file can be found at: {}\n".format(self.image_path))
         mock_datastore.assert_called_once()
 
+    @mock.patch('urllib2.urlopen')
     @mock.patch('ci_utils.data_store.create_for_ticket')
-    def test_get_ticket_image_not_found(self, mock_create_datastore):
-        urllib2.urlopen.side_effect = [
+    def test_get_ticket_image_not_found(self, mock_create_datastore,
+                                        mock_urlopen):
+        mock_urlopen.side_effect = [
             FakeUrlOpenReturn(200, json.dumps(artifact_data))]
         mock_datastore = mock.Mock()
         mock_datastore.get_file.side_effect = data_store.DataStoreException(
@@ -128,9 +130,11 @@
         with self.assertRaises(ImageObjectNotFound):
             args.func(args)
 
+    @mock.patch('urllib2.urlopen')
     @mock.patch('ci_utils.data_store.DataStore')
-    def test_get_ticket_image_already_exists(self, mock_data_store):
-        urllib2.urlopen.side_effect = [
+    def test_get_ticket_image_already_exists(self, mock_data_store,
+                                             mock_urlopen):
+        mock_urlopen.side_effect = [
             FakeUrlOpenReturn(200, json.dumps(artifact_data))]
         open(self.image_path, 'a').close()
         args = self.cli.parse_arguments(['get_image', '-t', '4', '-n',
@@ -140,9 +144,11 @@
                 "{} already exists.\n".format(self.image_path),
                 cm)
 
+    @mock.patch('urllib2.urlopen')
     @mock.patch('ci_utils.data_store.create_for_ticket')
-    def test_get_last_completed_ticket_image(self, mock_create_datastore):
-        urllib2.urlopen.side_effect = [
+    def test_get_last_completed_ticket_image(self, mock_create_datastore,
+                                             mock_urlopen):
+        mock_urlopen.side_effect = [
             FakeUrlOpenReturn(200, json.dumps(ticket_data)),
             FakeUrlOpenReturn(200, json.dumps(artifact_data))
         ]
@@ -158,9 +164,10 @@
                 "Image file can be found at: {}\n".format(self.image_path))
         mock_datastore.assert_called_once()
 
-    def test_get_last_completed_ticket_none_complete(self):
+    @mock.patch('urllib2.urlopen')
+    def test_get_last_completed_ticket_none_complete(self, mock_urlopen):
         data = {'objects': []}
-        urllib2.urlopen.side_effect = [
+        mock_urlopen.side_effect = [
             FakeUrlOpenReturn(200, json.dumps(data))]
         args = self.cli.parse_arguments(['get_image', '-n', self.image_path])
         with self.assertRaises(SystemExit) as cm:

=== modified file 'juju-deployer/deploy.py'
--- juju-deployer/deploy.py	2014-04-09 21:23:35 +0000
+++ juju-deployer/deploy.py	2014-04-15 12:18:41 +0000
@@ -1,19 +1,37 @@
 #!/usr/bin/python
+# Ubuntu Continuous Integration Engine
+# Copyright 2013, 2014 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import argparse
+import atexit
 import base64
+from  distutils import spawn
 import hashlib
+import itertools
+import logging
 import os
+import shutil
+import signal
+import stat
+import subprocess
 import sys
-import subprocess
-import stat
-import atexit
-import shutil
 import tempfile
 import textwrap
+import time
 import yaml
-from distutils import spawn
-from itertools import chain
+
 
 import bzrlib
 from bzrlib import (
@@ -21,9 +39,15 @@
     plugin,
     workingtree,
 )
-
-
+from novaclient.v1_1 import client as nova_client
+import novaclient.exceptions
+import swiftclient
+
+
+log = logging.getLogger(__name__)
 HERE = os.path.abspath(os.path.dirname(__file__))
+SSH_OPTS = ['-o', 'UserKnownHostsFile=/dev/null',
+            '-o', 'StrictHostKeyChecking=no']
 
 # the juju-deployer/ directory.
 deployer_dir = os.path.abspath(os.path.dirname(sys.argv[0]))
@@ -37,29 +61,11 @@
 plugin.load_plugins()
 
 
-def check_and_set_ppa_privacy(private_ppas_only):
-    current_value = os.environ['CI_PRIVATE_PPAS_ONLY']
-    if private_ppas_only:
-        value = 'True'
-        status = 'ENABLED'
-    elif current_value not in ['0', '1']:
-        print('Invalid CI_PRIVATE_PPAS_ONLY value. Options are 0 (disabled) '
-              'and 1 (enabled). Provided value is: "{}"'.format(current_value))
-        sys.exit(1)
-    else:
-        if current_value == '0':
-            value = 'False'
-            status = 'DISABLED'
-        else:
-            value = 'True'
-            status = 'ENABLED'
-    os.environ['CI_PRIVATE_PPAS_ONLY'] = value
-    print('Private PPAs only is {} (CI_PRIVATE_PPAS_ONLY: {})'.format(status,
-          value))
-
-
 def check_environment():
-    '''Cheetah requires environment variables for populating our templates'''
+    '''Cheetah requires environment variables for populating our templates.
+
+    Some default values are also set here as a side-effect.
+    '''
     cheetah_vars = {
         'OS_USERNAME': None,
         'CI_CODE_SOURCE': 'branch',
@@ -91,6 +97,258 @@
         exit(template.format('\n  '.join(missing_required)))
 
 
+def bootstrap():
+    """Bootstrap juju. Specify needed as bootstrap to just bootstrap, otherwise
+    tunnel to bootstrap and tunnel."""
+
+    args = ['juju', 'bootstrap', '--constraints=mem=1024M']
+    try:
+        subprocess.check_output(args)
+    except subprocess.CalledProcessError as e:
+        # FIXME: if check_output raise an exception, we don't get a proper
+        # output to check for the text below -- vila 2014-02-07
+        if 'environment is already bootstrapped' not in e.output:
+            raise
+
+
+def _get_control_bucket():
+    """Get the name of the bucket in Swift that juju uses for state
+    information."""
+
+    default_home = os.path.join(os.path.expanduser('~'), '.juju')
+    juju_home = os.environ.get('JUJU_HOME') or default_home
+
+    with open(os.path.join(juju_home, 'environments.yaml')) as fp:
+        envs = yaml.safe_load(fp.read())
+
+    juju_env = os.environ.get('JUJU_ENV')
+    if juju_env is None:
+        juju_env = envs.get('default')
+    if not juju_env:
+        log.error('No juju environment specified.')
+        sys.exit(1)
+
+    return envs['environments'][juju_env]['control-bucket']
+
+
+def _delete_control_bucket():
+    """Remove the juju control bucket and all of its files."""
+
+    bucket = _get_control_bucket()
+    conn = _connect_to_swift()
+    try:
+        files = [x['name'] for x in conn.get_container(bucket)[1]]
+        for f in files:
+            conn.delete_object(bucket, f)
+        conn.delete_container(bucket)
+    except swiftclient.exceptions.ClientException:
+        # Assume the bucket is not present
+        pass
+
+
+def _connect_to_swift():
+    opts = {'tenant_name': os.environ['OS_TENANT_NAME'],
+            'region_name': os.environ['OS_REGION_NAME'],
+            }
+    conn = swiftclient.client.Connection(
+        os.environ['OS_AUTH_URL'],
+        os.environ['OS_USERNAME'],
+        os.environ['OS_PASSWORD'],
+        os_options=opts,
+        auth_version='2.0')
+    return conn
+
+
+def _get_from_swift(bucket, path):
+    """Download the file path from swift at bucket."""
+
+    conn = _connect_to_swift()
+    # get_object return a tuple of (metadata, data)
+    return conn.get_object(bucket, path)[1]
+
+
+def get_bootstrap_ip(instance, max_tries=120):
+    """Wait for the juju bootstrap node to boot and obtain an IP address.
+
+    This is a pre-requisite to setup a tunnel.
+
+    :raises: NotFound if the instance has not been spawned.
+
+    :raises: Exception if no IP can be found after max_tries*3 seconds.
+    """
+
+    log.info("Getting the bootstrap IP address.")
+
+    args = [os.environ['OS_USERNAME'], os.environ['OS_PASSWORD'],
+            os.environ['OS_TENANT_NAME'], os.environ['OS_AUTH_URL']]
+    kwargs = {'region_name': os.environ['OS_REGION_NAME'],
+              'service_type': 'compute'}
+    nt = nova_client.Client(*args, **kwargs)
+
+    tries = 0
+    while tries < max_tries:
+        # Raises NotFound if the instance does not exist.
+        server = nt.servers.get(instance)
+        networks = server.networks.values()
+        if networks:
+            # We use a single network and don't care how it's named
+            net = networks[0]
+            if net:
+                # Either we have a single address or the last one has been
+                # added to make the instance reachable (floating or public).
+                net = net[-1]
+                # Don't sleep anymore. We have our IP, let's move on
+                return net
+
+        tries += 1
+        if tries != max_tries:
+            time.sleep(3)
+    raise Exception('Timed out waiting for bootstrap IP.')
+
+
+def get_bootstrap_nova_instance():
+    """Get the Nova instance identifier for the juju bootstrap node.
+
+    Raises swiftclient.exceptions.ClientException if the instance cannot be
+    found."""
+
+    bucket = _get_control_bucket()
+
+    state = yaml.safe_load(_get_from_swift(bucket, 'provider-state'))
+    if not state:
+        # Cause unknown (yet) but this has been seen in real life. I can't
+        # reproduce anymore but I suspect 'juju bootstrap' failed mid-way so
+        # returning None and letting the caller delete the bucket is the Right
+        # Thing to do -- vila 2014-02-07
+        log.warning("Empty 'provider-state' in the juju bucket")
+        return None
+    return state['state-instances'][0]
+
+
+def wait_for_node(ip):
+    """Wait for the bootstrap node to become available on ssh. We need to wait
+    for this state before attempting to set up a sshuttle."""
+
+    log.info("Waiting for the bootstrap node to respond to ssh.")
+
+    cmd = ['ssh'] + SSH_OPTS + ['ubuntu@%s' % ip, 'exit']
+    tries = 0
+    while tries < 120:
+        try:
+            subprocess.check_call(cmd)
+            return
+        except subprocess.CalledProcessError:
+            tries += 1
+            time.sleep(5)
+
+    raise Exception('Timed out waiting for bootstrap node.')
+
+
+def destroy_tunnel():
+    if os.path.exists('sshuttle.pid'):
+        with open('sshuttle.pid') as fp:
+            pid = int(fp.read())
+            os.kill(pid, signal.SIGTERM)
+
+
+def tunnel(ip):
+    """Tunnel to the bootstrap node using sshuttle."""
+
+    log.info("Tunneling to the bootstrap node.")
+
+    host = 'ubuntu@%s' % ip
+    mask = '10.55.0.0/16'
+    cmd = ['sshuttle', '-D', '-r', host, mask]
+    opt = ['-e', 'ssh %s' % ' '.join(SSH_OPTS)]
+    subprocess.check_call(cmd + opt)
+    atexit.register(destroy_tunnel)
+
+
+def wait_then_tunnel():
+    instance = get_bootstrap_nova_instance()
+    ip = get_bootstrap_ip(instance)
+    wait_for_node(ip)
+    tunnel(ip)
+
+
+def needs_bootstrap():
+    """Check to see if there's a Swift bucket for juju. If not, we need to
+    boostrap. Check to see if the IP of the bootstrap node is owned by a
+    running Nova instance. If not, we need to bootstrap."""
+
+    log.info("Checking to see if a bootstrap is needed.")
+
+    def inconsistent_state(msg):
+        log.warning(msg)
+        _delete_control_bucket()
+
+    try:
+        instance = get_bootstrap_nova_instance()
+    except swiftclient.exceptions.ClientException:
+        instance = None
+    if instance is None:
+        # We don't have an instance. We need to bootstrap.
+        inconsistent_state('No bootstrap instance in the swift bucket')
+        return True
+
+    try:
+        get_bootstrap_ip(instance, 1)
+    except novaclient.exceptions.NotFound:
+        # Juju claims we have an instance that does not exist. We're in an
+        # inconsistent state. Delete the bucket and bootstrap.
+        inconsistent_state('No IP for the bootstrap instance')
+        return True
+
+    return False
+
+
+def _get_pid_of_sshuttle(ip_addr):
+    try:
+        cmd = ['pgrep', '-f', 'sshuttle.*%s' % ip_addr]
+        pid = subprocess.check_output(cmd)
+        # pgrep returns a list of pids matching the search argument
+        pid.split('\n')[0]
+        return int(pid)
+    except subprocess.CalledProcessError:
+        return -1
+
+
+def private_IP(addr):
+    """Check whether an IP address can be routed.
+    :param addr: The IP address as a string.
+
+    :returns: True if the address is private (can't be routed). False
+        otherwise.
+
+    """
+    if addr.startswith('10.'):  # 10.0.0.0 - 10.255.255.255
+        return True
+    elif addr.startswith('192.168'):  # 192.168.0.0 - 192.168.255.255
+        return True
+    elif addr.startswith('172.'):  # 172.16.0.0 - 172.31.255.255
+        a, b, c, d = addr.split('.')
+        if 16 <= int(b) <= 31:
+            return True
+    return False
+
+
+def needs_tunnel():
+    """Check to see if a tunnel is required and up.
+
+    When required, it depends on juju's Swift bucket existing, and the IP it
+    refers to is pointed at by a sshuttle instance.
+    """
+    log.info("Checking to see if a tunnel needs to be setup.")
+
+    instance = get_bootstrap_nova_instance()
+    ip_addr = get_bootstrap_ip(instance, 1)
+    if private_IP(ip_addr):
+        sshuttle_pid = _get_pid_of_sshuttle(ip_addr)
+        return sshuttle_pid < 0
+    else:
+        return False
+
+
 def check_bootstrapped():
     try:
         # XXX: Added a print to give user feedback in case sshuttle isn't
@@ -116,6 +374,20 @@
             sys.exit(1)
 
 
+def populate_all_templates():
+    deployer_dir = os.path.join(HERE, '..', 'juju-deployer')
+    temp = tempfile.mkdtemp()
+    atexit.register(lambda: shutil.rmtree(temp))
+    # A place to put our generated .yaml files.
+    working_dir = os.path.join(temp, 'deployer')
+    # Copy all existing .yaml files, they don't need to be populated but
+    # juju-deployer expect them in 'working_dir'.
+    shutil.copytree(deployer_dir, working_dir)
+    # Create .yaml files from .yaml.tmpl files.
+    populate_templates(deployer_dir, working_dir)
+    return working_dir
+
+
 def populate_templates(deployer_dir, working_dir):
     # The command to populate the template.
     cmd_base = ['cheetah', 'fill', '--env', '-p']
@@ -123,8 +395,12 @@
     for tmpl in os.listdir(deployer_dir):
         path = os.path.join(deployer_dir, tmpl)
         if os.path.isdir(path):
+            # When called with a directory, recurse and process all templates
+            # found there.
             src = os.path.join(deployer_dir, tmpl)
             dst = os.path.join(working_dir, tmpl)
+            if not os.path.isdir(dst):
+                os.mkdir(dst)
             populate_templates(src, dst)
             continue
         if not tmpl.endswith('.yaml.tmpl'):
@@ -178,6 +454,31 @@
         cmd.run_argv_aliases(args)
 
 
+def install_charmworld(where=None):
+    """Recent versions of amulet need charmworldlib. Install it."""
+    if where is None:
+        where = os.path.join(HERE, '..', 'branches')
+    dest = os.path.abspath(os.path.join(where, 'charmworldlib'))
+    get_or_pull_branch('lp:charmworldlib', dest)
+    prepend_path('PYTHONPATH', dest)
+    # FIXME: This is a symptom that our setup story needs to receive some more
+    # love -- vila 2014-04-10
+    sys.path.insert(0, dest)
+    return dest
+
+
+def install_amulet(where=None):
+    if where is None:
+        where = os.path.join(HERE, '..', 'branches')
+    dest = os.path.abspath(os.path.join(where, 'amulet'))
+    get_or_pull_branch('lp:amulet', dest)
+    prepend_path('PYTHONPATH', dest)
+    # FIXME: This is a symptom that our setup story needs to receive some more
+    # love -- vila 2014-04-10
+    sys.path.insert(0, dest)
+    return dest
+
+
 def install_websocket_client(where=None):
     """Install python-websocket-client from bzr, as jujuclient needs it."""
     if where is None:
@@ -205,9 +506,10 @@
     if where is None:
         where = os.path.join(HERE, '..', 'branches')
     dest = os.path.abspath(os.path.join(where, 'juju-deployer'))
-    # For an obscure reason tag:0.3.1 is not in lp:juju-deployer
     get_or_pull_branch('lp:juju-deployer', dest, revision='revno:103')
     prepend_path('PYTHONPATH', dest)
+    # FIXME: This is a symptom that our setup story needs to receive some more
+    # love -- vila 2014-04-10
     sys.path.insert(0, dest)
     # FIXME: This duplicates setup.py and shouldn't be needed. This specific
     # piece (juju-deployer) is a weird beast, it installs from source (not pip
@@ -415,45 +717,74 @@
     return parser.parse_args(args)
 
 
+def setup(for_tests=False, list_only=False):
+    """Setup the environment for a deployment.
+
+    This includes:
+    - installing the source dependencies,
+    - uploading our source payload to swift,
+    - populating the templates,
+    - starting the juju boootstrap node if needed, with a tunnel if needed.
+    """
+    if for_tests:
+        install_amulet()
+        install_charmworld()
+    install_websocket_client()
+    install_jujuclient()
+    install_deployer()
+
+    check_environment()
+    if for_tests:
+        if not list_only:
+            upload_payload_to_swift()
+        working_dir = populate_all_templates()
+        if not list_only:
+            build_charms()
+        # FIXME: We use JUJU_DEPLOYER_DIR to tell deployers.DeployerTest where
+        # the populated templates are, there should be a better way than using
+        # an env var as a global variable -- vila 2014-04-11
+        os.environ['JUJU_DEPLOYER_DIR'] = working_dir
+    if for_tests:
+        if needs_bootstrap():
+            bootstrap()
+        if needs_tunnel():
+            wait_then_tunnel()
+
+
 def main():
     args = _get_args()
 
-    check_environment()
-
     # You can configure the system to use private PPAs only or by providing
     # the option --private-ppas-only or by setting the environment variable
     # CI_PRIVATE_PPAS_ONLY.
     # If provided, --private-ppas-only option overrides the environment
     # variable value.
-    check_and_set_ppa_privacy(args.private_ppas_only)
+    if args.private_ppas_only:
+        os.environ['CI_PRIVATE_PPAS_ONLY'] = '1'
 
     check_bootstrapped()
     check_dependencies()
 
-    install_websocket_client()
-    install_jujuclient()
-    install_deployer()
-
-    # A place to put our generated .yaml files.
-    temp_dir = tempfile.mkdtemp()
-    atexit.register(shutil.rmtree, temp_dir)
-    working_dir = os.path.join(temp_dir, 'deployer')
-    # Copy all existing .yaml files, they don't need to be populated but
-    # juju-deployer expect them in 'working_dir'.
-    shutil.copytree(deployer_dir, working_dir)
-
     if not args.branch_payload and not args.reuse_payload:
+        # We set some env variables below so this should happen before
+        # populating the templates
         upload_payload_to_swift()
 
-    # Create .yaml files from .yaml.tmpl files.
-    populate_templates(deployer_dir, working_dir)
+    setup()
+    priv = os.environ['CI_PRIVATE_PPAS_ONLY']
+    if priv == '0':
+        status = 'DISABLED'
+    else:
+        status = 'ENABLED'
+    print('Private PPAs only is {} (CI_PRIVATE_PPAS_ONLY: {})'.format(status,
+                                                                      priv))
+    working_dir = populate_all_templates()
+    build_charms()
 
     # Generate an argument list of -c FILE for each .yaml file in our working
     # directory.
     i = generate_argument_list(working_dir, args.specific)
-    generated = list(chain.from_iterable(i))
-
-    build_charms()
+    generated = list(itertools.chain.from_iterable(i))
 
     if args.upgrade:
         for x in generated:

=== modified file 'juju-deployer/test_deploy.py'
--- juju-deployer/test_deploy.py	2014-03-27 13:40:30 +0000
+++ juju-deployer/test_deploy.py	2014-04-15 12:18:41 +0000
@@ -1,14 +1,36 @@
 #!/usr/bin/env python
-
+# Ubuntu CI Engine
+# Copyright 2014 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from cStringIO import StringIO
 import mock
-import unittest
 import os
 import sys
 import tempfile
 import shutil
 import subprocess
-
-
+import unittest
+
+from novaclient import exceptions
+import swiftclient
+
+from bzrlib import (
+    branchbuilder,
+    errors,
+)
+from bzrlib.transport import memory
 from ucitests import fixtures
 
 
@@ -16,13 +38,24 @@
 import deploy
 
 
-class TestDeploy(unittest.TestCase):
+class TestCheckEnvironment(unittest.TestCase):
+
+    def setUp(self):
+        super(TestCheckEnvironment, self).setUp()
+        fixtures.set_uniq_cwd(self)
 
     def test_check_environment(self):
-        # If we haven't sourced a novarc, this program should bail out.
-        with mock.patch.dict('os.environ', {'OS_USERNAME': ''}):
-            with self.assertRaises(SystemExit):
-                deploy.check_environment()
+        # If we haven't sourced a valid novarc, this program should bail out.
+        fixtures.isolate_from_env(self, dict(OS_USERNAME=''))
+        with self.assertRaises(SystemExit):
+            deploy.check_environment()
+
+
+class TestCheckDependencies(unittest.TestCase):
+
+    def setUp(self):
+        super(TestCheckDependencies, self).setUp()
+        fixtures.set_uniq_cwd(self)
 
         # check that some of our values can be defaulted correctly:
         env_required = {
@@ -33,20 +66,82 @@
             'CI_LAUNCHPAD_USER': 'lp-user',
             'CI_LAUNCHPAD_PPA_OWNER': 'lp-ppa-owner',
         }
-        with mock.patch.dict('os.environ', env_required):
-            deploy.check_environment()
-            for key, val in env_required.iteritems():
-                self.assertEqual(val, os.environ[key])
-            # defaulted by code
-            self.assertEqual(os.environ['CI_CODE_SOURCE'], 'branch')
-            self.assertEqual(os.environ['CI_PRIVATE_PPAS_ONLY'], '0')
+        fixtures.isolate_from_env(self, env_required)
+        deploy.check_environment()
+        for key, val in env_required.iteritems():
+            self.assertEqual(val, os.environ[key])
+        # defaulted by code
+        self.assertEqual(os.environ['CI_CODE_SOURCE'], 'branch')
+        self.assertEqual(os.environ['CI_PRIVATE_PPAS_ONLY'], '0')
 
     def test_check_dependencies(self):
         # If we cannot find any of our dependencies, this program should bail
         # out.
-        with mock.patch.dict('os.environ', {'PATH': ''}):
-            with self.assertRaises(SystemExit):
-                deploy.check_dependencies()
+        fixtures.isolate_from_env(self, dict(PATH=''))
+        with self.assertRaises(SystemExit):
+            deploy.check_dependencies()
+
+
+class TestGetOrPullBranch(unittest.TestCase):
+
+    def setUp(self):
+        super(TestGetOrPullBranch, self).setUp()
+        fixtures.set_uniq_cwd(self)
+        fixtures.override_env(self, 'BZR_EMAIL', '[e-mail]')
+        builder = branchbuilder.BranchBuilder(
+            memory.MemoryTransport("memory:///"))
+        builder.start_series()
+        builder.build_snapshot(
+            'rev-id', None,
+            [('add', ('', 'root-id', 'directory', '')),
+             ('add', ('filename', 'f-id', 'file', 'content\n'))])
+        builder.build_snapshot('rev2-id', ['rev-id'],
+                               [('modify', ('f-id', 'new-content\n'))])
+        builder.finish_series()
+        # The tested API want local paths, let's create the branch on disk
+        tree = builder.get_branch().create_checkout('the_branch')
+        self.branch = tree.branch
+        # Capture output
+        self.out = StringIO()
+        self.err = StringIO()
+        # silence progress report
+        fixtures.override_env(self, 'BZR_PROGRESS_BAR', 'none')
+
+    def do_branch(self, *args):
+        deploy.get_or_pull_branch(*args, out=self.out, err=self.err)
+        self.assertEqual('', self.err.getvalue())
+
+    def test_get_branch(self):
+        self.do_branch('the_branch', 'here')
+        self.assertTrue(os.path.exists('here'))
+        self.assertEqual('', self.out.getvalue())
+
+    def test_pull_branch(self):
+        self.do_branch('the_branch', 'here')
+        self.do_branch('the_branch', 'here', 'revid:rev-id')
+        self.assertEqual('Now on revision 1.\n',   self.out.getvalue())
+        with open(os.path.join('here', 'filename')) as f:
+            self.assertEqual('content\n', f.read())
+
+    def assertErrorPath(self, dir_name, exc):
+        expected = os.path.join(self.uniq_dir, dir_name) + '/'
+        self.assertEqual(expected, exc.path)
+
+    def test_branch_bogus_src(self):
+        with self.assertRaises(errors.NotBranchError) as cm:
+            self.do_branch('I-dont-exist', 'here')
+        self.assertErrorPath('I-dont-exist', cm.exception)
+        self.assertEqual('', self.out.getvalue())
+
+    def test_pull_bogus_src(self):
+        deploy.get_or_pull_branch('the_branch', 'here')
+        with self.assertRaises(errors.NotBranchError) as cm:
+            self.do_branch('I-dont-exist', 'here')
+        self.assertErrorPath('I-dont-exist', cm.exception)
+        self.assertEqual('', self.out.getvalue())
+
+
+class TestBootStrap(unittest.TestCase):
 
     def test_check_not_bootstrapped(self):
         # If juju environment isn't bootstrapped, this program should bail
@@ -62,75 +157,95 @@
             # If juju status returns ok, this should just work.
             deploy.check_bootstrapped()
 
+
+class TestInstallDeployer(unittest.TestCase):
+
+    def setUp(self):
+        super(TestInstallDeployer, self).setUp()
+        fixtures.set_uniq_cwd(self)
+
     def test_install_deployer(self):
+        fixtures.isolate_from_env(self, dict(PYTHONPATH=''))
+        deployer_dir = deploy.install_deployer('.')
+        self.addCleanup(shutil.rmtree, deployer_dir)
+        self.assertTrue(os.path.exists(os.path.join(deployer_dir, '.bzr')))
+        self.assertEqual(deployer_dir, os.environ['PYTHONPATH'])
+        deployer_script = os.path.join(deployer_dir, 'juju-deployer')
+        self.assertTrue(os.access(deployer_script, os.X_OK))
+        self.assertTrue(os.path.exists(deployer_script))
+
+
+class TestGenerateArgumentList(unittest.TestCase):
+
+    def setUp(self):
+        super(TestGenerateArgumentList, self).setUp()
         fixtures.set_uniq_cwd(self)
-        with mock.patch.dict('os.environ', {'PYTHONPATH': ''}):
-            deployer_dir = deploy.install_deployer('.')
-            self.assertTrue(os.path.exists(os.path.join(deployer_dir, '.bzr')))
-            self.assertEqual(deployer_dir, os.environ['PYTHONPATH'])
-            deployer_script = os.path.join(deployer_dir, 'juju-deployer')
-            self.assertTrue(os.access(deployer_script, os.X_OK))
-            self.assertTrue(os.path.exists(deployer_script))
 
     def test_generate_empty_list(self):
-        working = tempfile.mkdtemp()
-        self.addCleanup(lambda: shutil.rmtree(working))
-
         # Calling without anything in the directory should produce an empty
         # list.
-        self.assertEqual(list(deploy.generate_argument_list(working)), [])
+        self.assertEqual([], list(deploy.generate_argument_list('.')))
 
     def test_generate_explicit_list(self):
-        working = tempfile.mkdtemp()
-        self.addCleanup(lambda: shutil.rmtree(working))
-
-        expected = os.path.join(working, 'foo.yaml')
+        expected = os.path.join('.', 'foo.yaml')
         with open(expected, 'w') as fp:
             fp.write('nothing')
         # Calling with a basic configuration should produce the full path to
         # that configuration in the form ('-c', path).
-        gen = deploy.generate_argument_list(working)
+        gen = deploy.generate_argument_list('.')
         self.assertEqual([('-c', expected)], list(gen))
 
+
+class TestPopulateTemplates(unittest.TestCase):
+
+    def setUp(self):
+        super(TestPopulateTemplates, self).setUp()
+        fixtures.set_uniq_cwd(self)
+        self.target = os.path.join('.', 'target')
+        os.mkdir(self.target)
+        self.source = os.path.join('.', 'source')
+        os.mkdir(self.source)
+
     def test_populate_templates_empty_list(self):
-        fake_deployer_dir = tempfile.mkdtemp()
-        self.addCleanup(lambda: shutil.rmtree(fake_deployer_dir))
-        working = tempfile.mkdtemp()
-        self.addCleanup(lambda: shutil.rmtree(working))
-
-        deploy.populate_templates(fake_deployer_dir, working)
+        deploy.populate_templates(self.source, self.target)
         # Nothing happened
-        self.assertEqual([], os.listdir(working))
-
-    @mock.patch.dict('os.environ', {'USER': 'foo'})
-    def test_populate_templates_explicit_list(self, *args):
-        fake_deployer_dir = tempfile.mkdtemp()
-        self.addCleanup(lambda: shutil.rmtree(fake_deployer_dir))
-        working = tempfile.mkdtemp()
-        self.addCleanup(lambda: shutil.rmtree(working))
-
-        test_files = ['config.yaml', 'configs/unit_config.yaml']
-        for f in test_files:
-            src = os.path.join(fake_deployer_dir, f)
-            dst = os.path.join(working, f)
-            if not os.path.exists(os.path.dirname(src)):
-                os.mkdir(os.path.dirname(src))
-            if not os.path.exists(os.path.dirname(dst)):
-                os.mkdir(os.path.dirname(dst))
-
-            with open(src + '.tmpl', 'w') as f:
-                f.write('me: ${USER}')
-
-        deploy.populate_templates(fake_deployer_dir, working)
-
-        for f in test_files:
-            f = os.path.join(working, f)
-            content = open(f).read()
-            self.assertEqual('me: {}'.format(os.environ['USER']), content)
+        self.assertEqual([], os.listdir(self.source))
+
+    def test_populate_templates_explicit_list(self):
+        fixtures.override_env(self, 'USER', 'foo')
+        stem = 'config.yaml'
+        with open(os.path.join(self.source, stem + '.tmpl'), 'w') as f:
+            f.write('me: ${USER}')
+        deploy.populate_templates(self.source, self.target)
+        with open(os.path.join(self.target, stem)) as f:
+            content = f.read()
+        self.assertEqual('me: {}'.format(os.environ['USER']), content)
+
+    def test_populate_recurse(self):
+        fixtures.override_env(self, 'USER', 'foo')
+        configs_dir = os.path.join(self.source, 'configs')
+        os.mkdir(configs_dir)
+        stem = 'config.yaml'
+        with open(os.path.join(configs_dir, stem + '.tmpl'), 'w') as f:
+            f.write('me: ${USER}')
+        # The only existing template is one level below self.source so
+        # populate_templates should recurse there.
+        deploy.populate_templates(self.source, self.target)
+        # Accordingly, the results should one level below self.target
+        with open(os.path.join(self.target, 'configs', stem)) as f:
+            content = f.read()
+        self.assertEqual('me: {}'.format(os.environ['USER']), content)
+
+
+class TestBuildCharms(unittest.TestCase):
+
+    def setUp(self):
+        super(TestBuildCharms, self).setUp()
+        fixtures.set_uniq_cwd(self)
 
     def test_build_charms(self):
         charmsdir = tempfile.mkdtemp()
-        self.addCleanup(lambda: shutil.rmtree(charmsdir))
+        self.addCleanup(shutil.rmtree, charmsdir)
         p = os.path.join(charmsdir, 'precise')
         os.mkdir(p)
         charm1 = os.path.join(p, 'charm1')
@@ -151,5 +266,64 @@
         self.assertItemsEqual(['charm1', 'charm2', 'webui'], os.listdir(p))
 
 
+class TestPrivateIP(unittest.TestCase):
+
+    def test_public(self):
+        self.assertFalse(deploy.private_IP('2.2.2.2'))
+
+    def test_class_A_private(self):
+        self.assertTrue(deploy.private_IP('10.0.4.5'))
+
+    def test_class_B_public(self):
+        self.assertFalse(deploy.private_IP('172.15.2.245'))
+
+    def test_class_B_private(self):
+        self.assertTrue(deploy.private_IP('172.24.12.36'))
+
+    def test_class_C_private(self):
+        self.assertTrue(deploy.private_IP('192.168.0.16'))
+
+
+class TestInstallAmulet(unittest.TestCase):
+
+    def setUp(self):
+        super(TestInstallAmulet, self).setUp()
+        fixtures.set_uniq_cwd(self)
+
+    def test_install_amulet(self):
+        fixtures.override_env(self, 'PYTHONPATH', '')
+        deploy.install_amulet('.')
+        self.assertTrue(os.path.exists(os.path.join('amulet', '.bzr')))
+        self.assertEqual(os.path.join(self.uniq_dir, 'amulet'),
+                         os.environ['PYTHONPATH'])
+
+
+class TestNeedsBotstrap(unittest.TestCase):
+
+    @mock.patch('deploy._get_control_bucket',
+                return_value='juju-d911b43e427c204cfcfd8c7276c4b2aa')
+    def test_needs_bootstrap(self, mock_get_control_bucket):
+        provider_state = '{ state-instances: ["abcdefg"] }'
+        with mock.patch('deploy._get_from_swift', return_value=provider_state):
+            # Bootstrap needed - the bucket exists but the instance does not.
+            e = exceptions.NotFound('Four oh four')
+            with mock.patch('deploy.get_bootstrap_ip', side_effect=e):
+                with mock.patch('deploy._delete_control_bucket') as dcb:
+                    self.assertTrue(deploy.needs_bootstrap())
+                    dcb.assert_called_with()
+
+            # Bootstrap not needed.
+            ip_addr = '127.0.0.1'
+            with mock.patch('deploy.get_bootstrap_ip', return_value=ip_addr):
+                self.assertFalse(deploy.needs_bootstrap())
+
+        # Bootstrap needed - the bucket does not exist.
+        e = swiftclient.exceptions.ClientException('not found')
+        with mock.patch('deploy._get_from_swift', side_effect=e):
+            with mock.patch('deploy._delete_control_bucket') as dcb:
+                self.assertTrue(deploy.needs_bootstrap())
+                dcb.assert_called_with()
+
+
 if __name__ == '__main__':
     unittest.main()

=== renamed file 'tests/test_run.py' => 'juju-deployer/test_run.py'
--- tests/test_run.py	2014-04-09 14:54:06 +0000
+++ juju-deployer/test_run.py	2014-04-15 12:18:41 +0000
@@ -1,26 +1,42 @@
 #!/usr/bin/env python
-
-import unittest
+# Ubuntu CI Engine
+# Copyright 2014 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import logging
 import mock
-import run
-import yaml
-import tempfile
-import shutil
 import os
-import logging
 import stat
-import swiftclient
-from novaclient.exceptions import NotFound
+import sys
+import tempfile
+import unittest
+import yaml
 
 
 from ucitests import fixtures
 
-
-# Disable all (almost) log output from the tests and the code they run
+# FIXME: Disable logging while running tests, this will be revisited once we
+# have a proper way to capture log in tests. -- vila 2014-02-12
 logging.disable(logging.CRITICAL)
 
-
-class TestRunTests(unittest.TestCase):
+mydir = os.path.dirname(__file__)
+sys.path.append(os.path.abspath(os.path.join( mydir, '..', 'tests')))
+import run
+
+
+class TestDestroyServices(unittest.TestCase):
+
     def _destroy_services(self, fake_status, destroy_machine, destroy_service):
         fake_status = yaml.safe_dump(fake_status)
 
@@ -43,60 +59,47 @@
         fake_status = {'services': {}, 'machines': {}}
         self._destroy_services(fake_status, destroy_machine, destroy_service)
 
-    def test_get_tests_no_tests(self):
-        temp_dir = tempfile.mkdtemp()
-        self.addCleanup(lambda: shutil.rmtree(temp_dir))
-        self.assertEqual(run.get_tests(temp_dir), [])
+
+class TestGetTests(unittest.TestCase):
+    """Minimally test get_tests until run-tests can replace ./tests/run.py.
+
+    We test against the real source but it's ok as we're read-only.
+    """
+
+    def setUp(self):
+        super(TestGetTests, self).setUp()
+        fixtures.set_uniq_cwd(self)
+
+    def create_file(self, name, content=''):
+        # We only care about files existing
+        with open(name, 'w') as f:
+            f.write(content)
+
+    def test_no_tests(self):
+        self.assertEqual([], run.get_tests('.'))
+
+    def test_matches_from_env(self):
+        fixtures.isolate_from_env(self, dict(TESTS='foo'))
+        self.create_file('test_foo.py')
+        self.assertEqual(['./test_foo.py'], run.get_tests('.'))
+
+    def test_filtered_from_env(self):
+        fixtures.isolate_from_env(self, dict(TESTS='foo'))
+        self.create_file('test_foo.py')
+        self.create_file('test_bar.py')
+        self.assertEqual(['./test_foo.py'], run.get_tests('.'))
 
     def test_get_tests(self):
-        temp_dir = tempfile.mkdtemp()
-        self.addCleanup(lambda: shutil.rmtree(temp_dir))
-        os.mkdir(os.path.join(temp_dir, 'foo'))
-        os.mkdir(os.path.join(temp_dir, 'bar'))
-        expect = [os.path.join(temp_dir, x, 'test.py') for x in ('foo', 'bar')]
-
-        for x in expect:
-            with open(x, 'w') as fp:
-                fp.write('#!/usr/bin/python\nprint "hello"\n')
-        with open(os.path.join(temp_dir, 'README'), 'w') as fp:
-            fp.write('Empty.')
-        os.mkdir(os.path.join(temp_dir, 'baz'))
-        with open(os.path.join(temp_dir, 'baz', 'README'), 'w') as fp:
-            fp.write('Empty.')
-
-        self.assertEqual(sorted(run.get_tests(temp_dir)), sorted(expect))
-
-    def test_install_amulet(self):
-        fixtures.set_uniq_cwd(self)
-        with mock.patch.dict('os.environ', {'PYTHONPATH': ''}):
-            amulet_dir = run.install_amulet('.')
-            self.addCleanup(shutil.rmtree, amulet_dir)
-            self.assertTrue(os.path.exists(os.path.join(amulet_dir, '.bzr')))
-            self.assertEqual(amulet_dir, os.environ['PYTHONPATH'])
-
-    @mock.patch('run._get_control_bucket',
-                return_value='juju-d911b43e427c204cfcfd8c7276c4b2aa')
-    def test_needs_bootstrap(self, mock_get_control_bucket):
-        provider_state = '{ state-instances: ["abcdefg"] }'
-        with mock.patch('run._get_from_swift', return_value=provider_state):
-            # Bootstrap needed - the bucket exists but the instance does not.
-            e = NotFound('Four oh four')
-            with mock.patch('run.get_bootstrap_ip', side_effect=e):
-                with mock.patch('run._delete_control_bucket') as dcb:
-                    self.assertTrue(run.needs_bootstrap())
-                    dcb.assert_called_with()
-
-            # Bootstrap not needed.
-            ip_addr = '127.0.0.1'
-            with mock.patch('run.get_bootstrap_ip', return_value=ip_addr):
-                self.assertFalse(run.needs_bootstrap())
-
-        # Bootstrap needed - the bucket does not exist.
-        e = swiftclient.exceptions.ClientException('not found')
-        with mock.patch('run._get_from_swift', side_effect=e):
-            with mock.patch('run._delete_control_bucket') as dcb:
-                self.assertTrue(run.needs_bootstrap())
-                dcb.assert_called_with()
+        expected = [os.path.join('.', 'test_{}.py'.format(x))
+                    for x in ('foo', 'bar')]
+
+        for x in expected:
+            self.create_file(x, '#!/usr/bin/python\nprint "hello"\n')
+        self.create_file('README')
+        self.assertEqual(sorted(expected), sorted(run.get_tests('.')))
+
+
+class TestRunTests(unittest.TestCase):
 
     def test_check_sudoers(self):
         with tempfile.NamedTemporaryFile() as fp:
@@ -139,24 +142,5 @@
         self.assertEqual(result, ['ci-juju-gui/0'])
 
 
-class TestPrivateIP(unittest.TestCase):
-
-    def test_public(self):
-        self.assertFalse(run.private_IP('2.2.2.2'))
-
-    def test_class_A_private(self):
-        self.assertTrue(run.private_IP('10.0.4.5'))
-
-    def test_class_B_public(self):
-        self.assertFalse(run.private_IP('172.15.2.245'))
-
-    def test_class_B_private(self):
-        self.assertTrue(run.private_IP('172.24.12.36'))
-
-    def test_class_C_private(self):
-        self.assertTrue(run.private_IP('192.168.0.16'))
-
-
-
 if __name__ == '__main__':
     unittest.main()

=== modified file 'juju-deployer/test_update.py'
--- juju-deployer/test_update.py	2014-02-19 14:48:22 +0000
+++ juju-deployer/test_update.py	2014-04-15 12:18:41 +0000
@@ -60,28 +60,26 @@
 class TestGetDeployerConfigs(unittest.TestCase):
 
     def setUp(self):
-        self.temp_dir = tempfile.mkdtemp()
-        self.addCleanup(lambda: shutil.rmtree(self.temp_dir))
+        super(TestGetDeployerConfigs, self).setUp()
+        fixtures.set_uniq_cwd(self)
 
     def test_get_deployer_no_configs(self):
-        configs = list(update.get_deployer_configs(self.temp_dir))
+        configs = list(update.get_deployer_configs('.'))
         self.assertEqual(configs, [])
 
     def test_get_deployer_two_configs(self):
         for x in ('foo.yaml', 'bar.yaml.tmpl'):
-            with open(os.path.join(self.temp_dir, x), 'w') as fp:
+            with open(x, 'w') as fp:
                 fp.write('nothing: here')
-        foo = os.path.join(self.temp_dir, 'foo.yaml')
-        bar = os.path.join(self.temp_dir, 'bar.yaml.tmpl')
-        self.assertEqual(sorted([foo, bar]),
-                         sorted(update.get_deployer_configs(self.temp_dir)))
+        self.assertEqual(sorted(['./bar.yaml.tmpl', './foo.yaml']),
+                         sorted(update.get_deployer_configs('.')))
 
 
 class TestUpdate(unittest.TestCase):
 
     def setUp(self):
-        self.temp_dir = tempfile.mkdtemp()
-        self.addCleanup(lambda: shutil.rmtree(self.temp_dir))
+        super(TestUpdate, self).setUp()
+        fixtures.set_uniq_cwd(self)
         fixtures.override_env(self, 'BZR_EMAIL', '[e-mail]')
         # We need a real branch with at least 2 revisions (so we can check out
         # of date references)
@@ -97,8 +95,7 @@
         builder.finish_series()
         # get_branches_and_revnos won't be able to use a in-memory branch,
         # let's create one on disk.
-        dest = os.path.join(self.temp_dir, 'the_branch')
-        tree = builder.get_branch().create_checkout(dest)
+        tree = builder.get_branch().create_checkout('the_branch')
         self.branch = tree.branch
 
     def get_branch_url(self):
@@ -120,14 +117,14 @@
 
     def test_ensure_all_branches_are_pinned(self):
         yaml_string = self.get_service_yaml(1)
-        path = os.path.join(self.temp_dir, 'foo.yaml')
+        path = 'foo.yaml'
         with open(path, 'w') as fp:
             fp.write(yaml.safe_dump(yaml_string))
         self.assertTrue(update.ensure_all_branches_are_pinned([path]))
 
     def test_ensure_some_branches_are_not_pinned(self):
         yaml_string = self.get_service_yaml()
-        path = os.path.join(self.temp_dir, 'foo.yaml')
+        path = 'foo.yaml'
         with open(path, 'w') as fp:
             fp.write(yaml.safe_dump(yaml_string))
         self.assertFalse(update.ensure_all_branches_are_pinned([path]))
@@ -136,7 +133,7 @@
         branch_url = self.get_branch_url()
         current = self.branch.revno()
         y = self.get_service_yaml(current - 1)
-        path = os.path.join(self.temp_dir, 'foo.yaml')
+        path = 'foo.yaml'
         with open(path, 'w') as fp:
             fp.write(yaml.safe_dump(y))
 
@@ -151,7 +148,7 @@
     def test_set_branches_and_revnos_on_naked_branches(self):
         branch_url = self.get_branch_url()
         y = self.get_service_yaml()
-        path = os.path.join(self.temp_dir, 'foo.yaml')
+        path = 'foo.yaml'
         with open(path, 'w') as fp:
             fp.write(yaml.safe_dump(y))
         update.set_branches_and_revnos([path], {branch_url: 1})
@@ -164,7 +161,7 @@
     def test_set_branches_and_revnos_on_pinned_branches(self):
         branch_url = self.get_branch_url()
         y = self.get_service_yaml(1)
-        path = os.path.join(self.temp_dir, 'foo.yaml')
+        path = 'foo.yaml'
         with open(path, 'w') as fp:
             fp.write(yaml.safe_dump(y))
         update.set_branches_and_revnos([path], {branch_url: 2})

=== modified file 'run-tests'
--- run-tests	2014-03-27 13:40:30 +0000
+++ run-tests	2014-04-15 12:18:41 +0000
@@ -22,6 +22,11 @@
 import sys
 
 
+HERE = os.path.abspath(os.path.dirname(__file__))
+
+# deploy.py isn't in our pythonpath when run as a script
+sys.path.append(os.path.join(HERE, 'juju-deployer'))
+import deploy
 from ucitests import (
     filters,
     loaders,
@@ -243,8 +248,46 @@
         return failures
 
 
-def run_regular_components(suite, out, result, options):
-    """Run tests for all the regular components.
+def load_orphan_tests(include_regexps, exclude_regexps=None):
+    """Load tests matching inclusive and exclusive regexps.
+
+    :param include_regexps: A list of regexps describing the tests to include.
+
+    :param exclude_regexps: A list of regexps describing the tests to exclude.
+
+    :return: The test suite for all collected tests.
+    """
+    components = ['juju-deployer']
+    loader = loaders.Loader()
+    suite = loader.suiteClass()
+    for c in components:
+        suite.addTests(load_component_tests(loader, c))
+    suite = filters.include_regexps(include_regexps, suite)
+    suite = filters.exclude_regexps(exclude_regexps, suite)
+    return suite
+
+
+def load_integration_tests(include_regexps, exclude_regexps=None):
+    """Load tests matching inclusive and exclusive regexps.
+
+    :param include_regexps: A list of regexps describing the tests to include.
+
+    :param exclude_regexps: A list of regexps describing the tests to exclude.
+
+    :return: The test suite for all collected tests.
+    """
+    components = ['tests']
+    loader = loaders.Loader()
+    suite = loader.suiteClass()
+    for c in components:
+        suite.addTests(load_component_tests(loader, c))
+    suite = filters.include_regexps(include_regexps, suite)
+    suite = filters.exclude_regexps(exclude_regexps, suite)
+    return suite
+
+
+def run_regular_tests(suite, out, result, options):
+    """Run the given test suite under some isolation.
 
     :param suite: The test suite to run.
 
@@ -289,16 +332,31 @@
     suite = load_regular_component_tests(options.include_regexps,
                                          options.exclude_regexps)
     # Regular components
-    ret = run_regular_components(suite, stdout, result, options)
+    ret = run_regular_tests(suite, stdout, result, options)
+    # orphan tests that don't have a proper home yet
+    suite = load_orphan_tests(options.include_regexps,
+                              options.exclude_regexps)
+    orphan_rc = run_regular_tests(suite, stdout, result, options)
+    ret = orphan_rc or ret
     # Django-based components
     with IsolatedImportAndLogging():
-        ppa =  run_django_tests('ppa-assigner', 'ppa_assigner', True,
-                                stdout, result, options)
-        ret = ppa or ret
+        ppa_rc =  run_django_tests('ppa-assigner', 'ppa_assigner', True,
+                                   stdout, result, options)
+        ret = ppa_rc or ret
     with IsolatedImportAndLogging():
-        ts = run_django_tests('ticket_system', 'ticket_system', True,
-                              stdout, result, options)
-        ret = ts or ret
+        ts_rc = run_django_tests('ticket_system', 'ticket_system', True,
+                                 stdout, result, options)
+        ret = ts_rc or ret
+    # integration tests
+    with IsolatedImportAndLogging(), SysPath(['ci-utils']):
+        # FIXME: We need some more setup here that tests themselves should do
+        # but they aren't ripe for that yet -- vila 2014-04-11
+        deploy.setup(for_tests=True, list_only=options.list_only)
+        suite = load_integration_tests(options.include_regexps,
+                                       options.exclude_regexps)
+        itg_rc = run_regular_tests(suite, stdout, result, options)
+        ret = itg_rc or ret
+
     if not options.list_only:
         # We did run tests, stop the run
         result.stopTestRun()

=== modified file 'tarmac.sh'
--- tarmac.sh	2014-03-26 15:35:02 +0000
+++ tarmac.sh	2014-04-15 12:18:41 +0000
@@ -58,7 +58,7 @@
 ./juju-deployer/update.py --assert-pinned
 ./juju-deployer/test_update.py
 ./juju-deployer/test_deploy.py
-./tests/test_run.py
+./juju-deployer/test_run.py
 '
 # Do not want to parse newlines as the internal field separator. It would break
 # any test with arguments.

=== modified file 'test_runner/tstrun/__init__.py'
--- test_runner/tstrun/__init__.py	2014-04-10 18:03:39 +0000
+++ test_runner/tstrun/__init__.py	2014-04-15 12:18:41 +0000
@@ -20,11 +20,14 @@
 HERE = os.path.abspath(os.path.dirname(__file__))
 
 
+HERE = os.path.abspath(os.path.dirname(__file__))
+
+
 # Creating the nova instance for the testbed and storing test results in the
 # data store require nova credentials.
 def get_auth_config(path=None):
     if path is None:
-        path = os.path.abspath(os.path.join(HERE, '../../unit_config'))
+        path = os.path.abspath(os.path.join(HERE, '..', '..', 'unit_config'))
     with open(path) as f:
         config = yaml.safe_load(f)
     return config

=== modified file 'test_runner/tstrun/tests/test_data_store.py'
--- test_runner/tstrun/tests/test_data_store.py	2014-02-17 10:13:29 +0000
+++ test_runner/tstrun/tests/test_data_store.py	2014-04-15 12:18:41 +0000
@@ -103,6 +103,10 @@
         # Get our content back
         self.assertEqual(content, self.store.get_file(rel_path))
 
+    def test_create_container_twice(self):
+        # swift is perfectly happy to create the same container twice.
+        self.store._create_container(self.store.container)
+
 
 if __name__ == '__main__':
     unittest.main()

=== removed directory 'tests/data_store'
=== modified file 'tests/deployers.py'
--- tests/deployers.py	2014-03-10 22:25:00 +0000
+++ tests/deployers.py	2014-04-15 12:18:41 +0000
@@ -15,16 +15,15 @@
 
 import base64
 import os
-import shutil
 import subprocess
 import unittest
 
+
+import amulet
 import yaml
 
-import amulet
 
-defdir = os.path.join(os.path.dirname(__file__), '../juju-deployer')
-DEPLOYER_DIR = os.environ.get('JUJU_DEPLOYER_DIR', defdir)
+HERE = os.path.dirname(__file__)
 
 
 class AmuletDeployment(object):
@@ -57,6 +56,9 @@
                             val = base64.b64encode(f.read())
                         service['options'][key] = val
 
+    def tearDown(self):
+        self.deployment.cleanup()
+
 
 class DeployerTest(unittest.TestCase):
     '''Base class for building juju deployer based tests.'''
@@ -66,19 +68,17 @@
 
     def setUp(self):
         super(DeployerTest, self).setUp()
-        self.deployer = AmuletDeployment(self.deployer_cfg)
-        self.addCleanup(self._clean_amulet)
-        if self.timeout:
-            self.deployer.setUp(self.timeout)
+        self.deployer = AmuletDeployment(
+            os.path.join(os.environ['JUJU_DEPLOYER_DIR'], self.deployer_cfg))
+        self.addCleanup(self.deployer.tearDown)
+        self.deployer.setUp(self.timeout)
         self.status = amulet.waiter.status(self.deployer.deployment.juju_env)
 
-    def _clean_amulet(self):
-        if self.deployer.deployment.deployer_dir:
-            shutil.rmtree(self.deployer.deployment.deployer_dir)
-
     def get_ip_and_port(self, service, unit=0):
         units = self.status['services'][service]['units']
         ip = units['%s/%d' % (service, unit)]['public-address']
+        # FIXME: Can fail with KeyError: 'open-ports'. Need to refresh status ?
+        # -- vila 2014-04-10
         port, _ = units['%s/%d' % (service, unit)]['open-ports'][0].split('/')
         return ip, int(port)
 

=== removed directory 'tests/image_builder'
=== removed directory 'tests/integration'
=== removed directory 'tests/juju_gui'
=== removed directory 'tests/ppa_assigner'
=== modified file 'tests/run.py'
--- tests/run.py	2014-04-10 14:55:27 +0000
+++ tests/run.py	2014-04-15 12:18:41 +0000
@@ -1,23 +1,28 @@
 #!/usr/bin/python
+# Ubuntu CI Engine
+# Copyright 2014 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import atexit
 import logging
 import os
-import shutil
-import signal
 import subprocess
 import sys
-import tempfile
 import time
 import yaml
 
-import swiftclient
-
-from novaclient.v1_1 import client
-from novaclient.exceptions import NotFound
-
-SSH_OPTS = ['-o', 'UserKnownHostsFile=/dev/null',
-            '-o', 'StrictHostKeyChecking=no']
+
 SUDOERS_FILE = '/etc/sudoers.d/sshuttle'
 SUDOERS = 'ALL = (root) NOPASSWD: /usr/bin/python /usr/lib/sshuttle/main.py *'
 
@@ -28,9 +33,8 @@
 
 HERE = os.path.abspath(os.path.dirname(__file__))
 
-
-# deploy.py probably isn't in our pythonpath
-sys.path.append(os.path.join(os.path.dirname(__file__), '../juju-deployer'))
+# deploy.py isn't in our pythonpath when run as a script
+sys.path.append(os.path.join(HERE, '../juju-deployer'))
 import deploy
 
 
@@ -86,6 +90,9 @@
     """Assert that there are no deployed services or nodes beyond the bootstrap
     node."""
 
+    # FIXME: This can block on ci-juju-gui in an error state (seen in real
+    # life). -- vila 2014-02-14
+
     # First ensure that anything dying has made the transition to the other
     # side.
     _wait_for_death()
@@ -117,33 +124,30 @@
         _destroy_service(service)
 
 
-def _find_test_files(test_dir, tests):
-    for test in tests:
-        actual_test = os.path.join(test_dir, test, 'test.py')
-        if not os.path.exists(actual_test):
-            msg = "Skipping %s as %s does not exist." % (test, actual_test)
-            log.error(msg)
-        else:
-            yield actual_test
-
-
-def _find_dirs_in_dir(test_dir):
-    """Return an iterator of only the directories in test_dir."""
-
-    for x in os.listdir(test_dir):
-        if os.path.isdir(os.path.join(test_dir, x)):
-            yield x
-
-
 def get_tests(test_dir):
     """Returns a list of full relative paths to test files."""
-
-    tests = os.environ.get('TESTS', '')
-    if tests:
-        tests = tests.split(' ')
-    else:
-        tests = _find_dirs_in_dir(test_dir)
-    return list(_find_test_files(test_dir, tests))
+    tests = []
+    for name in os.listdir(test_dir):
+        if name.startswith('test_') and name.endswith('.py'):
+            tests.append(name)
+    only = os.environ.get('TESTS', '')
+    if only:
+        selected = []
+        only = only.split()
+        # TESTS specifies which components should be tested, for compatibility
+        # with the previous layout where a 'test.py' file was under a directory
+        # named like the component, we keep matching on that.
+        for one in only:
+            file_name = 'test_{}.py'.format(one)
+            if  file_name in tests:
+                selected.append(file_name)
+            else:
+                msg = "Skipping {} as {} does not exist in {}"
+                log.error(msg.format(one, file_name, test_dir))
+        tests = selected
+    # Prepend the test dir to all selected tests
+    tests = [os.path.join(test_dir, name) for name in tests]
+    return tests
 
 
 def find_failed_units():
@@ -166,285 +170,6 @@
     subprocess.check_call(cmd)
 
 
-def install_charmworld(where=None):
-    """Recent versions of amulet need charmworldlib. Install it."""
-    if where is None:
-        where = os.path.join(HERE, '..', 'branches')
-    dest = os.path.abspath(os.path.join(where, 'charmworldlib'))
-    deploy.get_or_pull_branch('lp:charmworldlib', dest)
-    deploy.prepend_path('PYTHONPATH', dest)
-    return dest
-
-
-def install_amulet(where=None):
-    if where is None:
-        where = os.path.join(HERE, '..', 'branches')
-    dest = os.path.abspath(os.path.join(where, 'amulet'))
-    deploy.get_or_pull_branch('lp:amulet', dest)
-    deploy.prepend_path('PYTHONPATH', dest)
-    return dest
-
-
-def remove_branch(branch_dir):
-    """Clean up the bzr branch if we didn't already do so as a result of
-    bzr failing."""
-
-    if os.path.exists(branch_dir):
-        shutil.rmtree(branch_dir)
-
-
-def needs_bootstrap():
-    """Check to see if there's a Swift bucket for juju. If not, we need to
-    boostrap. Check to see if the IP of the bootstrap node is owned by a
-    running Nova instance. If not, we need to bootstrap."""
-
-    log.info("Checking to see if a bootstrap is needed.")
-
-    def inconsistent_state(msg):
-        log.warning(msg)
-        _delete_control_bucket()
-
-    try:
-        instance = get_bootstrap_nova_instance()
-    except swiftclient.exceptions.ClientException:
-        instance = None
-    if instance is None:
-        # We don't have an instance. We need to bootstrap.
-        inconsistent_state('No bootstrap instance in the swift bucket')
-        return True
-
-    try:
-        get_bootstrap_ip(instance, 1)
-    except NotFound:
-        # Juju claims we have an instance that does not exist. We're in an
-        # inconsistent state. Delete the bucket and bootstrap.
-        inconsistent_state('No IP for the bootstrap instance')
-        return True
-
-    return False
-
-
-def _get_pid_of_sshuttle(ip_addr):
-    try:
-        cmd = ['pgrep', '-f', 'sshuttle.*%s' % ip_addr]
-        pid = subprocess.check_output(cmd)
-        # pgrep returns a list of pids matching the search argument
-        pid.split('\n')[0]
-        return int(pid)
-    except subprocess.CalledProcessError:
-        return -1
-
-
-def needs_tunnel():
-    """Check to see if a tunnel is required and up.
-
-    When required, it depends on juju's Swift bucket existing, and the IP it
-    refers to is pointed at by a sshuttle instance.
-    """
-    log.info("Checking to see if a tunnel needs to be setup.")
-
-    instance = get_bootstrap_nova_instance()
-    ip_addr = get_bootstrap_ip(instance, 1)
-    if private_IP(ip_addr):
-        sshuttle_pid = _get_pid_of_sshuttle(ip_addr)
-        return sshuttle_pid < 0
-    else:
-        return False
-
-
-def _connect_to_swift():
-    opts = {'tenant_name': os.environ['OS_TENANT_NAME'],
-            'region_name': os.environ['OS_REGION_NAME'],
-            }
-    conn = swiftclient.client.Connection(
-        os.environ['OS_AUTH_URL'],
-        os.environ['OS_USERNAME'],
-        os.environ['OS_PASSWORD'],
-        os_options=opts,
-        auth_version='2.0')
-    return conn
-
-
-def _get_from_swift(bucket, path):
-    """Download the file path from swift at bucket."""
-
-    conn = _connect_to_swift()
-    # get_object return a tuple of (metadata, data)
-    return conn.get_object(bucket, path)[1]
-
-
-def _get_control_bucket():
-    """Get the name of the bucket in Swift that juju uses for state
-    information."""
-
-    default_home = os.path.join(os.path.expanduser('~'), '.juju')
-    juju_home = os.environ.get('JUJU_HOME') or default_home
-
-    with open(os.path.join(juju_home, 'environments.yaml')) as fp:
-        envs = yaml.safe_load(fp.read())
-
-    juju_env = os.environ.get('JUJU_ENV')
-    if juju_env is None:
-        juju_env = envs.get('default')
-    if not juju_env:
-        log.error('No juju environment specified.')
-        sys.exit(1)
-
-    return envs['environments'][juju_env]['control-bucket']
-
-
-def _delete_control_bucket():
-    """Remove the juju control bucket and all of its files."""
-
-    bucket = _get_control_bucket()
-    conn = _connect_to_swift()
-    try:
-        files = [x['name'] for x in conn.get_container(bucket)[1]]
-        for f in files:
-            conn.delete_object(bucket, f)
-        conn.delete_container(bucket)
-    except swiftclient.exceptions.ClientException:
-        # Assume the bucket is not present
-        pass
-
-
-def get_bootstrap_nova_instance():
-    """Get the Nova instance identifier for the juju bootstrap node.
-
-    Raises swiftclient.exceptions.ClientException if the instance cannot be
-    found."""
-
-    bucket = _get_control_bucket()
-
-    state = yaml.safe_load(_get_from_swift(bucket, 'provider-state'))
-    if not state:
-        # Cause unknown (yet) but this has been seen in real life. I can't
-        # reproduce anymore but I suspect 'juju bootstrap' failed mid-way so
-        # returning None and letting the caller delete the bucket is the Right
-        # Thing to do -- vila 2014-02-07
-        log.warning("Empty 'provider-state' in the juju bucket")
-        return None
-    return state['state-instances'][0]
-
-
-def private_IP(addr):
-    """Check whether an IP address can be routed.
-    :param addr: The IP address as a string.
-
-    :returns: True if the address is private (can't be routed). False
-        otherwise.
-
-    """
-    if addr.startswith('10.'):  # 10.0.0.0 - 10.255.255.255
-        return True
-    elif addr.startswith('192.168'):  # 192.168.0.0 - 192.168.255.255
-        return True
-    elif addr.startswith('172.'):  # 172.16.0.0 - 172.31.255.255
-        a, b, c, d = addr.split('.')
-        if 16 <= int(b) <= 31:
-            return True
-    return False
-
-
-def get_bootstrap_ip(instance, max_tries=120):
-    """Wait for the juju bootstrap node to boot and obtain an IP address.
-
-    This is a pre-requisite to setup a tunnel.
-
-    :raises: NotFound if the instance has not been spawned.
-
-    :raises: Exception if no IP can be found after max_tries*3 seconds.
-    """
-
-    log.info("Getting the bootstrap IP address.")
-
-    args = [os.environ['OS_USERNAME'], os.environ['OS_PASSWORD'],
-            os.environ['OS_TENANT_NAME'], os.environ['OS_AUTH_URL']]
-    kwargs = {'region_name': os.environ['OS_REGION_NAME'],
-              'service_type': 'compute'}
-    nt = client.Client(*args, **kwargs)
-
-    tries = 0
-    while tries < max_tries:
-        # Raises NotFound if the instance does not exist.
-        server = nt.servers.get(instance)
-        networks = server.networks.values()
-        if networks:
-            # We use a single network and don't care how it's named
-            net = networks[0]
-            if net:
-                # Either we have a single address or the last one has been
-                # added to make the instance reachable (floating or public).
-                net = net[-1]
-                # Don't sleep anymore. We have our IP, let's move on
-                return net
-
-        tries += 1
-        if tries != max_tries:
-            time.sleep(3)
-    raise Exception('Timed out waiting for bootstrap IP.')
-
-
-def destroy_tunnel():
-    if os.path.exists('sshuttle.pid'):
-        with open('sshuttle.pid') as fp:
-            pid = int(fp.read())
-            os.kill(pid, signal.SIGTERM)
-
-
-def tunnel(ip):
-    """Tunnel to the bootstrap node using sshuttle."""
-
-    log.info("Tunneling to the bootstrap node.")
-
-    host = 'ubuntu@%s' % ip
-    mask = '10.55.0.0/16'
-    cmd = ['sshuttle', '-D', '-r', host, mask]
-    opt = ['-e', 'ssh %s' % ' '.join(SSH_OPTS)]
-    subprocess.check_call(cmd + opt)
-    atexit.register(destroy_tunnel)
-
-
-def wait_for_node(ip):
-    """Wait for the bootstrap node to become available on ssh. We need to wait
-    for this state before attempting to set up a sshuttle."""
-
-    log.info("Waiting for the bootstrap node to respond to ssh.")
-
-    cmd = ['ssh'] + SSH_OPTS + ['ubuntu@%s' % ip, 'exit']
-    tries = 0
-    while tries < 120:
-        try:
-            subprocess.check_call(cmd)
-            return
-        except subprocess.CalledProcessError:
-            tries += 1
-            time.sleep(5)
-
-    raise Exception('Timed out waiting for bootstrap node.')
-
-
-def bootstrap():
-    """Bootstrap juju. Specify needed as bootstrap to just bootstrap, otherwise
-    tunnel to bootstrap and tunnel."""
-
-    args = ['juju', 'bootstrap', '--constraints=mem=1024M']
-    try:
-        subprocess.check_output(args)
-    except subprocess.CalledProcessError as e:
-        # FIXME: if check_output raise an exception, we don't get a proper
-        # output to check for the text below -- vila 2014-02-07
-        if 'environment is already bootstrapped' not in e.output:
-            raise
-
-
-def wait_then_tunnel():
-    instance = get_bootstrap_nova_instance()
-    ip = get_bootstrap_ip(instance)
-    wait_for_node(ip)
-    tunnel(ip)
-
-
 def check_sudoers(sudoers_path=SUDOERS_FILE):
     """Check to see if /etc/sudoers.d/sshuttle exists and is of the right
     form."""
@@ -492,23 +217,10 @@
     return rc == 0
 
 
-def _prepare_deployer():
-    deployer_dir = os.path.join(os.path.dirname(__file__), '../juju-deployer')
-    temp = tempfile.mkdtemp()
-    atexit.register(lambda: shutil.rmtree(temp))
-    working_dir = os.path.join(temp, 'deployer')
-    shutil.copytree(deployer_dir, working_dir)
-    deploy.upload_payload_to_swift()
-    deploy.populate_templates(deployer_dir, working_dir)
-    deploy.build_charms()
-    os.environ['JUJU_DEPLOYER_DIR'] = working_dir
-
-
 def main():
     check_nova_env()
 
     test_dir = os.path.dirname(sys.argv[0])
-    deploy.check_environment()
     tests = get_tests(test_dir)
     if not tests:
         # Running no tests is an error
@@ -524,32 +236,22 @@
         log.error(msg)
         sys.exit(1)
 
-    _prepare_deployer()
+    deploy.setup(for_tests=True)
 
-    install_amulet()
-    install_charmworld()
-    deploy.install_deployer()
-    deploy.install_jujuclient()
+    destroy_services()
+    assert_no_deployed_services()
+    atexit.register(destroy_services)
 
     # Now, insert the 'tests' dir in front of PYTHONPATH so we get our tests
     # without colliding with other 'tests' directories defined by other
     # packages (including our deps).
     deploy.prepend_path('PYTHONPATH', os.path.dirname(test_dir))
 
-    if needs_bootstrap():
-        bootstrap()
-    if needs_tunnel():
-        wait_then_tunnel()
-
-    destroy_services()
-    assert_no_deployed_services()
-    atexit.register(destroy_services)
-
     returncode = 0
     for test in tests:
-        log.error('========================================')
-        log.error('Testing %s' % test)
-        log.error('========================================')
+        log.info('========================================')
+        log.info('Testing %s' % test)
+        log.info('========================================')
         try:
             subprocess.check_call(['python', test])
         except subprocess.CalledProcessError as e:

=== renamed file 'tests/data_store/test.py' => 'tests/test_data_store.py'
--- tests/data_store/test.py	2014-02-17 10:13:29 +0000
+++ tests/test_data_store.py	2014-04-15 12:18:41 +0000
@@ -20,8 +20,7 @@
 import sys
 import yaml
 
-sys.path.append(os.path.join(
-    os.path.dirname(__file__), '..', '..', 'ci-utils'))
+sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'ci-utils'))
 
 from ci_utils import data_store
 

=== renamed file 'tests/image_builder/test.py' => 'tests/test_image_builder.py'
--- tests/image_builder/test.py	2014-04-02 11:56:54 +0000
+++ tests/test_image_builder.py	2014-04-15 12:18:41 +0000
@@ -1,18 +1,31 @@
 #!/usr/bin/env python
+# Ubuntu Continuous Integration Engine
+# Copyright 2014 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import json
-import os
 import unittest
 
 import httplib2
 
-from tests import deployers
+import deployers
 
 
 class TestTestRunner(deployers.DeployerTest):
     """Integration tests for image builder service run on a juju deployment"""
 
-    deployer_cfg = os.path.join(deployers.DEPLOYER_DIR, 'image-builder.yaml')
+    deployer_cfg = 'image-builder.yaml'
 
     def get_status(self):
         url = 'http://{}:{}/api/v1/status'.format(

=== renamed file 'tests/integration/test.py' => 'tests/test_integration.py'
--- tests/integration/test.py	2014-03-07 13:23:35 +0000
+++ tests/test_integration.py	2014-04-15 12:18:41 +0000
@@ -1,4 +1,18 @@
 #!/usr/bin/env python
+# Ubuntu Continuous Integration Engine
+# Copyright 2014 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
 import json
@@ -9,7 +23,10 @@
 
 import httplib2
 
-from tests.deployers import DEPLOYER_DIR, DeployerTest
+import deployers
+
+
+HERE = os.path.abspath(os.path.dirname(__file__))
 
 
 def create_virtual_env():
@@ -24,10 +41,9 @@
 
 def run_virtual_env(temp_dir, args, cwd=None):
     if args is 'create':
-        cwd = os.path.join(os.path.dirname(__file__), '../../cli')
-        changes = os.path.abspath(os.path.join(
-            os.path.dirname(__file__),
-            '../../cli/tests/data/foobar_0.1-1_source.changes'))
+        cwd = os.path.join(HERE, '../cli')
+        changes = os.path.abspath(os.path.join(HERE,
+            '../cli/tests/data/foobar_0.1-1_source.changes'))
         args = ('python ubuntu-ci create_ticket -t "New feature" -b 1234 -o '
                 '[e-mail] -d "This is a cool new feature" -a foo '
                 '-s %s' % changes)
@@ -39,18 +55,18 @@
         subprocess.check_call(args, shell=True)
 
 
-class IntegrationTest(DeployerTest):
+class IntegrationTest(deployers.DeployerTest):
     """Integration tests for ticket-system service run on a juju deployment"""
 
-    deployer_cfg = os.path.join(DEPLOYER_DIR, 'ticket-system.yaml')
+    deployer_cfg = 'ticket-system.yaml'
 
     def setUp(self):
         super(IntegrationTest, self).setUp()
         self.temp_dir = create_virtual_env()
-        p = os.path.join(os.path.dirname(__file__), '../../ci-utils/setup.py')
+        p = os.path.join(HERE, '../ci-utils/setup.py')
         args = '%s develop' % p
         run_virtual_env(self.temp_dir, args)
-        p = os.path.join(os.path.dirname(__file__), '../../cli/setup.py')
+        p = os.path.join(HERE, '../cli/setup.py')
         args = '%s develop' % p
         run_virtual_env(self.temp_dir, args)
         config_url = 'http://{}:{}'
@@ -94,5 +110,6 @@
         self.assertEqual(resp['status'], '200')
         self.assertIn('<a href="/" class="first ">Ticket List</a>', content)
 
+
 if __name__ == "__main__":
     unittest.main()

=== renamed file 'tests/juju_gui/test.py' => 'tests/test_juju_gui.py'
--- tests/juju_gui/test.py	2014-01-29 17:27:07 +0000
+++ tests/test_juju_gui.py	2014-04-15 12:18:41 +0000
@@ -1,17 +1,30 @@
 #!/usr/bin/env python
-
-import os
+# Ubuntu Continuous Integration Engine
+# Copyright 2014 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
 import unittest
 import urllib2
 
 
-from tests.deployers import DEPLOYER_DIR, DeployerTest
-
-
-class JujuGuiTest(DeployerTest):
+import deployers
+
+
+class JujuGuiTest(deployers.DeployerTest):
     """Integration tests for juju-gui service run on a juju deployment"""
 
-    deployer_cfg = os.path.join(DEPLOYER_DIR, 'juju-gui.yaml')
+    deployer_cfg = 'juju-gui.yaml'
 
     def get_unit(self):
         unit = self.status['services']['ci-juju-gui']['units'].values()[0]

=== renamed file 'tests/ppa_assigner/test.py' => 'tests/test_ppa_assigner.py'
--- tests/ppa_assigner/test.py	2014-03-14 10:37:28 +0000
+++ tests/test_ppa_assigner.py	2014-04-15 12:18:41 +0000
@@ -1,18 +1,32 @@
 #!/usr/bin/env python
-
+# Ubuntu Continuous Integration Engine
+# Copyright 2013, 2014 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import httplib2
 import json
 import os
 import unittest
 
-import httplib2
-
-from tests.deployers import DEPLOYER_DIR, DeployerTest
-
-
-class PPAAssignerTest(DeployerTest):
+
+import deployers
+
+
+class PPAAssignerTest(deployers.DeployerTest):
     """Integration tests for ppa-assigner service run on a juju deployment"""
 
-    deployer_cfg = os.path.join(DEPLOYER_DIR, 'ppa-assigner.yaml')
+    deployer_cfg = 'ppa-assigner.yaml'
 
     def get_server_status_and_content(self, url):
         final_url = url.format(*self.get_ip_and_port('ppa-django'))
@@ -134,5 +148,6 @@
             self.assertEqual('/api/v1/ppa/2/',
                              content['objects'][0]['resource_uri'])
 
+
 if __name__ == "__main__":
     unittest.main()

=== removed directory 'tests/test_runner'
=== renamed file 'tests/test_runner/test.py' => 'tests/test_test_runner.py'
--- tests/test_runner/test.py	2014-04-02 11:56:54 +0000
+++ tests/test_test_runner.py	2014-04-15 12:18:41 +0000
@@ -1,18 +1,31 @@
 #!/usr/bin/env python
+# Ubuntu Continuous Integration Engine
+# Copyright 2014 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import json
-import os
 import unittest
 
 import httplib2
 
-from tests import deployers
+import deployers
 
 
 class TestTestRunner(deployers.DeployerTest):
     """Integration tests for test runner service run on a juju deployment"""
 
-    deployer_cfg = os.path.join(deployers.DEPLOYER_DIR, 'test-runner.yaml')
+    deployer_cfg = 'test-runner.yaml'
 
     def post_test_image(self, ticket_id, image_id, package_list,
                         progress_trigger):

=== renamed file 'tests/ticket_system/test.py' => 'tests/test_ticket_system.py'
--- tests/ticket_system/test.py	2014-04-01 15:02:54 +0000
+++ tests/test_ticket_system.py	2014-04-15 12:18:41 +0000
@@ -1,18 +1,31 @@
 #!/usr/bin/env python
+# Ubuntu Continuous Integration Engine
+# Copyright 2014 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import json
-import os
 import unittest
 
 import httplib2
 
-from tests.deployers import DEPLOYER_DIR, DeployerTest
-
-
-class TicketSystemTest(DeployerTest):
+import deployers
+
+
+class TicketSystemTest(deployers.DeployerTest):
     """Integration tests for ticket-system service run on a juju deployment"""
 
-    deployer_cfg = os.path.join(DEPLOYER_DIR, 'ticket-system.yaml')
+    deployer_cfg = 'ticket-system.yaml'
 
     def get_server_status_and_content(self, url):
         final_url = url.format(*self.get_ip_and_port('ts-django'))

=== removed directory 'tests/ticket_system'
=== modified file 'ticket_system/people/tests.py'
--- ticket_system/people/tests.py	2014-02-08 10:43:30 +0000
+++ ticket_system/people/tests.py	2014-04-15 12:18:41 +0000
@@ -23,7 +23,7 @@
     if 'people' in settings.LOCAL_APPS:
         return std_tests
     else:
-        # if people is not in LOCAL_APPS the tests cannot pass so get rid of
+        # If 'people' is not in LOCAL_APPS the tests cannot pass so get rid of
         # them by returning an empty test suite.
         return loader.suiteClass()