Merge lp:~mthaddon/mojo/juju-intro-mojo-specs into lp:mojo/mojo-specs

[Tom Haddon]

Add a "juju-intro" service which uses the charms described on https://jujucharms.com/docs/stable/getting-started so that we can test these actually work in CI

[Tom Haddon (mthaddon)]

I'm going to test this in CI first before putting this MP up for review.

[Tom Haddon (mthaddon)]

This is now ready for review. Has run as follows: http://paste.ubuntu.com/12272171/

[lp:~mthaddon/mojo/juju-intro-mojo-specs]

70. By Tom Haddon on 2015-09-05

Just use the current promulgated series of each charm

71. By Tom Haddon on 2015-09-08

Add an e2e check for the site being up after initial install

[Paul Collins (pjdc)]

Approving, although I noticed:
 - running this spec with the local provider fails here due to mysql's default dataset-size of 80%; not sure how likely it is for someone to be using it, however
 - mojo-spec-helpers may need another refresh (or is itself stale); add-floating-ip was rewritten in 100% Python recently (see internal trunk)

review: Approve

=== added directory 'juju-intro'
=== added file 'juju-intro/README'
--- juju-intro/README	1970-01-01 00:00:00 +0000
+++ juju-intro/README	2015-09-08 08:23:55 +0000
@@ -0,0 +1,5 @@
+This spec contains the services that are included in the introduction to Juju
+as part of [1], and is intended to be used with CI to confirm that the charms
+we're pointing new users at for Juju are always in a working state.
+
+[1] https://jujucharms.com/docs/stable/getting-started

=== added symlink 'juju-intro/check-juju'
=== target is u'../mojo-spec-helpers/tests/check-juju'
=== added file 'juju-intro/collect'
--- juju-intro/collect	1970-01-01 00:00:00 +0000
+++ juju-intro/collect	2015-09-08 08:23:55 +0000
@@ -0,0 +1,5 @@
+wordpress   lp:charms/wordpress
+mysql       lp:charms/mysql
+
+# subordinates
+nrpe        lp:charms/nrpe-external-master

=== added file 'juju-intro/deploy'
--- juju-intro/deploy	1970-01-01 00:00:00 +0000
+++ juju-intro/deploy	2015-09-08 08:23:55 +0000
@@ -0,0 +1,14 @@
+wordpress:
+    series: {{ series }}
+    services:
+        wordpress:
+            charm: wordpress
+            expose: true
+        mysql:
+            charm: mysql
+        nrpe:
+            charm: nrpe
+    relations:
+        - ["wordpress", "mysql"]
+        - ["wordpress", "nrpe"]
+        - ["mysql", "nrpe"]

=== added file 'juju-intro/manifest'
--- juju-intro/manifest	1970-01-01 00:00:00 +0000
+++ juju-intro/manifest	2015-09-08 08:23:55 +0000
@@ -0,0 +1,10 @@
+collect
+deploy delay=0
+include config=manifest-verify
+
+## This isn't included in the main verify manifest because after we've installed
+## the service this won't work as expected so it's a one time thing for the
+## initial deployment
+
+# Verify the site is installed
+verify config=verify-installed

=== added file 'juju-intro/manifest-verify'
--- juju-intro/manifest-verify	1970-01-01 00:00:00 +0000
+++ juju-intro/manifest-verify	2015-09-08 08:23:55 +0000
@@ -0,0 +1,4 @@
+# Checking juju status
+verify config=check-juju
+# Running all naigos checks to confirm service is working as expected
+verify config=verify-nrpe

=== added file 'juju-intro/verify-installed'
--- juju-intro/verify-installed	1970-01-01 00:00:00 +0000
+++ juju-intro/verify-installed	2015-09-08 08:23:55 +0000
@@ -0,0 +1,10 @@
+#!/bin/bash
+
+set -e
+set -u
+
+# Check the service is actually up and we can get to the install page
+# This isn't included in the main verify manifest because after we've installed
+# the service this won't work as expected so it's a one time thing for the
+# initial deployment
+juju ssh wordpress/0 "/usr/lib/nagios/plugins/check_http -I 127.0.0.1 -H localhost -f follow -s '<title>WordPress &rsaquo; Installation</title>'"

=== added symlink 'juju-intro/verify-nrpe'
=== target is u'../mojo-spec-helpers/tests/verify-nrpe'
=== removed file 'mojo-how-to/devel/verify'
--- mojo-how-to/devel/verify	2015-01-26 16:23:38 +0000
+++ mojo-how-to/devel/verify	1970-01-01 00:00:00 +0000
@@ -1,26 +0,0 @@
-#!/bin/bash
-
-set -e
-
-# If we have any etc bzr nagios checks, we need to wait up to 15 minutes
-# for the cron to run to populate the check file, so just ignore those
-NAGIOS_OUTPUT=$(juju status | sed -rn 's/^ {8}public-address: //p'| xargs -I% ssh ubuntu@% 'egrep -oh /usr.*lib.* /etc/nagios/nrpe.d/check_* |grep -v check_etc_bzr.py |sed "s/.*/(set -x; &) || echo MOJO_NAGIOS_FAIL /"|sudo -u nagios -s bash |& sed "s/^/%: /"' 2>/dev/null)
-
-echo "${NAGIOS_OUTPUT}"
-
-NAGIOS_FAIL=$(echo "${NAGIOS_OUTPUT}" | grep MOJO_NAGIOS_FAIL) || true
-
-if [ -n "${NAGIOS_FAIL}" ]; then
-    echo "########################"
-    echo "# Nagios Checks Failed #"
-    echo "########################"
-    exit 1
-else
-    echo "########################"
-    echo "# Nagios Checks Passed #"
-    echo "########################"
-fi
-
-echo "#########################"
-echo "# Successfully verified #"
-echo "#########################"

=== modified file 'mojo-how-to/manifest-verify'
--- mojo-how-to/manifest-verify	2015-01-21 15:39:29 +0000
+++ mojo-how-to/manifest-verify	2015-09-08 08:23:55 +0000
@@ -1,2 +1,4 @@
+# Check juju
+verify config=check-juju
 # The service is up and running, let's verify it
-verify
+verify config=verify-nrpe

=== removed symlink 'mojo-how-to/production/verify'
=== target was u'../devel/verify'
=== added directory 'mojo-spec-helpers'
=== added directory 'mojo-spec-helpers/tests'
=== added file 'mojo-spec-helpers/tests/check-juju'
--- mojo-spec-helpers/tests/check-juju	1970-01-01 00:00:00 +0000
+++ mojo-spec-helpers/tests/check-juju	2015-09-08 08:23:55 +0000
@@ -0,0 +1,5 @@
+#!/usr/bin/python
+import utils.mojo_utils as mojo_utils
+
+mojo_utils.juju_check_hooks_complete()
+mojo_utils.juju_status_check_and_wait()

=== added symlink 'mojo-spec-helpers/tests/utils'
=== target is u'../utils'
=== added file 'mojo-spec-helpers/tests/verify-nrpe'
--- mojo-spec-helpers/tests/verify-nrpe	1970-01-01 00:00:00 +0000
+++ mojo-spec-helpers/tests/verify-nrpe	2015-09-08 08:23:55 +0000
@@ -0,0 +1,37 @@
+#!/bin/bash
+
+set -e
+
+# If we have any etc bzr nagios checks, we need to wait up to 15 minutes
+# for the cron to run to populate the check file, so just ignore those
+check() {
+    juju ssh $1 'egrep -oh /usr.*lib.* /etc/nagios/nrpe.d/check_* |\
+        grep -v check_etc_bzr.py |sed "s/.*/(set -x; &) || \
+        echo MOJO_NAGIOS_FAIL /"|sudo -u nagios -s bash' 2>/dev/null
+}
+
+NRPE_UNITS=$(juju status | sed -rn 's/^ *(nrpe\/[0-9]*):$/\1/p')
+NAGIOS_OUTPUT=$(
+    for unit in $NRPE_UNITS; do
+        check $unit | sed --e "s#^#$unit: #"
+    done
+)
+
+echo "${NAGIOS_OUTPUT}"
+
+NAGIOS_FAIL=$(echo "${NAGIOS_OUTPUT}" | grep MOJO_NAGIOS_FAIL) || true
+
+if [ -n "${NAGIOS_FAIL}" ]; then
+    echo "########################"
+    echo "# Nagios Checks Failed #"
+    echo "########################"
+    exit 1
+else
+    echo "########################"
+    echo "# Nagios Checks Passed #"
+    echo "########################"
+fi
+
+echo "########################"
+echo "# Succesfully verified #"
+echo "########################"

=== added directory 'mojo-spec-helpers/utils'
=== added file 'mojo-spec-helpers/utils/__init__.py'
=== added file 'mojo-spec-helpers/utils/add-floating-ip'
--- mojo-spec-helpers/utils/add-floating-ip	1970-01-01 00:00:00 +0000
+++ mojo-spec-helpers/utils/add-floating-ip	2015-09-08 08:23:55 +0000
@@ -0,0 +1,114 @@
+#!/bin/sh
+#
+# Author:       Paul Gear
+# Description:  Manage floating IP allocations in mojo local directory for a juju service or unit.
+# NOTE:         $MOJO_PROJECT and $MOJO_STAGE must be set before calling this script.
+#
+
+set -e
+set -u
+
+
+SECRETS_DIR="/srv/mojo/LOCAL/$MOJO_PROJECT/$MOJO_STAGE/"
+
+# Echo to standard error
+# Useful for printing from functions without polluting the "returned" output
+echo_stderr() { echo "$@" 1>&2; }
+
+# print the juju unit followed by the machine instance id
+get_juju_units()
+{
+    juju status "$@" | python -c '
+import sys, yaml
+status = yaml.safe_load(sys.stdin)
+for serv in status["services"]:
+    if status["services"][serv].get("units"):
+        for unit in status["services"][serv]["units"]:
+            machine = status["services"][serv]["units"][unit]["machine"]
+            instance = status["machines"][machine]["instance-id"]
+            print unit, instance
+'
+}
+
+
+# get the existing floating IP for this unit, or create a new one
+get_unit_floating_ip()
+{
+    UNIT="$1"
+    MACHINE="$2"
+    UNIT_FILE_NAME=$(echo "$UNIT" | tr '/' '_')
+    FLOATING_IP_FILE="$SECRETS_DIR/$UNIT_FILE_NAME"
+    IP=""
+    if [ -s "$FLOATING_IP_FILE" ]; then
+        IP=$(head -n 1 "$FLOATING_IP_FILE")
+
+    echo_stderr "- Found IP "$IP" for "$UNIT
+
+    # check how the IP is used now
+    ALLOCATION=$(nova floating-ip-list | awk -v IP="$IP" '$2 == IP {print $5}')
+    case $ALLOCATION in
+        "-")
+        # unallocated - we can use it
+        echo_stderr "- IP "$IP" is currently unallocated"
+        ;;
+        "")
+        # non-existent - we'll create one below
+        IP=""
+        echo_stderr "- No IP found for "$UNIT
+        ;;
+        *)
+        # allocated to a unit already
+        if nova show $MACHINE | awk '$3 == "network"' | sed -re 's/,? +/\n/g' | grep -q "^$IP$"; then
+            # it's allocated to us; do nothing
+            echo_stderr "- IP "$IP" is already allocated to "$UNIT
+            return
+        fi
+        # it's allocated to another unit - create one below
+        IP=""
+        echo_stderr "- IP "$IP" is allocated to another unit"
+        ;;
+    esac
+    fi
+    if [ -z "$IP" ]; then
+        IP=$(nova floating-ip-create | grep -wo '[0-9.a-f:]*')
+        echo "$IP" > "$FLOATING_IP_FILE"
+        echo_stderr "- Created new IP "$IP
+    fi
+    echo "$IP"
+}
+
+
+usage()
+{
+    cat <<EOF
+Usage: $0 {service|unit}
+
+# Add a floating IP to the apache2/0 unit:
+add-floating-ip apache2/0
+
+# Add floating IPs to all units the jenkins-slave service:
+add-floating-ip jenkins-slave
+
+# Add floating IPs to all units in the haproxy and squid services:
+add-floating-ip haproxy squid
+
+EOF
+    exit 2
+}
+
+
+if [ "$#" -lt 1 ]; then
+    usage
+fi
+
+for i in "$@"; do
+    get_juju_units "$i" | while read unit machine; do
+        echo_stderr ""
+        echo_stderr "Assigning IPs for "$unit
+        IP=$(get_unit_floating_ip "$unit" "$machine")
+        if [ -n "$IP" ]; then
+            echo_stderr "- Assigning IP "$IP" to "$unit
+            nova floating-ip-associate "$machine" "$IP"
+        fi
+    done
+done

=== added file 'mojo-spec-helpers/utils/cache_managers.py'
--- mojo-spec-helpers/utils/cache_managers.py	1970-01-01 00:00:00 +0000
+++ mojo-spec-helpers/utils/cache_managers.py	2015-09-08 08:23:55 +0000
@@ -0,0 +1,56 @@
+# System
+import os
+import json
+
+
+class JsonCache:
+    """
+    Store key value pairs in a JSON file
+    """
+
+    def __init__(self, cache_path):
+        self.cache_path = cache_path
+
+    def get_cache(self):
+        """
+        Get the dictionary from the cache file
+        or if it doesn't exist, return an empty dictionary
+        """
+
+        if os.path.isfile(self.cache_path):
+            with open(self.cache_path) as cache_file:
+                return json.load(cache_file)
+        else:
+            return {}
+
+    def put_cache(self, cache):
+        """
+        Save a dictionary to the JSON cache file
+        """
+
+        with open(self.cache_path, 'w') as cache_file:
+            json.dump(cache, cache_file)
+
+    def set(self, name, value):
+        """
+        Set a key value pair to the cache
+        """
+
+        cache = self.get_cache()
+        cache[name] = value
+        self.put_cache(cache)
+
+    def get(self, name):
+        """
+        Retrieve a value from the cache by key
+        """
+
+        return self.get_cache().get(name)
+
+    def wipe(self):
+        """
+        Remove the cache file
+        """
+
+        if os.path.isfile(self.cache_path):
+            os.remove(self.cache_path)

=== added file 'mojo-spec-helpers/utils/container_managers.py'
--- mojo-spec-helpers/utils/container_managers.py	1970-01-01 00:00:00 +0000
+++ mojo-spec-helpers/utils/container_managers.py	2015-09-08 08:23:55 +0000
@@ -0,0 +1,265 @@
+# Modules
+import requests
+
+
+class LocalEnvironmentSwiftContainer:
+    """
+    Manage a Swift container for the local deployment environment
+    for storing deployment settings
+
+    swift_connection: A working swiftclient.client.Connection object
+    container_name: The name of the swift container to use
+    """
+
+    previous_build_obj = 'previous-build-label'
+    deployed_build_obj = 'deployed-build-label'
+    deployed_revno_obj = 'deployed-spec-revno'
+    code_upgrade_succeeded_template = 'code-upgrade-{}-{}-succeeded'
+    mojo_run_succeeded_template = 'mojo-run-{}-succeeded'
+
+    def __init__(self, swift_connection, container_name):
+        self.swift_connection = swift_connection
+        self.container_name = container_name
+
+    def previous_build_label(self):
+        """
+        Get the build label that was previously deployed on this environment
+        From the Swift account associated with this environment
+        through environment variables "OS_AUTH_URL" etc.
+        """
+
+        return self.swift_connection.get_object(
+            container=self.container_name,
+            obj=self.previous_build_obj
+        )[1].strip()
+
+    def deployed_build_label(self):
+        """
+        Get the build label that is currently deployed on this environment
+        From the Swift account associated with this environment
+        through environment variables "OS_AUTH_URL" etc.
+        """
+
+        return self.swift_connection.get_object(
+            container=self.container_name,
+            obj=self.deployed_build_obj
+        )[1].strip()
+
+    def deployed_spec_revno(self):
+        """
+        Get the mojo spec revision number that is currently deployed
+        on this environment from the Swift account associated with this
+        environment through environment variables "OS_AUTH_URL" etc.
+        """
+
+        return self.swift_connection.get_object(
+            container=self.container_name,
+            obj=self.deployed_revno_obj
+        )[1].strip()
+
+    def save_code_upgrade_succeeded(
+        self, from_build_label, to_build_label
+    ):
+        """
+        Save an object into Swift
+        to signify that a code-upgrade succeeded
+        from one build_label to another
+        """
+
+        upgrade_object_name = self.code_upgrade_succeeded_template.format(
+            from_build_label, to_build_label
+        )
+
+        self.swift_connection.put_object(
+            container=self.container_name,
+            obj=upgrade_object_name,
+            contents='true'
+        )
+
+    def save_mojo_run_succeeded(self, spec_revno):
+        """
+        Save an object into Swift
+        to signifiy that a mojo-run succeeded with
+        a given revision number of the mojo spec
+        """
+
+        run_object_name = self.mojo_run_succeeded_template.format(
+            spec_revno
+        )
+
+        self.swift_connection.put_object(
+            container=self.container_name,
+            obj=run_object_name,
+            contents='true'
+        )
+
+    def save_deployed_build_label(self, build_label):
+        """
+        Save an object into swift containing
+        the build_label which is deployed to this environment
+        """
+
+        self.swift_connection.put_object(
+            container=self.container_name,
+            obj='deployed-build-label',
+            contents=build_label
+        )
+
+    def save_previous_build_label(self, build_label):
+        """
+        Save an object into swift
+        to record the previously deployed build_label
+        """
+
+        self.swift_connection.put_object(
+            container=self.container_name,
+            obj='previous-build-label',
+            contents=build_label
+        )
+
+    def save_mojo_spec_revno(self, spec_revno):
+        """
+        Save an object into swift containing
+        the current mojo spec revision number
+        """
+
+        self.swift_connection.put_object(
+            container=self.container_name,
+            obj='deployed-spec-revno',
+            contents=spec_revno
+        )
+
+
+class HttpContainer:
+    """
+    Methods for retrieving objects
+    from a Swift HTTP storage container
+    """
+
+    def __init__(self, container_url):
+        """
+        container_url: The storage URL path for the swift container
+        """
+
+        self.container_url = container_url
+
+    def get(self, object_name):
+        """
+        Retrieve the contents of an object
+        """
+
+        object_url = '{}/{}'.format(self.container_url, object_name)
+
+        response = requests.get(object_url)
+
+        try:
+            response.raise_for_status()
+        except requests.exceptions.HTTPError as http_error:
+            http_error.message += '. URL: {}'.format(object_url)
+            http_error.args += ('URL: {}'.format(object_url),)
+            raise http_error
+
+        response.raise_for_status()
+
+        return response.text
+
+    def head(self, object_name):
+        """
+        Retrieve the HEAD of an object
+        """
+
+        object_url = '{}/{}'.format(self.container_url, object_name)
+
+        return requests.head(object_url)
+
+    def exists(self, object_name):
+        """
+        Check an object exists
+        """
+
+        return self.head(object_name).ok
+
+
+class CIContainer(HttpContainer):
+    """
+    Methods for retrieving continuous integration
+    resources from an http swift store
+    """
+
+    code_upgrade_succeeded_template = 'code-upgrade-{}-{}-succeeded'
+    mojo_run_succeeded_template = 'mojo-run-{}-succeeded'
+
+    def has_code_upgrade_been_tested(
+        self,
+        from_build_label,
+        to_build_label
+    ):
+        """
+        Check if a specific code upgrade has been tested
+        (from one build_label to another)
+        by checking if a specially named Swift object exists
+        in the CI system's Swift HTTP store.
+        (This object will have been created
+        by self.save_code_upgrade_succeeded)
+        """
+
+        return self.exists(
+            self.code_upgrade_succeeded_template.format(
+                from_build_label,
+                to_build_label
+            )
+        )
+
+    def has_mojo_run_been_tested(self, spec_revno):
+        """
+        Check if a specific mojo spec revision number has been tested
+        by checking if a specially named Swift object exists
+        in the CI system's Swift HTTP store.
+        (This object will have been created by self.save_mojo_run_succeeded)
+        """
+
+        return self.exists(
+            self.mojo_run_succeeded_template.format(
+                spec_revno
+            )
+        )
+
+
+class BuildContainer(HttpContainer):
+    """
+    Methods for interacting with the HTTP Swift store
+    containing code builds
+    """
+
+    def latest_build_label(self):
+        """
+        Get the latest build label from the webteam's Swift HTTP object store
+        """
+
+        return self.get('latest-build-label').strip()
+
+
+class DeployedEnvironmentContainer(HttpContainer):
+    """
+    Methods for requesting information
+    from the HTTP swift store for a
+    depoyed environment (e.g. production, staging)
+    """
+
+    def deployed_build_label(self):
+        """
+        Get the build label which was most recently
+        deployed to this environment
+        from their Swift HTTP object store
+        """
+
+        return self.get('deployed-build-label').strip()
+
+    def deployed_spec_revno(self):
+        """
+        Get the version of the mojo spec
+        which was most recently run on this environment
+        from their Swift HTTP object store
+        """
+
+        return self.get('deployed-spec-revno').strip()

=== added file 'mojo-spec-helpers/utils/mojo_os_utils.py'
--- mojo-spec-helpers/utils/mojo_os_utils.py	1970-01-01 00:00:00 +0000
+++ mojo-spec-helpers/utils/mojo_os_utils.py	2015-09-08 08:23:55 +0000
@@ -0,0 +1,554 @@
+#!/usr/bin/python
+
+import swiftclient
+import glanceclient
+from keystoneclient.v2_0 import client as keystoneclient
+import mojo_utils
+from novaclient.v1_1 import client as novaclient
+from neutronclient.v2_0 import client as neutronclient
+import logging
+import re
+import sys
+import tempfile
+import urllib
+import os
+import time
+import subprocess
+import paramiko
+import StringIO
+
+
+# Openstack Client helpers
+def get_nova_creds(cloud_creds):
+    auth = {
+        'username': cloud_creds['OS_USERNAME'],
+        'api_key': cloud_creds['OS_PASSWORD'],
+        'auth_url': cloud_creds['OS_AUTH_URL'],
+        'project_id': cloud_creds['OS_TENANT_NAME'],
+        'region_name': cloud_creds['OS_REGION_NAME'],
+    }
+    return auth
+
+
+def get_ks_creds(cloud_creds):
+    auth = {
+        'username': cloud_creds['OS_USERNAME'],
+        'password': cloud_creds['OS_PASSWORD'],
+        'auth_url': cloud_creds['OS_AUTH_URL'],
+        'tenant_name': cloud_creds['OS_TENANT_NAME'],
+        'region_name': cloud_creds['OS_REGION_NAME'],
+    }
+    return auth
+
+
+def get_swift_creds(cloud_creds):
+    auth = {
+        'user': cloud_creds['OS_USERNAME'],
+        'key': cloud_creds['OS_PASSWORD'],
+        'authurl': cloud_creds['OS_AUTH_URL'],
+        'tenant_name': cloud_creds['OS_TENANT_NAME'],
+        'auth_version': '2.0',
+    }
+    return auth
+
+
+def get_nova_client(novarc_creds):
+    nova_creds = get_nova_creds(novarc_creds)
+    return novaclient.Client(**nova_creds)
+
+
+def get_neutron_client(novarc_creds):
+    neutron_creds = get_ks_creds(novarc_creds)
+    return neutronclient.Client(**neutron_creds)
+
+
+def get_keystone_client(novarc_creds):
+    keystone_creds = get_ks_creds(novarc_creds)
+    keystone_creds['insecure'] = True
+    return keystoneclient.Client(**keystone_creds)
+
+
+def get_swift_client(novarc_creds, insecure=True):
+    swift_creds = get_swift_creds(novarc_creds)
+    swift_creds['insecure'] = insecure
+    return swiftclient.client.Connection(**swift_creds)
+
+
+def get_glance_client(novarc_creds):
+    kc = get_keystone_client(novarc_creds)
+    glance_endpoint = kc.service_catalog.url_for(service_type='image',
+                                                 endpoint_type='publicURL')
+    return glanceclient.Client('1', glance_endpoint, token=kc.auth_token)
+
+
+# Glance Helpers
+def download_image(image, image_glance_name=None):
+    logging.info('Downloading ' + image)
+    tmp_dir = tempfile.mkdtemp(dir='/tmp')
+    if not image_glance_name:
+        image_glance_name = image.split('/')[-1]
+    local_file = os.path.join(tmp_dir, image_glance_name)
+    urllib.urlretrieve(image, local_file)
+    return local_file
+
+
+def upload_image(gclient, ifile, image_name, public, disk_format,
+                 container_format):
+    logging.info('Uploading %s to glance ' % (image_name))
+    with open(ifile) as fimage:
+        gclient.images.create(
+            name=image_name,
+            is_public=public,
+            disk_format=disk_format,
+            container_format=container_format,
+            data=fimage)
+
+
+def get_images_list(gclient):
+    return [image.name for image in gclient.images.list()]
+
+
+# Keystone helpers
+def tenant_create(kclient, tenants):
+    current_tenants = [tenant.name for tenant in kclient.tenants.list()]
+    for tenant in tenants:
+        if tenant in current_tenants:
+            logging.warning('Not creating tenant %s it already'
+                            'exists' % (tenant))
+        else:
+            logging.info('Creating tenant %s' % (tenant))
+            kclient.tenants.create(tenant_name=tenant)
+
+
+def user_create(kclient, users):
+    current_users = [user.name for user in kclient.users.list()]
+    for user in users:
+        if user['username'] in current_users:
+            logging.warning('Not creating user %s it already'
+                            'exists' % (user['username']))
+        else:
+            logging.info('Creating user %s' % (user['username']))
+            tenant_id = get_tenant_id(kclient, user['tenant'])
+            kclient.users.create(name=user['username'],
+                                 password=user['password'],
+                                 email=user['email'],
+                                 tenant_id=tenant_id)
+
+
+def get_roles_for_user(kclient, user_id, tenant_id):
+    roles = []
+    ksuser_roles = kclient.roles.roles_for_user(user_id, tenant_id)
+    for role in ksuser_roles:
+        roles.append(role.id)
+    return roles
+
+
+def add_users_to_roles(kclient, users):
+    for user_details in users:
+        tenant_id = get_tenant_id(kclient, user_details['tenant'])
+        for role_name in user_details['roles']:
+            role = kclient.roles.find(name=role_name)
+            user = kclient.users.find(name=user_details['username'])
+            users_roles = get_roles_for_user(kclient, user, tenant_id)
+            if role.id in users_roles:
+                logging.warning('Not adding role %s to %s it already has '
+                                'it' % (user_details['username'], role_name))
+            else:
+                logging.info('Adding %s to role %s for tenant'
+                             '%s' % (user_details['username'], role_name,
+                                     tenant_id))
+                kclient.roles.add_user_role(user_details['username'], role,
+                                            tenant_id)
+
+
+def get_tenant_id(ks_client, tenant_name):
+    for t in ks_client.tenants.list():
+        if t._info['name'] == tenant_name:
+            return t._info['id']
+    return None
+
+
+# Neutron Helpers
+def get_gateway_uuids():
+    gateway_config = mojo_utils.get_juju_status('neutron-gateway')
+    uuids = []
+    for machine in gateway_config['machines']:
+        uuids.append(gateway_config['machines'][machine]['instance-id'])
+    return uuids
+
+
+def get_net_uuid(neutron_client, net_name):
+    network = neutron_client.list_networks(name=net_name)['networks'][0]
+    return network['id']
+
+
+def configure_gateway_ext_port(novaclient):
+    uuids = get_gateway_uuids()
+    for uuid in uuids:
+        server = novaclient.servers.get(uuid)
+        mac_addrs = [a.mac_addr for a in server.interface_list()]
+        if len(mac_addrs) < 2:
+            logging.info('Adding additional port to Neutron Gateway')
+            server.interface_attach(port_id=None, net_id=None, fixed_ip=None)
+        else:
+            logging.warning('Neutron Gateway already has additional port')
+    if uuids:
+        logging.info('Seting Neutron Gateway external port to eth1')
+        mojo_utils.juju_set('neutron-gateway', 'ext-port=eth1')
+
+
+def create_tenant_network(neutron_client, tenant_id, net_name='private',
+                          shared=False, network_type='gre'):
+    networks = neutron_client.list_networks(name=net_name)
+    if len(networks['networks']) == 0:
+        logging.info('Creating network: %s',
+                     net_name)
+        network_msg = {
+            'network': {
+                'name': net_name,
+                'shared': shared,
+                'tenant_id': tenant_id,
+            }
+        }
+        if network_type == 'vxlan':
+            network_msg['network']['provider:segmentation_id'] = 1233
+            network_msg['network']['provider:network_type'] = network_type
+        network = neutron_client.create_network(network_msg)['network']
+    else:
+        logging.warning('Network %s already exists.', net_name)
+        network = networks['networks'][0]
+    return network
+
+
+def create_external_network(neutron_client, tenant_id, net_name='ext_net',
+                            network_type='gre'):
+    networks = neutron_client.list_networks(name=net_name)
+    if len(networks['networks']) == 0:
+        logging.info('Configuring external bridge')
+        network_msg = {
+            'name': net_name,
+            'router:external': True,
+            'tenant_id': tenant_id,
+        }
+        if network_type == 'vxlan':
+            network_msg['provider:segmentation_id'] = 1234
+            network_msg['provider:network_type'] = network_type
+
+        logging.info('Creating new external network definition: %s',
+                     net_name)
+        network = neutron_client.create_network(
+            {'network': network_msg})['network']
+        logging.info('New external network created: %s', network['id'])
+    else:
+        logging.warning('Network %s already exists.', net_name)
+        network = networks['networks'][0]
+    return network
+
+
+def create_tenant_subnet(neutron_client, tenant_id, network, cidr, dhcp=True,
+                         subnet_name='private_subnet'):
+    # Create subnet
+    subnets = neutron_client.list_subnets(name=subnet_name)
+    if len(subnets['subnets']) == 0:
+        logging.info('Creating subnet')
+        subnet_msg = {
+            'subnet': {
+                'name': subnet_name,
+                'network_id': network['id'],
+                'enable_dhcp': dhcp,
+                'cidr': cidr,
+                'ip_version': 4,
+                'tenant_id': tenant_id
+            }
+        }
+        subnet = neutron_client.create_subnet(subnet_msg)['subnet']
+    else:
+        logging.warning('Subnet %s already exists.', subnet_name)
+        subnet = subnets['subnets'][0]
+    return subnet
+
+
+def create_external_subnet(neutron_client, tenant_id, network,
+                           default_gateway=None, cidr=None,
+                           start_floating_ip=None, end_floating_ip=None,
+                           subnet_name='ext_net_subnet'):
+    subnets = neutron_client.list_subnets(name=subnet_name)
+    if len(subnets['subnets']) == 0:
+        subnet_msg = {
+            'name': subnet_name,
+            'network_id': network['id'],
+            'enable_dhcp': False,
+            'ip_version': 4,
+            'tenant_id': tenant_id
+        }
+
+        if default_gateway:
+            subnet_msg['gateway_ip'] = default_gateway
+        if cidr:
+            subnet_msg['cidr'] = cidr
+        if (start_floating_ip and end_floating_ip):
+            allocation_pool = {
+                'start': start_floating_ip,
+                'end': end_floating_ip,
+            }
+            subnet_msg['allocation_pools'] = [allocation_pool]
+
+        logging.info('Creating new subnet')
+        subnet = neutron_client.create_subnet({'subnet': subnet_msg})['subnet']
+        logging.info('New subnet created: %s', subnet['id'])
+    else:
+        logging.warning('Subnet %s already exists.', subnet_name)
+        subnet = subnets['subnets'][0]
+    return subnet
+
+
+def update_subnet_dns(neutron_client, subnet, dns_servers):
+    msg = {
+        'subnet': {
+            'dns_nameservers': dns_servers.split(',')
+        }
+    }
+    logging.info('Updating dns_nameservers (%s) for subnet',
+                 dns_servers)
+    neutron_client.update_subnet(subnet['id'], msg)
+
+
+def create_provider_router(neutron_client, tenant_id):
+    routers = neutron_client.list_routers(name='provider-router')
+    if len(routers['routers']) == 0:
+        logging.info('Creating provider router for external network access')
+        router_info = {
+            'router': {
+                'name': 'provider-router',
+                'tenant_id': tenant_id
+            }
+        }
+        router = neutron_client.create_router(router_info)['router']
+        logging.info('New router created: %s', (router['id']))
+    else:
+        logging.warning('Router provider-router already exists.')
+        router = routers['routers'][0]
+    return router
+
+
+def plug_extnet_into_router(neutron_client, router, network):
+    ports = neutron_client.list_ports(device_owner='network:router_gateway',
+                                      network_id=network['id'])
+    if len(ports['ports']) == 0:
+        logging.info('Plugging router into ext_net')
+        router = neutron_client.add_gateway_router(
+            router=router['id'],
+            body={'network_id': network['id']})
+        logging.info('Router connected')
+    else:
+        logging.warning('Router already connected')
+
+
+def plug_subnet_into_router(neutron_client, router, network, subnet):
+    routers = neutron_client.list_routers(name=router)
+    if len(routers['routers']) == 0:
+        logging.error('Unable to locate provider router %s', router)
+        sys.exit(1)
+    else:
+        # Check to see if subnet already plugged into router
+        ports = neutron_client.list_ports(
+            device_owner='network:router_interface',
+            network_id=network['id'])
+        if len(ports['ports']) == 0:
+            logging.info('Adding interface from subnet to %s' % (router))
+            router = routers['routers'][0]
+            neutron_client.add_interface_router(router['id'],
+                                                {'subnet_id': subnet['id']})
+        else:
+            logging.warning('Router already connected to subnet')
+
+
+# Nova Helpers
+def create_keypair(nova_client, keypair_name):
+    if nova_client.keypairs.findall(name=keypair_name):
+        _oldkey = nova_client.keypairs.find(name=keypair_name)
+        logging.info('Deleting key %s' % (keypair_name))
+        nova_client.keypairs.delete(_oldkey)
+    logging.info('Creating key %s' % (keypair_name))
+    new_key = nova_client.keypairs.create(name=keypair_name)
+    return new_key.private_key
+
+
+def boot_instance(nova_client, image_name, flavor_name, key_name):
+    image = nova_client.images.find(name=image_name)
+    flavor = nova_client.flavors.find(name=flavor_name)
+    net = nova_client.networks.find(label="private")
+    nics = [{'net-id': net.id}]
+    # Obviously time may not produce a unique name
+    vm_name = time.strftime("%Y%m%d%H%M%S")
+    logging.info('Creating %s %s '
+                 'instance %s' % (flavor_name, image_name, vm_name))
+    instance = nova_client.servers.create(name=vm_name,
+                                          image=image,
+                                          flavor=flavor,
+                                          key_name=key_name,
+                                          nics=nics)
+    return instance
+
+
+def wait_for_active(nova_client, vm_name, wait_time):
+    logging.info('Waiting %is for %s to reach ACTIVE '
+                 'state' % (wait_time, vm_name))
+    for counter in range(wait_time):
+        instance = nova_client.servers.find(name=vm_name)
+        if instance.status == 'ACTIVE':
+            logging.info('%s is ACTIVE' % (vm_name))
+            return True
+        elif instance.status != 'BUILD':
+            logging.error('instance %s in unknown '
+                          'state %s' % (instance.name, instance.status))
+            return False
+        time.sleep(1)
+    logging.error('instance %s failed to reach '
+                  'active state in %is' % (instance.name, wait_time))
+    return False
+
+
+def wait_for_cloudinit(nova_client, vm_name, bootstring, wait_time):
+    logging.info('Waiting %is for cloudinit on %s to '
+                 'complete' % (wait_time, vm_name))
+    instance = nova_client.servers.find(name=vm_name)
+    for counter in range(wait_time):
+        instance = nova_client.servers.find(name=vm_name)
+        console_log = instance.get_console_output()
+        if bootstring in console_log:
+            logging.info('Cloudinit for %s is complete' % (vm_name))
+            return True
+        time.sleep(1)
+    logging.error('cloudinit for instance %s failed '
+                  'to complete in %is' % (instance.name, wait_time))
+    return False
+
+
+def wait_for_boot(nova_client, vm_name, bootstring, active_wait,
+                  cloudinit_wait):
+    if not wait_for_active(nova_client, vm_name, active_wait):
+        raise Exception('Error initialising %s' % vm_name)
+    if not wait_for_cloudinit(nova_client, vm_name, bootstring,
+                              cloudinit_wait):
+        raise Exception('Cloudinit error %s' % vm_name)
+
+
+def wait_for_ping(ip, wait_time):
+    logging.info('Waiting for ping to %s' % (ip))
+    for counter in range(wait_time):
+        if ping(ip):
+            logging.info('Ping %s success' % (ip))
+            return True
+        time.sleep(1)
+    logging.error('Ping failed for %s' % (ip))
+    return False
+
+
+def assign_floating_ip(nova_client, vm_name):
+    floating_ip = nova_client.floating_ips.create()
+    logging.info('Assigning floating IP %s to %s' % (floating_ip.ip, vm_name))
+    instance = nova_client.servers.find(name=vm_name)
+    instance.add_floating_ip(floating_ip)
+    return floating_ip.ip
+
+
+def add_secgroup_rules(nova_client):
+    secgroup = nova_client.security_groups.find(name="default")
+    # Using presence of a 22 rule to indicate whether secgroup rules
+    # have been added
+    port_rules = [rule['to_port'] for rule in secgroup.rules]
+    if 22 in port_rules:
+        logging.warn('Security group rules for ssh already added')
+    else:
+        logging.info('Adding ssh security group rule')
+        nova_client.security_group_rules.create(secgroup.id,
+                                                ip_protocol="tcp",
+                                                from_port=22,
+                                                to_port=22)
+    if -1 in port_rules:
+        logging.warn('Security group rules for ping already added')
+    else:
+        logging.info('Adding ping security group rule')
+        nova_client.security_group_rules.create(secgroup.id,
+                                                ip_protocol="icmp",
+                                                from_port=-1,
+                                                to_port=-1)
+
+
+def ping(ip):
+    # Use the system ping command with count of 1 and wait time of 1.
+    ret = subprocess.call(['ping', '-c', '1', '-W', '1', ip],
+                          stdout=open('/dev/null', 'w'),
+                          stderr=open('/dev/null', 'w'))
+    return ret == 0
+
+
+def ssh_test(username, ip, vm_name, password=None, privkey=None):
+    logging.info('Attempting to ssh to %s(%s)' % (vm_name, ip))
+    ssh = paramiko.SSHClient()
+    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+    if privkey:
+        key = paramiko.RSAKey.from_private_key(StringIO.StringIO(privkey))
+        ssh.connect(ip, username=username, password='', pkey=key)
+    else:
+        ssh.connect(ip, username=username, password=password)
+    stdin, stdout, stderr = ssh.exec_command('uname -n')
+    return_string = stdout.readlines()[0].strip()
+    ssh.close()
+    if return_string == vm_name:
+        logging.info('SSH to %s(%s) succesfull' % (vm_name, ip))
+        return True
+    else:
+        logging.info('SSH to %s(%s) failed' % (vm_name, ip))
+        return False
+
+
+def boot_and_test(nova_client, image_name, flavor_name, number, privkey,
+                  active_wait=180, cloudinit_wait=180, ping_wait=180):
+    image_config = mojo_utils.get_mojo_config('images.yaml')
+    for counter in range(number):
+        instance = boot_instance(nova_client,
+                                 image_name=image_name,
+                                 flavor_name=flavor_name,
+                                 key_name='mojo')
+        wait_for_boot(nova_client, instance.name,
+                      image_config[image_name]['bootstring'], active_wait,
+                      cloudinit_wait)
+        ip = assign_floating_ip(nova_client, instance.name)
+        wait_for_ping(ip, ping_wait)
+        if not wait_for_ping(ip, ping_wait):
+            raise Exception('Ping of %s failed' % (ip))
+        ssh_test_args = {
+            'username': image_config[image_name]['username'],
+            'ip': ip,
+            'vm_name': instance.name,
+        }
+        if image_config[image_name]['auth_type'] == 'password':
+            ssh_test_args['password'] = image_config[image_name]['password']
+        elif image_config[image_name]['auth_type'] == 'privkey':
+            ssh_test_args['privkey'] = privkey
+        if not ssh_test(**ssh_test_args):
+            raise Exception('SSH failed' % (ip))
+
+
+# Hacluster helper
+def get_crm_leader(service, resource=None):
+    if not resource:
+        resource = 'res_.*_vip'
+    leader = set()
+    for unit in mojo_utils.get_juju_units(service=service):
+        crm_out = mojo_utils.remote_run(unit, 'sudo crm status')[0]
+        for line in crm_out.splitlines():
+            line = line.lstrip()
+            if re.match(resource, line):
+                leader.add(line.split()[-1])
+    if len(leader) != 1:
+        raise Exception('Unexpected leader count: ' + str(len(leader)))
+    return leader.pop().split('-')[-1]
+
+
+def delete_crm_leader(service, resource=None):
+    mach_no = get_crm_leader(service, resource)
+    unit = mojo_utils.convert_machineno_to_unit(mach_no)
+    mojo_utils.delete_unit(unit)

=== added file 'mojo-spec-helpers/utils/mojo_utils.py'
--- mojo-spec-helpers/utils/mojo_utils.py	1970-01-01 00:00:00 +0000
+++ mojo-spec-helpers/utils/mojo_utils.py	2015-09-08 08:23:55 +0000
@@ -0,0 +1,356 @@
+#!/usr/bin/python
+
+import subprocess
+import yaml
+import os
+import mojo
+import logging
+import time
+from collections import Counter
+from swiftclient.client import Connection
+
+JUJU_STATUSES = {
+    'good': ['ACTIVE', 'started'],
+    'bad': ['error'],
+    'transitional': ['pending', 'pending', 'down', 'installed', 'stopped'],
+}
+
+
+def get_juju_status(service=None):
+    cmd = ['juju', 'status']
+    if service:
+        cmd.append(service)
+    status_file = subprocess.Popen(cmd, stdout=subprocess.PIPE).stdout
+    return yaml.load(status_file)
+
+
+def get_juju_units(juju_status=None, service=None):
+    if not juju_status:
+        juju_status = get_juju_status()
+    units = []
+    if service:
+        services = [service]
+    else:
+        services = [juju_service for juju_service in juju_status['services']]
+    for svc in services:
+        if 'units' in juju_status['services'][svc]:
+            for unit in juju_status['services'][svc]['units']:
+                units.append(unit)
+    return units
+
+
+def convert_machineno_to_unit(machineno, juju_status=None):
+    if not juju_status:
+        juju_status = get_juju_status()
+    services = [service for service in juju_status['services']]
+    for svc in services:
+        if 'units' in juju_status['services'][svc]:
+            for unit in juju_status['services'][svc]['units']:
+                unit_info = juju_status['services'][svc]['units'][unit]
+                if unit_info['machine'] == machineno:
+                    return unit
+
+
+def remote_shell_check(unit):
+    cmd = ['juju', 'run', '--unit', unit, 'uname -a']
+    FNULL = open(os.devnull, 'w')
+    return not subprocess.call(cmd, stdout=FNULL, stderr=subprocess.STDOUT)
+
+
+def remote_run(unit, remote_cmd=None):
+    cmd = ['juju', 'run', '--unit', unit]
+    if remote_cmd:
+        cmd.append(remote_cmd)
+    else:
+        cmd.append('uname -a')
+    p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
+    output = p.communicate()
+    if p.returncode != 0:
+        raise Exception('Error running nagios checks')
+    return output
+
+
+def remote_upload(unit, script, remote_dir=None):
+    if remote_dir:
+        dst = unit + ':' + remote_dir
+    else:
+        dst = unit + ':/tmp/'
+    cmd = ['juju', 'scp', script, dst]
+    return subprocess.check_call(cmd)
+
+
+def delete_unit(unit):
+    service = unit.split('/')[0]
+    unit_count = len(get_juju_units(service=service))
+    logging.info('Removing unit ' + unit)
+    cmd = ['juju', 'destroy-unit', unit]
+    subprocess.check_call(cmd)
+    target_num = unit_count - 1
+    # Wait for the unit to disappear from juju status
+    while len(get_juju_units(service=service)) > target_num:
+        time.sleep(5)
+    juju_wait_finished()
+
+
+def add_unit(service, unit_num=None):
+    unit_count = len(get_juju_units(service=service))
+    if unit_num:
+        additional_units = int(unit_num)
+    else:
+        additional_units = 1
+    logging.info('Adding %i unit(s) to %s' % (additional_units, service))
+    cmd = ['juju', 'add-unit', service, '-n', str(additional_units)]
+    subprocess.check_call(cmd)
+    target_num = unit_count + additional_units
+    # Wait for the new unit to appear in juju status
+    while len(get_juju_units(service=service)) < target_num:
+        time.sleep(5)
+    juju_wait_finished()
+
+
+def juju_set(service, option):
+    subprocess.check_call(['juju', 'set', service, option])
+    juju_wait_finished()
+
+
+def juju_set_config_option(service, option_name, value):
+    option = "{}={}".format(option_name, value)
+    juju_set(service, option)
+
+
+def juju_get(service, option):
+    cmd = ['juju', 'get', service]
+    juju_get_output = subprocess.Popen(cmd, stdout=subprocess.PIPE).stdout
+    service_config = yaml.load(juju_get_output)
+    if 'value' in service_config['settings'][option]:
+        return service_config['settings'][option]['value']
+
+
+def get_undercload_auth():
+    juju_env = subprocess.check_output(['juju', 'switch']).strip('\n')
+    juju_env_file = open(os.environ['HOME'] + "/.juju/environments.yaml", 'r')
+    juju_env_contents = yaml.load(juju_env_file)
+    novarc_settings = juju_env_contents['environments'][juju_env]
+    auth_settings = {
+        'OS_AUTH_URL': novarc_settings['auth-url'],
+        'OS_TENANT_NAME': novarc_settings['tenant-name'],
+        'OS_USERNAME': novarc_settings['username'],
+        'OS_PASSWORD': novarc_settings['password'],
+        'OS_REGION_NAME': novarc_settings['region'],
+    }
+    return auth_settings
+
+
+# Openstack Client helpers
+def get_auth_url(juju_status=None):
+    if juju_get('keystone', 'vip'):
+        return juju_get('keystone', 'vip')
+    if not juju_status:
+        juju_status = get_juju_status()
+    unit = juju_status['services']['keystone']['units'].itervalues().next()
+    return unit['public-address']
+
+
+def get_overcloud_auth(juju_status=None):
+    if not juju_status:
+        juju_status = get_juju_status()
+    if juju_get('keystone', 'use-https').lower() == 'yes':
+        transport = 'https'
+        port = 35357
+    else:
+        transport = 'http'
+        port = 5000
+    address = get_auth_url()
+    auth_settings = {
+        'OS_AUTH_URL': '%s://%s:%i/v2.0' % (transport, address, port),
+        'OS_TENANT_NAME': 'admin',
+        'OS_USERNAME': 'admin',
+        'OS_PASSWORD': 'openstack',
+        'OS_REGION_NAME': 'RegionOne',
+    }
+    return auth_settings
+
+
+def get_mojo_file(filename):
+    spec = mojo.Spec(os.environ['MOJO_SPEC_DIR'])
+    return spec.get_config(filename, stage=os.environ['MOJO_STAGE'])
+
+
+def get_mojo_spec_revno():
+    """
+    Get the current revision number of the mojo spec
+    """
+
+    revno_command = 'bzr revno {}'.format(os.environ['MOJO_SPEC_DIR'])
+    return subprocess.check_output(revno_command.split()).strip()
+
+
+def get_mojo_config(filename):
+    config_file = get_mojo_file(filename)
+    logging.info('Using config %s' % (config_file))
+    return yaml.load(file(config_file, 'r'))
+
+
+def get_charm_dir():
+    return os.path.join(os.environ['MOJO_REPO_DIR'],
+                        os.environ['MOJO_SERIES'])
+
+
+def sync_charmhelpers(charmdir):
+    p = subprocess.Popen(['make', 'sync'], cwd=charmdir)
+    p.communicate()
+
+
+def sync_all_charmhelpers():
+    charm_base_dir = get_charm_dir()
+    for direc in os.listdir(charm_base_dir):
+        charm_dir = os.path.join(charm_base_dir, direc)
+        if os.path.isdir(charm_dir):
+            sync_charmhelpers(charm_dir)
+
+
+def parse_mojo_arg(options, mojoarg, multiargs=False):
+    if mojoarg.upper() in os.environ:
+        if multiargs:
+            return os.environ[mojoarg.upper()].split()
+        else:
+            return os.environ[mojoarg.upper()]
+    else:
+        return getattr(options, mojoarg)
+
+
+def get_machine_state(juju_status, state_type):
+    states = Counter()
+    for machine_no in juju_status['machines']:
+        if state_type in juju_status['machines'][machine_no]:
+            state = juju_status['machines'][machine_no][state_type]
+        else:
+            state = 'unknown'
+        states[state] += 1
+    return states
+
+
+def get_machine_agent_states(juju_status):
+    return get_machine_state(juju_status, 'agent-state')
+
+
+def get_machine_instance_states(juju_status):
+    return get_machine_state(juju_status, 'instance-state')
+
+
+def get_service_agent_states(juju_status):
+    service_state = Counter()
+    for service in juju_status['services']:
+        if 'units' in juju_status['services'][service]:
+            for unit in juju_status['services'][service]['units']:
+                unit_info = juju_status['services'][service]['units'][unit]
+                service_state[unit_info['agent-state']] += 1
+                if 'subordinates' in unit_info:
+                    for sub_unit in unit_info['subordinates']:
+                        sub_sstate = \
+                            unit_info['subordinates'][sub_unit]['agent-state']
+                        service_state[sub_sstate] += 1
+    return service_state
+
+
+def juju_status_summary(heading, statetype, states):
+    print heading
+    print "   " + statetype
+    for state in states:
+        print "    %s: %i" % (state, states[state])
+
+
+def juju_status_error_check(states):
+    for state in states:
+        if state in JUJU_STATUSES['bad']:
+            logging.error('Some statuses are in a bad state')
+            return True
+    logging.info('No statuses are in a bad state')
+    return False
+
+
+def juju_status_all_stable(states):
+    for state in states:
+        if state in JUJU_STATUSES['transitional']:
+            logging.info('Some statuses are in a transitional state')
+            return False
+    logging.info('Statuses are in a stable state')
+    return True
+
+
+def juju_status_check_and_wait():
+    checks = {
+        'Machines': [{
+                     'Heading': 'Instance State',
+                     'check_func': get_machine_instance_states,
+                     },
+                     {
+                     'Heading': 'Agent State',
+                     'check_func': get_machine_agent_states,
+                     }],
+        'Services': [{
+                     'Heading': 'Agent State',
+                     'check_func': get_service_agent_states,
+                     }]
+    }
+    stable_state = [False]
+    while False in stable_state:
+        juju_status = get_juju_status()
+        stable_state = []
+        for juju_objtype, check_info in checks.iteritems():
+            for check in check_info:
+                check_function = check['check_func']
+                states = check_function(juju_status)
+                if juju_status_error_check(states):
+                    raise Exception("Error in juju status")
+                stable_state.append(juju_status_all_stable(states))
+        time.sleep(5)
+    for juju_objtype, check_info in checks.iteritems():
+        for check in check_info:
+            check_function = check['check_func']
+            states = check_function(juju_status)
+            juju_status_summary(juju_objtype, check['Heading'], states)
+
+
+def remote_runs(units):
+    for unit in units:
+        if not remote_shell_check(unit):
+            raise Exception('Juju run failed on ' + unit)
+
+
+def juju_check_hooks_complete():
+    juju_units = get_juju_units()
+    remote_runs(juju_units)
+    remote_runs(juju_units)
+
+
+def juju_wait_finished():
+    # Wait till all statuses are green
+    juju_status_check_and_wait()
+    # juju status may report all has finished but hooks are still firing.
+    # So check..
+    juju_check_hooks_complete()
+    # Check nothing has subsequently gone bad
+    juju_status_check_and_wait()
+
+
+def build_swift_connection():
+    """
+    Create a Swift Connection object from the environment variables
+    OS_TENANT_NAME, OS_STORAGE_URL, OS_AUTH_URL, OS_USERNAME
+    OS_PASSWORD
+    """
+
+    # Get extra Swift options like tenant name and storage URL
+    os_options = {'tenant_name': os.environ.get('OS_TENANT_NAME')}
+    storage_url = os.environ.get('OS_STORAGE_URL')
+    if storage_url:
+        os_options['object_storage_url'] = storage_url
+
+    return Connection(
+        os.environ.get('OS_AUTH_URL'),
+        os.environ.get('OS_USERNAME'),
+        os.environ.get('OS_PASSWORD'),
+        auth_version='2.0',
+        os_options=os_options
+    )

=== added file 'mojo-spec-helpers/utils/shyaml.py'
--- mojo-spec-helpers/utils/shyaml.py	1970-01-01 00:00:00 +0000
+++ mojo-spec-helpers/utils/shyaml.py	2015-09-08 08:23:55 +0000
@@ -0,0 +1,219 @@
+#!/usr/bin/env python
+
+# Note: to launch test, you can use:
+#   python -m doctest -d shyaml.py
+# or
+#   nosetests
+
+from __future__ import print_function
+
+import sys
+import yaml
+import os.path
+import re
+
+EXNAME = os.path.basename(sys.argv[0])
+
+
+def tokenize(s):
+    r"""Returns an iterable in all subpart of a '.' separated string
+    So:
+        >>> list(tokenize('foo.bar.wiz'))
+        ['foo', 'bar', 'wiz']
+    this function has to deal with any type of data in the string. So it
+    actually interprets the string. Characters with meaning are '.' and '\'.
+    Both of these can be included in a token by quoting them with '\'.
+    So dot of slashes can be contained in token:
+        >>> print('\n'.join(tokenize(r'foo.dot<\.>.slash<\\>')))
+        foo
+        dot<.>
+        slash<\>
+    Notice that empty keys are also supported:
+        >>> list(tokenize(r'foo..bar'))
+        ['foo', '', 'bar']
+    Given an empty string:
+        >>> list(tokenize(r''))
+        ['']
+    And a None value:
+        >>> list(tokenize(None))
+        []
+    """
+    if s is None:
+        raise StopIteration
+    tokens = (re.sub(r'\\(\\|\.)', r'\1', m.group(0))
+              for m in re.finditer(r'((\\.|[^.\\])*)', s))
+    # an empty string superfluous token is added
+    # after all non-empty string token:
+    for token in tokens:
+        if len(token) != 0:
+            next(tokens)
+        yield token
+
+
+def mget(dct, key, default=None):
+    r"""Allow to get values deep in a dict with doted keys
+    Accessing leaf values is quite straightforward:
+        >>> dct = {'a': {'x': 1, 'b': {'c': 2}}}
+        >>> mget(dct, 'a.x')
+        1
+        >>> mget(dct, 'a.b.c')
+        2
+    But you can also get subdict if your key is not targeting a
+    leaf value:
+        >>> mget(dct, 'a.b')
+        {'c': 2}
+    As a special feature, list access is also supported by providing a
+    (possibily signed) integer, it'll be interpreted as usual python
+    sequence access using bracket notation:
+        >>> mget({'a': {'x': [1, 5], 'b': {'c': 2}}}, 'a.x.-1')
+        5
+        >>> mget({'a': {'x': 1, 'b': [{'c': 2}]}}, 'a.b.0.c')
+        2
+    Keys that contains '.' can be accessed by escaping them:
+        >>> dct = {'a': {'x': 1}, 'a.x': 3, 'a.y': 4}
+        >>> mget(dct, 'a.x')
+        1
+        >>> mget(dct, r'a\.x')
+        3
+        >>> mget(dct, r'a.y')
+        >>> mget(dct, r'a\.y')
+        4
+    As a consequence, if your key contains a '\', you should also escape it:
+        >>> dct = {r'a\x': 3, r'a\.x': 4, 'a.x': 5, 'a\\': {'x': 6}}
+        >>> mget(dct, r'a\\x')
+        3
+        >>> mget(dct, r'a\\\.x')
+        4
+        >>> mget(dct, r'a\\.x')
+        6
+        >>> mget({'a\\': {'b': 1}}, r'a\\.b')
+        1
+        >>> mget({r'a.b\.c': 1}, r'a\.b\\\.c')
+        1
+    And even empty strings key are supported:
+        >>> dct = {r'a': {'': {'y': 3}, 'y': 4}, 'b': {'': {'': 1}}, '': 2}
+        >>> mget(dct, r'a..y')
+        3
+        >>> mget(dct, r'a.y')
+        4
+        >>> mget(dct, r'')
+        2
+        >>> mget(dct, r'b..')
+        1
+    mget support also default value if the key is not found:
+        >>> mget({'a': 1}, 'b.y', default='N/A')
+        'N/A'
+    but will complain if you are trying to get into a leaf:
+        >>> mget({'a': 1}, 'a.y', default='N/A')  # doctest: +ELLIPSIS
+        Traceback (most recent call last):
+        ...
+        TypeError: 'int' object ...
+    if the key is None, the whole dct should be sent back:
+        >>> mget({'a': 1}, None)
+        {'a': 1}
+    """
+    return aget(dct, tokenize(key), default)
+
+
+def aget(dct, key, default=None):
+    r"""Allow to get values deep in a dict with iterable keys
+    Accessing leaf values is quite straightforward:
+        >>> dct = {'a': {'x': 1, 'b': {'c': 2}}}
+        >>> aget(dct, ('a', 'x'))
+        1
+        >>> aget(dct, ('a', 'b', 'c'))
+        2
+    If key is empty, it returns unchanged the ``dct`` value.
+        >>> aget({'x': 1}, ())
+        {'x': 1}
+    """
+    key = iter(key)
+    try:
+        head = next(key)
+    except StopIteration:
+        return dct
+    try:
+        value = dct[int(head)] if isinstance(dct, list) else dct[head]
+    except KeyError:
+        return default
+    return aget(value, key, default)
+
+
+def stderr(msg):
+    sys.stderr.write(msg + "\n")
+
+
+def die(msg, errlvl=1, prefix="Error: "):
+    stderr("%s%s" % (prefix, msg))
+    sys.exit(errlvl)
+
+SIMPLE_TYPES = (str, int, float)
+COMPLEX_TYPES = (list, dict)
+
+
+def dump(value):
+    return value if isinstance(value, SIMPLE_TYPES) \
+        else yaml.dump(value, default_flow_style=False)
+
+
+def type_name(value):
+    """Returns pseudo-YAML type name of given value."""
+    return "struct" if isinstance(value, dict) else \
+        "sequence" if isinstance(value, (tuple, list)) else \
+        type(value).__name__
+
+
+def stdout(value):
+    sys.stdout.write(value)
+
+
+def main(args):
+    usage = """usage:
+    %(exname)s {get-value{,-0},get-type,keys{,-0},values{,-0}} KEY DEFAULT
+    """ % {"exname": EXNAME}
+    if len(args) == 0:
+        die(usage, errlvl=0, prefix="")
+    action = args[0]
+    key_value = None if len(args) == 1 else args[1]
+    default = args[2] if len(args) > 2 else ""
+    contents = yaml.load(sys.stdin)
+    try:
+        value = mget(contents, key_value, default)
+    except IndexError:
+        die("list index error in path %r." % key_value)
+    except (KeyError, TypeError):
+        die("invalid path %r." % key_value)
+
+    tvalue = type_name(value)
+    termination = "\0" if action.endswith("-0") else "\n"
+
+    if action == "get-value":
+        print(dump(value), end='')
+    elif action in ("get-values", "get-values-0"):
+        if isinstance(value, dict):
+            for k, v in value.iteritems():
+                stdout("%s%s%s%s" % (dump(k), termination,
+                                     dump(v), termination))
+        elif isinstance(value, list):
+            for l in value:
+                stdout("%s%s" % (dump(l), termination))
+        else:
+            die("%s does not support %r type. "
+                "Please provide or select a sequence or struct."
+                % (action, tvalue))
+    elif action == "get-type":
+        print(tvalue)
+    elif action in ("keys", "keys-0", "values", "values-0"):
+        if isinstance(value, dict):
+            method = value.keys if action.startswith("keys") else value.values
+            for k in method():
+                stdout("%s%s" % (dump(k), termination))
+        else:
+            die("%s does not support %r type. "
+                "Please provide or select a struct." % (action, tvalue))
+    else:
+        die("Invalid argument.\n%s" % usage)
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv[1:]))

=== added directory 'mojo-spec-helpers/utils/tests'
=== added file 'mojo-spec-helpers/utils/tests/README.md'
--- mojo-spec-helpers/utils/tests/README.md	1970-01-01 00:00:00 +0000
+++ mojo-spec-helpers/utils/tests/README.md	2015-09-08 08:23:55 +0000
@@ -0,0 +1,67 @@
+Tests for python managers
+===
+
+These are tests for `container_managers.py`, `cache_managers.py`
+and some functions in `mojo_utils.py`.
+
+Setup
+---
+
+### A test container
+
+Tests for `container_managers.py` require a test container to be setup with objects named as follows:
+
+- latest-build-label
+- deployed-build-label
+- deployed-spec-revno
+- code-upgrade-test-build-label-01-test-build-label-02-succeeded
+- mojo-run-155-succeeded
+
+This container should be openly readable:
+
+``` bash
+swift post --read-acl .r:* ${TEST_CONTAINER_NAME}
+```
+
+You should then set the URL for this container in the environment variable `TEST_CONTAINER_URL`:
+
+``` bash
+export TEST_CONTAINER_URL=$(swift stat -v ${TEST_CONTAINER_NAME} | egrep -o 'http.*$')
+```
+
+### A local swift account
+
+You also need to make the credentials for connecting to a swift account available with `OS_*` environment variables:
+
+``` bash
+export OS_USERNAME=${USERNAME}
+export OS_TENANT_NAME=${TENANT_NAME}
+export OS_PASSWORD=${PASSWORD}
+export OS_STORAGE_URL=${OPTIONAL_STORAGE_URL}
+export OS_AUTH_URL=${AUTH_URL}
+export OS_REGION_NAME=${REGION_NAME}
+```
+
+### PYTHONPATH
+
+You also need to add the parent directory to your python path to run the tests:
+
+``` bash
+export PYTHONPATH=..
+```
+
+Running the tests
+---
+
+You can either run the tests with native python:
+
+``` bash
+./run_tests.py
+```
+
+Or with `pytest`:
+
+``` bash
+py.test
+```
+

=== added file 'mojo-spec-helpers/utils/tests/run_tests.py'
--- mojo-spec-helpers/utils/tests/run_tests.py	1970-01-01 00:00:00 +0000
+++ mojo-spec-helpers/utils/tests/run_tests.py	2015-09-08 08:23:55 +0000
@@ -0,0 +1,18 @@
+#!/usr/bin/env python
+
+"""
+These are tests for container_managers.py, cache_managers.py
+and some scripts in mojo_utils.py
+
+To run the container_tests, you'll need to set several environment
+variables - see test_container_managers.py
+"""
+
+from test_container_managers import container_tests
+from test_cache_managers import test_cache_managers
+from test_mojo_utils import test_get_mojo_spec_revno
+
+
+container_tests()
+test_cache_managers()
+test_get_mojo_spec_revno()

=== added file 'mojo-spec-helpers/utils/tests/test_cache_managers.py'
--- mojo-spec-helpers/utils/tests/test_cache_managers.py	1970-01-01 00:00:00 +0000
+++ mojo-spec-helpers/utils/tests/test_cache_managers.py	2015-09-08 08:23:55 +0000
@@ -0,0 +1,80 @@
+# System imports
+import os
+import subprocess
+import json
+
+# Local imports
+from cache_managers import JsonCache
+
+
+def _check_cache_file(cache, cache_path):
+    # Check file exists
+    assert os.path.isfile(cache_path), "Cache file not created"
+    print u"\u2713 Cache file exists: {}".format(cache_path)
+
+    # Check the data
+    with open(cache_path) as cache_file:
+        assert json.load(cache_file) == cache.get_cache(), "Bad data"
+        print u"\u2713 Cache file data is correct"
+
+
+def test_cache_managers():
+    print (
+        "\n===\n"
+        "Test JsonCache"
+        "\n===\n"
+    )
+
+    cache_path = subprocess.check_output(
+        'mktemp -u /tmp/cache-XXXX.json'.split()
+    ).strip()
+
+    cache = JsonCache(cache_path=cache_path)
+
+    fake_data = {
+        'sentence': 'hello world',
+        'number': 12,
+        'array': [1, 2, 3, "fish"]
+    }
+
+    # Insert items using "set"
+    for key, value in fake_data.iteritems():
+        cache.set(key, value)
+
+    # Check data against inserted data
+    assert fake_data == cache.get_cache(), "Data not set correctly"
+    print u"\u2713 Data correctly inserted"
+
+    # Verify cache file
+    _check_cache_file(cache, cache_path)
+
+    # Check retrieving each key with "get"
+    for key, value in fake_data.iteritems():
+        assert cache.get(key) == value
+
+    print u"\u2713 Successfully retrieved items with 'get'"
+
+    # Check wiping the cache
+    cache.wipe()
+
+    assert not os.path.isfile(cache_path), "Cache file shouldn't exist!"
+    assert not cache.get(fake_data.keys()[0]), "Cache still returning data"
+    print u"\u2713 Cache successfully wiped"
+
+    # Recreate cache with "put_cache"
+    cache.put_cache(fake_data)
+
+    # Check data against inserted data
+    assert fake_data == cache.get_cache()
+    print u"\u2713 Data correctly inserted"
+
+    # Check file integrity
+    _check_cache_file(cache, cache_path)
+
+    # Clean up
+    cache.wipe()
+    assert not os.path.isfile(cache_path), "Cache file shouldn't exist!"
+    print u"\u2713 Deleted {}".format(cache_path)
+
+if __name__ == "__main__":
+    test_cache_managers()

=== added file 'mojo-spec-helpers/utils/tests/test_container_managers.py'
--- mojo-spec-helpers/utils/tests/test_container_managers.py	1970-01-01 00:00:00 +0000
+++ mojo-spec-helpers/utils/tests/test_container_managers.py	2015-09-08 08:23:55 +0000
@@ -0,0 +1,188 @@
+# System
+import os
+
+# Local imports
+from container_managers import (
+    BuildContainer,
+    CIContainer,
+    DeployedEnvironmentContainer,
+    LocalEnvironmentSwiftContainer
+)
+from mojo_utils import build_swift_connection
+
+"""
+Tests for container_managers.
+
+To test these you'll need to set a TEST_CONTAINER_URL,
+which should be the HTTP URL to a swift container which contains
+the following publicly readable objects:
+- latest-build-label
+- deployed-build-label
+- deployed-spec-revno
+- code-upgrade-test-build-label-01-test-build-label-02-succeeded
+- mojo-run-155-succeeded
+
+You'll also need to have the following environment variables setup
+with credentials to connect to a valid (testing) Swift account:
+
+OS_AUTH_URL, OS_USERNAME, OS_PASSWORD, OS_TENANT_NAME, OS_REGION_NAME
+"""
+
+test_container_url = os.environ.get('TEST_CONTAINER_URL')
+
+
+def test_deployed_environment_container():
+    print (
+        "\n===\n"
+        "Test DeployedEnvironmentContainer"
+        "\n===\n"
+    )
+
+    deployed_env_container = DeployedEnvironmentContainer(
+        container_url=test_container_url,
+    )
+
+    build_label = deployed_env_container.deployed_build_label()
+    assert build_label, "Build label missing"
+    print u"\u2713 Found build label: {}".format(build_label)
+
+    spec_revno = deployed_env_container.deployed_spec_revno()
+    assert spec_revno, "Spec revno missing"
+    print u"\u2713 Found spec revno (from web): {}".format(spec_revno)
+
+
+def test_local_environment_swift_container():
+    print (
+        "\n===\n"
+        "Test LocalEnvironmentSwiftContainer"
+        "\n===\n"
+    )
+
+    swift_connection = build_swift_connection()
+
+    container_name = 'test-container'
+
+    swift_connection.put_container(container_name)
+
+    container = LocalEnvironmentSwiftContainer(
+        swift_connection=swift_connection,
+        container_name=container_name
+    )
+
+    deployed_build_label = "fiddlesticksandfishes"
+    previous_build_label = "fishesandfiddlesticks"
+    deployed_spec_revno = '111'
+
+    container.save_deployed_build_label(deployed_build_label)
+    container.save_previous_build_label(previous_build_label)
+    container.save_mojo_spec_revno(deployed_spec_revno)
+
+    assert container.deployed_build_label() == deployed_build_label
+    assert container.previous_build_label() == previous_build_label
+    assert container.deployed_spec_revno() == deployed_spec_revno
+    print u"\u2713 Saved and retrieved build_labels and spec_revno"
+
+    # Get names for objects
+    previous_obj = container.previous_build_obj
+    deployed_obj = container.deployed_build_obj
+    revno_obj = container.deployed_revno_obj
+    upgrade_object = container.code_upgrade_succeeded_template.format(
+        previous_build_label, deployed_build_label
+    )
+    run_object = container.mojo_run_succeeded_template.format(
+        deployed_spec_revno
+    )
+
+    # Remove the objects if they exist
+    for object_name in [previous_obj, deployed_obj, revno_obj, upgrade_object]:
+        try:
+            swift_connection.delete_object(container_name, object_name)
+        except:
+            pass
+
+    container.save_code_upgrade_succeeded(
+        previous_build_label, deployed_build_label
+    )
+
+    container.save_mojo_run_succeeded(deployed_spec_revno)
+
+    upgrade_head = swift_connection.head_object(container_name, upgrade_object)
+
+    run_head = swift_connection.head_object(container_name, run_object)
+
+    # Make sure the objects were saved to swift
+    assert int(upgrade_head['content-length']) > 0
+    assert int(run_head['content-length']) > 0
+
+    print u"\u2713 Code upgrade and mojo run successfully saved"
+
+
+def test_latest_build():
+    print (
+        "\n===\n"
+        "Test BuildContainer"
+        "\n===\n"
+    )
+
+    build_container = BuildContainer(
+        container_url=test_container_url
+    )
+
+    build_label = build_container.latest_build_label()
+    assert build_label, "Build label missing"
+    print u"\u2713 Found latest build label: {}".format(build_label)
+
+
+def test_ci_container():
+    print (
+        "\n===\n"
+        "Test CIContainer"
+        "\n===\n"
+    )
+
+    ci_container = CIContainer(
+        container_url=test_container_url
+    )
+
+    build_one = "test-build-label-01"
+    build_two = "test-build-label-02"
+
+    good_upgrade = ci_container.has_code_upgrade_been_tested(
+        from_build_label=build_one,
+        to_build_label=build_two
+    )
+    bad_upgrade = ci_container.has_code_upgrade_been_tested(
+        from_build_label=build_two,
+        to_build_label=build_one
+    )
+
+    good_upgrade = ci_container.has_code_upgrade_been_tested(
+        from_build_label=build_one,
+        to_build_label=build_two
+    )
+    bad_upgrade = ci_container.has_code_upgrade_been_tested(
+        from_build_label=build_two,
+        to_build_label=build_one
+    )
+
+    assert good_upgrade, "Good upgrade test missing"
+    assert bad_upgrade is False, "Bad upgrade isn't false"
+    print u"\u2713 Upgrades checked successfully"
+
+    good_mojo_test = ci_container.has_mojo_run_been_tested(spec_revno="155")
+    bad_mojo_test = ci_container.has_mojo_run_been_tested(spec_revno="99999")
+
+    assert good_mojo_test, "Good mojo test missing"
+    assert bad_mojo_test is False, "Bad mojo test isn't false"
+    print u"\u2713 Mojo runs checked succeeded"
+
+
+def container_tests():
+    test_deployed_environment_container()
+    test_local_environment_swift_container()
+    test_latest_build()
+    test_ci_container()
+
+
+if __name__ == "__main__":
+    container_tests()

=== added file 'mojo-spec-helpers/utils/tests/test_mojo_utils.py'
--- mojo-spec-helpers/utils/tests/test_mojo_utils.py	1970-01-01 00:00:00 +0000
+++ mojo-spec-helpers/utils/tests/test_mojo_utils.py	2015-09-08 08:23:55 +0000
@@ -0,0 +1,23 @@
+# System imports
+import os
+
+# Loca imports
+from mojo_utils import get_mojo_spec_revno
+
+
+def test_get_mojo_spec_revno():
+    print (
+        "\n===\n"
+        "Test get_mojo_spec_revno"
+        "\n===\n"
+    )
+
+    os.environ['MOJO_SPEC_DIR'] = os.path.abspath(
+        __file__ + '/../../../..'
+    )
+
+    assert get_mojo_spec_revno().isdigit()
+    print u"\u2713 Successfully retrieved revno"
+
+if __name__ == "__main__":
+    test_get_mojo_spec_revno()