Merge lp:~thumper/charms/trusty/python-django/support-1.7 into lp:charms/python-django

[Tim Penhey]

Well, this fixes the last merge I attempted. It was very broken.

In order to make sure I really didn't break it, I added some unit tests, which meant refactoring the hooks module so it had no global state. What I thought would take about and hour at the most became an eight hour marathon.

'make check' now runs the hook unit tests.

Two of the files in the hooks/tests directory were direct copies from gunicorn. I removed one and renamed the other so it doesn't get in the way.

Also, trying to run the tests indicated that the charmhandler/contrib module was attempting to bring in an unknown dependency. None of the contrib modules are used, so I removed them.

There is more cleanup to do, but I left some obvious things to fix as markers so the diff will show I didn't replace *everything*.

I have also tested manual deployment, and it works with postgresql, gunicorn and the django_settings relations.

[lp:~thumper/charms/trusty/python-django/support-1.7]

51. By Tim Penhey on 2015-05-25

lint fixes

52. By Tim Penhey on 2015-05-27

Merge prev

53. By Tim Penhey on 2015-05-29

mock out admin location

54. By Tim Penhey on 2015-05-31

Attempting to get bundletest working properly, adding 1.8 test.

55. By Tim Penhey on 2015-06-04

make check uses virtual env.

56. By Tim Penhey on 2015-06-04

Must update pip during upgrade too.

57. By Tim Penhey on 2015-06-04

Make sure to use --noinput for Django 1.7+ migrate.

58. By Tim Penhey on 2015-06-05

Make the lint target depend on flake8 in the virtual env.

59. By Tim Penhey on 2015-06-05

Make the django test executable.

60. By Tim Penhey on 2015-06-05

Tweak makefile and test calls.

61. By Tim Penhey on 2015-06-06

Fix the django 1.8 with postgresql test.
Seems that the autocommit option being used was removed in Django 1.8.

Unmerged revisions

=== modified file 'Makefile'
--- Makefile	2014-09-26 21:30:42 +0000
+++ Makefile	2015-05-27 22:03:05 +0000
@@ -28,3 +28,5 @@
 integration-test:
 	juju test --set-e -p SKIP_SLOW_TESTS,DEPLOYER_TARGET,JUJU_HOME,JUJU_ENV -v --timeout 3000s
 
+check:
+	nosetests hooks

=== modified file 'charm-helpers.yaml'
--- charm-helpers.yaml	2013-11-26 17:12:54 +0000
+++ charm-helpers.yaml	2015-05-27 22:03:05 +0000
@@ -3,4 +3,3 @@
 include:
     - core
     - fetch
-    - contrib

=== removed directory 'hooks/charmhelpers/contrib'
=== removed file 'hooks/charmhelpers/contrib/__init__.py'
=== removed directory 'hooks/charmhelpers/contrib/ansible'
=== removed file 'hooks/charmhelpers/contrib/ansible/__init__.py'
--- hooks/charmhelpers/contrib/ansible/__init__.py	2013-11-26 17:12:54 +0000
+++ hooks/charmhelpers/contrib/ansible/__init__.py	1970-01-01 00:00:00 +0000
@@ -1,165 +0,0 @@
-# Copyright 2013 Canonical Ltd.
-#
-# Authors:
-#  Charm Helpers Developers <[e-mail]>
-"""Charm Helpers ansible - declare the state of your machines.
-
-This helper enables you to declare your machine state, rather than
-program it procedurally (and have to test each change to your procedures).
-Your install hook can be as simple as:
-
-{{{
-import charmhelpers.contrib.ansible
-
-
-def install():
-    charmhelpers.contrib.ansible.install_ansible_support()
-    charmhelpers.contrib.ansible.apply_playbook('playbooks/install.yaml')
-}}}
-
-and won't need to change (nor will its tests) when you change the machine
-state.
-
-All of your juju config and relation-data are available as template
-variables within your playbooks and templates. An install playbook looks
-something like:
-
-{{{
----
-- hosts: localhost
-  user: root
-
-  tasks:
-    - name: Add private repositories.
-      template:
-        src: ../templates/private-repositories.list.jinja2
-        dest: /etc/apt/sources.list.d/private.list
-
-    - name: Update the cache.
-      apt: update_cache=yes
-
-    - name: Install dependencies.
-      apt: pkg={{ item }}
-      with_items:
-        - python-mimeparse
-        - python-webob
-        - sunburnt
-
-    - name: Setup groups.
-      group: name={{ item.name }} gid={{ item.gid }}
-      with_items:
-        - { name: 'deploy_user', gid: 1800 }
-        - { name: 'service_user', gid: 1500 }
-
-  ...
-}}}
-
-Read more online about playbooks[1] and standard ansible modules[2].
-
-[1] http://www.ansibleworks.com/docs/playbooks.html
-[2] http://www.ansibleworks.com/docs/modules.html
-"""
-import os
-import subprocess
-
-import charmhelpers.contrib.templating.contexts
-import charmhelpers.core.host
-import charmhelpers.core.hookenv
-import charmhelpers.fetch
-
-
-charm_dir = os.environ.get('CHARM_DIR', '')
-ansible_hosts_path = '/etc/ansible/hosts'
-# Ansible will automatically include any vars in the following
-# file in its inventory when run locally.
-ansible_vars_path = '/etc/ansible/host_vars/localhost'
-
-
-def install_ansible_support(from_ppa=True):
-    """Installs the ansible package.
-
-    By default it is installed from the PPA [1] linked from
-    the ansible website [2].
-
-    [1] https://launchpad.net/~rquillo/+archive/ansible
-    [2] http://www.ansibleworks.com/docs/gettingstarted.html#ubuntu-and-debian
-
-    If from_ppa is false, you must ensure that the package is available
-    from a configured repository.
-    """
-    if from_ppa:
-        charmhelpers.fetch.add_source('ppa:rquillo/ansible')
-        charmhelpers.fetch.apt_update(fatal=True)
-    charmhelpers.fetch.apt_install('ansible')
-    with open(ansible_hosts_path, 'w+') as hosts_file:
-        hosts_file.write('localhost ansible_connection=local')
-
-
-def apply_playbook(playbook, tags=None):
-    tags = tags or []
-    tags = ",".join(tags)
-    charmhelpers.contrib.templating.contexts.juju_state_to_yaml(
-        ansible_vars_path, namespace_separator='__',
-        allow_hyphens_in_keys=False)
-    call = [
-        'ansible-playbook',
-        '-c',
-        'local',
-        playbook,
-    ]
-    if tags:
-        call.extend(['--tags', '{}'.format(tags)])
-    subprocess.check_call(call)
-
-
-class AnsibleHooks(charmhelpers.core.hookenv.Hooks):
-    """Run a playbook with the hook-name as the tag.
-
-    This helper builds on the standard hookenv.Hooks helper,
-    but additionally runs the playbook with the hook-name specified
-    using --tags (ie. running all the tasks tagged with the hook-name).
-
-    Example:
-        hooks = AnsibleHooks(playbook_path='playbooks/my_machine_state.yaml')
-
-        # All the tasks within my_machine_state.yaml tagged with 'install'
-        # will be run automatically after do_custom_work()
-        @hooks.hook()
-        def install():
-            do_custom_work()
-
-        # For most of your hooks, you won't need to do anything other
-        # than run the tagged tasks for the hook:
-        @hooks.hook('config-changed', 'start', 'stop')
-        def just_use_playbook():
-            pass
-
-        # As a convenience, you can avoid the above noop function by specifying
-        # the hooks which are handled by ansible-only and they'll be registered
-        # for you:
-        # hooks = AnsibleHooks(
-        #     'playbooks/my_machine_state.yaml',
-        #     default_hooks=['config-changed', 'start', 'stop'])
-
-        if __name__ == "__main__":
-            # execute a hook based on the name the program is called by
-            hooks.execute(sys.argv)
-    """
-
-    def __init__(self, playbook_path, default_hooks=None):
-        """Register any hooks handled by ansible."""
-        super(AnsibleHooks, self).__init__()
-
-        self.playbook_path = playbook_path
-
-        default_hooks = default_hooks or []
-        noop = lambda *args, **kwargs: None
-        for hook in default_hooks:
-            self.register(hook, noop)
-
-    def execute(self, args):
-        """Execute the hook followed by the playbook using the hook as tag."""
-        super(AnsibleHooks, self).execute(args)
-        hook_name = os.path.basename(args[0])
-        charmhelpers.contrib.ansible.apply_playbook(
-            self.playbook_path, tags=[hook_name])

=== removed directory 'hooks/charmhelpers/contrib/charmhelpers'
=== removed file 'hooks/charmhelpers/contrib/charmhelpers/__init__.py'
--- hooks/charmhelpers/contrib/charmhelpers/__init__.py	2013-11-26 17:12:54 +0000
+++ hooks/charmhelpers/contrib/charmhelpers/__init__.py	1970-01-01 00:00:00 +0000
@@ -1,184 +0,0 @@
-# Copyright 2012 Canonical Ltd.  This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-import warnings
-warnings.warn("contrib.charmhelpers is deprecated", DeprecationWarning)
-
-"""Helper functions for writing Juju charms in Python."""
-
-__metaclass__ = type
-__all__ = [
-    #'get_config',             # core.hookenv.config()
-    #'log',                    # core.hookenv.log()
-    #'log_entry',              # core.hookenv.log()
-    #'log_exit',               # core.hookenv.log()
-    #'relation_get',           # core.hookenv.relation_get()
-    #'relation_set',           # core.hookenv.relation_set()
-    #'relation_ids',           # core.hookenv.relation_ids()
-    #'relation_list',          # core.hookenv.relation_units()
-    #'config_get',             # core.hookenv.config()
-    #'unit_get',               # core.hookenv.unit_get()
-    #'open_port',              # core.hookenv.open_port()
-    #'close_port',             # core.hookenv.close_port()
-    #'service_control',        # core.host.service()
-    'unit_info',              # client-side, NOT IMPLEMENTED
-    'wait_for_machine',       # client-side, NOT IMPLEMENTED
-    'wait_for_page_contents',  # client-side, NOT IMPLEMENTED
-    'wait_for_relation',      # client-side, NOT IMPLEMENTED
-    'wait_for_unit',          # client-side, NOT IMPLEMENTED
-]
-
-import operator
-from shelltoolbox import (
-    command,
-)
-import tempfile
-import time
-import urllib2
-import yaml
-
-SLEEP_AMOUNT = 0.1
-# We create a juju_status Command here because it makes testing much,
-# much easier.
-juju_status = lambda: command('juju')('status')
-
-# re-implemented as charmhelpers.fetch.configure_sources()
-#def configure_source(update=False):
-#    source = config_get('source')
-#    if ((source.startswith('ppa:') or
-#         source.startswith('cloud:') or
-#         source.startswith('http:'))):
-#        run('add-apt-repository', source)
-#    if source.startswith("http:"):
-#        run('apt-key', 'import', config_get('key'))
-#    if update:
-#        run('apt-get', 'update')
-
-
-# DEPRECATED: client-side only
-def make_charm_config_file(charm_config):
-    charm_config_file = tempfile.NamedTemporaryFile()
-    charm_config_file.write(yaml.dump(charm_config))
-    charm_config_file.flush()
-    # The NamedTemporaryFile instance is returned instead of just the name
-    # because we want to take advantage of garbage collection-triggered
-    # deletion of the temp file when it goes out of scope in the caller.
-    return charm_config_file
-
-
-# DEPRECATED: client-side only
-def unit_info(service_name, item_name, data=None, unit=None):
-    if data is None:
-        data = yaml.safe_load(juju_status())
-    service = data['services'].get(service_name)
-    if service is None:
-        # XXX 2012-02-08 gmb:
-        #     This allows us to cope with the race condition that we
-        #     have between deploying a service and having it come up in
-        #     `juju status`. We could probably do with cleaning it up so
-        #     that it fails a bit more noisily after a while.
-        return ''
-    units = service['units']
-    if unit is not None:
-        item = units[unit][item_name]
-    else:
-        # It might seem odd to sort the units here, but we do it to
-        # ensure that when no unit is specified, the first unit for the
-        # service (or at least the one with the lowest number) is the
-        # one whose data gets returned.
-        sorted_unit_names = sorted(units.keys())
-        item = units[sorted_unit_names[0]][item_name]
-    return item
-
-
-# DEPRECATED: client-side only
-def get_machine_data():
-    return yaml.safe_load(juju_status())['machines']
-
-
-# DEPRECATED: client-side only
-def wait_for_machine(num_machines=1, timeout=300):
-    """Wait `timeout` seconds for `num_machines` machines to come up.
-
-    This wait_for... function can be called by other wait_for functions
-    whose timeouts might be too short in situations where only a bare
-    Juju setup has been bootstrapped.
-
-    :return: A tuple of (num_machines, time_taken). This is used for
-             testing.
-    """
-    # You may think this is a hack, and you'd be right. The easiest way
-    # to tell what environment we're working in (LXC vs EC2) is to check
-    # the dns-name of the first machine. If it's localhost we're in LXC
-    # and we can just return here.
-    if get_machine_data()[0]['dns-name'] == 'localhost':
-        return 1, 0
-    start_time = time.time()
-    while True:
-        # Drop the first machine, since it's the Zookeeper and that's
-        # not a machine that we need to wait for. This will only work
-        # for EC2 environments, which is why we return early above if
-        # we're in LXC.
-        machine_data = get_machine_data()
-        non_zookeeper_machines = [
-            machine_data[key] for key in machine_data.keys()[1:]]
-        if len(non_zookeeper_machines) >= num_machines:
-            all_machines_running = True
-            for machine in non_zookeeper_machines:
-                if machine.get('instance-state') != 'running':
-                    all_machines_running = False
-                    break
-            if all_machines_running:
-                break
-        if time.time() - start_time >= timeout:
-            raise RuntimeError('timeout waiting for service to start')
-        time.sleep(SLEEP_AMOUNT)
-    return num_machines, time.time() - start_time
-
-
-# DEPRECATED: client-side only
-def wait_for_unit(service_name, timeout=480):
-    """Wait `timeout` seconds for a given service name to come up."""
-    wait_for_machine(num_machines=1)
-    start_time = time.time()
-    while True:
-        state = unit_info(service_name, 'agent-state')
-        if 'error' in state or state == 'started':
-            break
-        if time.time() - start_time >= timeout:
-            raise RuntimeError('timeout waiting for service to start')
-        time.sleep(SLEEP_AMOUNT)
-    if state != 'started':
-        raise RuntimeError('unit did not start, agent-state: ' + state)
-
-
-# DEPRECATED: client-side only
-def wait_for_relation(service_name, relation_name, timeout=120):
-    """Wait `timeout` seconds for a given relation to come up."""
-    start_time = time.time()
-    while True:
-        relation = unit_info(service_name, 'relations').get(relation_name)
-        if relation is not None and relation['state'] == 'up':
-            break
-        if time.time() - start_time >= timeout:
-            raise RuntimeError('timeout waiting for relation to be up')
-        time.sleep(SLEEP_AMOUNT)
-
-
-# DEPRECATED: client-side only
-def wait_for_page_contents(url, contents, timeout=120, validate=None):
-    if validate is None:
-        validate = operator.contains
-    start_time = time.time()
-    while True:
-        try:
-            stream = urllib2.urlopen(url)
-        except (urllib2.HTTPError, urllib2.URLError):
-            pass
-        else:
-            page = stream.read()
-            if validate(page, contents):
-                return page
-        if time.time() - start_time >= timeout:
-            raise RuntimeError('timeout waiting for contents of ' + url)
-        time.sleep(SLEEP_AMOUNT)

=== removed directory 'hooks/charmhelpers/contrib/charmsupport'
=== removed file 'hooks/charmhelpers/contrib/charmsupport/__init__.py'
=== removed file 'hooks/charmhelpers/contrib/charmsupport/nrpe.py'
--- hooks/charmhelpers/contrib/charmsupport/nrpe.py	2013-11-26 17:12:54 +0000
+++ hooks/charmhelpers/contrib/charmsupport/nrpe.py	1970-01-01 00:00:00 +0000
@@ -1,216 +0,0 @@
-"""Compatibility with the nrpe-external-master charm"""
-# Copyright 2012 Canonical Ltd.
-#
-# Authors:
-#  Matthew Wedgwood <[e-mail]>
-
-import subprocess
-import pwd
-import grp
-import os
-import re
-import shlex
-import yaml
-
-from charmhelpers.core.hookenv import (
-    config,
-    local_unit,
-    log,
-    relation_ids,
-    relation_set,
-)
-
-from charmhelpers.core.host import service
-
-# This module adds compatibility with the nrpe-external-master and plain nrpe
-# subordinate charms. To use it in your charm:
-#
-# 1. Update metadata.yaml
-#
-#   provides:
-#     (...)
-#     nrpe-external-master:
-#       interface: nrpe-external-master
-#       scope: container
-#
-#   and/or
-#
-#   provides:
-#     (...)
-#     local-monitors:
-#       interface: local-monitors
-#       scope: container
-
-#
-# 2. Add the following to config.yaml
-#
-#    nagios_context:
-#      default: "juju"
-#      type: string
-#      description: |
-#        Used by the nrpe subordinate charms.
-#        A string that will be prepended to instance name to set the host name
-#        in nagios. So for instance the hostname would be something like:
-#            juju-myservice-0
-#        If you're running multiple environments with the same services in them
-#        this allows you to differentiate between them.
-#
-# 3. Add custom checks (Nagios plugins) to files/nrpe-external-master
-#
-# 4. Update your hooks.py with something like this:
-#
-#    from charmsupport.nrpe import NRPE
-#    (...)
-#    def update_nrpe_config():
-#        nrpe_compat = NRPE()
-#        nrpe_compat.add_check(
-#            shortname = "myservice",
-#            description = "Check MyService",
-#            check_cmd = "check_http -w 2 -c 10 http://localhost"
-#            )
-#        nrpe_compat.add_check(
-#            "myservice_other",
-#            "Check for widget failures",
-#            check_cmd = "/srv/myapp/scripts/widget_check"
-#            )
-#        nrpe_compat.write()
-#
-#    def config_changed():
-#        (...)
-#        update_nrpe_config()
-#
-#    def nrpe_external_master_relation_changed():
-#        update_nrpe_config()
-#
-#    def local_monitors_relation_changed():
-#        update_nrpe_config()
-#
-# 5. ln -s hooks.py nrpe-external-master-relation-changed
-#    ln -s hooks.py local-monitors-relation-changed
-
-
-class CheckException(Exception):
-    pass
-
-
-class Check(object):
-    shortname_re = '[A-Za-z0-9-_]+$'
-    service_template = ("""
-#---------------------------------------------------
-# This file is Juju managed
-#---------------------------------------------------
-define service {{
-    use                             active-service
-    host_name                       {nagios_hostname}
-    service_description             {nagios_hostname}[{shortname}] """
-                        """{description}
-    check_command                   check_nrpe!{command}
-    servicegroups                   {nagios_servicegroup}
-}}
-""")
-
-    def __init__(self, shortname, description, check_cmd):
-        super(Check, self).__init__()
-        # XXX: could be better to calculate this from the service name
-        if not re.match(self.shortname_re, shortname):
-            raise CheckException("shortname must match {}".format(
-                Check.shortname_re))
-        self.shortname = shortname
-        self.command = "check_{}".format(shortname)
-        # Note: a set of invalid characters is defined by the
-        # Nagios server config
-        # The default is: illegal_object_name_chars=`~!$%^&*"|'<>?,()=
-        self.description = description
-        self.check_cmd = self._locate_cmd(check_cmd)
-
-    def _locate_cmd(self, check_cmd):
-        search_path = (
-            '/usr/lib/nagios/plugins',
-            '/usr/local/lib/nagios/plugins',
-        )
-        parts = shlex.split(check_cmd)
-        for path in search_path:
-            if os.path.exists(os.path.join(path, parts[0])):
-                command = os.path.join(path, parts[0])
-                if len(parts) > 1:
-                    command += " " + " ".join(parts[1:])
-                return command
-        log('Check command not found: {}'.format(parts[0]))
-        return ''
-
-    def write(self, nagios_context, hostname):
-        nrpe_check_file = '/etc/nagios/nrpe.d/{}.cfg'.format(
-            self.command)
-        with open(nrpe_check_file, 'w') as nrpe_check_config:
-            nrpe_check_config.write("# check {}\n".format(self.shortname))
-            nrpe_check_config.write("command[{}]={}\n".format(
-                self.command, self.check_cmd))
-
-        if not os.path.exists(NRPE.nagios_exportdir):
-            log('Not writing service config as {} is not accessible'.format(
-                NRPE.nagios_exportdir))
-        else:
-            self.write_service_config(nagios_context, hostname)
-
-    def write_service_config(self, nagios_context, hostname):
-        for f in os.listdir(NRPE.nagios_exportdir):
-            if re.search('.*{}.cfg'.format(self.command), f):
-                os.remove(os.path.join(NRPE.nagios_exportdir, f))
-
-        templ_vars = {
-            'nagios_hostname': hostname,
-            'nagios_servicegroup': nagios_context,
-            'description': self.description,
-            'shortname': self.shortname,
-            'command': self.command,
-        }
-        nrpe_service_text = Check.service_template.format(**templ_vars)
-        nrpe_service_file = '{}/service__{}_{}.cfg'.format(
-            NRPE.nagios_exportdir, hostname, self.command)
-        with open(nrpe_service_file, 'w') as nrpe_service_config:
-            nrpe_service_config.write(str(nrpe_service_text))
-
-    def run(self):
-        subprocess.call(self.check_cmd)
-
-
-class NRPE(object):
-    nagios_logdir = '/var/log/nagios'
-    nagios_exportdir = '/var/lib/nagios/export'
-    nrpe_confdir = '/etc/nagios/nrpe.d'
-
-    def __init__(self):
-        super(NRPE, self).__init__()
-        self.config = config()
-        self.nagios_context = self.config['nagios_context']
-        self.unit_name = local_unit().replace('/', '-')
-        self.hostname = "{}-{}".format(self.nagios_context, self.unit_name)
-        self.checks = []
-
-    def add_check(self, *args, **kwargs):
-        self.checks.append(Check(*args, **kwargs))
-
-    def write(self):
-        try:
-            nagios_uid = pwd.getpwnam('nagios').pw_uid
-            nagios_gid = grp.getgrnam('nagios').gr_gid
-        except:
-            log("Nagios user not set up, nrpe checks not updated")
-            return
-
-        if not os.path.exists(NRPE.nagios_logdir):
-            os.mkdir(NRPE.nagios_logdir)
-            os.chown(NRPE.nagios_logdir, nagios_uid, nagios_gid)
-
-        nrpe_monitors = {}
-        monitors = {"monitors": {"remote": {"nrpe": nrpe_monitors}}}
-        for nrpecheck in self.checks:
-            nrpecheck.write(self.nagios_context, self.hostname)
-            nrpe_monitors[nrpecheck.shortname] = {
-                "command": nrpecheck.command,
-            }
-
-        service('restart', 'nagios-nrpe-server')
-
-        for rid in relation_ids("local-monitors"):
-            relation_set(relation_id=rid, monitors=yaml.dump(monitors))

=== removed file 'hooks/charmhelpers/contrib/charmsupport/volumes.py'
--- hooks/charmhelpers/contrib/charmsupport/volumes.py	2013-11-26 17:12:54 +0000
+++ hooks/charmhelpers/contrib/charmsupport/volumes.py	1970-01-01 00:00:00 +0000
@@ -1,156 +0,0 @@
-'''
-Functions for managing volumes in juju units. One volume is supported per unit.
-Subordinates may have their own storage, provided it is on its own partition.
-
-Configuration stanzas:
-  volume-ephemeral:
-    type: boolean
-    default: true
-    description: >
-      If false, a volume is mounted as sepecified in "volume-map"
-      If true, ephemeral storage will be used, meaning that log data
-         will only exist as long as the machine. YOU HAVE BEEN WARNED.
-  volume-map:
-    type: string
-    default: {}
-    description: >
-      YAML map of units to device names, e.g:
-        "{ rsyslog/0: /dev/vdb, rsyslog/1: /dev/vdb }"
-      Service units will raise a configure-error if volume-ephemeral
-      is 'true' and no volume-map value is set. Use 'juju set' to set a
-      value and 'juju resolved' to complete configuration.
-
-Usage:
-    from charmsupport.volumes import configure_volume, VolumeConfigurationError
-    from charmsupport.hookenv import log, ERROR
-    def post_mount_hook():
-        stop_service('myservice')
-    def post_mount_hook():
-        start_service('myservice')
-
-    if __name__ == '__main__':
-        try:
-            configure_volume(before_change=pre_mount_hook,
-                             after_change=post_mount_hook)
-        except VolumeConfigurationError:
-            log('Storage could not be configured', ERROR)
-'''
-
-# XXX: Known limitations
-# - fstab is neither consulted nor updated
-
-import os
-from charmhelpers.core import hookenv
-from charmhelpers.core import host
-import yaml
-
-
-MOUNT_BASE = '/srv/juju/volumes'
-
-
-class VolumeConfigurationError(Exception):
-    '''Volume configuration data is missing or invalid'''
-    pass
-
-
-def get_config():
-    '''Gather and sanity-check volume configuration data'''
-    volume_config = {}
-    config = hookenv.config()
-
-    errors = False
-
-    if config.get('volume-ephemeral') in (True, 'True', 'true', 'Yes', 'yes'):
-        volume_config['ephemeral'] = True
-    else:
-        volume_config['ephemeral'] = False
-
-    try:
-        volume_map = yaml.safe_load(config.get('volume-map', '{}'))
-    except yaml.YAMLError as e:
-        hookenv.log("Error parsing YAML volume-map: {}".format(e),
-                    hookenv.ERROR)
-        errors = True
-    if volume_map is None:
-        # probably an empty string
-        volume_map = {}
-    elif not isinstance(volume_map, dict):
-        hookenv.log("Volume-map should be a dictionary, not {}".format(
-            type(volume_map)))
-        errors = True
-
-    volume_config['device'] = volume_map.get(os.environ['JUJU_UNIT_NAME'])
-    if volume_config['device'] and volume_config['ephemeral']:
-        # asked for ephemeral storage but also defined a volume ID
-        hookenv.log('A volume is defined for this unit, but ephemeral '
-                    'storage was requested', hookenv.ERROR)
-        errors = True
-    elif not volume_config['device'] and not volume_config['ephemeral']:
-        # asked for permanent storage but did not define volume ID
-        hookenv.log('Ephemeral storage was requested, but there is no volume '
-                    'defined for this unit.', hookenv.ERROR)
-        errors = True
-
-    unit_mount_name = hookenv.local_unit().replace('/', '-')
-    volume_config['mountpoint'] = os.path.join(MOUNT_BASE, unit_mount_name)
-
-    if errors:
-        return None
-    return volume_config
-
-
-def mount_volume(config):
-    if os.path.exists(config['mountpoint']):
-        if not os.path.isdir(config['mountpoint']):
-            hookenv.log('Not a directory: {}'.format(config['mountpoint']))
-            raise VolumeConfigurationError()
-    else:
-        host.mkdir(config['mountpoint'])
-    if os.path.ismount(config['mountpoint']):
-        unmount_volume(config)
-    if not host.mount(config['device'], config['mountpoint'], persist=True):
-        raise VolumeConfigurationError()
-
-
-def unmount_volume(config):
-    if os.path.ismount(config['mountpoint']):
-        if not host.umount(config['mountpoint'], persist=True):
-            raise VolumeConfigurationError()
-
-
-def managed_mounts():
-    '''List of all mounted managed volumes'''
-    return filter(lambda mount: mount[0].startswith(MOUNT_BASE), host.mounts())
-
-
-def configure_volume(before_change=lambda: None, after_change=lambda: None):
-    '''Set up storage (or don't) according to the charm's volume configuration.
-       Returns the mount point or "ephemeral". before_change and after_change
-       are optional functions to be called if the volume configuration changes.
-    '''
-
-    config = get_config()
-    if not config:
-        hookenv.log('Failed to read volume configuration', hookenv.CRITICAL)
-        raise VolumeConfigurationError()
-
-    if config['ephemeral']:
-        if os.path.ismount(config['mountpoint']):
-            before_change()
-            unmount_volume(config)
-            after_change()
-        return 'ephemeral'
-    else:
-        # persistent storage
-        if os.path.ismount(config['mountpoint']):
-            mounts = dict(managed_mounts())
-            if mounts.get(config['mountpoint']) != config['device']:
-                before_change()
-                unmount_volume(config)
-                mount_volume(config)
-                after_change()
-        else:
-            before_change()
-            mount_volume(config)
-            after_change()
-        return config['mountpoint']

=== removed directory 'hooks/charmhelpers/contrib/hahelpers'
=== removed file 'hooks/charmhelpers/contrib/hahelpers/__init__.py'
=== removed file 'hooks/charmhelpers/contrib/hahelpers/apache.py'
--- hooks/charmhelpers/contrib/hahelpers/apache.py	2014-05-09 20:11:59 +0000
+++ hooks/charmhelpers/contrib/hahelpers/apache.py	1970-01-01 00:00:00 +0000
@@ -1,59 +0,0 @@
-#
-# Copyright 2012 Canonical Ltd.
-#
-# This file is sourced from lp:openstack-charm-helpers
-#
-# Authors:
-#  James Page <[e-mail]>
-#  Adam Gandelman <[e-mail]>
-#
-
-import subprocess
-
-from charmhelpers.core.hookenv import (
-    config as config_get,
-    relation_get,
-    relation_ids,
-    related_units as relation_list,
-    log,
-    INFO,
-)
-
-
-def get_cert():
-    cert = config_get('ssl_cert')
-    key = config_get('ssl_key')
-    if not (cert and key):
-        log("Inspecting identity-service relations for SSL certificate.",
-            level=INFO)
-        cert = key = None
-        for r_id in relation_ids('identity-service'):
-            for unit in relation_list(r_id):
-                if not cert:
-                    cert = relation_get('ssl_cert',
-                                        rid=r_id, unit=unit)
-                if not key:
-                    key = relation_get('ssl_key',
-                                       rid=r_id, unit=unit)
-    return (cert, key)
-
-
-def get_ca_cert():
-    ca_cert = config_get('ssl_ca')
-    if ca_cert is None:
-        log("Inspecting identity-service relations for CA SSL certificate.",
-            level=INFO)
-        for r_id in relation_ids('identity-service'):
-            for unit in relation_list(r_id):
-                if ca_cert is None:
-                    ca_cert = relation_get('ca_cert',
-                                           rid=r_id, unit=unit)
-    return ca_cert
-
-
-def install_ca_cert(ca_cert):
-    if ca_cert:
-        with open('/usr/local/share/ca-certificates/keystone_juju_ca_cert.crt',
-                  'w') as crt:
-            crt.write(ca_cert)
-        subprocess.check_call(['update-ca-certificates', '--fresh'])

=== removed file 'hooks/charmhelpers/contrib/hahelpers/cluster.py'
--- hooks/charmhelpers/contrib/hahelpers/cluster.py	2014-05-09 20:11:59 +0000
+++ hooks/charmhelpers/contrib/hahelpers/cluster.py	1970-01-01 00:00:00 +0000
@@ -1,183 +0,0 @@
-#
-# Copyright 2012 Canonical Ltd.
-#
-# Authors:
-#  James Page <[e-mail]>
-#  Adam Gandelman <[e-mail]>
-#
-
-import subprocess
-import os
-
-from socket import gethostname as get_unit_hostname
-
-from charmhelpers.core.hookenv import (
-    log,
-    relation_ids,
-    related_units as relation_list,
-    relation_get,
-    config as config_get,
-    INFO,
-    ERROR,
-    unit_get,
-)
-
-
-class HAIncompleteConfig(Exception):
-    pass
-
-
-def is_clustered():
-    for r_id in (relation_ids('ha') or []):
-        for unit in (relation_list(r_id) or []):
-            clustered = relation_get('clustered',
-                                     rid=r_id,
-                                     unit=unit)
-            if clustered:
-                return True
-    return False
-
-
-def is_leader(resource):
-    cmd = [
-        "crm", "resource",
-        "show", resource
-    ]
-    try:
-        status = subprocess.check_output(cmd)
-    except subprocess.CalledProcessError:
-        return False
-    else:
-        if get_unit_hostname() in status:
-            return True
-        else:
-            return False
-
-
-def peer_units():
-    peers = []
-    for r_id in (relation_ids('cluster') or []):
-        for unit in (relation_list(r_id) or []):
-            peers.append(unit)
-    return peers
-
-
-def oldest_peer(peers):
-    local_unit_no = int(os.getenv('JUJU_UNIT_NAME').split('/')[1])
-    for peer in peers:
-        remote_unit_no = int(peer.split('/')[1])
-        if remote_unit_no < local_unit_no:
-            return False
-    return True
-
-
-def eligible_leader(resource):
-    if is_clustered():
-        if not is_leader(resource):
-            log('Deferring action to CRM leader.', level=INFO)
-            return False
-    else:
-        peers = peer_units()
-        if peers and not oldest_peer(peers):
-            log('Deferring action to oldest service unit.', level=INFO)
-            return False
-    return True
-
-
-def https():
-    '''
-    Determines whether enough data has been provided in configuration
-    or relation data to configure HTTPS
-    .
-    returns: boolean
-    '''
-    if config_get('use-https') == "yes":
-        return True
-    if config_get('ssl_cert') and config_get('ssl_key'):
-        return True
-    for r_id in relation_ids('identity-service'):
-        for unit in relation_list(r_id):
-            rel_state = [
-                relation_get('https_keystone', rid=r_id, unit=unit),
-                relation_get('ssl_cert', rid=r_id, unit=unit),
-                relation_get('ssl_key', rid=r_id, unit=unit),
-                relation_get('ca_cert', rid=r_id, unit=unit),
-            ]
-            # NOTE: works around (LP: #1203241)
-            if (None not in rel_state) and ('' not in rel_state):
-                return True
-    return False
-
-
-def determine_api_port(public_port):
-    '''
-    Determine correct API server listening port based on
-    existence of HTTPS reverse proxy and/or haproxy.
-
-    public_port: int: standard public port for given service
-
-    returns: int: the correct listening port for the API service
-    '''
-    i = 0
-    if len(peer_units()) > 0 or is_clustered():
-        i += 1
-    if https():
-        i += 1
-    return public_port - (i * 10)
-
-
-def determine_apache_port(public_port):
-    '''
-    Description: Determine correct apache listening port based on public IP +
-    state of the cluster.
-
-    public_port: int: standard public port for given service
-
-    returns: int: the correct listening port for the HAProxy service
-    '''
-    i = 0
-    if len(peer_units()) > 0 or is_clustered():
-        i += 1
-    return public_port - (i * 10)
-
-
-def get_hacluster_config():
-    '''
-    Obtains all relevant configuration from charm configuration required
-    for initiating a relation to hacluster:
-
-        ha-bindiface, ha-mcastport, vip, vip_iface, vip_cidr
-
-    returns: dict: A dict containing settings keyed by setting name.
-    raises: HAIncompleteConfig if settings are missing.
-    '''
-    settings = ['ha-bindiface', 'ha-mcastport', 'vip', 'vip_iface', 'vip_cidr']
-    conf = {}
-    for setting in settings:
-        conf[setting] = config_get(setting)
-    missing = []
-    [missing.append(s) for s, v in conf.iteritems() if v is None]
-    if missing:
-        log('Insufficient config data to configure hacluster.', level=ERROR)
-        raise HAIncompleteConfig
-    return conf
-
-
-def canonical_url(configs, vip_setting='vip'):
-    '''
-    Returns the correct HTTP URL to this host given the state of HTTPS
-    configuration and hacluster.
-
-    :configs    : OSTemplateRenderer: A config tempating object to inspect for
-                                      a complete https context.
-    :vip_setting:                str: Setting in charm config that specifies
-                                      VIP address.
-    '''
-    scheme = 'http'
-    if 'https' in configs.complete_contexts():
-        scheme = 'https'
-    if is_clustered():
-        addr = config_get(vip_setting)
-    else:
-        addr = unit_get('private-address')
-    return '%s://%s' % (scheme, addr)

=== removed directory 'hooks/charmhelpers/contrib/jujugui'
=== removed file 'hooks/charmhelpers/contrib/jujugui/__init__.py'
=== removed file 'hooks/charmhelpers/contrib/jujugui/utils.py'
--- hooks/charmhelpers/contrib/jujugui/utils.py	2013-11-26 17:12:54 +0000
+++ hooks/charmhelpers/contrib/jujugui/utils.py	1970-01-01 00:00:00 +0000
@@ -1,602 +0,0 @@
-"""Juju GUI charm utilities."""
-
-__all__ = [
-    'AGENT',
-    'APACHE',
-    'API_PORT',
-    'CURRENT_DIR',
-    'HAPROXY',
-    'IMPROV',
-    'JUJU_DIR',
-    'JUJU_GUI_DIR',
-    'JUJU_GUI_SITE',
-    'JUJU_PEM',
-    'WEB_PORT',
-    'bzr_checkout',
-    'chain',
-    'cmd_log',
-    'fetch_api',
-    'fetch_gui',
-    'find_missing_packages',
-    'first_path_in_dir',
-    'get_api_address',
-    'get_npm_cache_archive_url',
-    'get_release_file_url',
-    'get_staging_dependencies',
-    'get_zookeeper_address',
-    'legacy_juju',
-    'log_hook',
-    'merge',
-    'parse_source',
-    'prime_npm_cache',
-    'render_to_file',
-    'save_or_create_certificates',
-    'setup_apache',
-    'setup_gui',
-    'start_agent',
-    'start_gui',
-    'start_improv',
-    'write_apache_config',
-]
-
-from contextlib import contextmanager
-import errno
-import json
-import os
-import logging
-import shutil
-from subprocess import CalledProcessError
-import tempfile
-from urlparse import urlparse
-
-import apt
-import tempita
-
-from launchpadlib.launchpad import Launchpad
-from shelltoolbox import (
-    Serializer,
-    apt_get_install,
-    command,
-    environ,
-    install_extra_repositories,
-    run,
-    script_name,
-    search_file,
-    su,
-)
-from charmhelpers.core.host import (
-    service_start,
-)
-from charmhelpers.core.hookenv import (
-    log,
-    config,
-    unit_get,
-)
-
-
-AGENT = 'juju-api-agent'
-APACHE = 'apache2'
-IMPROV = 'juju-api-improv'
-HAPROXY = 'haproxy'
-
-API_PORT = 8080
-WEB_PORT = 8000
-
-CURRENT_DIR = os.getcwd()
-JUJU_DIR = os.path.join(CURRENT_DIR, 'juju')
-JUJU_GUI_DIR = os.path.join(CURRENT_DIR, 'juju-gui')
-JUJU_GUI_SITE = '/etc/apache2/sites-available/juju-gui'
-JUJU_GUI_PORTS = '/etc/apache2/ports.conf'
-JUJU_PEM = 'juju.includes-private-key.pem'
-BUILD_REPOSITORIES = ('ppa:chris-lea/node.js-legacy',)
-DEB_BUILD_DEPENDENCIES = (
-    'bzr', 'imagemagick', 'make', 'nodejs', 'npm',
-)
-DEB_STAGE_DEPENDENCIES = (
-    'zookeeper',
-)
-
-
-# Store the configuration from on invocation to the next.
-config_json = Serializer('/tmp/config.json')
-# Bazaar checkout command.
-bzr_checkout = command('bzr', 'co', '--lightweight')
-# Whether or not the charm is deployed using juju-core.
-# If juju-core has been used to deploy the charm, an agent.conf file must
-# be present in the charm parent directory.
-legacy_juju = lambda: not os.path.exists(
-    os.path.join(CURRENT_DIR, '..', 'agent.conf'))
-
-
-def _get_build_dependencies():
-    """Install deb dependencies for building."""
-    log('Installing build dependencies.')
-    cmd_log(install_extra_repositories(*BUILD_REPOSITORIES))
-    cmd_log(apt_get_install(*DEB_BUILD_DEPENDENCIES))
-
-
-def get_api_address(unit_dir):
-    """Return the Juju API address stored in the uniter agent.conf file."""
-    import yaml  # python-yaml is only installed if juju-core is used.
-    # XXX 2013-03-27 frankban bug=1161443:
-        # currently the uniter agent.conf file does not include the API
-        # address. For now retrieve it from the machine agent file.
-    base_dir = os.path.abspath(os.path.join(unit_dir, '..'))
-    for dirname in os.listdir(base_dir):
-        if dirname.startswith('machine-'):
-            agent_conf = os.path.join(base_dir, dirname, 'agent.conf')
-            break
-    else:
-        raise IOError('Juju agent configuration file not found.')
-    contents = yaml.load(open(agent_conf))
-    return contents['apiinfo']['addrs'][0]
-
-
-def get_staging_dependencies():
-    """Install deb dependencies for the stage (improv) environment."""
-    log('Installing stage dependencies.')
-    cmd_log(apt_get_install(*DEB_STAGE_DEPENDENCIES))
-
-
-def first_path_in_dir(directory):
-    """Return the full path of the first file/dir in *directory*."""
-    return os.path.join(directory, os.listdir(directory)[0])
-
-
-def _get_by_attr(collection, attr, value):
-    """Return the first item in collection having attr == value.
-
-    Return None if the item is not found.
-    """
-    for item in collection:
-        if getattr(item, attr) == value:
-            return item
-
-
-def get_release_file_url(project, series_name, release_version):
-    """Return the URL of the release file hosted in Launchpad.
-
-    The returned URL points to a release file for the given project, series
-    name and release version.
-    The argument *project* is a project object as returned by launchpadlib.
-    The arguments *series_name* and *release_version* are strings. If
-    *release_version* is None, the URL of the latest release will be returned.
-    """
-    series = _get_by_attr(project.series, 'name', series_name)
-    if series is None:
-        raise ValueError('%r: series not found' % series_name)
-    # Releases are returned by Launchpad in reverse date order.
-    releases = list(series.releases)
-    if not releases:
-        raise ValueError('%r: series does not contain releases' % series_name)
-    if release_version is not None:
-        release = _get_by_attr(releases, 'version', release_version)
-        if release is None:
-            raise ValueError('%r: release not found' % release_version)
-        releases = [release]
-    for release in releases:
-        for file_ in release.files:
-            if str(file_).endswith('.tgz'):
-                return file_.file_link
-    raise ValueError('%r: file not found' % release_version)
-
-
-def get_zookeeper_address(agent_file_path):
-    """Retrieve the Zookeeper address contained in the given *agent_file_path*.
-
-    The *agent_file_path* is a path to a file containing a line similar to the
-    following::
-
-        env JUJU_ZOOKEEPER="address"
-    """
-    line = search_file('JUJU_ZOOKEEPER', agent_file_path).strip()
-    return line.split('=')[1].strip('"')
-
-
-@contextmanager
-def log_hook():
-    """Log when a hook starts and stops its execution.
-
-    Also log to stdout possible CalledProcessError exceptions raised executing
-    the hook.
-    """
-    script = script_name()
-    log(">>> Entering {}".format(script))
-    try:
-        yield
-    except CalledProcessError as err:
-        log('Exception caught:')
-        log(err.output)
-        raise
-    finally:
-        log("<<< Exiting {}".format(script))
-
-
-def parse_source(source):
-    """Parse the ``juju-gui-source`` option.
-
-    Return a tuple of two elements representing info on how to deploy Juju GUI.
-    Examples:
-       - ('stable', None): latest stable release;
-       - ('stable', '0.1.0'): stable release v0.1.0;
-       - ('trunk', None): latest trunk release;
-       - ('trunk', '0.1.0+build.1'): trunk release v0.1.0 bzr revision 1;
-       - ('branch', 'lp:juju-gui'): release is made from a branch;
-       - ('url', 'http://example.com/gui'): release from a downloaded file.
-    """
-    if source.startswith('url:'):
-        source = source[4:]
-        # Support file paths, including relative paths.
-        if urlparse(source).scheme == '':
-            if not source.startswith('/'):
-                source = os.path.join(os.path.abspath(CURRENT_DIR), source)
-            source = "file://%s" % source
-        return 'url', source
-    if source in ('stable', 'trunk'):
-        return source, None
-    if source.startswith('lp:') or source.startswith('http://'):
-        return 'branch', source
-    if 'build' in source:
-        return 'trunk', source
-    return 'stable', source
-
-
-def render_to_file(template_name, context, destination):
-    """Render the given *template_name* into *destination* using *context*.
-
-    The tempita template language is used to render contents
-    (see http://pythonpaste.org/tempita/).
-    The argument *template_name* is the name or path of the template file:
-    it may be either a path relative to ``../config`` or an absolute path.
-    The argument *destination* is a file path.
-    The argument *context* is a dict-like object.
-    """
-    template_path = os.path.abspath(template_name)
-    template = tempita.Template.from_filename(template_path)
-    with open(destination, 'w') as stream:
-        stream.write(template.substitute(context))
-
-
-results_log = None
-
-
-def _setupLogging():
-    global results_log
-    if results_log is not None:
-        return
-    cfg = config()
-    logging.basicConfig(
-        filename=cfg['command-log-file'],
-        level=logging.INFO,
-        format="%(asctime)s: %(name)s@%(levelname)s %(message)s")
-    results_log = logging.getLogger('juju-gui')
-
-
-def cmd_log(results):
-    global results_log
-    if not results:
-        return
-    if results_log is None:
-        _setupLogging()
-    # Since 'results' may be multi-line output, start it on a separate line
-    # from the logger timestamp, etc.
-    results_log.info('\n' + results)
-
-
-def start_improv(staging_env, ssl_cert_path,
-                 config_path='/etc/init/juju-api-improv.conf'):
-    """Start a simulated juju environment using ``improv.py``."""
-    log('Setting up staging start up script.')
-    context = {
-        'juju_dir': JUJU_DIR,
-        'keys': ssl_cert_path,
-        'port': API_PORT,
-        'staging_env': staging_env,
-    }
-    render_to_file('config/juju-api-improv.conf.template', context, config_path)
-    log('Starting the staging backend.')
-    with su('root'):
-        service_start(IMPROV)
-
-
-def start_agent(
-        ssl_cert_path, config_path='/etc/init/juju-api-agent.conf',
-        read_only=False):
-    """Start the Juju agent and connect to the current environment."""
-    # Retrieve the Zookeeper address from the start up script.
-    unit_dir = os.path.realpath(os.path.join(CURRENT_DIR, '..'))
-    agent_file = '/etc/init/juju-{0}.conf'.format(os.path.basename(unit_dir))
-    zookeeper = get_zookeeper_address(agent_file)
-    log('Setting up API agent start up script.')
-    context = {
-        'juju_dir': JUJU_DIR,
-        'keys': ssl_cert_path,
-        'port': API_PORT,
-        'zookeeper': zookeeper,
-        'read_only': read_only
-    }
-    render_to_file('config/juju-api-agent.conf.template', context, config_path)
-    log('Starting API agent.')
-    with su('root'):
-        service_start(AGENT)
-
-
-def start_gui(
-        console_enabled, login_help, readonly, in_staging, ssl_cert_path,
-        charmworld_url, serve_tests, haproxy_path='/etc/haproxy/haproxy.cfg',
-        config_js_path=None, secure=True, sandbox=False):
-    """Set up and start the Juju GUI server."""
-    with su('root'):
-        run('chown', '-R', 'ubuntu:', JUJU_GUI_DIR)
-    # XXX 2013-02-05 frankban bug=1116320:
-        # External insecure resources are still loaded when testing in the
-        # debug environment. For now, switch to the production environment if
-        # the charm is configured to serve tests.
-    if in_staging and not serve_tests:
-        build_dirname = 'build-debug'
-    else:
-        build_dirname = 'build-prod'
-    build_dir = os.path.join(JUJU_GUI_DIR, build_dirname)
-    log('Generating the Juju GUI configuration file.')
-    is_legacy_juju = legacy_juju()
-    user, password = None, None
-    if (is_legacy_juju and in_staging) or sandbox:
-        user, password = 'admin', 'admin'
-    else:
-        user, password = None, None
-
-    api_backend = 'python' if is_legacy_juju else 'go'
-    if secure:
-        protocol = 'wss'
-    else:
-        log('Running in insecure mode! Port 80 will serve unencrypted.')
-        protocol = 'ws'
-
-    context = {
-        'raw_protocol': protocol,
-        'address': unit_get('public-address'),
-        'console_enabled': json.dumps(console_enabled),
-        'login_help': json.dumps(login_help),
-        'password': json.dumps(password),
-        'api_backend': json.dumps(api_backend),
-        'readonly': json.dumps(readonly),
-        'user': json.dumps(user),
-        'protocol': json.dumps(protocol),
-        'sandbox': json.dumps(sandbox),
-        'charmworld_url': json.dumps(charmworld_url),
-    }
-    if config_js_path is None:
-        config_js_path = os.path.join(
-            build_dir, 'juju-ui', 'assets', 'config.js')
-    render_to_file('config/config.js.template', context, config_js_path)
-
-    write_apache_config(build_dir, serve_tests)
-
-    log('Generating haproxy configuration file.')
-    if is_legacy_juju:
-        # The PyJuju API agent is listening on localhost.
-        api_address = '127.0.0.1:{0}'.format(API_PORT)
-    else:
-        # Retrieve the juju-core API server address.
-        api_address = get_api_address(os.path.join(CURRENT_DIR, '..'))
-    context = {
-        'api_address': api_address,
-        'api_pem': JUJU_PEM,
-        'legacy_juju': is_legacy_juju,
-        'ssl_cert_path': ssl_cert_path,
-        # In PyJuju environments, use the same certificate for both HTTPS and
-        # WebSocket connections. In juju-core the system already has the proper
-        # certificate installed.
-        'web_pem': JUJU_PEM,
-        'web_port': WEB_PORT,
-        'secure': secure
-    }
-    render_to_file('config/haproxy.cfg.template', context, haproxy_path)
-    log('Starting Juju GUI.')
-
-
-def write_apache_config(build_dir, serve_tests=False):
-    log('Generating the apache site configuration file.')
-    context = {
-        'port': WEB_PORT,
-        'serve_tests': serve_tests,
-        'server_root': build_dir,
-        'tests_root': os.path.join(JUJU_GUI_DIR, 'test', ''),
-    }
-    render_to_file('config/apache-ports.template', context, JUJU_GUI_PORTS)
-    render_to_file('config/apache-site.template', context, JUJU_GUI_SITE)
-
-
-def get_npm_cache_archive_url(Launchpad=Launchpad):
-    """Figure out the URL of the most recent NPM cache archive on Launchpad."""
-    launchpad = Launchpad.login_anonymously('Juju GUI charm', 'production')
-    project = launchpad.projects['juju-gui']
-    # Find the URL of the most recently created NPM cache archive.
-    npm_cache_url = get_release_file_url(project, 'npm-cache', None)
-    return npm_cache_url
-
-
-def prime_npm_cache(npm_cache_url):
-    """Download NPM cache archive and prime the NPM cache with it."""
-    # Download the cache archive and then uncompress it into the NPM cache.
-    npm_cache_archive = os.path.join(CURRENT_DIR, 'npm-cache.tgz')
-    cmd_log(run('curl', '-L', '-o', npm_cache_archive, npm_cache_url))
-    npm_cache_dir = os.path.expanduser('~/.npm')
-    # The NPM cache directory probably does not exist, so make it if not.
-    try:
-        os.mkdir(npm_cache_dir)
-    except OSError, e:
-        # If the directory already exists then ignore the error.
-        if e.errno != errno.EEXIST:  # File exists.
-            raise
-    uncompress = command('tar', '-x', '-z', '-C', npm_cache_dir, '-f')
-    cmd_log(uncompress(npm_cache_archive))
-
-
-def fetch_gui(juju_gui_source, logpath):
-    """Retrieve the Juju GUI release/branch."""
-    # Retrieve a Juju GUI release.
-    origin, version_or_branch = parse_source(juju_gui_source)
-    if origin == 'branch':
-        # Make sure we have the dependencies necessary for us to actually make
-        # a build.
-        _get_build_dependencies()
-        # Create a release starting from a branch.
-        juju_gui_source_dir = os.path.join(CURRENT_DIR, 'juju-gui-source')
-        log('Retrieving Juju GUI source checkout from %s.' % version_or_branch)
-        cmd_log(run('rm', '-rf', juju_gui_source_dir))
-        cmd_log(bzr_checkout(version_or_branch, juju_gui_source_dir))
-        log('Preparing a Juju GUI release.')
-        logdir = os.path.dirname(logpath)
-        fd, name = tempfile.mkstemp(prefix='make-distfile-', dir=logdir)
-        log('Output from "make distfile" sent to %s' % name)
-        with environ(NO_BZR='1'):
-            run('make', '-C', juju_gui_source_dir, 'distfile',
-                stdout=fd, stderr=fd)
-        release_tarball = first_path_in_dir(
-            os.path.join(juju_gui_source_dir, 'releases'))
-    else:
-        log('Retrieving Juju GUI release.')
-        if origin == 'url':
-            file_url = version_or_branch
-        else:
-            # Retrieve a release from Launchpad.
-            launchpad = Launchpad.login_anonymously(
-                'Juju GUI charm', 'production')
-            project = launchpad.projects['juju-gui']
-            file_url = get_release_file_url(project, origin, version_or_branch)
-        log('Downloading release file from %s.' % file_url)
-        release_tarball = os.path.join(CURRENT_DIR, 'release.tgz')
-        cmd_log(run('curl', '-L', '-o', release_tarball, file_url))
-    return release_tarball
-
-
-def fetch_api(juju_api_branch):
-    """Retrieve the Juju branch."""
-    # Retrieve Juju API source checkout.
-    log('Retrieving Juju API source checkout.')
-    cmd_log(run('rm', '-rf', JUJU_DIR))
-    cmd_log(bzr_checkout(juju_api_branch, JUJU_DIR))
-
-
-def setup_gui(release_tarball):
-    """Set up Juju GUI."""
-    # Uncompress the release tarball.
-    log('Installing Juju GUI.')
-    release_dir = os.path.join(CURRENT_DIR, 'release')
-    cmd_log(run('rm', '-rf', release_dir))
-    os.mkdir(release_dir)
-    uncompress = command('tar', '-x', '-z', '-C', release_dir, '-f')
-    cmd_log(uncompress(release_tarball))
-    # Link the Juju GUI dir to the contents of the release tarball.
-    cmd_log(run('ln', '-sf', first_path_in_dir(release_dir), JUJU_GUI_DIR))
-
-
-def setup_apache():
-    """Set up apache."""
-    log('Setting up apache.')
-    if not os.path.exists(JUJU_GUI_SITE):
-        cmd_log(run('touch', JUJU_GUI_SITE))
-        cmd_log(run('chown', 'ubuntu:', JUJU_GUI_SITE))
-        cmd_log(
-            run('ln', '-s', JUJU_GUI_SITE,
-                '/etc/apache2/sites-enabled/juju-gui'))
-
-    if not os.path.exists(JUJU_GUI_PORTS):
-        cmd_log(run('touch', JUJU_GUI_PORTS))
-        cmd_log(run('chown', 'ubuntu:', JUJU_GUI_PORTS))
-
-    with su('root'):
-        run('a2dissite', 'default')
-        run('a2ensite', 'juju-gui')
-
-
-def save_or_create_certificates(
-        ssl_cert_path, ssl_cert_contents, ssl_key_contents):
-    """Generate the SSL certificates.
-
-    If both *ssl_cert_contents* and *ssl_key_contents* are provided, use them
-    as certificates; otherwise, generate them.
-
-    Also create a pem file, suitable for use in the haproxy configuration,
-    concatenating the key and the certificate files.
-    """
-    crt_path = os.path.join(ssl_cert_path, 'juju.crt')
-    key_path = os.path.join(ssl_cert_path, 'juju.key')
-    if not os.path.exists(ssl_cert_path):
-        os.makedirs(ssl_cert_path)
-    if ssl_cert_contents and ssl_key_contents:
-        # Save the provided certificates.
-        with open(crt_path, 'w') as cert_file:
-            cert_file.write(ssl_cert_contents)
-        with open(key_path, 'w') as key_file:
-            key_file.write(ssl_key_contents)
-    else:
-        # Generate certificates.
-        # See http://superuser.com/questions/226192/openssl-without-prompt
-        cmd_log(run(
-            'openssl', 'req', '-new', '-newkey', 'rsa:4096',
-            '-days', '365', '-nodes', '-x509', '-subj',
-            # These are arbitrary test values for the certificate.
-            '/C=GB/ST=Juju/L=GUI/O=Ubuntu/CN=juju.ubuntu.com',
-            '-keyout', key_path, '-out', crt_path))
-    # Generate the pem file.
-    pem_path = os.path.join(ssl_cert_path, JUJU_PEM)
-    if os.path.exists(pem_path):
-        os.remove(pem_path)
-    with open(pem_path, 'w') as pem_file:
-        shutil.copyfileobj(open(key_path), pem_file)
-        shutil.copyfileobj(open(crt_path), pem_file)
-
-
-def find_missing_packages(*packages):
-    """Given a list of packages, return the packages which are not installed.
-    """
-    cache = apt.Cache()
-    missing = set()
-    for pkg_name in packages:
-        try:
-            pkg = cache[pkg_name]
-        except KeyError:
-            missing.add(pkg_name)
-            continue
-        if pkg.is_installed:
-            continue
-        missing.add(pkg_name)
-    return missing
-
-
-## Backend support decorators
-
-def chain(name):
-    """Helper method to compose a set of mixin objects into a callable.
-
-    Each method is called in the context of its mixin instance, and its
-    argument is the Backend instance.
-    """
-    # Chain method calls through all implementing mixins.
-    def method(self):
-        for mixin in self.mixins:
-            a_callable = getattr(type(mixin), name, None)
-            if a_callable:
-                a_callable(mixin, self)
-
-    method.__name__ = name
-    return method
-
-
-def merge(name):
-    """Helper to merge a property from a set of strategy objects
-    into a unified set.
-    """
-    # Return merged property from every providing mixin as a set.
-    @property
-    def method(self):
-        result = set()
-        for mixin in self.mixins:
-            segment = getattr(type(mixin), name, None)
-            if segment and isinstance(segment, (list, tuple, set)):
-                result |= set(segment)
-
-        return result
-    return method

=== removed directory 'hooks/charmhelpers/contrib/network'
=== removed file 'hooks/charmhelpers/contrib/network/__init__.py'
=== removed file 'hooks/charmhelpers/contrib/network/ip.py'
--- hooks/charmhelpers/contrib/network/ip.py	2014-05-09 20:11:59 +0000
+++ hooks/charmhelpers/contrib/network/ip.py	1970-01-01 00:00:00 +0000
@@ -1,69 +0,0 @@
-import sys
-
-from charmhelpers.fetch import apt_install
-from charmhelpers.core.hookenv import (
-    ERROR, log,
-)
-
-try:
-    import netifaces
-except ImportError:
-    apt_install('python-netifaces')
-    import netifaces
-
-try:
-    import netaddr
-except ImportError:
-    apt_install('python-netaddr')
-    import netaddr
-
-
-def _validate_cidr(network):
-    try:
-        netaddr.IPNetwork(network)
-    except (netaddr.core.AddrFormatError, ValueError):
-        raise ValueError("Network (%s) is not in CIDR presentation format" %
-                         network)
-
-
-def get_address_in_network(network, fallback=None, fatal=False):
-    """
-    Get an IPv4 address within the network from the host.
-
-    Args:
-        network (str): CIDR presentation format. For example,
-                       '192.168.1.0/24'.
-        fallback (str): If no address is found, return fallback.
-        fatal (boolean): If no address is found, fallback is not
-                         set and fatal is True then exit(1).
-    """
-
-    def not_found_error_out():
-        log("No IP address found in network: %s" % network,
-            level=ERROR)
-        sys.exit(1)
-
-    if network is None:
-        if fallback is not None:
-            return fallback
-        else:
-            if fatal:
-                not_found_error_out()
-
-    _validate_cidr(network)
-    for iface in netifaces.interfaces():
-        addresses = netifaces.ifaddresses(iface)
-        if netifaces.AF_INET in addresses:
-            addr = addresses[netifaces.AF_INET][0]['addr']
-            netmask = addresses[netifaces.AF_INET][0]['netmask']
-            cidr = netaddr.IPNetwork("%s/%s" % (addr, netmask))
-            if cidr in netaddr.IPNetwork(network):
-                return str(cidr.ip)
-
-    if fallback is not None:
-        return fallback
-
-    if fatal:
-        not_found_error_out()
-
-    return None

=== removed directory 'hooks/charmhelpers/contrib/network/ovs'
=== removed file 'hooks/charmhelpers/contrib/network/ovs/__init__.py'
--- hooks/charmhelpers/contrib/network/ovs/__init__.py	2013-11-26 17:12:54 +0000
+++ hooks/charmhelpers/contrib/network/ovs/__init__.py	1970-01-01 00:00:00 +0000
@@ -1,75 +0,0 @@
-''' Helpers for interacting with OpenvSwitch '''
-import subprocess
-import os
-from charmhelpers.core.hookenv import (
-    log, WARNING
-)
-from charmhelpers.core.host import (
-    service
-)
-
-
-def add_bridge(name):
-    ''' Add the named bridge to openvswitch '''
-    log('Creating bridge {}'.format(name))
-    subprocess.check_call(["ovs-vsctl", "--", "--may-exist", "add-br", name])
-
-
-def del_bridge(name):
-    ''' Delete the named bridge from openvswitch '''
-    log('Deleting bridge {}'.format(name))
-    subprocess.check_call(["ovs-vsctl", "--", "--if-exists", "del-br", name])
-
-
-def add_bridge_port(name, port):
-    ''' Add a port to the named openvswitch bridge '''
-    log('Adding port {} to bridge {}'.format(port, name))
-    subprocess.check_call(["ovs-vsctl", "--", "--may-exist", "add-port",
-                           name, port])
-    subprocess.check_call(["ip", "link", "set", port, "up"])
-
-
-def del_bridge_port(name, port):
-    ''' Delete a port from the named openvswitch bridge '''
-    log('Deleting port {} from bridge {}'.format(port, name))
-    subprocess.check_call(["ovs-vsctl", "--", "--if-exists", "del-port",
-                           name, port])
-    subprocess.check_call(["ip", "link", "set", port, "down"])
-
-
-def set_manager(manager):
-    ''' Set the controller for the local openvswitch '''
-    log('Setting manager for local ovs to {}'.format(manager))
-    subprocess.check_call(['ovs-vsctl', 'set-manager',
-                           'ssl:{}'.format(manager)])
-
-
-CERT_PATH = '/etc/openvswitch/ovsclient-cert.pem'
-
-
-def get_certificate():
-    ''' Read openvswitch certificate from disk '''
-    if os.path.exists(CERT_PATH):
-        log('Reading ovs certificate from {}'.format(CERT_PATH))
-        with open(CERT_PATH, 'r') as cert:
-            full_cert = cert.read()
-            begin_marker = "-----BEGIN CERTIFICATE-----"
-            end_marker = "-----END CERTIFICATE-----"
-            begin_index = full_cert.find(begin_marker)
-            end_index = full_cert.rfind(end_marker)
-            if end_index == -1 or begin_index == -1:
-                raise RuntimeError("Certificate does not contain valid begin"
-                                   " and end markers.")
-            full_cert = full_cert[begin_index:(end_index + len(end_marker))]
-            return full_cert
-    else:
-        log('Certificate not found', level=WARNING)
-        return None
-
-
-def full_restart():
-    ''' Full restart and reload of openvswitch '''
-    if os.path.exists('/etc/init/openvswitch-force-reload-kmod.conf'):
-        service('start', 'openvswitch-force-reload-kmod')
-    else:
-        service('force-reload-kmod', 'openvswitch-switch')

=== removed directory 'hooks/charmhelpers/contrib/openstack'
=== removed file 'hooks/charmhelpers/contrib/openstack/__init__.py'
=== removed file 'hooks/charmhelpers/contrib/openstack/alternatives.py'
--- hooks/charmhelpers/contrib/openstack/alternatives.py	2013-11-26 17:12:54 +0000
+++ hooks/charmhelpers/contrib/openstack/alternatives.py	1970-01-01 00:00:00 +0000
@@ -1,17 +0,0 @@
-''' Helper for managing alternatives for file conflict resolution '''
-
-import subprocess
-import shutil
-import os
-
-
-def install_alternative(name, target, source, priority=50):
-    ''' Install alternative configuration '''
-    if (os.path.exists(target) and not os.path.islink(target)):
-        # Move existing file/directory away before installing
-        shutil.move(target, '{}.bak'.format(target))
-    cmd = [
-        'update-alternatives', '--force', '--install',
-        target, name, source, str(priority)
-    ]
-    subprocess.check_call(cmd)

=== removed file 'hooks/charmhelpers/contrib/openstack/context.py'
--- hooks/charmhelpers/contrib/openstack/context.py	2014-05-09 20:11:59 +0000
+++ hooks/charmhelpers/contrib/openstack/context.py	1970-01-01 00:00:00 +0000
@@ -1,700 +0,0 @@
-import json
-import os
-import time
-
-from base64 import b64decode
-
-from subprocess import (
-    check_call
-)
-
-
-from charmhelpers.fetch import (
-    apt_install,
-    filter_installed_packages,
-)
-
-from charmhelpers.core.hookenv import (
-    config,
-    local_unit,
-    log,
-    relation_get,
-    relation_ids,
-    related_units,
-    unit_get,
-    unit_private_ip,
-    ERROR,
-)
-
-from charmhelpers.contrib.hahelpers.cluster import (
-    determine_apache_port,
-    determine_api_port,
-    https,
-    is_clustered
-)
-
-from charmhelpers.contrib.hahelpers.apache import (
-    get_cert,
-    get_ca_cert,
-)
-
-from charmhelpers.contrib.openstack.neutron import (
-    neutron_plugin_attribute,
-)
-
-CA_CERT_PATH = '/usr/local/share/ca-certificates/keystone_juju_ca_cert.crt'
-
-
-class OSContextError(Exception):
-    pass
-
-
-def ensure_packages(packages):
-    '''Install but do not upgrade required plugin packages'''
-    required = filter_installed_packages(packages)
-    if required:
-        apt_install(required, fatal=True)
-
-
-def context_complete(ctxt):
-    _missing = []
-    for k, v in ctxt.iteritems():
-        if v is None or v == '':
-            _missing.append(k)
-    if _missing:
-        log('Missing required data: %s' % ' '.join(_missing), level='INFO')
-        return False
-    return True
-
-
-def config_flags_parser(config_flags):
-    if config_flags.find('==') >= 0:
-        log("config_flags is not in expected format (key=value)",
-            level=ERROR)
-        raise OSContextError
-    # strip the following from each value.
-    post_strippers = ' ,'
-    # we strip any leading/trailing '=' or ' ' from the string then
-    # split on '='.
-    split = config_flags.strip(' =').split('=')
-    limit = len(split)
-    flags = {}
-    for i in xrange(0, limit - 1):
-        current = split[i]
-        next = split[i + 1]
-        vindex = next.rfind(',')
-        if (i == limit - 2) or (vindex < 0):
-            value = next
-        else:
-            value = next[:vindex]
-
-        if i == 0:
-            key = current
-        else:
-            # if this not the first entry, expect an embedded key.
-            index = current.rfind(',')
-            if index < 0:
-                log("invalid config value(s) at index %s" % (i),
-                    level=ERROR)
-                raise OSContextError
-            key = current[index + 1:]
-
-        # Add to collection.
-        flags[key.strip(post_strippers)] = value.rstrip(post_strippers)
-    return flags
-
-
-class OSContextGenerator(object):
-    interfaces = []
-
-    def __call__(self):
-        raise NotImplementedError
-
-
-class SharedDBContext(OSContextGenerator):
-    interfaces = ['shared-db']
-
-    def __init__(self,
-                 database=None, user=None, relation_prefix=None, ssl_dir=None):
-        '''
-        Allows inspecting relation for settings prefixed with relation_prefix.
-        This is useful for parsing access for multiple databases returned via
-        the shared-db interface (eg, nova_password, quantum_password)
-        '''
-        self.relation_prefix = relation_prefix
-        self.database = database
-        self.user = user
-        self.ssl_dir = ssl_dir
-
-    def __call__(self):
-        self.database = self.database or config('database')
-        self.user = self.user or config('database-user')
-        if None in [self.database, self.user]:
-            log('Could not generate shared_db context. '
-                'Missing required charm config options. '
-                '(database name and user)')
-            raise OSContextError
-        ctxt = {}
-
-        password_setting = 'password'
-        if self.relation_prefix:
-            password_setting = self.relation_prefix + '_password'
-
-        for rid in relation_ids('shared-db'):
-            for unit in related_units(rid):
-                rdata = relation_get(rid=rid, unit=unit)
-                ctxt = {
-                    'database_host': rdata.get('db_host'),
-                    'database': self.database,
-                    'database_user': self.user,
-                    'database_password': rdata.get(password_setting),
-                    'database_type': 'mysql'
-                }
-                if context_complete(ctxt):
-                    db_ssl(rdata, ctxt, self.ssl_dir)
-                    return ctxt
-        return {}
-
-
-class PostgresqlDBContext(OSContextGenerator):
-    interfaces = ['pgsql-db']
-
-    def __init__(self, database=None):
-        self.database = database
-
-    def __call__(self):
-        self.database = self.database or config('database')
-        if self.database is None:
-            log('Could not generate postgresql_db context. '
-                'Missing required charm config options. '
-                '(database name)')
-            raise OSContextError
-        ctxt = {}
-
-        for rid in relation_ids(self.interfaces[0]):
-            for unit in related_units(rid):
-                ctxt = {
-                    'database_host': relation_get('host', rid=rid, unit=unit),
-                    'database': self.database,
-                    'database_user': relation_get('user', rid=rid, unit=unit),
-                    'database_password': relation_get('password', rid=rid, unit=unit),
-                    'database_type': 'postgresql',
-                }
-                if context_complete(ctxt):
-                    return ctxt
-        return {}
-
-
-def db_ssl(rdata, ctxt, ssl_dir):
-    if 'ssl_ca' in rdata and ssl_dir:
-        ca_path = os.path.join(ssl_dir, 'db-client.ca')
-        with open(ca_path, 'w') as fh:
-            fh.write(b64decode(rdata['ssl_ca']))
-        ctxt['database_ssl_ca'] = ca_path
-    elif 'ssl_ca' in rdata:
-        log("Charm not setup for ssl support but ssl ca found")
-        return ctxt
-    if 'ssl_cert' in rdata:
-        cert_path = os.path.join(
-            ssl_dir, 'db-client.cert')
-        if not os.path.exists(cert_path):
-            log("Waiting 1m for ssl client cert validity")
-            time.sleep(60)
-        with open(cert_path, 'w') as fh:
-            fh.write(b64decode(rdata['ssl_cert']))
-        ctxt['database_ssl_cert'] = cert_path
-        key_path = os.path.join(ssl_dir, 'db-client.key')
-        with open(key_path, 'w') as fh:
-            fh.write(b64decode(rdata['ssl_key']))
-        ctxt['database_ssl_key'] = key_path
-    return ctxt
-
-
-class IdentityServiceContext(OSContextGenerator):
-    interfaces = ['identity-service']
-
-    def __call__(self):
-        log('Generating template context for identity-service')
-        ctxt = {}
-
-        for rid in relation_ids('identity-service'):
-            for unit in related_units(rid):
-                rdata = relation_get(rid=rid, unit=unit)
-                ctxt = {
-                    'service_port': rdata.get('service_port'),
-                    'service_host': rdata.get('service_host'),
-                    'auth_host': rdata.get('auth_host'),
-                    'auth_port': rdata.get('auth_port'),
-                    'admin_tenant_name': rdata.get('service_tenant'),
-                    'admin_user': rdata.get('service_username'),
-                    'admin_password': rdata.get('service_password'),
-                    'service_protocol':
-                    rdata.get('service_protocol') or 'http',
-                    'auth_protocol':
-                    rdata.get('auth_protocol') or 'http',
-                }
-                if context_complete(ctxt):
-                    # NOTE(jamespage) this is required for >= icehouse
-                    # so a missing value just indicates keystone needs
-                    # upgrading
-                    ctxt['admin_tenant_id'] = rdata.get('service_tenant_id')
-                    return ctxt
-        return {}
-
-
-class AMQPContext(OSContextGenerator):
-    interfaces = ['amqp']
-
-    def __init__(self, ssl_dir=None):
-        self.ssl_dir = ssl_dir
-
-    def __call__(self):
-        log('Generating template context for amqp')
-        conf = config()
-        try:
-            username = conf['rabbit-user']
-            vhost = conf['rabbit-vhost']
-        except KeyError as e:
-            log('Could not generate shared_db context. '
-                'Missing required charm config options: %s.' % e)
-            raise OSContextError
-        ctxt = {}
-        for rid in relation_ids('amqp'):
-            ha_vip_only = False
-            for unit in related_units(rid):
-                if relation_get('clustered', rid=rid, unit=unit):
-                    ctxt['clustered'] = True
-                    ctxt['rabbitmq_host'] = relation_get('vip', rid=rid,
-                                                         unit=unit)
-                else:
-                    ctxt['rabbitmq_host'] = relation_get('private-address',
-                                                         rid=rid, unit=unit)
-                ctxt.update({
-                    'rabbitmq_user': username,
-                    'rabbitmq_password': relation_get('password', rid=rid,
-                                                      unit=unit),
-                    'rabbitmq_virtual_host': vhost,
-                })
-
-                ssl_port = relation_get('ssl_port', rid=rid, unit=unit)
-                if ssl_port:
-                    ctxt['rabbit_ssl_port'] = ssl_port
-                ssl_ca = relation_get('ssl_ca', rid=rid, unit=unit)
-                if ssl_ca:
-                    ctxt['rabbit_ssl_ca'] = ssl_ca
-
-                if relation_get('ha_queues', rid=rid, unit=unit) is not None:
-                    ctxt['rabbitmq_ha_queues'] = True
-
-                ha_vip_only = relation_get('ha-vip-only',
-                                           rid=rid, unit=unit) is not None
-
-                if context_complete(ctxt):
-                    if 'rabbit_ssl_ca' in ctxt:
-                        if not self.ssl_dir:
-                            log(("Charm not setup for ssl support "
-                                 "but ssl ca found"))
-                            break
-                        ca_path = os.path.join(
-                            self.ssl_dir, 'rabbit-client-ca.pem')
-                        with open(ca_path, 'w') as fh:
-                            fh.write(b64decode(ctxt['rabbit_ssl_ca']))
-                            ctxt['rabbit_ssl_ca'] = ca_path
-                    # Sufficient information found = break out!
-                    break
-            # Used for active/active rabbitmq >= grizzly
-            if ('clustered' not in ctxt or ha_vip_only) \
-                    and len(related_units(rid)) > 1:
-                rabbitmq_hosts = []
-                for unit in related_units(rid):
-                    rabbitmq_hosts.append(relation_get('private-address',
-                                                       rid=rid, unit=unit))
-                ctxt['rabbitmq_hosts'] = ','.join(rabbitmq_hosts)
-        if not context_complete(ctxt):
-            return {}
-        else:
-            return ctxt
-
-
-class CephContext(OSContextGenerator):
-    interfaces = ['ceph']
-
-    def __call__(self):
-        '''This generates context for /etc/ceph/ceph.conf templates'''
-        if not relation_ids('ceph'):
-            return {}
-
-        log('Generating template context for ceph')
-
-        mon_hosts = []
-        auth = None
-        key = None
-        use_syslog = str(config('use-syslog')).lower()
-        for rid in relation_ids('ceph'):
-            for unit in related_units(rid):
-                mon_hosts.append(relation_get('private-address', rid=rid,
-                                              unit=unit))
-                auth = relation_get('auth', rid=rid, unit=unit)
-                key = relation_get('key', rid=rid, unit=unit)
-
-        ctxt = {
-            'mon_hosts': ' '.join(mon_hosts),
-            'auth': auth,
-            'key': key,
-            'use_syslog': use_syslog
-        }
-
-        if not os.path.isdir('/etc/ceph'):
-            os.mkdir('/etc/ceph')
-
-        if not context_complete(ctxt):
-            return {}
-
-        ensure_packages(['ceph-common'])
-
-        return ctxt
-
-
-class HAProxyContext(OSContextGenerator):
-    interfaces = ['cluster']
-
-    def __call__(self):
-        '''
-        Builds half a context for the haproxy template, which describes
-        all peers to be included in the cluster.  Each charm needs to include
-        its own context generator that describes the port mapping.
-        '''
-        if not relation_ids('cluster'):
-            return {}
-
-        cluster_hosts = {}
-        l_unit = local_unit().replace('/', '-')
-        cluster_hosts[l_unit] = unit_get('private-address')
-
-        for rid in relation_ids('cluster'):
-            for unit in related_units(rid):
-                _unit = unit.replace('/', '-')
-                addr = relation_get('private-address', rid=rid, unit=unit)
-                cluster_hosts[_unit] = addr
-
-        ctxt = {
-            'units': cluster_hosts,
-        }
-        if len(cluster_hosts.keys()) > 1:
-            # Enable haproxy when we have enough peers.
-            log('Ensuring haproxy enabled in /etc/default/haproxy.')
-            with open('/etc/default/haproxy', 'w') as out:
-                out.write('ENABLED=1\n')
-            return ctxt
-        log('HAProxy context is incomplete, this unit has no peers.')
-        return {}
-
-
-class ImageServiceContext(OSContextGenerator):
-    interfaces = ['image-service']
-
-    def __call__(self):
-        '''
-        Obtains the glance API server from the image-service relation.  Useful
-        in nova and cinder (currently).
-        '''
-        log('Generating template context for image-service.')
-        rids = relation_ids('image-service')
-        if not rids:
-            return {}
-        for rid in rids:
-            for unit in related_units(rid):
-                api_server = relation_get('glance-api-server',
-                                          rid=rid, unit=unit)
-                if api_server:
-                    return {'glance_api_servers': api_server}
-        log('ImageService context is incomplete. '
-            'Missing required relation data.')
-        return {}
-
-
-class ApacheSSLContext(OSContextGenerator):
-
-    """
-    Generates a context for an apache vhost configuration that configures
-    HTTPS reverse proxying for one or many endpoints.  Generated context
-    looks something like:
-    {
-        'namespace': 'cinder',
-        'private_address': 'iscsi.mycinderhost.com',
-        'endpoints': [(8776, 8766), (8777, 8767)]
-    }
-
-    The endpoints list consists of a tuples mapping external ports
-    to internal ports.
-    """
-    interfaces = ['https']
-
-    # charms should inherit this context and set external ports
-    # and service namespace accordingly.
-    external_ports = []
-    service_namespace = None
-
-    def enable_modules(self):
-        cmd = ['a2enmod', 'ssl', 'proxy', 'proxy_http']
-        check_call(cmd)
-
-    def configure_cert(self):
-        if not os.path.isdir('/etc/apache2/ssl'):
-            os.mkdir('/etc/apache2/ssl')
-        ssl_dir = os.path.join('/etc/apache2/ssl/', self.service_namespace)
-        if not os.path.isdir(ssl_dir):
-            os.mkdir(ssl_dir)
-        cert, key = get_cert()
-        with open(os.path.join(ssl_dir, 'cert'), 'w') as cert_out:
-            cert_out.write(b64decode(cert))
-        with open(os.path.join(ssl_dir, 'key'), 'w') as key_out:
-            key_out.write(b64decode(key))
-        ca_cert = get_ca_cert()
-        if ca_cert:
-            with open(CA_CERT_PATH, 'w') as ca_out:
-                ca_out.write(b64decode(ca_cert))
-            check_call(['update-ca-certificates'])
-
-    def __call__(self):
-        if isinstance(self.external_ports, basestring):
-            self.external_ports = [self.external_ports]
-        if (not self.external_ports or not https()):
-            return {}
-
-        self.configure_cert()
-        self.enable_modules()
-
-        ctxt = {
-            'namespace': self.service_namespace,
-            'private_address': unit_get('private-address'),
-            'endpoints': []
-        }
-        if is_clustered():
-            ctxt['private_address'] = config('vip')
-        for api_port in self.external_ports:
-            ext_port = determine_apache_port(api_port)
-            int_port = determine_api_port(api_port)
-            portmap = (int(ext_port), int(int_port))
-            ctxt['endpoints'].append(portmap)
-        return ctxt
-
-
-class NeutronContext(OSContextGenerator):
-    interfaces = []
-
-    @property
-    def plugin(self):
-        return None
-
-    @property
-    def network_manager(self):
-        return None
-
-    @property
-    def packages(self):
-        return neutron_plugin_attribute(
-            self.plugin, 'packages', self.network_manager)
-
-    @property
-    def neutron_security_groups(self):
-        return None
-
-    def _ensure_packages(self):
-        [ensure_packages(pkgs) for pkgs in self.packages]
-
-    def _save_flag_file(self):
-        if self.network_manager == 'quantum':
-            _file = '/etc/nova/quantum_plugin.conf'
-        else:
-            _file = '/etc/nova/neutron_plugin.conf'
-        with open(_file, 'wb') as out:
-            out.write(self.plugin + '\n')
-
-    def ovs_ctxt(self):
-        driver = neutron_plugin_attribute(self.plugin, 'driver',
-                                          self.network_manager)
-        config = neutron_plugin_attribute(self.plugin, 'config',
-                                          self.network_manager)
-        ovs_ctxt = {
-            'core_plugin': driver,
-            'neutron_plugin': 'ovs',
-            'neutron_security_groups': self.neutron_security_groups,
-            'local_ip': unit_private_ip(),
-            'config': config
-        }
-
-        return ovs_ctxt
-
-    def nvp_ctxt(self):
-        driver = neutron_plugin_attribute(self.plugin, 'driver',
-                                          self.network_manager)
-        config = neutron_plugin_attribute(self.plugin, 'config',
-                                          self.network_manager)
-        nvp_ctxt = {
-            'core_plugin': driver,
-            'neutron_plugin': 'nvp',
-            'neutron_security_groups': self.neutron_security_groups,
-            'local_ip': unit_private_ip(),
-            'config': config
-        }
-
-        return nvp_ctxt
-
-    def neutron_ctxt(self):
-        if https():
-            proto = 'https'
-        else:
-            proto = 'http'
-        if is_clustered():
-            host = config('vip')
-        else:
-            host = unit_get('private-address')
-        url = '%s://%s:%s' % (proto, host, '9696')
-        ctxt = {
-            'network_manager': self.network_manager,
-            'neutron_url': url,
-        }
-        return ctxt
-
-    def __call__(self):
-        self._ensure_packages()
-
-        if self.network_manager not in ['quantum', 'neutron']:
-            return {}
-
-        if not self.plugin:
-            return {}
-
-        ctxt = self.neutron_ctxt()
-
-        if self.plugin == 'ovs':
-            ctxt.update(self.ovs_ctxt())
-        elif self.plugin == 'nvp':
-            ctxt.update(self.nvp_ctxt())
-
-        alchemy_flags = config('neutron-alchemy-flags')
-        if alchemy_flags:
-            flags = config_flags_parser(alchemy_flags)
-            ctxt['neutron_alchemy_flags'] = flags
-
-        self._save_flag_file()
-        return ctxt
-
-
-class OSConfigFlagContext(OSContextGenerator):
-
-        """
-        Responsible for adding user-defined config-flags in charm config to a
-        template context.
-
-        NOTE: the value of config-flags may be a comma-separated list of
-              key=value pairs and some Openstack config files support
-              comma-separated lists as values.
-        """
-
-        def __call__(self):
-            config_flags = config('config-flags')
-            if not config_flags:
-                return {}
-
-            flags = config_flags_parser(config_flags)
-            return {'user_config_flags': flags}
-
-
-class SubordinateConfigContext(OSContextGenerator):
-
-    """
-    Responsible for inspecting relations to subordinates that
-    may be exporting required config via a json blob.
-
-    The subordinate interface allows subordinates to export their
-    configuration requirements to the principle for multiple config
-    files and multiple serivces.  Ie, a subordinate that has interfaces
-    to both glance and nova may export to following yaml blob as json:
-
-        glance:
-            /etc/glance/glance-api.conf:
-                sections:
-                    DEFAULT:
-                        - [key1, value1]
-            /etc/glance/glance-registry.conf:
-                    MYSECTION:
-                        - [key2, value2]
-        nova:
-            /etc/nova/nova.conf:
-                sections:
-                    DEFAULT:
-                        - [key3, value3]
-
-
-    It is then up to the principle charms to subscribe this context to
-    the service+config file it is interestd in.  Configuration data will
-    be available in the template context, in glance's case, as:
-        ctxt = {
-            ... other context ...
-            'subordinate_config': {
-                'DEFAULT': {
-                    'key1': 'value1',
-                },
-                'MYSECTION': {
-                    'key2': 'value2',
-                },
-            }
-        }
-
-    """
-
-    def __init__(self, service, config_file, interface):
-        """
-        :param service     : Service name key to query in any subordinate
-                             data found
-        :param config_file : Service's config file to query sections
-        :param interface   : Subordinate interface to inspect
-        """
-        self.service = service
-        self.config_file = config_file
-        self.interface = interface
-
-    def __call__(self):
-        ctxt = {}
-        for rid in relation_ids(self.interface):
-            for unit in related_units(rid):
-                sub_config = relation_get('subordinate_configuration',
-                                          rid=rid, unit=unit)
-                if sub_config and sub_config != '':
-                    try:
-                        sub_config = json.loads(sub_config)
-                    except:
-                        log('Could not parse JSON from subordinate_config '
-                            'setting from %s' % rid, level=ERROR)
-                        continue
-
-                    if self.service not in sub_config:
-                        log('Found subordinate_config on %s but it contained'
-                            'nothing for %s service' % (rid, self.service))
-                        continue
-
-                    sub_config = sub_config[self.service]
-                    if self.config_file not in sub_config:
-                        log('Found subordinate_config on %s but it contained'
-                            'nothing for %s' % (rid, self.config_file))
-                        continue
-
-                    sub_config = sub_config[self.config_file]
-                    for k, v in sub_config.iteritems():
-                        ctxt[k] = v
-
-        if not ctxt:
-            ctxt['sections'] = {}
-
-        return ctxt
-
-
-class SyslogContext(OSContextGenerator):
-
-    def __call__(self):
-        ctxt = {
-            'use_syslog': config('use-syslog')
-        }
-        return ctxt

=== removed file 'hooks/charmhelpers/contrib/openstack/neutron.py'
--- hooks/charmhelpers/contrib/openstack/neutron.py	2014-05-09 20:11:59 +0000
+++ hooks/charmhelpers/contrib/openstack/neutron.py	1970-01-01 00:00:00 +0000
@@ -1,171 +0,0 @@
-# Various utilies for dealing with Neutron and the renaming from Quantum.
-
-from subprocess import check_output
-
-from charmhelpers.core.hookenv import (
-    config,
-    log,
-    ERROR,
-)
-
-from charmhelpers.contrib.openstack.utils import os_release
-
-
-def headers_package():
-    """Ensures correct linux-headers for running kernel are installed,
-    for building DKMS package"""
-    kver = check_output(['uname', '-r']).strip()
-    return 'linux-headers-%s' % kver
-
-QUANTUM_CONF_DIR = '/etc/quantum'
-
-
-def kernel_version():
-    """ Retrieve the current major kernel version as a tuple e.g. (3, 13) """
-    kver = check_output(['uname', '-r']).strip()
-    kver = kver.split('.')
-    return (int(kver[0]), int(kver[1]))
-
-
-def determine_dkms_package():
-    """ Determine which DKMS package should be used based on kernel version """
-    # NOTE: 3.13 kernels have support for GRE and VXLAN native
-    if kernel_version() >= (3, 13):
-        return []
-    else:
-        return ['openvswitch-datapath-dkms']
-
-
-# legacy
-
-
-def quantum_plugins():
-    from charmhelpers.contrib.openstack import context
-    return {
-        'ovs': {
-            'config': '/etc/quantum/plugins/openvswitch/'
-                      'ovs_quantum_plugin.ini',
-            'driver': 'quantum.plugins.openvswitch.ovs_quantum_plugin.'
-                      'OVSQuantumPluginV2',
-            'contexts': [
-                context.SharedDBContext(user=config('neutron-database-user'),
-                                        database=config('neutron-database'),
-                                        relation_prefix='neutron',
-                                        ssl_dir=QUANTUM_CONF_DIR)],
-            'services': ['quantum-plugin-openvswitch-agent'],
-            'packages': [[headers_package()] + determine_dkms_package(),
-                         ['quantum-plugin-openvswitch-agent']],
-            'server_packages': ['quantum-server',
-                                'quantum-plugin-openvswitch'],
-            'server_services': ['quantum-server']
-        },
-        'nvp': {
-            'config': '/etc/quantum/plugins/nicira/nvp.ini',
-            'driver': 'quantum.plugins.nicira.nicira_nvp_plugin.'
-                      'QuantumPlugin.NvpPluginV2',
-            'contexts': [
-                context.SharedDBContext(user=config('neutron-database-user'),
-                                        database=config('neutron-database'),
-                                        relation_prefix='neutron',
-                                        ssl_dir=QUANTUM_CONF_DIR)],
-            'services': [],
-            'packages': [],
-            'server_packages': ['quantum-server',
-                                'quantum-plugin-nicira'],
-            'server_services': ['quantum-server']
-        }
-    }
-
-NEUTRON_CONF_DIR = '/etc/neutron'
-
-
-def neutron_plugins():
-    from charmhelpers.contrib.openstack import context
-    release = os_release('nova-common')
-    plugins = {
-        'ovs': {
-            'config': '/etc/neutron/plugins/openvswitch/'
-                      'ovs_neutron_plugin.ini',
-            'driver': 'neutron.plugins.openvswitch.ovs_neutron_plugin.'
-                      'OVSNeutronPluginV2',
-            'contexts': [
-                context.SharedDBContext(user=config('neutron-database-user'),
-                                        database=config('neutron-database'),
-                                        relation_prefix='neutron',
-                                        ssl_dir=NEUTRON_CONF_DIR)],
-            'services': ['neutron-plugin-openvswitch-agent'],
-            'packages': [[headers_package()] + determine_dkms_package(),
-                         ['neutron-plugin-openvswitch-agent']],
-            'server_packages': ['neutron-server',
-                                'neutron-plugin-openvswitch'],
-            'server_services': ['neutron-server']
-        },
-        'nvp': {
-            'config': '/etc/neutron/plugins/nicira/nvp.ini',
-            'driver': 'neutron.plugins.nicira.nicira_nvp_plugin.'
-                      'NeutronPlugin.NvpPluginV2',
-            'contexts': [
-                context.SharedDBContext(user=config('neutron-database-user'),
-                                        database=config('neutron-database'),
-                                        relation_prefix='neutron',
-                                        ssl_dir=NEUTRON_CONF_DIR)],
-            'services': [],
-            'packages': [],
-            'server_packages': ['neutron-server',
-                                'neutron-plugin-nicira'],
-            'server_services': ['neutron-server']
-        }
-    }
-    # NOTE: patch in ml2 plugin for icehouse onwards
-    if release >= 'icehouse':
-        plugins['ovs']['config'] = '/etc/neutron/plugins/ml2/ml2_conf.ini'
-        plugins['ovs']['driver'] = 'neutron.plugins.ml2.plugin.Ml2Plugin'
-        plugins['ovs']['server_packages'] = ['neutron-server',
-                                             'neutron-plugin-ml2']
-    return plugins
-
-
-def neutron_plugin_attribute(plugin, attr, net_manager=None):
-    manager = net_manager or network_manager()
-    if manager == 'quantum':
-        plugins = quantum_plugins()
-    elif manager == 'neutron':
-        plugins = neutron_plugins()
-    else:
-        log('Error: Network manager does not support plugins.')
-        raise Exception
-
-    try:
-        _plugin = plugins[plugin]
-    except KeyError:
-        log('Unrecognised plugin for %s: %s' % (manager, plugin), level=ERROR)
-        raise Exception
-
-    try:
-        return _plugin[attr]
-    except KeyError:
-        return None
-
-
-def network_manager():
-    '''
-    Deals with the renaming of Quantum to Neutron in H and any situations
-    that require compatability (eg, deploying H with network-manager=quantum,
-    upgrading from G).
-    '''
-    release = os_release('nova-common')
-    manager = config('network-manager').lower()
-
-    if manager not in ['quantum', 'neutron']:
-        return manager
-
-    if release in ['essex']:
-        # E does not support neutron
-        log('Neutron networking not supported in Essex.', level=ERROR)
-        raise Exception
-    elif release in ['folsom', 'grizzly']:
-        # neutron is named quantum in F and G
-        return 'quantum'
-    else:
-        # ensure accurate naming for all releases post-H
-        return 'neutron'

=== removed directory 'hooks/charmhelpers/contrib/openstack/templates'
=== removed file 'hooks/charmhelpers/contrib/openstack/templates/__init__.py'
--- hooks/charmhelpers/contrib/openstack/templates/__init__.py	2013-11-26 17:12:54 +0000
+++ hooks/charmhelpers/contrib/openstack/templates/__init__.py	1970-01-01 00:00:00 +0000
@@ -1,2 +0,0 @@
-# dummy __init__.py to fool syncer into thinking this is a syncable python
-# module

=== removed file 'hooks/charmhelpers/contrib/openstack/templating.py'
--- hooks/charmhelpers/contrib/openstack/templating.py	2013-11-26 17:12:54 +0000
+++ hooks/charmhelpers/contrib/openstack/templating.py	1970-01-01 00:00:00 +0000
@@ -1,280 +0,0 @@
-import os
-
-from charmhelpers.fetch import apt_install
-
-from charmhelpers.core.hookenv import (
-    log,
-    ERROR,
-    INFO
-)
-
-from charmhelpers.contrib.openstack.utils import OPENSTACK_CODENAMES
-
-try:
-    from jinja2 import FileSystemLoader, ChoiceLoader, Environment, exceptions
-except ImportError:
-    # python-jinja2 may not be installed yet, or we're running unittests.
-    FileSystemLoader = ChoiceLoader = Environment = exceptions = None
-
-
-class OSConfigException(Exception):
-    pass
-
-
-def get_loader(templates_dir, os_release):
-    """
-    Create a jinja2.ChoiceLoader containing template dirs up to
-    and including os_release.  If directory template directory
-    is missing at templates_dir, it will be omitted from the loader.
-    templates_dir is added to the bottom of the search list as a base
-    loading dir.
-
-    A charm may also ship a templates dir with this module
-    and it will be appended to the bottom of the search list, eg:
-    hooks/charmhelpers/contrib/openstack/templates.
-
-    :param templates_dir: str: Base template directory containing release
-                               sub-directories.
-    :param os_release   : str: OpenStack release codename to construct template
-                               loader.
-
-    :returns            : jinja2.ChoiceLoader constructed with a list of
-                          jinja2.FilesystemLoaders, ordered in descending
-                          order by OpenStack release.
-    """
-    tmpl_dirs = [(rel, os.path.join(templates_dir, rel))
-                 for rel in OPENSTACK_CODENAMES.itervalues()]
-
-    if not os.path.isdir(templates_dir):
-        log('Templates directory not found @ %s.' % templates_dir,
-            level=ERROR)
-        raise OSConfigException
-
-    # the bottom contains tempaltes_dir and possibly a common templates dir
-    # shipped with the helper.
-    loaders = [FileSystemLoader(templates_dir)]
-    helper_templates = os.path.join(os.path.dirname(__file__), 'templates')
-    if os.path.isdir(helper_templates):
-        loaders.append(FileSystemLoader(helper_templates))
-
-    for rel, tmpl_dir in tmpl_dirs:
-        if os.path.isdir(tmpl_dir):
-            loaders.insert(0, FileSystemLoader(tmpl_dir))
-        if rel == os_release:
-            break
-    log('Creating choice loader with dirs: %s' %
-        [l.searchpath for l in loaders], level=INFO)
-    return ChoiceLoader(loaders)
-
-
-class OSConfigTemplate(object):
-    """
-    Associates a config file template with a list of context generators.
-    Responsible for constructing a template context based on those generators.
-    """
-    def __init__(self, config_file, contexts):
-        self.config_file = config_file
-
-        if hasattr(contexts, '__call__'):
-            self.contexts = [contexts]
-        else:
-            self.contexts = contexts
-
-        self._complete_contexts = []
-
-    def context(self):
-        ctxt = {}
-        for context in self.contexts:
-            _ctxt = context()
-            if _ctxt:
-                ctxt.update(_ctxt)
-                # track interfaces for every complete context.
-                [self._complete_contexts.append(interface)
-                 for interface in context.interfaces
-                 if interface not in self._complete_contexts]
-        return ctxt
-
-    def complete_contexts(self):
-        '''
-        Return a list of interfaces that have atisfied contexts.
-        '''
-        if self._complete_contexts:
-            return self._complete_contexts
-        self.context()
-        return self._complete_contexts
-
-
-class OSConfigRenderer(object):
-    """
-    This class provides a common templating system to be used by OpenStack
-    charms.  It is intended to help charms share common code and templates,
-    and ease the burden of managing config templates across multiple OpenStack
-    releases.
-
-    Basic usage:
-        # import some common context generates from charmhelpers
-        from charmhelpers.contrib.openstack import context
-
-        # Create a renderer object for a specific OS release.
-        configs = OSConfigRenderer(templates_dir='/tmp/templates',
-                                   openstack_release='folsom')
-        # register some config files with context generators.
-        configs.register(config_file='/etc/nova/nova.conf',
-                         contexts=[context.SharedDBContext(),
-                                   context.AMQPContext()])
-        configs.register(config_file='/etc/nova/api-paste.ini',
-                         contexts=[context.IdentityServiceContext()])
-        configs.register(config_file='/etc/haproxy/haproxy.conf',
-                         contexts=[context.HAProxyContext()])
-        # write out a single config
-        configs.write('/etc/nova/nova.conf')
-        # write out all registered configs
-        configs.write_all()
-
-    Details:
-
-    OpenStack Releases and template loading
-    ---------------------------------------
-    When the object is instantiated, it is associated with a specific OS
-    release.  This dictates how the template loader will be constructed.
-
-    The constructed loader attempts to load the template from several places
-    in the following order:
-        - from the most recent OS release-specific template dir (if one exists)
-        - the base templates_dir
-        - a template directory shipped in the charm with this helper file.
-
-
-    For the example above, '/tmp/templates' contains the following structure:
-        /tmp/templates/nova.conf
-        /tmp/templates/api-paste.ini
-        /tmp/templates/grizzly/api-paste.ini
-        /tmp/templates/havana/api-paste.ini
-
-    Since it was registered with the grizzly release, it first seraches
-    the grizzly directory for nova.conf, then the templates dir.
-
-    When writing api-paste.ini, it will find the template in the grizzly
-    directory.
-
-    If the object were created with folsom, it would fall back to the
-    base templates dir for its api-paste.ini template.
-
-    This system should help manage changes in config files through
-    openstack releases, allowing charms to fall back to the most recently
-    updated config template for a given release
-
-    The haproxy.conf, since it is not shipped in the templates dir, will
-    be loaded from the module directory's template directory, eg
-    $CHARM/hooks/charmhelpers/contrib/openstack/templates.  This allows
-    us to ship common templates (haproxy, apache) with the helpers.
-
-    Context generators
-    ---------------------------------------
-    Context generators are used to generate template contexts during hook
-    execution.  Doing so may require inspecting service relations, charm
-    config, etc.  When registered, a config file is associated with a list
-    of generators.  When a template is rendered and written, all context
-    generates are called in a chain to generate the context dictionary
-    passed to the jinja2 template. See context.py for more info.
-    """
-    def __init__(self, templates_dir, openstack_release):
-        if not os.path.isdir(templates_dir):
-            log('Could not locate templates dir %s' % templates_dir,
-                level=ERROR)
-            raise OSConfigException
-
-        self.templates_dir = templates_dir
-        self.openstack_release = openstack_release
-        self.templates = {}
-        self._tmpl_env = None
-
-        if None in [Environment, ChoiceLoader, FileSystemLoader]:
-            # if this code is running, the object is created pre-install hook.
-            # jinja2 shouldn't get touched until the module is reloaded on next
-            # hook execution, with proper jinja2 bits successfully imported.
-            apt_install('python-jinja2')
-
-    def register(self, config_file, contexts):
-        """
-        Register a config file with a list of context generators to be called
-        during rendering.
-        """
-        self.templates[config_file] = OSConfigTemplate(config_file=config_file,
-                                                       contexts=contexts)
-        log('Registered config file: %s' % config_file, level=INFO)
-
-    def _get_tmpl_env(self):
-        if not self._tmpl_env:
-            loader = get_loader(self.templates_dir, self.openstack_release)
-            self._tmpl_env = Environment(loader=loader)
-
-    def _get_template(self, template):
-        self._get_tmpl_env()
-        template = self._tmpl_env.get_template(template)
-        log('Loaded template from %s' % template.filename, level=INFO)
-        return template
-
-    def render(self, config_file):
-        if config_file not in self.templates:
-            log('Config not registered: %s' % config_file, level=ERROR)
-            raise OSConfigException
-        ctxt = self.templates[config_file].context()
-
-        _tmpl = os.path.basename(config_file)
-        try:
-            template = self._get_template(_tmpl)
-        except exceptions.TemplateNotFound:
-            # if no template is found with basename, try looking for it
-            # using a munged full path, eg:
-            #   /etc/apache2/apache2.conf -> etc_apache2_apache2.conf
-            _tmpl = '_'.join(config_file.split('/')[1:])
-            try:
-                template = self._get_template(_tmpl)
-            except exceptions.TemplateNotFound as e:
-                log('Could not load template from %s by %s or %s.' %
-                    (self.templates_dir, os.path.basename(config_file), _tmpl),
-                    level=ERROR)
-                raise e
-
-        log('Rendering from template: %s' % _tmpl, level=INFO)
-        return template.render(ctxt)
-
-    def write(self, config_file):
-        """
-        Write a single config file, raises if config file is not registered.
-        """
-        if config_file not in self.templates:
-            log('Config not registered: %s' % config_file, level=ERROR)
-            raise OSConfigException
-
-        _out = self.render(config_file)
-
-        with open(config_file, 'wb') as out:
-            out.write(_out)
-
-        log('Wrote template %s.' % config_file, level=INFO)
-
-    def write_all(self):
-        """
-        Write out all registered config files.
-        """
-        [self.write(k) for k in self.templates.iterkeys()]
-
-    def set_release(self, openstack_release):
-        """
-        Resets the template environment and generates a new template loader
-        based on a the new openstack release.
-        """
-        self._tmpl_env = None
-        self.openstack_release = openstack_release
-        self._get_tmpl_env()
-
-    def complete_contexts(self):
-        '''
-        Returns a list of context interfaces that yield a complete context.
-        '''
-        interfaces = []
-        [interfaces.extend(i.complete_contexts())
-         for i in self.templates.itervalues()]
-        return interfaces

=== removed file 'hooks/charmhelpers/contrib/openstack/utils.py'
--- hooks/charmhelpers/contrib/openstack/utils.py	2014-05-09 20:11:59 +0000
+++ hooks/charmhelpers/contrib/openstack/utils.py	1970-01-01 00:00:00 +0000
@@ -1,450 +0,0 @@
-#!/usr/bin/python
-
-# Common python helper functions used for OpenStack charms.
-from collections import OrderedDict
-
-import apt_pkg as apt
-import subprocess
-import os
-import socket
-import sys
-
-from charmhelpers.core.hookenv import (
-    config,
-    log as juju_log,
-    charm_dir,
-    ERROR,
-    INFO
-)
-
-from charmhelpers.contrib.storage.linux.lvm import (
-    deactivate_lvm_volume_group,
-    is_lvm_physical_volume,
-    remove_lvm_physical_volume,
-)
-
-from charmhelpers.core.host import lsb_release, mounts, umount
-from charmhelpers.fetch import apt_install
-from charmhelpers.contrib.storage.linux.utils import is_block_device, zap_disk
-from charmhelpers.contrib.storage.linux.loopback import ensure_loopback_device
-
-CLOUD_ARCHIVE_URL = "http://ubuntu-cloud.archive.canonical.com/ubuntu"
-CLOUD_ARCHIVE_KEY_ID = '5EDB1B62EC4926EA'
-
-DISTRO_PROPOSED = ('deb http://archive.ubuntu.com/ubuntu/ %s-proposed '
-                   'restricted main multiverse universe')
-
-
-UBUNTU_OPENSTACK_RELEASE = OrderedDict([
-    ('oneiric', 'diablo'),
-    ('precise', 'essex'),
-    ('quantal', 'folsom'),
-    ('raring', 'grizzly'),
-    ('saucy', 'havana'),
-    ('trusty', 'icehouse')
-])
-
-
-OPENSTACK_CODENAMES = OrderedDict([
-    ('2011.2', 'diablo'),
-    ('2012.1', 'essex'),
-    ('2012.2', 'folsom'),
-    ('2013.1', 'grizzly'),
-    ('2013.2', 'havana'),
-    ('2014.1', 'icehouse'),
-])
-
-# The ugly duckling
-SWIFT_CODENAMES = OrderedDict([
-    ('1.4.3', 'diablo'),
-    ('1.4.8', 'essex'),
-    ('1.7.4', 'folsom'),
-    ('1.8.0', 'grizzly'),
-    ('1.7.7', 'grizzly'),
-    ('1.7.6', 'grizzly'),
-    ('1.10.0', 'havana'),
-    ('1.9.1', 'havana'),
-    ('1.9.0', 'havana'),
-    ('1.13.1', 'icehouse'),
-    ('1.13.0', 'icehouse'),
-    ('1.12.0', 'icehouse'),
-    ('1.11.0', 'icehouse'),
-])
-
-DEFAULT_LOOPBACK_SIZE = '5G'
-
-
-def error_out(msg):
-    juju_log("FATAL ERROR: %s" % msg, level='ERROR')
-    sys.exit(1)
-
-
-def get_os_codename_install_source(src):
-    '''Derive OpenStack release codename from a given installation source.'''
-    ubuntu_rel = lsb_release()['DISTRIB_CODENAME']
-    rel = ''
-    if src in ['distro', 'distro-proposed']:
-        try:
-            rel = UBUNTU_OPENSTACK_RELEASE[ubuntu_rel]
-        except KeyError:
-            e = 'Could not derive openstack release for '\
-                'this Ubuntu release: %s' % ubuntu_rel
-            error_out(e)
-        return rel
-
-    if src.startswith('cloud:'):
-        ca_rel = src.split(':')[1]
-        ca_rel = ca_rel.split('%s-' % ubuntu_rel)[1].split('/')[0]
-        return ca_rel
-
-    # Best guess match based on deb string provided
-    if src.startswith('deb') or src.startswith('ppa'):
-        for k, v in OPENSTACK_CODENAMES.iteritems():
-            if v in src:
-                return v
-
-
-def get_os_version_install_source(src):
-    codename = get_os_codename_install_source(src)
-    return get_os_version_codename(codename)
-
-
-def get_os_codename_version(vers):
-    '''Determine OpenStack codename from version number.'''
-    try:
-        return OPENSTACK_CODENAMES[vers]
-    except KeyError:
-        e = 'Could not determine OpenStack codename for version %s' % vers
-        error_out(e)
-
-
-def get_os_version_codename(codename):
-    '''Determine OpenStack version number from codename.'''
-    for k, v in OPENSTACK_CODENAMES.iteritems():
-        if v == codename:
-            return k
-    e = 'Could not derive OpenStack version for '\
-        'codename: %s' % codename
-    error_out(e)
-
-
-def get_os_codename_package(package, fatal=True):
-    '''Derive OpenStack release codename from an installed package.'''
-    apt.init()
-    cache = apt.Cache()
-
-    try:
-        pkg = cache[package]
-    except:
-        if not fatal:
-            return None
-        # the package is unknown to the current apt cache.
-        e = 'Could not determine version of package with no installation '\
-            'candidate: %s' % package
-        error_out(e)
-
-    if not pkg.current_ver:
-        if not fatal:
-            return None
-        # package is known, but no version is currently installed.
-        e = 'Could not determine version of uninstalled package: %s' % package
-        error_out(e)
-
-    vers = apt.upstream_version(pkg.current_ver.ver_str)
-
-    try:
-        if 'swift' in pkg.name:
-            swift_vers = vers[:5]
-            if swift_vers not in SWIFT_CODENAMES:
-                # Deal with 1.10.0 upward
-                swift_vers = vers[:6]
-            return SWIFT_CODENAMES[swift_vers]
-        else:
-            vers = vers[:6]
-            return OPENSTACK_CODENAMES[vers]
-    except KeyError:
-        e = 'Could not determine OpenStack codename for version %s' % vers
-        error_out(e)
-
-
-def get_os_version_package(pkg, fatal=True):
-    '''Derive OpenStack version number from an installed package.'''
-    codename = get_os_codename_package(pkg, fatal=fatal)
-
-    if not codename:
-        return None
-
-    if 'swift' in pkg:
-        vers_map = SWIFT_CODENAMES
-    else:
-        vers_map = OPENSTACK_CODENAMES
-
-    for version, cname in vers_map.iteritems():
-        if cname == codename:
-            return version
-    #e = "Could not determine OpenStack version for package: %s" % pkg
-    #error_out(e)
-
-
-os_rel = None
-
-
-def os_release(package, base='essex'):
-    '''
-    Returns OpenStack release codename from a cached global.
-    If the codename can not be determined from either an installed package or
-    the installation source, the earliest release supported by the charm should
-    be returned.
-    '''
-    global os_rel
-    if os_rel:
-        return os_rel
-    os_rel = (get_os_codename_package(package, fatal=False) or
-              get_os_codename_install_source(config('openstack-origin')) or
-              base)
-    return os_rel
-
-
-def import_key(keyid):
-    cmd = "apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 " \
-          "--recv-keys %s" % keyid
-    try:
-        subprocess.check_call(cmd.split(' '))
-    except subprocess.CalledProcessError:
-        error_out("Error importing repo key %s" % keyid)
-
-
-def configure_installation_source(rel):
-    '''Configure apt installation source.'''
-    if rel == 'distro':
-        return
-    elif rel == 'distro-proposed':
-        ubuntu_rel = lsb_release()['DISTRIB_CODENAME']
-        with open('/etc/apt/sources.list.d/juju_deb.list', 'w') as f:
-            f.write(DISTRO_PROPOSED % ubuntu_rel)
-    elif rel[:4] == "ppa:":
-        src = rel
-        subprocess.check_call(["add-apt-repository", "-y", src])
-    elif rel[:3] == "deb":
-        l = len(rel.split('|'))
-        if l == 2:
-            src, key = rel.split('|')
-            juju_log("Importing PPA key from keyserver for %s" % src)
-            import_key(key)
-        elif l == 1:
-            src = rel
-        with open('/etc/apt/sources.list.d/juju_deb.list', 'w') as f:
-            f.write(src)
-    elif rel[:6] == 'cloud:':
-        ubuntu_rel = lsb_release()['DISTRIB_CODENAME']
-        rel = rel.split(':')[1]
-        u_rel = rel.split('-')[0]
-        ca_rel = rel.split('-')[1]
-
-        if u_rel != ubuntu_rel:
-            e = 'Cannot install from Cloud Archive pocket %s on this Ubuntu '\
-                'version (%s)' % (ca_rel, ubuntu_rel)
-            error_out(e)
-
-        if 'staging' in ca_rel:
-            # staging is just a regular PPA.
-            os_rel = ca_rel.split('/')[0]
-            ppa = 'ppa:ubuntu-cloud-archive/%s-staging' % os_rel
-            cmd = 'add-apt-repository -y %s' % ppa
-            subprocess.check_call(cmd.split(' '))
-            return
-
-        # map charm config options to actual archive pockets.
-        pockets = {
-            'folsom': 'precise-updates/folsom',
-            'folsom/updates': 'precise-updates/folsom',
-            'folsom/proposed': 'precise-proposed/folsom',
-            'grizzly': 'precise-updates/grizzly',
-            'grizzly/updates': 'precise-updates/grizzly',
-            'grizzly/proposed': 'precise-proposed/grizzly',
-            'havana': 'precise-updates/havana',
-            'havana/updates': 'precise-updates/havana',
-            'havana/proposed': 'precise-proposed/havana',
-            'icehouse': 'precise-updates/icehouse',
-            'icehouse/updates': 'precise-updates/icehouse',
-            'icehouse/proposed': 'precise-proposed/icehouse',
-        }
-
-        try:
-            pocket = pockets[ca_rel]
-        except KeyError:
-            e = 'Invalid Cloud Archive release specified: %s' % rel
-            error_out(e)
-
-        src = "deb %s %s main" % (CLOUD_ARCHIVE_URL, pocket)
-        apt_install('ubuntu-cloud-keyring', fatal=True)
-
-        with open('/etc/apt/sources.list.d/cloud-archive.list', 'w') as f:
-            f.write(src)
-    else:
-        error_out("Invalid openstack-release specified: %s" % rel)
-
-
-def save_script_rc(script_path="scripts/scriptrc", **env_vars):
-    """
-    Write an rc file in the charm-delivered directory containing
-    exported environment variables provided by env_vars. Any charm scripts run
-    outside the juju hook environment can source this scriptrc to obtain
-    updated config information necessary to perform health checks or
-    service changes.
-    """
-    juju_rc_path = "%s/%s" % (charm_dir(), script_path)
-    if not os.path.exists(os.path.dirname(juju_rc_path)):
-        os.mkdir(os.path.dirname(juju_rc_path))
-    with open(juju_rc_path, 'wb') as rc_script:
-        rc_script.write(
-            "#!/bin/bash\n")
-        [rc_script.write('export %s=%s\n' % (u, p))
-         for u, p in env_vars.iteritems() if u != "script_path"]
-
-
-def openstack_upgrade_available(package):
-    """
-    Determines if an OpenStack upgrade is available from installation
-    source, based on version of installed package.
-
-    :param package: str: Name of installed package.
-
-    :returns: bool:    : Returns True if configured installation source offers
-                         a newer version of package.
-
-    """
-
-    src = config('openstack-origin')
-    cur_vers = get_os_version_package(package)
-    available_vers = get_os_version_install_source(src)
-    apt.init()
-    return apt.version_compare(available_vers, cur_vers) == 1
-
-
-def ensure_block_device(block_device):
-    '''
-    Confirm block_device, create as loopback if necessary.
-
-    :param block_device: str: Full path of block device to ensure.
-
-    :returns: str: Full path of ensured block device.
-    '''
-    _none = ['None', 'none', None]
-    if (block_device in _none):
-        error_out('prepare_storage(): Missing required input: '
-                  'block_device=%s.' % block_device, level=ERROR)
-
-    if block_device.startswith('/dev/'):
-        bdev = block_device
-    elif block_device.startswith('/'):
-        _bd = block_device.split('|')
-        if len(_bd) == 2:
-            bdev, size = _bd
-        else:
-            bdev = block_device
-            size = DEFAULT_LOOPBACK_SIZE
-        bdev = ensure_loopback_device(bdev, size)
-    else:
-        bdev = '/dev/%s' % block_device
-
-    if not is_block_device(bdev):
-        error_out('Failed to locate valid block device at %s' % bdev,
-                  level=ERROR)
-
-    return bdev
-
-
-def clean_storage(block_device):
-    '''
-    Ensures a block device is clean.  That is:
-        - unmounted
-        - any lvm volume groups are deactivated
-        - any lvm physical device signatures removed
-        - partition table wiped
-
-    :param block_device: str: Full path to block device to clean.
-    '''
-    for mp, d in mounts():
-        if d == block_device:
-            juju_log('clean_storage(): %s is mounted @ %s, unmounting.' %
-                     (d, mp), level=INFO)
-            umount(mp, persist=True)
-
-    if is_lvm_physical_volume(block_device):
-        deactivate_lvm_volume_group(block_device)
-        remove_lvm_physical_volume(block_device)
-    else:
-        zap_disk(block_device)
-
-
-def is_ip(address):
-    """
-    Returns True if address is a valid IP address.
-    """
-    try:
-        # Test to see if already an IPv4 address
-        socket.inet_aton(address)
-        return True
-    except socket.error:
-        return False
-
-
-def ns_query(address):
-    try:
-        import dns.resolver
-    except ImportError:
-        apt_install('python-dnspython')
-        import dns.resolver
-
-    if isinstance(address, dns.name.Name):
-        rtype = 'PTR'
-    elif isinstance(address, basestring):
-        rtype = 'A'
-    else:
-        return None
-
-    answers = dns.resolver.query(address, rtype)
-    if answers:
-        return str(answers[0])
-    return None
-
-
-def get_host_ip(hostname):
-    """
-    Resolves the IP for a given hostname, or returns
-    the input if it is already an IP.
-    """
-    if is_ip(hostname):
-        return hostname
-
-    return ns_query(hostname)
-
-
-def get_hostname(address, fqdn=True):
-    """
-    Resolves hostname for given IP, or returns the input
-    if it is already a hostname.
-    """
-    if is_ip(address):
-        try:
-            import dns.reversename
-        except ImportError:
-            apt_install('python-dnspython')
-            import dns.reversename
-
-        rev = dns.reversename.from_address(address)
-        result = ns_query(rev)
-        if not result:
-            return None
-    else:
-        result = address
-
-    if fqdn:
-        # strip trailing .
-        if result.endswith('.'):
-            return result[:-1]
-        else:
-            return result
-    else:
-        return result.split('.')[0]

=== removed directory 'hooks/charmhelpers/contrib/peerstorage'
=== removed file 'hooks/charmhelpers/contrib/peerstorage/__init__.py'
--- hooks/charmhelpers/contrib/peerstorage/__init__.py	2014-05-09 20:11:59 +0000
+++ hooks/charmhelpers/contrib/peerstorage/__init__.py	1970-01-01 00:00:00 +0000
@@ -1,83 +0,0 @@
-from charmhelpers.core.hookenv import (
-    relation_ids,
-    relation_get,
-    local_unit,
-    relation_set,
-)
-
-"""
-This helper provides functions to support use of a peer relation
-for basic key/value storage, with the added benefit that all storage
-can be replicated across peer units, so this is really useful for
-services that issue usernames/passwords to remote services.
-
-def shared_db_changed()
-    # Only the lead unit should create passwords
-    if not is_leader():
-        return
-    username = relation_get('username')
-    key = '{}.password'.format(username)
-    # Attempt to retrieve any existing password for this user
-    password = peer_retrieve(key)
-    if password is None:
-        # New user, create password and store
-        password = pwgen(length=64)
-        peer_store(key, password)
-    create_access(username, password)
-    relation_set(password=password)
-
-
-def cluster_changed()
-    # Echo any relation data other that *-address
-    # back onto the peer relation so all units have
-    # all *.password keys stored on their local relation
-    # for later retrieval.
-    peer_echo()
-
-"""
-
-
-def peer_retrieve(key, relation_name='cluster'):
-    """ Retrieve a named key from peer relation relation_name """
-    cluster_rels = relation_ids(relation_name)
-    if len(cluster_rels) > 0:
-        cluster_rid = cluster_rels[0]
-        return relation_get(attribute=key, rid=cluster_rid,
-                            unit=local_unit())
-    else:
-        raise ValueError('Unable to detect'
-                         'peer relation {}'.format(relation_name))
-
-
-def peer_store(key, value, relation_name='cluster'):
-    """ Store the key/value pair on the named peer relation relation_name """
-    cluster_rels = relation_ids(relation_name)
-    if len(cluster_rels) > 0:
-        cluster_rid = cluster_rels[0]
-        relation_set(relation_id=cluster_rid,
-                     relation_settings={key: value})
-    else:
-        raise ValueError('Unable to detect '
-                         'peer relation {}'.format(relation_name))
-
-
-def peer_echo(includes=None):
-    """Echo filtered attributes back onto the same relation for storage
-
-    Note that this helper must only be called within a peer relation
-    changed hook
-    """
-    rdata = relation_get()
-    echo_data = {}
-    if includes is None:
-        echo_data = rdata.copy()
-        for ex in ['private-address', 'public-address']:
-            if ex in echo_data:
-                echo_data.pop(ex)
-    else:
-        for attribute, value in rdata.iteritems():
-            for include in includes:
-                if include in attribute:
-                    echo_data[attribute] = value
-    if len(echo_data) > 0:
-        relation_set(relation_settings=echo_data)

=== removed directory 'hooks/charmhelpers/contrib/python'
=== removed file 'hooks/charmhelpers/contrib/python/__init__.py'
=== removed file 'hooks/charmhelpers/contrib/python/packages.py'
--- hooks/charmhelpers/contrib/python/packages.py	2014-05-09 20:11:59 +0000
+++ hooks/charmhelpers/contrib/python/packages.py	1970-01-01 00:00:00 +0000
@@ -1,76 +0,0 @@
-#!/usr/bin/env python
-# coding: utf-8
-
-__author__ = "Jorge Niedbalski <[e-mail]>"
-
-from charmhelpers.fetch import apt_install
-from charmhelpers.core.hookenv import log
-
-try:
-    from pip import main as pip_execute
-except ImportError:
-    apt_install('python-pip')
-    from pip import main as pip_execute
-
-
-def parse_options(given, available):
-    """Given a set of options, check if available"""
-    for key, value in given.items():
-        if key in available:
-            yield "--{0}={1}".format(key, value)
-
-
-def pip_install_requirements(requirements, **options):
-    """Install a requirements file """
-    command = ["install"]
-
-    available_options = ('proxy', 'src', 'log', )
-    for option in parse_options(options, available_options):
-        command.append(option)
-
-    command.append("-r {0}".format(requirements))
-    log("Installing from file: {} with options: {}".format(requirements,
-                                                           command))
-    pip_execute(command)
-
-
-def pip_install(package, fatal=False, **options):
-    """Install a python package"""
-    command = ["install"]
-
-    available_options = ('proxy', 'src', 'log', "index-url", )
-    for option in parse_options(options, available_options):
-        command.append(option)
-
-    if isinstance(package, list):
-        command.extend(package)
-    else:
-        command.append(package)
-
-    log("Installing {} package with options: {}".format(package,
-                                                        command))
-    pip_execute(command)
-
-
-def pip_uninstall(package, **options):
-    """Uninstall a python package"""
-    command = ["uninstall", "-q", "-y"]
-
-    available_options = ('proxy', 'log', )
-    for option in parse_options(options, available_options):
-        command.append(option)
-
-    if isinstance(package, list):
-        command.extend(package)
-    else:
-        command.append(package)
-
-    log("Uninstalling {} package with options: {}".format(package,
-                                                          command))
-    pip_execute(command)
-
-
-def pip_list():
-    """Returns the list of current python installed packages
-    """
-    return pip_execute(["list"])

=== removed file 'hooks/charmhelpers/contrib/python/version.py'
--- hooks/charmhelpers/contrib/python/version.py	2014-05-09 20:11:59 +0000
+++ hooks/charmhelpers/contrib/python/version.py	1970-01-01 00:00:00 +0000
@@ -1,18 +0,0 @@
-#!/usr/bin/env python
-# coding: utf-8
-
-__author__ = "Jorge Niedbalski <[e-mail]>"
-
-import sys
-
-
-def current_version():
-    """Current system python version"""
-    return sys.version_info
-
-
-def current_version_string():
-    """Current system python version as string major.minor.micro"""
-    return "{0}.{1}.{2}".format(sys.version_info.major,
-                                sys.version_info.minor,
-                                sys.version_info.micro)

=== removed directory 'hooks/charmhelpers/contrib/saltstack'
=== removed file 'hooks/charmhelpers/contrib/saltstack/__init__.py'
--- hooks/charmhelpers/contrib/saltstack/__init__.py	2013-11-26 17:12:54 +0000
+++ hooks/charmhelpers/contrib/saltstack/__init__.py	1970-01-01 00:00:00 +0000
@@ -1,102 +0,0 @@
-"""Charm Helpers saltstack - declare the state of your machines.
-
-This helper enables you to declare your machine state, rather than
-program it procedurally (and have to test each change to your procedures).
-Your install hook can be as simple as:
-
-{{{
-from charmhelpers.contrib.saltstack import (
-    install_salt_support,
-    update_machine_state,
-)
-
-
-def install():
-    install_salt_support()
-    update_machine_state('machine_states/dependencies.yaml')
-    update_machine_state('machine_states/installed.yaml')
-}}}
-
-and won't need to change (nor will its tests) when you change the machine
-state.
-
-It's using a python package called salt-minion which allows various formats for
-specifying resources, such as:
-
-{{{
-/srv/{{ basedir }}:
-    file.directory:
-        - group: ubunet
-        - user: ubunet
-        - require:
-            - user: ubunet
-        - recurse:
-            - user
-            - group
-
-ubunet:
-    group.present:
-        - gid: 1500
-    user.present:
-        - uid: 1500
-        - gid: 1500
-        - createhome: False
-        - require:
-            - group: ubunet
-}}}
-
-The docs for all the different state definitions are at:
-    http://docs.saltstack.com/ref/states/all/
-
-
-TODO:
-  * Add test helpers which will ensure that machine state definitions
-    are functionally (but not necessarily logically) correct (ie. getting
-    salt to parse all state defs.
-  * Add a link to a public bootstrap charm example / blogpost.
-  * Find a way to obviate the need to use the grains['charm_dir'] syntax
-    in templates.
-"""
-# Copyright 2013 Canonical Ltd.
-#
-# Authors:
-#  Charm Helpers Developers <[e-mail]>
-import subprocess
-
-import charmhelpers.contrib.templating.contexts
-import charmhelpers.core.host
-import charmhelpers.core.hookenv
-
-
-salt_grains_path = '/etc/salt/grains'
-
-
-def install_salt_support(from_ppa=True):
-    """Installs the salt-minion helper for machine state.
-
-    By default the salt-minion package is installed from
-    the saltstack PPA. If from_ppa is False you must ensure
-    that the salt-minion package is available in the apt cache.
-    """
-    if from_ppa:
-        subprocess.check_call([
-            '/usr/bin/add-apt-repository',
-            '--yes',
-            'ppa:saltstack/salt',
-        ])
-        subprocess.check_call(['/usr/bin/apt-get', 'update'])
-    # We install salt-common as salt-minion would run the salt-minion
-    # daemon.
-    charmhelpers.fetch.apt_install('salt-common')
-
-
-def update_machine_state(state_path):
-    """Update the machine state using the provided state declaration."""
-    charmhelpers.contrib.templating.contexts.juju_state_to_yaml(
-        salt_grains_path)
-    subprocess.check_call([
-        'salt-call',
-        '--local',
-        'state.template',
-        state_path,
-    ])

=== removed directory 'hooks/charmhelpers/contrib/ssl'
=== removed file 'hooks/charmhelpers/contrib/ssl/__init__.py'
--- hooks/charmhelpers/contrib/ssl/__init__.py	2013-11-26 17:12:54 +0000
+++ hooks/charmhelpers/contrib/ssl/__init__.py	1970-01-01 00:00:00 +0000
@@ -1,78 +0,0 @@
-import subprocess
-from charmhelpers.core import hookenv
-
-
-def generate_selfsigned(keyfile, certfile, keysize="1024", config=None, subject=None, cn=None):
-    """Generate selfsigned SSL keypair
-
-    You must provide one of the 3 optional arguments:
-    config, subject or cn
-    If more than one is provided the leftmost will be used
-
-    Arguments:
-    keyfile -- (required) full path to the keyfile to be created
-    certfile -- (required) full path to the certfile to be created
-    keysize -- (optional) SSL key length
-    config -- (optional) openssl configuration file
-    subject -- (optional) dictionary with SSL subject variables
-    cn -- (optional) cerfificate common name
-
-    Required keys in subject dict:
-    cn -- Common name (eq. FQDN)
-
-    Optional keys in subject dict
-    country -- Country Name (2 letter code)
-    state -- State or Province Name (full name)
-    locality -- Locality Name (eg, city)
-    organization -- Organization Name (eg, company)
-    organizational_unit -- Organizational Unit Name (eg, section)
-    email -- Email Address
-    """
-
-    cmd = []
-    if config:
-        cmd = ["/usr/bin/openssl", "req", "-new", "-newkey",
-               "rsa:{}".format(keysize), "-days", "365", "-nodes", "-x509",
-               "-keyout", keyfile,
-               "-out", certfile, "-config", config]
-    elif subject:
-        ssl_subject = ""
-        if "country" in subject:
-            ssl_subject = ssl_subject + "/C={}".format(subject["country"])
-        if "state" in subject:
-            ssl_subject = ssl_subject + "/ST={}".format(subject["state"])
-        if "locality" in subject:
-            ssl_subject = ssl_subject + "/L={}".format(subject["locality"])
-        if "organization" in subject:
-            ssl_subject = ssl_subject + "/O={}".format(subject["organization"])
-        if "organizational_unit" in subject:
-            ssl_subject = ssl_subject + "/OU={}".format(subject["organizational_unit"])
-        if "cn" in subject:
-            ssl_subject = ssl_subject + "/CN={}".format(subject["cn"])
-        else:
-            hookenv.log("When using \"subject\" argument you must "
-                        "provide \"cn\" field at very least")
-            return False
-        if "email" in subject:
-            ssl_subject = ssl_subject + "/emailAddress={}".format(subject["email"])
-
-        cmd = ["/usr/bin/openssl", "req", "-new", "-newkey",
-               "rsa:{}".format(keysize), "-days", "365", "-nodes", "-x509",
-               "-keyout", keyfile,
-               "-out", certfile, "-subj", ssl_subject]
-    elif cn:
-        cmd = ["/usr/bin/openssl", "req", "-new", "-newkey",
-               "rsa:{}".format(keysize), "-days", "365", "-nodes", "-x509",
-               "-keyout", keyfile,
-               "-out", certfile, "-subj", "/CN={}".format(cn)]
-
-    if not cmd:
-        hookenv.log("No config, subject or cn provided,"
-                    "unable to generate self signed SSL certificates")
-        return False
-    try:
-        subprocess.check_call(cmd)
-        return True
-    except Exception as e:
-        print "Execution of openssl command failed:\n{}".format(e)
-        return False

=== removed file 'hooks/charmhelpers/contrib/ssl/service.py'
--- hooks/charmhelpers/contrib/ssl/service.py	2014-05-09 20:11:59 +0000
+++ hooks/charmhelpers/contrib/ssl/service.py	1970-01-01 00:00:00 +0000
@@ -1,267 +0,0 @@
-import logging
-import os
-from os.path import join as path_join
-from os.path import exists
-import subprocess
-
-
-log = logging.getLogger("service_ca")
-
-logging.basicConfig(level=logging.DEBUG)
-
-STD_CERT = "standard"
-
-# Mysql server is fairly picky about cert creation
-# and types, spec its creation separately for now.
-MYSQL_CERT = "mysql"
-
-
-class ServiceCA(object):
-
-    default_expiry = str(365 * 2)
-    default_ca_expiry = str(365 * 6)
-
-    def __init__(self, name, ca_dir, cert_type=STD_CERT):
-        self.name = name
-        self.ca_dir = ca_dir
-        self.cert_type = cert_type
-
-    ###############
-    # Hook Helper API
-    @staticmethod
-    def get_ca(type=STD_CERT):
-        service_name = os.environ['JUJU_UNIT_NAME'].split('/')[0]
-        ca_path = os.path.join(os.environ['CHARM_DIR'], 'ca')
-        ca = ServiceCA(service_name, ca_path, type)
-        ca.init()
-        return ca
-
-    @classmethod
-    def get_service_cert(cls, type=STD_CERT):
-        service_name = os.environ['JUJU_UNIT_NAME'].split('/')[0]
-        ca = cls.get_ca()
-        crt, key = ca.get_or_create_cert(service_name)
-        return crt, key, ca.get_ca_bundle()
-
-    ###############
-
-    def init(self):
-        log.debug("initializing service ca")
-        if not exists(self.ca_dir):
-            self._init_ca_dir(self.ca_dir)
-            self._init_ca()
-
-    @property
-    def ca_key(self):
-        return path_join(self.ca_dir, 'private', 'cacert.key')
-
-    @property
-    def ca_cert(self):
-        return path_join(self.ca_dir, 'cacert.pem')
-
-    @property
-    def ca_conf(self):
-        return path_join(self.ca_dir, 'ca.cnf')
-
-    @property
-    def signing_conf(self):
-        return path_join(self.ca_dir, 'signing.cnf')
-
-    def _init_ca_dir(self, ca_dir):
-        os.mkdir(ca_dir)
-        for i in ['certs', 'crl', 'newcerts', 'private']:
-            sd = path_join(ca_dir, i)
-            if not exists(sd):
-                os.mkdir(sd)
-
-        if not exists(path_join(ca_dir, 'serial')):
-            with open(path_join(ca_dir, 'serial'), 'wb') as fh:
-                fh.write('02\n')
-
-        if not exists(path_join(ca_dir, 'index.txt')):
-            with open(path_join(ca_dir, 'index.txt'), 'wb') as fh:
-                fh.write('')
-
-    def _init_ca(self):
-        """Generate the root ca's cert and key.
-        """
-        if not exists(path_join(self.ca_dir, 'ca.cnf')):
-            with open(path_join(self.ca_dir, 'ca.cnf'), 'wb') as fh:
-                fh.write(
-                    CA_CONF_TEMPLATE % (self.get_conf_variables()))
-
-        if not exists(path_join(self.ca_dir, 'signing.cnf')):
-            with open(path_join(self.ca_dir, 'signing.cnf'), 'wb') as fh:
-                fh.write(
-                    SIGNING_CONF_TEMPLATE % (self.get_conf_variables()))
-
-        if exists(self.ca_cert) or exists(self.ca_key):
-            raise RuntimeError("Initialized called when CA already exists")
-        cmd = ['openssl', 'req', '-config', self.ca_conf,
-               '-x509', '-nodes', '-newkey', 'rsa',
-               '-days', self.default_ca_expiry,
-               '-keyout', self.ca_key, '-out', self.ca_cert,
-               '-outform', 'PEM']
-        output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
-        log.debug("CA Init:\n %s", output)
-
-    def get_conf_variables(self):
-        return dict(
-            org_name="juju",
-            org_unit_name="%s service" % self.name,
-            common_name=self.name,
-            ca_dir=self.ca_dir)
-
-    def get_or_create_cert(self, common_name):
-        if common_name in self:
-            return self.get_certificate(common_name)
-        return self.create_certificate(common_name)
-
-    def create_certificate(self, common_name):
-        if common_name in self:
-            return self.get_certificate(common_name)
-        key_p = path_join(self.ca_dir, "certs", "%s.key" % common_name)
-        crt_p = path_join(self.ca_dir, "certs", "%s.crt" % common_name)
-        csr_p = path_join(self.ca_dir, "certs", "%s.csr" % common_name)
-        self._create_certificate(common_name, key_p, csr_p, crt_p)
-        return self.get_certificate(common_name)
-
-    def get_certificate(self, common_name):
-        if not common_name in self:
-            raise ValueError("No certificate for %s" % common_name)
-        key_p = path_join(self.ca_dir, "certs", "%s.key" % common_name)
-        crt_p = path_join(self.ca_dir, "certs", "%s.crt" % common_name)
-        with open(crt_p) as fh:
-            crt = fh.read()
-        with open(key_p) as fh:
-            key = fh.read()
-        return crt, key
-
-    def __contains__(self, common_name):
-        crt_p = path_join(self.ca_dir, "certs", "%s.crt" % common_name)
-        return exists(crt_p)
-
-    def _create_certificate(self, common_name, key_p, csr_p, crt_p):
-        template_vars = self.get_conf_variables()
-        template_vars['common_name'] = common_name
-        subj = '/O=%(org_name)s/OU=%(org_unit_name)s/CN=%(common_name)s' % (
-            template_vars)
-
-        log.debug("CA Create Cert %s", common_name)
-        cmd = ['openssl', 'req', '-sha1', '-newkey', 'rsa:2048',
-               '-nodes', '-days', self.default_expiry,
-               '-keyout', key_p, '-out', csr_p, '-subj', subj]
-        subprocess.check_call(cmd)
-        cmd = ['openssl', 'rsa', '-in', key_p, '-out', key_p]
-        subprocess.check_call(cmd)
-
-        log.debug("CA Sign Cert %s", common_name)
-        if self.cert_type == MYSQL_CERT:
-            cmd = ['openssl', 'x509', '-req',
-                   '-in', csr_p, '-days', self.default_expiry,
-                   '-CA', self.ca_cert, '-CAkey', self.ca_key,
-                   '-set_serial', '01', '-out', crt_p]
-        else:
-            cmd = ['openssl', 'ca', '-config', self.signing_conf,
-                   '-extensions', 'req_extensions',
-                   '-days', self.default_expiry, '-notext',
-                   '-in', csr_p, '-out', crt_p, '-subj', subj, '-batch']
-        log.debug("running %s", " ".join(cmd))
-        subprocess.check_call(cmd)
-
-    def get_ca_bundle(self):
-        with open(self.ca_cert) as fh:
-            return fh.read()
-
-
-CA_CONF_TEMPLATE = """
-[ ca ]
-default_ca = CA_default
-
-[ CA_default ]
-dir                     = %(ca_dir)s
-policy                  = policy_match
-database                = $dir/index.txt
-serial                  = $dir/serial
-certs                   = $dir/certs
-crl_dir                 = $dir/crl
-new_certs_dir           = $dir/newcerts
-certificate             = $dir/cacert.pem
-private_key             = $dir/private/cacert.key
-RANDFILE                = $dir/private/.rand
-default_md              = default
-
-[ req ]
-default_bits            = 1024
-default_md              = sha1
-
-prompt                  = no
-distinguished_name      = ca_distinguished_name
-
-x509_extensions         = ca_extensions
-
-[ ca_distinguished_name ]
-organizationName        = %(org_name)s
-organizationalUnitName  = %(org_unit_name)s Certificate Authority
-
-
-[ policy_match ]
-countryName             = optional
-stateOrProvinceName     = optional
-organizationName        = match
-organizationalUnitName  = optional
-commonName              = supplied
-
-[ ca_extensions ]
-basicConstraints        = critical,CA:true
-subjectKeyIdentifier    = hash
-authorityKeyIdentifier  = keyid:always, issuer
-keyUsage                = cRLSign, keyCertSign
-"""
-
-
-SIGNING_CONF_TEMPLATE = """
-[ ca ]
-default_ca = CA_default
-
-[ CA_default ]
-dir                     = %(ca_dir)s
-policy                  = policy_match
-database                = $dir/index.txt
-serial                  = $dir/serial
-certs                   = $dir/certs
-crl_dir                 = $dir/crl
-new_certs_dir           = $dir/newcerts
-certificate             = $dir/cacert.pem
-private_key             = $dir/private/cacert.key
-RANDFILE                = $dir/private/.rand
-default_md              = default
-
-[ req ]
-default_bits            = 1024
-default_md              = sha1
-
-prompt                  = no
-distinguished_name      = req_distinguished_name
-
-x509_extensions         = req_extensions
-
-[ req_distinguished_name ]
-organizationName        = %(org_name)s
-organizationalUnitName  = %(org_unit_name)s machine resources
-commonName              = %(common_name)s
-
-[ policy_match ]
-countryName             = optional
-stateOrProvinceName     = optional
-organizationName        = match
-organizationalUnitName  = optional
-commonName              = supplied
-
-[ req_extensions ]
-basicConstraints        = CA:false
-subjectKeyIdentifier    = hash
-authorityKeyIdentifier  = keyid:always, issuer
-keyUsage                = digitalSignature, keyEncipherment, keyAgreement
-extendedKeyUsage        = serverAuth, clientAuth
-"""

=== removed directory 'hooks/charmhelpers/contrib/storage'
=== removed file 'hooks/charmhelpers/contrib/storage/__init__.py'
=== removed directory 'hooks/charmhelpers/contrib/storage/linux'
=== removed file 'hooks/charmhelpers/contrib/storage/linux/__init__.py'
=== removed file 'hooks/charmhelpers/contrib/storage/linux/ceph.py'
--- hooks/charmhelpers/contrib/storage/linux/ceph.py	2014-05-09 20:11:59 +0000
+++ hooks/charmhelpers/contrib/storage/linux/ceph.py	1970-01-01 00:00:00 +0000
@@ -1,387 +0,0 @@
-#
-# Copyright 2012 Canonical Ltd.
-#
-# This file is sourced from lp:openstack-charm-helpers
-#
-# Authors:
-#  James Page <[e-mail]>
-#  Adam Gandelman <[e-mail]>
-#
-
-import os
-import shutil
-import json
-import time
-
-from subprocess import (
-    check_call,
-    check_output,
-    CalledProcessError
-)
-
-from charmhelpers.core.hookenv import (
-    relation_get,
-    relation_ids,
-    related_units,
-    log,
-    INFO,
-    WARNING,
-    ERROR
-)
-
-from charmhelpers.core.host import (
-    mount,
-    mounts,
-    service_start,
-    service_stop,
-    service_running,
-    umount,
-)
-
-from charmhelpers.fetch import (
-    apt_install,
-)
-
-KEYRING = '/etc/ceph/ceph.client.{}.keyring'
-KEYFILE = '/etc/ceph/ceph.client.{}.key'
-
-CEPH_CONF = """[global]
- auth supported = {auth}
- keyring = {keyring}
- mon host = {mon_hosts}
- log to syslog = {use_syslog}
- err to syslog = {use_syslog}
- clog to syslog = {use_syslog}
-"""
-
-
-def install():
-    ''' Basic Ceph client installation '''
-    ceph_dir = "/etc/ceph"
-    if not os.path.exists(ceph_dir):
-        os.mkdir(ceph_dir)
-    apt_install('ceph-common', fatal=True)
-
-
-def rbd_exists(service, pool, rbd_img):
-    ''' Check to see if a RADOS block device exists '''
-    try:
-        out = check_output(['rbd', 'list', '--id', service,
-                            '--pool', pool])
-    except CalledProcessError:
-        return False
-    else:
-        return rbd_img in out
-
-
-def create_rbd_image(service, pool, image, sizemb):
-    ''' Create a new RADOS block device '''
-    cmd = [
-        'rbd',
-        'create',
-        image,
-        '--size',
-        str(sizemb),
-        '--id',
-        service,
-        '--pool',
-        pool
-    ]
-    check_call(cmd)
-
-
-def pool_exists(service, name):
-    ''' Check to see if a RADOS pool already exists '''
-    try:
-        out = check_output(['rados', '--id', service, 'lspools'])
-    except CalledProcessError:
-        return False
-    else:
-        return name in out
-
-
-def get_osds(service):
-    '''
-    Return a list of all Ceph Object Storage Daemons
-    currently in the cluster
-    '''
-    version = ceph_version()
-    if version and version >= '0.56':
-        return json.loads(check_output(['ceph', '--id', service,
-                                        'osd', 'ls', '--format=json']))
-    else:
-        return None
-
-
-def create_pool(service, name, replicas=2):
-    ''' Create a new RADOS pool '''
-    if pool_exists(service, name):
-        log("Ceph pool {} already exists, skipping creation".format(name),
-            level=WARNING)
-        return
-    # Calculate the number of placement groups based
-    # on upstream recommended best practices.
-    osds = get_osds(service)
-    if osds:
-        pgnum = (len(osds) * 100 / replicas)
-    else:
-        # NOTE(james-page): Default to 200 for older ceph versions
-        # which don't support OSD query from cli
-        pgnum = 200
-    cmd = [
-        'ceph', '--id', service,
-        'osd', 'pool', 'create',
-        name, str(pgnum)
-    ]
-    check_call(cmd)
-    cmd = [
-        'ceph', '--id', service,
-        'osd', 'pool', 'set', name,
-        'size', str(replicas)
-    ]
-    check_call(cmd)
-
-
-def delete_pool(service, name):
-    ''' Delete a RADOS pool from ceph '''
-    cmd = [
-        'ceph', '--id', service,
-        'osd', 'pool', 'delete',
-        name, '--yes-i-really-really-mean-it'
-    ]
-    check_call(cmd)
-
-
-def _keyfile_path(service):
-    return KEYFILE.format(service)
-
-
-def _keyring_path(service):
-    return KEYRING.format(service)
-
-
-def create_keyring(service, key):
-    ''' Create a new Ceph keyring containing key'''
-    keyring = _keyring_path(service)
-    if os.path.exists(keyring):
-        log('ceph: Keyring exists at %s.' % keyring, level=WARNING)
-        return
-    cmd = [
-        'ceph-authtool',
-        keyring,
-        '--create-keyring',
-        '--name=client.{}'.format(service),
-        '--add-key={}'.format(key)
-    ]
-    check_call(cmd)
-    log('ceph: Created new ring at %s.' % keyring, level=INFO)
-
-
-def create_key_file(service, key):
-    ''' Create a file containing key '''
-    keyfile = _keyfile_path(service)
-    if os.path.exists(keyfile):
-        log('ceph: Keyfile exists at %s.' % keyfile, level=WARNING)
-        return
-    with open(keyfile, 'w') as fd:
-        fd.write(key)
-    log('ceph: Created new keyfile at %s.' % keyfile, level=INFO)
-
-
-def get_ceph_nodes():
-    ''' Query named relation 'ceph' to detemine current nodes '''
-    hosts = []
-    for r_id in relation_ids('ceph'):
-        for unit in related_units(r_id):
-            hosts.append(relation_get('private-address', unit=unit, rid=r_id))
-    return hosts
-
-
-def configure(service, key, auth, use_syslog):
-    ''' Perform basic configuration of Ceph '''
-    create_keyring(service, key)
-    create_key_file(service, key)
-    hosts = get_ceph_nodes()
-    with open('/etc/ceph/ceph.conf', 'w') as ceph_conf:
-        ceph_conf.write(CEPH_CONF.format(auth=auth,
-                                         keyring=_keyring_path(service),
-                                         mon_hosts=",".join(map(str, hosts)),
-                                         use_syslog=use_syslog))
-    modprobe('rbd')
-
-
-def image_mapped(name):
-    ''' Determine whether a RADOS block device is mapped locally '''
-    try:
-        out = check_output(['rbd', 'showmapped'])
-    except CalledProcessError:
-        return False
-    else:
-        return name in out
-
-
-def map_block_storage(service, pool, image):
-    ''' Map a RADOS block device for local use '''
-    cmd = [
-        'rbd',
-        'map',
-        '{}/{}'.format(pool, image),
-        '--user',
-        service,
-        '--secret',
-        _keyfile_path(service),
-    ]
-    check_call(cmd)
-
-
-def filesystem_mounted(fs):
-    ''' Determine whether a filesytems is already mounted '''
-    return fs in [f for f, m in mounts()]
-
-
-def make_filesystem(blk_device, fstype='ext4', timeout=10):
-    ''' Make a new filesystem on the specified block device '''
-    count = 0
-    e_noent = os.errno.ENOENT
-    while not os.path.exists(blk_device):
-        if count >= timeout:
-            log('ceph: gave up waiting on block device %s' % blk_device,
-                level=ERROR)
-            raise IOError(e_noent, os.strerror(e_noent), blk_device)
-        log('ceph: waiting for block device %s to appear' % blk_device,
-            level=INFO)
-        count += 1
-        time.sleep(1)
-    else:
-        log('ceph: Formatting block device %s as filesystem %s.' %
-            (blk_device, fstype), level=INFO)
-        check_call(['mkfs', '-t', fstype, blk_device])
-
-
-def place_data_on_block_device(blk_device, data_src_dst):
-    ''' Migrate data in data_src_dst to blk_device and then remount '''
-    # mount block device into /mnt
-    mount(blk_device, '/mnt')
-    # copy data to /mnt
-    copy_files(data_src_dst, '/mnt')
-    # umount block device
-    umount('/mnt')
-    # Grab user/group ID's from original source
-    _dir = os.stat(data_src_dst)
-    uid = _dir.st_uid
-    gid = _dir.st_gid
-    # re-mount where the data should originally be
-    # TODO: persist is currently a NO-OP in core.host
-    mount(blk_device, data_src_dst, persist=True)
-    # ensure original ownership of new mount.
-    os.chown(data_src_dst, uid, gid)
-
-
-# TODO: re-use
-def modprobe(module):
-    ''' Load a kernel module and configure for auto-load on reboot '''
-    log('ceph: Loading kernel module', level=INFO)
-    cmd = ['modprobe', module]
-    check_call(cmd)
-    with open('/etc/modules', 'r+') as modules:
-        if module not in modules.read():
-            modules.write(module)
-
-
-def copy_files(src, dst, symlinks=False, ignore=None):
-    ''' Copy files from src to dst '''
-    for item in os.listdir(src):
-        s = os.path.join(src, item)
-        d = os.path.join(dst, item)
-        if os.path.isdir(s):
-            shutil.copytree(s, d, symlinks, ignore)
-        else:
-            shutil.copy2(s, d)
-
-
-def ensure_ceph_storage(service, pool, rbd_img, sizemb, mount_point,
-                        blk_device, fstype, system_services=[]):
-    """
-    NOTE: This function must only be called from a single service unit for
-          the same rbd_img otherwise data loss will occur.
-
-    Ensures given pool and RBD image exists, is mapped to a block device,
-    and the device is formatted and mounted at the given mount_point.
-
-    If formatting a device for the first time, data existing at mount_point
-    will be migrated to the RBD device before being re-mounted.
-
-    All services listed in system_services will be stopped prior to data
-    migration and restarted when complete.
-    """
-    # Ensure pool, RBD image, RBD mappings are in place.
-    if not pool_exists(service, pool):
-        log('ceph: Creating new pool {}.'.format(pool))
-        create_pool(service, pool)
-
-    if not rbd_exists(service, pool, rbd_img):
-        log('ceph: Creating RBD image ({}).'.format(rbd_img))
-        create_rbd_image(service, pool, rbd_img, sizemb)
-
-    if not image_mapped(rbd_img):
-        log('ceph: Mapping RBD Image {} as a Block Device.'.format(rbd_img))
-        map_block_storage(service, pool, rbd_img)
-
-    # make file system
-    # TODO: What happens if for whatever reason this is run again and
-    # the data is already in the rbd device and/or is mounted??
-    # When it is mounted already, it will fail to make the fs
-    # XXX: This is really sketchy!  Need to at least add an fstab entry
-    #      otherwise this hook will blow away existing data if its executed
-    #      after a reboot.
-    if not filesystem_mounted(mount_point):
-        make_filesystem(blk_device, fstype)
-
-        for svc in system_services:
-            if service_running(svc):
-                log('ceph: Stopping services {} prior to migrating data.'
-                    .format(svc))
-                service_stop(svc)
-
-        place_data_on_block_device(blk_device, mount_point)
-
-        for svc in system_services:
-            log('ceph: Starting service {} after migrating data.'
-                .format(svc))
-            service_start(svc)
-
-
-def ensure_ceph_keyring(service, user=None, group=None):
-    '''
-    Ensures a ceph keyring is created for a named service
-    and optionally ensures user and group ownership.
-
-    Returns False if no ceph key is available in relation state.
-    '''
-    key = None
-    for rid in relation_ids('ceph'):
-        for unit in related_units(rid):
-            key = relation_get('key', rid=rid, unit=unit)
-            if key:
-                break
-    if not key:
-        return False
-    create_keyring(service=service, key=key)
-    keyring = _keyring_path(service)
-    if user and group:
-        check_call(['chown', '%s.%s' % (user, group), keyring])
-    return True
-
-
-def ceph_version():
-    ''' Retrieve the local version of ceph '''
-    if os.path.exists('/usr/bin/ceph'):
-        cmd = ['ceph', '-v']
-        output = check_output(cmd)
-        output = output.split()
-        if len(output) > 3:
-            return output[2]
-        else:
-            return None
-    else:
-        return None

=== removed file 'hooks/charmhelpers/contrib/storage/linux/loopback.py'
--- hooks/charmhelpers/contrib/storage/linux/loopback.py	2013-11-26 17:12:54 +0000
+++ hooks/charmhelpers/contrib/storage/linux/loopback.py	1970-01-01 00:00:00 +0000
@@ -1,62 +0,0 @@
-
-import os
-import re
-
-from subprocess import (
-    check_call,
-    check_output,
-)
-
-
-##################################################
-# loopback device helpers.
-##################################################
-def loopback_devices():
-    '''
-    Parse through 'losetup -a' output to determine currently mapped
-    loopback devices. Output is expected to look like:
-
-        /dev/loop0: [0807]:961814 (/tmp/my.img)
-
-    :returns: dict: a dict mapping {loopback_dev: backing_file}
-    '''
-    loopbacks = {}
-    cmd = ['losetup', '-a']
-    devs = [d.strip().split(' ') for d in
-            check_output(cmd).splitlines() if d != '']
-    for dev, _, f in devs:
-        loopbacks[dev.replace(':', '')] = re.search('\((\S+)\)', f).groups()[0]
-    return loopbacks
-
-
-def create_loopback(file_path):
-    '''
-    Create a loopback device for a given backing file.
-
-    :returns: str: Full path to new loopback device (eg, /dev/loop0)
-    '''
-    file_path = os.path.abspath(file_path)
-    check_call(['losetup', '--find', file_path])
-    for d, f in loopback_devices().iteritems():
-        if f == file_path:
-            return d
-
-
-def ensure_loopback_device(path, size):
-    '''
-    Ensure a loopback device exists for a given backing file path and size.
-    If it a loopback device is not mapped to file, a new one will be created.
-
-    TODO: Confirm size of found loopback device.
-
-    :returns: str: Full path to the ensured loopback device (eg, /dev/loop0)
-    '''
-    for d, f in loopback_devices().iteritems():
-        if f == path:
-            return d
-
-    if not os.path.exists(path):
-        cmd = ['truncate', '--size', size, path]
-        check_call(cmd)
-
-    return create_loopback(path)

=== removed file 'hooks/charmhelpers/contrib/storage/linux/lvm.py'
--- hooks/charmhelpers/contrib/storage/linux/lvm.py	2013-11-26 17:12:54 +0000
+++ hooks/charmhelpers/contrib/storage/linux/lvm.py	1970-01-01 00:00:00 +0000
@@ -1,88 +0,0 @@
-from subprocess import (
-    CalledProcessError,
-    check_call,
-    check_output,
-    Popen,
-    PIPE,
-)
-
-
-##################################################
-# LVM helpers.
-##################################################
-def deactivate_lvm_volume_group(block_device):
-    '''
-    Deactivate any volume gruop associated with an LVM physical volume.
-
-    :param block_device: str: Full path to LVM physical volume
-    '''
-    vg = list_lvm_volume_group(block_device)
-    if vg:
-        cmd = ['vgchange', '-an', vg]
-        check_call(cmd)
-
-
-def is_lvm_physical_volume(block_device):
-    '''
-    Determine whether a block device is initialized as an LVM PV.
-
-    :param block_device: str: Full path of block device to inspect.
-
-    :returns: boolean: True if block device is a PV, False if not.
-    '''
-    try:
-        check_output(['pvdisplay', block_device])
-        return True
-    except CalledProcessError:
-        return False
-
-
-def remove_lvm_physical_volume(block_device):
-    '''
-    Remove LVM PV signatures from a given block device.
-
-    :param block_device: str: Full path of block device to scrub.
-    '''
-    p = Popen(['pvremove', '-ff', block_device],
-              stdin=PIPE)
-    p.communicate(input='y\n')
-
-
-def list_lvm_volume_group(block_device):
-    '''
-    List LVM volume group associated with a given block device.
-
-    Assumes block device is a valid LVM PV.
-
-    :param block_device: str: Full path of block device to inspect.
-
-    :returns: str: Name of volume group associated with block device or None
-    '''
-    vg = None
-    pvd = check_output(['pvdisplay', block_device]).splitlines()
-    for l in pvd:
-        if l.strip().startswith('VG Name'):
-            vg = ' '.join(l.split()).split(' ').pop()
-    return vg
-
-
-def create_lvm_physical_volume(block_device):
-    '''
-    Initialize a block device as an LVM physical volume.
-
-    :param block_device: str: Full path of block device to initialize.
-
-    '''
-    check_call(['pvcreate', block_device])
-
-
-def create_lvm_volume_group(volume_group, block_device):
-    '''
-    Create an LVM volume group backed by a given block device.
-
-    Assumes block device has already been initialized as an LVM PV.
-
-    :param volume_group: str: Name of volume group to create.
-    :block_device: str: Full path of PV-initialized block device.
-    '''
-    check_call(['vgcreate', volume_group, block_device])

=== removed file 'hooks/charmhelpers/contrib/storage/linux/utils.py'
--- hooks/charmhelpers/contrib/storage/linux/utils.py	2014-05-09 20:11:59 +0000
+++ hooks/charmhelpers/contrib/storage/linux/utils.py	1970-01-01 00:00:00 +0000
@@ -1,35 +0,0 @@
-from os import stat
-from stat import S_ISBLK
-
-from subprocess import (
-    check_call,
-    check_output,
-    call
-)
-
-
-def is_block_device(path):
-    '''
-    Confirm device at path is a valid block device node.
-
-    :returns: boolean: True if path is a block device, False if not.
-    '''
-    return S_ISBLK(stat(path).st_mode)
-
-
-def zap_disk(block_device):
-    '''
-    Clear a block device of partition table. Relies on sgdisk, which is
-    installed as pat of the 'gdisk' package in Ubuntu.
-
-    :param block_device: str: Full path of block device to clean.
-    '''
-    # sometimes sgdisk exits non-zero; this is OK, dd will clean up
-    call(['sgdisk', '--zap-all', '--mbrtogpt',
-          '--clear', block_device])
-    dev_end = check_output(['blockdev', '--getsz', block_device])
-    gpt_end = int(dev_end.split()[0]) - 100
-    check_call(['dd', 'if=/dev/zero', 'of=%s'%(block_device),
-                'bs=1M', 'count=1'])
-    check_call(['dd', 'if=/dev/zero', 'of=%s'%(block_device),
-                'bs=512', 'count=100', 'seek=%s'%(gpt_end)])

=== removed directory 'hooks/charmhelpers/contrib/templating'
=== removed file 'hooks/charmhelpers/contrib/templating/__init__.py'
=== removed file 'hooks/charmhelpers/contrib/templating/contexts.py'
--- hooks/charmhelpers/contrib/templating/contexts.py	2014-05-09 20:11:59 +0000
+++ hooks/charmhelpers/contrib/templating/contexts.py	1970-01-01 00:00:00 +0000
@@ -1,104 +0,0 @@
-# Copyright 2013 Canonical Ltd.
-#
-# Authors:
-#  Charm Helpers Developers <[e-mail]>
-"""A helper to create a yaml cache of config with namespaced relation data."""
-import os
-import yaml
-
-import charmhelpers.core.hookenv
-
-
-charm_dir = os.environ.get('CHARM_DIR', '')
-
-
-def dict_keys_without_hyphens(a_dict):
-    """Return the a new dict with underscores instead of hyphens in keys."""
-    return dict(
-        (key.replace('-', '_'), val) for key, val in a_dict.items())
-
-
-def update_relations(context, namespace_separator=':'):
-    """Update the context with the relation data."""
-    # Add any relation data prefixed with the relation type.
-    relation_type = charmhelpers.core.hookenv.relation_type()
-    relations = []
-    context['current_relation'] = {}
-    if relation_type is not None:
-        relation_data = charmhelpers.core.hookenv.relation_get()
-        context['current_relation'] = relation_data
-        # Deprecated: the following use of relation data as keys
-        # directly in the context will be removed.
-        relation_data = dict(
-            ("{relation_type}{namespace_separator}{key}".format(
-                relation_type=relation_type,
-                key=key,
-                namespace_separator=namespace_separator), val)
-            for key, val in relation_data.items())
-        relation_data = dict_keys_without_hyphens(relation_data)
-        context.update(relation_data)
-        relations = charmhelpers.core.hookenv.relations_of_type(relation_type)
-        relations = [dict_keys_without_hyphens(rel) for rel in relations]
-
-    if 'relations_deprecated' not in context:
-        context['relations_deprecated'] = {}
-    if relation_type is not None:
-        relation_type = relation_type.replace('-', '_')
-        context['relations_deprecated'][relation_type] = relations
-
-    context['relations'] = charmhelpers.core.hookenv.relations()
-
-
-def juju_state_to_yaml(yaml_path, namespace_separator=':',
-                       allow_hyphens_in_keys=True):
-    """Update the juju config and state in a yaml file.
-
-    This includes any current relation-get data, and the charm
-    directory.
-
-    This function was created for the ansible and saltstack
-    support, as those libraries can use a yaml file to supply
-    context to templates, but it may be useful generally to
-    create and update an on-disk cache of all the config, including
-    previous relation data.
-
-    By default, hyphens are allowed in keys as this is supported
-    by yaml, but for tools like ansible, hyphens are not valid [1].
-
-    [1] http://www.ansibleworks.com/docs/playbooks_variables.html#what-makes-a-valid-variable-name
-    """
-    config = charmhelpers.core.hookenv.config()
-
-    # Add the charm_dir which we will need to refer to charm
-    # file resources etc.
-    config['charm_dir'] = charm_dir
-    config['local_unit'] = charmhelpers.core.hookenv.local_unit()
-    config['unit_private_address'] = charmhelpers.core.hookenv.unit_private_ip()
-    config['unit_public_address'] = charmhelpers.core.hookenv.unit_get(
-        'public-address'
-    )
-
-    # Don't use non-standard tags for unicode which will not
-    # work when salt uses yaml.load_safe.
-    yaml.add_representer(unicode, lambda dumper,
-                         value: dumper.represent_scalar(
-                             u'tag:yaml.org,2002:str', value))
-
-    yaml_dir = os.path.dirname(yaml_path)
-    if not os.path.exists(yaml_dir):
-        os.makedirs(yaml_dir)
-
-    if os.path.exists(yaml_path):
-        with open(yaml_path, "r") as existing_vars_file:
-            existing_vars = yaml.load(existing_vars_file.read())
-    else:
-        existing_vars = {}
-
-    if not allow_hyphens_in_keys:
-        config = dict_keys_without_hyphens(config)
-    existing_vars.update(config)
-
-    update_relations(existing_vars, namespace_separator)
-
-    with open(yaml_path, "w+") as fp:
-        fp.write(yaml.dump(existing_vars, default_flow_style=False))

=== removed file 'hooks/charmhelpers/contrib/templating/pyformat.py'
--- hooks/charmhelpers/contrib/templating/pyformat.py	2013-11-26 17:12:54 +0000
+++ hooks/charmhelpers/contrib/templating/pyformat.py	1970-01-01 00:00:00 +0000
@@ -1,13 +0,0 @@
-'''
-Templating using standard Python str.format() method.
-'''
-
-from charmhelpers.core import hookenv
-
-
-def render(template, extra={}, **kwargs):
-    """Return the template rendered using Python's str.format()."""
-    context = hookenv.execution_environment()
-    context.update(extra)
-    context.update(kwargs)
-    return template.format(**context)

=== removed directory 'hooks/charmhelpers/contrib/unison'
=== removed file 'hooks/charmhelpers/contrib/unison/__init__.py'
--- hooks/charmhelpers/contrib/unison/__init__.py	2014-05-09 20:11:59 +0000
+++ hooks/charmhelpers/contrib/unison/__init__.py	1970-01-01 00:00:00 +0000
@@ -1,257 +0,0 @@
-# Easy file synchronization among peer units using ssh + unison.
-#
-# From *both* peer relation -joined and -changed, add a call to
-# ssh_authorized_peers() describing the peer relation and the desired
-# user + group.  After all peer relations have settled, all hosts should
-# be able to connect to on another via key auth'd ssh as the specified user.
-#
-# Other hooks are then free to synchronize files and directories using
-# sync_to_peers().
-#
-# For a peer relation named 'cluster', for example:
-#
-# cluster-relation-joined:
-# ...
-# ssh_authorized_peers(peer_interface='cluster',
-#                      user='juju_ssh', group='juju_ssh',
-#                      ensure_user=True)
-# ...
-#
-# cluster-relation-changed:
-# ...
-# ssh_authorized_peers(peer_interface='cluster',
-#                      user='juju_ssh', group='juju_ssh',
-#                      ensure_user=True)
-# ...
-#
-# Hooks are now free to sync files as easily as:
-#
-# files = ['/etc/fstab', '/etc/apt.conf.d/']
-# sync_to_peers(peer_interface='cluster',
-#                user='juju_ssh, paths=[files])
-#
-# It is assumed the charm itself has setup permissions on each unit
-# such that 'juju_ssh' has read + write permissions.  Also assumed
-# that the calling charm takes care of leader delegation.
-#
-# Additionally files can be synchronized only to an specific unit:
-# sync_to_peer(slave_address, user='juju_ssh',
-#              paths=[files], verbose=False)
-
-import os
-import pwd
-
-from copy import copy
-from subprocess import check_call, check_output
-
-from charmhelpers.core.host import (
-    adduser,
-    add_user_to_group,
-)
-
-from charmhelpers.core.hookenv import (
-    log,
-    hook_name,
-    relation_ids,
-    related_units,
-    relation_set,
-    relation_get,
-    unit_private_ip,
-    ERROR,
-)
-
-BASE_CMD = ['unison', '-auto', '-batch=true', '-confirmbigdel=false',
-            '-fastcheck=true', '-group=false', '-owner=false',
-            '-prefer=newer', '-times=true']
-
-
-def get_homedir(user):
-    try:
-        user = pwd.getpwnam(user)
-        return user.pw_dir
-    except KeyError:
-        log('Could not get homedir for user %s: user exists?', ERROR)
-        raise Exception
-
-
-def create_private_key(user, priv_key_path):
-    if not os.path.isfile(priv_key_path):
-        log('Generating new SSH key for user %s.' % user)
-        cmd = ['ssh-keygen', '-q', '-N', '', '-t', 'rsa', '-b', '2048',
-               '-f', priv_key_path]
-        check_call(cmd)
-    else:
-        log('SSH key already exists at %s.' % priv_key_path)
-    check_call(['chown', user, priv_key_path])
-    check_call(['chmod', '0600', priv_key_path])
-
-
-def create_public_key(user, priv_key_path, pub_key_path):
-    if not os.path.isfile(pub_key_path):
-        log('Generating missing ssh public key @ %s.' % pub_key_path)
-        cmd = ['ssh-keygen', '-y', '-f', priv_key_path]
-        p = check_output(cmd).strip()
-        with open(pub_key_path, 'wb') as out:
-            out.write(p)
-    check_call(['chown', user, pub_key_path])
-
-
-def get_keypair(user):
-    home_dir = get_homedir(user)
-    ssh_dir = os.path.join(home_dir, '.ssh')
-    priv_key = os.path.join(ssh_dir, 'id_rsa')
-    pub_key = '%s.pub' % priv_key
-
-    if not os.path.isdir(ssh_dir):
-        os.mkdir(ssh_dir)
-        check_call(['chown', '-R', user, ssh_dir])
-
-    create_private_key(user, priv_key)
-    create_public_key(user, priv_key, pub_key)
-
-    with open(priv_key, 'r') as p:
-        _priv = p.read().strip()
-
-    with open(pub_key, 'r') as p:
-        _pub = p.read().strip()
-
-    return (_priv, _pub)
-
-
-def write_authorized_keys(user, keys):
-    home_dir = get_homedir(user)
-    ssh_dir = os.path.join(home_dir, '.ssh')
-    auth_keys = os.path.join(ssh_dir, 'authorized_keys')
-    log('Syncing authorized_keys @ %s.' % auth_keys)
-    with open(auth_keys, 'wb') as out:
-        for k in keys:
-            out.write('%s\n' % k)
-
-
-def write_known_hosts(user, hosts):
-    home_dir = get_homedir(user)
-    ssh_dir = os.path.join(home_dir, '.ssh')
-    known_hosts = os.path.join(ssh_dir, 'known_hosts')
-    khosts = []
-    for host in hosts:
-        cmd = ['ssh-keyscan', '-H', '-t', 'rsa', host]
-        remote_key = check_output(cmd).strip()
-        khosts.append(remote_key)
-    log('Syncing known_hosts @ %s.' % known_hosts)
-    with open(known_hosts, 'wb') as out:
-        for host in khosts:
-            out.write('%s\n' % host)
-
-
-def ensure_user(user, group=None):
-    adduser(user)
-    if group:
-        add_user_to_group(user, group)
-
-
-def ssh_authorized_peers(peer_interface, user, group=None,
-                         ensure_local_user=False):
-    """
-    Main setup function, should be called from both peer -changed and -joined
-    hooks with the same parameters.
-    """
-    if ensure_local_user:
-        ensure_user(user, group)
-    priv_key, pub_key = get_keypair(user)
-    hook = hook_name()
-    if hook == '%s-relation-joined' % peer_interface:
-        relation_set(ssh_pub_key=pub_key)
-    elif hook == '%s-relation-changed' % peer_interface:
-        hosts = []
-        keys = []
-
-        for r_id in relation_ids(peer_interface):
-            for unit in related_units(r_id):
-                ssh_pub_key = relation_get('ssh_pub_key',
-                                           rid=r_id,
-                                           unit=unit)