Merge lp:~hopem/charms/trusty/mysql/contrib-database-mysql into lp:charms/trusty/mysql

[Edward Hope-Morley]

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #1560 mysql for hopem mp248743
    UNIT FAIL: unit-test missing

UNIT Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full unit test output: http://paste.ubuntu.com/10072413/
Build: http://10.245.162.77:8080/job/charm_unit_test/1560/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #1732 mysql for hopem mp248743
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/1732/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #1663 mysql for hopem mp248743
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/10072425/
Build: http://10.245.162.77:8080/job/charm_amulet_test/1663/

[lp:~hopem/charms/trusty/mysql/contrib-database-mysql]

146. By Edward Hope-Morley on 2015-02-10

fix charm-helpers yaml

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #1834 mysql for hopem mp248743
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/1834/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #1662 mysql for hopem mp248743
    UNIT FAIL: unit-test missing

UNIT Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full unit test output: http://paste.ubuntu.com/10157219/
Build: http://10.245.162.77:8080/job/charm_unit_test/1662/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #1853 mysql for hopem mp248743
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/10157234/
Build: http://10.245.162.77:8080/job/charm_amulet_test/1853/

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11006-results

review: Needs Fixing (automated testing)

[Cory Johns (johnsca)]

The automated test failure is only for Azure and seems spurious. I manually tested this relation with cs:trusty/lamp and it seems fine. +1

review: Approve

[Liam Young (gnuoy)]

Approve

review: Approve

=== modified file 'charm-helpers.yaml'
--- charm-helpers.yaml	2014-12-01 17:17:12 +0000
+++ charm-helpers.yaml	2015-02-10 11:19:03 +0000
@@ -5,3 +5,5 @@
     - core
     - fetch
     - contrib.network.ip
+    - contrib.database
+    - contrib.peerstorage
 
=== modified file 'hooks/charmhelpers/__init__.py'
--- hooks/charmhelpers/__init__.py	2014-12-01 17:17:12 +0000
+++ hooks/charmhelpers/__init__.py	2015-02-10 11:19:03 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 # Bootstrap charm-helpers, installing its dependencies if necessary using
 # only standard libraries.
 import subprocess
 
=== modified file 'hooks/charmhelpers/contrib/__init__.py'
--- hooks/charmhelpers/contrib/__init__.py	2014-05-08 10:22:43 +0000
+++ hooks/charmhelpers/contrib/__init__.py	2015-02-10 11:19:03 +0000
@@ -0,0 +1,15 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
 
=== added directory 'hooks/charmhelpers/contrib/database'
=== added file 'hooks/charmhelpers/contrib/database/__init__.py'
=== added file 'hooks/charmhelpers/contrib/database/mysql.py'
--- hooks/charmhelpers/contrib/database/mysql.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/contrib/database/mysql.py	2015-02-10 11:19:03 +0000
@@ -0,0 +1,372 @@
+"""Helper for working with a MySQL database"""
+import json
+import socket
+import re
+import sys
+import platform
+import os
+import glob
+
+from string import upper
+
+from charmhelpers.core.host import (
+    mkdir,
+    pwgen,
+    write_file
+)
+from charmhelpers.core.hookenv import (
+    relation_get,
+    related_units,
+    unit_get,
+    log,
+    DEBUG,
+    INFO,
+)
+from charmhelpers.core.hookenv import config as config_get
+from charmhelpers.fetch import (
+    apt_install,
+    apt_update,
+    filter_installed_packages,
+)
+from charmhelpers.contrib.peerstorage import (
+    peer_store,
+    peer_retrieve,
+)
+
+try:
+    import MySQLdb
+except ImportError:
+    apt_update(fatal=True)
+    apt_install(filter_installed_packages(['python-mysqldb']), fatal=True)
+    import MySQLdb
+
+
+class MySQLHelper(object):
+
+    def __init__(self, rpasswdf_template, upasswdf_template, host='localhost'):
+        self.host = host
+        # Password file path templates
+        self.root_passwd_file_template = rpasswdf_template
+        self.user_passwd_file_template = upasswdf_template
+
+    def connect(self, user='root', password=None):
+        self.connection = MySQLdb.connect(user=user, host=self.host,
+                                          passwd=password)
+
+    def database_exists(self, db_name):
+        cursor = self.connection.cursor()
+        try:
+            cursor.execute("SHOW DATABASES")
+            databases = [i[0] for i in cursor.fetchall()]
+        finally:
+            cursor.close()
+
+        return db_name in databases
+
+    def create_database(self, db_name):
+        cursor = self.connection.cursor()
+        try:
+            cursor.execute("CREATE DATABASE {} CHARACTER SET UTF8"
+                           .format(db_name))
+        finally:
+            cursor.close()
+
+    def grant_exists(self, db_name, db_user, remote_ip):
+        cursor = self.connection.cursor()
+        priv_string = "GRANT ALL PRIVILEGES ON `{}`.* " \
+                      "TO '{}'@'{}'".format(db_name, db_user, remote_ip)
+        try:
+            cursor.execute("SHOW GRANTS for '{}'@'{}'".format(db_user,
+                                                              remote_ip))
+            grants = [i[0] for i in cursor.fetchall()]
+        except MySQLdb.OperationalError:
+            return False
+        finally:
+            cursor.close()
+
+        # TODO: review for different grants
+        return priv_string in grants
+
+    def create_grant(self, db_name, db_user, remote_ip, password):
+        cursor = self.connection.cursor()
+        try:
+            # TODO: review for different grants
+            cursor.execute("GRANT ALL PRIVILEGES ON {}.* TO '{}'@'{}' "
+                           "IDENTIFIED BY '{}'".format(db_name,
+                                                       db_user,
+                                                       remote_ip,
+                                                       password))
+        finally:
+            cursor.close()
+
+    def create_admin_grant(self, db_user, remote_ip, password):
+        cursor = self.connection.cursor()
+        try:
+            cursor.execute("GRANT ALL PRIVILEGES ON *.* TO '{}'@'{}' "
+                           "IDENTIFIED BY '{}'".format(db_user,
+                                                       remote_ip,
+                                                       password))
+        finally:
+            cursor.close()
+
+    def cleanup_grant(self, db_user, remote_ip):
+        cursor = self.connection.cursor()
+        try:
+            cursor.execute("DROP FROM mysql.user WHERE user='{}' "
+                           "AND HOST='{}'".format(db_user,
+                                                  remote_ip))
+        finally:
+            cursor.close()
+
+    def execute(self, sql):
+        """Execute arbitary SQL against the database."""
+        cursor = self.connection.cursor()
+        try:
+            cursor.execute(sql)
+        finally:
+            cursor.close()
+
+    def migrate_passwords_to_peer_relation(self):
+        """Migrate any passwords storage on disk to cluster peer relation."""
+        dirname = os.path.dirname(self.root_passwd_file_template)
+        path = os.path.join(dirname, '*.passwd')
+        for f in glob.glob(path):
+            _key = os.path.basename(f)
+            with open(f, 'r') as passwd:
+                _value = passwd.read().strip()
+
+            try:
+                peer_store(_key, _value)
+                os.unlink(f)
+            except ValueError:
+                # NOTE cluster relation not yet ready - skip for now
+                pass
+
+    def get_mysql_password_on_disk(self, username=None, password=None):
+        """Retrieve, generate or store a mysql password for the provided
+        username on disk."""
+        if username:
+            template = self.user_passwd_file_template
+            passwd_file = template.format(username)
+        else:
+            passwd_file = self.root_passwd_file_template
+
+        _password = None
+        if os.path.exists(passwd_file):
+            with open(passwd_file, 'r') as passwd:
+                _password = passwd.read().strip()
+        else:
+            mkdir(os.path.dirname(passwd_file), owner='root', group='root',
+                  perms=0o770)
+            # Force permissions - for some reason the chmod in makedirs fails
+            os.chmod(os.path.dirname(passwd_file), 0o770)
+            _password = password or pwgen(length=32)
+            write_file(passwd_file, _password, owner='root', group='root',
+                       perms=0o660)
+
+        return _password
+
+    def get_mysql_password(self, username=None, password=None):
+        """Retrieve, generate or store a mysql password for the provided
+        username using peer relation cluster."""
+        self.migrate_passwords_to_peer_relation()
+        if username:
+            _key = 'mysql-{}.passwd'.format(username)
+        else:
+            _key = 'mysql.passwd'
+
+        try:
+            _password = peer_retrieve(_key)
+            if _password is None:
+                _password = password or pwgen(length=32)
+                peer_store(_key, _password)
+        except ValueError:
+            # cluster relation is not yet started; use on-disk
+            _password = self.get_mysql_password_on_disk(username, password)
+
+        return _password
+
+    def get_mysql_root_password(self, password=None):
+        """Retrieve or generate mysql root password for service units."""
+        return self.get_mysql_password(username=None, password=password)
+
+    def get_allowed_units(self, database, username, relation_id=None):
+        """Get list of units with access grants for database with username.
+
+        This is typically used to provide shared-db relations with a list of
+        which units have been granted access to the given database.
+        """
+        self.connect(password=self.get_mysql_root_password())
+        allowed_units = set()
+        for unit in related_units(relation_id):
+            settings = relation_get(rid=relation_id, unit=unit)
+            # First check for setting with prefix, then without
+            for attr in ["%s_hostname" % (database), 'hostname']:
+                hosts = settings.get(attr, None)
+                if hosts:
+                    break
+
+            if hosts:
+                # hostname can be json-encoded list of hostnames
+                try:
+                    hosts = json.loads(hosts)
+                except ValueError:
+                    hosts = [hosts]
+            else:
+                hosts = [settings['private-address']]
+
+            if hosts:
+                for host in hosts:
+                    if self.grant_exists(database, username, host):
+                        log("Grant exists for host '%s' on db '%s'" %
+                            (host, database), level=DEBUG)
+                        if unit not in allowed_units:
+                            allowed_units.add(unit)
+                    else:
+                        log("Grant does NOT exist for host '%s' on db '%s'" %
+                            (host, database), level=DEBUG)
+            else:
+                log("No hosts found for grant check", level=INFO)
+
+        return allowed_units
+
+    def configure_db(self, hostname, database, username, admin=False):
+        """Configure access to database for username from hostname."""
+        if config_get('prefer-ipv6'):
+            remote_ip = hostname
+        elif hostname != unit_get('private-address'):
+            try:
+                remote_ip = socket.gethostbyname(hostname)
+            except Exception:
+                # socket.gethostbyname doesn't support ipv6
+                remote_ip = hostname
+        else:
+            remote_ip = '127.0.0.1'
+
+        self.connect(password=self.get_mysql_root_password())
+        if not self.database_exists(database):
+            self.create_database(database)
+
+        password = self.get_mysql_password(username)
+        if not self.grant_exists(database, username, remote_ip):
+            if not admin:
+                self.create_grant(database, username, remote_ip, password)
+            else:
+                self.create_admin_grant(username, remote_ip, password)
+
+        return password
+
+
+class PerconaClusterHelper(object):
+
+    # Going for the biggest page size to avoid wasted bytes. InnoDB page size is
+    # 16MB
+    DEFAULT_PAGE_SIZE = 16 * 1024 * 1024
+
+    def human_to_bytes(self, human):
+        """Convert human readable configuration options to bytes."""
+        num_re = re.compile('^[0-9]+$')
+        if num_re.match(human):
+            return human
+
+        factors = {
+            'K': 1024,
+            'M': 1048576,
+            'G': 1073741824,
+            'T': 1099511627776
+        }
+        modifier = human[-1]
+        if modifier in factors:
+            return int(human[:-1]) * factors[modifier]
+
+        if modifier == '%':
+            total_ram = self.human_to_bytes(self.get_mem_total())
+            if self.is_32bit_system() and total_ram > self.sys_mem_limit():
+                total_ram = self.sys_mem_limit()
+            factor = int(human[:-1]) * 0.01
+            pctram = total_ram * factor
+            return int(pctram - (pctram % self.DEFAULT_PAGE_SIZE))
+
+        raise ValueError("Can only convert K,M,G, or T")
+
+    def is_32bit_system(self):
+        """Determine whether system is 32 or 64 bit."""
+        try:
+            return sys.maxsize < 2 ** 32
+        except OverflowError:
+            return False
+
+    def sys_mem_limit(self):
+        """Determine the default memory limit for the current service unit."""
+        if platform.machine() in ['armv7l']:
+            _mem_limit = self.human_to_bytes('2700M')  # experimentally determined
+        else:
+            # Limit for x86 based 32bit systems
+            _mem_limit = self.human_to_bytes('4G')
+
+        return _mem_limit
+
+    def get_mem_total(self):
+        """Calculate the total memory in the current service unit."""
+        with open('/proc/meminfo') as meminfo_file:
+            for line in meminfo_file:
+                key, mem = line.split(':', 2)
+                if key == 'MemTotal':
+                    mtot, modifier = mem.strip().split(' ')
+                    return '%s%s' % (mtot, upper(modifier[0]))
+
+    def parse_config(self):
+        """Parse charm configuration and calculate values for config files."""
+        config = config_get()
+        mysql_config = {}
+        if 'max-connections' in config:
+            mysql_config['max_connections'] = config['max-connections']
+
+        # Total memory available for dataset
+        dataset_bytes = self.human_to_bytes(config['dataset-size'])
+        mysql_config['dataset_bytes'] = dataset_bytes
+
+        if 'query-cache-type' in config:
+            # Query Cache Configuration
+            mysql_config['query_cache_size'] = config['query-cache-size']
+            if (config['query-cache-size'] == -1 and
+                    config['query-cache-type'] in ['ON', 'DEMAND']):
+                # Calculate the query cache size automatically
+                qcache_bytes = (dataset_bytes * 0.20)
+                qcache_bytes = int(qcache_bytes -
+                                   (qcache_bytes % self.DEFAULT_PAGE_SIZE))
+                mysql_config['query_cache_size'] = qcache_bytes
+                dataset_bytes -= qcache_bytes
+
+            # 5.5 allows the words, but not 5.1
+            if config['query-cache-type'] == 'ON':
+                mysql_config['query_cache_type'] = 1
+            elif config['query-cache-type'] == 'DEMAND':
+                mysql_config['query_cache_type'] = 2
+            else:
+                mysql_config['query_cache_type'] = 0
+
+        # Set a sane default key_buffer size
+        mysql_config['key_buffer'] = self.human_to_bytes('32M')
+
+        if 'preferred-storage-engine' in config:
+            # Storage engine configuration
+            preferred_engines = config['preferred-storage-engine'].split(',')
+            chunk_size = int(dataset_bytes / len(preferred_engines))
+            mysql_config['innodb_flush_log_at_trx_commit'] = 1
+            mysql_config['sync_binlog'] = 1
+            if 'InnoDB' in preferred_engines:
+                mysql_config['innodb_buffer_pool_size'] = chunk_size
+                if config['tuning-level'] == 'fast':
+                    mysql_config['innodb_flush_log_at_trx_commit'] = 2
+            else:
+                mysql_config['innodb_buffer_pool_size'] = 0
+
+            mysql_config['default_storage_engine'] = preferred_engines[0]
+            if 'MyISAM' in preferred_engines:
+                mysql_config['key_buffer'] = chunk_size
+
+            if config['tuning-level'] == 'fast':
+                mysql_config['sync_binlog'] = 0
+
+        return mysql_config
 
=== modified file 'hooks/charmhelpers/contrib/network/__init__.py'
--- hooks/charmhelpers/contrib/network/__init__.py	2014-09-21 21:48:22 +0000
+++ hooks/charmhelpers/contrib/network/__init__.py	2015-02-10 11:19:03 +0000
@@ -0,0 +1,15 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
 
=== modified file 'hooks/charmhelpers/contrib/network/ip.py'
--- hooks/charmhelpers/contrib/network/ip.py	2014-11-26 12:31:33 +0000
+++ hooks/charmhelpers/contrib/network/ip.py	2015-02-10 11:19:03 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 import glob
 import re
 import subprocess
 
=== added directory 'hooks/charmhelpers/contrib/peerstorage'
=== added file 'hooks/charmhelpers/contrib/peerstorage/__init__.py'
--- hooks/charmhelpers/contrib/peerstorage/__init__.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/contrib/peerstorage/__init__.py	2015-02-10 11:19:03 +0000
@@ -0,0 +1,148 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+import six
+from charmhelpers.core.hookenv import relation_id as current_relation_id
+from charmhelpers.core.hookenv import (
+    is_relation_made,
+    relation_ids,
+    relation_get,
+    local_unit,
+    relation_set,
+)
+
+
+"""
+This helper provides functions to support use of a peer relation
+for basic key/value storage, with the added benefit that all storage
+can be replicated across peer units.
+
+Requirement to use:
+
+To use this, the "peer_echo()" method has to be called form the peer
+relation's relation-changed hook:
+
[e-mail]("cluster-relation-changed") # Adapt the to your peer relation name
+def cluster_relation_changed():
+    peer_echo()
+
+Once this is done, you can use peer storage from anywhere:
+
[e-mail]("some-hook")
+def some_hook():
+    # You can store and retrieve key/values this way:
+    if is_relation_made("cluster"):  # from charmhelpers.core.hookenv
+        # There are peers available so we can work with peer storage
+        peer_store("mykey", "myvalue")
+        value = peer_retrieve("mykey")
+        print value
+    else:
+        print "No peers joind the relation, cannot share key/values :("
+"""
+
+
+def peer_retrieve(key, relation_name='cluster'):
+    """Retrieve a named key from peer relation `relation_name`."""
+    cluster_rels = relation_ids(relation_name)
+    if len(cluster_rels) > 0:
+        cluster_rid = cluster_rels[0]
+        return relation_get(attribute=key, rid=cluster_rid,
+                            unit=local_unit())
+    else:
+        raise ValueError('Unable to detect'
+                         'peer relation {}'.format(relation_name))
+
+
+def peer_retrieve_by_prefix(prefix, relation_name='cluster', delimiter='_',
+                            inc_list=None, exc_list=None):
+    """ Retrieve k/v pairs given a prefix and filter using {inc,exc}_list """
+    inc_list = inc_list if inc_list else []
+    exc_list = exc_list if exc_list else []
+    peerdb_settings = peer_retrieve('-', relation_name=relation_name)
+    matched = {}
+    for k, v in peerdb_settings.items():
+        full_prefix = prefix + delimiter
+        if k.startswith(full_prefix):
+            new_key = k.replace(full_prefix, '')
+            if new_key in exc_list:
+                continue
+            if new_key in inc_list or len(inc_list) == 0:
+                matched[new_key] = v
+    return matched
+
+
+def peer_store(key, value, relation_name='cluster'):
+    """Store the key/value pair on the named peer relation `relation_name`."""
+    cluster_rels = relation_ids(relation_name)
+    if len(cluster_rels) > 0:
+        cluster_rid = cluster_rels[0]
+        relation_set(relation_id=cluster_rid,
+                     relation_settings={key: value})
+    else:
+        raise ValueError('Unable to detect '
+                         'peer relation {}'.format(relation_name))
+
+
+def peer_echo(includes=None):
+    """Echo filtered attributes back onto the same relation for storage.
+
+    This is a requirement to use the peerstorage module - it needs to be called
+    from the peer relation's changed hook.
+    """
+    rdata = relation_get()
+    echo_data = {}
+    if includes is None:
+        echo_data = rdata.copy()
+        for ex in ['private-address', 'public-address']:
+            if ex in echo_data:
+                echo_data.pop(ex)
+    else:
+        for attribute, value in six.iteritems(rdata):
+            for include in includes:
+                if include in attribute:
+                    echo_data[attribute] = value
+    if len(echo_data) > 0:
+        relation_set(relation_settings=echo_data)
+
+
+def peer_store_and_set(relation_id=None, peer_relation_name='cluster',
+                       peer_store_fatal=False, relation_settings=None,
+                       delimiter='_', **kwargs):
+    """Store passed-in arguments both in argument relation and in peer storage.
+
+    It functions like doing relation_set() and peer_store() at the same time,
+    with the same data.
+
+    @param relation_id: the id of the relation to store the data on. Defaults
+                        to the current relation.
+    @param peer_store_fatal: Set to True, the function will raise an exception
+                             should the peer sotrage not be avialable."""
+
+    relation_settings = relation_settings if relation_settings else {}
+    relation_set(relation_id=relation_id,
+                 relation_settings=relation_settings,
+                 **kwargs)
+    if is_relation_made(peer_relation_name):
+        for key, value in six.iteritems(dict(list(kwargs.items()) +
+                                             list(relation_settings.items()))):
+            key_prefix = relation_id or current_relation_id()
+            peer_store(key_prefix + delimiter + key,
+                       value,
+                       relation_name=peer_relation_name)
+    else:
+        if peer_store_fatal:
+            raise ValueError('Unable to detect '
+                             'peer relation {}'.format(peer_relation_name))
 
=== modified file 'hooks/charmhelpers/core/__init__.py'
--- hooks/charmhelpers/core/__init__.py	2014-02-19 14:49:31 +0000
+++ hooks/charmhelpers/core/__init__.py	2015-02-10 11:19:03 +0000
@@ -0,0 +1,15 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
 
=== added file 'hooks/charmhelpers/core/decorators.py'
--- hooks/charmhelpers/core/decorators.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/core/decorators.py	2015-02-10 11:19:03 +0000
@@ -0,0 +1,57 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+#
+# Copyright 2014 Canonical Ltd.
+#
+# Authors:
+#  Edward Hope-Morley <[e-mail]>
+#
+
+import time
+
+from charmhelpers.core.hookenv import (
+    log,
+    INFO,
+)
+
+
+def retry_on_exception(num_retries, base_delay=0, exc_type=Exception):
+    """If the decorated function raises exception exc_type, allow num_retries
+    retry attempts before raise the exception.
+    """
+    def _retry_on_exception_inner_1(f):
+        def _retry_on_exception_inner_2(*args, **kwargs):
+            retries = num_retries
+            multiplier = 1
+            while True:
+                try:
+                    return f(*args, **kwargs)
+                except exc_type:
+                    if not retries:
+                        raise
+
+                delay = base_delay * multiplier
+                multiplier += 1
+                log("Retrying '%s' %d more times (delay=%s)" %
+                    (f.__name__, retries, delay), level=INFO)
+                retries -= 1
+                if delay:
+                    time.sleep(delay)
+
+        return _retry_on_exception_inner_2
+
+    return _retry_on_exception_inner_1
 
=== modified file 'hooks/charmhelpers/core/fstab.py'
--- hooks/charmhelpers/core/fstab.py	2014-11-26 12:31:33 +0000
+++ hooks/charmhelpers/core/fstab.py	2015-02-10 11:19:03 +0000
@@ -1,6 +1,22 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 __author__ = 'Jorge Niedbalski R. <[e-mail]>'
 
 import io
 
=== modified file 'hooks/charmhelpers/core/hookenv.py'
--- hooks/charmhelpers/core/hookenv.py	2014-11-26 12:31:33 +0000
+++ hooks/charmhelpers/core/hookenv.py	2015-02-10 11:19:03 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 "Interactions with the Juju environment"
 # Copyright 2013 Canonical Ltd.
 #
@@ -68,6 +84,8 @@
     command = ['juju-log']
     if level:
         command += ['-l', level]
+    if not isinstance(message, six.string_types):
+        message = repr(message)
     command += [message]
     subprocess.call(command)
 
@@ -394,21 +412,31 @@
 
 
 @cached
+def metadata():
+    """Get the current charm metadata.yaml contents as a python object"""
+    with open(os.path.join(charm_dir(), 'metadata.yaml')) as md:
+        return yaml.safe_load(md)
+
+
+@cached
 def relation_types():
     """Get a list of relation types supported by this charm"""
-    charmdir = os.environ.get('CHARM_DIR', '')
-    mdf = open(os.path.join(charmdir, 'metadata.yaml'))
-    md = yaml.safe_load(mdf)
     rel_types = []
+    md = metadata()
     for key in ('provides', 'requires', 'peers'):
         section = md.get(key)
         if section:
             rel_types.extend(section.keys())
-    mdf.close()
     return rel_types
 
 
 @cached
+def charm_name():
+    """Get the name of the current charm as is specified on metadata.yaml"""
+    return metadata().get('name')
+
+
+@cached
 def relations():
     """Get a nested dictionary of relation data for all related units"""
     rels = {}
 
=== modified file 'hooks/charmhelpers/core/host.py'
--- hooks/charmhelpers/core/host.py	2014-11-26 12:31:33 +0000
+++ hooks/charmhelpers/core/host.py	2015-02-10 11:19:03 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 """Tools for working with the host system"""
 # Copyright 2012 Canonical Ltd.
 #
@@ -101,6 +117,26 @@
     return user_info
 
 
+def add_group(group_name, system_group=False):
+    """Add a group to the system"""
+    try:
+        group_info = grp.getgrnam(group_name)
+        log('group {0} already exists!'.format(group_name))
+    except KeyError:
+        log('creating group {0}'.format(group_name))
+        cmd = ['addgroup']
+        if system_group:
+            cmd.append('--system')
+        else:
+            cmd.extend([
+                '--group',
+            ])
+        cmd.append(group_name)
+        subprocess.check_call(cmd)
+        group_info = grp.getgrnam(group_name)
+    return group_info
+
+
 def add_user_to_group(username, group):
     """Add a user to a group"""
     cmd = [
@@ -142,21 +178,24 @@
     uid = pwd.getpwnam(owner).pw_uid
     gid = grp.getgrnam(group).gr_gid
     realpath = os.path.abspath(path)
-    if os.path.exists(realpath):
-        if force and not os.path.isdir(realpath):
+    path_exists = os.path.exists(realpath)
+    if path_exists and force:
+        if not os.path.isdir(realpath):
             log("Removing non-directory file {} prior to mkdir()".format(path))
             os.unlink(realpath)
-    else:
+            os.makedirs(realpath, perms)
+    elif not path_exists:
         os.makedirs(realpath, perms)
     os.chown(realpath, uid, gid)
+    os.chmod(realpath, perms)
 
 
 def write_file(path, content, owner='root', group='root', perms=0o444):
-    """Create or overwrite a file with the contents of a string"""
+    """Create or overwrite a file with the contents of a byte string."""
     log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))
     uid = pwd.getpwnam(owner).pw_uid
     gid = grp.getgrnam(group).gr_gid
-    with open(path, 'w') as target:
+    with open(path, 'wb') as target:
         os.fchown(target.fileno(), uid, gid)
         os.fchmod(target.fileno(), perms)
         target.write(content)
@@ -322,7 +361,7 @@
         ip_output = (line for line in ip_output if line)
         for line in ip_output:
             if line.split()[1].startswith(int_type):
-                matched = re.search('.*: (bond[0-9]+\.[0-9]+)@.*', line)
+                matched = re.search('.*: (' + int_type + r'[0-9]+\.[0-9]+)@.*', line)
                 if matched:
                     interface = matched.groups()[0]
                 else:
@@ -366,10 +405,13 @@
     *  0 => Installed revno is the same as supplied arg
     * -1 => Installed revno is less than supplied arg
 
+    This function imports apt_cache function from charmhelpers.fetch if
+    the pkgcache argument is None. Be sure to add charmhelpers.fetch if
+    you call this function, or pass an apt_pkg.Cache() instance.
     '''
     import apt_pkg
-    from charmhelpers.fetch import apt_cache
     if not pkgcache:
+        from charmhelpers.fetch import apt_cache
         pkgcache = apt_cache()
     pkg = pkgcache[package]
     return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)
@@ -384,13 +426,21 @@
         os.chdir(cur)
 
 
-def chownr(path, owner, group):
+def chownr(path, owner, group, follow_links=True):
     uid = pwd.getpwnam(owner).pw_uid
     gid = grp.getgrnam(group).gr_gid
+    if follow_links:
+        chown = os.chown
+    else:
+        chown = os.lchown
 
     for root, dirs, files in os.walk(path):
         for name in dirs + files:
             full = os.path.join(root, name)
             broken_symlink = os.path.lexists(full) and not os.path.exists(full)
             if not broken_symlink:
-                os.chown(full, uid, gid)
+                chown(full, uid, gid)
+
+
+def lchownr(path, owner, group):
+    chownr(path, owner, group, follow_links=False)
 
=== modified file 'hooks/charmhelpers/core/services/__init__.py'
--- hooks/charmhelpers/core/services/__init__.py	2014-10-22 14:38:57 +0000
+++ hooks/charmhelpers/core/services/__init__.py	2015-02-10 11:19:03 +0000
@@ -1,2 +1,18 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 from .base import *  # NOQA
 from .helpers import *  # NOQA
 
=== modified file 'hooks/charmhelpers/core/services/base.py'
--- hooks/charmhelpers/core/services/base.py	2014-09-26 07:17:08 +0000
+++ hooks/charmhelpers/core/services/base.py	2015-02-10 11:19:03 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 import os
 import re
 import json
 
=== modified file 'hooks/charmhelpers/core/services/helpers.py'
--- hooks/charmhelpers/core/services/helpers.py	2014-11-26 12:31:33 +0000
+++ hooks/charmhelpers/core/services/helpers.py	2015-02-10 11:19:03 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 import os
 import yaml
 from charmhelpers.core import hookenv
 
=== modified file 'hooks/charmhelpers/core/sysctl.py'
--- hooks/charmhelpers/core/sysctl.py	2014-10-09 10:30:08 +0000
+++ hooks/charmhelpers/core/sysctl.py	2015-02-10 11:19:03 +0000
@@ -1,6 +1,22 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 __author__ = 'Jorge Niedbalski R. <[e-mail]>'
 
 import yaml
@@ -10,25 +26,31 @@
 from charmhelpers.core.hookenv import (
     log,
     DEBUG,
+    ERROR,
 )
 
 
 def create(sysctl_dict, sysctl_file):
     """Creates a sysctl.conf file from a YAML associative array
 
-    :param sysctl_dict: a dict of sysctl options eg { 'kernel.max_pid': 1337 }
-    :type sysctl_dict: dict
+    :param sysctl_dict: a YAML-formatted string of sysctl options eg "{ 'kernel.max_pid': 1337 }"
+    :type sysctl_dict: str
     :param sysctl_file: path to the sysctl file to be saved
     :type sysctl_file: str or unicode
     :returns: None
     """
-    sysctl_dict = yaml.load(sysctl_dict)
+    try:
+        sysctl_dict_parsed = yaml.safe_load(sysctl_dict)
+    except yaml.YAMLError:
+        log("Error parsing YAML sysctl_dict: {}".format(sysctl_dict),
+            level=ERROR)
+        return
 
     with open(sysctl_file, "w") as fd:
-        for key, value in sysctl_dict.items():
+        for key, value in sysctl_dict_parsed.items():
             fd.write("{}={}\n".format(key, value))
 
-    log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict),
+    log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict_parsed),
         level=DEBUG)
 
     check_call(["sysctl", "-p", sysctl_file])
 
=== modified file 'hooks/charmhelpers/core/templating.py'
--- hooks/charmhelpers/core/templating.py	2014-11-26 12:31:33 +0000
+++ hooks/charmhelpers/core/templating.py	2015-02-10 11:19:03 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 import os
 
 from charmhelpers.core import host
@@ -5,7 +21,7 @@
 
 
 def render(source, target, context, owner='root', group='root',
-           perms=0o444, templates_dir=None):
+           perms=0o444, templates_dir=None, encoding='UTF-8'):
     """
     Render a template.
 
@@ -48,5 +64,5 @@
                     level=hookenv.ERROR)
         raise e
     content = template.render(context)
-    host.mkdir(os.path.dirname(target))
-    host.write_file(target, content, owner, group, perms)
+    host.mkdir(os.path.dirname(target), owner, group, perms=0o755)
+    host.write_file(target, content.encode(encoding), owner, group, perms)
 
=== modified file 'hooks/charmhelpers/fetch/__init__.py'
--- hooks/charmhelpers/fetch/__init__.py	2014-11-26 12:31:33 +0000
+++ hooks/charmhelpers/fetch/__init__.py	2015-02-10 11:19:03 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 import importlib
 from tempfile import NamedTemporaryFile
 import time
@@ -64,9 +80,16 @@
     'trusty-juno/updates': 'trusty-updates/juno',
     'trusty-updates/juno': 'trusty-updates/juno',
     'juno/proposed': 'trusty-proposed/juno',
-    'juno/proposed': 'trusty-proposed/juno',
     'trusty-juno/proposed': 'trusty-proposed/juno',
     'trusty-proposed/juno': 'trusty-proposed/juno',
+    # Kilo
+    'kilo': 'trusty-updates/kilo',
+    'trusty-kilo': 'trusty-updates/kilo',
+    'trusty-kilo/updates': 'trusty-updates/kilo',
+    'trusty-updates/kilo': 'trusty-updates/kilo',
+    'kilo/proposed': 'trusty-proposed/kilo',
+    'trusty-kilo/proposed': 'trusty-proposed/kilo',
+    'trusty-proposed/kilo': 'trusty-proposed/kilo',
 }
 
 # The order of this list is very important. Handlers should be listed in from
 
=== modified file 'hooks/charmhelpers/fetch/archiveurl.py'
--- hooks/charmhelpers/fetch/archiveurl.py	2014-11-26 12:31:33 +0000
+++ hooks/charmhelpers/fetch/archiveurl.py	2015-02-10 11:19:03 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 import os
 import hashlib
 import re
 
=== modified file 'hooks/charmhelpers/fetch/bzrurl.py'
--- hooks/charmhelpers/fetch/bzrurl.py	2014-11-26 12:31:33 +0000
+++ hooks/charmhelpers/fetch/bzrurl.py	2015-02-10 11:19:03 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 import os
 from charmhelpers.fetch import (
     BaseFetchHandler,
@@ -11,10 +27,12 @@
 
 try:
     from bzrlib.branch import Branch
+    from bzrlib import bzrdir, workingtree, errors
 except ImportError:
     from charmhelpers.fetch import apt_install
     apt_install("python-bzrlib")
     from bzrlib.branch import Branch
+    from bzrlib import bzrdir, workingtree, errors
 
 
 class BzrUrlFetchHandler(BaseFetchHandler):
@@ -35,8 +53,14 @@
             from bzrlib.plugin import load_plugins
             load_plugins()
         try:
+            local_branch = bzrdir.BzrDir.create_branch_convenience(dest)
+        except errors.AlreadyControlDirError:
+            local_branch = Branch.open(dest)
+        try:
             remote_branch = Branch.open(source)
-            remote_branch.bzrdir.sprout(dest).open_branch()
+            remote_branch.push(local_branch)
+            tree = workingtree.WorkingTree.open(dest)
+            tree.update()
         except Exception as e:
             raise e
 
 
=== modified file 'hooks/charmhelpers/fetch/giturl.py'
--- hooks/charmhelpers/fetch/giturl.py	2014-11-26 12:31:33 +0000
+++ hooks/charmhelpers/fetch/giturl.py	2015-02-10 11:19:03 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 import os
 from charmhelpers.fetch import (
     BaseFetchHandler,
@@ -16,6 +32,8 @@
     apt_install("python-git")
     from git import Repo
 
+from git.exc import GitCommandError
+
 
 class GitUrlFetchHandler(BaseFetchHandler):
     """Handler for git branches via generic and github URLs"""
@@ -34,15 +52,20 @@
         repo = Repo.clone_from(source, dest)
         repo.git.checkout(branch)
 
-    def install(self, source, branch="master"):
+    def install(self, source, branch="master", dest=None):
         url_parts = self.parse_url(source)
         branch_name = url_parts.path.strip("/").split("/")[-1]
-        dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
-                                branch_name)
+        if dest:
+            dest_dir = os.path.join(dest, branch_name)
+        else:
+            dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
+                                    branch_name)
         if not os.path.exists(dest_dir):
             mkdir(dest_dir, perms=0o755)
         try:
             self.clone(source, dest_dir, branch)
+        except GitCommandError as e:
+            raise UnhandledSource(e.message)
         except OSError as e:
             raise UnhandledSource(e.strerror)
         return dest_dir
 
=== modified file 'hooks/common.py'
--- hooks/common.py	2015-01-07 14:18:57 +0000
+++ hooks/common.py	2015-02-10 11:19:03 +0000
@@ -6,6 +6,7 @@
 import shutil
 from charmhelpers.core import hookenv, host
 from charmhelpers.core.templating import render
+from charmhelpers.contrib.database.mysql import MySQLHelper
 
 
 def get_service_user_file(service):
@@ -60,71 +61,19 @@
 broken = os.path.exists(broken_path)
 
 
+def get_db_helper():
+    return MySQLHelper(rpasswdf_template='/var/lib/mysql/mysql.passwd',
+                       upasswdf_template='/var/lib/mysql/mysql-{}.passwd')
+
+
 def get_db_cursor():
     # Connect to mysql
-    passwd = open("/var/lib/mysql/mysql.passwd").read().strip()
+    db_helper = get_db_helper()
+    passwd = db_helper.get_mysql_root_password()
     connection = MySQLdb.connect(user="root", host="localhost", passwd=passwd)
     return connection.cursor()
 
 
-def database_exists(db_name):
-    cursor = get_db_cursor()
-    try:
-        cursor.execute("SHOW DATABASES")
-        databases = [i[0] for i in cursor.fetchall()]
-    finally:
-        cursor.close()
-    return db_name in databases
-
-
-def create_database(db_name):
-    cursor = get_db_cursor()
-    try:
-        cursor.execute("CREATE DATABASE {}".format(db_name))
-    finally:
-        cursor.close()
-
-
-def grant_exists(db_name, db_user, remote_ip):
-    cursor = get_db_cursor()
-    priv_string = "GRANT ALL PRIVILEGES ON `{}`.* " \
-                  "TO '{}'@'{}'".format(db_name, db_user, remote_ip)
-    try:
-        cursor.execute("SHOW GRANTS for '{}'@'{}'".format(db_user,
-                                                          remote_ip))
-        grants = [i[0] for i in cursor.fetchall()]
-    except MySQLdb.OperationalError:
-        print "No grants found"
-        return False
-    finally:
-        cursor.close()
-    return priv_string in grants
-
-
-def create_grant(db_name, db_user,
-                 remote_ip, password):
-    cursor = get_db_cursor()
-    try:
-        cursor.execute("GRANT ALL PRIVILEGES ON {}.* TO '{}'@'{}' "
-                       "IDENTIFIED BY '{}'".format(db_name,
-                                                   db_user,
-                                                   remote_ip,
-                                                   password))
-    finally:
-        cursor.close()
-
-
-def cleanup_grant(db_user,
-                  remote_ip):
-    cursor = get_db_cursor()
-    try:
-        cursor.execute("DROP FROM mysql.user WHERE user='{}' "
-                       "AND HOST='{}'".format(db_user,
-                                              remote_ip))
-    finally:
-        cursor.close()
-
-
 def migrate_to_mount(new_path):
     """Invoked when new mountpoint appears. This function safely migrates
     MySQL data from local disk to persistent storage (only if needed)
 
=== modified file 'hooks/ha_relations.py'
--- hooks/ha_relations.py	2014-09-21 22:03:41 +0000
+++ hooks/ha_relations.py	2015-02-10 11:19:03 +0000
@@ -7,6 +7,10 @@
 import lib.ceph_utils as ceph
 import lib.cluster_utils as cluster
 
+from charmhelpers.contrib.peerstorage import (
+    peer_echo,
+)
+
 # CEPH
 DATA_SRC_DST = '/var/lib/mysql'
 SERVICE_NAME = os.getenv('JUJU_UNIT_NAME').split('/')[0]
@@ -14,6 +18,11 @@
 LEADER_RES = 'res_mysql_vip'
 
 
+def cluster_changed():
+    # Echo any passwords placed on peer relation
+    peer_echo(includes=['.passwd'])
+
+
 def ha_relation_joined():
     vip = utils.config_get('vip')
     vip_iface = utils.config_get('vip_iface')
@@ -148,6 +157,7 @@
     "ha-relation-changed": ha_relation_changed,
     "ceph-relation-joined": ceph_joined,
     "ceph-relation-changed": ceph_changed,
+    "cluster-relation-changed": cluster_changed,
 }
 
 utils.do_hooks(hooks)
 
=== modified file 'hooks/shared_db_relations.py'
--- hooks/shared_db_relations.py	2014-11-26 16:35:19 +0000
+++ hooks/shared_db_relations.py	2015-02-10 11:19:03 +0000
@@ -7,18 +7,12 @@
 #
 # Author: Adam Gandelman <[e-mail]>
 
-
-from common import (
-    database_exists,
-    create_database,
-    grant_exists,
-    create_grant)
 import subprocess
 import json
-import socket
-import os
 import lib.utils as utils
 import lib.cluster_utils as cluster
+
+from common import get_db_helper
 from charmhelpers.core import hookenv
 from charmhelpers.contrib.network.ip import (
     get_ipv6_addr
@@ -50,71 +44,6 @@
 
 
 def shared_db_changed():
-
-    def get_allowed_units(database, username):
-        allowed_units = set()
-        for relid in hookenv.relation_ids('shared-db'):
-            for unit in hookenv.related_units(relid):
-                attr = "%s_%s" % (database, 'hostname')
-                hosts = hookenv.relation_get(attribute=attr, unit=unit,
-                                             rid=relid)
-                if not hosts:
-                    hosts = [hookenv.relation_get(attribute='private-address',
-                                                  unit=unit, rid=relid)]
-                else:
-                    # hostname can be json-encoded list of hostnames
-                    try:
-                        hosts = json.loads(hosts)
-                    except ValueError:
-                        pass
-
-                if not isinstance(hosts, list):
-                    hosts = [hosts]
-
-                if hosts:
-                    for host in hosts:
-                        utils.juju_log('INFO', "Checking host '%s' grant" %
-                                       (host))
-                        if grant_exists(database, username, host):
-                            if unit not in allowed_units:
-                                allowed_units.add(unit)
-                else:
-                    utils.juju_log('INFO', "No hosts found for grant check")
-
-        return allowed_units
-
-    def configure_db(hostname,
-                     database,
-                     username):
-        passwd_file = "/var/lib/mysql/mysql-{}.passwd".format(username)
-        if hostname != local_hostname:
-            try:
-                remote_ip = socket.gethostbyname(hostname)
-            except Exception:
-                # socket.gethostbyname doesn't support ipv6
-                remote_ip = hostname
-        else:
-            remote_ip = '127.0.0.1'
-
-        if not os.path.exists(passwd_file):
-            password = pwgen()
-            with open(passwd_file, 'w') as pfile:
-                pfile.write(password)
-                os.chmod(pfile.name, 0600)
-        else:
-            with open(passwd_file) as pfile:
-                password = pfile.read().strip()
-
-        if not database_exists(database):
-            create_database(database)
-        if not grant_exists(database,
-                            username,
-                            remote_ip):
-            create_grant(database,
-                         username,
-                         remote_ip, password)
-        return password
-
     if not cluster.eligible_leader(LEADER_RES):
         utils.juju_log('INFO',
                        'MySQL service is peered, bailing shared-db relation'
@@ -132,6 +61,8 @@
         'username',
         'hostname'])
 
+    db_helper = get_db_helper()
+
     if singleset.issubset(settings):
         # Process a single database configuration
         hostname = settings['hostname']
@@ -142,26 +73,23 @@
         try:
             hostname = json.loads(hostname)
         except ValueError:
-            pass
-
-        if isinstance(hostname, list):
-            for host in hostname:
-                password = configure_db(host, database, username)
-        else:
-            password = configure_db(hostname, database, username)
-
-        allowed_units = " ".join(unit_sorted(get_allowed_units(database,
-                                                               username)))
-
-        if not cluster.is_clustered():
-            utils.relation_set(db_host=local_hostname,
-                               password=password,
-                               allowed_units=allowed_units)
-        else:
-            utils.relation_set(db_host=utils.config_get("vip"),
-                               password=password,
-                               allowed_units=allowed_units)
-
+            hostname = [hostname]
+
+        for host in hostname:
+            password = db_helper.configure_db(host, database, username)
+
+        allowed_units = db_helper.get_allowed_units(database, username)
+        allowed_units = unit_sorted(allowed_units)
+        allowed_units = ' '.join(allowed_units)
+
+        if cluster.is_clustered():
+            db_host = utils.config_get("vip")
+        else:
+            db_host = local_hostname
+
+        utils.relation_set(db_host=db_host,
+                           password=password,
+                           allowed_units=allowed_units)
     else:
         # Process multiple database setup requests.
         # from incoming relation data:
@@ -195,22 +123,23 @@
                 database = databases[db]['database']
                 hostname = databases[db]['hostname']
                 username = databases[db]['username']
+
                 try:
+                    # Can be json-encoded list of hostnames
                     hostname = json.loads(hostname)
                 except ValueError:
-                    hostname = hostname
-
-                if isinstance(hostname, list):
-                    for host in hostname:
-                        password = configure_db(host, database, username)
-                else:
-                    password = configure_db(hostname, database, username)
-
-                return_data['_'.join([db, 'password'])] = password
-                allowed_units = unit_sorted(get_allowed_units(database,
-                                                              username))
-                return_data['_'.join([db, 'allowed_units'])] = \
-                    " ".join(allowed_units)
+                    # Otherwise expected to be single hostname
+                    hostname = [hostname]
+
+                for host in hostname:
+                    password = db_helper.configure_db(host, database, username)
+
+                a_units = db_helper.get_allowed_units(database, username)
+                a_units = ' '.join(unit_sorted(a_units))
+                return_data['%s_allowed_units' % (db)] = a_units
+
+                return_data['%s_password' % (db)] = password
+
         if len(return_data) > 0:
             utils.relation_set(**return_data)
         if not cluster.is_clustered():