Merge lp:~daniel-thewatkins/charms/trusty/ubuntu-repository-cache/update_charm-helpers into lp:charms/trusty/ubuntu-repository-cache

[Dan Watkins]

We need new charm-helpers for Nagios stuff and some other minor fixes; we were carrying some of those fixes as patches, which have been dropped.

[José Antonio Rey (jose)]

Hey Daniel!

Unfortunately there are several merge conflicts in this branch. Would you mind checking and fixing them so we can do a clean merge when approved?

Thanks for your contributions to the Charm Store!

review: Needs Fixing

[Dan Watkins (daniel-thewatkins)]

Thanks, bzr. Thzr.[0] I'll look at these today.

[0] https://www.youtube.com/watch?v=9jtU9BbReQk

[Dan Watkins (daniel-thewatkins)]

Thanks, bzr. Thzr.[0] I'll look at these today.

[0] https://www.youtube.com/watch?v=9jtU9BbReQk

[Dan Watkins (daniel-thewatkins)]

Thanks, bzr. Thzr.[0] I'll look at these today.

[0] https://www.youtube.com/watch?v=9jtU9BbReQk

[Dan Watkins (daniel-thewatkins)]

Thanks, bzr. Thzr.[0] I'll look at these today.

[0] https://www.youtube.com/watch?v=9jtU9BbReQk

[Dan Watkins (daniel-thewatkins)]

Thanks, bzr. Thzr.[0] I'll look at these today.

[0] https://www.youtube.com/watch?v=9jtU9BbReQk

[Dan Watkins (daniel-thewatkins)]

Right, this is fixed and ready for review now.

[Adam Israel (aisrael)]

Hi Daniel,

This merges cleanly for me. +1

review: Approve

[lp:~daniel-thewatkins/charms/trusty/ubuntu-repository-cache/update_charm-helpers]

194. By José Antonio Rey on 2015-06-04

Daniel Watkins 2015-06-01 Set PATH in templates/cron/ubuntu-repository-cache_rsync.cron.

[José Antonio Rey (jose)]

Daniel,

Thanks for fixing the merge conflicts. This now merges clean, so I'm going ahead and pushing.

+1 LGTM!

Thanks again for your contributions to the Charm Store!

review: Approve

[lp:~daniel-thewatkins/charms/trusty/ubuntu-repository-cache/update_charm-helpers]

195. By José Antonio Rey on 2015-06-04

Daniel Watkins 2015-06-02 Revert "storage.linux.util: Fix swapped is_device_mounted regex logic (lp:1370053)"
Daniel Watkins 2015-06-03 Update charm-helpers.
Daniel Watkins 2015-06-03 Remove superseded patches.
Daniel Watkins 2015-06-03 Refresh and apply remaining patches.

=== modified file 'lib/charmhelpers/contrib/charmsupport/__init__.py'
--- lib/charmhelpers/contrib/charmsupport/__init__.py	2014-09-18 21:30:22 +0000
+++ lib/charmhelpers/contrib/charmsupport/__init__.py	2015-06-04 13:22:19 +0000
@@ -0,0 +1,15 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.

=== modified file 'lib/charmhelpers/contrib/charmsupport/nrpe.py'
--- lib/charmhelpers/contrib/charmsupport/nrpe.py	2014-09-18 21:30:22 +0000
+++ lib/charmhelpers/contrib/charmsupport/nrpe.py	2015-06-04 13:22:19 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 """Compatibility with the nrpe-external-master charm"""
 # Copyright 2012 Canonical Ltd.
 #
@@ -8,6 +24,8 @@
 import pwd
 import grp
 import os
+import glob
+import shutil
 import re
 import shlex
 import yaml
@@ -18,6 +36,7 @@
     log,
     relation_ids,
     relation_set,
+    relations_of_type,
 )
 
 from charmhelpers.core.host import service
@@ -54,6 +73,12 @@
 #            juju-myservice-0
 #        If you're running multiple environments with the same services in them
 #        this allows you to differentiate between them.
+#    nagios_servicegroups:
+#      default: ""
+#      type: string
+#      description: |
+#        A comma-separated list of nagios servicegroups.
+#        If left empty, the nagios_context will be used as the servicegroup
 #
 # 3. Add custom checks (Nagios plugins) to files/nrpe-external-master
 #
@@ -138,7 +163,7 @@
         log('Check command not found: {}'.format(parts[0]))
         return ''
 
-    def write(self, nagios_context, hostname):
+    def write(self, nagios_context, hostname, nagios_servicegroups):
         nrpe_check_file = '/etc/nagios/nrpe.d/{}.cfg'.format(
             self.command)
         with open(nrpe_check_file, 'w') as nrpe_check_config:
@@ -150,16 +175,18 @@
             log('Not writing service config as {} is not accessible'.format(
                 NRPE.nagios_exportdir))
         else:
-            self.write_service_config(nagios_context, hostname)
+            self.write_service_config(nagios_context, hostname,
+                                      nagios_servicegroups)
 
-    def write_service_config(self, nagios_context, hostname):
+    def write_service_config(self, nagios_context, hostname,
+                             nagios_servicegroups):
         for f in os.listdir(NRPE.nagios_exportdir):
             if re.search('.*{}.cfg'.format(self.command), f):
                 os.remove(os.path.join(NRPE.nagios_exportdir, f))
 
         templ_vars = {
             'nagios_hostname': hostname,
-            'nagios_servicegroup': nagios_context,
+            'nagios_servicegroup': nagios_servicegroups,
             'description': self.description,
             'shortname': self.shortname,
             'command': self.command,
@@ -183,6 +210,10 @@
         super(NRPE, self).__init__()
         self.config = config()
         self.nagios_context = self.config['nagios_context']
+        if 'nagios_servicegroups' in self.config and self.config['nagios_servicegroups']:
+            self.nagios_servicegroups = self.config['nagios_servicegroups']
+        else:
+            self.nagios_servicegroups = self.nagios_context
         self.unit_name = local_unit().replace('/', '-')
         if hostname:
             self.hostname = hostname
@@ -208,12 +239,122 @@
         nrpe_monitors = {}
         monitors = {"monitors": {"remote": {"nrpe": nrpe_monitors}}}
         for nrpecheck in self.checks:
-            nrpecheck.write(self.nagios_context, self.hostname)
+            nrpecheck.write(self.nagios_context, self.hostname,
+                            self.nagios_servicegroups)
             nrpe_monitors[nrpecheck.shortname] = {
                 "command": nrpecheck.command,
             }
 
         service('restart', 'nagios-nrpe-server')
 
-        for rid in relation_ids("local-monitors"):
+        monitor_ids = relation_ids("local-monitors") + \
+            relation_ids("nrpe-external-master")
+        for rid in monitor_ids:
             relation_set(relation_id=rid, monitors=yaml.dump(monitors))
+
+
+def get_nagios_hostcontext(relation_name='nrpe-external-master'):
+    """
+    Query relation with nrpe subordinate, return the nagios_host_context
+
+    :param str relation_name: Name of relation nrpe sub joined to
+    """
+    for rel in relations_of_type(relation_name):
+        if 'nagios_hostname' in rel:
+            return rel['nagios_host_context']
+
+
+def get_nagios_hostname(relation_name='nrpe-external-master'):
+    """
+    Query relation with nrpe subordinate, return the nagios_hostname
+
+    :param str relation_name: Name of relation nrpe sub joined to
+    """
+    for rel in relations_of_type(relation_name):
+        if 'nagios_hostname' in rel:
+            return rel['nagios_hostname']
+
+
+def get_nagios_unit_name(relation_name='nrpe-external-master'):
+    """
+    Return the nagios unit name prepended with host_context if needed
+
+    :param str relation_name: Name of relation nrpe sub joined to
+    """
+    host_context = get_nagios_hostcontext(relation_name)
+    if host_context:
+        unit = "%s:%s" % (host_context, local_unit())
+    else:
+        unit = local_unit()
+    return unit
+
+
+def add_init_service_checks(nrpe, services, unit_name):
+    """
+    Add checks for each service in list
+
+    :param NRPE nrpe: NRPE object to add check to
+    :param list services: List of services to check
+    :param str unit_name: Unit name to use in check description
+    """
+    for svc in services:
+        upstart_init = '/etc/init/%s.conf' % svc
+        sysv_init = '/etc/init.d/%s' % svc
+        if os.path.exists(upstart_init):
+            nrpe.add_check(
+                shortname=svc,
+                description='process check {%s}' % unit_name,
+                check_cmd='check_upstart_job %s' % svc
+            )
+        elif os.path.exists(sysv_init):
+            cronpath = '/etc/cron.d/nagios-service-check-%s' % svc
+            cron_file = ('*/5 * * * * root '
+                         '/usr/local/lib/nagios/plugins/check_exit_status.pl '
+                         '-s /etc/init.d/%s status > '
+                         '/var/lib/nagios/service-check-%s.txt\n' % (svc,
+                                                                     svc)
+                         )
+            f = open(cronpath, 'w')
+            f.write(cron_file)
+            f.close()
+            nrpe.add_check(
+                shortname=svc,
+                description='process check {%s}' % unit_name,
+                check_cmd='check_status_file.py -f '
+                          '/var/lib/nagios/service-check-%s.txt' % svc,
+            )
+
+
+def copy_nrpe_checks():
+    """
+    Copy the nrpe checks into place
+
+    """
+    NAGIOS_PLUGINS = '/usr/local/lib/nagios/plugins'
+    nrpe_files_dir = os.path.join(os.getenv('CHARM_DIR'), 'hooks',
+                                  'charmhelpers', 'contrib', 'openstack',
+                                  'files')
+
+    if not os.path.exists(NAGIOS_PLUGINS):
+        os.makedirs(NAGIOS_PLUGINS)
+    for fname in glob.glob(os.path.join(nrpe_files_dir, "check_*")):
+        if os.path.isfile(fname):
+            shutil.copy2(fname,
+                         os.path.join(NAGIOS_PLUGINS, os.path.basename(fname)))
+
+
+def add_haproxy_checks(nrpe, unit_name):
+    """
+    Add checks for each service in list
+
+    :param NRPE nrpe: NRPE object to add check to
+    :param str unit_name: Unit name to use in check description
+    """
+    nrpe.add_check(
+        shortname='haproxy_servers',
+        description='Check HAProxy {%s}' % unit_name,
+        check_cmd='check_haproxy.sh')
+    nrpe.add_check(
+        shortname='haproxy_queue',
+        description='Check HAProxy queue depth {%s}' % unit_name,
+        check_cmd='check_haproxy_queue_depth.sh')

=== modified file 'lib/charmhelpers/contrib/hahelpers/__init__.py'
--- lib/charmhelpers/contrib/hahelpers/__init__.py	2014-08-26 13:59:01 +0000
+++ lib/charmhelpers/contrib/hahelpers/__init__.py	2015-06-04 13:22:19 +0000
@@ -0,0 +1,15 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.

=== modified file 'lib/charmhelpers/contrib/hahelpers/apache.py'
--- lib/charmhelpers/contrib/hahelpers/apache.py	2014-08-26 13:59:01 +0000
+++ lib/charmhelpers/contrib/hahelpers/apache.py	2015-06-04 13:22:19 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 #
 # Copyright 2012 Canonical Ltd.
 #
@@ -20,20 +36,27 @@
 )
 
 
-def get_cert():
+def get_cert(cn=None):
+    # TODO: deal with multiple https endpoints via charm config
     cert = config_get('ssl_cert')
     key = config_get('ssl_key')
     if not (cert and key):
         log("Inspecting identity-service relations for SSL certificate.",
             level=INFO)
         cert = key = None
+        if cn:
+            ssl_cert_attr = 'ssl_cert_{}'.format(cn)
+            ssl_key_attr = 'ssl_key_{}'.format(cn)
+        else:
+            ssl_cert_attr = 'ssl_cert'
+            ssl_key_attr = 'ssl_key'
         for r_id in relation_ids('identity-service'):
             for unit in relation_list(r_id):
                 if not cert:
-                    cert = relation_get('ssl_cert',
+                    cert = relation_get(ssl_cert_attr,
                                         rid=r_id, unit=unit)
                 if not key:
-                    key = relation_get('ssl_key',
+                    key = relation_get(ssl_key_attr,
                                        rid=r_id, unit=unit)
     return (cert, key)
 

=== modified file 'lib/charmhelpers/contrib/hahelpers/cluster.py'
--- lib/charmhelpers/contrib/hahelpers/cluster.py	2014-08-26 13:59:01 +0000
+++ lib/charmhelpers/contrib/hahelpers/cluster.py	2015-06-04 13:22:19 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 #
 # Copyright 2012 Canonical Ltd.
 #
@@ -16,6 +32,8 @@
 
 from socket import gethostname as get_unit_hostname
 
+import six
+
 from charmhelpers.core.hookenv import (
     log,
     relation_ids,
@@ -27,12 +45,24 @@
     WARNING,
     unit_get,
 )
+from charmhelpers.core.decorators import (
+    retry_on_exception,
+)
+from charmhelpers.core.strutils import (
+    bool_from_string,
+)
+
+DC_RESOURCE_NAME = 'DC'
 
 
 class HAIncompleteConfig(Exception):
     pass
 
 
+class CRMResourceNotFound(Exception):
+    pass
+
+
 def is_elected_leader(resource):
     """
     Returns True if the charm executing this is the elected cluster leader.
@@ -67,24 +97,53 @@
     return False
 
 
-def is_crm_leader(resource):
+def is_crm_dc():
+    """
+    Determine leadership by querying the pacemaker Designated Controller
+    """
+    cmd = ['crm', 'status']
+    try:
+        status = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+        if not isinstance(status, six.text_type):
+            status = six.text_type(status, "utf-8")
+    except subprocess.CalledProcessError:
+        return False
+    current_dc = ''
+    for line in status.split('\n'):
+        if line.startswith('Current DC'):
+            # Current DC: juju-lytrusty-machine-2 (168108163) - partition with quorum
+            current_dc = line.split(':')[1].split()[0]
+    if current_dc == get_unit_hostname():
+        return True
+    return False
+
+
+@retry_on_exception(5, base_delay=2, exc_type=CRMResourceNotFound)
+def is_crm_leader(resource, retry=False):
     """
     Returns True if the charm calling this is the elected corosync leader,
     as returned by calling the external "crm" command.
+
+    We allow this operation to be retried to avoid the possibility of getting a
+    false negative. See LP #1396246 for more info.
     """
-    cmd = [
-        "crm", "resource",
-        "show", resource
-    ]
+    if resource == DC_RESOURCE_NAME:
+        return is_crm_dc()
+    cmd = ['crm', 'resource', 'show', resource]
     try:
-        status = subprocess.check_output(cmd)
+        status = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+        if not isinstance(status, six.text_type):
+            status = six.text_type(status, "utf-8")
     except subprocess.CalledProcessError:
-        return False
-    else:
-        if get_unit_hostname() in status:
-            return True
-        else:
-            return False
+        status = None
+
+    if status and get_unit_hostname() in status:
+        return True
+
+    if status and "resource %s is NOT running" % (resource) in status:
+        raise CRMResourceNotFound("CRM resource %s not found" % (resource))
+
+    return False
 
 
 def is_leader(resource):
@@ -133,16 +192,16 @@
     .
     returns: boolean
     '''
-    if config_get('use-https') == "yes":
+    use_https = config_get('use-https')
+    if use_https and bool_from_string(use_https):
         return True
     if config_get('ssl_cert') and config_get('ssl_key'):
         return True
     for r_id in relation_ids('identity-service'):
         for unit in relation_list(r_id):
+            # TODO - needs fixing for new helper as ssl_cert/key suffixes with CN
             rel_state = [
                 relation_get('https_keystone', rid=r_id, unit=unit),
-                relation_get('ssl_cert', rid=r_id, unit=unit),
-                relation_get('ssl_key', rid=r_id, unit=unit),
                 relation_get('ca_cert', rid=r_id, unit=unit),
             ]
             # NOTE: works around (LP: #1203241)
@@ -151,54 +210,66 @@
     return False
 
 
-def determine_api_port(public_port):
+def determine_api_port(public_port, singlenode_mode=False):
     '''
     Determine correct API server listening port based on
     existence of HTTPS reverse proxy and/or haproxy.
 
     public_port: int: standard public port for given service
 
+    singlenode_mode: boolean: Shuffle ports when only a single unit is present
+
     returns: int: the correct listening port for the API service
     '''
     i = 0
-    if len(peer_units()) > 0 or is_clustered():
+    if singlenode_mode:
+        i += 1
+    elif len(peer_units()) > 0 or is_clustered():
         i += 1
     if https():
         i += 1
     return public_port - (i * 10)
 
 
-def determine_apache_port(public_port):
+def determine_apache_port(public_port, singlenode_mode=False):
     '''
     Description: Determine correct apache listening port based on public IP +
     state of the cluster.
 
     public_port: int: standard public port for given service
 
+    singlenode_mode: boolean: Shuffle ports when only a single unit is present
+
     returns: int: the correct listening port for the HAProxy service
     '''
     i = 0
-    if len(peer_units()) > 0 or is_clustered():
+    if singlenode_mode:
+        i += 1
+    elif len(peer_units()) > 0 or is_clustered():
         i += 1
     return public_port - (i * 10)
 
 
-def get_hacluster_config():
+def get_hacluster_config(exclude_keys=None):
     '''
     Obtains all relevant configuration from charm configuration required
     for initiating a relation to hacluster:
 
         ha-bindiface, ha-mcastport, vip
 
+    param: exclude_keys: list of setting key(s) to be excluded.
     returns: dict: A dict containing settings keyed by setting name.
     raises: HAIncompleteConfig if settings are missing.
     '''
     settings = ['ha-bindiface', 'ha-mcastport', 'vip']
     conf = {}
     for setting in settings:
+        if exclude_keys and setting in exclude_keys:
+            continue
+
         conf[setting] = config_get(setting)
     missing = []
-    [missing.append(s) for s, v in conf.iteritems() if v is None]
+    [missing.append(s) for s, v in six.iteritems(conf) if v is None]
     if missing:
         log('Insufficient config data to configure hacluster.', level=ERROR)
         raise HAIncompleteConfig

=== modified file 'lib/charmhelpers/contrib/storage/__init__.py'
--- lib/charmhelpers/contrib/storage/__init__.py	2014-09-15 14:42:27 +0000
+++ lib/charmhelpers/contrib/storage/__init__.py	2015-06-04 13:22:19 +0000
@@ -0,0 +1,15 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.

=== modified file 'lib/charmhelpers/contrib/storage/linux/__init__.py'
--- lib/charmhelpers/contrib/storage/linux/__init__.py	2014-09-15 14:42:27 +0000
+++ lib/charmhelpers/contrib/storage/linux/__init__.py	2015-06-04 13:22:19 +0000
@@ -0,0 +1,15 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.

=== modified file 'lib/charmhelpers/contrib/storage/linux/ceph.py'
--- lib/charmhelpers/contrib/storage/linux/ceph.py	2014-09-15 14:42:27 +0000
+++ lib/charmhelpers/contrib/storage/linux/ceph.py	2015-06-04 13:22:19 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 #
 # Copyright 2012 Canonical Ltd.
 #
@@ -16,19 +32,18 @@
 from subprocess import (
     check_call,
     check_output,
-    CalledProcessError
+    CalledProcessError,
 )
-
 from charmhelpers.core.hookenv import (
     relation_get,
     relation_ids,
     related_units,
     log,
+    DEBUG,
     INFO,
     WARNING,
-    ERROR
+    ERROR,
 )
-
 from charmhelpers.core.host import (
     mount,
     mounts,
@@ -37,7 +52,6 @@
     service_running,
     umount,
 )
-
 from charmhelpers.fetch import (
     apt_install,
 )
@@ -56,99 +70,85 @@
 
 
 def install():
-    ''' Basic Ceph client installation '''
+    """Basic Ceph client installation."""
     ceph_dir = "/etc/ceph"
     if not os.path.exists(ceph_dir):
         os.mkdir(ceph_dir)
+
     apt_install('ceph-common', fatal=True)
 
 
 def rbd_exists(service, pool, rbd_img):
-    ''' Check to see if a RADOS block device exists '''
+    """Check to see if a RADOS block device exists."""
     try:
-        out = check_output(['rbd', 'list', '--id', service,
-                            '--pool', pool])
+        out = check_output(['rbd', 'list', '--id',
+                            service, '--pool', pool]).decode('UTF-8')
     except CalledProcessError:
         return False
-    else:
-        return rbd_img in out
+
+    return rbd_img in out
 
 
 def create_rbd_image(service, pool, image, sizemb):
-    ''' Create a new RADOS block device '''
-    cmd = [
-        'rbd',
-        'create',
-        image,
-        '--size',
-        str(sizemb),
-        '--id',
-        service,
-        '--pool',
-        pool
-    ]
+    """Create a new RADOS block device."""
+    cmd = ['rbd', 'create', image, '--size', str(sizemb), '--id', service,
+           '--pool', pool]
     check_call(cmd)
 
 
 def pool_exists(service, name):
-    ''' Check to see if a RADOS pool already exists '''
+    """Check to see if a RADOS pool already exists."""
     try:
-        out = check_output(['rados', '--id', service, 'lspools'])
+        out = check_output(['rados', '--id', service,
+                            'lspools']).decode('UTF-8')
     except CalledProcessError:
         return False
-    else:
-        return name in out
+
+    return name in out
 
 
 def get_osds(service):
-    '''
-    Return a list of all Ceph Object Storage Daemons
-    currently in the cluster
-    '''
+    """Return a list of all Ceph Object Storage Daemons currently in the
+    cluster.
+    """
     version = ceph_version()
     if version and version >= '0.56':
         return json.loads(check_output(['ceph', '--id', service,
-                                        'osd', 'ls', '--format=json']))
-    else:
-        return None
-
-
-def create_pool(service, name, replicas=2):
-    ''' Create a new RADOS pool '''
+                                        'osd', 'ls',
+                                        '--format=json']).decode('UTF-8'))
+
+    return None
+
+
+def create_pool(service, name, replicas=3):
+    """Create a new RADOS pool."""
     if pool_exists(service, name):
         log("Ceph pool {} already exists, skipping creation".format(name),
             level=WARNING)
         return
+
     # Calculate the number of placement groups based
     # on upstream recommended best practices.
     osds = get_osds(service)
     if osds:
-        pgnum = (len(osds) * 100 / replicas)
+        pgnum = (len(osds) * 100 // replicas)
     else:
         # NOTE(james-page): Default to 200 for older ceph versions
         # which don't support OSD query from cli
         pgnum = 200
-    cmd = [
-        'ceph', '--id', service,
-        'osd', 'pool', 'create',
-        name, str(pgnum)
-    ]
+
+    cmd = ['ceph', '--id', service, 'osd', 'pool', 'create', name, str(pgnum)]
     check_call(cmd)
-    cmd = [
-        'ceph', '--id', service,
-        'osd', 'pool', 'set', name,
-        'size', str(replicas)
-    ]
+
+    cmd = ['ceph', '--id', service, 'osd', 'pool', 'set', name, 'size',
+           str(replicas)]
     check_call(cmd)
 
 
 def delete_pool(service, name):
-    ''' Delete a RADOS pool from ceph '''
-    cmd = [
-        'ceph', '--id', service,
-        'osd', 'pool', 'delete',
-        name, '--yes-i-really-really-mean-it'
-    ]
+    """Delete a RADOS pool from ceph."""
+    cmd = ['ceph', '--id', service, 'osd', 'pool', 'delete', name,
+           '--yes-i-really-really-mean-it']
     check_call(cmd)
 
 
@@ -161,44 +161,54 @@
 
 
 def create_keyring(service, key):
-    ''' Create a new Ceph keyring containing key'''
+    """Create a new Ceph keyring containing key."""
     keyring = _keyring_path(service)
     if os.path.exists(keyring):
-        log('ceph: Keyring exists at %s.' % keyring, level=WARNING)
+        log('Ceph keyring exists at %s.' % keyring, level=WARNING)
         return
-    cmd = [
-        'ceph-authtool',
-        keyring,
-        '--create-keyring',
-        '--name=client.{}'.format(service),
-        '--add-key={}'.format(key)
-    ]
+
+    cmd = ['ceph-authtool', keyring, '--create-keyring',
+           '--name=client.{}'.format(service), '--add-key={}'.format(key)]
     check_call(cmd)
-    log('ceph: Created new ring at %s.' % keyring, level=INFO)
+    log('Created new ceph keyring at %s.' % keyring, level=DEBUG)
+
+
+def delete_keyring(service):
+    """Delete an existing Ceph keyring."""
+    keyring = _keyring_path(service)
+    if not os.path.exists(keyring):
+        log('Keyring does not exist at %s' % keyring, level=WARNING)
+        return
+
+    os.remove(keyring)
+    log('Deleted ring at %s.' % keyring, level=INFO)
 
 
 def create_key_file(service, key):
-    ''' Create a file containing key '''
+    """Create a file containing key."""
     keyfile = _keyfile_path(service)
     if os.path.exists(keyfile):
-        log('ceph: Keyfile exists at %s.' % keyfile, level=WARNING)
+        log('Keyfile exists at %s.' % keyfile, level=WARNING)
         return
+
     with open(keyfile, 'w') as fd:
         fd.write(key)
-    log('ceph: Created new keyfile at %s.' % keyfile, level=INFO)
+
+    log('Created new keyfile at %s.' % keyfile, level=INFO)
 
 
 def get_ceph_nodes():
-    ''' Query named relation 'ceph' to detemine current nodes '''
+    """Query named relation 'ceph' to determine current nodes."""
     hosts = []
     for r_id in relation_ids('ceph'):
         for unit in related_units(r_id):
             hosts.append(relation_get('private-address', unit=unit, rid=r_id))
+
     return hosts
 
 
 def configure(service, key, auth, use_syslog):
-    ''' Perform basic configuration of Ceph '''
+    """Perform basic configuration of Ceph."""
     create_keyring(service, key)
     create_key_file(service, key)
     hosts = get_ceph_nodes()
@@ -211,17 +221,17 @@
 
 
 def image_mapped(name):
-    ''' Determine whether a RADOS block device is mapped locally '''
+    """Determine whether a RADOS block device is mapped locally."""
     try:
-        out = check_output(['rbd', 'showmapped'])
+        out = check_output(['rbd', 'showmapped']).decode('UTF-8')
     except CalledProcessError:
         return False
-    else:
-        return name in out
+
+    return name in out
 
 
 def map_block_storage(service, pool, image):
-    ''' Map a RADOS block device for local use '''
+    """Map a RADOS block device for local use."""
     cmd = [
         'rbd',
         'map',
@@ -235,31 +245,32 @@
 
 
 def filesystem_mounted(fs):
-    ''' Determine whether a filesytems is already mounted '''
+    """Determine whether a filesytems is already mounted."""
     return fs in [f for f, m in mounts()]
 
 
 def make_filesystem(blk_device, fstype='ext4', timeout=10):
-    ''' Make a new filesystem on the specified block device '''
+    """Make a new filesystem on the specified block device."""
     count = 0
     e_noent = os.errno.ENOENT
     while not os.path.exists(blk_device):
         if count >= timeout:
-            log('ceph: gave up waiting on block device %s' % blk_device,
+            log('Gave up waiting on block device %s' % blk_device,
                 level=ERROR)
             raise IOError(e_noent, os.strerror(e_noent), blk_device)
-        log('ceph: waiting for block device %s to appear' % blk_device,
-            level=INFO)
+
+        log('Waiting for block device %s to appear' % blk_device,
+            level=DEBUG)
         count += 1
         time.sleep(1)
     else:
-        log('ceph: Formatting block device %s as filesystem %s.' %
+        log('Formatting block device %s as filesystem %s.' %
             (blk_device, fstype), level=INFO)
         check_call(['mkfs', '-t', fstype, blk_device])
 
 
 def place_data_on_block_device(blk_device, data_src_dst):
-    ''' Migrate data in data_src_dst to blk_device and then remount '''
+    """Migrate data in data_src_dst to blk_device and then remount."""
     # mount block device into /mnt
     mount(blk_device, '/mnt')
     # copy data to /mnt
@@ -279,8 +290,8 @@
 
 # TODO: re-use
 def modprobe(module):
-    ''' Load a kernel module and configure for auto-load on reboot '''
-    log('ceph: Loading kernel module', level=INFO)
+    """Load a kernel module and configure for auto-load on reboot."""
+    log('Loading kernel module', level=INFO)
     cmd = ['modprobe', module]
     check_call(cmd)
     with open('/etc/modules', 'r+') as modules:
@@ -289,7 +300,7 @@
 
 
 def copy_files(src, dst, symlinks=False, ignore=None):
-    ''' Copy files from src to dst '''
+    """Copy files from src to dst."""
     for item in os.listdir(src):
         s = os.path.join(src, item)
         d = os.path.join(dst, item)
@@ -300,9 +311,9 @@
 
 
 def ensure_ceph_storage(service, pool, rbd_img, sizemb, mount_point,
-                        blk_device, fstype, system_services=[]):
-    """
-    NOTE: This function must only be called from a single service unit for
+                        blk_device, fstype, system_services=[],
+                        replicas=3):
+    """NOTE: This function must only be called from a single service unit for
     the same rbd_img otherwise data loss will occur.
 
     Ensures given pool and RBD image exists, is mapped to a block device,
@@ -316,15 +327,16 @@
     """
     # Ensure pool, RBD image, RBD mappings are in place.
     if not pool_exists(service, pool):
-        log('ceph: Creating new pool {}.'.format(pool))
-        create_pool(service, pool)
+        log('Creating new pool {}.'.format(pool), level=INFO)
+        create_pool(service, pool, replicas=replicas)
 
     if not rbd_exists(service, pool, rbd_img):
-        log('ceph: Creating RBD image ({}).'.format(rbd_img))
+        log('Creating RBD image ({}).'.format(rbd_img), level=INFO)
         create_rbd_image(service, pool, rbd_img, sizemb)
 
     if not image_mapped(rbd_img):
-        log('ceph: Mapping RBD Image {} as a Block Device.'.format(rbd_img))
+        log('Mapping RBD Image {} as a Block Device.'.format(rbd_img),
+            level=INFO)
         map_block_storage(service, pool, rbd_img)
 
     # make file system
@@ -339,45 +351,47 @@
 
         for svc in system_services:
             if service_running(svc):
-                log('ceph: Stopping services {} prior to migrating data.'
-                    .format(svc))
+                log('Stopping services {} prior to migrating data.'
+                    .format(svc), level=DEBUG)
                 service_stop(svc)
 
         place_data_on_block_device(blk_device, mount_point)
 
         for svc in system_services:
-            log('ceph: Starting service {} after migrating data.'
-                .format(svc))
+            log('Starting service {} after migrating data.'
+                .format(svc), level=DEBUG)
             service_start(svc)
 
 
 def ensure_ceph_keyring(service, user=None, group=None):
-    '''
-    Ensures a ceph keyring is created for a named service
-    and optionally ensures user and group ownership.
+    """Ensures a ceph keyring is created for a named service and optionally
+    ensures user and group ownership.
 
     Returns False if no ceph key is available in relation state.
-    '''
+    """
     key = None
     for rid in relation_ids('ceph'):
         for unit in related_units(rid):
             key = relation_get('key', rid=rid, unit=unit)
             if key:
                 break
+
     if not key:
         return False
+
     create_keyring(service=service, key=key)
     keyring = _keyring_path(service)
     if user and group:
         check_call(['chown', '%s.%s' % (user, group), keyring])
+
     return True
 
 
 def ceph_version():
-    ''' Retrieve the local version of ceph '''
+    """Retrieve the local version of ceph."""
     if os.path.exists('/usr/bin/ceph'):
         cmd = ['ceph', '-v']
-        output = check_output(cmd)
+        output = check_output(cmd).decode('US-ASCII')
         output = output.split()
         if len(output) > 3:
             return output[2]
@@ -385,3 +399,46 @@
             return None
     else:
         return None
+
+
+class CephBrokerRq(object):
+    """Ceph broker request.
+
+    Multiple operations can be added to a request and sent to the Ceph broker
+    to be executed.
+
+    Request is json-encoded for sending over the wire.
+
+    The API is versioned and defaults to version 1.
+    """
+    def __init__(self, api_version=1):
+        self.api_version = api_version
+        self.ops = []
+
+    def add_op_create_pool(self, name, replica_count=3):
+        self.ops.append({'op': 'create-pool', 'name': name,
+                         'replicas': replica_count})
+
+    @property
+    def request(self):
+        return json.dumps({'api-version': self.api_version, 'ops': self.ops})
+
+
+class CephBrokerRsp(object):
+    """Ceph broker response.
+
+    Response is json-decoded and contents provided as methods/properties.
+
+    The API is versioned and defaults to version 1.
+    """
+    def __init__(self, encoded_rsp):
+        self.api_version = None
+        self.rsp = json.loads(encoded_rsp)
+
+    @property
+    def exit_code(self):
+        return self.rsp.get('exit-code')
+
+    @property
+    def exit_msg(self):
+        return self.rsp.get('stderr')

=== modified file 'lib/charmhelpers/contrib/storage/linux/loopback.py'
--- lib/charmhelpers/contrib/storage/linux/loopback.py	2014-09-15 14:42:27 +0000
+++ lib/charmhelpers/contrib/storage/linux/loopback.py	2015-06-04 13:22:19 +0000
@@ -1,12 +1,28 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
 
 import os
 import re
-
 from subprocess import (
     check_call,
     check_output,
 )
 
+import six
+
 
 ##################################################
 # loopback device helpers.
@@ -37,7 +53,7 @@
     '''
     file_path = os.path.abspath(file_path)
     check_call(['losetup', '--find', file_path])
-    for d, f in loopback_devices().iteritems():
+    for d, f in six.iteritems(loopback_devices()):
         if f == file_path:
             return d
 
@@ -51,7 +67,7 @@
 
     :returns: str: Full path to the ensured loopback device (eg, /dev/loop0)
     '''
-    for d, f in loopback_devices().iteritems():
+    for d, f in six.iteritems(loopback_devices()):
         if f == path:
             return d
 

=== modified file 'lib/charmhelpers/contrib/storage/linux/lvm.py'
--- lib/charmhelpers/contrib/storage/linux/lvm.py	2014-09-15 14:42:27 +0000
+++ lib/charmhelpers/contrib/storage/linux/lvm.py	2015-06-04 13:22:19 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 from subprocess import (
     CalledProcessError,
     check_call,
@@ -61,6 +77,7 @@
     vg = None
     pvd = check_output(['pvdisplay', block_device]).splitlines()
     for l in pvd:
+        l = l.decode('UTF-8')
         if l.strip().startswith('VG Name'):
             vg = ' '.join(l.strip().split()[2:])
     return vg

=== modified file 'lib/charmhelpers/contrib/storage/linux/utils.py'
--- lib/charmhelpers/contrib/storage/linux/utils.py	2014-09-16 13:52:19 +0000
+++ lib/charmhelpers/contrib/storage/linux/utils.py	2015-06-04 13:22:19 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 import os
 import re
 from stat import S_ISBLK
@@ -30,7 +46,8 @@
     # sometimes sgdisk exits non-zero; this is OK, dd will clean up
     call(['sgdisk', '--zap-all', '--mbrtogpt',
           '--clear', block_device])
-    dev_end = check_output(['blockdev', '--getsz', block_device])
+    dev_end = check_output(['blockdev', '--getsz',
+                            block_device]).decode('UTF-8')
     gpt_end = int(dev_end.split()[0]) - 100
     check_call(['dd', 'if=/dev/zero', 'of=%s' % (block_device),
                 'bs=1M', 'count=1'])
@@ -47,7 +64,7 @@
         it doesn't.
     '''
     is_partition = bool(re.search(r".*[0-9]+\b", device))
-    out = check_output(['mount'])
-    if not is_partition:
+    out = check_output(['mount']).decode('UTF-8')
+    if is_partition:
         return bool(re.search(device + r"\b", out))
     return bool(re.search(device + r"[0-9]+\b", out))

=== modified file 'lib/charmhelpers/contrib/unison/__init__.py'
--- lib/charmhelpers/contrib/unison/__init__.py	2014-09-05 14:44:30 +0000
+++ lib/charmhelpers/contrib/unison/__init__.py	2015-06-04 13:22:19 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 # Easy file synchronization among peer units using ssh + unison.
 #
 # For the -joined, -changed, and -departed peer relations, add a call to
@@ -54,6 +70,7 @@
 from charmhelpers.core.host import (
     adduser,
     add_user_to_group,
+    pwgen,
 )
 
 from charmhelpers.core.hookenv import (
@@ -64,6 +81,7 @@
     relation_set,
     relation_get,
     unit_private_ip,
+    INFO,
     ERROR,
 )
 
@@ -72,13 +90,12 @@
             '-prefer=newer', '-times=true']
 
 
-def get_homedir(username):
+def get_homedir(user):
     try:
-        user = pwd.getpwnam(username)
-        log('get_homedir: {} homdir is {}'.format(username, user.pw_dir))
+        user = pwd.getpwnam(user)
         return user.pw_dir
     except KeyError:
-        log('Could not get homedir for user %s: user exists?', ERROR)
+        log('Could not get homedir for user %s: user exists?' % (user), ERROR)
         raise Exception
 
 
@@ -131,7 +148,7 @@
     ssh_dir = os.path.join(home_dir, '.ssh')
     auth_keys = os.path.join(ssh_dir, 'authorized_keys')
     log('Syncing authorized_keys @ %s.' % auth_keys)
-    with open(auth_keys, 'wb') as out:
+    with open(auth_keys, 'w') as out:
         for k in keys:
             out.write('%s\n' % k)
 
@@ -143,16 +160,16 @@
     khosts = []
     for host in hosts:
         cmd = ['ssh-keyscan', host]
-        remote_key = check_output(cmd).strip()
+        remote_key = check_output(cmd, universal_newlines=True).strip()
         khosts.append(remote_key)
     log('Syncing known_hosts @ %s.' % known_hosts)
-    with open(known_hosts, 'wb') as out:
+    with open(known_hosts, 'w') as out:
         for host in khosts:
             out.write('%s\n' % host)
 
 
 def ensure_user(user, group=None):
-    adduser(user)
+    adduser(user, pwgen())
     if group:
         add_user_to_group(user, group)
 
@@ -194,13 +211,14 @@
         relation_set(ssh_authorized_hosts=authed_hosts)
 
 
-def _run_as_user(user):
+def _run_as_user(user, gid=None):
     try:
         user = pwd.getpwnam(user)
     except KeyError:
         log('Invalid user: %s' % user)
         raise Exception
-    uid, gid = user.pw_uid, user.pw_gid
+    uid = user.pw_uid
+    gid = gid or user.pw_gid
     os.environ['HOME'] = user.pw_dir
 
     def _inner():
@@ -209,8 +227,8 @@
     return _inner
 
 
-def run_as_user(user, cmd):
-    return check_output(cmd, preexec_fn=_run_as_user(user), cwd='/')
+def run_as_user(user, cmd, gid=None):
+    return check_output(cmd, preexec_fn=_run_as_user(user, gid), cwd='/')
 
 
 def collect_authed_hosts(peer_interface):
@@ -225,19 +243,25 @@
                                         rid=r_id, unit=unit)
 
             if not authed_hosts:
-                log('Peer %s has not authorized *any* hosts yet, skipping.')
+                log('Peer %s has not authorized *any* hosts yet, skipping.' %
+                    (unit), level=INFO)
                 continue
 
             if unit_private_ip() in authed_hosts.split(':'):
                 hosts.append(private_addr)
             else:
-                log('Peer %s has not authorized *this* host yet, skipping.')
-
+                log('Peer %s has not authorized *this* host yet, skipping.' %
+                    (unit), level=INFO)
     return hosts
 
 
-def sync_path_to_host(path, host, user, verbose=False):
-    cmd = copy(BASE_CMD)
+def sync_path_to_host(path, host, user, verbose=False, cmd=None, gid=None,
+                      fatal=False):
+    """Sync path to an specific peer host
+
+    Propagates exception if operation fails and fatal=True.
+    """
+    cmd = cmd or copy(BASE_CMD)
     if not verbose:
         cmd.append('-silent')
 
@@ -250,17 +274,33 @@
 
     try:
         log('Syncing local path %s to %s@%s:%s' % (path, user, host, path))
-        run_as_user(user, cmd)
+        run_as_user(user, cmd, gid)
     except:
         log('Error syncing remote files')
-
-
-def sync_to_peer(host, user, paths=[], verbose=False):
-    '''Sync paths to an specific host'''
-    [sync_path_to_host(p, host, user, verbose) for p in paths]
-
-
-def sync_to_peers(peer_interface, user, paths=[], verbose=False):
-    '''Sync all hosts to an specific path'''
-    for host in collect_authed_hosts(peer_interface):
-        sync_to_peer(host, user, paths, verbose)
+        if fatal:
+            raise
+
+
+def sync_to_peer(host, user, paths=None, verbose=False, cmd=None, gid=None,
+                 fatal=False):
+    """Sync paths to an specific peer host
+
+    Propagates exception if any operation fails and fatal=True.
+    """
+    if paths:
+        for p in paths:
+            sync_path_to_host(p, host, user, verbose, cmd, gid, fatal)
+
+
+def sync_to_peers(peer_interface, user, paths=None, verbose=False, cmd=None,
+                  gid=None, fatal=False):
+    """Sync all hosts to an specific path
+
+    The type of group is integer, it allows user has permissions to
+    operate a directory have a different group id with the user id.
+
+    Propagates exception if any operation fails and fatal=True.
+    """
+    if paths:
+        for host in collect_authed_hosts(peer_interface):
+            sync_to_peer(host, user, paths, verbose, cmd, gid, fatal)

=== modified file 'lib/charmhelpers/core/__init__.py'
--- lib/charmhelpers/core/__init__.py	2014-08-17 21:29:19 +0000
+++ lib/charmhelpers/core/__init__.py	2015-06-04 13:22:19 +0000
@@ -0,0 +1,15 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.

=== modified file 'lib/charmhelpers/core/fstab.py'
--- lib/charmhelpers/core/fstab.py	2014-08-17 21:29:19 +0000
+++ lib/charmhelpers/core/fstab.py	2015-06-04 13:22:19 +0000
@@ -1,12 +1,29 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+import io
+import os
+
 __author__ = 'Jorge Niedbalski R. <[e-mail]>'
 
-import os
-
-
-class Fstab(file):
+
+class Fstab(io.FileIO):
     """This class extends file in order to implement a file reader/writer
     for file `/etc/fstab`
     """
@@ -24,8 +41,8 @@
                 options = "defaults"
 
             self.options = options
-            self.d = d
-            self.p = p
+            self.d = int(d)
+            self.p = int(p)
 
         def __eq__(self, o):
             return str(self) == str(o)
@@ -45,7 +62,7 @@
             self._path = path
         else:
             self._path = self.DEFAULT_PATH
-        file.__init__(self, self._path, 'r+')
+        super(Fstab, self).__init__(self._path, 'rb+')
 
     def _hydrate_entry(self, line):
         # NOTE: use split with no arguments to split on any
@@ -58,8 +75,9 @@
     def entries(self):
         self.seek(0)
         for line in self.readlines():
+            line = line.decode('us-ascii')
             try:
-                if not line.startswith("#"):
+                if line.strip() and not line.strip().startswith("#"):
                     yield self._hydrate_entry(line)
             except ValueError:
                 pass
@@ -75,18 +93,18 @@
         if self.get_entry_by_attr('device', entry.device):
             return False
 
-        self.write(str(entry) + '\n')
+        self.write((str(entry) + '\n').encode('us-ascii'))
         self.truncate()
         return entry
 
     def remove_entry(self, entry):
         self.seek(0)
 
-        lines = self.readlines()
+        lines = [l.decode('us-ascii') for l in self.readlines()]
 
         found = False
         for index, line in enumerate(lines):
-            if not line.startswith("#"):
+            if line.strip() and not line.strip().startswith("#"):
                 if self._hydrate_entry(line) == entry:
                     found = True
                     break
@@ -97,7 +115,7 @@
         lines.remove(line)
 
         self.seek(0)
-        self.write(''.join(lines))
+        self.write(''.join(lines).encode('us-ascii'))
         self.truncate()
         return True
 

=== modified file 'lib/charmhelpers/core/hookenv.py'
--- lib/charmhelpers/core/hookenv.py	2014-09-15 14:42:27 +0000
+++ lib/charmhelpers/core/hookenv.py	2015-06-04 13:22:19 +0000
@@ -1,17 +1,42 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 "Interactions with the Juju environment"
 # Copyright 2013 Canonical Ltd.
 #
 # Authors:
 #  Charm Helpers Developers <[e-mail]>
 
+from __future__ import print_function
+from functools import wraps
 import os
 import json
 import yaml
 import subprocess
 import sys
-import UserDict
+import errno
+import tempfile
 from subprocess import CalledProcessError
 
+import six
+if not six.PY3:
+    from UserDict import UserDict
+else:
+    from collections import UserDict
+
 CRITICAL = "CRITICAL"
 ERROR = "ERROR"
 WARNING = "WARNING"
@@ -35,15 +60,17 @@
 
     will cache the result of unit_get + 'test' for future calls.
     """
+    @wraps(func)
     def wrapper(*args, **kwargs):
         global cache
         key = str((func, args, kwargs))
         try:
             return cache[key]
         except KeyError:
-            res = func(*args, **kwargs)
-            cache[key] = res
-            return res
+            pass  # Drop out of the exception handler scope.
+        res = func(*args, **kwargs)
+        cache[key] = res
+        return res
     return wrapper
 
 
@@ -63,16 +90,29 @@
     command = ['juju-log']
     if level:
         command += ['-l', level]
+    if not isinstance(message, six.string_types):
+        message = repr(message)
     command += [message]
-    subprocess.call(command)
-
-
-class Serializable(UserDict.IterableUserDict):
+    # Missing juju-log should not cause failures in unit tests
+    # Send log output to stderr
+    try:
+        subprocess.call(command)
+    except OSError as e:
+        if e.errno == errno.ENOENT:
+            if level:
+                message = "{}: {}".format(level, message)
+            message = "juju-log: {}".format(message)
+            print(message, file=sys.stderr)
+        else:
+            raise
+
+
+class Serializable(UserDict):
     """Wrapper, an object that can be serialized to yaml or json"""
 
     def __init__(self, obj):
         # wrap the object
-        UserDict.IterableUserDict.__init__(self)
+        UserDict.__init__(self)
         self.data = obj
 
     def __getattr__(self, attr):
@@ -142,7 +182,7 @@
 
 def remote_unit():
     """The remote unit for the current relation hook"""
-    return os.environ['JUJU_REMOTE_UNIT']
+    return os.environ.get('JUJU_REMOTE_UNIT', None)
 
 
 def service_name():
@@ -214,6 +254,18 @@
         except KeyError:
             return (self._prev_dict or {})[key]
 
+    def get(self, key, default=None):
+        try:
+            return self[key]
+        except KeyError:
+            return default
+
+    def keys(self):
+        prev_keys = []
+        if self._prev_dict is not None:
+            prev_keys = self._prev_dict.keys()
+        return list(set(prev_keys + list(dict.keys(self))))
+
     def load_previous(self, path=None):
         """Load previous copy of config from disk.
 
@@ -263,7 +315,7 @@
 
         """
         if self._prev_dict:
-            for k, v in self._prev_dict.iteritems():
+            for k, v in six.iteritems(self._prev_dict):
                 if k not in self:
                     self[k] = v
         with open(self.path, 'w') as f:
@@ -278,7 +330,8 @@
         config_cmd_line.append(scope)
     config_cmd_line.append('--format=json')
     try:
-        config_data = json.loads(subprocess.check_output(config_cmd_line))
+        config_data = json.loads(
+            subprocess.check_output(config_cmd_line).decode('UTF-8'))
         if scope is not None:
             return config_data
         return Config(config_data)
@@ -297,10 +350,10 @@
     if unit:
         _args.append(unit)
     try:
-        return json.loads(subprocess.check_output(_args))
+        return json.loads(subprocess.check_output(_args).decode('UTF-8'))
     except ValueError:
         return None
-    except CalledProcessError, e:
+    except CalledProcessError as e:
         if e.returncode == 2:
             return None
         raise
@@ -310,18 +363,49 @@
     """Set relation information for the current unit"""
     relation_settings = relation_settings if relation_settings else {}
     relation_cmd_line = ['relation-set']
+    accepts_file = "--file" in subprocess.check_output(
+        relation_cmd_line + ["--help"], universal_newlines=True)
     if relation_id is not None:
         relation_cmd_line.extend(('-r', relation_id))
-    for k, v in (relation_settings.items() + kwargs.items()):
-        if v is None:
-            relation_cmd_line.append('{}='.format(k))
-        else:
-            relation_cmd_line.append('{}={}'.format(k, v))
-    subprocess.check_call(relation_cmd_line)
+    settings = relation_settings.copy()
+    settings.update(kwargs)
+    for key, value in settings.items():
+        # Force value to be a string: it always should, but some call
+        # sites pass in things like dicts or numbers.
+        if value is not None:
+            settings[key] = "{}".format(value)
+    if accepts_file:
+        # --file was introduced in Juju 1.23.2. Use it by default if
+        # available, since otherwise we'll break if the relation data is
+        # too big. Ideally we should tell relation-set to read the data from
+        # stdin, but that feature is broken in 1.23.2: Bug #1454678.
+        with tempfile.NamedTemporaryFile(delete=False) as settings_file:
+            settings_file.write(yaml.safe_dump(settings).encode("utf-8"))
+        subprocess.check_call(
+            relation_cmd_line + ["--file", settings_file.name])
+        os.remove(settings_file.name)
+    else:
+        for key, value in settings.items():
+            if value is None:
+                relation_cmd_line.append('{}='.format(key))
+            else:
+                relation_cmd_line.append('{}={}'.format(key, value))
+        subprocess.check_call(relation_cmd_line)
     # Flush cache of any relation-gets for local unit
     flush(local_unit())
 
 
+def relation_clear(r_id=None):
+    ''' Clears any relation data already set on relation r_id '''
+    settings = relation_get(rid=r_id,
+                            unit=local_unit())
+    for setting in settings:
+        if setting not in ['public-address', 'private-address']:
+            settings[setting] = None
+    relation_set(relation_id=r_id,
+                 **settings)
+
+
 @cached
 def relation_ids(reltype=None):
     """A list of relation_ids"""
@@ -329,7 +413,8 @@
     relid_cmd_line = ['relation-ids', '--format=json']
     if reltype is not None:
         relid_cmd_line.append(reltype)
-        return json.loads(subprocess.check_output(relid_cmd_line)) or []
+        return json.loads(
+            subprocess.check_output(relid_cmd_line).decode('UTF-8')) or []
     return []
 
 
@@ -340,7 +425,8 @@
     units_cmd_line = ['relation-list', '--format=json']
     if relid is not None:
         units_cmd_line.extend(('-r', relid))
-    return json.loads(subprocess.check_output(units_cmd_line)) or []
+    return json.loads(
+        subprocess.check_output(units_cmd_line).decode('UTF-8')) or []
 
 
 @cached
@@ -380,21 +466,31 @@
 
 
 @cached
+def metadata():
+    """Get the current charm metadata.yaml contents as a python object"""
+    with open(os.path.join(charm_dir(), 'metadata.yaml')) as md:
+        return yaml.safe_load(md)
+
+
+@cached
 def relation_types():
     """Get a list of relation types supported by this charm"""
-    charmdir = os.environ.get('CHARM_DIR', '')
-    mdf = open(os.path.join(charmdir, 'metadata.yaml'))
-    md = yaml.safe_load(mdf)
     rel_types = []
+    md = metadata()
     for key in ('provides', 'requires', 'peers'):
         section = md.get(key)
         if section:
             rel_types.extend(section.keys())
-    mdf.close()
     return rel_types
 
 
 @cached
+def charm_name():
+    """Get the name of the current charm as is specified on metadata.yaml"""
+    return metadata().get('name')
+
+
+@cached
 def relations():
     """Get a nested dictionary of relation data for all related units"""
     rels = {}
@@ -449,11 +545,16 @@
     """Get the unit ID for the remote unit"""
     _args = ['unit-get', '--format=json', attribute]
     try:
-        return json.loads(subprocess.check_output(_args))
+        return json.loads(subprocess.check_output(_args).decode('UTF-8'))
     except ValueError:
         return None
 
 
+def unit_public_ip():
+    """Get this unit's public IP address"""
+    return unit_get('public-address')
+
+
 def unit_private_ip():
     """Get this unit's private IP address"""
     return unit_get('private-address')
@@ -486,9 +587,10 @@
             hooks.execute(sys.argv)
     """
 
-    def __init__(self):
+    def __init__(self, config_save=True):
         super(Hooks, self).__init__()
         self._hooks = {}
+        self._config_save = config_save
 
     def register(self, name, function):
         """Register a hook"""
@@ -499,9 +601,10 @@
         hook_name = os.path.basename(args[0])
         if hook_name in self._hooks:
             self._hooks[hook_name]()
-            cfg = config()
-            if cfg.implicit_save:
-                cfg.save()
+            if self._config_save:
+                cfg = config()
+                if cfg.implicit_save:
+                    cfg.save()
         else:
             raise UnregisteredHookError(hook_name)
 
@@ -522,3 +625,120 @@
 def charm_dir():
     """Return the root directory of the current charm"""
     return os.environ.get('CHARM_DIR')
+
+
+@cached
+def action_get(key=None):
+    """Gets the value of an action parameter, or all key/value param pairs"""
+    cmd = ['action-get']
+    if key is not None:
+        cmd.append(key)
+    cmd.append('--format=json')
+    action_data = json.loads(subprocess.check_output(cmd).decode('UTF-8'))
+    return action_data
+
+
+def action_set(values):
+    """Sets the values to be returned after the action finishes"""
+    cmd = ['action-set']
+    for k, v in list(values.items()):
+        cmd.append('{}={}'.format(k, v))
+    subprocess.check_call(cmd)
+
+
+def action_fail(message):
+    """Sets the action status to failed and sets the error message.
+
+    The results set by action_set are preserved."""
+    subprocess.check_call(['action-fail', message])
+
+
+def status_set(workload_state, message):
+    """Set the workload state with a message
+
+    Use status-set to set the workload state with a message which is visible
+    to the user via juju status. If the status-set command is not found then
+    assume this is juju < 1.23 and juju-log the message unstead.
+
+    workload_state -- valid juju workload state.
+    message        -- status update message
+    """
+    valid_states = ['maintenance', 'blocked', 'waiting', 'active']
+    if workload_state not in valid_states:
+        raise ValueError(
+            '{!r} is not a valid workload state'.format(workload_state)
+        )
+    cmd = ['status-set', workload_state, message]
+    try:
+        ret = subprocess.call(cmd)
+        if ret == 0:
+            return
+    except OSError as e:
+        if e.errno != errno.ENOENT:
+            raise
+    log_message = 'status-set failed: {} {}'.format(workload_state,
+                                                    message)
+    log(log_message, level='INFO')
+
+
+def status_get():
+    """Retrieve the previously set juju workload state
+
+    If the status-set command is not found then assume this is juju < 1.23 and
+    return 'unknown'
+    """
+    cmd = ['status-get']
+    try:
+        raw_status = subprocess.check_output(cmd, universal_newlines=True)
+        status = raw_status.rstrip()
+        return status
+    except OSError as e:
+        if e.errno == errno.ENOENT:
+            return 'unknown'
+        else:
+            raise
+
+
+def translate_exc(from_exc, to_exc):
+    def inner_translate_exc1(f):
+        def inner_translate_exc2(*args, **kwargs):
+            try:
+                return f(*args, **kwargs)
+            except from_exc:
+                raise to_exc
+
+        return inner_translate_exc2
+
+    return inner_translate_exc1
+
+
+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
+def is_leader():
+    """Does the current unit hold the juju leadership
+
+    Uses juju to determine whether the current unit is the leader of its peers
+    """
+    cmd = ['is-leader', '--format=json']
+    return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
+
+
+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
+def leader_get(attribute=None):
+    """Juju leader get value(s)"""
+    cmd = ['leader-get', '--format=json'] + [attribute or '-']
+    return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
+
+
+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
+def leader_set(settings=None, **kwargs):
+    """Juju leader set value(s)"""
+    log("Juju leader-set '%s'" % (settings), level=DEBUG)
+    cmd = ['leader-set']
+    settings = settings or {}
+    settings.update(kwargs)
+    for k, v in settings.iteritems():
+        if v is None:
+            cmd.append('{}='.format(k))
+        else:
+            cmd.append('{}={}'.format(k, v))
+    subprocess.check_call(cmd)

=== modified file 'lib/charmhelpers/core/host.py'
--- lib/charmhelpers/core/host.py	2014-08-26 19:46:22 +0000
+++ lib/charmhelpers/core/host.py	2015-06-04 13:22:19 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 """Tools for working with the host system"""
 # Copyright 2012 Canonical Ltd.
 #
@@ -6,19 +22,20 @@
 #  Matthew Wedgwood <[e-mail]>
 
 import os
+import re
 import pwd
 import grp
 import random
 import string
 import subprocess
 import hashlib
-import shutil
 from contextlib import contextmanager
-
 from collections import OrderedDict
 
-from hookenv import log
-from fstab import Fstab
+import six
+
+from .hookenv import log
+from .fstab import Fstab
 
 
 def service_start(service_name):
@@ -54,7 +71,9 @@
 def service_running(service):
     """Determine whether a system service is running"""
     try:
-        output = subprocess.check_output(['service', service, 'status'], stderr=subprocess.STDOUT)
+        output = subprocess.check_output(
+            ['service', service, 'status'],
+            stderr=subprocess.STDOUT).decode('UTF-8')
     except subprocess.CalledProcessError:
         return False
     else:
@@ -67,9 +86,11 @@
 def service_available(service_name):
     """Determine whether a system service is available"""
     try:
-        subprocess.check_output(['service', service_name, 'status'], stderr=subprocess.STDOUT)
-    except subprocess.CalledProcessError:
-        return False
+        subprocess.check_output(
+            ['service', service_name, 'status'],
+            stderr=subprocess.STDOUT).decode('UTF-8')
+    except subprocess.CalledProcessError as e:
+        return b'unrecognized service' not in e.output
     else:
         return True
 
@@ -96,6 +117,26 @@
     return user_info
 
 
+def add_group(group_name, system_group=False):
+    """Add a group to the system"""
+    try:
+        group_info = grp.getgrnam(group_name)
+        log('group {0} already exists!'.format(group_name))
+    except KeyError:
+        log('creating group {0}'.format(group_name))
+        cmd = ['addgroup']
+        if system_group:
+            cmd.append('--system')
+        else:
+            cmd.extend([
+                '--group',
+            ])
+        cmd.append(group_name)
+        subprocess.check_call(cmd)
+        group_info = grp.getgrnam(group_name)
+    return group_info
+
+
 def add_user_to_group(username, group):
     """Add a user to a group"""
     cmd = [
@@ -115,7 +156,7 @@
     cmd.append(from_path)
     cmd.append(to_path)
     log(" ".join(cmd))
-    return subprocess.check_output(cmd).strip()
+    return subprocess.check_output(cmd).decode('UTF-8').strip()
 
 
 def symlink(source, destination):
@@ -130,28 +171,31 @@
     subprocess.check_call(cmd)
 
 
-def mkdir(path, owner='root', group='root', perms=0555, force=False):
+def mkdir(path, owner='root', group='root', perms=0o555, force=False):
     """Create a directory"""
     log("Making dir {} {}:{} {:o}".format(path, owner, group,
                                           perms))
     uid = pwd.getpwnam(owner).pw_uid
     gid = grp.getgrnam(group).gr_gid
     realpath = os.path.abspath(path)
-    if os.path.exists(realpath):
-        if force and not os.path.isdir(realpath):
+    path_exists = os.path.exists(realpath)
+    if path_exists and force:
+        if not os.path.isdir(realpath):
             log("Removing non-directory file {} prior to mkdir()".format(path))
             os.unlink(realpath)
-    else:
+            os.makedirs(realpath, perms)
+    elif not path_exists:
         os.makedirs(realpath, perms)
     os.chown(realpath, uid, gid)
-
-
-def write_file(path, content, owner='root', group='root', perms=0444):
-    """Create or overwrite a file with the contents of a string"""
+    os.chmod(realpath, perms)
+
+
+def write_file(path, content, owner='root', group='root', perms=0o444):
+    """Create or overwrite a file with the contents of a byte string."""
     log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))
     uid = pwd.getpwnam(owner).pw_uid
     gid = grp.getgrnam(group).gr_gid
-    with open(path, 'w') as target:
+    with open(path, 'wb') as target:
         os.fchown(target.fileno(), uid, gid)
         os.fchmod(target.fileno(), perms)
         target.write(content)
@@ -177,7 +221,7 @@
     cmd_args.extend([device, mountpoint])
     try:
         subprocess.check_output(cmd_args)
-    except subprocess.CalledProcessError, e:
+    except subprocess.CalledProcessError as e:
         log('Error mounting {} at {}\n{}'.format(device, mountpoint, e.output))
         return False
 
@@ -191,7 +235,7 @@
     cmd_args = ['umount', mountpoint]
     try:
         subprocess.check_output(cmd_args)
-    except subprocess.CalledProcessError, e:
+    except subprocess.CalledProcessError as e:
         log('Error unmounting {}\n{}'.format(mountpoint, e.output))
         return False
 
@@ -209,17 +253,42 @@
     return system_mounts
 
 
-def file_hash(path):
-    """Generate a md5 hash of the contents of 'path' or None if not found """
+def file_hash(path, hash_type='md5'):
+    """
+    Generate a hash checksum of the contents of 'path' or None if not found.
+
+    :param str hash_type: Any hash alrgorithm supported by :mod:`hashlib`,
+                          such as md5, sha1, sha256, sha512, etc.
+    """
     if os.path.exists(path):
-        h = hashlib.md5()
-        with open(path, 'r') as source:
-            h.update(source.read())  # IGNORE:E1101 - it does have update
+        h = getattr(hashlib, hash_type)()
+        with open(path, 'rb') as source:
+            h.update(source.read())
         return h.hexdigest()
     else:
         return None
 
 
+def check_hash(path, checksum, hash_type='md5'):
+    """
+    Validate a file using a cryptographic checksum.
+
+    :param str checksum: Value of the checksum used to validate the file.
+    :param str hash_type: Hash algorithm used to generate `checksum`.
+        Can be any hash alrgorithm supported by :mod:`hashlib`,
+        such as md5, sha1, sha256, sha512, etc.
+    :raises ChecksumError: If the file fails the checksum
+
+    """
+    actual_checksum = file_hash(path, hash_type)
+    if checksum != actual_checksum:
+        raise ChecksumError("'%s' != '%s'" % (checksum, actual_checksum))
+
+
+class ChecksumError(ValueError):
+    pass
+
+
 def restart_on_change(restart_map, stopstart=False):
     """Restart services based on configuration files changing
 
@@ -236,11 +305,11 @@
     ceph_client_changed function.
     """
     def wrap(f):
-        def wrapped_f(*args):
+        def wrapped_f(*args, **kwargs):
             checksums = {}
             for path in restart_map:
                 checksums[path] = file_hash(path)
-            f(*args)
+            f(*args, **kwargs)
             restarts = []
             for path in restart_map:
                 if checksums[path] != file_hash(path):
@@ -270,29 +339,39 @@
 def pwgen(length=None):
     """Generate a random pasword."""
     if length is None:
+        # A random length is ok to use a weak PRNG
         length = random.choice(range(35, 45))
     alphanumeric_chars = [
-        l for l in (string.letters + string.digits)
+        l for l in (string.ascii_letters + string.digits)
         if l not in 'l0QD1vAEIOUaeiou']
+    # Use a crypto-friendly PRNG (e.g. /dev/urandom) for making the
+    # actual password
+    random_generator = random.SystemRandom()
     random_chars = [
-        random.choice(alphanumeric_chars) for _ in range(length)]
+        random_generator.choice(alphanumeric_chars) for _ in range(length)]
     return(''.join(random_chars))
 
 
 def list_nics(nic_type):
     '''Return a list of nics of given type(s)'''
-    if isinstance(nic_type, basestring):
+    if isinstance(nic_type, six.string_types):
         int_types = [nic_type]
     else:
         int_types = nic_type
     interfaces = []
     for int_type in int_types:
         cmd = ['ip', 'addr', 'show', 'label', int_type + '*']
-        ip_output = subprocess.check_output(cmd).split('\n')
+        ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
         ip_output = (line for line in ip_output if line)
         for line in ip_output:
             if line.split()[1].startswith(int_type):
-                interfaces.append(line.split()[1].replace(":", ""))
+                matched = re.search('.*: (' + int_type + r'[0-9]+\.[0-9]+)@.*', line)
+                if matched:
+                    interface = matched.groups()[0]
+                else:
+                    interface = line.split()[1].replace(":", "")
+                interfaces.append(interface)
+
     return interfaces
 
 
@@ -304,7 +383,7 @@
 
 def get_nic_mtu(nic):
     cmd = ['ip', 'addr', 'show', nic]
-    ip_output = subprocess.check_output(cmd).split('\n')
+    ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
     mtu = ""
     for line in ip_output:
         words = line.split()
@@ -315,7 +394,7 @@
 
 def get_nic_hwaddr(nic):
     cmd = ['ip', '-o', '-0', 'addr', 'show', nic]
-    ip_output = subprocess.check_output(cmd)
+    ip_output = subprocess.check_output(cmd).decode('UTF-8')
     hwaddr = ""
     words = ip_output.split()
     if 'link/ether' in words:
@@ -330,10 +409,13 @@
     *  0 => Installed revno is the same as supplied arg
     * -1 => Installed revno is less than supplied arg
 
+    This function imports apt_cache function from charmhelpers.fetch if
+    the pkgcache argument is None. Be sure to add charmhelpers.fetch if
+    you call this function, or pass an apt_pkg.Cache() instance.
     '''
     import apt_pkg
-    from charmhelpers.fetch import apt_cache
     if not pkgcache:
+        from charmhelpers.fetch import apt_cache
         pkgcache = apt_cache()
     pkg = pkgcache[package]
     return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)

=== modified file 'lib/charmhelpers/core/services/__init__.py'
--- lib/charmhelpers/core/services/__init__.py	2014-08-17 21:29:19 +0000
+++ lib/charmhelpers/core/services/__init__.py	2015-06-04 13:22:19 +0000
@@ -1,2 +1,18 @@
-from .base import *
-from .helpers import *
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+from .base import *  # NOQA
+from .helpers import *  # NOQA

=== modified file 'lib/charmhelpers/core/services/base.py'
--- lib/charmhelpers/core/services/base.py	2014-09-02 17:52:32 +0000
+++ lib/charmhelpers/core/services/base.py	2015-06-04 13:22:19 +0000
@@ -1,7 +1,23 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 import os
-import re
 import json
-from collections import Iterable
+from inspect import getargspec
+from collections import Iterable, OrderedDict
 
 from charmhelpers.core import host
 from charmhelpers.core import hookenv
@@ -103,7 +119,7 @@
         """
         self._ready_file = os.path.join(hookenv.charm_dir(), 'READY-SERVICES.json')
         self._ready = None
-        self.services = {}
+        self.services = OrderedDict()
         for service in services or []:
             service_name = service['service']
             self.services[service_name] = service
@@ -116,8 +132,8 @@
         if hook_name == 'stop':
             self.stop_services()
         else:
+            self.reconfigure_services()
             self.provide_data()
-            self.reconfigure_services()
         cfg = hookenv.config()
         if cfg.implicit_save:
             cfg.save()
@@ -129,15 +145,36 @@
         A provider must have a `name` attribute, which indicates which relation
         to set data on, and a `provide_data()` method, which returns a dict of
         data to set.
+
+        The `provide_data()` method can optionally accept two parameters:
+
+          * ``remote_service`` The name of the remote service that the data will
+            be provided to.  The `provide_data()` method will be called once
+            for each connected service (not unit).  This allows the method to
+            tailor its data to the given service.
+          * ``service_ready`` Whether or not the service definition had all of
+            its requirements met, and thus the ``data_ready`` callbacks run.
+
+        Note that the ``provided_data`` methods are now called **after** the
+        ``data_ready`` callbacks are run.  This gives the ``data_ready`` callbacks
+        a chance to generate any data necessary for the providing to the remote
+        services.
         """
-        hook_name = hookenv.hook_name()
-        for service in self.services.values():
+        for service_name, service in self.services.items():
+            service_ready = self.is_ready(service_name)
             for provider in service.get('provided_data', []):
-                if re.match(r'{}-relation-(joined|changed)'.format(provider.name), hook_name):
-                    data = provider.provide_data()
-                    _ready = provider._is_ready(data) if hasattr(provider, '_is_ready') else data
-                    if _ready:
-                        hookenv.relation_set(None, data)
+                for relid in hookenv.relation_ids(provider.name):
+                    units = hookenv.related_units(relid)
+                    if not units:
+                        continue
+                    remote_service = units[0].split('/')[0]
+                    argspec = getargspec(provider.provide_data)
+                    if len(argspec.args) > 1:
+                        data = provider.provide_data(remote_service, service_ready)
+                    else:
+                        data = provider.provide_data()
+                    if data:
+                        hookenv.relation_set(relid, data)
 
     def reconfigure_services(self, *service_names):
         """

=== modified file 'lib/charmhelpers/core/services/helpers.py'
--- lib/charmhelpers/core/services/helpers.py	2014-08-17 21:29:19 +0000
+++ lib/charmhelpers/core/services/helpers.py	2015-06-04 13:22:19 +0000
@@ -1,3 +1,21 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import yaml
 from charmhelpers.core import hookenv
 from charmhelpers.core import templating
 
@@ -19,15 +37,23 @@
     the `name` attribute that are complete will used to populate the dictionary
     values (see `get_data`, below).
 
-    The generated context will be namespaced under the interface type, to prevent
-    potential naming conflicts.
+    The generated context will be namespaced under the relation :attr:`name`,
+    to prevent potential naming conflicts.
+
+    :param str name: Override the relation :attr:`name`, since it can vary from charm to charm
+    :param list additional_required_keys: Extend the list of :attr:`required_keys`
     """
     name = None
     interface = None
-    required_keys = []
-
-    def __init__(self, *args, **kwargs):
-        super(RelationContext, self).__init__(*args, **kwargs)
+
+    def __init__(self, name=None, additional_required_keys=None):
+        if not hasattr(self, 'required_keys'):
+            self.required_keys = []
+
+        if name is not None:
+            self.name = name
+        if additional_required_keys:
+            self.required_keys.extend(additional_required_keys)
         self.get_data()
 
     def __bool__(self):
@@ -101,11 +127,127 @@
         return {}
 
 
+class MysqlRelation(RelationContext):
+    """
+    Relation context for the `mysql` interface.
+
+    :param str name: Override the relation :attr:`name`, since it can vary from charm to charm
+    :param list additional_required_keys: Extend the list of :attr:`required_keys`
+    """
+    name = 'db'
+    interface = 'mysql'
+
+    def __init__(self, *args, **kwargs):
+        self.required_keys = ['host', 'user', 'password', 'database']
+        RelationContext.__init__(self, *args, **kwargs)
+
+
+class HttpRelation(RelationContext):
+    """
+    Relation context for the `http` interface.
+
+    :param str name: Override the relation :attr:`name`, since it can vary from charm to charm
+    :param list additional_required_keys: Extend the list of :attr:`required_keys`
+    """
+    name = 'website'
+    interface = 'http'
+
+    def __init__(self, *args, **kwargs):
+        self.required_keys = ['host', 'port']
+        RelationContext.__init__(self, *args, **kwargs)
+
+    def provide_data(self):
+        return {
+            'host': hookenv.unit_get('private-address'),
+            'port': 80,
+        }
+
+
+class RequiredConfig(dict):
+    """
+    Data context that loads config options with one or more mandatory options.
+
+    Once the required options have been changed from their default values, all
+    config options will be available, namespaced under `config` to prevent
+    potential naming conflicts (for example, between a config option and a
+    relation property).
+
+    :param list *args: List of options that must be changed from their default values.
+    """
+
+    def __init__(self, *args):
+        self.required_options = args
+        self['config'] = hookenv.config()
+        with open(os.path.join(hookenv.charm_dir(), 'config.yaml')) as fp:
+            self.config = yaml.load(fp).get('options', {})
+
+    def __bool__(self):
+        for option in self.required_options:
+            if option not in self['config']:
+                return False
+            current_value = self['config'][option]
+            default_value = self.config[option].get('default')
+            if current_value == default_value:
+                return False
+            if current_value in (None, '') and default_value in (None, ''):
+                return False
+        return True
+
+    def __nonzero__(self):
+        return self.__bool__()
+
+
+class StoredContext(dict):
+    """
+    A data context that always returns the data that it was first created with.
+
+    This is useful to do a one-time generation of things like passwords, that
+    will thereafter use the same value that was originally generated, instead
+    of generating a new value each time it is run.
+    """
+    def __init__(self, file_name, config_data):
+        """
+        If the file exists, populate `self` with the data from the file.
+        Otherwise, populate with the given data and persist it to the file.
+        """
+        if os.path.exists(file_name):
+            self.update(self.read_context(file_name))
+        else:
+            self.store_context(file_name, config_data)
+            self.update(config_data)
+
+    def store_context(self, file_name, config_data):
+        if not os.path.isabs(file_name):
+            file_name = os.path.join(hookenv.charm_dir(), file_name)
+        with open(file_name, 'w') as file_stream:
+            os.fchmod(file_stream.fileno(), 0o600)
+            yaml.dump(config_data, file_stream)
+
+    def read_context(self, file_name):
+        if not os.path.isabs(file_name):
+            file_name = os.path.join(hookenv.charm_dir(), file_name)
+        with open(file_name, 'r') as file_stream:
+            data = yaml.load(file_stream)
+            if not data:
+                raise OSError("%s is empty" % file_name)
+            return data
+
+
 class TemplateCallback(ManagerCallback):
     """
-    Callback class that will render a template, for use as a ready action.
+    Callback class that will render a Jinja2 template, for use as a ready
+    action.
+
+    :param str source: The template source file, relative to
+    `$CHARM_DIR/templates`
+
+    :param str target: The target to write the rendered template to
+    :param str owner: The owner of the rendered file
+    :param str group: The group of the rendered file
+    :param int perms: The permissions of the rendered file
     """
-    def __init__(self, source, target, owner='root', group='root', perms=0444):
+    def __init__(self, source, target,
+                 owner='root', group='root', perms=0o444):
         self.source = source
         self.target = target
         self.owner = owner

=== modified file 'lib/charmhelpers/core/templating.py'
--- lib/charmhelpers/core/templating.py	2014-08-17 21:29:19 +0000
+++ lib/charmhelpers/core/templating.py	2015-06-04 13:22:19 +0000
@@ -1,10 +1,27 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 import os
 
 from charmhelpers.core import host
 from charmhelpers.core import hookenv
 
 
-def render(source, target, context, owner='root', group='root', perms=0444, templates_dir=None):
+def render(source, target, context, owner='root', group='root',
+           perms=0o444, templates_dir=None, encoding='UTF-8'):
     """
     Render a template.
 
@@ -47,5 +64,5 @@
                     level=hookenv.ERROR)
         raise e
     content = template.render(context)
-    host.mkdir(os.path.dirname(target))
-    host.write_file(target, content, owner, group, perms)
+    host.mkdir(os.path.dirname(target), owner, group, perms=0o755)
+    host.write_file(target, content.encode(encoding), owner, group, perms)

=== modified file 'lib/charmhelpers/fetch/__init__.py'
--- lib/charmhelpers/fetch/__init__.py	2014-08-26 13:59:01 +0000
+++ lib/charmhelpers/fetch/__init__.py	2015-06-04 13:22:19 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 import importlib
 from tempfile import NamedTemporaryFile
 import time
@@ -5,10 +21,6 @@
 from charmhelpers.core.host import (
     lsb_release
 )
-from urlparse import (
-    urlparse,
-    urlunparse,
-)
 import subprocess
 from charmhelpers.core.hookenv import (
     config,
@@ -16,6 +28,12 @@
 )
 import os
 
+import six
+if six.PY3:
+    from urllib.parse import urlparse, urlunparse
+else:
+    from urlparse import urlparse, urlunparse
+
 
 CLOUD_ARCHIVE = """# Ubuntu Cloud Archive
 deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main
@@ -62,9 +80,16 @@
     'trusty-juno/updates': 'trusty-updates/juno',
     'trusty-updates/juno': 'trusty-updates/juno',
     'juno/proposed': 'trusty-proposed/juno',
-    'juno/proposed': 'trusty-proposed/juno',
     'trusty-juno/proposed': 'trusty-proposed/juno',
     'trusty-proposed/juno': 'trusty-proposed/juno',
+    # Kilo
+    'kilo': 'trusty-updates/kilo',
+    'trusty-kilo': 'trusty-updates/kilo',
+    'trusty-kilo/updates': 'trusty-updates/kilo',
+    'trusty-updates/kilo': 'trusty-updates/kilo',
+    'kilo/proposed': 'trusty-proposed/kilo',
+    'trusty-kilo/proposed': 'trusty-proposed/kilo',
+    'trusty-proposed/kilo': 'trusty-proposed/kilo',
 }
 
 # The order of this list is very important. Handlers should be listed in from
@@ -72,6 +97,7 @@
 FETCH_HANDLERS = (
     'charmhelpers.fetch.archiveurl.ArchiveUrlFetchHandler',
     'charmhelpers.fetch.bzrurl.BzrUrlFetchHandler',
+    'charmhelpers.fetch.giturl.GitUrlFetchHandler',
 )
 
 APT_NO_LOCK = 100  # The return code for "couldn't acquire lock" in APT.
@@ -132,7 +158,7 @@
 
 def apt_cache(in_memory=True):
     """Build and return an apt cache"""
-    import apt_pkg
+    from apt import apt_pkg
     apt_pkg.init()
     if in_memory:
         apt_pkg.config.set("Dir::Cache::pkgcache", "")
@@ -148,7 +174,7 @@
     cmd = ['apt-get', '--assume-yes']
     cmd.extend(options)
     cmd.append('install')
-    if isinstance(packages, basestring):
+    if isinstance(packages, six.string_types):
         cmd.append(packages)
     else:
         cmd.extend(packages)
@@ -181,7 +207,7 @@
 def apt_purge(packages, fatal=False):
     """Purge one or more packages"""
     cmd = ['apt-get', '--assume-yes', 'purge']
-    if isinstance(packages, basestring):
+    if isinstance(packages, six.string_types):
         cmd.append(packages)
     else:
         cmd.extend(packages)
@@ -192,7 +218,7 @@
 def apt_hold(packages, fatal=False):
     """Hold one or more packages"""
     cmd = ['apt-mark', 'hold']
-    if isinstance(packages, basestring):
+    if isinstance(packages, six.string_types):
         cmd.append(packages)
     else:
         cmd.extend(packages)
@@ -208,7 +234,8 @@
     """Add a package source to this system.
 
     @param source: a URL or sources.list entry, as supported by
-    add-apt-repository(1). Examples:
+    add-apt-repository(1). Examples::
+
         ppa:charmers/example
         deb https://stub:[e-mail]/ubuntu trusty main
 
@@ -217,6 +244,7 @@
         pocket for the release.
         'cloud:' may be used to activate official cloud archive pockets,
         such as 'cloud:icehouse'
+        'distro' may be used as a noop
 
     @param key: A key to be added to the system's APT keyring and used
     to verify the signatures on packages. Ideally, this should be an
@@ -250,12 +278,14 @@
         release = lsb_release()['DISTRIB_CODENAME']
         with open('/etc/apt/sources.list.d/proposed.list', 'w') as apt:
             apt.write(PROPOSED_POCKET.format(release))
+    elif source == 'distro':
+        pass
     else:
-        raise SourceConfigError("Unknown source: {!r}".format(source))
+        log("Unknown source: {!r}".format(source))
 
     if key:
         if '-----BEGIN PGP PUBLIC KEY BLOCK-----' in key:
-            with NamedTemporaryFile() as key_file:
+            with NamedTemporaryFile('w+') as key_file:
                 key_file.write(key)
                 key_file.flush()
                 key_file.seek(0)
@@ -292,14 +322,14 @@
     sources = safe_load((config(sources_var) or '').strip()) or []
     keys = safe_load((config(keys_var) or '').strip()) or None
 
-    if isinstance(sources, basestring):
+    if isinstance(sources, six.string_types):
         sources = [sources]
 
     if keys is None:
         for source in sources:
             add_source(source, None)
     else:
-        if isinstance(keys, basestring):
+        if isinstance(keys, six.string_types):
             keys = [keys]
 
         if len(sources) != len(keys):
@@ -311,22 +341,35 @@
         apt_update(fatal=True)
 
 
-def install_remote(source):
+def install_remote(source, *args, **kwargs):
     """
     Install a file tree from a remote source
 
     The specified source should be a url of the form:
         scheme://[host]/path[#[option=value][&...]]
 
-    Schemes supported are based on this modules submodules
-    Options supported are submodule-specific"""
+    Schemes supported are based on this modules submodules.
+    Options supported are submodule-specific.
+    Additional arguments are passed through to the submodule.
+
+    For example::
+
+        dest = install_remote('http://example.com/archive.tgz',
+                              checksum='deadbeef',
+                              hash_type='sha1')
+
+    This will download `archive.tgz`, validate it using SHA1 and, if
+    the file is ok, extract it and return the directory in which it
+    was extracted.  If the checksum fails, it will raise
+    :class:`charmhelpers.core.host.ChecksumError`.
+    """
     # We ONLY check for True here because can_handle may return a string
     # explaining why it can't handle a given source.
     handlers = [h for h in plugins() if h.can_handle(source) is True]
     installed_to = None
     for handler in handlers:
         try:
-            installed_to = handler.install(source)
+            installed_to = handler.install(source, *args, **kwargs)
         except UnhandledSource:
             pass
     if not installed_to:
@@ -383,7 +426,7 @@
         while result is None or result == APT_NO_LOCK:
             try:
                 result = subprocess.check_call(cmd, env=env)
-            except subprocess.CalledProcessError, e:
+            except subprocess.CalledProcessError as e:
                 retry_count = retry_count + 1
                 if retry_count > APT_NO_LOCK_RETRY_COUNT:
                     raise

=== modified file 'lib/charmhelpers/fetch/archiveurl.py'
--- lib/charmhelpers/fetch/archiveurl.py	2014-09-15 14:42:27 +0000
+++ lib/charmhelpers/fetch/archiveurl.py	2015-06-04 13:22:19 +0000
@@ -1,8 +1,22 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 import os
-import urllib2
-from urllib import urlretrieve
-import urlparse
 import hashlib
+import re
 
 from charmhelpers.fetch import (
     BaseFetchHandler,
@@ -12,21 +26,54 @@
     get_archive_handler,
     extract,
 )
-from charmhelpers.core.host import mkdir
-
-"""
-This class is a plugin for charmhelpers.fetch.install_remote.
-
-It grabs, validates and installs remote archives fetched over "http", "https", "ftp" or "file" protocols. The contents of the archive are installed in $CHARM_DIR/fetched/.
-
-Example usage:
-install_remote("https://example.com/some/archive.tar.gz")
-# Installs the contents of archive.tar.gz in $CHARM_DIR/fetched/.
-
-See charmhelpers.fetch.archiveurl.get_archivehandler for supported archive types.
-"""
+from charmhelpers.core.host import mkdir, check_hash
+
+import six
+if six.PY3:
+    from urllib.request import (
+        build_opener, install_opener, urlopen, urlretrieve,
+        HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler,
+    )
+    from urllib.parse import urlparse, urlunparse, parse_qs
+    from urllib.error import URLError
+else:
+    from urllib import urlretrieve
+    from urllib2 import (
+        build_opener, install_opener, urlopen,
+        HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler,
+        URLError
+    )
+    from urlparse import urlparse, urlunparse, parse_qs
+
+
+def splituser(host):
+    '''urllib.splituser(), but six's support of this seems broken'''
+    _userprog = re.compile('^(.*)@(.*)$')
+    match = _userprog.match(host)
+    if match:
+        return match.group(1, 2)
+    return None, host
+
+
+def splitpasswd(user):
+    '''urllib.splitpasswd(), but six's support of this is missing'''
+    _passwdprog = re.compile('^([^:]*):(.*)$', re.S)
+    match = _passwdprog.match(user)
+    if match:
+        return match.group(1, 2)
+    return user, None
+
+
 class ArchiveUrlFetchHandler(BaseFetchHandler):
-    """Handler for archives via generic URLs"""
+    """
+    Handler to download archive files from arbitrary URLs.
+
+    Can fetch from http, https, ftp, and file URLs.
+
+    Can install either tarballs (.tar, .tgz, .tbz2, etc) or zip files.
+
+    Installs the contents of the archive in $CHARM_DIR/fetched/.
+    """
     def can_handle(self, source):
         url_parts = self.parse_url(source)
         if url_parts.scheme not in ('http', 'https', 'ftp', 'file'):
@@ -36,22 +83,28 @@
         return False
 
     def download(self, source, dest):
+        """
+        Download an archive file.
+
+        :param str source: URL pointing to an archive file.
+        :param str dest: Local path location to download archive file to.
+        """
         # propogate all exceptions
         # URLError, OSError, etc
-        proto, netloc, path, params, query, fragment = urlparse.urlparse(source)
+        proto, netloc, path, params, query, fragment = urlparse(source)
         if proto in ('http', 'https'):
-            auth, barehost = urllib2.splituser(netloc)
+            auth, barehost = splituser(netloc)
             if auth is not None:
-                source = urlparse.urlunparse((proto, barehost, path, params, query, fragment))
-                username, password = urllib2.splitpasswd(auth)
-                passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
+                source = urlunparse((proto, barehost, path, params, query, fragment))
+                username, password = splitpasswd(auth)
+                passman = HTTPPasswordMgrWithDefaultRealm()
                 # Realm is set to None in add_password to force the username and password
                 # to be used whatever the realm
                 passman.add_password(None, source, username, password)
-                authhandler = urllib2.HTTPBasicAuthHandler(passman)
-                opener = urllib2.build_opener(authhandler)
-                urllib2.install_opener(opener)
-        response = urllib2.urlopen(source)
+                authhandler = HTTPBasicAuthHandler(passman)
+                opener = build_opener(authhandler)
+                install_opener(opener)
+        response = urlopen(source)
         try:
             with open(dest, 'w') as dest_file:
                 dest_file.write(response.read())
@@ -60,44 +113,49 @@
                 os.unlink(dest)
             raise e
 
-    def install(self, source):
+    # Mandatory file validation via Sha1 or MD5 hashing.
+    def download_and_validate(self, url, hashsum, validate="sha1"):
+        tempfile, headers = urlretrieve(url)
+        check_hash(tempfile, hashsum, validate)
+        return tempfile
+
+    def install(self, source, dest=None, checksum=None, hash_type='sha1'):
+        """
+        Download and install an archive file, with optional checksum validation.
+
+        The checksum can also be given on the `source` URL's fragment.
+        For example::
+
+            handler.install('http://example.com/file.tgz#sha1=deadbeef')
+
+        :param str source: URL pointing to an archive file.
+        :param str dest: Local destination path to install to. If not given,
+            installs to `$CHARM_DIR/archives/archive_file_name`.
+        :param str checksum: If given, validate the archive file after download.
+        :param str hash_type: Algorithm used to generate `checksum`.
+            Can be any hash alrgorithm supported by :mod:`hashlib`,
+            such as md5, sha1, sha256, sha512, etc.
+
+        """
         url_parts = self.parse_url(source)
         dest_dir = os.path.join(os.environ.get('CHARM_DIR'), 'fetched')
         if not os.path.exists(dest_dir):
-            mkdir(dest_dir, perms=0755)
+            mkdir(dest_dir, perms=0o755)
         dld_file = os.path.join(dest_dir, os.path.basename(url_parts.path))
         try:
             self.download(source, dld_file)
-        except urllib2.URLError as e:
+        except URLError as e:
             raise UnhandledSource(e.reason)
         except OSError as e:
             raise UnhandledSource(e.strerror)
-        return extract(dld_file)
-
-    # Mandatory file validation via Sha1 or MD5 hashing.
-    def download_and_validate(self, url, hashsum, validate="sha1"):
-        if validate == 'sha1' and len(hashsum) != 40:
-            raise ValueError("HashSum must be = 40 characters when using sha1"
-                             " validation")
-        if validate == 'md5' and len(hashsum) != 32:
-            raise ValueError("HashSum must be = 32 characters when using md5"
-                             " validation")
-        tempfile, headers = urlretrieve(url)
-        self.validate_file(tempfile, hashsum, validate)
-        return tempfile
-
-    # Predicate method that returns status of hash matching expected hash.
-    def validate_file(self, source, hashsum, vmethod='sha1'):
-        if vmethod != 'sha1' and vmethod != 'md5':
-            raise ValueError("Validation Method not supported")
-
-        if vmethod == 'md5':
-            m = hashlib.md5()
-        if vmethod == 'sha1':
-            m = hashlib.sha1()
-        with open(source) as f:
-            for line in f:
-                m.update(line)
-        if hashsum != m.hexdigest():
-            msg = "Hash Mismatch on {} expected {} got {}"
-            raise ValueError(msg.format(source, hashsum, m.hexdigest()))
+        options = parse_qs(url_parts.fragment)
+        for key, value in options.items():
+            if not six.PY3:
+                algorithms = hashlib.algorithms
+            else:
+                algorithms = hashlib.algorithms_available
+            if key in algorithms:
+                check_hash(dld_file, value, key)
+        if checksum:
+            check_hash(dld_file, checksum, hash_type)
+        return extract(dld_file, dest)

=== modified file 'lib/charmhelpers/fetch/bzrurl.py'
--- lib/charmhelpers/fetch/bzrurl.py	2014-08-17 21:29:19 +0000
+++ lib/charmhelpers/fetch/bzrurl.py	2015-06-04 13:22:19 +0000
@@ -1,3 +1,19 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
 import os
 from charmhelpers.fetch import (
     BaseFetchHandler,
@@ -5,12 +21,18 @@
 )
 from charmhelpers.core.host import mkdir
 
+import six
+if six.PY3:
+    raise ImportError('bzrlib does not support Python3')
+
 try:
     from bzrlib.branch import Branch
+    from bzrlib import bzrdir, workingtree, errors
 except ImportError:
     from charmhelpers.fetch import apt_install
     apt_install("python-bzrlib")
     from bzrlib.branch import Branch
+    from bzrlib import bzrdir, workingtree, errors
 
 
 class BzrUrlFetchHandler(BaseFetchHandler):
@@ -31,8 +53,14 @@
             from bzrlib.plugin import load_plugins
             load_plugins()
         try:
+            local_branch = bzrdir.BzrDir.create_branch_convenience(dest)
+        except errors.AlreadyControlDirError:
+            local_branch = Branch.open(dest)
+        try:
             remote_branch = Branch.open(source)
-            remote_branch.bzrdir.sprout(dest).open_branch()
+            remote_branch.push(local_branch)
+            tree = workingtree.WorkingTree.open(dest)
+            tree.update()
         except Exception as e:
             raise e
 
@@ -42,7 +70,7 @@
         dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
                                 branch_name)
         if not os.path.exists(dest_dir):
-            mkdir(dest_dir, perms=0755)
+            mkdir(dest_dir, perms=0o755)
         try:
             self.branch(source, dest_dir)
         except OSError as e:

=== removed file 'patches/hookenv-0001-fix_config_lookups.patch'
--- patches/hookenv-0001-fix_config_lookups.patch	2014-09-05 17:35:28 +0000
+++ patches/hookenv-0001-fix_config_lookups.patch	1970-01-01 00:00:00 +0000
@@ -1,30 +0,0 @@
-------------------------------------------------------------
-revno: 101
-committer: Robert Jennings <[e-mail]>
-branch nick: ubuntu-repository-cache
-timestamp: Fri 2014-09-05 12:33:45 -0500
-message:
-  Fix config lookups so that user-saved values are returned properly.
-  
-  https://code.launchpad.net/~tvansteenburgh/charm-helpers/fix-config-lookups/+merge/233558
-diff:
-=== modified file 'lib/charmhelpers/core/hookenv.py'
---- lib/charmhelpers/core/hookenv.py	2014-09-02 17:52:32 +0000
-+++ lib/charmhelpers/core/hookenv.py	2014-09-05 17:33:45 +0000
-@@ -203,6 +203,17 @@
-         if os.path.exists(self.path):
-             self.load_previous()
- 
-+    def __getitem__(self, key):
-+        """For regular dict lookups, check the current juju config first,
-+        the the previous (saved) copy. This ensures that user-saved values
-+        will be returned by a dict lookup.
-+
-+        """
-+        try:
-+            return dict.__getitem__(self, key)
-+        except KeyError:
-+            return (self._prev_dict or {})[key]
-+
-     def load_previous(self, path=None):
-         """Load previous copy of config from disk.

=== removed file 'patches/lchownr.patch'
--- patches/lchownr.patch	2014-09-02 17:52:32 +0000
+++ patches/lchownr.patch	1970-01-01 00:00:00 +0000
@@ -1,42 +0,0 @@
-------------------------------------------------------------
-revno: 37
-committer: Robert Jennings <[e-mail]>
-branch nick: ubuntu-repository-cache
-timestamp: Tue 2014-08-26 14:46:22 -0500
-message:
-  lchown() variant for charmhelpers.core.host.chownr()
-  
-  Port of MP#232306
-  https://code.launchpad.net/~rcj/charm-helpers/lchownr/+merge/232306
-  
-  Create hosts.lchownr() as a variant that doesn't follow links
-  lchownr() will walk the path without following links on the walk() and
-  will use os.lchown() to change owner/group on the link itself.
-diff:
-=== modified file 'lib/charmhelpers/core/host.py'
---- lib/charmhelpers/core/host.py	2014-08-26 13:59:01 +0000
-+++ lib/charmhelpers/core/host.py	2014-08-26 19:46:22 +0000
-@@ -348,13 +348,21 @@
-         os.chdir(cur)
- 
- 
--def chownr(path, owner, group):
-+def chownr(path, owner, group, follow_links=True):
-     uid = pwd.getpwnam(owner).pw_uid
-     gid = grp.getgrnam(group).gr_gid
-+    if follow_links:
-+        chown = os.chown
-+    else:
-+        chown = os.lchown
- 
-     for root, dirs, files in os.walk(path):
-         for name in dirs + files:
-             full = os.path.join(root, name)
-             broken_symlink = os.path.lexists(full) and not os.path.exists(full)
-             if not broken_symlink:
--                os.chown(full, uid, gid)
-+                chown(full, uid, gid)
-+
-+
-+def lchownr(path, owner, group):
-+    chownr(path, owner, group, follow_links=False)

=== removed file 'patches/mirror-0001-devel_reduce_dists.patch'
--- patches/mirror-0001-devel_reduce_dists.patch	2014-09-08 14:57:57 +0000
+++ patches/mirror-0001-devel_reduce_dists.patch	1970-01-01 00:00:00 +0000
@@ -1,16 +0,0 @@
-=== modified file 'lib/ubuntu_repository_cache/mirror.py'
---- lib/ubuntu_repository_cache/mirror.py	2014-09-08 14:55:51 +0000
-+++ lib/ubuntu_repository_cache/mirror.py	2014-09-08 14:56:47 +0000
-@@ -88,6 +88,11 @@
-                      '--exclude=dists/*/*/dist-upgrader-all',
-                      '--exclude=dists/*/*/installer*',
-                      '--exclude=dists/*/*/uefi',
-+                     '--exclude=dists/lucid-*',   # TODO Remove before flight
-+                     '--exclude=dists/precis*',   # TODO Remove before flight
-+                     '--exclude=dists/sauc*',     # TODO Remove before flight
-+                     '--exclude=dists/trust*',    # TODO Remove before flight
-+                     '--exclude=dists/utopic*',   # TODO Remove before flight
-                      )
-     host.rsync(rsync_source,
-                dest,
-

=== removed file 'patches/storage-0001-is_device_mounted.patch'
--- patches/storage-0001-is_device_mounted.patch	2014-09-16 13:52:19 +0000
+++ patches/storage-0001-is_device_mounted.patch	1970-01-01 00:00:00 +0000
@@ -1,31 +0,0 @@
-------------------------------------------------------------
-revno: 137
-committer: Robert Jennings <[e-mail]>
-branch nick: ubuntu-repository-cache
-timestamp: Tue 2014-09-16 08:50:02 -0500
-message:
-  storage.linux.util: Fix swapped is_device_mounted regex logic (lp:1370053)
-  
-  The is_device_mounted() function checks if the passed device is a
-  partition and tailors the regex based on this. However, the search
-  expressions were swapped and would fail. When a partition is provided,
-  a match will only be found if the parent device is mounted.
-  
-  For example, if we call is_device_mounted(device='/dev/sda1') and
-  '/dev/sda1' is in the mount command output, the result is false.
-  Similarly, is_device_mounted(device='/dev/sda') and '/dev/sda' is in
-  the mount command output, the result is false. Instead, if the device
-  is /dev/sda1 and /dev/sda is in the mount output, we get a True result
-  from the function.
-diff:
-=== modified file 'lib/charmhelpers/contrib/storage/linux/utils.py'
---- lib/charmhelpers/contrib/storage/linux/utils.py	2014-09-15 14:42:27 +0000
-+++ lib/charmhelpers/contrib/storage/linux/utils.py	2014-09-16 13:50:02 +0000
-@@ -48,6 +48,6 @@
-     '''
-     is_partition = bool(re.search(r".*[0-9]+\b", device))
-     out = check_output(['mount'])
--    if is_partition:
-+    if not is_partition:
-         return bool(re.search(device + r"\b", out))
-     return bool(re.search(device + r"[0-9]+\b", out))

=== modified file 'patches/unison-0002-keyscan.patch'
--- patches/unison-0002-keyscan.patch	2014-09-03 21:16:33 +0000
+++ patches/unison-0002-keyscan.patch	2015-06-04 13:22:19 +0000
@@ -14,12 +14,12 @@
 === modified file 'lib/charmhelpers/contrib/unison/__init__.py'
 --- lib/charmhelpers/contrib/unison/__init__.py	2014-09-03 19:44:16 +0000
 +++ lib/charmhelpers/contrib/unison/__init__.py	2014-09-03 19:47:58 +0000
-@@ -142,7 +142,7 @@
+@@ -159,7 +159,7 @@
      known_hosts = os.path.join(ssh_dir, 'known_hosts')
      khosts = []
      for host in hosts:
 -        cmd = ['ssh-keyscan', '-H', '-t', 'rsa', host]
 +        cmd = ['ssh-keyscan', host]
-         remote_key = check_output(cmd).strip()
+         remote_key = check_output(cmd, universal_newlines=True).strip()
          khosts.append(remote_key)
      log('Syncing known_hosts @ %s.' % known_hosts)