Merge lp:~niedbalski/charms/trusty/ceph-osd/sysctl-param into lp:~openstack-charmers-archive/charms/trusty/ceph-osd/next

[Jorge Niedbalski]

- Exposes the sysctl config option for update /etc/sysctl.d/50-charm-ceph-osd.conf using charmhelpers.core.sysctl.

- Linked with https://code.launchpad.net/~niedbalski/charms/trusty/ceph/sysctl-update/+merge/237379

[Corey Bryant (corey.bryant)]

Looks good. In terms of testing, lint passed. I didn't test anything else since I've already tested the same code for the ceph patch: https://code.launchpad.net/~niedbalski/charms/trusty/ceph/sysctl-update/+merge/237379.

review: Approve

[James Page (james-page)]

Feel like this is only part of what we need to make ceph do opininated things when deployed via charms - do we have some sane default tuning for sysctl that we can apply here?

review: Needs Fixing

[Ryan Beisner (1chb1n)]

UOSCI bot says:
This MP triggered a test on the Ubuntu OSCI system. Here is a summary of results.

#464 ceph-osd-next for niedbalski mp237381
charm_unit_test

This build:
http://10.98.191.181:8080/job/charm_unit_test/464/

MP URL:
https://code.launchpad.net/~niedbalski/charms/trusty/ceph-osd/sysctl-param/+merge/237381

Proposed branch:
lp:~niedbalski/charms/trusty/ceph-osd/sysctl-param

Results summary:
    UNIT FAIL: unit-test missing

UNIT Results not found.

Ubuntu OSCI Jenkins is currently in development on a Canonical private network, but we plan to publish results to a public instance soon. Tests are triggered if the proposed branch rev changes, or if the MP is placed into "Needs review" status after being otherwise for >= 1hr. Human review of results is still recommended.
http://10.98.191.181:8080/

[Ryan Beisner (1chb1n)]

UOSCI bot says:
This MP triggered a test on the Ubuntu OSCI system. Here is a summary of results.

#658 ceph-osd-next for niedbalski mp237381
charm_lint_check

This build:
http://10.98.191.181:8080/job/charm_lint_check/658/

MP URL:
https://code.launchpad.net/~niedbalski/charms/trusty/ceph-osd/sysctl-param/+merge/237381

Proposed branch:
lp:~niedbalski/charms/trusty/ceph-osd/sysctl-param

Results summary:
    LINT OK: believed to pass, but you should confirm results

LINT Results (max last 25 lines) from
/var/lib/jenkins/workspace/charm_lint_check/make-lint.658:
I: all charms should provide at least one thing
I: config.yaml: option osd-journal has no default value
I: config.yaml: option ephemeral-unmount has no default value
I: config.yaml: option osd-reformat has no default value
I: config.yaml: option ceph-public-network has no default value
I: config.yaml: option ceph-cluster-network has no default value
I: config.yaml: option key has no default value

Ubuntu OSCI Jenkins is currently in development on a Canonical private network, but we plan to publish results to a public instance soon. Tests are triggered if the proposed branch rev changes, or if the MP is placed into "Needs review" status after being otherwise for >= 1hr. Human review of results is still recommended.
http://10.98.191.181:8080/

[Ryan Beisner (1chb1n)]

UOSCI bot says:
This MP triggered a test on the Ubuntu OSCI system. Here is a summary of results.

#236 ceph-osd-next for niedbalski mp237381
charm_amulet_test

This build:
http://10.98.191.181:8080/job/charm_amulet_test/236/

MP URL:
https://code.launchpad.net/~niedbalski/charms/trusty/ceph-osd/sysctl-param/+merge/237381

Proposed branch:
lp:~niedbalski/charms/trusty/ceph-osd/sysctl-param

Results summary:
    AMULET FAIL: amulet-test failed

AMULET Results (max last 25 lines) from
/var/lib/jenkins/workspace/charm_amulet_test/make-test.236:
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 899, in run
    for change_set in self.watch:
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 266, in next
    return super(TimeoutWatcher, self).next()
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 211, in next
    'Id': self.watcher_id})
  File "/usr/lib/python2.7/dist-packages/jujuclient.py", line 140, in _rpc
    raw = self.conn.recv()
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 663, in recv
    opcode, data = self.recv_data()
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 678, in recv_data
    frame = self.recv_frame()
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 753, in recv_frame
    frame_buffer.recv_header(self._recv_strict)
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 257, in recv_header
    header = recv_fn(2)
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 860, in _recv_strict
    bytes = self._recv(shortage)
  File "/usr/lib/python2.7/dist-packages/websocket/_core.py", line 842, in _recv
    bytes = self.sock.recv(bufsize)
  File "/usr/lib/python2.7/ssl.py", line 341, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 260, in read
    return self._sslobj.read(len)
socket.error: [Errno 110] Connection timed out

Ubuntu OSCI Jenkins is currently in development on a Canonical private network, but we plan to publish results to a public instance soon. Tests are triggered if the proposed branch rev changes, or if the MP is placed into "Needs review" status after being otherwise for >= 1hr. Human review of results is still recommended.
http://10.98.191.181:8080/

[Jorge Niedbalski (niedbalski)]

Hello,

Any chance to get this merged?

[James Page (james-page)]

Not until feedback has been resolved! marked wip.

[Jorge Niedbalski (niedbalski)]

@james-page

Re-submitted according to your observations.

[uosci-testing-bot (uosci-testing-bot)]

UOSCI bot says:
charm_unit_test #995 ceph-osd-next for niedbalski mp242424
    UNIT FAIL: unit-test missing

UNIT Results (max last 5 lines):
INFO:root:Workspace dir: /var/lib/jenkins/workspace/charm_unit_test
INFO:root:Reading file: Makefile
INFO:root:Searching for: ['nosetest', 'unit.test']
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full unit test output: http://paste.ubuntu.com/9132357/
Build: http://10.98.191.181:8080/job/charm_unit_test/995/

[uosci-testing-bot (uosci-testing-bot)]

UOSCI bot says:
charm_lint_check #1161 ceph-osd-next for niedbalski mp242424
    LINT OK: passed

LINT Results (max last 5 lines):
  I: config.yaml: option ephemeral-unmount has no default value
  I: config.yaml: option osd-reformat has no default value
  I: config.yaml: option ceph-public-network has no default value
  I: config.yaml: option ceph-cluster-network has no default value
  I: config.yaml: option key has no default value

Full lint test output: http://paste.ubuntu.com/9132358/
Build: http://10.98.191.181:8080/job/charm_lint_check/1161/

[uosci-testing-bot (uosci-testing-bot)]

UOSCI bot says:
charm_amulet_test #503 ceph-osd-next for niedbalski mp242424
    AMULET FAIL: amulet-test failed

AMULET Results (max last 5 lines):
  ERROR waited for 10m0s without being able to connect: ssh: connect to host 10.217.3.218 port 22: No route to host
  juju-test.conductor WARNING : Could not bootstrap osci-sv07, got Bootstrap returned with an exit > 0. Skipping
  juju-test INFO : Results: 0 passed, 0 failed, 3 errored
  ERROR subprocess encountered error code 124
  make: *** [test] Error 124

Full amulet test output: http://paste.ubuntu.com/9132916/
Build: http://10.98.191.181:8080/job/charm_amulet_test/503/

[uosci-testing-bot (uosci-testing-bot)]

UOSCI bot says:
charm_unit_test #1052 ceph-osd-next for niedbalski mp242424
    UNIT FAIL: unit-test missing

UNIT Results (max last 5 lines):
INFO:root:Workspace dir: /var/lib/jenkins/workspace/charm_unit_test
INFO:root:Reading file: Makefile
INFO:root:Searching for: ['nosetest', 'unit.test']
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full unit test output: http://paste.ubuntu.com/9222288/
Build: http://10.98.191.181:8080/job/charm_unit_test/1052/

[uosci-testing-bot (uosci-testing-bot)]

UOSCI bot says:
charm_lint_check #1218 ceph-osd-next for niedbalski mp242424
    LINT OK: passed

LINT Results (max last 5 lines):
  I: config.yaml: option ephemeral-unmount has no default value
  I: config.yaml: option osd-reformat has no default value
  I: config.yaml: option ceph-public-network has no default value
  I: config.yaml: option ceph-cluster-network has no default value
  I: config.yaml: option key has no default value

Full lint test output: http://paste.ubuntu.com/9222289/
Build: http://10.98.191.181:8080/job/charm_lint_check/1218/

[uosci-testing-bot (uosci-testing-bot)]

UOSCI bot says:
charm_amulet_test #522 ceph-osd-next for niedbalski mp242424
    AMULET FAIL: amulet-test failed

AMULET Results (max last 5 lines):
  juju-test.conductor DEBUG : Calling "juju destroy-environment -y osci-sv07"
  WARNING cannot delete security group "juju-osci-sv07-0". Used by another environment?
  juju-test INFO : Results: 1 passed, 1 failed, 1 errored
  ERROR subprocess encountered error code 1
  make: *** [test] Error 1

Full amulet test output: http://paste.ubuntu.com/9222651/
Build: http://10.98.191.181:8080/job/charm_amulet_test/522/

=== modified file 'config.yaml'
--- config.yaml	2014-10-06 22:11:14 +0000
+++ config.yaml	2014-11-25 16:59:28 +0000
@@ -121,3 +121,9 @@
       order for this charm to function correctly, the privacy extension must be
       disabled and a non-temporary address must be configured/available on
       your network interface.
+  sysctl:
+    type: string
+    default:
+    description: |
+     YAML formatted associative array of sysctl values, e.g.:
+      '{ kernel.pid_max : 4194303 }'

=== modified file 'hooks/charmhelpers/contrib/network/ip.py'
--- hooks/charmhelpers/contrib/network/ip.py	2014-10-21 07:56:51 +0000
+++ hooks/charmhelpers/contrib/network/ip.py	2014-11-25 16:59:28 +0000
@@ -1,15 +1,12 @@
 import glob
 import re
 import subprocess
-import sys
 
 from functools import partial
 
 from charmhelpers.core.hookenv import unit_get
 from charmhelpers.fetch import apt_install
 from charmhelpers.core.hookenv import (
-    WARNING,
-    ERROR,
     log
 )
 
@@ -34,31 +31,28 @@
                          network)
 
 
+def no_ip_found_error_out(network):
+    errmsg = ("No IP address found in network: %s" % network)
+    raise ValueError(errmsg)
+
+
 def get_address_in_network(network, fallback=None, fatal=False):
-    """
-    Get an IPv4 or IPv6 address within the network from the host.
+    """Get an IPv4 or IPv6 address within the network from the host.
 
     :param network (str): CIDR presentation format. For example,
         '192.168.1.0/24'.
     :param fallback (str): If no address is found, return fallback.
     :param fatal (boolean): If no address is found, fallback is not
         set and fatal is True then exit(1).
-
     """
-
-    def not_found_error_out():
-        log("No IP address found in network: %s" % network,
-            level=ERROR)
-        sys.exit(1)
-
     if network is None:
         if fallback is not None:
             return fallback
+
+        if fatal:
+            no_ip_found_error_out(network)
         else:
-            if fatal:
-                not_found_error_out()
-            else:
-                return None
+            return None
 
     _validate_cidr(network)
     network = netaddr.IPNetwork(network)
@@ -70,6 +64,7 @@
             cidr = netaddr.IPNetwork("%s/%s" % (addr, netmask))
             if cidr in network:
                 return str(cidr.ip)
+
         if network.version == 6 and netifaces.AF_INET6 in addresses:
             for addr in addresses[netifaces.AF_INET6]:
                 if not addr['addr'].startswith('fe80'):
@@ -82,20 +77,20 @@
         return fallback
 
     if fatal:
-        not_found_error_out()
+        no_ip_found_error_out(network)
 
     return None
 
 
 def is_ipv6(address):
-    '''Determine whether provided address is IPv6 or not'''
+    """Determine whether provided address is IPv6 or not."""
     try:
         address = netaddr.IPAddress(address)
     except netaddr.AddrFormatError:
         # probably a hostname - so not an address at all!
         return False
-    else:
-        return address.version == 6
+
+    return address.version == 6
 
 
 def is_address_in_network(network, address):
@@ -113,11 +108,13 @@
     except (netaddr.core.AddrFormatError, ValueError):
         raise ValueError("Network (%s) is not in CIDR presentation format" %
                          network)
+
     try:
         address = netaddr.IPAddress(address)
     except (netaddr.core.AddrFormatError, ValueError):
         raise ValueError("Address (%s) is not in correct presentation format" %
                          address)
+
     if address in network:
         return True
     else:
@@ -147,6 +144,7 @@
                     return iface
                 else:
                     return addresses[netifaces.AF_INET][0][key]
+
         if address.version == 6 and netifaces.AF_INET6 in addresses:
             for addr in addresses[netifaces.AF_INET6]:
                 if not addr['addr'].startswith('fe80'):
@@ -160,41 +158,42 @@
                             return str(cidr).split('/')[1]
                         else:
                             return addr[key]
+
     return None
 
 
 get_iface_for_address = partial(_get_for_address, key='iface')
 
+
 get_netmask_for_address = partial(_get_for_address, key='netmask')
 
 
 def format_ipv6_addr(address):
-    """
-    IPv6 needs to be wrapped with [] in url link to parse correctly.
+    """If address is IPv6, wrap it in '[]' otherwise return None.
+
+    This is required by most configuration files when specifying IPv6
+    addresses.
     """
     if is_ipv6(address):
-        address = "[%s]" % address
-    else:
-        log("Not a valid ipv6 address: %s" % address, level=WARNING)
-        address = None
+        return "[%s]" % address
 
-    return address
+    return None
 
 
 def get_iface_addr(iface='eth0', inet_type='AF_INET', inc_aliases=False,
                    fatal=True, exc_list=None):
-    """
-    Return the assigned IP address for a given interface, if any, or [].
-    """
+    """Return the assigned IP address for a given interface, if any."""
     # Extract nic if passed /dev/ethX
     if '/' in iface:
         iface = iface.split('/')[-1]
+
     if not exc_list:
         exc_list = []
+
     try:
         inet_num = getattr(netifaces, inet_type)
     except AttributeError:
-        raise Exception('Unknown inet type ' + str(inet_type))
+        raise Exception("Unknown inet type '%s'" % str(inet_type))
 
     interfaces = netifaces.interfaces()
     if inc_aliases:
@@ -202,15 +201,18 @@
         for _iface in interfaces:
             if iface == _iface or _iface.split(':')[0] == iface:
                 ifaces.append(_iface)
+
         if fatal and not ifaces:
             raise Exception("Invalid interface '%s'" % iface)
+
         ifaces.sort()
     else:
         if iface not in interfaces:
             if fatal:
-                raise Exception("%s not found " % (iface))
+                raise Exception("Interface '%s' not found " % (iface))
             else:
                 return []
+
         else:
             ifaces = [iface]
 
@@ -221,10 +223,13 @@
             for entry in net_info[inet_num]:
                 if 'addr' in entry and entry['addr'] not in exc_list:
                     addresses.append(entry['addr'])
+
     if fatal and not addresses:
         raise Exception("Interface '%s' doesn't have any %s addresses." %
                         (iface, inet_type))
-    return addresses
+
+    return sorted(addresses)
+
 
 get_ipv4_addr = partial(get_iface_addr, inet_type='AF_INET')
 
@@ -241,6 +246,7 @@
                 raw = re.match(ll_key, _addr)
                 if raw:
                     _addr = raw.group(1)
+
                 if _addr == addr:
                     log("Address '%s' is configured on iface '%s'" %
                         (addr, iface))
@@ -251,8 +257,9 @@
 
 
 def sniff_iface(f):
-    """If no iface provided, inject net iface inferred from unit private
-    address.
+    """Ensure decorated function is called with a value for iface.
+
+    If no iface provided, inject net iface inferred from unit private address.
     """
     def iface_sniffer(*args, **kwargs):
         if not kwargs.get('iface', None):
@@ -295,7 +302,7 @@
         if global_addrs:
             # Make sure any found global addresses are not temporary
             cmd = ['ip', 'addr', 'show', iface]
-            out = subprocess.check_output(cmd)
+            out = subprocess.check_output(cmd).decode('UTF-8')
             if dynamic_only:
                 key = re.compile("inet6 (.+)/[0-9]+ scope global dynamic.*")
             else:
@@ -317,33 +324,28 @@
                 return addrs
 
     if fatal:
-        raise Exception("Interface '%s' doesn't have a scope global "
+        raise Exception("Interface '%s' does not have a scope global "
                         "non-temporary ipv6 address." % iface)
 
     return []
 
 
 def get_bridges(vnic_dir='/sys/devices/virtual/net'):
-    """
-    Return a list of bridges on the system or []
-    """
-    b_rgex = vnic_dir + '/*/bridge'
-    return [x.replace(vnic_dir, '').split('/')[1] for x in glob.glob(b_rgex)]
+    """Return a list of bridges on the system."""
+    b_regex = "%s/*/bridge" % vnic_dir
+    return [x.replace(vnic_dir, '').split('/')[1] for x in glob.glob(b_regex)]
 
 
 def get_bridge_nics(bridge, vnic_dir='/sys/devices/virtual/net'):
-    """
-    Return a list of nics comprising a given bridge on the system or []
-    """
-    brif_rgex = "%s/%s/brif/*" % (vnic_dir, bridge)
-    return [x.split('/')[-1] for x in glob.glob(brif_rgex)]
+    """Return a list of nics comprising a given bridge on the system."""
+    brif_regex = "%s/%s/brif/*" % (vnic_dir, bridge)
+    return [x.split('/')[-1] for x in glob.glob(brif_regex)]
 
 
 def is_bridge_member(nic):
-    """
-    Check if a given nic is a member of a bridge
-    """
+    """Check if a given nic is a member of a bridge."""
     for bridge in get_bridges():
         if nic in get_bridge_nics(bridge):
             return True
+
     return False

=== modified file 'hooks/charmhelpers/contrib/storage/linux/utils.py'
--- hooks/charmhelpers/contrib/storage/linux/utils.py	2014-09-28 21:26:48 +0000
+++ hooks/charmhelpers/contrib/storage/linux/utils.py	2014-11-25 16:59:28 +0000
@@ -30,7 +30,8 @@
     # sometimes sgdisk exits non-zero; this is OK, dd will clean up
     call(['sgdisk', '--zap-all', '--mbrtogpt',
           '--clear', block_device])
-    dev_end = check_output(['blockdev', '--getsz', block_device])
+    dev_end = check_output(['blockdev', '--getsz',
+                            block_device]).decode('UTF-8')
     gpt_end = int(dev_end.split()[0]) - 100
     check_call(['dd', 'if=/dev/zero', 'of=%s' % (block_device),
                 'bs=1M', 'count=1'])
@@ -47,7 +48,7 @@
         it doesn't.
     '''
     is_partition = bool(re.search(r".*[0-9]+\b", device))
-    out = check_output(['mount'])
+    out = check_output(['mount']).decode('UTF-8')
     if is_partition:
         return bool(re.search(device + r"\b", out))
     return bool(re.search(device + r"[0-9]+\b", out))

=== modified file 'hooks/charmhelpers/core/fstab.py'
--- hooks/charmhelpers/core/fstab.py	2014-07-24 09:41:12 +0000
+++ hooks/charmhelpers/core/fstab.py	2014-11-25 16:59:28 +0000
@@ -3,10 +3,11 @@
 
 __author__ = 'Jorge Niedbalski R. <[e-mail]>'
 
+import io
 import os
 
 
-class Fstab(file):
+class Fstab(io.FileIO):
     """This class extends file in order to implement a file reader/writer
     for file `/etc/fstab`
     """
@@ -24,8 +25,8 @@
                 options = "defaults"
 
             self.options = options
-            self.d = d
-            self.p = p
+            self.d = int(d)
+            self.p = int(p)
 
         def __eq__(self, o):
             return str(self) == str(o)
@@ -45,7 +46,7 @@
             self._path = path
         else:
             self._path = self.DEFAULT_PATH
-        file.__init__(self, self._path, 'r+')
+        super(Fstab, self).__init__(self._path, 'rb+')
 
     def _hydrate_entry(self, line):
         # NOTE: use split with no arguments to split on any
@@ -58,8 +59,9 @@
     def entries(self):
         self.seek(0)
         for line in self.readlines():
+            line = line.decode('us-ascii')
             try:
-                if not line.startswith("#"):
+                if line.strip() and not line.startswith("#"):
                     yield self._hydrate_entry(line)
             except ValueError:
                 pass
@@ -75,14 +77,14 @@
         if self.get_entry_by_attr('device', entry.device):
             return False
 
-        self.write(str(entry) + '\n')
+        self.write((str(entry) + '\n').encode('us-ascii'))
         self.truncate()
         return entry
 
     def remove_entry(self, entry):
         self.seek(0)
 
-        lines = self.readlines()
+        lines = [l.decode('us-ascii') for l in self.readlines()]
 
         found = False
         for index, line in enumerate(lines):
@@ -97,7 +99,7 @@
         lines.remove(line)
 
         self.seek(0)
-        self.write(''.join(lines))
+        self.write(''.join(lines).encode('us-ascii'))
         self.truncate()
         return True
 

=== modified file 'hooks/charmhelpers/core/hookenv.py'
--- hooks/charmhelpers/core/hookenv.py	2014-10-21 07:56:51 +0000
+++ hooks/charmhelpers/core/hookenv.py	2014-11-25 16:59:28 +0000
@@ -9,9 +9,14 @@
 import yaml
 import subprocess
 import sys
-import UserDict
 from subprocess import CalledProcessError
 
+import six
+if not six.PY3:
+    from UserDict import UserDict
+else:
+    from collections import UserDict
+
 CRITICAL = "CRITICAL"
 ERROR = "ERROR"
 WARNING = "WARNING"
@@ -67,12 +72,12 @@
     subprocess.call(command)
 
 
-class Serializable(UserDict.IterableUserDict):
+class Serializable(UserDict):
     """Wrapper, an object that can be serialized to yaml or json"""
 
     def __init__(self, obj):
         # wrap the object
-        UserDict.IterableUserDict.__init__(self)
+        UserDict.__init__(self)
         self.data = obj
 
     def __getattr__(self, attr):
@@ -269,7 +274,7 @@
 
         """
         if self._prev_dict:
-            for k, v in self._prev_dict.iteritems():
+            for k, v in six.iteritems(self._prev_dict):
                 if k not in self:
                     self[k] = v
         with open(self.path, 'w') as f:
@@ -284,7 +289,8 @@
         config_cmd_line.append(scope)
     config_cmd_line.append('--format=json')
     try:
-        config_data = json.loads(subprocess.check_output(config_cmd_line))
+        config_data = json.loads(
+            subprocess.check_output(config_cmd_line).decode('UTF-8'))
         if scope is not None:
             return config_data
         return Config(config_data)
@@ -303,10 +309,10 @@
     if unit:
         _args.append(unit)
     try:
-        return json.loads(subprocess.check_output(_args))
+        return json.loads(subprocess.check_output(_args).decode('UTF-8'))
     except ValueError:
         return None
-    except CalledProcessError, e:
+    except CalledProcessError as e:
         if e.returncode == 2:
             return None
         raise
@@ -318,7 +324,7 @@
     relation_cmd_line = ['relation-set']
     if relation_id is not None:
         relation_cmd_line.extend(('-r', relation_id))
-    for k, v in (relation_settings.items() + kwargs.items()):
+    for k, v in (list(relation_settings.items()) + list(kwargs.items())):
         if v is None:
             relation_cmd_line.append('{}='.format(k))
         else:
@@ -335,7 +341,8 @@
     relid_cmd_line = ['relation-ids', '--format=json']
     if reltype is not None:
         relid_cmd_line.append(reltype)
-        return json.loads(subprocess.check_output(relid_cmd_line)) or []
+        return json.loads(
+            subprocess.check_output(relid_cmd_line).decode('UTF-8')) or []
     return []
 
 
@@ -346,7 +353,8 @@
     units_cmd_line = ['relation-list', '--format=json']
     if relid is not None:
         units_cmd_line.extend(('-r', relid))
-    return json.loads(subprocess.check_output(units_cmd_line)) or []
+    return json.loads(
+        subprocess.check_output(units_cmd_line).decode('UTF-8')) or []
 
 
 @cached
@@ -455,7 +463,7 @@
     """Get the unit ID for the remote unit"""
     _args = ['unit-get', '--format=json', attribute]
     try:
-        return json.loads(subprocess.check_output(_args))
+        return json.loads(subprocess.check_output(_args).decode('UTF-8'))
     except ValueError:
         return None
 

=== modified file 'hooks/charmhelpers/core/host.py'
--- hooks/charmhelpers/core/host.py	2014-10-21 07:56:51 +0000
+++ hooks/charmhelpers/core/host.py	2014-11-25 16:59:28 +0000
@@ -14,11 +14,12 @@
 import subprocess
 import hashlib
 from contextlib import contextmanager
-
 from collections import OrderedDict
 
-from hookenv import log
-from fstab import Fstab
+import six
+
+from .hookenv import log
+from .fstab import Fstab
 
 
 def service_start(service_name):
@@ -54,7 +55,9 @@
 def service_running(service):
     """Determine whether a system service is running"""
     try:
-        output = subprocess.check_output(['service', service, 'status'], stderr=subprocess.STDOUT)
+        output = subprocess.check_output(
+            ['service', service, 'status'],
+            stderr=subprocess.STDOUT).decode('UTF-8')
     except subprocess.CalledProcessError:
         return False
     else:
@@ -67,7 +70,9 @@
 def service_available(service_name):
     """Determine whether a system service is available"""
     try:
-        subprocess.check_output(['service', service_name, 'status'], stderr=subprocess.STDOUT)
+        subprocess.check_output(
+            ['service', service_name, 'status'],
+            stderr=subprocess.STDOUT).decode('UTF-8')
     except subprocess.CalledProcessError as e:
         return 'unrecognized service' not in e.output
     else:
@@ -115,7 +120,7 @@
     cmd.append(from_path)
     cmd.append(to_path)
     log(" ".join(cmd))
-    return subprocess.check_output(cmd).strip()
+    return subprocess.check_output(cmd).decode('UTF-8').strip()
 
 
 def symlink(source, destination):
@@ -130,7 +135,7 @@
     subprocess.check_call(cmd)
 
 
-def mkdir(path, owner='root', group='root', perms=0555, force=False):
+def mkdir(path, owner='root', group='root', perms=0o555, force=False):
     """Create a directory"""
     log("Making dir {} {}:{} {:o}".format(path, owner, group,
                                           perms))
@@ -146,7 +151,7 @@
     os.chown(realpath, uid, gid)
 
 
-def write_file(path, content, owner='root', group='root', perms=0444):
+def write_file(path, content, owner='root', group='root', perms=0o444):
     """Create or overwrite a file with the contents of a string"""
     log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))
     uid = pwd.getpwnam(owner).pw_uid
@@ -177,7 +182,7 @@
     cmd_args.extend([device, mountpoint])
     try:
         subprocess.check_output(cmd_args)
-    except subprocess.CalledProcessError, e:
+    except subprocess.CalledProcessError as e:
         log('Error mounting {} at {}\n{}'.format(device, mountpoint, e.output))
         return False
 
@@ -191,7 +196,7 @@
     cmd_args = ['umount', mountpoint]
     try:
         subprocess.check_output(cmd_args)
-    except subprocess.CalledProcessError, e:
+    except subprocess.CalledProcessError as e:
         log('Error unmounting {}\n{}'.format(mountpoint, e.output))
         return False
 
@@ -218,8 +223,8 @@
     """
     if os.path.exists(path):
         h = getattr(hashlib, hash_type)()
-        with open(path, 'r') as source:
-            h.update(source.read())  # IGNORE:E1101 - it does have update
+        with open(path, 'rb') as source:
+            h.update(source.read())
         return h.hexdigest()
     else:
         return None
@@ -297,7 +302,7 @@
     if length is None:
         length = random.choice(range(35, 45))
     alphanumeric_chars = [
-        l for l in (string.letters + string.digits)
+        l for l in (string.ascii_letters + string.digits)
         if l not in 'l0QD1vAEIOUaeiou']
     random_chars = [
         random.choice(alphanumeric_chars) for _ in range(length)]
@@ -306,14 +311,14 @@
 
 def list_nics(nic_type):
     '''Return a list of nics of given type(s)'''
-    if isinstance(nic_type, basestring):
+    if isinstance(nic_type, six.string_types):
         int_types = [nic_type]
     else:
         int_types = nic_type
     interfaces = []
     for int_type in int_types:
         cmd = ['ip', 'addr', 'show', 'label', int_type + '*']
-        ip_output = subprocess.check_output(cmd).split('\n')
+        ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
         ip_output = (line for line in ip_output if line)
         for line in ip_output:
             if line.split()[1].startswith(int_type):
@@ -335,7 +340,7 @@
 
 def get_nic_mtu(nic):
     cmd = ['ip', 'addr', 'show', nic]
-    ip_output = subprocess.check_output(cmd).split('\n')
+    ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
     mtu = ""
     for line in ip_output:
         words = line.split()
@@ -346,7 +351,7 @@
 
 def get_nic_hwaddr(nic):
     cmd = ['ip', '-o', '-0', 'addr', 'show', nic]
-    ip_output = subprocess.check_output(cmd)
+    ip_output = subprocess.check_output(cmd).decode('UTF-8')
     hwaddr = ""
     words = ip_output.split()
     if 'link/ether' in words:

=== modified file 'hooks/charmhelpers/core/services/__init__.py'
--- hooks/charmhelpers/core/services/__init__.py	2014-09-26 08:25:57 +0000
+++ hooks/charmhelpers/core/services/__init__.py	2014-11-25 16:59:28 +0000
@@ -1,2 +1,2 @@
-from .base import *
-from .helpers import *
+from .base import *  # NOQA
+from .helpers import *  # NOQA

=== modified file 'hooks/charmhelpers/core/services/helpers.py'
--- hooks/charmhelpers/core/services/helpers.py	2014-09-26 08:25:57 +0000
+++ hooks/charmhelpers/core/services/helpers.py	2014-11-25 16:59:28 +0000
@@ -196,7 +196,7 @@
         if not os.path.isabs(file_name):
             file_name = os.path.join(hookenv.charm_dir(), file_name)
         with open(file_name, 'w') as file_stream:
-            os.fchmod(file_stream.fileno(), 0600)
+            os.fchmod(file_stream.fileno(), 0o600)
             yaml.dump(config_data, file_stream)
 
     def read_context(self, file_name):
@@ -211,15 +211,19 @@
 
 class TemplateCallback(ManagerCallback):
     """
-    Callback class that will render a Jinja2 template, for use as a ready action.
-
-    :param str source: The template source file, relative to `$CHARM_DIR/templates`
+    Callback class that will render a Jinja2 template, for use as a ready
+    action.
+
+    :param str source: The template source file, relative to
+    `$CHARM_DIR/templates`
+
     :param str target: The target to write the rendered template to
     :param str owner: The owner of the rendered file
     :param str group: The group of the rendered file
     :param int perms: The permissions of the rendered file
     """
-    def __init__(self, source, target, owner='root', group='root', perms=0444):
+    def __init__(self, source, target,
+                 owner='root', group='root', perms=0o444):
         self.source = source
         self.target = target
         self.owner = owner

=== added file 'hooks/charmhelpers/core/sysctl.py'
--- hooks/charmhelpers/core/sysctl.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/core/sysctl.py	2014-11-25 16:59:28 +0000
@@ -0,0 +1,34 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+__author__ = 'Jorge Niedbalski R. <[e-mail]>'
+
+import yaml
+
+from subprocess import check_call
+
+from charmhelpers.core.hookenv import (
+    log,
+    DEBUG,
+)
+
+
+def create(sysctl_dict, sysctl_file):
+    """Creates a sysctl.conf file from a YAML associative array
+
+    :param sysctl_dict: a dict of sysctl options eg { 'kernel.max_pid': 1337 }
+    :type sysctl_dict: dict
+    :param sysctl_file: path to the sysctl file to be saved
+    :type sysctl_file: str or unicode
+    :returns: None
+    """
+    sysctl_dict = yaml.load(sysctl_dict)
+
+    with open(sysctl_file, "w") as fd:
+        for key, value in sysctl_dict.items():
+            fd.write("{}={}\n".format(key, value))
+
+    log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict),
+        level=DEBUG)
+
+    check_call(["sysctl", "-p", sysctl_file])

=== removed file 'hooks/charmhelpers/core/sysctl.py'
--- hooks/charmhelpers/core/sysctl.py	2014-10-21 07:56:51 +0000
+++ hooks/charmhelpers/core/sysctl.py	1970-01-01 00:00:00 +0000
@@ -1,34 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-
-__author__ = 'Jorge Niedbalski R. <[e-mail]>'
-
-import yaml
-
-from subprocess import check_call
-
-from charmhelpers.core.hookenv import (
-    log,
-    DEBUG,
-)
-
-
-def create(sysctl_dict, sysctl_file):
-    """Creates a sysctl.conf file from a YAML associative array
-
-    :param sysctl_dict: a dict of sysctl options eg { 'kernel.max_pid': 1337 }
-    :type sysctl_dict: dict
-    :param sysctl_file: path to the sysctl file to be saved
-    :type sysctl_file: str or unicode
-    :returns: None
-    """
-    sysctl_dict = yaml.load(sysctl_dict)
-
-    with open(sysctl_file, "w") as fd:
-        for key, value in sysctl_dict.items():
-            fd.write("{}={}\n".format(key, value))
-
-    log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict),
-        level=DEBUG)
-
-    check_call(["sysctl", "-p", sysctl_file])

=== modified file 'hooks/charmhelpers/core/templating.py'
--- hooks/charmhelpers/core/templating.py	2014-09-26 08:25:57 +0000
+++ hooks/charmhelpers/core/templating.py	2014-11-25 16:59:28 +0000
@@ -4,7 +4,8 @@
 from charmhelpers.core import hookenv
 
 
-def render(source, target, context, owner='root', group='root', perms=0444, templates_dir=None):
+def render(source, target, context, owner='root', group='root',
+           perms=0o444, templates_dir=None):
     """
     Render a template.
 

=== modified file 'hooks/charmhelpers/fetch/__init__.py'
--- hooks/charmhelpers/fetch/__init__.py	2014-10-21 07:56:51 +0000
+++ hooks/charmhelpers/fetch/__init__.py	2014-11-25 16:59:28 +0000
@@ -5,10 +5,6 @@
 from charmhelpers.core.host import (
     lsb_release
 )
-from urlparse import (
-    urlparse,
-    urlunparse,
-)
 import subprocess
 from charmhelpers.core.hookenv import (
     config,
@@ -16,6 +12,12 @@
 )
 import os
 
+import six
+if six.PY3:
+    from urllib.parse import urlparse, urlunparse
+else:
+    from urlparse import urlparse, urlunparse
+
 
 CLOUD_ARCHIVE = """# Ubuntu Cloud Archive
 deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main
@@ -149,7 +151,7 @@
     cmd = ['apt-get', '--assume-yes']
     cmd.extend(options)
     cmd.append('install')
-    if isinstance(packages, basestring):
+    if isinstance(packages, six.string_types):
         cmd.append(packages)
     else:
         cmd.extend(packages)
@@ -182,7 +184,7 @@
 def apt_purge(packages, fatal=False):
     """Purge one or more packages"""
     cmd = ['apt-get', '--assume-yes', 'purge']
-    if isinstance(packages, basestring):
+    if isinstance(packages, six.string_types):
         cmd.append(packages)
     else:
         cmd.extend(packages)
@@ -193,7 +195,7 @@
 def apt_hold(packages, fatal=False):
     """Hold one or more packages"""
     cmd = ['apt-mark', 'hold']
-    if isinstance(packages, basestring):
+    if isinstance(packages, six.string_types):
         cmd.append(packages)
     else:
         cmd.extend(packages)
@@ -256,11 +258,11 @@
     elif source == 'distro':
         pass
     else:
-        raise SourceConfigError("Unknown source: {!r}".format(source))
+        log("Unknown source: {!r}".format(source))
 
     if key:
         if '-----BEGIN PGP PUBLIC KEY BLOCK-----' in key:
-            with NamedTemporaryFile() as key_file:
+            with NamedTemporaryFile('w+') as key_file:
                 key_file.write(key)
                 key_file.flush()
                 key_file.seek(0)
@@ -297,14 +299,14 @@
     sources = safe_load((config(sources_var) or '').strip()) or []
     keys = safe_load((config(keys_var) or '').strip()) or None
 
-    if isinstance(sources, basestring):
+    if isinstance(sources, six.string_types):
         sources = [sources]
 
     if keys is None:
         for source in sources:
             add_source(source, None)
     else:
-        if isinstance(keys, basestring):
+        if isinstance(keys, six.string_types):
             keys = [keys]
 
         if len(sources) != len(keys):
@@ -401,7 +403,7 @@
         while result is None or result == APT_NO_LOCK:
             try:
                 result = subprocess.check_call(cmd, env=env)
-            except subprocess.CalledProcessError, e:
+            except subprocess.CalledProcessError as e:
                 retry_count = retry_count + 1
                 if retry_count > APT_NO_LOCK_RETRY_COUNT:
                     raise

=== modified file 'hooks/charmhelpers/fetch/archiveurl.py'
--- hooks/charmhelpers/fetch/archiveurl.py	2014-09-28 21:26:48 +0000
+++ hooks/charmhelpers/fetch/archiveurl.py	2014-11-25 16:59:28 +0000
@@ -1,8 +1,23 @@
 import os
-import urllib2
-from urllib import urlretrieve
-import urlparse
 import hashlib
+import re
+
+import six
+if six.PY3:
+    from urllib.request import (
+        build_opener, install_opener, urlopen, urlretrieve,
+        HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler,
+    )
+    from urllib.parse import urlparse, urlunparse, parse_qs
+    from urllib.error import URLError
+else:
+    from urllib import urlretrieve
+    from urllib2 import (
+        build_opener, install_opener, urlopen,
+        HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler,
+        URLError
+    )
+    from urlparse import urlparse, urlunparse, parse_qs
 
 from charmhelpers.fetch import (
     BaseFetchHandler,
@@ -15,6 +30,24 @@
 from charmhelpers.core.host import mkdir, check_hash
 
 
+def splituser(host):
+    '''urllib.splituser(), but six's support of this seems broken'''
+    _userprog = re.compile('^(.*)@(.*)$')
+    match = _userprog.match(host)
+    if match:
+        return match.group(1, 2)
+    return None, host
+
+
+def splitpasswd(user):
+    '''urllib.splitpasswd(), but six's support of this is missing'''
+    _passwdprog = re.compile('^([^:]*):(.*)$', re.S)
+    match = _passwdprog.match(user)
+    if match:
+        return match.group(1, 2)
+    return user, None
+
+
 class ArchiveUrlFetchHandler(BaseFetchHandler):
     """
     Handler to download archive files from arbitrary URLs.
@@ -42,20 +75,20 @@
         """
         # propogate all exceptions
         # URLError, OSError, etc
-        proto, netloc, path, params, query, fragment = urlparse.urlparse(source)
+        proto, netloc, path, params, query, fragment = urlparse(source)
         if proto in ('http', 'https'):
-            auth, barehost = urllib2.splituser(netloc)
+            auth, barehost = splituser(netloc)
             if auth is not None:
-                source = urlparse.urlunparse((proto, barehost, path, params, query, fragment))
-                username, password = urllib2.splitpasswd(auth)
-                passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
+                source = urlunparse((proto, barehost, path, params, query, fragment))
+                username, password = splitpasswd(auth)
+                passman = HTTPPasswordMgrWithDefaultRealm()
                 # Realm is set to None in add_password to force the username and password
                 # to be used whatever the realm
                 passman.add_password(None, source, username, password)
-                authhandler = urllib2.HTTPBasicAuthHandler(passman)
-                opener = urllib2.build_opener(authhandler)
-                urllib2.install_opener(opener)
-        response = urllib2.urlopen(source)
+                authhandler = HTTPBasicAuthHandler(passman)
+                opener = build_opener(authhandler)
+                install_opener(opener)
+        response = urlopen(source)
         try:
             with open(dest, 'w') as dest_file:
                 dest_file.write(response.read())
@@ -91,17 +124,21 @@
         url_parts = self.parse_url(source)
         dest_dir = os.path.join(os.environ.get('CHARM_DIR'), 'fetched')
         if not os.path.exists(dest_dir):
-            mkdir(dest_dir, perms=0755)
+            mkdir(dest_dir, perms=0o755)
         dld_file = os.path.join(dest_dir, os.path.basename(url_parts.path))
         try:
             self.download(source, dld_file)
-        except urllib2.URLError as e:
+        except URLError as e:
             raise UnhandledSource(e.reason)
         except OSError as e:
             raise UnhandledSource(e.strerror)
-        options = urlparse.parse_qs(url_parts.fragment)
+        options = parse_qs(url_parts.fragment)
         for key, value in options.items():
-            if key in hashlib.algorithms:
+            if not six.PY3:
+                algorithms = hashlib.algorithms
+            else:
+                algorithms = hashlib.algorithms_available
+            if key in algorithms:
                 check_hash(dld_file, value, key)
         if checksum:
             check_hash(dld_file, checksum, hash_type)

=== modified file 'hooks/charmhelpers/fetch/bzrurl.py'
--- hooks/charmhelpers/fetch/bzrurl.py	2014-07-25 08:07:41 +0000
+++ hooks/charmhelpers/fetch/bzrurl.py	2014-11-25 16:59:28 +0000
@@ -5,6 +5,10 @@
 )
 from charmhelpers.core.host import mkdir
 
+import six
+if six.PY3:
+    raise ImportError('bzrlib does not support Python3')
+
 try:
     from bzrlib.branch import Branch
 except ImportError:
@@ -42,7 +46,7 @@
         dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
                                 branch_name)
         if not os.path.exists(dest_dir):
-            mkdir(dest_dir, perms=0755)
+            mkdir(dest_dir, perms=0o755)
         try:
             self.branch(source, dest_dir)
         except OSError as e:

=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2014-09-30 03:41:06 +0000
+++ hooks/hooks.py	2014-11-25 16:59:28 +0000
@@ -36,6 +36,8 @@
     filter_installed_packages,
 )
 
+from charmhelpers.core.sysctl import create as create_sysctl
+
 from utils import (
     render_template,
     get_host_ip,
@@ -110,6 +112,10 @@
     if config('prefer-ipv6'):
         assert_charm_supports_ipv6()
 
+    sysctl_dict = config('sysctl')
+    if sysctl_dict:
+        create_sysctl(sysctl_dict, '/etc/sysctl.d/50-ceph-osd-charm.conf')
+
     e_mountpoint = config('ephemeral-unmount')
     if (e_mountpoint and ceph.filesystem_mounted(e_mountpoint)):
         umount(e_mountpoint)

=== modified file 'tests/charmhelpers/contrib/amulet/deployment.py'
--- tests/charmhelpers/contrib/amulet/deployment.py	2014-09-27 17:18:26 +0000
+++ tests/charmhelpers/contrib/amulet/deployment.py	2014-11-25 16:59:28 +0000
@@ -1,6 +1,6 @@
 import amulet
-
 import os
+import six
 
 
 class AmuletDeployment(object):
@@ -52,12 +52,12 @@
 
     def _add_relations(self, relations):
         """Add all of the relations for the services."""
-        for k, v in relations.iteritems():
+        for k, v in six.iteritems(relations):
             self.d.relate(k, v)
 
     def _configure_services(self, configs):
         """Configure all of the services."""
-        for service, config in configs.iteritems():
+        for service, config in six.iteritems(configs):
             self.d.configure(service, config)
 
     def _deploy(self):

=== modified file 'tests/charmhelpers/contrib/amulet/utils.py'
--- tests/charmhelpers/contrib/amulet/utils.py	2014-09-27 17:18:26 +0000
+++ tests/charmhelpers/contrib/amulet/utils.py	2014-11-25 16:59:28 +0000
@@ -5,6 +5,8 @@
 import sys
 import time
 
+import six
+
 
 class AmuletUtils(object):
     """Amulet utilities.
@@ -58,7 +60,7 @@
            Verify the specified services are running on the corresponding
            service units.
            """
-        for k, v in commands.iteritems():
+        for k, v in six.iteritems(commands):
             for cmd in v:
                 output, code = k.run(cmd)
                 if code != 0:
@@ -100,11 +102,11 @@
            longs, or can be a function that evaluate a variable and returns a
            bool.
            """
-        for k, v in expected.iteritems():
+        for k, v in six.iteritems(expected):
             if k in actual:
-                if (isinstance(v, basestring) or
+                if (isinstance(v, six.string_types) or
                         isinstance(v, bool) or
-                        isinstance(v, (int, long))):
+                        isinstance(v, six.integer_types)):
                     if v != actual[k]:
                         return "{}:{}".format(k, actual[k])
                 elif not v(actual[k]):

=== modified file 'tests/charmhelpers/contrib/openstack/amulet/deployment.py'
--- tests/charmhelpers/contrib/openstack/amulet/deployment.py	2014-09-29 20:39:40 +0000
+++ tests/charmhelpers/contrib/openstack/amulet/deployment.py	2014-11-25 16:59:28 +0000
@@ -1,3 +1,4 @@
+import six
 from charmhelpers.contrib.amulet.deployment import (
     AmuletDeployment
 )
@@ -69,7 +70,7 @@
 
     def _configure_services(self, configs):
         """Configure all of the services."""
-        for service, config in configs.iteritems():
+        for service, config in six.iteritems(configs):
             self.d.configure(service, config)
 
     def _get_openstack_release(self):

=== modified file 'tests/charmhelpers/contrib/openstack/amulet/utils.py'
--- tests/charmhelpers/contrib/openstack/amulet/utils.py	2014-09-27 17:18:26 +0000
+++ tests/charmhelpers/contrib/openstack/amulet/utils.py	2014-11-25 16:59:28 +0000
@@ -7,6 +7,8 @@
 import keystoneclient.v2_0 as keystone_client
 import novaclient.v1_1.client as nova_client
 
+import six
+
 from charmhelpers.contrib.amulet.utils import (
     AmuletUtils
 )
@@ -60,7 +62,7 @@
            expected service catalog endpoints.
            """
         self.log.debug('actual: {}'.format(repr(actual)))
-        for k, v in expected.iteritems():
+        for k, v in six.iteritems(expected):
             if k in actual:
                 ret = self._validate_dict_data(expected[k][0], actual[k][0])
                 if ret: