Merge lp:~xfactor973/charms/trusty/ceph/erasure-wip into lp:~openstack-charmers-archive/charms/trusty/ceph/next

[James Page]

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #9157 ceph-next for james-page mp270983
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/12409460/
Build: http://10.245.162.77:8080/job/charm_unit_test/9157/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #9938 ceph-next for james-page mp270983
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/12409461/
Build: http://10.245.162.77:8080/job/charm_lint_check/9938/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #6422 ceph-next for james-page mp270983
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12410066/
Build: http://10.245.162.77:8080/job/charm_amulet_test/6422/

[Edward Hope-Morley (hopem)]

Hi Chris, not sure why but you have a lot of merge conflicts here (as well as lint unit and amulet fails which may be a consequence). Can you have a go at resyncing with /next?

review: Needs Fixing

[Chris Holcombe (xfactor973)]

I was actually just looking for feedback about my code. I wanted to know if the approach to the API looked ok. I'm not sure why it's being merged. There was probably some miscommunication.

[Edward Hope-Morley (hopem)]

Ack, well in any case it would be easier to read without the conflicts.

Unmerged revisions

111. By Chris Holcombe on 2015-08-25

Actions associated with the pool commands

110. By Chris Holcombe on 2015-08-25

WIP for erasure coding support

109. By Corey Bryant on 2015-08-10

[beisner,r=corey.bryant] Point charmhelper sync and amulet tests at stable branches.

108. By James Page on 2015-08-10

[gnuoy] 15.07 Charm release

107. By Liam Young on 2015-04-24

Point charmhelper sync and amulet tests at stable branches

=== modified file '.bzrignore'
--- .bzrignore	2014-10-01 20:08:33 +0000
+++ .bzrignore	2015-09-14 15:56:04 +0000
@@ -1,2 +1,3 @@
 bin
 .coverage
+.idea

=== added directory 'actions'
=== added file 'actions.yaml'
--- actions.yaml	1970-01-01 00:00:00 +0000
+++ actions.yaml	2015-09-14 15:56:04 +0000
@@ -0,0 +1,128 @@
+create-pool:
+  description:
+  params:
+    name: 
+      type: string
+      description: The name of the pool
+    placement-groups: 
+      type: integer
+      description: The total number of placement groups for the pool.
+    placement-purpose-groups: 
+      type: integer
+      description: The total number of placement groups for placement purposes. This should be equal to the total number of placement groups
+    profile-name:
+      type: String
+      description: The crush profile to use for this pool.  The ruleset must exist first.
+    pool-type:
+      type: string
+        kind:
+          type: string
+          enum: [Replicated, Erasure]
+      description: The pool type which may either be replicated to recover from lost OSDs by keeping multiple copies of the objects or erasure to get a kind of generalized RAID5 capability.
+  additionalProperties: false
+     
+create-erasure-profile:
+  description: Create a new erasure code profile to use on a pool.
+  additionalProperties: false
+
+get-erasure-profile:
+  description: Display an erasure code profile.
+  params:
+    name: 
+      type: string
+      description: The name of the profile
+  additionalProperties: false
+
+delete-erasure-profile:
+  description: Deletes an erasure code profile.
+  params:
+    name: 
+      type: string
+      description: The name of the profile
+  additionalProperties: false
+  
+list-erasure-profiles:
+  description: List the names of all erasure code profiles
+  additionalProperties: false
+
+list-pools:
+  description: List your cluster’s pools
+  additionalProperties: false
+
+set-pool-max-objects: 
+  description: Set pool quotas for the maximum number of objects per pool.
+  params:
+    max: 
+      type: integer
+      description: The name of the pool
+  additionalProperties: false
+
+set-pool-max-bytes: 
+  description: Set pool quotas for the maximum number of bytes.
+  params:
+    max: 
+      type: integer
+      description: The name of the pool
+  additionalProperties: false
+
+delete-pool:
+  description: Deletes the named pool
+  params:
+    name: 
+      type: string
+      description: The name of the pool
+  additionalProperties: false
+
+rename-pool:
+  description:
+  params:
+    name: 
+      type: string
+      description: The name of the pool
+  additionalProperties: false
+
+pool-statistics:
+  description: Show a pool’s utilization statistics
+  additionalProperties: false
+
+snapshot-pool:
+  description: 
+  params:
+    pool-name: 
+      type: string
+      description: The name of the pool
+    snapshot-name: 
+      type: string
+      description: The name of the snapshot
+  additionalProperties: false
+
+remove-pool-snapshot:
+  description:
+  params:
+    pool-name: 
+      type: string
+      description: The name of the pool
+    snapshot-name: 
+      type: string
+      description: The name of the snapshot
+  additionalProperties: false
+
+pool-set:
+  description:
+  params:
+    key:
+      type: string
+      description: Any valid Ceph key
+    value:
+      type: string
+      description: The value to set
+  additionalProperties: false
+
+pool-get:
+  description:
+  params:
+    key: 
+      type: string
+      description: Any valid Ceph key
+  additionalProperties: false
+

=== added file 'actions/create-erasure-profile'
=== added file 'actions/create-pool'
=== added file 'actions/delete-pool'
=== added file 'actions/list-pools'
=== added file 'actions/pool-get'
=== added file 'actions/pool-set'
=== added file 'actions/pool-statistics'
=== added file 'actions/remove-pool-snapshot'
=== added file 'actions/rename-pool'
=== added file 'actions/set-pool-max-bytes'
=== added file 'actions/set-pool-max-objects'
=== added file 'actions/snapshot-pool'
=== modified file 'charm-helpers-hooks.yaml'
--- charm-helpers-hooks.yaml	2015-09-07 08:23:57 +0000
+++ charm-helpers-hooks.yaml	2015-09-14 15:56:04 +0000
@@ -1,4 +1,4 @@
-branch: lp:charm-helpers
+branch: lp:~openstack-charmers/charm-helpers/stable
 destination: hooks/charmhelpers
 include:
     - core

=== modified file 'charm-helpers-tests.yaml'
--- charm-helpers-tests.yaml	2015-06-15 20:42:45 +0000
+++ charm-helpers-tests.yaml	2015-09-14 15:56:04 +0000
@@ -1,4 +1,4 @@
-branch: lp:charm-helpers
+branch: lp:~openstack-charmers/charm-helpers/stable
 destination: tests/charmhelpers
 include:
     - contrib.amulet

=== modified file 'config.yaml'
--- config.yaml	2015-07-10 14:14:18 +0000
+++ config.yaml	2015-09-14 15:56:04 +0000
@@ -7,6 +7,28 @@
       .
       This configuration element is mandatory and the service will fail on
       install if it is not provided.
+  pool-type:
+    type: string
+    default: Replicated
+    description: |
+      Ceph supports both Replicated and Erasure coded pools. If this option is
+      set to Erasure then two additional fields might need to be adjusted.
+      For more information see: http://ceph.com/docs/master/rados/operations/pools/#create-a-pool
+      Valid options are "Replicated", "Erasure", "LocalErasureCoded"
+  erasure-data-chunks:
+    type: int
+    default: 2
+    description: |
+      Each object is split in {k} data-chunks and stored on a different OSD
+  erasure-coding-chunks:
+    type: int
+    default: 3
+    description: |
+      Compute {m} parity chunks for each object.  The ratio of {k} to {m} will determine your
+      erasure coding overhead.  Example: erasure-data-chunks=10, erasure-coding-chunks=4.
+      Objects are divided into 10 chunks and an additional 4 chunks of parity are created.
+      40% overhead for an object that will not be lost unless 4 OSDs break at the same time.
+      A Replication pool would require 400% overhead to achieve the same failure tolerance.
   auth-supported:
     type: string
     default: cephx

=== modified file 'hooks/ceph_broker.py'
--- hooks/ceph_broker.py	2015-09-04 10:33:49 +0000
+++ hooks/ceph_broker.py	2015-09-14 15:56:04 +0000
@@ -75,7 +75,9 @@
         svc = 'admin'
         if op == "create-pool":
             params = {'pool': req.get('name'),
-                      'replicas': req.get('replicas')}
+                      'pool_type': req.get('pool_type'),
+                      }
+                      #'replicas': req.get('replicas')}
             if not all(params.iteritems()):
                 msg = ("Missing parameter(s): %s" %
                        (' '.join([k for k in params.iterkeys()
@@ -85,6 +87,7 @@
 
             pool = params['pool']
             replicas = params['replicas']
+            pool_type = params['pool-type']
             if not pool_exists(service=svc, name=pool):
                 log("Creating pool '%s' (replicas=%s)" % (pool, replicas),
                     level=INFO)

=== added directory 'hooks/charmhelpers/cli'
=== renamed directory 'hooks/charmhelpers/cli' => 'hooks/charmhelpers/cli.moved'
=== added file 'hooks/charmhelpers/cli/__init__.py'
--- hooks/charmhelpers/cli/__init__.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/cli/__init__.py	2015-09-14 15:56:04 +0000
@@ -0,0 +1,195 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+import inspect
+import argparse
+import sys
+
+from six.moves import zip
+
+from charmhelpers.core import unitdata
+
+
+class OutputFormatter(object):
+    def __init__(self, outfile=sys.stdout):
+        self.formats = (
+            "raw",
+            "json",
+            "py",
+            "yaml",
+            "csv",
+            "tab",
+        )
+        self.outfile = outfile
+
+    def add_arguments(self, argument_parser):
+        formatgroup = argument_parser.add_mutually_exclusive_group()
+        choices = self.supported_formats
+        formatgroup.add_argument("--format", metavar='FMT',
+                                 help="Select output format for returned data, "
+                                      "where FMT is one of: {}".format(choices),
+                                 choices=choices, default='raw')
+        for fmt in self.formats:
+            fmtfunc = getattr(self, fmt)
+            formatgroup.add_argument("-{}".format(fmt[0]),
+                                     "--{}".format(fmt), action='store_const',
+                                     const=fmt, dest='format',
+                                     help=fmtfunc.__doc__)
+
+    @property
+    def supported_formats(self):
+        return self.formats
+
+    def raw(self, output):
+        """Output data as raw string (default)"""
+        if isinstance(output, (list, tuple)):
+            output = '\n'.join(map(str, output))
+        self.outfile.write(str(output))
+
+    def py(self, output):
+        """Output data as a nicely-formatted python data structure"""
+        import pprint
+        pprint.pprint(output, stream=self.outfile)
+
+    def json(self, output):
+        """Output data in JSON format"""
+        import json
+        json.dump(output, self.outfile)
+
+    def yaml(self, output):
+        """Output data in YAML format"""
+        import yaml
+        yaml.safe_dump(output, self.outfile)
+
+    def csv(self, output):
+        """Output data as excel-compatible CSV"""
+        import csv
+        csvwriter = csv.writer(self.outfile)
+        csvwriter.writerows(output)
+
+    def tab(self, output):
+        """Output data in excel-compatible tab-delimited format"""
+        import csv
+        csvwriter = csv.writer(self.outfile, dialect=csv.excel_tab)
+        csvwriter.writerows(output)
+
+    def format_output(self, output, fmt='raw'):
+        fmtfunc = getattr(self, fmt)
+        fmtfunc(output)
+
+
+class CommandLine(object):
+    argument_parser = None
+    subparsers = None
+    formatter = None
+    exit_code = 0
+
+    def __init__(self):
+        if not self.argument_parser:
+            self.argument_parser = argparse.ArgumentParser(description='Perform common charm tasks')
+        if not self.formatter:
+            self.formatter = OutputFormatter()
+            self.formatter.add_arguments(self.argument_parser)
+        if not self.subparsers:
+            self.subparsers = self.argument_parser.add_subparsers(help='Commands')
+
+    def subcommand(self, command_name=None):
+        """
+        Decorate a function as a subcommand. Use its arguments as the
+        command-line arguments"""
+        def wrapper(decorated):
+            cmd_name = command_name or decorated.__name__
+            subparser = self.subparsers.add_parser(cmd_name,
+                                                   description=decorated.__doc__)
+            for args, kwargs in describe_arguments(decorated):
+                subparser.add_argument(*args, **kwargs)
+            subparser.set_defaults(func=decorated)
+            return decorated
+        return wrapper
+
+    def test_command(self, decorated):
+        """
+        Subcommand is a boolean test function, so bool return values should be
+        converted to a 0/1 exit code.
+        """
+        decorated._cli_test_command = True
+        return decorated
+
+    def no_output(self, decorated):
+        """
+        Subcommand is not expected to return a value, so don't print a spurious None.
+        """
+        decorated._cli_no_output = True
+        return decorated
+
+    def subcommand_builder(self, command_name, description=None):
+        """
+        Decorate a function that builds a subcommand. Builders should accept a
+        single argument (the subparser instance) and return the function to be
+        run as the command."""
+        def wrapper(decorated):
+            subparser = self.subparsers.add_parser(command_name)
+            func = decorated(subparser)
+            subparser.set_defaults(func=func)
+            subparser.description = description or func.__doc__
+        return wrapper
+
+    def run(self):
+        "Run cli, processing arguments and executing subcommands."
+        arguments = self.argument_parser.parse_args()
+        argspec = inspect.getargspec(arguments.func)
+        vargs = []
+        kwargs = {}
+        for arg in argspec.args:
+            vargs.append(getattr(arguments, arg))
+        if argspec.varargs:
+            vargs.extend(getattr(arguments, argspec.varargs))
+        if argspec.keywords:
+            for kwarg in argspec.keywords.items():
+                kwargs[kwarg] = getattr(arguments, kwarg)
+        output = arguments.func(*vargs, **kwargs)
+        if getattr(arguments.func, '_cli_test_command', False):
+            self.exit_code = 0 if output else 1
+            output = ''
+        if getattr(arguments.func, '_cli_no_output', False):
+            output = ''
+        self.formatter.format_output(output, arguments.format)
+        if unitdata._KV:
+            unitdata._KV.flush()
+
+
+cmdline = CommandLine()
+
+
+def describe_arguments(func):
+    """
+    Analyze a function's signature and return a data structure suitable for
+    passing in as arguments to an argparse parser's add_argument() method."""
+
+    argspec = inspect.getargspec(func)
+    # we should probably raise an exception somewhere if func includes **kwargs
+    if argspec.defaults:
+        positional_args = argspec.args[:-len(argspec.defaults)]
+        keyword_names = argspec.args[-len(argspec.defaults):]
+        for arg, default in zip(keyword_names, argspec.defaults):
+            yield ('--{}'.format(arg),), {'default': default}
+    else:
+        positional_args = argspec.args
+
+    for arg in positional_args:
+        yield (arg,), {}
+    if argspec.varargs:
+        yield (argspec.varargs,), {'nargs': '*'}

=== added file 'hooks/charmhelpers/cli/benchmark.py'
--- hooks/charmhelpers/cli/benchmark.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/cli/benchmark.py	2015-09-14 15:56:04 +0000
@@ -0,0 +1,36 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+from . import cmdline
+from charmhelpers.contrib.benchmark import Benchmark
+
+
[e-mail](command_name='benchmark-start')
+def start():
+    Benchmark.start()
+
+
[e-mail](command_name='benchmark-finish')
+def finish():
+    Benchmark.finish()
+
+
[e-mail]_builder('benchmark-composite', description="Set the benchmark composite score")
+def service(subparser):
+    subparser.add_argument("value", help="The composite score.")
+    subparser.add_argument("units", help="The units the composite score represents, i.e., 'reads/sec'.")
+    subparser.add_argument("direction", help="'asc' if a lower score is better, 'desc' if a higher score is better.")
+    return Benchmark.set_composite_score

=== added file 'hooks/charmhelpers/cli/commands.py'
--- hooks/charmhelpers/cli/commands.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/cli/commands.py	2015-09-14 15:56:04 +0000
@@ -0,0 +1,32 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+"""
+This module loads sub-modules into the python runtime so they can be
+discovered via the inspect module. In order to prevent flake8 from (rightfully)
+telling us these are unused modules, throw a ' # noqa' at the end of each import
+so that the warning is suppressed.
+"""
+
+from . import CommandLine  # noqa
+
+"""
+Import the sub-modules which have decorated subcommands to register with chlp.
+"""
+import host  # noqa
+import benchmark  # noqa
+import unitdata  # noqa
+from charmhelpers.core import hookenv  # noqa

=== added file 'hooks/charmhelpers/cli/host.py'
--- hooks/charmhelpers/cli/host.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/cli/host.py	2015-09-14 15:56:04 +0000
@@ -0,0 +1,31 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+from . import cmdline
+from charmhelpers.core import host
+
+
[e-mail]()
+def mounts():
+    "List mounts"
+    return host.mounts()
+
+
[e-mail]_builder('service', description="Control system services")
+def service(subparser):
+    subparser.add_argument("action", help="The action to perform (start, stop, etc...)")
+    subparser.add_argument("service_name", help="Name of the service to control")
+    return host.service

=== added file 'hooks/charmhelpers/cli/unitdata.py'
--- hooks/charmhelpers/cli/unitdata.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/cli/unitdata.py	2015-09-14 15:56:04 +0000
@@ -0,0 +1,39 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+from . import cmdline
+from charmhelpers.core import unitdata
+
+
[e-mail]_builder('unitdata', description="Store and retrieve data")
+def unitdata_cmd(subparser):
+    nested = subparser.add_subparsers()
+    get_cmd = nested.add_parser('get', help='Retrieve data')
+    get_cmd.add_argument('key', help='Key to retrieve the value of')
+    get_cmd.set_defaults(action='get', value=None)
+    set_cmd = nested.add_parser('set', help='Store data')
+    set_cmd.add_argument('key', help='Key to set')
+    set_cmd.add_argument('value', help='Value to store')
+    set_cmd.set_defaults(action='set')
+
+    def _unitdata_cmd(action, key, value):
+        if action == 'get':
+            return unitdata.kv().get(key)
+        elif action == 'set':
+            unitdata.kv().set(key, value)
+            unitdata.kv().flush()
+            return ''
+    return _unitdata_cmd

=== modified file 'hooks/charmhelpers/contrib/storage/linux/ceph.py'
--- hooks/charmhelpers/contrib/storage/linux/ceph.py	2015-09-10 09:29:50 +0000
+++ hooks/charmhelpers/contrib/storage/linux/ceph.py	2015-09-14 15:56:04 +0000
@@ -20,21 +20,30 @@
 # This file is sourced from lp:openstack-charm-helpers
 #
 # Authors:
-#  James Page <[e-mail]>
-#  Adam Gandelman <[e-mail]>
+# James Page <[e-mail]>
+# Adam Gandelman <[e-mail]>
 #
 
 import os
 import shutil
 import json
 import time
+<<<<<<< TREE
 import uuid
 
+=======
+from charmhelpers.fetch import (
+    apt_install,
+)
+>>>>>>> MERGE-SOURCE
 from subprocess import (
     check_call,
     check_output,
     CalledProcessError,
 )
+apt_install("python-enum")
+from enum import Enum
+
 from charmhelpers.core.hookenv import (
     local_unit,
     relation_get,
@@ -58,6 +67,8 @@
 from charmhelpers.fetch import (
     apt_install,
 )
+import math
+
 
 KEYRING = '/etc/ceph/ceph.client.{}.keyring'
 KEYFILE = '/etc/ceph/ceph.client.{}.key'
@@ -72,6 +83,40 @@
 """
 
 
+class PoolType(Enum):
+    Replicated = "replicated"
+    Erasure = "erasure"
+
+
+class Pool(object):
+    def __init__(self, name, pool_type):
+        self.PoolType = pool_type
+        self.name = name
+
+
+class ReplicatedPool(Pool):
+    def __init__(self, name, replicas=2):
+        super(ReplicatedPool, self).__init__(name=name, pool_type=PoolType.Replicated)
+        self.replicas = replicas
+
+
+class ErasurePool(Pool):
+    def __init__(self, name, erasure_code_profile="", data_chunks=2, coding_chunks=1, ):
+        super(ErasurePool, self).__init__(name=name, pool_type=PoolType.Erasure)
+        self.erasure_code_profile = erasure_code_profile
+        self.data_chunks = data_chunks
+        self.coding_chunks = coding_chunks
+
+
+class LocalRecoveryErasurePool(Pool):
+    def __init__(self, name, erasure_code_profile="", data_chunks=2, coding_chunks=1, local_chunks=1):
+        super(LocalRecoveryErasurePool, self).__init__(name=name, pool_type=PoolType.Erasure)
+        self.erasure_code_profile = erasure_code_profile
+        self.data_chunks = data_chunks
+        self.coding_chunks = coding_chunks
+        self.local_chunks = local_chunks
+
+
 def install():
     """Basic Ceph client installation."""
     ceph_dir = "/etc/ceph"
@@ -85,7 +130,7 @@
     """Check to see if a RADOS block device exists."""
     try:
         out = check_output(['rbd', 'list', '--id',
-                            service, '--pool', pool]).decode('UTF-8')
+                            service, '--pool', pool.name]).decode('UTF-8')
     except CalledProcessError:
         return False
 
@@ -95,11 +140,11 @@
 def create_rbd_image(service, pool, image, sizemb):
     """Create a new RADOS block device."""
     cmd = ['rbd', 'create', image, '--size', str(sizemb), '--id', service,
-           '--pool', pool]
+           '--pool', pool.name]
     check_call(cmd)
 
 
-def pool_exists(service, name):
+def pool_exists(service, pool):
     """Check to see if a RADOS pool already exists."""
     try:
         out = check_output(['rados', '--id', service,
@@ -107,6 +152,18 @@
     except CalledProcessError:
         return False
 
+    return pool.name in out
+
+
+def erasure_profile_exists(service, name):
+    """Check to see if an Erasure code profile already exists."""
+    try:
+        out = check_output(['ceph', '--id', service,
+                            'osd', 'erasure-code-profile', 'get',
+                            name]).decode('UTF-8')
+    except CalledProcessError:
+        return False
+
     return name in out
 
 
@@ -123,29 +180,77 @@
     return None
 
 
-def create_pool(service, name, replicas=3):
-    """Create a new RADOS pool."""
-    if pool_exists(service, name):
-        log("Ceph pool {} already exists, skipping creation".format(name),
+def create_erasure_profile(service, erasure_code_profile, data_chunks, coding_chunks):
+    cmd = ['ceph', '--id', service, 'osd', 'erasure-code-profile',
+           'set', erasure_code_profile, 'k=' + data_chunks, 'm=' + coding_chunks]
+    out = check_call(cmd)
+
+
+# NOTE: This is horribly slow
+def power_log(x):
+    return 2**(math.ceil(math.log(x, 2)))
+
+
+def create_pool(service, pool_class):
+    """Create a new RADOS pool.
+        pool=Pool object defined above
+    """
+    # Double check we have the right args here
+    assert isinstance(pool_class, Pool)
+    if pool_exists(service, pool_class.name):
+        log("Ceph pool {} already exists, skipping creation".format(pool_class.name),
             level=WARNING)
         return
 
     # Calculate the number of placement groups based
     # on upstream recommended best practices.
     osds = get_osds(service)
+    pgnum = 200 # NOTE(james-page) Default to 200 for older ceph versions which don't support OSD query
     if osds:
-        pgnum = (len(osds) * 100 // replicas)
-    else:
+        # TODO: What do i do about this?
+        if isinstance(pool_class, ErasurePool):
+            pgnum = (len(osds) * 100 // (pool_class.coding_chunks + pool_class.data_chunks))
+            pgnum = power_log(pgnum)  # Round to nearest power of 2 per Ceph docs
+        elif isinstance(pool_class, LocalRecoveryErasurePool):
+            pgnum = (len(osds) * 100 // (pool_class.coding_chunks + pool_class.data_chunks))
+            pgnum = power_log(pgnum)  # Round to nearest power of 2 per Ceph docs
+        elif isinstance(pool_class, ReplicatedPool):
+            pgnum = (len(osds) * 100 // pool_class.replicas)
+            pgnum = power_log(pgnum)  # Round to nearest power of 2 per Ceph docs
+    #else:
         # NOTE(james-page): Default to 200 for older ceph versions
         # which don't support OSD query from cli
-        pgnum = 200
-
-    cmd = ['ceph', '--id', service, 'osd', 'pool', 'create', name, str(pgnum)]
-    check_call(cmd)
-
-    cmd = ['ceph', '--id', service, 'osd', 'pool', 'set', name, 'size',
-           str(replicas)]
-    check_call(cmd)
+    #    pgnum = 200
+
+    cmd = ['ceph', '--id', service, 'osd', 'pool', 'create', pool_class.name, str(pgnum)]
+
+    if isinstance(pool_class, ErasurePool):
+        # Check to see if the profile exists.  If not we need to create it
+        log("Creating an erasure pool: " + str(pool_class))
+        if not erasure_profile_exists(service, pool_class.erasure_code_profile):
+            create_erasure_profile(service, pool_class.erasure_code_profile,
+                                   pool_class.data_chunks,
+                                   pool_class.coding_chunks)
+        cmd.append(pool_class.PoolType)
+        cmd.append(pool_class.erasure_code_profile)
+
+    elif isinstance(pool_class, LocalRecoveryErasurePool):
+        log("Creating a local recovery erasure pool: " + str(pool_class))
+        if not erasure_profile_exists(service, pool_class.erasure_code_profile):
+            create_erasure_profile(service, pool_class.erasure_code_profile,
+                                   pool_class.data_chunks,
+                                   pool_class.coding_chunks)
+        cmd.append(pool_class.PoolType)
+        cmd.append(pool_class.erasure_code_profile)
+
+    check_call(cmd)
+
+    if isinstance(pool_class, ReplicatedPool):
+        # This is the default
+        log("Created a replicated pool: " + str(pool_class))
+        cmd = ['ceph', '--id', service, 'osd', 'pool', 'set', pool_class.name, 'size',
+               str(pool_class.replicas)]
+        check_call(cmd)
 
 
 def delete_pool(service, name):
@@ -314,8 +419,7 @@
 
 
 def ensure_ceph_storage(service, pool, rbd_img, sizemb, mount_point,
-                        blk_device, fstype, system_services=[],
-                        replicas=3):
+                        blk_device, fstype, system_services=[]):
     """NOTE: This function must only be called from a single service unit for
     the same rbd_img otherwise data loss will occur.
 
@@ -328,10 +432,12 @@
     All services listed in system_services will be stopped prior to data
     migration and restarted when complete.
     """
+    log("ensure_ceph_storage")
+    assert isinstance(pool, Pool)
     # Ensure pool, RBD image, RBD mappings are in place.
     if not pool_exists(service, pool):
         log('Creating new pool {}.'.format(pool), level=INFO)
-        create_pool(service, pool, replicas=replicas)
+        create_pool(service, pool)
 
     if not rbd_exists(service, pool, rbd_img):
         log('Creating RBD image ({}).'.format(rbd_img), level=INFO)
@@ -347,8 +453,8 @@
     # the data is already in the rbd device and/or is mounted??
     # When it is mounted already, it will fail to make the fs
     # XXX: This is really sketchy!  Need to at least add an fstab entry
-    #      otherwise this hook will blow away existing data if its executed
-    #      after a reboot.
+    # otherwise this hook will blow away existing data if its executed
+    # after a reboot.
     if not filesystem_mounted(mount_point):
         make_filesystem(blk_device, fstype)
 
@@ -414,7 +520,12 @@
 
     The API is versioned and defaults to version 1.
     """
+<<<<<<< TREE
     def __init__(self, api_version=1, request_id=None):
+=======
+
+    def __init__(self, api_version=1):
+>>>>>>> MERGE-SOURCE
         self.api_version = api_version
         if request_id:
             self.request_id = request_id

=== added file 'hooks/charmhelpers/core/files.py'
--- hooks/charmhelpers/core/files.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/core/files.py	2015-09-14 15:56:04 +0000
@@ -0,0 +1,45 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+__author__ = 'Jorge Niedbalski <[e-mail]>'
+
+import os
+import subprocess
+
+
+def sed(filename, before, after, flags='g'):
+    """
+    Search and replaces the given pattern on filename.
+
+    :param filename: relative or absolute file path.
+    :param before: expression to be replaced (see 'man sed')
+    :param after: expression to replace with (see 'man sed')
+    :param flags: sed-compatible regex flags in example, to make
+    the  search and replace case insensitive, specify ``flags="i"``.
+    The ``g`` flag is always specified regardless, so you do not
+    need to remember to include it when overriding this parameter.
+    :returns: If the sed command exit code was zero then return,
+    otherwise raise CalledProcessError.
+    """
+    expression = r's/{0}/{1}/{2}'.format(before,
+                                         after, flags)
+
+    return subprocess.check_call(["sed", "-i", "-r", "-e",
+                                  expression,
+                                  os.path.expanduser(filename)])

=== renamed file 'hooks/charmhelpers/core/files.py' => 'hooks/charmhelpers/core/files.py.moved'
=== modified file 'hooks/charmhelpers/core/hookenv.py'
--- hooks/charmhelpers/core/hookenv.py	2015-09-03 09:42:00 +0000
+++ hooks/charmhelpers/core/hookenv.py	2015-09-14 15:56:04 +0000
@@ -34,6 +34,23 @@
 import tempfile
 from subprocess import CalledProcessError
 
+try:
+    from charmhelpers.cli import cmdline
+except ImportError as e:
+    # due to the anti-pattern of partially synching charmhelpers directly
+    # into charms, it's possible that charmhelpers.cli is not available;
+    # if that's the case, they don't really care about using the cli anyway,
+    # so mock it out
+    if str(e) == 'No module named cli':
+        class cmdline(object):
+            @classmethod
+            def subcommand(cls, *args, **kwargs):
+                def _wrap(func):
+                    return func
+                return _wrap
+    else:
+        raise
+
 import six
 if not six.PY3:
     from UserDict import UserDict
@@ -70,11 +87,18 @@
         try:
             return cache[key]
         except KeyError:
+<<<<<<< TREE
             pass  # Drop out of the exception handler scope.
         res = func(*args, **kwargs)
         cache[key] = res
         return res
     wrapper._wrapped = func
+=======
+            pass  # Drop out of the exception handler scope.
+        res = func(*args, **kwargs)
+        cache[key] = res
+        return res
+>>>>>>> MERGE-SOURCE
     return wrapper
 
 
@@ -174,19 +198,36 @@
     return os.environ.get('JUJU_RELATION', None)
 
 
-@cached
-def relation_id(relation_name=None, service_or_unit=None):
-    """The relation ID for the current or a specified relation"""
-    if not relation_name and not service_or_unit:
-        return os.environ.get('JUJU_RELATION_ID', None)
-    elif relation_name and service_or_unit:
-        service_name = service_or_unit.split('/')[0]
-        for relid in relation_ids(relation_name):
-            remote_service = remote_service_name(relid)
-            if remote_service == service_name:
-                return relid
-    else:
-        raise ValueError('Must specify neither or both of relation_name and service_or_unit')
+<<<<<<< TREE
+@cached
+def relation_id(relation_name=None, service_or_unit=None):
+    """The relation ID for the current or a specified relation"""
+    if not relation_name and not service_or_unit:
+        return os.environ.get('JUJU_RELATION_ID', None)
+    elif relation_name and service_or_unit:
+        service_name = service_or_unit.split('/')[0]
+        for relid in relation_ids(relation_name):
+            remote_service = remote_service_name(relid)
+            if remote_service == service_name:
+                return relid
+    else:
+        raise ValueError('Must specify neither or both of relation_name and service_or_unit')
+=======
[e-mail]()
+@cached
+def relation_id(relation_name=None, service_or_unit=None):
+    """The relation ID for the current or a specified relation"""
+    if not relation_name and not service_or_unit:
+        return os.environ.get('JUJU_RELATION_ID', None)
+    elif relation_name and service_or_unit:
+        service_name = service_or_unit.split('/')[0]
+        for relid in relation_ids(relation_name):
+            remote_service = remote_service_name(relid)
+            if remote_service == service_name:
+                return relid
+    else:
+        raise ValueError('Must specify neither or both of relation_name and service_or_unit')
+>>>>>>> MERGE-SOURCE
 
 
 def local_unit():
@@ -196,25 +237,47 @@
 
 def remote_unit():
     """The remote unit for the current relation hook"""
-    return os.environ.get('JUJU_REMOTE_UNIT', None)
-
-
+<<<<<<< TREE
+    return os.environ.get('JUJU_REMOTE_UNIT', None)
+
+
+=======
+    return os.environ.get('JUJU_REMOTE_UNIT', None)
+
+
[e-mail]()
+>>>>>>> MERGE-SOURCE
 def service_name():
     """The name service group this unit belongs to"""
     return local_unit().split('/')[0]
 
 
-@cached
-def remote_service_name(relid=None):
-    """The remote service name for a given relation-id (or the current relation)"""
-    if relid is None:
-        unit = remote_unit()
-    else:
-        units = related_units(relid)
-        unit = units[0] if units else None
-    return unit.split('/')[0] if unit else None
-
-
+<<<<<<< TREE
+@cached
+def remote_service_name(relid=None):
+    """The remote service name for a given relation-id (or the current relation)"""
+    if relid is None:
+        unit = remote_unit()
+    else:
+        units = related_units(relid)
+        unit = units[0] if units else None
+    return unit.split('/')[0] if unit else None
+
+
+=======
[e-mail]()
+@cached
+def remote_service_name(relid=None):
+    """The remote service name for a given relation-id (or the current relation)"""
+    if relid is None:
+        unit = remote_unit()
+    else:
+        units = related_units(relid)
+        unit = units[0] if units else None
+    return unit.split('/')[0] if unit else None
+
+
+>>>>>>> MERGE-SOURCE
 def hook_name():
     """The name of the currently executing hook"""
     return os.environ.get('JUJU_HOOK_NAME', os.path.basename(sys.argv[0]))
@@ -721,6 +784,7 @@
 
     The results set by action_set are preserved."""
     subprocess.check_call(['action-fail', message])
+<<<<<<< TREE
 
 
 def action_name():
@@ -896,3 +960,178 @@
     for callback, args, kwargs in reversed(_atexit):
         callback(*args, **kwargs)
     del _atexit[:]
+=======
+
+
+def action_name():
+    """Get the name of the currently executing action."""
+    return os.environ.get('JUJU_ACTION_NAME')
+
+
+def action_uuid():
+    """Get the UUID of the currently executing action."""
+    return os.environ.get('JUJU_ACTION_UUID')
+
+
+def action_tag():
+    """Get the tag for the currently executing action."""
+    return os.environ.get('JUJU_ACTION_TAG')
+
+
+def status_set(workload_state, message):
+    """Set the workload state with a message
+
+    Use status-set to set the workload state with a message which is visible
+    to the user via juju status. If the status-set command is not found then
+    assume this is juju < 1.23 and juju-log the message unstead.
+
+    workload_state -- valid juju workload state.
+    message        -- status update message
+    """
+    valid_states = ['maintenance', 'blocked', 'waiting', 'active']
+    if workload_state not in valid_states:
+        raise ValueError(
+            '{!r} is not a valid workload state'.format(workload_state)
+        )
+    cmd = ['status-set', workload_state, message]
+    try:
+        ret = subprocess.call(cmd)
+        if ret == 0:
+            return
+    except OSError as e:
+        if e.errno != errno.ENOENT:
+            raise
+    log_message = 'status-set failed: {} {}'.format(workload_state,
+                                                    message)
+    log(log_message, level='INFO')
+
+
+def status_get():
+    """Retrieve the previously set juju workload state
+
+    If the status-set command is not found then assume this is juju < 1.23 and
+    return 'unknown'
+    """
+    cmd = ['status-get']
+    try:
+        raw_status = subprocess.check_output(cmd, universal_newlines=True)
+        status = raw_status.rstrip()
+        return status
+    except OSError as e:
+        if e.errno == errno.ENOENT:
+            return 'unknown'
+        else:
+            raise
+
+
+def translate_exc(from_exc, to_exc):
+    def inner_translate_exc1(f):
+        def inner_translate_exc2(*args, **kwargs):
+            try:
+                return f(*args, **kwargs)
+            except from_exc:
+                raise to_exc
+
+        return inner_translate_exc2
+
+    return inner_translate_exc1
+
+
+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
+def is_leader():
+    """Does the current unit hold the juju leadership
+
+    Uses juju to determine whether the current unit is the leader of its peers
+    """
+    cmd = ['is-leader', '--format=json']
+    return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
+
+
+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
+def leader_get(attribute=None):
+    """Juju leader get value(s)"""
+    cmd = ['leader-get', '--format=json'] + [attribute or '-']
+    return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
+
+
+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
+def leader_set(settings=None, **kwargs):
+    """Juju leader set value(s)"""
+    # Don't log secrets.
+    # log("Juju leader-set '%s'" % (settings), level=DEBUG)
+    cmd = ['leader-set']
+    settings = settings or {}
+    settings.update(kwargs)
+    for k, v in settings.items():
+        if v is None:
+            cmd.append('{}='.format(k))
+        else:
+            cmd.append('{}={}'.format(k, v))
+    subprocess.check_call(cmd)
+
+
+@cached
+def juju_version():
+    """Full version string (eg. '1.23.3.1-trusty-amd64')"""
+    # Per https://bugs.launchpad.net/juju-core/+bug/1455368/comments/1
+    jujud = glob.glob('/var/lib/juju/tools/machine-*/jujud')[0]
+    return subprocess.check_output([jujud, 'version'],
+                                   universal_newlines=True).strip()
+
+
+@cached
+def has_juju_version(minimum_version):
+    """Return True if the Juju version is at least the provided version"""
+    return LooseVersion(juju_version()) >= LooseVersion(minimum_version)
+
+
+_atexit = []
+_atstart = []
+
+
+def atstart(callback, *args, **kwargs):
+    '''Schedule a callback to run before the main hook.
+
+    Callbacks are run in the order they were added.
+
+    This is useful for modules and classes to perform initialization
+    and inject behavior. In particular:
+
+        - Run common code before all of your hooks, such as logging
+          the hook name or interesting relation data.
+        - Defer object or module initialization that requires a hook
+          context until we know there actually is a hook context,
+          making testing easier.
+        - Rather than requiring charm authors to include boilerplate to
+          invoke your helper's behavior, have it run automatically if
+          your object is instantiated or module imported.
+
+    This is not at all useful after your hook framework as been launched.
+    '''
+    global _atstart
+    _atstart.append((callback, args, kwargs))
+
+
+def atexit(callback, *args, **kwargs):
+    '''Schedule a callback to run on successful hook completion.
+
+    Callbacks are run in the reverse order that they were added.'''
+    _atexit.append((callback, args, kwargs))
+
+
+def _run_atstart():
+    '''Hook frameworks must invoke this before running the main hook body.'''
+    global _atstart
+    for callback, args, kwargs in _atstart:
+        callback(*args, **kwargs)
+    del _atstart[:]
+
+
+def _run_atexit():
+    '''Hook frameworks must invoke this after the main hook body has
+    successfully completed. Do not invoke it if the hook fails.'''
+    global _atexit
+    for callback, args, kwargs in reversed(_atexit):
+        callback(*args, **kwargs)
+    del _atexit[:]
+>>>>>>> MERGE-SOURCE

=== modified file 'hooks/charmhelpers/core/host.py'
--- hooks/charmhelpers/core/host.py	2015-08-19 13:50:16 +0000
+++ hooks/charmhelpers/core/host.py	2015-09-14 15:56:04 +0000
@@ -63,36 +63,69 @@
     return service_result
 
 
-def service_pause(service_name, init_dir=None):
-    """Pause a system service.
-
-    Stop it, and prevent it from starting again at boot."""
-    if init_dir is None:
-        init_dir = "/etc/init"
-    stopped = service_stop(service_name)
-    # XXX: Support systemd too
-    override_path = os.path.join(
-        init_dir, '{}.override'.format(service_name))
-    with open(override_path, 'w') as fh:
-        fh.write("manual\n")
-    return stopped
-
-
-def service_resume(service_name, init_dir=None):
-    """Resume a system service.
-
-    Reenable starting again at boot. Start the service"""
-    # XXX: Support systemd too
-    if init_dir is None:
-        init_dir = "/etc/init"
-    override_path = os.path.join(
-        init_dir, '{}.override'.format(service_name))
-    if os.path.exists(override_path):
-        os.unlink(override_path)
-    started = service_start(service_name)
-    return started
-
-
+<<<<<<< TREE
+def service_pause(service_name, init_dir=None):
+    """Pause a system service.
+
+    Stop it, and prevent it from starting again at boot."""
+    if init_dir is None:
+        init_dir = "/etc/init"
+    stopped = service_stop(service_name)
+    # XXX: Support systemd too
+    override_path = os.path.join(
+        init_dir, '{}.override'.format(service_name))
+    with open(override_path, 'w') as fh:
+        fh.write("manual\n")
+    return stopped
+
+
+def service_resume(service_name, init_dir=None):
+    """Resume a system service.
+
+    Reenable starting again at boot. Start the service"""
+    # XXX: Support systemd too
+    if init_dir is None:
+        init_dir = "/etc/init"
+    override_path = os.path.join(
+        init_dir, '{}.override'.format(service_name))
+    if os.path.exists(override_path):
+        os.unlink(override_path)
+    started = service_start(service_name)
+    return started
+
+
+=======
+def service_pause(service_name, init_dir=None):
+    """Pause a system service.
+
+    Stop it, and prevent it from starting again at boot."""
+    if init_dir is None:
+        init_dir = "/etc/init"
+    stopped = service_stop(service_name)
+    # XXX: Support systemd too
+    override_path = os.path.join(
+        init_dir, '{}.conf.override'.format(service_name))
+    with open(override_path, 'w') as fh:
+        fh.write("manual\n")
+    return stopped
+
+
+def service_resume(service_name, init_dir=None):
+    """Resume a system service.
+
+    Reenable starting again at boot. Start the service"""
+    # XXX: Support systemd too
+    if init_dir is None:
+        init_dir = "/etc/init"
+    override_path = os.path.join(
+        init_dir, '{}.conf.override'.format(service_name))
+    if os.path.exists(override_path):
+        os.unlink(override_path)
+    started = service_start(service_name)
+    return started
+
+
+>>>>>>> MERGE-SOURCE
 def service(action, service_name):
     """Control a system service"""
     cmd = ['service', service_name, action]

=== modified file 'hooks/charmhelpers/core/services/helpers.py'
--- hooks/charmhelpers/core/services/helpers.py	2015-08-19 00:51:43 +0000
+++ hooks/charmhelpers/core/services/helpers.py	2015-09-14 15:56:04 +0000
@@ -241,14 +241,22 @@
     action.
 
     :param str source: The template source file, relative to
+<<<<<<< TREE
         `$CHARM_DIR/templates`
 
+=======
+        `$CHARM_DIR/templates`
+>>>>>>> MERGE-SOURCE
     :param str target: The target to write the rendered template to
     :param str owner: The owner of the rendered file
     :param str group: The group of the rendered file
     :param int perms: The permissions of the rendered file
+<<<<<<< TREE
     :param partial on_change_action: functools partial to be executed when
                                      rendered file changes
+=======
+
+>>>>>>> MERGE-SOURCE
     """
     def __init__(self, source, target,
                  owner='root', group='root', perms=0o444,

=== modified file 'hooks/charmhelpers/fetch/__init__.py'
=== modified file 'metadata.yaml'
--- metadata.yaml	2015-07-01 14:47:39 +0000
+++ metadata.yaml	2015-09-14 15:56:04 +0000
@@ -1,4 +1,4 @@
-name: ceph
+name: ceph-erasure
 summary: Highly scalable distributed storage
 maintainer: James Page <[e-mail]>
 description: |

=== modified file 'tests/basic_deployment.py'
--- tests/basic_deployment.py	2015-07-02 14:38:21 +0000
+++ tests/basic_deployment.py	2015-09-14 15:56:04 +0000
@@ -18,7 +18,7 @@
 class CephBasicDeployment(OpenStackAmuletDeployment):
     """Amulet tests on a basic ceph deployment."""
 
-    def __init__(self, series=None, openstack=None, source=None, stable=False):
+    def __init__(self, series=None, openstack=None, source=None, stable=True):
         """Deploy the entire test environment."""
         super(CephBasicDeployment, self).__init__(series, openstack, source,
                                                   stable)

=== modified file 'tests/charmhelpers/contrib/amulet/utils.py'
--- tests/charmhelpers/contrib/amulet/utils.py	2015-09-10 09:29:50 +0000
+++ tests/charmhelpers/contrib/amulet/utils.py	2015-09-14 15:56:04 +0000
@@ -14,15 +14,26 @@
 # You should have received a copy of the GNU Lesser General Public License
 # along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
 
+<<<<<<< TREE
+=======
+import amulet
+import ConfigParser
+import distro_info
+>>>>>>> MERGE-SOURCE
 import io
 import json
 import logging
 import os
 import re
+<<<<<<< TREE
 import socket
 import subprocess
+=======
+import six
+>>>>>>> MERGE-SOURCE
 import sys
 import time
+<<<<<<< TREE
 import uuid
 
 import amulet
@@ -33,6 +44,9 @@
     from urllib import parse as urlparse
 else:
     import urlparse
+=======
+import urlparse
+>>>>>>> MERGE-SOURCE
 
 
 class AmuletUtils(object):
@@ -107,6 +121,7 @@
         """Validate that lists of commands succeed on service units.  Can be
            used to verify system services are running on the corresponding
            service units.
+<<<<<<< TREE
 
         :param commands: dict with sentry keys and arbitrary command list vals
         :returns: None if successful, Failure string message otherwise
@@ -120,6 +135,21 @@
                       'validate_services_by_name instead of validate_services '
                       'due to init system differences.')
 
+=======
+
+        :param commands: dict with sentry keys and arbitrary command list vals
+        :returns: None if successful, Failure string message otherwise
+        """
+        self.log.debug('Checking status of system services...')
+
+        # /!\ DEPRECATION WARNING (beisner):
+        # New and existing tests should be rewritten to use
+        # validate_services_by_name() as it is aware of init systems.
+        self.log.warn('/!\\ DEPRECATION WARNING:  use '
+                      'validate_services_by_name instead of validate_services '
+                      'due to init system differences.')
+
+>>>>>>> MERGE-SOURCE
         for k, v in six.iteritems(commands):
             for cmd in v:
                 output, code = k.run(cmd)
@@ -130,6 +160,7 @@
                     return "command `{}` returned {}".format(cmd, str(code))
         return None
 
+<<<<<<< TREE
     def validate_services_by_name(self, sentry_services):
         """Validate system service status by service name, automatically
            detecting init system based on Ubuntu release codename.
@@ -169,6 +200,43 @@
                         cmd, output, str(code))
         return None
 
+=======
+    def validate_services_by_name(self, sentry_services):
+        """Validate system service status by service name, automatically
+           detecting init system based on Ubuntu release codename.
+
+        :param sentry_services: dict with sentry keys and svc list values
+        :returns: None if successful, Failure string message otherwise
+        """
+        self.log.debug('Checking status of system services...')
+
+        # Point at which systemd became a thing
+        systemd_switch = self.ubuntu_releases.index('vivid')
+
+        for sentry_unit, services_list in six.iteritems(sentry_services):
+            # Get lsb_release codename from unit
+            release, ret = self.get_ubuntu_release_from_sentry(sentry_unit)
+            if ret:
+                return ret
+
+            for service_name in services_list:
+                if (self.ubuntu_releases.index(release) >= systemd_switch or
+                        service_name == "rabbitmq-server"):
+                    # init is systemd
+                    cmd = 'sudo service {} status'.format(service_name)
+                elif self.ubuntu_releases.index(release) < systemd_switch:
+                    # init is upstart
+                    cmd = 'sudo status {}'.format(service_name)
+
+                output, code = sentry_unit.run(cmd)
+                self.log.debug('{} `{}` returned '
+                               '{}'.format(sentry_unit.info['unit_name'],
+                                           cmd, code))
+                if code != 0:
+                    return "command `{}` returned {}".format(cmd, str(code))
+        return None
+
+>>>>>>> MERGE-SOURCE
     def _get_config(self, unit, filename):
         """Get a ConfigParser object for parsing a unit's config file."""
         file_contents = unit.file_contents(filename)
@@ -470,6 +538,7 @@
 
     def endpoint_error(self, name, data):
         return 'unexpected endpoint data in {} - {}'.format(name, data)
+<<<<<<< TREE
 
     def get_ubuntu_releases(self):
         """Return a list of all Ubuntu releases in order of release."""
@@ -776,3 +845,123 @@
         output = _check_output(command, universal_newlines=True)
         data = json.loads(output)
         return data.get(u"status") == "completed"
+=======
+
+    def get_ubuntu_releases(self):
+        """Return a list of all Ubuntu releases in order of release."""
+        _d = distro_info.UbuntuDistroInfo()
+        _release_list = _d.all
+        self.log.debug('Ubuntu release list: {}'.format(_release_list))
+        return _release_list
+
+    def file_to_url(self, file_rel_path):
+        """Convert a relative file path to a file URL."""
+        _abs_path = os.path.abspath(file_rel_path)
+        return urlparse.urlparse(_abs_path, scheme='file').geturl()
+
+    def check_commands_on_units(self, commands, sentry_units):
+        """Check that all commands in a list exit zero on all
+        sentry units in a list.
+
+        :param commands:  list of bash commands
+        :param sentry_units:  list of sentry unit pointers
+        :returns: None if successful; Failure message otherwise
+        """
+        self.log.debug('Checking exit codes for {} commands on {} '
+                       'sentry units...'.format(len(commands),
+                                                len(sentry_units)))
+        for sentry_unit in sentry_units:
+            for cmd in commands:
+                output, code = sentry_unit.run(cmd)
+                if code == 0:
+                    self.log.debug('{} `{}` returned {} '
+                                   '(OK)'.format(sentry_unit.info['unit_name'],
+                                                 cmd, code))
+                else:
+                    return ('{} `{}` returned {} '
+                            '{}'.format(sentry_unit.info['unit_name'],
+                                        cmd, code, output))
+        return None
+
+    def get_process_id_list(self, sentry_unit, process_name):
+        """Get a list of process ID(s) from a single sentry juju unit
+        for a single process name.
+
+        :param sentry_unit: Pointer to amulet sentry instance (juju unit)
+        :param process_name: Process name
+        :returns: List of process IDs
+        """
+        cmd = 'pidof {}'.format(process_name)
+        output, code = sentry_unit.run(cmd)
+        if code != 0:
+            msg = ('{} `{}` returned {} '
+                   '{}'.format(sentry_unit.info['unit_name'],
+                               cmd, code, output))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+        return str(output).split()
+
+    def get_unit_process_ids(self, unit_processes):
+        """Construct a dict containing unit sentries, process names, and
+        process IDs."""
+        pid_dict = {}
+        for sentry_unit, process_list in unit_processes.iteritems():
+            pid_dict[sentry_unit] = {}
+            for process in process_list:
+                pids = self.get_process_id_list(sentry_unit, process)
+                pid_dict[sentry_unit].update({process: pids})
+        return pid_dict
+
+    def validate_unit_process_ids(self, expected, actual):
+        """Validate process id quantities for services on units."""
+        self.log.debug('Checking units for running processes...')
+        self.log.debug('Expected PIDs: {}'.format(expected))
+        self.log.debug('Actual PIDs: {}'.format(actual))
+
+        if len(actual) != len(expected):
+            return ('Unit count mismatch.  expected, actual: {}, '
+                    '{} '.format(len(expected), len(actual)))
+
+        for (e_sentry, e_proc_names) in expected.iteritems():
+            e_sentry_name = e_sentry.info['unit_name']
+            if e_sentry in actual.keys():
+                a_proc_names = actual[e_sentry]
+            else:
+                return ('Expected sentry ({}) not found in actual dict data.'
+                        '{}'.format(e_sentry_name, e_sentry))
+
+            if len(e_proc_names.keys()) != len(a_proc_names.keys()):
+                return ('Process name count mismatch.  expected, actual: {}, '
+                        '{}'.format(len(expected), len(actual)))
+
+            for (e_proc_name, e_pids_length), (a_proc_name, a_pids) in \
+                    zip(e_proc_names.items(), a_proc_names.items()):
+                if e_proc_name != a_proc_name:
+                    return ('Process name mismatch.  expected, actual: {}, '
+                            '{}'.format(e_proc_name, a_proc_name))
+
+                a_pids_length = len(a_pids)
+                if e_pids_length != a_pids_length:
+                    return ('PID count mismatch. {} ({}) expected, actual: '
+                            '{}, {} ({})'.format(e_sentry_name, e_proc_name,
+                                                 e_pids_length, a_pids_length,
+                                                 a_pids))
+                else:
+                    self.log.debug('PID check OK: {} {} {}: '
+                                   '{}'.format(e_sentry_name, e_proc_name,
+                                               e_pids_length, a_pids))
+        return None
+
+    def validate_list_of_identical_dicts(self, list_of_dicts):
+        """Check that all dicts within a list are identical."""
+        hashes = []
+        for _dict in list_of_dicts:
+            hashes.append(hash(frozenset(_dict.items())))
+
+        self.log.debug('Hashes: {}'.format(hashes))
+        if len(set(hashes)) == 1:
+            self.log.debug('Dicts within list are identical')
+        else:
+            return 'Dicts within list are not identical'
+
+        return None
+>>>>>>> MERGE-SOURCE

=== modified file 'tests/charmhelpers/contrib/openstack/amulet/deployment.py'
--- tests/charmhelpers/contrib/openstack/amulet/deployment.py	2015-09-10 09:29:50 +0000
+++ tests/charmhelpers/contrib/openstack/amulet/deployment.py	2015-09-14 15:56:04 +0000
@@ -94,9 +94,15 @@
         # Charms which should use the source config option
         use_source = ['mysql', 'mongodb', 'rabbitmq-server', 'ceph',
                       'ceph-osd', 'ceph-radosgw']
+<<<<<<< TREE
 
         # Charms which can not use openstack-origin, ie. many subordinates
         no_origin = ['cinder-ceph', 'hacluster', 'neutron-openvswitch', 'nrpe']
+=======
+        # Most OpenStack subordinate charms do not expose an origin option
+        # as that is controlled by the principle.
+        ignore = ['cinder-ceph', 'hacluster', 'neutron-openvswitch']
+>>>>>>> MERGE-SOURCE
 
         if self.openstack:
             for svc in services:

=== modified file 'tests/charmhelpers/contrib/openstack/amulet/utils.py'
--- tests/charmhelpers/contrib/openstack/amulet/utils.py	2015-09-10 09:29:50 +0000
+++ tests/charmhelpers/contrib/openstack/amulet/utils.py	2015-09-14 15:56:04 +0000
@@ -27,8 +27,12 @@
 import heatclient.v1.client as heat_client
 import keystoneclient.v2_0 as keystone_client
 import novaclient.v1_1.client as nova_client
+<<<<<<< TREE
 import pika
 import swiftclient
+=======
+import swiftclient
+>>>>>>> MERGE-SOURCE
 
 from charmhelpers.contrib.amulet.utils import (
     AmuletUtils
@@ -341,6 +345,7 @@
 
     def delete_instance(self, nova, instance):
         """Delete the specified instance."""
+<<<<<<< TREE
 
         # /!\ DEPRECATION WARNING
         self.log.warn('/!\\ DEPRECATION WARNING:  use '
@@ -961,3 +966,267 @@
         else:
             msg = 'No message retrieved.'
             amulet.raise_status(amulet.FAIL, msg)
+=======
+
+        # /!\ DEPRECATION WARNING
+        self.log.warn('/!\\ DEPRECATION WARNING:  use '
+                      'delete_resource instead of delete_instance.')
+        self.log.debug('Deleting instance ({})...'.format(instance))
+        return self.delete_resource(nova.servers, instance,
+                                    msg='nova instance')
+
+    def create_or_get_keypair(self, nova, keypair_name="testkey"):
+        """Create a new keypair, or return pointer if it already exists."""
+        try:
+            _keypair = nova.keypairs.get(keypair_name)
+            self.log.debug('Keypair ({}) already exists, '
+                           'using it.'.format(keypair_name))
+            return _keypair
+        except:
+            self.log.debug('Keypair ({}) does not exist, '
+                           'creating it.'.format(keypair_name))
+
+        _keypair = nova.keypairs.create(name=keypair_name)
+        return _keypair
+
+    def create_cinder_volume(self, cinder, vol_name="demo-vol", vol_size=1,
+                             img_id=None, src_vol_id=None, snap_id=None):
+        """Create cinder volume, optionally from a glance image, OR
+        optionally as a clone of an existing volume, OR optionally
+        from a snapshot.  Wait for the new volume status to reach
+        the expected status, validate and return a resource pointer.
+
+        :param vol_name: cinder volume display name
+        :param vol_size: size in gigabytes
+        :param img_id: optional glance image id
+        :param src_vol_id: optional source volume id to clone
+        :param snap_id: optional snapshot id to use
+        :returns: cinder volume pointer
+        """
+        # Handle parameter input and avoid impossible combinations
+        if img_id and not src_vol_id and not snap_id:
+            # Create volume from image
+            self.log.debug('Creating cinder volume from glance image...')
+            bootable = 'true'
+        elif src_vol_id and not img_id and not snap_id:
+            # Clone an existing volume
+            self.log.debug('Cloning cinder volume...')
+            bootable = cinder.volumes.get(src_vol_id).bootable
+        elif snap_id and not src_vol_id and not img_id:
+            # Create volume from snapshot
+            self.log.debug('Creating cinder volume from snapshot...')
+            snap = cinder.volume_snapshots.find(id=snap_id)
+            vol_size = snap.size
+            snap_vol_id = cinder.volume_snapshots.get(snap_id).volume_id
+            bootable = cinder.volumes.get(snap_vol_id).bootable
+        elif not img_id and not src_vol_id and not snap_id:
+            # Create volume
+            self.log.debug('Creating cinder volume...')
+            bootable = 'false'
+        else:
+            # Impossible combination of parameters
+            msg = ('Invalid method use - name:{} size:{} img_id:{} '
+                   'src_vol_id:{} snap_id:{}'.format(vol_name, vol_size,
+                                                     img_id, src_vol_id,
+                                                     snap_id))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        # Create new volume
+        try:
+            vol_new = cinder.volumes.create(display_name=vol_name,
+                                            imageRef=img_id,
+                                            size=vol_size,
+                                            source_volid=src_vol_id,
+                                            snapshot_id=snap_id)
+            vol_id = vol_new.id
+        except Exception as e:
+            msg = 'Failed to create volume: {}'.format(e)
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        # Wait for volume to reach available status
+        ret = self.resource_reaches_status(cinder.volumes, vol_id,
+                                           expected_stat="available",
+                                           msg="Volume status wait")
+        if not ret:
+            msg = 'Cinder volume failed to reach expected state.'
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        # Re-validate new volume
+        self.log.debug('Validating volume attributes...')
+        val_vol_name = cinder.volumes.get(vol_id).display_name
+        val_vol_boot = cinder.volumes.get(vol_id).bootable
+        val_vol_stat = cinder.volumes.get(vol_id).status
+        val_vol_size = cinder.volumes.get(vol_id).size
+        msg_attr = ('Volume attributes - name:{} id:{} stat:{} boot:'
+                    '{} size:{}'.format(val_vol_name, vol_id,
+                                        val_vol_stat, val_vol_boot,
+                                        val_vol_size))
+
+        if val_vol_boot == bootable and val_vol_stat == 'available' \
+                and val_vol_name == vol_name and val_vol_size == vol_size:
+            self.log.debug(msg_attr)
+        else:
+            msg = ('Volume validation failed, {}'.format(msg_attr))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        return vol_new
+
+    def delete_resource(self, resource, resource_id,
+                        msg="resource", max_wait=120):
+        """Delete one openstack resource, such as one instance, keypair,
+        image, volume, stack, etc., and confirm deletion within max wait time.
+
+        :param resource: pointer to os resource type, ex:glance_client.images
+        :param resource_id: unique name or id for the openstack resource
+        :param msg: text to identify purpose in logging
+        :param max_wait: maximum wait time in seconds
+        :returns: True if successful, otherwise False
+        """
+        self.log.debug('Deleting OpenStack resource '
+                       '{} ({})'.format(resource_id, msg))
+        num_before = len(list(resource.list()))
+        resource.delete(resource_id)
+
+        tries = 0
+        num_after = len(list(resource.list()))
+        while num_after != (num_before - 1) and tries < (max_wait / 4):
+            self.log.debug('{} delete check: '
+                           '{} [{}:{}] {}'.format(msg, tries,
+                                                  num_before,
+                                                  num_after,
+                                                  resource_id))
+            time.sleep(4)
+            num_after = len(list(resource.list()))
+            tries += 1
+
+        self.log.debug('{}:  expected, actual count = {}, '
+                       '{}'.format(msg, num_before - 1, num_after))
+
+        if num_after == (num_before - 1):
+            return True
+        else:
+            self.log.error('{} delete timed out'.format(msg))
+            return False
+
+    def resource_reaches_status(self, resource, resource_id,
+                                expected_stat='available',
+                                msg='resource', max_wait=120):
+        """Wait for an openstack resources status to reach an
+           expected status within a specified time.  Useful to confirm that
+           nova instances, cinder vols, snapshots, glance images, heat stacks
+           and other resources eventually reach the expected status.
+
+        :param resource: pointer to os resource type, ex: heat_client.stacks
+        :param resource_id: unique id for the openstack resource
+        :param expected_stat: status to expect resource to reach
+        :param msg: text to identify purpose in logging
+        :param max_wait: maximum wait time in seconds
+        :returns: True if successful, False if status is not reached
+        """
+
+        tries = 0
+        resource_stat = resource.get(resource_id).status
+        while resource_stat != expected_stat and tries < (max_wait / 4):
+            self.log.debug('{} status check: '
+                           '{} [{}:{}] {}'.format(msg, tries,
+                                                  resource_stat,
+                                                  expected_stat,
+                                                  resource_id))
+            time.sleep(4)
+            resource_stat = resource.get(resource_id).status
+            tries += 1
+
+        self.log.debug('{}:  expected, actual status = {}, '
+                       '{}'.format(msg, resource_stat, expected_stat))
+
+        if resource_stat == expected_stat:
+            return True
+        else:
+            self.log.debug('{} never reached expected status: '
+                           '{}'.format(resource_id, expected_stat))
+            return False
+
+    def get_ceph_osd_id_cmd(self, index):
+        """Produce a shell command that will return a ceph-osd id."""
+        return ("`initctl list | grep 'ceph-osd ' | "
+                "awk 'NR=={} {{ print $2 }}' | "
+                "grep -o '[0-9]*'`".format(index + 1))
+
+    def get_ceph_pools(self, sentry_unit):
+        """Return a dict of ceph pools from a single ceph unit, with
+        pool name as keys, pool id as vals."""
+        pools = {}
+        cmd = 'sudo ceph osd lspools'
+        output, code = sentry_unit.run(cmd)
+        if code != 0:
+            msg = ('{} `{}` returned {} '
+                   '{}'.format(sentry_unit.info['unit_name'],
+                               cmd, code, output))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        # Example output: 0 data,1 metadata,2 rbd,3 cinder,4 glance,
+        for pool in str(output).split(','):
+            pool_id_name = pool.split(' ')
+            if len(pool_id_name) == 2:
+                pool_id = pool_id_name[0]
+                pool_name = pool_id_name[1]
+                pools[pool_name] = int(pool_id)
+
+        self.log.debug('Pools on {}: {}'.format(sentry_unit.info['unit_name'],
+                                                pools))
+        return pools
+
+    def get_ceph_df(self, sentry_unit):
+        """Return dict of ceph df json output, including ceph pool state.
+
+        :param sentry_unit: Pointer to amulet sentry instance (juju unit)
+        :returns: Dict of ceph df output
+        """
+        cmd = 'sudo ceph df --format=json'
+        output, code = sentry_unit.run(cmd)
+        if code != 0:
+            msg = ('{} `{}` returned {} '
+                   '{}'.format(sentry_unit.info['unit_name'],
+                               cmd, code, output))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+        return json.loads(output)
+
+    def get_ceph_pool_sample(self, sentry_unit, pool_id=0):
+        """Take a sample of attributes of a ceph pool, returning ceph
+        pool name, object count and disk space used for the specified
+        pool ID number.
+
+        :param sentry_unit: Pointer to amulet sentry instance (juju unit)
+        :param pool_id: Ceph pool ID
+        :returns: List of pool name, object count, kb disk space used
+        """
+        df = self.get_ceph_df(sentry_unit)
+        pool_name = df['pools'][pool_id]['name']
+        obj_count = df['pools'][pool_id]['stats']['objects']
+        kb_used = df['pools'][pool_id]['stats']['kb_used']
+        self.log.debug('Ceph {} pool (ID {}): {} objects, '
+                       '{} kb used'.format(pool_name, pool_id,
+                                           obj_count, kb_used))
+        return pool_name, obj_count, kb_used
+
+    def validate_ceph_pool_samples(self, samples, sample_type="resource pool"):
+        """Validate ceph pool samples taken over time, such as pool
+        object counts or pool kb used, before adding, after adding, and
+        after deleting items which affect those pool attributes.  The
+        2nd element is expected to be greater than the 1st; 3rd is expected
+        to be less than the 2nd.
+
+        :param samples: List containing 3 data samples
+        :param sample_type: String for logging and usage context
+        :returns: None if successful, Failure message otherwise
+        """
+        original, created, deleted = range(3)
+        if samples[created] <= samples[original] or \
+                samples[deleted] >= samples[created]:
+            return ('Ceph {} samples ({}) '
+                    'unexpected.'.format(sample_type, samples))
+        else:
+            self.log.debug('Ceph {} samples (OK): '
+                           '{}'.format(sample_type, samples))
+            return None
+>>>>>>> MERGE-SOURCE

=== added file 'tests/tests.yaml'
--- tests/tests.yaml	1970-01-01 00:00:00 +0000
+++ tests/tests.yaml	2015-09-14 15:56:04 +0000
@@ -0,0 +1,18 @@
+bootstrap: true
+reset: true
+virtualenv: true
+makefile:
+    - lint
+    - test
+sources:
+    - ppa:juju/stable
+packages:
+    - amulet
+    - python-amulet
+    - python-cinderclient
+    - python-distro-info
+    - python-glanceclient
+    - python-heatclient
+    - python-keystoneclient
+    - python-novaclient
+    - python-swiftclient

=== renamed file 'tests/tests.yaml' => 'tests/tests.yaml.moved'