Merge lp:~xfactor973/charms/trusty/ceph-osd/sysctl-perf into lp:~openstack-charmers-archive/charms/trusty/ceph-osd/next

[James Page]

This branch adds some HDD and network adapter sysctl tuning. Lots of potential performance has been left on the table with our out of the box ceph setup. This aims to recapture some of that.

For the HDD portion: Linux has set very conservative settings when it comes to HDDs. While this is great for desktop responsiveness it isn't so kind to storage servers. I have exposed a configuration option if an administrator knows they have a RAID card with a lot of cache onboard. In a future patch set I will add some SSD tuning to take advantage of its unique characteristics.

Testing this has proven challenging. While it doesn't break anything on AWS that doesn't really prove anything either. If anyone has access to physical hardware to test this I would really appreciate it.

[Chris MacNaughton (chris.macnaughton)]

See comment on network interfaces. I'd also like to see some validation on varied hardware (SSD / HDD) that this doesn't break things either way.

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #18194 ceph-osd for xfactor973 mp284033
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/14678289/
Build: http://10.245.162.77:8080/job/charm_lint_check/18194/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #16953 ceph-osd for xfactor973 mp284033
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/16953/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #9068 ceph-osd for xfactor973 mp284033
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/14678328/
Build: http://10.245.162.77:8080/job/charm_amulet_test/9068/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #16956 ceph-osd for xfactor973 mp284033
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/16956/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #18198 ceph-osd for xfactor973 mp284033
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/18198/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #9071 ceph-osd for xfactor973 mp284033
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/14680501/
Build: http://10.245.162.77:8080/job/charm_amulet_test/9071/

[Chris Holcombe (xfactor973)]

@chris.macnaughton Ask and ye shall receive ;). This keys off of link speed now

[Chris Holcombe (xfactor973)]

Ignore the stupid line nits. That's from me using auto format in pycharms IDE

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #113 ceph-osd for xfactor973 mp284033
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/15002338/
Build: http://10.245.162.36:8080/job/charm_unit_test/113/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #130 ceph-osd for xfactor973 mp284033
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/15002365/
Build: http://10.245.162.36:8080/job/charm_lint_check/130/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #12 ceph-osd for xfactor973 mp284033
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/15002518/
Build: http://10.245.162.36:8080/job/charm_amulet_test/12/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #132 ceph-osd for xfactor973 mp284033
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/15002739/
Build: http://10.245.162.36:8080/job/charm_unit_test/132/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #140 ceph-osd for xfactor973 mp284033
    LINT OK: passed

Build: http://10.245.162.36:8080/job/charm_lint_check/140/

[Ryan Beisner (1chb1n)]

@xfactor973 The tests/* dir affects only Amulet test runs, and the 00-setup file is batch-maintained (may be overwritten).

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #27 ceph-osd for xfactor973 mp284033
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/15004320/
Build: http://10.245.162.36:8080/job/charm_amulet_test/27/

[James Page (james-page)]

Amulet test failure is genuine; install hook fails on 12.04:

2016-02-09 19:55:55 INFO install After this operation, 1467 kB of additional disk space will be used.
2016-02-09 19:55:55 INFO install Get:1 http://nova.clouds.archive.ubuntu.com/ubuntu/ precise/main python-setuptools all 0.6.24-1ubuntu1 [441 kB]
2016-02-09 19:55:55 INFO install Get:2 http://nova.clouds.archive.ubuntu.com/ubuntu/ precise/universe python-pip all 1.0-1build1 [95.1 kB]
2016-02-09 19:55:55 INFO install Fetched 536 kB in 0s (25.1 MB/s)
2016-02-09 19:55:55 INFO install Selecting previously unselected package python-setuptools.
2016-02-09 19:55:55 INFO install (Reading database ... 47935 files and directories currently installed.)
2016-02-09 19:55:55 INFO install Unpacking python-setuptools (from .../python-setuptools_0.6.24-1ubuntu1_all.deb) ...
2016-02-09 19:55:55 INFO install Selecting previously unselected package python-pip.
2016-02-09 19:55:55 INFO install Unpacking python-pip (from .../python-pip_1.0-1build1_all.deb) ...
2016-02-09 19:55:55 INFO install Processing triggers for man-db ...
2016-02-09 19:55:55 INFO install Setting up python-setuptools (0.6.24-1ubuntu1) ...
2016-02-09 19:55:55 INFO install Setting up python-pip (1.0-1build1) ...
2016-02-09 19:55:56 INFO install Package `python-enum34' is not installed and no info is available.
2016-02-09 19:55:56 INFO install Use dpkg --info (= dpkg-deb --info) to examine archive files,
2016-02-09 19:55:56 INFO install and dpkg --contents (= dpkg-deb --contents) to list their contents.
2016-02-09 19:55:56 INFO install Reading package lists...
2016-02-09 19:55:56 INFO install Building dependency tree...
2016-02-09 19:55:56 INFO install Reading state information...
2016-02-09 19:55:57 INFO install E: Unable to locate package python-enum34
2016-02-09 19:55:57 INFO install Traceback (most recent call last):
2016-02-09 19:55:57 INFO install File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/install.real", line 16, in <module>
2016-02-09 19:55:57 INFO install import ceph
2016-02-09 19:55:57 INFO install File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/ceph.py", line 9, in <module>
2016-02-09 19:55:57 INFO install from enum import Enum
2016-02-09 19:55:57 INFO install ImportError: No module named enum

review: Needs Fixing

[James Page (james-page)]

I'll have to defer to your knowledge on nic and block device tuning as I can't directly validate that - one question - are the changes make to interface and block device parameters persistent? I don't think that they are, so a server reboot will hit reset on tuned params.

review: Needs Information

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #315 ceph-osd for xfactor973 mp284033
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/15031007/
Build: http://10.245.162.36:8080/job/charm_unit_test/315/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #395 ceph-osd for xfactor973 mp284033
    LINT OK: passed

Build: http://10.245.162.36:8080/job/charm_lint_check/395/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #166 ceph-osd for xfactor973 mp284033
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/15031371/
Build: http://10.245.162.36:8080/job/charm_amulet_test/166/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #373 ceph-osd for xfactor973 mp284033
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/15032005/
Build: http://10.245.162.36:8080/job/charm_unit_test/373/

[Chris Holcombe (xfactor973)]

I dumped the enum dependency :)

On 02/11/2016 05:46 AM, James Page wrote:
> Review: Needs Fixing
>
> Amulet test failure is genuine; install hook fails on 12.04:
>
> 2016-02-09 19:55:55 INFO install After this operation, 1467 kB of additional disk space will be used.
> 2016-02-09 19:55:55 INFO install Get:1 http://nova.clouds.archive.ubuntu.com/ubuntu/ precise/main python-setuptools all 0.6.24-1ubuntu1 [441 kB]
> 2016-02-09 19:55:55 INFO install Get:2 http://nova.clouds.archive.ubuntu.com/ubuntu/ precise/universe python-pip all 1.0-1build1 [95.1 kB]
> 2016-02-09 19:55:55 INFO install Fetched 536 kB in 0s (25.1 MB/s)
> 2016-02-09 19:55:55 INFO install Selecting previously unselected package python-setuptools.
> 2016-02-09 19:55:55 INFO install (Reading database ... 47935 files and directories currently installed.)
> 2016-02-09 19:55:55 INFO install Unpacking python-setuptools (from .../python-setuptools_0.6.24-1ubuntu1_all.deb) ...
> 2016-02-09 19:55:55 INFO install Selecting previously unselected package python-pip.
> 2016-02-09 19:55:55 INFO install Unpacking python-pip (from .../python-pip_1.0-1build1_all.deb) ...
> 2016-02-09 19:55:55 INFO install Processing triggers for man-db ...
> 2016-02-09 19:55:55 INFO install Setting up python-setuptools (0.6.24-1ubuntu1) ...
> 2016-02-09 19:55:55 INFO install Setting up python-pip (1.0-1build1) ...
> 2016-02-09 19:55:56 INFO install Package `python-enum34' is not installed and no info is available.
> 2016-02-09 19:55:56 INFO install Use dpkg --info (= dpkg-deb --info) to examine archive files,
> 2016-02-09 19:55:56 INFO install and dpkg --contents (= dpkg-deb --contents) to list their contents.
> 2016-02-09 19:55:56 INFO install Reading package lists...
> 2016-02-09 19:55:56 INFO install Building dependency tree...
> 2016-02-09 19:55:56 INFO install Reading state information...
> 2016-02-09 19:55:57 INFO install E: Unable to locate package python-enum34
> 2016-02-09 19:55:57 INFO install Traceback (most recent call last):
> 2016-02-09 19:55:57 INFO install File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/install.real", line 16, in <module>
> 2016-02-09 19:55:57 INFO install import ceph
> 2016-02-09 19:55:57 INFO install File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/ceph.py", line 9, in <module>
> 2016-02-09 19:55:57 INFO install from enum import Enum
> 2016-02-09 19:55:57 INFO install ImportError: No module named enum
>
>

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #669 ceph-osd for xfactor973 mp284033
    LINT OK: passed

Build: http://10.245.162.36:8080/job/charm_lint_check/669/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #579 ceph-osd for xfactor973 mp284033
    UNIT OK: passed

Build: http://10.245.162.36:8080/job/charm_unit_test/579/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #277 ceph-osd for xfactor973 mp284033
    AMULET OK: passed

Build: http://10.245.162.36:8080/job/charm_amulet_test/277/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #730 ceph-osd-next for james-page mp286492
    UNIT OK: passed

Build: http://10.245.162.36:8080/job/charm_unit_test/730/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #829 ceph-osd-next for james-page mp286492
    LINT OK: passed

Build: http://10.245.162.36:8080/job/charm_lint_check/829/

[James Page (james-page)]

Some suggested refactoring to use charmhelpers.core.sysctl to manage the nic settings.

Also I'm not sure that max_sectors_kb is reboot persistent yet

review: Needs Fixing

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #329 ceph-osd-next for james-page mp286492
    AMULET OK: passed

Build: http://10.245.162.36:8080/job/charm_amulet_test/329/

Unmerged revisions

56. By Chris Holcombe on 2016-02-16

Removed enum dependency

55. By Chris Holcombe on 2016-02-12

Pull in the tox files from next

54. By Chris Holcombe on 2016-02-11

Persist the changes for reboots. Also catch IOError in all the posix operations

53. By Chris Holcombe on 2016-02-09

Address the lint issues and try to solve the python-enum34 package missing with the unit tests

52. By Chris Holcombe on 2016-02-03

Change over to using network link speed as the key to sysctl tuning instead of the driver name. This is a more generic approach because most 10Gb links need roughly the same set of sysctls

51. By Chris Holcombe on 2016-01-27

Lint spacing

50. By Chris Holcombe on 2016-01-26

Add a status notification to the network adapter tuning section

49. By Chris Holcombe on 2016-01-26

Added possible network adapter sysctl tuning

48. By Chris Holcombe on 2016-01-26

Add a config flag and a function to tune the read and write sectors for each osd block device

47. By Chris Holcombe on 2016-01-26

Sysctl and performance tweaks for ceph

=== modified file '.bzrignore'
--- .bzrignore	2015-10-30 02:23:36 +0000
+++ .bzrignore	2016-02-18 11:56:12 +0000
@@ -3,3 +3,6 @@
 .tox
 .testrepository
 bin
+.idea
+.tox
+.testrepository

=== added file '.coveragerc'
--- .coveragerc	1970-01-01 00:00:00 +0000
+++ .coveragerc	2016-02-18 11:56:12 +0000
@@ -0,0 +1,7 @@
+[report]
+# Regexes for lines to exclude from consideration
+exclude_lines =
+    if __name__ == .__main__.:
+include=
+    hooks/hooks.py
+    hooks/ceph*.py

=== renamed file '.coveragerc' => '.coveragerc.moved'
=== added file '.testr.conf'
--- .testr.conf	1970-01-01 00:00:00 +0000
+++ .testr.conf	2016-02-18 11:56:12 +0000
@@ -0,0 +1,8 @@
+[DEFAULT]
+test_command=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
+             OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
+             OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-60} \
+             ${PYTHON:-python} -m subunit.run discover -t ./ ./unit_tests $LISTOPT $IDOPTION
+
+test_id_option=--load-list $IDFILE
+test_list_option=--list

=== renamed file '.testr.conf' => '.testr.conf.moved'
=== modified file 'Makefile'
--- Makefile	2016-01-08 21:44:50 +0000
+++ Makefile	2016-02-18 11:56:12 +0000
@@ -13,7 +13,10 @@
 
 functional_test:
 	@echo Starting Amulet tests...
+<<<<<<< TREE
 	@tests/setup/00-setup
+=======
+>>>>>>> MERGE-SOURCE
 	@juju test -v -p AMULET_HTTP_PROXY,AMULET_OS_VIP --timeout 2700
 
 bin/charm_helpers_sync.py:

=== modified file 'charm-helpers-hooks.yaml'
--- charm-helpers-hooks.yaml	2015-08-03 14:53:05 +0000
+++ charm-helpers-hooks.yaml	2016-02-18 11:56:12 +0000
@@ -1,4 +1,4 @@
-branch: lp:charm-helpers
+branch: lp:~openstack-charmers/charm-helpers/stable
 destination: hooks/charmhelpers
 include:
     - core

=== modified file 'charm-helpers-tests.yaml'
--- charm-helpers-tests.yaml	2014-09-27 02:28:51 +0000
+++ charm-helpers-tests.yaml	2016-02-18 11:56:12 +0000
@@ -1,4 +1,4 @@
-branch: lp:charm-helpers
+branch: lp:~openstack-charmers/charm-helpers/stable
 destination: tests/charmhelpers
 include:
     - contrib.amulet

=== modified file 'config.yaml'
--- config.yaml	2016-01-13 12:48:57 +0000
+++ config.yaml	2016-02-18 11:56:12 +0000
@@ -128,7 +128,8 @@
   sysctl:
     type: string
     default: '{ kernel.pid_max : 2097152, vm.max_map_count : 524288,
-                kernel.threads-max: 2097152 }'
+                kernel.threads-max: 2097152, vm.vfs_cache_pressure: 1,
+                vm.swappiness: 1}'
     description: |
       YAML-formatted associative array of sysctl key/value pairs to be set
       persistently. By default we set pid_max, max_map_count and 
@@ -152,3 +153,11 @@
     description: |
       A comma-separated list of nagios servicegroups.
       If left empty, the nagios_context will be used as the servicegroup
+  max_sectors_kb:
+    default: 1048576
+    type: int
+    description: |
+      This parameter will adjust every block device in your server to allow
+      greater IO operation sizes.  If you have a RAID card with cache on it
+      consider tuning this much higher than the 1MB default.  1MB is a safe
+      default for spinning HDDs that don't have much cache.

=== modified file 'hooks/ceph.py'
--- hooks/ceph.py	2016-01-29 07:31:13 +0000
+++ hooks/ceph.py	2016-02-18 11:56:12 +0000
@@ -1,4 +1,4 @@
-
+# coding=utf-8
 #
 # Copyright 2012 Canonical Ltd.
 #
@@ -6,13 +6,16 @@
 #  James Page <[e-mail]>
 #  Paul Collins <[e-mail]>
 #
-
 import json
 import subprocess
 import time
 import os
+<<<<<<< TREE
 import re
 import sys
+=======
+
+>>>>>>> MERGE-SOURCE
 from charmhelpers.core.host import (
     mkdir,
     chownr,
@@ -22,6 +25,7 @@
 )
 from charmhelpers.core.hookenv import (
     log,
+<<<<<<< TREE
     ERROR,
     WARNING,
     DEBUG,
@@ -31,6 +35,11 @@
 from charmhelpers.fetch import (
     apt_cache
 )
+=======
+    ERROR, WARNING,
+    status_set,
+    config)
+>>>>>>> MERGE-SOURCE
 from charmhelpers.contrib.storage.linux.utils import (
     zap_disk,
     is_block_device,
@@ -46,6 +55,115 @@
 
 PACKAGES = ['ceph', 'gdisk', 'ntp', 'btrfs-tools', 'python-ceph', 'xfsprogs']
 
+LinkSpeed = {
+    "BASE_10": 10,
+    "BASE_100": 100,
+    "BASE_1000": 1000,
+    "GBASE_10": 10000,
+    "GBASE_40": 40000,
+    "GBASE_100": 100000,
+    "UNKNOWN": None
+}
+
+# Mapping of adapter name: sysctl settings
+NETWORK_ADAPTER_SYSCTLS = {
+    # 1Gb
+    LinkSpeed["BASE_1000"]: [
+        # Fill me in
+    ],
+    # 10Gb
+    LinkSpeed["GBASE_10"]: [
+        'net.core.rmem_default=524287',
+        'net.core.wmem_default=524287',
+        'net.core.rmem_max=524287',
+        'net.core.wmem_max=524287',
+        'net.core.optmem_max=524287',
+        'net.core.netdev_max_backlog=300000',
+        'net.ipv4.tcp_rmem=”10000000 10000000 10000000”',
+        'net.ipv4.tcp_wmem=”10000000 10000000 10000000”',
+        'net.ipv4.tcp_mem=”10000000 10000000 10000000”'
+    ],
+    # Mellanox 10/40Gb
+    LinkSpeed["GBASE_40"]: [
+        'net.ipv4.tcp_timestamps=0',
+        'net.ipv4.tcp_sack=1',
+        'net.core.netdev_max_backlog=250000',
+        'net.core.rmem_max=4194304',
+        'net.core.wmem_max=4194304',
+        'net.core.rmem_default=4194304',
+        'net.core.wmem_default=4194304',
+        'net.core.optmem_max=4194304',
+        'net.ipv4.tcp_rmem=”4096 87380 4194304”',
+        'net.ipv4.tcp_wmem=”4096 65536 4194304”',
+        'net.ipv4.tcp_low_latency=1',
+        'net.ipv4.tcp_adv_win_scale=1'
+    ]
+}
+
+
+def tune_nic(network_interface):
+    """
+    This will set optimal sysctls for the particular network adapter.
+    :param network_interface: The network adapter name.
+    """
+    speed = get_link_speed(network_interface)
+    if speed in NETWORK_ADAPTER_SYSCTLS:
+        status_set('maintenance', 'Tuning device {}'.format(network_interface))
+        for setting in NETWORK_ADAPTER_SYSCTLS[speed]:
+            try:
+                log("Setting network adapter sysctl {}".format(setting))
+                # Set this now for the running system
+                subprocess.check_call(["sysctl", "--write", setting])
+            except subprocess.CalledProcessError as e:
+                log("sysctl setting {} failed with error {}".format(
+                    setting,
+                    e.message))
+
+            # Persist this for future reboots
+            try:
+                with open("/etc/sysctl.conf", "a") as sysconf:
+                    sysconf.write(setting)
+            except IOError as e:
+                log("Write to /etc/sysctl.conf failed with error {}".format(
+                    e.message))
+    else:
+        log("No settings found for network adapter: {}".format(
+            network_interface))
+
+
+def get_link_speed(network_interface):
+    """
+    This will find the link speed for a given network device.  Returns None
+    if an error occurs.
+    :param network_interface:
+    :return: LinkSpeed
+    """
+    speed_path = os.path.join(os.sep, 'sys', 'class', 'net',
+                              network_interface, 'speed')
+    # I'm not sure where else we'd check if this doesn't exist
+    if not os.path.exists(speed_path):
+        return LinkSpeed["UNKNOWN"]
+
+    try:
+        with open(speed_path, 'r') as sysfs:
+            speed = sysfs.readlines()
+
+            # Did we actually read anything?
+            if not speed:
+                return LinkSpeed["UNKNOWN"]
+
+            # Try to find a sysctl match for this particular speed
+            for name, speed in LinkSpeed.items():
+                if speed == int(speed[0].strip()):
+                    return speed
+            # Default to UNKNOWN if we can't find a match
+            return LinkSpeed["UNKNOWN"]
+    except IOError as e:
+        log("Unable to open {path} because of error: {error}".format(
+            path=speed_path,
+            error=e.message))
+        return LinkSpeed["UNKNOWN"]
+
 
 def ceph_user():
     if get_version() > 1:
@@ -165,6 +283,7 @@
         # Ignore any errors for this call
         subprocess.call(cmd)
 
+
 DISK_FORMATS = [
     'xfs',
     'ext4',
@@ -182,10 +301,17 @@
         info = subprocess.check_output(['sgdisk', '-i', '1', dev])
         info = info.split("\n")  # IGNORE:E1103
         for line in info:
+<<<<<<< TREE
             for ptype in CEPH_PARTITIONS:
                 sig = 'Partition GUID code: {}'.format(ptype)
                 if line.startswith(sig):
                     return True
+=======
+            if line.startswith(
+                    'Partition GUID code: 4FBD7E29-9D25-41B8-AFD0-062C0CEFF05D'
+            ):
+                return True
+>>>>>>> MERGE-SOURCE
     except subprocess.CalledProcessError:
         pass
     return False
@@ -218,7 +344,7 @@
 
 
 def wait_for_bootstrap():
-    while (not is_bootstrapped()):
+    while not is_bootstrapped():
         time.sleep(3)
 
 
@@ -401,6 +527,7 @@
             pass
 
 
+<<<<<<< TREE
 def maybe_zap_journal(journal_dev):
     if (is_osd_disk(journal_dev)):
         log('Looks like {} is already an OSD data'
@@ -427,6 +554,94 @@
     return least[1]
 
 
+=======
+def tune_dev(block_dev):
+    """
+    Try to make some intelligent decisions with HDD tuning.  Future work will
+    include optimizing SSDs.
+    This function will change the read ahead sectors and the max write
+    sectors for each block device.
+    :param block_dev: A block device name: Example: /dev/sda
+    """
+    log('Tuning device {}'.format(block_dev))
+    status_set('maintenance', 'Tuning device {}'.format(block_dev))
+    try:
+        # Set the read ahead sectors to 256
+        log('Setting read ahead to 256 for device {}'.format(block_dev))
+        subprocess.check_output(['hdparm', '-a256', block_dev])
+    except subprocess.CalledProcessError as e:
+        log('hdparm failed with error: {}'.format(e.message))
+    # make sure that /sys/.../max_sectors_kb matches max_hw_sectors_kb or at
+    # least 1MB for spinning disks
+    # If the box has a RAID card with cache this could go much bigger.
+    dev_name = None
+    path_parts = os.path.split(block_dev)
+    if len(path_parts) == 2:
+        dev_name = path_parts[1]
+    else:
+        log('Unable to determine the block device name from path: {}'.format(
+            block_dev))
+        # Play it safe and bail
+        return
+    max_sectors_kb = 0
+    max_hw_sectors_kb = 0
+    max_sectors_kb_path = os.path.join('sys', 'block', dev_name, 'queue',
+                                       'max_sectors_kb')
+    max_hw_sectors_kb_path = os.path.join('sys', 'block', dev_name, 'queue',
+                                          'max_hw_sectors_kb')
+
+    # Read in what Linux has set by default
+    if os.path.exists(max_sectors_kb_path):
+        try:
+            with open(max_sectors_kb_path, 'r') as f:
+                max_sectors_kb = f.read().strip()
+        except IOError as e:
+            log('Failed to read max_sectors_kb to {}. Error: {}'.format(
+                max_sectors_kb_path, e.message))
+            # Bail.
+            return
+
+    # Read in what the hardware supports
+    if os.path.exists(max_hw_sectors_kb_path):
+        try:
+            with open(max_hw_sectors_kb_path, 'r') as f:
+                max_hw_sectors_kb = f.read().strip()
+        except IOError as e:
+            log('Failed to read max_hw_sectors_kb to {}. Error: {}'.format(
+                max_hw_sectors_kb_path, e.message))
+
+    # OK we have a situation where the hardware supports more than Linux is
+    # currently requesting
+    if max_sectors_kb < max_hw_sectors_kb:
+        config_max_sectors_kb = config('max_sectors_kb')
+        if config_max_sectors_kb < max_hw_sectors_kb:
+            # Set the max_sectors_kb to the config.yaml value if it is less
+            # than the max_hw_sectors_kb
+            log('Setting max_sectors_kb for device {} to {}'.format(
+                max_sectors_kb_path, config_max_sectors_kb))
+            try:
+                with open(max_sectors_kb_path, 'w') as f:
+                    f.write(config_max_sectors_kb)
+            except IOError as e:
+                log('Failed to write max_sectors_kb to {}. Error: {}'.format(
+                    max_sectors_kb_path, e.message))
+        else:
+            # Set to the max_hw_sectors_kb
+            log('Setting max_sectors_kb for device {} to {}'.format(
+                max_sectors_kb_path, max_hw_sectors_kb))
+            try:
+                with open(max_sectors_kb_path, 'w') as f:
+                    f.write(max_hw_sectors_kb)
+            except IOError as e:
+                log('Failed to write max_sectors_kb to {}. Error: {}'.format(
+                    max_sectors_kb_path, e.message))
+    else:
+        log('max_sectors_kb match max_hw_sectors_kb.  No change needed for '
+            'device: {}'.format(block_dev))
+    status_set('maintenance', 'Finished tuning device {}'.format(block_dev))
+
+
+>>>>>>> MERGE-SOURCE
 def osdize(dev, osd_format, osd_journal, reformat_osd=False,
            ignore_errors=False):
     if dev.startswith('/dev'):
@@ -445,17 +660,27 @@
         log('Path {} is not a block device - bailing'.format(dev))
         return
 
+<<<<<<< TREE
     if (is_osd_disk(dev) and not reformat_osd):
         log('Looks like {} is already an'
             ' OSD data or journal, skipping.'.format(dev))
+=======
+    if is_osd_disk(dev) and not reformat_osd:
+        log('Looks like {} is already an OSD, skipping.'.format(dev))
+>>>>>>> MERGE-SOURCE
         return
 
     if is_device_mounted(dev):
         log('Looks like {} is in use, skipping.'.format(dev))
         return
 
+<<<<<<< TREE
     status_set('maintenance', 'Initializing device {}'.format(dev))
     cmd = ['ceph-disk', 'prepare']
+=======
+    status_set('maintenance', 'Initializing device {}'.format(dev))
+    cmd = ['ceph-disk-prepare']
+>>>>>>> MERGE-SOURCE
     # Later versions of ceph support more options
     if cmp_pkgrevno('ceph', '0.48.3') >= 0:
         if osd_format:
@@ -509,6 +734,7 @@
 
 def filesystem_mounted(fs):
     return subprocess.call(['grep', '-wqs', fs, '/proc/mounts']) == 0
+<<<<<<< TREE
 
 
 def get_running_osds():
@@ -519,3 +745,15 @@
         return result.split()
     except subprocess.CalledProcessError:
         return []
+=======
+
+
+def get_running_osds():
+    """Returns a list of the pids of the current running OSD daemons"""
+    cmd = ['pgrep', 'ceph-osd']
+    try:
+        result = subprocess.check_output(cmd)
+        return result.split()
+    except subprocess.CalledProcessError:
+        return []
+>>>>>>> MERGE-SOURCE

=== added file 'hooks/ceph_hooks.py'
--- hooks/ceph_hooks.py	1970-01-01 00:00:00 +0000
+++ hooks/ceph_hooks.py	2016-02-18 11:56:12 +0000
@@ -0,0 +1,279 @@
+#!/usr/bin/python
+
+#
+# Copyright 2012 Canonical Ltd.
+#
+# Authors:
+#  James Page <[e-mail]>
+#
+
+import glob
+import os
+import shutil
+import sys
+import netifaces
+
+import ceph
+from charmhelpers.core.hookenv import (
+    log,
+    ERROR,
+    config,
+    relation_ids,
+    related_units,
+    relation_get,
+    Hooks,
+    UnregisteredHookError,
+    service_name,
+    status_set,
+)
+from charmhelpers.core.host import (
+    umount,
+    mkdir,
+    cmp_pkgrevno
+)
+from charmhelpers.fetch import (
+    add_source,
+    apt_install,
+    apt_update,
+    filter_installed_packages,
+)
+from charmhelpers.core.sysctl import create as create_sysctl
+from utils import (
+    render_template,
+    get_host_ip,
+    assert_charm_supports_ipv6
+)
+from charmhelpers.contrib.openstack.alternatives import install_alternative
+from charmhelpers.contrib.network.ip import (
+    get_ipv6_addr,
+    format_ipv6_addr
+)
+from charmhelpers.contrib.charmsupport import nrpe
+
+hooks = Hooks()
+
+
+def tune_network_adapters():
+    interfaces = netifaces.interfaces()
+    for interface in interfaces:
+        if interface == "lo":
+            # Skip the loopback
+            continue
+        log("Looking up {} for possible sysctl tuning.".format(interface))
+        ceph.tune_nic(interface)
+
+
+def install_upstart_scripts():
+    # Only install upstart configurations for older versions
+    if cmp_pkgrevno('ceph', "0.55.1") < 0:
+        for x in glob.glob('files/upstart/*.conf'):
+            shutil.copy(x, '/etc/init/')
+
+
[e-mail]('install.real')
+def install():
+    add_source(config('source'), config('key'))
+    apt_update(fatal=True)
+    apt_install(packages=ceph.PACKAGES, fatal=True)
+    install_upstart_scripts()
+    tune_network_adapters()
+
+
+def emit_cephconf():
+    mon_hosts = get_mon_hosts()
+    log('Monitor hosts are ' + repr(mon_hosts))
+
+    cephcontext = {
+        'auth_supported': get_auth(),
+        'mon_hosts': ' '.join(mon_hosts),
+        'fsid': get_fsid(),
+        'old_auth': cmp_pkgrevno('ceph', "0.51") < 0,
+        'osd_journal_size': config('osd-journal-size'),
+        'use_syslog': str(config('use-syslog')).lower(),
+        'ceph_public_network': config('ceph-public-network'),
+        'ceph_cluster_network': config('ceph-cluster-network'),
+    }
+
+    if config('prefer-ipv6'):
+        dynamic_ipv6_address = get_ipv6_addr()[0]
+        if not config('ceph-public-network'):
+            cephcontext['public_addr'] = dynamic_ipv6_address
+        if not config('ceph-cluster-network'):
+            cephcontext['cluster_addr'] = dynamic_ipv6_address
+
+    # Install ceph.conf as an alternative to support
+    # co-existence with other charms that write this file
+    charm_ceph_conf = "/var/lib/charm/{}/ceph.conf".format(service_name())
+    mkdir(os.path.dirname(charm_ceph_conf))
+    with open(charm_ceph_conf, 'w') as cephconf:
+        cephconf.write(render_template('ceph.conf', cephcontext))
+    install_alternative('ceph.conf', '/etc/ceph/ceph.conf',
+                        charm_ceph_conf, 90)
+
+
+JOURNAL_ZAPPED = '/var/lib/ceph/journal_zapped'
+
+
[e-mail]('config-changed')
+def config_changed():
+    # Pre-flight checks
+    if config('osd-format') not in ceph.DISK_FORMATS:
+        log('Invalid OSD disk format configuration specified', level=ERROR)
+        sys.exit(1)
+
+    if config('prefer-ipv6'):
+        assert_charm_supports_ipv6()
+
+    sysctl_dict = config('sysctl')
+    if sysctl_dict:
+        create_sysctl(sysctl_dict, '/etc/sysctl.d/50-ceph-osd-charm.conf')
+
+    e_mountpoint = config('ephemeral-unmount')
+    if e_mountpoint and ceph.filesystem_mounted(e_mountpoint):
+        umount(e_mountpoint)
+
+    osd_journal = config('osd-journal')
+    if osd_journal and not os.path.exists(JOURNAL_ZAPPED) and \
+            os.path.exists(osd_journal):
+        ceph.zap_disk(osd_journal)
+        with open(JOURNAL_ZAPPED, 'w') as zapped:
+            zapped.write('DONE')
+
+    if ceph.is_bootstrapped():
+        log('ceph bootstrapped, rescanning disks')
+        emit_cephconf()
+        for dev in get_devices():
+            ceph.osdize(dev, config('osd-format'),
+                        config('osd-journal'), config('osd-reformat'),
+                        config('ignore-device-errors'))
+            # Make it fast!
+            ceph.tune_dev(dev)
+        ceph.start_osds(get_devices())
+
+
+def get_mon_hosts():
+    hosts = []
+    for relid in relation_ids('mon'):
+        for unit in related_units(relid):
+            addr = relation_get(
+                'ceph-public-address',
+                unit,
+                relid) or get_host_ip(relation_get('private-address',
+                                                   unit,
+                                                   relid))
+
+            if addr:
+                hosts.append('{}:6789'.format(format_ipv6_addr(addr) or addr))
+
+    hosts.sort()
+    return hosts
+
+
+def get_fsid():
+    return get_conf('fsid')
+
+
+def get_auth():
+    return get_conf('auth')
+
+
+def get_conf(name):
+    for relid in relation_ids('mon'):
+        for unit in related_units(relid):
+            conf = relation_get(name,
+                                unit, relid)
+            if conf:
+                return conf
+    return None
+
+
+def reformat_osd():
+    if config('osd-reformat'):
+        return True
+    else:
+        return False
+
+
+def get_devices():
+    if config('osd-devices'):
+        return config('osd-devices').split(' ')
+    else:
+        return []
+
+
[e-mail]('mon-relation-changed',
+            'mon-relation-departed')
+def mon_relation():
+    bootstrap_key = relation_get('osd_bootstrap_key')
+    if get_fsid() and get_auth() and bootstrap_key:
+        log('mon has provided conf- scanning disks')
+        emit_cephconf()
+        ceph.import_osd_bootstrap_key(bootstrap_key)
+        for dev in get_devices():
+            ceph.osdize(dev, config('osd-format'),
+                        config('osd-journal'), config('osd-reformat'),
+                        config('ignore-device-errors'))
+            # Make it fast!
+            ceph.tune_dev(dev)
+        ceph.start_osds(get_devices())
+    else:
+        log('mon cluster has not yet provided conf')
+
+
[e-mail]('upgrade-charm')
+def upgrade_charm():
+    if get_fsid() and get_auth():
+        emit_cephconf()
+    install_upstart_scripts()
+    apt_install(packages=filter_installed_packages(ceph.PACKAGES),
+                fatal=True)
+
+
[e-mail]('nrpe-external-master-relation-joined',
+            'nrpe-external-master-relation-changed')
+def update_nrpe_config():
+    # python-dbus is used by check_upstart_job
+    apt_install('python-dbus')
+    hostname = nrpe.get_nagios_hostname()
+    current_unit = nrpe.get_nagios_unit_name()
+    nrpe_setup = nrpe.NRPE(hostname=hostname)
+    nrpe_setup.add_check(
+        shortname='ceph-osd',
+        description='process check {%s}' % current_unit,
+        check_cmd=('/bin/cat /var/lib/ceph/osd/ceph-*/whoami |'
+                   'xargs -I@ status ceph-osd id=@ && exit 0 || exit 2')
+    )
+    nrpe_setup.write()
+
+
+def assess_status():
+    """Assess status of current unit"""
+    # Check for mon relation
+    if len(relation_ids('mon')) < 1:
+        status_set('blocked', 'Missing relation: monitor')
+        return
+
+    # Check for monitors with presented addresses
+    # Check for bootstrap key presentation
+    monitors = get_mon_hosts()
+    if len(monitors) < 1 or not get_conf('osd_bootstrap_key'):
+        status_set('waiting', 'Incomplete relation: monitor')
+        return
+
+    # Check for OSD device creation parity i.e. at least some devices
+    # must have been presented and used for this charm to be operational
+    running_osds = ceph.get_running_osds()
+    if not running_osds:
+        status_set('blocked',
+                   'No block devices detected using current configuration')
+    else:
+        status_set('active',
+                   'Unit is ready ({} OSD)'.format(len(running_osds)))
+
+
+if __name__ == '__main__':
+    try:
+        hooks.execute(sys.argv)
+    except UnregisteredHookError as e:
+        log('Unknown hook {} - skipping.'.format(e))
+    assess_status()

=== renamed file 'hooks/ceph_hooks.py' => 'hooks/ceph_hooks.py.moved'
=== added directory 'hooks/charmhelpers/cli'
=== renamed directory 'hooks/charmhelpers/cli' => 'hooks/charmhelpers/cli.moved'
=== added file 'hooks/charmhelpers/cli/__init__.py'
--- hooks/charmhelpers/cli/__init__.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/cli/__init__.py	2016-02-18 11:56:12 +0000
@@ -0,0 +1,191 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+import inspect
+import argparse
+import sys
+
+from six.moves import zip
+
+from charmhelpers.core import unitdata
+
+
+class OutputFormatter(object):
+    def __init__(self, outfile=sys.stdout):
+        self.formats = (
+            "raw",
+            "json",
+            "py",
+            "yaml",
+            "csv",
+            "tab",
+        )
+        self.outfile = outfile
+
+    def add_arguments(self, argument_parser):
+        formatgroup = argument_parser.add_mutually_exclusive_group()
+        choices = self.supported_formats
+        formatgroup.add_argument("--format", metavar='FMT',
+                                 help="Select output format for returned data, "
+                                      "where FMT is one of: {}".format(choices),
+                                 choices=choices, default='raw')
+        for fmt in self.formats:
+            fmtfunc = getattr(self, fmt)
+            formatgroup.add_argument("-{}".format(fmt[0]),
+                                     "--{}".format(fmt), action='store_const',
+                                     const=fmt, dest='format',
+                                     help=fmtfunc.__doc__)
+
+    @property
+    def supported_formats(self):
+        return self.formats
+
+    def raw(self, output):
+        """Output data as raw string (default)"""
+        if isinstance(output, (list, tuple)):
+            output = '\n'.join(map(str, output))
+        self.outfile.write(str(output))
+
+    def py(self, output):
+        """Output data as a nicely-formatted python data structure"""
+        import pprint
+        pprint.pprint(output, stream=self.outfile)
+
+    def json(self, output):
+        """Output data in JSON format"""
+        import json
+        json.dump(output, self.outfile)
+
+    def yaml(self, output):
+        """Output data in YAML format"""
+        import yaml
+        yaml.safe_dump(output, self.outfile)
+
+    def csv(self, output):
+        """Output data as excel-compatible CSV"""
+        import csv
+        csvwriter = csv.writer(self.outfile)
+        csvwriter.writerows(output)
+
+    def tab(self, output):
+        """Output data in excel-compatible tab-delimited format"""
+        import csv
+        csvwriter = csv.writer(self.outfile, dialect=csv.excel_tab)
+        csvwriter.writerows(output)
+
+    def format_output(self, output, fmt='raw'):
+        fmtfunc = getattr(self, fmt)
+        fmtfunc(output)
+
+
+class CommandLine(object):
+    argument_parser = None
+    subparsers = None
+    formatter = None
+    exit_code = 0
+
+    def __init__(self):
+        if not self.argument_parser:
+            self.argument_parser = argparse.ArgumentParser(description='Perform common charm tasks')
+        if not self.formatter:
+            self.formatter = OutputFormatter()
+            self.formatter.add_arguments(self.argument_parser)
+        if not self.subparsers:
+            self.subparsers = self.argument_parser.add_subparsers(help='Commands')
+
+    def subcommand(self, command_name=None):
+        """
+        Decorate a function as a subcommand. Use its arguments as the
+        command-line arguments"""
+        def wrapper(decorated):
+            cmd_name = command_name or decorated.__name__
+            subparser = self.subparsers.add_parser(cmd_name,
+                                                   description=decorated.__doc__)
+            for args, kwargs in describe_arguments(decorated):
+                subparser.add_argument(*args, **kwargs)
+            subparser.set_defaults(func=decorated)
+            return decorated
+        return wrapper
+
+    def test_command(self, decorated):
+        """
+        Subcommand is a boolean test function, so bool return values should be
+        converted to a 0/1 exit code.
+        """
+        decorated._cli_test_command = True
+        return decorated
+
+    def no_output(self, decorated):
+        """
+        Subcommand is not expected to return a value, so don't print a spurious None.
+        """
+        decorated._cli_no_output = True
+        return decorated
+
+    def subcommand_builder(self, command_name, description=None):
+        """
+        Decorate a function that builds a subcommand. Builders should accept a
+        single argument (the subparser instance) and return the function to be
+        run as the command."""
+        def wrapper(decorated):
+            subparser = self.subparsers.add_parser(command_name)
+            func = decorated(subparser)
+            subparser.set_defaults(func=func)
+            subparser.description = description or func.__doc__
+        return wrapper
+
+    def run(self):
+        "Run cli, processing arguments and executing subcommands."
+        arguments = self.argument_parser.parse_args()
+        argspec = inspect.getargspec(arguments.func)
+        vargs = []
+        for arg in argspec.args:
+            vargs.append(getattr(arguments, arg))
+        if argspec.varargs:
+            vargs.extend(getattr(arguments, argspec.varargs))
+        output = arguments.func(*vargs)
+        if getattr(arguments.func, '_cli_test_command', False):
+            self.exit_code = 0 if output else 1
+            output = ''
+        if getattr(arguments.func, '_cli_no_output', False):
+            output = ''
+        self.formatter.format_output(output, arguments.format)
+        if unitdata._KV:
+            unitdata._KV.flush()
+
+
+cmdline = CommandLine()
+
+
+def describe_arguments(func):
+    """
+    Analyze a function's signature and return a data structure suitable for
+    passing in as arguments to an argparse parser's add_argument() method."""
+
+    argspec = inspect.getargspec(func)
+    # we should probably raise an exception somewhere if func includes **kwargs
+    if argspec.defaults:
+        positional_args = argspec.args[:-len(argspec.defaults)]
+        keyword_names = argspec.args[-len(argspec.defaults):]
+        for arg, default in zip(keyword_names, argspec.defaults):
+            yield ('--{}'.format(arg),), {'default': default}
+    else:
+        positional_args = argspec.args
+
+    for arg in positional_args:
+        yield (arg,), {}
+    if argspec.varargs:
+        yield (argspec.varargs,), {'nargs': '*'}

=== added file 'hooks/charmhelpers/cli/benchmark.py'
--- hooks/charmhelpers/cli/benchmark.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/cli/benchmark.py	2016-02-18 11:56:12 +0000
@@ -0,0 +1,36 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+from . import cmdline
+from charmhelpers.contrib.benchmark import Benchmark
+
+
[e-mail](command_name='benchmark-start')
+def start():
+    Benchmark.start()
+
+
[e-mail](command_name='benchmark-finish')
+def finish():
+    Benchmark.finish()
+
+
[e-mail]_builder('benchmark-composite', description="Set the benchmark composite score")
+def service(subparser):
+    subparser.add_argument("value", help="The composite score.")
+    subparser.add_argument("units", help="The units the composite score represents, i.e., 'reads/sec'.")
+    subparser.add_argument("direction", help="'asc' if a lower score is better, 'desc' if a higher score is better.")
+    return Benchmark.set_composite_score

=== added file 'hooks/charmhelpers/cli/commands.py'
--- hooks/charmhelpers/cli/commands.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/cli/commands.py	2016-02-18 11:56:12 +0000
@@ -0,0 +1,32 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+"""
+This module loads sub-modules into the python runtime so they can be
+discovered via the inspect module. In order to prevent flake8 from (rightfully)
+telling us these are unused modules, throw a ' # noqa' at the end of each import
+so that the warning is suppressed.
+"""
+
+from . import CommandLine  # noqa
+
+"""
+Import the sub-modules which have decorated subcommands to register with chlp.
+"""
+from . import host  # noqa
+from . import benchmark  # noqa
+from . import unitdata  # noqa
+from . import hookenv  # noqa

=== added file 'hooks/charmhelpers/cli/hookenv.py'
--- hooks/charmhelpers/cli/hookenv.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/cli/hookenv.py	2016-02-18 11:56:12 +0000
@@ -0,0 +1,23 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+from . import cmdline
+from charmhelpers.core import hookenv
+
+
+cmdline.subcommand('relation-id')(hookenv.relation_id._wrapped)
+cmdline.subcommand('service-name')(hookenv.service_name)
+cmdline.subcommand('remote-service-name')(hookenv.remote_service_name._wrapped)

=== added file 'hooks/charmhelpers/cli/host.py'
--- hooks/charmhelpers/cli/host.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/cli/host.py	2016-02-18 11:56:12 +0000
@@ -0,0 +1,31 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+from . import cmdline
+from charmhelpers.core import host
+
+
[e-mail]()
+def mounts():
+    "List mounts"
+    return host.mounts()
+
+
[e-mail]_builder('service', description="Control system services")
+def service(subparser):
+    subparser.add_argument("action", help="The action to perform (start, stop, etc...)")
+    subparser.add_argument("service_name", help="Name of the service to control")
+    return host.service

=== added file 'hooks/charmhelpers/cli/unitdata.py'
--- hooks/charmhelpers/cli/unitdata.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/cli/unitdata.py	2016-02-18 11:56:12 +0000
@@ -0,0 +1,39 @@
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+from . import cmdline
+from charmhelpers.core import unitdata
+
+
[e-mail]_builder('unitdata', description="Store and retrieve data")
+def unitdata_cmd(subparser):
+    nested = subparser.add_subparsers()
+    get_cmd = nested.add_parser('get', help='Retrieve data')
+    get_cmd.add_argument('key', help='Key to retrieve the value of')
+    get_cmd.set_defaults(action='get', value=None)
+    set_cmd = nested.add_parser('set', help='Store data')
+    set_cmd.add_argument('key', help='Key to set')
+    set_cmd.add_argument('value', help='Value to store')
+    set_cmd.set_defaults(action='set')
+
+    def _unitdata_cmd(action, key, value):
+        if action == 'get':
+            return unitdata.kv().get(key)
+        elif action == 'set':
+            unitdata.kv().set(key, value)
+            unitdata.kv().flush()
+            return ''
+    return _unitdata_cmd

=== modified file 'hooks/charmhelpers/contrib/charmsupport/nrpe.py'
=== modified file 'hooks/charmhelpers/contrib/network/ip.py'
=== added file 'hooks/charmhelpers/core/files.py'
--- hooks/charmhelpers/core/files.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/core/files.py	2016-02-18 11:56:12 +0000
@@ -0,0 +1,45 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+__author__ = 'Jorge Niedbalski <[e-mail]>'
+
+import os
+import subprocess
+
+
+def sed(filename, before, after, flags='g'):
+    """
+    Search and replaces the given pattern on filename.
+
+    :param filename: relative or absolute file path.
+    :param before: expression to be replaced (see 'man sed')
+    :param after: expression to replace with (see 'man sed')
+    :param flags: sed-compatible regex flags in example, to make
+    the  search and replace case insensitive, specify ``flags="i"``.
+    The ``g`` flag is always specified regardless, so you do not
+    need to remember to include it when overriding this parameter.
+    :returns: If the sed command exit code was zero then return,
+    otherwise raise CalledProcessError.
+    """
+    expression = r's/{0}/{1}/{2}'.format(before,
+                                         after, flags)
+
+    return subprocess.check_call(["sed", "-i", "-r", "-e",
+                                  expression,
+                                  os.path.expanduser(filename)])

=== renamed file 'hooks/charmhelpers/core/files.py' => 'hooks/charmhelpers/core/files.py.moved'
=== modified file 'hooks/charmhelpers/core/hookenv.py'
--- hooks/charmhelpers/core/hookenv.py	2016-01-22 22:29:34 +0000
+++ hooks/charmhelpers/core/hookenv.py	2016-02-18 11:56:12 +0000
@@ -491,6 +491,7 @@
 
 
 @cached
+<<<<<<< TREE
 def peer_relation_id():
     '''Get the peers relation id if a peers relation has been joined, else None.'''
     md = metadata()
@@ -561,6 +562,65 @@
 
 
 @cached
+=======
+def relation_to_interface(relation_name):
+    """
+    Given the name of a relation, return the interface that relation uses.
+
+    :returns: The interface name, or ``None``.
+    """
+    return relation_to_role_and_interface(relation_name)[1]
+
+
+@cached
+def relation_to_role_and_interface(relation_name):
+    """
+    Given the name of a relation, return the role and the name of the interface
+    that relation uses (where role is one of ``provides``, ``requires``, or ``peer``).
+
+    :returns: A tuple containing ``(role, interface)``, or ``(None, None)``.
+    """
+    _metadata = metadata()
+    for role in ('provides', 'requires', 'peer'):
+        interface = _metadata.get(role, {}).get(relation_name, {}).get('interface')
+        if interface:
+            return role, interface
+    return None, None
+
+
+@cached
+def role_and_interface_to_relations(role, interface_name):
+    """
+    Given a role and interface name, return a list of relation names for the
+    current charm that use that interface under that role (where role is one
+    of ``provides``, ``requires``, or ``peer``).
+
+    :returns: A list of relation names.
+    """
+    _metadata = metadata()
+    results = []
+    for relation_name, relation in _metadata.get(role, {}).items():
+        if relation['interface'] == interface_name:
+            results.append(relation_name)
+    return results
+
+
+@cached
+def interface_to_relations(interface_name):
+    """
+    Given an interface, return a list of relation names for the current
+    charm that use that interface.
+
+    :returns: A list of relation names.
+    """
+    results = []
+    for role in ('provides', 'requires', 'peer'):
+        results.extend(role_and_interface_to_relations(role, interface_name))
+    return results
+
+
+@cached
+>>>>>>> MERGE-SOURCE
 def charm_name():
     """Get the name of the current charm as is specified on metadata.yaml"""
     return metadata().get('name')
@@ -766,6 +826,7 @@
 
     The results set by action_set are preserved."""
     subprocess.check_call(['action-fail', message])
+<<<<<<< TREE
 
 
 def action_name():
@@ -976,3 +1037,180 @@
     for callback, args, kwargs in reversed(_atexit):
         callback(*args, **kwargs)
     del _atexit[:]
+=======
+
+
+def action_name():
+    """Get the name of the currently executing action."""
+    return os.environ.get('JUJU_ACTION_NAME')
+
+
+def action_uuid():
+    """Get the UUID of the currently executing action."""
+    return os.environ.get('JUJU_ACTION_UUID')
+
+
+def action_tag():
+    """Get the tag for the currently executing action."""
+    return os.environ.get('JUJU_ACTION_TAG')
+
+
+def status_set(workload_state, message):
+    """Set the workload state with a message
+
+    Use status-set to set the workload state with a message which is visible
+    to the user via juju status. If the status-set command is not found then
+    assume this is juju < 1.23 and juju-log the message unstead.
+
+    workload_state -- valid juju workload state.
+    message        -- status update message
+    """
+    valid_states = ['maintenance', 'blocked', 'waiting', 'active']
+    if workload_state not in valid_states:
+        raise ValueError(
+            '{!r} is not a valid workload state'.format(workload_state)
+        )
+    cmd = ['status-set', workload_state, message]
+    try:
+        ret = subprocess.call(cmd)
+        if ret == 0:
+            return
+    except OSError as e:
+        if e.errno != errno.ENOENT:
+            raise
+    log_message = 'status-set failed: {} {}'.format(workload_state,
+                                                    message)
+    log(log_message, level='INFO')
+
+
+def status_get():
+    """Retrieve the previously set juju workload state and message
+
+    If the status-get command is not found then assume this is juju < 1.23 and
+    return 'unknown', ""
+
+    """
+    cmd = ['status-get', "--format=json", "--include-data"]
+    try:
+        raw_status = subprocess.check_output(cmd)
+    except OSError as e:
+        if e.errno == errno.ENOENT:
+            return ('unknown', "")
+        else:
+            raise
+    else:
+        status = json.loads(raw_status.decode("UTF-8"))
+        return (status["status"], status["message"])
+
+
+def translate_exc(from_exc, to_exc):
+    def inner_translate_exc1(f):
+        def inner_translate_exc2(*args, **kwargs):
+            try:
+                return f(*args, **kwargs)
+            except from_exc:
+                raise to_exc
+
+        return inner_translate_exc2
+
+    return inner_translate_exc1
+
+
+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
+def is_leader():
+    """Does the current unit hold the juju leadership
+
+    Uses juju to determine whether the current unit is the leader of its peers
+    """
+    cmd = ['is-leader', '--format=json']
+    return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
+
+
+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
+def leader_get(attribute=None):
+    """Juju leader get value(s)"""
+    cmd = ['leader-get', '--format=json'] + [attribute or '-']
+    return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
+
+
+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
+def leader_set(settings=None, **kwargs):
+    """Juju leader set value(s)"""
+    # Don't log secrets.
+    # log("Juju leader-set '%s'" % (settings), level=DEBUG)
+    cmd = ['leader-set']
+    settings = settings or {}
+    settings.update(kwargs)
+    for k, v in settings.items():
+        if v is None:
+            cmd.append('{}='.format(k))
+        else:
+            cmd.append('{}={}'.format(k, v))
+    subprocess.check_call(cmd)
+
+
+@cached
+def juju_version():
+    """Full version string (eg. '1.23.3.1-trusty-amd64')"""
+    # Per https://bugs.launchpad.net/juju-core/+bug/1455368/comments/1
+    jujud = glob.glob('/var/lib/juju/tools/machine-*/jujud')[0]
+    return subprocess.check_output([jujud, 'version'],
+                                   universal_newlines=True).strip()
+
+
+@cached
+def has_juju_version(minimum_version):
+    """Return True if the Juju version is at least the provided version"""
+    return LooseVersion(juju_version()) >= LooseVersion(minimum_version)
+
+
+_atexit = []
+_atstart = []
+
+
+def atstart(callback, *args, **kwargs):
+    '''Schedule a callback to run before the main hook.
+
+    Callbacks are run in the order they were added.
+
+    This is useful for modules and classes to perform initialization
+    and inject behavior. In particular:
+
+        - Run common code before all of your hooks, such as logging
+          the hook name or interesting relation data.
+        - Defer object or module initialization that requires a hook
+          context until we know there actually is a hook context,
+          making testing easier.
+        - Rather than requiring charm authors to include boilerplate to
+          invoke your helper's behavior, have it run automatically if
+          your object is instantiated or module imported.
+
+    This is not at all useful after your hook framework as been launched.
+    '''
+    global _atstart
+    _atstart.append((callback, args, kwargs))
+
+
+def atexit(callback, *args, **kwargs):
+    '''Schedule a callback to run on successful hook completion.
+
+    Callbacks are run in the reverse order that they were added.'''
+    _atexit.append((callback, args, kwargs))
+
+
+def _run_atstart():
+    '''Hook frameworks must invoke this before running the main hook body.'''
+    global _atstart
+    for callback, args, kwargs in _atstart:
+        callback(*args, **kwargs)
+    del _atstart[:]
+
+
+def _run_atexit():
+    '''Hook frameworks must invoke this after the main hook body has
+    successfully completed. Do not invoke it if the hook fails.'''
+    global _atexit
+    for callback, args, kwargs in reversed(_atexit):
+        callback(*args, **kwargs)
+    del _atexit[:]
+>>>>>>> MERGE-SOURCE

=== modified file 'hooks/charmhelpers/core/host.py'
--- hooks/charmhelpers/core/host.py	2016-01-22 22:29:34 +0000
+++ hooks/charmhelpers/core/host.py	2016-02-18 11:56:12 +0000
@@ -63,6 +63,7 @@
     return service_result
 
 
+<<<<<<< TREE
 def service_pause(service_name, init_dir="/etc/init", initd_dir="/etc/init.d"):
     """Pause a system service.
 
@@ -117,6 +118,38 @@
     return started
 
 
+=======
+def service_pause(service_name, init_dir=None):
+    """Pause a system service.
+
+    Stop it, and prevent it from starting again at boot."""
+    if init_dir is None:
+        init_dir = "/etc/init"
+    stopped = service_stop(service_name)
+    # XXX: Support systemd too
+    override_path = os.path.join(
+        init_dir, '{}.override'.format(service_name))
+    with open(override_path, 'w') as fh:
+        fh.write("manual\n")
+    return stopped
+
+
+def service_resume(service_name, init_dir=None):
+    """Resume a system service.
+
+    Reenable starting again at boot. Start the service"""
+    # XXX: Support systemd too
+    if init_dir is None:
+        init_dir = "/etc/init"
+    override_path = os.path.join(
+        init_dir, '{}.override'.format(service_name))
+    if os.path.exists(override_path):
+        os.unlink(override_path)
+    started = service_start(service_name)
+    return started
+
+
+>>>>>>> MERGE-SOURCE
 def service(action, service_name):
     """Control a system service"""
     if init_is_systemd():
@@ -376,6 +409,7 @@
         return None
 
 
+<<<<<<< TREE
 def path_hash(path):
     """Generate a hash checksum of all files matching 'path'. Standard
     wildcards like '*' and '?' are supported, see documentation for the 'glob'
@@ -390,6 +424,23 @@
     }
 
 
+=======
+def path_hash(path):
+    """
+    Generate a hash checksum of all files matching 'path'. Standard wildcards
+    like '*' and '?' are supported, see documentation for the 'glob' module for
+    more information.
+
+    :return: dict: A { filename: hash } dictionary for all matched files.
+                   Empty if none found.
+    """
+    return {
+        filename: file_hash(filename)
+        for filename in glob.iglob(path)
+    }
+
+
+>>>>>>> MERGE-SOURCE
 def check_hash(path, checksum, hash_type='md5'):
     """Validate a file using a cryptographic checksum.
 
@@ -475,6 +526,7 @@
     return(''.join(random_chars))
 
 
+<<<<<<< TREE
 def is_phy_iface(interface):
     """Returns True if interface is not virtual, otherwise False."""
     if interface:
@@ -513,6 +565,46 @@
 
 def list_nics(nic_type=None):
     """Return a list of nics of given type(s)"""
+=======
+def is_phy_iface(interface):
+    """Returns True if interface is not virtual, otherwise False."""
+    if interface:
+        sys_net = '/sys/class/net'
+        if os.path.isdir(sys_net):
+            for iface in glob.glob(os.path.join(sys_net, '*')):
+                if '/virtual/' in os.path.realpath(iface):
+                    continue
+
+                if interface == os.path.basename(iface):
+                    return True
+
+    return False
+
+
+def get_bond_master(interface):
+    """Returns bond master if interface is bond slave otherwise None.
+
+    NOTE: the provided interface is expected to be physical
+    """
+    if interface:
+        iface_path = '/sys/class/net/%s' % (interface)
+        if os.path.exists(iface_path):
+            if '/virtual/' in os.path.realpath(iface_path):
+                return None
+
+            master = os.path.join(iface_path, 'master')
+            if os.path.exists(master):
+                master = os.path.realpath(master)
+                # make sure it is a bond master
+                if os.path.exists(os.path.join(master, 'bonding')):
+                    return os.path.basename(master)
+
+    return None
+
+
+def list_nics(nic_type=None):
+    '''Return a list of nics of given type(s)'''
+>>>>>>> MERGE-SOURCE
     if isinstance(nic_type, six.string_types):
         int_types = [nic_type]
     else:

=== added file 'hooks/charmhelpers/core/hugepage.py'
--- hooks/charmhelpers/core/hugepage.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/core/hugepage.py	2016-02-18 11:56:12 +0000
@@ -0,0 +1,62 @@
+# -*- coding: utf-8 -*-
+
+# Copyright 2014-2015 Canonical Limited.
+#
+# This file is part of charm-helpers.
+#
+# charm-helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# charm-helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public License
+# along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
+
+import yaml
+from charmhelpers.core import fstab
+from charmhelpers.core import sysctl
+from charmhelpers.core.host import (
+    add_group,
+    add_user_to_group,
+    fstab_mount,
+    mkdir,
+)
+
+
+def hugepage_support(user, group='hugetlb', nr_hugepages=256,
+                     max_map_count=65536, mnt_point='/run/hugepages/kvm',
+                     pagesize='2MB', mount=True):
+    """Enable hugepages on system.
+
+    Args:
+    user (str)  -- Username to allow access to hugepages to
+    group (str) -- Group name to own hugepages
+    nr_hugepages (int) -- Number of pages to reserve
+    max_map_count (int) -- Number of Virtual Memory Areas a process can own
+    mnt_point (str) -- Directory to mount hugepages on
+    pagesize (str) -- Size of hugepages
+    mount (bool) -- Whether to Mount hugepages
+    """
+    group_info = add_group(group)
+    gid = group_info.gr_gid
+    add_user_to_group(user, group)
+    sysctl_settings = {
+        'vm.nr_hugepages': nr_hugepages,
+        'vm.max_map_count': max_map_count,
+        'vm.hugetlb_shm_group': gid,
+    }
+    sysctl.create(yaml.dump(sysctl_settings), '/etc/sysctl.d/10-hugepage.conf')
+    mkdir(mnt_point, owner='root', group='root', perms=0o755, force=False)
+    lfstab = fstab.Fstab()
+    fstab_entry = lfstab.get_entry_by_attr('mountpoint', mnt_point)
+    if fstab_entry:
+        lfstab.remove_entry(fstab_entry)
+    entry = lfstab.Entry('nodev', mnt_point, 'hugetlbfs',
+                         'mode=1770,gid={},pagesize={}'.format(gid, pagesize), 0, 0)
+    lfstab.add_entry(entry)
+    if mount:
+        fstab_mount(mnt_point)

=== renamed file 'hooks/charmhelpers/core/hugepage.py' => 'hooks/charmhelpers/core/hugepage.py.moved'
=== modified file 'hooks/charmhelpers/core/services/helpers.py'
--- hooks/charmhelpers/core/services/helpers.py	2016-01-22 22:29:34 +0000
+++ hooks/charmhelpers/core/services/helpers.py	2016-02-18 11:56:12 +0000
@@ -247,22 +247,36 @@
     :param str owner: The owner of the rendered file
     :param str group: The group of the rendered file
     :param int perms: The permissions of the rendered file
+<<<<<<< TREE
     :param partial on_change_action: functools partial to be executed when
                                      rendered file changes
     :param jinja2 loader template_loader: A jinja2 template loader
 
     :return str: The rendered template
+=======
+    :param partial on_change_action: functools partial to be executed when
+                                     rendered file changes
+>>>>>>> MERGE-SOURCE
     """
     def __init__(self, source, target,
+<<<<<<< TREE
                  owner='root', group='root', perms=0o444,
                  on_change_action=None, template_loader=None):
+=======
+                 owner='root', group='root', perms=0o444,
+                 on_change_action=None):
+>>>>>>> MERGE-SOURCE
         self.source = source
         self.target = target
         self.owner = owner
         self.group = group
         self.perms = perms
+<<<<<<< TREE
         self.on_change_action = on_change_action
         self.template_loader = template_loader
+=======
+        self.on_change_action = on_change_action
+>>>>>>> MERGE-SOURCE
 
     def __call__(self, manager, service_name, event_name):
         pre_checksum = ''
@@ -272,6 +286,7 @@
         context = {'ctx': {}}
         for ctx in service.get('required_data', []):
             context.update(ctx)
+<<<<<<< TREE
             context['ctx'].update(ctx)
 
         result = templating.render(self.source, self.target, context,
@@ -286,6 +301,17 @@
                 self.on_change_action()
 
         return result
+=======
+        templating.render(self.source, self.target, context,
+                          self.owner, self.group, self.perms)
+        if self.on_change_action:
+            if pre_checksum == host.file_hash(self.target):
+                hookenv.log(
+                    'No change detected: {}'.format(self.target),
+                    hookenv.DEBUG)
+            else:
+                self.on_change_action()
+>>>>>>> MERGE-SOURCE
 
 
 # Convenience aliases for templates

=== modified file 'hooks/charmhelpers/fetch/__init__.py'
--- hooks/charmhelpers/fetch/__init__.py	2016-01-22 22:29:34 +0000
+++ hooks/charmhelpers/fetch/__init__.py	2016-02-18 11:56:12 +0000
@@ -90,6 +90,7 @@
     'kilo/proposed': 'trusty-proposed/kilo',
     'trusty-kilo/proposed': 'trusty-proposed/kilo',
     'trusty-proposed/kilo': 'trusty-proposed/kilo',
+<<<<<<< TREE
     # Liberty
     'liberty': 'trusty-updates/liberty',
     'trusty-liberty': 'trusty-updates/liberty',
@@ -106,6 +107,16 @@
     'mitaka/proposed': 'trusty-proposed/mitaka',
     'trusty-mitaka/proposed': 'trusty-proposed/mitaka',
     'trusty-proposed/mitaka': 'trusty-proposed/mitaka',
+=======
+    # Liberty
+    'liberty': 'trusty-updates/liberty',
+    'trusty-liberty': 'trusty-updates/liberty',
+    'trusty-liberty/updates': 'trusty-updates/liberty',
+    'trusty-updates/liberty': 'trusty-updates/liberty',
+    'liberty/proposed': 'trusty-proposed/liberty',
+    'trusty-liberty/proposed': 'trusty-proposed/liberty',
+    'trusty-proposed/liberty': 'trusty-proposed/liberty',
+>>>>>>> MERGE-SOURCE
 }
 
 # The order of this list is very important. Handlers should be listed in from
@@ -231,6 +242,7 @@
     _run_apt_command(cmd, fatal)
 
 
+<<<<<<< TREE
 def apt_mark(packages, mark, fatal=False):
     """Flag one or more packages using apt-mark"""
     log("Marking {} as {}".format(packages, mark))
@@ -246,6 +258,23 @@
         subprocess.call(cmd, universal_newlines=True)
 
 
+=======
+def apt_mark(packages, mark, fatal=False):
+    """Flag one or more packages using apt-mark"""
+    cmd = ['apt-mark', mark]
+    if isinstance(packages, six.string_types):
+        cmd.append(packages)
+    else:
+        cmd.extend(packages)
+    log("Holding {}".format(packages))
+
+    if fatal:
+        subprocess.check_call(cmd, universal_newlines=True)
+    else:
+        subprocess.call(cmd, universal_newlines=True)
+
+
+>>>>>>> MERGE-SOURCE
 def apt_hold(packages, fatal=False):
     return apt_mark(packages, 'hold', fatal=fatal)
 

=== modified file 'hooks/charmhelpers/fetch/archiveurl.py'
=== modified file 'hooks/charmhelpers/fetch/giturl.py'
--- hooks/charmhelpers/fetch/giturl.py	2016-01-22 22:29:34 +0000
+++ hooks/charmhelpers/fetch/giturl.py	2016-02-18 11:56:12 +0000
@@ -41,10 +41,15 @@
         else:
             return True
 
+<<<<<<< TREE
     def clone(self, source, dest, branch="master", depth=None):
+=======
+    def clone(self, source, dest, branch, depth=None):
+>>>>>>> MERGE-SOURCE
         if not self.can_handle(source):
             raise UnhandledSource("Cannot handle {}".format(source))
 
+<<<<<<< TREE
         if os.path.exists(dest):
             cmd = ['git', '-C', dest, 'pull', source, branch]
         else:
@@ -52,6 +57,12 @@
             if depth:
                 cmd.extend(['--depth', depth])
         check_call(cmd)
+=======
+        if depth:
+            Repo.clone_from(source, dest, branch=branch, depth=depth)
+        else:
+            Repo.clone_from(source, dest, branch=branch)
+>>>>>>> MERGE-SOURCE
 
     def install(self, source, branch="master", dest=None, depth=None):
         url_parts = self.parse_url(source)
@@ -62,9 +73,15 @@
             dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
                                     branch_name)
         try:
+<<<<<<< TREE
             self.clone(source, dest_dir, branch, depth)
         except CalledProcessError as e:
             raise UnhandledSource(e)
+=======
+            self.clone(source, dest_dir, branch, depth)
+        except GitCommandError as e:
+            raise UnhandledSource(e)
+>>>>>>> MERGE-SOURCE
         except OSError as e:
             raise UnhandledSource(e.strerror)
         return dest_dir

=== modified file 'hooks/install'
--- hooks/install	2015-09-22 13:35:49 +0000
+++ hooks/install	2016-02-18 11:56:12 +0000
@@ -1,3 +1,4 @@
+<<<<<<< TREE
 #!/bin/bash
 # Wrapper to deal with newer Ubuntu versions that don't have py2 installed
 # by default.
@@ -18,3 +19,25 @@
 done
 
 exec ./hooks/install.real
+=======
+#!/bin/bash
+# Wrapper to deal with newer Ubuntu versions that don't have py2 installed
+# by default.
+
+declare -a DEPS=('apt' 'netaddr' 'netifaces' 'pip' 'yaml' 'enum34')
+
+check_and_install() {
+    pkg="${1}-${2}"
+    if ! dpkg -s ${pkg} 2>&1 > /dev/null; then
+        apt-get -y install ${pkg}
+    fi
+}
+
+PYTHON="python"
+
+for dep in ${DEPS[@]}; do
+    check_and_install ${PYTHON} ${dep}
+done
+
+exec ./hooks/install.real
+>>>>>>> MERGE-SOURCE

=== added symlink 'hooks/install.real'
=== target is u'ceph_hooks.py'
=== renamed symlink 'hooks/install.real' => 'hooks/install.real.moved'
=== added symlink 'hooks/update-status'
=== target is u'ceph_hooks.py'
=== renamed symlink 'hooks/update-status' => 'hooks/update-status.moved'
=== modified file 'metadata.yaml'
=== added file 'requirements.txt'
--- requirements.txt	1970-01-01 00:00:00 +0000
+++ requirements.txt	2016-02-18 11:56:12 +0000
@@ -0,0 +1,11 @@
+# The order of packages is significant, because pip processes them in the order
+# of appearance. Changing the order has an impact on the overall integration
+# process, which may cause wedges in the gate later.
+PyYAML>=3.1.0
+simplejson>=2.2.0
+netifaces>=0.10.4
+netaddr>=0.7.12,!=0.7.16
+Jinja2>=2.6  # BSD License (3 clause)
+six>=1.9.0
+dnspython>=1.12.0
+psutil>=1.1.1,<2.0.0

=== renamed file 'requirements.txt' => 'requirements.txt.moved'
=== added file 'setup.cfg'
--- setup.cfg	1970-01-01 00:00:00 +0000
+++ setup.cfg	2016-02-18 11:56:12 +0000
@@ -0,0 +1,5 @@
+[nosetests]
+verbosity=2
+with-coverage=1
+cover-erase=1
+cover-package=hooks

=== renamed file 'setup.cfg' => 'setup.cfg.moved'
=== modified file 'templates/ceph.conf'
--- templates/ceph.conf	2016-01-13 12:48:57 +0000
+++ templates/ceph.conf	2016-02-18 11:56:12 +0000
@@ -41,3 +41,8 @@
 osd journal size = {{ osd_journal_size }}
 filestore xattr use omap = true
 
+# Improve read/write performance at the expense of backfill performance
+filestore merge threshold = 40
+filestore split multiple = 8
+osd op threads = 12
+

=== added file 'test-requirements.txt'
--- test-requirements.txt	1970-01-01 00:00:00 +0000
+++ test-requirements.txt	2016-02-18 11:56:12 +0000
@@ -0,0 +1,9 @@
+# The order of packages is significant, because pip processes them in the order
+# of appearance. Changing the order has an impact on the overall integration
+# process, which may cause wedges in the gate later.
+coverage>=3.6
+mock>=1.2
+flake8>=2.2.4,<=2.4.1
+os-testr>=0.4.1
+charm-tools
+enum

=== renamed file 'test-requirements.txt' => 'test-requirements.txt.moved'
=== modified file 'tests/README'
--- tests/README	2016-01-08 21:45:34 +0000
+++ tests/README	2016-02-18 11:56:12 +0000
@@ -1,3 +1,4 @@
+<<<<<<< TREE
 This directory provides Amulet tests to verify basic deployment functionality
 from the perspective of this charm, its requirements and its features, as
 exercised in a subset of the full OpenStack deployment test bundle topology.
@@ -22,6 +23,30 @@
     9xx restarts, config changes, actions and other final checks
 
 In order to run tests, charm-tools and juju must be installed:
+=======
+This directory provides Amulet tests that focus on verification of ceph-osd
+deployments.
+
+test_* methods are called in lexical sort order.
+
+Test name convention to ensure desired test order:
+    1xx service and endpoint checks
+    2xx relation checks
+    3xx config checks
+    4xx functional checks
+    9xx restarts and other final checks
+
+Common uses of ceph-osd relations in bundle deployments:
+    - - "ceph-osd:mon"
+      - "ceph:osd"
+
+More detailed relations of ceph-osd service in a common deployment:
+    relations:
+????
+
+In order to run tests, you'll need charm-tools installed (in addition to
+juju, of course):
+>>>>>>> MERGE-SOURCE
     sudo add-apt-repository ppa:juju/stable
     sudo apt-get update
     sudo apt-get install charm-tools juju juju-deployer amulet

=== modified file 'tests/basic_deployment.py'
--- tests/basic_deployment.py	2016-02-12 21:23:26 +0000
+++ tests/basic_deployment.py	2016-02-18 11:56:12 +0000
@@ -19,7 +19,7 @@
     """Amulet tests on a basic ceph-osd deployment."""
 
     def __init__(self, series=None, openstack=None, source=None,
-                 stable=False):
+                 stable=True):
         """Deploy the entire test environment."""
         super(CephOsdBasicDeployment, self).__init__(series, openstack,
                                                      source, stable)
@@ -116,10 +116,20 @@
         self.ceph1_sentry = self.d.sentry.unit['ceph/1']
         self.ceph2_sentry = self.d.sentry.unit['ceph/2']
         self.ceph_osd_sentry = self.d.sentry.unit['ceph-osd/0']
-        u.log.debug('openstack release val: {}'.format(
-            self._get_openstack_release()))
-        u.log.debug('openstack release str: {}'.format(
-            self._get_openstack_release_string()))
+<<<<<<< TREE
+        u.log.debug('openstack release val: {}'.format(
+            self._get_openstack_release()))
+        u.log.debug('openstack release str: {}'.format(
+            self._get_openstack_release_string()))
+=======
+        u.log.debug('openstack release val: {}'.format(
+            self._get_openstack_release()))
+        u.log.debug('openstack release str: {}'.format(
+            self._get_openstack_release_string()))
+
+        # Let things settle a bit original moving forward
+        time.sleep(30)
+>>>>>>> MERGE-SOURCE
 
         # Authenticate admin with keystone
         self.keystone = u.authenticate_keystone_admin(self.keystone_sentry,

=== modified file 'tests/charmhelpers/contrib/amulet/utils.py'
--- tests/charmhelpers/contrib/amulet/utils.py	2016-01-22 22:29:34 +0000
+++ tests/charmhelpers/contrib/amulet/utils.py	2016-02-18 11:56:12 +0000
@@ -19,8 +19,12 @@
 import logging
 import os
 import re
+<<<<<<< TREE
 import socket
 import subprocess
+=======
+import subprocess
+>>>>>>> MERGE-SOURCE
 import sys
 import time
 import uuid
@@ -107,6 +111,7 @@
         """Validate that lists of commands succeed on service units.  Can be
            used to verify system services are running on the corresponding
            service units.
+<<<<<<< TREE
 
         :param commands: dict with sentry keys and arbitrary command list vals
         :returns: None if successful, Failure string message otherwise
@@ -120,6 +125,21 @@
                       'validate_services_by_name instead of validate_services '
                       'due to init system differences.')
 
+=======
+
+        :param commands: dict with sentry keys and arbitrary command list vals
+        :returns: None if successful, Failure string message otherwise
+        """
+        self.log.debug('Checking status of system services...')
+
+        # /!\ DEPRECATION WARNING (beisner):
+        # New and existing tests should be rewritten to use
+        # validate_services_by_name() as it is aware of init systems.
+        self.log.warn('/!\\ DEPRECATION WARNING:  use '
+                      'validate_services_by_name instead of validate_services '
+                      'due to init system differences.')
+
+>>>>>>> MERGE-SOURCE
         for k, v in six.iteritems(commands):
             for cmd in v:
                 output, code = k.run(cmd)
@@ -501,6 +521,7 @@
 
     def endpoint_error(self, name, data):
         return 'unexpected endpoint data in {} - {}'.format(name, data)
+<<<<<<< TREE
 
     def get_ubuntu_releases(self):
         """Return a list of all Ubuntu releases in order of release."""
@@ -816,3 +837,176 @@
             return ("unknown", "")
         status = json.loads(raw_status)
         return (status["status"], status["message"])
+=======
+
+    def get_ubuntu_releases(self):
+        """Return a list of all Ubuntu releases in order of release."""
+        _d = distro_info.UbuntuDistroInfo()
+        _release_list = _d.all
+        self.log.debug('Ubuntu release list: {}'.format(_release_list))
+        return _release_list
+
+    def file_to_url(self, file_rel_path):
+        """Convert a relative file path to a file URL."""
+        _abs_path = os.path.abspath(file_rel_path)
+        return urlparse.urlparse(_abs_path, scheme='file').geturl()
+
+    def check_commands_on_units(self, commands, sentry_units):
+        """Check that all commands in a list exit zero on all
+        sentry units in a list.
+
+        :param commands:  list of bash commands
+        :param sentry_units:  list of sentry unit pointers
+        :returns: None if successful; Failure message otherwise
+        """
+        self.log.debug('Checking exit codes for {} commands on {} '
+                       'sentry units...'.format(len(commands),
+                                                len(sentry_units)))
+        for sentry_unit in sentry_units:
+            for cmd in commands:
+                output, code = sentry_unit.run(cmd)
+                if code == 0:
+                    self.log.debug('{} `{}` returned {} '
+                                   '(OK)'.format(sentry_unit.info['unit_name'],
+                                                 cmd, code))
+                else:
+                    return ('{} `{}` returned {} '
+                            '{}'.format(sentry_unit.info['unit_name'],
+                                        cmd, code, output))
+        return None
+
+    def get_process_id_list(self, sentry_unit, process_name,
+                            expect_success=True):
+        """Get a list of process ID(s) from a single sentry juju unit
+        for a single process name.
+
+        :param sentry_unit: Amulet sentry instance (juju unit)
+        :param process_name: Process name
+        :param expect_success: If False, expect the PID to be missing,
+            raise if it is present.
+        :returns: List of process IDs
+        """
+        cmd = 'pidof -x {}'.format(process_name)
+        if not expect_success:
+            cmd += " || exit 0 && exit 1"
+        output, code = sentry_unit.run(cmd)
+        if code != 0:
+            msg = ('{} `{}` returned {} '
+                   '{}'.format(sentry_unit.info['unit_name'],
+                               cmd, code, output))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+        return str(output).split()
+
+    def get_unit_process_ids(self, unit_processes, expect_success=True):
+        """Construct a dict containing unit sentries, process names, and
+        process IDs.
+
+        :param unit_processes: A dictionary of Amulet sentry instance
+            to list of process names.
+        :param expect_success: if False expect the processes to not be
+            running, raise if they are.
+        :returns: Dictionary of Amulet sentry instance to dictionary
+            of process names to PIDs.
+        """
+        pid_dict = {}
+        for sentry_unit, process_list in six.iteritems(unit_processes):
+            pid_dict[sentry_unit] = {}
+            for process in process_list:
+                pids = self.get_process_id_list(
+                    sentry_unit, process, expect_success=expect_success)
+                pid_dict[sentry_unit].update({process: pids})
+        return pid_dict
+
+    def validate_unit_process_ids(self, expected, actual):
+        """Validate process id quantities for services on units."""
+        self.log.debug('Checking units for running processes...')
+        self.log.debug('Expected PIDs: {}'.format(expected))
+        self.log.debug('Actual PIDs: {}'.format(actual))
+
+        if len(actual) != len(expected):
+            return ('Unit count mismatch.  expected, actual: {}, '
+                    '{} '.format(len(expected), len(actual)))
+
+        for (e_sentry, e_proc_names) in six.iteritems(expected):
+            e_sentry_name = e_sentry.info['unit_name']
+            if e_sentry in actual.keys():
+                a_proc_names = actual[e_sentry]
+            else:
+                return ('Expected sentry ({}) not found in actual dict data.'
+                        '{}'.format(e_sentry_name, e_sentry))
+
+            if len(e_proc_names.keys()) != len(a_proc_names.keys()):
+                return ('Process name count mismatch.  expected, actual: {}, '
+                        '{}'.format(len(expected), len(actual)))
+
+            for (e_proc_name, e_pids_length), (a_proc_name, a_pids) in \
+                    zip(e_proc_names.items(), a_proc_names.items()):
+                if e_proc_name != a_proc_name:
+                    return ('Process name mismatch.  expected, actual: {}, '
+                            '{}'.format(e_proc_name, a_proc_name))
+
+                a_pids_length = len(a_pids)
+                fail_msg = ('PID count mismatch. {} ({}) expected, actual: '
+                            '{}, {} ({})'.format(e_sentry_name, e_proc_name,
+                                                 e_pids_length, a_pids_length,
+                                                 a_pids))
+
+                # If expected is not bool, ensure PID quantities match
+                if not isinstance(e_pids_length, bool) and \
+                        a_pids_length != e_pids_length:
+                    return fail_msg
+                # If expected is bool True, ensure 1 or more PIDs exist
+                elif isinstance(e_pids_length, bool) and \
+                        e_pids_length is True and a_pids_length < 1:
+                    return fail_msg
+                # If expected is bool False, ensure 0 PIDs exist
+                elif isinstance(e_pids_length, bool) and \
+                        e_pids_length is False and a_pids_length != 0:
+                    return fail_msg
+                else:
+                    self.log.debug('PID check OK: {} {} {}: '
+                                   '{}'.format(e_sentry_name, e_proc_name,
+                                               e_pids_length, a_pids))
+        return None
+
+    def validate_list_of_identical_dicts(self, list_of_dicts):
+        """Check that all dicts within a list are identical."""
+        hashes = []
+        for _dict in list_of_dicts:
+            hashes.append(hash(frozenset(_dict.items())))
+
+        self.log.debug('Hashes: {}'.format(hashes))
+        if len(set(hashes)) == 1:
+            self.log.debug('Dicts within list are identical')
+        else:
+            return 'Dicts within list are not identical'
+
+        return None
+
+    def run_action(self, unit_sentry, action,
+                   _check_output=subprocess.check_output):
+        """Run the named action on a given unit sentry.
+
+        _check_output parameter is used for dependency injection.
+
+        @return action_id.
+        """
+        unit_id = unit_sentry.info["unit_name"]
+        command = ["juju", "action", "do", "--format=json", unit_id, action]
+        self.log.info("Running command: %s\n" % " ".join(command))
+        output = _check_output(command, universal_newlines=True)
+        data = json.loads(output)
+        action_id = data[u'Action queued with id']
+        return action_id
+
+    def wait_on_action(self, action_id, _check_output=subprocess.check_output):
+        """Wait for a given action, returning if it completed or not.
+
+        _check_output parameter is used for dependency injection.
+        """
+        command = ["juju", "action", "fetch", "--format=json", "--wait=0",
+                   action_id]
+        output = _check_output(command, universal_newlines=True)
+        data = json.loads(output)
+        return data.get(u"status") == "completed"
+>>>>>>> MERGE-SOURCE

=== modified file 'tests/charmhelpers/contrib/openstack/amulet/deployment.py'
--- tests/charmhelpers/contrib/openstack/amulet/deployment.py	2016-01-22 22:29:34 +0000
+++ tests/charmhelpers/contrib/openstack/amulet/deployment.py	2016-02-18 11:56:12 +0000
@@ -69,6 +69,7 @@
            Determine if the local branch being tested is derived from its
            stable or next (dev) branch, and based on this, use the corresonding
            stable or next branches for the other_services."""
+<<<<<<< TREE
 
         self.log.info('OpenStackAmuletDeployment:  determine branch locations')
 
@@ -82,6 +83,20 @@
 
         if self.series in ['precise', 'trusty']:
             base_series = self.series
+=======
+        base_charms = ['mysql', 'mongodb', 'nrpe']
+
+        if self.series in ['precise', 'trusty']:
+            base_series = self.series
+        else:
+            base_series = self.current_next
+
+        if self.stable:
+            for svc in other_services:
+                temp = 'lp:charms/{}/{}'
+                svc['location'] = temp.format(base_series,
+                                              svc['name'])
+>>>>>>> MERGE-SOURCE
         else:
             base_series = self.current_next
 
@@ -122,11 +137,17 @@
         # Charms which should use the source config option
         use_source = ['mysql', 'mongodb', 'rabbitmq-server', 'ceph',
                       'ceph-osd', 'ceph-radosgw']
+<<<<<<< TREE
 
         # Charms which can not use openstack-origin, ie. many subordinates
         no_origin = ['cinder-ceph', 'hacluster', 'neutron-openvswitch', 'nrpe',
                      'openvswitch-odl', 'neutron-api-odl', 'odl-controller',
                      'cinder-backup']
+=======
+        # Most OpenStack subordinate charms do not expose an origin option
+        # as that is controlled by the principle.
+        ignore = ['cinder-ceph', 'hacluster', 'neutron-openvswitch', 'nrpe']
+>>>>>>> MERGE-SOURCE
 
         if self.openstack:
             for svc in services:
@@ -224,11 +245,18 @@
         # Must be ordered by OpenStack release (not by Ubuntu release):
         (self.precise_essex, self.precise_folsom, self.precise_grizzly,
          self.precise_havana, self.precise_icehouse,
+<<<<<<< TREE
          self.trusty_icehouse, self.trusty_juno, self.utopic_juno,
          self.trusty_kilo, self.vivid_kilo, self.trusty_liberty,
          self.wily_liberty, self.trusty_mitaka,
          self.xenial_mitaka) = range(14)
 
+=======
+         self.trusty_icehouse, self.trusty_juno, self.utopic_juno,
+         self.trusty_kilo, self.vivid_kilo, self.trusty_liberty,
+         self.wily_liberty) = range(12)
+
+>>>>>>> MERGE-SOURCE
         releases = {
             ('precise', None): self.precise_essex,
             ('precise', 'cloud:precise-folsom'): self.precise_folsom,
@@ -238,12 +266,21 @@
             ('trusty', None): self.trusty_icehouse,
             ('trusty', 'cloud:trusty-juno'): self.trusty_juno,
             ('trusty', 'cloud:trusty-kilo'): self.trusty_kilo,
+<<<<<<< TREE
             ('trusty', 'cloud:trusty-liberty'): self.trusty_liberty,
             ('trusty', 'cloud:trusty-mitaka'): self.trusty_mitaka,
+=======
+            ('trusty', 'cloud:trusty-liberty'): self.trusty_liberty,
+>>>>>>> MERGE-SOURCE
             ('utopic', None): self.utopic_juno,
+<<<<<<< TREE
             ('vivid', None): self.vivid_kilo,
             ('wily', None): self.wily_liberty,
             ('xenial', None): self.xenial_mitaka}
+=======
+            ('vivid', None): self.vivid_kilo,
+            ('wily', None): self.wily_liberty}
+>>>>>>> MERGE-SOURCE
         return releases[(self.series, self.openstack)]
 
     def _get_openstack_release_string(self):
@@ -259,8 +296,12 @@
             ('trusty', 'icehouse'),
             ('utopic', 'juno'),
             ('vivid', 'kilo'),
+<<<<<<< TREE
             ('wily', 'liberty'),
             ('xenial', 'mitaka'),
+=======
+            ('wily', 'liberty'),
+>>>>>>> MERGE-SOURCE
         ])
         if self.openstack:
             os_origin = self.openstack.split(':')[1]

=== modified file 'tests/charmhelpers/contrib/openstack/amulet/utils.py'
--- tests/charmhelpers/contrib/openstack/amulet/utils.py	2016-01-22 22:29:34 +0000
+++ tests/charmhelpers/contrib/openstack/amulet/utils.py	2016-02-18 11:56:12 +0000
@@ -18,8 +18,12 @@
 import json
 import logging
 import os
+<<<<<<< TREE
 import re
 import six
+=======
+import six
+>>>>>>> MERGE-SOURCE
 import time
 import urllib
 
@@ -28,8 +32,12 @@
 import heatclient.v1.client as heat_client
 import keystoneclient.v2_0 as keystone_client
 import novaclient.v1_1.client as nova_client
+<<<<<<< TREE
 import pika
 import swiftclient
+=======
+import swiftclient
+>>>>>>> MERGE-SOURCE
 
 from charmhelpers.contrib.amulet.utils import (
     AmuletUtils
@@ -342,6 +350,7 @@
 
     def delete_instance(self, nova, instance):
         """Delete the specified instance."""
+<<<<<<< TREE
 
         # /!\ DEPRECATION WARNING
         self.log.warn('/!\\ DEPRECATION WARNING:  use '
@@ -983,3 +992,267 @@
         else:
             msg = 'No message retrieved.'
             amulet.raise_status(amulet.FAIL, msg)
+=======
+
+        # /!\ DEPRECATION WARNING
+        self.log.warn('/!\\ DEPRECATION WARNING:  use '
+                      'delete_resource instead of delete_instance.')
+        self.log.debug('Deleting instance ({})...'.format(instance))
+        return self.delete_resource(nova.servers, instance,
+                                    msg='nova instance')
+
+    def create_or_get_keypair(self, nova, keypair_name="testkey"):
+        """Create a new keypair, or return pointer if it already exists."""
+        try:
+            _keypair = nova.keypairs.get(keypair_name)
+            self.log.debug('Keypair ({}) already exists, '
+                           'using it.'.format(keypair_name))
+            return _keypair
+        except:
+            self.log.debug('Keypair ({}) does not exist, '
+                           'creating it.'.format(keypair_name))
+
+        _keypair = nova.keypairs.create(name=keypair_name)
+        return _keypair
+
+    def create_cinder_volume(self, cinder, vol_name="demo-vol", vol_size=1,
+                             img_id=None, src_vol_id=None, snap_id=None):
+        """Create cinder volume, optionally from a glance image, OR
+        optionally as a clone of an existing volume, OR optionally
+        from a snapshot.  Wait for the new volume status to reach
+        the expected status, validate and return a resource pointer.
+
+        :param vol_name: cinder volume display name
+        :param vol_size: size in gigabytes
+        :param img_id: optional glance image id
+        :param src_vol_id: optional source volume id to clone
+        :param snap_id: optional snapshot id to use
+        :returns: cinder volume pointer
+        """
+        # Handle parameter input and avoid impossible combinations
+        if img_id and not src_vol_id and not snap_id:
+            # Create volume from image
+            self.log.debug('Creating cinder volume from glance image...')
+            bootable = 'true'
+        elif src_vol_id and not img_id and not snap_id:
+            # Clone an existing volume
+            self.log.debug('Cloning cinder volume...')
+            bootable = cinder.volumes.get(src_vol_id).bootable
+        elif snap_id and not src_vol_id and not img_id:
+            # Create volume from snapshot
+            self.log.debug('Creating cinder volume from snapshot...')
+            snap = cinder.volume_snapshots.find(id=snap_id)
+            vol_size = snap.size
+            snap_vol_id = cinder.volume_snapshots.get(snap_id).volume_id
+            bootable = cinder.volumes.get(snap_vol_id).bootable
+        elif not img_id and not src_vol_id and not snap_id:
+            # Create volume
+            self.log.debug('Creating cinder volume...')
+            bootable = 'false'
+        else:
+            # Impossible combination of parameters
+            msg = ('Invalid method use - name:{} size:{} img_id:{} '
+                   'src_vol_id:{} snap_id:{}'.format(vol_name, vol_size,
+                                                     img_id, src_vol_id,
+                                                     snap_id))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        # Create new volume
+        try:
+            vol_new = cinder.volumes.create(display_name=vol_name,
+                                            imageRef=img_id,
+                                            size=vol_size,
+                                            source_volid=src_vol_id,
+                                            snapshot_id=snap_id)
+            vol_id = vol_new.id
+        except Exception as e:
+            msg = 'Failed to create volume: {}'.format(e)
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        # Wait for volume to reach available status
+        ret = self.resource_reaches_status(cinder.volumes, vol_id,
+                                           expected_stat="available",
+                                           msg="Volume status wait")
+        if not ret:
+            msg = 'Cinder volume failed to reach expected state.'
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        # Re-validate new volume
+        self.log.debug('Validating volume attributes...')
+        val_vol_name = cinder.volumes.get(vol_id).display_name
+        val_vol_boot = cinder.volumes.get(vol_id).bootable
+        val_vol_stat = cinder.volumes.get(vol_id).status
+        val_vol_size = cinder.volumes.get(vol_id).size
+        msg_attr = ('Volume attributes - name:{} id:{} stat:{} boot:'
+                    '{} size:{}'.format(val_vol_name, vol_id,
+                                        val_vol_stat, val_vol_boot,
+                                        val_vol_size))
+
+        if val_vol_boot == bootable and val_vol_stat == 'available' \
+                and val_vol_name == vol_name and val_vol_size == vol_size:
+            self.log.debug(msg_attr)
+        else:
+            msg = ('Volume validation failed, {}'.format(msg_attr))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        return vol_new
+
+    def delete_resource(self, resource, resource_id,
+                        msg="resource", max_wait=120):
+        """Delete one openstack resource, such as one instance, keypair,
+        image, volume, stack, etc., and confirm deletion within max wait time.
+
+        :param resource: pointer to os resource type, ex:glance_client.images
+        :param resource_id: unique name or id for the openstack resource
+        :param msg: text to identify purpose in logging
+        :param max_wait: maximum wait time in seconds
+        :returns: True if successful, otherwise False
+        """
+        self.log.debug('Deleting OpenStack resource '
+                       '{} ({})'.format(resource_id, msg))
+        num_before = len(list(resource.list()))
+        resource.delete(resource_id)
+
+        tries = 0
+        num_after = len(list(resource.list()))
+        while num_after != (num_before - 1) and tries < (max_wait / 4):
+            self.log.debug('{} delete check: '
+                           '{} [{}:{}] {}'.format(msg, tries,
+                                                  num_before,
+                                                  num_after,
+                                                  resource_id))
+            time.sleep(4)
+            num_after = len(list(resource.list()))
+            tries += 1
+
+        self.log.debug('{}:  expected, actual count = {}, '
+                       '{}'.format(msg, num_before - 1, num_after))
+
+        if num_after == (num_before - 1):
+            return True
+        else:
+            self.log.error('{} delete timed out'.format(msg))
+            return False
+
+    def resource_reaches_status(self, resource, resource_id,
+                                expected_stat='available',
+                                msg='resource', max_wait=120):
+        """Wait for an openstack resources status to reach an
+           expected status within a specified time.  Useful to confirm that
+           nova instances, cinder vols, snapshots, glance images, heat stacks
+           and other resources eventually reach the expected status.
+
+        :param resource: pointer to os resource type, ex: heat_client.stacks
+        :param resource_id: unique id for the openstack resource
+        :param expected_stat: status to expect resource to reach
+        :param msg: text to identify purpose in logging
+        :param max_wait: maximum wait time in seconds
+        :returns: True if successful, False if status is not reached
+        """
+
+        tries = 0
+        resource_stat = resource.get(resource_id).status
+        while resource_stat != expected_stat and tries < (max_wait / 4):
+            self.log.debug('{} status check: '
+                           '{} [{}:{}] {}'.format(msg, tries,
+                                                  resource_stat,
+                                                  expected_stat,
+                                                  resource_id))
+            time.sleep(4)
+            resource_stat = resource.get(resource_id).status
+            tries += 1
+
+        self.log.debug('{}:  expected, actual status = {}, '
+                       '{}'.format(msg, resource_stat, expected_stat))
+
+        if resource_stat == expected_stat:
+            return True
+        else:
+            self.log.debug('{} never reached expected status: '
+                           '{}'.format(resource_id, expected_stat))
+            return False
+
+    def get_ceph_osd_id_cmd(self, index):
+        """Produce a shell command that will return a ceph-osd id."""
+        return ("`initctl list | grep 'ceph-osd ' | "
+                "awk 'NR=={} {{ print $2 }}' | "
+                "grep -o '[0-9]*'`".format(index + 1))
+
+    def get_ceph_pools(self, sentry_unit):
+        """Return a dict of ceph pools from a single ceph unit, with
+        pool name as keys, pool id as vals."""
+        pools = {}
+        cmd = 'sudo ceph osd lspools'
+        output, code = sentry_unit.run(cmd)
+        if code != 0:
+            msg = ('{} `{}` returned {} '
+                   '{}'.format(sentry_unit.info['unit_name'],
+                               cmd, code, output))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        # Example output: 0 data,1 metadata,2 rbd,3 cinder,4 glance,
+        for pool in str(output).split(','):
+            pool_id_name = pool.split(' ')
+            if len(pool_id_name) == 2:
+                pool_id = pool_id_name[0]
+                pool_name = pool_id_name[1]
+                pools[pool_name] = int(pool_id)
+
+        self.log.debug('Pools on {}: {}'.format(sentry_unit.info['unit_name'],
+                                                pools))
+        return pools
+
+    def get_ceph_df(self, sentry_unit):
+        """Return dict of ceph df json output, including ceph pool state.
+
+        :param sentry_unit: Pointer to amulet sentry instance (juju unit)
+        :returns: Dict of ceph df output
+        """
+        cmd = 'sudo ceph df --format=json'
+        output, code = sentry_unit.run(cmd)
+        if code != 0:
+            msg = ('{} `{}` returned {} '
+                   '{}'.format(sentry_unit.info['unit_name'],
+                               cmd, code, output))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+        return json.loads(output)
+
+    def get_ceph_pool_sample(self, sentry_unit, pool_id=0):
+        """Take a sample of attributes of a ceph pool, returning ceph
+        pool name, object count and disk space used for the specified
+        pool ID number.
+
+        :param sentry_unit: Pointer to amulet sentry instance (juju unit)
+        :param pool_id: Ceph pool ID
+        :returns: List of pool name, object count, kb disk space used
+        """
+        df = self.get_ceph_df(sentry_unit)
+        pool_name = df['pools'][pool_id]['name']
+        obj_count = df['pools'][pool_id]['stats']['objects']
+        kb_used = df['pools'][pool_id]['stats']['kb_used']
+        self.log.debug('Ceph {} pool (ID {}): {} objects, '
+                       '{} kb used'.format(pool_name, pool_id,
+                                           obj_count, kb_used))
+        return pool_name, obj_count, kb_used
+
+    def validate_ceph_pool_samples(self, samples, sample_type="resource pool"):
+        """Validate ceph pool samples taken over time, such as pool
+        object counts or pool kb used, before adding, after adding, and
+        after deleting items which affect those pool attributes.  The
+        2nd element is expected to be greater than the 1st; 3rd is expected
+        to be less than the 2nd.
+
+        :param samples: List containing 3 data samples
+        :param sample_type: String for logging and usage context
+        :returns: None if successful, Failure message otherwise
+        """
+        original, created, deleted = range(3)
+        if samples[created] <= samples[original] or \
+                samples[deleted] >= samples[created]:
+            return ('Ceph {} samples ({}) '
+                    'unexpected.'.format(sample_type, samples))
+        else:
+            self.log.debug('Ceph {} samples (OK): '
+                           '{}'.format(sample_type, samples))
+            return None
+>>>>>>> MERGE-SOURCE

=== added file 'tests/tests.yaml'
--- tests/tests.yaml	1970-01-01 00:00:00 +0000
+++ tests/tests.yaml	2016-02-18 11:56:12 +0000
@@ -0,0 +1,20 @@
+bootstrap: true
+reset: true
+virtualenv: true
+makefile:
+    - lint
+    - test
+sources:
+    - ppa:juju/stable
+packages:
+    - amulet
+    - distro-info-data
+    - python-cinderclient
+    - python-distro-info
+    - python-glanceclient
+    - python-heatclient
+    - python-keystoneclient
+    - python-neutronclient
+    - python-novaclient
+    - python-pika
+    - python-swiftclient

=== renamed file 'tests/tests.yaml' => 'tests/tests.yaml.moved'
=== added file 'tox.ini'
--- tox.ini	1970-01-01 00:00:00 +0000
+++ tox.ini	2016-02-18 11:56:12 +0000
@@ -0,0 +1,29 @@
+[tox]
+envlist = lint,py27
+skipsdist = True
+
+[testenv]
+setenv = VIRTUAL_ENV={envdir}
+         PYTHONHASHSEED=0
+install_command =
+  pip install --allow-unverified python-apt {opts} {packages}
+commands = ostestr {posargs}
+
+[testenv:py27]
+basepython = python2.7
+deps = -r{toxinidir}/requirements.txt
+       -r{toxinidir}/test-requirements.txt
+
+[testenv:lint]
+basepython = python2.7
+deps = -r{toxinidir}/requirements.txt
+       -r{toxinidir}/test-requirements.txt
+commands = flake8 {posargs} hooks unit_tests tests
+           charm proof
+
+[testenv:venv]
+commands = {posargs}
+
+[flake8]
+ignore = E402,E226
+exclude = hooks/charmhelpers

=== renamed file 'tox.ini' => 'tox.ini.moved'
=== added file 'unit_tests/test_status.py'
--- unit_tests/test_status.py	1970-01-01 00:00:00 +0000
+++ unit_tests/test_status.py	2016-02-18 11:56:12 +0000
@@ -0,0 +1,56 @@
+import mock
+import test_utils
+
+import ceph_hooks as hooks
+
+TO_PATCH = [
+    'status_set',
+    'config',
+    'ceph',
+    'relation_ids',
+    'relation_get',
+    'related_units',
+    'get_conf',
+]
+
+CEPH_MONS = [
+    'ceph/0',
+    'ceph/1',
+    'ceph/2',
+]
+
+
+class ServiceStatusTestCase(test_utils.CharmTestCase):
+
+    def setUp(self):
+        super(ServiceStatusTestCase, self).setUp(hooks, TO_PATCH)
+        self.config.side_effect = self.test_config.get
+
+    def test_assess_status_no_monitor_relation(self):
+        self.relation_ids.return_value = []
+        hooks.assess_status()
+        self.status_set.assert_called_with('blocked', mock.ANY)
+
+    def test_assess_status_monitor_relation_incomplete(self):
+        self.relation_ids.return_value = ['mon:1']
+        self.related_units.return_value = CEPH_MONS
+        self.get_conf.return_value = None
+        hooks.assess_status()
+        self.status_set.assert_called_with('waiting', mock.ANY)
+
+    def test_assess_status_monitor_complete_no_disks(self):
+        self.relation_ids.return_value = ['mon:1']
+        self.related_units.return_value = CEPH_MONS
+        self.get_conf.return_value = 'monitor-bootstrap-key'
+        self.ceph.get_running_osds.return_value = []
+        hooks.assess_status()
+        self.status_set.assert_called_with('blocked', mock.ANY)
+
+    def test_assess_status_monitor_complete_disks(self):
+        self.relation_ids.return_value = ['mon:1']
+        self.related_units.return_value = CEPH_MONS
+        self.get_conf.return_value = 'monitor-bootstrap-key'
+        self.ceph.get_running_osds.return_value = ['12345',
+                                                   '67890']
+        hooks.assess_status()
+        self.status_set.assert_called_with('active', mock.ANY)

=== renamed file 'unit_tests/test_status.py' => 'unit_tests/test_status.py.moved'
=== added file 'unit_tests/test_utils.py'
--- unit_tests/test_utils.py	1970-01-01 00:00:00 +0000
+++ unit_tests/test_utils.py	2016-02-18 11:56:12 +0000
@@ -0,0 +1,121 @@
+import logging
+import unittest
+import os
+import yaml
+
+from contextlib import contextmanager
+from mock import patch, MagicMock
+
+
+def load_config():
+    '''
+    Walk backwords from __file__ looking for config.yaml, load and return the
+    'options' section'
+    '''
+    config = None
+    f = __file__
+    while config is None:
+        d = os.path.dirname(f)
+        if os.path.isfile(os.path.join(d, 'config.yaml')):
+            config = os.path.join(d, 'config.yaml')
+            break
+        f = d
+
+    if not config:
+        logging.error('Could not find config.yaml in any parent directory '
+                      'of %s. ' % f)
+        raise Exception
+
+    return yaml.safe_load(open(config).read())['options']
+
+
+def get_default_config():
+    '''
+    Load default charm config from config.yaml return as a dict.
+    If no default is set in config.yaml, its value is None.
+    '''
+    default_config = {}
+    config = load_config()
+    for k, v in config.iteritems():
+        if 'default' in v:
+            default_config[k] = v['default']
+        else:
+            default_config[k] = None
+    return default_config
+
+
+class CharmTestCase(unittest.TestCase):
+
+    def setUp(self, obj, patches):
+        super(CharmTestCase, self).setUp()
+        self.patches = patches
+        self.obj = obj
+        self.test_config = TestConfig()
+        self.test_relation = TestRelation()
+        self.patch_all()
+
+    def patch(self, method):
+        _m = patch.object(self.obj, method)
+        mock = _m.start()
+        self.addCleanup(_m.stop)
+        return mock
+
+    def patch_all(self):
+        for method in self.patches:
+            setattr(self, method, self.patch(method))
+
+
+class TestConfig(object):
+
+    def __init__(self):
+        self.config = get_default_config()
+
+    def get(self, attr=None):
+        if not attr:
+            return self.get_all()
+        try:
+            return self.config[attr]
+        except KeyError:
+            return None
+
+    def get_all(self):
+        return self.config
+
+    def set(self, attr, value):
+        if attr not in self.config:
+            raise KeyError
+        self.config[attr] = value
+
+
+class TestRelation(object):
+
+    def __init__(self, relation_data={}):
+        self.relation_data = relation_data
+
+    def set(self, relation_data):
+        self.relation_data = relation_data
+
+    def get(self, attr=None, unit=None, rid=None):
+        if attr is None:
+            return self.relation_data
+        elif attr in self.relation_data:
+            return self.relation_data[attr]
+        return None
+
+
+@contextmanager
+def patch_open():
+    '''Patch open() to allow mocking both open() itself and the file that is
+    yielded.
+
+    Yields the mock for "open" and "file", respectively.'''
+    mock_open = MagicMock(spec=open)
+    mock_file = MagicMock(spec=file)
+
+    @contextmanager
+    def stub_open(*args, **kwargs):
+        mock_open(*args, **kwargs)
+        yield mock_file
+
+    with patch('__builtin__.open', stub_open):
+        yield mock_open, mock_file

=== renamed file 'unit_tests/test_utils.py' => 'unit_tests/test_utils.py.moved'