Merge lp:~1chb1n/charms/trusty/cinder/next-amulet-updates into lp:~openstack-charmers-archive/charms/trusty/cinder/next

[Ryan Beisner]

Update amulet tests for vivid, prep for wily. Refactor existing tests. Move pieces to charmhelpers. Sync hooks/charmhelpers. Sync tests/charmhelpers.

Update amulet tests for vivid, prep for wily. Refactor existing tests. Move pieces to charmhelpers. Sync hooks/charmhelpers. Sync tests/charmhelpers.

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #5589 cinder-next for 1chb1n mp262772
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/5589/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #5221 cinder-next for 1chb1n mp262772
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/5221/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4755 cinder-next for 1chb1n mp262772
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/11765323/
Build: http://10.245.162.77:8080/job/charm_amulet_test/4755/

[Ryan Beisner (1chb1n)]

FYI, undercloud issue caused test failure for #4755.

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #5590 cinder-next for 1chb1n mp262772
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/5590/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #5222 cinder-next for 1chb1n mp262772
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/5222/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4777 cinder-next for 1chb1n mp262772
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/4777/

[Ryan Beisner (1chb1n)]

Flipped back to WIP re: tests/charmhelpers work in progress. Other things here are clear for review and input.

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #5675 cinder-next for 1chb1n mp262772
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/5675/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #5307 cinder-next for 1chb1n mp262772
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/5307/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4858 cinder-next for 1chb1n mp262772
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/4858/

[lp:~1chb1n/charms/trusty/cinder/next-amulet-updates]

105. By Ryan Beisner on 2015-06-29

add tests.yaml, update 00-setup

106. By Ryan Beisner on 2015-06-29

fix publish target in makefile

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #5683 cinder-next for 1chb1n mp262772
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/5683/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #5315 cinder-next for 1chb1n mp262772
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/5315/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4866 cinder-next for 1chb1n mp262772
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/11794686/
Build: http://10.245.162.77:8080/job/charm_amulet_test/4866/

[lp:~1chb1n/charms/trusty/cinder/next-amulet-updates]

107. By Ryan Beisner on 2015-06-29

fix 00-setup

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #5691 cinder-next for 1chb1n mp262772
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/5691/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #5323 cinder-next for 1chb1n mp262772
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/5323/

[Corey Bryant (corey.bryant)]

Looks good (one suggestion inline to ponder, not a blocker to merging). I'll approve once the corresponding c-h lands and these amulet tests are successful.

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4874 cinder-next for 1chb1n mp262772
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/11795712/
Build: http://10.245.162.77:8080/job/charm_amulet_test/4874/

[Ryan Beisner (1chb1n)]

Corey - Thanks for the review. Regarding the local helper methods, I also think they might have a place in c-h, but wanted to wait and get through this iteration of all of the os-charm updates to get a better feel for where or if they might be reusable elsewhere. ie. Erring on the side of keeping it out of c-h until there is a need.

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4881 cinder-next for 1chb1n mp262772
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/4881/

[Ryan Beisner (1chb1n)]

Test rig issue is causing failures in bootstrapping; will re-test when that's resolved.

[lp:~1chb1n/charms/trusty/cinder/next-amulet-updates]

108. By Ryan Beisner on 2015-07-01

update tags for consistency with other openstack charms

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #5701 cinder-next for 1chb1n mp262772
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/5701/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #5333 cinder-next for 1chb1n mp262772
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/5333/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4889 cinder-next for 1chb1n mp262772
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/4889/

[Corey Bryant (corey.bryant)]

review: Approve

=== modified file 'Makefile'
--- Makefile	2015-04-16 21:32:02 +0000
+++ Makefile	2015-07-01 14:48:09 +0000
@@ -2,17 +2,17 @@
 PYTHON := /usr/bin/env python
 
 lint:
-	@flake8 --exclude hooks/charmhelpers actions hooks unit_tests tests
+	@flake8 --exclude hooks/charmhelpers,tests/charmhelpers \
+        actions hooks unit_tests tests
 	@charm proof
 
-unit_test:
+test:
+	@# Bundletester expects unit tests here.
 	@echo Starting unit tests...
 	@$(PYTHON) /usr/bin/nosetests --nologcapture --with-coverage unit_tests
 
-test:
+functional_test:
 	@echo Starting amulet deployment tests...
-	#NOTE(beisner): can remove -v after bug 1320357 is fixed
-	#   https://bugs.launchpad.net/amulet/+bug/1320357
 	@juju test -v -p AMULET_HTTP_PROXY,AMULET_OS_VIP --timeout 2700
 
 bin/charm_helpers_sync.py:
@@ -24,6 +24,6 @@
 	@$(PYTHON) bin/charm_helpers_sync.py -c charm-helpers-hooks.yaml
 	@$(PYTHON) bin/charm_helpers_sync.py -c charm-helpers-tests.yaml
 
-publish: lint unit_test
+publish: lint test
 	bzr push lp:charms/cinder
 	bzr push lp:charms/trusty/cinder

=== modified file 'hooks/charmhelpers/core/hookenv.py'
--- hooks/charmhelpers/core/hookenv.py	2015-06-10 21:37:05 +0000
+++ hooks/charmhelpers/core/hookenv.py	2015-07-01 14:48:09 +0000
@@ -21,7 +21,9 @@
 #  Charm Helpers Developers <[e-mail]>
 
 from __future__ import print_function
+from distutils.version import LooseVersion
 from functools import wraps
+import glob
 import os
 import json
 import yaml
@@ -242,29 +244,7 @@
         self.path = os.path.join(charm_dir(), Config.CONFIG_FILE_NAME)
         if os.path.exists(self.path):
             self.load_previous()
-
-    def __getitem__(self, key):
-        """For regular dict lookups, check the current juju config first,
-        then the previous (saved) copy. This ensures that user-saved values
-        will be returned by a dict lookup.
-
-        """
-        try:
-            return dict.__getitem__(self, key)
-        except KeyError:
-            return (self._prev_dict or {})[key]
-
-    def get(self, key, default=None):
-        try:
-            return self[key]
-        except KeyError:
-            return default
-
-    def keys(self):
-        prev_keys = []
-        if self._prev_dict is not None:
-            prev_keys = self._prev_dict.keys()
-        return list(set(prev_keys + list(dict.keys(self))))
+        atexit(self._implicit_save)
 
     def load_previous(self, path=None):
         """Load previous copy of config from disk.
@@ -283,6 +263,9 @@
         self.path = path or self.path
         with open(self.path) as f:
             self._prev_dict = json.load(f)
+        for k, v in self._prev_dict.items():
+            if k not in self:
+                self[k] = v
 
     def changed(self, key):
         """Return True if the current value for this key is different from
@@ -314,13 +297,13 @@
         instance.
 
         """
-        if self._prev_dict:
-            for k, v in six.iteritems(self._prev_dict):
-                if k not in self:
-                    self[k] = v
         with open(self.path, 'w') as f:
             json.dump(self, f)
 
+    def _implicit_save(self):
+        if self.implicit_save:
+            self.save()
+
 
 @cached
 def config(scope=None):
@@ -587,10 +570,14 @@
             hooks.execute(sys.argv)
     """
 
-    def __init__(self, config_save=True):
+    def __init__(self, config_save=None):
         super(Hooks, self).__init__()
         self._hooks = {}
-        self._config_save = config_save
+
+        # For unknown reasons, we allow the Hooks constructor to override
+        # config().implicit_save.
+        if config_save is not None:
+            config().implicit_save = config_save
 
     def register(self, name, function):
         """Register a hook"""
@@ -598,13 +585,16 @@
 
     def execute(self, args):
         """Execute a registered hook based on args[0]"""
+        _run_atstart()
         hook_name = os.path.basename(args[0])
         if hook_name in self._hooks:
-            self._hooks[hook_name]()
-            if self._config_save:
-                cfg = config()
-                if cfg.implicit_save:
-                    cfg.save()
+            try:
+                self._hooks[hook_name]()
+            except SystemExit as x:
+                if x.code is None or x.code == 0:
+                    _run_atexit()
+                raise
+            _run_atexit()
         else:
             raise UnregisteredHookError(hook_name)
 
@@ -732,13 +722,79 @@
 @translate_exc(from_exc=OSError, to_exc=NotImplementedError)
 def leader_set(settings=None, **kwargs):
     """Juju leader set value(s)"""
-    log("Juju leader-set '%s'" % (settings), level=DEBUG)
+    # Don't log secrets.
+    # log("Juju leader-set '%s'" % (settings), level=DEBUG)
     cmd = ['leader-set']
     settings = settings or {}
     settings.update(kwargs)
-    for k, v in settings.iteritems():
+    for k, v in settings.items():
         if v is None:
             cmd.append('{}='.format(k))
         else:
             cmd.append('{}={}'.format(k, v))
     subprocess.check_call(cmd)
+
+
+@cached
+def juju_version():
+    """Full version string (eg. '1.23.3.1-trusty-amd64')"""
+    # Per https://bugs.launchpad.net/juju-core/+bug/1455368/comments/1
+    jujud = glob.glob('/var/lib/juju/tools/machine-*/jujud')[0]
+    return subprocess.check_output([jujud, 'version'],
+                                   universal_newlines=True).strip()
+
+
+@cached
+def has_juju_version(minimum_version):
+    """Return True if the Juju version is at least the provided version"""
+    return LooseVersion(juju_version()) >= LooseVersion(minimum_version)
+
+
+_atexit = []
+_atstart = []
+
+
+def atstart(callback, *args, **kwargs):
+    '''Schedule a callback to run before the main hook.
+
+    Callbacks are run in the order they were added.
+
+    This is useful for modules and classes to perform initialization
+    and inject behavior. In particular:
+        - Run common code before all of your hooks, such as logging
+          the hook name or interesting relation data.
+        - Defer object or module initialization that requires a hook
+          context until we know there actually is a hook context,
+          making testing easier.
+        - Rather than requiring charm authors to include boilerplate to
+          invoke your helper's behavior, have it run automatically if
+          your object is instantiated or module imported.
+
+    This is not at all useful after your hook framework as been launched.
+    '''
+    global _atstart
+    _atstart.append((callback, args, kwargs))
+
+
+def atexit(callback, *args, **kwargs):
+    '''Schedule a callback to run on successful hook completion.
+
+    Callbacks are run in the reverse order that they were added.'''
+    _atexit.append((callback, args, kwargs))
+
+
+def _run_atstart():
+    '''Hook frameworks must invoke this before running the main hook body.'''
+    global _atstart
+    for callback, args, kwargs in _atstart:
+        callback(*args, **kwargs)
+    del _atstart[:]
+
+
+def _run_atexit():
+    '''Hook frameworks must invoke this after the main hook body has
+    successfully completed. Do not invoke it if the hook fails.'''
+    global _atexit
+    for callback, args, kwargs in reversed(_atexit):
+        callback(*args, **kwargs)
+    del _atexit[:]

=== modified file 'hooks/charmhelpers/core/services/base.py'
--- hooks/charmhelpers/core/services/base.py	2015-06-10 21:37:05 +0000
+++ hooks/charmhelpers/core/services/base.py	2015-07-01 14:48:09 +0000
@@ -128,15 +128,18 @@
         """
         Handle the current hook by doing The Right Thing with the registered services.
         """
-        hook_name = hookenv.hook_name()
-        if hook_name == 'stop':
-            self.stop_services()
-        else:
-            self.reconfigure_services()
-            self.provide_data()
-        cfg = hookenv.config()
-        if cfg.implicit_save:
-            cfg.save()
+        hookenv._run_atstart()
+        try:
+            hook_name = hookenv.hook_name()
+            if hook_name == 'stop':
+                self.stop_services()
+            else:
+                self.reconfigure_services()
+                self.provide_data()
+        except SystemExit as x:
+            if x.code is None or x.code == 0:
+                hookenv._run_atexit()
+        hookenv._run_atexit()
 
     def provide_data(self):
         """

=== modified file 'metadata.yaml'
--- metadata.yaml	2015-01-09 16:02:39 +0000
+++ metadata.yaml	2015-07-01 14:48:09 +0000
@@ -3,8 +3,10 @@
 maintainer: Adam Gandelman <[e-mail]>
 description: |
  Cinder is a storage service for the Openstack project
-categories:
-  - miscellaneous
+tags:
+  - openstack
+  - storage
+  - misc
 provides:
   nrpe-external-master:
     interface: nrpe-external-master

=== modified file 'tests/00-setup'
--- tests/00-setup	2015-03-08 02:27:57 +0000
+++ tests/00-setup	2015-07-01 14:48:09 +0000
@@ -6,6 +6,9 @@
 sudo apt-get update --yes
 sudo apt-get install --yes python-amulet \
                            python-cinderclient \
+                           python-distro-info \
                            python-glanceclient \
+                           python-heatclient \
                            python-keystoneclient \
-                           python-novaclient
+                           python-novaclient \
+                           python-swiftclient

=== modified file 'tests/017-basic-trusty-kilo' (properties changed: -x to +x)
=== modified file 'tests/019-basic-vivid-kilo' (properties changed: -x to +x)
=== modified file 'tests/053-basic-trusty-kilo-git' (properties changed: -x to +x)
=== modified file 'tests/054-basic-vivid-kilo-git' (properties changed: -x to +x)
=== modified file 'tests/README'
--- tests/README	2014-10-07 18:32:59 +0000
+++ tests/README	2015-07-01 14:48:09 +0000
@@ -1,6 +1,15 @@
 This directory provides Amulet tests that focus on verification of Cinder
 deployments.
 
+test_* methods are called in lexical sort order.
+
+Test name convention to ensure desired test order:
+    1xx service and endpoint checks
+    2xx relation checks
+    3xx config checks
+    4xx functional checks
+    9xx restarts and other final checks
+
 In order to run tests, you'll need charm-tools installed (in addition to
 juju, of course):
     sudo add-apt-repository ppa:juju/stable

=== modified file 'tests/basic_deployment.py'
--- tests/basic_deployment.py	2015-05-12 14:49:27 +0000
+++ tests/basic_deployment.py	2015-07-01 14:48:09 +0000
@@ -2,19 +2,17 @@
 
 import amulet
 import os
-import types
-from time import sleep
+import time
 import yaml
-import cinderclient.v1.client as cinder_client
 
 from charmhelpers.contrib.openstack.amulet.deployment import (
     OpenStackAmuletDeployment
 )
 
-from charmhelpers.contrib.openstack.amulet.utils import (  # noqa
+from charmhelpers.contrib.openstack.amulet.utils import (
     OpenStackAmuletUtils,
     DEBUG,
-    ERROR
+    # ERROR
 )
 
 # Use DEBUG to turn on debug logging
@@ -22,17 +20,14 @@
 
 
 class CinderBasicDeployment(OpenStackAmuletDeployment):
-    '''Amulet tests on a basic lvm-backed cinder deployment. Verify
+    """Amulet tests on a basic lvm-backed cinder deployment. Verify
        relations, service status, users and endpoint service catalog.
        Create, clone, delete volumes. Create volume from glance image.
-       Create volume snapshot. Create volume from snapshot.'''
-
-    # NOTE(beisner):  Features and tests vary across Openstack releases.
-    # https://wiki.openstack.org/wiki/CinderSupportMatrix
+       Create volume snapshot. Create volume from snapshot."""
 
     def __init__(self, series=None, openstack=None, source=None, git=False,
                  stable=False):
-        '''Deploy the entire test environment.'''
+        """Deploy the entire test environment."""
         super(CinderBasicDeployment, self).__init__(series, openstack, source,
                                                     stable)
         self.git = git
@@ -56,7 +51,7 @@
                                                          other_services)
 
     def _add_relations(self):
-        '''Add relations for the services.'''
+        """Add relations for the services."""
         relations = {
             'keystone:shared-db': 'mysql:shared-db',
             'cinder:shared-db': 'mysql:shared-db',
@@ -70,7 +65,7 @@
         super(CinderBasicDeployment, self)._add_relations(relations)
 
     def _configure_services(self):
-        '''Configure all of the services.'''
+        """Configure all of the services."""
         cinder_config = {'block-device': 'vdb',
                          'glance-api-version': '2',
                          'overwrite': 'true'}
@@ -102,137 +97,211 @@
         super(CinderBasicDeployment, self)._configure_services(configs)
 
     def _initialize_tests(self):
-        '''Perform final initialization before tests get run.'''
+        """Perform final initialization before tests get run."""
         # Access the sentries for inspecting service units
         self.cinder_sentry = self.d.sentry.unit['cinder/0']
         self.glance_sentry = self.d.sentry.unit['glance/0']
         self.mysql_sentry = self.d.sentry.unit['mysql/0']
         self.keystone_sentry = self.d.sentry.unit['keystone/0']
         self.rabbitmq_sentry = self.d.sentry.unit['rabbitmq-server/0']
+        u.log.debug('openstack release val: {}'.format(
+            self._get_openstack_release()))
+        u.log.debug('openstack release str: {}'.format(
+            self._get_openstack_release_string()))
+
+        # Let things settle a bit original moving forward
+        time.sleep(30)
 
         # Authenticate admin with keystone
         self.keystone = u.authenticate_keystone_admin(self.keystone_sentry,
                                                       user='admin',
                                                       password='openstack',
                                                       tenant='admin')
+
         # Authenticate admin with cinder endpoint
-        self.cinder = self.authenticate_cinder_admin(username='admin',
-                                                     password='openstack',
-                                                     tenant='admin')
+        self.cinder = u.authenticate_cinder_admin(self.keystone_sentry,
+                                                  username='admin',
+                                                  password='openstack',
+                                                  tenant='admin')
+
         # Authenticate admin with glance endpoint
         self.glance = u.authenticate_glance_admin(self.keystone)
 
-        u.log.debug('openstack rel: {}'.format(self._get_openstack_release()))
-        # Wait for relations to settle
-        sleep(120)
-
-    def authenticate_cinder_admin(self, username, password, tenant):
-        """Authenticates admin user with cinder."""
-        # NOTE(beisner): need to move to charmhelpers, and adjust calls here.
-        # Probably useful on other charm tests.
-        service_ip = \
-            self.keystone_sentry.relation('shared-db',
-                                          'mysql:shared-db')['private-address']
-        ept = "http://{}:5000/v2.0".format(service_ip.strip().decode('utf-8'))
-        return cinder_client.Client(username, password, tenant, ept)
-
-    def force_list(self, obj):
-        '''Determine the object type and return a list.  Some Openstack
-           component API list methods return generators, some return lists.
-           Where obj is cinder.volumes, cinder.volume_snapshots, glance.images,
-           or other Openstack object with a list method.'''
-        # NOTE(beisner): need to move to charmhelpers, and adjust calls here.
-
-        # NOTE(beisner):  Beware - glance's list method returns a generator,
-        # and cinder's list method returns a list!
-        if isinstance(obj.list(), types.ListType):
-            return obj.list()
-        elif isinstance(obj.list(), types.GeneratorType):
-            return list(obj.list())
-        else:
-            u.log.debug('unhandled object type: {}'.format(type(obj.list())))
-            return False
-
-    def delete_all_objs(self, obj, item_desc='object', max_wait=60):
-        '''Delete all objects from openstack component, such as all volumes,
-           all images or all snapshots.  Waits and confirms deletion.'''
-        # NOTE(beisner): need to move to charmhelpers, and adjust calls here.
-        # Probably useful on other charm tests.
-
-        # Get list of objects to delete
-        obj_list = self.force_list(obj)
-        if obj_list is False:
-            return '{} list failed'.format(item_desc)
-
-        if len(obj_list) == 0:
-            u.log.debug('no {}(s) to delete'.format(item_desc))
-            return None
-
-        # Delete objects
-        for obj_this in obj_list:
-            u.log.debug('deleting {}: {}'.format(item_desc, obj_this.id))
+    def _extend_cinder_volume(self, vol_id, new_size=2):
+        """Extend an existing cinder volume size.
+
+        :param vol_id: existing cinder volume to extend
+        :param new_size: new size in gigabytes
+        :returns: None if successful; Failure message otherwise
+        """
+        # Extend existing volume size
+        try:
+            self.cinder.volumes.extend(vol_id, new_size)
+            vol_size_org = self.cinder.volumes.get(vol_id).size
+        except Exception as e:
+            msg = 'Failed to extend volume: {}'.format(e)
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        # Confirm that the volume reaches available status.
+        ret = u.resource_reaches_status(self.cinder.volumes, vol_id,
+                                        expected_stat="available",
+                                        msg="Volume status wait")
+        if not ret:
+            msg = ('Cinder volume failed to reach expected state '
+                   'while extending.')
+            return ret
+
+        # Validate volume size and status
+        u.log.debug('Validating volume attributes...')
+        vol_size_ext = self.cinder.volumes.get(vol_id).size
+        vol_stat = self.cinder.volumes.get(vol_id).status
+        msg_attr = ('Volume attributes - orig size:{} extended size:{} '
+                    'stat:{}'.format(vol_size_org, vol_size_ext, vol_stat))
+
+        if vol_size_ext > vol_size_org and vol_stat == 'available':
+            u.log.debug(msg_attr)
+        else:
+            msg = ('Volume validation failed, {}'.format(msg_attr))
+            return ret
+
+        return None
+
+    def _snapshot_cinder_volume(self, name='demo-snapshot', vol_id=None):
+        """Create a snapshot of an existing cinder volume.
+
+        :param name: display name to assign to snapshot
+        :param vol_id: existing cinder volume to snapshot
+        :returns: None if successful; Failure message otherwise
+        """
+        u.log.debug('Creating snapshot of volume ({})...'.format(vol_id))
+        # Create snapshot of an existing cinder volume
+        try:
+            snap_new = self.cinder.volume_snapshots.create(
+                volume_id=vol_id, display_name=name)
+            snap_id = snap_new.id
+        except Exception as e:
+            msg = 'Failed to snapshot the volume: {}'.format(e)
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        # Confirm that the volume reaches available status.
+        ret = u.resource_reaches_status(self.cinder.volume_snapshots,
+                                        snap_id,
+                                        expected_stat="available",
+                                        msg="Volume status wait")
+        if not ret:
+            msg = ('Cinder volume failed to reach expected state '
+                   'while snapshotting.')
+            return ret
+
+        # Validate snapshot
+        u.log.debug('Validating snapshot attributes...')
+        snap_name = self.cinder.volume_snapshots.get(snap_id).display_name
+        snap_stat = self.cinder.volume_snapshots.get(snap_id).status
+        snap_vol_id = self.cinder.volume_snapshots.get(snap_id).volume_id
+        msg_attr = ('Snapshot attributes - name:{} status:{} '
+                    'vol_id:{}'.format(snap_name, snap_stat, snap_vol_id))
+
+        if snap_name == name and snap_stat == 'available' \
+                and snap_vol_id == vol_id:
+            u.log.debug(msg_attr)
+        else:
+            msg = ('Snapshot validation failed, {}'.format(msg_attr))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        return snap_new
+
+    def _check_cinder_lvm(self):
+        """Inspect lvm on cinder unit, do basic validation against
+        cinder volumes and snapshots that exist."""
+        u.log.debug('Checking cinder volumes against lvm volumes...')
+        # Inspect
+        cmd = 'sudo lvs | grep cinder-volumes | awk \'{ print $1 }\''
+        output, code = self.cinder_sentry.run(cmd)
+        u.log.debug('{} `{}` returned '
+                    '{}'.format(self.cinder_sentry.info['unit_name'],
+                                cmd, code))
+        if code != 0:
+            return "command `{}` returned {}".format(cmd, str(code))
+
+        vol_list = self.cinder.volumes.list()
+        lv_id_list = output.split('\n')
+        lv_count = len(lv_id_list)
+        vol_count = len(vol_list)
+        snap_count = len(self.cinder.volume_snapshots.list())
+
+        # Expect cinder vol + snap count to match lvm log vol count
+        u.log.debug('vols:{} snaps:{} lvs:{}'.format(vol_count,
+                                                     snap_count,
+                                                     lv_count))
+        if (vol_count + snap_count) != len(lv_id_list):
+            msg = ('lvm volume count ({}) != cinder volume + snap count '
+                   '({})'.format(len(vol_list), len(lv_id_list)))
+            return msg
+
+        # Expect all cinder vol IDs to exist in the LVM volume list
+        for vol_this in vol_list:
             try:
-                obj_this.delete()
+                vol_id = vol_this.id
+                vol_name = vol_this.display_name
+                lv_id = 'volume-{}'.format(vol_id)
+                _index = lv_id_list.index(lv_id)
+                u.log.info('Volume ({}) correlates to lv '
+                           '{} ({})'.format(vol_name,
+                                            _index,
+                                            lv_id))
             except:
-                return '{} delete failed for {} with status {}'.format(
-                       item_desc, obj_this.id, obj_this.status)
-
-        # Wait for objects to disappear
-        obj_count = len(self.force_list(obj))
-        tries = 0
-        while obj_count != 0 and tries <= (max_wait/4):
-            u.log.debug('{} delete wait: {} {}'.format(item_desc,
-                                                       tries, obj_count))
-            sleep(4)
-            obj_count = len(self.force_list(obj))
-            tries += 1
-
-        if obj_count != 0:
-            return '{}(s) not deleted, {} remain.'.format(item_desc,
-                                                          obj_count)
-
-    def obj_is_status(self, obj, obj_id, stat='available',
-                      msg='openstack object status check', max_wait=120):
-        ''''Wait for an openstack object status to be as expected.
-            By default, expect an available status within 120s.  Useful
-            when confirming cinder volumes, snapshots, glance images, etc.
-            reach a certain state/status within a specified time.'''
-        # NOTE(beisner): need to move to charmhelpers, and adjust calls here.
-        # Probably useful on other charm tests.
-
-        obj_stat = obj.get(obj_id).status
-        tries = 0
-        while obj_stat != stat and tries < (max_wait/4):
-            u.log.debug(msg + ': {} [{}:{}] {}'.format(tries, obj_stat,
-                                                       stat, obj_id))
-            sleep(4)
-            obj_stat = obj.get(obj_id).status
-            tries += 1
-        if obj_stat == stat:
-            return True
-        else:
-            return False
-
-    def test_services(self):
-        '''Verify that the expected services are running on the
-           corresponding service units.'''
-        commands = {
-            self.cinder_sentry: ['status cinder-api',
-                                 'status cinder-scheduler',
-                                 'status cinder-volume'],
-            self.glance_sentry: ['status glance-registry',
-                                 'status glance-api'],
-            self.mysql_sentry: ['status mysql'],
-            self.keystone_sentry: ['status keystone'],
-            self.rabbitmq_sentry: ['sudo service rabbitmq-server status']
+                u.log.error('lvs output: {}'.format(output))
+                msg = ('Volume ID {} not found in '
+                       'LVM volume list.'.format(vol_this.id))
+                return msg
+
+        return None
+
+    def test_100_services(self):
+        """Verify that the expected services are running on the
+           corresponding service units."""
+        services = {
+            self.cinder_sentry: ['cinder-api',
+                                 'cinder-scheduler',
+                                 'cinder-volume'],
+            self.glance_sentry: ['glance-registry',
+                                 'glance-api'],
+            self.mysql_sentry: ['mysql'],
+            self.keystone_sentry: ['keystone'],
+            self.rabbitmq_sentry: ['rabbitmq-server']
         }
-        u.log.debug('commands: {}'.format(commands))
-        ret = u.validate_services(commands)
-        if ret:
-            amulet.raise_status(amulet.FAIL, msg=ret)
-
-    def test_service_catalog(self):
-        '''Verify that the service catalog endpoint data'''
+        ret = u.validate_services_by_name(services)
+        if ret:
+            amulet.raise_status(amulet.FAIL, msg=ret)
+
+    def test_110_users(self):
+        """Verify expected users."""
+        u.log.debug('Checking keystone users...')
+        user0 = {'name': 'cinder_cinderv2',
+                 'enabled': True,
+                 'tenantId': u.not_null,
+                 'id': u.not_null,
+                 'email': 'juju@localhost'}
+        user1 = {'name': 'admin',
+                 'enabled': True,
+                 'tenantId': u.not_null,
+                 'id': u.not_null,
+                 'email': 'juju@localhost'}
+        user2 = {'name': 'glance',
+                 'enabled': True,
+                 'tenantId': u.not_null,
+                 'id': u.not_null,
+                 'email': 'juju@localhost'}
+        expected = [user0, user1, user2]
+        actual = self.keystone.users.list()
+
+        ret = u.validate_user_data(expected, actual)
+        if ret:
+            amulet.raise_status(amulet.FAIL, msg=ret)
+
+    def test_112_service_catalog(self):
+        """Verify that the service catalog endpoint data"""
+        u.log.debug('Checking keystone service catalog...')
         endpoint_vol = {'adminURL': u.valid_url,
                         'region': 'RegionOne',
                         'publicURL': u.valid_url,
@@ -254,47 +323,66 @@
         if ret:
             amulet.raise_status(amulet.FAIL, msg=ret)
 
-    def test_cinder_glance_image_service_relation(self):
-        '''Verify the cinder:glance image-service relation data'''
+    def test_114_cinder_endpoint(self):
+        """Verify the cinder endpoint data."""
+        u.log.debug('Checking cinder endpoint...')
+        endpoints = self.keystone.endpoints.list()
+        admin_port = internal_port = public_port = '8776'
+        expected = {'id': u.not_null,
+                    'region': 'RegionOne',
+                    'adminurl': u.valid_url,
+                    'internalurl': u.valid_url,
+                    'publicurl': u.valid_url,
+                    'service_id': u.not_null}
+
+        ret = u.validate_endpoint_data(endpoints, admin_port, internal_port,
+                                       public_port, expected)
+        if ret:
+            amulet.raise_status(amulet.FAIL,
+                                msg='cinder endpoint: {}'.format(ret))
+
+    def test_202_cinder_glance_image_service_relation(self):
+        """Verify the cinder:glance image-service relation data"""
+        u.log.debug('Checking cinder:glance image-service relation data...')
         unit = self.cinder_sentry
         relation = ['image-service', 'glance:image-service']
         expected = {'private-address': u.valid_ip}
-        u.log.debug('')
         ret = u.validate_relation_data(unit, relation, expected)
         if ret:
             msg = u.relation_error('cinder image-service', ret)
             amulet.raise_status(amulet.FAIL, msg=msg)
 
-    def test_glance_cinder_image_service_relation(self):
-        '''Verify the glance:cinder image-service relation data'''
+    def test_203_glance_cinder_image_service_relation(self):
+        """Verify the glance:cinder image-service relation data"""
+        u.log.debug('Checking glance:cinder image-service relation data...')
         unit = self.glance_sentry
         relation = ['image-service', 'cinder:image-service']
         expected = {
             'private-address': u.valid_ip,
             'glance-api-server': u.valid_url
         }
-        u.log.debug('')
         ret = u.validate_relation_data(unit, relation, expected)
         if ret:
             msg = u.relation_error('glance image-service', ret)
             amulet.raise_status(amulet.FAIL, msg=msg)
 
-    def test_mysql_cinder_db_relation(self):
-        '''Verify the mysql:glance shared-db relation data'''
+    def test_204_mysql_cinder_db_relation(self):
+        """Verify the mysql:glance shared-db relation data"""
+        u.log.debug('Checking mysql:cinder db relation data...')
         unit = self.mysql_sentry
         relation = ['shared-db', 'cinder:shared-db']
         expected = {
             'private-address': u.valid_ip,
             'db_host': u.valid_ip
         }
-        u.log.debug('')
         ret = u.validate_relation_data(unit, relation, expected)
         if ret:
             msg = u.relation_error('mysql shared-db', ret)
             amulet.raise_status(amulet.FAIL, msg=msg)
 
-    def test_cinder_mysql_db_relation(self):
-        '''Verify the cinder:mysql shared-db relation data'''
+    def test_205_cinder_mysql_db_relation(self):
+        """Verify the cinder:mysql shared-db relation data"""
+        u.log.debug('Checking cinder:mysql db relation data...')
         unit = self.cinder_sentry
         relation = ['shared-db', 'mysql:shared-db']
         expected = {
@@ -303,14 +391,14 @@
             'username': 'cinder',
             'database': 'cinder'
         }
-        u.log.debug('')
         ret = u.validate_relation_data(unit, relation, expected)
         if ret:
             msg = u.relation_error('cinder shared-db', ret)
             amulet.raise_status(amulet.FAIL, msg=msg)
 
-    def test_keystone_cinder_id_relation(self):
-        '''Verify the keystone:cinder identity-service relation data'''
+    def test_206_keystone_cinder_id_relation(self):
+        """Verify the keystone:cinder identity-service relation data"""
+        u.log.debug('Checking keystone:cinder id relation data...')
         unit = self.keystone_sentry
         relation = ['identity-service',
                     'cinder:identity-service']
@@ -328,14 +416,14 @@
             'service_tenant_id': u.not_null,
             'service_host': u.valid_ip
         }
-        u.log.debug('')
         ret = u.validate_relation_data(unit, relation, expected)
         if ret:
             msg = u.relation_error('identity-service cinder', ret)
             amulet.raise_status(amulet.FAIL, msg=msg)
 
-    def test_cinder_keystone_id_relation(self):
-        '''Verify the cinder:keystone identity-service relation data'''
+    def test_207_cinder_keystone_id_relation(self):
+        """Verify the cinder:keystone identity-service relation data"""
+        u.log.debug('Checking cinder:keystone id relation data...')
         unit = self.cinder_sentry
         relation = ['identity-service',
                     'keystone:identity-service']
@@ -347,14 +435,14 @@
             'cinder_admin_url': u.valid_url,
             'private-address': u.valid_ip
         }
-        u.log.debug('')
         ret = u.validate_relation_data(unit, relation, expected)
         if ret:
             msg = u.relation_error('cinder identity-service', ret)
             amulet.raise_status(amulet.FAIL, msg=msg)
 
-    def test_rabbitmq_cinder_amqp_relation(self):
-        '''Verify the rabbitmq-server:cinder amqp relation data'''
+    def test_208_rabbitmq_cinder_amqp_relation(self):
+        """Verify the rabbitmq-server:cinder amqp relation data"""
+        u.log.debug('Checking rmq:cinder amqp relation data...')
         unit = self.rabbitmq_sentry
         relation = ['amqp', 'cinder:amqp']
         expected = {
@@ -362,14 +450,14 @@
             'password': u.not_null,
             'hostname': u.valid_ip
         }
-        u.log.debug('')
         ret = u.validate_relation_data(unit, relation, expected)
         if ret:
             msg = u.relation_error('amqp cinder', ret)
             amulet.raise_status(amulet.FAIL, msg=msg)
 
-    def test_cinder_rabbitmq_amqp_relation(self):
-        '''Verify the cinder:rabbitmq-server amqp relation data'''
+    def test_209_cinder_rabbitmq_amqp_relation(self):
+        """Verify the cinder:rabbitmq-server amqp relation data"""
+        u.log.debug('Checking cinder:rmq amqp relation data...')
         unit = self.cinder_sentry
         relation = ['amqp', 'rabbitmq-server:amqp']
         expected = {
@@ -377,71 +465,69 @@
             'vhost': 'openstack',
             'username': u.not_null
         }
-        u.log.debug('')
         ret = u.validate_relation_data(unit, relation, expected)
         if ret:
             msg = u.relation_error('cinder amqp', ret)
             amulet.raise_status(amulet.FAIL, msg=msg)
 
-    def test_cinder_default_config(self):
-        '''Verify default section configs in cinder.conf and
-           compare some of the parameters to relation data.'''
-        unit_ci = self.cinder_sentry
+    def test_300_cinder_config(self):
+        """Verify the data in the cinder.conf file."""
+        u.log.debug('Checking cinder config file data...')
+        unit = self.cinder_sentry
+        conf = '/etc/cinder/cinder.conf'
         unit_mq = self.rabbitmq_sentry
-        rel_ci_mq = unit_ci.relation('amqp', 'rabbitmq-server:amqp')
+        unit_ks = self.keystone_sentry
         rel_mq_ci = unit_mq.relation('amqp', 'cinder:amqp')
-        u.log.debug('actual ci:mq relation: {}'.format(rel_ci_mq))
-        u.log.debug('actual mq:ci relation: {}'.format(rel_mq_ci))
-        conf = '/etc/cinder/cinder.conf'
-        expected = {'use_syslog': 'False',
-                    'debug': 'False',
-                    'verbose': 'False',
-                    'iscsi_helper': 'tgtadm',
-                    'volume_group': 'cinder-volumes',
-                    'rabbit_userid': 'cinder',
-                    'rabbit_password': rel_mq_ci['password'],
-                    'rabbit_host': rel_mq_ci['hostname'],
-                    'auth_strategy': 'keystone',
-                    'volumes_dir': '/var/lib/cinder/volumes'}
-        section = 'DEFAULT'
-        u.log.debug('')
-        ret = u.validate_config_data(unit_ci, conf, section, expected)
-        if ret:
-            msg = 'cinder.conf default config error: {}'.format(ret)
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-    def test_cinder_auth_config(self):
-        '''Verify authtoken section config in cinder.conf or
-           api-paste.ini using glance/keystone relation data.'''
-        unit_ci = self.cinder_sentry
-        unit_ks = self.keystone_sentry
         rel_ks_ci = unit_ks.relation('identity-service',
                                      'cinder:identity-service')
-        u.log.debug('actual ks:ci relation: {}'.format(rel_ks_ci))
-
-        expected = {'admin_user': rel_ks_ci['service_username'],
-                    'admin_password': rel_ks_ci['service_password'],
-                    'admin_tenant_name': rel_ks_ci['service_tenant'],
-                    'auth_host': rel_ks_ci['auth_host']}
-
-        if self._get_openstack_release() >= self.precise_icehouse:
-            conf = '/etc/cinder/cinder.conf'
-            section = 'keystone_authtoken'
-            auth_uri = 'http://' + rel_ks_ci['auth_host'] + \
-                       ':' + rel_ks_ci['service_port'] + '/'
-            expected['auth_uri'] = auth_uri
+
+        auth_uri = 'http://' + rel_ks_ci['auth_host'] + \
+                   ':' + rel_ks_ci['service_port'] + '/'
+
+        expected = {
+            'DEFAULT': {
+                'use_syslog': 'False',
+                'debug': 'False',
+                'verbose': 'False',
+                'iscsi_helper': 'tgtadm',
+                'volume_group': 'cinder-volumes',
+                'auth_strategy': 'keystone',
+                'volumes_dir': '/var/lib/cinder/volumes'
+            },
+            'keystone_authtoken': {
+                'admin_user': rel_ks_ci['service_username'],
+                'admin_password': rel_ks_ci['service_password'],
+                'admin_tenant_name': rel_ks_ci['service_tenant'],
+                'auth_uri': auth_uri
+            }
+        }
+
+        expected_rmq = {
+            'rabbit_userid': 'cinder',
+            'rabbit_virtual_host': 'openstack',
+            'rabbit_password': rel_mq_ci['password'],
+            'rabbit_host': rel_mq_ci['hostname'],
+        }
+
+        if self._get_openstack_release() >= self.trusty_kilo:
+            # Kilo or later
+            expected['oslo_messaging_rabbit'] = expected_rmq
         else:
-            conf = '/etc/cinder/api-paste.ini'
-            section = 'filter:authtoken'
-
-        ret = u.validate_config_data(unit_ci, conf, section, expected)
-        if ret:
-            msg = "cinder auth config error: {}".format(ret)
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-    def test_cinder_logging_config(self):
-        ''' Inspect select sections and config pairs in logging.conf.'''
-        unit_ci = self.cinder_sentry
+            # Juno or earlier
+            expected['DEFAULT'].update(expected_rmq)
+            expected['keystone_authtoken']['auth_host'] = \
+                rel_ks_ci['auth_host']
+
+        for section, pairs in expected.iteritems():
+            ret = u.validate_config_data(unit, conf, section, pairs)
+            if ret:
+                message = "cinder config error: {}".format(ret)
+                amulet.raise_status(amulet.FAIL, msg=message)
+
+    def test_301_cinder_logging_config(self):
+        """Verify the data in the cinder logging conf file."""
+        u.log.debug('Checking cinder logging config file data...')
+        unit = self.cinder_sentry
         conf = '/etc/cinder/logging.conf'
 
         expected = {
@@ -460,347 +546,138 @@
         }
 
         for section, pairs in expected.iteritems():
-            ret = u.validate_config_data(unit_ci, conf, section, pairs)
+            ret = u.validate_config_data(unit, conf, section, pairs)
             if ret:
-                msg = "cinder logging config error: {}".format(ret)
-                amulet.raise_status(amulet.FAIL, msg=msg)
+                message = "cinder logging config error: {}".format(ret)
+                amulet.raise_status(amulet.FAIL, msg=message)
 
-    def test_cinder_rootwrap_config(self):
-        ''' Inspect select config pairs in rootwrap.conf. '''
-        unit_ci = self.cinder_sentry
+    def test_303_cinder_rootwrap_config(self):
+        """Inspect select config pairs in rootwrap.conf."""
+        u.log.debug('Checking cinder rootwrap config file data...')
+        unit = self.cinder_sentry
         conf = '/etc/cinder/rootwrap.conf'
-        expected = {'filters_path': '/etc/cinder/rootwrap.d,'
-                                    '/usr/share/cinder/rootwrap'}
         section = 'DEFAULT'
-
-        if self._get_openstack_release() >= self.precise_havana:
-            expected['use_syslog'] = 'False'
-            expected['exec_dirs'] = '/sbin,/usr/sbin,/bin,/usr/bin'
-
-        ret = u.validate_config_data(unit_ci, conf, section, expected)
+        expected = {
+            'filters_path': '/etc/cinder/rootwrap.d,'
+                            '/usr/share/cinder/rootwrap',
+            'use_syslog': 'False',
+        }
+
+        ret = u.validate_config_data(unit, conf, section, expected)
         if ret:
             msg = "cinder rootwrap config error: {}".format(ret)
             amulet.raise_status(amulet.FAIL, msg=msg)
 
-    def test_cinder_endpoint(self):
-        '''Verify the cinder endpoint data.'''
-        endpoints = self.keystone.endpoints.list()
-        admin_port = internal_port = public_port = '8776'
-        expected = {'id': u.not_null,
-                    'region': 'RegionOne',
-                    'adminurl': u.valid_url,
-                    'internalurl': u.valid_url,
-                    'publicurl': u.valid_url,
-                    'service_id': u.not_null}
-
-        ret = u.validate_endpoint_data(endpoints, admin_port, internal_port,
-                                       public_port, expected)
-        if ret:
-            amulet.raise_status(amulet.FAIL,
-                                msg='glance endpoint: {}'.format(ret))
-
-    def test_z_cinder_restart_on_config_change(self):
-        '''Verify cinder services are restarted when the config is changed.
-
-           Note(coreycb): The method name with the _z_ is a little odd
-           but it forces the test to run last.  It just makes things
-           easier because restarting services requires re-authorization.
-           '''
-        u.log.debug('making charm config change')
-        mtime = u.get_sentry_time(self.cinder_sentry)
-        self.d.configure('cinder', {'verbose': 'True', 'debug': 'True'})
-        if not u.validate_service_config_changed(self.cinder_sentry,
-                                                 mtime,
-                                                 'cinder-api',
-                                                 '/etc/cinder/cinder.conf'):
-            self.d.configure('cinder', {'verbose': 'False', 'debug': 'False'})
-            msg = "cinder-api service didn't restart after config change"
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-        if not u.validate_service_config_changed(self.cinder_sentry,
-                                                 mtime,
-                                                 'cinder-volume',
-                                                 '/etc/cinder/cinder.conf',
-                                                 sleep_time=0):
-            self.d.configure('cinder', {'verbose': 'False', 'debug': 'False'})
-            msg = "cinder-volume service didn't restart after conf change"
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-        u.log.debug('returning to original charm config')
-        self.d.configure('cinder', {'verbose': 'False', 'debug': 'False'})
-
-    def test_users(self):
-        '''Verify expected users.'''
-        user0 = {'name': 'cinder_cinderv2',
-                 'enabled': True,
-                 'tenantId': u.not_null,
-                 'id': u.not_null,
-                 'email': 'juju@localhost'}
-        user1 = {'name': 'admin',
-                 'enabled': True,
-                 'tenantId': u.not_null,
-                 'id': u.not_null,
-                 'email': 'juju@localhost'}
-        user2 = {'name': 'glance',
-                 'enabled': True,
-                 'tenantId': u.not_null,
-                 'id': u.not_null,
-                 'email': 'juju@localhost'}
-        expected = [user0, user1, user2]
-        actual = self.keystone.users.list()
-
-        ret = u.validate_user_data(expected, actual)
-        if ret:
-            amulet.raise_status(amulet.FAIL, msg=ret)
-
-    def test_000_delete_volumes_snapshots_images(self):
-        '''Delete all volumes, snapshots and images, if they exist,
-        as the first of the ordered tests.  Useful in re-run scenarios.'''
-        self.test_900_delete_all_snapshots()
-        self.test_900_glance_delete_all_images()
-        self.test_999_delete_all_volumes()
-
-    def test_100_create_and_extend_volume(self):
-        '''Add and confirm a new 1GB volume. In Havana and later,
-           extend that volume to 2GB.'''
-        # Create new volume
-        vol_new = self.cinder.volumes.create(display_name="demo-vol", size=1)
-        vol_id = vol_new.id
-
-        # Wait for volume status to be available
-        ret = self.obj_is_status(self.cinder.volumes, obj_id=vol_id,
-                                 stat='available',
-                                 msg='create vol status wait')
-        if not ret:
-            msg = 'volume create failed'
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-        # NOTE(beisner): Cinder extend is supported only in Havana or later
-        if self._get_openstack_release() < self.precise_havana:
-            u.log.debug('Skipping volume extend due to openstack release < H')
-            return
-
-        # Extend volume size
-        self.cinder.volumes.extend(vol_id, '2')
-
-        # Wait for extend
-        vol_size = self.cinder.volumes.get(vol_id).size
-        tries = 0
-        while vol_size != 2 and tries <= 15:
-            u.log.debug('volume extend size wait: {} {}'.format(tries,
-                                                                vol_id))
-            sleep(4)
-            vol_size = self.cinder.volumes.get(vol_id).size
-            tries += 1
-
-        if vol_size != 2:
-            msg = 'Failed to extend volume, size is {}'.format(vol_size)
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-    def test_100_glance_image_create(self):
-        '''Create new cirros glance image, to be referenced by
-           a cinder volume create tests in Havana or later.'''
-
-        # NOTE(beisner): Cinder create vol-from-img support for lvm and
-        # rbd(ceph) exists only in Havana or later
-        if self._get_openstack_release() < self.precise_havana:
-            u.log.debug('Skipping create glance img due to openstack rel < H')
-            return
-
-        # Create a new image
-        image_new = u.create_cirros_image(self.glance, 'cirros-image-1')
-
-        # Confirm image is created and has status of 'active'
-        if not image_new:
-            msg = 'image create failed'
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-    def test_200_clone_volume(self):
-        '''Create a new cinder volume, clone it to another cinder volume.'''
-        # Get volume object and ID
-        try:
-            vol = self.cinder.volumes.find(display_name="demo-vol")
-            vol_id = vol.id
-            vol_size = vol.size
-        except:
-            msg = ('Volume (demo-vol) not found.')
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-        if vol.status != 'available':
-            msg = ('volume status not == available: {}'.format(vol.status))
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-        # Create new clone volume from source volume
-        vol_clone = self.cinder.volumes.create(display_name="demo-vol-clone",
-                                               size=vol_size,
-                                               source_volid=vol_id)
-
-        ret = self.obj_is_status(self.cinder.volumes, obj_id=vol_clone.id,
-                                 stat='available',
-                                 msg='clone vol status wait')
-        if not ret:
-            msg = 'volume clone failed - from {}'.format(vol_id)
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-    def test_200_create_volume_from_glance_image(self):
-        '''Create new volume from glance cirros image (Havana and later),
-           check status and bootable flag.'''
-
-        # NOTE(beisner): Cinder create vol-from-img support for lvm and
-        # rbd(ceph) exists only in Havana or later
-        if self._get_openstack_release() < self.precise_havana:
-            u.log.debug('Skipping create vol from img, openstack rel < H')
-            return
-
-        # Get image object and id
-        expected_img_name = 'cirros-image-1'
-        img_list = list(self.glance.images.list())
-        img_count = len(img_list)
-
-        if img_count != 0:
-            # NOTE(beisner): glance api has no find method, presume 1st image
-            img_id = img_list[0].id
-        else:
-            msg = 'image not found'
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-        # Confirm image name
-        if img_list[0].name != expected_img_name:
-            msg = 'unexpected image name {}'.format(img_list[0].name)
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-        # Create new volume from glance image
-        vol_new = self.cinder.volumes.create(display_name="demo-vol-cirros",
-                                             size=1, imageRef=img_id)
-        vol_id = vol_new.id
-
-        # Wait for volume stat to be avail, check that it's flagged bootable
-        ret = self.obj_is_status(self.cinder.volumes, obj_id=vol_id,
-                                 stat='available',
-                                 msg='create vol from img status wait')
-        vol_boot = self.cinder.volumes.get(vol_id).bootable
-
-        if not ret or vol_boot != 'true':
-            vol_stat = self.cinder.volumes.get(vol_id).status
-            msg = ('vol create failed - from glance img:'
-                   ' id:{} stat:{} boot:{}'.format(vol_id,
-                                                   vol_stat,
-                                                   vol_boot))
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-    def test_300_cinder_create_snapshot(self):
-        '''Create a snapshot of a volume. Use a cirros-based volume where
-           supported (Havana and newer), and fall back to a vanilla
-           volume snapshot everywhere else.'''
-
-        if self._get_openstack_release() >= self.precise_havana:
-            vol_src_name = "demo-vol-cirros"
-        elif self._get_openstack_release() < self.precise_havana:
-            vol_src_name = "demo-vol"
-
-        u.log.debug('creating snapshot of volume: {}'.format(vol_src_name))
-
-        # Get volume object and id
-        try:
-            vol_src = self.cinder.volumes.find(display_name=vol_src_name)
-            vol_id = vol_src.id
-        except:
-            msg = ('volume not found while creating snapshot')
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-        if vol_src.status != 'available':
-            msg = ('volume status not == available: {}').format(vol_src.status)
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-        # Create new snapshot
-        snap_new = self.cinder.volume_snapshots.create(
-            volume_id=vol_id, display_name='demo-snapshot')
-        snap_id = snap_new.id
-
-        # Wait for snapshot status to become available
-        ret = self.obj_is_status(self.cinder.volume_snapshots, obj_id=snap_id,
-                                 stat='available',
-                                 msg='snapshot create status wait')
-        if not ret:
-            snap_stat = self.cinder.volume_snapshots.get(snap_id).status
-            msg = 'volume snapshot failed: {} {}'.format(snap_id,
-                                                         snap_stat)
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-    def test_310_create_volume_from_snapshot(self):
-        '''Create a new volume from a snapshot of a volume.'''
-        # Get snapshot object and ID
-        try:
-            snap = self.cinder.volume_snapshots.find(
-                display_name="demo-snapshot")
-            snap_id = snap.id
-            snap_size = snap.size
-        except:
-            msg = 'snapshot not found while creating volume'
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-        if snap.status != 'available':
-            msg = 'snapshot status not == available: {}'.format(snap.status)
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-        # Create new volume from snapshot
-        vol_new = self.cinder.volumes.create(
-            display_name="demo-vol-from-snap",
-            snapshot_id=snap_id,
-            size=snap_size)
-        vol_id = vol_new.id
-
-        # Wait for volume status to be == available
-        ret = self.obj_is_status(self.cinder.volumes, obj_id=vol_id,
-                                 stat='available',
-                                 msg='vol from snap create status wait')
-        if not ret:
-            vol_stat = self.cinder.volumes.get(vol_id).status
-            msg = 'volume create failed: {} {}'.format(vol_id,
-                                                       vol_stat)
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-    def test_900_confirm_lvm_volume_list(self):
-        '''Confirm cinder volume IDs with lvm logical volume IDs.
-           Expect a 1:1 relationship of lvm:cinder volumes.'''
-        commando = self.cinder_sentry.run('sudo lvs | grep cinder-volumes | '
-                                          'awk \'{ print $1 }\'')
-        vol_list = self.cinder.volumes.list()
-        lv_id_list = commando[0].split('\n')
-        vol_count = len(vol_list)
-        snap_count = len(self.cinder.volume_snapshots.list())
-
-        # Expect cinder vol + snap count to match lvm log vol count
-        if (vol_count + snap_count) != len(lv_id_list):
-            msg = ('lvm volume count ({}) != cinder volume + snap count '
-                   '({})'.format(len(vol_list), len(lv_id_list)))
-            amulet.raise_status(amulet.FAIL, msg=msg)
-
-        # Expect all cinder vol IDs to exist in the LVM volume list
-        for vol_this in vol_list:
-            try:
-                lv_id_list.index('volume-' + vol_this.id)
-            except:
-                msg = ('volume ID {} not found in '
-                       'LVM volume list.'.format(vol_this.id))
+    def test_400_cinder_api_connection(self):
+        """Simple api call to check service is up and responding"""
+        u.log.debug('Checking basic cinder api functionality...')
+        check = list(self.cinder.volumes.list())
+        u.log.debug('Cinder api check (volumes.list): {}'.format(check))
+        assert(check == [])
+
+    def test_401_create_delete_volume(self):
+        """Create a cinder volume and delete it."""
+        u.log.debug('Creating, checking and deleting cinder volume...')
+        vol_new = u.create_cinder_volume(self.cinder)
+        vol_id = vol_new.id
+        u.delete_resource(self.cinder.volumes, vol_id, msg="cinder volume")
+
+    def test_402_create_delete_volume_from_image(self):
+        """Create a cinder volume from a glance image, and delete it."""
+        u.log.debug('Creating, checking and deleting cinder volume'
+                    'from glance image...')
+        img_new = u.create_cirros_image(self.glance, "cirros-image-1")
+        img_id = img_new.id
+        vol_new = u.create_cinder_volume(self.cinder,
+                                         vol_name="demo-vol-cirros",
+                                         img_id=img_id)
+        vol_id = vol_new.id
+        u.delete_resource(self.glance.images, img_id, msg="glance image")
+        u.delete_resource(self.cinder.volumes, vol_id, msg="cinder volume")
+
+    def test_403_volume_snap_clone_extend_inspect(self):
+        """Create a cinder volume, clone it, extend its size, create a
+        snapshot of the volume, create a volume from a snapshot, check
+        status of each, inspect underlying lvm, then delete the resources."""
+        u.log.debug('Creating, snapshotting, cloning, extending a '
+                    'cinder volume...')
+        vols = []
+
+        # Create a 1GB volume
+        vol_new = u.create_cinder_volume(self.cinder, vol_size=1)
+        vols.append(vol_new)
+        vol_id = vol_new.id
+
+        # Snapshot the volume
+        snap = self._snapshot_cinder_volume(vol_id=vol_id)
+        snap_id = snap.id
+
+        # Create a volume from the snapshot
+        vol_from_snap = u.create_cinder_volume(self.cinder,
+                                               vol_name="demo-vol-from-snap",
+                                               snap_id=snap_id)
+        vols.append(vol_from_snap)
+
+        # Clone an existing volume
+        vol_clone = u.create_cinder_volume(self.cinder,
+                                           vol_name="demo-vol-clone",
+                                           src_vol_id=vol_id)
+        vols.append(vol_clone)
+        vol_clone_id = vol_clone.id
+
+        # Extend the cloned volume and confirm new size
+        ret = self._extend_cinder_volume(vol_clone_id, new_size=2)
+        if ret:
+            amulet.raise_status(amulet.FAIL, msg=ret)
+
+        # Inspect logical volumes (lvm) on cinder unit
+        ret = self._check_cinder_lvm()
+        if ret:
+            amulet.raise_status(amulet.FAIL, msg=ret)
+
+        # Cleanup
+        u.log.debug('Deleting snapshot {}...'.format(snap_id))
+        u.delete_resource(self.cinder.volume_snapshots,
+                          snap_id, msg="cinder volume")
+
+        for vol in vols:
+            u.log.debug('Deleting volume {}...'.format(vol.id))
+            u.delete_resource(self.cinder.volumes, vol.id, msg="cinder volume")
+
+    def test_900_restart_on_config_change(self):
+        """Verify that the specified services are restarted when the
+        config is changed."""
+
+        sentry = self.cinder_sentry
+        juju_service = 'cinder'
+
+        # Expected default and alternate values
+        set_default = {'debug': 'False'}
+        set_alternate = {'debug': 'True'}
+
+        # Config file affected by juju set config change
+        conf_file = '/etc/cinder/cinder.conf'
+
+        # Services which are expected to restart upon config change
+        services = [
+            'cinder-api',
+            'cinder-scheduler',
+            'cinder-volume'
+        ]
+
+        # Make config change, check for service restarts
+        u.log.debug('Making config change on {}...'.format(juju_service))
+        self.d.configure(juju_service, set_alternate)
+
+        sleep_time = 40
+        for s in services:
+            u.log.debug("Checking that service restarted: {}".format(s))
+            if not u.service_restarted(sentry, s,
+                                       conf_file, sleep_time=sleep_time,
+                                       pgrep_full=True):
+                self.d.configure(juju_service, set_default)
+                msg = "service {} didn't restart after config change".format(s)
                 amulet.raise_status(amulet.FAIL, msg=msg)
-
-    def test_900_glance_delete_all_images(self):
-        '''Delete all glance images and confirm deletion.'''
-        ret = self.delete_all_objs(self.glance.images, item_desc='image')
-        if ret:
-            amulet.raise_status(amulet.FAIL, msg=ret)
-
-    def test_900_delete_all_snapshots(self):
-        '''Delete all cinder volume snapshots and confirm deletion.'''
-        ret = self.delete_all_objs(self.cinder.volume_snapshots,
-                                   item_desc='snapshot')
-        if ret:
-            amulet.raise_status(amulet.FAIL, msg=ret)
-
-    def test_999_delete_all_volumes(self):
-        '''Delete all cinder volumes and confirm deletion,
-           as the last of the ordered tests.'''
-        ret = self.delete_all_objs(self.cinder.volumes, item_desc='volume')
-        if ret:
-            amulet.raise_status(amulet.FAIL, msg=ret)
+            sleep_time = 0
+
+        self.d.configure(juju_service, set_default)

=== modified file 'tests/charmhelpers/contrib/amulet/utils.py'
--- tests/charmhelpers/contrib/amulet/utils.py	2015-06-19 16:29:27 +0000
+++ tests/charmhelpers/contrib/amulet/utils.py	2015-07-01 14:48:09 +0000
@@ -14,6 +14,7 @@
 # You should have received a copy of the GNU Lesser General Public License
 # along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
 
+import amulet
 import ConfigParser
 import distro_info
 import io
@@ -173,6 +174,11 @@
 
            Verify that the specified section of the config file contains
            the expected option key:value pairs.
+
+           Compare expected dictionary data vs actual dictionary data.
+           The values in the 'expected' dictionary can be strings, bools, ints,
+           longs, or can be a function that evaluates a variable and returns a
+           bool.
            """
         self.log.debug('Validating config file data ({} in {} on {})'
                        '...'.format(section, config_file,
@@ -185,9 +191,20 @@
         for k in expected.keys():
             if not config.has_option(section, k):
                 return "section [{}] is missing option {}".format(section, k)
-            if config.get(section, k) != expected[k]:
+
+            actual = config.get(section, k)
+            v = expected[k]
+            if (isinstance(v, six.string_types) or
+                    isinstance(v, bool) or
+                    isinstance(v, six.integer_types)):
+                # handle explicit values
+                if actual != v:
+                    return "section [{}] {}:{} != expected {}:{}".format(
+                           section, k, actual, k, expected[k])
+            # handle function pointers, such as not_null or valid_ip
+            elif not v(actual):
                 return "section [{}] {}:{} != expected {}:{}".format(
-                       section, k, config.get(section, k), k, expected[k])
+                       section, k, actual, k, expected[k])
         return None
 
     def _validate_dict_data(self, expected, actual):
@@ -195,7 +212,7 @@
 
            Compare expected dictionary data vs actual dictionary data.
            The values in the 'expected' dictionary can be strings, bools, ints,
-           longs, or can be a function that evaluate a variable and returns a
+           longs, or can be a function that evaluates a variable and returns a
            bool.
            """
         self.log.debug('actual: {}'.format(repr(actual)))
@@ -206,8 +223,10 @@
                 if (isinstance(v, six.string_types) or
                         isinstance(v, bool) or
                         isinstance(v, six.integer_types)):
+                    # handle explicit values
                     if v != actual[k]:
                         return "{}:{}".format(k, actual[k])
+                # handle function pointers, such as not_null or valid_ip
                 elif not v(actual[k]):
                     return "{}:{}".format(k, actual[k])
             else:
@@ -406,3 +425,109 @@
         """Convert a relative file path to a file URL."""
         _abs_path = os.path.abspath(file_rel_path)
         return urlparse.urlparse(_abs_path, scheme='file').geturl()
+
+    def check_commands_on_units(self, commands, sentry_units):
+        """Check that all commands in a list exit zero on all
+        sentry units in a list.
+
+        :param commands:  list of bash commands
+        :param sentry_units:  list of sentry unit pointers
+        :returns: None if successful; Failure message otherwise
+        """
+        self.log.debug('Checking exit codes for {} commands on {} '
+                       'sentry units...'.format(len(commands),
+                                                len(sentry_units)))
+        for sentry_unit in sentry_units:
+            for cmd in commands:
+                output, code = sentry_unit.run(cmd)
+                if code == 0:
+                    self.log.debug('{} `{}` returned {} '
+                                   '(OK)'.format(sentry_unit.info['unit_name'],
+                                                 cmd, code))
+                else:
+                    return ('{} `{}` returned {} '
+                            '{}'.format(sentry_unit.info['unit_name'],
+                                        cmd, code, output))
+        return None
+
+    def get_process_id_list(self, sentry_unit, process_name):
+        """Get a list of process ID(s) from a single sentry juju unit
+        for a single process name.
+
+        :param sentry_unit: Pointer to amulet sentry instance (juju unit)
+        :param process_name: Process name
+        :returns: List of process IDs
+        """
+        cmd = 'pidof {}'.format(process_name)
+        output, code = sentry_unit.run(cmd)
+        if code != 0:
+            msg = ('{} `{}` returned {} '
+                   '{}'.format(sentry_unit.info['unit_name'],
+                               cmd, code, output))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+        return str(output).split()
+
+    def get_unit_process_ids(self, unit_processes):
+        """Construct a dict containing unit sentries, process names, and
+        process IDs."""
+        pid_dict = {}
+        for sentry_unit, process_list in unit_processes.iteritems():
+            pid_dict[sentry_unit] = {}
+            for process in process_list:
+                pids = self.get_process_id_list(sentry_unit, process)
+                pid_dict[sentry_unit].update({process: pids})
+        return pid_dict
+
+    def validate_unit_process_ids(self, expected, actual):
+        """Validate process id quantities for services on units."""
+        self.log.debug('Checking units for running processes...')
+        self.log.debug('Expected PIDs: {}'.format(expected))
+        self.log.debug('Actual PIDs: {}'.format(actual))
+
+        if len(actual) != len(expected):
+            return ('Unit count mismatch.  expected, actual: {}, '
+                    '{} '.format(len(expected), len(actual)))
+
+        for (e_sentry, e_proc_names) in expected.iteritems():
+            e_sentry_name = e_sentry.info['unit_name']
+            if e_sentry in actual.keys():
+                a_proc_names = actual[e_sentry]
+            else:
+                return ('Expected sentry ({}) not found in actual dict data.'
+                        '{}'.format(e_sentry_name, e_sentry))
+
+            if len(e_proc_names.keys()) != len(a_proc_names.keys()):
+                return ('Process name count mismatch.  expected, actual: {}, '
+                        '{}'.format(len(expected), len(actual)))
+
+            for (e_proc_name, e_pids_length), (a_proc_name, a_pids) in \
+                    zip(e_proc_names.items(), a_proc_names.items()):
+                if e_proc_name != a_proc_name:
+                    return ('Process name mismatch.  expected, actual: {}, '
+                            '{}'.format(e_proc_name, a_proc_name))
+
+                a_pids_length = len(a_pids)
+                if e_pids_length != a_pids_length:
+                    return ('PID count mismatch. {} ({}) expected, actual: '
+                            '{}, {} ({})'.format(e_sentry_name, e_proc_name,
+                                                 e_pids_length, a_pids_length,
+                                                 a_pids))
+                else:
+                    self.log.debug('PID check OK: {} {} {}: '
+                                   '{}'.format(e_sentry_name, e_proc_name,
+                                               e_pids_length, a_pids))
+        return None
+
+    def validate_list_of_identical_dicts(self, list_of_dicts):
+        """Check that all dicts within a list are identical."""
+        hashes = []
+        for _dict in list_of_dicts:
+            hashes.append(hash(frozenset(_dict.items())))
+
+        self.log.debug('Hashes: {}'.format(hashes))
+        if len(set(hashes)) == 1:
+            self.log.debug('Dicts within list are identical')
+        else:
+            return 'Dicts within list are not identical'
+
+        return None

=== modified file 'tests/charmhelpers/contrib/openstack/amulet/deployment.py'
--- tests/charmhelpers/contrib/openstack/amulet/deployment.py	2015-06-19 16:29:27 +0000
+++ tests/charmhelpers/contrib/openstack/amulet/deployment.py	2015-07-01 14:48:09 +0000
@@ -79,9 +79,9 @@
         services.append(this_service)
         use_source = ['mysql', 'mongodb', 'rabbitmq-server', 'ceph',
                       'ceph-osd', 'ceph-radosgw']
-        # Openstack subordinate charms do not expose an origin option as that
-        # is controlled by the principle
-        ignore = ['neutron-openvswitch']
+        # Most OpenStack subordinate charms do not expose an origin option
+        # as that is controlled by the principle.
+        ignore = ['cinder-ceph', 'hacluster', 'neutron-openvswitch']
 
         if self.openstack:
             for svc in services:
@@ -148,3 +148,36 @@
             return os_origin.split('%s-' % self.series)[1].split('/')[0]
         else:
             return releases[self.series]
+
+    def get_ceph_expected_pools(self, radosgw=False):
+        """Return a list of expected ceph pools in a ceph + cinder + glance
+        test scenario, based on OpenStack release and whether ceph radosgw
+        is flagged as present or not."""
+
+        if self._get_openstack_release() >= self.trusty_kilo:
+            # Kilo or later
+            pools = [
+                'rbd',
+                'cinder',
+                'glance'
+            ]
+        else:
+            # Juno or earlier
+            pools = [
+                'data',
+                'metadata',
+                'rbd',
+                'cinder',
+                'glance'
+            ]
+
+        if radosgw:
+            pools.extend([
+                '.rgw.root',
+                '.rgw.control',
+                '.rgw',
+                '.rgw.gc',
+                '.users.uid'
+            ])
+
+        return pools

=== modified file 'tests/charmhelpers/contrib/openstack/amulet/utils.py'
--- tests/charmhelpers/contrib/openstack/amulet/utils.py	2015-06-19 16:29:27 +0000
+++ tests/charmhelpers/contrib/openstack/amulet/utils.py	2015-07-01 14:48:09 +0000
@@ -14,16 +14,20 @@
 # You should have received a copy of the GNU Lesser General Public License
 # along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
 
+import amulet
+import json
 import logging
 import os
 import six
 import time
 import urllib
 
+import cinderclient.v1.client as cinder_client
 import glanceclient.v1.client as glance_client
 import heatclient.v1.client as heat_client
 import keystoneclient.v2_0 as keystone_client
 import novaclient.v1_1.client as nova_client
+import swiftclient
 
 from charmhelpers.contrib.amulet.utils import (
     AmuletUtils
@@ -171,6 +175,16 @@
         self.log.debug('Checking if tenant exists ({})...'.format(tenant))
         return tenant in [t.name for t in keystone.tenants.list()]
 
+    def authenticate_cinder_admin(self, keystone_sentry, username,
+                                  password, tenant):
+        """Authenticates admin user with cinder."""
+        # NOTE(beisner): cinder python client doesn't accept tokens.
+        service_ip = \
+            keystone_sentry.relation('shared-db',
+                                     'mysql:shared-db')['private-address']
+        ept = "http://{}:5000/v2.0".format(service_ip.strip().decode('utf-8'))
+        return cinder_client.Client(username, password, tenant, ept)
+
     def authenticate_keystone_admin(self, keystone_sentry, user, password,
                                     tenant):
         """Authenticates admin user with the keystone admin endpoint."""
@@ -212,9 +226,29 @@
         return nova_client.Client(username=user, api_key=password,
                                   project_id=tenant, auth_url=ep)
 
+    def authenticate_swift_user(self, keystone, user, password, tenant):
+        """Authenticates a regular user with swift api."""
+        self.log.debug('Authenticating swift user ({})...'.format(user))
+        ep = keystone.service_catalog.url_for(service_type='identity',
+                                              endpoint_type='publicURL')
+        return swiftclient.Connection(authurl=ep,
+                                      user=user,
+                                      key=password,
+                                      tenant_name=tenant,
+                                      auth_version='2.0')
+
     def create_cirros_image(self, glance, image_name):
-        """Download the latest cirros image and upload it to glance."""
-        self.log.debug('Creating glance image ({})...'.format(image_name))
+        """Download the latest cirros image and upload it to glance,
+        validate and return a resource pointer.
+
+        :param glance: pointer to authenticated glance connection
+        :param image_name: display name for new image
+        :returns: glance image pointer
+        """
+        self.log.debug('Creating glance cirros image '
+                       '({})...'.format(image_name))
+
+        # Download cirros image
         http_proxy = os.getenv('AMULET_HTTP_PROXY')
         self.log.debug('AMULET_HTTP_PROXY: {}'.format(http_proxy))
         if http_proxy:
@@ -223,33 +257,51 @@
         else:
             opener = urllib.FancyURLopener()
 
-        f = opener.open("http://download.cirros-cloud.net/version/released")
+        f = opener.open('http://download.cirros-cloud.net/version/released')
         version = f.read().strip()
-        cirros_img = "cirros-{}-x86_64-disk.img".format(version)
+        cirros_img = 'cirros-{}-x86_64-disk.img'.format(version)
         local_path = os.path.join('tests', cirros_img)
 
         if not os.path.exists(local_path):
-            cirros_url = "http://{}/{}/{}".format("download.cirros-cloud.net",
+            cirros_url = 'http://{}/{}/{}'.format('download.cirros-cloud.net',
                                                   version, cirros_img)
             opener.retrieve(cirros_url, local_path)
         f.close()
 
+        # Create glance image
         with open(local_path) as f:
             image = glance.images.create(name=image_name, is_public=True,
                                          disk_format='qcow2',
                                          container_format='bare', data=f)
-        count = 1
-        status = image.status
-        while status != 'active' and count < 10:
-            time.sleep(3)
-            image = glance.images.get(image.id)
-            status = image.status
-            self.log.debug('image status: {}'.format(status))
-            count += 1
-
-        if status != 'active':
-            self.log.error('image creation timed out')
-            return None
+
+        # Wait for image to reach active status
+        img_id = image.id
+        ret = self.resource_reaches_status(glance.images, img_id,
+                                           expected_stat='active',
+                                           msg='Image status wait')
+        if not ret:
+            msg = 'Glance image failed to reach expected state.'
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        # Re-validate new image
+        self.log.debug('Validating image attributes...')
+        val_img_name = glance.images.get(img_id).name
+        val_img_stat = glance.images.get(img_id).status
+        val_img_pub = glance.images.get(img_id).is_public
+        val_img_cfmt = glance.images.get(img_id).container_format
+        val_img_dfmt = glance.images.get(img_id).disk_format
+        msg_attr = ('Image attributes - name:{} public:{} id:{} stat:{} '
+                    'container fmt:{} disk fmt:{}'.format(
+                        val_img_name, val_img_pub, img_id,
+                        val_img_stat, val_img_cfmt, val_img_dfmt))
+
+        if val_img_name == image_name and val_img_stat == 'active' \
+                and val_img_pub is True and val_img_cfmt == 'bare' \
+                and val_img_dfmt == 'qcow2':
+            self.log.debug(msg_attr)
+        else:
+            msg = ('Volume validation failed, {}'.format(msg_attr))
+            amulet.raise_status(amulet.FAIL, msg=msg)
 
         return image
 
@@ -260,22 +312,7 @@
         self.log.warn('/!\\ DEPRECATION WARNING:  use '
                       'delete_resource instead of delete_image.')
         self.log.debug('Deleting glance image ({})...'.format(image))
-        num_before = len(list(glance.images.list()))
-        glance.images.delete(image)
-
-        count = 1
-        num_after = len(list(glance.images.list()))
-        while num_after != (num_before - 1) and count < 10:
-            time.sleep(3)
-            num_after = len(list(glance.images.list()))
-            self.log.debug('number of images: {}'.format(num_after))
-            count += 1
-
-        if num_after != (num_before - 1):
-            self.log.error('image deletion timed out')
-            return False
-
-        return True
+        return self.delete_resource(glance.images, image, msg='glance image')
 
     def create_instance(self, nova, image_name, instance_name, flavor):
         """Create the specified instance."""
@@ -308,22 +345,8 @@
         self.log.warn('/!\\ DEPRECATION WARNING:  use '
                       'delete_resource instead of delete_instance.')
         self.log.debug('Deleting instance ({})...'.format(instance))
-        num_before = len(list(nova.servers.list()))
-        nova.servers.delete(instance)
-
-        count = 1
-        num_after = len(list(nova.servers.list()))
-        while num_after != (num_before - 1) and count < 10:
-            time.sleep(3)
-            num_after = len(list(nova.servers.list()))
-            self.log.debug('number of instances: {}'.format(num_after))
-            count += 1
-
-        if num_after != (num_before - 1):
-            self.log.error('instance deletion timed out')
-            return False
-
-        return True
+        return self.delete_resource(nova.servers, instance,
+                                    msg='nova instance')
 
     def create_or_get_keypair(self, nova, keypair_name="testkey"):
         """Create a new keypair, or return pointer if it already exists."""
@@ -339,6 +362,88 @@
         _keypair = nova.keypairs.create(name=keypair_name)
         return _keypair
 
+    def create_cinder_volume(self, cinder, vol_name="demo-vol", vol_size=1,
+                             img_id=None, src_vol_id=None, snap_id=None):
+        """Create cinder volume, optionally from a glance image, OR
+        optionally as a clone of an existing volume, OR optionally
+        from a snapshot.  Wait for the new volume status to reach
+        the expected status, validate and return a resource pointer.
+
+        :param vol_name: cinder volume display name
+        :param vol_size: size in gigabytes
+        :param img_id: optional glance image id
+        :param src_vol_id: optional source volume id to clone
+        :param snap_id: optional snapshot id to use
+        :returns: cinder volume pointer
+        """
+        # Handle parameter input and avoid impossible combinations
+        if img_id and not src_vol_id and not snap_id:
+            # Create volume from image
+            self.log.debug('Creating cinder volume from glance image...')
+            bootable = 'true'
+        elif src_vol_id and not img_id and not snap_id:
+            # Clone an existing volume
+            self.log.debug('Cloning cinder volume...')
+            bootable = cinder.volumes.get(src_vol_id).bootable
+        elif snap_id and not src_vol_id and not img_id:
+            # Create volume from snapshot
+            self.log.debug('Creating cinder volume from snapshot...')
+            snap = cinder.volume_snapshots.find(id=snap_id)
+            vol_size = snap.size
+            snap_vol_id = cinder.volume_snapshots.get(snap_id).volume_id
+            bootable = cinder.volumes.get(snap_vol_id).bootable
+        elif not img_id and not src_vol_id and not snap_id:
+            # Create volume
+            self.log.debug('Creating cinder volume...')
+            bootable = 'false'
+        else:
+            # Impossible combination of parameters
+            msg = ('Invalid method use - name:{} size:{} img_id:{} '
+                   'src_vol_id:{} snap_id:{}'.format(vol_name, vol_size,
+                                                     img_id, src_vol_id,
+                                                     snap_id))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        # Create new volume
+        try:
+            vol_new = cinder.volumes.create(display_name=vol_name,
+                                            imageRef=img_id,
+                                            size=vol_size,
+                                            source_volid=src_vol_id,
+                                            snapshot_id=snap_id)
+            vol_id = vol_new.id
+        except Exception as e:
+            msg = 'Failed to create volume: {}'.format(e)
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        # Wait for volume to reach available status
+        ret = self.resource_reaches_status(cinder.volumes, vol_id,
+                                           expected_stat="available",
+                                           msg="Volume status wait")
+        if not ret:
+            msg = 'Cinder volume failed to reach expected state.'
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        # Re-validate new volume
+        self.log.debug('Validating volume attributes...')
+        val_vol_name = cinder.volumes.get(vol_id).display_name
+        val_vol_boot = cinder.volumes.get(vol_id).bootable
+        val_vol_stat = cinder.volumes.get(vol_id).status
+        val_vol_size = cinder.volumes.get(vol_id).size
+        msg_attr = ('Volume attributes - name:{} id:{} stat:{} boot:'
+                    '{} size:{}'.format(val_vol_name, vol_id,
+                                        val_vol_stat, val_vol_boot,
+                                        val_vol_size))
+
+        if val_vol_boot == bootable and val_vol_stat == 'available' \
+                and val_vol_name == vol_name and val_vol_size == vol_size:
+            self.log.debug(msg_attr)
+        else:
+            msg = ('Volume validation failed, {}'.format(msg_attr))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        return vol_new
+
     def delete_resource(self, resource, resource_id,
                         msg="resource", max_wait=120):
         """Delete one openstack resource, such as one instance, keypair,
@@ -350,6 +455,8 @@
         :param max_wait: maximum wait time in seconds
         :returns: True if successful, otherwise False
         """
+        self.log.debug('Deleting OpenStack resource '
+                       '{} ({})'.format(resource_id, msg))
         num_before = len(list(resource.list()))
         resource.delete(resource_id)
 
@@ -411,3 +518,87 @@
             self.log.debug('{} never reached expected status: '
                            '{}'.format(resource_id, expected_stat))
             return False
+
+    def get_ceph_osd_id_cmd(self, index):
+        """Produce a shell command that will return a ceph-osd id."""
+        return ("`initctl list | grep 'ceph-osd ' | "
+                "awk 'NR=={} {{ print $2 }}' | "
+                "grep -o '[0-9]*'`".format(index + 1))
+
+    def get_ceph_pools(self, sentry_unit):
+        """Return a dict of ceph pools from a single ceph unit, with
+        pool name as keys, pool id as vals."""
+        pools = {}
+        cmd = 'sudo ceph osd lspools'
+        output, code = sentry_unit.run(cmd)
+        if code != 0:
+            msg = ('{} `{}` returned {} '
+                   '{}'.format(sentry_unit.info['unit_name'],
+                               cmd, code, output))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+
+        # Example output: 0 data,1 metadata,2 rbd,3 cinder,4 glance,
+        for pool in str(output).split(','):
+            pool_id_name = pool.split(' ')
+            if len(pool_id_name) == 2:
+                pool_id = pool_id_name[0]
+                pool_name = pool_id_name[1]
+                pools[pool_name] = int(pool_id)
+
+        self.log.debug('Pools on {}: {}'.format(sentry_unit.info['unit_name'],
+                                                pools))
+        return pools
+
+    def get_ceph_df(self, sentry_unit):
+        """Return dict of ceph df json output, including ceph pool state.
+
+        :param sentry_unit: Pointer to amulet sentry instance (juju unit)
+        :returns: Dict of ceph df output
+        """
+        cmd = 'sudo ceph df --format=json'
+        output, code = sentry_unit.run(cmd)
+        if code != 0:
+            msg = ('{} `{}` returned {} '
+                   '{}'.format(sentry_unit.info['unit_name'],
+                               cmd, code, output))
+            amulet.raise_status(amulet.FAIL, msg=msg)
+        return json.loads(output)
+
+    def get_ceph_pool_sample(self, sentry_unit, pool_id=0):
+        """Take a sample of attributes of a ceph pool, returning ceph
+        pool name, object count and disk space used for the specified
+        pool ID number.
+
+        :param sentry_unit: Pointer to amulet sentry instance (juju unit)
+        :param pool_id: Ceph pool ID
+        :returns: List of pool name, object count, kb disk space used
+        """
+        df = self.get_ceph_df(sentry_unit)
+        pool_name = df['pools'][pool_id]['name']
+        obj_count = df['pools'][pool_id]['stats']['objects']
+        kb_used = df['pools'][pool_id]['stats']['kb_used']
+        self.log.debug('Ceph {} pool (ID {}): {} objects, '
+                       '{} kb used'.format(pool_name, pool_id,
+                                           obj_count, kb_used))
+        return pool_name, obj_count, kb_used
+
+    def validate_ceph_pool_samples(self, samples, sample_type="resource pool"):
+        """Validate ceph pool samples taken over time, such as pool
+        object counts or pool kb used, before adding, after adding, and
+        after deleting items which affect those pool attributes.  The
+        2nd element is expected to be greater than the 1st; 3rd is expected
+        to be less than the 2nd.
+
+        :param samples: List containing 3 data samples
+        :param sample_type: String for logging and usage context
+        :returns: None if successful, Failure message otherwise
+        """
+        original, created, deleted = range(3)
+        if samples[created] <= samples[original] or \
+                samples[deleted] >= samples[created]:
+            return ('Ceph {} samples ({}) '
+                    'unexpected.'.format(sample_type, samples))
+        else:
+            self.log.debug('Ceph {} samples (OK): '
+                           '{}'.format(sample_type, samples))
+            return None

=== added file 'tests/tests.yaml'
--- tests/tests.yaml	1970-01-01 00:00:00 +0000
+++ tests/tests.yaml	2015-07-01 14:48:09 +0000
@@ -0,0 +1,18 @@
+bootstrap: true
+reset: true
+virtualenv: true
+makefile:
+    - lint
+    - test
+sources:
+    - ppa:juju/stable
+packages:
+    - amulet
+    - python-amulet
+    - python-cinderclient
+    - python-distro-info
+    - python-glanceclient
+    - python-heatclient
+    - python-keystoneclient
+    - python-novaclient
+    - python-swiftclient