Merge lp:~james-page/charms/trusty/ceph-osd/status into lp:~openstack-charmers-archive/charms/trusty/ceph-osd/next

[James Page]

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #10667 ceph-osd-next for james-page mp273634
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/12701797/
Build: http://10.245.162.77:8080/job/charm_unit_test/10667/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #11480 ceph-osd-next for james-page mp273634
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/12702222/
Build: http://10.245.162.77:8080/job/charm_lint_check/11480/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7177 ceph-osd-next for james-page mp273634
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12703539/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7177/

[Ryan Beisner (1chb1n)]

FWIW, I've seen a ton of this in the past ~36hrs across various charm tests. One of the ceph units failed in the install hook due to:

2015-10-07 11:26:36 INFO install WARNING: The following packages cannot be authenticated!
2015-10-07 11:26:36 INFO install libjs-jquery python-werkzeug python-markupsafe python-jinja2
2015-10-07 11:26:36 INFO install python-itsdangerous python-flask libreadline5 xfsprogs libunwind8 libsnappy1
2015-10-07 11:26:36 INFO install libleveldb1 libopts25 btrfs-tools python-blinker python-pyinotify
2015-10-07 11:26:36 INFO install E: There are problems and -y was used without --force-yes

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #10673 ceph-osd-next for james-page mp273634
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/10673/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7184 ceph-osd-next for james-page mp273634
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/7184/

[David Ames (thedac)]

Lint test is failing.
Other than that this looks good.

[lp:~james-page/charms/trusty/ceph-osd/status]

54. By James Page on 2015-10-08

Tidy lint

[James Page (james-page)]

Lint tidied up.

review: Resubmit

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #11505 ceph-osd-next for james-page mp273634
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/11505/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #10698 ceph-osd-next for james-page mp273634
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/10698/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7210 ceph-osd-next for james-page mp273634
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/7210/

[Chris Holcombe (xfactor973)]

Looks good to me

review: Approve

[David Ames (thedac)]

Approved. Merging.

review: Approve

=== added file '.coveragerc'
--- .coveragerc	1970-01-01 00:00:00 +0000
+++ .coveragerc	2015-10-08 11:42:10 +0000
@@ -0,0 +1,7 @@
+[report]
+# Regexes for lines to exclude from consideration
+exclude_lines =
+    if __name__ == .__main__.:
+include=
+    hooks/hooks.py
+    hooks/ceph*.py

=== modified file '.pydevproject'
--- .pydevproject	2012-10-08 14:07:16 +0000
+++ .pydevproject	2015-10-08 11:42:10 +0000
@@ -4,5 +4,6 @@
 <pydev_property name="org.python.pydev.PYTHON_PROJECT_INTERPRETER">Default</pydev_property>
 <pydev_pathproperty name="org.python.pydev.PROJECT_SOURCE_PATH">
 <path>/ceph-osd/hooks</path>
+<path>/ceph-osd/unit_tests</path>
 </pydev_pathproperty>
 </pydev_project>

=== modified file 'Makefile'
--- Makefile	2015-07-02 15:08:10 +0000
+++ Makefile	2015-10-08 11:42:10 +0000
@@ -3,9 +3,14 @@
 
 lint:
 	@flake8 --exclude hooks/charmhelpers,tests/charmhelpers \
-        hooks tests unit_tests
+		hooks tests unit_tests
 	@charm proof
 
+test:
+	@# Bundletester expects unit tests here.
+	@echo Starting unit tests...
+	@$(PYTHON) /usr/bin/nosetests --nologcapture --with-coverage  unit_tests
+
 functional_test:
 	@echo Starting Amulet tests...
 	@juju test -v -p AMULET_HTTP_PROXY,AMULET_OS_VIP --timeout 2700
@@ -13,7 +18,7 @@
 bin/charm_helpers_sync.py:
 	@mkdir -p bin
 	@bzr cat lp:charm-helpers/tools/charm_helpers_sync/charm_helpers_sync.py \
-        > bin/charm_helpers_sync.py
+		> bin/charm_helpers_sync.py
 
 sync: bin/charm_helpers_sync.py
 	$(PYTHON) bin/charm_helpers_sync.py -c charm-helpers-hooks.yaml

=== modified file 'hooks/ceph.py'
--- hooks/ceph.py	2014-09-12 12:12:47 +0000
+++ hooks/ceph.py	2015-10-08 11:42:10 +0000
@@ -18,7 +18,8 @@
 )
 from charmhelpers.core.hookenv import (
     log,
-    ERROR, WARNING
+    ERROR, WARNING,
+    status_set,
 )
 from charmhelpers.contrib.storage.linux.utils import (
     zap_disk,
@@ -333,6 +334,7 @@
         log('Looks like {} is in use, skipping.'.format(dev))
         return
 
+    status_set('maintenance', 'Initializing device {}'.format(dev))
     cmd = ['ceph-disk-prepare']
     # Later versions of ceph support more options
     if cmp_pkgrevno('ceph', '0.48.3') >= 0:
@@ -382,3 +384,13 @@
 
 def filesystem_mounted(fs):
     return subprocess.call(['grep', '-wqs', fs, '/proc/mounts']) == 0
+
+
+def get_running_osds():
+    '''Returns a list of the pids of the current running OSD daemons'''
+    cmd = ['pgrep', 'ceph-osd']
+    try:
+        result = subprocess.check_output(cmd)
+        return result.split()
+    except subprocess.CalledProcessError:
+        return []

=== renamed file 'hooks/hooks.py' => 'hooks/ceph_hooks.py'
--- hooks/hooks.py	2015-09-22 13:35:49 +0000
+++ hooks/ceph_hooks.py	2015-10-08 11:42:10 +0000
@@ -22,7 +22,8 @@
     relation_get,
     Hooks,
     UnregisteredHookError,
-    service_name
+    service_name,
+    status_set,
 )
 from charmhelpers.core.host import (
     umount,
@@ -227,8 +228,34 @@
     nrpe_setup.write()
 
 
+def assess_status():
+    '''Assess status of current unit'''
+    # Check for mon relation
+    if len(relation_ids('mon')) < 1:
+        status_set('blocked', 'Missing relation: monitor')
+        return
+
+    # Check for monitors with presented addresses
+    # Check for bootstrap key presentation
+    monitors = get_mon_hosts()
+    if len(monitors) < 1 or not get_conf('osd_bootstrap_key'):
+        status_set('waiting', 'Incomplete relation: monitor')
+        return
+
+    # Check for OSD device creation parity i.e. at least some devices
+    # must have been presented and used for this charm to be operational
+    running_osds = ceph.get_running_osds()
+    if not running_osds:
+        status_set('blocked',
+                   'No block devices detected using current configuration')
+    else:
+        status_set('active',
+                   'Unit is ready ({} OSD)'.format(len(running_osds)))
+
+
 if __name__ == '__main__':
     try:
         hooks.execute(sys.argv)
     except UnregisteredHookError as e:
         log('Unknown hook {} - skipping.'.format(e))
+    assess_status()

=== modified symlink 'hooks/config-changed'
=== target changed u'hooks.py' => u'ceph_hooks.py'
=== modified symlink 'hooks/install.real'
=== target changed u'hooks.py' => u'ceph_hooks.py'
=== modified symlink 'hooks/mon-relation-changed'
=== target changed u'hooks.py' => u'ceph_hooks.py'
=== modified symlink 'hooks/mon-relation-departed'
=== target changed u'hooks.py' => u'ceph_hooks.py'
=== modified symlink 'hooks/nrpe-external-master-relation-changed'
=== target changed u'hooks.py' => u'ceph_hooks.py'
=== modified symlink 'hooks/nrpe-external-master-relation-joined'
=== target changed u'hooks.py' => u'ceph_hooks.py'
=== modified symlink 'hooks/start'
=== target changed u'hooks.py' => u'ceph_hooks.py'
=== modified symlink 'hooks/stop'
=== target changed u'hooks.py' => u'ceph_hooks.py'
=== added symlink 'hooks/update-status'
=== target is u'ceph_hooks.py'
=== modified symlink 'hooks/upgrade-charm'
=== target changed u'hooks.py' => u'ceph_hooks.py'
=== added file 'setup.cfg'
--- setup.cfg	1970-01-01 00:00:00 +0000
+++ setup.cfg	2015-10-08 11:42:10 +0000
@@ -0,0 +1,5 @@
+[nosetests]
+verbosity=2
+with-coverage=1
+cover-erase=1
+cover-package=hooks

=== modified file 'unit_tests/__init__.py'
--- unit_tests/__init__.py	2014-11-25 18:27:01 +0000
+++ unit_tests/__init__.py	2015-10-08 11:42:10 +0000
@@ -0,0 +1,2 @@
+import sys
+sys.path.append('hooks')

=== added file 'unit_tests/test_status.py'
--- unit_tests/test_status.py	1970-01-01 00:00:00 +0000
+++ unit_tests/test_status.py	2015-10-08 11:42:10 +0000
@@ -0,0 +1,56 @@
+import mock
+import test_utils
+
+import ceph_hooks as hooks
+
+TO_PATCH = [
+    'status_set',
+    'config',
+    'ceph',
+    'relation_ids',
+    'relation_get',
+    'related_units',
+    'get_conf',
+]
+
+CEPH_MONS = [
+    'ceph/0',
+    'ceph/1',
+    'ceph/2',
+]
+
+
+class ServiceStatusTestCase(test_utils.CharmTestCase):
+
+    def setUp(self):
+        super(ServiceStatusTestCase, self).setUp(hooks, TO_PATCH)
+        self.config.side_effect = self.test_config.get
+
+    def test_assess_status_no_monitor_relation(self):
+        self.relation_ids.return_value = []
+        hooks.assess_status()
+        self.status_set.assert_called_with('blocked', mock.ANY)
+
+    def test_assess_status_monitor_relation_incomplete(self):
+        self.relation_ids.return_value = ['mon:1']
+        self.related_units.return_value = CEPH_MONS
+        self.get_conf.return_value = None
+        hooks.assess_status()
+        self.status_set.assert_called_with('waiting', mock.ANY)
+
+    def test_assess_status_monitor_complete_no_disks(self):
+        self.relation_ids.return_value = ['mon:1']
+        self.related_units.return_value = CEPH_MONS
+        self.get_conf.return_value = 'monitor-bootstrap-key'
+        self.ceph.get_running_osds.return_value = []
+        hooks.assess_status()
+        self.status_set.assert_called_with('blocked', mock.ANY)
+
+    def test_assess_status_monitor_complete_disks(self):
+        self.relation_ids.return_value = ['mon:1']
+        self.related_units.return_value = CEPH_MONS
+        self.get_conf.return_value = 'monitor-bootstrap-key'
+        self.ceph.get_running_osds.return_value = ['12345',
+                                                   '67890']
+        hooks.assess_status()
+        self.status_set.assert_called_with('active', mock.ANY)

=== added file 'unit_tests/test_utils.py'
--- unit_tests/test_utils.py	1970-01-01 00:00:00 +0000
+++ unit_tests/test_utils.py	2015-10-08 11:42:10 +0000
@@ -0,0 +1,121 @@
+import logging
+import unittest
+import os
+import yaml
+
+from contextlib import contextmanager
+from mock import patch, MagicMock
+
+
+def load_config():
+    '''
+    Walk backwords from __file__ looking for config.yaml, load and return the
+    'options' section'
+    '''
+    config = None
+    f = __file__
+    while config is None:
+        d = os.path.dirname(f)
+        if os.path.isfile(os.path.join(d, 'config.yaml')):
+            config = os.path.join(d, 'config.yaml')
+            break
+        f = d
+
+    if not config:
+        logging.error('Could not find config.yaml in any parent directory '
+                      'of %s. ' % f)
+        raise Exception
+
+    return yaml.safe_load(open(config).read())['options']
+
+
+def get_default_config():
+    '''
+    Load default charm config from config.yaml return as a dict.
+    If no default is set in config.yaml, its value is None.
+    '''
+    default_config = {}
+    config = load_config()
+    for k, v in config.iteritems():
+        if 'default' in v:
+            default_config[k] = v['default']
+        else:
+            default_config[k] = None
+    return default_config
+
+
+class CharmTestCase(unittest.TestCase):
+
+    def setUp(self, obj, patches):
+        super(CharmTestCase, self).setUp()
+        self.patches = patches
+        self.obj = obj
+        self.test_config = TestConfig()
+        self.test_relation = TestRelation()
+        self.patch_all()
+
+    def patch(self, method):
+        _m = patch.object(self.obj, method)
+        mock = _m.start()
+        self.addCleanup(_m.stop)
+        return mock
+
+    def patch_all(self):
+        for method in self.patches:
+            setattr(self, method, self.patch(method))
+
+
+class TestConfig(object):
+
+    def __init__(self):
+        self.config = get_default_config()
+
+    def get(self, attr=None):
+        if not attr:
+            return self.get_all()
+        try:
+            return self.config[attr]
+        except KeyError:
+            return None
+
+    def get_all(self):
+        return self.config
+
+    def set(self, attr, value):
+        if attr not in self.config:
+            raise KeyError
+        self.config[attr] = value
+
+
+class TestRelation(object):
+
+    def __init__(self, relation_data={}):
+        self.relation_data = relation_data
+
+    def set(self, relation_data):
+        self.relation_data = relation_data
+
+    def get(self, attr=None, unit=None, rid=None):
+        if attr is None:
+            return self.relation_data
+        elif attr in self.relation_data:
+            return self.relation_data[attr]
+        return None
+
+
+@contextmanager
+def patch_open():
+    '''Patch open() to allow mocking both open() itself and the file that is
+    yielded.
+
+    Yields the mock for "open" and "file", respectively.'''
+    mock_open = MagicMock(spec=open)
+    mock_file = MagicMock(spec=file)
+
+    @contextmanager
+    def stub_open(*args, **kwargs):
+        mock_open(*args, **kwargs)
+        yield mock_file
+
+    with patch('__builtin__.open', stub_open):
+        yield mock_open, mock_file