Merge lp:~1chb1n/charms/trusty/cinder-ceph/next-amulet-relation-settle into lp:~openstack-charmers-archive/charms/trusty/cinder-ceph/next

[Ryan Beisner]

Amulet tests (in general) are sometimes racey when the test rig is under higher load, as the relations may not have settled before tests begin.

Instead of an arbitary sleep time after deployment is "complete," use the same logic we use in openstack-mojo-testing (a juju run -all XYZ check) before proceeding.

Ex. http://paste.ubuntu.com/11812413/ ... unexpected relation data in cinder cinder-ceph storage-backend - key 'broker_rsp' does not exist

This MP is dependent on c-h changes @ https://code.launchpad.net/~1chb1n/charm-helpers/amulet-relations-settled/+merge/263724

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #5729 cinder-ceph-next for 1chb1n mp263716
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/5729/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #5361 cinder-ceph-next for 1chb1n mp263716
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/5361/

[lp:~1chb1n/charms/trusty/cinder-ceph/next-amulet-relation-settle]

36. By Ryan Beisner on 2015-07-02

use juju run all (a la openstack-mojo-specs) instead of arbitrary sleep; resync tests/charmhelpers

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #5730 cinder-ceph-next for 1chb1n mp263716
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/5730/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #5362 cinder-ceph-next for 1chb1n mp263716
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/5362/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4923 cinder-ceph-next for 1chb1n mp263716
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/4923/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4928 cinder-ceph-next for 1chb1n mp263716
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/11813598/
Build: http://10.245.162.77:8080/job/charm_amulet_test/4928/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4925 cinder-ceph-next for 1chb1n mp263716
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/4925/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4927 cinder-ceph-next for 1chb1n mp263716
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/4927/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4926 cinder-ceph-next for 1chb1n mp263716
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/4926/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4924 cinder-ceph-next for 1chb1n mp263716
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/4924/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4993 cinder-ceph-next for 1chb1n mp263716
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/4993/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4996 cinder-ceph-next for 1chb1n mp263716
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/4996/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4995 cinder-ceph-next for 1chb1n mp263716
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/4995/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4994 cinder-ceph-next for 1chb1n mp263716
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/4994/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #4997 cinder-ceph-next for 1chb1n mp263716
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/4997/

[Ryan Beisner (1chb1n)]

FYI: To check the new wait logic, I repeatedly ran multiple simultaneous jobs while the system was under load, and all passed.

http://10.245.162.77:8080/view/Dashboards/view/Amulet/job/charm_amulet_test/4993/
http://10.245.162.77:8080/view/Dashboards/view/Amulet/job/charm_amulet_test/4994/
http://10.245.162.77:8080/view/Dashboards/view/Amulet/job/charm_amulet_test/4995/
http://10.245.162.77:8080/view/Dashboards/view/Amulet/job/charm_amulet_test/4996/
http://10.245.162.77:8080/view/Dashboards/view/Amulet/job/charm_amulet_test/4997/

[Ryan Beisner (1chb1n)]

Before these changes, in a looped local test run, appx 1 in 4 tests failed with the relation data issue.

With these changes, 15 of 15 local test runs succeeded.

[lp:~1chb1n/charms/trusty/cinder-ceph/next-amulet-relation-settle]

37. By Ryan Beisner on 2015-07-09

resync tests/charmhelpers

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #5901 cinder-ceph-next for 1chb1n mp263716
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/5901/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #5533 cinder-ceph-next for 1chb1n mp263716
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/5533/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5014 cinder-ceph-next for 1chb1n mp263716
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/5014/

[Corey Bryant (corey.bryant)]

Looks good, I'll approve once the c-h portion lands.

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5025 cinder-ceph-next for 1chb1n mp263716
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/11852192/
Build: http://10.245.162.77:8080/job/charm_amulet_test/5025/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5026 cinder-ceph-next for 1chb1n mp263716
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/11852198/
Build: http://10.245.162.77:8080/job/charm_amulet_test/5026/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5028 cinder-ceph-next for 1chb1n mp263716
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/11852202/
Build: http://10.245.162.77:8080/job/charm_amulet_test/5028/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5029 cinder-ceph-next for 1chb1n mp263716
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/11852210/
Build: http://10.245.162.77:8080/job/charm_amulet_test/5029/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5027 cinder-ceph-next for 1chb1n mp263716
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
Timeout occurred (2700s), printing juju status...environment: osci-sv05
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/11852211/
Build: http://10.245.162.77:8080/job/charm_amulet_test/5027/

[Ryan Beisner (1chb1n)]

^^ Amulet test builds #5025 - 5029 #undercloud-fail

Will re-run after confirming test rig's undercloud health...

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5032 cinder-ceph-next for 1chb1n mp263716
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/11855935/
Build: http://10.245.162.77:8080/job/charm_amulet_test/5032/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5033 cinder-ceph-next for 1chb1n mp263716
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/5033/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5034 cinder-ceph-next for 1chb1n mp263716
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/5034/

[Ryan Beisner (1chb1n)]

^ 1 of 3 failed with the relation settling race, with the 2nd remote run check removed, the test race condition re-appeared. So I added it back, which also keeps the approach in line with the same in openstack mojo specs.

[lp:~1chb1n/charms/trusty/cinder-ceph/next-amulet-relation-settle]

38. By Ryan Beisner on 2015-07-13

resync tests/charmhelpers

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #6111 cinder-ceph-next for 1chb1n mp263716
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/6111/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #5743 cinder-ceph-next for 1chb1n mp263716
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/5743/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5097 cinder-ceph-next for 1chb1n mp263716
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/11872749/
Build: http://10.245.162.77:8080/job/charm_amulet_test/5097/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5099 cinder-ceph-next for 1chb1n mp263716
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/11873035/
Build: http://10.245.162.77:8080/job/charm_amulet_test/5099/

[Ryan Beisner (1chb1n)]

Abandoning this as the juju run twice approach appears to no longer be effective.

I disapprove. ;-)

review: Disapprove

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5100 cinder-ceph-next for 1chb1n mp263716
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/5100/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #5098 cinder-ceph-next for 1chb1n mp263716
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/5098/

Unmerged revisions

38. By Ryan Beisner on 2015-07-13

resync tests/charmhelpers

37. By Ryan Beisner on 2015-07-09

resync tests/charmhelpers

36. By Ryan Beisner on 2015-07-02

use juju run all (a la openstack-mojo-specs) instead of arbitrary sleep; resync tests/charmhelpers

35. By Ryan Beisner on 2015-07-02

increase initial wait

=== modified file 'tests/basic_deployment.py'
--- tests/basic_deployment.py	2015-06-30 20:40:06 +0000
+++ tests/basic_deployment.py	2015-07-13 14:04:38 +0000
@@ -103,9 +103,6 @@
     def _initialize_tests(self):
         """Perform final initialization before tests get run."""
         # Access the sentries for inspecting service units
-        u.log.debug('!!!!!')
-        u.log.debug(dir(self.d.sentry))
-
         self.mysql_sentry = self.d.sentry.unit['mysql/0']
         self.keystone_sentry = self.d.sentry.unit['keystone/0']
         self.rabbitmq_sentry = self.d.sentry.unit['rabbitmq-server/0']
@@ -119,8 +116,8 @@
         u.log.debug('openstack release str: {}'.format(
             self._get_openstack_release_string()))
 
-        # Let things settle a bit original moving forward
-        time.sleep(30)
+        # Make sure relations have settled, raise exception if not
+        u.check_hooks_are_settled()
 
         # Authenticate admin with keystone
         self.keystone = u.authenticate_keystone_admin(self.keystone_sentry,

=== modified file 'tests/charmhelpers/contrib/amulet/utils.py'
--- tests/charmhelpers/contrib/amulet/utils.py	2015-06-29 14:33:37 +0000
+++ tests/charmhelpers/contrib/amulet/utils.py	2015-07-13 14:04:38 +0000
@@ -22,9 +22,11 @@
 import os
 import re
 import six
+import subprocess
 import sys
 import time
 import urlparse
+import yaml
 
 
 class AmuletUtils(object):
@@ -531,3 +533,47 @@
             return 'Dicts within list are not identical'
 
         return None
+
+    def _remote_shell_check(self, unit, timeout='2m0s'):
+        """Check that a juju run shell command succeeds on a juju unit."""
+        cmd = ['juju', 'run', '--timeout', timeout,
+               '--unit', unit, 'uname -a']
+        fnull = open(os.devnull, 'w')
+        return not subprocess.call(cmd, stdout=fnull, stderr=subprocess.STDOUT)
+
+    def _remote_runs(self, units, timeout='2m0s'):
+        """Check that juju run succeeds against a list of units."""
+        self.log.debug('Checking juju run command completion on '
+                       'units: {}'.format(units))
+        for unit in units:
+            if not self._remote_shell_check(unit, timeout):
+                msg = 'Juju run failed or timed out on {}'.format(unit)
+                amulet.raise_status(amulet.FAIL, msg=msg)
+        self.log.debug('Units completed juju run command checks OK.')
+
+    def get_juju_status(self):
+        """Get juju status yaml data, return as dictionary."""
+        self.log.debug('Getting juju status yaml data...')
+        cmd = ['juju', 'status', '--format', 'yaml']
+        status_file = subprocess.Popen(cmd, stdout=subprocess.PIPE).stdout
+        return yaml.load(status_file)
+
+    def get_juju_units(self):
+        """Get a list of juju units from juju status output."""
+        self.log.debug('Getting juju unit list from juju status data...')
+        juju_status = self.get_juju_status()
+        units = []
+        services = [svc for svc in juju_status['services']]
+        for svc in services:
+            if 'units' in juju_status['services'][svc]:
+                for unit in juju_status['services'][svc]['units']:
+                    units.append(unit)
+        return units
+
+    def check_hooks_are_settled(self, timeout='2m0s'):
+        """Use juju run (twice) against all units as an indicator that
+        hooks are done.  Raise exception if any unit fails or times out."""
+        self.log.info('Checking that juju hooks appear to be complete...')
+        juju_units = self.get_juju_units()
+        self._remote_runs(juju_units, timeout)
+        self._remote_runs(juju_units, timeout)