Merge lp:~1chb1n/charm-helpers/amulet-relations-settled into lp:charm-helpers

[Ryan Beisner]

Add check_hooks_are_settled() to amulet/utils.py, based on openstack-mojo-specs.

Prior to this, openstack amulet tests relied on an arbitrary sleep time after deployment returned "complete," to wait for hooks to settle. That was racey, as the sleep time was not always enough during higher testing load.

This proposal uses the logic that openstack-mojo-specs has been using to address this challenge.

# Charm test MPs which depend on this landing:
https://code.launchpad.net/~1chb1n/charms/trusty/cinder-ceph/next-amulet-relation-settle/+merge/263716

[Ryan Beisner (1chb1n)]

FYI: To check the new wait logic, I repeatedly ran multiple simultaneous jobs while the system was under load, and all passed.

http://10.245.162.77:8080/view/Dashboards/view/Amulet/job/charm_amulet_test/4993/
http://10.245.162.77:8080/view/Dashboards/view/Amulet/job/charm_amulet_test/4994/
http://10.245.162.77:8080/view/Dashboards/view/Amulet/job/charm_amulet_test/4995/
http://10.245.162.77:8080/view/Dashboards/view/Amulet/job/charm_amulet_test/4996/
http://10.245.162.77:8080/view/Dashboards/view/Amulet/job/charm_amulet_test/4997/

[Ryan Beisner (1chb1n)]

For reference, this is based on juju_check_hooks_complete() from:

http://bazaar.launchpad.net/~ost-maintainers/openstack-mojo-specs/mojo-openstack-specs/view/head:/helper/utils/mojo_utils.py#L456

[Corey Bryant (corey.bryant)]

Looks good but I have a couple comments.

[Ryan Beisner (1chb1n)]

Thanks, see replies, questions in line.

[Corey Bryant (corey.bryant)]

Responded inline

[lp:~1chb1n/charm-helpers/amulet-relations-settled]

402. By Ryan Beisner on 2015-07-09

Plumb timeout all the way through;
Remove 2nd _remote_runs check;
Adjust _remote_run failure feedback.

[Corey Bryant (corey.bryant)]

review: Approve

[lp:~1chb1n/charm-helpers/amulet-relations-settled]

403. By Ryan Beisner on 2015-07-13

add 2nd _remote_runs check back in

[Ryan Beisner (1chb1n)]

FYI, with the 2nd remote run check removed, the test race condition re-appeared. So I added it back, which also keeps the approach in line with the same in openstack mojo specs.

[Corey Bryant (corey.bryant)]

Sounds fine to me then.

[lp:~1chb1n/charm-helpers/amulet-relations-settled]

404. By Ryan Beisner on 2015-07-13

rebase

[Ryan Beisner (1chb1n)]

Abandoning this as the juju run twice approach appears to no longer be effective.

I disapprove. ;-)

review: Disapprove

Unmerged revisions

404. By Ryan Beisner on 2015-07-13

rebase

403. By Ryan Beisner on 2015-07-13

add 2nd _remote_runs check back in

402. By Ryan Beisner on 2015-07-09

Plumb timeout all the way through;
Remove 2nd _remote_runs check;
Adjust _remote_run failure feedback.

401. By Ryan Beisner on 2015-07-02

add juju_hooks_complete to amulet/utils.py (based on openstack-mojospecs)

=== modified file 'charmhelpers/contrib/amulet/utils.py'
--- charmhelpers/contrib/amulet/utils.py	2015-06-29 13:19:46 +0000
+++ charmhelpers/contrib/amulet/utils.py	2015-07-13 14:36:55 +0000
@@ -22,9 +22,11 @@
 import os
 import re
 import six
+import subprocess
 import sys
 import time
 import urlparse
+import yaml
 
 
 class AmuletUtils(object):
@@ -531,3 +533,47 @@
             return 'Dicts within list are not identical'
 
         return None
+
+    def _remote_shell_check(self, unit, timeout='2m0s'):
+        """Check that a juju run shell command succeeds on a juju unit."""
+        cmd = ['juju', 'run', '--timeout', timeout,
+               '--unit', unit, 'uname -a']
+        fnull = open(os.devnull, 'w')
+        return not subprocess.call(cmd, stdout=fnull, stderr=subprocess.STDOUT)
+
+    def _remote_runs(self, units, timeout='2m0s'):
+        """Check that juju run succeeds against a list of units."""
+        self.log.debug('Checking juju run command completion on '
+                       'units: {}'.format(units))
+        for unit in units:
+            if not self._remote_shell_check(unit, timeout):
+                msg = 'Juju run failed or timed out on {}'.format(unit)
+                amulet.raise_status(amulet.FAIL, msg=msg)
+        self.log.debug('Units completed juju run command checks OK.')
+
+    def get_juju_status(self):
+        """Get juju status yaml data, return as dictionary."""
+        self.log.debug('Getting juju status yaml data...')
+        cmd = ['juju', 'status', '--format', 'yaml']
+        status_file = subprocess.Popen(cmd, stdout=subprocess.PIPE).stdout
+        return yaml.load(status_file)
+
+    def get_juju_units(self):
+        """Get a list of juju units from juju status output."""
+        self.log.debug('Getting juju unit list from juju status data...')
+        juju_status = self.get_juju_status()
+        units = []
+        services = [svc for svc in juju_status['services']]
+        for svc in services:
+            if 'units' in juju_status['services'][svc]:
+                for unit in juju_status['services'][svc]['units']:
+                    units.append(unit)
+        return units
+
+    def check_hooks_are_settled(self, timeout='2m0s'):
+        """Use juju run (twice) against all units as an indicator that
+        hooks are done.  Raise exception if any unit fails or times out."""
+        self.log.info('Checking that juju hooks appear to be complete...')
+        juju_units = self.get_juju_units()
+        self._remote_runs(juju_units, timeout)
+        self._remote_runs(juju_units, timeout)