Merge lp:~1chb1n/openstack-mojo-specs/mojo-openstack-specs-juju-wait-for-workload-status into lp:openstack-mojo-specs

[Ryan Beisner]

Refresh with a fork of juju-wait which enables workload status blocking. If a unit is in a blocked state, keep waiting.
  From: https://code.launchpad.net/~1chb1n/juju-wait/workload-status
  Re: https://bugs.launchpad.net/juju-wait/+bug/1553312

Remove neutron-gateway:mysql shared-db relation from bundles
  Re: https://github.com/openstack/charm-neutron-gateway/commit/00f0edc70d68ce846db928ec2304d79fc6d1a5ae

Refresh with a fork of juju-wait which enables workload status blocking. If a unit is in a blocked state, keep waiting.
  From: https://code.launchpad.net/~1chb1n/juju-wait/workload-status
  Re: https://bugs.launchpad.net/juju-wait/+bug/1553312

Remove neutron-gateway:mysql shared-db relation from bundles
  Re: https://github.com/openstack/charm-neutron-gateway/commit/00f0edc70d68ce846db928ec2304d79fc6d1a5ae

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #1877 mojo-openstack-specs for 1chb1n mp288302
    LINT OK: passed

Build: http://10.245.162.36:8080/job/charm_lint_check/1877/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #1878 mojo-openstack-specs for 1chb1n mp288302
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/15321958/
Build: http://10.245.162.36:8080/job/charm_lint_check/1878/

[lp:~1chb1n/openstack-mojo-specs/mojo-openstack-specs-juju-wait-for-workload-status]

254. By Ryan Beisner on 2016-03-07

Lint cleanup

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #1879 mojo-openstack-specs for 1chb1n mp288302
    LINT OK: passed

Build: http://10.245.162.36:8080/job/charm_lint_check/1879/

[Ryan Beisner (1chb1n)]

FYI, specs now passing with this branch. Here are a couple of spot-checks:

http://10.245.162.36:8080/view/Dashboards/view/TestCharmPipeline/job/mojo_runner_baremetal/234/

http://10.245.162.36:8080/view/Dashboards/view/Mojo/job/mojo_runner/521/

[James Page (james-page)]

review: Approve

=== modified file 'helper/bundles/baremetal7.yaml'
--- helper/bundles/baremetal7.yaml	2016-01-22 15:43:03 +0000
+++ helper/bundles/baremetal7.yaml	2016-03-07 17:17:39 +0000
@@ -152,7 +152,6 @@
     - [ cinder, keystone ]
     - [ cinder, cinder-ceph ]
     - [ cinder-ceph, ceph ]
-    - [ neutron-gateway, mysql ]
     - [ neutron-gateway, nova-cloud-controller ]
     - [ openstack-dashboard, keystone ]
     - [ swift-proxy, keystone ]

=== modified file 'helper/bundles/full.yaml'
--- helper/bundles/full.yaml	2016-01-22 17:10:00 +0000
+++ helper/bundles/full.yaml	2016-03-07 17:17:39 +0000
@@ -113,7 +113,6 @@
     - [ cinder, nova-cloud-controller ]
     - [ cinder, keystone ]
     - [ cinder, ceph ]
-    - [ neutron-gateway, mysql ]
     - [ neutron-gateway, nova-cloud-controller ]
     - [ openstack-dashboard, keystone ]
     - [ swift-proxy, keystone ]

=== modified file 'helper/bundles/full_ssl.yaml'
--- helper/bundles/full_ssl.yaml	2015-08-05 08:40:27 +0000
+++ helper/bundles/full_ssl.yaml	2016-03-07 17:17:39 +0000
@@ -117,7 +117,6 @@
     - [ cinder, nova-cloud-controller ]
     - [ cinder, keystone ]
     - [ cinder, ceph ]
-    - [ neutron-gateway, mysql ]
     - [ neutron-gateway, nova-cloud-controller ]
     - [ openstack-dashboard, keystone ]
     - [ swift-proxy, keystone ]

=== modified file 'helper/bundles/full_vrrpha.yaml'
--- helper/bundles/full_vrrpha.yaml	2015-11-16 15:07:13 +0000
+++ helper/bundles/full_vrrpha.yaml	2016-03-07 17:17:39 +0000
@@ -64,7 +64,6 @@
     - [ cinder, rabbitmq-server ]
     - [ cinder, nova-cloud-controller ]
     - [ cinder, keystone ]
-    - [ neutron-gateway, mysql ]
     - [ neutron-gateway, nova-cloud-controller ]
     - [ openstack-dashboard, keystone ]
 openstack-singlerabbit:

=== modified file 'helper/bundles/ha.yaml'
--- helper/bundles/ha.yaml	2016-01-20 10:57:18 +0000
+++ helper/bundles/ha.yaml	2016-03-07 17:17:39 +0000
@@ -196,7 +196,6 @@
     - [ cinder, keystone ]
     - [ cinder, cinder-ceph ]
     - [ cinder-ceph, ceph ]
-    - [ neutron-gateway, mysql ]
     - [ neutron-gateway, nova-cloud-controller ]
     - [ openstack-dashboard, keystone ]
     - [ swift-proxy, keystone ]

=== modified file 'helper/bundles/haphase2.yaml'
--- helper/bundles/haphase2.yaml	2015-10-21 20:53:34 +0000
+++ helper/bundles/haphase2.yaml	2016-03-07 17:17:39 +0000
@@ -152,7 +152,6 @@
     - [ nova-compute, mysql ]
     - [ glance, mysql ]
     - [ cinder, mysql ]
-    - [ neutron-gateway, mysql ]
 openstack-mysql-database:
   inherits:
   - openstack-services

=== modified file 'helper/bundles/minimal.yaml'
--- helper/bundles/minimal.yaml	2015-08-05 08:40:27 +0000
+++ helper/bundles/minimal.yaml	2016-03-07 17:17:39 +0000
@@ -61,7 +61,6 @@
     - [ cinder, rabbitmq-server ]
     - [ cinder, nova-cloud-controller ]
     - [ cinder, keystone ]
-    - [ neutron-gateway, mysql ]
     - [ neutron-gateway, nova-cloud-controller ]
 openstack-singlerabbit:
 openstack-singlerabbit:

=== modified file 'helper/bundles/next-ha.yaml'
--- helper/bundles/next-ha.yaml	2015-08-05 08:40:27 +0000
+++ helper/bundles/next-ha.yaml	2016-03-07 17:17:39 +0000
@@ -173,7 +173,6 @@
     - [ cinder, nova-cloud-controller ]
     - [ cinder, keystone ]
     - [ cinder, ceph ]
-    - [ neutron-gateway, mysql ]
     - [ neutron-gateway, nova-cloud-controller ]
     - [ openstack-dashboard, keystone ]
     - [ swift-proxy, keystone ]

=== modified file 'helper/bundles/ssl-everywhere.yaml'
--- helper/bundles/ssl-everywhere.yaml	2014-12-18 10:28:32 +0000
+++ helper/bundles/ssl-everywhere.yaml	2016-03-07 17:17:39 +0000
@@ -117,7 +117,6 @@
     - [ cinder, nova-cloud-controller ]
     - [ cinder, keystone ]
     - [ cinder, ceph ]
-    - [ neutron-gateway, mysql ]
     - [ neutron-gateway, rabbitmq-server ]
     - [ neutron-gateway, nova-cloud-controller ]
     - [ openstack-dashboard, keystone ]

=== modified file 'helper/tests/check_juju.py'
--- helper/tests/check_juju.py	2015-11-02 11:31:41 +0000
+++ helper/tests/check_juju.py	2016-03-07 17:17:39 +0000
@@ -1,6 +1,6 @@
 #!/usr/bin/python
 import sys
-from utils.juju_wait import wait
+from utils.mojo_utils import juju_wait_finished
 
 if __name__ == '__main__':
-    sys.exit(wait())
+    sys.exit(juju_wait_finished())

=== modified file 'helper/utils/juju_wait.py'
--- helper/utils/juju_wait.py	2015-11-02 11:31:41 +0000
+++ helper/utils/juju_wait.py	2016-03-07 17:17:39 +0000
@@ -1,5 +1,10 @@
 #!/usr/bin/python3
 
+# NOTE(beisner):  this is a fork of juju-wait which enables workload
+# status blocking.  ie. If a unit is in a blocked state, keep waiting.
+#   From:  https://code.launchpad.net/~1chb1n/juju-wait/workload-status
+#   Re:    https://bugs.launchpad.net/juju-wait/+bug/1553312
+
 # This file is part of juju-wait, a juju plugin to wait for environment
 # steady state.
 #
@@ -82,7 +87,6 @@
     # using the environment variable.
     env = os.environ.copy()
     env['TZ'] = 'UTC'
-    env['JUJU_CLI_VERSION'] = '1'
     json_status = run_or_die(['juju', 'status', '--format=json'], env=env)
     if json_status is None:
         return None
@@ -107,6 +111,14 @@
 # still due to be run.
 IDLE_CONFIRMATION = timedelta(seconds=15)
 
+# If all units have one of the following workload status values,
+# consider them ready.  Not all charms use workload the status feature.
+# Those that do not will be represented by the 'unknown' value and
+# workload status will be ignored.  For charms that do, wait for an
+# 'active' workload status value.  FYI: blocked, waiting, maintenance
+# values indicate not-ready workload states.
+WORKLOAD_OK_STATES = ['active', 'unknown']
+
 
 def wait_cmd(args=sys.argv[1:]):
     description = dedent("""\
@@ -126,6 +138,9 @@
                         action='store_true', default=False)
     parser.add_argument('-v', '--verbose', dest='verbose',
                         action='store_true', default=False)
+    parser.add_argument('-w', '--workload', dest='wait_for_workload',
+                        help='Wait for unit workload status active state',
+                        action='store_true', default=False)
     args = parser.parse_args(args)
 
     # Parser did not exit, so continue.
@@ -138,7 +153,7 @@
     else:
         log.setLevel(logging.INFO)
     try:
-        wait(log)
+        wait(log, args.wait_for_workload)
         return 0
     except JujuWaitException as x:
         return x.args[0]
@@ -153,7 +168,7 @@
                 'logging-config=juju=WARNING;unit=INFO'])
 
 
-def wait(log=None):
+def wait(log=None, wait_for_workload=False):
     if log is None:
         log = logging.getLogger()
 
@@ -181,32 +196,50 @@
         # logs sniffed because they are running Juju 1.23 or earlier.
         ready_units = {}
 
-        # Flattened agent status for all units and subordinates that
-        # provide it. Note that 'agent status' is only available in
+        # Flattened agent and workload status for all units and subordinates
+        # that provide it. Note that 'agent status' is only available in
         # Juju 1.24 and later. This is easily confused with 'agent state'
         # which is available in earlier versions of Juju.
+        workload_status = {}
         agent_status = {}
         agent_version = {}
         for sname, service in status.get('services', {}).items():
             for uname, unit in service.get('units', {}).items():
                 all_units.add(uname)
                 agent_version[uname] = unit.get('agent-version')
-                if 'agent-status' in unit:
+                if ('workload-status' in unit and
+                        'current' in unit['workload-status']):
+                    workload_status[uname] = unit['workload-status']
+
+                if 'agent-status' in unit and unit['agent-status'] != {}:
                     agent_status[uname] = unit['agent-status']
                 else:
                     ready_units[uname] = unit  # Schedule for sniffing.
                 for subname, sub in unit.get('subordinates', {}).items():
+                    if ('workload-status' in sub and
+                            'current' in sub['workload-status']):
+                        workload_status[subname] = sub['workload-status']
+
                     agent_version[subname] = sub.get('agent-version')
-                    if 'agent-status' in sub:
+                    if 'agent-status' in sub and unit['agent-status'] != {}:
                         agent_status[subname] = sub['agent-status']
                     else:
                         ready_units[subname] = sub  # Schedule for sniffing.
 
+        for uname, wstatus in sorted(workload_status.items()):
+            current = wstatus['current']
+            since = parse_ts(wstatus['since'])
+            if current not in WORKLOAD_OK_STATES and wait_for_workload:
+                logging.debug('{} workload status is {} since '
+                              '{}Z'.format(uname, current, since))
+                ready = False
+
         for uname, astatus in sorted(agent_status.items()):
             current = astatus['current']
             since = parse_ts(astatus['since'])
-            logging.debug('{} is {} since {}Z'.format(uname, current, since))
             if current != 'idle':
+                logging.debug('{} agent status is {} since '
+                              '{}Z'.format(uname, current, since))
                 ready = False
 
         # Log storage to compare with prev_logs.
@@ -228,6 +261,11 @@
                 logging.debug('{} is {}'.format(uname, agent_state))
                 ready = False
             elif ready:
+                # We only start grabbing the logs once all the units
+                # are in a suitable lifecycle state. If we don't do this,
+                # we risk attempting to grab logs from units or subordinates
+                # that are not yet ready to respond, or have disappeared
+                # since we last checked the environment status.
                 if not logging_reset:
                     reset_logging()
                     logging_reset = True
@@ -248,9 +286,9 @@
             for uname, version in agent_version.items():
                 sname = uname.split('/', 1)[0]
                 services.add(sname)
-                if (sname not in services_with_leader and version
-                    and (LooseVersion(version) >= LooseVersion('1.23')
-                         or get_is_leader(uname) is True)):
+                if (sname not in services_with_leader and version and
+                    (LooseVersion(version) <= LooseVersion('1.23') or
+                        get_is_leader(uname) is True)):
                     services_with_leader.add(sname)
                     logging.debug('{} is lead by {}'.format(sname, uname))
             for sname in services:
@@ -283,4 +321,7 @@
     # Most installations will be using the setuptools generated
     # launchers.
     script = os.path.basename(sys.argv[0])
-    sys.exit(wait_cmd())
+    if script == 'juju-wait':
+        sys.exit(wait_cmd())
+    else:
+        raise RuntimeError('Unknown script {}'.format(script))

=== modified file 'helper/utils/mojo_utils.py'
--- helper/utils/mojo_utils.py	2016-03-07 10:37:26 +0000
+++ helper/utils/mojo_utils.py	2016-03-07 17:17:39 +0000
@@ -481,7 +481,13 @@
 
 
 def juju_wait_finished():
-    juju_wait.wait()
+    """Use juju-wait from local utils path to block until all service
+    units quiesce and satisfy workload status ready state."""
+    log = logging.getLogger()
+    log.setLevel(logging.DEBUG)
+    logging.info('Calling juju-wait')
+    juju_wait.wait(log, wait_for_workload=True)
+    logging.debug('End of juju-wait')
 
 
 def dict_to_yaml(dict_data):