Merge lp:~1chb1n/charms/trusty/neutron-gateway/next-amulet-15.10 into lp:~openstack-charmers-archive/charms/trusty/neutron-gateway/next

[Ryan Beisner]

Update amulet tests for Trusty-Liberty, Wily-Liberty.

Sync charmhelpers.

Add service and relations to satisfy workload status ready state.

Add new logic to wait for extended status message to confirm deploy is ready, before testing.

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #12026 neutron-gateway-next for 1chb1n mp274722
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/12026/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #11176 neutron-gateway-next for 1chb1n mp274722
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/11176/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7387 neutron-gateway-next for 1chb1n mp274722
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 124
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12800187/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7387/

[Ryan Beisner (1chb1n)]

^ bootstrap fail

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7388 neutron-gateway-next for 1chb1n mp274722
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12800424/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7388/

[lp:~1chb1n/charms/trusty/neutron-gateway/next-amulet-15.10]

154. By Ryan Beisner on 2015-10-16

fix relation check, remove varying unit data check

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #12031 neutron-gateway-next for 1chb1n mp274722
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/12031/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #11178 neutron-gateway-next for 1chb1n mp274722
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/11178/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7392 neutron-gateway-next for 1chb1n mp274722
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12803083/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7392/

[lp:~1chb1n/charms/trusty/neutron-gateway/next-amulet-15.10]

155. By Ryan Beisner on 2015-10-19

enable wily test; add debug output

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #12162 neutron-gateway-next for 1chb1n mp274722
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/12162/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #11293 neutron-gateway-next for 1chb1n mp274722
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/11293/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7432 neutron-gateway-next for 1chb1n mp274722
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/7432/

[Liam Young (gnuoy)]

Approved

review: Approve

=== modified file 'hooks/charmhelpers/contrib/openstack/amulet/deployment.py'
--- hooks/charmhelpers/contrib/openstack/amulet/deployment.py	2015-09-21 22:51:02 +0000
+++ hooks/charmhelpers/contrib/openstack/amulet/deployment.py	2015-10-19 14:53:22 +0000
@@ -14,6 +14,7 @@
 # You should have received a copy of the GNU Lesser General Public License
 # along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
 
+import re
 import six
 from collections import OrderedDict
 from charmhelpers.contrib.amulet.deployment import (
@@ -114,6 +115,45 @@
         for service, config in six.iteritems(configs):
             self.d.configure(service, config)
 
+    def _auto_wait_for_status(self, message=None, exclude_services=None,
+                              timeout=1800):
+        """Wait for all units to have a specific extended status, except
+        for any defined as excluded.  Unless specified via message, any
+        status containing any case of 'ready' will be considered a match.
+
+        Examples of message usage:
+
+          Wait for all unit status to CONTAIN any case of 'ready' or 'ok':
+              message = re.compile('.*ready.*|.*ok.*', re.IGNORECASE)
+
+          Wait for all units to reach this status (exact match):
+              message = 'Unit is ready'
+
+          Wait for all units to reach any one of these (exact match):
+              message = re.compile('Unit is ready|OK|Ready')
+
+          Wait for at least one unit to reach this status (exact match):
+              message = {'ready'}
+
+        See Amulet's sentry.wait_for_messages() for message usage detail.
+        https://github.com/juju/amulet/blob/master/amulet/sentry.py
+
+        :param message: Expected status match
+        :param exclude_services: List of juju service names to ignore
+        :param timeout: Maximum time in seconds to wait for status match
+        :returns: None.  Raises if timeout is hit.
+        """
+
+        if not message:
+            message = re.compile('.*ready.*', re.IGNORECASE)
+
+        if not exclude_services:
+            exclude_services = []
+
+        services = list(set(self.d.services.keys()) - set(exclude_services))
+        service_messages = {service: message for service in services}
+        self.d.sentry.wait_for_messages(service_messages, timeout=timeout)
+
     def _get_openstack_release(self):
         """Get openstack release.
 

=== modified file 'hooks/charmhelpers/contrib/openstack/amulet/utils.py'
--- hooks/charmhelpers/contrib/openstack/amulet/utils.py	2015-09-21 22:51:02 +0000
+++ hooks/charmhelpers/contrib/openstack/amulet/utils.py	2015-10-19 14:53:22 +0000
@@ -752,7 +752,7 @@
             self.log.debug('SSL is enabled  @{}:{} '
                            '({})'.format(host, port, unit_name))
             return True
-        elif not port and not conf_ssl:
+        elif not conf_ssl:
             self.log.debug('SSL not enabled @{}:{} '
                            '({})'.format(host, port, unit_name))
             return False

=== modified file 'hooks/charmhelpers/contrib/openstack/context.py'
--- hooks/charmhelpers/contrib/openstack/context.py	2015-09-29 20:59:50 +0000
+++ hooks/charmhelpers/contrib/openstack/context.py	2015-10-19 14:53:22 +0000
@@ -952,6 +952,19 @@
                     'config': config}
         return ovs_ctxt
 
+    def midonet_ctxt(self):
+        driver = neutron_plugin_attribute(self.plugin, 'driver',
+                                          self.network_manager)
+        midonet_config = neutron_plugin_attribute(self.plugin, 'config',
+                                                  self.network_manager)
+        mido_ctxt = {'core_plugin': driver,
+                     'neutron_plugin': 'midonet',
+                     'neutron_security_groups': self.neutron_security_groups,
+                     'local_ip': unit_private_ip(),
+                     'config': midonet_config}
+
+        return mido_ctxt
+
     def __call__(self):
         if self.network_manager not in ['quantum', 'neutron']:
             return {}
@@ -973,6 +986,8 @@
             ctxt.update(self.nuage_ctxt())
         elif self.plugin == 'plumgrid':
             ctxt.update(self.pg_ctxt())
+        elif self.plugin == 'midonet':
+            ctxt.update(self.midonet_ctxt())
 
         alchemy_flags = config('neutron-alchemy-flags')
         if alchemy_flags:
@@ -1105,7 +1120,7 @@
 
         ctxt = {
             ... other context ...
-            'subordinate_config': {
+            'subordinate_configuration': {
                 'DEFAULT': {
                     'key1': 'value1',
                 },
@@ -1146,22 +1161,23 @@
                     try:
                         sub_config = json.loads(sub_config)
                     except:
-                        log('Could not parse JSON from subordinate_config '
-                            'setting from %s' % rid, level=ERROR)
+                        log('Could not parse JSON from '
+                            'subordinate_configuration setting from %s'
+                            % rid, level=ERROR)
                         continue
 
                     for service in self.services:
                         if service not in sub_config:
-                            log('Found subordinate_config on %s but it contained'
-                                'nothing for %s service' % (rid, service),
-                                level=INFO)
+                            log('Found subordinate_configuration on %s but it '
+                                'contained nothing for %s service'
+                                % (rid, service), level=INFO)
                             continue
 
                         sub_config = sub_config[service]
                         if self.config_file not in sub_config:
-                            log('Found subordinate_config on %s but it contained'
-                                'nothing for %s' % (rid, self.config_file),
-                                level=INFO)
+                            log('Found subordinate_configuration on %s but it '
+                                'contained nothing for %s'
+                                % (rid, self.config_file), level=INFO)
                             continue
 
                         sub_config = sub_config[self.config_file]

=== modified file 'hooks/charmhelpers/contrib/openstack/neutron.py'
--- hooks/charmhelpers/contrib/openstack/neutron.py	2015-09-28 11:18:48 +0000
+++ hooks/charmhelpers/contrib/openstack/neutron.py	2015-10-19 14:53:22 +0000
@@ -209,6 +209,20 @@
             'server_packages': ['neutron-server',
                                 'neutron-plugin-plumgrid'],
             'server_services': ['neutron-server']
+        },
+        'midonet': {
+            'config': '/etc/neutron/plugins/midonet/midonet.ini',
+            'driver': 'midonet.neutron.plugin.MidonetPluginV2',
+            'contexts': [
+                context.SharedDBContext(user=config('neutron-database-user'),
+                                        database=config('neutron-database'),
+                                        relation_prefix='neutron',
+                                        ssl_dir=NEUTRON_CONF_DIR)],
+            'services': [],
+            'packages': [[headers_package()] + determine_dkms_package()],
+            'server_packages': ['neutron-server',
+                                'python-neutron-plugin-midonet'],
+            'server_services': ['neutron-server']
         }
     }
     if release >= 'icehouse':

=== modified file 'hooks/charmhelpers/contrib/openstack/utils.py'
--- hooks/charmhelpers/contrib/openstack/utils.py	2015-09-28 09:44:36 +0000
+++ hooks/charmhelpers/contrib/openstack/utils.py	2015-10-19 14:53:22 +0000
@@ -121,6 +121,7 @@
     ('2.2.2', 'kilo'),
     ('2.3.0', 'liberty'),
     ('2.4.0', 'liberty'),
+    ('2.5.0', 'liberty'),
 ])
 
 # >= Liberty version->codename mapping

=== modified file 'hooks/charmhelpers/core/host.py'
--- hooks/charmhelpers/core/host.py	2015-09-21 22:51:02 +0000
+++ hooks/charmhelpers/core/host.py	2015-10-19 14:53:22 +0000
@@ -566,7 +566,14 @@
         os.chdir(cur)
 
 
-def chownr(path, owner, group, follow_links=True):
+def chownr(path, owner, group, follow_links=True, chowntopdir=False):
+    """
+    Recursively change user and group ownership of files and directories
+    in given path. Doesn't chown path itself by default, only its children.
+
+    :param bool follow_links: Also Chown links if True
+    :param bool chowntopdir: Also chown path itself if True
+    """
     uid = pwd.getpwnam(owner).pw_uid
     gid = grp.getgrnam(group).gr_gid
     if follow_links:
@@ -574,6 +581,10 @@
     else:
         chown = os.lchown
 
+    if chowntopdir:
+        broken_symlink = os.path.lexists(path) and not os.path.exists(path)
+        if not broken_symlink:
+            chown(path, uid, gid)
     for root, dirs, files in os.walk(path):
         for name in dirs + files:
             full = os.path.join(root, name)

=== modified file 'tests/021-basic-wily-liberty' (properties changed: -x to +x)
=== modified file 'tests/basic_deployment.py'
--- tests/basic_deployment.py	2015-10-19 09:32:10 +0000
+++ tests/basic_deployment.py	2015-10-19 14:53:22 +0000
@@ -1,5 +1,3 @@
-#!/usr/bin/python
-
 import amulet
 import os
 import time
@@ -47,7 +45,9 @@
         other_services = [{'name': 'mysql'},
                           {'name': 'rabbitmq-server'},
                           {'name': 'keystone'},
+                          {'name': 'glance'},  # satisfy workload status
                           {'name': 'nova-cloud-controller'},
+                          {'name': 'nova-compute'},  # satisfy workload stat
                           {'name': 'neutron-api'}]
 
         super(NeutronGatewayBasicDeployment, self)._add_services(
@@ -68,7 +68,15 @@
             'neutron-api:shared-db': 'mysql:shared-db',
             'neutron-api:amqp': 'rabbitmq-server:amqp',
             'neutron-api:neutron-api': 'nova-cloud-controller:neutron-api',
-            'neutron-api:identity-service': 'keystone:identity-service'
+            'neutron-api:identity-service': 'keystone:identity-service',
+            'glance:identity-service': 'keystone:identity-service',
+            'glance:shared-db': 'mysql:shared-db',
+            'glance:amqp': 'rabbitmq-server:amqp',
+            'nova-cloud-controller:cloud-compute': 'nova-compute:'
+                                                   'cloud-compute',
+            'nova-compute:amqp': 'rabbitmq-server:amqp',
+            'nova-compute:image-service': 'glance:image-service',
+            'nova-cloud-controller:image-service': 'glance:image-service',
         }
         super(NeutronGatewayBasicDeployment, self)._add_relations(relations)
 
@@ -947,6 +955,7 @@
         self.neutron.create_network({'network': network})
 
         networks = self.neutron.list_networks(name=net_name)
+        u.log.debug('Networks: {}'.format(networks))
         net_len = len(networks['networks'])
         if net_len != 1:
             msg = "Expected 1 network, found {}".format(net_len)

=== modified file 'tests/charmhelpers/contrib/openstack/amulet/deployment.py'
--- tests/charmhelpers/contrib/openstack/amulet/deployment.py	2015-09-28 10:06:18 +0000
+++ tests/charmhelpers/contrib/openstack/amulet/deployment.py	2015-10-19 14:53:22 +0000
@@ -14,6 +14,7 @@
 # You should have received a copy of the GNU Lesser General Public License
 # along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
 
+import re
 import six
 from collections import OrderedDict
 from charmhelpers.contrib.amulet.deployment import (
@@ -114,6 +115,45 @@
         for service, config in six.iteritems(configs):
             self.d.configure(service, config)
 
+    def _auto_wait_for_status(self, message=None, exclude_services=None,
+                              timeout=1800):
+        """Wait for all units to have a specific extended status, except
+        for any defined as excluded.  Unless specified via message, any
+        status containing any case of 'ready' will be considered a match.
+
+        Examples of message usage:
+
+          Wait for all unit status to CONTAIN any case of 'ready' or 'ok':
+              message = re.compile('.*ready.*|.*ok.*', re.IGNORECASE)
+
+          Wait for all units to reach this status (exact match):
+              message = 'Unit is ready'
+
+          Wait for all units to reach any one of these (exact match):
+              message = re.compile('Unit is ready|OK|Ready')
+
+          Wait for at least one unit to reach this status (exact match):
+              message = {'ready'}
+
+        See Amulet's sentry.wait_for_messages() for message usage detail.
+        https://github.com/juju/amulet/blob/master/amulet/sentry.py
+
+        :param message: Expected status match
+        :param exclude_services: List of juju service names to ignore
+        :param timeout: Maximum time in seconds to wait for status match
+        :returns: None.  Raises if timeout is hit.
+        """
+
+        if not message:
+            message = re.compile('.*ready.*', re.IGNORECASE)
+
+        if not exclude_services:
+            exclude_services = []
+
+        services = list(set(self.d.services.keys()) - set(exclude_services))
+        service_messages = {service: message for service in services}
+        self.d.sentry.wait_for_messages(service_messages, timeout=timeout)
+
     def _get_openstack_release(self):
         """Get openstack release.