Merge lp:~1chb1n/charms/trusty/neutron-api/next-amulet-15.10 into lp:~openstack-charmers-archive/charms/trusty/neutron-api/next

[Ryan Beisner]

Update amulet tests for Trusty-Liberty, Wily-Liberty.

Sync charmhelpers.

Add service and relations to satisfy workload status ready state.

Add new logic to wait for extended status message to confirm deploy is ready, before testing.

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #12024 neutron-api-next for 1chb1n mp274709
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/12024/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #11175 neutron-api-next for 1chb1n mp274709
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/11175/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7386 neutron-api-next for 1chb1n mp274709
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12799468/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7386/

[lp:~1chb1n/charms/trusty/neutron-api/next-amulet-15.10]

157. By Ryan Beisner on 2015-10-16

fix relation check, remove varying unit data check

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #12028 neutron-api-next for 1chb1n mp274709
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/12028/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #11177 neutron-api-next for 1chb1n mp274709
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/11177/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7391 neutron-api-next for 1chb1n mp274709
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/7391/

[Liam Young (gnuoy)]

Approve

review: Approve

=== modified file 'hooks/charmhelpers/contrib/openstack/amulet/deployment.py'
--- hooks/charmhelpers/contrib/openstack/amulet/deployment.py	2015-09-25 14:35:35 +0000
+++ hooks/charmhelpers/contrib/openstack/amulet/deployment.py	2015-10-16 16:14:02 +0000
@@ -14,6 +14,7 @@
 # You should have received a copy of the GNU Lesser General Public License
 # along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
 
+import re
 import six
 from collections import OrderedDict
 from charmhelpers.contrib.amulet.deployment import (
@@ -114,6 +115,45 @@
         for service, config in six.iteritems(configs):
             self.d.configure(service, config)
 
+    def _auto_wait_for_status(self, message=None, exclude_services=None,
+                              timeout=1800):
+        """Wait for all units to have a specific extended status, except
+        for any defined as excluded.  Unless specified via message, any
+        status containing any case of 'ready' will be considered a match.
+
+        Examples of message usage:
+
+          Wait for all unit status to CONTAIN any case of 'ready' or 'ok':
+              message = re.compile('.*ready.*|.*ok.*', re.IGNORECASE)
+
+          Wait for all units to reach this status (exact match):
+              message = 'Unit is ready'
+
+          Wait for all units to reach any one of these (exact match):
+              message = re.compile('Unit is ready|OK|Ready')
+
+          Wait for at least one unit to reach this status (exact match):
+              message = {'ready'}
+
+        See Amulet's sentry.wait_for_messages() for message usage detail.
+        https://github.com/juju/amulet/blob/master/amulet/sentry.py
+
+        :param message: Expected status match
+        :param exclude_services: List of juju service names to ignore
+        :param timeout: Maximum time in seconds to wait for status match
+        :returns: None.  Raises if timeout is hit.
+        """
+
+        if not message:
+            message = re.compile('.*ready.*', re.IGNORECASE)
+
+        if not exclude_services:
+            exclude_services = []
+
+        services = list(set(self.d.services.keys()) - set(exclude_services))
+        service_messages = {service: message for service in services}
+        self.d.sentry.wait_for_messages(service_messages, timeout=timeout)
+
     def _get_openstack_release(self):
         """Get openstack release.
 
 
=== modified file 'hooks/charmhelpers/contrib/openstack/amulet/utils.py'
--- hooks/charmhelpers/contrib/openstack/amulet/utils.py	2015-09-25 14:35:35 +0000
+++ hooks/charmhelpers/contrib/openstack/amulet/utils.py	2015-10-16 16:14:02 +0000
@@ -752,7 +752,7 @@
             self.log.debug('SSL is enabled  @{}:{} '
                            '({})'.format(host, port, unit_name))
             return True
-        elif not port and not conf_ssl:
+        elif not conf_ssl:
             self.log.debug('SSL not enabled @{}:{} '
                            '({})'.format(host, port, unit_name))
             return False
 
=== modified file 'hooks/charmhelpers/contrib/openstack/context.py'
--- hooks/charmhelpers/contrib/openstack/context.py	2015-09-28 19:06:04 +0000
+++ hooks/charmhelpers/contrib/openstack/context.py	2015-10-16 16:14:02 +0000
@@ -14,6 +14,7 @@
 # You should have received a copy of the GNU Lesser General Public License
 # along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
 
+import glob
 import json
 import os
 import re
@@ -951,6 +952,19 @@
                     'config': config}
         return ovs_ctxt
 
+    def midonet_ctxt(self):
+        driver = neutron_plugin_attribute(self.plugin, 'driver',
+                                          self.network_manager)
+        midonet_config = neutron_plugin_attribute(self.plugin, 'config',
+                                                  self.network_manager)
+        mido_ctxt = {'core_plugin': driver,
+                     'neutron_plugin': 'midonet',
+                     'neutron_security_groups': self.neutron_security_groups,
+                     'local_ip': unit_private_ip(),
+                     'config': midonet_config}
+
+        return mido_ctxt
+
     def __call__(self):
         if self.network_manager not in ['quantum', 'neutron']:
             return {}
@@ -972,6 +986,8 @@
             ctxt.update(self.nuage_ctxt())
         elif self.plugin == 'plumgrid':
             ctxt.update(self.pg_ctxt())
+        elif self.plugin == 'midonet':
+            ctxt.update(self.midonet_ctxt())
 
         alchemy_flags = config('neutron-alchemy-flags')
         if alchemy_flags:
@@ -1104,7 +1120,7 @@
 
         ctxt = {
             ... other context ...
-            'subordinate_config': {
+            'subordinate_configuration': {
                 'DEFAULT': {
                     'key1': 'value1',
                 },
@@ -1145,22 +1161,23 @@
                     try:
                         sub_config = json.loads(sub_config)
                     except:
-                        log('Could not parse JSON from subordinate_config '
-                            'setting from %s' % rid, level=ERROR)
+                        log('Could not parse JSON from '
+                            'subordinate_configuration setting from %s'
+                            % rid, level=ERROR)
                         continue
 
                     for service in self.services:
                         if service not in sub_config:
-                            log('Found subordinate_config on %s but it contained'
-                                'nothing for %s service' % (rid, service),
-                                level=INFO)
+                            log('Found subordinate_configuration on %s but it '
+                                'contained nothing for %s service'
+                                % (rid, service), level=INFO)
                             continue
 
                         sub_config = sub_config[service]
                         if self.config_file not in sub_config:
-                            log('Found subordinate_config on %s but it contained'
-                                'nothing for %s' % (rid, self.config_file),
-                                level=INFO)
+                            log('Found subordinate_configuration on %s but it '
+                                'contained nothing for %s'
+                                % (rid, self.config_file), level=INFO)
                             continue
 
                         sub_config = sub_config[self.config_file]
@@ -1363,7 +1380,7 @@
             normalized.update({port: port for port in resolved
                                if port in ports})
             if resolved:
-                return {bridge: normalized[port] for port, bridge in
+                return {normalized[port]: bridge for port, bridge in
                         six.iteritems(portmap) if port in normalized.keys()}
 
         return None
@@ -1374,12 +1391,22 @@
     def __call__(self):
         ctxt = {}
         mappings = super(PhyNICMTUContext, self).__call__()
-        if mappings and mappings.values():
-            ports = mappings.values()
+        if mappings and mappings.keys():
+            ports = sorted(mappings.keys())
             napi_settings = NeutronAPIContext()()
             mtu = napi_settings.get('network_device_mtu')
+            all_ports = set()
+            # If any of ports is a vlan device, its underlying device must have
+            # mtu applied first.
+            for port in ports:
+                for lport in glob.glob("/sys/class/net/%s/lower_*" % port):
+                    lport = os.path.basename(lport)
+                    all_ports.add(lport.split('_')[1])
+
+            all_ports = list(all_ports)
+            all_ports.extend(ports)
             if mtu:
-                ctxt["devs"] = '\\n'.join(ports)
+                ctxt["devs"] = '\\n'.join(all_ports)
                 ctxt['mtu'] = mtu
 
         return ctxt
 
=== modified file 'hooks/charmhelpers/contrib/openstack/neutron.py'
--- hooks/charmhelpers/contrib/openstack/neutron.py	2015-09-04 11:03:14 +0000
+++ hooks/charmhelpers/contrib/openstack/neutron.py	2015-10-16 16:14:02 +0000
@@ -209,6 +209,20 @@
             'server_packages': ['neutron-server',
                                 'neutron-plugin-plumgrid'],
             'server_services': ['neutron-server']
+        },
+        'midonet': {
+            'config': '/etc/neutron/plugins/midonet/midonet.ini',
+            'driver': 'midonet.neutron.plugin.MidonetPluginV2',
+            'contexts': [
+                context.SharedDBContext(user=config('neutron-database-user'),
+                                        database=config('neutron-database'),
+                                        relation_prefix='neutron',
+                                        ssl_dir=NEUTRON_CONF_DIR)],
+            'services': [],
+            'packages': [[headers_package()] + determine_dkms_package()],
+            'server_packages': ['neutron-server',
+                                'python-neutron-plugin-midonet'],
+            'server_services': ['neutron-server']
         }
     }
     if release >= 'icehouse':
@@ -310,10 +324,10 @@
 def parse_data_port_mappings(mappings, default_bridge='br-data'):
     """Parse data port mappings.
 
-    Mappings must be a space-delimited list of port:bridge mappings.
+    Mappings must be a space-delimited list of bridge:port.
 
-    Returns dict of the form {port:bridge} where port may be an mac address or
-    interface name.
+    Returns dict of the form {port:bridge} where ports may be mac addresses or
+    interface names.
     """
 
     # NOTE(dosaboy): we use rvalue for key to allow multiple values to be
 
=== modified file 'hooks/charmhelpers/contrib/openstack/templates/ceph.conf'
--- hooks/charmhelpers/contrib/openstack/templates/ceph.conf	2015-07-16 20:17:53 +0000
+++ hooks/charmhelpers/contrib/openstack/templates/ceph.conf	2015-10-16 16:14:02 +0000
@@ -13,3 +13,9 @@
 err to syslog = {{ use_syslog }}
 clog to syslog = {{ use_syslog }}
 
+[client]
+{% if rbd_client_cache_settings -%}
+{% for key, value in rbd_client_cache_settings.iteritems() -%}
+{{ key }} = {{ value }}
+{% endfor -%}
+{%- endif %}
 \ No newline at end of file
 
=== modified file 'hooks/charmhelpers/contrib/openstack/utils.py'
--- hooks/charmhelpers/contrib/openstack/utils.py	2015-10-06 07:11:01 +0000
+++ hooks/charmhelpers/contrib/openstack/utils.py	2015-10-16 16:14:02 +0000
@@ -121,6 +121,7 @@
     ('2.2.2', 'kilo'),
     ('2.3.0', 'liberty'),
     ('2.4.0', 'liberty'),
+    ('2.5.0', 'liberty'),
 ])
 
 # >= Liberty version->codename mapping
 
=== modified file 'hooks/charmhelpers/core/hookenv.py'
--- hooks/charmhelpers/core/hookenv.py	2015-09-28 19:06:04 +0000
+++ hooks/charmhelpers/core/hookenv.py	2015-10-16 16:14:02 +0000
@@ -795,7 +795,6 @@
             raise
     log_message = 'status-set failed: {} {}'.format(workload_state,
                                                     message)
-    # XXX Fix this
     log(log_message, level='INFO')
 
 
 
=== modified file 'hooks/charmhelpers/core/host.py'
--- hooks/charmhelpers/core/host.py	2015-09-25 14:35:35 +0000
+++ hooks/charmhelpers/core/host.py	2015-10-16 16:14:02 +0000
@@ -566,7 +566,14 @@
         os.chdir(cur)
 
 
-def chownr(path, owner, group, follow_links=True):
+def chownr(path, owner, group, follow_links=True, chowntopdir=False):
+    """
+    Recursively change user and group ownership of files and directories
+    in given path. Doesn't chown path itself by default, only its children.
+
+    :param bool follow_links: Also Chown links if True
+    :param bool chowntopdir: Also chown path itself if True
+    """
     uid = pwd.getpwnam(owner).pw_uid
     gid = grp.getgrnam(group).gr_gid
     if follow_links:
@@ -574,6 +581,10 @@
     else:
         chown = os.lchown
 
+    if chowntopdir:
+        broken_symlink = os.path.lexists(path) and not os.path.exists(path)
+        if not broken_symlink:
+            chown(path, uid, gid)
     for root, dirs, files in os.walk(path):
         for name in dirs + files:
             full = os.path.join(root, name)
 
=== modified file 'tests/020-basic-trusty-liberty' (properties changed: -x to +x)
=== modified file 'tests/021-basic-wily-liberty' (properties changed: -x to +x)
=== modified file 'tests/basic_deployment.py'
--- tests/basic_deployment.py	2015-09-28 19:06:04 +0000
+++ tests/basic_deployment.py	2015-10-16 16:14:02 +0000
@@ -1,11 +1,5 @@
-#!/usr/bin/python
-"""
-Basic neutron-api functional test.
-"""
-
 import amulet
 import os
-import time
 import yaml
 
 from charmhelpers.contrib.openstack.amulet.deployment import (
@@ -35,6 +29,11 @@
         self._add_relations()
         self._configure_services()
         self._deploy()
+
+        u.log.info('Waiting on extended status checks...')
+        exclude_services = ['mysql']
+        self._auto_wait_for_status(exclude_services=exclude_services)
+
         self._initialize_tests()
 
     def _add_services(self):
@@ -48,6 +47,7 @@
         other_services = [{'name': 'mysql'},
                           {'name': 'rabbitmq-server'},
                           {'name': 'keystone'},
+                          {'name': 'glance'},  # to satisfy workload status
                           {'name': 'neutron-openvswitch'},
                           {'name': 'nova-cloud-controller'},
                           {'name': 'neutron-gateway'},
@@ -68,6 +68,19 @@
             'nova-compute:neutron-plugin': 'neutron-openvswitch:'
                                            'neutron-plugin',
             'nova-cloud-controller:shared-db': 'mysql:shared-db',
+            'neutron-gateway:amqp': 'rabbitmq-server:amqp',
+            'nova-cloud-controller:amqp': 'rabbitmq-server:amqp',
+            'nova-compute:amqp': 'rabbitmq-server:amqp',
+            'neutron-openvswitch:amqp': 'rabbitmq-server:amqp',
+            'nova-cloud-controller:identity-service': 'keystone:'
+                                                      'identity-service',
+            'nova-cloud-controller:cloud-compute': 'nova-compute:'
+                                                   'cloud-compute',
+            'glance:identity-service': 'keystone:identity-service',
+            'glance:shared-db': 'mysql:shared-db',
+            'glance:amqp': 'rabbitmq-server:amqp',
+            'nova-compute:image-service': 'glance:image-service',
+            'nova-cloud-controller:image-service': 'glance:image-service',
         }
 
         # NOTE(beisner): relate this separately due to the resulting
@@ -158,8 +171,6 @@
             self._get_openstack_release()))
         u.log.debug('openstack release str: {}'.format(
             self._get_openstack_release_string()))
-        # Let things settle a bit before moving forward
-        time.sleep(30)
 
     def test_100_services(self):
         """Verify the expected services are running on the corresponding
@@ -226,13 +237,6 @@
             'password': u.not_null
         }
 
-        if self._get_openstack_release() == self.precise_icehouse:
-            # Precise
-            expected['allowed_units'] = 'nova-cloud-controller/0 neutron-api/0'
-        else:
-            # Not Precise
-            expected['allowed_units'] = 'neutron-api/0'
-
         ret = u.validate_relation_data(unit, relation, expected)
         if ret:
             message = u.relation_error('mysql shared-db', ret)
 
=== modified file 'tests/charmhelpers/contrib/openstack/amulet/deployment.py'
--- tests/charmhelpers/contrib/openstack/amulet/deployment.py	2015-09-25 14:35:35 +0000
+++ tests/charmhelpers/contrib/openstack/amulet/deployment.py	2015-10-16 16:14:02 +0000
@@ -14,6 +14,7 @@
 # You should have received a copy of the GNU Lesser General Public License
 # along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
 
+import re
 import six
 from collections import OrderedDict
 from charmhelpers.contrib.amulet.deployment import (
@@ -114,6 +115,45 @@
         for service, config in six.iteritems(configs):
             self.d.configure(service, config)
 
+    def _auto_wait_for_status(self, message=None, exclude_services=None,
+                              timeout=1800):
+        """Wait for all units to have a specific extended status, except
+        for any defined as excluded.  Unless specified via message, any
+        status containing any case of 'ready' will be considered a match.
+
+        Examples of message usage:
+
+          Wait for all unit status to CONTAIN any case of 'ready' or 'ok':
+              message = re.compile('.*ready.*|.*ok.*', re.IGNORECASE)
+
+          Wait for all units to reach this status (exact match):
+              message = 'Unit is ready'
+
+          Wait for all units to reach any one of these (exact match):
+              message = re.compile('Unit is ready|OK|Ready')
+
+          Wait for at least one unit to reach this status (exact match):
+              message = {'ready'}
+
+        See Amulet's sentry.wait_for_messages() for message usage detail.
+        https://github.com/juju/amulet/blob/master/amulet/sentry.py
+
+        :param message: Expected status match
+        :param exclude_services: List of juju service names to ignore
+        :param timeout: Maximum time in seconds to wait for status match
+        :returns: None.  Raises if timeout is hit.
+        """
+
+        if not message:
+            message = re.compile('.*ready.*', re.IGNORECASE)
+
+        if not exclude_services:
+            exclude_services = []
+
+        services = list(set(self.d.services.keys()) - set(exclude_services))
+        service_messages = {service: message for service in services}
+        self.d.sentry.wait_for_messages(service_messages, timeout=timeout)
+
     def _get_openstack_release(self):
         """Get openstack release.
 
 
=== modified file 'tests/charmhelpers/contrib/openstack/amulet/utils.py'
--- tests/charmhelpers/contrib/openstack/amulet/utils.py	2015-09-25 14:35:35 +0000
+++ tests/charmhelpers/contrib/openstack/amulet/utils.py	2015-10-16 16:14:02 +0000
@@ -752,7 +752,7 @@
             self.log.debug('SSL is enabled  @{}:{} '
                            '({})'.format(host, port, unit_name))
             return True
-        elif not port and not conf_ssl:
+        elif not conf_ssl:
             self.log.debug('SSL not enabled @{}:{} '
                            '({})'.format(host, port, unit_name))
             return False