Merge lp:~ddellav/charms/trusty/nova-compute/upgrade-action into lp:~openstack-charmers-archive/charms/trusty/nova-compute/next

[David Della Vecchia]

Adding charmhelpers-based upgrade action to nova-compute.

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #11392 nova-compute-next for ddellav mp273560
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/11392/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #10584 nova-compute-next for ddellav mp273560
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/10584/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #10585 nova-compute-next for ddellav mp273560
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/10585/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #11394 nova-compute-next for ddellav mp273560
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/11394/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7141 nova-compute-next for ddellav mp273560
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12698231/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7141/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7140 nova-compute-next for ddellav mp273560
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12698727/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7140/

[Corey Bryant (corey.bryant)]

Reviewed and tested successfully. I approve. Just waiting on amulet tests to pass before landing.

review: Approve

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #11425 nova-compute-next for ddellav mp273560
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/11425/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #10616 nova-compute-next for ddellav mp273560
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/10616/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7168 nova-compute-next for ddellav mp273560
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [test] Error 124
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12700486/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7168/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #10733 nova-compute-next for ddellav mp273560
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/10733/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #11542 nova-compute-next for ddellav mp273560
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/11542/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7243 nova-compute-next for ddellav mp273560
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12718961/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7243/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7258 nova-compute-next for ddellav mp273560
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12720684/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7258/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #10797 nova-compute-next for ddellav mp273560
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/10797/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #11608 nova-compute-next for ddellav mp273560
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/11608/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #11618 nova-compute-next for ddellav mp273560
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/11618/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #10807 nova-compute-next for ddellav mp273560
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/10807/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #7266 nova-compute-next for ddellav mp273560
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12724034/
Build: http://10.245.162.77:8080/job/charm_amulet_test/7266/

[Corey Bryant (corey.bryant)]

The amulet failure is not related. Something is wrong with kilo deploying from source. So I'm going to merge this.

=== modified file 'actions.yaml'
--- actions.yaml	2015-04-15 14:21:42 +0000
+++ actions.yaml	2015-10-08 15:08:00 +0000
@@ -1,2 +1,4 @@
 git-reinstall:
     description: Reinstall nova-compute from the openstack-origin-git repositories.
+openstack-upgrade:
+  description: Perform openstack upgrades. Config option action-managed-upgrade must be set to True.
 
=== added symlink 'actions/openstack-upgrade'
=== target is u'openstack_upgrade.py'
=== added file 'actions/openstack_upgrade.py'
--- actions/openstack_upgrade.py	1970-01-01 00:00:00 +0000
+++ actions/openstack_upgrade.py	2015-10-08 15:08:00 +0000
@@ -0,0 +1,32 @@
+#!/usr/bin/python
+import sys
+
+sys.path.append('hooks/')
+
+from charmhelpers.contrib.openstack.utils import (
+    do_action_openstack_upgrade,
+)
+
+from nova_compute_utils import do_openstack_upgrade
+
+from nova_compute_hooks import (
+    config_changed,
+    CONFIGS
+)
+
+
+def openstack_upgrade():
+    """Upgrade packages to config-set Openstack version.
+
+    If the charm was installed from source we cannot upgrade it.
+    For backwards compatibility a config flag must be set for this
+    code to run, otherwise a full service level upgrade will fire
+    on config-changed."""
+
+    if (do_action_openstack_upgrade('nova-common',
+                                    do_openstack_upgrade,
+                                    CONFIGS)):
+        config_changed()
+
+if __name__ == '__main__':
+    openstack_upgrade()
 
=== modified file 'config.yaml'
--- config.yaml	2015-10-02 09:40:38 +0000
+++ config.yaml	2015-10-08 15:08:00 +0000
@@ -265,3 +265,13 @@
     description: |
       The pecentage of system memory to use for hugepages eg '10%' or the total
       number of 2M hugepages - eg "1024".
+  action-managed-upgrade:
+    type: boolean
+    default: False
+    description: |
+      If True enables openstack upgrades for this charm via juju actions.
+      You will still need to set openstack-origin to the new repository but
+      instead of an upgrade running automatically across all units, it will
+      wait for you to execute the openstack-upgrade action for this charm on
+      each unit. If False it will revert to existing behavior of upgrading
+      all units on config change.
 
=== modified file 'hooks/charmhelpers/contrib/network/ip.py'
--- hooks/charmhelpers/contrib/network/ip.py	2015-09-02 15:03:16 +0000
+++ hooks/charmhelpers/contrib/network/ip.py	2015-10-08 15:08:00 +0000
@@ -23,7 +23,7 @@
 from functools import partial
 
 from charmhelpers.core.hookenv import unit_get
-from charmhelpers.fetch import apt_install
+from charmhelpers.fetch import apt_install, apt_update
 from charmhelpers.core.hookenv import (
     log,
     WARNING,
@@ -32,13 +32,15 @@
 try:
     import netifaces
 except ImportError:
-    apt_install('python-netifaces')
+    apt_update(fatal=True)
+    apt_install('python-netifaces', fatal=True)
     import netifaces
 
 try:
     import netaddr
 except ImportError:
-    apt_install('python-netaddr')
+    apt_update(fatal=True)
+    apt_install('python-netaddr', fatal=True)
     import netaddr
 
 
 
=== modified file 'hooks/charmhelpers/contrib/openstack/amulet/utils.py'
--- hooks/charmhelpers/contrib/openstack/amulet/utils.py	2015-09-28 07:56:16 +0000
+++ hooks/charmhelpers/contrib/openstack/amulet/utils.py	2015-10-08 15:08:00 +0000
@@ -752,7 +752,7 @@
             self.log.debug('SSL is enabled  @{}:{} '
                            '({})'.format(host, port, unit_name))
             return True
-        elif not port and not conf_ssl:
+        elif not conf_ssl:
             self.log.debug('SSL not enabled @{}:{} '
                            '({})'.format(host, port, unit_name))
             return False
 
=== modified file 'hooks/charmhelpers/contrib/openstack/context.py'
--- hooks/charmhelpers/contrib/openstack/context.py	2015-09-28 07:56:16 +0000
+++ hooks/charmhelpers/contrib/openstack/context.py	2015-10-08 15:08:00 +0000
@@ -14,6 +14,7 @@
 # You should have received a copy of the GNU Lesser General Public License
 # along with charm-helpers.  If not, see <http://www.gnu.org/licenses/>.
 
+import glob
 import json
 import os
 import re
@@ -1363,7 +1364,7 @@
             normalized.update({port: port for port in resolved
                                if port in ports})
             if resolved:
-                return {bridge: normalized[port] for port, bridge in
+                return {normalized[port]: bridge for port, bridge in
                         six.iteritems(portmap) if port in normalized.keys()}
 
         return None
@@ -1374,12 +1375,22 @@
     def __call__(self):
         ctxt = {}
         mappings = super(PhyNICMTUContext, self).__call__()
-        if mappings and mappings.values():
-            ports = mappings.values()
+        if mappings and mappings.keys():
+            ports = sorted(mappings.keys())
             napi_settings = NeutronAPIContext()()
             mtu = napi_settings.get('network_device_mtu')
+            all_ports = set()
+            # If any of ports is a vlan device, its underlying device must have
+            # mtu applied first.
+            for port in ports:
+                for lport in glob.glob("/sys/class/net/%s/lower_*" % port):
+                    lport = os.path.basename(lport)
+                    all_ports.add(lport.split('_')[1])
+
+            all_ports = list(all_ports)
+            all_ports.extend(ports)
             if mtu:
-                ctxt["devs"] = '\\n'.join(ports)
+                ctxt["devs"] = '\\n'.join(all_ports)
                 ctxt['mtu'] = mtu
 
         return ctxt
 
=== modified file 'hooks/charmhelpers/contrib/openstack/neutron.py'
--- hooks/charmhelpers/contrib/openstack/neutron.py	2015-08-25 17:40:00 +0000
+++ hooks/charmhelpers/contrib/openstack/neutron.py	2015-10-08 15:08:00 +0000
@@ -310,10 +310,10 @@
 def parse_data_port_mappings(mappings, default_bridge='br-data'):
     """Parse data port mappings.
 
-    Mappings must be a space-delimited list of port:bridge mappings.
+    Mappings must be a space-delimited list of bridge:port.
 
-    Returns dict of the form {port:bridge} where port may be an mac address or
-    interface name.
+    Returns dict of the form {port:bridge} where ports may be mac addresses or
+    interface names.
     """
 
     # NOTE(dosaboy): we use rvalue for key to allow multiple values to be
 
=== modified file 'hooks/charmhelpers/core/hookenv.py'
--- hooks/charmhelpers/core/hookenv.py	2015-09-28 07:56:16 +0000
+++ hooks/charmhelpers/core/hookenv.py	2015-10-08 15:08:00 +0000
@@ -623,6 +623,38 @@
     return unit_get('private-address')
 
 
+@cached
+def storage_get(attribute="", storage_id=""):
+    """Get storage attributes"""
+    _args = ['storage-get', '--format=json']
+    if storage_id:
+        _args.extend(('-s', storage_id))
+    if attribute:
+        _args.append(attribute)
+    try:
+        return json.loads(subprocess.check_output(_args).decode('UTF-8'))
+    except ValueError:
+        return None
+
+
+@cached
+def storage_list(storage_name=""):
+    """List the storage IDs for the unit"""
+    _args = ['storage-list', '--format=json']
+    if storage_name:
+        _args.append(storage_name)
+    try:
+        return json.loads(subprocess.check_output(_args).decode('UTF-8'))
+    except ValueError:
+        return None
+    except OSError as e:
+        import errno
+        if e.errno == errno.ENOENT:
+            # storage-list does not exist
+            return []
+        raise
+
+
 class UnregisteredHookError(Exception):
     """Raised when an undefined hook is called"""
     pass
 
=== modified file 'hooks/nova_compute_hooks.py'
--- hooks/nova_compute_hooks.py	2015-10-07 20:45:02 +0000
+++ hooks/nova_compute_hooks.py	2015-10-08 15:08:00 +0000
@@ -117,10 +117,10 @@
         if config_value_changed('openstack-origin-git'):
             status_set('maintenance', 'Running Git install')
             git_install(config('openstack-origin-git'))
-    else:
+    elif not config('action-managed-upgrade'):
         if openstack_upgrade_available('nova-common'):
             status_set('maintenance', 'Running openstack upgrade')
-            CONFIGS = do_openstack_upgrade()
+            do_openstack_upgrade(CONFIGS)
 
     sysctl_dict = config('sysctl')
     if sysctl_dict:
 
=== modified file 'hooks/nova_compute_utils.py'
--- hooks/nova_compute_utils.py	2015-10-03 00:00:02 +0000
+++ hooks/nova_compute_utils.py	2015-10-08 15:08:00 +0000
@@ -537,7 +537,7 @@
             _keys.write('{}\n'.format(authorized_keys[index]))
 
 
-def do_openstack_upgrade():
+def do_openstack_upgrade(configs):
     # NOTE(jamespage) horrible hack to make utils forget a cached value
     import charmhelpers.contrib.openstack.utils as utils
     utils.os_rel = None
@@ -557,10 +557,8 @@
     apt_install(determine_packages(), fatal=True)
 
     # Regenerate configs in full for new release
-    configs = register_configs()
     configs.write_all()
     [service_restart(s) for s in services()]
-    return configs
 
 
 def import_keystone_ca_cert():
 
=== modified file 'tests/charmhelpers/contrib/amulet/utils.py'
--- tests/charmhelpers/contrib/amulet/utils.py	2015-09-28 07:56:16 +0000
+++ tests/charmhelpers/contrib/amulet/utils.py	2015-10-08 15:08:00 +0000
@@ -326,7 +326,7 @@
 
     def service_restarted_since(self, sentry_unit, mtime, service,
                                 pgrep_full=None, sleep_time=20,
-                                retry_count=2, retry_sleep_time=30):
+                                retry_count=30, retry_sleep_time=10):
         """Check if service was been started after a given time.
 
         Args:
@@ -334,8 +334,9 @@
           mtime (float): The epoch time to check against
           service (string): service name to look for in process table
           pgrep_full: [Deprecated] Use full command line search mode with pgrep
-          sleep_time (int): Seconds to sleep before looking for process
-          retry_count (int): If service is not found, how many times to retry
+          sleep_time (int): Initial sleep time (s) before looking for file
+          retry_sleep_time (int): Time (s) to sleep between retries
+          retry_count (int): If file is not found, how many times to retry
 
         Returns:
           bool: True if service found and its start time it newer than mtime,
@@ -359,11 +360,12 @@
                                                             pgrep_full)
                 self.log.debug('Attempt {} to get {} proc start time on {} '
                                'OK'.format(tries, service, unit_name))
-            except IOError:
+            except IOError as e:
                 # NOTE(beisner) - race avoidance, proc may not exist yet.
                 # https://bugs.launchpad.net/charm-helpers/+bug/1474030
                 self.log.debug('Attempt {} to get {} proc start time on {} '
-                               'failed'.format(tries, service, unit_name))
+                               'failed\n{}'.format(tries, service,
+                                                   unit_name, e))
                 time.sleep(retry_sleep_time)
                 tries += 1
 
@@ -383,35 +385,62 @@
             return False
 
     def config_updated_since(self, sentry_unit, filename, mtime,
-                             sleep_time=20):
+                             sleep_time=20, retry_count=30,
+                             retry_sleep_time=10):
         """Check if file was modified after a given time.
 
         Args:
           sentry_unit (sentry): The sentry unit to check the file mtime on
           filename (string): The file to check mtime of
           mtime (float): The epoch time to check against
-          sleep_time (int): Seconds to sleep before looking for process
+          sleep_time (int): Initial sleep time (s) before looking for file
+          retry_sleep_time (int): Time (s) to sleep between retries
+          retry_count (int): If file is not found, how many times to retry
 
         Returns:
           bool: True if file was modified more recently than mtime, False if
-                file was modified before mtime,
+                file was modified before mtime, or if file not found.
         """
-        self.log.debug('Checking %s updated since %s' % (filename, mtime))
+        unit_name = sentry_unit.info['unit_name']
+        self.log.debug('Checking that %s updated since %s on '
+                       '%s' % (filename, mtime, unit_name))
         time.sleep(sleep_time)
-        file_mtime = self._get_file_mtime(sentry_unit, filename)
+        file_mtime = None
+        tries = 0
+        while tries <= retry_count and not file_mtime:
+            try:
+                file_mtime = self._get_file_mtime(sentry_unit, filename)
+                self.log.debug('Attempt {} to get {} file mtime on {} '
+                               'OK'.format(tries, filename, unit_name))
+            except IOError as e:
+                # NOTE(beisner) - race avoidance, file may not exist yet.
+                # https://bugs.launchpad.net/charm-helpers/+bug/1474030
+                self.log.debug('Attempt {} to get {} file mtime on {} '
+                               'failed\n{}'.format(tries, filename,
+                                                   unit_name, e))
+                time.sleep(retry_sleep_time)
+                tries += 1
+
+        if not file_mtime:
+            self.log.warn('Could not determine file mtime, assuming '
+                          'file does not exist')
+            return False
+
         if file_mtime >= mtime:
             self.log.debug('File mtime is newer than provided mtime '
-                           '(%s >= %s)' % (file_mtime, mtime))
+                           '(%s >= %s) on %s (OK)' % (file_mtime,
+                                                      mtime, unit_name))
             return True
         else:
-            self.log.warn('File mtime %s is older than provided mtime %s'
-                          % (file_mtime, mtime))
+            self.log.warn('File mtime is older than provided mtime'
+                          '(%s < on %s) on %s' % (file_mtime,
+                                                  mtime, unit_name))
             return False
 
     def validate_service_config_changed(self, sentry_unit, mtime, service,
                                         filename, pgrep_full=None,
-                                        sleep_time=20, retry_count=2,
-                                        retry_sleep_time=30):
+                                        sleep_time=20, retry_count=30,
+                                        retry_sleep_time=10):
         """Check service and file were updated after mtime
 
         Args:
@@ -456,7 +485,9 @@
             sentry_unit,
             filename,
             mtime,
-            sleep_time=0)
+            sleep_time=sleep_time,
+            retry_count=retry_count,
+            retry_sleep_time=retry_sleep_time)
 
         return service_restart and config_update
 
 
=== modified file 'tests/charmhelpers/contrib/openstack/amulet/utils.py'
--- tests/charmhelpers/contrib/openstack/amulet/utils.py	2015-09-28 07:56:16 +0000
+++ tests/charmhelpers/contrib/openstack/amulet/utils.py	2015-10-08 15:08:00 +0000
@@ -752,7 +752,7 @@
             self.log.debug('SSL is enabled  @{}:{} '
                            '({})'.format(host, port, unit_name))
             return True
-        elif not port and not conf_ssl:
+        elif not conf_ssl:
             self.log.debug('SSL not enabled @{}:{} '
                            '({})'.format(host, port, unit_name))
             return False
 
=== added file 'unit_tests/test_actions_openstack_upgrade.py'
--- unit_tests/test_actions_openstack_upgrade.py	1970-01-01 00:00:00 +0000
+++ unit_tests/test_actions_openstack_upgrade.py	2015-10-08 15:08:00 +0000
@@ -0,0 +1,62 @@
+from mock import patch
+import os
+
+os.environ['JUJU_UNIT_NAME'] = 'nova_compute'
+
+with patch('charmhelpers.core.hookenv.config') as config:
+    config.return_value = 'nova'
+    import nova_compute_utils as utils  # noqa
+
+with patch('nova_compute_utils.restart_map'):
+    with patch('nova_compute_utils.register_configs'):
+        import openstack_upgrade
+
+from test_utils import (
+    CharmTestCase
+)
+
+TO_PATCH = [
+    'config_changed',
+    'do_openstack_upgrade'
+]
+
+
+class TestNovaComputeUpgradeActions(CharmTestCase):
+
+    def setUp(self):
+        super(TestNovaComputeUpgradeActions, self).setUp(openstack_upgrade,
+                                                         TO_PATCH)
+
+    @patch('charmhelpers.contrib.openstack.utils.config')
+    @patch('charmhelpers.contrib.openstack.utils.action_set')
+    @patch('charmhelpers.contrib.openstack.utils.git_install_requested')
+    @patch('charmhelpers.contrib.openstack.utils.openstack_upgrade_available')
+    @patch('charmhelpers.contrib.openstack.utils.juju_log')
+    def test_openstack_upgrade_true(self, log, upgrade_avail, git_requested,
+                                    action_set, config):
+
+        git_requested.return_value = False
+        upgrade_avail.return_value = True
+        config.return_value = True
+
+        openstack_upgrade.openstack_upgrade()
+
+        self.assertTrue(self.do_openstack_upgrade.called)
+        self.assertTrue(self.config_changed.called)
+
+    @patch('charmhelpers.contrib.openstack.utils.config')
+    @patch('charmhelpers.contrib.openstack.utils.action_set')
+    @patch('charmhelpers.contrib.openstack.utils.git_install_requested')  # noqa
+    @patch('charmhelpers.contrib.openstack.utils.openstack_upgrade_available')  # noqa
+    @patch('charmhelpers.contrib.openstack.utils.juju_log')
+    def test_openstack_upgrade_false(self,  log, upgrade_avail, git_requested,
+                                     action_set, config):
+
+        git_requested.return_value = False
+        upgrade_avail.return_value = True
+        config.return_value = False
+
+        openstack_upgrade.openstack_upgrade()
+
+        self.assertFalse(self.do_openstack_upgrade.called)
+        self.assertFalse(self.config_changed.called)
 
=== modified file 'unit_tests/test_nova_compute_hooks.py'
--- unit_tests/test_nova_compute_hooks.py	2015-10-06 09:04:58 +0000
+++ unit_tests/test_nova_compute_hooks.py	2015-10-08 15:08:00 +0000
@@ -115,6 +115,15 @@
         hooks.config_changed()
         self.assertTrue(self.do_openstack_upgrade.called)
 
+    @patch.object(hooks, 'git_install_requested')
+    def test_config_changed_with_openstack_upgrade_action(self, git_requested):
+        git_requested.return_value = False
+        self.openstack_upgrade_available.return_value = True
+        self.test_config.set('action-managed-upgrade', True)
+
+        hooks.config_changed()
+        self.assertFalse(self.do_openstack_upgrade.called)
+
     @patch.object(hooks, 'compute_joined')
     def test_config_changed_with_migration(self, compute_joined):
         self.git_install_requested.return_value = False