Merge lp:~plumgrid-team/charms/trusty/plumgrid-edge/trunk into lp:charms/trusty/plumgrid-edge

[Bilal Baqar]

Needs sync

[lp:~plumgrid-team/charms/trusty/plumgrid-edge/trunk]

21. By Bilal Baqar on 2016-01-26

Merge: setting hostname of lxc to the hostname of the node

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/2221/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/2200/

review: Needs Fixing (automated testing)

[lp:~plumgrid-team/charms/trusty/plumgrid-edge/trunk]

22. By Bilal Baqar on 2016-03-03

Fixing lint

23. By Bilal Baqar on 2016-03-14

Made the following changes:
1. Reordered file and module imports
2. Sorted director IPs
3. Added unit fqdn in /etc/hosts of plumgrid-lxc
4. Loading plumgrid specific iptables on install
5. Added temporary upgrade hook to load iptables
6. stop_pg() is being used in restart_pg()
7. persistant iptables

24. By Bilal Baqar on 2016-03-25

Improved config-changed hook to perform steps according to the config changed

25. By Bilal Baqar on 2016-03-28

OPSVM Changes - Ticket: [SOL-830]
- Getting OPSVM IP from director relation
- Making OPSVM specific changes
- Cleaned code in various functions
- Added restart_on_change decorater function that restarts plumgrid service only when there has been any change in the configuration files
- Removed restart of plumgrid service when only two directors are available
- Fixed unit tests accordingly

26. By Bilal Baqar on 2016-04-08

Adding status messages in charms - Ticket:[SOL-949]

27. By Bilal Baqar on 2016-04-25

Merge: Liberty/Mitaka support

28. By Bilal Baqar on 2016-05-18

Merge - Charmhelpers sync and improved pg-restart

29. By Bilal Baqar on 2016-05-28

5.1 changes
- configure-pg-sources added
- updated templates

30. By Junaid Ali on 2016-06-19

L3 fabric changes

31. By Junaid Ali on 2016-07-28

Changes:
   - Default value for management interface removed as
     'juju-br0' has been changed to 'br-eth0' in Juju 2.0
   - Updated get_mgmt_interface method

Unmerged revisions

31. By Junaid Ali on 2016-07-28

Changes:
   - Default value for management interface removed as
     'juju-br0' has been changed to 'br-eth0' in Juju 2.0
   - Updated get_mgmt_interface method

=== modified file 'Makefile'
--- Makefile	2015-08-24 16:22:15 +0000
+++ Makefile	2016-03-03 21:45:16 +0000
@@ -7,7 +7,7 @@
 	netaddr jinja2
 
 lint: virtualenv
-	.venv/bin/flake8 --exclude hooks/charmhelpers hooks unit_tests tests
+	.venv/bin/flake8 --exclude hooks/charmhelpers hooks unit_tests tests --ignore E402
 	@charm proof
 
 unit_test: virtualenv

=== modified file 'config.yaml'
--- config.yaml	2015-08-09 15:48:20 +0000
+++ config.yaml	2016-03-03 21:45:16 +0000
@@ -3,6 +3,28 @@
     default: 'null'
     type: string
     description: Public SSH key of PLUMgrid LCM which is running PG-Tools
+  mgmt-interface:
+    type: string
+    default: 'juju-br0'
+    description: The interface connected to PLUMgrid Managment network.
+  os-data-network:
+    type: string
+    default:
+    description: |
+      The IP address and netmask of the OpenStack Data network (e.g.,
+      192.168.0.0/24)
+      .
+      This network will be used for tenant network traffic in overlay
+      networks.
+  fabric-interfaces:
+    default: 'MANAGEMENT'
+    type: string
+    description: |
+       Interfaces that will provide fabric connectivity on the director nodes.
+       Provided in form of json in a string. These interfaces have to be connected 
+       to the os-data-network specified in the config. Default value is MANAGEMENT which
+       will configure the management interface as the fabric interface on each
+       director.
   network-device-mtu:
     type: string
     default: '1580'
@@ -15,6 +37,16 @@
     default: null
     type: string
     description: Provide the respective keys of the install sources
+  plumgrid-build:
+    default: 'latest'
+    type: string
+    description: |
+       Provide the build version of PLUMgrid packages that needs to be installed
+  iovisor-build:
+    default: 'latest'
+    type: string
+    description: |
+       Provide the build version of iovisor package that needs to be installed
   metadata-shared-key:
     default: 'plumgrid'
     type: string

=== modified file 'hooks/charmhelpers/contrib/openstack/neutron.py'
--- hooks/charmhelpers/contrib/openstack/neutron.py	2015-07-29 18:19:18 +0000
+++ hooks/charmhelpers/contrib/openstack/neutron.py	2016-03-03 21:45:16 +0000
@@ -204,8 +204,8 @@
                                         database=config('database'),
                                         ssl_dir=NEUTRON_CONF_DIR)],
             'services': [],
-            'packages': [['plumgrid-lxc'],
-                         ['iovisor-dkms']],
+            'packages': ['plumgrid-lxc',
+                         'iovisor-dkms'],
             'server_packages': ['neutron-server',
                                 'neutron-plugin-plumgrid'],
             'server_services': ['neutron-server']

=== modified file 'hooks/pg_edge_context.py'
--- hooks/pg_edge_context.py	2015-08-24 16:22:15 +0000
+++ hooks/pg_edge_context.py	2016-03-03 21:45:16 +0000
@@ -9,7 +9,7 @@
     relation_get,
 )
 from charmhelpers.contrib.openstack import context
-
+from charmhelpers.contrib.openstack.utils import get_host_ip
 from socket import gethostname as get_unit_hostname
 
 
@@ -21,7 +21,7 @@
     for rid in relation_ids('plumgrid'):
         for unit in related_units(rid):
             rdata = relation_get(rid=rid, unit=unit)
-            director_ips.append(str(rdata['private-address']))
+            director_ips.append(str(get_host_ip(rdata['private-address'])))
     return director_ips
 
 
@@ -69,11 +69,12 @@
             else:
                 pg_dir_ips = pg_dir_ips + ',' + str(ip)
         pg_ctxt['local_ip'] = pg_dir_ips
-        pg_ctxt['pg_hostname'] = "pg-edge"
-        from pg_edge_utils import check_interface_type
-        interface_type = check_interface_type()
-        pg_ctxt['interface'] = interface_type
-        pg_ctxt['label'] = get_unit_hostname()
+        unit_hostname = get_unit_hostname()
+        pg_ctxt['pg_hostname'] = unit_hostname
+        from pg_edge_utils import get_mgmt_interface, get_fabric_interface
+        pg_ctxt['interface'] = get_mgmt_interface()
+        pg_ctxt['fabric_interface'] = get_fabric_interface()
+        pg_ctxt['label'] = unit_hostname
         pg_ctxt['fabric_mode'] = 'host'
 
         return pg_ctxt

=== modified file 'hooks/pg_edge_hooks.py'
--- hooks/pg_edge_hooks.py	2015-08-16 19:07:22 +0000
+++ hooks/pg_edge_hooks.py	2016-03-03 21:45:16 +0000
@@ -32,6 +32,7 @@
     remove_iovisor,
     ensure_mtu,
     add_lcm_key,
+    fabric_interface_changed
 )
 
 hooks = Hooks()
@@ -89,11 +90,21 @@
     if add_lcm_key():
         log("PLUMgrid LCM Key added")
         return 1
+    charm_config = config()
+    if charm_config.changed('fabric-interfaces'):
+        if not fabric_interface_changed():
+            log("Fabric interface already set")
+            return 1
+    if charm_config.changed('os-data-network'):
+        if charm_config['fabric-interfaces'] == 'MANAGEMENT':
+            log('Fabric running on managment network')
+            return 1
     stop_pg()
     configure_sources(update=True)
     pkgs = determine_packages()
     for pkg in pkgs:
         apt_install(pkg, options=['--force-yes'], fatal=True)
+    remove_iovisor()
     load_iovisor()
     ensure_mtu()
     for rid in relation_ids('neutron-plugin'):

=== modified file 'hooks/pg_edge_utils.py'
--- hooks/pg_edge_utils.py	2015-11-12 11:55:13 +0000
+++ hooks/pg_edge_utils.py	2016-03-03 21:45:16 +0000
@@ -7,6 +7,14 @@
 from charmhelpers.core.hookenv import (
     log,
     config,
+    unit_get
+)
+from charmhelpers.contrib.network.ip import (
+    get_iface_from_addr,
+    get_bridges,
+    get_bridge_nics,
+    is_address_in_network,
+    get_iface_addr
 )
 from charmhelpers.core.host import (
     write_file,
@@ -14,6 +22,9 @@
     service_start,
     service_stop,
 )
+from charmhelpers.fetch import (
+    apt_cache
+)
 from charmhelpers.contrib.storage.linux.ceph import modprobe
 from charmhelpers.core.host import set_nic_mtu
 from charmhelpers.fetch import apt_install
@@ -22,11 +33,12 @@
 from charmhelpers.contrib.openstack.utils import (
     os_release,
 )
+from socket import gethostname as get_unit_hostname
 import pg_edge_context
 import subprocess
 import time
 import os
-import re
+import json
 
 SHARED_SECRET = "/etc/nova/secret.txt"
 LXC_CONF = '/etc/libvirt/lxc.conf'
@@ -72,7 +84,25 @@
     Returns list of packages required by PLUMgrid Edge as specified
     in the neutron_plugins dictionary in charmhelpers.
     '''
-    return neutron_plugin_attribute('plumgrid', 'packages', 'neutron')
+    pkgs = []
+    tag = 'latest'
+    for pkg in neutron_plugin_attribute('plumgrid', 'packages', 'neutron'):
+        if 'plumgrid' in pkg:
+            tag = config('plumgrid-build')
+        elif pkg == 'iovisor-dkms':
+            tag = config('iovisor-build')
+
+        if tag == 'latest':
+            pkgs.append(pkg)
+        else:
+            if tag in [i.ver_str for i in apt_cache()[pkg].version_list]:
+                pkgs.append('%s=%s' % (pkg, tag))
+            else:
+                error_msg = \
+                    "Build version '%s' for package '%s' not available" \
+                    % (tag, pkg)
+                raise ValueError(error_msg)
+    return pkgs
 
 
 def register_configs(release=None):
@@ -81,6 +111,9 @@
     the context required for all templates of this charm.
     '''
     release = release or os_release('nova-compute', base='kilo')
+    if release < 'kilo':
+        raise ValueError('OpenStack %s release not supported' % release)
+
     configs = templating.OSConfigRenderer(templates_dir=TEMPLATES,
                                           openstack_release=release)
     for cfg, rscs in resource_map().iteritems():
@@ -151,52 +184,116 @@
     '''
     _exec_cmd(cmd=['rmmod', 'iovisor'],
               error_msg='Error Loading Iovisor Kernel Module')
-
-
-def check_interface_type():
-    '''
-    Checks the interface. Support added for AWS deployments. There are 2
-    possible interfaces "juju-br0" and "eth0". The default being juju-br0
-    '''
-    log("Checking Interface Type")
-    default_interface = "juju-br0"
-    AWS_interface = "eth0"
-    shell_output = subprocess.check_output(['brctl', 'show', 'juju-br0'])
-    output = re.split(' |\n|\t', shell_output)
-    if output[10] == '':
-        return AWS_interface
-    else:
-        return default_interface
+    time.sleep(1)
+
+
+def interface_exists(interface):
+    '''
+    Checks if interface exists on node.
+    '''
+    try:
+        subprocess.check_call(['ip', 'link', 'show', interface],
+                              stdout=open(os.devnull, 'w'),
+                              stderr=subprocess.STDOUT)
+    except subprocess.CalledProcessError:
+        return False
+    return True
+
+
+def get_mgmt_interface():
+    '''
+    Returns the managment interface.
+    '''
+    mgmt_interface = config('mgmt-interface')
+    if interface_exists(mgmt_interface):
+        return mgmt_interface
+    else:
+        log('Provided managment interface %s does not exist'
+            % mgmt_interface)
+        return get_iface_from_addr(unit_get('private-address'))
+
+
+def fabric_interface_changed():
+    '''
+    Returns true if interface for node changed.
+    '''
+    fabric_interface = get_fabric_interface()
+    try:
+        with open(PG_IFCS_CONF, 'r') as ifcs:
+            for line in ifcs:
+                if 'fabric_core' in line:
+                    if line.split()[0] == fabric_interface:
+                        return False
+    except IOError:
+        return True
+    return True
+
+
+def get_fabric_interface():
+    '''
+    Returns the fabric interface.
+    '''
+    fabric_interfaces = config('fabric-interfaces')
+    if fabric_interfaces == 'MANAGEMENT':
+        return get_mgmt_interface()
+    else:
+        try:
+            all_fabric_interfaces = json.loads(fabric_interfaces)
+        except ValueError:
+            raise ValueError('Invalid json provided for fabric interfaces')
+        hostname = get_unit_hostname()
+        if hostname in all_fabric_interfaces:
+            node_fabric_interface = all_fabric_interfaces[hostname]
+        elif 'DEFAULT' in all_fabric_interfaces:
+            node_fabric_interface = all_fabric_interfaces['DEFAULT']
+        else:
+            raise ValueError('No fabric interface provided for node')
+        if interface_exists(node_fabric_interface):
+            if is_address_in_network(config('os-data-network'),
+                                     get_iface_addr(node_fabric_interface)[0]):
+                return node_fabric_interface
+            else:
+                raise ValueError('Fabric interface not in fabric network')
+        else:
+            log('Provided fabric interface %s does not exist'
+                % node_fabric_interface)
+            raise ValueError('Provided fabric interface does not exist')
+        return node_fabric_interface
 
 
 def ensure_mtu():
     '''
     Ensures required MTU of the underlying networking of the node.
     '''
-    log("Changing MTU of juju-br0 and all attached interfaces")
     interface_mtu = config('network-device-mtu')
-    interface_type = check_interface_type()
-    if interface_type == "juju-br0":
-        cmd = subprocess.check_output(["brctl", "show", interface_type])
-        words = cmd.split()
-        for word in words:
-            if 'eth' in word:
-                set_nic_mtu(word, interface_mtu)
-    set_nic_mtu(interface_type, interface_mtu)
-
-
-def _exec_cmd(cmd=None, error_msg='Command exited with ERRORs', fatal=False):
+    fabric_interface = get_fabric_interface()
+    if fabric_interface in get_bridges():
+        attached_interfaces = get_bridge_nics(fabric_interface)
+        for interface in attached_interfaces:
+            set_nic_mtu(interface, interface_mtu)
+    set_nic_mtu(fabric_interface, interface_mtu)
+
+
+def _exec_cmd(cmd=None, error_msg='Command exited with ERRORs', fatal=False,
+              verbose=False):
     '''
     Function to execute any bash command on the node.
     '''
     if cmd is None:
         log("No command specified")
     else:
-        if fatal:
+        if fatal and verbose:
+            subprocess.check_call(cmd, stdout=open(os.devnull, 'w'),
+                                  stderr=subprocess.STDOUT)
+        elif fatal and not verbose:
             subprocess.check_call(cmd)
         else:
             try:
-                subprocess.check_call(cmd)
+                if verbose:
+                    subprocess.check_call(cmd, stdout=open(os.devnull, 'w'),
+                                          stderr=subprocess.STDOUT)
+                else:
+                    subprocess.check_call(cmd)
             except subprocess.CalledProcessError:
                 log(error_msg)
 
@@ -208,7 +305,8 @@
     apt_install('apparmor-utils')
     apt_install('cgroup-bin')
     _exec_cmd(['sudo', 'aa-disable', '/usr/sbin/libvirtd'],
-              error_msg='Error disabling AppArmor profile of libvirtd')
+              error_msg='Error disabling AppArmor profile of libvirtd',
+              verbose=True)
     disable_apparmor()
     service_restart('libvirt-bin')
 

=== modified file 'templates/kilo/ifcs.conf'
--- templates/kilo/ifcs.conf	2015-07-29 18:19:18 +0000
+++ templates/kilo/ifcs.conf	2016-03-03 21:45:16 +0000
@@ -1,2 +1,2 @@
-{{ interface }} = fabric_core host
+{{ fabric_interface }} = fabric_core host
 

=== added file 'tests/files/plumgrid-edge-dense.yaml'
--- tests/files/plumgrid-edge-dense.yaml	1970-01-01 00:00:00 +0000
+++ tests/files/plumgrid-edge-dense.yaml	2016-03-03 21:45:16 +0000
@@ -0,0 +1,133 @@
+test:
+  series: 'trusty'
+  relations:
+  - - mysql
+    - keystone
+  - - nova-cloud-controller
+    - mysql
+  - - nova-cloud-controller
+    - rabbitmq-server
+  - - nova-cloud-controller
+    - glance
+  - - nova-cloud-controller
+    - keystone
+  - - nova-compute
+    - nova-cloud-controller
+  - - nova-compute
+    - mysql
+  - - nova-compute
+    - rabbitmq-server
+  - - nova-compute
+    - glance
+  - - glance
+    - mysql
+  - - glance
+    - keystone
+  - - glance
+    - cinder
+  - - mysql
+    - cinder
+  - - cinder
+    - rabbitmq-server
+  - - cinder
+    - nova-cloud-controller
+  - - cinder
+    - keystone
+  - - openstack-dashboard
+    - keystone
+  - - neutron-api
+    - mysql
+  - - neutron-api
+    - keystone
+  - - neutron-api
+    - rabbitmq-server
+  - - neutron-api
+    - nova-cloud-controller
+  - - neutron-api
+    - neutron-api-plumgrid
+  - - neutron-api-plumgrid
+    - plumgrid-edge
+  - - plumgrid-director
+    - plumgrid-edge
+  - - nova-compute
+    - plumgrid-edge
+  - - plumgrid-director
+    - plumgrid-gateway
+  services:
+    mysql:
+      charm: cs:trusty/mysql
+      num_units: 1
+      to: 'lxc:plumgrid-director=0'
+    rabbitmq-server:
+      charm: cs:trusty/rabbitmq-server
+      num_units: 1
+      to: 'lxc:plumgrid-director=0'
+    keystone:
+      charm: cs:trusty/keystone
+      num_units: 1
+      options:
+        admin-password: plumgrid
+        openstack-origin: cloud:trusty-kilo
+      to: 'lxc:plumgrid-director=0'
+    nova-cloud-controller:
+      charm: cs:trusty/nova-cloud-controller
+      num_units: 1
+      options:
+        console-access-protocol: novnc
+        network-manager: Neutron
+        openstack-origin: cloud:trusty-kilo
+        quantum-security-groups: 'yes'
+      to: 'lxc:plumgrid-director=0'
+    glance:
+      charm: cs:trusty/glance
+      num_units: 1
+      options:
+        openstack-origin: cloud:trusty-kilo
+      to: 'lxc:plumgrid-director=0'
+    openstack-dashboard:
+      charm: cs:trusty/openstack-dashboard
+      num_units: 1
+      options:
+        openstack-origin: cloud:trusty-kilo
+      to: 'lxc:plumgrid-director=0'
+    cinder:
+      charm: cs:trusty/cinder
+      num_units: 1
+      options:
+        openstack-origin: cloud:trusty-kilo
+      to: 'lxc:plumgrid-director=0'
+    neutron-api:
+      charm: cs:~plumgrid-team/trusty/neutron-api
+      num_units: 1
+      options:
+        neutron-plugin: plumgrid
+        neutron-security-groups: false
+        openstack-origin: cloud:trusty-kilo
+        plumgrid-password: plumgrid
+        plumgrid-username: plumgrid
+        plumgrid-virtual-ip: 192.168.100.250
+      to: 'lxc:plumgrid-director=0'
+    neutron-api-plumgrid:
+      charm: cs:~plumgrid-team/trusty/neutron-api-plumgrid
+      options:
+        enable-metadata: True
+    plumgrid-director:
+      charm: cs:~plumgrid-team/trusty/plumgrid-director
+      num_units: 1
+      constraints: "root-disk=30G mem=8G cpu-cores=8"
+      options:
+        plumgrid-virtual-ip: 192.168.100.250
+    nova-compute:
+      charm: cs:~plumgrid-team/trusty/nova-compute
+      num_units: 1
+      options:
+        enable-live-migration: true
+        enable-resize: true
+        migration-auth-type: ssh
+        openstack-origin: cloud:trusty-kilo
+      to: '0'
+    plumgrid-edge:
+      charm: cs:~plumgrid-team/trusty/plumgrid-edge
+    plumgrid-gateway:
+      charm: cs:~plumgrid-team/trusty/plumgrid-gateway
+      num_units: 1

=== renamed file 'tests/files/plumgrid-edge-dense.yaml' => 'tests/files/plumgrid-edge-dense.yaml.moved'
=== added file 'tests/tests.yaml'
--- tests/tests.yaml	1970-01-01 00:00:00 +0000
+++ tests/tests.yaml	2016-03-03 21:45:16 +0000
@@ -0,0 +1,2 @@
+makefile:
+  - lint

=== renamed file 'tests/tests.yaml' => 'tests/tests.yaml.moved'
=== modified file 'unit_tests/test_pg_edge_context.py'
--- unit_tests/test_pg_edge_context.py	2015-08-24 16:22:15 +0000
+++ unit_tests/test_pg_edge_context.py	2016-03-03 21:45:16 +0000
@@ -37,8 +37,10 @@
     @patch.object(context, '_pg_dir_settings')
     @patch.object(charmhelpers.contrib.openstack.context,
                   'neutron_plugin_attribute')
-    @patch.object(utils, 'check_interface_type')
-    def test_neutroncc_context_api_rel(self, _int_type, _npa, _pg_dir_settings,
+    @patch.object(utils, 'get_mgmt_interface')
+    @patch.object(utils, 'get_fabric_interface')
+    def test_neutroncc_context_api_rel(self, _fabric_int, _mgmt_int,
+                                       _npa, _pg_dir_settings,
                                        _save_flag_file, _config_flag,
                                        _unit_get, _unit_priv_ip, _config,
                                        _is_clus, _https, _ens_pkgs):
@@ -64,7 +66,8 @@
         _is_clus.return_value = False
         _config_flag.return_value = False
         _pg_dir_settings.return_value = {'pg_dir_ip': '192.168.100.201'}
-        _int_type.return_value = 'juju-br0'
+        _mgmt_int.return_value = 'juju-br0'
+        _fabric_int.return_value = 'juju-br0'
         napi_ctxt = context.PGEdgeContext()
         expect = {
             'config': 'neutron.randomconfig',
@@ -74,8 +77,9 @@
             'neutron_plugin': 'plumgrid',
             'neutron_security_groups': None,
             'neutron_url': 'https://192.168.100.201:9696',
-            'pg_hostname': 'pg-edge',
+            'pg_hostname': 'node0',
             'interface': 'juju-br0',
+            'fabric_interface': 'juju-br0',
             'label': 'node0',
             'fabric_mode': 'host',
             'neutron_alchemy_flags': False,

=== modified file 'unit_tests/test_pg_edge_hooks.py'
--- unit_tests/test_pg_edge_hooks.py	2015-08-16 19:07:22 +0000
+++ unit_tests/test_pg_edge_hooks.py	2016-03-03 21:45:16 +0000
@@ -83,21 +83,8 @@
         )
 
     def test_config_changed_hook(self):
-        _pkgs = ['plumgrid-lxc', 'iovisor-dkms']
-        self.add_lcm_key.return_value = 0
-        self.determine_packages.return_value = [_pkgs]
+        self.add_lcm_key.return_value = 1
         self._call_hook('config-changed')
-        self.stop_pg.assert_called_with()
-        self.configure_sources.assert_called_with(update=True)
-        self.apt_install.assert_has_calls([
-            call(_pkgs, fatal=True,
-                 options=['--force-yes']),
-        ])
-        self.load_iovisor.assert_called_with()
-        self.ensure_mtu.assert_called_with()
-        self.ensure_files.assert_called_with()
-        self.CONFIGS.write_all.assert_called_with()
-        self.restart_pg.assert_called_with()
 
     def test_stop(self):
         _pkgs = ['plumgrid-lxc', 'iovisor-dkms']