Merge lp:~plumgrid-team/charms/trusty/plumgrid-gateway/trunk into lp:charms/trusty/plumgrid-gateway

[Bilal Baqar]

Needs sync

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/2222/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/2201/

review: Needs Fixing (automated testing)

[lp:~plumgrid-team/charms/trusty/plumgrid-gateway/trunk]

21. By Bilal Baqar on 2016-03-03

Fixing lint

[Bilal Baqar (bbaqar)]

tests/files/plumgrid-edge-dense.yaml and tests/tests.yaml in both branches are identical. Dont know the reason for the conflict. Please resolve it before merging.

[lp:~plumgrid-team/charms/trusty/plumgrid-gateway/trunk]

22. By Bilal Baqar on 2016-03-14

Made the following changes:
1. Reordered file and module imports
2. Sorted director IPs
3. Added unit fqdn in /etc/hosts of plumgrid-lxc
4. Loading plumgrid specific iptables on install
5. Added temporary upgrade hook to load iptables
6. stop_pg() is being used in restart_pg()
7. persistant iptables

23. By Bilal Baqar on 2016-03-25

Improved config-changed hook to perform steps according to the config changed

24. By Bilal Baqar on 2016-03-28

OPSVM Changes - Ticket: [SOL-830]
- Getting OPSVM IP from director relation
- Making OPSVM specific changes
- Cleaned code in various functions
- Added restart_on_change decorater function that restarts plumgrid service only when there has been any change in the configuration files
- Removed restart of plumgrid service when only two directors are available
- Fixed unit tests accordingly

25. By Bilal Baqar on 2016-03-28

restart_on_stop Ticket: [SOL-950]
- Added restart_on_stop decorator function which starts plumgrid service is it has been stopped in the function
- Added restart_on_change to config-changed hook so that any config changes result in plugrid restart
- ifc_list_gateway only removed when change in ifcs.conf

26. By Bilal Baqar on 2016-04-08

Adding status messages in charms - Ticket:[SOL-949]

27. By Bilal Baqar on 2016-04-25

Merge: Liberty/Mitaka support

28. By Bilal Baqar on 2016-05-18

Merge - Charmhelpers sync and improved pg-restart

29. By Bilal Baqar on 2016-05-28

5.1 changes
- configure-pg-sources added
- updated templates

30. By Junaid Ali on 2016-06-19

L3 fabric changes

31. By Junaid Ali on 2016-07-28

Changes:
    - Default value for management interface removed as
      'juju-br0' has been changed to 'br-eth0' in Juju 2.0
    - Updated get_mgmt_interface method

Unmerged revisions

31. By Junaid Ali on 2016-07-28

Changes:
    - Default value for management interface removed as
      'juju-br0' has been changed to 'br-eth0' in Juju 2.0
    - Updated get_mgmt_interface method

=== modified file 'Makefile'
--- Makefile	2015-08-24 16:24:20 +0000
+++ Makefile	2016-04-08 08:36:52 +0000
@@ -7,7 +7,7 @@
         netaddr jinja2
 
 lint: virtualenv
-	.venv/bin/flake8 --exclude hooks/charmhelpers hooks unit_tests tests
+	.venv/bin/flake8 --exclude hooks/charmhelpers hooks unit_tests tests --ignore E402
 	@charm proof
 
 unit_test: virtualenv

=== modified file 'README.md'
--- README.md	2015-09-01 18:08:51 +0000
+++ README.md	2016-04-08 08:36:52 +0000
@@ -36,7 +36,7 @@
 Example Config
 
     plumgrid-gateway:
-        external-interface: eth1
+        external-interfaces: '{"node01":"eth5,eth2","node02":"eth4,eth8"}'
         install_sources: 'ppa:plumgrid-team/stable'
         install_keys: 'null'
     plumgrid-edge:
@@ -54,7 +54,7 @@
         neutron-plugin: "plumgrid"
         plumgrid-virtual-ip: "192.168.100.250"
 
-The "external-interface" config parameter should be the interface that will provide external connectivity. 
+The "external-interfaces" config parameter should be the interfaces that will provide external connectivity on each of the gateway nodes. Should be provided as a json in a string with hostname and interface names.
 Provide the source repo path for PLUMgrid Debs in 'install_sources' and the corresponding keys in 'install_keys'.
 The virtual IP passed on in the neutron-api charm has to be same as the one passed in the plumgrid-director charm.
 

=== modified file 'config.yaml'
--- config.yaml	2015-07-29 18:23:55 +0000
+++ config.yaml	2016-04-08 08:36:52 +0000
@@ -1,16 +1,40 @@
 options:
-  external-interface:
-    default: eth1
+  external-interfaces:
+    default: '{"hostname":"eth1,eth2"}'
     type: string
-    description: The interface that will provide external connectivity
+    description: |
+       One or multiple interfaces that will provide external connectivity on
+       each of the gateway nodes. Provided in form of json in a string.
   lcm-ssh-key:
     default: 'null'
     type: string
     description: Public SSH key of PLUMgrid LCM which is running PG-Tools
+  mgmt-interface:
+    type: string
+    default: 'juju-br0'
+    description: The interface connected to PLUMgrid Managment network.
+  os-data-network:
+    type: string
+    default:
+    description: |
+      The IP address and netmask of the OpenStack Data network (e.g.,
+      192.168.0.0/24)
+      .
+      This network will be used for tenant network traffic in overlay
+      networks.
+  fabric-interfaces:
+    default: 'MANAGEMENT'
+    type: string
+    description: |
+       Interfaces that will provide fabric connectivity on the gateway nodes.
+       Provided in form of json in a string. These interfaces have to be connected 
+       to the os-data-network specified in the config. Default value is MANAGEMENT which
+       will configure the management interface as the fabric interface on each
+       director.
   network-device-mtu:
     type: string
     default: '1580'
-    description: The MTU size for interfaces managed by director.
+    description: The MTU size for interfaces managed by gateway.
   install_sources:
     default: 'ppa:plumgrid-team/stable'
     type: string
@@ -19,3 +43,13 @@
     default: null
     type: string
     description: Provide the respective keys of the install sources
+  plumgrid-build:
+    default: 'latest'
+    type: string
+    description: |
+       Provide the build version of PLUMgrid packages that needs to be installed
+  iovisor-build:
+    default: 'latest'
+    type: string
+    description: |
+       Provide the build version of iovisor package that needs to be installed

=== modified file 'hooks/charmhelpers/contrib/openstack/neutron.py'
--- hooks/charmhelpers/contrib/openstack/neutron.py	2015-07-29 18:23:55 +0000
+++ hooks/charmhelpers/contrib/openstack/neutron.py	2016-04-08 08:36:52 +0000
@@ -204,8 +204,8 @@
                                         database=config('database'),
                                         ssl_dir=NEUTRON_CONF_DIR)],
             'services': [],
-            'packages': [['plumgrid-lxc'],
-                         ['iovisor-dkms']],
+            'packages': ['plumgrid-lxc',
+                         'iovisor-dkms'],
             'server_packages': ['neutron-server',
                                 'neutron-plugin-plumgrid'],
             'server_services': ['neutron-server']

=== modified file 'hooks/pg_gw_context.py'
--- hooks/pg_gw_context.py	2015-08-24 16:24:20 +0000
+++ hooks/pg_gw_context.py	2016-04-08 08:36:52 +0000
@@ -3,27 +3,36 @@
 # This file contains the class that generates context for
 # PLUMgrid template files.
 
+from charmhelpers.contrib.openstack import context
+from charmhelpers.contrib.openstack.utils import get_host_ip
 from charmhelpers.core.hookenv import (
     relation_ids,
     related_units,
     relation_get,
-    config,
-)
-from charmhelpers.contrib.openstack import context
-
-from socket import gethostname as get_unit_hostname
-
-
-def _pg_dir_settings():
+)
+from socket import (
+    gethostname,
+    getfqdn
+)
+
+
+def _pg_dir_context():
     '''
     Inspects relation with PLUMgrid director.
     '''
-    director_ips = []
+    ctxt = {
+        'opsvm_ip': '127.0.0.1',
+        'director_ips': [],
+    }
     for rid in relation_ids('plumgrid'):
         for unit in related_units(rid):
             rdata = relation_get(rid=rid, unit=unit)
-            director_ips.append(str(rdata['private-address']))
-    return director_ips
+            ctxt['director_ips'
+                 ].append(str(get_host_ip(rdata['private-address'])))
+            if "opsvm_ip" in rdata:
+                ctxt['opsvm_ip'] = \
+                    rdata['opsvm_ip']
+    return ctxt
 
 
 class PGGwContext(context.NeutronContext):
@@ -60,25 +69,29 @@
         if not pg_ctxt:
             return {}
 
-        conf = config()
-        pg_dir_ips = ''
-        pg_dir_settings = _pg_dir_settings()
-        single_ip = True
-        for ip in pg_dir_settings:
-            if single_ip:
-                pg_dir_ips = str(ip)
-                single_ip = False
-            else:
-                pg_dir_ips = pg_dir_ips + ',' + str(ip)
-        pg_ctxt['local_ip'] = pg_dir_ips
-        unit_hostname = get_unit_hostname()
+        pg_dir_context = _pg_dir_context()
+        pg_dir_ips = sorted(pg_dir_context['director_ips'])
+        dir_count = len(pg_dir_ips)
+        pg_ctxt['director_ips_string'] = (str(pg_dir_ips[0]) + ',' +
+                                          str(pg_dir_ips[1]) + ',' +
+                                          str(pg_dir_ips[2])
+                                          if dir_count == 3 else
+                                          str(pg_dir_ips[0])
+                                          if dir_count == 1 else
+                                          '')
+        unit_hostname = gethostname()
         pg_ctxt['pg_hostname'] = unit_hostname
-        from pg_gw_utils import check_interface_type
-        interface_type = check_interface_type()
-        pg_ctxt['interface'] = interface_type
+        pg_ctxt['pg_fqdn'] = getfqdn()
+        from pg_gw_utils import (
+            get_mgmt_interface,
+            get_gw_interfaces,
+            get_fabric_interface
+        )
+        pg_ctxt['interface'] = get_mgmt_interface()
+        pg_ctxt['fabric_interface'] = get_fabric_interface()
         pg_ctxt['label'] = unit_hostname
         pg_ctxt['fabric_mode'] = 'host'
-
-        pg_ctxt['ext_interface'] = conf['external-interface']
+        pg_ctxt['ext_interfaces'] = get_gw_interfaces()
+        pg_ctxt['opsvm_ip'] = pg_dir_context['opsvm_ip']
 
         return pg_ctxt

=== modified file 'hooks/pg_gw_hooks.py'
--- hooks/pg_gw_hooks.py	2015-08-16 19:08:38 +0000
+++ hooks/pg_gw_hooks.py	2016-04-08 08:36:52 +0000
@@ -6,29 +6,35 @@
 # in this file.
 
 import sys
-
+from charmhelpers.core.host import service_running
 from charmhelpers.core.hookenv import (
     Hooks,
     UnregisteredHookError,
     log,
+    config,
+    status_set
 )
 
 from charmhelpers.fetch import (
     apt_install,
-    apt_purge,
     configure_sources,
 )
 
 from pg_gw_utils import (
     register_configs,
     ensure_files,
-    restart_pg,
+    restart_map,
     stop_pg,
     determine_packages,
     load_iovisor,
     remove_iovisor,
     ensure_mtu,
     add_lcm_key,
+    fabric_interface_changed,
+    load_iptables,
+    restart_on_change,
+    restart_on_stop,
+    director_cluster_ready
 )
 
 hooks = Hooks()
@@ -40,7 +46,10 @@
     '''
     Install hook is run when the charm is first deployed on a node.
     '''
+    status_set('maintenance', 'Executing pre-install')
+    load_iptables()
     configure_sources(update=True)
+    status_set('maintenance', 'Installing apt packages')
     pkgs = determine_packages()
     for pkg in pkgs:
         apt_install(pkg, options=['--force-yes'], fatal=True)
@@ -50,40 +59,54 @@
     add_lcm_key()
 
 
[e-mail]('plumgrid-relation-joined')
 @hooks.hook('plumgrid-relation-changed')
-def plumgrid_joined():
+@restart_on_change(restart_map())
+def plumgrid_changed():
     '''
     This hook is run when relation between plumgrid-gateway and
     plumgrid-director is made.
     '''
-    ensure_mtu()
-    ensure_files()
-    add_lcm_key()
-    CONFIGS.write_all()
-    restart_pg()
+    if director_cluster_ready():
+        ensure_mtu()
+        CONFIGS.write_all()
 
 
 @hooks.hook('config-changed')
+@restart_on_stop()
+@restart_on_change(restart_map())
 def config_changed():
     '''
     This hook is run when a config parameter is changed.
     It also runs on node reboot.
     '''
-    if add_lcm_key():
-        log("PLUMgrid LCM Key added")
-        return 1
-    stop_pg()
-    configure_sources(update=True)
-    pkgs = determine_packages()
-    for pkg in pkgs:
-        apt_install(pkg, options=['--force-yes'], fatal=True)
-    load_iovisor()
-    ensure_mtu()
-    ensure_files()
-    add_lcm_key()
-    CONFIGS.write_all()
-    restart_pg()
+    charm_config = config()
+    if charm_config.changed('lcm-ssh-key'):
+        if add_lcm_key():
+            log("PLUMgrid LCM Key added")
+    if charm_config.changed('fabric-interfaces'):
+        if not fabric_interface_changed():
+            log("Fabric interface already set")
+    if (charm_config.changed('install_sources') or
+        charm_config.changed('plumgrid-build') or
+        charm_config.changed('install_keys') or
+            charm_config.changed('iovisor-build')):
+        stop_pg()
+        status_set('maintenance', 'Upgrading apt packages')
+        configure_sources(update=True)
+        pkgs = determine_packages()
+        for pkg in pkgs:
+            apt_install(pkg, options=['--force-yes'], fatal=True)
+            remove_iovisor()
+            load_iovisor()
+    ensure_mtu()
+    CONFIGS.write_all()
+
+
[e-mail]('upgrade-charm')
+@restart_on_change(restart_map())
+def upgrade_charm():
+    ensure_mtu()
+    CONFIGS.write_all()
 
 
 @hooks.hook('stop')
@@ -92,10 +115,14 @@
     This hook is run when the charm is destroyed.
     '''
     stop_pg()
-    remove_iovisor()
-    pkgs = determine_packages()
-    for pkg in pkgs:
-        apt_purge(pkg, fatal=False)
+
+
[e-mail]('update-status')
+def update_status():
+    if service_running('plumgrid'):
+        status_set('active', 'Unit is ready')
+    else:
+        status_set('blocked', 'plumgrid service not running')
 
 
 def main():

=== modified file 'hooks/pg_gw_utils.py'
--- hooks/pg_gw_utils.py	2015-08-24 16:24:20 +0000
+++ hooks/pg_gw_utils.py	2016-04-08 08:36:52 +0000
@@ -2,40 +2,56 @@
 
 # This file contains functions used by the hooks to deploy PLUMgrid Gateway.
 
+import pg_gw_context
+import subprocess
+import time
+import os
+import json
+from collections import OrderedDict
+from socket import gethostname as get_unit_hostname
+from copy import deepcopy
 from charmhelpers.contrib.openstack.neutron import neutron_plugin_attribute
-from copy import deepcopy
+from charmhelpers.contrib.storage.linux.ceph import modprobe
+from charmhelpers.contrib.openstack import templating
 from charmhelpers.core.hookenv import (
     log,
     config,
+    unit_get,
+    status_set
+)
+from charmhelpers.contrib.network.ip import (
+    get_iface_from_addr,
+    get_bridges,
+    get_bridge_nics,
+    is_address_in_network,
+    get_iface_addr
 )
 from charmhelpers.core.host import (
     write_file,
     service_start,
     service_stop,
-)
-from charmhelpers.contrib.storage.linux.ceph import modprobe
-from charmhelpers.core.host import set_nic_mtu
-from charmhelpers.contrib.openstack import templating
-from collections import OrderedDict
+    service_running,
+    path_hash,
+    set_nic_mtu
+)
+from charmhelpers.fetch import (
+    apt_cache,
+    apt_install
+)
 from charmhelpers.contrib.openstack.utils import (
     os_release,
 )
-import pg_gw_context
-import subprocess
-import time
-import os
-import re
 
 LXC_CONF = "/etc/libvirt/lxc.conf"
 TEMPLATES = 'templates/'
 PG_LXC_DATA_PATH = '/var/lib/libvirt/filesystems/plumgrid-data'
-
 PG_CONF = '%s/conf/pg/plumgrid.conf' % PG_LXC_DATA_PATH
 PG_HN_CONF = '%s/conf/etc/hostname' % PG_LXC_DATA_PATH
 PG_HS_CONF = '%s/conf/etc/hosts' % PG_LXC_DATA_PATH
 PG_IFCS_CONF = '%s/conf/pg/ifcs.conf' % PG_LXC_DATA_PATH
+OPS_CONF = '%s/conf/etc/00-pg.conf' % PG_LXC_DATA_PATH
 AUTH_KEY_PATH = '%s/root/.ssh/authorized_keys' % PG_LXC_DATA_PATH
-
+IFC_LIST_GW = '/var/run/plumgrid/lxc/ifc_list_gateway'
 SUDOERS_CONF = '/etc/sudoers.d/ifc_ctl_sudoers'
 
 BASE_RESOURCE_MAP = OrderedDict([
@@ -51,6 +67,10 @@
         'services': ['plumgrid'],
         'contexts': [pg_gw_context.PGGwContext()],
     }),
+    (OPS_CONF, {
+        'services': ['plumgrid'],
+        'contexts': [pg_gw_context.PGGwContext()],
+    }),
     (PG_IFCS_CONF, {
         'services': [],
         'contexts': [pg_gw_context.PGGwContext()],
@@ -63,7 +83,25 @@
     Returns list of packages required by PLUMgrid Gateway as specified
     in the neutron_plugins dictionary in charmhelpers.
     '''
-    return neutron_plugin_attribute('plumgrid', 'packages', 'neutron')
+    pkgs = []
+    tag = 'latest'
+    for pkg in neutron_plugin_attribute('plumgrid', 'packages', 'neutron'):
+        if 'plumgrid' in pkg:
+            tag = config('plumgrid-build')
+        elif pkg == 'iovisor-dkms':
+            tag = config('iovisor-build')
+
+        if tag == 'latest':
+            pkgs.append(pkg)
+        else:
+            if tag in [i.ver_str for i in apt_cache()[pkg].version_list]:
+                pkgs.append('%s=%s' % (pkg, tag))
+            else:
+                error_msg = \
+                    "Build version '%s' for package '%s' not available" \
+                    % (tag, pkg)
+                raise ValueError(error_msg)
+    return pkgs
 
 
 def register_configs(release=None):
@@ -103,17 +141,27 @@
     write_file(SUDOERS_CONF,
                "\nnova ALL=(root) NOPASSWD: /opt/pg/bin/ifc_ctl_pp *\n",
                owner='root', group='root', perms=0o644)
+    _exec_cmd(cmd=['rm', '-f', IFC_LIST_GW])
 
 
 def restart_pg():
     '''
     Stops and Starts PLUMgrid service after flushing iptables.
     '''
-    service_stop('plumgrid')
-    time.sleep(2)
-    _exec_cmd(cmd=['iptables', '-F'])
+    stop_pg()
     service_start('plumgrid')
-    time.sleep(5)
+    time.sleep(3)
+    if not service_running('plumgrid'):
+        if service_running('libvirt-bin'):
+            raise ValueError("plumgrid service couldn't be started")
+        else:
+            if service_start('libvirt-bin'):
+                time.sleep(3)
+                if not service_running('plumgrid'):
+                    raise ValueError("plumgrid service couldn't be started")
+            else:
+                raise ValueError("libvirt-bin service couldn't be started")
+    status_set('active', 'Unit is ready')
 
 
 def stop_pg():
@@ -136,39 +184,118 @@
     Removes iovisor kernel module.
     '''
     _exec_cmd(cmd=['rmmod', 'iovisor'],
-              error_msg='Error Loading Iovisor Kernel Module')
-
-
-def check_interface_type():
-    '''
-    Checks the interface. Support added for AWS deployments. There are 2
-    possible interfaces "juju-br0" and "eth0". The default being juju-br0
-    '''
-    log("Checking Interface Type")
-    default_interface = "juju-br0"
-    AWS_interface = "eth0"
-    shell_output = subprocess.check_output(['brctl', 'show', 'juju-br0'])
-    output = re.split(' |\n|\t', shell_output)
-    if output[10] == '':
-        return AWS_interface
-    else:
-        return default_interface
+              error_msg='Error Removing IOVisor Kernel Module')
+    time.sleep(1)
+
+
+def interface_exists(interface):
+    '''
+    Checks if interface exists on node.
+    '''
+    try:
+        subprocess.check_call(['ip', 'link', 'show', interface],
+                              stdout=open(os.devnull, 'w'),
+                              stderr=subprocess.STDOUT)
+    except subprocess.CalledProcessError:
+        return False
+    return True
+
+
+def get_mgmt_interface():
+    '''
+    Returns the managment interface.
+    '''
+    mgmt_interface = config('mgmt-interface')
+    if interface_exists(mgmt_interface):
+        return mgmt_interface
+    else:
+        log('Provided managment interface %s does not exist'
+            % mgmt_interface)
+        return get_iface_from_addr(unit_get('private-address'))
+
+
+def fabric_interface_changed():
+    '''
+    Returns true if interface for node changed.
+    '''
+    fabric_interface = get_fabric_interface()
+    try:
+        with open(PG_IFCS_CONF, 'r') as ifcs:
+            for line in ifcs:
+                if 'fabric_core' in line:
+                    if line.split()[0] == fabric_interface:
+                        return False
+    except IOError:
+        return True
+    return True
+
+
+def get_fabric_interface():
+    '''
+    Returns the fabric interface.
+    '''
+    fabric_interfaces = config('fabric-interfaces')
+    if fabric_interfaces == 'MANAGEMENT':
+        return get_mgmt_interface()
+    else:
+        try:
+            all_fabric_interfaces = json.loads(fabric_interfaces)
+        except ValueError:
+            raise ValueError('Invalid json provided for fabric interfaces')
+        hostname = get_unit_hostname()
+        if hostname in all_fabric_interfaces:
+            node_fabric_interface = all_fabric_interfaces[hostname]
+        elif 'DEFAULT' in all_fabric_interfaces:
+            node_fabric_interface = all_fabric_interfaces['DEFAULT']
+        else:
+            raise ValueError('No fabric interface provided for node')
+        if interface_exists(node_fabric_interface):
+            if is_address_in_network(config('os-data-network'),
+                                     get_iface_addr(node_fabric_interface)[0]):
+                return node_fabric_interface
+            else:
+                raise ValueError('Fabric interface not in fabric network')
+        else:
+            log('Provided fabric interface %s does not exist'
+                % node_fabric_interface)
+            raise ValueError('Provided fabric interface does not exist')
+        return node_fabric_interface
+
+
+def get_gw_interfaces():
+    '''
+    Gateway node can have multiple interfaces. This function parses json
+    provided in config to get all gateway interfaces for this node.
+    '''
+    node_interfaces = []
+    try:
+        all_interfaces = json.loads(config('external-interfaces'))
+    except ValueError:
+        raise ValueError("Invalid json provided for gateway interfaces")
+    hostname = get_unit_hostname()
+    if hostname in all_interfaces:
+        node_interfaces = all_interfaces[hostname].split(',')
+    elif 'DEFAULT' in all_interfaces:
+        node_interfaces = all_interfaces['DEFAULT'].split(',')
+    for interface in node_interfaces:
+        if not interface_exists(interface):
+            log('Provided gateway interface %s does not exist'
+                % interface)
+            raise ValueError('Provided gateway interface does not exist')
+    return node_interfaces
 
 
 def ensure_mtu():
     '''
     Ensures required MTU of the underlying networking of the node.
     '''
-    log("Changing MTU of juju-br0 and all attached interfaces")
     interface_mtu = config('network-device-mtu')
-    interface_type = check_interface_type()
-    if interface_type == "juju-br0":
-        cmd = subprocess.check_output(["brctl", "show", interface_type])
-        words = cmd.split()
-        for word in words:
-            if 'eth' in word:
-                set_nic_mtu(word, interface_mtu)
-    set_nic_mtu(interface_type, interface_mtu)
+    fabric_interface = get_fabric_interface()
+    if fabric_interface in get_bridges():
+        attached_interfaces = get_bridge_nics(fabric_interface)
+        for interface in attached_interfaces:
+            set_nic_mtu(interface, interface_mtu)
+    set_nic_mtu(fabric_interface, interface_mtu)
 
 
 def _exec_cmd(cmd=None, error_msg='Command exited with ERRORs', fatal=False):
@@ -216,3 +343,84 @@
     fa.write('\n')
     fa.close()
     return 1
+
+
+def load_iptables():
+    '''
+    Loads iptables rules to allow all PLUMgrid communication.
+    '''
+    network = get_cidr_from_iface(get_mgmt_interface())
+    if network:
+        _exec_cmd(['sudo', 'iptables', '-A', 'INPUT', '-p', 'tcp',
+                   '-j', 'ACCEPT', '-s', network, '-d',
+                   network, '-m', 'state', '--state', 'NEW'])
+        _exec_cmd(['sudo', 'iptables', '-A', 'INPUT', '-p', 'udp', '-j',
+                   'ACCEPT', '-s', network, '-d', network,
+                   '-m', 'state', '--state', 'NEW'])
+        apt_install('iptables-persistent')
+
+
+def get_cidr_from_iface(interface):
+    '''
+    Determines Network CIDR from interface.
+    '''
+    if not interface:
+        return None
+    apt_install('ohai')
+    try:
+        os_info = subprocess.check_output(['ohai', '-l', 'fatal'])
+    except OSError:
+        log('Unable to get operating system information')
+        return None
+    try:
+        os_info_json = json.loads(os_info)
+    except ValueError:
+        log('Unable to determine network')
+        return None
+    device = os_info_json['network']['interfaces'].get(interface)
+    if device is not None:
+        if device.get('routes'):
+            routes = device['routes']
+            for net in routes:
+                if 'scope' in net:
+                    return net.get('destination')
+        else:
+            return None
+    else:
+        return None
+
+
+def director_cluster_ready():
+    dirs_count = len(pg_gw_context._pg_dir_context()['director_ips'])
+    return True if dirs_count == 1 or dirs_count == 3 else False
+
+
+def restart_on_stop():
+    """
+    Starts plumgrid service if it is stopped
+    """
+    def wrap(f):
+        def wrapped_f(*args, **kwargs):
+            f(*args, **kwargs)
+            if not service_running('plumgrid'):
+                restart_pg()
+        return wrapped_f
+    return wrap
+
+
+def restart_on_change(restart_map):
+    """
+    Restart services based on configuration files changing
+    """
+    def wrap(f):
+        def wrapped_f(*args, **kwargs):
+            checksums = {path: path_hash(path) for path in restart_map}
+            f(*args, **kwargs)
+            for path in restart_map:
+                if path_hash(path) != checksums[path]:
+                    if path == PG_IFCS_CONF:
+                        ensure_files()
+                    restart_pg()
+                    break
+        return wrapped_f
+    return wrap

=== removed symlink 'hooks/plumgrid-relation-joined'
=== target was u'pg_gw_hooks.py'
=== added symlink 'hooks/update-status'
=== target is u'pg_gw_hooks.py'
=== added symlink 'hooks/upgrade-charm'
=== target is u'pg_gw_hooks.py'
=== added file 'templates/kilo/00-pg.conf'
--- templates/kilo/00-pg.conf	1970-01-01 00:00:00 +0000
+++ templates/kilo/00-pg.conf	2016-04-08 08:36:52 +0000
@@ -0,0 +1,2 @@
+$template ls_json,"{{'{'}}{{'%'}}timestamp:::date-rfc3339,jsonf:@timestamp%,%source:::jsonf:@source_host%,%msg:::json%}"
+:syslogtag,isequal,"pg:" @{{ opsvm_ip }}:6000;ls_json

=== modified file 'templates/kilo/hosts'
--- templates/kilo/hosts	2015-07-29 18:23:55 +0000
+++ templates/kilo/hosts	2016-04-08 08:36:52 +0000
@@ -1,5 +1,5 @@
 127.0.0.1   localhost
-127.0.1.1   {{ pg_hostname }}
+127.0.1.1   {{ pg_fqdn }}   {{ pg_hostname }}
 
 # The following lines are desirable for IPv6 capable hosts
 ::1     ip6-localhost ip6-loopback

=== modified file 'templates/kilo/ifcs.conf'
--- templates/kilo/ifcs.conf	2015-07-29 18:23:55 +0000
+++ templates/kilo/ifcs.conf	2016-04-08 08:36:52 +0000
@@ -1,6 +1,7 @@
-{{ interface }} = fabric_core host
-{% if ext_interface -%}
-{{ ext_interface }} = access_phys
-
+{{ fabric_interface }} = fabric_core host
+{% if ext_interfaces -%}
+{% for ip in ext_interfaces -%}
+{{ ip }} = access_phys
+{% endfor -%}
 {% endif -%}
 

=== modified file 'templates/kilo/plumgrid.conf'
--- templates/kilo/plumgrid.conf	2015-07-29 18:23:55 +0000
+++ templates/kilo/plumgrid.conf	2016-04-08 08:36:52 +0000
@@ -1,4 +1,4 @@
-plumgrid_ip={{ local_ip }}
+plumgrid_ip={{ director_ips_string }}
 plumgrid_port=8001
 mgmt_dev={{ interface }}
 label={{ label}}

=== added file 'tests/files/plumgrid-gateway-dense.yaml'
--- tests/files/plumgrid-gateway-dense.yaml	1970-01-01 00:00:00 +0000
+++ tests/files/plumgrid-gateway-dense.yaml	2016-04-08 08:36:52 +0000
@@ -0,0 +1,133 @@
+test:
+  series: 'trusty'
+  relations:
+  - - mysql
+    - keystone
+  - - nova-cloud-controller
+    - mysql
+  - - nova-cloud-controller
+    - rabbitmq-server
+  - - nova-cloud-controller
+    - glance
+  - - nova-cloud-controller
+    - keystone
+  - - nova-compute
+    - nova-cloud-controller
+  - - nova-compute
+    - mysql
+  - - nova-compute
+    - rabbitmq-server
+  - - nova-compute
+    - glance
+  - - glance
+    - mysql
+  - - glance
+    - keystone
+  - - glance
+    - cinder
+  - - mysql
+    - cinder
+  - - cinder
+    - rabbitmq-server
+  - - cinder
+    - nova-cloud-controller
+  - - cinder
+    - keystone
+  - - openstack-dashboard
+    - keystone
+  - - neutron-api
+    - mysql
+  - - neutron-api
+    - keystone
+  - - neutron-api
+    - rabbitmq-server
+  - - neutron-api
+    - nova-cloud-controller
+  - - neutron-api
+    - neutron-api-plumgrid
+  - - neutron-api-plumgrid
+    - plumgrid-edge
+  - - plumgrid-director
+    - plumgrid-edge
+  - - nova-compute
+    - plumgrid-edge
+  - - plumgrid-director
+    - plumgrid-gateway
+  services:
+    mysql:
+      charm: cs:trusty/mysql
+      num_units: 1
+      to: 'lxc:plumgrid-director=0'
+    rabbitmq-server:
+      charm: cs:trusty/rabbitmq-server
+      num_units: 1
+      to: 'lxc:plumgrid-director=0'
+    keystone:
+      charm: cs:trusty/keystone
+      num_units: 1
+      options:
+        admin-password: plumgrid
+        openstack-origin: cloud:trusty-kilo
+      to: 'lxc:plumgrid-director=0'
+    nova-cloud-controller:
+      charm: cs:trusty/nova-cloud-controller
+      num_units: 1
+      options:
+        console-access-protocol: novnc
+        network-manager: Neutron
+        openstack-origin: cloud:trusty-kilo
+        quantum-security-groups: 'yes'
+      to: 'lxc:plumgrid-director=0'
+    glance:
+      charm: cs:trusty/glance
+      num_units: 1
+      options:
+        openstack-origin: cloud:trusty-kilo
+      to: 'lxc:plumgrid-director=0'
+    openstack-dashboard:
+      charm: cs:trusty/openstack-dashboard
+      num_units: 1
+      options:
+        openstack-origin: cloud:trusty-kilo
+      to: 'lxc:plumgrid-director=0'
+    cinder:
+      charm: cs:trusty/cinder
+      num_units: 1
+      options:
+        openstack-origin: cloud:trusty-kilo
+      to: 'lxc:plumgrid-director=0'
+    neutron-api:
+      charm: cs:~plumgrid-team/trusty/neutron-api
+      num_units: 1
+      options:
+        neutron-plugin: plumgrid
+        neutron-security-groups: false
+        openstack-origin: cloud:trusty-kilo
+        plumgrid-password: plumgrid
+        plumgrid-username: plumgrid
+        plumgrid-virtual-ip: 192.168.100.250
+      to: 'lxc:plumgrid-director=0'
+    neutron-api-plumgrid:
+      charm: cs:~plumgrid-team/trusty/neutron-api-plumgrid
+      options:
+        enable-metadata: True
+    plumgrid-director:
+      charm: cs:~plumgrid-team/trusty/plumgrid-director
+      num_units: 1
+      constraints: "mem=8G root-disk=30G cpu-cores=8"
+      options:
+        plumgrid-virtual-ip: 192.168.100.250
+    nova-compute:
+      charm: cs:~plumgrid-team/trusty/nova-compute
+      num_units: 1
+      options:
+        enable-live-migration: true
+        enable-resize: true
+        migration-auth-type: ssh
+        openstack-origin: cloud:trusty-kilo
+      to: '0'
+    plumgrid-edge:
+      charm: cs:~plumgrid-team/trusty/plumgrid-edge
+    plumgrid-gateway:
+      charm: cs:~plumgrid-team/trusty/plumgrid-gateway
+      num_units: 1

=== renamed file 'tests/files/plumgrid-gateway-dense.yaml' => 'tests/files/plumgrid-gateway-dense.yaml.moved'
=== added file 'tests/tests.yaml'
--- tests/tests.yaml	1970-01-01 00:00:00 +0000
+++ tests/tests.yaml	2016-04-08 08:36:52 +0000
@@ -0,0 +1,3 @@
+makefile:
+  - lint
+

=== renamed file 'tests/tests.yaml' => 'tests/tests.yaml.moved'
=== modified file 'unit_tests/test_pg_gw_context.py'
--- unit_tests/test_pg_gw_context.py	2015-08-24 16:24:20 +0000
+++ unit_tests/test_pg_gw_context.py	2016-04-08 08:36:52 +0000
@@ -5,8 +5,8 @@
 import charmhelpers
 
 TO_PATCH = [
-    'config',
-    'get_unit_hostname',
+    'gethostname',
+    'getfqdn'
 ]
 
 
@@ -22,8 +22,6 @@
 
     def setUp(self):
         super(PGGwContextTest, self).setUp(context, TO_PATCH)
-        self.config.side_effect = self.test_config.get
-        self.test_config.set('external-interface', 'eth1')
 
     def tearDown(self):
         super(PGGwContextTest, self).tearDown()
@@ -37,11 +35,14 @@
     @patch.object(charmhelpers.contrib.openstack.context,
                   'config_flags_parser')
     @patch.object(context.PGGwContext, '_save_flag_file')
-    @patch.object(context, '_pg_dir_settings')
+    @patch.object(context, '_pg_dir_context')
     @patch.object(charmhelpers.contrib.openstack.context,
                   'neutron_plugin_attribute')
-    @patch.object(utils, 'check_interface_type')
-    def test_neutroncc_context_api_rel(self, _int_type, _npa, _pg_dir_settings,
+    @patch.object(utils, 'get_mgmt_interface')
+    @patch.object(utils, 'get_fabric_interface')
+    @patch.object(utils, 'get_gw_interfaces')
+    def test_neutroncc_context_api_rel(self, _gw_int, _fabric_int,
+                                       _mgmt_int, _npa, _pg_dir_context,
                                        _save_flag_file, _config_flag,
                                        _unit_get, _unit_priv_ip, _config,
                                        _is_clus, _https, _ens_pkgs):
@@ -50,38 +51,38 @@
                 return "neutron.randomdriver"
             if section == "config":
                 return "neutron.randomconfig"
-        config = {'external-interface': "eth1"}
-
-        def mock_config(key=None):
-            if key:
-                return config.get(key)
-
-            return config
 
         self.maxDiff = None
-        self.config.side_effect = mock_config
         _npa.side_effect = mock_npa
-        _unit_get.return_value = '192.168.100.201'
-        _unit_priv_ip.return_value = '192.168.100.201'
-        self.get_unit_hostname.return_value = 'node0'
+        _unit_get.return_value = '192.168.100.203'
+        _unit_priv_ip.return_value = '192.168.100.203'
+        self.gethostname.return_value = 'node0'
+        self.getfqdn.return_value = 'node0'
         _is_clus.return_value = False
         _config_flag.return_value = False
-        _pg_dir_settings.return_value = {'pg_dir_ip': '192.168.100.201'}
-        _int_type.return_value = 'juju-br0'
+        _pg_dir_context.return_value = {'director_ips': ['192.168.100.201'],
+                                        'opsvm_ip': '127.0.0.1'}
+        _mgmt_int.return_value = 'juju-br0'
+        _fabric_int.return_value = 'juju-br0'
+        _gw_int.return_value = ['eth1']
         napi_ctxt = context.PGGwContext()
         expect = {
-            'ext_interface': "eth1",
+            'ext_interfaces': ['eth1'],
             'config': 'neutron.randomconfig',
             'core_plugin': 'neutron.randomdriver',
-            'local_ip': 'pg_dir_ip',
+            'local_ip': '192.168.100.203',
+            'director_ips_string': '192.168.100.201',
             'network_manager': 'neutron',
             'neutron_plugin': 'plumgrid',
             'neutron_security_groups': None,
-            'neutron_url': 'https://192.168.100.201:9696',
+            'neutron_url': 'https://192.168.100.203:9696',
             'pg_hostname': 'node0',
+            'pg_fqdn': 'node0',
             'interface': 'juju-br0',
+            'fabric_interface': 'juju-br0',
             'label': 'node0',
             'fabric_mode': 'host',
             'neutron_alchemy_flags': False,
+            'opsvm_ip': '127.0.0.1',
         }
         self.assertEquals(expect, napi_ctxt())

=== modified file 'unit_tests/test_pg_gw_hooks.py'
--- unit_tests/test_pg_gw_hooks.py	2015-08-24 16:24:20 +0000
+++ unit_tests/test_pg_gw_hooks.py	2016-04-08 08:36:52 +0000
@@ -1,5 +1,6 @@
 from mock import MagicMock, patch, call
 from test_utils import CharmTestCase
+
 with patch('charmhelpers.core.hookenv.config') as config:
     config.return_value = 'neutron'
     import pg_gw_utils as utils
@@ -18,17 +19,17 @@
 TO_PATCH = [
     'remove_iovisor',
     'apt_install',
-    'apt_purge',
     'CONFIGS',
     'log',
     'configure_sources',
     'ensure_files',
     'stop_pg',
-    'restart_pg',
     'load_iovisor',
     'ensure_mtu',
     'add_lcm_key',
     'determine_packages',
+    'load_iptables',
+    'director_cluster_ready'
 ]
 NEUTRON_CONF_DIR = "/etc/neutron"
 
@@ -60,34 +61,12 @@
         self.ensure_files.assert_called_with()
         self.add_lcm_key.assert_called_with()
 
-    def test_plumgrid_joined(self):
-        self._call_hook('plumgrid-relation-joined')
-        self.ensure_mtu.assert_called_with()
-        self.ensure_files.assert_called_with()
-        self.add_lcm_key.assert_called_with()
-        self.CONFIGS.write_all.assert_called_with()
-        self.restart_pg.assert_called_with()
-
-    def test_config_changed_hook(self):
-        _pkgs = ['plumgrid-lxc', 'iovisor-dkms']
-        self.add_lcm_key.return_value = 0
-        self.determine_packages.return_value = [_pkgs]
-        self._call_hook('config-changed')
-        self.stop_pg.assert_called_with()
-        self.configure_sources.assert_called_with(update=True)
-        self.apt_install.assert_has_calls([
-            call(_pkgs, fatal=True,
-                 options=['--force-yes']),
-        ])
-        self.load_iovisor.assert_called_with()
-        self.ensure_mtu.assert_called_with()
-        self.ensure_files.assert_called_with()
-        self.CONFIGS.write_all.assert_called_with()
-        self.restart_pg.assert_called_with()
+    def test_plumgrid_changed(self):
+        self._call_hook('plumgrid-relation-changed')
+        self.director_cluster_ready.return_value = True
+        self.ensure_mtu.assert_called_with()
+        self.CONFIGS.write_all.assert_called_with()
 
     def test_stop(self):
-        _pkgs = ['plumgrid-lxc', 'iovisor-dkms']
         self._call_hook('stop')
         self.stop_pg.assert_called_with()
-        self.remove_iovisor.assert_called_with()
-        self.determine_packages.return_value = _pkgs

=== modified file 'unit_tests/test_pg_gw_utils.py'
--- unit_tests/test_pg_gw_utils.py	2015-08-24 16:24:20 +0000
+++ unit_tests/test_pg_gw_utils.py	2016-04-08 08:36:52 +0000
@@ -53,7 +53,8 @@
         confs = [nutils.PG_CONF,
                  nutils.PG_HN_CONF,
                  nutils.PG_HS_CONF,
-                 nutils.PG_IFCS_CONF]
+                 nutils.PG_IFCS_CONF,
+                 nutils.OPS_CONF]
         self.assertItemsEqual(_regconfs.configs, confs)
 
     def test_resource_map(self):
@@ -69,6 +70,7 @@
             (nutils.PG_CONF, ['plumgrid']),
             (nutils.PG_HN_CONF, ['plumgrid']),
             (nutils.PG_HS_CONF, ['plumgrid']),
+            (nutils.OPS_CONF, ['plumgrid']),
             (nutils.PG_IFCS_CONF, []),
         ])
         self.assertEqual(expect, _restart_map)