Merge lp:~sateesh-chodapuneedi/nova/esx-multi-nic into lp:~hudson-openstack/nova/trunk

[Sateesh]

Multi-NIC support for vmwareapi virt driver in nova.
Does injection of Multi-NIC information to instances with Operating system flavors Ubuntu, Windows and RHEL.
vmwareapi virt driver now relies on calls to network manager instead of nova db calls for network configuration information of instance.
Re-oranized VMWareVlanBridgeDriver and added session parmeter to methods to use existing session. Also removed session creation code as session comes as argument.
Added check for flat_inject flag before attempting an inject operation.

This branch resolves the following bugs:
  Bug #831497 in OpenStack Compute (nova): "Instance spawn operation fails on ESXi compute node"
  https://bugs.launchpad.net/nova/+bug/831497
  Bug #839383 in OpenStack Compute (nova): "ESX(i) VIFs and mac addresses"
  https://bugs.launchpad.net/nova/+bug/839383

[Vish Ishaya (vishvananda)]

Awesome work on this! It will be great to have esx working properly again. Minor nit:

63 - bridge_interface = network['bridge_interface']
64 + vlan_interface = FLAGS.vmwareapi_vlan_interface

can we just use the global vlan interface here?

review: Needs Information

[Sateesh (sateesh-chodapuneedi)]

@Vish thanks for the review.

> 63 - bridge_interface = network['bridge_interface']
> 64 + vlan_interface = FLAGS.vmwareapi_vlan_interface
>
> can we just use the global vlan interface here?

I choose to use separate flag because vlan_interface is not being used in the current code base (revision 1526), also post multi-nic nova code started using bridge_interface in networks table rather than depending on this flag. From the information I have, it seems to me that this flag is going to become stale. So I thought of having a separate flag specific vmwareapi virt driver which deals with it's specific case. Even if vlan_interface flag gets removed no functionality breakage occur in vmwareapi virt driver. So trying to keep this flag modular (or module specific).
Also ESX(i) names it NIC with prefix 'vmnic', different from other systems that use prefix 'eth'. So I thought if we classify the flag by it's value, it becomes specific to vmware subset.

[Vish Ishaya (vishvananda)]

We were actually planning on adding the flag back as an override. It is possible that the interface specified in the table is incorrect for some hosts. For example if you have two hosts, one using eth0 and the other using eth1. But this is a minor issue that can be fixed later. I'm happy leaving this as is for now.

review: Approve

[Dan Prince (dan-prince)]

Looks good to me.

review: Approve

[Sateesh (sateesh-chodapuneedi)]

Thanks Vish and Dan for your reviews.

[Sateesh (sateesh-chodapuneedi)]

Thanks Vish and Dan for your reviews.

~Sateesh

=== modified file 'nova/tests/vmwareapi/stubs.py'
--- nova/tests/vmwareapi/stubs.py	2011-07-27 00:40:50 +0000
+++ nova/tests/vmwareapi/stubs.py	2011-09-05 07:38:25 +0000
@@ -47,7 +47,5 @@
     stubs.Set(vmware_images, 'upload_image', fake.fake_upload_image)
     stubs.Set(vmwareapi_conn.VMWareAPISession, "_get_vim_object",
               fake_get_vim_object)
-    stubs.Set(vmwareapi_conn.VMWareAPISession, "_get_vim_object",
-              fake_get_vim_object)
     stubs.Set(vmwareapi_conn.VMWareAPISession, "_is_vim_object",
               fake_is_vim_object)
 
=== modified file 'nova/virt/vmwareapi/fake.py'
--- nova/virt/vmwareapi/fake.py	2011-07-27 00:40:50 +0000
+++ nova/virt/vmwareapi/fake.py	2011-09-05 07:38:25 +0000
@@ -409,7 +409,7 @@
 
 def fake_get_network(*args, **kwargs):
     """Fake get network."""
-    return [{'type': 'fake'}]
+    return {'type': 'fake'}
 
 
 def fake_fetch_image(image, instance, **kwargs):
 
=== modified file 'nova/virt/vmwareapi/vif.py'
--- nova/virt/vmwareapi/vif.py	2011-08-03 18:49:14 +0000
+++ nova/virt/vmwareapi/vif.py	2011-09-05 07:38:25 +0000
@@ -17,42 +17,35 @@
 
 """VIF drivers for VMWare."""
 
-from nova import db
 from nova import exception
 from nova import flags
 from nova import log as logging
-from nova import utils
 from nova.virt.vif import VIFDriver
-from nova.virt.vmwareapi_conn import VMWareAPISession
 from nova.virt.vmwareapi import network_utils
 
 
 LOG = logging.getLogger("nova.virt.vmwareapi.vif")
 
 FLAGS = flags.FLAGS
+FLAGS['vmwareapi_vlan_interface'].SetDefault('vmnic0')
 
 
 class VMWareVlanBridgeDriver(VIFDriver):
     """VIF Driver to setup bridge/VLAN networking using VMWare API."""
 
     def plug(self, instance, network, mapping):
+        """Plug the VIF to specified instance using information passed.
+        Currently we are plugging the VIF(s) during instance creation itself.
+        We can use this method when we add support to add additional NIC to
+        an existing instance."""
+        pass
+
+    def ensure_vlan_bridge(self, session, network):
         """Create a vlan and bridge unless they already exist."""
         vlan_num = network['vlan']
         bridge = network['bridge']
-        bridge_interface = network['bridge_interface']
+        vlan_interface = FLAGS.vmwareapi_vlan_interface
 
-        # Open vmwareapi session
-        host_ip = FLAGS.vmwareapi_host_ip
-        host_username = FLAGS.vmwareapi_host_username
-        host_password = FLAGS.vmwareapi_host_password
-        if not host_ip or host_username is None or host_password is None:
-            raise Exception(_('Must specify vmwareapi_host_ip, '
-                              'vmwareapi_host_username '
-                              'and vmwareapi_host_password to use '
-                              'connection_type=vmwareapi'))
-        session = VMWareAPISession(host_ip, host_username, host_password,
-                                   FLAGS.vmwareapi_api_retry_count)
-        vlan_interface = bridge_interface
         # Check if the vlan_interface physical network adapter exists on the
         # host.
         if not network_utils.check_if_vlan_interface_exists(session,
@@ -92,4 +85,6 @@
                                                pgroup=pg_vlanid)
 
     def unplug(self, instance, network, mapping):
+        """Cleanup operations like deleting port group if no instance
+        is associated with it."""
         pass
 
=== modified file 'nova/virt/vmwareapi/vm_util.py'
--- nova/virt/vmwareapi/vm_util.py	2011-07-27 00:40:50 +0000
+++ nova/virt/vmwareapi/vm_util.py	2011-09-05 07:38:25 +0000
@@ -39,8 +39,7 @@
 
 
 def get_vm_create_spec(client_factory, instance, data_store_name,
-                       network_name="vmnet0",
-                       os_type="otherGuest", network_ref=None):
+                       vif_infos, os_type="otherGuest"):
     """Builds the VM Create spec."""
     config_spec = client_factory.create('ns0:VirtualMachineConfigSpec')
     config_spec.name = instance.name
@@ -61,14 +60,12 @@
     config_spec.numCPUs = int(instance.vcpus)
     config_spec.memoryMB = int(instance.memory_mb)
 
-    mac_address = None
-    if instance['mac_addresses']:
-        mac_address = instance['mac_addresses'][0]['address']
-
-    nic_spec = create_network_spec(client_factory,
-                                    network_name, mac_address)
-
-    device_config_spec = [nic_spec]
+    vif_spec_list = []
+    for vif_info in vif_infos:
+        vif_spec = create_network_spec(client_factory, vif_info)
+        vif_spec_list.append(vif_spec)
+
+    device_config_spec = vif_spec_list
 
     config_spec.deviceChange = device_config_spec
     return config_spec
@@ -93,8 +90,7 @@
     return virtual_device_config
 
 
-def create_network_spec(client_factory, network_name, mac_address,
-                        network_ref=None):
+def create_network_spec(client_factory, vif_info):
     """
     Builds a config spec for the addition of a new network
     adapter to the VM.
@@ -109,6 +105,9 @@
     # NOTE(asomya): Only works on ESXi if the portgroup binding is set to
     # ephemeral. Invalid configuration if set to static and the NIC does
     # not come up on boot if set to dynamic.
+    network_ref = vif_info['network_ref']
+    network_name = vif_info['network_name']
+    mac_address = vif_info['mac_address']
     backing = None
     if (network_ref and
         network_ref['type'] == "DistributedVirtualPortgroup"):
@@ -295,11 +294,8 @@
     return config_spec
 
 
-def get_machine_id_change_spec(client_factory, mac, ip_addr, netmask,
-                               gateway, broadcast, dns):
+def get_machine_id_change_spec(client_factory, machine_id_str):
     """Builds the machine id change config spec."""
-    machine_id_str = "%s;%s;%s;%s;%s;%s" % (mac, ip_addr, netmask,
-                                            gateway, broadcast, dns)
     virtual_machine_config_spec = \
         client_factory.create('ns0:VirtualMachineConfigSpec')
 
 
=== modified file 'nova/virt/vmwareapi/vmops.py'
--- nova/virt/vmwareapi/vmops.py	2011-07-29 21:58:27 +0000
+++ nova/virt/vmwareapi/vmops.py	2011-09-05 07:38:25 +0000
@@ -27,7 +27,6 @@
 import uuid
 
 from nova import context as nova_context
-from nova import db
 from nova import exception
 from nova import flags
 from nova import log as logging
@@ -111,22 +110,6 @@
         client_factory = self._session._get_vim().client.factory
         service_content = self._session._get_vim().get_service_content()
 
-        network = db.network_get_by_instance(nova_context.get_admin_context(),
-                                            instance['id'])
-
-        net_name = network['bridge']
-
-        def _check_if_network_bridge_exists():
-            network_ref = \
-                network_utils.get_network_with_the_name(self._session,
-                                                        net_name)
-            if network_ref is None:
-                raise exception.NetworkNotFoundForBridge(bridge=net_name)
-            return network_ref
-
-        self.plug_vifs(instance, network_info)
-        network_obj = _check_if_network_bridge_exists()
-
         def _get_datastore_ref():
             """Get the datastore list and choose the first local storage."""
             data_stores = self._session._call_method(vim_util, "get_objects",
@@ -182,11 +165,36 @@
 
         vm_folder_mor, res_pool_mor = _get_vmfolder_and_res_pool_mors()
 
+        def _check_if_network_bridge_exists(network_name):
+            network_ref = \
+                network_utils.get_network_with_the_name(self._session,
+                                                        network_name)
+            if network_ref is None:
+                raise exception.NetworkNotFoundForBridge(bridge=network_name)
+            return network_ref
+
+        def _get_vif_infos():
+            vif_infos = []
+            for (network, mapping) in network_info:
+                mac_address = mapping['mac']
+                network_name = network['bridge']
+                if mapping.get('should_create_vlan'):
+                    network_ref = self._vif_driver.ensure_vlan_bridge(
+                                                        self._session, network)
+                else:
+                    network_ref = _check_if_network_bridge_exists(network_name)
+                vif_infos.append({'network_name': network_name,
+                                  'mac_address': mac_address,
+                                  'network_ref': network_ref,
+                                 })
+            return vif_infos
+
+        vif_infos = _get_vif_infos()
+
         # Get the create vm config spec
         config_spec = vm_util.get_vm_create_spec(
                             client_factory, instance,
-                            data_store_name, net_name, os_type,
-                            network_obj)
+                            data_store_name, vif_infos, os_type)
 
         def _execute_create_vm():
             """Create VM on ESX host."""
@@ -204,8 +212,10 @@
 
         _execute_create_vm()
 
-        # Set the machine id for the VM for setting the IP
-        self._set_machine_id(client_factory, instance)
+        # Set the machine.id parameter of the instance to inject
+        # the NIC configuration inside the VM
+        if FLAGS.flat_injected:
+            self._set_machine_id(client_factory, instance, network_info)
 
         # Naming the VM files in correspondence with the VM instance name
         # The flat vmdk file name
@@ -716,39 +726,45 @@
         """Return link to instance's ajax console."""
         return 'http://fakeajaxconsole/fake_url'
 
-    def _set_machine_id(self, client_factory, instance):
-        """
-        Set the machine id of the VM for guest tools to pick up and change
-        the IP.
-        """
-        admin_context = nova_context.get_admin_context()
+    def _set_machine_id(self, client_factory, instance, network_info):
+        """
+        Set the machine id of the VM for guest tools to pick up and reconfigure
+        the network interfaces.
+        """
         vm_ref = self._get_vm_ref_from_the_name(instance.name)
         if vm_ref is None:
             raise exception.InstanceNotFound(instance_id=instance.id)
-        network = db.network_get_by_instance(nova_context.get_admin_context(),
-                                            instance['id'])
-        mac_address = None
-        if instance['mac_addresses']:
-            mac_address = instance['mac_addresses'][0]['address']
-
-        net_mask = network["netmask"]
-        gateway = network["gateway"]
-        broadcast = network["broadcast"]
-        # TODO(vish): add support for dns2
-        dns = network["dns1"]
-
-        addresses = db.instance_get_fixed_addresses(admin_context,
-                                                    instance['id'])
-        ip_addr = addresses[0] if addresses else None
+
+        machine_id_str = ''
+        for (network, info) in network_info:
+            # TODO(vish): add support for dns2
+            # TODO(sateesh): add support for injection of ipv6 configuration
+            ip_v4 = ip_v6 = None
+            if 'ips' in info and len(info['ips']) > 0:
+                ip_v4 = info['ips'][0]
+            if 'ip6s' in info and len(info['ip6s']) > 0:
+                ip_v6 = info['ip6s'][0]
+            if len(info['dns']) > 0:
+                dns = info['dns'][0]
+            else:
+                dns = ''
+
+            interface_str = "%s;%s;%s;%s;%s;%s" % \
+                                            (info['mac'],
+                                             ip_v4 and ip_v4['ip'] or '',
+                                             ip_v4 and ip_v4['netmask'] or '',
+                                             info['gateway'],
+                                             info['broadcast'],
+                                             dns)
+            machine_id_str = machine_id_str + interface_str + '#'
 
         machine_id_change_spec = \
-            vm_util.get_machine_id_change_spec(client_factory, mac_address,
-                                               ip_addr, net_mask, gateway,
-                                               broadcast, dns)
+            vm_util.get_machine_id_change_spec(client_factory, machine_id_str)
+
         LOG.debug(_("Reconfiguring VM instance %(name)s to set the machine id "
                   "with ip - %(ip_addr)s") %
                   ({'name': instance.name,
-                   'ip_addr': ip_addr}))
+                   'ip_addr': ip_v4['ip']}))
         reconfig_task = self._session._call_method(self._session._get_vim(),
                            "ReconfigVM_Task", vm_ref,
                            spec=machine_id_change_spec)
@@ -756,7 +772,7 @@
         LOG.debug(_("Reconfigured VM instance %(name)s to set the machine id "
                   "with ip - %(ip_addr)s") %
                   ({'name': instance.name,
-                   'ip_addr': ip_addr}))
+                   'ip_addr': ip_v4['ip']}))
 
     def _get_datacenter_name_and_ref(self):
         """Get the datacenter name and the reference."""
 
=== modified file 'tools/esx/guest_tool.py'
--- tools/esx/guest_tool.py	2011-06-27 18:41:07 +0000
+++ tools/esx/guest_tool.py	2011-09-05 07:38:25 +0000
@@ -81,28 +81,34 @@
 
 def _parse_network_details(machine_id):
     """
-    Parse the machine.id field to get MAC, IP, Netmask and Gateway fields
-    machine.id is of the form MAC;IP;Netmask;Gateway;Broadcast;DNS1,DNS2
-    where ';' is the separator.
+    Parse the machine_id to get MAC, IP, Netmask and Gateway fields per NIC.
+    machine_id is of the form ('NIC_record#NIC_record#', '')
+    Each of the NIC will have record NIC_record in the form
+    'MAC;IP;Netmask;Gateway;Broadcast;DNS' where ';' is field separator.
+    Each record is separated by '#' from next record.
     """
+    logging.debug(_("Received machine_id from vmtools : %s") % machine_id[0])
     network_details = []
     if machine_id[1].strip() == "1":
         pass
     else:
-        network_info_list = machine_id[0].split(';')
-        assert len(network_info_list) % 6 == 0
-        no_grps = len(network_info_list) / 6
-        i = 0
-        while i < no_grps:
-            k = i * 6
-            network_details.append((
-                            network_info_list[k].strip().lower(),
-                            network_info_list[k + 1].strip(),
-                            network_info_list[k + 2].strip(),
-                            network_info_list[k + 3].strip(),
-                            network_info_list[k + 4].strip(),
-                            network_info_list[k + 5].strip().split(',')))
-            i += 1
+        for machine_id_str in machine_id[0].split('#'):
+            network_info_list = machine_id_str.split(';')
+            if len(network_info_list) % 6 != 0:
+                break
+            no_grps = len(network_info_list) / 6
+            i = 0
+            while i < no_grps:
+                k = i * 6
+                network_details.append((
+                                network_info_list[k].strip().lower(),
+                                network_info_list[k + 1].strip(),
+                                network_info_list[k + 2].strip(),
+                                network_info_list[k + 3].strip(),
+                                network_info_list[k + 4].strip(),
+                                network_info_list[k + 5].strip().split(',')))
+                i += 1
+    logging.debug(_("NIC information from vmtools : %s") % network_details)
     return network_details
 
 
@@ -279,6 +285,7 @@
 
 
 def _set_rhel_networking(network_details=None):
+    """Set IPv4 network settings for RHEL distros."""
     network_details = network_details or []
     all_dns_servers = []
     for network_detail in network_details:
@@ -320,31 +327,33 @@
 
 
 def _set_ubuntu_networking(network_details=None):
+    """Set IPv4 network settings for Ubuntu."""
     network_details = network_details or []
-    """ Set IPv4 network settings for Ubuntu """
     all_dns_servers = []
-    for network_detail in network_details:
+    interface_file_name = '/etc/network/interfaces'
+    # Remove file
+    os.remove(interface_file_name)
+    # Touch file
+    _execute(['touch', interface_file_name])
+    interface_file = open(interface_file_name, 'w')
+    for device, network_detail in enumerate(network_details):
         mac_address, ip_address, subnet_mask, gateway, broadcast,\
             dns_servers = network_detail
         all_dns_servers.extend(dns_servers)
         adapter_name, current_ip_address = \
                 _get_linux_adapter_name_and_ip_address(mac_address)
 
-        if adapter_name and not ip_address == current_ip_address:
-            interface_file_name = \
-                '/etc/network/interfaces'
-            # Remove file
-            os.remove(interface_file_name)
-            # Touch file
-            _execute(['touch', interface_file_name])
-            interface_file = open(interface_file_name, 'w')
+        if adapter_name:
             interface_file.write('\nauto %s' % adapter_name)
             interface_file.write('\niface %s inet static' % adapter_name)
             interface_file.write('\nbroadcast %s' % broadcast)
             interface_file.write('\ngateway %s' % gateway)
             interface_file.write('\nnetmask %s' % subnet_mask)
-            interface_file.write('\naddress %s' % ip_address)
-            interface_file.close()
+            interface_file.write('\naddress %s\n' % ip_address)
+        logging.debug(_("Successfully configured NIC %d with "
+                        "NIC info %s") % (device, network_detail))
+    interface_file.close()
+
     if all_dns_servers:
         dns_file_name = "/etc/resolv.conf"
         os.remove(dns_file_name)
@@ -355,7 +364,8 @@
         for dns_server in unique_entries:
             dns_file.write("\nnameserver %s" % dns_server)
         dns_file.close()
-    print "\nRestarting networking....\n"
+
+    logging.debug(_("Restarting networking....\n"))
     _execute(['/etc/init.d/networking', 'restart'])