Merge lp:~jfontan/cloud-init/opennebula-4.8 into lp:~cloud-init-dev/cloud-init/trunk

[Javi Fontan]

Changes to bring new functionality to the OpenNebula DataSource (ONE 4.8) and fix problems with the new interface naming scheme. It also addresses some problems with RHEL/CentOS 7 that made the old version incompatible.

There is a change that I'm not sure you are happy with it. I've added a ifdown command before ifup as in Ubuntu the interface did not get configured as the interface was already up. If there is a better way to fix this I will be happy to change the code.

[Scott Moser (smoser)]

I'm sorry for horribly delinquent behavior.

Javi, thanks for your submission.
If you're still around
a.) is this still relevant / needed?
b.) Can you sign the contributors agreement? http://www.ubuntu.com/legal/contributors please feel free to ping me in irc or email if you have questions.

[Javi Fontan (jfontan)]

Sorry for the late reply.

The changes are still needed. I'm not sure if they can be merged as is
as I have not been following cloud-init development recently. As
stated I'm not sure you will like the ifdown/ifup of interfaces.

I've already signed the agreement for a previous contribution to cloud-init.

I'm jfontan in freenode if you need anything.

Thanks

On Wed, Apr 6, 2016 at 9:38 PM, Scott Moser <email address hidden> wrote:
>
> I'm sorry for horribly delinquent behavior.
>
> Javi, thanks for your submission.
> If you're still around
> a.) is this still relevant / needed?
> b.) Can you sign the contributors agreement? http://www.ubuntu.com/legal/contributors please feel free to ping me in irc or email if you have questions.
>
> --
> https://code.launchpad.net/~jfontan/cloud-init/opennebula-4.8/+merge/234207
> You are the owner of lp:~jfontan/cloud-init/opennebula-4.8.

--
Javier Fontán Muiños
Developer
OpenNebula - Flexible Enterprise Cloud Made Simple
www.OpenNebula.org | @OpenNebula | github.com/jfontan

[Scott Moser (smoser)]

Hello,
Thank you for taking the time to contribute to cloud-init. Cloud-init has moved its revision control system to git. As a result, we are marking all bzr merge proposals as 'rejected'. If you would like to re-submit this proposal for review, please do so by following the current HACKING documentation at http://cloudinit.readthedocs.io/en/latest/topics/hacking.html .

Since network configuration has changed dramatically, I"m not sure if this change is still relevant.

Unmerged revisions

1012. By Javi Fontan on 2014-09-10

Update OpenNebula DataSource to the latest version

Now network interfaces are matched using MAC addresses instead of name,
this makes it work with the new interface naming scheme.

Also added support for IPv6, select interface for gateway, search domain
and fixes for RHEL/CentOS 7

1011. By Javi Fontan on 2014-09-10

Add distro functions for OpenNebula DataSource

OpenNebula datasource need to disable NetworkManager and use runuser instead
of sudo in RHEL/CentOS

1010. By Javi Fontan on 2014-09-10

Bring down interface before configuring it

If the interface is already up the ifup command will have no effect in
some distributions.

=== modified file 'cloudinit/distros/__init__.py'
--- cloudinit/distros/__init__.py	2014-09-10 18:32:37 +0000
+++ cloudinit/distros/__init__.py	2014-09-10 21:01:03 +0000
@@ -271,13 +271,19 @@
             util.write_file(self.hosts_fn, contents.getvalue(), mode=0644)
 
     def _bring_up_interface(self, device_name):
-        cmd = ['ifup', device_name]
-        LOG.debug("Attempting to run bring up interface %s using command %s",
-                   device_name, cmd)
+        down_cmd = ['ifdown', device_name]
+        up_cmd = ['ifup', device_name]
+        LOG.debug("Attempting to run bring up interface %s using commands %s, %s",
+                   device_name, down_cmd, up_cmd)
         try:
-            (_out, err) = util.subp(cmd)
-            if len(err):
-                LOG.warn("Running %s resulted in stderr output: %s", cmd, err)
+            (_out, err) = util.subp(down_cmd)
+            if len(err):
+                LOG.warn("Running %s resulted in stderr output: %s", down_cmd, err)
+
+            (_out, err) = util.subp(up_cmd)
+            if len(err):
+                LOG.warn("Running %s resulted in stderr output: %s", up_cmd, err)
+
             return True
         except util.ProcessExecutionError:
             util.logexc(LOG, "Running interface command %s failed", cmd)
@@ -525,6 +531,13 @@
                 util.subp(['usermod', '-a', '-G', name, member])
                 LOG.info("Added user '%s' to group '%s'" % (member, name))
 
+    # Command used to execute command with an specific user            
+    def switch_user_cmd(self, user):
+        return ['sudo', '-u', user]
+
+    def disable_network_manager(self):
+        pass
+
 
 def _get_package_mirror_info(mirror_info, availability_zone=None,
                              mirror_filter=util.search_for_mirror):

=== modified file 'cloudinit/distros/rhel.py'
--- cloudinit/distros/rhel.py	2014-02-03 22:03:14 +0000
+++ cloudinit/distros/rhel.py	2014-09-10 21:01:03 +0000
@@ -208,3 +208,15 @@
     def update_package_sources(self):
         self._runner.run("update-sources", self.package_command,
                          ["makecache"], freq=PER_INSTANCE)
+
+    def switch_user_cmd(self, user):
+        # In RHEL/CentOS sudo has requiretty defined and sudo can not be used
+        return ['runuser', '-u', user, '--']
+
+    def disable_network_manager(self):
+        # ifup fails if NetworkManager is running
+        try:
+            (out, _err) = util.subp(["systemctl", "stop", "NetworkManager"])
+            (out, _err) = util.subp(["systemctl", "disable", "NetworkManager"])
+        except util.ProcessExecutionError:
+            util.logexc(LOG, "Disable NetworkManager command failed")
\ No newline at end of file

=== modified file 'cloudinit/sources/DataSourceOpenNebula.py'
--- cloudinit/sources/DataSourceOpenNebula.py	2014-08-26 19:53:41 +0000
+++ cloudinit/sources/DataSourceOpenNebula.py	2014-09-10 21:01:03 +0000
@@ -42,7 +42,6 @@
 CONTEXT_DISK_FILES = ["context.sh"]
 VALID_DSMODES = ("local", "net", "disabled")
 
-
 class DataSourceOpenNebula(sources.DataSource):
     def __init__(self, sys_cfg, distro, paths):
         sources.DataSource.__init__(self, sys_cfg, distro, paths)
@@ -69,10 +68,12 @@
         for cdev in candidates:
             try:
                 if os.path.isdir(self.seed_dir):
-                    results = read_context_disk_dir(cdev, asuser=parseuser)
+                    results = read_context_disk_dir(cdev, { 'distro': self.distro,
+                                                            'asuser': parseuser })
                 elif cdev.startswith("/dev"):
                     results = util.mount_cb(cdev, read_context_disk_dir,
-                                            data=parseuser)
+                                            data={ 'distro': self.distro,
+                                                   'asuser': parseuser })
             except NonContextDiskDir:
                 continue
             except BrokenContextDiskDir as exc:
@@ -149,63 +150,99 @@
 
 class OpenNebulaNetwork(object):
     REG_DEV_MAC = re.compile(
-                    r'^\d+: (eth\d+):.*?link\/ether (..:..:..:..:..:..) ?',
+                    r'^\d+: (\w+):.*?link\/\w+ (..:..:..:..:..:..) ?',
                     re.MULTILINE | re.DOTALL)
 
-    def __init__(self, ip, context):
+    def __init__(self, ip, context, distro):
         self.ip = ip
         self.context = context
         self.ifaces = self.get_ifaces()
+        self.distro = distro
 
     def get_ifaces(self):
-        return self.REG_DEV_MAC.findall(self.ip)
+        list = self.REG_DEV_MAC.findall(self.ip)
+        ifaces = dict()
+        for l in list:
+            ifaces[l[1]] = l[0]
+        return ifaces
 
     def mac2ip(self, mac):
         components = mac.split(':')[2:]
         return [str(int(c, 16)) for c in components]
 
+    def get_iface_var(self, dev, var):
+        var_name = dev.upper() + '_' + var
+        if var_name in self.context:
+            return self.context[var_name]
+        else:
+            return None
+
     def get_ip(self, dev, components):
-        var_name = dev.upper() + '_IP'
-        if var_name in self.context:
-            return self.context[var_name]
+        var = self.get_iface_var(dev, 'IP')
+        if var:
+            return var
         else:
             return '.'.join(components)
 
     def get_mask(self, dev):
-        var_name = dev.upper() + '_MASK'
-        if var_name in self.context:
-            return self.context[var_name]
+        var = self.get_iface_var(dev, 'MASK')
+        if var:
+            return var
         else:
             return '255.255.255.0'
 
     def get_network(self, dev, components):
-        var_name = dev.upper() + '_NETWORK'
-        if var_name in self.context:
-            return self.context[var_name]
+        var = self.get_iface_var(dev, 'NETWORK')
+        if var:
+            return var
         else:
             return '.'.join(components[:-1]) + '.0'
 
+    def is_gateway_iface(self, dev):
+        if 'GATEWAY_IFACE' in self.context:
+            val=self.context['GATEWAY_IFACE']
+            dev_num=val.replace('ETH', '')
+            return ('ETH'+dev_num) == dev
+        else:
+            return True
+
     def get_gateway(self, dev):
-        var_name = dev.upper() + '_GATEWAY'
-        if var_name in self.context:
-            return self.context[var_name]
+        if self.is_gateway_iface(dev):
+            return self.get_iface_var(dev, 'GATEWAY')
         else:
             return None
 
     def get_dns(self, dev):
-        var_name = dev.upper() + '_DNS'
-        if var_name in self.context:
-            return self.context[var_name]
-        else:
-            return None
+        return self.get_iface_var(dev, 'DNS')
 
     def get_domain(self, dev):
-        var_name = dev.upper() + '_DOMAIN'
-        if var_name in self.context:
-            return self.context[var_name]
+        var = self.get_iface_var(dev, 'SEARCH_DOMAIN')
+        if var:
+            return var
+
+        return self.get_iface_var(dev, 'DOMAIN')
+
+    def get_force_ipv4(self, dev):
+        return self.get_iface_var(dev, 'CONTEXT_FORCE_IPV4')
+
+    def get_ip6(self, dev):
+        return self.get_iface_var(dev, 'IP6')
+
+    def get_gateway6(self, dev):
+        if self.is_gateway_iface(dev):
+            return self.get_iface_var(dev, 'GATEWAY6')
         else:
             return None
 
+    def get_context_interfaces(self):
+
+        def device_mac(t): return re.match(r"ETH\d+_MAC", t)
+
+        mac_vars = filter(device_mac, self.context.keys())
+
+        context_interfaces = [v.split('_')[0] for v in mac_vars]
+        return context_interfaces
+
     def gen_conf(self):
         global_dns = []
         if 'DNS' in self.context:
@@ -216,34 +253,65 @@
         conf.append('iface lo inet loopback')
         conf.append('')
 
-        for i in self.ifaces:
-            dev = i[0]
-            mac = i[1]
+        context_interfaces = self.get_context_interfaces()
+
+        if len(context_interfaces) and self.distro:
+            self.distro.disable_network_manager()
+
+        for interface in context_interfaces:
+            mac = self.context[interface+"_MAC"]
+            dev = self.ifaces[mac]
+
             ip_components = self.mac2ip(mac)
 
-            conf.append('auto ' + dev)
-            conf.append('iface ' + dev + ' inet static')
-            conf.append('  address ' + self.get_ip(dev, ip_components))
-            conf.append('  network ' + self.get_network(dev, ip_components))
-            conf.append('  netmask ' + self.get_mask(dev))
-
-            gateway = self.get_gateway(dev)
-            if gateway:
-                conf.append('  gateway ' + gateway)
-
-            domain = self.get_domain(dev)
-            if domain:
-                conf.append('  dns-search ' + domain)
-
-            # add global DNS servers to all interfaces
-            dns = self.get_dns(dev)
+            all_dns = None
+            dns = self.get_dns(interface)
             if global_dns or dns:
                 all_dns = global_dns
                 if dns:
                     all_dns.append(dns)
-                conf.append('  dns-nameservers ' + ' '.join(all_dns))
-
-            conf.append('')
+
+            ip6 = self.get_ip6(dev)
+            conf.append('auto ' + dev)
+
+            if self.get_force_ipv4(dev) or not ip6:
+                conf.append('iface ' + dev + ' inet static')
+                conf.append('  address ' + self.get_ip(interface, ip_components))
+                conf.append('  network ' + self.get_network(interface, ip_components))
+                conf.append('  netmask ' + self.get_mask(interface))
+
+                gateway = self.get_gateway(interface)
+                if gateway:
+                    conf.append('  gateway ' + gateway)
+
+                domain = self.get_domain(interface)
+                if domain:
+                    conf.append('  dns-search ' + domain)
+
+                # add global DNS servers to all interfaces
+                if all_dns:
+                    conf.append('  dns-nameservers ' + ' '.join(all_dns))
+
+                conf.append('')
+
+            if ip6:
+                conf.append('iface ' + dev + ' inet6 static')
+                conf.append('  address ' + ip6)
+                conf.append('  netmask 64')
+
+                gateway = self.get_gateway6(dev)
+                if gateway:
+                    conf.append('  gateway ' + gateway)
+
+                domain = self.get_domain(interface)
+                if domain:
+                    conf.append('  dns-search ' + domain)
+
+                # add global DNS servers to all interfaces
+                if all_dns:
+                    conf.append('  dns-nameservers ' + ' '.join(all_dns))
+
+                conf.append('')
 
         return "\n".join(conf)
 
@@ -353,12 +421,21 @@
     return ret
 
 
-def read_context_disk_dir(source_dir, asuser=None):
+def read_context_disk_dir(source_dir, options={}):
     """
     read_context_disk_dir(source_dir):
     read source_dir and return a tuple with metadata dict and user-data
     string populated.  If not a valid dir, raise a NonContextDiskDir
     """
+
+    asuser = None
+    if options.has_key('asuser'):
+        asuser = options['asuser']
+
+    distro = None
+    if options.has_key('distro'):
+        distro = options['distro']
+
     found = {}
     for af in CONTEXT_DISK_FILES:
         fn = os.path.join(source_dir, af)
@@ -382,7 +459,14 @@
             with open(os.path.join(source_dir, 'context.sh'), 'r') as f:
                 content = f.read().strip()
 
-            context = parse_shell_config(content, asuser=asuser)
+            if distro:
+                switch_cmd = distro.switch_user_cmd
+            else:
+                switch_cmd = None
+
+            context = parse_shell_config(content,
+                                         asuser=asuser,
+                                         switch_user_cb=switch_cmd)
         except util.ProcessExecutionError as e:
             raise BrokenContextDiskDir("Error processing context.sh: %s" % (e))
         except IOError as e:
@@ -409,7 +493,7 @@
 
     # custom hostname -- try hostname or leave cloud-init
     # itself create hostname from IP address later
-    for k in ('HOSTNAME', 'PUBLIC_IP', 'IP_PUBLIC', 'ETH0_IP'):
+    for k in ('SET_HOSTNAME', 'HOSTNAME', 'PUBLIC_IP', 'IP_PUBLIC', 'ETH0_IP'):
         if k in context:
             results['metadata']['local-hostname'] = context[k]
             break
@@ -436,7 +520,7 @@
     for k in context.keys():
         if re.match(r'^ETH\d+_IP$', k):
             (out, _) = util.subp(['/sbin/ip', 'link'])
-            net = OpenNebulaNetwork(out, context)
+            net = OpenNebulaNetwork(out, context, distro)
             results['network-interfaces'] = net.gen_conf()
             break
 
@@ -453,3 +537,4 @@
 # Return a list of data sources that match this set of dependencies
 def get_datasource_list(depends):
     return sources.list_from_depends(depends, datasources)
+

=== modified file 'tests/unittests/test_datasource/test_opennebula.py'
--- tests/unittests/test_datasource/test_opennebula.py	2014-07-23 16:50:45 +0000
+++ tests/unittests/test_datasource/test_opennebula.py	2014-09-10 21:01:03 +0000
@@ -234,29 +234,17 @@
         super(TestOpenNebulaNetwork, self).setUp()
 
     def test_lo(self):
-        net = ds.OpenNebulaNetwork('', {})
-        self.assertEqual(net.gen_conf(), u'''\
-auto lo
-iface lo inet loopback
-''')
-
-    def test_eth0(self):
-        net = ds.OpenNebulaNetwork(CMD_IP_OUT, {})
-        self.assertEqual(net.gen_conf(), u'''\
-auto lo
-iface lo inet loopback
-
-auto eth0
-iface eth0 inet static
-  address 10.18.1.1
-  network 10.18.1.0
-  netmask 255.255.255.0
+        net = ds.OpenNebulaNetwork('', {}, None)
+        self.assertEqual(net.gen_conf(), u'''\
+auto lo
+iface lo inet loopback
 ''')
 
     def test_eth0_override(self):
         context = {
             'DNS': '1.2.3.8',
             'ETH0_IP': '1.2.3.4',
+            'ETH0_MAC': '02:00:0a:12:01:01',
             'ETH0_NETWORK': '1.2.3.0',
             'ETH0_MASK': '255.255.0.0',
             'ETH0_GATEWAY': '1.2.3.5',
@@ -264,21 +252,103 @@
             'ETH0_DNS': '1.2.3.6 1.2.3.7'
         }
 
-        net = ds.OpenNebulaNetwork(CMD_IP_OUT, context)
-        self.assertEqual(net.gen_conf(), u'''\
-auto lo
-iface lo inet loopback
-
-auto eth0
-iface eth0 inet static
-  address 1.2.3.4
-  network 1.2.3.0
-  netmask 255.255.0.0
-  gateway 1.2.3.5
-  dns-search example.com
-  dns-nameservers 1.2.3.8 1.2.3.6 1.2.3.7
-''')
-
+        net = ds.OpenNebulaNetwork(CMD_IP_OUT, context, None)
+        self.assertEqual(net.gen_conf(), u'''\
+auto lo
+iface lo inet loopback
+
+auto eth0
+iface eth0 inet static
+  address 1.2.3.4
+  network 1.2.3.0
+  netmask 255.255.0.0
+  gateway 1.2.3.5
+  dns-search example.com
+  dns-nameservers 1.2.3.8 1.2.3.6 1.2.3.7
+''')
+
+    def test_eth0_override_with_gateway(self):
+        context = {
+            'DNS': '1.2.3.8',
+            'ETH0_IP': '1.2.3.4',
+            'ETH0_MAC': '02:00:0a:12:01:01',
+            'ETH0_NETWORK': '1.2.3.0',
+            'ETH0_MASK': '255.255.0.0',
+            'ETH0_GATEWAY': '1.2.3.5',
+            'ETH0_SEARCH_DOMAIN': 'example.com',
+            'ETH0_DNS': '1.2.3.6 1.2.3.7',
+            'GATEWAY_IFACE': 'ETH0'
+        }
+
+        net = ds.OpenNebulaNetwork(CMD_IP_OUT, context, None)
+        self.assertEqual(net.gen_conf(), u'''\
+auto lo
+iface lo inet loopback
+
+auto eth0
+iface eth0 inet static
+  address 1.2.3.4
+  network 1.2.3.0
+  netmask 255.255.0.0
+  gateway 1.2.3.5
+  dns-search example.com
+  dns-nameservers 1.2.3.8 1.2.3.6 1.2.3.7
+''')
+
+    def test_eth0_override_without_gateway(self):
+        context = {
+            'DNS': '1.2.3.8',
+            'ETH0_IP': '1.2.3.4',
+            'ETH0_MAC': '02:00:0a:12:01:01',
+            'ETH0_NETWORK': '1.2.3.0',
+            'ETH0_MASK': '255.255.0.0',
+            'ETH0_GATEWAY': '1.2.3.5',
+            'ETH0_DOMAIN': 'example.com',
+            'ETH0_DNS': '1.2.3.6 1.2.3.7',
+            'GATEWAY_IFACE': '1'
+        }
+
+        net = ds.OpenNebulaNetwork(CMD_IP_OUT, context, None)
+        self.assertEqual(net.gen_conf(), u'''\
+auto lo
+iface lo inet loopback
+
+auto eth0
+iface eth0 inet static
+  address 1.2.3.4
+  network 1.2.3.0
+  netmask 255.255.0.0
+  dns-search example.com
+  dns-nameservers 1.2.3.8 1.2.3.6 1.2.3.7
+''')
+
+    def test_eth0_override_with_ipv6(self):
+        context = {
+            'DNS': '1.2.3.8',
+            'ETH0_IP': '1.2.3.4',
+            'ETH0_MAC': '02:00:0a:12:01:01',
+            'ETH0_NETWORK': '1.2.3.0',
+            'ETH0_MASK': '255.255.0.0',
+            'ETH0_GATEWAY': '1.2.3.5',
+            'ETH0_DOMAIN': 'example.com',
+            'ETH0_DNS': '1.2.3.6 1.2.3.7',
+            'ETH0_IP6': '2001:470:801f::1',
+            'ETH0_GATEWAY6': '2001:470:1f05:15a::1'
+        }
+
+        net = ds.OpenNebulaNetwork(CMD_IP_OUT, context, None)
+        self.assertEqual(net.gen_conf(), u'''\
+auto lo
+iface lo inet loopback
+
+auto eth0
+iface eth0 inet6 static
+  address 2001:470:801f::1
+  netmask 64
+  gateway 2001:470:1f05:15a::1
+  dns-search example.com
+  dns-nameservers 1.2.3.8 1.2.3.6 1.2.3.7
+''')
 
 class TestParseShellConfig(MockerTestCase):
     def test_no_seconds(self):