Merge ~enolfc/cloud-init:bug-1716397 into cloud-init:master

[Enol Fernández]

OpenNebula: support network configuration with persistent named devices.

OpenNebula network configuration is provided in context with device
identifiers in the form ETH[0-9]. That assumes that the host can "know"
the name of the device in the guest, which is unreliable at best and
completely broken when predictable names are used.

The change here now utilizes context variables named ETH<N>_MAC that
provide the mac address for the 'ETH<N>' device.

LP: #1716397

Fixes bug #1716397

[Enol Fernández (enolfc)]

Tests are broken, will submit new commits soon

[Scott Moser (smoser)]

Thank you for your contribution to cloud-init. Contributions to
cloud-init require the developer to have signed the Canonical Contributors
Agreement [1]. Your launchpad id is not currently listed as a member of
the contributor-agreement-canonical group [2].

Membership in that group is currently managed by a manual process, so it
is possible that you have followed the process but were not added to the
group. If you have already signed the contributor agreement, please
simply say so here.

For more information on contribution see the hacking documentation in
cloud-init [3].

--
[1] https://www.ubuntu.com/legal/contributors
[2] https://launchpad.net/~contributor-agreement-canonical/+members
[3] http://cloudinit.readthedocs.io/en/latest/topics/hacking.html

[Scott Moser (smoser)]

is _MAC expected present in all OpenNebula releases ? or is that something only provided in newer releases ? It'd be good to comment about when it was present.

Thanks for raising this and helping out.

[Enol Fernández (enolfc)]

The first reference I could find to this is in OpenNebula 4.4:
http://docs.opennebula.org/4.4/user/virtual_machine_setup/cong.html#network-configuration

I can include a fallback if the _MAC is not available.

On 11 September 2017 at 17:11, Scott Moser <email address hidden> wrote:

> is _MAC expected present in all OpenNebula releases ? or is that something
> only provided in newer releases ? It'd be good to comment about when it
> was present.
>
> Thanks for raising this and helping out.
>
> --
> https://code.launchpad.net/~enolfc/cloud-init/+git/cloud-
> init/+merge/330518
> You are the owner of ~enolfc/cloud-init:bug-1716397.
>

--
Enol Fernández | Cloud Technologist | EGI Foundation
<email address hidden>

[Scott Moser (smoser)]

For my own reference, 4.4 was Dec 2013
 https://opennebula.org/software/release/

so that is "pretty old".

[Enol Fernández (enolfc)]

Hi,

I'm asking my company to sign the agreement. Once that's done, I should get
back to you to add me to the group?

Enol

On 11 September 2017 at 17:11, Scott Moser <email address hidden> wrote:

> Thank you for your contribution to cloud-init. Contributions to
> cloud-init require the developer to have signed the Canonical Contributors
> Agreement [1]. Your launchpad id is not currently listed as a member of
> the contributor-agreement-canonical group [2].
>
> Membership in that group is currently managed by a manual process, so it
> is possible that you have followed the process but were not added to the
> group. If you have already signed the contributor agreement, please
> simply say so here.
>
> For more information on contribution see the hacking documentation in
> cloud-init [3].
>
> --
> [1] https://www.ubuntu.com/legal/contributors
> [2] https://launchpad.net/~contributor-agreement-canonical/+members
> [3] http://cloudinit.readthedocs.io/en/latest/topics/hacking.html
>
> --
> https://code.launchpad.net/~enolfc/cloud-init/+git/cloud-
> init/+merge/330518
> You are the owner of ~enolfc/cloud-init:bug-1716397.
>

--
Enol Fernández | Cloud Technologist | EGI Foundation
<email address hidden>

[Scott Moser (smoser)]

Yes, please let me know when you or your company has signed. Unfortunately
there is some manual process still around, so as soon as that's done I'll
accept your code but you might not show in the group right away.
Thanks

On Wed, Sep 13, 2017 at 10:04 AM, Enol Fernández <email address hidden>
wrote:

> Hi,
>
> I'm asking my company to sign the agreement. Once that's done, I should get
> back to you to add me to the group?
>
> Enol
>
> On 11 September 2017 at 17:11, Scott Moser <email address hidden> wrote:
>
> > Thank you for your contribution to cloud-init. Contributions to
> > cloud-init require the developer to have signed the Canonical
> Contributors
> > Agreement [1]. Your launchpad id is not currently listed as a member of
> > the contributor-agreement-canonical group [2].
> >
> > Membership in that group is currently managed by a manual process, so it
> > is possible that you have followed the process but were not added to the
> > group. If you have already signed the contributor agreement, please
> > simply say so here.
> >
> > For more information on contribution see the hacking documentation in
> > cloud-init [3].
> >
> > --
> > [1] https://www.ubuntu.com/legal/contributors
> > [2] https://launchpad.net/~contributor-agreement-canonical/+members
> > [3] http://cloudinit.readthedocs.io/en/latest/topics/hacking.html
> >
> > --
> > https://code.launchpad.net/~enolfc/cloud-init/+git/cloud-
> > init/+merge/330518
> > You are the owner of ~enolfc/cloud-init:bug-1716397.
> >
>
>
>
> --
> Enol Fernández | Cloud Technologist | EGI Foundation
> <email address hidden>
>
> https://code.launchpad.net/~enolfc/cloud-init/+git/cloud-
> init/+merge/330518
> You are subscribed to branch cloud-init:master.
>

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:1e704904ffb283afddd87c05616f3967b896699d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/289/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/289/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:0f71ba25af6a754337ed700f0c55dde1b14679b0
https://jenkins.ubuntu.com/server/job/cloud-init-ci/319/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/319/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

Enol,
I'm sorry to bother you, but did you get anywhere on the CLA ?

Thanks,
Scott

[Enol Fernández (enolfc)]

Hi Scott,

I did not get a reply back yet. Will ping again.

E.

On 21 September 2017 at 02:15, Scott Moser <email address hidden> wrote:

> Enol,
> I'm sorry to bother you, but did you get anywhere on the CLA ?
>
> Thanks,
> Scott
> --
> https://code.launchpad.net/~enolfc/cloud-init/+git/cloud-
> init/+merge/330518
> You are the owner of ~enolfc/cloud-init:bug-1716397.
>

--
Enol Fernández | Cloud Technologist | EGI Foundation
<email address hidden>

[Scott Moser (smoser)]

Enol,

Just as housekeeping I'm going to put this into 'work in progress'.
Please feel free to comment and move it back.

Please feel free to ping me in email or in irc if you have any questions.

Thanks for your contribution.

Scott

Unmerged commits

0f71ba2... by Enol Fernández on 2017-09-11

OpenNebula: support network configuration with persistent named devices

OpenNebula network configuration is provided in context with device
identifiers in the form ETH[0-9]. That assumes that the host can "know"
the name of the device in the guest, which is unreliable at best and
completely broken when predictable names are used.

The change here now utilizes context variables named ETH<N>_MAC that
provide the mac address for the 'ETH<N>' device.

diff --git a/cloudinit/sources/DataSourceOpenNebula.py b/cloudinit/sources/DataSourceOpenNebula.py
index 5fdac19..2d2a6d0 100644
--- a/cloudinit/sources/DataSourceOpenNebula.py
+++ b/cloudinit/sources/DataSourceOpenNebula.py
@@ -19,6 +19,7 @@ import string
 
 from cloudinit import log as logging
 from cloudinit import net
+from cloudinit.net import eni
 from cloudinit import sources
 from cloudinit import util
 
@@ -86,7 +87,7 @@ class DataSourceOpenNebula(sources.DataSource):
             return False
 
         self.seed = seed
-        self.network_eni = results.get("network_config")
+        self.network_eni = results.get("network-interfaces")
         self.metadata = md
         self.userdata_raw = results.get('userdata')
         return True
@@ -99,6 +100,12 @@ class DataSourceOpenNebula(sources.DataSource):
                 resolve_ip = False
         return sources.DataSource.get_hostname(self, fqdn, resolve_ip)
 
+    @property
+    def network_config(self):
+        if self.network_eni:
+            return eni.convert_eni_data(self.network_eni)
+        return None
+
 
 class NonContextDiskDir(Exception):
     pass
@@ -161,6 +168,13 @@ class OpenNebulaNetwork(object):
         else:
             return None
 
+    def mac2dev(self, mac, dev=''):
+        for k in self.context:
+            m = re.match(r'(ETH\d+)_MAC$', k)
+            if self.context[k] == mac:
+                return m.groups()[0]
+        return dev
+
     def gen_conf(self):
         global_dns = []
         if 'DNS' in self.context:
@@ -174,22 +188,25 @@ class OpenNebulaNetwork(object):
         for mac, dev in self.ifaces.items():
             ip_components = self.mac2ip(mac)
 
+            one_dev = self.mac2dev(mac, dev)
+
             conf.append('auto ' + dev)
             conf.append('iface ' + dev + ' inet static')
-            conf.append('  address ' + self.get_ip(dev, ip_components))
-            conf.append('  network ' + self.get_network(dev, ip_components))
-            conf.append('  netmask ' + self.get_mask(dev))
+            conf.append('  address ' + self.get_ip(one_dev, ip_components))
+            conf.append('  network ' + self.get_network(one_dev,
+                                                        ip_components))
+            conf.append('  netmask ' + self.get_mask(one_dev))
 
-            gateway = self.get_gateway(dev)
+            gateway = self.get_gateway(one_dev)
             if gateway:
                 conf.append('  gateway ' + gateway)
 
-            domain = self.get_domain(dev)
+            domain = self.get_domain(one_dev)
             if domain:
                 conf.append('  dns-search ' + domain)
 
             # add global DNS servers to all interfaces
-            dns = self.get_dns(dev)
+            dns = self.get_dns(one_dev)
             if global_dns or dns:
                 all_dns = global_dns
                 if dns:
diff --git a/tests/unittests/test_datasource/test_opennebula.py b/tests/unittests/test_datasource/test_opennebula.py
index e7d5569..765c195 100644
--- a/tests/unittests/test_datasource/test_opennebula.py
+++ b/tests/unittests/test_datasource/test_opennebula.py
@@ -235,7 +235,8 @@ class TestOpenNebulaDataSource(TestCase):
 
 class TestOpenNebulaNetwork(unittest.TestCase):
 
-    system_nics = {'02:00:0a:12:01:01': 'eth0'}
+    eth0_system_nics = {'02:00:0a:12:01:01': 'eth0'}
+    ens0_system_nics = {'02:00:0a:12:01:01': 'ens0'}
 
     def test_lo(self):
         net = ds.OpenNebulaNetwork(context={}, system_nics_by_mac={})
@@ -244,9 +245,25 @@ auto lo
 iface lo inet loopback
 ''')
 
+    def test_dev2mac_eth0(self):
+        context = {
+            'ETH0_MAC': '02:00:0a:12:01:01'
+        }
+
+        net = ds.OpenNebulaNetwork(context,
+                                   system_nics_by_mac=self.eth0_system_nics)
+        one_dev = net.mac2dev('02:00:0a:12:01:01')
+        self.assertEqual(one_dev, 'ETH0')
+
+    def test_dev2mac_default(self):
+        net = ds.OpenNebulaNetwork({},
+                                   system_nics_by_mac=self.eth0_system_nics)
+        one_dev = net.mac2dev('02:00:0a:12:01:01', 'FOO')
+        self.assertEqual(one_dev, 'FOO')
+
     @mock.patch(DS_PATH + ".get_physical_nics_by_mac")
     def test_eth0(self, m_get_phys_by_mac):
-        m_get_phys_by_mac.return_value = self.system_nics
+        m_get_phys_by_mac.return_value = self.eth0_system_nics
         net = ds.OpenNebulaNetwork({})
         self.assertEqual(net.gen_conf(), u'''\
 auto lo
@@ -259,6 +276,61 @@ iface eth0 inet static
   netmask 255.255.255.0
 ''')
 
+    def test_no_eth_system_nic(self):
+        context = {
+            'DNS': '1.2.3.8',
+            'ETH0_IP': '1.2.3.4',
+            'ETH0_NETWORK': '1.2.3.0',
+            'ETH0_MASK': '255.255.0.0',
+            'ETH0_GATEWAY': '1.2.3.5',
+            'ETH0_DOMAIN': 'example.com',
+            'ETH0_DNS': '1.2.3.6 1.2.3.7',
+            'ETH0_MAC': '02:00:0a:12:01:01'
+        }
+
+        net = ds.OpenNebulaNetwork(context,
+                                   system_nics_by_mac=self.ens0_system_nics)
+        self.assertEqual(net.gen_conf(), u'''\
+auto lo
+iface lo inet loopback
+
+auto ens0
+iface ens0 inet static
+  address 1.2.3.4
+  network 1.2.3.0
+  netmask 255.255.0.0
+  gateway 1.2.3.5
+  dns-search example.com
+  dns-nameservers 1.2.3.8 1.2.3.6 1.2.3.7
+''')
+
+    def test_eth0_override_no_mac(self):
+        context = {
+            'DNS': '1.2.3.8',
+            'ETH0_IP': '1.2.3.4',
+            'ETH0_NETWORK': '1.2.3.0',
+            'ETH0_MASK': '255.255.0.0',
+            'ETH0_GATEWAY': '1.2.3.5',
+            'ETH0_DOMAIN': 'example.com',
+            'ETH0_DNS': '1.2.3.6 1.2.3.7',
+        }
+
+        net = ds.OpenNebulaNetwork(context,
+                                   system_nics_by_mac=self.eth0_system_nics)
+        self.assertEqual(net.gen_conf(), u'''\
+auto lo
+iface lo inet loopback
+
+auto eth0
+iface eth0 inet static
+  address 1.2.3.4
+  network 1.2.3.0
+  netmask 255.255.0.0
+  gateway 1.2.3.5
+  dns-search example.com
+  dns-nameservers 1.2.3.8 1.2.3.6 1.2.3.7
+''')
+
     def test_eth0_override(self):
         context = {
             'DNS': '1.2.3.8',
@@ -267,11 +339,12 @@ iface eth0 inet static
             'ETH0_MASK': '255.255.0.0',
             'ETH0_GATEWAY': '1.2.3.5',
             'ETH0_DOMAIN': 'example.com',
-            'ETH0_DNS': '1.2.3.6 1.2.3.7'
+            'ETH0_DNS': '1.2.3.6 1.2.3.7',
+            'ETH0_MAC': '02:00:0a:12:01:01'
         }
 
         net = ds.OpenNebulaNetwork(context,
-                                   system_nics_by_mac=self.system_nics)
+                                   system_nics_by_mac=self.eth0_system_nics)
         self.assertEqual(net.gen_conf(), u'''\
 auto lo
 iface lo inet loopback