Merge ~rjschwei/cloud-init:netV1ToTranslate into cloud-init:master

[Robert Schweikert]

Handle network configuration translation for the legacy path, do not drop gateway information

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:0acc956dc470fa201577c2b1267585217ed972d1
https://jenkins.ubuntu.com/server/job/cloud-init-ci/510/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/510/rebuild

review: Needs Fixing (continuous-integration)

4d3028d... by Robert Schweikert on 2017-11-19

- Style fixes
- Fix import

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:4d3028d8d083b3781335d5a062038a4ca3cd92b5
https://jenkins.ubuntu.com/server/job/cloud-init-ci/515/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    FAILED: MAAS Compatability Testing

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/515/rebuild

review: Needs Fixing (continuous-integration)

2f0c7c4... by Scott Moser on 2017-11-19

EC2: Kill dhclient process used in sandbox dhclient.

dhclient runs, obtains a address and then backgrounds itself.
cloud-init did not take care to kill it after it was done with it.
After it has run and created the leases, we can kill it.

LP: #1732964

0bee235... by Scott Moser on 2017-11-20

EC2: Fix bug using fallback_nic and metadata when restoring from cache.

If user upgraded to new cloud-init and attempted to run 'cloud-init init'
without rebooting, cloud-init restores the datasource object from pickle.
The older version pickled datasource object had no value for
_network_config or fallback_nic. This caused the Ec2 datasource to attempt
to reconfigure networking with a None fallback_nic. The pickled object
also cached an older version of ec2 metadata which didn't contain network
information.

This branch does two things:
 - Add a fallback_interface property to DatasourceEC2 to support reading the
   old .fallback_nic attribute if it was set. New versions will
   call net.find_fallback_nic() if there has not been one found.
 - Re-crawl metadata if we are on Ec2 and don't have a 'network' key in
   metadata

LP: #1732917

f8d7238... by Scott Moser on 2017-11-20

integration test: replace curtin test ppa with cloud-init test ppa.

Cloud-init integration tests should not depend on a curtin test ppa.
We already had a cloud-init test ppa for explicitly this purpose.
Just use it instead.

7d99045... by Ryan McCabe on 2017-11-20

sysconfig: Correctly render dns and dns search info.

Currently when dns and dns search info is provided, it is not rendered
when outputting to sysconfig format.

This patch causes the DNS and DOMAIN lines to be written out rendering
sysconfig.

LP: #1705804

36f7770... by Scott Moser on 2017-11-21

tests: Use apt-get to install a deb so that depends get resolved.

Instead of using 'dpkg -i' to install a package and then running
apt-get -f install, to hope that it would install needed dependencies
we can just use 'apt-get' directly to do the install.

The 'dpkg/apt-get -f' path was a problem if the installed deb was
older than the available deb. In that case it would get replaced.

462d4c1... by Joshua Powers on 2017-11-22

tests: Enable bionic in integration tests.

[Chad Smith (chad.smith)]

Looking over the failure, that MAAS compatibility testing hit an issue with an include url test unrelated to your branch. Can you git fetch on master and git rebase your branch against master and re-push that branch up. It will contain the fix 6ad23fe9b11f07e4404c8a1f2f1e9cba2640dceb which should resolve integration test issues hit on this branch.

ec12296... by Robert Schweikert on 2017-11-27

- Fix the regular expression for IPv4 address parsing
- Move the unit test into the "new" prefered locations

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:462d4c1710f08212d2b84f359e54242b2507f0b0
https://jenkins.ubuntu.com/server/job/cloud-init-ci/548/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/548/rebuild

review: Approve (continuous-integration)

[Robert Schweikert (rjschwei)]

@raharper and @chad.smith

Thanks for review and comments.

Fixed: Test location and IPv4 regular expression.

Comments and questions on the other info you provided.

As far as the test data is concerned, I kind of did a cut and paste from existing tests based on network configuration for the distros. I really have no idea what the various data sources can and cannot produce and what the debian/ubuntu format looks like. So test data that conforms to what the data sources produce would be better. If there is concrete data, that I should be using and have not used, please point me in that direction. Same can be applied to the suggested change of the search for "default gw", I agree that the proposed expression is more flexible, but does it need to be, i.e. do we have a data source where there is a varying space?

Yes there is always the risk that a new data source may format things differently, but the question would then be if we should compensate for such potential future discretions now or not.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:ec1229688e6fa7abef2b306d4e2ffd8f8f83cf3d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/550/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/550/rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

Robert, I agree on your suggestion to not be too flexible for future formats etc. So +1 on dropping my other re.match suggestion on 'default gw' handling. Yeah no need to waste resources planning for future-formats if we don't need to. Per your handling is duplicate (list type) entries in post-up, did you have specific use-cases in mind where multiple post-up commands are specified in /etc/network/interfaces in a single iface stanza?

review: Needs Information

Unmerged commits

ec12296... by Robert Schweikert on 2017-11-27

- Fix the regular expression for IPv4 address parsing
- Move the unit test into the "new" prefered locations

462d4c1... by Joshua Powers on 2017-11-22

tests: Enable bionic in integration tests.

36f7770... by Scott Moser on 2017-11-21

tests: Use apt-get to install a deb so that depends get resolved.

Instead of using 'dpkg -i' to install a package and then running
apt-get -f install, to hope that it would install needed dependencies
we can just use 'apt-get' directly to do the install.

The 'dpkg/apt-get -f' path was a problem if the installed deb was
older than the available deb. In that case it would get replaced.

7d99045... by Ryan McCabe on 2017-11-20

sysconfig: Correctly render dns and dns search info.

Currently when dns and dns search info is provided, it is not rendered
when outputting to sysconfig format.

This patch causes the DNS and DOMAIN lines to be written out rendering
sysconfig.

LP: #1705804

f8d7238... by Scott Moser on 2017-11-20

integration test: replace curtin test ppa with cloud-init test ppa.

Cloud-init integration tests should not depend on a curtin test ppa.
We already had a cloud-init test ppa for explicitly this purpose.
Just use it instead.

0bee235... by Scott Moser on 2017-11-20

EC2: Fix bug using fallback_nic and metadata when restoring from cache.

If user upgraded to new cloud-init and attempted to run 'cloud-init init'
without rebooting, cloud-init restores the datasource object from pickle.
The older version pickled datasource object had no value for
_network_config or fallback_nic. This caused the Ec2 datasource to attempt
to reconfigure networking with a None fallback_nic. The pickled object
also cached an older version of ec2 metadata which didn't contain network
information.

This branch does two things:
 - Add a fallback_interface property to DatasourceEC2 to support reading the
   old .fallback_nic attribute if it was set. New versions will
   call net.find_fallback_nic() if there has not been one found.
 - Re-crawl metadata if we are on Ec2 and don't have a 'network' key in
   metadata

LP: #1732917

2f0c7c4... by Scott Moser on 2017-11-19

EC2: Kill dhclient process used in sandbox dhclient.

dhclient runs, obtains a address and then backgrounds itself.
cloud-init did not take care to kill it after it was done with it.
After it has run and created the leases, we can kill it.

LP: #1732964

8cd5461... by Chad Smith on 2017-11-17

ntp: fix configuration template rendering for openSUSE and SLES

Add opensuse distro support to cc_ntp module.

LP: #1726572

c8604c4... by Chad Smith on 2017-11-17

centos: Provide the failed #include url in error messages

On python 2.7 and earlier (CentOS 6 & 7), UrlErrors raised by requests do
not report the url which failed. In such cases, append the url if not
present in the error message.

This fixes nightly CI failures at
https://jenkins.ubuntu.com/server/view/cloud-init/.

4d3028d... by Robert Schweikert on 2017-11-19

- Style fixes
- Fix import

diff --git a/cloudinit/distros/net_util.py b/cloudinit/distros/net_util.py
index 1ce1aa7..5f7394d 100644
--- a/cloudinit/distros/net_util.py
+++ b/cloudinit/distros/net_util.py
@@ -67,6 +67,11 @@
 #     }
 # }
 
+import re
+
+ipv4 = re.compile("\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}")
+
+
 def translate_network(settings):
     # Get the standard cmd, args from the ubuntu format
     entries = []
@@ -88,7 +93,14 @@ def translate_network(settings):
                 consume = {}
             consume[cmd] = args
         else:
-            consume[cmd] = args
+            if consume.get(cmd):
+                val = consume[cmd]
+                if isinstance(val, list):
+                    consume[cmd].append(args)
+                else:
+                    consume[cmd] = [val, args]
+            else:
+                consume[cmd] = args
     # Check if anything left over to consume
     absorb = False
     for (cmd, args) in consume.items():
@@ -148,6 +160,16 @@ def translate_network(settings):
                     hw_addr = hw_split[1]
                     if hw_addr:
                         iface_info['hwaddress'] = hw_addr
+            if 'post-up' in info:
+                routes = info['post-up']
+                if isinstance(routes, list):
+                    for route_info in routes:
+                        if 'default gw' in route_info:
+                            iface_info['gateway'] = ipv4.search(
+                                route_info).group(0)
+                elif 'default gw' in routes:
+                    iface_info['gateway'] = ipv4.search(routes).group(0)
+
         # If ipv6 is enabled, device will have multiple IPs, so we need to
         # update the dictionary instead of overwriting it...
         if dev_name in real_ifaces:
diff --git a/cloudinit/distros/tests/__init__.py b/cloudinit/distros/tests/__init__.py
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/cloudinit/distros/tests/__init__.py
diff --git a/cloudinit/distros/tests/test_net_util.py b/cloudinit/distros/tests/test_net_util.py
new file mode 100644
index 0000000..93426cf
--- /dev/null
+++ b/cloudinit/distros/tests/test_net_util.py
@@ -0,0 +1,103 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+
+from cloudinit.distros import net_util
+
+from cloudinit.tests.helpers import CiTestCase
+
+
+class TestNetworkConfigTransform(CiTestCase):
+
+    def test_basic_config_ipv4(self):
+        basic_conf = '''
+auto lo
+iface lo inet loopback
+
+auto eth0
+iface eth0 inet static
+    address 192.168.1.5
+    broadcast 192.168.1.0
+    gateway 192.168.1.254
+    netmask 255.255.255.0
+    network 192.168.0.0
+
+auto eth1
+iface eth1 inet dhcp
+'''
+        sysconfig = net_util.translate_network(basic_conf)
+        expected = {
+            'lo': {'auto': True, 'ipv6': {}},
+            'eth0': {
+                'auto': True,
+                'ipv6': {},
+                'broadcast': '192.168.1.0',
+                'netmask': '255.255.255.0',
+                'bootproto': 'static',
+                'address': '192.168.1.5',
+                'gateway': '192.168.1.254'
+            },
+            'eth1': {'auto': True, 'bootproto': 'dhcp', 'ipv6': {}}
+        }
+        self.assertEqual(sysconfig, expected)
+
+    def test_v1_confi_single_route_ipv4(self):
+        v1_conf = '''
+auto lo
+iface lo inet loopback
+
+auto eth0
+iface eth0 inet static
+    hwaddress fa:16:3e:ee:2b:97
+    address 192.168.168.30/24
+    mtu 1500
+    post-up route add default gw 192.168.168.1 || true
+    pre-down route del default gw 192.168.168.1 || true
+'''
+        sysconfig = net_util.translate_network(v1_conf)
+        expected = {
+            'lo': {'auto': True, 'ipv6': {}},
+            'eth0': {
+                'auto': True,
+                'bootproto': 'static',
+                'gateway': '192.168.168.1',
+                'address': '192.168.168.30/24',
+                'ipv6': {}
+            }
+        }
+        self.assertEqual(sysconfig, expected)
+
+    def test_v1_confi_multi_route_multi_nic_ipv4(self):
+        v1_conf = '''
+auto lo
+iface lo inet loopback
+
+auto eth0
+iface eth0 inet static
+    hwaddress fa:16:3e:ee:2b:97
+    address 192.168.168.30/24
+    mtu 1500
+    post-up route add default gw 192.168.168.1 || true
+    pre-down route del default gw 192.168.168.1 || true
+
+auto eth1
+iface eth1 inet dhcp
+    post-up route add 192.168.168.1 || true
+'''
+        sysconfig = net_util.translate_network(v1_conf)
+        expected = {
+            'lo': {'auto': True, 'ipv6': {}},
+            'eth0': {
+                'auto': True,
+                'bootproto': 'static',
+                'gateway': '192.168.168.1',
+                'address': '192.168.168.30/24',
+                'ipv6': {}
+            },
+            'eth1': {
+                'auto': True,
+                'bootproto': 'dhcp',
+                'ipv6': {}
+            }
+        }
+        self.assertEqual(sysconfig, expected)
+
+# vi: ts=4 expandtab