Merge ~chad.smith/cloud-init:bug/17746666-netplan-mtu-on-bonds into cloud-init:master

[Chad Smith]

netplan: fix mtu if provided by network config for all rendered types

When network configuration for any interface defines maximum transmission
values (MTU) the netplan, eni and sysconfig renders will take into account
any device-level, or subnet-level mtu values.

When network configuration has conflicting device-level and ipv4 subnet
mtu values, the subnet-specific value is honored and a warning will be
logged about any ignored device-level setting.

LP: #1774666

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:2240d2096536d914ebaed472d1c19ea98eda19ab
https://jenkins.ubuntu.com/server/job/cloud-init-ci/51/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/51/rebuild

review: Needs Fixing (continuous-integration)

6956c65... by Chad Smith on 2018-06-07

netplan: render bond and vlan mtu if provided by network config

When network configuration for vlan and bond interfaces declare
mtu values, the netplan renderer will pass those config values
into netplan.

LP: #1774666

4eb1912... by Chad Smith on 2018-06-07

rtd docs adding mtu to bond and vlan

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:885901fd730f4e1310f33256fd87e282d30e49bf
https://jenkins.ubuntu.com/server/job/cloud-init-ci/52/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/52/rebuild

review: Needs Fixing (continuous-integration)

[Ryan Harper (raharper)]

Let's doing this generically in the _extract_addresses method; that gets called on all types.

diff --git a/cloudinit/net/netplan.py b/cloudinit/net/netplan.py
index 6344348..817f415 100644
--- a/cloudinit/net/netplan.py
+++ b/cloudinit/net/netplan.py
@@ -124,6 +124,9 @@ def _extract_addresses(config, entry):

             addresses.append(addr)

+ if 'mtu' in config:
+ entry.update({'mtu': config.get('mtu')})
+
     if len(addresses) > 0:
         entry.update({'addresses': addresses})
     if len(routes) > 0:

review: Needs Fixing

51004fc... by Chad Smith on 2018-06-07

address review commment handle mtu config settings in comment _extract_addresses function

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:f9fa0a562ed33ad4ed96a715555e42d4b151e273
https://jenkins.ubuntu.com/server/job/cloud-init-ci/53/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/53/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:bf2c1d564f88919289f3688927d859c9a6c6eae1
https://jenkins.ubuntu.com/server/job/cloud-init-ci/55/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/55/rebuild

review: Approve (continuous-integration)

8470eef... by Chad Smith on 2018-06-07

use subnet's ipv4 mtu over device-level mtu config

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:8c6c6fbe3bd08f28e1cc510a68ead09dc3bfe6bb
https://jenkins.ubuntu.com/server/job/cloud-init-ci/56/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/56/rebuild

review: Needs Fixing (continuous-integration)

[Ryan Harper (raharper)]

This looks good, likely needs a sysconfig update for this path as well.

12c7a50... by Chad Smith on 2018-06-08

pycodestyle fix

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:12c7a50095cb77f4a6e0139d7968db0e09c7c3e1
https://jenkins.ubuntu.com/server/job/cloud-init-ci/60/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/60/rebuild

review: Approve (continuous-integration)

10a79a4... by Chad Smith on 2018-06-08

sysconfig should also emit a warning if device-level mtu is provided and ipv4-subnet also provided

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:10a79a40daeca34b39d8bce66a53a4646b92aa9d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/61/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/61/rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

Let's only warn if the mtu values differ between device/subnet values.

9b946e1... by Chad Smith on 2018-06-08

net render: only warn if device-level mtu != subnet-level mtu

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:9b946e16376ac062b305af1e6827be4dcd67e4b6
https://jenkins.ubuntu.com/server/job/cloud-init-ci/62/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/62/rebuild

review: Approve (continuous-integration)

e4486d4... by Chad Smith on 2018-06-11

Add an assert that we only emit warnings when device-level mtu does not match subnet-level mtu if both are present.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:e4486d4a33ae3c6cfcf68065685db337552d7cca
https://jenkins.ubuntu.com/server/job/cloud-init-ci/65/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/65/rebuild

review: Needs Fixing (continuous-integration)

[Ryan Harper (raharper)]

Let's update the commit message to indicate that generally we're fixing mtu value rendering in netplan (that covers all types), For the docs, move the MTU value under the Common settings; since it is indeed common in netplan (and now in how we apply config).

Lastly mention that we now detect and log a warning if a per-subnet mtu value differs from an mtu setting on the device.

0cef9d1... by Chad Smith on 2018-06-11

drop debug maxDiff

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:0cef9d16945e2d589fe1bfdda2c87db9459688a8
https://jenkins.ubuntu.com/server/job/cloud-init-ci/66/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/66/rebuild

review: Approve (continuous-integration)

5138860... by Chad Smith on 2018-06-12

add mtu config docs to common netplan rtd

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:51388607f3dd3e4b3ce9c8a50c804f01c0eabbed
https://jenkins.ubuntu.com/server/job/cloud-init-ci/70/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/70/rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=c3f1ad9a

diff --git a/cloudinit/net/eni.py b/cloudinit/net/eni.py
index c6a71d1..bd20a36 100644
--- a/cloudinit/net/eni.py
+++ b/cloudinit/net/eni.py
@@ -10,9 +10,12 @@ from . import ParserError
 from . import renderer
 from .network_state import subnet_is_ipv6
 
+from cloudinit import log as logging
 from cloudinit import util
 
 
+LOG = logging.getLogger(__name__)
+
 NET_CONFIG_COMMANDS = [
     "pre-up", "up", "post-up", "down", "pre-down", "post-down",
 ]
@@ -61,7 +64,7 @@ def _iface_add_subnet(iface, subnet):
 
 
 # TODO: switch to valid_map for attrs
-def _iface_add_attrs(iface, index):
+def _iface_add_attrs(iface, index, ipv4_subnet_mtu):
     # If the index is non-zero, this is an alias interface. Alias interfaces
     # represent additional interface addresses, and should not have additional
     # attributes. (extra attributes here are almost always either incorrect,
@@ -100,6 +103,13 @@ def _iface_add_attrs(iface, index):
             value = 'on' if iface[key] else 'off'
         if not value or key in ignore_map:
             continue
+        if key == 'mtu' and ipv4_subnet_mtu:
+            if value != ipv4_subnet_mtu:
+                LOG.warning(
+                    "Network config: ignoring %s device-level mtu:%s because"
+                    " ipv4 subnet-level mtu:%s provided.",
+                    iface['name'], value, ipv4_subnet_mtu)
+            continue
         if key in multiline_keys:
             for v in value:
                 content.append("    {0} {1}".format(renames.get(key, key), v))
@@ -377,12 +387,15 @@ class Renderer(renderer.Renderer):
         subnets = iface.get('subnets', {})
         if subnets:
             for index, subnet in enumerate(subnets):
+                ipv4_subnet_mtu = None
                 iface['index'] = index
                 iface['mode'] = subnet['type']
                 iface['control'] = subnet.get('control', 'auto')
                 subnet_inet = 'inet'
                 if subnet_is_ipv6(subnet):
                     subnet_inet += '6'
+                else:
+                    ipv4_subnet_mtu = subnet.get('mtu')
                 iface['inet'] = subnet_inet
                 if subnet['type'].startswith('dhcp'):
                     iface['mode'] = 'dhcp'
@@ -397,7 +410,7 @@ class Renderer(renderer.Renderer):
                     _iface_start_entry(
                         iface, index, render_hwaddress=render_hwaddress) +
                     _iface_add_subnet(iface, subnet) +
-                    _iface_add_attrs(iface, index)
+                    _iface_add_attrs(iface, index, ipv4_subnet_mtu)
                 )
                 for route in subnet.get('routes', []):
                     lines.extend(self._render_route(route, indent="    "))
@@ -409,7 +422,8 @@ class Renderer(renderer.Renderer):
             if 'bond-master' in iface or 'bond-slaves' in iface:
                 lines.append("auto {name}".format(**iface))
             lines.append("iface {name} {inet} {mode}".format(**iface))
-            lines.extend(_iface_add_attrs(iface, index=0))
+            lines.extend(
+                _iface_add_attrs(iface, index=0, ipv4_subnet_mtu=None))
             sections.append(lines)
         return sections
 
diff --git a/cloudinit/net/netplan.py b/cloudinit/net/netplan.py
index 6344348..4014363 100644
--- a/cloudinit/net/netplan.py
+++ b/cloudinit/net/netplan.py
@@ -34,7 +34,7 @@ def _get_params_dict_by_match(config, match):
                 if key.startswith(match))
 
 
-def _extract_addresses(config, entry):
+def _extract_addresses(config, entry, ifname):
     """This method parse a cloudinit.net.network_state dictionary (config) and
        maps netstate keys/values into a dictionary (entry) to represent
        netplan yaml.
@@ -124,6 +124,15 @@ def _extract_addresses(config, entry):
 
             addresses.append(addr)
 
+    if 'mtu' in config:
+        entry_mtu = entry.get('mtu')
+        if entry_mtu and config['mtu'] != entry_mtu:
+            LOG.warning(
+                "Network config: ignoring %s device-level mtu:%s because"
+                " ipv4 subnet-level mtu:%s provided.",
+                ifname, config['mtu'], entry_mtu)
+        else:
+            entry['mtu'] = config['mtu']
     if len(addresses) > 0:
         entry.update({'addresses': addresses})
     if len(routes) > 0:
@@ -262,10 +271,7 @@ class Renderer(renderer.Renderer):
                     else:
                         del eth['match']
                         del eth['set-name']
-                if 'mtu' in ifcfg:
-                    eth['mtu'] = ifcfg.get('mtu')
-
-                _extract_addresses(ifcfg, eth)
+                _extract_addresses(ifcfg, eth, ifname)
                 ethernets.update({ifname: eth})
 
             elif if_type == 'bond':
@@ -288,7 +294,7 @@ class Renderer(renderer.Renderer):
                 slave_interfaces = ifcfg.get('bond-slaves')
                 if slave_interfaces == 'none':
                     _extract_bond_slaves_by_name(interfaces, bond, ifname)
-                _extract_addresses(ifcfg, bond)
+                _extract_addresses(ifcfg, bond, ifname)
                 bonds.update({ifname: bond})
 
             elif if_type == 'bridge':
@@ -321,7 +327,7 @@ class Renderer(renderer.Renderer):
 
                 if len(br_config) > 0:
                     bridge.update({'parameters': br_config})
-                _extract_addresses(ifcfg, bridge)
+                _extract_addresses(ifcfg, bridge, ifname)
                 bridges.update({ifname: bridge})
 
             elif if_type == 'vlan':
@@ -333,7 +339,7 @@ class Renderer(renderer.Renderer):
                 macaddr = ifcfg.get('mac_address', None)
                 if macaddr is not None:
                     vlan['macaddress'] = macaddr.lower()
-                _extract_addresses(ifcfg, vlan)
+                _extract_addresses(ifcfg, vlan, ifname)
                 vlans.update({ifname: vlan})
 
         # inject global nameserver values under each all interface which
diff --git a/cloudinit/net/sysconfig.py b/cloudinit/net/sysconfig.py
index e53b9f1..3d71923 100644
--- a/cloudinit/net/sysconfig.py
+++ b/cloudinit/net/sysconfig.py
@@ -304,6 +304,13 @@ class Renderer(renderer.Renderer):
                     mtu_key = 'IPV6_MTU'
                     iface_cfg['IPV6INIT'] = True
                 if 'mtu' in subnet:
+                    mtu_mismatch = bool(mtu_key in iface_cfg and
+                                        subnet['mtu'] != iface_cfg[mtu_key])
+                    if mtu_mismatch:
+                        LOG.warning(
+                            'Network config: ignoring %s device-level mtu:%s'
+                            ' because ipv4 subnet-level mtu:%s provided.',
+                            iface_cfg.name, iface_cfg[mtu_key], subnet['mtu'])
                     iface_cfg[mtu_key] = subnet['mtu']
             elif subnet_type == 'manual':
                 # If the subnet has an MTU setting, then ONBOOT=True
diff --git a/doc/rtd/topics/network-config-format-v1.rst b/doc/rtd/topics/network-config-format-v1.rst
index 2f8ab54..3b0148c 100644
--- a/doc/rtd/topics/network-config-format-v1.rst
+++ b/doc/rtd/topics/network-config-format-v1.rst
@@ -130,6 +130,18 @@ the bond interfaces.
 The ``bond_interfaces`` key accepts a list of network device ``name`` values
 from the configuration.  This list may be empty.
 
+**mtu**: *<MTU SizeBytes>*
+
+The MTU key represents a device's Maximum Transmission Unit, the largest size
+packet or frame, specified in octets (eight-bit bytes), that can be sent in a
+packet- or frame-based network.  Specifying ``mtu`` is optional.
+
+.. note::
+
+  The possible supported values of a device's MTU is not available at
+  configuration time.  It's possible to specify a value too large or to
+  small for a device and may be ignored by the device.
+
 **params**:  *<Dictionary of key: value bonding parameter pairs>*
 
 The ``params`` key in a bond holds a dictionary of bonding parameters.
@@ -268,6 +280,21 @@ Type ``vlan`` requires the following keys:
 - ``vlan_link``: Specify the underlying link via its ``name``.
 - ``vlan_id``: Specify the VLAN numeric id.
 
+The following optional keys are supported:
+
+**mtu**: *<MTU SizeBytes>*
+
+The MTU key represents a device's Maximum Transmission Unit, the largest size
+packet or frame, specified in octets (eight-bit bytes), that can be sent in a
+packet- or frame-based network.  Specifying ``mtu`` is optional.
+
+.. note::
+
+  The possible supported values of a device's MTU is not available at
+  configuration time.  It's possible to specify a value too large or to
+  small for a device and may be ignored by the device.
+
+
 **VLAN Example**::
 
    network:
diff --git a/doc/rtd/topics/network-config-format-v2.rst b/doc/rtd/topics/network-config-format-v2.rst
index 335d236..ea370ef 100644
--- a/doc/rtd/topics/network-config-format-v2.rst
+++ b/doc/rtd/topics/network-config-format-v2.rst
@@ -174,6 +174,12 @@ recognized by ``inet_pton(3)``
 Example for IPv4: ``gateway4: 172.16.0.1``
 Example for IPv6: ``gateway6: 2001:4::1``
 
+**mtu**: *<MTU SizeBytes>*
+
+The MTU key represents a device's Maximum Transmission Unit, the largest size
+packet or frame, specified in octets (eight-bit bytes), that can be sent in a
+packet- or frame-based network.  Specifying ``mtu`` is optional.
+
 **nameservers**: *<(mapping)>*
 
 Set DNS servers and search domains, for manual address configuration. There
diff --git a/tests/unittests/test_net.py b/tests/unittests/test_net.py
index e13ca3c..5ab61cf 100644
--- a/tests/unittests/test_net.py
+++ b/tests/unittests/test_net.py
@@ -525,6 +525,7 @@ NETWORK_CONFIGS = {
             config:
               - type: 'physical'
                 name: 'iface0'
+                mtu: 8999
                 subnets:
                   - type: static
                     address: 192.168.14.2/24
@@ -660,8 +661,8 @@ iface eth0.101 inet static
     dns-nameservers 192.168.0.10 10.23.23.134
     dns-search barley.maas sacchromyces.maas brettanomyces.maas
     gateway 192.168.0.1
-    hwaddress aa:bb:cc:dd:ee:11
     mtu 1500
+    hwaddress aa:bb:cc:dd:ee:11
     vlan-raw-device eth0
     vlan_id 101
 
@@ -757,6 +758,7 @@ pre-down route del -net 10.0.0.0 netmask 255.0.0.0 gw 11.0.0.1 metric 3 || true
                         id: 101
                         link: eth0
                         macaddress: aa:bb:cc:dd:ee:11
+                        mtu: 1500
                         nameservers:
                             addresses:
                             - 192.168.0.10
@@ -920,6 +922,8 @@ pre-down route del -net 10.0.0.0 netmask 255.0.0.0 gw 11.0.0.1 metric 3 || true
                   mtu: 1500
                   subnets:
                     - type: static
+                      # When 'mtu' matches device-level mtu, no warnings
+                      mtu: 1500
                       address: 192.168.0.2/24
                       gateway: 192.168.0.1
                       dns_nameservers:
@@ -1028,6 +1032,7 @@ pre-down route del -net 10.0.0.0 netmask 255.0.0.0 gw 11.0.0.1 metric 3 || true
               - type: bond
                 name: bond0
                 mac_address: "aa:bb:cc:dd:e8:ff"
+                mtu: 9000
                 bond_interfaces:
                   - bond0s0
                   - bond0s1
@@ -1070,6 +1075,7 @@ pre-down route del -net 10.0.0.0 netmask 255.0.0.0 gw 11.0.0.1 metric 3 || true
                      interfaces:
                      - bond0s0
                      - bond0s1
+                     mtu: 9000
                      parameters:
                          mii-monitor-interval: 100
                          mode: active-backup
@@ -1157,6 +1163,7 @@ pre-down route del -net 10.0.0.0 netmask 255.0.0.0 gw 11.0.0.1 metric 3 || true
         IPADDR1=192.168.1.2
         IPV6ADDR=2001:1::1/92
         IPV6INIT=yes
+        MTU=9000
         NETMASK=255.255.255.0
         NETMASK1=255.255.255.0
         NM_CONTROLLED=no
@@ -1203,6 +1210,7 @@ pre-down route del -net 10.0.0.0 netmask 255.0.0.0 gw 11.0.0.1 metric 3 || true
                 name: en0
                 mac_address: "aa:bb:cc:dd:e8:00"
               - type: vlan
+                mtu: 2222
                 name: en0.99
                 vlan_link: en0
                 vlan_id: 99
@@ -1238,6 +1246,7 @@ pre-down route del -net 10.0.0.0 netmask 255.0.0.0 gw 11.0.0.1 metric 3 || true
                 IPV6ADDR=2001:1::bbbb/96
                 IPV6INIT=yes
                 IPV6_DEFAULTGW=2001:1::1
+                MTU=2222
                 NETMASK=255.255.255.0
                 NETMASK1=255.255.255.0
                 NM_CONTROLLED=no
@@ -1669,6 +1678,8 @@ iface eth1 inet dhcp
 
 class TestSysConfigRendering(CiTestCase):
 
+    with_logs = True
+
     scripts_dir = '/etc/sysconfig/network-scripts'
     header = ('# Created by cloud-init on instance boot automatically, '
               'do not edit.\n#\n')
@@ -1917,6 +1928,9 @@ USERCTL=no
         found = self._render_and_read(network_config=yaml.load(entry['yaml']))
         self._compare_files_to_expected(entry['expected_sysconfig'], found)
         self._assert_headers(found)
+        self.assertNotIn(
+            'WARNING: Network config: ignoring eth0.101 device-level mtu',
+            self.logs.getvalue())
 
     def test_small_config(self):
         entry = NETWORK_CONFIGS['small']
@@ -1929,6 +1943,10 @@ USERCTL=no
         found = self._render_and_read(network_config=yaml.load(entry['yaml']))
         self._compare_files_to_expected(entry['expected_sysconfig'], found)
         self._assert_headers(found)
+        expected_msg = (
+            'WARNING: Network config: ignoring iface0 device-level mtu:8999'
+            ' because ipv4 subnet-level mtu:9000 provided.')
+        self.assertIn(expected_msg, self.logs.getvalue())
 
     def test_dhcpv6_only_config(self):
         entry = NETWORK_CONFIGS['dhcpv6_only']
@@ -2410,6 +2428,7 @@ class TestNetplanRoundTrip(CiTestCase):
 
 
 class TestEniRoundTrip(CiTestCase):
+
     def _render_and_read(self, network_config=None, state=None, eni_path=None,
                          netrules_path=None, dir=None):
         if dir is None: