gre_sys set to default 1472 when using path_mtu > 1500 with ovs 2.8.x

Pinged upstream dev mailing list for openvswitch for feedback

Eric Garver:

"Most likely you are seeing this bug:

  https://bugzilla.redhat.com/show_bug.cgi?id=1488484

It's actually an issue with the kernel GRE driver ignoring IFLA_MTU when
the device is created."

Creating test package with proposed fix:

https://patchwork.ozlabs.org/patch/860192/

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1742505

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Mirroring some of the thread on the ovs dev ML:

From Christina

"Thanks Eric, that matches my findings, glad that there seems to be an
accepted fix already.
But it is fairly recent and only in since 4.15-rc8 levels afaik.

But OTOH its description at [1] reads pretty much like my notes so far.

@James - do you think you could test a super-recent mainline kernel
build from [2] in regard to this issue?

[1]: https://github.com/torvalds/linux/commit/cfddd4c33c254954927942599d299b3865743146
[2]: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc8/"

This kernel fix resolves a secondary issue I saw with the 4.13 kernel where the workaround we used for 4.4 did not work - the kernel was applying a 1500 max limit to all GRE devices.

This was fixed in [1] which I've confirmed on my test but the original OVS bug still exists - gre_sys is still configured with MTU 1472, implying the IFLA_MTU value provided via netlink is being ignored by the kernel.

Marking kernel tasks as confirmed as we have at least one related bug which we have confirmed is fixed under 4.15 (and looks like an easy pick for 4.13).

ovs-dev ML thread:

https://mail.openvswitch.org/pipermail/ovs-dev/2018-January/343069.html

Simple reproducer:

sudo add-apt-repository cloud-archive:pike
sudo apt update
sudo apt install openvswitch-switch
sudo ovs-vsctl add-br br-tun
sudo ovs-vsctl add-port br-tun gre0 -- set interface gre0 type=gre options:remote_ip=10.100.1.1

this will cause ovs to create the gre_sys device in the kernel; the remote_ip can be anything but is obviously not functional.

Reference bug 1743746 for MTU hardware limitation in 4.13 kernel

I've re-tested with the proposed changes upstream; MTU is correctly set to 65000 for all tunnel types including GRE on 4.4 kernels; the kernel fix for bug 1743746 is required for support on >= Artful or if the HWE edge kernel is in use on Xenial.

Test packages for Artful and Xenial in:

  ppa:james-page/pike

Leaving the kernel bug tasks open for now; there is an underlying issue in the kernel specific to GRE tunnel ports where the MTU passed during new device creation is not being used; the fixes to OVS workaround this issue rather than actually resolving it.

I've uploaded the fixes for this issue to bionic (currently in proposed awaiting builds across all archs) and for artful (stacked ontop of the current 2.8.1 stable release in proposed).

I'd like todo this fix ontop of the 2.8.1 release, rather than have end-users deal with two sets of updates which are disruptive to the data plane in an openstack cloud.

@james-page When will the 2.8.1 release be?

Hello David, or anyone else affected,

Accepted openvswitch into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvswitch/2.8.1-0ubuntu0.17.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello David, or anyone else affected,

Accepted openvswitch into pike-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:pike-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-pike-needed to verification-pike-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-pike-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Tested OK on Xenial with Pike UCA:

16: gre_sys@NONE: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc pfifo_fast master ovs-system state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 56:5c:6e:19:f5:4b brd ff:ff:ff:ff:ff:ff

However testing on artful currently blocked pending resolution of bug 1743746 which prevents an increase in the MTU of the gre devices.

This bug was fixed in the package openvswitch - 2.8.1-0ubuntu3

---------------
openvswitch (2.8.1-0ubuntu3) bionic; urgency=medium

  * Updates to systemd configuration:
    - Move to distinct units for ovsdb-server and ovs-vswitchd.
  * Drop obsolete upstart configuration file.
  * Bump nofiles to 1048576 for ovs daemons (LP: #1737866).
  * d/control: Bump minimum debhelper version to 10, drop BD on
    dh-systemd.
  * d/p/dpif-kernel-gre-mtu-workaround.patch,
    d/p/dpif-netlink-rtnl-Use-65000-instead-of-65535-as-tunnel-MTU.patch:
    Cherry pick in-flight fixes for workaround to correctly set MTU
    of GRE devices via netlink (LP: #1742505).

 -- James Page <email address hidden> Thu, 18 Jan 2018 15:26:41 +0200

Hello David, or anyone else affected,

Accepted openvswitch into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

successfully tested with bug 1743746

This bug was fixed in the package openvswitch - 2.8.1-0ubuntu0.17.10.2

---------------
openvswitch (2.8.1-0ubuntu0.17.10.2) artful; urgency=medium

  * d/p/dpif-kernel-gre-mtu-workaround.patch,
    d/p/dpif-netlink-rtnl-Use-65000-instead-of-65535-as-tunnel-MTU.patch:
    Cherry pick in-flight fixes for workaround to correctly set MTU
    of GRE devices via netlink (LP: #1742505).

openvswitch (2.8.1-0ubuntu0.17.10.1) artful; urgency=medium

  * New upstream stable release (LP: #1724622).

 -- James Page <email address hidden> Sat, 20 Jan 2018 10:22:31 +0000

The verification of the Stable Release Update for openvswitch has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

The verification of the Stable Release Update for openvswitch has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package openvswitch - 2.8.1-0ubuntu0.17.10.2~cloud0
---------------

 openvswitch (2.8.1-0ubuntu0.17.10.2~cloud0) xenial-pike; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 openvswitch (2.8.1-0ubuntu0.17.10.2) artful; urgency=medium
 .
   * d/p/dpif-kernel-gre-mtu-workaround.patch,
     d/p/dpif-netlink-rtnl-Use-65000-instead-of-65535-as-tunnel-MTU.patch:
     Cherry pick in-flight fixes for workaround to correctly set MTU
     of GRE devices via netlink (LP: #1742505).