Checksum drop of metadata traffic on isolated networks with DPDK

David - please see the link that was the reason I reverted this change, https://lore.kernel.org/patchwork/patch/824819/ - that is basically saying this rule has no effect for TCP, it was only meant for UDP, and was finally changed to log a warning in the kernel.

There is probably something else going on here causing issues, possibly outside of neutron.

Brian,

Thanks for getting back to me. It seems this is a duplicate of LP Bug #1722584 [0]. And the explanation for my running into it is that we have not yet pushed your reversion into our Ubuntu packaging.

Marking this bug a duplicate of LP Bug #1722584

[0] https://bugs.launchpad.net/cloud-archive/+bug/1722584

We de-duplicated this bug as we have narrowed the focus. This is DPDK specific.

When using isolated provider networks AND DPDK metadata is dropped due to incorrect TCP checksum. Specifically when the provider interface is a DPDK interface.

It is temporarily mitigated by adding the iptables rule:
iptables -t mangle -A OUTPUT -o ns-+ -p tcp --sport 80 -j CHECKSUM --checksum-fill
However this is not sustainable as any restart of openvswitch will clear this setting.

Here is an example of a tcpdump in the qdhcp netns:

    172.20.0.23.50060 > 169.254.169.254.80: Flags [S], cksum 0xb3c1 (correct), seq 3293184694, win 29200, options [mss 1460,sackOK,TS val 78090881 ecr 0,nop,wscale 7], length 0
23:36:43.932728 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)

    169.254.169.254.80 > 172.20.0.23.50060: Flags [S.], cksum 0x0057 (incorrect -> 0x2b02), seq 2915867972, ack 3293184695, win 28960, options [mss 1460,sackOK,TS val 4009971593 ecr 78090881,nop,wscale 7], length 0

Continuing with re-transmissions of the same. Note the incorrect cksum.

With the mangle rule in place:

    172.20.0.4.46706 > 169.254.169.254.80: Flags [S], cksum 0xbf25 (correct), seq 4115688639, win 29200, options [mss 1460,sackOK,TS val 2390126443 ecr 0,nop,wscale
7], length 0
23:40:01.510745 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    169.254.169.254.80 > 172.20.0.4.46706: Flags [S.], cksum 0xe3f3 (correct), seq 1829822633, ack 4115688640, win 28960, options [mss 1460,sackOK,TS val 812998113 e
cr 2390126443,nop,wscale 7], length 0
23:40:01.510919 IP (tos 0x0, ttl 64, id 38572, offset 0, flags [DF], proto TCP (6), length 52)
    172.20.0.4.46706 > 169.254.169.254.80: Flags [.], cksum 0x82fb (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 2390126443 ecr 812998113], length 0
23:40:01.510974 IP (tos 0x0, ttl 64, id 38573, offset 0, flags [DF], proto TCP (6), length 229)
    172.20.0.4.46706 > 169.254.169.254.80: Flags [P.], cksum 0x5e40 (correct), seq 1:178, ack 1, win 229, options [nop,nop,TS val 2390126443 ecr 812998113], length 1
77: HTTP, length: 177
        GET /openstack HTTP/1.1
        Host: 169.254.169.254
        User-Agent: Cloud-Init/19.1-1-gbaa47854-0ubuntu1~18.04.1
        Accept-Encoding: gzip, deflate
        Accept: */*
        Connection: keep-alive

23:40:01.553043 IP (tos 0x0, ttl 64, id 62471, offset 0, flags [DF], proto TCP (6), length 52)
    169.254.169.254.80 > 172.20.0.4.46706: Flags [.], cksum 0x8219 (correct), seq 1, ack 178, win 235, options [nop,nop,TS val 812998156 ecr 2390126443], length 0
23:40:02.036984 IP (tos 0x0, ttl 64, id 62472, offset 0, flags [DF], proto TCP (6), length 252)
    169.254.169.254.80 > 172.20.0.4.46706: Flags [P.], cksum 0xd303 (correct), seq 1:201, ack 178, win 235, options [nop,nop,TS val 812998640 ecr 2390126443], length
 200: HTTP, length: 200
        HTTP/1.1 200 OK
        Content-Type: text/plain; charset=UTF-8
        Content-Length: 83
        Date: Thu, 13 Jun 2019 23:40:02 GMT

        2012-08-10
        2013-04-04
        2013-10-17
        2015-10-15
        2016-06-30
        2016-10-06
        2017-02-22
        latest[!http]

We can provi...

Further testing shows the provider network is irrelevant. With DPDK and an isolated network (qdhcp only no qrouter) either GRE or provider, any traffic initiated by the qdhcp netns, including response traffic, gets an incorrect TCP checksum.

This packet gets put on the "wire" and it is the VM that drops the packet due to an invalid TCP checksum.

In a DPDK isolated network environment from the qdhcp netns you can see this in action with an arbitrary netcat call:

nc -vz $VM_IP 73 (Any TCP port)

tcpdump on the VM side and you can see

22:06:31.424716 IP (tos 0x0, ttl 64, id 14532, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.0.2.39784 > 172.20.0.6.73: Flags [S], cksum 0x585f (incorrect -> 0x0437), seq 114680395, win 26880, options [mss 8960,sackOK,TS val 1660321155 ecr 0,nop,wscale 7], length 0
22:06:39.616633 IP (tos 0x0, ttl 64, id 14533, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.0.2.39784 > 172.20.0.6.73: Flags [S], cksum 0x585f (incorrect -> 0xe436), seq 114680395, win 26880, options [mss 8960,sackOK,TS val 1660329347 ecr 0,nop,wscale 7], length 0
22:06:55.744502 IP (tos 0x0, ttl 64, id 14534, offset 0, flags [DF], proto TCP (6), length 60)
    172.20.0.2.39784 > 172.20.0.6.73: Flags [S], cksum 0x585f (incorrect -> 0xa536), seq 114680395, win 26880, options [mss 8960,sackOK,TS val 1660345475 ecr 0,nop,wscale 7], length 0

So the VM sees the response traffic from the qdhcp netns but drops it because the TCP checksum is invalid.

When we turn on DVR and create a virtual router (unused) the qrouter netns does not have this problem. I have not root caused why but there are a number of other iptables settings in the qrouter netns that are not in the qdhcp that may be required.

So what we are looking for is differences in the setup of the qdhcp netns from the qrouter netns.

Also, note the qrouter nets works with or without the neutron-l3-agent-POSTROUTING checksum fill rule. This is important as this checksum fill was removed from the neutron code.

A bit more research. I tried to find differences in iptables rules between qdhcp and qrouter netns-es.

I went so far as to restore iptables-save of qrouter in the qdhcp with no change.

Any TCP connection (or response) in the qdhcp netns generates an invalid TCP checksum.

The issue appears to be that ovs_use_veth=True was set to true in dhcp_agent.ini. This causes the qdhcp namespace to be connected to the bridge via a veth pair. This appears to leave checksum offloading enabled on the device in the qdhcp namespace. Manually turning off tx-checksum-ip-generic appears to fix the issue but for a permanent fix switch ovs_use_veth to False.

Liam - thanks for the information. Since the default value for ovs_use_veth is False in the neutron tree, was it the installer tools that changed the value?

Hi Brian, yes it was, I'll be proposing a change to fix that today.

Maybe we should add some warning about it to our docs too?

Related fix proposed to branch: master
Review: https://review.opendev.org/666277

Liam, can you clarify what change you're proposing and where?

I'm assuming setting ovs_use_veth=false in dhcp_agent.ini by default?

(and not disabling checksumming somehow).

Hi niveditasinghvi. Yes, I'm proposing ovs_use_veth be set to False rather than manually disabling checksumming. The changes are here: https://review.opendev.org/#/q/topic:bug/1832021+(status:open+OR+status:merged). We are having a few issue with CI but I hope to have them landed asap.

Reviewed: https://review.opendev.org/666277
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=051b58f566dd382f22df937d2e90c06e9ca2faad
Submitter: Zuul
Branch: master

commit 051b58f566dd382f22df937d2e90c06e9ca2faad
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jun 19 14:27:18 2019 +0200

    Update DPDK docs with note about using veth pairs

    In case when ovs-dpdk is used together with ``ovs_use_veth`` config
    option set to True, it cause invalid checksum on all packets send from
    qdhcp namespace.
    This commit adds short info about this limitation to ovs-dpdk config
    guide.

    Change-Id: I6237abab3d9e625440e95e75f5091d09a1ec44f0
    Related-Bug: #1832021

Reviewed: https://review.opendev.org/666293
Committed: https://git.openstack.org/cgit/openstack/charm-neutron-openvswitch/commit/?id=7578326c592a81e6cc6737f360fc2b15ed175cf0
Submitter: Zuul
Branch: master

commit 7578326c592a81e6cc6737f360fc2b15ed175cf0
Author: Liam Young <email address hidden>
Date: Wed Jun 19 13:29:18 2019 +0000

    Stop using veth pairs to connect qdhcp ns

    veth pairs are currently being used to connect the qdhcp namespace
    to the underlying bridge. This behaviour appears to only be needed
    for old kernels with limited namespaces support (pre trusty).

    Change-Id: If1f669de09e2499e74e88e2b72203047e7f9f957
    Closes-Bug: #1832021

Just a late note: the upgrade was applied to the customer in question and it indeed fixed the problem.

charm neutron-openvswitch-next-359 -> neutron-openvswitch-next-367

Reviewed: https://review.opendev.org/713719
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=11838a2bc50caa40e776bce211f5f2d1d16a14af
Submitter: Zuul
Branch: master

commit 11838a2bc50caa40e776bce211f5f2d1d16a14af
Author: Alexander Vlasov <email address hidden>
Date: Wed Mar 18 13:35:20 2020 -0500

    Workaround for TCP checksum issue with ovs-dpdk and veth pair

    The need for this change stems from following issues:
    1) When ovs_use_veth = False with ovs-dpdk issue with ovs
    was observed - after vswitch restart interface is not comming up.
    Meaning ovs-dpdk uses ovs internal ports and it is not able to bring
    them up on restart.
    2) When ovs_use_veth = True and ovs-dpkd is used, packets sent with
    incorrect checksum due to the fact that ovs-dpdk does not do checksum
    calculations for veth interface.

    This commit allows to use second option and resolve checksum issue by
    disabling checksum offload.

    Closes-Bug: #1832021
    Related-Bug: #1831935

    Change-Id: Iecce8d2c6c2c46718cc1020c6e8f914cd4560e4b

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/738483

Fix proposed to branch: stable/train
Review: https://review.opendev.org/738485

Reviewed: https://review.opendev.org/738485
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=85f95e0bd9a80b952c6591119372d24b79ac24d2
Submitter: Zuul
Branch: stable/train

commit 85f95e0bd9a80b952c6591119372d24b79ac24d2
Author: Alexander Vlasov <email address hidden>
Date: Wed Mar 18 13:35:20 2020 -0500

    Workaround for TCP checksum issue with ovs-dpdk and veth pair

    The need for this change stems from following issues:
    1) When ovs_use_veth = False with ovs-dpdk issue with ovs
    was observed - after vswitch restart interface is not comming up.
    Meaning ovs-dpdk uses ovs internal ports and it is not able to bring
    them up on restart.
    2) When ovs_use_veth = True and ovs-dpkd is used, packets sent with
    incorrect checksum due to the fact that ovs-dpdk does not do checksum
    calculations for veth interface.

    This commit allows to use second option and resolve checksum issue by
    disabling checksum offload.

    Closes-Bug: #1832021
    Related-Bug: #1831935

    Change-Id: Iecce8d2c6c2c46718cc1020c6e8f914cd4560e4b
    (cherry picked from commit 11838a2bc50caa40e776bce211f5f2d1d16a14af)

Reviewed: https://review.opendev.org/738483
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e79afe115cb4db588fdd6bbd7500775173fd88c5
Submitter: Zuul
Branch: stable/ussuri

commit e79afe115cb4db588fdd6bbd7500775173fd88c5
Author: Alexander Vlasov <email address hidden>
Date: Wed Mar 18 13:35:20 2020 -0500

    Workaround for TCP checksum issue with ovs-dpdk and veth pair

    The need for this change stems from following issues:
    1) When ovs_use_veth = False with ovs-dpdk issue with ovs
    was observed - after vswitch restart interface is not comming up.
    Meaning ovs-dpdk uses ovs internal ports and it is not able to bring
    them up on restart.
    2) When ovs_use_veth = True and ovs-dpkd is used, packets sent with
    incorrect checksum due to the fact that ovs-dpdk does not do checksum
    calculations for veth interface.

    This commit allows to use second option and resolve checksum issue by
    disabling checksum offload.

    Closes-Bug: #1832021
    Related-Bug: #1831935

    Change-Id: Iecce8d2c6c2c46718cc1020c6e8f914cd4560e4b
    (cherry picked from commit 11838a2bc50caa40e776bce211f5f2d1d16a14af)

Reviewed: https://review.opendev.org/713717
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=81dfa48466a594da78d93494e805b9ba97f94bc6
Submitter: Zuul
Branch: stable/stein

commit 81dfa48466a594da78d93494e805b9ba97f94bc6
Author: Alexander Vlasov <email address hidden>
Date: Wed Mar 18 13:35:20 2020 -0500

    Workaround for TCP checksum issue with ovs-dpdk and veth pair

    The need for this change stems from following issues:
    1) When ovs_use_veth = False with ovs-dpdk issue with ovs
    was observed - after vswitch restart interface is not comming up.
    Meaning ovs-dpdk uses ovs internal ports and it is not able to bring
    them up on restart.
    2) When ovs_use_veth = True and ovs-dpkd is used, packets sent with
    incorrect checksum due to the fact that ovs-dpdk does not do checksum
    calculations for veth interface.

    This commit allows to use second option and resolve checksum issue by
    disabling checksum offload.

    Closes-Bug: #1832021
    Related-Bug: #1831935

    Change-Id: Iecce8d2c6c2c46718cc1020c6e8f914cd4560e4b
    (cherry picked from commit 11838a2bc50caa40e776bce211f5f2d1d16a14af)

Hi, I have same issue on OVN, is there any bug fix for this on Openstack Ussuri

Attached debdiff ready for sru cloud review.

I think I don't have the permissions to properly set the revisions if someone can set it, please.
This should be added to:
  - UCA: queens
  - Distro: Bionic (queens)

The attachment "neutron-dpdk-fix.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Fixed in Ubuntu Focal.

---
neutron (2:16.1.0-0ubuntu1) focal; urgency=medium

  * New stable point release for OpenStack Ussuri (LP: #1892139).
  * d/control: Align (Build-)Depends with upstream.

 -- Chris MacNaughton <email address hidden> Thu, 27 Aug 2020 05:31:05 +0000

Fixed in neutron 17.0.0 and later.

Attached debdiff for Bionic Rocky

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-neutron-openvswitch/+/790354

Thanks Erlon, I've uploaded to rocky-staging and the bionic unapproved queue. The SRU team will need to review the bionic version, so I have subscribed them.

Hello David, or anyone else affected,

Accepted neutron into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:12.1.1-0ubuntu8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello David, or anyone else affected,

Accepted neutron into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello David, or anyone else affected,

Accepted neutron into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package neutron - 2:12.1.1-0ubuntu8

---------------
neutron (2:12.1.1-0ubuntu8) bionic; urgency=medium

  * Backport fix for TCP checksum issue (LP: #1832021)
    - d/p/0001-Workaround-for-TCP-checksum-issue-with-ovs-dpdk-and-.patch

 -- erlon <email address hidden> Mon, 26 Apr 2021 14:01:49 -0300

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package neutron - 2:13.0.7-0ubuntu1~cloud6
---------------

 neutron (2:13.0.7-0ubuntu1~cloud6) bionic-rocky; urgency=medium
 .
   * Backport fix for TCP checksum issue (LP: #1832021)
     - d/p/0001-Workaround-for-TCP-checksum-issue-with-ovs-dpdk-and-.patch

I tried to test this on xenial-queens, but the charm does not work and seems to be broken. See that the queens backport fails only on xenial. On bionic-queens, it worked and passed the tests. So, we should not add this package to the xenial-queens cloud archive.

@erlon, I can revert this, however if the bug in the charm can be hit with or without this SRU code, I think we should open a bug against the charm. Would you be able to look into that?

Reverting for Queens cloud archive only.

Sure, makes sense. I opened this: https://bugs.launchpad.net/charm-neutron-openvswitch/+bug/1935077

Reviewed: https://review.opendev.org/c/openstack/charm-neutron-openvswitch/+/790354
Committed: https://opendev.org/openstack/charm-neutron-openvswitch/commit/1f6e29fbe2c6cabbdb082f5d8899520809f9b7ae
Submitter: "Zuul (22348)"
Branch: master

commit 1f6e29fbe2c6cabbdb082f5d8899520809f9b7ae
Author: Erlon R. Cruz <email address hidden>
Date: Sun May 9 10:38:02 2021 -0300

    Adds Neutron OVS agent config to DHCP daemon

    The neutron-dhcp-agent might rely on options defined in
    openvswitch_agent.ini. By default this config file is not passed to
    neutron-dhcp-agent daemon, and therefore those options are not
    loaded and fall back for the default values and by that inhibits
    the desired behavior.

    Charm helpers' PR: https://github.com/juju/charm-helpers/pull/615

    Depends-on: I39024855c3e42ee135b6ad5e7618a770219b6994

    Closes-bug: #1832021
    Change-Id: I134c8077ee52ccdb4e383109ecbea27ed1633fb8

This issue was fixed in the openstack/neutron queens-eol release.

This issue was fixed in the openstack/neutron rocky-eol release.