Network loop between physical networks with DVR

This comment is no longer true
https://github.com/openstack/neutron/blob/5999716cfc4a00ac426e016eabbb51247ba0b190/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L1563-L1565

Because setup_rpc now runs after setup_physical_bridges
https://opendev.org/openstack/neutron/commit/d41bd58f31e259fe408c8c059b31299fdfe81127

The call to setup_dvr_flows() also installs the drop rule. But this is not getting to run because setup_rpc blocks on the has_alive_neutron_server rpc. Some stable branches don't have this rpc, and they block on get_dvr_mac_address instead.

https://github.com/openstack/neutron/blob/5999716cfc4a00ac426e016eabbb51247ba0b190/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py#L268-L286

Fix proposed to branch: master
Review: https://review.opendev.org/740506

# for BR in br-int br-physnet1 br-physnet2; do echo -e '\n' $BR; ovs-ofctl dump-flows $BR; done

 br-int
 cookie=0x45cd6ea336bf1713, duration=388.709s, table=0, n_packets=0, n_bytes=0, priority=65535,vlan_tci=0x0fff/0x1fff actions=drop
 cookie=0x45cd6ea336bf1713, duration=388.712s, table=0, n_packets=396, n_bytes=17144, priority=0 actions=resubmit(,60)
 cookie=0x45cd6ea336bf1713, duration=388.713s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x45cd6ea336bf1713, duration=388.710s, table=24, n_packets=0, n_bytes=0, priority=0 actions=drop
 cookie=0x45cd6ea336bf1713, duration=388.710s, table=60, n_packets=396, n_bytes=17144, priority=3 actions=NORMAL
 cookie=0x45cd6ea336bf1713, duration=388.708s, table=61, n_packets=0, n_bytes=0, priority=3 actions=NORMAL

 br-physnet1
 cookie=0x43cb683a62e439ee, duration=388.668s, table=0, n_packets=396, n_bytes=17144, priority=0 actions=NORMAL

 br-physnet2
 cookie=0xb9eab97d187e7034, duration=388.655s, table=0, n_packets=396, n_bytes=17144, priority=0 actions=NORMAL

Did I get the situation (with all the provided background) correct:

So, if setup rpc first, the rabbit gets muted and rpc cannot be checked before setting up bridges (no rules matching in secure mode - no flow).

And if setup bridges first, it blocks on rpc and no blocking rules got installed on bridges that causes ~1 mil/s packets arp storm.

Ultimately with the blocking rule (being removed in https://review.opendev.org/#/c/733568/), there is a dataplane downtime each time the neutron ovs agent restarted?

If so, the only solution I can think of (after reverting the latter patch) is make rpc check and the setup bridges to be exec'ed *real* parallel, and added a strict timeouts + a single retry for each of it?

Change abandoned by Darragh O'Reilly (<email address hidden>) on branch: master
Review: https://review.opendev.org/740506
Reason: Will create another patchset that fixes both issues.

Fix proposed to branch: master
Review: https://review.opendev.org/740724

Fix proposed to branch: master
Review: https://review.opendev.org/741444

Reviewed: https://review.opendev.org/740724
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8
Submitter: Zuul
Branch: master

commit c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8
Author: Darragh O'Reilly <email address hidden>
Date: Mon Jul 13 14:48:10 2020 +0000

    Ensure drop flows on br-int at agent startup for DVR too

    Commit 90212b12 changed the OVS agent so adding vital drop flows on
    br-int (table 0 priority 2) for packets from physical bridges was
    deferred until DVR initialization later on. But if br-int has no flows
    from a previous run (eg after host reboot), then these packets will hit
    the NORMAL flow in table 60. And if there is more than one physical
    bridge, then the physical interfaces from the different bridges are now
    essentially connected at layer 2 and a network loop is possible in the
    time before the flows are added by DVR. Also the DVR code won't add them
    until after RPC calls to the server, so a loop is more likely if the
    server is not available.

    This patch restores adding these flows to when the physical bridges are
    first configured. Also updated a comment that was no longer correct and
    updated the unit test.

    Change-Id: I42c33fefaae6a7bee134779c840f35632823472e
    Closes-Bug: #1887148
    Related-Bug: #1869808

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/742355

Fix proposed to branch: stable/train
Review: https://review.opendev.org/742359

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/742361

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/742364

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/742365

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/742366

Change abandoned by Rodolfo Alonso Hernandez (<email address hidden>) on branch: master
Review: https://review.opendev.org/741444
Reason: Superseded by https://review.opendev.org/#/c/740724/. Nice to see a better option.

Reviewed: https://review.opendev.org/742364
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=47ec363f5faefd85dfa33223c0087f4444afb5b9
Submitter: Zuul
Branch: stable/rocky

commit 47ec363f5faefd85dfa33223c0087f4444afb5b9
Author: Darragh O'Reilly <email address hidden>
Date: Mon Jul 13 14:48:10 2020 +0000

    Ensure drop flows on br-int at agent startup for DVR too

    Commit 90212b12 changed the OVS agent so adding vital drop flows on
    br-int (table 0 priority 2) for packets from physical bridges was
    deferred until DVR initialization later on. But if br-int has no flows
    from a previous run (eg after host reboot), then these packets will hit
    the NORMAL flow in table 60. And if there is more than one physical
    bridge, then the physical interfaces from the different bridges are now
    essentially connected at layer 2 and a network loop is possible in the
    time before the flows are added by DVR. Also the DVR code won't add them
    until after RPC calls to the server, so a loop is more likely if the
    server is not available.

    This patch restores adding these flows to when the physical bridges are
    first configured. Also updated a comment that was no longer correct and
    updated the unit test.

    Change-Id: I42c33fefaae6a7bee134779c840f35632823472e
    Closes-Bug: #1887148
    Related-Bug: #1869808
    (cherry picked from commit c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8)
    (cherry picked from commit 143fe8ff89ba776618ed6291af9d5e28e4662bdb)
    (cherry picked from commit 6a861b8c8c28e5675ec2208057298b811ba2b649)
    (cherry picked from commit 8181c5dbfe799ac6c832ab67b7eab3bcef4098b9)
    Conflicts:
            neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

Reviewed: https://review.opendev.org/742366
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=00466f41d690ca7c7a918bfd861878ef620bbec9
Submitter: Zuul
Branch: stable/pike

commit 00466f41d690ca7c7a918bfd861878ef620bbec9
Author: Darragh O'Reilly <email address hidden>
Date: Mon Jul 13 14:48:10 2020 +0000

    Ensure drop flows on br-int at agent startup for DVR too

    Commit 90212b12 changed the OVS agent so adding vital drop flows on
    br-int (table 0 priority 2) for packets from physical bridges was
    deferred until DVR initialization later on. But if br-int has no flows
    from a previous run (eg after host reboot), then these packets will hit
    the NORMAL flow in table 60. And if there is more than one physical
    bridge, then the physical interfaces from the different bridges are now
    essentially connected at layer 2 and a network loop is possible in the
    time before the flows are added by DVR. Also the DVR code won't add them
    until after RPC calls to the server, so a loop is more likely if the
    server is not available.

    This patch restores adding these flows to when the physical bridges are
    first configured. Also updated a comment that was no longer correct and
    updated the unit test.

    Change-Id: I42c33fefaae6a7bee134779c840f35632823472e
    Closes-Bug: #1887148
    Related-Bug: #1869808
    (cherry picked from commit c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8)
    (cherry picked from commit 143fe8ff89ba776618ed6291af9d5e28e4662bdb)
    (cherry picked from commit 6a861b8c8c28e5675ec2208057298b811ba2b649)
    (cherry picked from commit 8181c5dbfe799ac6c832ab67b7eab3bcef4098b9)
    Conflicts:
            neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
    (cherry picked from commit 47ec363f5faefd85dfa33223c0087f4444afb5b9)
    Conflicts:
            neutron/tests/unit/plugins/ml2/drivers/openvswitch/agent/test_ovs_neutron_agent.py
    (cherry picked from commit 8a173ec29ac1819c3d28c191814cd1402d272bb9)

Reviewed: https://review.opendev.org/742365
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8a173ec29ac1819c3d28c191814cd1402d272bb9
Submitter: Zuul
Branch: stable/queens

commit 8a173ec29ac1819c3d28c191814cd1402d272bb9
Author: Darragh O'Reilly <email address hidden>
Date: Mon Jul 13 14:48:10 2020 +0000

    Ensure drop flows on br-int at agent startup for DVR too

    Commit 90212b12 changed the OVS agent so adding vital drop flows on
    br-int (table 0 priority 2) for packets from physical bridges was
    deferred until DVR initialization later on. But if br-int has no flows
    from a previous run (eg after host reboot), then these packets will hit
    the NORMAL flow in table 60. And if there is more than one physical
    bridge, then the physical interfaces from the different bridges are now
    essentially connected at layer 2 and a network loop is possible in the
    time before the flows are added by DVR. Also the DVR code won't add them
    until after RPC calls to the server, so a loop is more likely if the
    server is not available.

    This patch restores adding these flows to when the physical bridges are
    first configured. Also updated a comment that was no longer correct and
    updated the unit test.

    Change-Id: I42c33fefaae6a7bee134779c840f35632823472e
    Closes-Bug: #1887148
    Related-Bug: #1869808
    (cherry picked from commit c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8)
    (cherry picked from commit 143fe8ff89ba776618ed6291af9d5e28e4662bdb)
    (cherry picked from commit 6a861b8c8c28e5675ec2208057298b811ba2b649)
    (cherry picked from commit 8181c5dbfe799ac6c832ab67b7eab3bcef4098b9)
    Conflicts:
            neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
    (cherry picked from commit 47ec363f5faefd85dfa33223c0087f4444afb5b9)
    Conflicts:
            neutron/tests/unit/plugins/ml2/drivers/openvswitch/agent/test_ovs_neutron_agent.py

Reviewed: https://review.opendev.org/742361
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8181c5dbfe799ac6c832ab67b7eab3bcef4098b9
Submitter: Zuul
Branch: stable/stein

commit 8181c5dbfe799ac6c832ab67b7eab3bcef4098b9
Author: Darragh O'Reilly <email address hidden>
Date: Mon Jul 13 14:48:10 2020 +0000

    Ensure drop flows on br-int at agent startup for DVR too

    Commit 90212b12 changed the OVS agent so adding vital drop flows on
    br-int (table 0 priority 2) for packets from physical bridges was
    deferred until DVR initialization later on. But if br-int has no flows
    from a previous run (eg after host reboot), then these packets will hit
    the NORMAL flow in table 60. And if there is more than one physical
    bridge, then the physical interfaces from the different bridges are now
    essentially connected at layer 2 and a network loop is possible in the
    time before the flows are added by DVR. Also the DVR code won't add them
    until after RPC calls to the server, so a loop is more likely if the
    server is not available.

    This patch restores adding these flows to when the physical bridges are
    first configured. Also updated a comment that was no longer correct and
    updated the unit test.

    Change-Id: I42c33fefaae6a7bee134779c840f35632823472e
    Closes-Bug: #1887148
    Related-Bug: #1869808
    (cherry picked from commit c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8)
    (cherry picked from commit 143fe8ff89ba776618ed6291af9d5e28e4662bdb)
    (cherry picked from commit 6a861b8c8c28e5675ec2208057298b811ba2b649)

Reviewed: https://review.opendev.org/742355
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=143fe8ff89ba776618ed6291af9d5e28e4662bdb
Submitter: Zuul
Branch: stable/ussuri

commit 143fe8ff89ba776618ed6291af9d5e28e4662bdb
Author: Darragh O'Reilly <email address hidden>
Date: Mon Jul 13 14:48:10 2020 +0000

    Ensure drop flows on br-int at agent startup for DVR too

    Commit 90212b12 changed the OVS agent so adding vital drop flows on
    br-int (table 0 priority 2) for packets from physical bridges was
    deferred until DVR initialization later on. But if br-int has no flows
    from a previous run (eg after host reboot), then these packets will hit
    the NORMAL flow in table 60. And if there is more than one physical
    bridge, then the physical interfaces from the different bridges are now
    essentially connected at layer 2 and a network loop is possible in the
    time before the flows are added by DVR. Also the DVR code won't add them
    until after RPC calls to the server, so a loop is more likely if the
    server is not available.

    This patch restores adding these flows to when the physical bridges are
    first configured. Also updated a comment that was no longer correct and
    updated the unit test.

    Change-Id: I42c33fefaae6a7bee134779c840f35632823472e
    Closes-Bug: #1887148
    Related-Bug: #1869808
    (cherry picked from commit c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8)

Reviewed: https://review.opendev.org/742359
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=17eded13595b18ab60af5256e0f63c57c3702296
Submitter: Zuul
Branch: stable/train

commit 17eded13595b18ab60af5256e0f63c57c3702296
Author: Darragh O'Reilly <email address hidden>
Date: Mon Jul 13 14:48:10 2020 +0000

    Ensure drop flows on br-int at agent startup for DVR too

    Commit 90212b12 changed the OVS agent so adding vital drop flows on
    br-int (table 0 priority 2) for packets from physical bridges was
    deferred until DVR initialization later on. But if br-int has no flows
    from a previous run (eg after host reboot), then these packets will hit
    the NORMAL flow in table 60. And if there is more than one physical
    bridge, then the physical interfaces from the different bridges are now
    essentially connected at layer 2 and a network loop is possible in the
    time before the flows are added by DVR. Also the DVR code won't add them
    until after RPC calls to the server, so a loop is more likely if the
    server is not available.

    This patch restores adding these flows to when the physical bridges are
    first configured. Also updated a comment that was no longer correct and
    updated the unit test.

    Change-Id: I42c33fefaae6a7bee134779c840f35632823472e
    Closes-Bug: #1887148
    Related-Bug: #1869808
    (cherry picked from commit c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8)
    (cherry picked from commit 143fe8ff89ba776618ed6291af9d5e28e4662bdb)

Hello Darragh, or anyone else affected,

Accepted neutron into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Darragh, or anyone else affected,

Accepted neutron into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:12.1.1-0ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Darragh, or anyone else affected,

Accepted neutron into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

All SRU verification completed and performed in https://bugs.launchpad.net/neutron/+bug/1869808 so please refer to that LP for the results.

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package neutron - 2:12.1.1-0ubuntu4

---------------
neutron (2:12.1.1-0ubuntu4) bionic; urgency=medium

  * Fix interrupt of VLAN traffic on reboot of neutron-ovs-agent:
  - d/p/0001-ovs-agent-signal-to-plugin-if-tunnel-refresh-needed.patch (LP: #1853613)
  - d/p/0002-Do-not-block-connection-between-br-int-and-br-phys-o.patch (LP: #1869808)
  - d/p/0003-Ensure-that-stale-flows-are-cleaned-from-phys_bridge.patch (LP: #1864822)
  - d/p/0004-DVR-Reconfigure-re-created-physical-bridges-for-dvr-.patch (LP: #1864822)
  - d/p/0005-Ensure-drop-flows-on-br-int-at-agent-startup-for-DVR.patch (LP: #1887148)
  - d/p/0006-Don-t-check-if-any-bridges-were-recrected-when-OVS-w.patch (LP: #1864822)
  - d/p/0007-Not-remove-the-running-router-when-MQ-is-unreachable.patch (LP: #1871850)

 -- Edward Hope-Morley <email address hidden> Mon, 22 Feb 2021 16:55:40 +0000

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package neutron - 2:12.1.1-0ubuntu4~cloud0
---------------

 neutron (2:12.1.1-0ubuntu4~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:12.1.1-0ubuntu4) bionic; urgency=medium
 .
   * Fix interrupt of VLAN traffic on reboot of neutron-ovs-agent:
   - d/p/0001-ovs-agent-signal-to-plugin-if-tunnel-refresh-needed.patch (LP: #1853613)
   - d/p/0002-Do-not-block-connection-between-br-int-and-br-phys-o.patch (LP: #1869808)
   - d/p/0003-Ensure-that-stale-flows-are-cleaned-from-phys_bridge.patch (LP: #1864822)
   - d/p/0004-DVR-Reconfigure-re-created-physical-bridges-for-dvr-.patch (LP: #1864822)
   - d/p/0005-Ensure-drop-flows-on-br-int-at-agent-startup-for-DVR.patch (LP: #1887148)
   - d/p/0006-Don-t-check-if-any-bridges-were-recrected-when-OVS-w.patch (LP: #1864822)
   - d/p/0007-Not-remove-the-running-router-when-MQ-is-unreachable.patch (LP: #1871850)

This issue was fixed in the openstack/neutron pike-eol release.

This issue was fixed in the openstack/neutron queens-eol release.

This issue was fixed in the openstack/neutron rocky-eol release.