reboot neutron-ovs-agent introduces a short interrupt of vlan traffic

Hi Norman - are you able to reproduce this on a later version of neutron like 15.0.x?

Since this is just causing a small impact and the restart of the ovs-agent isn't a frequent event, I'm going to set the priority to Low.

After restart of neutron-ovs-agent, it should installs all OF rules with new cookie_id and then delete old rules so duration in all of them should be reset.
Is your problem only on vlan network? Or it happens on other network types too?

Hi,
Please take a look at this fix: https://review.opendev.org/#/c/681462/. If your deployment does no have this fix, try to backport it and see if it helps.

Thank you very much for the reply. I've checked the code and we do indeed have this patch applied already.

but it seems to a different problem, because the short interrupt still exists.

I would like to remove drop_port calls inside setup_physical_bridges because it seems particularly overkill,
for example, such flows will be quickly overriden by dvr flows after several steps.

I would be appreciated to have some backgrounds on theses two calls and understand why they are put there. From our point of view, control plane failure should do little or no harm to user plane data flows. But blocking connections will have a huge backfire and making all vlans useless.

Besides it will also interrupt data flows during startup depending on how fast rpc call is made. So I'd really like to remove these two calls if there are no other security issues.

Proposed patch https://review.opendev.org/#/c/733568/

Reviewed: https://review.opendev.org/733568
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=90212b12cdf62e92d811997ebba699cab431d696
Submitter: Zuul
Branch: master

commit 90212b12cdf62e92d811997ebba699cab431d696
Author: shenjiatong <email address hidden>
Date: Thu Jun 18 15:33:13 2020 +0800

    Do not block connection between br-int and br-phys on startup

    Block traffic between br-int and br-physical is over kill
    and will at least

    1. interrupt vlan flow during startup, and is particularly
    so if dvr enabled
    2. if let's rabbitmq is not stable, it is possible data plane
    will be affected and vlan will never work.

    Using openstack on k8s particularly amplifies the problem
    because pod could be killed pretty easily by liveness
    probes.

    Change-Id: I51050c600ba7090fea71213687d94340bac0674a
    Closes-Bug: #1869808

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/738503

Fix proposed to branch: stable/train
Review: https://review.opendev.org/738504

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/738505

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/738511

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/738512

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/738523

Reviewed: https://review.opendev.org/738503
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=88e70a520acaca37db645c3ef1124df8c7d778d5
Submitter: Zuul
Branch: stable/ussuri

commit 88e70a520acaca37db645c3ef1124df8c7d778d5
Author: shenjiatong <email address hidden>
Date: Thu Jun 18 15:33:13 2020 +0800

    Do not block connection between br-int and br-phys on startup

    Block traffic between br-int and br-physical is over kill
    and will at least

    1. interrupt vlan flow during startup, and is particularly
    so if dvr enabled
    2. if let's rabbitmq is not stable, it is possible data plane
    will be affected and vlan will never work.

    Using openstack on k8s particularly amplifies the problem
    because pod could be killed pretty easily by liveness
    probes.

    Change-Id: I51050c600ba7090fea71213687d94340bac0674a
    Closes-Bug: #1869808
    (cherry picked from commit 90212b12cdf62e92d811997ebba699cab431d696)

Reviewed: https://review.opendev.org/738504
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4f501f405d1c44e00784df8450cbe83129da1ea7
Submitter: Zuul
Branch: stable/train

commit 4f501f405d1c44e00784df8450cbe83129da1ea7
Author: shenjiatong <email address hidden>
Date: Thu Jun 18 15:33:13 2020 +0800

    Do not block connection between br-int and br-phys on startup

    Block traffic between br-int and br-physical is over kill
    and will at least

    1. interrupt vlan flow during startup, and is particularly
    so if dvr enabled
    2. if let's rabbitmq is not stable, it is possible data plane
    will be affected and vlan will never work.

    Using openstack on k8s particularly amplifies the problem
    because pod could be killed pretty easily by liveness
    probes.

    Change-Id: I51050c600ba7090fea71213687d94340bac0674a
    Closes-Bug: #1869808
    (cherry picked from commit 90212b12cdf62e92d811997ebba699cab431d696)

Reviewed: https://review.opendev.org/738505
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6dfc35680fcc885d9ad449ca2b39225fb1bca898
Submitter: Zuul
Branch: stable/stein

commit 6dfc35680fcc885d9ad449ca2b39225fb1bca898
Author: shenjiatong <email address hidden>
Date: Thu Jun 18 15:33:13 2020 +0800

    Do not block connection between br-int and br-phys on startup

    Block traffic between br-int and br-physical is over kill
    and will at least

    1. interrupt vlan flow during startup, and is particularly
    so if dvr enabled
    2. if let's rabbitmq is not stable, it is possible data plane
    will be affected and vlan will never work.

    Using openstack on k8s particularly amplifies the problem
    because pod could be killed pretty easily by liveness
    probes.

    Change-Id: I51050c600ba7090fea71213687d94340bac0674a
    Closes-Bug: #1869808
    (cherry picked from commit 90212b12cdf62e92d811997ebba699cab431d696)

Reviewed: https://review.opendev.org/738512
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=131bbc9a53411033cf27664d8f1fd7afc72c57bf
Submitter: Zuul
Branch: stable/queens

commit 131bbc9a53411033cf27664d8f1fd7afc72c57bf
Author: shenjiatong <email address hidden>
Date: Thu Jun 18 15:33:13 2020 +0800

    Do not block connection between br-int and br-phys on startup

    Block traffic between br-int and br-physical is over kill
    and will at least

    1. interrupt vlan flow during startup, and is particularly
    so if dvr enabled
    2. if let's rabbitmq is not stable, it is possible data plane
    will be affected and vlan will never work.

    Using openstack on k8s particularly amplifies the problem
    because pod could be killed pretty easily by liveness
    probes.

    Conflicts:
        neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Change-Id: I51050c600ba7090fea71213687d94340bac0674a
    Closes-Bug: #1869808
    (cherry picked from commit 90212b12cdf62e92d811997ebba699cab431d696)

Reviewed: https://review.opendev.org/738511
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cc48edf85cf66277423b0eb52ae6353f8028d2a6
Submitter: Zuul
Branch: stable/rocky

commit cc48edf85cf66277423b0eb52ae6353f8028d2a6
Author: shenjiatong <email address hidden>
Date: Thu Jun 18 15:33:13 2020 +0800

    Do not block connection between br-int and br-phys on startup

    Block traffic between br-int and br-physical is over kill
    and will at least

    1. interrupt vlan flow during startup, and is particularly
    so if dvr enabled
    2. if let's rabbitmq is not stable, it is possible data plane
    will be affected and vlan will never work.

    Using openstack on k8s particularly amplifies the problem
    because pod could be killed pretty easily by liveness
    probes.

    Conflicts:
        neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Change-Id: I51050c600ba7090fea71213687d94340bac0674a
    Closes-Bug: #1869808
    (cherry picked from commit 90212b12cdf62e92d811997ebba699cab431d696)

Related fix proposed to branch: master
Review: https://review.opendev.org/740506

Change abandoned by Darragh O'Reilly (<email address hidden>) on branch: master
Review: https://review.opendev.org/740506
Reason: Will create another patchset that fixes both issues.

Related fix proposed to branch: master
Review: https://review.opendev.org/740724

Related fix proposed to branch: master
Review: https://review.opendev.org/741444

Reviewed: https://review.opendev.org/738523
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1f4f888ad34d54ec968d9c9f9f80c388f3ca0d12
Submitter: Zuul
Branch: stable/pike

commit 1f4f888ad34d54ec968d9c9f9f80c388f3ca0d12
Author: shenjiatong <email address hidden>
Date: Thu Jun 18 15:33:13 2020 +0800

    Do not block connection between br-int and br-phys on startup

    Block traffic between br-int and br-physical is over kill
    and will at least

    1. interrupt vlan flow during startup, and is particularly
    so if dvr enabled
    2. if let's rabbitmq is not stable, it is possible data plane
    will be affected and vlan will never work.

    Using openstack on k8s particularly amplifies the problem
    because pod could be killed pretty easily by liveness
    probes.

    Conflicts:
        neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

    Change-Id: I51050c600ba7090fea71213687d94340bac0674a
    Closes-Bug: #1869808
    (cherry picked from commit 90212b12cdf62e92d811997ebba699cab431d696)

Reviewed: https://review.opendev.org/740724
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8
Submitter: Zuul
Branch: master

commit c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8
Author: Darragh O'Reilly <email address hidden>
Date: Mon Jul 13 14:48:10 2020 +0000

    Ensure drop flows on br-int at agent startup for DVR too

    Commit 90212b12 changed the OVS agent so adding vital drop flows on
    br-int (table 0 priority 2) for packets from physical bridges was
    deferred until DVR initialization later on. But if br-int has no flows
    from a previous run (eg after host reboot), then these packets will hit
    the NORMAL flow in table 60. And if there is more than one physical
    bridge, then the physical interfaces from the different bridges are now
    essentially connected at layer 2 and a network loop is possible in the
    time before the flows are added by DVR. Also the DVR code won't add them
    until after RPC calls to the server, so a loop is more likely if the
    server is not available.

    This patch restores adding these flows to when the physical bridges are
    first configured. Also updated a comment that was no longer correct and
    updated the unit test.

    Change-Id: I42c33fefaae6a7bee134779c840f35632823472e
    Closes-Bug: #1887148
    Related-Bug: #1869808

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/742355

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/742359

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/742361

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/742364

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/742365

Related fix proposed to branch: stable/pike
Review: https://review.opendev.org/742366

Change abandoned by Rodolfo Alonso Hernandez (<email address hidden>) on branch: master
Review: https://review.opendev.org/741444
Reason: Superseded by https://review.opendev.org/#/c/740724/. Nice to see a better option.

Reviewed: https://review.opendev.org/742364
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=47ec363f5faefd85dfa33223c0087f4444afb5b9
Submitter: Zuul
Branch: stable/rocky

commit 47ec363f5faefd85dfa33223c0087f4444afb5b9
Author: Darragh O'Reilly <email address hidden>
Date: Mon Jul 13 14:48:10 2020 +0000

    Ensure drop flows on br-int at agent startup for DVR too

    Commit 90212b12 changed the OVS agent so adding vital drop flows on
    br-int (table 0 priority 2) for packets from physical bridges was
    deferred until DVR initialization later on. But if br-int has no flows
    from a previous run (eg after host reboot), then these packets will hit
    the NORMAL flow in table 60. And if there is more than one physical
    bridge, then the physical interfaces from the different bridges are now
    essentially connected at layer 2 and a network loop is possible in the
    time before the flows are added by DVR. Also the DVR code won't add them
    until after RPC calls to the server, so a loop is more likely if the
    server is not available.

    This patch restores adding these flows to when the physical bridges are
    first configured. Also updated a comment that was no longer correct and
    updated the unit test.

    Change-Id: I42c33fefaae6a7bee134779c840f35632823472e
    Closes-Bug: #1887148
    Related-Bug: #1869808
    (cherry picked from commit c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8)
    (cherry picked from commit 143fe8ff89ba776618ed6291af9d5e28e4662bdb)
    (cherry picked from commit 6a861b8c8c28e5675ec2208057298b811ba2b649)
    (cherry picked from commit 8181c5dbfe799ac6c832ab67b7eab3bcef4098b9)
    Conflicts:
            neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py

Reviewed: https://review.opendev.org/742366
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=00466f41d690ca7c7a918bfd861878ef620bbec9
Submitter: Zuul
Branch: stable/pike

commit 00466f41d690ca7c7a918bfd861878ef620bbec9
Author: Darragh O'Reilly <email address hidden>
Date: Mon Jul 13 14:48:10 2020 +0000

    Ensure drop flows on br-int at agent startup for DVR too

    Commit 90212b12 changed the OVS agent so adding vital drop flows on
    br-int (table 0 priority 2) for packets from physical bridges was
    deferred until DVR initialization later on. But if br-int has no flows
    from a previous run (eg after host reboot), then these packets will hit
    the NORMAL flow in table 60. And if there is more than one physical
    bridge, then the physical interfaces from the different bridges are now
    essentially connected at layer 2 and a network loop is possible in the
    time before the flows are added by DVR. Also the DVR code won't add them
    until after RPC calls to the server, so a loop is more likely if the
    server is not available.

    This patch restores adding these flows to when the physical bridges are
    first configured. Also updated a comment that was no longer correct and
    updated the unit test.

    Change-Id: I42c33fefaae6a7bee134779c840f35632823472e
    Closes-Bug: #1887148
    Related-Bug: #1869808
    (cherry picked from commit c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8)
    (cherry picked from commit 143fe8ff89ba776618ed6291af9d5e28e4662bdb)
    (cherry picked from commit 6a861b8c8c28e5675ec2208057298b811ba2b649)
    (cherry picked from commit 8181c5dbfe799ac6c832ab67b7eab3bcef4098b9)
    Conflicts:
            neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
    (cherry picked from commit 47ec363f5faefd85dfa33223c0087f4444afb5b9)
    Conflicts:
            neutron/tests/unit/plugins/ml2/drivers/openvswitch/agent/test_ovs_neutron_agent.py
    (cherry picked from commit 8a173ec29ac1819c3d28c191814cd1402d272bb9)

Reviewed: https://review.opendev.org/742365
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8a173ec29ac1819c3d28c191814cd1402d272bb9
Submitter: Zuul
Branch: stable/queens

commit 8a173ec29ac1819c3d28c191814cd1402d272bb9
Author: Darragh O'Reilly <email address hidden>
Date: Mon Jul 13 14:48:10 2020 +0000

    Ensure drop flows on br-int at agent startup for DVR too

    Commit 90212b12 changed the OVS agent so adding vital drop flows on
    br-int (table 0 priority 2) for packets from physical bridges was
    deferred until DVR initialization later on. But if br-int has no flows
    from a previous run (eg after host reboot), then these packets will hit
    the NORMAL flow in table 60. And if there is more than one physical
    bridge, then the physical interfaces from the different bridges are now
    essentially connected at layer 2 and a network loop is possible in the
    time before the flows are added by DVR. Also the DVR code won't add them
    until after RPC calls to the server, so a loop is more likely if the
    server is not available.

    This patch restores adding these flows to when the physical bridges are
    first configured. Also updated a comment that was no longer correct and
    updated the unit test.

    Change-Id: I42c33fefaae6a7bee134779c840f35632823472e
    Closes-Bug: #1887148
    Related-Bug: #1869808
    (cherry picked from commit c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8)
    (cherry picked from commit 143fe8ff89ba776618ed6291af9d5e28e4662bdb)
    (cherry picked from commit 6a861b8c8c28e5675ec2208057298b811ba2b649)
    (cherry picked from commit 8181c5dbfe799ac6c832ab67b7eab3bcef4098b9)
    Conflicts:
            neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py
    (cherry picked from commit 47ec363f5faefd85dfa33223c0087f4444afb5b9)
    Conflicts:
            neutron/tests/unit/plugins/ml2/drivers/openvswitch/agent/test_ovs_neutron_agent.py

Reviewed: https://review.opendev.org/742361
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8181c5dbfe799ac6c832ab67b7eab3bcef4098b9
Submitter: Zuul
Branch: stable/stein

commit 8181c5dbfe799ac6c832ab67b7eab3bcef4098b9
Author: Darragh O'Reilly <email address hidden>
Date: Mon Jul 13 14:48:10 2020 +0000

    Ensure drop flows on br-int at agent startup for DVR too

    Commit 90212b12 changed the OVS agent so adding vital drop flows on
    br-int (table 0 priority 2) for packets from physical bridges was
    deferred until DVR initialization later on. But if br-int has no flows
    from a previous run (eg after host reboot), then these packets will hit
    the NORMAL flow in table 60. And if there is more than one physical
    bridge, then the physical interfaces from the different bridges are now
    essentially connected at layer 2 and a network loop is possible in the
    time before the flows are added by DVR. Also the DVR code won't add them
    until after RPC calls to the server, so a loop is more likely if the
    server is not available.

    This patch restores adding these flows to when the physical bridges are
    first configured. Also updated a comment that was no longer correct and
    updated the unit test.

    Change-Id: I42c33fefaae6a7bee134779c840f35632823472e
    Closes-Bug: #1887148
    Related-Bug: #1869808
    (cherry picked from commit c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8)
    (cherry picked from commit 143fe8ff89ba776618ed6291af9d5e28e4662bdb)
    (cherry picked from commit 6a861b8c8c28e5675ec2208057298b811ba2b649)

Reviewed: https://review.opendev.org/742355
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=143fe8ff89ba776618ed6291af9d5e28e4662bdb
Submitter: Zuul
Branch: stable/ussuri

commit 143fe8ff89ba776618ed6291af9d5e28e4662bdb
Author: Darragh O'Reilly <email address hidden>
Date: Mon Jul 13 14:48:10 2020 +0000

    Ensure drop flows on br-int at agent startup for DVR too

    Commit 90212b12 changed the OVS agent so adding vital drop flows on
    br-int (table 0 priority 2) for packets from physical bridges was
    deferred until DVR initialization later on. But if br-int has no flows
    from a previous run (eg after host reboot), then these packets will hit
    the NORMAL flow in table 60. And if there is more than one physical
    bridge, then the physical interfaces from the different bridges are now
    essentially connected at layer 2 and a network loop is possible in the
    time before the flows are added by DVR. Also the DVR code won't add them
    until after RPC calls to the server, so a loop is more likely if the
    server is not available.

    This patch restores adding these flows to when the physical bridges are
    first configured. Also updated a comment that was no longer correct and
    updated the unit test.

    Change-Id: I42c33fefaae6a7bee134779c840f35632823472e
    Closes-Bug: #1887148
    Related-Bug: #1869808
    (cherry picked from commit c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8)

Reviewed: https://review.opendev.org/742359
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=17eded13595b18ab60af5256e0f63c57c3702296
Submitter: Zuul
Branch: stable/train

commit 17eded13595b18ab60af5256e0f63c57c3702296
Author: Darragh O'Reilly <email address hidden>
Date: Mon Jul 13 14:48:10 2020 +0000

    Ensure drop flows on br-int at agent startup for DVR too

    Commit 90212b12 changed the OVS agent so adding vital drop flows on
    br-int (table 0 priority 2) for packets from physical bridges was
    deferred until DVR initialization later on. But if br-int has no flows
    from a previous run (eg after host reboot), then these packets will hit
    the NORMAL flow in table 60. And if there is more than one physical
    bridge, then the physical interfaces from the different bridges are now
    essentially connected at layer 2 and a network loop is possible in the
    time before the flows are added by DVR. Also the DVR code won't add them
    until after RPC calls to the server, so a loop is more likely if the
    server is not available.

    This patch restores adding these flows to when the physical bridges are
    first configured. Also updated a comment that was no longer correct and
    updated the unit test.

    Change-Id: I42c33fefaae6a7bee134779c840f35632823472e
    Closes-Bug: #1887148
    Related-Bug: #1869808
    (cherry picked from commit c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8)
    (cherry picked from commit 143fe8ff89ba776618ed6291af9d5e28e4662bdb)

The attachment "debdiff for ubuntu cloud archive (queens)" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

SRU proposed for Ubuntu Bionic + Cloud Archive (Queens) for the following 3 bugs:
Bug #1869808 reboot neutron-ovs-agent introduces a short interrupt of vlan traffic
Bug #1887148 Network loop between physical networks with DVR (Fix for fix to Bug #1869808)
Bug #1871850 [L3] existing router resources are partial deleted unexpectedly when MQ is gone

SRU is only required for Bionic + Queens Cloud Archive, all other releases already have these patches.

==
reboot neutron-ovs-agent introduces a short interrupt of vlan traffic
https://bugs.launchpad.net/neutron/+bug/1869808

pike 1f4f888ad34d54ec968d9c9f9f80c388f3ca0d12 stable/pike [EOL]
queens 131bbc9a53411033cf27664d8f1fd7afc72c57bf stable/queens [Needed]
rocky cc48edf85cf66277423b0eb52ae6353f8028d2a6 stable/rocky [EOL]
stein 6dfc35680fcc885d9ad449ca2b39225fb1bca898 14.3.0 [Already done]
train 4f501f405d1c44e00784df8450cbe83129da1ea7 15.2.0 [Already done]
ussuri 88e70a520acaca37db645c3ef1124df8c7d778d5 16.1.0 [Already done]
master 90212b12cdf62e92d811997ebba699cab431d696 17.0.0 [Already done]

==
[L3] existing router resources are partial deleted unexpectedly when MQ is gone
https://bugs.launchpad.net/neutron/+bug/1871850

queens ec6c98060d78c97edf6382ede977209f007fdb81 stable/queens [Needed]
rocky 5ee377952badd94d08425aab41853916092acd07 stable/rocky [EOL]
stein 71f22834f2240834ca591e27a920f9444bac9689 14.4.0 [Already done]
train a96ad52c7e57664c63e3675b64718c5a288946fb 15.3.0 [Already done]
ussuri 5eeb98cdb51dc0dadd43128d1d0ed7d497606ded 16.2.0 [Already done]
master 12b9149e20665d80c11f1ef3d2283e1fa6f3b693 17.0.0 [Already done]

==
Network loop between physical networks with DVR (Fix for 1869808)
https://bugs.launchpad.net/neutron/+bug/1887148

pike 00466f41d690ca7c7a918bfd861878ef620bbec9 stable/pike [EOL]
queens 8a173ec29ac1819c3d28c191814cd1402d272bb9 stable/queens [Needed]
rocky 47ec363f5faefd85dfa33223c0087f4444afb5b9 stable/rocky [EOL]
stein 8181c5dbfe799ac6c832ab67b7eab3bcef4098b9 14.3.1 [Already done]
train 17eded13595b18ab60af5256e0f63c57c3702296 15.2.0 [Already done]
ussuri 143fe8ff89ba776618ed6291af9d5e28e4662bdb 16.1.0 [Already done]
master c1a77ef8b74bb9b5abbc5cb03fb3201383122eb8 17.0.0 [Already done]

Ubuntu SRU Justification

[Impact]

- When there is a RabbitMQ or neutron-api outage, the neutron-openvswitch-agent undergoes a "resync" process and temporarily blocks all VM traffic. This always happens for a short time period (maybe <1 second) but in some high scale environments this lasts for minutes. If RabbitMQ is down again during the re-sync, traffic will also be blocked until it can connect which may be for a long period. This also affects situations where neutron-openvswitch-agent is intentionally restarted while RabbitMQ is down. Bug #1869808 addresses this issue and Bug #1887148 is a fix for that fix to prevent network loops during DVR startup.

- In the same situation, the neutron-l3-agent can delete the L3 router (Bug #1871850)

[Test Case]

(1) Deploy Openstack Bionic-Queens with DVR and a *VLAN* tenant network (VXLAN or FLAT will not reproduce the issue). With a standard deployment, simply enabling DHCP on the ext_net subnet will allow VMs to be booted directly on the ext_net provider network. "openstack subnet set --dhcp ext_net and then deploy the VM directly to ext_net"

(2) Deploy a VM to the VLAN network

(3) Start pinging the VM from an external network

(4) Stop all RabbitMQ servers

(5) Restart neutron-openvswitch-agent

(6) Ping traffic should cease and not recover

(7) Start all RabbitMQ servers

(8) Ping traffic will recover after 30-60 seconds

[Where problems could occur]

These patches are all cherry-picked from the upstream stable branches, and have existed upstream including the stable/queens branch for many months and in Ubuntu all supported subsequent releases (Stein onwards) have also had these patches for many months with the exception of Queens.

There is a chance that not installing these drop flows during startup could have traffic go somewhere that's not expected when the network is in a partially setup case, this was the case for DVR and in setups where more than 1 DVR external network port existed a network loop was possibly temporarily created. This was already addressed with the included patch for Bug #1869808. Checked and could not locate any other merged changes to this drop_port logic that also need to be backported.

[Other Info]

Attaching revised SRU patch for Ubuntu Bionic, no code content changes but fixed the changelog to list all 3 bug numbers correctly.

@Trent, thanks for the patches. I've modified them slightly to be cherry-picked from upstream's stable/rocky branch.

Correction, stable/queens patches are the same as the attached debdiff. I created stable/rocky patches based on upstream stable/stein.

neutron 2:12.1.1-0ubuntu4 is in the bionic unapproved queue:
https://launchpad.net/ubuntu/bionic/+queue?queue_state=1&queue_text=neutron

neutron 2:13.0.7-0ubuntu1~cloud3 has been uploaded to rocky-staging

Hello norman, or anyone else affected,

Accepted neutron into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

After a further review of the patches surrounding this issue I decided to pull what looks more like a complete set of the associated patches from stable/queens and have been testing a build that I am now happy with. It behaves no differently to the current upload but supports hopefully all the edge cases around ovs agent restart and resync that have been resolved in stable/queens. I will attach a debdiff and would like to request that replace the currently uploaded package with this.

The new set of patches is as follows:

 d/p/0001-ovs-agent-signal-to-plugin-if-tunnel-refresh-needed.patch (LP: #1853613)
 d/p/0002-Do-not-block-connection-between-br-int-and-br-phys-o.patch (LP: #1869808)
 d/p/0003-Ensure-that-stale-flows-are-cleaned-from-phys_bridge.patch (LP: #1864822)
 d/p/0004-DVR-Reconfigure-re-created-physical-bridges-for-dvr-.patch (LP: #1864822)
 d/p/0005-Ensure-drop-flows-on-br-int-at-agent-startup-for-DVR.patch (LP: #1887148)
 d/p/0006-Don-t-check-if-any-bridges-were-recrected-when-OVS-w.patch (LP: #1864822)
 d/p/0007-Not-remove-the-running-router-when-MQ-is-unreachable.patch (LP: #1871850)

Same test case etc as current sru.

New build has been uploaded to bionic unapproved queue - https://launchpadlibrarian.net/526055760/neutron_12.1.1-0ubuntu4_source.changes. Waiting for approval and then will verify.

Looking to get this approved so that we can verify it, as needing this ideally released by the weekend of March 27th for some maintenance activity. Is something holding back the approval?

Hello norman, or anyone else affected,

Accepted neutron into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:12.1.1-0ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I have specifically verified that this bug (vlan traffic interruption during restart when rabbitmq is down) is fixed by the package in bionic-proposed. Followed my reproduction steps per the Test Case and all traffic to instances stops on 12.1.1-0ubuntu3 and does not stop on 12.1.1-0ubuntu4

But not completing verification yet as we need to perform more general testing on the package for regressions etc

Hello norman, or anyone else affected,

Accepted neutron into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Bionic-queens regression tests ran successfully:

23:25:18 ======
23:25:18 Totals
23:25:18 ======
23:25:18 Ran: 97 tests in 742.1731 sec.
23:25:18 - Passed: 89
23:25:18 - Skipped: 8
23:25:18 - Expected Fail: 0
23:25:18 - Unexpected Success: 0
23:25:18 - Failed: 0
23:25:18 Sum of execute time for each test: 586.7572 sec.

Verified Bionic queens using [Test Plan] with output as follows:

# apt-cache policy neutron-common
neutron-common:
  Installed: 2:12.1.1-0ubuntu4
  Candidate: 2:12.1.1-0ubuntu4
  Version table:
 *** 2:12.1.1-0ubuntu4 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2:12.1.1-0ubuntu3 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     2:12.0.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

I ran a ping for the duration of restarting both rabbit and neutron-openvswitch-agent and did not see any interruption.

Verified Bionic rocky (uca) using [Test Plan] with output as follows:

# apt-cache policy neutron-common
neutron-common:
  Installed: 2:13.0.7-0ubuntu1~cloud3
  Candidate: 2:13.0.7-0ubuntu1~cloud3
  Version table:
 *** 2:13.0.7-0ubuntu1~cloud3 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/rocky/main amd64 Packages
        100 /var/lib/dpkg/status
     2:12.1.1-0ubuntu3 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     2:12.0.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

I ran a ping for the duration of restarting both rabbit and neutron-openvswitch-agent and did not see any interruption.

Verified Xenial queens (uca) using [Test Plan] with output as follows:

# apt-cache policy neutron-common
neutron-common:
  Installed: 2:12.1.1-0ubuntu4~cloud0
  Candidate: 2:12.1.1-0ubuntu4~cloud0
  Version table:
 *** 2:12.1.1-0ubuntu4~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-proposed/queens/main amd64 Packages
        100 /var/lib/dpkg/status
     2:8.4.0-0ubuntu7.5 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
     2:8.4.0-0ubuntu7.4 500
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
     2:8.0.0-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

I ran a ping for the duration of restarting both rabbit and neutron-openvswitch-agent and did not see any interruption.

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package neutron - 2:12.1.1-0ubuntu4

---------------
neutron (2:12.1.1-0ubuntu4) bionic; urgency=medium

  * Fix interrupt of VLAN traffic on reboot of neutron-ovs-agent:
  - d/p/0001-ovs-agent-signal-to-plugin-if-tunnel-refresh-needed.patch (LP: #1853613)
  - d/p/0002-Do-not-block-connection-between-br-int-and-br-phys-o.patch (LP: #1869808)
  - d/p/0003-Ensure-that-stale-flows-are-cleaned-from-phys_bridge.patch (LP: #1864822)
  - d/p/0004-DVR-Reconfigure-re-created-physical-bridges-for-dvr-.patch (LP: #1864822)
  - d/p/0005-Ensure-drop-flows-on-br-int-at-agent-startup-for-DVR.patch (LP: #1887148)
  - d/p/0006-Don-t-check-if-any-bridges-were-recrected-when-OVS-w.patch (LP: #1864822)
  - d/p/0007-Not-remove-the-running-router-when-MQ-is-unreachable.patch (LP: #1871850)

 -- Edward Hope-Morley <email address hidden> Mon, 22 Feb 2021 16:55:40 +0000

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package neutron - 2:12.1.1-0ubuntu4~cloud0
---------------

 neutron (2:12.1.1-0ubuntu4~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:12.1.1-0ubuntu4) bionic; urgency=medium
 .
   * Fix interrupt of VLAN traffic on reboot of neutron-ovs-agent:
   - d/p/0001-ovs-agent-signal-to-plugin-if-tunnel-refresh-needed.patch (LP: #1853613)
   - d/p/0002-Do-not-block-connection-between-br-int-and-br-phys-o.patch (LP: #1869808)
   - d/p/0003-Ensure-that-stale-flows-are-cleaned-from-phys_bridge.patch (LP: #1864822)
   - d/p/0004-DVR-Reconfigure-re-created-physical-bridges-for-dvr-.patch (LP: #1864822)
   - d/p/0005-Ensure-drop-flows-on-br-int-at-agent-startup-for-DVR.patch (LP: #1887148)
   - d/p/0006-Don-t-check-if-any-bridges-were-recrected-when-OVS-w.patch (LP: #1864822)
   - d/p/0007-Not-remove-the-running-router-when-MQ-is-unreachable.patch (LP: #1871850)

This issue was fixed in the openstack/neutron pike-eol release.

This issue was fixed in the openstack/neutron queens-eol release.

This issue was fixed in the openstack/neutron rocky-eol release.