neutron

ovs agent is stuck with OVSFWTagNotFound when dealing with unbound port

Bug #1849098 reported by Oleg Bondarev on 2019-10-21

This bug affects 1 person

	Status	Importance	Assigned to
Ubuntu Cloud Archive	Fix Released	Undecided	Unassigned
Queens	Fix Released	Undecided	Unassigned
neutron	Fix Released	High	Oleg Bondarev
neutron (Ubuntu)	Fix Released	Undecided	Unassigned
Bionic	Fix Released	Undecided	Seyeong Kim

Bug Description

[Impact]

somehow port is unbounded, then neutron-openvswitch-agent raise
OVSFWTagNotFound, then creating new instance will be failed.

[Test Plan]
1. deploy bionic openstack env
2. launch one instance
3. modify neutron-openvswitch-agent code inside nova-compute
- https://pastebin.ubuntu.com/p/nBRKkXmjx8/
4. restart neutron-openvswitch-agent
5. check if there are a lot of cannot get tag for port ..
6. launch another instance.
7. It fails after vif_plugging_timeout, with "virtual interface creation failed"

[Where problems could occur]
while no regressions are expected, if they do occur it would be when getting or creating vif port

[Others]

Original description.

neutron-openvswitch-agent meets unbound port:

2019-10-17 11:32:21.868 135 WARNING neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Device ef34215f-e099-4fd0-935f-c9a42951d166 not defined on plugin or binding failed

Later when applying firewall rules:

2019-10-17 11:32:21.901 135 INFO neutron.agent.securitygroups_rpc [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Preparing filters for devices {'ef34215f-e099-4fd0-935f-c9a42951d166', 'e9c97cf0-1a5e-4d77-b57b-0ba474d12e29', 'fff1bb24-6423-4486-87c4-1fe17c552cca', '2e20f9ee-bcb5-445c-b31f-d70d276d45c9', '03a60047-cb07-42a4-8b49-619d5982a9bd', 'a452cea2-deaf-4411-bbae-ce83870cbad4', '79b03e5c-9be0-4808-9784-cb4878c3dbd5', '9b971e75-3c1b-463d-88cf-3f298105fa6e'}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Error while processing VIF ports: neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 530, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.sg_port_map.ports[port_id]
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'ef34215f-e099-4fd0-935f-c9a42951d166'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 81, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return int(other_config['tag'])
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent KeyError: 'tag'
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent During handling of the above exception, another exception occurred:
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2280, in rpc_loop
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1847, in process_network_ports
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info.get('updated', set()))
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 258, in setup_port_filters
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.prepare_devices_filter(new_devices)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 125, in decorated_function
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent *args, **kwargs)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 133, in prepare_devices_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self._apply_port_filter(device_ids)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/securitygroups_rpc.py", line 164, in _apply_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.firewall.prepare_port_filter(device)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 555, in prepare_port_filter
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent of_port = self.get_or_create_ofport(port)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 532, in get_or_create_ofport
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_vlan_id = self._get_port_vlan_tag(ovs_port.port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 516, in _get_port_vlan_tag
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return get_tag_from_other_config(self.int_br.br, port_name)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/var/lib/openstack/lib/python3.6/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py", line 84, in get_tag_from_other_config
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_name=port_name, other_config=other_config)
2019-10-17 11:32:21.906 135 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent neutron.agent.linux.openvswitch_firewall.exceptions.OVSFWTagNotFound: Cannot get tag for port o-hm0 from its other_config: {}
2019-10-17 11:32:21.909 135 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-aae68b42-a99f-4bb3-bcf6-a6d3c4ca9e31 - - - - -] Agent out of sync with plugin!

this happens in each agent cycle so agent can't do anything.

Need to handle OVSFWTagNotFound in prepare_port_filter() like was done for update_port_filter in https://review.opendev.org/#/c/630910/

See original description

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-21: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/689720

Changed in neutron:
status:	New → In Progress

Lajos Katona (lajos-katona) on 2019-10-22

tags:

added: ovs-fw

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-27: Fix merged to neutron (master)

Reviewed: https://review.opendev.org/689720
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e801159003978f7c5c67eb3e9fe36fa690656d77
Submitter: Zuul
Branch: master

commit e801159003978f7c5c67eb3e9fe36fa690656d77
Author: Oleg Bondarev <email address hidden>
Date: Mon Oct 21 15:40:12 2019 +0400

Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall

    This will prevent ovs agent from endless fail loop when dealing
    with unbound port: like when port was created in neutron before
    agent become alive, then agent gets online and and starts processing
    devices.
    This patch adds exception handling to prepare_port_filter() -
    same as done in update_port_filter().

Change-Id: I1137eb18efaf51c67fab145e645f58cbd3772e40
Closes-Bug: #1849098

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-28: Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/691604

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-28: Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/691605

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-29: Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/691861

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-29: Fix merged to neutron (stable/train)

Reviewed: https://review.opendev.org/691604
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c20c77feb98275e9d7337bb7fc02b188ca2a5daa
Submitter: Zuul
Branch: stable/train

commit c20c77feb98275e9d7337bb7fc02b188ca2a5daa
Author: Oleg Bondarev <email address hidden>
Date: Mon Oct 21 15:40:12 2019 +0400

Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall

    Change-Id: I1137eb18efaf51c67fab145e645f58cbd3772e40
    Closes-Bug: #1849098
    (cherry picked from commit e801159003978f7c5c67eb3e9fe36fa690656d77)

tags:

added: in-stable-train

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-30: Fix merged to neutron (stable/stein)

Reviewed: https://review.opendev.org/691605
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0fb7f8ddac0d55dfc64c0c0c2931b3c17cb00d37
Submitter: Zuul
Branch: stable/stein

commit 0fb7f8ddac0d55dfc64c0c0c2931b3c17cb00d37
Author: Oleg Bondarev <email address hidden>
Date: Mon Oct 21 15:40:12 2019 +0400

Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall

    Change-Id: I1137eb18efaf51c67fab145e645f58cbd3772e40
    Closes-Bug: #1849098
    (cherry picked from commit e801159003978f7c5c67eb3e9fe36fa690656d77)

tags:

added: in-stable-stein

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-11-11: Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/693681

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-11-15: Fix merged to neutron (stable/rocky)

Reviewed: https://review.opendev.org/693681
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=879bb9032857bf0109185e85095ef4b5c77bcbde
Submitter: Zuul
Branch: stable/rocky

commit 879bb9032857bf0109185e85095ef4b5c77bcbde
Author: Oleg Bondarev <email address hidden>
Date: Mon Oct 21 15:40:12 2019 +0400

Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall

    Change-Id: I1137eb18efaf51c67fab145e645f58cbd3772e40
    Closes-Bug: #1849098
    (cherry picked from commit e801159003978f7c5c67eb3e9fe36fa690656d77)
    (cherry picked from commit 0fb7f8ddac0d55dfc64c0c0c2931b3c17cb00d37)

tags:

added: in-stable-rocky

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-11-15: Fix merged to neutron (stable/queens)

#10

Reviewed: https://review.opendev.org/691861
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6d581e94aa9a42894ebb9161790e64b2b60828b0
Submitter: Zuul
Branch: stable/queens

commit 6d581e94aa9a42894ebb9161790e64b2b60828b0
Author: Oleg Bondarev <email address hidden>
Date: Mon Oct 21 15:40:12 2019 +0400

Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall

    Change-Id: I1137eb18efaf51c67fab145e645f58cbd3772e40
    Closes-Bug: #1849098
    (cherry picked from commit e801159003978f7c5c67eb3e9fe36fa690656d77)

tags:

added: in-stable-queens

Bernard Cafarelli (bcafarel) on 2019-11-28

tags:

added: neutron-proactive-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-12-19: Fix included in openstack/neutron 13.0.6

#11

This issue was fixed in the openstack/neutron 13.0.6 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-12-19: Fix included in openstack/neutron 14.0.4

#12

This issue was fixed in the openstack/neutron 14.0.4 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-12-19: Fix included in openstack/neutron 15.0.1

#13

This issue was fixed in the openstack/neutron 15.0.1 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-02-21: Fix included in openstack/neutron 16.0.0.0b1

#14

This issue was fixed in the openstack/neutron 16.0.0.0b1 development milestone.

Revision history for this message

Seyeong Kim (seyeongkim) wrote on 2021-05-03:

#15

lp1849098_bionic.debdiff Edit (6.8 KiB, text/plain)

Changed in neutron (Ubuntu):
status:	New → Fix Released
Changed in neutron (Ubuntu Bionic):
assignee:	nobody → Seyeong Kim (seyeongkim)
status:	New → In Progress

Seyeong Kim (seyeongkim) on 2021-05-03

description:	updated
tags:	added: sts

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2021-05-04:

#16

@Thanks Seyeong. I've picked the patch directly from the upstream stable/queens branch and uploaded to the bionic unapproved queue.

Dan Streetman (ddstreet) on 2021-05-04

description:	updated
description:	updated

Revision history for this message

Robie Basak (racb) wrote on 2021-05-05: Please test proposed package

#17

Hello Oleg, or anyone else affected,

Accepted neutron into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:12.1.1-0ubuntu7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in neutron (Ubuntu Bionic):
status:	In Progress → Fix Committed
tags:	added: verification-needed verification-needed-bionic

Revision history for this message

Hemanth Nakkina (hemanth-n) wrote on 2021-05-05:

#18

Verified the test case with the bionic-proposed (neutron 2:12.1.1-0ubuntu7) and test case is working fine

cirros-110548: VM launched after deployment
testvm-after-tagcomment: VM launched after changing the code as mentioned in test case and VM is in ERROR state and logs shows tag errrors
testvm-after-fix: VM launched after the neutron package is upgraded with the one in bionic-proposed.

tags:

added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic

Revision history for this message

Łukasz Zemczak (sil2100) wrote on 2021-05-13: Update Released

#19

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2021-05-13:

#20

This bug was fixed in the package neutron - 2:12.1.1-0ubuntu7

---------------
neutron (2:12.1.1-0ubuntu7) bionic; urgency=medium

  * Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall
    - d/p/0001-Handle-OVSFWPortNotFound-and-OVSFWTagNotFound-in-ovs.patch
      (LP: #1849098).

neutron (2:12.1.1-0ubuntu6) bionic; urgency=medium

* Do not initialize snat-ns twice (LP: #1850779)
- d/p/0001-Do-not-initialize-snat-ns-twice.patch

neutron (2:12.1.1-0ubuntu5) bionic; urgency=medium

* Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
- d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch

-- Seyeong Kim <email address hidden> Mon, 03 May 2021 17:15:28 +0900

Changed in neutron (Ubuntu Bionic):
status:	Fix Committed → Fix Released

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2021-05-13:

#21

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2021-05-13:

#22

This bug was fixed in the package neutron - 2:12.1.1-0ubuntu7~cloud0
---------------

neutron (2:12.1.1-0ubuntu7~cloud0) xenial-queens; urgency=medium
.
   * New update for the Ubuntu Cloud Archive.
.
neutron (2:12.1.1-0ubuntu7) bionic; urgency=medium
.
   * Handle OVSFWPortNotFound and OVSFWTagNotFound in ovs firewall
     - d/p/0001-Handle-OVSFWPortNotFound-and-OVSFWTagNotFound-in-ovs.patch
       (LP: #1849098).
.
neutron (2:12.1.1-0ubuntu6) bionic; urgency=medium
.
   * Do not initialize snat-ns twice (LP: #1850779)
     - d/p/0001-Do-not-initialize-snat-ns-twice.patch
.
neutron (2:12.1.1-0ubuntu5) bionic; urgency=medium
.
   * Backport fix for dvr-snat missig rfp interfaces (LP: #1894843)
     - d/p/0001-Fix-deletion-of-rfp-interfaces-when-router-is-re-ena.patch