networking-ovn

Race condition while updating ports

Bug #1800846 reported by Lucas Alvares Gomes
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
networking-ovn
Fix Released
Medium
Lucas Alvares Gomes

Bug Description

There's apparently a race condition while ports are being updated. This was found in a rally:

Oct 31 11:36:29.440659 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers [None req-1e509247-03e6-4628-9f96-8bacd548433c c_rally_f43d21d2_VgsUwkug c_rally_f43d21d2_vQ0lxiwM] Mechanism driver 'ovn' failed in update_port_postcommit: SubnetNotFound: Subnet b6e093a2-1cc4-4dcc-afdf-48fe27d79ec2 could not be found.
Oct 31 11:36:29.440869 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers Traceback (most recent call last):
Oct 31 11:36:29.441149 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/opt/stack/neutron/neutron/plugins/ml2/managers.py", line 429, in _call_on_drivers
Oct 31 11:36:29.441333 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers getattr(driver.obj, method_name)(context)
Oct 31 11:36:29.441485 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/opt/stack/networking-ovn/networking_ovn/ml2/mech_driver.py", line 542, in update_port_postcommit
Oct 31 11:36:29.441671 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers self._ovn_client.update_port(port, port_object=original_port)
Oct 31 11:36:29.441840 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/opt/stack/networking-ovn/networking_ovn/common/ovn_client.py", line 362, in update_port
Oct 31 11:36:29.442005 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers port_info = self._get_port_options(port, qos_options)
Oct 31 11:36:29.442154 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/opt/stack/networking-ovn/networking_ovn/common/ovn_client.py", line 217, in _get_port_options
Oct 31 11:36:29.442303 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers ip['subnet_id'])
Oct 31 11:36:29.442457 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/neutron_lib/db/api.py", line 233, in wrapped
Oct 31 11:36:29.442612 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers return method(*args, **kwargs)
Oct 31 11:36:29.442765 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/neutron_lib/db/api.py", line 140, in wrapped
Oct 31 11:36:29.442928 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers setattr(e, '_RETRY_EXCEEDED', True)
Oct 31 11:36:29.443081 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 31 11:36:29.443232 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers self.force_reraise()
Oct 31 11:36:29.443386 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 31 11:36:29.443541 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers six.reraise(self.type_, self.value, self.tb)
Oct 31 11:36:29.443733 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/neutron_lib/db/api.py", line 136, in wrapped
Oct 31 11:36:29.443880 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers return f(*args, **kwargs)
Oct 31 11:36:29.444023 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 154, in wrapper
Oct 31 11:36:29.444162 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers ectxt.value = e.inner_exc
Oct 31 11:36:29.444300 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 31 11:36:29.444442 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers self.force_reraise()
Oct 31 11:36:29.444584 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 31 11:36:29.444725 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers six.reraise(self.type_, self.value, self.tb)
Oct 31 11:36:29.444878 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 142, in wrapper
Oct 31 11:36:29.445020 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers return f(*args, **kwargs)
Oct 31 11:36:29.445161 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
Oct 31 11:36:29.445316 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers LOG.debug("Retry wrapper got retriable exception: %s", e)
Oct 31 11:36:29.445457 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Oct 31 11:36:29.445610 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers self.force_reraise()
Oct 31 11:36:29.445749 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Oct 31 11:36:29.445911 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers six.reraise(self.type_, self.value, self.tb)
Oct 31 11:36:29.446046 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/usr/local/lib/python2.7/dist-packages/neutron_lib/db/api.py", line 179, in wrapped
Oct 31 11:36:29.446951 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers return f(*dup_args, **dup_kwargs)
Oct 31 11:36:29.447142 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1061, in get_subnet
Oct 31 11:36:29.447290 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers subnet_obj = self._get_subnet_object(context, id)
Oct 31 11:36:29.447437 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers File "/opt/stack/neutron/neutron/db/db_base_plugin_common.py", line 240, in _get_subnet_object
Oct 31 11:36:29.447597 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers raise n_exc.SubnetNotFound(subnet_id=id)
Oct 31 11:36:29.447793 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers SubnetNotFound: Subnet b6e093a2-1cc4-4dcc-afdf-48fe27d79ec2 could not be found.
Oct 31 11:36:29.447945 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.managers
Oct 31 11:36:29.450047 ubuntu-xenial-ovh-bhs1-0000216197 neutron-server[9201]: ERROR neutron.plugins.ml2.plugin [None req-1e509247-03e6-4628-9f96-8bacd548433c c_rally_f43d21d2_VgsUwkug c_rally_f43d21d2_vQ0lxiwM] mechanism_manager.update_port_postcommit failed for port 8fbe7f2c-f273-4619-a04f-b138295bc505: MechanismDriverError

Logs (will expire eventually): http://logs.openstack.org/74/606674/4/check/networking-ovn-rally-task/c58d16d/controller/logs/screen-q-svc.txt.gz?#_Oct_31_11_36_29_447793

Rally logs (will expire eventually), create_and_update_ports failure due to this error: http://logs.openstack.org/74/606674/4/check/networking-ovn-rally-task/c58d16d/results/report.html.gz#/NeutronNetworks.create_and_update_ports

Lucas Alvares Gomes (lucasagomes)
Changed in networking-ovn:
status: New → Confirmed
importance: Undecided → Medium
assignee: nobody → Lucas Alvares Gomes (lucasagomes)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-ovn (master)

Fix proposed to branch: master
Review: https://review.openstack.org/614520

Changed in networking-ovn:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-ovn (master)

Reviewed: https://review.openstack.org/614520
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=18e566c9c96b3f815df4f2867cc2d459f527c5b5
Submitter: Zuul
Branch: master

commit 18e566c9c96b3f815df4f2867cc2d459f527c5b5
Author: Lucas Alvares Gomes <email address hidden>
Date: Wed Oct 31 14:14:22 2018 +0000

    Fix update_port() SubnetNotFound race conditional

    While in collecting port information in the update_port() method the
    code will loop thru the fixed_ips associated with the port and fetch the
    subnet the IP belongs to from the Neutron database but, in some cases,
    that subnet would have been deleted mid-operation. That would cause the
    whole update_port() to error out with a SubnetNotFound exception.

    This patch fixes the problem by handling such exception and not
    appending that IP to the list of addresses (in OVN DB terms) in case the
    subnet has already been deleted.

    Change-Id: Ia19324722425184da13a7a0383185486eed62e38
    Closes-Bug: #1800846
    Signed-off-by: Lucas Alvares Gomes <email address hidden>

Changed in networking-ovn:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to networking-ovn (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/617952

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to networking-ovn (stable/rocky)

Reviewed: https://review.openstack.org/617952
Committed: https://git.openstack.org/cgit/openstack/networking-ovn/commit/?id=1a02228cb9d1232101447983b8aa6e8e180fcad2
Submitter: Zuul
Branch: stable/rocky

commit 1a02228cb9d1232101447983b8aa6e8e180fcad2
Author: Lucas Alvares Gomes <email address hidden>
Date: Wed Oct 31 14:14:22 2018 +0000

    Fix update_port() SubnetNotFound race conditional

    While in collecting port information in the update_port() method the
    code will loop thru the fixed_ips associated with the port and fetch the
    subnet the IP belongs to from the Neutron database but, in some cases,
    that subnet would have been deleted mid-operation. That would cause the
    whole update_port() to error out with a SubnetNotFound exception.

    This patch fixes the problem by handling such exception and not
    appending that IP to the list of addresses (in OVN DB terms) in case the
    subnet has already been deleted.

    Change-Id: Ia19324722425184da13a7a0383185486eed62e38
    Closes-Bug: #1800846
    Signed-off-by: Lucas Alvares Gomes <email address hidden>
    (cherry picked from commit 18e566c9c96b3f815df4f2867cc2d459f527c5b5)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn 6.0.0.0b1

This issue was fixed in the openstack/networking-ovn 6.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn 5.1.0

This issue was fixed in the openstack/networking-ovn 5.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.