kuryr

VethDeletionFailure: Deleting the veth pair failed

Bug #1776035 reported by hongbin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kuryr
New
Undecided
hongbin
kuryr-libnetwork
New
Undecided
Unassigned

Bug Description

The gate consistently failed with the following error:

  "VethDeletionFailure: Deleting the veth pair failed."

This happened only on the gate. I couldn't reproduce it locally. Here is the full trace:

  http://paste.openstack.org/show/723041/

Here is gate log:

  http://logs.openstack.org/52/574052/1/check/zun-tempest-docker-etcd/200e9b9/logs/screen-kuryr-libnetwork.txt.gz#_Jun_09_20_40_41_420612

logstash:

  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22VethDeletionFailure%3A%20Deleting%20the%20veth%20pair%20failed%5C%22%20AND%20tags%3A%5C%22screen-kuryr-libnetwork.txt%5C%22

See original description
hongbin (hongbin034)
Changed in kuryr:
assignee: nobody → hongbin (hongbin034)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to kuryr (master)

Reviewed: https://review.openstack.org/574042
Committed: https://git.openstack.org/cgit/openstack/kuryr/commit/?id=2f38e07a38be59305c573311a04c8d4b6de5a912
Submitter: Zuul
Branch: master

commit 2f38e07a38be59305c573311a04c8d4b6de5a912
Author: Hongbin Lu <email address hidden>
Date: Sat Jun 9 17:27:19 2018 +0000

    Add logging for the original message

    On port binding/unbinding, Kuryr catches exception raised from
    pyroute2 and re-raise a new exception. As a result, the detailed
    information contained in the original exception loses. This patch
    adds a logging to record the original exception message and trace.

    Related-Bug: #1776035
    Change-Id: I4b13724a460d84a2b953f750b8b88c0f60cee97d

Revision history for this message
hongbin (hongbin034) wrote :
Download full text (3.2 KiB)

Sample failure in gate:

May 06 00:50:16.378720 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: 2019-05-06 00:50:16.377 7630 ERROR kuryr_libnetwork.utils [-] Unexpected error happened: Deleting the veth pair failed.: VethDeletionFailure: Deleting the veth pair failed.
May 06 00:50:16.378720 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: Traceback (most recent call last):
May 06 00:50:16.379043 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1813, in full_dispatch_request
May 06 00:50:16.381128 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: rv = self.dispatch_request()
May 06 00:50:16.381128 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 1799, in dispatch_request
May 06 00:50:16.381128 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: return self.view_functions[rule.endpoint](**req.view_args)
May 06 00:50:16.381600 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: File "/opt/stack/kuryr-libnetwork/kuryr_libnetwork/controllers.py", line 1291, in network_driver_delete_endpoint
May 06 00:50:16.382960 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: LOG.error('Cleaning the veth pair up was failed.')
May 06 00:50:16.382960 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
May 06 00:50:16.383588 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: self.force_reraise()
May 06 00:50:16.383588 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
May 06 00:50:16.383891 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: six.reraise(self.type_, self.value, self.tb)
May 06 00:50:16.383891 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: File "/opt/stack/kuryr-libnetwork/kuryr_libnetwork/controllers.py", line 1281, in network_driver_delete_endpoint
May 06 00:50:16.383891 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: endpoint_id, neutron_port)
May 06 00:50:16.383891 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: File "/opt/stack/kuryr-libnetwork/kuryr_libnetwork/port_driver/drivers/veth.py", line 81, in delete_host_iface
May 06 00:50:16.384451 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: return binding.port_unbind(endpoint_id, neutron_port)
May 06 00:50:16.384451 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: File "/usr/local/lib/python2.7/dist-packages/kuryr/lib/binding/__init__.py", line 75, in port_unbind
May 06 00:50:16.384762 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: return driver.port_unbind(endpoint_id, neutron_port, **kwargs)
May 06 00:50:16.384762 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: File "/usr/local/lib/python2.7/dist-packages/kuryr/lib/binding/drivers/veth.py", line 123, in port_unbind
May 06 00:50:16.384762 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: 'Deleting the veth pair failed.')
May 06 00:50:16.384762 ubuntu-bionic-rax-ord-0005988061 kuryr-server[7630]: VethDeletionF...

Read more...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.