portbinding update fails with ObjectDeletedError

Bug #1669528 reported by Boden R
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Kevin Benton

Bug Description

The gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial gate failed in a recent neutron change [1]. Digging into the logs [2] it appears to be DB related::

--
2017-03-01 15:47:17.800 1412 ERROR oslo_messaging.rpc.server ObjectDeletedError: Instance '<PortBinding at 0x7f9abdd7af10>' has been deleted, or its row is otherwise not present.
--

Based on logstash [3], this appears to be a frequent error. It's not clear to me if neutron is really the issue here, or if something else is happening and causing the neutron error.

[1] https://review.openstack.org/#/c/422210/
[2] http://logs.openstack.org/10/422210/9/check/gate-tempest-dsvm-neutron-linuxbridge-ubuntu-xenial/f639199/logs/screen-q-svc.txt.gz?
[3] http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22ObjectDeletedError%5C%22

Matt Riedemann (mriedem)
Changed in neutron:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Matt Riedemann (mriedem) wrote :
Revision history for this message
Matt Riedemann (mriedem) wrote :

Actually I think it goes back further than 3/2. Ihar pointed this out in IRC:

https://review.openstack.org/#/c/423581/6

From looking at the logstash results, it shows up a lot in successful runs too so it's probably not causing failures, but it shows up in a lot of the runs so there is a problem to be solved with updating/deleting ports.

Changed in neutron:
assignee: nobody → Kevin Benton (kevinbenton)
Revision history for this message
Kevin Benton (kevinbenton) wrote :

I don't think the locked_port_and_binding change will have had an impact. This exception is being thrown after the transaction is closed so the lock would have been released.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/441231

Changed in neutron:
status: Confirmed → In Progress
tags: added: db
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/441231
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fc563eaabec392281be90bd2472bed4f6506cb40
Submitter: Jenkins
Branch: master

commit fc563eaabec392281be90bd2472bed4f6506cb40
Author: Kevin Benton <email address hidden>
Date: Fri Mar 3 08:51:33 2017 -0800

    deepcopy binding and binding levels avoid expiration

    Perform a deepcopy on the sqla objects passed into the PortContext
    so we get detached versions of them safe to reference forever.
    This is necessary because the PortContexts outlive the
    transaction context managers they are creating in which means an
    object can be expired and result in a query after a commit
    (e.g. in bind_port_if_needed) that will fail and result in an
    exception.

    This required a few additional explicit session.merge calls to deal
    with cases where touching the mech context was implicitly expected
    to modify the DB state on the next commit.

    Closes-Bug: #1669528
    Change-Id: Ib5ba2daa80acba53c082bade1f61a3ee44ca41fc

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 11.0.0.0b1

This issue was fixed in the openstack/neutron 11.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/470268

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/470297

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/newton)

Reviewed: https://review.openstack.org/470297
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=cb773098d276cfbabc600361e91cb709ec846ba8
Submitter: Jenkins
Branch: stable/newton

commit cb773098d276cfbabc600361e91cb709ec846ba8
Author: Kevin Benton <email address hidden>
Date: Fri Mar 3 08:51:33 2017 -0800

    deepcopy binding and binding levels avoid expiration

    Perform a deepcopy on the sqla objects passed into the PortContext
    so we get detached versions of them safe to reference forever.
    This is necessary because the PortContexts outlive the
    transaction context managers they are creating in which means an
    object can be expired and result in a query after a commit
    (e.g. in bind_port_if_needed) that will fail and result in an
    exception.

    This required a few additional explicit session.merge calls to deal
    with cases where touching the mech context was implicitly expected
    to modify the DB state on the next commit.

    Conflicts:
     neutron/plugins/ml2/driver_context.py
     neutron/plugins/ml2/plugin.py
     neutron/tests/unit/plugins/ml2/test_port_binding.py

    Closes-Bug: #1669528
    Change-Id: Ib5ba2daa80acba53c082bade1f61a3ee44ca41fc
    (cherry picked from commit 20c1de9dc83875431e8bf995f3f4f85bd6d81955)

tags: added: in-stable-newton
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/ocata)

Change abandoned by Jakub Libosvar (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/470268
Reason: Abandoned as per Kevin's comments.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 9.4.1

This issue was fixed in the openstack/neutron 9.4.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.