tempest.api.orchestration.stacks.test_stacks times out during tearDownClass

I do see this db lock error in the engine log:

http://logs.openstack.org/47/86047/1/check/check-tempest-dsvm-postgres-full/0f04e55/logs/screen-h-eng.txt.gz?level=TRACE#_2014-04-08_21_33_07_587

WARNING heat.engine.stack_lock [-] Lock was already released on stack 58ce15a1-f0d3-4ffe-b322-ffd714002542!

OperationalError: (OperationalError) unknown error 'SELECT resource.created_at AS resource_created_at, resource.id AS resource_id, resource.action AS resource_action, resource.status AS resource_status, resource.name AS resource_name, resource.nova_instance AS resource_nova_instance, resource.status_reason AS resource_status_reason, resource.rsrc_metadata AS resource_rsrc_metadata, resource.stack_id AS resource_stack_id, resource.updated_at AS resource_updated_at \nFROM resource \nWHERE %(param_1)s = resource.stack_id' {'param_1': u'58ce15a1-f0d3-4ffe-b322-ffd714002542'}

There are 46 hits in 7 days on that, but it's showing up in a lot of success runs too:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiTG9jayB3YXMgYWxyZWFkeSByZWxlYXNlZCBvbiBzdGFja1wiIEFORCB0YWdzOnNjcmVlbi1oLWVuZy50eHQiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjEzOTcxNDQ4NzYzNzR9

e-r patch: https://review.openstack.org/#/c/86636/

Related fix proposed to branch: master
Review: https://review.openstack.org/99503

Related fix proposed to branch: master
Review: https://review.openstack.org/99517

Reviewed: https://review.openstack.org/99517
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=09d023887bc92c9e392f3923141113825a78a426
Submitter: Jenkins
Branch: master

commit 09d023887bc92c9e392f3923141113825a78a426
Author: Steve Baker <email address hidden>
Date: Thu Jun 12 00:23:22 2014 +0000

    Revert "Use auth_token_info to initialize heat_keystoneclient sessions"

    It looks like this change has revealed an unrelated race with locking
    during delete. While this issue is investigated then this change
    needs to be reverted to fix the gate.

    Related-Bug: #1306029

    This reverts commit 535dd4e77e2c439a8611f27e5e9fa821cc8d7589.

    Change-Id: I06be473d7c6961e399c4fd6128f374d24b9b0510

Reviewed: https://review.openstack.org/99503
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=02453adfae612aa53be2ec7e39a0ff5184dd414e
Submitter: Jenkins
Branch: master

commit 02453adfae612aa53be2ec7e39a0ff5184dd414e
Author: Clint Byrum <email address hidden>
Date: Wed Jun 11 16:09:45 2014 -0700

    Handle missing user_creds in stack.delete

    In some cases concurrent or interrupted and resumed stack delete
    operations will result in missing user_creds records referenced in a
    stack. This may be the cause of race conditions between deletes
    resulting in unexpected behavior and spurious failures.

    Change-Id: I76f3f8803a4f9171223783a9c6eea0fc20280010
    Related-Bug: #1306029

Did the fix above solve this issue, or do we have a critical unassigned bug which is still happening?

I think this is one of the symptoms for the delete race bug, so it may occur again if the yet-to-be-fixed race starts being triggered again.

Marking as committed. I have not seen this error since Zanes delete lock fix landed https://review.openstack.org/#/c/99562/

A race still exists, which is captured by bug #1328983