Fix magnum's gate, oslo.messaging broke it

Per IRC discussion, oslo.messaging===5.24.2 is buggy. We can avoid 5.24.2 and 5.25.0 until the problem is fixed.

Fix proposed to branch: master
Review: https://review.openstack.org/469518

please submit this to global-requirements.txt (in the requirements repo) instead so it can filter out to all projects.

There is no indication in here about what went into 5.24.2 that broke Magnum - has anyone sorted that out?

The description says https://review.openstack.org/#/c/463673/ but that went into 5.25.0, not 5.24.2, so why is 5.24.2 also bad - or isn't it?

Also note that oslo.messaging change was backported and merged to stable/ocata and proposed to stable/newton, but I put a -2 on the stable/newton change until this is sorted out:

https://review.openstack.org/#/q/I62b9e09513e3ebfebc64a941d4b21b6c053b511d,n,z

5.24.2 doesn't break magnum. Per IRC discussion (with sileht) 5.24.2 is considered buggy. Why 5.24.2 is bad, I can not answer. Magnum works fine with 5.24.2.

OK thanks for clarifying.

I did some troubleshooting on this, it looks the issue occurs if you are using oslo.messaging 5.25.0 and the 'blocking' executor. Change the executor to 'eventlet' will resolve the problem. I commented on the patch for details: https://review.openstack.org/#/c/463673/

The default executor of o.m is 'blocking', but after change [1],
o.m with blocking is broken. 'eventlet' and 'non-blocking' work
fine and most services use 'eventlet', so this is why it was
noticed by magnum which used the default one.

Magnum uses eventlet in one place [2], but wrongly blocking in another [3].

We must change the executor in magnum to be eventlet regardless of this bug, since it recommended by the oslo team and the 'blocking' executor has 0% test coverage.

Oslo side, the default behavior of the library *must* be functional. So, I suggest [4] to blacklist until 5.25.1 is out.

[1] https://review.openstack.org/#/c/463673/
[2] http://git.openstack.org/cgit/openstack/magnum/tree/magnum/common/rpc.py#n166
[3] http://git.openstack.org/cgit/openstack/magnum/tree/magnum/common/rpc_service.py#n52
[4] https://review.openstack.org/#/c/469539/

I assume that a 5.25.1 will be released to fix the blocking executor.

Fix proposed to branch: master
Review: https://review.openstack.org/469852

Change abandoned by Spyros Trigazis (strigazi) (<email address hidden>) on branch: master
Review: https://review.openstack.org/469518
Reason: In favor of https://review.openstack.org/#/c/469852/

Reviewed: https://review.openstack.org/469852
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=bd69b3fff6b50048813360627fb02044cf1bcb58
Submitter: Jenkins
Branch: master

commit bd69b3fff6b50048813360627fb02044cf1bcb58
Author: Spyros Trigazis (strigazi) <email address hidden>
Date: Thu Jun 1 12:20:26 2017 +0000

    Use eventlet executor in rpc_service

    The default executor of o.m is 'blocking', but after change [1],
    o.m with blocking is broken. 'eventlet' and 'non-blocking' work
    fine and most services use 'eventlet', so this is why it was
    noticed by magnum which used the default one.

    Use eventlet to not be affected by bug #1694728 , but more
    importantly, the oslo team suggests to not use blocking which
    has 0% test coverage and it is going to be deprecated and unset
    from the default configuration.

    [1] https://review.openstack.org/#/c/463673/

    Change-Id: I47da73787456c97f7d84fd4440404b551ff62528
    Closes-Bug: #1694728

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/470283

Reviewed: https://review.openstack.org/470283
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=a4203b79d77ceee3d884fbb376b1e32dfc54530f
Submitter: Jenkins
Branch: stable/ocata

commit a4203b79d77ceee3d884fbb376b1e32dfc54530f
Author: Matt Riedemann <email address hidden>
Date: Fri Jun 2 13:19:28 2017 +0000

    Revert "rabbit: restore synchronous ack/requeue"

    This reverts commit b3316669263ad5f76e03bb7b54f1704f64c8c17f.

    It was reported on master (pike) that this change broke
    the default "blocking" executor so we should revert it on
    stable/ocata.

    Change-Id: Ia4bd74aa3df059e00b209d66afa8e327b76fe6ca
    Related-Bug: #1694728

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/470759

Change abandoned by Mehdi Abaakouk (sileht) (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/470759

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/470762

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/471275

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/471276

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/471277

Reviewed: https://review.openstack.org/471277
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=a9c642af0ce75c75a8d1439d63e2c803eaedcc2c
Submitter: Jenkins
Branch: stable/mitaka

commit a9c642af0ce75c75a8d1439d63e2c803eaedcc2c
Author: Spyros Trigazis (strigazi) <email address hidden>
Date: Thu Jun 1 12:20:26 2017 +0000

    Use eventlet executor in rpc_service

    The default executor of o.m is 'blocking', but after change [1],
    o.m with blocking is broken. 'eventlet' and 'non-blocking' work
    fine and most services use 'eventlet', so this is why it was
    noticed by magnum which used the default one.

    Use eventlet to not be affected by bug #1694728 , but more
    importantly, the oslo team suggests to not use blocking which
    has 0% test coverage and it is going to be deprecated and unset
    from the default configuration.

    [1] https://review.openstack.org/#/c/463673/

    Change-Id: I47da73787456c97f7d84fd4440404b551ff62528
    Closes-Bug: #1694728
    (cherry picked from commit bd69b3fff6b50048813360627fb02044cf1bcb58)

Change abandoned by Mehdi Abaakouk (sileht) (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/470762
Reason: I will restore it when master is merged

Fix proposed to branch: master
Review: https://review.openstack.org/472233

Change abandoned by Mehdi Abaakouk (sileht) (<email address hidden>) on branch: master
Review: https://review.openstack.org/472233

Reviewed: https://review.openstack.org/469806
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=8ee5ae135a6ecb918f40619982e3dc7e38ed0bbf
Submitter: Jenkins
Branch: master

commit 8ee5ae135a6ecb918f40619982e3dc7e38ed0bbf
Author: Mehdi Abaakouk <email address hidden>
Date: Thu Jun 1 10:28:23 2017 +0200

    Fix rabbitmq driver with blocking executor

    We recently move ack/requeue of messages in main/polling thread
    of rabbitmq drivers. And break the blocking executor.

    This one is not tested by any tests and now deprecated.

    This change workaround the issue until we completely remove the
    blocking executor.

    Change-Id: Id479100f6ff364cf67a199e9b70f9f0c7bf7e1a9
    Closes-bug: #1694728

This issue was fixed in the openstack/oslo.messaging 5.27.0 release.

Change abandoned by Mehdi Abaakouk (sileht) (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/466787

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/474202

Change abandoned by Mehdi Abaakouk (sileht) (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/470762
Reason: Sorry I have missed that on I remove it.

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/474455

Change abandoned by Mehdi Abaakouk (sileht) (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/474202
Reason: The one with the new Change-ID: https://review.openstack.org/#/c/474455/

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/474456

Change abandoned by Mehdi Abaakouk (sileht) (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/470759
Reason: This change have a new changeID: https://review.openstack.org/474456

Change abandoned by Mehdi Abaakouk (sileht) (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/474456
Reason: Thanks a lot for catching this. A good reason for me to stop provide backport in advance...

Also changing all the Change-ID have created some mess, git-review refuses to update this patch. And it force me to update the old one: https://review.openstack.org/466787

So I have restored it and abandon this one.

Reviewed: https://review.openstack.org/474455
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=2775568e01b8429dc90fade4e38673085098da34
Submitter: Jenkins
Branch: stable/ocata

commit 2775568e01b8429dc90fade4e38673085098da34
Author: Mehdi Abaakouk <email address hidden>
Date: Wed May 10 09:19:38 2017 +0200

    rabbit: restore synchronous ack/requeue

    Note this change also contains the fix for the regression it
    introduced.

    In https://review.openstack.org/#/c/436958, we fix a thread safety
    issue. But we make the ack/requeue of message asynchronous. In nominal
    case, it works, but if network/rabbit connection issue occurs this
    can result to rpc call handle twice. By chance we double check already
    processed message ids, and drop duplicates, but that if the message
    goes to another node, the mitigation won't work.

    This restore the previous behavior, to ensure we run application
    callback of rpc.call/rpc.cast only when the message have been
    successfully ack.

    (cherry picked from commit da02bc2169b09959d857c605961ead1bbba1019d)

    Fix rabbitmq driver with blocking executor

    We recently move ack/requeue of messages in main/polling thread
    of rabbitmq drivers. And break the blocking executor.

    This one is not tested by any tests and now deprecated.

    This change workaround the issue until we completely remove the
    blocking executor.

    Closes-bug: #1694728
    (cherry picked from commit 8ee5ae135a6ecb918f40619982e3dc7e38ed0bbf)

    Change-Id: I62b9e09513e3ebfebc64a941d4b21b6c053b511d

Change abandoned by Spyros Trigazis (strigazi) (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/471275
Reason: Abandon, since o.m blocking is fixed

Change abandoned by Spyros Trigazis (strigazi) (<email address hidden>) on branch: stable/newton
Review: https://review.openstack.org/471276
Reason: Abandon, since o.m blocking is fixed

This issue was fixed in the openstack/oslo.messaging 5.17.2 release.

Reviewed: https://review.openstack.org/466787
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=c41f0ce709ea39717eb75748a6c5b7e9a69628cf
Submitter: Jenkins
Branch: stable/newton

commit c41f0ce709ea39717eb75748a6c5b7e9a69628cf
Author: Mehdi Abaakouk <email address hidden>
Date: Wed May 10 09:19:38 2017 +0200

    rabbit: restore synchronous ack/requeue

    Note this change also contains the fix for the regression it
    introduced.

    In https://review.openstack.org/#/c/436958, we fix a thread safety
    issue. But we make the ack/requeue of message asynchronous. In nominal
    case, it works, but if network/rabbit connection issue occurs this
    can result to rpc call handle twice. By chance we double check already
    processed message ids, and drop duplicates, but that if the message
    goes to another node, the mitigation won't work.

    This restore the previous behavior, to ensure we run application
    callback of rpc.call/rpc.cast only when the message have been
    successfully ack.

    (cherry picked from commit da02bc2169b09959d857c605961ead1bbba1019d)

    Fix rabbitmq driver with blocking executor

    We recently move ack/requeue of messages in main/polling thread
    of rabbitmq drivers. And break the blocking executor.

    This one is not tested by any tests and now deprecated.

    This change workaround the issue until we completely remove the
    blocking executor.

    Closes-bug: #1694728
    (cherry picked from commit 8ee5ae135a6ecb918f40619982e3dc7e38ed0bbf)

    Change-Id: I62b9e09513e3ebfebc64a941d4b21b6c053b511d

This issue was fixed in the openstack/oslo.messaging 5.10.2 release.

This issue was fixed in the openstack/magnum 5.0.0 release.