rabbit: drain_events is sometimes called with None

Bug #1408370 reported by Mehdi Abaakouk
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.messaging
Fix Released
High
Mehdi Abaakouk
oslo.messaging (Ubuntu)
Fix Released
Undecided
Unassigned
Trusty
Fix Released
High
James Page
Utopic
Fix Committed
Undecided
Unassigned

Bug Description

Hi,

In rabbit driver, kombu.connection.drain_events is sometimes called with None it should be a maximun of 1.

Cheers,

---- ---- ---- ---- ----

[Impact]

 * This patch along with those from LP #1400268 and LP #1338732 fixes rabbitmq reconnects

 * We are backporting this to Icehouse since oslo.messaging 1.3.0
   fails to reconnect to Rabbit properly, particularly nova-compute.

 * This patch alond with it's dependencies metnioend above, will ensure that
   multiple reconnect attempts happen by having connections timout and retry.

[Test Case]

 * Start a service that uses oslo.messaging with rabbitmq e.g. nova-compute

 * Stop rabbitmq while tail-F /var/log/nova/nova-compute.log

 * Observe that nova-compute amqp times out and it is trying to reconnect

 * Restart rabbitmq

 * Observe that rabbitmq connection has re-established

[Regression Potential]

 * None. I have tested in my local cloud environment and it appears to be
   reliable.

Mehdi Abaakouk (sileht)
Changed in oslo.messaging:
importance: Undecided → High
Changed in oslo.messaging:
assignee: nobody → Mehdi Abaakouk (sileht)
status: New → In Progress
Mehdi Abaakouk (sileht)
Changed in oslo.messaging:
milestone: none → next-kilo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.messaging (master)

Reviewed: https://review.openstack.org/145521
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=44132d4344902f98007e6e58ea3bee56c701b400
Submitter: Jenkins
Branch: master

commit 44132d4344902f98007e6e58ea3bee56c701b400
Author: Mehdi Abaakouk <email address hidden>
Date: Wed Jan 7 15:49:54 2015 +0100

    rabbit: fix timeout timer when duration is None

    When the duration of the timeout timer used in the rabbit driver.is
    None and we want that the timer return a maximum of N secs it return None
    (infinite) instead of N.

    This change fixes that.

    Closes-bug: #1408370

    Change-Id: I7f4cb3075f776c63aa7dc497173677f92b68c16d

Changed in oslo.messaging:
status: In Progress → Fix Committed
Changed in oslo.messaging:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.messaging (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/174457

description: updated
Mehdi Abaakouk (sileht)
tags: added: icehouse-backport-potential
Revision history for this message
Edward Hope-Morley (hopem) wrote :
James Page (james-page)
Changed in oslo.messaging (Ubuntu):
status: New → Fix Released
Changed in oslo.messaging (Ubuntu Trusty):
status: New → Triaged
importance: Undecided → High
James Page (james-page)
Changed in oslo.messaging (Ubuntu Trusty):
status: Triaged → In Progress
assignee: nobody → James Page (james-page)
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Mehdi, or anyone else affected,

Accepted oslo.messaging into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/oslo.messaging/1.3.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in oslo.messaging (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Ante Karamatić (ivoks) wrote :

We used 1.3.0-0ubuntu1.1 and we confirm it solves the problem.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package oslo.messaging - 1.3.0-0ubuntu1.1

---------------
oslo.messaging (1.3.0-0ubuntu1.1) trusty; urgency=medium

  * Backport fixes for reliable AMQP reconnect support, ensuring
    nova-compute instances re-connect and message correctly when
    RabbitMQ message brokers disappear is clustered configurations:
    - d/p/0001-rabbit-more-precise-iterconsume-timeout.patch:
      Improve precision of iterconsume timeouts (LP: #1400268).
    - d/p/0002-rabbit-fix-timeout-timer-when-duration-is-None.patch:
      Fix timeout timer when duration is set to None (LP: #1408370).
    - d/p/0003-Declare-DirectPublisher-exchanges-with-passive-True.patch:
      Ensure that message publishers fail and retry if the consumer has
      not yet declared a receiving queue (LP: #1338732).
 -- Edward Hope-Morley <email address hidden> Thu, 23 Apr 2015 15:56:08 +0100

Changed in oslo.messaging (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for oslo.messaging has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.messaging (stable/icehouse)

Reviewed: https://review.openstack.org/174457
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=106dd46101f536359df984394b5994f911df4912
Submitter: Jenkins
Branch: stable/icehouse

commit 106dd46101f536359df984394b5994f911df4912
Author: Mehdi Abaakouk <email address hidden>
Date: Wed Jan 7 15:49:54 2015 +0100

    rabbit: fix timeout timer when duration is None

    When the duration of the timeout timer used in the rabbit driver.is
    None and we want that the timer return a maximum of N secs it return None
    (infinite) instead of N.

    This change fixes that.

    Closes-bug: #1408370

    (cherry picked from commit 44132d4344902f98007e6e58ea3bee56c701b400)
    Change-Id: I7f4cb3075f776c63aa7dc497173677f92b68c16d

tags: added: in-stable-icehouse
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Mehdi, or anyone else affected,

Accepted oslo.messaging into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/oslo.messaging/1.4.1-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in oslo.messaging (Ubuntu Utopic):
status: New → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Jian Wen (wenjianhn)
tags: removed: icehouse-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.