Merge lp:~corey.bryant/ubuntu/trusty/oslo.messaging/lp1318721 into lp:ubuntu/trusty-updates/oslo.messaging

[Corey Bryant]

[James Page (james-page)]

Uploaded to trusty proposed for SRU team review.

review: Approve

[lp:~corey.bryant/ubuntu/trusty/oslo.messaging/lp1318721]

10. By Corey Bryant on 2016-01-21

Bump to 1.4

=== modified file 'debian/changelog'
--- debian/changelog	2015-06-25 09:59:42 +0000
+++ debian/changelog	2016-01-21 15:09:11 +0000
@@ -1,3 +1,11 @@
+oslo.messaging (1.3.0-0ubuntu1.4) trusty; urgency=medium
+
+  * Backport upstream fix (LP: #1318721):
+    - d/p/fix-reconnect-race-condition-with-rabbitmq-cluster.patch:
+      Redeclare if exception is catched after self.queue.declare() failed.
+
+ -- Hui Xiang <[e-mail]>  Thu, 17 Dec 2015 16:22:35 +0800
+
 oslo.messaging (1.3.0-0ubuntu1.2) trusty; urgency=medium
 
   * Detect when underlying kombu connection to rabbitmq server has been

=== added file 'debian/patches/fix-reconnect-race-condition-with-rabbitmq-cluster.patch'
--- debian/patches/fix-reconnect-race-condition-with-rabbitmq-cluster.patch	1970-01-01 00:00:00 +0000
+++ debian/patches/fix-reconnect-race-condition-with-rabbitmq-cluster.patch	2016-01-21 15:09:11 +0000
@@ -0,0 +1,75 @@
+Description: Fix reconnect race condition with RabbitMQ cluster
+
+    commit 7ad0d7eaf9cb095a14b07a08c814d9f1f9c8ff12
+    Author: Jens Rosenboom <[e-mail]>
+    Date: Fri Jun 27 16:46:47 2014 +0200
+
+    Retry Queue creation to workaround race condition
+    that may happen when both the client and broker race over
+    exchange creation and deletion respectively which happen only
+    when the Queue/Exchange were created with auto-delete flag.
+
+    Queues/Exchange declared with auto-delete instruct the Broker to
+    delete the Queue when the last Consumer disconnect from it, and
+    the Exchange when the last Queue is deleted from this Exchange.
+
+    Now in a RabbitMQ cluster setup, if the cluster node that we are
+    connected to go down, 2 things will happen:
+
+     1. From RabbitMQ side, the Queues w/ auto-delete will be deleted
+        from the other cluster nodes and then the Exchanges that the
+        Queues are bind to if they were also created w/ auto-delete.
+     2. From client side, client will reconnect to another cluster
+        node and call queue.declare() which create Exchanges then
+        Queues then Binding in that order.
+
+    Now in a happy path the queues/exchanges will be deleted from the
+    broker before client start re-creating them again, but it also
+    possible that the client first start by creating queues/exchange
+    as part of the queue.declare() call, which are no-op operations
+    b/c they alreay existed, but before it could bind Queue to
+    Exchange, RabbitMQ nodes just received the 'signal' that the
+    queue doesn't have any consumer so it should be delete, and the
+    same with exchanges, which will lead to binding fail with
+    NotFound error.
+
+    Illustration of the time line from Client and RabbitMQ cluster
+    respectively when the race condition happen:
+
+
+           e-declare(E) q-declare(Q) q-bind(Q, E)
+         -----+------------------+----------------+----------->
+                                   e-delete(E)
+         ------------------------------+---------------------->
+
+    Change-Id: Ideb73af6f246a8282780cdb204d675d5d4555bf0
+    Closes-Bug: #1318721
+
+Author: Jens Rosenboom <[e-mail]>
+Origin: backport, https://review.openstack.org/#/c/103157/
+Bug: https://bugs.launchpad.net/neutron/+bug/1318721
+
+--- a/oslo/messaging/_drivers/impl_rabbit.py
++++ b/oslo/messaging/_drivers/impl_rabbit.py
+@@ -159,7 +159,20 @@
+         self.channel = channel
+         self.kwargs['channel'] = channel
+         self.queue = kombu.entity.Queue(**self.kwargs)
+-        self.queue.declare()
++        try:
++            self.queue.declare()
++        except Exception as e:
++            # NOTE: This exception may be triggered by a race condition.
++            # Simply retrying will solve the error most of the time and
++            # should work well enough as a workaround until the race condition
++            # itself can be fixed.
++            # TODO(jrosenboom): In order to be able to match the Execption
++            # more specifically, we have to refactor ConsumerBase to use
++            # 'channel_errors' of the kombu connection object that
++            # has created the channel.
++            # See https://bugs.launchpad.net/neutron/+bug/1318721 for details.
++            LOG.exception(_("Declaring queue failed with (%s), retrying"), e)
++            self.queue.declare()
+ 
+     def _callback_handler(self, message, callback):
+         """Call callback with deserialized message.

=== modified file 'debian/patches/series'
--- debian/patches/series	2015-06-25 09:59:42 +0000
+++ debian/patches/series	2016-01-21 15:09:11 +0000
@@ -3,3 +3,4 @@
 0002-rabbit-fix-timeout-timer-when-duration-is-None.patch
 0003-Declare-DirectPublisher-exchanges-with-passive-True.patch
 redeclare-consumers-when-ack-requeue-fails.patch
+fix-reconnect-race-condition-with-rabbitmq-cluster.patch