Merge lp:~cprov/adt-result-checker/qos into lp:adt-result-checker

[Celso Providelo]

Set a fair QoS to our multi-worker environment (prefetch_count=1)

Set a fair QoS to our multi-worker environment (prefetch_count=1)

[Paul Larson (pwlars)]

I think the default is prefetch_count=0 right? Is the performance gain of setting this to 1 really helpful for us? I'm not super familiar with the impacts of various qos options in rabbit, so I'm mostly just curious that this is not going to impact us in some other way. Otherwise I'm fine with this.

review: Approve

[Celso Providelo (cprov)]

Paul,

prefetch_count=0 (along with default prefetch_size=0) means the consumer is telling rabbit to "reserve as many messages as you can for me", which is bad for our multi-worker environment, because workers will compete to reserve messages, but will take quite a long time to process (acknowledge)them. Other workers will starve and be idle most of the time.

prefetch_count=1, means that only one single message will be given to workers at time. So until the test is done, other messages in the queue will be available to other workers to pick them up.

See a more appropriate description at https://www.rabbitmq.com/consumer-prefetch.html

=== modified file 'adt_result_checker/__init__.py'
--- adt_result_checker/__init__.py	2015-03-20 02:23:17 +0000
+++ adt_result_checker/__init__.py	2015-03-20 19:18:06 +0000
@@ -54,6 +54,10 @@
 
     def get_consumers(self, Consumer, channel):
         """Return consumers instances for all configured queues."""
+        # Set a fair QoS for a multi-worker environment with
+        # late-acknowledgement. Each worker should only be fed a single
+        # message at time (prefetch_count).
+        channel.basic_qos(prefetch_size=0, prefetch_count=1, a_global=False)
         queues = [kombu.Queue('adt.results.{}'.format(API_VERSION))]
         return [Consumer(queues=queues, callbacks=[self.process])]