celery RabbitMQ queue slowly increasing on ackee

(I asked Haw to assign this to Aaron as he and Abel have been bringing
up the celery stack for LP).

The alert has been squelched for now, but if the growth rate were to
accelerate we'd run into rabbit memory issues - we've got a few days
grace I suspect.

Just got another alert - we're now at "CRITICALS: * in * has 945 messages"

Seems this will keep accumulating until we complete https://rt.admin.canonical.com//Ticket/Display.html?id=51940

Fixed in stable r15057 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15057>.

https://rt.admin.canonical.com//Ticket/Display.html?id=51940 has been completed. And ackee is on revno 15288

However the alert is at CRITICALS: * in * has 6585 messages still. What remains to be done?

What needs to be done depends on what queues still have messages.

It is the celery queue

#Vhost Name Messages_ready Messages_unacknowledged Messages Consumers Memory Time
/ celery 6585 0 6585 0 2914608 1337883002

It's not clear what to do, then. Since that queue is not being used, nothing will drain it. A full rabbit reset would clear all queues. We could perhaps write a custom script to purge the celery queue.

@abentley: "We could perhaps write a custom script to purge the celery queue." <- please do, as this should be part of a graceful cleanup in general (for deprecated "storage").

FTR downtimed +1wk in nagios.

If the queue isn't needed, you can just delete it via the rabbitctl UI.

If the queue *is* needed, then we need to have a consumer consuming from it ;)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 12-05-28 07:45 PM, Robert Collins wrote:
> If the queue isn't needed, you can just delete it via the rabbitctl
> UI.

Could you provide a specific invocation, please? I've looked, and I
couldn't find anything that would clear/delete one queue at a time.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk/EyuQACgkQ0F+nu1YWqI1GVQCdHZTorCnBYrhZsKLmk8kCz0T5
EuAAnRB6tydmdiuvICFC7f0WeCxZ443j
=1Mug
-----END PGP SIGNATURE-----

On Wed, May 30, 2012 at 1:11 AM, Aaron Bentley <email address hidden> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 12-05-28 07:45 PM, Robert Collins wrote:
>> If the queue isn't needed, you can just delete it via the rabbitctl
>> UI.
>
> Could you provide a specific invocation, please?  I've looked, and I
> couldn't find anything that would clear/delete one queue at a time.

Oh blah, you are right; I am thinking of functionality in the
management plugin CLI - http://www.rabbitmq.com/management-cli.html

We have, I believe, installed the management plugin, which will permit
this via both HTTP and the CLI tool it provides.

-Rob

I have purged and removed the celery queue from production and staging