Merge lp:~adeuring/lazr.jobrunner/slow-lane into lp:lazr.jobrunner

[Abel Deuring]

This branch implements a "fast lane/slow lane" mechanism in lazr.jobrunner.

The core idea is simple: If the configuration of a celeryd instance
has the parameter FALLBACK_QUEUE defined, a job that is aborted
by a SoftTimeLimitExceeded exception, it is put again into the
queue specified by FALLBACK_QUEUE.

Working on this change, I noticed that celery tasks which time
out leave a message in the result queue, if task.ignore_result == False.
I tried to consume this message via

    try:
        result.wait()
    except SoftTimeLimitExceeded:
        result.wait()

but this did not work, so the timeout related tests (test_timeout_long,
test_timeout_in_fast_lane_passes_in_slow_lane,
test_timeout_in_fast_lane_and_slow_lane) now use another Task, where
ignore_result is True. This requires a variant of the method
run_file_job(), which does not try to call result.wait(). The new
method run_file_job_ignore_result() wait instead for some time until
stops the celery instance.

To keep the AQMP server "clean", I added a check of the queues of the
local rabbitmq server in the methods setUp() and tearDown() of TestCeleryD.
It seems to me that AMQP does not have any mechanism to get a list
of all current queues from a server, so I used the rabbitmq webservice API
instead.

[Aaron Bentley (abentley)]

I suggested using "enqueue" rather than "queue" as the function name, but this is part of the Job API in LP, so that didn't make sense. I suggested using apply_async(ignore_result=True) instead of RunFileJobNoResult, but that was not effective. I asked why runJob was accepting a callback rather than using the reQueue method directly, and abel pointed out that reQueue is a Task method, and JobRunner doesn't know about tasks.

review: Approve

[lp:~adeuring/lazr.jobrunner/slow-lane]

28. By Abel Deuring on 2012-04-11

type fixed

29. By Abel Deuring on 2012-04-11

make checks of the RabbitMQ queue status optional