Ubuntu
maas package

The celery cluster worker is not properly stopped

Bug #1059453 reported by Raphaël Badin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Jeroen T. Vermeulen
maas (Ubuntu)
Fix Released
Critical
Unassigned

Bug Description

The celery cluster is started in upstream:src/provisioningserver/start_cluster_controller.py, called from the upstart script packaging:debian/maas-cluster-controller.maas-cluster-celery.upstart. When all the MAAS' services are stopped (for instance when all the MAAS packages are purged), the celery workers are not properly stopped.

How to recreate:
- install the MAAS package
- run `ps aux | grep celeryd` to get the list of all the celeryd services. After the synchronization phase, you should see the region workers and the cluster workers (http://paste.ubuntu.com/1253472/ the region workers listen to 'celery' and 'master' while the cluster workers listen to a queue named after a UUID)
- remove all the packages with 'sudo apt-get purge maas*'
- run `ps aux | grep celeryd`: the cluster workers are still running.

Related branches

lp:~jtv/maas/bug-1059453
Gavin Panella (community): Approve
Diff: 145 lines (+27/-40)
3 files modified
src/provisioningserver/start_cluster_controller.py (+11/-8)
src/provisioningserver/tests/test_start_cluster_controller.py (+15/-31)
src/provisioningserver/utils.py (+1/-1)
Raphaël Badin (rvb)
affects: maas → maas (Ubuntu)
Revision history for this message
Julian Edwards (julian-edwards) wrote :

I suspect there's a "expect fork" missing in the upstart conf

Changed in maas:
status: New → Triaged
importance: Undecided → Critical
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

Update for the record. As we have since found out, it's much worse than just needing an "expect fork." Somewhere along the line, before it gets around to running the start-cluster-controller code proper, maas-provision does a whole bunch of other forks. The question is whether we can make upstart track the right child process.

Revision history for this message
Julian Edwards (julian-edwards) wrote : Re: [Bug 1059453] Re: The celery cluster worker is not properly stopped

On Wednesday 03 October 2012 02:47:05 you wrote:
> Update for the record. As we have since found out, it's much worse than
> just needing an "expect fork." Somewhere along the line, before it gets
> around to running the start-cluster-controller code proper, maas-
> provision does a whole bunch of other forks. The question is whether we
> can make upstart track the right child process.

The answer to fix this is to make a completely new wrapper script outside of
maas-provision, I think. There's some weirdness going on in it.

Jeroen T. Vermeulen (jtv)
Changed in maas:
assignee: nobody → Jeroen T. Vermeulen (jtv)
status: Triaged → In Progress
MAAS Lander (maas-lander)
Changed in maas:
status: In Progress → Fix Committed
Julian Edwards (julian-edwards)
Changed in maas:
status: Fix Committed → Fix Released
Andres Rodriguez (andreserl)
Changed in maas (Ubuntu):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.