MAAS

Changing tag definition quickly could result in inconsistent data

Bug #1065454 reported by John A Meinel on 2012-10-11

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	MAAS	Fix Released	Low	Unassigned
	1.2	Fix Released	High	Julian Edwards	MAAS 12.10-stabilization

Bug Description

Because we might have multiple workers for one nodegroup, and rabbit doesn't ensure FIFO anyway, it is possible to change a tag definition quick enough that you have 2 workers working on the same nodes, but different definitions concurrently. With multiple nodegroups, this concurrency gets even worse.

Essentially there is a race condition, that whichever worker finishes first, will get their node <=> tag mapping overwritten by the one that finishes last. In practice, the last definition will start later, and will likely win.

A possible way to guarantee the last definition wins is to have the workers pass back the tag definition that they used for the processing. If it matches the current value in the database, accept it, otherwise reject or ignore it.

Related branches

lp:~jameinel/maas/tag-race-1065454

Merged into lp:~maas-committers/maas/trunk at revision 1302

Martin Packman (community): Approve on 2012-10-25

lp:~julian-edwards/maas/backport-r1302

Merged into lp:maas/1.2 at revision 1329

Julian Edwards (community): Approve on 2012-12-04

John A Meinel (jameinel) on 2012-10-11

summary:

- Changing tag definition quickly can result in inconsistent data
+ Changing tag definition quickly could result in inconsistent data

Revision history for this message

Julian Edwards (julian-edwards) wrote on 2012-10-17:

> Because we might have multiple workers for one nodegroup

This should never be the case, the architecture is explicit about one worker per nodegroup.

Does this affect your bug report?

Changed in maas:
status:	Triaged → Incomplete

Revision history for this message

John A Meinel (jameinel) wrote on 2012-10-17: Re: [Bug 1065454] Re: Changing tag definition quickly could result in inconsistent data

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 10/17/2012 5:24 AM, Julian Edwards wrote:
>> Because we might have multiple workers for one nodegroup
>
> This should never be the case, the architecture is explicit about
> one worker per nodegroup.
>
> Does this affect your bug report?
>
> ** Changed in: maas Status: Triaged => Incomplete
>

When doing 'apt-get install maas-cluster-controller' I get a celeryd
running with 'concurrency=2'. And I can show that if I issue multiple
'maascli M tag TAG rebuild' it will run 2 of them in parallel.

So while the architecture is 1 cluster controller, it does have more
than one worker.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (Cygwin)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iEYEARECAAYFAlB+jfoACgkQJdeBCYSNAAPJowCcD92k7/SPzMQmqHR1hhIyuLK4
xE8AoMfxyPQv9ey0O/XgM5jF3m9I9ZE/
=FAkT
-----END PGP SIGNATURE-----

Revision history for this message

Julian Edwards (julian-edwards) wrote on 2012-10-18:

> When doing 'apt-get install maas-cluster-controller' I get a celeryd
> running with 'concurrency=2'. And I can show that if I issue multiple
> 'maascli M tag TAG rebuild' it will run 2 of them in parallel.
>
> So while the architecture is 1 cluster controller, it does have more
> than one worker.
>
> John

Ah I see - your terminology confused me! There is only one celery "worker"
but it gets split into threads unless you tell it not to, so I see your
problem now.

Cheers.

Changed in maas:
status:	Incomplete → Triaged

MAAS Lander (maas-lander) on 2012-10-25

Changed in maas:
status:	Triaged → Fix Committed

Julian Edwards (julian-edwards) on 2012-12-04

Changed in maas:
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.