MAAS

Code review comment for lp:~allenap/maas/celery-in-dns-removal

celery-in-dns-removal
Merge into trunk

Revision history for this message

Julian Edwards (julian-edwards) wrote on 2014-11-18:

On Tuesday 18 Nov 2014 10:48:21 you wrote:
> > Just a thought, but if each region controller physical host runs a
> > bind daemon then a per-host lock will suffice to duplicate the jobs to
> > the right places.
>
> It will, but a web API request to, say, provision a node arrives at only
> one host (different each time maybe, but only one host sees each
> individual request), and that's when we trigger updates to DNS. The
> region controller that services that request needs to be able to tell at
> least one region controller on each other host in the region to update
> BIND's configuration and zone files.

Right, that's what I mean by a per-host lock. We just create a job on every
host.

>
> > This has the advantage of making our DNS HA.
>
> We have a "mesh" so that every region controller can talk to every
> cluster controller, and we could extend that so that every region
> controller can talk to every other region controller, and thus address
> the problem above.

We could also do PG triggers.

>
> Alternative #1: we could set up BIND to be HA on its own. Its config
> would be updated by a periodic job, say, but zone data updates would
> come via nsupdate. BIND would then sort out distributing those changes
> as required.

We would still have the problem of deciding from which appserver we send
updates / amend the zone file. But it would be nice to use its own
replication!

>
> Alternative #2: we create an independent BIND management service. The
> region calls out to it when a change is needed and it then encapsulates
> configuration changes, zone data changes, and HA, and the region doesn't
> need to know.
>
> Personally I think I like #2 most. I suspect it might also be the least
> work. Meshing together all the region controllers was originally my
> favourite, but I see many possibilities for deadlock there.

It's an idea. We should put something together to discuss all of these
options.

I suspect the trigger thing might be the least work but I've not thought hard
about it yet.

On Tuesday 18 Nov 2014 10:48:21 you wrote:
> > Just a thought, but if each region controller physical host runs a
> > bind daemon then a per-host lock will suffice to duplicate the jobs to
> > the right places.
> 
> It will, but a web API request to, say, provision a node arrives at only
> one host (different each time maybe, but only one host sees each
> individual request), and that's when we trigger updates to DNS. The
> region controller that services that request needs to be able to tell at
> least one region controller on each other host in the region to update
> BIND's configuration and zone files.

Right, that's what I mean by a per-host lock.  We just create a job on every 
host.

> 
> > This has the advantage of making our DNS HA.
> 
> We have a "mesh" so that every region controller can talk to every
> cluster controller, and we could extend that so that every region
> controller can talk to every other region controller, and thus address
> the problem above.

We could also do PG triggers.

> 
> Alternative #1: we could set up BIND to be HA on its own. Its config
> would be updated by a periodic job, say, but zone data updates would
> come via nsupdate. BIND would then sort out distributing those changes
> as required.

We would still have the problem of deciding from which appserver we send 
updates / amend the zone file.  But it would be nice to use its own 
replication!

> 
> Alternative #2: we create an independent BIND management service. The
> region calls out to it when a change is needed and it then encapsulates
> configuration changes, zone data changes, and HA, and the region doesn't
> need to know.
> 
> Personally I think I like #2 most. I suspect it might also be the least
> work. Meshing together all the region controllers was originally my
> favourite, but I see many possibilities for deadlock there.

It's an idea.  We should put something together to discuss all of these 
options.

I suspect the trigger thing might be the least work but I've not thought hard 
about it yet.

« Back to merge proposal