Merge lp:~allenap/maas/celery-in-dns-removal into lp:~maas-committers/maas/trunk

[Gavin Panella]

Remove all remaining Celery dependencies.

Much of this is mechanical, including name changes for several
functions, and fair amount of code has been removed.

I changed names because my brain was aching trying to follow all the
calls. The API had grown ad-hoc and there was no consistency. There are
still some warts (there are code comments to explain). Anyway, the main
things I changed are:

- In the region the functions to update DNS settings/options and zones
  are all prefixed with "dns_".

- In the cluster the functions to update BIND configurations and write
  zones files are all prefixed with "bind_". These are lower-level than
  the dns_* functions.

There's no RPC between region and cluster, so the division is really
just a conceptual one, for now. As before, we only ever update a single
DNS server which runs on the same machine as the region controller.

To be clear: MAAS does not support DNS on multiple region controllers,
and never has, so don't worry that that's missing here. The division of
responsibility between the dns_* and bind_* functions lends itself to
solving this later.

[Jeroen T. Vermeulen (jtv)]

Brilliant, thanks! I have a few notes inline, but nothing serious. Some things may be worth polishing in later branches.

review: Approve

[Gavin Panella (allenap)]

Thank you for taking this on, I really appreciate it.

[Gavin Panella (allenap)]

This is now ready to land. After discussion with Andres, I'm holding on this because it doesn't really fit with the goals of 1.7.1. It's up to the folk in Austin this week to decide when to land it :)

[Julian Edwards (julian-edwards)]

On Saturday 15 Nov 2014 11:46:41 you wrote:
> To be clear: MAAS does not support DNS on multiple region controllers,
> and never has, so don't worry that that's missing here. The division of
> responsibility between the dns_* and bind_* functions lends itself to
> solving this later.

Just a thought, but if each region controller physical host runs a bind daemon
then a per-host lock will suffice to duplicate the jobs to the right places.

This has the advantage of making our DNS HA.

[Gavin Panella (allenap)]

> Just a thought, but if each region controller physical host runs a
> bind daemon then a per-host lock will suffice to duplicate the jobs to
> the right places.

It will, but a web API request to, say, provision a node arrives at only
one host (different each time maybe, but only one host sees each
individual request), and that's when we trigger updates to DNS. The
region controller that services that request needs to be able to tell at
least one region controller on each other host in the region to update
BIND's configuration and zone files.

> This has the advantage of making our DNS HA.

We have a "mesh" so that every region controller can talk to every
cluster controller, and we could extend that so that every region
controller can talk to every other region controller, and thus address
the problem above.

Alternative #1: we could set up BIND to be HA on its own. Its config
would be updated by a periodic job, say, but zone data updates would
come via nsupdate. BIND would then sort out distributing those changes
as required.

Alternative #2: we create an independent BIND management service. The
region calls out to it when a change is needed and it then encapsulates
configuration changes, zone data changes, and HA, and the region doesn't
need to know.

Personally I think I like #2 most. I suspect it might also be the least
work. Meshing together all the region controllers was originally my
favourite, but I see many possibilities for deadlock there.

[Julian Edwards (julian-edwards)]

On Tuesday 18 Nov 2014 10:48:21 you wrote:
> > Just a thought, but if each region controller physical host runs a
> > bind daemon then a per-host lock will suffice to duplicate the jobs to
> > the right places.
>
> It will, but a web API request to, say, provision a node arrives at only
> one host (different each time maybe, but only one host sees each
> individual request), and that's when we trigger updates to DNS. The
> region controller that services that request needs to be able to tell at
> least one region controller on each other host in the region to update
> BIND's configuration and zone files.

Right, that's what I mean by a per-host lock. We just create a job on every
host.

>
> > This has the advantage of making our DNS HA.
>
> We have a "mesh" so that every region controller can talk to every
> cluster controller, and we could extend that so that every region
> controller can talk to every other region controller, and thus address
> the problem above.

We could also do PG triggers.

>
> Alternative #1: we could set up BIND to be HA on its own. Its config
> would be updated by a periodic job, say, but zone data updates would
> come via nsupdate. BIND would then sort out distributing those changes
> as required.

We would still have the problem of deciding from which appserver we send
updates / amend the zone file. But it would be nice to use its own
replication!

>
> Alternative #2: we create an independent BIND management service. The
> region calls out to it when a change is needed and it then encapsulates
> configuration changes, zone data changes, and HA, and the region doesn't
> need to know.
>
> Personally I think I like #2 most. I suspect it might also be the least
> work. Meshing together all the region controllers was originally my
> favourite, but I see many possibilities for deadlock there.

It's an idea. We should put something together to discuss all of these
options.

I suspect the trigger thing might be the least work but I've not thought hard
about it yet.

[MAAS Lander (maas-lander)]

Voting does not meet specified criteria. Required: Approve >= 1, Disapprove == 0. Got: .

[Gavin Panella (allenap)]

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~allenap/maas/celery-in-dns-removal into lp:maas failed. Below is the output from the failed tests.

Ign http://security.ubuntu.com trusty-security InRelease
Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Ign http://nova.clouds.archive.ubuntu.com trusty-updates InRelease
Get:2 http://security.ubuntu.com trusty-security Release [62.0 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty Release.gpg
Hit http://nova.clouds.archive.ubuntu.com trusty-updates Release.gpg
Hit http://nova.clouds.archive.ubuntu.com trusty Release
Hit http://nova.clouds.archive.ubuntu.com trusty-updates Release
Get:3 http://security.ubuntu.com trusty-security/main Sources [58.3 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main Sources
Get:4 http://security.ubuntu.com trusty-security/universe Sources [17.4 kB]
Get:5 http://security.ubuntu.com trusty-security/main amd64 Packages [186 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Sources
Get:6 http://security.ubuntu.com trusty-security/universe amd64 Packages [81.9 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main Sources
Get:7 http://security.ubuntu.com trusty-security/main Translation-en [93.3 kB]
Get:8 http://security.ubuntu.com trusty-security/universe Translation-en [44.3 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe Sources
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en
Ign http://nova.clouds.archive.ubuntu.com trusty/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 544 kB in 2s (194 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
     --no-install-recommends install apache2 authbind bind9 bind9utils build-essential bzr-builddeb curl daemontools debhelper dh-apport distro-info dnsutils firefox freeipmi-tools gjs ipython isc-dhcp-common libjs-raphael libjs-yui3-full libjs-yui3-min libpq-dev make pep8 postgresql pyflakes python-amqplib python-bzrlib python-convoy python-crochet python-cssselect python-curtin python-dev python-distro-info python-django python-django-piston python-django-south python-djorm-ext-pgarray python-docutils python-extras python-fixtures python-flake8 python-formencode python-hivex python-httplib2 python-jinja2 python-jsonschema python-lockfile python-lxml python-mimeparse python-mock python-netaddr python-netifaces python-nose python-oauth python-oops python-oops-amqp python-oops-datedir-repo python-oops-twisted python-oops-wsgi python-openssl python-paramiko python-pexpect python-pip python-pocket...

[MAAS Lander (maas-lander)]

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

=== modified file 'docs/development/lease-scanning-and-dns.rst'
--- docs/development/lease-scanning-and-dns.rst	2014-10-02 11:53:59 +0000
+++ docs/development/lease-scanning-and-dns.rst	2015-01-13 18:36:01 +0000
@@ -26,15 +26,11 @@
 stores the active leases in the DHCPLease table.
 
 
-Updating the DNS zone file
-==========================
-
-If a new lease is found then the dns.change_dns_zones() function gets called
-which invokes two tasks::
-
- #. ``write_dns_zone_config()``
- #. ``rndc_command()``
-
-The first is responsible for writing out a new zone file with the appropriate
-sequence number and timestamp, and then the second is chained on to that
-and sends an rndc message to the DNS server to reload the zone.
+Updating one or more DNS zone files
+===================================
+
+If a new lease is found then the dns.dns_update_zones() function gets called
+which takes two steps::
+
+ #. Write out updated zone files.
+ #. Ask BIND to reload the zone.

=== modified file 'docs/development/tagging.rst'
--- docs/development/tagging.rst	2014-08-16 10:34:55 +0000
+++ docs/development/tagging.rst	2015-01-13 18:36:01 +0000
@@ -16,6 +16,11 @@
 New or updated tag definition
 -----------------------------
 
+**Note** that this is somewhat outdated. See `bug 1372544`_ (*Tag changes will
+never be evaluated on unconnected clusters*) for more information.
+
+.. _bug 1372544: https://bugs.launchpad.net/maas/+bug/1372544
+
 When a new tag is created or an existing tag is modified, its
 expression must be evaluated for every node known to the region. It's
 a moderately computationally intensive process, so the work is spread

=== modified file 'etc/maas/templates/dns/named.conf.options.inside.maas.template'
--- etc/maas/templates/dns/named.conf.options.inside.maas.template	2014-08-29 04:27:00 +0000
+++ etc/maas/templates/dns/named.conf.options.inside.maas.template	2015-01-13 18:36:01 +0000
@@ -1,10 +1,11 @@
 {{if upstream_dns}}
-    forwarders {
-        {{upstream_dns}};
-    };
+forwarders {
+{{for upstream_dns_address in upstream_dns}}
+    {{upstream_dns_address}};
+{{endfor}}
+};
 {{endif}}
 
 allow-query { any; };
 allow-recursion { trusted; };
 allow-query-cache { trusted; };
-

=== modified file 'etc/maas/templates/dns/named.conf.template'
--- etc/maas/templates/dns/named.conf.template	2014-08-29 04:27:00 +0000
+++ etc/maas/templates/dns/named.conf.template	2015-01-13 18:36:01 +0000
@@ -11,7 +11,9 @@
 # Access control for recursive queries.  See named.conf.options.inside.maas
 # for the directives used on this ACL.
 acl "trusted" {
-    {{trusted_networks}}
+{{for trusted_network in trusted_networks}}
+    {{trusted_network}};
+{{endfor}}
     localnets;
     localhost;
 };

=== modified file 'required-packages/base'
--- required-packages/base	2014-09-24 13:05:22 +0000
+++ required-packages/base	2015-01-13 18:36:01 +0000
@@ -13,7 +13,6 @@
 postgresql
 python-amqplib
 python-bzrlib
-python-celery
 python-convoy
 python-crochet
 python-curtin

=== removed file 'src/maas/celeryconfig.py'
--- src/maas/celeryconfig.py	2014-09-29 15:51:04 +0000
+++ src/maas/celeryconfig.py	1970-01-01 00:00:00 +0000
@@ -1,34 +0,0 @@
-# Copyright 2012-2014 Canonical Ltd.  This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-"""Celery settings for the region controller."""
-
-from __future__ import (
-    absolute_import,
-    print_function,
-    unicode_literals,
-)
-
-str = None
-
-__metaclass__ = type
-
-
-# Each cluster should have its own queue created automatically by Celery.
-CELERY_CREATE_MISSING_QUEUES = True
-
-CELERY_IMPORTS = (
-    # Tasks.
-    "provisioningserver.tasks",
-    )
-
-CELERY_ACKS_LATE = True
-
-# Do not store the tasks' return values (aka tombstones);
-# This improves performance.
-CELERY_IGNORE_RESULT = True
-
-# Don't queue, always run tasks immediately, and propagate exceptions back to
-# the caller. This eliminates the need for a queue.
-CELERY_ALWAYS_EAGER = True
-CELERY_EAGER_PROPAGATES_EXCEPTIONS = True

=== modified file 'src/maas/settings.py'
--- src/maas/settings.py	2014-11-10 21:21:47 +0000
+++ src/maas/settings.py	2015-01-13 18:36:01 +0000
@@ -309,9 +309,6 @@
     'maasjson': 'maasserver.json',
 }
 
-# Set up celery to use the production settings.
-os.environ['CELERY_CONFIG_MODULE'] = 'maas.celeryconfig'
-
 # Allow the user to override settings in maas_local_settings.
 import_local_settings()
 

=== modified file 'src/maasserver/dns/config.py'
--- src/maasserver/dns/config.py	2014-10-15 10:28:01 +0000
+++ src/maasserver/dns/config.py	2015-01-13 18:36:01 +0000
@@ -13,15 +13,11 @@
 
 __metaclass__ = type
 __all__ = [
-    'add_zone',
-    'change_dns_zones',
-    'get_trusted_networks',
-    'is_dns_enabled',
-    'next_zone_serial',
-    'write_full_dns_config',
+    'dns_add_zones',
+    'dns_update_all_zones',
+    'dns_update_zones',
     ]
 
-
 from django.conf import settings
 from maasserver import locks
 from maasserver.dns.zonegenerator import ZoneGenerator
@@ -35,7 +31,15 @@
     Sequence,
     )
 from maasserver.utils import synchronised
-from provisioningserver import tasks
+from provisioningserver.dns.actions import (
+    bind_reconfigure,
+    bind_reload,
+    bind_reload_with_retries,
+    bind_reload_zone,
+    bind_write_configuration,
+    bind_write_options,
+    bind_write_zones,
+    )
 from provisioningserver.logger import get_maas_logger
 
 
@@ -67,88 +71,111 @@
 
 
 @synchronised(locks.dns)
-def change_dns_zones(nodegroups):
-    """Update the zone configuration for the given list of Nodegroups.
-
-    :param nodegroups: The list of nodegroups (or the nodegroup) for which the
-        zone should be updated.
-    :type nodegroups: list (or :class:`NodeGroup`)
-    """
-    if not (is_dns_enabled() and is_dns_in_use()):
-        return
-    serial = next_zone_serial()
-    for zone in ZoneGenerator(nodegroups, serial):
-        maaslog.info("Generating new DNS zone file for %s", zone.zone_name)
-        zone_reload_subtask = tasks.rndc_command.subtask(
-            args=[['reload', zone.zone_name]])
-        tasks.write_dns_zone_config.delay(
-            zones=[zone], callback=zone_reload_subtask)
-
-
-@synchronised(locks.dns)
-def add_zone(nodegroup):
-    """Add to the DNS server a new zone for the given `nodegroup`.
-
-    To do this we have to write a new configuration file for the zone
-    and update the master config to include this new configuration.
-    These are done in turn by chaining Celery subtasks.
-
-    :param nodegroup: The nodegroup for which the zone should be added.
-    :type nodegroup: :class:`NodeGroup`
-    """
-    if not (is_dns_enabled() and is_dns_in_use()):
-        return
+def dns_add_zones(clusters):
+    """Add zone files for the given cluster(s), and serve them.
+
+    Serving these new zone files means updating BIND's configuration to
+    include them, then asking it to load the new configuration.
+
+    :param clusters: The clusters(s) for which zones should be served.
+    :type clusters: :py:class:`NodeGroup`, or an iterable thereof.
+    """
+    if not (is_dns_enabled() and is_dns_in_use()):
+        return
+
     zones_to_write = ZoneGenerator(
-        nodegroup, serial_generator=next_zone_serial
-        ).as_list()
+        clusters, serial_generator=next_zone_serial).as_list()
     if len(zones_to_write) == 0:
         return None
     serial = next_zone_serial()
     # Compute non-None zones.
     zones = ZoneGenerator(NodeGroup.objects.all(), serial).as_list()
-    reconfig_subtask = tasks.rndc_command.subtask(args=[['reconfig']])
-    write_dns_config_subtask = tasks.write_dns_config.subtask(
-        kwargs={
-            'zones': zones, 'callback': reconfig_subtask,
-            'trusted_networks': get_trusted_networks()})
-    tasks.write_dns_zone_config.delay(
-        zones=zones_to_write, callback=write_dns_config_subtask)
-
-
-@synchronised(locks.dns)
-def write_full_dns_config(reload_retry=False, force=False):
-    """Write the DNS configuration.
-
-    :param reload_retry: Should the reload rndc command be retried in case
-        of failure?  Defaults to `False`.
+    bind_write_zones(zones_to_write)
+    bind_write_configuration(zones, trusted_networks=get_trusted_networks())
+    bind_reconfigure()
+
+
+@synchronised(locks.dns)
+def dns_update_zones(clusters):
+    """Update the zone files for the given cluster(s).
+
+    Once the new zone files have been written, BIND is asked to reload them.
+    It's assumed that BIND already serves the zones that are being written.
+
+    :param clusters: Those cluster(s) for which the zone should be updated.
+    :type clusters: A :py:class:`NodeGroup`, or an iterable thereof.
+    """
+    if not (is_dns_enabled() and is_dns_in_use()):
+        return
+
+    serial = next_zone_serial()
+    for zone in ZoneGenerator(clusters, serial):
+        maaslog.info("Generating new DNS zone file for %s", zone.zone_name)
+        bind_write_zones([zone])
+        bind_reload_zone(zone.zone_name)
+
+
+@synchronised(locks.dns)
+def dns_update_all_zones(reload_retry=False, force=False):
+    """Update all zone files for the given cluster(s), and serve them.
+
+    Serving these zone files means updating BIND's configuration to include
+    them, then asking it to load the new configuration.
+
+    :param reload_retry: Should the DNS server reload be retried in case
+        of failure? Defaults to `False`.
     :type reload_retry: bool
-    :param force: Write the configuration even if no interface is
-        configured to manage DNS.
+    :param force: Update the configuration even if no interface is configured
+        to manage DNS. This makes sense when deconfiguring an interface.
     :type force: bool
     """
-    write_conf = (
-        is_dns_enabled() and (force or is_dns_in_use()))
+    write_conf = is_dns_enabled() and (force or is_dns_in_use())
     if not write_conf:
         return
+
+    clusters = NodeGroup.objects.all()
     zones = ZoneGenerator(
-        NodeGroup.objects.all(), serial_generator=next_zone_serial
-        ).as_list()
+        clusters, serial_generator=next_zone_serial).as_list()
+    bind_write_zones(zones)
+
+    # We should not be calling bind_write_options() here; call-sites should be
+    # making a separate call. It's a historical legacy, where many sites now
+    # expect this side-effect from calling dns_update_all_zones(), and some
+    # that call it for this side-effect alone. At present all it does is set
+    # the upstream DNS servers, nothing to do with serving zones at all!
+    bind_write_options(upstream_dns=get_upstream_dns())
+
+    # Nor should we be rewriting ACLs that are related only to allowing
+    # recursive queries to the upstream DNS servers. Again, this is legacy,
+    # where the "trusted" ACL ended up in the same configuration file as the
+    # zone stanzas, and so both need to be rewritten at the same time.
+    bind_write_configuration(zones, trusted_networks=get_trusted_networks())
+
+    # Reloading with retries may be a legacy from Celery days, or it may be
+    # necessary to recover from races during start-up. We're not sure if it is
+    # actually needed but it seems safer to maintain this behaviour until we
+    # have a better understanding.
+    if reload_retry:
+        bind_reload_with_retries()
+    else:
+        bind_reload()
+
+
+def get_upstream_dns():
+    """Return the IP addresses of configured upstream DNS servers.
+
+    :return: A list of IP addresses.
+    """
     upstream_dns = Config.objects.get_config("upstream_dns")
-    tasks.write_full_dns_config.delay(
-        zones=zones,
-        callback=tasks.rndc_command.subtask(
-            args=[['reload'], reload_retry]),
-        upstream_dns=upstream_dns,
-        trusted_networks=get_trusted_networks())
+    return [] if upstream_dns is None else upstream_dns.split()
 
 
 def get_trusted_networks():
     """Return the CIDR representation of all the Networks we know about.
 
-    This must be a whitespace separated list, where each item ends in a
-    semicolon, or blank if there's no networks.
+    :return: A list of CIDR-format network specifications.
     """
-    networks = " ".join(
-        "%s;" % net.get_network().cidr
-        for net in Network.objects.all())
-    return networks
+    return [
+        unicode(net.get_network().cidr)
+        for net in Network.objects.all()
+    ]

=== modified file 'src/maasserver/dns/connect.py'
--- src/maasserver/dns/connect.py	2014-08-29 09:42:38 +0000
+++ src/maasserver/dns/connect.py	2015-01-13 18:36:01 +0000
@@ -36,13 +36,13 @@
 def dns_post_save_NodeGroup(sender, instance, created, **kwargs):
     """Create or update DNS zones related to the saved nodegroup."""
     from maasserver.dns.config import (
-        write_full_dns_config,
-        add_zone,
+        dns_update_all_zones,
+        dns_add_zones,
         )
     if created:
-        add_zone(instance)
+        dns_add_zones(instance)
     else:
-        write_full_dns_config()
+        dns_update_all_zones()
 
 
 # XXX rvb 2012-09-12: This is only needed because we use that
@@ -52,24 +52,24 @@
 def dns_post_save_NodeGroupInterface(sender, instance, created, **kwargs):
     """Create or update DNS zones related to the saved nodegroupinterface."""
     from maasserver.dns.config import (
-        write_full_dns_config,
-        add_zone,
+        dns_update_all_zones,
+        dns_add_zones,
         )
     if created:
-        add_zone(instance.nodegroup)
+        dns_add_zones(instance.nodegroup)
     else:
-        write_full_dns_config()
+        dns_update_all_zones()
 
 
 def dns_post_edit_management_NodeGroupInterface(instance, old_values, deleted):
     """Delete DNS zones related to the interface."""
-    from maasserver.dns.config import write_full_dns_config
+    from maasserver.dns.config import dns_update_all_zones
     [old_field] = old_values
     if old_field == NODEGROUPINTERFACE_MANAGEMENT.DHCP_AND_DNS:
         # Force the dns config to be written as this might have been
         # triggered by the last DNS-enabled interface being deleted
         # or switched off (i.e. management set to DHCP or UNMANAGED).
-        write_full_dns_config(force=True)
+        dns_update_all_zones(force=True)
 
 
 connect_to_field_change(
@@ -81,8 +81,8 @@
 def dns_post_delete_Node(sender, instance, **kwargs):
     """When a Node is deleted, update the Node's zone file."""
     try:
-        from maasserver.dns.config import change_dns_zones
-        change_dns_zones(instance.nodegroup)
+        from maasserver.dns.config import dns_update_zones
+        dns_update_zones(instance.nodegroup)
     except NodeGroup.DoesNotExist:
         # If this Node is being deleted because the whole NodeGroup
         # has been deleted, no need to update the zone file because
@@ -92,29 +92,29 @@
 
 def dns_post_edit_hostname_Node(instance, old_values, **kwargs):
     """When a Node has been flagged, update the related zone."""
-    from maasserver.dns.config import change_dns_zones
-    change_dns_zones(instance.nodegroup)
+    from maasserver.dns.config import dns_update_zones
+    dns_update_zones(instance.nodegroup)
 
 
 connect_to_field_change(dns_post_edit_hostname_Node, Node, ['hostname'])
 
 
 def dns_setting_changed(sender, instance, created, **kwargs):
-    from maasserver.dns.config import write_full_dns_config
-    write_full_dns_config()
+    from maasserver.dns.config import dns_update_all_zones
+    dns_update_all_zones()
 
 
 @receiver(post_save, sender=Network)
 def dns_post_save_Network(sender, instance, **kwargs):
     """When a network is added/changed, put it in the DNS trusted networks."""
-    from maasserver.dns.config import write_full_dns_config
-    write_full_dns_config()
+    from maasserver.dns.config import dns_update_all_zones
+    dns_update_all_zones()
 
 
 @receiver(post_delete, sender=Network)
 def dns_post_delete_Network(sender, instance, **kwargs):
-    from maasserver.dns.config import write_full_dns_config
-    write_full_dns_config()
+    from maasserver.dns.config import dns_update_all_zones
+    dns_update_all_zones()
 
 
 Config.objects.config_changed_connect("upstream_dns", dns_setting_changed)

=== modified file 'src/maasserver/dns/tests/test_config.py'
--- src/maasserver/dns/tests/test_config.py	2014-10-29 17:05:36 +0000
+++ src/maasserver/dns/tests/test_config.py	2015-01-13 18:36:01 +0000
@@ -18,20 +18,19 @@
 from itertools import islice
 import random
 
-from celery.task import task
 from django.conf import settings
 from django.core.management import call_command
 from maasserver import locks
 from maasserver.dns import config as dns_config_module
 from maasserver.dns.config import (
-    add_zone,
-    change_dns_zones,
+    dns_add_zones,
+    dns_update_all_zones,
+    dns_update_zones,
     get_trusted_networks,
+    get_upstream_dns,
     is_dns_enabled,
     is_dns_in_use,
     next_zone_serial,
-    tasks as dns_tasks,
-    write_full_dns_config,
     zone_serial,
     )
 from maasserver.enum import (
@@ -46,17 +45,16 @@
 from maasserver.testing.factory import factory
 from maasserver.testing.testcase import MAASServerTestCase
 from maastesting.fakemethod import FakeMethod
-from maastesting.matchers import MockCalledOnceWith
-from mock import (
-    ANY,
-    sentinel,
+from maastesting.matchers import (
+    MockCalledOnceWith,
+    MockNotCalled,
     )
+from mock import sentinel
 from netaddr import (
     IPAddress,
     IPNetwork,
     IPRange,
     )
-from provisioningserver import tasks
 from provisioningserver.dns.config import (
     compose_config_path,
     DNSConfig,
@@ -141,7 +139,7 @@
             interface.static_ip_range_low, interface.static_ip_range_high)
         static_ip = unicode(islice(ips, lease_number, lease_number + 1).next())
         staticaddress = factory.make_StaticIPAddress(ip=static_ip, mac=mac)
-        change_dns_zones([nodegroup])
+        dns_update_zones([nodegroup])
         return nodegroup, node, staticaddress
 
     def dig_resolve(self, fqdn, version=4):
@@ -184,57 +182,57 @@
         def check_dns_is_locked():
             self.assertTrue(
                 locks.dns.is_locked(), "locks.dns isn't locked")
-            return False  # Prevent change_dns_zones from continuing.
+            return False  # Prevent dns_update_zones from continuing.
 
         is_dns_enabled = self.patch_autospec(
             dns_config_module, 'is_dns_enabled')
         is_dns_enabled.side_effect = check_dns_is_locked
         return is_dns_enabled
 
-    def test_change_dns_zone_uses_lock(self):
-        is_dns_enabled = self.patch_is_dns_enabled()
-        dns_config_module.change_dns_zones(sentinel.nodegroup)
-        self.assertThat(is_dns_enabled, MockCalledOnceWith())
-
-    def test_add_zone_uses_lock(self):
-        is_dns_enabled = self.patch_is_dns_enabled()
-        dns_config_module.add_zone(sentinel.nodegroup)
-        self.assertThat(is_dns_enabled, MockCalledOnceWith())
-
-    def test_write_full_dns_config_uses_lock(self):
-        is_dns_enabled = self.patch_is_dns_enabled()
-        dns_config_module.write_full_dns_config()
+    def test_dns_update_zones_uses_lock(self):
+        is_dns_enabled = self.patch_is_dns_enabled()
+        dns_config_module.dns_update_zones(sentinel.nodegroup)
+        self.assertThat(is_dns_enabled, MockCalledOnceWith())
+
+    def test_dns_add_zones_uses_lock(self):
+        is_dns_enabled = self.patch_is_dns_enabled()
+        dns_config_module.dns_add_zones(sentinel.nodegroup)
+        self.assertThat(is_dns_enabled, MockCalledOnceWith())
+
+    def test_dns_update_all_zones_config_uses_lock(self):
+        is_dns_enabled = self.patch_is_dns_enabled()
+        dns_config_module.dns_update_all_zones()
         self.assertThat(is_dns_enabled, MockCalledOnceWith())
 
 
 class TestDNSConfigModifications(TestDNSServer):
 
-    def test_add_zone_loads_dns_zone(self):
+    def test_dns_add_zones_loads_dns_zone(self):
         nodegroup, node, static = self.create_nodegroup_with_static_ip()
         self.patch(settings, 'DNS_CONNECT', True)
-        add_zone(nodegroup)
+        dns_add_zones(nodegroup)
         self.assertDNSMatches(node.hostname, nodegroup.name, static.ip)
 
-    def test_add_zone_preserves_trusted_networks(self):
+    def test_dns_add_zones_preserves_trusted_networks(self):
         nodegroup, node, static = self.create_nodegroup_with_static_ip()
         trusted_network = factory.make_ipv4_address()
         get_trusted_networks_patch = self.patch(
             dns_config_module, 'get_trusted_networks')
-        get_trusted_networks_patch.return_value = trusted_network + ';'
+        get_trusted_networks_patch.return_value = [trusted_network]
         self.patch(settings, 'DNS_CONNECT', True)
-        add_zone(nodegroup)
+        dns_add_zones(nodegroup)
         self.assertThat(
             compose_config_path(DNSConfig.target_file_name),
             FileContains(matcher=Contains(trusted_network)))
 
-    def test_change_dns_zone_changes_dns_zone(self):
+    def test_dns_update_zones_changes_dns_zone(self):
         nodegroup, _, _ = self.create_nodegroup_with_static_ip()
         self.patch(settings, 'DNS_CONNECT', True)
-        write_full_dns_config()
+        dns_update_all_zones()
         nodegroup, new_node, new_static = (
             self.create_nodegroup_with_static_ip(
                 nodegroup=nodegroup, lease_number=2))
-        change_dns_zones(nodegroup)
+        dns_update_zones(nodegroup)
         self.assertDNSMatches(new_node.hostname, nodegroup.name, new_static.ip)
 
     def test_is_dns_enabled_return_false_if_DNS_CONNECT_False(self):
@@ -252,60 +250,57 @@
         self.create_managed_nodegroup()
         self.assertTrue(is_dns_in_use())
 
-    def test_write_full_dns_loads_full_dns_config(self):
+    def test_dns_update_all_zones_loads_full_dns_config(self):
         nodegroup, node, static = self.create_nodegroup_with_static_ip()
         self.patch(settings, 'DNS_CONNECT', True)
-        write_full_dns_config()
+        dns_update_all_zones()
         self.assertDNSMatches(node.hostname, nodegroup.name, static.ip)
 
-    def test_write_full_dns_passes_reload_retry_parameter(self):
+    def test_dns_update_all_zones_passes_reload_retry_parameter(self):
         self.patch(settings, 'DNS_CONNECT', True)
-        recorder = FakeMethod()
         self.create_managed_nodegroup()
-
-        @task
-        def recorder_task(*args, **kwargs):
-            return recorder(*args, **kwargs)
-        self.patch(tasks, 'rndc_command', recorder_task)
-        write_full_dns_config(reload_retry=True)
-        self.assertEqual(
-            ([(['reload'], True)]), recorder.extract_args())
-
-    def test_write_full_dns_passes_upstream_dns_parameter(self):
+        bind_reload_with_retries = self.patch_autospec(
+            dns_config_module, "bind_reload_with_retries")
+        dns_update_all_zones(reload_retry=True)
+        self.assertThat(bind_reload_with_retries, MockCalledOnceWith())
+
+    def test_dns_update_all_zones_passes_upstream_dns_parameter(self):
         self.patch(settings, 'DNS_CONNECT', True)
         self.create_managed_nodegroup()
         random_ip = factory.make_ipv4_address()
         Config.objects.set_config("upstream_dns", random_ip)
-        patched_task = self.patch(dns_tasks.write_full_dns_config, "delay")
-        write_full_dns_config()
-        self.assertThat(patched_task, MockCalledOnceWith(
-            zones=ANY, callback=ANY, trusted_networks=ANY,
-            upstream_dns=random_ip))
+        bind_write_options = self.patch_autospec(
+            dns_config_module, "bind_write_options")
+        dns_update_all_zones()
+        self.assertThat(
+            bind_write_options,
+            MockCalledOnceWith(upstream_dns=[random_ip]))
 
-    def test_write_full_dns_writes_trusted_networks_parameter(self):
+    def test_dns_update_all_zones_writes_trusted_networks_parameter(self):
         self.patch(settings, 'DNS_CONNECT', True)
         self.create_managed_nodegroup()
         trusted_network = factory.make_ipv4_address()
         get_trusted_networks_patch = self.patch(
             dns_config_module, 'get_trusted_networks')
-        get_trusted_networks_patch.return_value = trusted_network + ';'
-        write_full_dns_config()
+        get_trusted_networks_patch.return_value = [trusted_network]
+        dns_update_all_zones()
         self.assertThat(
             compose_config_path(DNSConfig.target_file_name),
             FileContains(matcher=Contains(trusted_network)))
 
-    def test_write_full_dns_doesnt_call_task_it_no_interface_configured(self):
+    def test_dns_update_all_zones_does_nada_if_no_interface_configured(self):
         self.patch(settings, 'DNS_CONNECT', True)
-        patched_task = self.patch(dns_tasks.write_full_dns_config, "delay")
-        write_full_dns_config()
-        self.assertEqual(0, patched_task.call_count)
+        bind_write_configuration = self.patch_autospec(
+            dns_config_module, "bind_write_configuration")
+        dns_update_all_zones()
+        self.assertThat(bind_write_configuration, MockNotCalled())
 
     def test_dns_config_has_NS_record(self):
         ip = factory.make_ipv4_address()
         self.patch(settings, 'DEFAULT_MAAS_URL', 'http://%s/' % ip)
         nodegroup, node, static = self.create_nodegroup_with_static_ip()
         self.patch(settings, 'DNS_CONNECT', True)
-        write_full_dns_config()
+        dns_update_all_zones()
         # Get the NS record for the zone 'nodegroup.name'.
         ns_record = dig_call(
             port=self.bind.config.port,
@@ -419,7 +414,7 @@
         ip = "%s" % random.choice(ip_range)
         lease = factory.make_DHCPLease(
             nodegroup=nodegroup, mac=mac.mac_address, ip=ip)
-        change_dns_zones([nodegroup])
+        dns_update_zones([nodegroup])
         self.assertDNSMatches(node.hostname, nodegroup.name, lease.ip)
 
 
@@ -435,19 +430,35 @@
             node.hostname, nodegroup.name, static.ip, version=6)
 
 
+class TestGetUpstreamDNS(MAASServerTestCase):
+    """Test for maasserver/dns/config.py:get_upstream_dns()"""
+
+    def test__returns_empty_list_if_not_set(self):
+        self.assertEqual([], get_upstream_dns())
+
+    def test__returns_list_of_one_address_if_set(self):
+        address = factory.make_ipv4_address()
+        Config.objects.set_config("upstream_dns", address)
+        self.assertEqual([address], get_upstream_dns())
+
+    def test__returns_list_of_many_address_if_set(self):
+        addresses = [factory.make_ipv4_address(), factory.make_ipv4_address()]
+        Config.objects.set_config("upstream_dns", " ".join(addresses))
+        self.assertEqual(addresses, get_upstream_dns())
+
+
 class TestGetTrustedNetworks(MAASServerTestCase):
     """Test for maasserver/dns/config.py:get_trusted_networks()"""
 
     def test__returns_empty_string_if_no_networks(self):
-        self.assertEqual("", get_trusted_networks())
+        self.assertEqual([], get_trusted_networks())
 
     def test__returns_single_network(self):
         net = factory.make_Network()
-        expected = unicode(net.get_network().cidr) + ';'
+        expected = [unicode(net.get_network().cidr)]
         self.assertEqual(expected, get_trusted_networks())
 
     def test__returns_many_networks(self):
         nets = [factory.make_Network() for _ in xrange(random.randint(1, 5))]
-        expected = "; ".join(unicode(net.get_network().cidr) for net in nets)
-        expected += ';'
+        expected = [unicode(net.get_network().cidr) for net in nets]
         self.assertEqual(expected, get_trusted_networks())

=== modified file 'src/maasserver/management/commands/write_dns_config.py'
--- src/maasserver/management/commands/write_dns_config.py	2014-08-15 09:44:42 +0000
+++ src/maasserver/management/commands/write_dns_config.py	2015-01-13 18:36:01 +0000
@@ -24,7 +24,7 @@
     ]
 
 from django.core.management.base import BaseCommand
-from maasserver.dns.config import write_full_dns_config
+from maasserver.dns.config import dns_update_all_zones
 
 
 class Command(BaseCommand):
@@ -33,4 +33,4 @@
         "this region has cluster controllers configured to manage DNS.")
 
     def handle(self, *args, **options):
-        write_full_dns_config()
+        dns_update_all_zones()

=== modified file 'src/maasserver/models/node.py'
--- src/maasserver/models/node.py	2015-01-13 17:10:46 +0000
+++ src/maasserver/models/node.py	2015-01-13 18:36:01 +0000
@@ -1427,8 +1427,8 @@
         """
         deallocated_ips = StaticIPAddress.objects.deallocate_by_node(self)
         self.delete_host_maps(deallocated_ips)
-        from maasserver.dns.config import change_dns_zones
-        change_dns_zones([self.nodegroup])
+        from maasserver.dns.config import dns_update_zones
+        dns_update_zones([self.nodegroup])
 
     def get_boot_purpose(self):
         """
@@ -1518,7 +1518,7 @@
         """
         # Avoid circular imports.
         from metadataserver.models import NodeUserData
-        from maasserver.dns.config import change_dns_zones
+        from maasserver.dns.config import dns_update_zones
 
         if not by_user.has_perm(NODE_PERMISSION.EDIT, self):
             # You can't stop a node you don't own unless you're an
@@ -1560,7 +1560,7 @@
             transition_timeout = None
 
         # Update the DNS zone with the new static IP info as necessary.
-        change_dns_zones(self.nodegroup)
+        dns_update_zones(self.nodegroup)
 
         power_info = self.get_effective_power_info()
         if not power_info.can_be_started:

=== modified file 'src/maasserver/models/tests/test_dhcplease.py'
--- src/maasserver/models/tests/test_dhcplease.py	2014-11-25 13:06:05 +0000
+++ src/maasserver/models/tests/test_dhcplease.py	2015-01-13 18:36:01 +0000
@@ -166,10 +166,10 @@
             map_leases(nodegroup))
 
     def test_update_leases_does_not_update_dns_zone_if_nothing_added(self):
-        self.patch(dns, 'change_dns_zones')
+        self.patch(dns, 'dns_update_zones')
         nodegroup = factory.make_NodeGroup()
         DHCPLease.objects.update_leases(nodegroup, {})
-        self.assertFalse(dns.change_dns_zones.called)
+        self.assertFalse(dns.dns_update_zones.called)
 
     def test_get_hostname_ip_mapping_returns_mapping(self):
         nodegroup = factory.make_NodeGroup()

=== modified file 'src/maasserver/models/tests/test_node.py'
--- src/maasserver/models/tests/test_node.py	2015-01-13 17:10:46 +0000
+++ src/maasserver/models/tests/test_node.py	2015-01-13 18:36:01 +0000
@@ -1228,7 +1228,7 @@
                 {node.nodegroup: expected}))
 
     def test_deallocate_static_ip_updates_dns(self):
-        change_dns_zones = self.patch(dns_config, 'change_dns_zones')
+        dns_update_zones = self.patch(dns_config, 'dns_update_zones')
         nodegroup = factory.make_NodeGroup(
             management=NODEGROUPINTERFACE_MANAGEMENT.DHCP_AND_DNS,
             status=NODEGROUP_STATUS.ACCEPTED)
@@ -1236,7 +1236,7 @@
             nodegroup=nodegroup, status=NODE_STATUS.ALLOCATED,
             owner=factory.make_User(), power_type='ether_wake')
         node.release()
-        self.assertThat(change_dns_zones, MockCalledOnceWith([node.nodegroup]))
+        self.assertThat(dns_update_zones, MockCalledOnceWith([node.nodegroup]))
 
     def test_release_logs_and_raises_errors_in_stopping(self):
         node = factory.make_Node(status=NODE_STATUS.DEPLOYED)
@@ -2424,12 +2424,12 @@
         user = factory.make_User()
         node = self.make_acquired_node_with_mac(user)
 
-        change_dns_zones = self.patch(dns_config, "change_dns_zones")
+        dns_update_zones = self.patch(dns_config, "dns_update_zones")
 
         node.start(user)
 
         self.assertThat(
-            change_dns_zones, MockCalledOnceWith(node.nodegroup))
+            dns_update_zones, MockCalledOnceWith(node.nodegroup))
 
     def test__starts_nodes(self):
         user = factory.make_User()

=== modified file 'src/maasserver/start_up.py'
--- src/maasserver/start_up.py	2014-10-10 13:35:47 +0000
+++ src/maasserver/start_up.py	2015-01-13 18:36:01 +0000
@@ -26,7 +26,7 @@
     security,
     )
 from maasserver.bootresources import ensure_boot_source_definition
-from maasserver.dns.config import write_full_dns_config
+from maasserver.dns.config import dns_update_all_zones
 from maasserver.fields import register_mac_type
 from maasserver.models import NodeGroup
 from provisioningserver.upgrade_cluster import create_gnupg_home
@@ -77,4 +77,4 @@
     ensure_boot_source_definition()
 
     # Regenerate MAAS's DNS configuration.  This should be reentrant, really.
-    write_full_dns_config(reload_retry=True)
+    dns_update_all_zones(reload_retry=True)

=== modified file 'src/maasserver/tests/test_commands_write_dns_config.py'
--- src/maasserver/tests/test_commands_write_dns_config.py	2014-09-23 21:51:24 +0000
+++ src/maasserver/tests/test_commands_write_dns_config.py	2015-01-13 18:36:01 +0000
@@ -26,7 +26,7 @@
 from maasserver.testing.factory import factory
 from maasserver.testing.testcase import MAASServerTestCase
 from netaddr import IPNetwork
-from provisioningserver import tasks
+from provisioningserver.dns import actions
 from provisioningserver.dns.testing import patch_dns_config_path
 from testtools.matchers import FileExists
 
@@ -37,8 +37,8 @@
         dns_conf_dir = self.make_dir()
         patch_dns_config_path(self, dns_conf_dir)
         self.patch(settings, 'DNS_CONNECT', True)
-        # Prevent rndc task dispatch.
-        self.patch(tasks, "rndc_command")
+        # Prevent rndc execution.
+        self.patch(actions, "execute_rndc_command")
         domain = factory.make_string()
         factory.make_NodeGroup(
             name=domain,

=== modified file 'src/maasserver/tests/test_forms_network.py'
--- src/maasserver/tests/test_forms_network.py	2014-12-11 15:14:32 +0000
+++ src/maasserver/tests/test_forms_network.py	2015-01-13 18:36:01 +0000
@@ -181,8 +181,8 @@
             form.errors)
 
     def test_writes_dns_when_network_edited(self):
-        write_full_dns_config = self.patch(
-            dns_config_module, "write_full_dns_config")
+        dns_update_all_zones = self.patch(
+            dns_config_module, "dns_update_all_zones")
         network = factory.make_ipv4_network()
         name = factory.make_name('network')
         definition = {
@@ -194,14 +194,14 @@
         }
         form = NetworkForm(data=definition)
         form.save()
-        self.assertThat(write_full_dns_config, MockCalledOnceWith())
+        self.assertThat(dns_update_all_zones, MockCalledOnceWith())
 
     def test_writes_dns_when_network_deleted(self):
         network = factory.make_Network()
-        write_full_dns_config = self.patch(
-            dns_config_module, "write_full_dns_config")
+        dns_update_all_zones = self.patch(
+            dns_config_module, "dns_update_all_zones")
         network.delete()
-        self.assertThat(write_full_dns_config, MockCalledOnceWith())
+        self.assertThat(dns_update_all_zones, MockCalledOnceWith())
 
 
 class TestCreateNetworkFromNodeGroupInterface(MAASServerTestCase):

=== modified file 'src/maasserver/tests/test_start_up.py'
--- src/maasserver/tests/test_start_up.py	2014-10-20 21:11:45 +0000
+++ src/maasserver/tests/test_start_up.py	2015-01-13 18:36:01 +0000
@@ -77,7 +77,7 @@
 
     def test__calls_write_full_dns_config(self):
         recorder = FakeMethod()
-        self.patch(start_up, 'write_full_dns_config', recorder)
+        self.patch(start_up, 'dns_update_all_zones', recorder)
         start_up.inner_start_up()
         self.assertEqual(
             (1, [()]),

=== removed file 'src/maastesting/celery.py'
--- src/maastesting/celery.py	2014-09-29 15:42:42 +0000
+++ src/maastesting/celery.py	1970-01-01 00:00:00 +0000
@@ -1,84 +0,0 @@
-# Copyright 2012 Canonical Ltd.  This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-"""A fixture to make Celery run tasks in a synchronous fashion."""
-
-from __future__ import (
-    absolute_import,
-    print_function,
-    unicode_literals,
-    )
-
-str = None
-
-__metaclass__ = type
-__all__ = [
-    'CeleryFixture',
-    ]
-
-from celery import (
-    current_app,
-    signals,
-    )
-from fixtures import Fixture
-from testtools.monkey import MonkeyPatcher
-
-
-class CeleryFixture(Fixture):
-    """This fixture will record Celery tasks as they're run.
-
-    This fixture can be used directly::
-
-      >>> from unittest import TestCase
-
-      >>> class CeleryTest1(TestCase):
-      ...
-      ...     def setUp(self):
-      ...         super(CeleryTest1, self).setUp()
-      ...         self.useFixture(CeleryFixture())
-
-    It can also be converted into a FixtureResource::
-
-      >>> from testresources import FixtureResource
-
-      >>> class CeleryTest2(TestCase):
-      ...
-      ...     resources = (
-      ...         ("celery", FixtureResource(CeleryFixture())),
-      ...         )
-      ...
-
-    """
-
-    def setUp(self):
-        super(CeleryFixture, self).setUp()
-        self.configure()
-        self.record_tasks()
-
-    def configure(self):
-        patcher = MonkeyPatcher()
-        patcher.add_patch(current_app.conf, 'CELERY_ALWAYS_EAGER', True)
-        patcher.add_patch(
-            current_app.conf, 'CELERY_EAGER_PROPAGATES_EXCEPTIONS', True)
-        self.addCleanup(patcher.restore)
-        patcher.patch()
-
-    def record_tasks(self):
-        self.tasks = []
-
-        def on_task_postrun(**kwargs):
-            self.tasks.append(kwargs)
-        signals.task_postrun.connect(on_task_postrun, weak=False)
-        self.addCleanup(lambda: self.cleanup_tasks())
-
-    def get_task_routing(self):
-        """Get a mapping between the name of the tasks and the queue they
-        were sent to.
-        """
-        return {
-            task_info['task'].name: task_info['task'].queue
-            for task_info in self.tasks
-        }
-
-    def cleanup_tasks(self):
-        self.tasks = []

=== removed file 'src/maastesting/tests/test_celery.py'
--- src/maastesting/tests/test_celery.py	2014-09-24 10:12:35 +0000
+++ src/maastesting/tests/test_celery.py	1970-01-01 00:00:00 +0000
@@ -1,67 +0,0 @@
-# Copyright 2012, 2013 Canonical Ltd.  This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-"""Test matchers."""
-
-from __future__ import (
-    absolute_import,
-    print_function,
-    unicode_literals,
-    )
-
-str = None
-
-__metaclass__ = type
-__all__ = []
-
-import random
-
-from celery.decorators import task
-from celery.result import EagerResult
-from maastesting.celery import CeleryFixture
-from maastesting.testcase import MAASTestCase
-
-
-@task()
-def task_add(x, y):
-    return x + y
-
-
-@task()
-def task_exception(x, y):
-    raise RuntimeError()
-
-
-class TestCeleryFixture(MAASTestCase):
-    """Tests `CeleryFixture`."""
-
-    def setUp(self):
-        super(TestCeleryFixture, self).setUp()
-        self.celery = self.useFixture(CeleryFixture())
-
-    def test_celery_eagerresult_contains_result(self):
-        # The result is an instance of EagerResult and it contains the actual
-        # result.
-        x = random.randrange(100)
-        y = random.randrange(100)
-        result = task_add.delay(x, y)
-        self.assertIsInstance(result, EagerResult)
-        self.assertEqual(x + y, result.result)
-
-    def test_celery_exception_raised(self):
-        self.assertRaises(RuntimeError, task_exception.delay, 1, 2)
-
-    def test_celery_records_tasks(self):
-        x = random.randrange(100)
-        y = random.randrange(100)
-        task_add.delay(x=x, y=y)
-        z = random.randrange(100)
-        t = random.randrange(100)
-        task_add.delay(x=z, y=t)
-        tasks = self.celery.tasks
-        self.assertEqual(2, len(tasks))
-        self.assertEqual(
-            ['maastesting.tests.test_celery.task_add'] * 2,
-            [task['task'].name for task in tasks])
-        self.assertEqual({'x': x, 'y': y}, tasks[0]['kwargs'])
-        self.assertEqual({'x': z, 'y': t}, tasks[1]['kwargs'])

=== added file 'src/provisioningserver/dns/actions.py'
--- src/provisioningserver/dns/actions.py	1970-01-01 00:00:00 +0000
+++ src/provisioningserver/dns/actions.py	2015-01-13 18:36:01 +0000
@@ -0,0 +1,142 @@
+# Copyright 2014 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Low-level actions to manage the DNS service, like reloading zones."""
+
+from __future__ import (
+    absolute_import,
+    print_function,
+    unicode_literals,
+    )
+
+str = None
+
+__metaclass__ = type
+__all__ = [
+    "bind_reconfigure",
+    "bind_reload",
+    "bind_reload_zone",
+    "bind_write_configuration",
+    "bind_write_options",
+    "bind_write_zones",
+]
+
+import collections
+from subprocess import CalledProcessError
+from time import sleep
+
+from provisioningserver.dns.config import (
+    DNSConfig,
+    execute_rndc_command,
+    set_up_options_conf,
+    )
+from provisioningserver.logger import get_maas_logger
+from provisioningserver.utils.shell import ExternalProcessError
+
+
+maaslog = get_maas_logger("dns")
+
+
+def bind_reconfigure():
+    """Ask BIND to reload its configuration and *new* zone files.
+
+    From rndc(8):
+
+      Reload the configuration file and load new zones, but do not reload
+      existing zone files even if they have changed. This is faster than a
+      full reload when there is a large number of zones because it avoids the
+      need to examine the modification times of the zones files.
+
+    """
+    try:
+        execute_rndc_command(("reconfig",))
+    except CalledProcessError as exc:
+        maaslog.error("Reloading BIND configuration failed: %s", exc)
+        # Log before upgrade so that the output does not go to maaslog.
+        ExternalProcessError.upgrade(exc)
+        raise
+
+
+def bind_reload():
+    """Ask BIND to reload its configuration and all zone files."""
+    try:
+        execute_rndc_command(("reload",))
+    except CalledProcessError as exc:
+        maaslog.error("Reloading BIND failed: %s", exc)
+        # Log before upgrade so that the output does not go to maaslog.
+        ExternalProcessError.upgrade(exc)
+        raise
+
+
+def bind_reload_with_retries(attempts=10, interval=2):
+    """Ask BIND to reload its configuration and all zone files.
+
+    :param attempts: The number of attempts.
+    :param interval: The time in seconds to sleep between each attempt.
+    """
+    for countdown in xrange(attempts - 1, -1, -1):
+        try:
+            bind_reload()
+        except CalledProcessError:
+            if countdown == 0:
+                raise
+            else:
+                sleep(interval)
+        else:
+            break
+
+
+def bind_reload_zone(zone_name):
+    """Ask BIND to reload the zone file for the given zone.
+
+    :param zone_name: The name of the zone to reload.
+    """
+    try:
+        execute_rndc_command(("reload", zone_name))
+    except CalledProcessError as exc:
+        maaslog.error("Reloading BIND zone %r failed: %s", zone_name, exc)
+        # Log before upgrade so that the output does not go to maaslog.
+        ExternalProcessError.upgrade(exc)
+        raise
+
+
+def bind_write_configuration(zones, trusted_networks):
+    """Write BIND's configuration.
+
+    :param zones: Those zones to include in main config.
+    :type zones: Sequence of :py:class:`DNSZoneData`.
+
+    :param trusted_networks: A sequence of CIDR network specifications that
+        are permitted to use the DNS server as a forwarder.
+    """
+    # trusted_networks was formerly specified as a single IP address with
+    # netmask. These assertions are here to prevent code that assumes that
+    # slipping through.
+    assert not isinstance(trusted_networks, (bytes, unicode))
+    assert isinstance(trusted_networks, collections.Sequence)
+
+    dns_config = DNSConfig(zones=zones)
+    dns_config.write_config(trusted_networks=trusted_networks)
+
+
+def bind_write_options(upstream_dns):
+    """Write BIND options.
+
+    :param upstream_dns: A sequence of upstream DNS servers.
+    """
+    # upstream_dns was formerly specified as a single IP address. These
+    # assertions are here to prevent code that assumes that slipping through.
+    assert not isinstance(upstream_dns, (bytes, unicode))
+    assert isinstance(upstream_dns, collections.Sequence)
+
+    set_up_options_conf(upstream_dns=upstream_dns)
+
+
+def bind_write_zones(zones):
+    """Write out DNS zones.
+
+    :param zones: Those zones to write.
+    :type zones: Sequence of :py:class:`DNSZoneData`.
+    """
+    for zone in zones:
+        zone.write_config()

=== added file 'src/provisioningserver/dns/tests/test_actions.py'
--- src/provisioningserver/dns/tests/test_actions.py	1970-01-01 00:00:00 +0000
+++ src/provisioningserver/dns/tests/test_actions.py	2015-01-13 18:36:01 +0000
@@ -0,0 +1,271 @@
+# Copyright 2014 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Tests for :py:module:`provisioningserver.dns.actions`."""
+
+from __future__ import (
+    absolute_import,
+    print_function,
+    unicode_literals,
+    )
+
+str = None
+
+__metaclass__ = type
+__all__ = []
+
+import os
+from os.path import join
+import random
+from random import randint
+from subprocess import CalledProcessError
+from textwrap import dedent
+
+from fixtures import FakeLogger
+from maastesting.factory import factory
+from maastesting.matchers import (
+    MockCalledOnceWith,
+    MockCallsMatch,
+    )
+from maastesting.testcase import MAASTestCase
+from mock import (
+    call,
+    sentinel,
+    )
+from netaddr import IPNetwork
+from provisioningserver.dns import actions
+from provisioningserver.dns.config import (
+    MAAS_NAMED_CONF_NAME,
+    MAAS_NAMED_CONF_OPTIONS_INSIDE_NAME,
+    )
+from provisioningserver.dns.testing import patch_dns_config_path
+from provisioningserver.dns.zoneconfig import (
+    DNSForwardZoneConfig,
+    DNSReverseZoneConfig,
+    )
+from provisioningserver.testing.testcase import PservTestCase
+from provisioningserver.utils.shell import ExternalProcessError
+from testtools.matchers import (
+    AllMatch,
+    Contains,
+    FileContains,
+    FileExists,
+    )
+
+
+class TestReconfigure(MAASTestCase):
+    """Tests for :py:func:`actions.bind_reconfigure`."""
+
+    def test__executes_rndc_command(self):
+        self.patch_autospec(actions, "execute_rndc_command")
+        actions.bind_reconfigure()
+        self.assertThat(
+            actions.execute_rndc_command,
+            MockCalledOnceWith(("reconfig",)))
+
+    def test__logs_subprocess_error(self):
+        erc = self.patch_autospec(actions, "execute_rndc_command")
+        erc.side_effect = factory.make_CalledProcessError()
+        with FakeLogger("maas") as logger:
+            self.assertRaises(CalledProcessError, actions.bind_reconfigure)
+        self.assertDocTestMatches(
+            "Reloading BIND configuration failed: "
+            "Command ... returned non-zero exit status ...",
+            logger.output)
+
+    def test__upgrades_subprocess_error(self):
+        erc = self.patch_autospec(actions, "execute_rndc_command")
+        erc.side_effect = factory.make_CalledProcessError()
+        self.assertRaises(ExternalProcessError, actions.bind_reconfigure)
+
+
+class TestReload(MAASTestCase):
+    """Tests for :py:func:`actions.bind_reload`."""
+
+    def test__executes_rndc_command(self):
+        self.patch_autospec(actions, "execute_rndc_command")
+        actions.bind_reload()
+        self.assertThat(
+            actions.execute_rndc_command,
+            MockCalledOnceWith(("reload",)))
+
+    def test__logs_subprocess_error(self):
+        erc = self.patch_autospec(actions, "execute_rndc_command")
+        erc.side_effect = factory.make_CalledProcessError()
+        with FakeLogger("maas") as logger:
+            self.assertRaises(CalledProcessError, actions.bind_reload)
+        self.assertDocTestMatches(
+            "Reloading BIND failed: "
+            "Command ... returned non-zero exit status ...",
+            logger.output)
+
+    def test__upgrades_subprocess_error(self):
+        erc = self.patch_autospec(actions, "execute_rndc_command")
+        erc.side_effect = factory.make_CalledProcessError()
+        self.assertRaises(ExternalProcessError, actions.bind_reload)
+
+
+class TestReloadWithRetries(MAASTestCase):
+    """Tests for :py:func:`actions.bind_reload_with_retries`."""
+
+    def test__calls_bind_reload_count_times(self):
+        self.patch_autospec(actions, "sleep")  # Disable.
+        bind_reload = self.patch_autospec(actions, "bind_reload")
+        bind_reload.side_effect = factory.make_CalledProcessError()
+        attempts = randint(3, 13)
+        self.assertRaises(
+            CalledProcessError, actions.bind_reload_with_retries,
+            attempts=attempts)
+        expected_calls = [call()] * attempts
+        self.assertThat(
+            actions.bind_reload,
+            MockCallsMatch(*expected_calls))
+
+    def test__returns_on_success(self):
+        self.patch_autospec(actions, "sleep")  # Disable.
+        bind_reload = self.patch(actions, "bind_reload")
+        bind_reload.side_effect = [
+            factory.make_CalledProcessError(),
+            factory.make_CalledProcessError(),
+            None,  # Success
+        ]
+        actions.bind_reload_with_retries(attempts=5)
+        expected_calls = [call(), call(), call()]
+        self.assertThat(
+            actions.bind_reload,
+            MockCallsMatch(*expected_calls))
+
+    def test__sleeps_interval_seconds_between_attempts(self):
+        self.patch_autospec(actions, "sleep")  # Disable.
+        bind_reload = self.patch_autospec(actions, "bind_reload")
+        bind_reload.side_effect = factory.make_CalledProcessError()
+        attempts = randint(3, 13)
+        self.assertRaises(
+            CalledProcessError, actions.bind_reload_with_retries,
+            attempts=attempts, interval=sentinel.interval)
+        expected_sleep_calls = [call(sentinel.interval)] * (attempts - 1)
+        self.assertThat(actions.sleep, MockCallsMatch(*expected_sleep_calls))
+
+
+class TestReloadZone(MAASTestCase):
+    """Tests for :py:func:`actions.bind_reload_zone`."""
+
+    def test__executes_rndc_command(self):
+        self.patch_autospec(actions, "execute_rndc_command")
+        actions.bind_reload_zone(sentinel.zone)
+        self.assertThat(
+            actions.execute_rndc_command,
+            MockCalledOnceWith(("reload", sentinel.zone)))
+
+    def test__logs_subprocess_error(self):
+        erc = self.patch_autospec(actions, "execute_rndc_command")
+        erc.side_effect = factory.make_CalledProcessError()
+        with FakeLogger("maas") as logger:
+            self.assertRaises(
+                CalledProcessError, actions.bind_reload_zone, sentinel.zone)
+        self.assertDocTestMatches(
+            "Reloading BIND zone ... failed: "
+            "Command ... returned non-zero exit status ...",
+            logger.output)
+
+    def test__upgrades_subprocess_error(self):
+        erc = self.patch_autospec(actions, "execute_rndc_command")
+        erc.side_effect = factory.make_CalledProcessError()
+        self.assertRaises(
+            ExternalProcessError, actions.bind_reload_zone, sentinel.zone)
+
+
+class TestConfiguration(PservTestCase):
+    """Tests for the `bind_write_*` functions."""
+
+    def setUp(self):
+        super(TestConfiguration, self).setUp()
+        # Ensure that files are written to a temporary directory.
+        self.dns_conf_dir = self.make_dir()
+        patch_dns_config_path(self, self.dns_conf_dir)
+        # Patch out calls to 'execute_rndc_command'.
+        self.patch_autospec(actions, 'execute_rndc_command')
+
+    def test_bind_write_configuration_writes_file(self):
+        domain = factory.make_string()
+        zones = [
+            DNSReverseZoneConfig(
+                domain, serial=random.randint(1, 100),
+                network=factory.make_ipv4_network()),
+            DNSReverseZoneConfig(
+                domain, serial=random.randint(1, 100),
+                network=factory.make_ipv6_network()),
+        ]
+        actions.bind_write_configuration(
+            zones=zones, trusted_networks=[])
+        self.assertThat(
+            os.path.join(self.dns_conf_dir, MAAS_NAMED_CONF_NAME),
+            FileExists())
+
+    def test_bind_write_configuration_writes_file_with_acl(self):
+        trusted_networks = [
+            factory.make_ipv4_network(),
+            factory.make_ipv6_network(),
+        ]
+        actions.bind_write_configuration(
+            zones=[], trusted_networks=trusted_networks)
+        expected_file = os.path.join(self.dns_conf_dir, MAAS_NAMED_CONF_NAME)
+        self.assertThat(expected_file, FileExists())
+        expected_content = dedent("""\
+        acl "trusted" {
+            %s;
+            %s;
+            localnets;
+            localhost;
+        };
+        """)
+        expected_content %= tuple(trusted_networks)
+        self.assertThat(expected_file, FileContains(
+            matcher=Contains(expected_content)))
+
+    def test_bind_write_zones_writes_file(self):
+        domain = factory.make_string()
+        network = IPNetwork('192.168.0.3/24')
+        dns_ip = factory.pick_ip_in_network(network)
+        ip = factory.pick_ip_in_network(network)
+        forward_zone = DNSForwardZoneConfig(
+            domain, serial=random.randint(1, 100),
+            mapping={factory.make_string(): [ip]},
+            dns_ip=dns_ip)
+        reverse_zone = DNSReverseZoneConfig(
+            domain, serial=random.randint(1, 100), network=network)
+        actions.bind_write_zones(zones=[forward_zone, reverse_zone])
+
+        forward_file_name = 'zone.%s' % domain
+        reverse_file_name = 'zone.0.168.192.in-addr.arpa'
+        expected_files = [
+            join(self.dns_conf_dir, forward_file_name),
+            join(self.dns_conf_dir, reverse_file_name),
+        ]
+        self.assertThat(expected_files, AllMatch(FileExists()))
+
+    def test_bind_write_options_sets_up_config(self):
+        # bind_write_configuration_and_zones writes the config file, writes
+        # the zone files, and reloads the dns service.
+        upstream_dns = [
+            factory.make_ipv4_address(),
+            factory.make_ipv4_address(),
+        ]
+        actions.bind_write_options(upstream_dns=upstream_dns)
+        expected_options_file = join(
+            self.dns_conf_dir, MAAS_NAMED_CONF_OPTIONS_INSIDE_NAME)
+        self.assertThat(expected_options_file, FileExists())
+        expected_options_content = dedent("""\
+        forwarders {
+            %s;
+            %s;
+        };
+
+        allow-query { any; };
+        allow-recursion { trusted; };
+        allow-query-cache { trusted; };
+        """)
+        expected_options_content %= tuple(upstream_dns)
+        self.assertThat(
+            expected_options_file,
+            FileContains(expected_options_content))

=== modified file 'src/provisioningserver/dns/tests/test_config.py'
--- src/provisioningserver/dns/tests/test_config.py	2014-11-17 11:02:09 +0000
+++ src/provisioningserver/dns/tests/test_config.py	2015-01-13 18:36:01 +0000
@@ -133,13 +133,14 @@
 
     def test_set_up_options_conf_writes_configuration(self):
         dns_conf_dir = patch_dns_config_path(self)
-        fake_dns = factory.make_ipv4_address()
+        fake_dns = [factory.make_ipv4_address(), factory.make_ipv4_address()]
         set_up_options_conf(upstream_dns=fake_dns)
         target_file = os.path.join(
             dns_conf_dir, MAAS_NAMED_CONF_OPTIONS_INSIDE_NAME)
         self.assertThat(
-            target_file,
-            FileContains(matcher=Contains(fake_dns)))
+            target_file, MatchesAll(*(
+                FileContains(matcher=Contains(address))
+                for address in fake_dns)))
 
     def test_set_up_options_conf_handles_no_upstream_dns(self):
         dns_conf_dir = patch_dns_config_path(self)

=== removed file 'src/provisioningserver/logger/utils.py'
--- src/provisioningserver/logger/utils.py	2014-09-17 08:51:31 +0000
+++ src/provisioningserver/logger/utils.py	1970-01-01 00:00:00 +0000
@@ -1,47 +0,0 @@
-# Copyright 2014 Canonical Ltd.  This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-"""Utilities for logging."""
-
-from __future__ import (
-    absolute_import,
-    print_function,
-    unicode_literals,
-    )
-
-str = None
-
-__metaclass__ = type
-__all__ = [
-    "log_call",
-]
-
-from functools import wraps
-import logging
-
-from provisioningserver.logger.log import get_maas_logger
-
-
-maaslog = get_maas_logger("calls")
-
-
-def log_call(level=logging.INFO):
-    """Log to the maaslog that something happened with a task.
-
-    :param event: The event that we want to log.
-    :param task_name: The name of the task.
-    :**kwargs: A dict of args passed to the task.
-    """
-    def _decorator(func):
-        @wraps(func)
-        def wrapper(*args, **kwargs):
-            arg_string = "%s %s" % (args, kwargs)
-            maaslog.log(
-                level, "Starting task '%s' with args: %s" %
-                (func.__name__, arg_string))
-            func(*args, **kwargs)
-            maaslog.log(
-                level, "Finished task '%s' with args: %s" %
-                (func.__name__, arg_string))
-        return wrapper
-    return _decorator

=== modified file 'src/provisioningserver/rpc/clusterservice.py'
--- src/provisioningserver/rpc/clusterservice.py	2014-12-17 20:55:55 +0000
+++ src/provisioningserver/rpc/clusterservice.py	2015-01-13 18:36:01 +0000
@@ -18,7 +18,6 @@
 
 from functools import partial
 import json
-import logging
 from os import urandom
 import random
 import re
@@ -42,7 +41,6 @@
 from provisioningserver.drivers.hardware.ucsm import probe_and_enlist_ucsm
 from provisioningserver.drivers.hardware.virsh import probe_virsh_and_enlist
 from provisioningserver.logger.log import get_maas_logger
-from provisioningserver.logger.utils import log_call
 from provisioningserver.network import discover_networks
 from provisioningserver.rpc import (
     cluster,
@@ -254,7 +252,6 @@
                 nameservers=nameservers, netmasks=netmasks),
             }
 
-    @log_call(level=logging.DEBUG)
     @cluster.PowerOn.responder
     def power_on(self, system_id, hostname, power_type, context):
         """Turn a node on."""
@@ -264,7 +261,6 @@
         d.addCallback(lambda _: {})
         return d
 
-    @log_call(level=logging.DEBUG)
     @cluster.PowerOff.responder
     def power_off(self, system_id, hostname, power_type, context):
         """Turn a node off."""

=== removed file 'src/provisioningserver/tasks.py'
--- src/provisioningserver/tasks.py	2014-11-14 17:18:10 +0000
+++ src/provisioningserver/tasks.py	1970-01-01 00:00:00 +0000
@@ -1,153 +0,0 @@
-# Copyright 2012-2014 Canonical Ltd.  This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-"""Provisioning server tasks that are run in Celery workers."""
-
-from __future__ import (
-    absolute_import,
-    print_function,
-    unicode_literals,
-    )
-
-str = None
-
-__metaclass__ = type
-__all__ = [
-    'rndc_command',
-    'write_dns_config',
-    'write_dns_zone_config',
-    'write_full_dns_config',
-    ]
-
-from functools import wraps
-from subprocess import CalledProcessError
-
-from celery.app import app_or_default
-from celery.task import task
-from provisioningserver.dns.config import (
-    DNSConfig,
-    execute_rndc_command,
-    set_up_options_conf,
-    )
-from provisioningserver.logger import get_maas_logger
-from provisioningserver.logger.utils import log_call
-
-
-celery_config = app_or_default().conf
-
-maaslog = get_maas_logger("tasks")
-
-
-# The tasks catch bare exceptions in an attempt to circumvent Celery's
-# bizarre exception handling which prints a stack trace but not the
-# error message contained in the exception itself!  The message is
-# printed and then the exception re-raised so that it marks the task as
-# failed - in doing so it logs the stack trace, which is why the code
-# does not do a simple maaslog.exception(exc).
-def log_exception_text(func):
-    """Wrap a function and log any exception text raised."""
-    @wraps(func)
-    def wrapper(*args, **kwargs):
-        try:
-            func(*args, **kwargs)
-        except Exception as e:
-            maaslog.error("%s: %s", func.__name__, unicode(e))
-            raise
-    return wrapper
-
-
-# =====================================================================
-# DNS-related tasks
-# =====================================================================
-
-# How many times should a rndc task be retried?
-RNDC_COMMAND_MAX_RETRY = 10
-
-# How long to wait between rndc tasks retries (in seconds)?
-RNDC_COMMAND_RETRY_DELAY = 2
-
-
-@task(max_retries=RNDC_COMMAND_MAX_RETRY)
-@log_call()
-@log_exception_text
-def rndc_command(arguments, retry=False, callback=None):
-    """Use rndc to execute a command.
-    :param arguments: Argument list passed down to the rndc command.
-    :type arguments : list
-    :param retry: Should this task be retried in case of failure?
-    :type retry: bool
-    :param callback: Callback subtask.
-    :type callback: callable
-    """
-    try:
-        execute_rndc_command(arguments)
-    except CalledProcessError as exc:
-        if retry:
-            return rndc_command.retry(
-                exc=exc, countdown=RNDC_COMMAND_RETRY_DELAY)
-        else:
-            maaslog.error("rndc_command failed: %s", unicode(exc))
-            raise
-    if callback is not None:
-        callback.delay()
-
-
-@task
-@log_call()
-@log_exception_text
-def write_full_dns_config(zones=None, callback=None, **kwargs):
-    """Write out the DNS configuration files: the main configuration
-    file and the zone files.
-    :param zones: List of zones to write.
-    :type zones: list of :class:`DNSZoneData`
-    :param callback: Callback subtask.
-    :type callback: callable
-    :param **kwargs: Keyword args passed to DNSConfig.write_config()
-    """
-    if zones is not None:
-        for zone in zones:
-            zone.write_config()
-    # Write main config file.
-    dns_config = DNSConfig(zones=zones)
-    dns_config.write_config(**kwargs)
-    # Write the included options file.
-    set_up_options_conf(**kwargs)
-    if callback is not None:
-        callback.delay()
-
-
-@task
-@log_call()
-@log_exception_text
-def write_dns_config(zones=(), callback=None, **kwargs):
-    """Write out the DNS configuration file.
-
-    :param zones: List of zones to include as part of the main
-        config.
-    :type zones: list of :class:`DNSZoneData`
-    :param callback: Callback subtask.
-    :type callback: callable
-    :param **kwargs: Keyword args passed to DNSConfig.write_config()
-    """
-    dns_config = DNSConfig(zones=zones)
-    dns_config.write_config(**kwargs)
-    if callback is not None:
-        callback.delay()
-
-
-@task
-@log_call()
-@log_exception_text
-def write_dns_zone_config(zones, callback=None, **kwargs):
-    """Write out DNS zones.
-
-    :param zone: The zone data to write the configuration for.
-    :type zone: :class:`DNSZoneData`
-    :param callback: Callback subtask.
-    :type callback: callable
-    :param **kwargs: Keyword args passed to DNSZoneConfig.write_config()
-    """
-    for zone in zones:
-        zone.write_config()
-    if callback is not None:
-        callback.delay()

=== removed file 'src/provisioningserver/tests/test_tasks.py'
--- src/provisioningserver/tests/test_tasks.py	2014-11-14 17:18:10 +0000
+++ src/provisioningserver/tests/test_tasks.py	1970-01-01 00:00:00 +0000
@@ -1,227 +0,0 @@
-# Copyright 2012-2014 Canonical Ltd.  This software is licensed under the
-# GNU Affero General Public License version 3 (see the file LICENSE).
-
-"""Tests for Celery tasks."""
-
-from __future__ import (
-    absolute_import,
-    print_function,
-    unicode_literals,
-    )
-
-str = None
-
-__metaclass__ = type
-__all__ = []
-
-import os
-import random
-from subprocess import CalledProcessError
-
-import celery
-from celery import states
-from maastesting.celery import CeleryFixture
-from maastesting.factory import factory
-from maastesting.fakemethod import (
-    FakeMethod,
-    MultiFakeMethod,
-    )
-from netaddr import IPNetwork
-from provisioningserver import tasks
-from provisioningserver.dns.config import (
-    MAAS_NAMED_CONF_NAME,
-    MAAS_NAMED_CONF_OPTIONS_INSIDE_NAME,
-    )
-from provisioningserver.dns.testing import patch_dns_config_path
-from provisioningserver.dns.zoneconfig import (
-    DNSForwardZoneConfig,
-    DNSReverseZoneConfig,
-    )
-from provisioningserver.tasks import (
-    rndc_command,
-    RNDC_COMMAND_MAX_RETRY,
-    write_dns_config,
-    write_dns_zone_config,
-    write_full_dns_config,
-    )
-from provisioningserver.testing.testcase import PservTestCase
-from provisioningserver.utils.shell import ExternalProcessError
-from testtools.matchers import (
-    Equals,
-    FileExists,
-    MatchesListwise,
-    )
-
-
-def assertTaskRetried(runner, result, nb_retries, task_name):
-    # In celery version 2.5 (in Saucy) a retried tasks that eventually
-    # succeeds comes out in a 'SUCCESS' state and in 3.1 (in Trusty) is comes
-    # out with a 'RETRY' state.
-    # In both cases the task is successfully retried.
-    if celery.VERSION[0] == 2:
-        runner.assertTrue(result.successful())
-    else:
-        runner.assertEqual(
-            len(runner.celery.tasks), nb_retries)
-        last_task = runner.celery.tasks[0]
-        # The last task succeeded.
-        runner.assertEqual(
-            (last_task['task'].name, last_task['state']),
-            (task_name, states.SUCCESS))
-
-
-class TestDNSTasks(PservTestCase):
-
-    def setUp(self):
-        super(TestDNSTasks, self).setUp()
-        # Patch DNS_CONFIG_DIR so that the configuration files will be
-        # written in a temporary directory.
-        self.dns_conf_dir = self.make_dir()
-        patch_dns_config_path(self, self.dns_conf_dir)
-        # Record the calls to 'execute_rndc_command' (instead of
-        # executing real rndc commands).
-        self.rndc_recorder = FakeMethod()
-        self.patch(tasks, 'execute_rndc_command', self.rndc_recorder)
-        self.celery = self.useFixture(CeleryFixture())
-
-    def test_write_dns_config_writes_file(self):
-        zone_names = [random.randint(1, 100), random.randint(1, 100)]
-        command = factory.make_string()
-        result = write_dns_config.delay(
-            zone_names=zone_names,
-            callback=rndc_command.subtask(args=[command]))
-
-        self.assertThat(
-            (
-                result.successful(),
-                os.path.join(self.dns_conf_dir, MAAS_NAMED_CONF_NAME),
-                self.rndc_recorder.calls,
-            ),
-            MatchesListwise(
-                (
-                    Equals(True),
-                    FileExists(),
-                    Equals([((command,), {})]),
-                )),
-            result)
-
-    def test_write_dns_zone_config_writes_file(self):
-        command = factory.make_string()
-        domain = factory.make_string()
-        network = IPNetwork('192.168.0.3/24')
-        dns_ip = factory.pick_ip_in_network(network)
-        ip = factory.pick_ip_in_network(network)
-        forward_zone = DNSForwardZoneConfig(
-            domain, serial=random.randint(1, 100),
-            mapping={factory.make_string(): [ip]},
-            dns_ip=dns_ip)
-        reverse_zone = DNSReverseZoneConfig(
-            domain, serial=random.randint(1, 100), network=network)
-        result = write_dns_zone_config.delay(
-            zones=[forward_zone, reverse_zone],
-            callback=rndc_command.subtask(args=[command]))
-
-        forward_file_name = 'zone.%s' % domain
-        reverse_file_name = 'zone.0.168.192.in-addr.arpa'
-        self.assertThat(
-            (
-                result.successful(),
-                os.path.join(self.dns_conf_dir, forward_file_name),
-                os.path.join(self.dns_conf_dir, reverse_file_name),
-                self.rndc_recorder.calls,
-            ),
-            MatchesListwise(
-                (
-                    Equals(True),
-                    FileExists(),
-                    FileExists(),
-                    Equals([((command, ), {})]),
-                )),
-            result)
-
-    def test_rndc_command_execute_command(self):
-        command = factory.make_string()
-        result = rndc_command.delay(command)
-
-        self.assertThat(
-            (result.successful(), self.rndc_recorder.calls),
-            MatchesListwise(
-                (
-                    Equals(True),
-                    Equals([((command,), {})]),
-                )))
-
-    def test_rndc_command_can_be_retried(self):
-        # The rndc_command task can be retried.
-        # Simulate a temporary failure.
-        number_of_failures = RNDC_COMMAND_MAX_RETRY
-        raised_exception = CalledProcessError(
-            factory.make_name('exception'), random.randint(100, 200))
-        simulate_failures = MultiFakeMethod(
-            [FakeMethod(failure=raised_exception)] * number_of_failures +
-            [FakeMethod()])
-        self.patch(tasks, 'execute_rndc_command', simulate_failures)
-        command = factory.make_string()
-        result = rndc_command.delay(command, retry=True)
-        assertTaskRetried(
-            self, result, RNDC_COMMAND_MAX_RETRY + 1,
-            'provisioningserver.tasks.rndc_command')
-
-    def test_rndc_command_is_retried_a_limited_number_of_times(self):
-        # If we simulate RNDC_COMMAND_MAX_RETRY + 1 failures, the
-        # task fails.
-        number_of_failures = RNDC_COMMAND_MAX_RETRY + 1
-        raised_exception = ExternalProcessError(
-            random.randint(100, 200), factory.make_name('exception'))
-        simulate_failures = MultiFakeMethod(
-            [FakeMethod(failure=raised_exception)] * number_of_failures +
-            [FakeMethod()])
-        self.patch(tasks, 'execute_rndc_command', simulate_failures)
-        command = factory.make_string()
-        self.assertRaises(
-            ExternalProcessError, rndc_command.delay,
-            command, retry=True)
-
-    def test_write_full_dns_config_sets_up_config(self):
-        # write_full_dns_config writes the config file, writes
-        # the zone files, and reloads the dns service.
-        domain = factory.make_string()
-        network = IPNetwork('192.168.0.3/24')
-        ip = factory.pick_ip_in_network(network)
-        dns_ip = factory.pick_ip_in_network(network)
-        zones = [
-            DNSForwardZoneConfig(
-                domain, serial=random.randint(1, 100),
-                mapping={factory.make_string(): [ip]},
-                dns_ip=dns_ip,
-            ),
-            DNSReverseZoneConfig(
-                domain, serial=random.randint(1, 100), network=network),
-        ]
-        command = factory.make_string()
-        result = write_full_dns_config.delay(
-            zones=zones,
-            callback=rndc_command.subtask(args=[command]),
-            upstream_dns=factory.make_ipv4_address())
-
-        forward_file_name = 'zone.%s' % domain
-        reverse_file_name = 'zone.0.168.192.in-addr.arpa'
-        self.assertThat(
-            (
-                result.successful(),
-                self.rndc_recorder.calls,
-                os.path.join(self.dns_conf_dir, forward_file_name),
-                os.path.join(self.dns_conf_dir, reverse_file_name),
-                os.path.join(self.dns_conf_dir, MAAS_NAMED_CONF_NAME),
-                os.path.join(
-                    self.dns_conf_dir, MAAS_NAMED_CONF_OPTIONS_INSIDE_NAME),
-            ),
-            MatchesListwise(
-                (
-                    Equals(True),
-                    Equals([((command,), {})]),
-                    FileExists(),
-                    FileExists(),
-                    FileExists(),
-                    FileExists(),
-                )))