Merge lp:~allenap/maas/dns-serials--bug-1571645--sequence into lp:~maas-committers/maas/trunk

[Gavin Panella]

Move zone_serial into the dnspublication module.

Also associate it in the database, ensure that the sequence in the database is set to cycle, and ratchet up the controls on imports in South migrations.

[Mike Pontillo (mpontillo)]

Looks good. Just one minor nit below.

review: Approve

[Gavin Panella (allenap)]

Thanks Mike!

[MAAS Lander (maas-lander)]

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

[MAAS Lander (maas-lander)]

The attempt to merge lp:~allenap/maas/dns-serials--bug-1571645--sequence into lp:maas failed. Below is the output from the failed tests.

Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [92.2 kB]
Hit:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [93.3 kB]
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:5 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [34.4 kB]
Get:6 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [7,972 B]
Fetched 228 kB in 0s (476 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-mock python3-netaddr python3-netifaces python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
build-essential is already the newest version (12.1ubuntu2).
bzr is already the newest version (2.7.0-2ubuntu1).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu12).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already ...

[Gavin Panella (allenap)]

Merged migrations.

=== modified file 'src/maasserver/api/domains.py'
--- src/maasserver/api/domains.py	2016-03-31 23:34:55 +0000
+++ src/maasserver/api/domains.py	2016-04-29 11:36:07 +0000
@@ -8,14 +8,12 @@
     operation,
     OperationsHandler,
 )
-from maasserver.dns.config import (
-    dns_force_reload,
-    zone_serial,
-)
+from maasserver.dns.config import dns_force_reload
 from maasserver.enum import NODE_PERMISSION
 from maasserver.exceptions import MAASAPIValidationError
 from maasserver.forms_domain import DomainForm
 from maasserver.models import Domain
+from maasserver.models.dnspublication import zone_serial
 from maasserver.sequence import INT_MAX
 from piston3.utils import rc
 

=== modified file 'src/maasserver/api/tests/test_domains.py'
--- src/maasserver/api/tests/test_domains.py	2016-03-28 13:54:47 +0000
+++ src/maasserver/api/tests/test_domains.py	2016-04-29 11:36:07 +0000
@@ -11,7 +11,7 @@
 
 from django.conf import settings
 from django.core.urlresolvers import reverse
-from maasserver.dns.config import zone_serial
+from maasserver.models.dnspublication import zone_serial
 from maasserver.models.domain import Domain
 from maasserver.sequence import INT_MAX
 from maasserver.testing.api import APITestCase

=== modified file 'src/maasserver/dns/config.py'
--- src/maasserver/dns/config.py	2016-03-31 23:34:55 +0000
+++ src/maasserver/dns/config.py	2016-04-29 11:36:07 +0000
@@ -13,12 +13,9 @@
 from maasserver.dns.zonegenerator import ZoneGenerator
 from maasserver.enum import RDNS_MODE
 from maasserver.models.config import Config
+from maasserver.models.dnspublication import zone_serial
 from maasserver.models.domain import Domain
 from maasserver.models.subnet import Subnet
-from maasserver.sequence import (
-    INT_MAX,
-    Sequence,
-)
 from provisioningserver.dns.actions import (
     bind_reload,
     bind_reload_with_retries,
@@ -32,13 +29,6 @@
 maaslog = get_maas_logger("dns")
 
 
-# A DNS zone's serial is a 32-bit integer.  Also, we start with the
-# value 1 because 0 has special meaning for some DNS servers.  Even if
-# we control the DNS server we use, better safe than sorry.
-zone_serial = Sequence(
-    'maasserver_zone_serial_seq', increment=1, minvalue=1, maxvalue=INT_MAX)
-
-
 def current_zone_serial():
     return '%0.10d' % zone_serial.current()
 

=== modified file 'src/maasserver/migrations/builtin/maasserver/0026_create_zone_serial_sequence.py'
--- src/maasserver/migrations/builtin/maasserver/0026_create_zone_serial_sequence.py	2016-02-10 20:24:35 +0000
+++ src/maasserver/migrations/builtin/maasserver/0026_create_zone_serial_sequence.py	2016-04-29 11:36:07 +0000
@@ -3,8 +3,10 @@
 
 from django.db import migrations
 
-# At the time of writing this should match the definition in
-# maasserver.dns.config.zone_serial, and vice-versa.
+# At the time of writing this should match the definition of zone_serial in
+# maasserver.dns.config. This is done with a snippet of PL/pgSQL to support
+# PostgreSQL before 9.5, which does not support `CREATE SEQUENCE ... IF NOT
+# EXISTS`.
 sequence_create = ("""\
 DO
 $$

=== added file 'src/maasserver/migrations/builtin/maasserver/0056_zone_serial_ownership.py'
--- src/maasserver/migrations/builtin/maasserver/0056_zone_serial_ownership.py	1970-01-01 00:00:00 +0000
+++ src/maasserver/migrations/builtin/maasserver/0056_zone_serial_ownership.py	2016-04-29 11:36:07 +0000
@@ -0,0 +1,30 @@
+# -*- coding: utf-8 -*-
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('maasserver', '0055_dns_publications'),
+    ]
+
+    operations = [
+        # Associate maasserver_zone_serial_seq with maasserver_dnspublication.
+        migrations.RunSQL(
+            # Forwards.
+            ("ALTER SEQUENCE maasserver_zone_serial_seq "
+             "OWNED BY maasserver_dnspublication.serial"),
+            # Reverse.
+            ("ALTER SEQUENCE maasserver_zone_serial_seq "
+             "OWNED BY NONE"),
+        ),
+        # Ensure that maasserver_zone_serial_seq cycles. It does not seem
+        # likely that anyone will exhaust all 2^32 zone serials, but if they
+        # do we should behave as the DNS server will expect, i.e. cycle.
+        migrations.RunSQL(
+            # Forwards.
+            "ALTER SEQUENCE maasserver_zone_serial_seq CYCLE",
+            # Reverse.
+            "ALTER SEQUENCE maasserver_zone_serial_seq NO CYCLE",
+        ),
+    ]

=== added file 'src/maasserver/migrations/builtin/maasserver/0057_merge.py'
--- src/maasserver/migrations/builtin/maasserver/0057_merge.py	1970-01-01 00:00:00 +0000
+++ src/maasserver/migrations/builtin/maasserver/0057_merge.py	2016-04-29 11:36:07 +0000
@@ -0,0 +1,13 @@
+# -*- coding: utf-8 -*-
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('maasserver', '0056_add_description_to_fabric_and_space'),
+        ('maasserver', '0056_zone_serial_ownership'),
+    ]
+
+    operations = [
+    ]

=== modified file 'src/maasserver/migrations/south/migrations/0011_add_dns_zone_serial_sequence.py'
--- src/maasserver/migrations/south/migrations/0011_add_dns_zone_serial_sequence.py	2015-11-25 15:32:43 +0000
+++ src/maasserver/migrations/south/migrations/0011_add_dns_zone_serial_sequence.py	2016-04-29 11:36:07 +0000
@@ -6,19 +6,38 @@
 import datetime
 
 from django.db import models
-from maasserver.dns.config import zone_serial
 from south.db import db
 from south.v2 import SchemaMigration
 
+# Stolen from the future, namely 0026_create_zone_serial_sequence in Django's
+# new built-in migrations. This is done with a snippet of PL/pgSQL to support
+# PostgreSQL before 9.5, which does not support `CREATE SEQUENCE ... IF NOT
+# EXISTS`.
+sequence_create = ("""\
+DO
+$$
+BEGIN
+    CREATE SEQUENCE maasserver_zone_serial_seq
+    MINVALUE {minvalue:d} MAXVALUE {maxvalue:d};
+EXCEPTION WHEN duplicate_table THEN
+    -- Do nothing, it already exists.
+END
+$$ LANGUAGE plpgsql;
+""").format(minvalue=1, maxvalue=((2 ** 32) - 1))
+
+sequence_drop = (
+    "DROP SEQUENCE IF EXISTS maasserver_zone_serial_seq"
+)
+
 
 class Migration(SchemaMigration):
 
     def forwards(self, orm):
-        zone_serial.create()
+        db.execute(sequence_create)
 
 
     def backwards(self, orm):
-        zone_serial.drop()
+        db.execute(sequence_drop)
 
 
     models = {

=== modified file 'src/maasserver/models/dnspublication.py'
--- src/maasserver/models/dnspublication.py	2016-04-21 16:32:47 +0000
+++ src/maasserver/models/dnspublication.py	2016-04-29 11:36:07 +0000
@@ -21,7 +21,17 @@
     IntegerField,
 )
 from maasserver import DefaultMeta
-from maasserver.dns.config import zone_serial
+from maasserver.sequence import (
+    INT_MAX,
+    Sequence,
+)
+
+# A DNS zone's serial is a 32-bit integer. Also, we start with the value 1
+# because 0 has special meaning for some DNS servers. Even if we control the
+# DNS server we use, better safe than sorry.
+zone_serial = Sequence(
+    'maasserver_zone_serial_seq', increment=1, minvalue=1, maxvalue=INT_MAX,
+    owner='maasserver_dnspublication.serial')
 
 
 def next_serial():

=== modified file 'src/maasserver/triggers/system.py'
--- src/maasserver/triggers/system.py	2016-04-11 16:23:26 +0000
+++ src/maasserver/triggers/system.py	2016-04-29 11:36:07 +0000
@@ -15,7 +15,7 @@
 
 from textwrap import dedent
 
-from maasserver.dns.config import zone_serial
+from maasserver.models.dnspublication import zone_serial
 from maasserver.triggers import (
     register_procedure,
     register_trigger,

=== modified file 'src/maasserver/triggers/tests/helper.py'
--- src/maasserver/triggers/tests/helper.py	2016-03-31 23:34:55 +0000
+++ src/maasserver/triggers/tests/helper.py	2016-04-29 11:36:07 +0000
@@ -8,7 +8,6 @@
 
 from crochet import wait_for
 from django.contrib.auth.models import User
-from maasserver.dns.config import zone_serial
 from maasserver.enum import (
     INTERFACE_TYPE,
     NODE_TYPE,
@@ -18,6 +17,7 @@
 from maasserver.models.cacheset import CacheSet
 from maasserver.models.dhcpsnippet import DHCPSnippet
 from maasserver.models.dnsdata import DNSData
+from maasserver.models.dnspublication import zone_serial
 from maasserver.models.dnsresource import DNSResource
 from maasserver.models.domain import Domain
 from maasserver.models.event import Event

=== modified file 'src/maasserver/triggers/tests/test_system.py'
--- src/maasserver/triggers/tests/test_system.py	2016-03-31 23:34:55 +0000
+++ src/maasserver/triggers/tests/test_system.py	2016-04-29 11:36:07 +0000
@@ -8,7 +8,7 @@
 from contextlib import closing
 
 from django.db import connection
-from maasserver.dns.config import zone_serial
+from maasserver.models.dnspublication import zone_serial
 from maasserver.testing.testcase import MAASServerTestCase
 from maasserver.triggers.system import register_system_triggers
 from maasserver.utils.orm import psql_array

=== modified file 'utilities/check-imports'
--- utilities/check-imports	2016-03-28 13:54:47 +0000
+++ utilities/check-imports	2016-04-29 11:36:07 +0000
@@ -360,7 +360,18 @@
     ),
     (
         RegionMigrationsSouth,
-        RegionControllerRule | Allow("south|south.**"),
+        Rule(
+            Allow("django|django.**"),
+            Allow("lxml|lxml.**"),
+            Allow("netaddr|netaddr.**"),
+            Allow("south|south.**"),
+            Allow(StandardLibraries),
+
+            # XXX: The following three permissions are temporary;
+            # they MUST be eliminated before MAAS 2.0 release.
+            Allow("maasserver|maasserver.**"),
+            Allow("metadataserver|metadataserver.**"),
+        )
     ),
     (
         RegionMigrationsBuiltin,