Merge lp:~allenap/maas/dns-serials--bug-1571645--model into lp:~maas-committers/maas/trunk

[Gavin Panella]

New DNSPublication model to ensure that zone serials are consistent across an HA region.

This is the first part of addressing the linked bug, just a new model for system triggers to populate.

[Blake Rouse (blake-rouse)]

Looks good.

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~allenap/maas/dns-serials--bug-1571645--model into lp:maas failed. Below is the output from the failed tests.

Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [92.2 kB]
Hit:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [92.2 kB]
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Fetched 184 kB in 0s (384 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-mock python3-netaddr python3-netifaces python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
bind9 is already the newest version (1:9.10.3.dfsg.P4-8).
bind9utils is already the newest version (1:9.10.3.dfsg.P4-8).
build-essential is already the newest version (12.1ubuntu2).
bzr is already the newest version (2.7.0-2ubuntu1).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
dnsutils is already the newest version (1:9.10.3.dfsg.P4-8).
firefox is already the newest version (45.0.2+build1-0ubuntu1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu12).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libj...

[MAAS Lander (maas-lander)]

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

[MAAS Lander (maas-lander)]

The attempt to merge lp:~allenap/maas/dns-serials--bug-1571645--model into lp:maas failed. Below is the output from the failed tests.

Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [92.2 kB]
Hit:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [92.2 kB]
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Fetched 184 kB in 0s (433 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-mock python3-netaddr python3-netifaces python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
bind9 is already the newest version (1:9.10.3.dfsg.P4-8).
bind9utils is already the newest version (1:9.10.3.dfsg.P4-8).
build-essential is already the newest version (12.1ubuntu2).
bzr is already the newest version (2.7.0-2ubuntu1).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
dnsutils is already the newest version (1:9.10.3.dfsg.P4-8).
firefox is already the newest version (45.0.2+build1-0ubuntu1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu12).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libj...

[Gavin Panella (allenap)]

Spurious failure that lp:~allenap/maas/maas-crochet-run-test will help to address. For now, trying again.

=== added file 'src/maasserver/migrations/builtin/maasserver/0055_dns_publications.py'
--- src/maasserver/migrations/builtin/maasserver/0055_dns_publications.py	1970-01-01 00:00:00 +0000
+++ src/maasserver/migrations/builtin/maasserver/0055_dns_publications.py	2016-04-26 16:41:19 +0000
@@ -0,0 +1,28 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+import django.core.validators
+from django.db import (
+    migrations,
+    models,
+)
+import maasserver.models.dnspublication
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('maasserver', '0054_controller'),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='DNSPublication',
+            fields=[
+                ('id', models.AutoField(auto_created=True, verbose_name='ID', serialize=False, primary_key=True)),
+                ('serial', models.IntegerField(editable=False, default=maasserver.models.dnspublication.next_serial, validators=(django.core.validators.MinValueValidator(1), django.core.validators.MaxValueValidator(4294967295)))),
+                ('created', models.DateTimeField(auto_now_add=True)),
+                ('source', models.CharField(blank=True, editable=False, max_length=255, help_text='A brief explanation why DNS was published.')),
+            ],
+        ),
+    ]

=== modified file 'src/maasserver/models/__init__.py'
--- src/maasserver/models/__init__.py	2016-04-15 22:14:33 +0000
+++ src/maasserver/models/__init__.py	2016-04-26 16:41:19 +0000
@@ -4,61 +4,69 @@
 """Model export and helpers for maasserver."""
 
 __all__ = [
-    'BMC',
     'Bcache',
     'BlockDevice',
+    'BMC',
+    'BondInterface',
     'BootResource',
     'BootResourceFile',
     'BootResourceSet',
     'BootSource',
     'BootSourceCache',
     'BootSourceSelection',
+    'BridgeInterface',
     'CacheSet',
     'ComponentError',
     'Config',
     'Controller',
+    'Device',
     'DHCPSnippet',
     'DNSData',
+    'DNSPublication',
     'DNSResource',
-    'Device',
     'Domain',
     'Event',
+    'EventType',
     'Fabric',
     'FanNetwork',
     'FileStorage',
     'Filesystem',
     'FilesystemGroup',
+    'Interface',
     'IPRange',
-    'Interface',
     'LargeFile',
     'LicenseKey',
+    'logger',
     'Machine',
     'Node',
+    'NodeGroupToRackController',
     'OwnerData',
     'Partition',
     'PartitionTable',
     'PhysicalBlockDevice',
     'PhysicalInterface',
+    'RackController',
     'RAID',
-    'RackController',
     'RegionController',
     'RegionControllerProcess',
     'RegionControllerProcessEndpoint',
     'RegionRackRPCConnection',
+    'Service',
+    'Space',
     'SSHKey',
     'SSLKey',
-    'Service',
-    'Space',
+    'StaticIPAddress',
     'Subnet',
     'Tag',
     'Template',
+    'UnknownInterface',
     'UserProfile',
-    'VLAN',
     'VersionedTextFile',
     'VirtualBlockDevice',
+    'VLAN',
+    'VLANInterface',
     'VolumeGroup',
     'Zone',
-    'logger',
     ]
 
 from django.contrib.auth.backends import ModelBackend
@@ -87,6 +95,7 @@
 from maasserver.models.config import Config
 from maasserver.models.dhcpsnippet import DHCPSnippet
 from maasserver.models.dnsdata import DNSData
+from maasserver.models.dnspublication import DNSPublication
 from maasserver.models.dnsresource import DNSResource
 from maasserver.models.domain import Domain
 from maasserver.models.event import Event
@@ -147,57 +156,6 @@
 from maasserver.utils import ignore_unused
 from piston3.doc import HandlerDocumentation
 
-# Suppress warning about symbols being imported, but only used for
-# export in __all__.
-ignore_unused(
-    BMC,
-    Bcache,
-    BondInterface,
-    BridgeInterface,
-    BootResource,
-    BootResourceFile,
-    BootResourceSet,
-    CacheSet,
-    ComponentError,
-    Config,
-    Controller,
-    DHCPSnippet,
-    Event,
-    EventType,
-    Fabric,
-    FileStorage,
-    Filesystem,
-    FilesystemGroup,
-    IPRange,
-    Interface,
-    LargeFile,
-    LicenseKey,
-    NodeGroupToRackController,
-    OwnerData,
-    Partition,
-    PartitionTable,
-    RAID,
-    RackController,
-    RegionController,
-    RegionControllerProcess,
-    RegionControllerProcessEndpoint,
-    SSHKey,
-    Service,
-    StaticIPAddress,
-    Tag,
-    Template,
-    UnknownInterface,
-    UserProfile,
-    VLAN,
-    VLANInterface,
-    VersionedTextFile,
-    VirtualBlockDevice,
-    VolumeGroup,
-    Zone,
-    logger,
-)
-
-
 # Connect the 'create_user' method to the post save signal of User.
 post_save.connect(create_user, sender=User)
 

=== added file 'src/maasserver/models/dnspublication.py'
--- src/maasserver/models/dnspublication.py	1970-01-01 00:00:00 +0000
+++ src/maasserver/models/dnspublication.py	2016-04-26 16:41:19 +0000
@@ -0,0 +1,74 @@
+# Copyright 2016 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""DNS publication model objects."""
+
+__all__ = [
+    "DNSPublication",
+]
+
+from django.core.validators import (
+    MaxValueValidator,
+    MinValueValidator,
+)
+from django.db.models import (
+    Manager,
+    Model,
+)
+from django.db.models.fields import (
+    CharField,
+    DateTimeField,
+    IntegerField,
+)
+from maasserver import DefaultMeta
+from maasserver.dns.config import zone_serial
+
+
+def next_serial():
+    return next(zone_serial)
+
+
+class DNSPublicationManager(Manager):
+    """Manager for DNS publishing records."""
+
+    def get_most_recent(self):
+        """Return the most recently inserted `DNSPublication`."""
+        return self.order_by("-id")[0]
+
+    def collect_garbage(self):
+        """Delete all but the most recently inserted `DNSPublication`."""
+        self.filter(id__lt=self.get_most_recent().id).delete()
+
+
+class DNSPublication(Model):
+    """A row in this table denotes a DNS publication request.
+
+    Typically this will be populated by a trigger within the database. A
+    listeners in regiond will be notified and consult the most recent record
+    in this table. This way we can consistently publish zones with the same
+    serial in an HA environment, and newly starting regiond processes can
+    immediately be consistent with their peers.
+    """
+
+    class Meta(DefaultMeta):
+        """Needed for South to recognize this model."""
+
+    objects = DNSPublicationManager()
+
+    # The serial number with which to publish the zone. We don't use the
+    # primary key for this because zone serials are allowed to cycle.
+    serial = IntegerField(
+        editable=False, null=False, default=next_serial, unique=False,
+        validators=(
+            MinValueValidator(zone_serial.minvalue),
+            MaxValueValidator(zone_serial.maxvalue),
+        ))
+
+    # This field is informational.
+    created = DateTimeField(
+        editable=False, null=False, auto_now=False, auto_now_add=True)
+
+    # This field is informational.
+    source = CharField(
+        editable=False, max_length=255, null=False, blank=True,
+        help_text="A brief explanation why DNS was published.")

=== added file 'src/maasserver/models/tests/test_dnspublication.py'
--- src/maasserver/models/tests/test_dnspublication.py	1970-01-01 00:00:00 +0000
+++ src/maasserver/models/tests/test_dnspublication.py	2016-04-26 16:41:19 +0000
@@ -0,0 +1,84 @@
+# Copyright 2016 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Tests for general DNS models."""
+
+__all__ = []
+
+from datetime import (
+    datetime,
+    timedelta,
+)
+from random import randint
+
+from maasserver.models.dnspublication import DNSPublication
+from maasserver.testing.factory import factory
+from maasserver.testing.testcase import MAASServerTestCase
+from testtools.matchers import (
+    Equals,
+    HasLength,
+    IsInstance,
+    MatchesAll,
+    MatchesStructure,
+    Not,
+)
+
+
+class TestDNSPublication(MAASServerTestCase):
+    """Test the `DNSPublication` model."""
+
+    def test_create_empty(self):
+        pub = DNSPublication()
+        pub.save()
+        self.assertThat(
+            pub, MatchesStructure(
+                serial=IsInstance(int),
+                created=IsInstance(datetime),
+                source=Equals(""),
+            ))
+
+    def test_create_with_values(self):
+        serial = randint(1, 5000)
+        created = datetime.now() - timedelta(minutes=1098)
+        source = factory.make_name("source")
+        pub = DNSPublication(serial=serial, created=created, source=source)
+        pub.save()
+        self.assertThat(
+            pub, MatchesStructure(
+                serial=Equals(serial),
+                created=MatchesAll(
+                    IsInstance(datetime),
+                    # `created` is always set; given values are ignored.
+                    Not(Equals(created)),
+                    first_only=True,
+                ),
+                source=Equals(source),
+            ))
+
+
+class TestDNSPublicationManager(MAASServerTestCase):
+    """Test `DNSPublicationManager`."""
+
+    def test_get_most_recent_returns_record_with_highest_id(self):
+        DNSPublication(serial=3).save()
+        DNSPublication(serial=30).save()
+        DNSPublication(serial=10).save()
+        self.assertThat(
+            DNSPublication.objects.get_most_recent(),
+            MatchesStructure(serial=Equals(10)))
+
+    def test_get_most_recent_crashes_when_no_publications(self):
+        # This is okay because we're going to ensure (using a migration) that
+        # there is never less than one publication in the table. If this crash
+        # happens we have bigger problems.
+        self.assertRaises(IndexError, DNSPublication.objects.get_most_recent)
+
+    def test_collect_garbage_removes_all_but_most_recent_record(self):
+        for serial in range(10):
+            DNSPublication(serial=serial).save()
+        self.assertThat(DNSPublication.objects.all(), HasLength(10))
+        DNSPublication.objects.collect_garbage()
+        self.assertThat(DNSPublication.objects.all(), HasLength(1))
+        self.assertThat(
+            DNSPublication.objects.get_most_recent(),
+            MatchesStructure(serial=Equals(serial)))