Merge lp:~lamont/maas/dns-failures-1.10 into lp:maas/1.10

[LaMont Jones]

Backport fixes for DNS test failures from 2.0 (trunk). BIND 9.10 (now in xenial) does not reload zone files that are older than the time of the last reload of that file, so we need to keep time correctly on the zone files, and possibly let it creep into the future in order to make sure that the file gets reloaded when there is heavy churn.

Backport fixes for DNS test failures from 2.0 (trunk). BIND 9.10 (now in xenial) does not reload zone files that are older than the time of the last reload of that file, so we need to keep time correctly on the zone files, and possibly let it creep into the future in order to make sure that the file gets reloaded when there is heavy churn.

[Lee Trager (ltrager)]

LGTM!

review: Approve

[MAAS Lander (maas-lander)]

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

[MAAS Lander (maas-lander)]

The attempt to merge lp:~lamont/maas/dns-failures-1.10 into lp:maas/1.10 failed. Below is the output from the failed tests.

Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 authbind bind9 bind9utils build-essential bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-apt python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-djorm-ext-pgarray python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-mock python3-netaddr python3-netifaces python3-oauth python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-seamicroclient python3-simplejson python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-1ubuntu1).
authbind is already the newest version (2.1.1+nmu1).
bind9 is already the newest version (1:9.10.3.dfsg.P2-4).
bind9utils is already the newest version (1:9.10.3.dfsg.P2-4).
build-essential is already the newest version (12.1ubuntu2).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu2).
dh-apport is already the newest version (2.20-0ubuntu3).
dh-systemd is already the newest version (1.28ubuntu2).
distro-info is already the newest version (0.14build1).
dnsutils is already the newest version (1:9.10.3.dfsg.P2-4).
firefox is already the newest version (44.0.2+build1-0ubuntu1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.0-1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu7).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-jquery-hotkeys is already the newest version (0~20130707+git2d51e3a9+dfsg-2ubuntu1).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
libpq...

[MAAS Lander (maas-lander)]

The attempt to merge lp:~lamont/maas/dns-failures-1.10 into lp:maas/1.10 failed. Below is the output from the failed tests.

Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Get:2 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial InRelease [95.8 kB]
Hit:3 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:5 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main Sources [1,109 kB]
Get:6 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1,443 kB]
Get:7 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/main Translation-en [842 kB]
Get:8 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7,347 kB]
Fetched 10.8 MB in 2s (3,900 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 authbind bind9 bind9utils build-essential bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-apt python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-djorm-ext-pgarray python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-mock python3-netaddr python3-netifaces python3-oauth python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-seamicroclient python3-simplejson python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-1ubuntu1).
authbind is already the newest version (2.1.1+nmu1).
bind9 is already the newest version (1:9.10.3.dfsg.P2-4).
bind9utils is already the newest version (1:9.10.3.dfsg.P2-4).
build-essential is already the newest version (12.1ubuntu2).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu2).
dh-apport is already the newest version (2.20-0ubuntu3).
dh-systemd is already the newest version (1.28ubuntu2).
distro-info is already the newest version (0.14build1).
dnsutils is already the newest version (1:9.10.3.dfsg.P2-4).
firefox is already the newest version (44.0.2+build1-0ubuntu1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already ...

=== modified file 'src/maasserver/dns/tests/test_config.py'
--- src/maasserver/dns/tests/test_config.py	2015-12-10 10:31:31 +0000
+++ src/maasserver/dns/tests/test_config.py	2016-02-26 01:38:09 +0000
@@ -413,6 +413,8 @@
     def assertDNSMatches(self, hostname, domain, ip, version=4):
         # A forward lookup on the hostname returns the IP address.
         fqdn = "%s.%s" % (hostname, domain)
+        # Give BIND enough time to process the rndc request.
+        time.sleep(0.2)
         forward_lookup_result = self.dig_resolve(fqdn, version=version)
         self.expectThat(
             forward_lookup_result, Contains(ip),

=== modified file 'src/provisioningserver/dns/tests/test_config.py'
--- src/provisioningserver/dns/tests/test_config.py	2015-12-01 18:12:59 +0000
+++ src/provisioningserver/dns/tests/test_config.py	2016-02-26 01:38:09 +0000
@@ -168,7 +168,8 @@
 class TestRNDCUtilities(MAASTestCase):
 
     def test_generate_rndc_returns_configurations(self):
-        rndc_content, named_content = generate_rndc()
+        rndc_content, named_content = generate_rndc(
+            include_default_controls=False)
         # rndc_content and named_content look right.
         self.assertIn('# Start of rndc.conf', rndc_content)
         self.assertIn('controls {', named_content)

=== modified file 'src/provisioningserver/utils/fs.py'
--- src/provisioningserver/utils/fs.py	2015-12-01 18:12:59 +0000
+++ src/provisioningserver/utils/fs.py	2016-02-26 01:38:09 +0000
@@ -201,7 +201,7 @@
         raise
 
 
-def pick_new_mtime(old_mtime=None, starting_age=1000):
+def pick_new_mtime(old_mtime=None):
     """Choose a new modification time for a file that needs it updated.
 
     This function is used to manage the modification time of files
@@ -216,29 +216,20 @@
     time in the past when it is first written, and
     increment it by 1 second on each subsequent write.
 
-    However we also want to be careful not to set the modification time
-    in the future, mostly because BIND does not deal with that very
-    well.
-
     :param old_mtime: File's previous modification time, as a number
         with a unity of one second, or None if it did not previously
         exist.
-    :param starting_age: If the file did not exist previously, set its
-        modification time this many seconds in the past.
     """
     now = time()
-    if old_mtime is None:
+    if old_mtime is None or old_mtime + 1 < now:
         # File is new.  Set modification time in the past to have room for
         # sub-second modifications.
-        return now - starting_age
-    elif old_mtime + 1 <= now:
-        # There is room to increment the file's mtime by one second
-        # without ending up in the future.
+        return now
+    else:
+        # Let the file timestamp march into the future, so that BIND will
+        # reload the zone.  Our future selves will notice that mtime is enough
+        # before the then-now to go back to having it match us.
         return old_mtime + 1
-    else:
-        # We can't increase the file's modification time.  Give up and
-        # return the previous modification time.
-        return old_mtime
 
 
 def incremental_write(content, filename, mode=0o600):

=== modified file 'src/provisioningserver/utils/tests/test_fs.py'
--- src/provisioningserver/utils/tests/test_fs.py	2015-12-01 18:12:59 +0000
+++ src/provisioningserver/utils/tests/test_fs.py	2016-02-26 01:38:09 +0000
@@ -258,8 +258,9 @@
         old_mtime = os.stat(filename).st_mtime - 10
         os.utime(filename, (old_mtime, old_mtime))
         incremental_write(content, filename)
+        new_time = time.time()
         self.assertAlmostEqual(
-            os.stat(filename).st_mtime, old_mtime + 1, delta=0.01)
+            os.stat(filename).st_mtime, new_time, delta=0.01)
 
     def test_incremental_write_sets_permissions(self):
         atomic_file = self.make_file()
@@ -295,25 +296,16 @@
 class TestPickNewMTime(MAASTestCase):
     """Test `pick_new_mtime`."""
 
-    def test_pick_new_mtime_applies_starting_age_to_new_file(self):
-        before = time.time()
-        starting_age = randint(0, 5)
-        recommended_age = pick_new_mtime(None, starting_age=starting_age)
-        now = time.time()
-        self.assertAlmostEqual(
-            now - starting_age,
-            recommended_age,
-            delta=(now - before))
-
     def test_pick_new_mtime_increments_mtime_if_possible(self):
-        past = time.time() - 2
-        self.assertEqual(past + 1, pick_new_mtime(past))
+        now = time.time()
+        new_time = pick_new_mtime(now - 2)
+        self.assertAlmostEqual(new_time, now, delta=0.0001)
 
-    def test_pick_new_mtime_refuses_to_move_mtime_into_the_future(self):
+    def test_pick_new_mtime_happily_moves_mtime_into_the_future(self):
         # Race condition: this will fail if the test gets held up for
         # a second between readings of the clock.
         now = time.time()
-        self.assertEqual(now, pick_new_mtime(now))
+        self.assertEqual(now + 1, pick_new_mtime(now))
 
 
 class TestGetMAASProvisionCommand(MAASTestCase):