Merge lp:~lamont/maas/bug-1553176-1.9 into lp:maas/1.9

[LaMont Jones]

Backport parts of r4750 from trunk: BIND compares zone file mtime to last load time and skips the load when the file is older, so our approach to slowly incrementing mtime breaks.

Stop trying to be smarter than BIND about zonefile modification times, since it uses last-load-time and file mtime to determine whether or not it's already loaded.

[Mike Pontillo (mpontillo)]

Looks good; let's wait to land this until the new version of bind makes it from -proposed to -updates.

review: Approve

[LaMont Jones (lamont)]

To follow up on the discussion in IRC, bind9 introduced a change (that landed in trusty) wherein it compares the mtime of the zone file to the system time when the zone was last loaded, and declares the zone to be current if it isn't newer. By setting the mtime into the past and then slowly creeping forward 1 second per update, unless there is a flurry of activity that never stops (and we maintain roughly 1 update per second after that flurry), the DNS will generally be wrong. As such, it makes sense to land this even if bind9 hasn't arrived in -updates yet.

[Mike Pontillo (mpontillo)]

Sounds good. You can go ahead and land it.

[MAAS Lander (maas-lander)]

The attempt to merge lp:~lamont/maas/bug-1553176-1.9 into lp:maas/1.9 failed. Below is the output from the failed tests.

Get:1 http://security.ubuntu.com trusty-security InRelease [65.9 kB]
Ign http://prodstack-zone-1.clouds.archive.ubuntu.com trusty InRelease
Get:2 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports InRelease
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty Release.gpg
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty Release
Get:3 http://security.ubuntu.com trusty-security/main Sources [124 kB]
Get:4 http://security.ubuntu.com trusty-security/universe Sources [48.6 kB]
Get:5 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/main Sources [390 kB]
Get:6 http://security.ubuntu.com trusty-security/main amd64 Packages [579 kB]
Get:7 http://security.ubuntu.com trusty-security/universe amd64 Packages [150 kB]
Get:8 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/restricted Sources [5,911 B]
Get:9 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/universe Sources [173 kB]
Get:10 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/multiverse Sources [7,535 B]
Get:11 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [944 kB]
Get:12 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/restricted amd64 Packages [16.4 kB]
Get:13 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [397 kB]
Get:14 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/multiverse amd64 Packages [14.0 kB]
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/main Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/restricted Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/universe Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/multiverse Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/main amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/restricted amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/universe amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/multiverse amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/main Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/restricted Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/universe Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/multiverse Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/multiverse amd64 Packages
Fetched 2,982 kB in 1s (1,638 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
     --no-...

[MAAS Lander (maas-lander)]

The attempt to merge lp:~lamont/maas/bug-1553176-1.9 into lp:maas/1.9 failed. Below is the output from the failed tests.

Ign http://prodstack-zone-1.clouds.archive.ubuntu.com trusty InRelease
Get:1 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Get:2 http://security.ubuntu.com trusty-security InRelease [65.9 kB]
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports InRelease
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty Release.gpg
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty Release
Get:3 http://security.ubuntu.com trusty-security/main Sources [124 kB]
Get:4 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/main Sources [391 kB]
Get:5 http://security.ubuntu.com trusty-security/universe Sources [49.3 kB]
Get:6 http://security.ubuntu.com trusty-security/main amd64 Packages [579 kB]
Get:7 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/restricted Sources [5,911 B]
Get:8 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/universe Sources [173 kB]
Get:9 http://security.ubuntu.com trusty-security/universe amd64 Packages [151 kB]
Get:10 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/multiverse Sources [7,535 B]
Get:11 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [947 kB]
Get:12 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/restricted amd64 Packages [16.4 kB]
Get:13 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [397 kB]
Get:14 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/multiverse amd64 Packages [14.0 kB]
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/main Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/restricted Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/universe Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/multiverse Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/main amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/restricted amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/universe amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/multiverse amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/main Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/restricted Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/universe Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/multiverse Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/multiverse amd64 Packages
Fetched 2,987 kB in 1s (1,662 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
     --no-...

[MAAS Lander (maas-lander)]

The attempt to merge lp:~lamont/maas/bug-1553176-1.9 into lp:maas/1.9 failed. Below is the output from the failed tests.

Ign http://prodstack-zone-1.clouds.archive.ubuntu.com trusty InRelease
Get:1 http://security.ubuntu.com trusty-security InRelease [65.9 kB]
Get:2 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports InRelease
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty Release.gpg
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty Release
Get:3 http://security.ubuntu.com trusty-security/main Sources [125 kB]
Get:4 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/main Sources [392 kB]
Get:5 http://security.ubuntu.com trusty-security/universe Sources [49.6 kB]
Get:6 http://security.ubuntu.com trusty-security/main amd64 Packages [580 kB]
Get:7 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/restricted Sources [5,911 B]
Get:8 http://security.ubuntu.com trusty-security/universe amd64 Packages [152 kB]
Get:9 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/universe Sources [174 kB]
Get:10 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/multiverse Sources [7,510 B]
Get:11 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [948 kB]
Get:12 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/restricted amd64 Packages [16.4 kB]
Get:13 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [398 kB]
Get:14 http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-updates/multiverse amd64 Packages [14.0 kB]
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/main Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/restricted Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/universe Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/multiverse Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/main amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/restricted amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/universe amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty-backports/multiverse amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/main Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/restricted Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/universe Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/multiverse Sources
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://prodstack-zone-1.clouds.archive.ubuntu.com trusty/multiverse amd64 Packages
Fetched 2,995 kB in 1s (1,555 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
     --no-...

=== modified file 'src/provisioningserver/utils/fs.py'
--- src/provisioningserver/utils/fs.py	2015-10-14 16:27:21 +0000
+++ src/provisioningserver/utils/fs.py	2017-02-08 18:01:00 +0000
@@ -52,10 +52,7 @@
 import sys
 import tempfile
 import threading
-from time import (
-    sleep,
-    time,
-)
+from time import sleep
 
 from provisioningserver.utils import sudo
 from provisioningserver.utils.shell import ExternalProcessError
@@ -207,69 +204,26 @@
         raise
 
 
-def pick_new_mtime(old_mtime=None, starting_age=1000):
-    """Choose a new modification time for a file that needs it updated.
-
-    This function is used to manage the modification time of files
-    for which we need to see an increment in the modification time
-    each time the file is modified.  This is the case for DNS zone
-    files which only get properly reloaded if BIND sees that the
-    modification time is > to the time it has in its database.
-
-    Modification time can have a resolution as low as one second in
-    some relevant environments (we have observed this with ext3).
-    To produce mtime changes regardless, we set a file's modification
-    time in the past when it is first written, and
-    increment it by 1 second on each subsequent write.
-
-    However we also want to be careful not to set the modification time
-    in the future, mostly because BIND does not deal with that very
-    well.
-
-    :param old_mtime: File's previous modification time, as a number
-        with a unity of one second, or None if it did not previously
-        exist.
-    :param starting_age: If the file did not exist previously, set its
-        modification time this many seconds in the past.
-    """
-    now = time()
-    if old_mtime is None:
-        # File is new.  Set modification time in the past to have room for
-        # sub-second modifications.
-        return now - starting_age
-    elif old_mtime + 1 <= now:
-        # There is room to increment the file's mtime by one second
-        # without ending up in the future.
-        return old_mtime + 1
-    else:
-        # We can't increase the file's modification time.  Give up and
-        # return the previous modification time.
-        return old_mtime
-
-
 def incremental_write(content, filename, mode=0o600):
-    """Write the given `content` into the file `filename` and
-    increment the modification time by 1 sec.
+    """Write the given `content` into the file `filename`.  In the past, this
+    would potentially change modification time to arbitrary values.
 
     :param mode: Access permissions for the file.
     """
-    old_mtime = get_mtime(filename)
+    # We used to change modification time on the files, in an attempt to out
+    # smart BIND into loading zones on file systems where time was not granular
+    # enough.  BIND got smarter about how it loads zones and remembers when it
+    # last loaded the zone: either the then-current time, or the mtime of the
+    # file, whichever is later.  When BIND then gets a reload request, it
+    # compares the current time to the loadtime for the domain, and skips it if
+    # the file is not new.  If we set mtime in the past, then zones don't load.
+    # If we set it in the future, then we WILL sometimes hit the race condition
+    # where BIND looks at the time after we atomic_write, but before we manage
+    # to set the time into the future.  N.B.: /etc on filesystems with 1-second
+    # granularity are no longer supported by MAAS.  The good news is that since
+    # 2.6, linux has supported nanosecond-granular time.  As of bind9
+    # 1:9.10.3.dfsg.P2-5, BIND even uses it.
     atomic_write(content, filename, mode=mode)
-    new_mtime = pick_new_mtime(old_mtime)
-    os.utime(filename, (new_mtime, new_mtime))
-
-
-def get_mtime(filename):
-    """Return a file's modification time, or None if it does not exist."""
-    try:
-        return os.stat(filename).st_mtime
-    except OSError as e:
-        if e.errno == errno.ENOENT:
-            # File does not exist.  Be helpful, return None.
-            return None
-        else:
-            # Other failure.  The caller will want to know.
-            raise
 
 
 def sudo_write_file(filename, contents, encoding='utf-8', mode=0o644):

=== modified file 'src/provisioningserver/utils/tests/test_fs.py'
--- src/provisioningserver/utils/tests/test_fs.py	2015-10-14 18:00:50 +0000
+++ src/provisioningserver/utils/tests/test_fs.py	2017-02-08 18:01:00 +0000
@@ -17,7 +17,6 @@
 from base64 import urlsafe_b64encode
 import os
 import os.path
-from random import randint
 import re
 from shutil import rmtree
 import stat
@@ -31,7 +30,6 @@
 
 from maastesting import root
 from maastesting.factory import factory
-from maastesting.fakemethod import FakeMethod
 from maastesting.matchers import (
     MockCalledOnceWith,
     MockCallsMatch,
@@ -52,9 +50,7 @@
     ensure_dir,
     FileLock,
     get_maas_provision_command,
-    get_mtime,
     incremental_write,
-    pick_new_mtime,
     read_text_file,
     RunLock,
     sudo_delete_file,
@@ -264,8 +260,22 @@
         old_mtime = os.stat(filename).st_mtime - 10
         os.utime(filename, (old_mtime, old_mtime))
         incremental_write(content, filename)
-        self.assertAlmostEqual(
-            os.stat(filename).st_mtime, old_mtime + 1, delta=0.01)
+        # should be much closer to new_time than to old_mtime.
+        new_time = time.time()
+        self.assertAlmostEqual(
+            os.stat(filename).st_mtime, new_time, delta=2.0)
+
+    def test_incremental_write_does_not_set_future_time(self):
+        content = factory.make_bytes()
+        filename = self.make_file(contents=factory.make_string())
+        # Pretend that this file is older than it is.  So that
+        # incrementing its mtime won't put it in the future.
+        old_mtime = os.stat(filename).st_mtime + 10
+        os.utime(filename, (old_mtime, old_mtime))
+        incremental_write(content, filename)
+        new_time = time.time()
+        self.assertAlmostEqual(
+            os.stat(filename).st_mtime, new_time, delta=2.0)
 
     def test_incremental_write_sets_permissions(self):
         atomic_file = self.make_file()
@@ -274,54 +284,6 @@
         self.assertEqual(mode, stat.S_IMODE(os.stat(atomic_file).st_mode))
 
 
-class TestGetMTime(MAASTestCase):
-    """Test `get_mtime`."""
-
-    def test_get_mtime_returns_None_for_nonexistent_file(self):
-        nonexistent_file = os.path.join(
-            self.make_dir(), factory.make_name('nonexistent-file'))
-        self.assertIsNone(get_mtime(nonexistent_file))
-
-    def test_get_mtime_returns_mtime(self):
-        existing_file = self.make_file()
-        mtime = os.stat(existing_file).st_mtime - randint(0, 100)
-        os.utime(existing_file, (mtime, mtime))
-        # Some small rounding/representation errors can happen here.
-        # That's just the way of floating-point numbers.  According to
-        # Gavin there's a conversion to fixed-point along the way, which
-        # would raise representability issues.
-        self.assertAlmostEqual(mtime, get_mtime(existing_file), delta=0.00001)
-
-    def test_get_mtime_passes_on_other_error(self):
-        forbidden_file = self.make_file()
-        self.patch(os, 'stat', FakeMethod(failure=OSError("Forbidden file")))
-        self.assertRaises(OSError, get_mtime, forbidden_file)
-
-
-class TestPickNewMTime(MAASTestCase):
-    """Test `pick_new_mtime`."""
-
-    def test_pick_new_mtime_applies_starting_age_to_new_file(self):
-        before = time.time()
-        starting_age = randint(0, 5)
-        recommended_age = pick_new_mtime(None, starting_age=starting_age)
-        now = time.time()
-        self.assertAlmostEqual(
-            now - starting_age,
-            recommended_age,
-            delta=(now - before))
-
-    def test_pick_new_mtime_increments_mtime_if_possible(self):
-        past = time.time() - 2
-        self.assertEqual(past + 1, pick_new_mtime(past))
-
-    def test_pick_new_mtime_refuses_to_move_mtime_into_the_future(self):
-        # Race condition: this will fail if the test gets held up for
-        # a second between readings of the clock.
-        now = time.time()
-        self.assertEqual(now, pick_new_mtime(now))
-
-
 class TestGetMAASProvisionCommand(MAASTestCase):
 
     def test__returns_just_command_for_production(self):