Merge into 1.9 : bug-1553176-1.9 : Code : MAAS

Status:

Merged

Approved by:

Mike Pontillo on 2017-02-17

Approved revision:

4598

Merged at revision:

4598

Proposed branch:

lp:~lamont/maas/bug-1553176-1.9

Merge into:

lp:maas/1.9

Diff against target:

210 lines (+32/-116)

2 files modified

src/provisioningserver/utils/fs.py (+16/-62)
src/provisioningserver/utils/tests/test_fs.py (+16/-54)

To merge this branch:

bzr merge lp:~lamont/maas/bug-1553176-1.9

Undecided

Fix Released

Link a bug report

Reviewer	Review Type	Date Requested	Status
Mike Pontillo (community)		2017-02-03	Approve on 2017-02-04
Review via email: mp+316384@code.launchpad.net

Commit message

Backport parts of r4750 from trunk: BIND compares zone file mtime to last load time and skips the load when the file is older, so our approach to slowly incrementing mtime breaks.

Description of the change

Stop trying to be smarter than BIND about zonefile modification times, since it uses last-load-time and file mtime to determine whether or not it's already loaded.

Mike Pontillo (mpontillo) wrote on 2017-02-04:

#

Looks good; let's wait to land this until the new version of bind makes it from -proposed to -updates.

review: Approve

LaMont Jones (lamont) wrote on 2017-02-04:

#

To follow up on the discussion in IRC, bind9 introduced a change (that landed in trusty) wherein it compares the mtime of the zone file to the system time when the zone was last loaded, and declares the zone to be current if it isn't newer. By setting the mtime into the past and then slowly creeping forward 1 second per update, unless there is a flurry of activity that never stops (and we maintain roughly 1 update per second after that flurry), the DNS will generally be wrong. As such, it makes sense to land this even if bind9 hasn't arrived in -updates yet.

Mike Pontillo (mpontillo) wrote on 2017-02-04:

#

Sounds good. You can go ahead and land it.

MAAS Lander (maas-lander) wrote on 2017-02-04:

#

Download full text (83.9 KiB)

The attempt to merge lp:~lamont/maas/bug-1553176-1.9 into lp:maas/1.9 failed. Below is the output from the failed tests.

MAAS Lander (maas-lander) wrote on 2017-02-08:

#

Download full text (1.4 MiB)

The attempt to merge lp:~lamont/maas/bug-1553176-1.9 into lp:maas/1.9 failed. Below is the output from the failed tests.

MAAS Lander (maas-lander) wrote on 2017-02-17:

#

Download full text (1.0 MiB)

The attempt to merge lp:~lamont/maas/bug-1553176-1.9 into lp:maas/1.9 failed. Below is the output from the failed tests.

MAAS

Merge lp:~lamont/maas/bug-1553176-1.9 into lp:maas/1.9

Commit message

Description of the change

Preview Diff

Subscribers

 === modified file 'src/provisioningserver/utils/fs.py'
 --- src/provisioningserver/utils/fs.py	2015-10-14 16:27:21 +0000
 +++ src/provisioningserver/utils/fs.py	2017-02-08 18:01:00 +0000
@@ -52,10 +52,7 @@
  import sys
  import tempfile
  import threading
--from time import (
--    sleep,
--    time,
--)
++from time import sleep
  from provisioningserver.utils import sudo
  from provisioningserver.utils.shell import ExternalProcessError
@@ -207,69 +204,26 @@
          raise
--def pick_new_mtime(old_mtime=None, starting_age=1000):
--    """Choose a new modification time for a file that needs it updated.
--
--    This function is used to manage the modification time of files
--    for which we need to see an increment in the modification time
--    each time the file is modified.  This is the case for DNS zone
--    files which only get properly reloaded if BIND sees that the
--    modification time is > to the time it has in its database.
--
--    Modification time can have a resolution as low as one second in
--    some relevant environments (we have observed this with ext3).
--    To produce mtime changes regardless, we set a file's modification
--    time in the past when it is first written, and
--    increment it by 1 second on each subsequent write.
--
--    However we also want to be careful not to set the modification time
--    in the future, mostly because BIND does not deal with that very
--    well.
--
--    :param old_mtime: File's previous modification time, as a number
--        with a unity of one second, or None if it did not previously
--        exist.
--    :param starting_age: If the file did not exist previously, set its
--        modification time this many seconds in the past.
--    """
--    now = time()
--    if old_mtime is None:
--        # File is new.  Set modification time in the past to have room for
--        # sub-second modifications.
--        return now - starting_age
--    elif old_mtime + 1 <= now:
--        # There is room to increment the file's mtime by one second
--        # without ending up in the future.
--        return old_mtime + 1
--    else:
--        # We can't increase the file's modification time.  Give up and
--        # return the previous modification time.
--        return old_mtime
--
--
  def incremental_write(content, filename, mode=0o600):
--    """Write the given `content` into the file `filename` and
--    increment the modification time by 1 sec.
++    """Write the given `content` into the file `filename`.  In the past, this
++    would potentially change modification time to arbitrary values.
      :param mode: Access permissions for the file.
      """
--    old_mtime = get_mtime(filename)
++    # We used to change modification time on the files, in an attempt to out
++    # smart BIND into loading zones on file systems where time was not granular
++    # enough.  BIND got smarter about how it loads zones and remembers when it
++    # last loaded the zone: either the then-current time, or the mtime of the
++    # file, whichever is later.  When BIND then gets a reload request, it
++    # compares the current time to the loadtime for the domain, and skips it if
++    # the file is not new.  If we set mtime in the past, then zones don't load.
++    # If we set it in the future, then we WILL sometimes hit the race condition
++    # where BIND looks at the time after we atomic_write, but before we manage
++    # to set the time into the future.  N.B.: /etc on filesystems with 1-second
++    # granularity are no longer supported by MAAS.  The good news is that since
++    # 2.6, linux has supported nanosecond-granular time.  As of bind9
++    # 1:9.10.3.dfsg.P2-5, BIND even uses it.
      atomic_write(content, filename, mode=mode)
--    new_mtime = pick_new_mtime(old_mtime)
--    os.utime(filename, (new_mtime, new_mtime))
--
--
--def get_mtime(filename):
--    """Return a file's modification time, or None if it does not exist."""
--    try:
--        return os.stat(filename).st_mtime
--    except OSError as e:
--        if e.errno == errno.ENOENT:
--            # File does not exist.  Be helpful, return None.
--            return None
--        else:
--            # Other failure.  The caller will want to know.
--            raise
  def sudo_write_file(filename, contents, encoding='utf-8', mode=0o644):
 === modified file 'src/provisioningserver/utils/tests/test_fs.py'
 --- src/provisioningserver/utils/tests/test_fs.py	2015-10-14 18:00:50 +0000
 +++ src/provisioningserver/utils/tests/test_fs.py	2017-02-08 18:01:00 +0000
@@ -17,7 +17,6 @@
  from base64 import urlsafe_b64encode
  import os
  import os.path
--from random import randint
  import re
  from shutil import rmtree
  import stat
@@ -31,7 +30,6 @@
  from maastesting import root
  from maastesting.factory import factory
--from maastesting.fakemethod import FakeMethod
  from maastesting.matchers import (
      MockCalledOnceWith,
      MockCallsMatch,
@@ -52,9 +50,7 @@
      ensure_dir,
      FileLock,
      get_maas_provision_command,
--    get_mtime,
      incremental_write,
--    pick_new_mtime,
      read_text_file,
      RunLock,
      sudo_delete_file,
@@ -264,8 +260,22 @@
          old_mtime = os.stat(filename).st_mtime - 10
          os.utime(filename, (old_mtime, old_mtime))
          incremental_write(content, filename)
--        self.assertAlmostEqual(
--            os.stat(filename).st_mtime, old_mtime + 1, delta=0.01)
++        # should be much closer to new_time than to old_mtime.
++        new_time = time.time()
++        self.assertAlmostEqual(
++            os.stat(filename).st_mtime, new_time, delta=2.0)
++
++    def test_incremental_write_does_not_set_future_time(self):
++        content = factory.make_bytes()
++        filename = self.make_file(contents=factory.make_string())
++        # Pretend that this file is older than it is.  So that
++        # incrementing its mtime won't put it in the future.
++        old_mtime = os.stat(filename).st_mtime + 10
++        os.utime(filename, (old_mtime, old_mtime))
++        incremental_write(content, filename)
++        new_time = time.time()
++        self.assertAlmostEqual(
++            os.stat(filename).st_mtime, new_time, delta=2.0)
      def test_incremental_write_sets_permissions(self):
          atomic_file = self.make_file()
@@ -274,54 +284,6 @@
          self.assertEqual(mode, stat.S_IMODE(os.stat(atomic_file).st_mode))
--class TestGetMTime(MAASTestCase):
--    """Test `get_mtime`."""
--
--    def test_get_mtime_returns_None_for_nonexistent_file(self):
--        nonexistent_file = os.path.join(
--            self.make_dir(), factory.make_name('nonexistent-file'))
--        self.assertIsNone(get_mtime(nonexistent_file))
--
--    def test_get_mtime_returns_mtime(self):
--        existing_file = self.make_file()
--        mtime = os.stat(existing_file).st_mtime - randint(0, 100)
--        os.utime(existing_file, (mtime, mtime))
--        # Some small rounding/representation errors can happen here.
--        # That's just the way of floating-point numbers.  According to
--        # Gavin there's a conversion to fixed-point along the way, which
--        # would raise representability issues.
--        self.assertAlmostEqual(mtime, get_mtime(existing_file), delta=0.00001)
--
--    def test_get_mtime_passes_on_other_error(self):
--        forbidden_file = self.make_file()
--        self.patch(os, 'stat', FakeMethod(failure=OSError("Forbidden file")))
--        self.assertRaises(OSError, get_mtime, forbidden_file)
--
--
--class TestPickNewMTime(MAASTestCase):
--    """Test `pick_new_mtime`."""
--
--    def test_pick_new_mtime_applies_starting_age_to_new_file(self):
--        before = time.time()
--        starting_age = randint(0, 5)
--        recommended_age = pick_new_mtime(None, starting_age=starting_age)
--        now = time.time()
--        self.assertAlmostEqual(
--            now - starting_age,
--            recommended_age,
--            delta=(now - before))
--
--    def test_pick_new_mtime_increments_mtime_if_possible(self):
--        past = time.time() - 2
--        self.assertEqual(past + 1, pick_new_mtime(past))
--
--    def test_pick_new_mtime_refuses_to_move_mtime_into_the_future(self):
--        # Race condition: this will fail if the test gets held up for
--        # a second between readings of the clock.
--        now = time.time()
--        self.assertEqual(now, pick_new_mtime(now))
--
--
  class TestGetMAASProvisionCommand(MAASTestCase):
      def test__returns_just_command_for_production(self):