Merge lp:~ltrager/maas/lp1603563 into lp:~maas-committers/maas/trunk

[Lee Trager]

Reset status_expires when a non-monitored status is set.

In LP1554636 we have multiple users who are reporting that when releasing a deployed node the node goes to failed releasing state. When you look through the attached logs you can see the node goes into a failed releasing state 1 second after the release request comes in. I haven't been able to reproduce this myself but I *think* I found a race condition which is causing this to happen. Node._release was setting the status_expires time in a post_commit hook before the status was set.

What I believe is happening is the status gets set, then the status_monitor service runs using an old expired_status, sees its expired and puts the node into a FAILED_RELEASING state. This happens before the post_commit hook to update the node status is run.

What this MP does is always set the status_expires field to None on save if the status is not monitored.

[Mike Pontillo (mpontillo)]

Looks good to me! (Needs a commit message though.)

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~ltrager/maas/lp1603563 into lp:maas failed. Below is the output from the failed tests.

Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB]
Hit:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [95.7 kB]
Get:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease [92.2 kB]
Get:5 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates/main Sources [201 kB]
Get:6 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [413 kB]
Fetched 896 kB in 0s (1,626 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind avahi-utils bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common isc-dhcp-server libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-attr python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
authbind is already the newest version (2.1.1+nmu1).
avahi-utils is already the newest version (0.6.32~rc+dfsg-1ubuntu2).
build-essential is already the newest version (12.1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
make is already the newest version (4.1-6).
postgresql is already the newest version (9.5+173).
pxelinux is already the newest version (3:6.03+dfsg-11ubuntu1).
python-f...

=== modified file 'src/maasserver/models/node.py'
--- src/maasserver/models/node.py	2016-10-22 05:40:08 +0000
+++ src/maasserver/models/node.py	2016-10-26 18:28:44 +0000
@@ -123,6 +123,7 @@
     COMMISSIONING_LIKE_STATUSES,
     get_failed_status,
     is_failed_status,
+    MONITORED_STATUSES,
     NODE_FAILURE_MONITORED_STATUS_TIMEOUTS,
     NODE_TRANSITIONS,
 )
@@ -1462,7 +1463,15 @@
                     self.hostname, error)
 
     def save(self, *args, **kwargs):
+        # Reset the status_expires if not a monitored status. This prevents
+        # a race condition seen in LP1603563 where an old status_expires caused
+        # the node to do in a FAILED_RELEASING state due to an old
+        # status_expire being set.
+        if self.status not in MONITORED_STATUSES:
+            self.status_expires = None
+
         super(Node, self).save(*args, **kwargs)
+
         # We let hostname be blank for the initial save, but fix it before the
         # save completes.  This is because set_random_hostname() operates by
         # trying to re-save the node with a random hostname, and retrying until

=== modified file 'src/maasserver/models/tests/test_node.py'
--- src/maasserver/models/tests/test_node.py	2016-10-22 05:50:06 +0000
+++ src/maasserver/models/tests/test_node.py	2016-10-26 18:28:44 +0000
@@ -2624,6 +2624,15 @@
             "Invalid transition: Retired -> Allocated.",
             node.save)
 
+    def test_save_resets_status_expires_on_non_monitored_status(self):
+        # Regression test for LP:1603563
+        node = factory.make_Node(status=NODE_STATUS.RELEASING)
+        Node._set_status_expires(node.system_id, 60)
+        node.status = NODE_STATUS.READY
+        node.save()
+        node = reload_object(node)
+        self.assertIsNone(node.status_expires)
+
     def test_full_clean_checks_architecture_for_installable_nodes(self):
         device = factory.make_Device(architecture='')
         # Set type here so we don't cause exception while creating object