Merge lp:~blake-rouse/maas/query-power-on-status-change into lp:~maas-committers/maas/trunk

[Blake Rouse]

Perform PowerQuery on a node 20 seconds after a status change.

This is done when nodes make a status transition that do not perform a power action. e.g. Commissioning -> Ready, does not perform a power action, updating the power status 20 seconds after changes gives the machine time to shutdown, while providing a more up-to-date power status.

[Julian Edwards (julian-edwards)]

Looks good, but marked needs-fixing because we can avoid the signal handler by using Node.clean_status() instead. See comments inline.

review: Needs Fixing

[Julian Edwards (julian-edwards)]

 review: approve

On Wednesday 22 Oct 2014 19:54:52 you wrote:
> This is the same approach that is done to log to the event log when a node
> status transitions. This was discussed and recommended by rvba, to do it
> this way.

Ah yes, rvb loves his signals. Ok let's leave that for now.

Thanks for the other changes, looks good to go.

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~blake-rouse/maas/query-power-on-status-change into lp:maas failed. Below is the output from the failed tests.

Ign http://security.ubuntu.com trusty-security InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Get:2 http://security.ubuntu.com trusty-security Release [59.7 kB]
Ign http://nova.clouds.archive.ubuntu.com trusty-updates InRelease
Hit http://nova.clouds.archive.ubuntu.com trusty Release.gpg
Get:3 http://nova.clouds.archive.ubuntu.com trusty-updates Release.gpg [933 B]
Hit http://nova.clouds.archive.ubuntu.com trusty Release
Get:4 http://nova.clouds.archive.ubuntu.com trusty-updates Release [59.7 kB]
Get:5 http://security.ubuntu.com trusty-security/main Sources [48.3 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main Sources
Get:6 http://security.ubuntu.com trusty-security/universe Sources [11.2 kB]
Get:7 http://security.ubuntu.com trusty-security/main amd64 Packages [149 kB]
Get:8 http://security.ubuntu.com trusty-security/universe amd64 Packages [50.4 kB]
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Sources
Hit http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en
Get:9 http://nova.clouds.archive.ubuntu.com trusty-updates/main Sources [129 kB]
Get:10 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Sources [88.2 kB]
Get:11 http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [343 kB]
Get:12 http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [214 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en
Ign http://nova.clouds.archive.ubuntu.com trusty/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 1,155 kB in 3s (354 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
     --no-install-recommends install apache2 authbind bind9 bind9utils build-essential bzr-builddeb curl daemontools debhelper dh-apport distro-info dnsutils firefox freeipmi-tools gjs ipython isc-dhcp-common libjs-raphael libjs-yui3-full libjs-yui3-min libpq-dev make pep8 postgresql pyflakes python-amqplib python-bzrlib python-celery python-convoy python-crochet python-cssselect python-curtin python-dev python-distro-info python-django python-django-piston python-django-south python-djorm-ext-pgarray python-docutils python-extras python-fixtures python-flake8 python-formencode python-hivex python-httplib2 python-jinja2 python-jsonschema python-lockfile python-lxml python-mimeparse python-mock python-netaddr python-netifaces python-nose python-oauth python-oops python-oops-amqp python-oops-datedir-repo python-oops-twisted python-oops-ws...

=== modified file 'src/maasserver/models/__init__.py'
--- src/maasserver/models/__init__.py	2014-09-29 15:26:08 +0000
+++ src/maasserver/models/__init__.py	2014-10-23 13:33:12 +0000
@@ -192,3 +192,6 @@
 
 from maasserver import event_connect
 ignore_unused(event_connect)
+
+from maasserver import node_query
+ignore_unused(node_query)

=== added file 'src/maasserver/node_query.py'
--- src/maasserver/node_query.py	1970-01-01 00:00:00 +0000
+++ src/maasserver/node_query.py	2014-10-23 13:33:12 +0000
@@ -0,0 +1,108 @@
+# Copyright 2014 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Query power status on node state changes."""
+
+from __future__ import (
+    absolute_import,
+    print_function,
+    unicode_literals,
+    )
+
+str = None
+
+__metaclass__ = type
+__all__ = []
+
+from datetime import timedelta
+
+from crochet import TimeoutError
+from maasserver.enum import POWER_STATE
+from maasserver.models import Node
+from maasserver.node_status import QUERY_TRANSITIONS
+from maasserver.rpc import getClientFor
+from maasserver.signals import connect_to_field_change
+from provisioningserver.logger import get_maas_logger
+from provisioningserver.power.poweraction import (
+    PowerActionFail,
+    UnknownPowerType,
+    )
+from provisioningserver.rpc.cluster import PowerQuery
+from provisioningserver.rpc.exceptions import NoConnectionsAvailable
+from provisioningserver.utils.twisted import synchronous
+from twisted.internet import reactor
+from twisted.internet.threads import deferToThread
+
+
+maaslog = get_maas_logger('node_query')
+
+# Amount of time to wait after a node status has been updated to
+# perform a power query.
+WAIT_TO_QUERY = timedelta(seconds=20)
+
+
+@synchronous
+def update_power_state_of_node(system_id):
+    """Update the power state of the given node."""
+    try:
+        node = Node.objects.get(system_id=system_id)
+    except Node.DoesNotExist:
+        # Just in case the Node has been deleted,
+        # before we get to this point.
+        return
+
+    try:
+        client = getClientFor(node.nodegroup.uuid)
+    except NoConnectionsAvailable:
+        maaslog.error(
+            "Unable to get RPC connection for cluster '%s' (%s)",
+            node.nodegroup.cluster_name, node.nodegroup.uuid)
+        return
+
+    try:
+        power_info = node.get_effective_power_info()
+    except UnknownPowerType:
+        return
+    if not power_info.can_be_started:
+        # Power state is not queryable
+        return
+
+    call = client(
+        PowerQuery, system_id=system_id, hostname=node.hostname,
+        power_type=power_info.power_type,
+        context=power_info.power_parameters)
+    try:
+        state = call.wait(30).get("state", POWER_STATE.ERROR)
+    except (TimeoutError, NotImplementedError, PowerActionFail):
+        state = POWER_STATE.ERROR
+    node.power_state = state
+    node.save()
+
+
+def wait_to_update_power_state_of_node(system_id, clock=reactor):
+    """Wait "WAIT_TO_QUERY" amount of time then update the power state of
+    the given node."""
+    clock.callLater(
+        WAIT_TO_QUERY.total_seconds(), deferToThread,
+        update_power_state_of_node, system_id)
+
+
+@synchronous
+def signal_update_power_state_of_node(instance, old_values, **kwargs):
+    """Updates the power state of a node, when its status changes."""
+    node = instance
+    [old_status] = old_values
+
+    # Check if this transition should even check for a new power state.
+    if old_status not in QUERY_TRANSITIONS:
+        return
+    if node.status not in QUERY_TRANSITIONS[old_status]:
+        return
+
+    # Update the power state of the node, after the waiting period.
+    wait_to_update_power_state_of_node(node.system_id)
+
+
+connect_to_field_change(
+    signal_update_power_state_of_node,
+    Node, ['status'], delete=False)

=== modified file 'src/maasserver/node_status.py'
--- src/maasserver/node_status.py	2014-10-15 13:55:21 +0000
+++ src/maasserver/node_status.py	2014-10-23 13:33:12 +0000
@@ -183,6 +183,26 @@
     NODE_STATUS.DISK_ERASING,
 ]
 
+# Node state transitions that perform query actions. This is to keep the
+# power state of the node up-to-date when transitions occur that do not
+# perform a power action directly.
+QUERY_TRANSITIONS = {
+    None: [
+        NODE_STATUS.NEW,
+        ],
+    NODE_STATUS.COMMISSIONING: [
+        NODE_STATUS.FAILED_COMMISSIONING,
+        NODE_STATUS.READY,
+        ],
+    NODE_STATUS.DEPLOYING: [
+        NODE_STATUS.FAILED_DEPLOYMENT,
+        NODE_STATUS.DEPLOYED,
+    ],
+    NODE_STATUS.DISK_ERASING: [
+        NODE_STATUS.FAILED_DISK_ERASING,
+        ],
+    }
+
 
 def get_failed_status(status):
     """Returns the failed status corresponding to the given status.

=== added file 'src/maasserver/tests/test_node_query.py'
--- src/maasserver/tests/test_node_query.py	1970-01-01 00:00:00 +0000
+++ src/maasserver/tests/test_node_query.py	2014-10-23 13:33:12 +0000
@@ -0,0 +1,132 @@
+# Copyright 2014 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Tests for node power status query when state changes."""
+
+from __future__ import (
+    absolute_import,
+    print_function,
+    unicode_literals,
+    )
+
+str = None
+
+__metaclass__ = type
+__all__ = []
+
+import random
+
+from maasserver.node_status import (
+    get_failed_status,
+    NODE_STATUS,
+    )
+from maasserver.rpc.testing.fixtures import MockLiveRegionToClusterRPCFixture
+from maasserver.testing.eventloop import (
+    RegionEventLoopFixture,
+    RunningEventLoopFixture,
+    )
+from maasserver.testing.factory import factory
+from maasserver.testing.orm import reload_object
+from maasserver.testing.testcase import MAASServerTestCase
+from maastesting.matchers import (
+    MockCalledOnceWith,
+    MockNotCalled,
+    )
+from provisioningserver.power.poweraction import PowerActionFail
+from provisioningserver.rpc import cluster as cluster_module
+from provisioningserver.rpc.testing import always_succeed_with
+from twisted.internet.task import Clock
+
+
+class TestStatusQueryEvent(MAASServerTestCase):
+
+    def setUp(self):
+        super(TestStatusQueryEvent, self).setUp()
+        # Circular imports.
+        from maasserver import node_query
+        self.node_query = node_query
+
+    def test_changing_status_of_node_emits_event(self):
+        mock_update = self.patch(
+            self.node_query, 'wait_to_update_power_state_of_node')
+        old_status = NODE_STATUS.COMMISSIONING
+        node = factory.make_Node(status=old_status, power_type='virsh')
+        node.status = get_failed_status(old_status)
+        node.save()
+        self.assertThat(
+            mock_update,
+            MockCalledOnceWith(node.system_id))
+
+    def test_changing_not_tracked_status_of_node_doesnt_emit_event(self):
+        mock_update = self.patch(
+            self.node_query, "wait_to_update_power_state_of_node")
+        old_status = NODE_STATUS.ALLOCATED
+        node = factory.make_Node(status=old_status, power_type="virsh")
+        node.status = NODE_STATUS.DEPLOYING
+        node.save()
+        self.assertThat(
+            mock_update,
+            MockNotCalled())
+
+
+class TestWaitToUpdatePowerStateOfNode(MAASServerTestCase):
+
+    def setUp(self):
+        super(TestWaitToUpdatePowerStateOfNode, self).setUp()
+        # Circular imports.
+        from maasserver import node_query
+        self.node_query = node_query
+
+    def test__calls_update_power_state_of_node_after_wait_time(self):
+        mock_defer_to_thread = self.patch(self.node_query, 'deferToThread')
+        node = factory.make_Node(power_type="virsh")
+        clock = Clock()
+        self.node_query.wait_to_update_power_state_of_node(
+            node.system_id, clock=clock)
+
+        self.expectThat(mock_defer_to_thread, MockNotCalled())
+        clock.advance(self.node_query.WAIT_TO_QUERY.total_seconds())
+        self.expectThat(
+            mock_defer_to_thread,
+            MockCalledOnceWith(
+                self.node_query.update_power_state_of_node, node.system_id))
+
+
+class TestUpdatePowerStateOfNode(MAASServerTestCase):
+
+    def setUp(self):
+        super(TestUpdatePowerStateOfNode, self).setUp()
+        # Circular imports.
+        from maasserver import node_query
+        self.node_query = node_query
+
+    def prepare_rpc(self, nodegroup, side_effect):
+        self.useFixture(RegionEventLoopFixture("rpc"))
+        self.useFixture(RunningEventLoopFixture())
+        self.rpc_fixture = self.useFixture(MockLiveRegionToClusterRPCFixture())
+        protocol = self.rpc_fixture.makeCluster(
+            nodegroup, cluster_module.PowerQuery)
+        protocol.PowerQuery.side_effect = side_effect
+
+    def test__updates_node_power_state(self):
+        node = factory.make_Node(power_type="virsh")
+        random_state = random.choice(["on", "off"])
+        self.prepare_rpc(
+            node.nodegroup,
+            side_effect=always_succeed_with({"state": random_state}))
+        self.node_query.update_power_state_of_node(node.system_id)
+        self.assertEqual(random_state, reload_object(node).power_state)
+
+    def test__handles_deleted_node(self):
+        node = factory.make_Node(power_type="virsh")
+        node.delete()
+        self.node_query.update_power_state_of_node(node.system_id)
+        #: Test is that no error is raised
+
+    def test__updates_node_power_state_to_error_if_failure(self):
+        node = factory.make_Node(power_type="virsh")
+        self.prepare_rpc(
+            node.nodegroup,
+            side_effect=PowerActionFail())
+        self.node_query.update_power_state_of_node(node.system_id)
+        self.assertEqual("error", reload_object(node).power_state)