Merge lp:~ltrager/maas/disable_delete_rack into lp:~maas-committers/maas/trunk

[Lee Trager]

Disable and stop the rackd service if running on deletion

Upon deletion of a rack controller if a secondary rack controller is still connected send an RPC call which disables and stops the maas-rackd service. A user can reenable it using systemctl. As maas-dhcpd{,6}.service is bound to maas-rackd.service stopping maas-rackd also stops maas-dhcpd{,6}.

Last Friday when testing this branch the delete failed because there still was a relation between the rack controller and RegionRackRPCConnection. Since merging trunk I haven't seen this error but I'm wondering if I should search and delete any of these relations.

For maas-rackd to be able to disable and stop the maas-rackd service I had to add permission to 99-maas-sudoers. Review at https://code.launchpad.net/~ltrager/maas/disable_delete_rackd_sudo/+merge/293593

[Blake Rouse (blake-rouse)]

This looks really good. Nice work.

review: Approve

[MAAS Lander (maas-lander)]

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

[MAAS Lander (maas-lander)]

The attempt to merge lp:~ltrager/maas/disable_delete_rack into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [94.5 kB]
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Fetched 94.5 kB in 0s (179 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-mock python3-netaddr python3-netifaces python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
build-essential is already the newest version (12.1ubuntu2).
bzr is already the newest version (2.7.0-2ubuntu1).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu12).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
libpq-dev is already the newest version (9.5.2-1).
make is already the newest version (4.1-6).
postgresql is already the newest version (9.5+173...

[MAAS Lander (maas-lander)]

The attempt to merge lp:~ltrager/maas/disable_delete_rack into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://security.ubuntu.com/ubuntu xenial-security InRelease
Get:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [94.5 kB]
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Fetched 94.5 kB in 0s (217 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-mock python3-netaddr python3-netifaces python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
build-essential is already the newest version (12.1ubuntu2).
bzr is already the newest version (2.7.0-2ubuntu1).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu12).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
libpq-dev is already the newest version (9.5.2-1).
make is already the newest version (4.1-6).
postgresql is already the newest version (9.5+173...

[MAAS Lander (maas-lander)]

The attempt to merge lp:~ltrager/maas/disable_delete_rack into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [94.5 kB]
Hit:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu xenial-security InRelease
Get:5 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [64.3 kB]
Get:6 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [15.0 kB]
Fetched 174 kB in 0s (397 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
build-essential is already the newest version (12.1ubuntu2).
bzr is already the newest version (2.7.0-2ubuntu1).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu12).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
l...

[MAAS Lander (maas-lander)]

The attempt to merge lp:~ltrager/maas/disable_delete_rack into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [94.5 kB]
Hit:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu xenial-security InRelease
Fetched 94.5 kB in 0s (231 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
build-essential is already the newest version (12.1ubuntu2).
bzr is already the newest version (2.7.0-2ubuntu1).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu12).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
libpq-dev is already the newest version (9.5.2-1).
make is already the newest version (4.1-6).
postgresql is already the newest version (9...

[Andres Rodriguez (andreserl)]

ERROR: provisioningserver.rpc.tests.test_clusterservice.TestClusterProtocol_Identify.test_identify_is_registered
----------------------------------------------------------------------
testtools.testresult.real._StringException: logged-error: {{{
Traceback (most recent call last):
provisioningserver.rpc.exceptions.CannotDisableAndShutoffRackd: failure
}}}

twisted-log: {{{
2016-05-10 18:30:20+0000 [-] Unhandled error in Deferred:
2016-05-10 18:30:20+0000 [-] Unhandled Error
 Traceback (most recent call last):
 Failure: provisioningserver.rpc.exceptions.CannotDisableAndShutoffRackd: failure
}}}

----------------------------------------------------------------------
Ran 2369 tests in 44.324s

=== modified file 'src/maasserver/models/node.py'
--- src/maasserver/models/node.py	2016-05-09 21:18:43 +0000
+++ src/maasserver/models/node.py	2016-05-10 23:45:39 +0000
@@ -162,6 +162,7 @@
 from provisioningserver.power import QUERY_POWER_TYPES
 from provisioningserver.rpc.cluster import (
     AddChassis,
+    DisableAndShutoffRackd,
     IsImportBootImagesRunning,
     RefreshRackControllerInfo,
 )
@@ -3658,15 +3659,6 @@
 
     def delete(self):
         """Delete this rack controller."""
-        # Avoid circular dependency.
-        from maasserver.models import RegionRackRPCConnection
-        connections = RegionRackRPCConnection.objects.filter(
-            rack_controller=self)
-        if len(connections) != 0:
-            raise ValidationError(
-                "Unable to delete %s as it's currently connected to one or "
-                "more regions." % self.hostname)
-
         primary_vlans = VLAN.objects.filter(primary_rack=self)
         if len(primary_vlans) != 0:
             raise ValidationError(
@@ -3675,6 +3667,15 @@
                 (self.hostname,
                     ', '.join([str(vlan) for vlan in primary_vlans])))
 
+        try:
+            client = getClientFor(self.system_id, timeout=1)
+            call = client(DisableAndShutoffRackd)
+            call.wait(30)
+        except NoConnectionsAvailable:
+            # NoConnectionsAvailable is always thrown. Either because the rack
+            # is currently disconnected or rackd was killed
+            pass
+
         for vlan in VLAN.objects.filter(secondary_rack=self):
             vlan.secondary_rack = None
             vlan.save()

=== modified file 'src/maasserver/models/tests/test_node.py'
--- src/maasserver/models/tests/test_node.py	2016-05-09 23:54:19 +0000
+++ src/maasserver/models/tests/test_node.py	2016-05-10 23:45:39 +0000
@@ -137,10 +137,12 @@
 from provisioningserver.power.schema import JSON_POWER_TYPE_PARAMETERS
 from provisioningserver.rpc.cluster import (
     AddChassis,
+    DisableAndShutoffRackd,
     IsImportBootImagesRunning,
     RefreshRackControllerInfo,
 )
 from provisioningserver.rpc.exceptions import (
+    CannotDisableAndShutoffRackd,
     NoConnectionsAvailable,
     UnknownPowerType,
 )
@@ -6992,11 +6994,34 @@
         rackcontroller.delete()
         self.assertIsNone(reload_object(rackcontroller))
 
-    def test_prevents_delete_when_connected(self):
-        rackcontroller = factory.make_RackController()
-        mock_filter = self.patch(RegionRackRPCConnection.objects, 'filter')
-        mock_filter.return_value = [rackcontroller]
-        self.assertRaises(ValidationError, rackcontroller.delete)
+    def test_disables_and_disconn_when_secondary_connected(self):
+        rackcontroller = factory.make_RackController()
+        factory.make_VLAN(secondary_rack=rackcontroller)
+
+        self.useFixture(RegionEventLoopFixture("rpc"))
+        self.useFixture(RunningEventLoopFixture())
+        fixture = self.useFixture(MockLiveRegionToClusterRPCFixture())
+        protocol = fixture.makeCluster(
+            rackcontroller, DisableAndShutoffRackd)
+        protocol.DisableAndShutoffRackd.return_value = defer.succeed({})
+
+        rackcontroller.delete()
+        self.expectThat(protocol.DisableAndShutoffRackd, MockCalledOnce())
+
+    def test_disables_and_disconn_when_secondary_connected_fails(self):
+        rackcontroller = factory.make_RackController()
+        factory.make_VLAN(secondary_rack=rackcontroller)
+
+        self.useFixture(RegionEventLoopFixture("rpc"))
+        self.useFixture(RunningEventLoopFixture())
+        fixture = self.useFixture(MockLiveRegionToClusterRPCFixture())
+        protocol = fixture.makeCluster(
+            rackcontroller, DisableAndShutoffRackd)
+        protocol.DisableAndShutoffRackd.return_value = defer.fail(
+            CannotDisableAndShutoffRackd())
+
+        self.assertRaises(CannotDisableAndShutoffRackd, rackcontroller.delete)
+        self.expectThat(protocol.DisableAndShutoffRackd, MockCalledOnce())
 
     def test_prevents_delete_when_primary_rack(self):
         rackcontroller = factory.make_RackController()

=== modified file 'src/provisioningserver/rpc/cluster.py'
--- src/provisioningserver/rpc/cluster.py	2016-04-29 22:12:18 +0000
+++ src/provisioningserver/rpc/cluster.py	2016-05-10 23:45:39 +0000
@@ -478,3 +478,15 @@
         (b"protocol", amp.Unicode(optional=True)),
     ]
     errors = {}
+
+
+class DisableAndShutoffRackd(amp.Command):
+    """Disable and shutdown the rackd service.
+
+    :since: 2.0
+    """
+    arguments = []
+    errors = {
+        exceptions.CannotDisableAndShutoffRackd: (
+            b"CannotDisableAndShutoffRackd"),
+    }

=== modified file 'src/provisioningserver/rpc/clusterservice.py'
--- src/provisioningserver/rpc/clusterservice.py	2016-04-29 22:12:18 +0000
+++ src/provisioningserver/rpc/clusterservice.py	2016-05-10 23:45:39 +0000
@@ -72,6 +72,10 @@
     get_maas_id,
     set_maas_id,
 )
+from provisioningserver.utils.shell import (
+    call_and_check,
+    ExternalProcessError,
+)
 from provisioningserver.utils.twisted import (
     DeferredValue,
     synchronous,
@@ -460,6 +464,26 @@
             maaslog.error(message)
         return {}
 
+    @cluster.DisableAndShutoffRackd.responder
+    def disable_and_shutoff_rackd(self):
+        """DisableAndShutoffRackd()
+
+        Implementation of
+        :py:class:`~provisioningserver.rpc.cluster.DisableAndShutoffRackd`.
+        """
+        maaslog.info("Rack deleted, disabling rackd service")
+        try:
+            # We can't use the --now flag as if the maas-rackd service is on
+            # but not enabled the service won't be stopped
+            call_and_check(
+                ['sudo', 'systemctl', 'disable', 'maas-rackd'])
+            call_and_check(
+                ['sudo', 'systemctl', 'stop', 'maas-rackd'])
+        except ExternalProcessError as e:
+            maaslog.error("Unable to disable and stop maas-rackd service")
+            raise exceptions.CannotDisableAndShutoffRackd(e.output_as_unicode)
+        return {}
+
 
 @implementer(IConnection)
 class ClusterClient(Cluster):

=== modified file 'src/provisioningserver/rpc/exceptions.py'
--- src/provisioningserver/rpc/exceptions.py	2016-04-11 16:23:26 +0000
+++ src/provisioningserver/rpc/exceptions.py	2016-05-10 23:45:39 +0000
@@ -7,6 +7,7 @@
     "AuthenticationFailed",
     "CannotConfigureDHCP",
     "CannotCreateHostMap",
+    "CannotDisableAndShutoffRackd",
     "CannotModifyHostMap",
     "CannotRegisterCluster",
     "CannotRemoveHostMap",
@@ -146,3 +147,7 @@
 
 class BootConfigNoResponse(Exception):
     """The region gave no response for the boot configuration."""
+
+
+class CannotDisableAndShutoffRackd(Exception):
+    """Rackd cannot be disabled and shutoff."""

=== modified file 'src/provisioningserver/rpc/tests/test_clusterservice.py'
--- src/provisioningserver/rpc/tests/test_clusterservice.py	2016-05-10 11:02:10 +0000
+++ src/provisioningserver/rpc/tests/test_clusterservice.py	2016-05-10 23:45:39 +0000
@@ -2392,3 +2392,28 @@
             'hostname': factory.make_hostname(),
             })
         self.assertEquals({}, response.result)
+
+
+class TestClusterProtocol_DisableAndShutoffRackd(MAASTestCase):
+
+    def test__is_registered(self):
+        protocol = Cluster()
+        responder = protocol.locateResponder(
+            cluster.DisableAndShutoffRackd.commandName)
+        self.assertIsNotNone(responder)
+
+    def test_issues_restart(self):
+        mock_call_and_check = self.patch(clusterservice, 'call_and_check')
+        response = call_responder(
+            Cluster(), cluster.DisableAndShutoffRackd, {})
+        self.assertEquals({}, response.result)
+        self.assertEquals(2, mock_call_and_check.call_count)
+
+    @inlineCallbacks
+    def test_raises_error_on_failure(self):
+        mock_call_and_check = self.patch(clusterservice, 'call_and_check')
+        mock_call_and_check.side_effect = ExternalProcessError(
+            1, 'systemctl', 'failure')
+        with ExpectedException(exceptions.CannotDisableAndShutoffRackd):
+            yield call_responder(
+                Cluster(), cluster.DisableAndShutoffRackd, {})