Merge lp:~newell-jensen/maas/2.2-fix-1694759 into lp:maas/2.2

[Newell Jensen]

Backport trunk r6074: Delete newly allocated RSD node if the assembling process fails.

[Newell Jensen (newell-jensen)]

Self review backport.

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~newell-jensen/maas/2.2-fix-1694759 into lp:maas/2.2 failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu xenial-security InRelease
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind avahi-utils bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common isc-dhcp-server libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libnss-wrapper libpq-dev make nodejs-legacy npm postgresql psmisc pxelinux python3-all python3-apt python3-attr python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
authbind is already the newest version (2.1.1+nmu1).
avahi-utils is already the newest version (0.6.32~rc+dfsg-1ubuntu2).
build-essential is already the newest version (12.1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
make is already the newest version (4.1-6).
postgresql is already the newest version (9.5+173).
psmisc is already the newest version (22.21-2.1build1).
pxelinux is already the newest version (3:6.03+dfsg-11ubuntu1).
python-formencode is already the newest version (1.3.0-0ubuntu5).
python-lxml is already the newest version (3.5.0-1build1).
python-netaddr is already the newest version (0.7.18-1).
python-netifaces is already the newest version (0.10.4-0.1build2).
python-psycopg2 is already the newest version (2.6.1-1...

[MAAS Lander (maas-lander)]

The attempt to merge lp:~newell-jensen/maas/2.2-fix-1694759 into lp:maas/2.2 failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
Get:4 http://prodstack-zone-1.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Fetched 306 kB in 0s (336 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind avahi-utils bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common isc-dhcp-server libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libnss-wrapper libpq-dev make nodejs-legacy npm postgresql psmisc pxelinux python3-all python3-apt python3-attr python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
authbind is already the newest version (2.1.1+nmu1).
avahi-utils is already the newest version (0.6.32~rc+dfsg-1ubuntu2).
build-essential is already the newest version (12.1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
make is already the newest version (4.1-6).
postgresql is already the newest version (9.5+173).
psmisc is already the newest version (22.21-2.1build1).
pxelinux is already the newest version (3:6.03+dfsg-11ubuntu1).
python-formencode is already the newest version (1.3.0-0ubuntu5).
python-lxml is already the newest version (3.5.0-1build1).
python-netaddr is already the newest version (0.7.18-1).
python-netifaces is already the newest version (0.10.4-0.1buil...

=== modified file 'src/provisioningserver/drivers/pod/rsd.py'
--- src/provisioningserver/drivers/pod/rsd.py	2017-05-09 19:10:12 +0000
+++ src/provisioningserver/drivers/pod/rsd.py	2017-06-07 17:24:55 +0000
@@ -1046,11 +1046,8 @@
             "Unable to allocate machine with requested resources.")
 
     @inlineCallbacks
-    def decompose(self, pod_id, context):
-        """Decompose machine."""
-        url = self.get_url(context)
-        node_id = context.get('node_id').encode('utf-8')
-        headers = self.make_auth_headers(**context)
+    def delete_node(self, url, node_id, headers):
+        """Delete node at node_id."""
         # Delete machine at node_id.
         endpoint = b"redfish/v1/Nodes/%s" % node_id
         try:
@@ -1060,12 +1057,17 @@
             # XXX newell 2017-02-27 bug=1667754:
             # Catch the 404 error when trying to decompose the
             # resource that has already been decomposed.
-            # This is a work around and will need to be handled
-            # differently on the region so we don't try to
-            # decompose a machine multiple times.
             if int(error.status) != HTTPStatus.NOT_FOUND:
                 raise
 
+    @inlineCallbacks
+    def decompose(self, pod_id, context):
+        """Decompose machine."""
+        url = self.get_url(context)
+        node_id = context.get('node_id').encode('utf-8')
+        headers = self.make_auth_headers(**context)
+        yield self.delete_node(url, node_id, headers)
+
         # Retrieve pod resources.
         discovered_pod = yield self.get_pod_resources(url, headers)
         # Retrive pod hints.
@@ -1114,7 +1116,8 @@
             yield self.redfish_request(
                 b"POST", join(url, endpoint), headers)
         elif node_state == 'Failed':
-            # Broken system.
+            # Broken system, delete allocated node.
+            yield self.delete_node(url, node_id, headers)
             raise PodFatalError(
                 "Composed machine at node ID %s has a ComposedNodeState"
                 " of Failed." % node_id)
@@ -1131,7 +1134,8 @@
         # Check one last time if the state has became `Failed`.
 
         if node_state == 'Failed':
-            # Broken system.
+            # Broken system, delete allocated node.
+            yield self.delete_node(url, node_id, headers)
             raise PodFatalError(
                 "Composed machine at node ID %s has a ComposedNodeState"
                 " of Failed." % node_id)
 
=== modified file 'src/provisioningserver/drivers/pod/tests/test_rsd.py'
--- src/provisioningserver/drivers/pod/tests/test_rsd.py	2017-05-09 21:53:26 +0000
+++ src/provisioningserver/drivers/pod/tests/test_rsd.py	2017-06-07 17:24:55 +0000
@@ -2002,74 +2002,77 @@
                 factory.make_name('system_id'), context, request)
 
     @inlineCallbacks
+    def test__delete_node(self):
+        driver = RSDPodDriver()
+        context = make_context()
+        url = driver.get_url(context)
+        node_id = context.get('node_id').encode('utf-8')
+        endpoint = b"redfish/v1/Nodes/%s" % node_id
+        headers = driver.make_auth_headers(**context)
+        mock_redfish_request = self.patch(driver, 'redfish_request')
+
+        yield driver.delete_node(url, node_id, headers)
+        self.assertThat(mock_redfish_request, MockCalledOnceWith(
+            b"DELETE", join(url, endpoint), headers))
+
+    @inlineCallbacks
+    def test_delete_node_continues_on_404_error(self):
+        driver = RSDPodDriver()
+        context = make_context()
+        url = driver.get_url(context)
+        node_id = context.get('node_id').encode('utf-8')
+        endpoint = b"redfish/v1/Nodes/%s" % node_id
+        headers = driver.make_auth_headers(**context)
+        mock_redfish_request = self.patch(driver, 'redfish_request')
+        error = PartialDownloadError(
+            response=json.dumps(SAMPLE_JSON_SYSTEMS).encode('utf-8'),
+            code=HTTPStatus.NOT_FOUND)
+        mock_redfish_request.side_effect = error
+
+        yield driver.delete_node(url, node_id, headers)
+        self.assertThat(mock_redfish_request, MockCalledOnceWith(
+            b"DELETE", join(url, endpoint), headers))
+
+    @inlineCallbacks
+    def test_delete_node_raises_when_not_404_error(self):
+        driver = RSDPodDriver()
+        context = make_context()
+        url = driver.get_url(context)
+        node_id = context.get('node_id').encode('utf-8')
+        endpoint = b"redfish/v1/Nodes/%s" % node_id
+        headers = driver.make_auth_headers(**context)
+        mock_redfish_request = self.patch(driver, 'redfish_request')
+        error = PartialDownloadError(
+            response=json.dumps(SAMPLE_JSON_SYSTEMS).encode('utf-8'),
+            code=HTTPStatus.BAD_REQUEST)
+        mock_redfish_request.side_effect = error
+
+        with ExpectedException(PartialDownloadError):
+            yield driver.delete_node(url, node_id, headers)
+        self.assertThat(mock_redfish_request, MockCalledOnceWith(
+            b"DELETE", join(url, endpoint), headers))
+
+    @inlineCallbacks
     def test__decompose(self):
         driver = RSDPodDriver()
         context = make_context()
         url = driver.get_url(context)
         node_id = context.get('node_id').encode('utf-8')
         headers = driver.make_auth_headers(**context)
-        endpoint = b"redfish/v1/Nodes/%s" % node_id
-        mock_redfish_request = self.patch(driver, 'redfish_request')
-        mock_get_pod_resources = self.patch(driver, 'get_pod_resources')
-        discovered_pod = make_discovered_pod()
-        mock_get_pod_resources.return_value = discovered_pod
-        mock_get_pod_hints = self.patch(driver, 'get_pod_hints')
-
-        yield driver.decompose(
-            factory.make_name('system_id'), context)
-        self.assertThat(mock_redfish_request, MockCalledOnceWith(
-            b"DELETE", join(url, endpoint), headers))
-        self.assertThat(mock_get_pod_resources, MockCalledOnceWith(
-            url, headers))
-        self.assertThat(mock_get_pod_hints, MockCalledOnceWith(
-            discovered_pod))
-
-    @inlineCallbacks
-    def test_decompose_continues_on_404_error(self):
-        driver = RSDPodDriver()
-        context = make_context()
-        url = driver.get_url(context)
-        node_id = context.get('node_id').encode('utf-8')
-        headers = driver.make_auth_headers(**context)
-        endpoint = b"redfish/v1/Nodes/%s" % node_id
-        mock_redfish_request = self.patch(driver, 'redfish_request')
-        error = PartialDownloadError(
-            response=json.dumps(SAMPLE_JSON_SYSTEMS).encode('utf-8'),
-            code=HTTPStatus.NOT_FOUND)
-        mock_redfish_request.side_effect = error
-        mock_get_pod_resources = self.patch(driver, 'get_pod_resources')
-        discovered_pod = make_discovered_pod()
-        mock_get_pod_resources.return_value = discovered_pod
-        mock_get_pod_hints = self.patch(driver, 'get_pod_hints')
-
-        yield driver.decompose(
-            factory.make_name('system_id'), context)
-        self.assertThat(mock_redfish_request, MockCalledOnceWith(
-            b"DELETE", join(url, endpoint), headers))
-        self.assertThat(mock_get_pod_resources, MockCalledOnceWith(
-            url, headers))
-        self.assertThat(mock_get_pod_hints, MockCalledOnceWith(
-            discovered_pod))
-
-    @inlineCallbacks
-    def test_decompose_raises_when_not_404_error(self):
-        driver = RSDPodDriver()
-        context = make_context()
-        url = driver.get_url(context)
-        node_id = context.get('node_id').encode('utf-8')
-        headers = driver.make_auth_headers(**context)
-        endpoint = b"redfish/v1/Nodes/%s" % node_id
-        mock_redfish_request = self.patch(driver, 'redfish_request')
-        error = PartialDownloadError(
-            response=json.dumps(SAMPLE_JSON_SYSTEMS).encode('utf-8'),
-            code=HTTPStatus.BAD_REQUEST)
-        mock_redfish_request.side_effect = error
-
-        with ExpectedException(PartialDownloadError):
-            yield driver.decompose(
-                factory.make_name('system_id'), context)
-        self.assertThat(mock_redfish_request, MockCalledOnceWith(
-            b"DELETE", join(url, endpoint), headers))
+        mock_delete_node = self.patch(driver, 'delete_node')
+        mock_get_pod_resources = self.patch(driver, 'get_pod_resources')
+        discovered_pod = make_discovered_pod()
+        mock_get_pod_resources.return_value = discovered_pod
+        mock_get_pod_hints = self.patch(driver, 'get_pod_hints')
+
+        yield driver.decompose(
+            factory.make_name('system_id'), context)
+        self.assertThat(mock_delete_node, MockCalledOnceWith(
+            url, node_id, headers))
+        self.assertThat(mock_get_pod_resources, MockCalledOnceWith(
+            url, headers))
+        self.assertThat(mock_get_pod_hints, MockCalledOnceWith(
+            discovered_pod))
 
     @inlineCallbacks
     def test__set_pxe_boot(self):
@@ -2164,12 +2167,15 @@
         mock_get_composed_node_state = self.patch(
             driver, 'get_composed_node_state')
         mock_get_composed_node_state.return_value = "Failed"
+        mock_delete_node = self.patch(driver, 'delete_node')
         mock_redfish_request = self.patch(driver, 'redfish_request')
 
         with ExpectedException(PodFatalError):
             yield driver.assemble_node(url, node_id, headers)
         self.assertThat(mock_get_composed_node_state, MockCalledOnceWith(
             url, node_id, headers))
+        self.assertThat(mock_delete_node, MockCalledOnceWith(
+            url, node_id, headers))
         self.assertThat(mock_redfish_request, MockNotCalled())
 
     @inlineCallbacks
@@ -2183,6 +2189,7 @@
             driver, 'get_composed_node_state')
         mock_get_composed_node_state.side_effect = [
             "Allocated", "Assembling", "Failed"]
+        mock_delete_node = self.patch(driver, 'delete_node')
         mock_redfish_request = self.patch(driver, 'redfish_request')
         endpoint = (
             b"redfish/v1/Nodes/%s/Actions/ComposedNode.Assemble" % node_id)
@@ -2196,6 +2203,8 @@
             call(url, node_id, headers)))
         self.assertThat(mock_redfish_request, MockCalledOnceWith(
             b"POST", uri, headers))
+        self.assertThat(mock_delete_node, MockCalledOnceWith(
+            url, node_id, headers))
 
     @inlineCallbacks
     def test_power_issues_power_reset(self):