Merge lp:~blake-rouse/maas/fix-1487135 into lp:maas/trunk

Proposed by Blake Rouse on 2015-11-04
Status: Merged
Approved by: Blake Rouse on 2015-11-05
Approved revision: 4460
Merged at revision: 4471
Proposed branch: lp:~blake-rouse/maas/fix-1487135
Merge into: lp:maas/trunk
Diff against target: 259 lines (+130/-12)
5 files modified
src/maasserver/api/nodes.py (+27/-0)
src/maasserver/api/tests/test_doc.py (+1/-0)
src/maasserver/api/tests/test_node.py (+42/-0)
src/maasserver/preseed.py (+26/-12)
src/maasserver/tests/test_preseed.py (+34/-0)
To merge this branch: bzr merge lp:~blake-rouse/maas/fix-1487135
Reviewer Review Type Date Requested Status
Mike Pontillo (community) 2015-11-04 Approve on 2015-11-05
Jeffrey C Jones (community) 2015-11-04 Approve on 2015-11-04
Review via email: mp+276689@code.launchpad.net

Commit message

Add get_curtin_config API to return the curtin configuration that was passed when the node started deployment.

To post a comment you must log in.
Blake Rouse (blake-rouse) wrote :

I did this because I kept needing a way to get the configuration when curtin deployments fail.

Jeffrey C Jones (trapnine) wrote :

looks good, just a comment typo

review: Approve
Mike Pontillo (mpontillo) wrote :

This is great. Thanks for adding this.

review: Approve
MAAS Lander (maas-lander) wrote :
Download full text (1013.7 KiB)

The attempt to merge lp:~blake-rouse/maas/fix-1487135 into lp:maas failed. Below is the output from the failed tests.

Get:1 http://security.ubuntu.com trusty-security InRelease [64.4 kB]
Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Get:2 http://nova.clouds.archive.ubuntu.com trusty-updates InRelease [64.4 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty Release.gpg
Hit http://nova.clouds.archive.ubuntu.com trusty Release
Get:3 http://security.ubuntu.com trusty-security/main Sources [98.0 kB]
Get:4 http://nova.clouds.archive.ubuntu.com trusty-updates/main Sources [242 kB]
Get:5 http://security.ubuntu.com trusty-security/universe Sources [31.0 kB]
Get:6 http://security.ubuntu.com trusty-security/main amd64 Packages [361 kB]
Get:7 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Sources [143 kB]
Get:8 http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [644 kB]
Get:9 http://security.ubuntu.com trusty-security/universe amd64 Packages [117 kB]
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Get:10 http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [326 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty/main Sources
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Sources
Hit http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en
Ign http://nova.clouds.archive.ubuntu.com trusty/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 2,092 kB in 4s (502 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
     --no-install-recommends install apache2 authbind bind9 bind9utils build-essential bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm pep8 phantomjs postgresql pyflakes python-apt python-bson python-bzrlib python-convoy python-coverage python-crochet python-cssselect python-curtin python-dev python-distro-info python-django python-django-piston python-django-south python-djorm-ext-pgarray python-docutils python-extras python-fixtures python-flake8 python-formencode python-hivex python-httplib2 python-jinja2 python-jsonschema python-lxml python-mock python-netaddr python-netifaces python-nose python-oauth python-openssl python-paramiko python-pexpect python-pip python-pocket-lint python-psycopg2 python-pyinotify python-pyparsing python-seamicroclient python-simplejson python-simplestreams python-sphinx python-subunit python-tempita python-testresources python-testscenarios python-te...

Andres Rodriguez (andreserl) wrote :

should we have get_storage_config and get_network_config or something along those lines?

Blake Rouse (blake-rouse) wrote :

No because it is the same config. It is shown in the output.

MAAS Lander (maas-lander) wrote :
Download full text (1013.7 KiB)

The attempt to merge lp:~blake-rouse/maas/fix-1487135 into lp:maas failed. Below is the output from the failed tests.

Get:1 http://security.ubuntu.com trusty-security InRelease [64.4 kB]
Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Get:2 http://nova.clouds.archive.ubuntu.com trusty-updates InRelease [64.4 kB]
Get:3 http://security.ubuntu.com trusty-security/main Sources [98.0 kB]
Get:4 http://security.ubuntu.com trusty-security/universe Sources [31.0 kB]
Get:5 http://security.ubuntu.com trusty-security/main amd64 Packages [361 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty Release.gpg
Hit http://nova.clouds.archive.ubuntu.com trusty Release
Get:6 http://nova.clouds.archive.ubuntu.com trusty-updates/main Sources [242 kB]
Get:7 http://security.ubuntu.com trusty-security/universe amd64 Packages [117 kB]
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Get:8 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Sources [143 kB]
Get:9 http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [644 kB]
Get:10 http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [326 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty/main Sources
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Sources
Hit http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en
Ign http://nova.clouds.archive.ubuntu.com trusty/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en_US
Fetched 2,092 kB in 4s (493 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
     --no-install-recommends install apache2 authbind bind9 bind9utils build-essential bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm pep8 phantomjs postgresql pyflakes python-apt python-bson python-bzrlib python-convoy python-coverage python-crochet python-cssselect python-curtin python-dev python-distro-info python-django python-django-piston python-django-south python-djorm-ext-pgarray python-docutils python-extras python-fixtures python-flake8 python-formencode python-hivex python-httplib2 python-jinja2 python-jsonschema python-lxml python-mock python-netaddr python-netifaces python-nose python-oauth python-openssl python-paramiko python-pexpect python-pip python-pocket-lint python-psycopg2 python-pyinotify python-pyparsing python-seamicroclient python-simplejson python-simplestreams python-sphinx python-subunit python-tempita python-testresources python-testscenarios python-te...

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'src/maasserver/api/nodes.py'
2--- src/maasserver/api/nodes.py 2015-11-04 23:38:04 +0000
3+++ src/maasserver/api/nodes.py 2015-11-05 14:17:35 +0000
4@@ -78,6 +78,7 @@
5 from maasserver.models.node import RELEASABLE_STATUSES
6 from maasserver.models.nodeprobeddetails import get_single_probed_details
7 from maasserver.node_constraint_filter_forms import AcquireNodeForm
8+from maasserver.preseed import get_curtin_merged_config
9 from maasserver.rpc import getClientFor
10 from maasserver.storage_layouts import (
11 StorageLayoutError,
12@@ -96,6 +97,7 @@
13 from provisioningserver.rpc.cluster import PowerQuery
14 from provisioningserver.rpc.exceptions import NoConnectionsAvailable
15 import simplejson as json
16+import yaml
17
18 # Node's fields exposed on the API.
19 DISPLAYED_NODE_FIELDS = (
20@@ -930,6 +932,10 @@
21 If the default gateways need to be specific for this node you can set
22 which interface and subnet's gateway to use when this node is deployed
23 with the `node-interfaces set-default-gateway` API.
24+
25+ Returns 404 if the node could not be found.
26+ Returns 403 if the user does not have permission to clear the default
27+ gateways.
28 """
29 node = Node.nodes.get_node_or_404(
30 system_id=system_id, user=request.user, perm=NODE_PERMISSION.ADMIN)
31@@ -938,6 +944,27 @@
32 node.save()
33 return node
34
35+ @operation(idempotent=True)
36+ def get_curtin_config(self, request, system_id):
37+ """Return the rendered curtin configuration for the node.
38+
39+ Returns 404 if the node could not be found.
40+ Returns 403 if the user does not have permission to get the curtin
41+ configuration.
42+ """
43+ node = Node.nodes.get_node_or_404(
44+ system_id=system_id, user=request.user, perm=NODE_PERMISSION.VIEW)
45+ if node.status not in [
46+ NODE_STATUS.DEPLOYING,
47+ NODE_STATUS.DEPLOYED,
48+ NODE_STATUS.FAILED_DEPLOYMENT]:
49+ raise MAASAPIBadRequest(
50+ "Node %s is not in a deployment state." % node.hostname)
51+ return HttpResponse(
52+ yaml.safe_dump(
53+ get_curtin_merged_config(node), default_flow_style=False),
54+ content_type='text/plain')
55+
56
57 def create_node(request):
58 """Service an http request to create a node.
59
60=== modified file 'src/maasserver/api/tests/test_doc.py'
61--- src/maasserver/api/tests/test_doc.py 2015-09-15 19:07:22 +0000
62+++ src/maasserver/api/tests/test_doc.py 2015-11-05 14:17:35 +0000
63@@ -323,6 +323,7 @@
64 "POST set_storage_layout op=set_storage_layout restful=False",
65 "POST clear_default_gateways op=clear_default_gateways "
66 "restful=False",
67+ "GET get_curtin_config op=get_curtin_config restful=False",
68 }
69 observed_actions = {
70 "%(method)s %(name)s op=%(op)s restful=%(restful)s" % action
71
72=== modified file 'src/maasserver/api/tests/test_node.py'
73--- src/maasserver/api/tests/test_node.py 2015-10-27 20:53:16 +0000
74+++ src/maasserver/api/tests/test_node.py 2015-11-05 14:17:35 +0000
75@@ -24,6 +24,7 @@
76 from django.core.urlresolvers import reverse
77 from django.db import transaction
78 from maasserver import forms
79+from maasserver.api import nodes as nodes_module
80 from maasserver.enum import (
81 INTERFACE_TYPE,
82 IPADDRESS_TYPE,
83@@ -83,6 +84,7 @@
84 HasLength,
85 Not,
86 )
87+import yaml
88
89
90 class NodeAnonAPITest(MAASServerTestCase):
91@@ -2186,3 +2188,43 @@
92 node = reload_object(node)
93 self.assertIsNone(node.gateway_link_ipv4)
94 self.assertIsNone(node.gateway_link_ipv6)
95+
96+
97+class TestGetCurtinConfig(APITestCase):
98+
99+ def get_node_uri(self, node):
100+ """Get the API URI for `node`."""
101+ return reverse('node_handler', args=[node.system_id])
102+
103+ def test__500_when_node_not_in_deployment_state(self):
104+ node = factory.make_Node(
105+ owner=self.logged_in_user,
106+ status=factory.pick_enum(
107+ NODE_STATUS, but_not=[
108+ NODE_STATUS.DEPLOYING,
109+ NODE_STATUS.DEPLOYED,
110+ NODE_STATUS.FAILED_DEPLOYMENT,
111+ ]))
112+ response = self.client.get(
113+ self.get_node_uri(node), {'op': 'get_curtin_config'})
114+ self.assertEquals(
115+ httplib.BAD_REQUEST, response.status_code, response.content)
116+
117+ def test__returns_curtin_config_in_yaml(self):
118+ node = factory.make_Node(
119+ owner=self.logged_in_user, status=NODE_STATUS.DEPLOYING)
120+ fake_config = {
121+ "config": factory.make_name("config")
122+ }
123+ mock_get_curtin_merged_config = self.patch(
124+ nodes_module, "get_curtin_merged_config")
125+ mock_get_curtin_merged_config.return_value = fake_config
126+ response = self.client.get(
127+ self.get_node_uri(node), {'op': 'get_curtin_config'})
128+ self.assertEquals(
129+ httplib.OK, response.status_code, response.content)
130+ self.assertEquals(
131+ yaml.safe_dump(fake_config, default_flow_style=False),
132+ response.content)
133+ self.assertThat(
134+ mock_get_curtin_merged_config, MockCalledOnceWith(node))
135
136=== modified file 'src/maasserver/preseed.py'
137--- src/maasserver/preseed.py 2015-10-30 16:14:33 +0000
138+++ src/maasserver/preseed.py 2015-11-05 14:17:35 +0000
139@@ -31,6 +31,7 @@
140
141 from crochet import TimeoutError
142 from curtin.commands import block_meta
143+from curtin.config import merge_config
144 from curtin.pack import pack_install
145 from django.conf import settings
146 from maasserver import logger
147@@ -216,14 +217,8 @@
148 return []
149
150
151-def get_curtin_userdata(node):
152- """Return the curtin user-data.
153-
154- :param node: The node for which to generate the user-data.
155- :return: The rendered user-data string.
156- :rtype: unicode.
157- """
158- installer_url = get_curtin_installer_url(node)
159+def get_curtin_yaml_config(node):
160+ """Return the curtin configration for the node."""
161 main_config = get_curtin_config(node)
162 reporter_config = compose_curtin_maas_reporter(node)
163 swap_config = compose_curtin_swap_preseed(node)
164@@ -253,13 +248,32 @@
165 else:
166 storage_config = []
167
168+ return (
169+ [main_config] + reporter_config + storage_config + network_config +
170+ swap_config + kernel_config + verbose_config)
171+
172+
173+def get_curtin_merged_config(node):
174+ """Return the merged curtin configuration for the node."""
175+ yaml_config = get_curtin_yaml_config(node)
176+ config = {}
177+ for cfg in yaml_config:
178+ merge_config(config, yaml.load(cfg))
179+ return config
180+
181+
182+def get_curtin_userdata(node):
183+ """Return the curtin user-data.
184+
185+ :param node: The node for which to generate the user-data.
186+ :return: The rendered user-data string.
187+ :rtype: unicode.
188+ """
189 # Pack the curtin and the configuration into a script to execute on the
190 # deploying node.
191 return pack_install(
192- configs=(
193- [main_config] + reporter_config + storage_config + network_config +
194- swap_config + kernel_config + verbose_config),
195- args=[installer_url])
196+ configs=get_curtin_yaml_config(node),
197+ args=[get_curtin_installer_url(node)])
198
199
200 def get_curtin_image(node):
201
202=== modified file 'src/maasserver/tests/test_preseed.py'
203--- src/maasserver/tests/test_preseed.py 2015-10-30 16:03:19 +0000
204+++ src/maasserver/tests/test_preseed.py 2015-11-05 14:17:35 +0000
205@@ -53,6 +53,7 @@
206 get_curtin_context,
207 get_curtin_image,
208 get_curtin_installer_url,
209+ get_curtin_merged_config,
210 get_curtin_userdata,
211 get_enlist_preseed,
212 get_netloc_and_path,
213@@ -83,6 +84,7 @@
214 MockNotCalled,
215 )
216 from metadataserver.models import NodeKey
217+from mock import sentinel
218 from provisioningserver.drivers.osystem.ubuntu import UbuntuOS
219 from provisioningserver.rpc.exceptions import NoConnectionsAvailable
220 from provisioningserver.utils.enum import map_enum
221@@ -733,6 +735,38 @@
222 }, yaml.load(preseed[0]))
223
224
225+class TestGetCurtinMergedConfig(MAASServerTestCase):
226+
227+ def test__merges_configs_together(self):
228+ configs = [
229+ yaml.safe_dump({
230+ "maas": {
231+ "test": "data"
232+ },
233+ "override": "data",
234+ }),
235+ yaml.safe_dump({
236+ "maas2": {
237+ "test": "data2"
238+ },
239+ "override": "data2",
240+ }),
241+ ]
242+ mock_yaml_config = self.patch_autospec(
243+ preseed_module, "get_curtin_yaml_config")
244+ mock_yaml_config.return_value = configs
245+ self.assertEquals({
246+ "maas": {
247+ "test": "data"
248+ },
249+ "maas2": {
250+ "test": "data2"
251+ },
252+ "override": "data2",
253+ }, get_curtin_merged_config(sentinel.node))
254+ self.assertThat(mock_yaml_config, MockCalledOnceWith(sentinel.node))
255+
256+
257 class TestGetCurtinUserData(
258 PreseedRPCMixin, BootImageHelperMixin, MAASServerTestCase):
259 """Tests for `get_curtin_userdata`."""