Merge ~smoser/maas:cleanup/render-curtin-config-internally into maas:master

[Scott Moser]

Internally handle most of curtin_userdata leaving templates to user.

The 'curtin_userdata' file was scary to look at, and hard to work
with for a user. Adding early_commands or late_commands meant that they
would have to edit an existing entry, and failure to do so would
break the install.

This takes what was previously in that those files and handles them
internally to maas. Thus, the user now has a blank slate to work on,
and an example of what could go there.

One change along the way was to render the writing of the binary
tmp file with the apt key via base64 rather than hex encoding.

This is a work in progress, I'm looking to see if it the general concept
is acceptable.

It is untested at this point, but I believe should be (close to) functional but does break third party drivers currently.

[Blake Rouse (blake-rouse)]

I think the general idea is great, but we need to think about the upgrade path.

What happens if they made changes to the sections that you have not made internal? Will there changes override the internal changes?

Also this is more of a future thing, but I would prefer to see the templates move into the database with an API, because at the moment with HA, these are a problem.

[Andres Rodriguez (andreserl)]

That's my only concern, we need to make sure the upgrade path doesn't break
and this continuous to work with those that modify their curtin scripts.
Which is a lot of people atm.

On Mon, Oct 23, 2017 at 9:16 AM, Blake Rouse <email address hidden>
wrote:

> I think the general idea is great, but we need to think about the upgrade
> path.
>
> What happens if they made changes to the sections that you have not made
> internal? Will there changes override the internal changes?
>
> Also this is more of a future thing, but I would prefer to see the
> templates move into the database with an API, because at the moment with
> HA, these are a problem.
> --
> https://code.launchpad.net/~smoser/maas/+git/maas/+merge/332573
> Your team MAAS Committers is subscribed to branch maas:master.
>

--
Andres Rodriguez
Engineering Manager, MAAS
Canonical USA, Inc.

[Scott Moser (smoser)]

Summary: over all, it will do reasonably well on "upgrade".

What is the upgrade path anyway, for *any* change to that file?
Do you have something in place other than debian conf-file upgrades?

If all you have is debian conf-file, then the admin running 'apt-get
dist-upgrade' is going to get prompted and see that those files are being
replaced and have the ability to diff them.

I'd consider that there are 2 likely paths:
a.) user does nothing, leaving the whole file in place as it was before.
b.) user looks at it, sees their changes and generally correctly merges
them by creating the file with *just* their changes inside.

So 'b' is the easy path, that should work or is generally not fixable
other than by the user.

for 'a', all the "builtin" config values are dictionaries.
When curtin merges dictionaries, the later one wins.
So as long as we keep the same keys in all cases, then nothing
would get run more than once and all should be well.
By 'key', i mean:
  early_commands/driver_*
and
  debconf_selections/maas

If you really wanted to safe-guard you could check the loaded file
and explicitly ignore the previously named keys, but that too could
actually cause breakage, if the user had modified those keys to
their liking.

[Scott Moser (smoser)]

Actually, what I think i'd do and suggest is to
a.) rename the keys from the names that did exist in that file
  early_commands/driver_* to early_commands/30_driver_*
  debconf_selections/maas_* to debconf_selections/maas_internal

b.) "clean" out the old names if they were present in the loaded config

this would mean that user's modifications to that file that modified *those* things would actually fail to have affect. But generally speaking they should not have done that, or we'd like to have bugs to see what was wrong.

then just warn in log that it is ignoring those things.
that'd keep their file basically working as expected other than some things
in it not actually having any affect.

[Andres Rodriguez (andreserl)]

For 2.3, this is too late, so we will target this for 2.4 once it opens.

[MAAS Lander (maas-lander)]

UNIT TESTS
-b cleanup/render-curtin-config-internally lp:~smoser/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci-jenkins.internal:8080/job/maas/job/branch-tester/519/console
COMMIT: d313a298a0eeb39535e8017b777db8481010f5e1

review: Needs Fixing

[Scott Moser (smoser)]

I addressed Ante's comment and updated to merge without conflicts on current master.

[MAAS Lander (maas-lander)]

UNIT TESTS
-b cleanup/render-curtin-config-internally lp:~smoser/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci-jenkins.internal:8080/job/maas/job/branch-tester/812/console
COMMIT: addfc9a36144635a35e1465f10cb7de905ae387d

review: Needs Fixing

[Scott Moser (smoser)]

I think this is ready for review at this point.
I've fixed up all the unit tests.

[Mike Pontillo (mpontillo)]

This branch is looking much better, but where are the tests that cover the new code?

review: Needs Information

[MAAS Lander (maas-lander)]

UNIT TESTS
-b cleanup/render-curtin-config-internally lp:~smoser/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci-jenkins.internal:8080/job/maas/job/branch-tester/823/console
COMMIT: 296fab455858b1b3970137fb23f27351db5b9d8e

review: Needs Fixing

[Scott Moser (smoser)]

well, we can add tests for the new code, but this is really just code motion (replace of template rendering with internal).
Ie, the net change is the same behavior. Whatever tests were there before (that I adjusted) should give the same total code coverage.

What specific things are you wanting to be tested?

[Scott Moser (smoser)]

Also, I can't see the results of the unit tests above.
roaksoax gave me a paste of them, but I'm not sure what woudl be causing the 'url'' to not get set (some help?)

[Andres Rodriguez (andreserl)]

Hi Scott,

We'll take a look at it next week. Right now the team is running on reduced
capacity.

On Fri, Dec 1, 2017 at 10:53 AM, Scott Moser <email address hidden>
wrote:

> Also, I can't see the results of the unit tests above.
> roaksoax gave me a paste of them, but I'm not sure what woudl be causing
> the 'url'' to not get set (some help?)
>
> --
> https://code.launchpad.net/~smoser/maas/+git/maas/+merge/332573
> You are reviewing the proposed merge of ~smoser/maas:cleanup/render-curtin-config-internally
> into maas:master.
>

--
Andres Rodriguez
Engineering Manager, MAAS
Canonical USA, Inc.

[MAAS Lander (maas-lander)]

UNIT TESTS
-b cleanup/render-curtin-config-internally lp:~smoser/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci-jenkins.internal:8080/job/maas/job/branch-tester/868/console
COMMIT: d33d9f6b6da6b2306e167357579fe6e4ae8764dd

review: Needs Fixing

[Scott Moser (smoser)]

Bump?

[Mike Pontillo (mpontillo)]

@smoser, I'll take a closer look. Thanks for getting this work going.

[Blake Rouse (blake-rouse)]

This has been sitting for a long time, closing up the old queue.

Unmerged commits

d33d9f6... by Scott Moser on 2017-11-27

fix per make format

1936d8c... by Scott Moser on 2017-11-27

fix flake8

bc88c60... by Scott Moser on 2017-11-27

drop key_b64 variable

ff2cc32... by Scott Moser on 2017-11-27

drop 'subst' helper, no longer used

cafa3eb... by Scott Moser on 2017-11-27

remove debug print

7d1215b... by Scott Moser on 2017-11-27

tests pass.

5930dfe... by Scott Moser on 2017-11-27

replace old templates with stock empty

51bd856... by Scott Moser on 2017-11-24

Internally handle most of curtin_userdata leaving templates to user.

The 'curtin_userdata' file was scary to look at, and hard to work
with for a user. Adding early_commands or late_commands meant that they
would have to edit an existing entry, and failure to do so would
break the install.

This takes what was previously in that those files and handles them
internally to maas. Thus, the user now has a blank slate to work on,
and an example of what could go there.

LP: #1734391

diff --git a/contrib/preseeds_v2/curtin_userdata b/contrib/preseeds_v2/curtin_userdata
index 4caddc4..a22a775 100644
--- a/contrib/preseeds_v2/curtin_userdata
+++ b/contrib/preseeds_v2/curtin_userdata
@@ -1,42 +1,18 @@
-#cloud-config
-debconf_selections:
- maas: |
-  {{for line in str(curtin_preseed).splitlines()}}
-  {{line}}
-  {{endfor}}
-early_commands:
-{{if third_party_drivers and driver}}
-  {{py: key_string = ''.join(['\\x%x' % x for x in driver['key_binary']])}}
-  {{if driver['key_binary'] and driver['repository'] and driver['package']}}
-  driver_00_get_key: /bin/echo -en '{{key_string}}' > /tmp/maas-{{driver['package']}}.gpg
-  driver_01_add_key: ["apt-key", "add", "/tmp/maas-{{driver['package']}}.gpg"]
-  {{endif}}
-  {{if driver['repository']}}
-  driver_02_add: ["add-apt-repository", "-y", "deb {{driver['repository']}} {{node.get_distro_series()}} main"]
-  {{endif}}
-  {{if driver['package']}}
-  driver_03_update_install: ["sh", "-c", "apt-get update --quiet && apt-get --assume-yes install {{driver['package']}}"]
-  {{endif}}
-  {{if driver['module']}}
-  driver_04_load: ["sh", "-c", "depmod && modprobe {{driver['module']}} || echo 'Warning: Failed to load module: {{driver['module']}}'"]
-  {{endif}}
-{{else}}
-  driver_00: ["sh", "-c", "echo third party drivers not installed or necessary."]
-{{endif}}
-late_commands:
-  maas: [wget, '--no-proxy', {{node_disable_pxe_url|escape.json}}, '--post-data', {{node_disable_pxe_data|escape.json}}, '-O', '/dev/null']
-{{if third_party_drivers and driver}}
-  {{if driver['key_binary'] and driver['repository'] and driver['package']}}
-  driver_00_key_get: curtin in-target -- sh -c "/bin/echo -en '{{key_string}}' > /tmp/maas-{{driver['package']}}.gpg"
-  driver_02_key_add: ["curtin", "in-target", "--", "apt-key", "add", "/tmp/maas-{{driver['package']}}.gpg"]
-  {{endif}}
-  {{if driver['repository']}}
-  driver_03_add: ["curtin", "in-target", "--", "add-apt-repository", "-y", "deb {{driver['repository']}} {{node.get_distro_series()}} main"]
-  {{endif}}
-  driver_04_update_install: ["curtin", "in-target", "--", "apt-get", "update", "--quiet"]
-  {{if driver['package']}}
-  driver_05_install: ["curtin", "in-target", "--", "apt-get", "-y", "install", "{{driver['package']}}"]
-  {{endif}}
-  driver_06_depmod: ["curtin", "in-target", "--", "depmod"]
-  driver_07_update_initramfs: ["curtin", "in-target", "--", "update-initramfs", "-u"]
-{{endif}}
+#curtin-config
+## Here you can put custom data that will go to each node.
+## The rendered output must be valid YAML.
+##
+## This file will be rendered as a tempita template.
+## variables available for the rendering are
+##   osystem: "ubuntu", "suse", "windows", ...
+##   release: "xenial", "trusty", ...
+##   server_host: the maas facing server host
+##   server_url: ...
+##   syslog_host_port: ...
+##   metadata_enlist_url: ...
+##   node: the 'node' object
+##   license_key
+##
+## As a simple example, consider:
+#late_commands:
+#  - ["curtin", "in-target", "--", "apt-get", "dist-upgrade"]
diff --git a/contrib/preseeds_v2/curtin_userdata_centos b/contrib/preseeds_v2/curtin_userdata_centos
index 3de9c99..2dd2023 100644
--- a/contrib/preseeds_v2/curtin_userdata_centos
+++ b/contrib/preseeds_v2/curtin_userdata_centos
@@ -1,9 +1,3 @@
-#cloud-config
-debconf_selections:
- maas: |
-  {{for line in str(curtin_preseed).splitlines()}}
-  {{line}}
-  {{endfor}}
-
-late_commands:
-  maas: [wget, '--no-proxy', '{{node_disable_pxe_url}}', '--post-data', '{{node_disable_pxe_data}}', '-O', '/dev/null']
+#curtin-config
+## This file is used only for centos deployments.
+## See curtin_userdata for more information.
diff --git a/contrib/preseeds_v2/curtin_userdata_windows b/contrib/preseeds_v2/curtin_userdata_windows
index 77e4a65..4b7d891 100644
--- a/contrib/preseeds_v2/curtin_userdata_windows
+++ b/contrib/preseeds_v2/curtin_userdata_windows
@@ -1,11 +1,3 @@
 #cloud-config
-debconf_selections:
- maas: |
-  {{for line in str(curtin_preseed).splitlines()}}
-  {{line}}
-  {{endfor}}
-
-late_commands:
-  maas: [wget, '--no-proxy', '{{node_disable_pxe_url}}', '--post-data', '{{node_disable_pxe_data}}', '-O', '/dev/null']
-
-license_key: {{node.get_effective_license_key()}}
+## This file is used only for windows deployments.
+## See curtin_userdata for more information.
diff --git a/src/maasserver/node_action.py b/src/maasserver/node_action.py
index 74c8df7..1830e2f 100644
--- a/src/maasserver/node_action.py
+++ b/src/maasserver/node_action.py
@@ -44,7 +44,7 @@ from maasserver.node_status import (
     is_failed_status,
     NON_MONITORED_STATUSES,
 )
-from maasserver.preseed import get_curtin_config
+from maasserver.preseed import get_curtin_configs
 from maasserver.utils.orm import post_commit_do
 from maasserver.utils.osystems import (
     validate_hwe_kernel,
@@ -347,7 +347,7 @@ class Deploy(NodeAction):
             raise NodeActionError(e)
 
         try:
-            get_curtin_config(self.node)
+            get_curtin_configs(self.node)
         except Exception as e:
             raise NodeActionError(
                 "Failed to retrieve curtin config: %s" % e)
diff --git a/src/maasserver/preseed.py b/src/maasserver/preseed.py
index f2ed29c..d983a80 100644
--- a/src/maasserver/preseed.py
+++ b/src/maasserver/preseed.py
@@ -15,6 +15,7 @@ __all__ = [
     'OS_WITH_IPv6_SUPPORT',
     ]
 
+import base64
 from collections import namedtuple
 import json
 import os.path
@@ -344,7 +345,8 @@ def compose_curtin_verbose_preseed():
 
 def get_curtin_yaml_config(node, default_region_ip=None):
     """Return the curtin configration for the node."""
-    main_config = get_curtin_config(node)
+    main_config = get_curtin_configs(node)
+    custom_config = get_curtin_custom_config(node)
     cloud_config = compose_curtin_cloud_config(node)
     archive_config = compose_curtin_archive_config(node)
     reporter_config = compose_curtin_maas_reporter(node)
@@ -389,9 +391,9 @@ def get_curtin_yaml_config(node, default_region_ip=None):
         storage_config = []
 
     return (
-        storage_config + [main_config] + archive_config + reporter_config +
+        storage_config + main_config + archive_config + reporter_config +
         network_config + swap_config + kernel_config + verbose_config +
-        cloud_config)
+        cloud_config + [custom_config])
 
 
 def get_curtin_merged_config(node):
@@ -475,7 +477,7 @@ def get_curtin_installer_url(node):
     return url_prepend + url
 
 
-def get_curtin_config(node, default_region_ip=None):
+def get_curtin_preseed_context(node, default_region_ip=None):
     """Return the curtin configuration to be used by curtin.pack_install.
 
     :param node: The node for which to generate the configuration.
@@ -483,8 +485,6 @@ def get_curtin_config(node, default_region_ip=None):
     """
     osystem = node.get_osystem()
     series = node.get_distro_series()
-    template = load_preseed_template(
-        node, USERDATA_TYPE.CURTIN, osystem, series)
     rack_controller = node.get_boot_rack_controller()
     context = get_preseed_context(
         osystem, series, rack_controller=rack_controller,
@@ -497,6 +497,131 @@ def get_curtin_config(node, default_region_ip=None):
         get_curtin_context(
             node, rack_controller=rack_controller,
             default_region_ip=default_region_ip))
+    return context
+
+
+def get_third_party_drivers_config_data(enabled, key, repo, package, module,
+                                        series):
+    ret = {'early_commands': {}, 'late_commands': {}}
+    pre = 'driver_0'
+
+    if not enabled:
+        msg = "echo third party drivers not installed or necessary."
+        ret['early_commands'][pre + '0'] = ["echo", msg]
+        ret['late_commands'][pre + '0'] = ["echo", msg]
+        return ret
+
+    early = ret['early_commands']
+    late = ret['late_commands']
+
+    in_target = ["curtin", "in-target", "--"]
+    if key and repo and package:
+        tmpf = "/tmp/maas-" + package
+        echo_key = [
+            'sh', '-c',
+            'echo %s | base64 --decode > %s' % (base64.b64encode(key).decode(),
+                                                tmpf)]
+        apt_add_key = ['apt-key', 'add', tmpf]
+        early[pre + '0_get_key'] = echo_key
+        early[pre + '1_add_key'] = apt_add_key
+        late[pre + '0_key_get'] = in_target + echo_key
+        late[pre + '2_key_add'] = in_target + apt_add_key
+    if repo:
+        add_apt_repo = ["add-apt-repository", "-y",
+                        "deb %s %s main" % (repo, series)]
+        early[pre + '2_add'] = add_apt_repo
+        late[pre + '3_add'] = in_target + add_apt_repo
+    if package:
+        early[pre + '3_update_install'] = [
+            "sh", "-c", ("apt-get update --quiet && "
+                         "apt-get --assume-yes install " + package)]
+        late[pre + '4_update_install'] = in_target + [
+            "apt-get", "update", "--quite"]
+        late[pre + '5_install'] = in_target + [
+            "apt-get", "-y", "install", package]
+    if module:
+        early[pre + '4_load'] = [
+            "sh", "-c",
+            ("depmod && modprobe %s || " % module +
+             "echo 'Warning: Failed to load module: %s" % module)]
+
+    late[pre + '6_depmod'] = in_target + ["depmod"]
+    late[pre + '7_update_initramfs'] = in_target + ["update-initramfs", "-u"]
+    return ret
+
+
+def _get_curtin_base_config_data(context):
+    config = {'debconf_selections': {'maas': context['curtin_preseed']}}
+    config['late_commands'] = {
+        'maas': ["wget", "--no-proxy", context["node_disable_pxe_url"],
+                 "--post-data", context["node_disable_pxe_data"],
+                 "-O", "/dev/null"]}
+
+    # Ensure we always set debconf_selections for grub to ensure it doesn't
+    # overwrite the config sent by MAAS. See LP: #1642298
+    grub2_debconf = {'grub2': 'grub2   grub2/update_nvram  boolean false'}
+    if 'debconf_selections' in config:
+        config['debconf_selections'].update(grub2_debconf)
+    else:
+        config['debconf_selections'] = grub2_debconf
+
+    # Precise does not support cloud-init performing the reboot, so curtin
+    # must have this statement.
+    if context['release'] == "precise":
+        config['power_state'] = {'mode': 'reboot'}
+
+    return config
+
+
+def get_curtin_configs_ubuntu(node):
+    """Return primary curtin config for ubuntu."""
+    context = get_curtin_preseed_context(node)
+    driver = context.get('driver', {})
+    series = node.get_distro_series()
+    return [yaml.safe_dump(c) for c in [
+        _get_curtin_base_config_data(context),
+        get_third_party_drivers_config_data(
+            enabled=context.get('third_party_drivers'),
+            key=driver.get('key_binary'),
+            repo=driver.get('repository'),
+            package=driver.get('package'),
+            module=driver.get('module'),
+            series=series)]]
+
+
+def get_curtin_configs_other(node):
+    """Return primary curtin config for non-ubuntu."""
+    context = get_curtin_preseed_context(node)
+    config = _get_curtin_base_config_data(context)
+    if context['osystem'] == "windows":
+        config['license_key'] = node.get_effective_license_key()
+
+    return [yaml.safe_dump(config)]
+
+
+def get_curtin_configs(node):
+    """Return the primary curtin configuration.
+
+    :param node: The node for which to generate the configuration.
+    :rtype: unicode."""
+    if node.get_osystem() == "ubuntu":
+        return get_curtin_configs_ubuntu(node)
+    return get_curtin_configs_other(node)
+
+
+def get_curtin_custom_config(node):
+    """Return the curtin configuration to be used by curtin.pack_install.
+
+    This reads and renders the template files in /etc/maas/preseeds.
+
+    :param node: The node for which to generate the configuration.
+    :rtype: unicode.
+    """
+    context = get_curtin_preseed_context(node)
+    osystem = node.get_osystem()
+    series = node.get_distro_series()
+    template = load_preseed_template(
+        node, USERDATA_TYPE.CURTIN, osystem, series)
     deprecated_context_variables = [
         'main_archive_hostname', 'main_archive_directory',
         'ports_archive_hostname', 'ports_archive_directory',
@@ -507,6 +632,8 @@ def get_curtin_config(node, default_region_ip=None):
             deprecated_context_variables.remove(var)
     context.update(get_node_deprecated_preseed_context())
     config = yaml.safe_load(template.substitute(**context))
+    if not config:
+        config = {}
     # Remove deprecated config from the curtin preseed.
     if 'power_state' in config:
         del config['power_state']
diff --git a/src/maasserver/tests/test_node_action.py b/src/maasserver/tests/test_node_action.py
index a9368e0..79c46ce 100644
--- a/src/maasserver/tests/test_node_action.py
+++ b/src/maasserver/tests/test_node_action.py
@@ -514,12 +514,12 @@ class TestDeployAction(MAASServerTestCase):
         node = factory.make_Node(
             interface=True, status=NODE_STATUS.ALLOCATED,
             power_type='manual', owner=user)
-        mock_get_curtin_config = self.patch(
-            node_action_module, 'get_curtin_config')
+        mock_get_curtin_configs = self.patch(
+            node_action_module, 'get_curtin_configs')
         mock_node_start = self.patch(node, 'start')
         Deploy(node, user).execute()
         self.expectThat(
-            mock_get_curtin_config, MockCalledOnceWith(node))
+            mock_get_curtin_configs, MockCalledOnceWith(node))
         self.expectThat(
             mock_node_start, MockCalledOnceWith(user))
 
@@ -528,9 +528,9 @@ class TestDeployAction(MAASServerTestCase):
         node = factory.make_Node(
             interface=True, status=NODE_STATUS.ALLOCATED,
             power_type='manual', owner=user)
-        mock_get_curtin_config = self.patch(
-            node_action_module, 'get_curtin_config')
-        mock_get_curtin_config.side_effect = NodeActionError('error')
+        mock_get_curtin_configs = self.patch(
+            node_action_module, 'get_curtin_configs')
+        mock_get_curtin_configs.side_effect = NodeActionError('error')
         error = self.assertRaises(
             NodeActionError, Deploy(node, user).execute)
         self.assertEqual(
@@ -559,8 +559,8 @@ class TestDeployAction(MAASServerTestCase):
         node = factory.make_Node(
             interface=True, status=NODE_STATUS.ALLOCATED,
             power_type='manual', owner=user)
-        mock_get_curtin_config = self.patch(
-            node_action_module, 'get_curtin_config')
+        mock_get_curtin_configs = self.patch(
+            node_action_module, 'get_curtin_configs')
         mock_node_start = self.patch(node, 'start')
         osystem = make_usable_osystem(self)
         os_name = osystem["name"]
@@ -571,7 +571,7 @@ class TestDeployAction(MAASServerTestCase):
         }
         Deploy(node, user).execute(**extra)
         self.expectThat(
-            mock_get_curtin_config, MockCalledOnceWith(node))
+            mock_get_curtin_configs, MockCalledOnceWith(node))
         self.expectThat(
             mock_node_start, MockCalledOnceWith(user))
         self.expectThat(node.osystem, Equals(os_name))
@@ -583,8 +583,8 @@ class TestDeployAction(MAASServerTestCase):
         node = factory.make_Node(
             interface=True, status=NODE_STATUS.ALLOCATED,
             power_type='manual', owner=user)
-        mock_get_curtin_config = self.patch(
-            node_action_module, 'get_curtin_config')
+        mock_get_curtin_configs = self.patch(
+            node_action_module, 'get_curtin_configs')
         mock_node_start = self.patch(node, 'start')
         osystem = make_usable_osystem(self)
         os_name = osystem["name"]
@@ -595,7 +595,7 @@ class TestDeployAction(MAASServerTestCase):
         }
         Deploy(node, user).execute(**extra)
         self.expectThat(
-            mock_get_curtin_config, MockCalledOnceWith(node))
+            mock_get_curtin_configs, MockCalledOnceWith(node))
         self.expectThat(
             mock_node_start, MockCalledOnceWith(user))
         self.expectThat(node.osystem, Equals(os_name))
@@ -607,8 +607,8 @@ class TestDeployAction(MAASServerTestCase):
         node = factory.make_Node(
             interface=True, status=NODE_STATUS.ALLOCATED,
             power_type='manual', owner=user)
-        mock_get_curtin_config = self.patch(
-            node_action_module, 'get_curtin_config')
+        mock_get_curtin_configs = self.patch(
+            node_action_module, 'get_curtin_configs')
         mock_node_start = self.patch(node, 'start')
         osystem = make_osystem_with_releases(self)
         extra = {
@@ -616,7 +616,7 @@ class TestDeployAction(MAASServerTestCase):
         }
         Deploy(node, user).execute(**extra)
         self.expectThat(
-            mock_get_curtin_config, MockCalledOnceWith(node))
+            mock_get_curtin_configs, MockCalledOnceWith(node))
         self.expectThat(
             mock_node_start, MockCalledOnceWith(user))
         self.expectThat(node.osystem, Equals(""))
@@ -627,8 +627,8 @@ class TestDeployAction(MAASServerTestCase):
         node = factory.make_Node(
             interface=True, status=NODE_STATUS.ALLOCATED,
             power_type='manual', owner=user)
-        mock_get_curtin_config = self.patch(
-            node_action_module, 'get_curtin_config')
+        mock_get_curtin_configs = self.patch(
+            node_action_module, 'get_curtin_configs')
         mock_node_start = self.patch(node, 'start')
         osystem = make_osystem_with_releases(self)
         extra = {
@@ -636,7 +636,7 @@ class TestDeployAction(MAASServerTestCase):
         }
         Deploy(node, user).execute(**extra)
         self.expectThat(
-            mock_get_curtin_config, MockCalledOnceWith(node))
+            mock_get_curtin_configs, MockCalledOnceWith(node))
         self.expectThat(
             mock_node_start, MockCalledOnceWith(user))
         self.expectThat(node.osystem, Equals(""))
@@ -645,14 +645,14 @@ class TestDeployAction(MAASServerTestCase):
     def test_Deploy_allocates_node_if_node_not_already_allocated(self):
         user = factory.make_User()
         node = factory.make_Node(status=NODE_STATUS.READY, with_boot_disk=True)
-        mock_get_curtin_config = self.patch(
-            node_action_module, 'get_curtin_config')
+        mock_get_curtin_configs = self.patch(
+            node_action_module, 'get_curtin_configs')
         mock_node_start = self.patch(node, 'start')
         action = Deploy(node, user)
         action.execute()
 
         self.expectThat(
-            mock_get_curtin_config, MockCalledOnceWith(node))
+            mock_get_curtin_configs, MockCalledOnceWith(node))
         self.expectThat(
             mock_node_start, MockCalledOnceWith(user))
         self.expectThat(user, Equals(node.owner))
diff --git a/src/maasserver/tests/test_preseed.py b/src/maasserver/tests/test_preseed.py
index 6f5e7d6..bde4bc9 100644
--- a/src/maasserver/tests/test_preseed.py
+++ b/src/maasserver/tests/test_preseed.py
@@ -13,6 +13,7 @@ from textwrap import dedent
 from unittest.mock import sentinel
 from urllib.parse import urlparse
 
+from curtin.config import merge_config
 from django.conf import settings
 from maasserver import preseed as preseed_module
 from maasserver.clusterrpc.testing.boot_images import make_rpc_boot_image
@@ -47,7 +48,7 @@ from maasserver.preseed import (
     curtin_maas_reporter,
     GENERIC_FILENAME,
     get_curtin_cloud_config,
-    get_curtin_config,
+    get_curtin_configs,
     get_curtin_context,
     get_curtin_image,
     get_curtin_installer_url,
@@ -126,6 +127,14 @@ class BootImageHelperMixin:
             preseed_module,
             'get_boot_images_for').return_value = [boot_image]
 
+    def get_merged_config(self, result):
+        """Return the fully merged config from the list of yaml_blobs
+           that is provided in result (as returned by get_curtin_configs)."""
+        ret = {}
+        for yaml_blob in result:
+            merge_config(ret, yaml.safe_load(yaml_blob))
+        return ret
+
 
 class TestSplitSubArch(MAASServerTestCase):
     """Tests for `split_subarch`."""
@@ -977,8 +986,9 @@ class TestRenderCurtinUserdataWithThirdPartyDrivers(
         get_third_party_driver = self.patch(
             preseed_module, "get_third_party_driver")
         get_third_party_driver.return_value = self.driver
-        curtin_config_text = get_curtin_config(node)
-        config = yaml.safe_load(curtin_config_text)
+        config = self.get_merged_config(get_curtin_configs(node))
+        self.assertThat(config, Contains('early_commands'))
+
         self.assertThat(
             config['early_commands'], Contains('driver_00_get_key'))
         self.assertThat(
@@ -1035,11 +1045,11 @@ class TestCurtinUtilities(
         node = factory.make_Node_with_Interface_on_Subnet(
             primary_rack=self.rpc_rack_controller)
         self.configure_get_boot_images_for_node(node, 'xinstall')
-        config = get_curtin_config(node)
-        self.assertThat(
-            config,
-            Contains("debconf_selections:"))
-        self.assertThat(config, Not(Contains('mode: reboot')))
+        config = self.get_merged_config(get_curtin_configs(node))
+        self.assertIsNotNone(config.get('early_commands'),
+                             "Did not find 'early_commands' entry")
+        self.assertNotEqual(
+            config.get('power_state', {}).get('mode', None), 'reboot')
 
     def test_get_curtin_config_removes_power_state(self):
         node = factory.make_Node_with_Interface_on_Subnet(
@@ -1051,8 +1061,9 @@ class TestCurtinUtilities(
         """)
         self.patch(preseed_module, "get_preseed_template").return_value = (
             factory.make_name("filename"), power_state_template)
-        config = get_curtin_config(node)
-        self.assertThat(config, Not(Contains('mode: reboot')))
+        config = self.get_merged_config(get_curtin_configs(node))
+        self.assertNotEqual(
+            'reboot', config.get('power_state', {}).get('mode'), 'reboot')
 
     def test_get_curtin_config_removes_apt_mirrors(self):
         node = factory.make_Node_with_Interface_on_Subnet(
@@ -1065,9 +1076,9 @@ class TestCurtinUtilities(
         """)
         self.patch(preseed_module, "get_preseed_template").return_value = (
             factory.make_name("filename"), apt_mirrors_template)
-        config = get_curtin_config(node)
-        self.assertThat(config, Not(Contains('ubuntu_archive')))
-        self.assertThat(config, Not(Contains('ubuntu_security')))
+        config = self.get_merged_config(get_curtin_configs(node))
+        self.assertNotIn('ubuntu_archive', config.get('apt_mirrors', {}))
+        self.assertNotIn('ubuntu_security', config.get('apt_mirrors', {}))
 
     def test_get_curtin_config_removes_apt_proxy(self):
         node = factory.make_Node_with_Interface_on_Subnet(
@@ -1078,8 +1089,9 @@ class TestCurtinUtilities(
         """)
         self.patch(preseed_module, "get_preseed_template").return_value = (
             factory.make_name("filename"), apt_proxy_template)
-        config = get_curtin_config(node)
-        self.assertThat(config, Not(Contains('127.0.0.1')))
+        config = self.get_merged_config(get_curtin_configs(node))
+        self.assertThat(config.get('apt_proxy', ''),
+                        Not(Contains('127.0.0.1')))
 
     def test_get_curtin_config_contains_reboot_for_precise(self):
         node = factory.make_Node_with_Interface_on_Subnet(
@@ -1087,8 +1099,8 @@ class TestCurtinUtilities(
         node.distro_series = "precise"
         node.save()
         self.configure_get_boot_images_for_node(node, 'xinstall')
-        config = get_curtin_config(node)
-        self.assertThat(config, Contains('mode: reboot'))
+        config = self.get_merged_config(get_curtin_configs(node))
+        self.assertEqual(config['power_state']['mode'], "reboot")
 
     def test_get_curtin_config_with_ipv4_rack_url(self):
         primary_rack = self.rpc_rack_controller
@@ -1097,13 +1109,12 @@ class TestCurtinUtilities(
         node = factory.make_Node_with_Interface_on_Subnet(
             primary_rack=primary_rack)
         self.configure_get_boot_images_for_node(node, 'xinstall')
-        config = get_curtin_config(node)
-        yaml_conf = yaml.safe_load(config)
+        config = self.get_merged_config(get_curtin_configs(node))
         self.assertEqual(
             "%smetadata/latest/by-id/%s/" % (
                 primary_rack.url, node.system_id),
-            yaml_conf['late_commands']['maas'][2])
-        self.assertTrue('debconf_selections' in yaml_conf)
+            config['late_commands']['maas'][2])
+        self.assertTrue('debconf_selections' in config)
 
     def test_get_curtin_config_with_ipv6_rack_url(self):
         primary_rack = self.rpc_rack_controller
@@ -1112,13 +1123,12 @@ class TestCurtinUtilities(
         node = factory.make_Node_with_Interface_on_Subnet(
             primary_rack=primary_rack)
         self.configure_get_boot_images_for_node(node, 'xinstall')
-        config = get_curtin_config(node)
-        yaml_conf = yaml.safe_load(config)
+        config = self.get_merged_config(get_curtin_configs(node))
         self.assertEqual(
             "%smetadata/latest/by-id/%s/" % (
                 primary_rack.url, node.system_id),
-            yaml_conf['late_commands']['maas'][2])
-        self.assertTrue('debconf_selections' in yaml_conf)
+            config['late_commands']['maas'][2])
+        self.assertTrue('debconf_selections' in config)
 
     def test_get_curtin_config_with_name_rack_url(self):
         primary_rack = self.rpc_rack_controller
@@ -1127,13 +1137,12 @@ class TestCurtinUtilities(
         node = factory.make_Node_with_Interface_on_Subnet(
             primary_rack=primary_rack)
         self.configure_get_boot_images_for_node(node, 'xinstall')
-        config = get_curtin_config(node)
-        yaml_conf = yaml.safe_load(config)
+        config = self.get_merged_config(get_curtin_configs(node))
         self.assertEqual(
             "%smetadata/latest/by-id/%s/" % (
                 primary_rack.url, node.system_id),
-            yaml_conf['late_commands']['maas'][2])
-        self.assertTrue('debconf_selections' in yaml_conf)
+            config['late_commands']['maas'][2])
+        self.assertTrue('debconf_selections' in config)
 
     def test_get_curtin_config_with_quote_rack_url(self):
         primary_rack = self.rpc_rack_controller
@@ -1142,22 +1151,21 @@ class TestCurtinUtilities(
         node = factory.make_Node_with_Interface_on_Subnet(
             primary_rack=primary_rack)
         self.configure_get_boot_images_for_node(node, 'xinstall')
-        config = get_curtin_config(node)
-        yaml_conf = yaml.safe_load(config)
+        config = self.get_merged_config(get_curtin_configs(node))
         self.assertEqual(
             "%smetadata/latest/by-id/%s/" % (primary_rack.url, node.system_id),
-            yaml_conf['late_commands']['maas'][2])
-        self.assertTrue('debconf_selections' in yaml_conf)
+            config['late_commands']['maas'][2])
+        self.assertTrue('debconf_selections' in config)
 
     def test_get_curtin_config_has_grub2_debconf_selections(self):
         node = factory.make_Node_with_Interface_on_Subnet(
             primary_rack=self.rpc_rack_controller)
         node.save()
         self.configure_get_boot_images_for_node(node, 'xinstall')
-        config = get_curtin_config(node)
-        self.assertThat(
-            config,
-            Contains('grub2: grub2   grub2/update_nvram  boolean false'))
+        config = self.get_merged_config(get_curtin_configs(node))
+        self.assertEqual(
+            config['debconf_selections']['grub2'].split(),
+            ['grub2', 'grub2/update_nvram', 'boolean', 'false'])
 
     def make_fastpath_node(self, main_arch=None):
         """Return a `Node`, with FPI enabled, and the given main architecture.