Merge ~mfo/maas:lp2020397-opt-out into maas:master

[Mauricio Faria de Oliveira]

Introduce an option to opt-out for custom image dependency validation in the curtin_userdata preseed.

custom_validation: 0

More details in the commit message.

[MAAS Lander (maas-lander)]

UNIT TESTS
-b lp2020397-opt-out lp:~mfo/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas-tester/2661/console
COMMIT: 26be7fdf406b47d138c592eaea0806b2d38fc072

review: Needs Fixing

[Mauricio Faria de Oliveira (mfo)]

jenkins: !test

[MAAS Lander (maas-lander)]

UNIT TESTS
-b lp2020397-opt-out lp:~mfo/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/maas-tester/2664/console
COMMIT: 6cf0398316addba1b0597d5acef8044519924d3c

review: Needs Fixing

[Mauricio Faria de Oliveira (mfo)]

jenkins: !test

[Mauricio Faria de Oliveira (mfo)]

jenkins: !test

[MAAS Lander (maas-lander)]

UNIT TESTS
-b lp2020397-opt-out lp:~mfo/maas/+git/maas into -b master lp:~maas-committers/maas

STATUS: SUCCESS
COMMIT: e1cd74f9583e91fd598f98d60a20a4191d225e8b

review: Approve

[Jack Lloyd-Walters (lloydwaltersj)]

+1

review: Approve

[Jack Lloyd-Walters (lloydwaltersj)]

review: Abstain

Unmerged commits

e1cd74f... by Mauricio Faria de Oliveira on 2023-06-05

LP: #2020397 - Opt-out for custom image dependency validation

Some custom images might not have any package manager at all,
thus fail the (curtin) late commands on deployment for custom
image dependency validation (cloud-init and netplan packages).

Even though these checks may be perceived as a regression in
such cases, it is likely better to keep them by default, for
more robust checks/assertions on deployment vs. boot errors.

As a workaround, users could add late commands to create/remove
symlinks of the package manager command to /bin/true to fake it,
but that is hacky, and might fail after source code changes.

So, let's provide a way for these users to opt-out of custom
image dependency validation with the curtin_userdata preseed
(since these checks are introduced as curtin late commands),
in an upstream-maintained manner.

In the curtin_userdata preseed/template, introduce this line:
custom_validation: 0

This would seem better than introducing per-custom image
properties (database changes involved, probably) or some
CLI/UI changes and pipe them to the deployment functions.

Tests
=====

Steps: download centos70 image, upload as custom, deploy.

Default:

    start: cmd-install/stage-late: executing late commands
    start: cmd-install/stage-late/98-validate-custom-image-has-cloud-init: running 'curtin in-target -- bash -c dpkg-query -s cloud-init || (echo "cloud-init not detected, ..." && exit 1)'
    ...
    bash: dpkg-query: command not found
    cloud-init not detected, MAAS will not be able to configure this machine properly
    ...
    98-validate-custom-image-has-cloud-init command failed
    finish: cmd-install/stage-late/98-validate-custom-image-has-cloud-init: FAIL: running 'curtin in-target -- bash -c dpkg-query -s cloud-init || (echo "cloud-init not detected, ..." && exit 1)'
    ...
    finish: cmd-install/stage-late: FAIL: executing late commands
    curtin: Installation failed with exception: Unexpected error while running command.

Opt-out:

    start: cmd-install/stage-late: executing late commands
    ...
    finish: cmd-install/stage-late: SUCCESS: executing late commands
    curtin: Installation finished.

Unit tests:

    $ make install-dependencies
    $ make bin/pytest
    $ bin/pytest src/maasserver/tests/test_preseed.py -k TestGetCustomImageDependencyValidation -v --no-header --no-summary
    ...
    src/maasserver/tests/test_preseed.py::TestGetCustomImageDependencyValidation::test_validation_for_custom_centos PASSED [ 14%]
    src/maasserver/tests/test_preseed.py::TestGetCustomImageDependencyValidation::test_validation_for_custom_rhel PASSED [ 28%]
    src/maasserver/tests/test_preseed.py::TestGetCustomImageDependencyValidation::test_validation_for_custom_ubuntu PASSED [ 42%]
    src/maasserver/tests/test_preseed.py::TestGetCustomImageDependencyValidation::test_validation_for_linuxes PASSED [ 57%]
    src/maasserver/tests/test_preseed.py::TestGetCustomImageDependencyValidation::test_validation_for_linuxes_skip_zero PASSED [ 71%]
    src/maasserver/tests/test_preseed.py::TestGetCustomImageDependencyValidation::test_validation_for_linuxes_skip_zero_only PASSED [ 85%]
    src/maasserver/tests/test_preseed.py::TestGetCustomImageDependencyValidation::test_validation_for_non_custom PASSED [100%]
    ...

Signed-off-by: Mauricio Faria de Oliveira <email address hidden>

diff --git a/src/maasserver/preseed.py b/src/maasserver/preseed.py
index 94d77c0..57bd72d 100644
--- a/src/maasserver/preseed.py
+++ b/src/maasserver/preseed.py
@@ -632,8 +632,18 @@ def get_curtin_config(request, node, base_osystem=None, base_series=None):
             }
         )
 
+    # Allow users to skip custom image dependency validation
+    # with 'custom_validation: 0' in curtin_userdata(_custom).
+    custom_validation_config = None
+    if "custom_validation" in config:
+        # The key is only valid on custom osystem, but may exist
+        # on non-custom (shared template, e.g., curtin_userdata).
+        if osystem == "custom":
+            custom_validation_config = config["custom_validation"]
+        del config["custom_validation"]
+
     custom_validation = get_custom_image_dependency_validation(
-        node, base_osystem
+        node, base_osystem, custom_validation_config
     )
     if custom_validation is not None:
         config["late_commands"].update(custom_validation)
@@ -655,9 +665,11 @@ PACKAGE_MANAGER_PER_OS = {
 }
 
 
-def get_custom_image_dependency_validation(node, base_osystem):
+def get_custom_image_dependency_validation(node, base_osystem, config=None):
     if node.get_osystem() != "custom":
         return None
+    if config == 0:
+        return None
 
     cmd = {}
     err_msg = "not detected, MAAS will not be able to configure this machine properly"
diff --git a/src/maasserver/tests/test_preseed.py b/src/maasserver/tests/test_preseed.py
index 8d36e2a..f068a4f 100644
--- a/src/maasserver/tests/test_preseed.py
+++ b/src/maasserver/tests/test_preseed.py
@@ -442,6 +442,46 @@ class TestGetCustomImageDependencyValidation(MAASServerTestCase):
                 get_custom_image_dependency_validation(machine, osystem)
             )
 
+    def test_validation_for_linuxes_skip_zero(self):
+        machine = factory.make_Machine(
+            status=NODE_STATUS.DEPLOYED,
+            architecture="amd64/generic",
+            osystem="custom",
+        )
+        for osystem in LINUX_OSYSTEMS:
+            distro_series = factory.make_name(osystem)
+            factory.make_BootResource(
+                name=f"custom/{distro_series}",
+                base_image=f"{osystem}/{osystem}",
+                architecture="amd64/generic",
+            )
+            machine.distro_series = distro_series
+            machine.save()
+            self.assertIsNone(
+                get_custom_image_dependency_validation(machine, osystem, 0)
+            )
+
+    def test_validation_for_linuxes_skip_zero_only(self):
+        machine = factory.make_Machine(
+            status=NODE_STATUS.DEPLOYED,
+            architecture="amd64/generic",
+            osystem="custom",
+        )
+        for osystem in LINUX_OSYSTEMS:
+            distro_series = factory.make_name(osystem)
+            factory.make_BootResource(
+                name=f"custom/{distro_series}",
+                base_image=f"{osystem}/{osystem}",
+                architecture="amd64/generic",
+            )
+            machine.distro_series = distro_series
+            machine.save()
+            self.assertIsNotNone(
+                get_custom_image_dependency_validation(
+                    machine, osystem, "not0"
+                )
+            )
+
 
 class TestLoadPreseedTemplate(MAASServerTestCase):
     """Tests for `load_preseed_template`."""