Merge ~lloydwaltersj/maas-ci/+git/system-tests:image_mapping-parity-with-packerMAAS into ~maas-committers/maas-ci/+git/system-tests:master

[Jack Lloyd-Walters]

Add images: Alma8, Alma9
Add Series: Ubuntu:Noble
Add Arches: Ubuntu:arm64, Debian:arm64
Add Arch support in gen_config and image_builder
Skips tests if an incompatible machine and image arch are generated

Packer MAAS has become slightly out of sync with the system tests, let's fix that:

- Add all the images packer maas supports to the image mapping
- Add all the arches we have confirmed working to the image mapping
- Add the ability to build said arches
- Add image testing skipping if the machine and image arch do not match

d5991c7... by Jack Lloyd-Walters on 2024-03-20

fix example in genconfig

[MAAS Lander (maas-lander)]

UNIT TESTS
-b image_mapping-parity-with-packerMAAS lp:~lloydwaltersj/maas-ci/+git/system-tests into -b master lp:~maas-committers/maas-ci/+git/system-tests

STATUS: SUCCESS
COMMIT: a7f9e364e0798b430698134844c58ed858c28e5f

review: Approve

[MAAS Lander (maas-lander)]

UNIT TESTS
-b image_mapping-parity-with-packerMAAS lp:~lloydwaltersj/maas-ci/+git/system-tests into -b master lp:~maas-committers/maas-ci/+git/system-tests

STATUS: SUCCESS
COMMIT: d5991c707c23949df63ed8006f30fca0a333ad7c

review: Approve

590856f... by Jack Lloyd-Walters on 2024-03-20

add alma and debian to pretty os name

[MAAS Lander (maas-lander)]

UNIT TESTS
-b image_mapping-parity-with-packerMAAS lp:~lloydwaltersj/maas-ci/+git/system-tests into -b master lp:~maas-committers/maas-ci/+git/system-tests

STATUS: SUCCESS
COMMIT: 590856f1110915e0f9b0b2a6657b43c0a0f945af

review: Approve

f63a5b5... by Jack Lloyd-Walters on 2024-03-20

better errors

[MAAS Lander (maas-lander)]

UNIT TESTS
-b image_mapping-parity-with-packerMAAS lp:~lloydwaltersj/maas-ci/+git/system-tests into -b master lp:~maas-committers/maas-ci/+git/system-tests

STATUS: SUCCESS
COMMIT: f63a5b5e7ff31e9b6116233085e75d2884a696e1

review: Approve

98d4c66... by Jack Lloyd-Walters on 2024-03-20

don't add arch to QEMU filename

[MAAS Lander (maas-lander)]

UNIT TESTS
-b image_mapping-parity-with-packerMAAS lp:~lloydwaltersj/maas-ci/+git/system-tests into -b master lp:~maas-committers/maas-ci/+git/system-tests

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/system-tests-tester/1031/console
COMMIT: 98d4c66dbff4aef5ad7811c1b5ad64c280fe12dd

review: Needs Fixing

8ccb147... by Jack Lloyd-Walters on 2024-03-20

add noble to image mapping

1b26db2... by Jack Lloyd-Walters on 2024-03-20

add qemu arm package

[MAAS Lander (maas-lander)]

UNIT TESTS
-b image_mapping-parity-with-packerMAAS lp:~lloydwaltersj/maas-ci/+git/system-tests into -b master lp:~maas-committers/maas-ci/+git/system-tests

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/system-tests-tester/1032/console
COMMIT: 1b26db2087c260dda383008ef827fb93e77f1558

review: Needs Fixing

c7b0b4f... by Jack Lloyd-Walters on 2024-03-20

add timeout option to packer

[MAAS Lander (maas-lander)]

UNIT TESTS
-b image_mapping-parity-with-packerMAAS lp:~lloydwaltersj/maas-ci/+git/system-tests into -b master lp:~maas-committers/maas-ci/+git/system-tests

STATUS: SUCCESS
COMMIT: c7b0b4f341c5ad6c444d1bc0e1667315fe20a902

review: Approve

91153a3... by Jack Lloyd-Walters on 2024-03-23

add more granular arch support

[MAAS Lander (maas-lander)]

UNIT TESTS
-b image_mapping-parity-with-packerMAAS lp:~lloydwaltersj/maas-ci/+git/system-tests into -b master lp:~maas-committers/maas-ci/+git/system-tests

STATUS: SUCCESS
COMMIT: 91153a35b87ddf614296238f10e7e939b5bcd446

review: Approve

26b8463... by Jack Lloyd-Walters on 2024-03-25

add arm64 to ubuntu-flat

035b101... by Jack Lloyd-Walters on 2024-03-26

add arch to build log file name

[MAAS Lander (maas-lander)]

UNIT TESTS
-b image_mapping-parity-with-packerMAAS lp:~lloydwaltersj/maas-ci/+git/system-tests into -b master lp:~maas-committers/maas-ci/+git/system-tests

STATUS: FAILED
LOG: http://maas-ci.internal:8080/job/system-tests-tester/1035/console
COMMIT: 035b10196e7e6b5c384f1adff13f07716f74649f

review: Needs Fixing

48bd337... by Jack Lloyd-Walters on 2024-03-26

linting

[MAAS Lander (maas-lander)]

UNIT TESTS
-b image_mapping-parity-with-packerMAAS lp:~lloydwaltersj/maas-ci/+git/system-tests into -b master lp:~maas-committers/maas-ci/+git/system-tests

STATUS: SUCCESS
COMMIT: 48bd337afbdf2647cd69eba87a9a798f82adbecb

review: Approve

[Jack Lloyd-Walters (lloydwaltersj)]

jenkins: !test

[MAAS Lander (maas-lander)]

UNIT TESTS
-b image_mapping-parity-with-packerMAAS lp:~lloydwaltersj/maas-ci/+git/system-tests into -b master lp:~maas-committers/maas-ci/+git/system-tests

STATUS: SUCCESS
COMMIT: 48bd337afbdf2647cd69eba87a9a798f82adbecb

review: Approve

15db22e... by Jack Lloyd-Walters on 2024-04-18

skip if architecture wrong

[MAAS Lander (maas-lander)]

UNIT TESTS
-b image_mapping-parity-with-packerMAAS lp:~lloydwaltersj/maas-ci/+git/system-tests into -b master lp:~maas-committers/maas-ci/+git/system-tests

STATUS: SUCCESS
COMMIT: 15db22ed6f7fd471369cc2a7c9c73bd947253128

review: Approve

Unmerged commits

15db22e... by Jack Lloyd-Walters on 2024-04-18

skip if architecture wrong

lint:0 (build)

1 → 1 of 1 result

First • Previous • Next • Last

48bd337... by Jack Lloyd-Walters on 2024-03-26

linting

lint:0 (build)

1 → 1 of 1 result

First • Previous • Next • Last

035b101... by Jack Lloyd-Walters on 2024-03-26

add arch to build log file name

lint:0 (build)

1 → 1 of 1 result

First • Previous • Next • Last

26b8463... by Jack Lloyd-Walters on 2024-03-25

add arm64 to ubuntu-flat

91153a3... by Jack Lloyd-Walters on 2024-03-23

add more granular arch support

lint:0 (build)

1 → 1 of 1 result

First • Previous • Next • Last

c7b0b4f... by Jack Lloyd-Walters on 2024-03-20

add timeout option to packer

lint:0 (build)

1 → 1 of 1 result

First • Previous • Next • Last

1b26db2... by Jack Lloyd-Walters on 2024-03-20

add qemu arm package

lint:0 (build)

1 → 1 of 1 result

First • Previous • Next • Last

8ccb147... by Jack Lloyd-Walters on 2024-03-20

add noble to image mapping

98d4c66... by Jack Lloyd-Walters on 2024-03-20

don't add arch to QEMU filename

lint:0 (build)

1 → 1 of 1 result

First • Previous • Next • Last

f63a5b5... by Jack Lloyd-Walters on 2024-03-20

better errors

lint:0 (build)

1 → 1 of 1 result

First • Previous • Next • Last

diff --git a/image_mapping.yaml b/image_mapping.yaml
index 69a0eea..32b2af7 100644
--- a/image_mapping.yaml
+++ b/image_mapping.yaml
@@ -15,14 +15,55 @@
 #        packer_target: $PACKER_TARGET
 #        source_iso: $ISO_NAME
 
-# Also includes a mapping between main packer image and series
-# name, for the images that accept that argument (ie: ubuntu)
-
+# Includes a mapping between main packer image and series: `images_with_series`
 # some images wrap multiple series, which are buildable by passing
 # `SERIES=$series` during make. This maps those images, along with
-# the valid series that are passable
+# the valid series that are passable. ie:
+# images_with_series:
+#  $IMAGE:
+#    - $SERIES_1
+#    - $SERIES_2
+
+# Includes a mapping between packer image and architecture: `images_with_arches`
+# some images are buildable for different architectures, which are
+# accessed by passing `ARCH=$arch` during make. This maps the architectures
+# buildable to the image, alongside any per-series support for an arch, ie:
+# images_with_arches:
+#  $IMAGE:
+#    arches:
+#      - $ARCH_1
+#      - $ARCH_2
+#    supported_series:
+#      $ARCH_2:
+#        - $SERIES_1
+#        - $SERIES_2
+#  $IMAGE_WITH_CONF:
+#    arches:
+#      - arch_2
+# if an arch is not found in supported_Series, it is assumed that all series
+# for the image support all arches. (If the image does not define a series, this
+# can obviously be ignored). If an image has a configuration option, we search
+# for that first, defaulting to the base image (ie: ubuntu-flat -> ubuntu)
 
 images:
+  alma8:
+    filename: alma8.tar.gz
+    filetype: tgz
+    architecture: amd64/generic
+    osystem: custom
+    oseries: "alma8"
+    packer_template: alma8
+    base_image: "rhel/8"
+    ssh_username: cloud-user
+  alma9:
+    filename: alma9.tar.gz
+    filetype: tgz
+    architecture: amd64/generic
+    osystem: custom
+    oseries: "alma9"
+    packer_template: alma9
+    base_image: "rhel/9"
+    ssh_username: cloud-user
   centos6:
     filename: centos6.tar.gz
     filetype: tgz
@@ -202,7 +243,12 @@ images:
     oseries: "ubuntu-raw-lvm"
     packer_template: ubuntu
     packer_target: custom-ubuntu-lvm.dd.gz
+
 images_with_series:
+  debian:
+    - buster
+    - bullseye
+    - bookworm
   ubuntu:
     - xenial
     - bionic
@@ -210,7 +256,21 @@ images_with_series:
     - jammy
     - lunar
     - mantic
+    - noble
+
+images_with_arches:
   debian:
-    - buster
-    - bullseye
-    - bookworm
+    arches:
+      - amd64
+      - arm64
+  ubuntu:
+    arches:
+      - amd64
+      - arm64
+  ubuntu-flat:
+    arches:
+      - amd64
+      - arm64
+  ubuntu-lvm:
+    arches:
+      - amd64
diff --git a/lxd_configs/packer.profile b/lxd_configs/packer.profile
index e2772ec..b3c5740 100644
--- a/lxd_configs/packer.profile
+++ b/lxd_configs/packer.profile
@@ -5,4 +5,3 @@ devices:
         path: /root/.cache/packer
         source: /home/ubuntu/packer/cache
         type: disk
-        shift: true
diff --git a/pyproject.toml b/pyproject.toml
index f8d44e5..5eb0f4a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -8,6 +8,7 @@ markers = [
     "skip_if_installed_from_deb_package", # Skips tests if MAAS is installed from package.
     "skip_if_packer_unconfigured", # Skips tests if Packer-MAAS is not configured.
     "skip_if_os_not_supported", # Skips tests if the OS chosen is not compatible with the machine architecture
+    "skip_if_arch_incompatible", # Skips test if the machine and image arches do not match
     "skip_if_terraform_unconfigured" # Skips tests if Terraform is not configured.
 ]
 log_level = "INFO"
diff --git a/systemtests/api.py b/systemtests/api.py
index 894f93b..1c31099 100644
--- a/systemtests/api.py
+++ b/systemtests/api.py
@@ -352,12 +352,14 @@ class AuthenticatedAPIClient:
 
     def pretty_os_name(self, osystem: str) -> str:
         oses = {
+            "alma": "AlmaLinux",
             "centos": "CentOS",
+            "debian": "Debian",
+            "esxi": "ESXi",
             "ol": "OracleLinux",
             "rhel": "RHEL",
             "rocky": "RockyLinux",
             "sles": "SLES",
-            "esxi": "ESXi",
             "ubuntu": "Ubuntu",
             "windows": "Windows",
         }
@@ -379,8 +381,8 @@ class AuthenticatedAPIClient:
             f"Available images: {', '.join(resources)}"
         )
 
-    def wait_for_images(self) -> None:
-        for retry_info in retries(timeout=20 * 60, delay=10):
+    def wait_for_images(self, timeout: int = 30 * 60) -> None:
+        for retry_info in retries(timeout=timeout, delay=10):
             if not self.is_importing_boot_resources():
                 break
         else:
@@ -389,9 +391,9 @@ class AuthenticatedAPIClient:
                 f" over {retry_info.elapsed} seconds."
             )
 
-    def import_and_wait_for_images(self) -> None:
+    def import_and_wait_for_images(self, timeout: int = 30 * 60) -> None:
         self.import_boot_resources()
-        self.wait_for_images()
+        self.wait_for_images(timeout=timeout)
 
     def list_machines(self, **kwargs: Any) -> list[Machine]:
         """
diff --git a/systemtests/conftest.py b/systemtests/conftest.py
index a7e9111..4453271 100644
--- a/systemtests/conftest.py
+++ b/systemtests/conftest.py
@@ -52,6 +52,7 @@ from .fixtures import (
     pool,
     skip_if_ansible_collection_unconfigured,
     skip_if_ansible_playbooks_unconfigured,
+    skip_if_arch_incompatible,
     skip_if_dns_unconfigured,
     skip_if_installed_from_deb_package,
     skip_if_installed_from_snap,
@@ -70,6 +71,7 @@ from .lxd import Instance, get_lxd
 from .machine_config import MachineConfig
 from .models import HardwareSyncMachine, MachineUnderTest
 from .o11y import add_o11y_header
+from .packer import add_packer_header
 from .state import (
     authenticated_admin,
     configured_maas,
@@ -114,6 +116,7 @@ __all__ = [
     "skip_if_installed_from_deb_package",
     "skip_if_installed_from_snap",
     "skip_if_os_not_supported",
+    "skip_if_arch_incompatible",
     "ssh_key",
     "tag_all",
     "terraform_main",
@@ -204,6 +207,7 @@ def pytest_report_header(config: pytest.Config) -> list[str]:
     add_vault_header(headers, systemtests_config)
     add_o11y_header(headers, systemtests_config)
     add_ansible_header(headers, systemtests_config)
+    add_packer_header(headers, systemtests_config)
     return headers + ["END_SYSTEMTESTS_META"]
 
 
diff --git a/systemtests/fixtures.py b/systemtests/fixtures.py
index 20f856f..473046c 100644
--- a/systemtests/fixtures.py
+++ b/systemtests/fixtures.py
@@ -17,6 +17,7 @@ from .ansible_collection import AnsibleCollectionMain
 from .api import AuthenticatedAPIClient, UnauthenticatedMAASAPIClient
 from .config import ADMIN_EMAIL, ADMIN_PASSWORD, ADMIN_USER
 from .dnstester import DNSTester
+from .image_config import TestableImage
 from .lxd import Instance, get_lxd
 from .machine_config import MachineConfig
 from .o11y import is_o11y_enabled, setup_o11y
@@ -705,6 +706,19 @@ def skip_if_os_not_supported(request: Any) -> None:
 
 
 @pytest.fixture(autouse=True)
+def skip_if_arch_incompatible(request: Any) -> None:
+    if marker := request.node.get_closest_marker("skip_if_arch_incompatible"):
+        # Avoid instantiating machine or image configurations unless marker applied
+        machine_config: MachineConfig = request.getfixturevalue("machine_config")
+        image_to_test: TestableImage = request.getfixturevalue("image_to_test")
+        reason = marker.args[0]
+        if image_to_test and (
+            image_to_test.architecture != machine_config.architecture
+        ):
+            pytest.skip(reason)
+
+
[e-mail](autouse=True)
 def skip_if_packer_unconfigured(request: Any, config: dict[str, Any]) -> None:
     """Skip tests that require Packer configuration."""
     marker = request.node.get_closest_marker("skip_if_packer_unconfigured")
diff --git a/systemtests/image_builder/test_packer.py b/systemtests/image_builder/test_packer.py
index 83f0302..c015f57 100644
--- a/systemtests/image_builder/test_packer.py
+++ b/systemtests/image_builder/test_packer.py
@@ -32,20 +32,36 @@ class TestPackerMAASConfig:
         deps = packer_main.get_dependencies(image_to_build.packer_template)
         testlog.info(f"Image Dependencies: {deps}")
 
-        image, preseed = packer_main.build_image(
+        image, alt_image, preseed = packer_main.build_image(
             image_to_build.packer_template,
             image_to_build.packer_target,
             image_to_build.packer_filename,
+            image_to_build.alternative_packer_filename,
             image_to_build.build_series,
+            image_to_build.build_architecture,
             image_to_build.preseed,
             image_to_build.source_iso,
         )
         assert image is not None
         img_file = packer_main._instance.files[image]
-        assert img_file.exists(), f"failed to produce the expected image ({img_file})"
+        if not img_file.exists() and alt_image and alt_image != image:
+            packer_main.logger.error(
+                f"Trying alternative filename {alt_image}, {image} doesn't exist."
+            )
+            img_file = packer_main._instance.files[alt_image]
+        assert (
+            img_file.exists()
+        ), f"Failed to produce the expected image ({img_file.path})"
 
         if image_to_build.filename:
             packer_main.upload_image(img_file, image_to_build.filename)
+            packer_main.logger.info(
+                f"{image_to_build.name} uploaded as {image_to_build.filename}"
+            )
+
         if preseed_filename := image_to_build.preseed_filename:
             preseed_file = packer_main._instance.files[preseed]
             packer_main.upload_preseed(preseed_file, preseed_filename)
+            packer_main.logger.info(
+                f"{image_to_build.name} preseed uploaded as {preseed_filename}"
+            )
diff --git a/systemtests/image_config.py b/systemtests/image_config.py
index 57bbe0f..556e728 100644
--- a/systemtests/image_config.py
+++ b/systemtests/image_config.py
@@ -31,11 +31,12 @@ class TestableImage:
     packer_template: str | None = None
     packer_target: str | None = None
     build_series: str | None = None
+    build_architecture: str | None = None
     preseed: str | None = None
     source_iso: str | None = None
 
     def __str__(self) -> str:
-        return f"{self.osystem}/{self.oseries}"
+        return self.filename.split(".")[0]
 
     @classmethod
     def from_config(
@@ -59,6 +60,14 @@ class TestableImage:
         return self.packer_target or f"{self.packer_template}.{ext}"
 
     @property
+    def alternative_packer_filename(self) -> str:
+        """If the packer name doesn't match, try this."""
+        ext = EXTENSION_MAP[self.filetype]
+        if self.packer_template is None:
+            return f"{self.name}.{ext}"
+        return self.packer_target or f"{self.packer_template}.{ext}"
+
+    @property
     def preseed_filename(self) -> str:
         if preseed := self.preseed:
             return "_".join(
diff --git a/systemtests/packer.py b/systemtests/packer.py
index e50314a..e33e962 100644
--- a/systemtests/packer.py
+++ b/systemtests/packer.py
@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import re
 from datetime import timedelta
 from pathlib import Path
 from typing import TYPE_CHECKING, Any
@@ -11,14 +12,26 @@ from systemtests.lxd import _FileWrapper
 if TYPE_CHECKING:
     from systemtests.lxd import CLILXD, Instance
 
+TIMEOUT_REGEX = re.compile(r"(?:(?P<d>\d+) days, )?(?P<h>\d+)\:(?P<m>\d+)\:\d+")
+
 
 class UnknowStorageBackendError(Exception):
     """We don't know how to handle this backend"""
 
 
+def add_packer_header(headers: list[str], config: dict[str, Any]) -> None:
+    if packer_images := config.get("image-tests", {}):
+        headers.append(f"packer images: {', '.join(packer_images.keys())}")
+    if packer_config := config.get("packer-maas", {}):
+        git_repo = packer_config.get(
+            "git-repo", "https://github.com/canonical/packer-maas"
+        )
+        git_branch = packer_config.get("git-branch", "main")
+        headers.append(f"packer: {git_repo}:{git_branch}")
+
+
 class PackerMain(GitBuild):
-    use_timeout = True
-    timeout = timedelta(minutes=15)
+    timeout = timedelta(hours=2)
 
     def __init__(
         self,
@@ -61,6 +74,7 @@ class PackerMain(GitBuild):
         self.apt_install(
             [
                 "cloud-image-utils",
+                "efibootmgr",
                 "fuse2fs",
                 "fusefat",
                 "libnbd-bin",
@@ -68,7 +82,13 @@ class PackerMain(GitBuild):
                 "nbdkit",
                 "ovmf",
                 "python3-pip",
+                "qemu-utils",
+                # boot
+                "qemu-efi",
+                "qemu-efi-aarch64",
+                # architectures
                 "qemu-system-x86",
+                "qemu-system-aarch64",
             ]
         )
         self.logger.info("Installing Packer")
@@ -110,13 +130,29 @@ class PackerMain(GitBuild):
         packer_template: str,
         packer_target: str | None,
         img_filename: str,
+        img_alt_filename: str | None,
         build_series: str | None,
+        build_architecture: str | None,
         preseed_file: str | None,
         source_iso: str | None,
-    ) -> tuple[str | None, str]:
+    ) -> tuple[str | None, str | None, str]:
         env = self._env.copy()
         env["SUDO"] = "sudo -E"
-        log_file = f"build-{packer_template}-{packer_target or 'all'}.log"
+        log_file = (
+            "-".join(
+                filter(
+                    None,
+                    [
+                        "build",
+                        packer_template,
+                        packer_target or "all",
+                        build_architecture,
+                    ],
+                )
+            )
+            + ".log"
+        )
+        env["FOREGROUND"] = "1"
         env["PACKER_LOG"] = "on"
         env["PACKER_LOG_PATH"] = f"{self.clone_path}/{log_file}"
         if source_iso:
@@ -129,9 +165,13 @@ class PackerMain(GitBuild):
             "-C",
             f"{self.clone_path}/{packer_template}",
             f"{packer_target or 'all'}",
+            "PACKER_LOG=1",
         ]
         if build_series:
             cmd += [f"SERIES={build_series}"]
+        if build_architecture:
+            cmd += [f"ARCH={build_architecture}", f"TIMEOUT={self.timeout_units}"]
+
         try:
             runtime = self.timed(
                 self._instance.execute,
@@ -149,11 +189,20 @@ class PackerMain(GitBuild):
 
         # if the image has a build series, the default file may not match
         image_file_path = f"{self.clone_path}/{packer_template}/{img_filename}"
-        return (
-            image_file_path.replace(f"-{build_series}", "")
-            if build_series
-            else image_file_path
-        ), preseed
+        alternate_file_path = f"{self.clone_path}/{packer_template}/{img_alt_filename}"
+
+        # the result does not contain an arch name, but the filename given might
+        if arch := build_architecture:
+            image_file_path = image_file_path.replace(f":{arch}", "")
+            alternate_file_path = alternate_file_path.replace(f":{arch}", "")
+
+        if build_series:
+            return (
+                image_file_path.replace(f"-{build_series}", ""),
+                alternate_file_path.replace(f"-{build_series}", ""),
+                preseed,
+            )
+        return image_file_path, alternate_file_path, preseed
 
     def get_dependencies(
         self,
@@ -173,3 +222,18 @@ class PackerMain(GitBuild):
 
     def __repr__(self) -> str:
         return f"<PackerMain in {self._instance}>"
+
+    @property
+    def timeout_units(self) -> str:
+        if search := TIMEOUT_REGEX.search(str(self.timeout)):
+            return "".join(
+                filter(
+                    None,
+                    [
+                        f"{value}{unit}"
+                        for unit, value in search.groupdict().items()
+                        if value
+                    ],
+                )
+            )
+        raise ValueError(f"{self.timeout} didn't match the expected form?")
diff --git a/systemtests/state.py b/systemtests/state.py
index 92aba9f..313da9d 100644
--- a/systemtests/state.py
+++ b/systemtests/state.py
@@ -65,12 +65,24 @@ def import_images_and_wait_until_synced(
 ) -> None:
     architectures = set()
     osystems: set[tuple[str, str]] = set()
-    for power_config in config.get("machines", {}).get("hardware", {}).values():
+
+    # fetch the configs
+    per_machine_config: dict[str, Any] = config.get("machines", {}).get("hardware", {})
+    all_vm_config: dict[str, Any] = config.get("machines", {})
+    per_vm_config: dict[str, Any] = all_vm_config.get("vms", {}).get("instances", {})
+
+    # determine osystems and arches per machine
+    for power_config in (per_machine_config | per_vm_config | all_vm_config).values():
         architectures.add(power_config.get("architecture", "amd64"))
         if set(("osystem", "oseries")).intersection(power_config.keys()):
-            osystem = power_config.get("osystem", "ubuntu")
-            oseries = power_config.get("oseries", "focal")
-            osystems.add((osystem, oseries))
+            osystems.add(
+                (
+                    power_config.get("osystem", "ubuntu"),
+                    power_config.get("oseries", "focal"),
+                )
+            )
+
+    # determine all osystems per terrafrom and ansible
     osystems.update(
         set(
             ("ubuntu", series)
@@ -82,6 +94,10 @@ def import_images_and_wait_until_synced(
             )
         )
     )
+    LOG.info(
+        f"{len(osystems)} OS Images selected for import: "
+        + ", ".join("-".join(osys) for osys in osystems)
+    )
 
     started_importing_regex = "^Started importing of boot images from"
     # Sometimes the region has already started importing images, we
@@ -138,31 +154,37 @@ def import_images_and_wait_until_synced(
         for name, cfg in config.get("image-tests", {}).items()
     ]
     for img in custom_images:
+        dst = authenticated_admin.api_client.maas_container.files[
+            f"/home/ubuntu/{img.filename}"
+        ]
+        preseed_dst = authenticated_admin.api_client.maas_container.files[
+            f"/home/ubuntu/{img.filename}"
+        ]
+
         scheme, filepath = get_filestore_filepath(file_store_cfg, img.filename)
         if scheme == "file":
-            dst = authenticated_admin.api_client.maas_container.files[
-                f"/home/ubuntu/{img.filename}"
-            ]
             dst.push(filepath, create_dirs=True, uid=1000, gid=1000)
+            LOG.info(f"Uploaded {img}")
         else:
             LOG.error(f"Unsupported file-store scheme '{scheme}'")
 
-        # upload the preseed if needed
         if not img.preseed:
             continue
+
         preseed_scheme, preseed_filepath = get_filestore_filepath(
             file_store_cfg, img.preseed_filename
         )
         if preseed_scheme == "file":
-            dst = authenticated_admin.api_client.maas_container.files[
-                f"/var/snap/maas/current/preseeds/{img.preseed_filename}"
-            ]
-            dst.push(preseed_filepath, create_dirs=True, uid=1000, gid=1000)
+            preseed_dst.push(preseed_filepath, create_dirs=True, uid=1000, gid=1000)
+            LOG.info(f"Uploaded {img} preseed")
         else:
             LOG.error(f"Unsupported file-store scheme '{preseed_scheme}'")
 
     region_start_point = time.time()
-    authenticated_admin.import_and_wait_for_images()
+    # allow up to 40 minutes per image
+    authenticated_admin.import_and_wait_for_images(
+        timeout=40 * 60 * max(1, len(custom_images))
+    )
     authenticated_admin.log_available_boot_resources()
 
     region_time_taken = time.time() - region_start_point
@@ -184,7 +206,7 @@ def import_images_and_wait_until_synced(
         LOG.info(f"Took {windows_time_taken:0.1f}s to upload Windows")
     # testing of custom (packer) images
     for img in custom_images:
-        architectures.add(img.architecture)
+        architectures.add(img.architecture.split("/")[0])
         osystems.add((img.osystem, img.oseries))
         custom_image_start_point = time.time()
         img_params = {
@@ -196,6 +218,7 @@ def import_images_and_wait_until_synced(
         }
         if img.base_image is not None:
             img_params["base_image"] = img.base_image
+
         authenticated_admin.create_boot_resource(**img_params)
         custom_images_time_taken = time.time() - custom_image_start_point
         LOG.info(f"Took {custom_images_time_taken:0.1f}s to upload {img.name}")
@@ -306,14 +329,22 @@ def ready_remote_maas(
             for image in boot_images["images"]
             if image["name"].startswith("bootloader/")
         }
+
         image_arches = {
             image["architecture"]
             for image in boot_images["images"]
             if not image["name"].startswith("bootloader/")
         }
-        assert needed_architectures <= bootloaders_arches
-        assert needed_architectures <= image_arches
-        assert boot_images["status"] == "synced"
+        # Make failure explicit for failure scanning
+        assert needed_architectures <= bootloaders_arches, (
+            "Image architectures missing! "
+            f"{needed_architectures.difference(bootloaders_arches)} "
+            f"not in {bootloaders_arches}"
+        )
+        assert needed_architectures <= image_arches, (
+            "Image architectures missing! "
+            f"{needed_architectures.difference(image_arches)} not in {image_arches}"
+        )
 
 
 # FIXME: scope should be function, and use: @pytest_steps.cross_steps_fixture
diff --git a/systemtests/tests_per_machine/test_hardware_sync.py b/systemtests/tests_per_machine/test_hardware_sync.py
index d2544cc..f5f853c 100644
--- a/systemtests/tests_per_machine/test_hardware_sync.py
+++ b/systemtests/tests_per_machine/test_hardware_sync.py
@@ -6,6 +6,7 @@ from itertools import chain, repeat
 from subprocess import CalledProcessError, CompletedProcess
 from typing import TYPE_CHECKING, Callable, Iterator
 
+import pytest
 from pytest_steps import test_steps
 from pytest_steps.steps_generator import optional_step
 from retry.api import retry, retry_call
@@ -24,6 +25,7 @@ if TYPE_CHECKING:
     from paramiko import PKey
 
     from ..api import AuthenticatedAPIClient, Machine
+    from ..image_config import TestableImage
     from ..lxd import Instance
 
 
@@ -101,12 +103,16 @@ def powered_off_vm(instance: Instance) -> Iterator[None]:
     "remove_device",
     "cleanup",
 )
[e-mail]_if_arch_incompatible(
+    "Machine architecture incompatible with image architecture"
+)
 def test_hardware_sync(
     maas_api_client: AuthenticatedAPIClient,
     hardware_sync_machine: HardwareSyncMachine,
     ready_remote_maas: None,
     ssh_key: PKey,
     testlog: Logger,
+    image_to_test: TestableImage | None,
 ) -> Iterator[None]:
     hardware_sync_machine.instance.logger = testlog
     maas_api_client.set_config("hardware_sync_interval", "5s")
diff --git a/systemtests/tests_per_machine/test_machine.py b/systemtests/tests_per_machine/test_machine.py
index 82a70b6..e272113 100644
--- a/systemtests/tests_per_machine/test_machine.py
+++ b/systemtests/tests_per_machine/test_machine.py
@@ -30,6 +30,9 @@ if TYPE_CHECKING:
 
 
 @pytest.mark.skip_if_os_not_supported("Machine hardware is not supported by OS")
[e-mail]_if_arch_incompatible(
+    "Machine architecture incompatible with image architecture"
+)
 @test_steps("enlist", "metadata", "commission", "deploy", "test_image", "rescue")
 def test_full_circle(
     maas_api_client: AuthenticatedAPIClient,
diff --git a/utils/gen_config.py b/utils/gen_config.py
index 9679636..56bf890 100755
--- a/utils/gen_config.py
+++ b/utils/gen_config.py
@@ -4,6 +4,7 @@
 from __future__ import annotations
 
 import argparse
+import re
 import sys
 from typing import Any, Optional
 
@@ -11,6 +12,9 @@ from ruamel.yaml import YAML
 
 yaml = YAML()
 
+IMAGE_ARCH_REGEX = re.compile(r"^(?P<name_tuple>[^\:]+)(?:\:(?P<arch>\w+))?$")
+IMAGE_SERIES_REGEX = re.compile(r"(?:^(?P<name_part>.*)\-)*(?P<series>[^\-]+)$")
+
 
 class WideHelpFormatter(argparse.HelpFormatter):
     def __init__(
@@ -151,7 +155,14 @@ def main(argv: list[str]) -> int:
         "--image-name",
         action="append",
         metavar="IMAGE_NAME",
-        help="Run tests for this image; can be repeated",
+        help="""
+        Run tests for this image; can be repeated.
+        If supported by the image, image names can contain series and arch information:
+        - base image:           centos7
+        - image with series:    debian-buster
+        - image with arch:      ubuntu:arm64
+        - series and arch:      ubuntu-noble:arm64
+        """,
     )
     parser.add_argument(
         "--image-filestore",
@@ -338,44 +349,91 @@ def main(argv: list[str]) -> int:
         config["image-tests"] = config.get("image-tests", {})
         with args.image_mapping as fh:
             image_cfg: dict[str, Any] = yaml.load(fh)
-        # fetch details from the image mapping cfg file
-        for image in args.image_name:
-            images = image_cfg.get("images", {})
-            series_images = image_cfg.get("images_with_series", {})
-
-            # is this an image with a series argument?
-            if image_search := set([image, image.split("-")[0]]).intersection(
-                series_images.keys()
-            ):
-                image_name = list(image_search)[0]
-                if "-" in image:
-                    packer_name, series = image.rsplit("-", 1)
-                    if series in series_images.get(image_name):
-                        config["image-tests"][image] = image_cfg["images"][packer_name]
-                        config["image-tests"][image]["build_series"] = series
-                        filename = config["image-tests"][image]["filename"].split(
-                            ".", 1
+
+        images = image_cfg.get("images", {})
+        series_images = image_cfg.get("images_with_series", {})
+        arches_images = image_cfg.get("images_with_arches", {})
+        mapping_filename = args.image_mapping.name
+
+        for image_name in args.image_name:
+
+            image, series, arch = None, None, None
+
+            if arch_search := IMAGE_ARCH_REGEX.search(image_name):
+
+                image, arch = arch_search.groups()
+
+                if image not in images and (
+                    series_search := IMAGE_SERIES_REGEX.search(image)
+                ):
+                    image, series = series_search.groups()
+
+                base_image = image.split("-")[0]
+
+                assert (
+                    image in images
+                ), f"{image.capitalize()} isn't defined in {mapping_filename}!"
+
+                this_image_cfg = dict(images[image])
+
+                if series:
+                    assert (
+                        base_image in series_images
+                    ), f"{base_image.capitalize()} doesn't define any series!"
+                    assert (
+                        series in series_images[base_image]
+                    ), f"{base_image.capitalize()} doesn't have a series {series}!"
+
+                    this_image_cfg["build_series"] = series
+
+                    name, extension = this_image_cfg["filename"].split(".", 1)
+                    this_image_cfg["filename"] = f"{name}-{series}.{extension}"
+
+                    this_image_cfg["oseries"] = (
+                        series
+                        if this_image_cfg["osystem"] == "custom"
+                        else f'{this_image_cfg["oseries"]}-{series}'
+                    )
+
+                if arch:
+                    image_key = image if image in arches_images else base_image
+                    assert (
+                        image_key in arches_images
+                    ), f"{image_key.capitalize()} doesn't define any architectures!"
+                    assert (
+                        arch in arches_images[image_key]["arches"]
+                    ), f"{image_key.capitalize()} doesn't have an architecture {arch}!"
+                    if (
+                        series
+                        and (
+                            supported_series := arches_images[image_key].get(
+                                "supported_series"
+                            )
+                        )
+                        and (arch_series_support := supported_series.get(arch))
+                    ):
+                        assert series in arch_series_support, (
+                            f"{image_key.capitalize()} doesn't support the "
+                            f"architecture {arch} on {series}!"
                         )
-                        config["image-tests"][image][
-                            "filename"
-                        ] = f"{filename[0]}-{series}.{filename[1]}"
-                        # append the series if the osystem is custom, else replace
-                        if config["image-tests"][image]["osystem"] == "custom":
-                            config["image-tests"][image]["oseries"] += f"-{series}"
-                        else:
-                            config["image-tests"][image]["oseries"] = series
+
+                    default_arch, subarch = this_image_cfg["architecture"].split("/")
+
+                    # only change stuff if we're not building the default arch
+                    if arch == default_arch:
+                        image_name = image_name.split(":")[0]
+
                     else:
-                        print(f"'{series}' is not a valid series for '{packer_name}'!")
-                else:
-                    print(
-                        f"'{image}' is invalid, {image_name}"
-                        " images require a series argument!"
-                    )
-            # This is a non-series image
-            elif image in images:
-                config["image-tests"][image] = image_cfg["images"][image]
-            else:
-                print(f"Image '{image}' does not have a known mapping!")
+                        this_image_cfg["architecture"] = f"{arch}/{subarch}"
+                        this_image_cfg["build_architecture"] = arch
+
+                        name, extension = this_image_cfg["filename"].split(".", 1)
+                        this_image_cfg["filename"] = f"{name}:{arch}.{extension}"
+
+                    if preseed := this_image_cfg.get("preseed"):
+                        this_image_cfg["preseed"] = preseed.replace(default_arch, arch)
+
+                config["image-tests"][image_name] = this_image_cfg
     else:
         config.pop("image-tests", None)