MAAS deploys fail if host has NIC w/ random MAC

It seems Ampere Mt. Jade Platform is also impacted by this issue.

Its dmidecode info (seems not provide useful information very much, unfortunately):

ubuntu@howzit:~$ sudo dmidecode -t bios -t 42; sudo dmidecode -t 42 -u
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.3.0 present.
# SMBIOS implementations newer than version 3.2.0 are not
# fully supported by this version of dmidecode.

Handle 0x0000, DMI type 0, 26 bytes
BIOS Information
        Vendor: Ampere(R)
        Version: 1.6.20210526 (SCP: 1.06.20210526)
        Release Date: 2021/05/26
        ROM Size: 7680 kB
        Characteristics:
                PCI is supported
                BIOS is upgradeable
                Boot from CD is supported
                Selectable boot is supported
                ACPI is supported
                UEFI is supported
        BIOS Revision: 5.15
        Firmware Revision: 1.6

Handle 0x0029, DMI type 13, 22 bytes
BIOS Language Information
        Language Description Format: Long
        Installable Languages: 1
                en|US|iso8859-1
        Currently Installed Language: en|US|iso8859-1

Handle 0x0055, DMI type 42, 17 bytes
Management Controller Host Interface
        Host Interface Type: OEM

# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.3.0 present.
# SMBIOS implementations newer than version 3.2.0 are not
# fully supported by this version of dmidecode.

Handle 0x0055, DMI type 42, 17 bytes
        Header and Data:
                2A 11 55 00 F0 04 FF 00 00 00 01 02 04 FF FF FF
                FF

From cloud-init's perspective, this is working as expected. If we're presented a network device through the metadata and then can't find it on the system, we intentionally complain about it.

We can of course add logic to ignore or make special cases for specific devices, but until we decide the best course of action, I'm going to mark this as incomplete for cloud-init.

Generated MAC addresses should have the U/L bit set (2nd-least-significant bit of the first octet):

x2‑xx‑xx‑xx‑xx‑xx
x6‑xx‑xx‑xx‑xx‑xx
xA‑xx‑xx‑xx‑xx‑xx
xE‑xx‑xx‑xx‑xx‑xx

MAAS could filter these devices when not configured.

Related to: https://bugs.launchpad.net/maas/+bug/1931735

Marking as invalid because this sounds like a feature request for MAAS, not a bug, e.g, MAAS is working as designed. You can (a) file a feature request in the "Features" category, or (b) help me understand why this is should be considered an actual error in the MAAS code.

Thanks Bill. I don't happen to know the criteria MAAS uses to put things into bug vs. feature request buckets, so I don't feel like I have the rules for making the (b) argument either way. What we know is that MAAS (and/or its dependencies) assumes that all NICs have static MAC addresses. We're now seeing that assumption be proven incorrect, and it's causing MAAS to not support industry standard servers from large HW vendors that otherwise work fine with Ubuntu.

To be clear, I'm not asking that MAAS learn to model these devices. Maybe there's a reason it could/should, but that certainly seems like a new feature. Rather, I'm asking that MAAS not fail deployments due to their presence. Ignoring these easily-detected (Comment #3) devices during commissioning seems like a reasonable fix IMO.

I attempted to fix this by ignoring NICs with the U/L bit set:
  https://code.launchpad.net/~dannf/maas/+git/maas/+merge/409001
As noted there, QEMU uses a prefix with this bit set, and that maybe true with other virtualization, so I don't think we can key on MAC. I think instead we may want to specifically try and detect Redfish NICs and ignore them.

Assuming it is a reasonable solution to hide NICs that can be determined to be RedFish NICs, where is the appropriate place to do this? Should the machine-resources binary be expanded to provide the necessary information, e.g. via a new record?

"resources": {
  "management_controller": [
    {
      "type": "redfish",
      "interface_type": "network_host_interface",
      "device_type": "usb_network_interface",
      "usb_network_interface": {
        "vendor_id": "1234",
        "product_id": "5678",
        "serial_number_type": "string",
        "serial_number": "abc123",
      [...]

Presumably then, update_interfaces() could introspect the data, figure out which interface it maps to, and omit it from the model.

I don't think this is a feature request. Ignoring the NIC in MAAS, might be reasonable. Although it's odd that the NIC doesn't have a MAC of its own. Is that a hardware feature, or is it the driver that doesn't surface the physical MAC?

Also, could you please provide the current output from the machine-resources resources binary for that machine?

I don't know for sure if the absent MAC is by design - you're right, it could be missing support in the driver. I'll see if I can find someone who knows the firmware's intent.

machine-resources output attached.

I agree that in MAAS we should skip devices without a MAC.

On Fri, Oct 22, 2021 at 3:35 AM Alberto Donato
<email address hidden> wrote:
>
> I agree that in MAAS we should skip devices without a MAC.

I'm not sure if there is a reliable way to detect a NIC without a MAC.
If the driver doesn't find a "burned in" MAC, it will generate a
random MAC for the device on every boot. The only thing we'll know for
sure about the random MAC is that it will have the "local assignment
bit" set. But we can't skip a device just because it has that bit set
in the MAC, because that bit is also set in MACs used by QEMU
instances, and likely other forms of virtualization. So my best
suggestion is to tackle the specific subset of devices with random
MACs that are causing real problems today - the RedFish NICs, which
can be easily identified from the SMBIOS table.

 -dann

Workarounds exist, no obvious way to deal with random MAC addresses in a general way.

For 3.3, we should investigate dropping the use of match/set-name for netplan config, since interface names are now stable. In that case, we wouldn't need to create a config stanza for interfaces that are unconfigured.

Ignoring NICs that the firmware tells you are RedFish controllers (and therefore not wired up to any routable network) seems like a reasonable way to tackle the known/biggest source of this project.

Regarding stable interface names, I don't think we've reached that panacea quite yet. Predictable names can change based on OS series - and even in HWE kernels within a single series. See bugs 1940860 and 1945225 for examples.

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3892

I've run into a similar problem: If a node was commissioned with a kernel that detects a particular network device, but the system is then deployed with a kernel that does not detect that device, the deployment fails with the same message as noted in the original bug report. I've encountered this most recently with a Redfish device on blubi (an Ericsson AB CRU 0201 server), but ISTR running into something similar with a "real" NIC in the past. I understand curtin complaining about this, but when the result is that the post-install reboot, and therefore the entire deployment, fails, it can be unclear to the user why the deployment failed -- MAAS logs show a successful installation followed by no output from the node. I had to use the system's remote KVM and watch the entire deployment to figure it out. I worked around it by deleting the Redfish device, which isn't important for us or MAAS; but if it were a real device, that solution would leave the device unconfigured, even when the node was deployed using a kernel that could use it. A more robust solution would enable the deployment to succeed, even if the device was inactive.

Is there any plan on creating a work around or fix for this issue?