[feature] Add ability to select the BIOS boot method for IPMI - To address issues with newer firmware.

The IPMI spec for boot options has two choices - either a system boots from EFI, or from PC Compatible (BIOS). See 'boot flags' in Table 28 from [1]. There is no way via IPMI to say "boot from BIOS/EFI according to how to the system's firmware is configured".

MAAS doesn't set anything, so freeipmi uses the default which is to tell the system to boot from BIOS.

The fix for this is to make MAAS aware of whether systems are BIOS or EFI, and to tell EFI systems to boot from EFI using the BIOS_Boot_Type field Jeff posted in the description.

[1] https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/ipmi-second-gen-interface-spec-v2-rev1-1.pdf

So i think there are different variables here:

1. boot order (e.g. pxe, disk, usb, cdrom, etc)
2. boot method (legacy or efi).

So, with this issue, what we are saying is that when using a tool to manage the BMC remotely, it will *NOT* respect the settings that have been manually set by the administrator in the BIOS itself. For example, if the administrator sets:

1. boot order - disk first, pxe second
2. boot method - efi
3. ipmitool -I lanplus -H <iLO ip> -U admin -P admin123 chassis bootdev pxe

And then the administrator uses ipmitool/freeipmi-tools to tell the machine to PXE boot the next time instead of boot from disk, then it will *NOT* respect the fact that the boot method is EFI, and it will boot in legacy mode instead.

To me, this seems like a critical issue that the firmware *automatically* decides to overwrite the setting in the *bios*, without the user specifically requesting to do so.

I would think this is something that needs to be fixed in the IPMI spec.

The problem isn't in the firmware, it's in the IPMI specification. It doesn't have a way to respect the firmware's boot settings - it requires either EFI or BIOS to be set in the IPMI boot options. The firmware could ignore that field, but that would break compatibility with IPMI and would leave no way to specify via IPMI that the machine should boot in BIOS or UEFI.

The firmware could gain the ability to, if the option is not present, then *respect* what's currently set in the BIOS. That would provide backwards compatibility.

The firmware can't gain that ability. The field is set via a bit in an parameter value that has been there all along in the IPMI spec. Its original reserved value was 0, and the value for BIOS in the spec is now 0. The firmware can't distinguish between '0' being reserved and not set purposefully, and '0' being set to indicate BIOS boot.

@Jeff,

Can you gather the output of for the machine with the newer firmware:

ipmi-chassis-config -D LAN_2_0 -u XX -p XX -h XX --checkout -e Chassis_Boot_Flags:BIOS_Boot_Type

Certainly.

$ ipmi-chassis-config -D LAN_2_0 -u xxxx-p xxxx-h x.x.x.x--checkout -e Chassis_Boot_Flags:BIOS_Boot_Type
Section Chassis_Boot_Flags
        ## Possible values: PC-COMPATIBLE/EFI
        BIOS_Boot_Type PC-COMPATIBLE
EndSection

Sucks. It reports the system is set to Legacy even though it is actually set to EFI.

That is basically what Narinder found. That the default posture is to PXE boot Legacy unless explicitly told otherwise. This will likely have broader ramifications ultimately.

The value reported in that IPMI command is not what the firmware is configured to do; it is just what was written to it by previous IPMI commands. The IPMI spec calls this "semi-volatile".

Also note that boot flags get reset after ~60 seconds if the Persistent option is not set, which MAAS does not set.

for Lenovo there's an option to force UEFI, and ignore legacy.
eg. ./OneCli config set BootModes.IgnoreLegacyFlagOnIpmi enabled --override --imm USERID:PASSW0RD@<ipaddress>

I am gonna ask that this be changed back to a bug like I originally had it. This is not a feature request.

I say this because I have seen it in 2 other environments now.

to put it another way, since this was discovered, I have had to modify /usr/lib/python3/dist-packages/provisioningserver/drivers/power/ipmi.py 100% of the time.

Hi Jeff,

This is not a bug in MAAS provided that it is not MAAS that has changed. It
is the firmware that has changed and requiring new behavior which requires
MAAS to add a new capability (feature) that was not previously supported.

On Mon, Mar 19, 2018 at 7:27 AM Jeff Hillman <email address hidden>
wrote:

> I am gonna ask that this be changed back to a bug like I originally had
> it. This is not a feature request.
>
> I say this because I have seen it in 2 other environments now.
>
>
> to put it another way, since this was discovered, I have had to modify
> /usr/lib/python3/dist-packages/provisioningserver/drivers/power/ipmi.py
> 100% of the time.
>
> --
> You received this bug notification because you are subscribed to MAAS.
> https://bugs.launchpad.net/bugs/1750622
>
> Title:
> [feature] Add ability to select the BIOS boot method for IPMI - To
> address issues with newer firmware.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1750622/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=maas; status=Triaged; importance=Undecided;
> assignee=None;
> Launchpad-Bug-Tags: cpe-onsite feature internal wishlist
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: andreserl gabor.meszaros jason-hobbs jhillman
> Launchpad-Bug-Reporter: Jeff Hillman (jhillman)
> Launchpad-Bug-Modifier: Jeff Hillman (jhillman)
> Launchpad-Message-Rationale: Subscriber (MAAS)
> Launchpad-Message-For: andreserl
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

I have installed maas 2.4.0-6981 on ubuntu 18.04 and the issue still persist. The flag is not in the file. Isn't the fix released from the beta?

Applying manually the patch fixes the issue but it would be better if fixed in the packages.
Could you please have a look?

Here the MAAS version

ii maas 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS client and command-line interface
ii maas-common 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server common files
ii maas-dhcp 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS DHCP server
ii maas-proxy 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS Caching Proxy
ii maas-rack-controller 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Rack Controller for MAAS
ii maas-region-api 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Region controller API service for MAAS
ii maas-region-controller 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Region Controller for MAAS
ii python3-django-maas 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server provisioning libraries (Python 3)

Hi Luca,

The fix is available in the release. I’ve double verified it and we also
have this under CI.

That said, don’t understand your statement of “the flag is not in the
file”. Can you clarify?

On Mon, Jul 30, 2018 at 11:30 PM Luca Cervigni <email address hidden>
wrote:

> I have installed maas 2.4.0-6981 on ubuntu 18.04 and the issue still
> persist. The flag is not in the file. Isn't the fix released from the
> beta?
>
>
> Applying manually the patch fixes the issue but it would be better if
> fixed in the packages.
> Could you please have a look?
>
> Here the MAAS version
>
> ii maas
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all "Metal as a Service" is
> a physical cloud and IPAM
> ii maas-cli
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS client and
> command-line interface
> ii maas-common
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server common
> files
> ii maas-dhcp
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS DHCP server
> ii maas-proxy
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS Caching Proxy
> ii maas-rack-controller
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Rack Controller for MAAS
> ii maas-region-api
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Region controller API
> service for MAAS
> ii maas-region-controller
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all Region Controller for
> MAAS
> ii python3-django-maas
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server Django web
> framework (Python 3)
> ii python3-maas-client
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS python API client
> (Python 3)
> ii python3-maas-provisioningserver
> 2.4.0-6981-g011e51b7a-0ubuntu1~18.04.1 all MAAS server
> provisioning libraries (Python 3)
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1750622
>
> Title:
> [feature] Add ability to select the BIOS boot method for IPMI - To
> address issues with newer firmware.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1750622/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=maas; milestone=2.4.0beta1; status=Fix Released;
> importance=Medium; <email address hidden>;
> Launchpad-Bug: product=maas; productseries=2.3; milestone=2.3.2;
> status=Fix Released; importance=Undecided; assignee=None;
> Launchpad-Bug-Tags: 4010 cpe-onsite feature internal wishlist
> Launchpad-Bug-Information-Type: Public
> Launchpad-Bug-Private: no
> Launchpad-Bug-Security-Vulnerability: no
> Launchpad-Bug-Commenters: andreserl cervigni gabor.meszaros jason-hobbs
> jhillman
> Launchpad-Bug-Reporter: Jeff Hillman (jhillman)
> Launchpad-Bug-Modifier: Luca Cervigni (cervigni)
> Launchpad-Message-Rationale: Assignee
> Launchpad-Message-For: andreserl
>
--
Andres Rodriguez (RoAkSoAx)
Ubuntu Server Developer
MSc. Telecom & Networking
Systems Engineer

Yes sorry. Around line 35:

Original file:
https://paste.ubuntu.com/p/Bpzh2TxrY2/

File with patch applied ( line 38):
https://paste.ubuntu.com/p/NJPTQvC3ZQ/

in the original file the line in the
   BIOS_Boot_Type EFI
is not present.

I added it manually and the problem is fixed.

Hi Luca,

What you are describing is not the solution its available on the product. See lines 42-47 on the original file above:

IPMI_CONFIG_WITH_BOOT_TYPE = """\
Section Chassis_Boot_Flags
        Boot_Flags_Persistent No
        BIOS_Boot_Type %s
        Boot_Device PXE
EndSection
"""

The section:
        BIOS_Boot_Type %s

Is used when either forcing auto/legacy/EFI. That is an option on the configuration of the power settings for the machine. YOu can select between:

'auto' will let the BMC do its own thing, and this is done for backwards compatibility with older BMC versions.
'legacy' - self explanatory.
'efi' - self explanatory.