2.9.2 enlist/power-on fail cipher suite unabailabe for Lenovo x3650 M5

Hi Chuan Li, thank you for trying MAAS 2.9 and reporting the bug.

As you can see in the release notes - https://maas.io/docs/2.9/release-notes#heading--bmc-param-additions - MAAS 2.9 included several changes around IPMI encryption. In your screenshot you show that Cipher ID 3 is being used, and the error message states that you should check BMC configuration.

What Ciphers does the Lenovo x3650 M5 support? Have you tried different ciphers? Is encryption important to the customer? What happens if you select 'freeipmi-tools default'?

We've seen something similar with drapion, a Lenovo x3650 M5, after upgrading from Ubuntu 18.04 and MAAS 2.6 to Ubuntu 20.04 and MAAS 2.9. As part of this upgrade, we ditched our old MAAS database and re-enlisted all the nodes. We are also unable to access the server's BMC using ipmitool:

$ ipmitool -I lanplus -H 10.245.129.49 -U username -P password chassis status
Error in open session response message : no matching cipher suite

The same ipmitool problem occurs when trying to access the BMC from another node deployed with Ubuntu 18.04 or even 16.04, which had led us to believe that the problem was a failure on the server (which had been fine prior to our MAAS upgrade); however, the same bug on another system after an upgrade to MAAS 2.9 brings that conclusion into doubt. Is it possible that MAAS enlisting the server is changing something in the BMC's configuration to create this problem?

Other things we've tried, without success, include:

* Rebooting the BMC
* Testing with every "Cipher Suite ID" option available in MAAS
* Upgrading the BMC's firmware to the latest version
* Upgrading the computer's main firmware to the latest version
* Enabling NIST SP 800-131A compliance mode in BMC (but this setting
  doesn't seem to have "taken")

The server's web UI remains accessible.

We're seeing the same problem on one other server, an IBM x3650 M2 (wildorange). This server is old enough that we weren't likely to keep it for long, but the newer M5 variant is still worth keeping.

One more thing I've just tried, with no effect:

* Deleting and re-enlisting the server from MAAS, after the BMC
  and firmware updates

I've made progress on a workaround. By setting the MAAS power type to "manual" and using the BMC's web UI to control it, I deployed Ubuntu on the affected node. I then found this, using ipmitool in-band:

# ipmitool lan print 1 | grep Cipher
RMCP+ Cipher Suites : 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16
Cipher Suite Priv Max : XXXaXXXXaXXXaXX

This output differed from another node that had no problems:

# ipmitool lan print 2 | grep Cipher
RMCP+ Cipher Suites : 0,1,2,3,6,7,8,11,12,15,16,17
Cipher Suite Priv Max : XaaaaaaaaaaaXXX

I tried setting the cipher suites on the x3650 to match those on the working system:

# ipmitool lan set 1 cipher_privs XaaaaaaaaaaaXXX

The node's BMC then began working, at least via ipmitool. (We're having unrelated problems with our MAAS server, so I'm not immediately able to test with MAAS.)

This is a rather blind change. I don't know much about IPMI cipher suites, so this configuration on the x3650 might be wrong in some significant way; but it at least works with ipmitool. Unfortunately, I don't know the node's configuration prior to the upgrade of our MAAS server, and documentation seems pretty sparse.

MAAS 2.9 added support for detecting and using the strongest IPMI cipher available. Most IPMI security ciphers have been found to be vulnerable. MAAS prefers 17 as its the strongest, but also supports 3, 8, and 12. 3 is the default ipmitool uses and is what was used before 2.9. MAAS 2.9 also secures your BMC by disabling insecure ciphers and modifying other settings to ensure your BMC is secure.

Please upload the output of the commissioning script "30-maas-01-bmc-config". Be sure to upload all runs, a previous run may contain the log of changing configuration.

MAAS does allow you to override 30-maas-01-bmc-config with a custom commissioning script. To do this create your own commissioning script which will run before 30-maas-01-bmc-config, e.g 00-my-bmc-config. MAAS sets an environment variable BMC_CONFIG_PATH. Your script needs to write BMC configuration data in a YAML format to that path. An easy way to do this is create a copy of 30-maas-01-bmc-config and modify it as you see fit.

The UA customer tried all options in the cipher suite drop-down list, but no luck.
The customer also found different behaviors of 'ipmitool lan print' between before/after adding the Lenovo x3650 M5 into MAAS.

Pre-enlistment:

root@bionic-maas:~# ipmitool -I lanplus -H 100.88.20.2 -U USERID -P PASSW0RD lan print
Set in Progress : Set Complete
Auth Type Support : NONE MD5 PASSWORD
Auth Type Enable : Callback :
                        : User : MD5 PASSWORD
                        : Operator : MD5 PASSWORD
                        : Admin : MD5 PASSWORD
                        : OEM :
IP Address Source : Static Address
IP Address : 100.88.20.2
Subnet Mask : 255.255.255.0
MAC Address : 08:94:ef:48:f0:55
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
Gratituous ARP Intrvl : 2.0 seconds
Default Gateway IP : 100.88.20.254
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16
Cipher Suite Priv Max : aaaaaaaaaaaaaaa
                        : X=Cipher Suite Unused
                        : c=CALLBACK
                        : u=USER
                        : o=OPERATOR
                        : a=ADMIN
                        : O=OEM
Bad Password Threshold : 0
Invalid password disable: no
Attempt Count Reset Int.: 0
User Lockout Interval : 0

Post-enlistment:

root@bionic-maas:~# ipmitool -I lanplus -H 100.88.20.2 -U USERID -P PASSW0RD lan print -vvv

>> Sending IPMI command payload
>> netfn : 0x06
>> command : 0x38
>> data : 0x8e 0x04

BUILDING A v1.5 COMMAND
>> IPMI Request Session Header
>> Authtype : NONE
>> Sequence : 0x00000000
>> Session ID : 0x00000000
>> IPMI Request Message Header
>> Rs Addr : 20
>> NetFn : 06
>> Rs LUN : 0
>> Rq Addr : 81
>> Rq Seq : 00
>> Rq Lun : 0
>> Command : 38
<< IPMI Response Session Header
<< Authtype : NONE
<< Payload type : IPMI (0)
<< Session ID : 0x00000000
<< Sequence : 0x00000000
<< IPMI Msg/Payload Length : 16
<< IPMI Response Message Header
<< Rq Addr : 81
<< NetFn : 07
<< Rq LUN : 0
<< Rs Addr : 20
<< Rq Seq : 00
<< Rs Lun : 0
<< Command : 38
<< Compl Code : 0x00
>> SENDING AN OPEN SESSION REQUEST

<<OPEN SESSION RESPONSE
<< Message tag : 0x00
<< RMCP+ status : no matching cipher suite
<< Maximum privilege level : Unknown (0x00)
<< Console Session ID : 0xa0a2a3a4
Error in open session response message : no matching cipher suite

Error: Unable to establish IPMI v2 / RMCP+ session

Hi Lee Trager,

Thank you for checking this issue. Could you please help to point out where we can find 'output of the commissioning script "30-maas-01-bmc-config"

I am sorry I don't have the environment. Because it's owned by a UA customer.

The customer is asking for a w/a for this issue. Appreciate if someone can help to provide a w/a.

Thank you in advance!

Lee, here's the requested output from drapion, our x3650 M5:

INFO: Loading IPMI kernel modules...
INFO: Checking for HP Moonshot...
INFO: Checking for IPMI...
INFO: IPMI detected!
INFO: Reading current IPMI BMC values...
INFO: Configuring IPMI Lan_Channel...
INFO: Configuring IPMI Lan_Channel_Auth...
INFO: Lan_Channel_Auth settings unavailable!
INFO: Configuring IPMI cipher suite ids...
INFO: MAAS will use IPMI cipher suite id "3" for BMC communication
WARNING: No K_g BMC key found or configured, communication with BMC will not use a session key!
INFO: Configuring IPMI Serial_Channel...
INFO: Serial_Channel settings unavailable!
INFO: Configuring IPMI SOL_Conf...
INFO: Found existing IPMI user "maas"!
INFO: Configuring IPMI BMC user "maas"...
INFO: IPMI user number - User3
INFO: IPMI user privilege level - Administrator
INFO: IPMI Version - LAN_2_0
INFO: IPMI boot type - efi

...and here's the same from wildorange, our x3650 M2:

INFO: Loading IPMI kernel modules...
INFO: Checking for HP Moonshot...
INFO: Checking for IPMI...
INFO: IPMI detected!
INFO: Reading current IPMI BMC values...
INFO: Configuring IPMI Lan_Channel...
INFO: Configuring IPMI Lan_Channel_Auth...
INFO: Lan_Channel_Auth settings unavailable!
INFO: Configuring IPMI cipher suite ids...
INFO: MAAS will use IPMI cipher suite id "3" for BMC communication
WARNING: No K_g BMC key found or configured, communication with BMC will not use a session key!
INFO: Configuring IPMI Serial_Channel...
INFO: Serial_Channel settings unavailable!
INFO: Configuring IPMI SOL_Conf...
INFO: Found existing IPMI user "maas"!
INFO: Configuring IPMI BMC user "maas"...
INFO: IPMI user number - User10
INFO: IPMI user privilege level - Administrator
INFO: IPMI Version - LAN_2_0
INFO: IPMI boot type - efi

Note that both of these required switching from IPMI to manual power control and then powering on via their BMCs' web UIs.

One more addition: I used the workaround described earlier, in #5 above, on wildorange, the older x3650 M2, and it worked, except that I needed to adjust the cipher_privs string to "XaaaXXXXXXXXXXX".

bmc_config.py is incorrect.

It assumes that the index into the string is equal to the cipher suite id.

In reality, the cipher suites that are implemented indicate the mapping from that string to cipher.

In Lenovo's case, we removed cipher suite 0 as it is unacceptably insecure and cannot be hardened:

RMCP+ Cipher Suites : 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16

So when bmc_config.py thinks it's being helpful disabling cipher suite 2, it is disabling cipher suite 3 (and also is force enabling cipher suite 4, an relatively less secure cipher suite).

As an aside, note that cipher suites apart from cipher suite 0 are only particularly insecure if someone explicitly uses them. Having them be passively available doesn't actually constitute a significant risk. They all require knowledge of the password or a crackable password (RAKP-1 offers a challenge first, so offline attack is possible against even the strongest cipher suites) before they can be activated.

I'm incorrect, I see now thtat you are mapping according to the supported_cipher_suite_ids, my guess is incorrect.

merge request opened; it looks like setting the cipher list using ipmitool has never worked; I guess since the default is to set the cipher acls with bmc-config it didn't come up before. Maybe this box is one of those where bmc-config doesn't properly show the ipmi cipher list.

This bug has cropped up on a third of our systems, jolteon, a Lenovo x3550 M5. On this system, I discovered that setting the "Power driver" in MAAS from "LAN_2_0 [IPMI 2.0]" to "LAN [IPMI 1.5]" works around the problem much more easily than the workaround noted in my comments #5 and #10. I have not tested this workaround on the two other affected systems in our (Server Certification's) possession.

Ok I see whats happening. IPMI is very finicky and there are two tools which interact with IPMI BMCs, bmc-config and ipmitool. When detecting IPMI ciphers sometimes bmc-config detects all supported cipher while other times ipmitool detects all ciphers. I discovered this in a bug while a user tested 2.9.0. I coded up a solution where MAAS runs both bmc-config and ipmitool to detect ciphers and then uses whatever tool detected more to make sure ciphers which MAAS support are enabled. This worked for the user I was working with, in the MAAS CI bmc-config always works.

The bug here is revealed in comment #5

RMCP+ Cipher Suites : 0,1,2,3,6,7,8,11,12,15,16,17
Cipher Suite Priv Max : XaaaaaaaaaaaXXX

ipmitool is reporting the ciphers it supports in "RMCP+ Cipher Suites" but skips 4, 5, 9, 10, 13, and 14. Cipher Suite Priv Max represents the status of each cipher. Each character is supposed to map to a cipher suite. As you can see ipmitool is reporting this BMC supports 12 different ciphers but has 15 different settings for ciphers.

To make matters worse ipmitool forces you to specify all cipher settings at once. We can't not disable ciphers because then we couldn't enable them either.

I believe the three extra settings are for vendor ciphers which are not supported by ipmitool. Thus MAAS should be able to ignore them.

So there are two things to fix
1. Cipher ids must be properly mapped to their settings
2. As dannf pointed out in his patch I used a '=' instead of a '+='

I just realized I used the working machine as the example for why things are broken. The reason the broken machine is broken is due to an off by one error

RMCP+ Cipher Suites : 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16
Cipher Suite Priv Max : XXXaXXXXaXXXaXX

Here MAAS thinks cipher 3, 8, and 12 are enabled so it tries to use cipher 3. Because its not accounting for the missing cipher 0 what is actually enabled is cipher 4, 9, and 13. Thus your seeing the access denied error. Even if MAAS never disabled a cipher and supported all 17 ciphers you'd still see the same error due to MAAS not accounting for cipher gaps.

> I believe the three extra settings are for vendor ciphers which are not
> supported by ipmitool.

no, you've misunderstood the ipmitool api to set the cipher privs. please check my patch that fixes it.

@ddstreet - Responded to your branch

Can someone test the attached script? If you upload it into MAAS it will be used instead of the built in script.

> Can someone test the attached script?

Lee, again, I don't think you understand the problem...you are misusing the ipmitool api.

The ipmitool cipher_privs mask is not a mask of the supported cipher numbers. it's a mask of ALL cipher numbers starting from 0. My MR fixes the code to properly create the mask, but you rejected my MR, so I'm not sure what else I can do. Hopefully this gets fixed properly, even if you don't use my patch, but the patch you already landed won't fix it.

I've tested BOTH Lee's and Dan's proposed changed bmc_config.py on jolteon, and I'm afraid that neither works -- both result in the same error message on the MAAS server ("Error:Access denied while performing power action: cipher suite unavailable. Check BMC configuration and try again.") I'm unable to test on drapion or wildorange, since they're controlled by a different MAAS server that's installed via snaps, so I'm unable to change the contents of MAAS on that server.

If it would help, I can perform more tests on jolteon, but I need to know what information/tests you want.

ipmitool is a red herring here. As I mentioned previously mentioned MAAS checks both bmc-config and ipmitool to see which detects the most amount of ciphers. Both detect 16 ciphers but bmc-config detects ciphers 0-15 while ipmitool detects ciphers 1-16. Because both detect the same amount of ciphers MAAS is using bmc-config for all configuration. The bmc-config code path is doing the right thing and makes sure cipher is enabled and available for us. I verified this by manually disabling all ciphers, running the bmc-config script, then checking `bmc-config --checkout -S Rmcpplus_Conf_Privilege` which shows cipher 3 is enabled.

I think some other setting is locking remote commands out.

I think I figured this out. This is a firmware bug. The firmware itself has an off by one error. As I mentioned MAAS uses either bmc-config or ipmitool to detect and configure ciphers. It picks whichever one discovers the highest support cipher. In the case of a tie it uses bmc-config. On Jolteon and Dan's customer case bmc-config detects ciphers 0-15 while ipmitool detects 1-16. Because there is a tie bmc-config is used. What bmc-config says is cipher 0 is actually cipher 1. So when bmc-config enabled cipher 3 it actually enabled cipher 4.

As you can see below bmc-config thinks only cipher 2 is enabled. ipmitool output is unclear but what it means is there is no cipher 0 so Cipher Suite Priv Max[0] is the config value for cipher 1.

I think the patch I landed will fix this but I have to create a follow on patch so ipmitool is always used.

# bmc-config --checkout -S Rmcpplus_Conf_Privilege
#
# Section Rmcpplus_Conf_Privilege Comments
#
# If your system supports IPMI 2.0 and Serial-over-LAN (SOL),cipher suite IDs
# may be configurable below. In the Rmcpplus_Conf_Privilege section, maximum
# user privilege levels allowed for authentication under IPMI 2.0 (including
# Serial-over-LAN) are set for each supported cipher suite ID. Each cipher suite
# ID supports different sets of authentication, integrity, and encryption
# algorithms for IPMI 2.0. Typically, the highest privilege level any username
# configured should set for support under a cipher suite ID. This is typically
# "Administrator".
#
Section Rmcpplus_Conf_Privilege
 ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
 Maximum_Privilege_Cipher_Suite_Id_0 Unused
 ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
 Maximum_Privilege_Cipher_Suite_Id_1 Unused
 ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
 Maximum_Privilege_Cipher_Suite_Id_2 Administrator
 ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
 Maximum_Privilege_Cipher_Suite_Id_3 Unused
 ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
 Maximum_Privilege_Cipher_Suite_Id_4 Unused
 ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
 Maximum_Privilege_Cipher_Suite_Id_5 Unused
 ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
 Maximum_Privilege_Cipher_Suite_Id_6 Unused
 ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
 Maximum_Privilege_Cipher_Suite_Id_7 Unused
 ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
 Maximum_Privilege_Cipher_Suite_Id_8 Unused
 ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
 Maximum_Privilege_Cipher_Suite_Id_9 Unused
 ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
 Maximum_Privilege_Cipher_Suite_Id_10 Unused
 ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
 Maximum_Privilege_Cipher_Suite_Id_11 Unused
 ## Possible values: Unused/User/Operator/Administrator/OEM_Proprietary
 Maximum_Privilege_Cipher_Suite_Id_12 Unu...

I've discovered another machine that appears to be affected by this bug: jehan, a QuantaGrid D52B-1U. The 30-maas-01-bmc-config script runs successfully, but MAAS reports the same cipher suite error as it does on the affected Lenovo servers. This system is relatively new; we received ours in early 2018.

Here is a long explanation of this problem. The TL;DR is, IMHO, disabling IPMI cipher suite privilege levels is extremely dangerous and can potentially disable access to the BMC’s IPMI interface completely, and additionally does not secure the BMC.

Details:

While only the Lenovo BMC engineers know exactly why the BMC behaves the way it does, this may be a result of misinterpretation of the IPMI spec.

The ipmi spec is available at:
https://www.intel.com/content/www/us/en/servers/ipmi/ipmi-technical-resources.html

References are to the latest version “IPMI v2.0, rev. 1.1 markup for Errata 7, April 21, 2015”

The relevant part of the spec to this issue is Table 23-4, LAN Configuration parameters. The important parameters are #23 (“RMCP+ Messaging Cipher Suite Entries”) and #24 (“Messaging Cipher Suite Privilege Levels”). Parameter 23 returns an array of cipher suite IDs that the channel supports, from 0-16 supported cipher suite ids. Parameter 24 allows getting, or setting, the mask of the maximum privilege level allowed for each cipher suite.

The Lenovo BMC engineers may have interpreted the parameter 24 mask as corresponding to the array of cipher suite IDs provided by parameter 23. That’s certainly a fair interpretation, and it makes sense. Unfortunately, that isn’t the interpretation of some other BMC manufacturers, and is not the way the mask is handled by both the ipmitool and freeipmi-tools utilities. Instead, the interpretation is that the mask applies to the cipher suite ID numbers. To be fair to the original BMC implementors, the first IPMI v2 spec only defined 15 cipher suite IDs, so the mask size covered all possible currently-existing cipher suites.

This is most simply demonstrated with the ipmitool ‘lan print’ command, as well as the ‘lan set cipher_privs’ command. The ‘lan set cipher_privs’ command allows setting the mask, and
the ‘lan print’ command prints the output of all lan configuration params, including 23 and 24, e.g.:

RMCP+ Cipher Suites : 1,2,3,6,7,8,11,12
Cipher Suite Priv Max : XXXaXXXXXXXXXXX

In that example, depending on the interpretation of the IPMI spec, either cipher suite 3 or 6 might be enabled for administrator access; if one interprets the mask as indexing the array of supported suites, the mask has cipher suite 6 enabled, while if one interprets the mask as referencing cipher suites by id number, the mask has suite 3 enabled.

This example comes from a Supermicro box, and their interpretation is that the mask shown enables cipher suite 3, not 6.

Unfortunately (or maybe fortunately), some BMC implementors seem to have chosen the simple approach of simply listing all defined cipher suites in order, so there is no difference between the “RMCP+ Cipher Suites” array and the list of defined suites, e.g.:

RMCP+ Cipher Suites : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14

In that case, there is no ambiguity; the spec can be interpreted either way, and the result is the same. This is the case for several systems I tested:

Dell PowerEdge R730xd
Dell PowerEdge T610
RMCP+ Cipher Suites : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14

HP ProLiant SL390s G7
HP ProLiant DL360e Gen8
HP ProLiant DL360 Gen9
HP ProLiant ...

I'll add one very ironic detail of this bug:

The maas bmc script has 2 ways to manage the cipher suites, either with freeipmi-tools, or with ipmitool. The interface to freeipmi-tools hides the implementation detail (described in detail in the last comment), while ipmitool does not.

In the maas bmc script, it almost always uses the freeipmi-tools program to set the cipher suite privileges. That causes this bug since freeipmi-tools takes the spec interpretation that I explained above.

However, the ipmitool program gives the user direct control over setting the privileges mask, so the spec interpretation is left up to the user. In the case of the maas bmc script, it takes the *opposite* interpretation of the spec, and if the maas bmc script avoided using the freeipmi-tools program, these boxes (that break due to this bug) would actually have their cipher suite privs set *correctly*! However, other boxes would break, that interpret the spec differently.

My current understanding is that @ltrager disagrees with me and believes the maas bmc script is doing the right thing.

In any case, my suggestion is to remove all the cipher suite privilege modification code, but this is the maas team's call on exactly how to handle things.

> I've discovered another machine that appears to be affected by this bug: jehan, a QuantaGrid D52B-1U.

That's interesting, as my check of the Quanta S910-X31E showed it should work, maybe there is a difference in BMC behavior between the two Quanta boxes. It's hard to tell for sure without hands-on examination though.

I took another look at this on the machine in question and did some more testing. I do now agree with Dan that ciphers are configured linearly. I have an updated branch[1] which now only configures with ipmitool linearly. What is confusing is some BMC firmware report supported ciphers out of order.

I don't see how MAAS can work around this firmware bug. MAAS can't always test BMC firmware settings as ciphers aren't used with ipmitool locally and many environments don't allow a machine to access the BMC network. Even when the machine does have network access to the BMC network iterating through ciphers can cause the BMC to lock the maas account due to rate limiting. I actually triggered that while testing on some machines that aren't effected by this bug. Even if MAAS works around that a vendor could release a firmware update which fixes the bug and breaks MAAS's access.

The work around is very easy. All you need to do is upload a custom commissioning script, which runs after 30-maas-01-bmc-config, that enables the right cipher. In this case that would be

ipmitool lan set channel 1 cipher_suite_privs XXaXXXXXXXXXXXXX

[1] https://code.launchpad.net/~ltrager/maas/+git/maas/+merge/400287

The attached script enables one cipher at a time and checks which cipher ids can be used to access the BMC. This requires the machine has access to the BMC network and rate limiting is disabled.