[2.0b7, API/CLI] allows you to add machines with power_type and no power_parameters

Bug #1593991 reported by Larry Michel
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Lee Trager
2.0
Fix Released
Critical
Lee Trager

Bug Description

There's a combination of issues:

1. MAAS allows you to add a machine only specifying power_type, e.g. "ipmi". (The only reason when this should be allowed is when power_type=manual). There's also a error that should be addressed in https://bugs.launchpad.net/maas/+bug/1600328

2. When you add multiple machines only specifying power_type (no power_parameters_<param>, it causes all machines to think they have the same BMC. (See bellow explanation).

How to solve
=============
1. When adding a machine with power_type=manual, and no power_parameters_XYZ it should succeed.
2. When adding a machine with power_type={ipmi,etc}, and no power_parameters_XYZ, it should fail and should show an error that no power parameters have been provided.

======================================================================

I am enlisting servers on maas through the CLI and I ended up with one power controller controlling 37 nodes. I wonder whether this had to do with me executing these commands as a script, because not all of them ended up on the same power controller. I saw another 2 servers that shared their power controller which was separate from the one controlling the 37 nodes.

I deleted one these servers and added it again through the CLI; it came up fine with its own power controller. I am guessing I can work around this by putting a sleep (1 or 2 seconds) between each commands?

maas root machines create architecture=amd64 subarchitecture=generic min_hwe_kernel=hwe-t mac_addresses=2c:59:e5:9a:d3:f4 hostname=reading domain=oilstaging power_type=ipmi
maas root machines create architecture=amd64 subarchitecture=generic min_hwe_kernel=hwe-t mac_addresses=d8:9d:67:6f:56:28 hostname=gulon domain=oilstaging power_type=ipmi
maas root machines create architecture=amd64 subarchitecture=generic min_hwe_kernel=hwe-t mac_addresses=2c:59:e5:41:a8:6c hostname=tucker domain=oilstaging power_type=ipmi

There are no error during the machines create. However, when I try to update the power parameters on the UI, I see this warning message:

This power controller manages 37 other nodes. Changing power parameters will affect these nodes.

Then I see that they all have same values. If update the power parameters for one, it updates for every nodes:

For example:
maas root machine update 4y3hed power_parameters_power_driver=LAN_2_0 power_parameters_power_address=192.168.224.44 power_parameters_power_pass=really? power_parameters_power_user=admin

    "4y3han": {
        "power_address": "192.168.224.44",
        "power_pass": "really?",
        "power_user": "admin",
        "power_driver": "LAN_2_0",
        "mac_address": ""
    },
    "4y3hed": {
        "power_address": "192.168.224.44",
        "power_pass": "really?",
        "power_user": "admin",
        "power_driver": "LAN_2_0",
        "mac_address": ""
    },

The same thing happens when updating from the CLI.

If I delete the server and add through the UI.

ubuntu@maas2-integration:~$ dpkg -l '*maas*'|cat
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===============================-====================================-============-=================================================
ii maas 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all "Metal as a Service" is a physical cloud and IPAM
ii maas-cli 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS client and command-line interface
un maas-cluster-controller <none> <none> (no description available)
ii maas-common 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS server common files
ii maas-dhcp 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS DHCP server
ii maas-dns 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS DNS server
ii maas-proxy 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS Caching Proxy
ii maas-rack-controller 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all Rack Controller for MAAS
ii maas-region-api 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all Region controller API service for MAAS
ii maas-region-controller 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all Region Controller for MAAS
un maas-region-controller-min <none> <none> (no description available)
un python-django-maas <none> <none> (no description available)
un python-maas-client <none> <none> (no description available)
un python-maas-provisioningserver <none> <none> (no description available)
ii python3-django-maas 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS server Django web framework (Python 3)
ii python3-maas-client 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS python API client (Python 3)
ii python3-maas-provisioningserver 2.0.0~beta7+bzr5112-0ubuntu1~xenial1 all MAAS server provisioning libraries (Python 3)
ubuntu@maas2-integration:~$

Tags: oil

Related branches

Revision history for this message
Larry Michel (lmic) wrote :
description: updated
summary: - 2.0 beta7: nodes enlisted manually through CLI added with a single power
+ [2.0b7] nodes enlisted manually through CLI added with a single power
controller controlling all the nodes for power type IPMI
Revision history for this message
Larry Michel (lmic) wrote : Re: [2.0b7] nodes enlisted manually through CLI added with a single power controller controlling all the nodes for power type IPMI

This is very easy to recreate:

https://pastebin.canonical.com/160427/

All the nodes of the same power type end up on a single controller. To workaround I am having to first save them with APU which breaks that bond, then going back to IPMI to edit. Without doing that, it ends up editing a single entry shared amount 70+ or so of the IPMI ones and 10+ of the AMT ones.

Revision history for this message
Larry Michel (lmic) wrote :

Clarification.. one power controller shared by IPMI nodes and one shared controller for AMT nodes.

Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

Here's two nodes that were created without power parameters, you can see they end up with the same BMC:

>>> suangi.bmc
<BMC: 92 (192.168.224.200:type=STICKY)>
>>> prunus.bmc
<BMC: 92 (192.168.224.200:type=STICKY)>
>>> bmc = prunus.bmc
>>> bmc.power_type
'ipmi'
>>> bmc.node_set.count()
47

You can see this bmc has 47 nodes attached to it.

When we go to set the power parameters on the nodes, it affects all the nodes.

Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

To work around this I had to do this from the maas-region-admin shell.

>>> for node in bmc.node_set.all():
... node.bmc = BMC(power_type='ipmi')
... node.bmc.save()
... node.save()

Followed by this:
>>> for node in Node.objects.all():
... if node.power_type is "":
... node.power_type = 'ipmi'
... node.save()

Revision history for this message
Larry Michel (lmic) wrote :

I recreated this from the GUI. Attaching screenshots.. Basically doing the equivalent of the maas CLI commands, specifying the power type and no parameters.

summary: - [2.0b7] nodes enlisted manually through CLI added with a single power
+ [2.0rc1] nodes enlisted manually through CLI added with a single power
controller controlling all the nodes for power type IPMI
description: updated
summary: - [2.0rc1] nodes enlisted manually through CLI added with a single power
- controller controlling all the nodes for power type IPMI
+ [2.0rc1] nodes enlisted manually as IPMI all have the same BMC
Revision history for this message
Jason Hobbs (jason-hobbs) wrote : Re: [2.0rc1] nodes enlisted manually as IPMI all have the same BMC

After setting the BMCs to seperate objects via the admin shell, they were switched back to having the same BMC object at some point, so our work around did not work and we are stuck.

Revision history for this message
Larry Michel (lmic) wrote :

I ran the following commands (see pastebin) for updating the power parameters and they ended up under the shared controller again:

For example:
ubuntu@maas2-production:~/lmic/fix$ maas root machine update 4y3hgt power_parameters_power_driver=LAN_2_0 power_parameters_power_address=192.168.224.45 power_parameters_power_user=admin power_parameters_power_pass=insecure
Success.
Machine-readable output follows:
{
    "address_ttl": null,
...
}

Here are all the commands:

https://pastebin.canonical.com/160588/

Revision history for this message
Larry Michel (lmic) wrote :

Here's screen shot attached.

Note that the warning message says 63 nodes but initially that message had said 64 nodes (I applied the workaround described earlier to 1 node).

Revision history for this message
Larry Michel (lmic) wrote :
Download full text (3.4 KiB)

Per Andres, the power parameters need to be specified at creation time as in example below.

I deleted gytrash entry which had the problem and recreated the node with:

$ maas root machines create architecture=amd64 subarchitecture=generic min_hwe_kernel=hwe-t mac_addresses=2c:59:e5:47:2e:84 domain=oil hostname=gytrash power_parameters_power_driver=LAN_2_0 power_parameters_power_address=192.168.224.45 power_parameters_power_user=admin power_parameters_power_pass=insecure power_type=ipmi
Success.
Machine-readable output follows:
{
    "virtualblockdevice_set": [],
    "owner_data": {},
    "architecture": "amd64/generic",
    "distro_series": "",
    "status": 1,
    "status_name": "Commissioning",
    "storage": 0.0,
    "physicalblockdevice_set": [],
    "cpu_count": 0,
    "fqdn": "gytrash.oil",
    "address_ttl": null,
    "memory": 0,
    "special_filesystems": [],
    "hostname": "gytrash",
    "interface_set": [
        {
            "children": [],
            "discovered": null,
            "enabled": true,
            "id": 1041,
            "parents": [],
            "tags": [],
            "mac_address": "2c:59:e5:47:2e:84",
            "vlan": {
                "external_dhcp": null,
                "secondary_rack": null,
                "dhcp_on": true,
                "id": 5001,
                "vid": 0,
                "mtu": 1500,
                "name": "untagged",
                "resource_uri": "/MAAS/api/2.0/vlans/5001/",
                "primary_rack": "4y3h7n",
                "fabric": "fabric-0"
            },
            "params": "",
            "effective_mtu": 1500,
            "type": "physical",
            "name": "eth0",
            "resource_uri": "/MAAS/api/2.0/nodes/4y3hxd/interfaces/1041/",
            "links": []
        }
    ],
    "status_action": "start commissioning",
    "disable_ipv4": false,
    "owner": "root",
    "node_type": 0,
    "min_hwe_kernel": "hwe-t",
    "power_type": "ipmi",
    "zone": {
        "description": "",
        "name": "default",
        "resource_uri": "/MAAS/api/2.0/zones/default/",
        "id": 1
    },
    "domain": {
        "ttl": null,
        "id": 0,
        "resource_record_count": 24,
        "name": "oil",
        "resource_uri": "/MAAS/api/2.0/domains/0/",
        "authoritative": true
    },
    "status_message": "(root)",
    "osystem": "",
    "node_type_name": "Machine",
    "hwe_kernel": "",
    "system_id": "4y3hxd",
    "ip_addresses": [],
    "tag_names": [],
    "boot_disk": null,
    "swap_size": null,
    "power_state": "unknown",
    "netboot": true,
    "resource_uri": "/MAAS/api/2.0/machines/4y3hxd/",
    "boot_interface": {
        "children": [],
        "discovered": null,
        "enabled": true,
        "id": 1041,
        "parents": [],
        "tags": [],
        "mac_address": "2c:59:e5:47:2e:84",
        "vlan": {
            "external_dhcp": null,
            "secondary_rack": null,
            "dhcp_on": true,
            "id": 5001,
            "vid": 0,
            "mtu": 1500,
            "name": "untagged",
            "resource_uri": "/MAAS/api/2.0/vlans/5001/",
            "primary_rack": "4y3h7n",
            "fabric": "fabric...

Read more...

description: updated
summary: - [2.0rc1] nodes enlisted manually as IPMI all have the same BMC
+ [2.0b7, API/CLI] allow's you to add machines with power_type and no
+ power_parameters
Changed in maas:
importance: Undecided → Medium
status: New → In Progress
assignee: nobody → Jeffrey C Jones (trapnine)
summary: - [2.0b7, API/CLI] allow's you to add machines with power_type and no
+ [2.0b7, API/CLI] allows you to add machines with power_type but no
power_parameters
summary: - [2.0b7, API/CLI] allows you to add machines with power_type but no
+ [2.0b7, API/CLI] allows you to add machines with power_type and no
power_parameters
Changed in maas:
importance: Medium → Critical
Changed in maas:
milestone: 2.1.0 → 2.1.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.