Failed to power on nodes on SM15k

Kenny,

Do you have the "System ID" saved in the Power section for the Node in the WebUI?

Yes I did. It just wasn't passed into the function for the extract
params. The fix is included at the bottom of the bug report.

Regards,
Kenny
On Nov 13, 2015 1:15 PM, "Newell Jensen" <email address hidden>
wrote:

> Kenny,
>
> Do you have the "System ID" saved in the Power section for the Node in
> the WebUI?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1515769
>
> Title:
> Failed to power on nodes on SM15k
>
> Status in MAAS:
> Triaged
>
> Bug description:
> When trying to commission a node on the SM15000 using MAAS 1.9 beta 2,
> I got error: "Failed to power on node - Node could not be powered on:
> Failed talking to node's BMC for an unknown reason." Further
> investigation show this trace in /var/log/maas/clusterd.log:
>
> 2015-11-11 17:36:29-0600 [ClusterClient,client] drafty-tramp: Power
> could not be turned on.
> Traceback (most recent call last):
> File
> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in
> callback
> self._startRunCallbacks(result)
> File
> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in
> _startRunCallbacks
> self._runCallbacks()
> File
> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in
> _runCallbacks
> current.result = callback(current.result, *args,
> **kw)
> File
> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1155, in
> gotResult
> _inlineCallbacks(r, g, deferred)
> --- <exception caught here> ---
> File
> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in
> _inlineCallbacks
> result = result.throwExceptionIntoGenerator(g)
> File
> "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in
> throwExceptionIntoGenerator
> return g.throw(self.type, self.value, self.tb)
> File
> "/usr/lib/python2.7/dist-packages/provisioningserver/power/change.py", line
> 287, in change_power_state
> system_id, hostname, power_type, power_change,
> context)
> File
> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in
> _inlineCallbacks
> result = result.throwExceptionIntoGenerator(g)
> File
> "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in
> throwExceptionIntoGenerator
> return g.throw(self.type, self.value, self.tb)
> File
> "/usr/lib/python2.7/dist-packages/provisioningserver/drivers/power/__init__.py",
> line 276, in perform_power
> power_func, system_id, **kwargs)
> File
> "/usr/lib/python2.7/dist-packages/twisted/python/threadpool.py", line 191,
> in _worker
> result = context.call(ctx, function, *args, **kwargs)
> File
> "/u...

Kenny,

I understand and see that. However, the context (kwargs are gathered from
the context of the user data that is saved from the power parameters of the
UI). Could you do me a favor and print out the system_id and kwargs that
are passed into the original power_on or power_off functions and list them
here?

Thanks

On Fri, Nov 13, 2015 at 11:23 AM, Kenny Ho <email address hidden> wrote:

> Yes I did. It just wasn't passed into the function for the extract
> params. The fix is included at the bottom of the bug report.
>
> Regards,
> Kenny
> On Nov 13, 2015 1:15 PM, "Newell Jensen" <email address hidden>
> wrote:
>
> > Kenny,
> >
> > Do you have the "System ID" saved in the Power section for the Node in
> > the WebUI?
> >
> > --
> > You received this bug notification because you are subscribed to the bug
> > report.
> > https://bugs.launchpad.net/bugs/1515769
> >
> > Title:
> > Failed to power on nodes on SM15k
> >
> > Status in MAAS:
> > Triaged
> >
> > Bug description:
> > When trying to commission a node on the SM15000 using MAAS 1.9 beta 2,
> > I got error: "Failed to power on node - Node could not be powered on:
> > Failed talking to node's BMC for an unknown reason." Further
> > investigation show this trace in /var/log/maas/clusterd.log:
> >
> > 2015-11-11 17:36:29-0600 [ClusterClient,client] drafty-tramp: Power
> > could not be turned on.
> > Traceback (most recent call last):
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382,
> in
> > callback
> > self._startRunCallbacks(result)
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490,
> in
> > _startRunCallbacks
> > self._runCallbacks()
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577,
> in
> > _runCallbacks
> > current.result = callback(current.result, *args,
> > **kw)
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1155,
> in
> > gotResult
> > _inlineCallbacks(r, g, deferred)
> > --- <exception caught here> ---
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097,
> in
> > _inlineCallbacks
> > result = result.throwExceptionIntoGenerator(g)
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389,
> in
> > throwExceptionIntoGenerator
> > return g.throw(self.type, self.value, self.tb)
> > File
> > "/usr/lib/python2.7/dist-packages/provisioningserver/power/change.py",
> line
> > 287, in change_power_state
> > system_id, hostname, power_type, power_change,
> > context)
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097,
> in
> > _inlineCallbacks
> > result = result.throwExceptionIntoGenerator(g)
> > File
> > "/usr/lib/python2.7/dist-packages/twisted/python/fa...

Sure (I blanked out some IP/mac address/username password, etc.):
2015-11-13 14:13:18-0600 [ClusterClient,client] power_on
2015-11-13 14:13:18-0600 [ClusterClient,client] {u'power_control':
u'restapi2', u'ipmipower': u'/usr/sbin/ipmipower', u'ipmitool':
u'/usr/bin/ipmitool', u'fence_cdu': u'/usr/sbin/fence_cdu', u'power_id':
u'node-*******', u'power_user': u'*******', u'username': u'',
u'ipmi_chassis_config': u'/usr/sbin/ipmi-chassis-config', u'power_address':
u'*********', u'mac_address': u'*******', u'power_driver': u'',
u'boot_mode': u'pxe', u'power_off_mode': u'', u'power_pass': u'*******',
u'ipmi_config': u'ipmi.conf'}
2015-11-13 14:13:29-0600 [ClusterClient,client] power_off
2015-11-13 14:13:29-0600 [ClusterClient,client] {u'power_control':
u'restapi2', u'ipmipower': u'/usr/sbin/ipmipower', u'ipmitool':
u'/usr/bin/ipmitool', u'fence_cdu': u'/usr/sbin/fence_cdu', u'power_id':
u'node-*******', u'power_user': u'*******', u'username': u'',
u'ipmi_chassis_config': u'/usr/sbin/ipmi-chassis-config', u'power_address':
u'*********', u'mac_address': u'*******', u'power_driver': u'',
u'boot_mode': u'pxe', u'power_off_mode': u'hard', u'power_pass':
u'*******', u'ipmi_config': u'ipmi.conf'}

On Fri, Nov 13, 2015 at 2:57 PM, Newell Jensen <email address hidden>
wrote:

> Kenny,
>
> I understand and see that. However, the context (kwargs are gathered from
> the context of the user data that is saved from the power parameters of the
> UI). Could you do me a favor and print out the system_id and kwargs that
> are passed into the original power_on or power_off functions and list them
> here?
>
> Thanks
>
> On Fri, Nov 13, 2015 at 11:23 AM, Kenny Ho <email address hidden> wrote:
>
> > Yes I did. It just wasn't passed into the function for the extract
> > params. The fix is included at the bottom of the bug report.
> >
> > Regards,
> > Kenny
> > On Nov 13, 2015 1:15 PM, "Newell Jensen" <email address hidden>
> > wrote:
> >
> > > Kenny,
> > >
> > > Do you have the "System ID" saved in the Power section for the Node in
> > > the WebUI?
> > >
> > > --
> > > You received this bug notification because you are subscribed to the
> bug
> > > report.
> > > https://bugs.launchpad.net/bugs/1515769
> > >
> > > Title:
> > > Failed to power on nodes on SM15k
> > >
> > > Status in MAAS:
> > > Triaged
> > >
> > > Bug description:
> > > When trying to commission a node on the SM15000 using MAAS 1.9 beta
> 2,
> > > I got error: "Failed to power on node - Node could not be powered on:
> > > Failed talking to node's BMC for an unknown reason." Further
> > > investigation show this trace in /var/log/maas/clusterd.log:
> > >
> > > 2015-11-11 17:36:29-0600 [ClusterClient,client] drafty-tramp: Power
> > > could not be turned on.
> > > Traceback (most recent call last):
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382,
> > in
> > > callback
> > > self._startRunCallbacks(result)
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490,
> > in
> > > _startRunCallbacks
> > > self._runC...

Newell,

The issue is that the power parameters for sm15k include a field named
system_id. Instead of passing all the parameters as a **kwarg just pass it
as a dictionary to all power types. That will prevent the collision.
On Nov 13, 2015 3:05 PM, "Newell Jensen" <email address hidden>
wrote:

> Kenny,
>
> I understand and see that. However, the context (kwargs are gathered from
> the context of the user data that is saved from the power parameters of the
> UI). Could you do me a favor and print out the system_id and kwargs that
> are passed into the original power_on or power_off functions and list them
> here?
>
> Thanks
>
> On Fri, Nov 13, 2015 at 11:23 AM, Kenny Ho <email address hidden> wrote:
>
> > Yes I did. It just wasn't passed into the function for the extract
> > params. The fix is included at the bottom of the bug report.
> >
> > Regards,
> > Kenny
> > On Nov 13, 2015 1:15 PM, "Newell Jensen" <email address hidden>
> > wrote:
> >
> > > Kenny,
> > >
> > > Do you have the "System ID" saved in the Power section for the Node in
> > > the WebUI?
> > >
> > > --
> > > You received this bug notification because you are subscribed to the
> bug
> > > report.
> > > https://bugs.launchpad.net/bugs/1515769
> > >
> > > Title:
> > > Failed to power on nodes on SM15k
> > >
> > > Status in MAAS:
> > > Triaged
> > >
> > > Bug description:
> > > When trying to commission a node on the SM15000 using MAAS 1.9 beta
> 2,
> > > I got error: "Failed to power on node - Node could not be powered on:
> > > Failed talking to node's BMC for an unknown reason." Further
> > > investigation show this trace in /var/log/maas/clusterd.log:
> > >
> > > 2015-11-11 17:36:29-0600 [ClusterClient,client] drafty-tramp: Power
> > > could not be turned on.
> > > Traceback (most recent call last):
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382,
> > in
> > > callback
> > > self._startRunCallbacks(result)
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490,
> > in
> > > _startRunCallbacks
> > > self._runCallbacks()
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577,
> > in
> > > _runCallbacks
> > > current.result = callback(current.result, *args,
> > > **kw)
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line
> 1155,
> > in
> > > gotResult
> > > _inlineCallbacks(r, g, deferred)
> > > --- <exception caught here> ---
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line
> 1097,
> > in
> > > _inlineCallbacks
> > > result = result.throwExceptionIntoGenerator(g)
> > > File
> > > "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389,
> > in
> > > throwExceptionIntoGenerator
> > > return g.throw(self.type, self.value, self.tb)
> > > File
> > > "/usr/lib/python2.7...

Ah, perhaps using (*args, **kwargs) would fix it; is the system_id being passed as a non-named parameter somewhere?

(Just saw Blake's comment; I like that better. Explicit is better than implicit.)

Okay we can either change the way the current implementation is done (as mentioned above) or we can "correct" the power schema that sm15k currently has by changing "System ID" to "Server ID" for the sm15k (which is what it should be). This would need a data migration. I had talked to Blake about changing the implementation, and already have branch for this. However, on further thought and seeing that sm15k is the only hardware that has system_id as one of its power parameters, it might be better to just change this power power parameter. Thoughts from the MAAS team?