comparing datetime to None in update-time handling

Bug #1096719 reported by Robert Collins
28
This bug affects 6 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Robert Collins
Folsom
Fix Released
High
Pádraig Brady

Bug Description

2013-01-05 08:02:23 WARNING nova.scheduler.manager [req-c243dee7-fc00-42d8-8654-603b899d471b admin demo] Failed to schedule_run_instance: can't compare datetime.datetime to NoneType
2013-01-05 08:02:23 WARNING nova.scheduler.manager [req-c243dee7-fc00-42d8-8654-603b899d471b admin demo] [instance: 6e4d25bd-9ad8-4e92-b5b4-00fabeed477d] Setting instance to ERROR state.
2013-01-05 08:02:23 ERROR nova.openstack.common.rpc.amqp [req-c243dee7-fc00-42d8-8654-603b899d471b admin demo] Exception during message handling
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 277, in _process_data
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 147, in dispatch
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/manager.py", line 121, in run_instance
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp context, ex, request_spec)
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/manager.py", line 109, in run_instance
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp requested_networks, is_first_time, filter_properties)
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 60, in schedule_run_instance
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp filter_properties, instance_uuids)
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 275, in _schedule
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp hosts = self.host_manager.get_all_host_states(elevated)
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/host_manager.py", line 386, in get_all_host_states
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp host_state.update_from_compute_node(compute)
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/host_manager.py", line 143, in update_from_compute_node
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp if self.updated and self.updated > compute['updated_at']:
2013-01-05 08:02:23 TRACE nova.openstack.common.rpc.amqp TypeError: can't compare datetime.datetime to NoneType

this is from HEAD + pending baremetal patches.

Tags: baremetal
Changed in nova:
assignee: nobody → Robert Collins (lifeless)
status: Triaged → Fix Committed
Revision history for this message
Haiwei Xu (xu-haiwei) wrote :

I met the same question when i used the "nova boot" command.

Revision history for this message
Haiwei Xu (xu-haiwei) wrote :
Download full text (4.4 KiB)

I have set tow nodes: precise64 and slave101, and both of them seems to work properly.

vagrant@precise64:~$ nova-manage service list
Binary Host Zone Status State Updated_At
nova-network slave101 nova enabled :-) 2013-01-08 05:28:24
nova-compute slave101 nova enabled :-) 2013-01-08 05:28:24
nova-conductor precise64 nova enabled :-) 2013-01-08 05:28:22
nova-cert precise64 nova enabled :-) 2013-01-08 05:28:24
nova-compute precise64 nova enabled :-) 2013-01-08 05:28:24
nova-network precise64 nova enabled :-) 2013-01-08 05:28:26
nova-scheduler precise64 nova enabled :-) 2013-01-08 05:28:26
nova-consoleauth precise64 nova enabled :-) 2013-01-08 05:28:24

when i try to create an instance with "nova boot",the error happens:
vagrant@precise64:~$ nova boot --hint force_hosts=slave101 --flavor 1 --image a566bbb7-ef41-4fea-8f60-157d86873b22 image01

2013-01-08 05:29:17 DEBUG nova.scheduler.filter_scheduler [req-dff32077-fe8c-49d9-8784-25be8f692d8a admin demo] Attempting to build 1 instance(s) from (pid=9080) schedule_run_instance /opt/stack/nova/nova/scheduler/filter_scheduler.py:53
2013-01-08 05:29:17 WARNING nova.scheduler.manager [req-dff32077-fe8c-49d9-8784-25be8f692d8a admin demo] Failed to schedule_run_instance: can't compare datetime.datetime to NoneType
2013-01-08 05:29:17 WARNING nova.scheduler.manager [req-dff32077-fe8c-49d9-8784-25be8f692d8a admin demo] [instance: 631f69ff-9827-427f-a5d0-e59359687519] Setting instance to ERROR state.
2013-01-08 05:29:17 ERROR nova.openstack.common.rpc.amqp [req-dff32077-fe8c-49d9-8784-25be8f692d8a admin demo] Exception during message handling
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 277, in _process_data
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 147, in dispatch
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/manager.py", line 121, in run_instance
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp context, ex, request_spec)
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-01-08 05:29:17 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/scheduler/manager.py", line 109, in run_...

Read more...

Revision history for this message
Haiwei Xu (xu-haiwei) wrote :

hi,

i have solved this problem by editing the /etc/nova/nova.conf

I deleted the line "allow_admin_api=True"

Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-2
status: Fix Committed → Fix Released
Revision history for this message
Joshua (josh-ni1sr89) wrote :

I have the exact same error, I did not have "allow_admin_api=True" in my nova.conf. Can someone point me to the code changes for this fix?

Revision history for this message
Richard Knight (richmgb) wrote :
Revision history for this message
Ryan Richard (ryan-richard) wrote :

This bug also appears to affect Folsom with the .1 backport branch.

I also confirmed that the patch appears to fix the issue.

Revision history for this message
Lorin Hochstein (lorinh) wrote :

I hit this in Folsom as well, any chance of a backport to fix this?

Revision history for this message
Robert Collins (lifeless) wrote :

I seem to recall someone backporting this in the last week or so.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/folsom)

Reviewed: https://review.openstack.org/23758
Committed: http://github.com/openstack/nova/commit/5b43cef510b68cff1f6e2f80742d3204b0b51e45
Submitter: Jenkins
Branch: stable/folsom

commit 5b43cef510b68cff1f6e2f80742d3204b0b51e45
Author: Robert Collins <email address hidden>
Date: Mon Jan 7 15:12:39 2013 +1300

    Handle compute node records with no timestamp.

    Commit 94560ab57d9fc23673f42017e6f2a78cb2b66b7a added a fast-path
    to avoid work when baremetal compute nodes haven't changed, but if
    the compute message has None for its updated-at timestamp (which
    can happen during startup) the comparison crashes.

    Fixes bug: 1096719
    Change-Id: Ib8add431d975eee133deefea62eb53479a97396e
    (cherry picked from commit fad69df25ffcea2a44cbf3ef636a68863a2d64d9)

Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-2 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.