OpenStack Compute (nova)

simply_tenant_usage stack traces due to lacking of flavor metadata

Bug #1161022 reported by Rafi Khardalian
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Dan Smith
OpenStack Compute (nova) 2013.1 "grizzly"

Bug Description

During the Grizzly cycle, we made a change such that flavor details are copied for each instance and stored in instance_system_metadata. There's an associated migration (153) for this, which accounts for *non-deleted* instances only. Thus, if querying usage data containing pre-Grizzly, deleted instances, simple_tenant_usage will stack track due to not having the necessary flavor metadata:

2013-03-27 03:41:00.883 ERROR nova.api.openstack [req-854b5c2a-da1e-47c1-bca9-e6a927642d49 af1711fbb8c84c73a1f245679699b419 62afa5ddaa2248d19e2518eb40b0fe8a] Caught error: 'instance_type_memory_
mb'
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack Traceback (most recent call last):
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 81, in __call__
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack return req.get_response(self.application)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack application, catch_exc_info=False)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 451, in __call__
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack return self.app(env, start_response)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack response = self.app(environ, start_response)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 890, in __call__
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack content_type, body, accept)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 942, in _process_stack
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1022, in dispatch
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack return method(req=request, **action_args)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/simple_tenant_usage.py", line 220, in index
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack detailed=detailed)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/simple_tenant_usage.py", line 122, in _tenant_usages_for_period
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack flavor = instance_types.extract_instance_type(instance)
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/instance_types.py", line 247, in extract_instance_type
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack instance_type[key] = type_fn(sys_meta[type_key])
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack KeyError: 'instance_type_memory_mb'
2013-03-27 03:41:00.883 10756 TRACE nova.api.openstack
2013-03-27 03:41:00.902 INFO nova.api.openstack [req-854b5c2a-da1e-47c1-bca9-e6a927642d49 af1711fbb8c84c73a1f245679699b419 62afa5ddaa2248d19e2518eb40b0fe8a]

We've discussed (via #openstack-nova) a possible short term fix for Grizzly RC2, which is to add handling in the code to fall back to the old flavor data retrieval mechanism. Then, as a longer term Havana solution, adding another migration inclusive of deleted data.

Either way we need a fix for Grizzly, otherwise the Horizon "overview" pages will generate exceptions and manifest as errors back to users.

See original description
Rafi Khardalian (rkhardalian)
description: updated
tags: added: grizzly-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/25550

Changed in nova:
assignee: nobody → Dan Smith (danms)
status: New → In Progress
Dan Smith (danms)
Changed in nova:
milestone: none → grizzly-rc2
importance: Undecided → High
Thierry Carrez (ttx)
tags: removed: grizzly-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/25550
Committed: http://github.com/openstack/nova/commit/30631d0077424001e2fa57674e6e9f2a12f0683e
Submitter: Jenkins
Branch: master

commit 30631d0077424001e2fa57674e6e9f2a12f0683e
Author: Dan Smith <email address hidden>
Date: Wed Mar 27 11:30:49 2013 -0700

    Make tenant_usage fall back to instance_type_id

    In order to expose historical usage information about instances that
    were deleted at the time migration #153 was run, we need to fall back
    to looking up type information by instance_type_id.

    This patch does that (only for deleted instances) and replicates the
    previous behavior of skipping instances that have an instance_type_id
    that points to a non-existent type.

    Fixes bug 1161022

    Change-Id: I39d30c59d1711d8f2a61c7a5e2545b3952e55dbe

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (milestone-proposed)

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/25614

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (milestone-proposed)

Reviewed: https://review.openstack.org/25614
Committed: http://github.com/openstack/nova/commit/f29ecbaa21fc51ce2c6c60ac395378cb8436e0c2
Submitter: Jenkins
Branch: milestone-proposed

commit f29ecbaa21fc51ce2c6c60ac395378cb8436e0c2
Author: Dan Smith <email address hidden>
Date: Wed Mar 27 11:30:49 2013 -0700

    Make tenant_usage fall back to instance_type_id

    In order to expose historical usage information about instances that
    were deleted at the time migration #153 was run, we need to fall back
    to looking up type information by instance_type_id.

    This patch does that (only for deleted instances) and replicates the
    previous behavior of skipping instances that have an instance_type_id
    that points to a non-existent type.

    Fixes bug 1161022

    (cherry-picked from 30631d0077424001e2fa57674e6e9f2a12f0683e)

    Change-Id: I39d30c59d1711d8f2a61c7a5e2545b3952e55dbe

Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-rc2 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.