OpenStack Compute (nova)

[SRU] Instance left stuck in transitional POWERING state

Bug #1304333 reported by David McNally on 2014-04-08

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
OpenStack Compute (nova)	Fix Released	Medium	David McNally	OpenStack Compute (nova) 2014.2 "juno"
Icehouse	Fix Released	Undecided	Unassigned	OpenStack Compute (nova) 2014.1.5
nova (Ubuntu)	Fix Released	Undecided	Unassigned
Trusty	Fix Released	Medium	James Page

Bug Description

If a compute manager is stopped / fails during POWERING-ON or POWERING-OFF operations then the instance will be left stuck in a this transitional task_state.

--- --- --- --- --- --- ---

[Impact]

* We are backporting this to Icehouse since nova currently fails to resolve
instance state when service is restarted. It is not expected to impact
normal operational behaviour in any way.

[Test Case]

* Deploy cloud incl. nova-compute and rabbitmq and create some instances.

* Perform actions on those instances that cause state to change

* Restart nova-compute and once restarted check that nova instances are in
expected state.

[Regression Potential]

* None that I can see. This is hopefully a very low impact patch and I have
tested in my local cloud environment with successful results.

See original description

Tags:

Related branches

lp:~ubuntu-server-dev/nova/icehouse

Revision history for this message

John Garbutt (johngarbutt) wrote on 2014-04-08:

https://review.openstack.org/#/c/62038/

tags:	added: compute
Changed in nova:
status:	New → Triaged
importance:	Undecided → Medium
assignee:	nobody → David McNally (dave-mcnally)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-04-09: Fix merged to nova (master)

Reviewed: https://review.openstack.org/62038
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8f7056f48c115ee0939c76ce08531dab0f14e354
Submitter: Jenkins
Branch: master

commit 8f7056f48c115ee0939c76ce08531dab0f14e354
Author: David McNally <email address hidden>
Date: Fri Nov 22 16:18:53 2013 +0000

Recover from POWERING-* state on compute manager start-up

    If a compute manager is stopped / fails during POWERING-ON or
    POWERING-OFF operations then the instance will be left stuck in
    a transitional task_state.

    This change handles both these states by simply either retriggering
    the power on or power off request on start-up of the compute manager.
    If the operation had previously completed then retrying will
    essentially be a no-op so it is safe to handle instances in this
    state in this way.

Closes bug: #1304333
Change-Id: I262be6b13a764bc1355be34cdc0180e8e87f260b

Changed in nova:
status:	Triaged → Fix Committed

Thierry Carrez (ttx) on 2014-06-11

Changed in nova:
milestone:	none → juno-1
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2014-10-16

Changed in nova:
milestone:	juno-1 → 2014.2

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-22: Fix proposed to nova (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/176234

Edward Hope-Morley (hopem) on 2015-04-22

tags:

added: icehouse-backport-potential

Revision history for this message

Edward Hope-Morley (hopem) wrote on 2015-04-23: Re: Instance left stuck in transitional POWERING state

lp1304333-icehouse-debdiff.patch Edit (6.9 KiB, text/plain)

description:

updated

James Page (james-page) on 2015-04-23

Changed in nova (Ubuntu):
status:	New → Fix Released
Changed in nova (Ubuntu Trusty):
importance:	Undecided → Medium
status:	New → Triaged

Revision history for this message

James Page (james-page) wrote on 2015-04-23:

patch applied to icehouse branch and updated package uploaded to trusty-proposed for SRU team review.

Thanks!

summary:	- Instance left stuck in transitional POWERING state + [SRU] Instance left stuck in transitional POWERING state
Changed in nova (Ubuntu Trusty):
status:	Triaged → In Progress
assignee:	nobody → James Page (james-page)

Revision history for this message

Chris J Arges (arges) wrote on 2015-04-29: Please test proposed package

Hello David, or anyone else affected,

Accepted nova into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/nova/1:2014.1.4-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in nova (Ubuntu Trusty):
status:	In Progress → Fix Committed
tags:	added: verification-needed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-30: Fix merged to nova (stable/icehouse)

Reviewed: https://review.openstack.org/176234
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d8853eef181fabebc13bdfdaa98818cfcef318be
Submitter: Jenkins
Branch: stable/icehouse

commit d8853eef181fabebc13bdfdaa98818cfcef318be
Author: David McNally <email address hidden>
Date: Fri Nov 22 16:18:53 2013 +0000

Recover from POWERING-* state on compute manager start-up

    If a compute manager is stopped / fails during POWERING-ON or
    POWERING-OFF operations then the instance will be left stuck in
    a transitional task_state.

Closes bug: #1304333

(cherry picked from commit 8f7056f48c115ee0939c76ce08531dab0f14e354)
Change-Id: I262be6b13a764bc1355be34cdc0180e8e87f260b

tags:

added: in-stable-icehouse

Revision history for this message

Edward Hope-Morley (hopem) wrote on 2015-05-26:

We've been running this a while in multiple Trusty Icehouse deployments so +1 for SRU.

tags:

added: verification-done
removed: verification-needed

Revision history for this message

Chris J Arges (arges) wrote on 2015-05-27: Update Released

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2015-05-27:

#10

This bug was fixed in the package nova - 1:2014.1.4-0ubuntu2.1

---------------
nova (1:2014.1.4-0ubuntu2.1) trusty; urgency=medium

  * Ensure that compute manager restarts during instance power
    operations don't leave instances stuck in transitional task
    states (LP: #1304333):
    - d/p/recover-from-power-state-on-compute.patch
      Cherry pick backport of upstream fix from OpenStack >= Juno.
-- Edward Hope-Morley <email address hidden> Wed, 22 Apr 2015 09:51:28 +0100