Masakari will fail to recovery instances on failure host if instance's task_state is not none

Bug #1721742 reported by takahara.kengo
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
masakari
Fix Released
Undecided
takahara.kengo

Bug Description

In Openstack version Ocata, evacuate API fails if instance's task_state is not none.

$ nova evacuate 2d2a8f35-7a8b-4201-ac0e-6ef12befcc39
ERROR (Conflict): Cannot 'evacuate' instance 2d2a8f35-7a8b-4201-ac0e-6ef12befcc39 while it is in task_state powering-off (HTTP 409) (Request-ID: req-0b694705-c75d-4a8e-9f2f-537057186592)
$

So masakari-engine will fail evacuation when it recovers instances which task_state is not none.

When a node failure occurs, masakari-engine should recover instances which vm_state/task_state is active/powering-off.
I think that we should add the following logic to masakari-engine.
1. If vm_state is active, error or stopped, get instance's task_state.
2. If task_state is none, execute evacuate api.
3. If task_state is not none, execute reset-state api to change task_state to none, and execute evacuate.
4. For an instance which is originally stopped and has been executed reset-state api, execute stop api after evacuate.

summary: - Evacuate API fails when instance's task_state is not none
+ Masakari will fail to recovery instances on failure host if instance's
+ task_state is not none
Changed in masakari:
assignee: nobody → takahara.kengo (takahara.kengo)
Revision history for this message
takahara.kengo (takahara.kengo) wrote :

Additional infomation for my proposal.
Reset-state api changes vm_state to error (or active by option) and task_state to none.
We can see this in the source code below.
https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/admin_actions.py#L72

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to masakari (master)

Reviewed: https://review.openstack.org/515013
Committed: https://git.openstack.org/cgit/openstack/masakari/commit/?id=d621267402cf85eb607e0a0f328958ca9c9c3754
Submitter: Zuul
Branch: master

commit d621267402cf85eb607e0a0f328958ca9c9c3754
Author: Kengo Takahara <email address hidden>
Date: Wed Oct 25 19:13:51 2017 +0900

    Evacuates instances which task_state is not none

    This patch added implementation so that masakari can evacuate
    instances which task_state is not none.
    After the instance evacuated, it is recovered with original
    vm_state. So if the instance's vm_state was 'stopped', it is
    recovered with 'stopped', and if 'error', it is recovered
    with 'error'.

    Change-Id: I7af8552de0ee77b948a071b7f787514a81ccebc3
    Closes-Bug: #1721742

Changed in masakari:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.