Fuel for OpenStack

reboot timeout way too short in fuel 11 stable

Bug #1671793 reported by Luca Cervigni
38
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Stanislaw Bogatkin
Fuel for OpenStack 12.0
Nominated for Ocata by Michael Polenchuk
Newton
Fix Released
Critical
Michael Polenchuk
Fuel for OpenStack 10.1

Bug Description

During the provisioning of servers, the timeout of node_reboot is set to 3 minutes.
This is way too short for bare metal servers. In our environment with UEFI enabled our IBM servers take at least 10 minutes for the fastest, up to 15 minutes for the slowest.

The timeout should be increased to at least 15 minutes from 3 or the provisioning fails with:

All nodes are finished. Failed tasks: Task[node_reboot/10], Task[node_reboot/26], Task[node_reboot/32], Task[node_reboot/34] Stopping the deployment process!

Expected result:
provisioning success

Actual result:
provisioning failed.

Impact: not able to provision servers (very high)

Stanislaw Bogatkin (sbogatkin)
Changed in fuel:
milestone: none → 11.0
assignee: nobody → Stanislaw Bogatkin (sbogatkin)
importance: Undecided → High
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/444826

Changed in fuel:
status: Confirmed → In Progress
Revision history for this message
Sergey Yezhkov (yezhkov) wrote :

Hello, I have the same problem.
I already try to change timeout in graphs/provision/tasks.yaml, but I can`t find how to sync this changes.
For plugins I usualy use "fuel2 plugins sync". But it not work for graph/provision.

Revision history for this message
Sergey Yezhkov (yezhkov) wrote :

I found how to sync changes:
fuel2 graph upload -r {releaseID} -t provision -f /etc/fuel/graphs/provision/tasks.yaml
or, for enviroment:
fuel2 graph upload -e {envID} -t provision -f /etc/fuel/graphs/provision/tasks.yaml

Revision history for this message
Michael Polenchuk (mpolenchuk) wrote :

The same for F10:
https://build.opnfv.org/ci/job/fuel-deploy-baremetal-daily-master/1960/console

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/445962

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/446357

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/444826
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=12e9b93a9d123f56c1500714fd97bdd35cf4bca0
Submitter: Jenkins
Branch: master

commit 12e9b93a9d123f56c1500714fd97bdd35cf4bca0
Author: Stanislaw Bogatkin <email address hidden>
Date: Mon Mar 13 14:13:34 2017 +0300

    Increase reboot task timeout

    To avoid broken provision on baremetal servers, increase reboot
    task timeout to 15 minutes.

    Change-Id: Ia7b9aa96407dae5819420146f5e1e934031b662c
    Closes-Bug: #1671793

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/newton)

Reviewed: https://review.openstack.org/445962
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=74acfc23f3bd69f93903fe257eb6c43bcbc2167d
Submitter: Jenkins
Branch: stable/newton

commit 74acfc23f3bd69f93903fe257eb6c43bcbc2167d
Author: Stanislaw Bogatkin <email address hidden>
Date: Mon Mar 13 14:13:34 2017 +0300

    Increase reboot task timeout

    To avoid broken provision on baremetal servers, increase reboot
    task timeout to 15 minutes.

    Change-Id: Ia7b9aa96407dae5819420146f5e1e934031b662c
    Closes-Bug: #1671793

Michael Polenchuk (mpolenchuk)
Changed in fuel:
milestone: 11.0 → 12.0
Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

It looks that this bug also affects 10.0 community build BVT:
https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/1337/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/ocata)

Reviewed: https://review.openstack.org/446357
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f4eda96d45469a42b3cc2f69466ef24eae0f4008
Submitter: Jenkins
Branch: stable/ocata

commit f4eda96d45469a42b3cc2f69466ef24eae0f4008
Author: Stanislaw Bogatkin <email address hidden>
Date: Mon Mar 13 14:13:34 2017 +0300

    Increase reboot task timeout

    To avoid broken provision on baremetal servers, increase reboot
    task timeout to 15 minutes.

    Change-Id: Ia7b9aa96407dae5819420146f5e1e934031b662c
    Closes-Bug: #1671793

tags: added: in-stable-ocata
Revision history for this message
Kyrylo Romanenko (kromanenko) wrote :

Please check this, it looks it is also subject for this bug: https://ci.fuel-infra.org/job/11.0-community.main.ubuntu.bvt_2/580/

Revision history for this message
Ekaterina Shutova (eshutova) wrote :

Verified for 10.0 #1512.
Latest runs are passed, problem is fixed.
https://ci.fuel-infra.org/job/10.0-community.main.ubuntu.bvt_2/1398/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library ocata-eol

This issue was fixed in the openstack/fuel-library ocata-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.