MAAS

Allow 'timeouts' (commissioning, deployment, release, etc) to be user configurable

Bug #1401939 reported by Jason Hobbs
58
This bug affects 9 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
High
Lee Trager
MAAS 2.5.0beta2

Bug Description

Some systems take longer to shutdown than others, but increasing the release timeout for everyone means it takes longer to detect failures for systems that should shutdown quickly.

So, users should be able to adjust the default release time in order to accomodate their systems with longer or shorter shutdown times. This should cascade down to settings like default_waiting_policy - how long we wait between checking if the power is off on a node.

Tags: sprint track

Related branches

~ltrager/maas:config_node_timeout
Andres Rodriguez (community): Approve
MAAS Lander: Approve
Mike Pontillo (community): Approve
Diff: 828 lines (+150/-177)
10 files modified
src/maasserver/api/tests/test_maas.py (+1/-1)
src/maasserver/forms/settings.py (+16/-0)
src/maasserver/models/config.py (+1/-0)
src/maasserver/models/node.py (+21/-72)
src/maasserver/models/tests/test_node.py (+9/-17)
src/maasserver/node_status.py (+17/-5)
src/maasserver/rpc/tests/test_boot.py (+8/-6)
src/maasserver/status_monitor.py (+27/-29)
src/maasserver/tests/test_status_monitor.py (+48/-44)
src/metadataserver/tests/test_api_twisted.py (+2/-3)
Christian Reis (kiko)
Changed in maas:
milestone: none → next
Blake Rouse (blake-rouse)
Changed in maas:
status: New → Triaged
importance: Undecided → Medium
Andres Rodriguez (andreserl)
Changed in maas:
importance: Medium → Wishlist
summary: - Allow 'release timeout' to be tunable by user
+ Allow 'timeouts' (commissioning, deployment, release, etc) to be user
+ configurable
Andres Rodriguez (andreserl)
Changed in maas:
milestone: next → 2.2.0
assignee: nobody → Andres Rodriguez (andreserl)
status: Triaged → In Progress
Revision history for this message
chaithu (chaithanyaprasad) wrote :

http://askubuntu.com/questions/878333/how-can-we-increase-commissioning-deploying-timeout-for-nodes-in-maas

Hi,

My deployment fails because of the timeout(40 mins). So changing the value to 80 mins worked for me.

Refer above link to change the timeout.

BR
Chaithanya

Revision history for this message
Lee Trager (ltrager) wrote :

MAAS 2.2 changes this a bit for commissioning and testing. The current timeout is used until commissioning or testing are started. Once a machine enters either state a heart beat begins. If MAAS no longer detects the heart beat(10 minutes of silence) it assumes the node died and puts it into a failed state. Individual scripts may have a timeout which is also checked.

We could use the heartbeat check for deployment as well if Curtin is noisy enough, or we can add a heart beat. This would allow machines to take however long it needs to successfully deploy.

Andres Rodriguez (andreserl)
Changed in maas:
milestone: 2.2.0 → 2.2.x
Andres Rodriguez (andreserl)
Changed in maas:
status: In Progress → Triaged
milestone: 2.2.x → next
Andres Rodriguez (andreserl)
Changed in maas:
assignee: Andres Rodriguez (andreserl) → Lee Trager (ltrager)
milestone: next → 2.5.0
status: Triaged → In Progress
Dean Henrichsmeyer (dean)
Changed in maas:
importance: Wishlist → High
Andres Rodriguez (andreserl)
tags: added: track
Andres Rodriguez (andreserl)
tags: removed: track
Andres Rodriguez (andreserl)
Changed in maas:
milestone: 2.5.0 → 2.5.0beta2
Andres Rodriguez (andreserl)
tags: added: sprint track
MAAS Lander (maas-lander)
Changed in maas:
status: In Progress → Fix Committed
Andres Rodriguez (andreserl)
Changed in maas:
status: Fix Committed → Fix Released
Revision history for this message
Jochen Wezel (jwezel) wrote :

I'm missing a how-to guide how to customize the timeout values which shall be configurable since https://bugs.launchpad.net/maas/+bug/1401939 was released.

The admin GUI of v2.5.1 doesn't show this setting, so I expect there must be another location where to change these values from e.g. 10 sec. to 25 sec. for the tests in commissioning procedure.

Appropriate location for this documentation could be at
https://docs.maas.io/2.5/en/troubleshoot-faq#nodes-hang-on-commissioning

Revision history for this message
Lee Trager (ltrager) wrote :

This option is not available in the UI. You must use the CLI to change it. For example to set the timeout to 60 minutes you would do

maas $PROFILE maas set-config name=node_timeout value=60

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.