Merge lp:~rvb/maas/tweak-timeouts into lp:~maas-committers/maas/trunk
Proposed by
Raphaël Badin
Status: | Merged |
---|---|
Approved by: | Raphaël Badin |
Approved revision: | no longer in the source branch. |
Merged at revision: | 4072 |
Proposed branch: | lp:~rvb/maas/tweak-timeouts |
Merge into: | lp:~maas-committers/maas/trunk |
Diff against target: |
193 lines (+44/-10) 6 files modified
src/maasserver/api/nodes.py (+2/-3) src/maasserver/plugin.py (+14/-0) src/maasserver/tests/test_plugin.py (+14/-2) src/maasserver/websockets/handlers/node.py (+3/-4) src/provisioningserver/drivers/power/__init__.py (+9/-0) src/provisioningserver/rpc/power.py (+2/-1) |
To merge this branch: | bzr merge lp:~rvb/maas/tweak-timeouts |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Gavin Panella (community) | Approve | ||
Review via email: mp+264018@code.launchpad.net |
Commit message
Set the maximum thread pool size to 100: this is meant as a workaround for deadlocks such as the one described in the linked bug. It's not meant to be a permanent solution but rather to buy us time to come up with a proper solution (like an "intelligent thread pool manager). Unify the power query timeout: use a unique timeout across the board and use a value that can accommodate seamicro's tendency to take 30s to reply to a power query.
To post a comment you must log in.
Remember last time we increased this we hit database connection issues. I
don't know if increasing the number of threads is a good idea. That is why
we lowered it back down.
Does Gavin's branch not fix the deadlock issue? Is there another deadlock
that needs this increase?
On Jul 7, 2015 6:27 AM, "Raphaël Badin" <email address hidden> wrote:
> Raphaël Badin has proposed merging lp:~rvb/maas/tweak-timeouts into /bugs.launchpad .net/maas/ +bug/1470013 /code.launchpad .net/~rvb/ maas/tweak- timeouts/ +merge/ 264018 /api/nodes. py' api/nodes. py 2015-06-25 10:26:40 +0000 api/nodes. py 2015-07-07 10:22:52 +0000 utils.orm import get_first ver.drivers. power import POWER_QUERY_TIMEOUT ver.power. poweraction import ( power_info. power_type, power_info. power_parameter s) POWER_QUERY_ TIMEOUT) TimeoutError: /plugin. py' plugin. py 2015-05-07 18:14:38 +0000 plugin. py 2015-07-07 10:22:52 +0000 g(description) : IServiceMaker, IPlugin)
> lp:maas.
>
> Commit message:
> Set the maximum thread pool size to 100: this is meant as a workaround for
> deadlocks such as the one described in the linked bug. It's not meant to be
> a permanent solution but rather to buy us time to come up with a proper
> solution (like an "intelligent thread pool manager). Unify the power query
> timeout: use a unique timeout across the board and use a value that can
> accommodate seamicro's tendency to take 30s to reply to a power query.
>
> Requested reviews:
> MAAS Maintainers (maas-maintainers)
> Related bugs:
> Bug #1470013 in MAAS: "Failed power off/status for multiple nodes within
> a SM15K chassis"
> https:/
>
> For more details, see:
> https:/
> --
> Your team MAAS Maintainers is requested to review the proposed merge of
> lp:~rvb/maas/tweak-timeouts into lp:maas.
>
> === modified file 'src/maasserver
> --- src/maasserver/
> +++ src/maasserver/
> @@ -76,6 +76,7 @@
> from maasserver.utils import find_nodegroup
> from maasserver.
> from piston.utils import rc
> +from provisioningser
> from provisioningser
> PowerActionFail,
> UnknownPowerType,
> @@ -690,9 +691,7 @@
> power_type=
> context=
> try:
> - # Allow 30 seconds for the power query max as we're holding
> - # up an appserver thread here.
> - state = call.wait(30)
> + state = call.wait(
> except crochet.
> maaslog.error(
> "%s: Timed out waiting for power response in
> Node.power_state",
>
> === modified file 'src/maasserver
> --- src/maasserver/
> +++ src/maasserver/
> @@ -24,6 +24,7 @@
> from twisted.plugin import IPlugin
> from twisted.python import usage
> from zope.interface import implementer
> +from twisted.internet import reactor
>
>
> def serverFromStrin
> @@ -45,6 +46,14 @@
> ]
>
>
> +# The maximum number of threads used by the default twisted thread pool.
> +# This value is a trade-off between a small value (such as the default:
> 10)
> +# which can create deadlocks (see 1470013) and a huge value which can
> cause
> +# MAAS to hit other limitations such as the number of open files or the
> +# number of concurrent database connexions.
> +MAX_THREADS = 100
> +
> +
> @implementer(
> class...