Fuel Plugins

Heka service deadlock with idle packs

Bug #1557388 reported by Denis Klepikov
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel Plugins
Fix Released
Undecided
LMA-Toolchain Fuel Plugins
Fuel Plugins 7.0
Nominated for 7.0 by Denis Klepikov
StackLight
Fix Released
High
Swann Croiset
StackLight 0.10.0
0.8
Fix Released
High
Swann Croiset
StackLight 0.8.1
0.9
Fix Released
High
Swann Croiset
StackLight 0.9.0

Bug Description

LMA toolchain plugins v0.8
Fuel 7.0

Heka service went into deadlock under load.

Logs contain a lot of entries like:

Diagnostics: 6 packs have been idle more than 120 seconds.
Diagnostics: 59 packs have been idle more than 120 seconds.

There is only one way to restart it:

kill -9 hekad on all controllers

See original description
Denis Klepikov (dklepikov)
tags: added: lma support
Irina Povolotskaya (ipovolotskaya)
Changed in fuel-plugins:
assignee: nobody → Simon Pasquier (simon-pasquier)
milestone: none → 7.0
Denis Klepikov (dklepikov)
description: updated
Irina Povolotskaya (ipovolotskaya)
tags: added: customer-found
Denis Klepikov (dklepikov)
description: updated
description: updated
Simon Pasquier (simon-pasquier)
Changed in fuel-plugins:
assignee: Simon Pasquier (simon-pasquier) → LMA-Toolchain Fuel Plugins (mos-lma-toolchain)
Simon Pasquier (simon-pasquier)
Changed in lma-toolchain:
milestone: none → 0.8.1
importance: Undecided → High
status: New → Triaged
Swann Croiset (swann-w)
Changed in lma-toolchain:
assignee: nobody → LMA-Toolchain Fuel Plugins (mos-lma-toolchain)
status: Triaged → Fix Committed
Revision history for this message
Swann Croiset (swann-w) wrote :

The Heka buffering is not enabled with 0.8.0 but in 0.8.1. This should fix this issue.

Patrick Petit (patrick-michel-petit)
Changed in fuel-plugins:
status: New → Confirmed
OpenStack Infra (hudson-openstack)
Changed in lma-toolchain:
assignee: LMA-Toolchain Fuel Plugins (mos-lma-toolchain) → Swann Croiset (swann-w)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-plugin-lma-collector (stable/0.8)

Fix proposed to branch: stable/0.8
Review: https://review.openstack.org/301821

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-plugin-lma-collector (master)

Reviewed: https://review.openstack.org/301496
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=96df47af7385578280a68863b55416da7dd8bfdb
Submitter: Jenkins
Branch: master

commit 96df47af7385578280a68863b55416da7dd8bfdb
Author: Swann Croiset <email address hidden>
Date: Tue Apr 5 09:53:24 2016 +0200

    Increase the Heka poolsize on controllers

    On controller nodes, the Heka poolsize must be increased to handle the load
    generated by derived metrics from logs otherwise a deadlock
    can happen in the filter plugins and block heka.

    Fixes-bug: #1557388

    Change-Id: I74362011d32d413f244c6cdb6e4625ed96759df0

Changed in lma-toolchain:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-plugin-lma-collector (stable/0.9)

Fix proposed to branch: stable/0.9
Review: https://review.openstack.org/302100

Simon Pasquier (simon-pasquier)
summary: - Heka service deadlock
+ Heka service deadlock with idle packs
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-plugin-lma-collector (stable/0.9)

Reviewed: https://review.openstack.org/302100
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=1f49e5b58e4ecfbc795bd18f2553948f6b578d2a
Submitter: Jenkins
Branch: stable/0.9

commit 1f49e5b58e4ecfbc795bd18f2553948f6b578d2a
Author: Swann Croiset <email address hidden>
Date: Tue Apr 5 09:53:24 2016 +0200

    Increase the Heka poolsize on controllers

    On controller nodes, the Heka poolsize must be increased to handle the load
    generated by derived metrics from logs otherwise a deadlock
    can happen in the filter plugins and block heka.

    Fixes-bug: #1557388

    Change-Id: I74362011d32d413f244c6cdb6e4625ed96759df0
    (cherry picked from commit 96df47af7385578280a68863b55416da7dd8bfdb)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-plugin-lma-collector (stable/0.8)

Reviewed: https://review.openstack.org/301821
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=346425f04d20526b9e8c23be9360e5456e79c9bb
Submitter: Jenkins
Branch: stable/0.8

commit 346425f04d20526b9e8c23be9360e5456e79c9bb
Author: Swann Croiset <email address hidden>
Date: Tue Apr 5 09:53:24 2016 +0200

    Increase the Heka poolsize on controllers

    On controller nodes, the Heka poolsize must be increased to handle the load
    generated by derived metrics from logs otherwise a deadlock
    can happen in the filter plugins and block heka.

    Fixes-bug: #1557388
    (cherry picked from commit 96df47af7385578280a68863b55416da7dd8bfdb)

    Conflicts:
     deployment_scripts/puppet/manifests/base.pp
     deployment_scripts/puppet/modules/lma_collector/manifests/init.pp

    Change-Id: I74362011d32d413f244c6cdb6e4625ed96759df0

Simon Pasquier (simon-pasquier)
no longer affects: lma-toolchain/1.0
Changed in lma-toolchain:
milestone: 1.0.0 → 0.10.0
Simon Pasquier (simon-pasquier)
Changed in fuel-plugins:
status: Confirmed → Fix Released
Simon Pasquier (simon-pasquier)
Changed in lma-toolchain:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.