Consumers for re-balanced pipeline queues are not cleaned

Bug #1458917 reported by Yurii Prokulevych
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.messaging
Fix Released
Undecided
gordon chung
Kilo
Fix Committed
Undecided
gordon chung
oslo.messaging (Ubuntu)
Fix Released
High
Unassigned
Vivid
Fix Released
High
James Page

Bug Description

2 instances of openstack-ceilometer-notification.service are running.
Coordination backend - redis

When pipelines are coordinated/re-coordinated they are still listening to the previous queue.

Scenario:

1. event:queu1:source_1and event:queue2:source2 are managed by Agent1
2. Agent2 is added
3. Pipeline are re-coordinated between 2 agents.
    Assuming event:queu1:source_1 is still managed by Agent1 and event:queue2:source2 is managed by Agent2
4. RMQ's UI reports that both Agent1 and Agent 2 are consumer for "ceilometer-pipe-event:queue2:source2.sample"

Channel Consumer tag Ack required Exclusive Prefetch count Arguments
192.168.100.60:43686 (1) 1 ● ○ 0
192.168.100.70:43693 (1) 1 ● ○ 0

At the same time 2 connection for 'not-moved' pipeline are present:

Channel Consumer tag Ack required Exclusive Prefetch count Arguments
192.168.100.60:43692 (1) 1 ● ○ 0
192.168.100.60:45623 (1) 1 ● ○ 0

ceilometer packages:
openstack-ceilometer-notification-2015.1.0-2.el7ost.noarch
openstack-ceilometer-common-2015.1.0-2.el7ost.noarch
python-ceilometerclient-1.0.13-1.el7ost.noarch
openstack-ceilometer-alarm-2015.1.0-2.el7ost.noarch
python-ceilometer-2015.1.0-2.el7ost.noarch
openstack-ceilometer-collector-2015.1.0-2.el7ost.noarch
openstack-ceilometer-central-2015.1.0-2.el7ost.noarch
openstack-ceilometer-api-2015.1.0-2.el7ost.noarch
openstack-ceilometer-compute-2015.1.0-2.el7ost.noarch

Thanks,
Yurii

Revision history for this message
gordon chung (chungg) wrote :

currently debugging but it seems like when .wait() is called on consumer (after stop) it fails.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.messaging (master)

Fix proposed to branch: master
Review: https://review.openstack.org/185693

Changed in oslo.messaging:
assignee: nobody → gordon chung (chungg)
status: New → In Progress
Revision history for this message
gordon chung (chungg) wrote :

ceilometer calls stop() then wait() when killing consumers...

when using 1.8.2: oslo.messaging is throwing "ERROR oslo_messaging._drivers.impl_rabbit [-] Failed to consume message from queue:" (with no error message) when caling wait()

against master: nothing is thrown... but consumers are not killed.

Changed in ceilometer:
status: New → Invalid
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo.messaging (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/185702

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.messaging (master)

Reviewed: https://review.openstack.org/185693
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=a45084126452c7ce2a7da43e09a2654f2ee9ec2d
Submitter: Jenkins
Branch: master

commit a45084126452c7ce2a7da43e09a2654f2ee9ec2d
Author: gordon chung <email address hidden>
Date: Tue May 26 14:36:23 2015 -0400

    consumer connections not closed properly

    heartbeat_thread is not set for listeners. when closing connection,
    it blindly checks heartbeat_thread and will throw an error causing
    connection to remain open. this patch explicitly sets
    heartbeat_thread to None.

    Change-Id: Ief3bf02f952882ecadf742cdd0bac8edd7812473
    Closes-Bug: #1458917

Changed in oslo.messaging:
status: In Progress → Fix Committed
Changed in oslo.messaging:
milestone: none → 1.12.0
status: Fix Committed → Fix Released
gordon chung (chungg)
no longer affects: ceilometer/kilo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.messaging (stable/kilo)

Reviewed: https://review.openstack.org/185702
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=41d0d875a10b48eea89f46ae3c426a023dfae27c
Submitter: Jenkins
Branch: stable/kilo

commit 41d0d875a10b48eea89f46ae3c426a023dfae27c
Author: gordon chung <email address hidden>
Date: Tue May 26 14:36:23 2015 -0400

    consumer connections not closed properly

    heartbeat_thread is not set for listeners. when closing connection,
    it blindly checks heartbeat_thread and will throw an error causing
    connection to remain open. this patch explicitly sets
    heartbeat_thread to None.

    Change-Id: Ief3bf02f952882ecadf742cdd0bac8edd7812473
    Closes-Bug: #1458917

James Page (james-page)
Changed in oslo.messaging (Ubuntu Wily):
status: New → Fix Released
importance: Undecided → High
Changed in oslo.messaging (Ubuntu Vivid):
importance: Undecided → High
status: New → Triaged
assignee: nobody → James Page (james-page)
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Yurii, or anyone else affected,

Accepted oslo.messaging into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/oslo.messaging/1.8.3-0ubuntu0.15.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in oslo.messaging (Ubuntu Vivid):
status: Triaged → Fix Committed
tags: added: verification-needed
James Page (james-page)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package oslo.messaging - 1.8.3-0ubuntu0.15.04.1

---------------
oslo.messaging (1.8.3-0ubuntu0.15.04.1) vivid; urgency=medium

  * New upstream point release (LP: #1467959):
    - RabbitMQ driver:
      + Adding publisher acknowledgements/confirms for better handling
        of messages during broker shutdown/network failure.
      + Ensure consumer connections closed properly (LP: #1458917).
      + Set timeout on the underlying socket (LP: #1436788).
      + Disable and mark heartbeat as experimental (LP: #1436769).
      + Fix ipv6 support.
    - ZeroMQ driver:
      + Don't raise Timeout on no-matchmaker results (LP: #1186310).
      + Fix issue with Redis not deleting expired keys (LP: #1417464).
      + d/p/Fix-changing-keys-during-iteration-in-matchmaker-hea.patch,
        d/p/Add-pluggability-for-matchmakers.patch: Dropped, included
        upstream.

 -- James Page <email address hidden> Tue, 23 Jun 2015 15:28:01 +0100

Changed in oslo.messaging (Ubuntu Vivid):
status: Fix Committed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for oslo.messaging has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Yurii, or anyone else affected,

Accepted oslo.messaging into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/oslo.messaging/1.8.3-0ubuntu0.15.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: removed: verification-done
tags: added: verification-needed
James Page (james-page)
tags: added: verification-done
removed: verification-needed
Mathew Hodson (mhodson)
no longer affects: oslo.messaging (Ubuntu Wily)
affects: ceilometer → ubuntu-translations
no longer affects: ubuntu-translations
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/oslo.messaging 1.8.3

This issue was fixed in the openstack/oslo.messaging 1.8.3 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.