OpenStack + Chef

[all] Can not set rabbit_max_retries and rabbit_retry_interval

Bug #1439968 reported by Wei Hu
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack + Chef
Fix Released
Medium
Ma Wen Cheng
OpenStack + Chef kilo-rc1

Bug Description

When using rabbitmq as the AMQP service, user can not assign rabbit_max_retries and rabbit_retry_interval value. rabbit_max_reties default to 0 which will let oslo.message retry forever when disconnect with rabbitmq-server. Sometimes, this is very dangerous, because those retry and AMQP queue can not connect error log will fill the whole disk.

Tags: all
Wei Hu (huwei-xtu)
Changed in openstack-chef:
assignee: nobody → Wei Hu (huwei-xtu)
Revision history for this message
Wei Hu (huwei-xtu) wrote :

See the options usage in oslo.message code.
https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L106

But I can find those options only in neutron.conf. The glance and nova relate configuration files do not contain those options.
https://github.com/openstack/glance/blob/master/etc/glance-registry.conf#L98

I guess those options should also be included in those configuration files. But not sure.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-common (master)

Fix proposed to branch: master
Review: https://review.openstack.org/171074

Changed in openstack-chef:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-network (master)

Fix proposed to branch: master
Review: https://review.openstack.org/171087

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-common (master)

Reviewed: https://review.openstack.org/171074
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-common/commit/?id=22c46294d6d5623f7a0b2c162c0aef69cb3be760
Submitter: Jenkins
Branch: master

commit 22c46294d6d5623f7a0b2c162c0aef69cb3be760
Author: Wei Hu <email address hidden>
Date: Tue Apr 7 15:41:55 2015 +0800

    Add rabbitmq max_retries and retry_interval attr

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Change-Id: I6bba768f4cf0c61b92fbc5d682b60f094afb7180
    Partial-bug: #1439968

Changed in openstack-chef:
assignee: Wei Hu (huwei-xtu) → Ma Wen Cheng (mars914)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-network (master)

Reviewed: https://review.openstack.org/171087
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-network/commit/?id=0becd087221b63ace01f4d80a465fc303885e64a
Submitter: Jenkins
Branch: master

commit 0becd087221b63ace01f4d80a465fc303885e64a
Author: Wei Hu <email address hidden>
Date: Tue Apr 7 16:50:42 2015 +0800

    Add rabbitmq max_retries and retry_interval in neutron.conf.erb

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Partial-bug: #1439968

    Change-Id: I8e19c3343627d4e2316f91b4fb084c5b6486dd84

Ma Wen Cheng (mars914)
Changed in openstack-chef:
assignee: Ma Wen Cheng (mars914) → nobody
Revision history for this message
Ma Wen Cheng (mars914) wrote : Re: Can not set rabbit_max_retries and rabbit_retry_interval

it is also needed to update all the other components for these two attributes.

tags: added: all
Wei Hu (huwei-xtu)
Changed in openstack-chef:
assignee: nobody → Wei Hu (huwei-xtu)
Mark Vanderwiel (vanderwl)
Changed in openstack-chef:
importance: Undecided → Medium
milestone: none → kilo-rc1
Mark Vanderwiel (vanderwl)
summary: - Can not set rabbit_max_retries and rabbit_retry_interval
+ [all] Can not set rabbit_max_retries and rabbit_retry_interval
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-compute (master)

Fix proposed to branch: master
Review: https://review.openstack.org/182173

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-compute (master)

Reviewed: https://review.openstack.org/182173
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-compute/commit/?id=8d1f4228d27c1ebfbebb20f3c6f9ccca6efb60e7
Submitter: Jenkins
Branch: master

commit 8d1f4228d27c1ebfbebb20f3c6f9ccca6efb60e7
Author: Wei Hu <email address hidden>
Date: Tue May 12 15:05:50 2015 +0800

    Add rabbitmq max_retries and retry_interval in nova.conf.erb

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Change-Id: Ie5ce28de16ed2b4b9c51a8b81e4accaca1466cac
    Partial-bug: #1439968

Revision history for this message
Wei Hu (huwei-xtu) wrote :

Fix nova and neutron. Keystone does not need this option.

Still have glance, ceilometer. cinder components need fix.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-identity (master)

Fix proposed to branch: master
Review: https://review.openstack.org/190948

Changed in openstack-chef:
assignee: Wei Hu (huwei-xtu) → Ma Wen Cheng (mars914)
Revision history for this message
Ma Wen Cheng (mars914) wrote :

Based on the current codes, keystone also needs these two options.

Revision history for this message
Ma Wen Cheng (mars914) wrote :

I will fix the left components instead of Wei Hu to move on this bug.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-image (master)

Fix proposed to branch: master
Review: https://review.openstack.org/190951

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-block-storage (master)

Fix proposed to branch: master
Review: https://review.openstack.org/190955

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-orchestration (master)

Fix proposed to branch: master
Review: https://review.openstack.org/190957

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-telemetry (master)

Fix proposed to branch: master
Review: https://review.openstack.org/190961

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cookbook-openstack-bare-metal (master)

Fix proposed to branch: master
Review: https://review.openstack.org/190975

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-identity (master)

Reviewed: https://review.openstack.org/190948
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-identity/commit/?id=141b2738b443873b0295870f75c9a2d16ab9c0d0
Submitter: Jenkins
Branch: master

commit 141b2738b443873b0295870f75c9a2d16ab9c0d0
Author: wenchma <email address hidden>
Date: Fri Jun 12 14:21:50 2015 +0800

    Add rabbitmq max_retries and retry_interval attributes in keystone conf

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Change-Id: I54236d1d0243e73eb27dd8984cfb86f8bbf7a271
    Closes-bug: #1439968

Changed in openstack-chef:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-orchestration (master)

Reviewed: https://review.openstack.org/190957
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-orchestration/commit/?id=5259486ff4fde915e0f3f866b6d9c44fc98dedeb
Submitter: Jenkins
Branch: master

commit 5259486ff4fde915e0f3f866b6d9c44fc98dedeb
Author: wenchma <email address hidden>
Date: Fri Jun 12 15:01:46 2015 +0800

    Add rabbitmq max_retries and retry_interval in heat conf

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Change-Id: I261d3f31aa89f845d88ee4cbd1acb56c9131ba47
    Closes-bug: #1439968

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-telemetry (master)

Reviewed: https://review.openstack.org/190961
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-telemetry/commit/?id=ff2d6f597dfc46a87b95dc6eeb8cd7c9fdcc7e2c
Submitter: Jenkins
Branch: master

commit ff2d6f597dfc46a87b95dc6eeb8cd7c9fdcc7e2c
Author: wenchma <email address hidden>
Date: Fri Jun 12 15:14:47 2015 +0800

    Add rabbitmq max_retries and retry_interval attributes in ceilometer conf

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Change-Id: I1fd9d4840a55453667eb2ed4c03e4c3cb72442b9
    Closes-bug: #1439968

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-image (master)

Reviewed: https://review.openstack.org/190951
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-image/commit/?id=c46945c8e98244f8c94b5d08dfdfbfb95f5b6c3d
Submitter: Jenkins
Branch: master

commit c46945c8e98244f8c94b5d08dfdfbfb95f5b6c3d
Author: wenchma <email address hidden>
Date: Fri Jun 12 14:35:41 2015 +0800

    Add rabbitmq max_retries and retry_interval attributes in glance conf

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Change-Id: Id205952829678a83b0ccd8e415c64cf1283e0e08
    Closes-bug: #1439968

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cookbook-openstack-block-storage (master)

Reviewed: https://review.openstack.org/190955
Committed: https://git.openstack.org/cgit/stackforge/cookbook-openstack-block-storage/commit/?id=3c0ceb9f4cd47b6030e9399f380a998bb0f89fc7
Submitter: Jenkins
Branch: master

commit 3c0ceb9f4cd47b6030e9399f380a998bb0f89fc7
Author: wenchma <email address hidden>
Date: Fri Jun 12 14:48:03 2015 +0800

    Add rabbitmq max_retries and retry_interval in cinder conf

    Oslo.message using rabbit_max_retries and rabbit_retry_interval to
    define reconnect rabbitmq server times and interval when can not
    connect to rabbitmq server.

    Change-Id: I449f7be68dd9c9adbad587a47cf9d9dcc3462369
    Closes-bug: #1439968

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cookbook-openstack-image ocata-eol

This issue was fixed in the openstack/cookbook-openstack-image ocata-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cookbook-openstack-orchestration ocata-eol

This issue was fixed in the openstack/cookbook-openstack-orchestration ocata-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.