[SRU] Failure to retry update_ha_routers_states

I've seen the same issue.

When I was looking at this issue I put LOG.debug("...") at line https://github.com/openstack/neutron/blob/master/neutron/agent/l3/keepalived_state_change.py#L106 and master has appeared as expected. I think this was coincidence and also I'm using just virtual env.

I'm not sure what will be the right solution here: check if we have all standby on server and notify agent to check whether one of them is the master?

Can you verify the version of keepalived being used? I believe the version was important for this feature.

Jesse - Looks like Keepalived v1.2.7 (08/14,2013)

I've saw this with v1.2.13. I don't think that version of Keepalived is related here as the issue is on the Neutron side.

Related patches could be https://review.openstack.org/#/c/300245/ and https://review.openstack.org/#/c/364803/

Checked both patches - issue still appearing.

Does infra03 house your only rabbitmq service? vrrp is operating normally it seems underneath. If you look at your state_path/neutron/ha_conf/{router_id}/state you'll probably see vrrp operating without issue. Now the only way the l3 agent will update neutron api is through rabbitmq and if it's not available, no update will be made until the service returns.

Not sure what can be done here if rabbitmq dies as well.

Yes, you are correct, if I check state_path/neutron/ha_conf/{router_id}/state it will show which agent is active for router. As I said in comment #2 probably the variant can be notify L3 agent to recheck states if for server all agents shown as standby, but I haven't checked something like this yet.

@Ann, so the main question I had didn't get answered as failure testing of infra01/02 is ok. The main thing we need to understand is if the rabbit node is getting killed during this testing on infra03. No matter what the solution you present, if there's no rpc, no states will get updated. The reason I ask this is infra01/02 failure testing shows that the states get updated just fine.

@Brian - This is an OSA/RPC build with RabbitMQ clustering across 3 infra nodes. In this test we simply wanted to kill the server where the router was 'master' and observe the behavior. As you pointed out, VRRP under the hood was operating correctly - one of the two remaining nodes took over as master and the user did not sustain a long period of downtime. However, there was no clear indication from the API which infra node had taken over, and there was no way to update that state without either restarting the L3 agent(s) or causing a transition through manual means.

Looks like the state notification will choose only one rpc node and if that become unavailable it seems there's not enough time to detect this impact before we fail the state change notification:

https://github.com/openstack/neutron/blob/master/neutron/agent/l3/agent.py#L679

Might be best to detect a failed state, maybe determine if other nodes are possible, and if so wait for a reconnect to rabbit before trying again. Might have to play with this a bit, testing out some solutions.

Was able to duplicate this in devstack. But, oslo handles the reconnect to the secondary rabbit host correctly and reissues the call to the newly connected host. Checking stable/mitaka for the differences.

@James, what is the olso.messaging version?

@James, and also what version is kombu?

So this is a very narrow window that is being hit:

https://github.com/openstack/neutron/blob/master/neutron/common/rpc.py#L127

When the oslo.messaging driver fetches the connection, it has to hit a tiny window when it views the socket as not timed out:

https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/amqpdriver.py#L419

The connection is validated and if it's viewed as a valid connection, the message is sent:

https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L705

Ultimately, we get the traceback and it times out waiting for the reply, and it's handled by just killing the request:

https://github.com/openstack/neutron/blob/master/neutron/common/rpc.py#L128

Now, we could add an rpc_ha_retry_limit to attempt another send, basically another param that we act on. But this narrow window that we are hitting explains why sometimes you see it and sometimes you don't, just depends on how quick everything falls into place.

Some questions though:
If the host dies, how can the amqp client view the connection as valid and send anything if it can't establish a connection to the rabbit host? How is it able to accept a connection?

Is it possible ensure_publishing isn't handling a disconnect well? or kombu?

https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L1156

Fix proposed to branch: master
Review: https://review.openstack.org/411784

In regards to the versions we run the following versions of kombu and olso:

kombu (3.0.34)
oslo.messaging (4.6.1)

Reviewed: https://review.openstack.org/411784
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1927da1bc7c4e56162dd3704d58d3b922d4ebce9
Submitter: Jenkins
Branch: master

commit 1927da1bc7c4e56162dd3704d58d3b922d4ebce9
Author: AKamyshnikova <email address hidden>
Date: Fri Dec 16 16:00:59 2016 +0400

    Add check for ha state

    If all agents are shown as a standby it is possible changing state
    were lost due to problems with RabbitMQ. Current change adds check
    for ha state in fetch_and_sync_all_routers. If state is different -
     notify server that state should be changed.

    Also change _get_bindings_and_update_router_state_for_dead_agents
    to set standby for dead agent only in case we have more than one
    active.

    Change-Id: If5596eb24041ea9fae1d5d2563dcaf655c5face7
    Closes-bug:#1648242

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/423055

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/423064

Reviewed: https://review.openstack.org/423055
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ff7c5c25d3845e4cbdd318389c02564290a7a0f2
Submitter: Jenkins
Branch: stable/newton

commit ff7c5c25d3845e4cbdd318389c02564290a7a0f2
Author: AKamyshnikova <email address hidden>
Date: Fri Dec 16 16:00:59 2016 +0400

    Add check for ha state

    If all agents are shown as a standby it is possible changing state
    were lost due to problems with RabbitMQ. Current change adds check
    for ha state in fetch_and_sync_all_routers. If state is different -
     notify server that state should be changed.

    Also change _get_bindings_and_update_router_state_for_dead_agents
    to set standby for dead agent only in case we have more than one
    active.

    (cherry picked from commit 1927da1bc7c4e56162dd3704d58d3b922d4ebce9)
    Change-Id: If5596eb24041ea9fae1d5d2563dcaf655c5face7
    Closes-bug:#1648242

Reviewed: https://review.openstack.org/423064
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c8a4fa46948e5026923ae22eb75aeeb4a93161c4
Submitter: Jenkins
Branch: stable/mitaka

commit c8a4fa46948e5026923ae22eb75aeeb4a93161c4
Author: AKamyshnikova <email address hidden>
Date: Fri Dec 16 16:00:59 2016 +0400

    Add check for ha state

    If all agents are shown as a standby it is possible changing state
    were lost due to problems with RabbitMQ. Current change adds check
    for ha state in fetch_and_sync_all_routers. If state is different -
     notify server that state should be changed.

    Also change _get_bindings_and_update_router_state_for_dead_agents
    to set standby for dead agent only in case we have more than one
    active.

    (cherry picked from commit 1927da1bc7c4e56162dd3704d58d3b922d4ebce9)
    Change-Id: If5596eb24041ea9fae1d5d2563dcaf655c5face7
    Closes-bug:#1648242

This issue was fixed in the openstack/neutron 10.0.0.0b3 development milestone.

This issue was fixed in the openstack/neutron 9.2.0 release.

Waiting for neutron 9.2.0 PR (bug 1664306) to hit N UCA (already landed in Ubuntu Y) before submitting for Xenial/Mitaka.

newton 9.2.0 now fully released so proceeding with mitaka sru

The attachment "lp1648242-xenial-mitaka.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Uploaded to xenial-proposed for SRU team review as part of 2:8.4.0-0ubuntu2

Hello James, or anyone else affected,

Accepted neutron into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:8.4.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Tested on Xenial Mitaka and lgtm.

Hello James, or anyone else affected,

Accepted neutron into mitaka-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:mitaka-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-mitaka-needed to verification-mitaka-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-mitaka-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package neutron - 2:8.4.0-0ubuntu2

---------------
neutron (2:8.4.0-0ubuntu2) xenial; urgency=medium

  [ Edward Hope-Morley ]
  * Backport fix for Failure to retry update_ha_routers_states (LP: #1648242)
    - d/p/add-check-for-ha-state.patch

  [ Chuck Short ]
  * d/neutron-common.install, d/neutron-dhcp-agent.install:
    Remove cron jobs since they will cause a race when
    using an L3 agent. The L3 agent cleans up after itself now.
    (LP: #1623664)

 -- Chuck Short <email address hidden> Wed, 19 Apr 2017 11:39:09 +0100

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package neutron - 2:8.4.0-0ubuntu2~cloud0
---------------

 neutron (2:8.4.0-0ubuntu2~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:8.4.0-0ubuntu2) xenial; urgency=medium
 .
   [ Edward Hope-Morley ]
   * Backport fix for Failure to retry update_ha_routers_states (LP: #1648242)
     - d/p/add-check-for-ha-state.patch
 .
   [ Chuck Short ]
   * d/neutron-common.install, d/neutron-dhcp-agent.install:
     Remove cron jobs since they will cause a race when
     using an L3 agent. The L3 agent cleans up after itself now.
     (LP: #1623664)