[SRU] Using SSL with rabbitmq prevents communication between nova-compute and conductor after latest nova updates

Upgraded nova-compute to 1:2014.1.5-0ubuntu1

The logs show more details:

2015-07-08 11:32:29.066 13437 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on amqp.wedgecnd.internal:5672
2015-07-08 11:32:38.062 13437 WARNING nova.conductor.api [req-9f555cd9-0e37-4114-919f-f3e5e164d724 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor?
2015-07-08 11:32:38.069 13437 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: <AMQPError: unknown error>
2015-07-08 11:32:38.069 13437 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2015-07-08 11:32:38.069 13437 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 624, in ensure
2015-07-08 11:32:38.069 13437 TRACE oslo.messaging._drivers.impl_rabbit return method(*args, **kwargs)
2015-07-08 11:32:38.069 13437 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 704, in _consume
2015-07-08 11:32:38.069 13437 TRACE oslo.messaging._drivers.impl_rabbit raise self.connection.recoverable_connection_errors[0]
2015-07-08 11:32:38.069 13437 TRACE oslo.messaging._drivers.impl_rabbit RecoverableConnectionError: <AMQPError: unknown error>
2015-07-08 11:32:38.069 13437 TRACE oslo.messaging._drivers.impl_rabbit
2015-07-08 11:32:38.069 13437 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on amqp.wedgecnd.internal:5672
2015-07-08 11:32:38.069 13437 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...

I added "oslo.messaging" as affected project.

If the patches that are carried in the Ubuntu packaging for oslo.messaging are removed and the nova services restarted then ssl + nova + rabbit seems to work.

eg

apt-get install --yes quilt
apt-get source oslo.messaging
cd oslo.messaging-1.3.0/
export QUILT_PATCHES=debian/patches
export QUILT_REFRESH_ARGS="-p ab --no-timestamps --no-index"
quilt pop -a

cp oslo/messaging/_drivers/impl_rabbit.py /usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py
cp oslo/messaging/_drivers/common.py /usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/common.py
cp oslo/messaging/_drivers/amqpdriver.py /usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py

cd /etc/init/; for i in nova-*conf; do service ${i/.conf/} restart; done

A bit more info from my end. I've been trying out different scenarios and it seems that this is constrained to Trusty Icehouse using python-oslo.messaging version 1.3.0-0ubuntu1.2 configured to connect to rabbitmq-server using ssl e.g. my nova.conf has:

rabbit_userid = nova
rabbit_virtual_host = openstack
rabbit_password = gr6Mx2FJhC8NH3P4dBRGH8tYT39s6LLcMfJChKM6dtb3rpN5wfkRWVBcMLdhqp58
rabbit_host = 10.5.6.86
rabbit_use_ssl = True
rabbit_port = 5671
kombu_ssl_ca_certs = /etc/nova/rabbit-client-ca.pem

I've played around with reverting back to 1.3.0-0ubuntu1 (which does not appear to exhibit the issue) and re-adding patches one-by-one and have found that simply adding the patch for bug 1400268 causes the issue to occur. So, question is what is it about that patch that causes these issues?

OK upon further investigation i have found some trace of a root cause. Oslo.messaging always uses a timeout of 1 second when polling queues and connections. This appears to be too small when using ssl and frequently results in SSLError/timeout which cause all threads to fail and reconnect and fail again repeatedly thus resulting in the number of connections rising fast and rpc not working, hence why compute and conductor are not able to communicate. I've played around with alternative timeout values and I get much better results even with a value of 2s instead of 1s. I'll propose an initial workaround patch shortly so we can get out of this bind for now but I think we'll ultimately need a more intelligent solution than what oslo.messaging support in this version.

Finally got to the bottom of this. The issue lies in python-amqp rather than python-oslo.messaging. The current trusty version of python-amqp (1.3.3) has a bug that is fixed in 1.4.4 (see http://amqp.readthedocs.org/en/latest/changelog.html#version-1-4-4). I tried backporting the Juno/Utopic version (1.4.5) for Trusty and everything works just fine now. I will shortly propose an SRU to get python-amqp fixed in Trusty.

Hello Dina, or anyone else affected,

Accepted python-amqp into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-amqp/1.3.3-1ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Tested python-amqp version 1.3.3-1ubuntu1.1 from the trusty-proposed and it fixes the bug.

I have run an ssl only icehouse deployment with this and can also verify that it does fix the bug.

This bug was fixed in the package python-amqp - 1.3.3-1ubuntu1.1

---------------
python-amqp (1.3.3-1ubuntu1.1) trusty; urgency=medium

  * Ensure SSL read timeouts raised properly (LP: #1472712):
    - d/p/dont-disconnect-transport-on-ssl-read-timeout.patch:
      Backport patch from 1.4.4.

 -- Edward Hope-Morley <email address hidden> Mon, 10 Aug 2015 15:57:44 +0100

The verification of the Stable Release Update for python-amqp has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This is effecting precise/icehouse deployments as they have the old version of python-amqp