Openstack API Charms have a naive haproxy connection timeout

Bug #1522130 reported by David Ames
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
ceilometer (Juju Charms Collection)
Fix Released
High
Unassigned
ceph-radosgw (Juju Charms Collection)
Fix Released
High
Unassigned
cinder (Juju Charms Collection)
Fix Released
High
Unassigned
glance (Juju Charms Collection)
Fix Released
High
Unassigned
keystone (Juju Charms Collection)
Fix Released
High
Unassigned
neutron-api (Juju Charms Collection)
Fix Released
High
Unassigned
nova-cloud-controller (Juju Charms Collection)
Fix Released
High
Unassigned
openstack-dashboard (Juju Charms Collection)
Fix Released
High
Unassigned
swift-proxy (Juju Charms Collection)
Fix Released
High
Unassigned

Bug Description

On OpenStack API charms haproxy sets timeout connection to 1 second. This is reasonable only in an ideal scenario. Connections to backends on a busy cloud may take much longer. Even haproxy docs suggest 4 to 5 seconds [1].

defaults
    log global
    mode tcp
    option tcplog
    option dontlognull
    retries 3
    timeout queue 1000
    timeout connect 1000

This causes mysterious errors like the following:

On neutron-api:
ConnectionRefused: Unable to establish connection to http://10.245.161.159:9696/v2.0/security-groups.json

On keystone:
ERROR: HTTPConnectionPool(host='10.5.51.133', port=5000): Max retries exceeded with url: /v2.0/tokens (Caused by <class 'httplib.BadStatusLine'>: '')

On nova-cloud-controller:
requests.exceptions.ConnectionError: HTTPConnectionPool(host='10.245.161.158', port=8774): Max retries exceeded with url: /v2/ba51369fa6444c6d9c2be7264675741f/servers/373ed532-9819-4f2b-b32a-4d27147d0338/os-interface (Caused by <class 'httplib.BadStatusLine'>: '')

When max_retries is forced > 1 it becomes only slightly less opaque:

urllib3.connectionpool - WARNING - Retrying (5 attempts remain) after connection broken by 'BadStatusLine("''",)': /v2/ba51369fa6444c6d9c2be7264675741f/servers/32eed1a0-6c9d-497f-b9fb-aac0fa748858/os-interface

Set timeout connect (and timeout queue while we are at it) to sane defaults and optionally make this configurable.

[1] https://cbonte.github.io/haproxy-dconv/configuration-1.5.html#4-timeout%20connect

Related branches

David Ames (thedac)
description: updated
affects: charms → nova-cloud-controller (Juju Charms Collection)
Revision history for this message
Ryan Beisner (1chb1n) wrote :

Yes, I think a charm config option (and/or a more sane timeout default value) would be great, for all os-charms which employ haproxy.

I've observed the "''" return during times of elevated load, in neutron-api, nova and dashboard.

David Ames (thedac)
description: updated
Revision history for this message
Ryan Beisner (1chb1n) wrote :

Observable a symptom when attempting to delete nova security groups as a normal user just using the cloud:

nova secgroup-delete XYZ
ERROR: ('Connection aborted.', BadStatusLine("''",))

http://paste.ubuntu.com/13623685/

Liam Young (gnuoy)
Changed in glance (Juju Charms Collection):
status: New → Invalid
Changed in nova-cloud-controller (Juju Charms Collection):
status: New → Invalid
Changed in cinder (Juju Charms Collection):
status: New → In Progress
Changed in keystone (Juju Charms Collection):
status: New → In Progress
Changed in neutron-api (Juju Charms Collection):
status: New → In Progress
Changed in cinder (Juju Charms Collection):
importance: Undecided → High
Changed in keystone (Juju Charms Collection):
importance: Undecided → High
Changed in neutron-api (Juju Charms Collection):
importance: Undecided → High
Changed in cinder (Juju Charms Collection):
assignee: nobody → Liam Young (gnuoy)
Changed in keystone (Juju Charms Collection):
assignee: nobody → Liam Young (gnuoy)
Changed in neutron-api (Juju Charms Collection):
assignee: nobody → Liam Young (gnuoy)
Changed in cinder (Juju Charms Collection):
milestone: none → 16.01
Changed in keystone (Juju Charms Collection):
milestone: none → 16.01
Changed in neutron-api (Juju Charms Collection):
milestone: none → 16.01
Revision history for this message
Liam Young (gnuoy) wrote :

I misread this and thought it was case of just adding haproxy-server-timeout and haproxy-client-timeout options to the config.yaml of the charms which do not have it already. I see now that the options that need changing aren't the ones controlled by those config options.

Changed in cinder (Juju Charms Collection):
status: In Progress → Confirmed
Changed in glance (Juju Charms Collection):
status: Invalid → Confirmed
Changed in keystone (Juju Charms Collection):
status: In Progress → Confirmed
Changed in neutron-api (Juju Charms Collection):
status: In Progress → Confirmed
Changed in nova-cloud-controller (Juju Charms Collection):
status: Invalid → Won't Fix
status: Won't Fix → Confirmed
Changed in glance (Juju Charms Collection):
importance: Undecided → High
Changed in nova-cloud-controller (Juju Charms Collection):
importance: Undecided → High
Changed in cinder (Juju Charms Collection):
assignee: Liam Young (gnuoy) → nobody
Changed in keystone (Juju Charms Collection):
assignee: Liam Young (gnuoy) → nobody
Changed in neutron-api (Juju Charms Collection):
assignee: Liam Young (gnuoy) → nobody
Changed in glance (Juju Charms Collection):
milestone: none → 16.01
Changed in nova-cloud-controller (Juju Charms Collection):
milestone: none → 16.01
Revision history for this message
David Ames (thedac) wrote :

For the record *all* of the timeout values could use reconfigurability. I wound up changing

    timeout queue 1000
    timeout connect 1000
    timeout client 30000
    timeout server 30000

to

    timeout queue 30000
    timeout connect 30000
    timeout client 90000
    timeout server 90000

This all in the haproxy.cfg template file
defaults
    log global
    mode tcp
    option tcplog
    option dontlognull
    retries 3
    timeout queue 1000
    timeout connect 1000
{% if haproxy_client_timeout -%}
    timeout client {{ haproxy_client_timeout }}
{% else -%}
    timeout client 30000
{% endif -%}

{% if haproxy_server_timeout -%}
    timeout server {{ haproxy_server_timeout }}
{% else -%}
    timeout server 30000
{% endif -%}

David Ames (thedac)
Changed in ceph-radosgw (Juju Charms Collection):
status: New → Confirmed
importance: Undecided → High
Changed in openstack-dashboard (Juju Charms Collection):
status: New → Confirmed
importance: Undecided → High
Changed in swift-proxy (Juju Charms Collection):
status: New → Confirmed
importance: Undecided → High
Changed in openstack-dashboard (Juju Charms Collection):
milestone: none → 16.01
Changed in swift-proxy (Juju Charms Collection):
milestone: none → 16.01
Changed in ceph-radosgw (Juju Charms Collection):
milestone: none → 16.01
David Ames (thedac)
Changed in ceilometer (Juju Charms Collection):
importance: Undecided → High
milestone: none → 16.01
status: New → Confirmed
James Page (james-page)
Changed in ceilometer (Juju Charms Collection):
status: Confirmed → Fix Committed
Changed in ceph-radosgw (Juju Charms Collection):
status: Confirmed → Fix Committed
Changed in cinder (Juju Charms Collection):
status: Confirmed → Fix Committed
Changed in glance (Juju Charms Collection):
status: Confirmed → Fix Committed
Changed in keystone (Juju Charms Collection):
status: Confirmed → Fix Committed
Changed in neutron-api (Juju Charms Collection):
status: Confirmed → Fix Committed
Changed in nova-cloud-controller (Juju Charms Collection):
status: Confirmed → Fix Committed
Changed in openstack-dashboard (Juju Charms Collection):
status: Confirmed → Fix Committed
Changed in swift-proxy (Juju Charms Collection):
status: Confirmed → Fix Committed
James Page (james-page)
Changed in nova-cloud-controller (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in neutron-api (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in keystone (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in glance (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in cinder (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in openstack-dashboard (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in ceph-radosgw (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in swift-proxy (Juju Charms Collection):
status: Fix Committed → Fix Released
Changed in ceilometer (Juju Charms Collection):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.