OpenStack Compute (nova)

413 error code doesn't always provide Retry-After

Bug #1079387 reported by Sean Dague
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Sean Dague
OpenStack Compute (nova) 2013.1 "grizzly"
Folsom
Fix Released
Medium
Sean Dague
OpenStack Compute (nova) 2012.2.1
nova (Ubuntu)
Fix Released
Undecided
Unassigned
Quantal
Fix Released
Undecided
Unassigned

Bug Description

In nova we use the 413 error code (Request Entity Too Large) for quota errors, not just the originally intended "server out or resources right now". As such, we don't always have a Retry-After set.

The code currently assumes that on any 413 we have set a Retry-After, and then dies in an exception because that key isn't found (an example being):

2012-11-15 13:48:18 DEBUG nova.quota [req-c9829c33-352d-49d3-9ae6-e102f7851c15 ServerMetadataTest-user ServerMetadataTest-tenant] Created reservations ['39653c9e-578c-46ca-8105-f2c9c6a1948f', 'b7042f9d-c361-4728-88de-676cfb45c1bf', 'fd679562-ef6d-4b3f-83b3-5d09204462cb'] reserve /opt/stack/nova/nova/quota.py:721
2012-11-15 13:48:18 WARNING nova.compute.api [req-c9829c33-352d-49d3-9ae6-e102f7851c15 ServerMetadataTest-user ServerMetadataTest-tenant] Metadata property key greater than 255 characters
2012-11-15 13:48:18 INFO nova.api.openstack.wsgi [req-c9829c33-352d-49d3-9ae6-e102f7851c15 ServerMetadataTest-user ServerMetadataTest-tenant] HTTP exception thrown: Invalid metadata size: Metadata property key greater than 255 characters
2012-11-15 13:48:18 INFO nova.api.openstack.wsgi [req-c9829c33-352d-49d3-9ae6-e102f7851c15 ServerMetadataTest-user ServerMetadataTest-tenant] http://127.0.0.1:8774/v2/650a0ae2e4ad465680ebac8a7ccaf36a/servers returned with HTTP 413
2012-11-15 13:48:18 ERROR nova.api.openstack [req-c9829c33-352d-49d3-9ae6-e102f7851c15 ServerMetadataTest-user ServerMetadataTest-tenant] Caught error: 'retry-after'
2012-11-15 13:48:18 24798 TRACE nova.api.openstack Traceback (most recent call last):
2012-11-15 13:48:18 24798 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/__init__.py", line 81, in __call__
2012-11-15 13:48:18 24798 TRACE nova.api.openstack return req.get_response(self.application)
2012-11-15 13:48:18 24798 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1053, in get_response
2012-11-15 13:48:18 24798 TRACE nova.api.openstack application, catch_exc_info=False)
2012-11-15 13:48:18 24798 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1022, in call_application
2012-11-15 13:48:18 24798 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2012-11-15 13:48:18 24798 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-11-15 13:48:18 24798 TRACE nova.api.openstack return resp(environ, start_response)
2012-11-15 13:48:18 24798 TRACE nova.api.openstack File "/opt/stack/keystone/keystone/middleware/auth_token.py", line 288, in __call__
2012-11-15 13:48:18 24798 TRACE nova.api.openstack return self.app(env, start_response)
2012-11-15 13:48:18 24798 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-11-15 13:48:18 24798 TRACE nova.api.openstack return resp(environ, start_response)
2012-11-15 13:48:18 24798 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-11-15 13:48:18 24798 TRACE nova.api.openstack return resp(environ, start_response)
2012-11-15 13:48:18 24798 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2012-11-15 13:48:18 24798 TRACE nova.api.openstack response = self.app(environ, start_response)
2012-11-15 13:48:18 24798 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-11-15 13:48:18 24798 TRACE nova.api.openstack return resp(environ, start_response)
2012-11-15 13:48:18 24798 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 159, in __call__
2012-11-15 13:48:18 24798 TRACE nova.api.openstack return resp(environ, start_response)
2012-11-15 13:48:18 24798 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 147, in __call__
2012-11-15 13:48:18 24798 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2012-11-15 13:48:18 24798 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 208, in call_func
2012-11-15 13:48:18 24798 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2012-11-15 13:48:18 24798 TRACE nova.api.openstack File "/opt/stack/nova/nova/api/openstack/wsgi.py", line 1183, in __call__
2012-11-15 13:48:18 24798 TRACE nova.api.openstack retry = self.wrapped_exc.headers['Retry-After']
2012-11-15 13:48:18 24798 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/webob/headers.py", line 16, in __getitem__
2012-11-15 13:48:18 24798 TRACE nova.api.openstack raise KeyError(key)
2012-11-15 13:48:18 24798 TRACE nova.api.openstack KeyError: 'retry-after'
2012-11-15 13:48:18 24798 TRACE nova.api.openstack

Sean Dague (sdague)
Changed in nova:
importance: Undecided → Medium
assignee: nobody → Sean Dague (sdague-b)
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/16273

Changed in nova:
status: Triaged → In Progress
Vish Ishaya (vishvananda)
tags: added: folsom-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/16273
Committed: http://github.com/openstack/nova/commit/3f95c7a4aaaf44e029d86ce2b7d46c1315e908e4
Submitter: Jenkins
Branch: master

commit 3f95c7a4aaaf44e029d86ce2b7d46c1315e908e4
Author: Sean Dague <email address hidden>
Date: Thu Nov 15 19:00:33 2012 -0500

    don't explode if a 413 didn't set Retry-After

    We use 413 error code not only for temporary out of memory
    conditions but also for over quota issues. These kinds of
    requests don't set a Retry-After header, but the code assumed
    that was a valid key, thus exploding on a KeyError. Issue
    caught in looking at tempest traces.

    Fixes bug #1079387

    Change-Id: Id406ab0799a35220b189b2e5292be092eb821ec5

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/16303

Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-1
status: Fix Committed → Fix Released
Vish Ishaya (vishvananda)
tags: removed: folsom-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/folsom)

Reviewed: https://review.openstack.org/16303
Committed: http://github.com/openstack/nova/commit/55d1412dc0adeae0ced23aa330213db2ce6d7f61
Submitter: Jenkins
Branch: stable/folsom

commit 55d1412dc0adeae0ced23aa330213db2ce6d7f61
Author: Sean Dague <email address hidden>
Date: Thu Nov 15 19:00:33 2012 -0500

    don't explode if a 413 didn't set Retry-After

    We use 413 error code not only for temporary out of memory
    conditions but also for over quota issues. These kinds of
    requests don't set a Retry-After header, but the code assumed
    that was a valid key, thus exploding on a KeyError. Issue
    caught in looking at tempest traces.

    Fixes bug #1079387

    Change-Id: Id406ab0799a35220b189b2e5292be092eb821ec5

Adam Gandelman (gandelman-a)
Changed in nova (Ubuntu):
status: New → Fix Released
Changed in nova (Ubuntu Quantal):
status: New → Confirmed
Revision history for this message
Clint Byrum (clint-fewbar) wrote : Please test proposed package

Hello Sean, or anyone else affected,

Accepted nova into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/nova/2012.2.1+stable-20121212-a99a802e-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in nova (Ubuntu Quantal):
status: Confirmed → Fix Committed
tags: added: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (8.3 KiB)

This bug was fixed in the package nova - 2012.2.1+stable-20121212-a99a802e-0ubuntu1

---------------
nova (2012.2.1+stable-20121212-a99a802e-0ubuntu1) quantal-proposed; urgency=low

  * Ubuntu updates:
    - debian/control: Ensure novaclient is upgraded with nova,
      require python-keystoneclient >= 1:2.9.0. (LP: #1073289)
    - d/p/avoid_setuptools_git_dependency.patch: Refresh.
  * Dropped patches, applied upstream:
    - debian/patches/CVE-2012-5625.patch: [a99a802]
  * Resynchronize with stable/folsom (b55014ca) (LP: #1085255):
    - [a99a802] create_lvm_image allocates dirty blocks (LP: #1070539)
    - [670b388] RPC exchange name defaults to 'openstack' (LP: #1083944)
    - [3ede373] disassociate_floating_ip with multi_host=True fails
      (LP: #1074437)
    - [22d7c3b] libvirt imagecache should handle shared image storage
      (LP: #1075018)
    - [e787786] Detached and deleted RBD volumes remain associated with insance
      (LP: #1083818)
    - [9265eb0] live_migration missing migrate_data parameter in Hyper-V driver
      (LP: #1066513)
    - [3d99848] use_single_default_gateway does not function correctly
      (LP: #1075859)
    - [65a2d0a] resize does not migrate DHCP host information (LP: #1065440)
    - [102c76b] Nova backup image fails (LP: #1065053)
    - [48a3521] Fix config-file overrides for nova-dhcpbridge
    - [69663ee] Cloudpipe in Folsom: no such option: cnt_vpn_clients
      (LP: #1069573)
    - [6e47cc8] DisassociateAddress can cause Internal Server Error
      (LP: #1080406)
    - [22c3d7b] API calls to dis-associate an auto-assigned floating IP should
      return proper warning (LP: #1061499)
    - [bd11d15] libvirt: if exception raised during volume_detach, volume state
      is inconsistent (LP: #1057756)
    - [dcb59c3] admin can't describe all images in ec2 api (LP: #1070138)
    - [78de622] Incorrect Exception raised during Create server when metadata
      over 255 characters (LP: #1004007)
    - [c313de4] Fixed IP isn't released before updating DHCP host file
      (LP: #1078718)
    - [f4ab42d] Enabling Return Reservation ID with XML create server request
      returns no body (LP: #1061124)
    - [3db2a38] 'BackupCreate' should accept rotation parameter greater than or
      equal to zero (LP: #1071168)
    - [f7e5dde] libvirt reboot sometimes fails to reattach volumes
      (LP: #1073720)
    - [ff776d4] libvirt: detaching volume may fail while terminating other
      instances on the same host concurrently (LP: #1060836)
    - [85a8bc2] Used instance uuid rather than id in remove-fixed-ip
    - [42a85c0] Fix error on invalid delete_on_termination value
    - [6a17579] xenapi migrations fail w/ swap (LP: #1064083)
    - [97649b8] attach-time field for volumes is not updated for detach volume
      (LP: #1056122)
    - [8f6a718] libvirt: rebuild is not using kernel and ramdisk associated with
      the new image (LP: #1060925)
    - [fbe835f] live-migration and volume host assignement (LP: #1066887)
    - [c2a9150] typo prevents volume_tmp_dir flag from working (LP: #1071536)
    - [93efa21] Instances deleted during spawn leak network allocations
      (LP: #1068716)
    - [ebabd02] After restartin...

Read more...

Changed in nova (Ubuntu Quantal):
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.