[nova][volumes] Exceeding volumes, gigabytes and floating_ips quotas returns general uninformative HTTP 500 error

Apart from obscuring the true source of the error, this generic-500 approach in the native openstack API is also inconsistent with the EC2 API, where the exception type and message are exposed in the /Response/Errors/Error/Code and /Response/Errors/Error/Message elements respectively.

I'm explicitly not submitting to gerrit, since a proper patch would cover all the other places where we would need a similar change. I can submit if wanted, though.

Fix proposed to branch: master
Review: https://review.openstack.org/9429

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/9430

Note that I've kept the 500 status, but with a more informative message, as opposed to switching to for example a 413 status - the reasoning here is that the Retry-After logic is not useful in a quota breach case, as opposed to a rate-limit throttling.

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/9446

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/9534

Reviewed: https://review.openstack.org/9429
Committed: http://github.com/openstack/nova/commit/e9d21589d39355ffc126e360cc2ba7311e014edb
Submitter: Jenkins
Branch: master

commit e9d21589d39355ffc126e360cc2ba7311e014edb
Author: Eoghan Glynn <email address hidden>
Date: Fri Jul 6 10:10:28 2012 +0000

    Expose over-quota exceptions via native API.

    Fixes bug LP 1021373.

    Previously an over-quota condition would be exposed via the EC2 API,
    but hidden in the corresponding call via the native API (in the sense
    of the exception detail being replaced with a generic 500 Server Error
    response).

    We now report any NovaException declared to be safe. In this patch,
    the set of safe exception types includes any subclass of QuotaError,
    but in subsequent patches the net should be widened to include
    exceptions that do not expose senstive information.

    Change-Id: I3cc36337c7e67cf487ca49de646c437c217ae538

Reviewed: https://review.openstack.org/9445
Committed: http://github.com/openstack/nova/commit/a3576bbb0c7090b97b0b02c88ffa81915db6290b
Submitter: Jenkins
Branch: master

commit a3576bbb0c7090b97b0b02c88ffa81915db6290b
Author: Eoghan Glynn <email address hidden>
Date: Tue Jul 10 08:16:27 2012 +0100

    Return 413 status on over-quota in the native API.

    Related to LP 1021373.

    Previously we returned a generic 500 Server Error on an over-quota
    conditions, whereas this is arguably more appropriately reported
    as a 413 status.

    Change-Id: I5c1cdc9db54804c512d60e4179c1faa13516d6f9

Reviewed: https://review.openstack.org/9430
Committed: http://github.com/openstack/nova/commit/4c49df742d1730b124f81e30f12e61602966a819
Submitter: Jenkins
Branch: stable/essex

commit 4c49df742d1730b124f81e30f12e61602966a819
Author: Eoghan Glynn <email address hidden>
Date: Fri Jul 6 10:10:28 2012 +0000

    Expose over-quota exceptions via native API.

    Fixes bug LP 1021373.

    Previously an over-quota condition would be exposed via the EC2 API,
    but hidden in the corresponding call via the native API (in the sense
    of the exception detail being replaced with a generic 500 Server Error
    response).

    We now report any NovaException declared to be safe. In this patch,
    the set of safe exception types includes any subclass of QuotaError,
    but in subsequent patches the net should be widened to include
    exceptions that do not expose senstive information.

    Change-Id: I3cc36337c7e67cf487ca49de646c437c217ae538

Reviewed: https://review.openstack.org/9446
Committed: http://github.com/openstack/nova/commit/6e873bccc078e1013987698cd71c69deebf2e99f
Submitter: Jenkins
Branch: stable/essex

commit 6e873bccc078e1013987698cd71c69deebf2e99f
Author: Eoghan Glynn <email address hidden>
Date: Mon Jul 9 21:09:06 2012 +0100

    Return 413 status on over-quota in the native API.

    Related to LP 1021373.

    Previously, we returned a generic 500 Server Error on an over-quota
    conditions, whereas this is arguably more appropriately reported
    as a 413 status.

    Change-Id: I5c1cdc9db54804c512d60e4179c1faa13516d6f9

Reviewed: https://review.openstack.org/9534
Committed: http://github.com/openstack/nova/commit/4b89b4fd3665a6eaaa63911102070fd93f20f8bc
Submitter: Jenkins
Branch: stable/essex

commit 4b89b4fd3665a6eaaa63911102070fd93f20f8bc
Author: Eoghan Glynn <email address hidden>
Date: Tue Jul 10 12:11:28 2012 +0100

    Handle local & remote exceptions consistently.

    Related to LP 1021373.

    When the floating_ips quota threshold is exceeded, this occurs remotely
    in the networking service and hence is reported back to the API service
    over RPC, via a RemoteError wrapping the original QuotaError exception.

    Previously the handling of such RemoteErrors in the native API was
    inconsistent with exceptions raised locally within the API service.

    Now in both cases, we expose the exception explantion, status etc. in the
    response, if the exception type is declared as safe.

    This fix is not required on master, as the Folsom RPC failure deserialization
    logic renders it unnecessary.

    Change-Id: Icf7dcafd1f942f5d81d12587775d6ade4a176932

Please find the attached test log from the Ubuntu Server Team's CI infrastructure. As part of the verification process for this bug, Nova has been deployed and configured across multiple nodes using precise-proposed as an installation source. After successful bring-up and configuration of the cluster, a number of exercises and smoke tests have be invoked to ensure the updated package did not introduce any regressions. A number of test iterations were carried out to catch any possible transient errors.

Please Note the list of installed packages at the top and bottom of the report.

For records of upstream test coverage of this update, please see the Jenkins links in the comments of the relevant upstream code-review(s):

Trunk review: https://review.openstack.org/9445
Stable review: https://review.openstack.org/9534

As per the provisional Micro Release Exception granted to this package by the Technical Board, we hope this contributes toward verification of this update.

Test coverage log.

Please find the attached test log from the Ubuntu Server Team's CI infrastructure. As part of the verification process for this bug, Nova has been deployed and configured across multiple nodes using precise-proposed as an installation source. After successful bring-up and configuration of the cluster, a number of exercises and smoke tests have be invoked to ensure the updated package did not introduce any regressions. A number of test iterations were carried out to catch any possible transient errors.

Please Note the list of installed packages at the top and bottom of the report.

For records of upstream test coverage of this update, please see the Jenkins links in the comments of the relevant upstream code-review(s):

Trunk review: https://review.openstack.org/9445
Stable review: https://review.openstack.org/9534

As per the provisional Micro Release Exception granted to this package by the Technical Board, we hope this contributes toward verification of this update.

Test coverage log.

This bug was fixed in the package nova - 2012.1.3+stable-20120827-4d2a4afe-0ubuntu1

---------------
nova (2012.1.3+stable-20120827-4d2a4afe-0ubuntu1) precise-proposed; urgency=low

  * New upstream snapshot, fixes FTBFS in -proposed. (LP: #1041120)
  * Resynchronize with stable/essex (4d2a4afe):
    - [5d63601] Inappropriate exception handling on kvm live/block migration
      (LP: #917615)
    - [ae280ca] Deleted floating ips can cause instance delete to fail
      (LP: #1038266)

nova (2012.1.3+stable-20120824-86fb7362-0ubuntu1) precise-proposed; urgency=low

  * New upstream snapshot. (LP: #1041120)
  * Dropped, superseded by new snapshot:
    - debian/patches/CVE-2012-3447.patch: [d9577ce]
    - debian/patches/CVE-2012-3371.patch: [25f5bd3]
    - debian/patches/CVE-2012-3360+3361.patch: [b0feaff]
  * Resynchronize with stable/essex (86fb7362):
    - [86fb736] Libvirt driver reports incorrect error when volume-detach fails
      (LP: #1029463)
    - [272b98d] nova delete lxc-instance umounts the wrong rootfs (LP: #971621)
    - [09217ab] Block storage connections are NOT restored on system reboot
      (LP: #1036902)
    - [d9577ce] CVE-2012-3361 not fully addressed (LP: #1031311)
    - [e8ef050] pycrypto is unused and the existing code is potentially insecure
      to use (LP: #1033178)
    - [3b4ac31] cannot umount guestfs (LP: #1013689)
    - [f8255f3] qpid_heartbeat setting in ineffective (LP: #1030430)
    - [413c641] Deallocation of fixed IP occurs before security group refresh
      leading to potential security issue in error / race conditions
      (LP: #1021352)
    - [219c5ca] Race condition in network/deallocate_for_instance() leads to
      security issue (LP: #1021340)
    - [f2bc403] cleanup_file_locks does not remove stale sentinel files
      (LP: #1018586)
    - [4c7d671] Deleting Flavor currently in use by instance creates error
      (LP: #994935)
    - [7e88e39] nova testsuite errors on newer versions of python-boto (e.g.
      2.5.2) (LP: #1027984)
    - [80d3026] NoMoreFloatingIps: Zero floating ips available after repeatedly
      creating and destroying instances over time (LP: #1017418)
    - [4d74631] Launching with source groups under load produces lazy load error
      (LP: #1018721)
    - [08e5128] API 'v1.1/{tenant_id}/os-hosts' does not return a list of hosts
      (LP: #1014925)
    - [801b94a] Restarting nova-compute removes ip packet filters (LP: #1027105)
    - [f6d1f55] instance live migration should create virtual_size disk image
      (LP: #977007)
    - [4b89b4f] [nova][volumes] Exceeding volumes, gigabytes and floating_ips
      quotas returns general uninformative HTTP 500 error (LP: #1021373)
    - [6e873bc] [nova][volumes] Exceeding volumes, gigabytes and floating_ips
      quotas returns general uninformative HTTP 500 error (LP: #1021373)
    - [7b215ed] Use default qemu-img cluster size in libvirt connection driver
    - [d3a87a2] Listing flavors with marker set returns 400 (LP: #956096)
    - [cf6a85a] nova-rootwrap hardcodes paths instead of using
      /sbin:/usr/sbin:/usr/bin:/bin (LP: #1013147)
    - [2efc87c] affinity filters don't work if scheduler_hints is None
      (LP: #1007573)
  ...

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.