nova quota-update command is executed without checking if the quota exceed

Bug #1160749 reported by David Geng
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
David Geng

Bug Description

1. Changed number of the cores from 100 to 15 with the following command:

[root@firstbox11-1 ˜]# nova quota-update --cores 15 2bff6058d6454dc0859cc545c2dc6d4f
[root@firstbox11-1 ˜]# nova quota-show 2bff6058d6454dc0859cc545c2dc6d4f
+-----------------------------+--------+
| Property | Value |
+-----------------------------+--------+
| cores | 15 |
| floating_ips | 40 |
| gigabytes | 1000 |
| injected_file_content_bytes | 10240 |
| injected_files | 5 |
| instances | 100 |
| metadata_items | 128 |
| ram | 204800 |
| volumes | 10 |
+-----------------------------+--------+

2. Perform a deploy using a flavor with 8 CPUs like this:

tm_fl: Memory: 2048MB, VCPUS: 8, Root: 8GB, Ephemeral: 0Gb, FlavorID: 69, Swap: 0MB, RXTX Factor: 1.0, public, ExtraSpecs {}

the deploy fails with the following message:
'Quota exceeded for cores: Requested 8, but already used 47 of 15 cores'

This is an issue, because 47 cores are already used, but the nova quota-update does not check and permits to decrease the cores value under
47.....in alternative if it is possible have a number of cores already used greater than of the cores defined on quota....it is an usability issue because minimum a warning should be issued when nova quota-update is issued.

Revision history for this message
David Geng (genggjh) wrote :

Currently we have two ways to update quotas:
1. nova quota-update
2. nova-manage project quota <tenant-id> .....

Looks like the easy way is update nova-manage to check whether the new quota exceeds the "in_use", but it will lead to inconsistent the above two CLI.
Does anyone has some better solution?

Changed in python-novaclient:
assignee: nobody → David Geng (genggjh)
Revision history for this message
Kevin L. Mitchell (klmitch) wrote :

I'm not sure that there is a good way of preventing this situation in the light of potential race conditions (allocation of the 16th core at the same time the quota is being set to 15, say). Depending on the quota being changed (default, per-quota-class, per-tenant), we could be having to search for the usage of thousands of tenants, too. Issuing a warning of some sort might be easier in the nova-manage case, but we're trying to deprecate nova-manage, and I know of no easy way of issuing warnings through the nova API response…and I definitely don't want to see novaclient sending a lot of traffic to nova-api just to check whether a quota change results in someone being over quota.

Revision history for this message
David Geng (genggjh) wrote :

@Kevin, is it possible to add those check in the nova server side and compare the qutoa usage info from database in the QuotaSetsController when update qutoa?

David Geng (genggjh)
Changed in python-novaclient:
status: New → In Progress
Revision history for this message
David Geng (genggjh) wrote :
Revision history for this message
David Geng (genggjh) wrote :

The code change is in nova side instead of nova client

affects: python-novaclient → nova
Changed in nova:
importance: Undecided → Medium
tags: added: grizzly-backport-potential
Revision history for this message
David Geng (genggjh) wrote :
Revision history for this message
Attila Fazekas (afazekas) wrote :

The issue with the force option it can break existing admin scripts and tools.

What is your opinion instead of using "force", you should use a "check" Boolean argument.

The behavior without "check" will be the existing behavior.

With "check" it will refuse the update if one of limits will be reached.

Or you can add the force option to the v3 API only.

It is part of the admin API , usually the admins know why they want to limit the quota, and usually if an admin wants to limit something he has strong reason.

Revision history for this message
David Geng (genggjh) wrote :

I understand your point on we need keep the back compatible to avoid break existing behavior, but if we do not control this by default, it will very hard for admins to be aware of the quotas he want to change are already exceed.

Maybe the force option to v3 API only is one choice, but how we handle this in our product which will adopt grizzly?

Revision history for this message
Attila Fazekas (afazekas) wrote :

Last time when the quota set changed it was related to security issue. (IP address over usage)

May be I do not fully understand why it is so important.
AFAIK: The admin usually changes quotas from requests from bosses or because the quota is not paid by the user/costumer.
What is the other frequent use cases ?

If we just consider the CLI usage as an use case.
You can have the CLI (just when used from CLI, not as library) to implicitly add an optional request parameter ('check') when it called without the ' --force' argument.

The above mentioned change is almost invisible from API side and makes the CLI to work as you want.

Revision history for this message
David Geng (genggjh) wrote :

Currently this is from our operation experience and customer requirement, we need let admin be aware of when the quotas will oversell, this is very important for operation.

You know currently there is no way to show how many quotas are in used from CLI (nova quota-show), if we have no any warning to show the quotas exceed when admin update the quotas, admin will not know the quotas were already oversold.

For the 'check' parameter, I still have concern regarding this. If we do not check this in code level, I really don't think admin will take the initiative to do the 'check'.

So consider to avoid break existing behavior, How about enable the force option to V3 API only?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/25887
Committed: http://github.com/openstack/nova/commit/d5bbfad3d06e8801d70f4efce84c9504519efbc3
Submitter: Jenkins
Branch: master

commit d5bbfad3d06e8801d70f4efce84c9504519efbc3
Author: gengjh <email address hidden>
Date: Mon Apr 1 22:11:50 2013 +0800

    Enhance the validation of the quotas update

    Need check whether the already used and reserved exceeds the new quota
    before update it.

    DocImpact
    Implements a validation to validate whether already used and reserved
    quota exceeds the new quota when run 'nova quota-update', it will throw
    error if the quota exceeds. This check will be ignored if admin want to
    force update when run 'nova quota-update' with additional option
    '--force'.
    This validation help admin to be aware of whether the quotas are
    oversold when they try to update quota and also provide an option
    '--force' to allow admin force update the quotas.

    Fix bug 1160749

    Change-Id: Iba3cee0f0d92cf2e6d64bc83830b0091992d1ee9

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
David Geng (genggjh) wrote :

The nova client need to be changes as well

Changed in nova:
status: Fix Committed → In Progress
Alan Pevec (apevec)
tags: removed: grizzly-backport-potential
Revision history for this message
Joe Gordon (jogo) wrote :

Not sure how the client can solve this.

Changed in nova:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.