Merge lp:~blamar/nova/lp728587 into lp:~hudson-openstack/nova/trunk

Proposed by Brian Lamar on 2011-03-16
Status: Merged
Approved by: Rick Harris on 2011-03-18
Approved revision: 806
Merged at revision: 830
Proposed branch: lp:~blamar/nova/lp728587
Merge into: lp:~hudson-openstack/nova/trunk
Diff against target: 1238 lines (+912/-169)
8 files modified
etc/api-paste.ini (+1/-1)
nova/api/openstack/__init__.py (+6/-0)
nova/api/openstack/faults.py (+39/-0)
nova/api/openstack/limits.py (+358/-0)
nova/tests/api/openstack/__init__.py (+1/-1)
nova/tests/api/openstack/fakes.py (+5/-5)
nova/tests/api/openstack/test_adminapi.py (+0/-1)
nova/tests/api/openstack/test_limits.py (+502/-161)
To merge this branch: bzr merge lp:~blamar/nova/lp728587
Reviewer Review Type Date Requested Status
Rick Harris (community) Approve on 2011-03-18
Paul Voccio (community) 2011-03-16 Approve on 2011-03-17
Titan 2011-03-16 Pending
Mark Washenberger 2011-03-16 Pending
Review via email: mp+53554@code.launchpad.net

Commit message

Re-implementation (or just implementation in many cases) of Limits in the OpenStack API. Limits is now available through /limits and the concept of a limit has been extended to include arbitrary regex / http verb combinations along with correct XML/JSON serialization. Tests included.

Description of the change

The addition of a /limits resource was a larger job than I had first thought. The original rate-limiting middleware and supporting classes were being tested through `time.sleep` calls instead of actually unit-testing the logic.

I've replaced most of the classes with something that is more testable and much more compatible with the output we're generating for the 1.1 API.

Overall we should have much more test coverage than with the previous limits implementation.

To post a comment you must log in.
lp:~blamar/nova/lp728587 updated on 2011-03-16
800. By Brian Lamar on 2011-03-16

Removed VIM specific stuff and changed copyright from 2010 to 2011.

801. By Brian Lamar on 2011-03-16

Added tests back for RateLimitingMiddleware which now throw correctly serialized
errors with correct error codes.

Removed some error printing, and simplified some other parts of the code with
suggestions from teammates.

802. By Brian Lamar on 2011-03-16

Added i18n to error message.

Jay Pipes (jaypipes) wrote :

Hi Brian! Just FYI, make sure you select "nova-core" as the reviewer when you do a merge request. You can add additional reviewers with the "Request another review" link on the merge proposal page (and I encourage you to do so, like you've done!)

The reason to do this is because nova-core sends email notifications to the dozen or so core committers, hopefully prodding more reviews :)

cheers!
jay

Jay Pipes (jaypipes) wrote :

Also, if you don't select *any* reviewer, it defaults to nova-core, just an FYI...

Brian Lamar (blamar) wrote :

Hey Jay, I was waiting to hear back from a teammate before submitting directly to nova-core. I'll add them now since I realize the deadline is coming up soonish! Thanks again.

lp:~blamar/nova/lp728587 updated on 2011-03-17
803. By Brian Lamar on 2011-03-17

Merged trunk.

Paul Voccio (pvo) wrote :

nova/api/openstack/faults.py:74:41: E202 whitespace before '}'
                "overLimitFault": "code"
                                        ^

nova/api/openstack/faults.py:97:80: E501 line too long (80 characters)
        self.wrapped_exc.body = serializer.serialize(self.content, content_type)

got a few pep8 nits to go over. Will look for the remerge.

review: Needs Fixing
lp:~blamar/nova/lp728587 updated on 2011-03-17
804. By Brian Lamar on 2011-03-17

Fixed pep8 violation.

Paul Voccio (pvo) wrote :

upgraded my pep8 to 0.6.1 and everything looked clean.

review: Approve
lp:~blamar/nova/lp728587 updated on 2011-03-17
805. By Brian Lamar on 2011-03-17

Pep8 error, oddly specific to pep8 v0.5 < x > v0.6

Rick Harris (rconradharris) wrote :

Great work, Brian.

Some femto-nits:

> 245 + self.remaining = math.floor((cap - water) / cap * val)

This might benefit from one more paren just to make it abundantly clear what's going on:

  self.remaining = math.floor(((cap - water) / cap) * val)

> 901 + return sum(filter(lambda x: x != None, results))

Might be clearer as:

  return sum(result for result in results if result)

As I understand it, the recommendation is to favor list-comprehensions rather than filter/map/reduce where possible.

Brian Lamar (blamar) wrote :

Thanks Rick. I hate to see the day that people don't quite get order of operations, but it's a great femto-nit (I'd even accept it as a pico-nit!).

As for the sum() line, I'd like to say your solution is more clear...but it is at least more efficient. I must not have thought about that line too long because now that I look at it:

---
return sum(filter(lambda x: x != None, results))

is the same as

return sum(filter(None, results))
---

That being said I understand list comprehensions (or actually in this case I think it ends up being a generator comprehension, which I just learned about!) are preferred so I've gone ahead and made these changes.

Thanks a lot for the review!

lp:~blamar/nova/lp728587 updated on 2011-03-17
806. By Brian Lamar on 2011-03-17

Better comment for fault. Improved readability of two small sections.

Rick Harris (rconradharris) wrote :

lgtm, thanks for the fixes.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'etc/api-paste.ini'
--- etc/api-paste.ini 2011-03-07 19:33:24 +0000
+++ etc/api-paste.ini 2011-03-17 20:28:35 +0000
@@ -79,7 +79,7 @@
79paste.filter_factory = nova.api.openstack.auth:AuthMiddleware.factory79paste.filter_factory = nova.api.openstack.auth:AuthMiddleware.factory
8080
81[filter:ratelimit]81[filter:ratelimit]
82paste.filter_factory = nova.api.openstack.ratelimiting:RateLimitingMiddleware.factory82paste.filter_factory = nova.api.openstack.limits:RateLimitingMiddleware.factory
8383
84[app:osapiapp]84[app:osapiapp]
85paste.app_factory = nova.api.openstack:APIRouter.factory85paste.app_factory = nova.api.openstack:APIRouter.factory
8686
=== modified file 'nova/api/openstack/__init__.py'
--- nova/api/openstack/__init__.py 2011-03-11 19:49:32 +0000
+++ nova/api/openstack/__init__.py 2011-03-17 20:28:35 +0000
@@ -33,6 +33,7 @@
33from nova.api.openstack import consoles33from nova.api.openstack import consoles
34from nova.api.openstack import flavors34from nova.api.openstack import flavors
35from nova.api.openstack import images35from nova.api.openstack import images
36from nova.api.openstack import limits
36from nova.api.openstack import servers37from nova.api.openstack import servers
37from nova.api.openstack import shared_ip_groups38from nova.api.openstack import shared_ip_groups
38from nova.api.openstack import users39from nova.api.openstack import users
@@ -114,12 +115,17 @@
114115
115 mapper.resource("image", "images", controller=images.Controller(),116 mapper.resource("image", "images", controller=images.Controller(),
116 collection={'detail': 'GET'})117 collection={'detail': 'GET'})
118
117 mapper.resource("flavor", "flavors", controller=flavors.Controller(),119 mapper.resource("flavor", "flavors", controller=flavors.Controller(),
118 collection={'detail': 'GET'})120 collection={'detail': 'GET'})
121
119 mapper.resource("shared_ip_group", "shared_ip_groups",122 mapper.resource("shared_ip_group", "shared_ip_groups",
120 collection={'detail': 'GET'},123 collection={'detail': 'GET'},
121 controller=shared_ip_groups.Controller())124 controller=shared_ip_groups.Controller())
122125
126 _limits = limits.LimitsController()
127 mapper.resource("limit", "limits", controller=_limits)
128
123 super(APIRouter, self).__init__(mapper)129 super(APIRouter, self).__init__(mapper)
124130
125131
126132
=== modified file 'nova/api/openstack/faults.py'
--- nova/api/openstack/faults.py 2011-03-09 20:08:11 +0000
+++ nova/api/openstack/faults.py 2011-03-17 20:28:35 +0000
@@ -61,3 +61,42 @@
61 content_type = req.best_match_content_type()61 content_type = req.best_match_content_type()
62 self.wrapped_exc.body = serializer.serialize(fault_data, content_type)62 self.wrapped_exc.body = serializer.serialize(fault_data, content_type)
63 return self.wrapped_exc63 return self.wrapped_exc
64
65
66class OverLimitFault(webob.exc.HTTPException):
67 """
68 Rate-limited request response.
69 """
70
71 _serialization_metadata = {
72 "application/xml": {
73 "attributes": {
74 "overLimitFault": "code",
75 },
76 },
77 }
78
79 def __init__(self, message, details, retry_time):
80 """
81 Initialize new `OverLimitFault` with relevant information.
82 """
83 self.wrapped_exc = webob.exc.HTTPForbidden()
84 self.content = {
85 "overLimitFault": {
86 "code": self.wrapped_exc.status_int,
87 "message": message,
88 "details": details,
89 },
90 }
91
92 @webob.dec.wsgify(RequestClass=wsgi.Request)
93 def __call__(self, request):
94 """
95 Return the wrapped exception with a serialized body conforming to our
96 error format.
97 """
98 serializer = wsgi.Serializer(self._serialization_metadata)
99 content_type = request.best_match_content_type()
100 content = serializer.serialize(self.content, content_type)
101 self.wrapped_exc.body = content
102 return self.wrapped_exc
64103
=== added file 'nova/api/openstack/limits.py'
--- nova/api/openstack/limits.py 1970-01-01 00:00:00 +0000
+++ nova/api/openstack/limits.py 2011-03-17 20:28:35 +0000
@@ -0,0 +1,358 @@
1# Copyright 2011 OpenStack LLC.
2# All Rights Reserved.
3#
4# Licensed under the Apache License, Version 2.0 (the "License"); you may
5# not use this file except in compliance with the License. You may obtain
6# a copy of the License at
7#
8# http://www.apache.org/licenses/LICENSE-2.0
9#
10# Unless required by applicable law or agreed to in writing, software
11# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
12# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
13# License for the specific language governing permissions and limitations
14# under the License.import datetime
15
16"""
17Module dedicated functions/classes dealing with rate limiting requests.
18"""
19
20import copy
21import httplib
22import json
23import math
24import re
25import time
26import urllib
27import webob.exc
28
29from collections import defaultdict
30
31from webob.dec import wsgify
32
33from nova import wsgi
34from nova.api.openstack import faults
35from nova.wsgi import Controller
36from nova.wsgi import Middleware
37
38
39# Convenience constants for the limits dictionary passed to Limiter().
40PER_SECOND = 1
41PER_MINUTE = 60
42PER_HOUR = 60 * 60
43PER_DAY = 60 * 60 * 24
44
45
46class LimitsController(Controller):
47 """
48 Controller for accessing limits in the OpenStack API.
49 """
50
51 _serialization_metadata = {
52 "application/xml": {
53 "attributes": {
54 "limit": ["verb", "URI", "regex", "value", "unit",
55 "resetTime", "remaining", "name"],
56 },
57 "plurals": {
58 "rate": "limit",
59 },
60 },
61 }
62
63 def index(self, req):
64 """
65 Return all global and rate limit information.
66 """
67 abs_limits = {}
68 rate_limits = req.environ.get("nova.limits", [])
69
70 return {
71 "limits": {
72 "rate": rate_limits,
73 "absolute": abs_limits,
74 },
75 }
76
77
78class Limit(object):
79 """
80 Stores information about a limit for HTTP requets.
81 """
82
83 UNITS = {
84 1: "SECOND",
85 60: "MINUTE",
86 60 * 60: "HOUR",
87 60 * 60 * 24: "DAY",
88 }
89
90 def __init__(self, verb, uri, regex, value, unit):
91 """
92 Initialize a new `Limit`.
93
94 @param verb: HTTP verb (POST, PUT, etc.)
95 @param uri: Human-readable URI
96 @param regex: Regular expression format for this limit
97 @param value: Integer number of requests which can be made
98 @param unit: Unit of measure for the value parameter
99 """
100 self.verb = verb
101 self.uri = uri
102 self.regex = regex
103 self.value = int(value)
104 self.unit = unit
105 self.unit_string = self.display_unit().lower()
106 self.remaining = int(value)
107
108 if value <= 0:
109 raise ValueError("Limit value must be > 0")
110
111 self.last_request = None
112 self.next_request = None
113
114 self.water_level = 0
115 self.capacity = self.unit
116 self.request_value = float(self.capacity) / float(self.value)
117 self.error_message = _("Only %(value)s %(verb)s request(s) can be "\
118 "made to %(uri)s every %(unit_string)s." % self.__dict__)
119
120 def __call__(self, verb, url):
121 """
122 Represents a call to this limit from a relevant request.
123
124 @param verb: string http verb (POST, GET, etc.)
125 @param url: string URL
126 """
127 if self.verb != verb or not re.match(self.regex, url):
128 return
129
130 now = self._get_time()
131
132 if self.last_request is None:
133 self.last_request = now
134
135 leak_value = now - self.last_request
136
137 self.water_level -= leak_value
138 self.water_level = max(self.water_level, 0)
139 self.water_level += self.request_value
140
141 difference = self.water_level - self.capacity
142
143 self.last_request = now
144
145 if difference > 0:
146 self.water_level -= self.request_value
147 self.next_request = now + difference
148 return difference
149
150 cap = self.capacity
151 water = self.water_level
152 val = self.value
153
154 self.remaining = math.floor(((cap - water) / cap) * val)
155 self.next_request = now
156
157 def _get_time(self):
158 """Retrieve the current time. Broken out for testability."""
159 return time.time()
160
161 def display_unit(self):
162 """Display the string name of the unit."""
163 return self.UNITS.get(self.unit, "UNKNOWN")
164
165 def display(self):
166 """Return a useful representation of this class."""
167 return {
168 "verb": self.verb,
169 "URI": self.uri,
170 "regex": self.regex,
171 "value": self.value,
172 "remaining": int(self.remaining),
173 "unit": self.display_unit(),
174 "resetTime": int(self.next_request or self._get_time()),
175 }
176
177# "Limit" format is a dictionary with the HTTP verb, human-readable URI,
178# a regular-expression to match, value and unit of measure (PER_DAY, etc.)
179
180DEFAULT_LIMITS = [
181 Limit("POST", "*", ".*", 10, PER_MINUTE),
182 Limit("POST", "*/servers", "^/servers", 50, PER_DAY),
183 Limit("PUT", "*", ".*", 10, PER_MINUTE),
184 Limit("GET", "*changes-since*", ".*changes-since.*", 3, PER_MINUTE),
185 Limit("DELETE", "*", ".*", 100, PER_MINUTE),
186]
187
188
189class RateLimitingMiddleware(Middleware):
190 """
191 Rate-limits requests passing through this middleware. All limit information
192 is stored in memory for this implementation.
193 """
194
195 def __init__(self, application, limits=None):
196 """
197 Initialize new `RateLimitingMiddleware`, which wraps the given WSGI
198 application and sets up the given limits.
199
200 @param application: WSGI application to wrap
201 @param limits: List of dictionaries describing limits
202 """
203 Middleware.__init__(self, application)
204 self._limiter = Limiter(limits or DEFAULT_LIMITS)
205
206 @wsgify(RequestClass=wsgi.Request)
207 def __call__(self, req):
208 """
209 Represents a single call through this middleware. We should record the
210 request if we have a limit relevant to it. If no limit is relevant to
211 the request, ignore it.
212
213 If the request should be rate limited, return a fault telling the user
214 they are over the limit and need to retry later.
215 """
216 verb = req.method
217 url = req.url
218 context = req.environ.get("nova.context")
219
220 if context:
221 username = context.user_id
222 else:
223 username = None
224
225 delay, error = self._limiter.check_for_delay(verb, url, username)
226
227 if delay:
228 msg = _("This request was rate-limited.")
229 retry = time.time() + delay
230 return faults.OverLimitFault(msg, error, retry)
231
232 req.environ["nova.limits"] = self._limiter.get_limits(username)
233
234 return self.application
235
236
237class Limiter(object):
238 """
239 Rate-limit checking class which handles limits in memory.
240 """
241
242 def __init__(self, limits):
243 """
244 Initialize the new `Limiter`.
245
246 @param limits: List of `Limit` objects
247 """
248 self.limits = copy.deepcopy(limits)
249 self.levels = defaultdict(lambda: copy.deepcopy(limits))
250
251 def get_limits(self, username=None):
252 """
253 Return the limits for a given user.
254 """
255 return [limit.display() for limit in self.levels[username]]
256
257 def check_for_delay(self, verb, url, username=None):
258 """
259 Check the given verb/user/user triplet for limit.
260
261 @return: Tuple of delay (in seconds) and error message (or None, None)
262 """
263 delays = []
264
265 for limit in self.levels[username]:
266 delay = limit(verb, url)
267 if delay:
268 delays.append((delay, limit.error_message))
269
270 if delays:
271 delays.sort()
272 return delays[0]
273
274 return None, None
275
276
277class WsgiLimiter(object):
278 """
279 Rate-limit checking from a WSGI application. Uses an in-memory `Limiter`.
280
281 To use:
282 POST /<username> with JSON data such as:
283 {
284 "verb" : GET,
285 "path" : "/servers"
286 }
287
288 and receive a 204 No Content, or a 403 Forbidden with an X-Wait-Seconds
289 header containing the number of seconds to wait before the action would
290 succeed.
291 """
292
293 def __init__(self, limits=None):
294 """
295 Initialize the new `WsgiLimiter`.
296
297 @param limits: List of `Limit` objects
298 """
299 self._limiter = Limiter(limits or DEFAULT_LIMITS)
300
301 @wsgify(RequestClass=wsgi.Request)
302 def __call__(self, request):
303 """
304 Handles a call to this application. Returns 204 if the request is
305 acceptable to the limiter, else a 403 is returned with a relevant
306 header indicating when the request *will* succeed.
307 """
308 if request.method != "POST":
309 raise webob.exc.HTTPMethodNotAllowed()
310
311 try:
312 info = dict(json.loads(request.body))
313 except ValueError:
314 raise webob.exc.HTTPBadRequest()
315
316 username = request.path_info_pop()
317 verb = info.get("verb")
318 path = info.get("path")
319
320 delay, error = self._limiter.check_for_delay(verb, path, username)
321
322 if delay:
323 headers = {"X-Wait-Seconds": "%.2f" % delay}
324 return webob.exc.HTTPForbidden(headers=headers, explanation=error)
325 else:
326 return webob.exc.HTTPNoContent()
327
328
329class WsgiLimiterProxy(object):
330 """
331 Rate-limit requests based on answers from a remote source.
332 """
333
334 def __init__(self, limiter_address):
335 """
336 Initialize the new `WsgiLimiterProxy`.
337
338 @param limiter_address: IP/port combination of where to request limit
339 """
340 self.limiter_address = limiter_address
341
342 def check_for_delay(self, verb, path, username=None):
343 body = json.dumps({"verb": verb, "path": path})
344 headers = {"Content-Type": "application/json"}
345
346 conn = httplib.HTTPConnection(self.limiter_address)
347
348 if username:
349 conn.request("POST", "/%s" % (username), body, headers)
350 else:
351 conn.request("POST", "/", body, headers)
352
353 resp = conn.getresponse()
354
355 if 200 >= resp.status < 300:
356 return None, None
357
358 return resp.getheader("X-Wait-Seconds"), resp.read() or None
0359
=== modified file 'nova/tests/api/openstack/__init__.py'
--- nova/tests/api/openstack/__init__.py 2011-02-23 19:56:37 +0000
+++ nova/tests/api/openstack/__init__.py 2011-03-17 20:28:35 +0000
@@ -20,7 +20,7 @@
2020
21from nova import context21from nova import context
22from nova import flags22from nova import flags
23from nova.api.openstack.ratelimiting import RateLimitingMiddleware23from nova.api.openstack.limits import RateLimitingMiddleware
24from nova.api.openstack.common import limited24from nova.api.openstack.common import limited
25from nova.tests.api.openstack import fakes25from nova.tests.api.openstack import fakes
26from webob import Request26from webob import Request
2727
=== modified file 'nova/tests/api/openstack/fakes.py'
--- nova/tests/api/openstack/fakes.py 2011-03-16 19:16:16 +0000
+++ nova/tests/api/openstack/fakes.py 2011-03-17 20:28:35 +0000
@@ -34,7 +34,7 @@
34import nova.api.openstack.auth34import nova.api.openstack.auth
35from nova.api import openstack35from nova.api import openstack
36from nova.api.openstack import auth36from nova.api.openstack import auth
37from nova.api.openstack import ratelimiting37from nova.api.openstack import limits
38from nova.auth.manager import User, Project38from nova.auth.manager import User, Project
39from nova.image import glance39from nova.image import glance
40from nova.image import local40from nova.image import local
@@ -77,7 +77,7 @@
77 inner_application = openstack.APIRouter()77 inner_application = openstack.APIRouter()
78 mapper = urlmap.URLMap()78 mapper = urlmap.URLMap()
79 api = openstack.FaultWrapper(auth.AuthMiddleware(79 api = openstack.FaultWrapper(auth.AuthMiddleware(
80 ratelimiting.RateLimitingMiddleware(inner_application)))80 limits.RateLimitingMiddleware(inner_application)))
81 mapper['/v1.0'] = api81 mapper['/v1.0'] = api
82 mapper['/'] = openstack.FaultWrapper(openstack.Versions())82 mapper['/'] = openstack.FaultWrapper(openstack.Versions())
83 return mapper83 return mapper
@@ -115,13 +115,13 @@
115115
116def stub_out_rate_limiting(stubs):116def stub_out_rate_limiting(stubs):
117 def fake_rate_init(self, app):117 def fake_rate_init(self, app):
118 super(ratelimiting.RateLimitingMiddleware, self).__init__(app)118 super(limits.RateLimitingMiddleware, self).__init__(app)
119 self.application = app119 self.application = app
120120
121 stubs.Set(nova.api.openstack.ratelimiting.RateLimitingMiddleware,121 stubs.Set(nova.api.openstack.limits.RateLimitingMiddleware,
122 '__init__', fake_rate_init)122 '__init__', fake_rate_init)
123123
124 stubs.Set(nova.api.openstack.ratelimiting.RateLimitingMiddleware,124 stubs.Set(nova.api.openstack.limits.RateLimitingMiddleware,
125 '__call__', fake_wsgi)125 '__call__', fake_wsgi)
126126
127127
128128
=== modified file 'nova/tests/api/openstack/test_adminapi.py'
--- nova/tests/api/openstack/test_adminapi.py 2011-03-08 17:18:13 +0000
+++ nova/tests/api/openstack/test_adminapi.py 2011-03-17 20:28:35 +0000
@@ -23,7 +23,6 @@
23from nova import flags23from nova import flags
24from nova import test24from nova import test
25from nova.api import openstack25from nova.api import openstack
26from nova.api.openstack import ratelimiting
27from nova.api.openstack import auth26from nova.api.openstack import auth
28from nova.tests.api.openstack import fakes27from nova.tests.api.openstack import fakes
2928
3029
=== renamed file 'nova/tests/api/openstack/test_ratelimiting.py' => 'nova/tests/api/openstack/test_limits.py'
--- nova/tests/api/openstack/test_ratelimiting.py 2011-02-23 19:56:37 +0000
+++ nova/tests/api/openstack/test_limits.py 2011-03-17 20:28:35 +0000
@@ -1,178 +1,520 @@
1# Copyright 2011 OpenStack LLC.
2# All Rights Reserved.
3#
4# Licensed under the Apache License, Version 2.0 (the "License"); you may
5# not use this file except in compliance with the License. You may obtain
6# a copy of the License at
7#
8# http://www.apache.org/licenses/LICENSE-2.0
9#
10# Unless required by applicable law or agreed to in writing, software
11# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
12# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
13# License for the specific language governing permissions and limitations
14# under the License.
15
16"""
17Tests dealing with HTTP rate-limiting.
18"""
19
1import httplib20import httplib
21import json
2import StringIO22import StringIO
23import stubout
3import time24import time
25import unittest
4import webob26import webob
527
6from nova import test28from xml.dom.minidom import parseString
7import nova.api.openstack.ratelimiting as ratelimiting29
830from nova.api.openstack import limits
931from nova.api.openstack.limits import Limit
10class LimiterTest(test.TestCase):32
1133
12 def setUp(self):34TEST_LIMITS = [
13 super(LimiterTest, self).setUp()35 Limit("GET", "/delayed", "^/delayed", 1, limits.PER_MINUTE),
14 self.limits = {36 Limit("POST", "*", ".*", 7, limits.PER_MINUTE),
15 'a': (5, ratelimiting.PER_SECOND),37 Limit("POST", "/servers", "^/servers", 3, limits.PER_MINUTE),
16 'b': (5, ratelimiting.PER_MINUTE),38 Limit("PUT", "*", "", 10, limits.PER_MINUTE),
17 'c': (5, ratelimiting.PER_HOUR),39 Limit("PUT", "/servers", "^/servers", 5, limits.PER_MINUTE),
18 'd': (1, ratelimiting.PER_SECOND),40]
19 'e': (100, ratelimiting.PER_SECOND)}41
20 self.rl = ratelimiting.Limiter(self.limits)42
2143class BaseLimitTestSuite(unittest.TestCase):
22 def exhaust(self, action, times_until_exhausted, **kwargs):44 """Base test suite which provides relevant stubs and time abstraction."""
23 for i in range(times_until_exhausted):45
24 when = self.rl.perform(action, **kwargs)46 def setUp(self):
25 self.assertEqual(when, None)47 """Run before each test."""
26 num, period = self.limits[action]48 self.time = 0.0
27 delay = period * 1.0 / num49 self.stubs = stubout.StubOutForTesting()
28 # Verify that we are now thoroughly delayed50 self.stubs.Set(limits.Limit, "_get_time", self._get_time)
29 for i in range(10):51
30 when = self.rl.perform(action, **kwargs)52 def tearDown(self):
31 self.assertAlmostEqual(when, delay, 2)53 """Run after each test."""
3254 self.stubs.UnsetAll()
33 def test_second(self):55
34 self.exhaust('a', 5)56 def _get_time(self):
35 time.sleep(0.2)57 """Return the "time" according to this test suite."""
36 self.exhaust('a', 1)58 return self.time
37 time.sleep(1)59
38 self.exhaust('a', 5)60
3961class LimitsControllerTest(BaseLimitTestSuite):
40 def test_minute(self):62 """
41 self.exhaust('b', 5)63 Tests for `limits.LimitsController` class.
4264 """
43 def test_one_per_period(self):65
44 def allow_once_and_deny_once():66 def setUp(self):
45 when = self.rl.perform('d')67 """Run before each test."""
46 self.assertEqual(when, None)68 BaseLimitTestSuite.setUp(self)
47 when = self.rl.perform('d')69 self.controller = limits.LimitsController()
48 self.assertAlmostEqual(when, 1, 2)70
49 return when71 def _get_index_request(self, accept_header="application/json"):
50 time.sleep(allow_once_and_deny_once())72 """Helper to set routing arguments."""
51 time.sleep(allow_once_and_deny_once())73 request = webob.Request.blank("/")
52 allow_once_and_deny_once()74 request.accept = accept_header
5375 request.environ["wsgiorg.routing_args"] = (None, {
54 def test_we_can_go_indefinitely_if_we_spread_out_requests(self):76 "action": "index",
55 for i in range(200):77 "controller": "",
56 when = self.rl.perform('e')78 })
57 self.assertEqual(when, None)79 return request
58 time.sleep(0.01)80
5981 def _populate_limits(self, request):
60 def test_users_get_separate_buckets(self):82 """Put limit info into a request."""
61 self.exhaust('c', 5, username='alice')83 _limits = [
62 self.exhaust('c', 5, username='bob')84 Limit("GET", "*", ".*", 10, 60).display(),
63 self.exhaust('c', 5, username='chuck')85 Limit("POST", "*", ".*", 5, 60 * 60).display(),
64 self.exhaust('c', 0, username='chuck')86 ]
65 self.exhaust('c', 0, username='bob')87 request.environ["nova.limits"] = _limits
66 self.exhaust('c', 0, username='alice')88 return request
6789
6890 def test_empty_index_json(self):
69class FakeLimiter(object):91 """Test getting empty limit details in JSON."""
70 """Fake Limiter class that you can tell how to behave."""92 request = self._get_index_request()
7193 response = request.get_response(self.controller)
72 def __init__(self, test):94 expected = {
73 self._action = self._username = self._delay = None95 "limits": {
74 self.test = test96 "rate": [],
7597 "absolute": {},
76 def mock(self, action, username, delay):98 },
77 self._action = action99 }
78 self._username = username100 body = json.loads(response.body)
79 self._delay = delay101 self.assertEqual(expected, body)
80102
81 def perform(self, action, username):103 def test_index_json(self):
82 self.test.assertEqual(action, self._action)104 """Test getting limit details in JSON."""
83 self.test.assertEqual(username, self._username)105 request = self._get_index_request()
84 return self._delay106 request = self._populate_limits(request)
85107 response = request.get_response(self.controller)
86108 expected = {
87class WSGIAppTest(test.TestCase):109 "limits": {
88110 "rate": [{
89 def setUp(self):111 "regex": ".*",
90 super(WSGIAppTest, self).setUp()112 "resetTime": 0,
91 self.limiter = FakeLimiter(self)113 "URI": "*",
92 self.app = ratelimiting.WSGIApp(self.limiter)114 "value": 10,
93115 "verb": "GET",
94 def test_invalid_methods(self):116 "remaining": 10,
95 requests = []117 "unit": "MINUTE",
96 for method in ['GET', 'PUT', 'DELETE']:118 },
97 req = webob.Request.blank('/limits/michael/breakdance',119 {
98 dict(REQUEST_METHOD=method))120 "regex": ".*",
99 requests.append(req)121 "resetTime": 0,
100 for req in requests:122 "URI": "*",
101 self.assertEqual(req.get_response(self.app).status_int, 405)123 "value": 5,
102124 "verb": "POST",
103 def test_invalid_urls(self):125 "remaining": 5,
104 requests = []126 "unit": "HOUR",
105 for prefix in ['limit', '', 'limiter2', 'limiter/limits', 'limiter/1']:127 }],
106 req = webob.Request.blank('/%s/michael/breakdance' % prefix,128 "absolute": {},
107 dict(REQUEST_METHOD='POST'))129 },
108 requests.append(req)130 }
109 for req in requests:131 body = json.loads(response.body)
110 self.assertEqual(req.get_response(self.app).status_int, 404)132 self.assertEqual(expected, body)
111133
112 def verify(self, url, username, action, delay=None):134 def test_empty_index_xml(self):
135 """Test getting limit details in XML."""
136 request = self._get_index_request("application/xml")
137 response = request.get_response(self.controller)
138
139 expected = "<limits><rate/><absolute/></limits>"
140 body = response.body.replace("\n", "").replace(" ", "")
141
142 self.assertEqual(expected, body)
143
144 def test_index_xml(self):
145 """Test getting limit details in XML."""
146 request = self._get_index_request("application/xml")
147 request = self._populate_limits(request)
148 response = request.get_response(self.controller)
149
150 expected = parseString("""
151 <limits>
152 <rate>
153 <limit URI="*" regex=".*" remaining="10" resetTime="0"
154 unit="MINUTE" value="10" verb="GET"/>
155 <limit URI="*" regex=".*" remaining="5" resetTime="0"
156 unit="HOUR" value="5" verb="POST"/>
157 </rate>
158 <absolute/>
159 </limits>
160 """.replace(" ", ""))
161 body = parseString(response.body.replace(" ", ""))
162
163 self.assertEqual(expected.toxml(), body.toxml())
164
165
166class LimitMiddlewareTest(BaseLimitTestSuite):
167 """
168 Tests for the `limits.RateLimitingMiddleware` class.
169 """
170
171 @webob.dec.wsgify
172 def _empty_app(self, request):
173 """Do-nothing WSGI app."""
174 pass
175
176 def setUp(self):
177 """Prepare middleware for use through fake WSGI app."""
178 BaseLimitTestSuite.setUp(self)
179 _limits = [
180 Limit("GET", "*", ".*", 1, 60),
181 ]
182 self.app = limits.RateLimitingMiddleware(self._empty_app, _limits)
183
184 def test_good_request(self):
185 """Test successful GET request through middleware."""
186 request = webob.Request.blank("/")
187 response = request.get_response(self.app)
188 self.assertEqual(200, response.status_int)
189
190 def test_limited_request_json(self):
191 """Test a rate-limited (403) GET request through middleware."""
192 request = webob.Request.blank("/")
193 response = request.get_response(self.app)
194 self.assertEqual(200, response.status_int)
195
196 request = webob.Request.blank("/")
197 response = request.get_response(self.app)
198 self.assertEqual(response.status_int, 403)
199
200 body = json.loads(response.body)
201 expected = "Only 1 GET request(s) can be made to * every minute."
202 value = body["overLimitFault"]["details"].strip()
203 self.assertEqual(value, expected)
204
205 def test_limited_request_xml(self):
206 """Test a rate-limited (403) response as XML"""
207 request = webob.Request.blank("/")
208 response = request.get_response(self.app)
209 self.assertEqual(200, response.status_int)
210
211 request = webob.Request.blank("/")
212 request.accept = "application/xml"
213 response = request.get_response(self.app)
214 self.assertEqual(response.status_int, 403)
215
216 root = parseString(response.body).childNodes[0]
217 expected = "Only 1 GET request(s) can be made to * every minute."
218
219 details = root.getElementsByTagName("details")
220 self.assertEqual(details.length, 1)
221
222 value = details.item(0).firstChild.data.strip()
223 self.assertEqual(value, expected)
224
225
226class LimitTest(BaseLimitTestSuite):
227 """
228 Tests for the `limits.Limit` class.
229 """
230
231 def test_GET_no_delay(self):
232 """Test a limit handles 1 GET per second."""
233 limit = Limit("GET", "*", ".*", 1, 1)
234 delay = limit("GET", "/anything")
235 self.assertEqual(None, delay)
236 self.assertEqual(0, limit.next_request)
237 self.assertEqual(0, limit.last_request)
238
239 def test_GET_delay(self):
240 """Test two calls to 1 GET per second limit."""
241 limit = Limit("GET", "*", ".*", 1, 1)
242 delay = limit("GET", "/anything")
243 self.assertEqual(None, delay)
244
245 delay = limit("GET", "/anything")
246 self.assertEqual(1, delay)
247 self.assertEqual(1, limit.next_request)
248 self.assertEqual(0, limit.last_request)
249
250 self.time += 4
251
252 delay = limit("GET", "/anything")
253 self.assertEqual(None, delay)
254 self.assertEqual(4, limit.next_request)
255 self.assertEqual(4, limit.last_request)
256
257
258class LimiterTest(BaseLimitTestSuite):
259 """
260 Tests for the in-memory `limits.Limiter` class.
261 """
262
263 def setUp(self):
264 """Run before each test."""
265 BaseLimitTestSuite.setUp(self)
266 self.limiter = limits.Limiter(TEST_LIMITS)
267
268 def _check(self, num, verb, url, username=None):
269 """Check and yield results from checks."""
270 for x in xrange(num):
271 yield self.limiter.check_for_delay(verb, url, username)[0]
272
273 def _check_sum(self, num, verb, url, username=None):
274 """Check and sum results from checks."""
275 results = self._check(num, verb, url, username)
276 return sum(item for item in results if item)
277
278 def test_no_delay_GET(self):
279 """
280 Simple test to ensure no delay on a single call for a limit verb we
281 didn"t set.
282 """
283 delay = self.limiter.check_for_delay("GET", "/anything")
284 self.assertEqual(delay, (None, None))
285
286 def test_no_delay_PUT(self):
287 """
288 Simple test to ensure no delay on a single call for a known limit.
289 """
290 delay = self.limiter.check_for_delay("PUT", "/anything")
291 self.assertEqual(delay, (None, None))
292
293 def test_delay_PUT(self):
294 """
295 Ensure the 11th PUT will result in a delay of 6.0 seconds until
296 the next request will be granced.
297 """
298 expected = [None] * 10 + [6.0]
299 results = list(self._check(11, "PUT", "/anything"))
300
301 self.assertEqual(expected, results)
302
303 def test_delay_POST(self):
304 """
305 Ensure the 8th POST will result in a delay of 6.0 seconds until
306 the next request will be granced.
307 """
308 expected = [None] * 7
309 results = list(self._check(7, "POST", "/anything"))
310 self.assertEqual(expected, results)
311
312 expected = 60.0 / 7.0
313 results = self._check_sum(1, "POST", "/anything")
314 self.failUnlessAlmostEqual(expected, results, 8)
315
316 def test_delay_GET(self):
317 """
318 Ensure the 11th GET will result in NO delay.
319 """
320 expected = [None] * 11
321 results = list(self._check(11, "GET", "/anything"))
322
323 self.assertEqual(expected, results)
324
325 def test_delay_PUT_servers(self):
326 """
327 Ensure PUT on /servers limits at 5 requests, and PUT elsewhere is still
328 OK after 5 requests...but then after 11 total requests, PUT limiting
329 kicks in.
330 """
331 # First 6 requests on PUT /servers
332 expected = [None] * 5 + [12.0]
333 results = list(self._check(6, "PUT", "/servers"))
334 self.assertEqual(expected, results)
335
336 # Next 5 request on PUT /anything
337 expected = [None] * 4 + [6.0]
338 results = list(self._check(5, "PUT", "/anything"))
339 self.assertEqual(expected, results)
340
341 def test_delay_PUT_wait(self):
342 """
343 Ensure after hitting the limit and then waiting for the correct
344 amount of time, the limit will be lifted.
345 """
346 expected = [None] * 10 + [6.0]
347 results = list(self._check(11, "PUT", "/anything"))
348 self.assertEqual(expected, results)
349
350 # Advance time
351 self.time += 6.0
352
353 expected = [None, 6.0]
354 results = list(self._check(2, "PUT", "/anything"))
355 self.assertEqual(expected, results)
356
357 def test_multiple_delays(self):
358 """
359 Ensure multiple requests still get a delay.
360 """
361 expected = [None] * 10 + [6.0] * 10
362 results = list(self._check(20, "PUT", "/anything"))
363 self.assertEqual(expected, results)
364
365 self.time += 1.0
366
367 expected = [5.0] * 10
368 results = list(self._check(10, "PUT", "/anything"))
369 self.assertEqual(expected, results)
370
371 def test_multiple_users(self):
372 """
373 Tests involving multiple users.
374 """
375 # User1
376 expected = [None] * 10 + [6.0] * 10
377 results = list(self._check(20, "PUT", "/anything", "user1"))
378 self.assertEqual(expected, results)
379
380 # User2
381 expected = [None] * 10 + [6.0] * 5
382 results = list(self._check(15, "PUT", "/anything", "user2"))
383 self.assertEqual(expected, results)
384
385 self.time += 1.0
386
387 # User1 again
388 expected = [5.0] * 10
389 results = list(self._check(10, "PUT", "/anything", "user1"))
390 self.assertEqual(expected, results)
391
392 self.time += 1.0
393
394 # User1 again
395 expected = [4.0] * 5
396 results = list(self._check(5, "PUT", "/anything", "user2"))
397 self.assertEqual(expected, results)
398
399
400class WsgiLimiterTest(BaseLimitTestSuite):
401 """
402 Tests for `limits.WsgiLimiter` class.
403 """
404
405 def setUp(self):
406 """Run before each test."""
407 BaseLimitTestSuite.setUp(self)
408 self.app = limits.WsgiLimiter(TEST_LIMITS)
409
410 def _request_data(self, verb, path):
411 """Get data decribing a limit request verb/path."""
412 return json.dumps({"verb": verb, "path": path})
413
414 def _request(self, verb, url, username=None):
113 """Make sure that POSTing to the given url causes the given username415 """Make sure that POSTing to the given url causes the given username
114 to perform the given action. Make the internal rate limiter return416 to perform the given action. Make the internal rate limiter return
115 delay and make sure that the WSGI app returns the correct response.417 delay and make sure that the WSGI app returns the correct response.
116 """418 """
117 req = webob.Request.blank(url, dict(REQUEST_METHOD='POST'))419 if username:
118 self.limiter.mock(action, username, delay)420 request = webob.Request.blank("/%s" % username)
119 resp = req.get_response(self.app)
120 if not delay:
121 self.assertEqual(resp.status_int, 200)
122 else:421 else:
123 self.assertEqual(resp.status_int, 403)422 request = webob.Request.blank("/")
124 self.assertEqual(resp.headers['X-Wait-Seconds'], "%.2f" % delay)423
125424 request.method = "POST"
126 def test_good_urls(self):425 request.body = self._request_data(verb, url)
127 self.verify('/limiter/michael/hoot', 'michael', 'hoot')426 response = request.get_response(self.app)
427
428 if "X-Wait-Seconds" in response.headers:
429 self.assertEqual(response.status_int, 403)
430 return response.headers["X-Wait-Seconds"]
431
432 self.assertEqual(response.status_int, 204)
433
434 def test_invalid_methods(self):
435 """Only POSTs should work."""
436 requests = []
437 for method in ["GET", "PUT", "DELETE", "HEAD", "OPTIONS"]:
438 request = webob.Request.blank("/")
439 request.body = self._request_data("GET", "/something")
440 response = request.get_response(self.app)
441 self.assertEqual(response.status_int, 405)
442
443 def test_good_url(self):
444 delay = self._request("GET", "/something")
445 self.assertEqual(delay, None)
128446
129 def test_escaping(self):447 def test_escaping(self):
130 self.verify('/limiter/michael/jump%20up', 'michael', 'jump up')448 delay = self._request("GET", "/something/jump%20up")
449 self.assertEqual(delay, None)
131450
132 def test_response_to_delays(self):451 def test_response_to_delays(self):
133 self.verify('/limiter/michael/hoot', 'michael', 'hoot', 1)452 delay = self._request("GET", "/delayed")
134 self.verify('/limiter/michael/hoot', 'michael', 'hoot', 1.56)453 self.assertEqual(delay, None)
135 self.verify('/limiter/michael/hoot', 'michael', 'hoot', 1000)454
455 delay = self._request("GET", "/delayed")
456 self.assertEqual(delay, '60.00')
457
458 def test_response_to_delays_usernames(self):
459 delay = self._request("GET", "/delayed", "user1")
460 self.assertEqual(delay, None)
461
462 delay = self._request("GET", "/delayed", "user2")
463 self.assertEqual(delay, None)
464
465 delay = self._request("GET", "/delayed", "user1")
466 self.assertEqual(delay, '60.00')
467
468 delay = self._request("GET", "/delayed", "user2")
469 self.assertEqual(delay, '60.00')
136470
137471
138class FakeHttplibSocket(object):472class FakeHttplibSocket(object):
139 """a fake socket implementation for httplib.HTTPResponse, trivial"""473 """
474 Fake `httplib.HTTPResponse` replacement.
475 """
140476
141 def __init__(self, response_string):477 def __init__(self, response_string):
478 """Initialize new `FakeHttplibSocket`."""
142 self._buffer = StringIO.StringIO(response_string)479 self._buffer = StringIO.StringIO(response_string)
143480
144 def makefile(self, _mode, _other):481 def makefile(self, _mode, _other):
145 """Returns the socket's internal buffer"""482 """Returns the socket's internal buffer."""
146 return self._buffer483 return self._buffer
147484
148485
149class FakeHttplibConnection(object):486class FakeHttplibConnection(object):
150 """A fake httplib.HTTPConnection487 """
488 Fake `httplib.HTTPConnection`.
489 """
151490
152 Requests made via this connection actually get translated and routed into491 def __init__(self, app, host):
153 our WSGI app, we then wait for the response and turn it back into492 """
154 an httplib.HTTPResponse.493 Initialize `FakeHttplibConnection`.
155 """494 """
156 def __init__(self, app, host, is_secure=False):
157 self.app = app495 self.app = app
158 self.host = host496 self.host = host
159497
160 def request(self, method, path, data='', headers={}):498 def request(self, method, path, body="", headers={}):
499 """
500 Requests made via this connection actually get translated and routed
501 into our WSGI app, we then wait for the response and turn it back into
502 an `httplib.HTTPResponse`.
503 """
161 req = webob.Request.blank(path)504 req = webob.Request.blank(path)
162 req.method = method505 req.method = method
163 req.body = data
164 req.headers = headers506 req.headers = headers
165 req.host = self.host507 req.host = self.host
166 # Call the WSGI app, get the HTTP response508 req.body = body
509
167 resp = str(req.get_response(self.app))510 resp = str(req.get_response(self.app))
168 # For some reason, the response doesn't have "HTTP/1.0 " prepended; I
169 # guess that's a function the web server usually provides.
170 resp = "HTTP/1.0 %s" % resp511 resp = "HTTP/1.0 %s" % resp
171 sock = FakeHttplibSocket(resp)512 sock = FakeHttplibSocket(resp)
172 self.http_response = httplib.HTTPResponse(sock)513 self.http_response = httplib.HTTPResponse(sock)
173 self.http_response.begin()514 self.http_response.begin()
174515
175 def getresponse(self):516 def getresponse(self):
517 """Return our generated response from the request."""
176 return self.http_response518 return self.http_response
177519
178520
@@ -208,36 +550,35 @@
208 httplib.HTTPConnection = HTTPConnectionDecorator(httplib.HTTPConnection)550 httplib.HTTPConnection = HTTPConnectionDecorator(httplib.HTTPConnection)
209551
210552
211class WSGIAppProxyTest(test.TestCase):553class WsgiLimiterProxyTest(BaseLimitTestSuite):
554 """
555 Tests for the `limits.WsgiLimiterProxy` class.
556 """
212557
213 def setUp(self):558 def setUp(self):
214 """Our WSGIAppProxy is going to call across an HTTPConnection to a559 """
215 WSGIApp running a limiter. The proxy will send input, and the proxy560 Do some nifty HTTP/WSGI magic which allows for WSGI to be called
216 should receive that same input, pass it to the limiter who gives a561 directly by something like the `httplib` library.
217 result, and send the expected result back.562 """
218563 BaseLimitTestSuite.setUp(self)
219 The HTTPConnection isn't real -- it's monkeypatched to point straight564 self.app = limits.WsgiLimiter(TEST_LIMITS)
220 at the WSGIApp. And the limiter isn't real -- it's a fake that565 wire_HTTPConnection_to_WSGI("169.254.0.1:80", self.app)
221 behaves the way we tell it to.566 self.proxy = limits.WsgiLimiterProxy("169.254.0.1:80")
222 """
223 super(WSGIAppProxyTest, self).setUp()
224 self.limiter = FakeLimiter(self)
225 app = ratelimiting.WSGIApp(self.limiter)
226 wire_HTTPConnection_to_WSGI('100.100.100.100:80', app)
227 self.proxy = ratelimiting.WSGIAppProxy('100.100.100.100:80')
228567
229 def test_200(self):568 def test_200(self):
230 self.limiter.mock('conquer', 'caesar', None)569 """Successful request test."""
231 when = self.proxy.perform('conquer', 'caesar')570 delay = self.proxy.check_for_delay("GET", "/anything")
232 self.assertEqual(when, None)571 self.assertEqual(delay, (None, None))
233572
234 def test_403(self):573 def test_403(self):
235 self.limiter.mock('grumble', 'proletariat', 1.5)574 """Forbidden request test."""
236 when = self.proxy.perform('grumble', 'proletariat')575 delay = self.proxy.check_for_delay("GET", "/delayed")
237 self.assertEqual(when, 1.5)576 self.assertEqual(delay, (None, None))
238577
239 def test_failure(self):578 delay, error = self.proxy.check_for_delay("GET", "/delayed")
240 def shouldRaise():579 error = error.strip()
241 self.limiter.mock('murder', 'brutus', None)580
242 self.proxy.perform('stab', 'brutus')581 expected = ("60.00", "403 Forbidden\n\nOnly 1 GET request(s) can be "\
243 self.assertRaises(AssertionError, shouldRaise)582 "made to /delayed every minute.")
583
584 self.assertEqual((delay, error), expected)