Merge lp:~blamar/nova/lp728587 into lp:~hudson-openstack/nova/trunk

Proposed by Brian Lamar on 2011-03-16
Status: Merged
Approved by: Rick Harris on 2011-03-18
Approved revision: 806
Merged at revision: 830
Proposed branch: lp:~blamar/nova/lp728587
Merge into: lp:~hudson-openstack/nova/trunk
Diff against target: 1238 lines (+912/-169)
8 files modified
etc/api-paste.ini (+1/-1)
nova/api/openstack/__init__.py (+6/-0)
nova/api/openstack/faults.py (+39/-0)
nova/api/openstack/limits.py (+358/-0)
nova/tests/api/openstack/__init__.py (+1/-1)
nova/tests/api/openstack/fakes.py (+5/-5)
nova/tests/api/openstack/test_adminapi.py (+0/-1)
nova/tests/api/openstack/test_limits.py (+502/-161)
To merge this branch: bzr merge lp:~blamar/nova/lp728587
Reviewer Review Type Date Requested Status
Rick Harris (community) Approve on 2011-03-18
Paul Voccio (community) 2011-03-16 Approve on 2011-03-17
Titan 2011-03-16 Pending
Mark Washenberger 2011-03-16 Pending
Review via email: mp+53554@code.launchpad.net

Commit message

Re-implementation (or just implementation in many cases) of Limits in the OpenStack API. Limits is now available through /limits and the concept of a limit has been extended to include arbitrary regex / http verb combinations along with correct XML/JSON serialization. Tests included.

Description of the change

The addition of a /limits resource was a larger job than I had first thought. The original rate-limiting middleware and supporting classes were being tested through `time.sleep` calls instead of actually unit-testing the logic.

I've replaced most of the classes with something that is more testable and much more compatible with the output we're generating for the 1.1 API.

Overall we should have much more test coverage than with the previous limits implementation.

To post a comment you must log in.
lp:~blamar/nova/lp728587 updated on 2011-03-16
800. By Brian Lamar on 2011-03-16

Removed VIM specific stuff and changed copyright from 2010 to 2011.

801. By Brian Lamar on 2011-03-16

Added tests back for RateLimitingMiddleware which now throw correctly serialized
errors with correct error codes.

Removed some error printing, and simplified some other parts of the code with
suggestions from teammates.

802. By Brian Lamar on 2011-03-16

Added i18n to error message.

Jay Pipes (jaypipes) wrote :

Hi Brian! Just FYI, make sure you select "nova-core" as the reviewer when you do a merge request. You can add additional reviewers with the "Request another review" link on the merge proposal page (and I encourage you to do so, like you've done!)

The reason to do this is because nova-core sends email notifications to the dozen or so core committers, hopefully prodding more reviews :)

cheers!
jay

Jay Pipes (jaypipes) wrote :

Also, if you don't select *any* reviewer, it defaults to nova-core, just an FYI...

Brian Lamar (blamar) wrote :

Hey Jay, I was waiting to hear back from a teammate before submitting directly to nova-core. I'll add them now since I realize the deadline is coming up soonish! Thanks again.

lp:~blamar/nova/lp728587 updated on 2011-03-17
803. By Brian Lamar on 2011-03-17

Merged trunk.

Paul Voccio (pvo) wrote :

nova/api/openstack/faults.py:74:41: E202 whitespace before '}'
                "overLimitFault": "code"
                                        ^

nova/api/openstack/faults.py:97:80: E501 line too long (80 characters)
        self.wrapped_exc.body = serializer.serialize(self.content, content_type)

got a few pep8 nits to go over. Will look for the remerge.

review: Needs Fixing
lp:~blamar/nova/lp728587 updated on 2011-03-17
804. By Brian Lamar on 2011-03-17

Fixed pep8 violation.

Paul Voccio (pvo) wrote :

upgraded my pep8 to 0.6.1 and everything looked clean.

review: Approve
lp:~blamar/nova/lp728587 updated on 2011-03-17
805. By Brian Lamar on 2011-03-17

Pep8 error, oddly specific to pep8 v0.5 < x > v0.6

Rick Harris (rconradharris) wrote :

Great work, Brian.

Some femto-nits:

> 245 + self.remaining = math.floor((cap - water) / cap * val)

This might benefit from one more paren just to make it abundantly clear what's going on:

  self.remaining = math.floor(((cap - water) / cap) * val)

> 901 + return sum(filter(lambda x: x != None, results))

Might be clearer as:

  return sum(result for result in results if result)

As I understand it, the recommendation is to favor list-comprehensions rather than filter/map/reduce where possible.

Brian Lamar (blamar) wrote :

Thanks Rick. I hate to see the day that people don't quite get order of operations, but it's a great femto-nit (I'd even accept it as a pico-nit!).

As for the sum() line, I'd like to say your solution is more clear...but it is at least more efficient. I must not have thought about that line too long because now that I look at it:

---
return sum(filter(lambda x: x != None, results))

is the same as

return sum(filter(None, results))
---

That being said I understand list comprehensions (or actually in this case I think it ends up being a generator comprehension, which I just learned about!) are preferred so I've gone ahead and made these changes.

Thanks a lot for the review!

lp:~blamar/nova/lp728587 updated on 2011-03-17
806. By Brian Lamar on 2011-03-17

Better comment for fault. Improved readability of two small sections.

Rick Harris (rconradharris) wrote :

lgtm, thanks for the fixes.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'etc/api-paste.ini'
2--- etc/api-paste.ini 2011-03-07 19:33:24 +0000
3+++ etc/api-paste.ini 2011-03-17 20:28:35 +0000
4@@ -79,7 +79,7 @@
5 paste.filter_factory = nova.api.openstack.auth:AuthMiddleware.factory
6
7 [filter:ratelimit]
8-paste.filter_factory = nova.api.openstack.ratelimiting:RateLimitingMiddleware.factory
9+paste.filter_factory = nova.api.openstack.limits:RateLimitingMiddleware.factory
10
11 [app:osapiapp]
12 paste.app_factory = nova.api.openstack:APIRouter.factory
13
14=== modified file 'nova/api/openstack/__init__.py'
15--- nova/api/openstack/__init__.py 2011-03-11 19:49:32 +0000
16+++ nova/api/openstack/__init__.py 2011-03-17 20:28:35 +0000
17@@ -33,6 +33,7 @@
18 from nova.api.openstack import consoles
19 from nova.api.openstack import flavors
20 from nova.api.openstack import images
21+from nova.api.openstack import limits
22 from nova.api.openstack import servers
23 from nova.api.openstack import shared_ip_groups
24 from nova.api.openstack import users
25@@ -114,12 +115,17 @@
26
27 mapper.resource("image", "images", controller=images.Controller(),
28 collection={'detail': 'GET'})
29+
30 mapper.resource("flavor", "flavors", controller=flavors.Controller(),
31 collection={'detail': 'GET'})
32+
33 mapper.resource("shared_ip_group", "shared_ip_groups",
34 collection={'detail': 'GET'},
35 controller=shared_ip_groups.Controller())
36
37+ _limits = limits.LimitsController()
38+ mapper.resource("limit", "limits", controller=_limits)
39+
40 super(APIRouter, self).__init__(mapper)
41
42
43
44=== modified file 'nova/api/openstack/faults.py'
45--- nova/api/openstack/faults.py 2011-03-09 20:08:11 +0000
46+++ nova/api/openstack/faults.py 2011-03-17 20:28:35 +0000
47@@ -61,3 +61,42 @@
48 content_type = req.best_match_content_type()
49 self.wrapped_exc.body = serializer.serialize(fault_data, content_type)
50 return self.wrapped_exc
51+
52+
53+class OverLimitFault(webob.exc.HTTPException):
54+ """
55+ Rate-limited request response.
56+ """
57+
58+ _serialization_metadata = {
59+ "application/xml": {
60+ "attributes": {
61+ "overLimitFault": "code",
62+ },
63+ },
64+ }
65+
66+ def __init__(self, message, details, retry_time):
67+ """
68+ Initialize new `OverLimitFault` with relevant information.
69+ """
70+ self.wrapped_exc = webob.exc.HTTPForbidden()
71+ self.content = {
72+ "overLimitFault": {
73+ "code": self.wrapped_exc.status_int,
74+ "message": message,
75+ "details": details,
76+ },
77+ }
78+
79+ @webob.dec.wsgify(RequestClass=wsgi.Request)
80+ def __call__(self, request):
81+ """
82+ Return the wrapped exception with a serialized body conforming to our
83+ error format.
84+ """
85+ serializer = wsgi.Serializer(self._serialization_metadata)
86+ content_type = request.best_match_content_type()
87+ content = serializer.serialize(self.content, content_type)
88+ self.wrapped_exc.body = content
89+ return self.wrapped_exc
90
91=== added file 'nova/api/openstack/limits.py'
92--- nova/api/openstack/limits.py 1970-01-01 00:00:00 +0000
93+++ nova/api/openstack/limits.py 2011-03-17 20:28:35 +0000
94@@ -0,0 +1,358 @@
95+# Copyright 2011 OpenStack LLC.
96+# All Rights Reserved.
97+#
98+# Licensed under the Apache License, Version 2.0 (the "License"); you may
99+# not use this file except in compliance with the License. You may obtain
100+# a copy of the License at
101+#
102+# http://www.apache.org/licenses/LICENSE-2.0
103+#
104+# Unless required by applicable law or agreed to in writing, software
105+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
106+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
107+# License for the specific language governing permissions and limitations
108+# under the License.import datetime
109+
110+"""
111+Module dedicated functions/classes dealing with rate limiting requests.
112+"""
113+
114+import copy
115+import httplib
116+import json
117+import math
118+import re
119+import time
120+import urllib
121+import webob.exc
122+
123+from collections import defaultdict
124+
125+from webob.dec import wsgify
126+
127+from nova import wsgi
128+from nova.api.openstack import faults
129+from nova.wsgi import Controller
130+from nova.wsgi import Middleware
131+
132+
133+# Convenience constants for the limits dictionary passed to Limiter().
134+PER_SECOND = 1
135+PER_MINUTE = 60
136+PER_HOUR = 60 * 60
137+PER_DAY = 60 * 60 * 24
138+
139+
140+class LimitsController(Controller):
141+ """
142+ Controller for accessing limits in the OpenStack API.
143+ """
144+
145+ _serialization_metadata = {
146+ "application/xml": {
147+ "attributes": {
148+ "limit": ["verb", "URI", "regex", "value", "unit",
149+ "resetTime", "remaining", "name"],
150+ },
151+ "plurals": {
152+ "rate": "limit",
153+ },
154+ },
155+ }
156+
157+ def index(self, req):
158+ """
159+ Return all global and rate limit information.
160+ """
161+ abs_limits = {}
162+ rate_limits = req.environ.get("nova.limits", [])
163+
164+ return {
165+ "limits": {
166+ "rate": rate_limits,
167+ "absolute": abs_limits,
168+ },
169+ }
170+
171+
172+class Limit(object):
173+ """
174+ Stores information about a limit for HTTP requets.
175+ """
176+
177+ UNITS = {
178+ 1: "SECOND",
179+ 60: "MINUTE",
180+ 60 * 60: "HOUR",
181+ 60 * 60 * 24: "DAY",
182+ }
183+
184+ def __init__(self, verb, uri, regex, value, unit):
185+ """
186+ Initialize a new `Limit`.
187+
188+ @param verb: HTTP verb (POST, PUT, etc.)
189+ @param uri: Human-readable URI
190+ @param regex: Regular expression format for this limit
191+ @param value: Integer number of requests which can be made
192+ @param unit: Unit of measure for the value parameter
193+ """
194+ self.verb = verb
195+ self.uri = uri
196+ self.regex = regex
197+ self.value = int(value)
198+ self.unit = unit
199+ self.unit_string = self.display_unit().lower()
200+ self.remaining = int(value)
201+
202+ if value <= 0:
203+ raise ValueError("Limit value must be > 0")
204+
205+ self.last_request = None
206+ self.next_request = None
207+
208+ self.water_level = 0
209+ self.capacity = self.unit
210+ self.request_value = float(self.capacity) / float(self.value)
211+ self.error_message = _("Only %(value)s %(verb)s request(s) can be "\
212+ "made to %(uri)s every %(unit_string)s." % self.__dict__)
213+
214+ def __call__(self, verb, url):
215+ """
216+ Represents a call to this limit from a relevant request.
217+
218+ @param verb: string http verb (POST, GET, etc.)
219+ @param url: string URL
220+ """
221+ if self.verb != verb or not re.match(self.regex, url):
222+ return
223+
224+ now = self._get_time()
225+
226+ if self.last_request is None:
227+ self.last_request = now
228+
229+ leak_value = now - self.last_request
230+
231+ self.water_level -= leak_value
232+ self.water_level = max(self.water_level, 0)
233+ self.water_level += self.request_value
234+
235+ difference = self.water_level - self.capacity
236+
237+ self.last_request = now
238+
239+ if difference > 0:
240+ self.water_level -= self.request_value
241+ self.next_request = now + difference
242+ return difference
243+
244+ cap = self.capacity
245+ water = self.water_level
246+ val = self.value
247+
248+ self.remaining = math.floor(((cap - water) / cap) * val)
249+ self.next_request = now
250+
251+ def _get_time(self):
252+ """Retrieve the current time. Broken out for testability."""
253+ return time.time()
254+
255+ def display_unit(self):
256+ """Display the string name of the unit."""
257+ return self.UNITS.get(self.unit, "UNKNOWN")
258+
259+ def display(self):
260+ """Return a useful representation of this class."""
261+ return {
262+ "verb": self.verb,
263+ "URI": self.uri,
264+ "regex": self.regex,
265+ "value": self.value,
266+ "remaining": int(self.remaining),
267+ "unit": self.display_unit(),
268+ "resetTime": int(self.next_request or self._get_time()),
269+ }
270+
271+# "Limit" format is a dictionary with the HTTP verb, human-readable URI,
272+# a regular-expression to match, value and unit of measure (PER_DAY, etc.)
273+
274+DEFAULT_LIMITS = [
275+ Limit("POST", "*", ".*", 10, PER_MINUTE),
276+ Limit("POST", "*/servers", "^/servers", 50, PER_DAY),
277+ Limit("PUT", "*", ".*", 10, PER_MINUTE),
278+ Limit("GET", "*changes-since*", ".*changes-since.*", 3, PER_MINUTE),
279+ Limit("DELETE", "*", ".*", 100, PER_MINUTE),
280+]
281+
282+
283+class RateLimitingMiddleware(Middleware):
284+ """
285+ Rate-limits requests passing through this middleware. All limit information
286+ is stored in memory for this implementation.
287+ """
288+
289+ def __init__(self, application, limits=None):
290+ """
291+ Initialize new `RateLimitingMiddleware`, which wraps the given WSGI
292+ application and sets up the given limits.
293+
294+ @param application: WSGI application to wrap
295+ @param limits: List of dictionaries describing limits
296+ """
297+ Middleware.__init__(self, application)
298+ self._limiter = Limiter(limits or DEFAULT_LIMITS)
299+
300+ @wsgify(RequestClass=wsgi.Request)
301+ def __call__(self, req):
302+ """
303+ Represents a single call through this middleware. We should record the
304+ request if we have a limit relevant to it. If no limit is relevant to
305+ the request, ignore it.
306+
307+ If the request should be rate limited, return a fault telling the user
308+ they are over the limit and need to retry later.
309+ """
310+ verb = req.method
311+ url = req.url
312+ context = req.environ.get("nova.context")
313+
314+ if context:
315+ username = context.user_id
316+ else:
317+ username = None
318+
319+ delay, error = self._limiter.check_for_delay(verb, url, username)
320+
321+ if delay:
322+ msg = _("This request was rate-limited.")
323+ retry = time.time() + delay
324+ return faults.OverLimitFault(msg, error, retry)
325+
326+ req.environ["nova.limits"] = self._limiter.get_limits(username)
327+
328+ return self.application
329+
330+
331+class Limiter(object):
332+ """
333+ Rate-limit checking class which handles limits in memory.
334+ """
335+
336+ def __init__(self, limits):
337+ """
338+ Initialize the new `Limiter`.
339+
340+ @param limits: List of `Limit` objects
341+ """
342+ self.limits = copy.deepcopy(limits)
343+ self.levels = defaultdict(lambda: copy.deepcopy(limits))
344+
345+ def get_limits(self, username=None):
346+ """
347+ Return the limits for a given user.
348+ """
349+ return [limit.display() for limit in self.levels[username]]
350+
351+ def check_for_delay(self, verb, url, username=None):
352+ """
353+ Check the given verb/user/user triplet for limit.
354+
355+ @return: Tuple of delay (in seconds) and error message (or None, None)
356+ """
357+ delays = []
358+
359+ for limit in self.levels[username]:
360+ delay = limit(verb, url)
361+ if delay:
362+ delays.append((delay, limit.error_message))
363+
364+ if delays:
365+ delays.sort()
366+ return delays[0]
367+
368+ return None, None
369+
370+
371+class WsgiLimiter(object):
372+ """
373+ Rate-limit checking from a WSGI application. Uses an in-memory `Limiter`.
374+
375+ To use:
376+ POST /<username> with JSON data such as:
377+ {
378+ "verb" : GET,
379+ "path" : "/servers"
380+ }
381+
382+ and receive a 204 No Content, or a 403 Forbidden with an X-Wait-Seconds
383+ header containing the number of seconds to wait before the action would
384+ succeed.
385+ """
386+
387+ def __init__(self, limits=None):
388+ """
389+ Initialize the new `WsgiLimiter`.
390+
391+ @param limits: List of `Limit` objects
392+ """
393+ self._limiter = Limiter(limits or DEFAULT_LIMITS)
394+
395+ @wsgify(RequestClass=wsgi.Request)
396+ def __call__(self, request):
397+ """
398+ Handles a call to this application. Returns 204 if the request is
399+ acceptable to the limiter, else a 403 is returned with a relevant
400+ header indicating when the request *will* succeed.
401+ """
402+ if request.method != "POST":
403+ raise webob.exc.HTTPMethodNotAllowed()
404+
405+ try:
406+ info = dict(json.loads(request.body))
407+ except ValueError:
408+ raise webob.exc.HTTPBadRequest()
409+
410+ username = request.path_info_pop()
411+ verb = info.get("verb")
412+ path = info.get("path")
413+
414+ delay, error = self._limiter.check_for_delay(verb, path, username)
415+
416+ if delay:
417+ headers = {"X-Wait-Seconds": "%.2f" % delay}
418+ return webob.exc.HTTPForbidden(headers=headers, explanation=error)
419+ else:
420+ return webob.exc.HTTPNoContent()
421+
422+
423+class WsgiLimiterProxy(object):
424+ """
425+ Rate-limit requests based on answers from a remote source.
426+ """
427+
428+ def __init__(self, limiter_address):
429+ """
430+ Initialize the new `WsgiLimiterProxy`.
431+
432+ @param limiter_address: IP/port combination of where to request limit
433+ """
434+ self.limiter_address = limiter_address
435+
436+ def check_for_delay(self, verb, path, username=None):
437+ body = json.dumps({"verb": verb, "path": path})
438+ headers = {"Content-Type": "application/json"}
439+
440+ conn = httplib.HTTPConnection(self.limiter_address)
441+
442+ if username:
443+ conn.request("POST", "/%s" % (username), body, headers)
444+ else:
445+ conn.request("POST", "/", body, headers)
446+
447+ resp = conn.getresponse()
448+
449+ if 200 >= resp.status < 300:
450+ return None, None
451+
452+ return resp.getheader("X-Wait-Seconds"), resp.read() or None
453
454=== modified file 'nova/tests/api/openstack/__init__.py'
455--- nova/tests/api/openstack/__init__.py 2011-02-23 19:56:37 +0000
456+++ nova/tests/api/openstack/__init__.py 2011-03-17 20:28:35 +0000
457@@ -20,7 +20,7 @@
458
459 from nova import context
460 from nova import flags
461-from nova.api.openstack.ratelimiting import RateLimitingMiddleware
462+from nova.api.openstack.limits import RateLimitingMiddleware
463 from nova.api.openstack.common import limited
464 from nova.tests.api.openstack import fakes
465 from webob import Request
466
467=== modified file 'nova/tests/api/openstack/fakes.py'
468--- nova/tests/api/openstack/fakes.py 2011-03-16 19:16:16 +0000
469+++ nova/tests/api/openstack/fakes.py 2011-03-17 20:28:35 +0000
470@@ -34,7 +34,7 @@
471 import nova.api.openstack.auth
472 from nova.api import openstack
473 from nova.api.openstack import auth
474-from nova.api.openstack import ratelimiting
475+from nova.api.openstack import limits
476 from nova.auth.manager import User, Project
477 from nova.image import glance
478 from nova.image import local
479@@ -77,7 +77,7 @@
480 inner_application = openstack.APIRouter()
481 mapper = urlmap.URLMap()
482 api = openstack.FaultWrapper(auth.AuthMiddleware(
483- ratelimiting.RateLimitingMiddleware(inner_application)))
484+ limits.RateLimitingMiddleware(inner_application)))
485 mapper['/v1.0'] = api
486 mapper['/'] = openstack.FaultWrapper(openstack.Versions())
487 return mapper
488@@ -115,13 +115,13 @@
489
490 def stub_out_rate_limiting(stubs):
491 def fake_rate_init(self, app):
492- super(ratelimiting.RateLimitingMiddleware, self).__init__(app)
493+ super(limits.RateLimitingMiddleware, self).__init__(app)
494 self.application = app
495
496- stubs.Set(nova.api.openstack.ratelimiting.RateLimitingMiddleware,
497+ stubs.Set(nova.api.openstack.limits.RateLimitingMiddleware,
498 '__init__', fake_rate_init)
499
500- stubs.Set(nova.api.openstack.ratelimiting.RateLimitingMiddleware,
501+ stubs.Set(nova.api.openstack.limits.RateLimitingMiddleware,
502 '__call__', fake_wsgi)
503
504
505
506=== modified file 'nova/tests/api/openstack/test_adminapi.py'
507--- nova/tests/api/openstack/test_adminapi.py 2011-03-08 17:18:13 +0000
508+++ nova/tests/api/openstack/test_adminapi.py 2011-03-17 20:28:35 +0000
509@@ -23,7 +23,6 @@
510 from nova import flags
511 from nova import test
512 from nova.api import openstack
513-from nova.api.openstack import ratelimiting
514 from nova.api.openstack import auth
515 from nova.tests.api.openstack import fakes
516
517
518=== renamed file 'nova/tests/api/openstack/test_ratelimiting.py' => 'nova/tests/api/openstack/test_limits.py'
519--- nova/tests/api/openstack/test_ratelimiting.py 2011-02-23 19:56:37 +0000
520+++ nova/tests/api/openstack/test_limits.py 2011-03-17 20:28:35 +0000
521@@ -1,178 +1,520 @@
522+# Copyright 2011 OpenStack LLC.
523+# All Rights Reserved.
524+#
525+# Licensed under the Apache License, Version 2.0 (the "License"); you may
526+# not use this file except in compliance with the License. You may obtain
527+# a copy of the License at
528+#
529+# http://www.apache.org/licenses/LICENSE-2.0
530+#
531+# Unless required by applicable law or agreed to in writing, software
532+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
533+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
534+# License for the specific language governing permissions and limitations
535+# under the License.
536+
537+"""
538+Tests dealing with HTTP rate-limiting.
539+"""
540+
541 import httplib
542+import json
543 import StringIO
544+import stubout
545 import time
546+import unittest
547 import webob
548
549-from nova import test
550-import nova.api.openstack.ratelimiting as ratelimiting
551-
552-
553-class LimiterTest(test.TestCase):
554-
555- def setUp(self):
556- super(LimiterTest, self).setUp()
557- self.limits = {
558- 'a': (5, ratelimiting.PER_SECOND),
559- 'b': (5, ratelimiting.PER_MINUTE),
560- 'c': (5, ratelimiting.PER_HOUR),
561- 'd': (1, ratelimiting.PER_SECOND),
562- 'e': (100, ratelimiting.PER_SECOND)}
563- self.rl = ratelimiting.Limiter(self.limits)
564-
565- def exhaust(self, action, times_until_exhausted, **kwargs):
566- for i in range(times_until_exhausted):
567- when = self.rl.perform(action, **kwargs)
568- self.assertEqual(when, None)
569- num, period = self.limits[action]
570- delay = period * 1.0 / num
571- # Verify that we are now thoroughly delayed
572- for i in range(10):
573- when = self.rl.perform(action, **kwargs)
574- self.assertAlmostEqual(when, delay, 2)
575-
576- def test_second(self):
577- self.exhaust('a', 5)
578- time.sleep(0.2)
579- self.exhaust('a', 1)
580- time.sleep(1)
581- self.exhaust('a', 5)
582-
583- def test_minute(self):
584- self.exhaust('b', 5)
585-
586- def test_one_per_period(self):
587- def allow_once_and_deny_once():
588- when = self.rl.perform('d')
589- self.assertEqual(when, None)
590- when = self.rl.perform('d')
591- self.assertAlmostEqual(when, 1, 2)
592- return when
593- time.sleep(allow_once_and_deny_once())
594- time.sleep(allow_once_and_deny_once())
595- allow_once_and_deny_once()
596-
597- def test_we_can_go_indefinitely_if_we_spread_out_requests(self):
598- for i in range(200):
599- when = self.rl.perform('e')
600- self.assertEqual(when, None)
601- time.sleep(0.01)
602-
603- def test_users_get_separate_buckets(self):
604- self.exhaust('c', 5, username='alice')
605- self.exhaust('c', 5, username='bob')
606- self.exhaust('c', 5, username='chuck')
607- self.exhaust('c', 0, username='chuck')
608- self.exhaust('c', 0, username='bob')
609- self.exhaust('c', 0, username='alice')
610-
611-
612-class FakeLimiter(object):
613- """Fake Limiter class that you can tell how to behave."""
614-
615- def __init__(self, test):
616- self._action = self._username = self._delay = None
617- self.test = test
618-
619- def mock(self, action, username, delay):
620- self._action = action
621- self._username = username
622- self._delay = delay
623-
624- def perform(self, action, username):
625- self.test.assertEqual(action, self._action)
626- self.test.assertEqual(username, self._username)
627- return self._delay
628-
629-
630-class WSGIAppTest(test.TestCase):
631-
632- def setUp(self):
633- super(WSGIAppTest, self).setUp()
634- self.limiter = FakeLimiter(self)
635- self.app = ratelimiting.WSGIApp(self.limiter)
636-
637- def test_invalid_methods(self):
638- requests = []
639- for method in ['GET', 'PUT', 'DELETE']:
640- req = webob.Request.blank('/limits/michael/breakdance',
641- dict(REQUEST_METHOD=method))
642- requests.append(req)
643- for req in requests:
644- self.assertEqual(req.get_response(self.app).status_int, 405)
645-
646- def test_invalid_urls(self):
647- requests = []
648- for prefix in ['limit', '', 'limiter2', 'limiter/limits', 'limiter/1']:
649- req = webob.Request.blank('/%s/michael/breakdance' % prefix,
650- dict(REQUEST_METHOD='POST'))
651- requests.append(req)
652- for req in requests:
653- self.assertEqual(req.get_response(self.app).status_int, 404)
654-
655- def verify(self, url, username, action, delay=None):
656+from xml.dom.minidom import parseString
657+
658+from nova.api.openstack import limits
659+from nova.api.openstack.limits import Limit
660+
661+
662+TEST_LIMITS = [
663+ Limit("GET", "/delayed", "^/delayed", 1, limits.PER_MINUTE),
664+ Limit("POST", "*", ".*", 7, limits.PER_MINUTE),
665+ Limit("POST", "/servers", "^/servers", 3, limits.PER_MINUTE),
666+ Limit("PUT", "*", "", 10, limits.PER_MINUTE),
667+ Limit("PUT", "/servers", "^/servers", 5, limits.PER_MINUTE),
668+]
669+
670+
671+class BaseLimitTestSuite(unittest.TestCase):
672+ """Base test suite which provides relevant stubs and time abstraction."""
673+
674+ def setUp(self):
675+ """Run before each test."""
676+ self.time = 0.0
677+ self.stubs = stubout.StubOutForTesting()
678+ self.stubs.Set(limits.Limit, "_get_time", self._get_time)
679+
680+ def tearDown(self):
681+ """Run after each test."""
682+ self.stubs.UnsetAll()
683+
684+ def _get_time(self):
685+ """Return the "time" according to this test suite."""
686+ return self.time
687+
688+
689+class LimitsControllerTest(BaseLimitTestSuite):
690+ """
691+ Tests for `limits.LimitsController` class.
692+ """
693+
694+ def setUp(self):
695+ """Run before each test."""
696+ BaseLimitTestSuite.setUp(self)
697+ self.controller = limits.LimitsController()
698+
699+ def _get_index_request(self, accept_header="application/json"):
700+ """Helper to set routing arguments."""
701+ request = webob.Request.blank("/")
702+ request.accept = accept_header
703+ request.environ["wsgiorg.routing_args"] = (None, {
704+ "action": "index",
705+ "controller": "",
706+ })
707+ return request
708+
709+ def _populate_limits(self, request):
710+ """Put limit info into a request."""
711+ _limits = [
712+ Limit("GET", "*", ".*", 10, 60).display(),
713+ Limit("POST", "*", ".*", 5, 60 * 60).display(),
714+ ]
715+ request.environ["nova.limits"] = _limits
716+ return request
717+
718+ def test_empty_index_json(self):
719+ """Test getting empty limit details in JSON."""
720+ request = self._get_index_request()
721+ response = request.get_response(self.controller)
722+ expected = {
723+ "limits": {
724+ "rate": [],
725+ "absolute": {},
726+ },
727+ }
728+ body = json.loads(response.body)
729+ self.assertEqual(expected, body)
730+
731+ def test_index_json(self):
732+ """Test getting limit details in JSON."""
733+ request = self._get_index_request()
734+ request = self._populate_limits(request)
735+ response = request.get_response(self.controller)
736+ expected = {
737+ "limits": {
738+ "rate": [{
739+ "regex": ".*",
740+ "resetTime": 0,
741+ "URI": "*",
742+ "value": 10,
743+ "verb": "GET",
744+ "remaining": 10,
745+ "unit": "MINUTE",
746+ },
747+ {
748+ "regex": ".*",
749+ "resetTime": 0,
750+ "URI": "*",
751+ "value": 5,
752+ "verb": "POST",
753+ "remaining": 5,
754+ "unit": "HOUR",
755+ }],
756+ "absolute": {},
757+ },
758+ }
759+ body = json.loads(response.body)
760+ self.assertEqual(expected, body)
761+
762+ def test_empty_index_xml(self):
763+ """Test getting limit details in XML."""
764+ request = self._get_index_request("application/xml")
765+ response = request.get_response(self.controller)
766+
767+ expected = "<limits><rate/><absolute/></limits>"
768+ body = response.body.replace("\n", "").replace(" ", "")
769+
770+ self.assertEqual(expected, body)
771+
772+ def test_index_xml(self):
773+ """Test getting limit details in XML."""
774+ request = self._get_index_request("application/xml")
775+ request = self._populate_limits(request)
776+ response = request.get_response(self.controller)
777+
778+ expected = parseString("""
779+ <limits>
780+ <rate>
781+ <limit URI="*" regex=".*" remaining="10" resetTime="0"
782+ unit="MINUTE" value="10" verb="GET"/>
783+ <limit URI="*" regex=".*" remaining="5" resetTime="0"
784+ unit="HOUR" value="5" verb="POST"/>
785+ </rate>
786+ <absolute/>
787+ </limits>
788+ """.replace(" ", ""))
789+ body = parseString(response.body.replace(" ", ""))
790+
791+ self.assertEqual(expected.toxml(), body.toxml())
792+
793+
794+class LimitMiddlewareTest(BaseLimitTestSuite):
795+ """
796+ Tests for the `limits.RateLimitingMiddleware` class.
797+ """
798+
799+ @webob.dec.wsgify
800+ def _empty_app(self, request):
801+ """Do-nothing WSGI app."""
802+ pass
803+
804+ def setUp(self):
805+ """Prepare middleware for use through fake WSGI app."""
806+ BaseLimitTestSuite.setUp(self)
807+ _limits = [
808+ Limit("GET", "*", ".*", 1, 60),
809+ ]
810+ self.app = limits.RateLimitingMiddleware(self._empty_app, _limits)
811+
812+ def test_good_request(self):
813+ """Test successful GET request through middleware."""
814+ request = webob.Request.blank("/")
815+ response = request.get_response(self.app)
816+ self.assertEqual(200, response.status_int)
817+
818+ def test_limited_request_json(self):
819+ """Test a rate-limited (403) GET request through middleware."""
820+ request = webob.Request.blank("/")
821+ response = request.get_response(self.app)
822+ self.assertEqual(200, response.status_int)
823+
824+ request = webob.Request.blank("/")
825+ response = request.get_response(self.app)
826+ self.assertEqual(response.status_int, 403)
827+
828+ body = json.loads(response.body)
829+ expected = "Only 1 GET request(s) can be made to * every minute."
830+ value = body["overLimitFault"]["details"].strip()
831+ self.assertEqual(value, expected)
832+
833+ def test_limited_request_xml(self):
834+ """Test a rate-limited (403) response as XML"""
835+ request = webob.Request.blank("/")
836+ response = request.get_response(self.app)
837+ self.assertEqual(200, response.status_int)
838+
839+ request = webob.Request.blank("/")
840+ request.accept = "application/xml"
841+ response = request.get_response(self.app)
842+ self.assertEqual(response.status_int, 403)
843+
844+ root = parseString(response.body).childNodes[0]
845+ expected = "Only 1 GET request(s) can be made to * every minute."
846+
847+ details = root.getElementsByTagName("details")
848+ self.assertEqual(details.length, 1)
849+
850+ value = details.item(0).firstChild.data.strip()
851+ self.assertEqual(value, expected)
852+
853+
854+class LimitTest(BaseLimitTestSuite):
855+ """
856+ Tests for the `limits.Limit` class.
857+ """
858+
859+ def test_GET_no_delay(self):
860+ """Test a limit handles 1 GET per second."""
861+ limit = Limit("GET", "*", ".*", 1, 1)
862+ delay = limit("GET", "/anything")
863+ self.assertEqual(None, delay)
864+ self.assertEqual(0, limit.next_request)
865+ self.assertEqual(0, limit.last_request)
866+
867+ def test_GET_delay(self):
868+ """Test two calls to 1 GET per second limit."""
869+ limit = Limit("GET", "*", ".*", 1, 1)
870+ delay = limit("GET", "/anything")
871+ self.assertEqual(None, delay)
872+
873+ delay = limit("GET", "/anything")
874+ self.assertEqual(1, delay)
875+ self.assertEqual(1, limit.next_request)
876+ self.assertEqual(0, limit.last_request)
877+
878+ self.time += 4
879+
880+ delay = limit("GET", "/anything")
881+ self.assertEqual(None, delay)
882+ self.assertEqual(4, limit.next_request)
883+ self.assertEqual(4, limit.last_request)
884+
885+
886+class LimiterTest(BaseLimitTestSuite):
887+ """
888+ Tests for the in-memory `limits.Limiter` class.
889+ """
890+
891+ def setUp(self):
892+ """Run before each test."""
893+ BaseLimitTestSuite.setUp(self)
894+ self.limiter = limits.Limiter(TEST_LIMITS)
895+
896+ def _check(self, num, verb, url, username=None):
897+ """Check and yield results from checks."""
898+ for x in xrange(num):
899+ yield self.limiter.check_for_delay(verb, url, username)[0]
900+
901+ def _check_sum(self, num, verb, url, username=None):
902+ """Check and sum results from checks."""
903+ results = self._check(num, verb, url, username)
904+ return sum(item for item in results if item)
905+
906+ def test_no_delay_GET(self):
907+ """
908+ Simple test to ensure no delay on a single call for a limit verb we
909+ didn"t set.
910+ """
911+ delay = self.limiter.check_for_delay("GET", "/anything")
912+ self.assertEqual(delay, (None, None))
913+
914+ def test_no_delay_PUT(self):
915+ """
916+ Simple test to ensure no delay on a single call for a known limit.
917+ """
918+ delay = self.limiter.check_for_delay("PUT", "/anything")
919+ self.assertEqual(delay, (None, None))
920+
921+ def test_delay_PUT(self):
922+ """
923+ Ensure the 11th PUT will result in a delay of 6.0 seconds until
924+ the next request will be granced.
925+ """
926+ expected = [None] * 10 + [6.0]
927+ results = list(self._check(11, "PUT", "/anything"))
928+
929+ self.assertEqual(expected, results)
930+
931+ def test_delay_POST(self):
932+ """
933+ Ensure the 8th POST will result in a delay of 6.0 seconds until
934+ the next request will be granced.
935+ """
936+ expected = [None] * 7
937+ results = list(self._check(7, "POST", "/anything"))
938+ self.assertEqual(expected, results)
939+
940+ expected = 60.0 / 7.0
941+ results = self._check_sum(1, "POST", "/anything")
942+ self.failUnlessAlmostEqual(expected, results, 8)
943+
944+ def test_delay_GET(self):
945+ """
946+ Ensure the 11th GET will result in NO delay.
947+ """
948+ expected = [None] * 11
949+ results = list(self._check(11, "GET", "/anything"))
950+
951+ self.assertEqual(expected, results)
952+
953+ def test_delay_PUT_servers(self):
954+ """
955+ Ensure PUT on /servers limits at 5 requests, and PUT elsewhere is still
956+ OK after 5 requests...but then after 11 total requests, PUT limiting
957+ kicks in.
958+ """
959+ # First 6 requests on PUT /servers
960+ expected = [None] * 5 + [12.0]
961+ results = list(self._check(6, "PUT", "/servers"))
962+ self.assertEqual(expected, results)
963+
964+ # Next 5 request on PUT /anything
965+ expected = [None] * 4 + [6.0]
966+ results = list(self._check(5, "PUT", "/anything"))
967+ self.assertEqual(expected, results)
968+
969+ def test_delay_PUT_wait(self):
970+ """
971+ Ensure after hitting the limit and then waiting for the correct
972+ amount of time, the limit will be lifted.
973+ """
974+ expected = [None] * 10 + [6.0]
975+ results = list(self._check(11, "PUT", "/anything"))
976+ self.assertEqual(expected, results)
977+
978+ # Advance time
979+ self.time += 6.0
980+
981+ expected = [None, 6.0]
982+ results = list(self._check(2, "PUT", "/anything"))
983+ self.assertEqual(expected, results)
984+
985+ def test_multiple_delays(self):
986+ """
987+ Ensure multiple requests still get a delay.
988+ """
989+ expected = [None] * 10 + [6.0] * 10
990+ results = list(self._check(20, "PUT", "/anything"))
991+ self.assertEqual(expected, results)
992+
993+ self.time += 1.0
994+
995+ expected = [5.0] * 10
996+ results = list(self._check(10, "PUT", "/anything"))
997+ self.assertEqual(expected, results)
998+
999+ def test_multiple_users(self):
1000+ """
1001+ Tests involving multiple users.
1002+ """
1003+ # User1
1004+ expected = [None] * 10 + [6.0] * 10
1005+ results = list(self._check(20, "PUT", "/anything", "user1"))
1006+ self.assertEqual(expected, results)
1007+
1008+ # User2
1009+ expected = [None] * 10 + [6.0] * 5
1010+ results = list(self._check(15, "PUT", "/anything", "user2"))
1011+ self.assertEqual(expected, results)
1012+
1013+ self.time += 1.0
1014+
1015+ # User1 again
1016+ expected = [5.0] * 10
1017+ results = list(self._check(10, "PUT", "/anything", "user1"))
1018+ self.assertEqual(expected, results)
1019+
1020+ self.time += 1.0
1021+
1022+ # User1 again
1023+ expected = [4.0] * 5
1024+ results = list(self._check(5, "PUT", "/anything", "user2"))
1025+ self.assertEqual(expected, results)
1026+
1027+
1028+class WsgiLimiterTest(BaseLimitTestSuite):
1029+ """
1030+ Tests for `limits.WsgiLimiter` class.
1031+ """
1032+
1033+ def setUp(self):
1034+ """Run before each test."""
1035+ BaseLimitTestSuite.setUp(self)
1036+ self.app = limits.WsgiLimiter(TEST_LIMITS)
1037+
1038+ def _request_data(self, verb, path):
1039+ """Get data decribing a limit request verb/path."""
1040+ return json.dumps({"verb": verb, "path": path})
1041+
1042+ def _request(self, verb, url, username=None):
1043 """Make sure that POSTing to the given url causes the given username
1044 to perform the given action. Make the internal rate limiter return
1045 delay and make sure that the WSGI app returns the correct response.
1046 """
1047- req = webob.Request.blank(url, dict(REQUEST_METHOD='POST'))
1048- self.limiter.mock(action, username, delay)
1049- resp = req.get_response(self.app)
1050- if not delay:
1051- self.assertEqual(resp.status_int, 200)
1052+ if username:
1053+ request = webob.Request.blank("/%s" % username)
1054 else:
1055- self.assertEqual(resp.status_int, 403)
1056- self.assertEqual(resp.headers['X-Wait-Seconds'], "%.2f" % delay)
1057-
1058- def test_good_urls(self):
1059- self.verify('/limiter/michael/hoot', 'michael', 'hoot')
1060+ request = webob.Request.blank("/")
1061+
1062+ request.method = "POST"
1063+ request.body = self._request_data(verb, url)
1064+ response = request.get_response(self.app)
1065+
1066+ if "X-Wait-Seconds" in response.headers:
1067+ self.assertEqual(response.status_int, 403)
1068+ return response.headers["X-Wait-Seconds"]
1069+
1070+ self.assertEqual(response.status_int, 204)
1071+
1072+ def test_invalid_methods(self):
1073+ """Only POSTs should work."""
1074+ requests = []
1075+ for method in ["GET", "PUT", "DELETE", "HEAD", "OPTIONS"]:
1076+ request = webob.Request.blank("/")
1077+ request.body = self._request_data("GET", "/something")
1078+ response = request.get_response(self.app)
1079+ self.assertEqual(response.status_int, 405)
1080+
1081+ def test_good_url(self):
1082+ delay = self._request("GET", "/something")
1083+ self.assertEqual(delay, None)
1084
1085 def test_escaping(self):
1086- self.verify('/limiter/michael/jump%20up', 'michael', 'jump up')
1087+ delay = self._request("GET", "/something/jump%20up")
1088+ self.assertEqual(delay, None)
1089
1090 def test_response_to_delays(self):
1091- self.verify('/limiter/michael/hoot', 'michael', 'hoot', 1)
1092- self.verify('/limiter/michael/hoot', 'michael', 'hoot', 1.56)
1093- self.verify('/limiter/michael/hoot', 'michael', 'hoot', 1000)
1094+ delay = self._request("GET", "/delayed")
1095+ self.assertEqual(delay, None)
1096+
1097+ delay = self._request("GET", "/delayed")
1098+ self.assertEqual(delay, '60.00')
1099+
1100+ def test_response_to_delays_usernames(self):
1101+ delay = self._request("GET", "/delayed", "user1")
1102+ self.assertEqual(delay, None)
1103+
1104+ delay = self._request("GET", "/delayed", "user2")
1105+ self.assertEqual(delay, None)
1106+
1107+ delay = self._request("GET", "/delayed", "user1")
1108+ self.assertEqual(delay, '60.00')
1109+
1110+ delay = self._request("GET", "/delayed", "user2")
1111+ self.assertEqual(delay, '60.00')
1112
1113
1114 class FakeHttplibSocket(object):
1115- """a fake socket implementation for httplib.HTTPResponse, trivial"""
1116+ """
1117+ Fake `httplib.HTTPResponse` replacement.
1118+ """
1119
1120 def __init__(self, response_string):
1121+ """Initialize new `FakeHttplibSocket`."""
1122 self._buffer = StringIO.StringIO(response_string)
1123
1124 def makefile(self, _mode, _other):
1125- """Returns the socket's internal buffer"""
1126+ """Returns the socket's internal buffer."""
1127 return self._buffer
1128
1129
1130 class FakeHttplibConnection(object):
1131- """A fake httplib.HTTPConnection
1132+ """
1133+ Fake `httplib.HTTPConnection`.
1134+ """
1135
1136- Requests made via this connection actually get translated and routed into
1137- our WSGI app, we then wait for the response and turn it back into
1138- an httplib.HTTPResponse.
1139- """
1140- def __init__(self, app, host, is_secure=False):
1141+ def __init__(self, app, host):
1142+ """
1143+ Initialize `FakeHttplibConnection`.
1144+ """
1145 self.app = app
1146 self.host = host
1147
1148- def request(self, method, path, data='', headers={}):
1149+ def request(self, method, path, body="", headers={}):
1150+ """
1151+ Requests made via this connection actually get translated and routed
1152+ into our WSGI app, we then wait for the response and turn it back into
1153+ an `httplib.HTTPResponse`.
1154+ """
1155 req = webob.Request.blank(path)
1156 req.method = method
1157- req.body = data
1158 req.headers = headers
1159 req.host = self.host
1160- # Call the WSGI app, get the HTTP response
1161+ req.body = body
1162+
1163 resp = str(req.get_response(self.app))
1164- # For some reason, the response doesn't have "HTTP/1.0 " prepended; I
1165- # guess that's a function the web server usually provides.
1166 resp = "HTTP/1.0 %s" % resp
1167 sock = FakeHttplibSocket(resp)
1168 self.http_response = httplib.HTTPResponse(sock)
1169 self.http_response.begin()
1170
1171 def getresponse(self):
1172+ """Return our generated response from the request."""
1173 return self.http_response
1174
1175
1176@@ -208,36 +550,35 @@
1177 httplib.HTTPConnection = HTTPConnectionDecorator(httplib.HTTPConnection)
1178
1179
1180-class WSGIAppProxyTest(test.TestCase):
1181+class WsgiLimiterProxyTest(BaseLimitTestSuite):
1182+ """
1183+ Tests for the `limits.WsgiLimiterProxy` class.
1184+ """
1185
1186 def setUp(self):
1187- """Our WSGIAppProxy is going to call across an HTTPConnection to a
1188- WSGIApp running a limiter. The proxy will send input, and the proxy
1189- should receive that same input, pass it to the limiter who gives a
1190- result, and send the expected result back.
1191-
1192- The HTTPConnection isn't real -- it's monkeypatched to point straight
1193- at the WSGIApp. And the limiter isn't real -- it's a fake that
1194- behaves the way we tell it to.
1195- """
1196- super(WSGIAppProxyTest, self).setUp()
1197- self.limiter = FakeLimiter(self)
1198- app = ratelimiting.WSGIApp(self.limiter)
1199- wire_HTTPConnection_to_WSGI('100.100.100.100:80', app)
1200- self.proxy = ratelimiting.WSGIAppProxy('100.100.100.100:80')
1201+ """
1202+ Do some nifty HTTP/WSGI magic which allows for WSGI to be called
1203+ directly by something like the `httplib` library.
1204+ """
1205+ BaseLimitTestSuite.setUp(self)
1206+ self.app = limits.WsgiLimiter(TEST_LIMITS)
1207+ wire_HTTPConnection_to_WSGI("169.254.0.1:80", self.app)
1208+ self.proxy = limits.WsgiLimiterProxy("169.254.0.1:80")
1209
1210 def test_200(self):
1211- self.limiter.mock('conquer', 'caesar', None)
1212- when = self.proxy.perform('conquer', 'caesar')
1213- self.assertEqual(when, None)
1214+ """Successful request test."""
1215+ delay = self.proxy.check_for_delay("GET", "/anything")
1216+ self.assertEqual(delay, (None, None))
1217
1218 def test_403(self):
1219- self.limiter.mock('grumble', 'proletariat', 1.5)
1220- when = self.proxy.perform('grumble', 'proletariat')
1221- self.assertEqual(when, 1.5)
1222-
1223- def test_failure(self):
1224- def shouldRaise():
1225- self.limiter.mock('murder', 'brutus', None)
1226- self.proxy.perform('stab', 'brutus')
1227- self.assertRaises(AssertionError, shouldRaise)
1228+ """Forbidden request test."""
1229+ delay = self.proxy.check_for_delay("GET", "/delayed")
1230+ self.assertEqual(delay, (None, None))
1231+
1232+ delay, error = self.proxy.check_for_delay("GET", "/delayed")
1233+ error = error.strip()
1234+
1235+ expected = ("60.00", "403 Forbidden\n\nOnly 1 GET request(s) can be "\
1236+ "made to /delayed every minute.")
1237+
1238+ self.assertEqual((delay, error), expected)