Merge lp:~sandy-walsh/nova/zones4 into lp:~hudson-openstack/nova/trunk
- zones4
- Merge into trunk
Status: | Merged |
---|---|
Approved by: | Rick Harris |
Approved revision: | 720 |
Merged at revision: | 874 |
Proposed branch: | lp:~sandy-walsh/nova/zones4 |
Merge into: | lp:~hudson-openstack/nova/trunk |
Diff against target: |
725 lines (+387/-15) 7 files modified
nova/api/openstack/servers.py (+20/-1) nova/api/openstack/zones.py (+8/-10) nova/compute/api.py (+16/-1) nova/db/api.py (+1/-0) nova/scheduler/api.py (+177/-0) nova/scheduler/zone_manager.py (+4/-3) nova/tests/test_scheduler.py (+161/-0) |
To merge this branch: | bzr merge lp:~sandy-walsh/nova/zones4 |
Related bugs: | |
Related blueprints: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Rick Harris (community) | Approve | ||
Matt Dietz (community) | Approve | ||
Review via email: mp+54760@code.launchpad.net |
This proposal supersedes a proposal from 2011-03-17.
Commit message
Description of the change
In this branch we are forwarding incoming requests to child zones when the requested resource is not found in the current zone.
For example: If 'nova pause 123' is issued against Zone 1, but instance 123 does not live in Zone 1, the call will be forwarded to all child zones hoping someone can deal with it.
NOTE: This currently only works with OpenStack API requests and routing checks are only being done against Compute/instance_id checks.
Specifically:
* servers.
* servers.create is pending for distributed scheduler
* servers.get_all will get added early in Diablo.
What I've been doing for testing:
1. Set up a Nova deployment in a VM (Zone0)
2. Clone the VM and set --zone_name=zone1 (and change all the IP addresses to the new address in nova.conf, glance.conf and novarc)
3. Set --enable_
4. use the --connection_
5. Add Zone1 as a child of Zone0 (nova zone-add)
(make sure the instance id's are different in each zone)
Example of calls being sent to child zones:
http://
Eric Day (eday) wrote : Posted in a previous version of this proposal | # |
Sandy Walsh (sandy-walsh) wrote : Posted in a previous version of this proposal | # |
Thanks Eric,
That was the way I was initially intending to go.
The closer I got to it, I noticed that all calls would need to be re-marshaled to be sent to the children. We'd need a new client library abstraction layer (since forwarded requests may be OS API or EC2 API). I was hoping to avoid all that by simply forwarding the already marshaled message.
This client library abstraction is likely to get a little unwieldy since it needs to support OS/EC2 and whomever else comes along (actually, sounds like another endorsement for DirectAPI.)
Also, with this approach I was assuming an idiom of "API checks parameters and bails if they're wrong. Work is done in the services." ... but I see the counter-argument.
Let me think more about client library abstraction. Since we have big API changes coming soon, might be timely to consider this.
Sandy Walsh (sandy-walsh) wrote : Posted in a previous version of this proposal | # |
Thinking a little more about it, but I've got a concern about that approach. Not for technical reasons, but for schedule/political reasons: Since API is such a touchy subject and still under development, this could turn in a major blocker against getting anything practical done with Zones for a long time.
Eric Day (eday) wrote : Posted in a previous version of this proposal | # |
FWIW, I think making the call to novaclient (and re-marshal) from nova.compute.* is the correct thing to do. I would move forward with the OpenStack API for this as well, as that seems to be where we are committed at the moment. If we need to change this in the future, it will be a simple HTTP formatting issue inside novaclient (the API into novaclient shouldn't change).
Sandy Walsh (sandy-walsh) wrote : Posted in a previous version of this proposal | # |
Yup ... that's the direction I'll take. Thanks again Eric.
Sandy Walsh (sandy-walsh) wrote : Posted in a previous version of this proposal | # |
Ok, all refactored out of middleware now. The <service>/api.py does the checking now and the external api just bails out after a redirect. It's all handled in decorators now.
Rick Harris (rconradharris) wrote : Posted in a previous version of this proposal | # |
Nice work, Sandy.
Some really minor nits/suggestions:
> 402 + try:
> 403 + manager = getattr(nova, collection)
The getattr can be outside of the try-block (AFAICT).
> 404 + if isinstance(item_id, int) or item_id.isdigit():
> 405 + result = manager.
> 406 + else:
> 407 + result = manager.
One common way of handling this is to just go ahead and cast to int and
handle the possible ValueError (EAFP), like:
try:
result = manager.
except ValueError:
result = manager.
> 371 +def _wrap_method(
> 372 + """Wrap method to supply self."""
> 373 + def _wrap(*args, **kwargs):
> 374 + return function(self, *args, **kwargs)
> 375 + return _wrap
For the newly added decorators, it might be worth using @functools.wraps so
that the outer-func inherits the inner-funcs attributes. Can make debugging a
little easier.
> 397 +def _issue_
> 398 + item_id):
Trailing '\' isn't needed.
Could line-up item_id with opening param.
> 264 + @scheduler_
> 265 def get_diagnostics
Should that be 'get_diagnostics'? I ask because it's the only one that differs
in that regard.
Matt Dietz (cerberus) wrote : Posted in a previous version of this proposal | # |
The exception raising idea is really interesting. Hard to follow at first, but the tests make it clear IMO. I think this looks pretty pretty promising.
Sandy Walsh (sandy-walsh) wrote : Posted in a previous version of this proposal | # |
Thanks rick ... great feedback. I've implemented your changes.
re: get_diagnostics, the name in the decorator is the name of the method in novaclient, so it may not map 1:1 to the method being wrapped (as in this case).
Basically the decorator is saying, "if this method can't find the instance, check with the child zones by calling <method> in novaclient"
Sandy Walsh (sandy-walsh) wrote : Posted in a previous version of this proposal | # |
(oh, push pending, will change WIP to let you know)
Rick Harris (rconradharris) wrote : Posted in a previous version of this proposal | # |
> 788 +def redirect_
> 648 +def _wrap_method(
Didn't like the idea of using @functools.wraps to propagate the inner func's metadata?
Sandy Walsh (sandy-walsh) wrote : Posted in a previous version of this proposal | # |
I do. Good suggestion. But I'll fix that up on the next branch. Haven't spent enough time with it and don't want to miss the window today.
Rick Harris (rconradharris) wrote : Posted in a previous version of this proposal | # |
Good deal, lgtm. Thanks!
OpenStack Infra (hudson-openstack) wrote : Posted in a previous version of this proposal | # |
No proposals found for merge of lp:~sandy-walsh/nova/zones3 into lp:nova.
- 720. By Sandy Walsh
-
trunk merge
Rick Harris (rconradharris) wrote : | # |
Rej....Approved!
Preview Diff
1 | === modified file 'nova/api/openstack/servers.py' |
2 | --- nova/api/openstack/servers.py 2011-03-24 17:59:25 +0000 |
3 | +++ nova/api/openstack/servers.py 2011-03-24 19:09:30 +0000 |
4 | @@ -37,6 +37,7 @@ |
5 | from nova.compute import instance_types |
6 | from nova.compute import power_state |
7 | import nova.api.openstack |
8 | +from nova.scheduler import api as scheduler_api |
9 | |
10 | |
11 | LOG = logging.getLogger('server') |
12 | @@ -87,15 +88,18 @@ |
13 | for inst in limited_list] |
14 | return dict(servers=servers) |
15 | |
16 | + @scheduler_api.redirect_handler |
17 | def show(self, req, id): |
18 | """ Returns server details by server id """ |
19 | try: |
20 | - instance = self.compute_api.get(req.environ['nova.context'], id) |
21 | + instance = self.compute_api.routing_get( |
22 | + req.environ['nova.context'], id) |
23 | builder = self._get_view_builder(req) |
24 | return builder.build(instance, is_detail=True) |
25 | except exception.NotFound: |
26 | return faults.Fault(exc.HTTPNotFound()) |
27 | |
28 | + @scheduler_api.redirect_handler |
29 | def delete(self, req, id): |
30 | """ Destroys a server """ |
31 | try: |
32 | @@ -226,6 +230,7 @@ |
33 | # if the original error is okay, just reraise it |
34 | raise error |
35 | |
36 | + @scheduler_api.redirect_handler |
37 | def update(self, req, id): |
38 | """ Updates the server name or password """ |
39 | if len(req.body) == 0: |
40 | @@ -251,6 +256,7 @@ |
41 | return faults.Fault(exc.HTTPNotFound()) |
42 | return exc.HTTPNoContent() |
43 | |
44 | + @scheduler_api.redirect_handler |
45 | def action(self, req, id): |
46 | """Multi-purpose method used to reboot, rebuild, or |
47 | resize a server""" |
48 | @@ -316,6 +322,7 @@ |
49 | return faults.Fault(exc.HTTPUnprocessableEntity()) |
50 | return exc.HTTPAccepted() |
51 | |
52 | + @scheduler_api.redirect_handler |
53 | def lock(self, req, id): |
54 | """ |
55 | lock the instance with id |
56 | @@ -331,6 +338,7 @@ |
57 | return faults.Fault(exc.HTTPUnprocessableEntity()) |
58 | return exc.HTTPAccepted() |
59 | |
60 | + @scheduler_api.redirect_handler |
61 | def unlock(self, req, id): |
62 | """ |
63 | unlock the instance with id |
64 | @@ -346,6 +354,7 @@ |
65 | return faults.Fault(exc.HTTPUnprocessableEntity()) |
66 | return exc.HTTPAccepted() |
67 | |
68 | + @scheduler_api.redirect_handler |
69 | def get_lock(self, req, id): |
70 | """ |
71 | return the boolean state of (instance with id)'s lock |
72 | @@ -360,6 +369,7 @@ |
73 | return faults.Fault(exc.HTTPUnprocessableEntity()) |
74 | return exc.HTTPAccepted() |
75 | |
76 | + @scheduler_api.redirect_handler |
77 | def reset_network(self, req, id): |
78 | """ |
79 | Reset networking on an instance (admin only). |
80 | @@ -374,6 +384,7 @@ |
81 | return faults.Fault(exc.HTTPUnprocessableEntity()) |
82 | return exc.HTTPAccepted() |
83 | |
84 | + @scheduler_api.redirect_handler |
85 | def inject_network_info(self, req, id): |
86 | """ |
87 | Inject network info for an instance (admin only). |
88 | @@ -388,6 +399,7 @@ |
89 | return faults.Fault(exc.HTTPUnprocessableEntity()) |
90 | return exc.HTTPAccepted() |
91 | |
92 | + @scheduler_api.redirect_handler |
93 | def pause(self, req, id): |
94 | """ Permit Admins to Pause the server. """ |
95 | ctxt = req.environ['nova.context'] |
96 | @@ -399,6 +411,7 @@ |
97 | return faults.Fault(exc.HTTPUnprocessableEntity()) |
98 | return exc.HTTPAccepted() |
99 | |
100 | + @scheduler_api.redirect_handler |
101 | def unpause(self, req, id): |
102 | """ Permit Admins to Unpause the server. """ |
103 | ctxt = req.environ['nova.context'] |
104 | @@ -410,6 +423,7 @@ |
105 | return faults.Fault(exc.HTTPUnprocessableEntity()) |
106 | return exc.HTTPAccepted() |
107 | |
108 | + @scheduler_api.redirect_handler |
109 | def suspend(self, req, id): |
110 | """permit admins to suspend the server""" |
111 | context = req.environ['nova.context'] |
112 | @@ -421,6 +435,7 @@ |
113 | return faults.Fault(exc.HTTPUnprocessableEntity()) |
114 | return exc.HTTPAccepted() |
115 | |
116 | + @scheduler_api.redirect_handler |
117 | def resume(self, req, id): |
118 | """permit admins to resume the server from suspend""" |
119 | context = req.environ['nova.context'] |
120 | @@ -432,6 +447,7 @@ |
121 | return faults.Fault(exc.HTTPUnprocessableEntity()) |
122 | return exc.HTTPAccepted() |
123 | |
124 | + @scheduler_api.redirect_handler |
125 | def rescue(self, req, id): |
126 | """Permit users to rescue the server.""" |
127 | context = req.environ["nova.context"] |
128 | @@ -443,6 +459,7 @@ |
129 | return faults.Fault(exc.HTTPUnprocessableEntity()) |
130 | return exc.HTTPAccepted() |
131 | |
132 | + @scheduler_api.redirect_handler |
133 | def unrescue(self, req, id): |
134 | """Permit users to unrescue the server.""" |
135 | context = req.environ["nova.context"] |
136 | @@ -454,6 +471,7 @@ |
137 | return faults.Fault(exc.HTTPUnprocessableEntity()) |
138 | return exc.HTTPAccepted() |
139 | |
140 | + @scheduler_api.redirect_handler |
141 | def get_ajax_console(self, req, id): |
142 | """ Returns a url to an instance's ajaxterm console. """ |
143 | try: |
144 | @@ -463,6 +481,7 @@ |
145 | return faults.Fault(exc.HTTPNotFound()) |
146 | return exc.HTTPAccepted() |
147 | |
148 | + @scheduler_api.redirect_handler |
149 | def diagnostics(self, req, id): |
150 | """Permit Admins to retrieve server diagnostics.""" |
151 | ctxt = req.environ["nova.context"] |
152 | |
153 | === modified file 'nova/api/openstack/zones.py' |
154 | --- nova/api/openstack/zones.py 2011-03-23 19:31:15 +0000 |
155 | +++ nova/api/openstack/zones.py 2011-03-24 19:09:30 +0000 |
156 | @@ -17,6 +17,7 @@ |
157 | |
158 | from nova import db |
159 | from nova import flags |
160 | +from nova import log as logging |
161 | from nova import wsgi |
162 | from nova.scheduler import api |
163 | |
164 | @@ -38,7 +39,8 @@ |
165 | |
166 | |
167 | def _scrub_zone(zone): |
168 | - return _filter_keys(zone, ('id', 'api_url')) |
169 | + return _exclude_keys(zone, ('username', 'password', 'created_at', |
170 | + 'deleted', 'deleted_at', 'updated_at')) |
171 | |
172 | |
173 | class Controller(wsgi.Controller): |
174 | @@ -53,12 +55,8 @@ |
175 | # Ask the ZoneManager in the Scheduler for most recent data, |
176 | # or fall-back to the database ... |
177 | items = api.get_zone_list(req.environ['nova.context']) |
178 | - if not items: |
179 | - items = db.zone_get_all(req.environ['nova.context']) |
180 | - |
181 | items = common.limited(items, req) |
182 | - items = [_exclude_keys(item, ['username', 'password']) |
183 | - for item in items] |
184 | + items = [_scrub_zone(item) for item in items] |
185 | return dict(zones=items) |
186 | |
187 | def detail(self, req): |
188 | @@ -81,23 +79,23 @@ |
189 | def show(self, req, id): |
190 | """Return data about the given zone id""" |
191 | zone_id = int(id) |
192 | - zone = db.zone_get(req.environ['nova.context'], zone_id) |
193 | + zone = api.zone_get(req.environ['nova.context'], zone_id) |
194 | return dict(zone=_scrub_zone(zone)) |
195 | |
196 | def delete(self, req, id): |
197 | zone_id = int(id) |
198 | - db.zone_delete(req.environ['nova.context'], zone_id) |
199 | + api.zone_delete(req.environ['nova.context'], zone_id) |
200 | return {} |
201 | |
202 | def create(self, req): |
203 | context = req.environ['nova.context'] |
204 | env = self._deserialize(req.body, req.get_content_type()) |
205 | - zone = db.zone_create(context, env["zone"]) |
206 | + zone = api.zone_create(context, env["zone"]) |
207 | return dict(zone=_scrub_zone(zone)) |
208 | |
209 | def update(self, req, id): |
210 | context = req.environ['nova.context'] |
211 | env = self._deserialize(req.body, req.get_content_type()) |
212 | zone_id = int(id) |
213 | - zone = db.zone_update(context, zone_id, env["zone"]) |
214 | + zone = api.zone_update(context, zone_id, env["zone"]) |
215 | return dict(zone=_scrub_zone(zone)) |
216 | |
217 | === modified file 'nova/compute/api.py' |
218 | --- nova/compute/api.py 2011-03-24 17:53:54 +0000 |
219 | +++ nova/compute/api.py 2011-03-24 19:09:30 +0000 |
220 | @@ -34,6 +34,7 @@ |
221 | from nova import utils |
222 | from nova import volume |
223 | from nova.compute import instance_types |
224 | +from nova.scheduler import api as scheduler_api |
225 | from nova.db import base |
226 | |
227 | FLAGS = flags.FLAGS |
228 | @@ -352,6 +353,7 @@ |
229 | rv = self.db.instance_update(context, instance_id, kwargs) |
230 | return dict(rv.iteritems()) |
231 | |
232 | + @scheduler_api.reroute_compute("delete") |
233 | def delete(self, context, instance_id): |
234 | LOG.debug(_("Going to try to terminate %s"), instance_id) |
235 | try: |
236 | @@ -384,6 +386,13 @@ |
237 | rv = self.db.instance_get(context, instance_id) |
238 | return dict(rv.iteritems()) |
239 | |
240 | + @scheduler_api.reroute_compute("get") |
241 | + def routing_get(self, context, instance_id): |
242 | + """Use this method instead of get() if this is the only |
243 | + operation you intend to to. It will route to novaclient.get |
244 | + if the instance is not found.""" |
245 | + return self.get(context, instance_id) |
246 | + |
247 | def get_all(self, context, project_id=None, reservation_id=None, |
248 | fixed_ip=None): |
249 | """Get all instances, possibly filtered by one of the |
250 | @@ -527,14 +536,17 @@ |
251 | "instance_id": instance_id, |
252 | "flavor_id": flavor_id}}) |
253 | |
254 | + @scheduler_api.reroute_compute("pause") |
255 | def pause(self, context, instance_id): |
256 | """Pause the given instance.""" |
257 | self._cast_compute_message('pause_instance', context, instance_id) |
258 | |
259 | + @scheduler_api.reroute_compute("unpause") |
260 | def unpause(self, context, instance_id): |
261 | """Unpause the given instance.""" |
262 | self._cast_compute_message('unpause_instance', context, instance_id) |
263 | |
264 | + @scheduler_api.reroute_compute("diagnostics") |
265 | def get_diagnostics(self, context, instance_id): |
266 | """Retrieve diagnostics for the given instance.""" |
267 | return self._call_compute_message( |
268 | @@ -546,18 +558,22 @@ |
269 | """Retrieve actions for the given instance.""" |
270 | return self.db.instance_get_actions(context, instance_id) |
271 | |
272 | + @scheduler_api.reroute_compute("suspend") |
273 | def suspend(self, context, instance_id): |
274 | """suspend the instance with instance_id""" |
275 | self._cast_compute_message('suspend_instance', context, instance_id) |
276 | |
277 | + @scheduler_api.reroute_compute("resume") |
278 | def resume(self, context, instance_id): |
279 | """resume the instance with instance_id""" |
280 | self._cast_compute_message('resume_instance', context, instance_id) |
281 | |
282 | + @scheduler_api.reroute_compute("rescue") |
283 | def rescue(self, context, instance_id): |
284 | """Rescue the given instance.""" |
285 | self._cast_compute_message('rescue_instance', context, instance_id) |
286 | |
287 | + @scheduler_api.reroute_compute("unrescue") |
288 | def unrescue(self, context, instance_id): |
289 | """Unrescue the given instance.""" |
290 | self._cast_compute_message('unrescue_instance', context, instance_id) |
291 | @@ -573,7 +589,6 @@ |
292 | |
293 | def get_ajax_console(self, context, instance_id): |
294 | """Get a url to an AJAX Console""" |
295 | - instance = self.get(context, instance_id) |
296 | output = self._call_compute_message('get_ajax_console', |
297 | context, |
298 | instance_id) |
299 | |
300 | === modified file 'nova/db/api.py' |
301 | --- nova/db/api.py 2011-03-24 18:02:04 +0000 |
302 | +++ nova/db/api.py 2011-03-24 19:09:30 +0000 |
303 | @@ -71,6 +71,7 @@ |
304 | """No more available blades""" |
305 | pass |
306 | |
307 | + |
308 | ################### |
309 | |
310 | |
311 | |
312 | === modified file 'nova/scheduler/api.py' |
313 | --- nova/scheduler/api.py 2011-03-23 19:31:15 +0000 |
314 | +++ nova/scheduler/api.py 2011-03-24 19:09:30 +0000 |
315 | @@ -17,11 +17,21 @@ |
316 | Handles all requests relating to schedulers. |
317 | """ |
318 | |
319 | +import novaclient |
320 | + |
321 | +from nova import db |
322 | +from nova import exception |
323 | from nova import flags |
324 | from nova import log as logging |
325 | from nova import rpc |
326 | |
327 | +from eventlet import greenpool |
328 | + |
329 | FLAGS = flags.FLAGS |
330 | +flags.DEFINE_bool('enable_zone_routing', |
331 | + False, |
332 | + 'When True, routing to child zones will occur.') |
333 | + |
334 | LOG = logging.getLogger('nova.scheduler.api') |
335 | |
336 | |
337 | @@ -45,9 +55,27 @@ |
338 | items = _call_scheduler('get_zone_list', context) |
339 | for item in items: |
340 | item['api_url'] = item['api_url'].replace('\\/', '/') |
341 | + if not items: |
342 | + items = db.zone_get_all(context) |
343 | return items |
344 | |
345 | |
346 | +def zone_get(context, zone_id): |
347 | + return db.zone_get(context, zone_id) |
348 | + |
349 | + |
350 | +def zone_delete(context, zone_id): |
351 | + return db.zone_delete(context, zone_id) |
352 | + |
353 | + |
354 | +def zone_create(context, data): |
355 | + return db.zone_create(context, data) |
356 | + |
357 | + |
358 | +def zone_update(context, zone_id, data): |
359 | + return db.zone_update(context, zone_id, data) |
360 | + |
361 | + |
362 | def get_zone_capabilities(context, service=None): |
363 | """Returns a dict of key, value capabilities for this zone, |
364 | or for a particular class of services running in this zone.""" |
365 | @@ -62,3 +90,152 @@ |
366 | args=dict(service_name=service_name, host=host, |
367 | capabilities=capabilities)) |
368 | return rpc.fanout_cast(context, 'scheduler', kwargs) |
369 | + |
370 | + |
371 | +def _wrap_method(function, self): |
372 | + """Wrap method to supply self.""" |
373 | + def _wrap(*args, **kwargs): |
374 | + return function(self, *args, **kwargs) |
375 | + return _wrap |
376 | + |
377 | + |
378 | +def _process(func, zone): |
379 | + """Worker stub for green thread pool. Give the worker |
380 | + an authenticated nova client and zone info.""" |
381 | + nova = novaclient.OpenStack(zone.username, zone.password, zone.api_url) |
382 | + nova.authenticate() |
383 | + return func(nova, zone) |
384 | + |
385 | + |
386 | +def child_zone_helper(zone_list, func): |
387 | + """Fire off a command to each zone in the list. |
388 | + The return is [novaclient return objects] from each child zone. |
389 | + For example, if you are calling server.pause(), the list will |
390 | + be whatever the response from server.pause() is. One entry |
391 | + per child zone called.""" |
392 | + green_pool = greenpool.GreenPool() |
393 | + return [result for result in green_pool.imap( |
394 | + _wrap_method(_process, func), zone_list)] |
395 | + |
396 | + |
397 | +def _issue_novaclient_command(nova, zone, collection, method_name, item_id): |
398 | + """Use novaclient to issue command to a single child zone. |
399 | + One of these will be run in parallel for each child zone.""" |
400 | + manager = getattr(nova, collection) |
401 | + result = None |
402 | + try: |
403 | + try: |
404 | + result = manager.get(int(item_id)) |
405 | + except ValueError, e: |
406 | + result = manager.find(name=item_id) |
407 | + except novaclient.NotFound: |
408 | + url = zone.api_url |
409 | + LOG.debug(_("%(collection)s '%(item_id)s' not found on '%(url)s'" % |
410 | + locals())) |
411 | + return None |
412 | + |
413 | + if method_name.lower() not in ['get', 'find']: |
414 | + result = getattr(result, method_name)() |
415 | + return result |
416 | + |
417 | + |
418 | +def wrap_novaclient_function(f, collection, method_name, item_id): |
419 | + """Appends collection, method_name and item_id to the incoming |
420 | + (nova, zone) call from child_zone_helper.""" |
421 | + def inner(nova, zone): |
422 | + return f(nova, zone, collection, method_name, item_id) |
423 | + |
424 | + return inner |
425 | + |
426 | + |
427 | +class RedirectResult(exception.Error): |
428 | + """Used to the HTTP API know that these results are pre-cooked |
429 | + and they can be returned to the caller directly.""" |
430 | + def __init__(self, results): |
431 | + self.results = results |
432 | + super(RedirectResult, self).__init__( |
433 | + message=_("Uncaught Zone redirection exception")) |
434 | + |
435 | + |
436 | +class reroute_compute(object): |
437 | + """Decorator used to indicate that the method should |
438 | + delegate the call the child zones if the db query |
439 | + can't find anything.""" |
440 | + def __init__(self, method_name): |
441 | + self.method_name = method_name |
442 | + |
443 | + def __call__(self, f): |
444 | + def wrapped_f(*args, **kwargs): |
445 | + collection, context, item_id = \ |
446 | + self.get_collection_context_and_id(args, kwargs) |
447 | + try: |
448 | + # Call the original function ... |
449 | + return f(*args, **kwargs) |
450 | + except exception.InstanceNotFound, e: |
451 | + LOG.debug(_("Instance %(item_id)s not found " |
452 | + "locally: '%(e)s'" % locals())) |
453 | + |
454 | + if not FLAGS.enable_zone_routing: |
455 | + raise |
456 | + |
457 | + zones = db.zone_get_all(context) |
458 | + if not zones: |
459 | + raise |
460 | + |
461 | + # Ask the children to provide an answer ... |
462 | + LOG.debug(_("Asking child zones ...")) |
463 | + result = self._call_child_zones(zones, |
464 | + wrap_novaclient_function(_issue_novaclient_command, |
465 | + collection, self.method_name, item_id)) |
466 | + # Scrub the results and raise another exception |
467 | + # so the API layers can bail out gracefully ... |
468 | + raise RedirectResult(self.unmarshall_result(result)) |
469 | + return wrapped_f |
470 | + |
471 | + def _call_child_zones(self, zones, function): |
472 | + """Ask the child zones to perform this operation. |
473 | + Broken out for testing.""" |
474 | + return child_zone_helper(zones, function) |
475 | + |
476 | + def get_collection_context_and_id(self, args, kwargs): |
477 | + """Returns a tuple of (novaclient collection name, security |
478 | + context and resource id. Derived class should override this.""" |
479 | + context = kwargs.get('context', None) |
480 | + instance_id = kwargs.get('instance_id', None) |
481 | + if len(args) > 0 and not context: |
482 | + context = args[1] |
483 | + if len(args) > 1 and not instance_id: |
484 | + instance_id = args[2] |
485 | + return ("servers", context, instance_id) |
486 | + |
487 | + def unmarshall_result(self, zone_responses): |
488 | + """Result is a list of responses from each child zone. |
489 | + Each decorator derivation is responsible to turning this |
490 | + into a format expected by the calling method. For |
491 | + example, this one is expected to return a single Server |
492 | + dict {'server':{k:v}}. Others may return a list of them, like |
493 | + {'servers':[{k,v}]}""" |
494 | + reduced_response = [] |
495 | + for zone_response in zone_responses: |
496 | + if not zone_response: |
497 | + continue |
498 | + |
499 | + server = zone_response.__dict__ |
500 | + |
501 | + for k in server.keys(): |
502 | + if k[0] == '_' or k == 'manager': |
503 | + del server[k] |
504 | + |
505 | + reduced_response.append(dict(server=server)) |
506 | + if reduced_response: |
507 | + return reduced_response[0] # first for now. |
508 | + return {} |
509 | + |
510 | + |
511 | +def redirect_handler(f): |
512 | + def new_f(*args, **kwargs): |
513 | + try: |
514 | + return f(*args, **kwargs) |
515 | + except RedirectResult, e: |
516 | + return e.results |
517 | + return new_f |
518 | |
519 | === modified file 'nova/scheduler/zone_manager.py' |
520 | --- nova/scheduler/zone_manager.py 2011-02-24 23:23:15 +0000 |
521 | +++ nova/scheduler/zone_manager.py 2011-03-24 19:09:30 +0000 |
522 | @@ -58,8 +58,9 @@ |
523 | child zone.""" |
524 | self.last_seen = datetime.now() |
525 | self.attempt = 0 |
526 | - self.name = zone_metadata["name"] |
527 | - self.capabilities = zone_metadata["capabilities"] |
528 | + self.name = zone_metadata.get("name", "n/a") |
529 | + self.capabilities = ", ".join(["%s=%s" % (k, v) |
530 | + for k, v in zone_metadata.iteritems() if k != 'name']) |
531 | self.is_active = True |
532 | |
533 | def to_dict(self): |
534 | @@ -104,7 +105,7 @@ |
535 | """Keeps the zone states updated.""" |
536 | def __init__(self): |
537 | self.last_zone_db_check = datetime.min |
538 | - self.zone_states = {} |
539 | + self.zone_states = {} # { <zone_id> : ZoneState } |
540 | self.service_states = {} # { <service> : { <host> : { cap k : v }}} |
541 | self.green_pool = greenpool.GreenPool() |
542 | |
543 | |
544 | === modified file 'nova/tests/test_scheduler.py' |
545 | --- nova/tests/test_scheduler.py 2011-03-10 06:23:13 +0000 |
546 | +++ nova/tests/test_scheduler.py 2011-03-24 19:09:30 +0000 |
547 | @@ -21,6 +21,9 @@ |
548 | |
549 | import datetime |
550 | import mox |
551 | +import novaclient.exceptions |
552 | +import stubout |
553 | +import webob |
554 | |
555 | from mox import IgnoreArg |
556 | from nova import context |
557 | @@ -32,6 +35,7 @@ |
558 | from nova import rpc |
559 | from nova import utils |
560 | from nova.auth import manager as auth_manager |
561 | +from nova.scheduler import api |
562 | from nova.scheduler import manager |
563 | from nova.scheduler import driver |
564 | from nova.compute import power_state |
565 | @@ -937,3 +941,160 @@ |
566 | db.instance_destroy(self.context, instance_id) |
567 | db.service_destroy(self.context, s_ref['id']) |
568 | db.service_destroy(self.context, s_ref2['id']) |
569 | + |
570 | + |
571 | +class FakeZone(object): |
572 | + def __init__(self, api_url, username, password): |
573 | + self.api_url = api_url |
574 | + self.username = username |
575 | + self.password = password |
576 | + |
577 | + |
578 | +def zone_get_all(context): |
579 | + return [ |
580 | + FakeZone('http://example.com', 'bob', 'xxx'), |
581 | + ] |
582 | + |
583 | + |
584 | +class FakeRerouteCompute(api.reroute_compute): |
585 | + def _call_child_zones(self, zones, function): |
586 | + return [] |
587 | + |
588 | + def get_collection_context_and_id(self, args, kwargs): |
589 | + return ("servers", None, 1) |
590 | + |
591 | + def unmarshall_result(self, zone_responses): |
592 | + return dict(magic="found me") |
593 | + |
594 | + |
595 | +def go_boom(self, context, instance): |
596 | + raise exception.InstanceNotFound("boom message", instance) |
597 | + |
598 | + |
599 | +def found_instance(self, context, instance): |
600 | + return dict(name='myserver') |
601 | + |
602 | + |
603 | +class FakeResource(object): |
604 | + def __init__(self, attribute_dict): |
605 | + for k, v in attribute_dict.iteritems(): |
606 | + setattr(self, k, v) |
607 | + |
608 | + def pause(self): |
609 | + pass |
610 | + |
611 | + |
612 | +class ZoneRedirectTest(test.TestCase): |
613 | + def setUp(self): |
614 | + super(ZoneRedirectTest, self).setUp() |
615 | + self.stubs = stubout.StubOutForTesting() |
616 | + |
617 | + self.stubs.Set(db, 'zone_get_all', zone_get_all) |
618 | + |
619 | + self.enable_zone_routing = FLAGS.enable_zone_routing |
620 | + FLAGS.enable_zone_routing = True |
621 | + |
622 | + def tearDown(self): |
623 | + self.stubs.UnsetAll() |
624 | + FLAGS.enable_zone_routing = self.enable_zone_routing |
625 | + super(ZoneRedirectTest, self).tearDown() |
626 | + |
627 | + def test_trap_found_locally(self): |
628 | + decorator = FakeRerouteCompute("foo") |
629 | + try: |
630 | + result = decorator(found_instance)(None, None, 1) |
631 | + except api.RedirectResult, e: |
632 | + self.fail(_("Successful database hit should succeed")) |
633 | + |
634 | + def test_trap_not_found_locally(self): |
635 | + decorator = FakeRerouteCompute("foo") |
636 | + try: |
637 | + result = decorator(go_boom)(None, None, 1) |
638 | + self.assertFail(_("Should have rerouted.")) |
639 | + except api.RedirectResult, e: |
640 | + self.assertEquals(e.results['magic'], 'found me') |
641 | + |
642 | + def test_routing_flags(self): |
643 | + FLAGS.enable_zone_routing = False |
644 | + decorator = FakeRerouteCompute("foo") |
645 | + try: |
646 | + result = decorator(go_boom)(None, None, 1) |
647 | + self.assertFail(_("Should have thrown exception.")) |
648 | + except exception.InstanceNotFound, e: |
649 | + self.assertEquals(e.message, 'boom message') |
650 | + |
651 | + def test_get_collection_context_and_id(self): |
652 | + decorator = api.reroute_compute("foo") |
653 | + self.assertEquals(decorator.get_collection_context_and_id( |
654 | + (None, 10, 20), {}), ("servers", 10, 20)) |
655 | + self.assertEquals(decorator.get_collection_context_and_id( |
656 | + (None, 11,), dict(instance_id=21)), ("servers", 11, 21)) |
657 | + self.assertEquals(decorator.get_collection_context_and_id( |
658 | + (None,), dict(context=12, instance_id=22)), ("servers", 12, 22)) |
659 | + |
660 | + def test_unmarshal_single_server(self): |
661 | + decorator = api.reroute_compute("foo") |
662 | + self.assertEquals(decorator.unmarshall_result([]), {}) |
663 | + self.assertEquals(decorator.unmarshall_result( |
664 | + [FakeResource(dict(a=1, b=2)), ]), |
665 | + dict(server=dict(a=1, b=2))) |
666 | + self.assertEquals(decorator.unmarshall_result( |
667 | + [FakeResource(dict(a=1, _b=2)), ]), |
668 | + dict(server=dict(a=1,))) |
669 | + self.assertEquals(decorator.unmarshall_result( |
670 | + [FakeResource(dict(a=1, manager=2)), ]), |
671 | + dict(server=dict(a=1,))) |
672 | + self.assertEquals(decorator.unmarshall_result( |
673 | + [FakeResource(dict(_a=1, manager=2)), ]), |
674 | + dict(server={})) |
675 | + |
676 | + |
677 | +class FakeServerCollection(object): |
678 | + def get(self, instance_id): |
679 | + return FakeResource(dict(a=10, b=20)) |
680 | + |
681 | + def find(self, name): |
682 | + return FakeResource(dict(a=11, b=22)) |
683 | + |
684 | + |
685 | +class FakeEmptyServerCollection(object): |
686 | + def get(self, f): |
687 | + raise novaclient.NotFound(1) |
688 | + |
689 | + def find(self, name): |
690 | + raise novaclient.NotFound(2) |
691 | + |
692 | + |
693 | +class FakeNovaClient(object): |
694 | + def __init__(self, collection): |
695 | + self.servers = collection |
696 | + |
697 | + |
698 | +class DynamicNovaClientTest(test.TestCase): |
699 | + def test_issue_novaclient_command_found(self): |
700 | + zone = FakeZone('http://example.com', 'bob', 'xxx') |
701 | + self.assertEquals(api._issue_novaclient_command( |
702 | + FakeNovaClient(FakeServerCollection()), |
703 | + zone, "servers", "get", 100).a, 10) |
704 | + |
705 | + self.assertEquals(api._issue_novaclient_command( |
706 | + FakeNovaClient(FakeServerCollection()), |
707 | + zone, "servers", "find", "name").b, 22) |
708 | + |
709 | + self.assertEquals(api._issue_novaclient_command( |
710 | + FakeNovaClient(FakeServerCollection()), |
711 | + zone, "servers", "pause", 100), None) |
712 | + |
713 | + def test_issue_novaclient_command_not_found(self): |
714 | + zone = FakeZone('http://example.com', 'bob', 'xxx') |
715 | + self.assertEquals(api._issue_novaclient_command( |
716 | + FakeNovaClient(FakeEmptyServerCollection()), |
717 | + zone, "servers", "get", 100), None) |
718 | + |
719 | + self.assertEquals(api._issue_novaclient_command( |
720 | + FakeNovaClient(FakeEmptyServerCollection()), |
721 | + zone, "servers", "find", "name"), None) |
722 | + |
723 | + self.assertEquals(api._issue_novaclient_command( |
724 | + FakeNovaClient(FakeEmptyServerCollection()), |
725 | + zone, "servers", "any", "name"), None) |
Hi Sandy,
I don't think WSGI middleware is the correct place to be handling zones forwarding. What if we write a tool directly against nova.compute.api or if some other component (like a compute or network worker) issues a nova.compute.api call? I think all logic and routing should instead be handled inside inside nova.compute and/or nova.scheduler. Nothing in nova.api should ever be aware of what is happening.