Merge lp:~sandy-walsh/nova/zones4 into lp:~hudson-openstack/nova/trunk

Proposed by Sandy Walsh
Status: Merged
Approved by: Rick Harris
Approved revision: 720
Merged at revision: 874
Proposed branch: lp:~sandy-walsh/nova/zones4
Merge into: lp:~hudson-openstack/nova/trunk
Diff against target: 725 lines (+387/-15)
7 files modified
nova/api/openstack/servers.py (+20/-1)
nova/api/openstack/zones.py (+8/-10)
nova/compute/api.py (+16/-1)
nova/db/api.py (+1/-0)
nova/scheduler/api.py (+177/-0)
nova/scheduler/zone_manager.py (+4/-3)
nova/tests/test_scheduler.py (+161/-0)
To merge this branch: bzr merge lp:~sandy-walsh/nova/zones4
Reviewer Review Type Date Requested Status
Rick Harris (community) Approve
Matt Dietz (community) Approve
Review via email: mp+54760@code.launchpad.net

This proposal supersedes a proposal from 2011-03-17.

Description of the change

In this branch we are forwarding incoming requests to child zones when the requested resource is not found in the current zone.

For example: If 'nova pause 123' is issued against Zone 1, but instance 123 does not live in Zone 1, the call will be forwarded to all child zones hoping someone can deal with it.

NOTE: This currently only works with OpenStack API requests and routing checks are only being done against Compute/instance_id checks.
Specifically:
* servers.get/pause/unpause/diagnostics/suspend/resume/rescue/unrescue/delete
* servers.create is pending for distributed scheduler
* servers.get_all will get added early in Diablo.

What I've been doing for testing:
1. Set up a Nova deployment in a VM (Zone0)
2. Clone the VM and set --zone_name=zone1 (and change all the IP addresses to the new address in nova.conf, glance.conf and novarc)
3. Set --enable_zone_routing=true on all zones
4. use the --connection_type=fake driver for compute to keep things easy
5. Add Zone1 as a child of Zone0 (nova zone-add)

(make sure the instance id's are different in each zone)

Example of calls being sent to child zones:
http://paste.openstack.org/show/964/

To post a comment you must log in.
Revision history for this message
Eric Day (eday) wrote : Posted in a previous version of this proposal

Hi Sandy,

I don't think WSGI middleware is the correct place to be handling zones forwarding. What if we write a tool directly against nova.compute.api or if some other component (like a compute or network worker) issues a nova.compute.api call? I think all logic and routing should instead be handled inside inside nova.compute and/or nova.scheduler. Nothing in nova.api should ever be aware of what is happening.

Revision history for this message
Sandy Walsh (sandy-walsh) wrote : Posted in a previous version of this proposal

Thanks Eric,

That was the way I was initially intending to go.

The closer I got to it, I noticed that all calls would need to be re-marshaled to be sent to the children. We'd need a new client library abstraction layer (since forwarded requests may be OS API or EC2 API). I was hoping to avoid all that by simply forwarding the already marshaled message.

This client library abstraction is likely to get a little unwieldy since it needs to support OS/EC2 and whomever else comes along (actually, sounds like another endorsement for DirectAPI.)

Also, with this approach I was assuming an idiom of "API checks parameters and bails if they're wrong. Work is done in the services." ... but I see the counter-argument.

Let me think more about client library abstraction. Since we have big API changes coming soon, might be timely to consider this.

Revision history for this message
Sandy Walsh (sandy-walsh) wrote : Posted in a previous version of this proposal

Thinking a little more about it, but I've got a concern about that approach. Not for technical reasons, but for schedule/political reasons: Since API is such a touchy subject and still under development, this could turn in a major blocker against getting anything practical done with Zones for a long time.

Revision history for this message
Eric Day (eday) wrote : Posted in a previous version of this proposal

FWIW, I think making the call to novaclient (and re-marshal) from nova.compute.* is the correct thing to do. I would move forward with the OpenStack API for this as well, as that seems to be where we are committed at the moment. If we need to change this in the future, it will be a simple HTTP formatting issue inside novaclient (the API into novaclient shouldn't change).

Revision history for this message
Sandy Walsh (sandy-walsh) wrote : Posted in a previous version of this proposal

Yup ... that's the direction I'll take. Thanks again Eric.

Revision history for this message
Sandy Walsh (sandy-walsh) wrote : Posted in a previous version of this proposal

Ok, all refactored out of middleware now. The <service>/api.py does the checking now and the external api just bails out after a redirect. It's all handled in decorators now.

Revision history for this message
Rick Harris (rconradharris) wrote : Posted in a previous version of this proposal

Nice work, Sandy.

Some really minor nits/suggestions:

> 402 + try:
> 403 + manager = getattr(nova, collection)

The getattr can be outside of the try-block (AFAICT).

> 404 + if isinstance(item_id, int) or item_id.isdigit():
> 405 + result = manager.get(int(item_id))
> 406 + else:
> 407 + result = manager.find(name=item_id)

One common way of handling this is to just go ahead and cast to int and
handle the possible ValueError (EAFP), like:

    try:
        result = manager.get(int(item_id))
    except ValueError:
        result = manager.find(name=item_id)

> 371 +def _wrap_method(function, self):
> 372 + """Wrap method to supply self."""
> 373 + def _wrap(*args, **kwargs):
> 374 + return function(self, *args, **kwargs)
> 375 + return _wrap

For the newly added decorators, it might be worth using @functools.wraps so
that the outer-func inherits the inner-funcs attributes. Can make debugging a
little easier.

> 397 +def _issue_novaclient_command(nova, zone, collection, method_name, \
> 398 + item_id):

Trailing '\' isn't needed.

Could line-up item_id with opening param.

> 264 + @scheduler_api.reroute_compute("diagnostics")
> 265 def get_diagnostics(self, context, instance_id):

Should that be 'get_diagnostics'? I ask because it's the only one that differs
in that regard.

review: Needs Fixing
Revision history for this message
Matt Dietz (cerberus) wrote : Posted in a previous version of this proposal

The exception raising idea is really interesting. Hard to follow at first, but the tests make it clear IMO. I think this looks pretty pretty promising.

review: Approve
Revision history for this message
Sandy Walsh (sandy-walsh) wrote : Posted in a previous version of this proposal

Thanks rick ... great feedback. I've implemented your changes.

re: get_diagnostics, the name in the decorator is the name of the method in novaclient, so it may not map 1:1 to the method being wrapped (as in this case).

Basically the decorator is saying, "if this method can't find the instance, check with the child zones by calling <method> in novaclient"

Revision history for this message
Sandy Walsh (sandy-walsh) wrote : Posted in a previous version of this proposal

(oh, push pending, will change WIP to let you know)

Revision history for this message
Rick Harris (rconradharris) wrote : Posted in a previous version of this proposal

> 788 +def redirect_handler(f):
> 648 +def _wrap_method(function, self):

Didn't like the idea of using @functools.wraps to propagate the inner func's metadata?

review: Needs Information
Revision history for this message
Sandy Walsh (sandy-walsh) wrote : Posted in a previous version of this proposal

I do. Good suggestion. But I'll fix that up on the next branch. Haven't spent enough time with it and don't want to miss the window today.

Revision history for this message
Rick Harris (rconradharris) wrote : Posted in a previous version of this proposal

Good deal, lgtm. Thanks!

review: Approve
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Posted in a previous version of this proposal

No proposals found for merge of lp:~sandy-walsh/nova/zones3 into lp:nova.

lp:~sandy-walsh/nova/zones4 updated
720. By Sandy Walsh

trunk merge

Revision history for this message
Matt Dietz (cerberus) wrote :

Trying again.

review: Approve
Revision history for this message
Rick Harris (rconradharris) wrote :

Rej....Approved!

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'nova/api/openstack/servers.py'
--- nova/api/openstack/servers.py 2011-03-24 17:59:25 +0000
+++ nova/api/openstack/servers.py 2011-03-24 19:09:30 +0000
@@ -37,6 +37,7 @@
37from nova.compute import instance_types37from nova.compute import instance_types
38from nova.compute import power_state38from nova.compute import power_state
39import nova.api.openstack39import nova.api.openstack
40from nova.scheduler import api as scheduler_api
4041
4142
42LOG = logging.getLogger('server')43LOG = logging.getLogger('server')
@@ -87,15 +88,18 @@
87 for inst in limited_list]88 for inst in limited_list]
88 return dict(servers=servers)89 return dict(servers=servers)
8990
91 @scheduler_api.redirect_handler
90 def show(self, req, id):92 def show(self, req, id):
91 """ Returns server details by server id """93 """ Returns server details by server id """
92 try:94 try:
93 instance = self.compute_api.get(req.environ['nova.context'], id)95 instance = self.compute_api.routing_get(
96 req.environ['nova.context'], id)
94 builder = self._get_view_builder(req)97 builder = self._get_view_builder(req)
95 return builder.build(instance, is_detail=True)98 return builder.build(instance, is_detail=True)
96 except exception.NotFound:99 except exception.NotFound:
97 return faults.Fault(exc.HTTPNotFound())100 return faults.Fault(exc.HTTPNotFound())
98101
102 @scheduler_api.redirect_handler
99 def delete(self, req, id):103 def delete(self, req, id):
100 """ Destroys a server """104 """ Destroys a server """
101 try:105 try:
@@ -226,6 +230,7 @@
226 # if the original error is okay, just reraise it230 # if the original error is okay, just reraise it
227 raise error231 raise error
228232
233 @scheduler_api.redirect_handler
229 def update(self, req, id):234 def update(self, req, id):
230 """ Updates the server name or password """235 """ Updates the server name or password """
231 if len(req.body) == 0:236 if len(req.body) == 0:
@@ -251,6 +256,7 @@
251 return faults.Fault(exc.HTTPNotFound())256 return faults.Fault(exc.HTTPNotFound())
252 return exc.HTTPNoContent()257 return exc.HTTPNoContent()
253258
259 @scheduler_api.redirect_handler
254 def action(self, req, id):260 def action(self, req, id):
255 """Multi-purpose method used to reboot, rebuild, or261 """Multi-purpose method used to reboot, rebuild, or
256 resize a server"""262 resize a server"""
@@ -316,6 +322,7 @@
316 return faults.Fault(exc.HTTPUnprocessableEntity())322 return faults.Fault(exc.HTTPUnprocessableEntity())
317 return exc.HTTPAccepted()323 return exc.HTTPAccepted()
318324
325 @scheduler_api.redirect_handler
319 def lock(self, req, id):326 def lock(self, req, id):
320 """327 """
321 lock the instance with id328 lock the instance with id
@@ -331,6 +338,7 @@
331 return faults.Fault(exc.HTTPUnprocessableEntity())338 return faults.Fault(exc.HTTPUnprocessableEntity())
332 return exc.HTTPAccepted()339 return exc.HTTPAccepted()
333340
341 @scheduler_api.redirect_handler
334 def unlock(self, req, id):342 def unlock(self, req, id):
335 """343 """
336 unlock the instance with id344 unlock the instance with id
@@ -346,6 +354,7 @@
346 return faults.Fault(exc.HTTPUnprocessableEntity())354 return faults.Fault(exc.HTTPUnprocessableEntity())
347 return exc.HTTPAccepted()355 return exc.HTTPAccepted()
348356
357 @scheduler_api.redirect_handler
349 def get_lock(self, req, id):358 def get_lock(self, req, id):
350 """359 """
351 return the boolean state of (instance with id)'s lock360 return the boolean state of (instance with id)'s lock
@@ -360,6 +369,7 @@
360 return faults.Fault(exc.HTTPUnprocessableEntity())369 return faults.Fault(exc.HTTPUnprocessableEntity())
361 return exc.HTTPAccepted()370 return exc.HTTPAccepted()
362371
372 @scheduler_api.redirect_handler
363 def reset_network(self, req, id):373 def reset_network(self, req, id):
364 """374 """
365 Reset networking on an instance (admin only).375 Reset networking on an instance (admin only).
@@ -374,6 +384,7 @@
374 return faults.Fault(exc.HTTPUnprocessableEntity())384 return faults.Fault(exc.HTTPUnprocessableEntity())
375 return exc.HTTPAccepted()385 return exc.HTTPAccepted()
376386
387 @scheduler_api.redirect_handler
377 def inject_network_info(self, req, id):388 def inject_network_info(self, req, id):
378 """389 """
379 Inject network info for an instance (admin only).390 Inject network info for an instance (admin only).
@@ -388,6 +399,7 @@
388 return faults.Fault(exc.HTTPUnprocessableEntity())399 return faults.Fault(exc.HTTPUnprocessableEntity())
389 return exc.HTTPAccepted()400 return exc.HTTPAccepted()
390401
402 @scheduler_api.redirect_handler
391 def pause(self, req, id):403 def pause(self, req, id):
392 """ Permit Admins to Pause the server. """404 """ Permit Admins to Pause the server. """
393 ctxt = req.environ['nova.context']405 ctxt = req.environ['nova.context']
@@ -399,6 +411,7 @@
399 return faults.Fault(exc.HTTPUnprocessableEntity())411 return faults.Fault(exc.HTTPUnprocessableEntity())
400 return exc.HTTPAccepted()412 return exc.HTTPAccepted()
401413
414 @scheduler_api.redirect_handler
402 def unpause(self, req, id):415 def unpause(self, req, id):
403 """ Permit Admins to Unpause the server. """416 """ Permit Admins to Unpause the server. """
404 ctxt = req.environ['nova.context']417 ctxt = req.environ['nova.context']
@@ -410,6 +423,7 @@
410 return faults.Fault(exc.HTTPUnprocessableEntity())423 return faults.Fault(exc.HTTPUnprocessableEntity())
411 return exc.HTTPAccepted()424 return exc.HTTPAccepted()
412425
426 @scheduler_api.redirect_handler
413 def suspend(self, req, id):427 def suspend(self, req, id):
414 """permit admins to suspend the server"""428 """permit admins to suspend the server"""
415 context = req.environ['nova.context']429 context = req.environ['nova.context']
@@ -421,6 +435,7 @@
421 return faults.Fault(exc.HTTPUnprocessableEntity())435 return faults.Fault(exc.HTTPUnprocessableEntity())
422 return exc.HTTPAccepted()436 return exc.HTTPAccepted()
423437
438 @scheduler_api.redirect_handler
424 def resume(self, req, id):439 def resume(self, req, id):
425 """permit admins to resume the server from suspend"""440 """permit admins to resume the server from suspend"""
426 context = req.environ['nova.context']441 context = req.environ['nova.context']
@@ -432,6 +447,7 @@
432 return faults.Fault(exc.HTTPUnprocessableEntity())447 return faults.Fault(exc.HTTPUnprocessableEntity())
433 return exc.HTTPAccepted()448 return exc.HTTPAccepted()
434449
450 @scheduler_api.redirect_handler
435 def rescue(self, req, id):451 def rescue(self, req, id):
436 """Permit users to rescue the server."""452 """Permit users to rescue the server."""
437 context = req.environ["nova.context"]453 context = req.environ["nova.context"]
@@ -443,6 +459,7 @@
443 return faults.Fault(exc.HTTPUnprocessableEntity())459 return faults.Fault(exc.HTTPUnprocessableEntity())
444 return exc.HTTPAccepted()460 return exc.HTTPAccepted()
445461
462 @scheduler_api.redirect_handler
446 def unrescue(self, req, id):463 def unrescue(self, req, id):
447 """Permit users to unrescue the server."""464 """Permit users to unrescue the server."""
448 context = req.environ["nova.context"]465 context = req.environ["nova.context"]
@@ -454,6 +471,7 @@
454 return faults.Fault(exc.HTTPUnprocessableEntity())471 return faults.Fault(exc.HTTPUnprocessableEntity())
455 return exc.HTTPAccepted()472 return exc.HTTPAccepted()
456473
474 @scheduler_api.redirect_handler
457 def get_ajax_console(self, req, id):475 def get_ajax_console(self, req, id):
458 """ Returns a url to an instance's ajaxterm console. """476 """ Returns a url to an instance's ajaxterm console. """
459 try:477 try:
@@ -463,6 +481,7 @@
463 return faults.Fault(exc.HTTPNotFound())481 return faults.Fault(exc.HTTPNotFound())
464 return exc.HTTPAccepted()482 return exc.HTTPAccepted()
465483
484 @scheduler_api.redirect_handler
466 def diagnostics(self, req, id):485 def diagnostics(self, req, id):
467 """Permit Admins to retrieve server diagnostics."""486 """Permit Admins to retrieve server diagnostics."""
468 ctxt = req.environ["nova.context"]487 ctxt = req.environ["nova.context"]
469488
=== modified file 'nova/api/openstack/zones.py'
--- nova/api/openstack/zones.py 2011-03-23 19:31:15 +0000
+++ nova/api/openstack/zones.py 2011-03-24 19:09:30 +0000
@@ -17,6 +17,7 @@
1717
18from nova import db18from nova import db
19from nova import flags19from nova import flags
20from nova import log as logging
20from nova import wsgi21from nova import wsgi
21from nova.scheduler import api22from nova.scheduler import api
2223
@@ -38,7 +39,8 @@
3839
3940
40def _scrub_zone(zone):41def _scrub_zone(zone):
41 return _filter_keys(zone, ('id', 'api_url'))42 return _exclude_keys(zone, ('username', 'password', 'created_at',
43 'deleted', 'deleted_at', 'updated_at'))
4244
4345
44class Controller(wsgi.Controller):46class Controller(wsgi.Controller):
@@ -53,12 +55,8 @@
53 # Ask the ZoneManager in the Scheduler for most recent data,55 # Ask the ZoneManager in the Scheduler for most recent data,
54 # or fall-back to the database ...56 # or fall-back to the database ...
55 items = api.get_zone_list(req.environ['nova.context'])57 items = api.get_zone_list(req.environ['nova.context'])
56 if not items:
57 items = db.zone_get_all(req.environ['nova.context'])
58
59 items = common.limited(items, req)58 items = common.limited(items, req)
60 items = [_exclude_keys(item, ['username', 'password'])59 items = [_scrub_zone(item) for item in items]
61 for item in items]
62 return dict(zones=items)60 return dict(zones=items)
6361
64 def detail(self, req):62 def detail(self, req):
@@ -81,23 +79,23 @@
81 def show(self, req, id):79 def show(self, req, id):
82 """Return data about the given zone id"""80 """Return data about the given zone id"""
83 zone_id = int(id)81 zone_id = int(id)
84 zone = db.zone_get(req.environ['nova.context'], zone_id)82 zone = api.zone_get(req.environ['nova.context'], zone_id)
85 return dict(zone=_scrub_zone(zone))83 return dict(zone=_scrub_zone(zone))
8684
87 def delete(self, req, id):85 def delete(self, req, id):
88 zone_id = int(id)86 zone_id = int(id)
89 db.zone_delete(req.environ['nova.context'], zone_id)87 api.zone_delete(req.environ['nova.context'], zone_id)
90 return {}88 return {}
9189
92 def create(self, req):90 def create(self, req):
93 context = req.environ['nova.context']91 context = req.environ['nova.context']
94 env = self._deserialize(req.body, req.get_content_type())92 env = self._deserialize(req.body, req.get_content_type())
95 zone = db.zone_create(context, env["zone"])93 zone = api.zone_create(context, env["zone"])
96 return dict(zone=_scrub_zone(zone))94 return dict(zone=_scrub_zone(zone))
9795
98 def update(self, req, id):96 def update(self, req, id):
99 context = req.environ['nova.context']97 context = req.environ['nova.context']
100 env = self._deserialize(req.body, req.get_content_type())98 env = self._deserialize(req.body, req.get_content_type())
101 zone_id = int(id)99 zone_id = int(id)
102 zone = db.zone_update(context, zone_id, env["zone"])100 zone = api.zone_update(context, zone_id, env["zone"])
103 return dict(zone=_scrub_zone(zone))101 return dict(zone=_scrub_zone(zone))
104102
=== modified file 'nova/compute/api.py'
--- nova/compute/api.py 2011-03-24 17:53:54 +0000
+++ nova/compute/api.py 2011-03-24 19:09:30 +0000
@@ -34,6 +34,7 @@
34from nova import utils34from nova import utils
35from nova import volume35from nova import volume
36from nova.compute import instance_types36from nova.compute import instance_types
37from nova.scheduler import api as scheduler_api
37from nova.db import base38from nova.db import base
3839
39FLAGS = flags.FLAGS40FLAGS = flags.FLAGS
@@ -352,6 +353,7 @@
352 rv = self.db.instance_update(context, instance_id, kwargs)353 rv = self.db.instance_update(context, instance_id, kwargs)
353 return dict(rv.iteritems())354 return dict(rv.iteritems())
354355
356 @scheduler_api.reroute_compute("delete")
355 def delete(self, context, instance_id):357 def delete(self, context, instance_id):
356 LOG.debug(_("Going to try to terminate %s"), instance_id)358 LOG.debug(_("Going to try to terminate %s"), instance_id)
357 try:359 try:
@@ -384,6 +386,13 @@
384 rv = self.db.instance_get(context, instance_id)386 rv = self.db.instance_get(context, instance_id)
385 return dict(rv.iteritems())387 return dict(rv.iteritems())
386388
389 @scheduler_api.reroute_compute("get")
390 def routing_get(self, context, instance_id):
391 """Use this method instead of get() if this is the only
392 operation you intend to to. It will route to novaclient.get
393 if the instance is not found."""
394 return self.get(context, instance_id)
395
387 def get_all(self, context, project_id=None, reservation_id=None,396 def get_all(self, context, project_id=None, reservation_id=None,
388 fixed_ip=None):397 fixed_ip=None):
389 """Get all instances, possibly filtered by one of the398 """Get all instances, possibly filtered by one of the
@@ -527,14 +536,17 @@
527 "instance_id": instance_id,536 "instance_id": instance_id,
528 "flavor_id": flavor_id}})537 "flavor_id": flavor_id}})
529538
539 @scheduler_api.reroute_compute("pause")
530 def pause(self, context, instance_id):540 def pause(self, context, instance_id):
531 """Pause the given instance."""541 """Pause the given instance."""
532 self._cast_compute_message('pause_instance', context, instance_id)542 self._cast_compute_message('pause_instance', context, instance_id)
533543
544 @scheduler_api.reroute_compute("unpause")
534 def unpause(self, context, instance_id):545 def unpause(self, context, instance_id):
535 """Unpause the given instance."""546 """Unpause the given instance."""
536 self._cast_compute_message('unpause_instance', context, instance_id)547 self._cast_compute_message('unpause_instance', context, instance_id)
537548
549 @scheduler_api.reroute_compute("diagnostics")
538 def get_diagnostics(self, context, instance_id):550 def get_diagnostics(self, context, instance_id):
539 """Retrieve diagnostics for the given instance."""551 """Retrieve diagnostics for the given instance."""
540 return self._call_compute_message(552 return self._call_compute_message(
@@ -546,18 +558,22 @@
546 """Retrieve actions for the given instance."""558 """Retrieve actions for the given instance."""
547 return self.db.instance_get_actions(context, instance_id)559 return self.db.instance_get_actions(context, instance_id)
548560
561 @scheduler_api.reroute_compute("suspend")
549 def suspend(self, context, instance_id):562 def suspend(self, context, instance_id):
550 """suspend the instance with instance_id"""563 """suspend the instance with instance_id"""
551 self._cast_compute_message('suspend_instance', context, instance_id)564 self._cast_compute_message('suspend_instance', context, instance_id)
552565
566 @scheduler_api.reroute_compute("resume")
553 def resume(self, context, instance_id):567 def resume(self, context, instance_id):
554 """resume the instance with instance_id"""568 """resume the instance with instance_id"""
555 self._cast_compute_message('resume_instance', context, instance_id)569 self._cast_compute_message('resume_instance', context, instance_id)
556570
571 @scheduler_api.reroute_compute("rescue")
557 def rescue(self, context, instance_id):572 def rescue(self, context, instance_id):
558 """Rescue the given instance."""573 """Rescue the given instance."""
559 self._cast_compute_message('rescue_instance', context, instance_id)574 self._cast_compute_message('rescue_instance', context, instance_id)
560575
576 @scheduler_api.reroute_compute("unrescue")
561 def unrescue(self, context, instance_id):577 def unrescue(self, context, instance_id):
562 """Unrescue the given instance."""578 """Unrescue the given instance."""
563 self._cast_compute_message('unrescue_instance', context, instance_id)579 self._cast_compute_message('unrescue_instance', context, instance_id)
@@ -573,7 +589,6 @@
573589
574 def get_ajax_console(self, context, instance_id):590 def get_ajax_console(self, context, instance_id):
575 """Get a url to an AJAX Console"""591 """Get a url to an AJAX Console"""
576 instance = self.get(context, instance_id)
577 output = self._call_compute_message('get_ajax_console',592 output = self._call_compute_message('get_ajax_console',
578 context,593 context,
579 instance_id)594 instance_id)
580595
=== modified file 'nova/db/api.py'
--- nova/db/api.py 2011-03-24 18:02:04 +0000
+++ nova/db/api.py 2011-03-24 19:09:30 +0000
@@ -71,6 +71,7 @@
71 """No more available blades"""71 """No more available blades"""
72 pass72 pass
7373
74
74###################75###################
7576
7677
7778
=== modified file 'nova/scheduler/api.py'
--- nova/scheduler/api.py 2011-03-23 19:31:15 +0000
+++ nova/scheduler/api.py 2011-03-24 19:09:30 +0000
@@ -17,11 +17,21 @@
17Handles all requests relating to schedulers.17Handles all requests relating to schedulers.
18"""18"""
1919
20import novaclient
21
22from nova import db
23from nova import exception
20from nova import flags24from nova import flags
21from nova import log as logging25from nova import log as logging
22from nova import rpc26from nova import rpc
2327
28from eventlet import greenpool
29
24FLAGS = flags.FLAGS30FLAGS = flags.FLAGS
31flags.DEFINE_bool('enable_zone_routing',
32 False,
33 'When True, routing to child zones will occur.')
34
25LOG = logging.getLogger('nova.scheduler.api')35LOG = logging.getLogger('nova.scheduler.api')
2636
2737
@@ -45,9 +55,27 @@
45 items = _call_scheduler('get_zone_list', context)55 items = _call_scheduler('get_zone_list', context)
46 for item in items:56 for item in items:
47 item['api_url'] = item['api_url'].replace('\\/', '/')57 item['api_url'] = item['api_url'].replace('\\/', '/')
58 if not items:
59 items = db.zone_get_all(context)
48 return items60 return items
4961
5062
63def zone_get(context, zone_id):
64 return db.zone_get(context, zone_id)
65
66
67def zone_delete(context, zone_id):
68 return db.zone_delete(context, zone_id)
69
70
71def zone_create(context, data):
72 return db.zone_create(context, data)
73
74
75def zone_update(context, zone_id, data):
76 return db.zone_update(context, zone_id, data)
77
78
51def get_zone_capabilities(context, service=None):79def get_zone_capabilities(context, service=None):
52 """Returns a dict of key, value capabilities for this zone,80 """Returns a dict of key, value capabilities for this zone,
53 or for a particular class of services running in this zone."""81 or for a particular class of services running in this zone."""
@@ -62,3 +90,152 @@
62 args=dict(service_name=service_name, host=host,90 args=dict(service_name=service_name, host=host,
63 capabilities=capabilities))91 capabilities=capabilities))
64 return rpc.fanout_cast(context, 'scheduler', kwargs)92 return rpc.fanout_cast(context, 'scheduler', kwargs)
93
94
95def _wrap_method(function, self):
96 """Wrap method to supply self."""
97 def _wrap(*args, **kwargs):
98 return function(self, *args, **kwargs)
99 return _wrap
100
101
102def _process(func, zone):
103 """Worker stub for green thread pool. Give the worker
104 an authenticated nova client and zone info."""
105 nova = novaclient.OpenStack(zone.username, zone.password, zone.api_url)
106 nova.authenticate()
107 return func(nova, zone)
108
109
110def child_zone_helper(zone_list, func):
111 """Fire off a command to each zone in the list.
112 The return is [novaclient return objects] from each child zone.
113 For example, if you are calling server.pause(), the list will
114 be whatever the response from server.pause() is. One entry
115 per child zone called."""
116 green_pool = greenpool.GreenPool()
117 return [result for result in green_pool.imap(
118 _wrap_method(_process, func), zone_list)]
119
120
121def _issue_novaclient_command(nova, zone, collection, method_name, item_id):
122 """Use novaclient to issue command to a single child zone.
123 One of these will be run in parallel for each child zone."""
124 manager = getattr(nova, collection)
125 result = None
126 try:
127 try:
128 result = manager.get(int(item_id))
129 except ValueError, e:
130 result = manager.find(name=item_id)
131 except novaclient.NotFound:
132 url = zone.api_url
133 LOG.debug(_("%(collection)s '%(item_id)s' not found on '%(url)s'" %
134 locals()))
135 return None
136
137 if method_name.lower() not in ['get', 'find']:
138 result = getattr(result, method_name)()
139 return result
140
141
142def wrap_novaclient_function(f, collection, method_name, item_id):
143 """Appends collection, method_name and item_id to the incoming
144 (nova, zone) call from child_zone_helper."""
145 def inner(nova, zone):
146 return f(nova, zone, collection, method_name, item_id)
147
148 return inner
149
150
151class RedirectResult(exception.Error):
152 """Used to the HTTP API know that these results are pre-cooked
153 and they can be returned to the caller directly."""
154 def __init__(self, results):
155 self.results = results
156 super(RedirectResult, self).__init__(
157 message=_("Uncaught Zone redirection exception"))
158
159
160class reroute_compute(object):
161 """Decorator used to indicate that the method should
162 delegate the call the child zones if the db query
163 can't find anything."""
164 def __init__(self, method_name):
165 self.method_name = method_name
166
167 def __call__(self, f):
168 def wrapped_f(*args, **kwargs):
169 collection, context, item_id = \
170 self.get_collection_context_and_id(args, kwargs)
171 try:
172 # Call the original function ...
173 return f(*args, **kwargs)
174 except exception.InstanceNotFound, e:
175 LOG.debug(_("Instance %(item_id)s not found "
176 "locally: '%(e)s'" % locals()))
177
178 if not FLAGS.enable_zone_routing:
179 raise
180
181 zones = db.zone_get_all(context)
182 if not zones:
183 raise
184
185 # Ask the children to provide an answer ...
186 LOG.debug(_("Asking child zones ..."))
187 result = self._call_child_zones(zones,
188 wrap_novaclient_function(_issue_novaclient_command,
189 collection, self.method_name, item_id))
190 # Scrub the results and raise another exception
191 # so the API layers can bail out gracefully ...
192 raise RedirectResult(self.unmarshall_result(result))
193 return wrapped_f
194
195 def _call_child_zones(self, zones, function):
196 """Ask the child zones to perform this operation.
197 Broken out for testing."""
198 return child_zone_helper(zones, function)
199
200 def get_collection_context_and_id(self, args, kwargs):
201 """Returns a tuple of (novaclient collection name, security
202 context and resource id. Derived class should override this."""
203 context = kwargs.get('context', None)
204 instance_id = kwargs.get('instance_id', None)
205 if len(args) > 0 and not context:
206 context = args[1]
207 if len(args) > 1 and not instance_id:
208 instance_id = args[2]
209 return ("servers", context, instance_id)
210
211 def unmarshall_result(self, zone_responses):
212 """Result is a list of responses from each child zone.
213 Each decorator derivation is responsible to turning this
214 into a format expected by the calling method. For
215 example, this one is expected to return a single Server
216 dict {'server':{k:v}}. Others may return a list of them, like
217 {'servers':[{k,v}]}"""
218 reduced_response = []
219 for zone_response in zone_responses:
220 if not zone_response:
221 continue
222
223 server = zone_response.__dict__
224
225 for k in server.keys():
226 if k[0] == '_' or k == 'manager':
227 del server[k]
228
229 reduced_response.append(dict(server=server))
230 if reduced_response:
231 return reduced_response[0] # first for now.
232 return {}
233
234
235def redirect_handler(f):
236 def new_f(*args, **kwargs):
237 try:
238 return f(*args, **kwargs)
239 except RedirectResult, e:
240 return e.results
241 return new_f
65242
=== modified file 'nova/scheduler/zone_manager.py'
--- nova/scheduler/zone_manager.py 2011-02-24 23:23:15 +0000
+++ nova/scheduler/zone_manager.py 2011-03-24 19:09:30 +0000
@@ -58,8 +58,9 @@
58 child zone."""58 child zone."""
59 self.last_seen = datetime.now()59 self.last_seen = datetime.now()
60 self.attempt = 060 self.attempt = 0
61 self.name = zone_metadata["name"]61 self.name = zone_metadata.get("name", "n/a")
62 self.capabilities = zone_metadata["capabilities"]62 self.capabilities = ", ".join(["%s=%s" % (k, v)
63 for k, v in zone_metadata.iteritems() if k != 'name'])
63 self.is_active = True64 self.is_active = True
6465
65 def to_dict(self):66 def to_dict(self):
@@ -104,7 +105,7 @@
104 """Keeps the zone states updated."""105 """Keeps the zone states updated."""
105 def __init__(self):106 def __init__(self):
106 self.last_zone_db_check = datetime.min107 self.last_zone_db_check = datetime.min
107 self.zone_states = {}108 self.zone_states = {} # { <zone_id> : ZoneState }
108 self.service_states = {} # { <service> : { <host> : { cap k : v }}}109 self.service_states = {} # { <service> : { <host> : { cap k : v }}}
109 self.green_pool = greenpool.GreenPool()110 self.green_pool = greenpool.GreenPool()
110111
111112
=== modified file 'nova/tests/test_scheduler.py'
--- nova/tests/test_scheduler.py 2011-03-10 06:23:13 +0000
+++ nova/tests/test_scheduler.py 2011-03-24 19:09:30 +0000
@@ -21,6 +21,9 @@
2121
22import datetime22import datetime
23import mox23import mox
24import novaclient.exceptions
25import stubout
26import webob
2427
25from mox import IgnoreArg28from mox import IgnoreArg
26from nova import context29from nova import context
@@ -32,6 +35,7 @@
32from nova import rpc35from nova import rpc
33from nova import utils36from nova import utils
34from nova.auth import manager as auth_manager37from nova.auth import manager as auth_manager
38from nova.scheduler import api
35from nova.scheduler import manager39from nova.scheduler import manager
36from nova.scheduler import driver40from nova.scheduler import driver
37from nova.compute import power_state41from nova.compute import power_state
@@ -937,3 +941,160 @@
937 db.instance_destroy(self.context, instance_id)941 db.instance_destroy(self.context, instance_id)
938 db.service_destroy(self.context, s_ref['id'])942 db.service_destroy(self.context, s_ref['id'])
939 db.service_destroy(self.context, s_ref2['id'])943 db.service_destroy(self.context, s_ref2['id'])
944
945
946class FakeZone(object):
947 def __init__(self, api_url, username, password):
948 self.api_url = api_url
949 self.username = username
950 self.password = password
951
952
953def zone_get_all(context):
954 return [
955 FakeZone('http://example.com', 'bob', 'xxx'),
956 ]
957
958
959class FakeRerouteCompute(api.reroute_compute):
960 def _call_child_zones(self, zones, function):
961 return []
962
963 def get_collection_context_and_id(self, args, kwargs):
964 return ("servers", None, 1)
965
966 def unmarshall_result(self, zone_responses):
967 return dict(magic="found me")
968
969
970def go_boom(self, context, instance):
971 raise exception.InstanceNotFound("boom message", instance)
972
973
974def found_instance(self, context, instance):
975 return dict(name='myserver')
976
977
978class FakeResource(object):
979 def __init__(self, attribute_dict):
980 for k, v in attribute_dict.iteritems():
981 setattr(self, k, v)
982
983 def pause(self):
984 pass
985
986
987class ZoneRedirectTest(test.TestCase):
988 def setUp(self):
989 super(ZoneRedirectTest, self).setUp()
990 self.stubs = stubout.StubOutForTesting()
991
992 self.stubs.Set(db, 'zone_get_all', zone_get_all)
993
994 self.enable_zone_routing = FLAGS.enable_zone_routing
995 FLAGS.enable_zone_routing = True
996
997 def tearDown(self):
998 self.stubs.UnsetAll()
999 FLAGS.enable_zone_routing = self.enable_zone_routing
1000 super(ZoneRedirectTest, self).tearDown()
1001
1002 def test_trap_found_locally(self):
1003 decorator = FakeRerouteCompute("foo")
1004 try:
1005 result = decorator(found_instance)(None, None, 1)
1006 except api.RedirectResult, e:
1007 self.fail(_("Successful database hit should succeed"))
1008
1009 def test_trap_not_found_locally(self):
1010 decorator = FakeRerouteCompute("foo")
1011 try:
1012 result = decorator(go_boom)(None, None, 1)
1013 self.assertFail(_("Should have rerouted."))
1014 except api.RedirectResult, e:
1015 self.assertEquals(e.results['magic'], 'found me')
1016
1017 def test_routing_flags(self):
1018 FLAGS.enable_zone_routing = False
1019 decorator = FakeRerouteCompute("foo")
1020 try:
1021 result = decorator(go_boom)(None, None, 1)
1022 self.assertFail(_("Should have thrown exception."))
1023 except exception.InstanceNotFound, e:
1024 self.assertEquals(e.message, 'boom message')
1025
1026 def test_get_collection_context_and_id(self):
1027 decorator = api.reroute_compute("foo")
1028 self.assertEquals(decorator.get_collection_context_and_id(
1029 (None, 10, 20), {}), ("servers", 10, 20))
1030 self.assertEquals(decorator.get_collection_context_and_id(
1031 (None, 11,), dict(instance_id=21)), ("servers", 11, 21))
1032 self.assertEquals(decorator.get_collection_context_and_id(
1033 (None,), dict(context=12, instance_id=22)), ("servers", 12, 22))
1034
1035 def test_unmarshal_single_server(self):
1036 decorator = api.reroute_compute("foo")
1037 self.assertEquals(decorator.unmarshall_result([]), {})
1038 self.assertEquals(decorator.unmarshall_result(
1039 [FakeResource(dict(a=1, b=2)), ]),
1040 dict(server=dict(a=1, b=2)))
1041 self.assertEquals(decorator.unmarshall_result(
1042 [FakeResource(dict(a=1, _b=2)), ]),
1043 dict(server=dict(a=1,)))
1044 self.assertEquals(decorator.unmarshall_result(
1045 [FakeResource(dict(a=1, manager=2)), ]),
1046 dict(server=dict(a=1,)))
1047 self.assertEquals(decorator.unmarshall_result(
1048 [FakeResource(dict(_a=1, manager=2)), ]),
1049 dict(server={}))
1050
1051
1052class FakeServerCollection(object):
1053 def get(self, instance_id):
1054 return FakeResource(dict(a=10, b=20))
1055
1056 def find(self, name):
1057 return FakeResource(dict(a=11, b=22))
1058
1059
1060class FakeEmptyServerCollection(object):
1061 def get(self, f):
1062 raise novaclient.NotFound(1)
1063
1064 def find(self, name):
1065 raise novaclient.NotFound(2)
1066
1067
1068class FakeNovaClient(object):
1069 def __init__(self, collection):
1070 self.servers = collection
1071
1072
1073class DynamicNovaClientTest(test.TestCase):
1074 def test_issue_novaclient_command_found(self):
1075 zone = FakeZone('http://example.com', 'bob', 'xxx')
1076 self.assertEquals(api._issue_novaclient_command(
1077 FakeNovaClient(FakeServerCollection()),
1078 zone, "servers", "get", 100).a, 10)
1079
1080 self.assertEquals(api._issue_novaclient_command(
1081 FakeNovaClient(FakeServerCollection()),
1082 zone, "servers", "find", "name").b, 22)
1083
1084 self.assertEquals(api._issue_novaclient_command(
1085 FakeNovaClient(FakeServerCollection()),
1086 zone, "servers", "pause", 100), None)
1087
1088 def test_issue_novaclient_command_not_found(self):
1089 zone = FakeZone('http://example.com', 'bob', 'xxx')
1090 self.assertEquals(api._issue_novaclient_command(
1091 FakeNovaClient(FakeEmptyServerCollection()),
1092 zone, "servers", "get", 100), None)
1093
1094 self.assertEquals(api._issue_novaclient_command(
1095 FakeNovaClient(FakeEmptyServerCollection()),
1096 zone, "servers", "find", "name"), None)
1097
1098 self.assertEquals(api._issue_novaclient_command(
1099 FakeNovaClient(FakeEmptyServerCollection()),
1100 zone, "servers", "any", "name"), None)