Merge lp:~sandy-walsh/nova/zones2 into lp:~hudson-openstack/nova/trunk
- zones2
- Merge into trunk
Status: | Merged |
---|---|
Approved by: | Jay Pipes |
Approved revision: | 673 |
Merged at revision: | 782 |
Proposed branch: | lp:~sandy-walsh/nova/zones2 |
Merge into: | lp:~hudson-openstack/nova/trunk |
Diff against target: |
655 lines (+443/-22) 9 files modified
nova/api/openstack/__init__.py (+3/-3) nova/api/openstack/zones.py (+20/-4) nova/flags.py (+4/-0) nova/scheduler/api.py (+49/-0) nova/scheduler/manager.py (+10/-0) nova/scheduler/zone_manager.py (+143/-0) nova/tests/api/openstack/test_zones.py (+41/-15) nova/tests/test_zones.py (+172/-0) tools/pip-requires (+1/-0) |
To merge this branch: | bzr merge lp:~sandy-walsh/nova/zones2 |
Related bugs: | |
Related blueprints: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Devin Carlen (community) | Approve | ||
Eric Day (community) | Approve | ||
Jay Pipes (community) | Approve | ||
Review via email: mp+50247@code.launchpad.net |
Commit message
Introduces the ZoneManager to the Scheduler which polls the child zones and caches their availability and capabilities.
Description of the change
This branch establishes a ZoneManager within the scheduler.
The ZoneManager takes the child zone structure created in the first phase of this bp and polls the child zones using the public API (and python-novatools, a new pip requirement).
From this polling, the parent zone determines the name & capabilities of the child zones. It also tracks if a zone is online or offline (after successive errors).
Additionally 'novatools zone-list' will now show the status of the child zones as gathered by the scheduler if available. It can take a minute or so for the first polling to occur (configurable).
Jay Pipes (jaypipes) wrote : | # |
Hi!
Good second round of coding! :)
72 + for item in items:
73 + item['api_url'] = item['api_
Are there a lot of URLs with \/ in them?
76 + if len(items) == 0:
Could just shorten the above to: if not items:
106 +DEFINE_
107 + 'comma-delimited list of tags which represent boolean'
108 + ' capabilities of this zone')
I was under the impression that we wanted to store these kind of things in the database, no? I think a flag of boolean-like capabilities is a bit limiting.
277 + def _poll_zones(self, context):
278 + """Try to connect to each child zone and get update."""
279 + green_pool = GreenPool()
280 + green_pool.
Would it be a bit more efficient to have the ZoneManager initialize a GreenPool in its constructure and have ZoneManager.
148 +# Copyright (c) 2010 Openstack, LLC.
s/2010/2011 :)
286 + logging.
Missed an i18n above
Other than those little nits, looks good. :)
I encourage you to add a bit of overview documentation to the doc/source/ RST files when you get a chance in a future commit.
-jay
Sandy Walsh (sandy-walsh) wrote : | # |
Good feedback guys ... I'm on it!
Sandy Walsh (sandy-walsh) wrote : | # |
> zone_capabilities flag - Should this be a list of k/v pairs rather than just a
> tag list? Something like "hypervisors=
> probably want to think about how this could be auto-generated from child
> zones/plugins. For example, when a compute worker configured for xen it
> automatically sets this in the scheduler/API so other zones can discover it.
Agreed. Phase 3 is going to be getting Compute, Volume, etc to send messages into the Scheduler service as well with their capabilities (so your API note is timely).
I meant to put the feature in this branch that aggregates the child zone capabilities collected into the /zone/info query ... forgot. I'll do that next in Phase 3.
Ed Leafe (ed-leafe) wrote : | # |
235 +def _call_novatools
236 + """Call novatools. Broken out for testing purposes."""
237 + os = novatools.
238 + return os.zones.
In this code, I would prefer a name that isn't likely to conflict with common Python module names. While 'os' isn't imported into the namespace for that code, it's a good practice not to use common names, especially when you can pick any name, as they can introduce hard-to-detect bugs.
272 + # Cleanup zones removed from db ...
273 + for zone_id in self.zone_
Simpler: for zone_id in self.zone_states:
(.keys() is redundant)
299 +def zone_get_
- and -
308 +def zone_get_
Is there a reason for three args named x,y,z?
393 +def exploding_
Love it!
454 + zone_state.
455 + username='user1', password='pass1'))
Fake URLs for testing should use 'example.com'. 'foo.com' is an actual domain name.
Sandy Walsh (sandy-walsh) wrote : | # |
> 73 + item['api_url'] = item['api_
The data coming back from the queue seems to be escaped/not unmarshalled properly. I need to investigate this further.
Jay Pipes (jaypipes) wrote : | # |
Hi Sandy! Please set the merge prop to Work In Progress while you work on any changes, then set back to Needs Review when you've pushed your changes up to LP. That way, reviewers are sent an email asking them to re-review the patches. Thanks!
Jay Pipes (jaypipes) wrote : | # |
excellent work, Sandy. looking forward to phase 3.
Eric Day (eday) wrote : | # |
36: still importing rpc, not needed
37,160,215,348: according to the hacking file, "thou shalt not import objects, only modules"
Otherwise, lgtm!
OpenStack Infra (hudson-openstack) wrote : | # |
The attempt to merge lp:~sandy-walsh/nova/zones2 into lp:nova failed. Below is the output from the failed tests.
AdminAPITest
test_
test_
APITest
test_
Test
test_
test_
test_bad_token ok
test_bad_user ok
test_no_user ok
test_
TestLimiter
test_
LimiterTest
test_
test_
test_
test_
test_
test_
test_
test_
test_
test_
test_
TestFaults
test_
test_raise ok
test_
FlavorsTest
test_
test_
GlanceImageServ
test_create ok
test_
test_delete ok
test_update ok
ImageController
test_
test_
LocalImageServi
test_create ok
test_
test_delete ok
test_update ok
LimiterTest
test_minute ok
test_
test_second ok
test_
test_
WSGIAppProxyTest
test_200 ok
test_403 ok
test_failure ...
Jay Pipes (jaypipes) wrote : | # |
we need to install novatools on Hudson...
Thierry Carrez (ttx) wrote : | # |
I'll quickly look into packaging it for Ubuntu, which should simplify this operation.
Todd Willey (xtoddx) wrote : | # |
Any updates on the state of hudson + novaclient? Also, why is it `class API:` and not `class API(object):`?
Sandy Walsh (sandy-walsh) wrote : | # |
Heh, thanks Todd. I missed that.
Latest news on novaclient: license issues are settled, just need to get a tarball to base the PPA on.
Thierry Carrez (ttx) wrote : | # |
python-novaclient is now packaged in PPA and available on Hudson, no more known blockers in getting this in.
Preview Diff
1 | === modified file 'nova/api/openstack/__init__.py' | |||
2 | --- nova/api/openstack/__init__.py 2011-03-03 01:50:48 +0000 | |||
3 | +++ nova/api/openstack/__init__.py 2011-03-09 12:55:18 +0000 | |||
4 | @@ -77,8 +77,8 @@ | |||
5 | 77 | 77 | ||
6 | 78 | server_members['pause'] = 'POST' | 78 | server_members['pause'] = 'POST' |
7 | 79 | server_members['unpause'] = 'POST' | 79 | server_members['unpause'] = 'POST' |
10 | 80 | server_members["diagnostics"] = "GET" | 80 | server_members['diagnostics'] = 'GET' |
11 | 81 | server_members["actions"] = "GET" | 81 | server_members['actions'] = 'GET' |
12 | 82 | server_members['suspend'] = 'POST' | 82 | server_members['suspend'] = 'POST' |
13 | 83 | server_members['resume'] = 'POST' | 83 | server_members['resume'] = 'POST' |
14 | 84 | server_members['rescue'] = 'POST' | 84 | server_members['rescue'] = 'POST' |
15 | @@ -87,7 +87,7 @@ | |||
16 | 87 | server_members['inject_network_info'] = 'POST' | 87 | server_members['inject_network_info'] = 'POST' |
17 | 88 | 88 | ||
18 | 89 | mapper.resource("zone", "zones", controller=zones.Controller(), | 89 | mapper.resource("zone", "zones", controller=zones.Controller(), |
20 | 90 | collection={'detail': 'GET'}) | 90 | collection={'detail': 'GET', 'info': 'GET'}), |
21 | 91 | 91 | ||
22 | 92 | mapper.resource("server", "servers", controller=servers.Controller(), | 92 | mapper.resource("server", "servers", controller=servers.Controller(), |
23 | 93 | collection={'detail': 'GET'}, | 93 | collection={'detail': 'GET'}, |
24 | 94 | 94 | ||
25 | === modified file 'nova/api/openstack/zones.py' | |||
26 | --- nova/api/openstack/zones.py 2011-02-21 07:16:10 +0000 | |||
27 | +++ nova/api/openstack/zones.py 2011-03-09 12:55:18 +0000 | |||
28 | @@ -1,4 +1,4 @@ | |||
30 | 1 | # Copyright 2010 OpenStack LLC. | 1 | # Copyright 2011 OpenStack LLC. |
31 | 2 | # All Rights Reserved. | 2 | # All Rights Reserved. |
32 | 3 | # | 3 | # |
33 | 4 | # Licensed under the Apache License, Version 2.0 (the "License"); you may | 4 | # Licensed under the Apache License, Version 2.0 (the "License"); you may |
34 | @@ -18,6 +18,7 @@ | |||
35 | 18 | from nova import flags | 18 | from nova import flags |
36 | 19 | from nova import wsgi | 19 | from nova import wsgi |
37 | 20 | from nova import db | 20 | from nova import db |
38 | 21 | from nova.scheduler import api | ||
39 | 21 | 22 | ||
40 | 22 | 23 | ||
41 | 23 | FLAGS = flags.FLAGS | 24 | FLAGS = flags.FLAGS |
42 | @@ -32,6 +33,10 @@ | |||
43 | 32 | return dict((k, v) for k, v in item.iteritems() if k in keys) | 33 | return dict((k, v) for k, v in item.iteritems() if k in keys) |
44 | 33 | 34 | ||
45 | 34 | 35 | ||
46 | 36 | def _exclude_keys(item, keys): | ||
47 | 37 | return dict((k, v) for k, v in item.iteritems() if k not in keys) | ||
48 | 38 | |||
49 | 39 | |||
50 | 35 | def _scrub_zone(zone): | 40 | def _scrub_zone(zone): |
51 | 36 | return _filter_keys(zone, ('id', 'api_url')) | 41 | return _filter_keys(zone, ('id', 'api_url')) |
52 | 37 | 42 | ||
53 | @@ -41,19 +46,30 @@ | |||
54 | 41 | _serialization_metadata = { | 46 | _serialization_metadata = { |
55 | 42 | 'application/xml': { | 47 | 'application/xml': { |
56 | 43 | "attributes": { | 48 | "attributes": { |
58 | 44 | "zone": ["id", "api_url"]}}} | 49 | "zone": ["id", "api_url", "name", "capabilities"]}}} |
59 | 45 | 50 | ||
60 | 46 | def index(self, req): | 51 | def index(self, req): |
61 | 47 | """Return all zones in brief""" | 52 | """Return all zones in brief""" |
63 | 48 | items = db.zone_get_all(req.environ['nova.context']) | 53 | # Ask the ZoneManager in the Scheduler for most recent data, |
64 | 54 | # or fall-back to the database ... | ||
65 | 55 | items = api.API().get_zone_list(req.environ['nova.context']) | ||
66 | 56 | if not items: | ||
67 | 57 | items = db.zone_get_all(req.environ['nova.context']) | ||
68 | 58 | |||
69 | 49 | items = common.limited(items, req) | 59 | items = common.limited(items, req) |
71 | 50 | items = [_scrub_zone(item) for item in items] | 60 | items = [_exclude_keys(item, ['username', 'password']) |
72 | 61 | for item in items] | ||
73 | 51 | return dict(zones=items) | 62 | return dict(zones=items) |
74 | 52 | 63 | ||
75 | 53 | def detail(self, req): | 64 | def detail(self, req): |
76 | 54 | """Return all zones in detail""" | 65 | """Return all zones in detail""" |
77 | 55 | return self.index(req) | 66 | return self.index(req) |
78 | 56 | 67 | ||
79 | 68 | def info(self, req): | ||
80 | 69 | """Return name and capabilities for this zone.""" | ||
81 | 70 | return dict(zone=dict(name=FLAGS.zone_name, | ||
82 | 71 | capabilities=FLAGS.zone_capabilities)) | ||
83 | 72 | |||
84 | 57 | def show(self, req, id): | 73 | def show(self, req, id): |
85 | 58 | """Return data about the given zone id""" | 74 | """Return data about the given zone id""" |
86 | 59 | zone_id = int(id) | 75 | zone_id = int(id) |
87 | 60 | 76 | ||
88 | === modified file 'nova/flags.py' | |||
89 | --- nova/flags.py 2011-02-25 01:04:25 +0000 | |||
90 | +++ nova/flags.py 2011-03-09 12:55:18 +0000 | |||
91 | @@ -354,3 +354,7 @@ | |||
92 | 354 | 354 | ||
93 | 355 | DEFINE_string('node_availability_zone', 'nova', | 355 | DEFINE_string('node_availability_zone', 'nova', |
94 | 356 | 'availability zone of this node') | 356 | 'availability zone of this node') |
95 | 357 | |||
96 | 358 | DEFINE_string('zone_name', 'nova', 'name of this zone') | ||
97 | 359 | DEFINE_string('zone_capabilities', 'kypervisor:xenserver;os:linux', | ||
98 | 360 | 'Key/Value tags which represent capabilities of this zone') | ||
99 | 357 | 361 | ||
100 | === added file 'nova/scheduler/api.py' | |||
101 | --- nova/scheduler/api.py 1970-01-01 00:00:00 +0000 | |||
102 | +++ nova/scheduler/api.py 2011-03-09 12:55:18 +0000 | |||
103 | @@ -0,0 +1,49 @@ | |||
104 | 1 | # Copyright (c) 2011 Openstack, LLC. | ||
105 | 2 | # All Rights Reserved. | ||
106 | 3 | # | ||
107 | 4 | # Licensed under the Apache License, Version 2.0 (the "License"); you may | ||
108 | 5 | # not use this file except in compliance with the License. You may obtain | ||
109 | 6 | # a copy of the License at | ||
110 | 7 | # | ||
111 | 8 | # http://www.apache.org/licenses/LICENSE-2.0 | ||
112 | 9 | # | ||
113 | 10 | # Unless required by applicable law or agreed to in writing, software | ||
114 | 11 | # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT | ||
115 | 12 | # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the | ||
116 | 13 | # License for the specific language governing permissions and limitations | ||
117 | 14 | # under the License. | ||
118 | 15 | |||
119 | 16 | """ | ||
120 | 17 | Handles all requests relating to schedulers. | ||
121 | 18 | """ | ||
122 | 19 | |||
123 | 20 | from nova import flags | ||
124 | 21 | from nova import log as logging | ||
125 | 22 | from nova import rpc | ||
126 | 23 | |||
127 | 24 | FLAGS = flags.FLAGS | ||
128 | 25 | LOG = logging.getLogger('nova.scheduler.api') | ||
129 | 26 | |||
130 | 27 | |||
131 | 28 | class API(object): | ||
132 | 29 | """API for interacting with the scheduler.""" | ||
133 | 30 | |||
134 | 31 | def _call_scheduler(self, method, context, params=None): | ||
135 | 32 | """Generic handler for RPC calls to the scheduler. | ||
136 | 33 | |||
137 | 34 | :param params: Optional dictionary of arguments to be passed to the | ||
138 | 35 | scheduler worker | ||
139 | 36 | |||
140 | 37 | :retval: Result returned by scheduler worker | ||
141 | 38 | """ | ||
142 | 39 | if not params: | ||
143 | 40 | params = {} | ||
144 | 41 | queue = FLAGS.scheduler_topic | ||
145 | 42 | kwargs = {'method': method, 'args': params} | ||
146 | 43 | return rpc.call(context, queue, kwargs) | ||
147 | 44 | |||
148 | 45 | def get_zone_list(self, context): | ||
149 | 46 | items = self._call_scheduler('get_zone_list', context) | ||
150 | 47 | for item in items: | ||
151 | 48 | item['api_url'] = item['api_url'].replace('\\/', '/') | ||
152 | 49 | return items | ||
153 | 0 | 50 | ||
154 | === modified file 'nova/scheduler/manager.py' | |||
155 | --- nova/scheduler/manager.py 2011-01-19 15:41:30 +0000 | |||
156 | +++ nova/scheduler/manager.py 2011-03-09 12:55:18 +0000 | |||
157 | @@ -29,6 +29,7 @@ | |||
158 | 29 | from nova import manager | 29 | from nova import manager |
159 | 30 | from nova import rpc | 30 | from nova import rpc |
160 | 31 | from nova import utils | 31 | from nova import utils |
161 | 32 | from nova.scheduler import zone_manager | ||
162 | 32 | 33 | ||
163 | 33 | LOG = logging.getLogger('nova.scheduler.manager') | 34 | LOG = logging.getLogger('nova.scheduler.manager') |
164 | 34 | FLAGS = flags.FLAGS | 35 | FLAGS = flags.FLAGS |
165 | @@ -43,12 +44,21 @@ | |||
166 | 43 | if not scheduler_driver: | 44 | if not scheduler_driver: |
167 | 44 | scheduler_driver = FLAGS.scheduler_driver | 45 | scheduler_driver = FLAGS.scheduler_driver |
168 | 45 | self.driver = utils.import_object(scheduler_driver) | 46 | self.driver = utils.import_object(scheduler_driver) |
169 | 47 | self.zone_manager = zone_manager.ZoneManager() | ||
170 | 46 | super(SchedulerManager, self).__init__(*args, **kwargs) | 48 | super(SchedulerManager, self).__init__(*args, **kwargs) |
171 | 47 | 49 | ||
172 | 48 | def __getattr__(self, key): | 50 | def __getattr__(self, key): |
173 | 49 | """Converts all method calls to use the schedule method""" | 51 | """Converts all method calls to use the schedule method""" |
174 | 50 | return functools.partial(self._schedule, key) | 52 | return functools.partial(self._schedule, key) |
175 | 51 | 53 | ||
176 | 54 | def periodic_tasks(self, context=None): | ||
177 | 55 | """Poll child zones periodically to get status.""" | ||
178 | 56 | self.zone_manager.ping(context) | ||
179 | 57 | |||
180 | 58 | def get_zone_list(self, context=None): | ||
181 | 59 | """Get a list of zones from the ZoneManager.""" | ||
182 | 60 | return self.zone_manager.get_zone_list() | ||
183 | 61 | |||
184 | 52 | def _schedule(self, method, context, topic, *args, **kwargs): | 62 | def _schedule(self, method, context, topic, *args, **kwargs): |
185 | 53 | """Tries to call schedule_* method on the driver to retrieve host. | 63 | """Tries to call schedule_* method on the driver to retrieve host. |
186 | 54 | 64 | ||
187 | 55 | 65 | ||
188 | === added file 'nova/scheduler/zone_manager.py' | |||
189 | --- nova/scheduler/zone_manager.py 1970-01-01 00:00:00 +0000 | |||
190 | +++ nova/scheduler/zone_manager.py 2011-03-09 12:55:18 +0000 | |||
191 | @@ -0,0 +1,143 @@ | |||
192 | 1 | # Copyright (c) 2011 Openstack, LLC. | ||
193 | 2 | # All Rights Reserved. | ||
194 | 3 | # | ||
195 | 4 | # Licensed under the Apache License, Version 2.0 (the "License"); you may | ||
196 | 5 | # not use this file except in compliance with the License. You may obtain | ||
197 | 6 | # a copy of the License at | ||
198 | 7 | # | ||
199 | 8 | # http://www.apache.org/licenses/LICENSE-2.0 | ||
200 | 9 | # | ||
201 | 10 | # Unless required by applicable law or agreed to in writing, software | ||
202 | 11 | # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT | ||
203 | 12 | # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the | ||
204 | 13 | # License for the specific language governing permissions and limitations | ||
205 | 14 | # under the License. | ||
206 | 15 | |||
207 | 16 | """ | ||
208 | 17 | ZoneManager oversees all communications with child Zones. | ||
209 | 18 | """ | ||
210 | 19 | |||
211 | 20 | import novaclient | ||
212 | 21 | import thread | ||
213 | 22 | import traceback | ||
214 | 23 | |||
215 | 24 | from datetime import datetime | ||
216 | 25 | from eventlet import greenpool | ||
217 | 26 | |||
218 | 27 | from nova import db | ||
219 | 28 | from nova import flags | ||
220 | 29 | from nova import log as logging | ||
221 | 30 | |||
222 | 31 | FLAGS = flags.FLAGS | ||
223 | 32 | flags.DEFINE_integer('zone_db_check_interval', 60, | ||
224 | 33 | 'Seconds between getting fresh zone info from db.') | ||
225 | 34 | flags.DEFINE_integer('zone_failures_to_offline', 3, | ||
226 | 35 | 'Number of consecutive errors before marking zone offline') | ||
227 | 36 | |||
228 | 37 | |||
229 | 38 | class ZoneState(object): | ||
230 | 39 | """Holds the state of all connected child zones.""" | ||
231 | 40 | def __init__(self): | ||
232 | 41 | self.is_active = True | ||
233 | 42 | self.name = None | ||
234 | 43 | self.capabilities = None | ||
235 | 44 | self.attempt = 0 | ||
236 | 45 | self.last_seen = datetime.min | ||
237 | 46 | self.last_exception = None | ||
238 | 47 | self.last_exception_time = None | ||
239 | 48 | |||
240 | 49 | def update_credentials(self, zone): | ||
241 | 50 | """Update zone credentials from db""" | ||
242 | 51 | self.zone_id = zone.id | ||
243 | 52 | self.api_url = zone.api_url | ||
244 | 53 | self.username = zone.username | ||
245 | 54 | self.password = zone.password | ||
246 | 55 | |||
247 | 56 | def update_metadata(self, zone_metadata): | ||
248 | 57 | """Update zone metadata after successful communications with | ||
249 | 58 | child zone.""" | ||
250 | 59 | self.last_seen = datetime.now() | ||
251 | 60 | self.attempt = 0 | ||
252 | 61 | self.name = zone_metadata["name"] | ||
253 | 62 | self.capabilities = zone_metadata["capabilities"] | ||
254 | 63 | self.is_active = True | ||
255 | 64 | |||
256 | 65 | def to_dict(self): | ||
257 | 66 | return dict(name=self.name, capabilities=self.capabilities, | ||
258 | 67 | is_active=self.is_active, api_url=self.api_url, | ||
259 | 68 | id=self.zone_id) | ||
260 | 69 | |||
261 | 70 | def log_error(self, exception): | ||
262 | 71 | """Something went wrong. Check to see if zone should be | ||
263 | 72 | marked as offline.""" | ||
264 | 73 | self.last_exception = exception | ||
265 | 74 | self.last_exception_time = datetime.now() | ||
266 | 75 | api_url = self.api_url | ||
267 | 76 | logging.warning(_("'%(exception)s' error talking to " | ||
268 | 77 | "zone %(api_url)s") % locals()) | ||
269 | 78 | |||
270 | 79 | max_errors = FLAGS.zone_failures_to_offline | ||
271 | 80 | self.attempt += 1 | ||
272 | 81 | if self.attempt >= max_errors: | ||
273 | 82 | self.is_active = False | ||
274 | 83 | logging.error(_("No answer from zone %(api_url)s " | ||
275 | 84 | "after %(max_errors)d " | ||
276 | 85 | "attempts. Marking inactive.") % locals()) | ||
277 | 86 | |||
278 | 87 | |||
279 | 88 | def _call_novaclient(zone): | ||
280 | 89 | """Call novaclient. Broken out for testing purposes.""" | ||
281 | 90 | client = novaclient.OpenStack(zone.username, zone.password, zone.api_url) | ||
282 | 91 | return client.zones.info()._info | ||
283 | 92 | |||
284 | 93 | |||
285 | 94 | def _poll_zone(zone): | ||
286 | 95 | """Eventlet worker to poll a zone.""" | ||
287 | 96 | logging.debug(_("Polling zone: %s") % zone.api_url) | ||
288 | 97 | try: | ||
289 | 98 | zone.update_metadata(_call_novaclient(zone)) | ||
290 | 99 | except Exception, e: | ||
291 | 100 | zone.log_error(traceback.format_exc()) | ||
292 | 101 | |||
293 | 102 | |||
294 | 103 | class ZoneManager(object): | ||
295 | 104 | """Keeps the zone states updated.""" | ||
296 | 105 | def __init__(self): | ||
297 | 106 | self.last_zone_db_check = datetime.min | ||
298 | 107 | self.zone_states = {} | ||
299 | 108 | self.green_pool = greenpool.GreenPool() | ||
300 | 109 | |||
301 | 110 | def get_zone_list(self): | ||
302 | 111 | """Return the list of zones we know about.""" | ||
303 | 112 | return [zone.to_dict() for zone in self.zone_states.values()] | ||
304 | 113 | |||
305 | 114 | def _refresh_from_db(self, context): | ||
306 | 115 | """Make our zone state map match the db.""" | ||
307 | 116 | # Add/update existing zones ... | ||
308 | 117 | zones = db.zone_get_all(context) | ||
309 | 118 | existing = self.zone_states.keys() | ||
310 | 119 | db_keys = [] | ||
311 | 120 | for zone in zones: | ||
312 | 121 | db_keys.append(zone.id) | ||
313 | 122 | if zone.id not in existing: | ||
314 | 123 | self.zone_states[zone.id] = ZoneState() | ||
315 | 124 | self.zone_states[zone.id].update_credentials(zone) | ||
316 | 125 | |||
317 | 126 | # Cleanup zones removed from db ... | ||
318 | 127 | keys = self.zone_states.keys() # since we're deleting | ||
319 | 128 | for zone_id in keys: | ||
320 | 129 | if zone_id not in db_keys: | ||
321 | 130 | del self.zone_states[zone_id] | ||
322 | 131 | |||
323 | 132 | def _poll_zones(self, context): | ||
324 | 133 | """Try to connect to each child zone and get update.""" | ||
325 | 134 | self.green_pool.imap(_poll_zone, self.zone_states.values()) | ||
326 | 135 | |||
327 | 136 | def ping(self, context=None): | ||
328 | 137 | """Ping should be called periodically to update zone status.""" | ||
329 | 138 | diff = datetime.now() - self.last_zone_db_check | ||
330 | 139 | if diff.seconds >= FLAGS.zone_db_check_interval: | ||
331 | 140 | logging.debug(_("Updating zone cache from db.")) | ||
332 | 141 | self.last_zone_db_check = datetime.now() | ||
333 | 142 | self._refresh_from_db(context) | ||
334 | 143 | self._poll_zones(context) | ||
335 | 0 | 144 | ||
336 | === modified file 'nova/tests/api/openstack/test_zones.py' | |||
337 | --- nova/tests/api/openstack/test_zones.py 2011-02-28 19:44:17 +0000 | |||
338 | +++ nova/tests/api/openstack/test_zones.py 2011-03-09 12:55:18 +0000 | |||
339 | @@ -1,4 +1,4 @@ | |||
341 | 1 | # Copyright 2010 OpenStack LLC. | 1 | # Copyright 2011 OpenStack LLC. |
342 | 2 | # All Rights Reserved. | 2 | # All Rights Reserved. |
343 | 3 | # | 3 | # |
344 | 4 | # Licensed under the Apache License, Version 2.0 (the "License"); you may | 4 | # Licensed under the Apache License, Version 2.0 (the "License"); you may |
345 | @@ -24,6 +24,7 @@ | |||
346 | 24 | from nova import test | 24 | from nova import test |
347 | 25 | from nova.api.openstack import zones | 25 | from nova.api.openstack import zones |
348 | 26 | from nova.tests.api.openstack import fakes | 26 | from nova.tests.api.openstack import fakes |
349 | 27 | from nova.scheduler import api | ||
350 | 27 | 28 | ||
351 | 28 | 29 | ||
352 | 29 | FLAGS = flags.FLAGS | 30 | FLAGS = flags.FLAGS |
353 | @@ -31,7 +32,7 @@ | |||
354 | 31 | 32 | ||
355 | 32 | 33 | ||
356 | 33 | def zone_get(context, zone_id): | 34 | def zone_get(context, zone_id): |
358 | 34 | return dict(id=1, api_url='http://foo.com', username='bob', | 35 | return dict(id=1, api_url='http://example.com', username='bob', |
359 | 35 | password='xxx') | 36 | password='xxx') |
360 | 36 | 37 | ||
361 | 37 | 38 | ||
362 | @@ -42,7 +43,7 @@ | |||
363 | 42 | 43 | ||
364 | 43 | 44 | ||
365 | 44 | def zone_update(context, zone_id, values): | 45 | def zone_update(context, zone_id, values): |
367 | 45 | zone = dict(id=zone_id, api_url='http://foo.com', username='bob', | 46 | zone = dict(id=zone_id, api_url='http://example.com', username='bob', |
368 | 46 | password='xxx') | 47 | password='xxx') |
369 | 47 | zone.update(values) | 48 | zone.update(values) |
370 | 48 | return zone | 49 | return zone |
371 | @@ -52,12 +53,26 @@ | |||
372 | 52 | pass | 53 | pass |
373 | 53 | 54 | ||
374 | 54 | 55 | ||
381 | 55 | def zone_get_all(context): | 56 | def zone_get_all_scheduler(*args): |
382 | 56 | return [ | 57 | return [ |
383 | 57 | dict(id=1, api_url='http://foo.com', username='bob', | 58 | dict(id=1, api_url='http://example.com', username='bob', |
384 | 58 | password='xxx'), | 59 | password='xxx'), |
385 | 59 | dict(id=2, api_url='http://blah.com', username='alice', | 60 | dict(id=2, api_url='http://example.org', username='alice', |
386 | 60 | password='qwerty')] | 61 | password='qwerty') |
387 | 62 | ] | ||
388 | 63 | |||
389 | 64 | |||
390 | 65 | def zone_get_all_scheduler_empty(*args): | ||
391 | 66 | return [] | ||
392 | 67 | |||
393 | 68 | |||
394 | 69 | def zone_get_all_db(context): | ||
395 | 70 | return [ | ||
396 | 71 | dict(id=1, api_url='http://example.com', username='bob', | ||
397 | 72 | password='xxx'), | ||
398 | 73 | dict(id=2, api_url='http://example.org', username='alice', | ||
399 | 74 | password='qwerty') | ||
400 | 75 | ] | ||
401 | 61 | 76 | ||
402 | 62 | 77 | ||
403 | 63 | class ZonesTest(test.TestCase): | 78 | class ZonesTest(test.TestCase): |
404 | @@ -74,7 +89,6 @@ | |||
405 | 74 | FLAGS.allow_admin_api = True | 89 | FLAGS.allow_admin_api = True |
406 | 75 | 90 | ||
407 | 76 | self.stubs.Set(nova.db, 'zone_get', zone_get) | 91 | self.stubs.Set(nova.db, 'zone_get', zone_get) |
408 | 77 | self.stubs.Set(nova.db, 'zone_get_all', zone_get_all) | ||
409 | 78 | self.stubs.Set(nova.db, 'zone_update', zone_update) | 92 | self.stubs.Set(nova.db, 'zone_update', zone_update) |
410 | 79 | self.stubs.Set(nova.db, 'zone_create', zone_create) | 93 | self.stubs.Set(nova.db, 'zone_create', zone_create) |
411 | 80 | self.stubs.Set(nova.db, 'zone_delete', zone_delete) | 94 | self.stubs.Set(nova.db, 'zone_delete', zone_delete) |
412 | @@ -84,7 +98,19 @@ | |||
413 | 84 | FLAGS.allow_admin_api = self.allow_admin | 98 | FLAGS.allow_admin_api = self.allow_admin |
414 | 85 | super(ZonesTest, self).tearDown() | 99 | super(ZonesTest, self).tearDown() |
415 | 86 | 100 | ||
417 | 87 | def test_get_zone_list(self): | 101 | def test_get_zone_list_scheduler(self): |
418 | 102 | self.stubs.Set(api.API, '_call_scheduler', zone_get_all_scheduler) | ||
419 | 103 | req = webob.Request.blank('/v1.0/zones') | ||
420 | 104 | res = req.get_response(fakes.wsgi_app()) | ||
421 | 105 | res_dict = json.loads(res.body) | ||
422 | 106 | |||
423 | 107 | self.assertEqual(res.status_int, 200) | ||
424 | 108 | self.assertEqual(len(res_dict['zones']), 2) | ||
425 | 109 | |||
426 | 110 | def test_get_zone_list_db(self): | ||
427 | 111 | self.stubs.Set(api.API, '_call_scheduler', | ||
428 | 112 | zone_get_all_scheduler_empty) | ||
429 | 113 | self.stubs.Set(nova.db, 'zone_get_all', zone_get_all_db) | ||
430 | 88 | req = webob.Request.blank('/v1.0/zones') | 114 | req = webob.Request.blank('/v1.0/zones') |
431 | 89 | res = req.get_response(fakes.wsgi_app()) | 115 | res = req.get_response(fakes.wsgi_app()) |
432 | 90 | res_dict = json.loads(res.body) | 116 | res_dict = json.loads(res.body) |
433 | @@ -98,7 +124,7 @@ | |||
434 | 98 | res_dict = json.loads(res.body) | 124 | res_dict = json.loads(res.body) |
435 | 99 | 125 | ||
436 | 100 | self.assertEqual(res_dict['zone']['id'], 1) | 126 | self.assertEqual(res_dict['zone']['id'], 1) |
438 | 101 | self.assertEqual(res_dict['zone']['api_url'], 'http://foo.com') | 127 | self.assertEqual(res_dict['zone']['api_url'], 'http://example.com') |
439 | 102 | self.assertFalse('password' in res_dict['zone']) | 128 | self.assertFalse('password' in res_dict['zone']) |
440 | 103 | self.assertEqual(res.status_int, 200) | 129 | self.assertEqual(res.status_int, 200) |
441 | 104 | 130 | ||
442 | @@ -109,7 +135,7 @@ | |||
443 | 109 | self.assertEqual(res.status_int, 200) | 135 | self.assertEqual(res.status_int, 200) |
444 | 110 | 136 | ||
445 | 111 | def test_zone_create(self): | 137 | def test_zone_create(self): |
447 | 112 | body = dict(zone=dict(api_url='http://blah.zoo', username='fred', | 138 | body = dict(zone=dict(api_url='http://example.com', username='fred', |
448 | 113 | password='fubar')) | 139 | password='fubar')) |
449 | 114 | req = webob.Request.blank('/v1.0/zones') | 140 | req = webob.Request.blank('/v1.0/zones') |
450 | 115 | req.method = 'POST' | 141 | req.method = 'POST' |
451 | @@ -120,7 +146,7 @@ | |||
452 | 120 | 146 | ||
453 | 121 | self.assertEqual(res.status_int, 200) | 147 | self.assertEqual(res.status_int, 200) |
454 | 122 | self.assertEqual(res_dict['zone']['id'], 1) | 148 | self.assertEqual(res_dict['zone']['id'], 1) |
456 | 123 | self.assertEqual(res_dict['zone']['api_url'], 'http://blah.zoo') | 149 | self.assertEqual(res_dict['zone']['api_url'], 'http://example.com') |
457 | 124 | self.assertFalse('username' in res_dict['zone']) | 150 | self.assertFalse('username' in res_dict['zone']) |
458 | 125 | 151 | ||
459 | 126 | def test_zone_update(self): | 152 | def test_zone_update(self): |
460 | @@ -134,5 +160,5 @@ | |||
461 | 134 | 160 | ||
462 | 135 | self.assertEqual(res.status_int, 200) | 161 | self.assertEqual(res.status_int, 200) |
463 | 136 | self.assertEqual(res_dict['zone']['id'], 1) | 162 | self.assertEqual(res_dict['zone']['id'], 1) |
465 | 137 | self.assertEqual(res_dict['zone']['api_url'], 'http://foo.com') | 163 | self.assertEqual(res_dict['zone']['api_url'], 'http://example.com') |
466 | 138 | self.assertFalse('username' in res_dict['zone']) | 164 | self.assertFalse('username' in res_dict['zone']) |
467 | 139 | 165 | ||
468 | === added file 'nova/tests/test_zones.py' | |||
469 | --- nova/tests/test_zones.py 1970-01-01 00:00:00 +0000 | |||
470 | +++ nova/tests/test_zones.py 2011-03-09 12:55:18 +0000 | |||
471 | @@ -0,0 +1,172 @@ | |||
472 | 1 | # Copyright 2010 United States Government as represented by the | ||
473 | 2 | # All Rights Reserved. | ||
474 | 3 | # | ||
475 | 4 | # Licensed under the Apache License, Version 2.0 (the "License"); you may | ||
476 | 5 | # not use this file except in compliance with the License. You may obtain | ||
477 | 6 | # a copy of the License at | ||
478 | 7 | # | ||
479 | 8 | # http://www.apache.org/licenses/LICENSE-2.0 | ||
480 | 9 | # | ||
481 | 10 | # Unless required by applicable law or agreed to in writing, software | ||
482 | 11 | # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT | ||
483 | 12 | # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the | ||
484 | 13 | # License for the specific language governing permissions and limitations | ||
485 | 14 | # under the License. | ||
486 | 15 | """ | ||
487 | 16 | Tests For ZoneManager | ||
488 | 17 | """ | ||
489 | 18 | |||
490 | 19 | import datetime | ||
491 | 20 | import mox | ||
492 | 21 | import novaclient | ||
493 | 22 | |||
494 | 23 | from nova import context | ||
495 | 24 | from nova import db | ||
496 | 25 | from nova import flags | ||
497 | 26 | from nova import service | ||
498 | 27 | from nova import test | ||
499 | 28 | from nova import rpc | ||
500 | 29 | from nova import utils | ||
501 | 30 | from nova.auth import manager as auth_manager | ||
502 | 31 | from nova.scheduler import zone_manager | ||
503 | 32 | |||
504 | 33 | FLAGS = flags.FLAGS | ||
505 | 34 | |||
506 | 35 | |||
507 | 36 | class FakeZone: | ||
508 | 37 | """Represents a fake zone from the db""" | ||
509 | 38 | def __init__(self, *args, **kwargs): | ||
510 | 39 | for k, v in kwargs.iteritems(): | ||
511 | 40 | setattr(self, k, v) | ||
512 | 41 | |||
513 | 42 | |||
514 | 43 | def exploding_novaclient(zone): | ||
515 | 44 | """Used when we want to simulate a novaclient call failing.""" | ||
516 | 45 | raise Exception("kaboom") | ||
517 | 46 | |||
518 | 47 | |||
519 | 48 | class ZoneManagerTestCase(test.TestCase): | ||
520 | 49 | """Test case for zone manager""" | ||
521 | 50 | def test_ping(self): | ||
522 | 51 | zm = zone_manager.ZoneManager() | ||
523 | 52 | self.mox.StubOutWithMock(zm, '_refresh_from_db') | ||
524 | 53 | self.mox.StubOutWithMock(zm, '_poll_zones') | ||
525 | 54 | zm._refresh_from_db(mox.IgnoreArg()) | ||
526 | 55 | zm._poll_zones(mox.IgnoreArg()) | ||
527 | 56 | |||
528 | 57 | self.mox.ReplayAll() | ||
529 | 58 | zm.ping(None) | ||
530 | 59 | self.mox.VerifyAll() | ||
531 | 60 | |||
532 | 61 | def test_refresh_from_db_new(self): | ||
533 | 62 | zm = zone_manager.ZoneManager() | ||
534 | 63 | |||
535 | 64 | self.mox.StubOutWithMock(db, 'zone_get_all') | ||
536 | 65 | db.zone_get_all(mox.IgnoreArg()).AndReturn([ | ||
537 | 66 | FakeZone(id=1, api_url='http://foo.com', username='user1', | ||
538 | 67 | password='pass1'), | ||
539 | 68 | ]) | ||
540 | 69 | |||
541 | 70 | self.assertEquals(len(zm.zone_states), 0) | ||
542 | 71 | |||
543 | 72 | self.mox.ReplayAll() | ||
544 | 73 | zm._refresh_from_db(None) | ||
545 | 74 | self.mox.VerifyAll() | ||
546 | 75 | |||
547 | 76 | self.assertEquals(len(zm.zone_states), 1) | ||
548 | 77 | self.assertEquals(zm.zone_states[1].username, 'user1') | ||
549 | 78 | |||
550 | 79 | def test_refresh_from_db_replace_existing(self): | ||
551 | 80 | zm = zone_manager.ZoneManager() | ||
552 | 81 | zone_state = zone_manager.ZoneState() | ||
553 | 82 | zone_state.update_credentials(FakeZone(id=1, api_url='http://foo.com', | ||
554 | 83 | username='user1', password='pass1')) | ||
555 | 84 | zm.zone_states[1] = zone_state | ||
556 | 85 | |||
557 | 86 | self.mox.StubOutWithMock(db, 'zone_get_all') | ||
558 | 87 | db.zone_get_all(mox.IgnoreArg()).AndReturn([ | ||
559 | 88 | FakeZone(id=1, api_url='http://foo.com', username='user2', | ||
560 | 89 | password='pass2'), | ||
561 | 90 | ]) | ||
562 | 91 | |||
563 | 92 | self.assertEquals(len(zm.zone_states), 1) | ||
564 | 93 | |||
565 | 94 | self.mox.ReplayAll() | ||
566 | 95 | zm._refresh_from_db(None) | ||
567 | 96 | self.mox.VerifyAll() | ||
568 | 97 | |||
569 | 98 | self.assertEquals(len(zm.zone_states), 1) | ||
570 | 99 | self.assertEquals(zm.zone_states[1].username, 'user2') | ||
571 | 100 | |||
572 | 101 | def test_refresh_from_db_missing(self): | ||
573 | 102 | zm = zone_manager.ZoneManager() | ||
574 | 103 | zone_state = zone_manager.ZoneState() | ||
575 | 104 | zone_state.update_credentials(FakeZone(id=1, api_url='http://foo.com', | ||
576 | 105 | username='user1', password='pass1')) | ||
577 | 106 | zm.zone_states[1] = zone_state | ||
578 | 107 | |||
579 | 108 | self.mox.StubOutWithMock(db, 'zone_get_all') | ||
580 | 109 | db.zone_get_all(mox.IgnoreArg()).AndReturn([]) | ||
581 | 110 | |||
582 | 111 | self.assertEquals(len(zm.zone_states), 1) | ||
583 | 112 | |||
584 | 113 | self.mox.ReplayAll() | ||
585 | 114 | zm._refresh_from_db(None) | ||
586 | 115 | self.mox.VerifyAll() | ||
587 | 116 | |||
588 | 117 | self.assertEquals(len(zm.zone_states), 0) | ||
589 | 118 | |||
590 | 119 | def test_refresh_from_db_add_and_delete(self): | ||
591 | 120 | zm = zone_manager.ZoneManager() | ||
592 | 121 | zone_state = zone_manager.ZoneState() | ||
593 | 122 | zone_state.update_credentials(FakeZone(id=1, api_url='http://foo.com', | ||
594 | 123 | username='user1', password='pass1')) | ||
595 | 124 | zm.zone_states[1] = zone_state | ||
596 | 125 | |||
597 | 126 | self.mox.StubOutWithMock(db, 'zone_get_all') | ||
598 | 127 | |||
599 | 128 | db.zone_get_all(mox.IgnoreArg()).AndReturn([ | ||
600 | 129 | FakeZone(id=2, api_url='http://foo.com', username='user2', | ||
601 | 130 | password='pass2'), | ||
602 | 131 | ]) | ||
603 | 132 | self.assertEquals(len(zm.zone_states), 1) | ||
604 | 133 | |||
605 | 134 | self.mox.ReplayAll() | ||
606 | 135 | zm._refresh_from_db(None) | ||
607 | 136 | self.mox.VerifyAll() | ||
608 | 137 | |||
609 | 138 | self.assertEquals(len(zm.zone_states), 1) | ||
610 | 139 | self.assertEquals(zm.zone_states[2].username, 'user2') | ||
611 | 140 | |||
612 | 141 | def test_poll_zone(self): | ||
613 | 142 | self.mox.StubOutWithMock(zone_manager, '_call_novaclient') | ||
614 | 143 | zone_manager._call_novaclient(mox.IgnoreArg()).AndReturn( | ||
615 | 144 | dict(name='zohan', capabilities='hairdresser')) | ||
616 | 145 | |||
617 | 146 | zone_state = zone_manager.ZoneState() | ||
618 | 147 | zone_state.update_credentials(FakeZone(id=2, | ||
619 | 148 | api_url='http://foo.com', username='user2', | ||
620 | 149 | password='pass2')) | ||
621 | 150 | zone_state.attempt = 1 | ||
622 | 151 | |||
623 | 152 | self.mox.ReplayAll() | ||
624 | 153 | zone_manager._poll_zone(zone_state) | ||
625 | 154 | self.mox.VerifyAll() | ||
626 | 155 | self.assertEquals(zone_state.attempt, 0) | ||
627 | 156 | self.assertEquals(zone_state.name, 'zohan') | ||
628 | 157 | |||
629 | 158 | def test_poll_zone_fails(self): | ||
630 | 159 | self.stubs.Set(zone_manager, "_call_novaclient", exploding_novaclient) | ||
631 | 160 | |||
632 | 161 | zone_state = zone_manager.ZoneState() | ||
633 | 162 | zone_state.update_credentials(FakeZone(id=2, | ||
634 | 163 | api_url='http://foo.com', username='user2', | ||
635 | 164 | password='pass2')) | ||
636 | 165 | zone_state.attempt = FLAGS.zone_failures_to_offline - 1 | ||
637 | 166 | |||
638 | 167 | self.mox.ReplayAll() | ||
639 | 168 | zone_manager._poll_zone(zone_state) | ||
640 | 169 | self.mox.VerifyAll() | ||
641 | 170 | self.assertEquals(zone_state.attempt, 3) | ||
642 | 171 | self.assertFalse(zone_state.is_active) | ||
643 | 172 | self.assertEquals(zone_state.name, None) | ||
644 | 0 | 173 | ||
645 | === modified file 'tools/pip-requires' | |||
646 | --- tools/pip-requires 2011-01-23 20:52:09 +0000 | |||
647 | +++ tools/pip-requires 2011-03-09 12:55:18 +0000 | |||
648 | @@ -10,6 +10,7 @@ | |||
649 | 10 | carrot==0.10.5 | 10 | carrot==0.10.5 |
650 | 11 | eventlet==0.9.12 | 11 | eventlet==0.9.12 |
651 | 12 | lockfile==0.8 | 12 | lockfile==0.8 |
652 | 13 | python-novaclient==2.3 | ||
653 | 13 | python-daemon==1.5.5 | 14 | python-daemon==1.5.5 |
654 | 14 | python-gflags==1.3 | 15 | python-gflags==1.3 |
655 | 15 | redis==2.0.0 | 16 | redis==2.0.0 |
Scheduler API - I would create a scheduler API much like we did for compute/ volume/ network, and use that API interface instead of making RPC calls directly from nova/api/ openstack/ zones.py. We want to keep nova.api modules thin wrappers over the core modules so we can reuse those core calls easily (say if some other nova.api module or another core component wants to grab the same info).
zone_capabilities flag - Should this be a list of k/v pairs rather than just a tag list? Something like "hypervisors= xen,kvm; volume= iscsi". Eventually we probably want to think about how this could be auto-generated from child zones/plugins. For example, when a compute worker configured for xen it automatically sets this in the scheduler/API so other zones can discover it. This isn't a blocker for merge since we can figure this out later.