Merge lp:~mwhudson/lava-scheduler/create-private-job into lp:lava-scheduler
- create-private-job
- Merge into trunk
Status: | Superseded |
---|---|
Proposed branch: | lp:~mwhudson/lava-scheduler/create-private-job |
Merge into: | lp:lava-scheduler |
Diff against target: |
433 lines (+148/-63) 4 files modified
lava_scheduler_app/models.py (+30/-11) lava_scheduler_app/tests.py (+112/-34) lava_scheduler_daemon/dbjobsource.py (+4/-18) setup.py (+2/-0) |
To merge this branch: | bzr merge lp:~mwhudson/lava-scheduler/create-private-job |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Zygmunt Krynicki (community) | Approve | ||
Review via email: mp+96703@code.launchpad.net |
Commit message
Description of the change
This branch enables the creation of private testjobs. It does so by looking for the bundle stream named by the submit_results action and copying the access data over to the new job. Because this means poking around in the JSON way more than before, I also finally implemented validation of the json at submit_job time -- this branch depends on https:/
So, what do you think? I think I more or less like this approach -- it avoids repetition of access information and the potential for inconsistency there. I do find grubbing through the actions for the submit_results action to be a bit strange still, both for the UX and the implementation -- I think I would be happier if the bundle_stream name and (ideally, optional, at least for jobs that run through the scheduler) xml-rpc endpoint were top level properties of the job description. But that can wait for another day, perhaps.
Cheers,
mwh
Zygmunt Krynicki (zyga) wrote : | # |
Zygmunt Krynicki (zyga) wrote : | # |
Otherwise this looks good :-) +1
Unmerged revisions
Preview Diff
1 | === modified file 'lava_scheduler_app/models.py' |
2 | --- lava_scheduler_app/models.py 2012-03-07 01:56:57 +0000 |
3 | +++ lava_scheduler_app/models.py 2012-03-09 04:16:33 +0000 |
4 | @@ -3,16 +3,17 @@ |
5 | from django.contrib.auth.models import User |
6 | from django.core.exceptions import ValidationError |
7 | from django.db import models |
8 | -from django.db.models.query import QuerySet |
9 | from django.utils.translation import ugettext as _ |
10 | |
11 | - |
12 | -from django_restricted_resource.managers import RestrictedResourceManager |
13 | from django_restricted_resource.models import RestrictedResource |
14 | -from django_restricted_resource.utils import filter_bogus_users |
15 | + |
16 | +from dashboard_app.models import BundleStream |
17 | + |
18 | +from lava_dispatcher.job import validate_job_data |
19 | |
20 | from linaro_django_xmlrpc.models import AuthToken |
21 | |
22 | + |
23 | class JSONDataError(ValueError): |
24 | """Error raised when JSON is syntactically valid but ill-formed.""" |
25 | |
26 | @@ -30,12 +31,9 @@ |
27 | def validate_job_json(data): |
28 | try: |
29 | ob = simplejson.loads(data) |
30 | + validate_job_data(ob) |
31 | except ValueError, e: |
32 | raise ValidationError(str(e)) |
33 | - else: |
34 | - if not isinstance(ob, dict): |
35 | - raise ValidationError( |
36 | - "job json must be an object, not %s" % type(ob).__name__) |
37 | |
38 | |
39 | class DeviceType(models.Model): |
40 | @@ -266,6 +264,7 @@ |
41 | @classmethod |
42 | def from_json_and_user(cls, json_data, user): |
43 | job_data = simplejson.loads(json_data) |
44 | + validate_job_data(job_data) |
45 | if 'target' in job_data: |
46 | target = Device.objects.get(hostname=job_data['target']) |
47 | device_type = None |
48 | @@ -279,6 +278,25 @@ |
49 | |
50 | is_check = job_data.get('health_check', False) |
51 | |
52 | + submitter = user |
53 | + group = None |
54 | + is_public = True |
55 | + |
56 | + for action in job_data['actions']: |
57 | + if not action['command'].startswith('submit_results'): |
58 | + continue |
59 | + stream = action['parameters']['stream'] |
60 | + try: |
61 | + bundle_stream = BundleStream.objects.get(pathname=stream) |
62 | + except BundleStream.DoesNotExist: |
63 | + raise ValueError("stream %s not found" % stream) |
64 | + if not bundle_stream.is_owned_by(submitter): |
65 | + raise ValueError( |
66 | + "you cannot submit to the stream %s" % stream) |
67 | + user, group, is_public = (bundle_stream.user, |
68 | + bundle_stream.group, |
69 | + bundle_stream.is_public) |
70 | + |
71 | tags = [] |
72 | for tag_name in job_data.get('device_tags', []): |
73 | try: |
74 | @@ -286,9 +304,10 @@ |
75 | except Tag.DoesNotExist: |
76 | raise JSONDataError("tag %r does not exist" % tag_name) |
77 | job = TestJob( |
78 | - definition=json_data, submitter=user, requested_device=target, |
79 | - requested_device_type=device_type, description=job_name, |
80 | - health_check=is_check, user=user) |
81 | + definition=json_data, submitter=submitter, |
82 | + requested_device=target, requested_device_type=device_type, |
83 | + description=job_name, health_check=is_check, user=user, |
84 | + group=group, is_public=is_public) |
85 | job.save() |
86 | for tag in tags: |
87 | job.tags.add(tag) |
88 | |
89 | === modified file 'lava_scheduler_app/tests.py' |
90 | --- lava_scheduler_app/tests.py 2012-02-28 03:53:16 +0000 |
91 | +++ lava_scheduler_app/tests.py 2012-03-09 04:16:33 +0000 |
92 | @@ -3,7 +3,9 @@ |
93 | import json |
94 | import xmlrpclib |
95 | |
96 | -from django.contrib.auth.models import Permission, User |
97 | +from dashboard_app.models import BundleStream |
98 | + |
99 | +from django.contrib.auth.models import Group, Permission, User |
100 | from django.test import TransactionTestCase |
101 | from django.test.client import Client |
102 | |
103 | @@ -84,9 +86,22 @@ |
104 | device.save() |
105 | return device |
106 | |
107 | + def make_job_data(self, actions=[], **kw): |
108 | + data = {'actions': actions, 'timeout': 1} |
109 | + data.update(kw) |
110 | + if 'target' not in data and 'device_type' not in data: |
111 | + if DeviceType.objects.all(): |
112 | + data['device_type'] = DeviceType.objects.all()[0].name |
113 | + else: |
114 | + data['device_type'] = self.ensure_device_type().name |
115 | + return data |
116 | + |
117 | + def make_job_json(self, **kw): |
118 | + return json.dumps(self.make_job_data(**kw)) |
119 | + |
120 | def make_testjob(self, definition=None, submitter=None, **kwargs): |
121 | if definition is None: |
122 | - definition = json.dumps({}) |
123 | + definition = self.make_job_json() |
124 | if submitter is None: |
125 | submitter = self.make_user() |
126 | if 'user' not in kwargs: |
127 | @@ -107,70 +122,68 @@ |
128 | class TestTestJob(TestCaseWithFactory): |
129 | |
130 | def test_from_json_and_user_sets_definition(self): |
131 | - self.factory.ensure_device_type(name='panda') |
132 | - definition = json.dumps({'device_type':'panda'}) |
133 | + definition = self.factory.make_job_json() |
134 | job = TestJob.from_json_and_user(definition, self.factory.make_user()) |
135 | self.assertEqual(definition, job.definition) |
136 | |
137 | def test_from_json_and_user_sets_submitter(self): |
138 | - self.factory.ensure_device_type(name='panda') |
139 | user = self.factory.make_user() |
140 | job = TestJob.from_json_and_user( |
141 | - json.dumps({'device_type':'panda'}), user) |
142 | + self.factory.make_job_json(), user) |
143 | self.assertEqual(user, job.submitter) |
144 | |
145 | def test_from_json_and_user_sets_device_type(self): |
146 | panda_type = self.factory.ensure_device_type(name='panda') |
147 | job = TestJob.from_json_and_user( |
148 | - json.dumps({'device_type':'panda'}), self.factory.make_user()) |
149 | + self.factory.make_job_json(device_type='panda'), |
150 | + self.factory.make_user()) |
151 | self.assertEqual(panda_type, job.requested_device_type) |
152 | |
153 | def test_from_json_and_user_sets_target(self): |
154 | panda_board = self.factory.make_device(hostname='panda01') |
155 | job = TestJob.from_json_and_user( |
156 | - json.dumps({'target':'panda01'}), self.factory.make_user()) |
157 | + self.factory.make_job_json(target='panda01'), |
158 | + self.factory.make_user()) |
159 | self.assertEqual(panda_board, job.requested_device) |
160 | |
161 | def test_from_json_and_user_does_not_set_device_type_from_target(self): |
162 | panda_type = self.factory.ensure_device_type(name='panda') |
163 | self.factory.make_device(device_type=panda_type, hostname='panda01') |
164 | job = TestJob.from_json_and_user( |
165 | - json.dumps({'target':'panda01'}), self.factory.make_user()) |
166 | + self.factory.make_job_json(target='panda01'), |
167 | + self.factory.make_user()) |
168 | self.assertEqual(None, job.requested_device_type) |
169 | |
170 | def test_from_json_and_user_sets_date_submitted(self): |
171 | - self.factory.ensure_device_type(name='panda') |
172 | before = datetime.datetime.now() |
173 | job = TestJob.from_json_and_user( |
174 | - json.dumps({'device_type':'panda'}), self.factory.make_user()) |
175 | + self.factory.make_job_json(), |
176 | + self.factory.make_user()) |
177 | after = datetime.datetime.now() |
178 | self.assertTrue(before < job.submit_time < after) |
179 | |
180 | def test_from_json_and_user_sets_status_to_SUBMITTED(self): |
181 | - self.factory.ensure_device_type(name='panda') |
182 | job = TestJob.from_json_and_user( |
183 | - json.dumps({'device_type':'panda'}), self.factory.make_user()) |
184 | + self.factory.make_job_json(), |
185 | + self.factory.make_user()) |
186 | self.assertEqual(job.status, TestJob.SUBMITTED) |
187 | |
188 | def test_from_json_and_user_sets_no_tags_if_no_tags(self): |
189 | - self.factory.ensure_device_type(name='panda') |
190 | job = TestJob.from_json_and_user( |
191 | - json.dumps({'device_type':'panda', 'device_tags':[]}), |
192 | + self.factory.make_job_json(device_tags=[]), |
193 | self.factory.make_user()) |
194 | self.assertEqual(set(job.tags.all()), set([])) |
195 | |
196 | def test_from_json_and_user_errors_on_unknown_tags(self): |
197 | - self.factory.ensure_device_type(name='panda') |
198 | self.assertRaises( |
199 | JSONDataError, TestJob.from_json_and_user, |
200 | - json.dumps({'device_type':'panda', 'device_tags':['unknown']}), |
201 | + self.factory.make_job_json(device_tags=['unknown']), |
202 | self.factory.make_user()) |
203 | |
204 | def test_from_json_and_user_sets_tag_from_device_tags(self): |
205 | - self.factory.ensure_device_type(name='panda') |
206 | self.factory.ensure_tag('tag') |
207 | job = TestJob.from_json_and_user( |
208 | - json.dumps({'device_type':'panda', 'device_tags':['tag']}), |
209 | + self.factory.make_job_json(device_tags=['tag']), |
210 | self.factory.make_user()) |
211 | self.assertEqual( |
212 | set(tag.name for tag in job.tags.all()), set(['tag'])) |
213 | @@ -180,7 +193,7 @@ |
214 | self.factory.ensure_tag('tag1') |
215 | self.factory.ensure_tag('tag2') |
216 | job = TestJob.from_json_and_user( |
217 | - json.dumps({'device_type':'panda', 'device_tags':['tag1', 'tag2']}), |
218 | + self.factory.make_job_json(device_tags=['tag1', 'tag2']), |
219 | self.factory.make_user()) |
220 | self.assertEqual( |
221 | set(tag.name for tag in job.tags.all()), set(['tag1', 'tag2'])) |
222 | @@ -189,15 +202,75 @@ |
223 | self.factory.ensure_device_type(name='panda') |
224 | self.factory.ensure_tag('tag') |
225 | job1 = TestJob.from_json_and_user( |
226 | - json.dumps({'device_type':'panda', 'device_tags':['tag']}), |
227 | + self.factory.make_job_json(device_tags=['tag']), |
228 | self.factory.make_user()) |
229 | job2 = TestJob.from_json_and_user( |
230 | - json.dumps({'device_type':'panda', 'device_tags':['tag']}), |
231 | + self.factory.make_job_json(device_tags=['tag']), |
232 | self.factory.make_user()) |
233 | self.assertEqual( |
234 | set(tag.pk for tag in job1.tags.all()), |
235 | set(tag.pk for tag in job2.tags.all())) |
236 | |
237 | + def test_from_json_and_user_rejects_invalid_json(self): |
238 | + self.assertRaises( |
239 | + ValueError, TestJob.from_json_and_user, '{', |
240 | + self.factory.make_user()) |
241 | + |
242 | + def test_from_json_and_user_rejects_invalid_job(self): |
243 | + # job data must have the 'actions' and 'timeout' properties, so this |
244 | + # will be rejected. |
245 | + self.assertRaises( |
246 | + ValueError, TestJob.from_json_and_user, '{}', |
247 | + self.factory.make_user()) |
248 | + |
249 | + def make_job_json_for_stream_name(self, stream_name): |
250 | + return self.factory.make_job_json( |
251 | + actions=[ |
252 | + { |
253 | + 'command':'submit_results', |
254 | + 'parameters': { |
255 | + 'server': '...', |
256 | + 'stream': stream_name, |
257 | + } |
258 | + } |
259 | + ]) |
260 | + |
261 | + def test_from_json_and_user_sets_group_from_bundlestream(self): |
262 | + group = Group.objects.create(name='group') |
263 | + user = self.factory.make_user() |
264 | + user.groups.add(group) |
265 | + b = BundleStream.objects.create( |
266 | + group=group, slug='blah', is_public=True) |
267 | + b.save() |
268 | + j = self.make_job_json_for_stream_name(b.pathname) |
269 | + job = TestJob.from_json_and_user(j, user) |
270 | + self.assertEqual(group, job.group) |
271 | + |
272 | + def test_from_json_and_user_sets_is_public_from_bundlestream(self): |
273 | + group = Group.objects.create(name='group') |
274 | + user = self.factory.make_user() |
275 | + user.groups.add(group) |
276 | + b = BundleStream.objects.create( |
277 | + group=group, slug='blah', is_public=False) |
278 | + b.save() |
279 | + j = self.make_job_json_for_stream_name(b.pathname) |
280 | + job = TestJob.from_json_and_user(j, user) |
281 | + self.assertEqual(False, job.is_public) |
282 | + |
283 | + def test_from_json_and_user_rejects_missing_bundlestream(self): |
284 | + user = self.factory.make_user() |
285 | + j = self.make_job_json_for_stream_name('no such stream') |
286 | + self.assertRaises(ValueError, TestJob.from_json_and_user, j, user) |
287 | + |
288 | + def test_from_json_and_user_rejects_inaccessible_bundlestream(self): |
289 | + stream_user = self.factory.make_user() |
290 | + job_user = self.factory.make_user() |
291 | + b = BundleStream.objects.create( |
292 | + user=stream_user, slug='blah', is_public=True) |
293 | + b.save() |
294 | + j = self.make_job_json_for_stream_name(b.pathname) |
295 | + self.assertRaises(ValueError, TestJob.from_json_and_user, j, job_user) |
296 | + |
297 | |
298 | class TestSchedulerAPI(TestCaseWithFactory): |
299 | |
300 | @@ -231,8 +304,7 @@ |
301 | Permission.objects.get(codename='add_testjob')) |
302 | user.save() |
303 | server = self.server_proxy('test', 'test') |
304 | - self.factory.ensure_device_type(name='panda') |
305 | - definition = json.dumps({'device_type':'panda'}) |
306 | + definition = self.factory.make_job_json() |
307 | job_id = server.scheduler.submit_job(definition) |
308 | job = TestJob.objects.get(id=job_id) |
309 | self.assertEqual(definition, job.definition) |
310 | @@ -296,14 +368,19 @@ |
311 | |
312 | def test_getJobForBoard_returns_json(self): |
313 | device = self.factory.make_device(hostname='panda01') |
314 | - definition = {'foo': 'bar', 'target': 'panda01'} |
315 | + definition = self.factory.make_job_data(target='panda01') |
316 | self.factory.make_testjob( |
317 | requested_device=device, definition=json.dumps(definition)) |
318 | self.assertEqual( |
319 | definition, self.source.getJobForBoard('panda01')) |
320 | |
321 | - health_job = json.dumps({'health_check': True}) |
322 | - ordinary_job = json.dumps({'health_check': False}) |
323 | + @property |
324 | + def health_job(self): |
325 | + return self.factory.make_job_json(health_check=True) |
326 | + |
327 | + @property |
328 | + def ordinary_job(self): |
329 | + return self.factory.make_job_json(health_check=False) |
330 | |
331 | def assertHealthJobAssigned(self, device): |
332 | job_data = self.source.getJobForBoard(device.hostname) |
333 | @@ -371,7 +448,7 @@ |
334 | def test_getJobForBoard_considers_device_type(self): |
335 | panda_type = self.factory.ensure_device_type(name='panda') |
336 | self.factory.make_device(hostname='panda01', device_type=panda_type) |
337 | - definition = {'foo': 'bar'} |
338 | + definition = self.factory.make_job_data() |
339 | self.factory.make_testjob( |
340 | requested_device_type=panda_type, |
341 | definition=json.dumps(definition)) |
342 | @@ -383,8 +460,8 @@ |
343 | panda_type = self.factory.ensure_device_type(name='panda') |
344 | panda01 = self.factory.make_device( |
345 | hostname='panda01', device_type=panda_type) |
346 | - first_definition = {'foo': 'bar', 'target': 'panda01'} |
347 | - second_definition = {'foo': 'baz', 'target': 'panda01'} |
348 | + first_definition = self.factory.make_job_data(foo='bar', target='panda01') |
349 | + second_definition = self.factory.make_job_data(foo='baz', target='panda01') |
350 | self.factory.make_testjob( |
351 | requested_device=panda01, definition=json.dumps(first_definition), |
352 | submit_time=datetime.datetime.now() - datetime.timedelta(days=1)) |
353 | @@ -399,12 +476,13 @@ |
354 | panda_type = self.factory.ensure_device_type(name='panda') |
355 | panda01 = self.factory.make_device( |
356 | hostname='panda01', device_type=panda_type) |
357 | - type_definition = {'foo': 'bar'} |
358 | + type_definition = self.factory.make_job_data() |
359 | self.factory.make_testjob( |
360 | requested_device_type=panda_type, |
361 | definition=json.dumps(type_definition), |
362 | submit_time=datetime.datetime.now() - datetime.timedelta(days=1)) |
363 | - device_definition = {'foo': 'baz', 'target': 'panda01'} |
364 | + device_definition = self.factory.make_job_data( |
365 | + foo='baz', target='panda01') |
366 | self.factory.make_testjob( |
367 | requested_device=panda01, |
368 | definition=json.dumps(device_definition)) |
369 | @@ -417,7 +495,7 @@ |
370 | panda01 = self.factory.make_device( |
371 | hostname='panda01', device_type=panda_type) |
372 | self.factory.make_device(hostname='panda02', device_type=panda_type) |
373 | - definition = {'foo': 'bar', 'target': 'panda01'} |
374 | + definition = self.factory.make_job_data(foo='bar', target='panda01') |
375 | self.factory.make_testjob( |
376 | requested_device=panda01, |
377 | definition=json.dumps(definition)) |
378 | @@ -515,7 +593,7 @@ |
379 | def test_getJobForBoard_inserts_target_into_json(self): |
380 | panda_type = self.factory.ensure_device_type(name='panda') |
381 | self.factory.make_device(hostname='panda01', device_type=panda_type) |
382 | - definition = {'foo': 'bar'} |
383 | + definition = self.factory.make_job_data(device_type='panda') |
384 | self.factory.make_testjob( |
385 | requested_device_type=panda_type, |
386 | definition=json.dumps(definition)) |
387 | |
388 | === modified file 'lava_scheduler_daemon/dbjobsource.py' |
389 | --- lava_scheduler_daemon/dbjobsource.py 2012-03-07 01:59:51 +0000 |
390 | +++ lava_scheduler_daemon/dbjobsource.py 2012-03-09 04:16:33 +0000 |
391 | @@ -88,25 +88,11 @@ |
392 | def _get_json_data(self, job): |
393 | json_data = json.loads(job.definition) |
394 | json_data['target'] = job.actual_device.hostname |
395 | - # The rather extreme paranoia in what follows could be much reduced if |
396 | - # we thoroughly validated job data in submit_job. We don't (yet?) |
397 | - # and there is no sane way to report errors at this stage, so, |
398 | - # paranoia (the dispatcher will choke on bogus input in a more |
399 | - # informative way). |
400 | - if 'actions' not in json_data: |
401 | - return json_data |
402 | - actions = json_data['actions'] |
403 | - for action in actions: |
404 | - if not isinstance(action, dict): |
405 | - continue |
406 | - if action.get('command') != 'submit_results': |
407 | - continue |
408 | - params = action.get('parameters') |
409 | - if not isinstance(params, dict): |
410 | - continue |
411 | + for action in json_data['actions']: |
412 | + if not action['command'].startswith('submit_results'): |
413 | + continue |
414 | + params = action['parameters'] |
415 | params['token'] = job.submit_token.secret |
416 | - if not 'server' in params or not isinstance(params['server'], unicode): |
417 | - continue |
418 | parsed = urlparse.urlsplit(params['server']) |
419 | netloc = job.submitter.username + '@' + parsed.hostname |
420 | parsed = list(parsed) |
421 | |
422 | === modified file 'setup.py' |
423 | --- setup.py 2012-03-07 03:49:20 +0000 |
424 | +++ setup.py 2012-03-09 04:16:33 +0000 |
425 | @@ -35,6 +35,8 @@ |
426 | install_requires=[ |
427 | "django-restricted-resource", |
428 | "django-tables2 >= 0.9.4", |
429 | + "lava-dashboard", |
430 | + "lava-dispatcher", |
431 | "lava-server >= 0.10", |
432 | "simplejson", |
433 | "south >= 0.7.3", |
I think you forgot the prerequisite branch - this makes reading it harder.
60 + try: objects. get(pathname= stream) DoesNotExist:
61 + bundle_stream = BundleStream.
62 + except BundleStream.
63 + raise ValueError("stream %s not found" % stream)
429 + "lava-dashboard",
So it's official (netcraft confirms it). The scheduler depends on the dashboard now. Can we please look at how to merge them? This is a bigger topic. I'd like to get rid of lots of things from the dashboard proper. Perhaps we should move the base models to lava-server (tree-wise) application. lava.silo or lava.testdata or something like that.