Merge lp:lava-dashboard/multinode into lp:lava-dashboard

Proposed by Neil Williams on 2013-08-20
Status: Merged
Approved by: Neil Williams on 2013-08-28
Approved revision: 411
Merged at revision: 415
Proposed branch: lp:lava-dashboard/multinode
Merge into: lp:lava-dashboard
Diff against target: 246 lines (+192/-4)
2 files modified
dashboard_app/templates/dashboard_app/_test_run_list_table.html (+6/-0)
dashboard_app/xmlrpc.py (+186/-4)
To merge this branch: bzr merge lp:lava-dashboard/multinode
Reviewer Review Type Date Requested Status
Neil Williams Approve on 2013-08-28
Review via email: mp+181101@code.launchpad.net

Description of the change

Landing MultiNode.

Handles the aggregation of MultiNode result bundles after the XMLRPC calls which are coordinated by LAVA Coordinator.

This branch applies without conflicts.

lava-dashboard will be the first shared MultiNode branch to be merged as these changes are not dependent on changes in the other shared branches.

To post a comment you must log in.
Antonio Terceiro (terceiro) wrote :

> === modified file 'dashboard_app/xmlrpc.py'
> --- dashboard_app/xmlrpc.py 2013-05-02 10:35:29 +0000
> +++ dashboard_app/xmlrpc.py 2013-08-20 16:56:49 +0000
> @@ -2,7 +2,7 @@
> #
> # Author: Zygmunt Krynicki <email address hidden>
> #
> -# This file is part of Launch Control.
> +# This file is part of LAVA Dashboard
> #
> # Launch Control is free software: you can redistribute it and/or modify
> # it under the terms of the GNU Affero General Public License version 3
> @@ -25,7 +25,9 @@
> import logging
> import re
> import xmlrpclib
> -
> +import hashlib
> +import json
> +import os
> from django.contrib.auth.models import User, Group
> from django.core.urlresolvers import reverse
> from django.db import IntegrityError, DatabaseError
> @@ -243,6 +245,152 @@
> 'dashboard_app.views.redirect_to_bundle',
> kwargs={'content_sha1':bundle.content_sha1}))
>
> + def put_pending(self, content, group_name):
> + """
[...]
> + """
> + try:
> + # add this to a list which put_group can use.
> + sha1 = hashlib.sha1()
> + sha1.update(content)
> + hexdigest = sha1.hexdigest()
> + groupfile = "/tmp/%s" % group_name
> + with open(groupfile, "a+") as grp_file:
> + grp_file.write("%s\n" % content)
> + return hexdigest
> + except Exception as e:
> + logging.debug("Dashboard pending submission caused an exception: %s" % e)

Is there a race condition here? It's fine for two or more processes to append
to the same file, but it's possible that depending on the size of the bundles
and on line buffering issues the contents of different bundles might get
intermingled. Maybe we should write each bundle to its own separate file, then
read them all on put_group.

Also, I miss some sort of authentication to avoid the risk of having attackers
submitting random crap into bundle streams for multinode groups. I guess
put_group already handles authentication because it uses the underlying bundle
stream access control?

Neil Williams (codehelp) wrote :

On Thu, 22 Aug 2013 01:27:18 -0000
Antonio Terceiro <email address hidden> wrote:

> > + def put_pending(self, content, group_name):
> > + try:
> > + # add this to a list which put_group can use.
> > + sha1 = hashlib.sha1()
> > + sha1.update(content)
> > + hexdigest = sha1.hexdigest()
> > + groupfile = "/tmp/%s" % group_name
> > + with open(groupfile, "a+") as grp_file:
> > + grp_file.write("%s\n" % content)
> > + return hexdigest
> > + except Exception as e:
> > + logging.debug("Dashboard pending submission caused an
> > exception: %s" % e)
>
> Is there a race condition here?

Possibly, only within that one group. The Coordinator already ensures
that put_group waits for a fixed period to allow the last put_pending
for this group to complete but large groups could have a problem here.
(There is a reboot into the master image between the last possible sync
operation and the XMLRPC call too.)

> It's fine for two or more processes
> to append to the same file, but it's possible that depending on the
> size of the bundles and on line buffering issues the contents of
> different bundles might get intermingled. Maybe we should write each
> bundle to its own separate file, then read them all on put_group.

> Also, I miss some sort of authentication to avoid the risk of having
> attackers submitting random crap into bundle streams for multinode
> groups. I guess put_group already handles authentication because it
> uses the underlying bundle stream access control?

put_group uses authentication, yes.

I'll have a look at some of the checks in _put - what we don't want is
to use the full _put function which causes the creation and
deserialization of the pending bundle.

--

Neil Williams
=============
http://www.linux.codehelp.co.uk/

lp:lava-dashboard/multinode updated on 2013-08-24
411. By Neil Williams on 2013-08-24

Neil Williams 2013-08-23 Add support for checking authentication of the
 pending bundles by porting the stream check code from _put without
 allowing the pending bundle into the database before aggregation.

Neil Williams (codehelp) wrote :

Approved with update for review comments.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'dashboard_app/templates/dashboard_app/_test_run_list_table.html'
--- dashboard_app/templates/dashboard_app/_test_run_list_table.html 2013-02-25 09:25:30 +0000
+++ dashboard_app/templates/dashboard_app/_test_run_list_table.html 2013-08-24 08:11:25 +0000
@@ -2,6 +2,7 @@
2<table class="demo_jui display" id="test_runs">2<table class="demo_jui display" id="test_runs">
3 <thead>3 <thead>
4 <tr>4 <tr>
5 <th>{% trans "Device" %}</th>
5 <th>{% trans "Test Run" %}</th>6 <th>{% trans "Test Run" %}</th>
6 <th>{% trans "Test" %}</th>7 <th>{% trans "Test" %}</th>
7 <th>{% trans "Passes" %}</th>8 <th>{% trans "Passes" %}</th>
@@ -13,6 +14,11 @@
13 <tbody>14 <tbody>
14 {% for test_run in test_run_list %}15 {% for test_run in test_run_list %}
15 <tr>16 <tr>
17 {% for attribute in test_run.attributes.all %}
18 {% if attribute.name == "target" %}
19 <td>{{ attribute.value }}</td>
20 {% endif %}
21 {% endfor %}
16 <td><a href="{{ test_run.get_absolute_url }}"><code>{{ test_run.test }} results<code/></a></td>22 <td><a href="{{ test_run.get_absolute_url }}"><code>{{ test_run.test }} results<code/></a></td>
17 <td>{{ test_run.test }}</td>23 <td>{{ test_run.test }}</td>
18 <td>{{ test_run.get_summary_results.pass }}</td>24 <td>{{ test_run.get_summary_results.pass }}</td>
1925
=== modified file 'dashboard_app/xmlrpc.py'
--- dashboard_app/xmlrpc.py 2013-05-02 10:35:29 +0000
+++ dashboard_app/xmlrpc.py 2013-08-24 08:11:25 +0000
@@ -2,7 +2,7 @@
2#2#
3# Author: Zygmunt Krynicki <zygmunt.krynicki@linaro.org>3# Author: Zygmunt Krynicki <zygmunt.krynicki@linaro.org>
4#4#
5# This file is part of Launch Control.5# This file is part of LAVA Dashboard
6#6#
7# Launch Control is free software: you can redistribute it and/or modify7# Launch Control is free software: you can redistribute it and/or modify
8# it under the terms of the GNU Affero General Public License version 38# it under the terms of the GNU Affero General Public License version 3
@@ -25,7 +25,9 @@
25import logging25import logging
26import re26import re
27import xmlrpclib27import xmlrpclib
2828import hashlib
29import json
30import os
29from django.contrib.auth.models import User, Group31from django.contrib.auth.models import User, Group
30from django.core.urlresolvers import reverse32from django.core.urlresolvers import reverse
31from django.db import IntegrityError, DatabaseError33from django.db import IntegrityError, DatabaseError
@@ -105,9 +107,9 @@
105 logging.debug("Getting bundle stream")107 logging.debug("Getting bundle stream")
106 bundle_stream = BundleStream.objects.accessible_by_principal(self.user).get(pathname=pathname)108 bundle_stream = BundleStream.objects.accessible_by_principal(self.user).get(pathname=pathname)
107 except BundleStream.DoesNotExist:109 except BundleStream.DoesNotExist:
108 logging.debug("Bundle stream does not exists, aborting")110 logging.debug("Bundle stream does not exist, aborting")
109 raise xmlrpclib.Fault(errors.NOT_FOUND,111 raise xmlrpclib.Fault(errors.NOT_FOUND,
110 "Bundle stream not found")112 "Bundle stream not found")
111 if not bundle_stream.can_upload(self.user):113 if not bundle_stream.can_upload(self.user):
112 raise xmlrpclib.Fault(114 raise xmlrpclib.Fault(
113 errors.FORBIDDEN, "You cannot upload to this stream")115 errors.FORBIDDEN, "You cannot upload to this stream")
@@ -243,6 +245,186 @@
243 'dashboard_app.views.redirect_to_bundle',245 'dashboard_app.views.redirect_to_bundle',
244 kwargs={'content_sha1':bundle.content_sha1}))246 kwargs={'content_sha1':bundle.content_sha1}))
245247
248 def put_pending(self, content, pathname, group_name):
249 """
250 Name
251 ----
252 `put_pending` (`content`, `pathname`, `group_name`)
253
254 Description
255 -----------
256 MultiNode internal call.
257
258 Stores the bundle until the coordinator allows the complete
259 bundle list to be aggregated from the list and submitted by put_group
260
261 Arguments
262 ---------
263 `content`: string
264 Full text of the bundle. This *MUST* be a valid JSON
265 document and it *SHOULD* match the "Dashboard Bundle Format
266 1.0" schema. The SHA1 of the content *MUST* be unique or a
267 ``Fault(409, "...")`` is raised. This is used to protect
268 from simple duplicate submissions.
269 `pathname`: string
270 Pathname of the bundle stream where a new bundle should
271 be created and stored. This argument *MUST* designate a
272 pre-existing bundle stream or a ``Fault(404, "...")`` exception
273 is raised. In addition the user *MUST* have access
274 permission to upload bundles there or a ``Fault(403, "...")``
275 exception is raised. See below for access rules.
276 `group_name`: string
277 Unique ID of the MultiNode group. Other pending bundles will
278 be aggregated into a single result bundle for this group.
279
280 Return value
281 ------------
282 If all goes well this function returns the SHA1 of the content.
283
284 Exceptions raised
285 -----------------
286 404
287 Either:
288
289 - Bundle stream not found
290 - Uploading to specified stream is not permitted
291 409
292 Duplicate bundle content
293
294 Rules for bundle stream access
295 ------------------------------
296 The following rules govern bundle stream upload access rights:
297 - all anonymous streams are accessible
298 - personal streams are accessible to owners
299 - team streams are accessible to team members
300
301 """
302 try:
303 logging.debug("Getting bundle stream")
304 bundle_stream = BundleStream.objects.accessible_by_principal(self.user).get(pathname=pathname)
305 except BundleStream.DoesNotExist:
306 logging.debug("Bundle stream does not exist, aborting")
307 raise xmlrpclib.Fault(errors.NOT_FOUND,
308 "Bundle stream not found")
309 if not bundle_stream.can_upload(self.user):
310 raise xmlrpclib.Fault(
311 errors.FORBIDDEN, "You cannot upload to this stream")
312 try:
313 # add this to a list which put_group can use.
314 sha1 = hashlib.sha1()
315 sha1.update(content)
316 hexdigest = sha1.hexdigest()
317 groupfile = "/tmp/%s" % group_name
318 with open(groupfile, "a+") as grp_file:
319 grp_file.write("%s\n" % content)
320 return hexdigest
321 except Exception as e:
322 logging.debug("Dashboard pending submission caused an exception: %s" % e)
323
324 def put_group(self, content, content_filename, pathname, group_name):
325 """
326 Name
327 ----
328 `put_group` (`content`, `content_filename`, `pathname`, `group_name`)
329
330 Description
331 -----------
332 MultiNode internal call.
333
334 Adds the final bundle to the list, aggregates the list
335 into a single group bundle and submits the group bundle.
336
337 Arguments
338 ---------
339 `content`: string
340 Full text of the bundle. This *MUST* be a valid JSON
341 document and it *SHOULD* match the "Dashboard Bundle Format
342 1.0" schema. The SHA1 of the content *MUST* be unique or a
343 ``Fault(409, "...")`` is raised. This is used to protect
344 from simple duplicate submissions.
345 `content_filename`: string
346 Name of the file that contained the text of the bundle. The
347 `content_filename` can be an arbitrary string and will be
348 stored along with the content for reference.
349 `pathname`: string
350 Pathname of the bundle stream where a new bundle should
351 be created and stored. This argument *MUST* designate a
352 pre-existing bundle stream or a ``Fault(404, "...")`` exception
353 is raised. In addition the user *MUST* have access
354 permission to upload bundles there or a ``Fault(403, "...")``
355 exception is raised. See below for access rules.
356 `group_name`: string
357 Unique ID of the MultiNode group. Other pending bundles will
358 be aggregated into a single result bundle for this group. At
359 least one other bundle must have already been submitted as
360 pending for the specified MultiNode group. LAVA Coordinator
361 causes the parent job to wait until all nodes have been marked
362 as having pending bundles, even if some bundles are empty.
363
364 Return value
365 ------------
366 If all goes well this function returns the full URL of the bundle.
367
368 Exceptions raised
369 -----------------
370 ValueError:
371 One or more bundles could not be converted to JSON prior
372 to aggregation.
373 404
374 Either:
375
376 - Bundle stream not found
377 - Uploading to specified stream is not permitted
378 409
379 Duplicate bundle content
380
381 Rules for bundle stream access
382 ------------------------------
383 The following rules govern bundle stream upload access rights:
384 - all anonymous streams are accessible
385 - personal streams are accessible to owners
386 - team streams are accessible to team members
387
388 """
389 grp_file = "/tmp/%s" % group_name
390 bundle_set = {}
391 bundle_set[group_name] = []
392 if os.path.isfile(grp_file):
393 with open(grp_file, "r") as grp_data:
394 grp_list = grp_data.readlines()
395 for testrun in grp_list:
396 bundle_set[group_name].append(json.loads(testrun))
397 # Note: now that we have the data from the group, the group data file could be re-used
398 # as an error log which is simpler than debugging through XMLRPC.
399 else:
400 raise ValueError("Aggregation failure for %s - check coordinator rpc_delay?" % group_name)
401 group_tests = []
402 try:
403 json_data = json.loads(content)
404 except ValueError:
405 logging.debug("Invalid JSON content within the sub_id zero bundle")
406 json_data = None
407 try:
408 bundle_set[group_name].append(json_data)
409 except Exception as e:
410 logging.debug("appending JSON caused exception %s" % e)
411 try:
412 for bundle_list in bundle_set[group_name]:
413 for test_run in bundle_list['test_runs']:
414 group_tests.append(test_run)
415 except Exception as e:
416 logging.debug("aggregating bundles caused exception %s" % e)
417 group_content = json.dumps({"test_runs": group_tests, "format": json_data['format']})
418 bundle = self._put(group_content, content_filename, pathname)
419 logging.debug("Returning permalink to aggregated bundle for %s" % group_name)
420 permalink = self._context.request.build_absolute_uri(
421 reverse('dashboard_app.views.redirect_to_bundle',
422 kwargs={'content_sha1': bundle.content_sha1}))
423 # only delete the group file when things go well.
424 if os.path.isfile(grp_file):
425 os.remove(grp_file)
426 return permalink
427
246 def get(self, content_sha1):428 def get(self, content_sha1):
247 """429 """
248 Name430 Name

Subscribers

People subscribed via source and target branches

to status/vote changes: