Merge lp:~canonical-platform-qa/ubuntu-community-testing/initial-reporting into lp:ubuntu-community-testing

Proposed by Christopher Lee on 2015-09-09
Status: Merged
Merged at revision: 30
Proposed branch: lp:~canonical-platform-qa/ubuntu-community-testing/initial-reporting
Merge into: lp:ubuntu-community-testing
Diff against target: 546 lines (+427/-11)
13 files modified
ubuntu_pt_community/__init__.py (+5/-1)
ubuntu_pt_community/api/v1.py (+4/-8)
ubuntu_pt_community/db/__init__.py (+5/-2)
ubuntu_pt_community/db/db.py (+6/-0)
ubuntu_pt_community/pages/__init__.py (+27/-0)
ubuntu_pt_community/pages/pages.py (+69/-0)
ubuntu_pt_community/pages/reports.py (+167/-0)
ubuntu_pt_community/templates/base.html (+30/-0)
ubuntu_pt_community/templates/results/all_testsuites.html (+23/-0)
ubuntu_pt_community/templates/results/index.html (+10/-0)
ubuntu_pt_community/templates/results/latest_uploads.html (+27/-0)
ubuntu_pt_community/tests/__init__.py (+17/-0)
ubuntu_pt_community/tests/test_reports.py (+37/-0)
To merge this branch: bzr merge lp:~canonical-platform-qa/ubuntu-community-testing/initial-reporting
Reviewer Review Type Date Requested Status
Brendan Donegan (community) Approve on 2015-09-17
Nicholas Skaggs (community) 2015-09-09 Approve on 2015-09-15
Review via email: mp+270480@code.launchpad.net

Commit message

Initial lot of reporting.

Description of the change

Initial reporting incl.
  - "Latest Uploads" - list of upload date with testsuites run and the users email
  - "All testsuites" - simple stats for each testsuite that we have upload details for (runs, pass/fail numbers, success rates).

These are the first run and we can build on it from here (incl. making it look better :-P as well as navigation etc.).

To post a comment you must log in.
Christopher Lee (veebers) wrote :

Just marked as WIP as I notice that somethings are incorrect. Adding tests and sorting out these issues.

Nicholas Skaggs (nskaggs) wrote :

This looks good from my perspective. Chris, it looks like only the results from today will display in the 'latest results?'. I didn't actually build and deploy this, so I might be mistaken :-)

If so however, I might see this as a problem when days rollover. When the day switches to a new day, I might not see my results, even though I submitted them say within the last hour. Could we just simply display the last X number of results? Or if you wish limit it to a more reasonable number of days, perhaps 3 instead of just 1. If you do use days, it might still be wise to limit the number of results displayed so we don't have a long list on the page.

Christopher Lee (veebers) wrote :

@balloons at the moment there is no limit to how many it displays and just shows all uploads with the latest at the top.

We can iterate on doing smarter things with this (i.e. pagination (using JIT requests), just show X days etc.)

Nicholas Skaggs (nskaggs) wrote :

Ack, sounds fine to me.

review: Approve
Brendan Donegan (brendan-donegan) wrote :

Mostly looks great - some questions about use of docstrings though

review: Needs Information
41. By Christopher Lee on 2015-09-17

Better and additional docstrings.

42. By Christopher Lee on 2015-09-17

Fix flake8 errors.

Christopher Lee (veebers) wrote :

> Mostly looks great - some questions about use of docstrings though
Good catch with the docstrings. Have added more and improved the existing.

Regarding docstrings for 'private' methods; I see no issue with having them, They are a well known format which can streamline a developer reading them and contain details regarding the intention of the method itself.

Brendan Donegan (brendan-donegan) wrote :

Ok looks better now. +1

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'ubuntu_pt_community/__init__.py'
2--- ubuntu_pt_community/__init__.py 2015-08-18 00:55:26 +0000
3+++ ubuntu_pt_community/__init__.py 2015-09-17 05:33:50 +0000
4@@ -18,7 +18,10 @@
5 import logging
6 import sys
7
8-from ubuntu_pt_community import api
9+from ubuntu_pt_community import (
10+ api,
11+ pages,
12+)
13
14 from flask import Flask
15
16@@ -40,3 +43,4 @@
17 app = Flask(__name__)
18
19 api.define_api_routes(app)
20+pages.define_page_routes(app)
21
22=== modified file 'ubuntu_pt_community/api/v1.py'
23--- ubuntu_pt_community/api/v1.py 2015-08-18 10:58:22 +0000
24+++ ubuntu_pt_community/api/v1.py 2015-09-17 05:33:50 +0000
25@@ -28,7 +28,9 @@
26 from qakit.practitest.report_checkbox_results_to_practitest import (
27 upload_results
28 )
29-from ubuntu_pt_community import auth, db
30+from ubuntu_pt_community import auth
31+from ubuntu_pt_community.db import get_results_collection
32+
33
34 logger = logging.getLogger(__name__)
35
36@@ -86,7 +88,7 @@
37 )
38
39 try:
40- collection = get_results_database()
41+ collection = get_results_collection()
42 insert_id = collection.insert(details)
43 logger.info(
44 'Inserted details for {} with id {}'.format(
45@@ -103,12 +105,6 @@
46 logger.error('Failed to get database connection: ', e)
47
48
49-def get_results_database():
50- database_name = 'community_practitest'
51- collection_name = 'uploaded_results'
52- return db.get_collection(database_name, collection_name)
53-
54-
55 def get_user_email_address(request):
56 try:
57 return request.form['uploader_email']
58
59=== modified file 'ubuntu_pt_community/db/__init__.py'
60--- ubuntu_pt_community/db/__init__.py 2015-08-13 05:52:38 +0000
61+++ ubuntu_pt_community/db/__init__.py 2015-09-17 05:33:50 +0000
62@@ -16,6 +16,9 @@
63 # along with this program. If not, see <http://www.gnu.org/licenses/>.
64 #
65
66-from ubuntu_pt_community.db.db import get_collection
67+from ubuntu_pt_community.db.db import (
68+ get_collection,
69+ get_results_collection,
70+)
71
72-__all__ = ['get_collection']
73+__all__ = ['get_collection', 'get_results_collection']
74
75=== modified file 'ubuntu_pt_community/db/db.py'
76--- ubuntu_pt_community/db/db.py 2015-08-17 06:19:29 +0000
77+++ ubuntu_pt_community/db/db.py 2015-09-17 05:33:50 +0000
78@@ -48,6 +48,12 @@
79 return db[collection_name]
80
81
82+def get_results_collection():
83+ database_name = 'community_practitest'
84+ collection_name = 'uploaded_results'
85+ return get_collection(database_name, collection_name)
86+
87+
88 def get_config_file_path():
89 try:
90 return os.path.join(
91
92=== added directory 'ubuntu_pt_community/pages'
93=== added file 'ubuntu_pt_community/pages/__init__.py'
94--- ubuntu_pt_community/pages/__init__.py 1970-01-01 00:00:00 +0000
95+++ ubuntu_pt_community/pages/__init__.py 2015-09-17 05:33:50 +0000
96@@ -0,0 +1,27 @@
97+#
98+# Ubuntu PractiTest Community results processor
99+# Copyright (C) 2015 Canonical
100+#
101+# This program is free software: you can redistribute it and/or modify
102+# it under the terms of the GNU General Public License as published by
103+# the Free Software Foundation, either version 3 of the License, or
104+# (at your option) any later version.
105+#
106+# This program is distributed in the hope that it will be useful,
107+# but WITHOUT ANY WARRANTY; without even the implied warranty of
108+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
109+# GNU General Public License for more details.
110+#
111+# You should have received a copy of the GNU General Public License
112+# along with this program. If not, see <http://www.gnu.org/licenses/>.
113+#
114+
115+from ubuntu_pt_community.pages import pages
116+
117+"""Prepare the available api routes."""
118+
119+__all__ = ['define_page_routes']
120+
121+
122+def define_page_routes(webapp):
123+ pages.define_routes(webapp)
124
125=== added file 'ubuntu_pt_community/pages/pages.py'
126--- ubuntu_pt_community/pages/pages.py 1970-01-01 00:00:00 +0000
127+++ ubuntu_pt_community/pages/pages.py 2015-09-17 05:33:50 +0000
128@@ -0,0 +1,69 @@
129+#
130+# Ubuntu PractiTest Community results processor
131+# Copyright (C) 2015 Canonical
132+#
133+# This program is free software: you can redistribute it and/or modify
134+# it under the terms of the GNU General Public License as published by
135+# the Free Software Foundation, either version 3 of the License, or
136+# (at your option) any later version.
137+#
138+# This program is distributed in the hope that it will be useful,
139+# but WITHOUT ANY WARRANTY; without even the implied warranty of
140+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
141+# GNU General Public License for more details.
142+#
143+# You should have received a copy of the GNU General Public License
144+# along with this program. If not, see <http://www.gnu.org/licenses/>.
145+#
146+
147+import logging
148+from flask import render_template
149+
150+from ubuntu_pt_community.pages import reports
151+
152+
153+logger = logging.getLogger(__name__)
154+
155+
156+class PageDefinition:
157+
158+ def __init__(self, name, route_name, route_func):
159+ """Encapsulate the report view details."""
160+ self.name = name
161+ self.route_name = route_name
162+ self.route_func = route_func
163+
164+ @property
165+ def route_function_name(self):
166+ """Return the name of the route function for use_url."""
167+ return self.route_func.__name__
168+
169+
170+# Keep a list of report pages so we don't have to duplicate things when listing
171+# them and adding routes.
172+Report_Pages = [
173+ PageDefinition(
174+ 'All Results',
175+ '/reports/all_results',
176+ reports.view_all_results
177+ ),
178+
179+ PageDefinition(
180+ 'Latest Uploads',
181+ '/reports/latest',
182+ reports.view_latest_uploads
183+ ),
184+]
185+
186+
187+def define_routes(webapp):
188+ """Setup all routes for available reports (incl. list of reports."""
189+ webapp.add_url_rule('/reports', view_func=view_reports)
190+
191+ for page in Report_Pages:
192+ webapp.add_url_rule(page.route_name, view_func=page.route_func)
193+
194+
195+def view_reports():
196+ """Render a simple list of links to available reports."""
197+ return render_template('results/index.html', reports=Report_Pages)
198
199=== added file 'ubuntu_pt_community/pages/reports.py'
200--- ubuntu_pt_community/pages/reports.py 1970-01-01 00:00:00 +0000
201+++ ubuntu_pt_community/pages/reports.py 2015-09-17 05:33:50 +0000
202@@ -0,0 +1,167 @@
203+#
204+# Ubuntu PractiTest Community results processor
205+
206+# Copyright (C) 2015 Canonical
207+#
208+# This program is free software: you can redistribute it and/or modify
209+# it under the terms of the GNU General Public License as published by
210+# the Free Software Foundation, either version 3 of the License, or
211+# (at your option) any later version.
212+#
213+# This program is distributed in the hope that it will be useful,
214+# but WITHOUT ANY WARRANTY; without even the implied warranty of
215+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
216+# GNU General Public License for more details.
217+#
218+# You should have received a copy of the GNU General Public License
219+# along with this program. If not, see <http://www.gnu.org/licenses/>.
220+#
221+
222+import json
223+import logging
224+from collections import (
225+ defaultdict,
226+ namedtuple,
227+)
228+from flask import render_template
229+
230+from ubuntu_pt_community import db
231+
232+logger = logging.getLogger(__name__)
233+
234+
235+def view_all_results():
236+ """Render report displaying all uploaded testsuites with stats.
237+
238+ Stats displayed are number of runs, number of passes and fails and the % of
239+ successful runs.
240+ """
241+ results_collection = db.get_results_collection()
242+ all_uploads = results_collection.find()
243+ simple_report_data = _get_simple_report_data(all_uploads)
244+ return render_template(
245+ 'results/all_testsuites.html',
246+ results=simple_report_data
247+ )
248+
249+
250+def _get_simple_report_data(all_uploads):
251+ """Produce a list of suite results.
252+
253+ :param all_uploads: list of dicts containing uploaded data details.
254+ :returns: a list of TestsuiteResult objects.
255+ """
256+ TestsuiteResult = namedtuple(
257+ 'TestsuiteResult',
258+ ['name', 'runs', 'passes', 'fails', 'success_rate']
259+ )
260+
261+ # all_uploads will be a list of dicts
262+ # dict will have keys:
263+ # - results: a json string with the result details.
264+ # - user_email: email of the user who uploaded the results
265+ # - uploaded: a date of uploading
266+ testsuites = defaultdict(list)
267+ for upload in all_uploads:
268+ upload_id = str(upload['_id'])
269+
270+ try:
271+ results = _result_dict_from_document(upload)
272+ except KeyError as e:
273+ logger.warning(e)
274+ continue
275+
276+ only_tests = _get_only_testcases(results)
277+
278+ for testname in only_tests:
279+ # outcome will be 'pass' or 'fail'
280+ try:
281+ testsuites[testname].append(only_tests[testname]['outcome'])
282+ except KeyError:
283+ logger.error(
284+ 'Testsuite has not outcome. Document ID: ',
285+ upload_id
286+ )
287+
288+ results = []
289+ for suite in testsuites:
290+ runs = len(testsuites[suite])
291+ passes = len([t for t in testsuites[suite] if t == 'pass'])
292+ fails = runs - passes
293+ success_rate = format(passes / runs * 100, '.2f')
294+
295+ results.append(
296+ TestsuiteResult(suite, runs, passes, fails, success_rate)
297+ )
298+ return results
299+
300+
301+def view_latest_uploads():
302+ """Render report displaying a list of uploaded results, latest first."""
303+ results_collection = db.get_results_collection()
304+ # Sorted by latest first. . .
305+ all_uploads = results_collection.find().sort('uploaded', -1)
306+
307+ # slight shim over the returned data to prepare it for presention.
308+ sanitised_uploads = []
309+ for upload in all_uploads:
310+ upload_date = upload['uploaded'].strftime('%Y-%b-%d %H:%M:%S')
311+ uploader_email = upload.get('user_email') or 'Anonymous'
312+
313+ try:
314+ results = _result_dict_from_document(upload)
315+ except KeyError as e:
316+ logger.warning(e)
317+ continue
318+
319+ testsuite_details = _get_only_testcases(results)
320+ uploaded_testsuite_names = testsuite_details.keys()
321+
322+ sanitised_uploads.append(
323+ dict(
324+ upload_date=upload_date,
325+ testsuites=uploaded_testsuite_names,
326+ uploader_email=uploader_email
327+ )
328+ )
329+
330+ return render_template(
331+ 'results/latest_uploads.html',
332+ all_uploads=sanitised_uploads
333+ )
334+
335+
336+def _result_dict_from_document(upload):
337+ """Return a dict of result details.
338+
339+ Parses the json result string and constructs a dict containing the details.
340+
341+ :param upload: dict containing upload details. Must contain the key
342+ 'results' to be successful.
343+ :raises KeyError: if no 'results' are found in the upload dict.
344+ """
345+ try:
346+ results_json = upload['results']
347+ return json.loads(results_json.decode('utf-8'))
348+ except KeyError:
349+ upload_id = str(upload.get('_id'), 'No ID.')
350+ logger.warning(
351+ 'Skipping document. No results or result_map available',
352+ upload_id,
353+ )
354+
355+
356+def _get_only_testcases(upload_details):
357+ """Return details only for testsuites.
358+
359+ Uploaded results can contain detials that aren't testsuite related
360+ (i.e. system details tests were run on.) Extract only the details that are
361+ for testsuitse.
362+
363+ :param uploaded_details: dict containing keys 'resource_map' and
364+ 'result_map'.
365+ """
366+ removal_keys = upload_details['resource_map'].keys()
367+
368+ return {k: v for k, v in upload_details['result_map'].items()
369+ if k not in removal_keys}
370
371=== added directory 'ubuntu_pt_community/templates'
372=== added file 'ubuntu_pt_community/templates/base.html'
373--- ubuntu_pt_community/templates/base.html 1970-01-01 00:00:00 +0000
374+++ ubuntu_pt_community/templates/base.html 2015-09-17 05:33:50 +0000
375@@ -0,0 +1,30 @@
376+<!doctype html>
377+<html lang="en">
378+ <head>
379+ <meta charset="utf-8">
380+ <meta http-equiv="X-UA-Compatible" content="IE=edge">
381+ <meta name="viewport" content="width=device-width, initial-scale=1">
382+
383+ <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
384+
385+ {% block head %}
386+ <title>{% block title %}{% endblock %}</title>
387+ {% endblock %}
388+ </head>
389+ <body>
390+
391+ <div id="content" class="container">
392+ {% block content %}{% endblock %}
393+ </div>
394+
395+ <footer class="footer">
396+ <div class="container">
397+ <p class="text-muted">
398+ {% block footer %}
399+ &copy; Copyright 2015 <a href="https://canonical.com">Canonical Ltd.</a>
400+ {% endblock %}
401+ </p>
402+ </div>
403+ </footer>
404+ </body>
405+</html>
406
407=== added directory 'ubuntu_pt_community/templates/results'
408=== added file 'ubuntu_pt_community/templates/results/all_testsuites.html'
409--- ubuntu_pt_community/templates/results/all_testsuites.html 1970-01-01 00:00:00 +0000
410+++ ubuntu_pt_community/templates/results/all_testsuites.html 2015-09-17 05:33:50 +0000
411@@ -0,0 +1,23 @@
412+{% extends "base.html" %}
413+{% block title %}All Result Details{% endblock %}
414+
415+{% block content %}
416+<h1 class="page-header">All testsuite statistics</h1>
417+
418+<!-- Could probably also add a list of users/email address that have done this report. -->
419+<table class="table table-striped table-bordered">
420+ <tr>
421+ <th>Testsuite</th>
422+ <th>Total Runs</th>
423+ <th>Success Rate</th>
424+ </tr>
425+ {% for report in results %}
426+ <tr>
427+ <td>{{report.name}}</td>
428+ <td>{{report.runs}}</td>
429+ <td>{{report.success_rate}}%</td>
430+ </tr>
431+ {% endfor %}
432+</table>
433+
434+{% endblock %}
435
436=== added file 'ubuntu_pt_community/templates/results/index.html'
437--- ubuntu_pt_community/templates/results/index.html 1970-01-01 00:00:00 +0000
438+++ ubuntu_pt_community/templates/results/index.html 2015-09-17 05:33:50 +0000
439@@ -0,0 +1,10 @@
440+{% extends "base.html" %}
441+{% block title %}Community Testing Reports{% endblock %}
442+{% block content %}
443+<h1 class="page-header">List of available reports</h1>
444+<ul>
445+ {% for report in reports %}
446+ <li><a href="{{ url_for(report.route_function_name) }}">{{ report.name }}</a></li>
447+ {% endfor %}
448+</ul>
449+{% endblock %}
450
451=== added file 'ubuntu_pt_community/templates/results/latest_uploads.html'
452--- ubuntu_pt_community/templates/results/latest_uploads.html 1970-01-01 00:00:00 +0000
453+++ ubuntu_pt_community/templates/results/latest_uploads.html 2015-09-17 05:33:50 +0000
454@@ -0,0 +1,27 @@
455+{% extends "base.html" %}
456+{% block title %}Latest Uploads{% endblock %}
457+
458+{% block content %}
459+<h1 class="page-header">Latest uploads</h1>
460+
461+<table class="table table-striped table-bordered">
462+ <tr>
463+ <th>Date Uploaded</th>
464+ <th>Testsuites run</th>
465+ <th>Uploader</th>
466+ </tr>
467+ {% for upload in all_uploads %}
468+ <tr>
469+ <td>{{ upload.upload_date }}</td>
470+ <td>
471+ <ul>
472+ {% for testsuite in upload.testsuites %}
473+ <li>{{ testsuite }}</li>
474+ {% endfor %}
475+ </ul>
476+ </td>
477+ <td>{{ upload.uploader_email }}</td>
478+ </tr>
479+ {% endfor%}
480+</table>
481+{% endblock %}
482
483=== added directory 'ubuntu_pt_community/tests'
484=== added file 'ubuntu_pt_community/tests/__init__.py'
485--- ubuntu_pt_community/tests/__init__.py 1970-01-01 00:00:00 +0000
486+++ ubuntu_pt_community/tests/__init__.py 2015-09-17 05:33:50 +0000
487@@ -0,0 +1,17 @@
488+#
489+# Ubuntu PractiTest Community results processor
490+# Copyright (C) 2015 Canonical
491+#
492+# This program is free software: you can redistribute it and/or modify
493+# it under the terms of the GNU General Public License as published by
494+# the Free Software Foundation, either version 3 of the License, or
495+# (at your option) any later version.
496+#
497+# This program is distributed in the hope that it will be useful,
498+# but WITHOUT ANY WARRANTY; without even the implied warranty of
499+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
500+# GNU General Public License for more details.
501+#
502+# You should have received a copy of the GNU General Public License
503+# along with this program. If not, see <http://www.gnu.org/licenses/>.
504+#
505
506=== added file 'ubuntu_pt_community/tests/test_reports.py'
507--- ubuntu_pt_community/tests/test_reports.py 1970-01-01 00:00:00 +0000
508+++ ubuntu_pt_community/tests/test_reports.py 2015-09-17 05:33:50 +0000
509@@ -0,0 +1,37 @@
510+#
511+# Ubuntu PractiTest Community results processor
512+# Copyright (C) 2015 Canonical
513+#
514+# This program is free software: you can redistribute it and/or modify
515+# it under the terms of the GNU General Public License as published by
516+# the Free Software Foundation, either version 3 of the License, or
517+# (at your option) any later version.
518+#
519+# This program is distributed in the hope that it will be useful,
520+# but WITHOUT ANY WARRANTY; without even the implied warranty of
521+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
522+# GNU General Public License for more details.
523+#
524+# You should have received a copy of the GNU General Public License
525+# along with this program. If not, see <http://www.gnu.org/licenses/>.
526+#
527+
528+import testtools
529+
530+from ubuntu_pt_community.pages import reports
531+
532+
533+class ReportHelpersTestCase(testtools.TestCase):
534+
535+ def test_get_only_testcases_returns_only_testcases(self):
536+ testdict = dict(
537+ result_map=dict(
538+ testcase1=True,
539+ testcase2=True,
540+ resourcedetails1=False,
541+ ),
542+ resource_map=dict(resourcedetails1=False)
543+ )
544+
545+ results = reports._get_only_testcases(testdict)
546+ self.assertDictEqual(results, dict(testcase1=True, testcase2=True))

Subscribers

People subscribed via source and target branches