Merge lp:~canonical-platform-qa/ubuntu-community-testing/initial-reporting into lp:ubuntu-community-testing

[Christopher Lee]

Initial lot of reporting.

Initial reporting incl.
  - "Latest Uploads" - list of upload date with testsuites run and the users email
  - "All testsuites" - simple stats for each testsuite that we have upload details for (runs, pass/fail numbers, success rates).

These are the first run and we can build on it from here (incl. making it look better :-P as well as navigation etc.).

[Christopher Lee (veebers)]

Just marked as WIP as I notice that somethings are incorrect. Adding tests and sorting out these issues.

[Nicholas Skaggs (nskaggs)]

This looks good from my perspective. Chris, it looks like only the results from today will display in the 'latest results?'. I didn't actually build and deploy this, so I might be mistaken :-)

If so however, I might see this as a problem when days rollover. When the day switches to a new day, I might not see my results, even though I submitted them say within the last hour. Could we just simply display the last X number of results? Or if you wish limit it to a more reasonable number of days, perhaps 3 instead of just 1. If you do use days, it might still be wise to limit the number of results displayed so we don't have a long list on the page.

[Christopher Lee (veebers)]

@balloons at the moment there is no limit to how many it displays and just shows all uploads with the latest at the top.

We can iterate on doing smarter things with this (i.e. pagination (using JIT requests), just show X days etc.)

[Nicholas Skaggs (nskaggs)]

Ack, sounds fine to me.

review: Approve

[Brendan Donegan (brendan-donegan)]

Mostly looks great - some questions about use of docstrings though

review: Needs Information

[lp:~canonical-platform-qa/ubuntu-community-testing/initial-reporting]

41. By Christopher Lee on 2015-09-17

Better and additional docstrings.

42. By Christopher Lee on 2015-09-17

Fix flake8 errors.

[Christopher Lee (veebers)]

> Mostly looks great - some questions about use of docstrings though
Good catch with the docstrings. Have added more and improved the existing.

Regarding docstrings for 'private' methods; I see no issue with having them, They are a well known format which can streamline a developer reading them and contain details regarding the intention of the method itself.

[Brendan Donegan (brendan-donegan)]

Ok looks better now. +1

review: Approve

=== modified file 'ubuntu_pt_community/__init__.py'
--- ubuntu_pt_community/__init__.py	2015-08-18 00:55:26 +0000
+++ ubuntu_pt_community/__init__.py	2015-09-17 05:33:50 +0000
@@ -18,7 +18,10 @@
 import logging
 import sys
 
-from ubuntu_pt_community import api
+from ubuntu_pt_community import (
+    api,
+    pages,
+)
 
 from flask import Flask
 
@@ -40,3 +43,4 @@
 app = Flask(__name__)
 
 api.define_api_routes(app)
+pages.define_page_routes(app)

=== modified file 'ubuntu_pt_community/api/v1.py'
--- ubuntu_pt_community/api/v1.py	2015-08-18 10:58:22 +0000
+++ ubuntu_pt_community/api/v1.py	2015-09-17 05:33:50 +0000
@@ -28,7 +28,9 @@
 from qakit.practitest.report_checkbox_results_to_practitest import (
     upload_results
 )
-from ubuntu_pt_community import auth, db
+from ubuntu_pt_community import auth
+from ubuntu_pt_community.db import get_results_collection
+
 
 logger = logging.getLogger(__name__)
 
@@ -86,7 +88,7 @@
     )
 
     try:
-        collection = get_results_database()
+        collection = get_results_collection()
         insert_id = collection.insert(details)
         logger.info(
             'Inserted details for {} with id {}'.format(
@@ -103,12 +105,6 @@
         logger.error('Failed to get database connection: ', e)
 
 
-def get_results_database():
-    database_name = 'community_practitest'
-    collection_name = 'uploaded_results'
-    return db.get_collection(database_name, collection_name)
-
-
 def get_user_email_address(request):
     try:
         return request.form['uploader_email']

=== modified file 'ubuntu_pt_community/db/__init__.py'
--- ubuntu_pt_community/db/__init__.py	2015-08-13 05:52:38 +0000
+++ ubuntu_pt_community/db/__init__.py	2015-09-17 05:33:50 +0000
@@ -16,6 +16,9 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-from ubuntu_pt_community.db.db import get_collection
+from ubuntu_pt_community.db.db import (
+    get_collection,
+    get_results_collection,
+)
 
-__all__ = ['get_collection']
+__all__ = ['get_collection', 'get_results_collection']

=== modified file 'ubuntu_pt_community/db/db.py'
--- ubuntu_pt_community/db/db.py	2015-08-17 06:19:29 +0000
+++ ubuntu_pt_community/db/db.py	2015-09-17 05:33:50 +0000
@@ -48,6 +48,12 @@
     return db[collection_name]
 
 
+def get_results_collection():
+    database_name = 'community_practitest'
+    collection_name = 'uploaded_results'
+    return get_collection(database_name, collection_name)
+
+
 def get_config_file_path():
     try:
         return os.path.join(

=== added directory 'ubuntu_pt_community/pages'
=== added file 'ubuntu_pt_community/pages/__init__.py'
--- ubuntu_pt_community/pages/__init__.py	1970-01-01 00:00:00 +0000
+++ ubuntu_pt_community/pages/__init__.py	2015-09-17 05:33:50 +0000
@@ -0,0 +1,27 @@
+#
+# Ubuntu PractiTest Community results processor
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+from ubuntu_pt_community.pages import pages
+
+"""Prepare the available api routes."""
+
+__all__ = ['define_page_routes']
+
+
+def define_page_routes(webapp):
+    pages.define_routes(webapp)

=== added file 'ubuntu_pt_community/pages/pages.py'
--- ubuntu_pt_community/pages/pages.py	1970-01-01 00:00:00 +0000
+++ ubuntu_pt_community/pages/pages.py	2015-09-17 05:33:50 +0000
@@ -0,0 +1,69 @@
+#
+# Ubuntu PractiTest Community results processor
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import logging
+from flask import render_template
+
+from ubuntu_pt_community.pages import reports
+
+
+logger = logging.getLogger(__name__)
+
+
+class PageDefinition:
+
+    def __init__(self, name, route_name, route_func):
+        """Encapsulate the report view details."""
+        self.name = name
+        self.route_name = route_name
+        self.route_func = route_func
+
+    @property
+    def route_function_name(self):
+        """Return the name of the route function for use_url."""
+        return self.route_func.__name__
+
+
+# Keep a list of report pages so we don't have to duplicate things when listing
+# them and adding routes.
+Report_Pages = [
+    PageDefinition(
+        'All Results',
+        '/reports/all_results',
+        reports.view_all_results
+    ),
+
+    PageDefinition(
+        'Latest Uploads',
+        '/reports/latest',
+        reports.view_latest_uploads
+    ),
+]
+
+
+def define_routes(webapp):
+    """Setup all routes for available reports (incl. list of reports."""
+    webapp.add_url_rule('/reports', view_func=view_reports)
+
+    for page in Report_Pages:
+        webapp.add_url_rule(page.route_name, view_func=page.route_func)
+
+
+def view_reports():
+    """Render a simple list of links to available reports."""
+    return render_template('results/index.html', reports=Report_Pages)

=== added file 'ubuntu_pt_community/pages/reports.py'
--- ubuntu_pt_community/pages/reports.py	1970-01-01 00:00:00 +0000
+++ ubuntu_pt_community/pages/reports.py	2015-09-17 05:33:50 +0000
@@ -0,0 +1,167 @@
+#
+# Ubuntu PractiTest Community results processor
+
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import json
+import logging
+from collections import (
+    defaultdict,
+    namedtuple,
+)
+from flask import render_template
+
+from ubuntu_pt_community import db
+
+logger = logging.getLogger(__name__)
+
+
+def view_all_results():
+    """Render report displaying all uploaded testsuites with stats.
+
+    Stats displayed are number of runs, number of passes and fails and the % of
+    successful runs.
+    """
+    results_collection = db.get_results_collection()
+    all_uploads = results_collection.find()
+    simple_report_data = _get_simple_report_data(all_uploads)
+    return render_template(
+        'results/all_testsuites.html',
+        results=simple_report_data
+    )
+
+
+def _get_simple_report_data(all_uploads):
+    """Produce a list of suite results.
+
+    :param all_uploads: list of dicts containing uploaded data details.
+    :returns: a list of TestsuiteResult objects.
+    """
+    TestsuiteResult = namedtuple(
+        'TestsuiteResult',
+        ['name', 'runs', 'passes', 'fails', 'success_rate']
+    )
+
+    # all_uploads will be a list of dicts
+    # dict will have keys:
+    #   - results: a json string with the result details.
+    #   - user_email: email of the user who uploaded the results
+    #   - uploaded: a date of uploading
+    testsuites = defaultdict(list)
+    for upload in all_uploads:
+        upload_id = str(upload['_id'])
+
+        try:
+            results = _result_dict_from_document(upload)
+        except KeyError as e:
+            logger.warning(e)
+            continue
+
+        only_tests = _get_only_testcases(results)
+
+        for testname in only_tests:
+            # outcome will be 'pass' or 'fail'
+            try:
+                testsuites[testname].append(only_tests[testname]['outcome'])
+            except KeyError:
+                logger.error(
+                    'Testsuite has not outcome. Document ID: ',
+                    upload_id
+                )
+
+    results = []
+    for suite in testsuites:
+        runs = len(testsuites[suite])
+        passes = len([t for t in testsuites[suite] if t == 'pass'])
+        fails = runs - passes
+        success_rate = format(passes / runs * 100, '.2f')
+
+        results.append(
+            TestsuiteResult(suite, runs, passes, fails, success_rate)
+        )
+    return results
+
+
+def view_latest_uploads():
+    """Render report displaying a list of uploaded results, latest first."""
+    results_collection = db.get_results_collection()
+    # Sorted by latest first. . .
+    all_uploads = results_collection.find().sort('uploaded', -1)
+
+    # slight shim over the returned data to prepare it for presention.
+    sanitised_uploads = []
+    for upload in all_uploads:
+        upload_date = upload['uploaded'].strftime('%Y-%b-%d %H:%M:%S')
+        uploader_email = upload.get('user_email') or 'Anonymous'
+
+        try:
+            results = _result_dict_from_document(upload)
+        except KeyError as e:
+            logger.warning(e)
+            continue
+
+        testsuite_details = _get_only_testcases(results)
+        uploaded_testsuite_names = testsuite_details.keys()
+
+        sanitised_uploads.append(
+            dict(
+                upload_date=upload_date,
+                testsuites=uploaded_testsuite_names,
+                uploader_email=uploader_email
+            )
+        )
+
+    return render_template(
+        'results/latest_uploads.html',
+        all_uploads=sanitised_uploads
+    )
+
+
+def _result_dict_from_document(upload):
+    """Return a dict of result details.
+
+    Parses the json result string and constructs a dict containing the details.
+
+    :param upload: dict containing upload details. Must contain the key
+      'results' to be successful.
+    :raises KeyError: if no 'results' are found in the upload dict.
+    """
+    try:
+        results_json = upload['results']
+        return json.loads(results_json.decode('utf-8'))
+    except KeyError:
+        upload_id = str(upload.get('_id'), 'No ID.')
+        logger.warning(
+            'Skipping document. No results or result_map available',
+            upload_id,
+        )
+
+
+def _get_only_testcases(upload_details):
+    """Return details only for testsuites.
+
+    Uploaded results can contain detials that aren't testsuite related
+    (i.e. system details tests were run on.) Extract only the details that are
+    for testsuitse.
+
+    :param uploaded_details: dict containing keys 'resource_map' and
+    'result_map'.
+    """
+    removal_keys = upload_details['resource_map'].keys()
+
+    return {k: v for k, v in upload_details['result_map'].items()
+            if k not in removal_keys}

=== added directory 'ubuntu_pt_community/templates'
=== added file 'ubuntu_pt_community/templates/base.html'
--- ubuntu_pt_community/templates/base.html	1970-01-01 00:00:00 +0000
+++ ubuntu_pt_community/templates/base.html	2015-09-17 05:33:50 +0000
@@ -0,0 +1,30 @@
+<!doctype html>
+<html lang="en">
+  <head>
+    <meta charset="utf-8">
+    <meta http-equiv="X-UA-Compatible" content="IE=edge">
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+
+    <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.5/css/bootstrap.min.css">
+
+    {% block head %}
+    <title>{% block title %}{% endblock %}</title>
+    {% endblock %}
+  </head>
+  <body>
+
+    <div id="content" class="container">
+      {% block content %}{% endblock %}
+    </div>
+
+    <footer class="footer">
+      <div class="container">
+        <p class="text-muted">
+          {% block footer %}
+          &copy; Copyright 2015 <a href="https://canonical.com">Canonical Ltd.</a>
+          {% endblock %}
+        </p>
+      </div>
+    </footer>
+  </body>
+</html>

=== added directory 'ubuntu_pt_community/templates/results'
=== added file 'ubuntu_pt_community/templates/results/all_testsuites.html'
--- ubuntu_pt_community/templates/results/all_testsuites.html	1970-01-01 00:00:00 +0000
+++ ubuntu_pt_community/templates/results/all_testsuites.html	2015-09-17 05:33:50 +0000
@@ -0,0 +1,23 @@
+{% extends "base.html" %}
+{% block title %}All Result Details{% endblock %}
+
+{% block content %}
+<h1 class="page-header">All testsuite statistics</h1>
+
+<!-- Could probably also add a list of users/email address that have done this report. -->
+<table class="table table-striped table-bordered">
+  <tr>
+    <th>Testsuite</th>
+    <th>Total Runs</th>
+    <th>Success Rate</th>
+  </tr>
+  {% for report in results %}
+  <tr>
+    <td>{{report.name}}</td>
+    <td>{{report.runs}}</td>
+    <td>{{report.success_rate}}%</td>
+  </tr>
+  {% endfor %}
+</table>
+
+{% endblock %}

=== added file 'ubuntu_pt_community/templates/results/index.html'
--- ubuntu_pt_community/templates/results/index.html	1970-01-01 00:00:00 +0000
+++ ubuntu_pt_community/templates/results/index.html	2015-09-17 05:33:50 +0000
@@ -0,0 +1,10 @@
+{% extends "base.html" %}
+{% block title %}Community Testing Reports{% endblock %}
+{% block content %}
+<h1 class="page-header">List of available reports</h1>
+<ul>
+  {% for report in reports %}
+  <li><a href="{{ url_for(report.route_function_name) }}">{{ report.name }}</a></li>
+  {% endfor %}
+</ul>
+{% endblock %}

=== added file 'ubuntu_pt_community/templates/results/latest_uploads.html'
--- ubuntu_pt_community/templates/results/latest_uploads.html	1970-01-01 00:00:00 +0000
+++ ubuntu_pt_community/templates/results/latest_uploads.html	2015-09-17 05:33:50 +0000
@@ -0,0 +1,27 @@
+{% extends "base.html" %}
+{% block title %}Latest Uploads{% endblock %}
+
+{% block content %}
+<h1 class="page-header">Latest uploads</h1>
+
+<table class="table table-striped table-bordered">
+  <tr>
+    <th>Date Uploaded</th>
+    <th>Testsuites run</th>
+    <th>Uploader</th>
+  </tr>
+  {% for upload in all_uploads %}
+  <tr>
+    <td>{{ upload.upload_date }}</td>
+    <td>
+      <ul>
+        {% for testsuite in upload.testsuites %}
+        <li>{{ testsuite }}</li>
+        {% endfor %}
+      </ul>
+    </td>
+    <td>{{ upload.uploader_email }}</td>
+  </tr>
+  {% endfor%}
+</table>
+{% endblock %}

=== added directory 'ubuntu_pt_community/tests'
=== added file 'ubuntu_pt_community/tests/__init__.py'
--- ubuntu_pt_community/tests/__init__.py	1970-01-01 00:00:00 +0000
+++ ubuntu_pt_community/tests/__init__.py	2015-09-17 05:33:50 +0000
@@ -0,0 +1,17 @@
+#
+# Ubuntu PractiTest Community results processor
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#

=== added file 'ubuntu_pt_community/tests/test_reports.py'
--- ubuntu_pt_community/tests/test_reports.py	1970-01-01 00:00:00 +0000
+++ ubuntu_pt_community/tests/test_reports.py	2015-09-17 05:33:50 +0000
@@ -0,0 +1,37 @@
+#
+# Ubuntu PractiTest Community results processor
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import testtools
+
+from ubuntu_pt_community.pages import reports
+
+
+class ReportHelpersTestCase(testtools.TestCase):
+
+    def test_get_only_testcases_returns_only_testcases(self):
+        testdict = dict(
+            result_map=dict(
+                testcase1=True,
+                testcase2=True,
+                resourcedetails1=False,
+            ),
+            resource_map=dict(resourcedetails1=False)
+        )
+
+        results = reports._get_only_testcases(testdict)
+        self.assertDictEqual(results, dict(testcase1=True, testcase2=True))