Merge lp:~canonical-platform-qa/qakit/app_startup_poc into lp:qakit

[Sergio Cazzolato]

Process to parse logs, upload data to mongo db and plot the app startup times for the last builds.

Dependencies:
Install pip by doing: "sudo apt-get install python-pip python-dev build-essential"

Download and Install eve (http://python-eve.org/install.html)
Run eve by doing python qakit/appstartup/eve/run.py (you can define the env variables, see in file settings.py)

Download and install mongodb (https://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/)
Create a user for mongo and remember to update the eve settings file
To create the mongo user type: "mongo" and then
db.createUser(
   {
     user: "ubuntuUser",
     pwd: "ubuntuPassword",
     roles: [ { role: "readWrite", db: "test" } ]
   }
)
Note: eve code is ready to run with python 2.7 and 3.X

Network configuration when the eve machine is hosted in canonistack:

  1- Allow ip routing by doing:
  sysctl -w net.ipv4.ip_forward=1

  2- Make sure the os allow to route to localnet by doing:
  sudo sysctl -w net.ipv4.conf.eth0.route_localnet=1

  3- configure iptables by doing:
  sudo iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 5000 -j DNAT
  --to-destination 127.0.0.1:5000

To run the import and plotting process it is needed to run the following tests with the command "./run-system-tests ubuntu_system_tests.tests.app_startup.test_application_startup" for the branch lp:~canonical-platform-qa/ubuntu-system-tests/app_startup_poc.

Then, having a test-results.subunit file, run as following: python3 qakit/appstartup/orchestrator.py ~/Desktop/app_startup/test-results.subunit

The html report will be generated in ~/Desktop/app_startup/app_startup_report.html

The eve service is currently available on:

  http://10.55.61.109:5000/appstartup

Jenkins job available on:

  https://platform-qa-jenkins.ubuntu.com/job/cachio-appstarup

[lp:~canonical-platform-qa/qakit/app_startup_poc]

48. By Sergio Cazzolato on 2016-01-22

Merge with trunk and conflicts resolved

[Richard Huddie (rhuddie)]

Could you add a README with all the setup steps listed similar to above description? Or update existing one?

I found it a little confusing as the existing qakit README mentions only python 2 support:
http://bazaar.launchpad.net/~canonical-platform-qa/qakit/trunk/view/head:/README

review: Needs Fixing

[lp:~canonical-platform-qa/qakit/app_startup_poc]

49. By Sergio Cazzolato on 2016-01-25

Supporting the run type

50. By Sergio Cazzolato on 2016-01-26

Adding env variable for qakit config file

51. By Sergio Cazzolato on 2016-01-26

Making configurable the ip and port used by eve

52. By Sergio Cazzolato on 2016-01-28

Change to access from external ip to the local service

53. By Sergio Cazzolato on 2016-01-28

Adding missing file to create a html report

54. By Sergio Cazzolato on 2016-01-28

Making url path configurable

55. By Sergio Cazzolato on 2016-01-29

Changing graphics colors

56. By Sergio Cazzolato on 2016-02-01

Adding some styles to the report

57. By Sergio Cazzolato on 2016-02-01

Minor changes in the report

58. By Sergio Cazzolato on 2016-02-10

Adding json report

59. By Sergio Cazzolato on 2016-02-11

Improving how reports are managed

60. By Sergio Cazzolato on 2016-02-12

Adding docs to the README

61. By Sergio Cazzolato on 2016-02-12

Adding missing docs to the README

62. By Sergio Cazzolato on 2016-02-12

Adding configs for reports and adding more info json report

63. By Sergio Cazzolato on 2016-02-12

updating json report

[Allan LeSage (allanlesage)]

I have yet to get this working all the way but some comments to deal with meanwhile:

When you're using requests.get or .post, just pass the data as a named keyword argument *params* with a dict of data, it's much clearer than formatting the URL yourself IMO.

For the HTML rendering, it'd be cleaner to use a jinja2 template, instead of concatenating the HTML together piecemeal. Happy to help get that going, or you could just drag the metrics stuff into a shared location? Would you have a look at qakit/metrics/reports/__init__.py:render_html ?

What does dao mean?

More comments tomorrow, just wanted to get started ;) .

review: Needs Fixing

[Sergio Cazzolato (sergio-j-cazzolato)]

Going to use params to format the url.

The html report is going to disappear once we have it in trunk as we are going to use the dashboard. SO I don't see the need to change this implementation, don't you?

[lp:~canonical-platform-qa/qakit/app_startup_poc]

64. By Sergio Cazzolato on 2016-02-18

Adding paramter to filter number of builds to consider and using params to format the urls

[Allan LeSage (allanlesage)]

Thanks for that change, propose to merge this--it's just a proof of concept after all ;) --and it doesn't affect any existing code.

review: Approve

=== modified file 'README'
--- README	2014-10-08 01:40:03 +0000
+++ README	2016-02-18 14:19:39 +0000
@@ -48,3 +48,51 @@
 	| ubuntu-rtm/landing-007 - qtbase-opensource-src,qtbase-opensource-src-gles : Mirv                                                                        | Need QA Sign-off |
 	| ubuntu-rtm/landing-009 - ubuntu-touch-session : tedg                                                                                                    | Need QA Sign-off |
 	+---------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+
+
+How do I process appstartup logs
+================================
+
+Environment configuration
+-------------------------
+
+Install pip by doing: "sudo apt-get install python-pip python-dev build-essential"
+
+Download and Install eve (http://python-eve.org/install.html)
+Run eve by doing python qakit/appstartup/eve/run.py (you can define the env variables, see in file settings.py)
+
+Download and install mongodb (https://docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/)
+Create a user for mongo and remember to update the eve settings file
+To create the mongo user type: "mongo" and then
+db.createUser(
+   {
+     user: "ubuntuUser",
+     pwd: "ubuntuPassword",
+     roles: [ { role: "readWrite", db: "test" } ]
+   }
+)
+Note: eve code is ready to run with python 2.7 and 3.X
+
+In case it is needed to open ports where the eve machine is hosted:
+
+  1- Allow ip routing by doing:
+  sysctl -w net.ipv4.ip_forward=1
+
+  2- Make sure the os allow to route to localnet by doing:
+  sudo sysctl -w net.ipv4.conf.eth0.route_localnet=1
+
+  3- configure iptables by doing:
+  sudo iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 5000 -j DNAT
+  --to-destination 127.0.0.1:5000
+
+Add to the config the following:
+    [APP_STARTUP]
+    EVE_HOST = [dest_ip]
+    EVE_PORT = [dest_port]
+    EVE_PATH = appstartup
+    HTML_REPORT = True
+    JSON_REPORT = True
+    BUILDS_TO_PLOT = 10
+
+Having a test-results.subunit file, run as following: python3 qakit/appstartup/orchestrator.py pathto/test-results.subunit
+
+HTML and json reports are generated in the subunit directory. All the pictures will be stored in the appstartup_data dir.

=== modified file 'debian/control'
--- debian/control	2016-01-12 12:11:16 +0000
+++ debian/control	2016-02-18 14:19:39 +0000
@@ -24,6 +24,8 @@
 	 python3-bson,
 	 python3-dateutil,
 	 python3-jinja2,
+	 python3-matplotlib,
+	 python3-numpy,
 	 python3-pymongo,
 	 python3-requests,
 	 python3-subunit,

=== added directory 'qakit/appstartup'
=== added file 'qakit/appstartup/__init__.py'
--- qakit/appstartup/__init__.py	1970-01-01 00:00:00 +0000
+++ qakit/appstartup/__init__.py	2016-02-18 14:19:39 +0000
@@ -0,0 +1,33 @@
+#!/usr/bin/python3
+# UESQA Metrics
+# Copyright (C) 2016 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import configparser
+
+import qakit.config as qakit_config
+
+allowed_apps = ['webbrowser', 'address_book', 'calculator', 'here', 'dialer',
+                'clock', 'messaging', 'ebay', 'camera', 'system_settings',
+                'music', 'gallery']
+
+
+def _read_config(config_filepath):
+    """ Parse the config at the given filepath and return it """
+    config_file = configparser.ConfigParser()
+    config_file.read(config_filepath)
+    return config_file
+
+config_dict = _read_config(qakit_config.get_config_file_location())

=== added file 'qakit/appstartup/dao.py'
--- qakit/appstartup/dao.py	1970-01-01 00:00:00 +0000
+++ qakit/appstartup/dao.py	2016-02-18 14:19:39 +0000
@@ -0,0 +1,102 @@
+#!/usr/bin/python3
+# UESQA Metrics
+# Copyright (C) 2016 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import logging
+import requests
+
+from qakit.appstartup import config_dict as config
+
+BASE_URL = 'http://{}:{}/{}'.format(config['APP_STARTUP']['EVE_HOST'],
+                                    config['APP_STARTUP']['EVE_PORT'],
+                                    config['APP_STARTUP']['EVE_PATH'])
+
+logger = logging.getLogger(__name__)
+
+
+def get_headers():
+    return {'Content-Type': 'application/json'}
+
+
+def _get_where_param(**params):
+    """ Get a where parameter used to filter when getting a query to mongo
+    :param params: A dict with all the parameters used to filter
+    """
+    where_param = "{"
+    for (k, v) in params.items():
+        where_param += '"{}":"{}",'.format(k, v)
+    return where_param[:-1] + '}'
+
+
+def save_record_to_db(global_info, startup_times):
+    """ Save the record information in the mongo db. The information saved is
+    composed by joining both dicts received as parameter
+    """
+    logger.info('Saving records to db')
+    for instance in startup_times:
+        instance.update(global_info)
+        r = requests.post('{}'.format(BASE_URL),
+                          json=instance,
+                          headers=get_headers())
+        if not r.status_code == requests.codes.created:
+            logger.error('Error uploading instance data: {}'.format(instance))
+            logger.error(r.status_code, r.reason)
+
+
+def is_run_saved(run_id):
+    """ Indicate if there is at least one record with the run_id saved in the
+    mongo db
+    """
+    where_param = _get_where_param(run_id=run_id)
+    r = requests.get(BASE_URL, params={'where': where_param},
+                     headers=get_headers())
+    if r.status_code == requests.codes.ok:
+        return r.json()['_meta']['total'] > 0
+    else:
+        logger.error('Error retrieving information for run: {}'.format(run_id))
+        logger.error(r.status_code, r.reason)
+
+
+def get_records_for_app(app, device, channel):
+    """ Retrieve the records from the mongo db for an specific app, device and
+    channel. The records by default are ordered by build number
+    :param app: The app used to filter the results
+    :param device: The device used to filter the results
+    :param channel: The channel used to filter the results
+    """
+    where_param = _get_where_param(app=app, device=device, channel=channel)
+
+    stop = False
+    page_iter = 1
+    records = []
+    while not stop:
+        r = requests.get(BASE_URL,
+                         params={'where': where_param, 'max_results': 50,
+                                 'page': page_iter},
+                         headers=get_headers())
+        if r.status_code == requests.codes.ok:
+            records.extend(r.json()['_items'])
+        else:
+            logger.error('Error retrieving data from url: {}'.format(r.url))
+            logger.error(r.status_code, r.reason)
+
+        metadata = r.json()['_meta']
+        if metadata['max_results'] * metadata['page'] >= metadata['total']:
+            stop = True
+        else:
+            page_iter += 1
+
+    return records

=== added file 'qakit/appstartup/data_processor.py'
--- qakit/appstartup/data_processor.py	1970-01-01 00:00:00 +0000
+++ qakit/appstartup/data_processor.py	2016-02-18 14:19:39 +0000
@@ -0,0 +1,74 @@
+#!/usr/bin/python3
+# UESQA Metrics
+# Copyright (C) 2016 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import statistics
+
+from qakit.appstartup import config_dict as config
+
+BUILDS_TO_PLOT = config['APP_STARTUP']['BUILDS_TO_PLOT']
+
+
+def get_times_by_build(app_records):
+    """ Calculate the times for the different builds
+    :return: [builds], [mean cold], [stdev cold], [mean hot], [stdev hot]
+    """
+    build_times = {}
+    for record in app_records:
+        build = record['build_number']
+        if build in build_times:
+            build_times[build].append(record)
+        else:
+            build_times[build] = [record]
+
+    times = _process_build_times(build_times)
+    if times:
+        return zip(*times)
+    else:
+        return (), (), (), (), ()
+
+
+def _process_build_times(build_times):
+    """ Process and filter build times """
+    times = []
+    for build in sorted(build_times)[- int(BUILDS_TO_PLOT):]:
+        hot_times = _get_times_by_type(build_times[build], 'hot')
+        cold_times = _get_times_by_type(build_times[build], 'cold')
+
+        times.append((build, _get_average(cold_times),
+                      _get_stdev(cold_times),
+                      _get_average(hot_times),
+                      _get_stdev(hot_times)))
+    return times
+
+
+def _get_average(times):
+    if times:
+        return statistics.mean(times)
+    else:
+        return 0
+
+
+def _get_stdev(times):
+    if len(times) >= 2:
+        return statistics.stdev(times)
+    else:
+        return 0
+
+
+def _get_times_by_type(app_records, type):
+    return [app_record['time'] for app_record in app_records if
+            app_record['type'] == type]

=== added directory 'qakit/appstartup/eve'
=== added file 'qakit/appstartup/eve/__init__.py'
--- qakit/appstartup/eve/__init__.py	1970-01-01 00:00:00 +0000
+++ qakit/appstartup/eve/__init__.py	2016-02-18 14:19:39 +0000
@@ -0,0 +1,16 @@
+#!/usr/bin/python3
+# UESQA Metrics
+# Copyright (C) 2016 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.

=== added file 'qakit/appstartup/eve/run.py'
--- qakit/appstartup/eve/run.py	1970-01-01 00:00:00 +0000
+++ qakit/appstartup/eve/run.py	2016-02-18 14:19:39 +0000
@@ -0,0 +1,22 @@
+#!/usr/bin/python3
+# UESQA Metrics
+# Copyright (C) 2016 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from eve import Eve
+app = Eve()
+
+if __name__ == '__main__':
+    app.run()

=== added file 'qakit/appstartup/eve/settings.py'
--- qakit/appstartup/eve/settings.py	1970-01-01 00:00:00 +0000
+++ qakit/appstartup/eve/settings.py	2016-02-18 14:19:39 +0000
@@ -0,0 +1,89 @@
+#!/usr/bin/python3
+# UESQA Metrics
+# Copyright (C) 2016 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+
+MONGO_HOST = os.getenv('EVE_MONGO_HOST', '127.0.0.1')
+MONGO_PORT = os.getenv('EVE_MONGO_PORT', '27017')
+MONGO_USERNAME = os.getenv('EVE_MONGO_USERNAME', 'ubuntuUser')
+MONGO_PASSWORD = os.getenv('EVE_MONGO_PASSWORD', 'ubuntuPassword')
+MONGO_DBNAME = os.getenv('EVE_MONGO_DBNAME', 'test')
+SERVER_NAME = os.getenv('EVE_SERVER_NAME', None)
+
+
+appstartup_schema = {
+    # Schema definition, based on Cerberus grammar. Check the Cerberus project
+    # (https://github.com/nicolaiarocci/cerberus) for details.
+    'run_id': {
+        'type': 'string',
+        'required': True,
+    },
+    'time': {
+        'type': 'number',
+        'required': True,
+    },
+    'type': {
+        'type': 'string',
+        'required': True,
+    },
+    'iteration': {
+        'type': 'number',
+        'required': True,
+    },
+    'app': {
+        'type': 'string',
+        'required': False,
+    },
+    'build_number': {
+        'type': 'string',
+        'required': True,
+    },
+    'device': {
+        'type': 'string',
+        'required': True,
+    },
+    'channel': {
+        'type': 'string',
+        'required': True,
+    },
+}
+
+appstartup = {
+    # 'title' tag used in item links. Defaults to the resource title minus
+    # the final, plural 's' (works fine in most cases but not for 'people')
+    'item_title': 'appstartup',
+
+    # by default the standard item entry point is defined as
+    # '/people/<ObjectId>'. We leave it untouched, and we also enable an
+    # additional read-only entry point. This way consumers can also perform
+    # GET requests at '/people/<lastname>'.
+    'additional_lookup': {
+        'url': 'regex("[\w]+")',
+        'field': 'app'
+    },
+
+    # We choose to override global cache-control directives for this resource.
+    'cache_control': 'max-age=10,must-revalidate',
+    'cache_expires': 10,
+
+    # most global settings can be overridden at resource level
+    'schema': appstartup_schema,
+}
+
+DOMAIN = {'appstartup': appstartup}
+RESOURCE_METHODS = ['GET', 'POST', 'DELETE']
+ITEM_METHODS = ['GET', 'PATCH', 'PUT', 'DELETE']

=== added file 'qakit/appstartup/orchestrator.py'
--- qakit/appstartup/orchestrator.py	1970-01-01 00:00:00 +0000
+++ qakit/appstartup/orchestrator.py	2016-02-18 14:19:39 +0000
@@ -0,0 +1,99 @@
+#!/usr/bin/python3
+# UESQA Metrics
+# Copyright (C) 2016 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from argparse import ArgumentParser
+import logging
+import os
+
+from qakit.appstartup import allowed_apps
+from qakit.appstartup import config_dict as config
+from qakit.appstartup import dao
+from qakit.appstartup import data_processor
+from qakit.appstartup import parser
+from qakit.appstartup import plotter
+from qakit.appstartup import report
+from qakit.practitest import subunit_results
+
+logger = logging.getLogger(__name__)
+
+HTML_REPORT = config['APP_STARTUP']['HTML_REPORT']
+JSON_REPORT = config['APP_STARTUP']['JSON_REPORT']
+
+
+def save_results_if_needed(global_info, startup_times):
+    if startup_times:
+        run_id = global_info['run_id']
+        if not dao.is_run_saved(run_id):
+            dao.save_record_to_db(global_info, startup_times)
+        else:
+            logger.warning(
+                'Test results already in the db for the run_id {}'.format(
+                    run_id))
+
+
+def _collect_records_by_app(global_info):
+    app_records = {}
+    device = global_info['device']
+    channel = global_info['channel']
+    for app in allowed_apps:
+        app_records[app] = dao.get_records_for_app(app, device, channel)
+    return app_records
+
+
+def _plot_records_by_apps(records_by_apps, results_dir):
+    app_plots = {}
+
+    for app in allowed_apps:
+        (builds, cold_times, cold_stdevs, hot_times, hot_stdevs) = \
+            data_processor.get_times_by_build(records_by_apps[app])
+        app_plots[app] = plotter.plot_app_records(app, builds, cold_times,
+                                                  cold_stdevs, hot_times,
+                                                  hot_stdevs, results_dir)
+    return app_plots
+
+
+def _create_graphics_dir(results_dir):
+    dir = os.path.join(results_dir, 'appstartup_data')
+    if not os.path.exists(dir):
+        os.mkdir(dir)
+    return dir
+
+
+def generate_reports(subunit_log, global_info, records_by_apps):
+    if HTML_REPORT or JSON_REPORT:
+        graphics_dir = _create_graphics_dir(os.path.dirname(subunit_log))
+        pics = _plot_records_by_apps(records_by_apps, graphics_dir)
+        if HTML_REPORT:
+            report.create_html_report(pics, os.path.dirname(subunit_log),
+                                      report_info=global_info)
+        if JSON_REPORT:
+            report.create_json_report(pics, os.path.dirname(subunit_log),
+                                      report_info=global_info)
+
+
+def main():
+    args_parser = ArgumentParser("Parse app startup times from logs.")
+    args_parser.add_argument("subunit_log")
+    args = args_parser.parse_args()
+    results_list = subunit_results.parse(args.subunit_log)
+    global_info, startup_times = parser.parse_startup_times(results_list)
+    save_results_if_needed(global_info, startup_times)
+    records_by_apps = _collect_records_by_app(global_info)
+    generate_reports(args.subunit_log, global_info, records_by_apps)
+
+if __name__ == "__main__":
+    main()

=== added file 'qakit/appstartup/parser.py'
--- qakit/appstartup/parser.py	1970-01-01 00:00:00 +0000
+++ qakit/appstartup/parser.py	2016-02-18 14:19:39 +0000
@@ -0,0 +1,65 @@
+#!/usr/bin/python3
+# UESQA Metrics
+# Copyright (C) 2016 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import ast
+
+
+def parse_startup_times(results):
+    """ Retrieve the global run information and a list with all the history
+    instances parsed from the results_list in the subunit test log
+    """
+    startup_times = []
+    tests_logs = _get_results_test_logs(results)
+    global_info = _get_global_information(tests_logs)
+
+    for test_log in tests_logs:
+        times = _get_times(test_log)
+        if global_info and times:
+            startup_times.extend(times)
+    return global_info, startup_times
+
+
+def _get_global_information(tests_logs):
+    for test_log in tests_logs:
+        markers = [line for line in test_log if '<=' in line]
+        if markers:
+            values = markers[0].split('<=')[1].strip()
+            return ast.literal_eval(values)
+    raise RuntimeError('Global info not provided for tests')
+
+
+def _get_results_test_logs(results):
+    test_logs = []
+    for result in results:
+        test_log = result['details'].get('test-log')
+        if test_log:
+            test_logs.append(test_log.as_text().split('\n'))
+    return test_logs
+
+
+def _get_times(test_log):
+    markers = [line for line in test_log if '=>' in line]
+    times = []
+
+    iteration = 1
+    for mark in markers:
+        values = mark.split('=>')[1].strip()
+        values_dict = ast.literal_eval(values)
+        values_dict['iteration'] = iteration
+        times.append(values_dict)
+        iteration += 1
+    return times

=== added file 'qakit/appstartup/plotter.py'
--- qakit/appstartup/plotter.py	1970-01-01 00:00:00 +0000
+++ qakit/appstartup/plotter.py	2016-02-18 14:19:39 +0000
@@ -0,0 +1,63 @@
+#!/usr/bin/python3
+# UESQA Metrics
+# Copyright (C) 2016 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import matplotlib.pyplot as plt
+import numpy as np
+import os
+
+
+def plot_app_records(app, builds, cold_times, cold_stdevs, hot_times,
+                     hot_stdevs, plots_dir):
+    """ Generate the graphic and return the graphic path.
+    It is supposed the records are already ordered by build number.
+    """
+
+    # Set descriptions
+    fig, ax = plt.subplots()
+    ax.set_title('Times for {} app'.format(app))
+    ax.set_xlabel('Build Number')
+    ax.set_ylabel('Time (seconds)')
+
+    # Config the graphic
+    ind = np.arange(len(builds))
+    width = 0.35
+    ax.set_xticks(ind + width)
+    ax.set_xticklabels(builds)
+    ax.set_xmargin(0.1)
+    ax.yaxis.grid(True)
+
+    rects_cold = ax.bar(ind, cold_times, width, color='#3366FF',
+                        yerr=cold_stdevs, ecolor='#FF1A00', capsize=5,
+                        error_kw={'elinewidth': 2})
+    rects_hot = ax.bar(ind + width, hot_times, width, color='#FF1A00',
+                       yerr=hot_stdevs, ecolor='#3366FF', capsize=5,
+                       error_kw={'elinewidth': 2})
+
+# Shrink current axis's height by 10% on the bottom
+    box = ax.get_position()
+    ax.set_position([box.x0, box.y0 + box.height * 0.1,
+                     box.width, box.height * 0.9])
+
+    if rects_cold and rects_hot:
+        ax.legend((rects_cold[0], rects_hot[0]), ('Cold Start', 'Hot Start'),
+                  loc='upper center', bbox_to_anchor=(0.85, -0.05),
+                  fancybox=True, shadow=True)
+
+    # Save the graphic
+    dst_file = os.path.join(plots_dir, 'barchart_{}.png'.format(app))
+    plt.savefig(dst_file)
+    return dst_file

=== added file 'qakit/appstartup/report.py'
--- qakit/appstartup/report.py	1970-01-01 00:00:00 +0000
+++ qakit/appstartup/report.py	2016-02-18 14:19:39 +0000
@@ -0,0 +1,72 @@
+#!/usr/bin/python3
+# UESQA Metrics
+# Copyright (C) 2016 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import json
+import os
+
+html_header = '<html><head><title>{}</title></head><body><div align="center"><table>'  # NOQA
+html_footer = '</table></div></body></html>'
+html_picture = '<tr><td><br/><br/><font size="7" color="#8B005A"><center>{}</center></font><br/><img src="{}"/></td></tr>'  # NOQA
+html_logo = '<img src="http://design.ubuntu.com/wp-content/uploads/canonical-logo1.png">'  # NOQA
+html_title = '<br/><br/><div><font size="7" color="#8B005A">Apps StartUp Execution Report</font></div>'  # NOQA
+html_info_title_header = '<br/><br/><div><font size="6" color="#8B005A">General information</font>'  # NOQA
+html_info_title_footer = '</div>'
+html_info = '<div><br/><font size="4">{}: {}</font></div>'
+
+
+def create_html_report(out_files, report_dir,
+                       report_file="app_startup_report.html",
+                       report_info=None):
+    report_path = os.path.join(report_dir, report_file)
+
+    with open(report_path, "w") as report:
+        # write out the html header
+        report.write(html_header.format('App startup times report'))
+        report.write(html_logo)
+
+        report.write(html_title)
+        report.write(html_info_title_header)
+        for info in sorted(report_info.keys()):
+            report.write(html_info.format(info.replace('_', ' ').title(),
+                                          report_info[info]))
+        report.write(html_info_title_footer)
+
+        for app, path in out_files.items():
+            report.write(html_picture.format(app.title(), path))
+
+        report.write(html_footer)
+
+    return report_path
+
+
+def create_json_report(out_files, report_dir,
+                       report_file="app_startup_report.json",
+                       report_info=None):
+
+    report_path = os.path.join(report_dir, report_file)
+    data = dict()
+    data['channel'] = report_info['channel']
+    data['device'] = report_info['device']
+    data['build_number'] = report_info['build_number']
+
+    charts = []
+    for app, path in out_files.items():
+        charts.append({"title": app.title(), "path": path})
+    data['charts'] = charts
+
+    with open(report_path, "w") as report:
+        report.write(json.dumps(data, ensure_ascii=False))

=== modified file 'qakit/config.py'
--- qakit/config.py	2014-10-08 01:30:28 +0000
+++ qakit/config.py	2016-02-18 14:19:39 +0000
@@ -21,5 +21,7 @@
 
 
 def get_config_file_location():
-    """Returns the full, absolute path to the config file location."""
-    return os.path.expanduser('~/.config/qakit.ini')
+    """ Return the full, absolute path to the config file location. When the
+    QAKIT_CONFIG env variable is defined, it is be used, otherwise it is
+    used the default config file in .config dir"""
+    return os.getenv('QAKIT_CONFIG', os.path.expanduser('~/.config/qakit.ini'))

=== modified file 'setup.py'
--- setup.py	2015-11-25 20:51:47 +0000
+++ setup.py	2016-02-18 14:19:39 +0000
@@ -38,6 +38,8 @@
         'console_scripts': [
             ('subunit-practitest-export = '
              'qakit.practitest.report_subunit_results_to_practitest:main'),
+            ('parse-app-startup = '
+             'qakit.appstartup.parse_startup_times:main'),
         ]
     }
 )