Merge lp:~canonical-platform-qa/ubuntu-test-cases/health-check_adding-nfss-file-generator-to-dep8 into lp:ubuntu-test-cases/touch

[Christopher Lee]

Added a script that takes the results from this test and generates a nfss-data.json file.

Added a script that takes the results from this test and generates a nfss-data.json file.

Currently as there are no existing graphs for this data I just included everything from the resulting health-check result files and bring them into a single file.

[Leo Arias (elopio)]

This looks good.
Just the same pita comment about using single quotes for strings.

You left this:
100 + import ipdb; ipdb.set_trace()

It would be nice if we find a way to run tests for these scripts. I think that's out of the scope of this sprint, and we won't have time anyway. Just something to think about, as I'm not sure it would make sense to add the scripts tests in the same place as the dep8 tests, and we don't have a make test step.

review: Needs Fixing

[lp:~canonical-platform-qa/ubuntu-test-cases/health-check_adding-nfss-file-generator-to-dep8]

349. By Christopher Lee on 2014-11-12

Remove debugging statement

350. By Christopher Lee on 2014-11-12

Made use of single quotes consistent.

[Richard Huddie (rhuddie)]

Hi Chris,
I have updated my health-check-to-dep8 branch to move the debian files under tests/health-check/debian. So you will now need to re-apply your changes from debian/tests/health-check to tests/health-check/debian/tests/health-check.

Thanks.

review: Needs Fixing

[lp:~canonical-platform-qa/ubuntu-test-cases/health-check_adding-nfss-file-generator-to-dep8]

351. By Christopher Lee on 2014-11-12

Undo changes for now

352. By Christopher Lee on 2014-11-12

Merge directory shuffle

353. By Christopher Lee on 2014-11-12

Re-introduce execution of nfss-gen script

354. By Christopher Lee on 2014-11-12

Write a different nfss data file if we're running on krillin

[Allan LeSage (allanlesage)]

These tests are still running on my mako but I was able to generate a large JSON file on the early output using veebers' script.

I have only teensy-minor comments here, balanced against our interest in landing this--you might add docstrings where they're missing, also maybe just use strip() instead of replace()? Approving nonetheless :) . . . .

review: Approve

[lp:~canonical-platform-qa/ubuntu-test-cases/health-check_adding-nfss-file-generator-to-dep8]

355. By Christopher Lee on 2014-11-13

As per MP comment, clarify some code and docstrings

[Leo Arias (elopio)]

review: Approve

[Richard Huddie (rhuddie)]

review: Approve

[Christopher Lee (veebers)]

Merged into: lp:ubuntu-health-check-tests

=== modified file 'tests/health-check/debian/tests/health-check'
--- tests/health-check/debian/tests/health-check	2014-11-13 03:56:24 +0000
+++ tests/health-check/debian/tests/health-check	2014-11-13 03:56:24 +0000
@@ -5,6 +5,7 @@
 GET_PIDS_SCRIPT=$LOCAL_SCRIPT_DIR"health-check-test-get-pids.py"
 RUN_PIDS_SCRIPT=$LOCAL_SCRIPT_DIR"health-check-test-run-pids.py"
 TEST_PID_SCRIPT=$LOCAL_SCRIPT_DIR"health-check-test-pid.py"
+NFSS_GENERATOR_SCRIPT=$LOCAL_SCRIPT_DIR"generator-nfss-data-file.py"
 PROCMAP_FILE=$LOCAL_SCRIPT_DIR"procmapping.txt"
 # generate the procmapping file and copy to logs
 python3 $GET_PIDS_SCRIPT
@@ -15,4 +16,8 @@
 RC=$?
 # copy json logs to artifacts dir
 cp *.json $ADT_ARTIFACTS
+
+# This script will leave a nfss-data.json file in $ADT_ARTIFACTS
+python3 $NFSS_GENERATOR_SCRIPT $ADT_ARTIFACTS
+
 exit $RC

=== added file 'tests/health-check/generator-nfss-data-file.py'
--- tests/health-check/generator-nfss-data-file.py	1970-01-01 00:00:00 +0000
+++ tests/health-check/generator-nfss-data-file.py	2014-11-13 03:56:24 +0000
@@ -0,0 +1,119 @@
+#!/usr/bin/python3
+
+"""This script takes all the resulting json files from the health-check run and
+creates a single data file that can be uploaded to nfss.
+
+This script takes 1 argument which is the source (which will also be the
+destination) directory in which to find all the .json files to process.
+
+The resulting output of this script will look like:
+
+[
+  {
+    project: 'health-check',
+    test: '<process-name of pid from file 1>',
+    data: { <contents of json file> }
+  },
+  {
+    project: 'health-check',
+    test: '<process-name of pid from file 2>',
+    data: { <contents of json file> }
+  },
+  ... and so on ...
+]
+
+"""
+import json
+import os
+import psutil
+import re
+import sys
+import subprocess
+from glob import glob
+
+
+def usage():
+    print('{} <source file path containing .json files>'.format(sys.argv[0]))
+
+
+def write_data_to_file(destination_path, run_data):
+    if device_is_krillin():
+        data_file_name = 'private-nfss-data.json'
+    else:
+        data_file_name = 'nfss-data.json'
+
+    file_output = os.path.join(destination_path, data_file_name)
+    print('Writing results to %s' % (file_output))
+    with open(file_output, 'w') as f:
+        json.dump(run_data, f)
+
+
+def device_is_krillin():
+    """Returns True if the device executing this code is a Krillin."""
+    try:
+        prop_output = subprocess.check_output(
+            ['getprop', 'ro.product.device'],
+            universal_newlines=True
+        ).strip('\n')
+        return prop_output == 'krillin'
+    except FileNotFoundError:
+        return False
+
+    return False
+
+
+def _get_process_name_from_filename(filename):
+    """Given a file, determine the process name from it.
+
+    The file should be in the format health-check-xxx.json where xxx is a pid.
+
+    Returns process name for the pid contained within the filename.
+    Returns the string "UNKNOWN PROCESS" if we fail to find the running pid.
+
+    """
+    pid_search = re.search(
+        'health-check-([0-9]*)\.json',
+        filename
+    )
+
+    try:
+        pid = pid_search.group(1)
+        p = psutil.Process(int(pid))
+        return p.name()
+    except (IndexError, psutil.NoSuchProcess):
+        print("PID %s no longer exists" % pid)
+        return 'UNKNOWN PROCESS'
+
+
+def main(source_path):
+    """Goes through all sub directories in *source_path* generating json
+    results from the results there and writing them all to a single file to be
+    uploaded to the nfss backend.
+
+    """
+    all_tests_data = []
+
+    file_name_pattern = os.path.join(source_path, 'health-check-*.json')
+    all_data_files = glob(file_name_pattern)
+    for json_file in all_data_files:
+        with open(json_file, 'r') as f:
+            process_name = _get_process_name_from_filename(json_file)
+            json_data = json.load(f)
+
+            all_tests_data.append(dict(
+                project='health-check',
+                test=process_name,
+                data=json_data
+            ))
+
+    write_data_to_file(source_path, all_tests_data)
+
+
+if __name__ == '__main__':
+    try:
+        source_path = sys.argv[1]
+    except IndexError:
+        usage()
+        exit(1)
+
+    main(source_path)