Merge ~djacobs98/checkbox-iiotg/+git/checkbox-provider-intliotg:cyclictest into ~checkbox-dev/checkbox-iiotg/+git/checkbox-provider-intliotg:master

[Doug Jacobs]

Initial commit.
Add cyclictest script and test case to stress test test plan for desktop and server.

[PeiYao Chang (baconyao)]

Doug, there are few things need you to fix, I list them below.

1. The style of Checkbox job
  - Always have the test-plan.pxu and well structure for each units.
  - Problem:
    - Missing the test-plan.pxu in cyclictest unit
    - Do not include the job into the intliotg-desktop-22-04-stress and intliotg-server-22-04-stress, you can reference the ishtp sample (https://git.launchpad.net/~checkbox-dev/checkbox-iiotg/+git/checkbox-provider-intliotg/tree/units/ishtp/test-plan.pxu)

2. There are some errors thrown by flake8
  - After setting up the dev environment of Checkbox, run flake8 to check python scripts
    - Checkbox dev environment: https://github.com/canonical/checkbox/blob/main/CONTRIBUTING.md
    - flake8 command: `python -m flake8 bin/run_cyclictest.py`
  - Errors
    ```
    bin/run_cyclictest.py:10:80: E501 line too long (80 > 79 characters)
    bin/run_cyclictest.py:24:1: F401 'sys' imported but unused
    bin/run_cyclictest.py:31:1: E302 expected 2 blank lines, found 1
    bin/run_cyclictest.py:31:32: W291 trailing whitespace
    bin/run_cyclictest.py:33:80: E501 line too long (91 > 79 characters)
    bin/run_cyclictest.py:34:5: F841 local variable 'test_timeout' is assigned to but never used
    bin/run_cyclictest.py:37:80: E501 line too long (183 > 79 characters)
    bin/run_cyclictest.py:50:11: E275 missing whitespace after keyword
    bin/run_cyclictest.py:57:1: E302 expected 2 blank lines, found 1
    bin/run_cyclictest.py:59:1: E115 expected an indented block (comment)
    bin/run_cyclictest.py:60:1: E115 expected an indented block (comment)
    bin/run_cyclictest.py:68:11: E275 missing whitespace after keyword
    bin/run_cyclictest.py:83:1: E302 expected 2 blank lines, found 1
    bin/run_cyclictest.py:96:22: E701 multiple statements on one line (colon)
    bin/run_cyclictest.py:103:80: E501 line too long (130 > 79 characters)
    bin/run_cyclictest.py:113:80: E501 line too long (93 > 79 characters)
    bin/run_cyclictest.py:120:11: E275 missing whitespace after keyword
    bin/run_cyclictest.py:131:63: E261 at least two spaces before inline comment
    ```

review: Needs Fixing

[Doug Jacobs (djacobs98)]

Sample output from a 1 second run:
https://pastebin.canonical.com/p/4fPRnbbFSQ/

[Doug Jacobs (djacobs98)]

Sample output of 1 second test from Checkbox take 2:
https://pastebin.canonical.com/p/KrG9qv55VJ/

Unmerged commits

cc3c353... by Doug Jacobs on 2023-04-11

Initial commit. Add cyclictest to stress tests for Desktop & Server.
The test is run as a python script in bin/

diff --git a/bin/run_cyclictest.py b/bin/run_cyclictest.py
new file mode 100755
index 0000000..9a6c1a3
--- /dev/null
+++ b/bin/run_cyclictest.py
@@ -0,0 +1,141 @@
+#!/usr/bin/env python3
+# Copyright Canonical 2023
+# run_cyclictest.py
+#
+# Uses the cylictest binary to run cyclictest and measure process latency.
+# This test needs to run for at least 24 hours in order to provide useful data.
+#
+# The output from the test, as well as any errors, are then dumped into a file
+# so the second part can check the results.  A passing test will depend on the
+# maximum latency observed and whether that is an acceptable number.  This value
+# depends on the DUT's platform and CPU.  These values are provided by Intel as
+# part of the Real Time Key Performance Indicator (RT KPI) document.
+#
+# Pass conditions:
+#   * Max latency is equal to or lower than the expected value for the DUT's HW
+#   * 0 overflows (caused by a process waiting too long. also called a "miss")
+#
+# Fail conditions:
+#   * Any errors from the test.
+#   * Unacceptable values for Max Latency or Overflows.
+
+import argparse
+import subprocess
+import sys
+import os
+
+# This function runs the cyclictest command, and captures output and also
+# stores the output in a file for processing.
+# filename - the path and filename of the output file.
+# duration - the desired duration of the test in seconds.
+def run_it(filename, duration): 
+
+    # timeout must be larger than the test duration, otherwise the test will be interrupted
+    test_timeout = duration + 1
+
+    result = subprocess.run(
+            ["sudo", "chrt", "-f", "99", "cyclictest", "-a2-3", "-t2", "-m", "-p99", "-i250", "-h700", "-q", "-D", str(duration)], capture_output=True, text=True, timeout=(duration+1)
+    )
+
+    print("stdout:", result.stdout)
+    print("stderr:", result.stderr)
+
+    with open(filename, 'w') as f:
+        f.write(result.stdout)
+        f.write(result.stderr)
+
+    # Note: A returncode of 0 simply means the command executed successfully.
+    # It does NOT mean the test passed (or failed.)
+    # That will be determined by a separate function.
+    return(result.returncode)
+
+# The expected maximum latency is based on the HW platform and CPU.
+# For instance, ADL-S with an i7-12700 has an expected maximum latency of 11ms
+#
+# However, some hardware may not proper identification yet, in which case just
+# return the largest latency listed in the document.
+def lookup_max_latency():
+
+# In case we cannot determine the HW or the CPU of the DUT, use the largest
+# value from the table of expected maximum latency.
+    MAX_ALLOWED_LATENCY = 17
+
+# TODO: Implement the methods to determine the platform and CPU, and use that
+#       to lookup the expected maximum latency.
+#
+#       For now, just return MAX_ALLOWED_LATENCY
+
+    return(MAX_ALLOWED_LATENCY)
+
+# This function takes a filename containing the output from a run of cyclictest
+# and verifies two things:
+# 1. The maximum latency from each thread did not exceed the maximum latency
+#    specified by Intel for the DUT's HW Platform and CPU.
+#
+# 2. The number of overflows (scheduling misses) was 0 for each thread.
+#
+# Sample output for these lines looks like:
+# # Max Latencies: 00041 00046
+# # Histogram Overflows: 00000 00000
+# The values after these strings are the results from each thread.
+#
+# All other lines are ignored.
+def verify_cyclictest_results(cyclictest_file):
+    max_latency = lookup_max_latency()
+    print("Maximum latency allowed: " + str(max_latency))
+    latencies = []
+    overflows = []
+    strings = []
+
+    return_code = 0
+
+    f = open(cyclictest_file, "r")
+
+    for line in f:
+        line = line.strip()
+        if line == "": continue
+
+        strings = line.split(":")
+        if strings[0] == "# Max Latencies":
+            latencies = strings[1].split()
+            for latency in latencies:
+                if int(latency) > max_latency:
+                    print("Latency : " + latency + " -- Maximum Latency value of: " + str(max_latency) + " exceeded. Test fails.")
+                    return_code = 1
+                else:
+                    print("Latency : " + latency + " -- OK.")
+            continue
+
+        if strings[0] == "# Histogram Overflows":
+            overflows = strings[1].split()
+            for overflow in overflows:
+                if int(overflow) > 0:
+                    print("Overflow: " + overflow + " -- 0 Overflows expected.  Test fails.")
+                    return_code = 1
+                else:
+                    print("Overflow: " + overflow + " -- OK.")
+            continue
+
+    f.close()
+    return(return_code)
+
+
+if __name__ == "__main__":
+    PLAINBOX_SESSION_SHARE = os.environ.get('PLAINBOX_SESSION_SHARE', '')
+    return_code = 0
+
+    # path inside Checkbox for files, plus the default filename
+    filename = PLAINBOX_SESSION_SHARE + "cyclictest.log"
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--duration", type=int, default=86400) # default = 24h
+    args = parser.parse_args()
+
+    return_code = run_it(filename, args.duration)
+    if return_code != 0:
+        print("Error occurred during execution - aborting")
+        exit(return_code)
+
+    return_code = verify_cyclictest_results(filename)
+
+    exit(return_code)
diff --git a/units/cyclictest/category.pxu b/units/cyclictest/category.pxu
new file mode 100644
index 0000000..41c847c
--- /dev/null
+++ b/units/cyclictest/category.pxu
@@ -0,0 +1,3 @@
+unit: category
+id: cyclictest-latency-performance-test
+_name: Cyclictest Latency Performance Test
diff --git a/units/cyclictest/jobs.pxu b/units/cyclictest/jobs.pxu
new file mode 100644
index 0000000..a352fba
--- /dev/null
+++ b/units/cyclictest/jobs.pxu
@@ -0,0 +1,26 @@
+# test based on Intel's document "Real-Time Key Performance Indicators (KPIs)
+# Latest version: https://drive.google.com/drive/u/0/folders/1oP5YaynNkB1TEUp8hz0iPPfswZK3N5_O
+#
+# Add to stress tests (for now)
+
+# Test case: cyclictest_24h_rt_latency
+# Purpose: Test scheduling latency by running the cyclictest tool for 24 hours.
+# Test described in Intel's document on Real-Time Key Performance Indicators.
+# 
+# The expected Maximum Latency will vary based on the Platform and CPU that the
+# test is run on.  However for now, just use the largest value of 17 microseconds.
+#
+# This test will pass if:
+#   * The maximum latency is 17 microseconds or lower.
+#   * There are no overflows.
+#
+# Otherwise the test will fails.
+
+id: stress/cyclictest_24h_rt_latency
+category_id: com.canonical.plainbox::stress
+_summary:
+    Run cyclictest for 24 hours to test the scheduling latency for RT kernel.
+plugin: shell
+user: root
+command:
+    run_cyclictest.py
diff --git a/units/test-plan-desktop.pxu b/units/test-plan-desktop.pxu
index 23c0390..877d87a 100644
--- a/units/test-plan-desktop.pxu
+++ b/units/test-plan-desktop.pxu
@@ -135,6 +135,7 @@ include:
 nested_part:
     com.canonical.certification::stress-iperf3-automated                       # keep if ethernet is supported
     com.canonical.certification::client-cert-iot-server-22-04-stress
+    com.canonical.certification::stress/cyclictest_24h_rt_latency
 exclude:
     com.canonical.certification::stress-tests/hibernate.*
 
diff --git a/units/test-plan-server.pxu b/units/test-plan-server.pxu
index d8c0f95..708b0d7 100644
--- a/units/test-plan-server.pxu
+++ b/units/test-plan-server.pxu
@@ -124,6 +124,7 @@ include:
 nested_part:
     com.canonical.certification::stress-iperf3-automated                       # keep if ethernet is supported
     com.canonical.certification::client-cert-iot-server-22-04-stress
+    com.canonical.certification::stress/cyclictest_24h_rt_latency
 exclude:
     com.canonical.certification::stress-tests/hibernate.*