Merge ~djacobs98/checkbox-iiotg/+git/checkbox-provider-intliotg-3:cyclictest_2 into ~checkbox-dev/checkbox-iiotg/+git/checkbox-provider-intliotg:master

[Doug Jacobs]

Initial checkin.
Add cyclictest for RT to checkbox.
This includes the script that runs the test, entry in checkbox and the updates to the desktop and server test plans. For now, this will be part of the Stress test plan.

[Vic Liu (zongminl)]

See my comments inline

review: Needs Fixing

[PeiYao Chang (baconyao)]

Hi @doug, please see my comments inline.

review: Needs Fixing

[Doug Jacobs (djacobs98)]

Squashed & Pushed changes.

Here's the new pastebin showing the updated run:
https://pastebin.canonical.com/p/fQvpMfwdbX/

[Vic Liu (zongminl)]

Few more suggestions inline

review: Needs Fixing

[PeiYao Chang (baconyao)]

LGTM. No flak8 error be observed.

review: Approve

[Vic Liu (zongminl)]

+1, thank you

review: Approve

diff --git a/bin/run_cyclictest.py b/bin/run_cyclictest.py
new file mode 100755
index 0000000..e7adc3f
--- /dev/null
+++ b/bin/run_cyclictest.py
@@ -0,0 +1,148 @@
+#!/usr/bin/env python3
+# Copyright Canonical 2023
+# run_cyclictest.py
+#
+# Uses the cyclictest binary to run cyclictest and measure process latency.
+# This test needs to run for at least 24 hours in order to provide useful data.
+#
+# The output from the test, as well as any errors, are then dumped into a file
+# so the second part can check the results.  A passing test will depend on the
+# maximum latency observed and whether that is an acceptable number.  This
+# value depends on the DUT's platform and CPU.  These values are provided by
+# Intel as part of the Real Time Key Performance Indicator (RT KPI) document.
+#
+# Pass conditions:
+#   * Max latency is equal to or lower than the expected value for the DUT's HW
+#   * 0 overflows (caused by a process waiting too long. also called a "miss")
+#
+# Fail conditions:
+#   * Any errors from the test.
+#   * Unacceptable values for Max Latency or Overflows.
+
+import argparse
+import subprocess
+
+
+def run_it(duration):
+    """
+    This function runs the cyclictest command, and captures output and also
+    stores the output in a file for processing.
+
+    Inputs:
+    filename - the path and filename of the output file.
+    duration - the desired duration of the test in seconds.
+    """
+
+    # timeout must be larger than the test duration, otherwise the test will be
+    # interrupted
+    test_timeout = duration + 1
+
+    cmd = ["chrt", "-f", "99", "cyclictest", "-a2-3", "-t2", "-m", "-p99",
+           "-i250", "-h700", "-q", "-D", str(duration)]
+
+    try:
+        result = subprocess.run(
+                cmd, capture_output=True, text=True, timeout=test_timeout,
+                check=True)
+    except Exception as err:
+        raise SystemExit(err)
+
+    # Note: A returncode of 0 simply means the command executed successfully.
+    # It does NOT mean the test passed (or failed.)
+    # That will be determined by a separate function.
+    return result
+
+
+def lookup_max_latency():
+    """
+    The expected maximum latency is based on the HW platform and CPU.
+    For instance, ADL-S with an i7-12700 has an expected maximum latency of
+    11ms
+
+    However, some hardware may not proper identification yet.
+    In this case just return the largest latency listed in the document.
+    """
+
+    # In case we cannot determine the HW or the CPU of the DUT, use the largest
+    # value from the table of expected maximum latency.
+    MAX_ALLOWED_LATENCY = 17
+
+    # TODO: Implement the methods to determine the platform and CPU, and use
+    #       that to lookup the expected maximum latency.
+    #
+    #       For now, just return MAX_ALLOWED_LATENCY
+
+    return MAX_ALLOWED_LATENCY
+
+
+def verify_cyclictest_results(result):
+    """
+    This function takes output from the call to subprocess containing the
+    output from a run of cyclictest and looks at std out to verify two things:
+    1. The maximum latency from each thread did not exceed the maximum latency
+       specified by Intel for the DUT's HW Platform and CPU.
+
+    2. The number of overflows (scheduling misses) was 0 for each thread.
+
+    Both conditions must be true for this function to return success (0)
+
+    Sample output for these lines looks like:
+    # Max Latencies: 00041 00046
+    # Histogram Overflows: 00000 00000
+    The values after these strings are the results from each thread.
+
+    All other lines are ignored.
+    """
+    max_latency = lookup_max_latency()
+    print("Maximum latency allowed: " + str(max_latency))
+    # list of latencies, from each thread
+    latencies = []
+
+    # list of overflows (scheduling misses) from each thread
+    overflows = []
+
+    return_code = 0
+
+    std_output = result.stdout.split("\n")
+    for line in std_output:
+
+        if "# Max Latencies" in line:
+            latencies = line.split(":")[1].split()
+            for latency in latencies:
+                if int(latency) > max_latency:
+                    print("Test fails.\tLatency : " + latency +
+                          "\t-- Maximum Latency value of: " +
+                          str(max_latency) + " exceeded.")
+                    return_code = 1
+                else:
+                    print("Test passes.\tLatency : " + latency + " -- OK.")
+
+        elif "# Histogram Overflows" in line:
+            overflows = line.split(":")[1].split()
+            for overflow in overflows:
+                if int(overflow) > 0:
+                    print("Test fails.\tOverflow: " + overflow +
+                          " -- 0 Overflows expected.")
+                    return_code = 1
+                else:
+                    print("Test passes.\tOverflow: " + overflow + " -- OK.")
+            continue
+
+    return return_code
+
+
+if __name__ == "__main__":
+    return_code = 0
+
+    parser = argparse.ArgumentParser()
+    # default duration is 86400 seconds = 24hr
+    parser.add_argument("--duration", type=int, default=86400)
+    args = parser.parse_args()
+
+    result = run_it(args.duration)
+    print("stdout: {}".format(result.stdout))
+    print("stderr: {}\n".format(result.stderr))
+
+    return_code = verify_cyclictest_results(result)
+
+    raise SystemExit(return_code)
diff --git a/units/rt-performance-tests/category.pxu b/units/rt-performance-tests/category.pxu
new file mode 100644
index 0000000..43649f2
--- /dev/null
+++ b/units/rt-performance-tests/category.pxu
@@ -0,0 +1,3 @@
+unit: category
+id: rt-performance-tests
+_name: RT Performance Tests
diff --git a/units/rt-performance-tests/jobs.pxu b/units/rt-performance-tests/jobs.pxu
new file mode 100644
index 0000000..43c02d8
--- /dev/null
+++ b/units/rt-performance-tests/jobs.pxu
@@ -0,0 +1,29 @@
+# test based on Intel's document "Real-Time Key Performance Indicators (KPIs)
+# Latest version: https://drive.google.com/drive/u/0/folders/1oP5YaynNkB1TEUp8hz0iPPfswZK3N5_O
+#
+# Add to stress tests (for now)
+
+# Test case: cyclictest__rt_latency
+# Purpose: Test scheduling latency by running the cyclictest tool. 
+# Test described in Intel's document on Real-Time Key Performance Indicators.
+#
+# It is suggested duration of this is 24hrs, or 86440 seconds.
+#
+# The expected Maximum Latency will vary based on the Platform and CPU that the
+# test is run on.  However for now, just use the largest value of 17 microseconds.
+#
+# This test will pass if:
+#   * The maximum latency is 17 microseconds or lower.
+#   * There are no overflows.
+#
+# Otherwise the test will fails.
+
+id: rt-performance-tests/cyclictest_rt_latency
+category_id: rt-performance-tests
+_summary:
+    Run cyclictest to test scheduling latency. Default run time is 24 hours.
+plugin: shell
+user: root
+environ: CYCLICTEST_DURATION
+command:
+    run_cyclictest.py --duration "${CYLICTEST_DURATION:-86400}"
diff --git a/units/rt-performance-tests/test-plan.pxu b/units/rt-performance-tests/test-plan.pxu
new file mode 100644
index 0000000..6e5c59e
--- /dev/null
+++ b/units/rt-performance-tests/test-plan.pxu
@@ -0,0 +1,27 @@
+id: rt-performance-tests-full
+_name: Realtime Peformance Tests (Full)
+unit: test plan
+include:
+nested_part:
+    rt-performance-tests-manual
+    rt-performance-tests-automated
+    rt-performance-tests-stress
+
+id: rt-performance-tests-manual
+_name: Realtime Performance Tests (Manual)
+unit: test plan
+include:
+bootstrap_include:
+
+id: rt-performance-tests-automated
+_name: Realtime Performance Tests (Automated)
+unit: test plan
+include:
+bootstrap_include:
+
+id: rt-performance-tests-stress
+_name: Realtime Performance Tests (Stress)
+unit: test plan
+include:
+    rt-performance-tests/cyclictest_rt_latency
+bootstrap_include:
diff --git a/units/test-plan-desktop.pxu b/units/test-plan-desktop.pxu
index 23c0390..506e3d2 100644
--- a/units/test-plan-desktop.pxu
+++ b/units/test-plan-desktop.pxu
@@ -135,6 +135,7 @@ include:
 nested_part:
     com.canonical.certification::stress-iperf3-automated                       # keep if ethernet is supported
     com.canonical.certification::client-cert-iot-server-22-04-stress
+    rt-performance-tests-stress
 exclude:
     com.canonical.certification::stress-tests/hibernate.*
 
diff --git a/units/test-plan-server.pxu b/units/test-plan-server.pxu
index d8c0f95..8e4288b 100644
--- a/units/test-plan-server.pxu
+++ b/units/test-plan-server.pxu
@@ -124,6 +124,7 @@ include:
 nested_part:
     com.canonical.certification::stress-iperf3-automated                       # keep if ethernet is supported
     com.canonical.certification::client-cert-iot-server-22-04-stress
+    rt-performance-tests-stress
 exclude:
     com.canonical.certification::stress-tests/hibernate.*