Merge ~canonical-kernel-team/+git/autotest-client-tests:phlin/ftrace-granularity into ~canonical-kernel-team/+git/autotest-client-tests:master

[Po-Hsu Lin]

Improve the ftracetest granularity by running sub-tests one-by-one.
With this patch, we will be able to hint specific known failures
without letting other regressions to slip through.

Test case listing is achieved by porting the find_testcases() code
in ftracetest script from tools/testing/selftests/ftrace of a kernel
tree. Test verbosity increased by using -vvv flag, this will increase
report file size but it will be easier for debugging.

As we're not running the whole test altogether, the timeout threshold
has been modified to 10 minutes for each case on non-riscv64 systems.
We might need to adjust this later.

I have also removed some leftovers when we copy this test from
ubuntu_kernel_selftests.

Patch tested on a Focal VM. The test number (88) is identical before and after applying this patch.

The verbosity flag comparison between -v, -vv and -vvv can be found here:
https://pastebin.ubuntu.com/p/bP7fhYTxpQ/

V2:
After testing on a RISCV instance, I recalled that we had this autotest timeout issue on RISCV (bug 1940080), the timeout threshold must be set to 0 consequently.

I have also remove the unused part for grepping test in sub-dir (if statement for cmd = 'grep SUB_DIRS {}'.format(mk_src)), which is only useful in ubuntu_kernel_selftests as we don't have sub-dir with Makefiles in ftrace test.

[Po-Hsu Lin (cypressyew)]

I will try to run this on an instance that will fail with this test.

[Po-Hsu Lin (cypressyew)]

I have this patch tested on node janbi with j-realtime, with 10 minute timeout and 30 minute timeout.

The results are identical:
* 10 min
  - timeout ubuntu_kselftests_ftrace.ftrace:test.d--direct--ftrace-direct.tc
  - timeout ubuntu_kselftests_ftrace.ftrace:test.d--event--subsystem-enable.tc
  - fail ubuntu_kselftests_ftrace.ftrace:test.d--ftrace--func_traceonoff_triggers.tc
* 30 min
  - timeout ubuntu_kselftests_ftrace.ftrace:test.d--direct--ftrace-direct.tc
  - timeout ubuntu_kselftests_ftrace.ftrace:test.d--event--subsystem-enable.tc
  - fail ubuntu_kselftests_ftrace.ftrace:test.d--ftrace--func_traceonoff_triggers.tc

They all timeout / fail on the same point, please find the log here:
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2027770/comments/1
https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2027770/comments/2

Therefore I think we can keep the timeout as 10min.

[Cory Todd (corytodd)]

> As we're not running the whole test altogether, the timeout threshold
has been modified to 10 minutes for each case on non-riscv64 systems.

I'm not seeing where we adjust for non-riscv64. Maybe that's handled in some other definition now?

[Po-Hsu Lin (cypressyew)]

> > As we're not running the whole test altogether, the timeout threshold
> has been modified to 10 minutes for each case on non-riscv64 systems.
>
> I'm not seeing where we adjust for non-riscv64. Maybe that's handled in some
> other definition now?
For riscv64, it will have arch_scale = 2 by this if statement:
    # Scale timeouts by 2 for riscv64, some tests timeout due to lack of
    # timeout, despite progressing fine
    if arch in ['riscv64']:
        arch_scale = 2

But this reminds me maybe I should give it a try on those RISCV64 first. I will do this tomorrow. Thanks!

[Cory Todd (corytodd)]

derp, there it is at the top of the control file thanks!

[Francis Ginther (fginther)]

The "-vvv" may end up being too verbose and end up with huge log files. But let's add it and see if it becomes a problem.

review: Approve

[Sean Feole (sfeole)]

review: Approve

[Po-Hsu Lin (cypressyew)]

Note: the -vvv flag will take the test output to about 36MB, but there is nothing really interesting with -v and -vv so...

[Po-Hsu Lin (cypressyew)]

Copy forward the acks.
Applied and pushed, thank for the feedback!

diff --git a/ubuntu_kselftests_ftrace/control b/ubuntu_kselftests_ftrace/control
index 7d27f65..3d4f04a 100644
--- a/ubuntu_kselftests_ftrace/control
+++ b/ubuntu_kselftests_ftrace/control
@@ -17,7 +17,7 @@ arch_scale = 1
 # Scale timeouts by 2 for riscv64, some tests timeout due to lack of
 # timeout, despite progressing fine
 if arch in ['riscv64']:
-   arch_scale = 2
+    arch_scale = 2
 
 result = job.run_test_detail(NAME, test_name='setup', tag='setup', timeout=arch_scale*60*45)
 if result == 'GOOD':
@@ -32,36 +32,26 @@ if result == 'GOOD':
         dir_src = os.path.join(dir_root, category)
         mk_src = os.path.join(dir_src, 'Makefile')
         os.chdir(dir_src)
-        cmd = 'grep SUB_DIRS {}'.format(mk_src)
-        timeout_threshold = arch_scale*60*30
-        if utils.system_output(cmd, verbose=False, ignore_status=True):
-            cmd = 'make -f {} -f {} getsubdirs'.format(mk_helper, mk_src)
-            subdirs = utils.system_output(cmd).split()
-            for subdir in subdirs:
-                dir_src = os.path.join(dir_root, category, subdir)
-                os.chdir(dir_src)
-                mk_src = os.path.join(dir_src, 'Makefile')
-                if os.path.isfile(mk_src):
-                   cmd = 'make -f {} -f {} gettests'.format(mk_src, mk_helper)
-                   tests = utils.system_output(cmd).split()
-                   for item in tests:
-                       test = "{}/{}:{}".format(category, subdir, item)
-                       job.run_test_detail(NAME, test_name=test, tag=test, timeout=timeout_threshold)
-        elif os.path.isfile(mk_src):
+        timeout_threshold = 60 * 10
+        if arch == 'riscv64':
+            # autotest timeout on riscv64 is incorrect (lp:1940080), disable it
+            timeout_threshold = 0
+
+        if os.path.isfile(mk_src):
             cmd = 'make -f {} -f {} gettests'.format(mk_src, mk_helper)
             tests = utils.system_output(cmd).split()
             for item in tests:
-                timeout_threshold = arch_scale*60*45
-                if item == 'ftracetest':
-                    if arch == 'riscv64':
-                        # autotest timeout on riscv64 is incorrect (lp:1940080), disable it
-                        # It takes about 22 mins on 5.15 and about 35 mins on 5.13
-                        timeout_threshold = 0
-                    else:
-                        # ftracetest will take about ~60 minutes to run on some instances (lp:2008063)
-                        timeout_threshold = 60 * 75
-                test = "{}:{}".format(category, item)
-                job.run_test_detail(NAME, test_name=test, tag=test, timeout=timeout_threshold)
+                # Get all sub-tests for ftracetest and run them one-by-one
+                cmd = 'find {}/test.d/ -name *.tc | sort'.format(dir_src)
+                ftrace_tests = utils.system_output(cmd).split()
+                for ftrace_sub_test in ftrace_tests:
+                    # Replace the '/' in test pathname with '--' (as '-' and '_' have been used)
+                    # to avoid dir name collisions, otherwise autotest will try to create a dir
+                    # for it. The next test in the same dir will error out with:
+                    # "multiple tests cannot run with the same subdirectory"
+                    clean_name = ftrace_sub_test.split('/selftests/ftrace/')[1].replace('/', '--')
+                    test = "{}:{}".format(category, clean_name)
+                    job.run_test_detail(NAME, test_name=test, tag=test, timeout=timeout_threshold)
 else:
     print("ERROR: test failed to build, skipping all the sub tests")
 
diff --git a/ubuntu_kselftests_ftrace/ubuntu_kselftests_ftrace.py b/ubuntu_kselftests_ftrace/ubuntu_kselftests_ftrace.py
index 787e120..0f14def 100644
--- a/ubuntu_kselftests_ftrace/ubuntu_kselftests_ftrace.py
+++ b/ubuntu_kselftests_ftrace/ubuntu_kselftests_ftrace.py
@@ -96,19 +96,11 @@ class ubuntu_kselftests_ftrace(test.test):
 
         category = test_name.split(':')[0]
         sub_test = test_name.split(':')[1]
-        dir_root = os.path.join(self.srcdir, 'linux', 'tools', 'testing', 'selftests')
+        dir_root = os.path.join(self.srcdir, 'linux', 'tools', 'testing', 'selftests', 'ftrace')
         os.chdir(dir_root)
-        cmd = "make run_tests -C {} TEST_PROGS={} TEST_GEN_PROGS='' TEST_CUSTOM_PROGS=''".format(category, sub_test)
+        # Run sub-tests with ftracetest script, convert test name back to path
+        test = sub_test.replace('--', '/')
+        cmd = './ftracetest -vvv {}'.format(test)
         result = utils.system_output(cmd, retain_output=True)
 
-        # Old pattern for Xenial
-        pattern = re.compile('selftests: *(?P<case>[\w\-\.]+) \[FAIL\]\n')
-        if re.search(pattern, result):
-            raise error.TestError(test_name + ' failed.')
-        # If the test was not end by previous check, check again with new pattern
-        pattern = re.compile('not ok [\d\.]* selftests: {}: {} # (?!.*SKIP)'.format(category, sub_test))
-        if re.search(pattern, result):
-            raise error.TestError(test_name + ' failed.')
-
-
 # vi:set ts=4 sw=4 expandtab syntax=python: