Merge ~canonical-kernel-team/+git/autotest-client-tests:ubuntu_stress_smoke_split_test into ~canonical-kernel-team/+git/autotest-client-tests:master

[Paolo Pisati]

Split ubuntu_stress_smoke_split_test on a per stressor basis, this way the result page should be easier to review and failed tests are easier to spot.

[Po-Hsu Lin (cypressyew)]

Hi Paolo, please find inline comments.

One side note.
I have this tested with Jammy kernel on openstack as well. The elapsed time will increase from 16 minutes to 90 minutes (provision will take about 8 minutes)

I think the bottleneck is swap on/off, the preparation for a single stressor will take about 15 seconds to run:

11:24:35 INFO | START ubuntu_stress_smoke_test.utime ubuntu_stress_smoke_test.utime timestamp=1701861875 timeout=2100 localtime=Dec 06 11:24:35
....
11:24:35 DEBUG| Running '/home/ubuntu/autotest/client/tests/ubuntu_stress_smoke_test/ubuntu_stress_single_smoke_test.sh utime'
....
11:24:50 DEBUG| [stdout] Tests took 0 seconds to run
11:24:51 INFO | GOOD ubuntu_stress_smoke_test.utime ubuntu_stress_smoke_test.utime timestamp=1701861891 localtime=Dec 06 11:24:51 completed successfully

This can be treated as a follow-up after this commit.

review: Needs Fixing

[Paolo Pisati (p-pisati)]

I removed all white spaces and converted s/$s/$STRESSOR/.

I wonder if could move swap creation/allocation into setup() and thus cut down execution time, let me give it a try.

[Paolo Pisati (p-pisati)]

Ok, i moved swap creation/deletion in ubuntu_stress_smoke_test.py, that should save us some time - give it a shot.

[Po-Hsu Lin (cypressyew)]

Cool! Thanks for fixing this.
The test case number is correct now (setup + 267 stressors + swapoff)

I just found out there is a "sleep 15" command in the shell script, right after setting the oom level. That's the real cause of this time-consuming issue. I think that one should be moved out as well. (It's still a good thing to not swap on/off repeatedly.)

Also, there is one issue with doing swap configuration in setup():

The logic for autotest is that the setup() will get triggered only once if the test case version didn't get bumped. So when you try to run the same test suite on the same machine without re-provisioning it or removing autotest/client/tmp/<test suite> directory, the swap won't be enabled on your second attempt. This happens when we need to debug some issues manually.

This is the workflow:
1. initialize()
2. setup()
3. run_once() with stressors
4. swapoff()

With a second run immediately after the first one:
5. initialize()
6. run_once() with stressors
7. swapoff()

I can think of two possible solutions now.
1. Like when we did for ubuntu_lttng_smoke_test [1], move these pre-test configuration codes from setup() to initialize(), in which it is guaranteed to be executed on the very beginning of every run. (The only problem is that this will begin before setup(), it will fail if any of these action depends on a package that needs to be installed in setup() first)
2. Create a new job like the swapoff() you made here, and run it before iterating through all of the stressors

Solution 2 might be more straightforward, we can break the smoke_test shell script into 3 parts:
1. Pre-test configuration like check_machine(), sleep 15 and set_max_oom_level into one script. And determine if we need to skip the stressors base on the check_machine() result there. (For example a VM with small ram. Now it will still iterate through all the tests but without doing any real things. These skipped tests will be treated as PASSED and it cloud give us a false impression that this has been tested and the result is good.)
2. The stressor execution part, which is done.
3. cleanup, swapoff, zswap and cgroup_mem restoration part.

Part 1 and 3 can be embedded in ubuntu_stress_smoke_test.py like what you have already done for swapoff() here. Or just stay in a script and call them within ubuntu_stress_smoke_test.py. The latter might give us more flexibility whenever we need to move away from autotest. But I am good with both choices since we're not planning to do this now.

Sorry for getting this comment too lengthy, please feel free to ping me is you have any question.

[1] https://git.launchpad.net/~canonical-kernel-team/+git/autotest-client-tests/commit/?id=ca3878e0ecac7a284ad6ee0175db72e1d9cefa9d

review: Needs Fixing

[Paolo Pisati (p-pisati)]

Ok, i moved the swap on / oom part into a separate script (ubuntu_stress_smoke_test_init.sh) and i'm executing it in the initializate() method, but now i noticed that initialize() is called every time run_once() is invoked (thus creating a swap file every time a single test is run): wasn't that supposed to run just at the start of autotest?

[Paolo Pisati (p-pisati)]

I moved the swapon/swapoff part into two separate scripts (ubuntu_stress_smoke_test_init.sh / ubuntu_stress_smoke_test_cleanup.sh) invoked from control.

Since setup() is already called after initialize() from autotest itself, i don't understand why we call it again from control, and we can probably remove it, but we can save that for a subsequent pull req.

[Po-Hsu Lin (cypressyew)]

> Ok, i moved the swap on / oom part into a separate script
> (ubuntu_stress_smoke_test_init.sh) and i'm executing it in the initializate()
> method, but now i noticed that initialize() is called every time run_once() is
> invoked (thus creating a swap file every time a single test is run): wasn't
> that supposed to run just at the start of autotest?

Hi Paolo,
yes you're right! I just noticed that I got confused by the ubuntu_lttng_smoke_test which only has just one single test in the control file.

The initalize() method is indeed being called EVERY TIME when the test is started. So the actual workflow is like this:
1. initialize()
2. setup()
3. run_once() with stressors
4. initialize()
5. run_once()

From the document [1]:
initialize() - This is run before everything, every time the test is run.

[1] https://github.com/autotest/autotest/blob/0862aa74389bf434533697a9d970cf91e03b09ce/documentation/source/main/local/AddingTest.rst#L32

[Po-Hsu Lin (cypressyew)]

> I moved the swapon/swapoff part into two separate scripts
> (ubuntu_stress_smoke_test_init.sh / ubuntu_stress_smoke_test_cleanup.sh)
> invoked from control.
Thanks!
This looks good.

>
> Since setup() is already called after initialize() from autotest itself, i
> don't understand why we call it again from control, and we can probably remove
> it, but we can save that for a subsequent pull req.
Hi,
Calling setup() explicitly give us the opportunity to:
1. Abort the test early when setup() has failed, otherwise all the sub-tests will try to run setup() again.
2. We can also abort the tests after setup() intentionally, to help us build the test for manual tests (see build-only branch)
3. Have an extra setup entry on the test report, when it fails we can tell it's a test build issue. Otherwise we will need to look into the result of sub-tests.
4. Make setup() itself has it's own timeout setting, sometimes it can take a while for setup() to run

[Po-Hsu Lin (cypressyew)]

+1 on this version, thanks for the hard work.

I will make follow-up change to remove set_max_oom_level() and sleep command in ubuntu_stress_single_smoke_test.sh

review: Approve

diff --git a/ubuntu_stress_smoke_test/control b/ubuntu_stress_smoke_test/control
index a57199e..394b9f9 100644
--- a/ubuntu_stress_smoke_test/control
+++ b/ubuntu_stress_smoke_test/control
@@ -10,10 +10,63 @@ TEST_CATEGORY = 'Functional'
 TEST_TYPE = "client"
 DOC = ""
 
+
+#
+# Tests that can lock up some kernels or are CPU / arch specific, so exclude them for now
+#
+lockup1 = ["rdrand", "numa", "quota", "apparmor", "cpu-online", "kcmp", "copy-file", "exec"]
+lockup2 = ["spawn", "remap", "stack", "oom-pipe", "resources", "opcode", "sockfd" ,"vforkmany"]
+lockup3 = ["sockpair", "bind-mount", "funccall", "ioport", "watchdog", "mlockmany", "idle-page", "clone"]
+lockup = lockup1 + lockup2 + lockup3
+
+#
+# Tests that are not kernel specific
+#
+generic1 = ["atomic", "bsearch", "heapsort", "hsearch", "longjmp", "lsearch", "matrix" ,"memcpy"]
+generic2 = ["rdrand", "str", "tsc", "vecmath", "wcs", "zlib", "matrix-3d", "l1cache"]
+generic3 = ["nop", "qsort"]
+generic = generic1 + generic2 + generic3
+
+#
+# Tests that are known to cause breakage
+#
+breakage = ["xattr", "efivar", "sysinfo", "sysinval"]
+
+#
+# Currenly a new stress test can causes problems with s390x on older kernels
+# https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1905438
+#
+legacy = ["uprobe"]
+
+exclude = lockup + generic + breakage + legacy
+
+#
+# Tests that should be skipped on KVM kernels
+#
+cmd = "uname -r | awk -F'-' '{print $NF}'"
+flavour = utils.system_output(cmd, retain_output=True)
+if flavour == 'kvm':
+    exclude.append('dnotify')
+
+
+test_dir = os.path.join(job.bindir, 'tests', NAME)
+cmd = '%s/ubuntu_stress_smoke_test_init.sh' % (test_dir)
+utils.system_output(cmd, retain_output=True)
+
 result = job.run_test_detail(NAME, test_name='setup', tag='setup', timeout=60*60)
 if result == 'GOOD':
-    results = job.run_test_detail(NAME, test_name='stress-smoke-test', tag='stress-smoke-test', timeout=35*60)
+    stressng_dir = os.path.join(job.bindir, 'tmp', NAME, 'src', 'stress-ng')
+    cmd = '%s/stress-ng --stressor' % (stressng_dir)
+    stressors = utils.system_output(cmd, retain_output=True).split()
 else:
-    print("ERROR: test failed to build, skipping all the sub tests")
+    print("ERROR: can't get a list of available stressors")
+    exit(1)
+
+for stress in stressors:
+    if stress not in exclude:
+        job.run_test_detail(NAME, test_name=stress , tag=stress, timeout=35*60)
+
+cmd = '%s/ubuntu_stress_smoke_test_cleanup.sh' % (test_dir)
+utils.system_output(cmd, retain_output=True)
 
 # vi:set ts=4 sw=4 expandtab syntax=python:
diff --git a/ubuntu_stress_smoke_test/ubuntu_stress_smoke_test.sh b/ubuntu_stress_smoke_test/ubuntu_stress_single_smoke_test.sh
similarity index 70%
rename from ubuntu_stress_smoke_test/ubuntu_stress_smoke_test.sh
rename to ubuntu_stress_smoke_test/ubuntu_stress_single_smoke_test.sh
index 729d474..e3be957 100755
--- a/ubuntu_stress_smoke_test/ubuntu_stress_smoke_test.sh
+++ b/ubuntu_stress_smoke_test/ubuntu_stress_single_smoke_test.sh
@@ -1,5 +1,8 @@
 #!/bin/bash 
 
+[ $# -lt 1 ] && echo "$0 requires STRESSOR name to be run" && exit 1
+STRESSOR=${1}
+
 # Maximum machine age in years
 MAX_AGE=5
 # minimum required memory in MB
@@ -59,45 +62,7 @@ INSTANCES=4
 #
 #STRESS_OPTIONS="--ignite-cpu --maximize --syslog --verbose --verify"
 STRESS_OPTIONS="--ignite-cpu --syslog --verbose --verify --oomable"
-#
-# Tests that can lock up some kernels or are CPU / arch specific, so exclude them for now
-#
-EXCLUDE="rdrand numa quota apparmor cpu-online kcmp copy-file exec "
-EXCLUDE+="spawn remap stack oom-pipe resources opcode sockfd vforkmany sockpair "
-EXCLUDE+="bind-mount funccall ioport watchdog mlockmany idle-page clone "
-#
-# Tests that are not kernel specific
-#
-EXCLUDE+="atomic bsearch heapsort hsearch longjmp lsearch matrix memcpy nop qsort "
-EXCLUDE+="rdrand str tsc vecmath wcs zlib matrix-3d l1cache "
-#
-# Tests that are known to cause breakage
-#
-EXCLUDE+="xattr efivar sysinfo sysinval "
-#
-# Currenly a new stress test can causes problems with s390x on older kernels
-# https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1905438
-#
-EXCLUDE+="uprobe "
-#
-# Tests that should be skipped on KVM kernels
-#
-[ "$(uname -r | awk -F'-' '{print $NF}')" == "kvm" ] && EXCLUDE+="dnotify "
-#
-# Tests that break on specific kernel versions and we won't fix
-#
-ver=$(uname -r | cut -d'.' -f1-2)
-if [ "$ver" == "4.20" ]; then
-	#
-	#  Broken on 4.20, fixed in 5.0-rc2
-	#
-	EXCLUDE+="dccp sctp "
-fi
 
-#
-# Get built-in stressor names
-#
-STRESSORS=$(./stress-ng --stressors)
 rc=0
 TMP_FILE=/tmp/stress-$$.log
 
@@ -185,18 +150,6 @@ secs_now()
 	date "+%s"
 }
 
-not_exclude()
-{
-	for x in $2
-	do
-		if [ $x == $1 ]
-		then
-			return 1
-		fi
-	done
-	return 0
-}
-
 #
 #  Try an ensure that this script and parent won't be oom'd
 #
@@ -239,25 +192,6 @@ else
 	SYS_ZSWAP_SETTING="N"
 fi
 
-#
-#  Always add 1GB of swap to ensure swapping is exercised
-#
-SWPIMG=$PWD/swap.img
-
-#
-#  Create 1GB swap file
-#
-swapoff ${SWPIMG} >& /dev/null
-fallocate -l 1G ${SWPIMG}
-if [ $? -ne 0 ]; then
-	echo "FAILED: Count not create 1GB swap file, file system:"
-	df
-	exit 1
-fi
-chmod 0600 ${SWPIMG}
-mkswap ${SWPIMG}
-swapon ${SWPIMG}
-
 echo " "
 echo "Machine Configuration"
 echo "Physical Pages:  $(getconf _PHYS_PAGES)"
@@ -295,96 +229,87 @@ fi
 
 count=0
 s1=$(secs_now)
-for s in ${STRESSORS}
-do
-	if not_exclude $s "$EXCLUDE"
-	then
-		count=$((count + 1))
-		dmesg -c >& /dev/null
-		echo "$s STARTING"
-		${RUN_STRESS} -v -t ${DURATION} --${s} ${INSTANCES} --${s}-ops ${MAX_BOGO_OPS} ${STRESS_OPTIONS} >& ${TMP_FILE}
-		ret=$?
-		echo "$s RETURNED $ret"
-
-		n=$(dmesg | grep "Out of memory:" | wc -l)
-		if [ $ret -ne 0 -a $n -gt 0 ]; then
-			ret=88
-		fi
+count=$((count + 1))
+dmesg -c >& /dev/null
+echo "$STRESSOR STARTING"
+${RUN_STRESS} -v -t ${DURATION} --${STRESSOR} ${INSTANCES} --${STRESSOR}-ops ${MAX_BOGO_OPS} ${STRESS_OPTIONS} >& ${TMP_FILE}
+ret=$?
+echo "$STRESSOR RETURNED $ret"
+
+n=$(dmesg | grep "Out of memory:" | wc -l)
+if [ $ret -ne 0 -a $n -gt 0 ]; then
+	ret=88
+fi
 
-		n=$(dmesg | grep "Oops" | wc -l)
-		if [ $n -gt 0 ]; then
-			ret=99
-		fi
+n=$(dmesg | grep "Oops" | wc -l)
+if [ $n -gt 0 ]; then
+	ret=99
+fi
 
-		case $ret in
-		0)
-			echo "$s PASSED"
-			passed="$passed $s"
-			;;
-		1)
-			echo "$s SKIPPED (test framework out of resources or test should not be run)"
-			skipped="$skipped $s"
-			;;
-		2)
-			echo "$s FAILED"
-			failed="$failed $s"
-			cat ${TMP_FILE}
-			echo " "
-			dmesg
-			echo " "
-			rc=1
-			;;
-		3)
-			echo "$s SKIPPED (stressor out of resources)"
-			skipped="$skipped $s"
-			;;
-		4)
-			echo "$s SKIPPED (stressor not implemented on this arch)"
-			skipped="$skipped $s"
-			;;
-		5)
-			echo "$s SKIPPED (premature signal killed stressor)"
-			skipped="$skipped $s"
-			;;
-		6)
-			echo "$s SKIPPED (premature child exit, this is a bug in the stress test)"
-			skipped="$skipped $s"
-			;;
-		7)
-			echo "$s PASSED (child bogo-ops metrics were not accurate)"
-			passed="$passed $s"
-			;;
-		88)
-			echo "$s OOMED (out of memory kills detected)"
-			oomed="$oomed $s"
-			;;
-		99)
-			echo "$s FAILED (kernel oopsed)"
-			oopsed="$oopsed $s"
-			dmesg
-			echo " "
-			rc=1
-			;;
-		137)
-			echo "$s OOMED (out of memory kills detected)"
-			oomed="$oomed $s"
-			;;
-		*)
-			echo "$s BADRET (unknown return status $ret)"
-			badret="$badret $s"
-			dmesg
-			echo " "
-			;;
-		esac
-		rm -f ${TMP_FILE}
-	fi
-done
+case $ret in
+0)
+	echo "$STRESSOR PASSED"
+	passed="$passed $STRESSOR"
+	;;
+1)
+	echo "$STRESSOR SKIPPED (test framework out of resources or test should not be run)"
+	skipped="$skipped $STRESSOR"
+	;;
+2)
+	echo "$STRESSOR FAILED"
+	failed="$failed $STRESSOR"
+	cat ${TMP_FILE}
+	echo " "
+	dmesg
+	echo " "
+	rc=1
+	;;
+3)
+	echo "$STRESSOR SKIPPED (stressor out of resources)"
+	skipped="$skipped $STRESSOR"
+	;;
+4)
+	echo "$STRESSOR SKIPPED (stressor not implemented on this arch)"
+	skipped="$skipped $STRESSOR"
+	;;
+5)
+	echo "$STRESSOR SKIPPED (premature signal killed stressor)"
+	skipped="$skipped $STRESSOR"
+	;;
+6)
+	echo "$STRESSOR SKIPPED (premature child exit, this is a bug in the stress test)"
+	skipped="$skipped $STRESSOR"
+	;;
+7)
+	echo "$STRESSOR PASSED (child bogo-ops metrics were not accurate)"
+	passed="$passed $STRESSOR"
+	;;
+88)
+	echo "$STRESSOR OOMED (out of memory kills detected)"
+	oomed="$oomed $STRESSOR"
+	;;
+99)
+	echo "$STRESSOR FAILED (kernel oopsed)"
+	oopsed="$oopsed $STRESSOR"
+	dmesg
+	echo " "
+	rc=1
+	;;
+137)
+	echo "$STRESSOR OOMED (out of memory kills detected)"
+	oomed="$oomed $STRESSOR"
+	;;
+*)
+	echo "$STRESSOR BADRET (unknown return status $ret)"
+	badret="$badret $STRESSOR"
+	dmesg
+	echo " "
+	;;
+esac
+rm -f ${TMP_FILE}
 s2=$(secs_now)
 dur=$((s2 - $s1))
 
-#kill -9 $pid >& /dev/null
-#wait $pid
-
 echo " "
 echo "Summary:"
 echo "  Stressors run: $count"
@@ -397,9 +322,6 @@ echo "  Badret:  $(echo $badret | wc -w), $badret"
 echo " "
 echo "Tests took $dur seconds to run"
 
-swapoff -a ${SWPIMG}
-rm ${SWPIMG}
-
 if [ -e ${SYS_ZSWAP_ENABLED} ]; then
 	echo ${ORIGINAL_SYS_ZSWAP_SETTING} > ${SYS_ZSWAP_ENABLED}
 fi
diff --git a/ubuntu_stress_smoke_test/ubuntu_stress_smoke_test.py b/ubuntu_stress_smoke_test/ubuntu_stress_smoke_test.py
index 4775e65..2d78cf7 100644
--- a/ubuntu_stress_smoke_test/ubuntu_stress_smoke_test.py
+++ b/ubuntu_stress_smoke_test/ubuntu_stress_smoke_test.py
@@ -82,7 +82,7 @@ class ubuntu_stress_smoke_test(test.test):
         if os.uname()[1] == '202008-28164-ZCU106':
             raise error.TestFail('Test marked as failed for ZCU106 as requested by portias, dev test hang (LP: #1998738)')
         os.chdir(os.path.join(self.srcdir, 'stress-ng'))
-        cmd = '%s/ubuntu_stress_smoke_test.sh' % (self.bindir)
+        cmd = '%s/ubuntu_stress_single_smoke_test.sh %s' % (self.bindir, test_name)
         self.results = utils.system_output(cmd, retain_output=True)
 
 # vi:set ts=4 sw=4 expandtab syntax=python:
diff --git a/ubuntu_stress_smoke_test/ubuntu_stress_smoke_test_cleanup.sh b/ubuntu_stress_smoke_test/ubuntu_stress_smoke_test_cleanup.sh
new file mode 100755
index 0000000..6c43281
--- /dev/null
+++ b/ubuntu_stress_smoke_test/ubuntu_stress_smoke_test_cleanup.sh
@@ -0,0 +1,11 @@
+#!/bin/bash 
+
+#
+#  Garbage collect swap file
+#
+SWPIMG=$PWD/swap.img
+
+swapoff ${SWPIMG}
+rm ${SWPIMG}
+
+exit $rc
diff --git a/ubuntu_stress_smoke_test/ubuntu_stress_smoke_test_init.sh b/ubuntu_stress_smoke_test/ubuntu_stress_smoke_test_init.sh
new file mode 100755
index 0000000..0471fb7
--- /dev/null
+++ b/ubuntu_stress_smoke_test/ubuntu_stress_smoke_test_init.sh
@@ -0,0 +1,47 @@
+#!/bin/bash 
+
+SYS_ZSWAP_ENABLED=/sys/module/zswap/parameters/enabled
+
+#
+#  Try an ensure that this script and parent won't be oom'd
+#
+set_max_oom_level()
+{
+	if [ -e /proc/self/oom_score_adj ]; then
+		echo -900 > /proc/self/oom_score_adj
+		echo -900 > /proc/$PPID/oom_score_adj
+	elif [ -e /proc/self/oom_adj ]; then
+		echo -14 > /proc/self/oom_adj
+		echo -14 > /proc/$PPID/oom_adj
+	fi
+	#
+	# Ensure oom killer kills the stressor hogs rather
+	# than the wrong random process (e.g. autotest!)
+	#
+	if [ -e /proc/sys/vm/oom_kill_allocating_task ]; then
+		echo 0 > /proc/sys/vm/oom_kill_allocating_task
+	fi
+}
+
+#
+#  Always add 1GB of swap to ensure swapping is exercised
+#
+SWPIMG=$PWD/swap.img
+
+#
+#  Create 1GB swap file
+#
+swapoff ${SWPIMG} >& /dev/null
+fallocate -l 1G ${SWPIMG}
+if [ $? -ne 0 ]; then
+	echo "FAILED: Count not create 1GB swap file, file system:"
+	df
+	exit 1
+fi
+chmod 0600 ${SWPIMG}
+mkswap ${SWPIMG}
+swapon ${SWPIMG}
+
+set_max_oom_level
+sleep 15
+sync