Merge ~paride/curtin:feature/jenkins-runner-shuf into curtin:master

[Paride Legovini]

tools/jenkins-runner: shuffle test-cases to randomize load

Enabling jenkins-runner to enumerate all of the tests to be
run and then run them through 'shuf' which will randomize
each test class for selection by the parallel threads of
execution. This will reduce the likelihood that we have
four executions of a Bcache or Raid or other heavy storage
which is what we get with current module level parallelism.

This behavior can be disabled by exporting:

CURTIN_VMTEST_SHUFFLE_TESTS=0

It is enabled by default.

[Paride Legovini (paride)]

This is

https://code.launchpad.net/~raharper/curtin/+git/curtin/+merge/381792

rebased and with the typo fixed. The original MP already had a +1 from Chad and me, and wasn't merged just because of the typo.

[Dan Watkins (oddbloke)]

review: Approve

[Server Team CI bot (server-team-bot)]

review: Approve (continuous-integration)

diff --git a/tools/jenkins-runner b/tools/jenkins-runner
index 253b722..375dc3c 100755
--- a/tools/jenkins-runner
+++ b/tools/jenkins-runner
@@ -5,6 +5,7 @@ topdir="${CURTIN_VMTEST_TOPDIR:-${WORKSPACE:-$PWD}/output}"
 pkeep=${CURTIN_VMTEST_KEEP_DATA_PASS:-logs,collect}
 fkeep=${CURTIN_VMTEST_KEEP_DATA_FAIL:-logs,collect}
 reuse=${CURTIN_VMTEST_REUSE_TOPDIR:-0}
+shuffle=${CURTIN_VMTEST_SHUFFLE_TESTS:-1}
 declare -i ltimeout=${CURTIN_VMTEST_IMAGE_LOCK_TIMEOUT:-"-1"}
 export CURTIN_VMTEST_TAR_DISKS=${CURTIN_VMTEST_TAR_DISKS:-0}
 export CURTIN_VMTEST_REUSE_TOPDIR=$reuse
@@ -14,6 +15,7 @@ export CURTIN_VMTEST_KEEP_DATA_FAIL=$fkeep
 export CURTIN_VMTEST_TOPDIR="$topdir"
 export CURTIN_VMTEST_LOG="${CURTIN_VMTEST_LOG:-$topdir/debug.log}"
 export CURTIN_VMTEST_PARALLEL=${CURTIN_VMTEST_PARALLEL:-0}
+export CURTIN_VMTEST_SHUFFLE_TESTS=$shuffle
 export IMAGE_DIR=${IMAGE_DIR:-/srv/images}
 
 # empty TGT_* variables in current env to avoid killing a pid we didn't start.
@@ -50,6 +52,15 @@ if [ "$reuse" != "1" ]; then
     mkdir -p "$topdir" || fail "failed mkdir $topdir"
 fi
 
+# Use 'shuf' to randomize test case execution order
+if [ "$shuffle" == "1" ]; then
+    SHUFFLE="shuf"
+else
+    # when disabled just repeat the input to output
+    SHUFFLE="tee"
+fi
+
+
 start_s=$(date +%s)
 parallel=${CURTIN_VMTEST_PARALLEL}
 ntfilters=( )
@@ -83,9 +94,10 @@ if [ "${#tests[@]}" -ne 0 -a "${#ntfilters[@]}" -ne 0 ]; then
     error "test arguments provided were: ${#tests[*]}"
     fail
 elif [ "${#tests[@]}" -eq 0 -a "${#ntfilters[@]}" -eq 0 ]; then
-    tests=( tests/vmtests )
+    # run filter without args to enumerate all tests and maybe shuffle
+    tests=( $(./tools/vmtest-filter | ${SHUFFLE}) )
 elif [ "${#ntfilters[@]}" -ne 0 ]; then
-    tests=( $(./tools/vmtest-filter "${ntfilters[@]}") )
+    tests=( $(./tools/vmtest-filter "${ntfilters[@]}" | ${SHUFFLE}) )
     if [ "${#tests[@]}" -eq 0 ]; then
         error "Failed to find any tests with filter(s): \"${ntfilters[*]}\""
         fail "Try testing filters with: ./tools/vmtest-filter ${ntfilters[*]}"