Merge lp:~canonical-platform-qa/qa-jenkins-jobs/retrieve-tests-to-run-by-practitest-filter into lp:qa-jenkins-jobs

[Allan LeSage]

Retrieve test lists by PractTest filter.

Instead of using the sanity or regression lists in the ubuntu-system-tests repo, query PractiTest by filter and use tests lists retrieved.

It would be better not to have to use the number of the filter of course, will need a separate fix from ust to do that, adding sbalda and rhuddie as they're involved in.

You can see this working (albeit for a broken build with many test failures) here: https://platform-qa-jenkins.ubuntu.com/job/ust_rc-proposed_krillin_sanity_test_deux/6/consoleFull .

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:166
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/40/
Executed test runs:
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/3070
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-amd64-package/3070/artifact/work/output/*zip*/output.zip
    SUCCESS: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/3070
        deb: https://platform-qa-jenkins.ubuntu.com/job/build-wily-i386-package/3070/artifact/work/output/*zip*/output.zip
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/569/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/qa-jenkins-jobs-ci/40/rebuild

review: Approve (continuous-integration)

[Max Brustkern (nuclearbob)]

Makes sense to me.

review: Approve

[Richard Huddie (rhuddie)]

Changes seem ok to me!

review: Approve

[lp:~canonical-platform-qa/qa-jenkins-jobs/retrieve-tests-to-run-by-practitest-filter]

167. By Allan LeSage on 2016-08-17

Merge trunk.

=== modified file 'jobs/ubuntu-system-tests/ubuntu-system-tests-builder.sh'
--- jobs/ubuntu-system-tests/ubuntu-system-tests-builder.sh	2016-08-09 19:15:10 +0000
+++ jobs/ubuntu-system-tests/ubuntu-system-tests-builder.sh	2016-08-17 21:02:38 +0000
@@ -21,18 +21,34 @@
 EOF
 cat $CONFIG_PATH
 
+# let's work in a virtualenv
+virtualenv --system-site-package --python=python3 ve3
+. ve3/bin/activate
+
+export https_proxy=https://squid.internal:3128/
+
+rm -rf ubuntu-system-tests
+bzr branch $UBUNTU_SYSTEM_TESTS_BRANCH ubuntu-system-tests
+cd ubuntu-system-tests
+python3 setup.py develop
+
+query_practitest_for_tests_to_run_by_filter () {
+    python3 -m ubuntu_system_tests.run list --filters filter:$1 --config=$CONFIG_PATH
+}
+
 case "$TESTS_TO_RUN" in
-  "regression")
-    TESTS_TO_RUN=$(bzr cat "$UBUNTU_SYSTEM_TESTS_BRANCH/test_lists/regression.tests" | tr '\n' ' ')
-    ;;
-  "sanity")
-    TESTS_TO_RUN=$(bzr cat "$UBUNTU_SYSTEM_TESTS_BRANCH/test_lists/sanity.tests" | tr '\n' ' ')
-    ;;
-  "")
-    # FIXME: restore error when we're able to install Conditional Build Step plugin!
-    echo "No tests specified, PASS!" >&2
-    exit 0
-    ;;
+    # yes we magically know the filter numbers
+    "regression")
+	TESTS_TO_RUN=$(query_practitest_for_tests_to_run_by_filter 117776)
+	;;
+    "sanity")
+	TESTS_TO_RUN=$(query_practitest_for_tests_to_run_by_filter 63805)
+	;;
+    "")
+	# FIXME: restore error when we're able to install Conditional Build Step plugin!
+	echo "No tests specified, PASS!" >&2
+	exit 0
+	;;
 esac
 
 # Jenkins param is a long string, transform to multi-line to do remaining_tests math
@@ -52,8 +68,12 @@
     comm -13 <(successfully_run_tests) <(initial_tests_to_run);
 }
 
-recover_device_flash_and_run_tests () {
-  ./qa-jenkins-jobs/scripts/ubuntu-system-tests/run-ubuntu-system-tests.sh "$1"
+run_tests () {
+    #./qa-jenkins-jobs/scripts/ubuntu-system-tests/run-ubuntu-system-tests.sh "$1"
+    # setup device
+    python3 -m ubuntu_system_tests.run setup -n -d -p -u -v --config=$CONFIG_PATH
+    # run tests
+    python3 -m ubuntu_system_tests.run run -n -v --config=$CONFIG_PATH "$1"
 }
 
 RETRY=0;
@@ -62,5 +82,7 @@
     # need to switch back to spaces instead of newlines
     TESTS_TO_RUN="$(echo $(remaining_tests) | tr '\n' ' ')";
     echo "Trial number $RETRY with tests $TESTS_TO_RUN";
-    recover_device_flash_and_run_tests "$(echo "$TESTS_TO_RUN")"
+    run_tests "$(echo "$TESTS_TO_RUN")"
 done;
+
+deactivate # our virtualenv

=== removed file 'scripts/ubuntu-system-tests/run-ubuntu-system-tests.sh'
--- scripts/ubuntu-system-tests/run-ubuntu-system-tests.sh	2016-06-28 13:24:08 +0000
+++ scripts/ubuntu-system-tests/run-ubuntu-system-tests.sh	1970-01-01 00:00:00 +0000
@@ -1,21 +0,0 @@
-#!/bin/sh
-set -x
-set +e
-
-export TESTS="$1"
-
-# let's work in a virtualenv
-virtualenv --python=python3 ve3
-. ve3/bin/activate
-
-rm -rf ubuntu-system-tests
-bzr branch $UBUNTU_SYSTEM_TESTS_BRANCH ubuntu-system-tests
-cd ubuntu-system-tests
-
-python3 setup.py develop
-# setup device
-python3 -m ubuntu_system_tests.run setup -n -d -p -u -v --config=$CONFIG_PATH
-# run tests
-python3 -m ubuntu_system_tests.run run -n -v --config=$CONFIG_PATH "$TESTS"
-
-deactivate # our virtualenv