Merge lp:~chris.gagnon/phablet-tools/run-test-without-autopilot into lp:phablet-tools

[Chris Gagnon]

run non-autopilot tests with phablet-test-run using -x flag

use the -x flag to run tests that are not autopilot tests

example:
phablet-test-run -x -o . mir-integration-tests
phablet-test-run -x -o . run-mir-demo --mir-demo mir_demo_client_egltriangle
phablet-test-run -x -o . ls -la

[lp:~chris.gagnon/phablet-tools/run-test-without-autopilot]

225. By Chris Gagnon on 2013-11-12

don't use [[ -z use [ -z so sh will work as well as bash

226. By Chris Gagnon on 2013-11-12

wrap $EXTRA_COMMAND in quotes and use test so it's consistant with ANDROID_SERIAL

[Francis Ginther (fginther)]

Found some issues with my test case:
"./phablet-test-run -x 'ls > /tmp/ls.log' -o outdir -a /tmp/ls.log"

1) sh style conditional test:
./phablet-test-run: 202: ./phablet-test-run: [[: not found

2) The script still tries to collect the autopilot result file, when '-x' is specified this should be skipped:
remote object '/tmp/test_results.xml' does not exist

3) The usage line (line 30) does not include '-x'.

When 1 is fixed, the test runs as expected and the artifact is copied.

review: Needs Fixing

[lp:~chris.gagnon/phablet-tools/run-test-without-autopilot]

227. By Chris Gagnon on 2013-11-12

fix issues 2 and 3 in MP

[Chris Gagnon (chris.gagnon)]

All issues have been fixed

review: Needs Resubmitting

[lp:~chris.gagnon/phablet-tools/run-test-without-autopilot]

228. By Chris Gagnon on 2013-11-12

make it so you don't need to quote the command when using -x

229. By Chris Gagnon on 2013-11-12

fix accidentally deleted d in line 33

230. By Chris Gagnon on 2013-11-12

refactor function names so it's easier to read

[Francis Ginther (fginther)]

This solves the immediate need to run non-autopilot tests. I could argue that it would be better to not touch phablet-test-run and instead put effort into make a python version that is better at supporting both generic and autopilot tests, but I feel that should really be a separate effort.

review: Approve

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:230
http://jenkins.qa.ubuntu.com/job/phablet-tools-ci/242/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-trusty-amd64-ci/14
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-trusty-armhf-ci/14
    SUCCESS: http://jenkins.qa.ubuntu.com/job/phablet-tools-trusty-i386-ci/14

Click here to trigger a rebuild:
http://s-jenkins:8080/job/phablet-tools-ci/242/rebuild

review: Approve (continuous-integration)

[Sergio Schvezov (sergiusens)]

review: Approve

=== modified file 'phablet-test-run'
--- phablet-test-run	2013-10-24 17:45:53 +0000
+++ phablet-test-run	2013-11-12 21:29:04 +0000
@@ -20,6 +20,7 @@
 RESULTDIR=""
 ARTIFACTS=""
 
+EXECUTE_COMMAND=0
 USER=phablet
 NOSHELL=0
 RETVAL=0
@@ -27,7 +28,7 @@
 
 print_usage() {
     cat << EOF
-usage: $0 [-s SERIAL] [-s] [-n] [-v] [-p PACKAGENAME] [-c PACKAGE] [-o DIR] [-a ARTIFACT] testsuite
+usage: $0 [-s SERIAL] [-s] [-n] [-x] [-v] [-p PACKAGENAME] [-c PACKAGE] [-o DIR] [-a ARTIFACT] testsuite
 
   Run the specified testsuite on the device
 
@@ -37,6 +38,7 @@
   -n               Stop the shell during the test run
   -o DIR           Test report and artifacts output dir
   -a ARTIFACT      Artifact to fetch after the test, may be used multiple times (requires -o)
+  -x               Execute command line, will not use autopilot by default
   -v               Run autopilot with -v option for more verbose output
 EOF
 }
@@ -86,7 +88,21 @@
     exec_with_adb_user initctl start unity8
 }
 
-run_test() {
+get_returncode() {
+    while read line; do
+        set +e
+        RETVAL=$(echo $line | grep -Po '(?<=ADB_RC=)\d+')
+        if [ $? -eq 0 ] ; then
+            set -e
+            return $RETVAL
+        fi
+        echo $line
+        set -e
+    done
+}
+
+
+run_autopilot_test() {
     exec_with_adb chmod 666 /dev/uinput
     # adb shell always returns 0, so we have to do some hackery to get the
     # actual RC from the test
@@ -99,16 +115,24 @@
         else
             adb shell "$apbase $TESTSUITE\"; echo ADB_RC=\$?"
         fi
-    } | while read line; do
-        set +e
-        RETVAL=$(echo $line | grep -Po '(?<=ADB_RC=)\d+')
-        if [ $? -eq 0 ] ; then
-            set -e
-            return $RETVAL
+    } | get_returncode
+}
+
+run_cli_command() {
+    apbase="sudo -iu $USER bash -ic \"cd /home/phablet; $TESTSUITE"
+    # adb shell always returns 0, so we have to do some hackery to get the
+    # actual RC from the test
+    {
+        if [ "$RESULTDIR" ]; then
+            result_output=/tmp/test_results.txt
+            adb shell 'rm -rf $result_output'
+            adb shell "$apbase | tee $result_output\"; echo ADB_RC=\$?"
+            adb pull $result_output $RESULTDIR
+        else
+            adb shell "$apbase\"; echo ADB_RC=\$?"
         fi
-        echo $line
-        set -e
-    done
+    } | get_returncode
+
 }
 
 fetch_artifacts() {
@@ -116,7 +140,9 @@
         mkdir -p $RESULTDIR
         echo "***** Artifacts ($RESULTDIR) *****"
         set +e
-        adb pull /tmp/test_results.xml $RESULTDIR
+        if [ $EXECUTE_COMMAND -eq 0 ]; then
+            adb pull /tmp/test_results.xml $RESULTDIR
+        fi
         for artifact in $ARTIFACTS; do
            adb pull $artifact $RESULTDIR
         done
@@ -124,7 +150,7 @@
     fi
 }
 
-while getopts a:c:hno:p:s:v opt; do
+while getopts a:c:hnxo:p:s:v opt; do
     case $opt in
     a)
         ARTIFACTS="$ARTIFACTS $OPTARG"
@@ -148,6 +174,9 @@
     s)
         ANDROID_SERIAL=$OPTARG
         ;;
+    x)
+        EXECUTE_COMMAND=1
+        ;;
     v)
         VERBOSE="-v"
         ;;
@@ -156,7 +185,11 @@
 
 shift $((OPTIND - 1))
 
-TESTSUITE=$1
+TESTSUITE=$@
+if test -z "$TESTSUITE"; then
+    echo To run a test please specify a test suite
+    exit 0
+fi
 
 if test -z $ANDROID_SERIAL; then
     ANDROID_SERIAL=$(adb devices -l | grep usb | cut -f 1 -d " " | head -n1)
@@ -165,10 +198,6 @@
 
 adb_root
 
-if test -z $TESTSUITE; then
-    echo To run a test please specify a test suite
-    exit 0
-fi
 
 install_packages
 
@@ -177,7 +206,12 @@
     disable_shell
 fi
 
-run_test || RETVAL=1
+if [ $EXECUTE_COMMAND -eq 1 ]; then
+    echo "running $TESTSUITE"
+    run_cli_command || RETVAL=1
+else
+    run_autopilot_test || RETVAL=1
+fi
 
 fetch_artifacts