Merge lp:~liuyq0307/lava-android-test/workload into lp:lava-android-test

[Yongqin Liu]

add test for workload on b.L system
according to the talk on IRC, change to use local git url as default, also add support for specification of git url

[Andy Doan (doanac)]

Before going too far into this, I wanted to mention my general thought:

This seems to be using ADB to generate workload intensive stuff. I think
this is probably the wrong way to be doing this. I think the bigLITTLE
switcher tests is a better approach where its written entirely for the
target. This seems way better, and will ultimately better fit with our
lava-test-shell efforts.

On 12/06/2012 07:06 AM, Yongqin Liu wrote:
> + git_url="ssh://<email address hidden>/srv/linaro-private.git.linaro.org/people/bhoj/workload-automation.git"

That's not going to work for anyone but you.

[Yongqin Liu (liuyq0307)]

> Before going too far into this, I wanted to mention my general thought:
>
> This seems to be using ADB to generate workload intensive stuff. I think
> this is probably the wrong way to be doing this. I think the bigLITTLE
> switcher tests is a better approach where its written entirely for the
> target. This seems way better, and will ultimately better fit with our
> lava-test-shell efforts.

not see the detail implementation of workload test, but it seems to download
the apk of nbench, and install it to device, and do other things.

and for this workload test, are all the steps need to be run on host side
able to be done with lava-test-shell?
  git clone
  update ip in config
  run python workload_setup_dependencies.py
  python workload.py ${config_file} ${outputdir}/

> On 12/06/2012 07:06 AM, Yongqin Liu wrote:
> > + git_url="ssh://<email address hidden>/srv/linaro-
> private.git.linaro.org/people/bhoj/workload-automation.git"
>
> That's not going to work for anyone but you.
Or, what you mean is to rewrite the workload test via lava-test-shell?

[Andy Doan (doanac)]

On 12/06/2012 09:02 PM, Yongqin Liu wrote:
>> Before going too far into this, I wanted to mention my general thought:
>>
>> This seems to be using ADB to generate workload intensive stuff. I think
>> this is probably the wrong way to be doing this. I think the bigLITTLE
>> switcher tests is a better approach where its written entirely for the
>> target. This seems way better, and will ultimately better fit with our
>> lava-test-shell efforts.
>
> not see the detail implementation of workload test, but it seems to download
> the apk of nbench, and install it to device, and do other things.

I think this could be a quick patch to lava-test-shell. We have "deps"
support for Ubuntu. We could do similar for Android.

> and for this workload test, are all the steps need to be run on host side
> able to be done with lava-test-shell?
> git clone
> update ip in config
> run python workload_setup_dependencies.py
> python workload.py ${config_file} ${outputdir}/

We don't have to do the exact steps. git-clone is supported. not sure
about the ip update. The setup deps could be done in an equivalent
manner. Instead of thinking "how do we do this on host in python", I'm
saying ask yourself - "how could I do this as a script that ran on the
target"? I don't think it would be that hard.

If you absolutely think I'm crazy, then continue with this approach, but
doing a "workload" test and requiring ADB seems like a bad idea.

>> On 12/06/2012 07:06 AM, Yongqin Liu wrote:
>>> + git_url="ssh://<email address hidden>/srv/linaro-
>> private.git.linaro.org/people/bhoj/workload-automation.git"
>>
>> That's not going to work for anyone but you.
> Or, what you mean is to rewrite the workload test via lava-test-shell?

No - i just meant that the git url you had there will only work for you
with your ssh keys :)

[vishal (vishalbhoj)]

On 7 December 2012 22:34, Andy Doan <email address hidden> wrote:

> On 12/06/2012 09:02 PM, Yongqin Liu wrote:
> >> Before going too far into this, I wanted to mention my general thought:
> >>
> >> This seems to be using ADB to generate workload intensive stuff. I think
> >> this is probably the wrong way to be doing this. I think the bigLITTLE
> >> switcher tests is a better approach where its written entirely for the
> >> target. This seems way better, and will ultimately better fit with our
> >> lava-test-shell efforts.
> >
> > not see the detail implementation of workload test, but it seems to
> download
> > the apk of nbench, and install it to device, and do other things.
>
> I think this could be a quick patch to lava-test-shell. We have "deps"
> support for Ubuntu. We could do similar for Android.
>
> > and for this workload test, are all the steps need to be run on host side
> > able to be done with lava-test-shell?
> > git clone
> > update ip in config
> > run python workload_setup_dependencies.py
> > python workload.py ${config_file} ${outputdir}/
>
> We don't have to do the exact steps. git-clone is supported. not sure
> about the ip update. The setup deps could be done in an equivalent
> manner. Instead of thinking "how do we do this on host in python", I'm
> saying ask yourself - "how could I do this as a script that ran on the
> target"? I don't think it would be that hard.
>

I get your point here. But the whole benchmark is written by ARM to run
over adb + they even take control of serial terminal similar to LAVA( we
don't execute that path but rely on LAVA to reboot etc).

>
> If you absolutely think I'm crazy, then continue with this approach, but
> doing a "workload" test and requiring ADB seems like a bad idea.
>

adb is used only to transfer the files before and after the benchmark is
run and not while the benchmark is running.

>
> >> On 12/06/2012 07:06 AM, Yongqin Liu wrote:
> >>> + git_url="ssh://
> <email address hidden>/srv/linaro-
> >> private.git.linaro.org/people/bhoj/workload-automation.git"
> >>
> >> That's not going to work for anyone but you.
> > Or, what you mean is to rewrite the workload test via lava-test-shell?
>
> No - i just meant that the git url you had there will only work for you
> with your ssh keys :)
>
>
> --
>
> https://code.launchpad.net/~liuyq0307/lava-android-test/workload/+merge/138448
> You are requested to review the proposed merge of
> lp:~liuyq0307/lava-android-test/workload into lp:lava-android-test.
>

[Andy Doan (doanac)]

On 12/07/2012 11:13 AM, vishal wrote:
> I get your point here. But the whole benchmark is written by ARM to run
> over adb + they even take control of serial terminal similar to LAVA( we
> don't execute that path but rely on LAVA to reboot etc).

So use lava-android-test.

Its a little frustrating though:

* Android guys complained for over a year about "no blackbox testing".

* Benchmark guys complained about all Android results being invalid
because of the same basic thing.

We now have a solution in place, and you guys don't seem to be
interested in using it.

>> >
>> >If you absolutely think I'm crazy, then continue with this approach, but
>> >doing a "workload" test and requiring ADB seems like a bad idea.
>> >
> adb is used only to transfer the files before and after the benchmark is
> run and not while the benchmark is running.

So that could be done in lava-test-shell :)

[Yongqin Liu (liuyq0307)]

> On 12/07/2012 11:13 AM, vishal wrote:
> > I get your point here. But the whole benchmark is written by ARM to run
> > over adb + they even take control of serial terminal similar to LAVA( we
> > don't execute that path but rely on LAVA to reboot etc).
>
> So use lava-android-test.
If this means you aggree this branch for now, please approve this MP.
>
> Its a little frustrating though:
>
> * Android guys complained for over a year about "no blackbox testing".
>
what I know about this is for the network test problem which the adb connection that lava-android-test relies on.
> * Benchmark guys complained about all Android results being invalid
> because of the same basic thing.
not know the details, but from the what I can guess it's not a problem lava can resolve,
normally benchmark tools only displayed the result with GUI where command tools are not able to get.
I don't think lava-test-shell can help on this, if it can do, I am glad to know how it implemented.

> We now have a solution in place, and you guys don't seem to be
> interested in using it.
Actually at least I am interesting in it,
but if you ask me to reimplement an existing tool which works well with adb connection,
then I will want to know why, and know how much time it will take to reimplement.

> >> >If you absolutely think I'm crazy, then continue with this approach, but
> >> >doing a "workload" test and requiring ADB seems like a bad idea.
> >> >
> > adb is used only to transfer the files before and after the benchmark is
> > run and not while the benchmark is running.
>
> So that could be done in lava-test-shell :)

[Andy Doan (doanac)]

On 12/10/2012 08:18 AM, Yongqin Liu wrote:
>> On 12/07/2012 11:13 AM, vishal wrote:
>>> I get your point here. But the whole benchmark is written by ARM to run
>>> over adb + they even take control of serial terminal similar to LAVA( we
>>> don't execute that path but rely on LAVA to reboot etc).
>>
>> So use lava-android-test.
> If this means you aggree this branch for now, please approve this MP.

yes. it seems fine.

NOTE: we still have to iron out the private git access part of this issue.

[Andy Doan (doanac)]

Actually wait a bit. I think your git url still won't work

[Yongqin Liu (liuyq0307)]

> Actually wait a bit. I think your git url still won't work
I will change it to use the local git repository tomorrow.

And BTW, I will need you to help to deploy this modification tomorrow then.
So in the design of lava-test-shell, can you make it possible that the deployment isno need when add new test?

[Andy Doan (doanac)]

On 12/10/2012 09:32 AM, Yongqin Liu wrote:
>> Actually wait a bit. I think your git url still won't work
> I will change it to use the local git repository tomorrow.
>
> And BTW, I will need you to help to deploy this modification tomorrow then.
> So in the design of lava-test-shell, can you make it possible that the deployment isno need when add new test?

Yep - lava-test-shell allows you to control your own test definitions
(and even keep them in git if you'd like)

[Andy Doan (doanac)]

review: Approve

=== modified file 'lava_android_test/provider.py'
--- lava_android_test/provider.py	2012-07-10 06:34:06 +0000
+++ lava_android_test/provider.py	2012-12-11 16:47:25 +0000
@@ -266,15 +266,13 @@
         test_sh_path = '%s/%s' % (mod.curdir, f_name)
 
         HOST_SHELL_STEPS = ['bash %s -s $(SERIAL) $(OPTIONS)' % test_sh_path]
-        PATTERN = ("^\s*(?P<test_case_id>\S+)="
-                   "(?P<result>(pass|fail|ok|ng|true|false|skip|done))\s*$")
 
         testobj = self.gen_testobj(
                     testname=test_name,
                     installer=testdef.AndroidTestInstaller(),
                     runner=testdef.AndroidTestRunner(
                                     steps_host_pre=HOST_SHELL_STEPS),
-                    parser=testdef.AndroidTestParser(PATTERN),
+                    parser=testdef.AndroidSimpleTestParser(),
                     adb=ADB(serial))
         return testobj
 

=== added file 'lava_android_test/test_definitions/hostshells/workload.sh'
--- lava_android_test/test_definitions/hostshells/workload.sh	1970-01-01 00:00:00 +0000
+++ lava_android_test/test_definitions/hostshells/workload.sh	2012-12-11 16:47:25 +0000
@@ -0,0 +1,147 @@
+#!/bin/bash
+# Copyright (c) 2012 Linaro
+
+# Author: Linaro Validation Team <[e-mail]>
+#
+# This file is part of LAVA Android Test.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+function parse_argv() {
+    # Parse command line arguments
+    while test -n "$1"; do
+        case "$1" in
+            --serial|-s)
+                SERIAL="$2"
+                if [ -n "${SERIAL}" ]; then
+                    shift 2
+                else
+                    show_usage
+                    exit 1
+                fi
+                ;;
+            --config|-c)
+                CONFIG="$2"
+                if [ -n "${CONFIG}" ]; then
+                    shift 2
+                else
+                    show_usage
+                    exit 1
+                fi
+                ;;
+            --git|-g)
+                GIT_URL="$2"
+                if [ -n "${GIT_URL}" ]; then
+                    shift 2
+                else
+                    show_usage
+                    exit 1
+                fi
+                ;;
+            --help|-h)
+                show_usage
+                exit 1
+                ;;
+            *)
+                if [ -n "${OPTIONS}" ]; then
+                    OPTIONS="${OPTIONS} $1"
+                else
+                    OPTIONS="$1"
+                fi
+                shift
+                ;;
+        esac
+    done
+}
+
+function show_usage(){
+    # Display the usage line
+    echo "Usage $(basename $0) [--serial|-s <serial>] [--config|-c <config_file>] [--git|g <git-url>] <other-option>"
+    echo "Usage $(basename $0) [--help|-h]"
+}
+
+function parse_output_result(){
+    result_f=${1}
+    if [ ! -f "${1}" ]; then
+        echo "There is no result file output/results.csv"
+        return
+    fi
+
+    file_tmp=${result_f}.tmp
+    sed 's/ /_/g'  ${result_f} > ${file_tmp}
+    keys=`head -n 1 ${file_tmp}`
+    values=`tail -n 1 ${file_tmp}`
+    for ((i=1; i<21; i++)); do
+        key=`echo ${keys}|cut -d , -f ${i}|sed 's/\r//'`
+        value=`echo ${values}|cut -d , -f ${i}|sed 's/\r//'`
+
+       echo ${value}|grep -P '^[.\d]+$' &>/dev/null
+       if [ $? -ne 0 ]; then
+            key="${key}_${value}"
+            value="true"
+        fi
+        echo ${key}=${value}
+    done
+    rm -f ${file_tmp}
+}
+
+function main(){
+    local_git="file:///home/bhoj/workload-automation.git"
+    branch="lava"
+    outputdir="outputdir"
+    result="${outputdir}/result.csv"
+
+    parse_argv "$@"
+
+    config_file="config.csv"
+    if [ -n "${CONFIG}" ]; then
+        config_file="${CONFIG}"
+    fi
+
+    if [ -n "${GIT_URL}" ]; then
+        git_url="${GIT_URL}"
+    else
+        git_url="${local_git}"
+    fi
+
+    git clone "${git_url}" -b ${branch}
+    if [ $? -ne 0 ]; then
+        echo "Failed to clone git repository: ${git_url}"
+        exit 1
+    fi
+    ip=`echo ${SERIAL}|sed 's/:5555//'`
+    cd "workload-automation"
+
+    #update the ip address and patch config.csv file
+    sed -i "s/192.168.1.38/${ip}/g" workload_config.py
+    sed -i "s/,1200000//g" "${config_file}" #for config_a15.csv and config.csv
+    sed -i "s/,1000000//g" "${config_file}" #for config_a7.csv
+
+    python workload_setup_dependencies.py
+    if [ $? -ne 0 ]; then
+        echo "Failed to run workload_setup_dependencies.py"
+        exit 1
+    fi
+
+    rm -fr ${outputdir}
+    python workload.py ${config_file} ${outputdir}/
+    if [ $? -ne 0 ]; then
+        echo "Failed to run workload.py config.csv outputdir/"
+        exit 1
+    fi
+    parse_output_result ${result}
+}
+
+main "$@"
+

=== modified file 'lava_android_test/testdef.py'
--- lava_android_test/testdef.py	2012-08-17 07:36:25 +0000
+++ lava_android_test/testdef.py	2012-12-11 16:47:25 +0000
@@ -429,8 +429,8 @@
     def real_parse(self, result_filename='stdout.log',
               output_filename='stdout.log', test_name=''):
         """Using the pattern to do the real parse operation
-        
-        generate the test_results elements from the result file by parsing 
+
+        generate the test_results elements from the result file by parsing
         with the pattern specified.
         """
         if not self.pattern:
@@ -599,6 +599,38 @@
         self.fixmeasurements()
         self.fixids()
 
+class AndroidSimpleTestParser(AndroidTestParser):
+
+    def real_parse(self, result_filename='stdout.log',
+              output_filename='stdout.log', test_name=''):
+        self.res_pattern = ("^\s*(?P<test_case_id>.+?)\s*="
+                   "\s*(?P<result>(pass|fail|ok|ng|true|false|skip|done))\s*$")
+        self.measurement_pattern = ("^\s*(?P<test_case_id>.+?)\s*="
+                        "\s*(?P<measurement>[\.\d]+)\s*$")
+        self.measurement_units_pattern = ("^\s*(?P<test_case_id>.+?)\s*="
+                        "\s*(?P<measurement>[\.\d]+)\s+(?P<units>\S+)\s*$")
+
+        res_pat = re.compile(self.res_pattern)
+        measurement_pat = re.compile(self.measurement_pattern)
+        measurement_units_pat = re.compile(self.measurement_units_pattern)
+
+        with open(output_filename, 'r') as stream:
+            for lineno, line in enumerate(stream, 1):
+                match = res_pat.search(line)
+                if not match:
+                    match = measurement_pat.search(line)
+                    if not match:
+                        match = measurement_units_pat.search(line)
+                        if not match:
+                            continue
+                data = match.groupdict()
+                data["log_filename"] = result_filename
+                data["log_lineno"] = lineno
+                if data.get('result') is None:
+                    data['result'] = 'pass'
+
+                self.results['test_results'].append(data)
+
 
 def _run_steps_host(steps=[], serial=None, option=None, resultsdir=None):
     adb = ADB(serial)