Merge lp:~doanac/lava-android-test/restricted-coremark into lp:lava-android-test

[Andy Doan]

Add support for restricted coremark bench

This adds the basic ability to run the coremark benchmark on
Android. Since coremark is a restricted program, the actual
binary is managed separate from this test. The script,
download_test.sh is used by coremark.py to download the proper
binary. Therefore, this test requires install options to be
passed to the install command for example:

   lava-android-test install -o "coremark_url=https://user:pass@1.1.1.1/coremark-linaro4.6" coremark

The above example would download from an HTTP password protected
directory the coremark-linaro4.6 binary.

[Paul Larson (pwlars)]

I think this looks fine to me, except we should add some documentation for the test (Yongqin has a branch in progress for this right now to add it to the others... it's pretty simple). I think at least the docstring in the test itself explaining what it is, and definitely mentioning that it does not include the coremark binaries which need to be obtained from somewhere else.

I cc'd the rest of the team and would like to get them to weigh in on this as well. The big issue that I'm concerned about I think is that it looks pretty inconvenient to have to create the binaries for coremark, host them somewhere, make sure we only run this in a private job once the ability to do that exists, and create all the jobs pointing to those binaries. I'd love it if lava already had a build step built into it that would accomodate something like this internally, but it doesn't right now. It might be worth considering if we could somehow handle the build part of this through Jenkins in the same style that we do ci testing. Something for you to think about I guess.

In any case, I wouldn't actually recommend running a job that uses this in lava until we land and install the rest of the privacy changes so that you can:
1. specify a private bundle stream to push the results to
2. make sure that nobody sees your user:password who is not in linaro - this will still be in the job log, and even the job json, but if the job is specifically marked private, it won't be visible to anyone outside of linaro.

[Andy Doan (doanac)]

On 03/07/2012 10:43 PM, Paul Larson wrote:
> I think this looks fine to me, except we should add some documentation for the test (Yongqin has a branch in progress for this right now to add it to the others... it's pretty simple). I think at least the docstring in the test itself explaining what it is, and definitely mentioning that it does not include the coremark binaries which need to be obtained from somewhere else.

sounds good.

> I cc'd the rest of the team and would like to get them to weigh in on this as well. The big issue that I'm concerned about I think is that it looks pretty inconvenient to have to create the binaries for coremark, host them somewhere, make sure we only run this in a private job once the ability to do that exists, and create all the jobs pointing to those binaries. I'd love it if lava already had a build step built into it that would accomodate something like this internally, but it doesn't right now. It might be worth considering if we could somehow handle the build part of this through Jenkins in the same style that we do ci testing. Something for you to think about I guess.

Its extremely inconvenient but that seems to be the nature of dealing
with these benchmark suites. The changing URL is pretty much a must,
because the actual coremark binary will always be changing.

It could be done from Jenkins. However, I'd have to create special
Android build job just for this, because the code base isn't in a public
git repository which Android requires. So that solution isn't really
trivial either.

[Andy Doan (doanac)]

I just discovered this won't work because AOSP builds don't include busybox. I have to come up with a new way for this test to be done.

review: Needs Fixing

[Paul Larson (pwlars)]

review: Abstain

Unmerged revisions

141. By Andy Doan on 2012-03-06

Add support for restricted coremark bench

This adds the basic ability to run the coremark benchmark on
Android. Since coremark is a restricted program, the actual
binary is managed separate from this test. The script,
download_test.sh is used by coremark.py to download the proper
binary. Therefore, this test requires install options to be
passed to the install command for example:

 lava-android-test install -o "coremark_url=https://user:pass@1.1.1.1/coremark-linaro4.6" coremark

The above example would download from an HTTP password protected
directory the coremark-linaro4.6 binary.

=== added file 'lava_android_test/test_definitions/coremark.py'
--- lava_android_test/test_definitions/coremark.py	1970-01-01 00:00:00 +0000
+++ lava_android_test/test_definitions/coremark.py	2012-03-06 23:49:17 +0000
@@ -0,0 +1,54 @@
+# Copyright (c) 2012 Linaro
+
+# Author: Linaro Validation Team <[e-mail]>
+#
+# This file is part of LAVA Android Test.
+#
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+import os
+import lava_android_test.config
+import lava_android_test.testdef
+
+test_name = 'coremark'
+
+config = lava_android_test.config.get_config()
+inst_dir = '%s/%s' % (config.installdir_android, test_name)
+
+options_file = "%s/install_options" % (inst_dir)
+coremark = '%s/coremark' % (inst_dir)
+
+curdir = os.path.realpath(os.path.dirname(__file__))
+script_host = '%s/download_test.sh' % (curdir)
+script_target = '%s/download_test.sh' % (inst_dir)
+
+INSTALL_STEPS_ADB_PRE = [
+    'push %s %s ' % (script_host, script_target),
+    'shell chmod 777 %s' % script_target,
+    'shell %s coremark %s %s' % (script_target, options_file, coremark),
+]
+
+ADB_SHELL_STEPS = [ '"%s 0x0 0x0 0x66 0 7 1 2000"' % coremark ]
+PATTERN = "^CoreMark 1.0\s*:\s+(?P<measurement>\d+\.\d+) / (?P<message>.*)"
+
+inst = lava_android_test.testdef.AndroidTestInstaller(
+                                steps_adb_pre=INSTALL_STEPS_ADB_PRE)
+run = lava_android_test.testdef.AndroidTestRunner(
+                                    adbshell_steps=ADB_SHELL_STEPS)
+parser = lava_android_test.testdef.AndroidTestParser(PATTERN,
+                                       appendall={'test_case_id':'Iterations/Sec'})
+testobj = lava_android_test.testdef.AndroidTest(testname=test_name,
+                                    installer=inst,
+                                    runner=run,
+                                    parser=parser)
 
=== added file 'lava_android_test/test_definitions/download_test.sh'
--- lava_android_test/test_definitions/download_test.sh	1970-01-01 00:00:00 +0000
+++ lava_android_test/test_definitions/download_test.sh	2012-03-06 23:49:17 +0000
@@ -0,0 +1,13 @@
+#!/system/bin/sh
+
+KEY=$1
+OPTIONS_FILE=$2
+OUTFILE=$3
+
+url=`busybox grep $KEY_url= $OPTIONS_FILE | busybox cut -d= -f2`
+if [ "a$url" == "a" ] ; then
+	echo "$KEY install command requires '-o ${KEY}_url=<url>'"
+	exit 1
+fi
+busybox wget -O $OUTFILE $url
+chmod 755 $OUTFILE