Merge lp:~pundiramit/lava-android-test/bctest into lp:lava-android-test

[Amit Pundir]

Adding binder test "bctest" available on Juice Android builds.

[Amit Pundir (pundiramit)]

Please verify if I have used the right PATTERN to parse the results.

PATTERN = "(?P<test_case_id>.*ioctl)\s(?P<result>(PASS|FAIL)?).*"

In case of success, STOUT is:
LAVA: (stdout) binder: ioctl PASS
LAVA: (stdout) binder_write: ioctl PASS

In case of failure, STDOUT is:
LAVA: (stdout) binder: ioctl FAIL (invalid argument)
LAVA: (stdout) binder_write: ioctl FAIL (invalid argument)

[Antonio Terceiro (terceiro)]

On Thu, Feb 21, 2013 at 03:17:21PM -0000, Amit Pundir wrote:
> Please verify if I have used the right PATTERN to parse the results.
>
> PATTERN = "(?P<test_case_id>.*ioctl)\s(?P<result>(PASS|FAIL)?).*"
>
> In case of success, STOUT is:
> LAVA: (stdout) binder: ioctl PASS
> LAVA: (stdout) binder_write: ioctl PASS
>
> In case of failure, STDOUT is:
> LAVA: (stdout) binder: ioctl FAIL (invalid argument)
> LAVA: (stdout) binder_write: ioctl FAIL (invalid argument)

With this expression, test_case_id will be something like
'LAVA: (stdout) binder: ioctl'

Is that what you want? It seems to me that the 'LAVA: (stdout)'
is not useful to have

If you try

PATTERN = "LAVA: \(stdout\) (?P<test_case_id>.*ioctl)\s(?P<result>(PASS|FAIL)?).*"

then you get those bits out (i.e. test_case_id will be 'binder: ioctl')

--
Antonio Terceiro
Software Engineer - Linaro
http://www.linaro.org

[Yongqin Liu (liuyq0307)]

> On Thu, Feb 21, 2013 at 03:17:21PM -0000, Amit Pundir wrote:
> > Please verify if I have used the right PATTERN to parse the results.
> >
> > PATTERN = "(?P<test_case_id>.*ioctl)\s(?P<result>(PASS|FAIL)?).*"
> >
> > In case of success, STOUT is:
> > LAVA: (stdout) binder: ioctl PASS
> > LAVA: (stdout) binder_write: ioctl PASS
> >
> > In case of failure, STDOUT is:
> > LAVA: (stdout) binder: ioctl FAIL (invalid argument)
> > LAVA: (stdout) binder_write: ioctl FAIL (invalid argument)
>
> With this expression, test_case_id will be something like
> 'LAVA: (stdout) binder: ioctl'

I guess this output is gotten from lava-android-test command.
LAVA: (stdout) should be the output of lava-android-test framework.

the output of bctest is something like this I think:
binder: ioctl PASS
binder_write: ioctl PASS

So for this test it's ok for me.

Hi, Amit
but I want to know why you don't implement it with lava-test-shell?

And if you still want to use lava-android-test, then I suggest creating a file like:
http://bazaar.launchpad.net/~linaro-validation/lava-android-test/trunk/view/head:/lava_android_test/test_definitions/commands/example.py

> Is that what you want? It seems to me that the 'LAVA: (stdout)'
> is not useful to have
>
> If you try
>
> PATTERN = "LAVA: \(stdout\)
> (?P<test_case_id>.*ioctl)\s(?P<result>(PASS|FAIL)?).*"
>
> then you get those bits out (i.e. test_case_id will be 'binder: ioctl')
>
> --
> Antonio Terceiro
> Software Engineer - Linaro
> http://www.linaro.org

[Amit Pundir (pundiramit)]

On 25 February 2013 20:01, Yongqin Liu <email address hidden> wrote:

> > On Thu, Feb 21, 2013 at 03:17:21PM -0000, Amit Pundir wrote:
> > > Please verify if I have used the right PATTERN to parse the results.
> > >
> > > PATTERN = "(?P<test_case_id>.*ioctl)\s(?P<result>(PASS|FAIL)?).*"
> > >
> > > In case of success, STOUT is:
> > > LAVA: (stdout) binder: ioctl PASS
> > > LAVA: (stdout) binder_write: ioctl PASS
> > >
> > > In case of failure, STDOUT is:
> > > LAVA: (stdout) binder: ioctl FAIL (invalid argument)
> > > LAVA: (stdout) binder_write: ioctl FAIL (invalid argument)
> >
> > With this expression, test_case_id will be something like
> > 'LAVA: (stdout) binder: ioctl'
>
> I guess this output is gotten from lava-android-test command.
> LAVA: (stdout) should be the output of lava-android-test framework.
>
> the output of bctest is something like this I think:
> binder: ioctl PASS
> binder_write: ioctl PASS
>

Yes as mentioned by Yongqin, "LAVA: (stdout) " is an output from lava
framework and it get filtered out internally. Test result bundle will look
like:

...<snip>...
"test_results": [
        {
          "log_filename": "stdout.log",
          "log_lineno": 1,
          "test_case_id": "binder_ioctl",
          "result": "pass"
        },
        {
          "log_filename": "stdout.log",
          "log_lineno": 2,
          "test_case_id": "binder_write_ioctl",
          "result": "pass"
        }
      ],
...<snip>...

>
> So for this test it's ok for me.
>
> Hi, Amit
> but I want to know why you don't implement it with lava-test-shell?
>

I have also submitted a patch for bctest implementation with
lava-test-shell to Senthil.

>
> And if you still want to use lava-android-test, then I suggest creating a
> file like:
>
> http://bazaar.launchpad.net/~linaro-validation/lava-android-test/trunk/view/head:/lava_android_test/test_definitions/commands/example.py

If this is the way to go then I'll implement it this way. But I do not see
how can I use "DEFAULT_OPTIONS" feature in any of the examples there?

>
> > Is that what you want? It seems to me that the 'LAVA: (stdout)'
> > is not useful to have
> >
> > If you try
> >
> > PATTERN = "LAVA: \(stdout\)
> > (?P<test_case_id>.*ioctl)\s(?P<result>(PASS|FAIL)?).*"
> >
> > then you get those bits out (i.e. test_case_id will be 'binder: ioctl')
> >
> > --
> > Antonio Terceiro
> > Software Engineer - Linaro
> > http://www.linaro.org
> --
>
> https://code.launchpad.net/~pundiramit/lava-android-test/bctest/+merge/149857
> You are the owner of lp:~pundiramit/lava-android-test/bctest.
>

[Yongqin Liu (liuyq0307)]

> On 25 February 2013 20:01, Yongqin Liu <email address hidden> wrote:
>
> > > On Thu, Feb 21, 2013 at 03:17:21PM -0000, Amit Pundir wrote:
> > > > Please verify if I have used the right PATTERN to parse the results.
> > > >
> > > > PATTERN = "(?P<test_case_id>.*ioctl)\s(?P<result>(PASS|FAIL)?).*"
> > > >
> > > > In case of success, STOUT is:
> > > > LAVA: (stdout) binder: ioctl PASS
> > > > LAVA: (stdout) binder_write: ioctl PASS
> > > >
> > > > In case of failure, STDOUT is:
> > > > LAVA: (stdout) binder: ioctl FAIL (invalid argument)
> > > > LAVA: (stdout) binder_write: ioctl FAIL (invalid argument)
> > >
> > > With this expression, test_case_id will be something like
> > > 'LAVA: (stdout) binder: ioctl'
> >
> > I guess this output is gotten from lava-android-test command.
> > LAVA: (stdout) should be the output of lava-android-test framework.
> >
> > the output of bctest is something like this I think:
> > binder: ioctl PASS
> > binder_write: ioctl PASS
> >
>
> Yes as mentioned by Yongqin, "LAVA: (stdout) " is an output from lava
> framework and it get filtered out internally. Test result bundle will look
> like:
>
> ...<snip>...
> "test_results": [
> {
> "log_filename": "stdout.log",
> "log_lineno": 1,
> "test_case_id": "binder_ioctl",
> "result": "pass"
> },
> {
> "log_filename": "stdout.log",
> "log_lineno": 2,
> "test_case_id": "binder_write_ioctl",
> "result": "pass"
> }
> ],
> ...<snip>...
>
>
> >
> > So for this test it's ok for me.
> >
> > Hi, Amit
> > but I want to know why you don't implement it with lava-test-shell?
> >
>
> I have also submitted a patch for bctest implementation with
> lava-test-shell to Senthil.
>
>
> >
> > And if you still want to use lava-android-test, then I suggest creating a
> > file like:
> >
> > http://bazaar.launchpad.net/~linaro-validation/lava-android-
> test/trunk/view/head:/lava_android_test/test_definitions/commands/example.py
>
>
> If this is the way to go then I'll implement it this way. But I do not see
> how can I use "DEFAULT_OPTIONS" feature in any of the examples there?
For support the DEFAULT_OPTIONS in the commands, there need some more modification on other files.
Well, it's OK now, I approve this.

review: Approve

=== added file 'lava_android_test/test_definitions/bctest.py'
--- lava_android_test/test_definitions/bctest.py	1970-01-01 00:00:00 +0000
+++ lava_android_test/test_definitions/bctest.py	2013-02-21 15:13:24 +0000
@@ -0,0 +1,49 @@
+# Copyright (c) 2012 Linaro
+
+# Author: Linaro Android Team <[e-mail]>
+#
+# This file is part of LAVA Android Test.
+#
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""
+binder test "bctest" that is pre-intalled on Juice Android builds.
+
+**URL:** https://wiki.linaro.org/Internal/Projects/Juice
+
+**Default options:** "publish 1"
+"""
+
+import lava_android_test.config
+import lava_android_test.testdef
+
+test_name = 'bctest'
+
+DEFAULT_OPTIONS='publish 1'
+
+INSTALL_STEPS_ADB_PRE = []
+ADB_SHELL_STEPS = ['bctest $(OPTIONS)']
+PATTERN = "(?P<test_case_id>.*ioctl)\s(?P<result>(PASS|FAIL)?).*"
+
+inst = lava_android_test.testdef.AndroidTestInstaller(
+                                steps_adb_pre=INSTALL_STEPS_ADB_PRE)
+run = lava_android_test.testdef.AndroidTestRunner(
+                                    adbshell_steps=ADB_SHELL_STEPS)
+parser = lava_android_test.testdef.AndroidTestParser(PATTERN)
+testobj = lava_android_test.testdef.AndroidTest(testname=test_name,
+                                    installer=inst,
+                                    runner=run,
+                                    parser=parser,
+                                    default_options=DEFAULT_OPTIONS)