Merge lp:~liuyq0307/lava-android-test/support-monkeyrunner into lp:lava-android-test

[Yongqin Liu]

add run-monkeyrunner sub command to make support monkeyrunner test easily.

[Zygmunt Krynicki (zyga)]

It's pretty hard to review this as I'm not that fluent in the android parts of lava.

287 === added file 'lava_android_test/repository.py'

We already have lava-vcs which supports git and bzr through single API. Perhaps it would be better to use that instead?

[Yongqin Liu (liuyq0307)]

On 25 April 2012 21:17, Zygmunt Krynicki <email address hidden>wrote:

> It's pretty hard to review this as I'm not that fluent in the android
> parts of lava.
>
> 287 === added file 'lava_android_test/repository.py'
>
> We already have lava-vcs which supports git and bzr through single API.
> Perhaps it would be better to use that instead?
>
This component is great.

Can it be used now?
And just one look, it has many files, seems a little complicate.
(Maybe it is easy to be used)

I'd like to change to use it after we merged it to lava-core

[Zygmunt Krynicki (zyga)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

W dniu 25.04.2012 16:08, Yongqin Liu pisze:
> On 25 April 2012 21:17, Zygmunt Krynicki
> <email address hidden>wrote:
>
>> It's pretty hard to review this as I'm not that fluent in the
>> android parts of lava.
>>
>> 287 === added file 'lava_android_test/repository.py'
>>
>> We already have lava-vcs which supports git and bzr through
>> single API. Perhaps it would be better to use that instead?
>>
> This component is great.
>
> Can it be used now?

Yes

> And just one look, it has many files, seems a little complicate.
> (Maybe it is easy to be used)

It's pretty easy. The abstraction has three objects: IVCS (like git
and bzr), IRemoteBranch which has limited functionality but allows you
to create a local copy / branch / clone and ILocalBranch that gives
you extra stuff that I needed to work on lava. If you need more then
we can easily extend it.

PS: look at lava/vcs/interfaces.py - this is what you can do on both
git and bzr.

from lava.vcs.adapters import get_vcs
git = get_vcs("git")
remote_branch = git.open_remote("git://...")
local_branch = remote_branch.sprout_to_pathname("some-place")
print local_branch.get_revision_id()

> I'd like to change to use it after we merged it to lava-core

Ok

Thanks
ZK

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJPmDD6AAoJEOvx/vtfL8ZXadQP/0ga6sQg+vojuR6+ZSxzD78t
JtPKaRfWPlKrK7FazNjZLVXl/M390bxR/fWbHP9VMCsf9ZgXXL2t8whA0wk4sxnB
tHeaGjqgc21S2XdwGMbQ/XHI9cdSKvwvzpdOTD0ClqcjCBCOhVzQcylZLGAXapiW
TQo6ZrEvHBJwI62soSxk4xTO1ry5POYy8ZoVreYlhrPuYYE0SH7wnLj/Q/hvodmk
MWV5wfxgsGjeUTykKtNSZ7GQV1bWo8Cq60BG8xrUIlI2QEtcahFL2LsvKU/kV3PG
3Ul3QYj+zNv/TYMgA2qkyK7xYG0Rd0Gk8vxMEfId2nndcXTgo8MCue/vn8eU2Q5d
XdLHN+KzwDT6tY2QnxwYIfYIejUyWIPxBIXVI1DTO7xiJyjNpT/WESugC2MHZdGx
0k6KFXEYiGpze9INgnFp55aanfCDWUd+TY6vW4I754CtTIxOpvmMRIzxghVNjNLT
PNglV3y3nLy22N7+XBf0yKl2n9mh0f8te4DYNtjckKi1/lUyY8F4rDkPdhBkYj/J
b4zfGuw1E50fZ1nc+VDGFD+MS7Odj1Kmu5aRTppRobuOFVlsOSqvd1mkZ0t/WMj9
5cq78RBcXboFQF9beZJowrK6pI+koHzM8n55sicdySUSPhh24KtXHgo2T2TBajyA
3rI4BWQy+2v9tuxRFSw8
=xea1
-----END PGP SIGNATURE-----

[Michael Hudson-Doyle (mwhudson)]

Yongqin Liu <email address hidden> writes:

> Yongqin Liu has proposed merging lp:~liuyq0307/lava-android-test/support-monkeyrunner into lp:lava-android-test.
>
> Requested reviews:
> Linaro Validation Team (linaro-validation)
>
> For more details, see:
> https://code.launchpad.net/~liuyq0307/lava-android-test/support-monkeyrunner/+merge/103452
>
> add run-monkeyrunner sub command to make support monkeyrunner test easily.

It has to be said, this whole approach feels like a way for the android
team to get out of the chore of having to define their tests
properly... but if that's what they want, they can have it I guess.

I have quite a lot of Python coding issues, please see inline comments.

> === modified file 'lava_android_test/commands.py'
> --- lava_android_test/commands.py 2012-04-11 05:58:13 +0000
> +++ lava_android_test/commands.py 2012-04-25 12:02:22 +0000
> @@ -21,16 +21,20 @@
> import urlparse
> import versiontools
>
> +from tempfile import mkdtemp
> +
> from lava_tool.interface import Command as LAVACommand
> from lava_tool.interface import LavaCommandError
> from linaro_dashboard_bundle.io import DocumentIO
>
> from lava_android_test.adb import ADB
> from lava_android_test.config import get_config
> +from lava_android_test.repository import GitRepository
> from lava_android_test.testdef import testloader, AndroidTest
> from lava_android_test.testdef import AndroidTestRunner, \
> AndroidTestInstaller, \
> AndroidTestParser
> +from lava_android_test import utils
>
>
> class Command(LAVACommand):
> @@ -146,6 +150,17 @@
> test_dir = os.path.join(self.config.installdir_android, test_id)
> return self.adb.exists(test_dir)
>
> + def get_device_serial(self):
> + if not self.args.serial:
> + serial_ary = ADB().run_cmd_host('adb get-serialno')[1]
> + serial = serial_ary[0].strip()
> + if not serial or serial == 'unknown':
> + return ''
> + else:
> + return serial
> + else:
> + return self.args.serial
> +
>
> class AndroidTestCommand(AndroidCommand):
> @classmethod
> @@ -435,6 +450,136 @@
> self.say_end(tip_msg)
>
>
> +class run_monkeyrunner(AndroidCommand):
> + """
> + Run the monkeyrunner scripts that stored in the specified git repository

> + program:: lava-android-test run-monkeyrunner -g giturl -r resultfilelist
> + """
> +
> + @classmethod
> + def register_arguments(cls, parser):
> + super(run_monkeyrunner, cls).register_arguments(parser)
> + parser.add_argument("url",
> + help="The repository url of the test scripts")
> + parser.add_argument('-t', '--repo-type',
> + default='git',
> + help=("Specify the type of the repository"))
> + group = parser.add_argument_group("specify the bundle output file")
> + group.add_argument("-o", "--output",
> + default=None,
> + metavar="FILE",
> + he...

[lp:~liuyq0307/lava-android-test/support-monkeyrunner]

152. By Yongqin Liu on 2012-05-23

merge with trunk

153. By Yongqin Liu on 2012-05-23

update according to the comment

154. By Yongqin Liu on 2012-05-24

update the typo of bundle

155. By Yongqin Liu on 2012-05-24

revert project files

156. By Yongqin Liu on 2012-05-24

add return serial for get_device_serial according to review comment

157. By Yongqin Liu on 2012-05-24

modify bundle_format definition bug that using os.environ.get to define

158. By Yongqin Liu on 2012-05-24

modify the auto rewriting error

Unmerged revisions

=== modified file 'lava_android_test/commands.py'
--- lava_android_test/commands.py	2012-05-08 05:32:19 +0000
+++ lava_android_test/commands.py	2012-05-23 03:54:17 +0000
@@ -21,16 +21,20 @@
 import urlparse
 import versiontools
 
+from tempfile import mkdtemp
+
 from lava_tool.interface import Command as LAVACommand
 from lava_tool.interface import LavaCommandError
 from linaro_dashboard_bundle.io import DocumentIO
 
 from lava_android_test.adb import ADB
 from lava_android_test.config import get_config
+from lava_android_test.repository import GitRepository
 from lava_android_test.testdef import testloader, AndroidTest
 from lava_android_test.testdef import AndroidTestRunner, \
                                       AndroidTestInstaller, \
                                       AndroidTestParser
+from lava_android_test import utils
 
 
 class Command(LAVACommand):
@@ -146,6 +150,13 @@
         test_dir = os.path.join(self.config.installdir_android, test_id)
         return self.adb.exists(test_dir)
 
+    def get_device_serial(self):
+        if not self.args.serial:
+            serial_ary = ADB().run_cmd_host('adb get-serialno')[1]
+            serial = serial_ary[0].strip()
+            if not serial or serial == 'unknown':
+                return ''
+
     def assertDeviceIsConnected(self):
         if not self.adb.isDeviceConnected():
             if self.adb.serial:
@@ -453,6 +464,136 @@
         self.say_end(tip_msg)
 
 
+class run_monkeyrunner(AndroidCommand):
+    """
+    Run the monkeyrunner scripts that stored in the specified git repository

+    program:: lava-android-test run-monkeyrunner -g giturl -r resultfilelist
+    """
+
+    @classmethod
+    def register_arguments(cls, parser):
+        super(run_monkeyrunner, cls).register_arguments(parser)
+        parser.add_argument("url",
+                            help="The repository url of the test scripts")
+        parser.add_argument('-t', '--repo-type',
+                            default='git',
+                            help=("Specify the type of the repository"))
+        group = parser.add_argument_group("specify the bundle output file")
+        group.add_argument("-o", "--output",
+                            default=None,
+                            metavar="FILE",
+                            help=("After running the test parse the result"
+                                 " artefacts, fuse them with the initial"
+                                 " bundle and finally save the complete bundle"
+                                 " to the  specified FILE."))
+
+    def invoke(self):
+        config = get_config()
+        if self.args.serial:
+            serial = self.args.serial
+        else:
+            serial = self.get_device_serial()
+            if not serial:
+                raise LavaCommandError("No device attached")
+        self.adb = ADB(serial)
+
+        if self.args.repo_type == 'git':
+            target_dir = mkdtemp(prefix='git_repo', dir=config.tempdir_host)
+            os.chmod(target_dir, 0755)
+            GitRepository(self.args.url).checkout(target_dir)
+
+        script_list = utils.find_files(target_dir, '.py')
+
+        test_id = self.args.url
+        if len(test_id) > 40:
+            test_id = '%s...' % (test_id[:40])
+        test_id = 'monkeyrunner_%s' % test_id
+
+        tip_msg = ("Run monkeyrunner scripts in following url on device(%s):"
+                       "\n\turl=%s") % (
+                       self.args.serial,
+                       self.args.url)
+
+        self.say_begin(tip_msg)
+        bundles = []
+        for script in script_list:
+            sub_bundle = {}
+            from datetime import datetime
+            starttime = datetime.utcnow()
+            test_case_id = script.replace('%s/' % target_dir, '')
+            if len(test_case_id) > 50:
+                test_case_id = '%s...' % (test_case_id[:50])
+            try:
+                sub_bundle = self.run_monkeyrunner_test(script, serial,
+                                                         test_case_id)
+                test_result = {"test_case_id": test_case_id,
+                               "result": 'pass'}
+                if sub_bundle:
+                    sub_bundle['test_runs'][0]['test_results'].append(
+                                                            test_result)
+            except Exception as strerror:
+                self.say('Failed to run script(%s) with error:\n%s' % (
+                                                                script,
+                                                                strerror))
+
+                test_result = {"test_case_id": test_case_id,
+                               "result": 'fail'}
+                from uuid import uuid4
+                TIMEFORMAT = '%Y-%m-%dT%H:%M:%SZ'
+                sub_bundle['test_runs'] = [{
+                                'test_results': [test_result],
+                                'test_id':
+                                    'monkeyrunner(%s)' % test_case_id,
+                                'time_check_performed': False,
+                                'analyzer_assigned_uuid':
+                                        str(uuid4()),
+                                'analyzer_assigned_date':
+                                    starttime.strftime(TIMEFORMAT)}]
+            if sub_bundle:
+                    bundles.append(sub_bundle)
+
+        if self.args.output:
+            output_dir = os.path.dirname(self.args.output)
+            if output_dir and (not os.path.exists(output_dir)):
+                os.makedirs(output_dir)
+            with open(self.args.output, "wt") as stream:
+                DocumentIO.dump(stream, merge_bundles(bundles))
+
+        self.say_end(tip_msg)
+
+    def run_monkeyrunner_test(self, script, serial, test_case_id=None):
+        config = get_config()
+
+        inst = AndroidTestInstaller()
+        run = AndroidTestRunner(steps_host_pre=[
+                                'monkeyrunner %s %s' % (script, serial)])
+        parser = AndroidTestParser()
+        test = AndroidTest(testname='monkeyrunner',
+                            installer=inst, runner=run, parser=parser)
+        test.parser.results = {'test_results': []}
+        test.setadb(self.adb)
+
+        if not self.test_installed(test.testname):
+            test.install()
+
+        bundle = {}
+        org_png_file_list = utils.find_files(config.tempdir_host,
+                                             '.%s' % 'png')
+        result_id = test.run(quiet=self.args.quiet)
+        if self.args.output:
+            cur_all_png_list = utils.find_files(config.tempdir_host,
+                                                '.%s' % 'png')
+            new_png_list = set(cur_all_png_list).difference(org_png_file_list)
+            test_id = 'monkeyrunner(%s)' % (test_case_id)
+            bundle = generate_bundle(self.args.serial,
+                    result_id, test=test,
+                    test_id=test_id,
+                    attachments=list(new_png_list))
+            utils.delete_files(new_png_list)
+
+        return bundle
+
+
 class parse(AndroidResultsCommand):
     """
     Parse the results of previous test that run on the specified device

@@ -518,19 +659,29 @@
     return bundle
 
 
-def generate_bundle(serial=None, result_id=None, test=None, test_id=None):
+def merge_bundles(bundles=[]):
+    merged_bundles = {"format": "Dashboard Bundle Format 1.2",
+                      'test_runs': []}
+    for bundle in bundles:
+        if bundle['test_runs']:
+            merged_bundles['test_runs'].append(bundle['test_runs'][0])
+    return merged_bundles
+
+
+def generate_bundle(serial=None, result_id=None, test=None,
+                    test_id=None, attachments=[]):
     if result_id is None:
         return {}
     config = get_config()
     adb = ADB(serial)
     resultdir = os.path.join(config.resultsdir_android, result_id)
     if not adb.exists(resultdir):
-        raise  LavaCommandError("The result (%s) is not existed." % result_id)
+        raise  Exception("The result (%s) is not existed." % result_id)
 
     bundle_text = adb.read_file(os.path.join(resultdir, "testdata.json"))
     bundle = DocumentIO.loads(bundle_text)[1]
     test_tmp = None
-    if bundle['test_runs'][0]['test_id'] == 'custom':
+    if test:
         test_tmp = test
     else:
         test_tmp = testloader(bundle['test_runs'][0]['test_id'], serial)
@@ -560,6 +711,7 @@
             "content":  base64.standard_b64encode(stderr_text)
         }
     ]
+
     screencap_path = os.path.join(resultdir, 'screencap.png')
     if adb.exists(screencap_path):
         tmp_path = os.path.join(config.tempdir_host, 'screencap.png')
@@ -571,7 +723,17 @@
             "pathname": 'screencap.png',
             "mime_type": 'image/png',
             "content": base64.standard_b64encode(data)})
+        os.unlink(tmp_path)
 
+    for attach in attachments:
+        if os.path.exists(attach):
+            with open(attach, 'rb') as stream:
+                data = stream.read()
+            if data:
+                bundle['test_runs'][0]["attachments"].append({
+                            "pathname": os.path.basename(attach),
+                            "mime_type": 'image/png',
+                            "content": base64.standard_b64encode(data)})
     return bundle
 
 

=== modified file 'lava_android_test/main.py'
--- lava_android_test/main.py	2012-03-09 03:31:21 +0000
+++ lava_android_test/main.py	2012-05-23 03:54:17 +0000
@@ -1,4 +1,4 @@
-# Copyright (c) 2010 Linaro
+# Copyright (c) 2010-2012 Linaro
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -14,9 +14,9 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 import os
 import sys
-import subprocess
 import shutil
 from tempfile import mkdtemp
+from lava_android_test import utils
 from lava_android_test.config import get_config, set_config
 from lava_tool.dispatcher import LavaDispatcher, run_with_dispatcher_class
 
@@ -33,8 +33,7 @@
 
 
 def check_adb_installed():
-        rc = subprocess.call(["which", "adb"], stdout=open('/dev/null', 'w'))
-        return rc == 0
+    return utils.check_command_exist('adb')
 
 
 def main():

=== added file 'lava_android_test/repository.py'
--- lava_android_test/repository.py	1970-01-01 00:00:00 +0000
+++ lava_android_test/repository.py	2012-05-23 03:54:17 +0000
@@ -0,0 +1,83 @@
+# Copyright (C) 2012 Linaro Limited
+#
+# Author: Linaro Validation Team <[e-mail]>
+#
+# This file is part of LAVA Android Test.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+import os
+import subprocess
+
+from lava_android_test import utils
+
+
+class RepositoryError(Exception):
+    """
+    Raise this for repository related errors
+    """
+
+
+class Repository(object):
+    '''
+    Base class for all repository class used to checkout files.
+    This class and sub classes are base the repository command.
+    '''
+
+    def __init__(self, url, repo_type, cmds=[]):
+        self.url = url
+        self.repo_type = repo_type
+        self.check_cmds_exist(cmds)
+
+    def checkout(self, target_dir=None):
+        """
+        Checkout this repository to the specified the target_dir directory
+        """
+        raise NotImplementedError()
+
+    def check_cmds_exist(self, cmds=[]):
+        """
+        check whether the necessary commands are existing.
+        """
+        for cmd in cmds:
+            if not utils.check_command_exist(cmd):
+                raise RepositoryError(("The necessary command(%) does not"
+                                      " exist, Or can't be seen from path")
+                                       % cmd)
+
+
+class GitRepository(Repository):
+
+    git_cmd = 'git'
+
+    def __init__(self, url):
+        super(GitRepository, self).__init__(url, 'git', [self.git_cmd])
+
+    def checkout(self, target_dir=None):
+        """
+        Checkout this git repository to the specified the target_dir directory
+        """
+        cmds = []
+        if not target_dir:
+            cmds = [self.git_cmd, 'clone', self.url]
+
+        else:
+            if not os.path.exists(target_dir):
+                os.makedirs(target_dir)
+            cmds = [self.git_cmd, 'clone', self.url, target_dir]
+
+        rc = subprocess.call(cmds)
+        if rc != 0:
+            raise RepositoryError(("Failed to clone the specified "
+                  "repository() with exist staus=%d")
+                                   % (self.url, rc))

=== modified file 'lava_android_test/test_definitions/tjbench.py'
--- lava_android_test/test_definitions/tjbench.py	2012-04-06 13:14:51 +0000
+++ lava_android_test/test_definitions/tjbench.py	2012-05-23 03:54:17 +0000
@@ -87,7 +87,7 @@
                     if not prefix_hash.get(test_case_preffix):
                         prefix_hash[test_case_preffix] = True
                         test_case_preffix = '%s_%s' % (test_case_preffix,
-                                                      'scale_half')    
+                                                      'scale_half')
                     common_data = {'log_filename': result_filename,
                                    'log_lineno': lineno,
                                    'result': 'pass'

=== modified file 'lava_android_test/test_definitions/v8.py'
--- lava_android_test/test_definitions/v8.py	2012-03-20 14:13:50 +0000
+++ lava_android_test/test_definitions/v8.py	2012-05-23 03:54:17 +0000
@@ -17,7 +17,6 @@
 #
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
 import lava_android_test.config
 import lava_android_test.testdef
 

=== modified file 'lava_android_test/utils.py'
--- lava_android_test/utils.py	2012-03-08 11:29:09 +0000
+++ lava_android_test/utils.py	2012-05-23 03:54:17 +0000
@@ -1,4 +1,4 @@
-# Copyright (c) 2010 Linaro
+# Copyright (c) 2010-2012 Linaro
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -152,3 +152,27 @@
     if rest is None or  rest == '':
         return host
     return os.path.basename(rest)
+
+
+def check_command_exist(command):
+    rc = subprocess.call(["which", command], stdout=open('/dev/null', 'w'))
+    return rc == 0
+
+
+def find_files(target_dir, ext):
+    file_list = []
+    for f in os.listdir(target_dir):
+        f_path = '%s/%s' % (target_dir, f)
+        if os.path.isdir(f_path):
+            file_list.extend(find_files(f_path, ext))
+        elif f_path.upper().endswith(ext.upper()):
+            file_list.append(f_path)
+
+    file_list.sort()
+    return file_list
+
+
+def delete_files(flist=[]):
+    for f in flist:
+        if os.path.exists(f):
+            os.unlink(f)

=== modified file 'setup.py'
--- setup.py	2012-03-07 10:09:01 +0000
+++ setup.py	2012-05-23 03:54:17 +0000
@@ -42,6 +42,7 @@
     uninstall=lava_android_test.commands:uninstall
     run=lava_android_test.commands:run
     run-custom=lava_android_test.commands:run_custom
+    run-monkeyrunner=lava_android_test.commands:run_monkeyrunner
     parse=lava_android_test.commands:parse
     parse-custom=lava_android_test.commands:parse_custom
     show=lava_android_test.commands:show