Merge lp:~afrantzis/lava-test/alf-testdefs into lp:lava-test/0.0

[Alexandros Frantzis]

Test definitions for Linaro User Platforms graphics related benchmarks.

[Paul Larson (pwlars)]

I haven't had time to fully look at all of these, but a few observations on what I've seen so far:
First off, the clutter tests that had been segfaulting for me previously are no longer doing that. Something, somewhere, got fixed on this already. yay!
Also, it would be much easier to review these separately if at all possible. Also, that way some of the simpler ones can get in quickly while we deal with issues of the others.

1 === added file 'abrek/test_definitions/average_parser.py'
We need to find a different place for these. Anything under test_definitions is assumed to be a test. If it's only used by a single test, then you can have a directory with the test name. If it's something shared among more than one test, then we really ought to put it in the common code somewhere.

277 === added file 'abrek/test_definitions/timed_test_runner.py'
same here

301 + def _runsteps(self, resultsdir, quiet=False):
302 + outputlog = os.path.join(resultsdir, 'testoutput.log')
303 + for (cmd, runtime, info) in zip(self.steps, self.runtime, self.info):
304 + # Use a pty to make sure output from the tests
305 + # is immediately available (line-buffering)
I've been looking at doing something like this for everything, so I want to take a look at moving this out as well.

[Alexandros Frantzis (afrantzis)]

Agreed, I will break this up into separate merge proposals.

About the shared runner/parser code, perhaps it is worth having a testdef utilities directory where we can put commonly used parser/runner classes, so that the main code remains clean.

[Paul Larson (pwlars)]

> Agreed, I will break this up into separate merge proposals.
>
> About the shared runner/parser code, perhaps it is worth having a testdef
> utilities directory where we can put commonly used parser/runner classes, so
> that the main code remains clean.
Yes, that sounds good

[Zygmunt Krynicki (zyga)]

Alf: is this still valid? I'm inclined to reject it as you are splitting those up to separate branches?

review: Needs Information

Unmerged revisions

41. By Alexandros Frantzis on 2010-09-29

Download qgears2 package from personal apt repo.

40. By Alexandros Frantzis on 2010-09-24

Update clutter-eglx-es20 test definition for new abrek API.

39. By Alexandros Frantzis on 2010-09-24

Update x11perf test definition for new abrek API.

38. By Alexandros Frantzis on 2010-09-24

Really append 'units' and 'result' fields to all gtkperf test cases.

37. By Alexandros Frantzis on 2010-09-24

Update render-bench test definition for new abrek API.

36. By Alexandros Frantzis on 2010-09-24

Update qgears test definitions for new abrek API.

35. By Alexandros Frantzis on 2010-09-24

Update glmemperf for new abrek API.

34. By Alexandros Frantzis on 2010-09-24

Update glmark2-es2 test for new abrek API.

33. By Alexandros Frantzis on 2010-09-24

Update es2gears test for new abrek API.

32. By Alexandros Frantzis on 2010-09-24

Update average_parser and timed_test_parser for new abrek API.

=== added file 'abrek/test_definitions/average_parser.py'
--- abrek/test_definitions/average_parser.py	1970-01-01 00:00:00 +0000
+++ abrek/test_definitions/average_parser.py	2010-09-29 14:13:40 +0000
@@ -0,0 +1,73 @@
+import re
+import abrek.testdef
+import pickle
+
+class AverageParser(abrek.testdef.AbrekTestParser):
+    """Single test average result parser
+    
+    Parses a series of results from the same test and computes the
+    average.
+    """
+    def parse(self):
+        filename = "testoutput.log"
+        pat = re.compile(self.pattern)
+        total = 0.0
+        count = 0
+        with open(filename, 'r') as fd:
+            for line in fd:
+                match = pat.search(line)
+                if match:
+                    d = match.groupdict()
+                    total = total + float(d['measurement'])
+                    count = count + 1
+
+        if count > 0:
+            avg_result = total/count
+        else:
+            avg_result = 0
+
+        avg = {'measurement':avg_result}
+        self.results['test_results'].append(avg)
+        if self.fixupdict:
+            self.fixresults(self.fixupdict)
+        if self.appendall:
+            self.appendtoall(self.appendall)
+
+class MultiAverageParser(abrek.testdef.AbrekTestParser):
+    """Multiple test average result parser
+    
+    Parses a series of results from multiple tests and computes the
+    average for each test.
+    """
+    def parse(self):
+        filename = "testoutput.log"
+        pat = re.compile(self.pattern)
+        pat_end = re.compile("^\$End AbrekTest\$ (?P<info>.*)")
+        total = 0.0
+        count = 0
+        with open(filename, 'r') as fd:
+            for line in fd:
+                match = pat.search(line)
+                if match:
+                    d = match.groupdict()
+                    total = total + float(d['measurement'])
+                    count = count + 1
+                else:
+                    match = pat_end.search(line)
+                    if match:
+                        d = match.groupdict()
+                        if count > 0:
+                            avg_result = total/count
+                        else:
+                            avg_result = 0
+                        avg = pickle.loads(d['info'].decode('string-escape')[1:])
+                        avg['measurement'] = avg_result
+                        self.results['test_results'].append(avg)
+                        total = 0.0
+                        count = 0
+
+        if self.fixupdict:
+            self.fixresults(self.fixupdict)
+        if self.appendall:
+            self.appendtoall(self.appendall)
+
 
=== added file 'abrek/test_definitions/clutter-eglx-es20.py'
--- abrek/test_definitions/clutter-eglx-es20.py	1970-01-01 00:00:00 +0000
+++ abrek/test_definitions/clutter-eglx-es20.py	2010-09-29 14:13:40 +0000
@@ -0,0 +1,33 @@
+import abrek.testdef
+import pickle
+from average_parser import MultiAverageParser
+from timed_test_runner import TimedTestRunner
+
+clutter_tests = [
+    ("test-interactive", "test-rotate", 10),
+    ("test-interactive", "test-actors", 10),
+    ("test-interactive", "test-fbo", 10),
+    ("test-interactive", "test-animator", 10),
+    ("test-interactive", "test-random-text", 10),
+]
+cmd_fmt = "CLUTTER_SHOW_FPS=1 /usr/lib/clutter-1.0/tests/eglx-es20/%s %s"
+
+def create_test_cmds(tests):
+    return [cmd_fmt % args[:2] for args in tests]
+
+def create_test_runtimes(tests):
+    return [args[2] for args in tests]
+
+def create_test_info(tests):
+    return ["%r" % pickle.dumps({'test_case_id':'%s.%s' % args[:2], 'units':'fps'}) for args in tests]
+        
+
+parse = MultiAverageParser(pattern="\*\*\* FPS: (?P<measurement>\d+) \*\*\*",
+                           appendall={'result':'pass'})
+inst = abrek.testdef.AbrekTestInstaller(deps=["clutter-eglx-es20-1.0-tests"])
+run = TimedTestRunner(create_test_cmds(clutter_tests),
+                      create_test_runtimes(clutter_tests),
+                      create_test_info(clutter_tests))
+
+testobj = abrek.testdef.AbrekTest(testname="clutter-eglx-es20", installer=inst,
+                                  runner=run, parser=parse)
 
=== added file 'abrek/test_definitions/es2gears.py'
--- abrek/test_definitions/es2gears.py	1970-01-01 00:00:00 +0000
+++ abrek/test_definitions/es2gears.py	2010-09-29 14:13:40 +0000
@@ -0,0 +1,13 @@
+import re
+import abrek.testdef
+from average_parser import AverageParser
+from timed_test_runner import TimedTestRunner
+
+parse = AverageParser(pattern="=\W+(?P<measurement>\d+\.\d+) FPS",
+                      appendall= {'test_case_id':'es2gears', 'units':'fps',
+                                  'result':'pass'})
+inst = abrek.testdef.AbrekTestInstaller(deps=["es2gears"])
+run = TimedTestRunner(["es2gears"], [16], [""])
+
+testobj = abrek.testdef.AbrekTest(testname="es2gears", installer=inst,
+                                  runner=run, parser=parse)
 
=== added file 'abrek/test_definitions/glmark2-es2.py'
--- abrek/test_definitions/glmark2-es2.py	1970-01-01 00:00:00 +0000
+++ abrek/test_definitions/glmark2-es2.py	2010-09-29 14:13:40 +0000
@@ -0,0 +1,40 @@
+import re
+import abrek.testdef
+
+RUNSTEPS = ["glmark2-es2"]
+
+class Glmark2Parser(abrek.testdef.AbrekTestParser):
+    def parse(self):
+        PAT1 = "^\W+(?P<subtest>.*?)\W+FPS:\W+(?P<measurement>\d+)"
+        filename = "testoutput.log"
+        pat1 = re.compile(PAT1)
+        in_results = False
+        cur_test = ""
+        with open(filename, 'r') as fd:
+            for line in fd.readlines():
+                if line.find("Precompilation") != -1:
+                    in_results = True
+                if in_results == True:
+                    match = pat1.search(line)
+                    if match:
+                        d = match.groupdict()
+                        d['test_case_id'] = "%s.%s" % (cur_test, d['subtest'])
+                        d.pop('subtest')
+                        self.results['test_results'].append(d)
+                    else:
+                        if line.startswith("==="):
+                            in_results = False
+                        else:
+                            cur_test = line.strip()
+
+        if self.fixupdict:
+            self.fixresults(self.fixupdict)
+        if self.appendall:
+            self.appendtoall(self.appendall)
+
+parse = Glmark2Parser(appendall={'units':'fps', 'result':'pass'})
+inst = abrek.testdef.AbrekTestInstaller(deps=["glmark2-es2"])
+run = abrek.testdef.AbrekTestRunner(RUNSTEPS)
+
+testobj = abrek.testdef.AbrekTest(testname="glmark2-es2", installer=inst,
+                                  runner=run, parser=parse)
 
=== added file 'abrek/test_definitions/glmemperf.py'
--- abrek/test_definitions/glmemperf.py	1970-01-01 00:00:00 +0000
+++ abrek/test_definitions/glmemperf.py	2010-09-29 14:13:40 +0000
@@ -0,0 +1,13 @@
+import abrek.testdef
+
+RUNSTEPS = ["glmemperf"]
+PATTERN = "^(?P<test_case_id>\w+):\W+(?P<measurement>\d+) fps"
+
+inst = abrek.testdef.AbrekTestInstaller(deps=["glmemperf"])
+run = abrek.testdef.AbrekTestRunner(RUNSTEPS)
+parse = abrek.testdef.AbrekTestParser(PATTERN,
+                                      appendall={'units':'fps',
+                                                 'result':'pass'})
+
+testobj = abrek.testdef.AbrekTest(testname="glmemperf", installer=inst,
+                                  runner=run, parser=parse)
 
=== modified file 'abrek/test_definitions/gtkperf.py'
--- abrek/test_definitions/gtkperf.py	2010-09-22 17:55:02 +0000
+++ abrek/test_definitions/gtkperf.py	2010-09-29 14:13:40 +0000
@@ -27,7 +27,9 @@
                     if match:
                         self.results['test_results'].append(match.groupdict())
 
-parse = GtkTestParser(appendall={'units':'seconds', 'result':'pass'})
+        self.appendtoall({'units':'seconds', 'result':'pass'})
+
+parse = GtkTestParser()
 inst = abrek.testdef.AbrekTestInstaller(deps=["gtkperf"])
 run = abrek.testdef.AbrekTestRunner(RUNSTEPS)
 
 
=== added file 'abrek/test_definitions/qgears.py'
--- abrek/test_definitions/qgears.py	1970-01-01 00:00:00 +0000
+++ abrek/test_definitions/qgears.py	2010-09-29 14:13:40 +0000
@@ -0,0 +1,21 @@
+import re
+import abrek.testdef
+from average_parser import AverageParser
+from timed_test_runner import TimedTestRunner
+
+INSTALL_STEPS = [
+    'sudo add-apt-repository "deb http://people.canonical.com/~afrantzis/packages/ ./"',
+    'sudo apt-get update',
+    'sudo apt-get install -y --force-yes qgears2'
+    ]
+
+parse = AverageParser(pattern="=\W+(?P<measurement>\d+\.\d+) FPS", 
+                      appendall={'test_case_id':'qgears', 'units':'fps',
+                                 'result':'pass'})
+
+inst = abrek.testdef.AbrekTestInstaller(INSTALL_STEPS,
+                                        deps=["python-software-properties"])
+run = TimedTestRunner(["qgears"], [10], [""])
+
+testobj = abrek.testdef.AbrekTest(testname="qgears", installer=inst,
+                                  runner=run, parser=parse)
 
=== added file 'abrek/test_definitions/render-bench.py'
--- abrek/test_definitions/render-bench.py	1970-01-01 00:00:00 +0000
+++ abrek/test_definitions/render-bench.py	2010-09-29 14:13:40 +0000
@@ -0,0 +1,33 @@
+import re
+import abrek.testdef
+
+class RenderBenchParser(abrek.testdef.AbrekTestParser):
+    def parse(self):
+        PAT1 = "^Test: (?P<test_case_id>.*)"
+        PAT2 = "^Time: (?P<measurement>\d+\.\d+)"
+        filename = "testoutput.log"
+        pat1 = re.compile(PAT1)
+        pat2 = re.compile(PAT2)
+        cur_test = None
+        with open(filename, 'r') as fd:
+            for line in fd:
+                match = pat1.search(line)
+                if match:
+                    cur_test = match.groupdict()['test_case_id']
+                else:
+                    match = pat2.search(line)
+                    if match:
+                        d = match.groupdict()
+                        d['test_case_id'] = cur_test
+                        self.results['test_results'].append(d)
+
+        self.appendtoall({'units':'seconds', 'result':'pass'})
+
+RUNSTEPS = ["render_bench"]
+
+inst = abrek.testdef.AbrekTestInstaller(deps=["render-bench"])
+run = abrek.testdef.AbrekTestRunner(RUNSTEPS)
+parse = RenderBenchParser()
+
+testobj = abrek.testdef.AbrekTest(testname="render-bench", installer=inst,
+                                  runner=run, parser=parse)
 
=== added file 'abrek/test_definitions/timed_test_runner.py'
--- abrek/test_definitions/timed_test_runner.py	1970-01-01 00:00:00 +0000
+++ abrek/test_definitions/timed_test_runner.py	2010-09-29 14:13:40 +0000
@@ -0,0 +1,51 @@
+import abrek.testdef
+import pty
+import time
+import os
+import subprocess
+import signal
+import sys
+
+class TimedTestRunner(abrek.testdef.AbrekTestRunner):
+    """Test runner class for running tests for specific amounts of time
+
+    steps - list of steps to be executed in a shell
+    runtime - list of runtimes for each step
+    info - informational string to append to start/end markers
+    """
+    def __init__(self, steps=[], runtime=[], info=[]):
+        super(TimedTestRunner, self).__init__(steps)
+        self.runtime = runtime
+        self.info = info
+
+    def _runsteps(self, resultsdir, quiet=False):
+        outputlog = os.path.join(resultsdir, 'testoutput.log')
+        for (cmd, runtime, info) in zip(self.steps, self.runtime, self.info):
+            # Use a pty to make sure output from the tests
+            # is immediately available (line-buffering)
+            (pid, ptyfd) = pty.fork()
+            if pid == 0:
+                try:
+                    print "$Start AbrekTest$ %s" % info
+                    subprocess.call(cmd, shell=True)
+                except:
+                    pass
+                    
+                print "$End AbrekTest$ %s" % info
+                sys.exit()
+            else:
+                time.sleep(runtime)
+                os.kill(pid, signal.SIGINT)
+                with open(outputlog, 'a') as fd:
+                    out = os.read(ptyfd, 1024)
+                    while out != "":
+                        fd.write(out)
+                        out = ""
+                        try:
+                            out = os.read(ptyfd, 1024)
+                        except:
+                            pass
+
+                os.close(ptyfd)
+
+
 
=== added file 'abrek/test_definitions/x11perf.py'
--- abrek/test_definitions/x11perf.py	1970-01-01 00:00:00 +0000
+++ abrek/test_definitions/x11perf.py	2010-09-29 14:13:40 +0000
@@ -0,0 +1,39 @@
+import re
+import abrek.testdef
+
+x11perf_options = "-repeat 3"
+
+x11perf_tests = [
+    # Antialiased text (using XFT)
+    "-aa10text",
+    "-aa24text",
+
+    # Antialiased drawing (using XRENDER)
+    "-aatrapezoid300",
+    "-aatrap2x300",
+
+    # Normal blitting
+    "-copypixwin500",
+    "-copypixpix500",
+
+    # Composited blitting
+    "-comppixwin500",
+
+    # SHM put image
+    "-shmput500",
+    "-shmputxy500",
+
+    "-scroll500",
+    ]
+
+RUNSTEPS = ["x11perf %s %s" % (x11perf_options,  " ".join(x11perf_tests))]
+PATTERN = "trep @.*\(\W*(?P<measurement>\d+.\d+)/sec\):\W+(?P<test_case_id>.+)"
+
+inst = abrek.testdef.AbrekTestInstaller(deps=["x11-apps"])
+run = abrek.testdef.AbrekTestRunner(RUNSTEPS)
+parse = abrek.testdef.AbrekTestParser(PATTERN,
+                                      appendall={'units':'reps/s', 
+                                                 'result':'pass'})
+
+testobj = abrek.testdef.AbrekTest(testname="x11perf", installer=inst,
+                                  runner=run, parser=parse)