Merge lp:~le-chi-thu/lava-test/add-tutorial-1 into lp:lava-test/0.0

[Le Chi Thu]

Added tutorial how to create new test and how to implement own test result parser.

[Zygmunt Krynicki (zyga)]

Hi ChiThu.

42 + run = TestRunner(RUNSTEPS,default_options=DEFAULT_OPTIONS)

not PEP-8 clean

86 +artifacts.stdout_pathname is the file stored the output of the test execution. By parsing this file and append the test
87 +result to the self.results['test_result'] list. The :meth:`~lava_test.core.parsers.TestParser.analyze_test_result` help
88 +method will do the sanity check and also perform the fixup of the result.

IMHO this is not easy to understand. The "fixup" is vague, what is it about, why is it necessary? (I know, but think about the readers)

115 + $ lava-test register http://abc.com/my-custom-test.json

Use example.{com,org,net} per RFC 2606

66 + class LTPParser(TestParser):
67 + def parse(self, artifacts):

Again PEP-8

72 + with open(filename, 'r') as fd:

This is a logical bug, open does not return a file descriptor. It returns a file-like object. That object happens to have (on some platforms) a fileno() method that returns a descriptor number. Call it 'stream' instead; this is just confusing the reader.

121 +To see how the json file look like, let see the examples/stream.json. This file shows how to write the stream test in
122 +json format. Same test is written in python format: lava_test/test_definitions/stream.py.

This is also confusing. We should explain what the JSON representation does. It tells lava-test how to construct a test object out of the available test classes. While learning by example is fine without explanation it _will_ lead to confusion.

63 +To show how to implement the new parser, let have a closer look at one of existing test - lava_test/test_definitions/ltp.py

(snip)
66 + class LTPParser(TestParser):
67 + def parse(self, artifacts):
68 + filename = artifacts.stdout_pathname
69 + print "filename=%s" %filename
70 + # PATTERN = "^(?P<test_case_id>\S+) (?P<subid>\d+) (?P<result>\w+) : (?P<message>.+)"

None of this code makes any sense to the reader. What you should have started with is a discussion on what the test parser is for. Start by showing what the test program output looks like. Highlighting the parts that we want to capture in LAVA and then constructing simple code that does that. Only then show how to wrap that code in a Parser class.

review: Needs Fixing

[Paul Larson (pwlars)]

bump - ChiThu are you planning to update this soon?

Unmerged revisions

119. By Le Chi Thu <email address hidden> <email address hidden> on 2012-02-22

Added tutorial how to add new test and some minor documentation update.

=== modified file 'doc/usage.rst'
--- doc/usage.rst	2011-09-12 09:19:10 +0000
+++ doc/usage.rst	2012-02-22 11:47:17 +0000
@@ -141,6 +141,89 @@
     nice to have a tutorial that walks the user through wrapping a
     simple pass/fail test. 
 
+Tutorial how to add a new test to LAVA
+======================================
+
+Below are the steps needed to add a new test.
+
+* add a new python file in lava_test/test_definitions folder.
+* add the document of new test in lava_test/doc-test/doc/tests.rst
+* add the new test to the BuiltInProvider static list in lava_test/core/providers.py
+* implement the new test
+
+To show how to implement the new test, let have a closer look at one of existing test - lava_test/test_definitions/stream.py
+
+First import the classes needed ::
+
+    from lava_test.core.installers import TestInstaller
+    from lava_test.core.parsers import TestParser
+    from lava_test.core.runners import TestRunner
+    from lava_test.core.tests import Test
+
+Create the :class:`~lava_test.core.installers.TestInstaller` object. We tell the installer to download the stream.c.
+Install the gcc and build-essential Ubuntu packages before compile the stream.c ::
+
+    URL="http://www.cs.virginia.edu/stream/FTP/Code/stream.c"
+    INSTALLSTEPS = ['cc stream.c -O2 -fopenmp -o stream']
+    DEPS = ['gcc', 'build-essential']
+
+    install = TestInstaller(INSTALLSTEPS, deps=DEPS, url=URL)
+
+Create the :class:`~lava_test.core.runners.TestRunner` object. We tell the runner to execute the stream command with
+empty default options ::
+
+    DEFAULT_OPTIONS = ""
+    RUNSTEPS = ['./stream $(OPTIONS)']
+
+    run = TestRunner(RUNSTEPS,default_options=DEFAULT_OPTIONS)
+
+Create the :class:`~lava_test.core.parsers.TestParser` object. We tell the parser to match each line with regular
+expression. When the output of the test is parse, the parser will have a list of results. Each result is a dictionary
+and we ask parser  to append the 'units':'MB/s' and 'result':'pass' to the result dictionary. ::
+
+    PATTERN = "^(?P<test_case_id>\w+):\W+(?P<measurement>\d+\.\d+)"
+    
+    parse = TestParser(PATTERN, appendall={'units':'MB/s', 'result':'pass'})
+
+Create the :class:`~lava_test.core.tests.Test` object and name it as testobj which is expected by the test loader.
+::
+
+    testobj = Test(test_id="stream", installer=install,runner=run, parser=parse)
+
+Tutorial how to create own result parser
+========================================
+
+If the :class:`~lava_test.core.parsers.TestParser` not fulfill the need to parse the test result, you can derive
+the :class:`~lava_test.core.parsers.TestParser` and overwrite the method :meth:`~lava_test.api.core.ITest.parse`.
+
+To show how to implement the new parser, let have a closer look at one of existing test - lava_test/test_definitions/ltp.py
+::
+
+    class LTPParser(TestParser):
+        def parse(self, artifacts):
+            filename = artifacts.stdout_pathname
+            print "filename=%s" %filename
+            # PATTERN = "^(?P<test_case_id>\S+)    (?P<subid>\d+)  (?P<result>\w+)  :  (?P<message>.+)"
+            pat = re.compile(self.pattern)
+            with open(filename, 'r') as fd:
+                for line in fd.readlines():
+                    match = pat.search(line)
+                    if match:
+                        results = match.groupdict()
+                        subid = results.pop('subid')
+                        #The .0 results in ltp are all TINFO, filtering them
+                        #should help eliminate meaningless, duplicate results
+                        if subid == '0':
+                            continue
+                        results['test_case_id'] += "." + subid
+                        self.results['test_results'].append(
+                            self.analyze_test_result(results))
+
+artifacts.stdout_pathname is the file stored the output of the test execution. By parsing this file and append the test
+result to the self.results['test_result'] list. The :meth:`~lava_test.core.parsers.TestParser.analyze_test_result` help
+method will do the sanity check and also perform the fixup of the result. 
+
+
 Maintaining out-of-tree tests
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -178,18 +261,13 @@
 Maintaining simple declarative tests
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     
-By registering pure declarative tests at runtime.
-
-.. todo::
-
-    Describe how to use declarative tests. It would be a nice
-    extension of the tutorial once the user feels comfortable with
-    the initial python-based version.
-
-Writing new tests from scratch 
-==============================
-
-.. todo::
-
-    Describe considerations for test writers. Using native test
-    format with human-readable output adapters.
+A declarative test is a json access able by lava-test. The following commands are used in a test scenario. ::
+
+ $ lava-test register http://abc.com/my-custom-test.json
+ $ lava-test install my-custom-test
+ $ lava-test run my-custom-test
+ $ lava-test uninstall my-custom-test
+ $ lava-test unregister http://abc.com/my-custom-test.json
+
+To see how the json file look like, let see the examples/stream.json. This file shows how to write the stream test in
+json format. Same test is written in python format: lava_test/test_definitions/stream.py.

=== modified file 'examples/stream.json'
--- examples/stream.json	2011-09-12 09:19:10 +0000
+++ examples/stream.json	2012-02-22 11:47:17 +0000
@@ -2,11 +2,13 @@
     "format": "LAVA-Test Test Definition Format",
     "test_id": "stream-json",
     "install": {
-        "url": "http://www.cs.virginia.edu/stream/FTP/Code/stream.c",
-        "steps": ["cc stream.c -O2 -fopenmp -o stream"]
+        "steps": ["cc stream.c -O2 -fopenmp -o stream"],
+        "deps": ['gcc', 'build-essential'],
+        "url": "http://www.cs.virginia.edu/stream/FTP/Code/stream.c"
     },
     "run": {
-        "steps": ["./stream"]
+        "steps": ["./stream"],
+        "default_options": ""
     },
     "parse": {
         "pattern": "^(?P<test_case_id>\\w+):\\W+(?P<measurement>\\d+\\.\\d+)",

=== modified file 'lava_test/core/runners.py'
--- lava_test/core/runners.py	2011-10-20 17:53:31 +0000
+++ lava_test/core/runners.py	2012-02-22 11:47:17 +0000
@@ -31,6 +31,10 @@
 
     :ivar steps:
         list of shell commands to execute
+
+    :ivar default_options:
+        The macro $(OPTIONS) in the run steps will be replaced by the command line argument -t
+        set $(OPTIONS) to this default options if the user not overwritten with the command line option -t
     """
     def __init__(self, steps=None, default_options=None):
         self.steps = steps or []

=== modified file 'lava_test/test_definitions/stream.py'
--- lava_test/test_definitions/stream.py	2011-10-20 12:12:15 +0000
+++ lava_test/test_definitions/stream.py	2012-02-22 11:47:17 +0000
@@ -18,18 +18,21 @@
 from lava_test.core.runners import TestRunner
 from lava_test.core.tests import Test
 
-URL="http://www.cs.virginia.edu/stream/FTP/Code/stream.c"
 INSTALLSTEPS = ['cc stream.c -O2 -fopenmp -o stream']
 DEPS = ['gcc', 'build-essential']
+URL="http://www.cs.virginia.edu/stream/FTP/Code/stream.c"
+
+install = TestInstaller(INSTALLSTEPS, deps=DEPS, url=URL)
+
 DEFAULT_OPTIONS = ""
 RUNSTEPS = ['./stream $(OPTIONS)']
+
+run     = TestRunner(RUNSTEPS,default_options=DEFAULT_OPTIONS)
+
 PATTERN = "^(?P<test_case_id>\w+):\W+(?P<measurement>\d+\.\d+)"
 
-streaminst = TestInstaller(INSTALLSTEPS, deps=DEPS, url=URL)
-streamrun = TestRunner(RUNSTEPS,default_options=DEFAULT_OPTIONS)
-streamparser = TestParser(PATTERN,
-               appendall={'units':'MB/s', 'result':'pass'})
-testobj = Test(test_id="stream", installer=streaminst,
-                                  runner=streamrun, parser=streamparser)
+parse   = TestParser(PATTERN, appendall={'units':'MB/s', 'result':'pass'})
+
+testobj = Test(test_id="stream", installer=install, runner=run, parser=parse)