Merge lp:~deeptik/lava-test/openposix into lp:lava-test/0.0

[Deepti B. Kalakeri]

This patch automates the installation, execution, and parsing of posix test suite. This patch has been tested on Manderick for installation/execution and parsing of results.

[Paul Larson (pwlars)]

Please add the boilerplate gpl3 + linaro copyright at the top of the file. You can just copy/paste out of another file.

18 +URL="http://sourceforge.net/projects/posixtest/files/posixtest/" \
19 + "posixtestsuite-%s/posixtestsuite-%s.tar.gz" % (VERSION, VERSION)
You may actually want to look at grabbing the LTP source rather than openposix. One thing I noticed is that openposix hasn't had a release since 2006, but several fixes have gone into the LTP version since then.

23 +PATTERN = "((?P<test_case_id>\A(\w+[/]+)+(\d+[-]\d+):) (?P<message>\D+:)" \
24 + " (?P<result>\D+))"
this results in build, link, execution, etc being the "message". Is that what you intend? You don't have to have a message, and it might make more sense for this to be combined with the test_case_id. Not sure. I'll leave it to your judgement here as to what you think makes sense, based on looking at posixtest results.

46 + results[key] = results[key].rstrip("\n")
47 + results[key] = results[key].rstrip(":")
48 + results[key] = results[key].rstrip()
All this should not be necessary, adjust your regexp so that extraneous characters are outside of the ()'s

51 + self.results['test_results'].append(results)
This is causing you to get double entries for each test result

50 + results['test_case_id'] = test_case_id.replace("/", ".")
For bonus points, this is something we're going to have a problem with in other tests as well. I was just looking at this last week in relation to another test. I think we may be better off here making some changes to the way results are gathered to check for things like this, and handle bad characters automatically. That way, fewer tests would have to extend the AbrekResultsParser class. Feel free to take a look at this if you like, otherwise we can come back and fix this test once it's done.

review: Needs Fixing

[Deepti B. Kalakeri (deeptik)]

> Please add the boilerplate gpl3 + linaro copyright at the top of the file.
> You can just copy/paste out of another file.
>

Ok I was not aware of this. I will do it.

> 18 +URL="http://sourceforge.net/projects/posixtest/files/posixtest/" \
> 19 + "posixtestsuite-%s/posixtestsuite-%s.tar.gz" % (VERSION, VERSION)
> You may actually want to look at grabbing the LTP source rather than
> openposix. One thing I noticed is that openposix hasn't had a release since
> 2006, but several fixes have gone into the LTP version since then.
>
Ok I will try the posix test suite available with the ltp.

> 23 +PATTERN = "((?P<test_case_id>\A(\w+[/]+)+(\d+[-]\d+):)
> (?P<message>\D+:)" \
> 24 + " (?P<result>\D+))"
> this results in build, link, execution, etc being the "message". Is that what
> you intend? You don't have to have a message, and it might make more sense
> for this to be combined with the test_case_id. Not sure. I'll leave it to
> your judgement here as to what you think makes sense, based on looking at
> posixtest results.

I ran the posix test suite available with the ltp today and the results are recorded in a different format than previously, so I will have to change the regex. I will also check if keeping the message will make sense.
>
> 46 + results[key] = results[key].rstrip("\n")
> 47 + results[key] = results[key].rstrip(":")
> 48 + results[key] = results[key].rstrip()
> All this should not be necessary, adjust your regexp so that extraneous
> characters are outside of the ()'s

Will try to include this in the regex.
>
> 51 + self.results['test_results'].append(results)
> This is causing you to get double entries for each test result
>
oops! I did not notice this. Thanks for catching this.

> 50 + results['test_case_id'] = test_case_id.replace("/", ".")
> For bonus points, this is something we're going to have a problem with in
> other tests as well. I was just looking at this last week in relation to
> another test. I think we may be better off here making some changes to the
> way results are gathered to check for things like this, and handle bad
> characters automatically. That way, fewer tests would have to extend the
> AbrekResultsParser class. Feel free to take a look at this if you like,
> otherwise we can come back and fix this test once it's done.

[lp:~deeptik/lava-test/openposix]

32. By Deepti B. Kalakeri on 2010-09-22

Script to automate installation, execution, and parsing of posix test suite

[Deepti B. Kalakeri (deeptik)]

This patch addresses the review comments and includes the following changes:

1) Added the boilerplate gpl3 + linaro copyright
2) Modified the patch to include the openposix testsuite that is part of ltp
3) Improved the Regular expression to take care of the new format of the logs that is generated by openposix present in ltp
4) Verified that the parsing does not report duplicate records.

review: Needs Resubmitting

[Paul Larson (pwlars)]

Looks good, thanks!

review: Approve

=== added file 'abrek/test_definitions/posixtestsuite.py'
--- abrek/test_definitions/posixtestsuite.py	1970-01-01 00:00:00 +0000
+++ abrek/test_definitions/posixtestsuite.py	2010-09-22 12:52:45 +0000
@@ -0,0 +1,67 @@
+# Copyright (c) 2010 Linaro
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""
+   This script automates the automate installation, execution, and 
+   results parsing for the OpenPosix test suite. 
+   The POSIX Test Suite is an open source test suite with the goal of
+   performing conformance, functional, and stress testing of the IEEE
+   1003.1-2001 System Interfaces specification However the automation here 
+   does not support the stress test runs.
+
+"""
+import re
+import abrek.testdef
+
+VERSION="20100831"
+URL= "http://downloads.sourceforge.net/project/ltp/LTP Source/ltp-%s/"\
+     "ltp-full-%s.bz2" % (VERSION, VERSION)
+MD5="6982c72429a62f3917c13b2d529ad1ce"
+INSTALLSTEPS = ['tar -xjf ltp-full-20100831.bz2']
+RUNSTEPS = ['cd ltp-full-20100831/testcases/open_posix_testsuite/ && make']
+
+PATTERN = "((?P<test_case_id>\A(\w+[/]+)+\w+[-]*\w*[-]*\w*) .*? (?P<result>\w+))"
+FIXUPS = {   
+            "FAILED"      :  "fail",
+            "INTERRUPTED" :  "skip",
+            "PASSED"      :  "pass",
+            "UNRESOLVED"  :  "unknown",
+            "UNSUPPORTED" :  "skip",
+            "UNTESTED"    :  "skip",
+            "SKIPPING"    :  "skip"
+         }
+
+
+class PosixParser(abrek.testdef.AbrekTestParser):
+    def parse(self):
+        filename = "testoutput.log"
+        pat = re.compile(self.pattern)
+        with open(filename, 'r') as fd:
+            for line in fd.readlines():
+                match = pat.match(line)
+                if match:
+                    results = match.groupdict()
+                    test_case_id = results['test_case_id']
+                    results['test_case_id'] = test_case_id.replace("/", ".")
+                    self.results['test_results'].append(results)
+        if self.fixupdict:
+            self.fixresults(self.fixupdict)
+
+posix_inst = abrek.testdef.AbrekTestInstaller(INSTALLSTEPS, url=URL, md5=MD5)
+posix_run = abrek.testdef.AbrekTestRunner(RUNSTEPS)
+posixparser = PosixParser(PATTERN, fixupdict = FIXUPS)
+testobj = abrek.testdef.AbrekTest(testname="posixtestsuite", version=VERSION,
+                                  installer=posix_inst, runner=posix_run,
+                                  parser=posixparser)