Merge lp:~asc/fluidity/testharnessregex into lp:fluidity

[Adam Candy]

New feature for the testharness - specify test cases to consider by a regex using the option -s.
For example, the following runs all tests beginning with 'iceshelf' or 'shallow':
  testharness.py -s '^(iceshelf|shallow).*\.xml

[lp:~asc/fluidity/testharnessregex]

3571. By Adam Candy on 2011-09-05

Removal of ugly blank lines that had been allowed to creep in.

[Patrick Farrell (pefarrell)]

Is it possible to satisfy your use case with the test tag system instead?

[Adam Candy (asc)]

Ah yes, I did look at the tags feature but thought these were for more formal groupings such as 'parallel', 'zoltan' and 'flml', but I do now see there are some more informal groupings. I'll look at adding an iceshelf group. Thanks for the suggestion.

I think a pattern search on the test name is still a nice feature to include.

[David Ham (david-ham)]

I tend to disagree. Test names should be descriptive but they aren't
designed to categorise tests (which is what a pattern matching search
is effectively doing). Including this feature just encourages people
to misuse test names instead of using the proper tags.

On 5 September 2011 16:21, Adam Candy <email address hidden> wrote:
> Ah yes, I did look at the tags feature but thought these were for more formal groupings such as 'parallel', 'zoltan' and 'flml', but I do now see there are some more informal groupings.  I'll look at adding an iceshelf group.  Thanks for the suggestion.
>
> I think a pattern search on the test name is still a nice feature to include.
> --
> https://code.launchpad.net/~asc/fluidity/testharnessregex/+merge/73913
> Your team Fluidity Core Team is requested to review the proposed merge of lp:~asc/fluidity/testharnessregex into lp:fluidity.
>

--
Dr David Ham
Applied Modelling and Computation Group
Department of Earth Science and Engineering
Imperial College London

http://www.imperial.ac.uk/people/david.ham

Unmerged revisions

3571. By Adam Candy on 2011-09-05

Removal of ugly blank lines that had been allowed to creep in.

3570. By Adam Candy on 2011-09-02

New feature for the testharness - specify test cases to consider by a regex using the option -s.
For example, the following runs all tests beginning with 'iceshelf' or 'shallow':
  testharness.py -s '^(iceshelf|shallow).*\.xml

=== modified file 'tools/testharness.py'
--- tools/testharness.py	2011-08-05 11:19:06 +0000
+++ tools/testharness.py	2011-09-05 09:15:41 +0000
@@ -4,6 +4,7 @@
 import os
 import os.path
 import glob
+import re
 import time
 import regressiontest
 import traceback
@@ -15,7 +16,7 @@
 import elementtree.ElementTree as etree
 
 class TestHarness:
-    def __init__(self, length="any", parallel=False, exclude_tags=None, tags=None, file="", verbose=True, justtest=False,
+    def __init__(self, length="any", parallel=False, exclude_tags=None, tags=None, file="", search="", verbose=True, justtest=False,
         valgrind=False):
         self.tests = []
         self.verbose = verbose
@@ -38,6 +39,7 @@
           print "parallel: ", parallel
           print "tags to include: ", tags
           print "tags to exclude: ", exclude_tags
+          print "search: ", search
           print "-" * 80
           print 
 
@@ -84,13 +86,14 @@
           prob_defn = p.findall("problem_definition")[0]
           prob_length = prob_defn.attrib["length"]
           prob_nprocs = int(prob_defn.attrib["nprocs"])
-          if prob_length == length or (length == "any" and prob_length not in ["special", "long"]):
-            if self.parallel is True:
-              if prob_nprocs > 1:
-                working_set.append(xml_file)
-            else:
-              if prob_nprocs == 1:
-                working_set.append(xml_file)
+          if search is None or (search is not None and re.search(search, os.path.split(xml_file)[1]) is not None):
+            if prob_length == length or (length == "any" and prob_length not in ["special", "long"]):
+              if self.parallel is True:
+                if prob_nprocs > 1:
+                  working_set.append(xml_file)
+              else:
+                if prob_nprocs == 1:
+                  working_set.append(xml_file)
                 
         def get_xml_file_tags(xml_file):
           p = etree.parse(xml_file)
@@ -325,6 +328,7 @@
     parser.add_option("-e", "--exclude-tags", dest="exclude_tags", help="run only tests that do not have specific tags (takes precidence over -t)", default=[], action="append")
     parser.add_option("-t", "--tags", dest="tags", help="run tests with specific tags", default=[], action="append")
     parser.add_option("-f", "--file", dest="file", help="specific test case to run (by filename)", default="")
+    parser.add_option("-s", "--search", dest="search", help="regex on test case filenames, e.g. -s '^(iceshelf|shallow).*\.xml'", default="")
     parser.add_option("-n", "--threads", dest="thread_count", type="int",
                       help="number of tests to run at the same time", default=1)
     parser.add_option("-v", "--valgrind", action="store_true", dest="valgrind")
@@ -364,7 +368,7 @@
     else:
       tags = options.tags
 
-    testharness = TestHarness(length=options.length, parallel=para, exclude_tags=exclude_tags, tags=tags, file=options.file, verbose=True,
+    testharness = TestHarness(length=options.length, parallel=para, exclude_tags=exclude_tags, tags=tags, file=options.file, search=options.search, verbose=True,
         justtest=options.justtest, valgrind=options.valgrind)
 
     if options.justlist: