Merge lp:~eeejay/mago/dry_run into lp:~mago-contributors/mago/mago-1.0

[Eitan Isaacson]

[Eitan Isaacson (eeejay)]

This branch does two things:
- Defines an environment variable, DT_PATH, that could be used to run tests from any working directory.
- Adds a --dry-run option that will simply log tests being run, but not actually run them.

[Javier Collado (javier.collado)]

Hello,

DT_PATH seems to be a useful option. Shouldn't be a good idea to add it as command-line option?

Regarding the dry_run option, despite it seems also useful, I'm not completely sure about the implementation because I think that the "dryrun" argument is passed in a deep hierarchy before being used. If I understood correctly, the option has been added just to log the name of the test cases and test suites that would be run. Then, maybe is a better option to separate the discovery part from the running part so that it isn't needed to pass the "dryrun" argument.

I'm just guessing so maybe what I'm saying it's not possible with the current code, but I would be thinking about something like:
# discovery part logs the suites and test cases found
test_suites = discover_test_suites(DT_PATH)
# running part runs the test cases for all test suites
if not dry_run:
    for test_suite in test_suites:
        test_suite.run()

Best regards,
    Javier

review: Needs Information

[Jason Cozens (jason-cozens)]

Hello,

I think these are good suggestions . . .

> but I would be thinking about something like:
> # discovery part logs the suites and test cases found
> test_suites = discover_test_suites(DT_PATH)

I definitely agree we need something like this. I was going to suggest an option such as --test-info that would return information about applications, test suites, test cases. Possibly with some options as to what is returned.

I think if we go down this route we would get a cleaner division between test discovery, test selection and running tests.

Cheers,
Jason.

[Eitan Isaacson (eeejay)]

I am not that excited of the way I implemented this either. I think the dry run option (as opposed to test discovery) option is useful, because a user could have a fairly long command that includes many suites and specific cases, then she could run a "dry run" to see that the right things will be run before actually running the tests.

I am not married to the "dry run" concept or the code, I just need a test discovery mechanism so we could integrate well with checkbox.

[Javier Collado (javier.collado)]

Hello,

Please take a look at the merge that I've just proposed:
https://code.launchpad.net/~javier.collado/ubuntu-desktop-testing/suite_discovery/+merge/7055

As it's mentioned in the merge proposal, there is still some work to be done in terms of the integration between the propose_suite_file function and the new discovery code, but I believe that this change is going to be beneficial.

Best regards,
    Javier

[Joker Wild (lajjr-deactivatedaccount)]

Nice features, but I think when it is implemented it will allow more features we are going in a good direction.
Regards,
Leo

review: Needs Information (test)

Unmerged revisions

80. By Eitan Isaacson on 2009-06-01

Made changes a bit more lightweight. Use loggin instead of printing to stdout.

79. By Eitan Isaacson on 2009-05-19

changed to DT_PATH, to avoid GDT/UDT diversion.

78. By Eitan Isaacson on 2009-05-19

Added UDT_PATH environment variable.

77. By Eitan Isaacson on 2009-05-19

Added a --dry-run option.

=== modified file 'bin/ubuntu-desktop-test'
--- bin/ubuntu-desktop-test	2009-05-05 11:10:25 +0000
+++ bin/ubuntu-desktop-test	2009-06-01 09:03:55 +0000
@@ -20,8 +20,10 @@
 from time import time, gmtime, strftime
 from shutil import move
 
+from desktoptesting.test_suite.main import TestSuite
+
 # Globals
-TESTS_SHARE = "."
+TESTS_SHARE = os.environ.get("DT_PATH", ".")
 TESTS_HOME  = os.environ["HOME"] + "/.ubuntu-desktop-test"
 SCREENSHOTS_SHARE = "/tmp/ldtp-screenshots"
 
@@ -168,13 +170,14 @@
 
     return suite_files
 
-def run_suite_file(suite_file, log_file, cases=None):
+def run_suite_file(suite_file, log_file, cases=None, dryrun=False):
     conf_file = os.path.join(TESTS_SHARE, "conffile.ini")
-    runner = TestSuiteRunner(suite_file)
+    runner = TestSuiteRunner(suite_file, dryrun=dryrun)
     results = runner.run(cases=cases)
-    f = open(log_file, 'w')
-    f.write(results)
-    f.close()
+    if not dryrun:
+        f = open(log_file, 'w')
+        f.write(results)
+        f.close()
 
 def convert_log_file(log_file):
 
@@ -208,7 +211,7 @@
         % (html_file, xsl_file, log_file)
     safe_run_command(command)
  
-def process_suite_file(suite_file, target_directory, cases=None):
+def process_suite_file(suite_file, target_directory, cases=None, dryrun=False):
     application_name = os.path.basename(os.path.dirname(suite_file))
     application_target = os.path.join(target_directory, application_name)
     safe_make_directory(application_target)
@@ -216,7 +219,7 @@
     suite_name = os.path.basename(suite_file)
     log_file = os.path.join(application_target,
         suite_name.replace(".xml", ".log"))
-    run_suite_file(suite_file, log_file, cases)
+    run_suite_file(suite_file, log_file, cases, dryrun)
 
     convert_log_file(log_file)
 
@@ -261,6 +264,11 @@
                       default=None,
                       help="Test cases to run (all, if not specified).")
 
+    parser.add_option("--dry-run",
+                      action="store_true",
+                      default=False,
+                      help="Dry run, just print suites and cases.")
+
     (options, args) = parser.parse_args(args[1:])
 
     # Set logging early
@@ -300,7 +308,8 @@
 
     # Run filtered suite file
     for suite_file in suite_files:
-        process_suite_file(suite_file, options.target, options.case)
+        process_suite_file(
+            suite_file, options.target, options.case, options.dry_run)
 
 
     return 0
@@ -349,20 +358,24 @@
 
 
 class TestCaseRunner(TestRunner):
-    def __init__(self, obj, xml_node):
+    def __init__(self, obj, xml_node, dryrun=False):
         TestRunner.__init__(self)
+        self.dryrun = dryrun
         self.xml_node = xml_node
         self.name = xml_node.getAttribute('name')
-        self.test_func = getattr(
-            obj, xml_node.getElementsByTagName('method')[0].firstChild.data)
+        method_name = \
+            xml_node.getElementsByTagName('method')[0].firstChild.data
+        self.test_func = getattr(obj, method_name, None)
         args_element = xml_node.getElementsByTagName('args')
         if args_element:
             self.get_args(args_element[0])
 
     def run(self, logger):
+        logging.debug("Running case %s" % self.name)
         starttime = time()
         try:
-            rv = self.test_func(**self.args)
+            if not self.dryrun:
+                rv = self.test_func(**self.args)
         except AssertionError, e:
             # The test failed.
             if len(e.args) > 1:
@@ -399,12 +412,15 @@
             self.set_result('time', time() - starttime)
         
         self.add_results_to_node(self.xml_node)
-        
+       
 class TestSuiteRunner(TestRunner):
-    def __init__(self, suite_file, loggerclass=None):
+    _test_case_runner_cls = TestCaseRunner
+    def __init__(self, suite_file, loggerclass=None, dryrun=False):
         TestRunner.__init__(self)
+        logging.debug("Loading suite file %s" % suite_file)
         self.dom = xml.dom.minidom.parse(suite_file)
         self._strip_whitespace(self.dom.documentElement)
+        self.name = self.dom.documentElement.getAttribute('name')
         clsname, modname = None, None
         case_runners = []
         for node in self.dom.documentElement.childNodes:
@@ -429,12 +445,17 @@
 
         cls = getattr(mod, clsname)
         
-        self.testsuite = cls(**self.args)
+        if dryrun:
+            self.testsuite = TestSuite()
+        else:
+            self.testsuite = cls(**self.args)
         
         self.case_runners = \
-            [TestCaseRunner(self.testsuite, c) for c in case_runners]
+            [self._test_case_runner_cls(self.testsuite, c, dryrun) \
+                 for c in case_runners]
     
     def run(self, loggerclass=None, setup_once=True, cases=None):
+        logging.debug("Running suite %s" % self.name)
         try:
             self._run(loggerclass, setup_once, cases)
         except Exception, e: