Merge lp:~javier.collado/mago/suite_discovery into lp:~mago-contributors/mago/mago-1.0

[Javier Collado]

[Javier Collado (javier.collado)]

Hello,

This branch is a refactoring of the code in ubuntu-desktop-test into a package called cmd that is part of the desktoptesting package. The code has been splitted in different modules to provide better maintainability.

In addition to this, some effort has been spent in writing new test discovery code. The summary of the changes is the following:
- Added -d, --directory option to append as many directories as needed where the test cases should be found
- Added -i, --info option to just display the name of the test cases that would be executed.
- Removed -f, --file options since it the functionality to filter by suite file is already covered by -s, --suite

As you'll see that the discovery is based on generators which means that it's being performed on demand (i.e. test cases aren't discovered until they are going to be executed) and that the time that it takes to start executing the first test case should be reduced (specially in an installation with a big set of test cases). I believe that other test runners such as nose are written using this design.

There's some overlapping between the process_suite_file function and the Case class (in discovery module) since both of them perform test case filtering. However, I expect to improve the integration of these two parts of code in the future since changing also that part would have made the change much bigger than I expected.

I hope you find this change useful.

Best regards,
    Javier

[Eitan Isaacson (eeejay)]

Ambitious, I like it. It is a major change, but it is the obvious next step.

One thing: when you talk about the overlap, do you mean that text execution and discovery don't follow the exact same code path? I think they should before this goes into trunk.

Also, I think an environment variable (besides -d) would be useful.

> Hello,
>
> This branch is a refactoring of the code in ubuntu-desktop-test into a package
> called cmd that is part of the desktoptesting package. The code has been
> splitted in different modules to provide better maintainability.
>
> In addition to this, some effort has been spent in writing new test discovery
> code. The summary of the changes is the following:
> - Added -d, --directory option to append as many directories as needed where
> the test cases should be found
> - Added -i, --info option to just display the name of the test cases that
> would be executed.
> - Removed -f, --file options since it the functionality to filter by suite
> file is already covered by -s, --suite
>
> As you'll see that the discovery is based on generators which means that it's
> being performed on demand (i.e. test cases aren't discovered until they are
> going to be executed) and that the time that it takes to start executing the
> first test case should be reduced (specially in an installation with a big set
> of test cases). I believe that other test runners such as nose are written
> using this design.
>
> There's some overlapping between the process_suite_file function and the Case
> class (in discovery module) since both of them perform test case filtering.
> However, I expect to improve the integration of these two parts of code in the
> future since changing also that part would have made the change much bigger
> than I expected.
>
> I hope you find this change useful.
>
> Best regards,
> Javier

[Javier Collado (javier.collado)]

Hello,

The overlapping is because the code in the discovery.py module parses the xml file to discover the test cases in it. This is what makes possible to do the following:
for app in apps:
    for suite in app.suites():
        for case in suite.cases():
            ....

On the other hand, the process_suite_file also parses the suite file to run the test suite and apply the same filtering (using -c command line options) so for a complete integration of the discovery mechanism I need to make some changes so that a Suite object, that takes care of the xml parsing, is used instead of a suite filename.

I don't expect this additional change to be very big, but I wanted to know you opinion about the whole idea before spending too much time in it. So unless you have any concern, I may continue working on this and ask for another review when the work in the branch is finished.

Also, I will add the environment variable so that it's used if present unless an option is specified, which will take precedence. Given the change in the project name, I think that a good name for the variable would be MAGO_PATH.

Do you agree on this (continue working and ask for a new review later and the name for the environment variable)?

Best regards,
    Javier

> Ambitious, I like it. It is a major change, but it is the obvious next step.
>
> One thing: when you talk about the overlap, do you mean that text execution
> and discovery don't follow the exact same code path? I think they should
> before this goes into trunk.
>
> Also, I think an environment variable (besides -d) would be useful.
>

[Eitan Isaacson (eeejay)]

I agree that these changes would be positive. Ara, what do you say?

> Hello,
>
> The overlapping is because the code in the discovery.py module parses the xml
> file to discover the test cases in it. This is what makes possible to do the
> following:
> for app in apps:
> for suite in app.suites():
> for case in suite.cases():
> ....
>
> On the other hand, the process_suite_file also parses the suite file to run
> the test suite and apply the same filtering (using -c command line options) so
> for a complete integration of the discovery mechanism I need to make some
> changes so that a Suite object, that takes care of the xml parsing, is used
> instead of a suite filename.
>
> I don't expect this additional change to be very big, but I wanted to know you
> opinion about the whole idea before spending too much time in it. So unless
> you have any concern, I may continue working on this and ask for another
> review when the work in the branch is finished.
>
> Also, I will add the environment variable so that it's used if present unless
> an option is specified, which will take precedence. Given the change in the
> project name, I think that a good name for the variable would be MAGO_PATH.
>
> Do you agree on this (continue working and ask for a new review later and the
> name for the environment variable)?
>
> Best regards,
> Javier
>
> > Ambitious, I like it. It is a major change, but it is the obvious next step.
> >
> > One thing: when you talk about the overlap, do you mean that text execution
> > and discovery don't follow the exact same code path? I think they should
> > before this goes into trunk.
> >
> > Also, I think an environment variable (besides -d) would be useful.
> >

[lp:~javier.collado/mago/suite_discovery]

64. By Javier Collado on 2009-06-08

process_suite_file changed to use suite objects instead of suite filenames (renamed to process_suite)
run_suite_file changed to use suite objects instead of suite filenames (renamed to run_suite)

65. By Javier Collado on 2009-06-08

Runner classes are not longer in charge of xml parsing and test case discovery

66. By Javier Collado on 2009-06-08

Fixed problem with result tags

67. By Javier Collado on 2009-06-08

Results storage moved to result.ResultDict class

68. By Javier Collado on 2009-06-08

Classes renamed to make it clear that they just contain data and not the implementation

69. By Javier Collado on 2009-06-08

Screenshot problem fixed

70. By Javier Collado on 2009-06-09

Removing cases parameter no longer used

71. By Javier Collado on 2009-06-09

New process_application function created to iterate directly over applications instead of over suites

72. By Javier Collado on 2009-06-09

If MAGO_PATH environment variable exists, it's used as the default list of directories to look for applications (unless -d option is passed explicitly)

[Javier Collado (javier.collado)]

Hello,

I've been worked a little more on the code and the integration is now complete:
- runner classes work with SuiteData or CaseData objects so they don't take care of discovery or xml parsing functionality
- MAGO_PATH environment variable is used as default list of directories (unless -d option is used)

Please take a look again at the code and let me know you opinion.

Best regards,
   Javier

[Eitan Isaacson (eeejay)]

I like it.. and need it.

review: Approve

[Eitan Isaacson (eeejay)]

Actually, logging seems to be broken. I am not seeing any logging output from safe_run_command()

debugging..

review: Needs Fixing

[Eitan Isaacson (eeejay)]

Wasn't a logger bug, here is the fix:

=== modified file 'desktoptesting/cmd/discovery.py'
--- desktoptesting/cmd/discovery.py 2009-06-09 07:23:10 +0000
+++ desktoptesting/cmd/discovery.py 2009-06-16 21:21:15 +0000
@@ -243,7 +243,7 @@
         # command line
         if cls.whitelist:
             suites = (suite for suite in suites
- if suite in cls.whitelist)
+ if suite.name in cls.whitelist)

         return suites

[Eitan Isaacson (eeejay)]

Here is another regression that I think needs fixing before merge:

2009-06-16 14:50:13,260 ERROR XSL file `/home/eitan/svn/canonical/udt/suite_discovery/share/ubuntu-desktop-tests/report.xsl' does not exist.

This has to do with the TESTS_SHARE global. I am not sure if it's value is a good choice. In any case, the XSL should probably be found using "-d" and MAGO_PATH.

[Joker Wild (lajjr-deactivatedaccount)]

If logging in implemented.

review: Approve (test)

[Javier Collado (javier.collado)]

Hello Eitan,

Thanks for your comments and your feedback. I'm afraid that the code is not free from defects so I really appreciate your help.

I've taken a look at that piece of code and the original version still looks fine to me. Let me explain it as clear as possible:
- suite is a SuiteData object that implements the __eq__ method for comparisons
- suite.name is the name of the test suite as written in the xml file
- suite.filename is the name of the xml file that contains the suite description data

My understanding was that in the main branch the suite filtering isn't performed by the suite name, but by the suite filename (with or without extension). Is that correct? In such a case that piece of code seems to be right since it's working that way thanks to the __eq__ method. For example:

(Pydb) suite
<desktoptesting.cmd.discovery.SuiteData instance at 0x8aead4c>
(Pydb) suite.name
'gedit chains'
(Pydb) suite.filename
'gedit_chains.xml'
(Pydb) suite in ['gedit chains']
False
(Pydb) suite in ['gedit_chains.xml']
True
(Pydb) suite in ['gedit_chains']
True

Please let me know if this is the way suite filtering is expected to work?

Anyway, I'll take a look at the safe_run_command output to make sure what happened to the logs.

Best regards,
    Javier

> Wasn't a logger bug, here is the fix:
>
> === modified file 'desktoptesting/cmd/discovery.py'
> --- desktoptesting/cmd/discovery.py 2009-06-09 07:23:10 +0000
> +++ desktoptesting/cmd/discovery.py 2009-06-16 21:21:15 +0000
> @@ -243,7 +243,7 @@
> # command line
> if cls.whitelist:
> suites = (suite for suite in suites
> - if suite in cls.whitelist)
> + if suite.name in cls.whitelist)
>
> return suites

[Javier Collado (javier.collado)]

Hello Eitan,

Since there are multiple directories that may contain test cases, I made the assumption that the xsl file is located in a path relative to the test runner script. For example, in my installation:
$ which ubuntu-desktop-test
/usr/local/bin/ubuntu-desktop-test
$ locate report.xsl | grep /usr/local
/usr/local/share/ubuntu-desktop-tests/report.xsl

So what I did is, at first, set TEST_SHARE to (globals.py):
TESTS_SHARE = "share/ubuntu-desktop-tests"

and once the script is called (main.py):
   globals.TESTS_SHARE = os.path.realpath(os.path.join(os.path.dirname(args[0]),
                                                        os.path.pardir,
                                                        globals.TESTS_SHARE))

To make this work, it is needed that arg[0] contains a full path so the following change was needed (ubuntu-desktop-test)
args[0] = __file__

From what I see in your log, you called the ubuntu-desktop-test script from the development branch instead of from an installed path and that's the reason the XSL file wasn't found (XSL relative path is different in development than in installation).

I could change the code in main.py to use a fallback path for the XSL file in case it's not found where expected. However, I'm not very fond of making a change to make code in the development branch work when the same code is working after being installed. Maybe we should try to look for a cleaner solution for the problem. I don't know if trying to incorporate the buildout changes from Markus would make this mismatch easier to solve.

Does anybody have any idea?

Best regards,
    Javier

> Here is another regression that I think needs fixing before merge:
>
> 2009-06-16 14:50:13,260 ERROR XSL file
> `/home/eitan/svn/canonical/udt/suite_discovery/share/ubuntu-desktop-
> tests/report.xsl' does not exist.
>
> This has to do with the TESTS_SHARE global. I am not sure if it's value is a
> good choice. In any case, the XSL should probably be found using "-d" and
> MAGO_PATH.

[Javier Collado (javier.collado)]

I've run the gedit test cases with debug log level and seen this:
2009-06-17 10:02:10,933 DEBUG Running command: xsltproc -o /home/javi/.ubuntu-desktop-test/gedit/gedit_chains.html /usr/local/share/ubuntu-desktop-tests/report.xsl /home/javi/.ubuntu-desktop-test/gedit/gedit_chains.log

Should I have seen something else? Maybe the problem you're having is related to the one with the XSL file.

> Actually, logging seems to be broken. I am not seeing any logging output from
> safe_run_command()
>
> debugging..

[Eitan Isaacson (eeejay)]

> Hello Eitan,
>
> Thanks for your comments and your feedback. I'm afraid that the code is not
> free from defects so I really appreciate your help.
>
> I've taken a look at that piece of code and the original version still looks
> fine to me. Let me explain it as clear as possible:
> - suite is a SuiteData object that implements the __eq__ method for
> comparisons
> - suite.name is the name of the test suite as written in the xml file
> - suite.filename is the name of the xml file that contains the suite
> description data
>
> My understanding was that in the main branch the suite filtering isn't
> performed by the suite name, but by the suite filename (with or without
> extension). Is that correct?

This has been correct. The main problem is that the "-i" option does not show suite files, but suite names. This defeats the purpose of the suites being discoverable, a user should be able to do this:

$ ubuntu-desktop-test -i
*suite/case list*
$ ubuntu-desktop-test -c <suite from list above>

I think both options should be available, by file or by suite. The suite filter is the most easy to use, but the file filter will allow things to not break when more than one suite have the same name.

> Please let me know if this is the way suite filtering is expected to work?

overriding __eq__ is fine, just see above.

> Anyway, I'll take a look at the safe_run_command output to make sure what
> happened to the logs.

That was my mistake, the logger is fine. The suite was simply not being run because I was supplying suite names, not files, see above.

[Javier Collado (javier.collado)]

To solve the problem, I think I could do the following:
- -s/--suite: Provides filtering for both file name and suite name

Display not only the name, but also the file name when displaying the suite information (also display the path for applications).

Would that be OK?

[Eitan Isaacson (eeejay)]

>
> I could change the code in main.py to use a fallback path for the XSL file in
> case it's not found where expected. However, I'm not very fond of making a
> change to make code in the development branch work when the same code is
> working after being installed.

I think being able to run things out of the build path is very important. And yes, the solution is probably hackish. Maybe search for the XSL in the parent directory of the suite?

[Eitan Isaacson (eeejay)]

> To solve the problem, I think I could do the following:
> - -s/--suite: Provides filtering for both file name and suite name

Maybe -f/--file? or -x/--xml? Having -s do both is confusing.

>
> Display not only the name, but also the file name when displaying the suite
> information (also display the path for applications).
>
That sounds great.

> Would that be OK?

medio :)

[lp:~javier.collado/mago/suite_discovery]

73. By Javier Collado on 2009-06-18

Suites may be filtered by either name or filename depending on the command line option used (-n/--suite_name, -f/--suite_file)

[Eitan Isaacson (eeejay)]

Ok, I have been thinking about this, and here is how I think the directory mess could be simplified:

1. Get rid of -d, it is confusing that there is more than one way to set the directory.
2. globals.TESTS_SHARE turns into globals.MAGO_SHARE, unless a environment variable of the same name overrides it, it points to $(prefix)/share/mago.
3. A new variable called globals.MAGO_PATH is introduced that is a list of paths to search for tests in. By default it is: [os.path.curdir, os.path.join(MAGO_SHARE, 'tests')]. It could also be overridden with a colon-delimited environment variable with the same name.

Some pseudo-code:

def _get_share_dir():
  # Return installed share directory path, or current build directory if not installed.

MAGO_SHARE = os.env.get('MAGO_SHARE', _get_share_dir())

if os.env.get('MAGO_PATH', None):
  MAGO_PATH = os.env['MAGO_PATH'].split(':')
else:
  MAGO_PATH = [os.path.curdir, os.path.join(MAGO_SHARE, 'tests')]

[Eitan Isaacson (eeejay)]

Just implemented in lp:~eeejay/mago/suite_discovery

Please review/merge/comment here.

[lp:~javier.collado/mago/suite_discovery]

74. By Javier Collado on 2009-06-19

Taking advantage of the testing_install_scripts class to set TESTS_SHARE global variable properly in installed code, while the original source code works fine directly in the branch

[Javier Collado (javier.collado)]

Hello Eitan,

I'm afraid I still don't feel comfortable with having a piece of code whose aim is to check if we are in a branch or in an installed path.

When I was thinking about the questions I sent to the mail list regarding the additional code in setup.py, I saw that another solution was already there. What we really need is to have a global variable that has a value for the code that is in the branch and another one for the installed code. This way, the code is the same (which is property a like) in both cases despite the value of the TESTS_SHARE global variable isn't. This can be accomplished using the testing_install_scripts class in setup.py. Please take a look at the last version of the branch and let me know what do you think.

Best regards,
   Javier

> Just implemented in lp:~eeejay/mago/suite_discovery
>
> Please review/merge/comment here.

[lp:~javier.collado/mago/suite_discovery]

75. By Javier Collado on 2009-06-19

Removing code that is no longer needed

[Eitan Isaacson (eeejay)]

> Hello Eitan,
>
> I'm afraid I still don't feel comfortable with having a piece of code whose
> aim is to check if we are in a branch or in an installed path.
>

I don't think we will agree here. The variable substitution you suggest seems error prone on many levels, and easily overlooked. While the solution I suggested might not be entirely bomb-proof, I think it is fairly robust, in one file, and easy to read.

Did you see my three points above? I think those are important. As to how the share dir is found, I will leave that up to you. In any case, I needed this in trunk last week, for the checkbox work.

[Eitan Isaacson (eeejay)]

Another suggestion: Let's not recursively search for tests. Since we could define arbitrary paths to search in, a user might put "/" in by mistake, or even just her home directory. I noticed that discovery takes a long time when running from $HOME. A depth of one is good enough, in my opinion.

Unmerged revisions

75. By Javier Collado on 2009-06-19

Removing code that is no longer needed

74. By Javier Collado on 2009-06-19

Taking advantage of the testing_install_scripts class to set TESTS_SHARE global variable properly in installed code, while the original source code works fine directly in the branch

=== modified file 'bin/ubuntu-desktop-test'
--- bin/ubuntu-desktop-test	2009-05-05 11:10:25 +0000
+++ bin/ubuntu-desktop-test	2009-06-03 10:12:35 +0000
@@ -1,502 +1,9 @@
 #!/usr/bin/env python
-
-import os
-
-os.environ['NO_GAIL'] = '1'
-os.environ['NO_AT_BRIDGE'] = '1'
-
-import re
-import sys
-import logging
-import xml.dom.minidom
-import ldtp
-import ldtputils
-import traceback
-
-from logging import StreamHandler, FileHandler, Formatter
-from optparse import OptionParser
-from stat import ST_MODE, S_IMODE
-from subprocess import Popen, PIPE
-from time import time, gmtime, strftime
-from shutil import move
-
-# Globals
-TESTS_SHARE = "."
-TESTS_HOME  = os.environ["HOME"] + "/.ubuntu-desktop-test"
-SCREENSHOTS_SHARE = "/tmp/ldtp-screenshots"
-
-# For the moment, all applications should be running in English
-ldtp.setlocale("C")
-
-
-def error(message, *args):
-    message = "Error: %s\n" % message
-    sys.stderr.write(message % args)
-    sys.exit(1)
-
-def safe_change_mode(path, mode):
-    if not os.path.exists(path):
-        error("Path does not exist: %s", path)
-
-    old_mode = os.stat(path)[ST_MODE]
-    if mode != S_IMODE(old_mode):
-        os.chmod(path, mode)
-
-def safe_make_directory(path, mode=0755):
-    if os.path.exists(path):
-        if not os.path.isdir(path):
-            error("Path is not a directory: %s", path)
-
-        safe_change_mode(path, mode)
-    else:
-        logging.debug("Creating directory: %s", path)
-        os.makedirs(path, mode)
-
-def safe_run_command(command):
-    logging.debug("Running command: %s" % command)
-    p = Popen(command, stdout=PIPE, shell=True)
-    (pid, status) = os.waitpid(p.pid, 0)
-    if status:
-        error("Command failed: %s", command)
-
-    return p.stdout.read()
-
-def is_valid_application_directory(application_directory, applications):
-    application = os.path.basename(application_directory)
-    if applications is not None and application not in applications:
-        logging.debug("Application name `%s' not in specified options: %s",
-            application, ", ".join(applications))
-        return False
-
-    pattern = r"[a-z0-9][-_a-z0-9+.]*"
-    if not re.match(pattern, application, re.I):
-        logging.debug("Application name `%s' does not match pattern: %s",
-            application, pattern)
-        return False
-
-    if not os.path.isdir(application_directory):
-        logging.debug("Application directory `%s' is not a directory",
-            application_directory)
-        return False
-
-    return True
-
-def is_valid_suite(suite_file, suites, files):
-   
-    # If the user has specified a file, but not a suite, skip
-    if suites is None and files is not None:
-        return False
-    
-    # Support specifying suites with or without the extension
-    # but do not include backup files LP#326177
-    if suites is not None:
-        suites_clean = []
-        for s in suites:
-            if os.path.splitext(s)[1] is '':
-                suites_clean.append(s + ".xml")
-            else:
-                suites_clean.append(s)
-    
-        suites = suites_clean
-
-    suite = os.path.basename(suite_file)
-    if suites is not None and suite not in suites:
-        logging.debug("Suite name `%s' not in specified options: %s",
-            suite, ", ".join(suites))
-        return False
-
-    pattern = r"[a-z0-9][-_a-z0-9+.]*.xml$"
-    if not re.match(pattern, suite, re.I):
-        logging.debug("Suite name `%s' does not match pattern: %s",
-            suite, pattern)
-        return False
-
-    if not os.path.isfile(suite_file):
-        logging.debug("Suite file `%s' is not a file",
-            suite_file)
-        return False
-
-    return True
-
-def is_valid_suite_file(file, suites):
-
-    suite = os.path.basename(file)
-
-    pattern = r"[a-z0-9][-_a-z0-9+.]*.xml$"
-    if not re.match(pattern, suite, re.I):
-        logging.debug("Suite name `%s' does not match pattern: %s",
-            file, pattern)
-        return False
-
-    if not os.path.isfile(file):
-        logging.debug("Suite file `%s' is not a file",
-            file)
-        return False
-
-    if file in suites:
-        logging.debug("Suite file is already in the list",
-                file)
-        return False
-    
-    return True
-
-
-def filter_suite_files(applications, suites, files, folder):
-
-    # Filter applications
-    suite_files = []
-   
-    if folder is not None:
-        for application in os.listdir(folder):
-            application_directory = os.path.join(folder, application)
-            if not is_valid_application_directory(application_directory, applications):
-                continue
-
-            # Filter suites
-            for suite in os.listdir(application_directory):
-                suite_file = os.path.join(application_directory, suite)
-                if not is_valid_suite(suite_file, suites, files):
-                    continue
-
-                suite_files.append(suite_file)
-
-    if files is not None and folder is None:
-        for file in files:
-            if not is_valid_suite_file(file, suite_files):
-                continue
-            suite_files.append(file)
-
-    return suite_files
-
-def run_suite_file(suite_file, log_file, cases=None):
-    conf_file = os.path.join(TESTS_SHARE, "conffile.ini")
-    runner = TestSuiteRunner(suite_file)
-    results = runner.run(cases=cases)
-    f = open(log_file, 'w')
-    f.write(results)
-    f.close()
-
-def convert_log_file(log_file):
-
-    if os.path.exists(SCREENSHOTS_SHARE):
-        screenshot_dir = os.path.dirname(log_file) + "/screenshots"
-
-        if not os.path.exists(screenshot_dir):            
-            safe_make_directory(screenshot_dir)
-
-        if len(os.listdir(SCREENSHOTS_SHARE)) > 0:
-            command = "mv " + SCREENSHOTS_SHARE + "/* " + screenshot_dir
-            safe_run_command(command)
-
-    log_file_tmp = log_file + ".tmp"
-    o = open(log_file_tmp, "w")
-    data = open(log_file).read()
-    o.write(re.sub(SCREENSHOTS_SHARE, "screenshots", data))
-    o.flush()
-    o.close()
-    
-    os.remove(log_file)
-    os.rename(log_file_tmp, log_file)
-
-    xsl_file = os.path.join(TESTS_SHARE, "report.xsl")
-    if not os.path.exists(xsl_file):
-        error("XSL file `%s' does not exist.", xsl_file)
-
-    html_file = log_file.replace(".log", ".html")
-
-    command = "xsltproc -o %s %s %s" \
-        % (html_file, xsl_file, log_file)
-    safe_run_command(command)
- 
-def process_suite_file(suite_file, target_directory, cases=None):
-    application_name = os.path.basename(os.path.dirname(suite_file))
-    application_target = os.path.join(target_directory, application_name)
-    safe_make_directory(application_target)
-
-    suite_name = os.path.basename(suite_file)
-    log_file = os.path.join(application_target,
-        suite_name.replace(".xml", ".log"))
-    run_suite_file(suite_file, log_file, cases)
-
-    convert_log_file(log_file)
-
-def main(args=sys.argv):
-    usage = "%prog [OPTIONS]"
-    parser = OptionParser(usage=usage)
-
-    default_target = "~/.ubuntu-desktop-test"
-    default_log_level = "critical"
-
-    parser.add_option("-l", "--log",
-                      metavar="FILE",
-                      help="The file to write the log to.")
-    parser.add_option("--log-level",
-                      default=default_log_level,
-                      help="One of debug, info, warning, error or critical.")
-    parser.add_option("-a", "--application",
-                      action="append",
-                      type="string",
-                      default=None,
-                      help="Application name to test. Option can be repeated "
-                           "and defaults to all applications")
-    parser.add_option("-s", "--suite",
-                      action="append",
-                      type="string",
-                      default=None,
-                      help="Suite name to test within applications. Option "
-                           "can be repeated and default to all suites")
-    parser.add_option("-f", "--file",
-                      action="append",
-                      type="string",
-                      default=None,
-                      help="XML file name of the suite to test within applications.")
-    parser.add_option("-t", "--target",
-                      metavar="FILE",
-                      default=default_target,
-                      help="Target directory for logs and reports. Defaults "
-                           "to: %default")
-    parser.add_option("-c", "--case",
-                      action="append",
-                      type="string",
-                      default=None,
-                      help="Test cases to run (all, if not specified).")
-
-    (options, args) = parser.parse_args(args[1:])
-
-    # Set logging early
-    log_level = logging.getLevelName(options.log_level.upper())
-    log_handlers = []
-    log_handlers.append(StreamHandler())
-    if options.log:
-        log_filename = options.log
-        log_handlers.append(FileHandler(log_filename))
-
-    format = ("%(asctime)s %(levelname)-8s %(message)s")
-    if log_handlers:
-        for handler in log_handlers:
-            handler.setFormatter(Formatter(format))
-            logging.getLogger().addHandler(handler)
-        if log_level:
-            logging.getLogger().setLevel(log_level)
-    elif not logging.getLogger().handlers:
-        logging.disable(logging.CRITICAL)
-
-    options.target = os.path.expanduser(options.target)
-    if os.path.exists(options.target) and not os.path.isdir(options.target):
-        parser.error("Target directory `%s' exists but is not a directory.",
-            options.target)
-
-    # Filter suite files from project directory
-
-    if not os.path.isdir(TESTS_SHARE): 
-        error("Share directory `%s' is not a directory.", TESTS_SHARE)
-    else:
-        suite_files = filter_suite_files(options.application, options.suite, options.file, TESTS_SHARE)
-    
-    if os.path.isdir(TESTS_HOME):
-        suite_files += filter_suite_files(options.application, options.suite, options.file, TESTS_HOME)
-
-    suite_files += filter_suite_files(options.application, options.suite, options.file, None)
-
-    # Run filtered suite file
-    for suite_file in suite_files:
-        process_suite_file(suite_file, options.target, options.case)
-
-
-    return 0
-
-class TestRunner:
-    def __init__(self):
-        self.result = {}
-        self.args = {}
-
-    def get_args(self, node):
-        for n in node.childNodes:
-            if n.nodeType != n.ELEMENT_NODE or not n.hasChildNodes():
-                continue
-            self.args[n.tagName.encode('ascii')] = n.firstChild.data
-
-    def add_results_to_node(self, node):
-        if self.result == {}: return
-        result = node.ownerDocument.createElement("result")
-        for key, val in self.result.items():
-            if not isinstance(val, list):
-                val = [val]
-            for item in val:
-                n = node.ownerDocument.createElement(key)
-                n.appendChild(n.ownerDocument.createTextNode(str(item)))
-                result.appendChild(n)
-                
-        node.appendChild(result)
-
-    def append_result(self, key, value):
-        l = self.result.get(key, [])
-        l.append(value)
-        self.result[key] = l
-
-    def set_result(self, key, value):
-        self.result[key] = value
-
-    def append_screenshot(self, screenshot_file=None):
-        _logFile = "%s/screenshot-%s.png" % (SCREENSHOTS_SHARE,
-                                             strftime ("%m-%d-%Y-%H-%M-%s"))
-        safe_make_directory(SCREENSHOTS_SHARE)
-        if screenshot_file is None:
-            ldtputils.imagecapture(outFile = _logFile)
-        else:
-            move(screenshot_file, _logFile)
-        self.append_result('screenshot', _logFile)
-
-
-class TestCaseRunner(TestRunner):
-    def __init__(self, obj, xml_node):
-        TestRunner.__init__(self)
-        self.xml_node = xml_node
-        self.name = xml_node.getAttribute('name')
-        self.test_func = getattr(
-            obj, xml_node.getElementsByTagName('method')[0].firstChild.data)
-        args_element = xml_node.getElementsByTagName('args')
-        if args_element:
-            self.get_args(args_element[0])
-
-    def run(self, logger):
-        starttime = time()
-        try:
-            rv = self.test_func(**self.args)
-        except AssertionError, e:
-            # The test failed.
-            if len(e.args) > 1:
-                self.append_result('message', e.args[0])
-                self.append_screenshot(e.args[1])
-            else:
-                self.append_result('message', str(e))
-                self.append_screenshot()
-            self.append_result('stacktrace', traceback.format_exc())
-            self.set_result('pass', 0)
-        except Exception, e:
-            # There was an unrelated error.
-            logging.warning(traceback.format_exc())
-            if len(e.args) > 1:
-                self.append_result('message', e.args[0])
-                self.append_screenshot(e.args[1])
-            else:
-                self.append_result('message', str(e))
-                self.append_screenshot()
-            self.append_result('stacktrace', traceback.format_exc())
-            self.set_result('error', 1)
-        else:
-            self.set_result('pass', 1)
-            try:
-                message, screenshot = rv
-            except:
-                pass
-            else:
-                if message:
-                    self.append_result('message', message)
-                if screenshot:
-                    self.append_screenshot(screenshot)
-        finally:
-            self.set_result('time', time() - starttime)
-        
-        self.add_results_to_node(self.xml_node)
-        
-class TestSuiteRunner(TestRunner):
-    def __init__(self, suite_file, loggerclass=None):
-        TestRunner.__init__(self)
-        self.dom = xml.dom.minidom.parse(suite_file)
-        self._strip_whitespace(self.dom.documentElement)
-        clsname, modname = None, None
-        case_runners = []
-        for node in self.dom.documentElement.childNodes:
-            if node.nodeType != node.ELEMENT_NODE:
-                continue
-            if node.tagName == 'class':
-                modname, clsname = node.firstChild.data.rsplit('.', 1)
-            elif node.tagName == 'case':
-                logging.debug("Adding case %s to current test suite.", 
-                              node.getAttribute("name"))
-                case_runners.append(node)
-            elif node.tagName == 'args':
-                self.get_args(node)
-        if None in (clsname, modname):
-            raise Exception, "Missing a suite class"
-
-        # Suite file and module are suppose to be in the same directory
-        sys.path.insert(1, os.path.dirname(suite_file))
-        logging.debug("Modname: %s", modname)
-        mod = __import__(modname)
-        sys.path.pop(1)
-
-        cls = getattr(mod, clsname)
-        
-        self.testsuite = cls(**self.args)
-        
-        self.case_runners = \
-            [TestCaseRunner(self.testsuite, c) for c in case_runners]
-    
-    def run(self, loggerclass=None, setup_once=True, cases=None):
-        try:
-            self._run(loggerclass, setup_once, cases)
-        except Exception, e:
-            logging.warning(traceback.format_exc())
-            # There was an unrelated error.
-            self.append_result('message', str(e))
-            self.append_result('stacktrace', traceback.format_exc())
-            self.set_result('error', 1)
-            try:
-                self.testsuite.teardown()
-            except:
-                pass
-
-        self.add_results_to_node(self.dom.documentElement)
-
-        return self.dom.toprettyxml('  ')
-            
-    def _run(self, loggerclass, setup_once, cases):
-        if loggerclass:
-            logger = loggerclass()
-        else:
-            logger = ldtp
-
-        if setup_once:
-            # Set up the environment.
-            self.testsuite.setup()
-
-        firsttest = True
-        for testcase in self.case_runners:
-            if cases and testcase.name not in cases:
-                continue
-            if not setup_once:
-                # Set up the app for each test, if requested.
-                self.testsuite.setup()
-            if not firsttest:
-                # Clean up from previous run.
-                self.testsuite.cleanup()
-            firsttest = False
-            testcase.run(logger)
-            if not setup_once:
-                # Teardown upthe app for each test, if requested.
-                self.testsuite.teardown()
-
-        if setup_once:
-            # Tear down after entire suite.
-            self.testsuite.teardown()
-
-    def _strip_whitespace(self, element):
-        if element is None: 
-            return
-        sibling = element.firstChild
-        while sibling:
-            nextSibling = sibling.nextSibling
-            if sibling.nodeType == sibling.TEXT_NODE:
-                stripped = sibling.data.strip()
-                if stripped == '':
-                    element.removeChild(sibling)
-                else:
-                    sibling.data = stripped
-            else:
-                self._strip_whitespace(sibling)
-            sibling = nextSibling
+import sys, os
+from desktoptesting.cmd.main import main
 
 if __name__ == "__main__":
-    sys.exit(main())
+    # Pass complete file name as program name
+    args = sys.argv
+    args[0] = __file__
+    sys.exit(main(args))

=== added directory 'desktoptesting/cmd'
=== added file 'desktoptesting/cmd/__init__.py'
=== added file 'desktoptesting/cmd/discovery.py'
--- desktoptesting/cmd/discovery.py	1970-01-01 00:00:00 +0000
+++ desktoptesting/cmd/discovery.py	2009-06-04 04:18:42 +0000
@@ -0,0 +1,226 @@
+"""
+This module contains the functionality related to the discovery of the test suites
+"""
+import os, re, logging
+import xml.etree.ElementTree as etree
+
+
+class Application:
+    """
+    Application description data
+    """
+    name_pattern = r"[a-z0-9][-_a-z0-9+.]*"
+    name_regex = re.compile(name_pattern)
+    whitelist = None
+
+    def __init__(self, path, filenames):
+        self.path = path
+        self.filenames = filenames
+
+        self.name = os.path.basename(path)
+
+
+    def __eq__(self, other):
+        """
+        Two applications are considered to be equal if they have the
+        same name
+        """
+        return (type(self) == type(other)
+                and self.name == other.name)
+
+
+    def name_matches(self):
+        """
+        Return True if the application name
+        honors the expected pattern
+        """
+        return self.name_regex.match(self.name)
+
+
+    def suites(self):
+        """
+        Return a generator for all suites
+        """
+        return Suite.discover(self)
+
+
+    @classmethod
+    def discover(cls, base_dirpaths):
+        """
+        Generator that discovers all applications under
+        a list of top directories
+        """
+        discovered_applications = []
+
+        for base_dirpath in base_dirpaths:
+            for dirpath, dirnames, filenames in os.walk(base_dirpath):
+                # Application directories are expected to honor
+                # the specified name pattern
+                app = cls(dirpath, filenames)
+                if not app.name_matches():
+                    logging.debug("Application name %s does not match pattern: %s"
+                                  % (app.name, app.name_pattern))
+                    continue
+
+                # This check makes sure that the same application
+                # isn't discovered twice. That is to say, when discovering
+                # applications from multiple directories, the test cases 
+                # from the application that is first found will be
+                # executed while the others will be discarded
+                if app in discovered_applications:
+                    logging.debug("Application name %s has been already discovered"
+                                  % app.name)
+                    continue
+
+                # Return application only if there is no whitelist
+                # or if it matches any of the whitelist names
+                if cls.whitelist and not app.name in cls.whitelist:
+                    logging.debug("Application name %s has not been whitelisted"
+                                  % app.name)
+                    continue
+
+                # At least one '.xml' file with a 'suite' root tag
+                # should be contained in the directory to be a valid application directory
+                if not any(app.suites()):
+                    logging.debug("Application directory %s does't seem to contain a valid suite file"
+                                  % app.path)
+                    continue
+
+                discovered_applications.append(app)
+                yield app
+
+
+class Suite:
+    """
+    Suite description data
+    """
+    whitelist = None
+
+    def __init__(self, application, filename):
+        self.application = application
+        self.filename = filename
+        self.fullname = os.path.join(application.path,
+                                     filename)
+
+        try:
+            self.tree = etree.parse(self.fullname)
+        except:
+            self.tree = None
+
+
+    @property
+    def name(self):
+        """
+        Return suite name as written in the xml file
+        """
+        return self.tree.getroot().attrib['name']
+
+
+    def cases(self):
+        """
+        Generator for all test cases in the suite
+        """
+        return Case.discover(self)
+
+
+    def __eq__(self, other):
+        """
+        A suite is compared against its filename
+        or against its own name (useful for filtering)
+        """
+        if type(other) == str:
+            other_name, other_ext = os.path.splitext(other)
+            
+            if other_ext:
+                return other == self.filename
+            else:
+                return other_name == os.path.splitext(self.filename)[0]
+        else:
+            return self.id == other.id
+
+    
+    def has_valid_xml(self):
+        """
+        Return true if xml could be parsed
+        and the root tag is 'suite'
+        """
+        return (self.tree
+                and self.tree.getroot().tag == 'suite')
+
+
+    @classmethod
+    def discover(cls, app):
+        """
+        Discover suites inside of an application
+        """
+        # All test suites will be defined by an xml file
+        xml_filenames = (filename
+                         for filename in app.filenames
+                         if filename.endswith('xml'))
+
+        # Discovered suites must contain a valid xml content
+        # and at least one test cases
+        suites = (suite for suite in (cls(app, filename)
+                                      for filename in xml_filenames)
+                  if suite.has_valid_xml() and any(suite.cases()))
+
+        
+        # Filter suites using the whitelist provide through the
+        # command line
+        if cls.whitelist:
+            suites = (suite for suite in suites
+                      if suite in cls.whitelist)
+
+        return suites
+
+
+class Case:
+    """
+    Test case description data
+    """
+    whitelist = None
+
+    def __init__(self, suite, case_tag):
+        self.suite = suite
+        self.case_tag = case_tag
+
+
+    @property
+    def name(self):
+        """
+        Return test case name
+        """
+        return self.case_tag.attrib['name']
+
+
+    @classmethod
+    def discover(cls, suite):
+        """
+        Discover all test cases in a suite
+        """
+        cases = (cls(suite, case_tag)
+                 for case_tag in suite.tree.findall('case'))
+
+        if cls.whitelist:
+            cases = (case for case in cases
+                     if case.name in cls.whitelist)
+
+        return cases
+
+
+def discover_applications(top_directories,
+                          filtering_applications,
+                          filtering_suites,
+                          filtering_cases):
+    """
+    Discover all applications and filter them properly
+    """
+    # Configure filtering options
+    Application.whitelist = filtering_applications
+    Suite.whitelist = filtering_suites
+    Case.whitelist = filtering_cases
+
+    # Discover all applications under top directories
+    discovered_apps = Application.discover(top_directories)
+
+    return discovered_apps

=== added file 'desktoptesting/cmd/globals.py'
--- desktoptesting/cmd/globals.py	1970-01-01 00:00:00 +0000
+++ desktoptesting/cmd/globals.py	2009-06-03 12:29:19 +0000
@@ -0,0 +1,6 @@
+"""
+This module just provides a namespace for global variables used across
+multiple modules in the package
+"""
+TESTS_SHARE = "share/ubuntu-desktop-tests"
+SCREENSHOTS_SHARE = "/tmp/ldtp-screenshots"

=== added file 'desktoptesting/cmd/main.py'
--- desktoptesting/cmd/main.py	1970-01-01 00:00:00 +0000
+++ desktoptesting/cmd/main.py	2009-06-04 04:15:25 +0000
@@ -0,0 +1,136 @@
+"""
+This module provides the main entry point for the execution automated
+desktop test cases
+"""
+
+import os, re, sys, logging
+from logging import StreamHandler, FileHandler, Formatter
+from itertools import chain
+import ldtp
+
+from . import globals
+from .runner import TestSuiteRunner
+from .parser import parse_options
+from .utils import safe_make_directory, safe_run_command
+from .discovery import discover_applications
+
+
+def run_suite_file(suite_file, log_file, cases=None):
+    logging.info("Running %s" % suite_file)
+    runner = TestSuiteRunner(suite_file)
+    results = runner.run(cases=cases)
+    f = open(log_file, 'w')
+    f.write(results)
+    f.close()
+
+
+def convert_log_file(log_file, target_directory):
+    if os.path.exists(globals.SCREENSHOTS_SHARE):
+        screenshot_dir = os.path.dirname(log_file) + "/screenshots"
+
+        if not os.path.exists(screenshot_dir):            
+            safe_make_directory(screenshot_dir)
+
+        if len(os.listdir(globals.SCREENSHOTS_SHARE)) > 0:
+            command = "mv " + globals.SCREENSHOTS_SHARE + "/* " + screenshot_dir
+            safe_run_command(command)
+
+    log_file_tmp = log_file + ".tmp"
+    o = open(log_file_tmp, "w")
+    data = open(log_file).read()
+    o.write(re.sub(globals.SCREENSHOTS_SHARE, "screenshots", data))
+    o.flush()
+    o.close()
+    
+    os.remove(log_file)
+    os.rename(log_file_tmp, log_file)
+
+    xsl_file = os.path.join(globals.TESTS_SHARE, "report.xsl")
+    if not os.path.exists(xsl_file):
+        logging.error("XSL file `%s' does not exist." % xsl_file)
+        sys.exit(1)
+
+    html_file = log_file.replace(".log", ".html")
+
+    command = "xsltproc -o %s %s %s" \
+        % (html_file, xsl_file, log_file)
+    safe_run_command(command)
+ 
+
+def process_suite_file(suite_file, target_directory, cases=None):
+    application_name = os.path.basename(os.path.dirname(suite_file))
+    application_target = os.path.join(target_directory, application_name)
+    safe_make_directory(application_target)
+
+    suite_name = os.path.basename(suite_file)
+    log_file = os.path.join(application_target,
+                            suite_name.replace(".xml", ".log"))
+    run_suite_file(suite_file, log_file, cases)
+    convert_log_file(log_file, target_directory)
+
+
+def configure_logging(log_level_str, log_filename):
+    """
+    Configure log handlers
+    """
+    log_level = logging.getLevelName(log_level_str.upper())
+    log_handlers = []
+    log_handlers.append(StreamHandler())
+    if log_filename:
+        log_handlers.append(FileHandler(log_filename))
+
+    format = ("%(asctime)s %(levelname)-8s %(message)s")
+    if log_handlers:
+        for handler in log_handlers:
+            handler.setFormatter(Formatter(format))
+            logging.getLogger().addHandler(handler)
+        if log_level:
+            logging.getLogger().setLevel(log_level)
+    elif not logging.getLogger().handlers:
+        logging.disable(logging.CRITICAL)
+
+
+def main(args=sys.argv):
+    """
+    Execute automated tests
+    """
+    os.environ['NO_GAIL'] = '1'
+    os.environ['NO_AT_BRIDGE'] = '1'
+
+    # For the moment, all applications should be running in English
+    ldtp.setlocale("C")
+
+    # Get shared directory based on the directory in which binary is located
+    globals.TESTS_SHARE = os.path.realpath(os.path.join(os.path.dirname(args[0]),
+                                                        os.path.pardir,
+                                                        globals.TESTS_SHARE))
+
+    options = parse_options(args)
+    configure_logging(options.log_level, options.log)
+
+    apps = discover_applications(options.directories,
+                                 options.applications,
+                                 options.suites,
+                                 options.cases)
+
+    # Execute test cases
+    if not apps:
+        logging.warning("No test applications found")
+    else:
+        if not options.info:
+            suites = chain.from_iterable(app.suites() for app in apps)
+            for suite in suites:
+                process_suite_file(suite.fullname, options.target, options.cases)
+        else:
+            for app in apps:
+                print "Application: %s" % app.name
+                for suite in app.suites():
+                    print "- Suite: %s" % suite.name
+                    for case in suite.cases():
+                        print " - Case: %s" % case.name
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

=== added file 'desktoptesting/cmd/parser.py'
--- desktoptesting/cmd/parser.py	1970-01-01 00:00:00 +0000
+++ desktoptesting/cmd/parser.py	2009-06-04 04:15:25 +0000
@@ -0,0 +1,118 @@
+"""
+This module contains the code needed to parse all the options
+"""
+import os
+from copy import copy
+from optparse import OptionParser, OptionGroup, OptionValueError
+from optparse import Option as OldOption
+
+from . import globals
+
+def check_dir(option, opt, value):
+    """
+    Check if an dir option string contains a real directory
+    """
+    value = os.path.realpath(os.path.expanduser(value))
+    if not os.path.isdir(value):
+        raise OptionValueError("option %s: "
+                               "Directory '%s' doesn't exist"
+                               % (opt, value))
+    return value
+
+        
+def check_dirname(option, opt, value):
+    """
+    Check if an dirname option string contains a valid directory name
+    """
+    value = os.path.realpath(os.path.expanduser(value))
+    if os.path.exists(value) and not os.path.isdir(value):
+        raise OptionValueError("option %s: "
+                               "Directory '%s' exists, but isn't a directory"
+                               % (opt, value))
+    return value
+
+class Option(OldOption):
+    """
+    Extended option class that adds two directory types:
+    - dirname: Valid directory name that isn't required to exist
+    - dir: Valid directory name that must exist
+    """
+    TYPES = OldOption.TYPES + ("dir", "dirname")
+    TYPE_CHECKER = copy(OldOption.TYPE_CHECKER)
+    TYPE_CHECKER["dir"] = check_dir
+    TYPE_CHECKER["dirname"] = check_dirname
+
+
+def parse_options(args):
+    """
+    Parse options passed through the command line
+    """
+    default_target = "~/.ubuntu-desktop-test"
+    default_log_level = "critical"
+    default_directories = [os.getcwd(), globals.TESTS_SHARE]
+
+    parser = OptionParser(description="Execute automated tests",
+                          option_class=Option)
+
+    parser.add_option('-i', '--info',
+                      action="store_true",
+                      help=("Display information about test cases "
+                            "without executing them"))
+    
+    group = OptionGroup(parser, "Test selection options")
+    group.add_option("-d", "--directory",
+                     dest="directories",
+                     # Default directories are used only if no directory
+                     # was passed through the command line
+                     default = [],
+                     metavar="DIR",
+                     action="append",
+                     type="dir",
+                     help=("Application directory locations. Option can be repeated "
+                           "and defaults to both shared test directory "
+                           "and current working directory"))
+    group.add_option("-a", "--application",
+                     dest="applications",
+                     action="append",
+                     type="string",
+                     default=None,
+                     help=("Application name to test. Option can be repeated "
+                           "and defaults to all applications"))
+    group.add_option("-s", "--suite",
+                     dest="suites",
+                     metavar="SUITE",
+                     action="append",
+                     type="string",
+                     default=None,
+                     help=("Suite name to test within applications. Option "
+                           "can be repeated and default to all suites"))
+    group.add_option("-c", "--case",
+                     dest="cases",
+                     metavar="CASE",
+                     action="append",
+                     type="string",
+                     default=None,
+                     help="Test cases to run (all, if not specified).")
+    parser.add_option_group(group)
+
+    group = OptionGroup(parser, "Logging options")
+    group.add_option("-l", "--log",
+                     metavar="FILE",
+                     help="The file to write the log to.")
+    group.add_option("--log-level",
+                     default=default_log_level,
+                     help="One of debug, info, warning, error or critical.")
+    group.add_option("-t", "--target",
+                     metavar="DIR",
+                     type="dirname",
+                     default=default_target,
+                     help=("Target directory for logs and reports. Defaults "
+                           "to: %default"))
+    parser.add_option_group(group)
+
+    (options, args) = parser.parse_args(args[1:])
+
+    if not options.directories:
+        options.directories = default_directories
+
+    return options

=== added file 'desktoptesting/cmd/runner.py'
--- desktoptesting/cmd/runner.py	1970-01-01 00:00:00 +0000
+++ desktoptesting/cmd/runner.py	2009-06-03 16:35:09 +0000
@@ -0,0 +1,206 @@
+"""
+This module contains different kinds of runner classes needed to
+execute test cases
+"""
+import os, sys, traceback, logging
+import xml.dom.minidom
+import ldtp, ldtputils
+from time import time, gmtime, strftime
+from shutil import move
+
+from . import globals
+from .utils import safe_make_directory
+
+class TestRunner:
+    def __init__(self):
+        self.result = {}
+        self.args = {}
+
+    def get_args(self, node):
+        for n in node.childNodes:
+            if n.nodeType != n.ELEMENT_NODE or not n.hasChildNodes():
+                continue
+            self.args[n.tagName.encode('ascii')] = n.firstChild.data
+
+    def add_results_to_node(self, node):
+        if self.result == {}: return
+        result = node.ownerDocument.createElement("result")
+        for key, val in self.result.items():
+            if not isinstance(val, list):
+                val = [val]
+            for item in val:
+                n = node.ownerDocument.createElement(key)
+                n.appendChild(n.ownerDocument.createTextNode(str(item)))
+                result.appendChild(n)
+                
+        node.appendChild(result)
+
+    def append_result(self, key, value):
+        l = self.result.get(key, [])
+        l.append(value)
+        self.result[key] = l
+
+    def set_result(self, key, value):
+        self.result[key] = value
+
+    def append_screenshot(self, screenshot_file=None):
+        _logFile = "%s/screenshot-%s.png" % (globals.SCREENSHOTS_SHARE,
+                                             strftime ("%m-%d-%Y-%H-%M-%s"))
+        safe_make_directory(globals.SCREENSHOTS_SHARE)
+        if not screenshot_file:
+            ldtputils.imagecapture(outFile = _logFile)
+        else:
+            move(screenshot_file, _logFile)
+        self.append_result('screenshot', _logFile)
+
+
+class TestCaseRunner(TestRunner):
+    def __init__(self, obj, xml_node):
+        TestRunner.__init__(self)
+        self.xml_node = xml_node
+        self.name = xml_node.getAttribute('name')
+        self.test_func = getattr(
+            obj, xml_node.getElementsByTagName('method')[0].firstChild.data)
+        args_element = xml_node.getElementsByTagName('args')
+        if args_element:
+            self.get_args(args_element[0])
+
+    def run(self, logger):
+        starttime = time()
+        try:
+            rv = self.test_func(**self.args)
+        except AssertionError, e:
+            # The test failed.
+            if len(e.args) > 1:
+                self.append_result('message', e.args[0])
+                self.append_screenshot(e.args[1])
+            else:
+                self.append_result('message', str(e))
+                self.append_screenshot()
+            self.append_result('stacktrace', traceback.format_exc())
+            self.set_result('pass', 0)
+        except Exception, e:
+            # There was an unrelated error.
+            logging.warning(traceback.format_exc())
+            if len(e.args) > 1:
+                self.append_result('message', e.args[0])
+                self.append_screenshot(e.args[1])
+            else:
+                self.append_result('message', str(e))
+                self.append_screenshot()
+            self.append_result('stacktrace', traceback.format_exc())
+            self.set_result('error', 1)
+        else:
+            self.set_result('pass', 1)
+            try:
+                message, screenshot = rv
+            except:
+                pass
+            else:
+                if message:
+                    self.append_result('message', message)
+                if screenshot:
+                    self.append_screenshot(screenshot)
+        finally:
+            self.set_result('time', time() - starttime)
+        
+        self.add_results_to_node(self.xml_node)
+
+        
+class TestSuiteRunner(TestRunner):
+    def __init__(self, suite_file, loggerclass=None):
+        TestRunner.__init__(self)
+        self.dom = xml.dom.minidom.parse(suite_file)
+        self._strip_whitespace(self.dom.documentElement)
+        clsname, modname = None, None
+        case_runners = []
+        for node in self.dom.documentElement.childNodes:
+            if node.nodeType != node.ELEMENT_NODE:
+                continue
+            if node.tagName == 'class':
+                modname, clsname = node.firstChild.data.rsplit('.', 1)
+            elif node.tagName == 'case':
+                logging.debug("Adding case %s to current test suite.", 
+                              node.getAttribute("name"))
+                case_runners.append(node)
+            elif node.tagName == 'args':
+                self.get_args(node)
+        if None in (clsname, modname):
+            raise Exception, "Missing a suite class"
+
+        # Suite file and module are suppose to be in the same directory
+        sys.path.insert(1, os.path.dirname(suite_file))
+        logging.debug("Modname: %s", modname)
+        mod = __import__(modname)
+        sys.path.pop(1)
+
+        cls = getattr(mod, clsname)
+        
+        self.testsuite = cls(**self.args)
+        
+        self.case_runners = \
+            [TestCaseRunner(self.testsuite, c) for c in case_runners]
+    
+    def run(self, loggerclass=None, setup_once=True, cases=None):
+        try:
+            self._run(loggerclass, setup_once, cases)
+        except Exception, e:
+            logging.warning(traceback.format_exc())
+            # There was an unrelated error.
+            self.append_result('message', str(e))
+            self.append_result('stacktrace', traceback.format_exc())
+            self.set_result('error', 1)
+            try:
+                self.testsuite.teardown()
+            except:
+                pass
+
+        self.add_results_to_node(self.dom.documentElement)
+
+        return self.dom.toprettyxml('  ')
+            
+    def _run(self, loggerclass, setup_once, cases):
+        if loggerclass:
+            logger = loggerclass()
+        else:
+            logger = ldtp
+
+        if setup_once:
+            # Set up the environment.
+            self.testsuite.setup()
+
+        firsttest = True
+        for testcase in self.case_runners:
+            if cases and testcase.name not in cases:
+                continue
+            if not setup_once:
+                # Set up the app for each test, if requested.
+                self.testsuite.setup()
+            if not firsttest:
+                # Clean up from previous run.
+                self.testsuite.cleanup()
+            firsttest = False
+            testcase.run(logger)
+            if not setup_once:
+                # Teardown upthe app for each test, if requested.
+                self.testsuite.teardown()
+
+        if setup_once:
+            # Tear down after entire suite.
+            self.testsuite.teardown()
+
+    def _strip_whitespace(self, element):
+        if not element: 
+            return
+        sibling = element.firstChild
+        while sibling:
+            nextSibling = sibling.nextSibling
+            if sibling.nodeType == sibling.TEXT_NODE:
+                stripped = sibling.data.strip()
+                if stripped == '':
+                    element.removeChild(sibling)
+                else:
+                    sibling.data = stripped
+            else:
+                self._strip_whitespace(sibling)
+            sibling = nextSibling

=== added file 'desktoptesting/cmd/utils.py'
--- desktoptesting/cmd/utils.py	1970-01-01 00:00:00 +0000
+++ desktoptesting/cmd/utils.py	2009-06-03 10:12:35 +0000
@@ -0,0 +1,38 @@
+"""
+This module contains utility functions for file management and command
+execution
+"""
+import os, logging
+from stat import ST_MODE, S_IMODE
+from subprocess import Popen, PIPE
+
+def safe_change_mode(path, mode):
+    if not os.path.exists(path):
+        logging.error("Path does not exist: %s" % path)
+        sys.exit(1)
+
+    old_mode = os.stat(path)[ST_MODE]
+    if mode != S_IMODE(old_mode):
+        os.chmod(path, mode)
+
+
+def safe_make_directory(path, mode=0755):
+    if os.path.exists(path):
+        if not os.path.isdir(path):
+            logging.error("Path is not a directory: %s" % path)
+            sys.exit(1)
+
+        safe_change_mode(path, mode)
+    else:
+        logging.debug("Creating directory: %s" % path)
+        os.makedirs(path, mode)
+
+
+def safe_run_command(command):
+    logging.debug("Running command: %s" % command)
+    p = Popen(command, stdout=PIPE, shell=True)
+    (pid, status) = os.waitpid(p.pid, 0)
+    if status:
+        logging.error("Command failed: %s" % command)
+
+    return p.stdout.read()

=== modified file 'gedit/gedit_chains.py'
--- gedit/gedit_chains.py	2009-04-21 10:15:09 +0000
+++ gedit/gedit_chains.py	2009-06-03 09:41:56 +0000
@@ -1,4 +1,5 @@
 # -*- coding: utf-8 -*-
+import os
 from time import time, gmtime, strftime
 
 from desktoptesting.test_suite.gnome import GEditTestSuite
@@ -12,6 +13,11 @@
         self.application.write_text(chain)
         self.application.save(test_file)
 
+        # oracle file path is assumed to be relative
+        # to test case code
+        oracle = os.path.join(os.path.dirname(__file__),
+                              oracle)
+
         testcheck = FileComparison(oracle, test_file)
 
         if testcheck.perform_test() == FAIL:

=== modified file 'gedit/gedit_chains.xml'
--- gedit/gedit_chains.xml	2009-03-11 16:35:00 +0000
+++ gedit/gedit_chains.xml	2009-06-03 09:41:56 +0000
@@ -8,7 +8,7 @@
     <method>testChain</method>
     <description>Test Unicode text saving.</description>
     <args>
-      <oracle>./gedit/data/utf8.txt</oracle>
+      <oracle>data/utf8.txt</oracle>
       <chain>This is a japanese string: 広告掲載 - ビジネス</chain>
     </args>
   </case>
@@ -16,7 +16,7 @@
     <method>testChain</method>
     <description>Test ASCII text saving.</description>
     <args>
-      <oracle>./gedit/data/ascii.txt</oracle>
+      <oracle>data/ascii.txt</oracle>
       <chain>This is a very basic string!</chain>
     </args>
   </case>

=== modified file 'setup.py'
--- setup.py	2009-05-06 08:00:55 +0000
+++ setup.py	2009-06-03 09:41:56 +0000
@@ -113,7 +113,8 @@
     scripts = ["bin/ubuntu-desktop-test"],
     packages = ["desktoptesting",
                 "desktoptesting.application",
-                "desktoptesting.test_suite"],
+                "desktoptesting.test_suite",
+                "desktoptesting.cmd"],
     cmdclass = {
         "install_data": testing_install_data,
         "install_scripts": testing_install_scripts,