Merge lp:~thomir-deactivatedaccount/autopilot/trunk-add-new-formats into lp:autopilot

[Thomi Richards]

Make autopilot support subunit bytestream output.

Make autopilot support subunit bytestream output.

[lp:~thomir-deactivatedaccount/autopilot/trunk-add-new-formats]

380. By Thomi Richards on 2013-12-02

Merged trunk.

381. By Thomi Richards on 2013-12-02

Fix commented out code and incorrect copyright header date.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:380
http://jenkins.qa.ubuntu.com/job/autopilot-ci/320/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-amd64-ci/46/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-armhf-ci/46/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-i386-ci/46/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/320/rebuild

review: Needs Fixing (continuous-integration)

[lp:~thomir-deactivatedaccount/autopilot/trunk-add-new-formats]

382. By Thomi Richards on 2013-12-02

Add junitxml as a build-dep.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:381
http://jenkins.qa.ubuntu.com/job/autopilot-ci/321/
Executed test runs:
    None: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-amd64-ci/47/console
    None: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-armhf-ci/47/console
    None: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-i386-ci/47/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/321/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:380
http://jenkins.qa.ubuntu.com/job/autopilot-ci/322/
Executed test runs:
    None: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-amd64-ci/48/console
    None: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-armhf-ci/48/console
    None: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-i386-ci/48/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/322/rebuild

review: Needs Fixing (continuous-integration)

[lp:~thomir-deactivatedaccount/autopilot/trunk-add-new-formats]

383. By Thomi Richards on 2013-12-02

Fix PEP8.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:382
http://jenkins.qa.ubuntu.com/job/autopilot-ci/323/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-amd64-ci/49/console
    None: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-armhf-ci/49/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-i386-ci/49/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/323/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:383
http://jenkins.qa.ubuntu.com/job/autopilot-ci/324/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-amd64-ci/50
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-armhf-ci/50
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-i386-ci/50

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/324/rebuild

review: Approve (continuous-integration)

[Christopher Lee (veebers)]

LGTM

review: Approve

=== modified file 'autopilot/__init__.py'
--- autopilot/__init__.py	2013-11-26 22:54:36 +0000
+++ autopilot/__init__.py	2013-12-02 02:12:42 +0000
@@ -20,20 +20,12 @@
 from argparse import ArgumentParser, REMAINDER, Action
 import subprocess
 
+from autopilot.testresult import get_output_formats, get_default_format
+from autopilot.exceptions import BackendException
+
 version = '1.4.0'
 
 
-class BackendException(RuntimeError):
-
-    """An error occured while trying to initialise an autopilot backend."""
-
-    def __init__(self, original_exception):
-        super(BackendException, self).__init__(
-            "Error while initialising backend. Original exception was: " +
-            str(original_exception))
-        self.original_exception = original_exception
-
-
 def parse_arguments(argv=None):
     """Parse command-line arguments, and return an argparse arguments
     object.
@@ -57,8 +49,9 @@
                             If given a directory instead of a file will \
                             write to a file in that directory named: \
                             <hostname>_<dd.mm.yyy_HHMMSS>.log')
-    parser_run.add_argument('-f', "--format", choices=['text', 'xml'],
-                            default='text',
+    available_formats = get_output_formats().keys()
+    parser_run.add_argument('-f', "--format", choices=available_formats,
+                            default=get_default_format(),
                             required=False,
                             help='Specify desired output format. \
                             Default is "text".')

=== modified file 'autopilot/run.py'
--- autopilot/run.py	2013-11-28 04:03:49 +0000
+++ autopilot/run.py	2013-12-02 02:12:42 +0000
@@ -41,11 +41,10 @@
 from unittest import TestLoader, TestSuite
 
 from testtools import iterate_tests
-from testtools import TextTestResult
 
 from autopilot import get_version_string, parse_arguments
 import autopilot.globals
-from autopilot.testresult import AutopilotVerboseResult
+from autopilot.testresult import get_output_formats
 from autopilot.utilities import DebugLogFilter, LogFormatter
 
 
@@ -69,29 +68,13 @@
     root_logger.info(get_version_string())
 
 
-def construct_test_runner(args):
-    kwargs = dict(
-        stdout=get_output_stream(args),
-        output_format=get_output_format(args.format),
+def construct_test_result(args):
+    formats = get_output_formats()
+    return formats[args.format](
+        stream=get_output_stream(args),
+        failfast=args.failfast,
     )
 
-    return ConfigurableTestRunner(**kwargs)
-
-
-def get_output_format(format):
-    """Return a Result object for each format we support."""
-
-    if format == "text":
-        return type('VerboseTextTestResult', (TextTestResult,),
-                    dict(AutopilotVerboseResult.__dict__))
-
-    elif format == "xml":
-        from junitxml import JUnitXmlResult
-        return type('VerboseXmlResult', (JUnitXmlResult,),
-                    dict(AutopilotVerboseResult.__dict__))
-
-    raise KeyError("Unknown format name '%s'" % format)
-
 
 def get_output_stream(args):
     global _output_stream
@@ -118,30 +101,6 @@
     return _output_stream
 
 
-class ConfigurableTestRunner(object):
-    """A configurable test runner class.
-
-    This class alows us to configure the output format and whether of not we
-    collect coverage information for the test run.
-
-    """
-
-    def __init__(self, stdout, output_format):
-        self.stdout = stdout
-        self.result_class = output_format
-
-    def run(self, test, failfast=False):
-        "Run the given test case or test suite."
-        result = self.result_class(self.stdout)
-        result.startTestRun()
-        result.failfast = failfast
-        try:
-            test_result = test.run(result)
-        finally:
-            result.stopTestRun()
-        return test_result
-
-
 def get_package_location(import_name):
     """Get the on-disk location of a package from a test id name.
 
@@ -218,6 +177,7 @@
         self.args = parse_arguments()
 
     def run(self):
+        setup_logging(self.args.verbose)
         if self.args.mode == 'list':
             self.list_tests()
         elif self.args.mode == 'run':
@@ -228,7 +188,6 @@
             self.launch_app()
 
     def run_vis(self):
-        setup_logging(self.args.verbose)
         # importing this requires that DISPLAY is set. Since we don't always
         # want that requirement, do the import here:
         from autopilot.vis import vis_main
@@ -251,7 +210,6 @@
             get_application_launcher_from_string_hint,
         )
 
-        setup_logging(self.args.verbose)
         app_name = self.args.application[0]
         if not os.path.isabs(app_name) or not os.path.exists(app_name):
             try:
@@ -325,9 +283,13 @@
         if self.args.verbose:
             autopilot.globals.set_log_verbose(True)
 
-        setup_logging(self.args.verbose)
-        runner = construct_test_runner(self.args)
-        test_result = runner.run(test_suite, self.args.failfast)
+        result = construct_test_result(self.args)
+        result.startTestRun()
+        try:
+            test_result = test_suite.run(result)
+        finally:
+            result.stopTestRun()
+
         if not test_result.wasSuccessful():
             exit(1)
 

=== modified file 'autopilot/testresult.py'
--- autopilot/testresult.py	2013-09-16 17:26:27 +0000
+++ autopilot/testresult.py	2013-12-02 02:12:42 +0000
@@ -25,11 +25,17 @@
 import logging
 
 from autopilot.globals import get_log_verbose
-
-
-class AutopilotVerboseResult(object):
-    """A result class that logs failures, errors and success via the python
-    logging framework."""
+from testtools import (
+    ExtendedToOriginalDecorator,
+    ExtendedToStreamDecorator,
+    TestResultDecorator,
+    TextTestResult,
+    try_import,
+)
+
+
+class LoggedTestResultDecorator(TestResultDecorator):
+    """A decorator that logs messages to python's logging system."""
 
     def _log(self, level, message):
         """Performs the actual message logging"""
@@ -46,40 +52,61 @@
             self._log(level, text)
 
     def addSuccess(self, test, details=None):
-        """Called for a successful test"""
-        # Allow for different calling syntax used by the base class.
-        if details is None:
-            super(type(self), self).addSuccess(test)
-        else:
-            super(type(self), self).addSuccess(test, details)
         self._log(logging.INFO, "OK: %s" % (test.id()))
+        return super(LoggedTestResultDecorator, self).addSuccess(test, details)
 
     def addError(self, test, err=None, details=None):
-        """Called for a test which failed with an error"""
-        # Allow for different calling syntax used by the base class.
-        # The xml path only uses 'err'. Use of 'err' can be
-        # forced by raising TypeError when it is not specified.
-        if err is None:
-            raise TypeError
-        if details is None:
-            super(type(self), self).addError(test, err)
-        else:
-            super(type(self), self).addError(test, err, details)
         self._log(logging.ERROR, "ERROR: %s" % (test.id()))
         if hasattr(test, "getDetails"):
             self._log_details(logging.ERROR, test.getDetails())
+        return super(type(self), self).addError(test, err, details)
 
     def addFailure(self, test, err=None, details=None):
         """Called for a test which failed an assert"""
-        # Allow for different calling syntax used by the base class.
-        # The xml path only uses 'err' or 'details'. Use of 'err' can be
-        # forced by raising TypeError when it is not specified.
-        if err is None:
-            raise TypeError
-        if details is None:
-            super(type(self), self).addFailure(test, err)
-        else:
-            super(type(self), self).addFailure(test, err, details)
         self._log(logging.ERROR, "FAIL: %s" % (test.id()))
         if hasattr(test, "getDetails"):
             self._log_details(logging.ERROR, test.getDetails())
+        return super(type(self), self).addFailure(test, err, details)
+
+
+def get_output_formats():
+    """Get information regarding the different output formats supported."""
+    supported_formats = {}
+
+    supported_formats['text'] = _construct_text
+
+    if try_import('junitxml'):
+        supported_formats['xml'] = _construct_xml
+    if try_import('subunit'):
+        supported_formats['subunit'] = _construct_subunit
+    return supported_formats
+
+
+def get_default_format():
+    return 'text'
+
+
+def _construct_xml(**kwargs):
+    from junitxml import JUnitXmlResult
+    stream = kwargs.pop('stream')
+    return LoggedTestResultDecorator(
+        ExtendedToOriginalDecorator(
+            JUnitXmlResult(stream)
+        )
+    )
+
+
+def _construct_text(**kwargs):
+    stream = kwargs.pop('stream')
+    failfast = kwargs.pop('failfast')
+    return LoggedTestResultDecorator(TextTestResult(stream, failfast))
+
+
+def _construct_subunit(**kwargs):
+    from subunit import StreamResultToBytes
+    stream = kwargs.pop('stream')
+    return LoggedTestResultDecorator(
+        ExtendedToStreamDecorator(
+            StreamResultToBytes(stream)
+        )
+    )

=== added file 'autopilot/tests/unit/test_testresults.py'
--- autopilot/tests/unit/test_testresults.py	1970-01-01 00:00:00 +0000
+++ autopilot/tests/unit/test_testresults.py	2013-12-02 02:12:42 +0000
@@ -0,0 +1,92 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+#
+# Autopilot Functional Test Tool
+# Copyright (C) 2013 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+from mock import Mock, patch
+from testtools import TestCase, PlaceHolder
+from testtools.content import text_content
+
+from autopilot.testresult import (
+    get_default_format,
+    get_output_formats,
+    LoggedTestResultDecorator,
+)
+
+
+class LoggedTestResultDecoratorTests(TestCase):
+
+    def construct_simple_content_object(self):
+        return text_content(self.getUniqueString)
+
+    def test_can_construct(self):
+        LoggedTestResultDecorator(Mock())
+
+    def test_addSuccess_calls_decorated_test(self):
+        wrapped = Mock()
+        result = LoggedTestResultDecorator(wrapped)
+        fake_test = PlaceHolder('fake_test')
+        fake_details = self.construct_simple_content_object()
+
+        result.addSuccess(fake_test, fake_details)
+
+        wrapped.addSuccess.assert_called_once_with(
+            fake_test,
+            details=fake_details
+        )
+
+    def test_addError_calls_decorated_test(self):
+        wrapped = Mock()
+        result = LoggedTestResultDecorator(wrapped)
+        fake_test = PlaceHolder('fake_test')
+        fake_error = object()
+        fake_details = self.construct_simple_content_object()
+
+        result.addError(fake_test, fake_error, fake_details)
+
+        wrapped.addError.assert_called_once_with(
+            fake_test,
+            fake_error,
+            details=fake_details
+        )
+
+    def test_addFailure_calls_decorated_test(self):
+        wrapped = Mock()
+        result = LoggedTestResultDecorator(wrapped)
+        fake_test = PlaceHolder('fake_test')
+        fake_error = object()
+        fake_details = self.construct_simple_content_object()
+
+        result.addFailure(fake_test, fake_error, fake_details)
+
+        wrapped.addFailure.assert_called_once_with(
+            fake_test,
+            fake_error,
+            details=fake_details
+        )
+
+
+class OutputFormatFactoryTests(TestCase):
+
+    def test_has_text_format(self):
+        self.assertTrue('text' in get_output_formats())
+
+    def test_has_xml_format(self):
+        self.assertTrue('xml' in get_output_formats())
+
+    def test_default_format_is_available(self):
+        self.assertTrue(get_default_format() in get_output_formats())

=== modified file 'debian/control'
--- debian/control	2013-11-27 20:15:34 +0000
+++ debian/control	2013-12-02 02:12:42 +0000
@@ -16,23 +16,25 @@
                python-debian,
                python-dev,
                python-gi,
+               python-junitxml,
                python-mock,
                python-psutil,
                python-setuptools,
-               python-subunit,
                python-six,
                python-sphinx,
+               python-subunit,
                python-testscenarios,
                python-testtools,
                python-xlib,
                python3-all-dev (>= 3.3),
                python3-dbus,
                python3-gi,
+               python3-junitxml,
                python3-mock,
                python3-psutil,
                python3-setuptools,
+               python3-six,
                python3-subunit,
-               python3-six,
                python3-testscenarios,
                python3-testtools,
                python3-xlib,