Merge into trunk : texttest-run-uprefresh : Code : Autopilot

Status:	Needs review
Proposed branch:	lp:~veebers/autopilot/texttest-run-uprefresh
Merge into:	lp:autopilot
Diff against target:	848 lines (+351/-130) 7 files modified autopilot/__init__.py (+1/-1) autopilot/run.py (+21/-19) autopilot/testresult.py (+85/-22) autopilot/tests/functional/test_autopilot_functional.py (+87/-54) autopilot/tests/unit/test_command_line_args.py (+10/-2) autopilot/tests/unit/test_run.py (+127/-30) autopilot/tests/unit/test_testresults.py (+20/-2)
To merge this branch:	bzr merge lp:~veebers/autopilot/texttest-run-uprefresh
Related bugs:	Link a bug report

Reviewer	Review Type	Date Requested	Status
platform-qa-bot	continuous-integration		Approve on 2016-01-19
PS Jenkins bot	continuous-integration	2014-07-02	Needs Fixing on 2014-07-22
prod-platform-qa	continuous-integration	2017-02-22	Pending
Christopher Lee			Pending
Thomi Richards		2014-07-02	Pending
Corey Goldberg		2014-07-02	Pending
Review via email: mp+225247@code.launchpad.net

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-03: Posted in a previous version of this proposal

#

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/537/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-03: Posted in a previous version of this proposal

#

FAILED: Continuous integration, rev:450
http://jenkins.qa.ubuntu.com/job/autopilot-ci/538/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-amd64-ci/264/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-armhf-ci/264/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-i386-ci/173/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-trusty/3638/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-trusty-amd64/3642/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/538/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-03: Posted in a previous version of this proposal

#

FAILED: Continuous integration, rev:452
http://jenkins.qa.ubuntu.com/job/autopilot-ci/539/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-amd64-ci/265/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-armhf-ci/265/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-i386-ci/174/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-trusty/3639/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-trusty-amd64/3643/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/539/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-04: Posted in a previous version of this proposal

#

FAILED: Continuous integration, rev:454
http://jenkins.qa.ubuntu.com/job/autopilot-ci/541/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-amd64-ci/267/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-armhf-ci/267/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-i386-ci/176/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-trusty/3658/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-trusty-amd64/3662/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/541/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-04: Posted in a previous version of this proposal

#

FAILED: Continuous integration, rev:456
http://jenkins.qa.ubuntu.com/job/autopilot-ci/542/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-amd64-ci/268/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-armhf-ci/268/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-i386-ci/177/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-trusty/3662/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-trusty-amd64/3667/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/542/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-04: Posted in a previous version of this proposal

#

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/543/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-05: Posted in a previous version of this proposal

#

FAILED: Continuous integration, rev:458
http://jenkins.qa.ubuntu.com/job/autopilot-ci/547/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-amd64-ci/273/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-armhf-ci/273/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-i386-ci/182/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-trusty/3683/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-trusty-amd64/3688/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/547/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-05: Posted in a previous version of this proposal

#

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/548/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

Corey Goldberg (coreygoldberg) wrote on 2014-03-05: Posted in a previous version of this proposal

#

from irc with thomi:

<thomi> you should flush stdout afdter startTest, since we want to see that output in the case where a test takes a long time to run
<cgoldberg> right
<thomi> also, I think the skip line should have the optional message in '( )'
so like:
test.id ... SKIP (Not runnable on the device)
for example
<cgoldberg> yup.. agree
<thomi> also, I think the status messages can be a bit more verbose.
particularly, please change:
XFAIL => EXPECTED FAIL
and
NOTOK => UNEXPECTED SUCCESS
<cgoldberg> right. makes sense. I copied the unittest runner's output, but there's no reason not to be a little more explicit
<thomi> also, you should generalise the code (diff lines 82-92) to go in a separate function, something like _wrap_result_with_output_decorator
which accepts a result object, and wraps it in either a LoggingResultDecorator or your new decorator
that way, it's easier to test, and we keep the complexity of the construct_XXX functions down
<thomi> cgoldberg: in your tests, I think you can make your 'assertOutput' lines more readable, by doing:
<cgoldberg> thomi, ok.. i can wrap it in a function
<thomi> self.assertOutput('{id} ... OK\n', 'pass')
one line is better than 3, since we read left->right
other than those minor quibbles above ^^, this looks great, but you're missing some tests still
<cgoldberg> thomi, hah.. will do
<thomi> I'd like to see an integration test that shows that when you specify the various result formats without the verbose flag, we get the correct result object & wrapper
similarly for when you do specify the verbose flag
<cgoldberg> gotcha.. yea i can add that too
<thomi> Finally, I think you should add 6 functional tests to the functional test suite
1 test fro each format, * 1 with verbose, and one without verbose
actually, the verbose case is almost certainly covered already (but please do check)
so maybe it's only 3 new functional tests :)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-05: Posted in a previous version of this proposal

#

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/549/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

Corey Goldberg (coreygoldberg) wrote on 2014-03-06: Posted in a previous version of this proposal

#

from thomi:

OK, so we need to change things. Seems like there's several things we're trying to control independantly of each other here:

1) The level of verbosity autopilot uses when logging. Ideally we'd make sure that all our log levels were sensible, and then this would control the logging framework verbosity level. 0 * '-v' => normal and higher. '-v' => info logs and higher. '-vv' => debug logs and higher. The log *always* gets attached to the test result for every test, regardless of the output file format.

2) The format we store run reports in. We have 'text', 'xml', 'subunit'. I think we need one more, which is 'verbosetext' or something similar. This causes the test log to get printed to stdoud as the test is runing. This is equivilent to the '-v' flag before this change.

3) The location of the output report. If the -o option is specified, the report is written to that file, in the format specified by the '-f' parameter.

Finally, if, after configuring all this, stdout is not used, the 'status' format should be printed to stdout as well.

review: Needs Fixing

Revision history for this message

Corey Goldberg (coreygoldberg) wrote on 2014-03-06: Posted in a previous version of this proposal

#

Four output formats will be:
* status (new format)
* xml
* subunit
* text
Every format can be sent to stdout or a file.
-v / -vv flag ONLY controls the verbosity of the python log.
make subclass of TextTestResult to print to stream

Revision history for this message

Corey Goldberg (coreygoldberg) wrote on 2014-03-06: Posted in a previous version of this proposal

#

make above fixes

review: Needs Fixing

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-11: Posted in a previous version of this proposal

#

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/565/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-11: Posted in a previous version of this proposal

#

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/568/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-11: Posted in a previous version of this proposal

#

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/569/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

Christopher Lee (veebers) wrote on 2014-03-11: Posted in a previous version of this proposal

#

Just a couple of minor things.

Initially I thought I misunderstood the verbose/output changes until I read the comment re: verbose output only going to a log file. I think that the -h output needs to be updated to state as such.
(actually I think I'm confused on this point now).

The help stats that the default format for -f is text where it is now 'status' (I was initially a little confused why my -v wasn't outputting the tapping details etc.).

line 488-491 has some odd indentation. I would put the opening [ on a new line i.e.
  self.run_autopilot(
      [ "run", "--failfast", "-f", "text", "tests"]
  )
# You could put the list args on new lines too if you wanted, but not needed here.

That's all for now, I'll try give a more functional review later on :-)

review: Needs Fixing

Revision history for this message

Corey Goldberg (coreygoldberg) wrote on 2014-03-12: Posted in a previous version of this proposal

#

thanks veebers!

> I think that the -h output needs to be updated to state as such.
> (actually I think I'm confused on this point now).

I will update -h commandline help, and /docs today with clear explanation of log and output handling.

> line 488-491 has some odd indentation.

I just pushed a fix for indentation in functional tests.

Revision history for this message

Corey Goldberg (coreygoldberg) wrote on 2014-03-12: Posted in a previous version of this proposal

#

updated help strings.

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-12: Posted in a previous version of this proposal

#

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/570/rebuild

review: Approve (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-12: Posted in a previous version of this proposal

#

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/571/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-13: Posted in a previous version of this proposal

#

FAILED: Continuous integration, rev:471
http://jenkins.qa.ubuntu.com/job/autopilot-ci/573/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-amd64-ci/299/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-armhf-ci/299/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-i386-ci/208/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-trusty-autopilot/25/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-trusty-amd64/3922/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/573/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-18: Posted in a previous version of this proposal

#

FAILED: Continuous integration, rev:472
http://jenkins.qa.ubuntu.com/job/autopilot-ci/588/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-amd64-ci/314/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-armhf-ci/314/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-i386-ci/223/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-trusty-autopilot/40/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-trusty-amd64/4072/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/588/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-18: Posted in a previous version of this proposal

#

FAILED: Continuous integration, rev:473
http://jenkins.qa.ubuntu.com/job/autopilot-ci/589/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-amd64-ci/315/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-armhf-ci/315/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-trusty-i386-ci/224/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-trusty-autopilot/41/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-trusty-amd64/4088/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/589/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

Thomi Richards (thomir-deactivatedaccount) wrote on 2014-03-18: Posted in a previous version of this proposal

#

Hi,

A few things:

40 - '-v', '--verbose', required=False, default=False, action='count',
41 - help="Show autopilot log messages. Set twice to also log data "
42 - "useful for debugging autopilot itself.")
43 + "-v", "--verbosity", action='count', default=0, required=False,
44 + help="Increase verbosity of test details and 'text' mode output. "
45 + "Set twice to also log data useful for debugging autopilot "
46 + "itself.")

This is incorrect, since this is for the vis tool. The original message was correct, I don't think you need to change it at all.

54 - '-v', '--verbose', required=False, default=False, action='count',
55 - help="Show autopilot log messages. Set twice to also log data useful "
56 - "for debugging autopilot itself.")
57 + "-v", "--verbosity", action='count', default=0, required=False,
58 + help="Increase verbosity of test details and 'text' mode output. "
59 + "Set twice to also log data useful for debugging autopilot "
60 + "itself.")

Same here - this is for the launch command.

A small thing, but this:

Should be "2012-2014", not "2012,2013,2014", as per legal advice.

102 from autopilot import get_version_string, parse_arguments
103 -import autopilot.globals
104 from autopilot._debug import get_all_debug_profiles
105 -from autopilot.testresult import get_output_formats
106 -from autopilot.utilities import DebugLogFilter, LogFormatter
107 from autopilot.application._launcher import (
108 _get_app_env_from_string_hint,
109 get_application_launcher_wrapper,
110 launch_process,
111 )
112 +import autopilot.globals
113 +from autopilot.testresult import get_output_formats
114 +from autopilot.utilities import DebugLogFilter, LogFormatter

This isn't the standard we've set in the AP codebase. We mix 'import FOO' and 'from FOO import BAR' statements in one block, and alphabeticise the module names.

565 + def test_failfast_text_mode(self):

Please move the test_failfast test to a new class, and use scenarios to make sure that *all* test result formats support failfast, not just text and status.

Here's the big change I think we need to make:

Currently, the logger is set up to log to the same stream as the result object. This works for the text format, but not the others. Instead, we should make it so the test log is added to the test result as a detail, for every format. This will help us with the subunit format, for example. TBH, I thought this already happend, but I can't see the code now. Feel free to point it out to me :)

Cheers

Hi,

A few things:

40	- '-v', '--verbose', required=False, default=False, action='count',
41	- help="Show autopilot log messages. Set twice to also log data "
42	- "useful for debugging autopilot itself.")
43	+ "-v", "--verbosity", action='count', default=0, required=False,
44	+ help="Increase verbosity of test details and 'text' mode output. "
45	+ "Set twice to also log data useful for debugging autopilot "
46	+ "itself.")

This is incorrect, since this is for the vis tool. The original message was correct, I don't think you need to change it at all.

54	- '-v', '--verbose', required=False, default=False, action='count',
55	- help="Show autopilot log messages. Set twice to also log data useful "
56	- "for debugging autopilot itself.")
57	+ "-v", "--verbosity", action='count', default=0, required=False,
58	+ help="Increase verbosity of test details and 'text' mode output. "
59	+ "Set twice to also log data useful for debugging autopilot "
60	+ "itself.")

Same here - this is for the launch command.

A small thing, but this:

Should be "2012-2014", not "2012,2013,2014", as per legal advice.

102	from autopilot import get_version_string, parse_arguments
103	-import autopilot.globals
104	from autopilot._debug import get_all_debug_profiles
105	-from autopilot.testresult import get_output_formats
106	-from autopilot.utilities import DebugLogFilter, LogFormatter
107	from autopilot.application._launcher import (
108	_get_app_env_from_string_hint,
109	get_application_launcher_wrapper,
110	launch_process,
111	)
112	+import autopilot.globals
113	+from autopilot.testresult import get_output_formats
114	+from autopilot.utilities import DebugLogFilter, LogFormatter

This isn't the standard we've set in the AP codebase. We mix 'import FOO' and 'from FOO import BAR' statements in one block, and alphabeticise the module names.

565	+ def test_failfast_text_mode(self):

Please move the test_failfast test to a new class, and use scenarios to make sure that *all* test result formats support failfast, not just text and status.

Here's the big change I think we need to make:

Currently, the logger is set up to log to the same stream as the result object. This works for the text format, but not the others. Instead, we should make it so the test log is added to the test result as a detail, for every format. This will help us with the subunit format, for example. TBH, I thought this already happend, but I can't see the code now. Feel free to point it out to me :)

Cheers

review: Needs Fixing

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-03-18: Posted in a previous version of this proposal

#

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/590/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

Corey Goldberg (coreygoldberg) wrote on 2014-03-20: Posted in a previous version of this proposal

#

fixed:

- fixed copyright date format.
- reverted help text for -v arg in launch and vis modes
- added scenaro tests for failfast to cover all output formatsbz

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-07-02:

#

FAILED: Continuous integration, rev:482
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~veebers/autopilot/texttest-run-uprefresh/+merge/225247/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/autopilot-ci/748/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-utopic-amd64-ci/22/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-utopic-armhf-ci/22/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-utopic-i386-ci/22/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-utopic-autopilot/147/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-utopic-amd64/1391/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/748/rebuild

review: Needs Fixing (continuous-integration)

Reply

lp:~veebers/autopilot/texttest-run-uprefresh updated on 2014-07-02

483. By Christopher Lee on 2014-07-02: Fix introduced flake8 issues

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-07-02:

#

FAILED: Continuous integration, rev:483
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~veebers/autopilot/texttest-run-uprefresh/+merge/225247/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/autopilot-ci/749/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-utopic-amd64-ci/23/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-utopic-armhf-ci/23/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-utopic-i386-ci/23/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-utopic-autopilot/148/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-utopic-amd64/1392/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/749/rebuild

review: Needs Fixing (continuous-integration)

Reply

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-07-03:

#

FAILED: Continuous integration, rev:483
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~veebers/autopilot/texttest-run-uprefresh/+merge/225247/+edit-commit-message

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/751/rebuild

review: Needs Fixing (continuous-integration)

Reply

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-07-03:

#

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/754/rebuild

review: Needs Fixing (continuous-integration)

Reply

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-07-03:

#

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/758/rebuild

review: Needs Fixing (continuous-integration)

Reply

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-07-09:

#

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/767/rebuild

review: Needs Fixing (continuous-integration)

Reply

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-07-14:

#

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/785/rebuild

review: Needs Fixing (continuous-integration)

Reply

lp:~veebers/autopilot/texttest-run-uprefresh updated on 2014-07-22

484. By Christopher Lee on 2014-07-14: Fix 3 failing tests with typo change
485. By Christopher Lee on 2014-07-22: Fix sp error in test (verbose vs verbosity)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-07-22:

#

FAILED: Continuous integration, rev:485
http://jenkins.qa.ubuntu.com/job/autopilot-ci/810/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-utopic-amd64-ci/85/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-utopic-armhf-ci/84/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-utopic-i386-ci/84/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-utopic-autopilot/225/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-utopic-amd64/2111/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/810/rebuild

review: Needs Fixing (continuous-integration)

Reply

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-07-22:

#

FAILED: Continuous integration, rev:485
http://jenkins.qa.ubuntu.com/job/autopilot-ci/808/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-utopic-amd64-ci/83/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-utopic-armhf-ci/82/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-utopic-i386-ci/82/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-utopic-autopilot/223/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-utopic-amd64/2109/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/808/rebuild

review: Needs Fixing (continuous-integration)

Reply

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-07-22:

#

FAILED: Continuous integration, rev:485
http://jenkins.qa.ubuntu.com/job/autopilot-ci/812/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-utopic-amd64-ci/87/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-utopic-armhf-ci/86/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/autopilot-utopic-i386-ci/86/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-utopic-autopilot/228/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-utopic-amd64/2114/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/812/rebuild

review: Needs Fixing (continuous-integration)

Reply

lp:~veebers/autopilot/texttest-run-uprefresh updated on 2014-07-22

486. By Christopher Lee on 2014-07-22: Merge trunk + fix conflicts

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2014-07-22:

#

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-ci/813/rebuild

review: Needs Fixing (continuous-integration)

Reply

Revision history for this message

platform-qa-bot (platform-qa-bot) wrote on 2016-01-19:

#

PASSED: Continuous integration, rev:486
https://platform-qa-jenkins.ubuntu.com/job/autopilot-ci/13/
Executed test runs:
None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/13/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/autopilot-ci/13/rebuild

review: Approve (continuous-integration)

Reply

Autopilot

Merge lp:~veebers/autopilot/texttest-run-uprefresh into lp:autopilot

Commit message

Description of the change

Unmerged revisions

Preview Diff

Subscribers

 === modified file 'autopilot/__init__.py'
 --- autopilot/__init__.py	2014-05-20 14:43:24 +0000
 +++ autopilot/__init__.py	2014-07-22 03:40:04 +0000
@@ -1,7 +1,7 @@
  # -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+ #
  # Autopilot Functional Test Tool
--# Copyright (C) 2012-2013 Canonical
++# Copyright (C) 2012-2014 Canonical
+ #
  # This program is free software: you can redistribute it and/or modify
  # it under the terms of the GNU General Public License as published by
 === modified file 'autopilot/run.py'
 --- autopilot/run.py	2014-07-02 21:07:33 +0000
 +++ autopilot/run.py	2014-07-22 03:40:04 +0000
@@ -31,15 +31,14 @@
  import os.path
  from platform import node
  from random import shuffle
--import six
  import subprocess
  import sys
  from unittest import TestLoader, TestSuite
++import six
  from testtools import iterate_tests
  from autopilot import get_version_string, have_vis
--import autopilot.globals
  from autopilot import _config as test_config
  from autopilot._debug import (
      get_all_debug_profiles,
@@ -52,6 +51,7 @@
      get_application_launcher_wrapper,
      launch_process,
+ )
++import autopilot.globals
  def _get_parser():
@@ -215,15 +215,18 @@
          setattr(namespace, self.dest, values)
--def setup_logging(verbose):
--    """Configure the root logger and verbose logging to stderr."""
++def setup_logging(verbosity, path=None, format=None):
++    """Configure the root logger"""
      root_logger = get_root_logger()
      root_logger.setLevel(logging.DEBUG)
--    if verbose == 0:
++    if format == 'text':
++        stream = get_output_stream(format, path)
++        set_stream_handler(root_logger, stream)
++    else:
          set_null_log_handler(root_logger)
--    if verbose >= 1:
--        set_stderr_stream_handler(root_logger)
--    if verbose >= 2:
++    if verbosity >= 1:
++        autopilot.globals.set_log_verbose(True)
++    if verbosity >= 2:
          enable_debug_log_messages()
      # log autopilot version
      root_logger.info(get_version_string())
@@ -237,11 +240,10 @@
      root_logger.addHandler(logging.NullHandler())
--def set_stderr_stream_handler(root_logger):
--    formatter = LogFormatter()
--    stderr_handler = logging.StreamHandler(stream=sys.stderr)
--    stderr_handler.setFormatter(formatter)
--    root_logger.addHandler(stderr_handler)
++def set_stream_handler(root_logger, stream):
++    handler = logging.StreamHandler(stream=stream)
++    handler.setFormatter(LogFormatter())
++    root_logger.addHandler(handler)
  def enable_debug_log_messages():
@@ -256,7 +258,7 @@
+     )
--def get_output_stream(format, path):
++def get_output_stream(format, path=None):
      """Get an output stream pointing to 'path' that's appropriate for format
      'format'.
@@ -270,7 +272,7 @@
          log_file = _get_log_file_path(path)
          if format == 'xml':
              return _get_text_mode_file_stream(log_file)
--        elif format == 'text':
++        elif format in ('status', 'text'):
              return _get_binary_mode_file_stream(log_file)
          else:
              return _get_raw_binary_mode_file_stream(log_file)
@@ -639,7 +641,10 @@
          self.args = defined_args or _parse_arguments()
      def run(self):
--        setup_logging(getattr(self.args, 'verbose', False))
++        verbosity = getattr(self.args, 'verbose', 0)
++        format = getattr(self.args, 'format', 'status')
++        path = getattr(self.args, 'output', '')
++        setup_logging(verbosity, path, format)
          action = None
          if self.args.mode == 'list':
@@ -712,9 +717,6 @@
              print("Error: %s" % str(e))
              exit(1)
--        if self.args.verbose:
--            autopilot.globals.set_log_verbose(True)
--
          result = construct_test_result(self.args)
          result.startTestRun()
          try:
 === modified file 'autopilot/testresult.py'
 --- autopilot/testresult.py	2014-07-14 23:30:58 +0000
 +++ autopilot/testresult.py	2014-07-22 03:40:04 +0000
@@ -1,7 +1,7 @@
  # -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+ #
  # Autopilot Functional Test Tool
--# Copyright (C) 2012-2013 Canonical
++# Copyright (C) 2012,2013,2014 Canonical
+ #
  # This program is free software: you can redistribute it and/or modify
  # it under the terms of the GNU General Public License as published by
@@ -32,18 +32,81 @@
      try_import,
+ )
--from autopilot.globals import get_log_verbose
  from autopilot.utilities import _raise_on_unknown_kwargs
++class StatusFormatTestResult(TextTestResult):
++    """A TextTestResult that prints status messages to a stream."""
++
++    def __init__(self, stream, failfast):
++        super(StatusFormatTestResult, self).__init__(stream, failfast)
++        self.stream = stream
++
++    def startTest(self, test):
++        self.stream.write(u'%s' % test.id())
++        self.stream.write(u' ... ')
++        self.stream.flush()
++        super(StatusFormatTestResult, self).startTest(test)
++
++    def stopTest(self, test):
++        self.stream.write(u'\n')
++        self.stream.flush()
++        super(StatusFormatTestResult, self).stopTest(test)
++
++    def stopTestRun(self):
++        self.stream.write(u'-' * 80)
++        self.stream.write(u'\n')
++        self.stream.flush()
++        super(StatusFormatTestResult, self).stopTestRun()
++
++    def addExpectedFailure(self, test, err=None, details=None):
++        self.stream.write(u'EXPECTED FAIL')
++        super(StatusFormatTestResult, self).addExpectedFailure(
++            test, err, details
++        )
++
++    def addError(self, test, err=None, details=None):
++        self.stream.write(u'ERROR')
++        super(StatusFormatTestResult, self).addError(
++            test, err, details
++        )
++
++    def addFailure(self, test, err=None, details=None):
++        self.stream.write(u'FAIL')
++        super(StatusFormatTestResult, self).addFailure(
++            test, err, details
++        )
++
++    def addSkip(self, test, reason, details=None):
++        if not reason:
++            reason_displayed = ''
++        else:
++            reason_displayed = ' (%s)' % reason
++        self.stream.write(u'SKIP%s' % reason_displayed)
++        super(StatusFormatTestResult, self).addSkip(
++            test, reason, details
++        )
++
++    def addSuccess(self, test, details=None):
++        self.stream.write(u'OK')
++        super(StatusFormatTestResult, self).addSuccess(
++            test, details
++        )
++
++    def addUnexpectedSuccess(self, test, details=None):
++        self.stream.write(u'UNEXPECTED SUCCESS')
++        super(StatusFormatTestResult, self).addUnexpectedSuccess(
++            test, details
++        )
++
++
  class LoggedTestResultDecorator(TestResultDecorator):
      """A decorator that logs messages to python's logging system."""
      def _log(self, level, message):
--        """Perform the actual message logging."""
--        if get_log_verbose():
--            logging.getLogger().log(level, message)
++        """Perform the actual message logging"""
++        logging.getLogger().log(level, message)
      def _log_details(self, level, details):
          """Log the relavent test details."""
@@ -87,9 +150,8 @@
      """
      supported_formats = {}
--
++    supported_formats['status'] = _construct_status
      supported_formats['text'] = _construct_text
--
      if try_import('junitxml'):
          supported_formats['xml'] = _construct_xml
      if try_import('subunit'):
@@ -98,7 +160,7 @@
  def get_default_format():
--    return 'text'
++    return 'status'
  def _construct_xml(**kwargs):
@@ -106,16 +168,21 @@
      stream = kwargs.pop('stream')
      failfast = kwargs.pop('failfast')
      _raise_on_unknown_kwargs(kwargs)
--    result_object = LoggedTestResultDecorator(
--        ExtendedToOriginalDecorator(
--            JUnitXmlResult(stream)
--        )
--    )
--    result_object.failfast = failfast
--    return result_object
++    result = ExtendedToOriginalDecorator(JUnitXmlResult(stream))
++    result.failfast = failfast
++    return result
++
++
++def _construct_status(**kwargs):
++    """Status (terse) text output."""
++    stream = kwargs.pop('stream')
++    failfast = kwargs.pop('failfast')
++    _raise_on_unknown_kwargs(kwargs)
++    return StatusFormatTestResult(stream, failfast)
  def _construct_text(**kwargs):
++    """Verbose text output."""
      stream = kwargs.pop('stream')
      failfast = kwargs.pop('failfast')
      _raise_on_unknown_kwargs(kwargs)
@@ -127,10 +194,6 @@
      stream = kwargs.pop('stream')
      failfast = kwargs.pop('failfast')
      _raise_on_unknown_kwargs(kwargs)
--    result_object = LoggedTestResultDecorator(
--        ExtendedToStreamDecorator(
--            StreamResultToBytes(stream)
--        )
--    )
--    result_object.failfast = failfast
--    return result_object
++    result = ExtendedToStreamDecorator(StreamResultToBytes(stream))
++    result.failfast = failfast
++    return result
 === modified file 'autopilot/tests/functional/test_autopilot_functional.py'
 --- autopilot/tests/functional/test_autopilot_functional.py	2014-07-14 04:07:05 +0000
 +++ autopilot/tests/functional/test_autopilot_functional.py	2014-07-22 03:40:04 +0000
@@ -25,9 +25,11 @@
  import os.path
  import re
  from tempfile import mktemp
++from textwrap import dedent
++
++from testscenarios import WithScenarios
  from testtools import skipIf
  from testtools.matchers import Contains, Equals, MatchesRegex, Not
--from textwrap import dedent
  from autopilot import platform
  from autopilot.tests.functional import AutopilotRunTestBase, remove_if_exists
@@ -770,15 +772,10 @@
  class AutopilotVerboseFunctionalTests(AutopilotFunctionalTestsBase):
--    """Scenarioed functional tests for autopilot's verbose logging."""
--
--    scenarios = [
--        ('text_format', dict(output_format='text')),
--        ('xml_format', dict(output_format='xml'))
--    ]
++    """Functional tests for autopilot's verbose logging."""
      def test_verbose_flag_works(self):
--        """Verbose flag must log to stderr."""
++        """Verbose flag must log to stdout."""
          self.create_test_file(
              "test_simple.py", dedent("""\
@@ -792,13 +789,13 @@
              """)
+         )
--        code, output, error = self.run_autopilot(["run",
--                                                  "-f", self.output_format,
--                                                  "-v", "tests"])
++        code, output, error = self.run_autopilot(
++            ["run", "-f", "text", "-v", "tests"]
++        )
          self.assertThat(code, Equals(0))
          self.assertThat(
--            error, Contains(
++            output, Contains(
                  "Starting test tests.test_simple.SimpleTest.test_simple"))
      def test_verbose_flag_shows_timestamps(self):
@@ -816,11 +813,11 @@
              """)
+         )
--        code, output, error = self.run_autopilot(["run",
--                                                  "-f", self.output_format,
--                                                  "-v", "tests"])
++        code, output, error = self.run_autopilot(
++            ["run", "-f", "text", "-v", "tests"]
++        )
--        self.assertThat(error, MatchesRegex("^\d\d:\d\d:\d\d\.\d\d\d"))
++        self.assertThat(output, MatchesRegex("^\d\d:\d\d:\d\d\.\d\d\d"))
      def test_verbose_flag_shows_success(self):
          """Verbose log must indicate successful tests (text format)."""
@@ -837,12 +834,12 @@
              """)
+         )
--        code, output, error = self.run_autopilot(["run",
--                                                  "-f", self.output_format,
--                                                  "-v", "tests"])
++        code, output, error = self.run_autopilot(
++            ["run", "-f", "text", "-v", "tests"]
++        )
          self.assertThat(
--            error, Contains("OK: tests.test_simple.SimpleTest.test_simple"))
++            output, Contains("OK: tests.test_simple.SimpleTest.test_simple"))
      def test_verbose_flag_shows_error(self):
          """Verbose log must indicate test error with a traceback."""
@@ -859,15 +856,17 @@
              """)
+         )
--        code, output, error = self.run_autopilot(["run",
--                                                  "-f", self.output_format,
--                                                  "-v", "tests"])
++        code, output, error = self.run_autopilot(
++            ["run", "-f", "text", "-v", "tests"]
++        )
          self.assertThat(
--            error, Contains("ERROR: tests.test_simple.SimpleTest.test_simple"))
--        self.assertThat(error, Contains("traceback:"))
++            output,
++            Contains("ERROR: tests.test_simple.SimpleTest.test_simple")
++        )
++        self.assertThat(output, Contains("traceback:"))
          self.assertThat(
--            error,
++            output,
              Contains("RuntimeError: Intentionally fail test.")
+         )
@@ -887,13 +886,13 @@
              """)
+         )
--        code, output, error = self.run_autopilot(["run",
--                                                  "-f", self.output_format,
--                                                  "-v", "tests"])
++        code, output, error = self.run_autopilot(
++            ["run", "-f", "text", "-v", "tests"]
++        )
--        self.assertIn("FAIL: tests.test_simple.SimpleTest.test_simple", error)
--        self.assertIn("traceback:", error)
--        self.assertIn("AssertionError: False is not true", error)
++        self.assertIn("FAIL: tests.test_simple.SimpleTest.test_simple", output)
++        self.assertIn("traceback:", output)
++        self.assertIn("AssertionError: False is not true", output)
      def test_verbose_flag_captures_nested_autopilottestcase_classes(self):
          """Verbose log must contain the log details of both the nested and
@@ -917,20 +916,20 @@
              """)
+         )
--        code, output, error = self.run_autopilot(["run",
--                                                  "-f", self.output_format,
--                                                  "-v", "tests"])
++        code, output, error = self.run_autopilot(
++            ["run", "-f", "text", "-v", "tests"])
          self.assertThat(code, Equals(0))
++
          self.assertThat(
--            error,
++            output,
              Contains(
                  "Starting test tests.test_simple.OuterTestCase."
                  "test_nested_classes"
+             )
+         )
          self.assertThat(
--            error,
++            output,
              Contains(
                  "Starting test tests.test_simple.InnerTestCase."
                  "test_produce_log_output"
@@ -953,11 +952,10 @@
              """)
+         )
--        code, output, error = self.run_autopilot(["run",
--                                                  "-f", self.output_format,
--                                                  "-vv", "tests"])
++        code, output, error = self.run_autopilot(
++            ["run", "-f", "text", "-vv", "tests"])
--        self.assertThat(error, Contains("Hello World"))
++        self.assertThat(output, Contains("Hello World"))
      def test_debug_output_not_shown_by_default(self):
          """Verbose log must not show debug messages unless we specify '-vv'."""
@@ -975,11 +973,11 @@
              """)
+         )
--        code, output, error = self.run_autopilot(["run",
--                                                  "-f", self.output_format,
--                                                  "-v", "tests"])
++        code, output, error = self.run_autopilot(
++            ["run", "-f", "text", "-v", "tests"]
++        )
--        self.assertThat(error, Not(Contains("Hello World")))
++        self.assertThat(output, Not(Contains("Hello World")))
      def test_verbose_flag_shows_autopilot_version(self):
          from autopilot import get_version_string
@@ -997,13 +995,23 @@
              """)
+         )
--        code, output, error = self.run_autopilot(["run",
--                                                  "-f", self.output_format,
--                                                  "-v", "tests"])
++        code, output, error = self.run_autopilot(
++            ["run", "-f", "text", "-v", "tests"]
++        )
          self.assertThat(
--            error, Contains(get_version_string()))
--
--    def test_failfast(self):
++            output, Contains(get_version_string()))
++
++
++class AutopilotFailFastTests(AutopilotRunTestBase, WithScenarios):
++
++    scenarios = [
++        ('status', dict(format='status')),
++        ('text', dict(format='text')),
++        ('xml', dict(format='xml')),
++        ('subunit', dict(format='subunit')),
++    ]
++
++    def test_failfast_text_mode(self):
          """Run stops after first error encountered."""
          self.create_test_file(
              'test_failfast.py', dedent("""\
@@ -1020,9 +1028,34 @@
                      raise Exception
              """)
+         )
--        code, output, error = self.run_autopilot(["run",
--                                                  "--failfast",
--                                                  "tests"])
++        code, output, error = self.run_autopilot(
++            ["run", "--failfast", "-f", self.format, "tests"]
++        )
          self.assertThat(code, Equals(1))
          self.assertIn("Ran 1 test", output)
          self.assertIn("FAILED (failures=1)", output)
++
++
++class AutopilotTestDetailsTests(AutopilotRunTestBase):
++
++    def test_logging_added_as_test_detail(self):
++        msg = "This is some information"
++        self.create_test_file(
++            'test_details.py', dedent("""\
++            import logging
++            from autopilot.testcase import AutopilotTestCase
++
++
++            logger = logging.getLogger(__name__)
++
++            class SimpleTest(AutopilotTestCase):
++
++                def test_one(self):
++                    logger.info('%s')
++
++            """ % msg)
++        )
++        code, output, error = self.run_autopilot(
++            ["run", "-f", "text", "tests"]
++        )
++        self.assertIn(msg, output)
 === modified file 'autopilot/tests/unit/test_command_line_args.py'
 --- autopilot/tests/unit/test_command_line_args.py	2014-05-23 11:28:21 +0000
 +++ autopilot/tests/unit/test_command_line_args.py	2014-07-22 03:40:04 +0000
@@ -76,7 +76,7 @@
      def test_launch_command_has_correct_default_verbosity(self):
          args = parse_args("launch app")
--        self.assertThat(args.verbose, Equals(False))
++        self.assertThat(args.verbose, Equals(0))
      def test_launch_command_can_specify_verbosity(self):
          args = parse_args("launch -v app")
@@ -211,7 +211,15 @@
      def test_run_command_default_format(self):
          args = parse_args('run foo')
--        self.assertThat(args.format, Equals("text"))
++        self.assertThat(args.format, Equals("status"))
++
++    def test_run_command_status_format_short_version(self):
++        args = parse_args('run -f status foo')
++        self.assertThat(args.format, Equals("status"))
++
++    def test_run_command_status_format_long_version(self):
++        args = parse_args('run --format status foo')
++        self.assertThat(args.format, Equals("status"))
      def test_run_command_text_format_short_version(self):
          args = parse_args('run -f text foo')
 === modified file 'autopilot/tests/unit/test_run.py'
 --- autopilot/tests/unit/test_run.py	2014-05-23 11:28:21 +0000
 +++ autopilot/tests/unit/test_run.py	2014-07-22 03:40:04 +0000
@@ -18,7 +18,7 @@
+ #
  from argparse import Namespace
--from unittest.mock import Mock, patch
++from io import StringIO
  import logging
  import os.path
  from shutil import rmtree
@@ -37,12 +37,17 @@
      Raises,
      StartsWith,
+ )
++from unittest.mock import Mock, patch
  if six.PY3:
      from contextlib import ExitStack
  else:
      from contextlib2 import ExitStack
++from autopilot.testresult import (
++    LoggedTestResultDecorator,
++    StatusFormatTestResult,
++)
  from autopilot import have_vis, run
@@ -595,27 +600,10 @@
                  logging.DEBUG
+             )
--    def test_set_null_log_handler(self):
--        mock_root_logger = Mock()
--        run.set_null_log_handler(mock_root_logger)
--
--        self.assertThat(
--            mock_root_logger.addHandler.call_args[0][0],
--            IsInstance(logging.NullHandler)
--        )
--
--    @patch.object(run, 'get_root_logger')
--    def test_verbse_level_zero_sets_null_handler(self, fake_get_logger):
--        with patch.object(run, 'set_null_log_handler') as fake_set_null:
--            run.setup_logging(0)
--
--            fake_set_null.assert_called_once_with(
--                fake_get_logger.return_value
--            )
--
--    def test_stderr_handler_sets_stream_handler_with_custom_formatter(self):
--        mock_root_logger = Mock()
--        run.set_stderr_stream_handler(mock_root_logger)
++    def test_stream_handler_sets_custom_formatter(self):
++        mock_root_logger = Mock()
++        stream = StringIO()
++        run.set_stream_handler(mock_root_logger, stream)
          self.assertThat(mock_root_logger.addHandler.call_count, Equals(1))
          created_handler = mock_root_logger.addHandler.call_args[0][0]
@@ -630,12 +618,16 @@
+         )
      @patch.object(run, 'get_root_logger')
--    def test_verbose_level_one_sets_stream_handler(self, fake_get_logger):
--        with patch.object(run, 'set_stderr_stream_handler') as stderr_handler:
--            run.setup_logging(1)
++    def test_text_format_sets_stream_handler(self, fake_get_logger):
++        verbosity = 0
++        path = None
++        # handler is set when running in 'text' format mode
++        with patch.object(run, 'set_stream_handler') as handler:
++            run.setup_logging(verbosity, path, format='text')
--            stderr_handler.assert_called_once_with(
--                fake_get_logger.return_value
++            handler.assert_called_once_with(
++                fake_get_logger.return_value,
++                run.get_output_stream('text')
+             )
      def test_enable_debug_log_messages_sets_debugFilter_attr(self):
@@ -650,7 +642,7 @@
      @patch.object(run, 'get_root_logger')
      def test_verbose_level_two_enables_debug_messages(self, fake_get_logger):
          with patch.object(run, 'enable_debug_log_messages') as enable_debug:
--            run.setup_logging(2)
++            run.setup_logging(2, None)
              enable_debug.assert_called_once_with()
@@ -791,6 +783,13 @@
              run.get_output_stream(format, output)
              pgts.assert_called_once_with(output)
++    def test_status_format_opens_binary_mode_stream(self):
++        output = tempfile.mktemp()
++        format = 'status'
++        with patch.object(run, '_get_binary_mode_file_stream') as pgbs:
++            run.get_output_stream(format, output)
++            pgbs.assert_called_once_with(output)
++
      def test_txt_format_opens_binary_mode_stream(self):
          output = tempfile.mktemp()
          format = 'text'
@@ -815,6 +814,14 @@
          expected = "Using default log filename: %s\n" % path
          self.assertThat(expected, Equals(output))
++    def test_text_result_uses_logged_decorator(self):
++        args = Namespace()
++        args.failfast = False
++        args.output = None
++        args.format = "text"
++        result = run.construct_test_result(args)
++        self.assertIsInstance(result, LoggedTestResultDecorator)
++
  class TestProgramTests(TestCase):
@@ -841,12 +848,12 @@
              self.assertThat(program.args, Equals(fake_args))
      def test_run_calls_setup_logging_with_verbose_arg(self):
--        fake_args = Namespace(verbose=1, mode='')
++        fake_args = Namespace(verbose=1, mode='text')
          program = run.TestProgram(fake_args)
          with patch.object(run, 'setup_logging') as patched_setup_logging:
              program.run()
--            patched_setup_logging.assert_called_once_with(True)
++            patched_setup_logging.assert_called_once_with(1, '', 'status')
      def test_list_command_calls_list_tests_method(self):
          fake_args = Namespace(mode='list')
@@ -966,3 +973,93 @@
+     )
      defaults.update(kwargs)
      return Namespace(**defaults)
++
++
++def get_case(kind):
++    # Define the class in a function so test loading doesn't
++    #   try to load it as a regular test class.
++    class Test(TestCase):
++
++        def test_pass(self):
++            pass
++
++        def test_fail(self):
++            raise self.failureException
++
++        def test_error(self):
++            raise SyntaxError
++
++        def test_skip(self):
++            self.skipTest('')
++
++        def test_skip_reason(self):
++            self.skipTest('Because')
++
++        def test_expected_failure(self):
++            # We expect the test to fail and it does
++            self.expectFailure("1 should be 0", self.assertEqual, 1, 0)
++
++        def test_unexpected_success(self):
++            # We expect the test to fail but it doesn't
++            self.expectFailure("1 is not 1", self.assertEqual, 1, 1)
++
++    test_method = 'test_%s' % kind
++    return Test(test_method)
++
++
++def expand_template_for_test(template, test):
++    """Expand common references in template.
++
++    Tests that check runs output can be simplified if they use templates
++    instead of literal expected strings.
++
++    :param template: A string where common strings have been replaced by a
++        keyword.
++
++    :param test: The test case under scrutiny.
++    """
++    kwargs = dict(id=test.id())
++    return template.format(**kwargs)
++
++
++class StatusFormatTestResultTests(TestCase):
++
++    def setUp(self):
++        super(StatusFormatTestResultTests, self).setUp()
++
++    def assertOutput(self, template, kind):
++        test = get_case(kind)
++        result = StatusFormatTestResult(StringIO(), False)
++        test.run(result)
++
++        expected = expand_template_for_test(template, test)
++        self.assertEquals(expected, result.stream.getvalue().strip())
++
++    def test_pass(self):
++        self.assertOutput('{id} ... OK', 'pass')
++
++    def test_fail(self):
++        self.assertOutput('{id} ... FAIL', 'fail')
++
++    def test_error(self):
++        self.assertOutput('{id} ... ERROR', 'error')
++
++    def test_skip(self):
++        self.assertOutput('{id} ... SKIP', 'skip')
++
++    def test_skip_reason(self):
++        self.assertOutput('{id} ... SKIP (Because)', 'skip_reason')
++
++    def test_expected_failure(self):
++        self.assertOutput('{id} ... EXPECTED FAIL', 'expected_failure')
++
++    def test_unexpected_success(self):
++        self.assertOutput('{id} ... UNEXPECTED SUCCESS', 'unexpected_success')
++
++    def test_result_uses_status_class(self):
++        args = Namespace()
++        args.failfast = False
++        args.output = None
++        args.format = 'status'
++        result = run.construct_test_result(args)
++        self.assertIsInstance(result, StatusFormatTestResult)
 === modified file 'autopilot/tests/unit/test_testresults.py'
 --- autopilot/tests/unit/test_testresults.py	2014-07-01 00:36:26 +0000
 +++ autopilot/tests/unit/test_testresults.py	2014-07-22 03:40:04 +0000
@@ -27,8 +27,7 @@
  from testscenarios import WithScenarios
  import unittest
--from autopilot import testresult
--from autopilot import run
++from autopilot import run, testresult
  class LoggedTestResultDecoratorTests(TestCase):
@@ -110,6 +109,9 @@
  class OutputFormatFactoryTests(TestCase):
++    def test_has_status_format(self):
++        self.assertTrue('status' in testresult.get_output_formats())
++
      def test_has_text_format(self):
          self.assertTrue('text' in testresult.get_output_formats())
@@ -278,6 +280,22 @@
          self.assertFalse(test_result.wasSuccessful())
          self.assertEqual(1, test_result.testsRun)
++    def test_traceback_added_as_detail_on_fail(self):
++        msg = "failure info"
++
++        class FailingTests(TestCase):
++
++            def test_fails(self):
++                self.fail(msg)
++
++        test = FailingTests('test_fails')
++        test_result, output_path = self.run_test_with_result(
++            test
++        )
++        self.assertFalse(test_result.wasSuccessful())
++        tb_detail = test.getDetails()['traceback'].as_text()
++        self.assertThat(tb_detail, Contains(msg))
++
  def remove_if_exists(path):
      if os.path.exists(path):