Merge lp:~jml/testtools/unexpected-success-2 into lp:~testtools-committers/testtools/trunk
- unexpected-success-2
- Merge into trunk
Status: | Merged | ||||
---|---|---|---|---|---|
Merged at revision: | 142 | ||||
Proposed branch: | lp:~jml/testtools/unexpected-success-2 | ||||
Merge into: | lp:~testtools-committers/testtools/trunk | ||||
Diff against target: |
570 lines (+233/-38) 5 files modified
NEWS (+6/-1) testtools/testresult/doubles.py (+17/-1) testtools/testresult/real.py (+38/-3) testtools/tests/test_compat.py (+8/-7) testtools/tests/test_testresult.py (+164/-26) |
||||
To merge this branch: | bzr merge lp:~jml/testtools/unexpected-success-2 | ||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Martin Packman | Needs Fixing | ||
Robert Collins | Approve | ||
Review via email: mp+42050@code.launchpad.net |
Commit message
Description of the change
This branch finished off the rest of bug 654474. It changes the contract of TestResult such that wasSuccessful returns False if there has ever been an unexpected success.
In addition, it updates the contract tests to test for the behaviour of wasSuccessful in many more cases, and adds contract tests for Python 2.6 and 2.7 which we run against our fake implementations.
- 145. By Jonathan Lange
-
Update NEWS
Jonathan Lange (jml) wrote : | # |
Agreed on both points. Will only update startTestRun for testtools
TestResult, since I'm not controlling Python.
Would be better with testscenarios, but not in this branch. Also, we need
to have a chat about dependencies.
Thanks,
jml
Martin Packman (gz) wrote : | # |
On Python 3 (with lp:~gz/testtools/raises_regressions_675327 merged) I get three failures. All in TestMultiTestre
There are actually four failures, but one is silent...
As noted by Martin Pool in bug 654474 comment 5 changing the outcome without also providing feedback is bad, so I think TextTestResult.
=== modified file 'testtools/
--- testtools/
+++ testtools/
@@ -258,6 +261,8 @@
stop = self._now()
+ for test in self.unexpected
+ self.stream.
@@ -267,7 +272,7 @@
- len(self.failures) + len(self.errors)))
+ len(self.failures) + len(self.errors) + len(self.
The silent failure that should then be revealed on Python 3 also needs fixing:
=== modified file 'testtools/
--- testtools/
+++ testtools/
@@ -19,6 +19,7 @@
)
from testtools.matchers import (
MatchesExc
+ Not,
Raises,
)
@@ -246,7 +247,7 @@
if newio:
- Raises(
+ Not(Raises(
Final nit:
+ def test_addUnexpec
+ # addUnexpectedSu
Verb?
Jonathan Lange (jml) wrote : | # |
mgz, thanks very much for the review. I've fixed the nits, TextTestResult, the MultiTestResult failure (map(f, xs) returns an iterator in Python 3) and the hidden failure in test_compat. I also added tests for the TextTestResult changes.
lifeless, I've made the startTestRun change.
I'm going to merge it now, but I'd welcome any comments you have on the incremental diff.
- 146. By Jonathan Lange
-
minor nits
- 147. By Jonathan Lange
-
Make MultiTestResult at all usable in Python 3
- 148. By Jonathan Lange
-
startTestRun resets unexpected successes and other errors
- 149. By Jonathan Lange
-
NEWS update
- 150. By Jonathan Lange
-
Actually report the error in TextTestResult (thanks mgz)
- 151. By Jonathan Lange
-
Fix a inaccurate unexpected success, hidden by our broken Python 3 support
Robert Collins (lifeless) wrote : | # |
+ len(self.failures) + len(self.errors)))
+ len(self.failures) + len(self.errors)
+ + len(self.
sum(map(len, (self, self.failures, self.errors,
self.
Would be cleaner, I think?
Preview Diff
1 | === modified file 'NEWS' | |||
2 | --- NEWS 2010-11-29 00:11:19 +0000 | |||
3 | +++ NEWS 2010-11-29 00:28:14 +0000 | |||
4 | @@ -8,7 +8,12 @@ | |||
5 | 8 | ------- | 8 | ------- |
6 | 9 | 9 | ||
7 | 10 | * addUnexpectedSuccess is translated to addFailure for test results that don't | 10 | * addUnexpectedSuccess is translated to addFailure for test results that don't |
9 | 11 | know about addUnexpectedSuccess. (Jonathan Lange, #654474) | 11 | know about addUnexpectedSuccess. Further, it fails the entire result for |
10 | 12 | all testtools TestResults (i.e. wasSuccessful() returns False after | ||
11 | 13 | addUnexpectedSuccess has been called). (Jonathan Lange, #654474) | ||
12 | 14 | |||
13 | 15 | * startTestRun will reset any errors on the result. That is, wasSuccessful() | ||
14 | 16 | will always return True immediately after startTestRun() is called. | ||
15 | 12 | 17 | ||
16 | 13 | * Responsibility for running test cleanups has been moved to ``RunTest``. | 18 | * Responsibility for running test cleanups has been moved to ``RunTest``. |
17 | 14 | This change does not affect public APIs and can be safely ignored by test | 19 | This change does not affect public APIs and can be safely ignored by test |
18 | 15 | 20 | ||
19 | === modified file 'testtools/testresult/doubles.py' | |||
20 | --- testtools/testresult/doubles.py 2009-12-31 03:15:19 +0000 | |||
21 | +++ testtools/testresult/doubles.py 2010-11-29 00:28:14 +0000 | |||
22 | @@ -1,4 +1,4 @@ | |||
24 | 1 | # Copyright (c) 2009 Jonathan M. Lange. See LICENSE for details. | 1 | # Copyright (c) 2009-2010 Jonathan M. Lange. See LICENSE for details. |
25 | 2 | 2 | ||
26 | 3 | """Doubles of test result objects, useful for testing unittest code.""" | 3 | """Doubles of test result objects, useful for testing unittest code.""" |
27 | 4 | 4 | ||
28 | @@ -15,15 +15,18 @@ | |||
29 | 15 | def __init__(self): | 15 | def __init__(self): |
30 | 16 | self._events = [] | 16 | self._events = [] |
31 | 17 | self.shouldStop = False | 17 | self.shouldStop = False |
32 | 18 | self._was_successful = True | ||
33 | 18 | 19 | ||
34 | 19 | 20 | ||
35 | 20 | class Python26TestResult(LoggingBase): | 21 | class Python26TestResult(LoggingBase): |
36 | 21 | """A precisely python 2.6 like test result, that logs.""" | 22 | """A precisely python 2.6 like test result, that logs.""" |
37 | 22 | 23 | ||
38 | 23 | def addError(self, test, err): | 24 | def addError(self, test, err): |
39 | 25 | self._was_successful = False | ||
40 | 24 | self._events.append(('addError', test, err)) | 26 | self._events.append(('addError', test, err)) |
41 | 25 | 27 | ||
42 | 26 | def addFailure(self, test, err): | 28 | def addFailure(self, test, err): |
43 | 29 | self._was_successful = False | ||
44 | 27 | self._events.append(('addFailure', test, err)) | 30 | self._events.append(('addFailure', test, err)) |
45 | 28 | 31 | ||
46 | 29 | def addSuccess(self, test): | 32 | def addSuccess(self, test): |
47 | @@ -38,6 +41,9 @@ | |||
48 | 38 | def stopTest(self, test): | 41 | def stopTest(self, test): |
49 | 39 | self._events.append(('stopTest', test)) | 42 | self._events.append(('stopTest', test)) |
50 | 40 | 43 | ||
51 | 44 | def wasSuccessful(self): | ||
52 | 45 | return self._was_successful | ||
53 | 46 | |||
54 | 41 | 47 | ||
55 | 42 | class Python27TestResult(Python26TestResult): | 48 | class Python27TestResult(Python26TestResult): |
56 | 43 | """A precisely python 2.7 like test result, that logs.""" | 49 | """A precisely python 2.7 like test result, that logs.""" |
57 | @@ -62,9 +68,11 @@ | |||
58 | 62 | """A test result like the proposed extended unittest result API.""" | 68 | """A test result like the proposed extended unittest result API.""" |
59 | 63 | 69 | ||
60 | 64 | def addError(self, test, err=None, details=None): | 70 | def addError(self, test, err=None, details=None): |
61 | 71 | self._was_successful = False | ||
62 | 65 | self._events.append(('addError', test, err or details)) | 72 | self._events.append(('addError', test, err or details)) |
63 | 66 | 73 | ||
64 | 67 | def addFailure(self, test, err=None, details=None): | 74 | def addFailure(self, test, err=None, details=None): |
65 | 75 | self._was_successful = False | ||
66 | 68 | self._events.append(('addFailure', test, err or details)) | 76 | self._events.append(('addFailure', test, err or details)) |
67 | 69 | 77 | ||
68 | 70 | def addExpectedFailure(self, test, err=None, details=None): | 78 | def addExpectedFailure(self, test, err=None, details=None): |
69 | @@ -80,6 +88,7 @@ | |||
70 | 80 | self._events.append(('addSuccess', test)) | 88 | self._events.append(('addSuccess', test)) |
71 | 81 | 89 | ||
72 | 82 | def addUnexpectedSuccess(self, test, details=None): | 90 | def addUnexpectedSuccess(self, test, details=None): |
73 | 91 | self._was_successful = False | ||
74 | 83 | if details is not None: | 92 | if details is not None: |
75 | 84 | self._events.append(('addUnexpectedSuccess', test, details)) | 93 | self._events.append(('addUnexpectedSuccess', test, details)) |
76 | 85 | else: | 94 | else: |
77 | @@ -88,8 +97,15 @@ | |||
78 | 88 | def progress(self, offset, whence): | 97 | def progress(self, offset, whence): |
79 | 89 | self._events.append(('progress', offset, whence)) | 98 | self._events.append(('progress', offset, whence)) |
80 | 90 | 99 | ||
81 | 100 | def startTestRun(self): | ||
82 | 101 | super(ExtendedTestResult, self).startTestRun() | ||
83 | 102 | self._was_successful = True | ||
84 | 103 | |||
85 | 91 | def tags(self, new_tags, gone_tags): | 104 | def tags(self, new_tags, gone_tags): |
86 | 92 | self._events.append(('tags', new_tags, gone_tags)) | 105 | self._events.append(('tags', new_tags, gone_tags)) |
87 | 93 | 106 | ||
88 | 94 | def time(self, time): | 107 | def time(self, time): |
89 | 95 | self._events.append(('time', time)) | 108 | self._events.append(('time', time)) |
90 | 109 | |||
91 | 110 | def wasSuccessful(self): | ||
92 | 111 | return self._was_successful | ||
93 | 96 | 112 | ||
94 | === modified file 'testtools/testresult/real.py' | |||
95 | --- testtools/testresult/real.py 2010-10-24 09:21:01 +0000 | |||
96 | +++ testtools/testresult/real.py 2010-11-29 00:28:14 +0000 | |||
97 | @@ -108,6 +108,18 @@ | |||
98 | 108 | """Called when a test was expected to fail, but succeed.""" | 108 | """Called when a test was expected to fail, but succeed.""" |
99 | 109 | self.unexpectedSuccesses.append(test) | 109 | self.unexpectedSuccesses.append(test) |
100 | 110 | 110 | ||
101 | 111 | def wasSuccessful(self): | ||
102 | 112 | """Has this result been successful so far? | ||
103 | 113 | |||
104 | 114 | If there have been any errors, failures or unexpected successes, | ||
105 | 115 | return False. Otherwise, return True. | ||
106 | 116 | |||
107 | 117 | Note: This differs from standard unittest in that we consider | ||
108 | 118 | unexpected successes to be equivalent to failures, rather than | ||
109 | 119 | successes. | ||
110 | 120 | """ | ||
111 | 121 | return not (self.errors or self.failures or self.unexpectedSuccesses) | ||
112 | 122 | |||
113 | 111 | if str_is_unicode: | 123 | if str_is_unicode: |
114 | 112 | # Python 3 and IronPython strings are unicode, use parent class method | 124 | # Python 3 and IronPython strings are unicode, use parent class method |
115 | 113 | _exc_info_to_unicode = unittest.TestResult._exc_info_to_string | 125 | _exc_info_to_unicode = unittest.TestResult._exc_info_to_string |
116 | @@ -148,6 +160,9 @@ | |||
117 | 148 | 160 | ||
118 | 149 | New in python 2.7 | 161 | New in python 2.7 |
119 | 150 | """ | 162 | """ |
120 | 163 | self.unexpectedSuccesses = [] | ||
121 | 164 | self.errors = [] | ||
122 | 165 | self.failures = [] | ||
123 | 151 | 166 | ||
124 | 152 | def stopTestRun(self): | 167 | def stopTestRun(self): |
125 | 153 | """Called after a test run completes | 168 | """Called after a test run completes |
126 | @@ -182,7 +197,7 @@ | |||
127 | 182 | 197 | ||
128 | 183 | def __init__(self, *results): | 198 | def __init__(self, *results): |
129 | 184 | TestResult.__init__(self) | 199 | TestResult.__init__(self) |
131 | 185 | self._results = map(ExtendedToOriginalDecorator, results) | 200 | self._results = list(map(ExtendedToOriginalDecorator, results)) |
132 | 186 | 201 | ||
133 | 187 | def _dispatch(self, message, *args, **kwargs): | 202 | def _dispatch(self, message, *args, **kwargs): |
134 | 188 | return tuple( | 203 | return tuple( |
135 | @@ -223,6 +238,13 @@ | |||
136 | 223 | def done(self): | 238 | def done(self): |
137 | 224 | return self._dispatch('done') | 239 | return self._dispatch('done') |
138 | 225 | 240 | ||
139 | 241 | def wasSuccessful(self): | ||
140 | 242 | """Was this result successful? | ||
141 | 243 | |||
142 | 244 | Only returns True if every constituent result was successful. | ||
143 | 245 | """ | ||
144 | 246 | return all(self._dispatch('wasSuccessful')) | ||
145 | 247 | |||
146 | 226 | 248 | ||
147 | 227 | class TextTestResult(TestResult): | 249 | class TextTestResult(TestResult): |
148 | 228 | """A TestResult which outputs activity to a text stream.""" | 250 | """A TestResult which outputs activity to a text stream.""" |
149 | @@ -258,6 +280,10 @@ | |||
150 | 258 | stop = self._now() | 280 | stop = self._now() |
151 | 259 | self._show_list('ERROR', self.errors) | 281 | self._show_list('ERROR', self.errors) |
152 | 260 | self._show_list('FAIL', self.failures) | 282 | self._show_list('FAIL', self.failures) |
153 | 283 | for test in self.unexpectedSuccesses: | ||
154 | 284 | self.stream.write( | ||
155 | 285 | "%sUNEXPECTED SUCCESS: %s\n%s" % ( | ||
156 | 286 | self.sep1, test.id(), self.sep2)) | ||
157 | 261 | self.stream.write("Ran %d test%s in %.3fs\n\n" % | 287 | self.stream.write("Ran %d test%s in %.3fs\n\n" % |
158 | 262 | (self.testsRun, plural, | 288 | (self.testsRun, plural, |
159 | 263 | self._delta_to_float(stop - self.__start))) | 289 | self._delta_to_float(stop - self.__start))) |
160 | @@ -267,7 +293,8 @@ | |||
161 | 267 | self.stream.write("FAILED (") | 293 | self.stream.write("FAILED (") |
162 | 268 | details = [] | 294 | details = [] |
163 | 269 | details.append("failures=%d" % ( | 295 | details.append("failures=%d" % ( |
165 | 270 | len(self.failures) + len(self.errors))) | 296 | len(self.failures) + len(self.errors) |
166 | 297 | + len(self.unexpectedSuccesses))) | ||
167 | 271 | self.stream.write(", ".join(details)) | 298 | self.stream.write(", ".join(details)) |
168 | 272 | self.stream.write(")\n") | 299 | self.stream.write(")\n") |
169 | 273 | super(TextTestResult, self).stopTestRun() | 300 | super(TextTestResult, self).stopTestRun() |
170 | @@ -363,6 +390,9 @@ | |||
171 | 363 | self._test_start = self._now() | 390 | self._test_start = self._now() |
172 | 364 | super(ThreadsafeForwardingResult, self).startTest(test) | 391 | super(ThreadsafeForwardingResult, self).startTest(test) |
173 | 365 | 392 | ||
174 | 393 | def wasSuccessful(self): | ||
175 | 394 | return self.result.wasSuccessful() | ||
176 | 395 | |||
177 | 366 | 396 | ||
178 | 367 | class ExtendedToOriginalDecorator(object): | 397 | class ExtendedToOriginalDecorator(object): |
179 | 368 | """Permit new TestResult API code to degrade gracefully with old results. | 398 | """Permit new TestResult API code to degrade gracefully with old results. |
180 | @@ -376,11 +406,13 @@ | |||
181 | 376 | 406 | ||
182 | 377 | def __init__(self, decorated): | 407 | def __init__(self, decorated): |
183 | 378 | self.decorated = decorated | 408 | self.decorated = decorated |
184 | 409 | self._was_successful = True | ||
185 | 379 | 410 | ||
186 | 380 | def __getattr__(self, name): | 411 | def __getattr__(self, name): |
187 | 381 | return getattr(self.decorated, name) | 412 | return getattr(self.decorated, name) |
188 | 382 | 413 | ||
189 | 383 | def addError(self, test, err=None, details=None): | 414 | def addError(self, test, err=None, details=None): |
190 | 415 | self._was_successful = False | ||
191 | 384 | self._check_args(err, details) | 416 | self._check_args(err, details) |
192 | 385 | if details is not None: | 417 | if details is not None: |
193 | 386 | try: | 418 | try: |
194 | @@ -405,6 +437,7 @@ | |||
195 | 405 | return addExpectedFailure(test, err) | 437 | return addExpectedFailure(test, err) |
196 | 406 | 438 | ||
197 | 407 | def addFailure(self, test, err=None, details=None): | 439 | def addFailure(self, test, err=None, details=None): |
198 | 440 | self._was_successful = False | ||
199 | 408 | self._check_args(err, details) | 441 | self._check_args(err, details) |
200 | 409 | if details is not None: | 442 | if details is not None: |
201 | 410 | try: | 443 | try: |
202 | @@ -431,6 +464,7 @@ | |||
203 | 431 | return addSkip(test, reason) | 464 | return addSkip(test, reason) |
204 | 432 | 465 | ||
205 | 433 | def addUnexpectedSuccess(self, test, details=None): | 466 | def addUnexpectedSuccess(self, test, details=None): |
206 | 467 | self._was_successful = False | ||
207 | 434 | outcome = getattr(self.decorated, 'addUnexpectedSuccess', None) | 468 | outcome = getattr(self.decorated, 'addUnexpectedSuccess', None) |
208 | 435 | if outcome is None: | 469 | if outcome is None: |
209 | 436 | try: | 470 | try: |
210 | @@ -487,6 +521,7 @@ | |||
211 | 487 | return self.decorated.startTest(test) | 521 | return self.decorated.startTest(test) |
212 | 488 | 522 | ||
213 | 489 | def startTestRun(self): | 523 | def startTestRun(self): |
214 | 524 | self._was_successful = True | ||
215 | 490 | try: | 525 | try: |
216 | 491 | return self.decorated.startTestRun() | 526 | return self.decorated.startTestRun() |
217 | 492 | except AttributeError: | 527 | except AttributeError: |
218 | @@ -517,7 +552,7 @@ | |||
219 | 517 | return method(a_datetime) | 552 | return method(a_datetime) |
220 | 518 | 553 | ||
221 | 519 | def wasSuccessful(self): | 554 | def wasSuccessful(self): |
223 | 520 | return self.decorated.wasSuccessful() | 555 | return self._was_successful |
224 | 521 | 556 | ||
225 | 522 | 557 | ||
226 | 523 | class _StringException(Exception): | 558 | class _StringException(Exception): |
227 | 524 | 559 | ||
228 | === modified file 'testtools/tests/test_compat.py' | |||
229 | --- testtools/tests/test_compat.py 2010-11-11 09:46:18 +0000 | |||
230 | +++ testtools/tests/test_compat.py 2010-11-29 00:28:14 +0000 | |||
231 | @@ -19,6 +19,7 @@ | |||
232 | 19 | ) | 19 | ) |
233 | 20 | from testtools.matchers import ( | 20 | from testtools.matchers import ( |
234 | 21 | MatchesException, | 21 | MatchesException, |
235 | 22 | Not, | ||
236 | 22 | Raises, | 23 | Raises, |
237 | 23 | ) | 24 | ) |
238 | 24 | 25 | ||
239 | @@ -196,34 +197,34 @@ | |||
240 | 196 | super(TestUnicodeOutputStream, self).setUp() | 197 | super(TestUnicodeOutputStream, self).setUp() |
241 | 197 | if sys.platform == "cli": | 198 | if sys.platform == "cli": |
242 | 198 | self.skip("IronPython shouldn't wrap streams to do encoding") | 199 | self.skip("IronPython shouldn't wrap streams to do encoding") |
244 | 199 | 200 | ||
245 | 200 | def test_no_encoding_becomes_ascii(self): | 201 | def test_no_encoding_becomes_ascii(self): |
246 | 201 | """A stream with no encoding attribute gets ascii/replace strings""" | 202 | """A stream with no encoding attribute gets ascii/replace strings""" |
247 | 202 | sout = _FakeOutputStream() | 203 | sout = _FakeOutputStream() |
248 | 203 | unicode_output_stream(sout).write(self.uni) | 204 | unicode_output_stream(sout).write(self.uni) |
249 | 204 | self.assertEqual([_b("pa???n")], sout.writelog) | 205 | self.assertEqual([_b("pa???n")], sout.writelog) |
251 | 205 | 206 | ||
252 | 206 | def test_encoding_as_none_becomes_ascii(self): | 207 | def test_encoding_as_none_becomes_ascii(self): |
253 | 207 | """A stream with encoding value of None gets ascii/replace strings""" | 208 | """A stream with encoding value of None gets ascii/replace strings""" |
254 | 208 | sout = _FakeOutputStream() | 209 | sout = _FakeOutputStream() |
255 | 209 | sout.encoding = None | 210 | sout.encoding = None |
256 | 210 | unicode_output_stream(sout).write(self.uni) | 211 | unicode_output_stream(sout).write(self.uni) |
257 | 211 | self.assertEqual([_b("pa???n")], sout.writelog) | 212 | self.assertEqual([_b("pa???n")], sout.writelog) |
259 | 212 | 213 | ||
260 | 213 | def test_bogus_encoding_becomes_ascii(self): | 214 | def test_bogus_encoding_becomes_ascii(self): |
261 | 214 | """A stream with a bogus encoding gets ascii/replace strings""" | 215 | """A stream with a bogus encoding gets ascii/replace strings""" |
262 | 215 | sout = _FakeOutputStream() | 216 | sout = _FakeOutputStream() |
263 | 216 | sout.encoding = "bogus" | 217 | sout.encoding = "bogus" |
264 | 217 | unicode_output_stream(sout).write(self.uni) | 218 | unicode_output_stream(sout).write(self.uni) |
265 | 218 | self.assertEqual([_b("pa???n")], sout.writelog) | 219 | self.assertEqual([_b("pa???n")], sout.writelog) |
267 | 219 | 220 | ||
268 | 220 | def test_partial_encoding_replace(self): | 221 | def test_partial_encoding_replace(self): |
269 | 221 | """A string which can be partly encoded correctly should be""" | 222 | """A string which can be partly encoded correctly should be""" |
270 | 222 | sout = _FakeOutputStream() | 223 | sout = _FakeOutputStream() |
271 | 223 | sout.encoding = "iso-8859-7" | 224 | sout.encoding = "iso-8859-7" |
272 | 224 | unicode_output_stream(sout).write(self.uni) | 225 | unicode_output_stream(sout).write(self.uni) |
273 | 225 | self.assertEqual([_b("pa?\xe8?n")], sout.writelog) | 226 | self.assertEqual([_b("pa?\xe8?n")], sout.writelog) |
275 | 226 | 227 | ||
276 | 227 | def test_unicode_encodings_not_wrapped(self): | 228 | def test_unicode_encodings_not_wrapped(self): |
277 | 228 | """A unicode encoding is left unwrapped as needs no error handler""" | 229 | """A unicode encoding is left unwrapped as needs no error handler""" |
278 | 229 | sout = _FakeOutputStream() | 230 | sout = _FakeOutputStream() |
279 | @@ -232,7 +233,7 @@ | |||
280 | 232 | sout = _FakeOutputStream() | 233 | sout = _FakeOutputStream() |
281 | 233 | sout.encoding = "utf-16-be" | 234 | sout.encoding = "utf-16-be" |
282 | 234 | self.assertIs(unicode_output_stream(sout), sout) | 235 | self.assertIs(unicode_output_stream(sout), sout) |
284 | 235 | 236 | ||
285 | 236 | def test_stringio(self): | 237 | def test_stringio(self): |
286 | 237 | """A StringIO object should maybe get an ascii native str type""" | 238 | """A StringIO object should maybe get an ascii native str type""" |
287 | 238 | try: | 239 | try: |
288 | @@ -246,7 +247,7 @@ | |||
289 | 246 | if newio: | 247 | if newio: |
290 | 247 | self.expectFailure("Python 3 StringIO expects text not bytes", | 248 | self.expectFailure("Python 3 StringIO expects text not bytes", |
291 | 248 | self.assertThat, lambda: soutwrapper.write(self.uni), | 249 | self.assertThat, lambda: soutwrapper.write(self.uni), |
293 | 249 | Raises(MatchesException(TypeError))) | 250 | Not(Raises(MatchesException(TypeError)))) |
294 | 250 | soutwrapper.write(self.uni) | 251 | soutwrapper.write(self.uni) |
295 | 251 | self.assertEqual("pa???n", sout.getvalue()) | 252 | self.assertEqual("pa???n", sout.getvalue()) |
296 | 252 | 253 | ||
297 | 253 | 254 | ||
298 | === modified file 'testtools/tests/test_testresult.py' | |||
299 | --- testtools/tests/test_testresult.py 2010-11-28 12:15:49 +0000 | |||
300 | +++ testtools/tests/test_testresult.py 2010-11-29 00:28:14 +0000 | |||
301 | @@ -49,8 +49,39 @@ | |||
302 | 49 | StringIO = try_imports(['StringIO.StringIO', 'io.StringIO']) | 49 | StringIO = try_imports(['StringIO.StringIO', 'io.StringIO']) |
303 | 50 | 50 | ||
304 | 51 | 51 | ||
307 | 52 | class TestTestResultContract(TestCase): | 52 | class Python26Contract(object): |
308 | 53 | """Tests for the contract of TestResults.""" | 53 | |
309 | 54 | def test_fresh_result_is_successful(self): | ||
310 | 55 | # A result is considered successful before any tests are run. | ||
311 | 56 | result = self.makeResult() | ||
312 | 57 | self.assertTrue(result.wasSuccessful()) | ||
313 | 58 | |||
314 | 59 | def test_addError_is_failure(self): | ||
315 | 60 | # addError fails the test run. | ||
316 | 61 | result = self.makeResult() | ||
317 | 62 | result.startTest(self) | ||
318 | 63 | result.addError(self, an_exc_info) | ||
319 | 64 | result.stopTest(self) | ||
320 | 65 | self.assertFalse(result.wasSuccessful()) | ||
321 | 66 | |||
322 | 67 | def test_addFailure_is_failure(self): | ||
323 | 68 | # addFailure fails the test run. | ||
324 | 69 | result = self.makeResult() | ||
325 | 70 | result.startTest(self) | ||
326 | 71 | result.addFailure(self, an_exc_info) | ||
327 | 72 | result.stopTest(self) | ||
328 | 73 | self.assertFalse(result.wasSuccessful()) | ||
329 | 74 | |||
330 | 75 | def test_addSuccess_is_success(self): | ||
331 | 76 | # addSuccess does not fail the test run. | ||
332 | 77 | result = self.makeResult() | ||
333 | 78 | result.startTest(self) | ||
334 | 79 | result.addSuccess(self) | ||
335 | 80 | result.stopTest(self) | ||
336 | 81 | self.assertTrue(result.wasSuccessful()) | ||
337 | 82 | |||
338 | 83 | |||
339 | 84 | class Python27Contract(Python26Contract): | ||
340 | 54 | 85 | ||
341 | 55 | def test_addExpectedFailure(self): | 86 | def test_addExpectedFailure(self): |
342 | 56 | # Calling addExpectedFailure(test, exc_info) completes ok. | 87 | # Calling addExpectedFailure(test, exc_info) completes ok. |
343 | @@ -58,6 +89,52 @@ | |||
344 | 58 | result.startTest(self) | 89 | result.startTest(self) |
345 | 59 | result.addExpectedFailure(self, an_exc_info) | 90 | result.addExpectedFailure(self, an_exc_info) |
346 | 60 | 91 | ||
347 | 92 | def test_addExpectedFailure_is_success(self): | ||
348 | 93 | # addExpectedFailure does not fail the test run. | ||
349 | 94 | result = self.makeResult() | ||
350 | 95 | result.startTest(self) | ||
351 | 96 | result.addExpectedFailure(self, an_exc_info) | ||
352 | 97 | result.stopTest(self) | ||
353 | 98 | self.assertTrue(result.wasSuccessful()) | ||
354 | 99 | |||
355 | 100 | def test_addSkipped(self): | ||
356 | 101 | # Calling addSkip(test, reason) completes ok. | ||
357 | 102 | result = self.makeResult() | ||
358 | 103 | result.startTest(self) | ||
359 | 104 | result.addSkip(self, _u("Skipped for some reason")) | ||
360 | 105 | |||
361 | 106 | def test_addSkip_is_success(self): | ||
362 | 107 | # addSkip does not fail the test run. | ||
363 | 108 | result = self.makeResult() | ||
364 | 109 | result.startTest(self) | ||
365 | 110 | result.addSkip(self, _u("Skipped for some reason")) | ||
366 | 111 | result.stopTest(self) | ||
367 | 112 | self.assertTrue(result.wasSuccessful()) | ||
368 | 113 | |||
369 | 114 | def test_addUnexpectedSuccess(self): | ||
370 | 115 | # Calling addUnexpectedSuccess(test) completes ok. | ||
371 | 116 | result = self.makeResult() | ||
372 | 117 | result.startTest(self) | ||
373 | 118 | result.addUnexpectedSuccess(self) | ||
374 | 119 | |||
375 | 120 | def test_addUnexpectedSuccess_was_successful(self): | ||
376 | 121 | # addUnexpectedSuccess does not fail the test run in Python 2.7. | ||
377 | 122 | result = self.makeResult() | ||
378 | 123 | result.startTest(self) | ||
379 | 124 | result.addUnexpectedSuccess(self) | ||
380 | 125 | result.stopTest(self) | ||
381 | 126 | self.assertTrue(result.wasSuccessful()) | ||
382 | 127 | |||
383 | 128 | def test_startStopTestRun(self): | ||
384 | 129 | # Calling startTestRun completes ok. | ||
385 | 130 | result = self.makeResult() | ||
386 | 131 | result.startTestRun() | ||
387 | 132 | result.stopTestRun() | ||
388 | 133 | |||
389 | 134 | |||
390 | 135 | class TestResultContract(Python27Contract): | ||
391 | 136 | """Tests for the contract of TestResults.""" | ||
392 | 137 | |||
393 | 61 | def test_addExpectedFailure_details(self): | 138 | def test_addExpectedFailure_details(self): |
394 | 62 | # Calling addExpectedFailure(test, details=xxx) completes ok. | 139 | # Calling addExpectedFailure(test, details=xxx) completes ok. |
395 | 63 | result = self.makeResult() | 140 | result = self.makeResult() |
396 | @@ -76,24 +153,12 @@ | |||
397 | 76 | result.startTest(self) | 153 | result.startTest(self) |
398 | 77 | result.addFailure(self, details={}) | 154 | result.addFailure(self, details={}) |
399 | 78 | 155 | ||
400 | 79 | def test_addSkipped(self): | ||
401 | 80 | # Calling addSkip(test, reason) completes ok. | ||
402 | 81 | result = self.makeResult() | ||
403 | 82 | result.startTest(self) | ||
404 | 83 | result.addSkip(self, _u("Skipped for some reason")) | ||
405 | 84 | |||
406 | 85 | def test_addSkipped_details(self): | 156 | def test_addSkipped_details(self): |
407 | 86 | # Calling addSkip(test, reason) completes ok. | 157 | # Calling addSkip(test, reason) completes ok. |
408 | 87 | result = self.makeResult() | 158 | result = self.makeResult() |
409 | 88 | result.startTest(self) | 159 | result.startTest(self) |
410 | 89 | result.addSkip(self, details={}) | 160 | result.addSkip(self, details={}) |
411 | 90 | 161 | ||
412 | 91 | def test_addUnexpectedSuccess(self): | ||
413 | 92 | # Calling addUnexpectedSuccess(test) completes ok. | ||
414 | 93 | result = self.makeResult() | ||
415 | 94 | result.startTest(self) | ||
416 | 95 | result.addUnexpectedSuccess(self) | ||
417 | 96 | |||
418 | 97 | def test_addUnexpectedSuccess_details(self): | 162 | def test_addUnexpectedSuccess_details(self): |
419 | 98 | # Calling addUnexpectedSuccess(test) completes ok. | 163 | # Calling addUnexpectedSuccess(test) completes ok. |
420 | 99 | result = self.makeResult() | 164 | result = self.makeResult() |
421 | @@ -106,32 +171,57 @@ | |||
422 | 106 | result.startTest(self) | 171 | result.startTest(self) |
423 | 107 | result.addSuccess(self, details={}) | 172 | result.addSuccess(self, details={}) |
424 | 108 | 173 | ||
433 | 109 | def test_startStopTestRun(self): | 174 | def test_addUnexpectedSuccess_was_successful(self): |
434 | 110 | # Calling startTestRun completes ok. | 175 | # addUnexpectedSuccess fails test run in testtools. |
435 | 111 | result = self.makeResult() | 176 | result = self.makeResult() |
436 | 112 | result.startTestRun() | 177 | result.startTest(self) |
437 | 113 | result.stopTestRun() | 178 | result.addUnexpectedSuccess(self) |
438 | 114 | 179 | result.stopTest(self) | |
439 | 115 | 180 | self.assertFalse(result.wasSuccessful()) | |
440 | 116 | class TestTestResultContract(TestTestResultContract): | 181 | |
441 | 182 | def test_startTestRun_resets_unexpected_success(self): | ||
442 | 183 | result = self.makeResult() | ||
443 | 184 | result.startTest(self) | ||
444 | 185 | result.addUnexpectedSuccess(self) | ||
445 | 186 | result.stopTest(self) | ||
446 | 187 | result.startTestRun() | ||
447 | 188 | self.assertTrue(result.wasSuccessful()) | ||
448 | 189 | |||
449 | 190 | def test_startTestRun_resets_failure(self): | ||
450 | 191 | result = self.makeResult() | ||
451 | 192 | result.startTest(self) | ||
452 | 193 | result.addFailure(self, an_exc_info) | ||
453 | 194 | result.stopTest(self) | ||
454 | 195 | result.startTestRun() | ||
455 | 196 | self.assertTrue(result.wasSuccessful()) | ||
456 | 197 | |||
457 | 198 | def test_startTestRun_resets_errors(self): | ||
458 | 199 | result = self.makeResult() | ||
459 | 200 | result.startTest(self) | ||
460 | 201 | result.addError(self, an_exc_info) | ||
461 | 202 | result.stopTest(self) | ||
462 | 203 | result.startTestRun() | ||
463 | 204 | self.assertTrue(result.wasSuccessful()) | ||
464 | 205 | |||
465 | 206 | class TestTestResultContract(TestCase, TestResultContract): | ||
466 | 117 | 207 | ||
467 | 118 | def makeResult(self): | 208 | def makeResult(self): |
468 | 119 | return TestResult() | 209 | return TestResult() |
469 | 120 | 210 | ||
470 | 121 | 211 | ||
472 | 122 | class TestMultiTestresultContract(TestTestResultContract): | 212 | class TestMultiTestResultContract(TestCase, TestResultContract): |
473 | 123 | 213 | ||
474 | 124 | def makeResult(self): | 214 | def makeResult(self): |
475 | 125 | return MultiTestResult(TestResult(), TestResult()) | 215 | return MultiTestResult(TestResult(), TestResult()) |
476 | 126 | 216 | ||
477 | 127 | 217 | ||
479 | 128 | class TestTextTestResultContract(TestTestResultContract): | 218 | class TestTextTestResultContract(TestCase, TestResultContract): |
480 | 129 | 219 | ||
481 | 130 | def makeResult(self): | 220 | def makeResult(self): |
482 | 131 | return TextTestResult(StringIO()) | 221 | return TextTestResult(StringIO()) |
483 | 132 | 222 | ||
484 | 133 | 223 | ||
486 | 134 | class TestThreadSafeForwardingResultContract(TestTestResultContract): | 224 | class TestThreadSafeForwardingResultContract(TestCase, TestResultContract): |
487 | 135 | 225 | ||
488 | 136 | def makeResult(self): | 226 | def makeResult(self): |
489 | 137 | result_semaphore = threading.Semaphore(1) | 227 | result_semaphore = threading.Semaphore(1) |
490 | @@ -139,6 +229,36 @@ | |||
491 | 139 | return ThreadsafeForwardingResult(target, result_semaphore) | 229 | return ThreadsafeForwardingResult(target, result_semaphore) |
492 | 140 | 230 | ||
493 | 141 | 231 | ||
494 | 232 | class TestExtendedTestResultContract(TestCase, TestResultContract): | ||
495 | 233 | |||
496 | 234 | def makeResult(self): | ||
497 | 235 | return ExtendedTestResult() | ||
498 | 236 | |||
499 | 237 | |||
500 | 238 | class TestPython26TestResultContract(TestCase, Python26Contract): | ||
501 | 239 | |||
502 | 240 | def makeResult(self): | ||
503 | 241 | return Python26TestResult() | ||
504 | 242 | |||
505 | 243 | |||
506 | 244 | class TestAdaptedPython26TestResultContract(TestCase, TestResultContract): | ||
507 | 245 | |||
508 | 246 | def makeResult(self): | ||
509 | 247 | return ExtendedToOriginalDecorator(Python26TestResult()) | ||
510 | 248 | |||
511 | 249 | |||
512 | 250 | class TestPython27TestResultContract(TestCase, Python27Contract): | ||
513 | 251 | |||
514 | 252 | def makeResult(self): | ||
515 | 253 | return Python27TestResult() | ||
516 | 254 | |||
517 | 255 | |||
518 | 256 | class TestAdaptedPython27TestResultContract(TestCase, TestResultContract): | ||
519 | 257 | |||
520 | 258 | def makeResult(self): | ||
521 | 259 | return ExtendedToOriginalDecorator(Python27TestResult()) | ||
522 | 260 | |||
523 | 261 | |||
524 | 142 | class TestTestResult(TestCase): | 262 | class TestTestResult(TestCase): |
525 | 143 | """Tests for `TestResult`.""" | 263 | """Tests for `TestResult`.""" |
526 | 144 | 264 | ||
527 | @@ -308,6 +428,12 @@ | |||
528 | 308 | self.fail("yo!") | 428 | self.fail("yo!") |
529 | 309 | return Test("failed") | 429 | return Test("failed") |
530 | 310 | 430 | ||
531 | 431 | def make_unexpectedly_successful_test(self): | ||
532 | 432 | class Test(TestCase): | ||
533 | 433 | def succeeded(self): | ||
534 | 434 | self.expectFailure("yo!", lambda: None) | ||
535 | 435 | return Test("succeeded") | ||
536 | 436 | |||
537 | 311 | def make_test(self): | 437 | def make_test(self): |
538 | 312 | class Test(TestCase): | 438 | class Test(TestCase): |
539 | 313 | def test(self): | 439 | def test(self): |
540 | @@ -393,9 +519,18 @@ | |||
541 | 393 | self.assertThat(self.getvalue(), | 519 | self.assertThat(self.getvalue(), |
542 | 394 | DocTestMatches("...\n\nFAILED (failures=1)\n", doctest.ELLIPSIS)) | 520 | DocTestMatches("...\n\nFAILED (failures=1)\n", doctest.ELLIPSIS)) |
543 | 395 | 521 | ||
544 | 522 | def test_stopTestRun_not_successful_unexpected_success(self): | ||
545 | 523 | test = self.make_unexpectedly_successful_test() | ||
546 | 524 | self.result.startTestRun() | ||
547 | 525 | test.run(self.result) | ||
548 | 526 | self.result.stopTestRun() | ||
549 | 527 | self.assertThat(self.getvalue(), | ||
550 | 528 | DocTestMatches("...\n\nFAILED (failures=1)\n", doctest.ELLIPSIS)) | ||
551 | 529 | |||
552 | 396 | def test_stopTestRun_shows_details(self): | 530 | def test_stopTestRun_shows_details(self): |
553 | 397 | self.result.startTestRun() | 531 | self.result.startTestRun() |
554 | 398 | self.make_erroring_test().run(self.result) | 532 | self.make_erroring_test().run(self.result) |
555 | 533 | self.make_unexpectedly_successful_test().run(self.result) | ||
556 | 399 | self.make_failing_test().run(self.result) | 534 | self.make_failing_test().run(self.result) |
557 | 400 | self.reset_output() | 535 | self.reset_output() |
558 | 401 | self.result.stopTestRun() | 536 | self.result.stopTestRun() |
559 | @@ -428,7 +563,10 @@ | |||
560 | 428 | self.fail("yo!") | 563 | self.fail("yo!") |
561 | 429 | AssertionError: yo! | 564 | AssertionError: yo! |
562 | 430 | ------------ | 565 | ------------ |
564 | 431 | ...""", doctest.ELLIPSIS)) | 566 | ====================================================================== |
565 | 567 | UNEXPECTED SUCCESS: testtools.tests.test_testresult.Test.succeeded | ||
566 | 568 | ---------------------------------------------------------------------- | ||
567 | 569 | ...""", doctest.ELLIPSIS | doctest.REPORT_NDIFF)) | ||
568 | 432 | 570 | ||
569 | 433 | 571 | ||
570 | 434 | class TestThreadSafeForwardingResult(TestWithFakeExceptions): | 572 | class TestThreadSafeForwardingResult(TestWithFakeExceptions): |
Seems to me that wasSuccessful() should be reset by startTestRun - what do you think?
Rather than a mixin, this looks like a job for testscenarios. Its fine without but clearly not as nice.