Merge lp:~jml/subunit/filter-tags into lp:subunit

Proposed by Jonathan Lange on 2012-04-20
Status: Merged
Merged at revision: 165
Proposed branch: lp:~jml/subunit/filter-tags
Merge into: lp:subunit
Prerequisite: lp:~jml/subunit/tag-collapsing-rigor
Diff against target: 798 lines (+431/-201) 3 files modified
To merge this branch: bzr merge lp:~jml/subunit/filter-tags
Reviewer Review Type Date Requested Status
Robert Collins 2012-04-20 Needs Fixing on 2012-05-02
Review via email: mp+102840@code.launchpad.net

Commit Message

Add options to filter streams by tags.

Description of the Change

This branch adds options to subunit-filter to have it filter by tags. It has three main components:

1. Separating out a _PredicateFilter from TestResultFilter.

This simplifies TestResultFilter, leaving it managing two concerns: assembling options into a predicate and transforming results. It will be much easier to separate out the 'transforming results' concern after this change.

It is likely that _PredicateFilter could also be written in terms of TestByTestResult at some point.

2. Extending the predicate function to take 'tags'.

I've tried to do this in a way that preserves backwards compatibility and allows any users of TestResultFilter who've had a predicate function that does not support tags to continue using their code unchanged. Perhaps this is not needed.

3. Cleaning up subunit-filter

It behaves a little differently to the other filter commands, and has a different set of options, but these cleanups share code where we can and make the differences -- I hope -- more obvious.

None of this works without the fixes to TagCollapsingDecorator.

jml

To post a comment you must log in.
lp:~jml/subunit/filter-tags updated on 2012-04-20
185. By Jonathan Lange on 2012-04-20

Factor a TagsMixin out of TagCollapsingDecorator

186. By Jonathan Lange on 2012-04-20

Use the TagsMixin on the predicate so local and global tags are tracked correctly.

Robert Collins (lifeless) wrote :

I don't get
353 + # XXX: ExtendedToOriginalDecorator doesn't properly wrap current_tags.
354 + # https://bugs.launchpad.net/testtools/+bug/978027

review: Needs Information
Jonathan Lange (jml) wrote :

On 26 April 2012 03:07, Robert Collins <email address hidden> wrote:
> Review: Needs Information
>
> I don't get
> 353     + # XXX: ExtendedToOriginalDecorator doesn't properly wrap current_tags.
> 354     + # https://bugs.launchpad.net/testtools/+bug/978027
>

Detritus from an earlier version. Have removed it.

jml

lp:~jml/subunit/filter-tags updated on 2012-04-26
187. By Jonathan Lange on 2012-04-26

Merge trunk

188. By Jonathan Lange on 2012-04-26

Fix up some XXX comments.

Robert Collins (lifeless) wrote :

I merged this to a worktree tree of trunk. There was a trivial conflict (mis-bound hunks in test_results.py), but tests failed:

======================================================================
FAIL: test_time_ordering_preserved (subunit.tests.test_subunit_filter.TestTestResultFilter)
subunit.tests.test_subunit_filter.TestTestResultFilter.test_time_ordering_preserved
----------------------------------------------------------------------
_StringException: Traceback (most recent call last):
  File "/home/robertc/source/unittest/subunit/working/python/subunit/tests/test_subunit_filter.py", line 240, in test_time_ordering_preserved
    ('time', date_c)], result._events)
AssertionError: Sequences differ: [('time', datetime.datetime(20... != [('time', datetime.datetime(20...

First differing element 1:
('time', datetime.datetime(2000, 1, 2, 0, 0, tzinfo=<subunit.iso8601.Utc object at 0x194af10>))
('startTest', <subunit.RemotedTestCase description='foo'>)

+ [('time', datetime.datetime(2000, 1, 1, 0, 0, tzinfo=<FixedOffset '+00:00'>)),
- [('time',
- datetime.datetime(2000, 1, 1, 0, 0, tzinfo=<subunit.iso8601.Utc object at 0x194af10>)),
- ('time',
- datetime.datetime(2000, 1, 2, 0, 0, tzinfo=<subunit.iso8601.Utc object at 0x194af10>)),
   ('startTest', <subunit.RemotedTestCase description='foo'>),
+ ('time', datetime.datetime(2000, 1, 2, 0, 0, tzinfo=<FixedOffset '+00:00'>)),
   ('addError', <subunit.RemotedTestCase description='foo'>, {}),
   ('stopTest', <subunit.RemotedTestCase description='foo'>),
+ ('time', datetime.datetime(2000, 1, 3, 0, 0, tzinfo=<FixedOffset '+00:00'>))]
- ('time',
- datetime.datetime(2000, 1, 3, 0, 0, tzinfo=<subunit.iso8601.Utc object at 0x194af10>))]

review: Needs Fixing
Robert Collins (lifeless) wrote :

The TestResultFilter :predicate: parameter docs need updating too:

        :param filter_predicate: A callable taking (test, outcome, err,
            details, tags) and returning True if the result should be passed
            through. err and details may be none if no error or extra
            metadata is available. outcome is the name of the outcome such
            as 'success' or 'failure'. tags is new in 0.0.8; 0.0.7 filters
            are still supported but should be updated to accept the tags
            parameter for efficiency.

Robert Collins (lifeless) wrote :

The FixedOffset class is also coming from iso8601.py.

Robert Collins (lifeless) wrote :

Ah, redundant time statement dropped.

Robert Collins (lifeless) wrote :

(where by dropped I mean 'I am tired and it moved').

Jonathan Lange (jml) wrote :

Thanks for landing this.

> "The FixedOffset class is also coming from iso8601.py."

I don't know what this means or is in relation to. Mentioned on IRC, but putting here as it's better suited to async comms.

Preview Diff

1=== modified file 'filters/subunit-filter'
2--- filters/subunit-filter 2011-06-30 11:58:30 +0000
3+++ filters/subunit-filter 2012-04-26 10:54:20 +0000
4@@ -36,41 +36,59 @@
5 TestProtocolClient,
6 read_test_list,
7 )
8-from subunit.test_results import TestResultFilter
9-
10-parser = OptionParser(description=__doc__)
11-parser.add_option("--error", action="store_false",
12- help="include errors", default=False, dest="error")
13-parser.add_option("-e", "--no-error", action="store_true",
14- help="exclude errors", dest="error")
15-parser.add_option("--failure", action="store_false",
16- help="include failures", default=False, dest="failure")
17-parser.add_option("-f", "--no-failure", action="store_true",
18- help="exclude failures", dest="failure")
19-parser.add_option("--passthrough", action="store_false",
20- help="Show all non subunit input.", default=False, dest="no_passthrough")
21-parser.add_option("--no-passthrough", action="store_true",
22- help="Hide all non subunit input.", default=False, dest="no_passthrough")
23-parser.add_option("-s", "--success", action="store_false",
24- help="include successes", dest="success")
25-parser.add_option("--no-success", action="store_true",
26- help="exclude successes", default=True, dest="success")
27-parser.add_option("--no-skip", action="store_true",
28- help="exclude skips", dest="skip")
29-parser.add_option("--xfail", action="store_false",
30- help="include expected falures", default=True, dest="xfail")
31-parser.add_option("--no-xfail", action="store_true",
32- help="exclude expected falures", default=True, dest="xfail")
33-parser.add_option("-m", "--with", type=str,
34- help="regexp to include (case-sensitive by default)",
35- action="append", dest="with_regexps")
36-parser.add_option("--fixup-expected-failures", type=str,
37- help="File with list of test ids that are expected to fail; on failure "
38- "their result will be changed to xfail; on success they will be "
39- "changed to error.", dest="fixup_expected_failures", action="append")
40-parser.add_option("--without", type=str,
41- help="regexp to exclude (case-sensitive by default)",
42- action="append", dest="without_regexps")
43+from subunit.filters import filter_by_result
44+from subunit.test_results import (
45+ and_predicates,
46+ _make_tag_filter,
47+ TestResultFilter,
48+ )
49+
50+
51+def make_options(description):
52+ parser = OptionParser(description=__doc__)
53+ parser.add_option("--error", action="store_false",
54+ help="include errors", default=False, dest="error")
55+ parser.add_option("-e", "--no-error", action="store_true",
56+ help="exclude errors", dest="error")
57+ parser.add_option("--failure", action="store_false",
58+ help="include failures", default=False, dest="failure")
59+ parser.add_option("-f", "--no-failure", action="store_true",
60+ help="exclude failures", dest="failure")
61+ parser.add_option("--passthrough", action="store_false",
62+ help="Show all non subunit input.", default=False, dest="no_passthrough")
63+ parser.add_option("--no-passthrough", action="store_true",
64+ help="Hide all non subunit input.", default=False, dest="no_passthrough")
65+ parser.add_option("-s", "--success", action="store_false",
66+ help="include successes", dest="success")
67+ parser.add_option("--no-success", action="store_true",
68+ help="exclude successes", default=True, dest="success")
69+ parser.add_option("--no-skip", action="store_true",
70+ help="exclude skips", dest="skip")
71+ parser.add_option("--xfail", action="store_false",
72+ help="include expected falures", default=True, dest="xfail")
73+ parser.add_option("--no-xfail", action="store_true",
74+ help="exclude expected falures", default=True, dest="xfail")
75+ parser.add_option(
76+ "--with-tag", type=str,
77+ help="include tests with these tags", action="append", dest="with_tags")
78+ parser.add_option(
79+ "--without-tag", type=str,
80+ help="exclude tests with these tags", action="append", dest="without_tags")
81+ parser.add_option("-m", "--with", type=str,
82+ help="regexp to include (case-sensitive by default)",
83+ action="append", dest="with_regexps")
84+ parser.add_option("--fixup-expected-failures", type=str,
85+ help="File with list of test ids that are expected to fail; on failure "
86+ "their result will be changed to xfail; on success they will be "
87+ "changed to error.", dest="fixup_expected_failures", action="append")
88+ parser.add_option("--without", type=str,
89+ help="regexp to exclude (case-sensitive by default)",
90+ action="append", dest="without_regexps")
91+ parser.add_option("-F", "--only-genuine-failures", action="callback",
92+ callback=only_genuine_failures_callback,
93+ help="Only pass through failures and exceptions.")
94+ return parser
95+
96
97 def only_genuine_failures_callback(option, opt, value, parser):
98 parser.rargs.insert(0, '--no-passthrough')
99@@ -78,11 +96,6 @@
100 parser.rargs.insert(0, '--no-skip')
101 parser.rargs.insert(0, '--no-success')
102
103-parser.add_option("-F", "--only-genuine-failures", action="callback",
104- callback=only_genuine_failures_callback,
105- help="Only pass through failures and exceptions.")
106-
107-(options, args) = parser.parse_args()
108
109 def _compile_re_from_list(l):
110 return re.compile("|".join(l), re.MULTILINE)
111@@ -97,7 +110,7 @@
112 with_re = with_regexps and _compile_re_from_list(with_regexps)
113 without_re = without_regexps and _compile_re_from_list(without_regexps)
114
115- def check_regexps(test, outcome, err, details):
116+ def check_regexps(test, outcome, err, details, tags):
117 """Check if this test and error match the regexp filters."""
118 test_str = str(test) + outcome + str(err) + str(details)
119 if with_re and not with_re.search(test_str):
120@@ -108,21 +121,38 @@
121 return check_regexps
122
123
124-regexp_filter = _make_regexp_filter(options.with_regexps,
125- options.without_regexps)
126-fixup_expected_failures = set()
127-for path in options.fixup_expected_failures or ():
128- fixup_expected_failures.update(read_test_list(path))
129-result = TestProtocolClient(sys.stdout)
130-result = TestResultFilter(result, filter_error=options.error,
131- filter_failure=options.failure, filter_success=options.success,
132- filter_skip=options.skip, filter_xfail=options.xfail,
133- filter_predicate=regexp_filter,
134- fixup_expected_failures=fixup_expected_failures)
135-if options.no_passthrough:
136- passthrough_stream = DiscardStream()
137-else:
138- passthrough_stream = None
139-test = ProtocolTestCase(sys.stdin, passthrough=passthrough_stream)
140-test.run(result)
141-sys.exit(0)
142+def _make_result(output, options, predicate):
143+ """Make the result that we'll send the test outcomes to."""
144+ fixup_expected_failures = set()
145+ for path in options.fixup_expected_failures or ():
146+ fixup_expected_failures.update(read_test_list(path))
147+ return TestResultFilter(
148+ TestProtocolClient(output),
149+ filter_error=options.error,
150+ filter_failure=options.failure,
151+ filter_success=options.success,
152+ filter_skip=options.skip,
153+ filter_xfail=options.xfail,
154+ filter_predicate=predicate,
155+ fixup_expected_failures=fixup_expected_failures)
156+
157+
158+def main():
159+ parser = make_options(__doc__)
160+ (options, args) = parser.parse_args()
161+
162+ regexp_filter = _make_regexp_filter(
163+ options.with_regexps, options.without_regexps)
164+ tag_filter = _make_tag_filter(options.with_tags, options.without_tags)
165+ filter_predicate = and_predicates([regexp_filter, tag_filter])
166+
167+ filter_by_result(
168+ lambda output_to: _make_result(sys.stdout, options, filter_predicate),
169+ output_path=None,
170+ passthrough=(not options.no_passthrough),
171+ forward=False)
172+ sys.exit(0)
173+
174+
175+if __name__ == '__main__':
176+ main()
177
178=== modified file 'python/subunit/test_results.py'
179--- python/subunit/test_results.py 2012-04-20 11:32:41 +0000
180+++ python/subunit/test_results.py 2012-04-26 10:54:20 +0000
181@@ -20,6 +20,7 @@
182 import datetime
183
184 import testtools
185+from testtools.compat import all
186 from testtools.content import (
187 text_content,
188 TracebackContent,
189@@ -39,6 +40,9 @@
190 or features by degrading them.
191 """
192
193+ # XXX: Since lp:testtools r250, this is in testtools. Once it's released,
194+ # we should gut this and just use that.
195+
196 def __init__(self, decorated):
197 """Create a TestResultDecorator forwarding to decorated."""
198 # Make every decorator degrade gracefully.
199@@ -205,48 +209,45 @@
200 return self.decorated.time(a_datetime)
201
202
203-class TagCollapsingDecorator(HookedTestResultDecorator):
204- """Collapses many 'tags' calls into one where possible."""
205+class TagsMixin(object):
206
207- def __init__(self, result):
208- super(TagCollapsingDecorator, self).__init__(result)
209+ def __init__(self):
210 self._clear_tags()
211
212 def _clear_tags(self):
213 self._global_tags = set(), set()
214- self._current_test_tags = None
215-
216- def _get_current_tags(self):
217- if self._current_test_tags:
218- return self._current_test_tags
219+ self._test_tags = None
220+
221+ def _get_active_tags(self):
222+ global_new, global_gone = self._global_tags
223+ if self._test_tags is None:
224+ return set(global_new)
225+ test_new, test_gone = self._test_tags
226+ return global_new.difference(test_gone).union(test_new)
227+
228+ def _get_current_scope(self):
229+ if self._test_tags:
230+ return self._test_tags
231 return self._global_tags
232
233- def startTestRun(self):
234- super(TagCollapsingDecorator, self).startTestRun()
235- self._clear_tags()
236-
237- def startTest(self, test):
238- """Start a test.
239-
240- Not directly passed to the client, but used for handling of tags
241- correctly.
242- """
243- super(TagCollapsingDecorator, self).startTest(test)
244- self._current_test_tags = set(), set()
245-
246- def stopTest(self, test):
247- super(TagCollapsingDecorator, self).stopTest(test)
248- self._current_test_tags = None
249-
250- def _before_event(self):
251- new_tags, gone_tags = self._get_current_tags()
252+ def _flush_current_scope(self, tag_receiver):
253+ new_tags, gone_tags = self._get_current_scope()
254 if new_tags or gone_tags:
255- self.decorated.tags(new_tags, gone_tags)
256- if self._current_test_tags:
257- self._current_test_tags = set(), set()
258+ tag_receiver.tags(new_tags, gone_tags)
259+ if self._test_tags:
260+ self._test_tags = set(), set()
261 else:
262 self._global_tags = set(), set()
263
264+ def startTestRun(self):
265+ self._clear_tags()
266+
267+ def startTest(self, test):
268+ self._test_tags = set(), set()
269+
270+ def stopTest(self, test):
271+ self._test_tags = None
272+
273 def tags(self, new_tags, gone_tags):
274 """Handle tag instructions.
275
276@@ -256,13 +257,27 @@
277 :param new_tags: Tags to add,
278 :param gone_tags: Tags to remove.
279 """
280- current_new_tags, current_gone_tags = self._get_current_tags()
281+ current_new_tags, current_gone_tags = self._get_current_scope()
282 current_new_tags.update(new_tags)
283 current_new_tags.difference_update(gone_tags)
284 current_gone_tags.update(gone_tags)
285 current_gone_tags.difference_update(new_tags)
286
287
288+class TagCollapsingDecorator(HookedTestResultDecorator, TagsMixin):
289+ """Collapses many 'tags' calls into one where possible."""
290+
291+ def __init__(self, result):
292+ super(TagCollapsingDecorator, self).__init__(result)
293+ self._clear_tags()
294+
295+ def _before_event(self):
296+ self._flush_current_scope(self.decorated)
297+
298+ def tags(self, new_tags, gone_tags):
299+ TagsMixin.tags(self, new_tags, gone_tags)
300+
301+
302 class TimeCollapsingDecorator(HookedTestResultDecorator):
303 """Only pass on the first and last of a consecutive sequence of times."""
304
305@@ -288,12 +303,132 @@
306 self._last_received_time = a_time
307
308
309-def all_true(bools):
310- """Return True if all of 'bools' are True. False otherwise."""
311- for b in bools:
312- if not b:
313- return False
314- return True
315+def and_predicates(predicates):
316+ """Return a predicate that is true iff all predicates are true."""
317+ # XXX: Should probably be in testtools to be better used by matchers. jml
318+ return lambda *args, **kwargs: all(p(*args, **kwargs) for p in predicates)
319+
320+
321+def _make_tag_filter(with_tags, without_tags):
322+ """Make a callback that checks tests against tags."""
323+
324+ with_tags = with_tags and set(with_tags) or None
325+ without_tags = without_tags and set(without_tags) or None
326+
327+ def check_tags(test, outcome, err, details, tags):
328+ if with_tags and not with_tags <= tags:
329+ return False
330+ if without_tags and bool(without_tags & tags):
331+ return False
332+ return True
333+
334+ return check_tags
335+
336+
337+class _PredicateFilter(TestResultDecorator, TagsMixin):
338+
339+ def __init__(self, result, predicate):
340+ super(_PredicateFilter, self).__init__(result)
341+ self._clear_tags()
342+ self.decorated = TimeCollapsingDecorator(
343+ TagCollapsingDecorator(self.decorated))
344+ self._predicate = predicate
345+ # The current test (for filtering tags)
346+ self._current_test = None
347+ # Has the current test been filtered (for outputting test tags)
348+ self._current_test_filtered = None
349+ # Calls to this result that we don't know whether to forward on yet.
350+ self._buffered_calls = []
351+
352+ def filter_predicate(self, test, outcome, error, details):
353+ return self._predicate(
354+ test, outcome, error, details, self._get_active_tags())
355+
356+ def addError(self, test, err=None, details=None):
357+ if (self.filter_predicate(test, 'error', err, details)):
358+ self._buffered_calls.append(
359+ ('addError', [test, err], {'details': details}))
360+ else:
361+ self._filtered()
362+
363+ def addFailure(self, test, err=None, details=None):
364+ if (self.filter_predicate(test, 'failure', err, details)):
365+ self._buffered_calls.append(
366+ ('addFailure', [test, err], {'details': details}))
367+ else:
368+ self._filtered()
369+
370+ def addSkip(self, test, reason=None, details=None):
371+ if (self.filter_predicate(test, 'skip', reason, details)):
372+ self._buffered_calls.append(
373+ ('addSkip', [test, reason], {'details': details}))
374+ else:
375+ self._filtered()
376+
377+ def addExpectedFailure(self, test, err=None, details=None):
378+ if self.filter_predicate(test, 'expectedfailure', err, details):
379+ self._buffered_calls.append(
380+ ('addExpectedFailure', [test, err], {'details': details}))
381+ else:
382+ self._filtered()
383+
384+ def addUnexpectedSuccess(self, test, details=None):
385+ self._buffered_calls.append(
386+ ('addUnexpectedSuccess', [test], {'details': details}))
387+
388+ def addSuccess(self, test, details=None):
389+ if (self.filter_predicate(test, 'success', None, details)):
390+ self._buffered_calls.append(
391+ ('addSuccess', [test], {'details': details}))
392+ else:
393+ self._filtered()
394+
395+ def _filtered(self):
396+ self._current_test_filtered = True
397+
398+ def startTest(self, test):
399+ """Start a test.
400+
401+ Not directly passed to the client, but used for handling of tags
402+ correctly.
403+ """
404+ TagsMixin.startTest(self, test)
405+ self._current_test = test
406+ self._current_test_filtered = False
407+ self._buffered_calls.append(('startTest', [test], {}))
408+
409+ def stopTest(self, test):
410+ """Stop a test.
411+
412+ Not directly passed to the client, but used for handling of tags
413+ correctly.
414+ """
415+ if not self._current_test_filtered:
416+ for method, args, kwargs in self._buffered_calls:
417+ getattr(self.decorated, method)(*args, **kwargs)
418+ self.decorated.stopTest(test)
419+ self._current_test = None
420+ self._current_test_filtered = None
421+ self._buffered_calls = []
422+ TagsMixin.stopTest(self, test)
423+
424+ def tags(self, new_tags, gone_tags):
425+ TagsMixin.tags(self, new_tags, gone_tags)
426+ if self._current_test is not None:
427+ self._buffered_calls.append(('tags', [new_tags, gone_tags], {}))
428+ else:
429+ return super(_PredicateFilter, self).tags(new_tags, gone_tags)
430+
431+ def time(self, a_time):
432+ if self._current_test is not None:
433+ self._buffered_calls.append(('time', [a_time], {}))
434+ else:
435+ return self.decorated.time(a_time)
436+
437+ def id_to_orig_id(self, id):
438+ if id.startswith("subunit.RemotedTestCase."):
439+ return id[len("subunit.RemotedTestCase."):]
440+ return id
441
442
443 class TestResultFilter(TestResultDecorator):
444@@ -326,129 +461,62 @@
445 :param fixup_expected_failures: Set of test ids to consider known
446 failing.
447 """
448- super(TestResultFilter, self).__init__(result)
449- self.decorated = TimeCollapsingDecorator(
450- TagCollapsingDecorator(self.decorated))
451 predicates = []
452 if filter_error:
453- predicates.append(lambda t, outcome, e, d: outcome != 'error')
454+ predicates.append(
455+ lambda t, outcome, e, d, tags: outcome != 'error')
456 if filter_failure:
457- predicates.append(lambda t, outcome, e, d: outcome != 'failure')
458+ predicates.append(
459+ lambda t, outcome, e, d, tags: outcome != 'failure')
460 if filter_success:
461- predicates.append(lambda t, outcome, e, d: outcome != 'success')
462+ predicates.append(
463+ lambda t, outcome, e, d, tags: outcome != 'success')
464 if filter_skip:
465- predicates.append(lambda t, outcome, e, d: outcome != 'skip')
466+ predicates.append(
467+ lambda t, outcome, e, d, tags: outcome != 'skip')
468 if filter_xfail:
469- predicates.append(lambda t, outcome, e, d: outcome != 'expectedfailure')
470+ predicates.append(
471+ lambda t, outcome, e, d, tags: outcome != 'expectedfailure')
472 if filter_predicate is not None:
473- predicates.append(filter_predicate)
474- self.filter_predicate = (
475- lambda test, outcome, err, details:
476- all_true(p(test, outcome, err, details) for p in predicates))
477- # The current test (for filtering tags)
478- self._current_test = None
479- # Has the current test been filtered (for outputting test tags)
480- self._current_test_filtered = None
481- # Calls to this result that we don't know whether to forward on yet.
482- self._buffered_calls = []
483+ def compat(test, outcome, error, details, tags):
484+ # 0.0.7 and earlier did not support the 'tags' parameter.
485+ try:
486+ return filter_predicate(
487+ test, outcome, error, details, tags)
488+ except TypeError:
489+ return filter_predicate(test, outcome, error, details)
490+ predicates.append(compat)
491+ predicate = and_predicates(predicates)
492+ super(TestResultFilter, self).__init__(
493+ _PredicateFilter(result, predicate))
494 if fixup_expected_failures is None:
495 self._fixup_expected_failures = frozenset()
496 else:
497 self._fixup_expected_failures = fixup_expected_failures
498
499 def addError(self, test, err=None, details=None):
500- if (self.filter_predicate(test, 'error', err, details)):
501- if self._failure_expected(test):
502- self._buffered_calls.append(
503- ('addExpectedFailure', [test, err], {'details': details}))
504- else:
505- self._buffered_calls.append(
506- ('addError', [test, err], {'details': details}))
507+ if self._failure_expected(test):
508+ self.addExpectedFailure(test, err=err, details=details)
509 else:
510- self._filtered()
511+ super(TestResultFilter, self).addError(
512+ test, err=err, details=details)
513
514 def addFailure(self, test, err=None, details=None):
515- if (self.filter_predicate(test, 'failure', err, details)):
516- if self._failure_expected(test):
517- self._buffered_calls.append(
518- ('addExpectedFailure', [test, err], {'details': details}))
519- else:
520- self._buffered_calls.append(
521- ('addFailure', [test, err], {'details': details}))
522- else:
523- self._filtered()
524-
525- def addSkip(self, test, reason=None, details=None):
526- if (self.filter_predicate(test, 'skip', reason, details)):
527- self._buffered_calls.append(
528- ('addSkip', [test, reason], {'details': details}))
529- else:
530- self._filtered()
531+ if self._failure_expected(test):
532+ self.addExpectedFailure(test, err=err, details=details)
533+ else:
534+ super(TestResultFilter, self).addFailure(
535+ test, err=err, details=details)
536
537 def addSuccess(self, test, details=None):
538- if (self.filter_predicate(test, 'success', None, details)):
539- if self._failure_expected(test):
540- self._buffered_calls.append(
541- ('addUnexpectedSuccess', [test], {'details': details}))
542- else:
543- self._buffered_calls.append(
544- ('addSuccess', [test], {'details': details}))
545- else:
546- self._filtered()
547-
548- def addExpectedFailure(self, test, err=None, details=None):
549- if self.filter_predicate(test, 'expectedfailure', err, details):
550- self._buffered_calls.append(
551- ('addExpectedFailure', [test, err], {'details': details}))
552- else:
553- self._filtered()
554-
555- def addUnexpectedSuccess(self, test, details=None):
556- self._buffered_calls.append(
557- ('addUnexpectedSuccess', [test], {'details': details}))
558-
559- def _filtered(self):
560- self._current_test_filtered = True
561+ if self._failure_expected(test):
562+ self.addUnexpectedSuccess(test, details=details)
563+ else:
564+ super(TestResultFilter, self).addSuccess(test, details=details)
565
566 def _failure_expected(self, test):
567 return (test.id() in self._fixup_expected_failures)
568
569- def startTest(self, test):
570- """Start a test.
571-
572- Not directly passed to the client, but used for handling of tags
573- correctly.
574- """
575- self._current_test = test
576- self._current_test_filtered = False
577- self._buffered_calls.append(('startTest', [test], {}))
578-
579- def stopTest(self, test):
580- """Stop a test.
581-
582- Not directly passed to the client, but used for handling of tags
583- correctly.
584- """
585- if not self._current_test_filtered:
586- # Tags to output for this test.
587- for method, args, kwargs in self._buffered_calls:
588- getattr(self.decorated, method)(*args, **kwargs)
589- self.decorated.stopTest(test)
590- self._current_test = None
591- self._current_test_filtered = None
592- self._buffered_calls = []
593-
594- def time(self, a_time):
595- if self._current_test is not None:
596- self._buffered_calls.append(('time', [a_time], {}))
597- else:
598- return self.decorated.time(a_time)
599-
600- def id_to_orig_id(self, id):
601- if id.startswith("subunit.RemotedTestCase."):
602- return id[len("subunit.RemotedTestCase."):]
603- return id
604-
605
606 class TestIdPrintingResult(testtools.TestResult):
607
608@@ -513,7 +581,8 @@
609 class TestByTestResult(testtools.TestResult):
610 """Call something every time a test completes."""
611
612- # XXX: Arguably belongs in testtools.
613+# XXX: In testtools since lp:testtools r249. Once that's released, just
614+# import that.
615
616 def __init__(self, on_test):
617 """Construct a ``TestByTestResult``.
618
619=== modified file 'python/subunit/tests/test_subunit_filter.py'
620--- python/subunit/tests/test_subunit_filter.py 2011-05-09 21:00:42 +0000
621+++ python/subunit/tests/test_subunit_filter.py 2012-04-26 10:54:20 +0000
622@@ -17,15 +17,18 @@
623 """Tests for subunit.TestResultFilter."""
624
625 from datetime import datetime
626+import os
627+import subprocess
628+import sys
629 from subunit import iso8601
630 import unittest
631
632 from testtools import TestCase
633-from testtools.compat import _b, BytesIO, StringIO
634+from testtools.compat import _b, BytesIO
635 from testtools.testresult.doubles import ExtendedTestResult
636
637 import subunit
638-from subunit.test_results import TestResultFilter
639+from subunit.test_results import _make_tag_filter, TestResultFilter
640
641
642 class TestTestResultFilter(TestCase):
643@@ -77,6 +80,40 @@
644 filtered_result.failures])
645 self.assertEqual(4, filtered_result.testsRun)
646
647+ def test_tag_filter(self):
648+ tag_filter = _make_tag_filter(['global'], ['local'])
649+ result = ExtendedTestResult()
650+ result_filter = TestResultFilter(
651+ result, filter_success=False, filter_predicate=tag_filter)
652+ self.run_tests(result_filter)
653+ tests_included = [
654+ event[1] for event in result._events if event[0] == 'startTest']
655+ tests_expected = map(
656+ subunit.RemotedTestCase,
657+ ['passed', 'error', 'skipped', 'todo'])
658+ self.assertEquals(tests_expected, tests_included)
659+
660+ def test_tags_tracked_correctly(self):
661+ tag_filter = _make_tag_filter(['a'], [])
662+ result = ExtendedTestResult()
663+ result_filter = TestResultFilter(
664+ result, filter_success=False, filter_predicate=tag_filter)
665+ input_stream = (
666+ "test: foo\n"
667+ "tags: a\n"
668+ "successful: foo\n"
669+ "test: bar\n"
670+ "successful: bar\n")
671+ self.run_tests(result_filter, input_stream)
672+ foo = subunit.RemotedTestCase('foo')
673+ self.assertEquals(
674+ [('startTest', foo),
675+ ('tags', set(['a']), set()),
676+ ('addSuccess', foo),
677+ ('stopTest', foo),
678+ ],
679+ result._events)
680+
681 def test_exclude_errors(self):
682 filtered_result = unittest.TestResult()
683 result_filter = TestResultFilter(filtered_result, filter_error=True)
684@@ -151,6 +188,8 @@
685
686 def test_filter_predicate(self):
687 """You can filter by predicate callbacks"""
688+ # 0.0.7 and earlier did not support the 'tags' parameter, so we need
689+ # to test that we still support behaviour without it.
690 filtered_result = unittest.TestResult()
691 def filter_cb(test, outcome, err, details):
692 return outcome == 'success'
693@@ -161,6 +200,18 @@
694 # Only success should pass
695 self.assertEqual(1, filtered_result.testsRun)
696
697+ def test_filter_predicate_with_tags(self):
698+ """You can filter by predicate callbacks that accept tags"""
699+ filtered_result = unittest.TestResult()
700+ def filter_cb(test, outcome, err, details, tags):
701+ return outcome == 'success'
702+ result_filter = TestResultFilter(filtered_result,
703+ filter_predicate=filter_cb,
704+ filter_success=False)
705+ self.run_tests(result_filter)
706+ # Only success should pass
707+ self.assertEqual(1, filtered_result.testsRun)
708+
709 def test_time_ordering_preserved(self):
710 # Passing a subunit stream through TestResultFilter preserves the
711 # relative ordering of 'time' directives and any other subunit
712@@ -202,6 +253,86 @@
713 ('stopTest', foo), ], result._events)
714
715
716+class TestFilterCommand(TestCase):
717+
718+ example_subunit_stream = _b("""\
719+tags: global
720+test passed
721+success passed
722+test failed
723+tags: local
724+failure failed
725+test error
726+error error [
727+error details
728+]
729+test skipped
730+skip skipped
731+test todo
732+xfail todo
733+""")
734+
735+ def run_command(self, args, stream):
736+ root = os.path.dirname(
737+ os.path.dirname(os.path.dirname(os.path.dirname(__file__))))
738+ script_path = os.path.join(root, 'filters', 'subunit-filter')
739+ command = [sys.executable, script_path] + list(args)
740+ ps = subprocess.Popen(
741+ command, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
742+ stderr=subprocess.PIPE)
743+ out, err = ps.communicate(stream)
744+ if ps.returncode != 0:
745+ raise RuntimeError("%s failed: %s" % (command, err))
746+ return out
747+
748+ def to_events(self, stream):
749+ test = subunit.ProtocolTestCase(BytesIO(stream))
750+ result = ExtendedTestResult()
751+ test.run(result)
752+ return result._events
753+
754+ def test_default(self):
755+ output = self.run_command([], (
756+ "test: foo\n"
757+ "skip: foo\n"
758+ ))
759+ events = self.to_events(output)
760+ foo = subunit.RemotedTestCase('foo')
761+ self.assertEqual(
762+ [('startTest', foo),
763+ ('addSkip', foo, {}),
764+ ('stopTest', foo)],
765+ events)
766+
767+ def test_tags(self):
768+ output = self.run_command(['-s', '--with-tag', 'a'], (
769+ "tags: a\n"
770+ "test: foo\n"
771+ "success: foo\n"
772+ "tags: -a\n"
773+ "test: bar\n"
774+ "success: bar\n"
775+ "test: baz\n"
776+ "tags: a\n"
777+ "success: baz\n"
778+ ))
779+ events = self.to_events(output)
780+ foo = subunit.RemotedTestCase('foo')
781+ baz = subunit.RemotedTestCase('baz')
782+ self.assertEqual(
783+ [('tags', set(['a']), set()),
784+ ('startTest', foo),
785+ ('addSuccess', foo),
786+ ('stopTest', foo),
787+ ('tags', set(), set(['a'])),
788+ ('startTest', baz),
789+ ('tags', set(['a']), set()),
790+ ('addSuccess', baz),
791+ ('stopTest', baz),
792+ ],
793+ events)
794+
795+
796 def test_suite():
797 loader = subunit.tests.TestUtil.TestLoader()
798 result = loader.loadTestsFromName(__name__)

Subscribers

People subscribed via source and target branches