Allow passing of "disputed" FOTS tests

Bug #1151965 reported by Chris Hillery
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Zorba
Fix Released
High
Sorin Marian Nasoi

Bug Description

There are several FOTS tests whose results we disagree with. FOTS allows us to submit results with those tests marked as "disputed", which will not count as "failing". We need FOTS driver to support this.

In order to do this, we need to be able to specify the test cases, but also to specify our believed-correct expected results. FOTS driver should still fail the test if we do not match our own expected results. IMHO it should still fail even if we match the FOTS-provided expected results, so that we can ensure our list of disputed tests is always up-to-date.

It would probably be best to have a FOTSDisputedTests.xml file checked into bzr with all of this information.

Also, the Ctest integration may need to be tweaked - at least when running individual test cases, we explicitly search for 'result="pass"' in the output, so it would fail if it said eg. 'result="disputed"'.

Related branches

tags: added: testsystem
Changed in zorba:
status: Confirmed → Fix Committed
status: Fix Committed → In Progress
Changed in zorba:
status: In Progress → Fix Committed
Changed in zorba:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.