Allow passing of "disputed" FOTS tests
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Zorba |
Fix Released
|
High
|
Sorin Marian Nasoi |
Bug Description
There are several FOTS tests whose results we disagree with. FOTS allows us to submit results with those tests marked as "disputed", which will not count as "failing". We need FOTS driver to support this.
In order to do this, we need to be able to specify the test cases, but also to specify our believed-correct expected results. FOTS driver should still fail the test if we do not match our own expected results. IMHO it should still fail even if we match the FOTS-provided expected results, so that we can ensure our list of disputed tests is always up-to-date.
It would probably be best to have a FOTSDisputedTes
Also, the Ctest integration may need to be tweaked - at least when running individual test cases, we explicitly search for 'result="pass"' in the output, so it would fail if it said eg. 'result=
Related branches
- Chris Hillery: Approve
- Sorin Marian Nasoi: Approve
-
Diff: 2325 lines (+653/-751)8 files modifiedtest/fots/CMakeLists.txt (+50/-30)
test/fots/ImportFOTS.cmake (+2/-0)
test/fots_driver/cli.xq (+29/-77)
test/fots_driver/environment.xq (+14/-53)
test/fots_driver/evaluate.xq (+7/-10)
test/fots_driver/feedback.xq (+104/-63)
test/fots_driver/fots-driver.xq (+404/-353)
test/fots_driver/reporting.xq (+43/-165)
tags: | added: testsystem |
Changed in zorba: | |
status: | Confirmed → Fix Committed |
status: | Fix Committed → In Progress |
Changed in zorba: | |
status: | In Progress → Fix Committed |
Changed in zorba: | |
status: | Fix Committed → Fix Released |