Merge lp:~zorba-coders/zorba/fix_bug_1134403 into lp:zorba

Proposed by Sorin Marian Nasoi on 2013-03-18
Status: Merged
Approved by: Chris Hillery on 2013-03-20
Approved revision: 11291
Merged at revision: 11294
Proposed branch: lp:~zorba-coders/zorba/fix_bug_1134403
Merge into: lp:zorba
Diff against target: 209 lines (+21/-99)
3 files modified
test/fots/ImportFOTS.cmake (+2/-2)
test/fots_driver/feedback.xq (+15/-89)
test/fots_driver/fots-driver.xq (+4/-8)
To merge this branch: bzr merge lp:~zorba-coders/zorba/fix_bug_1134403
Reviewer Review Type Date Requested Status
Chris Hillery 2013-03-18 Approve on 2013-03-20
Sorin Marian Nasoi Approve on 2013-03-18
Review via email: mp+153784@code.launchpad.net

Commit Message

- feedback:pass, feedback:not-run, feedback:not-applicable return *only* non-verbosed output
- the tests added in Ctest for the FOTS/test-sets are always run with verbosed output. This way if a test case failed or passed but is marked as expected failure one can always see what happened

Description of the Change

- feedback:pass, feedback:not-run, feedback:not-applicable return *only* non-verbosed output
- the tests added in Ctest for the FOTS/test-sets are always run with verbosed output. This way if a test case failed or passed but is marked as expected failure one can always see what happened

To post a comment you must log in.
review: Approve
Zorba Build Bot (zorba-buildbot) wrote :

Validation queue job fix_bug_1134403-2013-03-18T13-00-48.859Z is finished. The final status was:

All tests succeeded!

Zorba Build Bot (zorba-buildbot) wrote :

Voting does not meet specified criteria. Required: Approve > 1, Disapprove < 1, Needs Fixing < 1, Pending < 1, Needs Information < 1, Resubmit < 1. Got: 1 Approve, 1 Pending.

Chris Hillery (ceejatec) wrote :

The ctest tests should not be run with verbose enabled.

review: Needs Fixing

> The ctest tests should not be run with verbose enabled.
Why is that? The only time verbosed information is outputed is:
- when a test case failed
- when a test case passed because it was marked as expected failure, meaning it in fact failed

> The ctest tests should not be run with verbose enabled.
Please take a look at a CTest report for a test set that has failing tests cases: let's say "fn-format-date" and let me know what problems do you see there.

Chris Hillery (ceejatec) wrote :

What I wanted:
 - verbose=false => passing tests shown condensed, failing tests shown in full
 - verbose=true => all tests shown in full
 - ctest in "fots-activate-sets" mode uses verbose=false, and in "fots-activate-cases" mode uses verbose=true

What you did:
 - verbose=false => all tests shown condensed
 - verbose=true => passing tests shown condensed, failing tests shown in full
 - ctest always uses verbose=true

The result for "fots-activate-sets" mode is the same as what I wanted, which is good since that's the most common mode (including what the RQ does). That was the key thing I needed to get fixed.

The result for "fots-activate-cases" mode is not what I intended, because for tests that are passing there's no way at all to get the complete output report. However, honestly I'm not sure that I care all that much, so I'll go ahead and approve this as it stands. If anyone complains later (which they probably won't) we can revisit that decision. Maybe verbose could just be a bit different for run-test-case mode, but as I said, for now this is good enough. Thanks.

review: Approve
Zorba Build Bot (zorba-buildbot) wrote :

Validation queue job fix_bug_1134403-2013-03-20T06-40-44.146Z is finished. The final status was:

All tests succeeded!

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'test/fots/ImportFOTS.cmake'
2--- test/fots/ImportFOTS.cmake 2013-03-17 15:20:38 +0000
3+++ test/fots/ImportFOTS.cmake 2013-03-18 12:58:52 +0000
4@@ -23,7 +23,7 @@
5
6 # Change this to modify which elements in FOTS driver results are output
7 # as CDATA
8-SET (FOTS_CDATA_ELEMENTS "fots:query test assert-xml")
9+SET (FOTS_CDATA_ELEMENTS "fots:query fots:result test assert-xml")
10
11 # Download and unpack pre-created archive
12 SET (_outdir "${BUILDDIR}/test/fots")
13@@ -76,7 +76,7 @@
14 "-e fotsPath:=${_outdir}/2011/QT3-test-suite/catalog.xml "
15 "-e mode:=run-test-set -e testSetName:=${_testset} "
16 "-e expectedFailuresPath:=${BUILDDIR}/FOTSExpectedFailures.xml "
17- "-e verbose:=false "
18+ "-e verbose:=true "
19 "--disable-http-resolution --indent "
20 "-z \"cdata-section-elements=${FOTS_CDATA_ELEMENTS}\")\n"
21 "ZORBA_SET_TEST_PROPERTY (FOTS/${_testset} "
22
23=== modified file 'test/fots_driver/feedback.xq'
24--- test/fots_driver/feedback.xq 2013-02-26 06:21:18 +0000
25+++ test/fots_driver/feedback.xq 2013-03-18 12:58:52 +0000
26@@ -79,11 +79,7 @@
27 $env,
28 $verbose)
29 else feedback:pass( $case,
30- $result,
31- $zorbaQuery,
32- $env,
33- $duration,
34- $verbose)
35+ $result)
36 };
37
38
39@@ -136,51 +132,19 @@
40 : Gives feedback on a test case run with success.
41 :
42 : @param $case test case.
43- : @return the test case after certain information was added.
44+ : @param $result test result of running the test case.
45+ : @return info about test case that passed.
46 :)
47-declare %private %ann:sequential function feedback:pass(
48+declare %private function feedback:pass(
49 $case as element(fots:test-case),
50- $result as item()*,
51- $zorbaQuery as xs:string,
52- $env as element(fots:environment)?,
53- $duration as xs:dayTimeDuration,
54- $verbose as xs:boolean
55+ $result as item()*
56 ) as element(fots:test-case)?
57 {
58 let $status := if(exists($result/fots:errors))
59 then 'wrongError'
60 else 'pass'
61- return
62- if ($verbose)
63- then
64- {
65- {
66- (insert node attribute result{$status} as last into $case,
67- if (exists($result/fots:errors))
68- then
69- insert node
70- attribute comment{$result/fots:errors}
71- as last into $case
72- else (),
73- insert node attribute executionTime{$duration} as last into $case,
74- insert node
75- <fots:info>
76- {$env}
77- <fots:query>{$zorbaQuery}</fots:query>
78- {$result/fots:expected-result,
79- $result/fots:result,
80- $result/fots:errors}
81- </fots:info>
82- as last into $case,
83- delete node $case/fots:description,
84- delete node $case/fots:created,
85- delete node $case/fots:result);
86-
87- $case
88- }
89- }
90- else <fots:test-case name="{data($case/@name)}"
91- result="{$status}"/>
92+ return
93+ <fots:test-case name="{data($case/@name)}" result="{$status}"/>
94 };
95
96
97@@ -249,32 +213,14 @@
98 : @param $case test case.
99 : @return the test case.
100 :)
101-declare %ann:sequential function feedback:not-run(
102- $case as element(fots:test-case),
103- $verbose as xs:boolean
104+declare function feedback:not-run(
105+ $case as element(fots:test-case)
106 ) as element(fots:test-case)?
107 {
108 trace(data($case/@name), "processing test case :");
109 trace("Above test case was not run.","");
110
111- let $status := 'notRun'
112- return
113- if ($verbose)
114- then
115- {
116- {
117- (insert node attribute result {$status} as last into $case,
118- delete node $case/fots:description,
119- delete node $case/fots:created);
120-
121- $case
122- }
123- }
124- else
125- {
126- <fots:test-case name="{$case/@name}"
127- result="{$status}" />
128- }
129+ <fots:test-case name="{$case/@name}" result="notRun" />
130 };
131
132
133@@ -285,35 +231,15 @@
134 : @param $dependencyError test error returned by the dependency checking.
135 : @return the test case.
136 :)
137-declare %ann:sequential function feedback:not-applicable(
138+declare function feedback:not-applicable(
139 $case as element(fots:test-case),
140- $env as element(fots:environment)?,
141- $dependencyError as xs:string,
142- $verbose as xs:boolean
143+ $dependencyError as xs:string
144 ) as element(fots:test-case)?
145 {
146 trace(data($case/@name), "processing test case :");
147 trace($dependencyError, "Dependency error :");
148
149- let $status := 'n/a'
150- return
151- if ($verbose)
152- then
153- {
154- {
155- (insert node attribute result{$status} as last into $case,
156- insert node attribute comment{$dependencyError} as last into $case,
157- insert node
158- <fots:info>{$env}</fots:info>
159- as last into $case,
160- delete node $case/fots:description,
161- delete node $case/fots:created);
162-
163- $case
164- }
165- }
166- else
167- <fots:test-case name="{data($case/@name)}"
168- result="{$status}"
169- comment="{$dependencyError}" />
170+ <fots:test-case name="{data($case/@name)}"
171+ result="n/a"
172+ comment="{$dependencyError}" />
173 };
174
175=== modified file 'test/fots_driver/fots-driver.xq'
176--- test/fots_driver/fots-driver.xq 2013-03-13 01:13:07 +0000
177+++ test/fots_driver/fots-driver.xq 2013-03-18 12:58:52 +0000
178@@ -667,12 +667,10 @@
179 return
180 if ($isExcepted)
181 then
182- feedback:not-run($testCase, $verbose)
183+ feedback:not-run($testCase)
184 else
185 feedback:not-applicable($testCase,
186- $envTestSet,
187- string-join($depMet,''),
188- $verbose)
189+ string-join($depMet,''))
190 }
191 </fots:test-set>
192 }
193@@ -711,14 +709,12 @@
194 $verbose,
195 fn:false())
196 else
197- feedback:not-run($testCase, $verbose)
198+ feedback:not-run($testCase)
199 }
200 else if (exists($depMet)) then
201 {
202 feedback:not-applicable($testCase,
203- $envTestSet,
204- string-join($depMet, ''),
205- $verbose)
206+ string-join($depMet, ''))
207 }
208 else if (empty($envTestSet)) then
209 {

Subscribers

People subscribed via source and target branches