Merge lp:~zorba-coders/zorba/fixes_FOTS_driver_29_MAR into lp:zorba

Proposed by Sorin Marian Nasoi on 2013-03-29
Status: Merged
Approved by: Chris Hillery on 2013-04-02
Approved revision: 11332
Merged at revision: 11334
Proposed branch: lp:~zorba-coders/zorba/fixes_FOTS_driver_29_MAR
Merge into: lp:zorba
Diff against target: 308 lines (+100/-56)
7 files modified
test/fots/CMakeLists.txt (+0/-35)
test/fots/ImportFOTS.cmake (+1/-1)
test/fots_driver/FOTSZorbaManifest.xml (+3/-2)
test/fots_driver/environment.xq (+20/-0)
test/fots_driver/feedback.xq (+6/-2)
test/fots_driver/fots-driver.xq (+14/-2)
test/fots_driver/tools/dependencies.xq (+56/-14)
To merge this branch: bzr merge lp:~zorba-coders/zorba/fixes_FOTS_driver_29_MAR
Reviewer Review Type Date Requested Status
Chris Hillery Approve on 2013-03-29
Sorin Marian Nasoi Approve on 2013-03-29
Review via email: mp+156109@code.launchpad.net

Commit message

- updated FOTS snapshot to March 29th. Updated expected failures accordingly.
- if a test did not pass, we further check if the reason it failed is an environment dependency on setting of a default COLLECTION or COLLATION URI (other that Unicode Codepoint Collation). If so, we report the test case as "notRun" otherwise we report it as "fail". This change decreased the number of failing test cases by 34 (and increased the number of test cases "notRun" by the same amount).
- corrected errors in FOTSZorbaManifest.xml
- modified 'tools/dependencies.xq' to scan FOTS for all dependencies that are not appearing in FOTSZorbaManifest.xml and report them.

Description of the change

- updated FOTS snapshot to March 29th. Updated expected failures accordingly.
- if a test did not pass, we further check if the reason it failed is an environment dependency on setting of a default COLLECTION or COLLATION URI (other that Unicode Codepoint Collation). If so, we report the test case as "notRun" otherwise we report it as "fail". We first try running the test case because 130 test case actually PASS although their used environment requires setting a default COLECTION or COLLATION URI. This change decreased the number of failing test cases by 34 (and increased the number of test cases "notRun" by the same amount).

To post a comment you must log in.
review: Approve
Zorba Build Bot (zorba-buildbot) wrote :

Validation queue job fixes_FOTS_driver_29_MAR-2013-03-29T06-29-40.76Z is finished. The final status was:

All tests succeeded!

Zorba Build Bot (zorba-buildbot) wrote :

Voting does not meet specified criteria. Required: Approve > 1, Disapprove < 1, Needs Fixing < 1, Pending < 1, Needs Information < 1, Resubmit < 1. Got: 1 Approve.

Chris Hillery (ceejatec) wrote :

The changes seem OK for now; however, they don't actually *fix* the two related bugs, do they?

review: Approve

> The changes seem OK for now; however, they don't actually *fix* the two
> related bugs, do they?
They do actually because they correctly marked some bugs as notRun instead of fail (which according to W3C is OK).
If we decide we want to increase the number of test cases that PASS, additional work needs to be done in XQXQ for both of those bugs.

Please also see:
https://bugs.launchpad.net/zorba/+bug/1160111/comments/2

My understanding of the comment is this:
Step 1) minimise the number of failing test cases as much as possible
Step 2) maximize the number of passing test cases as much as possible

What is your opinion?

One more thing: even Mike Kay says that these 2 let's call them features (ability to define COLLECTIONS and to set COLLATION URI's) is like

"implicit dependency [...] An implementation that cannot set the context as required should behave in the same way as if a dependency has not been satisfied, ideally reporting the test as "notRun" with a reason."
taken from http://dev.w3.org/2011/QT3-test-suite/guide/running.html

So the proposed changed implement the ideal case where Zorba report's the test cases as "notRun" with information about the reason it did not run them.

Zorba Build Bot (zorba-buildbot) wrote :

The attempt to merge lp:~zorba-coders/zorba/fixes_FOTS_driver_29_MAR into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:275 (message):
  Validation queue job fixes_FOTS_driver_29_MAR-2013-04-02T08-36-45.465Z is
  finished. The final status was:

  2 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

Zorba Build Bot (zorba-buildbot) wrote :

Validation queue job fixes_FOTS_driver_29_MAR-2013-04-02T18-06-54.471Z is finished. The final status was:

All tests succeeded!

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'test/fots/CMakeLists.txt'
2--- test/fots/CMakeLists.txt 2013-03-27 17:11:24 +0000
3+++ test/fots/CMakeLists.txt 2013-03-29 08:37:25 +0000
4@@ -135,31 +135,12 @@
5 EXPECTED_FOTS_FAILURE (fn-available-environment-variables fn-available-environment-variables-011 0)
6 EXPECTED_FOTS_FAILURE (fn-avg cbcl-avg-002 0)
7 EXPECTED_FOTS_FAILURE (fn-avg cbcl-avg-003 0)
8-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-4 0)
9-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-4d 0)
10-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-5 0)
11-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-5d 0)
12-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-6 0)
13-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-8 0)
14-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-9 0)
15-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-10 0)
16-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-10d 0)
17-EXPECTED_FOTS_FAILURE (fn-compare compare-010 0)
18-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-56 0)
19-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-57 0)
20-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-58 0)
21-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-59 0)
22-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-60 0)
23-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-61 0)
24-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-62 0)
25-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-63 0)
26 EXPECTED_FOTS_FAILURE (fn-deep-equal K2-SeqDeepEqualFunc-36 0)
27 EXPECTED_FOTS_FAILURE (fn-deep-equal K2-SeqDeepEqualFunc-37 0)
28 EXPECTED_FOTS_FAILURE (fn-deep-equal K2-SeqDeepEqualFunc-38 0)
29 EXPECTED_FOTS_FAILURE (fn-deep-equal K2-SeqDeepEqualFunc-39 0)
30 EXPECTED_FOTS_FAILURE (fn-distinct-values cbcl-distinct-values-002 0)
31 EXPECTED_FOTS_FAILURE (fn-distinct-values cbcl-distinct-values-002b 0)
32-EXPECTED_FOTS_FAILURE (fn-distinct-values cbcl-distinct-values-003 0)
33 EXPECTED_FOTS_FAILURE (fn-environment-variable environment-variable-005 0)
34 EXPECTED_FOTS_FAILURE (fn-environment-variable environment-variable-006 0)
35 EXPECTED_FOTS_FAILURE (fn-environment-variable environment-variable-007 0)
36@@ -218,8 +199,6 @@
37 EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-004 0)
38 EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-005 0)
39 EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-006 0)
40-EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-012 0)
41-EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-013 0)
42 EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-014 0)
43 EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-015 0)
44 EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-017 0)
45@@ -285,8 +264,6 @@
46 EXPECTED_FOTS_FAILURE (fn-subsequence cbcl-subsequence-017 0)
47 EXPECTED_FOTS_FAILURE (fn-subsequence cbcl-subsequence-018 0)
48 EXPECTED_FOTS_FAILURE (fn-subsequence cbcl-subsequence-024 0)
49-EXPECTED_FOTS_FAILURE (fn-substring-after fn-substring-after-24 0)
50-EXPECTED_FOTS_FAILURE (fn-substring-before fn-substring-before-24 0)
51 EXPECTED_FOTS_FAILURE (fn-tokenize fn-tokenize-31 0)
52 EXPECTED_FOTS_FAILURE (fn-tokenize fn-tokenize-32 0)
53 EXPECTED_FOTS_FAILURE (fn-unparsed-text fn-unparsed-text-037 0)
54@@ -623,13 +600,7 @@
55 EXPECTED_FOTS_FAILURE (xs-token cbcl-token-002b 0)
56 EXPECTED_FOTS_FAILURE (xs-token cbcl-token-004 0)
57 EXPECTED_FOTS_FAILURE (fn-fold-left fold-left-009 0)
58-EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-018 0)
59-EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-020 0)
60 EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-062 0)
61-EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-370 0)
62-EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-372 0)
63-EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-374 0)
64-EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-376 0)
65 EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-402 0)
66 EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-404 0)
67 EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-494 0)
68@@ -658,13 +629,7 @@
69 EXPECTED_FOTS_FAILURE (prod-FunctionCall FunctionCall-038 0)
70 EXPECTED_FOTS_FAILURE (prod-FunctionCall FunctionCall-039 0)
71 EXPECTED_FOTS_FAILURE (prod-InstanceofExpr instanceof140 0)
72-EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-018 0)
73-EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-020 0)
74 EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-062 0)
75-EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-370 0)
76-EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-372 0)
77-EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-374 0)
78-EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-376 0)
79 EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-402 0)
80 EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-404 0)
81 EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-494 0)
82
83=== modified file 'test/fots/ImportFOTS.cmake'
84--- test/fots/ImportFOTS.cmake 2013-03-21 20:02:48 +0000
85+++ test/fots/ImportFOTS.cmake 2013-03-29 08:37:25 +0000
86@@ -19,7 +19,7 @@
87
88
89 # Change this to publish updated FOTS archives
90-SET (FOTS_ARCHIVE "FOTS_210313.tgz")
91+SET (FOTS_ARCHIVE "FOTS_290313.tgz")
92
93 # Change this to modify which elements in FOTS driver results are output
94 # as CDATA
95
96=== modified file 'test/fots_driver/FOTSZorbaManifest.xml'
97--- test/fots_driver/FOTSZorbaManifest.xml 2013-03-15 08:22:41 +0000
98+++ test/fots_driver/FOTSZorbaManifest.xml 2013-03-29 08:37:25 +0000
99@@ -1,6 +1,6 @@
100 <test-suite-result xmlns="http://www.w3.org/2010/09/qt-fots-catalog">
101- <dependency type="feature" value="collection-stability" satisfied="true"/> <!-- Is this correct? -->
102- <dependency type="feature" value="directory-as-collection-uri" satisfied="true"/> <!-- Is this correct? -->
103+ <dependency type="feature" value="collection-stability" satisfied="false"/>
104+ <dependency type="feature" value="directory-as-collection-uri" satisfied="false"/>
105 <dependency type="feature" value="higherOrderFunctions" satisfied="true"/>
106 <dependency type="feature" value="moduleImport" satisfied="true"/>
107 <dependency type="feature" value="namespace-axis" satisfied="false"/> <!-- Is this correct? -->
108@@ -14,6 +14,7 @@
109 <dependency type="spec" value="XP10 XQ10" satisfied="true"/>
110 <dependency type="spec" value="XP10+" satisfied="true"/>
111 <dependency type="spec" value="XP20 XQ10" satisfied="true"/>
112+ <dependency type="spec" value="XP20" satisfied="true"/>
113 <dependency type="spec" value="XP20+" satisfied="true"/>
114 <dependency type="spec" value="XP30+" satisfied="true"/>
115 <dependency type="spec" value="XP30+ XQ10+" satisfied="true"/>
116
117=== modified file 'test/fots_driver/environment.xq'
118--- test/fots_driver/environment.xq 2013-03-26 16:25:16 +0000
119+++ test/fots_driver/environment.xq 2013-03-29 08:37:25 +0000
120@@ -759,3 +759,23 @@
121 else ()
122 }
123 };
124+
125+(:~
126+ : Checks if a test case should be run. Possible reasons for not running a test
127+ : according to http://dev.w3.org/2011/QT3-test-suite/guide/running.html:
128+ : - the environment requires the setting of collections
129+ : - the environment requires the setting of collation URIs
130+ :
131+ :
132+ :)
133+declare function env:check-prerequisites(
134+ $case as element(fots:test-case),
135+ $env as element(fots:environment)?
136+) as xs:string?
137+{
138+ if(exists($case/fots:environment/fots:collection) or exists($env/fots:collection))
139+ then 'Default collection is always an empty sequence.'
140+ else if (exists($case/fots:environment/fots:collation) or exists($env/fots:collation))
141+ then 'Can not define any other collations (other than the Unicode Codepoint Collation).'
142+ else ()
143+};
144
145=== modified file 'test/fots_driver/feedback.xq'
146--- test/fots_driver/feedback.xq 2013-03-18 12:53:46 +0000
147+++ test/fots_driver/feedback.xq 2013-03-29 08:37:25 +0000
148@@ -211,16 +211,20 @@
149 : in the $exceptedTestCases global var).
150 :
151 : @param $case test case.
152+ : @param $error the reason for which the test case was not run.
153 : @return the test case.
154 :)
155 declare function feedback:not-run(
156- $case as element(fots:test-case)
157+ $case as element(fots:test-case),
158+ $error as xs:string?
159 ) as element(fots:test-case)?
160 {
161 trace(data($case/@name), "processing test case :");
162 trace("Above test case was not run.","");
163
164- <fots:test-case name="{$case/@name}" result="notRun" />
165+ if(exists($error))
166+ then <fots:test-case name="{$case/@name}" result="notRun" comment="{$error}"/>
167+ else <fots:test-case name="{$case/@name}" result="notRun" />
168 };
169
170
171
172=== modified file 'test/fots_driver/fots-driver.xq'
173--- test/fots_driver/fots-driver.xq 2013-03-21 18:43:40 +0000
174+++ test/fots_driver/fots-driver.xq 2013-03-29 08:37:25 +0000
175@@ -667,7 +667,7 @@
176 return
177 if ($isExcepted)
178 then
179- feedback:not-run($testCase)
180+ feedback:not-run($testCase, ())
181 else
182 feedback:not-applicable($testCase,
183 string-join($depMet,''))
184@@ -711,7 +711,7 @@
185 $verbose,
186 fn:false())
187 else
188- feedback:not-run($testCase)
189+ feedback:not-run($testCase,())
190 }
191 else if (exists($depMet)) then
192 {
193@@ -859,6 +859,8 @@
194 $testSetBaseURI);
195
196 variable $duration := (datetime:current-dateTime() - $startDateTime);
197+
198+ variable $prerequisitesError as xs:string? := env:check-prerequisites($case, $env);
199
200 if (feedback:check-pass($result, $queryName, $testSetName, $expectedFailure))
201 then
202@@ -869,6 +871,16 @@
203 $duration,
204 $verbose,
205 exists($expectedFailure))
206+ (:
207+ If the test case did not pass, we check to see if the failure is caused
208+ by a environment that requires setting of a COLLATION or COLLECTION.
209+ There are over 130 test cases that are using an environment that requires
210+ setting a COLLATION or COLLECTION but they still PASS even if this setting
211+ is not done. That is why we first run the test case.
212+ :)
213+ else if(exists($prerequisitesError))
214+ then
215+ feedback:not-run($case, $prerequisitesError)
216 else
217 feedback:fail($case,
218 $result,
219
220=== modified file 'test/fots_driver/tools/dependencies.xq'
221--- test/fots_driver/tools/dependencies.xq 2012-11-13 10:45:24 +0000
222+++ test/fots_driver/tools/dependencies.xq 2013-03-29 08:37:25 +0000
223@@ -1,27 +1,69 @@
224-(: this query loads a rezults.xml file and reports the number of 'non applicable' tests grouped by feature :)
225+(: this query loads a results.xml file and reports the number of 'non applicable' tests grouped by feature :)
226 (:
227- sample usage
228+ sample usage (please note that base-uri has to be an absolute URI)
229+
230 ./zorba -f -q ../../test/fots_driver/tools/dependencies.xq
231- -e pathFailures:=results_12_nov.xml
232+ -e pathFailures:=results.xml
233+ --base-uri "/home/spungi/work/zorba/repo/tmp/fix_fots_driver/build/test/fots/2011/QT3-test-suite/catalog.xml"
234 -o driver_dependencies.xml
235 --indent
236-:)
237+ :)
238
239 import module namespace file = "http://expath.org/ns/file";
240+ import schema namespace fots = "http://www.w3.org/2010/09/qt-fots-catalog" at "catalog-schema.xsd";
241
242 declare default element namespace "http://www.w3.org/2010/09/qt-fots-catalog";
243- declare namespace fots = "http://www.w3.org/2010/09/qt-fots-catalog";
244
245 declare variable $pathFailures external;
246-
247+ declare variable $pathFOTSZorbaManifest external :=
248+ "../../../../../test/fots_driver/FOTSZorbaManifest.xml";
249+
250+(:~
251+ : This function is actually written by Michael Kay, Saxonica 18 July 2011 and
252+ : is part of the http://dev.w3.org/cvsweb/2011/QT3-test-suite/tools/dependencies.xq
253+ :)
254+ declare function local:mk_dependencies($t as element(fots:test-case)) as element(fots:dependency)* {
255+ $t/fots:dependency | $t/../fots:dependency
256+ };
257+
258+(:~
259+ : This function is actually written by Michael Kay, Saxonica 18 July 2011 and
260+ : is part of the http://dev.w3.org/cvsweb/2011/QT3-test-suite/tools/dependencies.xq
261+ :)
262+ declare function local:mk_FOTS_dependencies() as element(fots:dependency)* {
263+ let $test-sets :=
264+ for $s in doc('catalog.xml')/fots:catalog/fots:test-set
265+ let $uri := resolve-uri($s/@file, base-uri($s))
266+ return doc($uri)/*
267+ for $d in distinct-values($test-sets//fots:dependency/@type)
268+ return
269+ <dependency type="{$d}"> {
270+ let $de := $test-sets//fots:dependency[@type = $d]
271+ for $v in distinct-values($de/@value)
272+ return
273+ <value number-of-tests="{count($test-sets/fots:test-case[local:mk_dependencies(.)[@type = $d and @value = $v]])}">
274+ {$v}
275+ </value>
276+ }</dependency>
277+ };
278+
279 variable $failures := fn:parse-xml(file:read-text($pathFailures));
280+ variable $FOTSZorbaManifest := doc(resolve-uri($pathFOTSZorbaManifest));
281+ variable $notDefinedInZorba :=
282+ for $dep in local:mk_FOTS_dependencies()
283+ for $depval in $dep/value
284+ where empty($FOTSZorbaManifest/fots:test-suite-result/fots:dependency[@type=$dep/@type and @value=$depval])
285+ return <dependency type="{$dep/@type}" value="{$depval/text()}" />;
286
287- <report> {
288- for $d in distinct-values(data($failures/fots:test-cases/fots:test-set/fots:test-case[@result="not applicable"]/@comment))
289- order by $d
290+ <report number-of-tests="{count($failures/fots:test-cases/fots:test-set/fots:test-case[@result="n/a"])}" > {
291+ (for $d in distinct-values(data($failures/fots:test-cases/fots:test-set/fots:test-case[@result="n/a"]/@comment))
292+ let $failures := count($failures/fots:test-cases//fots:test-set//fots:test-case[@result="n/a" and @comment=$d])
293+ order by $failures descending, $d
294 return
295- <dependency type="{$d}"> {
296- <value number-of-tests="{count($failures/fots:test-cases//fots:test-set//fots:test-case[@result="not applicable" and @comment=$d])}" />
297- }</dependency>
298- }</report>
299-
300\ No newline at end of file
301+ <dependency number-of-tests="{$failures}" type="{$d}" />,
302+ if (empty($notDefinedInZorba))
303+ then <!-- There are no dependencies in FOTS that are not declared in FOTSZorbaManifest.xml -->
304+ else (<!-- Following dependency from FOTS is not defined in FOTSZorbaManifest.xml -->,
305+ $notDefinedInZorba)
306+ )
307+ }</report>
308\ No newline at end of file

Subscribers

People subscribed via source and target branches