Merge lp:~zorba-coders/zorba/fixes_FOTS_driver_29_MAR into lp:zorba

[Sorin Marian Nasoi]

- updated FOTS snapshot to March 29th. Updated expected failures accordingly.
- if a test did not pass, we further check if the reason it failed is an environment dependency on setting of a default COLLECTION or COLLATION URI (other that Unicode Codepoint Collation). If so, we report the test case as "notRun" otherwise we report it as "fail". This change decreased the number of failing test cases by 34 (and increased the number of test cases "notRun" by the same amount).
- corrected errors in FOTSZorbaManifest.xml
- modified 'tools/dependencies.xq' to scan FOTS for all dependencies that are not appearing in FOTSZorbaManifest.xml and report them.

- updated FOTS snapshot to March 29th. Updated expected failures accordingly.
- if a test did not pass, we further check if the reason it failed is an environment dependency on setting of a default COLLECTION or COLLATION URI (other that Unicode Codepoint Collation). If so, we report the test case as "notRun" otherwise we report it as "fail". We first try running the test case because 130 test case actually PASS although their used environment requires setting a default COLECTION or COLLATION URI. This change decreased the number of failing test cases by 34 (and increased the number of test cases "notRun" by the same amount).

[Sorin Marian Nasoi (sorin.marian.nasoi)]

review: Approve

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/fixes_FOTS_driver_29_MAR-2013-03-29T06-29-40.76Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue job fixes_FOTS_driver_29_MAR-2013-03-29T06-29-40.76Z is finished. The final status was:

All tests succeeded!

[Zorba Build Bot (zorba-buildbot)]

Voting does not meet specified criteria. Required: Approve > 1, Disapprove < 1, Needs Fixing < 1, Pending < 1, Needs Information < 1, Resubmit < 1. Got: 1 Approve.

[Chris Hillery (ceejatec)]

The changes seem OK for now; however, they don't actually *fix* the two related bugs, do they?

review: Approve

[Sorin Marian Nasoi (sorin.marian.nasoi)]

> The changes seem OK for now; however, they don't actually *fix* the two
> related bugs, do they?
They do actually because they correctly marked some bugs as notRun instead of fail (which according to W3C is OK).
If we decide we want to increase the number of test cases that PASS, additional work needs to be done in XQXQ for both of those bugs.

[Sorin Marian Nasoi (sorin.marian.nasoi)]

Please also see:
https://bugs.launchpad.net/zorba/+bug/1160111/comments/2

My understanding of the comment is this:
Step 1) minimise the number of failing test cases as much as possible
Step 2) maximize the number of passing test cases as much as possible

What is your opinion?

[Sorin Marian Nasoi (sorin.marian.nasoi)]

One more thing: even Mike Kay says that these 2 let's call them features (ability to define COLLECTIONS and to set COLLATION URI's) is like

"implicit dependency [...] An implementation that cannot set the context as required should behave in the same way as if a dependency has not been satisfied, ideally reporting the test as "notRun" with a reason."
taken from http://dev.w3.org/2011/QT3-test-suite/guide/running.html

So the proposed changed implement the ideal case where Zorba report's the test cases as "notRun" with information about the reason it did not run them.

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/fixes_FOTS_driver_29_MAR-2013-04-02T08-36-45.465Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~zorba-coders/zorba/fixes_FOTS_driver_29_MAR into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:275 (message):
  Validation queue job fixes_FOTS_driver_29_MAR-2013-04-02T08-36-45.465Z is
  finished. The final status was:

  2 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/fixes_FOTS_driver_29_MAR-2013-04-02T18-06-54.471Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue job fixes_FOTS_driver_29_MAR-2013-04-02T18-06-54.471Z is finished. The final status was:

All tests succeeded!

=== modified file 'test/fots/CMakeLists.txt'
--- test/fots/CMakeLists.txt	2013-03-27 17:11:24 +0000
+++ test/fots/CMakeLists.txt	2013-03-29 08:37:25 +0000
@@ -135,31 +135,12 @@
 EXPECTED_FOTS_FAILURE (fn-available-environment-variables fn-available-environment-variables-011 0)
 EXPECTED_FOTS_FAILURE (fn-avg cbcl-avg-002 0)
 EXPECTED_FOTS_FAILURE (fn-avg cbcl-avg-003 0)
-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-4 0)
-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-4d 0)
-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-5 0)
-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-5d 0)
-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-6 0)
-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-8 0)
-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-9 0)
-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-10 0)
-EXPECTED_FOTS_FAILURE (fn-collection fn-collection-10d 0)
-EXPECTED_FOTS_FAILURE (fn-compare compare-010 0)
-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-56 0)
-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-57 0)
-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-58 0)
-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-59 0)
-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-60 0)
-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-61 0)
-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-62 0)
-EXPECTED_FOTS_FAILURE (fn-deep-equal K-SeqDeepEqualFunc-63 0)
 EXPECTED_FOTS_FAILURE (fn-deep-equal K2-SeqDeepEqualFunc-36 0)
 EXPECTED_FOTS_FAILURE (fn-deep-equal K2-SeqDeepEqualFunc-37 0)
 EXPECTED_FOTS_FAILURE (fn-deep-equal K2-SeqDeepEqualFunc-38 0)
 EXPECTED_FOTS_FAILURE (fn-deep-equal K2-SeqDeepEqualFunc-39 0)
 EXPECTED_FOTS_FAILURE (fn-distinct-values cbcl-distinct-values-002 0)
 EXPECTED_FOTS_FAILURE (fn-distinct-values cbcl-distinct-values-002b 0)
-EXPECTED_FOTS_FAILURE (fn-distinct-values cbcl-distinct-values-003 0)
 EXPECTED_FOTS_FAILURE (fn-environment-variable environment-variable-005 0)
 EXPECTED_FOTS_FAILURE (fn-environment-variable environment-variable-006 0)
 EXPECTED_FOTS_FAILURE (fn-environment-variable environment-variable-007 0)
@@ -218,8 +199,6 @@
 EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-004 0)
 EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-005 0)
 EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-006 0)
-EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-012 0)
-EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-013 0)
 EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-014 0)
 EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-015 0)
 EXPECTED_FOTS_FAILURE (fn-generate-id generate-id-017 0)
@@ -285,8 +264,6 @@
 EXPECTED_FOTS_FAILURE (fn-subsequence cbcl-subsequence-017 0)
 EXPECTED_FOTS_FAILURE (fn-subsequence cbcl-subsequence-018 0)
 EXPECTED_FOTS_FAILURE (fn-subsequence cbcl-subsequence-024 0)
-EXPECTED_FOTS_FAILURE (fn-substring-after fn-substring-after-24 0)
-EXPECTED_FOTS_FAILURE (fn-substring-before fn-substring-before-24 0)
 EXPECTED_FOTS_FAILURE (fn-tokenize fn-tokenize-31 0)
 EXPECTED_FOTS_FAILURE (fn-tokenize fn-tokenize-32 0)
 EXPECTED_FOTS_FAILURE (fn-unparsed-text fn-unparsed-text-037 0)
@@ -623,13 +600,7 @@
 EXPECTED_FOTS_FAILURE (xs-token cbcl-token-002b 0)
 EXPECTED_FOTS_FAILURE (xs-token cbcl-token-004 0)
 EXPECTED_FOTS_FAILURE (fn-fold-left fold-left-009 0)
-EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-018 0)
-EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-020 0)
 EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-062 0)
-EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-370 0)
-EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-372 0)
-EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-374 0)
-EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-376 0)
 EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-402 0)
 EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-404 0)
 EXPECTED_FOTS_FAILURE (fn-function-lookup fn-function-lookup-494 0)
@@ -658,13 +629,7 @@
 EXPECTED_FOTS_FAILURE (prod-FunctionCall FunctionCall-038 0)
 EXPECTED_FOTS_FAILURE (prod-FunctionCall FunctionCall-039 0)
 EXPECTED_FOTS_FAILURE (prod-InstanceofExpr instanceof140 0)
-EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-018 0)
-EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-020 0)
 EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-062 0)
-EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-370 0)
-EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-372 0)
-EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-374 0)
-EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-376 0)
 EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-402 0)
 EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-404 0)
 EXPECTED_FOTS_FAILURE (prod-NamedFunctionRef function-literal-494 0)

=== modified file 'test/fots/ImportFOTS.cmake'
--- test/fots/ImportFOTS.cmake	2013-03-21 20:02:48 +0000
+++ test/fots/ImportFOTS.cmake	2013-03-29 08:37:25 +0000
@@ -19,7 +19,7 @@
 
 
 # Change this to publish updated FOTS archives
-SET (FOTS_ARCHIVE "FOTS_210313.tgz")
+SET (FOTS_ARCHIVE "FOTS_290313.tgz")
 
 # Change this to modify which elements in FOTS driver results are output
 # as CDATA

=== modified file 'test/fots_driver/FOTSZorbaManifest.xml'
--- test/fots_driver/FOTSZorbaManifest.xml	2013-03-15 08:22:41 +0000
+++ test/fots_driver/FOTSZorbaManifest.xml	2013-03-29 08:37:25 +0000
@@ -1,6 +1,6 @@
 <test-suite-result xmlns="http://www.w3.org/2010/09/qt-fots-catalog">
-  <dependency type="feature" value="collection-stability" satisfied="true"/>          <!-- Is this correct? -->
-  <dependency type="feature" value="directory-as-collection-uri" satisfied="true"/>   <!-- Is this correct? -->
+  <dependency type="feature" value="collection-stability" satisfied="false"/>
+  <dependency type="feature" value="directory-as-collection-uri" satisfied="false"/>
   <dependency type="feature" value="higherOrderFunctions" satisfied="true"/>
   <dependency type="feature" value="moduleImport" satisfied="true"/>
   <dependency type="feature" value="namespace-axis" satisfied="false"/>               <!-- Is this correct? -->
@@ -14,6 +14,7 @@
   <dependency type="spec" value="XP10 XQ10" satisfied="true"/>
   <dependency type="spec" value="XP10+" satisfied="true"/>
   <dependency type="spec" value="XP20 XQ10" satisfied="true"/>
+  <dependency type="spec" value="XP20" satisfied="true"/>
   <dependency type="spec" value="XP20+" satisfied="true"/>
   <dependency type="spec" value="XP30+" satisfied="true"/>
   <dependency type="spec" value="XP30+ XQ10+" satisfied="true"/>

=== modified file 'test/fots_driver/environment.xq'
--- test/fots_driver/environment.xq	2013-03-26 16:25:16 +0000
+++ test/fots_driver/environment.xq	2013-03-29 08:37:25 +0000
@@ -759,3 +759,23 @@
       else ()
   }
 };
+
+(:~
+ : Checks if a test case should be run. Possible reasons for not running a test
+ : according to http://dev.w3.org/2011/QT3-test-suite/guide/running.html:
+ : - the environment requires the setting of collections
+ : - the environment requires the setting of collation URIs
+ :
+ :
+ :)
+declare function env:check-prerequisites(
+  $case as element(fots:test-case),
+  $env  as element(fots:environment)?
+) as xs:string?
+{
+  if(exists($case/fots:environment/fots:collection) or exists($env/fots:collection))
+  then 'Default collection is always an empty sequence.'
+  else if (exists($case/fots:environment/fots:collation) or exists($env/fots:collation))
+  then 'Can not define any other collations (other than the Unicode Codepoint Collation).'
+  else ()
+};

=== modified file 'test/fots_driver/feedback.xq'
--- test/fots_driver/feedback.xq	2013-03-18 12:53:46 +0000
+++ test/fots_driver/feedback.xq	2013-03-29 08:37:25 +0000
@@ -211,16 +211,20 @@
  : in the $exceptedTestCases global var).
  :
  : @param $case test case.
+ : @param $error the reason for which the test case was not run.
  : @return the test case.
  :)
 declare function feedback:not-run(
-  $case as element(fots:test-case)
+  $case   as element(fots:test-case),
+  $error  as xs:string?
 ) as element(fots:test-case)?
 {
   trace(data($case/@name), "processing test case :");
   trace("Above test case was not run.","");
 
-  <fots:test-case name="{$case/@name}" result="notRun" />
+  if(exists($error))
+  then <fots:test-case name="{$case/@name}" result="notRun" comment="{$error}"/>
+  else <fots:test-case name="{$case/@name}" result="notRun" />
 };
 
 

=== modified file 'test/fots_driver/fots-driver.xq'
--- test/fots_driver/fots-driver.xq	2013-03-21 18:43:40 +0000
+++ test/fots_driver/fots-driver.xq	2013-03-29 08:37:25 +0000
@@ -667,7 +667,7 @@
           return
             if ($isExcepted)
             then
-              feedback:not-run($testCase)
+              feedback:not-run($testCase, ())
             else
               feedback:not-applicable($testCase,
                                       string-join($depMet,''))
@@ -711,7 +711,7 @@
                               $verbose,
                               fn:false())
                else
-                 feedback:not-run($testCase)
+                 feedback:not-run($testCase,())
             }
             else if (exists($depMet)) then
             {
@@ -859,6 +859,8 @@
                                            $testSetBaseURI);
 
     variable $duration := (datetime:current-dateTime() - $startDateTime);
+    
+    variable $prerequisitesError as xs:string? := env:check-prerequisites($case, $env);
 
     if (feedback:check-pass($result, $queryName, $testSetName, $expectedFailure))
     then
@@ -869,6 +871,16 @@
                     $duration,
                     $verbose,
                     exists($expectedFailure))
+    (: 
+      If the test case did not pass, we check to see if the failure is caused
+      by a environment that requires setting of a COLLATION or COLLECTION.
+      There are over 130 test cases that are using an environment that requires
+      setting a COLLATION or COLLECTION but they still PASS even if this setting
+      is not done. That is why we first run the test case.
+     :)
+    else if(exists($prerequisitesError))
+    then
+      feedback:not-run($case, $prerequisitesError)
     else
       feedback:fail($case,
                     $result,

=== modified file 'test/fots_driver/tools/dependencies.xq'
--- test/fots_driver/tools/dependencies.xq	2012-11-13 10:45:24 +0000
+++ test/fots_driver/tools/dependencies.xq	2013-03-29 08:37:25 +0000
@@ -1,27 +1,69 @@
-(: this query loads a rezults.xml file and reports the number of 'non applicable' tests grouped by feature :)
+(: this query loads a results.xml file and reports the number of 'non applicable' tests grouped by feature :)
 (: 
-  sample usage
+  sample usage (please note that base-uri has to be an absolute URI)
+          
   ./zorba -f -q ../../test/fots_driver/tools/dependencies.xq
-          -e pathFailures:=results_12_nov.xml
+          -e pathFailures:=results.xml
+          --base-uri "/home/spungi/work/zorba/repo/tmp/fix_fots_driver/build/test/fots/2011/QT3-test-suite/catalog.xml"
           -o driver_dependencies.xml
           --indent
-:)
+ :)
 
   import module namespace file = "http://expath.org/ns/file";
+  import schema namespace fots = "http://www.w3.org/2010/09/qt-fots-catalog" at "catalog-schema.xsd";
   
   declare default element namespace "http://www.w3.org/2010/09/qt-fots-catalog";
-  declare namespace fots = "http://www.w3.org/2010/09/qt-fots-catalog";
   
   declare variable $pathFailures external;
-
+  declare variable $pathFOTSZorbaManifest external :=
+    "../../../../../test/fots_driver/FOTSZorbaManifest.xml";
+
+(:~
+ : This function is actually written by Michael Kay, Saxonica 18 July 2011 and
+ : is part of the http://dev.w3.org/cvsweb/2011/QT3-test-suite/tools/dependencies.xq
+ :)
+  declare function local:mk_dependencies($t as element(fots:test-case)) as element(fots:dependency)* {
+    $t/fots:dependency | $t/../fots:dependency
+  };
+
+(:~
+ : This function is actually written by Michael Kay, Saxonica 18 July 2011 and
+ : is part of the http://dev.w3.org/cvsweb/2011/QT3-test-suite/tools/dependencies.xq
+ :)
+  declare function local:mk_FOTS_dependencies() as element(fots:dependency)* {
+    let $test-sets := 
+    for $s in doc('catalog.xml')/fots:catalog/fots:test-set
+    let $uri := resolve-uri($s/@file, base-uri($s))
+    return doc($uri)/*
+      for $d in distinct-values($test-sets//fots:dependency/@type)
+      return 
+        <dependency type="{$d}"> {
+          let $de := $test-sets//fots:dependency[@type = $d]
+          for $v in distinct-values($de/@value)
+          return 
+             <value number-of-tests="{count($test-sets/fots:test-case[local:mk_dependencies(.)[@type = $d and @value = $v]])}">
+               {$v}
+             </value>
+     }</dependency>
+  };
+  
   variable $failures := fn:parse-xml(file:read-text($pathFailures));
+  variable $FOTSZorbaManifest := doc(resolve-uri($pathFOTSZorbaManifest));
+  variable $notDefinedInZorba :=
+      for $dep in local:mk_FOTS_dependencies()
+      for $depval in $dep/value
+      where empty($FOTSZorbaManifest/fots:test-suite-result/fots:dependency[@type=$dep/@type and @value=$depval])
+      return <dependency type="{$dep/@type}" value="{$depval/text()}" />;
   
-  <report> {
-    for $d in distinct-values(data($failures/fots:test-cases/fots:test-set/fots:test-case[@result="not applicable"]/@comment))
-    order by $d
+  <report number-of-tests="{count($failures/fots:test-cases/fots:test-set/fots:test-case[@result="n/a"])}" > {
+   (for $d in distinct-values(data($failures/fots:test-cases/fots:test-set/fots:test-case[@result="n/a"]/@comment))
+    let $failures := count($failures/fots:test-cases//fots:test-set//fots:test-case[@result="n/a" and @comment=$d])
+    order by $failures descending, $d
     return
-      <dependency type="{$d}"> {
-        <value number-of-tests="{count($failures/fots:test-cases//fots:test-set//fots:test-case[@result="not applicable" and @comment=$d])}" />
-      }</dependency>
- }</report>
- 
\ No newline at end of file
+      <dependency number-of-tests="{$failures}" type="{$d}" />,
+    if (empty($notDefinedInZorba))
+    then <!-- There are no dependencies in FOTS that are not declared in FOTSZorbaManifest.xml -->
+    else (<!-- Following dependency from FOTS is not defined in FOTSZorbaManifest.xml -->,
+          $notDefinedInZorba)
+    )
+ }</report>
\ No newline at end of file