Merge lp:~diogo-simoes89/zorba/data-cleaning-thesaurus into lp:zorba/data-cleaning-module

[Diogo Simões]

Added the capitalize function and the thesaurus-based module.

This revision includes a new normalization function: capitalize($string as xs:string) as xs:string.

It also includes the thesaurus-based module, with the check-related ( $s1 as xs:string, $s2 as xs:string, $uri as xs:string, $type as xs:string ) and the related-terms ( $s1 as xs:string, $uri as xs:string, $type as xs:string ) functions.

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/data-cleaning-thesaurus-2012-04-04T15-40-20.194Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~diogo-simoes89/zorba/data-cleaning-thesaurus into lp:zorba/data-cleaning-module failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job data-cleaning-thesaurus-2012-04-04T15-40-20.194Z is
  finished. The final status was:

  8 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/data-cleaning-thesaurus-2012-04-11T16-16-43.833Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~diogo-simoes89/zorba/data-cleaning-thesaurus into lp:zorba/data-cleaning-module failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job data-cleaning-thesaurus-2012-04-11T16-16-43.833Z is
  finished. The final status was:

  3 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/data-cleaning-thesaurus-2012-04-16T13-01-49.132Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~diogo-simoes89/zorba/data-cleaning-thesaurus into lp:zorba/data-cleaning-module failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job data-cleaning-thesaurus-2012-04-16T13-01-49.132Z is
  finished. The final status was:

  2 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/data-cleaning-thesaurus-2012-04-17T11-34-47.556Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~diogo-simoes89/zorba/data-cleaning-thesaurus into lp:zorba/data-cleaning-module failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job data-cleaning-thesaurus-2012-04-17T11-34-47.556Z is
  finished. The final status was:

  2 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/data-cleaning-thesaurus-2012-04-18T11-37-19.07Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~diogo-simoes89/zorba/data-cleaning-thesaurus into lp:zorba/data-cleaning-module failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job data-cleaning-thesaurus-2012-04-18T11-37-19.07Z is
  finished. The final status was:

  5 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Paul J. Lucas (paul-lucas)]

The capitalize() function apparently only works for English. IMHO, this is a severe limitation. That aside, it probably ought to be named something like title-capitalize().

The thesaurus:check-related() function doesn't seem right. First, the result of ft:lookup() does *not* return the relationship between two phrases. It returns the related phrases for a given phrase that *have* the relationship you give it (if you use the signature that takes a $relationship).

Second, you're (again) hard-coding English (and yet not hard-coding the URI). It seems like that's all related-terms() does. Why bother? It just adds an extra layer of function and documentation with very little utility, IMHO.

Third, the documentation for check-related() isn't clear. Describing $s1 as the first string and $s2 as the second string is insufficient. The order matters.

FYI: the way you currently specify a thesaurus (http://wordnet.princeton.edu:=$RBKT_BINARY_DIR/thesauri/wordnet-en.zth) is going to change.

[Chris Hillery (ceejatec)]

Per Helena, this is no longer needed (stale proposal and the existing module is sufficient).

Unmerged revisions

46. By Diogo Simões on 2012-04-18

Adding thesaurus-based module (CMakeLists)

45. By Diogo Simões on 2012-04-17

Added the capitalize function and the thesaurus-based module.

44. By Diogo Simões on 2012-04-15

thesaurus-based bugs fixing

43. By Diogo Simões on 2012-04-10

Capitalize funtion and thesaurus-based module

42. By Diogo Simões on 2012-04-03

Capitalize funtion and thesaurus-based module

41. By Diogo Simões on 2012-04-03

Capitalize and thesaurus

40. By Diogo Simões on 2012-04-03

Changes in documentation

39. By Diogo Simões on 2012-04-03

Capitalize function and implementation of the thesaurus-based module

38. By Diogo Simões on 2012-04-03

New normalization functions and implementation of the thesaurus-based module.

=== modified file 'src/com/zorba-xquery/www/modules/data-cleaning/CMakeLists.txt'
--- src/com/zorba-xquery/www/modules/data-cleaning/CMakeLists.txt	2011-08-07 20:36:50 +0000
+++ src/com/zorba-xquery/www/modules/data-cleaning/CMakeLists.txt	2012-04-18 10:50:24 +0000
@@ -38,3 +38,6 @@
 
 DECLARE_ZORBA_MODULE (URI "http://www.zorba-xquery.com/modules/data-cleaning/token-based-string-similarity"
                       VERSION 2.0 FILE "token-based-string-similarity.xq")
+
+DECLARE_ZORBA_MODULE (URI "http://www.zorba-xquery.com/modules/data-cleaning/thesaurus-based"
+                      VERSION 2.0 FILE "thesaurus-based.xq")

=== modified file 'src/com/zorba-xquery/www/modules/data-cleaning/normalization.xq'
--- src/com/zorba-xquery/www/modules/data-cleaning/normalization.xq	2012-04-11 09:50:34 +0000
+++ src/com/zorba-xquery/www/modules/data-cleaning/normalization.xq	2012-04-18 10:50:24 +0000
@@ -37,6 +37,28 @@
 declare option ver:module-version "2.0";
 
 (:~
+: Converts a given string into a capitalized representation.
+:
+: @param $string The string to be capitalized.
+:
+: @return The string resulting from the conversion.
+: @example test/Queries/data-cleaning/normalization/capitalize.xq
+:)
+declare function normalization:capitalize ($string as xs:string) as xs:string{
+  let $ttokens := tokenize ($string, " ")
+  let $non-capitalized-words := ("a", "an", "the", "but", "as", "if", "and", "or", "nor", "of")
+  let $cap-tokens :=
+    for $toks in $ttokens[position()>1]
+    let $capitalized-tokens := 
+      if (not($non-capitalized-words = $toks))
+      then concat(upper-case(substring($toks, 1,1)), substring(lower-case($toks), 2), " ")
+      else concat(lower-case($toks), " ")
+    return $capitalized-tokens
+  let $cap-string := concat(concat(upper-case(substring($ttokens[position()=1], 1,1)), substring(lower-case($ttokens[position()=1]), 2), " "), string-join($cap-tokens))
+  return substring($cap-string, 1, string-length($cap-string)-1)
+};
+
+(:~
  : Converts a given string representation of a date value into a date representation valid according 
  : to the corresponding XML Schema type.
  :

=== added file 'src/com/zorba-xquery/www/modules/data-cleaning/thesaurus-based.xq'
--- src/com/zorba-xquery/www/modules/data-cleaning/thesaurus-based.xq	1970-01-01 00:00:00 +0000
+++ src/com/zorba-xquery/www/modules/data-cleaning/thesaurus-based.xq	2012-04-18 10:50:24 +0000
@@ -0,0 +1,74 @@
+(:
+ : Copyright 2006-2009 The FLWOR Foundation.
+ :
+ : Licensed under the Apache License, Version 2.0 (the "License");
+ : you may not use this file except in compliance with the License.
+ : You may obtain a copy of the License at
+ :
+ : http://www.apache.org/licenses/LICENSE-2.0
+ :
+ : Unless required by applicable law or agreed to in writing, software
+ : distributed under the License is distributed on an "AS IS" BASIS,
+ : WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ : See the License for the specific language governing permissions and
+ : limitations under the License.
+ :)
+
+(:~
+ : This library module provides thesaurus functions for checking semantic relations between strings 
+ : and for checking abbreviations.
+
+ : These functions are particularly useful in tasks related to the creation of semantic mappings.
+ : 
+ :
+ : @author Bruno Martins and Diogo Simões
+ :)
+
+module namespace thesaurus = "http://www.zorba-xquery.com/modules/data-cleaning/thesaurus-based";
+
+import module namespace ft = "http://www.zorba-xquery.com/modules/full-text";
+
+(:~
+ : Checks if two strings have a relationship defined in a given thesaurus.
+ : The implementation of this function depends on the full-text module.
+ :
+ :
+ : @param $s1 The first string.
+ : @param $s2 The second string.
+ : @param $uri The uri of the thesaurus to be considered.
+ : @param $type An identifyer for the type of relationship.
+ :
+ : @return true if the first string has the provided relationship with the second string defined in the thesaurus and false otherwise.
+ : @example test/Queries/data-cleaning/thesaurus-based/check-related.xq 
+ : 
+ :)
+declare function thesaurus:check-related ( $s1 as xs:string, $s2 as xs:string, $uri as xs:string, $type as xs:string ) as xs:boolean {
+  let $relation := ft:thesaurus-lookup( $uri,
+                                      $s2,
+                                      xs:language("en"),
+                                      $type )
+  return $relation = $s1
+};
+
+(:~
+ : Returns a sequence with the strings that have a relationship, 
+ : defined in a given thesaurus, with the string provided as input.
+ : The implementation of this function depends on the full-text module.
+ :
+ :
+ : @param $s1 The string with the query term.
+ : @param $uri The uri of the thesaurus to be considered.
+ : @param $type An identifyer for the type of relationship.
+ :
+ : @return A sequence with the strings that have the provided relationship, defined in the thesaurus, with the query term.
+ : @example test/Queries/data-cleaning/thesaurus-based/related-terms.xq
+ :)
+declare function thesaurus:related-terms ( $s1 as xs:string, $uri as xs:string, $type as xs:string ) as xs:string* {
+ let $synonyms := ft:thesaurus-lookup( $uri,
+                                      $s1,
+                                      xs:language("en"),
+                                      $type )
+ return $synonyms
+};
+
+

=== added file 'test/ExpQueryResults/data-cleaning/normalization/capitalize.xml.res'
--- test/ExpQueryResults/data-cleaning/normalization/capitalize.xml.res	1970-01-01 00:00:00 +0000
+++ test/ExpQueryResults/data-cleaning/normalization/capitalize.xml.res	2012-04-18 10:50:24 +0000
@@ -0,0 +1,1 @@
+The Lord of the Rings

=== added directory 'test/ExpQueryResults/data-cleaning/thesaurus-based'
=== added file 'test/ExpQueryResults/data-cleaning/thesaurus-based/check-related.xml.res'
--- test/ExpQueryResults/data-cleaning/thesaurus-based/check-related.xml.res	1970-01-01 00:00:00 +0000
+++ test/ExpQueryResults/data-cleaning/thesaurus-based/check-related.xml.res	2012-04-18 10:50:24 +0000
@@ -0,0 +1,1 @@
+true

=== added file 'test/ExpQueryResults/data-cleaning/thesaurus-based/related-terms.xml.res'
--- test/ExpQueryResults/data-cleaning/thesaurus-based/related-terms.xml.res	1970-01-01 00:00:00 +0000
+++ test/ExpQueryResults/data-cleaning/thesaurus-based/related-terms.xml.res	2012-04-18 10:50:24 +0000
@@ -0,0 +1,1 @@
+chromatic color chromatic colour spectral color spectral colour clothing article of clothing vesture wear wearable habiliment organization organisation sky dye dyestuff amobarbital lycaenid lycaenid butterfly discolor discolour colour color coloring colouring covering consumer goods social group atmosphere coloring material colouring material barbiturate truth serum truth drug butterfly change

=== added file 'test/Queries/data-cleaning/normalization/capitalize.xq'
--- test/Queries/data-cleaning/normalization/capitalize.xq	1970-01-01 00:00:00 +0000
+++ test/Queries/data-cleaning/normalization/capitalize.xq	2012-04-18 10:50:24 +0000
@@ -0,0 +1,3 @@
+import module namespace normalization = "http://www.zorba-xquery.com/modules/data-cleaning/normalization";
+
+normalization:capitalize ("the lord of the rings")

=== added directory 'test/Queries/data-cleaning/thesaurus-based'
=== added file 'test/Queries/data-cleaning/thesaurus-based/check-related.xq'
--- test/Queries/data-cleaning/thesaurus-based/check-related.xq	1970-01-01 00:00:00 +0000
+++ test/Queries/data-cleaning/thesaurus-based/check-related.xq	2012-04-18 10:50:24 +0000
@@ -0,0 +1,25 @@
+import module namespace thesaurus = "http://www.zorba-xquery.com/modules/data-cleaning/thesaurus-based";
+
+thesaurus:check-related ( "animal", "dog", "http://wordnet.princeton.edu", "BT" )
+
+(: Example configuration (taken from zorba testsuite):
+ 
+Args: 
+--thesaurus 
+http://wordnet.princeton.edu:=$RBKT_BINARY_DIR/thesauri/wordnet-en.zth
+ 
+ 
+---------------------------------------------------------------------------------------
+Args:  --thesaurus  http://wordnet.princeton.edu:=$RBKT_BINARY_DIR/thesauri/wordnet-en.zth 
+ 
+---------------------------------------------------------------------------------------
+ 
+ 
+ 
+Expected output:
+ 
+true
+ 
+ 
+:)
+

=== added file 'test/Queries/data-cleaning/thesaurus-based/related-terms.xq'
--- test/Queries/data-cleaning/thesaurus-based/related-terms.xq	1970-01-01 00:00:00 +0000
+++ test/Queries/data-cleaning/thesaurus-based/related-terms.xq	2012-04-18 10:50:24 +0000
@@ -0,0 +1,18 @@
+import module namespace thesaurus = "http://www.zorba-xquery.com/modules/data-cleaning/thesaurus-based";
+
+thesaurus:related-terms( "blue", "http://wordnet.princeton.edu", "BT" )
+
+(: Example configuration (taken from zorba testsuite):
+ 
+Args: 
+--thesaurus 
+http://wordnet.princeton.edu:=$RBKT_BINARY_DIR/thesauri/wordnet-en.zth
+ 
+ 
+---------------------------------------------------------------------------------------
+Args:  --thesaurus  http://wordnet.princeton.edu:=$RBKT_BINARY_DIR/thesauri/wordnet-en.zth 
+ 
+---------------------------------------------------------------------------------------
+ 
+ 
+:)