Merge lp:~zorba-coders/zorba/feature-ft_module into lp:zorba

[Paul J. Lucas]

Documentation tweaks.

Documentation tweaks.

[Paul J. Lucas (paul-lucas)]

review: Approve

[Matthias Brantner (matthias-brantner)]

review: Approve

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-05-11T02-08-17.708Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~zorba-coders/zorba/feature-ft_module into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job feature-ft_module-2012-05-11T02-08-17.708Z is
  finished. The final status was:

  1 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-05-11T02-50-07.325Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue job feature-ft_module-2012-05-11T02-50-07.325Z is finished. The final status was:

All tests succeeded!

[lp:~zorba-coders/zorba/feature-ft_module]

10835. By Paul J. Lucas on 2012-05-11

Documentation tweaks. Approved: Matthias Brantner, Paul J. Lucas

=== modified file 'doc/zorba/ft_tokenizer.dox'
--- doc/zorba/ft_tokenizer.dox	2012-05-03 12:31:51 +0000
+++ doc/zorba/ft_tokenizer.dox	2012-05-10 23:53:19 +0000
@@ -10,9 +10,12 @@
 
 \section ft_tokenizer_tokization Tokenization
 
-Using the
-<a href="http://site.icu-project.org/">ICU library</a>,
-Zorba's implementation of tokenization
+By default,
+Zorba uses the
+<a href="http://site.icu-project.org/">ICU library</a>
+for tokenization.
+For Roman alphabets,
+Zorba (ICU)
 considers only alpha-numeric sequences of characters to be part of a token;
 whitespace and punctuation characters are not
 and separate tokens.
@@ -117,7 +120,9 @@
   <tr>
     <td>\c lang</td>
     <td>
-      The language of the string.
+      The
+      <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+      of the string.
     </td>
   </tr>
   <tr>
@@ -132,7 +137,7 @@
   <tr>
     <td>\c callback</td>
     <td>
-      The Callback to call once per token.
+      The \c Callback to call once per token.
     </td>
   </tr>
   <tr>
@@ -232,7 +237,9 @@
   <tr>
     <td>\c languages</td>
     <td>
-      The list of languages supported by the tokenizer.
+      The list of
+      <a href="http://www.w3.org/TR/xmlschema-2/#language">languages</a>
+      supported by the tokenizer.
     </td>
   </tr>
   <tr>
@@ -247,7 +254,9 @@
 
 In addition to a \c Tokenizer,
 you must also implement a \c TokenizerProvider
-that, given a language, provides a \c Tokenizer for that language:
+that,
+given a <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>,
+provides a \c Tokenizer for that language:
 
 \code
 class TokenizerProvider {
@@ -262,7 +271,11 @@
 <table>
   <tr>
     <td>\c lang</td>
-    <td>The language to tokenize.</td>
+    <td>
+      The
+      <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+      to tokenize.
+    </td>
   </tr>
   <tr>
     <td>\c num</td>