Merge lp:~zorba-coders/zorba/feature-ft_module into lp:zorba

[Paul J. Lucas]

1. Added a new full-text module.
2. Fixed semi-broken Thesaurus API.
3. Now supporting many more languages for tokenization including Chinese.
4. Many other full-text improvements.

1. Added a new full-text module.
2. Fixed semi-broken Thesaurus API.
3. Now supporting many more languages for tokenization including Chinese.
4. Many other full-text improvements.

[Matthias Brantner (matthias-brantner)]

What's the change in modules/com/zorba-xquery/www/modules/http-client.xq.src/http_request_handler.cpp supposed to do? Isn't this introducing a memory leak?

review: Needs Information

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-04-21T06-51-50.326Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-04-21T07-33-07.125Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-04-21T08-15-11.44Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-04-21T08-57-07.76Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-04-21T09-39-08.71Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-04-21T10-21-09.993Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-04-21T11-03-19.264Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-04-21T11-45-08.395Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-04-21T12-27-07.914Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-04-21T13-09-06.765Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-04-21T13-51-07.637Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-04-21T14-33-04.205Z/log.html

[Paul J. Lucas (paul-lucas)]

The Remote Queue seems to have gotten itself into an infinite loop. I've notified Chris. PLEASE DO NOT SET THE PROPOSAL TO "APPROVED" TO START THE REMOTE QUEUE UNTIL THIS IS FIXED.

@Matthias: The "theSerStream" change was do fix a crash in the module. No, I don't remember why it's fixed in this branch. I think the change is OK as it is because theSerStream is deleted via cleanUpBody() that's called from endBody(). However, I changed the code to set theSerStream to null in cleanUpBody() and put the delete back in the destructor. (Deleting a null pointer is guaranteed to be harmless in C++.)

[Chris Hillery (ceejatec)]

What seems to be happening with the queue is that the build/test is timing out after 40 minutes, which unfortunately doesn't log appropriately. I have increased the timeout to 60 minutes and we'll see if it runs. But before this gets merged, we should look and see if we can figure out why it's taking longer than normal to run with this build.

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-04-22T05-45-16.013Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-ft_module-2012-04-22T06-47-13.448Z/log.html

[Paul J. Lucas (paul-lucas)]

On Apr 21, 2012, at 10:44 PM, Chris Hillery wrote:

> What seems to be happening with the queue is that the build/test is timing out after 40 minutes, which unfortunately doesn't log appropriately. I have increased the timeout to 60 minutes and we'll see if it runs. But before this gets merged, we should look and see if we can figure out why it's taking longer than normal to run with this build.

FYI: On my machine, it took 73 minutes. The trunk build took 63 minutes.

- Paul

[Chris Hillery (ceejatec)]

Other branches have been ok on the RQ machine with a 40-minute timeout. Why
does this branch take so much longer? If it added a great many slow tests
then that's probably ok, but if something is causing everything to go
slower than that's a problem. Maybe ask on zorba-dev if it would be
possible to run the pdash tests on this branch?
On Apr 22, 2012 3:31 PM, "Paul J. Lucas" <email address hidden> wrote:

> On Apr 21, 2012, at 10:44 PM, Chris Hillery wrote:
>
> > What seems to be happening with the queue is that the build/test is
> timing out after 40 minutes, which unfortunately doesn't log appropriately.
> I have increased the timeout to 60 minutes and we'll see if it runs. But
> before this gets merged, we should look and see if we can figure out why
> it's taking longer than normal to run with this build.
>
> FYI: On my machine, it took 73 minutes. The trunk build took 63 minutes.
>
> - Paul
>
>
> --
>
> https://code.launchpad.net/~zorba-coders/zorba/feature-ft_module/+merge/102972
> Your team Zorba Coders is subscribed to branch lp:zorba.
>
> --
> Mailing list: https://launchpad.net/~zorba-coders
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~zorba-coders
> More help : https://help.launchpad.net/ListHelp
>

[Paul J. Lucas (paul-lucas)]

I diff'd the output times between the trunk and this branch and nothing jumps out at being significantly longer. (Some tests actually run in less time.) My latest running of the test suite on my machine took 69 minutes.

[Zorba Build Bot (zorba-buildbot)]

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

[lp:~zorba-coders/zorba/feature-ft_module]

10796. By Sorin Marian Nasoi <email address hidden> on 2012-04-24

Fix for lp:967284. Approved: Nicolae Brinza, Sorin Marian Nasoi

10797. By Ghislain Fourny on 2012-04-25

Fixing case where accessing an empty vector with position 0 in base64binary (causes assertion on Windows to be hit). Approved: Markos Zaharioudakis, Matthias Brantner

10798. By Markos Zaharioudakis on 2012-04-25

put back --compile-plan and --execute-plan options Approved: Markos Zaharioudakis

10799. By Juan Zacarias on 2012-04-25

Solves bug #928135
Segmentation fault on python Approved: Rodolfo Ochoa, Matthias Brantner

10800. By Matthias Brantner on 2012-04-25

fix for bug #988412 (date:current-dateTime daylight saving) Approved: William Candillon, Matthias Brantner

10801. By Paul J. Lucas on 2012-04-25

Better implementation of listing files in a directory. Approved: Rodolfo Ochoa, Cezar Andrei

10802. By Chris Hillery on 2012-04-26

Add zorba::Item::getTypeCode(), and make corresponding SchemaTypeCode enum part of the public API. Add zorba::ItemFactory::assignElementTypedValue() to allow construction of elements with typed values in the public API. As discussed, this API is not the prettiest, but it is the least intrusive while being the easiest for end-users to make use of. Approved: Markos Zaharioudakis, Chris Hillery, Matthias Brantner

10803. By Ghislain Fourny on 2012-04-27

Added an instruction to sort chained tests for deterministic order. Approved: Matthias Brantner, Markos Zaharioudakis

10804. By Paul J. Lucas on 2012-04-28

1. Added a new full-text module.
2. Fixed semi-broken Thesaurus API.
3. Now supporting many more languages for tokenization including Chinese.
4. Many other full-text improvements. Approved: Chris Hillery, Paul J. Lucas

10805. By Nicolae Brinza on 2012-04-28

Fixes for bugs #931501 and #866987 -- improved error messages for fn:format-number() Approved: Nicolae Brinza, Matthias Brantner

10806. By Rodolfo Ochoa on 2012-04-30

Collection Manager and Document Manager is ready on XQJ API. Approved: Cezar Andrei, Matthias Brantner

10807. By Chris Hillery on 2012-05-01

Fix remote queue failures due to change in 28msec.com. Approved: Till Westmann, Chris Hillery

10808. By Cezar Andrei <email address hidden> on 2012-05-01

Fix for Bug #857842 Assertion failed with simple content element with comments
Make use of XML_SCHEMA_NS definition from ns_consts.h. Approved: Matthias Brantner, Cezar Andrei

10809. By Sorin Marian Nasoi <email address hidden> on 2012-05-01

fix for lp:987830. Approved: Sorin Marian Nasoi, Markos Zaharioudakis

10810. By Sorin Marian Nasoi <email address hidden> on 2012-05-02

Added the STACK and QUEUE modules. Approved: Matthias Brantner, Sorin Marian Nasoi

10811. By Chris Hillery on 2012-05-02

Added note about re-running CMake if you add or remove .cpp files in your module source directory.
 Approved: Juan Zacarias, Chris Hillery

10812. By Nicolae Brinza on 2012-05-02

Added XQuery 3.0 Functions
path, has-children#0, nilled#0 Approved: Juan Zacarias, Matthias Brantner

10813. By Chris Hillery on 2012-05-02

Massively refactor, reformat, correct, and clean up Zorba's Build doc. Approved: William Candillon, Chris Hillery

10814. By Markos Zaharioudakis on 2012-05-02

1. Fixed memory leak in case of index truncation
2. Fixed bug in mergeUpdates() method
 Approved: Markos Zaharioudakis

10815. By Nicolae Brinza on 2012-05-03

Small optimization in the serializer to avoid a repeated string comparison Approved: Nicolae Brinza, David Graf

10816. By Markos Zaharioudakis on 2012-05-03

Fixed bug in MarkNodeCopyProps rule (static cast to replace_expr without chaing the expr kind first) Approved: Markos Zaharioudakis

10817. By Markos Zaharioudakis on 2012-05-03

fixed bug #966706 (key uniqueness of value equality index not enforced) Approved: Markos Zaharioudakis, Till Westmann, Matthias Brantner

10818. By Markos Zaharioudakis on 2012-05-03

Fixed bug #862971 (no error upon duplicate function declarations) Approved: Markos Zaharioudakis

10819. By Paul J. Lucas on 2012-05-03

Fixed warnings and build error. Approved: David Graf, Paul J. Lucas

10820. By David Graf on 2012-05-05

fixing windows build Approved: Ghislain Fourny, David Graf

10821. By Till Westmann on 2012-05-05

enable blocking of internal modules by running through URI mapping (but not through URL resolution) during translation Approved: Matthias Brantner, Markos Zaharioudakis, Chris Hillery

10822. By Markos Zaharioudakis on 2012-05-05

fixed memory leak in population of value index Approved: Markos Zaharioudakis

10823. By Paul J. Lucas on 2012-05-05

Fixed typos. Approved: Matthias Brantner, Paul J. Lucas

10824. By Matthias Brantner on 2012-05-07

optimized ft:tokenize (no validation of tokens + factorized creation of qnames) Approved: Paul J. Lucas, Matthias Brantner

10825. By Chris Hillery on 2012-05-07

Removed note about JSONiq (not in 2.5 after all). Approved: Matthias Brantner, Chris Hillery

10826. By Markos Zaharioudakis on 2012-05-08

fixed bug 867170$ Approved: Markos Zaharioudakis

10827. By Ghislain Fourny on 2012-05-08

Fixed bug 978254 (QName comparison in item sequence chainer ignored namespaces and took prefixes into account). Approved: Markos Zaharioudakis, Matthias Brantner

10828. By Paul J. Lucas on 2012-05-08

Fixes the stop-words core dump on 64-bit Linux. Approved: Matthias Brantner, Paul J. Lucas

10829. By Paul J. Lucas on 2012-05-08

1. Added fix for [\s] -- should now always throw FORX0002.
2. I think all regex tests "pass" using both pre- and post-4.0 of ICU. Approved: Ghislain Fourny, Paul J. Lucas

10830. By Matthias Brantner on 2012-05-09

no node copying during insertion into collection if the nodes are freshly constructed nodes Approved: Markos Zaharioudakis, Matthias Brantner

10831. By Matthias Brantner on 2012-05-09

- Add the ability to create a StreamableStringItem that re-uses the stream from another Streamable*Item (in a memory-ownership-safe way). Fixed bug #996084 (crash in Streamable*Item with file module)

- Add a base64:decode#2 function that also does transcoding to utf-8 Approved: Chris Hillery, Dennis Knochenwefel, William Candillon, Matthias Brantner

10832. By Paul J. Lucas on 2012-05-09

Added current-compare-options() function. Approved: Matthias Brantner, Paul J. Lucas

10833. By Matthias Brantner on 2012-05-10

note in the java documentation that it might be deprecated in favor of xqj Approved: Matthias Brantner, Rodolfo Ochoa

10834. By Paul J. Lucas on 2012-05-10

1. In transcoding streambufs, throwing std::invalid_argument for empty charsets.
2. In the HTTP code, now setting the charset to ISO-8859-1 in the constructor so it's set even when there's no Content-Type header. Approved: Dennis Knochenwefel, Paul J. Lucas

10835. By Paul J. Lucas on 2012-05-11

Documentation tweaks. Approved: Matthias Brantner, Paul J. Lucas

10836. By Nicolae Brinza on 2012-05-11

Documentation fixes. Approved: Matthias Brantner, Nicolae Brinza

10837. By Paul J. Lucas on 2012-05-11

Added link to full-text module.
Note that I have no way to test it. Approved: Matthias Brantner, Paul J. Lucas

10838. By Paul J. Lucas on 2012-05-13

Added check to see that JsonML (array form) really starts with '['. Approved: William Candillon, Paul J. Lucas

10839. By Sorin Marian Nasoi <email address hidden> on 2012-05-15

Updated the import scripts to use the W3C testsuites from 04-May-2012. Approved: Chris Hillery, Sorin Marian Nasoi

10840. By Rodolfo Ochoa on 2012-05-15

Complete XQJ Documentation Approved: Matthias Brantner, Cezar Andrei

10841. By Rodolfo Ochoa on 2012-05-15

BaseURI can now be cleared through a method.
When BaseUri is undefined it returns an empty string instead of asserting.
Fixed some compilation warnings to have a cleaner compiling.
Added #define stdafx.h to some files to fix the precompiled headers on Windows. Approved: Matthias Brantner, Chris Hillery

10842. By Matthias Brantner on 2012-05-16

use an instead of ann prefix in the documentation Approved: Matthias Brantner, Chris Hillery

10843. By Matthias Brantner on 2012-05-16

- no undo is done for collection truncate
- fix for bug #986377 "do not apply any updates on collection if it is to be truncated" Approved: Markos Zaharioudakis, Matthias Brantner

10844. By Nicolae Brinza on 2012-05-16

Documentation improvements. Fixed the type of the options parameter to the parse-fragment() function. Approved: Nicolae Brinza, Matthias Brantner

10845. By Paul J. Lucas on 2012-05-16

Renamed Tokenizer::Numbers to Tokenizer::State now (just prior to the 2.5 release) to give it a better name for the forthcoming addition of the ability to tokenize using include/exclude Item lists. At that time, State will most likely be expanded to include additional state information beyond just numbers, hence the name change.

(In the previous proposal, I had forgotten to update the documentation -- now done.) Approved: Matthias Brantner, Paul J. Lucas

10846. By Ghislain Fourny on 2012-05-16

Allowing general tree IDs (containing hexadecimal digits and dash) in structural URIs (test). Approved: Markos Zaharioudakis, Matthias Brantner

10847. By Matthias Brantner on 2012-05-16

replaced occurrences of XQuery version 1.1 with 3.0 Approved: Chris Hillery, Matthias Brantner

10848. By Chris Hillery on 2012-05-17

Fix (from Matthias) to ensure StaticContext used for invoke() lives as long as the ItemSequence returned from said invoke().
 Approved: Matthias Brantner, Chris Hillery

10849. By Matthias Brantner on 2012-05-17

- fixed itemfactory unit test on Windows
- disabled string unit test on Windows (because of bug #867271) Approved: Chris Hillery, Matthias Brantner

10850. By William Candillon on 2012-05-17

Remove dead links in the documentation. Approved: William Candillon, Matthias Brantner

10851. By Paul J. Lucas on 2012-05-17

Fixed the build error and also fixed several warnings that are new with Xcode 4.3.2. Approved: Matthias Brantner, Paul J. Lucas

10852. By Paul J. Lucas on 2012-05-18

Getting in another public API change for 2.5 for the full-text module since now's the time to do it. Renamed tokenize() to tokenize-node() for 2 reasons:

1. There already exists tokenize-string() and therefore tokenize-node() is a better name than just plain tokenize().

2. The forthcoming addition of the black & white tokenization function will most likely be called tokenize-nodes() -- plural. Approved: Matthias Brantner, Paul J. Lucas

10853. By William Candillon on 2012-05-18

Fix PHP build Approved: Paul J. Lucas, William Candillon

10854. By Sorin Marian Nasoi <email address hidden> on 2012-05-18

Update modules_svg generation target. Add mechanism for core build to detect whether a given non-core module exists (by URI). Use that method to check for graphviz before generating modules_svg. Approved: Sorin Marian Nasoi, Chris Hillery

10855. By Chris Hillery on 2012-05-18

Allow modules to specify their own libs for their Config file, in addition to the automatically-added libs from DECLARE_ZORBA_MODULE().
 Approved: Matthias Brantner, Chris Hillery

10856. By Matthias Brantner on 2012-05-19

- fetch:content-binary
- fetch:content#3 (with encoding parameter)
- StreamResource::isStreamSeekable to make sure the streamable strings returned by fetch are seekable.
- fixed two warnings in nodes_impl.cpp
- extended the C++ api to be symmetric to the fetch module Approved: Till Westmann, Chris Hillery, Matthias Brantner

10857. By Paul J. Lucas on 2012-05-19

Fixed a couple of warnings. Approved: Matthias Brantner, Paul J. Lucas

10858. By Matthias Brantner on 2012-05-21

fix for bug #1001463 (type not available during computation of function caching). Approved: Markos Zaharioudakis, Matthias Brantner

10859. By Matthias Brantner on 2012-05-21

Add support for function items test in the print xquery visitor. Approved: Matthias Brantner, William Candillon

10860. By Matthias Brantner on 2012-05-21

added stacks and queues to the data lifecycle documentation Approved: William Candillon, Matthias Brantner

10861. By Matthias Brantner on 2012-05-21

push-down of count(probe-index()) into the store Approved: Till Westmann, Matthias Brantner

10862. By Chris Hillery on 2012-05-22

Set CMAKE_INSTALL_RPATH_USE_LINK_PATH to TRUE to have correct RPATHs in installed binaries. Add comment about INSTALL_NAME_DIR. Approved: Matthias Brantner, Chris Hillery

10863. By Matthias Brantner on 2012-05-22

Update changelog, update module tags for Zorba 2.5. Approved: Matthias Brantner, Chris Hillery

10864. By Sorin Marian Nasoi <email address hidden> on 2012-05-22

Fixed bug lp:1001477. Approved: Rodolfo Ochoa, Sorin Marian Nasoi

10865. By Paul J. Lucas on 2012-05-23

No longer setting the charset of an overridden media type to a default value. Approved: David Graf, Matthias Brantner

10866. By Matthias Brantner on 2012-05-23

updated change log to reflect bug #1002867 Approved: Paul J. Lucas, Matthias Brantner

10867. By Matthias Brantner on 2012-05-30

added a cmake variable that allows to configure whether external jars are packaged or not (ZORBA_PACKAGE_EXTERNAL_JARS)
 Approved: Matthias Brantner, Chris Hillery

10868. By Paul J. Lucas on 2012-06-08

s/ZORBA_ASSERT/if/ Approved: Matthias Brantner, Paul J. Lucas

10869. By Paul J. Lucas on 2012-06-09

1. Fixed return type of ft:thesaurus-lookup().
2. Added a test to ensure that a look-up of a non-existant word works. Approved: Matthias Brantner, Paul J. Lucas

10870. By Rodolfo Ochoa on 2012-06-12

- Install added for XQJ bindings and documentation
- more documentation for all SWIG generated language bindings
- other installer related cleanups Approved: Matthias Brantner, Chris Hillery, Juan Zacarias

10871. By Markos Zaharioudakis on 2012-06-14

fixed bug #1006166 Approved: Markos Zaharioudakis

10872. By Markos Zaharioudakis on 2012-06-14

fixed bug 960083$ (improper error handling of NaN comparisons) + small optimization of comparison operations Approved: Markos Zaharioudakis

10873. By Carlos Manuel Lopez on 2012-06-14

Implements new group by syntax, as defined in the XQuery 3.0 Spec since September 2011 Approved: Markos Zaharioudakis

10874. By Markos Zaharioudakis on 2012-06-14

Fixed bug #1003023$ (optimizer problems due to common subexpression after var folding into if-then-else) Approved: Markos Zaharioudakis

10875. By Markos Zaharioudakis on 2012-06-14

fixed bug #854506 and partial fix for bug #867008 Approved: Markos Zaharioudakis

10876. By Markos Zaharioudakis on 2012-06-14

Improved hoist rule: tighter hoisting of expressions (also fixes bug #967428) Approved: Markos Zaharioudakis

10877. By Markos Zaharioudakis on 2012-06-14

Fixed bug #991088$ (raise XUST0001 in trycatch with mixed updating and simple clauses)
 Approved: Markos Zaharioudakis

10878. By Till Westmann on 2012-06-15

add location information to ZXQP0029_URI_ACCESS_DENIED Approved: Chris Hillery, Markos Zaharioudakis

10879. By Till Westmann on 2012-06-15

remove unnecessary annotations and clean-up indentation Approved: Ghislain Fourny, Till Westmann

10880. By Markos Zaharioudakis on 2012-06-15

Merged the JSONiq branch into the zorba trunk, with JSONiq deactivated. Approved: Matthias Brantner, Ghislain Fourny, Markos Zaharioudakis

10881. By Markos Zaharioudakis on 2012-06-15

renamed file test/unit/static_context.cpp to avoid conflict with src/context/static_contectx.cpp during debugging Approved: Markos Zaharioudakis

10882. By Markos Zaharioudakis on 2012-06-19

Fixed bug #1008082 (bug in transform expr when a copy var is not used anywhere) Approved: Markos Zaharioudakis

10883. By Markos Zaharioudakis on 2012-06-19

Fixed bug #932314 (non-comparable values must be treated as distinct by fn:distinct-values) Approved: Markos Zaharioudakis

10884. By Ghislain Fourny on 2012-06-20

The builtin schema type names, in the store, are now created directly from the pool (this is to prevent a cyclic dependency if another item factory than the simple item factory needs to access the store to produce the QNames, because at this point the store has not been initialized yet). Approved: Matthias Brantner, Markos Zaharioudakis

10885. By Markos Zaharioudakis on 2012-06-22

Plan serializer does not serialize expressions anymore. Approved: Markos Zaharioudakis

10886. By Markos Zaharioudakis on 2012-06-22

added serialize_csize function Approved: Markos Zaharioudakis

10887. By Rodolfo Ochoa on 2012-06-22

Error fixed on windows Approved: Chris Hillery, Cezar Andrei

10888. By Till Westmann on 2012-06-23

Added support for transient maps in the unordered-maps module. Approved: Matthias Brantner, Till Westmann

10889. By Nicolae Brinza on 2012-06-25

Improved parser error messages (bug #867357). Approved: David Graf, Matthias Brantner

10890. By Ghislain Fourny on 2012-06-26

Fixes csize serialization. Approved: Markos Zaharioudakis, Ghislain Fourny

10891. By Ghislain Fourny on 2012-06-26

Adding asserts in OrdPath::getLocalBitLength to prevent endless loops and possibly reproduce such a potential endless loop with more information. Approved: David Graf, Markos Zaharioudakis

10892. By Dennis Knochenwefel on 2012-06-26

Cleaning up attribute and element nodes' type handling and checking invariants, following an inconsistency discovered on Windows. Approved: Markos Zaharioudakis, Matthias Brantner, Dennis Knochenwefel

10893. By Nicolae Brinza on 2012-06-26

Fixed and enabled fn:parse-xml-fragment(). Approved: Juan Zacarias, Matthias Brantner

10894. By Paul J. Lucas on 2012-06-27

Added base64::streambuf class and replaced horribly inefficient base64 code. Approved: Matthias Brantner, Paul J. Lucas

10895. By Rodolfo Ochoa on 2012-06-27

Adding cacert.pem for Windows Approved: Cezar Andrei, Chris Hillery

10896. By Markos Zaharioudakis on 2012-06-27

A better fix for compilation problems involving integer types. Approved: Markos Zaharioudakis

10897. By Paul J. Lucas on 2012-06-28

Fixed warnings. Approved: Matthias Brantner, Paul J. Lucas

10898. By Markos Zaharioudakis on 2012-06-28

Optimized hash sets used by fn:distinct-values and nodes-distinct Approved: Markos Zaharioudakis

10899. By Till Westmann on 2012-06-28

modify fn:path to add support for fragments Approved: Juan Zacarias, Matthias Brantner, Till Westmann

10900. By Paul J. Lucas on 2012-06-29

Added C++98 subset of C++11's unordered_map. The configure process checks for a working C++11 unordered_map first: if found, it will be used instead. The goal is to phase out the old hash*map* classes over time. Approved: Markos Zaharioudakis, Matthias Brantner

10901. By Till Westmann on 2012-06-29

add ref:has-node-reference and ref:assign-node-reference Approved: Markos Zaharioudakis, Matthias Brantner

10902. By Paul J. Lucas on 2012-06-29

Added the missing specialization for hash<unique_ptr<T,D>>. Approved: Matthias Brantner, Paul J. Lucas

10903. By Matthias Brantner on 2012-06-29

- added API function Item:isSeekable
- fixed a problem in the file module where all exceptions being throw in the body were caught and rethrown as file exception Approved: Chris Hillery, Matthias Brantner

10904. By David Graf on 2012-06-29

add positional skipping to access of static and dynamic collections Approved: Matthias Brantner, Till Westmann

10905. By Markos Zaharioudakis on 2012-06-29

Fixed bug #1018673 + renamed misleading isXXXNode() methods. Approved: Markos Zaharioudakis

10906. By Markos Zaharioudakis on 2012-06-29

Optimized hash function used for nodes (fixes bug #1010051) + some hashmap/hashset cleanup Approved: Markos Zaharioudakis

10907. By Markos Zaharioudakis on 2012-06-29

Fixed bug #1016429 (scoping of copy variables in transform expr) Approved: Markos Zaharioudakis

10908. By Paul J. Lucas on 2012-06-29

Added tokenize-nodes() function. Approved: Matthias Brantner, Paul J. Lucas

10909. By David Graf on 2012-07-03

windows build fix Approved: Dennis Knochenwefel, David Graf

10910. By Ghislain Fourny on 2012-07-03

Differentiating between two reasons why references are unregistered. Approved: Markos Zaharioudakis, Matthias Brantner

10911. By Ghislain Fourny on 2012-07-03

Fixed a bug preventing from compiling Zorba single-threadedly. Approved: Markos Zaharioudakis, Ghislain Fourny

10912. By Dennis Knochenwefel on 2012-07-04

fix for bug #1020953 (access of freed object) Approved: Ghislain Fourny, Nicolae Brinza

10913. By Ghislain Fourny on 2012-07-04

Adding a NOT ZORBA_HAVE_UNIQUE_PTR guard for adding the unique_ptr test, for consistency. There are guards for including the corresponding resources, so that the test was failing by lack of them.
 Approved: Markos Zaharioudakis, Till Westmann

10914. By Dennis Knochenwefel on 2012-07-05

fixed memory leak reading data of zstring after destruction. Encoded illegal char in exception to not cause any problems. Approved: Till Westmann, Ghislain Fourny

10915. By Dennis Knochenwefel on 2012-07-05

fixed CHECK_CXX_SOURCE_COMPILES doesn't work with backslash-n using cmake 2.8.8 on windows. Approved: Paul J. Lucas, Dennis Knochenwefel

10916. By Paul J. Lucas on 2012-07-05

Now checking data size before accessing vector[0]. Approved: Dennis Knochenwefel, Paul J. Lucas

10917. By David Graf on 2012-07-06

Unifying .bat and .vcproj generation. Bug #1013075 Approved: Rodolfo Ochoa, Dennis Knochenwefel

10918. By Chris Hillery on 2012-07-09

EXPECTED_FAILURE()s for two tests failing due to flworfound.org changes. Approved: Matthias Brantner, Chris Hillery

10919. By Paul J. Lucas on 2012-07-09

1. Added missing "lang" attribute to tokens generated from tokenize-nodes().
2. Added a test. Approved: Chris Hillery, Paul J. Lucas

10920. By Paul J. Lucas on 2012-07-09

Moved URI resolution for the thesaurus into the translator. Approved: Matthias Brantner, Paul J. Lucas

10921. By Paul J. Lucas on 2012-07-09

Fixed warnings. Approved: Matthias Brantner, Paul J. Lucas

10922. By Markos Zaharioudakis on 2012-07-10

Fixed bug #1022557 (subsequence function applied on window variable) Approved: Markos Zaharioudakis

10923. By Ghislain Fourny on 2012-07-10

Several modifications in the implementation of JSONiq to make it in sync with the new specification, in particular:
- No more pairs,
- Minor changes to the update syntax
- Synced returned error codes
- Support for collections and indices, including automatic maintenance
- Serializer extended to allow mixed JDM and XDM
- Updated C++ API Approved: Matthias Brantner, Ghislain Fourny, Chris Hillery, Markos Zaharioudakis

10924. By Markos Zaharioudakis on 2012-07-11

added auditing in zorba cmd + fixed bug in auditing the parse time Approved: Markos Zaharioudakis

10925. By Markos Zaharioudakis on 2012-07-11

Fixed bug #1002993 (bug during revalidation after update due to improper condition for calling TypeOps::get_atomic_type_code() from SchemaValidatorImpl::isPossibleSimpleContentRevalImpl()) Approved: Cezar Andrei, Markos Zaharioudakis

10926. By Ghislain Fourny on 2012-07-11

Adding two tests for allowing two errors upon sequence of several JSON items and an XML node. Approved: Till Westmann, Matthias Brantner

10927. By Markos Zaharioudakis on 2012-07-11

Avoid (if possible) treat expr for checking that the value of a non-external global variable conforms to the type declaration of the vatiable Approved: Markos Zaharioudakis

10928. By Nicolae Brinza on 2012-07-11

Implemented the new EQName syntax. Approved: Matthias Brantner, Nicolae Brinza

10929. By Paul J. Lucas on 2012-07-11

Reverted previous "fix." Approved: Matthias Brantner, Paul J. Lucas

10930. By Markos Zaharioudakis on 2012-07-12

Streaming execution for tumbling windows (also fixes bug #1010051) Approved: Markos Zaharioudakis

10931. By Ghislain Fourny on 2012-07-12

Fixed build errors occurring on Windows by exposing structured item types even in JSONiq is deactivated. Approved: Matthias Brantner, Markos Zaharioudakis

10932. By Markos Zaharioudakis on 2012-07-12

Incremental maintenance for general indexes. Approved: Markos Zaharioudakis

10933. By Paul J. Lucas on 2012-07-13

Fixed typo. Approved: Chris Hillery, Paul J. Lucas

10934. By Sorin Marian Nasoi <email address hidden> on 2012-07-13

Updated the F&O 1.1 and 3.0 documents used in the comparison of the functions and corrected the util:download-and-write-spec function. Approved: Chris Hillery, Sorin Marian Nasoi

10935. By Chris Hillery on 2012-07-13

Add test case demonstrating bug 1010728. Approved: Markos Zaharioudakis, Chris Hillery

10936. By David Graf on 2012-07-13

Removed internal debug info from fn:trace out by using zorba serializer instead of the internal show function. Approved: Chris Hillery, David Graf

10937. By Dennis Knochenwefel on 2012-07-13

Added case for DT_UNKNOWN. Approved: Dennis Knochenwefel, Paul J. Lucas

10938. By Rodolfo Ochoa on 2012-07-14

Fix for precompiled headers on Windows Approved: Rodolfo Ochoa, Chris Hillery

10939. By Cezar Andrei <email address hidden> on 2012-07-14

Integrate:
  Fix for bug 1023120 xs:include ignored in schemas.
  Added a new test for this case. Approved: Chris Hillery, David Graf

10940. By Till Westmann on 2012-07-14

fix position for elements and processing-instructions in fn:path Approved: Markos Zaharioudakis, Matthias Brantner

10941. By Nicolae Brinza on 2012-07-16

Added support for the unix shebang script launcher (#!/path/interpreter) Approved: Nicolae Brinza, David Graf

10942. By Paul J. Lucas on 2012-07-16

Now using enable_if for more functions to make overload resolution work better. Approved: Matthias Brantner, Paul J. Lucas

10943. By Rodolfo Ochoa on 2012-07-17

Disabling "print stack trace" for windows Approved: Chris Hillery, Rodolfo Ochoa

10944. By Paul J. Lucas on 2012-07-17

Now handling UTF-16 surrogate pairs. Approved: Dennis Knochenwefel, Paul J. Lucas

10945. By Nicolae Brinza on 2012-07-17

The parse-fragment function now allows a DOCTYPE declaration in the input. Approved: Nicolae Brinza, Matthias Brantner

10946. By Paul J. Lucas on 2012-07-18

Added functions to test for and create UTF-16 surrogate pairs.
These will probably be needed by whoever fixes bug #1025622. Approved: Dennis Knochenwefel, Paul J. Lucas

10947. By Ghislain Fourny on 2012-07-19

Fix that gives precedence to an array replacement over an array deletion, and adds tests about multiple updates with same selector on arrays and on objects. Approved: Markos Zaharioudakis, Matthias Brantner

10948. By Paul J. Lucas on 2012-07-19

Now doing proper JSON serialization. Approved: Chris Hillery, Dennis Knochenwefel, Paul J. Lucas

10949. By Paul J. Lucas on 2012-07-20

No longer checking captured subgroups in replacement string when 'q' flag is given. Approved: Matthias Brantner, Paul J. Lucas

10950. By Matthias Brantner on 2012-07-21

removed a non-core module dependency from the reuse-stream test Approved: Chris Hillery, Matthias Brantner

10951. By Markos Zaharioudakis on 2012-07-24

removed some debugging code, which causes Windows compilation problems Approved: Markos Zaharioudakis

10952. By Paul J. Lucas on 2012-07-25

Removed zorbatypes/transcoder.h & .cpp. Approved: Matthias Brantner, Paul J. Lucas

10953. By Paul J. Lucas on 2012-07-26

Now properly serializing JSON for JsonML. Approved: Chris Hillery, Paul J. Lucas

10954. By Paul J. Lucas on 2012-07-26

1. Tweaked equals().
2. Added "const&" to std_string function arguments. Approved: Matthias Brantner, Paul J. Lucas

10955. By Matthias Brantner on 2012-07-27

fix for bug #898066 (Stringstream & fn:trace) Approved: Chris Hillery, Matthias Brantner

10956. By Sorin Marian Nasoi <email address hidden> on 2012-07-27

- added information about the Zorba and XQTS versions that were used in generating the reports
- updated the README.txt with info regarding conformance reports generation Approved: William Candillon, Sorin Marian Nasoi

10957. By Ghislain Fourny on 2012-07-27

Activating ZORBA_WITH_JSON by default. Approved: Chris Hillery, Matthias Brantner

10958. By Rodolfo Ochoa on 2012-07-27

Documentation fixed for Zorba binary package with PHP. Approved: Cezar Andrei, Chris Hillery

10959. By Paul J. Lucas on 2012-07-27

s/0/npos/

This probably fixes some as-of-yet-undiscovered bug. Approved: Matthias Brantner, Paul J. Lucas

10960. By Rodolfo Ochoa on 2012-07-30

C# API Binding Approved: Cezar Andrei, Chris Hillery

10961. By Ghislain Fourny on 2012-07-31

Corrects bug 1029836. Approved: Chris Hillery, Matthias Brantner

10962. By Ghislain Fourny on 2012-07-31

Fixing Windows build with JSONiq. Approved: Chris Hillery, Matthias Brantner

10963. By Paul J. Lucas on 2012-08-02

Removed JsonML-object. Approved: Ghislain Fourny, Paul J. Lucas

10964. By Matthias Brantner on 2012-08-03

new ItemFactory function that allows creating dateTime items without timezone Approved: Chris Hillery, Matthias Brantner

10965. By Ghislain Fourny on 2012-08-03

Fixes bug 1032166 (critical memory bug). Approved: Chris Hillery, Matthias Brantner

10966. By Paul J. Lucas on 2012-08-03

1. s/take/swap/
2. Removed unnecessary assignment. Approved: Matthias Brantner, Paul J. Lucas

10967. By Juan Zacarias on 2012-08-03

Fix of bugs
Bug #1014979: Make thesaurus optional component on Windows
Bug #1014981: Make default thesaurus available as Ubuntu package
Also Fixed FindJNI.cmake to be found-able in the Ubuntu Installer Approved: Juan Zacarias, Chris Hillery

10968. By David Graf on 2012-08-03

Make testdriver_mt work with boost version >1.49. Approved: David Graf, Chris Hillery

10969. By Matthias Brantner on 2012-08-03

Added a keys() function to the index dml module. This fixes bug #900677. Approved: Matthias Brantner, David Graf

10970. By Chris Hillery on 2012-08-05

Test temporarily disabled while http-client is being updated. Also get HEAD of OAuth module rather than zorba-2.5 tag for some reason. Approved: Juan Zacarias, Chris Hillery

10971. By Chris Hillery on 2012-08-06

Split image into image + graphviz; split data-converters into csv + html. Approved: Juan Zacarias, Chris Hillery

10972. By William Candillon on 2012-08-06

Fix make doc target when multiple version of the same module exists. Approved: Sorin Marian Nasoi, Matthias Brantner

10973. By Ghislain Fourny on 2012-08-06

Correct array insert expression syntax to include [] like append expressions. Approved: Matthias Brantner, Chris Hillery

10974. By Juan Zacarias on 2012-08-06

Fixed windows installer component options for c# Bindings Approved: Rodolfo Ochoa, Chris Hillery

10975. By Matthias Brantner on 2012-08-07

removed some hardcoded english words from error messages raised by the json parser Approved: Paul J. Lucas, Matthias Brantner

10976. By Chris Hillery on 2012-08-07

Regenerate scanner and parser with flex/bison 2.5 - apparently 2.6 creates code that Clang doesn't like. Approved: Matthias Brantner, Chris Hillery

10977. By Matthias Brantner on 2012-08-07

prepare ChangeLog for 2.6 release Approved: Chris Hillery, Matthias Brantner

10978. By Chris Hillery on 2012-08-08

Bump version number to 2.6; update tagged modules; add archive module. Approved: Cezar Andrei, Sorin Marian Nasoi, Chris Hillery

10979. By David Graf on 2012-08-09

Since OSX Mountain Lion, clang is the default c++ compiler on mac. Therefore, it is not enough to check if the c++ compiler executable is called clang. Approved: David Graf, Paul J. Lucas

10980. By Chris Hillery on 2012-08-10

Add "generation" of local copy of FlexLexer.h, to ensure it always stays in sync with the generated lexer .cpp file. Approved: Paul J. Lucas, Chris Hillery

10981. By Sorin Marian Nasoi <email address hidden> on 2012-08-13

Fixed the XQDoc documentation issue related to the formatting of the parameters for the higher order functions. Approved: William Candillon, Sorin Marian Nasoi

10982. By Chris Hillery on 2012-08-14

Try to find FlexLexer.h associated with the flex binary being used.
 Approved: David Graf, Chris Hillery

10983. By Markos Zaharioudakis on 2012-08-14

Fixed bug #1033407 (do not store var_expr rchandles in the static context) Approved: Markos Zaharioudakis

10984. By Markos Zaharioudakis on 2012-08-14

Fixed bugs #899364 and 899363 (throw XQST0103 in case of non-distinct window variables)
Fixed bug #899366 (enforce the type declaration of a window variable) Approved: Markos Zaharioudakis

10985. By Markos Zaharioudakis on 2012-08-14

Fixed bug #1024892 (index declaration references udf declared after the index) Approved: Markos Zaharioudakis

10986. By David Graf on 2012-08-14

remove clang warnings in auditing code Approved: Matthias Brantner, Till Westmann, David Graf

10987. By Chris Hillery on 2012-08-15

added the archive module to ExternalModules.conf and the ChangeLog Approved: Luis Rodriguez Gonzalez, Juan Zacarias, Chris Hillery, Matthias Brantner

10988. By David Graf on 2012-08-15

Because the jsoniq_emitter aggregates the xml_emitter, it needs to handover all the parameters. Inclusive aEmitAttributes. Approved: Till Westmann, David Graf

10989. By luisrod <luisrod@LUISROD-LAP> on 2012-08-16

- Added code for bug#1025564 "Deprecate -f argument to zorbacmd"

10990. By Markos Zaharioudakis on 2012-08-16

Allow prolog variables to be referenced before they are declared (XQuery 3.0 feature) (fixes bug #900688) Approved: Markos Zaharioudakis

10991. By Chris Hillery on 2012-08-17

Revert unintentional commit r10989. Approved: Luis Rodriguez Gonzalez, Chris Hillery

10992. By Ghislain Fourny on 2012-08-20

Updated JSONiq tutorial. Approved: William Candillon, Matthias Brantner

10993. By Paul J. Lucas on 2012-08-21

Miscellaneous changes, some a prerequisite for LLVM that should be done anyway and not have to wait for the far-in-the-future LLVM branch merge. Approved: Matthias Brantner, Paul J. Lucas

10994. By Ghislain Fourny on 2012-08-21

Specifying collection and property upon ZDST0006. Approved: Till Westmann, Matthias Brantner

10995. By Paul J. Lucas on 2012-08-21

QueryLoc clean-up:
1. Added all-argument constructor (needed for LLVM).
2. Removed pointless copy constructor since default is fine.
3. Removed pointless virtual destructor (there are no virtual functions!). Approved: Matthias Brantner, Paul J. Lucas

10996. By Nicolae Brinza on 2012-08-22

Dynamically computed strings can now be cast to xs:QName. Fixes bug #898792 Approved: Nicolae Brinza, Chris Hillery

10997. By Till Westmann on 2012-08-23

move appending of ${requiredlibs-store} to requiredlibs to a place where it works Approved: Ghislain Fourny, Till Westmann

10998. By Ghislain Fourny on 2012-08-24

Fixes a bug that makes Zorba crash upon inserting more than one pair. Approved: Till Westmann, Matthias Brantner

10999. By Ghislain Fourny on 2012-08-25

Adds a method isEncoded to user-typed atomic items. Approved: Till Westmann, Matthias Brantner

11000. By Sorin Marian Nasoi <email address hidden> on 2012-08-27

fix for lp:969251. Approved: Ghislain Fourny, Sorin Marian Nasoi

11001. By Ghislain Fourny on 2012-08-27

Made URI computation lazy in StructuralAnyUri. Approved: Markos Zaharioudakis, Matthias Brantner

11002. By Ghislain Fourny on 2012-08-27

Fixes a Windows compiler error (bug 1040558). Approved: Luis Rodriguez Gonzalez, Juan Zacarias

11003. By William Candillon on 2012-08-28

Enable XML output from doxygen by default. Approved: Chris Hillery, Matthias Brantner

11004. By Markos Zaharioudakis on 2012-08-28

Fixed bug #1038410 (Memory leaks in parser, trace iterator, and general index) Approved: Markos Zaharioudakis

11005. By Markos Zaharioudakis on 2012-08-28

Fixed bug #1042840 (qname pool free-list corruption) Approved: Markos Zaharioudakis

11006. By Chris Hillery on 2012-08-28

Restoring execute bit to a bunch of scripts. Approved: Juan Zacarias, Matthias Brantner, Chris Hillery

11007. By Carlos Manuel Lopez on 2012-08-29

New memory management for compiler expressions (fixes bug #1036111) Approved: Markos Zaharioudakis

11008. By Ghislain Fourny on 2012-08-29

Simplified JSON items class and fixed some Xml Node static casts to handle JSON items as well. Approved: Markos Zaharioudakis, Matthias Brantner

11009. By Markos Zaharioudakis on 2012-08-29

Fixed bug #866984 (better error message for an eval error) Approved: Markos Zaharioudakis

11010. By Ghislain Fourny on 2012-08-29

Fixing a memory leak in append update primitive. Approved: Matthias Brantner, Till Westmann

11011. By William Candillon on 2012-08-29

Remove deprecated reference to the old sourceforge mailing-list. Approved: Matthias Brantner, William Candillon

11012. By Nicolae Brinza on 2012-08-30

Fixes for bugs #1023170, #1024033, #1027270 Approved: Chris Hillery, Matthias Brantner

11013. By Markos Zaharioudakis on 2012-08-30

rchandle cleanup Approved: Markos Zaharioudakis

11014. By Paul J. Lucas on 2012-08-31

Replaced UUID with thin layer over native platform implementation. Approved: Chris Hillery, Rodolfo Ochoa, Matthias Brantner, Paul J. Lucas

11015. By Chris Hillery on 2012-08-31

Corrected HTML serialization of empty elements. Added test cases for XHTML.
 Approved: Matthias Brantner, Chris Hillery

11016. By Chris Hillery on 2012-08-31

Fixes debug mode crash because of missing dictionary entries. Approved: William Candillon, Chris Hillery

11017. By Ghislain Fourny on 2012-08-31

Fixed bug 1041411 (prefixed true/false/null should be interpreted as name tests). Approved: Matthias Brantner, Chris Hillery

11018. By Ghislain Fourny on 2012-09-04

Removing superfluous store/naive prefixes in store includes. Approved: Till Westmann, Matthias Brantner

11019. By Paul J. Lucas on 2012-09-04

Suppressed warnings; moved gcc diagnostic push macros to config.h. Approved: Matthias Brantner, Paul J. Lucas

11020. By Till Westmann on 2012-09-05

add dependency on libuuid

11021. By Till Westmann on 2012-09-05

ensure deterministic test results for keys of index on unordered collection by sorting

11022. By Till Westmann on 2012-09-05

Some fixes in TreeID API and some include cleanup.

11023. By Matthias Brantner on 2012-09-05

Adding missing JSONiq library functions.

11024. By Matthias Brantner on 2012-09-05

Updated JSONiq tutorial.

11025. By Matthias Brantner on 2012-09-08

implementation of parse-json#2 allowing multiple top-level items

11026. By Matthias Brantner on 2012-09-08

more tests for jn:parse-json

11027. By Matthias Brantner on 2012-09-08

adapted changelog regarding jn:parse-json

Unmerged revisions

=== modified file 'ChangeLog'
--- ChangeLog	2012-04-24 12:39:38 +0000
+++ ChangeLog	2012-04-24 21:06:20 +0000
@@ -10,6 +10,7 @@
   * fn:unparsed-text-available
   * Extended API for Python, Java, PHP and Ruby.
   * Add jvm classpath to zorbacmd and to Zorba API. Tracked by #931816
+  * Added full-text module.
   * Added support for NO_ICU (to not use ICU for unicode processing)
   * Added XQJ support.
 
@@ -88,6 +89,8 @@
   * Fixed bug 867509 (Can not handle largest xs:unsignedLong values)
   * Fixed bug 924063 (sentence is incorrectly incremented when token characters end without sentence terminator)
   * Fixed bug 909126 (bug in cloning of var_expr)
+  * Fixed bug 928631 (external builtin function were not executed in the module they
+    were declared)
   * Fixed bug in destruction of exit_catcher_expr
   * Fixed bug #867024 (error messages)
   * Fixed bug #957580 (stream read failure in StringToCodepointsIteartor)
 
=== modified file 'cmake_modules/FindICU.cmake'
--- cmake_modules/FindICU.cmake	2012-04-24 14:35:54 +0000
+++ cmake_modules/FindICU.cmake	2012-04-24 21:06:20 +0000
@@ -28,6 +28,8 @@
 #                       (note: in addition to ICU_LIBRARIES)
 #  ICU_DATA_LIBRARIES - Libraries to link against for ICU data
 #
+#  ICU_VERSION        - ICU's version number.
+#
 
 # Look for the header file.
 find_path(
 
=== modified file 'doc/zorba/ft_intro.dox'
--- doc/zorba/ft_intro.dox	2012-04-24 12:39:38 +0000
+++ doc/zorba/ft_intro.dox	2012-04-24 21:06:20 +0000
@@ -5,9 +5,9 @@
 specification.
 Additional documentation:
 
-  - \ref ft_stemmer
-  - \ref ft_thesaurus
-  - \ref ft_tokenizer
+- \ref ft_stemmer
+- \ref ft_thesaurus
+- \ref ft_tokenizer
 
 \section ft_unimplemented Unimplemented Features
 
@@ -16,11 +16,11 @@
 implemented.
 The features that are not (completely) implemented are:
 
-  - The <a href="http://www.w3.org/TR/xpath-full-text-10/#ftignoreoption">Ignore Option</a>
-    (bug <a href="https://bugs.launchpad.net/zorba/+bug/sf-3187470">3187470</a>).
-  - <a href="http://www.w3.org/TR/xpath-full-text-10/#section-score-variables">Score Variables</a>
-    and <a href="http://www.w3.org/TR/xpath-full-text-10/#section-using-weights">Using Weights Within a Scored FTContainsExpr</a>
-    (bug <a href="https://bugs.launchpad.net/zorba/+bug/sf-3187462">3187462</a>).
+- The <a href="http://www.w3.org/TR/xpath-full-text-10/#ftignoreoption">Ignore Option</a>
+  (bug <a href="https://bugs.launchpad.net/zorba/+bug/866924">866924</a>).
+- <a href="http://www.w3.org/TR/xpath-full-text-10/#section-score-variables">Score Variables</a>
+  and <a href="http://www.w3.org/TR/xpath-full-text-10/#section-using-weights">Using Weights Within a Scored FTContainsExpr</a>
+  (bug <a href="https://bugs.launchpad.net/zorba/+bug/866923">866923</a>).
 
 */
 /* vim:set et sw=2 ts=2: */
 
=== modified file 'doc/zorba/ft_stemmer.dox'
--- doc/zorba/ft_stemmer.dox	2012-04-24 12:39:38 +0000
+++ doc/zorba/ft_stemmer.dox	2012-04-24 21:06:20 +0000
@@ -56,7 +56,12 @@
 public:
   typedef /* implementation-defined */ ptr;
 
+  struct Properties {
+    char const *uri;
+  };
+
   virtual void destroy() const = 0;
+  virtual void properties( Properties *result ) const = 0;
   virtual void stem( String const &word, locale::iso639_1::type lang, String *result ) const = 0;
 protected:
   virtual ~Stemmer();
@@ -89,6 +94,8 @@
 Note that \c result should always be set to something.
 If your stemmer doesn't know how to stem the given word,
 you should set \c result to \c word.
+You also need to implement the \c properties() function
+and set the identifying URI of your stemmer.
 
 A very simple stemmer
 that stems the word "foobar" to "foo"
@@ -98,6 +105,7 @@
 class MyStemmer : public Stemmer {
 public:
   void destroy() const;
+  void properties( Properties *result ) const;
   void stem( String const &word, locale::iso639_1::type lang, String *result ) const;
 private:
   MyStemmer();
@@ -108,6 +116,10 @@
   // Do nothing since we statically allocate a singleton instance of our stemmer.
 }
 
+void MyStemmer::properties( Properties *props ) const {
+  props->uri = "http://my.example.com/zorba/full-text/stemmer";
+}
+
 void MyStemmer::stem( String const &word, locale::iso639_1::type lang, String *result ) const {
   if ( word == "foobar" )
     *result = "foo";
@@ -120,7 +132,6 @@
 or a dictionary look-up
 to stem many words,
 of course.
-
 Although not used in this simple example,
 \c lang can be used to allow a single stemmer instance
 to stem words in more than one language.
@@ -135,16 +146,24 @@
 class StemmerProvider {
 public:
   virtual ~StemmerProvider();
-  virtual Stemmer::ptr getStemmer( locale::iso639_1::type lang ) const = 0;
+  virtual bool getStemmer( locale::iso639_1::type lang, Stemmer::ptr *s = 0 ) const = 0;
 };
 \endcode
 
+The \c getStemmer() function should return \c true
+only if it can provide a \c Stemmer
+for the given language; \c false otherwise.
+If the \c Stemmer::ptr argument is \c null,
+the caller wants to check only whether the provider
+can provide a stemmer for the given language
+and doesn't want a \c Stemmer instance created or returned.
+
 A simple \c StemmerProvider for our simple stemmer can be implemented as:
 
 \code
 class MyStemmerProvider : public StemmerProvider {
 public:
-  Stemmer::ptr getStemmer( locale::iso639_1::type lang ) const;
+  bool getStemmer( locale::iso639_1::type lang Stemmer::ptr *s = 0 ) const;
 };
 
 Stemmer::ptr MyStemmerProvider::getStemmer( locale::iso639_1::type lang ) const {
@@ -154,15 +173,14 @@
     case iso639_1::en:
     case iso639_1::unknown: // Handle "unknown" language since, in many cases, the language is not known.
       result.reset( &stemmer );
-      break;
+      return true;
     default: 
       //
-      // We have no stemmer for the given language: leave the result as null to indicate this.
+      // We have no stemmer for the given language: return false.
       // Zorba will then use the built-in stemmer for the given language.
       //
-      break;
+      return false;
   }
-  resturn std::move( result );
 }
 \endcode
 
 
=== modified file 'doc/zorba/ft_thesaurus.dox'
--- doc/zorba/ft_thesaurus.dox	2012-04-24 12:39:38 +0000
+++ doc/zorba/ft_thesaurus.dox	2012-04-24 21:06:20 +0000
@@ -44,16 +44,16 @@
 To download and install the WordNet database on a Unix-like system,
 follow these steps:
 
-  -# Download the WordNet database from
-     <a href="http://wordnet.princeton.edu/wordnet/download/">here</a>.
-     All you really need are just the database files
-     (<code>WNdb-3.0.tar.gz</code>).
-  -# Un-gzip and untar the files.
-     This will result in a directory dict containing the database files.
-  -# Move the dict directory somewhere of your choosing,
-     e.g., <code>/usr/local/wordnet-3.0/dict</code>.
-  -# Compile the \c dict directory into a Zorba-compatible binary thesaurus
-     as described below.
+-# Download the WordNet database from
+   <a href="http://wordnet.princeton.edu/wordnet/download/">here</a>.
+   All you really need are just the database files
+   (<code>WNdb-3.0.tar.gz</code>).
+-# Un-gzip and untar the files.
+   This will result in a directory dict containing the database files.
+-# Move the dict directory somewhere of your choosing,
+   e.g., <code>/usr/local/wordnet-3.0/dict</code>.
+-# Compile the \c dict directory into a Zorba-compatible binary thesaurus
+   as described below.
 
 To compile the WordNet database files,
 use the \c zt-wn-compile script
@@ -65,12 +65,12 @@
 zt-wn-compile [-v] wordnet_dict_dir [thesaurus_file]
 \endcode
 
-  - The \c -v option specifies verbose output.
-  - The \e wordnet_dict_dir specifies the full path
-    of the WordNet \c dict directory.
-  - The \e thesaurus_file specifies the name of the resulting binary file.
-    If none is given, it defaults to \c wordnet-en.zth
-    ("en" for English and "zth" for "Zorba Thesaurus file").
+- The \c -v option specifies verbose output.
+- The \e wordnet_dict_dir specifies the full path
+  of the WordNet \c dict directory.
+- The \e thesaurus_file specifies the name of the resulting binary file.
+  If none is given, it defaults to \c wordnet-en.zth
+  ("en" for English and "zth" for "Zorba Thesaurus file").
 
 For example:
 
@@ -78,33 +78,39 @@
 zt-wn-compile -v /usr/local/wordnet-3.0/dict
 \endcode
 
-Move the \c wordnet-en.zth file to a location of your choosing.
+To install the \c wordnet-en.zth file,
+move it onto Zorba's <i>library path</i>:
+
+\code
+LIB_PATH/edu/princeton/wordnet/wordnet-en.zth
+\endcode
 
 \subsection ft_thesaurus_precompiled Downloading a Precompiled WordNet Database
 
 Alternatively,
-you can download a precompiled WordNet database from
+you can download a precompiled, little-endian (Intel) CPU WordNet database from
 <a href="http://www.zorba-xquery.com/downloads/WordNet-3.0/wordnet-en.zip">here</a>.
 
 \section ft_thesaurus_mappings Thesauri Mappings
 
 In order to use thesauri,
-you need to specify where they are to the Zorba engine
-via one or more thesaurus <i>mappings</i>.
-A <i>mapping</i> maps a symbolic URI to URI for an actual thesaurus.
+you need to specify what symbolic URI(s) <i>map</i>
+to what thesauri.
 A mapping is of the form:
 
-<i>from_uri</i><code>:=</code><b>[</b><i>implementation</i><code>|</code><b>]</b><i>to_uri</i>
+<i>from_uri</i><code>:=</code><i>implementation-scheme</i><code>:</code><i>to_uri</i>
 
 For example:
 
 \code
-http://wordnet.princeton.edu:=wordnet|/usr/local/zorba/thesauri/wordnet-en.zth
+http://wordnet.princeton.edu:=wordnet://wordnet.princeton.edu
 \endcode
 
 says that the symbolic URI \c http://wordnet.princeton.edu
 maps to the WordNet implementation
-having a database file at the given path.
+having a database file at the given sub-path
+\c edu/princeton/wordnet
+on Zorba's library path.
 Once a mapping is established for a symbolic URI,
 it can be used in a query:
 
@@ -114,13 +120,8 @@
   using thesaurus at "http://wordnet.princeton.edu"
 \endcode
 
-If the \e implementation is omitted,
-it defaults to \c wordnet.
 As a special-case,
-the \e from_uri can be \c default or 
-\code 
-##default
-\endcode
+the \e from_uri can be \c default or \c ##default
 to allow for specifying the default thesaurus
 as was done for the first example on this page.
 
@@ -130,7 +131,7 @@
 use one or more –thesaurus options:
 
 \code
-zorba --thesaurus default:=/usr/local/zorba/thesauri/wordnet-en.zth ...
+zorba --thesaurus default:=wordnet://wordnet.princeton.edu ...
 \endcode
 
 \section ft_thesaurus_rels Thesaurus Relationships
@@ -423,25 +424,26 @@
 
 If no levels are specified in a query,
 Zorba defaults the WordNet implementation to be 2 levels.
-The rationale can be found
-<a href="http://www.w3.org/Bugs/Public/show_bug.cgi?id=11444">here</a>.
+(The rationale can be found
+<a href="http://www.w3.org/Bugs/Public/show_bug.cgi?id=11444">here</a>.)
 
 \section ft_thesaurus_providing Providing Your Own Thesaurus
 
 Using the Zorba C++ API,
 you can provide your own thesaurus
-by deriving from three classes:
+by deriving from four classes:
 \c Thesaurus,
 \c Thesaurus::iterator,
+\c ThesaurusProvider,
 and
-\c ThesaurusProvider.
+\c URLResolver.
 
 \subsection ft_class_thesaurus The Thesaurus Class
 
 The \c Thesaurus class is:
 
 \code
-class Thesaurus : public Resource {
+class Thesaurus {
 public:
   typedef /* implementation-defined */ ptr;
   typedef /* implementation-defined */ range_type;
@@ -457,15 +459,15 @@
 
   virtual iterator::ptr lookup( String const &phrase, String const &relationship, range_type at_least, range_type at_most ) const = 0;
 
-  virtual void destroy() const = 0;     // interited from Resource
+  virtual void destroy() const = 0;
 protected:
   virtual ~Thesaurus();
 };
 \endcode
 
-For details about the \c ptr type,
-the \c destroy() function,
-and why the destructor is \c protected,
+For details about the \c ptr types,
+the \c destroy() functions,
+and why the destructors are \c protected,
 see the \ref memory_management document.
 
 To implement the \c Thesaurus
@@ -482,18 +484,19 @@
   </tr>
   <tr>
     <td>\c at_least</td>
-    <td>The The minimum number of levels within the thesaurus to be traversed.</td>
+    <td>The minimum number of levels within the thesaurus to be traversed.</td>
   </tr>
   <tr>
     <td>\c at_most</td>
-    <td>The The maximum number of levels within the thesaurus to be traversed.</td>
+    <td>The maximum number of levels within the thesaurus to be traversed.</td>
   </tr>
 </table>
 
 The \c lookup() function returns a pointer to an \c iterator
 that is used to iterate over the phrase's synonyms.
-
-A very simple thesaurus
+You also need to implement an \c iterator.
+A very simple \c Thesaurus
+and its \c iterator
 can be implemented as:
 
 \code
@@ -505,53 +508,49 @@
   //
   // Define a simple thesaurus data structure as a map from a phrase to a list of its synonyms.
   //
-  typedef std::list<String> synonyms_t;
-  typedef std::map<String,synonyms_t const*> thesaurus_t;
+  typedef std::list<String> synonyms_type;
+  typedef std::map<String,synonyms_type const*> thesaurus_data_type;
 
-  static thesaurus_t const& get_thesaurus();
+  static thesaurus_data_type const& get_thesaurus_data();
 
   class iterator : public Thesaurus::iterator {
   public:
-    iterator( synonyms_t const &s ) : synonyms_( s ), i_( s.begin() ) { }
+    iterator( synonyms_type const &s ) : synonyms_( s ), i_( s.begin() ) { }
     void destroy();
     bool next( String *synonym );
   private:
-    synonyms_t const &synonyms_;      // synonyms to iterate over
-    synonyms_t::const_iterator i_;    // current iterator position
+    synonyms_type const &synonyms_;     // synonyms to iterate over
+    synonyms_type::const_iterator i_;   // current iterator position
   };
 };
 
 void MyThesaurus::destroy() const {
-  // Do nothing since we statically allocate a singleton instance of our thesaurus.
+  // Do nothing since we statically allocate a singleton instance of our Thesaurus.
 }
 
-MyThesaurus::thesaurus_t const& MyThesaurus::get_thesaurus() {
-  static thesaurus_t thesaurus;
-  if ( thesaurus.empty() ) {
-    //
-    // Construct a thesaurus "by hand" for this example.  A real thesaurus would probably
-    // be read from disk.
-    //
+MyThesaurus::thesaurus_data_type const& MyThesaurus::get_thesaurus_data() {
+  static thesaurus_data_type thesaurus_data;
+  if ( thesaurus_data.empty() ) {
+    //
+    // Construct thesaurus data "by hand" for this example.  A real thesaurus would probably be read from disk.
     // Note that every list of synonyms must always include the original phrase.
     //
-    static synonyms_t synonyms;
+    static synonyms_type synonyms;
     synonyms.push_back( "foo" );
     synonyms.push_back( "foobar" );
-    thesaurus[ "foo"    ] = &synonyms;
-    thesaurus[ "foobar" ] = &synonyms;
+    thesaurus_data[ "foo"    ] = &synonyms;
+    thesaurus_data[ "foobar" ] = &synonyms;
   }
-  return thesaurus;
+  return thesaurus_data;
 }
-\endcode
 
-\code
 MyThesaurus::iterator::ptr MyThesaurus::lookup( String const &phrase, String const &relationship,
                                                 range_type at_least, range_type at_most ) const {
-  static thesaurus_t const &thesaurus = get_thesaurus();
-  thesaurus_t::const_iterator const i = thesaurus.find( phrase );
+  static thesaurus_data_type const &thesaurus_data = get_thesaurus_data();
+  thesaurus_data_type::const_iterator const entry = thesaurus_data.find( phrase );
   iterator::ptr result;
-  if ( i != thesaurus.end() )
-    result.reset( new iterator( *i->second ) );
+  if ( entry != thesaurus_data.end() )
+    result.reset( new iterator( *entry->second ) );
   return std::move( result );
 }
 
@@ -572,13 +571,71 @@
 A real thesaurus would load a large number of synonyms,
 of course.
 
+\subsection ft_class_thesaurus_provider The ThesaurusProvider Class
+
+The \c ThesaurusProvider class is:
+
+\code
+class ThesaurusProvider : public Resource {
+public:
+  typedef /* implementation-defined */ ptr;
+
+  virtual bool getThesaurus( locale::iso639_1::type lang, Thesaurus::ptr *thesaurus = 0 ) const = 0;
+  void destroy() const;                 // inherited from Resource
+};
+\endcode
+
+To implement a \c ThesaurusProvider,
+you need to implement the \c getThesaurus() function where:
+
+<table>
+  <tr>
+    <td>\c lang</td>
+    <td>The desired language of the thesaurus.</td>
+  </tr>
+  <tr>
+    <td>\c thesaurus</td>
+    <td>If not \c null, set to point to a thesaurus for \c lang.</td>
+  </tr>
+</table>
+
+The \c getThesaurus() function returns \c true
+only if it can provide a thesaurus for the given language.
+Continuing with the example,
+a very simple \c ThesaurusProvider
+can be implemented as:
+
+\code
+class MyThesaurusProvider : pulic ThesaurusProvider {
+public:
+  void destroy() const;
+  bool getThesaurus( iso639_1::type lang, Thesaurus::ptr* = 0 ) const;
+};
+
+void MyThesaurusProvider::destroy() const {
+  // Do nothing since we statically allocate a singleton instance of our ThesaurusProvider.
+}
+
+bool MyThesaurusProvider::getThesaurus( iso639_1::type lang, Thesaurus::ptr *result ) const {
+  //
+  // Since our tiny thesaurus contains only universally known words, we don't bother checking lang
+  // and always return true.
+  //
+  static MyThesaurus thesaurus;
+  if ( result )
+    result->reset( &thesaurus );
+  return true;
+}
+\endcode
+
 \subsection ft_class_thesaurus_resolver A Thesaurus URL Resolver Class
 
-In addition to a \c Thesaurus,
+In addition to a \c Thesaurus
+and \c ThesaurusProvider,
 you must also implement a "thesaurus resolver" class
 that,
-given a URL and a language,
-provides a \c Thesaurus for that language.
+given a URI,
+provides a \c ThesaurusProvider for that URI.
 A simple \c ThesaurusURLResolver
 for our simple thesaurus can be implemented as:
 
@@ -591,23 +648,12 @@
   String const url_;
 };
 
-Resource*
-ThesaurusURLResolver::resolveURL( String const &url, EntityData const *data ) const {
-  ThesaurusEntityData const *const t_data = dynamic_cast<ThesaurusEntityData const*>( data );
-  assert( t_data );
-  static MyThesaurus thesaurus;
-  if ( url == url_ )
-    switch ( t_data->getLanguage() ) {
-      case locale::iso639_1::en:
-      case locale::iso639_1::unknown:
-        //
-        // Here, we could test to ensure that the language of our thesaurus matches the
-        // language sought, but in our case, we want our thesaurus to be used for all
-        // languages since "foo" and "foobar" are universal.
-        //
-      default:
-        return &thesaurus;
-    }
+Resource* ThesaurusURLResolver::resolveURL( String const &url, EntityData const *data ) const {
+  if ( data->getKind() == EntityData::THESAURUS )
+    static MyThesaurusProvider provider;
+    if ( uri == uri_ )
+      return &provider;
+  }
   return 0;
 }
 \endcode
 
=== modified file 'doc/zorba/ft_tokenizer.dox'
--- doc/zorba/ft_tokenizer.dox	2012-04-24 12:39:38 +0000
+++ doc/zorba/ft_tokenizer.dox	2012-04-24 21:06:20 +0000
@@ -5,14 +5,25 @@
 The Zorba XQuery processor implements the
 <a href="http://www.w3.org/TR/xpath-full-text-10/">XQuery and XPath Full Text 1.0</a>
 specification that, among other things,
-tokenizes a string into a sequence of tokens.
-See
-<a href="http://www.w3.org/TR/xpath-full-text-10/#TokenizationSec">Tokenization</a>.
-
-The initial implementation of the toknenizer
-uses the one provided by the
-<a href="http://site.icu-project.org/">ICU library</a>.
-However, you can provide your own tokenizer instead.
+<a ref="http://www.w3.org/TR/xpath-full-text-10/#TokenizationSec">tokenizes</a>
+a string into a sequence of tokens.
+
+\section ft_tokenizer_tokization Tokenization
+
+Using the
+<a href="http://site.icu-project.org/">ICU library</a>,
+Zorba's implementation of tokenization
+considers only alpha-numeric sequences of characters to be part of a token;
+whitespace and punctuation characters are not
+and separate tokens.
+However, alpha-numeric sequences matching the regular expression
+<code>[0-9][.,][0-9]</code>
+are retained as part of a token, e.g.:
+"98.6" and "1,432.58" are tokens.
+
+Alternatively,
+you can implement your own tokenizer
+by deriving from the \c Tokenizer class.
 
 \section ft_class_tokenizer The Tokenizer Class
 
@@ -36,33 +47,43 @@
 
   class Callback {
   public:
-    typedef Tokenizer::size_type size_type;;
+    typedef Tokenizer::size_type size_type;
 
     virtual ~Callback();
 
-    virtual void operator()( char const *utf8_s, size_type utf8_len,
-                             size_type token_no, size_type sent_no, size_type para_no,
-                             void *payload = 0 ) = 0;
-  };
-
-  enum ElementTraceOptions {
-    trace_none  = 0x0,  // Trace no elements.
-    trace_begin = 0x1,  // Trace the beginning of elements.
-    trace_end   = 0x2   // Trace the ending of elements.
-  };
+    virtual void token( char const *utf8_s, size_type utf8_len, locale::iso639_1::type lang,
+                        size_type token_no, size_type sent_no, size_type para_no,
+                        Item const *item = 0 ) = 0;
+  };
+
+  struct Properties {
+    typedef std::vector<locale::iso639_1::type> languages_type;
+
+    bool comments_separate_tokens;
+    bool elements_separate_tokens;
+    bool processing_instructions_separate_tokens;
+    languages_type languages;
+    char const *uri;
+  };
+
+  virtual void properties( Properties *result ) const = 0;
 
   virtual void destroy() const = 0;
-  virtual void element( Item const &qname, int trace_options );
   Numbers& numbers();
   Numbers const& numbers() const;
-  int trace_options() const;
-
-  virtual void tokenize( char const *utf8_s, size_type utf8_len, locale::iso639_1::type lang,
-                         bool wildcards, Callback &callback, void *payload = 0 ) = 0;
+
+  void tokenize_node( Item const &node, locale::iso639_1::type lang, Callback &callback );
+
+  virtual void tokenize_string( char const *utf8_s, size_type utf8_len, locale::iso639_1::type lang,
+                                bool wildcards, Callback &callback, Item const *item = 0 ) = 0;
 
 protected:
-  Tokenizer( Numbers&, int trace_options = trace_none );
+  Tokenizer( Numbers& );
   virtual ~Tokenizer();
+
+  bool find_lang_attribute( Item const&, locale::iso639_1::type *lang );
+  virtual void item( Item const&, bool entering );
+  virtual void tokenize_node_impl( Item const&, locale::iso639_1::type, Callback&, bool tokenize_acp );
 };
 \endcode
 
@@ -76,8 +97,8 @@
 It simply keeps track of the current
 token, sentence, and paragraph numbers.
 
-To implement the \c Tokenizer,
-you need to implement the \c %tokenize() function where:
+To implement a \c Tokenizer,
+you need to implement the \c %tokenize_string() function where:
 
 <table>
   <tr>
@@ -115,9 +136,13 @@
     </td>
   </tr>
   <tr>
-    <td>\c payload</td>
+    <td>\c item</td>
     <td>
-      Optional implementation-defined data.
+      The \c Item whence this token came.
+      If the token occurred within an element,
+      the \c Item is the text node.
+      If the token occurred within an attribute,
+      the \c Item is the attribute node.
     </td>
   </tr>
 </table>
@@ -127,21 +152,30 @@
 However,
 the things a tokenizer should take into consideration include:
 
-  - Detecting sentence termination ('.', '?', and '!' characters).
-  - Handling floating-point numbers with possible thousands separators
-    in US and European formats, e.g. "98.7", "98,7", "10,000", etc.
-  - Distinguishing '.' used as a sentence terminator
-    from '.' used as a decimal point.
-  - Handling apostrophies, e.g., "men's".
-  - Handling acronyms, e.g., "AT&T".
-
-\subsection ft_paragraphs Paragraphs
+- Detecting sentence termination ('.', '?', and '!' characters).
+- Handling floating-point numbers with possible thousands separators
+  in US and European formats, e.g. "98.7", "98,7", "10,000", etc.
+- Distinguishing '.' used as a sentence terminator
+  from '.' used as a decimal point.
+- Handling apostrophies, e.g., "men's".
+- Handling acronyms, e.g., "AT&T".
+
+The task of iterating over an XML element's child nodes
+is done by \c tokenize_node_impl().
+Its default implementation
+treats XML elements, comments, and processing instructions
+as token separators.
+(See \ref ft_tokenizer_properties.)
+If you want to change that,
+you need to override \c tokenize_node_impl().
+
+\subsection ft_tokenizer_paragraphs Paragraphs
 
 By default,
 Zorba increments the current paragraph number once
 for each XML element encountered.
 However,
-this doens't work well for mixed content.
+this doesn't work well for mixed content.
 For example, in the XHTML:
 \code
 <p>The <em>best</em> thing ever!</p>
@@ -150,31 +184,65 @@
 but Zorba will consider that 3 paragraphs by default.
 
 Your tokenizer can take control over when the paragraph number is incremented
-by passing the bitwise-or
-of the \c ElementTraceOptions values
-to the constructor
-and overriding the \c element() function.
-The \c element() function is passed the QName of the current XML element
-and (depending on the initial value passed to the constructor)
-one of \c trace_begin or \c trace_end.
-Note that this function is called
-only if the trace options value
-passed to the constructor
-was non-zero.
+by overriding the \c item() function.
+The \c item() function is passed the \c Item of the current XML element
+and whether the item is being entered or exited.
 
 For example,
-the \c element() function for tokenizing XHTML
+the \c item() function for tokenizing XHTML
 would be along the lines of:
 \code
-void MyTokenizer::element( Item const &qname, int trace_options ) {
-  if ( trace_options & trace_end )
-    return;
-  String const name( qname.getLocalName() );
-  if ( /* qname is an XHTML block-level element */ )
-    ++numbers().para;
+void MyTokenizer::item( Item const &item, bool entering ) {
+  if ( entering && item.isNode() && item.getNodeKind() == store::StoreConsts::elementNode ) {
+    Item qname;
+    item.getNodeName( qname );
+    if ( /* qname matches an XHTML block-level element's name */ )
+      ++numbers().para;
 }
 \endcode
 
+\subsection ft_tokenizer_properties Properties
+
+To implement a \c Tokenizer,
+you need also to implement the \c %properties() function
+that fills in the \c Properties struct where:
+
+<table>
+  <tr>
+    <td>\c comments_separate_tokens</td>
+    <td>
+      If \c true, XML comments separate tokens.  For example,
+      <code>net&lt;!-- --&gt;work</code> would be 2 tokens instead of 1.
+    </td>
+  </tr>
+  <tr>
+    <td>\c elements_separate_tokens</td>
+    <td>
+      If \c true, XML elements separate tokens.  For example,
+      <code>&lt;b&gt;B&lt;/b&gt;old</code> would be 2 tokens instead of 1.
+    </td>
+  </tr>
+  <tr>
+    <td>\c processing_instructions_separate_tokens</td>
+    <td>
+      If \c true, XML processing instructions separate tokens.  For example,
+      <code>net&lt;?PI pi?&gt;work</code> would be 2 tokens instead of 1.
+    </td>
+  </tr>
+  <tr>
+    <td>\c languages</td>
+    <td>
+      The list of languages supported by the tokenizer.
+    </td>
+  </tr>
+  <tr>
+    <td>\c uri</td>
+    <td>
+      The URI that uniquely identifies the %Tokenizer.
+    </td>
+  </tr>
+</table>
+
 \section ft_class_tokenizer_provider The TokenizerProviderClass
 
 In addition to a \c Tokenizer,
@@ -185,20 +253,51 @@
 class TokenizerProvider {
 public:
   virtual ~TokenizerProvider();
-  virtual Tokenizer::ptr getTokenizer( locale::iso639_1::type lang, Tokenizer::Numbers &numbers ) const = 0;
+  virtual bool getTokenizer( locale::iso639_1::type lang, Tokenizer::Numbers *numbers = 0, Tokenizer::ptr* = 0 ) const = 0;
 };
 \endcode
 
+Specifically, you need to implement the \c getTokenizer() function where:
+
+<table>
+  <tr>
+    <td>\c lang</td>
+    <td>The language to tokenize.</td>
+  </tr>
+  <tr>
+    <td>\c num</td>
+    <td>
+      The \c Numbers to use.
+      If \c null,
+      \a t is not set.
+    </td>
+  </tr>
+  <tr>
+    <td>\c t</td>
+    <td>
+      If not \c null,
+      set to point to a Tokenizer for \a lang.
+    </td>
+  </tr>
+</table>
+
 A simple \c TokenizerProvider for our tokenizer can be implemented as:
 
 \code
 class MyTokenizerProvider : public TokenizerProvider {
 public:
-  Tokenizer::ptr getTokenizer( locale::iso639_1::type lang ) const;
+  getTokenizer( locale::iso639_1::type lang, Tokenizer::Numbers* = 0, Tokenizer::ptr* = 0 ) const;
 };
 
-Tokenizer::ptr MyTokenizerProvider::getTokenizer( locale::iso639_1::type lang const {
-  return Tokenizer::ptr( new MyTokenizer );
+bool MyTokenizerProvider::getTokenizer( locale::iso639_1::type lang, Tokenizer::Numbers *num, Tokenizer::ptr *t ) const {
+  switch ( lang ) {
+    case iso639_1::en:
+      if ( num && t )
+        t->reset( new MyTokenizer );
+      return true;
+    default:
+      return false;
+  }
 }
 \endcode
 
 
=== modified file 'include/zorba/locale.h'
--- include/zorba/locale.h	2012-04-24 12:39:38 +0000
+++ include/zorba/locale.h	2012-04-24 21:06:20 +0000
@@ -22,24 +22,198 @@
 
     /////////////////////////////////////////////////////////////////////////// 
 
+    /**
+     * Defines constants for all ISO 639-1 language codes.
+     */
     namespace iso639_1 {
       enum type {
         unknown,
-        da,   // Danish
-        de,   // German
-        en,   // English
-        es,   // Spanish
-        fi,   // Finnish
-        fr,   // French
-        hu,   // Hungarian
-        it,   // Italian
-        nl,   // Dutch
-        no,   // Norwegian
-        pt,   // Portuguese
-        ro,   // Romanian
-        ru,   // Russian
-        sv,   // Swedish
-        tr,   // Turkish
+        aa,   ///< Afar
+        ab,   ///< Abkhazian
+        ae,   ///< Avestan
+        af,   ///< Afrikaans
+        ak,   ///< Akan
+        am,   ///< Amharic
+        an,   ///< Aragonese
+        ar,   ///< Arabic
+        as,   ///< Assamese
+        av,   ///< Avaric
+        ay,   ///< Aymara
+        az,   ///< Azerbaijani
+        ba,   ///< Bashkir
+        be,   ///< Byelorussian
+        bg,   ///< Bulgarian
+        bh,   ///< Bihari
+        bi,   ///< Bislama
+        bm,   ///< Bambara
+        bn,   ///< Bengali; Bangla
+        bo,   ///< Tibetan
+        br,   ///< Breton
+        bs,   ///< Bosnian
+        ca,   ///< Catalan
+        ce,   ///< Chechen
+        ch,   ///< Chamorro
+        co,   ///< Corsican
+        cr,   ///< Cree
+        cs,   ///< Czech
+        cu,   ///< Church Slavic; Church Slavonic
+        cv,   ///< Chuvash
+        cy,   ///< Welsh
+        da,   ///< Danish
+        de,   ///< German
+        dv,   ///< Divehi
+        dz,   ///< Bhutani
+        ee,   ///< Ewe
+        el,   ///< Greek
+        en,   ///< English
+        eo,   ///< Esperanto
+        es,   ///< Spanish
+        et,   ///< Estonian
+        eu,   ///< Basque
+        fa,   ///< Persian
+        ff,   ///< Fulah
+        fi,   ///< Finnish
+        fj,   ///< Fiji
+        fo,   ///< Faroese
+        fr,   ///< French
+        fy,   ///< Frisian
+        ga,   ///< Irish
+        gd,   ///< Scots Gaelic
+        gl,   ///< Galician
+        gn,   ///< Guarani
+        gu,   ///< Gujarati
+        gv,   ///< Manx
+        ha,   ///< Hausa
+        he,   ///< Hebrew (formerly iw)
+        hi,   ///< Hindi
+        ho,   ///< Hiri Motu
+        hr,   ///< Croatian
+        ht,   ///< Haitian Creole
+        hu,   ///< Hungarian
+        hy,   ///< Armenian
+        hz,   ///< Herero
+        ia,   ///< Interlingua
+        id,   ///< Indonesian (formerly in)
+        ie,   ///< Interlingue
+        ig,   ///< Igbo
+        ii,   ///< Nuosu
+        ik,   ///< Inupiak
+        io,   ///< Ido
+        is,   ///< Icelandic
+        it,   ///< Italian
+        iu,   ///< Inuktitut
+        ja,   ///< Japanese
+        jv,   ///< Javanese
+        ka,   ///< Georgian
+        kg,   ///< Kongo
+        ki,   ///< Gikuyu
+        kj,   ///< Kuanyama
+        kk,   ///< Kazakh
+        kl,   ///< Greenlandic
+        km,   ///< Cambodian
+        kn,   ///< Kannada
+        ko,   ///< Korean
+        kr,   ///< Kanuri
+        ks,   ///< Kashmiri
+        ku,   ///< Kurdish
+        kv,   ///< Komi
+        kw,   ///< Cornish
+        ky,   ///< Kirghiz
+        la,   ///< Latin
+        lb,   ///< Letzeburgesch
+        lg,   ///< Ganda
+        li,   ///< Limburgan; Limburger; Limburgish
+        ln,   ///< Lingala
+        lo,   ///< Laothian
+        lt,   ///< Lithuanian
+        lu,   ///< Luba-Katanga
+        lv,   ///< Latvian
+        mg,   ///< Malagasy
+        mh,   ///< Marshallese
+        mi,   ///< Maori
+        mk,   ///< Macedonian
+        ml,   ///< Malayalam
+        mn,   ///< Mongolian
+        mo,   ///< Moldavian
+        mr,   ///< Marathi
+        ms,   ///< Malay
+        mt,   ///< Maltese
+        my,   ///< Burmese
+        na,   ///< Nauru
+        nb,   ///< Norwegian Bokmal
+        nd,   ///< Ndebele, North
+        ne,   ///< Nepali
+        ng,   ///< Ndonga
+        nl,   ///< Dutch
+        nn,   ///< Norwegian Nynorsk
+        no,   ///< Norwegian
+        nr,   ///< Ndebele, South
+        nv,   ///< Navajo; Navaho
+        ny,   ///< Chichewa; Chewa; Nyanja
+        oc,   ///< Occitan
+        oj,   ///< Ojibwa
+        om,   ///< Oromo
+        or_,  ///< Oriya
+        os,   ///< Ossetian; Ossetic
+        pa,   ///< Panjabi; Punjabi
+        pi,   ///< Pali
+        pl,   ///< Polish
+        ps,   ///< Pashto, Pushto
+        pt,   ///< Portuguese
+        qu,   ///< Quechua
+        rm,   ///< Romansh
+        rn,   ///< Kirundi
+        ro,   ///< Romanian
+        ru,   ///< Russian
+        rw,   ///< Kinyarwanda
+        sa,   ///< Sanskrit
+        sc,   ///< Sardinian
+        sd,   ///< Sindhi
+        se,   ///< Northern Sami
+        sg,   ///< Sangho
+        sh,   ///< Serbo-Croatian
+        si,   ///< Sinhalese
+        sk,   ///< Slovak
+        sl,   ///< Slovenian
+        sm,   ///< Samoan
+        sn,   ///< Shona
+        so,   ///< Somali
+        sq,   ///< Albanian
+        sr,   ///< Serbian
+        ss,   ///< Siswati
+        st,   ///< Sesotho
+        su,   ///< Sundanese
+        sv,   ///< Swedish
+        sw,   ///< Swahili
+        ta,   ///< Tamil
+        te,   ///< Telugu
+        tg,   ///< Tajik
+        th,   ///< Thai
+        ti,   ///< Tigrinya
+        tk,   ///< Turkmen
+        tl,   ///< Tagalog
+        tn,   ///< Setswana
+        to,   ///< Tonga
+        tr,   ///< Turkish
+        ts,   ///< Tsonga
+        tt,   ///< Tatar
+        tw,   ///< Twi
+        ty,   ///< Tahitian
+        ug,   ///< Uighur
+        uk,   ///< Ukrainian
+        ur,   ///< Urdu
+        uz,   ///< Uzbek
+        ve,   ///< Venda
+        vi,   ///< Vietnamese
+        vo,   ///< Volapuk
+        wa,   ///< Walloon
+        wo,   ///< Wolof
+        xh,   ///< Xhosa
+        yi,   ///< Yiddish
+        yo,   ///< Yoruba
+        za,   ///< Zhuang
+        zh,   ///< Chinese
+        zu,   ///< Zulu
         NUM_ENTRIES
       };
     }
 
=== modified file 'include/zorba/pregenerated/diagnostic_list.h'
--- include/zorba/pregenerated/diagnostic_list.h	2012-04-24 12:39:38 +0000
+++ include/zorba/pregenerated/diagnostic_list.h	2012-04-24 21:06:20 +0000
@@ -454,6 +454,14 @@
 extern ZORBA_DLL_PUBLIC ZorbaErrorCode ZXQP8402_THESAURUS_ENDIANNESS_MISMATCH;
 
 extern ZORBA_DLL_PUBLIC ZorbaErrorCode ZXQP8403_THESAURUS_DATA_ERROR;
+
+extern ZORBA_DLL_PUBLIC ZorbaErrorCode ZXQP8404_STEM_LANG_NOT_SUPPORTED;
+
+extern ZORBA_DLL_PUBLIC ZorbaErrorCode ZXQP8405_STOP_WORDS_LANG_NOT_SUPPORTED;
+
+extern ZORBA_DLL_PUBLIC ZorbaErrorCode ZXQP8406_THESAURUS_LANG_NOT_SUPPORTED;
+
+extern ZORBA_DLL_PUBLIC ZorbaErrorCode ZXQP8407_TOKENIZER_LANG_NOT_SUPPORTED;
 #endif
 
 extern ZORBA_DLL_PUBLIC ZorbaErrorCode ZXQD0001_PREFIX_NOT_DECLARED;
 
=== modified file 'include/zorba/stemmer.h'
--- include/zorba/stemmer.h	2012-04-24 12:39:38 +0000
+++ include/zorba/stemmer.h	2012-04-24 21:06:20 +0000
@@ -52,6 +52,23 @@
   virtual void destroy() const = 0;
 
   /**
+   * Various properties of this %Stemmer.
+   */
+  struct Properties {
+    /**
+     * The URI that uniquely identifies this %Stemmer.
+     */
+    char const *uri;
+  };
+
+  /**
+   * Gets the Properties of this %Stemmer.
+   *
+   * @param result The Properties to populate.
+   */
+  virtual void properties( Properties *result ) const = 0;
+
+  /**
    * Stems the given word.
    *
    * @param word The word to stem.
@@ -66,7 +83,7 @@
 };
 
 /**
- * A %StemmerProvider, given an language, provies a stemmer for it.
+ * A %StemmerProvider, given a language, provides a Stemmer for it.
  */
 class ZORBA_DLL_PUBLIC StemmerProvider {
 public:
@@ -76,10 +93,12 @@
    * Gets a Stemmer for the given language.
    *
    * @param lang The language to get a Stemmer for.
-   * @return The relevant Stemmer or \c NULL if no stemmer for the given
-   * language is available.
+   * @param s If not \c null, set to point to a Stemmer for \a lang.
+   * @return Returns \c true only if this provider can provide a stemmer for
+   * \a lang.
    */
-  virtual Stemmer::ptr getStemmer( locale::iso639_1::type lang ) const = 0;
+  virtual bool getStemmer( locale::iso639_1::type lang,
+                           Stemmer::ptr *s = 0 ) const = 0;
 };
 
 ///////////////////////////////////////////////////////////////////////////////
 
=== modified file 'include/zorba/thesaurus.h'
--- include/zorba/thesaurus.h	2012-04-24 12:39:38 +0000
+++ include/zorba/thesaurus.h	2012-04-24 21:06:20 +0000
@@ -32,25 +32,13 @@
 ///////////////////////////////////////////////////////////////////////////////
 
 /**
- * Contains additional data for URIMappers and URLResolvers
- * when mapping/resolving a Thesaurus URI.
- */
-class ZORBA_DLL_PUBLIC ThesaurusEntityData : public EntityData {
-public:
-  /**
-   * Gets the language for which a thesaurus is being requested.
-   *
-   * @return said language.
-   */
-  virtual locale::iso639_1::type getLanguage() const = 0;
-};
-
-/**
- * A %Thesaurus is-a Resource for thesaurus implementations.
- */
-class ZORBA_DLL_PUBLIC Thesaurus : public Resource {
-public:
-  typedef std::unique_ptr<Thesaurus,internal::ztd::destroy_delete<Thesaurus> >
+ * A %Thesaurus provides a way to look up related phrases for a given phrase.
+ */
+class ZORBA_DLL_PUBLIC Thesaurus {
+public:
+  typedef std::unique_ptr<
+            Thesaurus const,internal::ztd::destroy_delete<Thesaurus const>
+          >
           ptr;
 
   /**
@@ -88,11 +76,11 @@
    * Destroys this %Thesaurus.
    * This function is called by Zorba when the %Thesaurus is no longer needed.
    *
-   * If your URLResolver dynamically allocates %Thesaurus objects, then the
+   * If your implementation dynamically allocates %Thesaurus objects, then your
    * implementation can simply be (and usually is) <code>delete this</code>.
    *
-   * If your URLResolver returns a pointer to a static %Thesaurus object, then
-   * the implementation should do nothing.
+   * If your implementation returns a pointer to a static %Thesaurus object,
+   * then your implementation should do nothing.
    */
   virtual void destroy() const = 0;
 
@@ -119,6 +107,32 @@
 
 ///////////////////////////////////////////////////////////////////////////////
 
+/**
+ * A %ThesaurusProvider is-a Resource for providing thesauri for a given
+ * language.
+ */
+class ZORBA_DLL_PUBLIC ThesaurusProvider : public Resource {
+public:
+  typedef std::unique_ptr<
+            ThesaurusProvider const,
+            internal::ztd::destroy_delete<ThesaurusProvider const>
+          >
+          ptr;
+
+  /**
+   * Gets a Thesaurus for the given language.
+   *
+   * @param lang The desired language of the thesaurus.
+   * @param t If not \c null, set to point to a Thesaurus for \a lang.
+   * @return Returns \c true only if this provider can provide a thesaurus for
+   * \a lang.
+   */
+  virtual bool getThesaurus( locale::iso639_1::type lang,
+                             Thesaurus::ptr *t = 0 ) const = 0;
+};
+
+///////////////////////////////////////////////////////////////////////////////
+
 } // namespace zorba
 #endif /* ZORBA_NO_FULL_TEXT */
 #endif /* ZORBA_THESAURUS_API_H */
 
=== modified file 'include/zorba/tokenizer.h'
--- include/zorba/tokenizer.h	2012-04-24 12:39:38 +0000
+++ include/zorba/tokenizer.h	2012-04-24 21:06:20 +0000
@@ -18,6 +18,8 @@
 #ifndef ZORBA_TOKENIZER_API_H
 #define ZORBA_TOKENIZER_API_H
 
+#include <vector>
+
 #include <zorba/config.h>
 #include <zorba/locale.h>
 #include <zorba/internal/unique_ptr.h>
@@ -67,8 +69,6 @@
    * A %Callback is called once per token.
    * This is only internally by Zorba.
    * You do not need to derive from this class.
-   * The only thing you need to do is call the callback's \c operator() once
-   * for each token you parse in \c tokenize().
    */
   class Callback {
   public:
@@ -77,19 +77,75 @@
     virtual ~Callback();
 
     /**
+     * This member-function is called whenever an item that is being tokenized
+     * is entered or exited.
+     *
+     * @param item The item being entered or exited.
+     * @param entering If \c true, the item is being entered; if \c false, the
+     * item is being exited.
+     */
+    virtual void item( Item const &item, bool entering );
+
+    /**
      * This member-function is called once per token.
      *
      * @param utf8_s    The UTF-8 token string.  It is not null-terminated.
      * @param utf8_len  The number of bytes in the token string.
+     * @param lang      The language of the token.
      * @param token_no  The token number.  Token numbers start at 0.
      * @param sent_no   The sentence number.  Sentence numbers start at 1.
      * @param para_no   The paragraph number.  Paragraph numbers start at 1.
-     * @param payload   Optional user-defined data.
-     */
-    virtual void operator()( char const *utf8_s, size_type utf8_len,
-                             size_type token_no, size_type sent_no,
-                             size_type para_no, void *payload = 0 ) = 0;
-  };
+     * @param item      The Item this token is from, if any.
+     */
+    virtual void token( char const *utf8_s, size_type utf8_len,
+                        locale::iso639_1::type lang,
+                        size_type token_no, size_type sent_no,
+                        size_type para_no, Item const *item = 0 ) = 0;
+  };
+
+  /////////////////////////////////////////////////////////////////////////////
+
+  /**
+   * Various properties of this %Tokenizer.
+   */
+  struct Properties {
+    typedef std::vector<locale::iso639_1::type> languages_type;
+
+    /**
+     * If \c true, XML comments separate tokens.  For example,
+     * \c net&lt;!----&gt;work would be 2 tokens instead of 1.
+     */
+    bool comments_separate_tokens;
+
+    /**
+     * If \c true, XML elements separate tokens.  For example,
+     * \c &lt;b&gt;B&lt;/b&gt;old would be 2 tokens instead of 1.
+     */
+    bool elements_separate_tokens;
+
+    /**
+     * If \c true, XML processing instructions separate tokens.  For example,
+     * <code>net<?PI pi?>work</code> would be 2 tokens instead of 1.
+     */
+    bool processing_instructions_separate_tokens;
+
+    /**
+     * The set of languages supported.
+     */
+    languages_type languages;
+
+    /**
+     * The URI that uniquely identifies this %Tokenizer.
+     */
+    char const* uri;
+  };
+
+  /**
+   * Gets the Properties of this %Tokenizer.
+   *
+   * @param result The Properties to populate.
+   */
+  virtual void properties( Properties *result ) const = 0;
 
   /////////////////////////////////////////////////////////////////////////////
 
@@ -106,39 +162,6 @@
   virtual void destroy() const = 0;
 
   /**
-   * Trace options for XML elements combined via bitwise-or.
-   */
-  enum ElementTraceOptions {
-    trace_none  = 0x0,  ///< Trace no elements.
-    trace_begin = 0x1,  ///< Trace the beginning of elements.
-    trace_end   = 0x2   ///< Trace the ending of elements.
-  };
-
-  /**
-   * Gets the trace options.  If the value is \c trace_none, then the paragraph
-   * number will be incremented upon entering an XML element; if the value is
-   * anything other than \c trace_none, then the tokenizer assumes
-   * responsibility for incrementing the paragraph number.
-   *
-   * @return Returns said options.
-   */
-  int trace_options() const {
-    return trace_options_;
-  }
-
-  /**
-   * This function is called whenever an XML element is entered during
-   * tokenization.  Note that this function is called only if \c
-   * trace_options() returns non-zero.
-   *
-   * @param qname The element's QName.
-   * @param trace_options The bitwise-or of the trace option(s) in effect for a
-   * particular call.
-   * @see trace_options()
-   */
-  virtual void element( Item const &qname, int trace_options );
-
-  /**
    * Gets this %Tokenizer's associated Numbers.
    *
    * @return Returns said Numbers.
@@ -153,6 +176,16 @@
   Numbers const& numbers() const;
 
   /**
+   * Tokenizes the given node.
+   *
+   * @param node      The node to tokenize.
+   * @param lang      The default language to use.
+   * @param callback  The Callback to call once per token.
+   */
+  void tokenize_node( Item const &node, locale::iso639_1::type lang,
+                      Callback &callback );
+
+  /**
    * Tokenizes the given string.
    *
    * @param utf8_s    The UTF-8 string to tokenize.  It need not be
@@ -162,11 +195,11 @@
    * @param wildcards If \c true, allows XQuery wildcard syntax characters to
    *                  be part of tokens.
    * @param callback  The Callback to call once per token.
-   * @param payload   Optional user-defined data.
+   * @param item      The Item this string is from, if any.
    */
-  virtual void tokenize( char const *utf8_s, size_type utf8_len,
-                         locale::iso639_1::type lang, bool wildcards,
-                         Callback &callback, void *payload = 0 ) = 0;
+  virtual void tokenize_string( char const *utf8_s, size_type utf8_len,
+                                locale::iso639_1::type lang, bool wildcards,
+                                Callback &callback, Item const *item = 0 ) = 0;
 
   /////////////////////////////////////////////////////////////////////////////
 
@@ -175,27 +208,71 @@
    * Constructs a %Tokenizer.
    *
    * @param numbers the Numbers to use.
-   * @param trace_options The bitwise-or of the available trace options, if
-   * any.
    */
-  Tokenizer( Numbers &numbers, int trace_options = trace_none );
+  Tokenizer( Numbers &numbers );
 
   /**
    * Destroys a %Tokenizer.
    */
   virtual ~Tokenizer() = 0;
 
+  /**
+   * Given an element, finds its \c xml:lang attribute, if any, and gets its
+   * value.
+   *
+   * @param element The element to check.
+   * @param lang A pointer to where to put the found language, if any.
+   * @return Returns \c true only if an \c xml:lang attribute is found and the
+   * value is a known language.
+   */
+  bool find_lang_attribute( Item const &element, locale::iso639_1::type *lang );
+
+  /**
+   * This member-function is called whenever an item that is being tokenized is
+   * entered or exited.
+   *
+   * @param item      The item being entered or exited.
+   * @param entering  If \c true, the item is being entered; if \c false, the
+   *                  item is being exited.
+   */
+  virtual void item( Item const &item, bool entering );
+
+  /**
+   * Tokenizes the given node and all of its child nodes, if any.  For each
+   * node, it is required that this function call the item() member function of
+   * both this %Tokenizer and of the Callback twice, once each for entrance and
+   * exit.
+   *
+   * @param node          The node to tokenize.
+   * @param lang          The default language to use.
+   * @param callback      The Callback to call per token.
+   * @param tokenize_acp  If \c true, additionally tokenize all attribute,
+   *                      comment, and processing-instruction nodes encountered;
+   *                      if \c false, skip them.
+   */
+  virtual void tokenize_node_impl( Item const &node,
+                                   locale::iso639_1::type lang,
+                                   Callback &callback, bool tokenize_acp );
+
 private:
-  int trace_options_;
-  Numbers *no_;
+  Numbers *numbers_;
 };
 
+inline Tokenizer::Tokenizer( Numbers &numbers ) : numbers_( &numbers ) {
+}
+
 inline Tokenizer::Numbers& Tokenizer::numbers() {
-  return *no_;
+  return *numbers_;
 }
 
 inline Tokenizer::Numbers const& Tokenizer::numbers() const {
-  return *no_;
+  return *numbers_;
+}
+
+inline void Tokenizer::tokenize_node( Item const &item,
+                                      locale::iso639_1::type lang,
+                                      Callback &callback ) {
+  tokenize_node_impl( item, lang, callback, true );
 }
 
 ///////////////////////////////////////////////////////////////////////////////
@@ -211,11 +288,14 @@
    * Creates a new %Tokenizer.
    *
    * @param lang The language of the text that the tokenizer will tokenize.
-   * @param numbers The Numbers to use.
-   * @return Returns said %Tokenizer.
+   * @param numbers The Numbers to use.  If \c null, \a t is not set.
+   * @param t If not \c null, set to point to a Tokenizer for \a lang.
+   * @return Returns \c true only if this provider can provide a tokenizer for
+   * \a lang.
    */
-  virtual Tokenizer::ptr getTokenizer( locale::iso639_1::type lang,
-                                       Tokenizer::Numbers &numbers ) const = 0;
+  virtual bool getTokenizer( locale::iso639_1::type lang,
+                             Tokenizer::Numbers *numbers = 0,
+                             Tokenizer::ptr *t = 0 ) const = 0;
 };
 
 ///////////////////////////////////////////////////////////////////////////////
 
=== modified file 'include/zorba/uri_resolvers.h'
--- include/zorba/uri_resolvers.h	2012-04-24 12:39:38 +0000
+++ include/zorba/uri_resolvers.h	2012-04-24 21:06:20 +0000
@@ -50,7 +50,8 @@
 class ZORBA_DLL_PUBLIC Resource
 {
 public:
-  typedef std::unique_ptr<Resource,internal::ztd::destroy_delete<Resource> > ptr;
+  typedef std::unique_ptr<Resource,internal::ztd::destroy_delete<Resource> >
+          ptr;
 
   virtual ~Resource() = 0;
 
@@ -172,8 +173,8 @@
    * object itself will be discarded.
    *
    * In any case, if they create a Resource, Zorba will take memory
-   * ownership of the Resource and delete it when it is no longer
-   * needed.
+   * ownership of the Resource and delete it (by calling destroy() on it)
+   * when it is no longer needed.
    */
   virtual Resource* resolveURL(const zorba::String& aUrl,
     EntityData const* aEntityData) = 0;
 
=== modified file 'modules/com/zorba-xquery/www/modules/CMakeLists.txt'
--- modules/com/zorba-xquery/www/modules/CMakeLists.txt	2012-04-24 12:39:38 +0000
+++ modules/com/zorba-xquery/www/modules/CMakeLists.txt	2012-04-24 21:06:20 +0000
@@ -72,6 +72,13 @@
 DECLARE_ZORBA_MODULE(FILE xqdoc.xq VERSION 2.0
   URI "http://www.zorba-xquery.com/modules/xqdoc")
 
+IF(NOT ZORBA_NO_FULL_TEXT)
+  DECLARE_ZORBA_MODULE(FILE full-text.xq VERSION 2.0
+    URI "http://www.zorba-xquery.com/modules/full-text")
+  DECLARE_ZORBA_SCHEMA(FILE full-text.xsd
+    URI "http://www.zorba-xquery.com/modules/full-text")
+ENDIF(NOT ZORBA_NO_FULL_TEXT)
+
 # Subdirectories
 DECLARE_ZORBA_MODULE(FILE converters/base64.xq VERSION 2.0
   URI "http://www.zorba-xquery.com/modules/converters/base64")
 
=== added file 'modules/com/zorba-xquery/www/modules/full-text.xq'
--- modules/com/zorba-xquery/www/modules/full-text.xq	1970-01-01 00:00:00 +0000
+++ modules/com/zorba-xquery/www/modules/full-text.xq	2012-04-24 21:06:20 +0000
@@ -0,0 +1,872 @@
+xquery version "3.0";
+
+(:
+ : Copyright 2006-2011 The FLWOR Foundation.
+ :
+ : Licensed under the Apache License, Version 2.0 (the "License");
+ : you may not use this file except in compliance with the License.
+ : You may obtain a copy of the License at
+ :
+ : http://www.apache.org/licenses/LICENSE-2.0
+ :
+ : Unless required by applicable law or agreed to in writing, software
+ : distributed under the License is distributed on an "AS IS" BASIS,
+ : WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ : See the License for the specific language governing permissions and
+ : limitations under the License.
+ :)
+
+(:===========================================================================:)
+
+(:~
+ : This module provides an XQuery API to full-text functions.
+ : For general information about Zorba's implementation of the
+ : <a href="http://www.w3.org/TR/xpath-full-text-10/">XQuery and XPath Full Text 1.0 specification</a>
+ : as well as instructions for building an installing a thesaurus,
+ : see the <a href="http://www.zorba-xquery.com/html/documentation/latest/zorba/ft_thesaurus">Full Text Thesaurus documentation</a>.
+ : <h2>Notes on languages</h2>
+ : To refer to paricular human languages,
+ : Zorba uses both the
+ : <a href="http://en.wikipedia.org/wiki/ISO_639-1">ISO 639-1</a>
+ : and
+ : <a href="http://en.wikipedia.org/wiki/ISO_639-2">ISO 639-2</a>
+ : languages codes.
+ : Note that Zorba supports only a subset of the
+ : <a href="http://en.wikipedia.org/wiki/List_of_ISO_639-1_codes">complete list of language codes</a>
+ : and not every function supports the same subset.
+ : <p/>
+ : Most functions in this module take a language as a parameter
+ : using the
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>
+ : XML schema data type.
+ : <h2>Notes on stemming</h2>
+ : The <code>stem()</code> functions return the
+ : <a href="http://en.wikipedia.org/wiki/Word_stem">stem</a>
+ : of a word.
+ : In Zorba,
+ : the stem of a word itself, however, is not guaranteed to be a word.
+ : It is best to consider a stem as an opaque byte sequence.
+ : All that is guaranteed about a stem is that,
+ : for a given word,
+ : the stem of that word will always be the same byte sequence.
+ : Hence,
+ : you sould never compare the result of one of the <code>stem()</code>
+ : functions against a non-stemmed string,
+ : for example:
+ : <pre>
+ :  if ( ft:stem( "apples" ) eq "apple" )             ** WRONG **
+ : </pre>
+ : Instead do:
+ : <pre>
+ :  if ( ft:stem( "apples" ) eq ft:stem( "apple" ) )  ** CORRECT **
+ : </pre>
+ : <h2>Notes on the thesaurus</h2>
+ : The <code>thesaurus-lookup()</code> functions have "levels"
+ : and "relationship" parameters.
+ : The values for these are implementation-defined.
+ : Zorba's default implementation uses the
+ : <a href="http://wordnet.princeton.edu/">WordNet lexical database</a>,
+ : version 3.0.
+ : <p/>
+ : In WordNet,
+ : the number of "levels" that two phrases are apart
+ : are how many hierarchical meanings apart they are.
+ : For example,
+ : "canary" is 5 levels away from "vertebrate"
+ : (carary &gt; finch &gt; oscine &gt; passerine &gt; bird &gt; vertebrate).
+ : <p/>
+ : When using the WordNet implementation,
+ : Zorba supports all of the relationships (and their abbreviations)
+ : specified by
+ : <a href="http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=7776">ISO 2788</a>
+ : and
+ : <a href="http://www.niso.org/kst/reports/standards?step=2&amp;gid=&amp;project_key=7cc9b583cb5a62e8c15d3099e0bb46bbae9cf38a">ANSI/NISO Z39.19-2005</a>
+ : with the exceptions of "HN" (history note)
+ : and "X SN" (see scope note for).
+ : These relationships are:
+ :  <table>
+ :    <tr>
+ :      <th>Rel.</th>
+ :      <th>Meaning</th>
+ :      <th>WordNet Rel.</th>
+ :    </tr>
+ :    <tr>
+ :      <td>BT</td>
+ :      <td>broader term</td>
+ :      <td>hypernym</td>
+ :    </tr>
+ :    <tr>
+ :      <td>BTG</td>
+ :      <td>broader term generic</td>
+ :      <td>hypernym</td>
+ :    </tr>
+ :    <tr>
+ :      <td>BTI</td>
+ :      <td>broader term instance</td>
+ :      <td>instance hypernym</td>
+ :    </tr>
+ :    <tr>
+ :      <td>BTP</td>
+ :      <td>broader term partitive</td>
+ :      <td>part meronym</td>
+ :    </tr>
+ :    <tr>
+ :      <td>NT</td>
+ :      <td>narrower term</td>
+ :      <td>hyponym</td>
+ :    </tr>
+ :    <tr>
+ :      <td>NTG</td>
+ :      <td>narrower term generic</td>
+ :      <td>hyponym</td>
+ :    </tr>
+ :    <tr>
+ :      <td>NTI</td>
+ :      <td>narrower term instance</td>
+ :      <td>instance hyponym</td>
+ :    </tr>
+ :    <tr>
+ :      <td>NTP</td>
+ :      <td>narrower term partitive</td>
+ :      <td>part holonym</td>
+ :    </tr>
+ :    <tr>
+ :      <td>RT</td>
+ :      <td>related term</td>
+ :      <td>also see</td>
+ :    </tr>
+ :    <tr>
+ :      <td>SN</td>
+ :      <td>scope note</td>
+ :      <td>n/a</td>
+ :    </tr>
+ :    <tr>
+ :      <td>TT</td>
+ :      <td>top term</td>
+ :      <td>hypernym</td>
+ :    </tr>
+ :    <tr>
+ :      <td>UF</td>
+ :      <td>non-preferred term</td>
+ :      <td>n/a</td>
+ :    </tr>
+ :    <tr>
+ :      <td>USE</td>
+ :      <td>preferred term</td>
+ :      <td>n/a</td>
+ :    </tr>
+ :  </table>
+ : Note that you can specify relationships
+ : either by their abbreviation
+ : or their meaning.
+ : Relationships are case-insensitive.
+ :
+ : In addition to the
+ : <a href="http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=7776">ISO 2788</a>
+ : and
+ : <a href="http://www.niso.org/kst/reports/standards?step=2&amp;gid=&amp;project_key=7cc9b583cb5a62e8c15d3099e0bb46bbae9cf38a">ANSI/NISO Z39.19-2005</a>
+ : relationships,
+ : Zorba also supports all of the relationships offered by WordNet.
+ : These relationships are:
+ :  <table class="ft_rels">
+ :    <tr>
+ :      <th>Relationship</th>
+ :      <th>Meaning</th>
+ :    </tr>
+ :    <tr>
+ :      <td nowrap="nowrap">also see</td>
+ :      <td>
+ :        A word that is related to another,
+ :        e.g., for "varnished" (furniture)
+ :        one should <em>also see</em> "finished."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td>antonym</td>
+ :      <td>
+ :        A word opposite in meaning to another,
+ :        e.g., "light" is an <em>antonym</em> for "heavy."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td>attribute</td>
+ :      <td>
+ :        A noun for which adjectives express values,
+ :        e.g., "weight" is an <em>attribute</em>
+ :        for which the adjectives "light" and "heavy"
+ :        express values.
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td>cause</td>
+ :      <td>
+ :        A verb that causes another,
+ :        e.g., "show" is a <em>cause</em> of "see."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td nowrap="nowrap">derivationally related form</td>
+ :      <td>
+ :        A word that is derived from a root word,
+ :        e.g., "metric" is a <em>derivationally related form</em> of "meter."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td nowrap="nowrap">derived from adjective</td>
+ :      <td>
+ :        An adverb that is derived from an adjective,
+ :        e.g., "correctly" is <em>derived from the adjective</em> "correct."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td>entailment</td>
+ :      <td>
+ :        A verb that presupposes another,
+ :        e.g., "snoring" <em>entails</em> "sleeping."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td>hypernym</td>
+ :      <td>
+ :        A word with a broad meaning that more specific words fall under,
+ :        e.g., "meal" is a <em>hypernym</em> of "breakfast."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td>hyponym</td>
+ :      <td>
+ :        A word of more specific meaning than a general term applicable to it,
+ :        e.g., "breakfast" is a <em>hyponym</em> of "meal."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td nowrap="nowrap">instance hypernym</td>
+ :      <td>
+ :        A word that denotes a category of some specific instance,
+ :        e.g., "author" is an <em>instance hypernym</em> of "Asimov."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td nowrap="nowrap">instance hyponym</td>
+ :      <td>
+ :        A term that donotes a specific instance of some general category,
+ :        e.g., "Asimov" is an <em>instance hyponym</em> of "author."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td nowrap="nowrap">member holonym</td>
+ :      <td>
+ :        A word that denotes a collection of individuals,
+ :        e.g., "faculty" is a <em>member holonym</em> of "professor."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td nowrap="nowrap">member meronym</td>
+ :      <td>
+ :        A word that denotes a member of a larger group,
+ :        e.g., a "person" is a <em>member meronym</em> of a "crowd."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td nowrap="nowrap">part holonym</td>
+ :      <td>
+ :        A word that denotes a larger whole comprised of some part,
+ :        e.g., "car" is a <em>part holonym</em> of "engine."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td nowrap="nowrap">part meronym</td>
+ :      <td>
+ :        A word that denotes a part of a larger whole,
+ :        e.g., an "engine" is <em>part meronym</em> of a "car."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td nowrap="nowrap">participle of verb</td>
+ :      <td>
+ :        An adjective that is the participle of some verb,
+ :        e.g., "breaking" is the <em>participle of the verb</em> "break."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td>pertainym</td>
+ :      <td>
+ :        An adjective that classifies its noun,
+ :        e.g., "musical" is a <em>pertainym</em> in "musical instrument."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td nowrap="nowrap">similar to</td>
+ :      <td>
+ :        Similar, though not necessarily interchangeable, adjectives.
+ :        For example, "shiny" is <em>similar to</em> "bright",
+ :        but they have subtle differences.
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td nowrap="nowrap">substance holonym</td>
+ :      <td>
+ :        A word that denotes a larger whole containing some constituent
+ :        substance, e.g., "bread" is a <em>substance holonym</em> of "flour."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td nowrap="nowrap">substance meronym</td>
+ :      <td>
+ :        A word that denotes a constituant substance of some larger whole,
+ :        e.g., "flour" is a <em>substance meronym</em> of "bread."
+ :      </td>
+ :    </tr>
+ :    <tr>
+ :      <td nowrap="nowrap">verb group</td>
+ :      <td>
+ :        A verb that is a member of a group of similar verbs,
+ :        e.g., "live" is in the <em>verb group</em>
+ :        of "dwell", "live", "inhabit", etc.
+ :      </td>
+ :    </tr>
+ :  </table>
+ : <h2>Notes on tokenization</h2>
+ : For general information about Zorba's implementation of tokenization,
+ : including what constitutes a token,
+ : see the <a href="http://www.zorba-xquery.com/html/documentation/latest/zorba/ft_tokenizer">Full Text Tokenizer</a> documentation.
+ :)
+
+(:===========================================================================:)
+
+module namespace ft = "http://www.zorba-xquery.com/modules/full-text";
+
+import schema namespace ft-schema =
+  "http://www.zorba-xquery.com/modules/full-text";
+
+declare namespace err = "http://www.w3.org/2005/xqt-errors";
+declare namespace zerr = "http://www.zorba-xquery.com/errors";
+
+declare namespace ver = "http://www.zorba-xquery.com/options/versioning";
+declare option ver:module-version "2.0";
+
+(:===========================================================================:)
+
+(:~
+ : Predeclared constant for the Danish
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-da as xs:language := xs:language("da");
+
+(:~
+ : Predeclared constant for the German
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-de as xs:language := xs:language("de");
+
+(:~
+ : Predeclared constant for the English
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-en as xs:language := xs:language("en");
+
+(:~
+ : Predeclared constant for the Spanish
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-es as xs:language := xs:language("es");
+
+(:~
+ : Predeclared constant for the Finnish
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-fi as xs:language := xs:language("fi");
+
+(:~
+ : Predeclared constant for the French
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-fr as xs:language := xs:language("fr");
+
+(:~
+ : Predeclared constant for the Hungarian
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-hu as xs:language := xs:language("hu");
+
+(:~
+ : Predeclared constant for the Italian
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-it as xs:language := xs:language("it");
+
+(:~
+ : Predeclared constant for the Dutch
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-nl as xs:language := xs:language("nl");
+
+(:~
+ : Predeclared constant for the Norwegian
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-no as xs:language := xs:language("no");
+
+(:~
+ : Predeclared constant for the Portuguese
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-pt as xs:language := xs:language("pt");
+
+(:~
+ : Predeclared constant for the Romanian
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-ro as xs:language := xs:language("ro");
+
+(:~
+ : Predeclared constant for the Russian
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-ru as xs:language := xs:language("ru");
+
+(:~
+ : Predeclared constant for the Swedish
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-sv as xs:language := xs:language("sv");
+
+(:~
+ : Predeclared constant for the Turkish
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language"><code>xs:language</code></a>.
+ :)
+declare variable $ft:lang-tr as xs:language := xs:language("tr");
+
+(:===========================================================================:)
+
+(:~
+ : Gets the current
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>:
+ : either the langauge specified by the
+ : <code><a href="http://www.w3.org/TR/xpath-full-text-10/#doc-xquery10-FTOptionDecl">declare ft-option using</a>
+ : <a href="http://www.w3.org/TR/xpath-full-text-10/#ftlanguageoption">language</a></code>
+ : statement (if any)
+ : or the one returned by <code>ft:host-lang()</code> (if none).
+ :
+ : @return said language.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-current-lang-true-1.xq
+ :)
+declare function ft:current-lang()
+  as xs:language external;
+
+(:~
+ : Gets the host's current
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>.
+ : The "host" is the computer on which Zorba is running.
+ : The host's current language is obtained as follows:
+ :  <ul>
+ :    <li>
+ :      For *nix systems:
+ :      <ol>
+ :        <li>
+ :          If <a ref="http://www.cplusplus.com/reference/clibrary/clocale/setlocale/"><code>setlocale</code>(3)</a> returns non-null,
+ :          the language corresponding to that locale is used.
+ :        </li>
+ :        <li>
+ :          Else, if the <code>LANG</code> environment variable is set,
+ :          that language is ued.
+ :        </li>
+ :        <li>
+ :          Otherwise, there is no default language.
+ :        </li>
+ :      </ol>
+ :    </li>
+ :    <li>
+ :      For Windows systems,
+ :      the language corresponding to the locale returned by the
+ :      <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/dd318101(v=vs.85).aspx"><code>GetLocaleInfo()</code></a>
+ :      function is used.
+ :    </li>
+ :  </ul>
+ :
+ : @return said language.
+ :)
+declare function ft:host-lang()
+  as xs:language external;
+
+(:~
+ : Checks whether the given
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : is supported for stemming.
+ :
+ : @param $lang The language to check.
+ : @return <code>true</code> only if the language is supported.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stem-lang-es-supported-true.xq
+ :)
+declare function ft:is-stem-lang-supported( $lang as xs:language )
+  as xs:boolean external;
+
+(:~
+ : Checks whether the given word is a stop-word.
+ :
+ : @param $word The word to check.
+ : @param $lang The
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : of <code>$word</code>.
+ : @return <code>true</code> only if <code>$word</code> is a stop-word.
+ : @error zerr:ZXQP8405 if <code>$lang</code> is not supported for stop-words.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-1.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-3.xq
+ :)
+declare function ft:is-stop-word( $word as xs:string, $lang as xs:language )
+  as xs:boolean external;
+
+(:~
+ : Checks whether the given word is a stop-word.
+ :
+ : @param $word The word to check.
+ : The word's <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : is assumed to be the one returned by <code>ft:current-lang()</code>.
+ : @return <code>true</code> only if <code>$word</code> is a stop-word.
+ : @error err:FTST0009 if <code>ft:current-lang()</code> is not supported in
+ : general.
+ : @error zerr:ZXQP8405 if <code>ft:current-lang()</code> is not supported for
+ : stop-words specifically.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-2.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-true-4.xq
+ :)
+declare function ft:is-stop-word( $word as xs:string )
+  as xs:boolean external;
+
+(:~
+ : Checks whether the given
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : is supported for stop words.
+ :
+ : @param $lang The language to check.
+ : @return <code>true</code> only if the language is supported.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-en-supported-true.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-supported-false-1.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-stop-word-lang-supported-false-2.xq
+ :)
+declare function ft:is-stop-word-lang-supported( $lang as xs:language )
+  as xs:boolean external;
+
+(:~
+ : Checks whether the given
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : is supported for look-up using the default thesaurus.
+ :
+ : @param $lang The language to check.
+ : @return <code>true</code> only if the language is supported.
+ :)
+declare function ft:is-thesaurus-lang-supported( $lang as xs:language )
+  as xs:boolean external;
+
+(:~
+ : Checks whether the given
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : is supported for look-up using the thesaurus specified by the given URI.
+ :
+ : @param $uri The URI specifying the thesaurus to use.
+ : @param $lang The language to check.
+ : @return <code>true</code> only if the language is supported.
+ : @error err:FTST0018 if <code>$uri</code> refers to a thesaurus
+ : that is not found in the statically known thesauri.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-is-thesaurus-lang-supported-true-1.xq
+ :)
+declare function ft:is-thesaurus-lang-supported( $uri as xs:string,
+                                                 $lang as xs:language )
+  as xs:boolean external;
+
+(:~
+ : Checks whether the given
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : is supported for tokenization.
+ :
+ : @param $lang The language to check.
+ : @return <code>true</code> only if the language is supported.
+ :)
+declare function ft:is-tokenizer-lang-supported( $lang as xs:language )
+  as xs:boolean external;
+
+(:~
+ : Stems the given word.
+ :
+ : @param $word The word to stem.
+ : @param $lang The
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : of <code>$word</code>.
+ : @return the stem of <code>$word</code>.
+ : @error err:FTST0009 if <code>$lang</code> is not supported in general.
+ : @error zerr:ZXQP8404 if <code>$lang</code> is not supported for stemming
+ : specifically.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-1.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-2.xq
+ :)
+declare function ft:stem( $word as xs:string, $lang as xs:language )
+  as xs:string external;
+
+(:~
+ : Stems the given word.
+ :
+ : @param $word The word to stem.
+ : The word's <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : is assumed to be the one returned by <code>ft:current-lang()</code>.
+ : @return the stem of <code>$word</code>.
+ : @error err:FTST0009 if <code>ft:current-lang()</code> is not supported in
+ : general.
+ : @error zerr:ZXQP8404 if <code>ft:current-lang()</code> is not supported for
+ : stemming specifically.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-3.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-stem-4.xq
+ :)
+declare function ft:stem( $word as xs:string )
+  as xs:string external;
+
+(:~
+ : Strips all diacritical marks from all characters.
+ :
+ : @param $string The string to strip diacritical marks from.
+ : @return <code>$string</code> with diacritical marks stripped.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-strip-diacritics-1.xq
+ :)
+declare function ft:strip-diacritics( $string as xs:string )
+  as xs:string external;
+
+(:~
+ : Looks-up the given phrase in the default thesaurus.
+ :
+ : @param $phrase The phrase to look up.
+ : The phrase's
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : is assumed to be the one returned by <code>ft:current-lang()</code>.
+ : @return the original and related phrases.
+ : @error err:FTST0009 if <code>ft:current-lang()</code> is not supported in
+ : general.
+ : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
+ : by the currently running version of Zorba.
+ : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
+ : that of the CPU on which Zorba is currently running.
+ : @error zerr:ZXQP8403 if there was an error reading the thesaurus data.
+ : @error zerr:ZXQP8406 if <code>ft:current-lang()</code> is not supported for
+ : thesaurus look-up specifically.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-1.xq
+ :)
+declare function ft:thesaurus-lookup( $phrase as xs:string )
+  as xs:string+ external;
+
+(:~
+ : Looks-up the given phrase in the thesaurus specified by the given URI.
+ :
+ : @param $uri The URI specifying the thesaurus to use.
+ : @param $phrase The phrase to look up.
+ : @param $lang The
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : of <code>$phrase</code>.
+ : @return the original and related phrases.
+ : @error err:FTST0009 if <code>$lang</code> is not supported in general.
+ : @error err:FTST0018 if <code>$uri</code> refers to a thesaurus
+ : that is not found in the statically known thesauri.
+ : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
+ : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
+ : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
+ : by the currently running version of Zorba.
+ : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
+ : that of the CPU on which Zorba is currently running.
+ : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
+ : @error zerr:ZXQP8406 if <code>$lang</code> is not supported for thesaurus
+ : look-up specifically.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-2.xq
+ :)
+declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
+                                      $lang as xs:language )
+  as xs:string+ external;
+
+(:~
+ : Looks-up the given phrase in a thesaurus.
+ :
+ : @param $uri The URI specifying the thesaurus to use.
+ : @param $phrase The phrase to look up.
+ : The phrase's
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : is assumed to be the one the one returned by <code>ft:current-lang()</code>.
+ : @return the original and related phrases.
+ : @error err:FTST0009 if <code>ft:current-lang()</code> is unsupported in
+ : general.
+ : @error err:FTST0018 if <code>$uri</code> refers to a thesaurus
+ : that is not found in the statically known thesauri.
+ : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
+ : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
+ : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
+ : by the currently running version of Zorba.
+ : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
+ : that of the CPU on which Zorba is currently running.
+ : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
+ : @error zerr:ZXQP8406 if <code>ft:current-lang()</code> is not supported for
+ : thesaurus look-up specifically.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-3.xq
+ :)
+declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string )
+  as xs:string+ external;
+
+(:~
+ : Looks-up the given phrase in a thesaurus.
+ :
+ : @param $uri The URI specifying the thesaurus to use.
+ : @param $phrase The phrase to look up.
+ : @param $lang The
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : of <code>$phrase</code>.
+ : @param $relationship The relationship the results are to have to
+ : <code>$phrase</code>.
+ : @return the original and related phrases.
+ : @error err:FTST0018 if <code>$uri</code> refers to a thesaurus
+ : that is not found in the statically known thesauri.
+ : @error err:FTST0009 if <code>$lang</code> is not supported in general.
+ : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
+ : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
+ : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
+ : by the currently running version of Zorba.
+ : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
+ : that of the CPU on which Zorba is currently running.
+ : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
+ : @error zerr:ZXQP8406 if <code>$lang</code> is not supported for thesaurus
+ : look-up specifically.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-4.xq
+ :)
+declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
+                                      $lang as xs:language,
+                                      $relationship as xs:string )
+  as xs:string+ external;
+
+(:~
+ : Looks-up the given phrase in a thesaurus.
+ :
+ : @param $uri The URI specifying the thesaurus to use.
+ : @param $phrase The phrase to look up.
+ : @param $lang The
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : of <code>$phrase</code>.
+ : @param $relationship The relationship the results are to have to
+ : <code>$phrase</code>.
+ : @param $level-least The minimum number of levels within the thesaurus to be
+ : travers$ed.
+ : @param $level-most The maximum number of levels within the thesaurus to be
+ : traversed.
+ : @return the original and related phrases.
+ : @error err:FOCA0003 if either <code>$level-least</code> or
+ : <code>$level-most</code> is either negative or too large.
+ : @error err:FTST0018 if <code>$uri</code> refers to a thesaurus
+ : that is not found in the statically known thesauri.
+ : @error err:FTST0009 if <code>$lang</code> is not supported in general.
+ : @error zerr:ZOSE0001 if the thesaurus data file could not be found.
+ : @error zerr:ZOSE0002 if the thesaurus data file is not a plain file.
+ : @error zerr:ZXQP8401 if the thesaurus data file's version is not supported
+ : by the currently running version of Zorba.
+ : @error zerr:ZXQP8402 if the thesaurus data file's endianness does not match
+ : that of the CPU on which Zorba is currently running.
+ : @error zerr:ZXQP8403 if there was an error reading the thesaurus data file.
+ : @error zerr:ZXQP8406 if <code>$lang</code> is not supported for thesaurus
+ : look-up specifically.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-thesaurus-lookup-5.xq
+ :)
+declare function ft:thesaurus-lookup( $uri as xs:string, $phrase as xs:string,
+                                      $lang as xs:language,
+                                      $relationship as xs:string,
+                                      $level-least as xs:integer,
+                                      $level-most as xs:integer )
+  as xs:string+ external;
+
+(:~
+ : Tokenizes the given document.
+ :
+ : @param $node The node to tokenize.
+ : @param $lang The default
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : of <code>$node</code>.
+ : @return a (possibly empty) sequence of tokens.
+ : @error err:FTST0009 if <code>$lang</code> is not supported in general.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-1.xq
+ :)
+declare function ft:tokenize( $node as node(), $lang as xs:language )
+  as element(ft-schema:token)* external;
+
+(:~
+ : Tokenizes the given document.
+ :
+ : @param $node The node to tokenize.
+ : The document's default
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : is assumed to be the one returned by <code>ft:current-lang()</code>.
+ : @return a (possibly empty) sequence of tokens.
+ : @error err:FTST0009 if <code>ft:current-lang()</code> is not supported in
+ : general.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-2.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-3.xq
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-4.xq
+ :)
+declare function ft:tokenize( $node as node() )
+  as element(ft-schema:token)* external;
+
+(:~
+ : Tokenizes the given string.
+ :
+ : @param $string The string to tokenize.
+ : @param $lang The default
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : of <code>$string</code>.
+ : @return a (possibly empty) sequence of tokens.
+ : @error err:FTST0009 if <code>$lang</code> is not supported in general.
+ : @error zerr:ZXQP8407 if <code>$lang</code> is not supported for
+ : tokenization specifically.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-string-1.xq
+ :)
+declare function ft:tokenize-string( $string as xs:string,
+                                     $lang as xs:language )
+  as xs:string* external;
+
+(:~
+ : Tokenizes the given string.
+ :
+ : @param $string The string to tokenize.
+ : The string's default
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : is assumed to be the one returned by <code>ft:current-lang()</code>.
+ : @return a (possibly empty) sequence of tokens.
+ : @error err:FTST0009 if <code>ft:current-lang()</code> is not supported in
+ : general.
+ : @error zerr:ZXQP8407 if <code>ft:current_lang()</code> is not supported for
+ : tokenization specifically.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenize-string-2.xq
+ :)
+declare function ft:tokenize-string( $string as xs:string )
+  as xs:string* external;
+
+(:~
+ : Gets properties of the tokenizer for the given
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>.
+ :
+ : @param $lang The langauage of the tokenizer to get the properties of.
+ : @return said properties.
+ : @error err:FTST0009 if <code>$lang</code> is not supported in general.
+ : @error zerr:ZXQP8407 if <code>$lang</code> is not supported for
+ : tokenization specifically.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenizer-properties-1.xq
+ :)
+declare function ft:tokenizer-properties( $lang as xs:language )
+  as element(ft-schema:tokenizer-properties) external;
+
+(:~
+ : Gets properties of the tokenizer for the
+ : <a href="http://www.w3.org/TR/xmlschema-2/#language">language</a>
+ : returned by <code>ft:current-lang()</code>.
+ :
+ : @return said properties.
+ : @error err:FTST0009 if <code>ft:current-lang()</code> is not supported in
+ : general.
+ : @error zerr:ZXQP8407 if <code>ft:current_lang()</code> is not supported for
+ : tokenization specifically.
+ : @example test/rbkt/Queries/zorba/fulltext/ft-module-tokenizer-properties-2.xq
+ :)
+declare function ft:tokenizer-properties()
+  as element(ft-schema:tokenizer-properties) external;
+
+(:===========================================================================:)
+
+(: vim:set et sw=2 ts=2: :)
 
=== added file 'modules/com/zorba-xquery/www/modules/full-text.xsd'
--- modules/com/zorba-xquery/www/modules/full-text.xsd	1970-01-01 00:00:00 +0000
+++ modules/com/zorba-xquery/www/modules/full-text.xsd	2012-04-24 21:06:20 +0000
@@ -0,0 +1,134 @@
+<?xml version="1.0"?>
+<!--
+ ! Copyright 2006-2011 The FLWOR Foundation.
+ ! 
+ ! Licensed under the Apache License, Version 2.0 (the "License");
+ ! you may not use this file except in compliance with the License.
+ ! You may obtain a copy of the License at
+ ! 
+ ! http://www.apache.org/licenses/LICENSE-2.0
+ ! 
+ ! Unless required by applicable law or agreed to in writing, software
+ ! distributed under the License is distributed on an "AS IS" BASIS,
+ ! WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ! See the License for the specific language governing permissions and
+ ! limitations under the License.
+-->
+
+<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
+  targetNamespace="http://www.zorba-xquery.com/modules/full-text"
+  xmlns="http://www.zorba-xquery.com/modules/full-text"
+  elementFormDefault="qualified"
+  attributeFormDefault="unqualified">
+
+  <!--======================================================================-->
+
+  <xs:element name="compare-options">
+    <xs:complexType>
+      <xs:attributeGroup ref="compare-attributes"/>
+    </xs:complexType>
+  </xs:element>
+
+  <xs:attributeGroup name="compare-attributes">
+    <xs:attribute name="case" type="sensitivity" default="insensitive"/>
+    <xs:attribute name="diacritics" type="sensitivity" default="insensitive"/>
+    <xs:attribute name="stem" type="yes-no-both" default="no"/>
+  </xs:attributeGroup>
+
+  <xs:simpleType name="sensitivity">
+    <xs:restriction base="xs:string">
+      <xs:enumeration value="insensitive"/>
+      <xs:enumeration value="sensitive"/>
+      <xs:enumeration value="both"/>
+    </xs:restriction>
+  </xs:simpleType>
+
+  <xs:simpleType name="yes-no-both">
+    <xs:restriction base="xs:string">
+      <xs:enumeration value="yes"/>
+      <xs:enumeration value="no"/>
+      <xs:enumeration value="both"/>
+    </xs:restriction>
+  </xs:simpleType>
+
+  <xs:complexType name="boolean-value">
+    <xs:attribute name="value" type="xs:boolean" use="required"/>
+  </xs:complexType>
+
+  <!--======================================================================-->
+
+  <xs:element name="token">
+    <xs:complexType>
+
+      <!-- The language of the token. -->
+      <xs:attribute name="lang" type="xs:language"/>
+
+      <!-- The sentence number. -->
+      <xs:attribute name="sentence" type="xs:nonNegativeInteger" use="required"/>
+
+      <!-- The paragraph number. -->
+      <xs:attribute name="paragraph" type="xs:nonNegativeInteger" use="required"/>
+
+      <!-- The token string value. -->
+      <xs:attribute name="value" type="xs:string" use="required"/>
+
+      <!--
+       ! A reference to the originating node.  If the token occurred within an
+       ! element, the reference refers to the text node.  If the token occurred
+       ! within an attribute, the reference refers to the attribute node.
+      -->
+      <xs:attribute name="node-ref" type="xs:anyURI"/>
+
+    </xs:complexType>
+  </xs:element>
+
+  <!--======================================================================-->
+
+  <xs:element name="tokenizer-properties">
+    <xs:complexType>
+      <xs:all>
+
+        <!--
+         ! If true, XML comments separate tokens.  (No example can be provided
+         ! here because it is illegal to nest an XML comment inside an XML
+         ! comment.)
+        -->
+        <xs:element name="comments-separate-tokens" type="boolean-value"/>
+
+        <!--
+         ! If true, XML elements separate tokens.  For example,
+         ! <b>B</b>old would be 2 tokens instead of 1.
+        -->
+        <xs:element name="elements-separate-tokens" type="boolean-value"/>
+
+        <!--
+         ! If true, XML processing instructions separate tokens.  For example,
+         ! net<?PI pi?>work would be 2 tokens instead of 1.
+        -->
+        <xs:element name="processing-instructions-separate-tokens" type="boolean-value"/>
+
+        <!--
+         ! The list of languages that the tokenizer can tokenize.
+        -->
+        <xs:element name="supported-languages">
+          <xs:complexType>
+            <xs:sequence>
+              <xs:element name="lang" type="xs:language" maxOccurs="unbounded"/>
+            </xs:sequence>
+          </xs:complexType>
+        </xs:element>
+
+      </xs:all>
+
+      <!--
+       !  The tokenizer's identifying URI.
+      -->
+      <xs:attribute name="uri" type="xs:anyURI"/>
+
+    </xs:complexType>
+  </xs:element>
+
+  <!--======================================================================-->
+
+</xs:schema>
+<!-- vim:set et sw=2 ts=2: -->
 
=== modified file 'modules/com/zorba-xquery/www/modules/http-client.xq.src/http_request_handler.cpp'
--- modules/com/zorba-xquery/www/modules/http-client.xq.src/http_request_handler.cpp	2012-04-24 12:39:38 +0000
+++ modules/com/zorba-xquery/www/modules/http-client.xq.src/http_request_handler.cpp	2012-04-24 21:06:20 +0000
@@ -39,7 +39,6 @@
       theSerStream(NULL),
       thePost(NULL),
       theLast(NULL),
-      theLastSerializerOptions(NULL),
       theIsHeadRequest(false)
   {
     theHeaderLists.push_back(NULL);
@@ -260,6 +259,7 @@
   void HttpRequestHandler::cleanUpBody()
   {
     delete theSerStream;
+    theSerStream = 0;
     theLastBodyHadContent = false;
   }
 
 
=== modified file 'modules/com/zorba-xquery/www/modules/pregenerated/errors.xq'
--- modules/com/zorba-xquery/www/modules/pregenerated/errors.xq	2012-04-24 12:39:38 +0000
+++ modules/com/zorba-xquery/www/modules/pregenerated/errors.xq	2012-04-24 21:06:20 +0000
@@ -188,6 +188,7 @@
 
 (:~
  :
+ : The thesaurus data file's endianness does not match that of the CPU.
  : 
 :)
 declare variable $zerr:ZXQP8402 as xs:QName := fn:QName($zerr:NS, "zerr:ZXQP8402");
@@ -201,6 +202,22 @@
 
 (:~
 :)
+declare variable $zerr:ZXQP8404 as xs:QName := fn:QName($zerr:NS, "zerr:ZXQP8404");
+
+(:~
+:)
+declare variable $zerr:ZXQP8405 as xs:QName := fn:QName($zerr:NS, "zerr:ZXQP8405");
+
+(:~
+:)
+declare variable $zerr:ZXQP8406 as xs:QName := fn:QName($zerr:NS, "zerr:ZXQP8406");
+
+(:~
+:)
+declare variable $zerr:ZXQP8407 as xs:QName := fn:QName($zerr:NS, "zerr:ZXQP8407");
+
+(:~
+:)
 declare variable $zerr:ZXQD0001 as xs:QName := fn:QName($zerr:NS, "zerr:ZXQD0001");
 
 (:~
 
=== modified file 'modules/com/zorba-xquery/www/modules/xqdoc2xhtml/index.xq'
--- modules/com/zorba-xquery/www/modules/xqdoc2xhtml/index.xq	2012-04-24 12:39:38 +0000
+++ modules/com/zorba-xquery/www/modules/xqdoc2xhtml/index.xq	2012-04-24 21:06:20 +0000
@@ -839,9 +839,7 @@
     if(fn:matches($specLine, "Args:")) then
       let $arg_split := fn:substring-after($specLine, "-x")
       return
-      if(fn:string-length($arg_split) eq 0) then
-        fn:error($err:UE008, fn:concat("Unknown Args: in spec file for example <", $exampleSource,"> .
-        Add the example input and expected output by hand in the example, in a commentary that should also include the word 'output'."))
+      if(fn:string-length($arg_split) eq 0) then string-join($specLines, " ")
       else
         let $var_value := fn:tokenize($arg_split, "=")
         let $var_name := fn:normalize-space(fn:replace($var_value[1], ":$", ""))
 
=== modified file 'scripts/zt-wn-get'
--- scripts/zt-wn-get	2012-04-24 12:39:38 +0000
+++ scripts/zt-wn-get	2012-04-24 21:06:20 +0000
@@ -22,7 +22,7 @@
   echo 'Arguments: [--workdir <workdir>] [--builddir <builddir>]'
   echo '           [--thesaurusurl <thesaurusurl>]'
   echo '           <zorba_repository>'
-  echo '<zorba_repository> is the top-level SVN working copy.'
+  echo '<zorba_repository> is the top-level BZR working copy.'
   echo '<workdir> is a temp directory to download and unzip XQTS (default: /tmp).'
   echo '<builddir> is the directory Zorba has been built in'
   echo '           (default: <zorba_repository>/build)'
@@ -71,8 +71,8 @@
 echo Build dir is at $BUILD
 
 # Compile thesaurus to binary format
-mkdir -p $BUILD/test/rbkt/thesauri
-THESAURUS_DEST="$BUILD/test/rbkt/thesauri/wordnet-en.zth"
+mkdir -p $BUILD/LIB_PATH/edu/princeton/wordnet
+THESAURUS_DEST="$BUILD/LIB_PATH/edu/princeton/wordnet/wordnet-en.zth"
 echo "Compiling thesaurus to $THESAURUS_DEST..."
 untar_dir=`mktemp -d "$WORK/thesaurus.XXXXXX"`
 cd "$untar_dir"
 
=== modified file 'src/api/CMakeLists.txt'
--- src/api/CMakeLists.txt	2012-04-24 12:39:38 +0000
+++ src/api/CMakeLists.txt	2012-04-24 21:06:20 +0000
@@ -62,8 +62,9 @@
 IF (NOT ZORBA_NO_FULL_TEXT)
   LIST(APPEND API_SRCS
     stemmer.cpp
-    stemmer_wrapper.cpp
-    thesaurus.cpp)
+    stemmer_wrappers.cpp
+    thesaurus.cpp
+    thesaurus_wrappers.cpp)
 ENDIF (NOT ZORBA_NO_FULL_TEXT)
 
 ADD_SRC_SUBFOLDER(API_SRCS serialization API_SERIALIZATION_SRCS)
 
=== modified file 'src/api/staticcontextimpl.cpp'
--- src/api/staticcontextimpl.cpp	2012-04-24 12:39:38 +0000
+++ src/api/staticcontextimpl.cpp	2012-04-24 21:06:20 +0000
@@ -42,8 +42,8 @@
 #include "context/static_context.h"
 #include "context/static_context_consts.h"
 #ifndef ZORBA_NO_FULL_TEXT
-#include "context/stemmer_wrappers.h"
-#include "context/thesaurus_wrappers.h"
+#include "stemmer_wrappers.h"
+#include "thesaurus_wrappers.h"
 #endif /* ZORBA_NO_FULL_TEXT */
 #include "uri_resolver_wrappers.h"
 
@@ -65,7 +65,6 @@
 
 namespace zorba {
 
-
 /*******************************************************************************
   Create a StaticContextImpl obj as well as an internal static_context obj S.
   S is created as a child of the zorba root sctx. This constructor is used
 
=== renamed file 'src/api/stemmer_wrapper.cpp' => 'src/api/stemmer_wrappers.cpp'
--- src/api/stemmer_wrapper.cpp	2012-04-24 12:39:38 +0000
+++ src/api/stemmer_wrappers.cpp	2012-04-24 21:06:20 +0000
@@ -23,7 +23,7 @@
 #include "diagnostics/assert.h"
 #include "util/cxx_util.h"
 
-#include "stemmer_wrapper.h"
+#include "stemmer_wrappers.h"
 
 using namespace zorba::locale;
 
@@ -32,8 +32,8 @@
 
 ///////////////////////////////////////////////////////////////////////////////
 
-StemmerWrapper::StemmerWrapper( zorba::Stemmer::ptr p ) :
-  api_stemmer_( std::move( p ) )
+StemmerWrapper::StemmerWrapper( zorba::Stemmer::ptr api_stemmer ) :
+  api_stemmer_( std::move( api_stemmer ) )
 {
   ZORBA_ASSERT( api_stemmer_.get() );
 }
@@ -42,6 +42,12 @@
   api_stemmer_.release()->destroy();
 }
 
+void StemmerWrapper::properties( Properties *props ) const {
+  zorba::Stemmer::Properties api_props;
+  api_stemmer_->properties( &api_props );
+  props->uri = api_props.uri;
+}
+
 void StemmerWrapper::stem( zstring const &word, iso639_1::type lang,
                            zstring *result ) const {
   String const api_word( Unmarshaller::newString( word ) );
@@ -52,19 +58,22 @@
 ///////////////////////////////////////////////////////////////////////////////
 
 StemmerProviderWrapper::
-StemmerProviderWrapper( zorba::StemmerProvider const *p ) :
-  api_stemmer_provider_( p )
+StemmerProviderWrapper( zorba::StemmerProvider const *api_stemmer_provider ) :
+  api_stemmer_provider_( api_stemmer_provider )
 {
   ZORBA_ASSERT( api_stemmer_provider_ );
 }
 
-Stemmer::ptr
-StemmerProviderWrapper::get_stemmer( iso639_1::type lang ) const {
-  zorba::Stemmer::ptr p( api_stemmer_provider_->getStemmer( lang ) );
-  Stemmer::ptr result;
-  if ( p.get() )
-    result.reset( new StemmerWrapper( std::move( p ) ) );
-  return std::move( result );
+bool StemmerProviderWrapper::getStemmer( iso639_1::type lang,
+                                         Stemmer::ptr *result ) const {
+  zorba::Stemmer::ptr api_ptr;
+  zorba::Stemmer::ptr *const api_ptr_ptr = result ? &api_ptr : nullptr;
+  if ( api_stemmer_provider_->getStemmer( lang, api_ptr_ptr ) ) {
+    if ( result )
+      result->reset( new StemmerWrapper( std::move( api_ptr ) ) );
+    return true;
+  }
+  return false;
 }
 
 ///////////////////////////////////////////////////////////////////////////////
 
=== renamed file 'src/api/stemmer_wrapper.h' => 'src/api/stemmer_wrappers.h'
--- src/api/stemmer_wrapper.h	2012-04-24 12:39:38 +0000
+++ src/api/stemmer_wrappers.h	2012-04-24 21:06:20 +0000
@@ -35,6 +35,7 @@
 
   // inherited
   void destroy() const;
+  void properties( Properties* ) const;
   void stem( zstring const &word, locale::iso639_1::type lang,
              zstring *result ) const;
 private:
@@ -50,7 +51,7 @@
   }
 
   // inherited
-  Stemmer::ptr get_stemmer( locale::iso639_1::type lang ) const;
+  bool getStemmer( locale::iso639_1::type, Stemmer::ptr* = 0 ) const;
 private:
   zorba::StemmerProvider const *const api_stemmer_provider_;
 };
 
=== modified file 'src/api/thesaurus.cpp'
--- src/api/thesaurus.cpp	2012-04-24 12:39:38 +0000
+++ src/api/thesaurus.cpp	2012-04-24 21:06:20 +0000
@@ -25,9 +25,11 @@
   // out-of-line since it's virtual
 }
 
-//Thesaurus::iterator::~iterator() {
-//  // out-of-line since it's virtual
-//}
+#if 0
+Thesaurus::iterator::~iterator() {
+  // out-of-line since it's virtual
+}
+#endif
 
 ///////////////////////////////////////////////////////////////////////////////
 
 
=== renamed file 'src/context/thesaurus_wrappers.cpp' => 'src/api/thesaurus_wrappers.cpp'
--- src/context/thesaurus_wrappers.cpp	2012-04-24 12:39:38 +0000
+++ src/api/thesaurus_wrappers.cpp	2012-04-24 21:06:20 +0000
@@ -87,6 +87,27 @@
 
 ///////////////////////////////////////////////////////////////////////////////
 
+ThesaurusProviderWrapper::
+ThesaurusProviderWrapper( zorba::ThesaurusProvider const *p ) :
+  api_thesaurus_provider_( p )
+{
+  ZORBA_ASSERT( api_thesaurus_provider_ );
+}
+
+bool ThesaurusProviderWrapper::getThesaurus( iso639_1::type lang,
+                                             Thesaurus::ptr *result ) const {
+  zorba::Thesaurus::ptr api_ptr;
+  zorba::Thesaurus::ptr *const api_ptr_ptr = result ? &api_ptr : nullptr;
+  if ( api_thesaurus_provider_->getThesaurus( lang, api_ptr_ptr ) ) {
+    if ( result )
+      result->reset( new ThesaurusWrapper( std::move( api_ptr ) ) );
+    return true;
+  }
+  return false;
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
 } // namespace internal
 } // namespace zorba
 
 
=== renamed file 'src/context/thesaurus_wrappers.h' => 'src/api/thesaurus_wrappers.h'
--- src/context/thesaurus_wrappers.h	2012-04-24 12:39:38 +0000
+++ src/api/thesaurus_wrappers.h	2012-04-24 21:06:20 +0000
@@ -22,6 +22,7 @@
 #ifndef ZORBA_NO_FULL_TEXT
 
 #include <zorba/thesaurus.h>
+
 #include "runtime/full_text/thesaurus.h"
 
 namespace zorba {
@@ -54,6 +55,17 @@
   zorba::Thesaurus::ptr api_thesaurus_;
 };
 
+class ThesaurusProviderWrapper : public ThesaurusProvider {
+public:
+  ThesaurusProviderWrapper( zorba::ThesaurusProvider const* );
+
+  // inherited
+  bool getThesaurus( locale::iso639_1::type, Thesaurus::ptr* ) const;
+
+private:
+  zorba::ThesaurusProvider::ptr const api_thesaurus_provider_;
+};
+
 ///////////////////////////////////////////////////////////////////////////////
 
 } // namespace internal
 
=== modified file 'src/api/uri_resolver_wrappers.cpp'
--- src/api/uri_resolver_wrappers.cpp	2012-04-24 12:39:38 +0000
+++ src/api/uri_resolver_wrappers.cpp	2012-04-24 21:06:20 +0000
@@ -15,24 +15,20 @@
  */
 #include "stdafx.h"
 
+#include <zorba/thesaurus.h>
+
+#include "runtime/full_text/thesaurus.h"
+
+#include "thesaurus_wrappers.h"
+#include "unmarshaller.h"
 #include "uri_resolver_wrappers.h"
 #include "uriresolverimpl.h"
-#include "unmarshaller.h"
-#include <zorba/thesaurus.h>
-#include <runtime/full_text/thesaurus.h>
-#include <context/thesaurus_wrappers.h>
 
 namespace zorba
 {
   // "Convenience" class for passing an internal EntityData object to
-  // external mappers/resolvers. This can serve as a plain EntityData or
-  // a ThesaurusEntityData. However, when there's another EntityData subclass
-  // in future, this won't work as EntityData becomes an ambiguous base class...
-#ifndef ZORBA_NO_FULL_TEXT
-  class EntityDataWrapper : public ThesaurusEntityData
-#else
+  // external mappers/resolvers.
   class EntityDataWrapper : public EntityData
-#endif /* ZORBA_NO_FULL_TEXT */
   {
   public:
     static EntityDataWrapper const* create(internal::EntityData const* aData) {
@@ -45,12 +41,7 @@
         return new EntityDataWrapper(EntityData::SCHEMA);
 #ifndef ZORBA_NO_FULL_TEXT
       case internal::EntityData::THESAURUS:
-      {
-        EntityDataWrapper* retval = new EntityDataWrapper(EntityData::THESAURUS);
-        retval->theThesaurusLang =
-            dynamic_cast<const internal::ThesaurusEntityData*>(aData)->getLanguage();
-        return retval;
-      }
+        return new EntityDataWrapper(EntityData::THESAURUS);
       case internal::EntityData::STOP_WORDS:
         return new EntityDataWrapper(EntityData::STOP_WORDS);
 #endif /* ZORBA_NO_FULL_TEXT */
@@ -67,21 +58,12 @@
       return theKind;
     }
 
-#ifndef ZORBA_NO_FULL_TEXT
-    virtual zorba::locale::iso639_1::type getLanguage() const {
-      return theThesaurusLang;
-    }
-#endif /* ZORBA_NO_FULL_TEXT */
-
   private:
     EntityDataWrapper(EntityData::Kind aKind)
       : theKind(aKind)
     {}
 
     EntityData::Kind const theKind;
-#ifndef ZORBA_NO_FULL_TEXT
-    zorba::locale::iso639_1::type theThesaurusLang;
-#endif /* ZORBA_NO_FULL_TEXT */
   };
 
   URIMapperWrapper::URIMapperWrapper(zorba::URIMapper& aUserMapper)
@@ -169,13 +151,13 @@
       }
 #ifndef ZORBA_NO_FULL_TEXT
       else {
-        Thesaurus* lUserThesaurus = dynamic_cast<Thesaurus*>(lUserPtr.get());
-        if (lUserThesaurus != NULL) {
-          // Here we pass memory ownership of the actual Thesaurus to the
-          // internal ThesaurusWrapper.
-          lRetval = new internal::ThesaurusWrapper
-              (Thesaurus::ptr(lUserThesaurus));
-          lUserPtr.release();
+        ThesaurusProvider* lUserThesaurusProvider =
+          dynamic_cast<ThesaurusProvider*>(lUserPtr.get());
+        if (lUserThesaurusProvider) {
+          // Here we pass memory ownership of the actual ThesaurusProvider to
+          // the internal ThesaurusWrapper.
+          lRetval = new internal::ThesaurusProviderWrapper
+              (lUserThesaurusProvider);
         }
         else {
           assert(false);
 
=== modified file 'src/api/xmldatamanagerimpl.cpp'
--- src/api/xmldatamanagerimpl.cpp	2012-04-24 12:39:38 +0000
+++ src/api/xmldatamanagerimpl.cpp	2012-04-24 21:06:20 +0000
@@ -47,7 +47,7 @@
 #include "runtime/util/flowctl_exception.h"
 
 #ifndef ZORBA_NO_FULL_TEXT
-#include "stemmer_wrapper.h"
+#include "stemmer_wrappers.h"
 #endif /* ZORBA_NO_FULL_TEXT */
 
 namespace zorba {
 
=== modified file 'src/api/xmldatamanagerimpl.h'
--- src/api/xmldatamanagerimpl.h	2012-04-24 12:39:38 +0000
+++ src/api/xmldatamanagerimpl.h	2012-04-24 21:06:20 +0000
@@ -27,7 +27,7 @@
 #include "util/singleton.h"
 
 #ifndef ZORBA_NO_FULL_TEXT
-#include "stemmer_wrapper.h"
+#include "stemmer_wrappers.h"
 #endif /* ZORBA_NO_FULL_TEXT */
 
 namespace zorba {
 
=== modified file 'src/compiler/codegen/plan_visitor.cpp'
--- src/compiler/codegen/plan_visitor.cpp	2012-04-24 12:39:38 +0000
+++ src/compiler/codegen/plan_visitor.cpp	2012-04-24 21:06:20 +0000
@@ -250,7 +250,7 @@
 class plan_ftnode_visitor : public ftnode_visitor 
 {
 public:
-  typedef std::list<PlanIter_t> PlanIter_list_t;
+  typedef std::vector<PlanIter_t> PlanIter_list_t;
 
   plan_ftnode_visitor( plan_visitor* v ) : plan_visitor_( v ) { }
 
 
=== modified file 'src/compiler/expression/expr_put.cpp'
--- src/compiler/expression/expr_put.cpp	2012-04-24 12:39:38 +0000
+++ src/compiler/expression/expr_put.cpp	2012-04-24 21:06:20 +0000
@@ -41,6 +41,7 @@
 #include "compiler/expression/function_item_expr.h"
 #include "compiler/parser/parse_constants.h"
 
+#include "diagnostics/assert.h"
 #include "functions/function.h"
 #include "functions/udf.h"
 
 
=== modified file 'src/compiler/translator/translator.cpp'
--- src/compiler/translator/translator.cpp	2012-04-24 12:39:38 +0000
+++ src/compiler/translator/translator.cpp	2012-04-24 21:06:20 +0000
@@ -68,6 +68,7 @@
 #include "functions/signature.h"
 #include "functions/udf.h"
 #include "functions/external_function.h"
+#include "functions/func_ft_module.h"
 
 #include "annotations/annotations.h"
 
@@ -859,7 +860,7 @@
 {
   ZORBA_ASSERT(count >= 0);
 
-  ftnode *n = NULL;
+  ftnode *n = nullptr;
   while ( count-- > 0 )
   {
     ZORBA_FATAL( !theFTNodeStack.empty(), "" );
@@ -3294,6 +3295,41 @@
                                     qnameItem->getLocalName())));
         }
 
+#ifndef ZORBA_NO_FULL_TEXT
+        if (qnameItem->getNamespace() == static_context::ZORBA_FULL_TEXT_FN_NS &&
+            (qnameItem->getLocalName() == "tokenizer-properties" ||
+             qnameItem->getLocalName() == "tokenize"))
+        {
+          FunctionConsts::FunctionKind kind;
+
+          if (qnameItem->getLocalName() == "tokenizer-properties")
+          {
+            assert(numParams <= 1);
+
+            if (numParams == 1)
+              kind = FunctionConsts::FULL_TEXT_TOKENIZER_PROPERTIES_1;
+            else
+              kind = FunctionConsts::FULL_TEXT_TOKENIZER_PROPERTIES_0;
+
+            f = new full_text_tokenizer_properties(f->getSignature(), kind);
+          }
+          else 
+          {
+            assert(numParams == 1 || numParams == 2);
+
+            if (numParams == 2)
+              kind = FunctionConsts::FULL_TEXT_TOKENIZE_2;
+            else
+              kind = FunctionConsts::FULL_TEXT_TOKENIZE_1;
+
+            f = new full_text_tokenize(f->getSignature(), kind);
+          }
+
+          f->setStaticContext(theRootSctx);
+          bind_fn(f, numParams, loc);
+        }
+#endif /* ZORBA_NO_FULL_TEXT */
+
         f->setAnnotations(theAnnotations);
         theAnnotations = NULL; // important to reset
 
@@ -12512,7 +12548,7 @@
 {
   TRACE_VISIT ();
 #ifndef ZORBA_NO_FULL_TEXT
-  push_ftstack( NULL ); // sentinel
+  push_ftstack( nullptr ); // sentinel
 #endif /* ZORBA_NO_FULL_TEXT */
   return no_state;
 }
@@ -12756,7 +12792,7 @@
 void *begin_visit (const FTMildNot& v) {
   TRACE_VISIT ();
 #ifndef ZORBA_NO_FULL_TEXT
-  push_ftstack( NULL ); // sentinel
+  push_ftstack( nullptr ); // sentinel
 #endif /* ZORBA_NO_FULL_TEXT */
   return no_state;
 }
@@ -12799,7 +12835,7 @@
 void *begin_visit (const FTOr& v) {
   TRACE_VISIT ();
 #ifndef ZORBA_NO_FULL_TEXT
-  push_ftstack( NULL ); // sentinel
+  push_ftstack( nullptr ); // sentinel
 #endif /* ZORBA_NO_FULL_TEXT */
   return no_state;
 }
@@ -13058,7 +13094,7 @@
     levels = dynamic_cast<ftrange*>( pop_ftstack() );
     ZORBA_ASSERT( levels );
   } else
-    levels = NULL;
+    levels = nullptr;
 
   ftthesaurus_id *const tid = new ftthesaurus_id(
     loc, v.get_uri(), v.get_relationship(), levels
@@ -13070,7 +13106,7 @@
 void *begin_visit (const FTThesaurusOption& v) {
   TRACE_VISIT ();
 #ifndef ZORBA_NO_FULL_TEXT
-  push_ftstack( NULL ); // sentinel
+  push_ftstack( nullptr ); // sentinel
 #endif /* ZORBA_NO_FULL_TEXT */
   return no_state;
 }
@@ -13078,10 +13114,8 @@
 void end_visit (const FTThesaurusOption& v, void* /*visit_state*/) {
   TRACE_VISIT_OUT ();
 #ifndef ZORBA_NO_FULL_TEXT
-  ftthesaurus_id *default_tid = NULL;
-  if ( v.includes_default() ) {
-    default_tid = new ftthesaurus_id( loc, "##default" );
-  }
+  ftthesaurus_id *const default_tid = v.includes_default() ?
+    new ftthesaurus_id( loc, "##default" ) : nullptr;
 
   ftthesaurus_option::thesaurus_id_list_t list;
   while ( true ) {
 
=== modified file 'src/context/CMakeLists.txt'
--- src/context/CMakeLists.txt	2012-04-24 12:39:38 +0000
+++ src/context/CMakeLists.txt	2012-04-24 21:06:20 +0000
@@ -32,11 +32,6 @@
     features.cpp
     )
 
-IF (NOT ZORBA_NO_FULL_TEXT)
-  LIST(APPEND CONTEXT_SRCS
-    thesaurus_wrappers.cpp)
-ENDIF (NOT ZORBA_NO_FULL_TEXT)
-
 SET(CONTEXT_BUILD_SRCS
   ${CMAKE_CURRENT_BINARY_DIR}/context/root_static_context_init.cpp
   )
 
=== modified file 'src/context/default_url_resolvers.cpp'
--- src/context/default_url_resolvers.cpp	2012-04-24 12:39:38 +0000
+++ src/context/default_url_resolvers.cpp	2012-04-24 21:06:20 +0000
@@ -17,6 +17,7 @@
 
 
 #include "context/default_url_resolvers.h"
+#include "util/cxx_util.h"
 #include "util/uri_util.h"
 #include "util/http_util.h"
 #include "util/fs_util.h"
@@ -41,8 +42,15 @@
 HTTPURLResolver::resolveURL
 (zstring const& aUrl, EntityData const* aEntityData)
 {
-  if (aEntityData->getKind() == EntityData::COLLECTION)
-    return NULL;
+  switch ( aEntityData->getKind() ) {
+    case EntityData::COLLECTION:
+#ifndef ZORBA_NO_FULL_TEXT
+    case EntityData::THESAURUS:
+#endif /* ZORBA_NO_FULL_TEXT */
+      return nullptr;
+    default:
+      break;
+  }
 
   uri::scheme lScheme = uri::get_scheme(aUrl);
   switch (lScheme) {
@@ -82,8 +90,15 @@
 FileURLResolver::resolveURL
 (zstring const& aUrl, EntityData const* aEntityData)
 {
-  if (aEntityData->getKind() == EntityData::COLLECTION)
-    return NULL;
+  switch ( aEntityData->getKind() ) {
+    case EntityData::COLLECTION:
+#ifndef ZORBA_NO_FULL_TEXT
+    case EntityData::THESAURUS:
+#endif /* ZORBA_NO_FULL_TEXT */
+      return nullptr;
+    default:
+      break;
+  }
 
   uri::scheme lScheme = uri::get_scheme(aUrl);
   if (lScheme != uri::file) {
@@ -111,7 +126,6 @@
 {
   if (aEntityData->getKind() != EntityData::COLLECTION)
     return NULL;
-
   store::Item_t lName;
   GENV_STORE.getItemFactory()->createQName(lName, aUrl.c_str(), "", "zorba-internal-name-for-w3c-collections");
   store::Collection_t lColl = GENV_STORE.getCollection(lName.getp(), true);
 
=== modified file 'src/context/static_context.cpp'
--- src/context/static_context.cpp	2012-04-24 12:39:38 +0000
+++ src/context/static_context.cpp	2012-04-24 21:06:20 +0000
@@ -378,11 +378,16 @@
 static_context::ZORBA_XML_FN_NS =
 "http://www.zorba-xquery.com/modules/xml";
 
+#ifndef ZORBA_NO_FULL_TEXT
+const char*
+static_context::ZORBA_FULL_TEXT_FN_NS =
+"http://www.zorba-xquery.com/modules/full-text";
+#endif /* ZORBA_NO_FULL_TEXT */
+
 const char*
 static_context::ZORBA_XML_FN_OPTIONS_NS =
 "http://www.zorba-xquery.com/modules/xml-options";
 
-
 /***************************************************************************//**
   Target namespaces of zorba reserved modules
 ********************************************************************************/
@@ -451,8 +456,11 @@
             ns == ZORBA_JSON_FN_NS ||
             ns == ZORBA_FETCH_FN_NS ||
             ns == ZORBA_NODE_FN_NS ||
+#ifndef ZORBA_NO_FULL_TEXT
+            ns == ZORBA_FULL_TEXT_FN_NS ||
+#endif /* ZORBA_NO_FULL_TEXT */
             ns == ZORBA_XML_FN_NS);
-  }
+  } 
   else if (ns == W3C_FN_NS || ns == XQUERY_MATH_FN_NS)
   {
     return true;
@@ -1585,7 +1593,7 @@
     std::auto_ptr<internal::Resource>& oResource,
     zstring& oErrorMessage) const
 {
-  oErrorMessage = "";
+  oErrorMessage.clear();
 
   // Iterate through all candidate URLs...
   for (std::vector<zstring>::iterator url = aUrls.begin();
@@ -1621,7 +1629,7 @@
         }
         catch (const std::exception& e)
         {
-          if (oErrorMessage == "")
+          if (oErrorMessage.empty()) 
           {
             // Really no point in saving anything more than the first message
             oErrorMessage = e.what();
 
=== modified file 'src/context/static_context.h'
--- src/context/static_context.h	2012-04-24 12:39:38 +0000
+++ src/context/static_context.h	2012-04-24 21:06:20 +0000
@@ -471,6 +471,9 @@
   static const char* ZORBA_FETCH_FN_NS;
   static const char* ZORBA_NODE_FN_NS;
   static const char* ZORBA_XML_FN_NS;
+#ifndef ZORBA_NO_FULL_TEXT
+  static const char* ZORBA_FULL_TEXT_FN_NS;
+#endif /* ZORBA_NO_FULL_TEXT */
   static const char* ZORBA_XML_FN_OPTIONS_NS;
 
   // Namespaces of virtual modules declaring zorba builtin functions
 
=== removed file 'src/context/stemmer_wrappers.cpp'
--- src/context/stemmer_wrappers.cpp	2012-04-24 12:39:38 +0000
+++ src/context/stemmer_wrappers.cpp	1970-01-01 00:00:00 +0000
@@ -1,74 +0,0 @@
-/*
- * Copyright 2006-2008 The FLWOR Foundation.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- * 
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-#include "stdafx.h"
-
-#include <zorba/config.h>
-
-#ifndef ZORBA_NO_FULL_TEXT
-
-#include "api/unmarshaller.h"
-#include "diagnostics/assert.h"
-#include "util/cxx_util.h"
-
-#include "stemmer_wrappers.h"
-
-using namespace zorba::locale;
-
-namespace zorba {
-namespace internal {
-
-///////////////////////////////////////////////////////////////////////////////
-
-StemmerWrapper::StemmerWrapper( zorba::Stemmer const *s ) :
-  api_stemmer_( s )
-{
-  ZORBA_ASSERT( api_stemmer_ );
-}
-
-void StemmerWrapper::stem( zstring const &word, iso639_1::type lang,
-                           zstring *result ) const {
-  String const api_word( Unmarshaller::newString( word ) );
-  String api_result( Unmarshaller::newString( *result ) );
-  api_stemmer_->stem( api_word, lang, &api_result );
-}
-
-///////////////////////////////////////////////////////////////////////////////
-
-StemmerProviderWrapper::
-StemmerProviderWrapper( zorba::StemmerProvider const *p ) :
-  api_stemmer_provider_( p )
-{
-  ZORBA_ASSERT( api_stemmer_provider_ );
-}
-
-Stemmer const*
-StemmerProviderWrapper::get_stemmer( iso639_1::type lang ) const {
-  zorba::Stemmer const *const s = api_stemmer_provider_->getStemmer( lang );
-  return s ? new StemmerWrapper( s ) : nullptr;
-}
-
-///////////////////////////////////////////////////////////////////////////////
-
-} // namespace internal
-} // namespace zorba
-
-#endif /* ZORBA_NO_FULL_TEXT */
-/*
- * Local variables:
- * mode: c++
- * End:
- */
-/* vim:set et sw=2 ts=2: */
 
=== removed file 'src/context/stemmer_wrappers.h'
--- src/context/stemmer_wrappers.h	2012-04-24 12:39:38 +0000
+++ src/context/stemmer_wrappers.h	1970-01-01 00:00:00 +0000
@@ -1,63 +0,0 @@
-/*
- * Copyright 2006-2008 The FLWOR Foundation.
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- * 
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-#pragma once
-#ifndef ZORBA_STEMMER_WRAPPERS_H
-#define ZORBA_STEMMER_WRAPPERS_H
-
-#include <zorba/config.h>
-
-#if 0
-#ifndef ZORBA_NO_FULL_TEXT
-
-#include <zorba/stemmer.h>
-#include "zorbautils/stemmer.h"
-
-namespace zorba {
-namespace internal {
-
-///////////////////////////////////////////////////////////////////////////////
-
-class StemmerWrapper : public Stemmer {
-public:
-  StemmerWrapper( zorba::Stemmer const *api_stemmer );
-  void stem( zstring const &word, locale::iso639_1::type lang,
-             zstring *result ) const;
-private:
-  zorba::Stemmer const *const api_stemmer_;
-};
-
-class StemmerProviderWrapper : public StemmerProvider {
-public:
-  StemmerProviderWrapper( zorba::StemmerProvider const *p );
-  Stemmer const* get_stemmer( locale::iso639_1::type lang ) const;
-private:
-  zorba::StemmerProvider const *const api_stemmer_provider_;
-};
-
-///////////////////////////////////////////////////////////////////////////////
-
-} // namespace internal
-} // namespace zorba
-
-#endif /* ZORBA_NO_FULL_TEXT */
-#endif
-#endif /* ZORBA_STEMMER_WRAPPERS_H */
-/*
- * Local variables:
- * mode: c++
- * End:
- */
-/* vim:set et sw=2 ts=2: */
 
=== modified file 'src/context/uri_resolver.cpp'
--- src/context/uri_resolver.cpp	2012-04-24 12:39:38 +0000
+++ src/context/uri_resolver.cpp	2012-04-24 21:06:20 +0000
@@ -117,19 +117,6 @@
   {
   }
 
-#ifndef ZORBA_NO_FULL_TEXT
-  ThesaurusEntityData::ThesaurusEntityData(locale::iso639_1::type aLang)
-    : EntityData(EntityData::THESAURUS),
-      theLang(aLang)
-  {
-  }
-
-  locale::iso639_1::type ThesaurusEntityData::getLanguage() const
-  {
-    return theLang;
-  }
-#endif /* ZORBA_NO_FULL_TEXT */
-
 /*************
  * URIMapper is an abstract class, but we have to define its vtbl and
  * base destructor somewhere.
 
=== modified file 'src/context/uri_resolver.h'
--- src/context/uri_resolver.h	2012-04-24 12:39:38 +0000
+++ src/context/uri_resolver.h	2012-04-24 21:06:20 +0000
@@ -55,21 +55,21 @@
   /**
    * @brief Return the URL used to load this Resource.
    */
-  zstring getUrl() { return theUrl; }
+  zstring const& getUrl() const { return theUrl; }
 
   virtual ~Resource() = 0;
 
-  protected:
+protected:
 
   Resource();
 
-  private:
+private:
 
   /**
    * Used by static_context to populate the URL.
    */
+  void setUrl(zstring const &aUrl) { theUrl = aUrl; }
   friend class zorba::static_context;
-  void setUrl(zstring aUrl) { theUrl = aUrl; }
 
   zstring theUrl;
 };
@@ -193,25 +193,6 @@
   Kind const theKind;
 };
 
-#ifndef ZORBA_NO_FULL_TEXT
-/**
- * @brief The class containing additional data for URIMappers and URLResolvers
- * when mapping/resolving a Thesaurus URI.
- */
-class ThesaurusEntityData : public EntityData
-{
-public:
-  ThesaurusEntityData(locale::iso639_1::type aLang);
-  /**
-   * @brief Return the language for which a thesaurus is being requested.
-   */
-  virtual locale::iso639_1::type getLanguage() const;
-
-private:
-  locale::iso639_1::type const theLang;
-};
-#endif /* ZORBA_NO_FULL_TEXT */
-
 /**
  * @brief Interface for URL resolving.
  *
 
=== modified file 'src/diagnostics/assert.cpp'
--- src/diagnostics/assert.cpp	2012-04-24 12:39:38 +0000
+++ src/diagnostics/assert.cpp	2012-04-24 21:06:20 +0000
@@ -68,7 +68,7 @@
     file, 
     line, 
     zerr::ZXQP0002_ASSERT_FAILED, 
-    ( msg ? ERROR_PARAMS( condition, msg ) : ERROR_PARAMS( condition ))
+    ( msg ? ERROR_PARAMS( condition, msg ) : ERROR_PARAMS( condition ) )
   );
 }
 
 
=== modified file 'src/diagnostics/assert.h'
--- src/diagnostics/assert.h	2012-04-24 12:39:38 +0000
+++ src/diagnostics/assert.h	2012-04-24 21:06:20 +0000
@@ -20,6 +20,10 @@
 #ifndef ZORBA_ASSERT_H
 #define ZORBA_ASSERT_H
 
+#include <sstream>
+
+#include "util/cxx_util.h"
+
 namespace zorba {
 
 /**
@@ -35,7 +39,7 @@
 void assertion_failed( char const *condition,
                        char const *file, 
                        int line, 
-                       char const *msg = 0);
+                       char const *msg = nullptr );
 
 /**
  * Zorba version of the standard assert(3) macro.
 
=== modified file 'src/diagnostics/diagnostic_en.xml'
--- src/diagnostics/diagnostic_en.xml	2012-04-24 12:39:38 +0000
+++ src/diagnostics/diagnostic_en.xml	2012-04-24 21:06:20 +0000
@@ -1746,7 +1746,7 @@
     <diagnostic code="ZXQP8401" name="THESAURUS_VERSION_MISMATCH"
       if="!defined(ZORBA_NO_FULL_TEXT)">
       <comment>
-       The version of the thesaurus is not the expected version.
+        The version of the thesaurus is not the expected version.
       </comment>
      <value>"$1": wrong WordNet file version; should be "$2"</value>
     </diagnostic>
@@ -1754,19 +1754,39 @@
     <diagnostic code="ZXQP8402" name="THESAURUS_ENDIANNESS_MISMATCH"
       if="!defined(ZORBA_NO_FULL_TEXT)">
       <comment>
+        The thesaurus data file's endianness does not match that of the CPU.
       </comment>
      <value>thesaurus data endianness does not match CPU</value>
-       The thesaurus data file's endianness does not match that of the CPU.
     </diagnostic>
 
     <diagnostic code="ZXQP8403" name="THESAURUS_DATA_ERROR"
       if="!defined(ZORBA_NO_FULL_TEXT)">
       <comment>
-       The thesaurus data contains an unexpected value.
+        The thesaurus data contains an unexpected value.
       </comment>
      <value>thesaurus data error${: 1}</value>
     </diagnostic>
 
+    <diagnostic code="ZXQP8404" name="STEM_LANG_NOT_SUPPORTED"
+      if="!defined(ZORBA_NO_FULL_TEXT)">
+      <value>"$1": langauge not supported for stemming</value>
+    </diagnostic>
+
+    <diagnostic code="ZXQP8405" name="STOP_WORDS_LANG_NOT_SUPPORTED"
+      if="!defined(ZORBA_NO_FULL_TEXT)">
+      <value>"$1": langauge not supported for stop-words</value>
+    </diagnostic>
+
+    <diagnostic code="ZXQP8406" name="THESAURUS_LANG_NOT_SUPPORTED"
+      if="!defined(ZORBA_NO_FULL_TEXT)">
+      <value>"$1": langauge not supported for thesaurus</value>
+    </diagnostic>
+
+    <diagnostic code="ZXQP8407" name="TOKENIZER_LANG_NOT_SUPPORTED"
+      if="!defined(ZORBA_NO_FULL_TEXT)">
+      <value>"$1": langauge not supported for tokenizer</value>
+    </diagnostic>
+
     <diagnostic code="ZXQD0001" name="PREFIX_NOT_DECLARED">
       <value>"$1": prefix not declared when calling function "$2" from $3</value>
     </diagnostic>
 
=== modified file 'src/diagnostics/pregenerated/diagnostic_list.cpp'
--- src/diagnostics/pregenerated/diagnostic_list.cpp	2012-04-24 12:39:38 +0000
+++ src/diagnostics/pregenerated/diagnostic_list.cpp	2012-04-24 21:06:20 +0000
@@ -660,6 +660,18 @@
 
 
 ZorbaErrorCode ZXQP8403_THESAURUS_DATA_ERROR( "ZXQP8403" );
+
+
+ZorbaErrorCode ZXQP8404_STEM_LANG_NOT_SUPPORTED( "ZXQP8404" );
+
+
+ZorbaErrorCode ZXQP8405_STOP_WORDS_LANG_NOT_SUPPORTED( "ZXQP8405" );
+
+
+ZorbaErrorCode ZXQP8406_THESAURUS_LANG_NOT_SUPPORTED( "ZXQP8406" );
+
+
+ZorbaErrorCode ZXQP8407_TOKENIZER_LANG_NOT_SUPPORTED( "ZXQP8407" );
 #endif
 
 
 
=== modified file 'src/diagnostics/pregenerated/dict_en.cpp'
--- src/diagnostics/pregenerated/dict_en.cpp	2012-04-24 12:39:38 +0000
+++ src/diagnostics/pregenerated/dict_en.cpp	2012-04-24 21:06:20 +0000
@@ -434,6 +434,18 @@
 #if !defined(ZORBA_NO_FULL_TEXT)
   { "ZXQP8403", "thesaurus data error${: 1}" },
 #endif
+#if !defined(ZORBA_NO_FULL_TEXT)
+  { "ZXQP8404", "\"$1\": langauge not supported for stemming" },
+#endif
+#if !defined(ZORBA_NO_FULL_TEXT)
+  { "ZXQP8405", "\"$1\": langauge not supported for stop-words" },
+#endif
+#if !defined(ZORBA_NO_FULL_TEXT)
+  { "ZXQP8406", "\"$1\": langauge not supported for thesaurus" },
+#endif
+#if !defined(ZORBA_NO_FULL_TEXT)
+  { "ZXQP8407", "\"$1\": langauge not supported for tokenizer" },
+#endif
   { "~AllMatchesHasExcludes", "AllMatches contains StringExclude" },
   { "~AlreadySpecified", "already specified" },
   { "~ArithOpNotDefinedBetween_23", "arithmetic operation not defined between types \"$2\" and \"$3\"" },
 
=== modified file 'src/functions/CMakeLists.txt'
--- src/functions/CMakeLists.txt	2012-04-24 12:39:38 +0000
+++ src/functions/CMakeLists.txt	2012-04-24 21:06:20 +0000
@@ -83,3 +83,7 @@
     func_apply.cpp
     func_serialize_impl.cpp
 )
+
+IF (NOT ZORBA_NO_FULL_TEXT)
+  LIST(APPEND FUNCTIONS_SRCS func_ft_module_impl.cpp)
+ENDIF (NOT ZORBA_NO_FULL_TEXT)
 
=== modified file 'src/functions/external_function.cpp'
--- src/functions/external_function.cpp	2012-04-24 12:39:38 +0000
+++ src/functions/external_function.cpp	2012-04-24 21:06:20 +0000
@@ -45,12 +45,12 @@
   :
   function(sig, FunctionConsts::FN_UNKNOWN),
   theLoc(loc),
-  theModuleSctx(modSctx),
   theNamespace(ns),
   theScriptingKind(scriptingType),
   theImpl(impl)
 {
   resetFlag(FunctionConsts::isBuiltin);
+  theModuleSctx = modSctx;
 }
 
 
@@ -62,7 +62,6 @@
   zorba::serialization::serialize_baseclass(ar, (function*)this);
 
   ar & theLoc;
-  ar & theModuleSctx;
   ar & theNamespace;
   ar & theScriptingKind;
 
 
=== added file 'src/functions/func_ft_module_impl.cpp'
--- src/functions/func_ft_module_impl.cpp	1970-01-01 00:00:00 +0000
+++ src/functions/func_ft_module_impl.cpp	2012-04-24 21:06:20 +0000
@@ -0,0 +1,110 @@
+/*
+ * Copyright 2006-2008 The FLWOR Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "stdafx.h"
+
+#include "functions/func_ft_module.h"
+
+#include "runtime/full_text/ft_module.h"
+
+#define FT_MODULE_NS "http://www.zorba-xquery.com/modules/full-text"
+
+namespace zorba {
+
+///////////////////////////////////////////////////////////////////////////////
+
+void populate_context_ft_module_impl( static_context *sctx ) {
+
+  xqtref_t tokenize_return_type =
+    GENV_TYPESYSTEM.create_node_type(
+      store::StoreConsts::elementNode,
+      createQName( FT_MODULE_NS, "", "token" ),
+      NULL,
+      TypeConstants::QUANT_STAR,
+      false,
+      false
+    );
+  {
+    DECL_WITH_KIND( sctx, full_text_tokenize,
+      (createQName( FT_MODULE_NS, "", "tokenize"),
+        GENV_TYPESYSTEM.ANY_NODE_TYPE_ONE,
+        tokenize_return_type),
+      FunctionConsts::FULL_TEXT_TOKENIZE_1
+    );
+  }
+  {
+    DECL_WITH_KIND( sctx, full_text_tokenize,
+      (createQName( FT_MODULE_NS, "", "tokenize"),
+        GENV_TYPESYSTEM.ANY_NODE_TYPE_ONE,
+        GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
+        tokenize_return_type),
+      FunctionConsts::FULL_TEXT_TOKENIZE_2
+    );
+  }
+
+  xqtref_t tokenizer_properties_return_type =
+    GENV_TYPESYSTEM.create_node_type(
+      store::StoreConsts::elementNode,
+      createQName( FT_MODULE_NS, "", "tokenizer-properties" ),
+      NULL,
+      TypeConstants::QUANT_ONE,
+      false,
+      false
+    );
+  {
+    DECL_WITH_KIND( sctx, full_text_tokenizer_properties,
+      (createQName( FT_MODULE_NS, "", "tokenizer-properties"),
+        tokenizer_properties_return_type),
+      FunctionConsts::FULL_TEXT_TOKENIZER_PROPERTIES_0
+    );
+  }
+  {
+    DECL_WITH_KIND( sctx, full_text_tokenizer_properties,
+      (createQName( FT_MODULE_NS, "", "tokenizer-properties"),
+        GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE,
+        tokenizer_properties_return_type),
+      FunctionConsts::FULL_TEXT_TOKENIZER_PROPERTIES_1
+    );
+  }
+
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+PlanIter_t full_text_tokenizer_properties::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc& loc,
+  std::vector<PlanIter_t>& argv,
+  expr& ann) const
+{
+  return new TokenizerPropertiesIterator(theModuleSctx, loc, argv);
+}
+
+
+PlanIter_t full_text_tokenize::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc& loc,
+  std::vector<PlanIter_t>& argv,
+  expr& ann) const
+{
+  return new TokenizeIterator(theModuleSctx, loc, argv);
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+} // namespace zorba
+/* vim:set et sw=2 ts=2: */
 
=== modified file 'src/functions/function.cpp'
--- src/functions/function.cpp	2012-04-24 12:39:38 +0000
+++ src/functions/function.cpp	2012-04-24 21:06:20 +0000
@@ -43,6 +43,7 @@
   theSignature(sig),
   theKind(kind),
   theFlags(0),
+  theModuleSctx(NULL),
   theXQueryVersion(StaticContextConsts::xquery_version_1_0)
 {
   setFlag(FunctionConsts::isBuiltin);
@@ -70,6 +71,7 @@
   SERIALIZE_ENUM(FunctionConsts::FunctionKind, theKind);
   ar & theFlags;
   ar & theAnnotationList;
+  ar & theModuleSctx;
   SERIALIZE_ENUM(StaticContextConsts::xquery_version_t, theXQueryVersion);
 }
 
@@ -92,6 +94,7 @@
   return n == VARIADIC_SIG_SIZE || argv.size() == n;
 }
 
+
 /*******************************************************************************
 
 ********************************************************************************/
 
=== modified file 'src/functions/function.h'
--- src/functions/function.h	2012-04-24 12:39:38 +0000
+++ src/functions/function.h	2012-04-24 21:06:20 +0000
@@ -42,7 +42,10 @@
 
 
 /*******************************************************************************
-
+  theModuleContext:
+  -----------------
+  The root sctx of the module containing the declaration. It is NULL for 
+  functions that must be executed in the static context of the caller.
 ********************************************************************************/
 class function : public SimpleRCObject
 {
@@ -51,6 +54,7 @@
   FunctionConsts::FunctionKind theKind;
   uint32_t                     theFlags;
   AnnotationList_t             theAnnotationList;
+  static_context             * theModuleSctx;
 
   StaticContextConsts::xquery_version_t theXQueryVersion;
 
@@ -89,6 +93,10 @@
 
   bool isVariadic() const { return theSignature.isVariadic(); }
 
+  static_context* getStaticContext() const { return theModuleSctx; }
+
+  void setStaticContext(static_context* sctx) { theModuleSctx = sctx; }
+
   void setFlag(FunctionConsts::AnnotationFlags flag)
   {
     theFlags |= flag;
 
=== modified file 'src/functions/library.cpp'
--- src/functions/library.cpp	2012-04-24 12:39:38 +0000
+++ src/functions/library.cpp	2012-04-24 21:06:20 +0000
@@ -68,6 +68,10 @@
 #include "functions/func_reflection.h"
 #include "functions/func_apply.h"
 #include "functions/func_fetch.h"
+#ifndef ZORBA_NO_FULL_TEXT
+#include "functions/func_ft_module.h"
+#include "runtime/full_text/ft_module_impl.h"
+#endif /* ZORBA_NO_FULL_TEXT */
 
 #include "functions/func_function_item_iter.h"
 
@@ -144,6 +148,10 @@
   populate_context_apply(sctx);
 
   populate_context_fetch(sctx);
+#ifndef ZORBA_NO_FULL_TEXT
+  populate_context_ft_module(sctx);
+  populate_context_ft_module_impl(sctx);
+#endif /* ZORBA_NO_FULL_TEXT */
 
   ar.set_loading_hardcoded_objects(false);
 }
 
=== added file 'src/functions/pregenerated/func_ft_module.cpp'
--- src/functions/pregenerated/func_ft_module.cpp	1970-01-01 00:00:00 +0000
+++ src/functions/pregenerated/func_ft_module.cpp	2012-04-24 21:06:20 +0000
@@ -0,0 +1,496 @@
+/*
+ * Copyright 2006-2008 The FLWOR Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+// ******************************************
+// *                                        *
+// * THIS IS A GENERATED FILE. DO NOT EDIT! *
+// * SEE .xml FILE WITH SAME NAME           *
+// *                                        *
+// ******************************************
+
+
+#include "stdafx.h"
+#include "runtime/full_text/ft_module.h"
+#include "functions/func_ft_module.h"
+
+
+namespace zorba{
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_current_lang::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc& loc,
+  std::vector<PlanIter_t>& argv,
+  expr& ann) const
+{
+  return new CurrentLangIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_host_lang::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc& loc,
+  std::vector<PlanIter_t>& argv,
+  expr& ann) const
+{
+  return new HostLangIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_is_stem_lang_supported::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc& loc,
+  std::vector<PlanIter_t>& argv,
+  expr& ann) const
+{
+  return new IsStemLangSupportedIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_is_stop_word::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc& loc,
+  std::vector<PlanIter_t>& argv,
+  expr& ann) const
+{
+  return new IsStopWordIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_is_stop_word_lang_supported::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc& loc,
+  std::vector<PlanIter_t>& argv,
+  expr& ann) const
+{
+  return new IsStopWordLangSupportedIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_is_thesaurus_lang_supported::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc& loc,
+  std::vector<PlanIter_t>& argv,
+  expr& ann) const
+{
+  return new IsThesaurusLangSupportedIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_is_tokenizer_lang_supported::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc& loc,
+  std::vector<PlanIter_t>& argv,
+  expr& ann) const
+{
+  return new IsTokenizerLangSupportedIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_stem::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc& loc,
+  std::vector<PlanIter_t>& argv,
+  expr& ann) const
+{
+  return new StemIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_strip_diacritics::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc& loc,
+  std::vector<PlanIter_t>& argv,
+  expr& ann) const
+{
+  return new StripDiacriticsIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_thesaurus_lookup::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc& loc,
+  std::vector<PlanIter_t>& argv,
+  expr& ann) const
+{
+  return new ThesaurusLookupIterator(sctx, loc, argv);
+}
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+PlanIter_t full_text_tokenize_string::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc& loc,
+  std::vector<PlanIter_t>& argv,
+  expr& ann) const
+{
+  return new TokenizeStringIterator(sctx, loc, argv);
+}
+
+#endif
+
+void populate_context_ft_module(static_context* sctx)
+{
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_current_lang,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","current-lang"), 
+        GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE),
+        FunctionConsts::FULL_TEXT_CURRENT_LANG_0);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_host_lang,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","host-lang"), 
+        GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE),
+        FunctionConsts::FULL_TEXT_HOST_LANG_0);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_is_stem_lang_supported,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","is-stem-lang-supported"), 
+        GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE, 
+        GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+        FunctionConsts::FULL_TEXT_IS_STEM_LANG_SUPPORTED_1);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_is_stop_word,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","is-stop-word"), 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+        FunctionConsts::FULL_TEXT_IS_STOP_WORD_1);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_is_stop_word,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","is-stop-word"), 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE, 
+        GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+        FunctionConsts::FULL_TEXT_IS_STOP_WORD_2);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_is_stop_word_lang_supported,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","is-stop-word-lang-supported"), 
+        GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE, 
+        GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+        FunctionConsts::FULL_TEXT_IS_STOP_WORD_LANG_SUPPORTED_1);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_is_thesaurus_lang_supported,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","is-thesaurus-lang-supported"), 
+        GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE, 
+        GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+        FunctionConsts::FULL_TEXT_IS_THESAURUS_LANG_SUPPORTED_1);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_is_thesaurus_lang_supported,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","is-thesaurus-lang-supported"), 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE, 
+        GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+        FunctionConsts::FULL_TEXT_IS_THESAURUS_LANG_SUPPORTED_2);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_is_tokenizer_lang_supported,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","is-tokenizer-lang-supported"), 
+        GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE, 
+        GENV_TYPESYSTEM.BOOLEAN_TYPE_ONE),
+        FunctionConsts::FULL_TEXT_IS_TOKENIZER_LANG_SUPPORTED_1);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_stem,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","stem"), 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE),
+        FunctionConsts::FULL_TEXT_STEM_1);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_stem,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","stem"), 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE),
+        FunctionConsts::FULL_TEXT_STEM_2);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_strip_diacritics,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","strip-diacritics"), 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE),
+        FunctionConsts::FULL_TEXT_STRIP_DIACRITICS_1);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_thesaurus_lookup,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","thesaurus-lookup"), 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_PLUS),
+        FunctionConsts::FULL_TEXT_THESAURUS_LOOKUP_1);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_thesaurus_lookup,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","thesaurus-lookup"), 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_PLUS),
+        FunctionConsts::FULL_TEXT_THESAURUS_LOOKUP_2);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_thesaurus_lookup,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","thesaurus-lookup"), 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_PLUS),
+        FunctionConsts::FULL_TEXT_THESAURUS_LOOKUP_3);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_thesaurus_lookup,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","thesaurus-lookup"), 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_PLUS),
+        FunctionConsts::FULL_TEXT_THESAURUS_LOOKUP_4);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_thesaurus_lookup,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","thesaurus-lookup"), 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.INTEGER_TYPE_ONE, 
+        GENV_TYPESYSTEM.INTEGER_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_PLUS),
+        FunctionConsts::FULL_TEXT_THESAURUS_LOOKUP_6);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_tokenize_string,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","tokenize-string"), 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_STAR),
+        FunctionConsts::FULL_TEXT_TOKENIZE_STRING_1);
+
+  }
+
+
+#endif
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+  {
+    
+
+    DECL_WITH_KIND(sctx, full_text_tokenize_string,
+        (createQName("http://www.zorba-xquery.com/modules/full-text","","tokenize-string"), 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.LANGUAGE_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_STAR),
+        FunctionConsts::FULL_TEXT_TOKENIZE_STRING_2);
+
+  }
+
+
+#endif
+}
+
+
+}
+
+
+
 
=== added file 'src/functions/pregenerated/func_ft_module.h'
--- src/functions/pregenerated/func_ft_module.h	1970-01-01 00:00:00 +0000
+++ src/functions/pregenerated/func_ft_module.h	2012-04-24 21:06:20 +0000
@@ -0,0 +1,259 @@
+/*
+ * Copyright 2006-2008 The FLWOR Foundation.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ 
+// ******************************************
+// *                                        *
+// * THIS IS A GENERATED FILE. DO NOT EDIT! *
+// * SEE .xml FILE WITH SAME NAME           *
+// *                                        *
+// ******************************************
+
+
+#ifndef ZORBA_FUNCTIONS_FT_MODULE_H
+#define ZORBA_FUNCTIONS_FT_MODULE_H
+
+
+#include "common/shared_types.h"
+#include "functions/function_impl.h"
+
+
+namespace zorba {
+
+
+void populate_context_ft_module(static_context* sctx);
+
+
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:current-lang
+class full_text_current_lang : public function
+{
+public:
+  full_text_current_lang(const signature& sig, FunctionConsts::FunctionKind kind)
+    : 
+    function(sig, kind)
+  {
+
+  }
+
+  CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:host-lang
+class full_text_host_lang : public function
+{
+public:
+  full_text_host_lang(const signature& sig, FunctionConsts::FunctionKind kind)
+    : 
+    function(sig, kind)
+  {
+
+  }
+
+  CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:is-stem-lang-supported
+class full_text_is_stem_lang_supported : public function
+{
+public:
+  full_text_is_stem_lang_supported(const signature& sig, FunctionConsts::FunctionKind kind)
+    : 
+    function(sig, kind)
+  {
+
+  }
+
+  CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:is-stop-word
+class full_text_is_stop_word : public function
+{
+public:
+  full_text_is_stop_word(const signature& sig, FunctionConsts::FunctionKind kind)
+    : 
+    function(sig, kind)
+  {
+
+  }
+
+  CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:is-stop-word-lang-supported
+class full_text_is_stop_word_lang_supported : public function
+{
+public:
+  full_text_is_stop_word_lang_supported(const signature& sig, FunctionConsts::FunctionKind kind)
+    : 
+    function(sig, kind)
+  {
+
+  }
+
+  CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:is-thesaurus-lang-supported
+class full_text_is_thesaurus_lang_supported : public function
+{
+public:
+  full_text_is_thesaurus_lang_supported(const signature& sig, FunctionConsts::FunctionKind kind)
+    : 
+    function(sig, kind)
+  {
+
+  }
+
+  CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:is-tokenizer-lang-supported
+class full_text_is_tokenizer_lang_supported : public function
+{
+public:
+  full_text_is_tokenizer_lang_supported(const signature& sig, FunctionConsts::FunctionKind kind)
+    : 
+    function(sig, kind)
+  {
+
+  }
+
+  CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:stem
+class full_text_stem : public function
+{
+public:
+  full_text_stem(const signature& sig, FunctionConsts::FunctionKind kind)
+    : 
+    function(sig, kind)
+  {
+
+  }
+
+  CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:strip-diacritics
+class full_text_strip_diacritics : public function
+{
+public:
+  full_text_strip_diacritics(const signature& sig, FunctionConsts::FunctionKind kind)
+    : 
+    function(sig, kind)
+  {
+
+  }
+
+  CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:thesaurus-lookup
+class full_text_thesaurus_lookup : public function
+{
+public:
+  full_text_thesaurus_lookup(const signature& sig, FunctionConsts::FunctionKind kind)
+    : 
+    function(sig, kind)
+  {
+
+  }
+
+  CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:tokenize
+class full_text_tokenize : public function
+{
+public:
+  full_text_tokenize(const signature& sig, FunctionConsts::FunctionKind kind)
+    : 
+    function(sig, kind)
+  {
+
+  }
+
+  CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:tokenizer-properties
+class full_text_tokenizer_properties : public function
+{
+public:
+  full_text_tokenizer_properties(const signature& sig, FunctionConsts::FunctionKind kind)
+    : 
+    function(sig, kind)
+  {
+
+  }
+
+  bool accessesDynCtx() const { return true; }
+
+  CODEGEN_DECL();
+};
+#endif
+#ifndef ZORBA_NO_FULL_TEXT
+
+//full-text:tokenize-string
+class full_text_tokenize_string : public function
+{
+public:
+  full_text_tokenize_string(const signature& sig, FunctionConsts::FunctionKind kind)
+    : 
+    function(sig, kind)
+  {
+
+  }
+
+  CODEGEN_DECL();
+};
+#endif
+
+
+} //namespace zorba
+
+
+#endif
+/*
+ * Local variables:
+ * mode: c++
+ * End:
+ */ 
 
=== modified file 'src/functions/pregenerated/function_enum.h'
--- src/functions/pregenerated/function_enum.h	2012-04-24 12:39:38 +0000
+++ src/functions/pregenerated/function_enum.h	2012-04-24 21:06:20 +0000
@@ -138,6 +138,29 @@
   FN_ZORBA_FETCH_CONTENT_2,
   FN_ZORBA_FETCH_CONTENT_TYPE_1,
   FN_PUT_2,
+  FULL_TEXT_CURRENT_LANG_0,
+  FULL_TEXT_HOST_LANG_0,
+  FULL_TEXT_IS_STEM_LANG_SUPPORTED_1,
+  FULL_TEXT_IS_STOP_WORD_1,
+  FULL_TEXT_IS_STOP_WORD_2,
+  FULL_TEXT_IS_STOP_WORD_LANG_SUPPORTED_1,
+  FULL_TEXT_IS_THESAURUS_LANG_SUPPORTED_1,
+  FULL_TEXT_IS_THESAURUS_LANG_SUPPORTED_2,
+  FULL_TEXT_IS_TOKENIZER_LANG_SUPPORTED_1,
+  FULL_TEXT_STEM_1,
+  FULL_TEXT_STEM_2,
+  FULL_TEXT_STRIP_DIACRITICS_1,
+  FULL_TEXT_THESAURUS_LOOKUP_1,
+  FULL_TEXT_THESAURUS_LOOKUP_2,
+  FULL_TEXT_THESAURUS_LOOKUP_3,
+  FULL_TEXT_THESAURUS_LOOKUP_4,
+  FULL_TEXT_THESAURUS_LOOKUP_6,
+  FULL_TEXT_TOKENIZE_1,
+  FULL_TEXT_TOKENIZE_2,
+  FULL_TEXT_TOKENIZER_PROPERTIES_0,
+  FULL_TEXT_TOKENIZER_PROPERTIES_1,
+  FULL_TEXT_TOKENIZE_STRING_1,
+  FULL_TEXT_TOKENIZE_STRING_2,
   FN_FUNCTION_NAME_1,
   FN_FUNCTION_ARITY_1,
   FN_PARTIAL_APPLY_2,
 
=== modified file 'src/runtime/full_text/CMakeLists.txt'
--- src/runtime/full_text/CMakeLists.txt	2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/CMakeLists.txt	2012-04-24 21:06:20 +0000
@@ -13,6 +13,7 @@
 # limitations under the License.
 
 SET(FULLTEXT_SRCS
+    ft_util.cpp
     ft_match.cpp
     ft_query_item.cpp
     ft_single_token_iterator.cpp
@@ -40,6 +41,7 @@
     thesaurus.cpp
     tokenizer.cpp
     default_tokenizer.cpp
+    ft_module.cpp
     )
 
 IF (ZORBA_NO_ICU)
@@ -51,5 +53,5 @@
 ADD_SRC_SUBFOLDER(FULLTEXT_SRCS stemmer LIBSTEMMER_SRCS)
 
 IF (ZORBA_WITH_FILE_ACCESS)
-    ADD_SRC_SUBFOLDER(FULLTEXT_SRCS thesauri THESAURUS_SRCS)
+  ADD_SRC_SUBFOLDER(FULLTEXT_SRCS thesauri THESAURUS_SRCS)
 ENDIF (ZORBA_WITH_FILE_ACCESS)
 
=== modified file 'src/runtime/full_text/apply.cpp'
--- src/runtime/full_text/apply.cpp	2012-04-24 12:39:38 +0000
+++ src/runtime/full_text/apply.cpp	2012-04-24 21:06:20 +0000
@@ -26,13 +26,14 @@
 #include "diagnostics/dict.h"
 #include "diagnostics/xquery_diagnostics.h"
 #include "store/api/item.h"
+#include "store/api/item_factory.h"
 #include "store/api/store.h"
-#include "store/api/item_factory.h"
 #include "system/globalenv.h"
 #include "util/cxx_util.h"
 #include "util/indent.h"
 #include "util/stl_util.h"
 #include "zorbamisc/ns_consts.h"
+#include "zorbautils/locale.h"
 
 #ifndef NDEBUG
 # include "system/properties.h"
@@ -1184,11 +1185,10 @@
   {
   }
 
-  void operator()( char const *utf8_s, size_type utf8_len, size_type,
-                   size_type, size_type, void* ) {
-    FTToken const t( utf8_s, (int)utf8_len, token_no_, lang_ );
-    tokens_.push_back( t );
-  }
+  // inherited
+  void item( Item const&, bool );
+  void token( char const*, size_type, iso639_1::type, size_type, size_type,
+               size_type, Item const* );
 
 private:
   FTTokenSeqIterator::FTTokens &tokens_;
@@ -1196,51 +1196,72 @@
   iso639_1::type const lang_;
 };
 
+void thesaurus_callback::item( Item const&, bool ) {
+  // out-of-line since it's virtual
+}
+
+void thesaurus_callback::token( char const *utf8_s, size_type utf8_len,
+                                iso639_1::type, size_type, size_type,
+                                size_type, Item const* ) {
+  FTToken const t( utf8_s, (int)utf8_len, token_no_, lang_ );
+  tokens_.push_back( t );
+}
+
 } // anonymous namespace
 
 void ftcontains_visitor::
-lookup_thesaurus( ftthesaurus_id const &tid, zstring const &query_phrase,
+lookup_thesaurus( ftthesaurus_id const &t_id, zstring const &query_phrase,
                   FTToken const &qt0, query_item_star_t &result ) {
   ft_int at_least, at_most;
-  if ( ftrange const *const levels = tid.get_levels() )
+  if ( ftrange const *const levels = t_id.get_levels() )
     eval_ftrange( *levels, &at_least, &at_most );
   else
     at_least = 0, at_most = numeric_limits<ft_int>::max();
 
-  zstring const &uri = tid.get_uri();
+  zstring const &uri = t_id.get_uri();
 
   zstring error_msg;
   auto_ptr<internal::Resource> rsrc = static_ctx_.resolve_uri(
-    uri, internal::ThesaurusEntityData( qt0.lang() ), error_msg
+    uri, internal::EntityData::THESAURUS, error_msg
   );
   if ( !rsrc.get() )
     throw XQUERY_EXCEPTION( err::FTST0018, ERROR_PARAMS( uri ) );
 
-  internal::Thesaurus::ptr thesaurus(
-    dynamic_cast<internal::Thesaurus*>( rsrc.release() )
-  );
-  if ( !thesaurus )
-    throw XQUERY_EXCEPTION( err::FTST0018, ERROR_PARAMS( uri ) );
-
-  internal::Thesaurus::iterator::ptr tresult(
+  internal::ThesaurusProvider const *const t_provider =
+    dynamic_cast<internal::ThesaurusProvider const*>( rsrc.get() );
+  ZORBA_ASSERT( t_provider );
+
+  internal::Thesaurus::ptr thesaurus;
+  if ( !t_provider->getThesaurus( qt0.lang(), &thesaurus ) )
+    throw XQUERY_EXCEPTION(
+      zerr::ZXQP8406_THESAURUS_LANG_NOT_SUPPORTED,
+      ERROR_PARAMS( iso639_1::string_of[ qt0.lang() ] )
+    );
+
+  internal::Thesaurus::iterator::ptr t_synonyms(
     thesaurus->lookup(
-      query_phrase, tid.get_relationship(), at_least, at_most
+      query_phrase, t_id.get_relationship(), at_least, at_most
     )
   );
-  if ( !tresult )
+  if ( !t_synonyms )
     return;
 
   FTTokenSeqIterator::FTTokens synonyms;
   thesaurus_callback cb( qt0.pos(), qt0.lang(), synonyms );
 
-  Tokenizer::Numbers tno;
-  Tokenizer::ptr tokenizer(
-    GENV_STORE.getTokenizerProvider()->getTokenizer( qt0.lang(), tno )
-  );
+  Tokenizer::Numbers t_num;
+  TokenizerProvider const *const provider = GENV_STORE.getTokenizerProvider();
+  ZORBA_ASSERT( provider );
+  Tokenizer::ptr tokenizer;
+  if ( !provider->getTokenizer( qt0.lang(), &t_num, &tokenizer ) )
+    throw XQUERY_EXCEPTION(
+      zerr::ZXQP8407_TOKENIZER_LANG_NOT_SUPPORTED,
+      ERROR_PARAMS( iso639_1::string_of[ qt0.lang() ] )
+    );
 
-  for ( zstring synonym; tresult->next( &synonym ); ) {
+  for ( zstring synonym; t_synonyms->next( &synonym ); ) {
     synonyms.clear();
-    tokenizer->tokenize(
+    tokenizer->tokenize_string(
       synonym.data(), synonym.size(), qt0.lang(), false, cb
     );
     query_item_t const query_item( new FTTokenSeqIterator( synonyms ) );
 
=== added file 'src/runtime/full_text/ft_module_impl.cpp'
--- src/runtime/full_text/ft_module_impl.cpp	1970-01-01 00:00:00 +0000
+++ src/runtime/full_text/ft_module_impl.cpp	2012-04-24 21:06:20 +0000
@@ -0,0 +1,843 @@
+/*
+ * Copyright 2006-2008 The FLWOR Foundation.
+ * 
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ * 
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * 
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <zorba/config.h>
+
+#ifndef ZORBA_NO_FULL_TEXT
+
+# include <limits>
+# include <typeinfo>
+
+# include <zorba/diagnostic_list.h>
+
+# include "api/unmarshaller.h"
+# include "context/namespace_context.h"
+# include "context/static_context.h"
+# include "diagnostics/assert.h"
+# include "diagnostics/xquery_diagnostics.h"
+# include "store/api/index.h"
+# include "store/api/item.h"
+# include "store/api/item_factory.h"
+# include "store/api/iterator.h"
+# include "store/api/store.h"
+# include "system/globalenv.h"
+# include "types/casting.h"
+# include "types/typeimpl.h"
+# include "types/typeops.h"
+# include "util/utf8_util.h"
+# include "zorbatypes/URI.h"
+# include "zorbautils/locale.h"
+
+# include "ft_stop_words_set.h"
+# include "ft_token_seq_iterator.h"
+# include "ft_util.h"
+# include "thesaurus.h"
+
+#endif /* ZORBA_NO_FULL_TEXT */
+
+#include "runtime/full_text/ft_module.h"
+
+using namespace std;
+using namespace zorba::locale;
+
+namespace zorba {
+
+///////////////////////////////////////////////////////////////////////////////
+
+#ifndef ZORBA_NO_FULL_TEXT
+inline iso639_1::type get_lang_from( static_context const *sctx ) {
+  iso639_1::type const lang = get_lang_from( sctx->get_match_options() );
+  return lang ? lang : get_host_lang();
+}
+
+static iso639_1::type get_lang_from( store::Item_t lang_item,
+                                     QueryLoc const &loc ) {
+  zstring lang_string;
+  lang_item->getStringValue2( lang_string );
+
+  if ( !GenericCast::instance()->castableToLanguage( lang_string ) )
+    throw XQUERY_EXCEPTION(
+      err::XPTY0004,
+      ERROR_PARAMS(
+        ZED( BadType_23o ), lang_string, ZED( NoCastTo_45o ), "xs:language"
+      ),
+      ERROR_LOC( loc )
+    );
+  if ( iso639_1::type const lang = find_lang( lang_string.c_str() ) )
+    return lang;
+  throw XQUERY_EXCEPTION(
+    err::FTST0009, ERROR_PARAMS( lang_string ), ERROR_LOC( loc )
+  );
+}
+#endif /* ZORBA_NO_FULL_TEXT */
+
+///////////////////////////////////////////////////////////////////////////////
+
+bool CurrentLangIterator::nextImpl( store::Item_t &result,
+                                    PlanState &plan_state ) const {
+#ifdef ZORBA_NO_FULL_TEXT
+  return false;
+#else
+  iso639_1::type const lang = get_lang_from( getStaticContext() );
+  zstring lang_string( iso639_1::string_of[ lang ] );
+
+  PlanIteratorState *state;
+  DEFAULT_STACK_INIT( PlanIteratorState, state, plan_state );
+
+  GENV_ITEMFACTORY->createLanguage( result, lang_string );
+  STACK_PUSH( true, state );
+
+  STACK_END( state );
+#endif /* ZORBA_NO_FULL_TEXT */
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+bool HostLangIterator::nextImpl( store::Item_t &result,
+                                 PlanState &plan_state ) const {
+#ifdef ZORBA_NO_FULL_TEXT
+  return false;
+#else
+  iso639_1::type const lang = get_host_lang();
+  zstring lang_string = iso639_1::string_of[ lang ];
+
+  PlanIteratorState *state;
+  DEFAULT_STACK_INIT( PlanIteratorState, state, plan_state );
+
+  GENV_ITEMFACTORY->createLanguage( result, lang_string );
+  STACK_PUSH( true, state );
+
+  STACK_END( state );
+#endif /* ZORBA_NO_FULL_TEXT */
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+bool IsStemLangSupportedIterator::nextImpl( store::Item_t &result,
+                                            PlanState &plan_state ) const {
+#ifdef ZORBA_NO_FULL_TEXT
+  return false;
+#else
+  bool is_supported;
+  store::Item_t item;
+
+  PlanIteratorState *state;
+  DEFAULT_STACK_INIT( PlanIteratorState, state, plan_state );
+
+  consumeNext( item, theChildren[0], plan_state );
+  try {
+    internal::StemmerProvider const *const provider =
+      GENV_STORE.getStemmerProvider();
+    is_supported = provider->getStemmer( get_lang_from( item, loc ) );
+  }
+  catch ( XQueryException const &e ) {
+    if ( e.diagnostic() != err::FTST0009 )
+      throw;
+    is_supported = false;
+  }
+
+  GENV_ITEMFACTORY->createBoolean( result, is_supported );
+  STACK_PUSH( true, state );
+
+  STACK_END( state );
+#endif /* ZORBA_NO_FULL_TEXT */
+}
+
+///////////////////////////////////////////////////////////////////////////////
+
+bool IsStopWordIterator::nextImpl( store::Item_t &result,
+                                   PlanState &plan_state ) const {
+#ifdef ZORBA_NO_FULL_TEXT
+  return false;
+#else
+  store::Item_t item;
+  iso639_1::type lang;