Merge lp:~zorba-coders/zorba/no_unicode into lp:zorba

[Rodolfo Ochoa]

"No Unicode" is now "No ICU."
Added a a q-flag fix for an undiscovered bug.

"No Unicode" is now "No ICU."
Added a a q-flag fix for an undiscovered bug.

[Matthias Brantner (matthias-brantner)]

Compiling with ZORBA_NO_ICU=ON fails on Linux:

[ 1%] Building CXX object src/CMakeFiles/zorba_simplestore.dir/api/zorba_string.cpp.o
In file included from /home/mbrantner/zorba/sandbox/src/util/regex.h:501:0,
                 from /home/mbrantner/zorba/sandbox/src/api/zorba_string.cpp:23:
/home/mbrantner/zorba/sandbox/src/util/regex_xquery.h:209:3: error: a class-key must be used when declaring a friend
/home/mbrantner/zorba/sandbox/src/util/regex_xquery.h:209:3: error: friend declaration does not name a class or function
/home/mbrantner/zorba/sandbox/src/util/regex_xquery.h:253:3: error: a class-key must be used when declaring a friend
/home/mbrantner/zorba/sandbox/src/util/regex_xquery.h:253:3: error: friend declaration does not name a class or function
make[2]: *** [src/CMakeFiles/zorba_simplestore.dir/api/zorba_string.cpp.o] Erro

[Matthias Brantner (matthias-brantner)]

review: Needs Fixing

[Zorba Build Bot (zorba-buildbot)]

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

[Matthias Brantner (matthias-brantner)]

The test suite doesn't run clean on my system (Linux) without ICU. This prevents us from adding the built to the remote queue. For example, the following tests fail without ICU (some of them also seem to fail with ICU):

 1294 - test/rbkt/zorba/string/Regex/regex_a10 (Failed)
 1548 - test/rbkt/zorba/fulltext/ft-wildcard-true-2 (Failed)
 1560 - test/rbkt/zorba/fulltext/ft-wildcard-true-4 (Failed)
 1574 - test/rbkt/zorba/fulltext/ft-same-sentence-true-4 (Failed)
 1581 - test/rbkt/zorba/fulltext/ft-wildcard-true-3 (Failed)
 1587 - test/rbkt/zorba/fulltext/ft-wildcard-true-9 (Failed)
 1600 - test/rbkt/zorba/fulltext/ft-diacritics-insensitive-true-1 (Failed)
 1605 - test/rbkt/zorba/fulltext/ft-wildcard-true-8 (Failed)
 1612 - test/rbkt/zorba/fulltext/ft-wildcard-true-10 (Failed)
 1635 - test/rbkt/zorba/fulltext/ft-wildcard-true-7 (Failed)
 1637 - test/rbkt/zorba/fulltext/ft-wildcard-true-11 (Failed)
 1643 - test/rbkt/zorba/fulltext/ft-wildcard-FTDY0020-3 (Failed)
 1789 - test/rbkt/zorba/index/numbers (Failed)
 2345 - test/unit/string_test (Failed)
 2534 - test/update/zorba/store/sc3 (Failed)
 2544 - doc/cxx/examples/context.cpp (Failed)

Please make sure the test suite runs clean.

review: Needs Fixing

[Paul J. Lucas (paul-lucas)]

Try it now.

[Daniel Turcanu (danielturcanu)]

Before commiting this branch, the branch lp:~danielturcanu/zorba/my_conv_module should be merged.

[Chris Hillery (ceejatec)]

FWIW, I've skimmed the change for CMake-related changes, and they all look fine (mostly quite trivial).

[Zorba Build Bot (zorba-buildbot)]

Attempt to merge into lp:zorba failed due to conflicts:

text conflict in ChangeLog

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/no_unicode-2012-03-30T19-15-23.23Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~zorba-coders/zorba/no_unicode into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job no_unicode-2012-03-30T19-15-23.23Z is finished. The
  final status was:

  6 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/no_unicode-2012-04-03T15-17-37.639Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~zorba-coders/zorba/no_unicode into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job no_unicode-2012-04-03T15-17-37.639Z is finished. The
  final status was:

  6 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/no_unicode-2012-04-06T00-21-13.829Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~zorba-coders/zorba/no_unicode into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job no_unicode-2012-04-06T00-21-13.829Z is finished. The
  final status was:

  6 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Zorba Build Bot (zorba-buildbot)]

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

[Markos Zaharioudakis (markos-za)]

review: Approve

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/no_unicode-2012-04-11T22-01-43.083Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~zorba-coders/zorba/no_unicode into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job no_unicode-2012-04-11T22-01-43.083Z is finished. The
  final status was:

  No tests were run - build or configure step must have failed.

  Not commiting changes.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/no_unicode-2012-04-11T23-27-15.591Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~zorba-coders/zorba/no_unicode into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job no_unicode-2012-04-11T23-27-15.591Z is finished. The
  final status was:

  No tests were run - build or configure step must have failed.

  Not commiting changes.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Zorba Build Bot (zorba-buildbot)]

Attempt to merge into lp:zorba failed due to conflicts:

text conflict in src/zorbautils/hashmap_itemh.h

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/no_unicode-2012-04-12T02-15-23.414Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~zorba-coders/zorba/no_unicode into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job no_unicode-2012-04-12T02-15-23.414Z is finished. The
  final status was:

  6 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/no_unicode-2012-04-13T23-51-19.988Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue job no_unicode-2012-04-13T23-51-19.988Z is finished. The final status was:

All tests succeeded!

[Zorba Build Bot (zorba-buildbot)]

Voting does not meet specified criteria. Required: Approve > 1, Disapprove < 1, Needs Fixing < 1, Pending < 1. Got: 2 Approve, 1 Pending.

[Rodolfo Ochoa (rodolfo-ochoa)]

review: Approve

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/no_unicode-2012-04-14T01-06-17.625Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue job no_unicode-2012-04-14T01-06-17.625Z is finished. The final status was:

All tests succeeded!

=== modified file 'CMakeConfiguration.txt'
--- CMakeConfiguration.txt	2012-03-28 05:19:57 +0000
+++ CMakeConfiguration.txt	2012-04-13 19:45:38 +0000
@@ -135,14 +135,14 @@
 SET (ZORBA_DEBUG_STRING ${ZORBA_DEBUG_STRING} CACHE BOOL "debug strings")
 MESSAGE (STATUS "ZORBA_DEBUG_STRING:                   " ${ZORBA_DEBUG_STRING})
 
-SET(ZORBA_NO_UNICODE OFF CACHE BOOL "disable ICU")
-MESSAGE(STATUS "ZORBA_NO_UNICODE:                     " ${ZORBA_NO_UNICODE})
+SET(ZORBA_NO_ICU OFF CACHE BOOL "disable ICU")
+MESSAGE(STATUS "ZORBA_NO_ICU:                         " ${ZORBA_NO_ICU})
 
-IF (ZORBA_NO_UNICODE)
+IF (ZORBA_NO_ICU)
   SET (no_full_text ON)
-ELSE (ZORBA_NO_UNICODE)
+ELSE (ZORBA_NO_ICU)
   SET (no_full_text OFF)
-ENDIF (ZORBA_NO_UNICODE)
+ENDIF (ZORBA_NO_ICU)
 SET (ZORBA_NO_FULL_TEXT ${no_full_text} CACHE BOOL "disable XQuery Full-Text support")
 MESSAGE(STATUS "ZORBA_NO_FULL_TEXT:                   " ${ZORBA_NO_FULL_TEXT})
 

=== modified file 'CMakeLists.txt'
--- CMakeLists.txt	2012-03-28 05:19:57 +0000
+++ CMakeLists.txt	2012-04-13 19:45:38 +0000
@@ -123,10 +123,14 @@
 CHECK_TYPE_SIZE("int64_t" ZORBA_HAVE_INT64_T) 
 
 CHECK_CXX_SOURCE_COMPILES ("#include <type_traits>\nint main() { std::enable_if<true,int> x; }" ZORBA_CXX_ENABLE_IF)
-CHECK_CXX_SOURCE_COMPILES ("int main() { int *p = nullptr; }" ZORBA_CXX_NULLPTR)
-CHECK_CXX_SOURCE_COMPILES ("int main() { static_assert(1,\"\"); }" ZORBA_CXX_STATIC_ASSERT)
+SET(CMAKE_EXTRA_INCLUDE_FILES wchar.h)
+CHECK_TYPE_SIZE("wchar_t" ZORBA_SIZEOF_WCHAR_T)
+SET(CMAKE_EXTRA_INCLUDE_FILES)
 CHECK_CXX_SOURCE_COMPILES ("#include <memory>\nint main() { std::unique_ptr<int> p; }" ZORBA_CXX_UNIQUE_PTR)
 
+CHECK_CXX_SOURCE_COMPILES("int main() { int *p = nullptr; }" ZORBA_CXX_NULLPTR)
+CHECK_CXX_SOURCE_COMPILES("int main() { static_assert(1,\"\"); }" ZORBA_CXX_STATIC_ASSERT)
+
 ################################################################################
 # Various cmake macros
 

=== modified file 'ChangeLog'
--- ChangeLog	2012-04-13 11:34:54 +0000
+++ ChangeLog	2012-04-13 19:45:38 +0000
@@ -4,6 +4,7 @@
 
 New Features:
   * Extended API for Python, Java, PHP and Ruby.
+  * Added support for NO_ICU (to not use ICU for unicode processing)
 
 Optimization:
 
@@ -154,7 +155,9 @@
   * Fixed bug when parsing a document with a base-uri attribute.
   * Fixed bug #863320 (Sentence is incorrectly incremented when token characters end without sentence terminator)
   * Fixed bug #863730 (static delete-node* functions don't raise ZDDY0012)
+  * Implemented the probe-index-range-value for general indexes
   * Removed ZSTR0005 and ZSTR0006 error codes
+  * Fixed bug #867662 ("nullptr" warning)
   * Fixed bug #868258 (Assertion failure with two delete collection)
   * Fixed bug #871623 and #871629 (assertion failures with insertions in dynamic collections)
   * Fixed bug #867262 (allow reuse of iterator over ExtFuncArgItemSequence)
@@ -163,6 +166,8 @@
   * New node-reference module. References can be obtained for any node, and
 	different nodes cannot have the same identifier.
   * Fixed bug #872697  (segmentation fault with validation of NMTOKENS)
+  * General index cannot be declared as unique if the type of its key is
+    xs:anyAtomicType or xs:untypedAtomic.
   * Added undo for node revalidation
   * Optimization for count(collection()) expressions
   * Fixed bug #872796  (validate-in-place can interfere with other update primitives)
@@ -181,6 +186,8 @@
   * Fixed bug #855715 (Invalid escaped characters in regex not caught)
   * Fixed bug #862089 (Split binary/xq install directories for modules) by
   splitting "module path" into separate URI and Library paths
+  * New node-position module. This module allows to obtain a representation of a node position, which
+    can be used to assess structural relationships with other nodes.   
   * Fixed bug #872502 (validation of the JSON module xqdoc fails)
   * Fixed bug #897619 (testdriver_mt can not run the XQueryX tests)
   * Fixed bug #867107 (xqdoc dependency to zorba is wrong)

=== modified file 'KNOWN_ISSUES.txt'
--- KNOWN_ISSUES.txt	2012-03-28 05:19:57 +0000
+++ KNOWN_ISSUES.txt	2012-04-13 19:45:38 +0000
@@ -37,7 +37,7 @@
 * The serializer currently doesn't implement character maps as specified
   (http://www.w3.org/TR/xslt-xquery-serialization/#character-maps)
 
-* In the 2.0 release, setting the CMake variables ZORBA_NO_UNICODE to
+* In the 2.0 release, setting the CMake variables ZORBA_NO_ICU to
   ON is not supported.
 
 * The PHP language binding is not supported on Mac OS X. For details,

=== modified file 'doc/cxx/examples/context.cpp'
--- doc/cxx/examples/context.cpp	2012-03-28 05:19:57 +0000
+++ doc/cxx/examples/context.cpp	2012-04-13 19:45:38 +0000
@@ -149,7 +149,11 @@
     outStream2 << lQuery << std::endl;
     std::cout << outStream2.str() << std::endl;
 
+#ifndef ZORBA_NO_ICU
     if (outStream2.str() != "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\nBook 1.1\n")
+#else
+    if (outStream2.str() != "<?xml version=\"1.0\"?>\nBook 1.1\n")
+#endif /* ZORBA_NO_ICU */
     {
       std::cerr << "Test 4 failed with a wrong result : " << std::endl
                 << outStream2.str() << std::endl;

=== modified file 'include/zorba/config.h.cmake'
--- include/zorba/config.h.cmake	2012-03-28 05:19:57 +0000
+++ include/zorba/config.h.cmake	2012-04-13 19:45:38 +0000
@@ -96,6 +96,8 @@
 typedef __int64 int64_t;
 #endif /* ZORBA_HAVE_INT64_T */
 
+#cmakedefine ZORBA_SIZEOF_WCHAR_T @ZORBA_SIZEOF_WCHAR_T@
+
 // Compiler
 #cmakedefine CLANG
 #cmakedefine MSVC
@@ -148,7 +150,7 @@
 
 // Zorba features
 #cmakedefine ZORBA_NO_FULL_TEXT
-#cmakedefine ZORBA_NO_UNICODE
+#cmakedefine ZORBA_NO_ICU
 #cmakedefine ZORBA_NO_XMLSCHEMA
 #cmakedefine ZORBA_NUMERIC_OPTIMIZATION
 #cmakedefine ZORBA_VERIFY_PEER_SSL_CERTIFICATE

=== modified file 'include/zorba/static_context.h'
--- include/zorba/static_context.h	2012-04-13 09:11:32 +0000
+++ include/zorba/static_context.h	2012-04-13 19:45:38 +0000
@@ -26,9 +26,13 @@
 #include <zorba/function.h>
 #include <zorba/annotation.h>
 #include <zorba/smart_ptr.h>
+#include <zorba/smart_ptr.h>
 #ifndef ZORBA_NO_FULL_TEXT
 #include <zorba/thesaurus.h>
 #endif /* ZORBA_NO_FULL_TEXT */
+#include <zorba/zorba.h>
+#include <zorba/store_manager.h>
+#include <zorba/zorba_exception.h>
 
 namespace zorba {
 

=== modified file 'include/zorba/util/time.h'
--- include/zorba/util/time.h	2012-03-28 05:19:57 +0000
+++ include/zorba/util/time.h	2012-04-13 19:45:38 +0000
@@ -178,7 +178,7 @@
 	
     inline long get_walltime_in_millis(const walltime& t)
     {
-      return t.time * 1000 + t.millitm;
+      return (long)(t.time * 1000 + t.millitm);
     }
 
 #else /* not Windows, and no clock_gettime() */

=== modified file 'src/CMakeLists.txt'
--- src/CMakeLists.txt	2012-03-28 05:19:57 +0000
+++ src/CMakeLists.txt	2012-04-13 19:45:38 +0000
@@ -59,7 +59,10 @@
 #
 # Next, add the files to be compiled into the library
 #
+
+MESSAGE(STATUS  "PRECOMPILED HEADERS: " ${ZORBA_PRECOMPILED_HEADERS})
 SET(ZORBA_PRECOMPILED_HEADERS OFF CACHE BOOL "Activate Zorba precompiled headers.")
+MESSAGE(STATUS  "PRECOMPILED HEADERS: " ${ZORBA_PRECOMPILED_HEADERS})
 
 SET(ZORBA_SRCS)
 ADD_SRC_SUBFOLDER(ZORBA_SRCS api API_SRCS)
@@ -97,6 +100,7 @@
 ENDIF(ZORBA_WITH_DEBUGGER)
 ADD_SRC_SUBFOLDER(ZORBA_SRCS unit_tests UNIT_TEST_SRCS)
 
+MESSAGE(STATUS  "PRECOMPILED HEADERS: " ${ZORBA_PRECOMPILED_HEADERS})
 IF(ZORBA_PRECOMPILED_HEADERS)
   ADD_SRC_SUBFOLDER(ZORBA_SRCS precompiled ZORBAMISC_SRCS)
   INCLUDE_DIRECTORIES("${CMAKE_SOURCE_DIR}/src/precompiled")

=== modified file 'src/api/serialization/serializer.cpp'
--- src/api/serialization/serializer.cpp	2012-03-28 05:19:57 +0000
+++ src/api/serialization/serializer.cpp	2012-04-13 19:45:38 +0000
@@ -180,7 +180,6 @@
   for (; chars < chars_end; chars++ )
   {
 
-#ifndef ZORBA_NO_UNICODE
     // the input string is UTF-8
     int char_length = utf8::char_length(*chars);
     if (char_length == 0)
@@ -217,7 +216,6 @@
 
       continue;
     }
-#endif//ZORBA_NO_UNICODE
 
     // raise an error iff (1) the serialization format is XML 1.0 and (2) the given character is an invalid XML 1.0 character
     if (ser && ser->method == PARAMETER_VALUE_XML &&
@@ -332,14 +330,12 @@
     {
       tr << (char)0xEF << (char)0xBB << (char)0xBF;
     }
-#ifndef ZORBA_NO_UNICODE
     else if (ser->encoding == PARAMETER_VALUE_UTF_16)
     {
       // Little-endian
       tr.verbatim((char)0xFF);
       tr.verbatim((char)0xFE);
     }
-#endif
   }
 }
 
@@ -862,13 +858,17 @@
   emitter::emit_declaration();
 
   if (ser->omit_xml_declaration == PARAMETER_VALUE_NO) {
-    tr << "<?xml version=\"" << ser->version << "\" encoding=\"";
-    if (ser->encoding == PARAMETER_VALUE_UTF_8) {
-      tr << "UTF-8";
-#ifndef ZORBA_NO_UNICODE
-    } else if (ser->encoding == PARAMETER_VALUE_UTF_16) {
-      tr << "UTF-16";
-#endif
+    tr << "<?xml version=\"" << ser->version;
+    switch (ser->encoding) {
+      case PARAMETER_VALUE_UTF_8:
+      case PARAMETER_VALUE_UTF_16:
+        tr << "\" encoding=\"";
+        switch (ser->encoding) {
+          case PARAMETER_VALUE_UTF_8 : tr << "UTF-8" ; break;
+          case PARAMETER_VALUE_UTF_16: tr << "UTF-16"; break;
+          default                    : ZORBA_ASSERT(false);
+        }
+        break;
     }
     tr << "\"";
 
@@ -1174,14 +1174,18 @@
       }
 
       tr << "<meta http-equiv=\"content-type\" content=\""
-         << ser->media_type << "; charset=";
-
-      if (ser->encoding == PARAMETER_VALUE_UTF_8)
-        tr << "UTF-8";
-#ifndef ZORBA_NO_UNICODE
-      else if (ser->encoding == PARAMETER_VALUE_UTF_16)
-        tr << "UTF-16";
-#endif
+         << ser->media_type;
+      switch (ser->encoding) {
+        case PARAMETER_VALUE_UTF_8:
+        case PARAMETER_VALUE_UTF_16:
+          tr << "\" charset=\"";
+          switch (ser->encoding) {
+            case PARAMETER_VALUE_UTF_8 : tr << "UTF-8" ; break;
+            case PARAMETER_VALUE_UTF_16: tr << "UTF-16"; break;
+            default                    : ZORBA_ASSERT(false);
+          }
+          break;
+      }
       tr << "\"";
       // closed_parent_tag = 1;
     }
@@ -1371,14 +1375,18 @@
         }
 
         tr << "<meta http-equiv=\"content-type\" content=\""
-           << ser->media_type << "; charset=";
-
-        if (ser->encoding == PARAMETER_VALUE_UTF_8)
-          tr << "UTF-8";
-#ifndef ZORBA_NO_UNICODE
-        else if (ser->encoding == PARAMETER_VALUE_UTF_16)
-          tr << "UTF-16";
-#endif
+           << ser->media_type;
+        switch (ser->encoding) {
+          case PARAMETER_VALUE_UTF_8:
+          case PARAMETER_VALUE_UTF_16:
+            tr << "\" charset=\"";
+            switch (ser->encoding) {
+              case PARAMETER_VALUE_UTF_8 : tr << "UTF-8" ; break;
+              case PARAMETER_VALUE_UTF_16: tr << "UTF-16"; break;
+              default                    : ZORBA_ASSERT(false);
+            }
+            break;
+        }
         tr << "\"/";
         //closed_parent_tag = 1;
       }
@@ -2098,10 +2106,8 @@
   {
     if (!strcmp(aValue, "UTF-8"))
       encoding = PARAMETER_VALUE_UTF_8;
-#ifndef ZORBA_NO_UNICODE
     else if (!strcmp(aValue, "UTF-16"))
       encoding = PARAMETER_VALUE_UTF_16;
-#endif
     else
       throw XQUERY_EXCEPTION(
         err::SEPM0016, ERROR_PARAMS( aValue, aName, ZED( GoodValuesAreUTF8 ) )
@@ -2210,16 +2216,13 @@
   {
     tr = new transcoder(os, false);
   }
-#ifndef ZORBA_NO_UNICODE
   else if (encoding == PARAMETER_VALUE_UTF_16)
   {
     tr = new transcoder(os, true);
   }
-#endif
   else
   {
-    ZORBA_ASSERT(0);
-    return false;
+    ZORBA_ASSERT(false);
   }
 
   if (method == PARAMETER_VALUE_XML)

=== modified file 'src/api/serialization/serializer.h'
--- src/api/serialization/serializer.h	2012-03-28 05:19:57 +0000
+++ src/api/serialization/serializer.h	2012-04-13 19:45:38 +0000
@@ -70,10 +70,8 @@
     PARAMETER_VALUE_TEXT,
     PARAMETER_VALUE_BINARY,
 
-    PARAMETER_VALUE_UTF_8
-#ifndef ZORBA_NO_UNICODE
-    ,PARAMETER_VALUE_UTF_16
-#endif
+    PARAMETER_VALUE_UTF_8,
+    PARAMETER_VALUE_UTF_16
   } PARAMETER_VALUE_TYPE;
 
 protected:

=== modified file 'src/diagnostics/diagnostic_en.xml'
--- src/diagnostics/diagnostic_en.xml	2012-04-10 13:10:22 +0000
+++ src/diagnostics/diagnostic_en.xml	2012-04-13 19:45:38 +0000
@@ -2517,11 +2517,11 @@
       <value>attribute node</value>
     </entry>
 
-    <entry key="BackRef0Illegal">
+    <entry key="BackRef0Illegal" if="!defined(ZORBA_NO_ICU)">
       <value>"0": illegal backreference</value>
     </entry>
 
-    <entry key="BackRefIllegalInCharClass">
+    <entry key="BackRefIllegalInCharClass" if="!defined(ZORBA_NO_ICU)">
       <value>backreference illegal in character class</value>
     </entry>
 
@@ -2569,7 +2569,7 @@
       <value>invalid library module</value>
     </entry>
 
-    <entry key="BadRegexEscape_3">
+    <entry key="BadRegexEscape_3" if="!defined(ZORBA_NO_ICU)">
       <value>"$3": illegal escape character</value>
     </entry>
 
@@ -3029,7 +3029,7 @@
       <value>nodeid component too big for encoding</value>
     </entry>
 
-    <entry key="NonClosedBackRef_3">
+    <entry key="NonClosedBackRef_3" if="!defined(ZORBA_NO_ICU)">
       <value>'$$3': non-closed backreference</value>
     </entry>
 
@@ -3041,7 +3041,7 @@
       <value>non-localhost authority</value>
     </entry>
 
-    <entry key="NonexistentBackRef_3">
+    <entry key="NonexistentBackRef_3" if="!defined(ZORBA_NO_ICU)">
       <value>'$$3': non-existent backreference</value>
     </entry>
 
@@ -3193,94 +3193,183 @@
       <value>item type is not a subtype of "$3"</value>
     </entry>
 
-    <entry key="U_REGEX_BAD_ESCAPE_SEQUENCE" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_BAD_ESCAPE_SEQUENCE" if="!defined(ZORBA_NO_ICU)">
       <value>unrecognized backslash escape sequence</value>
     </entry>
 
-    <entry key="U_REGEX_BAD_INTERVAL" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_BAD_INTERVAL" if="!defined(ZORBA_NO_ICU)">
       <value>error in {min,max} interval</value>
     </entry>
 
-    <entry key="U_REGEX_INTERNAL_ERROR" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_INTERNAL_ERROR" if="!defined(ZORBA_NO_ICU)">
       <value>an internal ICU error (bug) was detected</value>
     </entry>
 
-    <entry key="U_REGEX_INVALID_BACK_REF" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_INVALID_BACK_REF" if="!defined(ZORBA_NO_ICU)">
       <value>backreference to a non-existent capture group</value>
     </entry>
 
-    <entry key="U_REGEX_INVALID_FLAG" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_INVALID_FLAG" if="!defined(ZORBA_NO_ICU)">
       <value>invalid value for match mode flags</value>
     </entry>
 
-    <entry key="U_REGEX_INVALID_RANGE" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_INVALID_RANGE" if="!defined(ZORBA_NO_ICU)">
       <value>in character range [x-y], x is greater than y</value>
     </entry>
 
-    <entry key="U_REGEX_INVALID_STATE" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_INVALID_STATE" if="!defined(ZORBA_NO_ICU)">
       <value>RegexMatcher in invalid state for requested operation</value>
     </entry>
 
-    <entry key="U_REGEX_LOOK_BEHIND_LIMIT" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_LOOK_BEHIND_LIMIT" if="!defined(ZORBA_NO_ICU)">
       <value>look-behind pattern matches must have a bounded maximum length</value>
     </entry>
 
-    <entry key="U_REGEX_MAX_LT_MIN" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_MAX_LT_MIN" if="!defined(ZORBA_NO_ICU)">
       <value>in {min,max}, max is less than min</value>
     </entry>
 
-    <entry key="U_REGEX_MISMATCHED_PAREN" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_MISMATCHED_PAREN" if="!defined(ZORBA_NO_ICU)">
       <value>incorrectly nested parentheses</value>
     </entry>
 
-    <entry key="U_REGEX_MISSING_CLOSE_BRACKET" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_MISSING_CLOSE_BRACKET" if="!defined(ZORBA_NO_ICU)">
       <value>missing ']'</value>
     </entry>
 
-    <entry key="U_REGEX_NUMBER_TOO_BIG" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_NUMBER_TOO_BIG" if="!defined(ZORBA_NO_ICU)">
       <value>decimal number is too large</value>
     </entry>
 
-    <entry key="U_REGEX_OCTAL_TOO_BIG" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_OCTAL_TOO_BIG" if="!defined(ZORBA_NO_ICU)">
       <value>octal character constants must be &lt;= 0377</value>
     </entry>
 
-    <entry key="U_REGEX_PROPERTY_SYNTAX" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_PROPERTY_SYNTAX" if="!defined(ZORBA_NO_ICU)">
       <value>incorrect Unicode property</value>
     </entry>
 
-    <entry key="U_REGEX_RULE_SYNTAX" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_RULE_SYNTAX" if="!defined(ZORBA_NO_ICU)">
       <value>syntax error</value>
     </entry>
 
-    <entry key="U_REGEX_SET_CONTAINS_STRING" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_SET_CONTAINS_STRING" if="!defined(ZORBA_NO_ICU)">
       <value>can not have UnicodeSets containing strings</value>
     </entry>
 
-    <entry key="U_REGEX_STACK_OVERFLOW" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_STACK_OVERFLOW" if="!defined(ZORBA_NO_ICU)">
       <value>backtrack stack overflow</value>
     </entry>
 
-    <entry key="U_REGEX_STOPPED_BY_CALLER" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_STOPPED_BY_CALLER" if="!defined(ZORBA_NO_ICU)">
       <value>matching operation aborted by user callback fn</value>
     </entry>
 
-    <entry key="U_REGEX_TIME_OUT" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_TIME_OUT" if="!defined(ZORBA_NO_ICU)">
       <value>maximum allowed match time exceeded</value>
     </entry>
 
-    <entry key="U_REGEX_UNIMPLEMENTED" if="!defined(ZORBA_NO_UNICODE)">
-      <value>use of regular expression feature that is not yet implemented</value>
+    <entry key="U_REGEX_UNIMPLEMENTED" if="!defined(ZORBA_NO_ICU)">
+      <value>use of regular expression feature that is not yet implemented</value>
+    </entry>
+
+    <!-- Regex Ascii error messages-->
+    <entry key="REGEX_UNIMPLEMENTED" if="defined(ZORBA_NO_ICU)">
+      <value>use of regular expression feature that is not yet implemented</value>
+    </entry>
+
+    <entry key="REGEX_MISMATCHED_PAREN" if="defined(ZORBA_NO_ICU)">
+      <value>incorrectly nested parentheses</value>
+    </entry>
+
+    <entry key="REGEX_BROKEN_P_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>broken \\p construct</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PL_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{L?} category; supported categories: L, Lu, Ll, Lt, Lm, Lo</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PM_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{M?} category; supported categories: M, Mn, Mc, Me</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PN_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{N?} category; supported categories: N, Nd, Nl, No</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PP_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{P?} category; supported categories: P, Pc, Pd, Ps, Pe, Pi, Pf, Po</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PZ_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{Z?} category; supported categories: Z, Zs, Zl, Zp</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PS_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{S?} category; supported categories: S, Sm, Sc, Sk, So</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PC_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{C?} category; supported categories: C, Cc, Cf, Co, Cn(for not assigned)</value>
+    </entry>
+
+    <entry key="REGEX_BROKEN_PIs_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>broken \\p{Is} construct; valid characters are [a-zA-Z0-9-]</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PIs_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{Is} category block; see supported block escapes here: http://www.w3.org/TR/xmlschema-2/#charcter-classes</value>
+    </entry>
+
+    <entry key="REGEX_INVALID_UNICODE_CODEPOINT_u" if="defined(ZORBA_NO_ICU)">
+      <value>invalid unicode hex, should be in form \\uXXXX or \\UXXXXXXXX</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_ESC_CHAR" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\? escape char; supported escapes are: \\[nrt\\|.?*+(){}[]-^$] for char escapes, \\[pP] for categories and \\[sSiIcCdDwW] for multichar groups</value>
+    </entry>
+
+    <entry key="REGEX_INVALID_BACK_REF" if="defined(ZORBA_NO_ICU)">
+      <value>\\$3 backreference to a non-existent capture group ($4 groups so far)</value>
+    </entry>
+
+    <entry key="REGEX_INVALID_ATOM_CHAR" if="defined(ZORBA_NO_ICU)">
+      <value>'$3': invalid character for an atom; forbidden characters are: [{}?*+|^]</value>
+    </entry>
+
+    <entry key="REGEX_INVALID_SUBCLASS" if="defined(ZORBA_NO_ICU)">
+      <value>malformed class subtraction</value>
+    </entry>
+
+    <entry key="REGEX_INVALID_USE_OF_SUBCLASS" if="defined(ZORBA_NO_ICU)">
+      <value>improper use of class subtraction: it must be the last construct in a class group [xxx-[yyy]]</value>
+    </entry>
+
+    <entry key="REGEX_MULTICHAR_IN_CHAR_RANGE" if="defined(ZORBA_NO_ICU)">
+      <value>multichars or char categories cannot be part of a char range</value>
+    </entry>
+
+    <entry key="REGEX_MISSING_CLOSE_BRACKET" if="defined(ZORBA_NO_ICU)">
+      <value>missing ']' in character group</value>
+    </entry>
+
+    <entry key="REGEX_MAX_LT_MIN" if="defined(ZORBA_NO_ICU)">
+      <value>in {min,max}, max is less than min</value>
     </entry>
 
     <entry key="UnaryArithOp">
       <value>unary arithmetic operator</value>
     </entry>
 
-    <entry key="UnbalancedChar_3">
+    <entry key="UnbalancedChar_3" if="!defined(ZORBA_NO_ICU)">
       <value>missing '$3'</value>
     </entry>
 
+    <entry key="UnescapedChar_3" if="!defined(ZORBA_NO_ICU)">
+      <value>character '$3' must be escaped here</value>
+    </entry>
+
     <entry key="UnexpectedElement">
       <value>unexpected element</value>
     </entry>

=== modified file 'src/diagnostics/pregenerated/dict_en.cpp'
--- src/diagnostics/pregenerated/dict_en.cpp	2012-04-10 13:10:22 +0000
+++ src/diagnostics/pregenerated/dict_en.cpp	2012-04-13 19:45:38 +0000
@@ -437,8 +437,12 @@
   { "~AtomizationOfGroupByMakesMoreThanOneItem", "atomization of groupby variable produces more than one item" },
   { "~AttributeName", "attribute name" },
   { "~AttributeNode", "attribute node" },
+#if !defined(ZORBA_NO_ICU)
   { "~BackRef0Illegal", "\"0\": illegal backreference" },
+#endif
+#if !defined(ZORBA_NO_ICU)
   { "~BackRefIllegalInCharClass", "backreference illegal in character class" },
+#endif
   { "~BadAnyURI", "invalid xs:anyURI" },
   { "~BadArgTypeForFn_2o34o", "${\"2\": }invalid argument type for function $3()${: 4}" },
   { "~BadCharAfter_34", "'$3': illegal character after '$4'" },
@@ -451,7 +455,9 @@
   { "~BadIterator", "invalid iterator" },
   { "~BadLibraryModule", "invalid library module" },
   { "~BadPath", "invalid path" },
+#if !defined(ZORBA_NO_ICU)
   { "~BadRegexEscape_3", "\"$3\": illegal escape character" },
+#endif
   { "~BadStreamState", "bad I/O stream state" },
   { "~BadTokenInBraces_3", "\"$3\": illegal token within { }" },
   { "~BadTraceStream", "trace stream not retrievable using SerializationCallback" },
@@ -567,10 +573,14 @@
   { "~NoUntypedKeyNodeValue_2", "node with untyped key value found during probe on index \"$2\"" },
   { "~NodeIDNeedsBytes_2", "nodeid requires more than $2 bytes" },
   { "~NodeIDTooBig", "nodeid component too big for encoding" },
+#if !defined(ZORBA_NO_ICU)
   { "~NonClosedBackRef_3", "'$$3': non-closed backreference" },
+#endif
   { "~NonFileThesaurusURI", "non-file thesaurus URI" },
   { "~NonLocalhostAuthority", "non-localhost authority" },
+#if !defined(ZORBA_NO_ICU)
   { "~NonexistentBackRef_3", "'$$3': non-existent backreference" },
+#endif
   { "~NotAllowedForTypeName", "not allowed for typeName (use xsd:untyped instead)" },
   { "~NotAmongInScopeSchemaTypes", "not among in-scope schema types" },
   { "~NotDefInDynamicCtx", "not defined in dynamic context" },
@@ -589,6 +599,69 @@
   { "~ParserNoCreateTree", "XML tree creation failed" },
   { "~PromotionImpossible", "promotion not possible" },
   { "~QuotedColon_23", "\"$2\": $3" },
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_BROKEN_PIs_CONSTRUCT", "broken \\p{Is} construct; valid characters are [a-zA-Z0-9-]" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_BROKEN_P_CONSTRUCT", "broken \\p construct" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_INVALID_ATOM_CHAR", "'$3': invalid character for an atom; forbidden characters are: [{}?*+|^]" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_INVALID_BACK_REF", "\\$3 backreference to a non-existent capture group ($4 groups so far)" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_INVALID_SUBCLASS", "malformed class subtraction" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_INVALID_UNICODE_CODEPOINT_u", "invalid unicode hex, should be in form \\uXXXX or \\UXXXXXXXX" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_INVALID_USE_OF_SUBCLASS", "improper use of class subtraction: it must be the last construct in a class group [xxx-[yyy]]" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_MAX_LT_MIN", "in {min,max}, max is less than min" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_MISMATCHED_PAREN", "incorrectly nested parentheses" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_MISSING_CLOSE_BRACKET", "missing ']' in character group" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_MULTICHAR_IN_CHAR_RANGE", "multichars or char categories cannot be part of a char range" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNIMPLEMENTED", "use of regular expression feature that is not yet implemented" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_ESC_CHAR", "unknown \\? escape char; supported escapes are: \\[nrt\\|.?*+(){}[]-^$] for char escapes, \\[pP] for categories and \\[sSiIcCdDwW] for multichar groups" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PC_CONSTRUCT", "unknown \\p{C?} category; supported categories: C, Cc, Cf, Co, Cn(for not assigned)" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PIs_CONSTRUCT", "unknown \\p{Is} category block; see supported block escapes here: http://www.w3.org/TR/xmlschema-2/#charcter-classes" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PL_CONSTRUCT", "unknown \\p{L?} category; supported categories: L, Lu, Ll, Lt, Lm, Lo" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PM_CONSTRUCT", "unknown \\p{M?} category; supported categories: M, Mn, Mc, Me" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PN_CONSTRUCT", "unknown \\p{N?} category; supported categories: N, Nd, Nl, No" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PP_CONSTRUCT", "unknown \\p{P?} category; supported categories: P, Pc, Pd, Ps, Pe, Pi, Pf, Po" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PS_CONSTRUCT", "unknown \\p{S?} category; supported categories: S, Sm, Sc, Sk, So" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PZ_CONSTRUCT", "unknown \\p{Z?} category; supported categories: Z, Zs, Zl, Zp" },
+#endif
   { "~SEPM0009_Not10", "the version parameter has a value other than \"1.0\" and the doctype-system parameter is specified" },
   { "~SEPM0009_NotOmit", "the standalone attribute has a value other than \"omit\"" },
   { "~SchemaAttributeName", "schema-attribute name" },
@@ -610,68 +683,73 @@
   { "~TwoDecimalFormatsSameName_2", "\"$2\": two decimal formats with this name" },
   { "~TwoDefaultDecimalFormats", "two default decimal formats" },
   { "~TypeIsNotSubtype", "item type is not a subtype of \"$3\"" },
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_BAD_ESCAPE_SEQUENCE", "unrecognized backslash escape sequence" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_BAD_INTERVAL", "error in {min,max} interval" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_INTERNAL_ERROR", "an internal ICU error (bug) was detected" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_INVALID_BACK_REF", "backreference to a non-existent capture group" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_INVALID_FLAG", "invalid value for match mode flags" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_INVALID_RANGE", "in character range [x-y], x is greater than y" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_INVALID_STATE", "RegexMatcher in invalid state for requested operation" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_LOOK_BEHIND_LIMIT", "look-behind pattern matches must have a bounded maximum length" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_MAX_LT_MIN", "in {min,max}, max is less than min" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_MISMATCHED_PAREN", "incorrectly nested parentheses" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_MISSING_CLOSE_BRACKET", "missing ']'" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_NUMBER_TOO_BIG", "decimal number is too large" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_OCTAL_TOO_BIG", "octal character constants must be <= 0377" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_PROPERTY_SYNTAX", "incorrect Unicode property" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_RULE_SYNTAX", "syntax error" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_SET_CONTAINS_STRING", "can not have UnicodeSets containing strings" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_STACK_OVERFLOW", "backtrack stack overflow" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_STOPPED_BY_CALLER", "matching operation aborted by user callback fn" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_TIME_OUT", "maximum allowed match time exceeded" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_UNIMPLEMENTED", "use of regular expression feature that is not yet implemented" },
 #endif
   { "~UnaryArithOp", "unary arithmetic operator" },
+#if !defined(ZORBA_NO_ICU)
   { "~UnbalancedChar_3", "missing '$3'" },
+#endif
+#if !defined(ZORBA_NO_ICU)
+  { "~UnescapedChar_3", "character '$3' must be escaped here" },
+#endif
   { "~UnexpectedElement", "unexpected element" },
   { "~VarValMustBeSingleItem_2", "\"$2\": variable value must be single item" },
   { "~Variable", "variable" },

=== modified file 'src/precompiled/stdafx.h'
--- src/precompiled/stdafx.h	2012-03-28 05:19:57 +0000
+++ src/precompiled/stdafx.h	2012-04-13 19:45:38 +0000
@@ -15,363 +15,81 @@
 
  */
  
-#if defined STDAFX
-#include <iostream>
-#include <stdexcept>
-#include <cassert>
-#include <cstring>
-#include <memory>
-
-#include <sstream>
-#include <xfwrap>
-#include <xfwrap1>
-#include <istream>
-#include <cstdio>
-#include <xxshared>
-#include <crtdefs.h>
-#include <map>
-#include <set>
-//#include <poppack.h>
-//#include <xxtype_traits>
-//#include <xxcallwrap>
-
-// #include <xxcallpmf>
-// //#include <xxbind0>
-// //#include <xxbind1>
-// //#include <xxresult>
-// #include <zorba/audit.h>
-// #include "api/auditimpl.h"
-// #include <zorba/audit.h>
-
- //#include "unicode/unistr.h"
- #include "runtime/sequences/sequences.h"
- #include "diagnostics/xquery_diagnostics.h"
- #include "xercesc/util/xercesdefs.hpp"
- #include "runtime/collections/collections.h"
- #include "unicode/utypes.h"
- #include "zorba/config.h"
- #include "store/api/store.h"
- #include "zorba/zorba.h"
- #include "zorba/api_shared_types.h"
- #include "compiler/parsetree/parsenodes.h"
- #include "compiler/parser/parse_constants.h"
- //#include "compiler/api/compilercb.h"
- #include "zorbautils/checked_vector.h"
- #include "compiler/parser/xquery_driver.h"
- #include "util/sorter.h"
- #include "compiler/xqueryx/xqueryx_to_xquery.h"
-// #include "compiler/xqueryx/xqueryx_xslt.h"
-//#include "compiler/parser/xquery_scanner.h"
-//#include "compiler/parsetree/parsenode_base.h"
-//#include "compiler/parsetree/parsenode_visitor.h"
-// #include "runtime/core/flwor_iterator.h"
-// #include "context/static_context.h"
-// #include "zorbautils/fatal.h"
-// #include "runtime/base/unarybase.h"
-// #include "compiler/expression/expr_consts.h"
-// #include "api/iterator_singleton.h"
-// #include "runtime/visitors/printer_visitor_api.h"
-// //#include "compiler/parsetree/parsenode_print_dot_visitor.h"
-// //#include "compiler/parsetree/parsenode_print_dot_visitor.h"
-// //#include "runtime/visitors/planiter_visitor_impl_code.h"
-// //#include "runtime/visitors/planiter_visitor_impl_include.h"
-// //#include "runtime/visitors/printer_visitor_impl.h"
-// //#include "runtime/core/path.h"
-// #include "compiler/expression/ft_expr.h"
-// #include "compiler/expression/ftnode.h"
-// #include "compiler/parser/query_loc.h"
+#ifdef STDAFX
+
+  #include <fstream>
+  #include <iostream>
+  #include <stdexcept>
+  #include <cassert>
+  #include <cstring>
+  #include <memory>
+
+  #include <sstream>
+  #include <xfwrap>
+  #include <xfwrap1>
+  #include <istream>
+  #include <cstdio>
+  #include <xxshared>
+  #include <crtdefs.h>
+  #include <map>
+  #include <set>
+
+  #include "runtime/sequences/sequences.h"
+  #include "diagnostics/xquery_diagnostics.h"
+  #include "xercesc/util/xercesdefs.hpp"
+  #include "runtime/collections/collections.h"
+  #include "unicode/utypes.h"
+  #include "zorba/config.h"
+  #include "store/api/store.h"
+  #include "zorba/zorba.h"
+  #include "zorba/api_shared_types.h"
+  #include "compiler/parsetree/parsenodes.h"
+  #include "compiler/parser/parse_constants.h"
+  #include "zorbautils/checked_vector.h"
+  #include "compiler/parser/xquery_driver.h"
+  #include "util/sorter.h"
+  #include "compiler/xqueryx/xqueryx_to_xquery.h"
+  #include <zorba/store_manager.h>
+  #include <zorba/xquery.h>
+  #include <zorba/xquery_exception.h>
   #include "util/cxx_util.h"
-// #include "util/indent.h"
-// #include "util/stl_util.h"
-// #include "diagnostics/xquery_diagnostics.h"
-// #include "zorbatypes/numconversions.h"
+  #include "diagnostics/assert.h"
+  #include "zorbatypes/mapm/m_apm_lc.h"
+  #include "zorbatypes/datetime/parse.h"
+  #include "zorbatypes/chartype.h"
+  #include "zorbatypes/collation_manager.h"
+  #include "zorbatypes/ft_token.h"
+  #include "zorbatypes/m_apm.h"
+  #include "zorbatypes/rclock.h"
+  #include "zorbatypes/schema_types.h"
+  #include "zorbatypes/timezone.h"
+  #include "zorbatypes/transcoder.h"
+  #include "zorbatypes/URI.h"
+  #include "zorbatypes/xerces_xmlcharray.h"
+  #include "zorbatypes/zorbatypes_decl.h"
+  #include "zorbatypes/zstring.h"
+  #include "zorbautils/condition.h"
+  #include "zorbautils/hashfun.h"
+  #include "zorbautils/hashmap.h"
+  #include "zorbautils/hashmap_itemp.h"
+  #include "zorbautils/hashmap_str_obj.h"
+  #include "zorbautils/hashmap_zstring.h"
+  #include "zorbautils/hashset.h"
+  #include "zorbautils/hashset_itemh.h"
+  #include "zorbautils/latch.h"
+  #include "zorbautils/locale.h"
+  #include "zorbautils/lock.h"
+  #include "zorbautils/mutex.h"
+  #include "zorbautils/runnable.h"
+  #include "zorbautils/SAXParser.h"
+  #include "zorbautils/stack.h"
+  #include "zorbautils/string_util.h"
+  #include "unit_tests/unit_test_list.h"
+  #include "zorba/diagnostic_handler.h"
+  #include "zorba/xquery_warning.h"
+  #include "runtime/full_text/ftcontains_visitor.h"
+  #include "store/api/ft_token_iterator.h"
+  #include "store/naive/ft_token_store.h"
 
-// #include "api/serialization/serializable.h"
-// #include "api/serialization/serializer.h"
-// #include "api/collectionimpl.h"
-// #include "api/dynamiccontextimpl.h"
-// #include "api/fileimpl.h"
-// #include "api/functionimpl.h"
-// #include "api/invoke_item_sequence.h"
-// #include "api/itemfactoryimpl.h"
-// #include "api/resultiteratorchainer.h"
-// #include "api/resultiteratorimpl.h"
-// #include "api/sax2impl.h"
-// #include "api/serializerimpl.h"
-// #include "api/staticcontextimpl.h"
-// #include "api/storeiteratorimpl.h"
-// #include "api/unmarshaller.h"
-// #include "api/uri_resolver_wrappers.h"
-// #include "api/vectoriterator.h"
-// #include "api/xmldatamanagerimpl.h"
-// //#include "api/xqueryimpl.h"
-// #include "api/zorbaimpl.h"
-// #include "capi/cdynamic_context.h"
-// #include "capi/cexpression.h"
-// #include "capi/cexternal_function.h"
-// #include "capi/cimplementation.h"
-// #include "capi/csequence.h"
-// #include "capi/cstatic_context.h"
-// #include "capi/error.h"
-// #include "capi/external_module.h"
-// #include "capi/single_item_sequence.h"
-// #include "capi/user_item_sequence.h"
-// #include "compiler/parser/flexlexer.h"
-// #include "compiler/parser/ft_types.h"
-// #include "compiler/parser/symbol_table.h"
-// #include "compiler/parser/xqdoc_comment.h"
-// #include "compiler/parsetree/parsenode_print_xml_visitor.h"
-// #include "compiler/parsetree/parsenode_print_xqdoc_visitor.h"
-// #include "compiler/parsetree/parsenode_print_xquery_visitor.h"
-// #include "compiler/parsetree/parsenode_xqdoc_visitor.h"
-// #include "compiler/translator/prolog_graph.h"
-// #include "compiler/translator/translator.h"
-// #include "compiler/codegen/plan_visitor.h"
-// #include "compiler/expression/abstract_expr_visitor.h"
-// #include "compiler/expression/expr.h"
-// #include "compiler/expression/expr_annotations.h"
-// #include "compiler/expression/expr_base.h"
-// #include "compiler/expression/expr_classes.h"
-// #include "compiler/expression/expr_iter.h"
-// #include "compiler/expression/expr_utils.h"
-// #include "compiler/expression/expr_visitor.h"
-// #include "compiler/expression/flwor_expr.h"
-// //#include "compiler/expression/fo_expr.h"
-// #include "compiler/expression/ftnode_classes.h"
-// #include "compiler/expression/ftnode_visitor.h"
-// #include "compiler/expression/function_item_expr.h"
-// #include "compiler/expression/path_expr.h"
-// #include "compiler/expression/script_exprs.h"
-// #include "compiler/expression/update_exprs.h"
-// #include "compiler/expression/var_expr.h"
-// #include "compiler/rewriter/framework/rewriter.h"
-// #include "compiler/rewriter/framework/rewriter_context.h"
-// #include "compiler/rewriter/framework/rule_driver.h"
-// #include "compiler/rewriter/framework/sequential_rewriter.h"
-// #include "compiler/rewriter/rewriters/common_rewriter.h"
-// #include "compiler/rewriter/rewriters/default_optimizer.h"
-// #include "compiler/rewriter/rewriters/phase1_rewriter.h"
-// #include "compiler/rewriter/rules/ruleset.h"
-// #include "compiler/rewriter/rules/rule_base.h"
-// #include "compiler/rewriter/rules/type_rules.h"
-// #include "compiler/rewriter/tools/dataflow_annotations.h"
-// #include "compiler/rewriter/tools/expr_tools.h"
-// #include "compiler/rewriter/tools/udf_graph.h"
-// #include "compiler/xqddf/collection_decl.h"
-// #include "compiler/xqddf/value_ic.h"
-// #include "compiler/xqddf/value_index.h"
-// #include "compiler/semantic_annotations/annotations.h"
-// #include "compiler/semantic_annotations/annotation_holder.h"
-// #include "compiler/semantic_annotations/annotation_keys.h"
-// #include "compiler/api/compiler_api.h"
-// #include "compiler/api/compiler_api_impl.h"
-// #include "system/globalenv.h"
-// #include "system/properties.h"
-// #include "system/zorba_properties.h"
-// #include "context/decimal_format.h"
-// #include "context/default_uri_mappers.h"
-// #include "context/default_url_resolvers.h"
-// #include "context/dynamic_context.h"
-// #include "context/dynamic_loader.h"
-// #include "context/internal_uri_resolvers.h"
-// //#include "context/namespace_context.h"
-// #include "context/root_static_context.h"
-// #include "context/sctx_map_iterator.h"
-// #include "context/standard_uri_resolvers.h"
-// #include "context/static_context_consts.h"
-// #include "context/stemmer_wrappers.h"
-// #include "context/uri_resolver.h"
-// #include "context/uri_resolver_wrapper.h"
-#include "diagnostics/assert.h"
-// #include "diagnostics/diagnostic.h"
-// #include "diagnostics/dict.h"
-// #include "diagnostics/dict_impl.h"
-// #include "diagnostics/StackWalker.h"
-// #include "diagnostics/user_error.h"
-// #include "diagnostics/user_exception.h"
-// #include "diagnostics/xquery_exception.h"
-// #include "diagnostics/xquery_stack_trace.h"
-// #include "diagnostics/xquery_warning.h"
-// #include "diagnostics/zorba_exception.h"
-// //#include "functions/annotation.h"
-// #include "functions/external_function.h"
-// #include "functions/function.h"
-// #include "functions/function_consts.h"
-// #include "functions/function_impl.h"
-// #include "functions/func_accessors_impl.h"
-// #include "functions/func_apply.h"
-// #include "functions/func_arithmetic.h"
-// #include "functions/func_booleans_impl.h"
-// #include "functions/func_durations_dates_times_impl.h"
-// #include "functions/func_enclosed.h"
-// #include "functions/func_eval.h"
-// #include "functions/func_hoist.h"
-// #include "functions/func_index_ddl.h"
-// #include "functions/func_node_sort_distinct.h"
-// #include "functions/func_numerics_impl.h"
-// #include "functions/func_reflection.h"
-// #include "functions/func_sequences_impl.h"
-// #include "functions/func_var_decl.h"
-// #include "functions/library.h"
-// #include "functions/signature.h"
-// #include "functions/udf.h"
-// #include "runtime/full_text/thesauri/decode_base128.h"
-// #include "runtime/full_text/thesauri/encoded_list.h"
-// #include "runtime/full_text/thesauri/iso2788.h"
-// #include "runtime/full_text/thesauri/wn_db_segment.h"
-// #include "runtime/full_text/thesauri/wn_synset.h"
-// #include "runtime/full_text/thesauri/wn_thesaurus.h"
-// #include "runtime/full_text/thesauri/wn_types.h"
-// #include "runtime/full_text/thesauri/xqftts_relationship.h"
-// #include "runtime/full_text/thesauri/xqftts_thesaurus.h"
-// #include "runtime/full_text/ft_match.h"
-// #include "runtime/full_text/ft_query_item.h"
-// #include "runtime/full_text/ft_single_token_iterator.h"
-// #include "runtime/full_text/ft_stop_words_set.h"
-// #include "runtime/full_text/ft_thesaurus.h"
-// #include "runtime/full_text/ft_token_matcher.h"
-// #include "runtime/full_text/ft_token_seq_iterator.h"
-// #include "runtime/full_text/ft_token_span.h"
-// #include "runtime/full_text/ft_wildcard.h"
-// #include "runtime/full_text/full_text.h"
-// #include "runtime/full_text/apply.h"
-// #include "runtime/full_text/ft_util.h"
-// #include "runtime/collections/collections_base.h"
-// #include "runtime/core/apply_updates.h"
-// #include "runtime/core/arithmetic_impl.h"
-// #include "runtime/core/constructors.h"
-// #include "runtime/core/fncall_iterator.h"
-// #include "runtime/core/internal_operators.h"
-// #include "runtime/core/item_iterator.h"
-// #include "runtime/core/nodeid_iterators.h"
-// #include "runtime/core/path_iterators.h"
-// #include "runtime/core/sequencetypes.h"
-// #include "runtime/core/trycatch.h"
-// #include "runtime/core/var_iterators.h"
-// #include "runtime/numerics/NumericsImpl.h"
-// #include "runtime/booleans/BooleanImpl.h"
-// #include "runtime/base/binarybase.h"
-// #include "runtime/base/narybase.h"
-// #include "runtime/base/noarybase.h"
-// #include "runtime/base/plan_iterator.h"
-// #include "runtime/sequences/SequencesImpl.h"
-// #include "runtime/visitors/iterprinter.h"
-// #include "runtime/misc/materialize.h"
-// #include "runtime/scripting/scripting.h"
-// #include "types/schema/EventSchemaValidator.h"
-// #include "types/schema/LoadSchemaErrorHandler.h"
-// #include "types/schema/PrintSchema.h"
-// #include "types/schema/revalidateUtils.h"
-// #include "types/schema/schema.h"
-// #include "types/schema/SchemaValidatorFilter.h"
-// #include "types/schema/StrX.h"
-// #include "types/schema/validate.h"
-// #include "types/schema/ValidationEventHandler.h"
-// #include "types/schema/xercesIncludes.h"
-// #include "types/schema/XercesParseUtils.h"
-// #include "types/schema/XercSchemaValidator.h"
-// #include "types/casting.h"
-// #include "types/collation.h"
-// #include "types/node_test.h"
-// #include "types/root_typemanager.h"
-// #include "types/typeconstants.h"
-// #include "types/typeimpl.h"
-// #include "types/typemanager.h"
-// #include "types/typemanagerimpl.h"
-// #include "types/typeops.h"
-// #include "util/fx/fxarray.h"
-// #include "util/fx/fxcharheap.h"
-// #include "util/ascii_util.h"
-// #include "util/atomic_int.h"
-// #include "util/auto_vector.h"
-// #include "util/curl_util.h"
-// #include "util/dir.h"
-// #include "util/dynamic_bitset.h"
-// #include "util/empty.h"
-// #include "util/error_util.h"
-// #include "util/fs_util.h"
-// #include "util/hashmap.h"
-// //#include "util/hashmap32.h"
-// #include "util/less.h"
-// #include "util/mmap_file.h"
-// #include "util/nonatomic_int.h"
-// #include "util/omanip.h"
-// #include "util/oseparator.h"
-// #include "util/regex.h"
-// #include "util/singleton.h"
-// #include "util/string_util.h"
-// #include "util/threads.h"
-// #include "util/tokenbuf.h"
-// #include "util/tracer.h"
-// #include "util/triple.h"
-// #include "util/unicode_categories.h"
-// #include "util/unicode_util.h"
-// #include "util/uri_util.h"
-// #include "util/utf8_string.h"
-// #include "util/utf8_util.h"
-// #include "util/utf8_util_base.h"
-// #include "util/void_int.h"
-// #include "util/xml_util.h"
-// #include "zorbamisc/config/platform.h"
-// //#include "zorbaserialization/archiver.h"
-// #include "zorbaserialization/base64impl.h"
-// #include "zorbaserialization/bin_archiver.h"
-// //#include "zorbaserialization/class_serializer.h"
-// #include "zorbaserialization/mem_archiver.h"
-// #include "zorbaserialization/serialization_engine.h"
-// #include "zorbaserialization/template_serializer.h"
-// #include "zorbaserialization/xml_archiver.h"
-// #include "zorbaserialization/zorba_class_serializer.h"
- #include "zorbatypes/mapm/m_apm_lc.h"
- #include "zorbatypes/datetime/parse.h"
- //#include "zorbatypes/binary.h"
- #include "zorbatypes/chartype.h"
- #include "zorbatypes/collation_manager.h"
- //#include "zorbatypes/datetime.h"
- //#include "zorbatypes/decimal.h"
- //#include "zorbatypes/duration.h"
- //#include "zorbatypes/floatimpl.h"
- #include "zorbatypes/ft_token.h"
- //#include "zorbatypes/integer.h"
- #include "zorbatypes/libicu.h"
- #include "zorbatypes/m_apm.h"
- //#include "zorbatypes/rchandle.h"
- #include "zorbatypes/rclock.h"
- //#include "zorbatypes/regex_ascii.h"
- #include "zorbatypes/schema_types.h"
- #include "zorbatypes/timezone.h"
- #include "zorbatypes/transcoder.h"
- #include "zorbatypes/URI.h"
- #include "zorbatypes/xerces_xmlcharray.h"
- #include "zorbatypes/zorbatypes_decl.h"
- #include "zorbatypes/zstring.h"
- //#include "zorbautils/stemmer/sb_stemmer.h"
- #include "zorbautils/condition.h"
- #include "zorbautils/hashfun.h"
- #include "zorbautils/hashmap.h"
- #include "zorbautils/hashmap_itemp.h"
- #include "zorbautils/hashmap_str_obj.h"
- #include "zorbautils/hashmap_zstring.h"
- #include "zorbautils/hashset.h"
- #include "zorbautils/hashset_itemh.h"
- //#include "zorbautils/icu_tokenizer.h"
- #include "zorbautils/latch.h"
- #include "zorbautils/locale.h"
- #include "zorbautils/lock.h"
- #include "zorbautils/mutex.h"
- #include "zorbautils/runnable.h"
- #include "zorbautils/SAXParser.h"
- #include "zorbautils/stack.h"
-// #include "zorbautils/stemmer.h"
- #include "zorbautils/string_util.h"
- //#include "zorbautils/synchronous_logger.h"
- //#include "zorbautils/tokenizer.h"
- #include "unit_tests/unit_test_list.h"
- #include "zorba/diagnostic_handler.h"
- #include "zorba/xquery_warning.h"
- #include "runtime/full_text/ftcontains_visitor.h"
- #include "store/naive/naive_ft_token_iterator.h"
- #include "store/api/ft_token_iterator.h"
- #include "store/naive/ft_token_store.h"
 #endif
 /* vim:set et sw=2 ts=2: */

=== modified file 'src/runtime/full_text/CMakeLists.txt'
--- src/runtime/full_text/CMakeLists.txt	2012-03-28 05:19:57 +0000
+++ src/runtime/full_text/CMakeLists.txt	2012-04-13 19:45:38 +0000
@@ -42,11 +42,11 @@
     default_tokenizer.cpp
     )
 
-IF (ZORBA_NO_UNICODE)
+IF (ZORBA_NO_ICU)
   LIST(APPEND FULLTEXT_SRCS latin_tokenizer.cpp)
-ELSE (ZORBA_NO_UNICODE)
+ELSE (ZORBA_NO_ICU)
   LIST(APPEND FULLTEXT_SRCS icu_tokenizer.cpp)
-ENDIF (ZORBA_NO_UNICODE)
+ENDIF (ZORBA_NO_ICU)
 
 ADD_SRC_SUBFOLDER(FULLTEXT_SRCS stemmer LIBSTEMMER_SRCS)
 

=== modified file 'src/runtime/full_text/default_tokenizer.cpp'
--- src/runtime/full_text/default_tokenizer.cpp	2012-03-28 05:19:57 +0000
+++ src/runtime/full_text/default_tokenizer.cpp	2012-04-13 19:45:38 +0000
@@ -19,22 +19,22 @@
 #include <zorba/config.h>
 
 #include "default_tokenizer.h"
-#ifdef ZORBA_NO_UNICODE
+#ifdef ZORBA_NO_ICU
 # include "latin_tokenizer.h"
 #else
 # include "icu_tokenizer.h"
-#endif /* ZORBA_NO_UNICODE */
+#endif /* ZORBA_NO_ICU */
 
 namespace zorba {
 
 ///////////////////////////////////////////////////////////////////////////////
 
 TokenizerProvider const& default_tokenizer_provider() {
-#ifdef ZORBA_NO_UNICODE
+#ifdef ZORBA_NO_ICU
   static LatinTokenizerProvider const instance;
 #else
   static ICU_TokenizerProvider const instance;
-#endif /* ZORBA_NO_UNICODE */
+#endif /* ZORBA_NO_ICU */
   return instance;
 };
 

=== modified file 'src/runtime/full_text/latin_tokenizer.cpp'
--- src/runtime/full_text/latin_tokenizer.cpp	2012-03-28 05:19:57 +0000
+++ src/runtime/full_text/latin_tokenizer.cpp	2012-04-13 19:45:38 +0000
@@ -18,8 +18,9 @@
 #include <functional>
 
 #include <zorba/diagnostic_list.h>
-#include <zorba/xquery_exception.h>
-#include <zorba/zorba.h>
+
+#include "diagnostics/dict.h"
+#include "diagnostics/xquery_exception.h"
 
 #include "latin_tokenizer.h"
 

=== modified file 'src/runtime/full_text/latin_tokenizer.h'
--- src/runtime/full_text/latin_tokenizer.h	2012-03-28 05:19:57 +0000
+++ src/runtime/full_text/latin_tokenizer.h	2012-04-13 19:45:38 +0000
@@ -14,12 +14,12 @@
  * limitations under the License.
  */
 
-#ifndef ZORBA_WESTERN_TOKENIZER_H
-#define ZORBA_WESTERN_TOKENIZER_H
+#ifndef ZORBA_LATIN_TOKENIZER_H
+#define ZORBA_LATIN_TOKENIZER_H
 
 #include <zorba/config.h>
 
-#ifdef ZORBA_NO_FULL_TEXT
+#ifdef ZORBA_NO_ICU
 
 #include <zorba/tokenizer.h>
 #include "zorbatypes/zstring.h"
@@ -38,8 +38,8 @@
 
   // inherited
   void destroy() const;
-  void tokenize( char const*, size_type, iso639_1::type, bool, Callback&,
-                 void* );
+  void tokenize( char const*, size_type, locale::iso639_1::type, bool,
+                 Callback&, void* );
 
 private:
   typedef zstring string_type;
@@ -64,13 +64,14 @@
 class LatinTokenizerProvider : public TokenizerProvider {
 public:
   // inherited
-  Tokenizer::ptr getTokenizer( iso639_1::type, Tokenizer::Numbers& ) const;
+  Tokenizer::ptr getTokenizer( locale::iso639_1::type,
+                               Tokenizer::Numbers& ) const;
 };
 
 ///////////////////////////////////////////////////////////////////////////////
 
 } // namespace zorba
 
-#endif /* ZORBA_NO_FULL_TEXT */
-#endif /* ZORBA_WESTERN_TOKENIZER_H */
+#endif /* ZORBA_NO_ICU */
+#endif /* ZORBA_LATIN_TOKENIZER_H */
 /* vim:set et sw=2 ts=2: */

=== modified file 'src/runtime/numerics/format_integer_impl.cpp'
--- src/runtime/numerics/format_integer_impl.cpp	2012-03-28 05:19:57 +0000
+++ src/runtime/numerics/format_integer_impl.cpp	2012-04-13 19:45:38 +0000
@@ -881,7 +881,7 @@
             utf8_result += (*valueit);
           }
           else
-            utf8_result += (0x2080 + *valueit - '0');
+            utf8_result += (unicode::code_point)(0x2080 + *valueit - '0');
         }
       }
       else if((c0 == 0x2460) || //CIRCLED DIGIT ONE  (1-20)

=== modified file 'src/runtime/numerics/numerics_impl.cpp'
--- src/runtime/numerics/numerics_impl.cpp	2012-03-28 05:19:57 +0000
+++ src/runtime/numerics/numerics_impl.cpp	2012-04-13 19:45:38 +0000
@@ -462,7 +462,7 @@
     minus( "-" )
   {
     utf8_string<zstring> u_per_mille( per_mille );
-    u_per_mille = 0x2030;
+    u_per_mille = (unicode::code_point)0x2030;
   }
 
   void readFormat(const DecimalFormat_t& df_t)

=== modified file 'src/runtime/strings/strings_impl.cpp'
--- src/runtime/strings/strings_impl.cpp	2012-03-28 05:19:57 +0000
+++ src/runtime/strings/strings_impl.cpp	2012-04-13 19:45:38 +0000
@@ -810,7 +810,9 @@
   zstring normForm;
   zstring resStr;
   unicode::normalization::type normType;
+#ifndef ZORBA_NO_ICU
   bool success;
+#endif /* ZORBA_NO_ICU */
 
   PlanIteratorState* state;
   DEFAULT_STACK_INIT(PlanIteratorState, state, planState);
@@ -860,10 +862,10 @@
     }
 
     item0->getStringValue2(resStr);
-#ifndef ZORBA_NO_UNICODE
+#ifndef ZORBA_NO_ICU
     success = utf8::normalize(resStr, normType, &resStr);
     ZORBA_ASSERT(success);
-#endif//#ifndef ZORBA_NO_UNICODE
+#endif//#ifndef ZORBA_NO_ICU
     STACK_PUSH(GENV_ITEMFACTORY->createString(result, resStr), state );
   }
   else
@@ -992,7 +994,7 @@
         trans_map[ *map_i ] = *trans_i;
 
       for ( ; map_i != map_end; ++map_i )
-        trans_map[ *map_i ] = ~0;
+        trans_map[ *map_i ] = static_cast<unicode::code_point>( ~0 );
     }
 
     utf8_string<zstring> u_result_string( result_string );
@@ -1007,7 +1009,7 @@
       cp_map_type::const_iterator const found_i = trans_map.find( cp );
       if ( found_i != trans_map.end() ) {
         cp = found_i->second;
-        if ( cp == ~0 )
+        if ( cp == static_cast<unicode::code_point>( ~0 ) )
           continue;
       }
       u_result_string += cp;
@@ -1795,16 +1797,33 @@
                           int &utf8start,
                           unsigned int &bytestart,
                           int utf8end,
+                          unsigned int byteend,
                           zstring &out)
 {
+#ifndef ZORBA_NO_ICU
   utf8::size_type clen;
-  while(utf8start < utf8end)
-  {
-    clen = utf8::char_length(*sin);
-    out.append(sin, clen);
-    utf8start++;
-    bytestart += clen;
-    sin += clen;
+  if(utf8end)
+  {
+    while(utf8start < utf8end)
+    {
+      clen = utf8::char_length(*sin);
+      if(clen == 0)
+        clen = 1;
+      out.append(sin, clen);
+      utf8start++;
+      bytestart += clen;
+      sin += clen;
+    }
+  }
+  else
+#endif
+  {
+    if(!utf8end)
+      utf8end = byteend;
+    out.append(sin, utf8end-bytestart);
+    sin += utf8end-bytestart;
+    utf8start = utf8end;
+    bytestart = utf8end;
   }
 }
 
@@ -1812,6 +1831,7 @@
                                int &match_end1,
                                unsigned int &match_end1_bytes,
                                int match_start2,
+                               unsigned int match_start2_bytes,
                                const char *&strin)
 {
   store::Item_t non_match_elem;
@@ -1833,7 +1853,7 @@
   //  utf8_it++;
   //  match_end1++;
   //}
-  copyUtf8Chars(strin, match_end1, match_end1_bytes, match_start2, non_match_str);
+  copyUtf8Chars(strin, match_end1, match_end1_bytes, match_start2, match_start2_bytes, non_match_str);
   store::Item_t non_match_text_item;
   GENV_ITEMFACTORY->createTextNode(non_match_text_item, non_match_elem, non_match_str);
 }
@@ -1864,19 +1884,31 @@
       i--;
       break;
     }
+#ifndef ZORBA_NO_ICU
     match_startg = rx.get_match_start(i+1);
     if((match_startg < 0) && (gparent < 0))
       continue;
+#else
+    int temp_endg;
+    match_startg = -1;
+    temp_endg = -1;
+    if(!rx.get_match_start_end_bytes(i+1, &match_startg, &temp_endg) && (gparent < 0))
+      continue;
+#endif
     if(match_endgood < match_startg)
     {
       //add non-group match text
       zstring                non_group_str;
 
-      copyUtf8Chars(sin, match_endgood, match_end1_bytes, match_startg, non_group_str);
+      copyUtf8Chars(sin, match_endgood, match_end1_bytes, match_startg, 0, non_group_str);
       store::Item_t non_group_text_item;
       GENV_ITEMFACTORY->createTextNode(non_group_text_item, parent.getp(), non_group_str);
     }
+#ifndef ZORBA_NO_ICU
     match_endg = rx.get_match_end(i+1);
+#else
+    match_endg = temp_endg;
+#endif
     //add group match text
     GENV_ITEMFACTORY->createQName(group_element_name,
                                   static_context::W3C_FN_NS, "fn", "group");
@@ -1907,7 +1939,7 @@
     }
     zstring                group_str;
 
-    copyUtf8Chars(sin, match_startg, match_end1_bytes, match_endg, group_str);
+    copyUtf8Chars(sin, match_startg, match_end1_bytes, match_endg, 0, group_str);
     store::Item_t group_text_item;
     GENV_ITEMFACTORY->createTextNode(group_text_item, group_elem.getp(), group_str);
   }
@@ -1916,7 +1948,7 @@
   {
     zstring                non_group_str;
 
-    copyUtf8Chars(sin, match_endgood, match_end1_bytes, match_end2, non_group_str);
+    copyUtf8Chars(sin, match_endgood, match_end1_bytes, match_end2, 0, non_group_str);
     store::Item_t non_group_text_item;
     GENV_ITEMFACTORY->createTextNode(non_group_text_item, parent, non_group_str);
   }
@@ -2144,8 +2176,14 @@
       reachedEnd = false;
       while(rx.find_next_match(&reachedEnd))
       {
-        int    match_start2 = rx.get_match_start();
-        int    match_end2 = rx.get_match_end();
+        int    match_start2;
+        int    match_end2;
+#ifndef ZORBA_NO_ICU
+        match_start2 = rx.get_match_start();
+        match_end2 = rx.get_match_end();
+#else
+        rx.get_match_start_end_bytes(0, &match_start2, &match_end2);
+#endif
         ZORBA_ASSERT(match_start2 >= 0);
 
         if(is_input_stream && reachedEnd && !instream->eof())
@@ -2157,7 +2195,7 @@
         //construct the fn:non-match
         if(match_start2 > match_end1)
         {
-          addNonMatchElement(result, match_end1, match_end1_bytes, match_start2, instr);
+          addNonMatchElement(result, match_end1, match_end1_bytes, match_start2, 0, instr);
         }
 
         //construct the fn:match
@@ -2165,7 +2203,7 @@
         match_end1 = match_end2;
       }
 
-      if(is_input_stream && reachedEnd && !instream->eof())
+      if(is_input_stream && !instream->eof())
       {
         //load some more data, maybe the match will be different
         if(match_end1_bytes)
@@ -2213,7 +2251,7 @@
       else
       {
         if(match_end1_bytes < streambuf_read)
-          addNonMatchElement(result, match_end1, match_end1_bytes, streambuf_read, instr);
+          addNonMatchElement(result, match_end1, match_end1_bytes, 0, streambuf_read, instr);
         if(is_input_stream && instream->eof())
           reachedEnd = true;
       }

=== modified file 'src/store/api/store.h'
--- src/store/api/store.h	2012-04-10 20:59:34 +0000
+++ src/store/api/store.h	2012-04-13 19:45:38 +0000
@@ -16,7 +16,7 @@
 #ifndef ZORBA_STORE_STORE_H
 #define ZORBA_STORE_STORE_H
 
-#include <zorba/config.h>
+#include "zorba/config.h"
 #include "zorbatypes/schema_types.h"
 
 #include "store/api/shared_types.h"

=== modified file 'src/store/naive/simple_store.h'
--- src/store/naive/simple_store.h	2012-03-28 23:58:23 +0000
+++ src/store/naive/simple_store.h	2012-04-13 19:45:38 +0000
@@ -16,7 +16,11 @@
 #ifndef ZORBA_SIMPLE_STORE
 #define ZORBA_SIMPLE_STORE
 
-#include "store.h"
+#include "store/naive/store.h"
+
+#include "store/naive/node_factory.h"
+#include "store/naive/pul_primitive_factory.h"
+#include "store/naive/tree_id_generator.h"
 
 namespace zorba {
 namespace simplestore {
@@ -72,7 +76,7 @@
 
   NodeFactory* createNodeFactory() const;
 
-  void destroyNodeFactory(NodeFactory*) const;
+  void destroyNodeFactory(zorba::simplestore::NodeFactory*) const;
 
   store::ItemFactory* createItemFactory() const;
 
@@ -84,7 +88,7 @@
 
   PULPrimitiveFactory* createPULFactory() const;
 
-  void destroyPULFactory(PULPrimitiveFactory*) const;
+  void destroyPULFactory(zorba::simplestore::PULPrimitiveFactory*) const;
 
   CollectionSet* createCollectionSet() const;
 

=== modified file 'src/store/naive/store.h'
--- src/store/naive/store.h	2012-03-28 22:09:36 +0000
+++ src/store/naive/store.h	2012-04-13 19:45:38 +0000
@@ -16,10 +16,18 @@
 #ifndef ZORBA_SIMPLESTORE_STORE_H
 #define ZORBA_SIMPLESTORE_STORE_H
 
+#include "store/api/store.h"
+
 #include "shared_types.h"
 #include "store_defs.h"
 #include "hashmap_nodep.h"
 #include "tree_id.h"
+#include "store/util/hashmap_stringbuf.h"
+#include "zorbautils/mutex.h"
+#include "zorbautils/lock.h"
+#include "zorbautils/hashmap.h"
+#include "zorbautils/hashmap_itemp.h"
+#include "zorbautils/hashmap_zstring_nonserializable.h"
 
 #if (defined (WIN32) || defined (WINCE))
 #include "node_items.h"
@@ -28,14 +36,7 @@
 #include "store/api/ic.h"
 #endif
 
-#include "store/api/store.h"
-
-#include "store/util/hashmap_stringbuf.h"
-
-#include "zorbautils/mutex.h"
-#include "zorbautils/lock.h"
-#include "zorbautils/hashmap_itemp.h"
-#include "zorbautils/hashmap_zstring_nonserializable.h"
+using namespace zorba;
 
 namespace zorba
 {
@@ -63,9 +64,9 @@
 class TreeIdGeneratorFactory;
 class TreeIdGenerator;
 
-typedef zorba::HashMapZString<XmlNode_t> DocumentSet;
-typedef ItemPointerHashMap<store::Index_t> IndexSet;
-typedef ItemPointerHashMap<store::IC_t> ICSet;
+typedef HashMapZString<XmlNode_t> DocumentSet;
+typedef zorba::ItemPointerHashMap<store::Index_t> IndexSet;
+typedef zorba::ItemPointerHashMap<store::IC_t> ICSet;
 
 
 

=== modified file 'src/system/globalenv.cpp'
--- src/system/globalenv.cpp	2012-04-12 09:21:02 +0000
+++ src/system/globalenv.cpp	2012-04-13 19:45:38 +0000
@@ -17,11 +17,11 @@
 
 #include "common/common.h"
 
-#ifndef ZORBA_NO_UNICODE
+#ifndef ZORBA_NO_ICU
 # include <unicode/uclean.h>
 # include <unicode/utypes.h>
 # include <unicode/udata.h>
-#endif /* ZORBA_NO_UNICODE */
+#endif /* ZORBA_NO_ICU */
 
 #ifdef ZORBA_WITH_BIG_INTEGER
 # include "zorbatypes/m_apm.h"
@@ -206,7 +206,7 @@
   // from one thread only
   // see http://www.icu-project.org/userguide/design.html#Init_and_Termination
   // and http://www.icu-project.org/apiref/icu4c/uclean_8h.html
-#ifndef ZORBA_NO_UNICODE
+#ifndef ZORBA_NO_ICU
 #  if defined U_STATIC_IMPLEMENTATION && (defined WIN32 || defined WINCE)
   {
     TCHAR    self_path[1024];
@@ -236,13 +236,13 @@
     udata_setCommonData(icu_appdata, &data_err);
     ZORBA_ASSERT(data_err == U_ZERO_ERROR);
   
-      //  u_setDataDirectory(self_path);
+    // u_setDataDirectory(self_path);
   }
 #  endif
   UErrorCode lICUInitStatus = U_ZERO_ERROR;
   u_init(&lICUInitStatus);
   ZORBA_ASSERT(lICUInitStatus == U_ZERO_ERROR);
-#endif//ifndef ZORBA_NO_UNICODE
+#endif /* ZORBA_NO_ICU */
 }
 
 
@@ -254,12 +254,12 @@
   // releases statically initialized memory and prevents
   // valgrind from reporting those problems at the end
   // see http://www.icu-project.org/apiref/icu4c/uclean_8h.html#93f27d0ddc7c196a1da864763f2d8920
-#ifndef ZORBA_NO_UNICODE
+#ifndef ZORBA_NO_ICU
   u_cleanup();
 # if defined U_STATIC_IMPLEMENTATION && (defined WIN32 || defined WINCE)
   delete[] icu_appdata;
 # endif
-#endif//ifndef ZORBA_NO_UNICODE
+#endif /* ZORBA_NO_ICU */
 }
 
 

=== modified file 'src/unit_tests/CMakeLists.txt'
--- src/unit_tests/CMakeLists.txt	2012-03-28 05:19:57 +0000
+++ src/unit_tests/CMakeLists.txt	2012-04-13 19:45:38 +0000
@@ -29,9 +29,9 @@
     tokenizer.cpp)
 ENDIF (NOT ZORBA_NO_FULL_TEXT)
 
-IF (NOT ZORBA_NO_UNICODE)
+IF (NOT ZORBA_NO_ICU)
   LIST (APPEND UNIT_TEST_SRCS
     test_icu_streambuf.cpp)
-ENDIF (NOT ZORBA_NO_UNICODE)
+ENDIF (NOT ZORBA_NO_ICU)
 
 # vim:set et sw=2 tw=2:

=== modified file 'src/unit_tests/string.cpp'
--- src/unit_tests/string.cpp	2012-03-28 05:19:57 +0000
+++ src/unit_tests/string.cpp	2012-04-13 19:45:38 +0000
@@ -569,6 +569,7 @@
   ASSERT_TRUE( t == s );
 }
 
+#ifndef ZORBA_NO_ICU
 template<class StringType>
 static void test_to_string_from_wchar_t() {
   wchar_t const w[] = L"hello";
@@ -578,6 +579,7 @@
   for ( string::size_type i = 0; i < s.length(); ++i )
     ASSERT_TRUE( s[i] == w[i] );
 }
+#endif /* ZORBA_NO_ICU */
 
 template<class StringType>
 static void test_to_upper() {
@@ -605,6 +607,7 @@
   }
 }
 
+#ifndef ZORBA_NO_ICU
 static void test_to_wchar_t() {
   string const s = "hello";
   wchar_t *w;
@@ -616,6 +619,7 @@
     ASSERT_TRUE( w[i] == s[i] );
   delete[] w;
 }
+#endif /* ZORBA_NO_ICU */
 
 static void test_trim_start() {
   char const *s;
@@ -873,16 +877,20 @@
   test_to_string_from_utf8<zstring>();
   test_to_string_from_utf8<zstring_p>();
 
+#ifndef ZORBA_NO_ICU
   test_to_string_from_wchar_t<string>();
   test_to_string_from_wchar_t<zstring>();
   test_to_string_from_wchar_t<zstring_p>();
+#endif /* ZORBA_NO_ICU */
 
   test_to_upper<string>();
   test_to_upper<zstring>();
   test_to_upper<zstring_p>();
   test_to_upper<String>();
 
+#ifndef ZORBA_NO_ICU
   test_to_wchar_t();
+#endif /* ZORBA_NO_ICU */
 
   test_trim_start();
   test_trim_end();

=== modified file 'src/unit_tests/unit_test_list.h'
--- src/unit_tests/unit_test_list.h	2012-03-28 05:19:57 +0000
+++ src/unit_tests/unit_test_list.h	2012-04-13 19:45:38 +0000
@@ -36,9 +36,9 @@
     /**
      * ADD NEW UNIT TESTS HERE
      */
-#ifndef ZORBA_NO_UNICODE
+#ifndef ZORBA_NO_ICU
     int test_icu_streambuf( int, char*[] );
-#endif /* ZORBA_NO_UNICODE */
+#endif /* ZORBA_NO_ICU */
     int json_parser( int, char*[] );
 
     void initializeTestList();

=== modified file 'src/unit_tests/unit_tests.cpp'
--- src/unit_tests/unit_tests.cpp	2012-03-28 05:19:57 +0000
+++ src/unit_tests/unit_tests.cpp	2012-04-13 19:45:38 +0000
@@ -39,9 +39,9 @@
     void initializeTestList() {
       libunittests["string"] = test_string;
       libunittests["uri"] = runUriTest;
-#ifndef ZORBA_NO_UNICODE
+#ifndef ZORBA_NO_ICU
       libunittests["icu_streambuf"] = test_icu_streambuf;
-#endif /* ZORBA_NO_UNICODE */
+#endif /* ZORBA_NO_ICU */
       libunittests["json_parser"] = json_parser;
       libunittests["unique_ptr"] = test_unique_ptr;
 #ifndef ZORBA_NO_FULL_TEXT

=== modified file 'src/util/CMakeLists.txt'
--- src/util/CMakeLists.txt	2012-03-28 05:19:57 +0000
+++ src/util/CMakeLists.txt	2012-04-13 19:45:38 +0000
@@ -40,14 +40,14 @@
   LIST(APPEND UTIL_SRCS mmap_file.cpp)
 ENDIF(ZORBA_WITH_FILE_ACCESS)
 
-IF(ZORBA_NO_UNICODE)
+IF(ZORBA_NO_ICU)
   LIST(APPEND UTIL_SRCS
-    regex_ascii.cpp
+    regex_xquery.cpp
     passthru_streambuf.cpp)
-ELSE(ZORBA_NO_UNICODE)
+ELSE(ZORBA_NO_ICU)
   LIST(APPEND UTIL_SRCS
     icu_streambuf.cpp)
-ENDIF(ZORBA_NO_UNICODE)
+ENDIF(ZORBA_NO_ICU)
 
 HEADER_GROUP_SUBFOLDER(UTIL_SRCS fx)
 HEADER_GROUP_SUBFOLDER(UTIL_SRCS win32)

=== modified file 'src/util/icu_streambuf.h'
--- src/util/icu_streambuf.h	2012-02-04 01:26:18 +0000
+++ src/util/icu_streambuf.h	2012-04-13 19:45:38 +0000
@@ -17,6 +17,7 @@
 #ifndef ZORBA_ICU_STREAMBUF_H
 #define ZORBA_ICU_STREAMBUF_H
 
+#include <unicode/ucnv.h>
 #include <zorba/transcode_stream.h>
 
 #include "util/utf8_util.h"

=== modified file 'src/util/passthru_streambuf.cpp'
--- src/util/passthru_streambuf.cpp	2012-02-04 01:26:18 +0000
+++ src/util/passthru_streambuf.cpp	2012-04-13 19:45:38 +0000
@@ -14,8 +14,8 @@
  * limitations under the License.
  */
 
+#include "stdafx.h"
 #include "passthru_streambuf.h"
-
 using namespace std;
 
 namespace zorba {
@@ -47,7 +47,7 @@
 }
 
 bool passthru_streambuf::is_supported( char const *cc_charset ) {
-  return !is_necessary( charset );
+  return !is_necessary( cc_charset );
 }
 
 passthru_streambuf::pos_type

=== modified file 'src/util/passthru_streambuf.h'
--- src/util/passthru_streambuf.h	2012-02-02 18:37:24 +0000
+++ src/util/passthru_streambuf.h	2012-04-13 19:45:38 +0000
@@ -17,8 +17,9 @@
 #ifndef ZORBA_PASSTHRU_STREAMBUF_H
 #define ZORBA_PASSTHRU_STREAMBUF_H
 
-#include <zorba/transcode_streambuf.h>
-
+#include <zorba/transcode_stream.h>
+#include "zorbatypes/zstring.h"
+#include "util/ascii_util.h"
 namespace zorba {
 
 ///////////////////////////////////////////////////////////////////////////////
@@ -48,6 +49,13 @@
    * @return \c true only if the character encoding is supported.
    */
   static bool is_supported( char const *charset );
+  static bool is_necessary( char const *cc_charset );
+
+  typedef std::streambuf::char_type char_type;
+  typedef std::streambuf::int_type int_type;
+  typedef std::streambuf::off_type off_type;
+  typedef std::streambuf::pos_type pos_type;
+  typedef std::streambuf::traits_type traits_type;
 
 protected:
   void imbue( std::locale const& );

=== modified file 'src/util/regex.cpp'
--- src/util/regex.cpp	2012-03-28 05:19:57 +0000
+++ src/util/regex.cpp	2012-04-13 19:45:38 +0000
@@ -15,8 +15,6 @@
  */
 #include "stdafx.h"
 
-#include "regex.h"
-
 #include <cstring>
 #include <vector>
 
@@ -28,13 +26,13 @@
 
 #include "ascii_util.h"
 #include "cxx_util.h"
+#include "regex.h"
 #include "stl_util.h"
 
 #define INVALID_RE_EXCEPTION(...) \
   XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS( __VA_ARGS__ ) )
 
-
-#ifndef ZORBA_NO_UNICODE
+#ifndef ZORBA_NO_ICU
 # include <unicode/uversion.h>
 U_NAMESPACE_USE
 
@@ -103,6 +101,7 @@
 
   bool got_backslash = false;
   bool in_char_class = false;           // within [...]
+  bool is_first_char = true;            // to check ^ placement
 
   bool in_backref = false;              // '\'[1-9][0-9]*
   unsigned backref_no = 0;              // 1-based
@@ -231,6 +230,8 @@
             ++open_cap_subs;
             cap_sub.push_back( true );
             cur_cap_sub = cap_sub.size();
+            is_first_char = true;
+            goto append;
           }
           break;
         case ')':
@@ -245,8 +246,10 @@
         case '[':
           if ( q_flag )
             *icu_re += '\\';
-          else
+          else {
             in_char_class = true;
+            goto append;
+          }
           break;
         case ']':
           if ( q_flag )
@@ -254,6 +257,19 @@
           else
             in_char_class = false;
           break;
+        case '^':
+          if ( q_flag )
+            *icu_re += '\\';
+          else if ( !is_first_char && !in_char_class )
+            throw INVALID_RE_EXCEPTION( xq_re, ZED( UnescapedChar_3 ), *xq_c );
+          break;
+        case '|':
+          if ( q_flag )
+            *icu_re += '\\';
+          else {
+            is_first_char = true;
+            goto append;
+          }
         default:
           if ( x_flag && ascii::is_space( *xq_c ) ) {
             if ( !in_char_class )
@@ -265,37 +281,42 @@
             //
             *icu_re += '\\';
           }
-      }
-    }
+      } // switch
+    } // else
+    is_first_char = false;
+append:
     *icu_re += *xq_c;
   } // FOR_EACH
 
-  if ( i_flag ) {
-    //
-    // XQuery 3.0 F&O 5.6.1.1: All other constructs are unaffected by the "i"
-    // flag.  For example, "\p{Lu}" continues to match upper-case letters only.
-    //
-    // However, ICU lower-cases everything for the 'i' flag; hence we have to
-    // turn off the 'i' flag for just the \p{Lu}.
-    //
-    // Note that the "6" and "12" below are correct since "\\" represents a
-    // single '\'.
-    //
-    ascii::replace_all( *icu_re, "\\p{Lu}", 6, "(?-i:\\p{Lu})", 12 );
-  }
+  if ( !q_flag ) {
+    if ( i_flag ) {
+      //
+      // XQuery 3.0 F&O 5.6.1.1: All other constructs are unaffected by the "i"
+      // flag.  For example, "\p{Lu}" continues to match upper-case letters
+      // only.
+      //
+      // However, ICU lower-cases everything for the 'i' flag; hence we have to
+      // turn off the 'i' flag for just the \p{Lu}.
+      //
+      // Note that the "6" and "12" below are correct since "\\" represents a
+      // single '\'.
+      //
+      ascii::replace_all( *icu_re, "\\p{Lu}", 6, "(?-i:\\p{Lu})", 12 );
+    }
 
-  //
-  // XML Schema Part 2 F.1.1: [Unicode Database] groups code points into a
-  // number of blocks such as Basic Latin (i.e., ASCII), Latin-1 Supplement,
-  // Hangul Jamo, CJK Compatibility, etc. The set containing all characters
-  // that have block name X (with all white space stripped out), can be
-  // identified with a block escape \p{IsX}.
-  //
-  // However, ICU uses \p{InX} rather than \p{IsX}.
-  //
-  // Note that the "5" below is correct since "\\" represents a single '\'.
-  //
-  ascii::replace_all( *icu_re, "\\p{Is", 5, "\\p{In", 5 );
+    //
+    // XML Schema Part 2 F.1.1: [Unicode Database] groups code points into a
+    // number of blocks such as Basic Latin (i.e., ASCII), Latin-1 Supplement,
+    // Hangul Jamo, CJK Compatibility, etc. The set containing all characters
+    // that have block name X (with all white space stripped out), can be
+    // identified with a block escape \p{IsX}.
+    //
+    // However, ICU uses \p{InX} rather than \p{IsX}.
+    //
+    // Note that the "5" below is correct since "\\" represents a single '\'.
+    //
+    ascii::replace_all( *icu_re, "\\p{Is", 5, "\\p{In", 5 );
+  } // q_flag
 }
 
 ///////////////////////////////////////////////////////////////////////////////
@@ -442,11 +463,11 @@
 }
 
 } // namespace unicode
-
-}//namespace zorba
-
-
-#else /* ZORBA_NO_UNICODE */
+} // namespace zorba
+
+///////////////////////////////////////////////////////////////////////////////
+
+#else /* ZORBA_NO_ICU */
 
 #include "zorbatypes/zstring.h"
 
@@ -470,7 +491,7 @@
     case 'i': flags |= REGEX_ASCII_CASE_INSENSITIVE; break;
     case 's': flags |= REGEX_ASCII_DOTALL; break;
     case 'm': flags |= REGEX_ASCII_MULTILINE; break;
-    case 'x': flags |= REGEX_ASCII_COMMENTS; break;
+    case 'x': flags |= REGEX_ASCII_NO_WHITESPACE; break;
     case 'q': flags |= REGEX_ASCII_LITERAL; break;
     default:
       throw XQUERY_EXCEPTION( err::FORX0001, ERROR_PARAMS( *p ) );
@@ -483,6 +504,7 @@
 void regex::compile( char const *pattern, char const *flags)
 {
   parsed_flags = parse_regex_flags(flags);
+  regex_xquery::CRegexXQuery_parser regex_parser;
   regex_matcher = regex_parser.parse(pattern, parsed_flags);
   if(!regex_matcher)
     throw INVALID_RE_EXCEPTION(pattern);
@@ -517,6 +539,8 @@
 bool regex::next_token( char const *s, size_type *pos, zstring *token,
                   bool *matched)
 {
+  if(!s[*pos])
+    return false;
   bool  retval;
   int   match_pos;
   int   matched_len;
@@ -528,14 +552,8 @@
       token->assign(s+*pos, match_pos);
     *pos += match_pos + matched_len;
     if(matched)
-      if(match_pos)
-        *matched = true;
-      else
-        *matched = false;
-    if(match_pos)
-      return true;
-    else
-      return false;
+      *matched = true;
+    return true;
   }
   else
   {
@@ -544,7 +562,7 @@
     *pos += strlen(s+*pos);
     if(matched)
       *matched = false;
-    return s[*pos] != 0;
+    return true;
   }
 }
 
@@ -554,13 +572,9 @@
   int   matched_pos;
   int   matched_len;
 
-  bool prev_align = regex_matcher->set_align_begin(true);
-  retval = regex_matcher->match_from(s, parsed_flags, &matched_pos, &matched_len);
-  regex_matcher->set_align_begin(prev_align);
+  retval = regex_matcher->match_anywhere(s, parsed_flags|REGEX_ASCII_WHOLE_MATCH, &matched_pos, &matched_len);
   if(!retval)
     return false;
-  if(matched_len != strlen(s))
-    return false;
   return true;
 }
 
@@ -587,14 +601,19 @@
       //look for dollars
       if(*temprepl == '\\')
       {
-        temprepl++;
-        if(!*temprepl || (*temprepl != '\\') || (*temprepl != '$'))//Invalid replacement string.
-          throw XQUERY_EXCEPTION( err::FORX0004, ERROR_PARAMS( replacement ) );
+        if(!(parsed_flags & REGEX_ASCII_LITERAL))
+        {
+          temprepl++;
+          if(!*temprepl) 
+            temprepl--;
+          else if((*temprepl != '\\') && (*temprepl != '$'))//Invalid replacement string.
+            throw XQUERY_EXCEPTION( err::FORX0004, ERROR_PARAMS( replacement ) );
+        }
         result->append(1, *temprepl);
         temprepl++;
         continue;
       }
-      if(*temprepl == '$')
+      if((*temprepl == '$') && !(parsed_flags & REGEX_ASCII_LITERAL))
       {
         temprepl++;
         index = 0;
@@ -648,7 +667,7 @@
   if(retval)
   {
     m_match_pos += m_pos;
-    m_pos = m_match_pos = m_matched_len;
+    m_pos = m_match_pos + m_matched_len;
   }
   else
   {
@@ -666,35 +685,30 @@
   return (int)regex_matcher->get_indexed_regex_count();
 }
 
-int regex::get_match_start( int groupId )
-{
-  if(groupId == 0)
-    return m_match_pos;
-  if(groupId > (int)regex_matcher->get_indexed_regex_count())
-    return -1;
-  const char *submatched_source;
-  int   submatched_len;
-  if(!regex_matcher->get_indexed_match(groupId, &submatched_source, &submatched_len))
-    return -1;
-  return submatched_source - s_in_.c_str();
-}
-
-int regex::get_match_end( int groupId )
-{
-  if(groupId == 0)
-    return m_match_pos + m_matched_len;
-  if(groupId > (int)regex_matcher->get_indexed_regex_count())
-    return -1;
-  const char *submatched_source;
-  int   submatched_len;
-  if(!regex_matcher->get_indexed_match(groupId, &submatched_source, &submatched_len))
-    return -1;
-  return submatched_source - s_in_.c_str() + submatched_len;
+bool regex::get_match_start_end_bytes( int groupId, int *start, int *end )
+{
+  *start = -1;
+  *end = -1;
+  if(groupId == 0)
+  {
+    *start = m_match_pos;
+    *end = m_match_pos + m_matched_len;
+    return true;
+  }
+  if(groupId > (int)regex_matcher->get_indexed_regex_count())
+    return false;
+  const char *submatched_source;
+  int   submatched_len;
+  if(!regex_matcher->get_indexed_match(groupId, &submatched_source, &submatched_len))
+    return false;
+  *start = submatched_source - s_in_.c_str();
+  *end = *start + submatched_len;
+  return true;
 }
 
 } // namespace unicode
 } // namespace zorba
-#endif /* ZORBA_NO_UNICODE */
+#endif /* ZORBA_NO_ICU */
 
 ///////////////////////////////////////////////////////////////////////////////
 

=== modified file 'src/util/regex.h'
--- src/util/regex.h	2012-03-28 05:19:57 +0000
+++ src/util/regex.h	2012-04-13 19:45:38 +0000
@@ -17,15 +17,13 @@
 #ifndef ZORBA_REGEX_H
 #define ZORBA_REGEX_H
 
-#ifndef ZORBA_NO_UNICODE
-#include <unicode/regex.h>
-#endif
-
 #include "cxx_util.h"
 #include "unicode_util.h"
 #include "zorbatypes/zstring.h"
 
-#ifndef ZORBA_NO_UNICODE
+#ifndef ZORBA_NO_ICU
+
+#include <unicode/regex.h>
 
 namespace zorba {
 
@@ -496,15 +494,17 @@
 } // namespace unicode
 } // namespace zorba
 
-#else ///ZORBA_NO_UNICODE (ascii part:)
-
-#include "util/regex_ascii.h"
+///////////////////////////////////////////////////////////////////////////////
+
+#else /* ZORBA_NO_ICU */
+
+#include "util/regex_xquery.h"
 #include <string>
 
 namespace zorba{
 /**
  * Converts an XQuery regular expression to the form used by the regular
- * expression library Zorba is using (here regex_ascii).
+ * expression library Zorba is using (here regex_xquery).
  *
  * @param xq_re The XQuery regular expression.
  * @param lib_re A pointer to the resuling library regular expression.
@@ -525,7 +525,7 @@
   /**
    * Constructs a %regex.
    */
-  regex() : regex_matcher( NULL ) { }
+  regex() : regex_matcher( nullptr ) { }
 
   /**
    * Destroys a %regex.
@@ -835,31 +835,21 @@
 
   /**
    * Get the start position of the matched group.
-   * If groupId is zero, then the start position of the whole match is returned.
-   * If groupId is non-zero, then the start position of that group is returned.
-   * If that group has not been matched, -1 is returned.
+   * If groupId is zero, then the start and end position of the whole match is returned.
+   * If groupId is non-zero, then the start and end position of that group is returned.
+   * If that group has not been matched, false is returned.
    *
    * @param groupId the id of the group, either zero for the entire regex,
    *  or [1 .. group_count] for that specific group
-   * @return the start position, zero based, or -1 if that group didn't match
+   * @param start to return start position in bytes
+   * @param end to return end position in bytes
+   * @return true if that group exists and has been matched
    */
-  int get_match_start( int groupId = 0 );
+  bool get_match_start_end_bytes( int groupId, int *start, int *end );
 
-  /**
-   * Get the end position of the matched group.
-   * If groupId is zero, then the end position of the whole match is returned.
-   * If groupId is non-zero, then the end position of that group is returned.
-   * If that group has not been matched, -1 is returned.
-   *
-   * @param groupId the id of the group, either zero for the entire regex,
-   *  or [1 .. group_count] for that specific group
-   * @return the end position, zero based, or -1 if that group didn't match
-   */
-  int get_match_end( int groupId = 0 );
 
 private:
-  regex_ascii::CRegexAscii_parser regex_parser;
-  regex_ascii::CRegexAscii_regex  *regex_matcher;
+  regex_xquery::CRegexXQuery_regex  *regex_matcher;
   uint32_t    parsed_flags;
 
   zstring s_in_;
@@ -873,15 +863,13 @@
   regex( regex const& );
   regex& operator=( regex const& );
 };
+
+///////////////////////////////////////////////////////////////////////////////
+
 } // namespace unicode
 } // namespace zorba
 
-#endif /* ZORBA_NO_UNICODE */
-
-
-///////////////////////////////////////////////////////////////////////////////
-
-
+#endif /* ZORBA_NO_ICU */
 #endif /* ZORBA_REGEX_H */
 /*
  * Local variables:

=== renamed file 'src/util/regex_ascii.cpp' => 'src/util/regex_xquery.cpp'
--- src/util/regex_ascii.cpp	2012-03-28 05:19:57 +0000
+++ src/util/regex_xquery.cpp	2012-04-13 19:45:38 +0000
@@ -1,4 +1,4 @@
-a/*
+/*
  * Copyright 2006-2008 The FLWOR Foundation.
  * 
  * Licensed under the Apache License, Version 2.0 (the "License");
@@ -18,12 +18,15 @@
 
 #include "diagnostics/xquery_diagnostics.h"
 
-#include "regex_ascii.h"
+#include "regex_xquery.h"
 #include <string.h>
 #include "zorbatypes/chartype.h"
+#include "util/unicode_categories.h"
+#include "util/ascii_util.h"
+#include "util/utf8_string.h"
 
 namespace zorba {
-  namespace regex_ascii{
+  namespace regex_xquery{
 //ascii regular expression matching
 
 /*http://www.w3.org/TR/xmlschema-2/#regexs
@@ -62,96 +65,138 @@
 + http://www.w3.org/TR/xquery-operators/#regex-syntax (not implemented)
 */
 
+
+static bool compare_ascii_i(const char *str1, const char *str2)
+{
+  while(*str1 && *str2)
+  {
+    if(ascii::to_lower(*str1) != ascii::to_lower(*str2))
+      return false;
+    str1++;
+    str2++;
+  }
+  if(*str1 || *str2)
+    return false;
+  return true;
+}
+
+static bool compare_unicode_ni(const char *str1, const char *str2, int len)
+{
+  while(len > 0)
+  {
+    const char *temp_str1 = str1;
+    const char *temp_str2 = str2;
+    unicode::code_point cp1 = unicode::to_upper(utf8::next_char(temp_str1));
+    unicode::code_point cp2 = unicode::to_upper(utf8::next_char(temp_str2));
+    if(cp1 != cp2)
+      return false;
+    len -= temp_str1-str1;
+    str1 = temp_str1;
+    str2 = temp_str2;
+  }
+  return true;
+}
+static utf8::size_type myutf8len(const char *source)
+{
+  utf8::size_type  len = utf8::char_length(*source);
+  if(!len)
+    return 1;
+  else
+    return len;
+}
 ////////////////////////////////////
 ////Regular expression parsing and building of the tree
 ////////////////////////////////////
 
-CRegexAscii_regex* CRegexAscii_parser::parse(const char *pattern, unsigned int flags)
+CRegexXQuery_regex* CRegexXQuery_parser::parse(const char *pattern, unsigned int flags)
 {
   this->flags = flags;
-  bool align_begin = false;
   
-  if(!(flags & REGEX_ASCII_LITERAL) && (pattern[0] == '^'))
-    align_begin = true;
-
   int   regex_len;
-  CRegexAscii_regex*  regex = parse_regexp(pattern + (align_begin?1:0), &regex_len);
+  CRegexXQuery_regex*  regex = parse_regexp(pattern, &regex_len);
   
-  if(regex)
-    regex->set_align_begin(align_begin);
-
   return regex;
 }
 
 //until '\0' or ')'
-CRegexAscii_regex* CRegexAscii_parser::parse_regexp(const char *pattern, 
+CRegexXQuery_regex* CRegexXQuery_parser::parse_regexp(const char *pattern, 
                                                     int *regex_len)
 {
   *regex_len = 0;
   int   branch_len;
   regex_depth++;
-  CRegexAscii_regex *regex = new CRegexAscii_regex(current_regex);
+  std::auto_ptr<CRegexXQuery_regex>  regex(new CRegexXQuery_regex(current_regex));
   if(!current_regex)
-    current_regex = regex;
+    current_regex = regex.get();
   if(regex_depth >= 2)
   {
     //mark this as group if it does not start with ?:
     if(pattern[0] != '?' || pattern[1] != ':')
-      current_regex->subregex.push_back(regex);
+      current_regex->subregex.push_back(regex.get());
     else
       *regex_len = 2;
   }
-  CRegexAscii_branch  *branch;
+  CRegexXQuery_branch  *branch;
+  bool must_read_another_branch = true;
   while(pattern[*regex_len] && (pattern[*regex_len] != ')'))
   {
     branch = parse_branch(pattern+*regex_len, &branch_len);
     if(!branch)
     {
       regex_depth--;
-      delete regex;
       return NULL;
     }
     regex->add_branch(branch);
     *regex_len += branch_len;
+    if(pattern[*regex_len] == '|')
+      (*regex_len)++;
+    else
+      must_read_another_branch = false;
   }
-  if((current_regex == regex) && (pattern[*regex_len] == ')'))
+  if((current_regex == regex.get()) && (pattern[*regex_len] == ')'))
   {
-    throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(U_REGEX_MISMATCHED_PAREN)) );
+    throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_MISMATCHED_PAREN)) );
   }
   if(pattern[*regex_len])
     (*regex_len)++;
+  if(must_read_another_branch)
+    regex->add_branch(new CRegexXQuery_branch(current_regex));//add empty branch
   regex->flags = 0;//finished initialization
   regex_depth--;
-  return regex;
+  return regex.release();
 }
 
-CRegexAscii_branch* CRegexAscii_parser::parse_branch(const char *pattern, int *branch_len)
+CRegexXQuery_branch* CRegexXQuery_parser::parse_branch(const char *pattern, int *branch_len)
 {
   int piece_len;
 
-  CRegexAscii_branch    *branch = new CRegexAscii_branch(current_regex);
-  CRegexAscii_piece     *piece;
+  std::auto_ptr<CRegexXQuery_branch>    branch(new CRegexXQuery_branch(current_regex));
+  CRegexXQuery_piece     *piece;
   *branch_len = 0;
   while(pattern[*branch_len] && (pattern[*branch_len] != '|') && (pattern[*branch_len] != ')'))
   {
     piece = parse_piece(pattern+*branch_len, &piece_len);
     if(!piece)
     {
-      delete branch;
       return NULL;
     }
+    if(branch->piece_list.size() && dynamic_cast<CRegexXQuery_pinstart*>(piece->atom))
+    {
+      //found ^ that is not at the beginning of branch
+      throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_ATOM_CHAR), '^') );
+    }
     branch->add_piece(piece);
     *branch_len += piece_len;
   }
-  if(pattern[*branch_len] == '|')
-    (*branch_len)++;
-  return branch;
+  //if(pattern[*branch_len] == '|')
+  //  (*branch_len)++;
+  return branch.release();
 }
 
 //piece = atom + quantifier
-CRegexAscii_piece* CRegexAscii_parser::parse_piece(const char *pattern, int *piece_len)
+CRegexXQuery_piece* CRegexXQuery_parser::parse_piece(const char *pattern, int *piece_len)
 {
-  CRegexAscii_piece *piece = new CRegexAscii_piece;
+  std::auto_ptr<CRegexXQuery_piece>  piece(new CRegexXQuery_piece);
   IRegexAtom  *atom;
   *piece_len = 0;
 
@@ -160,19 +205,18 @@
   atom = read_atom(pattern, &atom_len);
   if(!atom)
   {
-    delete piece;
     return NULL;
   }
   piece->set_atom(atom);
   if(!(flags & REGEX_ASCII_LITERAL))
-    read_quantifier(piece, pattern+atom_len, &quantif_len);
+    read_quantifier(piece.get(), pattern+atom_len, &quantif_len);
 
   *piece_len += atom_len + quantif_len;
 
-  return piece;
+  return piece.release();
 }
 
-char CRegexAscii_parser::myishex(char c)
+char CRegexXQuery_parser::myishex(char c)
 {
   if((c >= '0') && (c <= '9'))
     return c-'0'+1;
@@ -183,26 +227,125 @@
   return 0;//not a hex
 }
 
-bool CRegexAscii_parser::myisdigit(char c)
-{
-  return (c >= '0') || (c <= '9');
-}
-
-char CRegexAscii_parser::readChar(const char *pattern, int *char_len, bool *is_multichar)
+bool CRegexXQuery_parser::myisdigit(char c)
+{
+  return (c >= '0') && (c <= '9');
+}
+
+bool CRegexXQuery_parser::myisletterAZ(char c)
+{
+  return ((c >= 'a') && (c <= 'z')) || ((c >= 'A') && (c <= 'Z'));
+}
+
+static const unicode::code_point specials_extcp[] = {0xFFF0, 0xFFFD, 0};
+
+static CRegexXQuery_parser::block_escape_t block_escape[] = 
+{
+{{0x0000, 0x007F}, NULL, "BasicLatin"},
+{{0x0080, 0x00FF}, NULL, "Latin-1Supplement"},
+{{0x0100, 0x017F}, NULL, "LatinExtended-A"},
+{{0x0180, 0x024F}, NULL, "LatinExtended-B"},
+{{0x0250, 0x02AF}, NULL, "IPAExtensions"},
+{{0x02B0, 0x02FF}, NULL, "SpacingModifierLetters"},
+{{0x0300, 0x036F}, NULL, "CombiningDiacriticalMarks"},
+{{0x0370, 0x03FF}, NULL, "Greek"},
+{{0x0400, 0x04FF}, NULL, "Cyrillic"},
+{{0x0530, 0x058F}, NULL, "Armenian"},
+{{0x0590, 0x05FF}, NULL, "Hebrew"},
+{{0x0600, 0x06FF}, NULL, "Arabic"},
+{{0x0700, 0x074F}, NULL, "Syriac"},
+{{0x0780, 0x07BF}, NULL, "Thaana"},
+{{0x0900, 0x097F}, NULL, "Devanagari"},
+{{0x0980, 0x09FF}, NULL, "Bengali"},
+{{0x0A00, 0x0A7F}, NULL, "Gurmukhi"},
+{{0x0A80, 0x0AFF}, NULL, "Gujarati"},
+{{0x0B00, 0x0B7F}, NULL, "Oriya"},
+{{0x0B80, 0x0BFF}, NULL, "Tamil"},
+{{0x0C00, 0x0C7F}, NULL, "Telugu"},
+{{0x0C80, 0x0CFF}, NULL, "Kannada"},
+{{0x0D00, 0x0D7F}, NULL, "Malayalam"},
+{{0x0D80, 0x0DFF}, NULL, "Sinhala"},
+{{0x0E00, 0x0E7F}, NULL, "Thai"},
+{{0x0E80, 0x0EFF}, NULL, "Lao"},
+{{0x0F00, 0x0FFF}, NULL, "Tibetan"},
+{{0x1000, 0x109F}, NULL, "Myanmar"},
+{{0x10A0, 0x10FF}, NULL, "Georgian"},
+{{0x1100, 0x11FF}, NULL, "HangulJamo"},
+{{0x1200, 0x137F}, NULL, "Ethiopic"},
+{{0x13A0, 0x13FF}, NULL, "Cherokee"},
+{{0x1400, 0x167F}, NULL, "UnifiedCanadianAboriginalSyllabics"},
+{{0x1680, 0x169F}, NULL, "Ogham"},
+{{0x16A0, 0x16FF}, NULL, "Runic"},
+{{0x1780, 0x17FF}, NULL, "Khmer"},
+{{0x1800, 0x18AF}, NULL, "Mongolian"},
+{{0x1E00, 0x1EFF}, NULL, "LatinExtendedAdditional"},
+{{0x1F00, 0x1FFF}, NULL, "GreekExtended"},
+{{0x2000, 0x206F}, NULL, "GeneralPunctuation"},
+{{0x2070, 0x209F}, NULL, "SuperscriptsandSubscripts"},
+{{0x20A0, 0x20CF}, NULL, "CurrencySymbols"},
+{{0x20D0, 0x20FF}, NULL, "CombiningMarksforSymbols"},
+{{0x2100, 0x214F}, NULL, "LetterlikeSymbols"},
+{{0x2150, 0x218F}, NULL, "NumberForms"},
+{{0x2190, 0x21FF}, NULL, "Arrows"},
+{{0x2200, 0x22FF}, NULL, "MathematicalOperators"},
+{{0x2300, 0x23FF}, NULL, "MiscellaneousTechnical"},
+{{0x2400, 0x243F}, NULL, "ControlPictures"},
+{{0x2440, 0x245F}, NULL, "OpticalCharacterRecognition"},
+{{0x2460, 0x24FF}, NULL, "EnclosedAlphanumerics"},
+{{0x2500, 0x257F}, NULL, "BoxDrawing"},
+{{0x2580, 0x259F}, NULL, "BlockElements"},
+{{0x25A0, 0x25FF}, NULL, "GeometricShapes"},
+{{0x2600, 0x26FF}, NULL, "MiscellaneousSymbols"},
+{{0x2700, 0x27BF}, NULL, "Dingbats"},
+{{0x2800, 0x28FF}, NULL, "BraillePatterns"},
+{{0x2E80, 0x2EFF}, NULL, "CJKRadicalsSupplement"},
+{{0x2F00, 0x2FDF}, NULL, "KangxiRadicals"},
+{{0x2FF0, 0x2FFF}, NULL, "IdeographicDescriptionCharacters"},
+{{0x3000, 0x303F}, NULL, "CJKSymbolsandPunctuation"},
+{{0x3040, 0x309F}, NULL, "Hiragana"},
+{{0x30A0, 0x30FF}, NULL, "Katakana"},
+{{0x3100, 0x312F}, NULL, "Bopomofo"},
+{{0x3130, 0x318F}, NULL, "HangulCompatibilityJamo"},
+{{0x3190, 0x319F}, NULL, "Kanbun"},
+{{0x31A0, 0x31BF}, NULL, "BopomofoExtended"},
+{{0x3200, 0x32FF}, NULL, "EnclosedCJKLettersandMonths"},
+{{0x3300, 0x33FF}, NULL, "CJKCompatibility"},
+{{0x3400, 0x4DB5}, NULL, "CJKUnifiedIdeographsExtensionA"},
+{{0x4E00, 0x9FFF}, NULL, "CJKUnifiedIdeographs"},
+{{0xA000, 0xA48F}, NULL, "YiSyllables"},
+{{0xA490, 0xA4CF}, NULL, "YiRadicals"},
+{{0xAC00, 0xD7A3}, NULL, "HangulSyllables"},
+{{0xE000, 0xF8FF}, NULL, "PrivateUse"},
+{{0xF900, 0xFAFF}, NULL, "CJKCompatibilityIdeographs"},
+{{0xFB00, 0xFB4F}, NULL, "AlphabeticPresentationForms"},
+{{0xFB50, 0xFDFF}, NULL, "ArabicPresentationForms-A"},
+{{0xFE20, 0xFE2F}, NULL, "CombiningHalfMarks"},
+{{0xFE30, 0xFE4F}, NULL, "CJKCompatibilityForms"},
+{{0xFE50, 0xFE6F}, NULL, "SmallFormVariants"},
+{{0xFE70, 0xFEFE}, NULL, "ArabicPresentationForms-B"},
+{{0xFEFF, 0xFEFF}, specials_extcp, "Specials"},
+{{0xFF00, 0xFFEF}, NULL, "HalfwidthandFullwidthForms"}
+};
+
+CRegexXQuery_charmatch* CRegexXQuery_parser::readChar(const char *pattern, 
+                                  int *char_len,
+                                  enum CHARGROUP_t *multichar_type)
 {
   char  c = 0;
   *char_len = 0;
-  *is_multichar = false;
+  *multichar_type = CHARGROUP_NO_MULTICHAR;
   switch(pattern[*char_len])
   {
   case '\\':
-  {  (*char_len)++;
+  {
+    (*char_len)++;
     switch(pattern[*char_len])
     {
-    case 'n': c = '\n';break;
-    case 'r': c = '\r';break;
-    case 't': c = '\t';break;
+    case 'n': c = '\n';(*char_len)++;return new CRegexXQuery_char_ascii(current_regex, c);
+    case 'r': c = '\r';(*char_len)++;return new CRegexXQuery_char_ascii(current_regex, c);
+    case 't': c = '\t';(*char_len)++;return new CRegexXQuery_char_ascii(current_regex, c);
     case '\\':
+    case '/'://+
     case '|':
     case '.':
     case '?':
@@ -216,19 +359,205 @@
     case '['://#x5B
     case ']'://#x5D
     case '^'://#x5E
+    case '$'://+
        c = pattern[*char_len];
-       break;
+      (*char_len)++;
+      *multichar_type = CHARGROUP_FLAGS_ONECHAR_ASCII;
+      return new CRegexXQuery_char_ascii(current_regex, c);
     case 'p'://catEsc
     case 'P'://complEsc
+    {
       //ignore the prop for now
-      c = pattern[*char_len];
-      *is_multichar = true;
-      if(pattern[*char_len+1] == '{')
-      {
-        while(pattern[*char_len] != '}')
+      *multichar_type = CHARGROUP_FLAGS_MULTICHAR_p;//(CHARGROUP_t)((pattern[*char_len] == 'P') ? 128 : 0);
+      bool is_reverse = (pattern[*char_len] == 'P');
+      c = 0;
+      if(pattern[(*char_len)+1] != '{')
+      {
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_BROKEN_P_CONSTRUCT)) );
+      }
+      (*char_len) += 2;
+      switch(pattern[*char_len])
+      {//IsCategory
+      case 'L':
+      {
+        switch(pattern[(*char_len)+1])
+        {
+        case '}':
+          c = unicode::UNICODE_Ll + 50;break;
+        case 'u':
+          c = unicode::UNICODE_Lu; (*char_len)++;break;
+        case 'l':
+          c = unicode::UNICODE_Ll; (*char_len)++;break;
+        case 't':
+          c = unicode::UNICODE_Lt; (*char_len)++;break;
+        case 'm':
+          c = unicode::UNICODE_Lm; (*char_len)++;break;
+        case 'o':
+          c = unicode::UNICODE_Lo; (*char_len)++;break;
+        default:
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PL_CONSTRUCT)) );
+        }
+      }break;
+      case 'M':
+      {
+        switch(pattern[(*char_len)+1])
+        {
+        case '}':
+          c = unicode::UNICODE_Mc + 50;break;
+        case 'n':
+          c = unicode::UNICODE_Mn; (*char_len)++;break;
+        case 'c':
+          c = unicode::UNICODE_Mc; (*char_len)++;break;
+        case 'e':
+          c = unicode::UNICODE_Me; (*char_len)++;break;
+        default:
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PM_CONSTRUCT)) );
+        }
+      }break;
+      case 'N':
+      {
+        switch(pattern[(*char_len)+1])
+        {
+        case '}':
+          c = unicode::UNICODE_Nd + 50;break;
+        case 'd':
+          c = unicode::UNICODE_Nd; (*char_len)++;break;
+        case 'l':
+          c = unicode::UNICODE_Nl; (*char_len)++;break;
+        case 'o':
+          c = unicode::UNICODE_No; (*char_len)++;break;
+        default:
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PN_CONSTRUCT)) );
+        }
+      }break;
+      case 'P':
+      {
+        switch(pattern[(*char_len)+1])
+        {
+        case '}':
+          c = unicode::UNICODE_Pc + 50;break;
+        case 'c':
+          c = unicode::UNICODE_Pc; (*char_len)++;break;
+        case 'd':
+          c = unicode::UNICODE_Pd; (*char_len)++;break;
+        case 's':
+          c = unicode::UNICODE_Ps; (*char_len)++;break;
+        case 'e':
+          c = unicode::UNICODE_Pe; (*char_len)++;break;
+        case 'i':
+          c = unicode::UNICODE_Pi; (*char_len)++;break;
+        case 'f':
+          c = unicode::UNICODE_Pf; (*char_len)++;break;
+        case 'o':
+          c = unicode::UNICODE_Po; (*char_len)++;break;
+        default:
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PP_CONSTRUCT)) );
+        }
+      }break;
+      case 'Z':
+      {
+        switch(pattern[(*char_len)+1])
+        {
+        case '}':
+          c = unicode::UNICODE_Zl + 50;break;
+        case 's':
+          c = unicode::UNICODE_Zs; (*char_len)++;break;
+        case 'l':
+          c = unicode::UNICODE_Zl; (*char_len)++;break;
+        case 'p':
+          c = unicode::UNICODE_Zp; (*char_len)++;break;
+        default:
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PZ_CONSTRUCT)) );
+        }
+      }break;
+      case 'S':
+      {
+        switch(pattern[(*char_len)+1])
+        {
+        case '}':
+          c = unicode::UNICODE_Sc + 50;break;
+        case 'm':
+          c = unicode::UNICODE_Sm; (*char_len)++;break;
+        case 'c':
+          c = unicode::UNICODE_Sc; (*char_len)++;break;
+        case 'k':
+          c = unicode::UNICODE_Sk; (*char_len)++;break;
+        case 'o':
+          c = unicode::UNICODE_So; (*char_len)++;break;
+        default:
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PS_CONSTRUCT)) );
+        }
+      }break;
+      case 'C':
+      {
+        switch(pattern[(*char_len)+1])
+        {
+        case '}':
+          c = unicode::UNICODE_Cc + 50;break;
+        case 'c':
+          c = unicode::UNICODE_Cc; (*char_len)++;break;
+        case 'f':
+          c = unicode::UNICODE_Cf; (*char_len)++;break;
+        case 'o':
+          c = unicode::UNICODE_Co; (*char_len)++;break;
+        case 'n':
+          c = unicode::UNICODE_Cn; (*char_len)++;break;
+        default:
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PC_CONSTRUCT)) );
+        }
+      }break;
+      }//end switch
+      if(c)
+      {
+        if(pattern[(*char_len) + 1] != '}')
+            throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_BROKEN_P_CONSTRUCT)) );
+        (*char_len)++;
+        (*char_len)++;
+        return new CRegexXQuery_multicharP(current_regex, c, is_reverse);
+      }
+      if(pattern[*char_len] == 'I')
+      {
+        if(pattern[(*char_len)+1] == 's')//IsBlock
+        {
+          *multichar_type = CHARGROUP_FLAGS_MULTICHAR_Is;
+          (*char_len) += 2;
+          zstring block_name;
+          char tempc = pattern[(*char_len)];
+          while(tempc && (tempc != '}'))
+          {
+            if(!myisletterAZ(tempc) && !myisdigit(tempc) && (tempc != '-'))
+              throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_BROKEN_PIs_CONSTRUCT)) );
+            block_name.append(1, tempc);
+            (*char_len)++;
+            tempc = pattern[(*char_len)];
+          }
+          if(!tempc)
+            throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_BROKEN_PIs_CONSTRUCT)) );
+          //search for the block name
+          int i;
+          int nr_blocks = sizeof(block_escape)/sizeof(CRegexXQuery_parser::block_escape_t);
+          for(i=0;i<nr_blocks;i++)
+          {
+            if(compare_ascii_i(block_name.c_str(), block_escape[i].group_name))
+            {
+              c = i;
+              break;
+            }
+          }
+          if(i==nr_blocks)
+            throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PIs_CONSTRUCT)) );
           (*char_len)++;
-      }
-      break;
+          return new CRegexXQuery_multicharIs(current_regex, i, is_reverse);
+        }
+        else
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_BROKEN_PIs_CONSTRUCT)) );
+      }
+      else
+      {
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_BROKEN_P_CONSTRUCT)) );
+      }
+      break;//unreachable
+    }//end case 'p'
       //multiCharEsc
     case 's':
     case 'S':
@@ -240,40 +569,104 @@
     case 'D':
     case 'w':
     case 'W':
-      *is_multichar = true;
+       *multichar_type = CHARGROUP_FLAGS_MULTICHAR_OTHER;
        c = pattern[*char_len];
-       break;
-    }
-    break;
-  }
-  case '#':///might be #xXX
-  {
-    if((pattern[*char_len+1] == 'x') &&
-      myishex(pattern[*char_len+2]) && myishex(pattern[*char_len+3]))
-    {
-      c = (myishex(pattern[*char_len+2])-1)<<4 | (myishex(pattern[*char_len+3])-1);
-      *char_len += 3;
-      break;
-    }
-  }
+       (*char_len)++;
+       return new CRegexXQuery_multicharOther(current_regex, c);
+    case 'u'://unicode codepoint \uXXXX
+    {
+      unicode::code_point utf8c = 0;
+      (*char_len)++;
+      for(int i=0;i<4;i++)
+      {
+        char hex = myishex(pattern[*char_len]);
+        if(!hex)
+        {
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_UNICODE_CODEPOINT_u)) );
+        }
+        utf8c <<= 4;
+        utf8c |= (hex-1) & 0x0f;
+        (*char_len)++;
+      }
+      return create_charmatch(utf8c, NULL, 0, multichar_type);
+    }
+    case 'U'://unicode codepoint \UXXXXXXXX
+    {
+      unicode::code_point utf8c = 0;
+      (*char_len)++;
+      for(int i=0;i<8;i++)
+      {
+        char hex = myishex(pattern[*char_len]);
+        if(!hex)
+        {
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_UNICODE_CODEPOINT_u)) );
+        }
+        utf8c <<= 4;
+        utf8c |= (hex-1) & 0x0f;
+        (*char_len)++;
+      }
+      return create_charmatch(utf8c, NULL, 0, multichar_type);
+    }
+    default:
+      throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_ESC_CHAR)) );
+    }
+    assert(false);
+    break;//unreachable
+  }//end case '\'
   default:
-     c = pattern[*char_len];
-     break;
-  }
-
-  (*char_len)++;
-  return c;
-}
-
-
-
-IRegexAtom* CRegexAscii_parser::read_atom(const char *pattern, int *atom_len)
+  {
+    const char *temp_pattern = pattern;
+    unicode::code_point utf8c = utf8::next_char(temp_pattern);
+    (*char_len) = temp_pattern - pattern;
+    return create_charmatch(utf8c, pattern, *char_len, multichar_type);
+  }
+  }
+  return NULL;
+}
+
+CRegexXQuery_charmatch *CRegexXQuery_parser::create_charmatch(unicode::code_point utf8c,
+                                                              const char *pattern, int utf8len,
+                                                              enum CHARGROUP_t *multichar_type)
+{
+  if(utf8c <= 0x7F)
+  {
+    *multichar_type = CHARGROUP_FLAGS_ONECHAR_ASCII;
+    if(flags & REGEX_ASCII_CASE_INSENSITIVE)
+      return new CRegexXQuery_char_ascii_i(current_regex, (char)utf8c);
+    else
+      return new CRegexXQuery_char_ascii(current_regex, (char)utf8c);
+  }
+  else
+  {
+    *multichar_type = CHARGROUP_FLAGS_ONECHAR_UNICODE;
+    if(flags & REGEX_ASCII_CASE_INSENSITIVE)
+      return new CRegexXQuery_char_unicode_i(current_regex, utf8c);
+    else
+    {
+      if(pattern)
+        return new CRegexXQuery_char_unicode(current_regex, pattern, utf8len);
+      else
+        return new CRegexXQuery_char_unicode_cp(current_regex, utf8c);
+    }
+  }
+}
+
+IRegexAtom* CRegexXQuery_parser::read_atom(const char *pattern, int *atom_len)
 {
   *atom_len = 0;
-  char  c;
-  bool is_end_line = false;
-  c = pattern[*atom_len];
-  if((!(flags & REGEX_ASCII_LITERAL)) && (c == '\\'))
+  if(flags & REGEX_ASCII_LITERAL)
+  {
+    unicode::code_point  utf8c;
+    //bool is_end_line = false;
+    const char *temp_pattern = pattern;
+    utf8c = utf8::next_char(temp_pattern);
+    *atom_len = temp_pattern - pattern;
+    enum CHARGROUP_t multichar_type;
+    return create_charmatch(utf8c, pattern, *atom_len, &multichar_type);
+  }
+
+  char c = *pattern;
+  if(c == '\\')
   {
     //check for back reference
     if(myisdigit(pattern[(*atom_len)+1]))
@@ -281,13 +674,13 @@
       (*atom_len)++;
       if(pattern[*atom_len] == '0')
       {
-        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(U_REGEX_INVALID_BACK_REF)) );
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_BACK_REF), 0, current_regex->subregex.size()) );
       }
       unsigned int backref = pattern[*atom_len] - '0';
       if((backref > current_regex->subregex.size()) ||
         (current_regex->subregex.at(backref-1)->flags != 0))
       {
-        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(U_REGEX_INVALID_BACK_REF)) );
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_BACK_REF), backref, current_regex->subregex.size()) );
       }
       while(current_regex->subregex.size() >= backref*10)
       {
@@ -303,70 +696,86 @@
             break;
         }
       }
-      return new CRegexAscii_backref(current_regex, backref);
+      (*atom_len)++;
+      return new CRegexXQuery_backref(current_regex, backref);
     }
   }
+  if(c == '^')
+  {
+    (*atom_len)++;
+    return new CRegexXQuery_pinstart(current_regex);
+  }
+  if((c == '}') || (c == '{') || (c == '?') || (c == '*') || (c == '+') || (c == '|'))
+  {
+    throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_ATOM_CHAR), c) );
+  }
   switch(c)
   {
   case '[':
   {
-    if(!(flags & REGEX_ASCII_LITERAL))
-    {
-      (*atom_len)++;
-      CRegexAscii_chargroup *chargroup = NULL;
-      int chargroup_len;
-      chargroup = readchargroup(pattern+*atom_len, &chargroup_len);
-      *atom_len += chargroup_len;
-      return chargroup;
-    }
+    (*atom_len)++;
+    CRegexXQuery_chargroup *chargroup = NULL;
+    int chargroup_len;
+    chargroup = readchargroup(pattern+*atom_len, &chargroup_len);
+    *atom_len += chargroup_len;
+    return chargroup;
   }
   case '.'://WildCharEsc
   {
-    if(!(flags & REGEX_ASCII_LITERAL))
-    {
-      CRegexAscii_wildchar  *wildchar = new CRegexAscii_wildchar(current_regex);
-      (*atom_len)++;
-      return wildchar;
-    }
+    (*atom_len)++;
+    return new CRegexXQuery_wildchar(current_regex);
   }
   case '('://begin an embedded reg exp
   {  
-    if(!(flags & REGEX_ASCII_LITERAL))
-    {
-      (*atom_len)++;
-      CRegexAscii_regex *emb_regex = NULL;
-      int   regex_len;
-      emb_regex = parse_regexp(pattern + *atom_len, &regex_len);
-      *atom_len += regex_len;
-      return emb_regex;
-    }
+    (*atom_len)++;
+    CRegexXQuery_regex *emb_regex = NULL;
+    int   regex_len;
+    emb_regex = parse_regexp(pattern + *atom_len, &regex_len);
+    *atom_len += regex_len;
+    return emb_regex;
   }
   case '$'://end line
-    if(!(flags & REGEX_ASCII_LITERAL))
-    {
-      is_end_line = true;
-    }
+    //is_end_line = true;
+    (*atom_len)++;
+    return new CRegexXQuery_endline(current_regex);
   default:
   {  
-    char  c;
+    //char  c;
+    CRegexXQuery_charmatch *charmatch = NULL;
     int   c_len;
-    bool  is_multichar = false;
-    if(!(flags & REGEX_ASCII_LITERAL))
-      c = readChar(pattern+*atom_len, &c_len, &is_multichar);
-    else
+    CHARGROUP_t   multichar_type = CHARGROUP_NO_MULTICHAR;
+    *atom_len = 0;
+    while(pattern[*atom_len])
     {
-      c = pattern[*atom_len];
-      c_len = 1;
+      charmatch = readChar(pattern+*atom_len, &c_len, &multichar_type);
+      *atom_len += c_len;
+      if((flags & REGEX_ASCII_NO_WHITESPACE) && (multichar_type == CHARGROUP_FLAGS_ONECHAR_ASCII))
+      {
+        char c = (char)charmatch->get_c();
+        if((c == ' ') || (c == '\t') || (c == '\r') || (c == '\n'))
+        {
+          //ignore this whitespace
+          delete charmatch;
+          continue;
+        }
+        else
+          break;
+      }
+      else
+        break;
     }
-    CRegexAscii_chargroup *chargroup = new CRegexAscii_chargroup(current_regex);
-    if(is_multichar)
-      chargroup->addMultiChar(c);
+    /*
+    std::auto_ptr<CRegexXQuery_chargroup> chargroup(new CRegexXQuery_chargroup(current_regex));
+    if(multichar_type)
+      chargroup->addMultiChar(c, multichar_type);
     else if(is_end_line)
       chargroup->addEndLine();
     else
-      chargroup->addCharRange(c, c);
+      chargroup->addOneChar(c);
     *atom_len += c_len;
-    return chargroup;
+    return chargroup.release();
+    */
+    return charmatch;
   }
   }
 }
@@ -374,81 +783,119 @@
 //read until ']'
 //posCharGroup  ::=   ( charRange | charClassEsc )+  
 //charRange     ::=    seRange | XmlCharIncDash
-CRegexAscii_chargroup* CRegexAscii_parser::readchargroup(const char *pattern, int *chargroup_len)
+CRegexXQuery_chargroup* CRegexXQuery_parser::readchargroup(const char *pattern, int *chargroup_len)
 {
-  CRegexAscii_chargroup *chargroup = NULL;
+  std::auto_ptr<CRegexXQuery_chargroup> chargroup;
   *chargroup_len = 0;
   if(pattern[*chargroup_len] == '^')//negative group
   {
     (*chargroup_len)++;
-    chargroup = new CRegexAscii_negchargroup(current_regex);
+    chargroup.reset(new CRegexXQuery_negchargroup(current_regex));
   }
   else
-    chargroup = new CRegexAscii_chargroup(current_regex);
+    chargroup.reset(new CRegexXQuery_chargroup(current_regex));
   while(pattern[*chargroup_len] && (pattern[*chargroup_len]!=']'))
   {
-    char  c1, c2;
-    bool  is_multichar;
+    //char  c1, c2;
+    CHARGROUP_t  multichar_type = CHARGROUP_NO_MULTICHAR;
     int   c1_len;
-    c1 = pattern[*chargroup_len];
-    c2 = pattern[*chargroup_len+1];
-    if((c1 == '-') && (c2 == '['))//charClassSub
+    if((pattern[*chargroup_len] == '-') && (pattern[(*chargroup_len)+1] == '['))//charClassSub
     {
       int classsub_len;
-      CRegexAscii_chargroup *classsub = readchargroup(pattern + *chargroup_len+1 + 1, &classsub_len);
+      CRegexXQuery_chargroup *classsub = readchargroup(pattern + (*chargroup_len)+1 + 1, &classsub_len);
       if(!classsub)
       {
-        delete chargroup;
-        return NULL;
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_SUBCLASS)) );
       }
       chargroup->addClassSub(classsub);
       *chargroup_len += 2 + classsub_len + 1;
       if(pattern[*chargroup_len-1] != ']')
       {
-        delete chargroup;
-        return NULL;
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_USE_OF_SUBCLASS)) );
       }
-      return chargroup;
+      return chargroup.release();
     }
 
-    c1 = readChar(pattern+*chargroup_len, &c1_len, &is_multichar);
-    if(is_multichar)//first char is multichar
+    std::unique_ptr<CRegexXQuery_charmatch> charmatch(readChar(pattern+*chargroup_len, &c1_len, &multichar_type));
+    if((multichar_type == CHARGROUP_FLAGS_MULTICHAR_p) ||
+      (multichar_type == CHARGROUP_FLAGS_MULTICHAR_Is) ||
+      (multichar_type == CHARGROUP_FLAGS_MULTICHAR_OTHER))//first char is multichar
     {
-      chargroup->addMultiChar(c1);
+      if((pattern[*chargroup_len+c1_len] == '-') &&///should not be a range
+        (pattern[*chargroup_len+c1_len+1] != ']'))
+      {
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_MULTICHAR_IN_CHAR_RANGE)) );
+      }
+      //chargroup->addMultiChar(c1, multichar_type);
+      chargroup->addCharMatch(charmatch.release());
       *chargroup_len += c1_len;
       continue;
     }
-    if(pattern[*chargroup_len+c1_len] == '-')///might be a range
+    (*chargroup_len) += c1_len;
+    if(pattern[*chargroup_len] == '-')///might be a range
     {
-      if(pattern[*chargroup_len+c1_len+1] == ']')//no range, just the last char is '-'
+      if(pattern[(*chargroup_len)+1] == ']')//no range, just the last char is '-'
       {
-        chargroup->addCharRange(c1, c1);
-        chargroup->addCharRange('-', '-');
-        *chargroup_len += c1_len + 1;
+        //chargroup->addOneChar(c1);
+        //chargroup->addOneChar('-');
+        chargroup->addCharMatch(charmatch.release());
+        chargroup->addCharMatch(new CRegexXQuery_char_ascii(current_regex, '-'));
+        (*chargroup_len)++;
         continue;
       }
-      else
+      else if(pattern[(*chargroup_len)+1] != '[')
       {
         //it is a range
-        char c3;
-        int  c3_len;
-        c3 = readChar(pattern+*chargroup_len+c1_len+1, &c3_len, &is_multichar);
-        if(is_multichar)
-          return NULL;//error
-        chargroup->addCharRange(c1, c3);
-        *chargroup_len += c1_len + 1 + c3_len;
+        (*chargroup_len)++;
+        std::unique_ptr<CRegexXQuery_charmatch>  charmatch2;
+        CHARGROUP_t  multichar_type2 = CHARGROUP_NO_MULTICHAR;
+        int  c2_len;
+        charmatch2.reset(readChar(pattern+(*chargroup_len), &c2_len, &multichar_type2));
+        if((multichar_type2 != CHARGROUP_FLAGS_ONECHAR_ASCII) &&
+          (multichar_type2 != CHARGROUP_FLAGS_ONECHAR_ASCII))//second char in range is multichar
+        {
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_MULTICHAR_IN_CHAR_RANGE)) );
+        }
+        //chargroup->addCharRange(c1, c3);
+        if((multichar_type == CHARGROUP_FLAGS_ONECHAR_ASCII) && (multichar_type2 == CHARGROUP_FLAGS_ONECHAR_ASCII))
+        {
+          if(flags & REGEX_ASCII_CASE_INSENSITIVE)
+            chargroup->addCharMatch(new CRegexXQuery_char_range_ascii_i(current_regex, 
+                                                                    (char)charmatch->get_c(),
+                                                                    (char)charmatch2->get_c()));
+          else
+            chargroup->addCharMatch(new CRegexXQuery_char_range_ascii(current_regex, 
+                                                                    (char)charmatch->get_c(),
+                                                                    (char)charmatch2->get_c()));
+        }
+        else
+        {
+          if(flags & REGEX_ASCII_CASE_INSENSITIVE)
+            chargroup->addCharMatch(new CRegexXQuery_char_range_unicode_i(current_regex, 
+                                                                    charmatch->get_c(),
+                                                                    charmatch2->get_c()));
+          else
+            chargroup->addCharMatch(new CRegexXQuery_char_range_unicode(current_regex, 
+                                                                    charmatch->get_c(),
+                                                                    charmatch2->get_c()));
+        }
+        *chargroup_len += c2_len;
         continue;
       }
     }
-    chargroup->addCharRange(c1, c1);
-    *chargroup_len += c1_len;
+    //chargroup->addOneChar(c1);
+    chargroup->addCharMatch(charmatch.release());
   }
   if(pattern[*chargroup_len])
     (*chargroup_len)++;
-  return chargroup;
+  else
+  {
+    throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_MISSING_CLOSE_BRACKET)) );
+  }
+  return chargroup.release();
 }
 
-void CRegexAscii_parser::read_quantifier(CRegexAscii_piece *piece,
+void CRegexXQuery_parser::read_quantifier(CRegexXQuery_piece *piece,
                                          const char *pattern, int *quantif_len)
 {
   *quantif_len = 0;
@@ -496,6 +943,10 @@
         max = max*10 + pattern[*quantif_len] - '0';
         (*quantif_len)++;
       }
+      if(max < min)
+      {
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_MAX_LT_MIN)) );
+      }
       piece->set_quantifier_min_max(min, max, true);
     }
     while(pattern[*quantif_len] && (pattern[*quantif_len] != '}'))
@@ -524,23 +975,25 @@
 ///Constructors and destructors and internal functions
 ////////////////////////////
 
-CRegexAscii_regex::CRegexAscii_regex(CRegexAscii_regex *topregex) : IRegexAtom(topregex?topregex:this)
+CRegexXQuery_regex::CRegexXQuery_regex(CRegexXQuery_regex *topregex) : IRegexAtom(topregex?topregex:this)
 {
   matched_source = NULL;
   matched_len = 0;
+//  backup_matched_source = NULL;
+//  backup_matched_len = 0;
   flags = 128;//set to 0 after initialization
 }
 
-CRegexAscii_regex::~CRegexAscii_regex()
+CRegexXQuery_regex::~CRegexXQuery_regex()
 {
-  std::list<CRegexAscii_branch*>::iterator  branch_it;
+  std::list<CRegexXQuery_branch*>::iterator  branch_it;
 
   for(branch_it = branch_list.begin(); branch_it != branch_list.end(); branch_it++)
   {
     delete (*branch_it);
   }
 /*
-  std::vector<CRegexAscii_regex*>::iterator   subregex_it;
+  std::vector<CRegexXQuery_regex*>::iterator   subregex_it;
   for(subregex_it = subregex.begin(); subregex_it != subregex.end(); subregex_it++)
   {
     delete (*subregex_it);
@@ -548,25 +1001,18 @@
 */
 }
 
-bool CRegexAscii_regex::set_align_begin(bool align_begin)
-{
-  bool prev_align = this->align_begin;
-  this->align_begin = align_begin;
-  return prev_align;
-}
-
-void CRegexAscii_regex::add_branch(CRegexAscii_branch *branch)
+void CRegexXQuery_regex::add_branch(CRegexXQuery_branch *branch)
 {
   branch_list.push_back(branch);
 }
 
-bool  CRegexAscii_regex::get_indexed_match(int index, 
+bool  CRegexXQuery_regex::get_indexed_match(int index, 
                                            const char **matched_source, 
                                            int *matched_len)
 {
   if(!index || index > (int)subregex.size())
     return false;
-  CRegexAscii_regex *subr = subregex[index-1];
+  CRegexXQuery_regex *subr = subregex[index-1];
   *matched_source = subr->matched_source;
   if(!*matched_source)
     return false;
@@ -574,145 +1020,209 @@
   return true;
 }
 
-unsigned int CRegexAscii_regex::get_indexed_regex_count()
+unsigned int CRegexXQuery_regex::get_indexed_regex_count()
 {
   return subregex.size();
 }
 
-CRegexAscii_branch::CRegexAscii_branch(CRegexAscii_regex* regex) :
-      IRegexMatcher(regex)
+CRegexXQuery_branch::CRegexXQuery_branch(CRegexXQuery_regex* regex) 
+      //:
+      //IRegexMatcher(regex)
 {
 }
 
-CRegexAscii_branch::~CRegexAscii_branch()
+CRegexXQuery_branch::~CRegexXQuery_branch()
 {
-  std::list<CRegexAscii_piece*>::iterator  piece_it;
+  std::list<RegexAscii_pieceinfo>::iterator  piece_it;
 
   for(piece_it = piece_list.begin(); piece_it != piece_list.end(); piece_it++)
   {
-    delete (*piece_it);
+    delete (*piece_it).piece;
   }
 }
 
-void CRegexAscii_branch::add_piece(CRegexAscii_piece *piece)
+void CRegexXQuery_branch::add_piece(CRegexXQuery_piece *piece)
 {
   piece_list.push_back(piece);
 }
 
-CRegexAscii_piece::CRegexAscii_piece()
+CRegexXQuery_piece::CRegexXQuery_piece()
 {
+  atom = NULL;
+  regex_atom = NULL;
 }
 
-CRegexAscii_piece::~CRegexAscii_piece()
+CRegexXQuery_piece::~CRegexXQuery_piece()
 {
   delete atom;
 }
 
-void CRegexAscii_piece::set_atom(IRegexAtom *atom)
+void CRegexXQuery_piece::set_atom(IRegexAtom *atom)
 {
   this->atom = atom;
+  this->regex_atom = dynamic_cast<CRegexXQuery_regex*>(atom);
 }
 
-void CRegexAscii_piece::set_quantifier_min_max(int min, int max, bool strict_max)
+void CRegexXQuery_piece::set_quantifier_min_max(int min, int max, bool strict_max)
 {
   this->min = min;
   this->max = max;
   this->strict_max = strict_max;
 }
-void CRegexAscii_piece::set_is_reluctant(bool is_reluctant)
+void CRegexXQuery_piece::set_is_reluctant(bool is_reluctant)
 {
   this->is_reluctant = is_reluctant;
 }
-void CRegexAscii_piece::get_quantifier(int *min, int *max, bool *strict_max)
+void CRegexXQuery_piece::get_quantifier(int *min, int *max, bool *strict_max)
 {
   *min = this->min;
   *max = this->max;
   *strict_max = this->strict_max;
 }
-bool CRegexAscii_piece::get_is_reluctant()
+bool CRegexXQuery_piece::get_is_reluctant()
 {
+  if(atom->regex_intern->flags & REGEX_ASCII_MINIMAL_MATCH)
+    return true;
   return is_reluctant;
 }
 
 
-CRegexAscii_chargroup::CRegexAscii_chargroup(CRegexAscii_regex* regex) :
+CRegexXQuery_charmatch::CRegexXQuery_charmatch(CRegexXQuery_regex* regex) :
+    IRegexAtom(regex)
+{
+}
+CRegexXQuery_multicharP::CRegexXQuery_multicharP(CRegexXQuery_regex* regex, char type, bool is_reverse) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->multichar_type = type; this->is_reverse = is_reverse;
+}
+CRegexXQuery_multicharIs::CRegexXQuery_multicharIs(CRegexXQuery_regex* regex, int block_index, bool is_reverse) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->block_index = block_index; this->is_reverse = is_reverse;
+}
+CRegexXQuery_multicharOther::CRegexXQuery_multicharOther(CRegexXQuery_regex* regex, char type) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->multichar_type = type;
+}
+CRegexXQuery_char_ascii::CRegexXQuery_char_ascii(CRegexXQuery_regex* regex, char c) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->c = c;
+}
+CRegexXQuery_char_ascii_i::CRegexXQuery_char_ascii_i(CRegexXQuery_regex* regex, char c) :
+    CRegexXQuery_char_ascii(regex, toupper(c))
+{
+}
+CRegexXQuery_char_range_ascii::CRegexXQuery_char_range_ascii(CRegexXQuery_regex* regex, char c1, char c2) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->c1 = c1; this->c2 = c2;
+}
+CRegexXQuery_char_range_ascii_i::CRegexXQuery_char_range_ascii_i(CRegexXQuery_regex* regex, char c1, char c2) :
+    CRegexXQuery_char_range_ascii(regex, toupper(c1), toupper(c2))
+{
+}
+CRegexXQuery_char_unicode::CRegexXQuery_char_unicode(CRegexXQuery_regex* regex, const char *source, int len) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->len = len;
+  memcpy(c, source, len);
+}
+CRegexXQuery_char_unicode_cp::CRegexXQuery_char_unicode_cp(CRegexXQuery_regex* regex, unicode::code_point c) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->c = c;
+}
+CRegexXQuery_char_unicode_i::CRegexXQuery_char_unicode_i(CRegexXQuery_regex* regex, unicode::code_point c) :
+    CRegexXQuery_char_unicode_cp(regex, unicode::to_upper(c))
+{
+}
+CRegexXQuery_char_range_unicode::CRegexXQuery_char_range_unicode(CRegexXQuery_regex* regex, unicode::code_point c1, unicode::code_point c2) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->c1 = c1; this->c2 = c2;
+}
+CRegexXQuery_char_range_unicode_i::CRegexXQuery_char_range_unicode_i(CRegexXQuery_regex* regex, unicode::code_point c1, unicode::code_point c2) :
+    CRegexXQuery_char_range_unicode(regex, unicode::to_upper(c1), unicode::to_upper(c2))
+{
+}
+CRegexXQuery_endline::CRegexXQuery_endline(CRegexXQuery_regex* regex) :
+    CRegexXQuery_charmatch(regex)
+{
+}
+
+unicode::code_point CRegexXQuery_char_unicode::get_c()
+{
+  const char *temp_c = (const char*)c;
+  return utf8::next_char(temp_c);
+}
+
+
+CRegexXQuery_chargroup::CRegexXQuery_chargroup(CRegexXQuery_regex* regex) :
     IRegexAtom(regex)
 {
   classsub = NULL;
 }
 
-CRegexAscii_chargroup::~CRegexAscii_chargroup()
+CRegexXQuery_chargroup::~CRegexXQuery_chargroup()
 {
   delete classsub;
-}
-
-void CRegexAscii_chargroup::addMultiChar(char c)
-{
-  chargroup_t cgt;
-  cgt.flags = CHARGROUP_FLAGS_MULTICHAR;
-  cgt.c1 = c;
-  cgt.c2 = 0;
-  chargroup_list.push_back(cgt);
-}
-
-void CRegexAscii_chargroup::addEndLine()
-{
-  chargroup_t cgt;
-  cgt.flags = CHARGROUP_FLAGS_ENDLINE;
-  cgt.c1 = '$';
-  cgt.c2 = 0;
-  chargroup_list.push_back(cgt);
-}
-
-void CRegexAscii_chargroup::addCharRange(char c1, char c2)
-{
-  chargroup_t cgt;
-  cgt.flags = 0;
-  cgt.c1 = c1;
-  cgt.c2 = c2;
-  chargroup_list.push_back(cgt);
-}
-
-void CRegexAscii_chargroup::addClassSub(CRegexAscii_chargroup* classsub)
+  std::list<CRegexXQuery_charmatch* >::iterator charmatch_it;
+  for(charmatch_it=chargroup_list.begin(); charmatch_it != chargroup_list.end(); charmatch_it++)
+    delete (*charmatch_it);
+}
+
+void CRegexXQuery_chargroup::addCharMatch(CRegexXQuery_charmatch *charmatch)
+{
+  chargroup_list.push_back(charmatch);
+}
+void CRegexXQuery_chargroup::addClassSub(CRegexXQuery_chargroup* classsub)
 {
   this->classsub = classsub;
 }
 
-CRegexAscii_negchargroup::CRegexAscii_negchargroup(CRegexAscii_regex* regex) :
-  CRegexAscii_chargroup(regex)
-{
-}
-
-CRegexAscii_negchargroup::~CRegexAscii_negchargroup()
-{
-}
-
-CRegexAscii_wildchar::CRegexAscii_wildchar(CRegexAscii_regex* regex) :
+CRegexXQuery_negchargroup::CRegexXQuery_negchargroup(CRegexXQuery_regex* regex) :
+  CRegexXQuery_chargroup(regex)
+{
+}
+
+CRegexXQuery_negchargroup::~CRegexXQuery_negchargroup()
+{
+}
+
+CRegexXQuery_wildchar::CRegexXQuery_wildchar(CRegexXQuery_regex* regex) :
       IRegexAtom(regex)
 {
 }
 
-CRegexAscii_wildchar::~CRegexAscii_wildchar()
+CRegexXQuery_wildchar::~CRegexXQuery_wildchar()
 {
 }
 
-CRegexAscii_backref::CRegexAscii_backref(CRegexAscii_regex* regex, unsigned int backref_) :
+CRegexXQuery_backref::CRegexXQuery_backref(CRegexXQuery_regex* regex, unsigned int backref_) :
       IRegexAtom(regex),
       backref(backref_)
 {
 }
 
-CRegexAscii_backref::~CRegexAscii_backref()
-{
-}
-
-CRegexAscii_parser::CRegexAscii_parser()
+CRegexXQuery_backref::~CRegexXQuery_backref()
+{
+}
+
+CRegexXQuery_pinstart::CRegexXQuery_pinstart(CRegexXQuery_regex* regex):
+      IRegexAtom(regex)
+{
+}
+
+CRegexXQuery_parser::CRegexXQuery_parser()
 {
   current_regex = NULL;
   regex_depth = 0;
 }
 
-CRegexAscii_parser::~CRegexAscii_parser()
+CRegexXQuery_parser::~CRegexXQuery_parser()
 {
 }
 
@@ -720,9 +1230,68 @@
 //////////////////////////////////////////
 ////Matching the pattern on a string
 /////////////////////////////////////////
+static std::list<RegexAscii_pieceinfo> empty_pieces;//empty list of pieces
+/*
+std::list<RegexAscii_pieceinfo>::iterator  
+IRegexAtom::choose_next_piece(const char *source, int *matched_len, 
+                              std::list<RegexAscii_pieceinfo>::iterator this_piece,
+                              std::list<RegexAscii_pieceinfo>::iterator end_piece)
+{
+  //if this_piece is repetition, repeat until max, then go to next piece
+  int min, max;
+  bool strict_max;
+  while(this_piece != end_piece)
+  {
+    (*this_piece).piece->get_quantifier(&min, &max, &strict_max);
+    if(max <= ((*this_piece).nr_matches))//finished this piece
+    {
+      this_piece++;
+    }
+    else
+      break;
+  }
+  return this_piece;
+}
+*/
+
+bool IRegexAtom::match(const char *source, int *start_from_branch, int *matched_len,
+                  std::list<RegexAscii_pieceinfo>::iterator this_piece,
+                  std::list<RegexAscii_pieceinfo>::iterator end_piece)
+{
+  *start_from_branch = 0;
+  bool retmatch;
+  retmatch = match_internal(source, start_from_branch, matched_len);
+  if(!retmatch)
+    return false;
+
+  if(this_piece == end_piece)
+    return true;
+
+  (*this_piece).nr_matches++;
+  int min,max;
+  bool strict_max;
+  (*this_piece).piece->get_quantifier(&min, &max, &strict_max);
+  std::list<RegexAscii_pieceinfo>::iterator init_piece = this_piece;
+  if(((min == 1) && (max == 1)) || //the simple common case
+    ((*matched_len == 0) && ((*this_piece).nr_matches>=min)))//to avoid infinite loop
+  {
+    this_piece++;
+    if(this_piece == end_piece)
+      return true;
+  }
+  int matched_len2;
+  retmatch = (*this_piece).piece->match_piece(this_piece, end_piece, source + *matched_len, &matched_len2);
+  if(!retmatch)
+  {
+    (*init_piece).nr_matches--;
+    return false;
+  }
+  *matched_len += matched_len2;
+  return true;
+}
 
 //try every position in source to match the pattern
-bool CRegexAscii_regex::match_anywhere(const char *source, unsigned int flags,
+bool CRegexXQuery_regex::match_anywhere(const char *source, unsigned int flags,
                                        int *match_pos, int *matched_len)
 {
   *match_pos = 0;
@@ -730,43 +1299,66 @@
   return match_from(source, flags, match_pos, matched_len);
 }
 
-bool CRegexAscii_regex::match_from(const char *source, unsigned int flags,
+bool CRegexXQuery_regex::match_from(const char *source, unsigned int flags,
                                        int *match_pos, int *matched_len)
 {
   this->flags = flags;
+  this->source_start = source;
   reachedEnd = false;
 
-  std::vector<CRegexAscii_regex*>::iterator regex_it;
+  std::vector<CRegexXQuery_regex*>::iterator regex_it;
   for(regex_it = subregex.begin(); regex_it != subregex.end(); regex_it++)
   {
     (*regex_it)->matched_source = NULL;
   }
-//  if(!source[0])
-//  {
-//    if(branch_list.empty())
-//      return true;
-//    else
-//      return false;
-//  }
-
-  bool  skip_first_match = false;
-  if(*match_pos && align_begin)
-    skip_first_match = true;
+
+  std::vector<std::pair<const char*, int> >  saved_subregex;
+
+  if(*match_pos && (flags & REGEX_ASCII_WHOLE_MATCH))
+    return false;
+
   do
   {
-    if(!skip_first_match)
-    {
-      if(match(source + *match_pos, matched_len))
-        return true;
-    }
-    skip_first_match = false;
-    if(align_begin)
+    int   start_from_branch = 0;
+    int   longest_match = -1;
+    while(1)
+    {
+      if(!match(source + *match_pos, &start_from_branch, matched_len, empty_pieces.begin(), empty_pieces.end()))
+        break;
+      if(longest_match < *matched_len)
+      {
+        longest_match = *matched_len;
+        if(start_from_branch && (flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+          save_subregex_list(saved_subregex);
+      }
+      if(!start_from_branch || !(flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+        break;
+      //else try the other branches to see which is longer
+    }
+    if(longest_match != -1)
+    {
+      *matched_len = longest_match;
+      if(saved_subregex.size())
+        load_subregex_list(saved_subregex);
+      if(flags & REGEX_ASCII_WHOLE_MATCH)
+      {
+        if(!source[*match_pos+*matched_len])
+          return true;
+        if((flags & REGEX_ASCII_MULTILINE) && 
+          ((source[*match_pos+*matched_len] == '\n') || (source[*match_pos+*matched_len] == '\r')))
+          return true;
+        return false;
+      }
+      return true;
+    }
+
+    if(flags & REGEX_ASCII_WHOLE_MATCH)
     {
       if(flags & REGEX_ASCII_MULTILINE)
       {
-        //goto the next line
+        //go to next line
         while(source[*match_pos] && (source[*match_pos] != '\n') && (source[*match_pos] != '\r'))
-          (*match_pos)++;
+          (*match_pos) += myutf8len(source);
         if(source[*match_pos] == '\n')
         {
           (*match_pos)++;
@@ -780,190 +1372,1039 @@
             (*match_pos)++;
         }
         if(!source[*match_pos])
-          return false;
+          break;
         continue;
       }
-      return false;
+      break;
     }
     if(!source[*match_pos])
       break;
-    (*match_pos)++;
+    (*match_pos) += myutf8len(source);
   }
   while(source[*match_pos]);
+//  if(!source[*match_pos])
+//  {
+//    reachedEnd = true;
+//  }
   return false;
 }
 
+void CRegexXQuery_regex::reset_match()
+{
+//  this->backup_matched_source = this->matched_source;
+//  this->backup_matched_len = this->matched_len;
+  this->matched_source = NULL;
+  this->matched_len = 0;
+  std::list<CRegexXQuery_branch*>::iterator  branch_it;
+  for(branch_it = branch_list.begin(); branch_it != branch_list.end(); branch_it++)
+  {
+    (*branch_it)->reset();
+  }
+}
+/*
+void CRegexXQuery_regex::restore_match()
+{
+  this->matched_source = this->backup_matched_source;
+  this->matched_len = this->backup_matched_len;
+  std::list<CRegexXQuery_branch*>::iterator  branch_it;
+  for(branch_it = branch_list.begin(); branch_it != branch_list.end(); branch_it++)
+  {
+    (*branch_it)->restore();
+  }
+}
+*/
 //match any of the branches
-bool CRegexAscii_regex::match(const char *source, int *matched_len)
+bool CRegexXQuery_regex::match(const char *source, int *start_from_branch, int *matched_len,
+                              std::list<RegexAscii_pieceinfo>::iterator next_piece,
+                              std::list<RegexAscii_pieceinfo>::iterator end_piece)
 {
   reachedEnd = false;
-  std::list<CRegexAscii_branch*>::iterator  branch_it;
-
-  for(branch_it = branch_list.begin(); branch_it != branch_list.end(); branch_it++)
-  {
-    if((*branch_it)->match(source, matched_len))
-    {
-      matched_source = source;
-      this->matched_len = *matched_len;
+  if(!(flags & REGEX_ASCII_GROUPING_LEN_WHOLE_PIECE) || 
+    (this->matched_source == NULL) || ((this->matched_source + this->matched_len) != source))
+    this->matched_source = source;
+  *matched_len = 0;
+  std::list<CRegexXQuery_branch*>::iterator  branch_it;
+
+  if(*start_from_branch == 0)
+  {
+    for(branch_it = branch_list.begin(); branch_it != branch_list.end(); branch_it++)
+    {
+      (*branch_it)->reset();
+    }
+  }
+
+  branch_it = branch_list.begin();
+  if(*start_from_branch)
+  {
+    for(int i=0;i<*start_from_branch;i++)
+      branch_it++;
+  }
+  (*start_from_branch)++;
+  for(; branch_it != branch_list.end(); branch_it++,(*start_from_branch)++)
+  {
+    if((*branch_it)->match(source, matched_len, this, next_piece, end_piece))
+    {
+      //matched_source = source;
+      //this->matched_len = *matched_len;
       return true;
     }
   }
-  matched_source = NULL;
-  matched_len = 0;
+  *start_from_branch = 0;
+  if(this->matched_source == source)
+    this->matched_source = NULL;
+  *matched_len = 0;
   return false;
 }
 
+void CRegexXQuery_regex::save_subregex_list(std::vector<std::pair<const char*, int> > &saved_subregex)
+{
+  saved_subregex.resize(0);
+  saved_subregex.reserve(subregex.size());
+  std::vector<CRegexXQuery_regex*>::iterator   it;
+  for(it=subregex.begin(); it != subregex.end(); it++)
+  {
+    saved_subregex.push_back(std::pair<const char*, int>((*it)->matched_source, (*it)->matched_len));
+  }
+}
+
+void CRegexXQuery_regex::load_subregex_list(std::vector<std::pair<const char*, int> > &saved_subregex)
+{
+  std::vector<std::pair<const char*, int> >::iterator   it;
+  std::vector<CRegexXQuery_regex*>::iterator            subit;
+  for(it=saved_subregex.begin(), subit = subregex.begin(); it != saved_subregex.end(); it++, subit++)
+  {
+    (*subit)->matched_source = (*it).first;
+    (*subit)->matched_len = (*it).second;
+  }
+}
+
+void CRegexXQuery_branch::reset()
+{
+  std::list<RegexAscii_pieceinfo>::iterator  piece_it;
+  for(piece_it = piece_list.begin(); piece_it != piece_list.end(); piece_it++)
+  {
+    (*piece_it).piece->atom->reset_match();
+  }
+}
+/*
+void CRegexXQuery_branch::restore()
+{
+  std::list<RegexAscii_pieceinfo>::iterator  piece_it;
+  for(piece_it = piece_list.begin(); piece_it != piece_list.end(); piece_it++)
+  {
+    (*piece_it).piece->atom->restore_match();
+  }
+}
+*/
 //match all the pieces
-bool CRegexAscii_branch::match(const char *source, int *matched_len)
+bool CRegexXQuery_branch::match(const char *source, int *matched_len,
+                              CRegexXQuery_regex* group_regex,
+                              std::list<RegexAscii_pieceinfo>::iterator next_piece,
+                              std::list<RegexAscii_pieceinfo>::iterator end_piece)
 {
-  std::list<CRegexAscii_piece*>::iterator  piece_it;
+  std::list<RegexAscii_pieceinfo>::iterator  piece_it;
 
   piece_it = piece_list.begin(); 
+  //if(piece_it == piece_list.end())
+    //if(!source[0])
+  //    return true;
+    //else
+    //  return false;
   if(piece_it == piece_list.end())
-    if(source[0])
-      return false;
+  {
+    piece_it = next_piece;
+    if(next_piece == end_piece)
+    {
+      group_regex->matched_len = 0;
+      return true;
+    }
+  }
+
+  std::list<RegexAscii_pieceinfo>   temp_pieces(piece_list);
+  temp_pieces.push_back(group_regex);//this will be used to store the group match
+  temp_pieces.insert(temp_pieces.end(), next_piece, end_piece);
+
+  return (*piece_it).piece->match_piece(temp_pieces.begin(), temp_pieces.end(), source, matched_len);
+}
+
+bool CRegexXQuery_piece::match_piece(std::list<RegexAscii_pieceinfo>::iterator piece_it,
+                                    std::list<RegexAscii_pieceinfo>::iterator end_it,
+                                    const char *source, int *matched_len)
+{
+  if((*piece_it).nr_matches < 0)
+  {
+    //special case, store the group match
+    (*piece_it).group_regex->matched_len = source - (*piece_it).group_regex->matched_source;
+    piece_it++;
+    if(piece_it == end_it)
+      return true;
     else
-      return true;
-  if(!(*piece_it)->get_is_reluctant())
-    return match_piece_iter_normal(piece_it, source, matched_len);
+      return (*piece_it).piece->match_piece(piece_it, end_it, source, matched_len);
+  }
+
+  if(!get_is_reluctant())
+    return match_piece_iter_normal(piece_it, end_it, source, matched_len);
   else
-    return match_piece_iter_reluctant(piece_it, source, matched_len);
-}
-
-//match as less as possible
-bool CRegexAscii_branch::match_piece_iter_reluctant(
-                                        std::list<CRegexAscii_piece*>::iterator piece_it,
+    return match_piece_iter_reluctant(piece_it, end_it, source, matched_len);
+}
+
+int CRegexXQuery_piece::choose_another_branch(std::vector<std::pair<int,int> > &match_lens)
+{
+  int i = match_lens.size()-1;
+  i--;
+  while((i >= 0) && (match_lens.at(i).second == 0))
+    i--;
+  if(i < 0)
+    return -1;//no more branches
+  match_lens.resize(i+1);
+  i++;
+  return i;
+}
+
+bool CRegexXQuery_piece::is_regex_atom()
+{
+  return regex_atom != NULL;
+}
+
+//match as less as possible (shortest string)
+bool CRegexXQuery_piece::match_piece_iter_reluctant(
+                                        std::list<RegexAscii_pieceinfo>::iterator piece_it,
+                                        std::list<RegexAscii_pieceinfo>::iterator end_it,
                                         const char *source, int *matched_len)
 {
   *matched_len = 0;
-  if(piece_it == piece_list.end())
+  if(piece_it == end_it)
     return true;
 
   int min, max;
   bool  strict_max;
   //std::vector<int>    match_lens;
-  (*piece_it)->get_quantifier(&min, &max, &strict_max);
-  if(strict_max && (max >= 0))
+  (*piece_it).piece->get_quantifier(&min, &max, &strict_max);
+
+  std::vector<std::pair<const char*, int> >  saved_subregex;
+
+  if(is_regex_atom())
   {
-    int   timeslen;
-    //check if the piece doesn't exceed the max match
-    if((*piece_it)->match_piece_times(source, &timeslen, max+1, NULL))
-      return false;///too many matches
+    //recursive
+    bool retmatch;
+    atom->regex_intern->save_subregex_list(saved_subregex);
+    if((*piece_it).nr_matches >= min)
+    {
+      //go to next piece
+      std::list<RegexAscii_pieceinfo>::iterator next_it = piece_it;
+      next_it++;
+      if(next_it == end_it)
+        return true;
+      retmatch = (*next_it).piece->match_piece(next_it, end_it, source, matched_len);
+      if(retmatch)
+        return true;
+    }
+    if(((max == -1) || ((*piece_it).nr_matches < max)) &&//try further with this piece
+      (((*piece_it).nr_matches < min) || ((*piece_it).nr_matches == 0) || ((*piece_it).piece->regex_atom->matched_len)))//if matched_len is zero, avoid infinite loop
+    {
+      int start_from_branch = 0;
+      int shortest_len = -1;
+      bool branch_saved = false;
+      //try all branches to get the shortest len
+      (*piece_it).nr_matches++;
+      while(atom->match(source, &start_from_branch, matched_len, piece_it, end_it))
+      {
+        if((shortest_len == -1) || (shortest_len > *matched_len))
+        {
+          shortest_len = *matched_len;
+          if(start_from_branch && (atom->regex_intern->flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+          {
+            atom->regex_intern->save_subregex_list(saved_subregex);
+            branch_saved = true;
+          }
+        }
+        if(!start_from_branch || !(atom->regex_intern->flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+          break;
+      }
+      if(shortest_len != -1)
+      {
+        *matched_len = shortest_len;
+        if(branch_saved)
+          atom->regex_intern->load_subregex_list(saved_subregex);
+        return true;
+      }
+      else
+      {
+        (*piece_it).nr_matches--;
+        atom->regex_intern->load_subregex_list(saved_subregex);
+        return false;
+      }
+    }
+    else
+    {
+      atom->regex_intern->load_subregex_list(saved_subregex);
+      return false;
+    }
   }
 
-  int i=min;
-  std::list<CRegexAscii_piece*>::iterator next_it = piece_it;
+  int i=0;
+  int shortest_len = -1;
+  int otherpieces_shortest = -1;
+  int i_shortest = -1;
+  std::list<RegexAscii_pieceinfo>::iterator next_it = piece_it;
+  std::vector<std::pair<int,int> >    match_lens;
   next_it++;
   int pieceslen = 0;
   while(1)
   {
-    if((max > 0) && (i>max))
-      break;
-   int piecelen = 0;
-   if((*piece_it)->match_piece_times(source+pieceslen, &piecelen, !pieceslen ? i : 1, NULL))
-   {
-      pieceslen += piecelen;
+    int piecelen = 0;
+    bool retmatch;
+    retmatch = match_piece_times(source, &piecelen, i < min ? min : i, &match_lens);
+    i = match_lens.size()-1;//number of matches
+    if(i<0)
+      i = 0;
+    if((i>=min))
+    {
+      pieceslen = piecelen;
+      if((shortest_len >= 0) && (shortest_len <= pieceslen))//this branch is longer
+      {//try another branch
+        i = choose_another_branch(match_lens);
+        if(i >= 0)
+          continue;//try another branch
+        else
+          break;
+      }
       int   otherpieces = 0;
-      if((next_it == piece_list.end()) ||
-        ((*next_it)->get_is_reluctant() && match_piece_iter_reluctant(next_it, source+pieceslen, &otherpieces)) ||
-        (!(*next_it)->get_is_reluctant() && match_piece_iter_normal(next_it, source+pieceslen, &otherpieces)))
-      {
-        *matched_len = pieceslen + otherpieces;
-        return true;
-      }
+      if((next_it == end_it) ||
+        (*next_it).piece->match_piece(next_it, end_it, source+pieceslen, &otherpieces)
+        )
+      {
+        if((i == pieceslen) || (match_lens.at(0).second == 0) ||//minimum achieved already, cannot go lower than that
+            !(atom->regex_intern->flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+        {
+          *matched_len = pieceslen + otherpieces;
+          return true;
+        }
+        if((shortest_len < 0) || (shortest_len > pieceslen))
+        {
+          shortest_len = pieceslen;
+          otherpieces_shortest = otherpieces;
+          i_shortest = i;
+          if(match_lens.at(0).second != 0)
+            atom->regex_intern->save_subregex_list(saved_subregex);
+        }
+        i = choose_another_branch(match_lens);
+        if(i >= 0)
+          continue;//try another branch
+        else
+          break;
+      }
+      else
+      {
+        //try further
+        if(retmatch)
+        {
+          i++;
+          if((max < 0) || (i<=max))
+            continue;
+          i--;
+        }
+      }
+    }
+    
+    if(i==0)
+    {
+      break;
     }
     else
-      break;
-    i++;
+    {
+      i = choose_another_branch(match_lens);
+      if(i >= 0)
+        continue;//try another branch
+      else
+        break;
+    }
   }
 
+  if(shortest_len >= 0)
+  {
+    if(strict_max && (max>=0) && (i_shortest > max))
+      return false;
+    *matched_len = shortest_len + otherpieces_shortest;
+    if(saved_subregex.size())
+      atom->regex_intern->load_subregex_list(saved_subregex);
+    return true;
+  }
   return false;
 }
 
 //match as much as possible
-bool CRegexAscii_branch::match_piece_iter_normal(
-                                        std::list<CRegexAscii_piece*>::iterator piece_it,
+bool CRegexXQuery_piece::match_piece_iter_normal(
+                                        std::list<RegexAscii_pieceinfo>::iterator piece_it,
+                                        std::list<RegexAscii_pieceinfo>::iterator end_it,
                                         const char *source, int *matched_len)
 {
   *matched_len = 0;
 
   int min, max;
   bool  strict_max;
-  std::vector<int>    match_lens;
-  (*piece_it)->get_quantifier(&min, &max, &strict_max);
-  int   timeslen;
-  if(strict_max && (max >= 0))
+  std::vector<std::pair<int,int> >    match_lens;
+  (*piece_it).piece->get_quantifier(&min, &max, &strict_max);
+  int   timeslen = 0;
+  std::vector<std::pair<const char*, int> >  saved_subregex;
+
+  if(is_regex_atom())
   {
-    //check if the piece doesn't exceed the max match
-    //if((*piece_it)->match_piece_times(source, &timeslen, max+1, &match_lens))
-    //  return false;///too many matches
-    (*piece_it)->match_piece_times(source, &timeslen, max, &match_lens);
+    //recursive
+    bool retmatch;
+    atom->regex_intern->save_subregex_list(saved_subregex);
+    if(((max == -1) || ((*piece_it).nr_matches < max)) && //try further with this piece
+      (((*piece_it).nr_matches < min) || ((*piece_it).nr_matches == 0) || ((*piece_it).piece->regex_atom->matched_len)))//if matched_len is zero, avoid infinite loop
+    {
+      int start_from_branch = 0;
+      int longest_len = -1;
+      bool branch_saved = false;
+      //try all branches to get the longest len
+      (*piece_it).nr_matches++;
+      while(atom->match(source, &start_from_branch, matched_len, piece_it, end_it))
+      {
+        if((longest_len < *matched_len))
+        {
+          longest_len = *matched_len;
+          if(start_from_branch && (atom->regex_intern->flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+          {
+            atom->regex_intern->save_subregex_list(saved_subregex);
+            branch_saved = true;
+          }
+        }
+        if(!start_from_branch || !(atom->regex_intern->flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+          break;
+      }
+      if(longest_len != -1)
+      {
+        *matched_len = longest_len;
+        if(branch_saved)
+          atom->regex_intern->load_subregex_list(saved_subregex);
+        return true;
+      }
+      else
+      {
+        atom->regex_intern->load_subregex_list(saved_subregex);
+        (*piece_it).nr_matches--;
+      }
+    }
+    if((*piece_it).nr_matches >= min)
+    {
+      //go to next piece
+      std::list<RegexAscii_pieceinfo>::iterator next_it = piece_it;
+      next_it++;
+      if(next_it == end_it)
+        return true;
+      retmatch = (*next_it).piece->match_piece(next_it, end_it, source, matched_len);
+      if(!retmatch)
+        atom->regex_intern->load_subregex_list(saved_subregex);
+      return retmatch;
+    }
+    else
+    {
+    //  regex_atom->restore_match();
+      atom->regex_intern->load_subregex_list(saved_subregex);
+      return false;
+    }
   }
-  else if(!strict_max && (max >= 0))
-    (*piece_it)->match_piece_times(source, &timeslen, max, &match_lens);
-  else
-    (*piece_it)->match_piece_times(source, &timeslen, -1, &match_lens);
 
-  int i;
-  std::list<CRegexAscii_piece*>::iterator next_it = piece_it;
+  int longest_len = -1;
+  int otherpieces_longest = -1;
+  int i_longest = -1;
+  int i = max;
+  std::list<RegexAscii_pieceinfo>::iterator next_it = piece_it;
   next_it++;
-  if(next_it == piece_list.end())
+
+  bool retmatch;
+  while(1)
   {
-    if((int)match_lens.size() > min)
-    {
-      *matched_len = timeslen;
-      return true;
+    retmatch = match_piece_times(source, &timeslen, i, &match_lens);
+    i=match_lens.size()-1;//number of matches
+    if((i>=min))
+    {
+      if(timeslen < longest_len)
+      {//this branch is no use
+        i = choose_another_branch(match_lens);
+        if(i >= 0)
+        {
+          i = max;
+          continue;//try another branch
+        }
+        else
+          break;
+      }
+      //int piecelen = 0;
+      int   otherpieces = 0;
+      if((next_it == end_it) ||
+        (*next_it).piece->match_piece(next_it, end_it, source+timeslen, &otherpieces)
+        )
+      {
+        if(timeslen > longest_len)
+        {
+          longest_len = timeslen;
+          otherpieces_longest = otherpieces;
+          i_longest = i;
+          if(!(atom->regex_intern->flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+          {
+            *matched_len = longest_len + otherpieces_longest;
+            return true;
+          }
+          else
+          {
+            if(match_lens.at(0).second)
+              atom->regex_intern->save_subregex_list(saved_subregex);
+          }
+        }
+      }
+      else
+      {
+        if(!match_lens.at(0).second)
+        {
+          match_lens.resize(match_lens.size()-1);
+          i--;
+          if(i >= 0)
+            continue;//try smaller 
+          else
+            break;
+        }
+        else
+        {
+          i = choose_another_branch(match_lens);
+          if(i >= 0)
+            continue;//try another branch
+          else
+            break;
+        }
+      }
+    }
+    //now try another branch
+    i = choose_another_branch(match_lens);
+    if(i >= 0)
+    {
+      i = max;
+      continue;//try another branch
     }
     else
-      return false;
-  }
-  for(i=match_lens.size()-1; i>=min; i--)
+      break;
+  }//end while
+
+  if(longest_len >= 0)
   {
-    int piecelen = 0;
-    int   otherpieces = 0;
-    if(((*next_it)->get_is_reluctant() && match_piece_iter_reluctant(next_it, source+match_lens[i]+piecelen, &otherpieces)) ||
-      (!(*next_it)->get_is_reluctant() && match_piece_iter_normal(next_it, source+match_lens[i]+piecelen, &otherpieces)))
-    {
-      *matched_len = match_lens[i] + piecelen + otherpieces;
-      return true;
-    }
+    *matched_len = longest_len + otherpieces_longest;
+    if(saved_subregex.size())
+      atom->regex_intern->load_subregex_list(saved_subregex);
+    return true;
   }
 
   return false;
 }
 
-bool CRegexAscii_piece::match_piece_times(const char *source, 
+bool CRegexXQuery_piece::match_piece_times(const char *source, 
                                           int *piecelen, 
                                           int times,
-                                          std::vector<int>    *match_lens)
+                                          std::vector<std::pair<int,int> >    *match_lens)
 {
-  *piecelen = 0;
-  for(int i=0;(times < 0) || (i<times);i++)
-  {
+  int i=0;
+  if(match_lens && match_lens->size())
+  {
+    i = match_lens->size()-1;
+  }
+  if(match_lens && match_lens->size())
+    *piecelen = match_lens->at(match_lens->size()-1).first;
+  else
+    *piecelen = 0;
+  if((times >= 0) && (i>=times))
+    return true;
+  for(;(times < 0) || (i<times);i++)
+  {
+    int   atomlen;
+    int   start_from_branch = 0;
+    if(match_lens && (i<(int)match_lens->size()))
+      start_from_branch = match_lens->at(i).second;
+    bool first_branch = (start_from_branch == 0);
+    if(!atom->match(source+*piecelen, &start_from_branch, &atomlen, empty_pieces.begin(), empty_pieces.end()))
+    {
+      if(match_lens)
+      {
+        if(i >= (int)match_lens->size())
+          match_lens->push_back(std::pair<int,int>(*piecelen, 0));
+        else
+          (*match_lens)[i] = std::pair<int,int>(*piecelen, 0);
+      }
+      return false;
+    }
     if(match_lens)
-      match_lens->push_back(*piecelen);
-    int   atomlen;
-    if(!atom->match(source+*piecelen, &atomlen))
-      return false;
+    {
+      if(i >= (int)match_lens->size())
+        match_lens->push_back(std::pair<int,int>(*piecelen, start_from_branch));
+      else
+        (*match_lens)[i] = std::pair<int,int>(*piecelen, start_from_branch);
+    }
     *piecelen += atomlen;
     if(!atomlen && !source[*piecelen])
     {
-      atom->regex_intern->reachedEnd = true;
+    //  atom->regex_intern->set_reachedEnd(source);
+      break;
+    }
+    if(first_branch && (atomlen == 0))//avoid infinite loop
+    {
       break;
     }
   }
   if(match_lens)
-    match_lens->push_back(*piecelen);
+  {
+  //  if(i >= match_lens->size())
+      match_lens->push_back(std::pair<int,int>(*piecelen, 0));
+  //  else
+  //    (*match_lens)[i] = std::pair<int,int>(*piecelen, 0);
+  }
 
   return true;
 }
 
+bool CRegexXQuery_multicharP::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  bool found = false;
+  const char *temp_source = source;
+  unicode::code_point utf8c = utf8::next_char(temp_source);
+  switch(multichar_type)
+  {
+  case unicode::UNICODE_Ll + 50:
+    if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Ll) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Lm) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Lo) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Lt) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Lu))
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  case unicode::UNICODE_Mc + 50:
+    if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Mn) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Mc) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Me))
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  case unicode::UNICODE_Nd + 50:
+    if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Nd) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Nl) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_No))
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  case unicode::UNICODE_Pc + 50:
+    if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pc) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pd) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Ps) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pe) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pi) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pf) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Po))
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  case unicode::UNICODE_Zl + 50:
+    if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Zs) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Zl) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Zp))
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  case unicode::UNICODE_Sc + 50:
+    if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Sm) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Sc) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Sk) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_So))
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  case unicode::UNICODE_Cc + 50:
+    if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Cc) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Cf) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Co))//ignore unicode::UNICODE_Cn
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  default:
+    if(unicode::check_codepoint_category(utf8c, (unicode::category)multichar_type))
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  }
+
+  if(found)
+  {
+    *matched_len = temp_source - source;
+  }
+  return found;
+}
+
+bool CRegexXQuery_multicharIs::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  bool found = false;
+  const char *temp_source = source;
+  unicode::code_point utf8c = utf8::next_char(temp_source);
+  const unicode::code_point *cp = block_escape[block_index].cp;
+  if((utf8c >= cp[0]) && (utf8c <= cp[1]))
+  {
+    if(!is_reverse)
+      found = true;
+  }
+  else if(block_escape[block_index].ext_cp)
+  {
+    cp = block_escape[block_index].ext_cp;
+    while(*cp)
+    {
+      if((utf8c >= cp[0]) && (utf8c <= cp[1]))
+        break;
+      cp += 2;
+    }
+    if(*cp)
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+  }
+  else
+  {
+    if(is_reverse)
+      found = true;
+  }
+  if(found)
+  {
+    *matched_len = temp_source - source;
+  }
+  return found;
+}
+
+bool CRegexXQuery_multicharOther::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  bool found = false;
+  bool value_true = true;
+  const char *temp_source = source;
+  unicode::code_point utf8c = utf8::next_char(temp_source);
+  switch(multichar_type)
+  {
+    case 'S':value_true = false;//[^\s]
+    case 's'://[#x20\t\n\r]
+      switch(utf8c)
+      {
+      case '\t':
+      case '\r':
+      case '\n':
+      case ' ':
+        found = true;
+      default:
+        break;
+      }
+      break;
+    case 'I':value_true = false;//[^\i]
+    case 'i'://the set of initial name characters, those matched by Letter | '_' | ':'
+      if((utf8c == '_') ||
+        (utf8c == ':') ||
+        XQCharType::isLetter(utf8c))
+      {
+        found = true;
+      }
+      break;
+    case 'C':value_true = false;//[^\c]
+    case 'c'://the set of name characters, those matched by NameChar
+      if(XQCharType::isNameChar(utf8c))
+      {
+        found = true;
+      }
+      break;
+    case 'D':value_true = false;//[^\d]
+    case 'd':
+      if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Nd))
+        found = true;
+      break;
+    case 'W':value_true = false;//[^\w]
+    case 'w':
+      found = !(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pc) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pd) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Ps) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pe) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pi) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pf) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Po) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Zs) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Zl) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Zp) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Cc) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Cf) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Co));//ignore unicode::UNICODE_Cn
+      break;
+    default:
+      throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(source, ZED(REGEX_UNIMPLEMENTED)) );
+  }
+  if((found && value_true) || (!found && !value_true))
+  {
+    *matched_len = temp_source - source;
+    return true;
+  }
+  else
+  {
+    return false;
+  }
+}
+
+bool CRegexXQuery_char_ascii::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  if(source[0] == c)
+  {
+    *matched_len = 1;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_ascii_i::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  char  sup = toupper(source[0]);
+  if(sup == c)
+  {
+    *matched_len = 1;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_range_ascii::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  if((source[0] >= c1) && (source[0] <= c2))
+  {
+    *matched_len = 1;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_range_ascii_i::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  char  sup = toupper(source[0]);
+  if((sup >= c1) && (sup <= c2))
+  {
+    *matched_len = 1;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_unicode::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  if(!memcmp(source, c, len))
+  {
+    *matched_len = len;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_unicode_cp::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  const char *temp_source = source;
+  unicode::code_point  utf8c = utf8::next_char(temp_source);
+  if(utf8c == c)
+  {
+    *matched_len = temp_source - source;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_unicode_i::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  const char *temp_source = source;
+  unicode::code_point  sup = unicode::to_upper(utf8::next_char(temp_source));
+  if(sup == c)
+  {
+    *matched_len = temp_source - source;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_range_unicode::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  const char *temp_source = source;
+  unicode::code_point  utf8c = utf8::next_char(temp_source);
+  if((utf8c >= c1) && (utf8c <= c2))
+  {
+    *matched_len = temp_source - source;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_range_unicode_i::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  const char *temp_source = source;
+  unicode::code_point  sup = unicode::to_upper(utf8::next_char(temp_source));
+  if((sup >= c1) && (sup <= c2))
+  {
+    *matched_len = temp_source - source;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_endline::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  *matched_len = 0;
+  if(!source[0])
+  {
+  //  regex_intern->reachedEnd = true;
+    return true;
+  }
+  if((source[0] == 0x0A) || ((source[0] == 0x0D) && (source[1] == 0x0A)))
+  {
+    if(regex_intern->get_flags() & REGEX_ASCII_MULTILINE)
+    {
+    //  regex_intern->reachedEnd = true;
+      return true;
+    }
+  }
+  return false;