Merge lp:~zorba-coders/zorba/no_unicode into lp:zorba

[Daniel Turcanu]

"No Unicode" is now "No ICU."

"No Unicode" is now "No ICU."

[Matthias Brantner (matthias-brantner)]

Compiling with ZORBA_NO_ICU=ON fails on Linux:

[ 1%] Building CXX object src/CMakeFiles/zorba_simplestore.dir/api/zorba_string.cpp.o
In file included from /home/mbrantner/zorba/sandbox/src/util/regex.h:501:0,
                 from /home/mbrantner/zorba/sandbox/src/api/zorba_string.cpp:23:
/home/mbrantner/zorba/sandbox/src/util/regex_xquery.h:209:3: error: a class-key must be used when declaring a friend
/home/mbrantner/zorba/sandbox/src/util/regex_xquery.h:209:3: error: friend declaration does not name a class or function
/home/mbrantner/zorba/sandbox/src/util/regex_xquery.h:253:3: error: a class-key must be used when declaring a friend
/home/mbrantner/zorba/sandbox/src/util/regex_xquery.h:253:3: error: friend declaration does not name a class or function
make[2]: *** [src/CMakeFiles/zorba_simplestore.dir/api/zorba_string.cpp.o] Erro

[Matthias Brantner (matthias-brantner)]

review: Needs Fixing

[Zorba Build Bot (zorba-buildbot)]

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

[Matthias Brantner (matthias-brantner)]

The test suite doesn't run clean on my system (Linux) without ICU. This prevents us from adding the built to the remote queue. For example, the following tests fail without ICU (some of them also seem to fail with ICU):

 1294 - test/rbkt/zorba/string/Regex/regex_a10 (Failed)
 1548 - test/rbkt/zorba/fulltext/ft-wildcard-true-2 (Failed)
 1560 - test/rbkt/zorba/fulltext/ft-wildcard-true-4 (Failed)
 1574 - test/rbkt/zorba/fulltext/ft-same-sentence-true-4 (Failed)
 1581 - test/rbkt/zorba/fulltext/ft-wildcard-true-3 (Failed)
 1587 - test/rbkt/zorba/fulltext/ft-wildcard-true-9 (Failed)
 1600 - test/rbkt/zorba/fulltext/ft-diacritics-insensitive-true-1 (Failed)
 1605 - test/rbkt/zorba/fulltext/ft-wildcard-true-8 (Failed)
 1612 - test/rbkt/zorba/fulltext/ft-wildcard-true-10 (Failed)
 1635 - test/rbkt/zorba/fulltext/ft-wildcard-true-7 (Failed)
 1637 - test/rbkt/zorba/fulltext/ft-wildcard-true-11 (Failed)
 1643 - test/rbkt/zorba/fulltext/ft-wildcard-FTDY0020-3 (Failed)
 1789 - test/rbkt/zorba/index/numbers (Failed)
 2345 - test/unit/string_test (Failed)
 2534 - test/update/zorba/store/sc3 (Failed)
 2544 - doc/cxx/examples/context.cpp (Failed)

Please make sure the test suite runs clean.

review: Needs Fixing

[lp:~zorba-coders/zorba/no_unicode]

10512. By Paul J. Lucas on 2012-01-16

Merge from trunk.

10513. By Daniel Turcanu on 2012-01-18

Fix bug in FnAnalyzeStringIterator

10514. By Daniel Turcanu on 2012-01-18

Removed regex_err12 test from expected failures

10515. By Paul J. Lucas on 2012-01-18

Merge from trunk.

10516. By Paul J. Lucas on 2012-01-18

Merge from trunk.

[Paul J. Lucas (paul-lucas)]

Try it now.

[lp:~zorba-coders/zorba/no_unicode]

10517. By Paul J. Lucas on 2012-02-21

Merge from trunk.

10518. By Paul J. Lucas on 2012-02-28

Merge from trunk.

10519. By Paul J. Lucas on 2012-03-07

Merge from trunk.

10520. By Paul J. Lucas on 2012-03-13

Merge from trunk.

10521. By Rodolfo Ochoa on 2012-03-28

-Fixes for windows compiling

10522. By Rodolfo Ochoa on 2012-03-28

Merge from trunk

10523. By Rodolfo Ochoa on 2012-03-30

- Fixes for commiting on windows with ZORBA_NO_ICU
- Fixes for commiting on windows with ZORBA_NO_FULL_TEXT

10524. By Rodolfo Ochoa on 2012-03-30

-Fix for no precompiled headers usage

10525. By Rodolfo Ochoa on 2012-03-30

Merge from trunk

10526. By Rodolfo Ochoa on 2012-04-02

fixes for windows

10527. By Rodolfo Ochoa on 2012-04-02

Fix for linux

10528. By Rodolfo Ochoa on 2012-04-03

more fixes for linux

10529. By Rodolfo Ochoa on 2012-04-03

Final fix for windows

10530. By Paul J. Lucas on 2012-04-05

1. Added fix for not catching bad regexs like "^^".
2. Added if="!defined(ZORBA_NO_ICU)" for some entries in the diagnostics
   dictionary.

10531. By Paul J. Lucas on 2012-04-05

Merge from trunk.

10532. By Paul J. Lucas on 2012-04-06

Fix for '^' bug.

10533. By Paul J. Lucas on 2012-04-06

1. Fixed yet another '^' bug.
2. Marked some regex tests as expected failure with correct bug numbers.

10534. By Paul J. Lucas on 2012-04-07

No longer doing some stuff when q_flag is set.

10535. By Paul J. Lucas on 2012-04-07

Tweaked one error message.

10536. By Paul J. Lucas on 2012-04-07

Merge from trunk.

10537. By Rodolfo Ochoa on 2012-04-09

Merge from trunk

10538. By Rodolfo Ochoa on 2012-04-11

Strange error on include guards

10539. By Rodolfo Ochoa on 2012-04-12

merge from trunk

10540. By Rodolfo Ochoa on 2012-04-13

fix for regex errors in RQ

Unmerged revisions

=== modified file 'CMakeConfiguration.txt'
--- CMakeConfiguration.txt	2012-01-11 17:30:25 +0000
+++ CMakeConfiguration.txt	2012-01-18 18:33:36 +0000
@@ -135,14 +135,14 @@
 SET (ZORBA_DEBUG_STRING ${ZORBA_DEBUG_STRING} CACHE BOOL "debug strings")
 MESSAGE (STATUS "ZORBA_DEBUG_STRING:                   " ${ZORBA_DEBUG_STRING})
 
-SET(ZORBA_NO_UNICODE OFF CACHE BOOL "disable ICU")
-MESSAGE(STATUS "ZORBA_NO_UNICODE:                     " ${ZORBA_NO_UNICODE})
+SET(ZORBA_NO_ICU OFF CACHE BOOL "disable ICU")
+MESSAGE(STATUS "ZORBA_NO_ICU:                         " ${ZORBA_NO_ICU})
 
-IF (ZORBA_NO_UNICODE)
+IF (ZORBA_NO_ICU)
   SET (no_full_text ON)
-ELSE (ZORBA_NO_UNICODE)
+ELSE (ZORBA_NO_ICU)
   SET (no_full_text OFF)
-ENDIF (ZORBA_NO_UNICODE)
+ENDIF (ZORBA_NO_ICU)
 SET (ZORBA_NO_FULL_TEXT ${no_full_text} CACHE BOOL "disable XQuery Full-Text support")
 MESSAGE(STATUS "ZORBA_NO_FULL_TEXT:                   " ${ZORBA_NO_FULL_TEXT})
 

=== modified file 'CMakeLists.txt'
--- CMakeLists.txt	2012-01-04 09:47:54 +0000
+++ CMakeLists.txt	2012-01-18 18:33:36 +0000
@@ -123,10 +123,14 @@
 CHECK_TYPE_SIZE("int64_t" ZORBA_HAVE_INT64_T) 
 
 CHECK_CXX_SOURCE_COMPILES ("#include <type_traits>\nint main() { std::enable_if<true,int> x; }" ZORBA_CXX_ENABLE_IF)
-CHECK_CXX_SOURCE_COMPILES ("int main() { int *p = nullptr; }" ZORBA_CXX_NULLPTR)
-CHECK_CXX_SOURCE_COMPILES ("int main() { static_assert(1,\"\"); }" ZORBA_CXX_STATIC_ASSERT)
+SET(CMAKE_EXTRA_INCLUDE_FILES wchar.h)
+CHECK_TYPE_SIZE("wchar_t" ZORBA_SIZEOF_WCHAR_T)
+SET(CMAKE_EXTRA_INCLUDE_FILES)
 CHECK_CXX_SOURCE_COMPILES ("#include <memory>\nint main() { std::unique_ptr<int> p; }" ZORBA_CXX_UNIQUE_PTR)
 
+CHECK_CXX_SOURCE_COMPILES("int main() { int *p = nullptr; }" ZORBA_CXX_NULLPTR)
+CHECK_CXX_SOURCE_COMPILES("int main() { static_assert(1,\"\"); }" ZORBA_CXX_STATIC_ASSERT)
+
 ################################################################################
 # Various cmake macros
 

=== modified file 'ChangeLog'
--- ChangeLog	2012-01-18 12:18:59 +0000
+++ ChangeLog	2012-01-18 18:33:36 +0000
@@ -1,5 +1,9 @@
 Zorba - The XQuery Processor
 
+version 2.x
+
+  * Added support for NO_ICU (to not use ICU for unicode processing)
+  
 version 2.2
 
   * No-copy optimization: avoids copying nodes during node-constructor expressions.
@@ -78,7 +82,9 @@
   * Fixed bug when parsing a document with a base-uri attribute.
   * Fixed bug #863320 (Sentence is incorrectly incremented when token characters end without sentence terminator)
   * Fixed bug #863730 (static delete-node* functions don't raise ZDDY0012)
+  * Implemented the probe-index-range-value for general indexes
   * Removed ZSTR0005 and ZSTR0006 error codes
+  * Fixed bug #867662 ("nullptr" warning)
   * Fixed bug #868258 (Assertion failure with two delete collection)
   * Fixed bug #871623 and #871629 (assertion failures with insertions in dynamic collections)
   * Fixed bug #867262 (allow reuse of iterator over ExtFuncArgItemSequence)
@@ -87,6 +93,8 @@
   * New node-reference module. References can be obtained for any node, and
 	different nodes cannot have the same identifier.
   * Fixed bug #872697  (segmentation fault with validation of NMTOKENS)
+  * General index cannot be declared as unique if the type of its key is
+    xs:anyAtomicType or xs:untypedAtomic.
   * Added undo for node revalidation
   * Optimization for count(collection()) expressions
   * Fixed bug #872796  (validate-in-place can interfere with other update primitives)
@@ -105,6 +113,8 @@
   * Fixed bug #855715 (Invalid escaped characters in regex not caught)
   * Fixed bug #862089 (Split binary/xq install directories for modules) by
   splitting "module path" into separate URI and Library paths
+  * New node-position module. This module allows to obtain a representation of a node position, which
+    can be used to assess structural relationships with other nodes.   
   * Fixed bug #872502 (validation of the JSON module xqdoc fails)
   * Fixed bug #897619 (testdriver_mt can not run the XQueryX tests)
   * Fixed bug #867107 (xqdoc dependency to zorba is wrong)

=== modified file 'KNOWN_ISSUES.txt'
--- KNOWN_ISSUES.txt	2011-10-07 08:28:43 +0000
+++ KNOWN_ISSUES.txt	2012-01-18 18:33:36 +0000
@@ -37,7 +37,7 @@
 * The serializer currently doesn't implement character maps as specified
   (http://www.w3.org/TR/xslt-xquery-serialization/#character-maps)
 
-* In the 2.0 release, setting the CMake variables ZORBA_NO_UNICODE to
+* In the 2.0 release, setting the CMake variables ZORBA_NO_ICU to
   ON is not supported.
 
 * The PHP language binding is not supported on Mac OS X. For details,

=== modified file 'doc/cxx/examples/context.cpp'
--- doc/cxx/examples/context.cpp	2011-07-22 08:12:31 +0000
+++ doc/cxx/examples/context.cpp	2012-01-18 18:33:36 +0000
@@ -149,7 +149,11 @@
     outStream2 << lQuery << std::endl;
     std::cout << outStream2.str() << std::endl;
 
+#ifndef ZORBA_NO_ICU
     if (outStream2.str() != "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\nBook 1.1\n")
+#else
+    if (outStream2.str() != "<?xml version=\"1.0\"?>\nBook 1.1\n")
+#endif /* ZORBA_NO_ICU */
     {
       std::cerr << "Test 4 failed with a wrong result : " << std::endl
                 << outStream2.str() << std::endl;

=== modified file 'include/zorba/config.h.cmake'
--- include/zorba/config.h.cmake	2012-01-11 17:30:25 +0000
+++ include/zorba/config.h.cmake	2012-01-18 18:33:36 +0000
@@ -93,6 +93,8 @@
 typedef __int64 int64_t;
 #endif /* ZORBA_HAVE_INT64_T */
 
+#cmakedefine ZORBA_SIZEOF_WCHAR_T @ZORBA_SIZEOF_WCHAR_T@
+
 // Compiler
 #cmakedefine CLANG
 #cmakedefine MSVC
@@ -145,7 +147,7 @@
 
 // Zorba features
 #cmakedefine ZORBA_NO_FULL_TEXT
-#cmakedefine ZORBA_NO_UNICODE
+#cmakedefine ZORBA_NO_ICU
 #cmakedefine ZORBA_NO_XMLSCHEMA
 #cmakedefine ZORBA_NUMERIC_OPTIMIZATION
 #cmakedefine ZORBA_VERIFY_PEER_SSL_CERTIFICATE

=== modified file 'include/zorba/util/time.h'
--- include/zorba/util/time.h	2011-06-16 16:40:44 +0000
+++ include/zorba/util/time.h	2012-01-18 18:33:36 +0000
@@ -178,7 +178,7 @@
 	
     inline long get_walltime_in_millis(const walltime& t)
     {
-      return t.time * 1000 + t.millitm;
+      return (long)(t.time * 1000 + t.millitm);
     }
 
 #else /* not Windows, and no clock_gettime() */

=== modified file 'src/api/serialization/serializer.cpp'
--- src/api/serialization/serializer.cpp	2012-01-11 17:30:25 +0000
+++ src/api/serialization/serializer.cpp	2012-01-18 18:33:36 +0000
@@ -180,7 +180,6 @@
   for (; chars < chars_end; chars++ )
   {
 
-#ifndef ZORBA_NO_UNICODE
     // the input string is UTF-8
     int char_length = utf8::char_length(*chars);
     if (char_length == 0)
@@ -217,7 +216,6 @@
 
       continue;
     }
-#endif//ZORBA_NO_UNICODE
 
     // raise an error iff (1) the serialization format is XML 1.0 and (2) the given character is an invalid XML 1.0 character
     if (ser && ser->method == PARAMETER_VALUE_XML &&
@@ -332,14 +330,12 @@
     {
       tr << (char)0xEF << (char)0xBB << (char)0xBF;
     }
-#ifndef ZORBA_NO_UNICODE
     else if (ser->encoding == PARAMETER_VALUE_UTF_16)
     {
       // Little-endian
       tr.verbatim((char)0xFF);
       tr.verbatim((char)0xFE);
     }
-#endif
   }
 }
 
@@ -834,13 +830,17 @@
   emitter::emit_declaration();
 
   if (ser->omit_xml_declaration == PARAMETER_VALUE_NO) {
-    tr << "<?xml version=\"" << ser->version << "\" encoding=\"";
-    if (ser->encoding == PARAMETER_VALUE_UTF_8) {
-      tr << "UTF-8";
-#ifndef ZORBA_NO_UNICODE
-    } else if (ser->encoding == PARAMETER_VALUE_UTF_16) {
-      tr << "UTF-16";
-#endif
+    tr << "<?xml version=\"" << ser->version;
+    switch (ser->encoding) {
+      case PARAMETER_VALUE_UTF_8:
+      case PARAMETER_VALUE_UTF_16:
+        tr << "\" encoding=\"";
+        switch (ser->encoding) {
+          case PARAMETER_VALUE_UTF_8 : tr << "UTF-8" ; break;
+          case PARAMETER_VALUE_UTF_16: tr << "UTF-16"; break;
+          default                    : ZORBA_ASSERT(false);
+        }
+        break;
     }
     tr << "\"";
 
@@ -1146,14 +1146,18 @@
       }
 
       tr << "<meta http-equiv=\"content-type\" content=\""
-         << ser->media_type << "; charset=";
-
-      if (ser->encoding == PARAMETER_VALUE_UTF_8)
-        tr << "UTF-8";
-#ifndef ZORBA_NO_UNICODE
-      else if (ser->encoding == PARAMETER_VALUE_UTF_16)
-        tr << "UTF-16";
-#endif
+         << ser->media_type;
+      switch (ser->encoding) {
+        case PARAMETER_VALUE_UTF_8:
+        case PARAMETER_VALUE_UTF_16:
+          tr << "\" charset=\"";
+          switch (ser->encoding) {
+            case PARAMETER_VALUE_UTF_8 : tr << "UTF-8" ; break;
+            case PARAMETER_VALUE_UTF_16: tr << "UTF-16"; break;
+            default                    : ZORBA_ASSERT(false);
+          }
+          break;
+      }
       tr << "\"";
       // closed_parent_tag = 1;
     }
@@ -1343,14 +1347,18 @@
         }
 
         tr << "<meta http-equiv=\"content-type\" content=\""
-           << ser->media_type << "; charset=";
-
-        if (ser->encoding == PARAMETER_VALUE_UTF_8)
-          tr << "UTF-8";
-#ifndef ZORBA_NO_UNICODE
-        else if (ser->encoding == PARAMETER_VALUE_UTF_16)
-          tr << "UTF-16";
-#endif
+           << ser->media_type;
+        switch (ser->encoding) {
+          case PARAMETER_VALUE_UTF_8:
+          case PARAMETER_VALUE_UTF_16:
+            tr << "\" charset=\"";
+            switch (ser->encoding) {
+              case PARAMETER_VALUE_UTF_8 : tr << "UTF-8" ; break;
+              case PARAMETER_VALUE_UTF_16: tr << "UTF-16"; break;
+              default                    : ZORBA_ASSERT(false);
+            }
+            break;
+        }
         tr << "\"/";
         //closed_parent_tag = 1;
       }
@@ -2052,10 +2060,8 @@
   {
     if (!strcmp(aValue, "UTF-8"))
       encoding = PARAMETER_VALUE_UTF_8;
-#ifndef ZORBA_NO_UNICODE
     else if (!strcmp(aValue, "UTF-16"))
       encoding = PARAMETER_VALUE_UTF_16;
-#endif
     else
       throw XQUERY_EXCEPTION(
         err::SEPM0016, ERROR_PARAMS( aValue, aName, ZED( GoodValuesAreUTF8 ) )
@@ -2164,16 +2170,13 @@
   {
     tr = new transcoder(os, false);
   }
-#ifndef ZORBA_NO_UNICODE
   else if (encoding == PARAMETER_VALUE_UTF_16)
   {
     tr = new transcoder(os, true);
   }
-#endif
   else
   {
-    ZORBA_ASSERT(0);
-    return false;
+    ZORBA_ASSERT(false);
   }
 
   if (method == PARAMETER_VALUE_XML)

=== modified file 'src/api/serialization/serializer.h'
--- src/api/serialization/serializer.h	2011-11-11 07:44:01 +0000
+++ src/api/serialization/serializer.h	2012-01-18 18:33:36 +0000
@@ -70,10 +70,8 @@
     PARAMETER_VALUE_TEXT,
     PARAMETER_VALUE_BINARY,
 
-    PARAMETER_VALUE_UTF_8
-#ifndef ZORBA_NO_UNICODE
-    ,PARAMETER_VALUE_UTF_16
-#endif
+    PARAMETER_VALUE_UTF_8,
+    PARAMETER_VALUE_UTF_16
   } PARAMETER_VALUE_TYPE;
 
 protected:

=== modified file 'src/diagnostics/diagnostic_en.xml'
--- src/diagnostics/diagnostic_en.xml	2011-12-21 14:40:33 +0000
+++ src/diagnostics/diagnostic_en.xml	2012-01-18 18:33:36 +0000
@@ -3080,85 +3080,171 @@
       <value>item type is not a subtype of "$3"</value>
     </entry>
 
-    <entry key="U_REGEX_BAD_ESCAPE_SEQUENCE" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_BAD_ESCAPE_SEQUENCE" if="!defined(ZORBA_NO_ICU)">
       <value>unrecognized backslash escape sequence</value>
     </entry>
 
-    <entry key="U_REGEX_BAD_INTERVAL" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_BAD_INTERVAL" if="!defined(ZORBA_NO_ICU)">
       <value>error in {min,max} interval</value>
     </entry>
 
-    <entry key="U_REGEX_INTERNAL_ERROR" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_INTERNAL_ERROR" if="!defined(ZORBA_NO_ICU)">
       <value>an internal ICU error (bug) was detected</value>
     </entry>
 
-    <entry key="U_REGEX_INVALID_BACK_REF" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_INVALID_BACK_REF" if="!defined(ZORBA_NO_ICU)">
       <value>backreference to a non-existent capture group</value>
     </entry>
 
-    <entry key="U_REGEX_INVALID_FLAG" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_INVALID_FLAG" if="!defined(ZORBA_NO_ICU)">
       <value>invalid value for match mode flags</value>
     </entry>
 
-    <entry key="U_REGEX_INVALID_RANGE" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_INVALID_RANGE" if="!defined(ZORBA_NO_ICU)">
       <value>in character range [x-y], x is greater than y</value>
     </entry>
 
-    <entry key="U_REGEX_INVALID_STATE" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_INVALID_STATE" if="!defined(ZORBA_NO_ICU)">
       <value>RegexMatcher in invalid state for requested operation</value>
     </entry>
 
-    <entry key="U_REGEX_LOOK_BEHIND_LIMIT" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_LOOK_BEHIND_LIMIT" if="!defined(ZORBA_NO_ICU)">
       <value>look-behind pattern matches must have a bounded maximum length</value>
     </entry>
 
-    <entry key="U_REGEX_MAX_LT_MIN" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_MAX_LT_MIN" if="!defined(ZORBA_NO_ICU)">
       <value>in {min,max}, max is less than min</value>
     </entry>
 
-    <entry key="U_REGEX_MISMATCHED_PAREN" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_MISMATCHED_PAREN" if="!defined(ZORBA_NO_ICU)">
       <value>incorrectly nested parentheses</value>
     </entry>
 
-    <entry key="U_REGEX_MISSING_CLOSE_BRACKET" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_MISSING_CLOSE_BRACKET" if="!defined(ZORBA_NO_ICU)">
       <value>missing ']'</value>
     </entry>
 
-    <entry key="U_REGEX_NUMBER_TOO_BIG" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_NUMBER_TOO_BIG" if="!defined(ZORBA_NO_ICU)">
       <value>decimal number is too large</value>
     </entry>
 
-    <entry key="U_REGEX_OCTAL_TOO_BIG" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_OCTAL_TOO_BIG" if="!defined(ZORBA_NO_ICU)">
       <value>octal character constants must be &lt;= 0377</value>
     </entry>
 
-    <entry key="U_REGEX_PROPERTY_SYNTAX" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_PROPERTY_SYNTAX" if="!defined(ZORBA_NO_ICU)">
       <value>incorrect Unicode property</value>
     </entry>
 
-    <entry key="U_REGEX_RULE_SYNTAX" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_RULE_SYNTAX" if="!defined(ZORBA_NO_ICU)">
       <value>syntax error</value>
     </entry>
 
-    <entry key="U_REGEX_SET_CONTAINS_STRING" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_SET_CONTAINS_STRING" if="!defined(ZORBA_NO_ICU)">
       <value>can not have UnicodeSets containing strings</value>
     </entry>
 
-    <entry key="U_REGEX_STACK_OVERFLOW" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_STACK_OVERFLOW" if="!defined(ZORBA_NO_ICU)">
       <value>backtrack stack overflow</value>
     </entry>
 
-    <entry key="U_REGEX_STOPPED_BY_CALLER" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_STOPPED_BY_CALLER" if="!defined(ZORBA_NO_ICU)">
       <value>matching operation aborted by user callback fn</value>
     </entry>
 
-    <entry key="U_REGEX_TIME_OUT" if="!defined(ZORBA_NO_UNICODE)">
+    <entry key="U_REGEX_TIME_OUT" if="!defined(ZORBA_NO_ICU)">
       <value>maximum allowed match time exceeded</value>
     </entry>
 
-    <entry key="U_REGEX_UNIMPLEMENTED" if="!defined(ZORBA_NO_UNICODE)">
-      <value>use of regular expression feature that is not yet implemented</value>
-    </entry>
+    <entry key="U_REGEX_UNIMPLEMENTED" if="!defined(ZORBA_NO_ICU)">
+      <value>use of regular expression feature that is not yet implemented</value>
+    </entry>
+
+    <!-- Regex Ascii error messages-->
+    <entry key="REGEX_UNIMPLEMENTED" if="defined(ZORBA_NO_ICU)">
+      <value>use of regular expression feature that is not yet implemented</value>
+    </entry>
+
+    <entry key="REGEX_MISMATCHED_PAREN" if="defined(ZORBA_NO_ICU)">
+      <value>incorrectly nested parentheses</value>
+    </entry>
+
+    <entry key="REGEX_BROKEN_P_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>broken \\p construct</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PL_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{L?} category; supported categories: L, Lu, Ll, Lt, Lm, Lo</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PM_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{M?} category; supported categories: M, Mn, Mc, Me</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PN_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{N?} category; supported categories: N, Nd, Nl, No</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PP_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{P?} category; supported categories: P, Pc, Pd, Ps, Pe, Pi, Pf, Po</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PZ_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{Z?} category; supported categories: Z, Zs, Zl, Zp</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PS_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{S?} category; supported categories: S, Sm, Sc, Sk, So</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PC_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{C?} category; supported categories: C, Cc, Cf, Co, Cn(for not assigned)</value>
+    </entry>
+
+    <entry key="REGEX_BROKEN_PIs_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>broken \\p{Is} construct; valid characters are [a-zA-Z0-9-]</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_PIs_CONSTRUCT" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\p{Is} category block; see supported block escapes here: http://www.w3.org/TR/xmlschema-2/#charcter-classes</value>
+    </entry>
+
+    <entry key="REGEX_INVALID_UNICODE_CODEPOINT_u" if="defined(ZORBA_NO_ICU)">
+      <value>invalid unicode hex, should be in form \\uXXXX or \\UXXXXXXXX</value>
+    </entry>
+
+    <entry key="REGEX_UNKNOWN_ESC_CHAR" if="defined(ZORBA_NO_ICU)">
+      <value>unknown \\? escape char; supported escapes are: \\[nrt\\|.?*+(){}[]-^$] for char escapes, \\[pP] for categories and \\[sSiIcCdDwW] for multichar groups</value>
+    </entry>
+
+    <entry key="REGEX_INVALID_BACK_REF" if="defined(ZORBA_NO_ICU)">
+      <value>\\$3 backreference to a non-existent capture group ($4 groups so far)</value>
+    </entry>
+
+    <entry key="REGEX_INVALID_ATOM_CHAR" if="defined(ZORBA_NO_ICU)">
+      <value>$3 - invalid character for an atom; forbidden characters are: [{}?*+|^]</value>
+    </entry>
+
+    <entry key="REGEX_INVALID_SUBCLASS" if="defined(ZORBA_NO_ICU)">
+      <value>malformed class subtraction</value>
+    </entry>
+
+    <entry key="REGEX_INVALID_USE_OF_SUBCLASS" if="defined(ZORBA_NO_ICU)">
+      <value>improper use of class subtraction: it must be the last construct in a class group [xxx-[yyy]]</value>
+    </entry>
+
+    <entry key="REGEX_MULTICHAR_IN_CHAR_RANGE" if="defined(ZORBA_NO_ICU)">
+      <value>multichars or char categories cannot be part of a char range</value>
+    </entry>
+
+    <entry key="REGEX_MISSING_CLOSE_BRACKET" if="defined(ZORBA_NO_ICU)">
+      <value>missing close bracket in char group</value>
+    </entry>
+
+    <entry key="REGEX_MAX_LT_MIN" if="defined(ZORBA_NO_ICU)">
+      <value>in {min,max}, max is less than min</value>
+    </entry>
+
 
     <entry key="UnaryArithOp">
       <value>unary arithmetic operator</value>

=== modified file 'src/diagnostics/pregenerated/dict_en.cpp'
--- src/diagnostics/pregenerated/dict_en.cpp	2011-12-21 14:40:33 +0000
+++ src/diagnostics/pregenerated/dict_en.cpp	2012-01-18 18:33:36 +0000
@@ -565,6 +565,69 @@
   { "~ParserNoCreateTree", "XML tree creation failed" },
   { "~PromotionImpossible", "promotion not possible" },
   { "~QuotedColon_23", "\"$2\": $3" },
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_BROKEN_PIs_CONSTRUCT", "broken \\p{Is} construct; valid characters are [a-zA-Z0-9-]" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_BROKEN_P_CONSTRUCT", "broken \\p construct" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_INVALID_ATOM_CHAR", "$3 - invalid character for an atom; forbidden characters are: [{}?*+|^]" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_INVALID_BACK_REF", "\\$3 backreference to a non-existent capture group ($4 groups so far)" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_INVALID_SUBCLASS", "malformed class subtraction" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_INVALID_UNICODE_CODEPOINT_u", "invalid unicode hex, should be in form \\uXXXX or \\UXXXXXXXX" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_INVALID_USE_OF_SUBCLASS", "improper use of class subtraction: it must be the last construct in a class group [xxx-[yyy]]" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_MAX_LT_MIN", "in {min,max}, max is less than min" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_MISMATCHED_PAREN", "incorrectly nested parentheses" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_MISSING_CLOSE_BRACKET", "missing close bracket in char group" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_MULTICHAR_IN_CHAR_RANGE", "multichars or char categories cannot be part of a char range" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNIMPLEMENTED", "use of regular expression feature that is not yet implemented" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_ESC_CHAR", "unknown \\? escape char; supported escapes are: \\[nrt\\|.?*+(){}[]-^$] for char escapes, \\[pP] for categories and \\[sSiIcCdDwW] for multichar groups" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PC_CONSTRUCT", "unknown \\p{C?} category; supported categories: C, Cc, Cf, Co, Cn(for not assigned)" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PIs_CONSTRUCT", "unknown \\p{Is} category block; see supported block escapes here: http://www.w3.org/TR/xmlschema-2/#charcter-classes" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PL_CONSTRUCT", "unknown \\p{L?} category; supported categories: L, Lu, Ll, Lt, Lm, Lo" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PM_CONSTRUCT", "unknown \\p{M?} category; supported categories: M, Mn, Mc, Me" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PN_CONSTRUCT", "unknown \\p{N?} category; supported categories: N, Nd, Nl, No" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PP_CONSTRUCT", "unknown \\p{P?} category; supported categories: P, Pc, Pd, Ps, Pe, Pi, Pf, Po" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PS_CONSTRUCT", "unknown \\p{S?} category; supported categories: S, Sm, Sc, Sk, So" },
+#endif
+#if defined(ZORBA_NO_ICU)
+  { "~REGEX_UNKNOWN_PZ_CONSTRUCT", "unknown \\p{Z?} category; supported categories: Z, Zs, Zl, Zp" },
+#endif
   { "~SEPM0009_Not10", "the version parameter has a value other than \"1.0\" and the doctype-system parameter is specified" },
   { "~SEPM0009_NotOmit", "the standalone attribute has a value other than \"omit\"" },
   { "~SchemaAttributeName", "schema-attribute name" },
@@ -588,64 +651,64 @@
   { "~TwoDecimalFormatsSameName_2", "\"$2\": two decimal formats with this name" },
   { "~TwoDefaultDecimalFormats", "two default decimal formats" },
   { "~TypeIsNotSubtype", "item type is not a subtype of \"$3\"" },
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_BAD_ESCAPE_SEQUENCE", "unrecognized backslash escape sequence" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_BAD_INTERVAL", "error in {min,max} interval" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_INTERNAL_ERROR", "an internal ICU error (bug) was detected" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_INVALID_BACK_REF", "backreference to a non-existent capture group" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_INVALID_FLAG", "invalid value for match mode flags" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_INVALID_RANGE", "in character range [x-y], x is greater than y" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_INVALID_STATE", "RegexMatcher in invalid state for requested operation" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_LOOK_BEHIND_LIMIT", "look-behind pattern matches must have a bounded maximum length" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_MAX_LT_MIN", "in {min,max}, max is less than min" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_MISMATCHED_PAREN", "incorrectly nested parentheses" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_MISSING_CLOSE_BRACKET", "missing ']'" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_NUMBER_TOO_BIG", "decimal number is too large" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_OCTAL_TOO_BIG", "octal character constants must be <= 0377" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_PROPERTY_SYNTAX", "incorrect Unicode property" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_RULE_SYNTAX", "syntax error" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_SET_CONTAINS_STRING", "can not have UnicodeSets containing strings" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_STACK_OVERFLOW", "backtrack stack overflow" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_STOPPED_BY_CALLER", "matching operation aborted by user callback fn" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_TIME_OUT", "maximum allowed match time exceeded" },
 #endif
-#if !defined(ZORBA_NO_UNICODE)
+#if !defined(ZORBA_NO_ICU)
   { "~U_REGEX_UNIMPLEMENTED", "use of regular expression feature that is not yet implemented" },
 #endif
   { "~UnaryArithOp", "unary arithmetic operator" },

=== modified file 'src/runtime/full_text/CMakeLists.txt'
--- src/runtime/full_text/CMakeLists.txt	2011-08-31 13:17:59 +0000
+++ src/runtime/full_text/CMakeLists.txt	2012-01-18 18:33:36 +0000
@@ -42,11 +42,11 @@
     default_tokenizer.cpp
     )
 
-IF (ZORBA_NO_UNICODE)
+IF (ZORBA_NO_ICU)
   LIST(APPEND FULLTEXT_SRCS latin_tokenizer.cpp)
-ELSE (ZORBA_NO_UNICODE)
+ELSE (ZORBA_NO_ICU)
   LIST(APPEND FULLTEXT_SRCS icu_tokenizer.cpp)
-ENDIF (ZORBA_NO_UNICODE)
+ENDIF (ZORBA_NO_ICU)
 
 ADD_SRC_SUBFOLDER(FULLTEXT_SRCS stemmer LIBSTEMMER_SRCS)
 

=== modified file 'src/runtime/full_text/default_tokenizer.cpp'
--- src/runtime/full_text/default_tokenizer.cpp	2011-08-31 02:53:07 +0000
+++ src/runtime/full_text/default_tokenizer.cpp	2012-01-18 18:33:36 +0000
@@ -19,22 +19,22 @@
 #include <zorba/config.h>
 
 #include "default_tokenizer.h"
-#ifdef ZORBA_NO_UNICODE
+#ifdef ZORBA_NO_ICU
 # include "latin_tokenizer.h"
 #else
 # include "icu_tokenizer.h"
-#endif /* ZORBA_NO_UNICODE */
+#endif /* ZORBA_NO_ICU */
 
 namespace zorba {
 
 ///////////////////////////////////////////////////////////////////////////////
 
 TokenizerProvider const& default_tokenizer_provider() {
-#ifdef ZORBA_NO_UNICODE
+#ifdef ZORBA_NO_ICU
   static LatinTokenizerProvider const instance;
 #else
   static ICU_TokenizerProvider const instance;
-#endif /* ZORBA_NO_UNICODE */
+#endif /* ZORBA_NO_ICU */
   return instance;
 };
 

=== modified file 'src/runtime/full_text/latin_tokenizer.cpp'
--- src/runtime/full_text/latin_tokenizer.cpp	2011-08-31 03:39:32 +0000
+++ src/runtime/full_text/latin_tokenizer.cpp	2012-01-18 18:33:36 +0000
@@ -18,8 +18,9 @@
 #include <functional>
 
 #include <zorba/diagnostic_list.h>
-#include <zorba/xquery_exception.h>
-#include <zorba/zorba.h>
+
+#include "diagnostics/dict.h"
+#include "diagnostics/xquery_exception.h"
 
 #include "latin_tokenizer.h"
 

=== modified file 'src/runtime/full_text/latin_tokenizer.h'
--- src/runtime/full_text/latin_tokenizer.h	2011-08-31 03:39:32 +0000
+++ src/runtime/full_text/latin_tokenizer.h	2012-01-18 18:33:36 +0000
@@ -14,12 +14,12 @@
  * limitations under the License.
  */
 
-#ifndef ZORBA_WESTERN_TOKENIZER_H
-#define ZORBA_WESTERN_TOKENIZER_H
+#ifndef ZORBA_LATIN_TOKENIZER_H
+#define ZORBA_LATIN_TOKENIZER_H
 
 #include <zorba/config.h>
 
-#ifdef ZORBA_NO_FULL_TEXT
+#ifdef ZORBA_NO_ICU
 
 #include <zorba/tokenizer.h>
 #include "zorbatypes/zstring.h"
@@ -38,8 +38,8 @@
 
   // inherited
   void destroy() const;
-  void tokenize( char const*, size_type, iso639_1::type, bool, Callback&,
-                 void* );
+  void tokenize( char const*, size_type, locale::iso639_1::type, bool,
+                 Callback&, void* );
 
 private:
   typedef zstring string_type;
@@ -64,13 +64,14 @@
 class LatinTokenizerProvider : public TokenizerProvider {
 public:
   // inherited
-  Tokenizer::ptr getTokenizer( iso639_1::type, Tokenizer::Numbers& ) const;
+  Tokenizer::ptr getTokenizer( locale::iso639_1::type,
+                               Tokenizer::Numbers& ) const;
 };
 
 ///////////////////////////////////////////////////////////////////////////////
 
 } // namespace zorba
 
-#endif /* ZORBA_NO_FULL_TEXT */
-#endif /* ZORBA_WESTERN_TOKENIZER_H */
+#endif /* ZORBA_NO_ICU */
+#endif /* ZORBA_LATIN_TOKENIZER_H */
 /* vim:set et sw=2 ts=2: */

=== modified file 'src/runtime/numerics/format_integer_impl.cpp'
--- src/runtime/numerics/format_integer_impl.cpp	2011-07-07 12:47:14 +0000
+++ src/runtime/numerics/format_integer_impl.cpp	2012-01-18 18:33:36 +0000
@@ -881,7 +881,7 @@
             utf8_result += (*valueit);
           }
           else
-            utf8_result += (0x2080 + *valueit - '0');
+            utf8_result += (unicode::code_point)(0x2080 + *valueit - '0');
         }
       }
       else if((c0 == 0x2460) || //CIRCLED DIGIT ONE  (1-20)

=== modified file 'src/runtime/numerics/numerics_impl.cpp'
--- src/runtime/numerics/numerics_impl.cpp	2011-07-10 14:55:46 +0000
+++ src/runtime/numerics/numerics_impl.cpp	2012-01-18 18:33:36 +0000
@@ -490,7 +490,7 @@
     minus( "-" )
   {
     utf8_string<zstring> u_per_mille( per_mille );
-    u_per_mille = 0x2030;
+    u_per_mille = (unicode::code_point)0x2030;
   }
 
   void readFormat(const DecimalFormat_t& df_t)

=== modified file 'src/runtime/strings/strings_impl.cpp'
--- src/runtime/strings/strings_impl.cpp	2012-01-11 17:30:25 +0000
+++ src/runtime/strings/strings_impl.cpp	2012-01-18 18:33:36 +0000
@@ -806,7 +806,9 @@
   zstring normForm;
   zstring resStr;
   unicode::normalization::type normType;
+#ifndef ZORBA_NO_ICU
   bool success;
+#endif /* ZORBA_NO_ICU */
 
   PlanIteratorState* state;
   DEFAULT_STACK_INIT(PlanIteratorState, state, planState);
@@ -856,10 +858,10 @@
     }
 
     item0->getStringValue2(resStr);
-#ifndef ZORBA_NO_UNICODE
+#ifndef ZORBA_NO_ICU
     success = utf8::normalize(resStr, normType, &resStr);
     ZORBA_ASSERT(success);
-#endif//#ifndef ZORBA_NO_UNICODE
+#endif//#ifndef ZORBA_NO_ICU
     STACK_PUSH(GENV_ITEMFACTORY->createString(result, resStr), state );
   }
   else
@@ -988,7 +990,7 @@
         trans_map[ *map_i ] = *trans_i;
 
       for ( ; map_i != map_end; ++map_i )
-        trans_map[ *map_i ] = ~0;
+        trans_map[ *map_i ] = static_cast<unicode::code_point>( ~0 );
     }
 
     utf8_string<zstring> u_result_string( result_string );
@@ -1003,7 +1005,7 @@
       cp_map_type::const_iterator const found_i = trans_map.find( cp );
       if ( found_i != trans_map.end() ) {
         cp = found_i->second;
-        if ( cp == ~0 )
+        if ( cp == static_cast<unicode::code_point>( ~0 ) )
           continue;
       }
       u_result_string += cp;
@@ -1791,16 +1793,33 @@
                           int &utf8start,
                           unsigned int &bytestart,
                           int utf8end,
+                          unsigned int byteend,
                           zstring &out)
 {
+#ifndef ZORBA_NO_ICU
   utf8::size_type clen;
-  while(utf8start < utf8end)
-  {
-    clen = utf8::char_length(*sin);
-    out.append(sin, clen);
-    utf8start++;
-    bytestart += clen;
-    sin += clen;
+  if(utf8end)
+  {
+    while(utf8start < utf8end)
+    {
+      clen = utf8::char_length(*sin);
+      if(clen == 0)
+        clen = 1;
+      out.append(sin, clen);
+      utf8start++;
+      bytestart += clen;
+      sin += clen;
+    }
+  }
+  else
+#endif
+  {
+    if(!utf8end)
+      utf8end = byteend;
+    out.append(sin, utf8end-bytestart);
+    sin += utf8end-bytestart;
+    utf8start = utf8end;
+    bytestart = utf8end;
   }
 }
 
@@ -1808,6 +1827,7 @@
                                int &match_end1,
                                unsigned int &match_end1_bytes,
                                int match_start2,
+                               unsigned int match_start2_bytes,
                                const char *&strin)
 {
   store::Item_t non_match_elem;
@@ -1829,7 +1849,7 @@
   //  utf8_it++;
   //  match_end1++;
   //}
-  copyUtf8Chars(strin, match_end1, match_end1_bytes, match_start2, non_match_str);
+  copyUtf8Chars(strin, match_end1, match_end1_bytes, match_start2, match_start2_bytes, non_match_str);
   store::Item_t non_match_text_item;
   GENV_ITEMFACTORY->createTextNode(non_match_text_item, non_match_elem, non_match_str);
 }
@@ -1860,19 +1880,31 @@
       i--;
       break;
     }
+#ifndef ZORBA_NO_ICU
     match_startg = rx.get_match_start(i+1);
     if((match_startg < 0) && (gparent < 0))
       continue;
+#else
+    int temp_endg;
+    match_startg = -1;
+    temp_endg = -1;
+    if(!rx.get_match_start_end_bytes(i+1, &match_startg, &temp_endg) && (gparent < 0))
+      continue;
+#endif
     if(match_endgood < match_startg)
     {
       //add non-group match text
       zstring                non_group_str;
 
-      copyUtf8Chars(sin, match_endgood, match_end1_bytes, match_startg, non_group_str);
+      copyUtf8Chars(sin, match_endgood, match_end1_bytes, match_startg, 0, non_group_str);
       store::Item_t non_group_text_item;
       GENV_ITEMFACTORY->createTextNode(non_group_text_item, parent.getp(), non_group_str);
     }
+#ifndef ZORBA_NO_ICU
     match_endg = rx.get_match_end(i+1);
+#else
+    match_endg = temp_endg;
+#endif
     //add group match text
     GENV_ITEMFACTORY->createQName(group_element_name,
                                   static_context::W3C_FN_NS, "fn", "group");
@@ -1903,7 +1935,7 @@
     }
     zstring                group_str;
 
-    copyUtf8Chars(sin, match_startg, match_end1_bytes, match_endg, group_str);
+    copyUtf8Chars(sin, match_startg, match_end1_bytes, match_endg, 0, group_str);
     store::Item_t group_text_item;
     GENV_ITEMFACTORY->createTextNode(group_text_item, group_elem.getp(), group_str);
   }
@@ -1912,7 +1944,7 @@
   {
     zstring                non_group_str;
 
-    copyUtf8Chars(sin, match_endgood, match_end1_bytes, match_end2, non_group_str);
+    copyUtf8Chars(sin, match_endgood, match_end1_bytes, match_end2, 0, non_group_str);
     store::Item_t non_group_text_item;
     GENV_ITEMFACTORY->createTextNode(non_group_text_item, parent, non_group_str);
   }
@@ -2140,8 +2172,14 @@
       reachedEnd = false;
       while(rx.find_next_match(&reachedEnd))
       {
-        int    match_start2 = rx.get_match_start();
-        int    match_end2 = rx.get_match_end();
+        int    match_start2;
+        int    match_end2;
+#ifndef ZORBA_NO_ICU
+        match_start2 = rx.get_match_start();
+        match_end2 = rx.get_match_end();
+#else
+        rx.get_match_start_end_bytes(0, &match_start2, &match_end2);
+#endif
         ZORBA_ASSERT(match_start2 >= 0);
 
         if(is_input_stream && reachedEnd && !instream->eof())
@@ -2153,7 +2191,7 @@
         //construct the fn:non-match
         if(match_start2 > match_end1)
         {
-          addNonMatchElement(result, match_end1, match_end1_bytes, match_start2, instr);
+          addNonMatchElement(result, match_end1, match_end1_bytes, match_start2, 0, instr);
         }
 
         //construct the fn:match
@@ -2161,7 +2199,7 @@
         match_end1 = match_end2;
       }
 
-      if(is_input_stream && reachedEnd && !instream->eof())
+      if(is_input_stream && !instream->eof())
       {
         //load some more data, maybe the match will be different
         if(match_end1_bytes)
@@ -2209,7 +2247,7 @@
       else
       {
         if(match_end1_bytes < streambuf_read)
-          addNonMatchElement(result, match_end1, match_end1_bytes, streambuf_read, instr);
+          addNonMatchElement(result, match_end1, match_end1_bytes, 0, streambuf_read, instr);
         if(is_input_stream && instream->eof())
           reachedEnd = true;
       }

=== modified file 'src/system/globalenv.cpp'
--- src/system/globalenv.cpp	2012-01-11 17:30:25 +0000
+++ src/system/globalenv.cpp	2012-01-18 18:33:36 +0000
@@ -17,11 +17,11 @@
 
 #include "common/common.h"
 
-#ifndef ZORBA_NO_UNICODE
+#ifndef ZORBA_NO_ICU
 # include <unicode/uclean.h>
 # include <unicode/utypes.h>
 # include <unicode/udata.h>
-#endif /* ZORBA_NO_UNICODE */
+#endif /* ZORBA_NO_ICU */
 
 #ifdef ZORBA_WITH_BIG_INTEGER
 # include "zorbatypes/m_apm.h"
@@ -208,7 +208,7 @@
   // from one thread only
   // see http://www.icu-project.org/userguide/design.html#Init_and_Termination
   // and http://www.icu-project.org/apiref/icu4c/uclean_8h.html
-#ifndef ZORBA_NO_UNICODE
+#ifndef ZORBA_NO_ICU
 #  if defined U_STATIC_IMPLEMENTATION && (defined WIN32 || defined WINCE)
   {
     TCHAR    self_path[1024];
@@ -238,13 +238,13 @@
     udata_setCommonData(icu_appdata, &data_err);
     ZORBA_ASSERT(data_err == U_ZERO_ERROR);
   
-      //  u_setDataDirectory(self_path);
+    // u_setDataDirectory(self_path);
   }
 #  endif
   UErrorCode lICUInitStatus = U_ZERO_ERROR;
   u_init(&lICUInitStatus);
   ZORBA_ASSERT(lICUInitStatus == U_ZERO_ERROR);
-#endif//ifndef ZORBA_NO_UNICODE
+#endif /* ZORBA_NO_ICU */
 }
 
 
@@ -256,12 +256,12 @@
   // releases statically initialized memory and prevents
   // valgrind from reporting those problems at the end
   // see http://www.icu-project.org/apiref/icu4c/uclean_8h.html#93f27d0ddc7c196a1da864763f2d8920
-#ifndef ZORBA_NO_UNICODE
+#ifndef ZORBA_NO_ICU
   u_cleanup();
 # if defined U_STATIC_IMPLEMENTATION && (defined WIN32 || defined WINCE)
   delete[] icu_appdata;
 # endif
-#endif//ifndef ZORBA_NO_UNICODE
+#endif /* ZORBA_NO_ICU */
 }
 
 

=== modified file 'src/util/CMakeLists.txt'
--- src/util/CMakeLists.txt	2011-07-18 14:25:21 +0000
+++ src/util/CMakeLists.txt	2012-01-18 18:33:36 +0000
@@ -38,9 +38,9 @@
   LIST(APPEND UTIL_SRCS mmap_file.cpp)
 ENDIF(ZORBA_WITH_FILE_ACCESS)
 
-IF(ZORBA_NO_UNICODE)
-  LIST(APPEND UTIL_SRCS regex_ascii.cpp)
-ENDIF(ZORBA_NO_UNICODE)
+IF(ZORBA_NO_ICU)
+  LIST(APPEND UTIL_SRCS regex_xquery.cpp)
+ENDIF(ZORBA_NO_ICU)
 
 HEADER_GROUP_SUBFOLDER(UTIL_SRCS fx)
 HEADER_GROUP_SUBFOLDER(UTIL_SRCS win32)

=== modified file 'src/util/regex.cpp'
--- src/util/regex.cpp	2011-09-24 00:16:36 +0000
+++ src/util/regex.cpp	2012-01-18 18:33:36 +0000
@@ -33,8 +33,7 @@
 #define INVALID_RE_EXCEPTION(...) \
   XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS( __VA_ARGS__ ) )
 
-
-#ifndef ZORBA_NO_UNICODE
+#ifndef ZORBA_NO_ICU
 # include <unicode/uversion.h>
 U_NAMESPACE_USE
 
@@ -442,11 +441,11 @@
 }
 
 } // namespace unicode
-
-}//namespace zorba
-
-
-#else /* ZORBA_NO_UNICODE */
+} // namespace zorba
+
+///////////////////////////////////////////////////////////////////////////////
+
+#else /* ZORBA_NO_ICU */
 
 #include "zorbatypes/zstring.h"
 
@@ -470,7 +469,7 @@
     case 'i': flags |= REGEX_ASCII_CASE_INSENSITIVE; break;
     case 's': flags |= REGEX_ASCII_DOTALL; break;
     case 'm': flags |= REGEX_ASCII_MULTILINE; break;
-    case 'x': flags |= REGEX_ASCII_COMMENTS; break;
+    case 'x': flags |= REGEX_ASCII_NO_WHITESPACE; break;
     case 'q': flags |= REGEX_ASCII_LITERAL; break;
     default:
       throw XQUERY_EXCEPTION( err::FORX0001, ERROR_PARAMS( *p ) );
@@ -483,6 +482,7 @@
 void regex::compile( char const *pattern, char const *flags)
 {
   parsed_flags = parse_regex_flags(flags);
+  regex_xquery::CRegexXQuery_parser regex_parser;
   regex_matcher = regex_parser.parse(pattern, parsed_flags);
   if(!regex_matcher)
     throw INVALID_RE_EXCEPTION(pattern);
@@ -517,6 +517,8 @@
 bool regex::next_token( char const *s, size_type *pos, zstring *token,
                   bool *matched)
 {
+  if(!s[*pos])
+    return false;
   bool  retval;
   int   match_pos;
   int   matched_len;
@@ -528,14 +530,8 @@
       token->assign(s+*pos, match_pos);
     *pos += match_pos + matched_len;
     if(matched)
-      if(match_pos)
-        *matched = true;
-      else
-        *matched = false;
-    if(match_pos)
-      return true;
-    else
-      return false;
+      *matched = true;
+    return true;
   }
   else
   {
@@ -544,7 +540,7 @@
     *pos += strlen(s+*pos);
     if(matched)
       *matched = false;
-    return s[*pos] != 0;
+    return true;
   }
 }
 
@@ -554,13 +550,9 @@
   int   matched_pos;
   int   matched_len;
 
-  bool prev_align = regex_matcher->set_align_begin(true);
-  retval = regex_matcher->match_from(s, parsed_flags, &matched_pos, &matched_len);
-  regex_matcher->set_align_begin(prev_align);
+  retval = regex_matcher->match_anywhere(s, parsed_flags|REGEX_ASCII_WHOLE_MATCH, &matched_pos, &matched_len);
   if(!retval)
     return false;
-  if(matched_len != strlen(s))
-    return false;
   return true;
 }
 
@@ -587,14 +579,19 @@
       //look for dollars
       if(*temprepl == '\\')
       {
-        temprepl++;
-        if(!*temprepl || (*temprepl != '\\') || (*temprepl != '$'))//Invalid replacement string.
-          throw XQUERY_EXCEPTION( err::FORX0004, ERROR_PARAMS( replacement ) );
+        if(!(parsed_flags & REGEX_ASCII_LITERAL))
+        {
+          temprepl++;
+          if(!*temprepl) 
+            temprepl--;
+          else if((*temprepl != '\\') && (*temprepl != '$'))//Invalid replacement string.
+            throw XQUERY_EXCEPTION( err::FORX0004, ERROR_PARAMS( replacement ) );
+        }
         result->append(1, *temprepl);
         temprepl++;
         continue;
       }
-      if(*temprepl == '$')
+      if((*temprepl == '$') && !(parsed_flags & REGEX_ASCII_LITERAL))
       {
         temprepl++;
         index = 0;
@@ -648,7 +645,7 @@
   if(retval)
   {
     m_match_pos += m_pos;
-    m_pos = m_match_pos = m_matched_len;
+    m_pos = m_match_pos + m_matched_len;
   }
   else
   {
@@ -666,35 +663,30 @@
   return (int)regex_matcher->get_indexed_regex_count();
 }
 
-int regex::get_match_start( int groupId )
-{
-  if(groupId == 0)
-    return m_match_pos;
-  if(groupId > (int)regex_matcher->get_indexed_regex_count())
-    return -1;
-  const char *submatched_source;
-  int   submatched_len;
-  if(!regex_matcher->get_indexed_match(groupId, &submatched_source, &submatched_len))
-    return -1;
-  return submatched_source - s_in_.c_str();
-}
-
-int regex::get_match_end( int groupId )
-{
-  if(groupId == 0)
-    return m_match_pos + m_matched_len;
-  if(groupId > (int)regex_matcher->get_indexed_regex_count())
-    return -1;
-  const char *submatched_source;
-  int   submatched_len;
-  if(!regex_matcher->get_indexed_match(groupId, &submatched_source, &submatched_len))
-    return -1;
-  return submatched_source - s_in_.c_str() + submatched_len;
+bool regex::get_match_start_end_bytes( int groupId, int *start, int *end )
+{
+  *start = -1;
+  *end = -1;
+  if(groupId == 0)
+  {
+    *start = m_match_pos;
+    *end = m_match_pos + m_matched_len;
+    return true;
+  }
+  if(groupId > (int)regex_matcher->get_indexed_regex_count())
+    return false;
+  const char *submatched_source;
+  int   submatched_len;
+  if(!regex_matcher->get_indexed_match(groupId, &submatched_source, &submatched_len))
+    return false;
+  *start = submatched_source - s_in_.c_str();
+  *end = *start + submatched_len;
+  return true;
 }
 
 } // namespace unicode
 } // namespace zorba
-#endif /* ZORBA_NO_UNICODE */
+#endif /* ZORBA_NO_ICU */
 
 ///////////////////////////////////////////////////////////////////////////////
 

=== modified file 'src/util/regex.h'
--- src/util/regex.h	2011-07-18 14:25:21 +0000
+++ src/util/regex.h	2012-01-18 18:33:36 +0000
@@ -17,15 +17,13 @@
 #ifndef ZORBA_REGEX_H
 #define ZORBA_REGEX_H
 
-#ifndef ZORBA_NO_UNICODE
-#include <unicode/regex.h>
-#endif
-
 #include "cxx_util.h"
 #include "unicode_util.h"
 #include "zorbatypes/zstring.h"
 
-#ifndef ZORBA_NO_UNICODE
+#ifndef ZORBA_NO_ICU
+
+#include <unicode/regex.h>
 
 namespace zorba {
 
@@ -496,15 +494,17 @@
 } // namespace unicode
 } // namespace zorba
 
-#else ///ZORBA_NO_UNICODE (ascii part:)
-
-#include "util/regex_ascii.h"
+///////////////////////////////////////////////////////////////////////////////
+
+#else /* ZORBA_NO_ICU */
+
+#include "util/regex_xquery.h"
 #include <string>
 
 namespace zorba{
 /**
  * Converts an XQuery regular expression to the form used by the regular
- * expression library Zorba is using (here regex_ascii).
+ * expression library Zorba is using (here regex_xquery).
  *
  * @param xq_re The XQuery regular expression.
  * @param lib_re A pointer to the resuling library regular expression.
@@ -525,7 +525,7 @@
   /**
    * Constructs a %regex.
    */
-  regex() : regex_matcher( NULL ) { }
+  regex() : regex_matcher( nullptr ) { }
 
   /**
    * Destroys a %regex.
@@ -835,31 +835,21 @@
 
   /**
    * Get the start position of the matched group.
-   * If groupId is zero, then the start position of the whole match is returned.
-   * If groupId is non-zero, then the start position of that group is returned.
-   * If that group has not been matched, -1 is returned.
+   * If groupId is zero, then the start and end position of the whole match is returned.
+   * If groupId is non-zero, then the start and end position of that group is returned.
+   * If that group has not been matched, false is returned.
    *
    * @param groupId the id of the group, either zero for the entire regex,
    *  or [1 .. group_count] for that specific group
-   * @return the start position, zero based, or -1 if that group didn't match
+   * @param start to return start position in bytes
+   * @param end to return end position in bytes
+   * @return true if that group exists and has been matched
    */
-  int get_match_start( int groupId = 0 );
+  bool get_match_start_end_bytes( int groupId, int *start, int *end );
 
-  /**
-   * Get the end position of the matched group.
-   * If groupId is zero, then the end position of the whole match is returned.
-   * If groupId is non-zero, then the end position of that group is returned.
-   * If that group has not been matched, -1 is returned.
-   *
-   * @param groupId the id of the group, either zero for the entire regex,
-   *  or [1 .. group_count] for that specific group
-   * @return the end position, zero based, or -1 if that group didn't match
-   */
-  int get_match_end( int groupId = 0 );
 
 private:
-  regex_ascii::CRegexAscii_parser regex_parser;
-  regex_ascii::CRegexAscii_regex  *regex_matcher;
+  regex_xquery::CRegexXQuery_regex  *regex_matcher;
   uint32_t    parsed_flags;
 
   zstring s_in_;
@@ -873,15 +863,13 @@
   regex( regex const& );
   regex& operator=( regex const& );
 };
+
+///////////////////////////////////////////////////////////////////////////////
+
 } // namespace unicode
 } // namespace zorba
 
-#endif /* ZORBA_NO_UNICODE */
-
-
-///////////////////////////////////////////////////////////////////////////////
-
-
+#endif /* ZORBA_NO_ICU */
 #endif /* ZORBA_REGEX_H */
 /*
  * Local variables:

=== renamed file 'src/util/regex_ascii.cpp' => 'src/util/regex_xquery.cpp'
--- src/util/regex_ascii.cpp	2011-08-05 02:21:55 +0000
+++ src/util/regex_xquery.cpp	2012-01-18 18:33:36 +0000
@@ -1,4 +1,4 @@
-a/*
+/*
  * Copyright 2006-2008 The FLWOR Foundation.
  * 
  * Licensed under the Apache License, Version 2.0 (the "License");
@@ -18,12 +18,15 @@
 
 #include "diagnostics/xquery_diagnostics.h"
 
-#include "regex_ascii.h"
+#include "regex_xquery.h"
 #include <string.h>
 #include "zorbatypes/chartype.h"
+#include "util/unicode_categories.h"
+#include "util/ascii_util.h"
+#include "util/utf8_string.h"
 
 namespace zorba {
-  namespace regex_ascii{
+  namespace regex_xquery{
 //ascii regular expression matching
 
 /*http://www.w3.org/TR/xmlschema-2/#regexs
@@ -62,96 +65,138 @@
 + http://www.w3.org/TR/xquery-operators/#regex-syntax (not implemented)
 */
 
+
+static bool compare_ascii_i(const char *str1, const char *str2)
+{
+  while(*str1 && *str2)
+  {
+    if(ascii::to_lower(*str1) != ascii::to_lower(*str2))
+      return false;
+    str1++;
+    str2++;
+  }
+  if(*str1 || *str2)
+    return false;
+  return true;
+}
+
+static bool compare_unicode_ni(const char *str1, const char *str2, int len)
+{
+  while(len > 0)
+  {
+    const char *temp_str1 = str1;
+    const char *temp_str2 = str2;
+    unicode::code_point cp1 = unicode::to_upper(utf8::next_char(temp_str1));
+    unicode::code_point cp2 = unicode::to_upper(utf8::next_char(temp_str2));
+    if(cp1 != cp2)
+      return false;
+    len -= temp_str1-str1;
+    str1 = temp_str1;
+    str2 = temp_str2;
+  }
+  return true;
+}
+static utf8::size_type myutf8len(const char *source)
+{
+  utf8::size_type  len = utf8::char_length(*source);
+  if(!len)
+    return 1;
+  else
+    return len;
+}
 ////////////////////////////////////
 ////Regular expression parsing and building of the tree
 ////////////////////////////////////
 
-CRegexAscii_regex* CRegexAscii_parser::parse(const char *pattern, unsigned int flags)
+CRegexXQuery_regex* CRegexXQuery_parser::parse(const char *pattern, unsigned int flags)
 {
   this->flags = flags;
-  bool align_begin = false;
   
-  if(!(flags & REGEX_ASCII_LITERAL) && (pattern[0] == '^'))
-    align_begin = true;
-
   int   regex_len;
-  CRegexAscii_regex*  regex = parse_regexp(pattern + (align_begin?1:0), &regex_len);
+  CRegexXQuery_regex*  regex = parse_regexp(pattern, &regex_len);
   
-  if(regex)
-    regex->set_align_begin(align_begin);
-
   return regex;
 }
 
 //until '\0' or ')'
-CRegexAscii_regex* CRegexAscii_parser::parse_regexp(const char *pattern, 
+CRegexXQuery_regex* CRegexXQuery_parser::parse_regexp(const char *pattern, 
                                                     int *regex_len)
 {
   *regex_len = 0;
   int   branch_len;
   regex_depth++;
-  CRegexAscii_regex *regex = new CRegexAscii_regex(current_regex);
+  std::auto_ptr<CRegexXQuery_regex>  regex(new CRegexXQuery_regex(current_regex));
   if(!current_regex)
-    current_regex = regex;
+    current_regex = regex.get();
   if(regex_depth >= 2)
   {
     //mark this as group if it does not start with ?:
     if(pattern[0] != '?' || pattern[1] != ':')
-      current_regex->subregex.push_back(regex);
+      current_regex->subregex.push_back(regex.get());
     else
       *regex_len = 2;
   }
-  CRegexAscii_branch  *branch;
+  CRegexXQuery_branch  *branch;
+  bool must_read_another_branch = true;
   while(pattern[*regex_len] && (pattern[*regex_len] != ')'))
   {
     branch = parse_branch(pattern+*regex_len, &branch_len);
     if(!branch)
     {
       regex_depth--;
-      delete regex;
       return NULL;
     }
     regex->add_branch(branch);
     *regex_len += branch_len;
+    if(pattern[*regex_len] == '|')
+      (*regex_len)++;
+    else
+      must_read_another_branch = false;
   }
-  if((current_regex == regex) && (pattern[*regex_len] == ')'))
+  if((current_regex == regex.get()) && (pattern[*regex_len] == ')'))
   {
-    throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(U_REGEX_MISMATCHED_PAREN)) );
+    throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_MISMATCHED_PAREN)) );
   }
   if(pattern[*regex_len])
     (*regex_len)++;
+  if(must_read_another_branch)
+    regex->add_branch(new CRegexXQuery_branch(current_regex));//add empty branch
   regex->flags = 0;//finished initialization
   regex_depth--;
-  return regex;
+  return regex.release();
 }
 
-CRegexAscii_branch* CRegexAscii_parser::parse_branch(const char *pattern, int *branch_len)
+CRegexXQuery_branch* CRegexXQuery_parser::parse_branch(const char *pattern, int *branch_len)
 {
   int piece_len;
 
-  CRegexAscii_branch    *branch = new CRegexAscii_branch(current_regex);
-  CRegexAscii_piece     *piece;
+  std::auto_ptr<CRegexXQuery_branch>    branch(new CRegexXQuery_branch(current_regex));
+  CRegexXQuery_piece     *piece;
   *branch_len = 0;
   while(pattern[*branch_len] && (pattern[*branch_len] != '|') && (pattern[*branch_len] != ')'))
   {
     piece = parse_piece(pattern+*branch_len, &piece_len);
     if(!piece)
     {
-      delete branch;
       return NULL;
     }
+    if(branch->piece_list.size() && dynamic_cast<CRegexXQuery_pinstart*>(piece->atom))
+    {
+      //found ^ that is not at the beginning of branch
+      throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_ATOM_CHAR), '^') );
+    }
     branch->add_piece(piece);
     *branch_len += piece_len;
   }
-  if(pattern[*branch_len] == '|')
-    (*branch_len)++;
-  return branch;
+  //if(pattern[*branch_len] == '|')
+  //  (*branch_len)++;
+  return branch.release();
 }
 
 //piece = atom + quantifier
-CRegexAscii_piece* CRegexAscii_parser::parse_piece(const char *pattern, int *piece_len)
+CRegexXQuery_piece* CRegexXQuery_parser::parse_piece(const char *pattern, int *piece_len)
 {
-  CRegexAscii_piece *piece = new CRegexAscii_piece;
+  std::auto_ptr<CRegexXQuery_piece>  piece(new CRegexXQuery_piece);
   IRegexAtom  *atom;
   *piece_len = 0;
 
@@ -160,19 +205,18 @@
   atom = read_atom(pattern, &atom_len);
   if(!atom)
   {
-    delete piece;
     return NULL;
   }
   piece->set_atom(atom);
   if(!(flags & REGEX_ASCII_LITERAL))
-    read_quantifier(piece, pattern+atom_len, &quantif_len);
+    read_quantifier(piece.get(), pattern+atom_len, &quantif_len);
 
   *piece_len += atom_len + quantif_len;
 
-  return piece;
+  return piece.release();
 }
 
-char CRegexAscii_parser::myishex(char c)
+char CRegexXQuery_parser::myishex(char c)
 {
   if((c >= '0') && (c <= '9'))
     return c-'0'+1;
@@ -183,26 +227,125 @@
   return 0;//not a hex
 }
 
-bool CRegexAscii_parser::myisdigit(char c)
-{
-  return (c >= '0') || (c <= '9');
-}
-
-char CRegexAscii_parser::readChar(const char *pattern, int *char_len, bool *is_multichar)
+bool CRegexXQuery_parser::myisdigit(char c)
+{
+  return (c >= '0') && (c <= '9');
+}
+
+bool CRegexXQuery_parser::myisletterAZ(char c)
+{
+  return ((c >= 'a') && (c <= 'z')) || ((c >= 'A') && (c <= 'Z'));
+}
+
+static const unicode::code_point specials_extcp[] = {0xFFF0, 0xFFFD, 0};
+
+static CRegexXQuery_parser::block_escape_t block_escape[] = 
+{
+{{0x0000, 0x007F}, NULL, "BasicLatin"},
+{{0x0080, 0x00FF}, NULL, "Latin-1Supplement"},
+{{0x0100, 0x017F}, NULL, "LatinExtended-A"},
+{{0x0180, 0x024F}, NULL, "LatinExtended-B"},
+{{0x0250, 0x02AF}, NULL, "IPAExtensions"},
+{{0x02B0, 0x02FF}, NULL, "SpacingModifierLetters"},
+{{0x0300, 0x036F}, NULL, "CombiningDiacriticalMarks"},
+{{0x0370, 0x03FF}, NULL, "Greek"},
+{{0x0400, 0x04FF}, NULL, "Cyrillic"},
+{{0x0530, 0x058F}, NULL, "Armenian"},
+{{0x0590, 0x05FF}, NULL, "Hebrew"},
+{{0x0600, 0x06FF}, NULL, "Arabic"},
+{{0x0700, 0x074F}, NULL, "Syriac"},
+{{0x0780, 0x07BF}, NULL, "Thaana"},
+{{0x0900, 0x097F}, NULL, "Devanagari"},
+{{0x0980, 0x09FF}, NULL, "Bengali"},
+{{0x0A00, 0x0A7F}, NULL, "Gurmukhi"},
+{{0x0A80, 0x0AFF}, NULL, "Gujarati"},
+{{0x0B00, 0x0B7F}, NULL, "Oriya"},
+{{0x0B80, 0x0BFF}, NULL, "Tamil"},
+{{0x0C00, 0x0C7F}, NULL, "Telugu"},
+{{0x0C80, 0x0CFF}, NULL, "Kannada"},
+{{0x0D00, 0x0D7F}, NULL, "Malayalam"},
+{{0x0D80, 0x0DFF}, NULL, "Sinhala"},
+{{0x0E00, 0x0E7F}, NULL, "Thai"},
+{{0x0E80, 0x0EFF}, NULL, "Lao"},
+{{0x0F00, 0x0FFF}, NULL, "Tibetan"},
+{{0x1000, 0x109F}, NULL, "Myanmar"},
+{{0x10A0, 0x10FF}, NULL, "Georgian"},
+{{0x1100, 0x11FF}, NULL, "HangulJamo"},
+{{0x1200, 0x137F}, NULL, "Ethiopic"},
+{{0x13A0, 0x13FF}, NULL, "Cherokee"},
+{{0x1400, 0x167F}, NULL, "UnifiedCanadianAboriginalSyllabics"},
+{{0x1680, 0x169F}, NULL, "Ogham"},
+{{0x16A0, 0x16FF}, NULL, "Runic"},
+{{0x1780, 0x17FF}, NULL, "Khmer"},
+{{0x1800, 0x18AF}, NULL, "Mongolian"},
+{{0x1E00, 0x1EFF}, NULL, "LatinExtendedAdditional"},
+{{0x1F00, 0x1FFF}, NULL, "GreekExtended"},
+{{0x2000, 0x206F}, NULL, "GeneralPunctuation"},
+{{0x2070, 0x209F}, NULL, "SuperscriptsandSubscripts"},
+{{0x20A0, 0x20CF}, NULL, "CurrencySymbols"},
+{{0x20D0, 0x20FF}, NULL, "CombiningMarksforSymbols"},
+{{0x2100, 0x214F}, NULL, "LetterlikeSymbols"},
+{{0x2150, 0x218F}, NULL, "NumberForms"},
+{{0x2190, 0x21FF}, NULL, "Arrows"},
+{{0x2200, 0x22FF}, NULL, "MathematicalOperators"},
+{{0x2300, 0x23FF}, NULL, "MiscellaneousTechnical"},
+{{0x2400, 0x243F}, NULL, "ControlPictures"},
+{{0x2440, 0x245F}, NULL, "OpticalCharacterRecognition"},
+{{0x2460, 0x24FF}, NULL, "EnclosedAlphanumerics"},
+{{0x2500, 0x257F}, NULL, "BoxDrawing"},
+{{0x2580, 0x259F}, NULL, "BlockElements"},
+{{0x25A0, 0x25FF}, NULL, "GeometricShapes"},
+{{0x2600, 0x26FF}, NULL, "MiscellaneousSymbols"},
+{{0x2700, 0x27BF}, NULL, "Dingbats"},
+{{0x2800, 0x28FF}, NULL, "BraillePatterns"},
+{{0x2E80, 0x2EFF}, NULL, "CJKRadicalsSupplement"},
+{{0x2F00, 0x2FDF}, NULL, "KangxiRadicals"},
+{{0x2FF0, 0x2FFF}, NULL, "IdeographicDescriptionCharacters"},
+{{0x3000, 0x303F}, NULL, "CJKSymbolsandPunctuation"},
+{{0x3040, 0x309F}, NULL, "Hiragana"},
+{{0x30A0, 0x30FF}, NULL, "Katakana"},
+{{0x3100, 0x312F}, NULL, "Bopomofo"},
+{{0x3130, 0x318F}, NULL, "HangulCompatibilityJamo"},
+{{0x3190, 0x319F}, NULL, "Kanbun"},
+{{0x31A0, 0x31BF}, NULL, "BopomofoExtended"},
+{{0x3200, 0x32FF}, NULL, "EnclosedCJKLettersandMonths"},
+{{0x3300, 0x33FF}, NULL, "CJKCompatibility"},
+{{0x3400, 0x4DB5}, NULL, "CJKUnifiedIdeographsExtensionA"},
+{{0x4E00, 0x9FFF}, NULL, "CJKUnifiedIdeographs"},
+{{0xA000, 0xA48F}, NULL, "YiSyllables"},
+{{0xA490, 0xA4CF}, NULL, "YiRadicals"},
+{{0xAC00, 0xD7A3}, NULL, "HangulSyllables"},
+{{0xE000, 0xF8FF}, NULL, "PrivateUse"},
+{{0xF900, 0xFAFF}, NULL, "CJKCompatibilityIdeographs"},
+{{0xFB00, 0xFB4F}, NULL, "AlphabeticPresentationForms"},
+{{0xFB50, 0xFDFF}, NULL, "ArabicPresentationForms-A"},
+{{0xFE20, 0xFE2F}, NULL, "CombiningHalfMarks"},
+{{0xFE30, 0xFE4F}, NULL, "CJKCompatibilityForms"},
+{{0xFE50, 0xFE6F}, NULL, "SmallFormVariants"},
+{{0xFE70, 0xFEFE}, NULL, "ArabicPresentationForms-B"},
+{{0xFEFF, 0xFEFF}, specials_extcp, "Specials"},
+{{0xFF00, 0xFFEF}, NULL, "HalfwidthandFullwidthForms"}
+};
+
+CRegexXQuery_charmatch* CRegexXQuery_parser::readChar(const char *pattern, 
+                                  int *char_len,
+                                  enum CHARGROUP_t *multichar_type)
 {
   char  c = 0;
   *char_len = 0;
-  *is_multichar = false;
+  *multichar_type = CHARGROUP_NO_MULTICHAR;
   switch(pattern[*char_len])
   {
   case '\\':
-  {  (*char_len)++;
+  {
+    (*char_len)++;
     switch(pattern[*char_len])
     {
-    case 'n': c = '\n';break;
-    case 'r': c = '\r';break;
-    case 't': c = '\t';break;
+    case 'n': c = '\n';(*char_len)++;return new CRegexXQuery_char_ascii(current_regex, c);
+    case 'r': c = '\r';(*char_len)++;return new CRegexXQuery_char_ascii(current_regex, c);
+    case 't': c = '\t';(*char_len)++;return new CRegexXQuery_char_ascii(current_regex, c);
     case '\\':
+    case '/'://+
     case '|':
     case '.':
     case '?':
@@ -216,19 +359,205 @@
     case '['://#x5B
     case ']'://#x5D
     case '^'://#x5E
+    case '$'://+
        c = pattern[*char_len];
-       break;
+      (*char_len)++;
+      *multichar_type = CHARGROUP_FLAGS_ONECHAR_ASCII;
+      return new CRegexXQuery_char_ascii(current_regex, c);
     case 'p'://catEsc
     case 'P'://complEsc
+    {
       //ignore the prop for now
-      c = pattern[*char_len];
-      *is_multichar = true;
-      if(pattern[*char_len+1] == '{')
-      {
-        while(pattern[*char_len] != '}')
+      *multichar_type = CHARGROUP_FLAGS_MULTICHAR_p;//(CHARGROUP_t)((pattern[*char_len] == 'P') ? 128 : 0);
+      bool is_reverse = (pattern[*char_len] == 'P');
+      c = 0;
+      if(pattern[(*char_len)+1] != '{')
+      {
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_BROKEN_P_CONSTRUCT)) );
+      }
+      (*char_len) += 2;
+      switch(pattern[*char_len])
+      {//IsCategory
+      case 'L':
+      {
+        switch(pattern[(*char_len)+1])
+        {
+        case '}':
+          c = unicode::UNICODE_Ll + 50;break;
+        case 'u':
+          c = unicode::UNICODE_Lu; (*char_len)++;break;
+        case 'l':
+          c = unicode::UNICODE_Ll; (*char_len)++;break;
+        case 't':
+          c = unicode::UNICODE_Lt; (*char_len)++;break;
+        case 'm':
+          c = unicode::UNICODE_Lm; (*char_len)++;break;
+        case 'o':
+          c = unicode::UNICODE_Lo; (*char_len)++;break;
+        default:
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PL_CONSTRUCT)) );
+        }
+      }break;
+      case 'M':
+      {
+        switch(pattern[(*char_len)+1])
+        {
+        case '}':
+          c = unicode::UNICODE_Mc + 50;break;
+        case 'n':
+          c = unicode::UNICODE_Mn; (*char_len)++;break;
+        case 'c':
+          c = unicode::UNICODE_Mc; (*char_len)++;break;
+        case 'e':
+          c = unicode::UNICODE_Me; (*char_len)++;break;
+        default:
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PM_CONSTRUCT)) );
+        }
+      }break;
+      case 'N':
+      {
+        switch(pattern[(*char_len)+1])
+        {
+        case '}':
+          c = unicode::UNICODE_Nd + 50;break;
+        case 'd':
+          c = unicode::UNICODE_Nd; (*char_len)++;break;
+        case 'l':
+          c = unicode::UNICODE_Nl; (*char_len)++;break;
+        case 'o':
+          c = unicode::UNICODE_No; (*char_len)++;break;
+        default:
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PN_CONSTRUCT)) );
+        }
+      }break;
+      case 'P':
+      {
+        switch(pattern[(*char_len)+1])
+        {
+        case '}':
+          c = unicode::UNICODE_Pc + 50;break;
+        case 'c':
+          c = unicode::UNICODE_Pc; (*char_len)++;break;
+        case 'd':
+          c = unicode::UNICODE_Pd; (*char_len)++;break;
+        case 's':
+          c = unicode::UNICODE_Ps; (*char_len)++;break;
+        case 'e':
+          c = unicode::UNICODE_Pe; (*char_len)++;break;
+        case 'i':
+          c = unicode::UNICODE_Pi; (*char_len)++;break;
+        case 'f':
+          c = unicode::UNICODE_Pf; (*char_len)++;break;
+        case 'o':
+          c = unicode::UNICODE_Po; (*char_len)++;break;
+        default:
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PP_CONSTRUCT)) );
+        }
+      }break;
+      case 'Z':
+      {
+        switch(pattern[(*char_len)+1])
+        {
+        case '}':
+          c = unicode::UNICODE_Zl + 50;break;
+        case 's':
+          c = unicode::UNICODE_Zs; (*char_len)++;break;
+        case 'l':
+          c = unicode::UNICODE_Zl; (*char_len)++;break;
+        case 'p':
+          c = unicode::UNICODE_Zp; (*char_len)++;break;
+        default:
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PZ_CONSTRUCT)) );
+        }
+      }break;
+      case 'S':
+      {
+        switch(pattern[(*char_len)+1])
+        {
+        case '}':
+          c = unicode::UNICODE_Sc + 50;break;
+        case 'm':
+          c = unicode::UNICODE_Sm; (*char_len)++;break;
+        case 'c':
+          c = unicode::UNICODE_Sc; (*char_len)++;break;
+        case 'k':
+          c = unicode::UNICODE_Sk; (*char_len)++;break;
+        case 'o':
+          c = unicode::UNICODE_So; (*char_len)++;break;
+        default:
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PS_CONSTRUCT)) );
+        }
+      }break;
+      case 'C':
+      {
+        switch(pattern[(*char_len)+1])
+        {
+        case '}':
+          c = unicode::UNICODE_Cc + 50;break;
+        case 'c':
+          c = unicode::UNICODE_Cc; (*char_len)++;break;
+        case 'f':
+          c = unicode::UNICODE_Cf; (*char_len)++;break;
+        case 'o':
+          c = unicode::UNICODE_Co; (*char_len)++;break;
+        case 'n':
+          c = unicode::UNICODE_Cn; (*char_len)++;break;
+        default:
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PC_CONSTRUCT)) );
+        }
+      }break;
+      }//end switch
+      if(c)
+      {
+        if(pattern[(*char_len) + 1] != '}')
+            throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_BROKEN_P_CONSTRUCT)) );
+        (*char_len)++;
+        (*char_len)++;
+        return new CRegexXQuery_multicharP(current_regex, c, is_reverse);
+      }
+      if(pattern[*char_len] == 'I')
+      {
+        if(pattern[(*char_len)+1] == 's')//IsBlock
+        {
+          *multichar_type = CHARGROUP_FLAGS_MULTICHAR_Is;
+          (*char_len) += 2;
+          zstring block_name;
+          char tempc = pattern[(*char_len)];
+          while(tempc && (tempc != '}'))
+          {
+            if(!myisletterAZ(tempc) && !myisdigit(tempc) && (tempc != '-'))
+              throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_BROKEN_PIs_CONSTRUCT)) );
+            block_name.append(1, tempc);
+            (*char_len)++;
+            tempc = pattern[(*char_len)];
+          }
+          if(!tempc)
+            throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_BROKEN_PIs_CONSTRUCT)) );
+          //search for the block name
+          int i;
+          int nr_blocks = sizeof(block_escape)/sizeof(CRegexXQuery_parser::block_escape_t);
+          for(i=0;i<nr_blocks;i++)
+          {
+            if(compare_ascii_i(block_name.c_str(), block_escape[i].group_name))
+            {
+              c = i;
+              break;
+            }
+          }
+          if(i==nr_blocks)
+            throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_PIs_CONSTRUCT)) );
           (*char_len)++;
-      }
-      break;
+          return new CRegexXQuery_multicharIs(current_regex, i, is_reverse);
+        }
+        else
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_BROKEN_PIs_CONSTRUCT)) );
+      }
+      else
+      {
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_BROKEN_P_CONSTRUCT)) );
+      }
+      break;//unreachable
+    }//end case 'p'
       //multiCharEsc
     case 's':
     case 'S':
@@ -240,40 +569,104 @@
     case 'D':
     case 'w':
     case 'W':
-      *is_multichar = true;
+       *multichar_type = CHARGROUP_FLAGS_MULTICHAR_OTHER;
        c = pattern[*char_len];
-       break;
-    }
-    break;
-  }
-  case '#':///might be #xXX
-  {
-    if((pattern[*char_len+1] == 'x') &&
-      myishex(pattern[*char_len+2]) && myishex(pattern[*char_len+3]))
-    {
-      c = (myishex(pattern[*char_len+2])-1)<<4 | (myishex(pattern[*char_len+3])-1);
-      *char_len += 3;
-      break;
-    }
-  }
+       (*char_len)++;
+       return new CRegexXQuery_multicharOther(current_regex, c);
+    case 'u'://unicode codepoint \uXXXX
+    {
+      unicode::code_point utf8c = 0;
+      (*char_len)++;
+      for(int i=0;i<4;i++)
+      {
+        char hex = myishex(pattern[*char_len]);
+        if(!hex)
+        {
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_UNICODE_CODEPOINT_u)) );
+        }
+        utf8c <<= 4;
+        utf8c |= (hex-1) & 0x0f;
+        (*char_len)++;
+      }
+      return create_charmatch(utf8c, NULL, 0, multichar_type);
+    }
+    case 'U'://unicode codepoint \UXXXXXXXX
+    {
+      unicode::code_point utf8c = 0;
+      (*char_len)++;
+      for(int i=0;i<8;i++)
+      {
+        char hex = myishex(pattern[*char_len]);
+        if(!hex)
+        {
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_UNICODE_CODEPOINT_u)) );
+        }
+        utf8c <<= 4;
+        utf8c |= (hex-1) & 0x0f;
+        (*char_len)++;
+      }
+      return create_charmatch(utf8c, NULL, 0, multichar_type);
+    }
+    default:
+      throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_UNKNOWN_ESC_CHAR)) );
+    }
+    assert(false);
+    break;//unreachable
+  }//end case '\'
   default:
-     c = pattern[*char_len];
-     break;
-  }
-
-  (*char_len)++;
-  return c;
-}
-
-
-
-IRegexAtom* CRegexAscii_parser::read_atom(const char *pattern, int *atom_len)
+  {
+    const char *temp_pattern = pattern;
+    unicode::code_point utf8c = utf8::next_char(temp_pattern);
+    (*char_len) = temp_pattern - pattern;
+    return create_charmatch(utf8c, pattern, *char_len, multichar_type);
+  }
+  }
+  return NULL;
+}
+
+CRegexXQuery_charmatch *CRegexXQuery_parser::create_charmatch(unicode::code_point utf8c,
+                                                              const char *pattern, int utf8len,
+                                                              enum CHARGROUP_t *multichar_type)
+{
+  if(utf8c <= 0x7F)
+  {
+    *multichar_type = CHARGROUP_FLAGS_ONECHAR_ASCII;
+    if(flags & REGEX_ASCII_CASE_INSENSITIVE)
+      return new CRegexXQuery_char_ascii_i(current_regex, (char)utf8c);
+    else
+      return new CRegexXQuery_char_ascii(current_regex, (char)utf8c);
+  }
+  else
+  {
+    *multichar_type = CHARGROUP_FLAGS_ONECHAR_UNICODE;
+    if(flags & REGEX_ASCII_CASE_INSENSITIVE)
+      return new CRegexXQuery_char_unicode_i(current_regex, utf8c);
+    else
+    {
+      if(pattern)
+        return new CRegexXQuery_char_unicode(current_regex, pattern, utf8len);
+      else
+        return new CRegexXQuery_char_unicode_cp(current_regex, utf8c);
+    }
+  }
+}
+
+IRegexAtom* CRegexXQuery_parser::read_atom(const char *pattern, int *atom_len)
 {
   *atom_len = 0;
-  char  c;
-  bool is_end_line = false;
-  c = pattern[*atom_len];
-  if((!(flags & REGEX_ASCII_LITERAL)) && (c == '\\'))
+  if(flags & REGEX_ASCII_LITERAL)
+  {
+    unicode::code_point  utf8c;
+    //bool is_end_line = false;
+    const char *temp_pattern = pattern;
+    utf8c = utf8::next_char(temp_pattern);
+    *atom_len = temp_pattern - pattern;
+    enum CHARGROUP_t multichar_type;
+    return create_charmatch(utf8c, pattern, *atom_len, &multichar_type);
+  }
+
+  char c = *pattern;
+  if(c == '\\')
   {
     //check for back reference
     if(myisdigit(pattern[(*atom_len)+1]))
@@ -281,13 +674,13 @@
       (*atom_len)++;
       if(pattern[*atom_len] == '0')
       {
-        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(U_REGEX_INVALID_BACK_REF)) );
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_BACK_REF), 0, current_regex->subregex.size()) );
       }
       unsigned int backref = pattern[*atom_len] - '0';
       if((backref > current_regex->subregex.size()) ||
         (current_regex->subregex.at(backref-1)->flags != 0))
       {
-        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(U_REGEX_INVALID_BACK_REF)) );
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_BACK_REF), backref, current_regex->subregex.size()) );
       }
       while(current_regex->subregex.size() >= backref*10)
       {
@@ -303,70 +696,86 @@
             break;
         }
       }
-      return new CRegexAscii_backref(current_regex, backref);
+      (*atom_len)++;
+      return new CRegexXQuery_backref(current_regex, backref);
     }
   }
+  if(c == '^')
+  {
+    (*atom_len)++;
+    return new CRegexXQuery_pinstart(current_regex);
+  }
+  if((c == '}') || (c == '{') || (c == '?') || (c == '*') || (c == '+') || (c == '|'))
+  {
+    throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_ATOM_CHAR), c) );
+  }
   switch(c)
   {
   case '[':
   {
-    if(!(flags & REGEX_ASCII_LITERAL))
-    {
-      (*atom_len)++;
-      CRegexAscii_chargroup *chargroup = NULL;
-      int chargroup_len;
-      chargroup = readchargroup(pattern+*atom_len, &chargroup_len);
-      *atom_len += chargroup_len;
-      return chargroup;
-    }
+    (*atom_len)++;
+    CRegexXQuery_chargroup *chargroup = NULL;
+    int chargroup_len;
+    chargroup = readchargroup(pattern+*atom_len, &chargroup_len);
+    *atom_len += chargroup_len;
+    return chargroup;
   }
   case '.'://WildCharEsc
   {
-    if(!(flags & REGEX_ASCII_LITERAL))
-    {
-      CRegexAscii_wildchar  *wildchar = new CRegexAscii_wildchar(current_regex);
-      (*atom_len)++;
-      return wildchar;
-    }
+    (*atom_len)++;
+    return new CRegexXQuery_wildchar(current_regex);
   }
   case '('://begin an embedded reg exp
   {  
-    if(!(flags & REGEX_ASCII_LITERAL))
-    {
-      (*atom_len)++;
-      CRegexAscii_regex *emb_regex = NULL;
-      int   regex_len;
-      emb_regex = parse_regexp(pattern + *atom_len, &regex_len);
-      *atom_len += regex_len;
-      return emb_regex;
-    }
+    (*atom_len)++;
+    CRegexXQuery_regex *emb_regex = NULL;
+    int   regex_len;
+    emb_regex = parse_regexp(pattern + *atom_len, &regex_len);
+    *atom_len += regex_len;
+    return emb_regex;
   }
   case '$'://end line
-    if(!(flags & REGEX_ASCII_LITERAL))
-    {
-      is_end_line = true;
-    }
+    //is_end_line = true;
+    (*atom_len)++;
+    return new CRegexXQuery_endline(current_regex);
   default:
   {  
-    char  c;
+    //char  c;
+    CRegexXQuery_charmatch *charmatch = NULL;
     int   c_len;
-    bool  is_multichar = false;
-    if(!(flags & REGEX_ASCII_LITERAL))
-      c = readChar(pattern+*atom_len, &c_len, &is_multichar);
-    else
+    CHARGROUP_t   multichar_type = CHARGROUP_NO_MULTICHAR;
+    *atom_len = 0;
+    while(pattern[*atom_len])
     {
-      c = pattern[*atom_len];
-      c_len = 1;
+      charmatch = readChar(pattern+*atom_len, &c_len, &multichar_type);
+      *atom_len += c_len;
+      if((flags & REGEX_ASCII_NO_WHITESPACE) && (multichar_type == CHARGROUP_FLAGS_ONECHAR_ASCII))
+      {
+        char c = (char)charmatch->get_c();
+        if((c == ' ') || (c == '\t') || (c == '\r') || (c == '\n'))
+        {
+          //ignore this whitespace
+          delete charmatch;
+          continue;
+        }
+        else
+          break;
+      }
+      else
+        break;
     }
-    CRegexAscii_chargroup *chargroup = new CRegexAscii_chargroup(current_regex);
-    if(is_multichar)
-      chargroup->addMultiChar(c);
+    /*
+    std::auto_ptr<CRegexXQuery_chargroup> chargroup(new CRegexXQuery_chargroup(current_regex));
+    if(multichar_type)
+      chargroup->addMultiChar(c, multichar_type);
     else if(is_end_line)
       chargroup->addEndLine();
     else
-      chargroup->addCharRange(c, c);
+      chargroup->addOneChar(c);
     *atom_len += c_len;
-    return chargroup;
+    return chargroup.release();
+    */
+    return charmatch;
   }
   }
 }
@@ -374,81 +783,119 @@
 //read until ']'
 //posCharGroup  ::=   ( charRange | charClassEsc )+  
 //charRange     ::=    seRange | XmlCharIncDash
-CRegexAscii_chargroup* CRegexAscii_parser::readchargroup(const char *pattern, int *chargroup_len)
+CRegexXQuery_chargroup* CRegexXQuery_parser::readchargroup(const char *pattern, int *chargroup_len)
 {
-  CRegexAscii_chargroup *chargroup = NULL;
+  std::auto_ptr<CRegexXQuery_chargroup> chargroup;
   *chargroup_len = 0;
   if(pattern[*chargroup_len] == '^')//negative group
   {
     (*chargroup_len)++;
-    chargroup = new CRegexAscii_negchargroup(current_regex);
+    chargroup.reset(new CRegexXQuery_negchargroup(current_regex));
   }
   else
-    chargroup = new CRegexAscii_chargroup(current_regex);
+    chargroup.reset(new CRegexXQuery_chargroup(current_regex));
   while(pattern[*chargroup_len] && (pattern[*chargroup_len]!=']'))
   {
-    char  c1, c2;
-    bool  is_multichar;
+    //char  c1, c2;
+    CHARGROUP_t  multichar_type = CHARGROUP_NO_MULTICHAR;
     int   c1_len;
-    c1 = pattern[*chargroup_len];
-    c2 = pattern[*chargroup_len+1];
-    if((c1 == '-') && (c2 == '['))//charClassSub
+    if((pattern[*chargroup_len] == '-') && (pattern[(*chargroup_len)+1] == '['))//charClassSub
     {
       int classsub_len;
-      CRegexAscii_chargroup *classsub = readchargroup(pattern + *chargroup_len+1 + 1, &classsub_len);
+      CRegexXQuery_chargroup *classsub = readchargroup(pattern + (*chargroup_len)+1 + 1, &classsub_len);
       if(!classsub)
       {
-        delete chargroup;
-        return NULL;
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_SUBCLASS)) );
       }
       chargroup->addClassSub(classsub);
       *chargroup_len += 2 + classsub_len + 1;
       if(pattern[*chargroup_len-1] != ']')
       {
-        delete chargroup;
-        return NULL;
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_INVALID_USE_OF_SUBCLASS)) );
       }
-      return chargroup;
+      return chargroup.release();
     }
 
-    c1 = readChar(pattern+*chargroup_len, &c1_len, &is_multichar);
-    if(is_multichar)//first char is multichar
+    std::unique_ptr<CRegexXQuery_charmatch> charmatch(readChar(pattern+*chargroup_len, &c1_len, &multichar_type));
+    if((multichar_type == CHARGROUP_FLAGS_MULTICHAR_p) ||
+      (multichar_type == CHARGROUP_FLAGS_MULTICHAR_Is) ||
+      (multichar_type == CHARGROUP_FLAGS_MULTICHAR_OTHER))//first char is multichar
     {
-      chargroup->addMultiChar(c1);
+      if((pattern[*chargroup_len+c1_len] == '-') &&///should not be a range
+        (pattern[*chargroup_len+c1_len+1] != ']'))
+      {
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_MULTICHAR_IN_CHAR_RANGE)) );
+      }
+      //chargroup->addMultiChar(c1, multichar_type);
+      chargroup->addCharMatch(charmatch.release());
       *chargroup_len += c1_len;
       continue;
     }
-    if(pattern[*chargroup_len+c1_len] == '-')///might be a range
+    (*chargroup_len) += c1_len;
+    if(pattern[*chargroup_len] == '-')///might be a range
     {
-      if(pattern[*chargroup_len+c1_len+1] == ']')//no range, just the last char is '-'
+      if(pattern[(*chargroup_len)+1] == ']')//no range, just the last char is '-'
       {
-        chargroup->addCharRange(c1, c1);
-        chargroup->addCharRange('-', '-');
-        *chargroup_len += c1_len + 1;
+        //chargroup->addOneChar(c1);
+        //chargroup->addOneChar('-');
+        chargroup->addCharMatch(charmatch.release());
+        chargroup->addCharMatch(new CRegexXQuery_char_ascii(current_regex, '-'));
+        (*chargroup_len)++;
         continue;
       }
-      else
+      else if(pattern[(*chargroup_len)+1] != '[')
       {
         //it is a range
-        char c3;
-        int  c3_len;
-        c3 = readChar(pattern+*chargroup_len+c1_len+1, &c3_len, &is_multichar);
-        if(is_multichar)
-          return NULL;//error
-        chargroup->addCharRange(c1, c3);
-        *chargroup_len += c1_len + 1 + c3_len;
+        (*chargroup_len)++;
+        std::unique_ptr<CRegexXQuery_charmatch>  charmatch2;
+        CHARGROUP_t  multichar_type2 = CHARGROUP_NO_MULTICHAR;
+        int  c2_len;
+        charmatch2.reset(readChar(pattern+(*chargroup_len), &c2_len, &multichar_type2));
+        if((multichar_type2 != CHARGROUP_FLAGS_ONECHAR_ASCII) &&
+          (multichar_type2 != CHARGROUP_FLAGS_ONECHAR_ASCII))//second char in range is multichar
+        {
+          throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_MULTICHAR_IN_CHAR_RANGE)) );
+        }
+        //chargroup->addCharRange(c1, c3);
+        if((multichar_type == CHARGROUP_FLAGS_ONECHAR_ASCII) && (multichar_type2 == CHARGROUP_FLAGS_ONECHAR_ASCII))
+        {
+          if(flags & REGEX_ASCII_CASE_INSENSITIVE)
+            chargroup->addCharMatch(new CRegexXQuery_char_range_ascii_i(current_regex, 
+                                                                    (char)charmatch->get_c(),
+                                                                    (char)charmatch2->get_c()));
+          else
+            chargroup->addCharMatch(new CRegexXQuery_char_range_ascii(current_regex, 
+                                                                    (char)charmatch->get_c(),
+                                                                    (char)charmatch2->get_c()));
+        }
+        else
+        {
+          if(flags & REGEX_ASCII_CASE_INSENSITIVE)
+            chargroup->addCharMatch(new CRegexXQuery_char_range_unicode_i(current_regex, 
+                                                                    charmatch->get_c(),
+                                                                    charmatch2->get_c()));
+          else
+            chargroup->addCharMatch(new CRegexXQuery_char_range_unicode(current_regex, 
+                                                                    charmatch->get_c(),
+                                                                    charmatch2->get_c()));
+        }
+        *chargroup_len += c2_len;
         continue;
       }
     }
-    chargroup->addCharRange(c1, c1);
-    *chargroup_len += c1_len;
+    //chargroup->addOneChar(c1);
+    chargroup->addCharMatch(charmatch.release());
   }
   if(pattern[*chargroup_len])
     (*chargroup_len)++;
-  return chargroup;
+  else
+  {
+    throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_MISSING_CLOSE_BRACKET)) );
+  }
+  return chargroup.release();
 }
 
-void CRegexAscii_parser::read_quantifier(CRegexAscii_piece *piece,
+void CRegexXQuery_parser::read_quantifier(CRegexXQuery_piece *piece,
                                          const char *pattern, int *quantif_len)
 {
   *quantif_len = 0;
@@ -496,6 +943,10 @@
         max = max*10 + pattern[*quantif_len] - '0';
         (*quantif_len)++;
       }
+      if(max < min)
+      {
+        throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(pattern, ZED(REGEX_MAX_LT_MIN)) );
+      }
       piece->set_quantifier_min_max(min, max, true);
     }
     while(pattern[*quantif_len] && (pattern[*quantif_len] != '}'))
@@ -524,23 +975,25 @@
 ///Constructors and destructors and internal functions
 ////////////////////////////
 
-CRegexAscii_regex::CRegexAscii_regex(CRegexAscii_regex *topregex) : IRegexAtom(topregex?topregex:this)
+CRegexXQuery_regex::CRegexXQuery_regex(CRegexXQuery_regex *topregex) : IRegexAtom(topregex?topregex:this)
 {
   matched_source = NULL;
   matched_len = 0;
+//  backup_matched_source = NULL;
+//  backup_matched_len = 0;
   flags = 128;//set to 0 after initialization
 }
 
-CRegexAscii_regex::~CRegexAscii_regex()
+CRegexXQuery_regex::~CRegexXQuery_regex()
 {
-  std::list<CRegexAscii_branch*>::iterator  branch_it;
+  std::list<CRegexXQuery_branch*>::iterator  branch_it;
 
   for(branch_it = branch_list.begin(); branch_it != branch_list.end(); branch_it++)
   {
     delete (*branch_it);
   }
 /*
-  std::vector<CRegexAscii_regex*>::iterator   subregex_it;
+  std::vector<CRegexXQuery_regex*>::iterator   subregex_it;
   for(subregex_it = subregex.begin(); subregex_it != subregex.end(); subregex_it++)
   {
     delete (*subregex_it);
@@ -548,25 +1001,18 @@
 */
 }
 
-bool CRegexAscii_regex::set_align_begin(bool align_begin)
-{
-  bool prev_align = this->align_begin;
-  this->align_begin = align_begin;
-  return prev_align;
-}
-
-void CRegexAscii_regex::add_branch(CRegexAscii_branch *branch)
+void CRegexXQuery_regex::add_branch(CRegexXQuery_branch *branch)
 {
   branch_list.push_back(branch);
 }
 
-bool  CRegexAscii_regex::get_indexed_match(int index, 
+bool  CRegexXQuery_regex::get_indexed_match(int index, 
                                            const char **matched_source, 
                                            int *matched_len)
 {
   if(!index || index > (int)subregex.size())
     return false;
-  CRegexAscii_regex *subr = subregex[index-1];
+  CRegexXQuery_regex *subr = subregex[index-1];
   *matched_source = subr->matched_source;
   if(!*matched_source)
     return false;
@@ -574,145 +1020,209 @@
   return true;
 }
 
-unsigned int CRegexAscii_regex::get_indexed_regex_count()
+unsigned int CRegexXQuery_regex::get_indexed_regex_count()
 {
   return subregex.size();
 }
 
-CRegexAscii_branch::CRegexAscii_branch(CRegexAscii_regex* regex) :
-      IRegexMatcher(regex)
+CRegexXQuery_branch::CRegexXQuery_branch(CRegexXQuery_regex* regex) 
+      //:
+      //IRegexMatcher(regex)
 {
 }
 
-CRegexAscii_branch::~CRegexAscii_branch()
+CRegexXQuery_branch::~CRegexXQuery_branch()
 {
-  std::list<CRegexAscii_piece*>::iterator  piece_it;
+  std::list<RegexAscii_pieceinfo>::iterator  piece_it;
 
   for(piece_it = piece_list.begin(); piece_it != piece_list.end(); piece_it++)
   {
-    delete (*piece_it);
+    delete (*piece_it).piece;
   }
 }
 
-void CRegexAscii_branch::add_piece(CRegexAscii_piece *piece)
+void CRegexXQuery_branch::add_piece(CRegexXQuery_piece *piece)
 {
   piece_list.push_back(piece);
 }
 
-CRegexAscii_piece::CRegexAscii_piece()
+CRegexXQuery_piece::CRegexXQuery_piece()
 {
+  atom = NULL;
+  regex_atom = NULL;
 }
 
-CRegexAscii_piece::~CRegexAscii_piece()
+CRegexXQuery_piece::~CRegexXQuery_piece()
 {
   delete atom;
 }
 
-void CRegexAscii_piece::set_atom(IRegexAtom *atom)
+void CRegexXQuery_piece::set_atom(IRegexAtom *atom)
 {
   this->atom = atom;
+  this->regex_atom = dynamic_cast<CRegexXQuery_regex*>(atom);
 }
 
-void CRegexAscii_piece::set_quantifier_min_max(int min, int max, bool strict_max)
+void CRegexXQuery_piece::set_quantifier_min_max(int min, int max, bool strict_max)
 {
   this->min = min;
   this->max = max;
   this->strict_max = strict_max;
 }
-void CRegexAscii_piece::set_is_reluctant(bool is_reluctant)
+void CRegexXQuery_piece::set_is_reluctant(bool is_reluctant)
 {
   this->is_reluctant = is_reluctant;
 }
-void CRegexAscii_piece::get_quantifier(int *min, int *max, bool *strict_max)
+void CRegexXQuery_piece::get_quantifier(int *min, int *max, bool *strict_max)
 {
   *min = this->min;
   *max = this->max;
   *strict_max = this->strict_max;
 }
-bool CRegexAscii_piece::get_is_reluctant()
+bool CRegexXQuery_piece::get_is_reluctant()
 {
+  if(atom->regex_intern->flags & REGEX_ASCII_MINIMAL_MATCH)
+    return true;
   return is_reluctant;
 }
 
 
-CRegexAscii_chargroup::CRegexAscii_chargroup(CRegexAscii_regex* regex) :
+CRegexXQuery_charmatch::CRegexXQuery_charmatch(CRegexXQuery_regex* regex) :
+    IRegexAtom(regex)
+{
+}
+CRegexXQuery_multicharP::CRegexXQuery_multicharP(CRegexXQuery_regex* regex, char type, bool is_reverse) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->multichar_type = type; this->is_reverse = is_reverse;
+}
+CRegexXQuery_multicharIs::CRegexXQuery_multicharIs(CRegexXQuery_regex* regex, int block_index, bool is_reverse) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->block_index = block_index; this->is_reverse = is_reverse;
+}
+CRegexXQuery_multicharOther::CRegexXQuery_multicharOther(CRegexXQuery_regex* regex, char type) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->multichar_type = type;
+}
+CRegexXQuery_char_ascii::CRegexXQuery_char_ascii(CRegexXQuery_regex* regex, char c) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->c = c;
+}
+CRegexXQuery_char_ascii_i::CRegexXQuery_char_ascii_i(CRegexXQuery_regex* regex, char c) :
+    CRegexXQuery_char_ascii(regex, toupper(c))
+{
+}
+CRegexXQuery_char_range_ascii::CRegexXQuery_char_range_ascii(CRegexXQuery_regex* regex, char c1, char c2) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->c1 = c1; this->c2 = c2;
+}
+CRegexXQuery_char_range_ascii_i::CRegexXQuery_char_range_ascii_i(CRegexXQuery_regex* regex, char c1, char c2) :
+    CRegexXQuery_char_range_ascii(regex, toupper(c1), toupper(c2))
+{
+}
+CRegexXQuery_char_unicode::CRegexXQuery_char_unicode(CRegexXQuery_regex* regex, const char *source, int len) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->len = len;
+  memcpy(c, source, len);
+}
+CRegexXQuery_char_unicode_cp::CRegexXQuery_char_unicode_cp(CRegexXQuery_regex* regex, unicode::code_point c) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->c = c;
+}
+CRegexXQuery_char_unicode_i::CRegexXQuery_char_unicode_i(CRegexXQuery_regex* regex, unicode::code_point c) :
+    CRegexXQuery_char_unicode_cp(regex, unicode::to_upper(c))
+{
+}
+CRegexXQuery_char_range_unicode::CRegexXQuery_char_range_unicode(CRegexXQuery_regex* regex, unicode::code_point c1, unicode::code_point c2) :
+    CRegexXQuery_charmatch(regex)
+{
+  this->c1 = c1; this->c2 = c2;
+}
+CRegexXQuery_char_range_unicode_i::CRegexXQuery_char_range_unicode_i(CRegexXQuery_regex* regex, unicode::code_point c1, unicode::code_point c2) :
+    CRegexXQuery_char_range_unicode(regex, unicode::to_upper(c1), unicode::to_upper(c2))
+{
+}
+CRegexXQuery_endline::CRegexXQuery_endline(CRegexXQuery_regex* regex) :
+    CRegexXQuery_charmatch(regex)
+{
+}
+
+unicode::code_point CRegexXQuery_char_unicode::get_c()
+{
+  const char *temp_c = (const char*)c;
+  return utf8::next_char(temp_c);
+}
+
+
+CRegexXQuery_chargroup::CRegexXQuery_chargroup(CRegexXQuery_regex* regex) :
     IRegexAtom(regex)
 {
   classsub = NULL;
 }
 
-CRegexAscii_chargroup::~CRegexAscii_chargroup()
+CRegexXQuery_chargroup::~CRegexXQuery_chargroup()
 {
   delete classsub;
-}
-
-void CRegexAscii_chargroup::addMultiChar(char c)
-{
-  chargroup_t cgt;
-  cgt.flags = CHARGROUP_FLAGS_MULTICHAR;
-  cgt.c1 = c;
-  cgt.c2 = 0;
-  chargroup_list.push_back(cgt);
-}
-
-void CRegexAscii_chargroup::addEndLine()
-{
-  chargroup_t cgt;
-  cgt.flags = CHARGROUP_FLAGS_ENDLINE;
-  cgt.c1 = '$';
-  cgt.c2 = 0;
-  chargroup_list.push_back(cgt);
-}
-
-void CRegexAscii_chargroup::addCharRange(char c1, char c2)
-{
-  chargroup_t cgt;
-  cgt.flags = 0;
-  cgt.c1 = c1;
-  cgt.c2 = c2;
-  chargroup_list.push_back(cgt);
-}
-
-void CRegexAscii_chargroup::addClassSub(CRegexAscii_chargroup* classsub)
+  std::list<CRegexXQuery_charmatch* >::iterator charmatch_it;
+  for(charmatch_it=chargroup_list.begin(); charmatch_it != chargroup_list.end(); charmatch_it++)
+    delete (*charmatch_it);
+}
+
+void CRegexXQuery_chargroup::addCharMatch(CRegexXQuery_charmatch *charmatch)
+{
+  chargroup_list.push_back(charmatch);
+}
+void CRegexXQuery_chargroup::addClassSub(CRegexXQuery_chargroup* classsub)
 {
   this->classsub = classsub;
 }
 
-CRegexAscii_negchargroup::CRegexAscii_negchargroup(CRegexAscii_regex* regex) :
-  CRegexAscii_chargroup(regex)
-{
-}
-
-CRegexAscii_negchargroup::~CRegexAscii_negchargroup()
-{
-}
-
-CRegexAscii_wildchar::CRegexAscii_wildchar(CRegexAscii_regex* regex) :
+CRegexXQuery_negchargroup::CRegexXQuery_negchargroup(CRegexXQuery_regex* regex) :
+  CRegexXQuery_chargroup(regex)
+{
+}
+
+CRegexXQuery_negchargroup::~CRegexXQuery_negchargroup()
+{
+}
+
+CRegexXQuery_wildchar::CRegexXQuery_wildchar(CRegexXQuery_regex* regex) :
       IRegexAtom(regex)
 {
 }
 
-CRegexAscii_wildchar::~CRegexAscii_wildchar()
+CRegexXQuery_wildchar::~CRegexXQuery_wildchar()
 {
 }
 
-CRegexAscii_backref::CRegexAscii_backref(CRegexAscii_regex* regex, unsigned int backref_) :
+CRegexXQuery_backref::CRegexXQuery_backref(CRegexXQuery_regex* regex, unsigned int backref_) :
       IRegexAtom(regex),
       backref(backref_)
 {
 }
 
-CRegexAscii_backref::~CRegexAscii_backref()
-{
-}
-
-CRegexAscii_parser::CRegexAscii_parser()
+CRegexXQuery_backref::~CRegexXQuery_backref()
+{
+}
+
+CRegexXQuery_pinstart::CRegexXQuery_pinstart(CRegexXQuery_regex* regex):
+      IRegexAtom(regex)
+{
+}
+
+CRegexXQuery_parser::CRegexXQuery_parser()
 {
   current_regex = NULL;
   regex_depth = 0;
 }
 
-CRegexAscii_parser::~CRegexAscii_parser()
+CRegexXQuery_parser::~CRegexXQuery_parser()
 {
 }
 
@@ -720,9 +1230,68 @@
 //////////////////////////////////////////
 ////Matching the pattern on a string
 /////////////////////////////////////////
+static std::list<RegexAscii_pieceinfo> empty_pieces;//empty list of pieces
+/*
+std::list<RegexAscii_pieceinfo>::iterator  
+IRegexAtom::choose_next_piece(const char *source, int *matched_len, 
+                              std::list<RegexAscii_pieceinfo>::iterator this_piece,
+                              std::list<RegexAscii_pieceinfo>::iterator end_piece)
+{
+  //if this_piece is repetition, repeat until max, then go to next piece
+  int min, max;
+  bool strict_max;
+  while(this_piece != end_piece)
+  {
+    (*this_piece).piece->get_quantifier(&min, &max, &strict_max);
+    if(max <= ((*this_piece).nr_matches))//finished this piece
+    {
+      this_piece++;
+    }
+    else
+      break;
+  }
+  return this_piece;
+}
+*/
+
+bool IRegexAtom::match(const char *source, int *start_from_branch, int *matched_len,
+                  std::list<RegexAscii_pieceinfo>::iterator this_piece,
+                  std::list<RegexAscii_pieceinfo>::iterator end_piece)
+{
+  *start_from_branch = 0;
+  bool retmatch;
+  retmatch = match_internal(source, start_from_branch, matched_len);
+  if(!retmatch)
+    return false;
+
+  if(this_piece == end_piece)
+    return true;
+
+  (*this_piece).nr_matches++;
+  int min,max;
+  bool strict_max;
+  (*this_piece).piece->get_quantifier(&min, &max, &strict_max);
+  std::list<RegexAscii_pieceinfo>::iterator init_piece = this_piece;
+  if(((min == 1) && (max == 1)) || //the simple common case
+    ((*matched_len == 0) && ((*this_piece).nr_matches>=min)))//to avoid infinite loop
+  {
+    this_piece++;
+    if(this_piece == end_piece)
+      return true;
+  }
+  int matched_len2;
+  retmatch = (*this_piece).piece->match_piece(this_piece, end_piece, source + *matched_len, &matched_len2);
+  if(!retmatch)
+  {
+    (*init_piece).nr_matches--;
+    return false;
+  }
+  *matched_len += matched_len2;
+  return true;
+}
 
 //try every position in source to match the pattern
-bool CRegexAscii_regex::match_anywhere(const char *source, unsigned int flags,
+bool CRegexXQuery_regex::match_anywhere(const char *source, unsigned int flags,
                                        int *match_pos, int *matched_len)
 {
   *match_pos = 0;
@@ -730,43 +1299,66 @@
   return match_from(source, flags, match_pos, matched_len);
 }
 
-bool CRegexAscii_regex::match_from(const char *source, unsigned int flags,
+bool CRegexXQuery_regex::match_from(const char *source, unsigned int flags,
                                        int *match_pos, int *matched_len)
 {
   this->flags = flags;
+  this->source_start = source;
   reachedEnd = false;
 
-  std::vector<CRegexAscii_regex*>::iterator regex_it;
+  std::vector<CRegexXQuery_regex*>::iterator regex_it;
   for(regex_it = subregex.begin(); regex_it != subregex.end(); regex_it++)
   {
     (*regex_it)->matched_source = NULL;
   }
-//  if(!source[0])
-//  {
-//    if(branch_list.empty())
-//      return true;
-//    else
-//      return false;
-//  }
-
-  bool  skip_first_match = false;
-  if(*match_pos && align_begin)
-    skip_first_match = true;
+
+  std::vector<std::pair<const char*, int> >  saved_subregex;
+
+  if(*match_pos && (flags & REGEX_ASCII_WHOLE_MATCH))
+    return false;
+
   do
   {
-    if(!skip_first_match)
-    {
-      if(match(source + *match_pos, matched_len))
-        return true;
-    }
-    skip_first_match = false;
-    if(align_begin)
+    int   start_from_branch = 0;
+    int   longest_match = -1;
+    while(1)
+    {
+      if(!match(source + *match_pos, &start_from_branch, matched_len, empty_pieces.begin(), empty_pieces.end()))
+        break;
+      if(longest_match < *matched_len)
+      {
+        longest_match = *matched_len;
+        if(start_from_branch && (flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+          save_subregex_list(saved_subregex);
+      }
+      if(!start_from_branch || !(flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+        break;
+      //else try the other branches to see which is longer
+    }
+    if(longest_match != -1)
+    {
+      *matched_len = longest_match;
+      if(saved_subregex.size())
+        load_subregex_list(saved_subregex);
+      if(flags & REGEX_ASCII_WHOLE_MATCH)
+      {
+        if(!source[*match_pos+*matched_len])
+          return true;
+        if((flags & REGEX_ASCII_MULTILINE) && 
+          ((source[*match_pos+*matched_len] == '\n') || (source[*match_pos+*matched_len] == '\r')))
+          return true;
+        return false;
+      }
+      return true;
+    }
+
+    if(flags & REGEX_ASCII_WHOLE_MATCH)
     {
       if(flags & REGEX_ASCII_MULTILINE)
       {
-        //goto the next line
+        //go to next line
         while(source[*match_pos] && (source[*match_pos] != '\n') && (source[*match_pos] != '\r'))
-          (*match_pos)++;
+          (*match_pos) += myutf8len(source);
         if(source[*match_pos] == '\n')
         {
           (*match_pos)++;
@@ -780,190 +1372,1039 @@
             (*match_pos)++;
         }
         if(!source[*match_pos])
-          return false;
+          break;
         continue;
       }
-      return false;
+      break;
     }
     if(!source[*match_pos])
       break;
-    (*match_pos)++;
+    (*match_pos) += myutf8len(source);
   }
   while(source[*match_pos]);
+//  if(!source[*match_pos])
+//  {
+//    reachedEnd = true;
+//  }
   return false;
 }
 
+void CRegexXQuery_regex::reset_match()
+{
+//  this->backup_matched_source = this->matched_source;
+//  this->backup_matched_len = this->matched_len;
+  this->matched_source = NULL;
+  this->matched_len = 0;
+  std::list<CRegexXQuery_branch*>::iterator  branch_it;
+  for(branch_it = branch_list.begin(); branch_it != branch_list.end(); branch_it++)
+  {
+    (*branch_it)->reset();
+  }
+}
+/*
+void CRegexXQuery_regex::restore_match()
+{
+  this->matched_source = this->backup_matched_source;
+  this->matched_len = this->backup_matched_len;
+  std::list<CRegexXQuery_branch*>::iterator  branch_it;
+  for(branch_it = branch_list.begin(); branch_it != branch_list.end(); branch_it++)
+  {
+    (*branch_it)->restore();
+  }
+}
+*/
 //match any of the branches
-bool CRegexAscii_regex::match(const char *source, int *matched_len)
+bool CRegexXQuery_regex::match(const char *source, int *start_from_branch, int *matched_len,
+                              std::list<RegexAscii_pieceinfo>::iterator next_piece,
+                              std::list<RegexAscii_pieceinfo>::iterator end_piece)
 {
   reachedEnd = false;
-  std::list<CRegexAscii_branch*>::iterator  branch_it;
-
-  for(branch_it = branch_list.begin(); branch_it != branch_list.end(); branch_it++)
-  {
-    if((*branch_it)->match(source, matched_len))
-    {
-      matched_source = source;
-      this->matched_len = *matched_len;
+  if(!(flags & REGEX_ASCII_GROUPING_LEN_WHOLE_PIECE) || 
+    (this->matched_source == NULL) || ((this->matched_source + this->matched_len) != source))
+    this->matched_source = source;
+  *matched_len = 0;
+  std::list<CRegexXQuery_branch*>::iterator  branch_it;
+
+  if(*start_from_branch == 0)
+  {
+    for(branch_it = branch_list.begin(); branch_it != branch_list.end(); branch_it++)
+    {
+      (*branch_it)->reset();
+    }
+  }
+
+  branch_it = branch_list.begin();
+  if(*start_from_branch)
+  {
+    for(int i=0;i<*start_from_branch;i++)
+      branch_it++;
+  }
+  (*start_from_branch)++;
+  for(; branch_it != branch_list.end(); branch_it++,(*start_from_branch)++)
+  {
+    if((*branch_it)->match(source, matched_len, this, next_piece, end_piece))
+    {
+      //matched_source = source;
+      //this->matched_len = *matched_len;
       return true;
     }
   }
-  matched_source = NULL;
-  matched_len = 0;
+  *start_from_branch = 0;
+  if(this->matched_source == source)
+    this->matched_source = NULL;
+  *matched_len = 0;
   return false;
 }
 
+void CRegexXQuery_regex::save_subregex_list(std::vector<std::pair<const char*, int> > &saved_subregex)
+{
+  saved_subregex.resize(0);
+  saved_subregex.reserve(subregex.size());
+  std::vector<CRegexXQuery_regex*>::iterator   it;
+  for(it=subregex.begin(); it != subregex.end(); it++)
+  {
+    saved_subregex.push_back(std::pair<const char*, int>((*it)->matched_source, (*it)->matched_len));
+  }
+}
+
+void CRegexXQuery_regex::load_subregex_list(std::vector<std::pair<const char*, int> > &saved_subregex)
+{
+  std::vector<std::pair<const char*, int> >::iterator   it;
+  std::vector<CRegexXQuery_regex*>::iterator            subit;
+  for(it=saved_subregex.begin(), subit = subregex.begin(); it != saved_subregex.end(); it++, subit++)
+  {
+    (*subit)->matched_source = (*it).first;
+    (*subit)->matched_len = (*it).second;
+  }
+}
+
+void CRegexXQuery_branch::reset()
+{
+  std::list<RegexAscii_pieceinfo>::iterator  piece_it;
+  for(piece_it = piece_list.begin(); piece_it != piece_list.end(); piece_it++)
+  {
+    (*piece_it).piece->atom->reset_match();
+  }
+}
+/*
+void CRegexXQuery_branch::restore()
+{
+  std::list<RegexAscii_pieceinfo>::iterator  piece_it;
+  for(piece_it = piece_list.begin(); piece_it != piece_list.end(); piece_it++)
+  {
+    (*piece_it).piece->atom->restore_match();
+  }
+}
+*/
 //match all the pieces
-bool CRegexAscii_branch::match(const char *source, int *matched_len)
+bool CRegexXQuery_branch::match(const char *source, int *matched_len,
+                              CRegexXQuery_regex* group_regex,
+                              std::list<RegexAscii_pieceinfo>::iterator next_piece,
+                              std::list<RegexAscii_pieceinfo>::iterator end_piece)
 {
-  std::list<CRegexAscii_piece*>::iterator  piece_it;
+  std::list<RegexAscii_pieceinfo>::iterator  piece_it;
 
   piece_it = piece_list.begin(); 
+  //if(piece_it == piece_list.end())
+    //if(!source[0])
+  //    return true;
+    //else
+    //  return false;
   if(piece_it == piece_list.end())
-    if(source[0])
-      return false;
+  {
+    piece_it = next_piece;
+    if(next_piece == end_piece)
+    {
+      group_regex->matched_len = 0;
+      return true;
+    }
+  }
+
+  std::list<RegexAscii_pieceinfo>   temp_pieces(piece_list);
+  temp_pieces.push_back(group_regex);//this will be used to store the group match
+  temp_pieces.insert(temp_pieces.end(), next_piece, end_piece);
+
+  return (*piece_it).piece->match_piece(temp_pieces.begin(), temp_pieces.end(), source, matched_len);
+}
+
+bool CRegexXQuery_piece::match_piece(std::list<RegexAscii_pieceinfo>::iterator piece_it,
+                                    std::list<RegexAscii_pieceinfo>::iterator end_it,
+                                    const char *source, int *matched_len)
+{
+  if((*piece_it).nr_matches < 0)
+  {
+    //special case, store the group match
+    (*piece_it).group_regex->matched_len = source - (*piece_it).group_regex->matched_source;
+    piece_it++;
+    if(piece_it == end_it)
+      return true;
     else
-      return true;
-  if(!(*piece_it)->get_is_reluctant())
-    return match_piece_iter_normal(piece_it, source, matched_len);
+      return (*piece_it).piece->match_piece(piece_it, end_it, source, matched_len);
+  }
+
+  if(!get_is_reluctant())
+    return match_piece_iter_normal(piece_it, end_it, source, matched_len);
   else
-    return match_piece_iter_reluctant(piece_it, source, matched_len);
-}
-
-//match as less as possible
-bool CRegexAscii_branch::match_piece_iter_reluctant(
-                                        std::list<CRegexAscii_piece*>::iterator piece_it,
+    return match_piece_iter_reluctant(piece_it, end_it, source, matched_len);
+}
+
+int CRegexXQuery_piece::choose_another_branch(std::vector<std::pair<int,int> > &match_lens)
+{
+  int i = match_lens.size()-1;
+  i--;
+  while((i >= 0) && (match_lens.at(i).second == 0))
+    i--;
+  if(i < 0)
+    return -1;//no more branches
+  match_lens.resize(i+1);
+  i++;
+  return i;
+}
+
+bool CRegexXQuery_piece::is_regex_atom()
+{
+  return regex_atom != NULL;
+}
+
+//match as less as possible (shortest string)
+bool CRegexXQuery_piece::match_piece_iter_reluctant(
+                                        std::list<RegexAscii_pieceinfo>::iterator piece_it,
+                                        std::list<RegexAscii_pieceinfo>::iterator end_it,
                                         const char *source, int *matched_len)
 {
   *matched_len = 0;
-  if(piece_it == piece_list.end())
+  if(piece_it == end_it)
     return true;
 
   int min, max;
   bool  strict_max;
   //std::vector<int>    match_lens;
-  (*piece_it)->get_quantifier(&min, &max, &strict_max);
-  if(strict_max && (max >= 0))
+  (*piece_it).piece->get_quantifier(&min, &max, &strict_max);
+
+  std::vector<std::pair<const char*, int> >  saved_subregex;
+
+  if(is_regex_atom())
   {
-    int   timeslen;
-    //check if the piece doesn't exceed the max match
-    if((*piece_it)->match_piece_times(source, &timeslen, max+1, NULL))
-      return false;///too many matches
+    //recursive
+    bool retmatch;
+    atom->regex_intern->save_subregex_list(saved_subregex);
+    if((*piece_it).nr_matches >= min)
+    {
+      //go to next piece
+      std::list<RegexAscii_pieceinfo>::iterator next_it = piece_it;
+      next_it++;
+      if(next_it == end_it)
+        return true;
+      retmatch = (*next_it).piece->match_piece(next_it, end_it, source, matched_len);
+      if(retmatch)
+        return true;
+    }
+    if(((max == -1) || ((*piece_it).nr_matches < max)) &&//try further with this piece
+      (((*piece_it).nr_matches < min) || ((*piece_it).nr_matches == 0) || ((*piece_it).piece->regex_atom->matched_len)))//if matched_len is zero, avoid infinite loop
+    {
+      int start_from_branch = 0;
+      int shortest_len = -1;
+      bool branch_saved = false;
+      //try all branches to get the shortest len
+      (*piece_it).nr_matches++;
+      while(atom->match(source, &start_from_branch, matched_len, piece_it, end_it))
+      {
+        if((shortest_len == -1) || (shortest_len > *matched_len))
+        {
+          shortest_len = *matched_len;
+          if(start_from_branch && (atom->regex_intern->flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+          {
+            atom->regex_intern->save_subregex_list(saved_subregex);
+            branch_saved = true;
+          }
+        }
+        if(!start_from_branch || !(atom->regex_intern->flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+          break;
+      }
+      if(shortest_len != -1)
+      {
+        *matched_len = shortest_len;
+        if(branch_saved)
+          atom->regex_intern->load_subregex_list(saved_subregex);
+        return true;
+      }
+      else
+      {
+        (*piece_it).nr_matches--;
+        atom->regex_intern->load_subregex_list(saved_subregex);
+        return false;
+      }
+    }
+    else
+    {
+      atom->regex_intern->load_subregex_list(saved_subregex);
+      return false;
+    }
   }
 
-  int i=min;
-  std::list<CRegexAscii_piece*>::iterator next_it = piece_it;
+  int i=0;
+  int shortest_len = -1;
+  int otherpieces_shortest = -1;
+  int i_shortest = -1;
+  std::list<RegexAscii_pieceinfo>::iterator next_it = piece_it;
+  std::vector<std::pair<int,int> >    match_lens;
   next_it++;
   int pieceslen = 0;
   while(1)
   {
-    if((max > 0) && (i>max))
-      break;
-   int piecelen = 0;
-   if((*piece_it)->match_piece_times(source+pieceslen, &piecelen, !pieceslen ? i : 1, NULL))
-   {
-      pieceslen += piecelen;
+    int piecelen = 0;
+    bool retmatch;
+    retmatch = match_piece_times(source, &piecelen, i < min ? min : i, &match_lens);
+    i = match_lens.size()-1;//number of matches
+    if(i<0)
+      i = 0;
+    if((i>=min))
+    {
+      pieceslen = piecelen;
+      if((shortest_len >= 0) && (shortest_len <= pieceslen))//this branch is longer
+      {//try another branch
+        i = choose_another_branch(match_lens);
+        if(i >= 0)
+          continue;//try another branch
+        else
+          break;
+      }
       int   otherpieces = 0;
-      if((next_it == piece_list.end()) ||
-        ((*next_it)->get_is_reluctant() && match_piece_iter_reluctant(next_it, source+pieceslen, &otherpieces)) ||
-        (!(*next_it)->get_is_reluctant() && match_piece_iter_normal(next_it, source+pieceslen, &otherpieces)))
-      {
-        *matched_len = pieceslen + otherpieces;
-        return true;
-      }
+      if((next_it == end_it) ||
+        (*next_it).piece->match_piece(next_it, end_it, source+pieceslen, &otherpieces)
+        )
+      {
+        if((i == pieceslen) || (match_lens.at(0).second == 0) ||//minimum achieved already, cannot go lower than that
+            !(atom->regex_intern->flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+        {
+          *matched_len = pieceslen + otherpieces;
+          return true;
+        }
+        if((shortest_len < 0) || (shortest_len > pieceslen))
+        {
+          shortest_len = pieceslen;
+          otherpieces_shortest = otherpieces;
+          i_shortest = i;
+          if(match_lens.at(0).second != 0)
+            atom->regex_intern->save_subregex_list(saved_subregex);
+        }
+        i = choose_another_branch(match_lens);
+        if(i >= 0)
+          continue;//try another branch
+        else
+          break;
+      }
+      else
+      {
+        //try further
+        if(retmatch)
+        {
+          i++;
+          if((max < 0) || (i<=max))
+            continue;
+          i--;
+        }
+      }
+    }
+    
+    if(i==0)
+    {
+      break;
     }
     else
-      break;
-    i++;
+    {
+      i = choose_another_branch(match_lens);
+      if(i >= 0)
+        continue;//try another branch
+      else
+        break;
+    }
   }
 
+  if(shortest_len >= 0)
+  {
+    if(strict_max && (max>=0) && (i_shortest > max))
+      return false;
+    *matched_len = shortest_len + otherpieces_shortest;
+    if(saved_subregex.size())
+      atom->regex_intern->load_subregex_list(saved_subregex);
+    return true;
+  }
   return false;
 }
 
 //match as much as possible
-bool CRegexAscii_branch::match_piece_iter_normal(
-                                        std::list<CRegexAscii_piece*>::iterator piece_it,
+bool CRegexXQuery_piece::match_piece_iter_normal(
+                                        std::list<RegexAscii_pieceinfo>::iterator piece_it,
+                                        std::list<RegexAscii_pieceinfo>::iterator end_it,
                                         const char *source, int *matched_len)
 {
   *matched_len = 0;
 
   int min, max;
   bool  strict_max;
-  std::vector<int>    match_lens;
-  (*piece_it)->get_quantifier(&min, &max, &strict_max);
-  int   timeslen;
-  if(strict_max && (max >= 0))
+  std::vector<std::pair<int,int> >    match_lens;
+  (*piece_it).piece->get_quantifier(&min, &max, &strict_max);
+  int   timeslen = 0;
+  std::vector<std::pair<const char*, int> >  saved_subregex;
+
+  if(is_regex_atom())
   {
-    //check if the piece doesn't exceed the max match
-    //if((*piece_it)->match_piece_times(source, &timeslen, max+1, &match_lens))
-    //  return false;///too many matches
-    (*piece_it)->match_piece_times(source, &timeslen, max, &match_lens);
+    //recursive
+    bool retmatch;
+    atom->regex_intern->save_subregex_list(saved_subregex);
+    if(((max == -1) || ((*piece_it).nr_matches < max)) && //try further with this piece
+      (((*piece_it).nr_matches < min) || ((*piece_it).nr_matches == 0) || ((*piece_it).piece->regex_atom->matched_len)))//if matched_len is zero, avoid infinite loop
+    {
+      int start_from_branch = 0;
+      int longest_len = -1;
+      bool branch_saved = false;
+      //try all branches to get the longest len
+      (*piece_it).nr_matches++;
+      while(atom->match(source, &start_from_branch, matched_len, piece_it, end_it))
+      {
+        if((longest_len < *matched_len))
+        {
+          longest_len = *matched_len;
+          if(start_from_branch && (atom->regex_intern->flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+          {
+            atom->regex_intern->save_subregex_list(saved_subregex);
+            branch_saved = true;
+          }
+        }
+        if(!start_from_branch || !(atom->regex_intern->flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+          break;
+      }
+      if(longest_len != -1)
+      {
+        *matched_len = longest_len;
+        if(branch_saved)
+          atom->regex_intern->load_subregex_list(saved_subregex);
+        return true;
+      }
+      else
+      {
+        atom->regex_intern->load_subregex_list(saved_subregex);
+        (*piece_it).nr_matches--;
+      }
+    }
+    if((*piece_it).nr_matches >= min)
+    {
+      //go to next piece
+      std::list<RegexAscii_pieceinfo>::iterator next_it = piece_it;
+      next_it++;
+      if(next_it == end_it)
+        return true;
+      retmatch = (*next_it).piece->match_piece(next_it, end_it, source, matched_len);
+      if(!retmatch)
+        atom->regex_intern->load_subregex_list(saved_subregex);
+      return retmatch;
+    }
+    else
+    {
+    //  regex_atom->restore_match();
+      atom->regex_intern->load_subregex_list(saved_subregex);
+      return false;
+    }
   }
-  else if(!strict_max && (max >= 0))
-    (*piece_it)->match_piece_times(source, &timeslen, max, &match_lens);
-  else
-    (*piece_it)->match_piece_times(source, &timeslen, -1, &match_lens);
 
-  int i;
-  std::list<CRegexAscii_piece*>::iterator next_it = piece_it;
+  int longest_len = -1;
+  int otherpieces_longest = -1;
+  int i_longest = -1;
+  int i = max;
+  std::list<RegexAscii_pieceinfo>::iterator next_it = piece_it;
   next_it++;
-  if(next_it == piece_list.end())
+
+  bool retmatch;
+  while(1)
   {
-    if((int)match_lens.size() > min)
-    {
-      *matched_len = timeslen;
-      return true;
+    retmatch = match_piece_times(source, &timeslen, i, &match_lens);
+    i=match_lens.size()-1;//number of matches
+    if((i>=min))
+    {
+      if(timeslen < longest_len)
+      {//this branch is no use
+        i = choose_another_branch(match_lens);
+        if(i >= 0)
+        {
+          i = max;
+          continue;//try another branch
+        }
+        else
+          break;
+      }
+      //int piecelen = 0;
+      int   otherpieces = 0;
+      if((next_it == end_it) ||
+        (*next_it).piece->match_piece(next_it, end_it, source+timeslen, &otherpieces)
+        )
+      {
+        if(timeslen > longest_len)
+        {
+          longest_len = timeslen;
+          otherpieces_longest = otherpieces;
+          i_longest = i;
+          if(!(atom->regex_intern->flags & REGEX_ASCII_GET_LONGEST_BRANCH))
+          {
+            *matched_len = longest_len + otherpieces_longest;
+            return true;
+          }
+          else
+          {
+            if(match_lens.at(0).second)
+              atom->regex_intern->save_subregex_list(saved_subregex);
+          }
+        }
+      }
+      else
+      {
+        if(!match_lens.at(0).second)
+        {
+          match_lens.resize(match_lens.size()-1);
+          i--;
+          if(i >= 0)
+            continue;//try smaller 
+          else
+            break;
+        }
+        else
+        {
+          i = choose_another_branch(match_lens);
+          if(i >= 0)
+            continue;//try another branch
+          else
+            break;
+        }
+      }
+    }
+    //now try another branch
+    i = choose_another_branch(match_lens);
+    if(i >= 0)
+    {
+      i = max;
+      continue;//try another branch
     }
     else
-      return false;
-  }
-  for(i=match_lens.size()-1; i>=min; i--)
+      break;
+  }//end while
+
+  if(longest_len >= 0)
   {
-    int piecelen = 0;
-    int   otherpieces = 0;
-    if(((*next_it)->get_is_reluctant() && match_piece_iter_reluctant(next_it, source+match_lens[i]+piecelen, &otherpieces)) ||
-      (!(*next_it)->get_is_reluctant() && match_piece_iter_normal(next_it, source+match_lens[i]+piecelen, &otherpieces)))
-    {
-      *matched_len = match_lens[i] + piecelen + otherpieces;
-      return true;
-    }
+    *matched_len = longest_len + otherpieces_longest;
+    if(saved_subregex.size())
+      atom->regex_intern->load_subregex_list(saved_subregex);
+    return true;
   }
 
   return false;
 }
 
-bool CRegexAscii_piece::match_piece_times(const char *source, 
+bool CRegexXQuery_piece::match_piece_times(const char *source, 
                                           int *piecelen, 
                                           int times,
-                                          std::vector<int>    *match_lens)
+                                          std::vector<std::pair<int,int> >    *match_lens)
 {
-  *piecelen = 0;
-  for(int i=0;(times < 0) || (i<times);i++)
-  {
+  int i=0;
+  if(match_lens && match_lens->size())
+  {
+    i = match_lens->size()-1;
+  }
+  if(match_lens && match_lens->size())
+    *piecelen = match_lens->at(match_lens->size()-1).first;
+  else
+    *piecelen = 0;
+  if((times >= 0) && (i>=times))
+    return true;
+  for(;(times < 0) || (i<times);i++)
+  {
+    int   atomlen;
+    int   start_from_branch = 0;
+    if(match_lens && (i<(int)match_lens->size()))
+      start_from_branch = match_lens->at(i).second;
+    bool first_branch = (start_from_branch == 0);
+    if(!atom->match(source+*piecelen, &start_from_branch, &atomlen, empty_pieces.begin(), empty_pieces.end()))
+    {
+      if(match_lens)
+      {
+        if(i >= (int)match_lens->size())
+          match_lens->push_back(std::pair<int,int>(*piecelen, 0));
+        else
+          (*match_lens)[i] = std::pair<int,int>(*piecelen, 0);
+      }
+      return false;
+    }
     if(match_lens)
-      match_lens->push_back(*piecelen);
-    int   atomlen;
-    if(!atom->match(source+*piecelen, &atomlen))
-      return false;
+    {
+      if(i >= (int)match_lens->size())
+        match_lens->push_back(std::pair<int,int>(*piecelen, start_from_branch));
+      else
+        (*match_lens)[i] = std::pair<int,int>(*piecelen, start_from_branch);
+    }
     *piecelen += atomlen;
     if(!atomlen && !source[*piecelen])
     {
-      atom->regex_intern->reachedEnd = true;
+    //  atom->regex_intern->set_reachedEnd(source);
+      break;
+    }
+    if(first_branch && (atomlen == 0))//avoid infinite loop
+    {
       break;
     }
   }
   if(match_lens)
-    match_lens->push_back(*piecelen);
+  {
+  //  if(i >= match_lens->size())
+      match_lens->push_back(std::pair<int,int>(*piecelen, 0));
+  //  else
+  //    (*match_lens)[i] = std::pair<int,int>(*piecelen, 0);
+  }
 
   return true;
 }
 
+bool CRegexXQuery_multicharP::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  bool found = false;
+  const char *temp_source = source;
+  unicode::code_point utf8c = utf8::next_char(temp_source);
+  switch(multichar_type)
+  {
+  case unicode::UNICODE_Ll + 50:
+    if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Ll) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Lm) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Lo) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Lt) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Lu))
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  case unicode::UNICODE_Mc + 50:
+    if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Mn) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Mc) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Me))
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  case unicode::UNICODE_Nd + 50:
+    if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Nd) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Nl) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_No))
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  case unicode::UNICODE_Pc + 50:
+    if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pc) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pd) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Ps) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pe) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pi) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pf) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Po))
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  case unicode::UNICODE_Zl + 50:
+    if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Zs) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Zl) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Zp))
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  case unicode::UNICODE_Sc + 50:
+    if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Sm) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Sc) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Sk) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_So))
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  case unicode::UNICODE_Cc + 50:
+    if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Cc) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Cf) ||
+        unicode::check_codepoint_category(utf8c, unicode::UNICODE_Co))//ignore unicode::UNICODE_Cn
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  default:
+    if(unicode::check_codepoint_category(utf8c, (unicode::category)multichar_type))
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+    break;
+  }
+
+  if(found)
+  {
+    *matched_len = temp_source - source;
+  }
+  return found;
+}
+
+bool CRegexXQuery_multicharIs::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  bool found = false;
+  const char *temp_source = source;
+  unicode::code_point utf8c = utf8::next_char(temp_source);
+  const unicode::code_point *cp = block_escape[block_index].cp;
+  if((utf8c >= cp[0]) && (utf8c <= cp[1]))
+  {
+    if(!is_reverse)
+      found = true;
+  }
+  else if(block_escape[block_index].ext_cp)
+  {
+    cp = block_escape[block_index].ext_cp;
+    while(*cp)
+    {
+      if((utf8c >= cp[0]) && (utf8c <= cp[1]))
+        break;
+      cp += 2;
+    }
+    if(*cp)
+    {
+      if(!is_reverse)
+        found = true;
+    }
+    else
+    {
+      if(is_reverse)
+        found = true;
+    }
+  }
+  else
+  {
+    if(is_reverse)
+      found = true;
+  }
+  if(found)
+  {
+    *matched_len = temp_source - source;
+  }
+  return found;
+}
+
+bool CRegexXQuery_multicharOther::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  bool found = false;
+  bool value_true = true;
+  const char *temp_source = source;
+  unicode::code_point utf8c = utf8::next_char(temp_source);
+  switch(multichar_type)
+  {
+    case 'S':value_true = false;//[^\s]
+    case 's'://[#x20\t\n\r]
+      switch(utf8c)
+      {
+      case '\t':
+      case '\r':
+      case '\n':
+      case ' ':
+        found = true;
+      default:
+        break;
+      }
+      break;
+    case 'I':value_true = false;//[^\i]
+    case 'i'://the set of initial name characters, those matched by Letter | '_' | ':'
+      if((utf8c == '_') ||
+        (utf8c == ':') ||
+        XQCharType::isLetter(utf8c))
+      {
+        found = true;
+      }
+      break;
+    case 'C':value_true = false;//[^\c]
+    case 'c'://the set of name characters, those matched by NameChar
+      if(XQCharType::isNameChar(utf8c))
+      {
+        found = true;
+      }
+      break;
+    case 'D':value_true = false;//[^\d]
+    case 'd':
+      if(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Nd))
+        found = true;
+      break;
+    case 'W':value_true = false;//[^\w]
+    case 'w':
+      found = !(unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pc) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pd) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Ps) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pe) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pi) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Pf) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Po) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Zs) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Zl) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Zp) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Cc) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Cf) ||
+                unicode::check_codepoint_category(utf8c, unicode::UNICODE_Co));//ignore unicode::UNICODE_Cn
+      break;
+    default:
+      throw XQUERY_EXCEPTION( err::FORX0002, ERROR_PARAMS(source, ZED(REGEX_UNIMPLEMENTED)) );
+  }
+  if((found && value_true) || (!found && !value_true))
+  {
+    *matched_len = temp_source - source;
+    return true;
+  }
+  else
+  {
+    return false;
+  }
+}
+
+bool CRegexXQuery_char_ascii::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  if(source[0] == c)
+  {
+    *matched_len = 1;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_ascii_i::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  char  sup = toupper(source[0]);
+  if(sup == c)
+  {
+    *matched_len = 1;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_range_ascii::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  if((source[0] >= c1) && (source[0] <= c2))
+  {
+    *matched_len = 1;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_range_ascii_i::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  char  sup = toupper(source[0]);
+  if((sup >= c1) && (sup <= c2))
+  {
+    *matched_len = 1;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_unicode::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  if(!memcmp(source, c, len))
+  {
+    *matched_len = len;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_unicode_cp::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  const char *temp_source = source;
+  unicode::code_point  utf8c = utf8::next_char(temp_source);
+  if(utf8c == c)
+  {
+    *matched_len = temp_source - source;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_unicode_i::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  const char *temp_source = source;
+  unicode::code_point  sup = unicode::to_upper(utf8::next_char(temp_source));
+  if(sup == c)
+  {
+    *matched_len = temp_source - source;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_range_unicode::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  const char *temp_source = source;
+  unicode::code_point  utf8c = utf8::next_char(temp_source);
+  if((utf8c >= c1) && (utf8c <= c2))
+  {
+    *matched_len = temp_source - source;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_char_range_unicode_i::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  const char *temp_source = source;
+  unicode::code_point  sup = unicode::to_upper(utf8::next_char(temp_source));
+  if((sup >= c1) && (sup <= c2))
+  {
+    *matched_len = temp_source - source;
+    return true;
+  }
+  else
+    return false;
+}
+
+bool CRegexXQuery_endline::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  *matched_len = 0;
+  if(!source[0])
+  {
+  //  regex_intern->reachedEnd = true;
+    return true;
+  }
+  if((source[0] == 0x0A) || ((source[0] == 0x0D) && (source[1] == 0x0A)))
+  {
+    if(regex_intern->get_flags() & REGEX_ASCII_MULTILINE)
+    {
+    //  regex_intern->reachedEnd = true;
+      return true;
+    }
+  }
+  return false;
+}
+
+
 //match any of chargroups
-bool CRegexAscii_chargroup::match(const char *source, int *matched_len)
+bool CRegexXQuery_chargroup::match_internal(const char *source, int *start_from_branch, int *matched_len)
 {
   *matched_len = 0;
-  std::list<chargroup_t>::iterator  cgt_it;
-
+  std::list<CRegexXQuery_charmatch* >::iterator  cgt_it;
+/*
   if(!source[0])
   {
     regex_intern->reachedEnd = true;
@@ -975,113 +2416,21 @@
       return false;
   }
 
-  if(source[0] == 0x0A)
+  if((source[0] == 0x0A) || ((source[0] == 0x0D) && (source[1] == 0x0A)))
   {
     if((regex_intern->flags & REGEX_ASCII_MULTILINE) &&
         (chargroup_list.size() == 1) && (chargroup_list.begin()->flags == CHARGROUP_FLAGS_ENDLINE))
     {
-      *matched_len = 1;
+      // *matched_len = 1;
       return true;
     }
   }
-
+*/
+  //bool found = false;
   for(cgt_it = chargroup_list.begin(); cgt_it != chargroup_list.end(); cgt_it++)
   {
-    if(cgt_it->flags == CHARGROUP_FLAGS_MULTICHAR)
-    {
-      switch(cgt_it->c1)
-      {
-        case 'p'://catEsc
-        case 'P'://complEsc
-          //ignore the prop for now
-          throw XQUERY_EXCEPTION( err::FORX0002 );
-        case 's'://[#x20\t\n\r]
-          switch(source[0])
-          {
-          case '\t':
-          case '\r':
-          case '\n':
-          case ' ':
-            *matched_len = 1;
-            return true;
-          default:
-            return false;
-          }
-        case 'S'://[^\s]
-          switch(source[0])
-          {
-          case 0:
-            regex_intern->reachedEnd = true;
-          case '\t':
-          case '\r':
-          case '\n':
-          case ' ':
-            return false;
-          default:
-            *matched_len = 1;
-            return true;
-          }
-        case 'i'://the set of initial name characters, those matched by Letter | '_' | ':'
-          if((source[0] == '_') ||
-            (source[0] == ':') ||
-            XQCharType::isLetter(source[0]))
-          {
-            *matched_len = 1;
-            return true;
-          }
-          return false;
-        case 'I':
-          if((source[0] == '_') ||
-            (source[0] == ':') ||
-            XQCharType::isLetter(source[0]))
-          {
-            return false;
-          }
-          *matched_len = 1;
-          return true;
-        case 'c'://the set of name characters, those matched by NameChar
-          if(XQCharType::isNameChar(source[0]))
-          {
-            *matched_len = 1;
-            return true;
-          }
-          return false;
-        case 'C':
-          if(XQCharType::isNameChar(source[0]))
-          {
-            return false;
-          }
-          *matched_len = 1;
-          return true;
-        case 'd':
-        case 'D':
-        case 'w':
-        case 'W':
-        default:
-          throw XQUERY_EXCEPTION( err::FORX0002 );
-      }
-      return false;
-    }
-    else if(cgt_it->flags == CHARGROUP_FLAGS_ENDLINE)
-    {
-      return false;
-    }
-    else
-    {
-      if(regex_intern->flags & REGEX_ASCII_CASE_INSENSITIVE)
-      {
-        char  sup = toupper(source[0]);
-        if((sup >= toupper(cgt_it->c1)) &&
-          (sup <= toupper(cgt_it->c2)))
-          break;
-      }
-      else
-      {
-        if((source[0] >= cgt_it->c1) &&
-          (source[0] <= cgt_it->c2))
-          break;
-      }
-    }
+    if((*cgt_it)->match_internal(source, start_from_branch, matched_len))
+      break;
   }
   if(cgt_it == chargroup_list.end())
     return false;
@@ -1089,53 +2438,48 @@
   if(classsub)
   {
     int   classsub_len;
-    if(classsub->match(source, &classsub_len))
+    if(classsub->match_internal(source, NULL, &classsub_len))
       return false;
   }
 
-  *matched_len = 1;
+  //*matched_len = 1;
   return true;
 }
 
-bool CRegexAscii_negchargroup::match(const char *source, int *matched_len)
+bool CRegexXQuery_negchargroup::match_internal(const char *source, int *start_from_branch, int *matched_len)
 {
   if(!source[0])
   {
-    regex_intern->reachedEnd = true;
+    regex_intern->set_reachedEnd(source);
     return false;
   }
-  if(!CRegexAscii_chargroup::match(source, matched_len))
+  if(!CRegexXQuery_chargroup::match_internal(source, start_from_branch, matched_len))
   {
-    *matched_len = 1;
+    *matched_len = myutf8len(source);
     return true;
   }
   return false;
 }
 
-bool CRegexAscii_wildchar::match(const char *source, int *matched_len)
+bool CRegexXQuery_wildchar::match_internal(const char *source, int *start_from_branch, int *matched_len)
 {
   *matched_len = 0;
-  if(source[0])
-  {
-    if((regex_intern->flags & REGEX_ASCII_DOTALL) || 
-      (source[0] != '\n') && (source[0] != '\r'))
-    {
-      *matched_len = 1;
-      return true;
-    }
-    else
-      return false;
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
+  if((regex_intern->flags & REGEX_ASCII_DOTALL) || 
+    ((source[0] != '\n') && (source[0] != '\r')))
+  {
+    *matched_len = myutf8len(source);
+    return true;
   }
   else
-  {
-    if(!source[0])
-      regex_intern->reachedEnd = true;
-    *matched_len = 0;
     return false;
-  }
 }
 
-bool CRegexAscii_backref::match(const char *source, int *matched_len)
+bool CRegexXQuery_backref::match_internal(const char *source, int *start_from_branch, int *matched_len)
 {
   const char *submatch = regex_intern->subregex.at(backref-1)->matched_source;
   if(!submatch)
@@ -1143,15 +2487,42 @@
     *matched_len = 0;
     return true;
   }
+  if(!source[0])
+  {
+    regex_intern->set_reachedEnd(source);
+    return false;
+  }
   *matched_len = regex_intern->subregex.at(backref-1)->matched_len;
-  if(!strncmp(source, submatch, *matched_len))
-  {
-    return true;
-  }
-  *matched_len = 0;
-  return false;
-}
-
-  }//end namespace regex_ascii
+  if(regex_intern->flags & REGEX_ASCII_CASE_INSENSITIVE)
+  {
+    if(compare_unicode_ni(source, submatch, *matched_len)) 
+    {
+      return true;
+    }
+  }
+  else
+  {
+    if(!memcmp(source, submatch, *matched_len))
+    {
+      return true;
+    }
+  }
+  *matched_len = 0;
+  return false;
+}
+
+bool CRegexXQuery_pinstart::match_internal(const char *source, int *start_from_branch, int *matched_len)
+{
+  *matched_len = 0;
+  if(source == regex_intern->source_start)
+    return true;
+  if((regex_intern->flags & REGEX_ASCII_MULTILINE) &&
+    ((source[-1] == '\n') || (source[-1] == '\r')))
+    return true;
+  
+  return false;
+}
+
+  }//end namespace regex_xquery
 }//end namespace zorba
 /* vim:set et sw=2 ts=2: */

=== renamed file 'src/util/regex_ascii.h' => 'src/util/regex_xquery.h'
--- src/util/regex_ascii.h	2011-07-18 14:25:21 +0000
+++ src/util/regex_xquery.h	2012-01-18 18:33:36 +0000
@@ -21,103 +21,142 @@
 #include <vector>
 
 #include <zorba/config.h>
+#include <zorba/internal/unique_ptr.h>
+#include "util/unicode_util.h"
 
 namespace zorba {
-  namespace regex_ascii{
+  namespace regex_xquery{
 
 //matching flags
-#define REGEX_ASCII_CASE_INSENSITIVE    1
-#define REGEX_ASCII_DOTALL              2
-#define REGEX_ASCII_MULTILINE           4
-#define REGEX_ASCII_COMMENTS            8
-#define REGEX_ASCII_LITERAL             16
-
-class CRegexAscii_regex;
-
-class IRegexMatcher
+#define REGEX_ASCII_CASE_INSENSITIVE    1   //i
+#define REGEX_ASCII_DOTALL              2   //s
+#define REGEX_ASCII_MULTILINE           4   //m
+#define REGEX_ASCII_NO_WHITESPACE       8   //x
+#define REGEX_ASCII_LITERAL             16  //q
+
+#define REGEX_ASCII_GET_LONGEST_BRANCH  32   //try all branches and get the longest match (or shortest for reluctant pieces)
+#define REGEX_ASCII_MINIMAL_MATCH       64   //consider all pieces as reluctant
+#define REGEX_ASCII_WHOLE_MATCH         128  //match only all string, like having "^regex$"
+#define REGEX_ASCII_GROUPING_LEN_WHOLE_PIECE    256  //compute the len of a grouping as for the whole piece ( for example (a)+ when matching "aa" and referred as $1 will get string len 2 instead of last 1)
+
+class CRegexXQuery_regex;
+class CRegexXQuery_piece;
+
+struct RegexAscii_pieceinfo
 {
-public:
-  CRegexAscii_regex *regex_intern;
-public:
-  IRegexMatcher(CRegexAscii_regex* regex) : regex_intern(regex) {}
-  virtual ~IRegexMatcher() {}
+  union
+  {
+    CRegexXQuery_piece*  piece;
+    CRegexXQuery_regex*  group_regex;
+  };
+  int nr_matches;
 
-  virtual bool match(const char *source, int *matched_len) = 0;
+  RegexAscii_pieceinfo(CRegexXQuery_piece* piece) {nr_matches=0;this->piece=piece;}
+  RegexAscii_pieceinfo(CRegexXQuery_regex* group_regex) {nr_matches=-1;this->group_regex=group_regex;}
 };
 
-class IRegexAtom : public IRegexMatcher
+
+class IRegexAtom
 {
+protected:
+  friend class CRegexXQuery_piece;
+  CRegexXQuery_regex *regex_intern;
 public:
-  IRegexAtom(CRegexAscii_regex* regex) : IRegexMatcher(regex) {}
+  IRegexAtom(CRegexXQuery_regex* regex)  : regex_intern(regex) {}
   virtual ~IRegexAtom() {}
+
+  virtual bool match(const char *source, int *start_from_branch, int *matched_len,
+                    std::list<RegexAscii_pieceinfo>::iterator next_piece,
+                    std::list<RegexAscii_pieceinfo>::iterator end_piece);
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len) = 0;
+  virtual void reset_match() {}
+//  virtual void restore_match() {}
 };
 
-class CRegexAscii_branch;
-class CRegexAscii_piece;
-class CRegexAscii_chargroup;
-class CRegexAscii_parser;
+class CRegexXQuery_branch;
+class CRegexXQuery_piece;
+class CRegexXQuery_chargroup;
+class CRegexXQuery_parser;
 
-class CRegexAscii_regex : public IRegexAtom
+class CRegexXQuery_regex : public IRegexAtom
 {
-  friend class CRegexAscii_parser;
-  friend class CRegexAscii_branch;
-  friend class CRegexAscii_piece;
-  friend class CRegexAscii_chargroup;
-  friend class CRegexAscii_negchargroup;
-  friend class CRegexAscii_wildchar;
-  friend class CRegexAscii_backref;
+  friend class CRegexXQuery_parser;
+  friend class CRegexXQuery_branch;
+  friend class CRegexXQuery_piece;
+  friend class CRegexXQuery_chargroup;
+  friend class CRegexXQuery_negchargroup;
+  friend class CRegexXQuery_wildchar;
+  friend class CRegexXQuery_backref;
+  friend class CRegexXQuery_pinstart;
 public:
-  CRegexAscii_regex(CRegexAscii_regex *);
-  virtual ~CRegexAscii_regex();
+  CRegexXQuery_regex(CRegexXQuery_regex *);
+  virtual ~CRegexXQuery_regex();
 
   bool match_anywhere(const char *source, unsigned int flags, int *match_pos, int *matched_len);
   bool match_from(const char *source, unsigned int flags, int *match_pos, int *matched_len);
-  virtual bool match(const char *source, int *matched_len);
 
   //for replace $1, $2 ...
   bool  get_indexed_match(int index, const char **matched_source, int *matched_len);
   unsigned int get_indexed_regex_count();
 
   bool get_reachedEnd() {return reachedEnd;}
-  bool set_align_begin(bool align_begin);
+  void set_reachedEnd(const char *source) {if(source > source_start) reachedEnd = true;}
+  unsigned int get_flags() {return flags;}
+public:
+  virtual bool match(const char *source, int *start_from_branch, int *matched_len,
+                    std::list<RegexAscii_pieceinfo>::iterator next_piece,
+                    std::list<RegexAscii_pieceinfo>::iterator end_piece);
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len) {return false;}//not impl
+  virtual void reset_match();
+//  virtual void restore_match();
 private:
-  void add_branch(CRegexAscii_branch *branch);
+  void add_branch(CRegexXQuery_branch *branch);
+
+  void save_subregex_list(std::vector<std::pair<const char*, int> > &saved_subregex);
+  void load_subregex_list(std::vector<std::pair<const char*, int> > &saved_subregex);
 private:
   unsigned int flags;
-  std::list<CRegexAscii_branch*>   branch_list;
-  bool  align_begin;
+  std::list<CRegexXQuery_branch*>   branch_list;
+
+  const char  *source_start;
 
   const char  *matched_source;
   int         matched_len;
-  std::vector<CRegexAscii_regex*>    subregex;//for grouping
+//  const unicode::code_point  *backup_matched_source;
+//  int         backup_matched_len;
+  std::vector<CRegexXQuery_regex*>    subregex;//for grouping
 
   bool        reachedEnd;
 };
 
-class CRegexAscii_branch : public IRegexMatcher
+class CRegexXQuery_branch
 {
-  friend class CRegexAscii_parser;
+  friend class CRegexXQuery_parser;
 public:
-  CRegexAscii_branch(CRegexAscii_regex* regex);
-  ~CRegexAscii_branch();
+  CRegexXQuery_branch(CRegexXQuery_regex* regex);
+  ~CRegexXQuery_branch();
 
-  virtual bool match(const char *source, int *matched_len);
-private:
-  std::list<CRegexAscii_piece*>   piece_list;
-private:
-  void add_piece(CRegexAscii_piece *piece);
+  bool match(const char *source, int *matched_len,
+              CRegexXQuery_regex* group_regex,
+              std::list<RegexAscii_pieceinfo>::iterator next_piece,
+              std::list<RegexAscii_pieceinfo>::iterator end_piece);
+  void reset();
+//  void restore();
+private:
+  std::list<RegexAscii_pieceinfo>   piece_list;
+private:
+  void add_piece(CRegexXQuery_piece *piece);
  
-  bool match_piece_iter_reluctant(std::list<CRegexAscii_piece*>::iterator piece_it,
-                        const char *source, int *matched_len);
-  bool match_piece_iter_normal(std::list<CRegexAscii_piece*>::iterator piece_it,
-                        const char *source, int *matched_len);
 };
 
-class CRegexAscii_piece //: public IRegexMatcher
+class CRegexXQuery_piece //: public IRegexMatcher
 {
-  friend class CRegexAscii_parser;
-public:
+  friend class CRegexXQuery_parser;
+  friend class CRegexXQuery_branch;
+
   IRegexAtom *atom;
+  CRegexXQuery_regex *regex_atom;
+
   //quantifier
   bool  strict_max;
   int   min;
@@ -125,8 +164,8 @@
   bool  is_reluctant;
 
 public:
-  CRegexAscii_piece();
-  ~CRegexAscii_piece();
+  CRegexXQuery_piece();
+  ~CRegexXQuery_piece();
 public:
   void set_atom(IRegexAtom *atom);
   void set_quantifier_min_max(int min, int max, bool strict_max);
@@ -134,95 +173,294 @@
   void get_quantifier(int *min, int *max, bool *strict_max);
   bool get_is_reluctant();
 //  bool match(const char *source, int *matched_len);
+  bool match_piece(std::list<RegexAscii_pieceinfo>::iterator next_piece,
+                   std::list<RegexAscii_pieceinfo>::iterator end_piece,
+                   const char *source, int *matched_len);
+protected:
   bool match_piece_times(const char *source, 
                          int *piecelen, 
                          int times,
-                         std::vector<int>    *match_lens);
-};
-
-#define   CHARGROUP_FLAGS_MULTICHAR   1
-#define   CHARGROUP_FLAGS_ENDLINE     2
-
-class CRegexAscii_chargroup : public IRegexAtom
-{
-  friend class CRegexAscii_parser;
-public:
-  CRegexAscii_chargroup(CRegexAscii_regex* regex);
-  virtual ~CRegexAscii_chargroup();
+                         std::vector<std::pair<int,int> >    *match_lens);
+  int  choose_another_branch(std::vector<std::pair<int,int> > &match_lens);
+  bool match_piece_iter_reluctant(std::list<RegexAscii_pieceinfo>::iterator next_piece,
+                        std::list<RegexAscii_pieceinfo>::iterator end_piece,
+                        const char *source, int *matched_len);
+  bool match_piece_iter_normal(std::list<RegexAscii_pieceinfo>::iterator next_piece,
+                        std::list<RegexAscii_pieceinfo>::iterator end_piece,
+                        const char *source, int *matched_len);
+  bool is_regex_atom();
+};
+
+
+enum CHARGROUP_t
+{
+CHARGROUP_NO_MULTICHAR = 0,
+//CHARGROUP_FLAGS_CHAR_RANGE,
+CHARGROUP_FLAGS_MULTICHAR_p,
+CHARGROUP_FLAGS_MULTICHAR_Is,
+CHARGROUP_FLAGS_MULTICHAR_OTHER,
+CHARGROUP_FLAGS_ONECHAR_ASCII,
+CHARGROUP_FLAGS_ONECHAR_UNICODE
+//CHARGROUP_FLAGS_ENDLINE
+};
+
+
+class CRegexXQuery_charmatch : public IRegexAtom
+{
+  friend class CRegexXQuery_parser;
+protected:
+  //enum CHARGROUP_t  type;
+public:
+  CRegexXQuery_charmatch(CRegexXQuery_regex* regex);//, enum CHARGROUP_t  type);
+  virtual ~CRegexXQuery_charmatch() {}
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len) = 0;
+  virtual unicode::code_point get_c() {return 0;}
+};
+
+class CRegexXQuery_multicharP : public CRegexXQuery_charmatch
+{
+  char multichar_type;
+  bool is_reverse;
+public:
+  CRegexXQuery_multicharP(CRegexXQuery_regex* regex, char type, bool is_reverse);
+  virtual ~CRegexXQuery_multicharP() {}
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+};
+
+class CRegexXQuery_multicharIs : public CRegexXQuery_charmatch
+{
+  int block_index;
+  bool is_reverse;
+public:
+  CRegexXQuery_multicharIs(CRegexXQuery_regex* regex, int block_index, bool is_reverse);
+  virtual ~CRegexXQuery_multicharIs() {}
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+};
+
+class CRegexXQuery_multicharOther : public CRegexXQuery_charmatch
+{
+  char multichar_type;
+public:
+  CRegexXQuery_multicharOther(CRegexXQuery_regex* regex, char type);
+  virtual ~CRegexXQuery_multicharOther() {}
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+};
+
+class CRegexXQuery_char_ascii : public CRegexXQuery_charmatch
+{
+  friend class CRegexXQuery_parser;
+protected:
+  char c;
+public:
+  CRegexXQuery_char_ascii(CRegexXQuery_regex* regex, char c);
+  virtual ~CRegexXQuery_char_ascii() {}
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+  virtual unicode::code_point get_c() {return c;}
+};
+
+class CRegexXQuery_char_ascii_i : public CRegexXQuery_char_ascii
+{
+public:
+  CRegexXQuery_char_ascii_i(CRegexXQuery_regex* regex, char c);
+  virtual ~CRegexXQuery_char_ascii_i() {}
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+  virtual unicode::code_point get_c()  {return c;}
+};
+
+class CRegexXQuery_char_range_ascii : public CRegexXQuery_charmatch
+{
+protected:
+  char c1;
+  char c2;
+public:
+  CRegexXQuery_char_range_ascii(CRegexXQuery_regex* regex, char c1, char c2);
+  virtual ~CRegexXQuery_char_range_ascii() {}
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+};
+
+class CRegexXQuery_char_range_ascii_i : public CRegexXQuery_char_range_ascii
+{
+public:
+  CRegexXQuery_char_range_ascii_i(CRegexXQuery_regex* regex, char c1, char c2);
+  virtual ~CRegexXQuery_char_range_ascii_i() {}
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+};
+
+class CRegexXQuery_char_unicode : public CRegexXQuery_charmatch
+{
+  unsigned char c[6];
+  int len;
+public:
+  CRegexXQuery_char_unicode(CRegexXQuery_regex* regex, const char *c, int len);
+  virtual ~CRegexXQuery_char_unicode() {}
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+  virtual unicode::code_point get_c();
+};
+
+class CRegexXQuery_char_unicode_cp : public CRegexXQuery_charmatch
+{
+protected:
+  unicode::code_point c;
+public:
+  CRegexXQuery_char_unicode_cp(CRegexXQuery_regex* regex, unicode::code_point c);
+  virtual ~CRegexXQuery_char_unicode_cp() {}
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+  virtual unicode::code_point get_c()  {return c;}
+};
+
+class CRegexXQuery_char_unicode_i : public CRegexXQuery_char_unicode_cp
+{
+public:
+  CRegexXQuery_char_unicode_i(CRegexXQuery_regex* regex, unicode::code_point c);
+  virtual ~CRegexXQuery_char_unicode_i() {}
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+  virtual unicode::code_point get_c()  {return c;}
+};
+
+class CRegexXQuery_char_range_unicode : public CRegexXQuery_charmatch
+{
+protected:
+  unicode::code_point c1;
+  unicode::code_point c2;
+public:
+  CRegexXQuery_char_range_unicode(CRegexXQuery_regex* regex, unicode::code_point c1, unicode::code_point c2);
+  virtual ~CRegexXQuery_char_range_unicode() {}
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+};
+
+class CRegexXQuery_char_range_unicode_i : public CRegexXQuery_char_range_unicode
+{
+public:
+  CRegexXQuery_char_range_unicode_i(CRegexXQuery_regex* regex, unicode::code_point c1, unicode::code_point c2);
+  virtual ~CRegexXQuery_char_range_unicode_i() {}
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+};
+
+class CRegexXQuery_endline : public CRegexXQuery_charmatch
+{
+public:
+  CRegexXQuery_endline(CRegexXQuery_regex* regex);
+  virtual ~CRegexXQuery_endline() {}
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+};
+
+
+class CRegexXQuery_chargroup : public IRegexAtom
+{
+  friend class CRegexXQuery_parser;
+public:
+  CRegexXQuery_chargroup(CRegexXQuery_regex* regex);
+  virtual ~CRegexXQuery_chargroup();
 private:
-  typedef struct
+/*  typedef struct
   {
-    unsigned char  flags;
+    CHARGROUP_t  flags;
     char  c1;
     char  c2;
   }chargroup_t;
-  std::list<chargroup_t>    chargroup_list;
-  CRegexAscii_chargroup *classsub;
-public:
-  void addMultiChar(char c);
-  void addEndLine();
-  void addCharRange(char c1, char c2);
-  void addClassSub(CRegexAscii_chargroup* classsub);
-
-  virtual bool match(const char *source, int *matched_len);
-};
-
-class CRegexAscii_negchargroup : public CRegexAscii_chargroup
-{
-public:
-  CRegexAscii_negchargroup(CRegexAscii_regex* regex);
-  virtual ~CRegexAscii_negchargroup();
-
-  virtual bool match(const char *source, int *matched_len);
-};
-
-class CRegexAscii_wildchar : public IRegexAtom
-{
-public:
-  CRegexAscii_wildchar(CRegexAscii_regex* regex);
-  virtual ~CRegexAscii_wildchar();
-
-  virtual bool match(const char *source, int *matched_len);
-};
-
-class CRegexAscii_backref : public IRegexAtom
-{
-public:
-  CRegexAscii_backref(CRegexAscii_regex* regex, unsigned int backref);
-  virtual ~CRegexAscii_backref();
-
-  virtual bool match(const char *source, int *matched_len);
+*/
+  std::list<CRegexXQuery_charmatch* >    chargroup_list;
+  CRegexXQuery_chargroup *classsub;
+public:
+  //void addMultiChar(char c, CHARGROUP_t multichar_type);
+  //void addEndLine();
+  //void addCharRange(char c1, char c2);
+  //void addOneChar(char c);
+  void addCharMatch(CRegexXQuery_charmatch *charmatch);
+  void addClassSub(CRegexXQuery_chargroup* classsub);
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+};
+
+class CRegexXQuery_negchargroup : public CRegexXQuery_chargroup
+{
+public:
+  CRegexXQuery_negchargroup(CRegexXQuery_regex* regex);
+  virtual ~CRegexXQuery_negchargroup();
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+};
+
+class CRegexXQuery_wildchar : public IRegexAtom
+{
+public:
+  CRegexXQuery_wildchar(CRegexXQuery_regex* regex);
+  virtual ~CRegexXQuery_wildchar();
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+};
+
+class CRegexXQuery_backref : public IRegexAtom
+{
+public:
+  CRegexXQuery_backref(CRegexXQuery_regex* regex, unsigned int backref);
+  virtual ~CRegexXQuery_backref();
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
 private:
   unsigned int backref;
 };
 
-class CRegexAscii_parser
-{
-public:
-  CRegexAscii_parser();
-  ~CRegexAscii_parser();
-
-public:
-  CRegexAscii_regex* parse(const char *pattern, unsigned int flags);
+class CRegexXQuery_pinstart : public IRegexAtom
+{
+public:
+  CRegexXQuery_pinstart(CRegexXQuery_regex* regex);
+
+  virtual bool match_internal(const char *source, int *start_from_branch, int *matched_len);
+};
+
+class CRegexXQuery_parser
+{
+public:
+  typedef struct
+  {
+    const unicode::code_point    cp[2];//in pairs start, end
+    const unicode::code_point    *ext_cp;
+    const char    *group_name;
+  }block_escape_t;
+
+  CRegexXQuery_parser();
+  ~CRegexXQuery_parser();
+
+public:
+  CRegexXQuery_regex* parse(const char *pattern, unsigned int flags);
 
 protected:
-  CRegexAscii_regex* parse_regexp(const char *pattern, int *regex_len);
-  CRegexAscii_branch* parse_branch(const char *pattern, int *branch_len);
-  CRegexAscii_piece* parse_piece(const char *pattern, int *piece_len);
+  CRegexXQuery_regex* parse_regexp(const char *pattern, int *regex_len);
+  CRegexXQuery_branch* parse_branch(const char *pattern, int *branch_len);
+  CRegexXQuery_piece* parse_piece(const char *pattern, int *piece_len);
   char myishex(char c);
   bool myisdigit(char c);
-  char readChar(const char *pattern, int *char_len, bool *is_multichar);
+  bool myisletterAZ(char c);
+  CRegexXQuery_charmatch* readChar(const char *pattern, int *char_len, CHARGROUP_t *multichar_type);
+  CRegexXQuery_charmatch *create_charmatch(unicode::code_point utf8c,
+                                            const char *pattern, int utf8len,
+                                            enum CHARGROUP_t *multichar_type);
   IRegexAtom* read_atom(const char *pattern, int *atom_len);
-  CRegexAscii_chargroup* readchargroup(const char *pattern, int *chargroup_len);
-  void read_quantifier(CRegexAscii_piece *piece, const char *pattern, int *quantif_len);
+  CRegexXQuery_chargroup* readchargroup(const char *pattern, int *chargroup_len);
+  void read_quantifier(CRegexXQuery_piece *piece, const char *pattern, int *quantif_len);
 
 private:
-  CRegexAscii_regex   *current_regex;
+  CRegexXQuery_regex   *current_regex;
   int   regex_depth;
   unsigned int flags;
 };
 
-}}//end namespace zorba::regex_ascii
+}
+}//end namespace zorba::regex_xquery
 
 #endif
 /* vim:set et sw=2 ts=2: */

=== modified file 'src/util/unicode_categories.cpp'
--- src/util/unicode_categories.cpp	2011-06-14 17:26:33 +0000
+++ src/util/unicode_categories.cpp	2012-01-18 18:33:36 +0000
@@ -65812,7 +65812,7 @@
   { 0x100000, 0x100000, UNICODE_Co},
 };
 
-bool check_codepoint_category(code_point cp, UnicodeCategoriesEnum categ)
+bool check_codepoint_category(code_point cp, category categ)
 {
   if(cp < 0x10000)
     return codepoints_categories[cp] == categ;
@@ -65824,10 +65824,10 @@
       if(cp >= codepoints_categories2[i].cp1)
         return codepoints_categories2[i].category == categ;
       else
-        return false;
+        return categ ? false : true;
     }
   }
-  return false;
+  return categ ? false : true;
 }
 
 /*

=== modified file 'src/util/unicode_categories.h'
--- src/util/unicode_categories.h	2011-06-14 17:26:33 +0000
+++ src/util/unicode_categories.h	2012-01-18 18:33:36 +0000
@@ -22,46 +22,53 @@
 namespace zorba {
 namespace unicode {
 
-//Unicode codepoint categories, as from http://www.fileformat.info/info/unicode/category/index.htm
+///////////////////////////////////////////////////////////////////////////////
 
-enum UnicodeCategoriesEnum {
-UNICODE_Cc, //Other, Control
-UNICODE_Cf, //Other, Format
-UNICODE_Co, //Other, Private Use
-UNICODE_Cs, //Other, Surrogate
-UNICODE_Ll, //Letter, Lowercase
-UNICODE_Lm, //Letter, Modifier
-UNICODE_Lo, //Letter, Other
-UNICODE_Lt, //Letter, Titlecase
-UNICODE_Lu, //Letter, Uppercase
-UNICODE_Mc, //Mark, Spacing Combining
-UNICODE_Me, //Mark, Enclosing
-UNICODE_Mn, //Mark, Nonspacing
-UNICODE_Nd, //Number, Decimal Digit
-UNICODE_Nl, //Number, Letter
-UNICODE_No, //Number, Other
-UNICODE_Pc, //Punctuation, Connector
-UNICODE_Pd, //Punctuation, Dash
-UNICODE_Pe, //Punctuation, Close
-UNICODE_Pf, //Punctuation, Final quote (may behave like Ps or Pe depending on usage)
-UNICODE_Pi, //Punctuation, Initial quote (may behave like Ps or Pe depending on usage)
-UNICODE_Po, //Punctuation, Other
-UNICODE_Ps, //Punctuation, Open
-UNICODE_Sc, //Symbol, Currency
-UNICODE_Sk, //Symbol, Modifier
-UNICODE_Sm, //Symbol, Math
-UNICODE_So, //Symbol, Other
-UNICODE_Zl, //Separator, Line
-UNICODE_Zp, //Separator, Paragraph
-UNICODE_Zs  //Separator, Space
+/**
+ * Unicode codepoint categories.
+ * See: http://www.fileformat.info/info/unicode/category/
+ */
+enum category {
+  UNICODE_Cn, // Not Assigned
+  UNICODE_Cc, // Other, Control
+  UNICODE_Cf, // Other, Format
+  UNICODE_Co, // Other, Private Use
+  UNICODE_Cs, // Other, Surrogate
+  UNICODE_Ll, // Letter, Lowercase
+  UNICODE_Lm, // Letter, Modifier
+  UNICODE_Lo, // Letter, Other
+  UNICODE_Lt, // Letter, Titlecase
+  UNICODE_Lu, // Letter, Uppercase
+  UNICODE_Mc, // Mark, Spacing Combining
+  UNICODE_Me, // Mark, Enclosing
+  UNICODE_Mn, // Mark, Nonspacing
+  UNICODE_Nd, // Number, Decimal Digit
+  UNICODE_Nl, // Number, Letter
+  UNICODE_No, // Number, Other
+  UNICODE_Pc, // Punctuation, Connector
+  UNICODE_Pd, // Punctuation, Dash
+  UNICODE_Pe, // Punctuation, Close
+  UNICODE_Pf, // Punctuation, Final quote (like Ps or Pe depending on usage)
+  UNICODE_Pi, // Punctuation, Initial quote (like Ps or Pe depending on usage)
+  UNICODE_Po, // Punctuation, Other
+  UNICODE_Ps, // Punctuation, Open
+  UNICODE_Sc, // Symbol, Currency
+  UNICODE_Sk, // Symbol, Modifier
+  UNICODE_Sm, // Symbol, Math
+  UNICODE_So, // Symbol, Other
+  UNICODE_Zl, // Separator, Line
+  UNICODE_Zp, // Separator, Paragraph
+  UNICODE_Zs  // Separator, Space
 };
 
 bool is_UnicodeNd(code_point cp, code_point *ret_zero);
 
-bool check_codepoint_category(code_point cp, UnicodeCategoriesEnum categ);
-
-}
-}
-
-#endif
+bool check_codepoint_category(code_point cp, category categ);
+
+///////////////////////////////////////////////////////////////////////////////
+
+} // namespace unicode
+} // namespaec zorba
+
+#endif /* ZORBA_UNICODE_CATEGORIES */
 /* vim:set et sw=2 ts=2: */

=== modified file 'src/util/unicode_util.cpp'
--- src/util/unicode_util.cpp	2011-07-17 00:10:56 +0000
+++ src/util/unicode_util.cpp	2012-01-18 18:33:36 +0000
@@ -22,15 +22,19 @@
 #include <functional>                   /* for binary_function */
 #include <utility>                      /* for pair */
 
-#include <unicode/normlzr.h>
-#include <unicode/ustring.h>
+#ifndef ZORBA_NO_ICU
+# include <unicode/normlzr.h>
+# include <unicode/ustring.h>
+#endif /* ZORBA_NO_ICU */
 
 #include "cxx_util.h"
 #include "unicode_util.h"
 #include "utf8_util.h"
 
 using namespace std;
+#ifndef ZORBA_NO_ICU
 U_NAMESPACE_USE
+#endif /* ZORBA_NO_ICU */
 
 namespace zorba {
 namespace unicode {
@@ -2208,6 +2212,8 @@
   return to_case<upper>( c );
 }
 
+#ifndef ZORBA_NO_ICU
+
 bool normalize( string const &in, normalization::type n, string *out ) {
   UErrorCode status = U_ZERO_ERROR;
   UNormalizationMode icu_mode;
@@ -2230,8 +2236,11 @@
   return U_SUCCESS( status ) == TRUE;
 }