Merge lp:~zorba-coders/zorba/bug-1026250 into lp:zorba

[Matthias Brantner]

fix for bug lp:1026250

- annotation to turn on sequential flwor streaming
- introduce support for pragmas and allows the body of a pragma to be a statement (recent scripting extension)
- collections are versioned in order to avoid deadlocks when modifying a collection that is currently read from (instead raise an error)

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/bug-1026250-2012-12-05T01-32-06.304Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~zorba-coders/zorba/bug-1026250 into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:275 (message):
  Validation queue job bug-1026250-2012-12-05T01-32-06.304Z is finished. The
  final status was:

  13 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Paul J. Lucas (paul-lucas)]

Of the failing tests, only some have been modified. If you changed the default behavior, shouldn't you also have to change every affected test by adding the pragma?

[lp:~zorba-coders/zorba/bug-1026250]

11088. By Paul J. Lucas on 2012-12-06

Cosmetic change.

11089. By Paul J. Lucas on 2012-12-06

Cleaned-up assert message.

11090. By Paul J. Lucas on 2012-12-06

Added operator<< for flwor_expr::ClauseKind.

11091. By Paul J. Lucas on 2012-12-06

Cleaned-up assert message.

[Paul J. Lucas (paul-lucas)]

I modified test flwor10.xq to be:

declare namespace ext = "http://www.zorba-xquery.com/extensions";

variable $x := 0;

(# ext:materialize #) {
  for $i in (1, 2, 3)
  let $y := { $x := $x + $i; () }
  return $x
}

ran it and got:

Zorba error [zerr:ZXQP0002]: "false": assertion failed with the message: ClauseKind = materialize_clause; raised at /Users/pjl/src/flwor/zorba/repo/bug-1026250/zorba/src/compiler/codegen/plan_visitor.cpp:1028

The switch statement doesn't have a case for materialize_clause. Either it should, or it should never be the case that a materialize_clause should occur here.

FYI: I modified the code on the branch to overload operator<< for ClauseKind -- makes it handy for debugging.

[Chris Hillery (ceejatec)]

Please put a more descriptive commit message for this change.

[Chris Hillery (ceejatec)]

Matthias will get this scheduled within 28msec, hopefully soon.

[lp:~zorba-coders/zorba/bug-1026250]

11092. By Federico Cavalieri on 2013-08-12

Merged trunk into the branch

11093. By Federico Cavalieri on 2013-08-12

Fixed compilation

11094. By Federico Cavalieri on 2013-08-12

Inverted annotation semantic

11095. By Federico Cavalieri on 2013-08-14

Implemented no-streaming extension

[Federico Cavalieri (fcavalieri)]

Ready for review

[Ghislain Fourny (gislenius)]

One of the tests uses ext:nomaterialization without dash. Is this intented?

[Federico Cavalieri (fcavalieri)]

nope, let me fix it

On Wed, Aug 14, 2013 at 3:14 PM, Ghislain Fourny <[e-mail]
> wrote:

> One of the tests uses ext:nomaterialization without dash. Is this intented?
> --
> https://code.launchpad.net/~zorba-coders/zorba/bug-1026250/+merge/138044
> Your team Zorba Coders is subscribed to branch lp:zorba.
>
> --
> Mailing list: https://launchpad.net/~zorba-coders
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~zorba-coders
> More help : https://help.launchpad.net/ListHelp
>

[lp:~zorba-coders/zorba/bug-1026250]

11096. By Federico Cavalieri on 2013-08-14

Fixed test28

[Federico Cavalieri (fcavalieri)]

Fixed and pushed

On Wed, Aug 14, 2013 at 9:36 PM, Federico Cavalieri <
[e-mail]> wrote:

> nope, let me fix it
>
>
> On Wed, Aug 14, 2013 at 3:14 PM, Ghislain Fourny <
> [e-mail]> wrote:
>
>> One of the tests uses ext:nomaterialization without dash. Is this
>> intented?
>> --
>> https://code.launchpad.net/~zorba-coders/zorba/bug-1026250/+merge/138044
>> Your team Zorba Coders is subscribed to branch lp:zorba.
>>
>> --
>> Mailing list: https://launchpad.net/~zorba-coders
>> Post to : <email address hidden>
>> Unsubscribe : https://launchpad.net/~zorba-coders
>> More help : https://help.launchpad.net/ListHelp
>>
>
>

[Matthias Brantner (matthias-brantner)]

Nice change.

The modifications in the following files don't seem to be needed:
- test/rbkt/Queries/zorba/updates/flwor1.xq
- test/rbkt/Queries/zorba/scripting/flwor6.xq
- test/rbkt/Queries/zorba/updates/flwor1.xq
- test/rbkt/Queries/zorba/no-copy/dataguide-c.xq

I'll add Markos as a second reviewer.

[Matthias Brantner (matthias-brantner)]

review: Approve

[lp:~zorba-coders/zorba/bug-1026250]

11097. By Federico Cavalieri on 2013-08-16

Completely reverted unnecessary test modifications

11098. By Federico Cavalieri on 2013-08-16

Reverted additional unnecessary test modification

11099. By Federico Cavalieri on 2013-08-16

Reverted further unnecessary modifications

[Federico Cavalieri (fcavalieri)]

Thanks for the review.
Can you explain me why
SYNC_CODE(theCollection->theLatch.rlock();)
and
SYNC_CODE(theCollection->theLatch.unlock();)
has been commented in this branch?
They were already commented in the branch when I started working on it.

[Matthias Brantner (matthias-brantner)]

> Thanks for the review.
> Can you explain me why
> SYNC_CODE(theCollection->theLatch.rlock();)
> and
> SYNC_CODE(theCollection->theLatch.unlock();)
> has been commented in this branch?
> They were already commented in the branch when I started working on it.
They were causing a deadlock when the collection being read from was modified. This is now fixed using collection versioning. The two lines can be removed.

[Markos Zaharioudakis (markos-za)]

I did not like very much the versioning solution for the collection deadlock. It certainly does not work with multi-threading. But this is not the only thing that is wrong for multi-threading, so I will approve.

I think a better solution would be that any dml:collection() function that is executed within a nonmaterialize pragma acquires the collection lock in exclussive mode. Of course that would require changes in the compiler, runtime, and store api.

review: Approve

[Markos Zaharioudakis (markos-za)]

I did not like very much the versioning solution for the collection deadlock. It certainly does not work with multi-threading. But this is not the only thing that is wrong for multi-threading, so I will approve.

I think a better solution would be that any dml:collection() function that is executed within a nonmaterialize pragma acquires the collection lock in exclussive mode. Of course that would require changes in the compiler, runtime, and store api.

review: Approve

[Federico Cavalieri (fcavalieri)]

Thanks for the feedback.
Merging

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for the following merge proposals:
https://code.launchpad.net/~zorba-coders/zorba/bug-1026250/+merge/138044

Progress dashboard at http://jenkins.lambda.nu/view/ValidationQueue

[Zorba Build Bot (zorba-buildbot)]

Validation queue succeeded - proposal merged!

=== modified file 'include/zorba/pregenerated/diagnostic_list.h'
--- include/zorba/pregenerated/diagnostic_list.h	2013-08-01 09:41:39 +0000
+++ include/zorba/pregenerated/diagnostic_list.h	2013-08-16 07:08:59 +0000
@@ -692,6 +692,8 @@
 
 extern ZORBA_DLL_PUBLIC ZorbaErrorCode ZDDY0040_INCONSISTENT_EDIT;
 
+extern ZORBA_DLL_PUBLIC ZorbaErrorCode ZDDY0041_CONCURRENT_MODIFICATION;
+
 extern ZORBA_DLL_PUBLIC ZorbaErrorCode ZDST0001_COLLECTION_ALREADY_DECLARED;
 
 extern ZORBA_DLL_PUBLIC ZorbaErrorCode ZDST0002_COLLECTION_ALREADY_IMPORTED;
 
=== modified file 'modules/pregenerated/zorba-errors.xq'
--- modules/pregenerated/zorba-errors.xq	2013-08-07 05:28:39 +0000
+++ modules/pregenerated/zorba-errors.xq	2013-08-16 07:08:59 +0000
@@ -602,6 +602,10 @@
 
 (:~
 :)
+declare variable $zerr:ZDDY0041 as xs:QName := fn:QName($zerr:NS, "zerr:ZDDY0041");
+
+(:~
+:)
 declare variable $zerr:ZDST0001 as xs:QName := fn:QName($zerr:NS, "zerr:ZDST0001");
 
 (:~
 
=== modified file 'src/compiler/codegen/plan_visitor.cpp'
--- src/compiler/codegen/plan_visitor.cpp	2013-07-01 18:59:06 +0000
+++ src/compiler/codegen/plan_visitor.cpp	2013-08-16 07:08:59 +0000
@@ -935,6 +935,9 @@
     
   if (v.is_sequential())
   {
+    pragma* pr = 0;
+    theCCB->lookup_pragma(&v, "no-materialization", pr);
+
     if (!isGeneral)
     {
       if (v.has_sequential_clauses())
@@ -983,15 +986,13 @@
 
       // Note: a materialize clause may exist already in case plan serialization
       // is on (see comment in materialize_clause::clone)
-      if (!isGeneral &&
+      if (!pr && !isGeneral &&
           v.get_return_expr()->is_sequential() &&
           v.get_clause(numClauses-1)->get_kind() != flwor_clause::materialize_clause &&
           (v.get_order_clause() != NULL || v.get_group_clause() == NULL))
       {
-        materialize_clause* mat =
-        theCCB->theEM->create_materialize_clause(v.get_sctx(),
+        materialize_clause* mat = theCCB->theEM->create_materialize_clause(v.get_sctx(),
                                                  v.get_return_expr()->get_loc());
-
         v.add_clause(mat);
         ++numClauses;
       }
@@ -1027,7 +1028,7 @@
               ++numForClauses;
           }
 
-          if (domExpr->is_sequential() &&
+          if (!pr && domExpr->is_sequential() &&
               (k == flwor_clause::for_clause ||
                k == flwor_clause::window_clause ||
                numForClauses > 0))
@@ -1066,8 +1067,8 @@
           break;
         }
         default:
-          ZORBA_ASSERT(false);
-        }
+          ZORBA_ASSERT_WITH_MSG(false, "ClauseKind = " << k);
+        } // switch
 
         ++i;
       }
 
=== modified file 'src/compiler/expression/flwor_expr.cpp'
--- src/compiler/expression/flwor_expr.cpp	2013-06-14 04:50:39 +0000
+++ src/compiler/expression/flwor_expr.cpp	2013-08-16 07:08:59 +0000
@@ -52,6 +52,27 @@
 /*******************************************************************************
 
 ********************************************************************************/
+std::ostream& operator<<( std::ostream &o, flwor_clause::ClauseKind k ) {
+  char const *const s[] = {
+    "for_clause",
+    "let_clause",
+    "window_clause",
+    "group_clause",
+    "order_clause",
+    "count_clause",
+    "where_clause",
+    "materialize_clause"
+  };
+  if ( k >= flwor_clause::for_clause && k <= flwor_clause::materialize_clause )
+    o << s[k];
+  else
+    o << "[invalid ClauseKind: " << (int)k << ']';
+  return o;
+}
+
+/*******************************************************************************
+
+********************************************************************************/
 forletwin_clause::forletwin_clause(
     static_context* sctx,
     CompilerCB* ccb,
@@ -671,7 +692,7 @@
     expr::substitution_t& subst) const
 {
   // we will reach here under the following scenario:
-  // 1. We do plan seriazation
+  // 1. We do plan serialization
   // 2. getPlan is called on udf A; this causes a mat clause to be created
   //    during the codegen on A's body
   // 3. getPlan is called on udf B, which invokes A, and A's body is
 
=== modified file 'src/compiler/expression/flwor_expr.h'
--- src/compiler/expression/flwor_expr.h	2013-06-14 04:50:39 +0000
+++ src/compiler/expression/flwor_expr.h	2013-08-16 07:08:59 +0000
@@ -50,7 +50,7 @@
 public:
   typedef var_rebind_list_t rebind_list_t;
 
-  typedef enum
+  enum ClauseKind
   {
     for_clause,
     let_clause,
@@ -60,7 +60,8 @@
     count_clause,
     where_clause,
     materialize_clause
-  } ClauseKind;
+    // if you change this, also ensure that operator<< for it is still correct
+  };
 
 protected:
   static_context          * theContext;
@@ -106,6 +107,8 @@
       expr::substitution_t& substitution) const = 0;
 };
 
+// for debugging
+std::ostream& operator<<( std::ostream&, flwor_clause::ClauseKind );
 
 /***************************************************************************//**
 
 
=== modified file 'src/compiler/parser/parser.y'
--- src/compiler/parser/parser.y	2013-06-17 20:05:21 +0000
+++ src/compiler/parser/parser.y	2013-08-16 07:08:59 +0000
@@ -4048,16 +4048,10 @@
 
 // [64]
 ExtensionExpr :
-        Pragma_list LBRACE RBRACE
-        {
-            $$ = new ExtensionExpr(
-                LOC(@$), dynamic_cast<PragmaList*>($1), NULL
-            );
-        }
-    |   Pragma_list LBRACE Expr RBRACE
-        {
-            $$ = new ExtensionExpr(
-                LOC(@$), dynamic_cast<PragmaList*>($1), $3
+        Pragma_list BlockExpr
+        {
+            $$ = new ExtensionExpr(
+                LOC(@$), dynamic_cast<PragmaList*>($1), $2
             );
         }
     ;
 
=== modified file 'src/compiler/parsetree/parsenodes.h'
--- src/compiler/parsetree/parsenodes.h	2013-08-05 13:20:52 +0000
+++ src/compiler/parsetree/parsenodes.h	2013-08-16 07:08:59 +0000
@@ -3506,7 +3506,7 @@
 
 
 /*******************************************************************************
-  [88] ExtensionExpr ::= PragmaList "{" Expr? "}"
+  [88] ExtensionExpr ::= PragmaList BlockExpr
 ********************************************************************************/
 class ExtensionExpr : public exprnode
 {
 
=== modified file 'src/compiler/translator/translator.cpp'
--- src/compiler/translator/translator.cpp	2013-08-12 09:28:02 +0000
+++ src/compiler/translator/translator.cpp	2013-08-16 07:08:59 +0000
@@ -2451,6 +2451,31 @@
 }
 
 
+/*******************************************************************************
+********************************************************************************/
+void
+recognizePragma(expr* e, const zstring& aLocalName)
+{
+  for (std::vector<pragma*>::const_iterator lIter = theScopedPragmas.begin();
+       lIter != theScopedPragmas.end();
+       ++lIter)
+  {
+    pragma* p = *lIter;
+    if (p->theQName->getNamespace() == ZORBA_EXTENSIONS_NS)
+    {
+      if (p->theQName->getLocalName() == aLocalName)
+      {
+        e->get_ccb()->add_pragma(e, p);
+        e->setContainsPragma(ANNOTATION_TRUE);
+        break;
+      }
+    }
+  }
+}
+
+
+
+
 /////////////////////////////////////////////////////////////////////////////////
 //                                                                             //
 //  Module, VersionDecl, MainModule, LibraryModule, ModuleDecl                 //
@@ -6938,6 +6963,8 @@
 
   theFlworClausesStack.resize(curClausePos);
 
+  recognizePragma(flwor, "no-materialization");
+
   push_nodestack(flwor);
 }
 
 
=== modified file 'src/diagnostics/diagnostic_en.xml'
--- src/diagnostics/diagnostic_en.xml	2013-08-14 04:05:45 +0000
+++ src/diagnostics/diagnostic_en.xml	2013-08-16 07:08:59 +0000
@@ -2279,7 +2279,7 @@
        A Zorba programming assertion failed.  If this error occurs, it is a bug
        and should be reported.
       </comment>
-      <value>"$1": assertion failed.${ The following information might help: 2}</value>
+      <value>"$1": assertion failed${ with the message: 2}</value>
     </diagnostic>
 
     <diagnostic code="ZXQP0003" name="INTERNAL_ERROR">
@@ -2847,6 +2847,10 @@
     <diagnostic code="ZDDY0040" name="INCONSISTENT_EDIT">
       <value>attempt to edit an item with a template of a different kind (for example an object with a node).</value>
     </diagnostic>
+    
+    <diagnostic code="ZDDY0041" name="CONCURRENT_MODIFICATION">
+      <value>"$1": collection was modified while reading</value>
+    </diagnostic>
 
     <diagnostic code="ZDST0001" name="COLLECTION_ALREADY_DECLARED">
       <value>"$1": collection already declared</value>
 
=== modified file 'src/diagnostics/pregenerated/diagnostic_list.cpp'
--- src/diagnostics/pregenerated/diagnostic_list.cpp	2013-08-01 09:41:39 +0000
+++ src/diagnostics/pregenerated/diagnostic_list.cpp	2013-08-16 07:08:59 +0000
@@ -1017,6 +1017,9 @@
 ZorbaErrorCode ZDDY0040_INCONSISTENT_EDIT( "ZDDY0040" );
 
 
+ZorbaErrorCode ZDDY0041_CONCURRENT_MODIFICATION( "ZDDY0041" );
+
+
 ZorbaErrorCode ZDST0001_COLLECTION_ALREADY_DECLARED( "ZDST0001" );
 
 
 
=== modified file 'src/diagnostics/pregenerated/dict_en.cpp'
--- src/diagnostics/pregenerated/dict_en.cpp	2013-08-14 04:12:57 +0000
+++ src/diagnostics/pregenerated/dict_en.cpp	2013-08-16 07:08:59 +0000
@@ -356,6 +356,7 @@
   { "ZDDY0038", "\"$1\": illegal edit from queue collection" },
   { "ZDDY0039", "attempt to edit non-root node in collection \"$1\"" },
   { "ZDDY0040", "attempt to edit an item with a template of a different kind (for example an object with a node)." },
+  { "ZDDY0041", "\"$1\": collection was modified while reading" },
   { "ZDST0001", "\"$1\": collection already declared" },
   { "ZDST0002", "\"$1\": collection already imported into module \"$2\"" },
   { "ZDST0003", "\"$1\": collection declaration not allowed in main module" },
@@ -462,7 +463,7 @@
   { "ZXQD0006", "\"$1\": invalid UTF-8 byte sequence" },
   { "ZXQP0000", "no error" },
   { "ZXQP0001", "dynamic runtime error${: 1}" },
-  { "ZXQP0002", "\"$1\": assertion failed.${ The following information might help: 2}" },
+  { "ZXQP0002", "\"$1\": assertion failed${ with the message: 2}" },
   { "ZXQP0003", "internal error${: 1}" },
   { "ZXQP0004", "not yet implemented: $1" },
   { "ZXQP0005", "\"$1\": feature not enabled" },
 
=== modified file 'src/store/naive/simple_collection.cpp'
--- src/store/naive/simple_collection.cpp	2013-06-15 02:57:08 +0000
+++ src/store/naive/simple_collection.cpp	2013-08-16 07:08:59 +0000
@@ -41,8 +41,9 @@
     bool isDynamic)
   : 
   Collection(name),
-  theIsDynamic(isDynamic),
-  theAnnotations(annotations)
+  theIsDynamic(isDynamic),  
+  theAnnotations(annotations),
+  theVersion(0)
 {
   theId = GET_STORE().createCollectionId();
   theTreeIdGenerator = GET_STORE().getTreeIdGeneratorFactory().createTreeGenerator(0);
@@ -55,7 +56,8 @@
 ********************************************************************************/
 SimpleCollection::SimpleCollection()
   : 
-  theIsDynamic(false)
+  theIsDynamic(false),
+  theVersion(0)
 {
   theTreeIdGenerator = GET_STORE().getTreeIdGeneratorFactory().createTreeGenerator(0);
 }
@@ -282,7 +284,8 @@
         ERROR_PARAMS(ZED(ZXQD0004_NOT_WITHIN_RANGE), position)
       );
   }
-
+  
+  ++theVersion;
 }
 
 
@@ -382,6 +385,8 @@
     theTrees[targetPos + i].transfer(items[i]);
   }
 
+  ++theVersion;
+
   return xs_integer(targetPos);
 }
 
@@ -415,6 +420,7 @@
     {
       csize pos = to_xs_unsignedInt(position);
       theTrees.erase(theTrees.begin() + pos);
+      ++theVersion;
       return true;
     }
     catch (const std::range_error&)
@@ -468,6 +474,7 @@
     structuredItem->detachFromCollection();
 
     theTrees.erase(theTrees.begin() + pos);
+    ++theVersion;
     return true;
   }
 }
@@ -533,6 +540,7 @@
       theTrees.erase(theTrees.begin() + pos);
     }
 
+    ++theVersion;
     return xs_integer(last - pos);
   }
 }
@@ -623,11 +631,14 @@
 ********************************************************************************/
 void SimpleCollection::CollectionIter::open()
 {
-  SYNC_CODE(theCollection->theLatch.rlock();)
+  //SYNC_CODE(theCollection->theLatch.rlock();)
   theHaveLock = true;
 
   theIterator = theCollection->theTrees.begin();
   theEnd = theCollection->theTrees.end();
+  
+  theVersion = theCollection->theVersion;
+
   skip();
 }
 
@@ -637,6 +648,12 @@
 ********************************************************************************/
 bool SimpleCollection::CollectionIter::next(store::Item_t& result)
 {
+  if (theVersion != theCollection->theVersion)
+  {
+    throw ZORBA_EXCEPTION(zerr::ZDDY0041_CONCURRENT_MODIFICATION,
+    ERROR_PARAMS(theCollection->getName()->getStringValue()));
+  }
+
   if (!theHaveLock) 
   {
     throw ZORBA_EXCEPTION(zerr::ZDDY0019_COLLECTION_ITERATOR_NOT_OPEN,
@@ -663,6 +680,8 @@
 {
   theIterator = theCollection->theTrees.begin();
   theEnd = theCollection->theTrees.end();
+
+  theVersion = theCollection->theVersion;
   skip();
 }
 
@@ -674,7 +693,7 @@
 {
   assert(theHaveLock);
   theHaveLock = false;
-  SYNC_CODE(theCollection->theLatch.unlock();)
+  //SYNC_CODE(theCollection->theLatch.unlock();)
 }
 
 } // namespace simplestore
 
=== modified file 'src/store/naive/simple_collection.h'
--- src/store/naive/simple_collection.h	2013-02-26 04:12:43 +0000
+++ src/store/naive/simple_collection.h	2013-08-16 07:08:59 +0000
@@ -84,6 +84,7 @@
     checked_vector<store::Item_t>::iterator theEnd;
     bool                                    theHaveLock;
     csize                                   theSkip;
+    ulong                                   theVersion;
 
   public:
     CollectionIter(SimpleCollection* collection, const xs_integer& skip);
@@ -111,6 +112,8 @@
 
   const std::vector<store::Annotation_t> theAnnotations;
 
+  ulong                                  theVersion;
+
   SYNC_CODE(Latch                        theLatch;)
 
 protected:
 
=== added file 'test/rbkt/ExpCompilerResults/IterPlan/zorba/flwor/flwor24.iter'
--- test/rbkt/ExpCompilerResults/IterPlan/zorba/flwor/flwor24.iter	1970-01-01 00:00:00 +0000
+++ test/rbkt/ExpCompilerResults/IterPlan/zorba/flwor/flwor24.iter	2013-08-16 07:08:59 +0000
@@ -0,0 +1,56 @@
+Iterator tree for const-folded expr:
+<FunctionTraceIterator>
+  <FnConcatIterator/>
+</FunctionTraceIterator>
+
+Iterator tree for main query:
+<SequentialIterator>
+  <CtxVarDeclareIterator varid="4" varname="xmlcontents">
+    <UDFunctionCallIterator function="fetch:content">
+      <SingletonIterator value="xs:string($RBKT_SRC_DIR/Queries/zorba/flwor/flwor24.xml)"/>
+    </UDFunctionCallIterator>
+  </CtxVarDeclareIterator>
+  <flwor::FLWORIterator>
+    <ForVariable name="w">
+      <FnZorbaParseXmlFragmentIterator>
+        <PromoteIterator type="xs:string">
+          <FnDataIterator>
+            <CtxVarIterator varid="4" varname="xmlcontents" varkind="local"/>
+          </FnDataIterator>
+        </PromoteIterator>
+        <TreatIterator type="[NodeXQType elementNode nametest=[uri: http://zorba.io/modules/xml-options, local: options] content=[XQType ANY_TYPE_KIND*]]" quant="?">
+          <ElementIterator>
+            <SingletonIterator value="xs:QName(http://zorba.io/modules/xml-options,opt,options)"/>
+            <ElementIterator>
+              <SingletonIterator value="xs:QName(http://zorba.io/modules/xml-options,opt,parse-external-parsed-entity)"/>
+              <AttributeIterator qname="xs:QName(http://zorba.io/modules/xml-options,opt,skip-root-nodes)">
+                <SingletonIterator value="xs:string(1)"/>
+              </AttributeIterator>
+            </ElementIterator>
+          </ElementIterator>
+        </TreatIterator>
+      </FnZorbaParseXmlFragmentIterator>
+    </ForVariable>
+    <MaterializeClause>
+      <MaterializeForVariable inputVar="w : "/>
+    </MaterializeClause>
+    <ReturnClause>
+      <SequentialIterator>
+        <ApplyIterator>
+          <FnConcatIterator/>
+        </ApplyIterator>
+        <ForVarIterator varname="w"/>
+      </SequentialIterator>
+    </ReturnClause>
+  </flwor::FLWORIterator>
+</SequentialIterator>
+
+Iterator tree for fetch:content:
+<FunctionTraceIterator>
+  <FetchContentIterator>
+    <LetVarIterator varname="uri"/>
+    <SingletonIterator value="xs:string(SOME_CONTENT)"/>
+    <SingletonIterator value="xs:string(UTF-8)"/>
+  </FetchContentIterator>
+</FunctionTraceIterator>
+
 
=== added file 'test/rbkt/ExpCompilerResults/IterPlan/zorba/flwor/flwor25.iter'
--- test/rbkt/ExpCompilerResults/IterPlan/zorba/flwor/flwor25.iter	1970-01-01 00:00:00 +0000
+++ test/rbkt/ExpCompilerResults/IterPlan/zorba/flwor/flwor25.iter	2013-08-16 07:08:59 +0000
@@ -0,0 +1,56 @@
+Iterator tree for const-folded expr:
+<FunctionTraceIterator>
+  <FnConcatIterator/>
+</FunctionTraceIterator>
+
+Iterator tree for main query:
+<SequentialIterator>
+  <CtxVarDeclareIterator varid="4" varname="xmlcontents">
+    <UDFunctionCallIterator function="fetch:content">
+      <SingletonIterator value="xs:string($RBKT_SRC_DIR/Queries/zorba/flwor/flwor24.xml)"/>
+    </UDFunctionCallIterator>
+  </CtxVarDeclareIterator>
+  <CtxVarDeclareIterator varid="5" varname="contents">
+    <FnZorbaParseXmlFragmentIterator>
+      <PromoteIterator type="xs:string">
+        <FnDataIterator>
+          <CtxVarIterator varid="4" varname="xmlcontents" varkind="local"/>
+        </FnDataIterator>
+      </PromoteIterator>
+      <TreatIterator type="[NodeXQType elementNode nametest=[uri: http://zorba.io/modules/xml-options, local: options] content=[XQType ANY_TYPE_KIND*]]" quant="?">
+        <ElementIterator>
+          <SingletonIterator value="xs:QName(http://zorba.io/modules/xml-options,opt,options)"/>
+          <ElementIterator>
+            <SingletonIterator value="xs:QName(http://zorba.io/modules/xml-options,opt,parse-external-parsed-entity)"/>
+            <AttributeIterator qname="xs:QName(http://zorba.io/modules/xml-options,opt,skip-root-nodes)">
+              <SingletonIterator value="xs:string(1)"/>
+            </AttributeIterator>
+          </ElementIterator>
+        </ElementIterator>
+      </TreatIterator>
+    </FnZorbaParseXmlFragmentIterator>
+  </CtxVarDeclareIterator>
+  <flwor::FLWORIterator>
+    <ForVariable name="w">
+      <CtxVarIterator varid="5" varname="contents" varkind="local"/>
+    </ForVariable>
+    <ReturnClause>
+      <SequentialIterator>
+        <ApplyIterator>
+          <FnConcatIterator/>
+        </ApplyIterator>
+        <ForVarIterator varname="w"/>
+      </SequentialIterator>
+    </ReturnClause>
+  </flwor::FLWORIterator>
+</SequentialIterator>
+
+Iterator tree for fetch:content:
+<FunctionTraceIterator>
+  <FetchContentIterator>
+    <LetVarIterator varname="uri"/>
+    <SingletonIterator value="xs:string(SOME_CONTENT)"/>
+    <SingletonIterator value="xs:string(UTF-8)"/>
+  </FetchContentIterator>
+</FunctionTraceIterator>
+
 
=== added file 'test/rbkt/ExpCompilerResults/IterPlan/zorba/flwor/flwor26.iter'
--- test/rbkt/ExpCompilerResults/IterPlan/zorba/flwor/flwor26.iter	1970-01-01 00:00:00 +0000
+++ test/rbkt/ExpCompilerResults/IterPlan/zorba/flwor/flwor26.iter	2013-08-16 07:08:59 +0000
@@ -0,0 +1,66 @@
+Iterator tree for main query:
+<SequentialIterator>
+  <CtxVarDeclareIterator varid="4" varname="coll">
+    <SingletonIterator value="xs:QName(http://www.zorba-xquery.com/modules/store/dynamic/collections/ddl,ddl,coll1)"/>
+  </CtxVarDeclareIterator>
+  <SequentialIterator>
+    <ApplyIterator>
+      <ZorbaCreateCollectionIterator>
+        <CtxVarIterator varid="4" varname="coll" varkind="global"/>
+      </ZorbaCreateCollectionIterator>
+    </ApplyIterator>
+    <ApplyIterator>
+      <ZorbaInsertNodesLastIterator is-dynamic="true" need-to-copy="true">
+        <CtxVarIterator varid="4" varname="coll" varkind="global"/>
+        <FnConcatIterator>
+          <ElementIterator>
+            <SingletonIterator value="xs:QName(,,a)"/>
+          </ElementIterator>
+          <ElementIterator>
+            <SingletonIterator value="xs:QName(,,c)"/>
+          </ElementIterator>
+          <ElementIterator>
+            <SingletonIterator value="xs:QName(,,b)"/>
+          </ElementIterator>
+        </FnConcatIterator>
+      </ZorbaInsertNodesLastIterator>
+    </ApplyIterator>
+    <flwor::FLWORIterator>
+      <ForVariable name="w">
+        <ZorbaCollectionIterator>
+          <CtxVarIterator varid="4" varname="coll" varkind="global"/>
+        </ZorbaCollectionIterator>
+      </ForVariable>
+      <ReturnClause>
+        <SequentialIterator>
+          <ApplyIterator>
+            <FunctionTraceIterator>
+              <IfThenElseIterator>
+                <TypedValueCompareIterator_INTEGER>
+                  <ForVarIterator varname="pos"/>
+                  <SingletonIterator value="xs:integer(1)"/>
+                </TypedValueCompareIterator_INTEGER>
+                <ApplyIterator>
+                  <ZorbaInsertNodesLastIterator is-dynamic="true" need-to-copy="true">
+                    <CtxVarIterator varid="4" varname="coll" varkind="global"/>
+                    <ElementIterator>
+                      <SingletonIterator value="xs:QName(,,d)"/>
+                      <EnclosedIterator attr_cont="false">
+                        <ForVarIterator varname="pos"/>
+                      </EnclosedIterator>
+                    </ElementIterator>
+                  </ZorbaInsertNodesLastIterator>
+                </ApplyIterator>
+                <ApplyIterator>
+                  <FnConcatIterator/>
+                </ApplyIterator>
+              </IfThenElseIterator>
+            </FunctionTraceIterator>
+          </ApplyIterator>
+          <ForVarIterator varname="w"/>
+        </SequentialIterator>
+      </ReturnClause>
+    </flwor::FLWORIterator>
+  </SequentialIterator>
+</SequentialIterator>
+
 
=== added file 'test/rbkt/ExpCompilerResults/IterPlan/zorba/flwor/flwor27.iter'
--- test/rbkt/ExpCompilerResults/IterPlan/zorba/flwor/flwor27.iter	1970-01-01 00:00:00 +0000
+++ test/rbkt/ExpCompilerResults/IterPlan/zorba/flwor/flwor27.iter	2013-08-16 07:08:59 +0000
@@ -0,0 +1,63 @@
+Iterator tree for const-folded expr:
+<FunctionTraceIterator>
+  <FnConcatIterator/>
+</FunctionTraceIterator>
+
+Iterator tree for main query:
+<SequentialIterator>
+  <CtxVarDeclareIterator varid="4" varname="xmlcontents">
+    <UDFunctionCallIterator function="fetch:content">
+      <SingletonIterator value="xs:string($RBKT_SRC_DIR/Queries/zorba/flwor/flwor24.xml)"/>
+    </UDFunctionCallIterator>
+  </CtxVarDeclareIterator>
+  <CtxVarDeclareIterator varid="5" varname="contents">
+    <FnZorbaParseXmlFragmentIterator>
+      <PromoteIterator type="xs:string">
+        <FnDataIterator>
+          <CtxVarIterator varid="4" varname="xmlcontents" varkind="local"/>
+        </FnDataIterator>
+      </PromoteIterator>
+      <TreatIterator type="[NodeXQType elementNode nametest=[uri: http://zorba.io/modules/xml-options, local: options] content=[XQType ANY_TYPE_KIND*]]" quant="?">
+        <ElementIterator>
+          <SingletonIterator value="xs:QName(http://zorba.io/modules/xml-options,opt,options)"/>
+          <ElementIterator>
+            <SingletonIterator value="xs:QName(http://zorba.io/modules/xml-options,opt,parse-external-parsed-entity)"/>
+            <AttributeIterator qname="xs:QName(http://zorba.io/modules/xml-options,opt,skip-root-nodes)">
+              <SingletonIterator value="xs:string(1)"/>
+            </AttributeIterator>
+          </ElementIterator>
+        </ElementIterator>
+      </TreatIterator>
+    </FnZorbaParseXmlFragmentIterator>
+  </CtxVarDeclareIterator>
+  <flwor::FLWORIterator>
+    <ForVariable name="w">
+      <CtxVarIterator varid="5" varname="contents" varkind="local"/>
+    </ForVariable>
+    <OrderBySpec>
+      <FnLocalNameIterator>
+        <TreatIterator type="[NodeXQType anyNode content=[XQType ANY_TYPE_KIND*]]" quant="?">
+          <ForVarIterator varname="w"/>
+        </TreatIterator>
+      </FnLocalNameIterator>
+    </OrderBySpec>
+    <ReturnClause>
+      <SequentialIterator>
+        <ApplyIterator>
+          <FnConcatIterator/>
+        </ApplyIterator>
+        <ForVarIterator varname="w"/>
+      </SequentialIterator>
+    </ReturnClause>
+  </flwor::FLWORIterator>
+</SequentialIterator>
+
+Iterator tree for fetch:content:
+<FunctionTraceIterator>
+  <FetchContentIterator>
+    <LetVarIterator varname="uri"/>
+    <SingletonIterator value="xs:string(SOME_CONTENT)"/>
+    <SingletonIterator value="xs:string(UTF-8)"/>
+  </FetchContentIterator>
+</FunctionTraceIterator>
+
 
=== added file 'test/rbkt/ExpCompilerResults/IterPlan/zorba/flwor/flwor28.iter'
--- test/rbkt/ExpCompilerResults/IterPlan/zorba/flwor/flwor28.iter	1970-01-01 00:00:00 +0000
+++ test/rbkt/ExpCompilerResults/IterPlan/zorba/flwor/flwor28.iter	2013-08-16 07:08:59 +0000
@@ -0,0 +1,57 @@
+Iterator tree for const-folded expr:
+<FunctionTraceIterator>
+  <FnConcatIterator/>
+</FunctionTraceIterator>
+
+Iterator tree for main query:
+<SequentialIterator>
+  <CtxVarDeclareIterator varid="4" varname="coll">
+    <SingletonIterator value="xs:QName(http://www.zorba-xquery.com/modules/store/dynamic/collections/ddl,ddl,coll1)"/>
+  </CtxVarDeclareIterator>
+  <SequentialIterator>
+    <ApplyIterator>
+      <ZorbaCreateCollectionIterator>
+        <CtxVarIterator varid="4" varname="coll" varkind="global"/>
+      </ZorbaCreateCollectionIterator>
+    </ApplyIterator>
+    <ApplyIterator>
+      <ZorbaInsertNodesLastIterator is-dynamic="true" need-to-copy="true">
+        <CtxVarIterator varid="4" varname="coll" varkind="global"/>
+        <FnConcatIterator>
+          <ElementIterator>
+            <SingletonIterator value="xs:QName(,,a)"/>
+          </ElementIterator>
+          <ElementIterator>
+            <SingletonIterator value="xs:QName(,,c)"/>
+          </ElementIterator>
+          <ElementIterator>
+            <SingletonIterator value="xs:QName(,,b)"/>
+          </ElementIterator>
+        </FnConcatIterator>
+      </ZorbaInsertNodesLastIterator>
+    </ApplyIterator>
+    <flwor::FLWORIterator>
+      <ForVariable name="w">
+        <ZorbaCollectionIterator>
+          <CtxVarIterator varid="4" varname="coll" varkind="global"/>
+        </ZorbaCollectionIterator>
+      </ForVariable>
+      <OrderBySpec>
+        <FnLocalNameIterator>
+          <TreatIterator type="[NodeXQType anyNode content=[XQType ANY_TYPE_KIND*]]" quant="?">
+            <ForVarIterator varname="w"/>
+          </TreatIterator>
+        </FnLocalNameIterator>
+      </OrderBySpec>
+      <ReturnClause>
+        <SequentialIterator>
+          <ApplyIterator>
+            <FnConcatIterator/>
+          </ApplyIterator>
+          <ForVarIterator varname="w"/>
+        </SequentialIterator>
+      </ReturnClause>
+    </flwor::FLWORIterator>
+  </SequentialIterator>
+</SequentialIterator>
+
 
=== added file 'test/rbkt/ExpQueryResults/zorba/flwor/flwor24.xml.res'
--- test/rbkt/ExpQueryResults/zorba/flwor/flwor24.xml.res	1970-01-01 00:00:00 +0000
+++ test/rbkt/ExpQueryResults/zorba/flwor/flwor24.xml.res	2013-08-16 07:08:59 +0000
@@ -0,0 +1,1 @@
+<a/><c/><b/>
 \ No newline at end of file
 
=== added file 'test/rbkt/ExpQueryResults/zorba/flwor/flwor25.xml.res'
--- test/rbkt/ExpQueryResults/zorba/flwor/flwor25.xml.res	1970-01-01 00:00:00 +0000
+++ test/rbkt/ExpQueryResults/zorba/flwor/flwor25.xml.res	2013-08-16 07:08:59 +0000
@@ -0,0 +1,1 @@
+<a/><c/><b/>
 \ No newline at end of file
 
=== added file 'test/rbkt/ExpQueryResults/zorba/flwor/flwor27.xml.res'
--- test/rbkt/ExpQueryResults/zorba/flwor/flwor27.xml.res	1970-01-01 00:00:00 +0000
+++ test/rbkt/ExpQueryResults/zorba/flwor/flwor27.xml.res	2013-08-16 07:08:59 +0000
@@ -0,0 +1,1 @@
+<c/><b/><a/>
 \ No newline at end of file
 
=== added file 'test/rbkt/ExpQueryResults/zorba/flwor/flwor28.xml.res'
--- test/rbkt/ExpQueryResults/zorba/flwor/flwor28.xml.res	1970-01-01 00:00:00 +0000
+++ test/rbkt/ExpQueryResults/zorba/flwor/flwor28.xml.res	2013-08-16 07:08:59 +0000
@@ -0,0 +1,1 @@
+<c/><b/><a/>
 \ No newline at end of file
 
=== added file 'test/rbkt/ExpQueryResults/zorba/scripting/flwor2.xml.res'
--- test/rbkt/ExpQueryResults/zorba/scripting/flwor2.xml.res	1970-01-01 00:00:00 +0000
+++ test/rbkt/ExpQueryResults/zorba/scripting/flwor2.xml.res	2013-08-16 07:08:59 +0000
@@ -0,0 +1,2 @@
+<?xml version="1.0" encoding="UTF-8"?>
+true
 
=== removed file 'test/rbkt/ExpQueryResults/zorba/scripting/flwor2.xml.res'
--- test/rbkt/ExpQueryResults/zorba/scripting/flwor2.xml.res	2013-02-07 17:24:36 +0000
+++ test/rbkt/ExpQueryResults/zorba/scripting/flwor2.xml.res	1970-01-01 00:00:00 +0000
@@ -1,2 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-true
 
=== added file 'test/rbkt/Queries/zorba/flwor/flwor24.xml'
--- test/rbkt/Queries/zorba/flwor/flwor24.xml	1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/flwor/flwor24.xml	2013-08-16 07:08:59 +0000
@@ -0,0 +1,5 @@
+<root>
+  <a/>
+  <c/>
+  <b/>
+</root>
 \ No newline at end of file
 
=== added file 'test/rbkt/Queries/zorba/flwor/flwor24.xq'
--- test/rbkt/Queries/zorba/flwor/flwor24.xq	1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/flwor/flwor24.xq	2013-08-16 07:08:59 +0000
@@ -0,0 +1,25 @@
+import module namespace parse-xml = "http://zorba.io/modules/xml";
+import schema namespace opt = "http://zorba.io/modules/xml-options";
+import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch";
+
+declare namespace an = "http://zorba.io/annotations";
+
+declare %an:sequential function local:test-sequential()
+{
+  
+};
+ 
+variable $xmlcontents := fetch:content("$RBKT_SRC_DIR/Queries/zorba/flwor/flwor24.xml");
+
+let $contents := parse-xml:parse(
+      $xmlcontents,
+      <opt:options>
+        <opt:parse-external-parsed-entity opt:skip-root-nodes="1" />
+      </opt:options>)
+
+for $w at $pos in $contents
+return
+{
+    local:test-sequential();
+    $w
+}
 \ No newline at end of file
 
=== added file 'test/rbkt/Queries/zorba/flwor/flwor25.xq'
--- test/rbkt/Queries/zorba/flwor/flwor25.xq	1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/flwor/flwor25.xq	2013-08-16 07:08:59 +0000
@@ -0,0 +1,29 @@
+import module namespace parse-xml = "http://zorba.io/modules/xml";
+import schema namespace opt = "http://zorba.io/modules/xml-options";
+import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch";
+
+declare namespace an = "http://zorba.io/annotations";
+declare namespace ext = "http://zorba.io/extensions";
+
+declare %an:sequential function local:test-sequential()
+{
+  
+};
+ 
+variable $xmlcontents := fetch:content("$RBKT_SRC_DIR/Queries/zorba/flwor/flwor24.xml");
+
+variable $contents := parse-xml:parse(
+      $xmlcontents,
+      <opt:options>
+        <opt:parse-external-parsed-entity opt:skip-root-nodes="1" />
+      </opt:options>);
+
+(# ext:no-materialization #)
+{
+  for $w at $pos in $contents
+  return
+  {
+    local:test-sequential();
+    $w
+  }
+}
 \ No newline at end of file
 
=== added file 'test/rbkt/Queries/zorba/flwor/flwor26.spec'
--- test/rbkt/Queries/zorba/flwor/flwor26.spec	1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/flwor/flwor26.spec	2013-08-16 07:08:59 +0000
@@ -0,0 +1,1 @@
+Error: http://zorba.io/errors:ZDDY0041
 \ No newline at end of file
 
=== added file 'test/rbkt/Queries/zorba/flwor/flwor26.xq'
--- test/rbkt/Queries/zorba/flwor/flwor26.xq	1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/flwor/flwor26.xq	2013-08-16 07:08:59 +0000
@@ -0,0 +1,30 @@
+import module namespace parse-xml = "http://zorba.io/modules/xml";
+import schema namespace opt = "http://zorba.io/modules/xml-options";
+import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch";
+import module namespace ddl = "http://www.zorba-xquery.com/modules/store/dynamic/collections/ddl";
+import module namespace dml = "http://www.zorba-xquery.com/modules/store/dynamic/collections/dml";
+
+declare namespace an = "http://zorba.io/annotations";
+declare namespace ext = "http://zorba.io/extensions";
+
+declare variable $coll := xs:QName("ddl:coll1");
+
+declare %an:sequential function local:test-sequential($pos)
+{
+  if ($pos eq 1)
+  then dml:insert-nodes-last($coll, <d>{$pos}</d>);
+  else ();
+};
+ 
+ddl:create($coll);
+dml:insert-nodes-last($coll, (<a/>,<c/>,<b/>));
+
+(# ext:no-materialization #)
+{
+  for $w at $pos in dml:collection($coll)
+  return
+  {
+    local:test-sequential($pos);
+    $w
+  }
+}
 \ No newline at end of file
 
=== added file 'test/rbkt/Queries/zorba/flwor/flwor27.xq'
--- test/rbkt/Queries/zorba/flwor/flwor27.xq	1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/flwor/flwor27.xq	2013-08-16 07:08:59 +0000
@@ -0,0 +1,30 @@
+import module namespace parse-xml = "http://zorba.io/modules/xml";
+import schema namespace opt = "http://zorba.io/modules/xml-options";
+import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch";
+
+declare namespace an = "http://zorba.io/annotations";
+declare namespace ext = "http://zorba.io/extensions";
+
+declare %an:sequential function local:test-sequential($pos)
+{
+  
+};
+ 
+variable $xmlcontents := fetch:content("$RBKT_SRC_DIR/Queries/zorba/flwor/flwor24.xml");
+
+variable $contents := parse-xml:parse(
+      $xmlcontents,
+      <opt:options>
+        <opt:parse-external-parsed-entity opt:skip-root-nodes="1" />
+      </opt:options>);
+
+(# ext:no-materialization #)
+{
+  for $w at $pos in $contents
+  order by local-name($w) descending
+  return
+  {
+    local:test-sequential($pos);
+    $w
+  }
+}
 \ No newline at end of file
 
=== added file 'test/rbkt/Queries/zorba/flwor/flwor28.xq'
--- test/rbkt/Queries/zorba/flwor/flwor28.xq	1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/flwor/flwor28.xq	2013-08-16 07:08:59 +0000
@@ -0,0 +1,29 @@
+import module namespace parse-xml = "http://zorba.io/modules/xml";
+import schema namespace opt = "http://zorba.io/modules/xml-options";
+import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch";
+import module namespace ddl = "http://www.zorba-xquery.com/modules/store/dynamic/collections/ddl";
+import module namespace dml = "http://www.zorba-xquery.com/modules/store/dynamic/collections/dml";
+
+declare namespace an = "http://zorba.io/annotations";
+declare namespace ext = "http://zorba.io/extensions";
+
+declare variable $coll := xs:QName("ddl:coll1");
+
+declare %an:sequential function local:test-sequential($pos)
+{
+  
+};
+ 
+ddl:create($coll);
+dml:insert-nodes-last($coll, (<a/>,<c/>,<b/>));
+
+(# ext:no-materialization #)
+{
+  for $w at $pos in dml:collection($coll)
+  order by local-name($w) descending
+  return
+  {
+    local:test-sequential($pos);
+    $w
+  }
+}
 \ No newline at end of file
 
=== added file 'test/rbkt/Queries/zorba/scripting/flwor2.spec'
--- test/rbkt/Queries/zorba/scripting/flwor2.spec	1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/scripting/flwor2.spec	2013-08-16 07:08:59 +0000
@@ -0,0 +1,1 @@
+Error: http://www.zorba-xquery.com/errors:ZDDY0037