Merge lp:~zorba-coders/zorba/feature-fetch_binary into lp:zorba

[Matthias Brantner]

- fetch:content-binary
- fetch:content#3 (with encoding parameter)
- StreamResource::isStreamSeekable to make sure the streamable strings returned by fetch are seekable.

- fetch:content-binary
- fetch:content#3 (with encoding parameter)
- StreamResource::isStreamSeekable to make sure the streamable strings returned by fetch are seekable.

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/feature-fetch_binary-2012-05-09T01-28-08.563Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~zorba-coders/zorba/feature-fetch_binary into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job feature-fetch_binary-2012-05-09T01-28-08.563Z is
  finished. The final status was:

  13 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[lp:~zorba-coders/zorba/feature-fetch_binary]

10834. By Matthias Brantner on 2012-05-09

- documentation
- more robust tests

10835. By Matthias Brantner on 2012-05-09

fixed unintended indentation change

10836. By Matthias Brantner on 2012-05-10

improved error messages + warning fixes

10837. By Matthias Brantner on 2012-05-10

increase major version number of fetch module

10838. By Matthias Brantner on 2012-05-11

fetch became non-pure

10839. By Matthias Brantner on 2012-05-11

expose the SOME_CONTENT entity in the api

10840. By Matthias Brantner on 2012-05-11

merge + changelog

10841. By Matthias Brantner on 2012-05-16

change version of the fetch module (3.0 => 2.1)

10842. By Matthias Brantner on 2012-05-16

improved documentation

10843. By Matthias Brantner on 2012-05-18

code improvements as suggested by review

10844. By Matthias Brantner on 2012-05-18

added StaticContext::fetchBinary + StaticContext::fetch with encoding to C++ api

10845. By Matthias Brantner on 2012-05-18

documentation improvements

10846. By Matthias Brantner on 2012-05-18

merge

Unmerged revisions

=== modified file 'ChangeLog'
=== modified file 'include/zorba/uri_resolvers.h'
--- include/zorba/uri_resolvers.h	2012-05-03 12:31:51 +0000
+++ include/zorba/uri_resolvers.h	2012-05-09 21:52:19 +0000
@@ -84,9 +84,11 @@
    * @param aStreamReleaser A function pointer which is invoked once
    *        the StreamResource is destroyed. Normally this function will delete
    *        the std::istream object passed to it.
+   * @param aIsStreamSeekable Determines whether the given stream is seekable.
    */
   static StreamResource* create(std::istream* aStream,
-                                StreamReleaser aStreamReleaser);
+                                StreamReleaser aStreamReleaser,
+                                bool aIsStreamSeekable = false);
   
   /**
    * @brief Retrieve the istream associated with this Resource.
@@ -99,6 +101,8 @@
   virtual StreamReleaser getStreamReleaser() = 0;
 
   virtual ~StreamResource() = 0;
+
+  virtual bool isStreamSeekable() const = 0;
 };
 
 /**

=== modified file 'modules/com/zorba-xquery/www/modules/CMakeLists.txt'
--- modules/com/zorba-xquery/www/modules/CMakeLists.txt	2012-05-03 12:31:51 +0000
+++ modules/com/zorba-xquery/www/modules/CMakeLists.txt	2012-05-09 21:52:19 +0000
@@ -43,7 +43,7 @@
 DECLARE_ZORBA_SCHEMA(FILE xqdoc.xsd URI "http://www.xqdoc.org/1.0")
 DECLARE_ZORBA_MODULE(FILE datetime.xq VERSION 2.0
   URI "http://www.zorba-xquery.com/modules/datetime")
-DECLARE_ZORBA_MODULE(FILE fetch.xq VERSION 2.0
+DECLARE_ZORBA_MODULE(FILE fetch.xq VERSION 2.1
   URI "http://www.zorba-xquery.com/modules/fetch")
 DECLARE_ZORBA_MODULE(FILE math.xq VERSION 2.0
   URI "http://www.zorba-xquery.com/modules/math")

=== modified file 'modules/com/zorba-xquery/www/modules/fetch.xq'
--- modules/com/zorba-xquery/www/modules/fetch.xq	2012-05-03 12:31:51 +0000
+++ modules/com/zorba-xquery/www/modules/fetch.xq	2012-05-09 21:52:19 +0000
@@ -17,15 +17,13 @@
 :)
 
 (:~
- : This module provides functions to fetch the contents or the type
- : of the content for a resource identified by a URI.
- : For example, it fetches content for file or http resources if Zorba
- : allows file or http access, respectively.
- :
- : <p>The errors raised by functions of this module have the namespace
- : <tt>http://www.zorba-xquery.com/errors</tt> (associated with prefix zerr).</p>
- :
- : @see <a href="www.zorba-xquery.com_errors.html">http://www.zorba-xquery.com/errors</a>
+ : <p>This module provides functions to fetch the content of a resource identified
+ : by a URI. For example, it fetches the content of file or http resources.</p>
+ :
+ : <p>In order to retrieve such content, the functions use the
+ : URI resolution and URL resolver process as documented at
+ : <a href="../../html/uriresolvers.html">
+ : URI Resolvers</a>.</p>
  :
  : @author Matthias Brantner
  :
@@ -38,23 +36,15 @@
 declare namespace zerr = "http://www.zorba-xquery.com/errors";
 
 declare namespace ver = "http://www.zorba-xquery.com/options/versioning";
-declare option ver:module-version "2.0";
+declare option ver:module-version "2.1";
 
 (:~
  : <p>Tries to fetch the resource referred to by the given URI.</p>
  :
- : <p>In order to retrieve the content, the functions uses the
- : URI resolution and URL resolver process as documented at
- : <a href="../../html/uriresolvers.html">
- : URI Resolvers</a>. Therefore, it queries all URI mappers
- : and resolvers with kind <tt>EntityData::SOME_CONTENT</tt>.</p>
+ : <p>It queries all URI mappers and resolvers with kind
+ : <tt>EntityData::SOME_CONTENT</tt>.</p>
  :
- : <p>The function is annotated with the <tt>an:streamable</tt>
- : annotation, that is it returns a streamable string. A streamable
- : string can only be consumed once. Please see section "Streamable Strings"
- : in the <a href="../../html/options_and_annotations.html">
- : documentation of Zorba's annotations</a>.
- : </p>
+ : <p>The content is assumed to be UTF-8 encoded.</p>
  :
  : @param $uri the resource to fetch.
  : @return the resource referred to by the given URI as streamble string.
@@ -73,28 +63,85 @@
 (:~
  : <p>Tries to fetch the resource referred to by the given URI.</p>
  :
- : <p>In order to retrieve the content, the functions uses the
- : URI resolution and URL resolver process as documented at
- : <a href="../../html/uriresolvers.html">
- : URI Resolvers</a>. Therefore, it queries all URI mappers
- : and resolvers with the specified entity kind.</p>
- :
- : <p>The function is annotated with the <tt>an:streamable</tt>
- : annotation, that is it returns a streamable string. A streamable
- : string can only be consumed once. Please see section "Streamable Strings"
- : in the <a href="../../html/options_and_annotations.html">
- : documentation of Zorba's annotations</a>.
- : </p>
- :
- : @param $uri the resource to fetch.
- : @param $entityKind the kind of resource to fetch.
- : @return the resource referred to by the given URI as streamble string.
- :
- : @error zerr:ZXQP0025 if the URI could not be resolved
- :   or did not resolve to a <tt>StreamResource</tt>.
- :
- : @see <a href="../../html/uriresolvers.html">URI Resolvers</a>.
- : @see <a href="../../html/options_and_annotations.html">Documentation of Zorba's annotations</a>.
- :)
-
-declare %an:streamable function fetch:content($uri as xs:string, $entityKind as xs:string) as xs:string external;
+ : <p>It queries all URI mappers and resolvers with kind the specified
+ : entity kind.</p>
+ :
+ : <p>The content is assumed to be UTF-8 encoded.</p>
+ :
+ : @param $uri the resource to fetch.
+ : @param $entity-kind the kind of resource to fetch.
+ : @return the resource referred to by the given URI as streamble string.
+ :
+ : @error zerr:ZXQP0025 if the URI could not be resolved
+ :   or did not resolve to a <tt>StreamResource</tt>.
+ :
+ : @see <a href="../../html/uriresolvers.html">URI Resolvers</a>.
+ : @see <a href="../../html/options_and_annotations.html">Documentation of Zorba's annotations</a>.
+ :)
+declare %an:streamable function fetch:content($uri as xs:string, $entity-kind as xs:string)
+as xs:string
+{
+  fetch:content($uri, $entity-kind, "UTF-8")
+};
+
+(:~
+ : <p>Tries to fetch the resource referred to by the given URI.</p>
+ :
+ : <p>It queries all URI mappers and resolvers with kind the specified
+ : entity kind.</p>
+ :
+ : @param $uri the resource to fetch.
+ : @param $entity-kind the kind of resource to fetch.
+ : @param $encoding the encoding of the content
+ : @return the resource referred to by the given URI as streamble string.
+ :
+ : @error zerr:ZXQP0025 if the URI could not be resolved
+ :   or did not resolve to a <tt>StreamResource</tt>.
+ : @error zerr:ZXQP0006 if the given encoding is invalid or not supported.
+ :
+ : @see <a href="../../html/uriresolvers.html">URI Resolvers</a>.
+ : @see <a href="../../html/options_and_annotations.html">Documentation of Zorba's annotations</a>.
+ :)
+declare %an:streamable function fetch:content(
+  $uri as xs:string,
+  $entity-kind as xs:string,
+  $encoding as xs:string)
+as xs:string external;
+
+(:~
+ : <p>Tries to fetch the resource referred to by the given URI and
+ : returning it as base64Binary.</p>
+ :
+ : <p>It queries all URI mappers and resolvers with kind
+ : <tt>EntityData::SOME_CONTENT</tt>.</p>
+
+ : @param $uri the resource to fetch.
+ : @return the resource referred to by the given URI as streamble base64Binary.
+ :
+ : @error zerr:ZXQP0025 if the URI could not be resolved
+ :   or did not resolve to a <tt>StreamResource</tt>.
+ :)
+declare %an:streamable function fetch:content-binary($uri as xs:string)
+as xs:base64Binary
+{
+  fetch:content-binary($uri, "SOME_CONTENT")
+};
+
+(:~
+ : <p>Tries to fetch the resource referred to by the given URI and
+ : returning it as base64Binary.</p>
+ :
+ : <p>It queries all URI mappers and resolvers with kind the specified
+ : entity kind.</p>
+ :
+ : @param $uri the resource to fetch.
+ : @param $entity-kind the kind of resource to fetch.
+ : @return the resource referred to by the given URI as streamble base64Binary.
+ :
+ : @error zerr:ZXQP0025 if the URI could not be resolved
+ :   or did not resolve to a <tt>StreamResource</tt>.
+ :)
+declare %an:streamable function fetch:content-binary(
+  $uri as xs:string,
+  $entity-kind as xs:string)
+as xs:base64Binary external;

=== modified file 'src/api/uri_resolver_wrappers.cpp'
--- src/api/uri_resolver_wrappers.cpp	2012-05-03 12:31:51 +0000
+++ src/api/uri_resolver_wrappers.cpp	2012-05-09 21:52:19 +0000
@@ -146,7 +146,9 @@
         // StreamResource, by passing the StreamReleaser to it and setting the
         // user's StreamResource's StreamReleaser to nullptr.
         lRetval = new internal::StreamResource(lUserStream->getStream(),
-                                           lUserStream->getStreamReleaser());
+                                           lUserStream->getStreamReleaser(),
+                                           "",
+                                           lUserStream->isStreamSeekable());
         lUserStream->setStreamReleaser(nullptr);
       }
 #ifndef ZORBA_NO_FULL_TEXT

=== modified file 'src/api/uriresolverimpl.cpp'
--- src/api/uriresolverimpl.cpp	2012-05-03 12:31:51 +0000
+++ src/api/uriresolverimpl.cpp	2012-05-09 21:52:19 +0000
@@ -43,15 +43,18 @@
   }
 
   StreamResource* StreamResource::create(std::istream* aStream,
-                                         StreamReleaser aStreamReleaser)
+                                         StreamReleaser aStreamReleaser,
+                                         bool aIsStreamSeekable)
   {
-    return new StreamResourceImpl(aStream, aStreamReleaser);
+    return new StreamResourceImpl(aStream, aStreamReleaser, aIsStreamSeekable);
   }
 
   StreamResourceImpl::StreamResourceImpl(std::istream* aStream,
-                                         StreamReleaser aStreamReleaser)
+                                         StreamReleaser aStreamReleaser,
+                                         bool aIsStreamSeekable)
     : theStream(aStream),
-      theStreamReleaser(aStreamReleaser)
+      theStreamReleaser(aStreamReleaser),
+      theIsStreamSeekable(aIsStreamSeekable)
   {
   }
 

=== modified file 'src/api/uriresolverimpl.h'
--- src/api/uriresolverimpl.h	2012-05-03 12:31:51 +0000
+++ src/api/uriresolverimpl.h	2012-05-09 21:52:19 +0000
@@ -39,14 +39,21 @@
 
   virtual void destroy() const;
 
+  virtual bool isStreamSeekable() const { return theIsStreamSeekable; }
+
 private:
 
-  StreamResourceImpl(std::istream* aStream, StreamReleaser aStreamReleaser);
+  StreamResourceImpl(
+      std::istream* aStream,
+      StreamReleaser aStreamReleaser,
+      bool aIsStreamSeekable);
 
   friend StreamResource* StreamResource::create(std::istream *aStream,
-                                                StreamReleaser aStreamReleaser);
+                                                StreamReleaser aStreamReleaser,
+                                                bool aIsStreamSeekable);
   std::istream* theStream;
   StreamReleaser theStreamReleaser;
+  bool theIsStreamSeekable;
 
 };
 

=== modified file 'src/context/default_url_resolvers.cpp'
--- src/context/default_url_resolvers.cpp	2012-05-03 12:31:51 +0000
+++ src/context/default_url_resolvers.cpp	2012-05-09 21:52:19 +0000
@@ -107,7 +107,8 @@
   zstring lPath = fs::get_normalized_path(aUrl);
   if (fs::get_type(lPath) == fs::file) {
     std::ifstream* lStream = new std::ifstream(lPath.c_str());
-    return new StreamResource(lStream, &fileStreamReleaser);
+    return new StreamResource(
+        lStream, &fileStreamReleaser, "", true /* seekable */);
   }
   return NULL;
 }

=== modified file 'src/context/uri_resolver.cpp'
--- src/context/uri_resolver.cpp	2012-05-03 12:31:51 +0000
+++ src/context/uri_resolver.cpp	2012-05-09 21:52:19 +0000
@@ -46,11 +46,13 @@
 
   StreamResource::StreamResource
   (std::istream* aStream, StreamReleaser aStreamReleaser,
-   zstring aStreamUrl /* = "" */)
+   zstring aStreamUrl /* = "" */,
+   bool aIsStreamSeekable)
     : Resource(),
       theStream(aStream),
       theStreamReleaser(aStreamReleaser),
-      theStreamUrl(aStreamUrl)
+      theStreamUrl(aStreamUrl),
+      theIsStreamSeekable(aIsStreamSeekable)
   {}
   
   StreamResource::~StreamResource()

=== modified file 'src/context/uri_resolver.h'
--- src/context/uri_resolver.h	2012-05-03 12:31:51 +0000
+++ src/context/uri_resolver.h	2012-05-09 21:52:19 +0000
@@ -97,10 +97,13 @@
    * are certain unusual circumstances where a URLResolver may wish to
    * return a stream over some other URL than the one passed to it. In
    * that case, the URLResolver may pass the true URL here.
+   * @param aIsStreamSeekable determines whether the stream passed as first
+   * argument is seekable.
    */
   StreamResource(std::istream* aStream,
                  StreamReleaser aStreamReleaser,
-                 zstring aStreamUrl = "");
+                 zstring aStreamUrl = "",
+                 bool aIsStreamSeekable = false);
   
   virtual ~StreamResource();
 
@@ -127,11 +130,18 @@
    */
   zstring getStreamUrl();
 
+  /**
+   * @brief Returns true if the stream returned by getStream is seekable,
+   * false otherwise.
+   */
+  bool isStreamSeekable() const { return theIsStreamSeekable; }
+
 private:
 
   std::istream* theStream;
   StreamReleaser theStreamReleaser;
   zstring theStreamUrl;
+  bool theIsStreamSeekable;
 };
 
 /**

=== modified file 'src/functions/pregenerated/func_fetch.cpp'
--- src/functions/pregenerated/func_fetch.cpp	2012-05-03 12:31:51 +0000
+++ src/functions/pregenerated/func_fetch.cpp	2012-05-09 21:52:19 +0000
@@ -41,6 +41,16 @@
   return new FetchContentIterator(sctx, loc, argv);
 }
 
+PlanIter_t fn_zorba_fetch_content_binary::codegen(
+  CompilerCB*,
+  static_context* sctx,
+  const QueryLoc& loc,
+  std::vector<PlanIter_t>& argv,
+  expr& ann) const
+{
+  return new FetchContentBinaryIterator(sctx, loc, argv);
+}
+
 PlanIter_t fn_zorba_fetch_content_type::codegen(
   CompilerCB*,
   static_context* sctx,
@@ -60,8 +70,22 @@
         (createQName("http://www.zorba-xquery.com/modules/fetch","","content"), 
         GENV_TYPESYSTEM.STRING_TYPE_ONE, 
         GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
         GENV_TYPESYSTEM.STRING_TYPE_ONE),
-        FunctionConsts::FN_ZORBA_FETCH_CONTENT_2);
+        FunctionConsts::FN_ZORBA_FETCH_CONTENT_3);
+
+  }
+
+
+  {
+    
+
+    DECL_WITH_KIND(sctx, fn_zorba_fetch_content_binary,
+        (createQName("http://www.zorba-xquery.com/modules/fetch","","content-binary"), 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.STRING_TYPE_ONE, 
+        GENV_TYPESYSTEM.BASE64BINARY_TYPE_ONE),
+        FunctionConsts::FN_ZORBA_FETCH_CONTENT_BINARY_2);
 
   }
 

=== modified file 'src/functions/pregenerated/func_fetch.h'
--- src/functions/pregenerated/func_fetch.h	2012-05-03 12:31:51 +0000
+++ src/functions/pregenerated/func_fetch.h	2012-05-09 21:52:19 +0000
@@ -55,6 +55,23 @@
 };
 
 
+//fn-zorba-fetch:content-binary
+class fn_zorba_fetch_content_binary : public function
+{
+public:
+  fn_zorba_fetch_content_binary(const signature& sig, FunctionConsts::FunctionKind kind)
+    : 
+    function(sig, kind)
+  {
+
+  }
+
+  bool accessesDynCtx() const { return true; }
+
+  CODEGEN_DECL();
+};
+
+
 //fn-zorba-fetch:content-type
 class fn_zorba_fetch_content_type : public function
 {

=== modified file 'src/functions/pregenerated/function_enum.h'
--- src/functions/pregenerated/function_enum.h	2012-05-08 03:09:12 +0000
+++ src/functions/pregenerated/function_enum.h	2012-05-09 21:52:19 +0000
@@ -137,7 +137,8 @@
   FN_TRACE_2,
   OP_ZORBA_READ_LINE_0,
   FN_ZORBA_UTIL_PRINT_1,
-  FN_ZORBA_FETCH_CONTENT_2,
+  FN_ZORBA_FETCH_CONTENT_3,
+  FN_ZORBA_FETCH_CONTENT_BINARY_2,
   FN_ZORBA_FETCH_CONTENT_TYPE_1,
   FN_PUT_2,
   FULL_TEXT_CURRENT_LANG_0,

=== modified file 'src/runtime/fetch/fetch_impl.cpp'
--- src/runtime/fetch/fetch_impl.cpp	2012-05-03 12:31:51 +0000
+++ src/runtime/fetch/fetch_impl.cpp	2012-05-09 21:52:19 +0000
@@ -15,6 +15,8 @@
  */
 #include "stdafx.h"
 
+#include <zorba/transcode_stream.h>
+
 #include "diagnostics/assert.h"
 #include "diagnostics/xquery_diagnostics.h"
 
@@ -30,33 +32,17 @@
 
 /*******************************************************************************
 ********************************************************************************/
-void
-FetchContentIterator::destroyStream(std::istream& aStream)
-{
-  delete &aStream;
-}
-
-bool
-FetchContentIterator::nextImpl(
-    store::Item_t& result,
-    PlanState& aPlanState) const
-{
-  store::Item_t lUri;
-  store::Item_t lEntityKind;
+std::auto_ptr<internal::Resource>
+getFetchResource(
+    const store::Item_t& aUri,
+    const store::Item_t& aKind,
+    const static_context* aSctx,
+    const QueryLoc& aLoc)
+{
   internal::EntityData::Kind lKind;
-  zstring lKindStr;
-  zstring lErrorMessage;
-  std::auto_ptr<internal::Resource> lRes;
-  internal::StreamResource* lStreamRes;
-
-  PlanIteratorState* state;
-  DEFAULT_STACK_INIT(PlanIteratorState, state, aPlanState);
-
-  consumeNext(lUri, theChildren[0].getp(), aPlanState);
-  consumeNext(lEntityKind, theChildren[1].getp(), aPlanState);
+  zstring lKindStr = aKind->getStringValue();
 
   // Figure out the EntityKind (any better way to do this?)
-  lKindStr = lEntityKind->getStringValue();
   if ( ! lKindStr.compare("SOME_CONTENT")) {
     lKind = internal::EntityData::SOME_CONTENT;
   }
@@ -70,7 +56,7 @@
   else if ( ! lKindStr.compare("THESAURUS")) {
     lKind = internal::EntityData::THESAURUS;
   }
-  else if ( ! lKindStr.compare("STOP_WORDS")) {
+  else if ( ! lKindStr.compare("STOP_WORDS")){
     lKind = internal::EntityData::STOP_WORDS;
   }
 #endif /* ZORBA_NO_FULL_TEXT */
@@ -78,14 +64,15 @@
     throw XQUERY_EXCEPTION(
           zerr::ZXQP0026_INVALID_ENUM_VALUE,
           ERROR_PARAMS(lKindStr, "entityKind"),
-          ERROR_LOC(loc));
+          ERROR_LOC(aLoc));
   }
 
   try {
     // ask the uri mappers and resolvers to give
     // me a resource of specified kind
-    lRes = theSctx->resolve_uri(
-      lUri->getStringValue(),
+    zstring lErrorMessage;
+    return aSctx->resolve_uri(
+      aUri->getStringValue(),
       lKind,
       lErrorMessage);
 
@@ -93,9 +80,41 @@
     throw XQUERY_EXCEPTION(
       zerr::ZXQP0025_ITEM_CREATION_FAILED,
       ERROR_PARAMS( e.what() ),
-      ERROR_LOC( loc )
+      ERROR_LOC( aLoc )
     );
   }
+}
+
+/*******************************************************************************
+********************************************************************************/
+void
+FetchContentIterator::destroyStream(std::istream& aStream)
+{
+  delete &aStream;
+}
+
+bool
+FetchContentIterator::nextImpl(
+    store::Item_t& result,
+    PlanState& aPlanState) const
+{
+  store::Item_t lUri;
+  store::Item_t lEntityKind;
+  store::Item_t lEncoding;
+  zstring lEncodingStr("UTF-8");
+  std::auto_ptr<internal::Resource> lRes;
+  internal::StreamResource* lStreamRes;
+
+  PlanIteratorState* state;
+  DEFAULT_STACK_INIT(PlanIteratorState, state, aPlanState);
+
+  consumeNext(lUri, theChildren[0].getp(), aPlanState);
+  consumeNext(lEntityKind, theChildren[1].getp(), aPlanState);
+  consumeNext(lEncoding, theChildren[2].getp(), aPlanState);
+
+  lEncodingStr = lEncoding->getStringValue();
+
+  lRes = getFetchResource(lUri, lEntityKind, theSctx, loc);
 
   lStreamRes = dynamic_cast<internal::StreamResource*>(lRes.get());
   if ( !lStreamRes ) {
@@ -106,13 +125,80 @@
     );
   }
 
+  if (transcode::is_necessary(lEncodingStr.c_str()))
+  {
+    if (!transcode::is_supported(lEncodingStr.c_str()))
+    {
+      throw XQUERY_EXCEPTION(
+          zerr::ZXQP0006_UNKNOWN_ENCODING,
+          ERROR_PARAMS( lEncodingStr.c_str() ),
+          ERROR_LOC( loc )
+        );
+    }
+    transcode::attach(*lStreamRes->getStream(), lEncodingStr.c_str());
+  }
+
   // return the resource in a streamable string. This transfers memory
   // ownership of the istream (via its StreamReleaser) to the StreamableString
   // object, so we then remove the StreamReleaser from the StreamResource.
   GENV_ITEMFACTORY->createStreamableString(
         result,
         *lStreamRes->getStream(),
-        lStreamRes->getStreamReleaser()
+        lStreamRes->getStreamReleaser(),
+        lStreamRes->isStreamSeekable()
+  );
+  lStreamRes->setStreamReleaser(nullptr);
+
+  STACK_PUSH(result != NULL, state);
+
+  STACK_END(state);
+}
+
+
+/*******************************************************************************
+********************************************************************************/
+void
+FetchContentBinaryIterator::destroyStream(std::istream& aStream)
+{
+  delete &aStream;
+}
+
+bool
+FetchContentBinaryIterator::nextImpl(
+    store::Item_t& result,
+    PlanState& aPlanState) const
+{
+  store::Item_t lUri;
+  store::Item_t lEntityKind;
+  std::auto_ptr<internal::Resource> lRes;
+  internal::StreamResource* lStreamRes;
+
+  PlanIteratorState* state;
+  DEFAULT_STACK_INIT(PlanIteratorState, state, aPlanState);
+
+  consumeNext(lUri, theChildren[0].getp(), aPlanState);
+  consumeNext(lEntityKind, theChildren[1].getp(), aPlanState);
+
+  lRes = getFetchResource(lUri, lEntityKind, theSctx, loc);
+
+  lStreamRes = dynamic_cast<internal::StreamResource*>(lRes.get());
+  if ( !lStreamRes ) {
+    throw XQUERY_EXCEPTION(
+      zerr::ZXQP0025_ITEM_CREATION_FAILED,
+      ERROR_PARAMS( "Resource not available." ),
+      ERROR_LOC( loc )
+    );
+  }
+
+  // return the resource in a streamable base64. This transfers memory
+  // ownership of the istream (via its StreamReleaser) to the StreamableBase64BinaryItem
+  // object, so we then remove the StreamReleaser from the StreamResource.
+  GENV_ITEMFACTORY->createStreamableBase64Binary(
+        result,
+        *lStreamRes->getStream(),
+        lStreamRes->getStreamReleaser(),
+        lStreamRes->isStreamSeekable(),
+        false
   );
   lStreamRes->setStreamReleaser(nullptr);
 

=== modified file 'src/runtime/fetch/pregenerated/fetch.cpp'
--- src/runtime/fetch/pregenerated/fetch.cpp	2012-05-03 12:31:51 +0000
+++ src/runtime/fetch/pregenerated/fetch.cpp	2012-05-09 21:52:19 +0000
@@ -54,6 +54,28 @@
 // </FetchContentIterator>
 
 
+// <FetchContentBinaryIterator>
+FetchContentBinaryIterator::class_factory<FetchContentBinaryIterator>
+FetchContentBinaryIterator::g_class_factory;
+
+
+void FetchContentBinaryIterator::accept(PlanIterVisitor& v) const {
+  v.beginVisit(*this);
+
+  std::vector<PlanIter_t>::const_iterator lIter = theChildren.begin();
+  std::vector<PlanIter_t>::const_iterator lEnd = theChildren.end();
+  for ( ; lIter != lEnd; ++lIter ){
+    (*lIter)->accept(v);
+  }
+
+  v.endVisit(*this);
+}
+
+FetchContentBinaryIterator::~FetchContentBinaryIterator() {}
+
+// </FetchContentBinaryIterator>
+
+
 // <FetchContentTypeIterator>
 FetchContentTypeIterator::class_factory<FetchContentTypeIterator>
 FetchContentTypeIterator::g_class_factory;

=== modified file 'src/runtime/fetch/pregenerated/fetch.h'
--- src/runtime/fetch/pregenerated/fetch.h	2012-05-03 12:31:51 +0000
+++ src/runtime/fetch/pregenerated/fetch.h	2012-05-09 21:52:19 +0000
@@ -73,6 +73,42 @@
  * 
  * Author: Matthias Brantner
  */
+class FetchContentBinaryIterator : public NaryBaseIterator<FetchContentBinaryIterator, PlanIteratorState>
+{ 
+public:
+  SERIALIZABLE_CLASS(FetchContentBinaryIterator);
+
+  SERIALIZABLE_CLASS_CONSTRUCTOR2T(FetchContentBinaryIterator,
+    NaryBaseIterator<FetchContentBinaryIterator, PlanIteratorState>);
+
+  void serialize( ::zorba::serialization::Archiver& ar)
+  {
+    serialize_baseclass(ar,
+    (NaryBaseIterator<FetchContentBinaryIterator, PlanIteratorState>*)this);
+  }
+
+  FetchContentBinaryIterator(
+    static_context* sctx,
+    const QueryLoc& loc,
+    std::vector<PlanIter_t>& children)
+    : 
+    NaryBaseIterator<FetchContentBinaryIterator, PlanIteratorState>(sctx, loc, children)
+  {}
+
+  virtual ~FetchContentBinaryIterator();
+
+public:
+  static void destroyStream(std::istream& aStream);
+  void accept(PlanIterVisitor& v) const;
+
+  bool nextImpl(store::Item_t& result, PlanState& aPlanState) const;
+};
+
+
+/**
+ * 
+ * Author: Matthias Brantner
+ */
 class FetchContentTypeIterator : public NaryBaseIterator<FetchContentTypeIterator, PlanIteratorState>
 { 
 public:

=== modified file 'src/runtime/nodes/nodes_impl.cpp'
--- src/runtime/nodes/nodes_impl.cpp	2012-05-03 12:31:51 +0000
+++ src/runtime/nodes/nodes_impl.cpp	2012-05-09 21:52:19 +0000
@@ -638,11 +638,13 @@
   lIterator->open();
   while(lIterator->next(lItem))
   {
-    if(lItem->getNodeKind() == aNode->getNodeKind())
+    if (lItem->getNodeKind() == aNode->getNodeKind())
+    {
       if(lItem->equals(aNode))
         break;
       else
         count++;
+    }
   }
   lIterator->close();
   return count;
@@ -663,7 +665,7 @@
   PlanIteratorState* state;
   DEFAULT_STACK_INIT(PlanIteratorState, state, planState);
 
-  if (consumeNext(inNode, theChildren[0], planState));
+  if (consumeNext(inNode, theChildren[0], planState))
   {
     do
     { 

=== modified file 'src/runtime/spec/fetch/fetch.xml'
--- src/runtime/spec/fetch/fetch.xml	2012-05-03 12:31:51 +0000
+++ src/runtime/spec/fetch/fetch.xml	2012-05-09 21:52:19 +0000
@@ -16,7 +16,8 @@
   <zorba:function>
     <zorba:signature localname="content" prefix="fn-zorba-fetch">
       <zorba:param>xs:string</zorba:param>
-      <zorba:param>xs:string</zorba:param>
+      <zorba:param>xs:string</zorba:param> <!-- entityKind -->
+      <zorba:param>xs:string</zorba:param> <!-- encoding -->
       <zorba:output>xs:string</zorba:output>
     </zorba:signature>
 
@@ -25,8 +26,33 @@
     </zorba:methods>
   </zorba:function>
 
-  <zorba:method static="true" name="destroyStream"
-    return="void">
+  <zorba:method static="true" name="destroyStream" return="void">
+    <zorba:param type="std::istream&amp;" name="aStream"/>
+  </zorba:method>
+
+</zorba:iterator>
+
+<!--
+/*********************************************************************
+*********************************************************************/
+-->
+<zorba:iterator name="FetchContentBinaryIterator">
+
+  <zorba:description author="Matthias Brantner"></zorba:description>
+
+  <zorba:function>
+    <zorba:signature localname="content-binary" prefix="fn-zorba-fetch">
+      <zorba:param>xs:string</zorba:param>
+      <zorba:param>xs:string</zorba:param>
+      <zorba:output>xs:base64Binary</zorba:output>
+    </zorba:signature>
+
+    <zorba:methods>
+      <zorba:accessesDynCtx returnValue="true"/>
+    </zorba:methods>
+  </zorba:function>
+
+  <zorba:method static="true" name="destroyStream" return="void">
     <zorba:param type="std::istream&amp;" name="aStream"/>
   </zorba:method>
 

=== modified file 'src/runtime/visitors/pregenerated/planiter_visitor.h'
--- src/runtime/visitors/pregenerated/planiter_visitor.h	2012-05-03 12:31:51 +0000
+++ src/runtime/visitors/pregenerated/planiter_visitor.h	2012-05-09 21:52:19 +0000
@@ -189,6 +189,8 @@
 
     class FetchContentIterator;
 
+    class FetchContentBinaryIterator;
+
     class FetchContentTypeIterator;
 
     class FnPutIterator;
@@ -897,6 +899,9 @@
     virtual void beginVisit ( const FetchContentIterator& ) = 0;
     virtual void endVisit   ( const FetchContentIterator& ) = 0;
 
+    virtual void beginVisit ( const FetchContentBinaryIterator& ) = 0;
+    virtual void endVisit   ( const FetchContentBinaryIterator& ) = 0;
+
     virtual void beginVisit ( const FetchContentTypeIterator& ) = 0;
     virtual void endVisit   ( const FetchContentTypeIterator& ) = 0;
 

=== modified file 'src/runtime/visitors/pregenerated/printer_visitor.cpp'
--- src/runtime/visitors/pregenerated/printer_visitor.cpp	2012-05-03 12:31:51 +0000
+++ src/runtime/visitors/pregenerated/printer_visitor.cpp	2012-05-09 21:52:19 +0000
@@ -1219,6 +1219,20 @@
 // </FetchContentIterator>
 
 
+// <FetchContentBinaryIterator>
+void PrinterVisitor::beginVisit ( const FetchContentBinaryIterator& a) {
+  thePrinter.startBeginVisit("FetchContentBinaryIterator", ++theId);
+  printCommons( &a, theId );
+  thePrinter.endBeginVisit( theId );
+}
+
+void PrinterVisitor::endVisit ( const FetchContentBinaryIterator& ) {
+  thePrinter.startEndVisit();
+  thePrinter.endEndVisit();
+}
+// </FetchContentBinaryIterator>
+
+
 // <FetchContentTypeIterator>
 void PrinterVisitor::beginVisit ( const FetchContentTypeIterator& a) {
   thePrinter.startBeginVisit("FetchContentTypeIterator", ++theId);

=== modified file 'src/runtime/visitors/pregenerated/printer_visitor.h'
--- src/runtime/visitors/pregenerated/printer_visitor.h	2012-05-03 12:31:51 +0000
+++ src/runtime/visitors/pregenerated/printer_visitor.h	2012-05-09 21:52:19 +0000
@@ -286,6 +286,9 @@
     void beginVisit( const FetchContentIterator& );
     void endVisit  ( const FetchContentIterator& );
 
+    void beginVisit( const FetchContentBinaryIterator& );
+    void endVisit  ( const FetchContentBinaryIterator& );
+
     void beginVisit( const FetchContentTypeIterator& );
     void endVisit  ( const FetchContentTypeIterator& );
 

=== added file 'test/rbkt/ExpQueryResults/zorba/fetch/fetch_seekable.xml.res'
--- test/rbkt/ExpQueryResults/zorba/fetch/fetch_seekable.xml.res	1970-01-01 00:00:00 +0000
+++ test/rbkt/ExpQueryResults/zorba/fetch/fetch_seekable.xml.res	2012-05-09 21:52:19 +0000
@@ -0,0 +1,1 @@
+true true true

=== added file 'test/rbkt/ExpQueryResults/zorba/fetch/fetch_seekable_binary.xml.res'
--- test/rbkt/ExpQueryResults/zorba/fetch/fetch_seekable_binary.xml.res	1970-01-01 00:00:00 +0000
+++ test/rbkt/ExpQueryResults/zorba/fetch/fetch_seekable_binary.xml.res	2012-05-09 21:52:19 +0000
@@ -0,0 +1,1 @@
+8 5Pb8Cg==

=== added file 'test/rbkt/ExpQueryResults/zorba/fetch/fetch_some_transcode.xml.res'
--- test/rbkt/ExpQueryResults/zorba/fetch/fetch_some_transcode.xml.res	1970-01-01 00:00:00 +0000
+++ test/rbkt/ExpQueryResults/zorba/fetch/fetch_some_transcode.xml.res	2012-05-09 21:52:19 +0000
@@ -0,0 +1,1 @@
+äöü

=== modified file 'test/rbkt/Queries/zorba/fetch/fetch_bogus1.xq'
--- test/rbkt/Queries/zorba/fetch/fetch_bogus1.xq	2012-05-03 12:31:51 +0000
+++ test/rbkt/Queries/zorba/fetch/fetch_bogus1.xq	2012-05-09 21:52:19 +0000
@@ -1,4 +1,4 @@
 (: Fetch with unknown entity kind :)
-import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch#2.0";
+import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch";
 
 fetch:content("http://www.zorba-xquery.com/modules/ext", "NOTHING")

=== modified file 'test/rbkt/Queries/zorba/fetch/fetch_bogus2.xq'
--- test/rbkt/Queries/zorba/fetch/fetch_bogus2.xq	2012-05-03 12:31:51 +0000
+++ test/rbkt/Queries/zorba/fetch/fetch_bogus2.xq	2012-05-09 21:52:19 +0000
@@ -1,4 +1,4 @@
 (: Fetch a valid module URI but as SOME_CONTENT :)
-import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch#2.0";
+import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch";
 
 fetch:content("http://www.flworfound.org/modules/ext", "SOME_CONTENT")

=== modified file 'test/rbkt/Queries/zorba/fetch/fetch_module1.xq'
--- test/rbkt/Queries/zorba/fetch/fetch_module1.xq	2012-05-03 12:31:51 +0000
+++ test/rbkt/Queries/zorba/fetch/fetch_module1.xq	2012-05-09 21:52:19 +0000
@@ -1,4 +1,4 @@
 (: Fetch a module's content :)
-import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch#2.0";
+import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch";
 
 fetch:content("http://www.zorba-xquery.com/modules/ext", "MODULE")

=== modified file 'test/rbkt/Queries/zorba/fetch/fetch_module2.xq'
--- test/rbkt/Queries/zorba/fetch/fetch_module2.xq	2012-05-03 12:31:51 +0000
+++ test/rbkt/Queries/zorba/fetch/fetch_module2.xq	2012-05-09 21:52:19 +0000
@@ -1,4 +1,4 @@
 (: Fetch a module's content with versioning :)
-import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch#2.0";
+import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch";
 
 fetch:content("http://www.zorba-xquery.com/modules/ext#1.0", "MODULE")

=== modified file 'test/rbkt/Queries/zorba/fetch/fetch_random_file.xq'
--- test/rbkt/Queries/zorba/fetch/fetch_random_file.xq	2012-02-01 17:13:01 +0000
+++ test/rbkt/Queries/zorba/fetch/fetch_random_file.xq	2012-05-09 21:52:19 +0000
@@ -1,4 +1,4 @@
 (: Fetch a random file from the filesystem :)
-import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch#2.0";
+import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch";
 
 fetch:content("http://www.zorba-xquery.com/random-file", "SOME_CONTENT")

=== modified file 'test/rbkt/Queries/zorba/fetch/fetch_schema1.xq'
--- test/rbkt/Queries/zorba/fetch/fetch_schema1.xq	2012-05-03 12:31:51 +0000
+++ test/rbkt/Queries/zorba/fetch/fetch_schema1.xq	2012-05-09 21:52:19 +0000
@@ -1,5 +1,5 @@
 (: Fetch a schemas's content :)
-import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch#2.0";
+import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch";
 
 fn:parse-xml(
   fetch:content("http://www.zorba-xquery.com/modules/theschema", "SCHEMA")

=== added file 'test/rbkt/Queries/zorba/fetch/fetch_seekable.xml.res'
--- test/rbkt/Queries/zorba/fetch/fetch_seekable.xml.res	1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/fetch/fetch_seekable.xml.res	2012-05-09 21:52:19 +0000
@@ -0,0 +1,1 @@
+true true true

=== added file 'test/rbkt/Queries/zorba/fetch/fetch_seekable.xq'
--- test/rbkt/Queries/zorba/fetch/fetch_seekable.xq	1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/fetch/fetch_seekable.xq	2012-05-09 21:52:19 +0000
@@ -0,0 +1,8 @@
+import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch";
+
+import module namespace s = "http://www.zorba-xquery.com/modules/string";
+
+(: make sure the returned string is streamable and can be consumed twice, i.e. is seekable :)
+let $x := fetch:content(fn:resolve-uri("iso-8859-1.txt"))
+return (s:is-streamable($x), string-length($x) gt 0, string-length($x) gt 0)
+

=== added file 'test/rbkt/Queries/zorba/fetch/fetch_seekable_binary.xq'
--- test/rbkt/Queries/zorba/fetch/fetch_seekable_binary.xq	1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/fetch/fetch_seekable_binary.xq	2012-05-09 21:52:19 +0000
@@ -0,0 +1,8 @@
+import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch";
+
+import module namespace b = "http://www.zorba-xquery.com/modules/converters/base64";
+
+(: make sure the returned string is streamable and can be consumed twice, i.e. is seekable :)
+let $x := fetch:content-binary(fn:resolve-uri("iso-8859-1.txt"))
+return (string-length(xs:string($x)), $x)
+

=== added file 'test/rbkt/Queries/zorba/fetch/fetch_some_transcode.xq'
--- test/rbkt/Queries/zorba/fetch/fetch_some_transcode.xq	1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/fetch/fetch_some_transcode.xq	2012-05-09 21:52:19 +0000
@@ -0,0 +1,3 @@
+import module namespace fetch = "http://www.zorba-xquery.com/modules/fetch";
+
+fetch:content(resolve-uri("iso-8859-1.txt"), "SOME_CONTENT", "ISO-8859-1")

=== added file 'test/rbkt/Queries/zorba/fetch/iso-8859-1.txt'
--- test/rbkt/Queries/zorba/fetch/iso-8859-1.txt	1970-01-01 00:00:00 +0000
+++ test/rbkt/Queries/zorba/fetch/iso-8859-1.txt	2012-05-09 21:52:19 +0000
@@ -0,0 +1,1 @@
+äöü