Merge lp:~zorba-coders/zorba/bug935639 into lp:zorba

[Rodolfo Ochoa]

This fixes the error:
n strings remain in the pool.
and change it to a more comprehensive error with reference to document that helps to prevent the error.

The problem can't be solved by technical means, the developer have to take care of resource usage, as in C++ the user must release all objects before shutdown the store, the same thing applies for XQJ.
So. the best workaround this problem is to show a more friendly message around the Strings Pool and create documentation that explain this issue. At the end, this issue affects every API usage from our clients.
Please check my changes and help me with English usage in the new document I created.

[Cezar Andrei (cezar-andrei)]

review: Approve

[Matthias Brantner (matthias-brantner)]

The Zorba XQuery Processor manage => The Zorba XQuery Processor manages

automaticall => automatically

review: Needs Fixing

[Markos Zaharioudakis (markos-za)]

review: Approve

[Zorba Build Bot (zorba-buildbot)]

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

[Matthias Brantner (matthias-brantner)]

review: Approve

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/bug935639-2012-03-07T22-33-06.388Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~zorba-coders/zorba/bug935639 into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job bug935639-2012-03-07T22-33-06.388Z is finished. The
  final status was:

  No tests were run - build or configure step must have failed.

  Not commiting changes.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Zorba Build Bot (zorba-buildbot)]

There are additional revisions which have not been approved in review. Please seek review and approval of these new revisions.

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/bug935639-2012-03-11T01-05-09.922Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~zorba-coders/zorba/bug935639 into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job bug935639-2012-03-11T01-05-09.922Z is finished. The
  final status was:

  3 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/bug935639-2012-03-12T01-12-03.818Z/log.html

[Zorba Build Bot (zorba-buildbot)]

The attempt to merge lp:~zorba-coders/zorba/bug935639 into lp:zorba failed. Below is the output from the failed tests.

CMake Error at /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake:274 (message):
  Validation queue job bug935639-2012-03-12T01-12-03.818Z is finished. The
  final status was:

  1 tests did not succeed - changes not commited.

Error in read script: /home/ceej/zo/testing/zorbatest/tester/TarmacLander.cmake

[lp:~zorba-coders/zorba/bug935639]

10691. By Rodolfo Ochoa on 2012-03-12

fix for errors.xq in CTESTs

10692. By Rodolfo Ochoa on 2012-03-12

bug from the editor

[Zorba Build Bot (zorba-buildbot)]

Validation queue starting for merge proposal.
Log at: http://zorbatest.lambda.nu:8080/remotequeue/bug935639-2012-03-12T23-14-08.66Z/log.html

[Zorba Build Bot (zorba-buildbot)]

Validation queue job bug935639-2012-03-12T23-14-08.66Z is finished. The final status was:

All tests succeeded!

=== modified file 'doc/zorba/indexpage.dox.in'
--- doc/zorba/indexpage.dox.in	2012-03-07 17:13:34 +0000
+++ doc/zorba/indexpage.dox.in	2012-03-12 23:11:19 +0000
@@ -28,6 +28,8 @@
   - \ref ft_intro
   - \ref errors_warnings
   - \ref impl_dep_features
+  - \ref memory_management
+  - \ref memory_leaks
 
 \section xqlib XQuery Modules
  - <a href="../xqdoc/xhtml/index.html">XQuery Module Library</a>

=== added file 'doc/zorba/memory_leaks.dox'
--- doc/zorba/memory_leaks.dox	1970-01-01 00:00:00 +0000
+++ doc/zorba/memory_leaks.dox	2012-03-12 23:11:19 +0000
@@ -0,0 +1,141 @@
+/** \page memory_leaks Memory Leaks
+
+\section memory_leaks_intro Introduction
+
+The Zorba XQuery Processor manages automatically its own memory, but still, there are cases where you may receive the following message:
+
+\code
+Zorba did not close properly, objects may still in memory while shutdown the store. 
+[n] referenced URIs remain in the string pool.
+For help avoiding this message please refer to http://www.zorba-xquery.com/html/documentation in section General Architecture -> Memory Leaks.
+\endcode
+
+This message means that not all the resources (e.g. XQuery objects) 
+were released before Zorba and its store were shutdown.
+While this might be a bug/memory leak in Zorba itself, it's also quite likely that the leak is in the application.
+For the application, the rule is that before shutting down the store 
+all resources must have been released.
+
+So, strictly speaking, your program is leaking resources.
+
+This problem might occur even though the application's code (C++ or other languages) looks good. Here is an example:
+
+\section memory_leaks_example_c Example in C++
+In the following example, the StaticContext and XQuery objects are not destroyed before Zorba and the store are shutdown.
+
+\code
+int main(int argc, char* argv[])
+{
+  void* lStore = zorba::StoreManager::getStore();
+  Zorba* lZorba = Zorba::getInstance(lStore);
+  
+  StaticContext_t lContext = lZorba->createStaticContext();
+  lContext->addNamespace("foo", "http://www.foo.com");
+  XQuery_t lQuery = lZorba->compileQuery("<foo:e/>", lContext);
+  lQuery->execute();
+  
+  lZorba->shutdown();
+  zorba::StoreManager::shutdownStore(lStore);
+  return 0;
+}
+\endcode
+
+So, for this scenario, the ideal way to solve the problem is through scopes, 
+where you implement certain part of code in methods and release them when the objects loose their reference. 
+Alternatively, you can explicitly set the variable to null, in which case the object will automatically be released.
+
+Solution example:
+\code
+int main(int argc, char* argv[])
+{
+  void* lStore = zorba::StoreManager::getStore();
+  Zorba* lZorba = Zorba::getInstance(lStore);
+  
+  {  // Use a scope
+
+  StaticContext_t lContext = lZorba->createStaticContext();
+    lContext->addNamespace("foo", "http://www.foo.com");
+    XQuery_t lQuery = lZorba->compileQuery("<foo:e/>", lContext);
+    lQuery->execute();
+    
+    lContext = NULL;  // or explicitely set free the resources
+    lQuery = NULL;
+  }
+  lZorba->shutdown();
+  zorba::StoreManager::shutdownStore(lStore);
+  return 0;
+}
+\endcode
+
+
+\section memory_leaks_example_java Example in Java
+This particular error is specially notorious when a memory managed language 
+shows the error because you expect the language frees all memory, here is an example:
+
+\code
+public static void main ( String argv[] )
+{
+  InMemoryStore store = InMemoryStore.getInstance();
+  Zorba zorba = Zorba.getInstance ( store );
+
+  StaticContext context = zorba.createStaticContext();
+  context.addNamespace("foo", "http://www.foo.com");
+  XQuery query = zorba.compileQuery("<foo:e/>", context);
+  String result = query.execute();
+  
+  zorba.shutdown();
+  InMemoryStore.shutdown ( store );
+  
+}    
+\endcode
+
+In this example, and for the rest of the languages because Zorba is created in C++, 
+we have created in the Zorba API the method \c destroy() \e that will set free the object
+that could be pointing to any resource from the store, this method is in every object that need to be released.
+
+*Java Note: Because Java is a garbage collected language you cannot predict when (or even if) an object will be destroyed. 
+Hence there is no direct equivalent of a destructor.
+There is an inherited method called finalize, but this is called entirely at the discretion of the garbage collector.
+So, destroy() is the best practice for any language including Java.
+
+Solution example:
+\code
+public static void main ( String argv[] )
+{
+  InMemoryStore store = InMemoryStore.getInstance();
+  Zorba zorba = Zorba.getInstance ( store );
+
+  StaticContext context = zorba.createStaticContext();
+  context.addNamespace("foo", "http://www.foo.com");
+  XQuery query = zorba.compileQuery("<foo:e/>", context);
+  String result = query.execute();
+
+  query.destroy();   // Release memory for this XQuery
+  context.destroy(); // Release memory for this StaticContext
+
+  zorba.shutdown();
+  InMemoryStore.shutdown ( store );
+  
+}
+\endcode
+
+
+\section memory_leaks_example_xqj Example in XQJ
+
+XQJ standard provides specific \c close() \e methods for this specific purpose:
+\code
+  XQDataSource xqdatasource = new XQDataSource();
+  XQConnection xqconnection = xqdatasource.getConnection();
+  XQStaticContext staticContext = xqconnection.getStaticContext();
+  staticContext.declareNamespace("foo", "http://www.foo.com");
+  xqconnection.setStaticContext(staticContext);
+  XQExpression xqexpression = xqconnection.createExpression();
+  XQSequence xqsequence = xqexpression.executeQuery("<foo:e/>");
+  // code to show the output
+
+  xqconnection.close();  // Closing connection frees all related resources
+\endcode
+
+
+*/
+/* vim:set et sw=2 ts=2: */

=== modified file 'include/zorba/pregenerated/diagnostic_list.h'
--- include/zorba/pregenerated/diagnostic_list.h	2012-03-08 18:51:25 +0000
+++ include/zorba/pregenerated/diagnostic_list.h	2012-03-12 23:11:19 +0000
@@ -734,6 +734,8 @@
 
 extern ZORBA_DLL_PUBLIC ZorbaErrorCode ZSTR0060_RANGE_EXCEPTION;
 
+extern ZORBA_DLL_PUBLIC ZorbaErrorCode ZSTR0065_STRINGS_IN_POOL;
+
 extern ZORBA_DLL_PUBLIC ZorbaErrorCode XSST0001;
 
 extern ZORBA_DLL_PUBLIC ZorbaErrorCode XSST0002;

=== modified file 'modules/com/zorba-xquery/www/modules/pregenerated/errors.xq'
--- modules/com/zorba-xquery/www/modules/pregenerated/errors.xq	2012-03-08 18:51:25 +0000
+++ modules/com/zorba-xquery/www/modules/pregenerated/errors.xq	2012-03-12 23:11:19 +0000
@@ -764,6 +764,10 @@
 
 (:~
 :)
+declare variable $zerr:ZSTR0065 as xs:QName := fn:QName($zerr:NS, "zerr:ZSTR0065");
+
+(:~
+:)
 declare variable $zerr:XSST0001 as xs:QName := fn:QName($zerr:NS, "zerr:XSST0001");
 
 (:~

=== modified file 'src/diagnostics/diagnostic_en.xml'
--- src/diagnostics/diagnostic_en.xml	2012-03-08 18:51:25 +0000
+++ src/diagnostics/diagnostic_en.xml	2012-03-12 23:11:19 +0000
@@ -2334,6 +2334,10 @@
       <value>out of range: $1</value>
     </diagnostic>
 
+    <diagnostic code="ZSTR0065" name="STRINGS_IN_POOL">
+      <value>Zorba did not close properly, objects may still in memory.\n$1 referenced URI(s) remain in the string pool.\nFor help avoiding this message please refer to http://www.zorba-xquery.com/html/documentation in section General Architecture -> Memory Leaks.</value>
+    </diagnostic>
+
     <!--////////// XQuery Scripting ////////////////////////////////////////-->
 
     <diagnostic code="XSST0001">

=== modified file 'src/diagnostics/pregenerated/diagnostic_list.cpp'
--- src/diagnostics/pregenerated/diagnostic_list.cpp	2012-03-08 18:51:25 +0000
+++ src/diagnostics/pregenerated/diagnostic_list.cpp	2012-03-12 23:11:19 +0000
@@ -1079,6 +1079,9 @@
 ZorbaErrorCode ZSTR0060_RANGE_EXCEPTION( "ZSTR0060" );
 
 
+ZorbaErrorCode ZSTR0065_STRINGS_IN_POOL( "ZSTR0065" );
+
+
 ZorbaErrorCode XSST0001( "XSST0001" );
 
 

=== modified file 'src/diagnostics/pregenerated/dict_en.cpp'
--- src/diagnostics/pregenerated/dict_en.cpp	2012-03-08 18:51:25 +0000
+++ src/diagnostics/pregenerated/dict_en.cpp	2012-03-12 23:11:19 +0000
@@ -378,6 +378,7 @@
   { "ZSTR0050", "\"$1\" not implemented for item type \"$2\"" },
   { "ZSTR0055", "streamable string has already been consumed" },
   { "ZSTR0060", "out of range: $1" },
+  { "ZSTR0065", "Zorba did not close properly, objects may still in memory.\n$1 referenced URI(s) remain in the string pool.\nFor help avoiding this message please refer to http://www.zorba-xquery.com/html/documentation in section General Architecture -> Memory Leaks." },
   { "ZWST0002", "\"$1\": unknown or unsupported annotation" },
   { "ZWST0003", "\"$1\": function declared sequential, but has non-sequential body" },
   { "ZWST0004", "Sequential FLWOR expr may not have the semantics you expect" },

=== modified file 'src/store/naive/node_items.cpp'
--- src/store/naive/node_items.cpp	2012-03-07 14:22:29 +0000
+++ src/store/naive/node_items.cpp	2012-03-12 23:11:19 +0000
@@ -722,7 +722,7 @@
   }
   catch (...)
   {
-    ZORBA_FATAL(false, "Unexpectd exception");
+    ZORBA_FATAL(false, "Unexpected exception");
   }
 }
 

=== modified file 'src/store/naive/string_pool.cpp'
--- src/store/naive/string_pool.cpp	2012-02-15 10:25:02 +0000
+++ src/store/naive/string_pool.cpp	2012-03-12 23:11:19 +0000
@@ -15,9 +15,10 @@
  */
 #include "stdafx.h"
 
-#include "zorbautils/fatal.h"
 #include "util/string_util.h"
 #include "string_pool.h"
+#include "zorba/zorba_exception.h"
+#include "diagnostics/xquery_diagnostics.h"
 
 namespace zorba { namespace simplestore {
 
@@ -32,14 +33,14 @@
   {
     if (theHashTab[i].theItem.is_shared())
     {
-      std::cerr << "i = " << i << " String " << theHashTab[i].theItem
-                << " is still in the pool" << std::endl;
+      //std::cerr << "ID: " << i << "   Referenced URI: " << theHashTab[i].theItem << std::endl;
       //delete theHashTab[i].theString.getp();
-      count++;
+      count ++;
     }
   }
-
-  ZORBA_FATAL(count == 0, count << " strings remain in the string pool");
+  if (count>0) {
+      throw ZORBA_EXCEPTION(zerr::ZSTR0065_STRINGS_IN_POOL, ERROR_PARAMS(count));
+  }
 }