Merge lp:~pbeaman/akiban-persistit/sphinxdoc-release-notes into lp:akiban-persistit

[Peter Beaman]

This branch includes some work previously done but never proposed in lp:~pbeaman/akiban-persistit/sphinxdoc-1, and new work on the release notes.

This branch only changes the doc directory.

Changes:

Convert all the asciidoc source files to sphinx. Files xxx.txt are removed and different files xxx.rst are added with formatting encoded for sphinx.

Content edits in some files resulting from the change.

New doc/build/build.sh script which now places all output in the target directory.

With these changes you must have sphinx (v 1.1.3) to run the build process. A new Java program, SphinxDocPrep does the appropriate text replacements in the sphinx-formatted files. The result of running build.sh is:

    target/sphinx/source - .rst source files with links to the apidocs

    target/sphinx/html - html files resulting from running sphinx-build. Included: ReleaseNotes.html in HTML format.

    target/sphinx/text/ReleaseNotes - releases notes in flat text, suitable for pasting into Launchpad.

[Nathan Williams (nwilliams)]

Looks plausible.

review: Approve

=== added file 'doc/BasicAPI.rst'
--- doc/BasicAPI.rst	1970-01-01 00:00:00 +0000
+++ doc/BasicAPI.rst	2012-05-30 18:23:19 +0000
@@ -0,0 +1,332 @@
+.. _Basic-API:
+
+Basic API
+=========
+
+Akiban Persistit stores data as key-value pairs in highly optimized B-Tree. (Actually, implements `B-Link Tree <http://www.cs.cornell.edu/courses/cs4411/2009sp/blink.pdf>`_ trees for greater concurrency). Like a Java Map implementation, Persistit associates at most one value with each unique instance of a Key value.
+
+Persistit provides classes and methods to access and modify keys and their associated values. Application code calls Persistit API methods to store, fetch, traverse and remove keys and records to and from the database.
+
+Persistit permits efficient multi-threaded concurrent access to database volumes. It is designed to minimize contention for critical resources and to maximize throughput on multi-processor machines. Concurrent ACID transactions are supported with multi-value concurrency control (MVCC).
+
+The Persistit Instance
+----------------------
+
+To access Persistit, the application first constructs an instance of the ``com.persistit.Persistit`` class and initializes it. This instance is the keystone of all subsequent operations.  It holds references to the buffers, maps, transaction contexts and other structures needed to access B-trees. The life cycle of a Persistit instance should be managed as follows:
+
+.. code-block:: java
+
+  final Persistit db = new Persistit();
+  //
+  // register any Coder and Renderer instances before initialization
+  //
+  db.initialize(configProperties);
+  try {
+      // do application work
+  } finally {
+      db.close();
+  }
+
+The ``configProperties`` describe the memory allocation, initial set of volumes, the journal, and other settings needed to get Persistit started. See :ref:`Configuration` for details.
+
+The ``com.persistit.Persistit#close`` method gracefully flushes all modified data to disk, stops background threads and unregisters JMX MBeans. 
+
+.. note:: 
+
+  The Persistit background threads are not daemon threads, so if your application returns 
+  from its static main method without calling ``close``, the JVM will not automatically exit.
+
+Although normal shutdown should always invoke ``close``, Persistit is designed to recover a consistent database state in the event of an abrupt shutdown or crash. See :ref:`Recovery`.
+
+.. _Key:
+
+Key
+---
+
+The content of a ``com.persistit.Key`` is the unique identifier for a key/value pair within a tree. Internally a ``Key`` contains an array of bytes that constitute the physical identity of the key/value pair within a tree. Logically, a key consists of a sequence of zero or more Java values, each of which is called a *key segment*. 
+
+The following value types are implicitly supported in keys::
+
+  null
+  boolean (and Boolean)
+  byte (and Byte)
+  short (and Short)
+  char (and Character)
+  int (and Integer)
+  long (and Long)
+  float (and Float)
+  double (and Double)
+  java.lang.String
+  java.math.BigInteger
+  java.math.BigDecimal
+  java.util.Date
+  byte[]
+
+In addition, you may register custom implementations of the ``com.persistit.encoding.KeyCoder`` interface to support encoding of other object classes. By default, String values are encoded in UTF-8 format.
+
+Appending and Decoding Key Segments
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``Key`` class provides methods to encode and decode each of these types to and from a key segment. For each type listed above, there is an ``append`` method, a ``to`` method and a ``decode`` method. For example, for the long type, there are methods
+
+.. code-block:: java
+
+  public void append(long v)
+  public void to(long v)
+  public long decodeLong()
+
+The ``to`` methods replaces the final key segment with a different value (unless the key is empty, in which case it works the same as ``append``).
+
+For example:
+
+.. code-block:: java
+
+  key.clear();         	// clear any previous key segments
+  key.append("Atlantic");  // append segment "Atlantic"
+  key.to("Pacific");   	// replace "Atlantic" with "Pacific"
+  key.reset();         	// reset index to beginning
+  String s = key.decode(); // s contains "Pacific"
+
+The Key class also provides methods to encode and decode Object values to and from a key. Strings, Dates, objects of the corresponding wrapper classes for the primitive types listed above, and objects supported by registered instances of ``com.persistit.encoding.KeyCoder`` are permitted. Primitive values are automatically boxed and unboxed as needed. The following code fragment demonstrates key manipulation with automatic conversion of primitive types and their wrappers.
+
+.. code-block:: java
+
+  key.clear();              	// clear any previous key segments
+  key.append(new Integer(1234));
+  key.append("Atlantic");
+  key.append(1.23d);
+  key.reset();              	// reset index to beginning for decoding
+  int v = key.decodeInt();  	// v will be 1234
+  String s = (String)key.decode(); // s will be "Atlantic"
+  Double d = (Double)decode();    // d will be 1.23d as a Double
+
+In this code segment, an object of type Integer is appended to the key’s value sequence, and then the same value is later decoded as a primitive int value. A String is appended and then decoded into a String. Finally, a primitive double value is appended and then decoded as an object of class Double.
+
+The maximum size of a serialized ``Key`` is 2,047 bytes.
+
+For further information, see ``com.persistit.Key``.
+
+
+.. _Value:
+
+Value
+-----
+
+A ``com.persistit.Value`` object holds a value. Unlike keys, Value objects have no restriction on the types of data they can represent, and they can hold much larger objects. In particular, a Value may contain null, any of the primitive types, or an object of any class.
+
+The backing store of a ``Value`` is a byte array that is written to a B-Tree data page, or in the case of a long record, multiple pages. The ``com.persistit.Value#put`` method variants encode (serialize) a Java primitive or Object value into the backing store, and the ``com.persistit.Value#get`` method variants decode (deserialize) the value.
+
+For example, in ``HelloWorld.java``, the line
+
+.. code-block:: java
+
+  dbex.getValue().put("World");
+
+serializes the String “World”, and the expression
+
+.. code-block:: java
+
+  dbex.getValue().get()
+
+decodes it. Persistit does not intrinsically cache decoded object values, nor does it track an object's state changes.  Each call to the ``get()`` method returns a new instance of the object. However, you can use a ``com.persistit.encoding.ObjectCache`` to cache object values. ``ObjectCache`` is designed specifically to cache objects fetched from Persistit.
+
+Value Types
+^^^^^^^^^^^
+
+``Value`` provides optimized predefined representations for the following types::
+
+  null
+  all primitive types
+  all arrays
+  java.math.BigInteger
+  java.math.BigDecimal
+  java.lang.String
+  java.util.Date
+
+In general, Persistit uses one of four mechanisms to encode a Java value into a Value object:
+
+- If the value is one of the predefined types listed above, Persistit uses its own internal serialization logic.
+- If there is a registered ``com.persistit.encoding.ValueCoder`` for the object's class, Persistit delegates to it.
+- If enabled, Persistit uses an accelerated serialization/deserialization mechanism to encode and decode objects.
+- Otherwise, for classes that implement java.io.Serializable, Persistit attempts to perform default Java serialization and deserialization.
+
+A Value may also be in the undefined state, which results from performing a fetch operation on a key for which no value is present in the database. The undefined state is distinct from the value ``null`` and can be tested with the ``isDefined()`` method.
+
+See :ref:`Serialization` for additional information.
+
+Large Values
+^^^^^^^^^^^^
+
+Persistit stores large values, in the current version up to 64MB in size. For example, it is possible to store an image’s backing bytes as a single value in the database. The size of the value to be stored is constrained by available heap memory; the entire value must be able to be serialized into an in-memory byte array in order for Persistit to store or retrieve it. Use ``com.persistit.Value#setMaximumSize`` to specify a the size constraint. Large values are broken up across multiple data pages and are not necessarily stored in contiguous file areas.
+
+The definition of “large” depends on the configuration properties. for example, for a volume with a page size of 16K bytes the threshold occurs at 6,108 bytes. A value having a serialized size smaller than this is stored in a single data page while a larger value is broken up and stored in multiple pages. For a smaller pages size the threshold is lower.
+
+On occasion it may be desirable to fetch only part of a large value. For example, it may be useful to extract summary information from the beginning of a the backing byte array for an Image. Variants versions of the ``fetch`` and ``traverse`` accept a minimum byte count parameter. When these methods are used only the specified minimum number bytes of the backing store are retrieved from the database. This technique can prevent Persistit from reading large numbers of pages from the disk in order to examine only a small portion of the record.
+
+.. _Exchange:
+
+Exchange
+--------
+
+The primary low-level interface for interacting with Persistit is ``com.persistit.Exchange``. The Exchange class provides all methods for storing, deleting, fetching and traversing key/value pairs. These methods are summarized here and described in detail in the Javadoc API documentation.
+
+An Exchange instance contains references to a ``Key`` and a ``Value``. The methods ``com.persistit.Exchange.getKey()`` and ``com.persistit.Exchange.getValue()`` access these instances.
+
+To construct an Exchange you specify a Volume (or alias) and a tree name in its constructor. The constructor will optionally create a new tree in that Volume if a tree having the specified name has not already been created. An application may construct an arbitrary number of Exchange objects. Creating a new Exchange has no effect on the database if the specified tree already exists. Tree creation is thread-safe: multiple threads concurrently constructing Exchanges using the same Tree name will safely result in the creation of only one new tree.
+
+An Exchange is a moderately complex object that can consume tens of kilobytes to megabytes (depending on the sizes of the Key and Value) of heap space. Memory-constrained applications should construct Exchanges in moderatation.
+
+Persistit offers Exchange pooling to avoid rapidly creating and destroying Exchange objects in multi-threaded applications.  An application may use the ``com.persistit.Persistit#getExchange`` and ``com.persistit.Persistit#releaseExchange`` methods to take and return an Exchange from and to a thread-local pool.
+
+An Exchange internally maintains some optimization information such that references to nearby Keys within a tree are accelerated. Performance may benefit from using a different Exchange for each area of the Tree being accessed.
+
+Concurrent Operations on Exchanges
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Although the underlying Persistit database is designed for highly concurrent multi-threaded operation, the ``Exchange`` class and its associated ``Key`` and ``Value`` instances are *not* thread-safe. Each thread should acquire and use its own Exchange objects when accessing the database. Nonetheless, multiple threads can execute database operations on overlapping data concurrently using their thread-private ``Exchange`` instances.
+
+Because Persistit permits concurrent operations by multiple threads, there is no guarantee that the underlying database will remain unchanged after an Exchange fetches or modifies its data. However, each operation on an Exchange is atomic, meaning that the inputs and outputs of each method are consistent with some valid state of the underlying Persistit backing store at some instant in time. The Exchange’s Value and Key objects represent that consistent state even if another thread subsequently modifies the database. Transactions, described below, allow multiple database operations to be performed atomically and consistently.
+
+Exchange API
+^^^^^^^^^^^^
+
+An Exchange has permanent references to a ``com.persistit.Key`` and a ``com.persistit.Value``. Typically you work with an Exchange in one of the following patterns:
+
+- Modify the Key, perform a ``fetch`` operation, and extract the Value.
+- Modify the Key, modify the Value, and then perform a ``store`` operation.
+- Modify the Key, and then perform a ``remove`` operation.
+- Optionally modify the Key, perform a ``traverse`` operation, then read the resulting Key and/or Value.
+
+These four methods, plus a few other methods listed here, are the primary low-level interface to the database. Semantics are as follows:
+
+``fetch``
+    Reads the stored value associated with this Exchange's Key and modifies the Exchange’s Value to reflect that value.
+``store``
+    Inserts or replaces the key/value pair for the specified key in the Tree either by replacing the former value, if there was one, or inserting a new value.
+``fetchAndStore``
+    Reads and then replaces the stored value. Upon completion, Value reflects the formerly stored value for the current Key. This operation is atomic.
+``remove``, ``removeAll``, ``removeKeyRange``
+    Removes key/value pairs from the Tree. Versions of this method specify either a single key or a range of keys to be removed.
+``fetchAndRemove``
+    Fetches and then removes the stored value. Upon completion, Value reflects the formerly stored value for the current Key. This operation is atomic.
+``traverse``, ``next``, ``previous``
+    Modifies the Exchange’s Key and Value to reflect a successor or predecessor key within the tree. See ``com.persistit.Key`` for detailed information on the order of traversal.
+``hasNext``, ``hasPrevious``
+    Indicates, without modifying the Exchange’s Value or Key objects, whether there is a successor or predecessor key in the Tree.
+``hasChildren``
+    Indicates whether there are records having keys that are logical children. A *logical child* of some key *P* is any key that can be constructed by appending one or more key segments to *P*.
+
+For convenience, Exchange delegates ``append`` and ``to`` methods to ``com.persistit.Key``. For example, Exchange provides the following methods that delegate to the identically named methods of Key :
+
+.. code-block:: java
+
+  public Exchange append(long v)
+  public Exchange append(String v)
+  ...
+
+To allow code call-chaining these methods of Exchange return the same Exchange. For example, it is valid to write code such as
+
+.. code-block:: java
+
+  exchange.clear().append(" Pacific").append("Ocean").append(123).fetch();
+
+This example fetches the value associated with the concatenated key
+``{“Pacific”, ”Ocean”, 123}``.
+
+Exchange also delegates other key manipulation methods. (See ``com.persistit.Exchange`` for detailed API documentation.)
+
+Traversing and Querying Collections of Data
+-------------------------------------------
+
+An Exchange provides a number of methods for traversing a collection of records in the Persistit database. These include variations of the ``com.persistit.Exchange#traverse``, ``com.persistit.Exchange#next`` and ``com.persistit.Exchange#previous``. For all of these methods, Persistit does two things: it modifies the Exchange's ``Key`` to reflect a new key that is before or after the current key, and it modifies the ``Value`` associated with the Exchange to reflect the database value associated with that key.
+
+For example, this code from ``HelloWorld.java`` prints out the key and value of each record in a tree:
+
+.. code-block:: java
+
+       	dbex.getKey().to(Key.BEFORE);
+       	while (dbex.next())
+       	{
+           	System.out.println(
+               	dbex.getKey().indexTo(0).decode() + " " +
+               	dbex.getValue().get());
+       	}
+
+In general, the traversal methods let you find a key in a tree related to the key you supply. In Persistit programs you frequently prime a key value by appending either ``com.persistit.Key#BEFORE`` or ``com.persistit.Key#AFTER``. A key containing either of these special values can never be stored in a tree; these are reserved to represent positions in key traversal order before the first valid key and after the last valid key, respectively. You then invoke next or previous, or any of the other traverse family variants, to enumerate keys within the tree.
+
+You can specify whether traversal is *deep* or *shallow*.  Deep traversal traverses the logical children (see com.persistit.Key) of a key. Shallow traversal traverses only the logical siblings.
+
+.. _KeyFilter:
+
+Selecting key values with a KeyFilter
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A ``com.persistit.KeyFilter`` defines a subset of all possible key values. For example, a KeyFilter can select keys with certain fixed segment values, sets of values or ranges of values.  Calling ``traverse``, ``next`` or ``previous`` with a KeyFilter efficiently traverses the subset of all keys in a Tree that match the filter.
+
+You construct a KeyFilter either by adding selection terms to it, or by calling the ``com.persistit.KeyParser#parseKeyFilter`` method of the ``com.persistit.KeyParser`` class to construct one from a string representation.
+
+Use of a KeyFilter is illustrated by the following code fragment:
+
+.. code-block:: java
+
+  Exchange ex = new Exchange("myVolume", "myTree", true);
+  KeyFilter kf = new KeyFilter("{\"Bellini\":\"Britten\"}");
+  ex.append(Key.BEFORE);
+  while (ex.next(kf)){
+      System.out.println(ex.getKey().reset().decodeString());
+  }
+
+This simple example emits the string-valued keys within Tree “myTree” whose values fall alphabetically between “Bellini” and “Britten”, inclusive.
+
+
+You will find an example with a KeyFilter in the examples/FindFileDemo directory.
+
+.. _PersistitMap:
+
+PersistitMap
+------------
+
+In addition to low-level access methods on keys and values, Persistit provides ``com.persistit.PersistitMap``, which implements the ``java.util.SortedMap`` interface. PersistitMap uses the Persistit database as a backing store so that key/value pairs are persistent, potentially shared with all threads, and limited in number only by disk storage.
+
+Keys and Values for PersistitMap must conform to the constraints described above under :ref:`Key` and :ref:`Value`.
+
+The constructor for PersistitMap takes an Exchange as its sole parameter. All key/value pairs of the Map are stored within the tree identified by this Exchange. The Key supplied by the Exchange becomes the root of a logical tree. For example:
+
+.. code-block:: java
+
+  Exchange ex = new Exchange("myVolume", "myTree", true);
+  ex.append("USA").append("MA");
+  PersistitMap<String, String> map = new PersistitMap<String, String>(ex);
+  map.put("Boston", "Hub");
+
+places a key/value pair into Tree “myTree” with the concatenated key ``{"USA ","MA","Boston"}`` and a value ``"Hub"``.
+
+Generally the expected behavior for an Iterator on a Map collection view is to throw a ``ConcurrentModificationException`` if the underlying collection changes. This is known as “fail-fast” behavior. PersistitMap implements this behavior by throwing a ``ConcurrentModificationException`` in the event the Tree containing the map changes after the Iterator is constructed.
+
+However, sometimes it may be desirable to use PersistitMap and its collections view interfaces to iterate across changing data, especially for large databases. PersistitMap provides the method ``com.persistit.PersistitMap#setAllowConcurrentModification`` to control whether changes made by other threads are permitted. By default, concurrent modifications are not allowed.
+
+.. note:: When ``PersistitMap`` is used within a transaction updates generated by other concurrent transactions are not visible and   
+   therefore cannot cause a ConcurrentModificationException.  However, to avoid unpredictable results an Iterator created within the scope 
+   of a transaction must be used only within that transaction.
+
+
+Exceptions in PersistitMap
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Persistit operations throw a variety of exceptions that are subclasses of ``com.persistit.exception.PersistitException``. However, the methods of the SortedMap interface do not permit arbitrary checked exceptions to be thrown. Therefore, PersistitMap wraps any PersistitException generated by the underlying database methods within a ``com.persistit.PersistitMap.PersistitMapException``. This exception is unchecked and can therefore be thrown by methods of the Map interface. Applications using PersistitMap should catch and handle PersistitMap.PersistitMapException.
+
+Applying a KeyFilter to a PersistitMap Iterator
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can specify a ``com.persistit.KeyFilter`` for the Iterator returned by the ``keySet()``, ``entrySet()`` and ``values()`` methods of ``com.persistit.PersistitMap``.  The KeyFilter restricts the range of keys traversed by the Iterator. To set the KeyFilter, you must cast the Iterator to the inner class PersistitMap.ExchangeIterator, as shown here:
+
+.. code-block:: java
+
+	PersistitMap map = new PersistitMap(exchange);
+	PersistitMap.ExchangeIterator iterator =
+   	(PersistitMap.ExchangeIterator)map.entrySet().iterator();
+	iterator.setFilterTerm(KeyFilter.rangeTerm("A", "M"));
+
+In this example, the iterator will only access String-valued keys between “A” and “M”.
+
+

=== removed file 'doc/BasicAPI.txt'
--- doc/BasicAPI.txt	2012-04-30 22:09:31 +0000
+++ doc/BasicAPI.txt	1970-01-01 00:00:00 +0000
@@ -1,320 +0,0 @@
-[[BasicAPI]]
-= Basic API
-
-Akiban Persistit stores data as key-value pairs in highly optimized B-Tree footnote:[Technically, Persistit implements http://www.cs.cornell.edu/courses/cs4411/2009sp/blink.pdf[B-Link] trees for greater concurrency]. Like a Java Map implementation, Persistit associates at most one value with each unique instance of a Key value.
-
-Persistit provides classes and methods to access and modify keys and their associated values. Application code calls Persistit API methods to store, fetch, traverse and remove keys and records to and from the database.
-
-Persistit permits efficient multi-threaded concurrent access to database volumes. It is designed to minimize contention for critical resources and to maximize throughput on multi-processor machines. Concurrent ACID transactions are supported with multi-value concurrency control (MVCC).
-
-== The Persistit Instance
-
-To access Persistit, the application first constructs an instance of the +com.persistit.Persistit+ class and initializes it. This instance is the keystone of all subsequent operations.  It holds references to the buffers, maps, transaction contexts and other structures needed to access B-trees. The life cycle of a Persistit instance should be managed as follows:
-
-[source,java]
-----
-final Persistit db = new Persistit();
-//
-// register any Coder and Renderer instances before initialization
-//
-db.initialize(configProperties);
-try {
-    // do application work
-} finally {
-    db.close();
-}
-----
-
-The +configProperties+ describe the memory allocation, initial set of volumes, the journal, and other settings needed to get Persistit started. See <<Configuration>> for details.
-
-The +com.persistit.Persistit#close+ method gracefully flushes all modified data to disk, stops background threads and unregisters JMX MBeans. 
-****
-The Persistit background threads are not daemon threads, so if your application returns from its static main method without calling +close+, the JVM will not automically exit.
-****
-
-Although normal shutdown should always invoke +close+, Persistit is designed to recover a consistent database state in the event of an abrupt shutdown or crash. See <<Recovery>>.
-
-[[Key]]
-== Key
-
-The content of a +com.persistit.Key+ is the unique identifier for a key/value pair within a tree. Internally a +Key+ contains an array of bytes that constitute the physical identity of the key/value pair within a tree. Logically, a key consists of a sequence of zero or more Java values, each of which is called a _key segment_. 
-
-The following value types are implicitly supported in keys:
-
-.Types Supported in +com.persistit.Key+
-----
-null
-boolean (and Boolean)
-byte (and Byte)
-short (and Short)
-char (and Character)
-int (and Integer)
-long (and Long)
-float (and Float)
-double (and Double)
-java.lang.String
-java.math.BigInteger
-java.math.BigDecimal
-java.util.Date
-byte[]
-----
-
-In addition, you may register custom implementations of the +com.persistit.encoding.KeyCoder+ interface to support encoding of other object classes. By default, String values are encoded in UTF-8 format.
-****
-TODO: Collation
-****
-
-=== Appending and Decoding Key Segments
-
-The +Key+ class provides methods to encode and decode each of these types to and from a key segment. For each type listed above, there is an +append+ method, a +to+ method and a +decode+ method. For example, for the long type, there are methods
-
-[source,java]
-----
-public void append(long v)
-public void to(long v)
-public long decodeLong()
-----
-
-The +to+ methods replaces the final key segment with a different value (unless the key is empty, in which case it works the same as +append+).
-
-For example:
-
-[source,java]
-----
-key.clear();         	// clear any previous key segments
-key.append("Atlantic");  // append segment "Atlantic"
-key.to("Pacific");   	// replace "Atlantic" with "Pacific"
-key.reset();         	// reset index to beginning
-String s = key.decode(); // s contains "Pacific"
-----
-
-The Key class also provides methods to encode and decode Object values to and from a key. Strings, Dates, objects of the corresponding wrapper classes for the primitive types listed above, and objects supported by registered instances of +com.persistit.encoding.KeyCoder+ are permitted. Primitive values are automatically boxed and unboxed as needed. The following code fragment demonstrates key manipulation with automatic conversion of primitive types and their wrappers.
-
-[source,java]
-----
-key.clear();              	// clear any previous key segments
-key.append(new Integer(1234));
-key.append("Atlantic");
-key.append(1.23d);
-key.reset();              	// reset index to beginning for decoding
-int v = key.decodeInt();  	// v will be 1234
-String s = (String)key.decode(); // s will be "Atlantic"
-Double d = (Double)decode();    // d will be 1.23d as a Double
-----
-
-In this code segment, an object of type Integer is appended to the key’s value sequence, and then the same value is later decoded as a primitive int value. A String is appended and then decoded into a String. Finally, a primitive double value is appended and then decoded as an object of class Double.
-
-The maximum size of a serialized +Key+ is 2,047 bytes.
-
-For further information, see +com.persistit.Key+.
-
-
-[[Value]]
-== Value
-
-A +com.persistit.Value+ object holds a value. Unlike keys, Value objects have no restriction on the types of data they can represent, and they can hold much larger objects. In particular, a Value may contain null, any of the primitive types, or an object of any class.
-
-The backing store of a +Value+ is a byte array that is written to a B-Tree data page, or in the case of a long record, multiple pages. The +com.persistit.Value#put+ method variants encode (serialize) a Java primitive or Object value into the backing store, and the +com.persistit.Value#get+ method variants decode (deserialize) the value.
-
-For example, in +HelloWorld.java+, the line
-
-[source,java]
-----
-dbex.getValue().put("World");
-----
-
-serializes the String “World”, and the expression
-
-[source,java]
-----
-dbex.getValue().get()
-----
-
-decodes it. Persistit does not intrinsically cache decoded object values, nor does it track an object's state changes.  Each call to the +get()+ method returns a new instance of the object. However, you can use a +com.persistit.encoding.ObjectCache+ to cache object values. +ObjectCache+ is designed specifically to cache objects fetched from Persistit.
-
-=== Value Types
-
-+Value+ provides optimized predefined representations for the following types:
-
-.Types Implicitly Supported by +com.persistit.Value+
-----
-null
-all primitive types
-all arrays
-java.math.BigInteger
-java.math.BigDecimal
-java.lang.String
-java.util.Date
-----
-In general, Persistit uses one of four mechanisms to encode a Java value into a Value object:
-
-- If the value is one of the predefined types listed above, Persistit uses its own internal serialization logic.
-- If there is a registered +com.persistit.encoding.ValueCoder+ for the object's class, Persistit delegates to it.
-- If enabled, Persistit uses an accelerated serialization/deserialization mechanism to encode and decode objects.
-- Otherwise, for classes that implement java.io.Serializable, Persistit attempts to perform default Java serialization and deserialization.
-
-A Value may also be in the undefined state, which results from performing a fetch operation on a key for which no value is present in the database. The undefined state is distinct from the value +null+ and can be tested with the +isDefined()+ method.
-
-See <<Serialization>> for additional information.
-
-=== Large Values
-
-Persistit stores large values, in the current version up to 64MB in size. For example, it is possible to store an image’s backing bytes as a single value in the database. The size of the value to be stored is constrained by available heap memory; the entire value must be able to be serialized into an in-memory byte array in order for Persistit to store or retrieve it. Use +com.persistit.Value#setMaximumSize+ to specify a the size constraint. Large values are broken up across multiple data pages and are not necessarily stored in contiguous file areas.
-
-The definition of “large” depends on the configuration properties. for example, for a volume with a page size of 16K bytes the threshold occurs at 6,108 bytes. A value having a serialized size smaller than this is stored in a single data page while a larger value is broken up and stored in multiple pages. For a smaller pages size the threshold is lower.
-
-On occasion it may be desirable to fetch only part of a large value. For example, it may be useful to extract summary information from the beginning of a the backing byte array for an Image. Variants versions of the +fetch+ and +traverse+ accept a minimum byte count parameter. When these methods are used only the specified minimum number bytes of the backing store are retrieved from the database. This technique can prevent Persistit from reading large numbers of pages from the disk in order to examine only a small portion of the record.
-
-[[Exchange]]
-== Exchange
-
-The primary low-level interface for interacting with Persistit is +com.persistit.Exchange+. The Exchange class provides all methods for storing, deleting, fetching and traversing key/value pairs. These methods are summarized here and described in detail in the Javadoc API documentation.
-
-An Exchange instance contains references to a +Key+ and a +Value+. The methods +com.persistit.Exchange.getKey()+ and +com.persistit.Exchange.getValue()+ access these instances.
-
-To construct an Exchange you specify a Volume (or alias) and a tree name in its constructor. The constructor will optionally create a new tree in that Volume if a tree having the specified name has not already been created. An application may construct an arbitrary number of Exchange objects. Creating a new Exchange has no effect on the database if the specified tree already exists. Tree creation is thread-safe: multiple threads concurrently constructing Exchanges using the same Tree name will safely result in the creation of only one new tree.
-
-An Exchange is a moderately complex object that can consume tens of kilobytes to megabytes (depending on the sizes of the Key and Value) of heap space. Memory-constrained applications should construct Exchanges in moderatation.
-
-Persistit offers Exchange pooling to avoid rapidly creating and destroying Exchange objects in multi-threaded applications.  An application may use the +com.persistit.Persistit#getExchange+ and +com.persistit.Persistit#releaseExchange+ methods to take and return an Exchange from and to a thread-local pool.
-
-An Exchange internally maintains some optimization information such that references to nearby Keys within a tree are accelerated. Performance may benefit from using a different Exchange for each area of the Tree being accessed.
-
-=== Concurrent Operations on Exchanges
-
-Although the underlying Persistit database is designed for highly concurrent multi-threaded operation, the +Exchange+ class and its associated +Key+ and +Value+ instances are _not_ thread-safe. Each thread should acquire and use its own Exchange objects when accessing the database. Nonetheless, multiple threads can execute database operations on overlapping data concurrently using their thread-private +Exchange+ instances.
-
-Because Persistit permits concurrent operations by multiple threads, there is no guarantee that the underlying database will remain unchanged after an Exchange fetches or modifies its data. However, each operation on an Exchange is atomic, meaning that the inputs and outputs of each method are consistent with some valid state of the underlying Persistit backing store at some instant in time. The Exchange’s Value and Key objects represent that consistent state even if another thread subsequently modifies the database. Transactions, described below, allow multiple database operations to be performed atomically and consistently.
-
-=== Exchange API
-
-An Exchange has permanent references to a +com.persistit.Key+ and a +com.persistit.Value+. Typically you work with an Exchange in one of the following patterns:
-
-- Modify the Key, perform a +fetch+ operation, and extract the Value.
-- Modify the Key, modify the Value, and then perform a +store+ operation.
-- Modify the Key, and then perform a +remove+ operation.
-- Optionally modify the Key, perform a +traverse+ operation, then read the resulting Key and/or Value.
-
-These four methods, plus a few other methods listed here, are the primary low-level interface to the database. Semantics are as follows:
-
-[horizontal]
-+fetch+:: Reads the stored value associated with this Exchange's Key and modifies the Exchange’s Value to reflect that value.
-+store+:: Inserts or replaces the key/value pair for the specified key in the Tree either by replacing the former value, if there was one, or inserting a new value.
-+fetchAndStore+:: Reads and then replaces the stored value. Upon completion, Value reflects the formerly stored value for the current Key. This operation is atomic.
-+remove+, +removeAll+, +removeKeyRange+:: Removes key/value pairs from the Tree. Versions of this method specify either a single key or a range of keys to be removed.
-+fetchAndRemove+:: Fetches and then removes the stored value. Upon completion, Value reflects the formerly stored value for the current Key. This operation is atomic.
-+traverse+, +next+, +previous+:: Modifies the Exchange’s Key and Value to reflect a successor or predecessor key within the tree. See +com.persistit.Key+ for detailed information on the order of traversal.
-+hasNext+, +hasPrevious+:: Indicates, without modifying the Exchange’s Value or Key objects, whether there is a successor or predecessor key in the Tree.
-+hasChildren+:: Indicates whether there are records having keys that are logical children. A _logical child_of some key _P_ is any key that can be constructed by appending one or more key segments to _P_.
-
-
-For convenience, Exchange delegates +append+ and +to+ methods to +com.persistit.Key+. For example, Exchange provides the following methods that delegate to the identically named methods of Key :
-
-[source,java]
-----
-public Exchange append(long v)
-public Exchange append(String v)
-...
-----
-To allow code call-chaining these methods of Exchange return the same Exchange. For example, it is valid to write code such as
-
-[source,java]
-----
-exchange.clear().append(" Pacific").append("Ocean").append(123).fetch();
-----
-
-This example fetches the value associated with the concatenated key
-+{“Pacific”, ”Ocean”, 123}+.
-
-Exchange also delegates other key manipulation methods. (See +com.persistit.Exchange+ for detailed API documentation.)
-
-=== Traversing and Querying Collections of Data
-
-An Exchange provides a number of methods for traversing a collection of records in the Persistit database. These include variations of the +com.persistit.Exchange#traverse+, +com.persistit.Exchange#next+ and +com.persistit.Exchange#previous+. For all of these methods, Persistit does two things: it modifies the Exchange's +Key+ to reflect a new key that is before or after the current key, and it modifies the +Value+ associated with the Exchange to reflect the database value associated with that key.
-
-For example, this code from +HelloWorld.java+ prints out the key and value of each record in a tree:
-
-[source,java]
-----
-       	dbex.getKey().to(Key.BEFORE);
-       	while (dbex.next())
-       	{
-           	System.out.println(
-               	dbex.getKey().indexTo(0).decode() + " " +
-               	dbex.getValue().get());
-       	}
-----
-
-In general, the traversal methods let you find a key in a tree related to the key you supply. In Persistit programs you frequently prime a key value by appending either +com.persistit.Key#BEFORE+ or +com.persistit.Key#AFTER+. A key containing either of these special values can never be stored in a tree; these are reserved to represent positions in key traversal order before the first valid key and after the last valid key, respectively. You then invoke next or previous, or any of the other traverse family variants, to enumerate keys within the tree.
-
-You can specify whether traversal is _deep_ or _shallow_.  Deep traversal traverses the logical children (see com.persistit.Key) of a key. Shallow traversal traverses only the logical siblings.
-
-[[KeyFilter]]
-=== Selecting key values with a KeyFilter
-
-A +com.persistit.KeyFilter+ defines a subset of all possible key values. For example, a KeyFilter can select keys with certain fixed segment values, sets of values or ranges of values.  Calling +traverse+, +next+ or +previous+ with a KeyFilter efficiently traverses the subset of all keys in a Tree that match the filter.
-
-You construct a KeyFilter either by adding selection terms to it, or by calling the +com.persistit.KeyParser#parseKeyFilter+ method of the +com.persistit.KeyParser+ class to construct one from a string representation.
-
-Use of a KeyFilter is illustrated by the following code fragment:
-
-[source,java]
-----
-Exchange ex = new Exchange("myVolume", "myTree", true);
-KeyFilter kf = new KeyFilter("{\"Bellini\":\"Britten\"}");
-ex.append(Key.BEFORE);
-while (ex.next(kf)){
-    System.out.println(ex.getKey().reset().decodeString());
-}
-----
-
-This simple example emits the string-valued keys within Tree “myTree” whose values fall alphabetically between “Bellini” and “Britten”, inclusive.
-
-
-You will find an example with a KeyFilter in the examples/FindFileDemo directory.
-
-[[PersistitMap]]
-=== PersistitMap
-
-In addition to low-level access methods on keys and values, Persistit provides +com.persistit.PersistitMap+, which implements the +java.util.SortedMap+ interface. PersistitMap uses the Persistit database as a backing store so that key/value pairs are persistent, potentially shared with all threads, and limited in number only by disk storage.
-
-Keys and Values for PersistitMap must conform to the constraints described above under <<Key>> and <<Value>>.
-
-The constructor for PersistitMap takes an Exchange as its sole parameter. All key/value pairs of the Map are stored within the tree identified by this Exchange. The Key supplied by the Exchange becomes the root of a logical tree. For example:
-
-[source,java]
-----
-Exchange ex = new Exchange("myVolume", "myTree", true);
-ex.append("USA").append("MA");
-PersistitMap<String, String> map = new PersistitMap<String, String>(ex);
-map.put("Boston", "Hub");
-----
-
-places a key/value pair into Tree “myTree” with the concatenated key +{"USA ","MA","Boston"}+ and a value +"Hub"+.
-
-Generally the expected behavior for an Iterator on a Map collection view is to throw a +ConcurrentModificationException+ if the underlying collection changes. This is known as “fail-fast” behavior. PersistitMap implements this behavior by throwing a +ConcurrentModificationException+ in the event the Tree containing the map changes after the Iterator is constructed.
-
-However, sometimes it may be desirable to use PersistitMap and its collections view interfaces to iterate across changing data, especially for large databases. PersistitMap provides the method +com.persistit.PersistitMap#setAllowConcurrentModification+ to control whether changes made by other threads are permitted. By default, concurrent modifications are not allowed.
-
-NOTE: when +PersistitMap+ is used within a transaction updates generated by other concurrent transactions are not visible and therefore cannot cause a ConcurrentModificationException.  However, to avoid unpredictable results an Iterator created within the scope of a transaction must be used only within that transaction.
-
-
-=== Exceptions in PersistitMap
-
-Persistit operations throw a variety of exceptions that are subclasses of +com.persistit.exception.PersistitException+. However, the methods of the SortedMap interface do not permit arbitrary checked exceptions to be thrown. Therefore, PersistitMap wraps any PersistitException generated by the underlying database methods within a +com.persistit.PersistitMap.PersistitMapException+. This exception is unchecked and can therefore be thrown by methods of the Map interface. Applications using PersistitMap should catch and handle PersistitMap.PersistitMapException.
-
-=== Applying a KeyFilter to a PersistitMap Iterator
-
-You can specify a +com.persistit.KeyFilter+ for the Iterator returned by the +keySet()+, +entrySet()+ and +values()+ methods of +com.persistit.PersistitMap+.  The KeyFilter restricts the range of keys traversed by the Iterator. To set the KeyFilter, you must cast the Iterator to the inner class PersistitMap.ExchangeIterator, as shown here:
-
-[source,java]
-----
-	PersistitMap map = new PersistitMap(exchange);
-	PersistitMap.ExchangeIterator iterator =
-   	(PersistitMap.ExchangeIterator)map.entrySet().iterator();
-	iterator.setFilterTerm(KeyFilter.rangeTerm("A", "M"));
-----
-
-In this example, the iterator will only access String-valued keys between “A” and “M”.
-
-

=== added file 'doc/Configuration.rst'
--- doc/Configuration.rst	1970-01-01 00:00:00 +0000
+++ doc/Configuration.rst	2012-05-30 18:23:19 +0000
@@ -0,0 +1,273 @@
+.. _Configuration:
+
+Configuration
+=============
+
+To initialize Akiban Persistit the embedding application defines a configuration and then invokes one of the  ``com.persistit.Persistit#initialize`` methods. The configuration defines parameters used to determine locations of files, sizes of buffer pool and journal files, policies and other elements required when Persistit starts up. These parameters are managed by the ``com.persistit.Configuration`` class.
+
+An application can define the configuration in one of two equivalent ways:
+
+- Create a ``com.persistit.Configuration`` instance and set its properties through methods such as ``com.persistit.Configuration#setJournalPath``.
+- Specify properties by name in a ``java.util.Properties`` instance and then pass the ``Properties`` to a ``Configuration`` constructor.
+
+The following code samples show different ways of using the ``com.persistit.Persistit#initialize`` method to configure and start Persistit:
+
+.. code-block:: java
+
+  final Persistit db = new Persistit();
+  final Properties p = new Properties();
+  p.setProperty("buffer.count.16384", "1000");
+  p.setProperty("journalpath", "/home/akiban/data/journal");
+  ...
+  db.initialize(p);
+
+.. code-block:: java
+
+  final Persistit db = new Persistit();
+  db.initialize("/home/akiban/my_config.properties");
+
+.. code-block:: java
+
+  final Persistit db = new Persistit();
+  final Configuration c = new Configuration();
+  c.getBufferPoolMap().get(16384).setCount(1000);
+  c.setJournalPath("/home/akiban/data/journal");
+  ...
+  db.initialize(c);
+
+There are three essential elements in a Persistit configuration:
+
+- Memory for the buffer pool(s)
+- Specifications for ``com.persistit.Volume`` instances
+- Journal file path
+
+Configuring the Buffer Pool
+---------------------------
+
+During initialization Persistit allocates a fixed amount of heap memory for use as buffers. Depending on the application, it is usually desirable to allocate most of a server’s physical memory to the JVM heap (using the ``-Xmx`` and ``-Xms`` JVM options) and then to allocate a large fraction of the heap to Persistit buffers. The buffer pool allocation is determined during initialization and remains constant until the embedding application calls ``com.persistit.Persistit#close`` to release all resources.
+
+The number of buffers, and therefore the heap memory consumed, is determined during initialization by the available heap memory and configuration parameters. The heap memory size is obtained from the platform MemoryMXBean which in turn supplies the value given by the ``-Xmx`` JVM property. Configuration parameters are specified by a collection of five ``com.persistit.Configuration.BufferPoolConfiguration`` objects, one for each of the possible buffer sizes 1,024, 2,048, 4,096, 8,192 and 16,384 bytes. The method ``com.persistit.Configuration#getBufferPoolMap(int)`` gets the ``BufferPoolConfiguration`` for a specified buffer size.
+
+A ``BufferPoolConfiguration`` contains the following attributes:
+
+  ``minimumCount``
+      lower bound on the number of buffers. (Default is zero.)
+  ``maximumCount``
+      upper bound on the number of buffers. (Default is zero.)
+  ``minimumMemory``
+      lower bound on memory to allocate. (Default is zero bytes.)
+  ``maximumMemory``
+      upper bound on memory to allocate. (Default is Long.MAX_VALUE bytes.)
+  ``reservedMemory``
+      minimum number of bytes to reserve for use other than buffers.  (Default is zero.)
+  ``fraction``
+      floating point value between 0.0f and 1.0f indicating how much of available memory too allocate.  (Default is 1.0f.)
+
+Persistit uses the following algorithm to determine the number of buffers to allocate for each buffer size:
+
+.. code-block:: java
+
+  memoryToUse = fraction * (maxHeap - reservedMemory)
+  memoryToUse = min(memoryToUse, maximumMemory)
+  bufferCount = memoryToUse / bufferSizeWithOverhead
+  bufferCount = max(minimumCount, min(maximumCount, count))
+  if (bufferCount * bufferSize > maximumMemory) then FAIL
+  if ((bufferCount + 1) * bufferSize < minimumMemory) then FAIL
+  allocate bufferCount buffers
+
+In other words, Persistit computes a buffer count based on the memory parameters, bounds it by ``minimumCount`` and ``maximumCount`` and then checks whether the resulting allocation fits within the memory constraints. Note that ``bufferSizeWithOverhead`` is about 14% larger than the buffer size; the additional memory is reserved for indexing data and other overhead associated with the buffer.
+
+Typically an application uses a single buffer size, specifying either an absolute count or memory-based constraints for that size. This can be done by setting the attributes of the appropriate ``BufferPoolConfiguration`` object directly, or using Property values.
+
+The property named ``buffer.count.SSSS`` where ``SSSS`` is “1024”, “2048”, “4096”, “8192” or “16384” specifies an absolute count.  For example,
+
+.. code-block:: java
+
+  buffer.count.8192 = 10000
+
+causes Persistit to allocate 10,000 buffers of size 8192.
+
+The property ``buffer.memory.SSSS`` specifies memory constraints as shown in this example
+
+.. code-block:: java
+
+  buffer.memory.8192 = 512K,20M,4M,0.6
+
+where 512K, 20M, 4M and 0.6 are the ``minimumMemory``, ``maximumMemory``, ``reservedMemory`` and ``fraction``, respectively.
+
+The MemoryMXBean supplies as its maximum heap size value the size given by the ``-Xmx`` JVM parameter.
+
+Heap Tuning
+-----------
+
+This section pertains to the Oracle HotSpot(tm) Java virtual machine.
+
+.. note:: 
+
+   Buffer instances are long-lived objects. To avoid severe garbage collector overhead it is important for all of them    
+   to fit in the heap’s tenured generation. This issue becomes especially significant with multi-gigabyte heaps.
+
+By default the HotSpot server JVM allocates 1/3 of the heap to the new generation and 2/3 to the tenured generation, meaning that allocating more than 2/3 of the heap to buffers will result in bad performance.
+
+You can increase the fraction by specifying ``-XX:NewRatio=N`` where ``N`` indicates the ratio of tenured generation space to new generation space, or by using the ``-Xmn`` parameter to specify an absolute amount of memory for the new generation.  Also, setting ``-Xms`` equal to ``-Xmx`` will avoid numerous garbage collection cycles during the start-up process.
+
+See [http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html] for further information on tuning the heap and garbage collector for the HotSpot JVM.
+
+Multiple Buffer Pools
+---------------------
+
+In some cases it may be desirable to allocate two or more buffer pools having buffers of different sizes. For example, it may be beneficial to use a large number of small buffers to hold secondary index pages.
+
+When specifying multiple memory constraints for multiple buffer pools, the ``fraction`` property applies to the available memory before any buffers are allocated. So, for example,
+
+.. code-block:: java
+
+  buffer.memory.2048=64M,512G,2G,.2
+  buffer.memory.16384=64M,512G,2G,.5
+
+results in two buffer pools having buffers of size 2,048 bytes and 16,384 bytes, respectively. Assuming that the ``-Xmx`` value is 12G, then 2,048 byte buffers will be allocated to fill 20% of 10GByte, 16,384 byte buffers will be allocated to fill 50% of 10GByte, and approximately 5GByte (30% of 10GByte plus 2GByte reserved) will be available to application code.
+
+Configuring Volumes
+-------------------
+
+Persistit creates and/or opens a set of database volume files during start-up. An application can create, open and close additional volumes, but it is often convenient for volumes to be defined in the confiuration, outside of application code.
+
+The ``com.persistit.Configuration#getVolumeList`` method returns a List of ``com.persistit.VolumeSpecification`` objects. An application can construct and add new ``VolumeSpecification`` instances to this list before calling ``com.persistit.Persistit#initialize(Configuration)``.  Alternatively, the application can define volume specifications as property values using the syntax:
+
+``volume.N = path[,attrname[:attrvalue]]...``
+
+where ``N`` is an arbitrary integer, ``path`` is the path specification of the volume file, and ``attrnames`` include:
+
+- ``pageSize``: Fixed length unit representing one page. Value must be one of 1024, 2048, 4096, 8192 or 16384. To open and use the Volume, the buffer pool must have available buffers of the same size.
+
+- ``create``: Persistit attempts to open an existing volume file with the specified *path*, or create a new one if the file does not exist.
+
+- ``createOnly``: Persistit throw a VolumeAlreadyExistsException if the file specified by path already exists. Otherwise it creates a new file with the specified path.
+
+- ``readOnly``: Opens a volume in read-only mode. An attempt to modify the volume results in a ReadOnlyVolumeException.
+
+- ``initialPages`` or ``initialSize``: Specifies the initial size of the newly created volume file, either as the count of pages or as the size in bytes.
+
+- ``extensionPages`` or ``extensionSize``: Specifies the extension size of the newly created volume, either as the count of pages or as the size in bytes. This is the size by which the volume file will expand when the volume needs to be enlarged.
+
+- ``maximumPages`` or ``maximumSize``: An upper limit on the number of pages this Volume may hold, either as the count of pages or as the size in bytes. An attempt to further enlarge the Volume will generate a VolumeFullException.
+
+- ``alias``: The name of this Volume used in constructing ``Exchange`` instances.  If unspecified, the name is the simple file name given in the *path*, not including its dotted suffix.
+
+For example::
+
+  volume.1=/home/akiban/ffdemo,create,pageSize:16K,\
+      initialSize:10M,extensionSize:10M,maximumSize:1G
+
+specifies a volume having the name “ffdemo” in the /home/akiban directory. A new volume will be created if there is no existing volume file, and when created it will have the initial, extension and maximum sizes of 10MByte, 10MByte and 1GByte, respectively. Its page size will be 16KByte, meaning that the configuration must also have a buffer pool of 16KByte buffers.
+
+System Volume
+-------------
+
+One volume in a Persistit configuration must be designated as the system volume. It contains class meta data for objects stored serialized in Persistit Values. When a configuration specifies only one volume, that volume implicitly becomes the system volume by default. However, when a configuration specifies multiple volumes, you must indicate which volume will serve as the system volume. There are two ways to do this. By default, Persistit looks for a unique volume named “_system”. You can simply create a volume whose file name is “_system”.
+
+Alternatively, you can specify a system volume name explicitly with the ``sysvolume`` property (or ``com.persistit.Configuration#setSysVolume``). The value is the name or alias of the selected volume.
+
+Configuring the Journal Path
+----------------------------
+
+The :ref:`Journal` consists of a series of sequentially numbered files located in directory specified by the configuration parameter ``journalpath``. The application can set this property by calling ``com.persistit.Configuration#setJournalPath`` prior to initializing Persistit or through the property::
+
+  journalpath=/ssd/data/my_app_journal
+
+The value specified can be either a 
+
+- directory, in which case files named ``persistit_journal.NNNNNNNNNNNN`` will be created, 
+- or a file name, in which case journal files will be created by appending the suffix ``.NNNNNNNNNNNN``.
+
+Recommendations for Physical Media
+----------------------------------
+
+The journal is written by appending records to the end of the highest-numbered file. Read operations occur while copying page images from the journal to their home volume files. While copying, Persistit attempts to perform large sequential read operations from the journal. Read operations also occur at random when Persistit needs to reload the image of a previously evicted page.
+
+Because of these characteristics a modern SSD (solid disk drive) is ideally suited for maintaining the journal. If no SSD is available in the server, placing the journal on a different physical disk drive than the volume file(s) can significantly improve performance.
+
+Other Configuration Parameters
+------------------------------
+
+The following additional properties are defined for Persistit. Other properties may also reside in the Properties object or its backing file; Persistit simply ignores any property not listed here.
+
+  ``journalsize``: (``com.persistit.Configuration#setJournalSize``) 
+      Journal file block size. Default is 1,000,000,000 bytes. A new Persistit rolls over to a new journal file when this 
+      size is reached. Generally there is no reason to adjust this setting.
+
+  ``appendonly``: (``com.persistit.Configuration#setAppendOnly``), True or false (default).  
+      When true, Persistit’s journal starts up in *append-only* mode in which modified pages are only written to the 
+      journal and not copied to their home volumes. As a consequence, all existing journal files are preserved, and new 
+      modifications are written only to newly created journal files. The append-only flag can also be enabled or disabled 
+      by application code and through the JMX and RMI interfaces.
+
+  ``rmiport``: (``com.persistit.Configuration#setRmiPort``) 
+      Specifies a port number on which Persistit will create a temporary Remote Method Invocation registry.  If this 
+      property is specified, Persistit creates a registry and registers a ``com.persistit.Management`` server on it. This 
+      allows remote access to management facilities within Persistit and permits the Swing-based administrative utility to 
+      attach to and manage a Persistit instance running on a headless server. The ``rmihost`` and ``rmiport`` properties 
+      are mutually exclusive.
+
+  ``rmihost``: (``com.persistit.Configuration#setRmiHost``) 
+      Specifies the URL of an Remote Method Invocation registry.  If present, Persistit registers its a server for its 
+      ``com.persistit.Management`` interface at the specified external registry. The ``rmihost`` and ``rmiport`` 
+      properties are mutually exclusive.
+
+  ``jmx``: (``com.persistit.Configuration#setJmxEnabled``), True or false (default). 
+      Specifies whether Persistit registers MXBeans with the platform MBean server. Set this value to ``true`` to enable 
+      access from ``jconsole`` and other management tools.
+
+  ``serialOverride``, ``constructorOverride``: (``com.persistit.Configuration#setSerialOverride`` ``com.persistit.Configuration#setConstructorOverride``) 
+      Control aspects of object serialization. See :ref:`Serialization`.
+
+  ``showgui``: (``com.persistit.Configuration#setShowGUI``), True of False.  
+      If true, Persistit attempts to create and display an instance of the AdminUI utility panel within the current JVM. 
+      Alternatively, AdminUI uses RMI and can be launched and run remotely if ``rmiport`` or ``rmihost`` has been 
+      specified.
+
+  ``logfile``: (``com.persistit.Configuration#setLogFile``) 
+      Name of a log file to which Persistit’s default logger will write diagnostic log entries. Applications generally 
+      install a logging adapter to reroute messages through Log4J, SLF4J or other logger. The ``logfile`` property is used 
+      only when no adapter has been installed.
+
+For all integer-valued properties, the suffix “K” may be used to represent kilo, “M” for mega, “G” for giga and “T” for tera. For example, “2M” represents the value 2,097,152.
+
+A Configuration Example
+-----------------------
+
+Following is an example of a Persistit configuration properties file::
+
+  datapath = /var/opt/persistit/data
+  logpath = /var/log/persistit
+  logfile = ${logpath}/${timestamp}.log
+
+  buffer.count.16384 = 5000
+
+  volume.1 = ${datapath}/demo_data, create, pageSize:16K, \
+  	  initialSize:1M, extensionSize:1M, maximumSize:10G, alias:data
+
+  volume.2 = ${datapath}/demo_system, create, pageSize:16K, \
+	  initialSize:100K, extensionSize:100K, maximumSize:1G
+
+  sysvolume = demo_system
+
+  journalpath = /ssd/persistit_journal
+
+With this configuration there will be 5,000 16K buffers in the buffer pool consuming heap space of approximately 93MB including overhead. Persistit will open or create volume files named ``/var/opt/persistit/data/demo_data`` and ``/var/opt/persistit/data/demo_system`` and a journal file named ``/ssd/persistit_journal.0000000000000000``. Persistit will write diagnostic logging output to a file such as ``/var/log/persistit/20110523172213.log``.
+
+The ``demo_data`` volume has the alias ``data``. Application code uses the name "data" to refer to it. The ``sysvolume`` property specifies that the ``demo_system`` volume is designated to hold class meta data for serialized objects.
+
+Property Value Substitution
+---------------------------
+
+This example also illustrates how property value substitution can be used within a Persistit configuration.  The value of the ``datapath`` replaces ``$\{datapath\}`` in the volume specification. The property name ``datapath`` is arbitrary; you may use any valid property name as a substitution variable. Similarly, the value of ``logpath`` replaces ``$\{logpath\}`` and the pseudo-property ``$\{timestamp\}`` expands to a timestamp in the form ``*yyyyMMddHHmm*`` to provides a unique time-based log file name.
+
+Incorporating Java System Properties
+------------------------------------
+
+You may also specify any configuration property as a Java system property with the prefix ``com.persisit.`` System properties override values specified as properties. For example, you can override the value of ``buffer.count.8192`` specifying::
+
+  java -Dcom.persistit.buffer.count.8192=10K -jar MyJar
+
+This is also true for substitution property values. For example, ``-Dcom.persistit.logpath=/tmp/`` will place the log files in the ``/tmp`` directory rather than ``/var/log/persistit`` as specified by the configuration file.

=== removed file 'doc/Configuration.txt'
--- doc/Configuration.txt	2012-04-30 22:09:31 +0000
+++ doc/Configuration.txt	1970-01-01 00:00:00 +0000
@@ -1,246 +0,0 @@
-[[Configuration]]
-= Configuration
-
-To initialize Akiban Persistit the embedding application defines a configuration and then invokes one of the  +com.persistit.Persistit#initialize+ methods. The configuration defines parameters used to determine locations of files, sizes of buffer pool and journal files, policies and other elements required when Persistit starts up. These parameters are managed by the +com.persistit.Configuration+ class.
-
-An application can define the configuration in one of two equivalent ways:
-
-- Create a +com.persistit.Configuration+ instance and set its properties through methods such as +com.persistit.Configuration#setJournalPath+.
-- Specify properties by name in a +java.util.Properties+ instance and then pass the +Properties+ to a +Configuration+ constructor.
-
-The following code samples show different ways of using the +com.persistit.Persistit#initialize+ method to configure and start Persistit:
-
-[source,java]
-----
-final Persistit db = new Persistit();
-final Properties p = new Properties();
-p.setProperty("buffer.count.16384", "1000");
-p.setProperty("journalpath", "/home/akiban/data/journal");
-...
-db.initialize(p);
-----
-
-[source,java]
-----
-final Persistit db = new Persistit();
-db.initialize("/home/akiban/my_config.properties");
-----
-
-[source,java]
-----
-final Persistit db = new Persistit();
-final Configuration c = new Configuration();
-c.getBufferPoolMap().get(16384).setCount(1000);
-c.setJournalPath("/home/akiban/data/journal");
-...
-db.initialize(c);
-----
-
-There are three essential elements in a Persistit configuration:
-
-- Memory for the buffer pool(s)
-- Specifications for +com.persistit.Volume+ instances
-- Journal file path
-
-== Configuring the Buffer Pool
-
-During initialization Persistit allocates a fixed amount of heap memory for use as buffers. Depending on the application, it is usually desirable to allocate most of a server’s physical memory to the JVM heap (using the +-Xmx+ and +-Xms+ JVM options) and then to allocate a large fraction of the heap to Persistit buffers. The buffer pool allocation is determined during initialization and remains constant until the embedding application calls +com.persistit.Persistit#close+ to release all resources.
-
-The number of buffers, and therefore the heap memory consumed, is determined during initialization by the available heap memory and configuration parameters. The heap memory size is obtained from the platform MemoryMXBean which in turn supplies the value given by the +-Xmx+ JVM property. Configuration parameters are specified by a collection of five +com.persistit.Configuration.BufferPoolConfiguration+ objects, one for each of the possible buffer sizes 1,024, 2,048, 4,096, 8,192 and 16,384 bytes. The method +com.persistit.Configuration#getBufferPoolMap(int)+ gets the +BufferPoolConfiguration+ for a specified buffer size.
-
-A +BufferPoolConfiguration+ contains the following attributes:
-
-[horizontal]
-minimumCount:: lower bound on the number of buffers. (Default is zero.)
-maximumCount:: upper bound on the number of buffers. (Default is zero.)
-minimumMemory:: lower bound on memory to allocate. (Default is zero bytes.)
-maximumMemory:: upper bound on memory to allocate. (Default is Long.MAX_VALUE bytes.)
-reservedMemory:: minimum number of bytes to reserve for use other than buffers.  (Default is zero.)
-fraction:: floating point value between 0.0f and 1.0f indicating how much of available memory too allocate.  (Default is 1.0f.)
-
-Persistit uses the following algorithm to determine the number of buffers to allocate for each buffer size:
-
-[source,java]
-----
-memoryToUse = fraction * (maxHeap - reservedMemory)
-memoryToUse = min(memoryToUse, maximumMemory)
-bufferCount = memoryToUse / bufferSizeWithOverhead
-bufferCount = max(minimumCount, min(maximumCount, count))
-if (bufferCount * bufferSize > maximumMemory) then FAIL
-if ((bufferCount + 1) * bufferSize < minimumMemory) then FAIL
-allocate bufferCount buffers
-----
-
-In other words, Persistit computes a buffer count based on the memory parameters, bounds it by +minimumCount+ and +maximumCount+ and then checks whether the resulting allocation fits within the memory constraints. Note that +bufferSizeWithOverhead+ is about 14% larger than the buffer size; the additional memory is reserved for indexing data and other overhead associated with the buffer.
-
-Typically an application uses a single buffer size, specifying either an absolute count or memory-based constraints for that size. This can be done by setting the attributes of the appropriate +BufferPoolConfiguration+ object directly, or using Property values.
-
-The property named +buffer.count._SSSS_+ where _SSSS_ is “1024”, “2048”, “4096”, “8192” or “16384” specifies an absolute count.  For example,
-
-----
-buffer.count.8192 = 10000
-----
-
-causes Persistit to allocate 10,000 buffers of size 8192.
-
-The property +buffer.memory._SSSS_+ specifies memory constraints as shown in this example
-
-----
-buffer.memory.8192 = 512K,20M,4M,0.6
-----
-
-where 512K, 20M, 4M and 0.6 are the +minimumMemory+, +maximumMemory+, +reservedMemory+ and +fraction+, respectively.
-
-The MemoryMXBean supplies as its maximum heap size value the size given by the +-Xmx+ JVM parameter.
-
-=== Heap Tuning
-
-This section pertains to the Oracle HotSpot(tm) Java virtual machine.
-
-****
-Buffer instances are long-lived objects. To avoid severe garbage collector overhead it is important for all of them to fit in the heap’s tenured generation. This issue becomes especially significant with multi-gigabyte heaps.
-****
-
-By default the HotSpot server JVM allocates 1/3 of the heap to the new generation and 2/3 to the tenured generation, meaning that allocating more than 2/3 of the heap to buffers will result in bad performance.
-
-You can increase the fraction by specifying +-XX:NewRatio=_N_+ where _N_ indicates the ratio of tenured generation space to new generation space, or by using the +-Xmn+ parameter to specify an absolute amount of memory for the new generation.  Also, setting +-Xms+ equal to +-Xmx+ will avoid numerous garbage collection cycles during the start-up process.
-
-See [http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html] for further information on tuning the heap and garbage collector for the HotSpot JVM.
-
-=== Multiple Buffer Pools
-
-In some cases it may be desirable to allocate two or more buffer pools having buffers of different sizes. For example, it may be beneficial to use a large number of small buffers to hold secondary index pages.
-
-When specifying multiple memory constraints for multiple buffer pools, the +fraction+ property applies to the available memory before any buffers are allocated. So, for example,
-
-
-----
-buffer.memory.2048=64M,512G,2G,.2
-buffer.memory.16384=64M,512G,2G,.5
-----
-
-results in two buffer pools having buffers of size 2,048 bytes and 16,384 bytes, respectively. Assuming that the +-Xmx+ value is 12G, then 2,048 byte buffers will be allocated to fill 20% of 10GByte, 16,384 byte buffers will be allocated to fill 50% of 10GByte, and approximately 5GByte (30% of 10GByte plus 2GByte reserved) will be available to application code.
-
-== Configuring Volumes
-
-Persistit creates and/or opens a set of database volume files during start-up. An application can create, open and close additional volumes, but it is often convenient for volumes to be defined in the confiuration, outside of application code.
-
-The +com.persistit.Configuration#getVolumeList+ method returns a List of +com.persistit.VolumeSpecification+ objects. An application can construct and add new +VolumeSpecification+ instances to this list before calling +com.persistit.Persistit#initialize(Configuration)+.  Alternatively, the application can define volume specifications as property values using the syntax:
-
-+volume._N_ = _path_[,_attrname_[:_attrvalue_]]...+
-
-where _N_ is an arbitrary integer, _path_ is the path specification of the volume file, and _attrnames_ include:
-
-- +pageSize+: Fixed length unit representing one page. Value must be one of 1024, 2048, 4096, 8192 or 16384. To open and use the Volume, the buffer pool must have available buffers of the same size.
-
-- +create+: Persistit attempts to open an existing volume file with the specified _path_, or create a new one if the file does not exist.
-
-- +createOnly+: Persistit throw a VolumeAlreadyExistsException if the file specified by path already exists. Otherwise it creates a new file with the specified path.
-
-- +readOnly+: Opens a volume in read-only mode. An attempt to modify the volume results in a ReadOnlyVolumeException.
-
-- +initialPages+ or +initialSize+: Specifies the initial size of the newly created volume file, either as the count of pages or as the size in bytes.
-
-- +extensionPages+ or +extensionSize+: Specifies the extension size of the newly created volume, either as the count of pages or as the size in bytes. This is the size by which the volume file will expand when the volume needs to be enlarged.
-
-- +maximumPages+ or +maximumSize+: An upper limit on the number of pages this Volume may hold, either as the count of pages or as the size in bytes. An attempt to further enlarge the Volume will generate a VolumeFullException.
-
-- +alias+: The name of this Volume used in constructing +Exchange+ instances.  If unspecified, the name is the simple file name given in the _path_, not including its dotted suffix.
-
-For example:
-
-----
-volume.1=/home/akiban/ffdemo,create,pageSize:16K,\
-    initialSize:10M,extensionSize:10M,maximumSize:1G
-----
-
-specifies a volume having the name “ffdemo” in the /home/akiban directory. A new volume will be created if there is no existing volume file, and when created it will have the initial, extension and maximum sizes of 10MByte, 10MByte and 1GByte, respectively. Its page size will be 16KByte, meaning that the configuration must also have a buffer pool of 16KByte buffers.
-
-=== System Volume
-
-One volume in a Persistit configuration must be designated as the system volume. It contains class meta data for objects stored serialized in Persistit Values. When a configuration specifies only one volume, that volume implicitly becomes the system volume by default. However, when a configuration specifies multiple volumes, you must indicate which volume will serve as the system volume. There are two ways to do this. By default, Persistit looks for a unique volume named “_system”. You can simply create a volume whose file name is “_system”.
-
-Alternatively, you can specify a system volume name explicitly with the +sysvolume+ property (or +com.persistit.Configuration#setSysVolume+). The value is the name or alias of the selected volume.
-
-== Configuring the Journal Path
-
-The <<Journal>> consists of a series of sequentially numbered files located in directory specified by the configuration parameter +journalpath+. The application can set this property by calling +com.persistit.Configuration#setJournalPath+ prior to initializing Persistit or through the property
-
-----
-journalpath=/ssd/data/my_app_journal
----- 
-
-The value specified can be either a 
-
-- directory, in which case files named +persistit_journal._NNNNNNNNNNNN_+ will be created, 
-
-- or a file name, in which case journal files will be created by appending the suffix +._NNNNNNNNNNNN_+.
-
-=== Recommendations for Physical Media
-
-The journal is written by appending records to the end of the highest-numbered file. Read operations occur while copying page images from the journal to their home volume files. While copying, Persistit attempts to perform large sequential read operations from the journal. Read operations also occur at random when Persistit needs to reload the image of a previously evicted page.
-
-Because of these characteristics a modern SSD (solid disk drive) is ideally suited for maintaining the journal. If no SSD is available in the server, placing the journal on a different physical disk drive than the volume file(s) can significantly improve performance.
-
-== Other Configuration Parameters
-
-The following additional properties are defined for Persistit. Other properties may also reside in the Properties object or its backing file; Persistit simply ignores any property not listed here.
-
-[horizontal]
-+journalsize+:: (+com.persistit.Configuration#setJournalSize+) Journal file block size. Default is 1,000,000,000 bytes. A new Persistit rolls over to a new journal file when this size is reached. Generally there is no reason to adjust this setting.
-
-+appendonly+:: (+com.persistit.Configuration#setAppendOnly+) True or false (default).  When true, Persistit’s journal starts up in _append-only_ mode in which modified pages are only written to the journal and not copied to their home volumes. As a consequence, all existing journal files are preserved, and new modifications are written only to newly created journal files. The append-only flag can also be enabled or disabled by application code and through the JMX and RMI interfaces.
-
-+rmiport+:: (+com.persistit.Configuration#setRmiPort+) Specifies a port number on which Persistit will create a temporary Remote Method Invocation registry.  If this property is specified, Persistit creates a registry and registers a +com.persistit.Management+ server on it. This allows remote access to management facilities within Persistit and permits the Swing-based administrative utility to attach to and manage a Persistit instance running on a headless server. The +rmihost+ and +rmiport+ properties are mutually exclusive.
-
-+rmihost+:: (+com.persistit.Configuration#setRmiHost+) Specifies the URL of an Remote Method Invocation registry.  If present, Persistit registers its a server for its +com.persistit.Management+ interface at the specified external registry. The +rmihost+ and +rmiport+ properties are mutually exclusive.
-
-+jmx+:: (+com.persistit.Configuration#setJmxEnabled+) True or false (default). Specifies whether Persistit registers MXBeans with the platform MBean server. Set this value to +true+ to enable access from +jconsole+ and other management tools.
-
-+serialOverride+, +constructorOverride+:: (+com.persistit.Configuration#setSerialOverride+ +com.persistit.Configuration#setConstructorOverride+) Control aspects of object serialization. See <<Serialization>>.
-
-+showgui+:: ((+com.persistit.Configuration#setShowGUI+) True of False.  If true, Persistit attempts to create and display an instance of the AdminUI utility panel within the current JVM. (Alternatively, AdminUI uses RMI and can be launched and run remotely if +rmiport+ or +rmihost+ has been specified.
-
-+logfile+:: (+com.persistit.Configuration#setLogFile+) Name of a log file to which Persistit’s default logger will write diagnostic log entries. Applications generally install a logging adapter to reroute messages through Log4J, SLF4J or other logger. The +logfile+ property is used only when no adapter has been installed.
-
-For all integer-valued properties, the suffix “K” may be used to represent kilo, “M” for mega, “G” for giga and “T” for tera. For example, “2M” represents the value 2,097,152.
-
-== A Configuration Example
-
-Following is an example of a Persistit configuration properties file:
-
-----
-datapath = /var/opt/persistit/data
-logpath = /var/log/persistit
-logfile = ${logpath}/${timestamp}.log
-
-buffer.count.16384 = 5000
-
-volume.1 = ${datapath}/demo_data, create, pageSize:16K, \
-	initialSize:1M, extensionSize:1M, maximumSize:10G, alias:data
-
-volume.2 = ${datapath}/demo_system, create, pageSize:16K, \
-	initialSize:100K, extensionSize:100K, maximumSize:1G
-
-sysvolume = demo_system
-
-journalpath = /ssd/persistit_journal
-----
-
-With this configuration there will be 5,000 16K buffers in the buffer pool consuming heap space of approximately 93MB including overhead. Persistit will open or create volume files named +/var/opt/persistit/data/demo_data+ and +/var/opt/persistit/data/demo_system+ and a journal file named +/ssd/persistit_journal.0000000000000000+. Persistit will write diagnostic logging output to a file such as +/var/log/persistit/20110523172213.log+.
-
-The +demo_data+ volume has the alias +data+. Application code uses the name "data" to refer to it. The +sysvolume+ property specifies that the +demo_system+ volume is designated to hold class meta data for serialized objects.
-
-=== Property Value Substitution
-
-This example also illustrates how property value substitution can be used within a Persistit configuration.  The value of the +datapath+ replaces +$\{datapath\}+ in the volume specification. The property name +datapath+ is arbitrary; you may use any valid property name as a substitution variable. Similarly, the value of +logpath+ replaces +$\{logpath\}+ and the pseudo-property +$\{timestamp\}+ expands to a timestamp in the form +_yyyyMMddHHmm_+ to provides a unique time-based log file name.
-
-=== Incorporating Java System Properties
-
-You may also specify any configuration property as a Java system property with the prefix +com.persisit.+ System properties override values specified as properties. For example, you can override the value of +buffer.count.8192+ specifying
-
-----
-java -Dcom.persistit.buffer.count.8192=10K -jar MyJar
-----
-
-This is also true for substitution property values. For example, +-Dcom.persistit.logpath=/tmp/+ will place the log files in the +/tmp+ directory rather than +/var/log/persistit+ as specified by the configuration file.

=== added file 'doc/GettingStarted.rst'
--- doc/GettingStarted.rst	1970-01-01 00:00:00 +0000
+++ doc/GettingStarted.rst	2012-05-30 18:23:19 +0000
@@ -0,0 +1,270 @@
+
+Getting Started with Akiban Persistit
+=====================================
+
+Welcome!
+
+We have worked hard to make Akiban Persistit(TM) exceptionally fast, reliable, simple and lightweight. We hope you will enjoy learning more about it and using it.
+
+Akiban Persistit is a key/value data storage library written in Java(TM). Key features include:
+
+- support for highly concurrent transaction processing with multi-version concurrency control
+- optimized serialization and deserialization mechanism for Java primitives and objects
+- multi-segment (compound) keys to enable a natural logical key hierarchy
+- support for long records (megabytes)
+- implementation of a persistent SortedMap
+- extensive management capability including command-line and GUI tools
+
+This chapter briefly and informally introduces and demonstrates various Persistit features through examples. Subsequent chapters and the Javadoc API documentation provides a detailed reference guide to the product.
+
+Download and Install
+--------------------
+
+Download ``akiban-persistit-3.xx.yy.zip`` from http://www.akiban.com/persistit/download.html.
+
+Unpack the distribution kit into a convenient directory using any unzip utility. For example, use ``jar`` to unpack the distribution kit to the current working directory as follows:
+.. code-block::
+
+  jar xvzf akiban-persistit-core-3.xx.yy.zip
+
+Review the ``LICENSE.html`` and ``README.html`` files located in the root of the installation directory. Persistit is licensed under the GNU Affero General Public License. By installing, copying or otherwise using the Software contained in the distribution kit, you agree to be bound by the terms of the license agreement. If you do not agree to these terms, remove and destroy all copies of the software in your possession immediately.
+
+Working with Persistit
+----------------------
+
+Add the ``akiban-persistit-core-3.xx.yy.jar`` from the ``lib`` directory of the distribution kit to your project's classpath. For example, copy it to ``jre/lib/ext`` in your Java Runtime Environment, or add it to your classpath environment variable. 
+
+That's it. You are ready to work with Persistit.
+
+Examples
+^^^^^^^^
+
+Review the ``examples`` directory. Here you will find functional examples of varying complexity.
+
+  ``examples/HelloWorld``
+      source code for the example illustrating this chapter
+  ``examples/SimpleDemo``
+      short, simple program similar to HelloWorld
+  ``examples/SimpleBench``
+      a small micro-benchmark measuring the speed of insert, traversal, random updates, etc.
+  ``examples/SimpleTransaction``
+      example demonstrating use of Persisit’s multi-version currency control (MVCC) transactions
+  ``examples/FindFile``
+      a somewhat larger example that uses Persistit as the backing store for file finder utility
+  ``example/PersistitMapDemo``
+      demonstration of the PersistitMap interface
+
+HelloWorld
+----------
+
+Before going further let's honor tradition with a small program that stores, fetches and displays the phrase “Hello World.” In this program we will create a record with the key “Hello” and the value “World”.
+
+.. HelloWorld.java
+
+.. code-block:: java
+
+  import com.persistit.Exchange;
+  import com.persistit.Key;
+  import com.persistit.Persistit;
+
+  public class HelloWorld {
+      public static void main(String[] args) throws Exception {
+          Persistit db = new Persistit();
+          try {
+              // Read configuration from persistit.properties, allocate
+              // buffers, open Volume, and perform recovery processing
+              // if necessary.
+              //
+              db.initialize();
+              //
+              // Create an Exchange, which is a thread-private facade for
+              // accessing data in a Persistit Tree. This Exchange will
+              // access a Tree called "greetings" in a Volume called
+              // "hwdemo". It will create a new Tree by that name
+              // if one does not already exist.
+              //
+              Exchange dbex = db.getExchange("hwdemo", "greetings", true);
+              //
+              // Set up the Value field of the Exchange.
+              //
+              dbex.getValue().put("World");
+              //
+              // Set up the Key field of the Exchange.
+              //
+              dbex.getKey().append("Hello");
+              //
+              // Ask Persistit to put this key/value pair into the Tree.
+              // Until this point, the changes to the Exchange are local
+              // to this thread.
+              //
+              dbex.store();
+              //
+              // Prepare to traverse all the keys in the Tree (of which there
+              // is currently only one!) and for each key display its value.
+              //
+              dbex.getKey().to(Key.BEFORE);
+              while (dbex.next()) {
+                  System.out.println(dbex.getKey().indexTo(0).decode() + " "
+                          + dbex.getValue().get());
+              }
+              db.releaseExchange(dbex);
+          } finally {
+              // Always close Persistit. If the application does not do
+              // this, Persistit's background threads will keep the JVM from
+              // terminating.
+              //
+              db.close();
+          }
+      }
+  }
+
+Concepts
+--------
+
+Although ``HelloWorld.java`` is not very useful, it demonstrates several of the basic building blocks of the Persistit API.
+
+Initialization and Configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Before accessing any data, ``HelloWorld.java`` calls one of the ``com.persistit.Persistit#initialize`` methods of ``com.persistit.Persistit``. This sets up the memory configuration for buffers and the path names of Persistit volume and journal files. Alternative versions of the initialize method accept configuration information from a ``java.util.Properties`` object, from a specified properties file, or by default from the file named ``persistit.properties``.
+
+In this example, ``persistit.properties`` looks like this:: 
+
+  datapath=.
+  buffer.count.8192=32
+  volume.1=${datapath}/hwdemo,create,pageSize:8192,\
+	  initialPages:5,extensionPages:5,maximumPages:100000
+  journalpath=${datapath}/hwdemo_journal
+
+See :ref:`Configuration` for additional information about Persistit configuration properties.
+
+Volumes and Trees
+^^^^^^^^^^^^^^^^^
+
+A configuration defines one or more volume files that will contain stored Persistit data. Usually you will specify the ``create`` flag, which allows Persistit to create a new volume if the file does not already exist. Creating a new file also establishes the initial size and growth parameters for that volume.
+
+Each volume may contain an unlimited number of named trees. Each tree within a volume embodies a logically distinct B-Tree index structure. Think of a tree as simply a named key space within a volume.
+
+``HelloWorld.java`` stores its key/value pair in a tree called “greetings” in a volume named “hwdemo”. This is specified by constructing an Exchange.
+
+Exchanges
+---------
+
+The ``com.persistit.Exchange`` class is the primary facade for interacting with Persistit data. It is so-named because it allows an application to exchange information with the database. An Exchange provides methods for storing, deleting, fetching and traversing key/value pairs.
+
+The method
+
+.. code-block:: java
+
+  Exchange dbex = db.getExchange("hwdemo", "greetings", true);
+
+in ``HelloWorld.java`` finds a volume named "hwdemo" and attempts to find a tree in it named "greetings". If there is no such tree, ``getExchange`` creates it.
+
+Methods ``com.persistit.Persistit#getExchange`` and ``com.persistit.Persistit#releaseExchange`` maintain a pool of reusable Exchange objects designed for use by multi-threaded applications such as web applications. If a suitable exchange already exists, ``getExchange`` returns it; otherwise it constructs a new one.
+
+The Exchange looks up the volume name “hwdemo” by matching it against the volumes specified in the configuration. The match is based on the simple file name of the volume after removing its final dotted suffix.  For example, the volume name “hwdemo” matches the volume specification ``${datapath}/hwdemo.v00``.
+
+Each Exchange is implicitly associated with a ``com.persistit.Key`` and a ``com.persistit.Value``. Typically you work with an Exchange in one of the following patterns:
+
+- Modify the Key, modify the Value and then perform a ``com.persistit.Exchange#store`` operation.
+- Modify the Key, perform a ``com.persistit.Exchange#fetch`` operation and then read the Value.
+- Modify the Key and then perform a ``com.persistit.Exchange#remove`` operation.
+- Optionally modify the Key, perform a ``com.persistit.Exchange#next``, ``com.persistit.Exchange#previous`` or ``com.persistit.Exchange#traverse`` operation, then read the resulting Key and/or Value.
+
+These methods and their variants provide the foundation for using Persistit.
+
+Records
+^^^^^^^
+
+In Persistit, a database record consists of a Key and a Value. The terms “record” and “key/value pair” are used interchangeably.
+
+When you store a record, Persistit searches for a previously stored record having the same key. If there is such a record, Persistit replaces its value.  If there is no such record, Persistit inserts a new one.  Like a Java Map, Persistit stores at most one value per key, and every record in a Tree has a unique key value.
+
+Keys
+^^^^
+
+A Key contains a unique identifier for key/value pair - or record - in a tree. The identifier consists of a sequence of one or more Java values encoded into an array of bytes stored in the volume file.
+
+Key instances are mutable. Your application typically changes an Exchange's Key in preparation for fetching or retrieving data. In particular, you can append, remove or replace one or more values in a Key. Each value you append is called a *key segment*. You append multiple key segments to implement concatenated keys. See ``com.persistit.Key`` for additional information on constructing keys and the ordering of key traversal within a tree.
+
+The ``HelloWorld.java`` example appends “Hello” to the Exchange’s Key object in this line:
+
+.. code-block:: java
+
+            dbex.getKey().append("Hello");
+
+The result is a key with a single key segment.
+
+Values
+^^^^^^
+
+A Value object represents the serialized state of a Java object or a primitive value. It is a staging area for data being transferred from or to the database by ``fetch``, ``traverse`` and ``store`` operations.
+
+Value instances are mutable. The ``fetch`` and ``traverse`` operations modify the state of an Exchange's Value instance to represent the value associated with some Key. Your application executes methods to modify the state of the Value instance in preparation for storing new data values into the database.
+
+Numerous methods allow you to serialize and deserialize primitives values and objects into and from a Value object. For example, in ``HelloWorld.java``, the statement
+
+.. code-block:: java
+
+            dbex.getValue().put("World");
+
+serializes the string “World” into the backing byte array of the Exchange’s Value object and
+
+.. code-block:: java
+
+            	System.out.println(
+                	dbex.getKey().indexTo(0).decode() + " " +
+                	dbex.getValue().get());
+
+deserializes and prints an object value from the Key and another object value from the Value. Value also has methods such as ``getInt``, ``getLong``, ``getByteArray`` to decode primitive and array values directly.
+
+Storing and Fetching Data
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Finally, it is these two methods in ``HelloWorld.java`` that cause the Exchange object to share data with the B-Tree, making it persistent and potentially available to other threads:
+
+.. code-block:: java
+
+            dbex.store();
+            ...
+            while (dbex.next()) { ... }
+
+Closing Persistit
+^^^^^^^^^^^^^^^^^
+
+Persistit creates one or more background threads that lazily write data to the Volume files and perform other maintenance activities. Be sure to invoke the ``com.persistit.Persistit#close`` method to allow these threads to finish their work and exit properly. The pattern illustrated in ``HelloWorld.java``, using a *try/finally* block to invoke ``close``, is strongly recommended.
+
+The ``com.persistit.Persistit#close(boolean)`` method optionally flushes all data to disk from the buffer pool before shutting down. Specifying the ``false`` option will close Persistit more quickly will lose recent updates if they were not performed inside of transactions, or will potentially require a longer recovery process during the next startup to reapply committed transactions.
+Additional Topics
+-----------------
+
+PersistitMap
+^^^^^^^^^^^^
+A particularly easy way to get started with Persistit is to use its built-in ``com.persistit.PersistitMap`` implementation. PersistitMap implements the ``java.util.SortedMap`` interface, so it can directly replace ``java.util.TreeMap`` or other kinds of Map in existing Java code.
+
+See :ref:`PersistitMap`.
+
+KeyFilters
+^^^^^^^^^^
+
+A ``com.persistit.KeyFilter`` can be supplied to restrict the results traversal operation in a convenient and  
+
+Transactions
+^^^^^^^^^^^^
+
+Persistit provides ACID Transaction support with multi-version concurrency control (MCC) and adjustable durability policy.
+
+See :ref:`Transactions`.
+
+Managing Persistit
+^^^^^^^^^^^^^^^^^^
+
+Persistit provides several mechanisms for managing Persistit operation within an application. These include
+
+- JMX MXBeans
+- The ``com.persistit.Management`` object which provides programmatic access to many management operations
+- The ``com.persistit.CLI`` object which provides a command-line interface for various management operations
+- The AdminUI tool which provides a graphical client interface for examining records and other resources
+- Logging interface design for easy embedding in host applications
+
+See :ref:`Management`.

=== removed file 'doc/GettingStarted.txt'
--- doc/GettingStarted.txt	2012-04-24 02:37:12 +0000
+++ doc/GettingStarted.txt	1970-01-01 00:00:00 +0000
@@ -1,255 +0,0 @@
-
-= Getting Started with Akiban Persistit
-
-
-Welcome!
-
-We have worked hard to make Akiban Persistit(TM) exceptionally fast, reliable, simple and lightweight. We hope you will enjoy learning more about it and using it.
-
-Akiban Persistit is a key/value data storage library written in Java(TM). Key features include:
-
-- support for highly concurrent transaction processing with multi-version concurrency control
-- optimized serialization and deserialization mechanism for Java primitives and objects
-- multi-segment (compound) keys to enable a natural logical key hierarchy
-- support for long records (megabytes)
-- implementation of a persistent SortedMap
-- extensive management capability including command-line and GUI tools
-
-This chapter briefly and informally introduces and demonstrates various Persistit features through examples. Subsequent chapters and the Javadoc API documentation provides a detailed reference guide to the product.
-
-== Download and Install
-
-Download +akiban-persistit-3.xx.yy.zip+ from http://www.akiban.com/persistit/download.html.
-
-Unpack the distribution kit into a convenient directory using any unzip utility. For example, use +jar+ to unpack the distribution kit to the current working directory as follows:
-----
-jar xvzf akiban-persistit-core-3.xx.yy.zip
-----
-
-Review the +LICENSE.html+ and +README.html+ files located in the root of the installation directory. Persistit is licensed under the XXXXXX XXXXXXX License. By installing, copying or otherwise using the Software contained in the distribution kit, you agree to be bound by the terms of the license agreement. If you do not agree to these terms, remove and destroy all copies of the software in your possession immediately.
-
-== Working with Persistit
-
-Add the +akiban-persistit-core-3.xx.yy.jar+ from the +lib+ directory of the distribution kit to your project's classpath. For example, copy it to +jre/lib/ext+ in your Java Runtime Environment, or add it to your classpath environment variable. 
-
-That's it. You are ready to work with Persistit.
-
-=== Examples
-
-Review the +examples+ directory. Here you will find functional examples of varying complexity.
-
-[horizontal]
-+examples/HelloWorld+:: source code for the example illustrating this chapter
-+examples/SimpleDemo+:: short, simple program similar to HelloWorld
-+examples/SimpleBench+:: a small micro-benchmark measuring the speed of insert, traversal, random updates, etc.
-+examples/SimpleTransaction+:: example demonstrating use of Persisit’s multi-version currency control (MVCC) transactions
-+examples/FindFile+:: a somewhat larger example that uses Persistit as the backing store for file finder utility
-+example/PersistitMapDemo+:: demonstration of the PersistitMap interface
-
-== HelloWorld
-
-Before going further let's honor tradition with a small program that stores, fetches and displays the phrase “Hello World.” In this program we will create a record with the key “Hello” and the value “World”.
-
-.HelloWorld.java
-[source,java]
-----
-import com.persistit.Exchange;
-import com.persistit.Key;
-import com.persistit.Persistit;
-
-public class HelloWorld {
-    public static void main(String[] args) throws Exception {
-        Persistit db = new Persistit();
-        try {
-            // Read configuration from persistit.properties, allocate
-            // buffers, open Volume, and perform recovery processing
-            // if necessary.
-            //
-            db.initialize();
-            //
-            // Create an Exchange, which is a thread-private facade for
-            // accessing data in a Persistit Tree. This Exchange will
-            // access a Tree called "greetings" in a Volume called
-            // "hwdemo". It will create a new Tree by that name
-            // if one does not already exist.
-            //
-            Exchange dbex = db.getExchange("hwdemo", "greetings", true);
-            //
-            // Set up the Value field of the Exchange.
-            //
-            dbex.getValue().put("World");
-            //
-            // Set up the Key field of the Exchange.
-            //
-            dbex.getKey().append("Hello");
-            //
-            // Ask Persistit to put this key/value pair into the Tree.
-            // Until this point, the changes to the Exchange are local
-            // to this thread.
-            //
-            dbex.store();
-            //
-            // Prepare to traverse all the keys in the Tree (of which there
-            // is currently only one!) and for each key display its value.
-            //
-            dbex.getKey().to(Key.BEFORE);
-            while (dbex.next()) {
-                System.out.println(dbex.getKey().indexTo(0).decode() + " "
-                        + dbex.getValue().get());
-            }
-            db.releaseExchange(dbex);
-        } finally {
-            // Always close Persistit. If the application does not do
-            // this, Persistit's background threads will keep the JVM from
-            // terminating.
-            //
-            db.close();
-        }
-    }
-}
-----
-
-== Concepts
-
-Although +HelloWorld.java+ is not very useful, it demonstrates several of the basic building blocks of the Persistit API.
-
-=== Initialization and Configuration
-
-Before accessing any data, +HelloWorld.java+ calls one of the +com.persistit.Persistit#initialize+ methods of +com.persistit.Persistit+. This sets up the memory configuration for buffers and the path names of Persistit volume and journal files. Alternative versions of the initialize method accept configuration information from a +java.util.Properties+ object, from a specified properties file, or by default from the file named +persistit.properties+.
-
-In this example, +persistit.properties+ looks like this:
-
-----
-datapath=.
-buffer.count.8192=32
-volume.1=${datapath}/hwdemo,create,pageSize:8192,\
-	initialPages:5,extensionPages:5,maximumPages:100000
-journalpath=${datapath}/hwdemo_journal
-----
-
-See <<Configuration>> for additional information about Persistit configuration properties.
-
-=== Volumes and Trees
-
-A configuration defines one or more volume files that will contain stored Persistit data. Usually you will specify the +create+ flag, which allows Persistit to create a new volume if the file does not already exist. Creating a new file also establishes the initial size and growth parameters for that volume.
-
-Each volume may contain an unlimited number of named trees. Each tree within a volume embodies a logically distinct B-Tree index structure. Think of a tree as simply a named key space within a volume.
-
-+HelloWorld.java+ stores its key/value pair in a tree called “greetings” in a volume named “hwdemo”. This is specified by constructing an Exchange.
-
-=== Exchanges
-
-The +com.persistit.Exchange+ class is the primary facade for interacting with Persistit data. It is so-named because it allows an application to exchange information with the database. An Exchange provides methods for storing, deleting, fetching and traversing key/value pairs.
-
-The method
-
-[source,java]
-----
-Exchange dbex = db.getExchange("hwdemo", "greetings", true);
-----
-
-in +HelloWorld.java+ finds a volume named "hwdemo" and attempts to find a tree in it named "greetings". If there is no such tree, +getExchange+ creates it.
-
-Methods +com.persistit.Persistit#getExchange+ and +com.persistit.Persistit#releaseExchange+ maintain a pool of reusable Exchange objects designed for use by multi-threaded applications such as web applications. If a suitable exchange already exists, +getExchange+ returns it; otherwise it constructs a new one.
-
-The Exchange looks up the volume name “hwdemo” by matching it against the volumes specified in the configuration. The match is based on the simple file name of the volume after removing its final dotted suffix.  For example, the volume name “hwdemo” matches the volume specification +${datapath}/hwdemo.v00+.
-
-Each Exchange is implicitly associated with a +com.persistit.Key+ and a +com.persistit.Value+. Typically you work with an Exchange in one of the following patterns:
-
-- Modify the Key, modify the Value and then perform a +com.persistit.Exchange#store+ operation.
-- Modify the Key, perform a +com.persistit.Exchange#fetch+ operation and then read the Value.
-- Modify the Key and then perform a +com.persistit.Exchange#remove+ operation.
-- Optionally modify the Key, perform a +com.persistit.Exchange#next+, +com.persistit.Exchange#previous+ or +com.persistit.Exchange#traverse+ operation, then read the resulting Key and/or Value.
-
-These methods and their variants provide the foundation for using Persistit.
-
-=== Records
-
-In Persistit, a database record consists of a Key and a Value. The terms “record” and “key/value pair” are used interchangeably.
-
-When you store a record, Persistit searches for a previously stored record having the same key. If there is such a record, Persistit replaces its value.  If there is no such record, Persistit inserts a new one.  Like a Java Map, Persistit stores at most one value per key, and every record in a Tree has a unique key value.
-
-=== Keys
-
-A Key contains a unique identifier for key/value pair - or record - in a tree. The identifier consists of a sequence of one or more Java values encoded into an array of bytes stored in the volume file.
-
-Key instances are mutable. Your application typically changes an Exchange's Key in preparation for fetching or retrieving data. In particular, you can append, remove or replace one or more values in a Key. Each value you append is called a _key segment_. You append multiple key segments to implement concatenated keys. See +com.persistit.Key+ for additional information on constructing keys and the ordering of key traversal within a tree.
-
-The +HelloWorld.java+ example appends “Hello” to the Exchange’s Key object in this line:
-
-[source,java]
-----
-            dbex.getKey().append("Hello");
-----
-
-The result is a key with a single key segment.
-
-=== Values
-
-A Value object represents the serialized state of a Java object or a primitive value. It is a staging area for data being transferred from or to the database by +fetch+, +traverse+ and +store+ operations.
-
-Value instances are mutable. The +fetch+ and +traverse+ operations modify the state of an Exchange's Value instance to represent the value associated with some Key. Your application executes methods to modify the state of the Value instance in preparation for storing new data values into the database.
-
-Numerous methods allow you to serialize and deserialize primitives values and objects into and from a Value object. For example, in +HelloWorld.java+, the statement
-
-[source,java]
-----
-            dbex.getValue().put("World");
-----
-serializes the string “World” into the backing byte array of the Exchange’s Value object and
-
-[source,java]
-----
-            	System.out.println(
-                	dbex.getKey().indexTo(0).decode() + " " +
-                	dbex.getValue().get());
-----
-deserializes and prints an object value from the Key and another object value from the Value. Value also has methods such as +getInt+, +getLong+, +getByteArray+ to decode primitive and array values directly.
-
-=== Storing and Fetching Data
-
-Finally, it is these two methods in +HelloWorld.java+ that cause the Exchange object to share data with the B-Tree, making it persistent and potentially available to other threads:
-
-[source,java]
-----
-            dbex.store();
-            ...
-            while (dbex.next()) { ... }
-----
-
-=== Closing Persistit
-
-Persistit creates one or more background threads that lazily write data to the Volume files and perform other maintenance activities. Be sure to invoke the +com.persistit.Persistit.close+ method to allow these threads to finish their work and exit properly. The pattern illustrated in +HelloWorld.java+, using a _try/finally_ block to invoke +close+, is strongly recommended.
-
-The +close+ flushes all data from 
-
-
-== Additional Topics
-
-=== PersistitMap
-
-A particularly easy way to get started with Persistit is to use its built-in +com.persistit.PersistitMap+ implementation. PersistitMap implements the +java.util.SortedMap+ interface, so it can directly replace +java.util.TreeMap+ or other kinds of Map in existing Java code.
-
-See <<PersistitMap>>.
-
-=== KeyFilters
-
-A +com.persistit.KeyFilter+ can be supplied to restrict the results traversal operation in a convenient and  
-
-=== Transactions
-
-Persistit provides ACID Transaction support with multi-version concurrency control (MCC) and adjustable durability policy.
-
-See <<Transactions>>.
-
-=== Managing Persistit
-
-Persistit provides several mechanisms for managing Persistit operation within an application. These include
-
-- JMX MXBeans
-- The +com.persistit.Management+ object which provides programmatic access to many management operations
-- The +com.persistit.CLI+ object which provides a command-line interface for various management operations
-- The AdminUI tool which provides a graphical client interface for examining records and other resources
-- Logging interface design for easy embedding in host applications
-
-See <<Management>>.

=== added file 'doc/Management.rst'
--- doc/Management.rst	1970-01-01 00:00:00 +0000
+++ doc/Management.rst	2012-05-30 18:23:19 +0000
@@ -0,0 +1,446 @@
+.. _Management:
+
+Management
+==========
+
+Akiban Persistit provides three main avenues for measuring and managing its internal resources: an RMI interface, a JMX interface and a command-line interface capable of launching various utility tasks. 
+
+The RMI interface is primarily intended for the com.persistit.ui.AdminUI utility. AdminUI is a JFC/Swing program that runs on a device with graphical UI capabilities.  For example, in Linux and Unix it requires an XServer. Since production servers are usually headless it is often necessary to run AdminUI remotely, via its RMI interface. To do this, the Persistit configuration must specify either the ``rmiport`` or ``rmihost`` property so that it can start an RMI server.
+
+Suppose a Persistit-based application is running on a host named “somehost” and has specified the configuration property ``rmiport=1099`` in its configuration.  Then the AdminUI can be launched as follows to connect with it:
+
+.. code-block:: java
+
+  java -cp classpath  com.persistit.ui.AdminUI somehost:1099
+
+where classpath includes the Persistit ``com.persistit.ui`` package. 
+
+The JMX interface can be used by third-party management utilities, from applications such as ``jconsole`` and ``visualvm``, and from command-line JMX clients such as ``jmxterm``. To enable JMX access, the configuration must specify the property ``jmx=true``.  This causes Persistit to register several MBeans with the platform MBean server during initialization.
+
+MXBeans
+-------
+The following JMX MXBeans are available:
+
+  ``com.persistit:type=Persistit``
+      See ``com.persistit.mxbeans.ManagementMXBean``
+  ``com.persistit:type=Persistit,class=AlertMonitorMXBean``
+      Accumulates, logs and emits notifications about abnormal events such as IOExceptions and measurements outside of 
+      expected thresholds.
+  ``com.persistit:type=Persistit,class=CleanupManagerMXBean``
+      View current state of the Cleanup Manager. The Cleanup Manager performs background pruning and tree maintenance 
+      activities.
+  ``com.persistit:type=Persistit,class=IOMeter``
+      Maintains statistics on file system I/O operations.
+  ``com.persistit.type=Persistit,class=JournalManager``
+      Views current journal status.
+  ``com.persistit.type=Persistit,class=RecoveryManager``
+      Views current status of the recovery process. Attributes of this MXBean change only during the recovery process.
+  ``com.persistit:type=Persistit,class=TransactionIndexMXBean``
+      View internal state of transaction index queues and tables.
+  ``com.persistit.type=Persistit,class=BufferPool.*SSSS*``
+      where *SSSS* is a buffer size (512, 1024, 2048, 4096 or 16394). View utilization statistics for buffers of the 
+      selected size.
+
+
+For details see the JavaDoc API documentation for each MXBean interface.
+
+Management Tasks
+----------------
+
+Persistit provides several ways to launch and administer ``com.persistit.Task`` instances.  A ``Task`` is a management operation that may take a significant amount of time and usually runs in a separate thread. For example, ``com.persistit.IntegrityCheck`` is a ``Task`` that verifies the internal structural integrity of one or more trees and can run for minutes to hours, depending on the size of the database.  The :ref:`AdminUI` tool, ``com.persistit.mxbeans.ManagementMXBean`` and the command-line interface (:ref:`CLI`) provide mechanisms to launch, suspend or stop a task, and to monitor a task’s progress.
+
+Currently the following built-in Tasks are available:
+
+  ``icheck``
+      Check the integrity of one or more trees or volumes.
+  ``save``
+      Save selected key-value pairs from one or more trees to a flat file.
+  ``load``
+      Load selected key-value pairs from a flat file written by ``save``.
+  ``backup``
+      Control and/or perform a concurrent backup of one more more volumes.
+  ``stat``
+      Aggregate various performance statistics and either return them immediately, or write them periodically to a file.
+  ``task``
+      Check the status of an existing task.  This task can also suspend, resume or stop an existing task. This task, which 
+      immediately returns status information, can be used by external tools to poll the status of other tasks.
+  ``cliserver``
+      Start a simple command-line server on a specified port.  This enables a client program to execute commands sending 
+      them directly to that port.
+  *other tasks*
+      Various commands allow you to select and view pages and journal records.
+
+
+Executing a Task from an Application
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``com.persistit.mxbeans.ManagementMXBean#execute`` and ``com.persistit.mxbeans.ManagementMXBean#launch`` methods both take a single String-valued argument, parse it to set up a ``Task`` and return a String-valued result. For example:
+
+.. code-block:: java
+
+  String taskId = db.getManagement().launch(“backup -z file=/tmp/mybackup.zip”);
+  String status = db.getManagement().execute(“task -v -m -c taskId=” + taskId);
+
+launches the backup task and then queries its status.
+
+Executing a Task from a JMX Client
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``com.persistit.mxbeans.ManagementMXBean#execute`` and ``com.persistit.mxbeans.ManagementMXBean#launch`` methods are exposed as operations on the ``com.persistit.mxbeans.ManagementMXBean``.  You can invoke tasks
+
+- via ``jconsole`` by typing the desired command line as the argument of the ``execute`` operation.
+- via a third-party JMX client such as ``jmxterm``.
+- via the ``cliserver`` feature
+
+Executing a Task Using a Third-Party JMX client
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can use the ``jmxterm`` program, for example, (see [http://www.cyclopsgroup.org/projects/jmxterm]) to execute commands with the following shell script::
+
+  #!/bin/sh
+  java -jar jmxterm-1.0-alpha-4-uber.jar --verbose silent --noninteract --url $1 <<EOF
+  run -d com.persistit -b com.persistit:type=Persistit execute $2
+  EOF
+
+To use this script, specify either the JMX URL or the process ID as the first command argument, and the command line as the second argument.  Example::
+
+  peter:~/workspace/sandbox$ jmxterm-execute 1234 ‘stat\ -a’
+  hit=3942334 miss=14 new=7364 evict=0 jwrite=81810 jread=2 jcopy=63848 tcommit=0 troll=0 CC=0 RV=12 RJ=2 WJ=81810 EV=0 FJ=529 IOkbytes=1134487 TOTAL
+
+This command invokes the ``stat`` task with the flag ``-a`` on a JVM running with process id 1234.  Note that with jxmterm white-space must be quoted by backslash (‘\’) even though the argument list is also enclosed in single-quotes.  The backslash marshals the space character through ``jmxterm``’s parser. Commas and other delimiters also need to be quoted.
+
+.. _cliserver:
+
+Executing a Task Using the Built-In ``cliserver``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``cliserver`` is a simple text-based server that receives a command line as a text string and emits the generated output as its response. To start it, enter the command::
+
+  cliserver port=9999
+
+programmatically or through JMX. (You may specify any valid, available port.) Then use a command-line client to send command lines to that port and display their results. Persistit includes a primitive command-line client within the ``com.persistit.CLI`` class itself.  Create a script to invoke it as follows::
+
+  #!/bin/sh
+  java -cp classpath com.persistit.CLI localhost:9999 $*
+
+Where ``classpath`` includes the Persistit library. Assuming the name of the script is ``pcli`` you can then invoke commands from a shell as shown in this example::
+
+  /home/akiban:~$ pcli icheck -v -c "trees=*:Acc*"
+  Volume,Tree,Faults,IndexPages,IndexBytes,DataPages,DataBytes,LongRecordPages,LongRecordBytes,MvvPages,MvvRecords,MvvOverhead,MvvAntiValues,IndexHoles,PrunedPages
+  "persistit","AccumulatorRecoveryTest",0,3,24296,1519,15560788,0,0,1506,52192,721521,2397,0,0
+  "*","*",0,3,24296,1519,15560788,0,0,1506,52192,721521,2397,0,0
+  /home/akiban:~$
+
+Alternatively, you can use ``curl`` as follows::
+
+  #!/bin/sh
+  echo "$*" | curl --silent --show-error telnet://localhost:9999
+
+to issue commands.
+
+.. caution::
+   
+   ``cliserver`` has no access control and sends potentially sensitive data in cleartext form. Therefore it should be used with care and only in a secure 
+   network environment. Its primary mission is to allow easy inspection of internal data structures within Persistit.
+
+.. _CLI:
+
+The Command-Line Interface
+--------------------------
+
+The String value passed to the ``execute`` and ``launch`` operations specifies the name of a task and its arguments. The general form is::
+
+  commandname -flag -flag argname=value argname=value
+
+where the order of arguments and flags is not significant.
+
+
+Command: ``icheck``
+^^^^^^^^^^^^^^^^^^^
+
+Performs a com.persistit.IntegrityCheck task. Arguments:
+
+  ``trees``
+      Specifies volumes and/or trees to check. See com.persistit.TreeSelector for details syntax. Default is all trees in all volumes.
+  ``-r``
+      Tree specification uses Java RegEx syntax (Default is to treat ‘*’ and ‘?’ as standard single-character and multi-character wildcards.
+  ``-u``
+      Don't freeze updates (Default is to freeze updates)
+  ``-h``
+      Fix index holes. An *index hole* is an anomaly that occurs rarely in normal operation such that a page does not have an index entry in the index page level 
+      immediately above it
+  ``-p``
+      Prune obsolete MVV (multi-version value) instances while checking.
+  ``-P``
+      Prune obsolete MVV instances, and clear any remaining aborted TransactionStatus instances.  Use with care.
+  ``-v``
+      Emit verbose output. For example, emit statistics for each tree.
+  ``-c``
+      Display tree statistics in comma-separated-variable format suitable for import into a spreadsheet program.
+
+Example::
+
+  icheck trees=vehicles/* -h
+
+Checks all trees in the ``vehicles`` volume and repairs index holes.
+
+Command: ``save``
+^^^^^^^^^^^^^^^^^
+
+Starts a com.persistit.StreamSaver task. Arguments:
+
+  ``file``
+      Name of file to save records to (required)
+  ``trees``
+      Specifies volumes and/or trees to save. See com.persistit.TreeSelector for details syntax. Default is all trees in all volumes.
+  ``-r``
+      Tree specification uses Java RegEx syntax (Default is to treat ‘*’ and ‘?’ as standard single-character and multi-character wildcards.)
+  ``-v``
+      emit verbose output
+  
+...‘*’ and ‘?’ are standard wildcards.
+
+Example::
+
+  save -v file=/home/akiban/save.dat trees=vehicles/*{[“Edsel”:”Yugo”]}
+
+Saves the records for “Edsel” through “Yugo”, inclusive, from any tree in the volume named ``vehicles``. See com.persistit.TreeSelector for selection syntax details.
+
+Command: ``load``
+^^^^^^^^^^^^^^^^^
+
+Starts a com.persistit.StreamLoader task. Arguments:
+
+  ``file``
+      Name of file to load records from
+  ``trees``
+      Specifies volumes and/or trees to load. See com.persistit.TreeSelector for details syntax. Default is all trees in all volumes.
+  ``-r``
+      Tree specification uses Java RegEx syntax (Default is to treat ‘*’ and ‘?’ as standard single-character and multi-character wildcards.)
+  ``-n``
+      Don't create missing Volumes (Default is to create them)
+  ``-t``
+      Don't create missing Trees (Default is to create them)
+  ``-v``
+      Emit verbose output
+
+...‘*’ and ‘?’ are standard wildcards.
+
+Example::
+
+  load file=/home/akiban/save.dat trees=*/*{[“Falcon”:”Firebird”]}
+
+For any tree in any volume, this command loads all records having keys between “Falcon” and “Firebird”, inclusive.
+
+Command: ``backup``
+^^^^^^^^^^^^^^^^^^^
+
+Starts a ``com.persistit.BackupTask`` task to perform concurrent (hot) backup. Arguments:
+
+  ``file``
+      Archive file path. If this argument is specified, BackupTask will back up the database in .zip format to the specified file.  This is intended only for small 
+      databases. It is expected that ``backup`` will be used in conjunction with high-speed third-party data copying utilities for production use. The ``-a`` and       
+  ``-e`` 
+      flags are incompatible with operation when the ``file`` argument is specified and are ignored.
+  ``-a``
+      Start appendOnly mode - for use with third-party backup tools.  ``backup -a`` should be invoked before data copying begins.
+  ``-e``
+      End appendOnly mode - for use with third-party backup tools.  ``backup -e`` should be invoked after data copying ends.
+  ``-c``
+      Request checkpoint before backup.
+  ``-z``
+      Compress output to ZIP format - meaningful only in conjunction with the ``file`` argument.
+  ``-f``
+      Emit a list of files that need to be copied. In this form the task immediately returns with a list of files currently comprising the Persistit database,  
+      including Volume and journal files.
+  ``-y``
+      Copy pages from journal to Volumes before starting backup.  This reduces the number of journal files in the backup set.
+
+Example::
+
+    backup -y -a -c -y -f
+    … invoke third-party backup tool to copy the database files
+    backup -e
+
+Uses the ``backup`` task twice, once to set *append-only* mode, checkpoint the journal and perform a full copy-back cycle (a process that attempts to shorten the journal), and then write out a list of files that need to be copied. The second call to ``backup`` restores normal operation.  Between these two calls a third party backup tool is used to copy the data.
+
+Example::
+
+    backup -z file=/tmp/my_backup.zip
+
+Uses the built-in file copy feature with ZIP compression.
+
+Command: ``task``
+^^^^^^^^^^^^^^^^^
+
+Queries, stops, suspends or resumes a background task.  Arguments:
+
+  ``taskId``
+      Task ID to to check, or -1 for all
+  ``-v``
+      Verbose - returns detailed status messages from the selected task(s)
+  ``-m``
+      Keep previously delivered messages. Default is to remove messages once reported.
+  ``-k``
+      Keep the selected task or tasks even if completed.  Default is to remove tasks once reported.
+  ``-x``
+      Stop the selected task or tasks
+  ``-u``
+      Suspend the selected task or tasks
+  ``-r``
+      Resume the selected task or tasks
+
+Unlike other commands, the ``task`` command always runs immediately even if invoked through the ``launch`` method. 
+
+You can use the ``task`` command to poll and display progress of long-running tasks. Invoke::
+
+  task  -v -m -c taskId=nnn
+
+until the result is empty.
+
+Command: ``cliserver``
+^^^^^^^^^^^^^^^^^^^^^^
+
+Starts a simple text-based server that receives a command line as a text string and emits the generated output as its response. Argument:
+
+  ``port``
+      Port number on which to listen for commands.
+
+Command: ``exit``
+^^^^^^^^^^^^^^^^^
+
+Ends a running ``cliserver`` instance.
+
+Commands for Viewing Data
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The following commands execute immediately, even if invoked through the ``launch`` method.  They provide a mechanism to examine individual database pages or journal records.
+
+Command: ``select``
+^^^^^^^^^^^^^^^^^^^
+
+Selects a volume and optionally a tree for subsequent operations such as ``view``. Arguments:
+
+  ``tree``
+      Specifies volume and/or tree to select as context for subsequent operations. See com.persistit.TreeSelector for details syntax.
+  ``-r``
+      Tree specification uses Java RegEx syntax (Default is to treat ‘*’ and ‘?’ as standard single-character and multi-character wildcards.)
+
+Command: ``list``
+^^^^^^^^^^^^^^^^^
+
+Lists volumes and trees.  Arguments:
+
+  ``trees``
+      Specifies volumes and/or trees to list. See com.persistit.TreeSelector for details syntax. Default is all trees in all volumes.
+  ``-r``
+      Tree specification uses Java RegEx syntax (Default is to treat ‘*’ and ‘?’ as standard single-character and multi-character wildcards.
+
+All volumes, and all trees within those volumes, that match the ``trees`` specification are listed. By default, this command lists all trees in all volumes.
+
+Command: ``pview``
+^^^^^^^^^^^^^^^^^^
+
+Displays contents of a database page. Arguments:
+
+  ``page``
+      page address
+  ``jaddr``
+      journal address - displays a page version stored at the specified journal address
+  ``key``
+      a key specified as a String defined in the com.persistit.Key class
+  ``level``
+      tree level of the desired page
+  ``find``
+      selected records in an index page surrounding a key that points to the specified page address
+  ``-a``
+      all records. If specified, all records in the page will be displayed.  Otherwise the output is abbreviated to no more than 20 lines.
+  ``-s``
+      summary - only header information in the page is displayed
+
+The ``pview`` command identifies a page in one of three distinct ways: by page address, by journal address, or by key.  Only one of the three parameters ``page``, ``jaddr`` or ``key`` (with ``level``) may be used.
+
+``page`` specifies the current version of a page having the specified address.  If there is a copy of the page in the buffer pool, that copy is displayed even if it contains updates that are not yet written to disk.
+
+``jaddr`` specifies an address in the journal. Typical use is to invoke the ``jview`` command to view a list of journal records, and then to see a detailed view of one page record in the journal, invoke the ``pview`` command with its journal address.
+
+``key`` specifies a key. By default the data page associated with that key will be displayed.  The data page is defined as level 0. The ``level`` parameter allows pages at various index levels to be viewed; for example ``level=1`` refers to the index page that points to the data page containing the specified key.
+
+When examining an index page with potentially hundreds of records it is sometimes convenient to find the record that points to a particular child page, and also the records immediately before and after. Specifying the ``find`` parameter when viewing an index page abbreviates the displayed records to include just the first and last records in the page, plus a small range of records surrounding the one that points to the specified page. This mechanism provides a convenient way to find sibling pages.
+
+
+Command: ``path``
+^^^^^^^^^^^^^^^^^
+
+For a specified key displays the sequence of pages from root of the tree to the data page containing they key. Argument:
+
+  ``key``
+      a key specified as a String defined in the com.persistit.Key class
+
+
+Command: ``jview``
+^^^^^^^^^^^^^^^^^^
+
+Displays journal records.  Arguments:
+
+  ``start``
+      starting journal address (default = 0)
+  ``end``
+      end journal address (address = infinite)
+  ``timestamps``
+      range selection of timestamp values, e.g., “132466-132499” for records having timestamps between these two numbers, inclusive. Default is all timestamps.
+  ``types``
+      comma-delimted list of two-character record types, e.g., “JH,IV,IT,CP” to select only Journal Header, Identify Volume, Identify Tree and Check Point records 
+      (see ``com.persistit.JournalRecord`` for definitions of all types.) Default value is all types.
+  ``pages``
+      range selection of page address for PA (Page) records, e.g., “1,2,13-16” to include pages, 1, 2, 13, 14, 15 or 16.
+  ``maxkey``
+      maximum display length of key values in the output. Default value is 42.
+  ``maxvalue``
+      maximum display length of values in the output. Default value is 42.
+  ``path``
+      journal file path. Default is the journal file path of the currently instantiated Persistit instance.
+  ``-v``
+      verbose format. If specified, causes PM (Page Map) and TM (TransactionMap) records to be be display all map elements.
+
+
+Note that the journal on a busy system contains a large number of records, so entering the ``journal`` command without constraining the address range or record types may result in extremely lengthy output.
+
+Command: ``open``
+^^^^^^^^^^^^^^^^^
+
+Opens a Persistit database for analysis. This task can only be used to examine a copy of a Persistit database that is not currently in use by an application. It works by attempting to open the volume and journal files using a synthesized configuration. It finds a collection of journal files and volume files specified by the ``datapath``, ``journalpath`` and ``volumepath`` arguments; from these it derives a set of properties that will allow it to examine those journals and volumes. By default all volumes are opened in read-only mode and cannot be changed by operations executed from the command-line interface.
+
+If there already is an open Persistit instance, this command detaches it. For example, if you start ``cliserver`` from a live Persistit instance and then issue the ``open`` command, the live instance will continue to operate but ``cliserver`` will no longer be attached to it.
+
+Note that you cannot ``open`` volumes that are already open in a running Persistit instance due to their file locks. However, you can copy open volumes and journal files to another location and ``open`` the copy. This is the primary use case for the ``open`` command: to analyze a copy of a database (for example a copy recovered from backup) without having to a launch the application software that embeds Persistit.
+
+Arguments:
+
+  ``datapath``
+      a directory path for volume and journal files to be analyzed
+  ``volumepath``
+      overrides ``datapath`` to specify an alternative location for volume files.
+  ``journalpath``
+      overrides ``datapath`` to specify an alternative location for journal files.
+  ``rmiport``
+      specifies an RMI port to which an instance of the AdminUI can attach.
+  ``-g``
+      launch a local copy of AdminUI
+  ``-y``
+      attempt to recover committed transactions .
+
+Note that even if you specify ``-y`` to recover transactions, the volume files will not be modified. But the ``open`` command will add a new journal file containing modifications caused by the recovery process. You can simply delete that file when done.
+
+Command: ``close``
+^^^^^^^^^^^^^^^^^^
+
+Detach and close the current Persistit instance. If the CLI was started with a live Persistit instance then this command merely detaches from it; if the instance was created with the ``open`` command then ``close`` closes it and releases all related file locks, buffers, etc.
+
+Command: ``source``
+^^^^^^^^^^^^^^^^^^^
+
+Execute command lines from a specified text file. Argument:
+
+  ``file``
+      file name of command input file

=== removed file 'doc/Management.txt'
--- doc/Management.txt	2012-04-30 22:09:31 +0000
+++ doc/Management.txt	1970-01-01 00:00:00 +0000
@@ -1,360 +0,0 @@
-[[Management]]
-= Management
-
-Akiban Persistit provides three main avenues for measuring and managing its internal resources: an RMI interface, a JMX interface and a command-line interface capable of launching various utility tasks. 
-
-The RMI interface is primarily intended for the com.persistit.ui.AdminUI utility. AdminUI is a JFC/Swing program that runs on a device with graphical UI capabilities.  For example, in Linux and Unix it requires an XServer. Since production servers are usually headless it is often necessary to run AdminUI remotely, via its RMI interface. To do this, the Persistit configuration must specify either the +rmiport+ or +rmihost+ property so that it can start an RMI server.
-
-Suppose a Persistit-based application is running on a host named “somehost” and has specified the configuration property +rmiport=1099+ in its configuration.  Then the AdminUI can be launched as follows to connect with it:
-
-----
-java -cp <classpath>  com.persistit.ui.AdminUI somehost:1099
-----
-
-where <classpath> includes the Persistit com.persistit.ui package. 
-
-The JMX interface can be used by third-party management utilities, from applications such as +jconsole+ and +visualvm+, and from command-line JMX clients such as +jmxterm+. To enable JMX access, the configuration must specify the property +jmx=true+.  This causes Persistit to register several MBeans with the platform MBean server during initialization.
-
-== MXBeans
-
-The following JMX MXBeans are available:
-
-[horizontal]
-+com.persistit:type=Persistit+:: See +com.persistit.mxbeans.ManagementMXBean+
-+com.persistit:type=Persistit,class=AlertMonitorMXBean+:: Accumulates, logs and emits notifications about abnormal events such as IOExceptions and measurements outside of expected thresholds.
-+com.persistit:type=Persistit,class=CleanupManagerMXBean+:: View current state of the Cleanup Manager. The Cleanup Manager performs background pruning and tree maintenance activities.
-+com.persistit:type=Persistit,class=IOMeter+:: Maintains statistics on file system I/O operations.
-+com.persistit.type=Persistit,class=JournalManager+:: Views current journal status.
-+com.persistit.type=Persistit,class=RecoveryManager+:: Views current status of the recovery process. Attributes of this MXBean change only during the recovery process.
-+com.persistit:type=Persistit,class=TransactionIndexMXBean+:: View internal state of transaction index queues and tables.
-+com.persistit.type=Persistit,class=BufferPool._SSSS_+:: where _SSSS_ is a buffer size (512, 1024, 2048, 4096 or 16394). View utilization statistics for buffers of the selected size.
-
-
-For details see the JavaDoc API documentation for each MXBean interface.
-
-== Management Tasks
-
-Persistit provides several ways to launch and administer +com.persistit.Task+ instances.  A +Task+ is a management operation that may take a significant amount of time and usually runs in a separate thread. For example, +com.persistit.IntegrityCheck+ is a +Task+ that verifies the internal structural integrity of one or more trees and can run for minutes to hours, depending on the size of the database.  The <<AdminUI>> tool, +com.persistit.ManagementMXBean+ and the command-line interface (<<CLI>>) provide mechanisms to launch, suspend or stop a task, and to monitor a task’s progress.
-
-Currently the following built-in Tasks are available:
-
-[horizontal]
-+icheck+:: Check the integrity of one or more trees or volumes.
-+save+:: Save selected key-value pairs from one or more trees to a flat file.
-+load+:: Load selected key-value pairs from a flat file written by +save+.
-+backup+:: Control and/or perform a concurrent backup of one more more volumes.
-+stat+:: Aggregate various performance statistics and either return them immediately, or write them periodically to a file.
-+task+:: Check the status of an existing task.  This task can also suspend, resume or stop an existing task. This task, which immediately returns status information, can be used by external tools to poll the status of other tasks.
-+cliserver+:: Start a simple command-line server on a specified port.  This enables a client program to execute commands sending them directly to that port.
-+_other tasks_+:: Various commands allow you to select and view pages and journal records.
-
-
-=== Executing a Task from an Application
-
-The +com.persistit.mxbeans.ManagementMXBean#execute+ and +com.persistit.mxbeans.ManagementMXBean#launch+ methods both take a single String-valued argument, parse it to set up a +Task+ and return a String-valued result. For example:
-
-[source,java]
-----
-String taskId = db.getManagement().launch(“backup -z file=/tmp/mybackup.zip”);
-String status = db.getManagement().execute(“task -v -m -c taskId=” + taskId);
-----
-
-launches the backup task and then queries its status.
-
-=== Executing a Task from a JMX Client
-
-The +com.persistit.mxbeans.ManagementMXBean#execute+ and +com.persistit.mxbeans.ManagementMXBean#launch+ methods are exposed as operations on the +com.persistit.mxbeans.ManagementMXBean+.  You can invoke tasks
-
-- via +jconsole+ by typing the desired command line as the argument of the +execute+ operation.
-- via a third-party JMX client such as +jmxterm+.
-- via the +cliserver+ feature
-
-==== Executing a Task Using a Third-Party JMX client.
-
-You can use the +jmxterm+ program, for example, (see [http://www.cyclopsgroup.org/projects/jmxterm]) to execute commands with the following shell script:
-
-[source,bash]
-----
-#!/bin/sh
-java -jar jmxterm-1.0-alpha-4-uber.jar --verbose silent --noninteract --url $1 <<EOF
-run -d com.persistit -b com.persistit:type=Persistit execute $2
-EOF
-----
-
-To use this script, specify either the JMX URL or the process ID as the first command argument, and the command line as the second argument.  Example
-
-----
-peter:~/workspace/sandbox$ jmxterm-execute 1234 ‘stat\ -a’
-hit=3942334 miss=14 new=7364 evict=0 jwrite=81810 jread=2 jcopy=63848 tcommit=0 troll=0 CC=0 RV=12 RJ=2 WJ=81810 EV=0 FJ=529 IOkbytes=1134487 TOTAL
-----
-
-This command invokes the +stat+ task with the flag +-a+ on a JVM running with process id 1234.  Note that with jxmterm white-space must be quoted by backslash (‘\’) even though the argument list is also enclosed in single-quotes.  The backslash marshals the space character through +jmxterm+’s parser. Commas and other delimiters also need to be quoted.
-
-[[cliserver]]
-=== Executing a Task Using the Built-In +cliserver+
-
-+cliserver+ is a simple text-based server that receives a command line as a text string and emits the generated output as its response. To start it, enter the command
-----
-cliserver port=9999
-----
-programmatically or through JMX. (You may specify any valid, available port.) Then use a command-line client to send command lines to that port and display their results. Persistit includes a primitive command-line client within the +com.persistit.CLI+ class itself.  Create a script to invoke it as follows:
-
-[source,bash]
-----
-#!/bin/sh
-java -cp <classpath> com.persistit.CLI localhost:9999 $*
-----
-
-Where +<classpath>+ includes the Persistit library. Assuming the name of the script is +pcli+ you can then invoke commands from a shell as shown in this example:
-
-----
-/home/akiban:~$ pcli icheck -v -c "trees=*:Acc*"
-Volume,Tree,Faults,IndexPages,IndexBytes,DataPages,DataBytes,LongRecordPages,LongRecordBytes,MvvPages,MvvRecords,MvvOverhead,MvvAntiValues,IndexHoles,PrunedPages
-"persistit","AccumulatorRecoveryTest",0,3,24296,1519,15560788,0,0,1506,52192,721521,2397,0,0
-"*","*",0,3,24296,1519,15560788,0,0,1506,52192,721521,2397,0,0
-/home/akiban:~$
-----
-
-Alternatively, you can use +curl+ as follows:
-
-[source,bash]
-----
-#!/bin/sh
-echo "$*" | curl --silent --show-error telnet://localhost:9999
-----
-to issue commands.
-
-CAUTION: Warning: +cliserver+ has no access control and sends potentially sensitive data in cleartext form. Therefore it should be used with care and only in a secure network environment. Its primary mission is to allow easy inspection of internal data structures within Persistit.
-
-[[CLI]]
-== The Command-Line Interface
-
-The String value passed to the +execute+ and +launch+ operations specifies the name of a task and its arguments. The general form is
-
-----
-commandname -flag -flag argname=value argname=value
-----
-
-where the order of arguments and flags is not significant.
-
-
-=== Command: +icheck+
-
-Performs a com.persistit.IntegrityCheck task. Arguments:
-
-[horizontal]
-+trees+:: Specifies volumes and/or trees to check. See com.persistit.TreeSelector for details syntax. Default is all trees in all volumes.
-+-r+:: Tree specification uses Java RegEx syntax (Default is to treat ‘*’ and ‘?’ as standard single-character and multi-character wildcards.
-+-u+:: Don't freeze updates (Default is to freeze updates)
-+-h+:: Fix index holes. An _index hole_ is an anomaly that occurs rarely in normal operation such that a page does not have an index entry in the index page level immediately above it
-+-p+:: Prune obsolete MVV (multi-version value) instances while checking.
-+-P+:: Prune obsolete MVV instances, and clear any remaining aborted TransactionStatus instances.  Use with care.
-+-v+:: Emit verbose output. For example, emit statistics for each tree.
-+-c+:: Display tree statistics in comma-separated-variable format suitable for import into a spreadsheet program.
-
-Example:
-----
-icheck trees=vehicles/* -h
-----
-Checks all trees in the +vehicles+ volume and repairs index holes.
-
-=== Command: +save+
-
-Starts a com.persistit.StreamSaver task. Arguments:
-
-[horizontal]
-+file+:: Name of file to save records to (required)
-+trees+:: Specifies volumes and/or trees to save. See com.persistit.TreeSelector for details syntax. Default is all trees in all volumes.
-+-r+:: Tree specification uses Java RegEx syntax (Default is to treat ‘*’ and ‘?’ as standard single-character and multi-character wildcards.)
-+-v+:: emit verbose output
-‘*’ and ‘?’ as standard wildcards.)
-
-Example:
-----
-save -v file=/home/akiban/save.dat trees=vehicles/*{[“Edsel”:”Yugo”]}
-----
-
-Saves the records for “Edsel” through “Yugo”, inclusive, from any tree in the volume named +vehicles+. See com.persistit.TreeSelector for selection syntax details.
-
-=== Command: +load+
-
-Starts a com.persistit.StreamLoader task. Arguments:
-
-[horizontal]
-+file+:: Name of file to load records from
-+trees+:: Specifies volumes and/or trees to load. See com.persistit.TreeSelector for details syntax. Default is all trees in all volumes.
-+-r+:: Tree specification uses Java RegEx syntax (Default is to treat ‘*’ and ‘?’ as standard single-character and multi-character wildcards.)
-+-n+:: Don't create missing Volumes (Default is to create them)
-+-t+:: Don't create missing Trees (Default is to create them)
-‘*’ and ‘?’ as standard wildcards.)
-+-v+:: emit verbose output
-
-Example:
-----
-load file=/home/akiban/save.dat trees=*/*{[“Falcon”:”Firebird”]}
-----
-
-For any tree in any volume, this command loads all records having keys between “Falcon” and “Firebird”, inclusive.
-
-=== Command: +backup+
-
-Starts a +com.persistit.BackupTask+ task to perform concurrent (hot) backup. Arguments:
-
-[horizontal]
-+file+:: Archive file path. If this argument is specified, BackupTask will back up the database in .zip format to the specified file.  This is intended only for small databases. It is expected that +backup+ will be used in conjunction with high-speed third-party data copying utilities for production use. The +-a+ and +-e+ flags are incompatible with operation when the +file+ argument is specified and are ignored.
-+-a+:: Start appendOnly mode - for use with third-party backup tools.  +backup -a+ should be invoked before data copying begins.
-+-e+:: End appendOnly mode - for use with third-party backup tools.  +backup -e+ should be invoked after data copying ends.
-+-c+:: Request checkpoint before backup.
-+-z+:: Compress output to ZIP format - meaningful only in conjunction with the +file+ argument.
-+-f+:: Emit a list of files that need to be copied. In this form the task immediately returns with a list of files currently comprising the Persistit database, including Volume and journal files.
-+-y+:: Copy pages from journal to Volumes before starting backup.  This reduces the number of journal files in the backup set.
-
-Examples:
-----
-backup -y -a -c -y -f
-… invoke third-party backup tool to copy the database files
-backup -e
-----
-Uses the +backup+ task twice, once to set _append-only_ mode, checkpoint the journal and perform a full copy-back cycle (a process that attempts to shorten the journal), and then write out a list of files that need to be copied. The second call to +backup+ restores normal operation.  Between these two calls a third party backup tool is used to copy the data.
-
-----
-backup -z file=/tmp/my_backup.zip
-----
-Uses the built-in file copy feature with ZIP compression.
-
-=== Command: +task+
-
-Queries, stops, suspends or resumes a background task.  Arguments:
-
-[horizontal]
-+taskId+:: Task ID to to check, or -1 for all
-+-v+:: Verbose - returns detailed status messages from the selected task(s)
-+-m+:: Keep previously delivered messages. Default is to remove messages once reported.
-+-k+:: Keep the selected task or tasks even if completed.  Default is to remove tasks once reported.
-+-x+:: Stop the selected task or tasks
-+-u+:: Suspend the selected task or tasks
-+-r+:: Resume the selected task or tasks
-
-Unlike other commands, the +task+ command always runs immediately even if invoked through the +launch+ method. 
-
-You can use the +task+ command to poll and display progress of long-running tasks. Invoke
-
-----
-task  -v -m -c taskId=nnn
-----
-
-until the result is empty.
-
-=== Command: +cliserver+
-
-Starts a simple text-based server that receives a command line as a text string and emits the generated output as its response. Argument:
-
-[horizontal]
-+port+:: Port number on which to listen for commands.
-
-=== Command: +exit+
-
-Ends a running +cliserver+ instance.
-
-== Commands for Viewing Data
-
-The following commands execute immediately, even if invoked through the +launch+ method.  They provide a mechanism to examine individual database pages or journal records.
-
-=== Command: +select+
-
-Selects a volume and optionally a tree for subsequent operations such as +view+. Arguments:
-
-[horizontal]
-+tree+:: Specifies volume and/or tree to select as context for subsequent operations. See com.persistit.TreeSelector for details syntax.
-+-r+:: Tree specification uses Java RegEx syntax (Default is to treat ‘*’ and ‘?’ as standard single-character and multi-character wildcards.)
-
-=== Command: +list+
-
-Lists volumes and trees.  Arguments:
-
-[horizontal]
-+trees+:: Specifies volumes and/or trees to list. See com.persistit.TreeSelector for details syntax. Default is all trees in all volumes.
-+-r+:: Tree specification uses Java RegEx syntax (Default is to treat ‘*’ and ‘?’ as standard single-character and multi-character wildcards.
-
-All volumes, and all trees within those volumes, that match the +trees+ specification are listed. By default, this command lists all trees in all volumes.
-
-=== Command: +pview+
-
-Displays contents of a database page. Arguments:
-
-[horizontal]
-+page+:: page address
-+jaddr+:: journal address - displays a page version stored at the specified journal address
-+key+:: a key specified as a String defined in the com.persistit.Key class
-+level+:: tree level of the desired page
-+find+:: selected records in an index page surrounding a key that points to the specified page address
-+-a+:: all records. If specified, all records in the page will be displayed.  Otherwise the output is abbreviated to no more than 20 lines.
-+-s+:: summary - only header information in the page is displayed
-
-The +pview+ command identifies a page in one of three distinct ways: by page address, by journal address, or by key.  Only one of the three parameters +page+, +jaddr+ or +key+ (with +level+) may be used.
-
-+page+ specifies the current version of a page having the specified address.  If there is a copy of the page in the buffer pool, that copy is displayed even if it contains updates that are not yet written to disk.
-
-+jaddr+ specifies an address in the journal. Typical use is to invoke the +jview+ command to view a list of journal records, and then to see a detailed view of one page record in the journal, invoke the +pview+ command with its journal address.
-
-+key+ specifies a key. By default the data page associated with that key will be displayed.  The data page is defined as level 0. The +level+ parameter allows pages at various index levels to be viewed; for example +level=1+ refers to the index page that points to the data page containing the specified key.
-
-When examining an index page with potentially hundreds of records it is sometimes convenient to find the record that points to a particular child page, and also the records immediately before and after. Specifying the +find+ parameter when viewing an index page abbreviates the displayed records to include just the first and last records in the page, plus a small range of records surrounding the one that points to the specified page. This mechanism provides a convenient way to find sibling pages.
-
-
-=== Command: +path+
-
-For a specified key displays the sequence of pages from root of the tree to the data page containing they key. Argument:
-
-[horizontal]
-+key+:: a key specified as a String defined in the com.persistit.Key class
-
-
-=== Command: +jview+
-
-Displays journal records.  Arguments:
-
-[horizontal]
-+start+:: starting journal address (default = 0)
-+end+:: end journal address (address = infinite)
-+timestamps+:: range selection of timestamp values, e.g., “132466-132499” for records having timestamps between these two numbers, inclusive. Default is all timestamps.
-+types+:: comma-delimted list of two-character record types, e.g., “JH,IV,IT,CP” to select only Journal Header, Identify Volume, Identify Tree and Check Point records (see com.persistit.JournalRecord for definitions of all types.) Default value is all types.
-+pages+:: range selection of page address for PA (Page) records, e.g., “1,2,13-16” to include pages, 1, 2, 13, 14, 15 or 16.
-+maxkey+:: maximum display length of key values in the output. Default value is 42.
-+maxvalue+:: maximum display length of values in the output. Default value is 42.
-+path+:: journal file path. Default is the journal file path of the currently instantiated Persistit instance.
-+-v+:: verbose format. If specified, causes PM (Page Map) and TM (TransactionMap) records to be be display all map elements.
-
-
-Note that the journal on a busy system contains a large number of records, so entering the +journal+ command without constraining the address range or record types may result in extremely lengthy output.
-
-=== Command: +open+
-
-Opens a Persistit database for analysis. This task can only be used to examine a copy of a Persistit database that is not currently in use by an application. It works by attempting to open the volume and journal files using a synthesized configuration. It finds a collection of journal files and volume files specified by the +datapath+, +journalpath+ and +volumepath+ arguments; from these it derives a set of properties that will allow it to examine those journals and volumes. By default all volumes are opened in read-only mode and cannot be changed by operations executed from the command-line interface.
-
-If there already is an open Persistit instance, this command detaches it. For example, if you start +cliserver+ from a live Persistit instance and then issue the +open+ command, the live instance will continue to operate but +cliserver+ will no longer be attached to it.
-
-Note that you cannot +open+ volumes that are already open in a running Persistit instance due to their file locks. However, you can copy open volumes and journal files to another location and +open+ the copy. This is the primary use case for the +open+ command: to analyze a copy of a database (for example a copy recovered from backup) without having to a launch the application software that embeds Persistit.
-
-Arguments:
-
-[horizontal]
-+datapath+:: a directory path for volume and journal files to be analyzed
-+volumepath+:: overrides +datapath+ to specify an alternative location for volume files.
-+journalpath+:: overrides +datapath+ to specify an alternative location for journal files.
-+rmiport+:: specifies an RMI port to which an instance of the AdminUI can attach.
-+-g+:: launch a local copy of AdminUI
-+-y+:: attempt to recover committed transactions .
-
-Note that even if you specify +-y+ to recover transactions, the volume files will not be modified. But the +open+ command will add a new journal file containing modifications caused by the recovery process. You can simply delete that file when done.
-
-=== Command: +close+
-
-Detach and close the current Persistit instance. If the CLI was started with a live Persistit instance then this command merely detaches from it; if the instance was created with the +open+ command then +close+ closes it and releases all related file locks, buffers, etc.
-
-=== Command: +source+
-
-Execute command lines from a specified text file. Argument:
-
-[horizontal]
-+file+:: file name of command input file

=== added file 'doc/Miscellaneous.rst'
--- doc/Miscellaneous.rst	1970-01-01 00:00:00 +0000
+++ doc/Miscellaneous.rst	2012-05-30 18:23:19 +0000
@@ -0,0 +1,36 @@
+.. _Miscellaneous:
+
+Miscellaneous Topics
+====================
+
+Following are some short items you may find useful as you explore Akiban Persistit. Follow links to the API documentation for more details.
+
+Histograms
+----------
+
+The method ``com.persistit.Exchange#computeHistogram`` class provides a way to sample and summarize a set of keys in a ``Tree``.  It works by traversing keys in index pages near the root of the tree.  Because only a small fraction of all the keys in the tree are represented in the index, this can result in relatively small sample set of keys relatively quickly. The result can be used to estimate the actual number of keys.
+
+Temporary Volumes
+-----------------
+
+A Persistit temporary volume is a special kind of Volume that is deleted when Persistit is closed. The update mechanism for temporary volumes avoids writing to disk whenever possible, and its contents are not recoverable Persistit shuts down. Therefore in some cases database operations on temporary volumes are faster.
+
+The primary use case for a temporary volume is an application that needs the unlimited size, but not the persistence of normal Persistit volumes.
+
+See the ``com.persistit.Persistit#createTemporaryVolume`` method for additional details.
+
+Logging
+-------
+
+By default Persistit emits log messages to a file called persistit.log  and also writes high level log messages to System.out.  You can change this behavior by plugging in a different logging implementation. In particular, Persistit provides pluggable adapters for various other logging implementations, including Log4J, SLF4J, and the Java logging API introduced in JDK 1.4. For details see the API documentation for com.persistit.logging.AbstractPersistitLogger.
+
+Using one of these logging frameworks is simple.  For example, the following code connects Persistit to an application-supplied SLF4J logger:
+
+.. code-block:: java
+
+  db.setPersistitLogger(new Slf4jAdapter(LOG))
+
+where ``db`` is the Persistit instance and ``LOG`` is a Logger supplied by SLF4J. This method should be called before the ``initialize`` method.
+
+
+

=== removed file 'doc/Miscellaneous.txt'
--- doc/Miscellaneous.txt	2012-04-30 22:09:31 +0000
+++ doc/Miscellaneous.txt	1970-01-01 00:00:00 +0000
@@ -1,32 +0,0 @@
-[[Miscellaneous]]
-= Miscellaneous Topics
-
-Following are some short items you may find useful as you explore Akiban Persistit. Follow links to the API documentation for more details.
-
-== Histograms
-
-The method +com.persistit.Exchange#computeHistogram+ class provides a way to sample and summarize a set of keys in a +Tree+.  It works by traversing keys in index pages near the root of the tree.  Because only a small fraction of all the keys in the tree are represented in the index, this can result in relatively small sample set of keys relatively quickly. The result can be used to estimate the actual number of keys.
-
-== Temporary Volumes
-
-A Persistit temporary volume is a special kind of Volume that is deleted when Persistit is closed. The update mechanism for temporary volumes avoids writing to disk whenever possible, and its contents are not recoverable Persistit shuts down. Therefore in some cases database operations on temporary volumes are faster.
-
-The primary use case for a temporary volume is an application that needs the unlimited size, but not the persistence of normal Persistit volumes.
-
-See the +com.persistit.Persistit#createTemporaryVolume+ method for additional details.
-
-== Logging
-
-By default Persistit emits log messages to a file called persistit.log  and also writes high level log messages to System.out.  You can change this behavior by plugging in a different logging implementation. In particular, Persistit provides pluggable adapters for various other logging implementations, including Log4J, SLF4J, and the Java logging API introduced in JDK 1.4. For details see the API documentation for com.persistit.logging.AbstractPersistitLogger.
-
-Using one of these logging frameworks is simple.  For example, the following code connects Persistit to an application-supplied SLF4J logger:
-
-[source,java]
-----
-db.setPersistitLogger(new Slf4jAdapter(LOG))
-----
-
-where +db+ is the Persistit instance and +LOG+ is a Logger supplied by SLF4J. This method should be called before the +initialize+ method.
-
-
-

=== added file 'doc/PhysicalStorage.rst'
--- doc/PhysicalStorage.rst	1970-01-01 00:00:00 +0000
+++ doc/PhysicalStorage.rst	2012-05-30 18:23:19 +0000
@@ -0,0 +1,142 @@
+.. _PhysicalStorage:
+
+Physical B-Tree Representation
+==============================
+
+This chapter describes the physical structures used to represent Akiban Persistit records on disk and in memory.
+
+Files
+-----
+
+Following is a directory listing illustrating a working Persistit database::
+
+  -rw-r--r--. 1 demo demo  24G Feb  8 13:18 akiban_data
+  -rw-r--r--. 1 demo demo  48K Feb  8 13:19 akiban_system
+  -rw-r--r--. 1 demo demo 954M Feb  8 13:18 akiban_journal.000000000225
+  -rw-r--r--. 1 demo demo 954M Feb  8 13:19 akiban_journal.000000000226
+  -rw-r--r--. 1 demo demo 954M Feb  8 13:19 akiban_journal.000000000227
+  -rw-r--r--. 1 demo demo 662M Feb  8 13:19 akiban_journal.000000000228
+
+This database contains two *volume* files, ``akiban_data`` and ``akiban_system`` and four files that constitute part of the *journal*. As explained below, Persistit records are usually stored in a combination of volume and journal files.
+
+.. _Journal:
+
+The Journal
+-----------
+
+The *journal* is a set of files containing variable length records. The journal is append-only. New records are written only at the end; existing records are never overwritten. The journal consists of a numbered series of files having a configurable maximum size. When a journal file becomes full Persistit closes it and begins a new file with the next counter value. The maximum size of a journal file is determined by a configuration property called its block size.  The default block size value is 1,000,000,000 bytes which works well with today’s standard server hardware.
+
+Every record in the journal has a 64-bit integer *journal address*. The journal address denotes which file contains the record and the record’s offset within that file. Journal addresses start at zero in a new database instance and grow perpetually. footnote:[Even on a system executing 1 million transactions per second the address space is large enough to last for hundreds of years.]
+
+Persistit writes two major types of records to journal files.
+
+- For each committed update transaction, Persistit writes a record containing sufficient information to replay the transaction during recovery. For example, when Persistit stores a key/value pair during a transaction, it writes a record to the journal containing the key and value.
+- Persistit also writes all updated page images to the journal. Some of these are eventually copied to volume files, as described below. This write/copy mechanism is critical to Persistit’s crash-recovery mechanism (see :ref:`Recovery`).
+
+As updates are applied, Persistit constantly appends new information- both transaction records and modified page images - to the end of the highest-numbered file. To prevent the aggregation of a large number of journal files Persistit also works to copy or remove information from older journal files so that they can be deleted. The background thread responsible for this activity is called the ``JOURNAL_COPIER`` thread. The JOURNAL_COPIER copies pages from the journal back into their home volume files, allowing old files to be deleted. Normally a Persistit system at rest gradually copies all update page images and perform checkpoints so that only one small journal file remains. Applications can accelerate that process by calling the ``com.persistit.Persistit#copyBackPages`` method.
+
+The journal is critical to ensuring Persistit can recover structurally intact B-Trees and apply all committed transactions after a system failure. For this reason, unless the JOURNAL_COPIER is entirely caught up, any attempt to save the state of a Persistit database must include both the volume and journal files.
+
+The journal also plays a critical role during concurrent backup. To back up a running Persistit database, the ``com.persistit.BackupTask`` does the following:
+
+- Enables ``appendOnly`` mode to suspend the copying of updated page images.
+- Copies the appropriate volume and journal files
+- Disables ``appendOnly`` mode to allow JOURNAL_COPIER to continue.
+
+For more details on the journal, checkpoints and transactions, see :ref:`Recovery`. For more information on concurrent backup and other management tasks, see :ref:`Management`.
+
+Pages and Volumes
+-----------------
+
+Persistit ultimately stores its data in one or more Volume files. Persistit manages volume files internally in sections called pages. Every page within one volume has the same size. The page size is configurable and may be 1,024, 2,048, 4,096, 8,192, or 16,384 (recommended) bytes long. Once the page size for a volume has been established, it cannot be changed. See :ref:`Configuration` for details of how to assign the page size for a new volume.
+
+Directory Tree
+^^^^^^^^^^^^^^
+
+Within a volume there can be an unlimited number of B-Trees. (B-Trees are also called simply “trees” in this document.) A tree consists of a set of pages including a *root page*, *index pages* and *data pages*. The root page can be data page if the tree is trivial and contains only small number of records. Usually the root page is an index page which contains references to other index pages which in turn may refer to data pages.
+
+Persistit manages a potentially large number of trees by maintaining a tree of trees called ``_directory``.  The ``_directory`` tree contains the name, root page address, ``com.persistit.Accumulator`` data and ``com.persistit.TreeStatistics`` data for all the other trees in the volume. The tree name ``_directory`` is reserved and may not be used when creating an Exchange.
+
+Data Pages
+^^^^^^^^^^
+
+A data page contains a representation of one or more variable-length key/value pairs. The number of key/value pairs depends on the page size, and the sizes of the serialized keys and values. The first key in each data page is stored in its entirety, while subsequent keys are stored with *prefix compression* to reduce storage footprint and accelerate searches. Therefore the storage size of the second and subsequent keys in a data page depend on how many of the leading bytes of its serialized form match its predecessor. (See :ref:`Key` and :ref:`Value` for information on how Persistit encodes logical Java values into the byte arrays stored in a data page.)
+
+Index Pages
+^^^^^^^^^^^
+
+An index page has a structure similar to a data page except that instead of holding serialized value data, it instead contains page addresses of subordinate pages within the tree.
+
+.. TODO - diagram of B-Tree, page layouts, etc
+
+.. _Recovery:
+
+Recovery
+========
+
+Akiban Persistit is designed, implemented and tested to ensure that whether the application shuts down gracefully or crashes without cleanly closing the database, the database remains structurally intact and internally consistent after restart.
+
+To do this, Persistit performs a process called *recovery* every time it starts up.  The recovery process is generally very fast after a normal shutdown. However, it can take a considerable amount of time after a crash because many committed transactions may need to be executed.
+
+Recovery performs two major activities:
+
+- Restores all B-Trees to an internally consistent state with a known timestamp.
+- Replays all transaction that committed after that timestamp.
+- Prunes multi-version values belonging to certain aborted transactions (see :ref:`Pruning`).
+
+To accomplish this, Persistit writes all updates first to the :ref:`Journal`. Persistit also periodically writes *checkpoint* records to the journal. During recovery, Persistit finds the last valid checkpoint written before shutdown or crash, restores B-Trees to state consistent with that checkpoint, and then replays transactions that committed after the checkpoint.
+
+Recovery depends on the availability of the volume and journal files as they existed prior to abrupt termination. If these are modified or destroyed outside of Persistit, successful recovery is unlikely.
+
+Timestamps and Checkpoints
+--------------------------
+
+Persistit maintains a universal counter called the *timestamp* counter. Every update operation assigns a new, larger timestamp, and every record in the journal includes the timestamp assigned to the operation writing the record. The timestamp counter is unrelated to clock time.  It is merely a counter.
+
+A *checkpoint* is simply a timestamp for which a valid recovery is possible. Periodically Persistit chooses a timestamp to be a new checkpoint. Over time it then ensures that all pages updated before the checkpoint have been written to the journal, and then writes a checkpoint marker. By default checkpoints occur once every two minutes. Normal shutdown through ``com.persistit.Persistit#close`` writes a final checkpoint to the journal regardless of when the last checkpoint cycle occurred. That final checkpoint is what allows recovery after a normal shutdown to be very fast.
+
+Upon start-up Persistit starts by finding the last valid checkpoint timestamp, and then recovers only those page images from the journal that were written prior to it. The result is that all B-Trees are internally consistent and contain all the updates that were issued and committed to disk before the checkpoint timestamp and none the occurred after the checkpoint timestamp.
+
+Then Persistit locates and reapplies all transaction records in the journal for transactions that committed after the last valid checkpoint timestamp. These transactions are reapplied to the database, with the result that:
+
+- The B-Tree index and data structures are intact. All store, fetch, remove and traverse operations will complete successfully. footnote:[Persistit provides the utility class com.persistit.IntegrityCheck to verify the integrity of a Volume.]
+- All committed transactions are present in the recovered database.  (See :ref:`Transactions` for durability determined by ``CommitPolicy``.)
+
+For updates occurring outside of a transaction the resulting state is identical to some consistent, reasonably recent state prior to the termination. (“Reasonably recent” depends on the checkpoint interval, which by default is set to two minutes.)
+
+Flush/Force/Checkpoint
+^^^^^^^^^^^^^^^^^^^^^^
+
+An application may require certainty at various points that all pending updates have been fully written to disk. The ``com.persistit.Persistit`` class provides three methods to ensure that updates have been written:
+
+  ``com.persistit.Persistit#flush``
+      causes Persistit to write all pending updates to the journal. Upon successful completion of flush any pages that needed writing prior to the call to flush are 
+      guaranteed to have been written to their respective volume files.
+  ``com.persistit.Persistit#force``
+      forces the underlying operating system to write pending updates from the operating system’s write-behind cache to the actual disk. (This operation relies on 
+      the underlying ``java.io.Filechannel#force(boolean)`` method.)
+  ``com.persistit.Persistit#checkpoint``
+      causes Persistit to allocate a new checkpoint timestamp and then wait for all updates that happened before that timestamp to be committed to disk.
+
+However, typical applications, especially those using :ref:`Transactions`, do not need to invoke these methods. Once a Transaction is durable, so are all other transactions that occurred at timestamps earlier than the transaction’s commit timestamp and no other method calls are required.
+
+
+The Buffer Pool
+---------------
+
+Persistit maintains a cache of page copies in memory called the *buffer pool*. The buffer pool is a critical resource in reducing disk I/O and providing good run-time performance. After performing a relatively expensive disk operation to read a copy of a page into the buffer pool, Persistit retains that copy to allow potentially many fetch and update operations to be performed against keys and values stored in that page.
+
+Persistit optimizes update operations by writing updated database pages lazily, generally a few seconds to minutes after the update has been performed on the in-memory copy of the page cached in the buffer pool. By writing lazily, Persistit allows many update operations to be completed on each page before incurring a relatively expensive disk I/O operation to write the updated version of the page to the Volume.
+
+In Persistit the buffer pool is a collection of buffers allocated from the heap for the duration of Persistit’s operation. The buffers are allocated by the ``com.persistit.Persistit#initialize`` method and are released when the application invokes close. Because buffers are allocated for the life of the Persistit instance, they impose no garbage collection overhead. (However, especially when using large buffer pool allocation in a JVM with a large heap, there are some special memory configuration issues to consider.  See :ref:`Configuration` for details.)
+
+Persistit allocates buffers from the buffer pool in approximately  least-recently-used (LRU) order. Most applications exhibit behavior in which data, having been accessed once, is read or updated several more times before the application moves to a different area of the database (locality of reference). LRU is an allocation strategy the yields reasonably good overall throughput by maintaining pages that are likely to be used again in the buffer pool in preference to pages that have not been used for a relatively long time.
+
+Generally, allocating more buffers in the buffer pool increases the likelihood that a page will be found in the pool rather than having to be reloaded from disk. Since disk I/O is relatively expensive, this means that enlarging the buffer pool is a good strategy for reducing disk I/O and thereby increasing throughput. Persistit is designed to manage extremely large buffer pools very efficiently, so if memory is available, it is generally a good strategy to maximum buffer pool size.
+
+Tools
+-----
+
+The command-line interface (see :ref:`CLI`) includes tools you can use to examine pages in volumes and records in the journal. Two of these include the ``jview`` and ``pview`` tasks. The ``jview`` command displays journal records selected within an address range, by type, by page address, and using other selection criteria in a readable form.  The ``pview`` command displays the contents of pages selected by page address or key from a volume, or by journal address from the journal.
+
+

=== removed file 'doc/PhysicalStorage.txt'
--- doc/PhysicalStorage.txt	2012-05-04 13:33:49 +0000
+++ doc/PhysicalStorage.txt	1970-01-01 00:00:00 +0000
@@ -1,126 +0,0 @@
-[[PhysicalStorage]]
-= Physical B-Tree Representation
-
-This chapter describes the physical structures used to represent Akiban Persistit records on disk and in memory.
-
-== Files
-
-Following is a directory listing illustrating a working Persistit database:
-
-----
--rw-r--r--. 1 demo demo  24G Feb  8 13:18 akiban_data
--rw-r--r--. 1 demo demo  48K Feb  8 13:19 akiban_system
--rw-r--r--. 1 demo demo 954M Feb  8 13:18 akiban_journal.000000000225
--rw-r--r--. 1 demo demo 954M Feb  8 13:19 akiban_journal.000000000226
--rw-r--r--. 1 demo demo 954M Feb  8 13:19 akiban_journal.000000000227
--rw-r--r--. 1 demo demo 662M Feb  8 13:19 akiban_journal.000000000228
-----
-
-This database contains two _volume_ files, +akiban_data+ and +akiban_system+ and four files that constitute part of the _journal_. As explained below, Persistit records are usually stored in a combination of volume and journal files.
-
-[[Journal]]
-== The Journal
-
-The _journal_ is a set of files containing variable length records. The journal is append-only. New records are written only at the end; existing records are never overwritten. The journal consists of a numbered series of files having a configurable maximum size. When a journal file becomes full Persistit closes it and begins a new file with the next counter value. The maximum size of a journal file is determined by a configuration property called its block size.  The default block size value is 1,000,000,000 bytes which works well with today’s standard server hardware.
-
-Every record in the journal has a 64-bit integer _journal address_. The journal address denotes which file contains the record and the record’s offset within that file. Journal addresses start at zero in a new database instance and grow perpetually. footnote:[Even on a system executing 1 million transactions per second the address space is large enough to last for hundreds of years.]
-
-Persistit writes two major types of records to journal files.
-
-- For each committed update transaction, Persistit writes a record containing sufficient information to replay the transaction during recovery. For example, when Persistit stores a key/value pair during a transaction, it writes a record to the journal containing the key and value.
-- Persistit also writes all updated page images to the journal. Some of these are eventually copied to volume files, as described below. This write/copy mechanism is critical to Persistit’s crash-recovery mechanism (see <<Recovery>>).
-
-As updates are applied, Persistit constantly appends new information- both transaction records and modified page images - to the end of the highest-numbered file. To prevent the aggregation of a large number of journal files Persistit also works to copy or remove information from older journal files so that they can be deleted. The background thread responsible for this activity is called the +JOURNAL_COPIER+ thread. The JOURNAL_COPIER copies pages from the journal back into their home volume files, allowing old files to be deleted. Normally a Persistit system at rest gradually copies all update page images and perform checkpoints so that only one small journal file remains. Applications can accelerate that process by calling the +com.persistit.Persistit#copyBackPages+ method.
-
-The journal is critical to ensuring Persistit can recover structurally intact B-Trees and apply all committed transactions after a system failure. For this reason, unless the JOURNAL_COPIER is entirely caught up, any attempt to save the state of a Persistit database must include both the volume and journal files.
-
-The journal also plays a critical role during concurrent backup. To back up a running Persistit database, the +com.persistit.BackupTask+ does the following:
-
-- Enables +appendOnly+ mode to suspend the copying of updated page images.
-- Copies the appropriate volume and journal files
-- Disables +appendOnly+ mode to allow JOURNAL_COPIER to continue.
-
-For more details on the journal, checkpoints and transactions, see <<Recovery>>. For more information on concurrent backup and other management tasks, see <<Management>>.
-
-== Pages and Volumes
-
-Persistit ultimately stores its data in one or more Volume files. Persistit manages volume files internally in sections called pages. Every page within one volume has the same size. The page size is configurable and may be 1,024, 2,048, 4,096, 8,192, or 16,384 (recommended) bytes long. Once the page size for a volume has been established, it cannot be changed. See <<Configuration>> for details of how to assign the page size for a new volume.
-
-=== Directory Tree
-
-Within a volume there can be an unlimited number of B-Trees. (B-Trees are also called simply “trees” in this document.) A tree consists of a set of pages including a _root page_, _index pages_ and _data pages_. The root page can be data page if the tree is trivial and contains only small number of records. Usually the root page is an index page which contains references to other index pages which in turn may refer to data pages.
-
-Persistit manages a potentially large number of trees by maintaining a tree of trees called +_directory+.  The +_directory+ tree contains the name, root page address, +com.persistit.Accumulator+ data and +com.persistit.TreeStatistics+ data for all the other trees in the volume. The tree name +_directory+ is reserved and may not be used when creating an Exchange.
-
-=== Data Pages
-
-A data page contains a representation of one or more variable-length key/value pairs. The number of key/value pairs depends on the page size, and the sizes of the serialized keys and values. The first key in each data page is stored in its entirety, while subsequent keys are stored with _prefix compression_ to reduce storage footprint and accelerate searches. Therefore the storage size of the second and subsequent keys in a data page depend on how many of the leading bytes of its serialized form match its predecessor. (See <<Key>> and <<Value>> for information on how Persistit encodes logical Java values into the byte arrays stored in a data page.)
-
-=== Index Pages
-
-An index page has a structure similar to a data page except that instead of holding serialized value data, it instead contains page addresses of subordinate pages within the tree.
-
-****
-TODO - diagram of B-Tree, page layouts, etc
-****
-[[Recovery]]
-== Recovery
-
-Akiban Persistit is designed, implemented and tested to ensure that whether the application shuts down gracefully or crashes without cleanly closing the database, the database remains structurally intact and internally consistent after restart.
-
-To do this, Persistit performs a process called _recovery_ every time it starts up.  The recovery process is generally very fast after a normal shutdown. However, it can take a considerable amount of time after a crash because many committed transactions may need to be executed.
-
-Recovery performs two major activities:
-
-- Restores all B-Trees to an internally consistent state with a known timestamp.
-- Replays all transaction that committed after that timestamp.
-- Prunes multi-version values belonging to certain aborted transactions (see <<Pruning>>).
-
-To accomplish this, Persistit writes all updates first to the <<Journal>>. Persistit also periodically writes _checkpoint_ records to the journal. During recovery, Persistit finds the last valid checkpoint written before shutdown or crash, restores B-Trees to state consistent with that checkpoint, and then replays transactions that committed after the checkpoint.
-
-Recovery depends on the availability of the volume and journal files as they existed prior to abrupt termination. If these are modified or destroyed outside of Persistit, successful recovery is unlikely.
-
-=== Timestamps and Checkpoints
-
-Persistit maintains a universal counter called the _timestamp_ counter. Every update operation assigns a new, larger timestamp, and every record in the journal includes the timestamp assigned to the operation writing the record. The timestamp counter is unrelated to clock time.  It is merely a counter.
-
-A _checkpoint_ is simply a timestamp for which a valid recovery is possible. Periodically Persistit chooses a timestamp to be a new checkpoint. Over time it then ensures that all pages updated before the checkpoint have been written to the journal, and then writes a checkpoint marker. By default checkpoints occur once every two minutes. Normal shutdown through +com.persistit.Persistit#close+ writes a final checkpoint to the journal regardless of when the last checkpoint cycle occurred. That final checkpoint is what allows recovery after a normal shutdown to be very fast.
-
-Upon start-up Persistit starts by finding the last valid checkpoint timestamp, and then recovers only those page images from the journal that were written prior to it. The result is that all B-Trees are internally consistent and contain all the updates that were issued and committed to disk before the checkpoint timestamp and none the occurred after the checkpoint timestamp.
-
-Then Persistit locates and reapplies all transaction records in the journal for transactions that committed after the last valid checkpoint timestamp. These transactions are reapplied to the database, with the result that:
-
-- The B-Tree index and data structures are intact. All store, fetch, remove and traverse operations will complete successfully. footnote:[Persistit provides the utility class com.persistit.IntegrityCheck to verify the integrity of a Volume.]
-- All committed transactions are present in the recovered database.  (See <<Transactions>> for durability determined by +CommitPolicy+.)
-
-For updates occurring outside of a transaction the resulting state is identical to some consistent, reasonably recent state prior to the termination. (“Reasonably recent” depends on the checkpoint interval, which by default is set to two minutes.)
-
-=== Flush/Force/Checkpoint
-
-An application may require certainty at various points that all pending updates have been fully written to disk. The +com.persistit.Persistit+ class provides three methods to ensure that updates have been written:
-
-[horizontal]
-+com.persistit.Persistit#flush+:: causes Persistit to write all pending updates to the journal. Upon successful completion of flush any pages that needed writing prior to the call to flush are guaranteed to have been written to their respective volume files.
-+com.persistit.Persistit#force+:: forces the underlying operating system to write pending updates from the operating system’s write-behind cache to the actual disk. (This operation relies on the underlying +java.io.Filechannel#force(boolean)+ method.)
-+com.persistit.Persistit#checkpoint+:: causes Persistit to allocate a new checkpoint timestamp and then wait for all updates that happened before that timestamp to be committed to disk.
-
-However, typical applications, especially those using <<Transactions>>, do not need to invoke these methods. Once a Transaction is durable, so are all other transactions that occurred at timestamps earlier than the transaction’s commit timestamp and no other method calls are required.
-
-
-== The Buffer Pool
-
-Persistit maintains a cache of page copies in memory called the _buffer pool_. The buffer pool is a critical resource in reducing disk I/O and providing good run-time performance. After performing a relatively expensive disk operation to read a copy of a page into the buffer pool, Persistit retains that copy to allow potentially many fetch and update operations to be performed against keys and values stored in that page.
-
-Persistit optimizes update operations by writing updated database pages lazily, generally a few seconds to minutes after the update has been performed on the in-memory copy of the page cached in the buffer pool. By writing lazily, Persistit allows many update operations to be completed on each page before incurring a relatively expensive disk I/O operation to write the updated version of the page to the Volume.
-
-In Persistit the buffer pool is a collection of buffers allocated from the heap for the duration of Persistit’s operation. The buffers are allocated by the +com.persistit.Persistit#initialize+ method and are released when the application invokes close. Because buffers are allocated for the life of the Persistit instance, they impose no garbage collection overhead. (However, especially when using large buffer pool allocation in a JVM with a large heap, there are some special memory configuration issues to consider.  See <<Configuration>> for details.)
-
-Persistit allocates buffers from the buffer pool in approximately  least-recently-used (LRU) order. Most applications exhibit behavior in which data, having been accessed once, is read or updated several more times before the application moves to a different area of the database (locality of reference). LRU is an allocation strategy the yields reasonably good overall throughput by maintaining pages that are likely to be used again in the buffer pool in preference to pages that have not been used for a relatively long time.
-
-Generally, allocating more buffers in the buffer pool increases the likelihood that a page will be found in the pool rather than having to be reloaded from disk. Since disk I/O is relatively expensive, this means that enlarging the buffer pool is a good strategy for reducing disk I/O and thereby increasing throughput. Persistit is designed to manage extremely large buffer pools very efficiently, so if memory is available, it is generally a good strategy to maximum buffer pool size.
-
-== Tools
-
-The command-line interface (see <<CLI>>) includes tools you can use to examine pages in volumes and records in the journal. Two of these include the +jview+ and +pview+ tasks. The +jview+ command displays journal records selected within an address range, by type, by page address, and using other selection criteria in a readable form.  The +pview+ command displays the contents of pages selected by page address or key from a volume, or by journal address from the journal.
-
-

=== added file 'doc/ReleaseNotes.rst'
--- doc/ReleaseNotes.rst	1970-01-01 00:00:00 +0000
+++ doc/ReleaseNotes.rst	2012-05-30 18:23:19 +0000
@@ -0,0 +1,84 @@
+************************************
+Akiban-Persistit 3.1.1 Release Notes
+************************************
+
+Release Date
+============
+May 25, 2012
+
+Overview
+========
+This is the first open source release of the Persistit project (https://launchpad.net/akiban-persistit).  
+
+See http://www.akiban.com/akiban-persistit for a summary of features and benefits, licensing information and how to get support.
+
+Documentation
+=============
+Users Guide (http://www.akiban.com/ak-docs/admin/persistit)
+JavaDoc  (http://www.akiban.com/ak-docs/admin/persistit/apidocs)
+
+Building Akiban-Persistit
+=========================
+Use Maven (maven.apache.org) to build Persistit.
+
+To build::
+
+  mvn install
+
+The resulting jar files are in the ``target`` directory. To build the Javadoc::
+
+  mvn javadoc:javadoc
+
+The resulting Javadoc HTML files are in ``target/site/apidocs``.
+
+Building and Running the Examples
+---------------------------------
+
+Small examples are located in the ``examples`` directory. Each has a short README file describing the example, and an Ant build script (http://ant.apache.org). After building the main akiban-persisit jar file using Maven, you may run::
+
+  ant run
+
+in each of the examples subdirectories to build and run the examples.
+
+Known Issues
+============
+
+Transactional Tree Management
+-----------------------------
+
+All operations within Trees such as store, fetch, remove and traverse are correctly supported within transactions. However, the operations to create and delete Tree instances currently do not respect transaction boundaries. For example, if a transaction creates a new Tree, it is immediately visible within other Transactions and will continue to exist even if the original transaction aborts.  (However, records inserted or modified by the original transaction will not be visible until the transaction commits.) Prior to creating/removing trees, transaction processing should be quiesced and allowed to complete.
+
+Problems with Disk Full - Bug 916071
+------------------------------------
+
+There are rare cases where Persistit will generate exceptions other than java.io.IOException: No space left on device when a disk volume containing the journal or volume file fills up. The database will be intact upon recovery, but the application may receive unexpected exceptions.
+
+Out of Memory Error, Direct Memory Buffer - Bug 985117
+------------------------------------------------------
+
+Out of Memory Error, Direct Memory Buffer.  Can cause failed transactions under extreme load conditions as a result of threads getting backed up writing to the journal file. However, this error is transient and recoverable by by retrying the failed transaction.
+
+* Workaround: Ensure your application has the ability to retry failed transactions
+
+
+Tree#getChangeCount may return inaccurate result - Bug 986465
+-------------------------------------------------------------
+
+The getChangeCount method may return inaccurate results as its not currently transactional.  The primary consumer is the PersistitMap. As a result of this bug Persistit may not generate java.util.ConcurrentModiciationException when it is supposed to.
+
+Multi-Version-Values sometimes not fully pruned - Bug 1000331
+-------------------------------------------------------------
+
+Multi-version values are not always pruned properly causing volume growth.  The number of MVV records and their overhead size can be obtaining by running the IntegrityCheck task. 
+
+* Workaround 1: Run the IntegrityCheck task (CLI command icheck) with the -P option which will prune the MVVs. This will remove obsolete MVV instances and in many cases free up pages in which new data can be stored.  However, it will not reduce the actual size of the volume file.
+
+* Workaround 2: To reduce the size of the volume you can use the CLI commands save  and load to offload and then reload the data into a newly created volume file. See http://www.akiban.com/ak-docs/admin/persistit/Management.html#management for more information about these operations.
+
+
+Buffer Pool Configuration
+=========================
+For optimal performance, proper configuration of the Persistit buffer pool is required.  See section "Configuring the Buffer Pool" in the configuration document http://www.akiban.com/ak-docs/admin/persistit/Configuration.html
+
+.. note:: Especially when used with multi-gigabyte heaps, the default Hotspot JVM server heuristics are be suboptimal for Persistit applications. Persistit is usually configured to allocate a large fraction of the heap to Buffer instances that are allocated at startup and held for the duration of the Persistit instance. For efficient operation, all of the Buffer instances must fit in the tenured (old) generation of the heap to avoid very significant garbage collector overhead.  Use either -XX:NewSize or -Xmn to adjust the relative sizes of the new and old generations.
+

=== added file 'doc/Security.rst'
--- doc/Security.rst	1970-01-01 00:00:00 +0000
+++ doc/Security.rst	2012-05-30 18:23:19 +0000
@@ -0,0 +1,93 @@
+.. _Security:
+
+Security Notes
+==============
+
+Akiban Persistit provides no built-in data access control because it is intended for embedded use in applications that provide their own logical access control. Security-conscious applications must also prevent unauthorized access to Persistit's physical files and to the Persistit API. The following resources must be protected from unauthorized access:
+
+- Persistit volume files
+- configuration properties file, if one is used
+- access by unauthorized code to the API exposed by the Persistit library
+- the port opened and exposed by Persistit when either the ``com.persistit.rmiport`` or ``com.persistit.rmihost`` property is set. If you are using Persistit's remote administration feature, be sure to block unauthorized access to the RMI port.
+- the :ref:`cliserver` if instantiated
+
+In addition to these general deployment considerations, Persistit requires certain permissions in an environment controlled by a security manager.
+
+Java programs run from the command line typically do not install a security manager, and therefore implicitly grant all permissions. However, when Persistit is used within an Applet, or within any framework that installs a security manager, it is important to understand what permissions are required.
+
+Security Domains
+----------------
+
+This section assumes a basic understanding of the Java security model. See http://download.oracle.com/javase/1.5.0/docs/guide/security/spec/security-spec.doc2.html[New Protection Mechanisms - Overview of Basic Concepts] for further information.
+
+Note that when Java is started from the command line, as is often the case in server applications, all security privileges are granted by default. The information in this section is intended for cases where security privileges need to be controlled.
+
+Akiban Persistit performs various security-sensitive operations: it reads and writes files, it reads system properties, it optionally opens a TCP/IP socket, and it performs various security-sensitive operations related to reflection and serialization. These operations are divided into two categories: those required only by the Persistit library itself, and those required by the application that is using Persistit. For example, the application code must have permission to read and write files, but it does not require permission to access private fields through reflection; only the Persistit library itself needs this permission. The latter operation is called a “privileged” operation because Persistit invokes the access controller's doPrivileged method to establish its permission to perform the privileged operation.
+
+At a practical level, this means you can create two separate security domains for applications embedding Persistit.  One domain is specific to the Persistit library itself, and grants all the permissions required by Persistit, including the privileged permissions. The other domain includes the application, and grants only the non-privileged permissions.
+
+Using the default java.lang.SecurityManager implementation, you define domains and the permissions available to them in a policy file stored in the user's home directory.  For those already familiar with the policy file format, here is a security policy file that illustrates these concepts:
+
+.. code-block:: java
+
+  grant codeBase "file:/appdir/myapplication.jar" {
+    permission java.io.FilePermission "e:\\data\\*", "read, write, delete";
+    permission java.io.FilePermission "e:\\logs\\*", "read, write, delete";
+    permission java.net.SocketPermission "localhost",
+           	  "accept, connect, listen, resolve";
+  };
+
+  grant codeBase "file:/lib/akiban-persistit.jar" {
+    permission java.io.FilePermission "<<ALL FILES>>", "read, write, delete";
+    permission java.net.SocketPermission "*:1099-",
+         	  "accept, connect, listen, resolve";
+    permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
+    permission java.io.SerializablePermission "enableSubclassImplementation";
+    permission java.util.PropertyPermission "com.persistit.*", "read";
+    permission java.lang.RuntimePermission "accessDeclaredMembers";
+  };
+
+This policy file sets up two security domains. One covers the application code in ``myapplication.jar`` and grants just the restricted set of permissions needed by the application, while the other grants additional permissions to the Persistit library.
+
+Although the file and socket permissions granted by the privileged domain are less restrictive than those granted to ``myapplication.jar``, the actual permission granted to running code is the intersection of these two and is therefore restricted to just the set of files and sockets granted to myapplication.jar.
+
+Permissions Required for DefaultValueCoder
+------------------------------------------
+
+DefaultValueCoder performs three security-sensitive operations:
+
+- It enumerates the declared fields of the class being serialized,
+- It reads and writes data from and to those fields using reflection even if they are private, and
+- It overrides the default implementations of java.io.ObjectInputStream and java.io.ObjectOutputStream.
+
+If a SecurityManager is installed then three permissions must be granted to enable this new mechanism::
+
+  java.lang.RuntimePermission "accessDeclaredMembers";
+  java.lang.reflect.ReflectPermission("suppressAccessChecks")
+  java.io.SerializablePermission("enableSubclassImplementation")
+
+Persistit acquires these permissions through privileged operations, meaning that only the Persistit library domain needs to have them – they do not need to be and should not be granted to the application domain.
+
+Permission Required for Reading System Properties
+-------------------------------------------------
+
+Persistit attempts to read system properties whose names begin with “com.persistit.” Specific permission to do this is granted by the line::
+
+   permission java.util.PropertyPermission "com.persistit.*", "read";
+
+Again, only the Persistit library domain needs to have this permission. If this permission is not granted, Persistit ignores all system properties.
+
+Permissions Required for File and Socket I/O
+--------------------------------------------
+
+Persistit needs permission to read and write its volume and journal files, to read a configuration properties file and (optionally) write to a log file.  File I/O permissions also apply to the source and destination files specified for Import and Export tasks available within the AdminUI utility. In addition, if you specify either the rmihost or rmiport property to enable remote administration, Persistit needs permission to create RMI connections.
+
+These are not privileged operations, meaning that if the policy establishes separate domains for the application and the Persistit library, both domains must grant permission for all I/O operations. (If they were privileged operations, an untrusted application code could use the Persistit library as a proxy to perform malicious file I/O.) As is defined by the Java security mechanism, when Persistit attempts to open a file, permissions of both the application domain and the Persistit library domains are checked; if the operation is denied by either domain then the attempt fails with a java.security.AccessControlException.
+
+As specified in the sample policy file above, the Persistit library domain has been granted permissions on `<<ALL FILES>>`. This means that the application domain controls what subset of the file system is accessible.  In the example, files may only be read and written to the e:\data and e:\logs directories on a Windows box.  (See the Java documentation on Permissions for details on how to construct File and Socket permissions.)
+
+Deploying Persistit as an Installed Optional Package
+----------------------------------------------------
+
+A convenient way to grant Persistit the permissions required to perform its privileged operations is to install it as an optional package. The Sun Java Runtime Environment treats JAR files located in the <jre-home>/lib/ext directory as optional Java extension classes, and by default grants them the same privileges as Java system classes. If the Persistit library is loaded from this directory then only the application File and Socket privileges need to be granted explicitly through a security policy. To deploy Persistit in this manner simply copy the Persistit library jar file to the appropriate ``*jre-home*/lib/ext`` directory.
+

=== removed file 'doc/Security.txt'
--- doc/Security.txt	2012-04-30 22:09:31 +0000
+++ doc/Security.txt	1970-01-01 00:00:00 +0000
@@ -1,91 +0,0 @@
-[[Security]]
-= Security Notes
-
-Akiban Persistit provides no built-in data access control because it is intended for embedded use in applications that provide their own logical access control. Security-conscious applications must also prevent unauthorized access to Persistit's physical files and to the Persistit API. The following resources must be protected from unauthorized access:
-
-- Persistit volume files
-- configuration properties file, if one is used
-- access by unauthorized code to the API exposed by the Persistit library
-- the port opened and exposed by Persistit when either the +com.persistit.rmiport+ or +com.persistit.rmihost+ property is set. If you are using Persistit's remote administration feature, be sure to block unauthorized access to the RMI port.
-- the <<cliserver>> if instantiated
-
-In addition to these general deployment considerations, Persistit requires certain permissions in an environment controlled by a security manager.
-
-Java programs run from the command line typically do not install a security manager, and therefore implicitly grant all permissions. However, when Persistit is used within an Applet, or within any framework that installs a security manager, it is important to understand what permissions are required.
-
-== Security Domains
-
-This section assumes a basic understanding of the Java security model. See http://download.oracle.com/javase/1.5.0/docs/guide/security/spec/security-spec.doc2.html[New Protection Mechanisms - Overview of Basic Concepts] for further information.
-
-Note that when Java is started from the command line, as is often the case in server applications, all security privileges are granted by default. The information in this section is intended for cases where security privileges need to be controlled.
-
-Akiban Persistit performs various security-sensitive operations: it reads and writes files, it reads system properties, it optionally opens a TCP/IP socket, and it performs various security-sensitive operations related to reflection and serialization. These operations are divided into two categories: those required only by the Persistit library itself, and those required by the application that is using Persistit. For example, the application code must have permission to read and write files, but it does not require permission to access private fields through reflection; only the Persistit library itself needs this permission. The latter operation is called a “privileged” operation because Persistit invokes the access controller's doPrivileged method to establish its permission to perform the privileged operation.
-
-At a practical level, this means you can create two separate security domains for applications embedding Persistit.  One domain is specific to the Persistit library itself, and grants all the permissions required by Persistit, including the privileged permissions. The other domain includes the application, and grants only the non-privileged permissions.
-
-Using the default java.lang.SecurityManager implementation, you define domains and the permissions available to them in a policy file stored in the user's home directory.  For those already familiar with the policy file format, here is a security policy file that illustrates these concepts:
-
-.Example .java.policy File
-----
-grant codeBase "file:/appdir/myapplication.jar" {
-  permission java.io.FilePermission "e:\\data\\*", "read, write, delete";
-  permission java.io.FilePermission "e:\\logs\\*", "read, write, delete";
-  permission java.net.SocketPermission "localhost",
-         	"accept, connect, listen, resolve";
-};
-
-grant codeBase "file:/lib/akiban-persistit.jar" {
-  permission java.io.FilePermission "<<ALL FILES>>", "read, write, delete";
-  permission java.net.SocketPermission "*:1099-",
-         	"accept, connect, listen, resolve";
-  permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
-  permission java.io.SerializablePermission "enableSubclassImplementation";
-  permission java.util.PropertyPermission "com.persistit.*", "read";
-  permission java.lang.RuntimePermission "accessDeclaredMembers";
-};
-----
-
-This policy file sets up two security domains. One covers the application code in +myapplication.jar+ and grants just the restricted set of permissions needed by the application, while the other grants additional permissions to the Persistit library.
-
-Although the file and socket permissions granted by the privileged domain are less restrictive than those granted to +myapplication.jar+, the actual permission granted to running code is the intersection of these two and is therefore restricted to just the set of files and sockets granted to myapplication.jar.
-
-== Permissions Required for DefaultValueCoder
-
-DefaultValueCoder performs three security-sensitive operations:
-
-- It enumerates the declared fields of the class being serialized,
-- It reads and writes data from and to those fields using reflection even if they are private, and
-- It overrides the default implementations of java.io.ObjectInputStream and java.io.ObjectOutputStream.
-
-If a SecurityManager is installed then three permissions must be granted to enable this new mechanism:
-
-----
-java.lang.RuntimePermission "accessDeclaredMembers";
-java.lang.reflect.ReflectPermission("suppressAccessChecks")
-java.io.SerializablePermission("enableSubclassImplementation")
-----
-
-Persistit acquires these permissions through privileged operations, meaning that only the Persistit library domain needs to have them – they do not need to be and should not be granted to the application domain.
-
-== Permission Required for Reading System Properties
-
-Persistit attempts to read system properties whose names begin with “com.persistit.” Specific permission to do this is granted by the line
-
-----
-   permission java.util.PropertyPermission "com.persistit.*", "read";
-----
-
-Again, only the Persistit library domain needs to have this permission. If this permission is not granted, Persistit ignores all system properties.
-
-== Permissions Required for File and Socket I/O
-
-Persistit needs permission to read and write its volume and journal files, to read a configuration properties file and (optionally) write to a log file.  File I/O permissions also apply to the source and destination files specified for Import and Export tasks available within the AdminUI utility. In addition, if you specify either the rmihost or rmiport property to enable remote administration, Persistit needs permission to create RMI connections.
-
-These are not privileged operations, meaning that if the policy establishes separate domains for the application and the Persistit library, both domains must grant permission for all I/O operations. (If they were privileged operations, an untrusted application code could use the Persistit library as a proxy to perform malicious file I/O.) As is defined by the Java security mechanism, when Persistit attempts to open a file, permissions of both the application domain and the Persistit library domains are checked; if the operation is denied by either domain then the attempt fails with a java.security.AccessControlException.
-
-As specified in the sample policy file above, the Persistit library domain has been granted permissions on `<<ALL FILES>>`. This means that the application domain controls what subset of the file system is accessible.  In the example, files may only be read and written to the e:\data and e:\logs directories on a Windows box.  (See the Java documentation on Permissions for details on how to construct File and Socket permissions.)
-
-== Deploying Persistit as an Installed Optional Package
-
-A convenient way to grant Persistit the permissions required to perform its privileged operations is to install it as an optional package. The Sun Java Runtime Environment treats JAR files located in the <jre-home>/lib/ext directory as optional Java extension classes, and by default grants them the same privileges as Java system classes. If the Persistit library is loaded from this directory then only the application File and Socket privileges need to be granted explicitly through a security policy. To deploy Persistit in this manner simply copy the Persistit library jar file to the appropriate +_jre-home_/lib/ext+ directory.
-

=== added file 'doc/Serialization.rst'
--- doc/Serialization.rst	1970-01-01 00:00:00 +0000
+++ doc/Serialization.rst	2012-05-30 18:23:19 +0000
@@ -0,0 +1,250 @@
+.. _Serialization:
+
+Serializing Object Values
+=========================
+
+Akiban Persistit uses one of several mechanisms to serialize a Java Object into a ``com.persistit.Value``.
+
+* For the following classes, Persistit provides built-in optimized serialization logic that cannot be overridden:
+
+  * ``java.lang.String``
+  * ``java.util.Date``
+  * ``java.math.BigInteger``
+  * ``java.math.BigDecimal``
+  * Wrapper classes for primitive values (``Boolean``, ``Byte``, ``Short``, etc.)
+  * All arrays (however, the mechanisms described here apply to array elements).
+
+* An application can register a custom ``com.persistit.encoding.ValueCoder`` to handle serialization of a particular class
+* Default serialization using Persistit's built-in serialization mechanism described below, or
+* Standard Java serialization as described in http://download.oracle.com/javase/1.5.0/docs/guide/serialization/spec/serial-arch.html[Java Object Serialization Specification].
+
+Persistit's default serialization method serializes objects into approximately 33% fewer bytes, and depending on the structure of objects being serialized, is about 40% faster than Java serialization.
+
+Storing Objects in Persistit
+----------------------------
+
+To store an object value into a Persistit database, you put the object into the Value field of an Exchange, and then invoke the Exchange's store method as shown in this code fragment:
+
+.. code-block:: java
+
+    exchange.getValue().put(myObject);
+    exchange.store();
+
+Of course, Persistit cannot actually store a live object on disk.  Instead it creates and stores a byte array containing state information about the object. Subsequently you fetch an object from Persistit as follows:
+
+.. code-block:: java
+
+    exchange.fetch();
+    MyClass myObject = (MyClass)exchange.getValue().get();
+
+The resulting MyClass instance is a newly constructed object instance that is equivalent - subject to the accuracy of the serialization code - to the original object. This process is equivalent to the serialization and deserialization capabilities provided by java.io.ObjectOutputStream and java.io.ObjectInputStream.
+
+Persistit makes use of helper classes called “coders” to marshal data between live objects and their stored byte-array representations. Value coders, which implement ``com.persistit.encoding.ValueCoder``, marshal data to and from Value objects; ``com.persistit.encoding.KeyCoder`` implementations do the same for ``com.persistit.Key``s. A value coder provides capability somewhat like the custom serialization logic implemented through ``readObject``, ``writeObject``, ``readExternal`` and ``writeExternal``. However, a value coder can provide this logic for any class without modifying the class itself, which may be important if the class is part of a closed library.
+
+You may create and register a value coder for almost any class, including classes that are not marked Serializable. The exceptions are those listed which have built-in, non-overridable serialization logic.
+
+DefaultValueCoder and SerialValueCoder
+--------------------------------------
+
+When required to serialize or deserialize class with no explicitly defined ``ValueCoder``, Persistit automatically creates and registers one of the following two default ``ValueCoder`` implementations:
+
+``com.persistit.DefaultValueCoder``::  uses introspection to determine which fields to serialize, and reflection to access and update the fields
+``com.persistit.encoding.SerialValueCoder``:: creates instances of ObjectInputStream and ObjectOutputStream to serialize and deserialize the object.
+
+DefaultValueCoder uses a more compact storage format and is significantly faster than standard Java serialization; however, it imposes certain limitations and trade-offs described below. By default, Persistit will use a DefaultValueCoder. However, you can identify classes that should instead be serialized and deserialized by ``SerialValueCoder`` by specifying the ``serialOverride`` configuration property, which is described below.
+
+DefaultValueCoder
+-----------------
+
+A DefaultValueCoder uses Java reflection to access and update the fields of an arbitrary object. The set of fields is defined by the Java Object Serialization Specification. By default, these include all non-static, non-transient fields of the current class and its Serializable superclasses.  A class may override this default set by specifying an array of ``java.io.ObjectStreamField`` objects in a private final static field named ``serialPersistentFields``, as described in the specification.
+
+``DefaultValueCoder`` invokes the special methods ``readResolve``, ``writeReplace``, ``readObject`` and ``writeObject``,  (or for Externalizable classes,  ``writeExternal`` and ``readExternal``) to provide the compatible custom serialization support. To support the ``readObject``/``readExternal`` and ``writeObject``/``writeExternal`` methods, Persistit creates extended implementations of ``java.io.ObjectOutputStream`` and ``java.io.ObjectInputStream``. These use a custom serialization format optimized for writing to a Value's backing byte array. For example, they do not organize data into 1,024-byte blocks, and they factor meta data about classes into a separate class information database so that this information is not repeated in multiple records containing instances of the same class.
+
+Currently, ``DefaultValueCoder`` does not support the following elements of the serialization API:
+
+- the ``readObjectNoData`` custom serialization method
+- the ``PutFields``/``GetFields`` API of ``ObjectOutputStream`` and ``ObjectInputStream``
+- the ``readLine`` method of ``ObjectInputStream``.
+
+Constructing Objects upon Deserialization
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When deserializing a value, ``DefaultValueCoder`` combines information about the original object's class and the stored field data to reconstruct an object equivalent to the original. To do so it must first construct a new instance of class and then decode and set its serialized fields.
+
+For compatibility with standard Java serialization, ``DefaultValueCoder`` constructs new object instances of Serializable classes using the same logic as ``ObjectInputStream``, namely:
+
+If the class is Externalizable, ``DefaultValueCoder`` invokes its public no-argument constructor. (The specification for Externalizable requires the class to have such a constructor.)
+
+Otherwise, if the class is Serializable, ``DefaultValueCoder`` invokes the no-argument constructor of its nearest non-serializable superclass.
+
+``DefaultValueCoder`` must use platform-specific logic when constructing instances of Serializable classes: specifically, it invokes the same internal, non-public method as ``ObjectInputStream``. We have verified correct behavior on a wide range of Java runtime environments, but because the implementation uses private methods within various JRE versions, it is possible (though unlikely) that a future JRE will not provide a comparable capability.
+
+To avoid using platform-specific API calls, you can specify the configuration property::
+
+    constructorOverride=true
+
+When this property is ``true``, ``DefaultValueCoder`` requires each object being serialized or deserialized to have a no-argument constructor through which instances will be constructed during deserialization. Unless the class implements Externalizable, that constructor may be private, package-private, protected or public.
+
+Extending DefaultValueCoder
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+You can register an extended ``DefaultValueCoder`` to provide custom behavior, including custom logic for constructing instances of a class, as shown here:
+
+.. code-block:: java
+
+	Persistit.getInstance().getCoderManager().registerValueCoder(MyClass.class, new DefaultValueCoder(MyClass.class) {
+  public Object get(Value value, Class clazz, CoderContext context) throws ConversionException {
+
+    // Construct the object being deserialized.
+    Object instance = new MyClass(...custom arguments...);
+
+    // See "registering objects while deserializing" below
+    value.registerEncodedObject(instance);
+    
+    // Load the non-transient, non-static fields
+    render(value, instance, clazz, context);
+    
+    return instance;
+        	}
+ });
+
+
+
+Security Policy Requirements for DefaultValueCoder
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+DefaultValueCoder performs security-sensitive operations: (a) it reads and writes data from and to private fields using reflection, and (b) it overrides the default implementations of java.io.ObjectInputStream and java.io.ObjectOutputStream. If a SecurityManager is installed then three permissions must be granted to enable the new mechanism::
+
+  java.lang.RuntimePermission "accessDeclaredMembers";
+  java.lang.reflect.ReflectPermission("suppressAccessChecks")
+  java.io.SerializablePermission("enableSubclassImplementation")
+
+See :ref:`Security` for an extended discussion on security policy issues for Persistit.
+
+SerialValueCoder
+----------------
+
+``SerialValueCoder`` uses standard Java serialization to store and retrieve object values. Typically this results in slower performance and a more verbose storage format than ``DefaultValueCoder``, but there are a number of reasons why a particular application might require standard Java serialization, including:
+
+- the security context into which the application will be deployed does not grant the permissions noted above that are required for ``DefaultValueCoder``,
+- to avoid Persistit's use of private API calls to construct object instances during deserialization,
+- a preference for the use of a standard format defined within the Java platform rather than Persistit's custom format,
+- limitations documented above on the API elements available during custom deserialization within DefaultValueCoder, for example non-support of GetField and PutField.
+
+Your application can specify ``SerialValueCoders`` for specific classes either by explicitly creating and registering them, or by naming them in the com.persistit.serialOverride property.
+
+To explicitly register a ``SerialValueCoder`` for the class ``MyClass``, do this:
+
+.. code-block:: java
+
+	...
+	Persistit.getInstance().getCoderManager().registerValueCoder(
+    	MyClass.class,
+    	new SerialValueCoder(MyClass.class));
+	...
+
+
+The ``com.persistit.serialOverride`` Configuration Property
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``serialOverride`` property specifies classes that are to be serialized by ``SerialValueCoder`` rather than ``DefaultValueCoder``. This property affects how Persistit assigns a value coder when none has previously been registered. It does not override or affect explicitly registered coders.
+
+Names are separated by commas and may contain wild cards.
+
+The following are valid patterns:
+
+  ``java.io.File``
+      Just the File class.
+  ``java.io.*``
+      All classes in the java.io package.
+  ``java.awt.**``
+      All classes in the java.awt package and its sub-packages
+  ``java.util.*Map``
+      All of the Map classes in the java.util.
+  ``**``
+      All classes in all packages
+
+More precisely, ``serialOverride`` specifies a comma-delimited list of zero or more patterns, each of which is either a fully-qualified class name or pattern that has within it exactly one wild card. The wild card “\*” replaces any sequence of characters other than a period (“.”), while “\*\*” replaces any sequence of characters including periods.  For example::
+
+  ``serialOverride=org.apache.**,com.mypkg.serialstuff.*,com.mypkg.MyClass``
+
+Like all configuration properties, you may specify this in the persistit.properties file or as a system property through a Java command-line argument in the form::
+
+  ``-Dcom.persistit.serialOverride=...``
+
+Registering Objects in a Custom ``ValueCoder``
+----------------------------------------------
+
+In a custom ``ValueCoder`` implementation, the ``get`` method is responsible for constructing and populating an instance of an object. The following pattern should be used when implementing the get method:
+
+.. code-block:: java
+
+  public void get(Value value, Class clazz, CoderContext context) throws ConversionException {
+    	// Construct the object being deserialized.
+    	//
+    	Object instance = ...constructor for the object...
+
+    	// Associate a handle with the newly
+    	// created instance.
+    	//
+    	value.registerEncodedObject(instance);
+
+    	// Populate the object's internal state
+    	//
+    	... load the fields – for example, by calling render...
+
+    	return instance;
+  }
+
+The purpose of the ``registerEncodedObject`` method is to record the association between the newly created object and an internal integer-valued handle that may be used subsequently in the serialization stream to refer to that object. This mechanism supports objects that may have fields that refer either indirectly or indirectly back to the same object – i.e., that participate in a cyclical reference graph.
+
+As a concrete example, consider a Person class with a spouse field such that for married couple p and q,  p.spouse is q and q.spouse is p. When Persistit serializes p it also serializes q, but when it serializes q's spouse field, it records a reference handle associated with the already-serialized instance of p rather than writing a new copy of p in the serialization stream. Upon deserializing q, Persistit looks up the object for the recorded handle to correctly associate the already-deserialized p instance with q.
+
+Whenever you implement a custom ``get()`` method in any ``ValueCoder``, you must notify the underlying Value object about the newly created object by calling registerEncodedObject before deserializing its fields so that any back-references made within serialized fields of that object can find the object correctly.
+
+``Value.toString()`` and ``decodeDisplayable``
+----------------------------------------------
+
+In many cases it is not very useful simply to display the result of evaluating ``toString()`` on an object. The default toString method inherited from Object conveys just a class name and a memory handle. In addition, for remote operations of AdminUI, it may not even be feasible to construct a deserialized object for each record. Therefore, ``com.persistit.Value`` provides a specialized ``toString()`` method to render an arbitrary object value into a legible string. The AdminUI utility uses this facility to summarize the data contained in a Tree.
+
+Persistit creates a String value loading the object's class, using the following algorithm:
+
+- If the state represented by this Value is undefined, then return "undefined".
+- If the state is null or a boolean, return "null" "false", or "true".
+- If the value represents a primitive type, return the string representation of the value, prefixed by "(byte)", "(short)", "(char)", "(long)", or "(float)" for the corresponding types. Values of type int and double are presented without prefix to reduce clutter.
+- If the value represents a String, return a modified form of the string enclosed in double quotes. For each character of the string, if it is a double quote replace it by "\"", otherwise if it is outside of the printable ASCII character set replace the character in the modified string by "\b", "\t", "\n", "\r" or "\uNNNN" such that the modified string would be a valid Java string constant.
+- If the value represents a wrapper for a primitive value (i.e., a java.lang.Boolean, java.lang.Byte, etc.) return the string representation of the value prefixed by "(Boolean)", "(Byte)", "(Short)", "(Character)", "(Integer)", "(Long)", "(Float)" or "(Double)".  The package name java.lang is removed to reduce clutter.
+- If the value represents a java.util.Date, return a formatted representation of the date using the format specified by Key.SDF. This is a readable format that displays the date with full precision, including milliseconds.
+- If the value represents an array, return a list of comma-separated element values surrounded by square brackets.
+- If the value represents one of the standard Collection implementations in the java.util package, then return a comma-separated list of values surrounded by square brackets.
+- If the value represents one of the standard Map implementations in the java.util package, then return a comma-separated list of key/value pairs surrounded by square brackets. Each key/value pair is represented by a string in the form key->value.
+- If the value represents an object of a class for which there is a registered com.persistit.encoding.ValueDisplayer, invoke the displayer's display method to format a displayable representation of the object.
+- If the value represents an object that has been stored using the version default serialization mechanism described above, return the class name of the object followed by a comma-separated tuple, enclosed within curly brace characters, representing the value of each field of the object.
+- If the value represents an object encoded through standard Java serialization, return the string "(Serialized-object)" followed by a sequence of hex digits representing the serialized bytes. Note that this process does not attempt to deserialize the object.
+- If the value represents an object that has already been represented within the formatted result - for example, if a Collection contains two references to the same object - then instead of creating an additional string representing the second or subsequent instance, emit a back reference pointer in the form @NNN where NNN is the character offset within the displayable string where the first instance was found. (This does not apply to strings and the primitive wrapper classes.)
+
+For example, consider a Person having for date of birth, first name, last name, salary and friends, an array of other Person objects. The result returned by toString() on a Value representing Mary Smith who has a friend John Smith, might appear as follows::
+
+ (Person){(Date)19490826000000.000-0400,"Mary","Jones",(long)75000,[
+	(Person){(Date)19550522000000.000-0400,"John","Smith",(long)68000,[@0]}]}
+
+In this example, John Smith's friends array contains a back reference to Mary Jones in the form "@0" because Mary's displayable reference starts at the beginning of the string.
+
+
+PersistitReference
+------------------
+
+In general, serializing an object that contains references to other objects requires all the referenced objects also to be serialized. For an object connected to a large reference graph, it may be impractical or even semantically incorrect to serialize the entire graph.
+
+One way to control the serialization graph for such an object is to write a custom ValueCoder; the custom ValueCoder can store key values for looking up the referenced object, rather than the object itself.  The ValueCoderDemo.java program demonstrates how this can be done.
+
+The ``com.persistit.ref.PersistitReference`` interface, and its abstract subclasses, provide an alternative mechanism for breaking up an object reference graph.  It requires no custom ValueCoder, but does impact the design of application classes.  In addition, you will need to write a concrete implementation of either com.persistit.ref.AbstractReference or com.persistit.ref.AbstractWeakReference based on the actual storage structure of your object graph.
+
+ObjectCache
+-----------
+
+A ``com.persistit.Value`` object holds the serialized, encoded state of a primitive value of an object.  Each time you invoke the get method on a Value, Persistit generates a new copy of the object deserialized from this Value.  Persistit does not implicitly cache deserialized objects. However, the ``com.persistit.encoding.ObjectCache`` class provides a simple mechanism for applications that need to maintain an in-memory cache of of objects from Persistit. ``ObjectCache`` works somewhat like a specialized version of java.util.WeakHashMap.
+
+``ObjectCache`` has ``put``, ``get`` and ``remove`` methods much like a normal Map implementation.  However, when storing an object value with the supplied ``com.persistit.Key``, ``ObjectCache`` constructs a new, immutable ``com.persistit.KeyState`` object to hold as an internal key. This is necessary because ``Key`` objects change value as they are used.
+
+Each ``ObjectCache`` entry holds its object value as a ``SoftReference``, making it available for garbage collection when space is needed.

=== removed file 'doc/Serialization.txt'
--- doc/Serialization.txt	2012-04-30 22:09:31 +0000
+++ doc/Serialization.txt	1970-01-01 00:00:00 +0000
@@ -1,246 +0,0 @@
-[[Serialization]]
-= Serializing Object Values
-
-Akiban Persistit uses one of several mechanisms to serialize a Java Object into a +com.persistit.Value+.
-
-- For the following classes, Persistit provides built-in optimized serialization logic that cannot be overridden:
-
-* +java.lang.String+
-* +java.util.Date+
-* +java.math.BigInteger+
-* +java.math.BigDecimal+
-* Wrapper classes for primitive values (+Boolean+, +Byte+, +Short+, etc.)
-* All arrays (however, the mechanisms described here apply to array elements).
-
-- An application can register a custom +com.persistit.encoding.ValueCoder+ to handle serialization of a particular class
-- Default serialization using Persistit's built-in serialization mechanism described below, or
-- Standard Java serialization as described in http://download.oracle.com/javase/1.5.0/docs/guide/serialization/spec/serial-arch.html[Java Object Serialization Specification].
-
-Persistit's default serialization method serializes objects into approximately 33% fewer bytes, and depending on the structure of objects being serialized, is about 40% faster than Java serialization.
-
-== Storing Objects in Persistit
-
-To store an object value into a Persistit database, you put the object into the Value field of an Exchange, and then invoke the Exchange's store method as shown in this code fragment:
-
-[source,java]
-----
-    exchange.getValue().put(myObject);
-    exchange.store();
-----
-
-Of course, Persistit cannot actually store a live object on disk.  Instead it creates and stores a byte array containing state information about the object. Subsequently you fetch an object from Persistit as follows:
-
-[source,java]
-----
-    exchange.fetch();
-    MyClass myObject = (MyClass)exchange.getValue().get();
-----
-
-The resulting MyClass instance is a newly constructed object instance that is equivalent - subject to the accuracy of the serialization code - to the original object. This process is equivalent to the serialization and deserialization capabilities provided by java.io.ObjectOutputStream and java.io.ObjectInputStream.
-
-Persistit makes use of helper classes called “coders” to marshal data between live objects and their stored byte-array representations. Value coders, which implement +com.persistit.encoding.ValueCoder+, marshal data to and from Value objects; +com.persistit.encoding.KeyCoder+ implementations do the same for +com.persistit.Key+ s. A value coder provides capability somewhat like the custom serialization logic implemented through +readObject+, +writeObject+, +readExternal+ and +writeExternal+. However, a value coder can provide this logic for any class without modifying the class itself, which may be important if the class is part of a closed library.
-
-You may create and register a value coder for almost any class, including classes that are not marked Serializable. The exceptions are those listed which have built-in, non-overridable serialization logic.
-
-=== DefaultValueCoder and SerialValueCoder
-
-When required to serialize or deserialize class with no explicitly defined +ValueCoder+, Persistit automatically creates and registers one of the following two default +ValueCoder+ implementations:
-
-+com.persistit.DefaultValueCoder+::  uses introspection to determine which fields to serialize, and reflection to access and update the fields
-+com.persistit.encoding.SerialValueCoder+:: creates instances of ObjectInputStream and ObjectOutputStream to serialize and deserialize the object.
-
-DefaultValueCoder uses a more compact storage format and is significantly faster than standard Java serialization; however, it imposes certain limitations and trade-offs described below. By default, Persistit will use a DefaultValueCoder. However, you can identify classes that should instead be serialized and deserialized by +SerialValueCoder+ by specifying the +serialOverride+ configuration property, which is described below.
-
-== DefaultValueCoder
-
-A DefaultValueCoder uses Java reflection to access and update the fields of an arbitrary object. The set of fields is defined by the Java Object Serialization Specification. By default, these include all non-static, non-transient fields of the current class and its Serializable superclasses.  A class may override this default set by specifying an array of +java.io.ObjectStreamField+ objects in a private final static field named +serialPersistentFields+, as described in the specification.
-
-+DefaultValueCoder+ invokes the special methods +readResolve+, +writeReplace+, +readObject+ and +writeObject+,  (or for Externalizable classes,  +writeExternal+ and +readExternal+) to provide the compatible custom serialization support. To support the +readObject+/+readExternal+ and +writeObject+/+writeExternal+ methods, Persistit creates extended implementations of +java.io.ObjectOutputStream+ and +java.io.ObjectInputStream+. These use a custom serialization format optimized for writing to a Value's backing byte array. For example, they do not organize data into 1,024-byte blocks, and they factor meta data about classes into a separate class information database so that this information is not repeated in multiple records containing instances of the same class.
-
-Currently, +DefaultValueCoder+ does not support the following elements of the serialization API:
-
-- the +readObjectNoData+ custom serialization method
-- the +PutFields+/+GetFields+ API of +ObjectOutputStream+ and +ObjectInputStream+
-- the +readLine+ method of +ObjectInputStream+.
-
-=== Constructing Objects upon Deserialization
-
-When deserializing a value, +DefaultValueCoder+ combines information about the original object's class and the stored field data to reconstruct an object equivalent to the original. To do so it must first construct a new instance of class and then decode and set its serialized fields.
-
-For compatibility with standard Java serialization, +DefaultValueCoder+ constructs new object instances of Serializable classes using the same logic as +ObjectInputStream+, namely:
-
-If the class is Externalizable, +DefaultValueCoder+ invokes its public no-argument constructor. (The specification for Externalizable requires the class to have such a constructor.)
-
-Otherwise, if the class is Serializable, +DefaultValueCoder+ invokes the no-argument constructor of its nearest non-serializable superclass.
-
-+DefaultValueCoder+ must use platform-specific logic when constructing instances of Serializable classes: specifically, it invokes the same internal, non-public method as +ObjectInputStream+. We have verified correct behavior on a wide range of Java runtime environments, but because the implementation uses private methods within various JRE versions, it is possible (though unlikely) that a future JRE will not provide a comparable capability.
-
-To avoid using platform-specific API calls, you can specify the configuration property
-
-----
-    constructorOverride=true
-----
-
-When this property is +true+, +DefaultValueCoder+ requires each object being serialized or deserialized to have a no-argument constructor through which instances will be constructed during deserialization. Unless the class implements Externalizable, that constructor may be private, package-private, protected or public.
-
-=== Extending DefaultValueCoder
-
-You can register an extended +DefaultValueCoder+ to provide custom behavior, including custom logic for constructing instances of a class, as shown here:
-
-[source,java]
------
-	Persistit.getInstance().getCoderManager().registerValueCoder(MyClass.class, new DefaultValueCoder(MyClass.class) {
-  public Object get(Value value, Class clazz, CoderContext context) throws ConversionException {
-
-    // Construct the object being deserialized.
-    Object instance = new MyClass(...custom arguments...);
-
-    // See "registering objects while deserializing" below
-    value.registerEncodedObject(instance);
-    
-    // Load the non-transient, non-static fields
-    render(value, instance, clazz, context);
-    
-    return instance;
-        	}
- });
-----
-
-
-
-=== Security Policy Requirements for DefaultValueCoder
-
-DefaultValueCoder performs security-sensitive operations: (a) it reads and writes data from and to private fields using reflection, and (b) it overrides the default implementations of java.io.ObjectInputStream and java.io.ObjectOutputStream. If a SecurityManager is installed then three permissions must be granted to enable the new mechanism:
-
-----
-java.lang.RuntimePermission "accessDeclaredMembers";
-java.lang.reflect.ReflectPermission("suppressAccessChecks")
-java.io.SerializablePermission("enableSubclassImplementation")
-----
-
-See <<Security>> for an extended discussion on security policy issues for Persistit.
-
-== SerialValueCoder
-
-+SerialValueCoder+ uses standard Java serialization to store and retrieve object values. Typically this results in slower performance and a more verbose storage format than +DefaultValueCoder+, but there are a number of reasons why a particular application might require standard Java serialization, including:
-
-- the security context into which the application will be deployed does not grant the permissions noted above that are required for +DefaultValueCoder+,
-- to avoid Persistit's use of private API calls to construct object instances during deserialization,
-- a preference for the use of a standard format defined within the Java platform rather than Persistit's custom format,
-- limitations documented above on the API elements available during custom deserialization within DefaultValueCoder, for example non-support of GetField and PutField.
-
-Your application can specify +SerialValueCoders+ for specific classes either by explicitly creating and registering them, or by naming them in the com.persistit.serialOverride property.
-
-To explicitly register a +SerialValueCoder+ for the class +MyClass+, do this:
-
-[source,java]
-----
-	...
-	Persistit.getInstance().getCoderManager().registerValueCoder(
-    	MyClass.class,
-    	new SerialValueCoder(MyClass.class));
-	...
-----
-
-
-=== The +com.persistit.serialOverride+ Configuration Property
-
-The +serialOverride+ property specifies classes that are to be serialized by +SerialValueCoder+ rather than +DefaultValueCoder+. This property affects how Persistit assigns a value coder when none has previously been registered. It does not override or affect explicitly registered coders.
-
-Names are separated by commas and may contain wild cards.
-
-The following are valid patterns:
-
-+java.io.File+:: Just the File class.
-+java.io.*+:: All classes in the java.io package.
-+java.awt.**+:: All classes in the java.awt package and its sub-packages
-+java.util.*Map+:: All of the Map classes in the java.util.
-+**+:: All classes in all packages
-
-More precisely, +serialOverride+ specifies a comma-delimited list of zero or more patterns, each of which is either a fully-qualified class name or pattern that has within it exactly one wild card. The wild card “\*” replaces any sequence of characters other than a period (“.”), while “\*\*” replaces any sequence of characters including periods.  For example:
-
-----
-+serialOverride=org.apache.**,com.mypkg.serialstuff.*,com.mypkg.MyClass+
-----
-
-Like all configuration properties, you may specify this in the persistit.properties file or as a system property through a Java command-line argument in the form:
-
-----
-+-Dcom.persistit.serialOverride=...+
-----
-
-== Registering Objects in a Custom +ValueCoder+
-
-In a custom +ValueCoder+ implementation, the +get+ method is responsible for constructing and populating an instance of an object. The following pattern should be used when implementing the get method:
-
-[source,java]
-----
-public void get(Value value, Class clazz, CoderContext context) throws ConversionException {
-    	// Construct the object being deserialized.
-    	//
-    	Object instance = ...constructor for the object...
-
-    	// Associate a handle with the newly
-    	// created instance.
-    	//
-    	value.registerEncodedObject(instance);
-
-    	// Populate the object's internal state
-    	//
-    	... load the fields – for example, by calling render...
-
-    	return instance;
-}
-----
-
-The purpose of the +registerEncodedObject+ method is to record the association between the newly created object and an internal integer-valued handle that may be used subsequently in the serialization stream to refer to that object. This mechanism supports objects that may have fields that refer either indirectly or indirectly back to the same object – i.e., that participate in a cyclical reference graph.
-
-As a concrete example, consider a Person class with a spouse field such that for married couple p and q,  p.spouse is q and q.spouse is p. When Persistit serializes p it also serializes q, but when it serializes q's spouse field, it records a reference handle associated with the already-serialized instance of p rather than writing a new copy of p in the serialization stream. Upon deserializing q, Persistit looks up the object for the recorded handle to correctly associate the already-deserialized p instance with q.
-
-Whenever you implement a custom +get()+ method in any +ValueCoder+, you must notify the underlying Value object about the newly created object by calling registerEncodedObject before deserializing its fields so that any back-references made within serialized fields of that object can find the object correctly.
-
-== +Value.toString()+ and +decodeDisplayable+
-
-In many cases it is not very useful simply to display the result of evaluating +toString()+ on an object. The default toString method inherited from Object conveys just a class name and a memory handle. In addition, for remote operations of AdminUI, it may not even be feasible to construct a deserialized object for each record. Therefore, +com.persistit.Value+ provides a specialized +toString()+ method to render an arbitrary object value into a legible string. The AdminUI utility uses this facility to summarize the data contained in a Tree.
-
-Persistit creates a String value loading the object's class, using the following algorithm:
-
-- If the state represented by this Value is undefined, then return "undefined".
-- If the state is null or a boolean, return "null" "false", or "true".
-- If the value represents a primitive type, return the string representation of the value, prefixed by "(byte)", "(short)", "(char)", "(long)", or "(float)" for the corresponding types. Values of type int and double are presented without prefix to reduce clutter.
-- If the value represents a String, return a modified form of the string enclosed in double quotes. For each character of the string, if it is a double quote replace it by "\"", otherwise if it is outside of the printable ASCII character set replace the character in the modified string by "\b", "\t", "\n", "\r" or "\uNNNN" such that the modified string would be a valid Java string constant.
-- If the value represents a wrapper for a primitive value (i.e., a java.lang.Boolean, java.lang.Byte, etc.) return the string representation of the value prefixed by "(Boolean)", "(Byte)", "(Short)", "(Character)", "(Integer)", "(Long)", "(Float)" or "(Double)".  The package name java.lang is removed to reduce clutter.
-- If the value represents a java.util.Date, return a formatted representation of the date using the format specified by Key.SDF. This is a readable format that displays the date with full precision, including milliseconds.
-- If the value represents an array, return a list of comma-separated element values surrounded by square brackets.
-- If the value represents one of the standard Collection implementations in the java.util package, then return a comma-separated list of values surrounded by square brackets.
-- If the value represents one of the standard Map implementations in the java.util package, then return a comma-separated list of key/value pairs surrounded by square brackets. Each key/value pair is represented by a string in the form key->value.
-- If the value represents an object of a class for which there is a registered com.persistit.encoding.ValueDisplayer, invoke the displayer's display method to format a displayable representation of the object.
-- If the value represents an object that has been stored using the version default serialization mechanism described above, return the class name of the object followed by a comma-separated tuple, enclosed within curly brace characters, representing the value of each field of the object.
-- If the value represents an object encoded through standard Java serialization, return the string "(Serialized-object)" followed by a sequence of hex digits representing the serialized bytes. Note that this process does not attempt to deserialize the object.
-- If the value represents an object that has already been represented within the formatted result - for example, if a Collection contains two references to the same object - then instead of creating an additional string representing the second or subsequent instance, emit a back reference pointer in the form @NNN where NNN is the character offset within the displayable string where the first instance was found. (This does not apply to strings and the primitive wrapper classes.)
-
-For example, consider a Person having for date of birth, first name, last name, salary and friends, an array of other Person objects. The result returned by toString() on a Value representing Mary Smith who has a friend John Smith, might appear as follows:
-
-----
- (Person){(Date)19490826000000.000-0400,"Mary","Jones",(long)75000,[
-	(Person){(Date)19550522000000.000-0400,"John","Smith",(long)68000,[@0]}]}
-----
-
-In this example, John Smith's friends array contains a back reference to Mary Jones in the form "@0" because Mary's displayable reference starts at the beginning of the string.
-
-
-== PersistitReference
-
-In general, serializing an object that contains references to other objects requires all the referenced objects also to be serialized. For an object connected to a large reference graph, it may be impractical or even semantically incorrect to serialize the entire graph.
-
-One way to control the serialization graph for such an object is to write a custom ValueCoder; the custom ValueCoder can store key values for looking up the referenced object, rather than the object itself.  The ValueCoderDemo.java program demonstrates how this can be done.
-
-The +com.persistit.ref.PersistitReference+ interface, and its abstract subclasses, provide an alternative mechanism for breaking up an object reference graph.  It requires no custom ValueCoder, but does impact the design of application classes.  In addition, you will need to write a concrete implementation of either com.persistit.ref.AbstractReference or com.persistit.ref.AbstractWeakReference based on the actual storage structure of your object graph.
-
-== ObjectCache
-
-A +com.persistit.Value+ object holds the serialized, encoded state of a primitive value of an object.  Each time you invoke the get method on a Value, Persistit generates a new copy of the object deserialized from this Value.  Persistit does not implicitly cache deserialized objects. However, the +com.persistit.encoding.ObjectCache+ class provides a simple mechanism for applications that need to maintain an in-memory cache of of objects from Persistit. +ObjectCache+ works somewhat like a specialized version of java.util.WeakHashMap.
-
-+ObjectCache+ has +put+, +get+ and +remove+ methods much like a normal Map implementation.  However, when storing an object value with the supplied +com.persistit.Key+, +ObjectCache+ constructs a new, immutable +com.persistit.KeyState+ object to hold as an internal key. This is necessary because +Key+ objects change value as they are used.
-
-Each +ObjectCache+ entry holds its object value as a +SoftReference+, making it available for garbage collection when space is needed.

=== removed file 'doc/TOC.txt'
--- doc/TOC.txt	2012-04-25 18:13:07 +0000
+++ doc/TOC.txt	1970-01-01 00:00:00 +0000
@@ -1,19 +0,0 @@
-Akiban Persistit User Guide
-===========================
-Peter Beaman <[e-mail]>
-:toc:
-:doctype: book
-:icons:
-:numbered:
-:website: http://www.akiban.com/persistit/
-
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
[e-mail]
-

=== added file 'doc/Transactions.rst'
--- doc/Transactions.rst	1970-01-01 00:00:00 +0000
+++ doc/Transactions.rst	2012-05-30 18:23:19 +0000
@@ -0,0 +1,200 @@
+.. _Transactions:
+
+Transactions
+============
+
+Akiban Persistit supports transactions with multi-version concurrency control (MVCC) using a protocol called Snapshot Isolation (SI). An application calls ``com.persistit.Transaction#begin``, ``com.persistit.Transaction#commit``, ``com.persistit.Transaction#rollback# and ``com.persistit.Transaction#end`` methods to control the current transaction scope explicitly.  A Transaction allows an application to execute multiple database operations in an atomic, consistent, isolated and durable (ACID) manner.
+
+Applications manage transactions through an instance of a ``com.persistit.Transaction`` object. ``Transaction`` does not represent a single transaction, but is instead a context in which a thread may perform many sequential transactions. The general pattern is that the application gets the current thread’s ``Transaction`` instance, calls its ``begin`` method, performs work, calls ``commit`` and finally ``end``.  The thread uses the same ``Transaction`` instance repeatedly. Generally each thread has one ``Transaction`` that lasts for the entire life of the thread (but see com.persistit.Transaction#_threadManagement for a mechanism that allows a transaction to be serviced by multiple threads). 
+
+Using Transactions
+------------------
+
+The following code fragment performs two store operations within the scope of a transaction:
+
+.. code-block:: java
+
+  //
+  // Get the transaction context for the current thread.
+  //
+  Transaction txn = myExchange.getTransaction();
+  int remainingRetries = RETRY_COUNT;
+  for (;;) {
+      txn.begin();
+      try {
+          myExchange.getValue().put("First value");
+          myExchange.clear().append(1).store();
+          myExchange.getValue().put("Second value");
+          myExchange.clear().append(2).store();
+          // Required to commit the transaction
+          txn.commit();
+          break;
+      } catch (RollbackException re) {
+          // perform any special rollback handling
+          // allow loop to repeat until commit succeeds or retries
+          // too many times.
+          if (--remainingRetries < 0) {
+            throw new TransactionFailedException();
+          }
+      } catch (PersistitException pe) {
+          // handle other Persistit exception
+      } finally {
+          // Required to end the scope of a transaction.
+          txn.end();
+      }
+  }
+
+This example catches ``com.persistit.exception.RollbackException`` which can be thrown by any Persistit operation within the scope of a transaction, including ``commit``. Any code explicitly running within the scope of a transaction should be designed to handle rollbacks.
+
+This example also uses a *try/finally* block to ensure every call to ``begin`` has a matching call to ``end``. This code pattern is mandatory: it is critical to correct transaction nesting behavior.
+
+One convenient way to do this is to encapsulate the logic of a transaction in an implementation of ``com.persisitit.TransactionRunnable`` interface. The ``com.persistit.Transaction#run`` method automatically provides logic to begin the transaction, execute the TransactionRunnable and commit the transaction, repeating the process until no rollback is thrown or a maximum retry count is reached. For example, the code fragment shown above can be rewritten as:
+
+.. code-block:: java
+
+  //
+  // Get the transaction context for the current thread.
+  //
+  Transaction txn = myExchange.getTransaction();
+  //
+  // Perform the transaction with the following parameters:
+  // - try to commit it up to 10 times
+  // - delay 2 milliseconds before each retry
+  // - use the group commit durability policy
+  //
+  txn.run(new TransactionRunnable() {
+      public void run() throws PersistitException {
+          myExchange.getValue().put("First value");
+          myExchange.clear().append(1).store();
+          myExchange.getValue().put("Second value");
+          myExchange.clear().append(2).store();
+      }
+  }, 10, 2, CommitPolicy.GROUP);
+
+Mixing Transactional and Non-Transactional Operations
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Database operations running outside the scope of an explicitly defined transaction are never subject to rollback and therefore do not require retry logic. However, such operations are also not guaranteed to be durable in the event of a system crash. Further, such operations are not isolated. Read operations performed outside of a transaction can read uncommitted updates, and updates performed outside of a transaction are visible within transactions. In other words, non-transactional reads and writes may break both the durability and isolation of concurrently executing transactions.  Therefore it is strongly recommended that in an application that relies on transactions, all interactions with the database should use transactions. 
+
+Optimistic Transaction Scheduling
+---------------------------------
+
+To achieve high performance and scalability, Persistit supports an optimistic transaction scheduling protocol called MVCC with http://wikipedia.org/wiki/Snapshot_isolation[Snapshot Isolation]. Under this protocol multiple threads are permitted to execute transactions at full speed without blocking until a potentially inconsistent state is recognized. At that point a transaction suspected of causing the inconsistent state is automatically forced to roll back.
+
+Optimistic scheduling works because transactions usually do not collide, especially when individual database operations are fast, and so in practice transactions are seldom rolled back. But because any transaction may be rolled back at any point, applications must be designed carefully to avoid unintended side-effects. For example, a transaction should never perform non-repeatable or externally visible operations such as file or network I/O within its scope.
+
+Snapshot Isolation
+^^^^^^^^^^^^^^^^^^
+
+Persistit schedules concurrently executing transactions optimistically, without locking any database records. Instead, Persistit uses the well-known Snapshot Isolation protocol to achieve atomicity and isolation. While transactions are modifying data, Persistit maintains multiple versions of values being modified. Each version is labeled with the commit timestamp of the transaction that modified it. Whenever a transaction reads a value that has been modified by other transactions, it gets the latest version that was committed before its own start timestamp. In other words, all read operations are performed as if from a "snapshot" of the state of the database made at the transaction's start timestamp - hence the name "Snapshot Isolation."
+
+.. _Pruning:
+
+Pruning 
+^^^^^^^
+
+Given that all updates written through transactions are created as versions within the MVCC scheme, a large number of versions can accumulate over time. Persistit reduces this proliferation through an activity called "pruning." Pruning resolves the final state of each version by removing any versions created by aborted transactions and removing obsolete versions no longer needed by other transactions. If a value contains only one version and the commit timestamp of the transaction that created it is before the start of any currently running transaction, that value is called *primordial*. The goal of pruning is to reduce most or all values in the database to their primordial states because updating and reading primordial values is more efficient than than managing multiple version values. Pruning happens automatically and is generally not visible to the application.
+
+Rollbacks
+^^^^^^^^^
+
+Usually Snapshot Isolation allows concurrent transactions to commit without interference but this is not always the case. Two concurrent transactions that attempt to modify the same Persistit key/value pair before they commit are said to have a "write-write dependency". To avoid anomalous results one of them must abort, rolling back any other updates it may also have created, and retry. Persistit implements a "first updater wins" policy in which if two transactions attempt to update the same record, the first transaction "wins" by being allowed to continue, while the second transaction "loses" and is required to abort.
+
+Once a transaction has aborted, any subsequent database operation it attempts throws a ``RollbackException``. Application code should catch and handle this Exception. Usually the correct and desired behavior is simply to retry the transaction as shown in the code samples above.
+
+A transaction can also voluntarily roll back. For example, transaction logic could detect an error condition that it chooses to handle by throwing an exception back to the application. In this case the transaction should invoke the ``rollback`` method to explicitly declare its intent to abort the transaction.
+
+Read-Only Transactions
+^^^^^^^^^^^^^^^^^^^^^^
+ 
+Under Snapshot Isolation, transactions that read but do not modify data cannot generate any write-write dependencies and are therefore not subject to  being rolled back because of the actions of other transactions. However, even though it modifies no data, a long-running read-only transaction can force Persistit to retain old value versions from other transactions for its duration in order to provide a snapshot view. This behavior can cause congestion and performance degradation by preventing very old values from being pruned. The degree to which this is a problem depends on the volume of update transactions being processed and the duration of long-running transactions.
+
+Snapshot Isolation is not Serializable
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+It is well-known that transactions executing under SI are not necessarily serializable. Under SI, so-called *write-skew* anomalies can happen with transactions that have certain kinds of interactions.  Write-skew can be avoided by (a) explicit application-level locking or (b) structuring transactions to add write-write dependencies where write-skew otherwise could occur.
+
+Note that many common transaction patterns, including those defined by the TPC-C benchmark, do not experience write-skew and therefore *are* serializable under SI.
+
+Durability Options: ``CommitPolicy``
+------------------------------------
+
+Persistit provides three policies that determine the durability of a transaction after it has executed the ``com.persistit.Transaction#commit`` method. These are:
+
+  ``HARD``
+      The ``commit`` method does not return until all updates created by the transaction have been written to non-volatile storage (e.g., disk storage).
+  ``GROUP``
+      The ``commit`` method does not return until all updates created by the transaction have been written to non-volatile storage. In addition, the committing 
+      transaction waits briefly in an attempt to recruit other transactions running in other threads to write their updates with the same physical I/O operation.
+  ``SOFT``
+      The ``commit`` method returns *before* the updates have been recorded on non-volatile storage. Persistit attempts to write them within 100 milliseconds, but 
+      this interval is not guaranteed.
+
+You can specify a default policy in the Persistit initialization properties using the ``txnpolicy`` property or under program control using ``com.persistit.Persistit#setDefaultTransactionCommitPolicy``. The default policy applies whenever the application calls the ``commit()`` method. You can override the default policy using ``commit(CommitPolicy)``.
+
+HARD and GROUP ensure each transaction is written durably to non-volatile storage before the ``commit`` method returns. The difference is that GROUP can improve throughput in multi-threaded applications because the average number of I/O operations needed to commit *N* transactions can be smaller than *N*. However, for one or a small number of concurrent threads, GROUP reduces throughput because it works by introducing a delay to allow other concurrent transactions to commit within a single I/O operation.
+
+SOFT commits are generally much faster than HARD or GROUP commits, especially for single-threaded applications, because the results of numerous transactions committed from a single thread can be aggregated and written to disk in a single I/O operation. However, transactions written with the SOFT commit policy are not immediately durable and it is possible that the recovered state of a database will be missing transactions that reported they were committed shortly before a crash.
+
+For SOFT commits, the state of the database after restart is such that for any committed transaction T, either all or none of its modifications will be present in the recovered database. Further, if a transaction T2 reads or updates data that was written by any other transaction T1, and if T2 is present in the recovered database, then so is T1. Any transaction that was in progress, but had not been committed at the time of the failure, is guaranteed not to be present in the recovered database. SOFT commits are designed to be durable within 100 milliseconds after ``commit`` returns. However, this interval is determined by computing the average duration of recent I/O operations to predict the completion time of the I/O that will write the transaction to disk, and therefore the interval cannot be guaranteed.
+
+Nested Transactions
+-------------------
+
+A nested transaction occurs when code that is already executing within the scope of a transaction executes the ``begin`` method to start a new transaction. This might happen, for example, if an application’s transaction logic calls a method that also uses transactions. In this case, the commit processing of the inner transaction scope is deferred until the outermost transaction commits. At that point, all the updates performed within the inner and outer transaction scopes are committed to the database. Similarly, a rollback initiated by the inner transaction causes both it and the outermost transaction to roll back.
+
+Accumulators
+------------
+
+Consider an application in which concurrently running transactions share a counter. For example, suppose each transaction is responsible for allocating a unique integer as a primary key for a database record. One way to do this would be to store the counter in a Persistit key/value pair, reading the value at the start of each transaction and committing an update at the end.
+
+The problem with this approach is that under SI, concurrent transactions running in a multi-threaded application would experience very frequent write-write dependencies on the counter value; in fact, the only way to complete any transactions would be serially, one at a time.
+
+Persistit provides the ``com.persistit.Accumulator`` class to avoid this problem.  An accumulator is designed to manage contributions from multiple concurrent transactions without causing write-write dependencies. Accumulators are durable in the sense that each transaction’s contribution is made durable with the transaction itself, and Persistit automatically recovers a correct state for each Accumulator in the event of a system crash.
+
+There are four types of accumulator in Persistit. Each a concrete subclass of the abstract ``com.persistit.Accumulator`` class:
+
+  ``SUM``
+      Tallies a count or sum of contributions by each transaction
+  ``MIN``
+      Finds the minimum value contributed by all transactions
+  ``MAX``
+      Finds the maximum value contributed by all transactions
+  ``SEQ``
+      Special case of the SUM accumulator used to generate sequence numbers
+
+Accumulator instances are associated with a ``com.persistit.Tree``.  Each ``Tree`` may have up to 64 accumulators. The following code fragment creates and/or acquires a ``SumAccumulator``, reads its snapshot value and then adds one to it:
+
+.. code-block:: java
+
+  final Exchange ex = _persistit.getExchange(volume, treeName, true);
+  final Transaction txn = ex.getTransaction();
+  txn.begin();
+  try {
+      final Accumulator acc =
+          ex.getTree().getAccumulator(Accumulator.Type.SUM, 17);
+      long snap = acc.getSnapshotValue(txn);
+      acc.update(1, txn);
+      txn.commit();
+  } finally {
+      txn.end();
+  }
+
+The value 17 is simply an arbitrary index number between 0 and 63, inclusive. The application is responsible for allocating and managing accumulator indexes.
+
+The snapshot value of an accumulator obtained through ``com.persistit.Accumulator#getSnapshotValue()`` is the value computed from all updates contributed by transactions that had committed at the time the current transaction started, plus the transaction’s own as-yet uncommitted updates. In other words, the snapshot value of the accumulator is consistent with the snapshot view of all other data visible within the transaction.
+
+An accumulator has two ways of accessing its accumulated value:
+
+  ``getSnapshotValue()``
+      Is a value computed from updates that were committed at the start of the current transaction. This method may be called only within the scope of a 
+      Transaction.
+  ``getLiveValue()``
+      Is an ephemeral value reflecting all updates performed by all transactions, including concurrent and aborted transactions.
+
+The snapshot value is a precise, consistent tally, while the live value is approximate. For a ``SumAccumulator``, ``MaxAccumulator`` or ``SeqAccumulator``, if all updates are have non-negative arguments, then the live value is always greater than or equal to the snapshot value.
+
+SeqAccumulator
+^^^^^^^^^^^^^^
+
+The ``SeqAccumulator`` class has a special role in allocating unique identifier numbers, e.g., synthetic primary keys.  The goal of the ``SeqAccumulator`` is to ensure that every committed transaction has received a unique value integer in all circumstances, including after recovery from a crash. See ``com.persistit.Accumulator`` for details.

=== removed file 'doc/Transactions.txt'
--- doc/Transactions.txt	2012-04-30 22:09:31 +0000
+++ doc/Transactions.txt	1970-01-01 00:00:00 +0000
@@ -1,179 +0,0 @@
-[[Transactions]]
-= Transactions
-
-Akiban Persistit supports transactions with multi-version concurrency control (MVCC) using a protocol called Snapshot Isolation (SI). An application calls +com.persistit.Transaction#begin+, +com.persistit.Transaction#commit+, +com.persistit.Transaction#rollback# and +com.persistit.Transaction#end+ methods to control the current transaction scope explicitly.  A Transaction allows an application to execute multiple database operations in an atomic, consistent, isolated and durable (ACID) manner.
-
-Applications manage transactions through an instance of a +com.persistit.Transaction+ object. +Transaction+ does not represent a single transaction, but is instead a context in which a thread may perform many sequential transactions. The general pattern is that the application gets the current thread’s +Transaction+ instance, calls its +begin+ method, performs work, calls +commit+ and finally +end+.  The thread uses the same +Transaction+ instance repeatedly. Generally each thread has one +Transaction+ that lasts for the entire life of the thread (but see com.persistit.Transaction#_threadManagement[Thread Management] for a mechanism that allows a transaction to be serviced by multiple threads). 
-
-== Using Transactions
-
-The following code fragment performs two store operations within the scope of a transaction:
-
-[source,java]
-----
-//
-// Get the transaction context for the current thread.
-//
-Transaction txn = myExchange.getTransaction();
-int remainingRetries = RETRY_COUNT;
-for (;;) {
-    txn.begin();
-    try {
-        myExchange.getValue().put("First value");
-        myExchange.clear().append(1).store();
-        myExchange.getValue().put("Second value");
-        myExchange.clear().append(2).store();
-        // Required to commit the transaction
-        txn.commit();
-        break;
-    } catch (RollbackException re) {
-        // perform any special rollback handling
-        // allow loop to repeat until commit succeeds or retries
-        // too many times.
-        if (--remainingRetries < 0) {
-            throw new TransactionFailedException();
-        }
-    } catch (PersistitException pe) {
-        // handle other Persistit exception
-    } finally {
-        // Required to end the scope of a transaction.
-        txn.end();
-    }
-}
-----
-
-This example catches +com.persistit.exception.RollbackException+ which can be thrown by any Persistit operation within the scope of a transaction, including +commit+. Any code explicitly running within the scope of a transaction should be designed to handle rollbacks.
-
-This example also uses a _try/finally_ block to ensure every call to +begin+ has a matching call to +end+. This code pattern is mandatory: it is critical to correct transaction nesting behavior.
-
-One convenient way to do this is to encapsulate the logic of a transaction in an implementation of +com.persisitit.TransactionRunnable+ interface. The +com.persistit.Transaction#run+ method automatically provides logic to begin the transaction, execute the TransactionRunnable and commit the transaction, repeating the process until no rollback is thrown or a maximum retry count is reached. For example, the code fragment shown above can be rewritten as:
-
-[source,java]
-----
-//
-// Get the transaction context for the current thread.
-//
-Transaction txn = myExchange.getTransaction();
-//
-// Perform the transaction with the following parameters:
-// - try to commit it up to 10 times
-// - delay 2 milliseconds before each retry
-// - use the group commit durability policy
-//
-txn.run(new TransactionRunnable() {
-    public void run() throws PersistitException {
-        myExchange.getValue().put("First value");
-        myExchange.clear().append(1).store();
-        myExchange.getValue().put("Second value");
-        myExchange.clear().append(2).store();
-    }
-}, 10, 2, CommitPolicy.GROUP);
-----
-
-=== Mixing Transactional and Non-Transactional Operations
-
-Database operations running outside the scope of an explicitly defined transaction are never subject to rollback and therefore do not require retry logic. However, such operations are also not guaranteed to be durable in the event of a system crash. Further, such operations are not isolated. Read operations performed outside of a transaction can read uncommitted updates, and updates performed outside of a transaction are visible within transactions. In other words, non-transactional reads and writes may break both the durability and isolation of concurrently executing transactions.  Therefore it is strongly recommended that in an application that relies on transactions, all interactions with the database should use transactions. 
-
-== Optimistic Transaction Scheduling
-
-To achieve high performance and scalability, Persistit supports an optimistic transaction scheduling protocol called MVCC with http://wikipedia.org/wiki/Snapshot_isolation[Snapshot Isolation]. Under this protocol multiple threads are permitted to execute transactions at full speed without blocking until a potentially inconsistent state is recognized. At that point a transaction suspected of causing the inconsistent state is automatically forced to roll back.
-
-Optimistic scheduling works because transactions usually do not collide, especially when individual database operations are fast, and so in practice transactions are seldom rolled back. But because any transaction may be rolled back at any point, applications must be designed carefully to avoid unintended side-effects. For example, a transaction should never perform non-repeatable or externally visible operations such as file or network I/O within its scope.
-
-=== Snapshot Isolation
-
-Persistit schedules concurrently executing transactions optimistically, without locking any database records. Instead, Persistit uses the well-known Snapshot Isolation protocol to achieve atomicity and isolation. While transactions are modifying data, Persistit maintains multiple versions of values being modified. Each version is labeled with the commit timestamp of the transaction that modified it. Whenever a transaction reads a value that has been modified by other transactions, it gets the latest version that was committed before its own start timestamp. In other words, all read operations are performed as if from a "snapshot" of the state of the database made at the transaction's start timestamp - hence the name "Snapshot Isolation."
-
-[[Pruning]]
-=== Pruning 
-
-Given that all updates written through transactions are created as versions within the MVCC scheme, a large number of versions can accumulate over time. Persistit reduces this proliferation through an activity called "pruning." Pruning resolves the final state of each version by removing any versions created by aborted transactions and removing obsolete versions no longer needed by other transactions. If a value contains only one version and the commit timestamp of the transaction that created it is before the start of any currently running transaction, that value is called _primordial_. The goal of pruning is to reduce most or all values in the database to their primordial states because updating and reading primordial values is more efficient than than managing multiple version values. Pruning happens automatically and is generally not visible to the application.
-
-=== Rollbacks
-
-Usually Snapshot Isolation allows concurrent transactions to commit without interference but this is not always the case. Two concurrent transactions that attempt to modify the same Persistit key/value pair before they commit are said to have a "write-write dependency". To avoid anomalous results one of them must abort, rolling back any other updates it may also have created, and retry. Persistit implements a "first updater wins" policy in which if two transactions attempt to update the same record, the first transaction "wins" by being allowed to continue, while the second transaction "loses" and is required to abort.
-
-Once a transaction has aborted, any subsequent database operation it attempts throws a +RollbackException+. Application code should catch and handle this Exception. Usually the correct and desired behavior is simply to retry the transaction as shown in the code samples above.
-
-A transaction can also voluntarily roll back. For example, transaction logic could detect an error condition that it chooses to handle by throwing an exception back to the application. In this case the transaction should invoke the +rollback+ method to explicitly declare its intent to abort the transaction.
-
-=== Read-Only Transactions
- 
-Under Snapshot Isolation, transactions that read but do not modify data cannot generate any write-write dependencies and are therefore not subject to  being rolled back because of the actions of other transactions. However, even though it modifies no data, a long-running read-only transaction can force Persistit to retain old value versions from other transactions for its duration in order to provide a snapshot view. This behavior can cause congestion and performance degradation by preventing very old values from being pruned. The degree to which this is a problem depends on the volume of update transactions being processed and the duration of long-running transactions.
-
-=== Snapshot Isolation is not Serializable
-
-It is well-known that transactions executing under SI are not necessarily serializable. Under SI, so-called _write-skew_ anomalies can happen with transactions that have certain kinds of interactions.  Write-skew can be avoided by (a) explicit application-level locking or (b) structuring transactions to add write-write dependencies where write-skew otherwise could occur.
-
-Note that many common transaction patterns, including those defined by the TPC-C benchmark, do not experience write-skew and therefore _are_ serializable under SI.
-
-== Durability Options: +CommitPolicy+
-
-Persistit provides three policies that determine the durability of a transaction after it has executed the +com.persistit.Transaction#commit+ method. These are:
-
-[horizontal]
-+HARD+:: The +commit+ method does not return until all updates created by the transaction have been written to non-volatile storage (e.g., disk storage).
-+GROUP+:: The +commit+ method does not return until all updates created by the transaction have been written to non-volatile storage. In addition, the committing transaction waits briefly in an attempt to recruit other transactions running in other threads to write their updates with the same physical I/O operation.
-+SOFT+:: The +commit+ method returns _before_ the updates have been recorded on non-volatile storage. Persistit attempts to write them within 100 milliseconds, but this interval is not guaranteed.
-
-You can specify a default policy in the Persistit initialization properties using the +txnpolicy+ property or under program control using +com.persistit.Persistit#setDefaultTransactionCommitPolicy+. The default policy applies whenever the application calls the +commit()+ method. You can override the default policy using +commit(CommitPolicy)+.
-
-HARD and GROUP ensure each transaction is written durably to non-volatile storage before the +commit+ method returns. The difference is that GROUP can improve throughput in multi-threaded applications because the average number of I/O operations needed to commit _N_ transactions can be smaller than _N_. However, for one or a small number of concurrent threads, GROUP reduces throughput because it works by introducing a delay to allow other concurrent transactions to commit within a single I/O operation.
-
-SOFT commits are generally much faster than HARD or GROUP commits, especially for single-threaded applications, because the results of numerous transactions committed from a single thread can be aggregated and written to disk in a single I/O operation. However, transactions written with the SOFT commit policy are not immediately durable and it is possible that the recovered state of a database will be missing transactions that reported they were committed shortly before a crash.
-
-For SOFT commits, the state of the database after restart is such that for any committed transaction T, either all or none of its modifications will be present in the recovered database. Further, if a transaction T2 reads or updates data that was written by any other transaction T1, and if T2 is present in the recovered database, then so is T1. Any transaction that was in progress, but had not been committed at the time of the failure, is guaranteed not to be present in the recovered database. SOFT commits are designed to be durable within 100 milliseconds after +commit+ returns. However, this interval is determined by computing the average duration of recent I/O operations to predict the completion time of the I/O that will write the transaction to disk, and therefore the interval cannot be guaranteed.
-
-== Nested Transactions
-
-A nested transaction occurs when code that is already executing within the scope of a transaction executes the +begin+ method to start a new transaction. This might happen, for example, if an application’s transaction logic calls a method that also uses transactions. In this case, the commit processing of the inner transaction scope is deferred until the outermost transaction commits. At that point, all the updates performed within the inner and outer transaction scopes are committed to the database. Similarly, a rollback initiated by the inner transaction causes both it and the outermost transaction to roll back.
-
-== Accumulators
-
-Consider an application in which concurrently running transactions share a counter. For example, suppose each transaction is responsible for allocating a unique integer as a primary key for a database record. One way to do this would be to store the counter in a Persistit key/value pair, reading the value at the start of each transaction and committing an update at the end.
-
-The problem with this approach is that under SI, concurrent transactions running in a multi-threaded application would experience very frequent write-write dependencies on the counter value; in fact, the only way to complete any transactions would be serially, one at a time.
-
-Persistit provides the +com.persistit.Accumulator+ class to avoid this problem.  An accumulator is designed to manage contributions from multiple concurrent transactions without causing write-write dependencies. Accumulators are durable in the sense that each transaction’s contribution is made durable with the transaction itself, and Persistit automatically recovers a correct state for each Accumulator in the event of a system crash.
-
-There are four types of accumulator in Persistit. Each a concrete subclass of the abstract +com.persistit.Accumulator+ class:
-
-[horizontal]
-+SUM+:: Tallies a count or sum of contributions by each transaction
-+MIN+:: Finds the minimum value contributed by all transactions
-+MAX+:: Finds the maximum value contributed by all transactions
-+SEQ+:: Special case of the SUM accumulator used to generate sequence numbers
-
-Accumulator instances are associated with a +com.persistit.Tree+.  Each +Tree+ may have up to 64 accumulators. The following code fragment creates and/or acquires a +SumAccumulator+, reads its snapshot value and then adds one to it:
-
-[source,java]
-----
-final Exchange ex = _persistit.getExchange(volume, treeName, true);
-final Transaction txn = ex.getTransaction();
-txn.begin();
-try {
-    final Accumulator acc =
-        ex.getTree().getAccumulator(Accumulator.Type.SUM, 17);
-    long snap = acc.getSnapshotValue(txn);
-    acc.update(1, txn);
-    txn.commit();
-} finally {
-    txn.end();
-}
-----
-
-The value 17 is simply an arbitrary index number between 0 and 63, inclusive. The application is responsible for allocating and managing accumulator indexes.
-
-The snapshot value of an accumulator obtained through +com.persistit.Accumulator#getSnapshotValue()+ is the value computed from all updates contributed by transactions that had committed at the time the current transaction started, plus the transaction’s own as-yet uncommitted updates. In other words, the snapshot value of the accumulator is consistent with the snapshot view of all other data visible within the transaction.
-
-An accumulator has two ways of accessing its accumulated value:
-
-[horizontal]
-+getSnapshotValue()+:: Is a value computed from updates that were committed at the start of the current transaction. This method may be called only within the scope of a Transaction.
-+getLiveValue()+:: Is an ephemeral value reflecting all updates performed by all transactions, including concurrent and aborted transactions.
-
-The snapshot value is a precise, consistent tally, while the live value is approximate. For a +SumAccumulator+, +MaxAccumulator+ or +SeqAccumulator+, if all updates are have non-negative arguments, then the live value is always greater than or equal to the snapshot value.
-
-=== SeqAccumulator
-
-The +SeqAccumulator+ class has a special role in allocating unique identifier numbers, e.g., synthetic primary keys.  The goal of the +SeqAccumulator+ is to ensure that every committed transaction has received a unique value integer in all circumstances, including after recovery from a crash. See +com.persistit.Accumulator+ for details.

=== modified file 'doc/build/build-doc.sh'
--- doc/build/build-doc.sh	2012-05-25 18:50:59 +0000
+++ doc/build/build-doc.sh	2012-05-30 18:23:19 +0000
@@ -37,12 +37,29 @@
 # The end-product files, user_guide.html and user_guide.xml are written
 # there.
 #
-rm -rf /tmp/akiban-persistit-doc
-mkdir /tmp/akiban-persistit-doc
-javac -d /tmp/akiban-persistit-doc -cp ../../core/target/classes/ src/*.java
-java -cp /tmp/akiban-persistit-doc:../../core/target/classes AsciiDocPrep in=../TOC.txt out=/tmp/akiban-persistit-doc/doc.txt base=apidocs index=../../core/target/site/apidocs/index-all.html
-asciidoc -a toc -n -d book -b xhtml11 -o /tmp/akiban-persistit-doc/doc.html /tmp/akiban-persistit-doc/doc.txt
-asciidoc -a toc -n -d book -b docbook -o /tmp/akiban-persistit-doc/doc.xml /tmp/akiban-persistit-doc/doc.txt
-sed s/\`/\ / /tmp/akiban-persistit-doc/doc.html > /tmp/akiban-persistit-doc/user_guide.html
-sed s/\`/\ / /tmp/akiban-persistit-doc/doc.xml > /tmp/akiban-persistit-doc/user_guide.xml
+rm -rf ../../target/sphinx/source
+mkdir -p ../../target/sphinx/source
+mkdir -p ../../target/sphinx/classes
+mkdir -p ../../target/sphinx/html
+mkdir -p ../../target/sphinx/text
+
+cp ../index.rst ../../target/sphinx/source
+cp ../conf.py ../../target/sphinx/source
+
+javac -d ../../target/sphinx/classes -cp ../../target/classes/ src/*.java
+
+java -cp ../../target/sphinx/classes:../../target/classes SphinxDocPrep in=../ReleaseNotes.rst out=../../target/sphinx/source/ReleaseNotes.rst base=http://www.akiban.com/ak-docs/admin/persistit/apidocs index=../../target/site/apidocs/index-all.html
+java -cp ../../target/sphinx/classes:../../target/classes SphinxDocPrep in=../BasicAPI.rst out=../../target/sphinx/source/BasicAPI.rst base=http://www.akiban.com/ak-docs/admin/persistit/apidocs index=../../target/site/apidocs/index-all.html
+java -cp ../../target/sphinx/classes:../../target/classes SphinxDocPrep in=../Configuration.rst out=../../target/sphinx/source/Configuration.rst base=http://www.akiban.com/ak-docs/admin/persistit/apidocs index=../../target/site/apidocs/index-all.html
+java -cp ../../target/sphinx/classes:../../target/classes SphinxDocPrep in=../GettingStarted.rst out=../../target/sphinx/source/GettingStarted.rst base=http://www.akiban.com/ak-docs/admin/persistit/apidocs index=../../target/site/apidocs/index-all.html
+java -cp ../../target/sphinx/classes:../../target/classes SphinxDocPrep in=../Management.rst out=../../target/sphinx/source/Management.rst base=http://www.akiban.com/ak-docs/admin/persistit/apidocs index=../../target/site/apidocs/index-all.html
+java -cp ../../target/sphinx/classes:../../target/classes SphinxDocPrep in=../Miscellaneous.rst out=../../target/sphinx/source/Miscellaneous.rst base=http://www.akiban.com/ak-docs/admin/persistit/apidocs index=../../target/site/apidocs/index-all.html
+java -cp ../../target/sphinx/classes:../../target/classes SphinxDocPrep in=../PhysicalStorage.rst out=../../target/sphinx/source/PhysicalStorage.rst base=http://www.akiban.com/ak-docs/admin/persistit/apidocs index=../../target/site/apidocs/index-all.html
+java -cp ../../target/sphinx/classes:../../target/classes SphinxDocPrep in=../Security.rst out=../../target/sphinx/source/Security.rst base=http://www.akiban.com/ak-docs/admin/persistit/apidocs index=../../target/site/apidocs/index-all.html
+java -cp ../../target/sphinx/classes:../../target/classes SphinxDocPrep in=../Serialization.rst out=../../target/sphinx/source/Serialization.rst base=http://www.akiban.com/ak-docs/admin/persistit/apidocs index=../../target/site/apidocs/index-all.html
+java -cp ../../target/sphinx/classes:../../target/classes SphinxDocPrep in=../Transactions.rst out=../../target/sphinx/source/Transactions.rst base=http://www.akiban.com/ak-docs/admin/persistit/apidocs index=../../target/site/apidocs/index-all.html
+
+sphinx-build -a  ../../target/sphinx/source ../../target/sphinx/html
+
+fold -s ../../target/sphinx/source/ReleaseNotes.rst | sed 's/``//g' | sed 's/\.\. note:/NOTE/' | sed 's/::/:/' > ../../target/sphinx/text/ReleaseNotes
 

=== added file 'doc/build/build-doc.sh.orig'
--- doc/build/build-doc.sh.orig	1970-01-01 00:00:00 +0000
+++ doc/build/build-doc.sh.orig	2012-05-30 18:23:19 +0000
@@ -0,0 +1,48 @@
+#/bin/sh
+#
+# Copyright © 2011-2012 Akiban Technologies, Inc.  All rights reserved.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, version 3 (only) of the
+# License.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+# 
+# This program may also be available under different license terms. For more
+# information, see www.akiban.com or contact [e-mail].
+# 
+
+# ---------------------
+#
+# Builds the Akiban Persistit doc set.  Currently this process is based on
+# the asciidoc tool (http://www.methods.co.nz/asciidoc/).
+#
+# Here are the steps:
+# 1. Run a Java program AsciiDocPrep to prepare a text asciidoc file.
+#    Among other things, AsciiDocPrep fills in JavaDoc hyperlinks.
+# 2. Run asciidoc to generate an html file.
+# 3. Use sed to replace some characters.  Turns out asciidoc doesn't like
+#    to link to URLs having spaces, so AsciDocPrep replaces those spaces
+#    with the "`" character.  This step converts those back to spaces.
+#
+# Run this script from the root of the persistit source directory. This
+# script writes changes only into a directory /tmp/akiban-persistit-doc.
+# The end-product files, user_guide.html and user_guide.xml are written
+# there.
+#
+rm -rf /tmp/akiban-persistit-doc
+mkdir /tmp/akiban-persistit-doc
+javac -d /tmp/akiban-persistit-doc -cp ../../core/target/classes/ src/*.java
+java -cp /tmp/akiban-persistit-doc:../../core/target/classes AsciiDocPrep in=../TOC.txt out=/tmp/akiban-persistit-doc/doc.txt base=apidocs index=../../core/target/site/apidocs/index-all.html
+asciidoc -a toc -n -d book -b xhtml11 -o /tmp/akiban-persistit-doc/doc.html /tmp/akiban-persistit-doc/doc.txt
+asciidoc -a toc -n -d book -b docbook -o /tmp/akiban-persistit-doc/doc.xml /tmp/akiban-persistit-doc/doc.txt
+sed s/\`/\ / /tmp/akiban-persistit-doc/doc.html > /tmp/akiban-persistit-doc/user_guide.html
+sed s/\`/\ / /tmp/akiban-persistit-doc/doc.xml > /tmp/akiban-persistit-doc/user_guide.xml
+

=== added file 'doc/build/src/SphinxDocPrep.java'
--- doc/build/src/SphinxDocPrep.java	1970-01-01 00:00:00 +0000
+++ doc/build/src/SphinxDocPrep.java	2012-05-30 18:23:19 +0000
@@ -0,0 +1,147 @@
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.FileWriter;
+import java.io.PrintWriter;
+import java.util.SortedMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import com.persistit.util.ArgParser;
+
+public class SphinxDocPrep {
+
+    private final static Pattern PERSISTIT_PATTERN = Pattern
+            .compile("(``)?(com\\.persistit(?:\\.[a-z]\\w*)*(?:\\.[A-Z]\\w*)+)(?:#(\\w+(?:[\\(\\)\\,a-zA-Z]*)))?(``)?");
+
+    private final static String[] ARG_TEMPLATE = { "in|string:|Input file", "out|string:|Output file",
+            "index|string:|Pathname of index-all.html file", "base|string:|Base of generated URLs", };
+
+    enum BlockState {
+        OUT, WAIT_FIRST_BLANK_LINE, WAIT_SECOND_BLANK_LINE
+    }
+
+    private AsciiDocIndex index;
+    private BlockState block = BlockState.OUT;
+    private PrintWriter writer;
+    private String base;
+    private String indexPath;
+
+    private void prepare(final String[] args) throws Exception {
+        ArgParser ap = new ArgParser("SphinxDocPrep", args, ARG_TEMPLATE);
+        final String inPath = ap.getStringValue("in");
+        final String outPath = ap.getStringValue("out");
+
+        writer = outPath.isEmpty() ? new PrintWriter(System.out) : new PrintWriter(new FileWriter(outPath));
+
+        base = ap.getStringValue("base");
+        if (base.isEmpty()) {
+            base = "http://akiban.com/persistit/doc/apidocs";
+        }
+        indexPath = ap.getStringValue("index");
+        if (indexPath.isEmpty()) {
+            indexPath = "/home/peter/website/apidocs/index-all.html";
+        }
+
+        index = new AsciiDocIndex();
+        System.out.print("Building JavaDoc index..");
+        index.buildIndex(indexPath, base);
+        System.out.println("done");
+
+        processFile(new File(inPath), 0);
+        writer.close();
+    }
+
+    public void processFile(final File file, final int level) throws Exception {
+        BufferedReader reader = new BufferedReader(new FileReader(file));
+        System.out.print("Processing file " + file);
+        String line;
+        while ((line = reader.readLine()) != null) {
+            if (line.startsWith("@")) {
+                processFile(new File(file.getParentFile(), line.substring(1)), level + 1);
+            } else {
+                processLine(line);
+            }
+        }
+        writer.println();
+        System.out.println(" - done");
+    }
+
+    private void processLine(final String line) throws Exception {
+        if (line.contains(".. code-block:")) {
+            block = BlockState.WAIT_FIRST_BLANK_LINE;
+        } else if (line.isEmpty()) {
+            switch (block) {
+            case WAIT_FIRST_BLANK_LINE:
+                block = BlockState.WAIT_SECOND_BLANK_LINE;
+                break;
+
+            case WAIT_SECOND_BLANK_LINE:
+                block = BlockState.OUT;
+                break;
+
+            default:
+                // no change
+            }
+        }
+
+        if (block == BlockState.OUT) {
+            final StringBuffer sb = new StringBuffer();
+            if (line.startsWith("=")) {
+                sb.append("=");
+            }
+            final Matcher matcher = PERSISTIT_PATTERN.matcher(line);
+            while (matcher.find()) {
+                processMatch(matcher, sb);
+            }
+            matcher.appendTail(sb);
+            writer.println(sb.toString());
+        } else {
+            writer.println(line);
+            writer.flush();
+        }
+    }
+
+    private void processMatch(final Matcher matcher, final StringBuffer sb) {
+        String className = matcher.group(2);
+        String methodName = matcher.group(3);
+
+        String replacement;
+        if (methodName == null) {
+            String url = index.getClassMap().get(className);
+            if (url == null || url.isEmpty()) {
+                replacement = "<<<Missing class: " + className + ">>>";
+            } else {
+                replacement = "`" + className + " <" + url + ">`_";
+            }
+        } else {
+            String from = className + "#" + methodName.split("\\(")[0];
+            final SortedMap<String, String> map = index.getMethodMap().tailMap(from);
+            String url;
+            if (map.isEmpty()) {
+                replacement = "<<<Missing method: " + methodName + ">>>";
+            } else {
+                final String first = map.firstKey();
+                url = map.get(first);
+                url = url.replace(" ", "%20");
+                String text = first.split("#")[1];
+                text = text.replace("com.persistit.encoding.", "");
+                text = text.replace("com.persistit.exception.", "");
+                text = text.replace("com.persistit.logging.", "");
+                text = text.replace("com.persistit.mxbeans.", "");
+                text = text.replace("com.persistit.ref.", "");
+                text = text.replace("com.persistit.ui.", "");
+                text = text.replace("com.persistit.", "");
+                text = text.replace("java.lang.", "");
+                text = text.replace("java.util.", "");
+                replacement= "`" + text + " <" + url + ">`_";
+            }
+        }
+
+        matcher.appendReplacement(sb, Matcher.quoteReplacement(replacement));
+    }
+
+    public static void main(final String[] args) throws Exception {
+        new SphinxDocPrep().prepare(args);
+    }
+}

=== added file 'doc/conf.py'
--- doc/conf.py	1970-01-01 00:00:00 +0000
+++ doc/conf.py	2012-05-30 18:23:19 +0000
@@ -0,0 +1,285 @@
+# -*- coding: utf-8 -*-
+#
+# PersistitDoc documentation build configuration file, created by
+# sphinx-quickstart on Fri May 18 15:19:04 2012.
+#
+# This file is execfile()d with the current directory set to its containing dir.
+#
+# Note that not all possible configuration values are present in this
+# autogenerated file.
+#
+# All configuration values have a default; values that are commented out
+# serve to show the default.
+
+import sys, os
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#sys.path.insert(0, os.path.abspath('.'))
+
+# -- General configuration -----------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be extensions
+# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
+extensions = ['sphinx.ext.todo']
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix of source filenames.
+source_suffix = '.rst'
+
+# The encoding of source files.
+#source_encoding = 'utf-8-sig'
+
+# The master toctree document.
+master_doc = 'index'
+
+# General information about the project.
+project = u'PersistitDoc'
+copyright = u'2012, Akiban Technologies'
+
+# The version info for the project you're documenting, acts as replacement for
+# |version| and |release|, also used in various other places throughout the
+# built documents.
+#
+# The short X.Y version.
+version = '1'
+# The full version, including alpha/beta/rc tags.
+release = '1'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#language = None
+
+# There are two options for replacing |today|: either, you set today to some
+# non-false value, then it is used:
+#today = ''
+# Else, today_fmt is used as the format for a strftime call.
+#today_fmt = '%B %d, %Y'
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+exclude_patterns = []
+
+# The reST default role (used for this markup: `text`) to use for all documents.
+#default_role = None
+
+# If true, '()' will be appended to :func: etc. cross-reference text.
+#add_function_parentheses = True
+
+# If true, the current module name will be prepended to all description
+# unit titles (such as .. function::).
+#add_module_names = True
+
+# If true, sectionauthor and moduleauthor directives will be shown in the
+# output. They are ignored by default.
+#show_authors = False
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+# A list of ignored prefixes for module index sorting.
+#modindex_common_prefix = []
+
+
+# -- Options for HTML output ---------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+html_theme = 'default'
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#html_theme_options = {}
+
+# Add any paths that contain custom themes here, relative to this directory.
+#html_theme_path = []
+
+# The name for this set of Sphinx documents.  If None, it defaults to
+# "<project> v<release> documentation".
+#html_title = None
+
+# A shorter title for the navigation bar.  Default is the same as html_title.
+#html_short_title = None
+
+# The name of an image file (relative to this directory) to place at the top
+# of the sidebar.
+#html_logo = None
+
+# The name of an image file (within the static path) to use as favicon of the
+# docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
+# pixels large.
+#html_favicon = None
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
+# using the given strftime format.
+#html_last_updated_fmt = '%b %d, %Y'
+
+# If true, SmartyPants will be used to convert quotes and dashes to
+# typographically correct entities.
+#html_use_smartypants = True
+
+# Custom sidebar templates, maps document names to template names.
+#html_sidebars = {}
+
+# Additional templates that should be rendered to pages, maps page names to
+# template names.
+#html_additional_pages = {}
+
+# If false, no module index is generated.
+#html_domain_indices = True
+
+# If false, no index is generated.
+#html_use_index = True
+
+# If true, the index is split into individual pages for each letter.
+#html_split_index = False
+
+# If true, links to the reST sources are added to the pages.
+#html_show_sourcelink = True
+
+# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
+#html_show_sphinx = True
+
+# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
+#html_show_copyright = True
+
+# If true, an OpenSearch description file will be output, and all pages will
+# contain a <link> tag referring to it.  The value of this option must be the
+# base URL from which the finished HTML is served.
+#html_use_opensearch = ''
+
+# This is the file name suffix for HTML files (e.g. ".xhtml").
+#html_file_suffix = None
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'PersistitDocdoc'
+
+
+# -- Options for LaTeX output --------------------------------------------------
+
+latex_elements = {
+# The paper size ('letterpaper' or 'a4paper').
+#'papersize': 'letterpaper',
+
+# The font size ('10pt', '11pt' or '12pt').
+#'pointsize': '10pt',
+
+# Additional stuff for the LaTeX preamble.
+#'preamble': '',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title, author, documentclass [howto/manual]).
+latex_documents = [
+  ('index', 'PersistitDoc.tex', u'PersistitDoc Documentation',
+   u'Akiban Technologies', 'manual'),
+]
+
+# The name of an image file (relative to this directory) to place at the top of
+# the title page.
+#latex_logo = None
+
+# For "manual" documents, if this is true, then toplevel headings are parts,
+# not chapters.
+#latex_use_parts = False
+
+# If true, show page references after internal links.
+#latex_show_pagerefs = False
+
+# If true, show URL addresses after external links.
+#latex_show_urls = False
+
+# Documents to append as an appendix to all manuals.
+#latex_appendices = []
+
+# If false, no module index is generated.
+#latex_domain_indices = True
+
+
+# -- Options for manual page output --------------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    ('index', 'persistitdoc', u'PersistitDoc Documentation',
+     [u'Akiban Technologies'], 1)
+]
+
+# If true, show URL addresses after external links.
+#man_show_urls = False
+
+
+# -- Options for Texinfo output ------------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+  ('index', 'PersistitDoc', u'PersistitDoc Documentation',
+   u'Akiban Technologies', 'PersistitDoc', 'One line description of project.',
+   'Miscellaneous'),
+]
+
+# Documents to append as an appendix to all manuals.
+#texinfo_appendices = []
+
+# If false, no module index is generated.
+#texinfo_domain_indices = True
+
+# How to display URL addresses: 'footnote', 'no', or 'inline'.
+#texinfo_show_urls = 'footnote'
+
+
+# -- Options for Epub output ---------------------------------------------------
+
+# Bibliographic Dublin Core info.
+epub_title = u'PersistitDoc'
+epub_author = u'Akiban Technologies'
+epub_publisher = u'Akiban Technologies'
+epub_copyright = u'2012, Akiban Technologies'
+
+# The language of the text. It defaults to the language option
+# or en if the language is not set.
+#epub_language = ''
+
+# The scheme of the identifier. Typical schemes are ISBN or URL.
+#epub_scheme = ''
+
+# The unique identifier of the text. This can be a ISBN number
+# or the project homepage.
+#epub_identifier = ''
+
+# A unique identification for the text.
+#epub_uid = ''
+
+# A tuple containing the cover image and cover page html template filenames.
+#epub_cover = ()
+
+# HTML files that should be inserted before the pages created by sphinx.
+# The format is a list of tuples containing the path and title.
+#epub_pre_files = []
+
+# HTML files shat should be inserted after the pages created by sphinx.
+# The format is a list of tuples containing the path and title.
+#epub_post_files = []
+
+# A list of files that should not be packed into the epub file.
+#epub_exclude_files = []
+
+# The depth of the table of contents in toc.ncx.
+#epub_tocdepth = 3
+
+# Allow duplicate toc entries.
+#epub_tocdup = True

=== added file 'doc/index.rst'
--- doc/index.rst	1970-01-01 00:00:00 +0000
+++ doc/index.rst	2012-05-30 18:23:19 +0000
@@ -0,0 +1,15 @@
+Akiban Persistit User Guide
+===========================
+.. toctree::
+   :maxdepth: 1
+
+   GettingStarted
+   BasicAPI
+   Transactions
+   PhysicalStorage
+   Configuration
+   Management
+   Security
+   Serialization
+   Miscellaneous
+

=== removed file 'doc/overview-summary.txt'
--- doc/overview-summary.txt	2011-04-26 02:10:48 +0000
+++ doc/overview-summary.txt	1970-01-01 00:00:00 +0000
@@ -1,107 +0,0 @@
-
-Persistit 2.2
-=============
-
-Persistit(TM) is a small, lightweight Java(TM) library that provides simple, fast and reliable data persistence for Java applications. It is designed to be embedded in Java application programs and to operate free of administration by the end-user.
-
-This section provides a brief overview.  See http://com.akiban.com/persistit/documentation/ for complete documentation.
-
-== API Overview
-
-Persistit stores data as key-value pairs in highly optimized B-Tree structures. Much like a Java Map implementation, Persistit associates at most one value with each unique instance of a key.
-
-Persistit provides interfaces to access and modify keys and their associated values. The developer writes code to construct key and value instances and to store, fetch, traverse and remove keys and records to and from the database. Persistit permits efficient multi-threaded concurrent access to database volumes. It is designed to minimize contention for critical resources and to maximize throughput on multi-processor machines.
-
-In addition to low-level access methods on keys and values, Persistit provides com.persistit.PersistitMap, which implements the java.util.SortedMap interface. PersistitMap uses the Persistit database as a backing store so that key/value pairs are persistent, potentially shared with all threads, and limited in number only by disk storage. (See PersistitMap.)
-
-Within Persistit, key values are _segmented_ and _ordered_. Segmented means that you can append multiple primitive values or Strings to construct a concatenated key. Ordered means that the methods that enumerate key values within a Persistit database do so in a specified natural order. (See Keys).
-
-A Persistit value may be any primitive value, any Serializable Java object, or an object of any class supported by a custom serialization helper class. When stored in the B-Tree, keys and values are represented by sequences of bytes. The byte sequence that represents a value may be of arbitrary length, bounded only by available heap memory. (See Values.)
-
-=== Access Methods
-
-The primary low-level interface for interacting with Persistit is +com.persistit.Exchange+. The Exchange class provides all methods for storing, deleting, fetching and traversing key/value pairs. These methods are summarized here and described in detail in the API documentation.
-
-Although the underlying Persistit database is designed for highly concurrent multi-threaded operation, the Exchange object itself is not thread-safe. Each thread should create and use its own Exchange object(s) when accessing the database.
-
-To create an Exchange you provide a Volume name (or alias) and a tree name in its constructor. The constructor will optionally create a new tree in that Volume if a tree having the specified name is not found. An application may construct an arbitrary number of Exchange objects. Creating a new Exchange has no effect on the database if the specified tree already exists. Tree creation is thread-safe: multiple threads concurrently constructing Exchanges using the same Tree name will safely result in the creation of only one new tree.
-
-An Exchange is a moderately complex object that requires several thousand bytes of heap space. Memory-constrained applications should construct Exchanges in moderate numbers. An Exchange internally maintains some optimization information such that references to nearby Keys within a tree are accelerated. Performance may benefit from using a different Exchange for each area of the Tree being accessed.
-
-Persistit offers Exchange pooling to avoid rapidly creating and destroying Exchange objects in multi-threaded applications.  In particular, web applications may benefit from using the Exchange pool.
-
-An Exchange is always associated with a com.persistit.Key and a com.persistit.Value. Typically you work with an Exchange in one of the following patterns:
-. Modify the Key, perform a +fetch+ operation, and extract the Value.
-. Modify the Key, modify the Value, and then perform a +store+ operation.
-. Modify the Key, and then perform a +remove+ operation.
-. Optionally modify the Key, perform a +traverse+ operation, then read the resulting Key and/or Value.
-
-These four methods, plus a few other methods listed here, are the primary low-level interface to the database. Semantics are as follows:
-
-[horizontal]
-+fetch+:: Reads the stored value associated with this Exchange's Key and modifies the Exchange’s Value to reflect that value.
-+store+:: Inserts or replaces the key/value pair for the specified key in the Tree either by replacing the former value, if there was one, or inserting a new value.
-+fetchAndStore+:: Fetches and then replaces the stored value. Upon completion, Value reflects the formerly stored value for the current Key. This operation is atomic, as opposed to sequential calls to fetch and store.
-+remove+, +removeAll+, +removeKeyRange+:: Removes key/value pairs from the Tree. Versions of this method specify either a single key or a range of keys to be removed.
-+fetchAndRemove+:: Fetches and then removes the stored value. Upon completion, Value reflects the formerly stored value for the current Key. This operation is atomic, as opposed to sequential calls to fetch and remove.
-+traverse+, +next+, +previous+:: Modifies the Exchange’s Key and Value to reflect a successor or predecessor key within the tree. (See API documentation for com.persistit.Key for information on the order of traversal.)
-+incrementValue+:: Atomically increments or decrements a long (64-bit integer) value associated with the current Key, and returns the modified value. If there is currently no value associated with the key then incrementValue creates one and assigns an initial value to it. This operation provides a convenient way for concurrent threads to safely allocate unique long integers without an explicit transaction scope.
-+hasNext+, +hasPrevious+:: Indicates, without modifying the Exchange’s Value or Key objects, whether there is a successor or predecessor key in the Tree.
-+getChangeCount+:: Number of times the Tree for this Exchange has changed. This count may be used as a reliable indicator of whether the Tree has changed since some earlier instant in time. For example, it is used to detect concurrent modifications by PersistitMap.
-
-Because Persistit permits concurrent operations by multiple threads, there is no guarantee that the underlying database will remain unchanged after any of these operations is completed. However, each of these methods operates atomically. That is, the inputs and outputs of each method are consistent with some valid state of the underlying Persistit backing store at some instant in time. The Value and Key objects for the Exchange represent that consistent state even if some other thread subsequently modifies the underlying database.
-
-=== PersistitMap
-
-Persistit provides an implementation of the java.util.SortedMap interface called com.persistit.PersistitMap. PersistitMap uses Persistit as its backing store, permitting large maps to be stored efficiently on disk using constant heap memory space.
-
-Keys for PersistitMap must conform to the constraints described above under Keys. Values must conform to the constraints described for Values.
-
-The constructor for PersistitMap takes an Exchange as its sole parameter. All key/value pairs of the Map are stored within the tree identified by this Exchange. The Key supplied by the Exchange becomes the root of a logical tree. For example:
-
-[source,java]
-----
-Exchange ex = new Exchange("myVolume", "myTree", true);
-ex.append("USA").append("MA");
-PersistitMap<String, String> map = new PersistitMap<String, String>(ex);
-map.put("Boston", "Hub");
-----
-
-places a key/value pair into the myTree with the concatenated key +{"USA ","MA","Boston"}+ and a value of +"Hub"+.
-
-Because Persistit is designed for concurrent operation it is possible (and often intended) for the backing store of PersistitMap to be changed by other threads while a java.util.Iterator is in use. Generally the expected behavior for an Iterator on a Map collection view is to throw a ConcurrentModificationException if the underlying collection changes. This is known as fail-fast behavior. PersistitMap implements this behavior by throwing a ConcurrentModificationException in the event the Tree containing the map changes. An application can detect that the map may have changed due to a programming error in case the design contract calls for it to remain unchanged by catching this exception.
-
-However, sometimes it may be desirable to use PersistitMap and its collections view interfaces to iterate across changing data. Internally, Persistit uses the traverse method to retrieve the next highest key in the key sort order in order to implement the Iterator’s hasNext and next methods. The result will depend on the content of the database at the instant these operations are performed. PersistitMap provides the method setAllowConcurrentModification to enable this behavior. By default, concurrent modifications are not allowed.
-=== KeyFilter
-
-A +com.persistit.KeyFilter+ defines a subset of all possible key values. You can supply a KeyFilter to the traverse methods of an Exchange.  You can also specify a KeyFilter for any Iterator returned by the collection views of a PersistitMap.  In either case, the key/value pairs covered by traversing the database or iterating over the collection view are restricted to those selected by the KeyFilter.
-
-Use of a KeyFilter is illustrated by the following code fragment:
-
-[source,java]
-----
-Exchange ex = new Exchange("myVolume", "myTree", true);
-KeyFilter kf = new KeyFilter("{\"Bellini\":\"Busoni\"}");
-ex.append(Key.BEFORE);
-while (ex.next(kf))
-{
-  System.out.println(ex.getKey().reset().decodeString());
-}
-----
-
-This simple example emits the string-valued keys within myTree whose values fall alphabetically between “Bellini” and “Busoni”, inclusive.
-
-
-== Transactions
-
-Persistit supports transactions with full isolation and optimistic concurrency control. An application may begin, commit or roll back the current transaction scope explicitly, executing multiple database operations in an atomic, consistent, isolated and (optionally) durable (ACID) manner. If the application does not explicitly define the scope of a transaction, each database operation implicitly runs within the scope of a separate transaction. Each Persistit transaction may optionally be committed to either memory or disk. Transactions committed to memory are much faster, but are not immediately durable. (See Transactions.)
-
-== Configuration
-
-To initialize Persistit the embedding application invokes one of the initialize methods of +com.persistit.Persistit+, passing either a +java.util.Properties+ object or the name of a properties file from which the Properties object derives its content. The following properties are defined for Persistit. Other properties may also reside in the Properties object or its backing file; Persistit simply ignores any property not listed here.
-
-== Logging API
-
-Persistit is writes various diagnostic and informational messages to a log. By default, the log is written as text to the file +persistit.log+ in the current working directory. However, a container application will usually have a logging architecture already in place, and Persistit provides a simple way to redirect its log output to the container application’s log. Adapters for Log4J and the Java Logging API are included; other logging systems are easy to adapt.
-
-