Merge lp:~pbeaman/akiban-persistit/release-notes-3.2.0 into lp:akiban-persistit

[Peter Beaman]

Release notes updated for 3.2.0 release.

LBJ: NO_BUILD

[Timothy Wegner (timmwegner)]

Looks great

[Timothy Wegner (timmwegner)]

review: Approve

=== modified file 'doc/BugList'
--- doc/BugList	2012-09-11 21:34:59 +0000
+++ doc/BugList	2012-10-15 21:27:24 +0000
@@ -1,3 +1,89 @@
+1065677 : 3.2.0
+
+`Persistit not recoverable after hard crash`
+
+A system experienced a sequence of events, including being killed several times at 
+inopportune times by the OOM Killer, that left its journal in an unrecoverable state.
+Specifically, there was a journal file that contained lots of updated pages, but no
+checkpoint record.  Due to a bug in the RecoveryManager code, the system timestamp
+was set incorrectly and led to an out-of-order page update sequence.  While this bug
+would be difficult to recreate by any natural circumstances, it is critical because
+of potential data loss.
+
+1064565 : 3.2.0
+
+`Accumulator state sometimes missing from checkpoint`
+
+Occasionally the state of an Accumulator was incorrect after normal system restart.
+The cause was a race in the checkpoint code. This is a critical bug due to potential
+data loss.
+
+1062315 : 3.2.0
+
+`Assertion failure in TransactionIndexBucket#allocateTransactionStatus`
+
+An assert was observed in an 8-hour stress test run.  The underlying cause was
+a brief race condition and was harmless. The race has been fixed. 
+
+1058650 : 3.2.0
+
+`IllegalStateException due to out-of-order MVVs`
+
+Stress tests experienced an occasional occurrence of an IllegalStateException due
+to out-of-order versions in an MVV value.  The condition was transient and did not
+cause data loss. The underlying cause has been fixed.
+
+1058254 : 3.2.0
+
+`Join on range delete can change successor keys`
+
+A range delete that crosses at least two page boundaries can write an incorrect key 
+when merging records; this causes it and subsequent keys within the page to be wrong 
+and also leads to a discontinuity between the max key in the page and the first key 
+of its right sibling.
+
+1056489 : 3.2.0
+
+`MVV and step visibility and pruning errors`
+
+Transactions using +step+ values (see com.persisit.Transaction#setStep) sometimes resulted
+in inconsistent states.
+
+1053680 : 3.2.0
+
+`Page deallocation can permanently lose pages`
+
+When removing records causes deallocation of chain of pages configured in a particular way, 
+a garbage page, and possibly many pages linked to it, may be stranded in a volume. 
+The result is irrecoverable loss of space in the volume file.
+
+1047973 : 3.2.0
+
+`O(N*N) Loop in JournalManager#cleanupPageList`
+
+Bad code in this method causes periodic performance degradation in 4-hour TPCC test. The 
+CPU cost of this algorithm scales as the square of the number of pages in the page list, 
+and the work is done within a globally blocking synchronized block.
+
+1045971 : 3.2.0
+
+`Dynamically loaded volumes not recovered "cleanly"`
+
+When a program creates new volumes using VolumeSpecifications, they subsequently 
+cause a "missing volume" warning from the journal copier and prevent the removal 
+of journal files containing the references. The journal files accumulate seemingly 
+without end, and each initialization of the database seems to process more and 
+more of the old transactions.
+
+See discussion about regarding the useOldVSpec property.
+
+1046049 : 3.2.0
+
+`Filtered traverse in LTEQ direction appears not to work`
+
+A bug in the com.persistit.KeyFilter caused reverse-direction traversal using
+a KeyFilter with a range term to return incorrect results.
+
 1044513 : 3.1.7
 
 `AssertionError in IntegrityCheckTest`

=== modified file 'doc/ReleaseNotes.rst'
--- doc/ReleaseNotes.rst	2012-09-11 21:46:00 +0000
+++ doc/ReleaseNotes.rst	2012-10-15 21:27:24 +0000
@@ -1,5 +1,5 @@
 ************************************
-Akiban Persistit
+Akiban Persistit Version 3.2.0
 ************************************
 
 Overview
@@ -50,6 +50,10 @@
 +---------+--------------------+--------------------------------------------------------------------------+
 | Version | Release Date       |  Summary                                                                 |
 +=========+====================+==========================================================================+
+| 3.2.0   | October 15, 2012   | Fix several critical and other bugs, one of which requires a modified    |
+|         |                    | journal file format. This version also significantly improves            |
+|         |                    | performance for I/O-intensive concurrent transaction loads.              |
++---------+--------------------+--------------------------------------------------------------------------+
 | 3.1.7   | September 11, 2012 | Fix several bugs, add buffer pool preload ("warm-up"),                   |
 |         |                    | reformat code base                                                       |
 +---------+--------------------+--------------------------------------------------------------------------+
@@ -73,50 +77,111 @@
 Changes and New Features
 ========================
 
+Persistit 3.2.0 - Default Journal File Format Changed
+-----------------------------------------------------
+
+Version 3.2.0 fixes problems related to Volumes created and opened by the com.persistit#loadVolume 
+method rather than being specified by the initial system configuration. In previous versions, journal files 
+contained insufficient information to properly recover such volumes, even during normal startup.
+
+To solve this problem the format of the IV (identify volume) journal record changed to include 
+the com.persistit.VolumeSpecification rather than the volume name alone. By default, journal 
+files written by earlier versions of Persistit continue to be supported, but once Persistit 3.2.0 has 
+added one or more new journal files to a database, earlier versions of Persistit are unable to 
+open the database.
+
+In most cases it is never necessary to revert to an earlier version, but to support sites where 
+backward-compatibility may be important, it is possible to specify a new configuration parameter 
+(see com.persistit.Configuration#setUseOldVSpec). When this property is +true+, Persistit writes 
+journal files that are backward-compatible, but incapable of supporting dynamically created volumes.
+
+Persistit 3.2.0 - Performance Improvements
+------------------------------------------
+
+Version 3.2.0 significantly improves I/O performance for concurrent transactions and
+better controls the number of journal files created during by very aggressive loads. A new attribute
+in the com.persistit.mxbeans.JournalManagerMXBean class called urgentFileCountThreshold controls the
+maximum number of journal files Persistit will create before ramping the up the urgency of copying pages to
+allow purging old files. Several other issues related to I/O scheduling and management of 
+dirty pages were resolved.
+
+Version 3.2.0 significantly improves scheduling of version pruning operations.  Pruning is the process by
+which obsolete versions are removed from multi-version values (MVVs). Better scheduling results in a
+significant reduction in the amount of space consumed by obsolete version and also results in better
+transaction throughput.
+
 Persistit 3.1.7 - Code Base Reformatted
 ---------------------------------------
 
-To simplify diffs and improve legibility, the entire code base was reformatted and "cleaned up" by the Eclipse code formatting tool. The maven build now automatically formats all
-source to ensure coherent diffs in the future.  The settings for formatting and code style cleanup by Eclipse are found in the ``src/etc`` directory.
+To simplify diffs and improve legibility, the entire code base was reformatted and "cleaned up" 
+by the Eclipse code formatting tool. The maven build now automatically formats all
+source to ensure coherent diffs in the future.  The settings for formatting and code style 
+cleanup by Eclipse are found in the ``src/etc`` directory.
 
 Persistit 3.1.7 - Buffer Pool Preload
 -------------------------------------
 
-On a server with a large buffer pool (many gigabytes), a Persistit instance can run for a long time before the buffer pool becomes populated with a suitable working set of database pages. Until then performance is degraded due to a potentially large number of random reads. For a production server the result may be poor performance for minutes to hours after restart.
-
-The preload facility periodically records an inventory of the pages currently in the buffer pool(s) and optionally reloads the same set of pages when Persistit is restarted. During the preload process Persistit attempts to read pages in approximate file-address order to accelerate reads. In one of our experiments Persistit preloads a buffer pool with over 800,000 16Kbyte buffers in about a minute, which is orders of magnitude faster than the same process would take with reads performed incrementally at random.
-
-Two new configuration properties com.persistit.Configuration#setBufferInventoryEnabled and com.persistit.Configuration#setBufferPreloadEnabled control this behavior. These settings are turned off by default in version 3.1.7.
+On a server with a large buffer pool (many gigabytes), a Persistit instance can run for a long 
+time before the buffer pool becomes populated with a suitable working set of database pages. 
+Until then performance is degraded due to a potentially large number of random reads. 
+For a production server the result may be poor performance for minutes to hours after restart.
+
+The preload facility periodically records an inventory of the pages currently in the buffer 
+pool(s) and optionally reloads the same set of pages when Persistit is restarted. During the 
+preload process Persistit attempts to read pages in approximate file-address order to 
+accelerate reads. In one of our experiments Persistit preloads a buffer pool with over 
+800,000 16Kbyte buffers in about a minute, which is orders of magnitude faster than the 
+same process would take with reads performed incrementally at random.
+
+Two new configuration properties com.persistit.Configuration#setBufferInventoryEnabled and 
+com.persistit.Configuration#setBufferPreloadEnabled control this behavior. These settings 
+are turned off by default in version 3.1.7.
 
 Persistit 3.1.4 - Detecting and Ignoring Missing Volumes
 --------------------------------------------------------
 
-Every time Persistit writes a modified page to disk, it does so first to the journal. During recovery processing, the page images from the journal are analyzed and reinserted into volumes in such a way that all B+Trees are restored to a consistent state. The issue addressed in this change is how Persistit behaves during recovery if it discovers that a volume referred to by a page in the journal no longer exists.
+Every time Persistit writes a modified page to disk, it does so first to the journal. 
+During recovery processing, the page images from the journal are analyzed and reinserted 
+into volumes in such a way that all B+Trees are restored to a consistent state. The issue 
+addressed in this change is how Persistit behaves during recovery if it discovers that a 
+volume referred to by a page in the journal no longer exists.
 
-Recognizing that under some circumstances an administrator may indeed wish to remove a volume from an existing Database, this change provides a configurable switch to optionally allow pages from missing volumes to be skipped (with logged warning messages) during recovery processing.  The switch can be enabled by setting the configuration parameter com.persistit.Configuration#setIgnoreMissingViolumes to true.
+Recognizing that under some circumstances an administrator may indeed wish to remove a 
+volume from an existing Database, this change provides a configurable switch to optionally 
+allow pages from missing volumes to be skipped (with logged warning messages) during recovery 
+processing.  The switch can be enabled by setting the configuration parameter 
+com.persistit.Configuration#setIgnoreMissingViolumes to true.
 
 
 Persistit 3.1.4 - Reduce KeyCoder Serialized Object Size
 --------------------------------------------------------
 
 .. note::
-   Any Database containing objects serialized by a custom KeyCoder from a previous version of Persistit is incompatible with this change
-
-Minimize the per-instance overhead for application objects written into Persistit Keys by reducing the size of the internal identifier.
-
-Persistit has rich support for serializing standard Java primitive and object types into a Key. Additionally, the KeyCoder class allows for any application level object to also be appended to a Key right next to any other type. This is tagged internally with per-class handles. This change lowers the initial offset to reduce and in many cases halve the serialized size. 
+   Any Database containing objects serialized by a custom KeyCoder from a previous version 
+   of Persistit is incompatible with this change
+
+Minimize the per-instance overhead for application objects written into Persistit Keys by 
+reducing the size of the internal identifier.
+
+Persistit has rich support for serializing standard Java primitive and object types into a 
+Key. Additionally, the KeyCoder class allows for any application level object to also be 
+appended to a Key right next to any other type. This is tagged internally with per-class 
+handles. This change lowers the initial offset to reduce and in many cases halve the serialized size. 
 
 Persistit 3.1.4 - Maven POM Changes For Eclipse Juno 
 ----------------------------------------------------
 
-The latest version of Eclipse, code named Juno, features a wide array of changes, including a new release of the m2eclipse plugin. In an effort to make getting started with Persistit as easy as possible, we have included the required m2e configuration sections in our pom.
+The latest version of Eclipse, code named Juno, features a wide array of changes, including a 
+new release of the m2eclipse plugin. In an effort to make getting started with Persistit as 
+easy as possible, we have included the required m2e configuration sections in our pom.
 
 Please contact Akiban if you have encounter any issues getting up and running with Persistit.   
 
 Persistit 3.1.2 - Asserts Added to Check for Correct Exchange Thread Behavior
 -----------------------------------------------------------------------------
 
-A bug in the Akiban Server code caused an Exchange to be used concurrently by two Threads, causing serious and seemingly unrelated failures in Persistit including instances of IllegalMonitorException and IllegalStateException. To guard against future occurrences, asserts were added to catch such concurrent use by multiple threads.  Applications should be tested with asserts enabled to verify correct thread usage.
+A bug in the Akiban Server code caused an Exchange to be used concurrently by two Threads, 
+causing serious and seemingly unrelated failures in Persistit including instances of IllegalMonitorException and IllegalStateException. To guard against future occurrences, asserts were added to catch such concurrent use by multiple threads.  Applications should be tested with asserts enabled to verify correct thread usage.
 
 
 
@@ -126,21 +191,23 @@
 Transactional Tree Management
 -----------------------------
 
-All operations within Trees such as store, fetch, remove and traverse are correctly supported within transactions. However, the operations to create and delete Tree instances currently do not respect transaction boundaries. For example, if a transaction creates a new Tree, it is immediately visible within other Transactions and will continue to exist even if the original transaction aborts.  (However, records inserted or modified by the original transaction will not be visible until the transaction commits.) Prior to creating/removing trees, transaction processing should be quiesced and allowed to complete.
-
-Problems with Disk Full
-------------------------------------
-
-https://bugs.launchpad.net/akiban-persistit/+bug/916071
-
-There are rare cases where Persistit will generate exceptions other than java.io.IOException: No space left on device when a disk volume containing the journal or volume file fills up. The database will be intact upon recovery, but the application may receive unexpected exceptions.
+All operations within Trees such as store, fetch, remove and traverse are correctly supported 
+within transactions. However, the operations to create and delete Tree instances currently do 
+not respect transaction boundaries. For example, if a transaction creates a new Tree, it is 
+immediately visible within other Transactions and will continue to exist even if the original 
+transaction aborts.  (However, records inserted or modified by the original transaction will 
+not be visible until the transaction commits.) Prior to creating/removing trees, transaction 
+processing should be quiesced and allowed to complete.
+
 
 Out of Memory Error, Direct Memory Buffer
 ------------------------------------------------------
 
 https://bugs.launchpad.net/akiban-persistit/+bug/985117
 
-Out of Memory Error, Direct Memory Buffer.  Can cause failed transactions under extreme load conditions as a result of threads getting backed up writing to the journal file. However, this error is transient and recoverable by by retrying the failed transaction.
+Out of Memory Error, Direct Memory Buffer.  Can cause failed transactions under extreme load 
+conditions as a result of threads getting backed up writing to the journal file. However, 
+this error is transient and recoverable by by retrying the failed transaction.
 
 * Workaround: Ensure your application has the ability to retry failed transactions
 
@@ -149,16 +216,23 @@
 
 https://bugs.launchpad.net/akiban-persistit/+bug/986465
 
-The getChangeCount method may return inaccurate results as its not currently transactional.  The primary consumer is the PersistitMap. As a result of this bug Persistit may not generate java.util.ConcurrentModiciationException when it is supposed to.
+The getChangeCount method may return inaccurate results as its not currently transactional.  
+The primary consumer is the PersistitMap. As a result of this bug Persistit may not generate 
+java.util.ConcurrentModificationException when it is supposed to.
 
 Multi-Version-Values sometimes not fully pruned
 -------------------------------------------------------------
 
 https://bugs.launchpad.net/akiban-persistit/+bug/1000331
 
-Multi-version values are not always pruned properly causing volume growth.  The number of MVV records and their overhead size can be obtaining by running the IntegrityCheck task. 
+Multi-version values are not always pruned properly causing volume growth.  The number of 
+MVV records and their overhead size can be obtaining by running the IntegrityCheck task. 
 
 * Workaround 1: Run the IntegrityCheck task (CLI command icheck) with the -P option which will prune the MVVs. This will remove obsolete MVV instances and in many cases free up pages in which new data can be stored.  However, it will not reduce the actual size of the volume file.
 
 * Workaround 2: To reduce the size of the volume you can use the CLI commands ``save`` and ``load`` to reload the data into a newly created volume file. See http://www.akiban.com/ak-docs/admin/persistit/Management.html#management for more information about these operations.
 
+Note: although the methods described here may be helpful in reducing MVV clutter, Persistit Version 3.2.0 significantly improves the algorithms used to prune obsolete versions and the 
+techniques described here are unlikely to be necessary.
+
+

=== modified file 'src/main/java/com/persistit/Configuration.java'
--- src/main/java/com/persistit/Configuration.java	2012-10-03 17:21:10 +0000
+++ src/main/java/com/persistit/Configuration.java	2012-10-15 21:27:24 +0000
@@ -1936,8 +1936,9 @@
      * Journal files created by version 3.1.8 with this mode enabled can be used
      * by earlier versions of Persistit. However, doing so prevents volumes
      * created dynamically using
-     * {@link Persistit#loadVolume(VolumeSpecification))} from being recovered
-     * properly. (See https://bugs.launchpad.net/akiban-persistit/+bug/1045971)
+     * {@link Persistit#loadVolume(com.persistit.VolumeSpecification)} from
+     * being recovered properly. (See
+     * https://bugs.launchpad.net/akiban-persistit/+bug/1045971)
      * </p>
      * <p>
      * This method and configuration parameter is deprecated and will be removed