Merge lp:~pbeaman/akiban-persistit/buffer-pool-warmup-sorted into lp:akiban-persistit

[Peter Beaman]

This proposal is a re-work of the original buffer pool warm-up code. Instead of creating a text file, this version stores the inventory in the system volume. It also loads pages in file-address order to reduce I/O time. There is a new stress test that pushes this into a medium-sized buffer pool (25K pages) but we still need to test it with a 1M page pool in TPCC or some other big test.

There are two config options, one to turn on inventory recording and the other to enable pre-loading pages. They are separate so that a server can start up without waiting for the pre-load but still be set to perform the inventory function.

Other changes include Configuration, Configuration.rst (documentation), etc.

[Nathan Williams (nwilliams)]

This has a pre-req of the reformatting, but it doesn't look to have kicked in.

Could you try resubmitting? If that doesn't work then I'll do the diff manually for review.

[Peter Beaman (pbeaman)]

Resubmitted to try to get a clean diff.

[Nathan Williams (nwilliams)]

The diff on LP is still wonky, but I checked out locally and everything is fine.

Changes look good, just a few questions below.

The inventory is taken in the checkpoint thread. How does the run time of the inventory scale with the size of the pool? You mentioned a 25K test, but the 1M is also a realistic size (even in our nightlies).

Along the same lines, have you tried this under any load?

Lastly, it is also done on close(). If this takes even in the 10s of seconds, we might want add an option for that.

review: Needs Information

[Nathan Williams (nwilliams)]

Talked with Peter about this offline and the points I raised were what he alluded to in the merge description.

Summary:
- Writing 1M records, non-transactionally, is a sub-second activity
- Start-up reading of the pages will be the slow part, perhaps minutes
- There are progress messages and this is the tradeoff to not have it spread out over hours
- And most importantly, this defaults to OFF

Given all that, putting this in is low risk and timing on large pools can be done soon^TM.

review: Approve

[Akiban Build User (build-akiban)]

There were 3 failures during build/test:

* job persistit-build failed at build number 408: http://172.16.20.104:8080/job/persistit-build/408/

* view must-pass failed: sql-parser-build is blue_anime

* view must-pass failed: persistit-build is yellow

review: Needs Fixing

[Peter Beaman (pbeaman)]

Fix WarmupTest

[Akiban Build User (build-akiban)]

There were 2 failures during build/test:

* job persistit-build failed at build number 410: http://172.16.20.104:8080/job/persistit-build/410/

* view must-pass failed: persistit-build is yellow

review: Needs Fixing

[lp:~pbeaman/akiban-persistit/buffer-pool-warmup-sorted]

385. By Peter Beaman on 2012-08-27

Fix WarmupTest (again)

=== modified file 'doc/Configuration.rst'
--- doc/Configuration.rst	2012-06-11 21:25:37 +0000
+++ doc/Configuration.rst	2012-08-27 21:56:20 +0000
@@ -221,7 +221,7 @@
   ``serialOverride``, ``constructorOverride``: (``com.persistit.Configuration#setSerialOverride`` ``com.persistit.Configuration#setConstructorOverride``) 
       Control aspects of object serialization. See :ref:`Serialization`.
 
-  ``showgui``: (``com.persistit.Configuration#setShowGUI``), True of False.  
+  ``showgui``: (``com.persistit.Configuration#setShowGUI``), True or False (default).  
       If true, Persistit attempts to create and display an instance of the AdminUI utility panel within the current JVM. 
       Alternatively, AdminUI uses RMI and can be launched and run remotely if ``rmiport`` or ``rmihost`` has been 
       specified.
@@ -231,6 +231,18 @@
       install a logging adapter to reroute messages through Log4J, SLF4J or other logger. The ``logfile`` property is used 
       only when no adapter has been installed.
 
+  ``bufferinventory``: (``com.persistit.Configuration#setBufferInventoryEnabled``), True or False (default).
+      If true, Persistit periodically records an inventory of all the buffes in the buffers pools to the System Volume. The inventory
+      enables Persistit to preload the buffer pools then next time it starts up with approximately the same pages that were present
+      before shutdown. To enable buffer preloading, the bufferpreload property must also be true.
+      
+  ``bufferpreload``: (``com.persistit.Configuration#setBufferPreloadEnabled``), True or False (default).
+      If true, and if a buffer pool inventory was previously recorded, Persistit attempts to "warm up" the buffer pool
+      by preloading pages that were present in the buffer pool when Persistit last shut down. This may allow a freshly started
+      Persistit instance to begin service a workload similar to what it had previously been processing without incurring the
+      cost of many random disk reads to load pages.
+        
+
 For all integer-valued properties, the suffix “K” may be used to represent kilo, “M” for mega, “G” for giga and “T” for tera. For example, “2M” represents the value 2,097,152.
 
 A Configuration Example

=== modified file 'src/main/java/com/persistit/BufferPool.java'
--- src/main/java/com/persistit/BufferPool.java	2012-08-15 16:11:36 +0000
+++ src/main/java/com/persistit/BufferPool.java	2012-08-27 21:56:20 +0000
@@ -18,13 +18,20 @@
 import java.io.BufferedReader;
 import java.io.BufferedWriter;
 import java.io.DataOutputStream;
+<<<<<<< TREE
 import java.io.File;
 import java.io.FileReader;
 import java.io.FileWriter;
 import java.io.IOException;
+=======
+import java.io.IOException;
+>>>>>>> MERGE-SOURCE
 import java.nio.ByteBuffer;
+import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.HashSet;
+import java.util.List;
 import java.util.Set;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicInteger;
@@ -32,6 +39,7 @@
 import java.util.concurrent.atomic.AtomicLongArray;
 import java.util.concurrent.locks.ReentrantLock;
 
+import com.persistit.JournalManager.PageNode;
 import com.persistit.exception.InUseException;
 import com.persistit.exception.InvalidPageAddressException;
 import com.persistit.exception.InvalidPageStructureException;
@@ -92,6 +100,17 @@
      */
     private final static int WRITE_AGE_THRESHOLD_RATIO = 4;
 
+    private final static String INVENTORY_TREE_NAME = "_buffers";
+    /**
+     * Maximum number of buffer inventory versions to retain
+     */
+    private final static int INVENTORY_VERSIONS = 3;
+
+    /**
+     * Preload log multiple
+     */
+    private final static int INVENTORY_PRELOAD_LOG_MESSAGE_MULTIPLE = 10000;
+
     /**
      * The Persistit instance that references this BufferPool.
      */
@@ -201,16 +220,21 @@
     private volatile long _writerPollInterval = DEFAULT_WRITER_POLL_INTERVAL;
 
     private volatile int _pageWriterTrancheSize = PAGE_WRITER_TRANCHE_SIZE;
+<<<<<<< TREE
 
     /**
      * Polling interval for PageCacher
      */
     private volatile long _cacherPollInterval;
 
+=======
+
+>>>>>>> MERGE-SOURCE
     /**
      * The PAGE_WRITER IOTaskRunnable
      */
     private PageWriter _writer;
+<<<<<<< TREE
 
     /**
      * The PAGE_CACHER IOTaskRunnable
@@ -219,6 +243,9 @@
 
     private String _defaultLogPath;
 
+=======
+
+>>>>>>> MERGE-SOURCE
     /**
      * Construct a BufferPool with the specified count of <code>Buffer</code>s
      * of the specified size.
@@ -289,6 +316,7 @@
             throw e;
         }
         _writer = new PageWriter();
+<<<<<<< TREE
         _cacher = new PageCacher();
     }
 
@@ -320,6 +348,8 @@
         } catch (final IOException e) {
             throw new PersistitException(e);
         }
+=======
+>>>>>>> MERGE-SOURCE
     }
 
     void startThreads() throws PersistitException {
@@ -329,9 +359,7 @@
     void close() {
         _closed.set(true);
         _persistit.waitForIOTaskStop(_writer);
-        _persistit.waitForIOTaskStop(_cacher);
         _writer = null;
-        _cacher = null;
     }
 
     /**
@@ -340,7 +368,6 @@
      */
     void crash() {
         IOTaskRunnable.crash(_writer);
-        IOTaskRunnable.crash(_cacher);
     }
 
     void flush(final long timestamp) throws PersistitInterruptedException {
@@ -432,6 +459,7 @@
             buffer.populateInfo(array[index]);
         }
     }
+<<<<<<< TREE
 
     private void populateWarmupFile() throws PersistitException {
         final File file = new File(_defaultLogPath);
@@ -461,6 +489,8 @@
             throw new PersistitException(e);
         }
     }
+=======
+>>>>>>> MERGE-SOURCE
 
     private boolean selected(final Buffer buffer, final int includeMask, final int excludeMask) {
         return ((includeMask == 0) || (buffer.getStatus() & includeMask) != 0)
@@ -1402,6 +1432,7 @@
             return isFlushing() ? 0 : _writerPollInterval;
         }
     }
+<<<<<<< TREE
 
     /**
      * Implementation of PAGE_CACHER thread
@@ -1431,6 +1462,8 @@
             return isFlushing() ? 0 : _cacherPollInterval;
         }
     }
+=======
+>>>>>>> MERGE-SOURCE
 
     @Override
     public String toString() {
@@ -1486,4 +1519,133 @@
         }
         stream.flush();
     }
+
+    void recordBufferInventory(final long timestamp) throws PersistitException {
+        final Exchange exchange = getBufferInventoryExchange();
+        /*
+         * Advisory only - transaction integrity not needed
+         */
+        exchange.ignoreTransactions();
+        try {
+            int total = 0;
+            exchange.clear().append(_bufferSize).append(timestamp).append(Key.BEFORE);
+            final Value value = exchange.getValue();
+            final int clockValueBefore = _clock.get();
+            for (int index = 0; index < _buffers.length; index++) {
+                final Buffer buffer = _buffers[index];
+                long page1 = -1, page2 = -1;
+                Volume volume1 = null, volume2 = null;
+                if (buffer != null && buffer.isValid()) {
+                    while (true) {
+                        page1 = buffer.getPageAddress();
+                        volume1 = buffer.getVolume();
+                        page2 = buffer.getPageAddress();
+                        volume2 = buffer.getVolume();
+                        if (page1 == page2 && volume1 == volume2) {
+                            break;
+                        }
+                        Util.spinSleep();
+                    }
+                    if (volume1 != null && !volume1.isTemporary()) {
+                        value.clear().setStreamMode(true);
+                        value.put(volume1.getHandle());
+                        value.put(page1);
+                        exchange.to(index).store();
+                        total++;
+                    }
+                }
+            }
+            final int clockValueAfter = _clock.get();
+            exchange.cut();
+            value.clear().setStreamMode(true);
+            value.put(_bufferCount);
+            value.put(total);
+            value.put(clockValueBefore);
+            value.put(clockValueAfter);
+            value.put(System.currentTimeMillis());
+            exchange.store();
+            int count = 0;
+            while (exchange.previous()) {
+                if (++count > INVENTORY_VERSIONS) {
+                    exchange.remove(Key.GTEQ);
+                }
+            }
+        } catch (final PersistitException e) {
+            _persistit.getLogBase().bufferInventoryException.log(e);
+        }
+    }
+
+    void preloadBufferInventory() {
+        int count = 0;
+        int total = 0;
+        try {
+            final JournalManager jman = _persistit.getJournalManager();
+            final Exchange exchange = getBufferInventoryExchange();
+            final Value value = exchange.getValue();
+            final List<PageNode> pageNodes = new ArrayList<PageNode>();
+            boolean foundInventory = false;
+            exchange.clear().append(_bufferSize).append(Key.AFTER);
+            while (exchange.previous()) {
+                if (exchange.getValue().isDefined()) {
+                    foundInventory = true;
+                    break;
+                }
+            }
+            if (!foundInventory) {
+                return;
+            }
+            value.setStreamMode(true);
+            /* int bufferCount = */value.getInt();
+            total = value.getInt();
+            /* int clockValueBefore = */value.getInt();
+            /* int clockValueAfter = */value.getInt();
+            final long systemTime = value.getLong();
+
+            _persistit.getLogBase().bufferInventoryLoad.log(systemTime);
+
+            exchange.append(Key.BEFORE);
+
+            while (exchange.next()) {
+                value.setStreamMode(true);
+                final int volumeHandle = value.getInt();
+                final long pageAddress = value.getLong();
+                final PageNode pn = new PageNode(volumeHandle, pageAddress);
+                pageNodes.add(pn);
+            }
+
+            Collections.sort(pageNodes, PageNode.READ_COMPARATOR);
+            for (final PageNode pn : pageNodes) {
+                final Volume vol = jman.volumeForHandle(pn.getVolumeHandle());
+                if (vol == null) {
+                    continue;
+                }
+                try {
+                    final Buffer buff = get(vol, pn.getPageAddress(), false, true);
+                    buff.release();
+                    count++;
+                    if ((count % INVENTORY_PRELOAD_LOG_MESSAGE_MULTIPLE) == 0) {
+                        _persistit.getLogBase().bufferInventoryProgress.log(count, total);
+                    }
+                    if (count >= _bufferCount) {
+                        //
+                        // If the buffer pool is now smaller, no need to load
+                        // more pages
+                        //
+                        break;
+                    }
+                } catch (final PersistitException e) {
+                    // ignore it
+                }
+            }
+        } catch (final PersistitException e) {
+            _persistit.getLogBase().bufferInventoryException.log(e);
+        } finally {
+            _persistit.getLogBase().bufferInventoryProgress.log(count, total);
+        }
+    }
+
+    private Exchange getBufferInventoryExchange() throws PersistitException {
+        final Volume sysvol = _persistit.getSystemVolume();
+        return _persistit.getExchange(sysvol, INVENTORY_TREE_NAME, true);
+    }
 }

=== modified file 'src/main/java/com/persistit/CheckpointManager.java'
--- src/main/java/com/persistit/CheckpointManager.java	2012-08-15 16:11:36 +0000
+++ src/main/java/com/persistit/CheckpointManager.java	2012-08-27 21:56:20 +0000
@@ -201,6 +201,7 @@
     void pollCreateCheckpoint() throws PersistitException {
         final long now = System.nanoTime();
         if (_lastCheckpointNanos + _checkpointIntervalNanos < now) {
+            _persistit.recordBufferPoolInventory();
             createCheckpoint();
         }
     }
@@ -252,12 +253,17 @@
                 _currentCheckpoint = new Checkpoint(txn.getStartTimestamp(), System.currentTimeMillis());
                 _outstandingCheckpoints.add(_currentCheckpoint);
                 _persistit.getLogBase().checkpointProposed.log(_currentCheckpoint);
+<<<<<<< TREE
                 return _currentCheckpoint;
             } catch (final InterruptedException ie) {
+=======
+            } catch (final InterruptedException ie) {
+>>>>>>> MERGE-SOURCE
                 throw new PersistitInterruptedException(ie);
             } finally {
                 txn.end();
             }
+            return _currentCheckpoint;
         } finally {
             _persistit.setSessionId(saveSessionId);
         }

=== modified file 'src/main/java/com/persistit/Configuration.java'
--- src/main/java/com/persistit/Configuration.java	2012-08-15 16:11:36 +0000
+++ src/main/java/com/persistit/Configuration.java	2012-08-27 21:56:20 +0000
@@ -263,8 +263,13 @@
     /**
      * Property name for the "append only" property.
      */
+<<<<<<< TREE
     public final static String APPEND_ONLY_PROPERTY = "appendonly";
 
+=======
+    public final static String APPEND_ONLY_PROPERTY_NAME = "appendonly";
+
+>>>>>>> MERGE-SOURCE
     /**
      * Property name for the "ignore missing volumes" property.
      */
@@ -274,13 +279,27 @@
      * Property name to specify the default {@link SplitPolicy}.
      */
     public final static String SPLIT_POLICY_PROPERTY_NAME = "splitpolicy";
+<<<<<<< TREE
 
     /**
      * Property name to specify the"buffer inventory" property name.
+=======
+
+    /**
+     * Property name to specify whether buffer preloading is enabled.
+     */
+    public final static String BUFFER_PRELOAD_PROPERTY_NAME = "bufferpreload";
+
+    /**
+     * Property name to specify whether buffer inventory is enabled.
+>>>>>>> MERGE-SOURCE
      */
     public final static String BUFFER_INVENTORY_PROPERTY_NAME = "bufferinventory";
+<<<<<<< TREE
 
     public final static String BUFFER_POLLING_INTERVAL_PROPERTY = "bufferpollinginterval";
+=======
+>>>>>>> MERGE-SOURCE
 
     /**
      * Property name to specify the default {@link JoinPolicy}.
@@ -634,9 +653,14 @@
     private int rmiServerPort;
     private boolean jmx = true;
     private boolean appendOnly;
+<<<<<<< TREE
     private String bufferInventoryPathName;
     private long bufferInventoryPollInterval = 3000000; // default five minute
                                                         // polling
+=======
+    private boolean bufferInventoryEnabled;
+    private boolean bufferPreloadEnabled;
+>>>>>>> MERGE-SOURCE
     private boolean ignoreMissingVolumes;
     private String tmpVolDir;
     private int tmpVolPageSize;
@@ -715,9 +739,7 @@
     }
 
     void loadProperties() throws InvalidVolumeSpecificationException {
-        setBufferInventoryPathName(getProperty(BUFFER_INVENTORY_PROPERTY_NAME));
-        setBufferInventoryPollingInterval(getLongProperty(BUFFER_POLLING_INTERVAL_PROPERTY, bufferInventoryPollInterval));
-        setAppendOnly(getBooleanProperty(APPEND_ONLY_PROPERTY, false));
+        setAppendOnly(getBooleanProperty(APPEND_ONLY_PROPERTY_NAME, false));
         setCommitPolicy(getProperty(COMMIT_POLICY_PROPERTY_NAME));
         setConstructorOverride(getBooleanProperty(CONSTRUCTOR_OVERRIDE_PROPERTY_NAME, false));
         setIgnoreMissingVolumes(getBooleanProperty(IGNORE_MISSING_VOLUMES_PROPERTY, false));
@@ -737,6 +759,8 @@
         setShowGUI(getBooleanProperty(SHOW_GUI_PROPERTY_NAME, false));
         setSplitPolicy(getProperty(SPLIT_POLICY_PROPERTY_NAME));
         setSysVolume(getProperty(SYSTEM_VOLUME_PROPERTY_NAME, DEFAULT_SYSTEM_VOLUME_NAME));
+        setBufferInventoryEnabled(getBooleanProperty(BUFFER_INVENTORY_PROPERTY_NAME, false));
+        setBufferPreloadEnabled(getBooleanProperty(BUFFER_PRELOAD_PROPERTY_NAME, false));
 
         loadPropertiesBufferSpecifications();
         loadPropertiesVolumeSpecifications();
@@ -1548,8 +1572,7 @@
     /**
      * <p>
      * Set a pattern that identifies classes to be serialized using standard
-     * Java serialization rather than Persistit's default serialization. TODO
-     * Link to Serialization section of user_guide.html.
+     * Java serialization rather than Persistit's default serialization.
      * </p>
      * <p>
      * Default value is <code>null</code><br />
@@ -1559,6 +1582,8 @@
      * @param serialOverride
      *            the serial override pattern to set
      * @see DefaultCoderManager
+     * @see <a
+     *      href="http://www.akiban.com/ak-docs/admin/persistit/Serialization.html">Serialization</a>
      */
     public void setSerialOverride(final String serialOverride) {
         this.serialOverride = serialOverride;
@@ -1579,8 +1604,7 @@
      * a public no-argument constructor. If so, then that constructor is used
      * when deserializing in the {@link DefaultObjectCoder}; if not then
      * Persistit uses private methods within the JDK to emulate standard Java
-     * serialization logic. TODO Link to Serialization section of
-     * user_guide.html.
+     * serialization logic.
      * </p>
      * <p>
      * Default value is <code>false</code><br />
@@ -1589,6 +1613,8 @@
      * 
      * @param constructorOverride
      *            the constructorOverride to set
+     * @see <a
+     *      href="http://www.akiban.com/ak-docs/admin/persistit/Serialization.html">Serialization</a>
      */
     public void setConstructorOverride(final boolean constructorOverride) {
         this.constructorOverride = constructorOverride;
@@ -1791,7 +1817,7 @@
     /**
      * Return the value defined by {@link #setAppendOnly}
      * 
-     * @return the whether to start Persistit in append-only mode
+     * @return <true>true</code> if append-only mode is enabled at startup
      */
     public boolean isAppendOnly() {
         return appendOnly;
@@ -1807,17 +1833,18 @@
      * </p>
      * <p>
      * Default value is <code>false</code><br />
-     * Property name is {@value #APPEND_ONLY_PROPERTY}
+     * Property name is {@value #APPEND_ONLY_PROPERTY_NAME}
      * </p>
      * 
      * @param appendOnly
-     *            <code>true</code> to start Persistit in append-only only
+     *            <code>true</code> to start Persistit in append-only mode
      */
     public void setAppendOnly(final boolean appendOnly) {
         this.appendOnly = appendOnly;
     }
 
     /**
+<<<<<<< TREE
      * Return the path name defined by {@link #getBufferInventoryPathName}
      * 
      * @return the path where file to warm-up Persistit with sample buffer data
@@ -1874,10 +1901,64 @@
         bufferInventoryPollInterval = Util.rangeCheck(seconds, 60L, Long.MAX_VALUE) * 1000L;
     }
 
+=======
+     * Return the value defined by {@link #setBufferInventoryEnabled}
+     * 
+     * @return <code>true</code> if periodic buffer pool inventory recording is
+     *         enabled
+     */
+    public boolean isBufferInventoryEnabled() {
+        return bufferInventoryEnabled;
+    }
+
+    /**
+     * <p>
+     * Control whether Persistit periodically records an inventory of its buffer
+     * pools to enable buffer pool preloading on a subsequent startup.
+     * </p>
+     * <p>
+     * Default value is <code>false</code><br />
+     * Property name is {@value #BUFFER_INVENTORY_PROPERTY_NAME}
+     * 
+     * @param bufferInventoryEnabled
+     *            <code>true</code> to enable periodic buffer inventory behavior
+     */
+    public void setBufferInventoryEnabled(final boolean bufferInventoryEnabled) {
+        this.bufferInventoryEnabled = bufferInventoryEnabled;
+    }
+
+    /**
+     * Return the value defined by {@link #setBufferPreloadEnabled}
+     * 
+     * @return <code>true</code> if the option to preload buffer pools is
+     *         enabled
+     */
+    public boolean isBufferPreloadEnabled() {
+        return bufferPreloadEnabled;
+    }
+
+    /**
+     * <p>
+     * Control whether Persistit attempts to preload (warm up) the buffer pools
+     * by preloading pages recorded in a previously generated inventory.
+     * </p>
+     * <p>
+     * Default value is <code>false</code><br />
+     * Property name is {@value #BUFFER_INVENTORY_PROPERTY_NAME}
+     * 
+     * @param bufferPreloadEnabled
+     *            <code>true</code> to enable buffer pool preloading during
+     *            startup
+     */
+    public void setBufferPreloadEnabled(final boolean bufferPreloadEnabled) {
+        this.bufferPreloadEnabled = bufferPreloadEnabled;
+    }
+
+>>>>>>> MERGE-SOURCE
     /**
      * Return the value defined by {@link #setIgnoreMissingVolumes(boolean)}
      * 
-     * @return the whether to start Persistit in ignore-missing-volumes mode
+     * @return <code>true</code>to enable ignore-missing-volumes mode
      */
     public boolean isIgnoreMissingVolumes() {
         return ignoreMissingVolumes;

=== modified file 'src/main/java/com/persistit/JournalManager.java'
--- src/main/java/com/persistit/JournalManager.java	2012-08-23 15:24:42 +0000
+++ src/main/java/com/persistit/JournalManager.java	2012-08-27 21:56:20 +0000
@@ -690,6 +690,29 @@
         final ByteBuffer bb = buffer.getByteBuffer();
 
         final Volume volume = buffer.getVolume();
+        final PageNode pn = lookupUpPageNode(pageAddress, volume);
+        if (pn == null) {
+            return false;
+        }
+        bb.position(0);
+        final long recordPageAddress = readPageBufferFromJournal(pn, bb);
+        _persistit.getIOMeter().chargeReadPageFromJournal(volume, pageAddress, bufferSize, pn.getJournalAddress(),
+                buffer.getIndex());
+
+        if (pageAddress != recordPageAddress) {
+            throw new CorruptJournalException("Record at " + pn + " is not volume/page " + buffer.toString());
+        }
+
+        if (bb.limit() != bufferSize) {
+            throw new CorruptJournalException("Record at " + pn + " is wrong size: expected/actual=" + bufferSize + "/"
+                    + bb.limit());
+        }
+        _readPageCount++;
+        buffer.getVolume().getStatistics().bumpReadCounter();
+        return true;
+    }
+
+    PageNode lookupUpPageNode(final long pageAddress, final Volume volume) {
         PageNode pnLookup = null;
         synchronized (this) {
             final Integer volumeHandle = _volumeToHandleMap.get(volume);
@@ -699,7 +722,7 @@
         }
 
         if (pnLookup == null) {
-            return false;
+            return null;
         }
 
         final PageNode pn = new PageNode(pnLookup.getVolumeHandle(), pnLookup.getPageAddress(),
@@ -714,6 +737,7 @@
          * new checkpoints and that keeps the copier from deleting it.
          */
         if (pnLookup.isInvalid()) {
+<<<<<<< TREE
             return false;
         }
 
@@ -733,6 +757,11 @@
         _readPageCount++;
         buffer.getVolume().getStatistics().bumpReadCounter();
         return true;
+=======
+            return null;
+        }
+        return pn;
+>>>>>>> MERGE-SOURCE
     }
 
     private long readPageBufferFromJournal(final PageNode pn, final ByteBuffer bb) throws PersistitIOException,
@@ -1829,6 +1858,10 @@
 
         PageNode _previous;
 
+        PageNode(final int volumeHandle, final long pageAddress) {
+            this(volumeHandle, pageAddress, Long.MIN_VALUE, -1);
+        }
+
         PageNode(final int volumeHandle, final long pageAddress, final long journalAddress, final long timestamp) {
             this._volumeHandle = volumeHandle;
             this._pageAddress = pageAddress;
@@ -1949,9 +1982,27 @@
         final static Comparator<PageNode> READ_COMPARATOR = new Comparator<PageNode>() {
 
             @Override
+<<<<<<< TREE
             public int compare(final PageNode a, final PageNode b) {
                 return a.getJournalAddress() > b.getJournalAddress() ? 1 : a.getJournalAddress() < b
                         .getJournalAddress() ? -1 : 0;
+=======
+            public int compare(final PageNode a, final PageNode b) {
+                if (!a.isInvalid() && !b.isInvalid()) {
+                    return a.getJournalAddress() > b.getJournalAddress() ? 1 : a.getJournalAddress() < b
+                            .getJournalAddress() ? -1 : 0;
+                }
+                if (a.isInvalid() && !b.isInvalid()) {
+                    return -1;
+                }
+                if (!a.isInvalid() && b.isInvalid()) {
+                    return 1;
+                }
+                if (a._volumeHandle != b._volumeHandle) {
+                    return a._volumeHandle - b._volumeHandle;
+                }
+                return a._pageAddress > b._pageAddress ? 1 : a._pageAddress < b._pageAddress ? -1 : 0;
+>>>>>>> MERGE-SOURCE
             }
         };
 

=== modified file 'src/main/java/com/persistit/MediatedFileChannel.java'
--- src/main/java/com/persistit/MediatedFileChannel.java	2012-08-15 16:11:36 +0000
+++ src/main/java/com/persistit/MediatedFileChannel.java	2012-08-27 21:56:20 +0000
@@ -342,7 +342,7 @@
      * 
      * @param channel
      */
-    void setErrorInjectingChannelForTests(final FileChannel channel) {
+    void injectChannelForTests(final FileChannel channel) {
         ((TestChannelInjector) channel).setChannel(_channel);
         _channel = channel;
     }

=== modified file 'src/main/java/com/persistit/Persistit.java'
--- src/main/java/com/persistit/Persistit.java	2012-08-15 16:11:36 +0000
+++ src/main/java/com/persistit/Persistit.java	2012-08-27 21:56:20 +0000
@@ -403,6 +403,7 @@
 
     private final AtomicBoolean _suspendShutdown = new AtomicBoolean(false);
     private final AtomicBoolean _suspendUpdates = new AtomicBoolean(false);
+    private final AtomicBoolean _enableBufferInventory = new AtomicBoolean(false);
 
     private UtilControl _localGUI;
 
@@ -588,9 +589,7 @@
                 initializeVolumes();
                 startJournal();
                 startBufferPools();
-                if (_configuration.getBufferInventoryPathName() != null) {
-                    warmupBufferPools();
-                }
+                preloadBufferPools();
                 finishRecovery();
                 startCheckpointManager();
                 startTransactionIndexPollTask();
@@ -699,6 +698,7 @@
         _defaultSplitPolicy = _configuration.getSplitPolicy();
         _defaultJoinPolicy = _configuration.getJoinPolicy();
         _defaultCommitPolicy = _configuration.getCommitPolicy();
+        _enableBufferInventory.set(_configuration.isBufferInventoryEnabled());
     }
 
     void startCheckpointManager() {
@@ -718,11 +718,33 @@
             pool.startThreads();
         }
     }
+<<<<<<< TREE
 
     void warmupBufferPools() throws PersistitException {
         final String pathName = _configuration.getBufferInventoryPathName();
         for (final BufferPool pool : _bufferPoolTable.values()) {
             pool.warmupBufferPool(pathName, pool.toString());
+=======
+
+    void recordBufferPoolInventory() {
+        final long timestamp = _timestampAllocator.getCurrentTimestamp();
+        if (_enableBufferInventory.get()) {
+            for (final BufferPool pool : _bufferPoolTable.values()) {
+                try {
+                    pool.recordBufferInventory(timestamp);
+                } catch (PersistitException e) {
+                    getLogBase().bufferInventoryException.log(e);
+                }
+            }
+        }
+    }
+
+    void preloadBufferPools() throws PersistitException {
+        if (_configuration.isBufferPreloadEnabled()) {
+            for (final BufferPool pool : _bufferPoolTable.values()) {
+                pool.preloadBufferInventory();
+            }
+>>>>>>> MERGE-SOURCE
         }
     }
 
@@ -974,9 +996,13 @@
      * 
      * @throws IllegalStateException
      */
+<<<<<<< TREE
 
     // TODO - why not one pool.
     public void releaseExchange(final Exchange exchange, final boolean secure) {
+=======
+    public void releaseExchange(final Exchange exchange, final boolean secure) {
+>>>>>>> MERGE-SOURCE
         if (exchange == null) {
             return;
         }
@@ -1638,6 +1664,7 @@
                     }
                 }
             }
+            recordBufferPoolInventory();
 
             /*
              * The copier is responsible for background pruning of aborted

=== modified file 'src/main/java/com/persistit/logging/LogBase.java'
--- src/main/java/com/persistit/logging/LogBase.java	2012-08-15 16:11:36 +0000
+++ src/main/java/com/persistit/logging/LogBase.java	2012-08-27 21:56:20 +0000
@@ -244,6 +244,15 @@
     @Message("ERROR|Too many journal files %,d")
     public final LogItem tooManyJournalFilesError = PersistitLogMessage.empty();
 
+    @Message("INFO|Preloading buffer pool inventory recorded at %tc")
+    public final LogItem bufferInventoryLoad = PersistitLogMessage.empty();
+
+    @Message("INFO|Preloaded %,d of %,d buffers")
+    public final LogItem bufferInventoryProgress = PersistitLogMessage.empty();
+
+    @Message("WARNING|Exception while writing buffer pool inventory %s")
+    public final LogItem bufferInventoryException = PersistitLogMessage.empty();
+
     public static String recurring(final String message, final int count, final long duration) {
         return String.format(RECURRING, message, count, duration);
     }

=== modified file 'src/test/java/com/persistit/IOFailureTest.java'
--- src/test/java/com/persistit/IOFailureTest.java	2012-08-15 16:11:36 +0000
+++ src/test/java/com/persistit/IOFailureTest.java	2012-08-27 21:56:20 +0000
@@ -57,7 +57,7 @@
 
     private ErrorInjectingFileChannel errorInjectingChannel(final FileChannel channel) {
         final ErrorInjectingFileChannel eimfc = new ErrorInjectingFileChannel();
-        ((MediatedFileChannel) channel).setErrorInjectingChannelForTests(eimfc);
+        ((MediatedFileChannel) channel).injectChannelForTests(eimfc);
         return eimfc;
     }
 

=== modified file 'src/test/java/com/persistit/StressRunner.java'
--- src/test/java/com/persistit/StressRunner.java	2012-08-15 16:11:36 +0000
+++ src/test/java/com/persistit/StressRunner.java	2012-08-27 21:56:20 +0000
@@ -31,6 +31,7 @@
 import com.persistit.stress.MixtureTxn1;
 import com.persistit.stress.MixtureTxn2;
 import com.persistit.stress.PersistitMap1;
+import com.persistit.stress.PreloadMixtureTxn1;
 import com.persistit.stress.StartStop;
 import com.persistit.stress.Stress10Suite;
 import com.persistit.stress.Stress12txnSuite;
@@ -79,6 +80,7 @@
         _classes.add(Stress12txnSuite.class);
         _classes.add(Stress4Suite.class);
         _classes.add(Stress8txnSuite.class);
+        _classes.add(PreloadMixtureTxn1.class);
     }
 
     private final static String DURATION_PARAM = "duration=";

=== added file 'src/test/java/com/persistit/TrackingFileChannel.java'
--- src/test/java/com/persistit/TrackingFileChannel.java	1970-01-01 00:00:00 +0000
+++ src/test/java/com/persistit/TrackingFileChannel.java	2012-08-27 21:56:20 +0000
@@ -0,0 +1,175 @@
+/**
+ * Copyright © 2012 Akiban Technologies, Inc.  All rights reserved.
+ * 
+ * This program and the accompanying materials are made available
+ * under the terms of the Eclipse Public License v1.0 which
+ * accompanies this distribution, and is available at
+ * http://www.eclipse.org/legal/epl-v10.html
+ * 
+ * This program may also be available under different license terms.
+ * For more information, see www.akiban.com or contact [e-mail].
+ * 
+ * Contributors:
+ * Akiban Technologies, Inc.
+ */
+
+package com.persistit;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.nio.MappedByteBuffer;
+import java.nio.channels.FileChannel;
+import java.nio.channels.FileLock;
+import java.nio.channels.ReadableByteChannel;
+import java.nio.channels.WritableByteChannel;
+import java.util.ArrayList;
+import java.util.List;
+
+import com.persistit.MediatedFileChannel.TestChannelInjector;
+
+/**
+ * <p>
+ * A {@link FileChannel} implementation that simulates IOExceptions under
+ * control of a unit test program. This class implements only those methods used
+ * by Persistit; many methods of FileChannel throw
+ * {@link UnsupportedOperationException}.
+ * </p>
+ * 
+ * @author peter
+ * 
+ */
+class TrackingFileChannel extends FileChannel implements TestChannelInjector {
+
+    volatile FileChannel _channel;
+
+    final List<Long> _writePositions = new ArrayList<Long>();
+
+    final List<Long> _readPositions = new ArrayList<Long>();
+
+    @Override
+    public void setChannel(final FileChannel channel) {
+        _channel = channel;
+    }
+
+    @Override
+    protected void implCloseChannel() throws IOException {
+        _channel.close();
+    }
+
+    @Override
+    public void force(final boolean metaData) throws IOException {
+        _channel.force(metaData);
+    }
+
+    @Override
+    public int read(final ByteBuffer byteBuffer, final long position) throws IOException {
+        _readPositions.add(position);
+        return _channel.read(byteBuffer, position);
+    }
+
+    @Override
+    public long size() throws IOException {
+        return _channel.size();
+    }
+
+    @Override
+    public FileChannel truncate(final long size) throws IOException {
+        return _channel.truncate(size);
+    }
+
+    @Override
+    public synchronized FileLock tryLock(final long position, final long size, final boolean shared) throws IOException {
+        return _channel.tryLock(position, size, shared);
+    }
+
+    @Override
+    public int write(final ByteBuffer byteBuffer, final long position) throws IOException {
+        _writePositions.add(position);
+        final int written = _channel.write(byteBuffer, position);
+        return written;
+    }
+
+    /*
+     * --------------------------------
+     * 
+     * Persistit does not use these methods and so they are Unsupported. Note
+     * that it would be difficult to support the relative read/write methods
+     * because the channel size is unavailable after it is closed. Therefore a
+     * client of this class must maintain its own position counter and cannot
+     * use the relative-addressing calls.
+     * 
+     * --------------------------------
+     */
+    @Override
+    public FileLock lock(final long position, final long size, final boolean shared) throws IOException {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public MappedByteBuffer map(final MapMode arg0, final long arg1, final long arg2) throws IOException {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public long position() throws IOException {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public FileChannel position(final long arg0) throws IOException {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public int read(final ByteBuffer byteBuffer) throws IOException {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public long read(final ByteBuffer[] arg0, final int arg1, final int arg2) throws IOException {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public long transferFrom(final ReadableByteChannel arg0, final long arg1, final long arg2) throws IOException {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public long transferTo(final long arg0, final long arg1, final WritableByteChannel arg2) throws IOException {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public int write(final ByteBuffer byteBuffer) throws IOException {
+        throw new UnsupportedOperationException();
+    }
+
+    @Override
+    public long write(final ByteBuffer[] arg0, final int arg1, final int arg2) throws IOException {
+        throw new UnsupportedOperationException();
+    }
+
+    public List<Long> getWritePositionList() {
+        return _writePositions;
+    }
+
+    public List<Long> getReadPositionList() {
+        return _readPositions;
+    }
+
+    public void assertOrdered(boolean read, boolean forward) {
+        final List<Long> list = read ? _readPositions : _writePositions;
+        long previous = forward ? -1 : Long.MAX_VALUE;
+        for (final Long position : list) {
+            if (forward) {
+                assertTrue("Position should be larger", position > previous);
+            } else {
+                assertTrue("Position should be smaller", position < previous);
+            }
+        }
+    }
+}

=== modified file 'src/test/java/com/persistit/WarmupTest.java'
--- src/test/java/com/persistit/WarmupTest.java	2012-08-15 16:11:36 +0000
+++ src/test/java/com/persistit/WarmupTest.java	2012-08-27 21:56:20 +0000
@@ -15,8 +15,14 @@
 
 package com.persistit;
 
-import static org.junit.Assert.assertEquals;
-
+<<<<<<< TREE
+import static org.junit.Assert.assertEquals;
+
+=======
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+>>>>>>> MERGE-SOURCE
 import java.util.Properties;
 
 import org.junit.Test;
@@ -24,14 +30,28 @@
 import com.persistit.unit.PersistitUnitTestCase;
 
 public class WarmupTest extends PersistitUnitTestCase {
-
+<<<<<<< TREE
+
+=======
+
+    @Override
+    protected Properties getProperties(final boolean cleanup) {
+        final Properties p = super.getProperties(cleanup);
+        p.setProperty("bufferinventory", "true");
+        p.setProperty("bufferpreload", "true");
+        return p;
+    }
+
+>>>>>>> MERGE-SOURCE
     @Test
     public void testWarmup() throws Exception {
         Exchange ex = _persistit.getExchange("persistit", "WarmupTest", true);
+        BufferPool pool = ex.getBufferPool();
         for (int i = 1; i <= 1000; i++) {
             ex.getValue().put(RED_FOX);
             ex.clear().append(i).store();
         }
+<<<<<<< TREE
 
         // Assumption: only one buffer pool is created
         int poolCount = 0;
@@ -63,5 +83,87 @@
             }
         }
         assertEquals(poolCount, poolCount1);
+=======
+
+        final Buffer[] buff = new Buffer[100];
+        for (int i = 0; i < pool.getBufferCount(); ++i) {
+            buff[i] = pool.getBufferCopy(i);
+        }
+
+        final Configuration config = _persistit.getConfiguration();
+        ex = null;
+        _persistit.close();
+
+        _persistit = new Persistit();
+        _persistit.initialize(config);
+        ex = _persistit.getExchange("persistit", "WarmupTest", false);
+        pool = ex.getBufferPool();
+
+        for (int i = 0; i < pool.getBufferCount(); ++i) {
+            final Buffer bufferCopy = pool.getBufferCopy(i);
+            assertEquals(bufferCopy.getPageAddress(), buff[i].getPageAddress());
+            assertEquals(bufferCopy.getPageType(), buff[i].getPageType());
+            assertEquals(bufferCopy.getBufferSize(), buff[i].getBufferSize());
+        }
+    }
+
+    @Test
+    public void readOrderIsSequential() throws Exception {
+
+        Exchange ex = _persistit.getExchange("persistit", "WarmupTest", true);
+        BufferPool pool = ex.getBufferPool();
+
+        final int full = pool.getBufferCount() * (pool.getBufferSize() / RED_FOX.length());
+        /*
+         * Overflow the buffer pool
+         */
+        for (int i = 1; i <= full * 3; i++) {
+            ex.getValue().put(RED_FOX);
+            ex.clear().append(i).store();
+        }
+        /*
+         * Pull some low-address pages in to scramble the pool
+         */
+        for (int i = full * 2; i >= 0; i -= 1000) {
+            ex.clear().append(i).fetch();
+        }
+        /*
+         * Verify that buffers in pool now have somewhat scrambled page
+         * addresses
+         */
+        int breaks = 0;
+        long previous = -1;
+
+        for (int i = 0; i < pool.getBufferCount(); i++) {
+            final Buffer b = pool.getBufferCopy(i);
+            assertTrue("Every buffer should be valid at this point", b.isValid());
+            if (b.getPageAddress() < previous) {
+                breaks++;
+            }
+            previous = b.getPageAddress();
+        }
+
+        assertTrue("Buffer pool should have scrambled page address", breaks > 0);
+
+        final Configuration config = _persistit.getConfiguration();
+        ex = null;
+        pool = null;
+        _persistit.copyBackPages();
+        _persistit.close();
+
+        _persistit = new Persistit();
+        config.setBufferInventoryEnabled(false);
+        config.setBufferPreloadEnabled(false);
+        _persistit.initialize(config);
+
+        final Volume volume = _persistit.getVolume("persistit");
+        final MediatedFileChannel mfc = (MediatedFileChannel) volume.getStorage().getChannel();
+        final TrackingFileChannel tfc = new TrackingFileChannel();
+        mfc.injectChannelForTests(tfc);
+        pool = volume.getStructure().getPool();
+        pool.preloadBufferInventory();
+        assertTrue("Preload should have loaded pages from journal file", tfc.getReadPositionList().size() > 0);
+        tfc.assertOrdered(true, true);
+>>>>>>> MERGE-SOURCE
     }
 }

=== added file 'src/test/java/com/persistit/stress/PreloadMixtureTxn1.java'
--- src/test/java/com/persistit/stress/PreloadMixtureTxn1.java	1970-01-01 00:00:00 +0000
+++ src/test/java/com/persistit/stress/PreloadMixtureTxn1.java	2012-08-27 21:56:20 +0000
@@ -0,0 +1,82 @@
+/**
+ * Copyright © 2012 Akiban Technologies, Inc.  All rights reserved.
+ * 
+ * This program and the accompanying materials are made available
+ * under the terms of the Eclipse Public License v1.0 which
+ * accompanies this distribution, and is available at
+ * http://www.eclipse.org/legal/epl-v10.html
+ * 
+ * This program may also be available under different license terms.
+ * For more information, see www.akiban.com or contact [e-mail].
+ * 
+ * Contributors:
+ * Akiban Technologies, Inc.
+ */
+
+package com.persistit.stress;
+
+import com.persistit.Configuration;
+import com.persistit.Persistit;
+import com.persistit.Transaction.CommitPolicy;
+import com.persistit.stress.unit.Stress1;
+import com.persistit.stress.unit.Stress2txn;
+import com.persistit.stress.unit.Stress3;
+import com.persistit.stress.unit.Stress3txn;
+import com.persistit.stress.unit.Stress5;
+import com.persistit.stress.unit.Stress6;
+import com.persistit.stress.unit.Stress8txn;
+
+public class PreloadMixtureTxn1 extends AbstractSuite {
+    private final static int CYCLES = 4;
+
+    static String name() {
+        return PreloadMixtureTxn1.class.getSimpleName();
+    }
+
+    public static void main(final String[] args) throws Exception {
+        new PreloadMixtureTxn1(args).runTest();
+    }
+
+    public PreloadMixtureTxn1(final String[] args) {
+        super(name(), args);
+        setDuration(getDuration() / CYCLES);
+    }
+
+    @Override
+    public void runTest() throws Exception {
+
+        deleteFiles(substitute("$datapath$/persistit*"));
+
+        for (int iteration = 0; iteration < CYCLES; iteration++) {
+            clear();
+            add(new Stress1("repeat=10 count=25000"));
+            add(new Stress1("repeat=10 count=25000"));
+            add(new Stress2txn("repeat=10 count=2500 size=4000 seed=118"));
+            add(new Stress2txn("repeat=2 count=25000 seed=119"));
+            add(new Stress3("repeat=5 count=25000 seed=119"));
+            add(new Stress3txn("repeat=5 count=25000 seed=120"));
+            add(new Stress3txn("repeat=5 count=25000"));
+            add(new Stress5("repeat=5 count=25000"));
+            add(new Stress6("repeat=5 count=1000 size=250"));
+            add(new Stress6("repeat=10 count=1000 size=250"));
+            add(new Stress8txn("repeat=2 count=1000 size=1000 seed=1"));
+            add(new Stress8txn("repeat=2 count=1000 size=1000 seed=2"));
+            add(new Stress8txn("repeat=2 count=1000 size=1000 seed=3"));
+            add(new Stress8txn("repeat=2 count=1000 size=1000 seed=4"));
+
+            final Configuration configuration = makeConfiguration(16384, "25000", CommitPolicy.SOFT);
+            configuration.setLogFile(configuration.getLogFile() + "_" + iteration);
+            configuration.setBufferInventoryEnabled(true);
+            configuration.setBufferPreloadEnabled(true);
+
+            final Persistit persistit = new Persistit();
+            persistit.initialize(configuration);
+
+            try {
+                execute(persistit);
+            } finally {
+                persistit.close();
+            }
+        }
+    }
+}

=== modified file 'src/test/java/com/persistit/unit/PersistitUnitTestCase.java'
--- src/test/java/com/persistit/unit/PersistitUnitTestCase.java	2012-08-15 16:11:36 +0000
+++ src/test/java/com/persistit/unit/PersistitUnitTestCase.java	2012-08-27 21:56:20 +0000
@@ -47,7 +47,8 @@
     protected Persistit _persistit = new Persistit();
 
     protected Properties getProperties(final boolean cleanup) {
-        return UnitTestProperties.getProperties(cleanup);
+        final Properties p = UnitTestProperties.getProperties(cleanup);
+        return p;
     }
 
     @Before

=== modified file 'src/test/java/com/persistit/unit/UnitTestProperties.java'
--- src/test/java/com/persistit/unit/UnitTestProperties.java	2012-08-07 21:11:28 +0000
+++ src/test/java/com/persistit/unit/UnitTestProperties.java	2012-08-27 21:56:20 +0000
@@ -43,7 +43,6 @@
         p.setProperty("tmpvoldir", "${datapath}");
         p.setProperty("rmiport", System.getProperty("rmiport", "8081"));
         p.setProperty("jmx", "true");
-        p.setProperty("bufferinventory", "/tmp/persistit_test_data");
         return p;
     }