Merge lp:~pbeaman/akiban-persistit/fix-1126868-lock-table-pruning into lp:akiban-persistit

[Peter Beaman]

Fixes bug1126868. The bug is that in a typical data loading scenario, or in any process where update transactions are performing a large volume of locks, the lock volume can be overwhelmed with new entries in such a way that obsolete entries are never pruned. When that happens the lock volume grows rapidly, and in one server test exceeds the 2GB sanity limit for temporary volumes.

A new test in ExchangeLockTest demonstrates the phenomenon by simply creating a large number of locks (10,000) in each of 500 transactions. Prior to the fix this test causes runaway growth of the lock volume.

This proposal fixes this phenomenon by adding a new pruning mode in which all locks are synchronously pruned upon when a transaction that has used any locks either commits or aborts.

The majority of the changes provide plumbing so that a call to Buffer#pruneMvvValues can return a List of consequent CleanupAction instances rather than enqueuing them on the CleanupManager. These are the applied during the call to Transaction#end so that the transaction cannot leave unpruned locks. In general this is efficient because the lock volume _should be_ small, and therefore memory resident. We hope never to see lock volume pages written to disk.

This branch also fixes a couple of related issues discovered while diagnosing and fixing the problem.

(1) Temporary volumes are not designed to be threadsafe. So there is now a new VolumeStorageL2 class that combines behavior of VolumeStorageT2 and VolumeStorageV2. In particular, VolumeStorageL2 has a head page that is used for synchronization. Also, since there is at most a single lock volume in the system, it now has a distinguished name, and it appears on the visible volumes list in the UI.

(2) The UI was using the volume's path as its identifier. This has been wrong, but unnoticed, for some time. The UI now uses the name as the unique identifier, and it now possible to examine trees in the lock volume using the UI. One minor change in the AdminUI.properties file swaps the name and path columns of the UI's main page.

[Nathan Williams (nwilliams)]

Does pruneLockPages() really need to call updateActiveTransactionCache()? It already happens every 10ms and doing it synchronously seems unnecessary.

VolumeStorageT2 and L2 are 99% identical. Can we pull some of that out into a base class, particularly the non-trivial stuff like read and write page? The javadoc also wasn't changed for L2.

Otherwise looks fine.

review: Needs Fixing

[Peter Beaman (pbeaman)]

Yes, removing the call to updateActiveTransactionCache makes the test fail. My hope is that by adding an update timestamp condition to it, there will be many instances where the update will happen asynchronously while the commit is being completed so that on a busy system we don't add an overwhelming number of synchronous executions.

Good point about L2. It is now a subclass of T2 with almost no code and corrected Javadoc.

[Nathan Williams (nwilliams)]

Making transactions that perform writes slower by doing the cache update synchronously seems like a bad conscious choice to make. Bucket locking, which recomputing the cache does, is exclusive right now. I have a feeling this will really hurt concurrency.

Do we have any perf tests for this or am I being too pessimistic?

[Peter Beaman (pbeaman)]

I don't like it either, but I couldn't find another way to solve the
problem. I guess the question we need to ask is how many transactions will
actually lock something. Certainly no read transactions. Can you suggest
a good multi-thread concurrent server benchmark?

On Mon, Mar 11, 2013 at 4:08 PM, Nathan Williams <email address hidden>wrote:

> Making transactions that perform writes slower by doing the cache update
> synchronously seems like a bad conscious choice to make. Bucket locking,
> which recomputing the cache does, is exclusive right now. I have a feeling
> this will really hurt concurrency.
>
> Do we have any perf tests for this or am I being too pessimistic?
> --
>
> https://code.launchpad.net/~pbeaman/akiban-persistit/fix-1126868-lock-table-pruning/+merge/152063
> You are the owner of
> lp:~pbeaman/akiban-persistit/fix-1126868-lock-table-pruning.
>

[Thomas Jones-Low (tjoneslo)]

The Multi-Thread Test (MTT) in the akiban-build-scripts repository is designed for this purpose.

[Nathan Williams (nwilliams)]

Peter and I discussed this offline. Merging this now makes sense and we can double check the data-loading-test works as desired.

We can confirm concurrency impact in parallel/after and if there is a problem, there is a bucket level change that probably makes sense.

review: Approve

[Akiban Build User (build-akiban)]

There were 2 failures during build/test:

* job server-build failed at build number 3869: http://172.16.20.104:8080/job/server-build/3869/

* view must-pass failed: server-build is yellow

review: Needs Fixing

[Peter Beaman (pbeaman)]

Approving this again. The failure is strange, and the line numbers displayed in the stack traces don't match the code. I will watch this build.

[Akiban Build User (build-akiban)]

There were 2 failures during build/test:

* job server-build failed at build number 3879: http://172.16.20.104:8080/job/server-build/3879/

* view must-pass failed: server-build is yellow

review: Needs Fixing

[Peter Beaman (pbeaman)]

Server tests run fine on local machine, but fail on build machine. Adding better logging and Approving in attempt to diagnose the problem.

[Akiban Build User (build-akiban)]

There were 2 failures during build/test:

* job server-build failed at build number 3881: http://172.16.20.104:8080/job/server-build/3881/

* view must-pass failed: server-build is yellow

review: Needs Fixing

[lp:~pbeaman/akiban-persistit/fix-1126868-lock-table-pruning]

434. By Peter Beaman on 2013-03-13

Correct BufferSizeUnavailableException cases

435. By Peter Beaman on 2013-03-13

Get buffer pool from volume structure instead of lookup by page size

[Peter Beaman (pbeaman)]

Trying yet again with some additional changes.

=== modified file 'src/main/java/com/persistit/Buffer.java'
--- src/main/java/com/persistit/Buffer.java	2013-01-19 17:52:20 +0000
+++ src/main/java/com/persistit/Buffer.java	2013-03-13 17:42:20 +0000
@@ -26,6 +26,7 @@
 import java.util.List;
 import java.util.Set;
 
+import com.persistit.CleanupManager.CleanupAction;
 import com.persistit.CleanupManager.CleanupAntiValue;
 import com.persistit.Exchange.Sequence;
 import com.persistit.JournalRecord.IV;
@@ -522,7 +523,7 @@
         final Volume volume = getVolume();
         if (volume != null) {
             if (prune) {
-                pruneMvvValues(null, false);
+                pruneMvvValues(null, false, null);
             }
             clearSlack();
             save();
@@ -3599,7 +3600,8 @@
      *         buffer
      * @throws PersistitException
      */
-    boolean pruneMvvValues(final Tree tree, final boolean pruneLongMVVs) throws PersistitException {
+    boolean pruneMvvValues(final Tree tree, final boolean pruneLongMVVs, final List<CleanupAction> cleanupActions)
+            throws PersistitException {
         boolean changed = false;
         try {
             boolean hasLongMvvRecords = false;
@@ -3611,7 +3613,7 @@
                 final long timestamp = _persistit.getTimestampAllocator().updateTimestamp();
                 _mvvCount = 0;
                 writePageOnCheckpoint(timestamp);
-                final int flags = pruneMvvValuesHelper(tree);
+                final int flags = pruneMvvValuesHelper(tree, cleanupActions);
                 changed = (flags & PRUNE_MVV_HELPER_CHANGED) != 0;
                 hasLongMvvRecords = (flags & PRUNE_MVV_HELPER_HAS_LONG) != 0;
 
@@ -3627,7 +3629,7 @@
 
                     final Buffer copy = new Buffer(this);
                     final boolean copyChanged = copy.pruneLongMvvValues(tree, prunedVersions, deferredExceptions,
-                            oldChainsToDeallocate);
+                            oldChainsToDeallocate, cleanupActions);
                     if (copyChanged) {
                         changed = true;
                         final long copyTimestamp = _persistit.getTimestampAllocator().updateTimestamp();
@@ -3662,7 +3664,8 @@
         return changed;
     }
 
-    private int pruneMvvValuesHelper(final Tree tree) throws PersistitException {
+    private int pruneMvvValuesHelper(final Tree tree, final List<CleanupAction> cleanupActions)
+            throws PersistitException {
         boolean changed = false;
         boolean hasLongMvvRecords = false;
         final List<PrunedVersion> prunedVersions = new ArrayList<PrunedVersion>();
@@ -3708,7 +3711,7 @@
                     incCountIfMvv(_bytes, offset, newSize);
                 }
 
-                if (pruneAntiValue(valueByte, p, tree)) {
+                if (pruneAntiValue(valueByte, p, tree, cleanupActions)) {
                     changed = true;
                     p -= KEYBLOCK_LENGTH;
                 }
@@ -3726,7 +3729,8 @@
      * successful completion of this process.
      */
     private boolean pruneLongMvvValues(final Tree tree, final List<PrunedVersion> prunedVersions,
-            final List<PersistitException> deferredExceptions, final List<Long> toDeallocate) {
+            final List<PersistitException> deferredExceptions, final List<Long> toDeallocate,
+            final List<CleanupAction> cleanupActions) {
 
         boolean changed = false;
         for (int p = KEY_BLOCK_START; p < _keyBlockEnd; p += KEYBLOCK_LENGTH) {
@@ -3768,7 +3772,7 @@
                         value.changeLongRecordMode(false);
                     }
                 }
-                if (pruneAntiValue(valueByte, p, tree)) {
+                if (pruneAntiValue(valueByte, p, tree, cleanupActions)) {
                     changed = true;
                     p -= KEYBLOCK_LENGTH;
                 }
@@ -3781,13 +3785,16 @@
         return changed;
     }
 
-    private boolean pruneAntiValue(final int valueByte, final int p, final Tree tree) {
+    private boolean pruneAntiValue(final int valueByte, final int p, final Tree tree,
+            final List<CleanupAction> cleanupActions) {
         if (valueByte == MVV.TYPE_ANTIVALUE) {
             if (p == KEY_BLOCK_START) {
                 if (tree != null) {
-                    if (!_enqueuedForAntiValuePruning) {
-                        final int treeHandle = tree.getHandle();
-                        assert treeHandle != 0 : "MVV found in a temporary tree " + tree;
+                    final int treeHandle = tree.getHandle();
+                    assert treeHandle != 0 : "MVV found in a temporary tree " + tree;
+                    if (cleanupActions != null) {
+                        cleanupActions.add(new CleanupAntiValue(treeHandle, getPageAddress()));
+                    } else if (!_enqueuedForAntiValuePruning) {
                         if (_persistit.getCleanupManager().offer(new CleanupAntiValue(treeHandle, getPageAddress()))) {
                             _enqueuedForAntiValuePruning = true;
                         }

=== modified file 'src/main/java/com/persistit/BufferPool.java'
--- src/main/java/com/persistit/BufferPool.java	2013-01-28 10:42:29 +0000
+++ src/main/java/com/persistit/BufferPool.java	2013-03-13 17:42:20 +0000
@@ -789,7 +789,7 @@
                     // page will find it.
                     //
                     buffer.setValid();
-                    if (vol.isTemporary()) {
+                    if (vol.isTemporary() || vol.isLockVolume()) {
                         buffer.setTemporary();
                     } else {
                         buffer.clearTemporary();
@@ -1468,7 +1468,7 @@
                         }
                         Util.spinSleep();
                     }
-                    if (volume1 != null && !volume1.isTemporary()) {
+                    if (volume1 != null && !volume1.isTemporary() && !volume1.isLockVolume()) {
                         value.clear().setStreamMode(true);
                         value.put(volume1.getHandle());
                         value.put(page1);

=== modified file 'src/main/java/com/persistit/CleanupManager.java'
--- src/main/java/com/persistit/CleanupManager.java	2012-11-26 17:19:45 +0000
+++ src/main/java/com/persistit/CleanupManager.java	2013-03-13 17:42:20 +0000
@@ -33,7 +33,7 @@
 
     interface CleanupAction extends Comparable<CleanupAction> {
 
-        void performAction(Persistit persistit) throws PersistitException;
+        void performAction(Persistit persistit, List<CleanupAction> consequentActions) throws PersistitException;
     }
 
     final static long DEFAULT_CLEANUP_INTERVAL_MS = 1000;
@@ -177,7 +177,7 @@
 
         for (final CleanupAction action : workList) {
             try {
-                action.performAction(_persistit);
+                action.performAction(_persistit, null);
                 _performed.incrementAndGet();
             } catch (final PersistitException e) {
                 lastException(e);
@@ -227,7 +227,11 @@
             } else {
                 return false;
             }
+        }
 
+        @Override
+        public int hashCode() {
+            return (int) (_treeHandle ^ _page);
         }
 
         @Override
@@ -275,10 +279,11 @@
         }
 
         @Override
-        public void performAction(final Persistit persistit) throws PersistitException {
+        public void performAction(final Persistit persistit, final List<CleanupAction> consequentActions)
+                throws PersistitException {
             final Exchange exchange = getExchange(persistit);
             if (exchange != null) {
-                exchange.pruneLeftEdgeValue(_page);
+                exchange.pruneLeftEdgeValue(_page, consequentActions);
             }
         }
     }
@@ -290,10 +295,11 @@
         }
 
         @Override
-        public void performAction(final Persistit persistit) throws PersistitException {
+        public void performAction(final Persistit persistit, final List<CleanupAction> consequentActions)
+                throws PersistitException {
             final Exchange exchange = getExchange(persistit);
             if (exchange != null) {
-                exchange.prune(_page);
+                exchange.prune(_page, consequentActions);
             }
         }
     }
@@ -307,7 +313,8 @@
         }
 
         @Override
-        public void performAction(final Persistit persistit) throws PersistitException {
+        public void performAction(final Persistit persistit, final List<CleanupAction> consequentActions)
+                throws PersistitException {
             final Exchange exchange = getExchange(persistit);
             if (exchange != null) {
                 exchange.fixIndexHole(_page, _level);

=== modified file 'src/main/java/com/persistit/Exchange.java'
--- src/main/java/com/persistit/Exchange.java	2013-01-24 18:18:40 +0000
+++ src/main/java/com/persistit/Exchange.java	2013-03-13 17:42:20 +0000
@@ -40,11 +40,13 @@
 import java.util.ArrayList;
 import java.util.List;
 
+import com.persistit.CleanupManager.CleanupAction;
 import com.persistit.Key.Direction;
 import com.persistit.MVV.PrunedVersion;
 import com.persistit.ValueHelper.MVVValueWriter;
 import com.persistit.ValueHelper.RawValueWriter;
 import com.persistit.VolumeStructure.Chain;
+import com.persistit.exception.BufferSizeUnavailableException;
 import com.persistit.exception.CorruptVolumeException;
 import com.persistit.exception.InUseException;
 import com.persistit.exception.PersistitException;
@@ -400,6 +402,7 @@
      * 
      * @param tree
      *            The <code>Tree</code> to access.
+     * @throws BufferSizeUnavailableException
      */
     public Exchange(final Tree tree) {
         this(tree._persistit);
@@ -427,7 +430,8 @@
         final Volume volume = tree.getVolume();
         _ignoreTransactions = volume.isTemporary();
         _ignoreMVCCFetch = false;
-        _pool = _persistit.getBufferPool(volume.getPageSize());
+        _pool = volume.getStructure().getPool();
+
         _transaction = _persistit.getTransaction();
         _key.clear();
         _value.clear();
@@ -1640,7 +1644,7 @@
                     if (splitRequired && !treeClaimAcquired) {
                         if (!didPrune && buffer.isDataPage()) {
                             didPrune = true;
-                            if (buffer.pruneMvvValues(_tree, false)) {
+                            if (buffer.pruneMvvValues(_tree, false, null)) {
                                 continue;
                             }
                         }
@@ -1856,7 +1860,6 @@
         Debug.$assert0.t((buffer.getStatus() & SharedResource.WRITER_MASK) != 0
                 && (buffer.getStatus() & SharedResource.CLAIMED_MASK) != 0);
         final Sequence sequence = lc.sequence(foundAt);
-
         long timestamp = timestamp();
         buffer.writePageOnCheckpoint(timestamp);
 
@@ -2853,6 +2856,9 @@
         lockExchange.getValue().clear().putAntiValueMVV();
         final int options = StoreOptions.WAIT | StoreOptions.DONT_JOURNAL | StoreOptions.MVCC;
         lockExchange.storeInternal(lockExchange.getKey(), lockExchange.getValue(), 0, options);
+        final long page = lockExchange._levelCache[0]._page;
+        _transaction.addLockPage(page, lockExchange.getTree().getHandle());
+        _persistit.releaseExchange(lockExchange);
     }
 
     /**
@@ -3993,7 +3999,7 @@
             search(key, true);
             buffer = _levelCache[0]._buffer;
             if (buffer != null) {
-                return buffer.pruneMvvValues(_tree, true);
+                return buffer.pruneMvvValues(_tree, true, null);
             } else {
                 return false;
             }
@@ -4015,7 +4021,7 @@
 
             while (buffer != null) {
                 checkPageType(buffer, Buffer.PAGE_TYPE_DATA, false);
-                pruned |= buffer.pruneMvvValues(_tree, true);
+                pruned |= buffer.pruneMvvValues(_tree, true, null);
                 final int foundAt = buffer.findKey(key2);
                 if (!buffer.isAfterRightEdge(foundAt)) {
                     break;
@@ -4037,11 +4043,11 @@
         return pruned;
     }
 
-    boolean prune(final long page) throws PersistitException {
+    boolean prune(final long page, final List<CleanupAction> consequentActions) throws PersistitException {
         Buffer buffer = null;
         try {
             buffer = _pool.get(_volume, page, true, true);
-            return buffer.pruneMvvValues(_tree, true);
+            return buffer.pruneMvvValues(_tree, true, consequentActions);
         } finally {
             if (buffer != null) {
                 buffer.release();
@@ -4049,7 +4055,7 @@
         }
     }
 
-    boolean pruneLeftEdgeValue(final long page) throws PersistitException {
+    boolean pruneLeftEdgeValue(final long page, final List<CleanupAction> consequentActions) throws PersistitException {
         _ignoreTransactions = true;
         Buffer buffer = null;
         try {

=== modified file 'src/main/java/com/persistit/IntegrityCheck.java'
--- src/main/java/com/persistit/IntegrityCheck.java	2012-09-12 22:02:22 +0000
+++ src/main/java/com/persistit/IntegrityCheck.java	2013-03-13 17:42:20 +0000
@@ -1170,7 +1170,7 @@
                     _counters._mvvPageCount++;
                     if (_prune && !_currentVolume.isReadOnly() && _counters._pruningErrorCount < MAX_PRUNING_ERRORS) {
                         try {
-                            buffer.pruneMvvValues(tree, true);
+                            buffer.pruneMvvValues(tree, true, null);
                             _counters._prunedPageCount++;
                         } catch (final PersistitException e) {
                             _counters._pruningErrorCount++;

=== modified file 'src/main/java/com/persistit/Management.java'
--- src/main/java/com/persistit/Management.java	2012-08-31 14:06:47 +0000
+++ src/main/java/com/persistit/Management.java	2013-03-13 17:42:20 +0000
@@ -464,7 +464,7 @@
      *         occurred while attempting to acquire the Class.
      * @throws RemoteException
      */
-    public Class getRemoteClass(String className) throws RemoteException;
+    public Class<?> getRemoteClass(String className) throws RemoteException;
 
     /**
      * Return the <code>VolumeInfo</code> for the volume specified by the
@@ -1450,7 +1450,7 @@
      * returns information about all open Volumes in an array of
      * <code>VolumeInfo</code> elements.
      */
-    public static class VolumeInfo extends AcquisitionTimeBase implements Serializable {
+    public static class VolumeInfo extends AcquisitionTimeBase implements Serializable, Comparable<VolumeInfo> {
         public final static long serialVersionUID = -1231320633942497896L;
 
         int pageSize;
@@ -1785,7 +1785,12 @@
         @Override
         public boolean equals(final Object object) {
             return object instanceof VolumeInfo && ((VolumeInfo) object).getId() == id
-                    && ((VolumeInfo) object).getPath().equals(path);
+                    && ((VolumeInfo) object).getName().equals(name);
+        }
+
+        @Override
+        public int compareTo(final VolumeInfo volumeInfo) {
+            return name.compareTo(volumeInfo.name);
         }
     }
 

=== modified file 'src/main/java/com/persistit/ManagementImpl.java'
--- src/main/java/com/persistit/ManagementImpl.java	2012-08-24 13:57:19 +0000
+++ src/main/java/com/persistit/ManagementImpl.java	2013-03-13 17:42:20 +0000
@@ -22,6 +22,7 @@
 import java.rmi.registry.LocateRegistry;
 import java.rmi.server.UnicastRemoteObject;
 import java.util.ArrayList;
+import java.util.Arrays;
 import java.util.HashMap;
 import java.util.Iterator;
 import java.util.List;
@@ -798,6 +799,7 @@
         for (int index = 0; index < volumes.size(); index++) {
             result[index] = new VolumeInfo(volumes.get(index));
         }
+        Arrays.sort(result);
         return result;
     }
 

=== modified file 'src/main/java/com/persistit/Persistit.java'
--- src/main/java/com/persistit/Persistit.java	2013-02-26 17:23:35 +0000
+++ src/main/java/com/persistit/Persistit.java	2013-03-13 17:42:20 +0000
@@ -1079,15 +1079,7 @@
      * @throws PersistitException
      */
     public Volume createTemporaryVolume() throws PersistitException {
-        int pageSize = _configuration.getTmpVolPageSize();
-        if (pageSize == 0) {
-            for (final int size : _bufferPoolTable.keySet()) {
-                if (size > pageSize) {
-                    pageSize = size;
-                }
-            }
-        }
-        return createTemporaryVolume(pageSize);
+        return createTemporaryVolume(temporaryVolumePageSize());
     }
 
     /**
@@ -1115,6 +1107,18 @@
         return Volume.createTemporaryVolume(this, pageSize, directory);
     }
 
+    private int temporaryVolumePageSize() {
+        int pageSize = _configuration.getTmpVolPageSize();
+        if (pageSize == 0) {
+            for (final int size : _bufferPoolTable.keySet()) {
+                if (size > pageSize) {
+                    pageSize = size;
+                }
+            }
+        }
+        return pageSize;
+    }
+
     /**
      * Delete a volume currently loaded volume and remove it from the list
      * returned by {@link #getVolumes()}.
@@ -1296,8 +1300,14 @@
         checkInitialized();
         checkClosed();
         if (_lockVolume == null) {
-            _lockVolume = createTemporaryVolume();
-            _lockVolume.setHandle(Volume.LOCK_VOLUME_HANDLE);
+            final int pageSize = temporaryVolumePageSize();
+            if (!Volume.isValidPageSize(pageSize)) {
+                throw new IllegalArgumentException("Invalid page size " + pageSize);
+            }
+            final String directoryName = getConfiguration().getTmpVolDir();
+            final File directory = directoryName == null ? null : new File(directoryName);
+            _lockVolume = Volume.createLockVolume(this, pageSize, directory);
+            _volumes.add(_lockVolume);
         }
         return _lockVolume;
     }

=== modified file 'src/main/java/com/persistit/Transaction.java'
--- src/main/java/com/persistit/Transaction.java	2013-02-20 16:52:45 +0000
+++ src/main/java/com/persistit/Transaction.java	2013-03-13 17:42:20 +0000
@@ -21,8 +21,14 @@
 import static com.persistit.util.ThreadSequencer.sequence;
 
 import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
 
 import com.persistit.Accumulator.Delta;
+import com.persistit.CleanupManager.CleanupAction;
+import com.persistit.CleanupManager.CleanupPruneAction;
 import com.persistit.JournalRecord.D0;
 import com.persistit.JournalRecord.D1;
 import com.persistit.JournalRecord.DR;
@@ -434,6 +440,8 @@
 
     private String _threadName;
 
+    private final Set<CleanupAction> _lockCleanupActions = new HashSet<CleanupAction>();
+
     public static enum CommitPolicy {
         /**
          * The {@link Transaction#commit} method returns before all updates have
@@ -675,6 +683,11 @@
                 _commitCount++;
                 _rollbacksSinceLastCommit = 0;
             }
+            try {
+                pruneLockPages();
+            } catch (Exception e) {
+                _persistit.getLogBase().pruneException.log(e, "locks");
+            }
             _transactionStatus = null;
             _rollbackPending = false;
             _threadName = null;
@@ -1317,6 +1330,32 @@
         return treeHandle;
     }
 
+    void addLockPage(final Long page, final int treeHandle) {
+        _lockCleanupActions.add(new CleanupPruneAction(treeHandle, page));
+    }
+
+    void pruneLockPages() {
+        if (_lockCleanupActions.isEmpty()) {
+            return;
+        }
+
+        _persistit.getTransactionIndex().updateActiveTransactionCache(_commitTimestamp);
+        List<CleanupAction> actions = new ArrayList<CleanupAction>(_lockCleanupActions);
+        _lockCleanupActions.clear();
+
+        while (!actions.isEmpty()) {
+            final List<CleanupAction> consequentActions = new ArrayList<CleanupAction>();
+            for (final CleanupAction cleanupAction : actions) {
+                try {
+                    cleanupAction.performAction(_persistit, consequentActions);
+                } catch (final PersistitException pe) {
+                    _persistit.getLogBase().pruneException.log(pe, this);
+                }
+                actions = consequentActions;
+            }
+        }
+    }
+
     /**
      * For unit tests only
      * 

=== modified file 'src/main/java/com/persistit/TransactionIndex.java'
--- src/main/java/com/persistit/TransactionIndex.java	2012-10-26 19:44:17 +0000
+++ src/main/java/com/persistit/TransactionIndex.java	2013-03-13 17:42:20 +0000
@@ -969,6 +969,18 @@
         }
     }
 
+    /**
+     * Update the floor only if it is smaller than the supplied timestamp.
+     * 
+     * @param ts
+     *            The timestamp
+     */
+    public void updateActiveTransactionCache(final long ts) {
+        if (_atCache._floor < ts) {
+            updateActiveTransactionCache();
+        }
+    }
+
     /*
      * (non-Javadoc)
      * 

=== modified file 'src/main/java/com/persistit/Tree.java'
--- src/main/java/com/persistit/Tree.java	2013-01-19 17:42:22 +0000
+++ src/main/java/com/persistit/Tree.java	2013-03-13 17:42:20 +0000
@@ -244,8 +244,8 @@
      */
     @Override
     public String toString() {
-        return "<Tree " + _name + " rootPageAddr=" + _rootPageAddr + " depth=" + _depth + " status="
-                + getStatusDisplayString() + ">";
+        return "<Tree " + _name + " in volume " + _volume.getName() + " rootPageAddr=" + _rootPageAddr + " depth="
+                + _depth + " status=" + getStatusDisplayString() + ">";
     }
 
     /**

=== modified file 'src/main/java/com/persistit/Volume.java'
--- src/main/java/com/persistit/Volume.java	2013-01-19 17:42:22 +0000
+++ src/main/java/com/persistit/Volume.java	2013-03-13 17:42:20 +0000
@@ -20,6 +20,7 @@
 import java.util.concurrent.atomic.AtomicInteger;
 import java.util.concurrent.atomic.AtomicReference;
 
+import com.persistit.exception.BufferSizeUnavailableException;
 import com.persistit.exception.InUseException;
 import com.persistit.exception.PersistitException;
 import com.persistit.exception.ReadOnlyVolumeException;
@@ -49,6 +50,7 @@
 public class Volume {
 
     final static int LOCK_VOLUME_HANDLE = Integer.MAX_VALUE;
+    final static String LOCK_VOLUME_NAME = "_lock";
 
     private final String _name;
     private long _id;
@@ -86,7 +88,21 @@
         final Volume volume = new Volume(
                 Thread.currentThread().getName() + TEMP_VOLUME_NAME_SUFFIX_FOR_FIXUP_DETECTION,
                 TEMP_VOLUME_ID_FOR_FIXUP_DETECTION);
-        volume.openTemporary(persistit, pageSize, tempDirectory);
+        volume.openInternal(persistit, pageSize);
+
+        volume._storage = new VolumeStorageT2(persistit, volume, tempDirectory);
+        volume._storage.create();
+
+        return volume;
+    }
+
+    static Volume createLockVolume(final Persistit persistit, final int pageSize, final File tempDirectory)
+            throws PersistitException {
+        final Volume volume = new Volume(LOCK_VOLUME_NAME, 0);
+        volume.setHandle(Volume.LOCK_VOLUME_HANDLE);
+        volume.openInternal(persistit, pageSize);
+        volume._storage = new VolumeStorageL2(persistit, volume, tempDirectory);
+        volume._storage.create();
         return volume;
     }
 
@@ -452,7 +468,7 @@
             throw new UnderSpecifiedVolumeException(getName());
         }
         if (persistit.getBufferPool(_specification.getPageSize()) == null) {
-            throw new IllegalStateException(getName());
+            throw new BufferSizeUnavailableException(getName());
         }
         final boolean exists = VolumeHeader.verifyVolumeHeader(_specification, persistit.getCurrentTimestamp());
 
@@ -485,21 +501,16 @@
         persistit.addVolume(this);
     }
 
-    void openTemporary(final Persistit persistit, final int pageSize, final File tempDirectory)
-            throws PersistitException {
+    private void openInternal(final Persistit persistit, final int pageSize) throws PersistitException {
         checkClosing();
         if (_storage != null) {
             throw new IllegalStateException("This volume has already been opened");
         }
         if (persistit.getBufferPool(pageSize) == null) {
-            throw new IllegalStateException("There is no buffer pool for pages of size " + pageSize);
+            throw new BufferSizeUnavailableException("There is no buffer pool for pages of size " + pageSize);
         }
-
         _structure = new VolumeStructure(persistit, this, pageSize);
-        _storage = new VolumeStorageT2(persistit, this, tempDirectory);
         _statistics = new VolumeStatistics();
-
-        _storage.create();
     }
 
     public String getName() {

=== added file 'src/main/java/com/persistit/VolumeStorageL2.java'
--- src/main/java/com/persistit/VolumeStorageL2.java	1970-01-01 00:00:00 +0000
+++ src/main/java/com/persistit/VolumeStorageL2.java	2013-03-13 17:42:20 +0000
@@ -0,0 +1,89 @@
+/**
+ * Copyright © 2011-2012 Akiban Technologies, Inc.  All rights reserved.
+ * 
+ * This program and the accompanying materials are made available
+ * under the terms of the Eclipse Public License v1.0 which
+ * accompanies this distribution, and is available at
+ * http://www.eclipse.org/legal/epl-v10.html
+ * 
+ * This program may also be available under different license terms.
+ * For more information, see www.akiban.com or contact [e-mail].
+ * 
+ * Contributors:
+ * Akiban Technologies, Inc.
+ */
+
+package com.persistit;
+
+import java.io.File;
+
+import com.persistit.exception.InUseException;
+import com.persistit.exception.PersistitException;
+
+/**
+ * Manage all details of file I/O for a special <code>Volume</code> that backs
+ * the {@link Exchange#lock()} method. The lock volume differs from a temporary
+ * volume in the following ways:
+ * <ul>
+ * <li>It has a defined name and is visible in the
+ * {@link Persistit#getVolumes()} array.</li>
+ * <li>It is threadsafe.</li>
+ * </ul>
+ * Like a temporary volume, it is intended that pages of the lock volume should
+ * seldom, if ever, need to be written to disk. And also like a temporary
+ * volume, the contents of the lock volume are not recovered during startup.
+ * 
+ * @author peter
+ */
+class VolumeStorageL2 extends VolumeStorageT2 {
+
+    final static String LOCK_VOLUME_FILE_PREFIX = "persistit_lockvol_";
+    private Buffer _headBuffer;
+
+    VolumeStorageL2(final Persistit persistit, final Volume volume, final File tempDirectory) {
+        super(persistit, volume, tempDirectory);
+    }
+
+    /**
+     * Indicate whether this is a temporary volume. The Lock volume is not
+     * temporary, although it inherits most of the characteristics of a
+     * temporary volume.
+     * 
+     * @return <code>true</code> if this volume is temporary
+     */
+    @Override
+    boolean isTemp() {
+        return false;
+    }
+
+    @Override
+    void truncate() throws PersistitException {
+        if (!claim(true, 0)) {
+            throw new InUseException("Unable to acquire claim on " + this);
+        }
+        try {
+            _headBuffer = _volume.getStructure().getPool().get(_volume, 0, true, false);
+            _headBuffer.init(Buffer.PAGE_TYPE_HEAD);
+            _headBuffer.setFixed();
+
+            truncateInternal();
+            releaseHeadBuffer();
+
+        } finally {
+            release();
+        }
+    }
+
+    @Override
+    void claimHeadBuffer() throws PersistitException {
+        if (!_headBuffer.claim(true)) {
+            throw new InUseException("Unable to acquire claim on " + _headBuffer);
+        }
+    }
+
+    @Override
+    void releaseHeadBuffer() {
+        _headBuffer.release();
+    }
+
+}

=== modified file 'src/main/java/com/persistit/VolumeStorageT2.java'
--- src/main/java/com/persistit/VolumeStorageT2.java	2013-01-04 00:12:33 +0000
+++ src/main/java/com/persistit/VolumeStorageT2.java	2013-03-13 17:42:20 +0000
@@ -44,7 +44,6 @@
 class VolumeStorageT2 extends VolumeStorage {
 
     final static String TEMP_FILE_PREFIX = "persistit_tempvol_";
-    private final static String TEMP_FILE_UNCREATED_NAME = "temp_volume_file_not_created_yet";
     private final File _tempDirectory;
     private long _maxPages;
     private volatile String _path;
@@ -117,7 +116,7 @@
     void create() throws PersistitException {
         final long maxSize = _persistit.getConfiguration().getTmpVolMaxSize();
         _maxPages = maxSize / _volume.getStructure().getPageSize();
-        _path = TEMP_FILE_UNCREATED_NAME;
+        _path = "";
         _channel = null;
         truncate();
         _opened = true;
@@ -220,21 +219,23 @@
             throw new InUseException("Unable to acquire claim on " + this);
         }
         try {
-            final VolumeStatistics stat = _volume.getStatistics();
-            final VolumeStructure struc = _volume.getStructure();
-
-            final long now = System.currentTimeMillis();
-            stat.setCreateTime(now);
-            stat.setOpenTime(now);
-
-            _nextAvailablePage = 1;
-
-            struc.init(0, 0);
+            truncateInternal();
         } finally {
             release();
         }
-        flushMetaData();
-
+    }
+
+    protected void truncateInternal() throws PersistitException {
+        final VolumeStatistics stat = _volume.getStatistics();
+        final VolumeStructure struc = _volume.getStructure();
+
+        final long now = System.currentTimeMillis();
+        stat.setCreateTime(now);
+        stat.setOpenTime(now);
+
+        _nextAvailablePage = 1;
+
+        struc.init(0, 0);
     }
 
     @Override

=== modified file 'src/main/java/com/persistit/VolumeStructure.java'
--- src/main/java/com/persistit/VolumeStructure.java	2012-11-28 17:13:21 +0000
+++ src/main/java/com/persistit/VolumeStructure.java	2013-03-13 17:42:20 +0000
@@ -26,6 +26,7 @@
 
 import com.persistit.AlertMonitor.AlertLevel;
 import com.persistit.AlertMonitor.Event;
+import com.persistit.exception.BufferSizeUnavailableException;
 import com.persistit.exception.CorruptVolumeException;
 import com.persistit.exception.InUseException;
 import com.persistit.exception.PersistitException;
@@ -128,13 +129,13 @@
         _persistit.getJournalManager().truncate(_volume, timestamp);
     }
 
-    Exchange directoryExchange() {
+    Exchange directoryExchange() throws BufferSizeUnavailableException {
         final Exchange ex = new Exchange(_directoryTree);
         ex.ignoreTransactions();
         return ex;
     }
 
-    Exchange accumulatorExchange() {
+    Exchange accumulatorExchange() throws BufferSizeUnavailableException {
         return new Exchange(_directoryTree);
     }
 
@@ -502,7 +503,9 @@
                     return buffer;
                 } finally {
                     garbageBuffer = releaseBuffer(garbageBuffer);
-                    deallocateGarbageChain(chains);
+                    if (!chains.isEmpty()) {
+                        deallocateGarbageChain(chains);
+                    }
                 }
             }
         } finally {
@@ -517,7 +520,6 @@
         buffer.init(Buffer.PAGE_TYPE_UNALLOCATED);
         Debug.$assert0.t(buffer.getPageAddress() != 0);
         return buffer;
-
     }
 
     void deallocateGarbageChain(final long left, final long right) throws PersistitException {

=== modified file 'src/main/java/com/persistit/ui/AdminUISummaryPanel.java'
--- src/main/java/com/persistit/ui/AdminUISummaryPanel.java	2012-08-24 13:57:19 +0000
+++ src/main/java/com/persistit/ui/AdminUISummaryPanel.java	2013-03-13 17:42:20 +0000
@@ -250,7 +250,7 @@
                     final Management.VolumeInfo[] array = (Management.VolumeInfo[]) _volumeInfoArrayModel
                             .getInfoArray();
                     if (array != null && index < array.length) {
-                        _selectedVolumeName = array[index].getPath();
+                        _selectedVolumeName = array[index].getName();
                     } else {
                         _selectedVolumeName = null;
                     }
@@ -423,7 +423,7 @@
                 final Management.VolumeInfo[] via = management.getVolumeInfoArray();
 
                 if (_selectedVolumeName == null && via.length > 0) {
-                    _selectedVolumeName = via[0].getPath();
+                    _selectedVolumeName = via[0].getName();
                 }
 
                 for (int index = 0; index < via.length; index++) {

=== modified file 'src/main/java/com/persistit/ui/AdminUITreePanel.java'
--- src/main/java/com/persistit/ui/AdminUITreePanel.java	2012-08-24 13:57:19 +0000
+++ src/main/java/com/persistit/ui/AdminUITreePanel.java	2013-03-13 17:42:20 +0000
@@ -263,7 +263,7 @@
     }
 
     private void selectVolume(final Management.VolumeInfo volumeInfo) {
-        final String newName = volumeInfo == null ? null : volumeInfo.getPath();
+        final String newName = volumeInfo == null ? null : volumeInfo.getName();
         if (!equals(newName, _selectedVolumeName)) {
             _logicalRecordArrayModel.setInfoArray(null);
             _selectedTreeName = null;

=== modified file 'src/main/java/com/persistit/ui/VTComboBoxModel.java'
--- src/main/java/com/persistit/ui/VTComboBoxModel.java	2012-08-24 13:57:19 +0000
+++ src/main/java/com/persistit/ui/VTComboBoxModel.java	2013-03-13 17:42:20 +0000
@@ -74,7 +74,7 @@
             if (_parent == null) {
                 final VolumeInfo[] volumes = _management.getVolumeInfoArray();
                 for (int index = 0; index < volumes.length; index++) {
-                    _cachedList.add(volumes[index].getPath());
+                    _cachedList.add(volumes[index].getName());
                 }
             } else {
                 final String volumeName = (String) _parent.getSelectedItem();

=== modified file 'src/main/resources/com/persistit/ui/AdminUI.properties'
--- src/main/resources/com/persistit/ui/AdminUI.properties	2012-11-08 15:04:52 +0000
+++ src/main/resources/com/persistit/ui/AdminUI.properties	2013-03-13 17:42:20 +0000
@@ -37,8 +37,8 @@
 lnf = com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel
 
 VolumeInfo.flags = A:All,S:Size,P:Performance,T:Time
-VolumeInfo.column.0 =  getPath:180:A:Path
-VolumeInfo.column.1 =  getName:100:A:Name
+VolumeInfo.column.0 =  getName:100:A:Name
+VolumeInfo.column.1 =  getPath:180:A:Path
 VolumeInfo.column.2 =  getPageSize:100:A:Page Size
 VolumeInfo.column.3 =  getCurrentSize:100:A:Current Size
 VolumeInfo.column.4 =  getMaximumSize:100:A:Max Size

=== modified file 'src/test/java/com/persistit/Bug1041293Test.java'
--- src/test/java/com/persistit/Bug1041293Test.java	2012-10-06 02:44:20 +0000
+++ src/test/java/com/persistit/Bug1041293Test.java	2013-03-13 17:42:20 +0000
@@ -17,6 +17,7 @@
 
 import org.junit.Test;
 
+import com.persistit.exception.BufferSizeUnavailableException;
 import com.persistit.exception.UnderSpecifiedVolumeException;
 
 /**
@@ -39,7 +40,7 @@
         volume.open(_persistit);
     }
 
-    @Test(expected = IllegalStateException.class)
+    @Test(expected = BufferSizeUnavailableException.class)
     public void mismatchedVolumeSpecificationNPE() throws Exception {
         final Configuration config = _persistit.getConfiguration();
         final VolumeSpecification vspec = config.volumeSpecification("${datapath}/test,pageSize:2048,create,"
@@ -48,7 +49,7 @@
         volume.open(_persistit);
     }
 
-    @Test(expected = IllegalStateException.class)
+    @Test(expected = BufferSizeUnavailableException.class)
     public void mismatchedTemporaryVolumePageSize() throws Exception {
         _persistit.createTemporaryVolume(2048);
     }

=== modified file 'src/test/java/com/persistit/CleanupManagerTest.java'
--- src/test/java/com/persistit/CleanupManagerTest.java	2012-08-24 13:57:19 +0000
+++ src/test/java/com/persistit/CleanupManagerTest.java	2013-03-13 17:42:20 +0000
@@ -19,6 +19,7 @@
 import static org.junit.Assert.assertTrue;
 
 import java.lang.ref.WeakReference;
+import java.util.List;
 
 import org.junit.Test;
 
@@ -48,7 +49,8 @@
         }
 
         @Override
-        public void performAction(final Persistit persistit) throws PersistitException {
+        public void performAction(final Persistit persistit, final List<CleanupAction> consequentActions)
+                throws PersistitException {
             assertEquals(_last + 1, _sequence);
             _last = _sequence;
             _counter++;

=== modified file 'src/test/java/com/persistit/ExchangeLockTest.java'
--- src/test/java/com/persistit/ExchangeLockTest.java	2013-02-15 15:39:42 +0000
+++ src/test/java/com/persistit/ExchangeLockTest.java	2013-03-13 17:42:20 +0000
@@ -19,6 +19,8 @@
 import static org.junit.Assert.assertTrue;
 import static org.junit.Assert.fail;
 
+import java.util.Properties;
+import java.util.Random;
 import java.util.concurrent.Semaphore;
 
 import org.junit.Test;
@@ -26,11 +28,17 @@
 import com.persistit.exception.InUseException;
 import com.persistit.exception.InvalidKeyException;
 import com.persistit.exception.PersistitException;
+import com.persistit.unit.UnitTestProperties;
 
 public class ExchangeLockTest extends PersistitUnitTestCase {
     private final static long DMILLIS = SharedResource.DEFAULT_MAX_WAIT_TIME;
     private final Semaphore _coordinator = new Semaphore(0);
 
+    @Override
+    public Properties getProperties(final boolean cleanup) {
+        return UnitTestProperties.getBiggerProperties(cleanup);
+    }
+
     @Test
     public void singleThreadedLock() throws Exception {
         final Exchange ex = _persistit.getExchange("persistit", "ExchangeLockTest", true);
@@ -231,37 +239,34 @@
         assertEquals(1, succeeded);
     }
 
+    /**
+     * This test is intended to exercise lock management for transactions
+     * executed sequentially, each of which performs lots of locks (similar to
+     * DataLoadingTest.)
+     * 
+     * @throws Exception
+     */
     @Test
     public void lockTablePruning() throws Exception {
         final Exchange ex = _persistit.getExchange("persistit", "ExchangeLockTest", true);
+        final Random random = new Random();
         final Transaction txn = ex.getTransaction();
-        txn.begin();
-        for (int i = 0; i < 10000; i++) {
-            ex.clear().append(i).append(RED_FOX).lock();
+        for (int j = 0; j < 100; j++) {
+            txn.begin();
+            for (int i = 0; i < 10000; i++) {
+                final int k = random.nextInt(100000);
+                ex.clear().append(k + (j * 100000)).append(RED_FOX).lock();
+            }
+            txn.commit();
+            txn.end();
         }
-        txn.commit();
-        txn.end();
+        assertTrue("Too many lock volume pages uses",
+                _persistit.getLockVolume().getStorage().getNextAvailablePage() < 100);
 
         final Exchange lockExchange = new Exchange(_persistit.getLockVolume().getTree("ExchangeLockTest", false));
-        lockExchange.ignoreMVCCFetch(true);
-
-        final int count1 = keyCount(lockExchange);
-        assertTrue(count1 > 0);
-
-        _persistit.getTransactionIndex().updateActiveTransactionCache();
-
-        for (int i = 0; i < 10000; i++) {
-            lockExchange.clear().append(i).append(RED_FOX).prune();
-        }
-
-        final int count2 = keyCount(lockExchange);
-        assertTrue(count2 < count1);
-
-        _persistit.getCleanupManager().poll();
-
-        final int count3 = keyCount(lockExchange);
-        assertEquals(0, count3);
-
+        final int count = keyCount(lockExchange);
+
+        assertEquals("Unpruned lock records", 0, count);
     }
 
     private int keyCount(final Exchange ex) throws PersistitException {

=== modified file 'src/test/java/com/persistit/MVCCPruneBufferTest.java'
--- src/test/java/com/persistit/MVCCPruneBufferTest.java	2012-09-26 13:31:13 +0000
+++ src/test/java/com/persistit/MVCCPruneBufferTest.java	2013-03-13 17:42:20 +0000
@@ -43,7 +43,7 @@
         final int available = buffer1.getAvailableSize();
         final int keys = buffer1.getKeyCount();
         buffer1.claim(true);
-        buffer1.pruneMvvValues(null, true);
+        buffer1.pruneMvvValues(null, true, null);
         assertEquals(keys, buffer1.getKeyCount());
         assertTrue(buffer1.getMvvCount() > 0);
         assertEquals(available, buffer1.getAvailableSize());
@@ -53,7 +53,7 @@
 
         _persistit.getTransactionIndex().updateActiveTransactionCache();
 
-        buffer1.pruneMvvValues(null, true);
+        buffer1.pruneMvvValues(null, true, null);
         assertEquals(0, _persistit.getCleanupManager().getAcceptedCount());
         assertTrue("Pruning should have removed primordial Anti-values", keys > buffer1.getKeyCount());
         // mvvCount is 1 because there is still a leading primordial AntiValue
@@ -78,7 +78,7 @@
         final int available = buffer1.getAvailableSize();
         final int keys = buffer1.getKeyCount();
         buffer1.claim(true);
-        buffer1.pruneMvvValues(null, true);
+        buffer1.pruneMvvValues(null, true, null);
         buffer1.release();
         assertEquals(keys, buffer1.getKeyCount());
         assertTrue(buffer1.getMvvCount() > 0);
@@ -102,7 +102,7 @@
         for (long page = 1; page < pageCount; page++) {
             final Buffer buffer = ex1.getBufferPool().get(ex1.getVolume(), page, true, true);
             try {
-                buffer.pruneMvvValues(null, true);
+                buffer.pruneMvvValues(null, true, null);
             } finally {
                 buffer.release();
             }
@@ -123,7 +123,7 @@
         for (long page = 1; page < pageCount; page++) {
             final Buffer buffer = ex1.getBufferPool().get(ex1.getVolume(), page, true, true);
             try {
-                buffer.pruneMvvValues(ex1.getTree(), true);
+                buffer.pruneMvvValues(ex1.getTree(), true, null);
             } finally {
                 buffer.release();
             }

=== modified file 'src/test/java/com/persistit/TestShim.java'
--- src/test/java/com/persistit/TestShim.java	2012-08-24 13:57:19 +0000
+++ src/test/java/com/persistit/TestShim.java	2013-03-13 17:42:20 +0000
@@ -18,6 +18,7 @@
 import java.nio.ByteBuffer;
 import java.nio.channels.FileChannel;
 
+import com.persistit.exception.BufferSizeUnavailableException;
 import com.persistit.exception.InvalidKeyException;
 import com.persistit.exception.PersistitException;
 import com.persistit.exception.PersistitInterruptedException;
@@ -104,7 +105,7 @@
         jman.copyBack();
     }
 
-    public static Exchange directoryExchange(final Volume volume) {
+    public static Exchange directoryExchange(final Volume volume) throws BufferSizeUnavailableException {
         return volume.getStructure().directoryExchange();
     }