Merge lp:~pbeaman/akiban-persistit/remove_fast_index_ratio into lp:akiban-persistit

[Peter Beaman]

While testing another branch I detected an anomalous state in the association of a Buffer with its FastIndex. This branch eliminates that possibility.

The code originally allowed there to be fewer FastIndex instances than Buffer instances and would shuttle FastIndexes between Buffers as needed. But after reducing the size of the FastIndex to 12.5% of the buffer and benchmarks that prove having a 1:1 ratio is optimal, we set the FAST_INDEX_RATIO in BufferPool to 1.0f and have no inclination to change it. This is equivalent to permanently assigning a FastIndex instance to each Buffer.

That's what this branch does: it allocates a unique, permanent FastIndex when a Buffer is constructed and removes all the code that permitting the reallocation of these structures on the fly.

Advantages: simpler, faster, less prone to bugs due to complex code path.

[Nathan Williams (nwilliams)]

Makes sense.

review: Approve

[Akiban Build User (build-akiban)]

There were 2 failures during build/test:

* job persistit-build failed at build number 360: http://172.16.20.104:8080/job/persistit-build/360/

* view must-pass failed: persistit-build is yellow

review: Needs Fixing

[Peter Beaman (pbeaman)]

New constructor for FastIndex was off by one.

[Nathan Williams (nwilliams)]

Was hung in teardown, again:
http://172.16.20.104:8080/job/persistit-build/361/

[Peter Beaman (pbeaman)]

Yup, there was an actual bug in the modified version of Buffer. As a consequence, FastIndexTest#testFastIndexRecompute throw an AssertionError. But FastIndexTest was not written defensively to remove buffer claims even in the event of an error, so the tearDown() method, which calls Persistit#close() hung due to inability to claim the page.

Three issues to resolve:

1. New bug in Buffer (rewrite failed to recompute an invalid FastIndex in the getFastIndex() method).

2. FastIndexTest needs try/finally blocks to ensure the buffer gets released.

3. Volume#close() needs to time out if unable to invalidate the volume within a reasonable length of time.

Adding this changes and resubmitting.

[Nathan Williams (nwilliams)]

The timeout in Volume#close() is probably a good thing to do in general and the other changes make sense as well.

Third times the charm.

review: Approve

[Akiban Build User (build-akiban)]

There were 3 failures during build/test:

* job persistit-build failed at build number 365: http://172.16.20.104:8080/job/persistit-build/365/

* view must-pass failed: server-build is blue_anime

* view must-pass failed: persistit-build is yellow

review: Needs Fixing

[Nathan Williams (nwilliams)]

Real failure, but in an unrelated area (TransactionTest2WithInterrupts, add a few revs ago). Trying again.

[Nathan Williams (nwilliams)]

Well, this has failed twice now in the server histogramsIT test with output like:

com.akiban.server.error.PersistitAdapterException: PERSISTIT_ERROR: Persistit Data Layer error: Volume akiban_data(/tmp/akserver-junit/akserver-unitdata7395450818281228504/akiban_data.v01) level=0 page=16 key=<{"Bob",null,"0245",null}> is before left edge

http://172.16.20.104:8080/job/server-build/2333/
http://172.16.20.104:8080/job/server-build/2334/

review: Needs Fixing

[Peter Beaman (pbeaman)]

Ouch. Thanks for passing this on. I don't have vpn access.
On Jun 16, 2012 9:57 AM, "Nathan Williams" <email address hidden> wrote:

> Review: Needs Fixing
>
> Well, this has failed twice now in the server histogramsIT test with
> output like:
>
> com.akiban.server.error.PersistitAdapterException: PERSISTIT_ERROR:
> Persistit Data Layer error: Volume
> akiban_data(/tmp/akserver-junit/akserver-unitdata7395450818281228504/akiban_data.v01)
> level=0 page=16 key=<{"Bob",null,"0245",null}> is before left edge
>
> http://172.16.20.104:8080/job/server-build/2333/
> http://172.16.20.104:8080/job/server-build/2334/
>
>
> --
>
> https://code.launchpad.net/~pbeaman/akiban-persistit/remove_fast_index_ratio/+merge/110391
> You are the owner of lp:~pbeaman/akiban-persistit/remove_fast_index_ratio.
>

[Peter Beaman (pbeaman)]

The failures were due to two subtle bugs in the branch. In Buffer#split there was a conditional statement missing a condition needed to correctly set a flag. The incorrectly set flag caused FastIndex#insertKeyBlock to misbehave.

But, the other bug is that the FastIndex should not have been recomputed in Buffer#putValue. The previous version of this branch did so by using a call to getFastIndex() rather than simply using the field value.

Finally, these issues were discovered in IndexHistogramIT because that test inserts both null- and String-valued key segments. This behavior leads to a sequence of keys having different data types and therefore differeing in their first byte. We need to add unit and stress tests to cover this; at the moment all Persistit tests have keys with the same data type in their first key segment.

[Nathan Williams (nwilliams)]

> We need to add unit and stress tests to cover this; at the
> moment all Persistit tests have keys with the same data type
> in their first key segment.

Any reason not to add that before merging this?

[Peter Beaman (pbeaman)]

Added unit test BufferTest2#splitAndJoinBuffersWithZeroEbcKeys().

[Nathan Williams (nwilliams)]

Thanks for the new test.

review: Approve

[Akiban Build User (build-akiban)]

There was one failure during build/test:

* bzr error: These branches have diverged. Use the missing command to see how.
Use the merge command to reconcile them.

review: Needs Fixing

[Nathan Williams (nwilliams)]

Looks like /build got out of sync with trunk.

Luckily it only looks like 2 revs are off and build is in order so I'll push it up to trunk.

=== modified file 'src/main/java/com/persistit/Buffer.java'
--- src/main/java/com/persistit/Buffer.java	2012-06-08 00:06:12 +0000
+++ src/main/java/com/persistit/Buffer.java	2012-06-25 15:43:24 +0000
@@ -269,7 +269,7 @@
     public final static int MAX_KEY_RATIO = 16;
 
     private final static int BINARY_SEARCH_THRESHOLD = 6;
-
+    
     abstract static class VerifyVisitor {
 
         protected void visitPage(long timestamp, Volume volume, long page, int type, int bufferSize, int keyBlockStart,
@@ -318,13 +318,23 @@
     /**
      * A ByteBuffer facade for this Buffer used in NIO operations
      */
-    private ByteBuffer _byteBuffer;
+    private final ByteBuffer _byteBuffer;
+
+    /**
+     * The size of this buffer
+     */
+    private final int _bufferSize;
 
     /**
      * The bytes in this buffer. Note these bytes are also the backing store of
      * _byteBuffer.
      */
     private byte[] _bytes;
+    
+    /**
+     * FastIndex structure used for rapid page searching
+     */
+    private final FastIndex _fastIndex;
 
     /**
      * The right sibling page address
@@ -332,16 +342,6 @@
     private volatile long _rightSibling;
 
     /**
-     * The size of this buffer
-     */
-    private int _bufferSize;
-
-    /**
-     * FastIndex structure used for rapid page searching
-     */
-    private FastIndex _fastIndex;
-
-    /**
      * Type code for this page.
      */
     private volatile int _type;
@@ -405,6 +405,7 @@
         _byteBuffer = ByteBuffer.allocate(size);
         _bytes = _byteBuffer.array();
         _bufferSize = size;
+        _fastIndex = new FastIndex(this, 1 + (size - HEADER_SIZE) / MAX_KEY_RATIO);
     }
 
     Buffer(Buffer original) {
@@ -423,20 +424,6 @@
         System.arraycopy(original._bytes, 0, _bytes, 0, _bytes.length);
     }
 
-    Buffer(Persistit persistit, Volume vol, long page, byte[] bytes) throws InvalidPageStructureException,
-            PersistitInterruptedException {
-        this(bytes.length, -1, persistit.getBufferPool(bytes.length), persistit);
-        System.arraycopy(bytes, 0, _bytes, 0, bytes.length);
-        _vol = vol;
-        _page = page;
-        claim(true);
-        try {
-            load();
-        } finally {
-            release();
-        }
-    }
-
     /**
      * Initializes the buffer so that it contains no keys or data.
      */
@@ -508,11 +495,11 @@
                 } else if (isIndexPage()) {
                     _tailHeaderSize = TAILBLOCK_HDR_SIZE_INDEX;
                 }
-                invalidateFastIndex();
             }
         } else {
             _type = PAGE_TYPE_HEAD;
         }
+        invalidateFastIndex();
         bumpGeneration();
     }
 
@@ -1406,7 +1393,6 @@
         if (Debug.ENABLED) {
             assertVerify();
         }
-        final FastIndex fastIndex = _fastIndex;
 
         boolean exactMatch = (foundAt & EXACT_MASK) > 0;
         int p = foundAt & P_MASK;
@@ -1522,11 +1508,10 @@
                         _bytes.length); // TODO limit
                 incCountIfMvv(_bytes, newTail + _tailHeaderSize + klength, storedLength & MVV.STORE_LENGTH_MASK);
             }
-
-            if (fastIndex != null) {
-                fastIndex.insertKeyBlock(p, ebcSuccessor, fixupSuccessor);
-            }
-
+            //
+            // Correct not to call getFastIndex()
+            //
+            _fastIndex.insertKeyBlock(p, ebcSuccessor, fixupSuccessor);
             bumpGeneration();
 
             if (p > KEY_BLOCK_START) {
@@ -2294,7 +2279,7 @@
             if (foundAtPosition >= splitAtPosition && (!lastLeft || foundAtPosition > splitAtPosition)) {
                 foundAt -= (splitAtPosition - KEY_BLOCK_START);
                 if (firstRight && !fixupSuccessor) {
-                    foundAt = (foundAt & P_MASK) | FIXUP_MASK | (ebc << DEPTH_SHIFT);
+                    foundAt = (foundAt & P_MASK) | (ebc > 0 ? FIXUP_MASK : 0) | (ebc << DEPTH_SHIFT);
                 }
                 final int t = rightSibling.putValue(key, valueHelper, foundAt, true);
                 whereInserted = -foundAt;
@@ -2313,8 +2298,7 @@
             }
             //
             // It is really bad if whereInserted is less than 0. Means that we
-            // failed
-            // to replace the value.
+            // failed to replace the value.
             //
             Debug.$assert0.t(whereInserted > 0);
             if (whereInserted <= 0) {
@@ -2830,38 +2814,17 @@
 
     }
 
-    void invalidateFastIndex() {
-        if (_fastIndex != null) {
+    synchronized void invalidateFastIndex() {
             _fastIndex.invalidate();
-        }
     }
 
-    synchronized FastIndex getFastIndex() throws PersistitInterruptedException {
-        // TODO - replace synchronized with CAS instructions
-        if (_fastIndex == null) {
-            _fastIndex = _pool.allocFastIndex();
-            _fastIndex.setBuffer(this);
-        }
+    synchronized FastIndex getFastIndex() {
         if (!_fastIndex.isValid()) {
             _fastIndex.recompute();
         }
-        _fastIndex.setTouched();
         return _fastIndex;
     }
 
-    synchronized void takeFastIndex() {
-        _fastIndex = null;
-    }
-
-    /**
-     * Only for unit tests.
-     * 
-     * @param fastIndex
-     */
-    void setFastIndex(final FastIndex fastIndex) {
-        _fastIndex = fastIndex;
-    }
-
     private void reduceEbc(int p, int newEbc, byte[] indexKeyBytes) {
         int kbData = getInt(p);
         int oldDb = decodeKeyBlockDb(kbData);
@@ -3305,8 +3268,7 @@
 
     static int bufferSizeWithOverhead(final int bufferSize) {
         int fastIndexSize = ((bufferSize - HEADER_SIZE) / MAX_KEY_RATIO) * FastIndex.BYTES_PER_ENTRY;
-        int fastIndexOverhead = (int) (fastIndexSize * BufferPool.FAST_INDEX_RATIO);
-        return bufferSize + fastIndexOverhead + ESTIMATED_FIXED_BUFFER_OVERHEAD;
+        return bufferSize + fastIndexSize + ESTIMATED_FIXED_BUFFER_OVERHEAD;
     }
 
     /**

=== modified file 'src/main/java/com/persistit/BufferPool.java'
--- src/main/java/com/persistit/BufferPool.java	2012-05-25 18:50:59 +0000
+++ src/main/java/com/persistit/BufferPool.java	2012-06-25 15:43:24 +0000
@@ -76,10 +76,6 @@
      * Maximum number of buffers this pool may have
      */
     public final static int MAXIMUM_POOL_COUNT = Integer.MAX_VALUE;
-    /**
-     * Ratio of FastIndex to buffers
-     */
-    final static float FAST_INDEX_RATIO = 1.0f;
 
     /**
      * The maximum number of lock buckets
@@ -126,15 +122,6 @@
     private final int _bufferSize;
 
     /**
-     * Count of FastIndex instances, computed as a fraction of buffer count
-     */
-    private final int _fastIndexCount;
-    /**
-     * FastIndex array
-     */
-    private final FastIndex[] _fastIndexes;
-
-    /**
      * Bit map for invalidated pages. Elements in this array, one bit per page,
      * indicate buffers that have been invalidated and are therefore able to be
      * allocated without evicting a valid page.
@@ -154,11 +141,6 @@
     private final AtomicInteger _clock = new AtomicInteger();
 
     /**
-     * Pointer to next FastIndex to allocate
-     */
-    private final AtomicInteger _fastIndexClock = new AtomicInteger();
-
-    /**
      * Count of buffer pool misses (buffer not found in pool)
      */
     private final AtomicLong _missCounter = new AtomicLong();
@@ -260,15 +242,12 @@
         _hashTable = new Buffer[_bufferCount * HASH_MULTIPLE];
         _hashLocks = new ReentrantLock[HASH_LOCKS];
         _maxKeys = (_bufferSize - Buffer.HEADER_SIZE) / Buffer.MAX_KEY_RATIO;
-        _fastIndexCount = (int) (count * FAST_INDEX_RATIO);
-        _fastIndexes = new FastIndex[_fastIndexCount];
 
         for (int index = 0; index < HASH_LOCKS; index++) {
             _hashLocks[index] = new ReentrantLock();
         }
 
         int buffers = 0;
-        int fastIndexes = 0;
         //
         // Allocate this here so that in the event of an OOME we can release it
         // to free enough memory to write the error information out.
@@ -280,11 +259,6 @@
                 _buffers[index] = buffer;
                 buffers++;
             }
-            for (int index = 0; index < _fastIndexCount; index++) {
-                _fastIndexes[index] = new FastIndex(_maxKeys + 1);
-                _fastIndexes[index].setBuffer(_buffers[index]);
-                fastIndexes++;
-            }
         } catch (OutOfMemoryError e) {
             //
             // Note: written this way to try to avoid another OOME.
@@ -297,12 +271,7 @@
             System.err.print(buffers);
             System.err.print("/");
             System.err.print(_bufferCount);
-            System.err.print(" buffers and ");
-            System.err.print(fastIndexes);
-            System.err.print("/");
-            System.err.print(_fastIndexCount);
-            System.err.print(" fast indexes ");
-            System.err.print(" from maximum heap ");
+            System.err.print(" buffers from maximum heap ");
             System.err.println(_persistit.getAvailableHeap());
             throw e;
         }
@@ -1016,30 +985,6 @@
         throw new IllegalStateException("No available Buffers");
     }
 
-    FastIndex allocFastIndex() throws PersistitInterruptedException {
-        for (int retry = 0; retry < _fastIndexCount * 2;) {
-            int clock = _fastIndexClock.get();
-            if (!_fastIndexClock.compareAndSet(clock, (clock + 1) % _fastIndexCount)) {
-                continue;
-            }
-            FastIndex findex = _fastIndexes[clock];
-
-            if (findex.testTouched()) {
-                findex.clearTouched();
-            } else {
-                Buffer buffer = findex.getBuffer();
-                if (buffer.claim(true, 0)) {
-                    buffer.takeFastIndex();
-                    buffer.release();
-                    findex.invalidate();
-                    return findex;
-                }
-            }
-            retry++;
-        }
-        throw new IllegalStateException("No FastIndex buffers available");
-    }
-
     enum Result {
         WRITTEN, UNAVAILABLE, ERROR
     };

=== modified file 'src/main/java/com/persistit/Exchange.java'
--- src/main/java/com/persistit/Exchange.java	2012-06-08 00:06:12 +0000
+++ src/main/java/com/persistit/Exchange.java	2012-06-25 15:43:24 +0000
@@ -23,15 +23,9 @@
 import static com.persistit.Buffer.EXACT_MASK;
 import static com.persistit.Buffer.HEADER_SIZE;
 import static com.persistit.Buffer.KEYBLOCK_LENGTH;
-import static com.persistit.Buffer.LONGREC_PREFIX_OFFSET;
-import static com.persistit.Buffer.LONGREC_PREFIX_SIZE;
-import static com.persistit.Buffer.LONGREC_SIZE;
-import static com.persistit.Buffer.LONGREC_TYPE;
-import static com.persistit.Buffer.MAX_LONG_RECORD_CHAIN;
 import static com.persistit.Buffer.MAX_VALID_PAGE_ADDR;
 import static com.persistit.Buffer.PAGE_TYPE_DATA;
 import static com.persistit.Buffer.PAGE_TYPE_INDEX_MIN;
-import static com.persistit.Buffer.PAGE_TYPE_LONG_RECORD;
 import static com.persistit.Buffer.P_MASK;
 import static com.persistit.Buffer.TAILBLOCK_HDR_SIZE_INDEX;
 import static com.persistit.Key.AFTER;

=== modified file 'src/main/java/com/persistit/FastIndex.java'
--- src/main/java/com/persistit/FastIndex.java	2012-05-25 18:50:59 +0000
+++ src/main/java/com/persistit/FastIndex.java	2012-06-25 15:43:24 +0000
@@ -24,7 +24,6 @@
 
 import com.persistit.util.Debug;
 
-
 /**
  * FastIndex is a wrapper for an array of integers that contain information used
  * in searching for a key in a page. There is one meaningful element in that
@@ -154,11 +153,6 @@
      */
     private final short[] _findexElements;
     /**
-     * AtomicInteger used to house the "touched" bit for the clock replacement
-     * algorithm in BufferPool.
-     */
-    private final AtomicInteger _bits = new AtomicInteger();
-    /**
      * Indicates whether the _findexElements array is valid
      */
     private boolean _isValid;
@@ -166,9 +160,10 @@
     /**
      * The buffer this fast index is associated with.
      */
-    private Buffer _buffer;
+    private final Buffer _buffer;
 
-    FastIndex(int elementSize) {
+    FastIndex(Buffer buffer, int elementSize) {
+        _buffer = buffer;
         _findexElements = new short[elementSize];
         _isValid = false;
     }
@@ -607,26 +602,6 @@
         return start - end + 1;
     }
 
-    void setBuffer(final Buffer buffer) {
-        _buffer = buffer;
-    }
-
-    Buffer getBuffer() {
-        return _buffer;
-    }
-
-    void setTouched() {
-        _bits.set(1);
-    }
-
-    void clearTouched() {
-        _bits.set(0);
-    }
-
-    boolean testTouched() {
-        return (_bits.get()) != 0;
-    }
-
     // TODO - unnecessary after debugging
     public String toString() {
         StringBuilder sb = new StringBuilder();

=== modified file 'src/main/java/com/persistit/Volume.java'
--- src/main/java/com/persistit/Volume.java	2012-05-25 18:50:59 +0000
+++ src/main/java/com/persistit/Volume.java	2012-06-25 15:43:24 +0000
@@ -144,19 +144,36 @@
     /**
      * Release all resources for this <code>Volume</code> and invalidate all its
      * buffers in the {@link BufferPool}. Exchanges based on this
+     * <code>Volume</code> may no longer be used after this call. Waits up to
+     * {@value com.persistit.SharedResource#DEFAULT_MAX_WAIT_TIME} milliseconds
+     * for other threads to relinquish access to the volume.
+     * 
+     * @throws PersistitException
+     */
+    public void close() throws PersistitException {
+        close(SharedResource.DEFAULT_MAX_WAIT_TIME);
+    }
+
+    /**
+     * Release all resources for this <code>Volume</code> and invalidate all its
+     * buffers in the {@link BufferPool}. Exchanges based on this
      * <code>Volume</code> may no longer be used after this call.
      * 
+     * @param timeout
+     *            Maximum time in milliseconds to wait for other threads to
+     *            relinquish access to the volume.
      * @throws PersistitException
      */
-    public void close() throws PersistitException {
+    public void close(final long timeout) throws PersistitException {
         closing();
+        final long expiration = System.currentTimeMillis() + timeout;
         for (;;) {
             //
             // Prevents read/write operations from starting while the
             // volume is being closed.
             //
             final VolumeStorage storage = getStorage();
-            if (!storage.claim(true)) {
+            if (!storage.claim(true, timeout)) {
                 throw new InUseException("Unable to acquire claim on " + this);
             }
             try {
@@ -174,6 +191,9 @@
             } finally {
                 storage.release();
             }
+            if (System.currentTimeMillis() >= expiration) {
+                throw new InUseException("Unable invalidate all pages on " + this);
+            }
             Util.sleep(Persistit.SHORT_DELAY);
         }
     }

=== modified file 'src/test/java/com/persistit/BufferTest2.java'
--- src/test/java/com/persistit/BufferTest2.java	2012-05-25 18:50:59 +0000
+++ src/test/java/com/persistit/BufferTest2.java	2012-06-25 15:43:24 +0000
@@ -29,6 +29,8 @@
 import static org.junit.Assert.assertTrue;
 import static org.junit.Assert.fail;
 
+import java.util.Arrays;
+
 import org.junit.Test;
 
 import com.persistit.ValueHelper.RawValueWriter;
@@ -331,6 +333,53 @@
         b1.removeKeys(foundAt1 & ~EXACT_MASK, foundAt2, ex.getAuxiliaryKey1());
     }
 
+    @Test
+    public void splitAndJoinBuffersWithZeroEbcKeys() throws Exception {
+        b1.init(Buffer.PAGE_TYPE_DATA);
+        b2.init(Buffer.PAGE_TYPE_DATA);
+        setUpValue(true, b1.getBufferSize() / 100);
+
+        int a;
+        for (a = 1;; a++) {
+            setUpShallowKey(a);
+            if (b1.putValue(ex.getKey(), vw) == -1) {
+                break;
+            }
+        }
+        setUpShallowKey(a / 2);
+        ex.getKey().getEncodedBytes()[6] = 0x7F;
+        ex.getKey().setEncodedSize(7);
+        int foundAt = b1.findKey(ex.getKey());
+        int splitAt = b1.split(b2, ex.getKey(), vw, foundAt, ex.getAuxiliaryKey1(), Exchange.Sequence.NONE,
+                SplitPolicy.EVEN_BIAS);
+
+        assertTrue("Split failed", splitAt != -1);
+        assertTrue("Verify failed", b1.verify(null, null) == null);
+        assertTrue("Verify failed", b2.verify(null, null) == null);
+        
+        for (int b = 1; b < a; b++) {
+            setUpShallowKey(b);
+            if (b <= a / 2) {
+                b1.fetch(b1.findKey(ex.getKey()), ex.getValue());
+            } else {
+                b2.fetch(b2.findKey(ex.getKey()), ex.getValue());
+            }
+            assertTrue("Value should be defined", ex.getValue().isDefined());
+        }
+        
+        setUpShallowKey(a / 2);
+        ex.getKey().getEncodedBytes()[6] = 0x7F;
+        ex.getKey().setEncodedSize(7);
+        int foundAt1 = b1.findKey(ex.getKey());
+        int foundAt2 = b2.findKey(ex.getKey());
+        assertTrue("Should have found exact match", (foundAt2 & EXACT_MASK) > 0);
+        foundAt2 = b2.nextKeyBlock(foundAt2);
+        boolean rebalanced = b1.join(b2, foundAt1, foundAt2, ex.getKey(), ex.getAuxiliaryKey1(), JoinPolicy.EVEN_BIAS);
+        assertTrue("Should have joined pages", !rebalanced);
+        assertTrue("Verify failed", b1.verify(null, null) == null);
+        assertTrue("Verify failed", b2.verify(null, null) == null);
+    }
+
     private void setUpPrettyFullBufferWithChangingEbc(final int valueLength) throws PersistitException {
         b1.init(Buffer.PAGE_TYPE_DATA);
         setUpValue(false, valueLength);
@@ -401,6 +450,26 @@
         ex.getValue().setPointerValue(pointer);
     }
 
+    private void setUpShallowKey(final int a) {
+        setUpShallowKey(ex, a, 0);
+    }
+
+    private void setUpShallowKey(Exchange ex, int value, long pointer) {
+        byte[] bytes = ex.getKey().clear().getEncodedBytes();
+        Arrays.fill(bytes, (byte) 0);
+        /*
+         * Convert int to little-endian form so that byte 0 varies fastest. 7
+         * bits per byte, no zeroes.
+         */
+        bytes[0] = (byte) (((value >>> 0) & 0x7F) + 1);
+        bytes[1] = (byte) (((value >>> 7) & 0x7F) + 1);
+        bytes[2] = (byte) (((value >>> 14) & 0x7F) + 1);
+        bytes[3] = (byte) (((value >>> 21) & 0x7F) + 1);
+        bytes[5] = (byte) (((value >>> 28) & 0x7F) + 1);
+        ex.getKey().setEncodedSize(6);
+        ex.getValue().setPointerValue(pointer);
+    }
+
     String keyString(char fill, int length, int prefix, int width, int k) {
         StringBuilder sb = new StringBuilder();
         for (int i = 0; i < prefix && i < length; i++) {

=== modified file 'src/test/java/com/persistit/FastIndexTest.java'
--- src/test/java/com/persistit/FastIndexTest.java	2012-05-25 18:50:59 +0000
+++ src/test/java/com/persistit/FastIndexTest.java	2012-06-25 15:43:24 +0000
@@ -39,78 +39,88 @@
     @Test
     public void testIndexSize() throws Exception {
         final Buffer b1 = getABuffer();
-        FastIndex fi = b1.getFastIndex();
-        /*
-         * BUFFER_SIZE - HEADER_SIZE / MAX_KEY_RATIO BUFFER_SIZE = 16384
-         * HEADER_SIZE = 32 MAX_KEY_RATIO = 16 16384 - 32 / 16 = 16352 / 16 =
-         * 1022 size passed to FastIndex constructor is 1022 + 1 when parameters
-         * are as listed above.
-         */
-        final int expectedSize = (16384 - Buffer.HEADER_SIZE) / Buffer.MAX_KEY_RATIO;
-        assertEquals(expectedSize + 1, fi.size());
-        b1.release();
+        try {
+            FastIndex fi = b1.getFastIndex();
+            /*
+             * BUFFER_SIZE - HEADER_SIZE / MAX_KEY_RATIO BUFFER_SIZE = 16384
+             * HEADER_SIZE = 32 MAX_KEY_RATIO = 16 16384 - 32 / 16 = 16352 / 16
+             * = 1022 size passed to FastIndex constructor is 1022 + 1 when
+             * parameters are as listed above.
+             */
+            final int expectedSize = (16384 - Buffer.HEADER_SIZE) / Buffer.MAX_KEY_RATIO;
+            assertEquals(expectedSize + 1, fi.size());
+        } finally {
+            b1.release();
+        }
     }
 
     @Test
     public void testIndexValidity() throws Exception {
         final Buffer b1 = getABuffer();
-        b1.init(Buffer.PAGE_TYPE_GARBAGE);
-        FastIndex fi = b1.getFastIndex();
-        fi.invalidate();
-        assertEquals(false, fi.isValid());
-        fi.recompute();
-        assertEquals(false, fi.isValid());
-        b1.init(Buffer.PAGE_TYPE_DATA);
-        fi.recompute();
-        assertEquals(true, fi.isValid());
-        assertEquals(true, fi.verify());
-        fi.invalidate();
-        assertEquals(false, fi.isValid());
-        assertEquals(false, fi.verify());
-        b1.release();
+        try {
+            b1.init(Buffer.PAGE_TYPE_GARBAGE);
+            FastIndex fi = b1.getFastIndex();
+            fi.invalidate();
+            assertEquals(false, fi.isValid());
+            fi.recompute();
+            assertEquals(false, fi.isValid());
+            b1.init(Buffer.PAGE_TYPE_DATA);
+            fi.recompute();
+            assertEquals(true, fi.isValid());
+            assertEquals(true, fi.verify());
+            fi.invalidate();
+            assertEquals(false, fi.isValid());
+            assertEquals(false, fi.verify());
+        } finally {
+            b1.release();
+        }
     }
 
     @Test
     public void testFastIndexRecompute() throws Exception {
         Buffer b1 = getABuffer();
-        b1.init(Buffer.PAGE_TYPE_DATA);
-        Key key = new Key(_persistit);
-        Value value = new Value(_persistit);
-        RawValueWriter vwriter = new RawValueWriter();
-        vwriter.init(value);
-        fakeKey(key, "A");
-        b1.putValue(key, vwriter);
-        fakeKey(key, "ABC");
-        b1.putValue(key, vwriter);
-        fakeKey(key, "ABK");
-        b1.putValue(key, vwriter);
-        fakeKey(key, "ABZ");
-        b1.putValue(key, vwriter);
-        fakeKey(key, "AC");
-        b1.putValue(key, vwriter);
-        fakeKey(key, "C");
-        b1.putValue(key, vwriter);
-        fakeKey(key, "B");
-        b1.putValue(key, vwriter);
-        fakeKey(key, "E");
-        b1.putValue(key, vwriter);
-        fakeKey(key, "D");
-        b1.putValue(key, vwriter);
-        fakeKey(key, "DA");
-        b1.putValue(key, vwriter);
-        fakeKey(key, "ABB");
-        b1.putValue(key, vwriter);
-        fakeKey(key, "ABA");
-        b1.putValue(key, vwriter);
-        fakeKey(key, "ABJ");
-        b1.putValue(key, vwriter);
+        try {
+            b1.init(Buffer.PAGE_TYPE_DATA);
+            Key key = new Key(_persistit);
+            Value value = new Value(_persistit);
+            RawValueWriter vwriter = new RawValueWriter();
+            FastIndex fi = b1.getFastIndex();
+            fi.recompute();
+            vwriter.init(value);
+            fakeKey(key, "A");
+            b1.putValue(key, vwriter);
+            fakeKey(key, "ABC");
+            b1.putValue(key, vwriter);
+            fakeKey(key, "ABK");
+            b1.putValue(key, vwriter);
+            fakeKey(key, "ABZ");
+            b1.putValue(key, vwriter);
+            fakeKey(key, "AC");
+            b1.putValue(key, vwriter);
+            fakeKey(key, "C");
+            b1.putValue(key, vwriter);
+            fakeKey(key, "B");
+            b1.putValue(key, vwriter);
+            fakeKey(key, "E");
+            b1.putValue(key, vwriter);
+            fakeKey(key, "D");
+            b1.putValue(key, vwriter);
+            fakeKey(key, "DA");
+            b1.putValue(key, vwriter);
+            fakeKey(key, "ABB");
+            b1.putValue(key, vwriter);
+            fakeKey(key, "ABA");
+            b1.putValue(key, vwriter);
+            fakeKey(key, "ABJ");
+            b1.putValue(key, vwriter);
 
-        FastIndex fi = b1.getFastIndex();
-        String inserteds = fi.toString();
-        fi.recompute();
-        String computeds = fi.toString();
-        assertEquals(inserteds, computeds);
-        b1.release();
+            String inserteds = fi.toString();
+            fi.recompute();
+            String computeds = fi.toString();
+            assertEquals(inserteds, computeds);
+        } finally {
+            b1.release();
+        }
     }
 
     private void fakeKey(final Key key, final String v) {
@@ -123,31 +133,30 @@
     public void testRandomInsert() throws Exception {
         Random random = new Random(3);
         Buffer b1 = getABuffer();
-        FastIndex f1 = b1.getFastIndex();
-
-        b1.init(Buffer.PAGE_TYPE_DATA);
-        Key key = new Key(_persistit);
-        Value value = new Value(_persistit);
-        for (int i = 0; i < 1000; i++) {
-            int size = random.nextInt(10) + 2;
-            byte[] bytes = new byte[size];
-            random.nextBytes(bytes);
-            System.arraycopy(bytes, 0, key.getEncodedBytes(), 0, size);
-            key.setEncodedSize(size);
-            RawValueWriter vwriter = new RawValueWriter();
-            vwriter.init(value);
-            b1.putValue(key, vwriter);
-
-            String s1 = f1.toString();
-            f1.recompute();
-            String s2 = f1.toString();
-            assertEquals(s1, s2);
+        try {
+            FastIndex f1 = b1.getFastIndex();
+
+            b1.init(Buffer.PAGE_TYPE_DATA);
+            Key key = new Key(_persistit);
+            Value value = new Value(_persistit);
+            for (int i = 0; i < 1000; i++) {
+                int size = random.nextInt(10) + 2;
+                byte[] bytes = new byte[size];
+                random.nextBytes(bytes);
+                System.arraycopy(bytes, 0, key.getEncodedBytes(), 0, size);
+                key.setEncodedSize(size);
+                RawValueWriter vwriter = new RawValueWriter();
+                vwriter.init(value);
+                b1.putValue(key, vwriter);
+
+                String s1 = f1.toString();
+                f1.recompute();
+                String s2 = f1.toString();
+                assertEquals(s1, s2);
+            }
+        } finally {
+            b1.release();
         }
-        b1.release();
-    }
-
-    @Override
-    public void runAllTests() {
     }
 
 }

=== modified file 'src/test/java/com/persistit/VolumeTest.java'
--- src/test/java/com/persistit/VolumeTest.java	2012-05-25 18:50:59 +0000
+++ src/test/java/com/persistit/VolumeTest.java	2012-06-25 15:43:24 +0000
@@ -35,6 +35,7 @@
 import org.junit.Test;
 
 import com.persistit.exception.CorruptVolumeException;
+import com.persistit.exception.InUseException;
 import com.persistit.exception.InvalidVolumeSpecificationException;
 import com.persistit.exception.VolumeFullException;
 import com.persistit.unit.PersistitUnitTestCase;
@@ -184,12 +185,12 @@
         final Volume vol2 = _persistit.loadVolume(vs);
         final long statTimestamp = vol2.getStatistics().getLastGlobalTimestamp();
         // Greater than is ok (other activity may have occurred)
-        if(statTimestamp < MARKER) {
+        if (statTimestamp < MARKER) {
             assertEquals("Saved and loaded timestamp", MARKER, statTimestamp);
         }
     }
 
-    @Test(expected=CorruptVolumeException.class)
+    @Test(expected = CorruptVolumeException.class)
     public void volumeFromFutureIsRejected() throws Exception {
         final int RECORDS = 100;
 
@@ -227,6 +228,25 @@
         _persistit.initialize(properties);
     }
 
+    @Test
+    public void timeoutWhenPageIsInUse() throws Exception {
+        final Exchange exchange = _persistit.getExchange(UnitTestProperties.VOLUME_NAME, "VolumeTest", true);
+        final Buffer buffer = exchange.getBufferPool().get(exchange.getVolume(), 1, false, true);
+        try {
+            final long start = System.currentTimeMillis();
+            try {
+                exchange.getVolume().close(5000);
+                fail("Expect an InUseException");
+            } catch (InUseException e) {
+                final long elapsed = System.currentTimeMillis() - start;
+                assertTrue("Expected InUseException to happen at 5000 ms but was " + elapsed, elapsed > 4000
+                        && elapsed < 10000);
+            }
+        } finally {
+            buffer.release();
+        }
+    }
+
     private VolumeSpecification validVolumeSpecification(final String specification) throws Exception {
         try {
             return _persistit.getConfiguration().volumeSpecification(specification);
@@ -245,10 +265,4 @@
         }
     }
 
-    @Override
-    public void runAllTests() throws Exception {
-        // TODO Auto-generated method stub
-
-    }
-
 }