Merge lp:~pbeaman/akiban-persistit/buffer-pool-warmup-sorted into lp:akiban-persistit
- buffer-pool-warmup-sorted
- Merge into trunk
Status: | Superseded |
---|---|
Proposed branch: | lp:~pbeaman/akiban-persistit/buffer-pool-warmup-sorted |
Merge into: | lp:akiban-persistit |
Prerequisite: | lp:~pbeaman/akiban-persistit/eclipse-canonical-format |
Diff against target: |
1315 lines (+654/-247) 15 files modified
doc/Configuration.rst (+13/-1) src/main/java/com/persistit/BufferPool.java (+144/-109) src/main/java/com/persistit/CheckpointManager.java (+2/-1) src/main/java/com/persistit/Configuration.java (+72/-72) src/main/java/com/persistit/JournalManager.java (+45/-22) src/main/java/com/persistit/MediatedFileChannel.java (+1/-1) src/main/java/com/persistit/Persistit.java (+22/-9) src/main/java/com/persistit/logging/LogBase.java (+9/-0) src/test/java/com/persistit/IOFailureTest.java (+1/-1) src/test/java/com/persistit/StressRunner.java (+2/-0) src/test/java/com/persistit/TrackingFileChannel.java (+175/-0) src/test/java/com/persistit/WarmupTest.java (+84/-29) src/test/java/com/persistit/stress/PreloadMixtureTxn1.java (+82/-0) src/test/java/com/persistit/unit/PersistitUnitTestCase.java (+2/-1) src/test/java/com/persistit/unit/UnitTestProperties.java (+0/-1) |
To merge this branch: | bzr merge lp:~pbeaman/akiban-persistit/buffer-pool-warmup-sorted |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Akiban Technologies | Pending | ||
Review via email: mp+121091@code.launchpad.net |
This proposal has been superseded by a proposal from 2012-08-24.
Commit message
Description of the change
This proposal is a re-work of the original buffer pool warm-up code. Instead of creating a text file, this version stores the inventory in the system volume. It also loads pages in file-address order to reduce I/O time. There is a new stress test that pushes this into a medium-sized buffer pool (25K pages) but we still need to test it with a 1M page pool in TPCC or some other big test.
There are two config options, one to turn on inventory recording and the other to enable pre-loading pages. They are separate so that a server can start up without waiting for the pre-load but still be set to perform the inventory function.
Other changes include Configuration, Configuration.rst (documentation), etc.
- 383. By Peter Beaman
-
Merge from release-notes-3.16
- 384. By Peter Beaman
-
Fix failures in WarmupTest
- 385. By Peter Beaman
-
Fix WarmupTest (again)
Unmerged revisions
Preview Diff
1 | === modified file 'doc/Configuration.rst' | |||
2 | --- doc/Configuration.rst 2012-06-11 21:25:37 +0000 | |||
3 | +++ doc/Configuration.rst 2012-08-23 21:28:19 +0000 | |||
4 | @@ -221,7 +221,7 @@ | |||
5 | 221 | ``serialOverride``, ``constructorOverride``: (``com.persistit.Configuration#setSerialOverride`` ``com.persistit.Configuration#setConstructorOverride``) | 221 | ``serialOverride``, ``constructorOverride``: (``com.persistit.Configuration#setSerialOverride`` ``com.persistit.Configuration#setConstructorOverride``) |
6 | 222 | Control aspects of object serialization. See :ref:`Serialization`. | 222 | Control aspects of object serialization. See :ref:`Serialization`. |
7 | 223 | 223 | ||
9 | 224 | ``showgui``: (``com.persistit.Configuration#setShowGUI``), True of False. | 224 | ``showgui``: (``com.persistit.Configuration#setShowGUI``), True or False (default). |
10 | 225 | If true, Persistit attempts to create and display an instance of the AdminUI utility panel within the current JVM. | 225 | If true, Persistit attempts to create and display an instance of the AdminUI utility panel within the current JVM. |
11 | 226 | Alternatively, AdminUI uses RMI and can be launched and run remotely if ``rmiport`` or ``rmihost`` has been | 226 | Alternatively, AdminUI uses RMI and can be launched and run remotely if ``rmiport`` or ``rmihost`` has been |
12 | 227 | specified. | 227 | specified. |
13 | @@ -231,6 +231,18 @@ | |||
14 | 231 | install a logging adapter to reroute messages through Log4J, SLF4J or other logger. The ``logfile`` property is used | 231 | install a logging adapter to reroute messages through Log4J, SLF4J or other logger. The ``logfile`` property is used |
15 | 232 | only when no adapter has been installed. | 232 | only when no adapter has been installed. |
16 | 233 | 233 | ||
17 | 234 | ``bufferinventory``: (``com.persistit.Configuration#setBufferInventoryEnabled``), True or False (default). | ||
18 | 235 | If true, Persistit periodically records an inventory of all the buffes in the buffers pools to the System Volume. The inventory | ||
19 | 236 | enables Persistit to preload the buffer pools then next time it starts up with approximately the same pages that were present | ||
20 | 237 | before shutdown. To enable buffer preloading, the bufferpreload property must also be true. | ||
21 | 238 | |||
22 | 239 | ``bufferpreload``: (``com.persistit.Configuration#setBufferPreloadEnabled``), True or False (default). | ||
23 | 240 | If true, and if a buffer pool inventory was previously recorded, Persistit attempts to "warm up" the buffer pool | ||
24 | 241 | by preloading pages that were present in the buffer pool when Persistit last shut down. This may allow a freshly started | ||
25 | 242 | Persistit instance to begin service a workload similar to what it had previously been processing without incurring the | ||
26 | 243 | cost of many random disk reads to load pages. | ||
27 | 244 | |||
28 | 245 | |||
29 | 234 | For all integer-valued properties, the suffix “K” may be used to represent kilo, “M” for mega, “G” for giga and “T” for tera. For example, “2M” represents the value 2,097,152. | 246 | For all integer-valued properties, the suffix “K” may be used to represent kilo, “M” for mega, “G” for giga and “T” for tera. For example, “2M” represents the value 2,097,152. |
30 | 235 | 247 | ||
31 | 236 | A Configuration Example | 248 | A Configuration Example |
32 | 237 | 249 | ||
33 | === modified file 'src/main/java/com/persistit/BufferPool.java' | |||
34 | --- src/main/java/com/persistit/BufferPool.java 2012-08-23 21:28:19 +0000 | |||
35 | +++ src/main/java/com/persistit/BufferPool.java 2012-08-23 21:28:19 +0000 | |||
36 | @@ -15,16 +15,14 @@ | |||
37 | 15 | 15 | ||
38 | 16 | package com.persistit; | 16 | package com.persistit; |
39 | 17 | 17 | ||
40 | 18 | import java.io.BufferedReader; | ||
41 | 19 | import java.io.BufferedWriter; | ||
42 | 20 | import java.io.DataOutputStream; | 18 | import java.io.DataOutputStream; |
43 | 21 | import java.io.File; | ||
44 | 22 | import java.io.FileReader; | ||
45 | 23 | import java.io.FileWriter; | ||
46 | 24 | import java.io.IOException; | 19 | import java.io.IOException; |
47 | 25 | import java.nio.ByteBuffer; | 20 | import java.nio.ByteBuffer; |
48 | 21 | import java.util.ArrayList; | ||
49 | 26 | import java.util.Arrays; | 22 | import java.util.Arrays; |
50 | 23 | import java.util.Collections; | ||
51 | 27 | import java.util.HashSet; | 24 | import java.util.HashSet; |
52 | 25 | import java.util.List; | ||
53 | 28 | import java.util.Set; | 26 | import java.util.Set; |
54 | 29 | import java.util.concurrent.atomic.AtomicBoolean; | 27 | import java.util.concurrent.atomic.AtomicBoolean; |
55 | 30 | import java.util.concurrent.atomic.AtomicInteger; | 28 | import java.util.concurrent.atomic.AtomicInteger; |
56 | @@ -32,6 +30,7 @@ | |||
57 | 32 | import java.util.concurrent.atomic.AtomicLongArray; | 30 | import java.util.concurrent.atomic.AtomicLongArray; |
58 | 33 | import java.util.concurrent.locks.ReentrantLock; | 31 | import java.util.concurrent.locks.ReentrantLock; |
59 | 34 | 32 | ||
60 | 33 | import com.persistit.JournalManager.PageNode; | ||
61 | 35 | import com.persistit.exception.InUseException; | 34 | import com.persistit.exception.InUseException; |
62 | 36 | import com.persistit.exception.InvalidPageAddressException; | 35 | import com.persistit.exception.InvalidPageAddressException; |
63 | 37 | import com.persistit.exception.InvalidPageStructureException; | 36 | import com.persistit.exception.InvalidPageStructureException; |
64 | @@ -92,6 +91,17 @@ | |||
65 | 92 | */ | 91 | */ |
66 | 93 | private final static int WRITE_AGE_THRESHOLD_RATIO = 4; | 92 | private final static int WRITE_AGE_THRESHOLD_RATIO = 4; |
67 | 94 | 93 | ||
68 | 94 | private final static String INVENTORY_TREE_NAME = "_buffers"; | ||
69 | 95 | /** | ||
70 | 96 | * Maximum number of buffer inventory versions to retain | ||
71 | 97 | */ | ||
72 | 98 | private final static int INVENTORY_VERSIONS = 3; | ||
73 | 99 | |||
74 | 100 | /** | ||
75 | 101 | * Preload log multiple | ||
76 | 102 | */ | ||
77 | 103 | private final static int INVENTORY_PRELOAD_LOG_MESSAGE_MULTIPLE = 10000; | ||
78 | 104 | |||
79 | 95 | /** | 105 | /** |
80 | 96 | * The Persistit instance that references this BufferPool. | 106 | * The Persistit instance that references this BufferPool. |
81 | 97 | */ | 107 | */ |
82 | @@ -203,23 +213,11 @@ | |||
83 | 203 | private volatile int _pageWriterTrancheSize = PAGE_WRITER_TRANCHE_SIZE; | 213 | private volatile int _pageWriterTrancheSize = PAGE_WRITER_TRANCHE_SIZE; |
84 | 204 | 214 | ||
85 | 205 | /** | 215 | /** |
86 | 206 | * Polling interval for PageCacher | ||
87 | 207 | */ | ||
88 | 208 | private volatile long _cacherPollInterval; | ||
89 | 209 | |||
90 | 210 | /** | ||
91 | 211 | * The PAGE_WRITER IOTaskRunnable | 216 | * The PAGE_WRITER IOTaskRunnable |
92 | 212 | */ | 217 | */ |
93 | 213 | private PageWriter _writer; | 218 | private PageWriter _writer; |
94 | 214 | 219 | ||
95 | 215 | /** | 220 | /** |
96 | 216 | * The PAGE_CACHER IOTaskRunnable | ||
97 | 217 | */ | ||
98 | 218 | private PageCacher _cacher; | ||
99 | 219 | |||
100 | 220 | private String _defaultLogPath; | ||
101 | 221 | |||
102 | 222 | /** | ||
103 | 223 | * Construct a BufferPool with the specified count of <code>Buffer</code>s | 221 | * Construct a BufferPool with the specified count of <code>Buffer</code>s |
104 | 224 | * of the specified size. | 222 | * of the specified size. |
105 | 225 | * | 223 | * |
106 | @@ -289,37 +287,6 @@ | |||
107 | 289 | throw e; | 287 | throw e; |
108 | 290 | } | 288 | } |
109 | 291 | _writer = new PageWriter(); | 289 | _writer = new PageWriter(); |
110 | 292 | _cacher = new PageCacher(); | ||
111 | 293 | } | ||
112 | 294 | |||
113 | 295 | void warmupBufferPool(final String pathName, final String fname) throws PersistitException { | ||
114 | 296 | final File file = new File(pathName, fname + ".log"); | ||
115 | 297 | _defaultLogPath = file.getAbsolutePath(); | ||
116 | 298 | |||
117 | 299 | try { | ||
118 | 300 | if (!file.exists()) { | ||
119 | 301 | file.createNewFile(); | ||
120 | 302 | } | ||
121 | 303 | |||
122 | 304 | final BufferedReader reader = new BufferedReader(new FileReader(file)); | ||
123 | 305 | String currLine; | ||
124 | 306 | while ((currLine = reader.readLine()) != null) { | ||
125 | 307 | final String[] info = currLine.split(" "); | ||
126 | 308 | if (info.length == 2) { | ||
127 | 309 | final Volume vol = _persistit.getVolume(info[1]); | ||
128 | 310 | if (vol != null) { | ||
129 | 311 | final long page = Long.parseLong(info[0]); | ||
130 | 312 | final Buffer buff = get(vol, page, false, true); | ||
131 | 313 | buff.release(); | ||
132 | 314 | } | ||
133 | 315 | } | ||
134 | 316 | } | ||
135 | 317 | reader.close(); | ||
136 | 318 | _cacherPollInterval = _persistit.getConfiguration().getBufferInventoryPollingInterval(); | ||
137 | 319 | _cacher.start(); | ||
138 | 320 | } catch (final IOException e) { | ||
139 | 321 | throw new PersistitException(e); | ||
140 | 322 | } | ||
141 | 323 | } | 290 | } |
142 | 324 | 291 | ||
143 | 325 | void startThreads() throws PersistitException { | 292 | void startThreads() throws PersistitException { |
144 | @@ -329,9 +296,7 @@ | |||
145 | 329 | void close() { | 296 | void close() { |
146 | 330 | _closed.set(true); | 297 | _closed.set(true); |
147 | 331 | _persistit.waitForIOTaskStop(_writer); | 298 | _persistit.waitForIOTaskStop(_writer); |
148 | 332 | _persistit.waitForIOTaskStop(_cacher); | ||
149 | 333 | _writer = null; | 299 | _writer = null; |
150 | 334 | _cacher = null; | ||
151 | 335 | } | 300 | } |
152 | 336 | 301 | ||
153 | 337 | /** | 302 | /** |
154 | @@ -340,7 +305,6 @@ | |||
155 | 340 | */ | 305 | */ |
156 | 341 | void crash() { | 306 | void crash() { |
157 | 342 | IOTaskRunnable.crash(_writer); | 307 | IOTaskRunnable.crash(_writer); |
158 | 343 | IOTaskRunnable.crash(_cacher); | ||
159 | 344 | } | 308 | } |
160 | 345 | 309 | ||
161 | 346 | void flush(final long timestamp) throws PersistitInterruptedException { | 310 | void flush(final long timestamp) throws PersistitInterruptedException { |
162 | @@ -433,35 +397,6 @@ | |||
163 | 433 | } | 397 | } |
164 | 434 | } | 398 | } |
165 | 435 | 399 | ||
166 | 436 | private void populateWarmupFile() throws PersistitException { | ||
167 | 437 | final File file = new File(_defaultLogPath); | ||
168 | 438 | |||
169 | 439 | try { | ||
170 | 440 | final BufferedWriter writer = new BufferedWriter(new FileWriter(file)); | ||
171 | 441 | for (int i = 0; i < _buffers.length; ++i) { | ||
172 | 442 | final Buffer b = _buffers[i]; | ||
173 | 443 | if (b != null && b.isValid() && !b.isDirty()) { | ||
174 | 444 | final long page = b.getPageAddress(); | ||
175 | 445 | final Volume volume = b.getVolume(); | ||
176 | 446 | final long page2 = b.getPageAddress(); | ||
177 | 447 | final Volume volume2 = b.getVolume(); | ||
178 | 448 | |||
179 | 449 | // Check if buffer has changed while reading | ||
180 | 450 | if (page == page2 && volume == volume2 && volume != null) { | ||
181 | 451 | final String addr = Long.toString(page); | ||
182 | 452 | final String vol = volume.getName(); | ||
183 | 453 | writer.append(addr + " " + vol); | ||
184 | 454 | writer.newLine(); | ||
185 | 455 | writer.flush(); | ||
186 | 456 | } | ||
187 | 457 | } | ||
188 | 458 | } | ||
189 | 459 | writer.close(); | ||
190 | 460 | } catch (final IOException e) { | ||
191 | 461 | throw new PersistitException(e); | ||
192 | 462 | } | ||
193 | 463 | } | ||
194 | 464 | |||
195 | 465 | private boolean selected(final Buffer buffer, final int includeMask, final int excludeMask) { | 400 | private boolean selected(final Buffer buffer, final int includeMask, final int excludeMask) { |
196 | 466 | return ((includeMask == 0) || (buffer.getStatus() & includeMask) != 0) | 401 | return ((includeMask == 0) || (buffer.getStatus() & includeMask) != 0) |
197 | 467 | && (buffer.getStatus() & excludeMask) == 0; | 402 | && (buffer.getStatus() & excludeMask) == 0; |
198 | @@ -1403,35 +1338,6 @@ | |||
199 | 1403 | } | 1338 | } |
200 | 1404 | } | 1339 | } |
201 | 1405 | 1340 | ||
202 | 1406 | /** | ||
203 | 1407 | * Implementation of PAGE_CACHER thread | ||
204 | 1408 | */ | ||
205 | 1409 | class PageCacher extends IOTaskRunnable { | ||
206 | 1410 | |||
207 | 1411 | PageCacher() { | ||
208 | 1412 | super(BufferPool.this._persistit); | ||
209 | 1413 | } | ||
210 | 1414 | |||
211 | 1415 | void start() { | ||
212 | 1416 | start("PAGE_CACHER:" + _bufferSize, _cacherPollInterval); | ||
213 | 1417 | } | ||
214 | 1418 | |||
215 | 1419 | @Override | ||
216 | 1420 | public void runTask() throws Exception { | ||
217 | 1421 | populateWarmupFile(); | ||
218 | 1422 | } | ||
219 | 1423 | |||
220 | 1424 | @Override | ||
221 | 1425 | protected boolean shouldStop() { | ||
222 | 1426 | return _closed.get() && !isFlushing(); | ||
223 | 1427 | } | ||
224 | 1428 | |||
225 | 1429 | @Override | ||
226 | 1430 | protected long pollInterval() { | ||
227 | 1431 | return isFlushing() ? 0 : _cacherPollInterval; | ||
228 | 1432 | } | ||
229 | 1433 | } | ||
230 | 1434 | |||
231 | 1435 | @Override | 1341 | @Override |
232 | 1436 | public String toString() { | 1342 | public String toString() { |
233 | 1437 | return "BufferPool[" + _bufferCount + "@" + _bufferSize + (_closed.get() ? ":closed" : "") + "]"; | 1343 | return "BufferPool[" + _bufferCount + "@" + _bufferSize + (_closed.get() ? ":closed" : "") + "]"; |
234 | @@ -1486,4 +1392,133 @@ | |||
235 | 1486 | } | 1392 | } |
236 | 1487 | stream.flush(); | 1393 | stream.flush(); |
237 | 1488 | } | 1394 | } |
238 | 1395 | |||
239 | 1396 | void recordBufferInventory(final long timestamp) throws PersistitException { | ||
240 | 1397 | final Exchange exchange = getBufferInventoryExchange(); | ||
241 | 1398 | /* | ||
242 | 1399 | * Advisory only - transaction integrity not needed | ||
243 | 1400 | */ | ||
244 | 1401 | exchange.ignoreTransactions(); | ||
245 | 1402 | try { | ||
246 | 1403 | int total = 0; | ||
247 | 1404 | exchange.clear().append(_bufferSize).append(timestamp).append(Key.BEFORE); | ||
248 | 1405 | final Value value = exchange.getValue(); | ||
249 | 1406 | final int clockValueBefore = _clock.get(); | ||
250 | 1407 | for (int index = 0; index < _buffers.length; index++) { | ||
251 | 1408 | final Buffer buffer = _buffers[index]; | ||
252 | 1409 | long page1 = -1, page2 = -1; | ||
253 | 1410 | Volume volume1 = null, volume2 = null; | ||
254 | 1411 | if (buffer != null && buffer.isValid()) { | ||
255 | 1412 | while (true) { | ||
256 | 1413 | page1 = buffer.getPageAddress(); | ||
257 | 1414 | volume1 = buffer.getVolume(); | ||
258 | 1415 | page2 = buffer.getPageAddress(); | ||
259 | 1416 | volume2 = buffer.getVolume(); | ||
260 | 1417 | if (page1 == page2 && volume1 == volume2) { | ||
261 | 1418 | break; | ||
262 | 1419 | } | ||
263 | 1420 | Util.spinSleep(); | ||
264 | 1421 | } | ||
265 | 1422 | if (volume1 != null && !volume1.isTemporary()) { | ||
266 | 1423 | value.clear().setStreamMode(true); | ||
267 | 1424 | value.put(volume1.getHandle()); | ||
268 | 1425 | value.put(page1); | ||
269 | 1426 | exchange.to(index).store(); | ||
270 | 1427 | total++; | ||
271 | 1428 | } | ||
272 | 1429 | } | ||
273 | 1430 | } | ||
274 | 1431 | final int clockValueAfter = _clock.get(); | ||
275 | 1432 | exchange.cut(); | ||
276 | 1433 | value.clear().setStreamMode(true); | ||
277 | 1434 | value.put(_bufferCount); | ||
278 | 1435 | value.put(total); | ||
279 | 1436 | value.put(clockValueBefore); | ||
280 | 1437 | value.put(clockValueAfter); | ||
281 | 1438 | value.put(System.currentTimeMillis()); | ||
282 | 1439 | exchange.store(); | ||
283 | 1440 | int count = 0; | ||
284 | 1441 | while (exchange.previous()) { | ||
285 | 1442 | if (++count > INVENTORY_VERSIONS) { | ||
286 | 1443 | exchange.remove(Key.GTEQ); | ||
287 | 1444 | } | ||
288 | 1445 | } | ||
289 | 1446 | } catch (final PersistitException e) { | ||
290 | 1447 | _persistit.getLogBase().bufferInventoryException.log(e); | ||
291 | 1448 | } | ||
292 | 1449 | } | ||
293 | 1450 | |||
294 | 1451 | void preloadBufferInventory() { | ||
295 | 1452 | int count = 0; | ||
296 | 1453 | int total = 0; | ||
297 | 1454 | try { | ||
298 | 1455 | final JournalManager jman = _persistit.getJournalManager(); | ||
299 | 1456 | final Exchange exchange = getBufferInventoryExchange(); | ||
300 | 1457 | final Value value = exchange.getValue(); | ||
301 | 1458 | final List<PageNode> pageNodes = new ArrayList<PageNode>(); | ||
302 | 1459 | boolean foundInventory = false; | ||
303 | 1460 | exchange.clear().append(_bufferSize).append(Key.AFTER); | ||
304 | 1461 | while (exchange.previous()) { | ||
305 | 1462 | if (exchange.getValue().isDefined()) { | ||
306 | 1463 | foundInventory = true; | ||
307 | 1464 | break; | ||
308 | 1465 | } | ||
309 | 1466 | } | ||
310 | 1467 | if (!foundInventory) { | ||
311 | 1468 | return; | ||
312 | 1469 | } | ||
313 | 1470 | value.setStreamMode(true); | ||
314 | 1471 | /* int bufferCount = */value.getInt(); | ||
315 | 1472 | total = value.getInt(); | ||
316 | 1473 | /* int clockValueBefore = */value.getInt(); | ||
317 | 1474 | /* int clockValueAfter = */value.getInt(); | ||
318 | 1475 | final long systemTime = value.getLong(); | ||
319 | 1476 | |||
320 | 1477 | _persistit.getLogBase().bufferInventoryLoad.log(systemTime); | ||
321 | 1478 | |||
322 | 1479 | exchange.append(Key.BEFORE); | ||
323 | 1480 | |||
324 | 1481 | while (exchange.next()) { | ||
325 | 1482 | value.setStreamMode(true); | ||
326 | 1483 | final int volumeHandle = value.getInt(); | ||
327 | 1484 | final long pageAddress = value.getLong(); | ||
328 | 1485 | final PageNode pn = new PageNode(volumeHandle, pageAddress); | ||
329 | 1486 | pageNodes.add(pn); | ||
330 | 1487 | } | ||
331 | 1488 | |||
332 | 1489 | Collections.sort(pageNodes, PageNode.READ_COMPARATOR); | ||
333 | 1490 | for (final PageNode pn : pageNodes) { | ||
334 | 1491 | final Volume vol = jman.volumeForHandle(pn.getVolumeHandle()); | ||
335 | 1492 | if (vol == null) { | ||
336 | 1493 | continue; | ||
337 | 1494 | } | ||
338 | 1495 | try { | ||
339 | 1496 | final Buffer buff = get(vol, pn.getPageAddress(), false, true); | ||
340 | 1497 | buff.release(); | ||
341 | 1498 | count++; | ||
342 | 1499 | if ((count % INVENTORY_PRELOAD_LOG_MESSAGE_MULTIPLE) == 0) { | ||
343 | 1500 | _persistit.getLogBase().bufferInventoryProgress.log(count, total); | ||
344 | 1501 | } | ||
345 | 1502 | if (count >= _bufferCount) { | ||
346 | 1503 | // | ||
347 | 1504 | // If the buffer pool is now smaller, no need to load | ||
348 | 1505 | // more pages | ||
349 | 1506 | // | ||
350 | 1507 | break; | ||
351 | 1508 | } | ||
352 | 1509 | } catch (final PersistitException e) { | ||
353 | 1510 | // ignore it | ||
354 | 1511 | } | ||
355 | 1512 | } | ||
356 | 1513 | } catch (final PersistitException e) { | ||
357 | 1514 | _persistit.getLogBase().bufferInventoryException.log(e); | ||
358 | 1515 | } finally { | ||
359 | 1516 | _persistit.getLogBase().bufferInventoryProgress.log(count, total); | ||
360 | 1517 | } | ||
361 | 1518 | } | ||
362 | 1519 | |||
363 | 1520 | private Exchange getBufferInventoryExchange() throws PersistitException { | ||
364 | 1521 | final Volume sysvol = _persistit.getSystemVolume(); | ||
365 | 1522 | return _persistit.getExchange(sysvol, INVENTORY_TREE_NAME, true); | ||
366 | 1523 | } | ||
367 | 1489 | } | 1524 | } |
368 | 1490 | 1525 | ||
369 | === modified file 'src/main/java/com/persistit/CheckpointManager.java' | |||
370 | --- src/main/java/com/persistit/CheckpointManager.java 2012-08-23 21:28:19 +0000 | |||
371 | +++ src/main/java/com/persistit/CheckpointManager.java 2012-08-23 21:28:19 +0000 | |||
372 | @@ -201,6 +201,7 @@ | |||
373 | 201 | void pollCreateCheckpoint() throws PersistitException { | 201 | void pollCreateCheckpoint() throws PersistitException { |
374 | 202 | final long now = System.nanoTime(); | 202 | final long now = System.nanoTime(); |
375 | 203 | if (_lastCheckpointNanos + _checkpointIntervalNanos < now) { | 203 | if (_lastCheckpointNanos + _checkpointIntervalNanos < now) { |
376 | 204 | _persistit.recordBufferPoolInventory(); | ||
377 | 204 | createCheckpoint(); | 205 | createCheckpoint(); |
378 | 205 | } | 206 | } |
379 | 206 | } | 207 | } |
380 | @@ -252,12 +253,12 @@ | |||
381 | 252 | _currentCheckpoint = new Checkpoint(txn.getStartTimestamp(), System.currentTimeMillis()); | 253 | _currentCheckpoint = new Checkpoint(txn.getStartTimestamp(), System.currentTimeMillis()); |
382 | 253 | _outstandingCheckpoints.add(_currentCheckpoint); | 254 | _outstandingCheckpoints.add(_currentCheckpoint); |
383 | 254 | _persistit.getLogBase().checkpointProposed.log(_currentCheckpoint); | 255 | _persistit.getLogBase().checkpointProposed.log(_currentCheckpoint); |
384 | 255 | return _currentCheckpoint; | ||
385 | 256 | } catch (final InterruptedException ie) { | 256 | } catch (final InterruptedException ie) { |
386 | 257 | throw new PersistitInterruptedException(ie); | 257 | throw new PersistitInterruptedException(ie); |
387 | 258 | } finally { | 258 | } finally { |
388 | 259 | txn.end(); | 259 | txn.end(); |
389 | 260 | } | 260 | } |
390 | 261 | return _currentCheckpoint; | ||
391 | 261 | } finally { | 262 | } finally { |
392 | 262 | _persistit.setSessionId(saveSessionId); | 263 | _persistit.setSessionId(saveSessionId); |
393 | 263 | } | 264 | } |
394 | 264 | 265 | ||
395 | === modified file 'src/main/java/com/persistit/Configuration.java' | |||
396 | --- src/main/java/com/persistit/Configuration.java 2012-08-23 21:28:19 +0000 | |||
397 | +++ src/main/java/com/persistit/Configuration.java 2012-08-23 21:28:19 +0000 | |||
398 | @@ -263,7 +263,7 @@ | |||
399 | 263 | /** | 263 | /** |
400 | 264 | * Property name for the "append only" property. | 264 | * Property name for the "append only" property. |
401 | 265 | */ | 265 | */ |
403 | 266 | public final static String APPEND_ONLY_PROPERTY = "appendonly"; | 266 | public final static String APPEND_ONLY_PROPERTY_NAME = "appendonly"; |
404 | 267 | 267 | ||
405 | 268 | /** | 268 | /** |
406 | 269 | * Property name for the "ignore missing volumes" property. | 269 | * Property name for the "ignore missing volumes" property. |
407 | @@ -276,12 +276,15 @@ | |||
408 | 276 | public final static String SPLIT_POLICY_PROPERTY_NAME = "splitpolicy"; | 276 | public final static String SPLIT_POLICY_PROPERTY_NAME = "splitpolicy"; |
409 | 277 | 277 | ||
410 | 278 | /** | 278 | /** |
412 | 279 | * Property name to specify the"buffer inventory" property name. | 279 | * Property name to specify whether buffer preloading is enabled. |
413 | 280 | */ | ||
414 | 281 | public final static String BUFFER_PRELOAD_PROPERTY_NAME = "bufferpreload"; | ||
415 | 282 | |||
416 | 283 | /** | ||
417 | 284 | * Property name to specify whether buffer inventory is enabled. | ||
418 | 280 | */ | 285 | */ |
419 | 281 | public final static String BUFFER_INVENTORY_PROPERTY_NAME = "bufferinventory"; | 286 | public final static String BUFFER_INVENTORY_PROPERTY_NAME = "bufferinventory"; |
420 | 282 | 287 | ||
421 | 283 | public final static String BUFFER_POLLING_INTERVAL_PROPERTY = "bufferpollinginterval"; | ||
422 | 284 | |||
423 | 285 | /** | 288 | /** |
424 | 286 | * Property name to specify the default {@link JoinPolicy}. | 289 | * Property name to specify the default {@link JoinPolicy}. |
425 | 287 | */ | 290 | */ |
426 | @@ -634,9 +637,8 @@ | |||
427 | 634 | private int rmiServerPort; | 637 | private int rmiServerPort; |
428 | 635 | private boolean jmx = true; | 638 | private boolean jmx = true; |
429 | 636 | private boolean appendOnly; | 639 | private boolean appendOnly; |
433 | 637 | private String bufferInventoryPathName; | 640 | private boolean bufferInventoryEnabled; |
434 | 638 | private long bufferInventoryPollInterval = 3000000; // default five minute | 641 | private boolean bufferPreloadEnabled; |
432 | 639 | // polling | ||
435 | 640 | private boolean ignoreMissingVolumes; | 642 | private boolean ignoreMissingVolumes; |
436 | 641 | private String tmpVolDir; | 643 | private String tmpVolDir; |
437 | 642 | private int tmpVolPageSize; | 644 | private int tmpVolPageSize; |
438 | @@ -715,9 +717,7 @@ | |||
439 | 715 | } | 717 | } |
440 | 716 | 718 | ||
441 | 717 | void loadProperties() throws InvalidVolumeSpecificationException { | 719 | void loadProperties() throws InvalidVolumeSpecificationException { |
445 | 718 | setBufferInventoryPathName(getProperty(BUFFER_INVENTORY_PROPERTY_NAME)); | 720 | setAppendOnly(getBooleanProperty(APPEND_ONLY_PROPERTY_NAME, false)); |
443 | 719 | setBufferInventoryPollingInterval(getLongProperty(BUFFER_POLLING_INTERVAL_PROPERTY, bufferInventoryPollInterval)); | ||
444 | 720 | setAppendOnly(getBooleanProperty(APPEND_ONLY_PROPERTY, false)); | ||
446 | 721 | setCommitPolicy(getProperty(COMMIT_POLICY_PROPERTY_NAME)); | 721 | setCommitPolicy(getProperty(COMMIT_POLICY_PROPERTY_NAME)); |
447 | 722 | setConstructorOverride(getBooleanProperty(CONSTRUCTOR_OVERRIDE_PROPERTY_NAME, false)); | 722 | setConstructorOverride(getBooleanProperty(CONSTRUCTOR_OVERRIDE_PROPERTY_NAME, false)); |
448 | 723 | setIgnoreMissingVolumes(getBooleanProperty(IGNORE_MISSING_VOLUMES_PROPERTY, false)); | 723 | setIgnoreMissingVolumes(getBooleanProperty(IGNORE_MISSING_VOLUMES_PROPERTY, false)); |
449 | @@ -737,6 +737,8 @@ | |||
450 | 737 | setShowGUI(getBooleanProperty(SHOW_GUI_PROPERTY_NAME, false)); | 737 | setShowGUI(getBooleanProperty(SHOW_GUI_PROPERTY_NAME, false)); |
451 | 738 | setSplitPolicy(getProperty(SPLIT_POLICY_PROPERTY_NAME)); | 738 | setSplitPolicy(getProperty(SPLIT_POLICY_PROPERTY_NAME)); |
452 | 739 | setSysVolume(getProperty(SYSTEM_VOLUME_PROPERTY_NAME, DEFAULT_SYSTEM_VOLUME_NAME)); | 739 | setSysVolume(getProperty(SYSTEM_VOLUME_PROPERTY_NAME, DEFAULT_SYSTEM_VOLUME_NAME)); |
453 | 740 | setBufferInventoryEnabled(getBooleanProperty(BUFFER_INVENTORY_PROPERTY_NAME, false)); | ||
454 | 741 | setBufferPreloadEnabled(getBooleanProperty(BUFFER_PRELOAD_PROPERTY_NAME, false)); | ||
455 | 740 | 742 | ||
456 | 741 | loadPropertiesBufferSpecifications(); | 743 | loadPropertiesBufferSpecifications(); |
457 | 742 | loadPropertiesVolumeSpecifications(); | 744 | loadPropertiesVolumeSpecifications(); |
458 | @@ -1548,8 +1550,7 @@ | |||
459 | 1548 | /** | 1550 | /** |
460 | 1549 | * <p> | 1551 | * <p> |
461 | 1550 | * Set a pattern that identifies classes to be serialized using standard | 1552 | * Set a pattern that identifies classes to be serialized using standard |
464 | 1551 | * Java serialization rather than Persistit's default serialization. TODO | 1553 | * Java serialization rather than Persistit's default serialization. |
463 | 1552 | * Link to Serialization section of user_guide.html. | ||
465 | 1553 | * </p> | 1554 | * </p> |
466 | 1554 | * <p> | 1555 | * <p> |
467 | 1555 | * Default value is <code>null</code><br /> | 1556 | * Default value is <code>null</code><br /> |
468 | @@ -1559,6 +1560,8 @@ | |||
469 | 1559 | * @param serialOverride | 1560 | * @param serialOverride |
470 | 1560 | * the serial override pattern to set | 1561 | * the serial override pattern to set |
471 | 1561 | * @see DefaultCoderManager | 1562 | * @see DefaultCoderManager |
472 | 1563 | * @see <a | ||
473 | 1564 | * href="http://www.akiban.com/ak-docs/admin/persistit/Serialization.html">Serialization</a> | ||
474 | 1562 | */ | 1565 | */ |
475 | 1563 | public void setSerialOverride(final String serialOverride) { | 1566 | public void setSerialOverride(final String serialOverride) { |
476 | 1564 | this.serialOverride = serialOverride; | 1567 | this.serialOverride = serialOverride; |
477 | @@ -1579,8 +1582,7 @@ | |||
478 | 1579 | * a public no-argument constructor. If so, then that constructor is used | 1582 | * a public no-argument constructor. If so, then that constructor is used |
479 | 1580 | * when deserializing in the {@link DefaultObjectCoder}; if not then | 1583 | * when deserializing in the {@link DefaultObjectCoder}; if not then |
480 | 1581 | * Persistit uses private methods within the JDK to emulate standard Java | 1584 | * Persistit uses private methods within the JDK to emulate standard Java |
483 | 1582 | * serialization logic. TODO Link to Serialization section of | 1585 | * serialization logic. |
482 | 1583 | * user_guide.html. | ||
484 | 1584 | * </p> | 1586 | * </p> |
485 | 1585 | * <p> | 1587 | * <p> |
486 | 1586 | * Default value is <code>false</code><br /> | 1588 | * Default value is <code>false</code><br /> |
487 | @@ -1589,6 +1591,8 @@ | |||
488 | 1589 | * | 1591 | * |
489 | 1590 | * @param constructorOverride | 1592 | * @param constructorOverride |
490 | 1591 | * the constructorOverride to set | 1593 | * the constructorOverride to set |
491 | 1594 | * @see <a | ||
492 | 1595 | * href="http://www.akiban.com/ak-docs/admin/persistit/Serialization.html">Serialization</a> | ||
493 | 1592 | */ | 1596 | */ |
494 | 1593 | public void setConstructorOverride(final boolean constructorOverride) { | 1597 | public void setConstructorOverride(final boolean constructorOverride) { |
495 | 1594 | this.constructorOverride = constructorOverride; | 1598 | this.constructorOverride = constructorOverride; |
496 | @@ -1791,7 +1795,7 @@ | |||
497 | 1791 | /** | 1795 | /** |
498 | 1792 | * Return the value defined by {@link #setAppendOnly} | 1796 | * Return the value defined by {@link #setAppendOnly} |
499 | 1793 | * | 1797 | * |
501 | 1794 | * @return the whether to start Persistit in append-only mode | 1798 | * @return <true>true</code> if append-only mode is enabled at startup |
502 | 1795 | */ | 1799 | */ |
503 | 1796 | public boolean isAppendOnly() { | 1800 | public boolean isAppendOnly() { |
504 | 1797 | return appendOnly; | 1801 | return appendOnly; |
505 | @@ -1807,77 +1811,73 @@ | |||
506 | 1807 | * </p> | 1811 | * </p> |
507 | 1808 | * <p> | 1812 | * <p> |
508 | 1809 | * Default value is <code>false</code><br /> | 1813 | * Default value is <code>false</code><br /> |
510 | 1810 | * Property name is {@value #APPEND_ONLY_PROPERTY} | 1814 | * Property name is {@value #APPEND_ONLY_PROPERTY_NAME} |
511 | 1811 | * </p> | 1815 | * </p> |
512 | 1812 | * | 1816 | * |
513 | 1813 | * @param appendOnly | 1817 | * @param appendOnly |
515 | 1814 | * <code>true</code> to start Persistit in append-only only | 1818 | * <code>true</code> to start Persistit in append-only mode |
516 | 1815 | */ | 1819 | */ |
517 | 1816 | public void setAppendOnly(final boolean appendOnly) { | 1820 | public void setAppendOnly(final boolean appendOnly) { |
518 | 1817 | this.appendOnly = appendOnly; | 1821 | this.appendOnly = appendOnly; |
519 | 1818 | } | 1822 | } |
520 | 1819 | 1823 | ||
521 | 1820 | /** | 1824 | /** |
576 | 1821 | * Return the path name defined by {@link #getBufferInventoryPathName} | 1825 | * Return the value defined by {@link #setBufferInventoryEnabled} |
577 | 1822 | * | 1826 | * |
578 | 1823 | * @return the path where file to warm-up Persistit with sample buffer data | 1827 | * @return <code>true</code> if periodic buffer pool inventory recording is |
579 | 1824 | * is stored | 1828 | * enabled |
580 | 1825 | */ | 1829 | */ |
581 | 1826 | public String getBufferInventoryPathName() { | 1830 | public boolean isBufferInventoryEnabled() { |
582 | 1827 | return bufferInventoryPathName; | 1831 | return bufferInventoryEnabled; |
583 | 1828 | } | 1832 | } |
584 | 1829 | 1833 | ||
585 | 1830 | /** | 1834 | /** |
586 | 1831 | * <p> | 1835 | * <p> |
587 | 1832 | * Control where Persistit stores its buffer inventory. In this mode | 1836 | * Control whether Persistit periodically records an inventory of its buffer |
588 | 1833 | * Persistit restarts with information from the last run. This method | 1837 | * pools to enable buffer pool preloading on a subsequent startup. |
589 | 1834 | * initializes the warm-up file at the specified location, if none is | 1838 | * </p> |
590 | 1835 | * specified the buffer pool is not warmed up on start-up. | 1839 | * <p> |
591 | 1836 | * </p> | 1840 | * Default value is <code>false</code><br /> |
592 | 1837 | * <p> | 1841 | * Property name is {@value #BUFFER_INVENTORY_PROPERTY_NAME} |
593 | 1838 | * Default value is <code>null</code><br /> | 1842 | * |
594 | 1839 | * Property name is {@value #BUFFER_INVENTORY_PROPERTY_NAME} | 1843 | * @param bufferInventoryEnabled |
595 | 1840 | * </p> | 1844 | * <code>true</code> to enable periodic buffer inventory behavior |
596 | 1841 | * | 1845 | */ |
597 | 1842 | * @param pathName | 1846 | public void setBufferInventoryEnabled(final boolean bufferInventoryEnabled) { |
598 | 1843 | * the name of the path to the warm-up file | 1847 | this.bufferInventoryEnabled = bufferInventoryEnabled; |
599 | 1844 | */ | 1848 | } |
600 | 1845 | public void setBufferInventoryPathName(final String pathName) { | 1849 | |
601 | 1846 | bufferInventoryPathName = pathName; | 1850 | /** |
602 | 1847 | 1851 | * Return the value defined by {@link #setBufferPreloadEnabled} | |
603 | 1848 | } | 1852 | * |
604 | 1849 | 1853 | * @return <code>true</code> if the option to preload buffer pools is | |
605 | 1850 | /** | 1854 | * enabled |
606 | 1851 | * Return polling interval defined by | 1855 | */ |
607 | 1852 | * {@link #getBufferInventoryPollingInterval} | 1856 | public boolean isBufferPreloadEnabled() { |
608 | 1853 | * | 1857 | return bufferPreloadEnabled; |
609 | 1854 | * @return the number of seconds wait between warm-up polls | 1858 | } |
610 | 1855 | */ | 1859 | |
611 | 1856 | public long getBufferInventoryPollingInterval() { | 1860 | /** |
612 | 1857 | return bufferInventoryPollInterval; | 1861 | * <p> |
613 | 1858 | } | 1862 | * Control whether Persistit attempts to preload (warm up) the buffer pools |
614 | 1859 | 1863 | * by preloading pages recorded in a previously generated inventory. | |
615 | 1860 | /** | 1864 | * </p> |
616 | 1861 | * <p> | 1865 | * <p> |
617 | 1862 | * Control the number of seconds between each poll for the cache warm-up | 1866 | * Default value is <code>false</code><br /> |
618 | 1863 | * option in Persistit. | 1867 | * Property name is {@value #BUFFER_INVENTORY_PROPERTY_NAME} |
619 | 1864 | * </p> | 1868 | * |
620 | 1865 | * <p> | 1869 | * @param bufferPreloadEnabled |
621 | 1866 | * Default value is <code>3000</code><br /> | 1870 | * <code>true</code> to enable buffer pool preloading during |
622 | 1867 | * Property name is {@value #BUFFER_POLLING_INTERVAL_PROPERTY} | 1871 | * startup |
623 | 1868 | * </p> | 1872 | */ |
624 | 1869 | * | 1873 | public void setBufferPreloadEnabled(final boolean bufferPreloadEnabled) { |
625 | 1870 | * @param seconds | 1874 | this.bufferPreloadEnabled = bufferPreloadEnabled; |
572 | 1871 | * the number of seconds between polls | ||
573 | 1872 | */ | ||
574 | 1873 | public void setBufferInventoryPollingInterval(final long seconds) { | ||
575 | 1874 | bufferInventoryPollInterval = Util.rangeCheck(seconds, 60L, Long.MAX_VALUE) * 1000L; | ||
626 | 1875 | } | 1875 | } |
627 | 1876 | 1876 | ||
628 | 1877 | /** | 1877 | /** |
629 | 1878 | * Return the value defined by {@link #setIgnoreMissingVolumes(boolean)} | 1878 | * Return the value defined by {@link #setIgnoreMissingVolumes(boolean)} |
630 | 1879 | * | 1879 | * |
632 | 1880 | * @return the whether to start Persistit in ignore-missing-volumes mode | 1880 | * @return <code>true</code>to enable ignore-missing-volumes mode |
633 | 1881 | */ | 1881 | */ |
634 | 1882 | public boolean isIgnoreMissingVolumes() { | 1882 | public boolean isIgnoreMissingVolumes() { |
635 | 1883 | return ignoreMissingVolumes; | 1883 | return ignoreMissingVolumes; |
636 | 1884 | 1884 | ||
637 | === modified file 'src/main/java/com/persistit/JournalManager.java' | |||
638 | --- src/main/java/com/persistit/JournalManager.java 2012-08-23 21:28:19 +0000 | |||
639 | +++ src/main/java/com/persistit/JournalManager.java 2012-08-23 21:28:19 +0000 | |||
640 | @@ -690,6 +690,29 @@ | |||
641 | 690 | final ByteBuffer bb = buffer.getByteBuffer(); | 690 | final ByteBuffer bb = buffer.getByteBuffer(); |
642 | 691 | 691 | ||
643 | 692 | final Volume volume = buffer.getVolume(); | 692 | final Volume volume = buffer.getVolume(); |
644 | 693 | final PageNode pn = lookupUpPageNode(pageAddress, volume); | ||
645 | 694 | if (pn == null) { | ||
646 | 695 | return false; | ||
647 | 696 | } | ||
648 | 697 | bb.position(0); | ||
649 | 698 | final long recordPageAddress = readPageBufferFromJournal(pn, bb); | ||
650 | 699 | _persistit.getIOMeter().chargeReadPageFromJournal(volume, pageAddress, bufferSize, pn.getJournalAddress(), | ||
651 | 700 | buffer.getIndex()); | ||
652 | 701 | |||
653 | 702 | if (pageAddress != recordPageAddress) { | ||
654 | 703 | throw new CorruptJournalException("Record at " + pn + " is not volume/page " + buffer.toString()); | ||
655 | 704 | } | ||
656 | 705 | |||
657 | 706 | if (bb.limit() != bufferSize) { | ||
658 | 707 | throw new CorruptJournalException("Record at " + pn + " is wrong size: expected/actual=" + bufferSize + "/" | ||
659 | 708 | + bb.limit()); | ||
660 | 709 | } | ||
661 | 710 | _readPageCount++; | ||
662 | 711 | buffer.getVolume().getStatistics().bumpReadCounter(); | ||
663 | 712 | return true; | ||
664 | 713 | } | ||
665 | 714 | |||
666 | 715 | PageNode lookupUpPageNode(final long pageAddress, final Volume volume) { | ||
667 | 693 | PageNode pnLookup = null; | 716 | PageNode pnLookup = null; |
668 | 694 | synchronized (this) { | 717 | synchronized (this) { |
669 | 695 | final Integer volumeHandle = _volumeToHandleMap.get(volume); | 718 | final Integer volumeHandle = _volumeToHandleMap.get(volume); |
670 | @@ -699,7 +722,7 @@ | |||
671 | 699 | } | 722 | } |
672 | 700 | 723 | ||
673 | 701 | if (pnLookup == null) { | 724 | if (pnLookup == null) { |
675 | 702 | return false; | 725 | return null; |
676 | 703 | } | 726 | } |
677 | 704 | 727 | ||
678 | 705 | final PageNode pn = new PageNode(pnLookup.getVolumeHandle(), pnLookup.getPageAddress(), | 728 | final PageNode pn = new PageNode(pnLookup.getVolumeHandle(), pnLookup.getPageAddress(), |
679 | @@ -714,25 +737,9 @@ | |||
680 | 714 | * new checkpoints and that keeps the copier from deleting it. | 737 | * new checkpoints and that keeps the copier from deleting it. |
681 | 715 | */ | 738 | */ |
682 | 716 | if (pnLookup.isInvalid()) { | 739 | if (pnLookup.isInvalid()) { |
702 | 717 | return false; | 740 | return null; |
703 | 718 | } | 741 | } |
704 | 719 | 742 | return pn; | |
686 | 720 | bb.position(0); | ||
687 | 721 | final long recordPageAddress = readPageBufferFromJournal(pn, bb); | ||
688 | 722 | _persistit.getIOMeter().chargeReadPageFromJournal(volume, pageAddress, bufferSize, pn.getJournalAddress(), | ||
689 | 723 | buffer.getIndex()); | ||
690 | 724 | |||
691 | 725 | if (pageAddress != recordPageAddress) { | ||
692 | 726 | throw new CorruptJournalException("Record at " + pn + " is not volume/page " + buffer.toString()); | ||
693 | 727 | } | ||
694 | 728 | |||
695 | 729 | if (bb.limit() != bufferSize) { | ||
696 | 730 | throw new CorruptJournalException("Record at " + pn + " is wrong size: expected/actual=" + bufferSize + "/" | ||
697 | 731 | + bb.limit()); | ||
698 | 732 | } | ||
699 | 733 | _readPageCount++; | ||
700 | 734 | buffer.getVolume().getStatistics().bumpReadCounter(); | ||
701 | 735 | return true; | ||
705 | 736 | } | 743 | } |
706 | 737 | 744 | ||
707 | 738 | private long readPageBufferFromJournal(final PageNode pn, final ByteBuffer bb) throws PersistitIOException, | 745 | private long readPageBufferFromJournal(final PageNode pn, final ByteBuffer bb) throws PersistitIOException, |
708 | @@ -1829,6 +1836,10 @@ | |||
709 | 1829 | 1836 | ||
710 | 1830 | PageNode _previous; | 1837 | PageNode _previous; |
711 | 1831 | 1838 | ||
712 | 1839 | PageNode(final int volumeHandle, final long pageAddress) { | ||
713 | 1840 | this(volumeHandle, pageAddress, Long.MIN_VALUE, -1); | ||
714 | 1841 | } | ||
715 | 1842 | |||
716 | 1832 | PageNode(final int volumeHandle, final long pageAddress, final long journalAddress, final long timestamp) { | 1843 | PageNode(final int volumeHandle, final long pageAddress, final long journalAddress, final long timestamp) { |
717 | 1833 | this._volumeHandle = volumeHandle; | 1844 | this._volumeHandle = volumeHandle; |
718 | 1834 | this._pageAddress = pageAddress; | 1845 | this._pageAddress = pageAddress; |
719 | @@ -1950,8 +1961,20 @@ | |||
720 | 1950 | 1961 | ||
721 | 1951 | @Override | 1962 | @Override |
722 | 1952 | public int compare(final PageNode a, final PageNode b) { | 1963 | public int compare(final PageNode a, final PageNode b) { |
725 | 1953 | return a.getJournalAddress() > b.getJournalAddress() ? 1 : a.getJournalAddress() < b | 1964 | if (!a.isInvalid() && !b.isInvalid()) { |
726 | 1954 | .getJournalAddress() ? -1 : 0; | 1965 | return a.getJournalAddress() > b.getJournalAddress() ? 1 : a.getJournalAddress() < b |
727 | 1966 | .getJournalAddress() ? -1 : 0; | ||
728 | 1967 | } | ||
729 | 1968 | if (a.isInvalid() && !b.isInvalid()) { | ||
730 | 1969 | return -1; | ||
731 | 1970 | } | ||
732 | 1971 | if (!a.isInvalid() && b.isInvalid()) { | ||
733 | 1972 | return 1; | ||
734 | 1973 | } | ||
735 | 1974 | if (a._volumeHandle != b._volumeHandle) { | ||
736 | 1975 | return a._volumeHandle - b._volumeHandle; | ||
737 | 1976 | } | ||
738 | 1977 | return a._pageAddress > b._pageAddress ? 1 : a._pageAddress < b._pageAddress ? -1 : 0; | ||
739 | 1955 | } | 1978 | } |
740 | 1956 | }; | 1979 | }; |
741 | 1957 | 1980 | ||
742 | 1958 | 1981 | ||
743 | === modified file 'src/main/java/com/persistit/MediatedFileChannel.java' | |||
744 | --- src/main/java/com/persistit/MediatedFileChannel.java 2012-08-23 21:28:19 +0000 | |||
745 | +++ src/main/java/com/persistit/MediatedFileChannel.java 2012-08-23 21:28:19 +0000 | |||
746 | @@ -342,7 +342,7 @@ | |||
747 | 342 | * | 342 | * |
748 | 343 | * @param channel | 343 | * @param channel |
749 | 344 | */ | 344 | */ |
751 | 345 | void setErrorInjectingChannelForTests(final FileChannel channel) { | 345 | void injectChannelForTests(final FileChannel channel) { |
752 | 346 | ((TestChannelInjector) channel).setChannel(_channel); | 346 | ((TestChannelInjector) channel).setChannel(_channel); |
753 | 347 | _channel = channel; | 347 | _channel = channel; |
754 | 348 | } | 348 | } |
755 | 349 | 349 | ||
756 | === modified file 'src/main/java/com/persistit/Persistit.java' | |||
757 | --- src/main/java/com/persistit/Persistit.java 2012-08-23 21:28:19 +0000 | |||
758 | +++ src/main/java/com/persistit/Persistit.java 2012-08-23 21:28:19 +0000 | |||
759 | @@ -403,6 +403,7 @@ | |||
760 | 403 | 403 | ||
761 | 404 | private final AtomicBoolean _suspendShutdown = new AtomicBoolean(false); | 404 | private final AtomicBoolean _suspendShutdown = new AtomicBoolean(false); |
762 | 405 | private final AtomicBoolean _suspendUpdates = new AtomicBoolean(false); | 405 | private final AtomicBoolean _suspendUpdates = new AtomicBoolean(false); |
763 | 406 | private final AtomicBoolean _enableBufferInventory = new AtomicBoolean(false); | ||
764 | 406 | 407 | ||
765 | 407 | private UtilControl _localGUI; | 408 | private UtilControl _localGUI; |
766 | 408 | 409 | ||
767 | @@ -588,9 +589,7 @@ | |||
768 | 588 | initializeVolumes(); | 589 | initializeVolumes(); |
769 | 589 | startJournal(); | 590 | startJournal(); |
770 | 590 | startBufferPools(); | 591 | startBufferPools(); |
774 | 591 | if (_configuration.getBufferInventoryPathName() != null) { | 592 | preloadBufferPools(); |
772 | 592 | warmupBufferPools(); | ||
773 | 593 | } | ||
775 | 594 | finishRecovery(); | 593 | finishRecovery(); |
776 | 595 | startCheckpointManager(); | 594 | startCheckpointManager(); |
777 | 596 | startTransactionIndexPollTask(); | 595 | startTransactionIndexPollTask(); |
778 | @@ -699,6 +698,7 @@ | |||
779 | 699 | _defaultSplitPolicy = _configuration.getSplitPolicy(); | 698 | _defaultSplitPolicy = _configuration.getSplitPolicy(); |
780 | 700 | _defaultJoinPolicy = _configuration.getJoinPolicy(); | 699 | _defaultJoinPolicy = _configuration.getJoinPolicy(); |
781 | 701 | _defaultCommitPolicy = _configuration.getCommitPolicy(); | 700 | _defaultCommitPolicy = _configuration.getCommitPolicy(); |
782 | 701 | _enableBufferInventory.set(_configuration.isBufferInventoryEnabled()); | ||
783 | 702 | } | 702 | } |
784 | 703 | 703 | ||
785 | 704 | void startCheckpointManager() { | 704 | void startCheckpointManager() { |
786 | @@ -719,10 +719,24 @@ | |||
787 | 719 | } | 719 | } |
788 | 720 | } | 720 | } |
789 | 721 | 721 | ||
794 | 722 | void warmupBufferPools() throws PersistitException { | 722 | void recordBufferPoolInventory() { |
795 | 723 | final String pathName = _configuration.getBufferInventoryPathName(); | 723 | final long timestamp = _timestampAllocator.getCurrentTimestamp(); |
796 | 724 | for (final BufferPool pool : _bufferPoolTable.values()) { | 724 | if (_enableBufferInventory.get()) { |
797 | 725 | pool.warmupBufferPool(pathName, pool.toString()); | 725 | for (final BufferPool pool : _bufferPoolTable.values()) { |
798 | 726 | try { | ||
799 | 727 | pool.recordBufferInventory(timestamp); | ||
800 | 728 | } catch (PersistitException e) { | ||
801 | 729 | getLogBase().bufferInventoryException.log(e); | ||
802 | 730 | } | ||
803 | 731 | } | ||
804 | 732 | } | ||
805 | 733 | } | ||
806 | 734 | |||
807 | 735 | void preloadBufferPools() throws PersistitException { | ||
808 | 736 | if (_configuration.isBufferPreloadEnabled()) { | ||
809 | 737 | for (final BufferPool pool : _bufferPoolTable.values()) { | ||
810 | 738 | pool.preloadBufferInventory(); | ||
811 | 739 | } | ||
812 | 726 | } | 740 | } |
813 | 727 | } | 741 | } |
814 | 728 | 742 | ||
815 | @@ -974,8 +988,6 @@ | |||
816 | 974 | * | 988 | * |
817 | 975 | * @throws IllegalStateException | 989 | * @throws IllegalStateException |
818 | 976 | */ | 990 | */ |
819 | 977 | |||
820 | 978 | // TODO - why not one pool. | ||
821 | 979 | public void releaseExchange(final Exchange exchange, final boolean secure) { | 991 | public void releaseExchange(final Exchange exchange, final boolean secure) { |
822 | 980 | if (exchange == null) { | 992 | if (exchange == null) { |
823 | 981 | return; | 993 | return; |
824 | @@ -1638,6 +1650,7 @@ | |||
825 | 1638 | } | 1650 | } |
826 | 1639 | } | 1651 | } |
827 | 1640 | } | 1652 | } |
828 | 1653 | recordBufferPoolInventory(); | ||
829 | 1641 | 1654 | ||
830 | 1642 | /* | 1655 | /* |
831 | 1643 | * The copier is responsible for background pruning of aborted | 1656 | * The copier is responsible for background pruning of aborted |
832 | 1644 | 1657 | ||
833 | === modified file 'src/main/java/com/persistit/logging/LogBase.java' | |||
834 | --- src/main/java/com/persistit/logging/LogBase.java 2012-08-23 21:28:19 +0000 | |||
835 | +++ src/main/java/com/persistit/logging/LogBase.java 2012-08-23 21:28:19 +0000 | |||
836 | @@ -244,6 +244,15 @@ | |||
837 | 244 | @Message("ERROR|Too many journal files %,d") | 244 | @Message("ERROR|Too many journal files %,d") |
838 | 245 | public final LogItem tooManyJournalFilesError = PersistitLogMessage.empty(); | 245 | public final LogItem tooManyJournalFilesError = PersistitLogMessage.empty(); |
839 | 246 | 246 | ||
840 | 247 | @Message("INFO|Preloading buffer pool inventory recorded at %tc") | ||
841 | 248 | public final LogItem bufferInventoryLoad = PersistitLogMessage.empty(); | ||
842 | 249 | |||
843 | 250 | @Message("INFO|Preloaded %,d of %,d buffers") | ||
844 | 251 | public final LogItem bufferInventoryProgress = PersistitLogMessage.empty(); | ||
845 | 252 | |||
846 | 253 | @Message("WARNING|Exception while writing buffer pool inventory %s") | ||
847 | 254 | public final LogItem bufferInventoryException = PersistitLogMessage.empty(); | ||
848 | 255 | |||
849 | 247 | public static String recurring(final String message, final int count, final long duration) { | 256 | public static String recurring(final String message, final int count, final long duration) { |
850 | 248 | return String.format(RECURRING, message, count, duration); | 257 | return String.format(RECURRING, message, count, duration); |
851 | 249 | } | 258 | } |
852 | 250 | 259 | ||
853 | === modified file 'src/test/java/com/persistit/IOFailureTest.java' | |||
854 | --- src/test/java/com/persistit/IOFailureTest.java 2012-08-23 21:28:19 +0000 | |||
855 | +++ src/test/java/com/persistit/IOFailureTest.java 2012-08-23 21:28:19 +0000 | |||
856 | @@ -57,7 +57,7 @@ | |||
857 | 57 | 57 | ||
858 | 58 | private ErrorInjectingFileChannel errorInjectingChannel(final FileChannel channel) { | 58 | private ErrorInjectingFileChannel errorInjectingChannel(final FileChannel channel) { |
859 | 59 | final ErrorInjectingFileChannel eimfc = new ErrorInjectingFileChannel(); | 59 | final ErrorInjectingFileChannel eimfc = new ErrorInjectingFileChannel(); |
861 | 60 | ((MediatedFileChannel) channel).setErrorInjectingChannelForTests(eimfc); | 60 | ((MediatedFileChannel) channel).injectChannelForTests(eimfc); |
862 | 61 | return eimfc; | 61 | return eimfc; |
863 | 62 | } | 62 | } |
864 | 63 | 63 | ||
865 | 64 | 64 | ||
866 | === modified file 'src/test/java/com/persistit/StressRunner.java' | |||
867 | --- src/test/java/com/persistit/StressRunner.java 2012-08-23 21:28:19 +0000 | |||
868 | +++ src/test/java/com/persistit/StressRunner.java 2012-08-23 21:28:19 +0000 | |||
869 | @@ -31,6 +31,7 @@ | |||
870 | 31 | import com.persistit.stress.MixtureTxn1; | 31 | import com.persistit.stress.MixtureTxn1; |
871 | 32 | import com.persistit.stress.MixtureTxn2; | 32 | import com.persistit.stress.MixtureTxn2; |
872 | 33 | import com.persistit.stress.PersistitMap1; | 33 | import com.persistit.stress.PersistitMap1; |
873 | 34 | import com.persistit.stress.PreloadMixtureTxn1; | ||
874 | 34 | import com.persistit.stress.StartStop; | 35 | import com.persistit.stress.StartStop; |
875 | 35 | import com.persistit.stress.Stress10Suite; | 36 | import com.persistit.stress.Stress10Suite; |
876 | 36 | import com.persistit.stress.Stress12txnSuite; | 37 | import com.persistit.stress.Stress12txnSuite; |
877 | @@ -79,6 +80,7 @@ | |||
878 | 79 | _classes.add(Stress12txnSuite.class); | 80 | _classes.add(Stress12txnSuite.class); |
879 | 80 | _classes.add(Stress4Suite.class); | 81 | _classes.add(Stress4Suite.class); |
880 | 81 | _classes.add(Stress8txnSuite.class); | 82 | _classes.add(Stress8txnSuite.class); |
881 | 83 | _classes.add(PreloadMixtureTxn1.class); | ||
882 | 82 | } | 84 | } |
883 | 83 | 85 | ||
884 | 84 | private final static String DURATION_PARAM = "duration="; | 86 | private final static String DURATION_PARAM = "duration="; |
885 | 85 | 87 | ||
886 | === added file 'src/test/java/com/persistit/TrackingFileChannel.java' | |||
887 | --- src/test/java/com/persistit/TrackingFileChannel.java 1970-01-01 00:00:00 +0000 | |||
888 | +++ src/test/java/com/persistit/TrackingFileChannel.java 2012-08-23 21:28:19 +0000 | |||
889 | @@ -0,0 +1,175 @@ | |||
890 | 1 | /** | ||
891 | 2 | * Copyright © 2012 Akiban Technologies, Inc. All rights reserved. | ||
892 | 3 | * | ||
893 | 4 | * This program and the accompanying materials are made available | ||
894 | 5 | * under the terms of the Eclipse Public License v1.0 which | ||
895 | 6 | * accompanies this distribution, and is available at | ||
896 | 7 | * http://www.eclipse.org/legal/epl-v10.html | ||
897 | 8 | * | ||
898 | 9 | * This program may also be available under different license terms. | ||
899 | 10 | * For more information, see www.akiban.com or contact licensing@akiban.com. | ||
900 | 11 | * | ||
901 | 12 | * Contributors: | ||
902 | 13 | * Akiban Technologies, Inc. | ||
903 | 14 | */ | ||
904 | 15 | |||
905 | 16 | package com.persistit; | ||
906 | 17 | |||
907 | 18 | import static org.junit.Assert.assertEquals; | ||
908 | 19 | import static org.junit.Assert.assertTrue; | ||
909 | 20 | import static org.junit.Assert.fail; | ||
910 | 21 | import java.io.IOException; | ||
911 | 22 | import java.nio.ByteBuffer; | ||
912 | 23 | import java.nio.MappedByteBuffer; | ||
913 | 24 | import java.nio.channels.FileChannel; | ||
914 | 25 | import java.nio.channels.FileLock; | ||
915 | 26 | import java.nio.channels.ReadableByteChannel; | ||
916 | 27 | import java.nio.channels.WritableByteChannel; | ||
917 | 28 | import java.util.ArrayList; | ||
918 | 29 | import java.util.List; | ||
919 | 30 | |||
920 | 31 | import com.persistit.MediatedFileChannel.TestChannelInjector; | ||
921 | 32 | |||
922 | 33 | /** | ||
923 | 34 | * <p> | ||
924 | 35 | * A {@link FileChannel} implementation that simulates IOExceptions under | ||
925 | 36 | * control of a unit test program. This class implements only those methods used | ||
926 | 37 | * by Persistit; many methods of FileChannel throw | ||
927 | 38 | * {@link UnsupportedOperationException}. | ||
928 | 39 | * </p> | ||
929 | 40 | * | ||
930 | 41 | * @author peter | ||
931 | 42 | * | ||
932 | 43 | */ | ||
933 | 44 | class TrackingFileChannel extends FileChannel implements TestChannelInjector { | ||
934 | 45 | |||
935 | 46 | volatile FileChannel _channel; | ||
936 | 47 | |||
937 | 48 | final List<Long> _writePositions = new ArrayList<Long>(); | ||
938 | 49 | |||
939 | 50 | final List<Long> _readPositions = new ArrayList<Long>(); | ||
940 | 51 | |||
941 | 52 | @Override | ||
942 | 53 | public void setChannel(final FileChannel channel) { | ||
943 | 54 | _channel = channel; | ||
944 | 55 | } | ||
945 | 56 | |||
946 | 57 | @Override | ||
947 | 58 | protected void implCloseChannel() throws IOException { | ||
948 | 59 | _channel.close(); | ||
949 | 60 | } | ||
950 | 61 | |||
951 | 62 | @Override | ||
952 | 63 | public void force(final boolean metaData) throws IOException { | ||
953 | 64 | _channel.force(metaData); | ||
954 | 65 | } | ||
955 | 66 | |||
956 | 67 | @Override | ||
957 | 68 | public int read(final ByteBuffer byteBuffer, final long position) throws IOException { | ||
958 | 69 | _readPositions.add(position); | ||
959 | 70 | return _channel.read(byteBuffer, position); | ||
960 | 71 | } | ||
961 | 72 | |||
962 | 73 | @Override | ||
963 | 74 | public long size() throws IOException { | ||
964 | 75 | return _channel.size(); | ||
965 | 76 | } | ||
966 | 77 | |||
967 | 78 | @Override | ||
968 | 79 | public FileChannel truncate(final long size) throws IOException { | ||
969 | 80 | return _channel.truncate(size); | ||
970 | 81 | } | ||
971 | 82 | |||
972 | 83 | @Override | ||
973 | 84 | public synchronized FileLock tryLock(final long position, final long size, final boolean shared) throws IOException { | ||
974 | 85 | return _channel.tryLock(position, size, shared); | ||
975 | 86 | } | ||
976 | 87 | |||
977 | 88 | @Override | ||
978 | 89 | public int write(final ByteBuffer byteBuffer, final long position) throws IOException { | ||
979 | 90 | _writePositions.add(position); | ||
980 | 91 | final int written = _channel.write(byteBuffer, position); | ||
981 | 92 | return written; | ||
982 | 93 | } | ||
983 | 94 | |||
984 | 95 | /* | ||
985 | 96 | * -------------------------------- | ||
986 | 97 | * | ||
987 | 98 | * Persistit does not use these methods and so they are Unsupported. Note | ||
988 | 99 | * that it would be difficult to support the relative read/write methods | ||
989 | 100 | * because the channel size is unavailable after it is closed. Therefore a | ||
990 | 101 | * client of this class must maintain its own position counter and cannot | ||
991 | 102 | * use the relative-addressing calls. | ||
992 | 103 | * | ||
993 | 104 | * -------------------------------- | ||
994 | 105 | */ | ||
995 | 106 | @Override | ||
996 | 107 | public FileLock lock(final long position, final long size, final boolean shared) throws IOException { | ||
997 | 108 | throw new UnsupportedOperationException(); | ||
998 | 109 | } | ||
999 | 110 | |||
1000 | 111 | @Override | ||
1001 | 112 | public MappedByteBuffer map(final MapMode arg0, final long arg1, final long arg2) throws IOException { | ||
1002 | 113 | throw new UnsupportedOperationException(); | ||
1003 | 114 | } | ||
1004 | 115 | |||
1005 | 116 | @Override | ||
1006 | 117 | public long position() throws IOException { | ||
1007 | 118 | throw new UnsupportedOperationException(); | ||
1008 | 119 | } | ||
1009 | 120 | |||
1010 | 121 | @Override | ||
1011 | 122 | public FileChannel position(final long arg0) throws IOException { | ||
1012 | 123 | throw new UnsupportedOperationException(); | ||
1013 | 124 | } | ||
1014 | 125 | |||
1015 | 126 | @Override | ||
1016 | 127 | public int read(final ByteBuffer byteBuffer) throws IOException { | ||
1017 | 128 | throw new UnsupportedOperationException(); | ||
1018 | 129 | } | ||
1019 | 130 | |||
1020 | 131 | @Override | ||
1021 | 132 | public long read(final ByteBuffer[] arg0, final int arg1, final int arg2) throws IOException { | ||
1022 | 133 | throw new UnsupportedOperationException(); | ||
1023 | 134 | } | ||
1024 | 135 | |||
1025 | 136 | @Override | ||
1026 | 137 | public long transferFrom(final ReadableByteChannel arg0, final long arg1, final long arg2) throws IOException { | ||
1027 | 138 | throw new UnsupportedOperationException(); | ||
1028 | 139 | } | ||
1029 | 140 | |||
1030 | 141 | @Override | ||
1031 | 142 | public long transferTo(final long arg0, final long arg1, final WritableByteChannel arg2) throws IOException { | ||
1032 | 143 | throw new UnsupportedOperationException(); | ||
1033 | 144 | } | ||
1034 | 145 | |||
1035 | 146 | @Override | ||
1036 | 147 | public int write(final ByteBuffer byteBuffer) throws IOException { | ||
1037 | 148 | throw new UnsupportedOperationException(); | ||
1038 | 149 | } | ||
1039 | 150 | |||
1040 | 151 | @Override | ||
1041 | 152 | public long write(final ByteBuffer[] arg0, final int arg1, final int arg2) throws IOException { | ||
1042 | 153 | throw new UnsupportedOperationException(); | ||
1043 | 154 | } | ||
1044 | 155 | |||
1045 | 156 | public List<Long> getWritePositionList() { | ||
1046 | 157 | return _writePositions; | ||
1047 | 158 | } | ||
1048 | 159 | |||
1049 | 160 | public List<Long> getReadPositionList() { | ||
1050 | 161 | return _readPositions; | ||
1051 | 162 | } | ||
1052 | 163 | |||
1053 | 164 | public void assertSequential(boolean read, boolean forward) { | ||
1054 | 165 | final List<Long> list = read ? _readPositions : _writePositions; | ||
1055 | 166 | long previous = forward ? -1 : Long.MAX_VALUE; | ||
1056 | 167 | for (final Long position : list) { | ||
1057 | 168 | if (forward) { | ||
1058 | 169 | assertTrue("Position should be larger", position > previous); | ||
1059 | 170 | } else { | ||
1060 | 171 | assertTrue("Position should be smaller", position < previous); | ||
1061 | 172 | } | ||
1062 | 173 | } | ||
1063 | 174 | } | ||
1064 | 175 | } | ||
1065 | 0 | 176 | ||
1066 | === modified file 'src/test/java/com/persistit/WarmupTest.java' | |||
1067 | --- src/test/java/com/persistit/WarmupTest.java 2012-08-23 21:28:19 +0000 | |||
1068 | +++ src/test/java/com/persistit/WarmupTest.java 2012-08-23 21:28:19 +0000 | |||
1069 | @@ -16,6 +16,7 @@ | |||
1070 | 16 | package com.persistit; | 16 | package com.persistit; |
1071 | 17 | 17 | ||
1072 | 18 | import static org.junit.Assert.assertEquals; | 18 | import static org.junit.Assert.assertEquals; |
1073 | 19 | import static org.junit.Assert.assertTrue; | ||
1074 | 19 | 20 | ||
1075 | 20 | import java.util.Properties; | 21 | import java.util.Properties; |
1076 | 21 | 22 | ||
1077 | @@ -25,43 +26,97 @@ | |||
1078 | 25 | 26 | ||
1079 | 26 | public class WarmupTest extends PersistitUnitTestCase { | 27 | public class WarmupTest extends PersistitUnitTestCase { |
1080 | 27 | 28 | ||
1081 | 29 | @Override | ||
1082 | 30 | protected Properties getProperties(final boolean cleanup) { | ||
1083 | 31 | final Properties p = super.getProperties(cleanup); | ||
1084 | 32 | p.setProperty("bufferinventory", "true"); | ||
1085 | 33 | p.setProperty("bufferpreload", "true"); | ||
1086 | 34 | return p; | ||
1087 | 35 | } | ||
1088 | 36 | |||
1089 | 28 | @Test | 37 | @Test |
1090 | 29 | public void testWarmup() throws Exception { | 38 | public void testWarmup() throws Exception { |
1091 | 30 | Exchange ex = _persistit.getExchange("persistit", "WarmupTest", true); | 39 | Exchange ex = _persistit.getExchange("persistit", "WarmupTest", true); |
1092 | 40 | final BufferPool pool = ex.getBufferPool(); | ||
1093 | 31 | for (int i = 1; i <= 1000; i++) { | 41 | for (int i = 1; i <= 1000; i++) { |
1094 | 32 | ex.getValue().put(RED_FOX); | 42 | ex.getValue().put(RED_FOX); |
1095 | 33 | ex.clear().append(i).store(); | 43 | ex.clear().append(i).store(); |
1096 | 34 | } | 44 | } |
1097 | 35 | 45 | ||
1098 | 36 | // Assumption: only one buffer pool is created | ||
1099 | 37 | int poolCount = 0; | ||
1100 | 38 | String pathName = ""; | ||
1101 | 39 | final Buffer[] buff = new Buffer[100]; | 46 | final Buffer[] buff = new Buffer[100]; |
1128 | 40 | for (final BufferPool p : _persistit.getBufferPoolHashMap().values()) { | 47 | for (int i = 0; i < pool.getBufferCount(); ++i) { |
1129 | 41 | poolCount = p.getBufferCount(); | 48 | buff[i] = pool.getBufferCopy(i); |
1130 | 42 | pathName = p.toString(); | 49 | } |
1131 | 43 | for (int i = 0; i < poolCount; ++i) { | 50 | |
1132 | 44 | buff[i] = p.getBufferCopy(i); | 51 | final Configuration config = _persistit.getConfiguration(); |
1133 | 45 | } | 52 | ex = null; |
1134 | 46 | } | 53 | _persistit.close(); |
1135 | 47 | 54 | ||
1136 | 48 | final Properties properties = _persistit.getProperties(); | 55 | _persistit = new Persistit(); |
1137 | 49 | ex = null; | 56 | _persistit.initialize(config); |
1138 | 50 | _persistit.close(); | 57 | |
1139 | 51 | 58 | for (int i = 0; i < pool.getBufferCount(); ++i) { | |
1140 | 52 | _persistit = new Persistit(); | 59 | final Buffer bufferCopy = pool.getBufferCopy(i); |
1141 | 53 | _persistit.initialize(properties); | 60 | assertEquals(bufferCopy.getPageAddress(), buff[i].getPageAddress()); |
1142 | 54 | 61 | assertEquals(bufferCopy.getPageType(), buff[i].getPageType()); | |
1143 | 55 | int poolCount1 = 0; | 62 | assertEquals(bufferCopy.getBufferSize(), buff[i].getBufferSize()); |
1144 | 56 | for (final BufferPool p : _persistit.getBufferPoolHashMap().values()) { | 63 | } |
1145 | 57 | poolCount1 = p.getBufferCount(); | 64 | } |
1146 | 58 | for (int i = 0; i < poolCount1; ++i) { | 65 | |
1147 | 59 | final Buffer bufferCopy = p.getBufferCopy(i); | 66 | @Test |
1148 | 60 | assertEquals(bufferCopy.getPageAddress(), buff[i].getPageAddress()); | 67 | public void readOrderIsSequential() throws Exception { |
1149 | 61 | assertEquals(bufferCopy.getPageType(), buff[i].getPageType()); | 68 | |
1150 | 62 | assertEquals(bufferCopy.getBufferSize(), buff[i].getBufferSize()); | 69 | Exchange ex = _persistit.getExchange("persistit", "WarmupTest", true); |
1151 | 63 | } | 70 | BufferPool pool = ex.getBufferPool(); |
1152 | 64 | } | 71 | |
1153 | 65 | assertEquals(poolCount, poolCount1); | 72 | final int full = pool.getBufferCount() * (pool.getBufferSize() / RED_FOX.length()); |
1154 | 73 | /* | ||
1155 | 74 | * Overflow the buffer pool | ||
1156 | 75 | */ | ||
1157 | 76 | for (int i = 1; i <= full * 3; i++) { | ||
1158 | 77 | ex.getValue().put(RED_FOX); | ||
1159 | 78 | ex.clear().append(i).store(); | ||
1160 | 79 | } | ||
1161 | 80 | /* | ||
1162 | 81 | * Pull some low-address pages in to scramble the pool | ||
1163 | 82 | */ | ||
1164 | 83 | for (int i = full * 2; i >= 0; i -= 1000) { | ||
1165 | 84 | ex.clear().append(i).fetch(); | ||
1166 | 85 | } | ||
1167 | 86 | /* | ||
1168 | 87 | * Verify that buffers in pool now have somewhat scrambled page | ||
1169 | 88 | * addresses | ||
1170 | 89 | */ | ||
1171 | 90 | int breaks = 0; | ||
1172 | 91 | long previous = -1; | ||
1173 | 92 | |||
1174 | 93 | for (int i = 0; i < pool.getBufferCount(); i++) { | ||
1175 | 94 | final Buffer b = pool.getBufferCopy(i); | ||
1176 | 95 | assertTrue("Every buffer should be valid at this point", b.isValid()); | ||
1177 | 96 | if (b.getPageAddress() < previous) { | ||
1178 | 97 | breaks++; | ||
1179 | 98 | } | ||
1180 | 99 | previous = b.getPageAddress(); | ||
1181 | 100 | } | ||
1182 | 101 | |||
1183 | 102 | assertTrue("Buffer pool should have scrambled page address", breaks > 0); | ||
1184 | 103 | |||
1185 | 104 | final Configuration config = _persistit.getConfiguration(); | ||
1186 | 105 | ex = null; | ||
1187 | 106 | pool = null; | ||
1188 | 107 | _persistit.close(); | ||
1189 | 108 | |||
1190 | 109 | _persistit = new Persistit(); | ||
1191 | 110 | config.setBufferPreloadEnabled(false); | ||
1192 | 111 | _persistit.initialize(config); | ||
1193 | 112 | |||
1194 | 113 | final Volume volume = _persistit.getVolume("persistit"); | ||
1195 | 114 | final MediatedFileChannel mfc = (MediatedFileChannel) volume.getStorage().getChannel(); | ||
1196 | 115 | final TrackingFileChannel tfc = new TrackingFileChannel(); | ||
1197 | 116 | mfc.injectChannelForTests(tfc); | ||
1198 | 117 | pool = volume.getStructure().getPool(); | ||
1199 | 118 | pool.preloadBufferInventory(); | ||
1200 | 119 | assertTrue("Preload should have loaded pages from journal file", tfc.getReadPositionList().size() > 0); | ||
1201 | 120 | tfc.assertSequential(true, true); | ||
1202 | 66 | } | 121 | } |
1203 | 67 | } | 122 | } |
1204 | 68 | 123 | ||
1205 | === added file 'src/test/java/com/persistit/stress/PreloadMixtureTxn1.java' | |||
1206 | --- src/test/java/com/persistit/stress/PreloadMixtureTxn1.java 1970-01-01 00:00:00 +0000 | |||
1207 | +++ src/test/java/com/persistit/stress/PreloadMixtureTxn1.java 2012-08-23 21:28:19 +0000 | |||
1208 | @@ -0,0 +1,82 @@ | |||
1209 | 1 | /** | ||
1210 | 2 | * Copyright © 2012 Akiban Technologies, Inc. All rights reserved. | ||
1211 | 3 | * | ||
1212 | 4 | * This program and the accompanying materials are made available | ||
1213 | 5 | * under the terms of the Eclipse Public License v1.0 which | ||
1214 | 6 | * accompanies this distribution, and is available at | ||
1215 | 7 | * http://www.eclipse.org/legal/epl-v10.html | ||
1216 | 8 | * | ||
1217 | 9 | * This program may also be available under different license terms. | ||
1218 | 10 | * For more information, see www.akiban.com or contact licensing@akiban.com. | ||
1219 | 11 | * | ||
1220 | 12 | * Contributors: | ||
1221 | 13 | * Akiban Technologies, Inc. | ||
1222 | 14 | */ | ||
1223 | 15 | |||
1224 | 16 | package com.persistit.stress; | ||
1225 | 17 | |||
1226 | 18 | import com.persistit.Configuration; | ||
1227 | 19 | import com.persistit.Persistit; | ||
1228 | 20 | import com.persistit.Transaction.CommitPolicy; | ||
1229 | 21 | import com.persistit.stress.unit.Stress1; | ||
1230 | 22 | import com.persistit.stress.unit.Stress2txn; | ||
1231 | 23 | import com.persistit.stress.unit.Stress3; | ||
1232 | 24 | import com.persistit.stress.unit.Stress3txn; | ||
1233 | 25 | import com.persistit.stress.unit.Stress5; | ||
1234 | 26 | import com.persistit.stress.unit.Stress6; | ||
1235 | 27 | import com.persistit.stress.unit.Stress8txn; | ||
1236 | 28 | |||
1237 | 29 | public class PreloadMixtureTxn1 extends AbstractSuite { | ||
1238 | 30 | private final static int CYCLES = 4; | ||
1239 | 31 | |||
1240 | 32 | static String name() { | ||
1241 | 33 | return PreloadMixtureTxn1.class.getSimpleName(); | ||
1242 | 34 | } | ||
1243 | 35 | |||
1244 | 36 | public static void main(final String[] args) throws Exception { | ||
1245 | 37 | new PreloadMixtureTxn1(args).runTest(); | ||
1246 | 38 | } | ||
1247 | 39 | |||
1248 | 40 | public PreloadMixtureTxn1(final String[] args) { | ||
1249 | 41 | super(name(), args); | ||
1250 | 42 | setDuration(getDuration() / CYCLES); | ||
1251 | 43 | } | ||
1252 | 44 | |||
1253 | 45 | @Override | ||
1254 | 46 | public void runTest() throws Exception { | ||
1255 | 47 | |||
1256 | 48 | deleteFiles(substitute("$datapath$/persistit*")); | ||
1257 | 49 | |||
1258 | 50 | for (int iteration = 0; iteration < CYCLES; iteration++) { | ||
1259 | 51 | clear(); | ||
1260 | 52 | add(new Stress1("repeat=10 count=25000")); | ||
1261 | 53 | add(new Stress1("repeat=10 count=25000")); | ||
1262 | 54 | add(new Stress2txn("repeat=10 count=2500 size=4000 seed=118")); | ||
1263 | 55 | add(new Stress2txn("repeat=2 count=25000 seed=119")); | ||
1264 | 56 | add(new Stress3("repeat=5 count=25000 seed=119")); | ||
1265 | 57 | add(new Stress3txn("repeat=5 count=25000 seed=120")); | ||
1266 | 58 | add(new Stress3txn("repeat=5 count=25000")); | ||
1267 | 59 | add(new Stress5("repeat=5 count=25000")); | ||
1268 | 60 | add(new Stress6("repeat=5 count=1000 size=250")); | ||
1269 | 61 | add(new Stress6("repeat=10 count=1000 size=250")); | ||
1270 | 62 | add(new Stress8txn("repeat=2 count=1000 size=1000 seed=1")); | ||
1271 | 63 | add(new Stress8txn("repeat=2 count=1000 size=1000 seed=2")); | ||
1272 | 64 | add(new Stress8txn("repeat=2 count=1000 size=1000 seed=3")); | ||
1273 | 65 | add(new Stress8txn("repeat=2 count=1000 size=1000 seed=4")); | ||
1274 | 66 | |||
1275 | 67 | final Configuration configuration = makeConfiguration(16384, "25000", CommitPolicy.SOFT); | ||
1276 | 68 | configuration.setLogFile(configuration.getLogFile() + "_" + iteration); | ||
1277 | 69 | configuration.setBufferInventoryEnabled(true); | ||
1278 | 70 | configuration.setBufferPreloadEnabled(true); | ||
1279 | 71 | |||
1280 | 72 | final Persistit persistit = new Persistit(); | ||
1281 | 73 | persistit.initialize(configuration); | ||
1282 | 74 | |||
1283 | 75 | try { | ||
1284 | 76 | execute(persistit); | ||
1285 | 77 | } finally { | ||
1286 | 78 | persistit.close(); | ||
1287 | 79 | } | ||
1288 | 80 | } | ||
1289 | 81 | } | ||
1290 | 82 | } | ||
1291 | 0 | 83 | ||
1292 | === modified file 'src/test/java/com/persistit/unit/PersistitUnitTestCase.java' | |||
1293 | --- src/test/java/com/persistit/unit/PersistitUnitTestCase.java 2012-08-23 21:28:19 +0000 | |||
1294 | +++ src/test/java/com/persistit/unit/PersistitUnitTestCase.java 2012-08-23 21:28:19 +0000 | |||
1295 | @@ -47,7 +47,8 @@ | |||
1296 | 47 | protected Persistit _persistit = new Persistit(); | 47 | protected Persistit _persistit = new Persistit(); |
1297 | 48 | 48 | ||
1298 | 49 | protected Properties getProperties(final boolean cleanup) { | 49 | protected Properties getProperties(final boolean cleanup) { |
1300 | 50 | return UnitTestProperties.getProperties(cleanup); | 50 | final Properties p = UnitTestProperties.getProperties(cleanup); |
1301 | 51 | return p; | ||
1302 | 51 | } | 52 | } |
1303 | 52 | 53 | ||
1304 | 53 | @Before | 54 | @Before |
1305 | 54 | 55 | ||
1306 | === modified file 'src/test/java/com/persistit/unit/UnitTestProperties.java' | |||
1307 | --- src/test/java/com/persistit/unit/UnitTestProperties.java 2012-08-07 21:11:28 +0000 | |||
1308 | +++ src/test/java/com/persistit/unit/UnitTestProperties.java 2012-08-23 21:28:19 +0000 | |||
1309 | @@ -43,7 +43,6 @@ | |||
1310 | 43 | p.setProperty("tmpvoldir", "${datapath}"); | 43 | p.setProperty("tmpvoldir", "${datapath}"); |
1311 | 44 | p.setProperty("rmiport", System.getProperty("rmiport", "8081")); | 44 | p.setProperty("rmiport", System.getProperty("rmiport", "8081")); |
1312 | 45 | p.setProperty("jmx", "true"); | 45 | p.setProperty("jmx", "true"); |
1313 | 46 | p.setProperty("bufferinventory", "/tmp/persistit_test_data"); | ||
1314 | 47 | return p; | 46 | return p; |
1315 | 48 | } | 47 | } |
1316 | 49 | 48 |