Merge lp:~nfernandez/akiban-persistit/persistit-warmup-capability into lp:akiban-persistit
- persistit-warmup-capability
- Merge into trunk
Status: | Merged |
---|---|
Approved by: | Peter Beaman |
Approved revision: | 363 |
Merged at revision: | 352 |
Proposed branch: | lp:~nfernandez/akiban-persistit/persistit-warmup-capability |
Merge into: | lp:akiban-persistit |
Diff against target: |
438 lines (+259/-11) 6 files modified
src/main/java/com/persistit/BufferPool.java (+114/-9) src/main/java/com/persistit/Configuration.java (+66/-1) src/main/java/com/persistit/Persistit.java (+10/-0) src/test/java/com/persistit/WarmupTest.java (+66/-0) src/test/java/com/persistit/unit/PersistitUnitTestCase.java (+2/-1) src/test/java/com/persistit/unit/UnitTestProperties.java (+1/-0) |
To merge this branch: | bzr merge lp:~nfernandez/akiban-persistit/persistit-warmup-capability |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Peter Beaman | Approve | ||
Nohemi Fernandez (community) | Needs Resubmitting | ||
Review via email: mp+117780@code.launchpad.net |
Commit message
Description of the change
Optimization: Creates a file of sample page data to insert into the next 'cold' start of Persistit.
Nohemi Fernandez (nfernandez) wrote : | # |
- 356. By Nohemi Fernandez
-
revert changes to irrelevant files
- 357. By Nohemi Fernandez
-
merge from trunk
- 358. By Nohemi Fernandez
-
clean-up diff
Peter Beaman (pbeaman) wrote : | # |
I've been thinking some more about the general problem and have some suggestions.
This version represents good progress. It passes all tests and makes the warmup process optional.
Here are some additional issues I'd like to address, perhaps over time:
Writing the buffer pool inventory should be optional. I suggest using the same configuration property, isBufferWarmupE
When we load a large buffer pool, say 1M pages of 16Kb each (which we already have on largish machines) it will be important to optimize the order in which these are read to reduce physical I/O delays. About the best we can do is to read pages in file offset order which is approximately linear with page address order - "approximately" because some pages may be read from the journal instead of the volume file. We probably want to sort the list of pages being read by physical file address order before reading anything.
We may benefit from repopulating the BufferPool in "clock" order. This will generally not be the same order in which we load the pages. The idea would be to read the pages in physical file order but put them in the _buffers array in an order that approximates their distance from the "clock" pointer at the time the inventory was taken. To do this we'll need a method other than BufferPool#get(...) to read and install the pages. I'm giving this some thought and will probably propose something later. We can do without this for now.
I wonder if there is any benefit in reloading only pages that have been in the BufferPool for awhile. For example, if we inventory the BufferPool during a data load operating we'll record lots of pages in the warmup file that when reloaded will never be used again. One possibility is to snapshot the inventory somewhat more frequently, say once per minute, and only include pages that have been in place for two or three cycles.
Small detail - should have brought this up earlier. Each Volume in the BufferPool has a small integer handle available through Volume#getHandle(). This is a good alternative to recording the volume name.
Finally, we can't put this in trunk without resolving where the warmup file goes; clearly we can't keep it in a fixed location in the /tmp directory. I think if we can resolve the file location, make the writing of the file optional through configuration, fix the Javadoc for setBufferWarmup
- 359. By Nohemi Fernandez
-
change configuration property for buffer pools
- 360. By Nohemi Fernandez
-
add configuration setting for warm-up polling time
Nohemi Fernandez (nfernandez) wrote : | # |
I added configuration properties for the buffer pool warm-up as follows:
1. "bufferinventory" --> path name where buffer pool information will be stored, if null then pool does not get warmed up
2. "bufferpollingi
Peter Beaman (pbeaman) wrote : | # |
Does the PAGE_CACHER thread try to call populateWarmuFile using a path of null when? I didn't see logic to disable the thread when the bufferinventoryfile property is missing.
Nohemi Fernandez (nfernandez) wrote : | # |
316 + if (_configuration
317 + warmupBufferPoo
318 + }
The page cacher thread is not started unless the above method is called, so the path in BufferPool is unnecessary (null) if there is no path name in the configuration.
Peter Beaman (pbeaman) wrote : | # |
Suggestions:
- Merge from trunk
- Range-check the buffer poll interval - buffer inventory poll interval. Decide and document in javadoc whether it is specified in ms, seconds or minutes.
- Move property from PersistitUnitTe
- 361. By Nohemi Fernandez
-
range check buffer inventory poll interval
- 362. By Nohemi Fernandez
-
merge from trunk
- 363. By Nohemi Fernandez
-
update WarmupTest header
Nohemi Fernandez (nfernandez) wrote : | # |
Suggestions followed.
Peter Beaman (pbeaman) wrote : | # |
Looks okay to me. Thanks for the changes.
Preview Diff
1 | === modified file 'src/main/java/com/persistit/BufferPool.java' | |||
2 | --- src/main/java/com/persistit/BufferPool.java 2012-08-04 19:19:44 +0000 | |||
3 | +++ src/main/java/com/persistit/BufferPool.java 2012-08-07 21:14:18 +0000 | |||
4 | @@ -16,7 +16,6 @@ | |||
5 | 16 | package com.persistit; | 16 | package com.persistit; |
6 | 17 | 17 | ||
7 | 18 | import java.io.DataOutputStream; | 18 | import java.io.DataOutputStream; |
8 | 19 | import java.io.IOException; | ||
9 | 20 | import java.nio.ByteBuffer; | 19 | import java.nio.ByteBuffer; |
10 | 21 | import java.util.Arrays; | 20 | import java.util.Arrays; |
11 | 22 | import java.util.HashSet; | 21 | import java.util.HashSet; |
12 | @@ -37,6 +36,7 @@ | |||
13 | 37 | import com.persistit.exception.VolumeClosedException; | 36 | import com.persistit.exception.VolumeClosedException; |
14 | 38 | import com.persistit.util.Debug; | 37 | import com.persistit.util.Debug; |
15 | 39 | import com.persistit.util.Util; | 38 | import com.persistit.util.Util; |
16 | 39 | import java.io.*; | ||
17 | 40 | 40 | ||
18 | 41 | /** | 41 | /** |
19 | 42 | * A pool of {@link Buffer} objects, maintained on various lists that permit | 42 | * A pool of {@link Buffer} objects, maintained on various lists that permit |
20 | @@ -52,7 +52,7 @@ | |||
21 | 52 | private final static long DEFAULT_WRITER_POLL_INTERVAL = 5000; | 52 | private final static long DEFAULT_WRITER_POLL_INTERVAL = 5000; |
22 | 53 | 53 | ||
23 | 54 | private final static int PAGE_WRITER_TRANCHE_SIZE = 5000; | 54 | private final static int PAGE_WRITER_TRANCHE_SIZE = 5000; |
25 | 55 | 55 | ||
26 | 56 | /** | 56 | /** |
27 | 57 | * Sleep time when buffers are exhausted | 57 | * Sleep time when buffers are exhausted |
28 | 58 | */ | 58 | */ |
29 | @@ -196,11 +196,24 @@ | |||
30 | 196 | private volatile long _writerPollInterval = DEFAULT_WRITER_POLL_INTERVAL; | 196 | private volatile long _writerPollInterval = DEFAULT_WRITER_POLL_INTERVAL; |
31 | 197 | 197 | ||
32 | 198 | private volatile int _pageWriterTrancheSize = PAGE_WRITER_TRANCHE_SIZE; | 198 | private volatile int _pageWriterTrancheSize = PAGE_WRITER_TRANCHE_SIZE; |
33 | 199 | |||
34 | 200 | /** | ||
35 | 201 | * Polling interval for PageCacher | ||
36 | 202 | */ | ||
37 | 203 | private volatile long _cacherPollInterval; | ||
38 | 204 | |||
39 | 199 | /** | 205 | /** |
40 | 200 | * The PAGE_WRITER IOTaskRunnable | 206 | * The PAGE_WRITER IOTaskRunnable |
41 | 201 | */ | 207 | */ |
42 | 202 | private PageWriter _writer; | 208 | private PageWriter _writer; |
44 | 203 | 209 | ||
45 | 210 | /** | ||
46 | 211 | * The PAGE_CACHER IOTaskRunnable | ||
47 | 212 | */ | ||
48 | 213 | private PageCacher _cacher; | ||
49 | 214 | |||
50 | 215 | private String _defaultLogPath; | ||
51 | 216 | |||
52 | 204 | /** | 217 | /** |
53 | 205 | * Construct a BufferPool with the specified count of <code>Buffer</code>s | 218 | * Construct a BufferPool with the specified count of <code>Buffer</code>s |
54 | 206 | * of the specified size. | 219 | * of the specified size. |
55 | @@ -237,7 +250,7 @@ | |||
56 | 237 | _hashTable = new Buffer[_bufferCount * HASH_MULTIPLE]; | 250 | _hashTable = new Buffer[_bufferCount * HASH_MULTIPLE]; |
57 | 238 | _hashLocks = new ReentrantLock[HASH_LOCKS]; | 251 | _hashLocks = new ReentrantLock[HASH_LOCKS]; |
58 | 239 | _maxKeys = (_bufferSize - Buffer.HEADER_SIZE) / Buffer.MAX_KEY_RATIO; | 252 | _maxKeys = (_bufferSize - Buffer.HEADER_SIZE) / Buffer.MAX_KEY_RATIO; |
60 | 240 | 253 | ||
61 | 241 | for (int index = 0; index < HASH_LOCKS; index++) { | 254 | for (int index = 0; index < HASH_LOCKS; index++) { |
62 | 242 | _hashLocks[index] = new ReentrantLock(); | 255 | _hashLocks[index] = new ReentrantLock(); |
63 | 243 | } | 256 | } |
64 | @@ -271,25 +284,59 @@ | |||
65 | 271 | throw e; | 284 | throw e; |
66 | 272 | } | 285 | } |
67 | 273 | _writer = new PageWriter(); | 286 | _writer = new PageWriter(); |
72 | 274 | 287 | _cacher = new PageCacher(); | |
73 | 275 | } | 288 | } |
74 | 276 | 289 | ||
75 | 277 | void startThreads() { | 290 | void warmupBufferPool(String pathName, String fname) throws PersistitException { |
76 | 291 | File file = new File(pathName, fname + ".log"); | ||
77 | 292 | _defaultLogPath = file.getAbsolutePath(); | ||
78 | 293 | |||
79 | 294 | try { | ||
80 | 295 | if (!file.exists()) { | ||
81 | 296 | file.createNewFile(); | ||
82 | 297 | } | ||
83 | 298 | |||
84 | 299 | BufferedReader reader = new BufferedReader(new FileReader(file)); | ||
85 | 300 | String currLine; | ||
86 | 301 | while ((currLine = reader.readLine()) != null) { | ||
87 | 302 | String[] info = currLine.split(" "); | ||
88 | 303 | if (info.length == 2) { | ||
89 | 304 | Volume vol = _persistit.getVolume(info[1]); | ||
90 | 305 | if (vol != null) { | ||
91 | 306 | long page = Long.parseLong(info[0]); | ||
92 | 307 | Buffer buff = get(vol, page, false, true); | ||
93 | 308 | buff.release(); | ||
94 | 309 | } | ||
95 | 310 | } | ||
96 | 311 | } | ||
97 | 312 | reader.close(); | ||
98 | 313 | _cacherPollInterval = _persistit.getConfiguration().getBufferInventoryPollingInterval(); | ||
99 | 314 | _cacher.start(); | ||
100 | 315 | } | ||
101 | 316 | catch (IOException e) { | ||
102 | 317 | throw new PersistitException(e); | ||
103 | 318 | } | ||
104 | 319 | } | ||
105 | 320 | |||
106 | 321 | void startThreads() throws PersistitException { | ||
107 | 278 | _writer.start(); | 322 | _writer.start(); |
108 | 279 | } | 323 | } |
109 | 280 | 324 | ||
110 | 281 | void close() { | 325 | void close() { |
111 | 282 | _closed.set(true); | 326 | _closed.set(true); |
112 | 283 | _persistit.waitForIOTaskStop(_writer); | 327 | _persistit.waitForIOTaskStop(_writer); |
113 | 328 | _persistit.waitForIOTaskStop(_cacher); | ||
114 | 284 | _writer = null; | 329 | _writer = null; |
115 | 330 | _cacher = null; | ||
116 | 285 | } | 331 | } |
117 | 286 | 332 | ||
118 | 287 | /** | 333 | /** |
120 | 288 | * Abruptly stop (using {@link Thread#stop()}) the writer and collector | 334 | * Abruptly stop (using {@link Thread#stop()}) the writer, cacher, and collector |
121 | 289 | * threads. This method should be used only by tests. | 335 | * threads. This method should be used only by tests. |
122 | 290 | */ | 336 | */ |
123 | 291 | void crash() { | 337 | void crash() { |
124 | 292 | IOTaskRunnable.crash(_writer); | 338 | IOTaskRunnable.crash(_writer); |
125 | 339 | IOTaskRunnable.crash(_cacher); | ||
126 | 293 | } | 340 | } |
127 | 294 | 341 | ||
128 | 295 | void flush(final long timestamp) throws PersistitInterruptedException { | 342 | void flush(final long timestamp) throws PersistitInterruptedException { |
129 | @@ -380,6 +427,35 @@ | |||
130 | 380 | buffer.populateInfo(array[index]); | 427 | buffer.populateInfo(array[index]); |
131 | 381 | } | 428 | } |
132 | 382 | } | 429 | } |
133 | 430 | |||
134 | 431 | private void populateWarmupFile() throws PersistitException { | ||
135 | 432 | File file = new File(_defaultLogPath); | ||
136 | 433 | |||
137 | 434 | try { | ||
138 | 435 | BufferedWriter writer = new BufferedWriter(new FileWriter(file)); | ||
139 | 436 | for (int i = 0; i < _buffers.length; ++i) { | ||
140 | 437 | Buffer b = _buffers[i]; | ||
141 | 438 | if (b != null && b.isValid() && !b.isDirty()) { | ||
142 | 439 | long page = b.getPageAddress(); | ||
143 | 440 | Volume volume = b.getVolume(); | ||
144 | 441 | long page2 = b.getPageAddress(); | ||
145 | 442 | Volume volume2 = b.getVolume(); | ||
146 | 443 | |||
147 | 444 | // Check if buffer has changed while reading | ||
148 | 445 | if (page == page2 && volume == volume2 && volume != null) { | ||
149 | 446 | String addr = Long.toString(page); | ||
150 | 447 | String vol = volume.getName(); | ||
151 | 448 | writer.append(addr + " " + vol); | ||
152 | 449 | writer.newLine(); | ||
153 | 450 | writer.flush(); | ||
154 | 451 | } | ||
155 | 452 | } | ||
156 | 453 | } | ||
157 | 454 | writer.close(); | ||
158 | 455 | } catch (IOException e) { | ||
159 | 456 | throw new PersistitException(e); | ||
160 | 457 | } | ||
161 | 458 | } | ||
162 | 383 | 459 | ||
163 | 384 | private boolean selected(Buffer buffer, int includeMask, int excludeMask) { | 460 | private boolean selected(Buffer buffer, int includeMask, int excludeMask) { |
164 | 385 | return ((includeMask == 0) || (buffer.getStatus() & includeMask) != 0) | 461 | return ((includeMask == 0) || (buffer.getStatus() & includeMask) != 0) |
165 | @@ -1315,6 +1391,35 @@ | |||
166 | 1315 | return isFlushing() ? 0 : _writerPollInterval; | 1391 | return isFlushing() ? 0 : _writerPollInterval; |
167 | 1316 | } | 1392 | } |
168 | 1317 | } | 1393 | } |
169 | 1394 | |||
170 | 1395 | /** | ||
171 | 1396 | * Implementation of PAGE_CACHER thread | ||
172 | 1397 | */ | ||
173 | 1398 | class PageCacher extends IOTaskRunnable { | ||
174 | 1399 | |||
175 | 1400 | PageCacher() { | ||
176 | 1401 | super(BufferPool.this._persistit); | ||
177 | 1402 | } | ||
178 | 1403 | |||
179 | 1404 | void start() { | ||
180 | 1405 | start("PAGE_CACHER:" + _bufferSize, _cacherPollInterval); | ||
181 | 1406 | } | ||
182 | 1407 | |||
183 | 1408 | @Override | ||
184 | 1409 | public void runTask() throws Exception { | ||
185 | 1410 | populateWarmupFile(); | ||
186 | 1411 | } | ||
187 | 1412 | |||
188 | 1413 | @Override | ||
189 | 1414 | protected boolean shouldStop() { | ||
190 | 1415 | return _closed.get() && !isFlushing(); | ||
191 | 1416 | } | ||
192 | 1417 | |||
193 | 1418 | @Override | ||
194 | 1419 | protected long pollInterval() { | ||
195 | 1420 | return isFlushing() ? 0 : _cacherPollInterval; | ||
196 | 1421 | } | ||
197 | 1422 | } | ||
198 | 1318 | 1423 | ||
199 | 1319 | @Override | 1424 | @Override |
200 | 1320 | public String toString() { | 1425 | public String toString() { |
201 | 1321 | 1426 | ||
202 | === modified file 'src/main/java/com/persistit/Configuration.java' | |||
203 | --- src/main/java/com/persistit/Configuration.java 2012-08-02 14:19:26 +0000 | |||
204 | +++ src/main/java/com/persistit/Configuration.java 2012-08-07 21:14:18 +0000 | |||
205 | @@ -264,15 +264,23 @@ | |||
206 | 264 | * Property name for the "append only" property. | 264 | * Property name for the "append only" property. |
207 | 265 | */ | 265 | */ |
208 | 266 | public final static String APPEND_ONLY_PROPERTY = "appendonly"; | 266 | public final static String APPEND_ONLY_PROPERTY = "appendonly"; |
210 | 267 | 267 | ||
211 | 268 | /** | 268 | /** |
212 | 269 | * Property name for the "ignore missing volumes" property. | 269 | * Property name for the "ignore missing volumes" property. |
213 | 270 | */ | 270 | */ |
214 | 271 | public final static String IGNORE_MISSING_VOLUMES_PROPERTY = "ignoremissingvolumes"; | 271 | public final static String IGNORE_MISSING_VOLUMES_PROPERTY = "ignoremissingvolumes"; |
215 | 272 | |||
216 | 272 | /** | 273 | /** |
217 | 273 | * Property name to specify the default {@link SplitPolicy}. | 274 | * Property name to specify the default {@link SplitPolicy}. |
218 | 274 | */ | 275 | */ |
219 | 275 | public final static String SPLIT_POLICY_PROPERTY_NAME = "splitpolicy"; | 276 | public final static String SPLIT_POLICY_PROPERTY_NAME = "splitpolicy"; |
220 | 277 | |||
221 | 278 | /** | ||
222 | 279 | * Property name to specify the"buffer inventory" property name. | ||
223 | 280 | */ | ||
224 | 281 | public final static String BUFFER_INVENTORY_PROPERTY_NAME = "bufferinventory"; | ||
225 | 282 | |||
226 | 283 | public final static String BUFFER_POLLING_INTERVAL_PROPERTY = "bufferpollinginterval"; | ||
227 | 276 | 284 | ||
228 | 277 | /** | 285 | /** |
229 | 278 | * Property name to specify the default {@link JoinPolicy}. | 286 | * Property name to specify the default {@link JoinPolicy}. |
230 | @@ -624,6 +632,8 @@ | |||
231 | 624 | private int rmiServerPort; | 632 | private int rmiServerPort; |
232 | 625 | private boolean jmx = true; | 633 | private boolean jmx = true; |
233 | 626 | private boolean appendOnly; | 634 | private boolean appendOnly; |
234 | 635 | private String bufferInventoryPathName; | ||
235 | 636 | private long bufferInventoryPollInterval = 3000000; // default five minute polling | ||
236 | 627 | private boolean ignoreMissingVolumes; | 637 | private boolean ignoreMissingVolumes; |
237 | 628 | private String tmpVolDir; | 638 | private String tmpVolDir; |
238 | 629 | private int tmpVolPageSize; | 639 | private int tmpVolPageSize; |
239 | @@ -702,6 +712,8 @@ | |||
240 | 702 | } | 712 | } |
241 | 703 | 713 | ||
242 | 704 | void loadProperties() throws InvalidVolumeSpecificationException { | 714 | void loadProperties() throws InvalidVolumeSpecificationException { |
243 | 715 | setBufferInventoryPathName(getProperty(BUFFER_INVENTORY_PROPERTY_NAME)); | ||
244 | 716 | setBufferInventoryPollingInterval(getLongProperty(BUFFER_POLLING_INTERVAL_PROPERTY, bufferInventoryPollInterval)); | ||
245 | 705 | setAppendOnly(getBooleanProperty(APPEND_ONLY_PROPERTY, false)); | 717 | setAppendOnly(getBooleanProperty(APPEND_ONLY_PROPERTY, false)); |
246 | 706 | setCommitPolicy(getProperty(COMMIT_POLICY_PROPERTY_NAME)); | 718 | setCommitPolicy(getProperty(COMMIT_POLICY_PROPERTY_NAME)); |
247 | 707 | setConstructorOverride(getBooleanProperty(CONSTRUCTOR_OVERRIDE_PROPERTY_NAME, false)); | 719 | setConstructorOverride(getBooleanProperty(CONSTRUCTOR_OVERRIDE_PROPERTY_NAME, false)); |
248 | @@ -1802,6 +1814,59 @@ | |||
249 | 1802 | } | 1814 | } |
250 | 1803 | 1815 | ||
251 | 1804 | /** | 1816 | /** |
252 | 1817 | * Return the path name defined by {@link #getBufferInventoryPathName} | ||
253 | 1818 | * @return the path where file to warm-up Persistit with sample buffer data is stored | ||
254 | 1819 | */ | ||
255 | 1820 | public String getBufferInventoryPathName() { | ||
256 | 1821 | return bufferInventoryPathName; | ||
257 | 1822 | } | ||
258 | 1823 | |||
259 | 1824 | /** | ||
260 | 1825 | * <p> | ||
261 | 1826 | * Control where Persistit stores its buffer inventory. In this mode | ||
262 | 1827 | * Persistit restarts with information from the last run. This method initializes | ||
263 | 1828 | * the warm-up file at the specified location, if none is specified the buffer | ||
264 | 1829 | * pool is not warmed up on start-up. | ||
265 | 1830 | * </p> | ||
266 | 1831 | * <p> | ||
267 | 1832 | * Default value is <code>null</code><br /> | ||
268 | 1833 | * Property name is {@value #BUFFER_INVENTORY_PROPERTY_NAME} | ||
269 | 1834 | * </p> | ||
270 | 1835 | * | ||
271 | 1836 | * @param pathName | ||
272 | 1837 | * the name of the path to the warm-up file | ||
273 | 1838 | */ | ||
274 | 1839 | public void setBufferInventoryPathName(String pathName) { | ||
275 | 1840 | bufferInventoryPathName = pathName; | ||
276 | 1841 | |||
277 | 1842 | } | ||
278 | 1843 | |||
279 | 1844 | /** | ||
280 | 1845 | * Return polling interval defined by {@link #getBufferInventoryPollingInterval} | ||
281 | 1846 | * @return the number of seconds wait between warm-up polls | ||
282 | 1847 | */ | ||
283 | 1848 | public long getBufferInventoryPollingInterval() { | ||
284 | 1849 | return bufferInventoryPollInterval; | ||
285 | 1850 | } | ||
286 | 1851 | |||
287 | 1852 | /** | ||
288 | 1853 | * <p> | ||
289 | 1854 | * Control the number of seconds between each poll for the | ||
290 | 1855 | * cache warm-up option in Persistit. | ||
291 | 1856 | * </p> | ||
292 | 1857 | * <p> | ||
293 | 1858 | * Default value is <code>3000</code><br /> | ||
294 | 1859 | * Property name is {@value #BUFFER_POLLING_INTERVAL_PROPERTY} | ||
295 | 1860 | * </p> | ||
296 | 1861 | * | ||
297 | 1862 | * @param seconds | ||
298 | 1863 | * the number of seconds between polls | ||
299 | 1864 | */ | ||
300 | 1865 | public void setBufferInventoryPollingInterval(long seconds) { | ||
301 | 1866 | bufferInventoryPollInterval = Util.rangeCheck(seconds, 60L, Long.MAX_VALUE) * 1000L; | ||
302 | 1867 | } | ||
303 | 1868 | |||
304 | 1869 | /** | ||
305 | 1805 | * Return the value defined by {@link #setIgnoreMissingVolumes(boolean)} | 1870 | * Return the value defined by {@link #setIgnoreMissingVolumes(boolean)} |
306 | 1806 | * | 1871 | * |
307 | 1807 | * @return the whether to start Persistit in ignore-missing-volumes mode | 1872 | * @return the whether to start Persistit in ignore-missing-volumes mode |
308 | 1808 | 1873 | ||
309 | === modified file 'src/main/java/com/persistit/Persistit.java' | |||
310 | --- src/main/java/com/persistit/Persistit.java 2012-08-02 14:19:26 +0000 | |||
311 | +++ src/main/java/com/persistit/Persistit.java 2012-08-07 21:14:18 +0000 | |||
312 | @@ -589,6 +589,9 @@ | |||
313 | 589 | initializeVolumes(); | 589 | initializeVolumes(); |
314 | 590 | startJournal(); | 590 | startJournal(); |
315 | 591 | startBufferPools(); | 591 | startBufferPools(); |
316 | 592 | if (_configuration.getBufferInventoryPathName() != null) { | ||
317 | 593 | warmupBufferPools(); | ||
318 | 594 | } | ||
319 | 592 | finishRecovery(); | 595 | finishRecovery(); |
320 | 593 | startCheckpointManager(); | 596 | startCheckpointManager(); |
321 | 594 | startTransactionIndexPollTask(); | 597 | startTransactionIndexPollTask(); |
322 | @@ -716,6 +719,13 @@ | |||
323 | 716 | pool.startThreads(); | 719 | pool.startThreads(); |
324 | 717 | } | 720 | } |
325 | 718 | } | 721 | } |
326 | 722 | |||
327 | 723 | void warmupBufferPools() throws PersistitException { | ||
328 | 724 | String pathName = _configuration.getBufferInventoryPathName(); | ||
329 | 725 | for (final BufferPool pool : _bufferPoolTable.values()) { | ||
330 | 726 | pool.warmupBufferPool(pathName, pool.toString()); | ||
331 | 727 | } | ||
332 | 728 | } | ||
333 | 719 | 729 | ||
334 | 720 | void startJournal() throws PersistitException { | 730 | void startJournal() throws PersistitException { |
335 | 721 | _journalManager.startJournal(); | 731 | _journalManager.startJournal(); |
336 | 722 | 732 | ||
337 | === added file 'src/test/java/com/persistit/WarmupTest.java' | |||
338 | --- src/test/java/com/persistit/WarmupTest.java 1970-01-01 00:00:00 +0000 | |||
339 | +++ src/test/java/com/persistit/WarmupTest.java 2012-08-07 21:14:18 +0000 | |||
340 | @@ -0,0 +1,66 @@ | |||
341 | 1 | /** | ||
342 | 2 | * Copyright © 2012 Akiban Technologies, Inc. All rights reserved. | ||
343 | 3 | * | ||
344 | 4 | * This program and the accompanying materials are made available | ||
345 | 5 | * under the terms of the Eclipse Public License v1.0 which | ||
346 | 6 | * accompanies this distribution, and is available at | ||
347 | 7 | * http://www.eclipse.org/legal/epl-v10.html | ||
348 | 8 | * | ||
349 | 9 | * This program may also be available under different license terms. | ||
350 | 10 | * For more information, see www.akiban.com or contact licensing@akiban.com. | ||
351 | 11 | * | ||
352 | 12 | * Contributors: | ||
353 | 13 | * Akiban Technologies, Inc. | ||
354 | 14 | */ | ||
355 | 15 | |||
356 | 16 | package com.persistit; | ||
357 | 17 | |||
358 | 18 | import static org.junit.Assert.*; | ||
359 | 19 | import java.util.Properties; | ||
360 | 20 | |||
361 | 21 | import org.junit.Test; | ||
362 | 22 | |||
363 | 23 | import com.persistit.unit.PersistitUnitTestCase; | ||
364 | 24 | |||
365 | 25 | public class WarmupTest extends PersistitUnitTestCase { | ||
366 | 26 | |||
367 | 27 | @Test | ||
368 | 28 | public void testWarmup() throws Exception { | ||
369 | 29 | Exchange ex = _persistit.getExchange("persistit", "WarmupTest", true); | ||
370 | 30 | for (int i = 1; i <= 1000; i++) { | ||
371 | 31 | ex.getValue().put(RED_FOX); | ||
372 | 32 | ex.clear().append(i).store(); | ||
373 | 33 | } | ||
374 | 34 | |||
375 | 35 | // Assumption: only one buffer pool is created | ||
376 | 36 | int poolCount = 0; | ||
377 | 37 | String pathName = ""; | ||
378 | 38 | Buffer[] buff = new Buffer[100]; | ||
379 | 39 | for (BufferPool p: _persistit.getBufferPoolHashMap().values()) { | ||
380 | 40 | poolCount = p.getBufferCount(); | ||
381 | 41 | pathName = p.toString(); | ||
382 | 42 | for (int i = 0; i < poolCount; ++i) { | ||
383 | 43 | buff[i] = p.getBufferCopy(i); | ||
384 | 44 | } | ||
385 | 45 | } | ||
386 | 46 | |||
387 | 47 | Properties properties = _persistit.getProperties(); | ||
388 | 48 | ex = null; | ||
389 | 49 | _persistit.close(); | ||
390 | 50 | |||
391 | 51 | _persistit = new Persistit(); | ||
392 | 52 | _persistit.initialize(properties); | ||
393 | 53 | |||
394 | 54 | int poolCount1 = 0; | ||
395 | 55 | for (BufferPool p: _persistit.getBufferPoolHashMap().values()) { | ||
396 | 56 | poolCount1 = p.getBufferCount(); | ||
397 | 57 | for (int i = 0; i < poolCount1; ++i) { | ||
398 | 58 | Buffer bufferCopy = p.getBufferCopy(i); | ||
399 | 59 | assertEquals(bufferCopy.getPageAddress(), buff[i].getPageAddress()); | ||
400 | 60 | assertEquals(bufferCopy.getPageType(), buff[i].getPageType()); | ||
401 | 61 | assertEquals(bufferCopy.getBufferSize(), buff[i].getBufferSize()); | ||
402 | 62 | } | ||
403 | 63 | } | ||
404 | 64 | assertEquals(poolCount, poolCount1); | ||
405 | 65 | } | ||
406 | 66 | } | ||
407 | 0 | 67 | ||
408 | === modified file 'src/test/java/com/persistit/unit/PersistitUnitTestCase.java' | |||
409 | --- src/test/java/com/persistit/unit/PersistitUnitTestCase.java 2012-08-02 04:45:28 +0000 | |||
410 | +++ src/test/java/com/persistit/unit/PersistitUnitTestCase.java 2012-08-07 21:14:18 +0000 | |||
411 | @@ -53,6 +53,7 @@ | |||
412 | 53 | @Before | 53 | @Before |
413 | 54 | public void setUp() throws Exception { | 54 | public void setUp() throws Exception { |
414 | 55 | checkNoPersistitThreads(); | 55 | checkNoPersistitThreads(); |
415 | 56 | |||
416 | 56 | _persistit.initialize(getProperties(true)); | 57 | _persistit.initialize(getProperties(true)); |
417 | 57 | } | 58 | } |
418 | 58 | 59 | ||
419 | @@ -83,7 +84,7 @@ | |||
420 | 83 | } catch (final Throwable t) { | 84 | } catch (final Throwable t) { |
421 | 84 | t.printStackTrace(); | 85 | t.printStackTrace(); |
422 | 85 | } finally { | 86 | } finally { |
424 | 86 | tearDown(); | 87 | tearDown(); |
425 | 87 | } | 88 | } |
426 | 88 | } | 89 | } |
427 | 89 | 90 | ||
428 | 90 | 91 | ||
429 | === modified file 'src/test/java/com/persistit/unit/UnitTestProperties.java' | |||
430 | --- src/test/java/com/persistit/unit/UnitTestProperties.java 2012-08-02 04:45:28 +0000 | |||
431 | +++ src/test/java/com/persistit/unit/UnitTestProperties.java 2012-08-07 21:14:18 +0000 | |||
432 | @@ -43,6 +43,7 @@ | |||
433 | 43 | p.setProperty("tmpvoldir", "${datapath}"); | 43 | p.setProperty("tmpvoldir", "${datapath}"); |
434 | 44 | p.setProperty("rmiport", System.getProperty("rmiport", "8081")); | 44 | p.setProperty("rmiport", System.getProperty("rmiport", "8081")); |
435 | 45 | p.setProperty("jmx", "true"); | 45 | p.setProperty("jmx", "true"); |
436 | 46 | p.setProperty("bufferinventory", "/tmp/persistit_test_data"); | ||
437 | 46 | return p; | 47 | return p; |
438 | 47 | } | 48 | } |
439 | 48 | 49 |
Some optimizations to consider for this branch:
1. Add checks for corrupt files
2. Store temporary back-up file of data in case of a system crash
3. Define names for multiple buffer pools-- this will be implemented if we switch from the current file I/O implementation to one using Exchange objects.