Merge lp:~akopytov/percona-server/i26611-bug1059812-5.5 into lp:percona-server/5.5

Proposed by Alexey Kopytov
Status: Merged
Approved by: Laurynas Biveinis
Approved revision: no longer in the source branch.
Merged at revision: 330
Proposed branch: lp:~akopytov/percona-server/i26611-bug1059812-5.5
Merge into: lp:percona-server/5.5
Diff against target: 441 lines (+177/-10)
15 files modified
Percona-Server/mysql-test/r/percona_server_variables_debug.result (+1/-0)
Percona-Server/mysql-test/r/percona_server_variables_release.result (+1/-0)
Percona-Server/mysql-test/suite/sys_vars/r/innodb_buffer_pool_populate_basic.result (+12/-0)
Percona-Server/mysql-test/suite/sys_vars/t/innodb_buffer_pool_populate_basic-master.opt (+1/-0)
Percona-Server/mysql-test/suite/sys_vars/t/innodb_buffer_pool_populate_basic.test (+16/-0)
Percona-Server/scripts/mysqld_safe.sh (+68/-0)
Percona-Server/storage/innobase/buf/buf0buf.c (+7/-4)
Percona-Server/storage/innobase/handler/ha_innodb.cc (+7/-0)
Percona-Server/storage/innobase/include/buf0buf.h (+1/-0)
Percona-Server/storage/innobase/include/os0proc.h (+2/-1)
Percona-Server/storage/innobase/include/srv0srv.h (+1/-0)
Percona-Server/storage/innobase/os/os0proc.c (+55/-3)
Percona-Server/storage/innobase/row/row0merge.c (+1/-1)
Percona-Server/storage/innobase/srv/srv0srv.c (+2/-0)
Percona-Server/storage/innobase/srv/srv0start.c (+2/-1)
To merge this branch: bzr merge lp:~akopytov/percona-server/i26611-bug1059812-5.5
Reviewer Review Type Date Requested Status
Laurynas Biveinis (community) Approve
Davi Arnaut (community) Needs Fixing
Review via email: mp+130320@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Alexey Kopytov (akopytov) wrote :
Revision history for this message
Laurynas Biveinis (laurynas-biveinis) wrote :

Alexey -

It's really uncommon to see my_bool instead of ibool outside of handler/ and to lesser extent srv/. I would contain it to these two dirs by changing srv0start.c to

buf_pool_init (... srv_buf_pool_populate ? TRUE : FALSE, ... );

and making the rest of changes use ibool. I think that this horrible one line is still better than mixing bool types in the lower InnoDB subsystems.

The 2.6.23 version check is something I'd rather not see, but brief checking seems to show that this indeed the only way to find out if MAP_PRIVATE | MAP_POPULATE works :(

review: Needs Fixing
Revision history for this message
Jeremy Cole (jeremycole) wrote :

BTW, it seems that the get_time/set_time changes in sql/sp_head.cc and sql/sql_class.h are not related to the main bug being fixed here.

Revision history for this message
Davi Arnaut (davi) wrote :
review: Needs Fixing
Revision history for this message
Alexey Kopytov (akopytov) wrote :

Merged the bug fix, thanks!

Revision history for this message
Davi Arnaut (davi) wrote :

The my_bool/ibool cleanup suggestion is good, will take it.

Revision history for this message
Laurynas Biveinis (laurynas-biveinis) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'Percona-Server/mysql-test/r/percona_server_variables_debug.result'
2--- Percona-Server/mysql-test/r/percona_server_variables_debug.result 2012-10-12 06:24:13 +0000
3+++ Percona-Server/mysql-test/r/percona_server_variables_debug.result 2012-10-18 18:03:21 +0000
4@@ -88,6 +88,7 @@
5 INNODB_AUTOINC_LOCK_MODE
6 INNODB_BLOCKING_BUFFER_POOL_RESTORE
7 INNODB_BUFFER_POOL_INSTANCES
8+INNODB_BUFFER_POOL_POPULATE
9 INNODB_BUFFER_POOL_RESTORE_AT_STARTUP
10 INNODB_BUFFER_POOL_SHM_CHECKSUM
11 INNODB_BUFFER_POOL_SHM_KEY
12
13=== modified file 'Percona-Server/mysql-test/r/percona_server_variables_release.result'
14--- Percona-Server/mysql-test/r/percona_server_variables_release.result 2012-09-28 01:36:45 +0000
15+++ Percona-Server/mysql-test/r/percona_server_variables_release.result 2012-10-18 18:03:21 +0000
16@@ -86,6 +86,7 @@
17 INNODB_AUTOINC_LOCK_MODE
18 INNODB_BLOCKING_BUFFER_POOL_RESTORE
19 INNODB_BUFFER_POOL_INSTANCES
20+INNODB_BUFFER_POOL_POPULATE
21 INNODB_BUFFER_POOL_RESTORE_AT_STARTUP
22 INNODB_BUFFER_POOL_SHM_CHECKSUM
23 INNODB_BUFFER_POOL_SHM_KEY
24
25=== added file 'Percona-Server/mysql-test/suite/sys_vars/r/innodb_buffer_pool_populate_basic.result'
26--- Percona-Server/mysql-test/suite/sys_vars/r/innodb_buffer_pool_populate_basic.result 1970-01-01 00:00:00 +0000
27+++ Percona-Server/mysql-test/suite/sys_vars/r/innodb_buffer_pool_populate_basic.result 2012-10-18 18:03:21 +0000
28@@ -0,0 +1,12 @@
29+CALL mtr.add_suppression(".* Forcing preallocation by faulting in pages.");
30+SELECT @@GLOBAL.innodb_buffer_pool_populate;
31+@@GLOBAL.innodb_buffer_pool_populate
32+1
33+1 Expected
34+SET @@GLOBAL.innodb_buffer_pool_populate=0;
35+ERROR HY000: Variable 'innodb_buffer_pool_populate' is a read only variable
36+Expected error 'Read only variable'
37+SELECT @@GLOBAL.innodb_buffer_pool_populate;
38+@@GLOBAL.innodb_buffer_pool_populate
39+1
40+1 Expected
41
42=== added file 'Percona-Server/mysql-test/suite/sys_vars/t/innodb_buffer_pool_populate_basic-master.opt'
43--- Percona-Server/mysql-test/suite/sys_vars/t/innodb_buffer_pool_populate_basic-master.opt 1970-01-01 00:00:00 +0000
44+++ Percona-Server/mysql-test/suite/sys_vars/t/innodb_buffer_pool_populate_basic-master.opt 2012-10-18 18:03:21 +0000
45@@ -0,0 +1,1 @@
46+--innodb-buffer-pool-populate=true
47
48=== added file 'Percona-Server/mysql-test/suite/sys_vars/t/innodb_buffer_pool_populate_basic.test'
49--- Percona-Server/mysql-test/suite/sys_vars/t/innodb_buffer_pool_populate_basic.test 1970-01-01 00:00:00 +0000
50+++ Percona-Server/mysql-test/suite/sys_vars/t/innodb_buffer_pool_populate_basic.test 2012-10-18 18:03:21 +0000
51@@ -0,0 +1,16 @@
52+--source include/have_innodb.inc
53+
54+CALL mtr.add_suppression(".* Forcing preallocation by faulting in pages.");
55+
56+# Display current value of innodb_use_sys_malloc
57+SELECT @@GLOBAL.innodb_buffer_pool_populate;
58+--echo 1 Expected
59+
60+# Variable should be read-only
61+--error ER_INCORRECT_GLOBAL_LOCAL_VAR
62+SET @@GLOBAL.innodb_buffer_pool_populate=0;
63+--echo Expected error 'Read only variable'
64+
65+SELECT @@GLOBAL.innodb_buffer_pool_populate;
66+--echo 1 Expected
67+
68
69=== modified file 'Percona-Server/scripts/mysqld_safe.sh'
70--- Percona-Server/scripts/mysqld_safe.sh 2011-11-10 13:46:51 +0000
71+++ Percona-Server/scripts/mysqld_safe.sh 2012-10-18 18:03:21 +0000
72@@ -17,6 +17,8 @@
73 niceness=0
74 mysqld_ld_preload=
75 mysqld_ld_library_path=
76+flush_caches=0
77+numa_interleave=0
78
79 # Initial logging status: error log is not open, and not using syslog
80 logging=init
81@@ -60,6 +62,10 @@
82 --syslog Log messages to syslog with 'logger'
83 --skip-syslog Log messages to error log (default)
84 --syslog-tag=TAG Pass -t "mysqld-TAG" to 'logger'
85+ --flush-caches Flush and purge buffers/caches before
86+ starting the server
87+ --numa-interleave Run mysqld with its memory interleaved
88+ on all NUMA nodes
89
90 All other options are passed to the mysqld program.
91
92@@ -205,6 +211,8 @@
93 --skip-syslog) want_syslog=0 ;;
94 --syslog-tag=*) syslog_tag="$val" ;;
95 --timezone=*) TZ="$val"; export TZ; ;;
96+ --flush-caches) flush_caches=1 ;;
97+ --numa-interleave) numa_interleave=1 ;;
98
99 --help) usage ;;
100
101@@ -711,6 +719,41 @@
102 fi
103
104 #
105+# Flush and purge buffers/caches.
106+#
107+
108+if @TARGET_LINUX@ && test $flush_caches -eq 1
109+then
110+ # Locate sync, ensure it exists.
111+ if ! my_which sync > /dev/null 2>&1
112+ then
113+ log_error "sync command not found, required for --flush-caches"
114+ exit 1
115+ # Flush file system buffers.
116+ elif ! sync
117+ then
118+ # Huh, the sync() function is always successful...
119+ log_error "sync failed, check if sync is properly installed"
120+ fi
121+
122+ # Locate sysctl, ensure it exists.
123+ if ! my_which sysctl > /dev/null 2>&1
124+ then
125+ log_error "sysctl command not found, required for --flush-caches"
126+ exit 1
127+ # Purge page cache, dentries and inodes.
128+ elif ! sysctl -q -w vm.drop_caches=3
129+ then
130+ log_error "sysctl failed, check the error message for details"
131+ exit 1
132+ fi
133+elif test $flush_caches -eq 1
134+then
135+ log_error "--flush-caches is not supported on this platform"
136+ exit 1
137+fi
138+
139+#
140 # Uncomment the following lines if you want all tables to be automatically
141 # checked and repaired during startup. You should add sensible key_buffer
142 # and sort_buffer values to my.cnf to improve check performance or require
143@@ -730,6 +773,31 @@
144
145 cmd="`mysqld_ld_preload_text`$NOHUP_NICENESS"
146
147+#
148+# Set mysqld's memory interleave policy.
149+#
150+
151+if @TARGET_LINUX@ && test $numa_interleave -eq 1
152+then
153+ # Locate numactl, ensure it exists.
154+ if ! my_which numactl > /dev/null 2>&1
155+ then
156+ log_error "numactl command not found, required for --numa-interleave"
157+ exit 1
158+ # Attempt to run a command, ensure it works.
159+ elif ! numactl --interleave=all true
160+ then
161+ log_error "numactl failed, check if numactl is properly installed"
162+ fi
163+
164+ # Launch mysqld with numactl.
165+ cmd="$cmd numactl --interleave=all"
166+elif test $numa_interleave -eq 1
167+then
168+ log_error "--numa-interleave is not supported on this platform"
169+ exit 1
170+fi
171+
172 for i in "$ledir/$MYSQLD" "$defaults" "--basedir=$MY_BASEDIR_VERSION" \
173 "--datadir=$DATADIR" "--plugin-dir=$plugin_dir" "$USER_OPTION"
174 do
175
176=== modified file 'Percona-Server/storage/innobase/buf/buf0buf.c'
177--- Percona-Server/storage/innobase/buf/buf0buf.c 2012-10-12 06:24:13 +0000
178+++ Percona-Server/storage/innobase/buf/buf0buf.c 2012-10-18 18:03:21 +0000
179@@ -1028,7 +1028,8 @@
180 /*===========*/
181 buf_pool_t* buf_pool, /*!< in: buffer pool instance */
182 buf_chunk_t* chunk, /*!< out: chunk of buffers */
183- ulint mem_size) /*!< in: requested size in bytes */
184+ ulint mem_size, /*!< in: requested size in bytes */
185+ ibool populate) /*!< in: virtual page preallocation */
186 {
187 buf_block_t* block;
188 byte* frame;
189@@ -1044,7 +1045,7 @@
190 + (UNIV_PAGE_SIZE - 1), UNIV_PAGE_SIZE);
191
192 chunk->mem_size = mem_size;
193- chunk->mem = os_mem_alloc_large(&chunk->mem_size);
194+ chunk->mem = os_mem_alloc_large(&chunk->mem_size, populate);
195
196 if (UNIV_UNLIKELY(chunk->mem == NULL)) {
197
198@@ -1254,6 +1255,7 @@
199 /*===================*/
200 buf_pool_t* buf_pool, /*!< in: buffer pool instance */
201 ulint buf_pool_size, /*!< in: size in bytes */
202+ ibool populate, /*!< in: virtual page preallocation */
203 ulint instance_no) /*!< in: id of the instance */
204 {
205 ulint i;
206@@ -1286,7 +1288,7 @@
207
208 UT_LIST_INIT(buf_pool->free);
209
210- if (!buf_chunk_init(buf_pool, chunk, buf_pool_size)) {
211+ if (!buf_chunk_init(buf_pool, chunk, buf_pool_size, populate)) {
212 mem_free(chunk);
213 mem_free(buf_pool);
214
215@@ -1381,6 +1383,7 @@
216 buf_pool_init(
217 /*==========*/
218 ulint total_size, /*!< in: size of the total pool in bytes */
219+ ibool populate, /*!< in: virtual page preallocation */
220 ulint n_instances) /*!< in: number of instances */
221 {
222 ulint i;
223@@ -1398,7 +1401,7 @@
224 for (i = 0; i < n_instances; i++) {
225 buf_pool_t* ptr = &buf_pool_ptr[i];
226
227- if (buf_pool_init_instance(ptr, size, i) != DB_SUCCESS) {
228+ if (buf_pool_init_instance(ptr, size, populate, i) != DB_SUCCESS) {
229
230 /* Free all the instances created so far. */
231 buf_pool_free(i);
232
233=== modified file 'Percona-Server/storage/innobase/handler/ha_innodb.cc'
234--- Percona-Server/storage/innobase/handler/ha_innodb.cc 2012-10-01 00:49:13 +0000
235+++ Percona-Server/storage/innobase/handler/ha_innodb.cc 2012-10-18 18:03:21 +0000
236@@ -12494,6 +12494,12 @@
237 "The size of the memory buffer InnoDB uses to cache data and indexes of its tables.",
238 NULL, NULL, 128*1024*1024L, 32*1024*1024L, LONGLONG_MAX, 1024*1024L);
239
240+static MYSQL_SYSVAR_BOOL(buffer_pool_populate, srv_buf_pool_populate,
241+ PLUGIN_VAR_NOCMDARG | PLUGIN_VAR_READONLY,
242+ "Preallocate (pre-fault) the page frames required for the mapping "
243+ "established by the buffer pool memory region. Disabled by default.",
244+ NULL, NULL, FALSE);
245+
246 static MYSQL_SYSVAR_LONG(buffer_pool_instances, innobase_buffer_pool_instances,
247 PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY,
248 "Number of buffer pool instances, set to higher value on high-end machines to increase scalability",
249@@ -12874,6 +12880,7 @@
250 MYSQL_SYSVAR(additional_mem_pool_size),
251 MYSQL_SYSVAR(autoextend_increment),
252 MYSQL_SYSVAR(buffer_pool_size),
253+ MYSQL_SYSVAR(buffer_pool_populate),
254 MYSQL_SYSVAR(buffer_pool_instances),
255 MYSQL_SYSVAR(buffer_pool_shm_key),
256 MYSQL_SYSVAR(buffer_pool_shm_checksum),
257
258=== modified file 'Percona-Server/storage/innobase/include/buf0buf.h'
259--- Percona-Server/storage/innobase/include/buf0buf.h 2012-08-07 06:10:00 +0000
260+++ Percona-Server/storage/innobase/include/buf0buf.h 2012-10-18 18:03:21 +0000
261@@ -233,6 +233,7 @@
262 buf_pool_init(
263 /*=========*/
264 ulint size, /*!< in: Size of the total pool in bytes */
265+ ibool populate, /*!< in: Force virtual page preallocation */
266 ulint n_instances); /*!< in: Number of instances */
267 /********************************************************************//**
268 Frees the buffer pool at shutdown. This must not be invoked before
269
270=== modified file 'Percona-Server/storage/innobase/include/os0proc.h'
271--- Percona-Server/storage/innobase/include/os0proc.h 2010-06-22 15:58:28 +0000
272+++ Percona-Server/storage/innobase/include/os0proc.h 2012-10-18 18:03:21 +0000
273@@ -58,7 +58,8 @@
274 void*
275 os_mem_alloc_large(
276 /*===============*/
277- ulint* n); /*!< in/out: number of bytes */
278+ ulint* n, /*!< in/out: number of bytes */
279+ ibool populate); /*!< in: virtual page preallocation */
280 /****************************************************************//**
281 Frees large pages memory. */
282 UNIV_INTERN
283
284=== modified file 'Percona-Server/storage/innobase/include/srv0srv.h'
285--- Percona-Server/storage/innobase/include/srv0srv.h 2012-09-28 07:53:51 +0000
286+++ Percona-Server/storage/innobase/include/srv0srv.h 2012-10-18 18:03:21 +0000
287@@ -179,6 +179,7 @@
288 extern ibool srv_use_sys_malloc;
289 #endif /* UNIV_HOTBACKUP */
290 extern ulint srv_buf_pool_size; /*!< requested size in bytes */
291+extern my_bool srv_buf_pool_populate; /*!< virtual page preallocation */
292 extern ulint srv_buf_pool_instances; /*!< requested number of buffer pool instances */
293 extern ulint srv_buf_pool_old_size; /*!< previously requested size */
294 extern ulint srv_buf_pool_curr_size; /*!< current size in bytes */
295
296=== modified file 'Percona-Server/storage/innobase/os/os0proc.c'
297--- Percona-Server/storage/innobase/os/os0proc.c 2011-12-28 10:31:18 +0000
298+++ Percona-Server/storage/innobase/os/os0proc.c 2012-10-18 18:03:21 +0000
299@@ -32,6 +32,12 @@
300 #include "ut0mem.h"
301 #include "ut0byte.h"
302
303+/* Linux release version */
304+#if defined(UNIV_LINUX) && defined(_GNU_SOURCE)
305+#include <string.h> /* strverscmp() */
306+#include <sys/utsname.h> /* uname() */
307+#endif
308+
309 /* FreeBSD for example has only MAP_ANON, Linux has MAP_ANONYMOUS and
310 MAP_ANON but MAP_ANON is marked as deprecated */
311 #if defined(MAP_ANONYMOUS)
312@@ -40,6 +46,13 @@
313 #define OS_MAP_ANON MAP_ANON
314 #endif
315
316+/* Linux's MAP_POPULATE */
317+#if defined(MAP_POPULATE)
318+#define OS_MAP_POPULATE MAP_POPULATE
319+#else
320+#define OS_MAP_POPULATE 0
321+#endif
322+
323 UNIV_INTERN ibool os_use_large_pages;
324 /* Large page size. This may be a boot-time option on some platforms */
325 UNIV_INTERN ulint os_large_page_size;
326@@ -63,13 +76,32 @@
327 }
328
329 /****************************************************************//**
330+Retrieve and compare operating system release.
331+@return TRUE if the OS release is equal to, or later than release. */
332+UNIV_INTERN
333+ibool
334+os_compare_release(
335+/*===============*/
336+ const char* release /*!< in: OS release */
337+ __attribute__((unused)))
338+{
339+#if defined(UNIV_LINUX) && defined(_GNU_SOURCE)
340+ struct utsname name;
341+ return uname(&name) == 0 && strverscmp(name.release, release) >= 0;
342+#else
343+ return 0;
344+#endif
345+}
346+
347+/****************************************************************//**
348 Allocates large pages memory.
349 @return allocated memory */
350 UNIV_INTERN
351 void*
352 os_mem_alloc_large(
353 /*===============*/
354- ulint* n) /*!< in/out: number of bytes */
355+ ulint* n, /*!< in/out: number of bytes */
356+ ibool populate) /*!< in: virtual page preallocation */
357 {
358 void* ptr;
359 ulint size;
360@@ -155,12 +187,13 @@
361 ut_ad(ut_is_2pow(size));
362 size = *n = ut_2pow_round(*n + (size - 1), size);
363 ptr = mmap(NULL, size, PROT_READ | PROT_WRITE,
364- MAP_PRIVATE | OS_MAP_ANON, -1, 0);
365+ MAP_PRIVATE | OS_MAP_ANON |
366+ (populate ? OS_MAP_POPULATE : 0), -1, 0);
367 if (UNIV_UNLIKELY(ptr == (void*) -1)) {
368 fprintf(stderr, "InnoDB: mmap(%lu bytes) failed;"
369 " errno %lu\n",
370 (ulong) size, (ulong) errno);
371- ptr = NULL;
372+ return(NULL);
373 } else {
374 os_fast_mutex_lock(&ut_list_mutex);
375 ut_total_allocated_memory += size;
376@@ -168,6 +201,25 @@
377 UNIV_MEM_ALLOC(ptr, size);
378 }
379 #endif
380+
381+#if OS_MAP_ANON && OS_MAP_POPULATE
382+ /* MAP_POPULATE is only supported for private mappings
383+ since Linux 2.6.23. */
384+ populate = populate && !os_compare_release("2.6.23");
385+
386+ if (populate) {
387+ fprintf(stderr, "InnoDB: Warning: mmap(MAP_POPULATE) "
388+ "is not supported for private mappings. "
389+ "Forcing preallocation by faulting in pages.\n");
390+ }
391+#endif
392+
393+ /* Initialize the entire buffer to force the allocation
394+ of physical memory page frames. */
395+ if (populate) {
396+ memset(ptr, '\0', size);
397+ }
398+
399 return(ptr);
400 }
401
402
403=== modified file 'Percona-Server/storage/innobase/row/row0merge.c'
404--- Percona-Server/storage/innobase/row/row0merge.c 2012-07-13 11:34:09 +0000
405+++ Percona-Server/storage/innobase/row/row0merge.c 2012-10-18 18:03:21 +0000
406@@ -2720,7 +2720,7 @@
407
408 merge_files = mem_alloc(n_indexes * sizeof *merge_files);
409 block_size = 3 * merge_sort_block_size;
410- block_mem = os_mem_alloc_large(&block_size);
411+ block_mem = os_mem_alloc_large(&block_size, FALSE);
412
413 for (i = 0; i < UT_ARR_SIZE(block); i++) {
414 block[i] = (row_merge_block_t ) ((byte *) block_mem +
415
416=== modified file 'Percona-Server/storage/innobase/srv/srv0srv.c'
417--- Percona-Server/storage/innobase/srv/srv0srv.c 2012-09-28 07:53:51 +0000
418+++ Percona-Server/storage/innobase/srv/srv0srv.c 2012-10-18 18:03:21 +0000
419@@ -232,6 +232,8 @@
420 UNIV_INTERN my_bool srv_use_sys_malloc = TRUE;
421 /* requested size in kilobytes */
422 UNIV_INTERN ulint srv_buf_pool_size = ULINT_MAX;
423+/* force virtual page preallocation (prefault) */
424+UNIV_INTERN my_bool srv_buf_pool_populate = FALSE;
425 /* requested number of buffer pool instances */
426 UNIV_INTERN ulint srv_buf_pool_instances = 1;
427 /* previously requested size */
428
429=== modified file 'Percona-Server/storage/innobase/srv/srv0start.c'
430--- Percona-Server/storage/innobase/srv/srv0start.c 2012-09-28 07:53:51 +0000
431+++ Percona-Server/storage/innobase/srv/srv0start.c 2012-10-18 18:03:21 +0000
432@@ -1540,7 +1540,8 @@
433 ((double) srv_buf_pool_size) / (1024 * 1024));
434 }
435
436- err = buf_pool_init(srv_buf_pool_size, srv_buf_pool_instances);
437+ err = buf_pool_init(srv_buf_pool_size, (ibool) srv_buf_pool_populate,
438+ srv_buf_pool_instances);
439
440 ut_print_timestamp(stderr);
441 fprintf(stderr,

Subscribers

People subscribed via source and target branches