Percona Server moved to https://jira.percona.com/projects/PS

Merge lp:~percona-dev/percona-server/release-5.1.50-12-adjust_buffer_pool_shm into lp:percona-server/release-5.1.50-12

  1. release-5.1.50-12-adjust_buffer_pool_shm
  2. Merge into release-5.1.50-12
Proposed by Yasufumi Kinoshita
Status: Merged
Approved by: Vadim Tkachenko
Approved revision: no longer in the source branch.
Merged at revision: 123
Proposed branch: lp:~percona-dev/percona-server/release-5.1.50-12-adjust_buffer_pool_shm
Merge into: lp:percona-server/release-5.1.50-12
Diff against target: 291 lines (+77/-39)
1 file modified
innodb_buffer_pool_shm.patch (+77/-39)
To merge this branch: bzr merge lp:~percona-dev/percona-server/release-5.1.50-12-adjust_buffer_pool_shm
Reviewer Review Type Date Requested Status
Vadim Tkachenko Approve
Fred Linhoss (community) documentation Approve
Review via email: mp+36971@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Vadim Tkachenko (vadim-tk) wrote :

For Fred: document bug and release notes

Revision history for this message
Fred Linhoss (fred-linhoss) wrote :

  Vadim,

I assume innodb_buffer_pool_shm is the page where Yasufumi's new
variable should be documented.

    https://intranet.percona.com/dokuwiki/per:fred:innodb_buffer_pool_shm

I have only changed the System Variables section to add the new variable.

Questions:

    1. Is the variable name correct, or does "innodb_" need to be added
to the front?
    2. I believe that "PLUGIN_VAR_NOCMDARG | PLUGIN_VAR_READONLY" means
that, in the table, "Command Line"=No and "Dynamic Variable"=No. But do
I have "Config File" and "Variable Scope" correct?
    3. Is the sentence describing what the variable does correct? (When
is the checksum validation done? Should I say anything about that?)

Thanks,
Fred

On 9/29/2010 1:01 PM, Vadim Tkachenko wrote:
> For Fred: document bug and release notes

Revision history for this message
Vadim Tkachenko (vadim-tk) wrote :

Fred,

it is innodb_buffer_pool_shm_checksum.

Usually if variable can be set in config file - it is also accepted in
command line.

You should say:
"Checksum validation is performed during start or shutdown. Checksum
check makes process slower but
adds additional protection that memory region is not corrupted".

On Thu, Sep 30, 2010 at 10:18 AM, Fred Linhoss <email address hidden> wrote:
>  Vadim,
>
> I assume innodb_buffer_pool_shm is the page where Yasufumi's new
> variable should be documented.
>
>    https://intranet.percona.com/dokuwiki/per:fred:innodb_buffer_pool_shm
>
> I have only changed the System Variables section to add the new variable.
>
> Questions:
>
>    1. Is the variable name correct, or does "innodb_" need to be added
> to the front?
>    2. I believe that "PLUGIN_VAR_NOCMDARG | PLUGIN_VAR_READONLY" means
> that, in the table, "Command Line"=No and "Dynamic Variable"=No. But do
> I have "Config File" and "Variable Scope" correct?
>    3. Is the sentence describing what the variable does correct? (When
> is the checksum validation done? Should I say anything about that?)
>
> Thanks,
> Fred
>
> On 9/29/2010 1:01 PM, Vadim Tkachenko wrote:
>> For Fred: document bug and release notes
> --
> https://code.launchpad.net/~percona-dev/percona-server/release-5.1.50-12-adjust_buffer_pool_shm/+merge/36971
> Your team Percona developers is requested to review the proposed merge of lp:~percona-dev/percona-server/release-5.1.50-12-adjust_buffer_pool_shm into lp:percona-server.
>

--
Vadim Tkachenko, CTO, Percona Inc.
Phone +1-888-401-3403,  Skype: vadimtk153
Schedule meeting: http://tungle.me/VadimTkachenko

Revision history for this message
Fred Linhoss (fred-linhoss) wrote :

On 9/30/2010 1:24 PM, Vadim Tkachenko wrote:
> Fred,
>
> it is innodb_buffer_pool_shm_checksum.
>
> Usually if variable can be set in config file - it is also accepted in
> command line.
 *But, "PLUGIN_VAR_NOCMDARG | PLUGIN_VAR_READONLY" means it _cannot_**
take it from the command line, correct?
If that's true, it makes me wonder how you can set it. Or it that wrong
and it does take it from the command line?
*
> You should say:
> "Checksum validation is performed during start or shutdown. Checksum check makes process slower but adds additional protection that memory region is not corrupted".
 *Done. Changes are now in this version:*

http://www.percona.com/docs/wiki/percona-server:features:innodb_buffer_pool_shm
>
> On Thu, Sep 30, 2010 at 10:18 AM, Fred Linhoss<email address hidden> wrote:
>> Vadim,
>>
>> I assume innodb_buffer_pool_shm is the page where Yasufumi's new
>> variable should be documented.
>>
>> https://intranet.percona.com/dokuwiki/per:fred:innodb_buffer_pool_shm
>>
>> I have only changed the System Variables section to add the new variable.
>>
>> Questions:
>>
>> 1. Is the variable name correct, or does "innodb_" need to be added
>> to the front?
>> 2. I believe that "PLUGIN_VAR_NOCMDARG | PLUGIN_VAR_READONLY" means
>> that, in the table, "Command Line"=No and "Dynamic Variable"=No. But do
>> I have "Config File" and "Variable Scope" correct?
>> 3. Is the sentence describing what the variable does correct? (When
>> is the checksum validation done? Should I say anything about that?)
>>
>> Thanks,
>> Fred
>>
>> On 9/29/2010 1:01 PM, Vadim Tkachenko wrote:
>>> For Fred: document bug and release notes
>> --
>> https://code.launchpad.net/~percona-dev/percona-server/release-5.1.50-12-adjust_buffer_pool_shm/+merge/36971
>> Your team Percona developers is requested to review the proposed merge of lp:~percona-dev/percona-server/release-5.1.50-12-adjust_buffer_pool_shm into lp:percona-server.
>>
>
>

Revision history for this message
Fred Linhoss (fred-linhoss) wrote :

Added new system variable innodb_buffer_pool_shm_checksum to [[percona_server::innodb_buffer_pool_shm]].

review: Approve (documentation)
Revision history for this message
Vadim Tkachenko (vadim-tk) :
review: Approve
Revision history for this message
Baron Schwartz (baron-xaprb) wrote :

Fred,

>> Usually if variable can be set in config file - it is also accepted in
>> command line.
> But, "PLUGIN_VAR_NOCMDARG | PLUGIN_VAR_READONLY" means it _cannot_**
> take it from the command line, correct?
> If that's true, it makes me wonder how you can set it. Or it that wrong
> and it does take it from the command line?

I haven't checked, but it looks like you are right. Do you have a
build that you can test to see?

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'innodb_buffer_pool_shm.patch'
2--- innodb_buffer_pool_shm.patch 2010-09-27 09:59:34 +0000
3+++ innodb_buffer_pool_shm.patch 2010-09-29 11:18:45 +0000
4@@ -96,7 +96,7 @@
5 /********************************************************************//**
6 Allocates a chunk of buffer frames.
7 @return chunk, or NULL on failure */
8-@@ -768,26 +835,174 @@
9+@@ -768,26 +835,184 @@
10 {
11 buf_block_t* block;
12 byte* frame;
13@@ -161,7 +161,7 @@
14 + if (UNIV_UNLIKELY(chunk->mem == NULL)) {
15 + return(NULL);
16 + }
17-+
18++init_again:
19 +#ifdef UNIV_SET_MEM_TO_ZERO
20 + if (is_new) {
21 + memset(chunk->mem, '\0', chunk->mem_size);
22@@ -204,16 +204,6 @@
23 + "InnoDB: Error: The shared memory was not initialized yet.\n");
24 + return(NULL);
25 + }
26-+ if (!shm_info->clean) {
27-+ fprintf(stderr,
28-+ "InnoDB: Error: The shared memory was not shut down cleanly.\n");
29-+ return(NULL);
30-+ }
31-+ if (!shm_info->reusable) {
32-+ fprintf(stderr,
33-+ "InnoDB: Error: The shared memory has unrecoverable contents.\n");
34-+ return(NULL);
35-+ }
36 + if (shm_info->buf_pool_size != srv_buf_pool_size) {
37 + fprintf(stderr,
38 + "InnoDB: Error: srv_buf_pool_size is different (shm=%lu current=%lu).\n",
39@@ -226,14 +216,34 @@
40 + shm_info->page_size, srv_page_size);
41 + return(NULL);
42 + }
43++ if (!shm_info->reusable) {
44++ fprintf(stderr,
45++ "InnoDB: Warning: The shared memory has unrecoverable contents.\n"
46++ "InnoDB: The shared memory segment is initialized.\n");
47++ is_new = TRUE;
48++ goto init_again;
49++ }
50++ if (!shm_info->clean) {
51++ fprintf(stderr,
52++ "InnoDB: Warning: The shared memory was not shut down cleanly.\n"
53++ "InnoDB: The shared memory segment is initialized.\n");
54++ is_new = TRUE;
55++ goto init_again;
56++ }
57 +
58 + ut_a(shm_info->zip_hash_offset == chunk->mem_size - zip_hash_mem_size);
59 + ut_a(shm_info->zip_hash_n == zip_hash_n);
60 +
61 + /* check checksum */
62-+ checksum = ut_fold_binary_32((byte*)chunk->mem + sizeof(buf_shm_info_t),
63-+ chunk->mem_size - sizeof(buf_shm_info_t));
64-+ if (shm_info->checksum != checksum) {
65++ if (srv_buffer_pool_shm_checksum) {
66++ checksum = ut_fold_binary_32((byte*)chunk->mem + sizeof(buf_shm_info_t),
67++ chunk->mem_size - sizeof(buf_shm_info_t));
68++ } else {
69++ checksum = BUF_NO_CHECKSUM_MAGIC;
70++ }
71++
72++ if (shm_info->checksum != BUF_NO_CHECKSUM_MAGIC
73++ && shm_info->checksum != checksum) {
74 + fprintf(stderr,
75 + "InnoDB: Error: checksum of the shared memory is not match. "
76 + "(stored=%lu calculated=%lu)\n",
77@@ -271,7 +281,7 @@
78
79 /* Align a pointer to the first frame. Note that when
80 os_large_page_size is smaller than UNIV_PAGE_SIZE,
81-@@ -795,8 +1010,13 @@
82+@@ -795,8 +1020,13 @@
83 it is bigger, we may allocate more blocks than requested. */
84
85 frame = ut_align(chunk->mem, UNIV_PAGE_SIZE);
86@@ -285,7 +295,7 @@
87
88 /* Subtract the space needed for block descriptors. */
89 {
90-@@ -810,6 +1030,98 @@
91+@@ -810,6 +1040,98 @@
92 chunk->size = size;
93 }
94
95@@ -305,7 +315,7 @@
96 + }
97 +
98 + chunk->size = shm_info->chunk_backup.size;
99-+ phys_offset = frame - (byte*)chunk->mem + shm_info->frame_offset;
100++ phys_offset = frame - ((byte*)chunk->mem + shm_info->frame_offset);
101 + logi_offset = frame - chunk->blocks[0].frame;
102 + previous_frame_address = chunk->blocks[0].frame;
103 + blocks_offset = (byte*)chunk->blocks - (byte*)shm_info->chunk_backup.blocks;
104@@ -339,17 +349,17 @@
105 + " Done.\n");
106 + }
107 +
108++ /* buf_block_t */
109++ block = chunk->blocks;
110++ for (i = chunk->size; i--; ) {
111++ buf_block_reuse(block, logi_offset);
112++ block++;
113++ }
114++
115 + if (logi_offset || blocks_offset) {
116 + fprintf(stderr,
117 + "InnoDB: Aligning logical offset...");
118 +
119-+ /* buf_block_t */
120-+ block = chunk->blocks;
121-+
122-+ for (i = chunk->size; i--; ) {
123-+ buf_block_reuse(block, logi_offset);
124-+ block++;
125-+ }
126 +
127 + /* buf_pool_t buf_pool_backup */
128 + UT_LIST_OFFSET(flush_list, buf_page_t, shm_info->buf_pool_backup.flush_list,
129@@ -384,7 +394,7 @@
130 /* Init block structs and assign frames for them. Then we
131 assign the frames to the first blocks (we already mapped the
132 memory above). */
133-@@ -833,6 +1145,11 @@
134+@@ -833,6 +1155,11 @@
135 block++;
136 frame += UNIV_PAGE_SIZE;
137 }
138@@ -396,7 +406,7 @@
139
140 return(chunk);
141 }
142-@@ -1014,6 +1331,8 @@
143+@@ -1014,6 +1341,8 @@
144 UNIV_MEM_UNDESC(block);
145 }
146
147@@ -405,7 +415,7 @@
148 os_mem_free_large(chunk->mem, chunk->mem_size);
149 }
150
151-@@ -1063,7 +1382,10 @@
152+@@ -1063,7 +1392,10 @@
153 srv_buf_pool_curr_size = buf_pool->curr_size * UNIV_PAGE_SIZE;
154
155 buf_pool->page_hash = hash_create(2 * buf_pool->curr_size);
156@@ -416,7 +426,7 @@
157
158 buf_pool->last_printout_time = time(NULL);
159
160-@@ -1078,6 +1400,86 @@
161+@@ -1078,6 +1410,86 @@
162 --------------------------- */
163 /* All fields are initialized by mem_zalloc(). */
164
165@@ -503,7 +513,7 @@
166 mutex_exit(&LRU_list_mutex);
167 rw_lock_x_unlock(&page_hash_latch);
168 buf_pool_mutex_exit();
169-@@ -1102,6 +1504,30 @@
170+@@ -1102,6 +1514,34 @@
171 buf_chunk_t* chunk;
172 buf_chunk_t* chunks;
173
174@@ -524,8 +534,12 @@
175 + memcpy(&(shm_info->chunk_backup), chunk, sizeof(buf_chunk_t));
176 +
177 + if (srv_fast_shutdown < 2) {
178-+ shm_info->checksum = ut_fold_binary_32((byte*)chunk->mem + sizeof(buf_shm_info_t),
179-+ chunk->mem_size - sizeof(buf_shm_info_t));
180++ if (srv_buffer_pool_shm_checksum) {
181++ shm_info->checksum = ut_fold_binary_32((byte*)chunk->mem + sizeof(buf_shm_info_t),
182++ chunk->mem_size - sizeof(buf_shm_info_t));
183++ } else {
184++ shm_info->checksum = BUF_NO_CHECKSUM_MAGIC;
185++ }
186 + shm_info->clean = TRUE;
187 + }
188 +
189@@ -534,7 +548,7 @@
190 chunks = buf_pool->chunks;
191 chunk = chunks + buf_pool->n_chunks;
192
193-@@ -1110,10 +1536,13 @@
194+@@ -1110,10 +1550,13 @@
195 would fail at shutdown. */
196 os_mem_free_large(chunk->mem, chunk->mem_size);
197 }
198@@ -548,7 +562,7 @@
199 mem_free(buf_pool);
200 buf_pool = NULL;
201 }
202-@@ -1308,6 +1737,11 @@
203+@@ -1308,6 +1751,11 @@
204 //buf_pool_mutex_enter();
205 mutex_enter(&LRU_list_mutex);
206
207@@ -560,7 +574,7 @@
208 shrink_again:
209 if (buf_pool->n_chunks <= 1) {
210
211-@@ -1551,6 +1985,11 @@
212+@@ -1551,6 +1999,11 @@
213 buf_pool_resize(void)
214 /*=================*/
215 {
216@@ -649,7 +663,23 @@
217 diff -ruN a/storage/innodb_plugin/handler/ha_innodb.cc b/storage/innodb_plugin/handler/ha_innodb.cc
218 --- a/storage/innodb_plugin/handler/ha_innodb.cc 2010-07-14 16:34:18.597725479 +0900
219 +++ b/storage/innodb_plugin/handler/ha_innodb.cc 2010-07-14 16:40:16.159323612 +0900
220-@@ -11340,6 +11340,11 @@
221+@@ -197,6 +197,7 @@
222+ static my_bool innobase_create_status_file = FALSE;
223+ static my_bool innobase_stats_on_metadata = TRUE;
224+ static my_bool innobase_use_sys_stats_table = FALSE;
225++static my_bool innobase_buffer_pool_shm_checksum = TRUE;
226+
227+ static char* internal_innobase_data_file_path = NULL;
228+
229+@@ -2417,6 +2418,7 @@
230+ srv_use_doublewrite_buf = (ibool) innobase_use_doublewrite;
231+ srv_use_checksums = (ibool) innobase_use_checksums;
232+ srv_fast_checksum = (ibool) innobase_fast_checksum;
233++ srv_buffer_pool_shm_checksum = (ibool) innobase_buffer_pool_shm_checksum;
234+
235+ #ifdef HAVE_LARGE_PAGES
236+ if ((os_use_large_pages = (ibool) my_use_large_pages))
237+@@ -11340,6 +11342,16 @@
238 "The size of the memory buffer InnoDB uses to cache data and indexes of its tables.",
239 NULL, NULL, 128*1024*1024L, 32*1024*1024L, LONGLONG_MAX, 1024*1024L);
240
241@@ -658,14 +688,20 @@
242 + "[experimental] The key value of shared memory segment for the buffer pool. 0 (default) disables the feature.",
243 + NULL, NULL, 0, 0, INT_MAX32, 0);
244 +
245++static MYSQL_SYSVAR_BOOL(buffer_pool_shm_checksum, innobase_buffer_pool_shm_checksum,
246++ PLUGIN_VAR_NOCMDARG | PLUGIN_VAR_READONLY,
247++ "Enable buffer_pool_shm checksum validation (enabled by default).",
248++ NULL, NULL, TRUE);
249++
250 static MYSQL_SYSVAR_ULONG(commit_concurrency, innobase_commit_concurrency,
251 PLUGIN_VAR_RQCMDARG,
252 "Helps in performance tuning in heavily concurrent environments.",
253-@@ -11599,6 +11604,7 @@
254+@@ -11599,6 +11611,8 @@
255 MYSQL_SYSVAR(additional_mem_pool_size),
256 MYSQL_SYSVAR(autoextend_increment),
257 MYSQL_SYSVAR(buffer_pool_size),
258 + MYSQL_SYSVAR(buffer_pool_shm_key),
259++ MYSQL_SYSVAR(buffer_pool_shm_checksum),
260 MYSQL_SYSVAR(checksums),
261 MYSQL_SYSVAR(fast_checksum),
262 MYSQL_SYSVAR(commit_concurrency),
263@@ -816,11 +852,12 @@
264 diff -ruN a/storage/innodb_plugin/include/srv0srv.h b/storage/innodb_plugin/include/srv0srv.h
265 --- a/storage/innodb_plugin/include/srv0srv.h 2010-07-14 16:32:49.695323045 +0900
266 +++ b/storage/innodb_plugin/include/srv0srv.h 2010-07-14 16:40:16.171325784 +0900
267-@@ -156,6 +156,8 @@
268+@@ -156,6 +156,9 @@
269 extern ulint srv_mem_pool_size;
270 extern ulint srv_lock_table_size;
271
272 +extern uint srv_buffer_pool_shm_key;
273++extern ibool srv_buffer_pool_shm_checksum;
274 +
275 extern ibool srv_thread_concurrency_timer_based;
276
277@@ -1057,12 +1094,13 @@
278 diff -ruN a/storage/innodb_plugin/srv/srv0srv.c b/storage/innodb_plugin/srv/srv0srv.c
279 --- a/storage/innodb_plugin/srv/srv0srv.c 2010-07-14 16:33:23.848391648 +0900
280 +++ b/storage/innodb_plugin/srv/srv0srv.c 2010-07-14 16:40:16.177323553 +0900
281-@@ -211,6 +211,9 @@
282+@@ -211,6 +211,10 @@
283 UNIV_INTERN ulint srv_mem_pool_size = ULINT_MAX;
284 UNIV_INTERN ulint srv_lock_table_size = ULINT_MAX;
285
286 +/* key value for shm */
287 +UNIV_INTERN uint srv_buffer_pool_shm_key = 0;
288++UNIV_INTERN ibool srv_buffer_pool_shm_checksum = TRUE;
289 +
290 /* This parameter is deprecated. Use srv_n_io_[read|write]_threads
291 instead. */

Subscribers

People subscribed via source and target branches

to all changes: