Percona Server moved to https://jira.percona.com/projects/PS

Merge lp:~percona-dev/percona-server/5.1.54-fix_expand_import_and_dict_size_limit into lp:percona-server/release-5.1.54-12

  1. 5.1.54-fix_expand_import_and_dict_size_limit
  2. Merge into 5.1.54
Proposed by Yasufumi Kinoshita
Status: Merged
Approved by: Valentine Gostev
Approved revision: no longer in the source branch.
Merged at revision: 192
Proposed branch: lp:~percona-dev/percona-server/5.1.54-fix_expand_import_and_dict_size_limit
Merge into: lp:percona-server/release-5.1.54-12
Diff against target: 307 lines (+133/-104)
3 files modified
innodb_dict_size_limit.patch (+127/-80)
innodb_expand_import.patch (+2/-2)
innodb_split_buf_pool_mutex.patch (+4/-22)
To merge this branch: bzr merge lp:~percona-dev/percona-server/5.1.54-fix_expand_import_and_dict_size_limit
Reviewer Review Type Date Requested Status
Valentine Gostev (community) qa Approve
Fred Linhoss (community) documentation Approve
Percona developers Pending
Review via email: mp+48102@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Vadim Tkachenko (vadim-tk) wrote :

Looks good for me, asking documentation approval

Revision history for this message
Vadim Tkachenko (vadim-tk) wrote :

Also we need test case to cover import with the same space_id.
Asking Valentine.

Revision history for this message
Fred Linhoss (fred-linhoss) wrote :

Will include in next release notes for 5.1 and 5.5 and update user doc pages for innodb_expand_import and innodb_dict_size_limit.

review: Approve (documentation)
Revision history for this message
Valentine Gostev (longbow) wrote :
Download full text (3.6 KiB)

when we forget to specify --export option during prepare xb_export.sh test fails with:

2011-02-03 19:50:38: xb_export.sh: ===> xtrabackup --datadir=/root/percona-xtrabackup/test/var/mysql --prepare --export --target-dir=/tmp/xb_export_backup
xtrabackup: cd to /tmp/xb_export_backup
xtrabackup: This target seems to be not prepared yet.
xtrabackup: Temporary instance for recovery is set as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 2097152
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.3
InnoDB: Warning: innodb_file_io_threads is deprecated. Please use innodb_read_io_threads and innodb_write_io_threads instead
110203 19:50:38 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 38470
110203 19:50:38 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Doing recovery: scanned up to log sequence number 58034 (1 %)
110203 19:50:38 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
110203 19:50:39 Percona XtraDB (http://www.percona.com) 1.0.13-12.4 started; log sequence number 58034
xtrabackup: starting shutdown with innodb_fast_shutdown = 0
110203 19:50:39 InnoDB: Starting shutdown...
110203 19:50:39 InnoDB: Shutdown completed; log sequence number 58034
xtrabackup Ver 1.5 Rev 203 for 5.1.53 unknown-linux-gnu (x86_64)
xtrabackup: xtrabackup_logfile detected: size=2097152, start_lsn=(38470)
xtrabackup: export option is specified.
xtrabackup: export metadata of table 'incremental_sample/test' to file `./incremental_sample/test.exp` (1 indexes)
xtrabackup: name=GEN_CLUST_INDEX, id.low=15, page=3

[notice (again)]
  If you use binary log and don't use any hack of group commit,
  the binary log position seems to be:

2011-02-03 19:50:39: xb_export.sh: ===> /root/percona-xtrabackup/test/var/test/bin/mysql --no-defaults --socket=/tmp/xtrabackup.mysql.sock --user=root -e alter table test import tablespace incremental_sample
InnoDB: free limit of ./incremental_sample/test.ibd is larger than its real size.
InnoDB: import: extended import of incremental_sample/test is started.
InnoDB: import: 1 indexes are detected.
InnoDB: Progress in %: 16 33 50 66 83 100 done.
110203 19:50:39 InnoDB: Error: file './incremental_sample/test.ibd' seems to be corrupt.
InnoDB: anyway, all not corrupt pages were tried to be converted to salvage.
InnoDB: ##### CAUTION #####
InnoDB: ## The .ibd must cause to crash InnoDB, though re-import would seem to be succeeded.
InnoDB: ## If you don't have knowledge about salvaging data from .ibd, you should not use the f...

Read more...

review: Needs Fixing (qa)
Revision history for this message
Valentine Gostev (longbow) wrote :

Please disregard my previous comment, I will file a separate bug

Revision history for this message
Valentine Gostev (longbow) wrote :

Please disregard prev comment - will file a separate bug

review: Approve (qa)
Revision history for this message
Yasufumi Kinoshita (yasufumi-kinoshita) wrote :

Thanks, we are progressing surely...

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'innodb_dict_size_limit.patch'
--- innodb_dict_size_limit.patch 2010-12-16 11:35:26 +0000
+++ innodb_dict_size_limit.patch 2011-02-01 02:13:15 +0000
@@ -8,7 +8,7 @@
8diff -ruN a/storage/innodb_plugin/btr/btr0sea.c b/storage/innodb_plugin/btr/btr0sea.c8diff -ruN a/storage/innodb_plugin/btr/btr0sea.c b/storage/innodb_plugin/btr/btr0sea.c
9--- a/storage/innodb_plugin/btr/btr0sea.c 2010-08-04 02:24:19.000000000 +09009--- a/storage/innodb_plugin/btr/btr0sea.c 2010-08-04 02:24:19.000000000 +0900
10+++ b/storage/innodb_plugin/btr/btr0sea.c 2010-08-27 16:09:42.926020757 +090010+++ b/storage/innodb_plugin/btr/btr0sea.c 2010-08-27 16:09:42.926020757 +0900
11@@ -1173,6 +1173,126 @@11@@ -1173,6 +1173,173 @@
12 mem_free(folds);12 mem_free(folds);
13 }13 }
14 14
@@ -36,96 +36,143 @@
36+ ulint i;36+ ulint i;
37+ mem_heap_t* heap = NULL;37+ mem_heap_t* heap = NULL;
38+ ulint* offsets;38+ ulint* offsets;
39+ ibool released_search_latch;
39+40+
40+ rw_lock_x_lock(&btr_search_latch);41+ rw_lock_s_lock(&btr_search_latch);
41+ buf_pool_mutex_enter();
42+42+
43+ table = btr_search_sys->hash_index;43+ table = btr_search_sys->hash_index;
44+44+
45+ bpage = UT_LIST_GET_LAST(buf_pool->LRU);45+ do {
46+46+ buf_chunk_t* chunks = buf_pool->chunks;
47+ while (bpage != NULL) {47+ buf_chunk_t* chunk = chunks + buf_pool->n_chunks;
48+ block = (buf_block_t*) bpage;48+
49+ if (block->index == index && block->is_hashed) {49+ released_search_latch = FALSE;
50+ page = block->frame;50+
51+51+ while (--chunk >= chunks) {
52+ /* from btr_search_drop_page_hash_index() */52+ block = chunk->blocks;
53+ n_fields = block->curr_n_fields;53+ i = chunk->size;
54+ n_bytes = block->curr_n_bytes;54+
55+55+retry:
56+ ut_a(n_fields + n_bytes > 0);56+ for (; i--; block++) {
57+57+ if (buf_block_get_state(block)
58+ n_recs = page_get_n_recs(page);58+ != BUF_BLOCK_FILE_PAGE
59+59+ || block->index != index
60+ /* Calculate and cache fold values into an array for fast deletion60+ || !block->is_hashed) {
61+ from the hash index */61+ continue;
62+62+ }
63+ folds = mem_alloc(n_recs * sizeof(ulint));63+
64+64+ page = block->frame;
65+ n_cached = 0;65+
66+66+ /* from btr_search_drop_page_hash_index() */
67+ rec = page_get_infimum_rec(page);67+ n_fields = block->curr_n_fields;
68+ rec = page_rec_get_next_low(rec, page_is_comp(page));68+ n_bytes = block->curr_n_bytes;
69+69+
70+ index_id = btr_page_get_index_id(page);70+
71+ /* keeping latch order */
72+ rw_lock_s_unlock(&btr_search_latch);
73+ released_search_latch = TRUE;
74+ rw_lock_x_lock(&block->lock);
75+
76+
77+ ut_a(n_fields + n_bytes > 0);
78+
79+ n_recs = page_get_n_recs(page);
80+
81+ /* Calculate and cache fold values into an array for fast deletion
82+ from the hash index */
83+
84+ folds = mem_alloc(n_recs * sizeof(ulint));
85+
86+ n_cached = 0;
87+
88+ rec = page_get_infimum_rec(page);
89+ rec = page_rec_get_next_low(rec, page_is_comp(page));
90+
91+ index_id = btr_page_get_index_id(page);
71+ 92+
72+ ut_a(0 == ut_dulint_cmp(index_id, index->id));93+ ut_a(0 == ut_dulint_cmp(index_id, index->id));
73+94+
74+ prev_fold = 0;95+ prev_fold = 0;
75+96+
76+ offsets = NULL;97+ offsets = NULL;
77+98+
78+ while (!page_rec_is_supremum(rec)) {99+ while (!page_rec_is_supremum(rec)) {
79+ offsets = rec_get_offsets(rec, index, offsets,100+ offsets = rec_get_offsets(rec, index, offsets,
80+ n_fields + (n_bytes > 0), &heap);101+ n_fields + (n_bytes > 0), &heap);
81+ ut_a(rec_offs_n_fields(offsets) == n_fields + (n_bytes > 0));102+ ut_a(rec_offs_n_fields(offsets) == n_fields + (n_bytes > 0));
82+ fold = rec_fold(rec, offsets, n_fields, n_bytes, index_id);103+ fold = rec_fold(rec, offsets, n_fields, n_bytes, index_id);
83+104+
84+ if (fold == prev_fold && prev_fold != 0) {105+ if (fold == prev_fold && prev_fold != 0) {
85+106+
86+ goto next_rec;107+ goto next_rec;
87+ }108+ }
88+109+
89+ /* Remove all hash nodes pointing to this page from the110+ /* Remove all hash nodes pointing to this page from the
90+ hash chain */111+ hash chain */
91+112+
92+ folds[n_cached] = fold;113+ folds[n_cached] = fold;
93+ n_cached++;114+ n_cached++;
94+next_rec:115+next_rec:
95+ rec = page_rec_get_next_low(rec, page_rec_is_comp(rec));116+ rec = page_rec_get_next_low(rec, page_rec_is_comp(rec));
96+ prev_fold = fold;117+ prev_fold = fold;
97+ }118+ }
98+119+
99+ for (i = 0; i < n_cached; i++) {120+ if (UNIV_LIKELY_NULL(heap)) {
100+121+ mem_heap_empty(heap);
101+ ha_remove_all_nodes_to_page(table, folds[i], page);122+ }
102+ }123+
103+124+ rw_lock_x_lock(&btr_search_latch);
104+ ut_a(index->search_info->ref_count > 0);125+
105+ index->search_info->ref_count--;126+ if (UNIV_UNLIKELY(!block->is_hashed)) {
106+127+ goto cleanup;
107+ block->is_hashed = FALSE;128+ }
108+ block->index = NULL;129+
109+ 130+ ut_a(block->index == index);
131+
132+ if (UNIV_UNLIKELY(block->curr_n_fields != n_fields)
133+ || UNIV_UNLIKELY(block->curr_n_bytes != n_bytes)) {
134+ rw_lock_x_unlock(&btr_search_latch);
135+ rw_lock_x_unlock(&block->lock);
136+
137+ mem_free(folds);
138+
139+ rw_lock_s_lock(&btr_search_latch);
140+ goto retry;
141+ }
142+
143+ for (i = 0; i < n_cached; i++) {
144+
145+ ha_remove_all_nodes_to_page(table, folds[i], page);
146+ }
147+
148+ ut_a(index->search_info->ref_count > 0);
149+ index->search_info->ref_count--;
150+
151+ block->is_hashed = FALSE;
152+ block->index = NULL;
153+
154+cleanup:
110+#if defined UNIV_AHI_DEBUG || defined UNIV_DEBUG155+#if defined UNIV_AHI_DEBUG || defined UNIV_DEBUG
111+ if (UNIV_UNLIKELY(block->n_pointers)) {156+ if (UNIV_UNLIKELY(block->n_pointers)) {
112+ /* Corruption */157+ /* Corruption */
113+ ut_print_timestamp(stderr);158+ ut_print_timestamp(stderr);
114+ fprintf(stderr,159+ fprintf(stderr,
115+" InnoDB: Corruption of adaptive hash index. After dropping\n"160+" InnoDB: Corruption of adaptive hash index. After dropping\n"
116+"InnoDB: the hash index to a page of %s, still %lu hash nodes remain.\n",161+"InnoDB: the hash index to a page of %s, still %lu hash nodes remain.\n",
117+ index->name, (ulong) block->n_pointers);162+ index->name, (ulong) block->n_pointers);
118+ }163+ }
119+#endif /* UNIV_AHI_DEBUG || UNIV_DEBUG */164+#endif /* UNIV_AHI_DEBUG || UNIV_DEBUG */
120+165+ rw_lock_x_unlock(&btr_search_latch);
121+ mem_free(folds);166+ rw_lock_x_unlock(&block->lock);
167+
168+ mem_free(folds);
169+
170+ rw_lock_s_lock(&btr_search_latch);
171+ }
122+ }172+ }
123+173+ } while (released_search_latch);
124+ bpage = UT_LIST_GET_PREV(LRU, bpage);174+
125+ }175+ rw_lock_s_unlock(&btr_search_latch);
126+
127+ buf_pool_mutex_exit();
128+ rw_lock_x_unlock(&btr_search_latch);
129+176+
130+ if (UNIV_LIKELY_NULL(heap)) {177+ if (UNIV_LIKELY_NULL(heap)) {
131+ mem_heap_free(heap);178+ mem_heap_free(heap);
132179
=== modified file 'innodb_expand_import.patch'
--- innodb_expand_import.patch 2010-12-16 11:35:26 +0000
+++ innodb_expand_import.patch 2011-02-01 02:13:15 +0000
@@ -34,8 +34,8 @@
34 space_id = fsp_header_get_space_id(page);34 space_id = fsp_header_get_space_id(page);
35 space_flags = fsp_header_get_flags(page);35 space_flags = fsp_header_get_flags(page);
36 36
37+ if (srv_expand_import37+ if (srv_expand_import) {
38+ && (space_id != id || space_flags != (flags & ~(~0 << DICT_TF_BITS)))) {38+
39+ ibool file_is_corrupt = FALSE;39+ ibool file_is_corrupt = FALSE;
40+ byte* buf3;40+ byte* buf3;
41+ byte* descr_page;41+ byte* descr_page;
4242
=== modified file 'innodb_split_buf_pool_mutex.patch'
--- innodb_split_buf_pool_mutex.patch 2010-12-16 11:35:26 +0000
+++ innodb_split_buf_pool_mutex.patch 2011-02-01 02:13:15 +0000
@@ -48,25 +48,7 @@
48diff -ruN a/storage/innodb_plugin/btr/btr0sea.c b/storage/innodb_plugin/btr/btr0sea.c48diff -ruN a/storage/innodb_plugin/btr/btr0sea.c b/storage/innodb_plugin/btr/btr0sea.c
49--- a/storage/innodb_plugin/btr/btr0sea.c 2010-08-27 16:11:12.151975789 +090049--- a/storage/innodb_plugin/btr/btr0sea.c 2010-08-27 16:11:12.151975789 +0900
50+++ b/storage/innodb_plugin/btr/btr0sea.c 2010-08-27 16:11:40.593021205 +090050+++ b/storage/innodb_plugin/btr/btr0sea.c 2010-08-27 16:11:40.593021205 +0900
51@@ -1199,7 +1199,7 @@51@@ -1925,7 +1925,8 @@
52 ulint* offsets;
53
54 rw_lock_x_lock(&btr_search_latch);
55- buf_pool_mutex_enter();
56+ mutex_enter(&LRU_list_mutex);
57
58 table = btr_search_sys->hash_index;
59
60@@ -1285,7 +1285,7 @@
61 bpage = UT_LIST_GET_PREV(LRU, bpage);
62 }
63
64- buf_pool_mutex_exit();
65+ mutex_exit(&LRU_list_mutex);
66 rw_lock_x_unlock(&btr_search_latch);
67
68 if (UNIV_LIKELY_NULL(heap)) {
69@@ -1878,7 +1878,8 @@
70 rec_offs_init(offsets_);52 rec_offs_init(offsets_);
71 53
72 rw_lock_x_lock(&btr_search_latch);54 rw_lock_x_lock(&btr_search_latch);
@@ -76,7 +58,7 @@
76 58
77 cell_count = hash_get_n_cells(btr_search_sys->hash_index);59 cell_count = hash_get_n_cells(btr_search_sys->hash_index);
78 60
79@@ -1886,11 +1887,13 @@61@@ -1933,11 +1934,13 @@
80 /* We release btr_search_latch every once in a while to62 /* We release btr_search_latch every once in a while to
81 give other queries a chance to run. */63 give other queries a chance to run. */
82 if ((i != 0) && ((i % chunk_size) == 0)) {64 if ((i != 0) && ((i % chunk_size) == 0)) {
@@ -92,7 +74,7 @@
92 }74 }
93 75
94 node = hash_get_nth_cell(btr_search_sys->hash_index, i)->node;76 node = hash_get_nth_cell(btr_search_sys->hash_index, i)->node;
95@@ -1997,11 +2000,13 @@77@@ -2044,11 +2047,13 @@
96 /* We release btr_search_latch every once in a while to78 /* We release btr_search_latch every once in a while to
97 give other queries a chance to run. */79 give other queries a chance to run. */
98 if (i != 0) {80 if (i != 0) {
@@ -108,7 +90,7 @@
108 }90 }
109 91
110 if (!ha_validate(btr_search_sys->hash_index, i, end_index)) {92 if (!ha_validate(btr_search_sys->hash_index, i, end_index)) {
111@@ -2009,7 +2014,8 @@93@@ -2056,7 +2061,8 @@
112 }94 }
113 }95 }
114 96

Subscribers

People subscribed via source and target branches

to all changes: