Merge lp:~percona-dev/percona-server/5.1.54-fix_expand_import_and_dict_size_limit into lp:percona-server/release-5.1.54-12

[Yasufumi Kinoshita]

[Vadim Tkachenko (vadim-tk)]

Looks good for me, asking documentation approval

[Vadim Tkachenko (vadim-tk)]

Also we need test case to cover import with the same space_id.
Asking Valentine.

[Fred Linhoss (fred-linhoss)]

Will include in next release notes for 5.1 and 5.5 and update user doc pages for innodb_expand_import and innodb_dict_size_limit.

review: Approve (documentation)

[Valentine Gostev (longbow)]

when we forget to specify --export option during prepare xb_export.sh test fails with:

2011-02-03 19:50:38: xb_export.sh: ===> xtrabackup --datadir=/root/percona-xtrabackup/test/var/mysql --prepare --export --target-dir=/tmp/xb_export_backup
xtrabackup: cd to /tmp/xb_export_backup
xtrabackup: This target seems to be not prepared yet.
xtrabackup: Temporary instance for recovery is set as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 2097152
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Compressed tables use zlib 1.2.3
InnoDB: Warning: innodb_file_io_threads is deprecated. Please use innodb_read_io_threads and innodb_write_io_threads instead
110203 19:50:38 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 38470
110203 19:50:38 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Doing recovery: scanned up to log sequence number 58034 (1 %)
110203 19:50:38 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
110203 19:50:39 Percona XtraDB (http://www.percona.com) 1.0.13-12.4 started; log sequence number 58034
xtrabackup: starting shutdown with innodb_fast_shutdown = 0
110203 19:50:39 InnoDB: Starting shutdown...
110203 19:50:39 InnoDB: Shutdown completed; log sequence number 58034
xtrabackup Ver 1.5 Rev 203 for 5.1.53 unknown-linux-gnu (x86_64)
xtrabackup: xtrabackup_logfile detected: size=2097152, start_lsn=(38470)
xtrabackup: export option is specified.
xtrabackup: export metadata of table 'incremental_sample/test' to file `./incremental_sample/test.exp` (1 indexes)
xtrabackup: name=GEN_CLUST_INDEX, id.low=15, page=3

[notice (again)]
  If you use binary log and don't use any hack of group commit,
  the binary log position seems to be:

2011-02-03 19:50:39: xb_export.sh: ===> /root/percona-xtrabackup/test/var/test/bin/mysql --no-defaults --socket=/tmp/xtrabackup.mysql.sock --user=root -e alter table test import tablespace incremental_sample
InnoDB: free limit of ./incremental_sample/test.ibd is larger than its real size.
InnoDB: import: extended import of incremental_sample/test is started.
InnoDB: import: 1 indexes are detected.
InnoDB: Progress in %: 16 33 50 66 83 100 done.
110203 19:50:39 InnoDB: Error: file './incremental_sample/test.ibd' seems to be corrupt.
InnoDB: anyway, all not corrupt pages were tried to be converted to salvage.
InnoDB: ##### CAUTION #####
InnoDB: ## The .ibd must cause to crash InnoDB, though re-import would seem to be succeeded.
InnoDB: ## If you don't have knowledge about salvaging data from .ibd, you should not use the f...

review: Needs Fixing (qa)

[Valentine Gostev (longbow)]

Please disregard my previous comment, I will file a separate bug

[Valentine Gostev (longbow)]

Please disregard prev comment - will file a separate bug

review: Approve (qa)

[Yasufumi Kinoshita (yasufumi-kinoshita)]

Thanks, we are progressing surely...

=== modified file 'innodb_dict_size_limit.patch'
--- innodb_dict_size_limit.patch	2010-12-16 11:35:26 +0000
+++ innodb_dict_size_limit.patch	2011-02-01 02:13:15 +0000
@@ -8,7 +8,7 @@
 diff -ruN a/storage/innodb_plugin/btr/btr0sea.c b/storage/innodb_plugin/btr/btr0sea.c
 --- a/storage/innodb_plugin/btr/btr0sea.c	2010-08-04 02:24:19.000000000 +0900
 +++ b/storage/innodb_plugin/btr/btr0sea.c	2010-08-27 16:09:42.926020757 +0900
-@@ -1173,6 +1173,126 @@
+@@ -1173,6 +1173,173 @@
  	mem_free(folds);
  }
  
@@ -36,96 +36,143 @@
 +	ulint		i;
 +	mem_heap_t*	heap	= NULL;
 +	ulint*		offsets;
++	ibool		released_search_latch;
 +
-+	rw_lock_x_lock(&btr_search_latch);
-+	buf_pool_mutex_enter();
++	rw_lock_s_lock(&btr_search_latch);
 +
 +	table = btr_search_sys->hash_index;
 +
-+	bpage = UT_LIST_GET_LAST(buf_pool->LRU);
-+
-+	while (bpage != NULL) {
-+		block = (buf_block_t*) bpage;
-+		if (block->index == index && block->is_hashed) {
-+			page = block->frame;
-+
-+			/* from btr_search_drop_page_hash_index() */
-+			n_fields = block->curr_n_fields;
-+			n_bytes = block->curr_n_bytes;
-+
-+			ut_a(n_fields + n_bytes > 0);
-+
-+			n_recs = page_get_n_recs(page);
-+
-+			/* Calculate and cache fold values into an array for fast deletion
-+			from the hash index */
-+
-+			folds = mem_alloc(n_recs * sizeof(ulint));
-+
-+			n_cached = 0;
-+
-+			rec = page_get_infimum_rec(page);
-+			rec = page_rec_get_next_low(rec, page_is_comp(page));
-+
-+			index_id = btr_page_get_index_id(page);
++	do {
++		buf_chunk_t*	chunks	= buf_pool->chunks;
++		buf_chunk_t*	chunk	= chunks + buf_pool->n_chunks;
++
++		released_search_latch = FALSE;
++
++		while (--chunk >= chunks) {
++			block	= chunk->blocks;
++			i	= chunk->size;
++
++retry:
++			for (; i--; block++) {
++				if (buf_block_get_state(block)
++				    != BUF_BLOCK_FILE_PAGE
++				    || block->index != index
++				    || !block->is_hashed) {
++					continue;
++				}
++
++				page = block->frame;
++
++				/* from btr_search_drop_page_hash_index() */
++				n_fields = block->curr_n_fields;
++				n_bytes = block->curr_n_bytes;
++
++
++				/* keeping latch order */
++				rw_lock_s_unlock(&btr_search_latch);
++				released_search_latch = TRUE;
++				rw_lock_x_lock(&block->lock);
++
++
++				ut_a(n_fields + n_bytes > 0);
++
++				n_recs = page_get_n_recs(page);
++
++				/* Calculate and cache fold values into an array for fast deletion
++				from the hash index */
++
++				folds = mem_alloc(n_recs * sizeof(ulint));
++
++				n_cached = 0;
++
++				rec = page_get_infimum_rec(page);
++				rec = page_rec_get_next_low(rec, page_is_comp(page));
++
++				index_id = btr_page_get_index_id(page);
 +	
-+			ut_a(0 == ut_dulint_cmp(index_id, index->id));
-+
-+			prev_fold = 0;
-+
-+			offsets = NULL;
-+
-+			while (!page_rec_is_supremum(rec)) {
-+				offsets = rec_get_offsets(rec, index, offsets,
-+							n_fields + (n_bytes > 0), &heap);
-+				ut_a(rec_offs_n_fields(offsets) == n_fields + (n_bytes > 0));
-+				fold = rec_fold(rec, offsets, n_fields, n_bytes, index_id);
-+
-+				if (fold == prev_fold && prev_fold != 0) {
-+
-+					goto next_rec;
-+				}
-+
-+				/* Remove all hash nodes pointing to this page from the
-+				hash chain */
-+
-+				folds[n_cached] = fold;
-+				n_cached++;
++				ut_a(0 == ut_dulint_cmp(index_id, index->id));
++
++				prev_fold = 0;
++
++				offsets = NULL;
++
++				while (!page_rec_is_supremum(rec)) {
++					offsets = rec_get_offsets(rec, index, offsets,
++								n_fields + (n_bytes > 0), &heap);
++					ut_a(rec_offs_n_fields(offsets) == n_fields + (n_bytes > 0));
++					fold = rec_fold(rec, offsets, n_fields, n_bytes, index_id);
++
++					if (fold == prev_fold && prev_fold != 0) {
++
++						goto next_rec;
++					}
++
++					/* Remove all hash nodes pointing to this page from the
++					hash chain */
++
++					folds[n_cached] = fold;
++					n_cached++;
 +next_rec:
-+				rec = page_rec_get_next_low(rec, page_rec_is_comp(rec));
-+				prev_fold = fold;
-+			}
-+
-+			for (i = 0; i < n_cached; i++) {
-+
-+				ha_remove_all_nodes_to_page(table, folds[i], page);
-+			}
-+
-+			ut_a(index->search_info->ref_count > 0);
-+			index->search_info->ref_count--;
-+
-+			block->is_hashed = FALSE;
-+			block->index = NULL;
-+	
++					rec = page_rec_get_next_low(rec, page_rec_is_comp(rec));
++					prev_fold = fold;
++				}
++
++				if (UNIV_LIKELY_NULL(heap)) {
++					mem_heap_empty(heap);
++				}
++
++				rw_lock_x_lock(&btr_search_latch);
++
++				if (UNIV_UNLIKELY(!block->is_hashed)) {
++					goto cleanup;
++				}
++
++				ut_a(block->index == index);
++
++				if (UNIV_UNLIKELY(block->curr_n_fields != n_fields)
++				    || UNIV_UNLIKELY(block->curr_n_bytes != n_bytes)) {
++					rw_lock_x_unlock(&btr_search_latch);
++					rw_lock_x_unlock(&block->lock);
++
++					mem_free(folds);
++
++					rw_lock_s_lock(&btr_search_latch);
++					goto retry;
++				}
++
++				for (i = 0; i < n_cached; i++) {
++
++					ha_remove_all_nodes_to_page(table, folds[i], page);
++				}
++
++				ut_a(index->search_info->ref_count > 0);
++				index->search_info->ref_count--;
++
++				block->is_hashed = FALSE;
++				block->index = NULL;
++
++cleanup:	
 +#if defined UNIV_AHI_DEBUG || defined UNIV_DEBUG
-+			if (UNIV_UNLIKELY(block->n_pointers)) {
-+				/* Corruption */
-+				ut_print_timestamp(stderr);
-+				fprintf(stderr,
++				if (UNIV_UNLIKELY(block->n_pointers)) {
++					/* Corruption */
++					ut_print_timestamp(stderr);
++					fprintf(stderr,
 +"  InnoDB: Corruption of adaptive hash index. After dropping\n"
 +"InnoDB: the hash index to a page of %s, still %lu hash nodes remain.\n",
-+					index->name, (ulong) block->n_pointers);
-+			}
++						index->name, (ulong) block->n_pointers);
++				}
 +#endif /* UNIV_AHI_DEBUG || UNIV_DEBUG */
-+
-+			mem_free(folds);
++				rw_lock_x_unlock(&btr_search_latch);
++				rw_lock_x_unlock(&block->lock);
++
++				mem_free(folds);
++
++				rw_lock_s_lock(&btr_search_latch);
++			}
 +		}
-+
-+		bpage = UT_LIST_GET_PREV(LRU, bpage);
-+	}
-+
-+	buf_pool_mutex_exit();
-+	rw_lock_x_unlock(&btr_search_latch);
++	} while (released_search_latch);
++
++	rw_lock_s_unlock(&btr_search_latch);
 +
 +	if (UNIV_LIKELY_NULL(heap)) {
 +		mem_heap_free(heap);

=== modified file 'innodb_expand_import.patch'
--- innodb_expand_import.patch	2010-12-16 11:35:26 +0000
+++ innodb_expand_import.patch	2011-02-01 02:13:15 +0000
@@ -34,8 +34,8 @@
  	space_id = fsp_header_get_space_id(page);
  	space_flags = fsp_header_get_flags(page);
  
-+	if (srv_expand_import
-+	    && (space_id != id || space_flags != (flags & ~(~0 << DICT_TF_BITS)))) {
++	if (srv_expand_import) {
++
 +		ibool		file_is_corrupt = FALSE;
 +		byte*		buf3;
 +		byte*		descr_page;

=== modified file 'innodb_split_buf_pool_mutex.patch'
--- innodb_split_buf_pool_mutex.patch	2010-12-16 11:35:26 +0000
+++ innodb_split_buf_pool_mutex.patch	2011-02-01 02:13:15 +0000
@@ -48,25 +48,7 @@
 diff -ruN a/storage/innodb_plugin/btr/btr0sea.c b/storage/innodb_plugin/btr/btr0sea.c
 --- a/storage/innodb_plugin/btr/btr0sea.c	2010-08-27 16:11:12.151975789 +0900
 +++ b/storage/innodb_plugin/btr/btr0sea.c	2010-08-27 16:11:40.593021205 +0900
-@@ -1199,7 +1199,7 @@
- 	ulint*		offsets;
- 
- 	rw_lock_x_lock(&btr_search_latch);
--	buf_pool_mutex_enter();
-+	mutex_enter(&LRU_list_mutex);
- 
- 	table = btr_search_sys->hash_index;
- 
-@@ -1285,7 +1285,7 @@
- 		bpage = UT_LIST_GET_PREV(LRU, bpage);
- 	}
- 
--	buf_pool_mutex_exit();
-+	mutex_exit(&LRU_list_mutex);
- 	rw_lock_x_unlock(&btr_search_latch);
- 
- 	if (UNIV_LIKELY_NULL(heap)) {
-@@ -1878,7 +1878,8 @@
+@@ -1925,7 +1925,8 @@
  	rec_offs_init(offsets_);
  
  	rw_lock_x_lock(&btr_search_latch);
@@ -76,7 +58,7 @@
  
  	cell_count = hash_get_n_cells(btr_search_sys->hash_index);
  
-@@ -1886,11 +1887,13 @@
+@@ -1933,11 +1934,13 @@
  		/* We release btr_search_latch every once in a while to
  		give other queries a chance to run. */
  		if ((i != 0) && ((i % chunk_size) == 0)) {
@@ -92,7 +74,7 @@
  		}
  
  		node = hash_get_nth_cell(btr_search_sys->hash_index, i)->node;
-@@ -1997,11 +2000,13 @@
+@@ -2044,11 +2047,13 @@
  		/* We release btr_search_latch every once in a while to
  		give other queries a chance to run. */
  		if (i != 0) {
@@ -108,7 +90,7 @@
  		}
  
  		if (!ha_validate(btr_search_sys->hash_index, i, end_index)) {
-@@ -2009,7 +2014,8 @@
+@@ -2056,7 +2061,8 @@
  		}
  	}