Merge lp:~akopytov/percona-xtrabackup/bug1079700-2.1 into lp:percona-xtrabackup/2.1

[Alexey Kopytov]

Bug #1079700: Issues with renaming/rotating tables during the backup
              stage

The problem was in the way XtraBackup handled DDL on individual
tablespaces on the backup stage. It first created a list of tablespaces
to copy which is represented by fil_system, but when starting the actual
tablespace copy operation later, it opened tablespace files by name. If
a tablespace file could not be opened, XtraBackup assumed the tablespace
got removed after the fil_system list was created and ignored the
tablespaces. This naturally only worked for cases when the tablespace
got dropped. A renamed tablespace would be missing in the resulting
backup, and tablespace rotations resulted in a missing tablespace and a
duplicate copy of another tablespace participating in rotation.

The idea of the fix is to make sure that once a tablespace is added to
fil_system, the underlying file is copied to backup with the same name
and space ID, regardless of what operations have been performed on the
tablespace/file during the backup procedure.

The only way to achieve that is to reuse file handles created when
opening tablespaces and adding them to fil_system, i.e. never attempt to
access tablespaces by name and rely on the fact the an open file handle
will always point to the same inode, even if the file is unlink()ed or
rename()d along the way. In other words, exclude the case when we are
going to copy a tablespace, but the file does not exist already.

This requires changes to fil_load_single_table_tablespace() to not close
the file handle after adding a space/node to fil_system, but keep the
node open and assign the handle to node->handle. We could use
fil_node_open_file() for that, but that function creates another handle
and open the tablespace file by name, which would still leave some room
for a race condition during the time when fil_system is populated. This
also requires XtraBackup to close the node correctly basically to comply
with various invariants enforced by fil_validate() in debug builds. This
part is implemented in XtraBackup in xb_fil_node_close_file().

We also want to reuse file handles in fil_load_single_table_tablespace()
only at the backup stage. Historically, XtraBackup patches used
recv_recovery_on to detect whether we are currently in the 'backup' or
'recovery' mode. That doesn't always work reliably. For example, we may
initialize fil_system before we start recovery to apply an incremental
backup. So recovery is still not started, but we are not in the backup
mode either. To circumvent that, the patch introduces another global
variable srv_backup_mode which is set by XtraBackup only in
xtrabackup_backup_func(). This patch also changes remote tablespaces
support in innodb56.patch to use srv_backup_mode instead of
recv_recovery_on.

Another change necessitated by the patch is ignoring deleted tablespaces
on recovery, which has been implemented in XB patches to
recv_apply_hashed_log_recs(). After MLOG_FILE_DELETE is replayed on
recovery and the underlying file is deleted, possible updates to the
same tablespace done before deleting the tablespace are left in the redo
log. They will be ignored by the recovery code. This can never cause any
problems for server, because file creation/removal is done immediately,
so on recovery InnoDB detects missing tablespace files and does not the
corresponding log records to the hash table (see the first lines in
recv_add_to_hash_table). The situation is different with XtraBackup: the
underlying file will be present when recovery starts, but then get
deleted when MLOG_FILE_DELETE is replayed, i.e. after all log records
from the current batch have been added to the hash table. As a result,
log records corresponding to the deleted tablespace are ignore, but are
still left in the hash table, so recv_apply_hashed_log_recs() hangs
forever waiting for recv_sys->n_addrs to become zero. This was also
possible with XtraBackup before this fix, i.e. when a tablespace is
removed after it has been copied by XtraBackup. This fix just makes this
condition more likely to occur, as we always copy all tablespaces rather
than ignore those removed before the copy operation is tarted.

To fix the above case, we also check for deleted tablespaces in
recv_apply_hashed_log_recs(). Those corresponding to previously deleted
tablespaces are marked as processed and recv_sys->n_addrs is decremented
accordingly to not leave spurious unprocessed log records in the hash
table.

Finally, now that we reuse fil_system file handles again, bug #870119
needs another fix. I.e. the number of open file handles grows to the
number of tablespaces we want to copy, so we want to prevent InnoDB LRU
policies from kicking in and closing/reusing file handles. There are
multiple ways to achieve that. Patching InnoDB code to disable
fil_system LRU and file closing policies in InnoDB appeared to be too
risky. The easiest one is to set the allowed number of open InnoDB files
to some vary large value unconditionally (i.e. override
innodb_open_files with the maximum possible value, LONG_MAX). InnoDB
does not allocate any resource for each srv_max_n_open_files
increment. It's just the maximum possible LRU list length, so this
change does not incur any additional resource consumption.

This revision also extends bug722638.sh (since there's 95% overlap
between that one and the test case for this bug), and renames
bug722638.sh to ddl.sh to better reflect the contents. It also backports
record_db_state() / verify_db_state() from the 2.1 test suite, because
messing with individual table checksums and checksum_table looks rather
cumbersome with a higher number of tables in the test.

http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.1-param/299/

[Laurynas Biveinis (laurynas-biveinis)]

Same comments as for the 2.0 MP.

review: Needs Information

[Laurynas Biveinis (laurynas-biveinis)]

review: Approve

=== modified file 'patches/innodb51.patch'
--- patches/innodb51.patch	2013-04-16 14:34:55 +0000
+++ patches/innodb51.patch	2013-05-08 05:18:19 +0000
@@ -474,7 +474,60 @@
  	if (size < FIL_IBD_FILE_INITIAL_SIZE * UNIV_PAGE_SIZE) {
  		fprintf(stderr,
  			"InnoDB: Error: the size of single-table tablespace"
-@@ -3425,7 +3530,7 @@
+@@ -3414,7 +3519,51 @@
+ 
+ 	fil_node_create(filepath, 0, space_id, FALSE);
+ func_exit:
+-	os_file_close(file);
++	/* We reuse file handles on the backup stage in XtraBackup to avoid
++	inconsistencies between the file name and the actual tablespace contents
++	if a DDL occurs between a fil_load_single_table_tablespaces() call and
++	the actual copy operation. */
++	if (srv_backup_mode) {
++
++		fil_node_t*	node;
++		fil_space_t*	space;
++
++		mutex_enter(&fil_system->mutex);
++
++		space = fil_space_get_by_id(space_id);
++
++		if (space) {
++			node = UT_LIST_GET_LAST(space->chain);
++
++			/* The handle will be closed by xtrabackup in
++			xtrabackup_copy_datafile(). We set node->open to TRUE to
++			make sure no one calls fil_node_open_file()
++			(i.e. attempts to reopen the tablespace by name) during
++			the backup stage. */
++
++			node->open = TRUE;
++			node->handle = file;
++
++			/* The following is copied from fil_node_open_file() to
++			pass fil_system validity checks. We cannot use
++			fil_node_open_file() directly, as that would re-open the
++			file by name and create another file handle. */
++
++			fil_system->n_open++;
++
++			if (space->purpose == FIL_TABLESPACE &&
++			    space->id != 0) {
++
++				/* Put the node to the LRU list */
++				UT_LIST_ADD_FIRST(LRU, fil_system->LRU, node);
++			}
++		}
++
++		mutex_exit(&fil_system->mutex);
++	} else {
++
++		os_file_close(file);
++	}
+ 	ut_free(buf2);
+ 	mem_free(filepath);
+ }
+@@ -3425,7 +3574,7 @@
  idea is to read as much good data as we can and jump over bad data.
  @return 0 if ok, -1 if error even after the retries, 1 if at the end
  of the directory */
@@ -483,7 +536,7 @@
  int
  fil_file_readdir_next_file(
  /*=======================*/
-@@ -3469,7 +3574,7 @@
+@@ -3469,7 +3618,7 @@
  @return	DB_SUCCESS or error number */
  UNIV_INTERN
  ulint
@@ -492,7 +545,7 @@
  /*===================================*/
  {
  	int		ret;
-@@ -3525,7 +3630,9 @@
+@@ -3525,7 +3674,9 @@
  			dbinfo.name);
  		srv_normalize_path_for_win(dbpath);
  
@@ -503,7 +556,7 @@
  
  		if (dbdir != NULL) {
  			/* printf("Opened dir %s\n", dbinfo.name); */
-@@ -3551,8 +3658,11 @@
+@@ -3551,8 +3702,11 @@
  						   ".ibd")) {
  					/* The name ends in .ibd; try opening
  					the file */
@@ -516,7 +569,7 @@
  				}
  next_file_item:
  				ret = fil_file_readdir_next_file(&err,
-@@ -3724,15 +3834,97 @@
+@@ -3724,15 +3878,97 @@
  				"InnoDB: in InnoDB data dictionary"
  				" has tablespace id %lu,\n"
  				"InnoDB: but tablespace with that id"
@@ -622,7 +675,7 @@
  		} else {
  			ut_print_timestamp(stderr);
  			fputs("  InnoDB: Error: table ", stderr);
-@@ -4121,7 +4313,7 @@
+@@ -4121,7 +4357,7 @@
  off the LRU list if it is in the LRU list. The caller must hold the fil_sys
  mutex. */
  static
@@ -631,7 +684,7 @@
  fil_node_prepare_for_io(
  /*====================*/
  	fil_node_t*	node,	/*!< in: file node */
-@@ -4141,10 +4333,13 @@
+@@ -4141,10 +4377,13 @@
  	}
  
  	if (node->open == FALSE) {
@@ -646,7 +699,7 @@
  	}
  
  	if (node->n_pending == 0 && space->purpose == FIL_TABLESPACE
-@@ -4157,6 +4352,8 @@
+@@ -4157,6 +4396,8 @@
  	}
  
  	node->n_pending++;
@@ -655,7 +708,7 @@
  }
  
  /********************************************************************//**
-@@ -4292,7 +4489,9 @@
+@@ -4292,7 +4533,9 @@
  	ut_ad(recv_no_ibuf_operations || (type == OS_FILE_WRITE)
  	      || !ibuf_bitmap_page(zip_size, block_offset)
  	      || sync || is_log);
@@ -666,7 +719,7 @@
  	      || ibuf_page(space_id, zip_size, block_offset, NULL));
  # endif /* UNIV_LOG_DEBUG */
  	if (sync) {
-@@ -4341,6 +4540,16 @@
+@@ -4341,6 +4584,16 @@
  
  	ut_ad((mode != OS_AIO_IBUF) || (space->purpose == FIL_TABLESPACE));
  
@@ -740,7 +793,7 @@
  
 --- a/storage/innodb_plugin/include/srv0srv.h
 +++ b/storage/innodb_plugin/include/srv0srv.h
-@@ -202,6 +202,10 @@
+@@ -202,6 +202,11 @@
  extern ulong	srv_max_purge_lag;
  
  extern ulong	srv_replication_delay;
@@ -748,6 +801,7 @@
 +extern ibool	srv_read_only;
 +extern ibool	srv_fake_write;
 +extern ibool	srv_apply_log_only;
++extern ibool	srv_backup_mode;
  /*-------------------------------------------*/
  
  extern ulint	srv_n_rows_inserted;
@@ -961,7 +1015,35 @@
  
  			ib_uint64_t	end_lsn;
  
-@@ -2297,7 +2306,7 @@
+@@ -1762,6 +1771,18 @@
+ 			ulint	zip_size = fil_space_get_zip_size(space);
+ 			ulint	page_no = recv_addr->page_no;
+ 
++			/* By now we have replayed all DDL log records from the
++			current batch. Check if the space ID is still valid in
++			the entry being processed, and ignore it if it is not.*/
++			if (fil_tablespace_deleted_or_being_deleted_in_mem(space, -1)) {
++
++				ut_a(recv_sys->n_addrs);
++
++				recv_addr->state = RECV_PROCESSED;
++				recv_sys->n_addrs--;
++
++				goto next;
++			}
+ 			if (recv_addr->state == RECV_NOT_PROCESSED) {
+ 				if (!has_printed) {
+ 					ut_print_timestamp(stderr);
+@@ -1795,7 +1816,7 @@
+ 
+ 				mutex_enter(&(recv_sys->mutex));
+ 			}
+-
++next:
+ 			recv_addr = HASH_GET_NEXT(addr_hash, recv_addr);
+ 		}
+ 
+@@ -2297,7 +2318,7 @@
  			   || type == MLOG_FILE_RENAME
  			   || type == MLOG_FILE_DELETE) {
  			ut_a(space);
@@ -970,7 +1052,7 @@
  			if (recv_replay_file_ops) {
  
  				/* In ibbackup --apply-log, replay an .ibd file
-@@ -2320,7 +2329,7 @@
+@@ -2320,7 +2341,7 @@
  					ut_error;
  				}
  			}
@@ -979,7 +1061,7 @@
  			/* In normal mysqld crash recovery we do not try to
  			replay file operations */
  #ifdef UNIV_LOG_LSN_DEBUG
-@@ -2737,8 +2746,11 @@
+@@ -2737,8 +2758,11 @@
  
  			fprintf(stderr,
  				"InnoDB: Doing recovery: scanned up to"
@@ -993,7 +1075,7 @@
  		}
  	}
  
-@@ -2841,7 +2845,7 @@
+@@ -2841,7 +2865,7 @@
  		"InnoDB: Reading tablespace information"
  		" from the .ibd files...\n");
  
@@ -1002,7 +1084,7 @@
  
  	/* If we are using the doublewrite method, we will
  	check if there are half-written pages in data files,
-@@ -2850,12 +2862,14 @@
+@@ -2850,12 +2874,14 @@
  
  	if (srv_force_recovery < SRV_FORCE_NO_LOG_REDO) {
  
@@ -1018,7 +1100,7 @@
  	}
  }
  
-@@ -3005,6 +3019,7 @@
+@@ -3005,6 +3031,7 @@
  		recv_sys->recovered_lsn = checkpoint_lsn;
  
  		srv_start_lsn = checkpoint_lsn;
@@ -1026,7 +1108,7 @@
  	}
  
  	contiguous_lsn = ut_uint64_align_down(recv_sys->scanned_lsn,
-@@ -3286,6 +3301,7 @@
+@@ -3286,6 +3313,7 @@
  	that the data dictionary tables will be free of any locks.
  	The data dictionary latch should guarantee that there is at
  	most one data dictionary transaction active at a time. */
@@ -1111,17 +1193,18 @@
  
 --- a/storage/innodb_plugin/srv/srv0srv.c
 +++ b/storage/innodb_plugin/srv/srv0srv.c
-@@ -374,6 +374,9 @@
+@@ -374,6 +374,10 @@
  
  UNIV_INTERN ulong	srv_replication_delay		= 0;
  
 +UNIV_INTERN ibool	srv_read_only   = FALSE;
 +UNIV_INTERN ibool	srv_fake_write  = FALSE;
 +UNIV_INTERN ibool	srv_apply_log_only = FALSE;
++UNIV_INTERN ibool	srv_backup_mode = FALSE;
  /*-------------------------------------------*/
  UNIV_INTERN ulong	srv_n_spin_wait_rounds	= 30;
  UNIV_INTERN ulong	srv_n_free_tickets_to_enter = 500;
-@@ -1002,7 +1005,7 @@
+@@ -1002,7 +1006,7 @@
  	}
  
  	/* Initialize some INFORMATION SCHEMA internal structures */
@@ -1130,7 +1213,7 @@
  }
  
  /*********************************************************************//**
-@@ -1013,6 +1016,7 @@
+@@ -1013,6 +1017,7 @@
  /*==========*/
  {
  	os_fast_mutex_free(&srv_conc_mutex);
@@ -1138,7 +1221,7 @@
  	mem_free(srv_conc_slots);
  	srv_conc_slots = NULL;
  
-@@ -1026,6 +1030,7 @@
+@@ -1026,6 +1031,7 @@
  	srv_mysql_table = NULL;
  
  	trx_i_s_cache_free(trx_i_s_cache);
@@ -1410,3 +1493,14 @@
  
  	UNIV_MEM_FREE(buf, n);
  }
+--- a/configure.in
++++ b/configure.in
+@@ -22,7 +22,7 @@
+ AM_INIT_AUTOMAKE([1.9 tar-ustar])
+ AC_PROG_LIBTOOL
+ 
+-AM_CONFIG_HEADER([include/config.h])
++AC_CONFIG_HEADERS([include/config.h])
+ 
+ # Request support for automake silent-rules if available.
+ # Default to verbose output. One can use the configure-time

=== modified file 'patches/innodb55.patch'
--- patches/innodb55.patch	2013-04-16 14:34:55 +0000
+++ patches/innodb55.patch	2013-05-08 05:18:19 +0000
@@ -464,7 +464,60 @@
  	if (size < FIL_IBD_FILE_INITIAL_SIZE * UNIV_PAGE_SIZE) {
  		fprintf(stderr,
  			"InnoDB: Error: the size of single-table tablespace"
-@@ -3476,7 +3581,7 @@
+@@ -3465,7 +3570,51 @@
+ 
+ 	fil_node_create(filepath, 0, space_id, FALSE);
+ func_exit:
+-	os_file_close(file);
++	/* We reuse file handles on the backup stage in XtraBackup to avoid
++	inconsistencies between the file name and the actual tablespace contents
++	if a DDL occurs between a fil_load_single_table_tablespaces() call and
++	the actual copy operation. */
++	if (srv_backup_mode) {
++
++		fil_node_t*	node;
++		fil_space_t*	space;
++
++		mutex_enter(&fil_system->mutex);
++
++		space = fil_space_get_by_id(space_id);
++
++		if (space) {
++			node = UT_LIST_GET_LAST(space->chain);
++
++			/* The handle will be closed by xtrabackup in
++			xtrabackup_copy_datafile(). We set node->open to TRUE to
++			make sure no one calls fil_node_open_file()
++			(i.e. attempts to reopen the tablespace by name) during
++			the backup stage. */
++
++			node->open = TRUE;
++			node->handle = file;
++
++			/* The following is copied from fil_node_open_file() to
++			pass fil_system validity checks. We cannot use
++			fil_node_open_file() directly, as that would re-open the
++			file by name and create another file handle. */
++
++			fil_system->n_open++;
++
++			if (space->purpose == FIL_TABLESPACE &&
++			    space->id != 0) {
++
++				/* Put the node to the LRU list */
++				UT_LIST_ADD_FIRST(LRU, fil_system->LRU, node);
++			}
++		}
++
++		mutex_exit(&fil_system->mutex);
++	} else {
++
++		os_file_close(file);
++	}
+ 	ut_free(buf2);
+ 	mem_free(filepath);
+ }
+@@ -3476,7 +3625,7 @@
  idea is to read as much good data as we can and jump over bad data.
  @return 0 if ok, -1 if error even after the retries, 1 if at the end
  of the directory */
@@ -473,7 +526,7 @@
  int
  fil_file_readdir_next_file(
  /*=======================*/
-@@ -3520,7 +3625,7 @@
+@@ -3520,7 +3669,7 @@
  @return	DB_SUCCESS or error number */
  UNIV_INTERN
  ulint
@@ -482,7 +535,7 @@
  /*===================================*/
  {
  	int		ret;
-@@ -3576,7 +3681,9 @@
+@@ -3576,7 +3725,9 @@
  			dbinfo.name);
  		srv_normalize_path_for_win(dbpath);
  
@@ -493,7 +546,7 @@
  
  		if (dbdir != NULL) {
  			/* printf("Opened dir %s\n", dbinfo.name); */
-@@ -3602,8 +3709,11 @@
+@@ -3602,8 +3753,11 @@
  						   ".ibd")) {
  					/* The name ends in .ibd; try opening
  					the file */
@@ -506,7 +559,7 @@
  				}
  next_file_item:
  				ret = fil_file_readdir_next_file(&err,
-@@ -3775,15 +3885,97 @@
+@@ -3775,15 +3929,97 @@
  				"InnoDB: in InnoDB data dictionary"
  				" has tablespace id %lu,\n"
  				"InnoDB: but tablespace with that id"
@@ -612,7 +665,7 @@
  		} else {
  			ut_print_timestamp(stderr);
  			fputs("  InnoDB: Error: table ", stderr);
-@@ -4172,7 +4364,7 @@
+@@ -4172,7 +4408,7 @@
  off the LRU list if it is in the LRU list. The caller must hold the fil_sys
  mutex. */
  static
@@ -621,7 +674,7 @@
  fil_node_prepare_for_io(
  /*====================*/
  	fil_node_t*	node,	/*!< in: file node */
-@@ -4192,10 +4384,13 @@
+@@ -4192,10 +4428,13 @@
  	}
  
  	if (node->open == FALSE) {
@@ -636,7 +689,7 @@
  	}
  
  	if (node->n_pending == 0 && space->purpose == FIL_TABLESPACE
-@@ -4208,6 +4403,8 @@
+@@ -4208,6 +4447,8 @@
  	}
  
  	node->n_pending++;
@@ -645,7 +698,7 @@
  }
  
  /********************************************************************//**
-@@ -4390,6 +4587,16 @@
+@@ -4390,6 +4631,16 @@
  
  	ut_ad((mode != OS_AIO_IBUF) || (space->purpose == FIL_TABLESPACE));
  
@@ -697,7 +750,7 @@
  Returns TRUE if a single-table tablespace does not exist in the memory cache,
 --- a/storage/innobase/include/srv0srv.h
 +++ b/storage/innobase/include/srv0srv.h
-@@ -218,6 +218,10 @@
+@@ -218,6 +218,11 @@
  extern ulong	srv_max_purge_lag;
  
  extern ulong	srv_replication_delay;
@@ -705,6 +758,7 @@
 +extern ibool	srv_read_only;
 +extern ibool	srv_fake_write;
 +extern ibool	srv_apply_log_only;
++extern ibool	srv_backup_mode;
  /*-------------------------------------------*/
  
  extern ulint	srv_n_rows_inserted;
@@ -874,7 +928,35 @@
  
  			ib_uint64_t	end_lsn;
  
-@@ -2308,7 +2317,7 @@
+@@ -1774,6 +1783,18 @@
+ 			ulint	zip_size = fil_space_get_zip_size(space);
+ 			ulint	page_no = recv_addr->page_no;
+ 
++			/* By now we have replayed all DDL log records from the
++			current batch. Check if the space ID is still valid in
++			the entry being processed, and ignore it if it is not.*/
++			if (fil_tablespace_deleted_or_being_deleted_in_mem(space, -1)) {
++
++				ut_a(recv_sys->n_addrs);
++
++				recv_addr->state = RECV_PROCESSED;
++				recv_sys->n_addrs--;
++
++				goto next;
++			}
+ 			if (recv_addr->state == RECV_NOT_PROCESSED) {
+ 				if (!has_printed) {
+ 					ut_print_timestamp(stderr);
+@@ -1807,7 +1828,7 @@
+ 
+ 				mutex_enter(&(recv_sys->mutex));
+ 			}
+-
++next:
+ 			recv_addr = HASH_GET_NEXT(addr_hash, recv_addr);
+ 		}
+ 
+@@ -2308,7 +2329,7 @@
  			   || type == MLOG_FILE_RENAME
  			   || type == MLOG_FILE_DELETE) {
  			ut_a(space);
@@ -883,7 +965,7 @@
  			if (recv_replay_file_ops) {
  
  				/* In ibbackup --apply-log, replay an .ibd file
-@@ -2331,7 +2340,7 @@
+@@ -2331,7 +2352,7 @@
  					ut_error;
  				}
  			}
@@ -892,7 +974,7 @@
  			/* In normal mysqld crash recovery we do not try to
  			replay file operations */
  #ifdef UNIV_LOG_LSN_DEBUG
-@@ -2748,8 +2757,11 @@
+@@ -2748,8 +2769,11 @@
  
  			fprintf(stderr,
  				"InnoDB: Doing recovery: scanned up to"
@@ -906,7 +988,7 @@
  		}
  	}
  
-@@ -2854,7 +2858,7 @@
+@@ -2854,7 +2878,7 @@
  		"InnoDB: Reading tablespace information"
  		" from the .ibd files...\n");
  
@@ -915,7 +997,7 @@
  
  	/* If we are using the doublewrite method, we will
  	check if there are half-written pages in data files,
-@@ -2863,12 +2875,14 @@
+@@ -2863,12 +2887,14 @@
  
  	if (srv_force_recovery < SRV_FORCE_NO_LOG_REDO) {
  
@@ -931,7 +1013,7 @@
  	}
  }
  
-@@ -3018,6 +3032,7 @@
+@@ -3018,6 +3044,7 @@
  		recv_sys->recovered_lsn = checkpoint_lsn;
  
  		srv_start_lsn = checkpoint_lsn;
@@ -939,7 +1021,7 @@
  	}
  
  	contiguous_lsn = ut_uint64_align_down(recv_sys->scanned_lsn,
-@@ -3299,6 +3314,7 @@
+@@ -3299,6 +3326,7 @@
  	that the data dictionary tables will be free of any locks.
  	The data dictionary latch should guarantee that there is at
  	most one data dictionary transaction active at a time. */
@@ -1023,17 +1105,18 @@
  }
 --- a/storage/innobase/srv/srv0srv.c
 +++ b/storage/innobase/srv/srv0srv.c
-@@ -399,6 +399,9 @@
+@@ -399,6 +399,10 @@
  
  UNIV_INTERN ulong	srv_replication_delay		= 0;
  
 +UNIV_INTERN ibool	srv_read_only   = FALSE;
 +UNIV_INTERN ibool	srv_fake_write  = FALSE;
 +UNIV_INTERN ibool	srv_apply_log_only = FALSE;
++UNIV_INTERN ibool	srv_backup_mode = FALSE;
  /*-------------------------------------------*/
  UNIV_INTERN ulong	srv_n_spin_wait_rounds	= 30;
  UNIV_INTERN ulong	srv_n_free_tickets_to_enter = 500;
-@@ -1078,7 +1081,7 @@
+@@ -1078,7 +1082,7 @@
  	}
  
  	/* Initialize some INFORMATION SCHEMA internal structures */
@@ -1042,7 +1125,7 @@
  }
  
  /*********************************************************************//**
-@@ -1089,6 +1092,7 @@
+@@ -1089,6 +1093,7 @@
  /*==========*/
  {
  	os_fast_mutex_free(&srv_conc_mutex);
@@ -1050,7 +1133,7 @@
  	mem_free(srv_conc_slots);
  	srv_conc_slots = NULL;
  
-@@ -1102,6 +1106,7 @@
+@@ -1102,6 +1107,7 @@
  	srv_mysql_table = NULL;
  
  	trx_i_s_cache_free(trx_i_s_cache);
@@ -1058,7 +1141,7 @@
  }
  
  /*********************************************************************//**
-@@ -1717,7 +1722,7 @@
+@@ -1717,7 +1723,7 @@
  		}
  
  		/* Record the lock wait time for this thread */

=== modified file 'patches/innodb56.patch'
--- patches/innodb56.patch	2013-05-04 05:26:19 +0000
+++ patches/innodb56.patch	2013-05-08 05:18:19 +0000
@@ -375,7 +375,7 @@
  
 +	/* Ignore .isl files on XtraBackup recovery. All tablespaces must be
 +	local. */
-+	if (!recv_recovery_on) {
++	if (srv_backup_mode) {
  	/* Check for a link file which locates a remote tablespace. */
  	remote.success = fil_open_linked_file(
  		tablename, &remote.filepath, &remote.file);
@@ -396,7 +396,60 @@
  		ib_logf(IB_LOG_LEVEL_ERROR,
  			"The size of single-table tablespace file %s "
  			"is only " UINT64PF ", should be at least %lu!",
-@@ -4263,7 +4390,7 @@
+@@ -4243,7 +4370,51 @@
+ 	}
+ 
+ func_exit:
+-	os_file_close(fsp->file);
++	/* We reuse file handles on the backup stage in XtraBackup to avoid
++	inconsistencies between the file name and the actual tablespace contents
++	if a DDL occurs between a fil_load_single_table_tablespaces() call and
++	the actual copy operation. */
++	if (srv_backup_mode) {
++
++		fil_node_t*	node;
++		fil_space_t*	space;
++
++		mutex_enter(&fil_system->mutex);
++
++		space = fil_space_get_by_id(fsp->id);
++
++		if (space) {
++			node = UT_LIST_GET_LAST(space->chain);
++
++			/* The handle will be closed by xtrabackup in
++			xtrabackup_copy_datafile(). We set node->open to TRUE to
++			make sure no one calls fil_node_open_file()
++			(i.e. attempts to reopen the tablespace by name) during
++			the backup stage. */
++
++			node->open = TRUE;
++			node->handle = fsp->file;
++
++			/* The following is copied from fil_node_open_file() to
++			pass fil_system validaty checks. We cannot use
++			fil_node_open_file() directly, as that would re-open the
++			file by name and create another file handle. */
++
++			fil_system->n_open++;
++			fil_n_file_opened++;
++
++			if (fil_space_belongs_in_lru(space)) {
++
++				/* Put the node to the LRU list */
++				UT_LIST_ADD_FIRST(LRU, fil_system->LRU, node);
++			}
++		}
++
++		mutex_exit(&fil_system->mutex);
++	} else {
++
++		os_file_close(fsp->file);
++	}
+ 
+ #ifdef UNIV_HOTBACKUP
+ func_exit_after_close:
+@@ -4263,7 +4434,7 @@
  idea is to read as much good data as we can and jump over bad data.
  @return 0 if ok, -1 if error even after the retries, 1 if at the end
  of the directory */
@@ -405,7 +458,7 @@
  int
  fil_file_readdir_next_file(
  /*=======================*/
-@@ -4303,7 +4430,7 @@
+@@ -4303,7 +4474,7 @@
  @return	DB_SUCCESS or error number */
  UNIV_INTERN
  dberr_t
@@ -414,7 +467,7 @@
  /*===================================*/
  {
  	int		ret;
-@@ -4359,7 +4486,9 @@
+@@ -4359,7 +4530,9 @@
  			    "%s/%s", fil_path_to_mysql_datadir, dbinfo.name);
  		srv_normalize_path_for_win(dbpath);
  
@@ -425,7 +478,7 @@
  
  		if (dbdir != NULL) {
  
-@@ -4380,9 +4509,15 @@
+@@ -4380,9 +4553,15 @@
  				    && (0 == strcmp(fileinfo.name
  						   + strlen(fileinfo.name) - 4,
  						   ".ibd")
@@ -433,7 +486,7 @@
 +					/* Ignore .isl files on XtraBackup
 +					recovery, all tablespaces must be
 +					local. */
-+					|| (!recv_recovery_on &&
++					|| (srv_backup_mode &&
 +					    0 == strcmp(fileinfo.name
  						   + strlen(fileinfo.name) - 4,
 -						   ".isl"))) {
@@ -443,7 +496,7 @@
  					/* The name ends in .ibd or .isl;
  					try opening the file */
  					fil_load_single_table_tablespace(
-@@ -4538,6 +4673,7 @@
+@@ -4538,6 +4717,7 @@
  {
  	fil_space_t*	fnamespace;
  	fil_space_t*	space;
@@ -451,7 +504,7 @@
  
  	ut_ad(fil_system);
  
-@@ -4615,6 +4751,10 @@
+@@ -4615,6 +4795,10 @@
  		if (fnamespace == NULL) {
  			if (print_error_if_does_not_exist) {
  				fil_report_missing_tablespace(name, id);
@@ -462,7 +515,7 @@
  			}
  		} else {
  			ut_print_timestamp(stderr);
-@@ -4638,6 +4778,10 @@
+@@ -4638,6 +4822,10 @@
  
  		mutex_exit(&fil_system->mutex);
  
@@ -473,7 +526,7 @@
  		return(FALSE);
  	}
  
-@@ -4728,6 +4872,7 @@
+@@ -4728,6 +4916,7 @@
  	ulint		page_size;
  	ulint		pages_added;
  	ibool		success;
@@ -481,7 +534,7 @@
  
  	ut_ad(!srv_read_only_mode);
  
-@@ -4772,13 +4917,17 @@
+@@ -4772,13 +4961,17 @@
  		goto retry;
  	}
  
@@ -500,7 +553,7 @@
  	start_page_no = space->size;
  	file_start_page_no = space->size - node->size;
  
-@@ -5024,7 +5173,7 @@
+@@ -5024,7 +5217,7 @@
  off the LRU list if it is in the LRU list. The caller must hold the fil_sys
  mutex. */
  static
@@ -509,7 +562,7 @@
  fil_node_prepare_for_io(
  /*====================*/
  	fil_node_t*	node,	/*!< in: file node */
-@@ -5044,9 +5193,12 @@
+@@ -5044,9 +5237,12 @@
  	}
  
  	if (node->open == FALSE) {
@@ -523,7 +576,7 @@
  	}
  
  	if (node->n_pending == 0 && fil_space_belongs_in_lru(space)) {
-@@ -5058,6 +5210,8 @@
+@@ -5058,6 +5254,8 @@
  	}
  
  	node->n_pending++;
@@ -532,7 +585,7 @@
  }
  
  /********************************************************************//**
-@@ -5259,6 +5413,16 @@
+@@ -5259,6 +5457,16 @@
  
  	ut_ad(mode != OS_AIO_IBUF || space->purpose == FIL_TABLESPACE);
  
@@ -549,7 +602,7 @@
  	node = UT_LIST_GET_FIRST(space->chain);
  
  	for (;;) {
-@@ -5290,7 +5454,11 @@
+@@ -5290,7 +5498,11 @@
  	}
  
  	/* Open file if closed */
@@ -562,7 +615,7 @@
  
  	/* Check that at least the start offset is within the bounds of a
  	single-table tablespace, including rollback tablespaces. */
-@@ -6164,6 +6332,7 @@
+@@ -6164,6 +6376,7 @@
  	return(err);
  }
  
@@ -570,7 +623,7 @@
  /****************************************************************//**
  Generate redo logs for swapping two .ibd files */
  UNIV_INTERN
-@@ -6187,4 +6356,4 @@
+@@ -6187,4 +6400,4 @@
  			 0, 0, new_name, old_name, &mtr);
  	mtr_commit(&mtr);
  }
@@ -713,13 +766,15 @@
  Returns TRUE if a single-table tablespace does not exist in the memory cache,
 --- a/storage/innobase/include/srv0srv.h
 +++ b/storage/innobase/include/srv0srv.h
-@@ -353,6 +353,9 @@
+@@ -353,6 +353,11 @@
  extern ulong	srv_max_purge_lag_delay;
  
  extern ulong	srv_replication_delay;
 +
 +extern ibool	srv_apply_log_only;
 +
++extern ibool	srv_backup_mode;
++
  /*-------------------------------------------*/
  
  extern ibool	srv_print_innodb_monitor;
@@ -814,7 +869,26 @@
  
  			lsn_t	end_lsn;
  
-@@ -2413,7 +2424,7 @@
+@@ -1868,6 +1879,18 @@
+ 			ulint	zip_size = fil_space_get_zip_size(space);
+ 			ulint	page_no = recv_addr->page_no;
+ 
++			/* By now we have replayed all DDL log records from the
++			current batch. Check if the space ID is still valid in
++			the entry being processed, and ignore it if it is not.*/
++			if (fil_tablespace_deleted_or_being_deleted_in_mem(space, -1)) {
++
++				ut_a(recv_sys->n_addrs);
++
++				recv_addr->state = RECV_PROCESSED;
++				recv_sys->n_addrs--;
++
++				continue;
++			}
+ 			if (recv_addr->state == RECV_NOT_PROCESSED) {
+ 				if (!has_printed) {
+ 					ib_logf(IB_LOG_LEVEL_INFO,
+@@ -2413,7 +2436,7 @@
  			   || type == MLOG_FILE_RENAME
  			   || type == MLOG_FILE_DELETE) {
  			ut_a(space);
@@ -823,7 +897,7 @@
  			if (recv_replay_file_ops) {
  
  				/* In ibbackup --apply-log, replay an .ibd file
-@@ -2436,7 +2447,7 @@
+@@ -2436,7 +2459,7 @@
  					ut_error;
  				}
  			}
@@ -832,7 +906,7 @@
  			/* In normal mysqld crash recovery we do not try to
  			replay file operations */
  #ifdef UNIV_LOG_LSN_DEBUG
-@@ -2863,8 +2874,14 @@
+@@ -2863,8 +2886,14 @@
  
  			fprintf(stderr,
  				"InnoDB: Doing recovery: scanned up to"
@@ -849,7 +923,7 @@
  		}
  	}
  
-@@ -2964,7 +2981,7 @@
+@@ -2964,7 +2993,7 @@
  	ib_logf(IB_LOG_LEVEL_INFO,
  		"Reading tablespace information from the .ibd files...");
  
@@ -858,7 +932,7 @@
  
  	/* If we are using the doublewrite method, we will
  	check if there are half-written pages in data files,
-@@ -3456,7 +3473,8 @@
+@@ -3456,7 +3485,8 @@
  	that the data dictionary tables will be free of any locks.
  	The data dictionary latch should guarantee that there is at
  	most one data dictionary transaction active at a time. */
@@ -895,16 +969,18 @@
  	tested with following crash point */
 --- a/storage/innobase/srv/srv0srv.cc
 +++ b/storage/innobase/srv/srv0srv.cc
-@@ -349,6 +349,8 @@
+@@ -349,6 +349,10 @@
  
  UNIV_INTERN ulong	srv_replication_delay		= 0;
  
 +UNIV_INTERN ibool	srv_apply_log_only	= FALSE;
 +
++UNIV_INTERN ibool	srv_backup_mode	= FALSE;
++
  /*-------------------------------------------*/
  UNIV_INTERN ulong	srv_n_spin_wait_rounds	= 30;
  UNIV_INTERN ulong	srv_spin_wait_delay	= 6;
-@@ -1808,7 +1810,8 @@
+@@ -1808,7 +1812,8 @@
  	if (ret == SRV_NONE
  	    && srv_shutdown_state != SRV_SHUTDOWN_NONE
  	    && trx_purge_state() != PURGE_STATE_DISABLED

=== modified file 'patches/xtradb51.patch'
--- patches/xtradb51.patch	2013-04-16 14:34:55 +0000
+++ patches/xtradb51.patch	2013-05-08 05:18:19 +0000
@@ -357,7 +357,60 @@
  	if (size < FIL_IBD_FILE_INITIAL_SIZE * (lint)UNIV_PAGE_SIZE) {
  		fprintf(stderr,
  			"InnoDB: Error: the size of single-table tablespace"
-@@ -4247,7 +4349,7 @@
+@@ -4236,7 +4338,51 @@
+ 
+ 	fil_node_create(filepath, 0, space_id, FALSE);
+ func_exit:
+-	os_file_close(file);
++	/* We reuse file handles on the backup stage in XtraBackup to avoid
++	inconsistencies between the file name and the actual tablespace contents
++	if a DDL occurs between a fil_load_single_table_tablespaces() call and
++	the actual copy operation. */
++	if (srv_backup_mode) {
++
++		fil_node_t*	node;
++		fil_space_t*	space;
++
++		mutex_enter(&fil_system->mutex);
++
++		space = fil_space_get_by_id(space_id);
++
++		if (space) {
++			node = UT_LIST_GET_LAST(space->chain);
++
++			/* The handle will be closed by xtrabackup in
++			xtrabackup_copy_datafile(). We set node->open to TRUE to
++			make sure no one calls fil_node_open_file()
++			(i.e. attempts to reopen the tablespace by name) during
++			the backup stage. */
++
++			node->open = TRUE;
++			node->handle = file;
++
++			/* The following is copied from fil_node_open_file() to
++			pass fil_system validity checks. We cannot use
++			fil_node_open_file() directly, as that would re-open the
++			file by name and create another file handle. */
++
++			fil_system->n_open++;
++
++			if (space->purpose == FIL_TABLESPACE &&
++			    space->id != 0) {
++
++				/* Put the node to the LRU list */
++				UT_LIST_ADD_FIRST(LRU, fil_system->LRU, node);
++			}
++		}
++
++		mutex_exit(&fil_system->mutex);
++	} else {
++
++		os_file_close(file);
++	}
+ 	ut_free(buf2);
+ 	mem_free(filepath);
+ }
+@@ -4247,7 +4393,7 @@
  idea is to read as much good data as we can and jump over bad data.
  @return 0 if ok, -1 if error even after the retries, 1 if at the end
  of the directory */
@@ -366,7 +419,7 @@
  int
  fil_file_readdir_next_file(
  /*=======================*/
-@@ -4291,7 +4393,7 @@
+@@ -4291,7 +4437,7 @@
  @return	DB_SUCCESS or error number */
  UNIV_INTERN
  ulint
@@ -375,7 +428,7 @@
  /*===================================*/
  {
  	int		ret;
-@@ -4347,7 +4449,9 @@
+@@ -4347,7 +4493,9 @@
  			dbinfo.name);
  		srv_normalize_path_for_win(dbpath);
  
@@ -386,7 +439,7 @@
  
  		if (dbdir != NULL) {
  			/* printf("Opened dir %s\n", dbinfo.name); */
-@@ -4373,8 +4477,11 @@
+@@ -4373,8 +4521,11 @@
  						   ".ibd")) {
  					/* The name ends in .ibd; try opening
  					the file */
@@ -399,7 +452,7 @@
  				}
  next_file_item:
  				ret = fil_file_readdir_next_file(&err,
-@@ -4546,15 +4653,97 @@
+@@ -4546,15 +4697,97 @@
  				"InnoDB: in InnoDB data dictionary"
  				" has tablespace id %lu,\n"
  				"InnoDB: but tablespace with that id"
@@ -505,7 +558,7 @@
  		} else {
  			ut_print_timestamp(stderr);
  			fputs("  InnoDB: Error: table ", stderr);
-@@ -4953,7 +5142,7 @@
+@@ -4953,7 +5186,7 @@
  off the LRU list if it is in the LRU list. The caller must hold the fil_sys
  mutex. */
  static
@@ -514,7 +567,7 @@
  fil_node_prepare_for_io(
  /*====================*/
  	fil_node_t*	node,	/*!< in: file node */
-@@ -4973,10 +5162,13 @@
+@@ -4973,10 +5206,13 @@
  	}
  
  	if (node->open == FALSE) {
@@ -529,7 +582,7 @@
  	}
  
  	if (node->n_pending == 0 && space->purpose == FIL_TABLESPACE
-@@ -4989,6 +5181,8 @@
+@@ -4989,6 +5225,8 @@
  	}
  
  	node->n_pending++;
@@ -538,7 +591,7 @@
  }
  
  /********************************************************************//**
-@@ -5125,7 +5319,9 @@
+@@ -5125,7 +5363,9 @@
  	ut_ad(recv_no_ibuf_operations || (type == OS_FILE_WRITE)
  	      || !ibuf_bitmap_page(zip_size, block_offset)
  	      || sync || is_log);
@@ -549,7 +602,7 @@
  	      || ibuf_page(space_id, zip_size, block_offset, NULL));
  # endif /* UNIV_LOG_DEBUG */
  	if (sync) {
-@@ -5190,6 +5386,16 @@
+@@ -5190,6 +5430,16 @@
  
  	ut_ad((mode != OS_AIO_IBUF) || (space->purpose == FIL_TABLESPACE));
  
@@ -681,13 +734,14 @@
  	return(buf);
 --- a/storage/innodb_plugin/include/srv0srv.h
 +++ b/storage/innodb_plugin/include/srv0srv.h
-@@ -226,6 +226,10 @@
+@@ -226,6 +226,11 @@
  
  extern ulong	srv_replication_delay;
  
 +extern ibool	srv_read_only;
 +extern ibool	srv_fake_write;
 +extern ibool	srv_apply_log_only;
++extern ibool	srv_backup_mode;
 +
  extern long long	srv_ibuf_max_size;
  extern ulint	srv_ibuf_active_contract;
@@ -876,7 +930,35 @@
  
  			ib_uint64_t	end_lsn;
  
-@@ -2369,7 +2377,7 @@
+@@ -1830,6 +1838,18 @@
+ 			ulint	zip_size = fil_space_get_zip_size(space);
+ 			ulint	page_no = recv_addr->page_no;
+ 
++			/* By now we have replayed all DDL log records from the
++			current batch. Check if the space ID is still valid in
++			the entry being processed, and ignore it if it is not.*/
++			if (fil_tablespace_deleted_or_being_deleted_in_mem(space, -1)) {
++
++				ut_a(recv_sys->n_addrs);
++
++				recv_addr->state = RECV_PROCESSED;
++				recv_sys->n_addrs--;
++
++				goto next;
++			}
+ 			if (recv_addr->state == RECV_NOT_PROCESSED) {
+ 				if (!has_printed) {
+ 					ut_print_timestamp(stderr);
+@@ -1863,7 +1883,7 @@
+ 
+ 				mutex_enter(&(recv_sys->mutex));
+ 			}
+-
++next:
+ 			recv_addr = HASH_GET_NEXT(addr_hash, recv_addr);
+ 		}
+ 
+@@ -2369,7 +2389,7 @@
  			   || type == MLOG_FILE_RENAME
  			   || type == MLOG_FILE_DELETE) {
  			ut_a(space);
@@ -885,7 +967,7 @@
  			if (recv_replay_file_ops) {
  
  				/* In ibbackup --apply-log, replay an .ibd file
-@@ -2392,7 +2400,7 @@
+@@ -2392,7 +2412,7 @@
  					ut_error;
  				}
  			}
@@ -894,7 +976,7 @@
  			/* In normal mysqld crash recovery we do not try to
  			replay file operations */
  #ifdef UNIV_LOG_LSN_DEBUG
-@@ -2809,8 +2817,11 @@
+@@ -2809,8 +2829,11 @@
  
  			fprintf(stderr,
  				"InnoDB: Doing recovery: scanned up to"
@@ -908,7 +990,7 @@
  		}
  	}
  
-@@ -2913,7 +2916,7 @@
+@@ -2913,7 +2936,7 @@
  		"InnoDB: Reading tablespace information"
  		" from the .ibd files...\n");
  
@@ -917,7 +999,7 @@
  
  	/* If we are using the doublewrite method, we will
  	check if there are half-written pages in data files,
-@@ -2922,12 +2933,14 @@
+@@ -2922,12 +2945,14 @@
  
  	if (srv_force_recovery < SRV_FORCE_NO_LOG_REDO) {
  
@@ -933,7 +1015,7 @@
  	}
  }
  
-@@ -3095,6 +3108,7 @@
+@@ -3095,6 +3120,7 @@
  		recv_sys->recovered_lsn = checkpoint_lsn;
  
  		srv_start_lsn = checkpoint_lsn;
@@ -941,7 +1023,7 @@
  	}
  
  	contiguous_lsn = ut_uint64_align_down(recv_sys->scanned_lsn,
-@@ -3453,6 +3467,7 @@
+@@ -3453,6 +3479,7 @@
  	that the data dictionary tables will be free of any locks.
  	The data dictionary latch should guarantee that there is at
  	most one data dictionary transaction active at a time. */
@@ -1038,18 +1120,19 @@
  /* prototypes for new functions added to ha_innodb.cc */
  ibool	innobase_get_slow_log();
  
-@@ -415,6 +410,10 @@
+@@ -415,6 +410,11 @@
  
  UNIV_INTERN ulong	srv_replication_delay		= 0;
  
 +UNIV_INTERN ibool	srv_read_only	= FALSE;
 +UNIV_INTERN ibool	srv_fake_write	= FALSE;
 +UNIV_INTERN ibool	srv_apply_log_only = FALSE;
++UNIV_INTERN ibool	srv_backup_mode = FALSE;
 +
  UNIV_INTERN long long	srv_ibuf_max_size = 0;
  UNIV_INTERN ulint	srv_ibuf_active_contract = 0; /* 0:disable 1:enable */
  UNIV_INTERN ulint	srv_ibuf_accel_rate = 100;
-@@ -1065,7 +1064,7 @@
+@@ -1065,7 +1065,7 @@
  	}
  
  	/* Initialize some INFORMATION SCHEMA internal structures */
@@ -1058,7 +1141,7 @@
  }
  
  /*********************************************************************//**
-@@ -1076,6 +1075,7 @@
+@@ -1076,6 +1076,7 @@
  /*==========*/
  {
  	os_fast_mutex_free(&srv_conc_mutex);
@@ -1066,7 +1149,7 @@
  	mem_free(srv_conc_slots);
  	srv_conc_slots = NULL;
  
-@@ -1089,6 +1089,7 @@
+@@ -1089,6 +1090,7 @@
  	srv_mysql_table = NULL;
  
  	trx_i_s_cache_free(trx_i_s_cache);
@@ -1074,7 +1157,7 @@
  }
  
  /*********************************************************************//**
-@@ -2570,36 +2571,6 @@
+@@ -2570,36 +2572,6 @@
  		old_sema = sema;
  	}
  
@@ -1163,7 +1246,6 @@
  
  	if (srv_auto_extend_last_data_file
  	    && sum_of_data_file_sizes < tablespace_size_in_header) {
--
 +		/* extend table space size aligning with header */
 +		ulint	actual_size;
 +		fil_extend_space_to_desired_size(&actual_size, 0, tablespace_size_in_header);
@@ -1174,7 +1256,7 @@
 +				(ulong) actual_size,
 +				(ulong) tablespace_size_in_header);
 +		}
-+
+ 
 +#ifdef UNDEFINED
  		fprintf(stderr,
  			"InnoDB: Error: tablespace size stored in header"
@@ -1497,3 +1579,14 @@
  
  	UNIV_MEM_FREE(buf, n);
  }
+--- a/configure.in
++++ b/configure.in
+@@ -22,7 +22,7 @@
+ AM_INIT_AUTOMAKE([1.9 tar-ustar])
+ AC_PROG_LIBTOOL
+ 
+-AM_CONFIG_HEADER([include/config.h])
++AC_CONFIG_HEADERS([include/config.h])
+ 
+ # Request support for automake silent-rules if available.
+ # Default to verbose output. One can use the configure-time

=== modified file 'patches/xtradb55.patch'
--- patches/xtradb55.patch	2013-04-16 14:34:55 +0000
+++ patches/xtradb55.patch	2013-05-08 05:18:19 +0000
@@ -356,7 +356,60 @@
  	if (size < FIL_IBD_FILE_INITIAL_SIZE * (lint)UNIV_PAGE_SIZE) {
  		fprintf(stderr,
  			"InnoDB: Error: the size of single-table tablespace"
-@@ -4299,7 +4402,7 @@
+@@ -4288,7 +4391,51 @@
+ 
+ 	fil_node_create(filepath, 0, space_id, FALSE);
+ func_exit:
+-	os_file_close(file);
++	/* We reuse file handles on the backup stage in XtraBackup to avoid
++	inconsistencies between the file name and the actual tablespace contents
++	if a DDL occurs between a fil_load_single_table_tablespaces() call and
++	the actual copy operation. */
++	if (srv_backup_mode) {
++
++		fil_node_t*	node;
++		fil_space_t*	space;
++
++		mutex_enter(&fil_system->mutex);
++
++		space = fil_space_get_by_id(space_id);
++
++		if (space) {
++			node = UT_LIST_GET_LAST(space->chain);
++
++			/* The handle will be closed by xtrabackup in
++			xtrabackup_copy_datafile(). We set node->open to TRUE to
++			make sure no one calls fil_node_open_file()
++			(i.e. attempts to reopen the tablespace by name) during
++			the backup stage. */
++
++			node->open = TRUE;
++			node->handle = file;
++
++			/* The following is copied from fil_node_open_file() to
++			pass fil_system validity checks. We cannot use
++			fil_node_open_file() directly, as that would re-open the
++			file by name and create another file handle. */
++
++			fil_system->n_open++;
++
++			if (space->purpose == FIL_TABLESPACE &&
++			    space->id != 0) {
++
++				/* Put the node to the LRU list */
++				UT_LIST_ADD_FIRST(LRU, fil_system->LRU, node);
++			}
++		}
++
++		mutex_exit(&fil_system->mutex);
++	} else {
++
++		os_file_close(file);
++	}
+ 	ut_free(buf2);
+ 	mem_free(filepath);
+ }
+@@ -4299,7 +4446,7 @@
  idea is to read as much good data as we can and jump over bad data.
  @return 0 if ok, -1 if error even after the retries, 1 if at the end
  of the directory */
@@ -365,7 +418,7 @@
  int
  fil_file_readdir_next_file(
  /*=======================*/
-@@ -4343,7 +4446,7 @@
+@@ -4343,7 +4490,7 @@
  @return	DB_SUCCESS or error number */
  UNIV_INTERN
  ulint
@@ -374,7 +427,7 @@
  /*===================================*/
  {
  	int		ret;
-@@ -4399,7 +4502,9 @@
+@@ -4399,7 +4546,9 @@
  			dbinfo.name);
  		srv_normalize_path_for_win(dbpath);
  
@@ -385,7 +438,7 @@
  
  		if (dbdir != NULL) {
  			/* printf("Opened dir %s\n", dbinfo.name); */
-@@ -4425,8 +4530,11 @@
+@@ -4425,8 +4574,11 @@
  						   ".ibd")) {
  					/* The name ends in .ibd; try opening
  					the file */
@@ -398,7 +451,7 @@
  				}
  next_file_item:
  				ret = fil_file_readdir_next_file(&err,
-@@ -4598,15 +4706,97 @@
+@@ -4598,15 +4750,97 @@
  				"InnoDB: in InnoDB data dictionary"
  				" has tablespace id %lu,\n"
  				"InnoDB: but tablespace with that id"
@@ -504,7 +557,7 @@
  		} else {
  			ut_print_timestamp(stderr);
  			fputs("  InnoDB: Error: table ", stderr);
-@@ -5005,7 +5195,7 @@
+@@ -5005,7 +5239,7 @@
  off the LRU list if it is in the LRU list. The caller must hold the fil_sys
  mutex. */
  static
@@ -513,7 +566,7 @@
  fil_node_prepare_for_io(
  /*====================*/
  	fil_node_t*	node,	/*!< in: file node */
-@@ -5025,10 +5215,13 @@
+@@ -5025,10 +5259,13 @@
  	}
  
  	if (node->open == FALSE) {
@@ -528,7 +581,7 @@
  	}
  
  	if (node->n_pending == 0 && space->purpose == FIL_TABLESPACE
-@@ -5041,6 +5234,8 @@
+@@ -5041,6 +5278,8 @@
  	}
  
  	node->n_pending++;
@@ -537,7 +590,7 @@
  }
  
  /********************************************************************//**
-@@ -5240,6 +5435,16 @@
+@@ -5240,6 +5479,16 @@
  
  	ut_ad((mode != OS_AIO_IBUF) || (space->purpose == FIL_TABLESPACE));
  
@@ -691,7 +744,7 @@
  extern char	srv_adaptive_flushing;
  
  
-@@ -247,6 +246,10 @@
+@@ -247,6 +246,11 @@
  extern ulint	srv_dict_size_limit;
  
  extern ulint	srv_lazy_drop_table;
@@ -699,6 +752,7 @@
 +extern ibool	srv_read_only;
 +extern ibool	srv_fake_write;
 +extern ibool	srv_apply_log_only;
++extern ibool	srv_backup_mode;
  /*-------------------------------------------*/
  
  extern ulint	srv_n_rows_inserted;
@@ -840,7 +894,35 @@
  
  			ib_uint64_t	end_lsn;
  
-@@ -2380,7 +2388,7 @@
+@@ -1842,6 +1850,18 @@
+ 			ulint	zip_size = fil_space_get_zip_size(space);
+ 			ulint	page_no = recv_addr->page_no;
+ 
++			/* By now we have replayed all DDL log records from the
++			current batch. Check if the space ID is still valid in
++			the entry being processed, and ignore it if it is not.*/
++			if (fil_tablespace_deleted_or_being_deleted_in_mem(space, -1)) {
++
++				ut_a(recv_sys->n_addrs);
++
++				recv_addr->state = RECV_PROCESSED;
++				recv_sys->n_addrs--;
++
++				goto next;
++			}
+ 			if (recv_addr->state == RECV_NOT_PROCESSED) {
+ 				if (!has_printed) {
+ 					ut_print_timestamp(stderr);
+@@ -1875,7 +1895,7 @@
+ 
+ 				mutex_enter(&(recv_sys->mutex));
+ 			}
+-
++next:
+ 			recv_addr = HASH_GET_NEXT(addr_hash, recv_addr);
+ 		}
+ 
+@@ -2380,7 +2400,7 @@
  			   || type == MLOG_FILE_RENAME
  			   || type == MLOG_FILE_DELETE) {
  			ut_a(space);
@@ -849,7 +931,7 @@
  			if (recv_replay_file_ops) {
  
  				/* In ibbackup --apply-log, replay an .ibd file
-@@ -2403,7 +2411,7 @@
+@@ -2403,7 +2423,7 @@
  					ut_error;
  				}
  			}
@@ -858,7 +940,7 @@
  			/* In normal mysqld crash recovery we do not try to
  			replay file operations */
  #ifdef UNIV_LOG_LSN_DEBUG
-@@ -2820,8 +2828,11 @@
+@@ -2820,8 +2840,11 @@
  
  			fprintf(stderr,
  				"InnoDB: Doing recovery: scanned up to"
@@ -872,7 +954,7 @@
  		}
  	}
  
-@@ -2926,7 +2929,7 @@
+@@ -2926,7 +2949,7 @@
  		"InnoDB: Reading tablespace information"
  		" from the .ibd files...\n");
  
@@ -881,7 +963,7 @@
  
  	/* If we are using the doublewrite method, we will
  	check if there are half-written pages in data files,
-@@ -2935,12 +2946,14 @@
+@@ -2935,12 +2958,14 @@
  
  	if (srv_force_recovery < SRV_FORCE_NO_LOG_REDO) {
  
@@ -897,7 +979,7 @@
  	}
  }
  
-@@ -3109,6 +3122,7 @@
+@@ -3109,6 +3134,7 @@
  		recv_sys->recovered_lsn = checkpoint_lsn;
  
  		srv_start_lsn = checkpoint_lsn;
@@ -905,7 +987,7 @@
  	}
  
  	contiguous_lsn = ut_uint64_align_down(recv_sys->scanned_lsn,
-@@ -3474,6 +3488,7 @@
+@@ -3474,6 +3500,7 @@
  	that the data dictionary tables will be free of any locks.
  	The data dictionary latch should guarantee that there is at
  	most one data dictionary transaction active at a time. */
@@ -1029,7 +1111,7 @@
  
  /* Try to flush dirty pages so as to avoid IO bursts at
  the checkpoints. */
-@@ -457,6 +451,10 @@
+@@ -457,6 +451,11 @@
  UNIV_INTERN ulint	srv_dict_size_limit = 0;
  
  UNIV_INTERN ulint	srv_lazy_drop_table = 0;
@@ -1037,10 +1119,11 @@
 +UNIV_INTERN ibool	srv_read_only   = FALSE;
 +UNIV_INTERN ibool	srv_fake_write  = FALSE;
 +UNIV_INTERN ibool	srv_apply_log_only = FALSE;
++UNIV_INTERN ibool	srv_backup_mode = FALSE;
  /*-------------------------------------------*/
  UNIV_INTERN ulong	srv_n_spin_wait_rounds	= 30;
  UNIV_INTERN ulong	srv_n_free_tickets_to_enter = 500;
-@@ -1137,7 +1135,7 @@
+@@ -1137,7 +1136,7 @@
  	}
  
  	/* Initialize some INFORMATION SCHEMA internal structures */
@@ -1049,7 +1132,7 @@
  }
  
  /*********************************************************************//**
-@@ -1148,6 +1146,7 @@
+@@ -1148,6 +1147,7 @@
  /*==========*/
  {
  	os_fast_mutex_free(&srv_conc_mutex);
@@ -1057,7 +1140,7 @@
  	mem_free(srv_conc_slots);
  	srv_conc_slots = NULL;
  
-@@ -1161,6 +1160,7 @@
+@@ -1161,6 +1161,7 @@
  	srv_mysql_table = NULL;
  
  	trx_i_s_cache_free(trx_i_s_cache);
@@ -1065,7 +1148,7 @@
  }
  
  /*********************************************************************//**
-@@ -1886,7 +1886,7 @@
+@@ -1886,7 +1887,7 @@
  		}
  
  		/* Record the lock wait time for this thread */
@@ -1074,7 +1157,7 @@
  	}
  
  	if (trx->was_chosen_as_deadlock_victim) {
-@@ -2842,36 +2842,6 @@
+@@ -2842,36 +2843,6 @@
  		old_sema = sema;
  	}
  

=== modified file 'src/fil_cur.cc'
--- src/fil_cur.cc	2013-04-29 06:43:29 +0000
+++ src/fil_cur.cc	2013-05-08 05:18:19 +0000
@@ -74,6 +74,55 @@
 
 }
 
+/**********************************************************************//**
+Closes a file. */
+static
+void
+xb_fil_node_close_file(
+/*===================*/
+	fil_node_t*	node)	/*!< in: file node */
+{
+	ibool	ret;
+
+	mutex_enter(&fil_system->mutex);
+
+	ut_ad(node);
+	ut_a(node->n_pending == 0);
+	ut_a(node->n_pending_flushes == 0);
+#if MYSQL_VERSION_ID >= 50600
+	ut_a(!node->being_extended);
+#endif
+
+	if (!node->open) {
+
+		mutex_exit(&fil_system->mutex);
+
+		return;
+	}
+
+	ret = os_file_close(node->handle);
+	ut_a(ret);
+
+	node->open = FALSE;
+
+	ut_a(fil_system->n_open > 0);
+	fil_system->n_open--;
+#if MYSQL_VERSION_ID >= 50600
+	fil_n_file_opened--;
+#endif
+
+	if (node->space->purpose == FIL_TABLESPACE &&
+	    !trx_sys_sys_space(node->space->id)) {
+
+		ut_a(UT_LIST_GET_LEN(fil_system->LRU) > 0);
+
+		/* The node is in the LRU list, remove it */
+		UT_LIST_REMOVE(LRU, fil_system->LRU, node);
+	}
+
+	mutex_exit(&fil_system->mutex);
+}
+
 /************************************************************************
 Open a source file cursor and initialize the associated read filter.
 
@@ -90,11 +139,12 @@
 	ulint	page_size;
 	ulint	page_size_shift;
 	ulint	zip_size;
+	ibool	success;
 
 	/* Initialize these first so xb_fil_cur_close() handles them correctly
 	in case of error */
 	cursor->orig_buf = NULL;
-	cursor->file = XB_FILE_UNDEFINED;
+	cursor->node = NULL;
 
 	cursor->space_id = node->space->id;
 	cursor->is_system = trx_sys_sys_space(node->space->id);
@@ -109,31 +159,56 @@
 		xb_get_relative_path(cursor->abs_path, cursor->is_system),
 		sizeof(cursor->rel_path));
 
-	/* Open the file */
-
-	if (my_stat(node->name, &cursor->statinfo, MYF(MY_WME)) == NULL) {
-		msg("[%02u] xtrabackup: Warning: cannot stat %s\n",
-		    thread_n, node->name);
-		return(XB_FIL_CUR_SKIP);
-	}
-
-	ibool success;
-
-	cursor->file =
-		xb_file_create_no_error_handling(node->name,
+	/* In the backup mode we should already have a tablespace handle created
+	by fil_load_single_table_tablespace() unless it is a system
+	tablespace. Otherwise we open the file here. */
+	if (cursor->is_system || !srv_backup_mode) {
+		node->handle =
+			xb_file_create_no_error_handling(node->name,
 						 OS_FILE_OPEN,
 						 OS_FILE_READ_ONLY,
 						 &success);
-	if (!success) {
-		/* The following call prints an error message */
-		os_file_get_last_error(TRUE);
-
-		msg("[%02u] xtrabackup: Warning: cannot open %s\n"
-		    "[%02u] xtrabackup: Warning: We assume the "
-		    "table was dropped or renamed during "
-		    "xtrabackup execution and ignore the file.\n",
-		    thread_n, node->name, thread_n);
-		return(XB_FIL_CUR_SKIP);
+		if (!success) {
+			/* The following call prints an error message */
+			os_file_get_last_error(TRUE);
+
+			msg("[%02u] xtrabackup: error: cannot open "
+			    "tablespace %s\n",
+			    thread_n, cursor->abs_path);
+
+			return(XB_FIL_CUR_ERROR);
+		}
+		mutex_enter(&fil_system->mutex);
+
+		node->open = TRUE;
+
+		fil_system->n_open++;
+#if MYSQL_VERSION_ID >= 50600
+		fil_n_file_opened++;
+#endif
+
+		if (node->space->purpose == FIL_TABLESPACE &&
+		    !trx_sys_sys_space(node->space->id)) {
+
+			/* Put the node to the LRU list */
+			UT_LIST_ADD_FIRST(LRU, fil_system->LRU, node);
+		}
+
+		mutex_exit(&fil_system->mutex);
+	}
+
+	ut_ad(node->open);
+
+	cursor->node = node;
+	cursor->file = node->handle;
+
+	if (my_fstat(cursor->file, &cursor->statinfo, MYF(MY_WME))) {
+		msg("[%02u] xtrabackup: error: cannot stat %s\n",
+		    thread_n, cursor->abs_path);
+
+		xb_fil_cur_close(cursor);
+
+		return(XB_FIL_CUR_ERROR);
 	}
 
 	xb_file_set_nocache(cursor->file, node->name, "OPEN");
@@ -142,7 +217,7 @@
 	/* Determine the page size */
 	zip_size = xb_get_zip_size(cursor->file);
 	if (zip_size == ULINT_UNDEFINED) {
-		os_file_close(cursor->file);
+		xb_fil_cur_close(cursor);
 		return(XB_FIL_CUR_SKIP);
 	} else if (zip_size) {
 		page_size = zip_size;
@@ -230,7 +305,8 @@
 	cursor->buf_offset = offset;
 	cursor->buf_page_no = (ulint) (offset >> cursor->page_size_shift);
 
-	success = xb_os_file_read(cursor->file, cursor->buf, offset, to_read);
+	success = xb_os_file_read(cursor->file, cursor->buf, offset,
+				  to_read);
 	if (!success) {
 		return(XB_FIL_CUR_ERROR);
 	}
@@ -294,7 +370,8 @@
 	if (cursor->orig_buf != NULL) {
 		ut_free(cursor->orig_buf);
 	}
-	if (cursor->file != XB_FILE_UNDEFINED) {
-		os_file_close(cursor->file);
+	if (cursor->node != NULL) {
+		xb_fil_node_close_file(cursor->node);
+		cursor->file = XB_FILE_UNDEFINED;
 	}
 }

=== modified file 'src/fil_cur.h'
--- src/fil_cur.h	2013-04-23 07:44:24 +0000
+++ src/fil_cur.h	2013-05-08 05:18:19 +0000
@@ -31,6 +31,7 @@
 
 struct xb_fil_cur_t {
 	os_file_t	file;		/*!< source file handle */
+	fil_node_t*	node;		/*!< source tablespace node */
 	char		rel_path[FN_REFLEN];
 					/*!< normalized file path */
 	char		abs_path[FN_REFLEN];

=== modified file 'src/xtrabackup.cc'
--- src/xtrabackup.cc	2013-05-07 13:28:22 +0000
+++ src/xtrabackup.cc	2013-05-08 05:18:19 +0000
@@ -1813,6 +1813,55 @@
 	return(zip_size);
 }
 
+/**********************************************************************//**
+Closes a file. */
+static
+void
+xb_fil_node_close_file(
+/*===================*/
+	fil_node_t*	node)	/*!< in: file node */
+{
+	ibool	ret;
+
+	mutex_enter(&fil_system->mutex);
+
+	ut_ad(node);
+	ut_a(node->n_pending == 0);
+	ut_a(node->n_pending_flushes == 0);
+#if MYSQL_VERSION_ID >= 50600
+	ut_a(!node->being_extended);
+#endif
+
+	if (!node->open) {
+
+		mutex_exit(&fil_system->mutex);
+
+		return;
+	}
+
+	ret = os_file_close(node->handle);
+	ut_a(ret);
+
+	node->open = FALSE;
+
+	ut_a(fil_system->n_open > 0);
+	fil_system->n_open--;
+#if MYSQL_VERSION_ID >= 50600
+	fil_n_file_opened--;
+#endif
+
+	if (node->space->purpose == FIL_TABLESPACE &&
+	    !trx_sys_sys_space(node->space->id)) {
+
+		ut_a(UT_LIST_GET_LEN(fil_system->LRU) > 0);
+
+		/* The node is in the LRU list, remove it */
+		UT_LIST_REMOVE(LRU, fil_system->LRU, node);
+	}
+
+	mutex_exit(&fil_system->mutex);
+}
+
 /* TODO: We may tune the behavior (e.g. by fil_aio)*/
 
 static
@@ -2497,8 +2546,7 @@
 		    srv_n_write_io_threads,
 		    SRV_MAX_N_PENDING_SYNC_IOS);
 
-	fil_init(srv_file_per_table ? 50000 : 5000,
-		 srv_max_n_open_files);
+	fil_init(srv_file_per_table ? 50000 : 5000, LONG_MAX);
 
 	fsp_init();
 
@@ -2541,11 +2589,6 @@
 		return(DB_ERROR);
 	}
 
-	err = fil_load_single_table_tablespaces(xb_check_if_open_tablespace);
-	if (err != DB_SUCCESS) {
-		return(err);
-	}
-
 #if MYSQL_VERSION_ID >= 50600
 	/* Add separate undo tablespaces to fil_system */
 
@@ -2558,6 +2601,15 @@
 	}
 #endif
 
+	/* It is important to call fil_load_single_table_tablespace() after
+	srv_undo_tablespaces_init(), because fil_is_user_tablespace_id() *
+	relies on srv_undo_tablespaces_open to be properly initialized */
+
+	err = fil_load_single_table_tablespaces(xb_check_if_open_tablespace);
+	if (err != DB_SUCCESS) {
+		return(err);
+	}
+
 	return(DB_SUCCESS);
 }
 
@@ -2851,6 +2903,8 @@
 
 	xb_set_innodb_read_only();
 
+	srv_backup_mode = TRUE;
+
 	/* initialize components */
         if(innodb_init_param())
                 exit(EXIT_FAILURE);

=== renamed file 'test/t/bug722638.sh' => 'test/t/ddl.sh'
--- test/t/bug722638.sh	2012-06-04 18:29:48 +0000
+++ test/t/ddl.sh	2013-05-08 05:18:19 +0000
@@ -1,7 +1,9 @@
-########################################################################
-# Bug #722638: xtrabackup: taking backup while tables are droped and 
+############################################################################
+# Bug #722638: xtrabackup: taking backup while tables are droped and
 #              created breaks backup
-########################################################################
+#
+# Bug #1079700: Issues with renaming/rotating tables during the backup stage
+############################################################################
 
 . inc/common.sh
 
@@ -10,7 +12,11 @@
     exit $SKIPPED_EXIT_CODE
 fi
 
-start_server --innodb_file_per_table
+MYSQLD_EXTRA_MY_CNF_OPTS="
+innodb_file_per_table
+"
+
+start_server
 
 run_cmd $MYSQL $MYSQL_ARGS test <<EOF
 
@@ -26,8 +32,19 @@
 CREATE TABLE t4_old(a INT) ENGINE=InnoDB;
 INSERT INTO t4_old VALUES (1), (2), (3);
 
+CREATE TABLE t5(a INT) ENGINE=InnoDB;
+INSERT INTO t5 VALUES (1), (2), (3);
+
+CREATE TABLE t6(c CHAR(1)) ENGINE=InnoDB;
+INSERT INTO t6 VALUES ('a'), ('b'), ('c');
+
 EOF
 
+# Make a checkpoint so that original tablespace creation events are not in the
+# xtrabackup log
+shutdown_server
+start_server
+
 mkdir -p $topdir/backup
 
 # Backup
@@ -70,13 +87,18 @@
 ALTER TABLE t4_old RENAME t4;
 INSERT INTO t4 VALUES (7), (8), (9);
 
+INSERT INTO t5 VALUES (4), (5), (6);
+INSERT INTO t6 VALUES ('d'), ('e'), ('f');
+
+# Rotate tables t5 and t6
+RENAME TABLE t5 TO temp, t6 TO t5, temp TO t6;
+
+INSERT INTO t5 VALUES ('g'), ('h'), ('i');
+INSERT INTO t6 VALUES (7), (8), (9);
+
 EOF
 
-# Calculate checksums
-checksum_t1=`checksum_table test t1`
-checksum_t2=`checksum_table test t2`
-checksum_t3=`checksum_table test t3`
-checksum_t4=`checksum_table test t4`
+record_db_state test
 
 # Resume xtrabackup
 vlog "Resuming xtrabackup"
@@ -84,6 +106,8 @@
 
 run_cmd wait $job_pid
 
+# exit 1
+
 # Prepare
 xtrabackup --datadir=$mysql_datadir --prepare --target-dir=$topdir/backup
 
@@ -94,25 +118,7 @@
     $mysql_datadir/test/*.ibd
 cp -r $topdir/backup/* $mysql_datadir
 
-start_server --innodb_file_per_table
-
-# Verify checksums
-checksum_t1_new=`checksum_table test t1`
-checksum_t2_new=`checksum_table test t2`
-checksum_t3_new=`checksum_table test t3`
-checksum_t4_new=`checksum_table test t4`
-vlog "Checksums (old/new):"
-vlog "t1: $checksum_t1/$checksum_t1_new"
-vlog "t2: $checksum_t2/$checksum_t2_new"
-vlog "t3: $checksum_t3/$checksum_t3_new"
-vlog "t4: $checksum_t4/$checksum_t4_new"
-
-if [ "$checksum_t1" = "$checksum_t1_new" -a \
-     "$checksum_t2" = "$checksum_t2_new" -a \
-     "$checksum_t3" = "$checksum_t3_new" -a \
-     "$checksum_t4" = "$checksum_t4_new" ]; then
-    exit 0
-fi
-
-vlog "Checksums do not match"
-exit -1
+start_server
+
+# Verify backup
+verify_db_state test