Merge lp:~sergei.glushchenko/percona-xtrabackup/buffer_pool_dump into lp:percona-xtrabackup/2.0

[Sergei Glushchenko]

Support for buffer pool dump MySQL 5.6

   - MySQL 5.6 adds an ability to dump the buffer pool into file for it to be
     possible to load it into memory at startup or on demand. The location of
     buffer pool file is set by global variable innodb_buffer_pool_filename
     and is relative to MySQL data directory.
   - The patch adds an ability to take a backup of buffer pool dump. It has been
     made exactly the same way as XtraDB LRU dump is backed up.
   - Two tests have been added for this feature. One takes regular backup and
     after takes streaming backup. Another one takes backup with --remote-host
     option. Second test has not been ported to 2.1 because --remote-host
     option is not supported in 2.1.
   - Also test for these features to work correctly with incremental backups
     is implemented.

http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.0-param/419/

[Alexey Kopytov (akopytov)]

Sergei,

  - string values for most options in --print-param are enclosed in
    double quotes. It only matters when path or file names contain
    spaces.
  - MP and revision comments refer to a --remote-host test, but I don't
    see it in the code (would also be rather tricky to test in the test
    suite). I thought it was a typo for --rsync, but the rsync test is
    also present in the 2.1 MP as it should be.
  - 5.6 version checks are different in test case. I suggest to use
    "${MYSQL_VERSION:0:3}" != "5.6", the other one looks a bit
    unreliable
  - please rebase it on current trunks. my.cnf is handled differently
    after recent merges, and the way it is done in these tests will
    conflict. You can now specify MYSQLD_EXTRA_MY_CNF_OPTS and it will
    be used by both server (no need for extra start_server arguments)
    and innobackupex/xtrabackup

review: Needs Fixing

[Sergei Glushchenko (sergei.glushchenko)]

Alexei,

  - string values for most options in --print-param are enclosed in
    double quotes. It only matters when path or file names contain
    spaces.

> I added quotes

  - MP and revision comments refer to a --remote-host test, but I don't
    see it in the code (would also be rather tricky to test in the test
    suite). I thought it was a typo for --rsync, but the rsync test is
    also present in the 2.1 MP as it should be.

> Indeed, I mixed many things in one. And also forgot to add test for incremental backup.

  - 5.6 version checks are different in test case. I suggest to use
    "${MYSQL_VERSION:0:3}" != "5.6", the other one looks a bit
    unreliable

> I modified version check to be comparison with "5.6"

  - please rebase it on current trunks. my.cnf is handled differently
    after recent merges, and the way it is done in these tests will
    conflict. You can now specify MYSQLD_EXTRA_MY_CNF_OPTS and it will
    be used by both server (no need for extra start_server arguments)
    and innobackupex/xtrabackup

> done.

[Sergei Glushchenko (sergei.glushchenko)]

Alexey, sorry for typo in name.

[Alexey Kopytov (akopytov)]

review: Approve

=== modified file 'innobackupex'
--- innobackupex	2013-04-25 08:27:18 +0000
+++ innobackupex	2013-04-25 14:34:24 +0000
@@ -392,6 +392,8 @@
 #
 sub backup {
     my $orig_datadir = get_option(\%config, $option_defaults_group, 'datadir');
+    my $buffer_pool_filename = get_option(\%config, $option_defaults_group,
+                                          'innodb_buffer_pool_filename');
 
     # check that we can connect to the database. This done by
     # connecting, issuing a query, and closing the connection.
@@ -447,21 +449,24 @@
     # Close the DB connection
     mysql_close();
 
-    # copy ib_lru_dump
-    if (-e "$orig_datadir/ib_lru_dump") {
-        if ($option_remote_host) {
-            print STDERR "$prefix Backing up file 'ib_lru_dump'\n";
-            system("scp $option_scp_opt '$orig_datadir/ib_lru_dump' '$option_remote_host:$backup_dir/ib_lru_dump'")
-                and Die "Failed to scp file 'ib_lru_dump': $!";
-        } elsif ($option_stream) {
-            print STDERR "$prefix Backing up as tar stream 'ib_lru_dump'\n";
-            system("cd $orig_datadir; $stream_cmd ib_lru_dump")
-                and Die "Failed to stream 'ib_lru_dump': $!";
-        } elsif (!$option_rsync) {
-            my $src_name = escape_path("$orig_datadir/ib_lru_dump");
-            my $dst_name = escape_path("$backup_dir/ib_lru_dump");
-            system("$CP_CMD \"$src_name\" \"$dst_name\"")
-                and Die "Failed to copy file 'ib_lru_dump': $!";
+    # Copy buffer poll dump and/or LRU dump
+    foreach my $dump_name ($buffer_pool_filename, 'ib_lru_dump') {
+        if (-e "$orig_datadir/$dump_name") {
+            if ($option_remote_host) {
+                print STDERR "$prefix Backing up file '$dump_name'\n";
+                system("scp $option_scp_opt '$orig_datadir/$dump_name' " .
+                       "'$option_remote_host:$backup_dir/$dump_name'")
+                    and Die "Failed to scp file '$dump_name': $!";
+            } elsif ($option_stream) {
+                print STDERR "$prefix Backing up as tar stream '$dump_name'\n";
+                system("cd $orig_datadir; $stream_cmd $dump_name")
+                    and Die "Failed to stream '$dump_name': $!";
+            } elsif (!$option_rsync) {
+                my $src_name = escape_path("$orig_datadir/$dump_name");
+                my $dst_name = escape_path("$backup_dir/$dump_name");
+                system("$CP_CMD \"$src_name\" \"$dst_name\"")
+                    and Die "Failed to copy file '$dump_name': $!";
+            }
         }
     }
 
@@ -2204,6 +2209,8 @@
 sub backup_files {
     my $prep_mode = shift;
     my $source_dir = get_option(\%config, $option_defaults_group, 'datadir');
+    my $buffer_pool_filename = get_option(\%config, $option_defaults_group,
+                                          'innodb_buffer_pool_filename');
     my @list;
     my $file;
     my $database;
@@ -2338,10 +2345,12 @@
     closedir(DIR);
 
     if ($option_rsync) {
-        if (-e "$source_dir/ib_lru_dump") {
-            print RSYNC "ib_lru_dump\n";
-            if (!$prep_mode) {
-                $rsync_files_hash{"ib_lru_dump"} = 1;
+        foreach my $dump_name ($buffer_pool_filename, 'ib_lru_dump') {
+            if (-e "$source_dir/$dump_name") {
+                print RSYNC "$dump_name\n";
+                if (!$prep_mode) {
+                    $rsync_files_hash{"$dump_name"} = 1;
+                }
             }
         }
 	close(RSYNC);

=== modified file 'src/xtrabackup.cc'
--- src/xtrabackup.cc	2013-04-24 14:09:42 +0000
+++ src/xtrabackup.cc	2013-04-25 14:34:24 +0000
@@ -1436,6 +1436,7 @@
 my_bool innobase_fast_checksum = FALSE;
 my_bool	innobase_extra_undoslots = FALSE;
 char*	innobase_doublewrite_file = NULL;
+char*	innobase_buffer_pool_filename = NULL;
 
 longlong innobase_buffer_pool_size = 8*1024*1024L;
 longlong innobase_log_file_size = DEFAULT_LOG_FILE_SIZE;
@@ -1633,6 +1634,7 @@
   OPT_INNODB_EXTRA_UNDOSLOTS,
   OPT_INNODB_DOUBLEWRITE_FILE,
 #endif
+  OPT_INNODB_BUFFER_POOL_FILENAME,
   OPT_INNODB_FORCE_RECOVERY,
   OPT_INNODB_LOCK_WAIT_TIMEOUT,
   OPT_INNODB_LOG_BUFFER_SIZE,
@@ -1961,6 +1963,11 @@
    (G_PTR*) &innobase_doublewrite_file, (G_PTR*) &innobase_doublewrite_file,
    0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0},
 #endif
+  {"innodb_buffer_pool_filename", OPT_INNODB_BUFFER_POOL_FILENAME,
+   "Filename to/from which to dump/load the InnoDB buffer pool",
+   (G_PTR*) &innobase_buffer_pool_filename,
+   (G_PTR*) &innobase_buffer_pool_filename,
+   0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0},
 
 #ifndef __WIN__
   {"debug-sync", OPT_XTRA_DEBUG_SYNC,
@@ -8040,6 +8047,10 @@
 		}
 		printf("innodb_undo_tablespaces = %lu\n", srv_undo_tablespaces);
 #endif
+		printf("innodb_buffer_pool_filename = \"%s\"\n",
+			innobase_buffer_pool_filename ?
+				innobase_buffer_pool_filename :
+				"ib_buffer_pool");
 		exit(EXIT_SUCCESS);
 	}
 

=== added file 'test/t/ib_buffer_pool.sh'
--- test/t/ib_buffer_pool.sh	1970-01-01 00:00:00 +0000
+++ test/t/ib_buffer_pool.sh	2013-04-25 14:34:24 +0000
@@ -0,0 +1,57 @@
+########################################################################
+# Blueprint: Support InnoDB buffer pool dumps in MySQL 5.6
+########################################################################
+
+. inc/common.sh
+
+if [ "${MYSQL_VERSION:0:3}" != "5.6" ]; then
+    echo "Requires MySQL 5.6" > $SKIPPED_REASON
+    exit $SKIPPED_EXIT_CODE
+fi
+
+MYSQLD_EXTRA_MY_CNF_OPTS="
+innodb_buffer_pool_filename=pool/dump
+"
+
+start_server
+
+# test presence of innodb_buffer_pool_filename
+${XB_BIN} --defaults-file=$topdir/my.cnf --print-param | \
+                            grep -q "innodb_buffer_pool_filename"
+
+mkdir $mysql_datadir/pool
+
+# produce buffer pool dump
+${MYSQL} ${MYSQL_ARGS} -e "SET GLOBAL innodb_buffer_pool_dump_now=ON;"
+
+# take a backup
+innobackupex --no-timestamp $topdir/backup
+
+if [ -f $topdir/backup/pool/dump ] ; then
+    vlog "Buffer pool dump has been backed up"
+else
+    vlog "Buffer pool dump has not been backed up"
+    exit -1
+fi
+
+# take streaming backup
+mkdir -p $topdir/backup
+innobackupex --stream=tar $topdir/backup > $topdir/backup/stream.tar
+
+if $TAR itf $topdir/backup/stream.tar | grep 'pool/dump' ; then
+    vlog "Buffer pool dump has been restored"
+else
+    vlog "Buffer pool dump has not been restored"
+    exit -1
+fi
+
+# restore from backup
+rm -rf $mysql_datadir/*
+innobackupex --copy-back $topdir/backup
+
+if [ -f $mysql_datadir/pool/dump ] ; then
+    vlog "Buffer pool dump has been restored"
+else
+    vlog "Buffer pool dump has not been restored"
+    exit -1
+fi

=== added file 'test/t/ib_buffer_pool_dump_incremental.sh'
--- test/t/ib_buffer_pool_dump_incremental.sh	1970-01-01 00:00:00 +0000
+++ test/t/ib_buffer_pool_dump_incremental.sh	2013-04-25 14:34:24 +0000
@@ -0,0 +1,58 @@
+########################################################################
+# Blueprint: Support InnoDB buffer pool dumps in MySQL 5.6
+########################################################################
+
+. inc/common.sh
+
+if [ "${MYSQL_VERSION:0:3}" != "5.6" ]; then
+    echo "Requires MySQL 5.6" > $SKIPPED_REASON
+    exit $SKIPPED_EXIT_CODE
+fi
+
+MYSQLD_EXTRA_MY_CNF_OPTS="
+innodb_buffer_pool_filename=pool/dump
+"
+
+start_server
+
+# test presence of innodb_buffer_pool_filename
+${XB_BIN} --defaults-file=$topdir/my.cnf --print-param | \
+                            grep -q "innodb_buffer_pool_filename"
+
+mkdir $mysql_datadir/pool
+
+# take a backup
+innobackupex --no-timestamp $topdir/backup
+
+# produce buffer pool dump
+${MYSQL} ${MYSQL_ARGS} -e "SET GLOBAL innodb_buffer_pool_dump_now=ON;"
+
+# incremental backup
+innobackupex --no-timestamp --incremental --incremental-basedir=$topdir/backup \
+    $topdir/incremental
+
+if [ -f $topdir/incremental/pool/dump ] ; then
+    vlog "Buffer pool dump has been backed up"
+else
+    vlog "Buffer pool dump has not been backed up"
+    exit -1
+fi
+
+# prepare
+innobackupex --apply-log --redo-only $topdir/backup
+
+innobackupex --apply-log --redo-only --incremental-dir=$topdir/incremental \
+    $topdir/backup
+
+innobackupex --apply-log $topdir/backup
+
+# restore from backup
+rm -rf $mysql_datadir/*
+innobackupex --copy-back $topdir/backup
+
+if [ -f $mysql_datadir/pool/dump ] ; then
+    vlog "Buffer pool dump has been restored"
+else
+    vlog "Buffer pool dump has not been restored"
+    exit -1
+fi

=== added file 'test/t/ib_buffer_pool_rsync.sh'
--- test/t/ib_buffer_pool_rsync.sh	1970-01-01 00:00:00 +0000
+++ test/t/ib_buffer_pool_rsync.sh	2013-04-25 14:34:24 +0000
@@ -0,0 +1,40 @@
+########################################################################
+# Blueprint: Support InnoDB buffer pool dumps in MySQL 5.6
+########################################################################
+
+. inc/common.sh
+
+if ! which rsync > /dev/null 2>&1 ; then
+    echo "Requires rsync to be installed" > $SKIPPED_REASON
+    exit $SKIPPED_EXIT_CODE
+fi
+
+if [ "${MYSQL_VERSION:0:3}" != "5.6" ]; then
+    echo "Requires MySQL 5.6" > $SKIPPED_REASON
+    exit $SKIPPED_EXIT_CODE
+fi
+
+MYSQLD_EXTRA_MY_CNF_OPTS="
+innodb_buffer_pool_filename=pool/dump
+"
+
+start_server
+
+# test presence of innodb_buffer_pool_filename
+${XB_BIN} --defaults-file=$topdir/my.cnf --print-param | \
+                            grep -q "innodb_buffer_pool_filename"
+
+mkdir $mysql_datadir/pool
+
+# produce buffer pool dump
+${MYSQL} ${MYSQL_ARGS} -e "SET GLOBAL innodb_buffer_pool_dump_now=ON;"
+
+# take a backup with rsync mode
+innobackupex --rsync --no-timestamp $topdir/backup
+
+if [ -f $topdir/backup/pool/dump ] ; then
+    vlog "Buffer pool dump has been backed up"
+else
+    vlog "Buffer pool dump has not been backed up"
+    exit -1
+fi