Merge lp:~akopytov/percona-xtrabackup/bug803636-2.0 into lp:percona-xtrabackup/2.0

[Alexey Kopytov]

[Alexey Kopytov (akopytov)]

http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.0-param/256/

[Laurynas Biveinis (laurynas-biveinis)]

review: Approve

=== modified file 'doc/source/innobackupex/how_innobackupex_works.rst'
--- doc/source/innobackupex/how_innobackupex_works.rst	2012-02-14 08:31:46 +0000
+++ doc/source/innobackupex/how_innobackupex_works.rst	2012-09-06 09:53:29 +0000
@@ -71,3 +71,10 @@
 |innobackupex| will read the read from the :file:`my.cnf` the variables :term:`datadir`, :term:`innodb_data_home_dir`, :term:`innodb_data_file_path`, :term:`innodb_log_group_home_dir` and check that the directories exist.
 
 It will copy the |MyISAM| tables, indexes, etc. (:term:`.frm`, :term:`.MRG`, :term:`.MYD`, :term:`.MYI`, :term:`.TRG`, :term:`.TRN`, :term:`.ARM`, :term:`.ARZ`, :term:`.CSM`, :term:`.CSV` and :term:`.opt` files) first, |InnoDB| tables and indexes next and the log files at last. It will preserve file's attributes when copying them, you may have to change the files' ownership to ``mysql`` before starting the database server, as they will be owned by the user who created the backup.
+
+Alternatively, the :option:`--move-back` option may be used to restore a
+backup. This option is similar to :option:`--copy-back` with the only
+difference that instead of copying files it moves them to their target
+locations. As this option removes backup files, it must be used with
+caution. It is useful in cases when there is not enough free disk space
+to hold both data files and their backup copies.

=== modified file 'doc/source/innobackupex/innobackupex_option_reference.rst'
--- doc/source/innobackupex/innobackupex_option_reference.rst	2012-08-02 12:38:15 +0000
+++ doc/source/innobackupex/innobackupex_option_reference.rst	2012-09-06 09:53:29 +0000
@@ -30,6 +30,12 @@
 
     Copy all the files in a previously made backup from the backup directory to their original locations.
 
+.. option:: --move-back
+
+    Move all the files in a previously made backup from the backup
+    directory to their original locations. As this option removes
+    backup files, it must be used with caution.
+
 .. option:: --include=REGEXP
 
    This option is a regular expression to be matched against table names in ``databasename.tablename`` format. It is passed directly to :program:`xtrabackup` 's :option:`xtrabackup --tables` option. See the :program:`xtrabackup` documentation for details.
@@ -101,7 +107,7 @@
 
 .. option:: --ibbackup = 'autodetect'
 
-   This option accepts a string argument that specifies which xtrabackup binary should be used. The string should be the command used to run *XtraBackup*. The option can be useful if the :program:`xtrabackup` binary is not in your search path or working directory and the database server is not accessible at the moment. If this option is not specified, :program:`innobackupex` attempts to determine the binary to use automatically. By default, :program:`xtrabackup` is the command used. When option :option:`--apply-log` is specified, the binary is used whose name is in the file :file:`xtrabackup_binary` in the backup directory, if that file exists, or will attempt to autodetect it. However, if :option:`--copy-back` is selected, :program:`xtrabackup` is used unless other is specified.
+   This option accepts a string argument that specifies which xtrabackup binary should be used. The string should be the command used to run *XtraBackup*. The option can be useful if the :program:`xtrabackup` binary is not in your search path or working directory and the database server is not accessible at the moment. If this option is not specified, :program:`innobackupex` attempts to determine the binary to use automatically. By default, :program:`xtrabackup` is the command used. When option :option:`--apply-log` is specified, the binary is used whose name is in the file :file:`xtrabackup_binary` in the backup directory, if that file exists, or will attempt to autodetect it. However, if :option:`--copy-back` or :option:`--move-back` is used, :program:`xtrabackup` is used unless other is specified.
 
 .. option:: --incremental
 

=== modified file 'innobackupex'
--- innobackupex	2012-08-02 12:38:15 +0000
+++ innobackupex	2012-09-06 09:53:29 +0000
@@ -72,6 +72,7 @@
 my $option_apply_log = '';
 my $option_redo_only = '';
 my $option_copy_back = '';
+my $option_move_back = '';
 my $option_include = '';
 my $option_databases = '';
 my $option_tables_file = '';
@@ -237,7 +238,7 @@
 print_version();
 
 # initialize global variables and perform some checks
-if ($option_copy_back) {
+if ($option_copy_back || $option_move_back) {
     $option_ibbackup_binary = 'xtrabackup' if ($option_ibbackup_binary eq 'autodetect');
 } elsif ($option_apply_log) {
 	# Read XtraBackup version from backup dir
@@ -279,7 +280,10 @@
 
 if ($option_copy_back) {
     # copy files from backup directory back to their original locations
-    copy_back();
+    copy_back(0);
+} elsif ($option_move_back) {
+    # move files from backup directory back to their original locations
+    copy_back(1);
 } elsif ($option_apply_log) {
     # expand data files in backup directory by applying log files to them
     apply_log();
@@ -574,10 +578,41 @@
 }
 
 #
-# Callback function for copy_dir_recursively that actually does all
-# the copying work
-#
-sub copy_file() {
+# copy() wrapper with error handling
+#
+sub copy_file {
+    my $src_path = shift;
+    my $dst_path = shift;
+
+    print STDERR "$prefix Copying '$src_path' to '$dst_path'\n";
+    copy($src_path, $dst_path) or Die "copy failed: $!";
+}
+
+#
+# move() wrapper with error handling
+#
+sub move_file {
+    my $src_path = shift;
+    my $dst_path = shift;
+
+    print STDERR "$prefix Moving '$src_path' to '$dst_path'\n";
+    move($src_path, $dst_path) or Die "move failed: $!";
+}
+
+
+#
+# Auxiliary function called from find() callbacks to copy or move files and create directories
+# when necessary.
+#
+# SYNOPSIS
+#
+#     process_file(\&process_func);
+#
+#       &process_func is a code reference that does the actual file operation
+#
+sub process_file {
+    my $process_func = shift;
+
     if (/$copy_dir_exclude_regexp/) {
 	return;
     }
@@ -593,16 +628,29 @@
     } else {
 	# Don't overwrite files unless $copy_dir_overwrite is 1
 	if (!$copy_dir_overwrite && -e "$copy_dir_dst/$_") {
-	    Die "Failed to copy file $File::Find::name: " .
-		"file $copy_dir_dst/$_ exists";
+	    Die "Failed to process file $File::Find::name: " .
+		"not overwriting file $copy_dir_dst/$_";
 	}
 
-	print STDERR "$prefix Copying '$File::Find::name' to '$dst_path'\n";
-	copy($File::Find::name, $dst_path) or Die "copy failed: $!";
+	&$process_func($File::Find::name, $dst_path);
     }
 }
 
 #
+# find() callback to copy files
+#
+sub copy_file_callback {
+    process_file(\&copy_file);
+}
+
+#
+# find() callback to move files
+#
+sub move_file_callback {
+    process_file(\&move_file);
+}
+
+#
 # copy_dir_recursively subroutine does a recursive copy of a specified
 # directory excluding files matching a specifies regexp. If $overwrite
 # is 1, it overwrites the existing files.
@@ -623,7 +671,29 @@
     $copy_dir_exclude_regexp = shift;
     $copy_dir_overwrite = shift;
 
-    find(\&copy_file, $copy_dir_src);
+    find(\&copy_file_callback, $copy_dir_src);
+}
+
+#
+# Similar to copy_dir_recursively, but moves files instead.
+#
+# SYNOPSIS
+#
+#     move_dir_recursively($src_dir, $dst_dir, $exclude_regexp,
+#                          $overwrite);
+#
+# TODO
+#
+#     use rsync when --rsync is specified
+#
+sub move_dir_recursively {
+    # Clean paths and remove trailing slashes if any
+    $copy_dir_src = File::Spec->canonpath(shift);
+    $copy_dir_dst = File::Spec->canonpath(shift);
+    $copy_dir_exclude_regexp = shift;
+    $copy_dir_overwrite = shift;
+
+    find(\&move_file_callback, $copy_dir_src);
 }
 
 #
@@ -631,6 +701,7 @@
 # back to their original locations.
 #
 sub copy_back {
+    my $move_flag = shift;
     my $orig_datadir = get_option(\%config, $option_defaults_group, 'datadir');
     my $orig_ibdata_dir = 
         get_option(\%config, $option_defaults_group, 'innodb_data_home_dir');
@@ -647,6 +718,14 @@
     my $compressed_data_file = '.*\.ibz$';
     my $file;
     my $backup_innodb_data_file_path;
+
+    # check whether files should be copied or moved to dest directory
+    my $move_or_copy_file = $move_flag ? \&move_file : \&copy_file;
+    my $move_or_copy_dir = $move_flag ?
+        \&move_dir_recursively : \&copy_dir_recursively;
+    my $operation = $move_flag ? "move" : "copy";
+
+
     # check that original data directories exist and they are empty
     if_directory_exists_and_empty($orig_datadir, "Original data");
     if_directory_exists_and_empty($orig_ibdata_dir, "Original InnoDB data");
@@ -678,7 +757,8 @@
                 Die "Backup data file '$backup_dir/$filename' does not exist, but "
                   . "its compressed copy '${path}.ibz' exists. Check "
                   . "that you have run '$innobackup_script --apply-log --uncompress "
-                  . "...' before attempting '$innobackup_script --copy-back ...' !";
+                  . "...' before attempting '$innobackup_script --copy-back ...'  "
+                  . "or '$innobackup_script --move-back ...' !";
             } else {
                 Die "Backup data file '$backup_dir/$filename' does not exist.";
             }
@@ -691,38 +771,34 @@
 
     # copy files to original data directory
     my $excluded_regexp = '^(' . $excluded_files . ')$';
-    print STDERR "$prefix Starting to copy files in '$backup_dir'\n"; 
+    print STDERR "$prefix Starting to $operation files in '$backup_dir'\n"; 
     print STDERR "$prefix back to original data directory '$orig_datadir'\n";
-    copy_dir_recursively($backup_dir, $orig_datadir, $excluded_regexp, 0);
+    &$move_or_copy_dir($backup_dir, $orig_datadir, $excluded_regexp, 0);
 
     # copy InnoDB data files to original InnoDB data directory
-    print STDERR "\n$prefix Starting to copy InnoDB system tablespace\n";
+    print STDERR "\n$prefix Starting to $operation InnoDB system tablespace\n";
     print STDERR "$prefix in '$backup_dir'\n";
     print STDERR "$prefix back to original InnoDB data directory '$orig_ibdata_dir'\n";
     foreach my $a (split(/;/, $orig_innodb_data_file_path)) {
         # get the relative pathname of a data file
         my $path = (split(/:/,$a))[0];
         my $filename = (split(/\/+/, $path))[-1];
-        print STDERR "$prefix Copying file '$backup_dir/$filename'\n";
         $src_name = escape_path("$backup_dir/$filename");
         $dst_name = escape_path("$orig_ibdata_dir/$path");
-        system("$CP_CMD \"$src_name\" \"$dst_name\"")
-            and Die "Failed to copy file '$filename': $!";
+        &$move_or_copy_file($src_name, $dst_name);
     }
 
     # copy InnoDB log files to original InnoDB log directory
     opendir(DIR, $backup_dir) 
         || Die "Can't open directory '$backup_dir': $!\n";
-    print STDERR "\n$prefix Starting to copy InnoDB log files\n";
+    print STDERR "\n$prefix Starting to $operation InnoDB log files\n";
     print STDERR "$prefix in '$backup_dir'\n";
     print STDERR "$prefix back to original InnoDB log directory '$orig_iblog_dir'\n";
     while (defined($file = readdir(DIR))) {
         if ($file =~ /'^' . $iblog_files . '$'/ && -f "$backup_dir/$file") {
-            print STDERR "$prefix Copying file '$backup_dir/$file'\n";
             $src_name = escape_path("$backup_dir/$file");
             $dst_name = escape_path("$orig_iblog_dir");
-            system("$CP_CMD \"$src_name\" \"$dst_name\"")
-                and Die "Failed to copy file '$file': $!";
+            &$move_or_copy_file($src_name, $dst_name);
         }
     }
     closedir(DIR);
@@ -1528,10 +1604,12 @@
     my $run = '';
 
     # print some instructions to the user
-    if (!$option_apply_log && !$option_copy_back) {
+    if (!$option_apply_log && !$option_copy_back && !$option_move_back) {
         $run = 'backup';
     } elsif ($option_copy_back) {
         $run = 'copy-back';
+    } elsif ($option_move_back) {
+        $run = 'move-back';
     } else {
         $run = 'apply-log';
     }
@@ -1541,7 +1619,7 @@
 
     # check that MySQL client program and InnoDB Hot Backup program
     # are runnable via shell
-    if (!$option_copy_back) {
+    if (!$option_copy_back && !$option_move_back) {
         # we are making a backup or applying log to backup
         if (!$option_apply_log) {
             # we are making a backup, we need mysql server
@@ -1589,7 +1667,7 @@
     #$innodb_log_group_home_dir = 
     #    get_option(\%config, 'mysqld', 'innodb_log_group_home_dir');
 
-    if (!$option_apply_log && !$option_copy_back) {
+    if (!$option_apply_log && !$option_copy_back && !$option_move_back) {
         # we are making a backup, create a new backup directory
         if (!$option_remote_host) {
             $backup_dir = File::Spec->rel2abs(make_backup_dir());
@@ -1619,7 +1697,7 @@
         }
         }
         write_backup_config_file($backup_config_file);
-    } elsif ($option_copy_back) {
+    } elsif ($option_copy_back || $option_move_back) {
         #$backup_config_file = $backup_dir . '/backup-my.cnf';
         #read_config_file($backup_config_file, \%backup_config);
     }         
@@ -1715,6 +1793,7 @@
                         'apply-log' => \$option_apply_log,
                         'redo-only' => \$option_redo_only,
                         'copy-back' => \$option_copy_back,
+                        'move-back' => \$option_move_back,
                         'include=s' => \$option_include,
                         'databases=s' => \$option_databases,
                         'tables-file=s', => \$option_tables_file,
@@ -1765,6 +1844,11 @@
         exit(0);
     }
 
+    if ($option_copy_back && $option_move_back) {
+        print STDERR "$prefix --copy-back and --move-back options are " .
+            "mutually exclusive";
+    }
+
     if ($option_compress == 0) {
         # compression level no specified, use default level
         $option_compress = $default_compression_level;
@@ -1792,7 +1876,7 @@
     # get options file name
     #$config_file = $ARGV[0];
 
-    if (!$option_apply_log && !$option_copy_back) {
+    if (!$option_apply_log && !$option_copy_back && !$option_move_back) {
         # we are making a backup, get backup root directory
         $backup_root = $ARGV[0];
         if ($option_incremental && !$option_incremental_lsn) {
@@ -2635,6 +2719,8 @@
 
 innobackupex --copy-back [--defaults-file=MY.CNF] [--defaults-group=GROUP-NAME] BACKUP-DIR
 
+innobackupex --move-back [--defaults-file=MY.CNF] [--defaults-group=GROUP-NAME] BACKUP-DIR
+
 =head1 DESCRIPTION
 
 The first command line above makes a hot backup of a MySQL database.
@@ -2663,6 +2749,11 @@
 The MY.CNF options file defines the original location of the database.
 The BACKUP-DIR is the path to a backup directory created by xtrabackup.
 
+The --move-back command is similar to --copy-back with the only difference that
+it moves files to their original locations rather than copies them. As this
+option removes backup files, it must be used with caution. It may be useful in
+cases when there is not enough free disk space to copy files.
+
 On success the exit code innobackupex is 0. A non-zero exit code 
 indicates an error.
 
@@ -2691,6 +2782,10 @@
 
 Copy all the files in a previously made backup from the backup directory to their original locations.
 
+=item --move-back
+
+Move all the files in a previously made backup from the backup directory to the actual datadir location. Use with caution, as it removes backup files.
+
 =item --databases=LIST
 
 This option specifies the list of databases that innobackupex should back up.  The option accepts a string argument. The list is of the form "databasename1[.table_name1] databasename2[.table_name2] . . .". If this option is not specified, all databases containing MyISAM and InnoDB tables will be backed up. Please make sure that --databases contains all of the InnoDB databases and tables, so that all of the innodb.frm files are also backed up. In case the list is very long, this can be specified in a file, and the full path of the file can be specified instead of the list. (See option --tables-file.)

=== added file 'test/t/bug803636.sh'
--- test/t/bug803636.sh	1970-01-01 00:00:00 +0000
+++ test/t/bug803636.sh	2012-09-06 09:53:29 +0000
@@ -0,0 +1,31 @@
+########################################################################
+# Bug #803636: "moves files" option needed with --copy-back
+########################################################################
+
+. inc/common.sh
+
+start_server --innodb_file_per_table
+
+load_sakila
+
+# Backup
+innobackupex --no-timestamp $topdir/backup
+
+stop_server
+
+rm -r $mysql_datadir
+
+# Prepare
+innobackupex --apply-log $topdir/backup
+
+# Restore
+mkdir -p $mysql_datadir
+innobackupex --move-back $topdir/backup
+
+# Check that there are no data files in the backup directory
+run_cmd_expect_failure ls -R $topdir/backup/*/*.{ibd,MYD,MYI,frm}
+
+start_server
+
+# Verify data
+run_cmd ${MYSQL} ${MYSQL_ARGS} -e "SELECT count(*) from actor" sakila