Merge lp:~igor-tverdovskiy/percona-xtrabackup/move-back_opt into lp:percona-xtrabackup/2.0

[Igor Tverdovskiy]

move-back opt was added. It does the same as copy-back, but moves files. - useful if not much of free disk space is available.

[Stewart Smith (stewart)]

The only thing missing here is a test case. It will be fairly easy to write one though. Check out some of the ib_ tests in test/t - they're shell scripts that use a bunch of helper functions provided by the test suite.

review: Needs Fixing

[Vojtech Kurka (vojtech-kurka)]

Please how is the status of this patch? When recovering a compressed backup of 1TB of data, it's not acceptable to wait for copying the data, it can take hours - moving is the only option.
Thank you, Vojtech

[Stewart Smith (stewart)]

> Please how is the status of this patch? When recovering a compressed backup of
> 1TB of data, it's not acceptable to wait for copying the data, it can take
> hours - moving is the only option.
> Thank you, Vojtech

If you're able to supply a test case, we can merge it a lot quicker.

[lp:~igor-tverdovskiy/percona-xtrabackup/move-back_opt]

394. By Igor Tverdovskiy on 2012-08-29

changes applied to the latest innobackupex, test case added

395. By Igor Tverdovskiy on 2012-08-29

test case added

[Igor Tverdovskiy (igor-tverdovskiy)]

Hi All,

> > Please how is the status of this patch? When recovering a compressed backup
> of
> > 1TB of data, it's not acceptable to wait for copying the data, it can take
> > hours - moving is the only option.
> > Thank you, Vojtech
>
> If you're able to supply a test case, we can merge it a lot quicker.

Test case has been added (bug803636.sh)

All cases are passed or skipped:

> sudo ./run.sh -d /usr
2012-08-29 16:56:50: run.sh: Stopping server with id=1...
2012-08-29 16:56:51: run.sh: Server with id=1 has been stopped
Running against MySQL 5.1.52-log (InnoDB )
Using 'xtrabackup_51' as xtrabackup binary

========================================================================

t/bug1002688.sh [passed]
t/bug1022562.sh [passed]
t/bug1028949.sh [skipped] Requires InnoDB plugin or XtraDB
t/bug483827.sh [passed]
t/bug489290.sh [passed]
t/bug514068.sh [passed]
t/bug606981.sh [passed]
t/bug722638.sh [passed]
t/bug723097.sh [passed]
t/bug723318.sh [passed]
t/bug729843.sh [passed]
t/bug733651.sh [passed]
t/bug759225.sh [skipped] Requires XtraDB
t/bug759701.sh [passed]
t/bug766033.sh [passed]
t/bug766607.sh [passed]
t/bug803636.sh [passed]
t/bug810269.sh [skipped] Requires InnoDB plugin or XtraDB
t/bug817132.sh [skipped] Requires xtradb51
t/bug884737.sh [passed]
t/bug891496.sh [passed]
t/bug932623.sh [passed]
t/bug972169.sh [passed]
t/bug976945.sh [skipped] Requires XtraDB
t/bug977101.sh [passed]
t/bug977652.sh [skipped] Requires qpress to be installed
t/bug983685.sh [passed]
t/bug983695.sh [skipped] Requires qpress to be installed
t/bug983720_galerainfo.sh [skipped] Requires WSREP enabled
t/bug983720_lrudump.sh [skipped] Requires XtraDB
t/bug989397.sh [passed]
t/bug996493.sh [passed]
t/bug999750.sh [skipped] Requires InnoDB plugin or XtraDB
t/ib_binlog_info.sh [passed]
t/ib_csm_csv.sh [passed]
t/ib_empty_dir.sh [passed]
t/ib_incremental.sh [passed]
t/ib_lru_dump_basic.sh [skipped] Requires XtraDB
t/ib_lru_dump_rsync.sh [skipped] Requires XtraDB
t/ib_lru_dump_stream.sh [skipped] Requires XtraDB
t/ib_rsync.sh [passed]
t/ib_slave_info.sh [passed]
t/ib_specialchar.sh [passed]
t/ib_stream_compress.sh [skipped] ...

[Igor Tverdovskiy (igor-tverdovskiy)]

PS: I have committed the latest innobackupex file with my changes.

[Laurynas Biveinis (laurynas-biveinis)]

Unfortunately there are merge conflicts against the current 2.0 trunk, see the diff below.

review: Needs Fixing

[Igor Tverdovskiy (igor-tverdovskiy)]

> Unfortunately there are merge conflicts against the current 2.0 trunk, see the
> diff below.

Okay, I'm going to add modifications to innobackupex from lp:percona-xtrabackup/2.0 branch. I'll be back soon.

[lp:~igor-tverdovskiy/percona-xtrabackup/move-back_opt]

396. By Igor Tverdovskiy on 2012-09-02

option added to 2.0 trunk

[Igor Tverdovskiy (igor-tverdovskiy)]

done please check

[Alexey Kopytov (akopytov)]

There's still one conflict in the pod section of innobackupex:

176 -innobackupex --copy-back [--defaults-file=MY.CNF] [--defaults-group=GROUP-NAME] BACKUP-DIR
177 +<<<<<<< TREE
178 +innobackupex --copy-back [--defaults-file=MY.CNF] [--defaults-group=GROUP-NAME] BACKUP-DIR
179 +=======
180 +innobackupex --copy-back [--defaults-file=MY.CNF] [--defaults-group=GROUP-NAME] BACKUP-DIR
181 + --move-back [--defaults-file=MY.CNF] [--defaults-group=GROUP-NAME] BACKUP-DIR
182 +>>>>>>> MERGE-SOURCE

I would create a separate line in the help text, rather than add --move-back to the existing one describing the --copy-back synopsis, i.e.:

innobackupex --copy-back [--defaults-file=MY.CNF] [--defaults-group=GROUP-NAME] BACKUP-DIR

innobackupex --move-back [--defaults-file=MY.CNF] [--defaults-group=GROUP-NAME] BACKUP-DIR

review: Needs Fixing

[lp:~igor-tverdovskiy/percona-xtrabackup/move-back_opt]

397. By Igor Tverdovskiy on 2012-09-02

move-back hepl text has been separated

[Igor Tverdovskiy (igor-tverdovskiy)]

committed please check

[Laurynas Biveinis (laurynas-biveinis)]

> committed please check

There are still merge conflicts shown in the diff: around lines 58 and 176.

review: Needs Fixing

[Igor Tverdovskiy (igor-tverdovskiy)]

> > committed please check
>
> There are still merge conflicts shown in the diff: around lines 58 and 176.

Hi,

I have looked through, however do not see any conflicts there.. Just a single new line of code was added in both cases.

[Alexey Kopytov (akopytov)]

Igor,

You can find the remaining conflicts if you look for "<<<" in the Preview Diff. But don't worry, I'll fix it myself in a separate MP along with some minor edits to the patch. Closing this MP.

review: Needs Resubmitting

[Igor Tverdovskiy (igor-tverdovskiy)]

I saw these conflicts <<<, however I do not understand what is wrong there... One line was replaced by two lines of code...

[Laurynas Biveinis (laurynas-biveinis)]

Probably the reason for all these conflicts is that the submitted branch has been branched off XtraBackup trunk a long time ago, so it is very likely they have diverged.

[Igor Tverdovskiy (igor-tverdovskiy)]

It has been branched on 2nd of September from lp:percona-xtrabackup/2.0 branch. Is this branch wrong? I should have used another one?

thanks

[Laurynas Biveinis (laurynas-biveinis)]

It does not seem to be branched off lp:percona-xtrabackup/2.0: the last parent branch commit on your branch is rev 392, dated in February. lp:percona-xtrabackup/2.0 currently is at rev 464, made on August 22.

[Igor Tverdovskiy (igor-tverdovskiy)]

Well my branch is pretty old because I have added this option in February, however on 2nd September I downloaded innobackupex from 2.0 branch, added my code, replaced old innobackupex in my branch with new one and committed changes.

[Laurynas Biveinis (laurynas-biveinis)]

I see, that probably confused bzr into thinking that there are conflicts. One of the correct procedures would be to branch freshly from recent 2.0 and make your changes - this might come handy in the future, as Alexey said he will take care of this MP.

Thanks for working on this.

[Igor Tverdovskiy (igor-tverdovskiy)]

ok, thanks for clarification

[Laurynas Biveinis (laurynas-biveinis)]

New MP at https://code.launchpad.net/~akopytov/percona-xtrabackup/bug803636-2.0/+merge/123040

Unmerged revisions

397. By Igor Tverdovskiy on 2012-09-02

move-back hepl text has been separated

396. By Igor Tverdovskiy on 2012-09-02

option added to 2.0 trunk

395. By Igor Tverdovskiy on 2012-08-29

test case added

394. By Igor Tverdovskiy on 2012-08-29

changes applied to the latest innobackupex, test case added

393. By Igor Tverdovskiy on 2012-02-18

added new option --move-back, just the same as --copy-back, but moves files instead.

=== modified file 'innobackupex'
--- innobackupex	2012-08-02 12:38:15 +0000
+++ innobackupex	2012-09-02 15:43:18 +0000
@@ -72,6 +72,7 @@
 my $option_apply_log = '';
 my $option_redo_only = '';
 my $option_copy_back = '';
+my $option_move_back = '';
 my $option_include = '';
 my $option_databases = '';
 my $option_tables_file = '';
@@ -217,6 +218,8 @@
 my $dst_name;
 my $win = ($^O eq 'MSWin32' ? 1 : 0);
 my $CP_CMD = ($win eq 1 ? "copy /Y" : "cp -p");
+my $MV_CMD = ($win eq 1 ? "move /Y" : "mv -f");
+my $MV_CP_CMD = '';
 my $xtrabackup_binary_file = 'xtrabackup_binary';
 my $xtrabackup_pid_file = 'xtrabackup_pid';
 my %rsync_files_hash;
@@ -237,7 +240,7 @@
 print_version();
 
 # initialize global variables and perform some checks
-if ($option_copy_back) {
+if ($option_copy_back || $option_move_back) {
     $option_ibbackup_binary = 'xtrabackup' if ($option_ibbackup_binary eq 'autodetect');
 } elsif ($option_apply_log) {
 	# Read XtraBackup version from backup dir
@@ -280,6 +283,9 @@
 if ($option_copy_back) {
     # copy files from backup directory back to their original locations
     copy_back();
+} elsif ($option_move_back) {
+    # move files from backup directory back to their original locations
+    move_back();
 } elsif ($option_apply_log) {
     # expand data files in backup directory by applying log files to them
     apply_log();
@@ -627,11 +633,25 @@
 }
 
 #
+# move_back subroutine uses copy_back subroutine to move data and index files 
+# from backup directory back to their original locations.
+#
+sub move_back {
+    my $move = 'yes';
+    copy_back($move);
+}
+
+#
 # copy_back subroutine copies data and index files from backup directory 
 # back to their original locations.
 #
 sub copy_back {
-    my $orig_datadir = get_option(\%config, $option_defaults_group, 'datadir');
+<<<<<<< TREE
+    my $orig_datadir = get_option(\%config, $option_defaults_group, 'datadir');
+=======
+	my($move) = @_;
+    my $orig_datadir = get_option(\%config, $option_defaults_group, 'datadir');
+>>>>>>> MERGE-SOURCE
     my $orig_ibdata_dir = 
         get_option(\%config, $option_defaults_group, 'innodb_data_home_dir');
     my $orig_innodb_data_file_path = 
@@ -647,6 +667,14 @@
     my $compressed_data_file = '.*\.ibz$';
     my $file;
     my $backup_innodb_data_file_path;
+    
+    # check whether files should be copied or moved to dest directory
+    if ($move eq 'yes') {
+        $MV_CP_CMD = $MV_CMD;
+    } else {
+        $MV_CP_CMD = $CP_CMD;
+    }
+    
     # check that original data directories exist and they are empty
     if_directory_exists_and_empty($orig_datadir, "Original data");
     if_directory_exists_and_empty($orig_ibdata_dir, "Original InnoDB data");
@@ -678,7 +706,8 @@
                 Die "Backup data file '$backup_dir/$filename' does not exist, but "
                   . "its compressed copy '${path}.ibz' exists. Check "
                   . "that you have run '$innobackup_script --apply-log --uncompress "
-                  . "...' before attempting '$innobackup_script --copy-back ...' !";
+                  . "...' before attempting '$innobackup_script --copy-back ...'  "
+                  . "...'                or '$innobackup_script --move-back ...' !";
             } else {
                 Die "Backup data file '$backup_dir/$filename' does not exist.";
             }
@@ -706,8 +735,8 @@
         print STDERR "$prefix Copying file '$backup_dir/$filename'\n";
         $src_name = escape_path("$backup_dir/$filename");
         $dst_name = escape_path("$orig_ibdata_dir/$path");
-        system("$CP_CMD \"$src_name\" \"$dst_name\"")
-            and Die "Failed to copy file '$filename': $!";
+		system("$MV_CP_CMD \"$src_name\" \"$dst_name\"")
+            and Die "Failed to copy/move file '$filename': $!";
     }
 
     # copy InnoDB log files to original InnoDB log directory
@@ -721,8 +750,8 @@
             print STDERR "$prefix Copying file '$backup_dir/$file'\n";
             $src_name = escape_path("$backup_dir/$file");
             $dst_name = escape_path("$orig_iblog_dir");
-            system("$CP_CMD \"$src_name\" \"$dst_name\"")
-                and Die "Failed to copy file '$file': $!";
+            system("$MV_CP_CMD \"$src_name\" \"$dst_name\"")
+                and Die "Failed to copy/move file '$file': $!";
         }
     }
     closedir(DIR);
@@ -1528,10 +1557,12 @@
     my $run = '';
 
     # print some instructions to the user
-    if (!$option_apply_log && !$option_copy_back) {
+    if (!$option_apply_log && !$option_copy_back && !$option_move_back) {
         $run = 'backup';
     } elsif ($option_copy_back) {
         $run = 'copy-back';
+    } elsif ($option_move_back) {
+        $run = 'move-back';
     } else {
         $run = 'apply-log';
     }
@@ -1541,7 +1572,7 @@
 
     # check that MySQL client program and InnoDB Hot Backup program
     # are runnable via shell
-    if (!$option_copy_back) {
+    if (!$option_copy_back && !$option_move_back) {
         # we are making a backup or applying log to backup
         if (!$option_apply_log) {
             # we are making a backup, we need mysql server
@@ -1589,7 +1620,7 @@
     #$innodb_log_group_home_dir = 
     #    get_option(\%config, 'mysqld', 'innodb_log_group_home_dir');
 
-    if (!$option_apply_log && !$option_copy_back) {
+    if (!$option_apply_log && !$option_copy_back && !$option_move_back) {
         # we are making a backup, create a new backup directory
         if (!$option_remote_host) {
             $backup_dir = File::Spec->rel2abs(make_backup_dir());
@@ -1619,7 +1650,7 @@
         }
         }
         write_backup_config_file($backup_config_file);
-    } elsif ($option_copy_back) {
+    } elsif ($option_copy_back || $option_move_back) {
         #$backup_config_file = $backup_dir . '/backup-my.cnf';
         #read_config_file($backup_config_file, \%backup_config);
     }         
@@ -1715,6 +1746,7 @@
                         'apply-log' => \$option_apply_log,
                         'redo-only' => \$option_redo_only,
                         'copy-back' => \$option_copy_back,
+                        'move-back' => \$option_move_back,
                         'include=s' => \$option_include,
                         'databases=s' => \$option_databases,
                         'tables-file=s', => \$option_tables_file,
@@ -1792,7 +1824,7 @@
     # get options file name
     #$config_file = $ARGV[0];
 
-    if (!$option_apply_log && !$option_copy_back) {
+    if (!$option_apply_log && !$option_copy_back && !$option_move_back) {
         # we are making a backup, get backup root directory
         $backup_root = $ARGV[0];
         if ($option_incremental && !$option_incremental_lsn) {
@@ -2633,7 +2665,13 @@
              [--export] [--redo-only] [--ibbackup=IBBACKUP-BINARY]
              BACKUP-DIR
 
-innobackupex --copy-back [--defaults-file=MY.CNF] [--defaults-group=GROUP-NAME] BACKUP-DIR
+<<<<<<< TREE
+innobackupex --copy-back [--defaults-file=MY.CNF] [--defaults-group=GROUP-NAME] BACKUP-DIR
+=======
+innobackupex --copy-back [--defaults-file=MY.CNF] [--defaults-group=GROUP-NAME] BACKUP-DIR
+
+innobackupex --move-back [--defaults-file=MY.CNF] [--defaults-group=GROUP-NAME] BACKUP-DIR
+>>>>>>> MERGE-SOURCE
 
 =head1 DESCRIPTION
 
@@ -2663,6 +2701,10 @@
 The MY.CNF options file defines the original location of the database.
 The BACKUP-DIR is the path to a backup directory created by xtrabackup.
 
+The --move-back command does just the same as --copy-back. The only difference is
+that files are moved to data directory instead of be copied. Useful in case you 
+have not much free space at your destination server.
+
 On success the exit code innobackupex is 0. A non-zero exit code 
 indicates an error.
 
@@ -2691,6 +2733,10 @@
 
 Copy all the files in a previously made backup from the backup directory to their original locations.
 
+=item --move-back
+ 
+Move all the files in a previously made backup from the backup directory to the actual datadir location.
+
 =item --databases=LIST
 
 This option specifies the list of databases that innobackupex should back up.  The option accepts a string argument. The list is of the form "databasename1[.table_name1] databasename2[.table_name2] . . .". If this option is not specified, all databases containing MyISAM and InnoDB tables will be backed up. Please make sure that --databases contains all of the InnoDB databases and tables, so that all of the innodb.frm files are also backed up. In case the list is very long, this can be specified in a file, and the full path of the file can be specified instead of the list. (See option --tables-file.)
 
=== added file 'test/t/bug803636.sh'
--- test/t/bug803636.sh	1970-01-01 00:00:00 +0000
+++ test/t/bug803636.sh	2012-09-02 15:43:18 +0000
@@ -0,0 +1,33 @@
+. inc/common.sh
+
+start_server
+
+load_dbase_schema sakila
+load_dbase_data sakila
+
+mkdir -p $topdir/backup
+innobackupex $topdir/backup
+backup_dir=`grep "innobackupex: Backup created in directory" $OUTFILE | awk -F\' '{ print $2}'`
+vlog "Backup created in directory $backup_dir"
+
+stop_server
+# Remove datadir
+rm -r $mysql_datadir
+#init_mysql_dir
+# Restore sakila
+vlog "Applying log"
+vlog "###########"
+vlog "# PREPARE #"
+vlog "###########"
+innobackupex --apply-log $backup_dir
+vlog "Restoring MySQL datadir"
+mkdir -p $mysql_datadir
+vlog "###########"
+vlog "# RESTORE #"
+vlog "###########"
+innobackupex --move-back $backup_dir
+
+start_server
+# Check sakila
+echo ${MYSQL}
+run_cmd ${MYSQL} ${MYSQL_ARGS} -e "SELECT count(*) from actor" sakila