Merge lp:~sergei.glushchenko/percona-xtrabackup/xb_bug483827 into lp:percona-xtrabackup/2.0

[Sergei Glushchenko]

Bug #483827. Support for mysqld_multi.
Fix based on work of Daniël van Eeden (lp:~dveeden/percona-xtrabackup/lp483827)
--section option added to innobackupex, specifying which section of my.cnf to handle.
--mysqld-section option added to xtrabackup for the same purpose.
Tescase does not run mysqld_multi, it just tests correctness of --section option.

http://jenkins.percona.com/view/Percona%20Xtrabackup/job/percona-xtrabackup-param/168/

[lp:~sergei.glushchenko/percona-xtrabackup/xb_bug483827]

409. By Hrvoje Matijakovic on 2012-03-29

* merged lp:~hrvojem/percona-xtrabackup/release-notes-1.9.2

410. By Hrvoje Matijakovic on 2012-03-29

* merged lp:~hrvojem/percona-xtrabackup/theme-update-2.0

411. By Stewart Smith on 2012-04-02

merge 1.9.2 release branch

412. By Hrvoje Matijakovic on 2012-04-03

* merged lp:~hrvojem/percona-xtrabackup/bug972185

413. By Hrvoje Matijakovic on 2012-04-04

* merged lp:~hrvojem/percona-xtrabackup/fresh-theme-2.0

414. By Stewart Smith on 2012-04-04

merge 1.6.6 release notes

415. By Stewart Smith on 2012-04-04

null merge of update 1.6.6 release date

416. By Stewart Smith on 2012-04-04

add 2.0.0 release notes

417. By Hrvoje Matijakovic on 2012-04-12

* merged lp:~hrvojem/percona-xtrabackup/theme-fix-2.0

418. By Stewart Smith on 2012-04-14

merge spelling fix in error msg

419. By Stewart Smith on 2012-04-14

Merge 2.0.0 release branch

420. By Alexey Kopytov on 2012-04-18

Automerge of lp:~akopytov/percona-xtrabackup/bug977995

421. By Alexey Kopytov on 2012-04-22

Bug #977998: --stream=tar fails (archive_write_header failed)

The problem was in the USTAR format used by XtraBackup which is
incompatible with files larger than 8 GB.

Fixed by changing the libarchive format to 'restricted PAX' which avoids
the limitation. However, the PAX format generates some annoying (and
benign) warnings when the GNU tar format is used due to extended
attributes written to the tar stream by libarchive in the PAX mode. This
problem was fixed by modifying the bundled libarchive code to suppress
the extended attributes.

422. By Stewart Smith on 2012-04-25

merge XB 2.0 docs update

423. By Stewart Smith on 2012-04-30

merge Sergei's fix for: Bug #983720: ib_lru_dump and --galera-info fail with --stream=xbstream

424. By Stewart Smith on 2012-04-30

Merge Sergei's fix for: Bug 983685: innodb_data_file_path is not written to backup-my.cnf
Regression introduced by fix for bug #733651.
Along with innodb_data_file_path, innodb_log_group_home_dir is also missed.

425. By Alexey Kopytov on 2012-05-14

Null merge of doc theme changes from 1.6.

[Alexey Kopytov (akopytov)]

Sergei,

   - sections in my.cnf are called groups in MySQL documentation. So
     let's be consistent and call them groups as well, and name the
     options (both the innobackupex and the xtrabackup one)
     --defaults-group, rather than --section and --mysqld-section

   - it looks like get_option is always passed the same value as its
     second argument, so I wonder if it really needs that argument, or
     rather just use $option_mysqld_section (or whatever it will be
     renamed to, i.e. $option_defaults_group) internally.

   - s/laod/load/
   - s/accetps/accepts/
   - s/rstore_args/restore_args/. Though you don't really need that
     function, as tests are executed in a separate shell process, so
     modifications to variables have no effect on other tests anyway.

   - I know that code in main() that scan options for "--mysqld-section"
     was copy-pasted, but please format it according to InnoDB
     style. Because currently it's a terrible mix of all possible
     formatting styles.

   - in the same code, I don't think strstr() is necessary, because you
     already have the pointer to '=' (or terminating zero) in optend.

review: Needs Fixing

[lp:~sergei.glushchenko/percona-xtrabackup/xb_bug483827]

426. By Sergei Glushchenko on 2012-05-17

Bug 996493: innobackupex --apply-log doesn't read config from backup-my.cnf
Following have been done:
1. Remove datadir and innodb_data_home_dir from backup-my.cnf
2. When do --apply-log read settings from backup-my.cnf
3. When do --copy-back read settings from /etc/my.cnf

427. By Sergei Glushchenko on 2012-05-17

Bug #483827. Support for mysqld_multi.
Based on work of Daniël van Eeden (lp:~dveeden/percona-xtrabackup/lp483827)
--defaults-group option added to both innobackupex and xtrabackup, specifying
which section of my.cnf to handle.

Unmerged revisions

427. By Sergei Glushchenko on 2012-05-17

Bug #483827. Support for mysqld_multi.
Based on work of Daniël van Eeden (lp:~dveeden/percona-xtrabackup/lp483827)
--defaults-group option added to both innobackupex and xtrabackup, specifying
which section of my.cnf to handle.

=== modified file 'doc/source/how-tos.rst'
--- doc/source/how-tos.rst	2011-07-07 05:32:50 +0000
+++ doc/source/how-tos.rst	2012-05-17 09:29:18 +0000
@@ -13,6 +13,7 @@
    howtos/recipes_ibkx_local
    howtos/recipes_ibkx_stream
    howtos/recipes_ibkx_inc
+   howtos/recipes_ibkx_multi
 
 
 .. _recipes-xbk:

=== modified file 'doc/source/innobackupex/innobackupex_option_reference.rst'
--- doc/source/innobackupex/innobackupex_option_reference.rst	2012-05-15 09:30:50 +0000
+++ doc/source/innobackupex/innobackupex_option_reference.rst	2012-05-17 09:29:18 +0000
@@ -70,6 +70,10 @@
 
    This option accepts a string argument that specifies the port to use when connecting to the database server with TCP/IP. It is passed to the :command:`mysql` child process. It is passed to the :command:`mysql` child process without alteration. See :command:`mysql --help` for details.
 
+.. option:: --defaults-group
+
+   This option accepts a string argument that specifies the group which should be read from the configuration file. This is needed if you use mysqld_multi.
+
 .. option:: --socket
 
    This option accepts a string argument that specifies the socket to use when connecting to the local database server with a UNIX domain socket. It is passed to the mysql child process without alteration. See :command:`mysql --help` for details.

=== modified file 'doc/source/innobackupex/restoring_a_backup_ibk.rst'
--- doc/source/innobackupex/restoring_a_backup_ibk.rst	2011-07-07 05:32:50 +0000
+++ doc/source/innobackupex/restoring_a_backup_ibk.rst	2012-05-17 09:29:18 +0000
@@ -6,7 +6,7 @@
 
   $ innobackupex --copy-back /path/to/BACKUP-DIR
 
-It will copy all the data-related files back to the server's :term:`datadir`, determined by the :file:`backup-my.cnf` configuration file in the directory. You should check the last line of the output for a success message::
+It will copy all the data-related files back to the server's :term:`datadir`, determined by the server's :file:`my.cnf` configuration file. You should check the last line of the output for a success message::
 
   innobackupex: Finished copying back files.
 

=== modified file 'doc/source/xtrabackup_bin/xbk_option_reference.rst'
--- doc/source/xtrabackup_bin/xbk_option_reference.rst	2012-04-21 07:12:28 +0000
+++ doc/source/xtrabackup_bin/xbk_option_reference.rst	2012-05-17 09:29:18 +0000
@@ -101,6 +101,10 @@
     --innodb-read-io-threads
     --innodb-write-io-threads
 
+.. option:: --defaults-group
+
+   This option is to set the group which should be read from the configuration file. This is used by innobackupex if you use the `--defaults-group` option. It is needed for mysqld_multi deployments. It must be the first option.
+
 .. option:: --log-stream
 
    Makes xtrabackup not copy data files, and output the contents of the InnoDB log files to STDOUT until the :option:`--suspend-at-end` file is deleted. This option enables :option:`--suspend-at-end` automatically.

=== modified file 'innobackupex'
--- innobackupex	2012-05-15 09:30:50 +0000
+++ innobackupex	2012-05-17 09:29:18 +0000
@@ -84,6 +84,7 @@
 my $option_mysql_port = '';
 my $option_mysql_socket = '';
 my $option_mysql_host = '';
+my $option_defaults_group = 'mysqld';
 my $option_no_timestamp = '';
 my $option_slave_info = '';
 my $option_galera_info = '';
@@ -377,7 +378,7 @@
 # process.
 #
 sub backup {
-    my $orig_datadir = get_option(\%config, 'mysqld', 'datadir');
+    my $orig_datadir = get_option(\%config, $option_defaults_group, 'datadir');
 
     # check that we can connect to the database. This done by
     # connecting, issuing a query, and closing the connection.
@@ -626,13 +627,13 @@
 # back to their original locations.
 #
 sub copy_back {
-    my $orig_datadir = get_option(\%config, 'mysqld', 'datadir');
+    my $orig_datadir = get_option(\%config, $option_defaults_group, 'datadir');
     my $orig_ibdata_dir = 
-        get_option(\%config, 'mysqld', 'innodb_data_home_dir');
+        get_option(\%config, $option_defaults_group, 'innodb_data_home_dir');
     my $orig_innodb_data_file_path = 
-        get_option(\%config, 'mysqld', 'innodb_data_file_path');
+        get_option(\%config, $option_defaults_group, 'innodb_data_file_path');
     my $orig_iblog_dir =
-        get_option(\%config, 'mysqld', 'innodb_log_group_home_dir');
+        get_option(\%config, $option_defaults_group, 'innodb_log_group_home_dir');
     my $iblog_files = 'ib_logfile.*';
     my $excluded_files = 
         '\.\.?|backup-my\.cnf|xtrabackup_logfile|' .
@@ -736,6 +737,12 @@
 
     if ($option_defaults_file) {
         $options = $options . " --defaults-file=\"$option_defaults_file\" ";
+    } else {
+        $options = $options . " --defaults-file=\"${backup_dir}/backup-my.cnf\" ";
+    }
+
+    if ($option_defaults_group) {
+        $options = $options . " --defaults-group=\"$option_defaults_group\" ";
     }
 
     $options = $options . "--prepare --target-dir=$backup_dir";
@@ -861,6 +868,10 @@
         $options = $options . " --defaults-file=\"$option_defaults_file\" ";
     }
 
+    if ($option_defaults_group) {
+        $options = $options . " --defaults-group=\"$option_defaults_group\" ";
+    }
+
     $options = $options . "--backup --suspend-at-end";
 
     if (!$option_remote_host && !$option_stream) {
@@ -923,13 +934,13 @@
 
             if($option_remote_host) {
                 #direct copy to remote
-                my $orig_datadir = get_option(\%config, 'mysqld', 'datadir');
+                my $orig_datadir = get_option(\%config, $option_defaults_group, 'datadir');
                 my $orig_ibdata_dir =
-                    get_option(\%config, 'mysqld', 'innodb_data_home_dir');
+                    get_option(\%config, $option_defaults_group, 'innodb_data_home_dir');
                 my $orig_innodb_data_file_path =
-                    get_option(\%config, 'mysqld', 'innodb_data_file_path');
+                    get_option(\%config, $option_defaults_group, 'innodb_data_file_path');
                 my $innodb_flush_method = 
-                    get_option(\%config, 'mysqld', 'innodb_flush_method');
+                    get_option(\%config, $option_defaults_group, 'innodb_flush_method');
                 my $innodb_use_odirect;
                 $innodb_use_odirect = 1 if $innodb_flush_method =~ m/^(ALL_)?O_DIRECT$/i;
 
@@ -1580,7 +1591,7 @@
     read_config_file(\%config);
 
     if(!$option_tmpdir) {
-        $option_tmpdir = get_option(\%config, 'mysqld', 'tmpdir');
+        $option_tmpdir = get_option(\%config, $option_defaults_group, 'tmpdir');
     }
 
     # get innodb log home directory from options file
@@ -1633,22 +1644,10 @@
 #
 sub write_backup_config_file {
     my $filename = shift;
-    my $innodb_data_file_path = 
-        get_option(\%config, 'mysqld', 'innodb_data_file_path');
-    my $root;
-
-    my @array = split(/;/, $innodb_data_file_path);
-    for (my $i = 0; $i <= $#array; $i++) {
-        my @tmp = split(/\/+/, $array[$i]);
-        $array[$i] = $tmp[-1];
-    }
-    $innodb_data_file_path = join(";", @array);
 
     if (!$option_remote_host) {
-        $root = $backup_dir;
         open(FILE, "> $filename") || Die "Failed to open file '$filename': $!";
     } else {
-        $root = `ssh $option_ssh_opt $option_remote_host 'cd $backup_dir; pwd'`;
         open(FILE, "| ssh $option_ssh_opt $option_remote_host 'cat > $filename'")
             || Die "Failed to open file '$option_remote_host:$filename': $!";
     }
@@ -1664,15 +1663,12 @@
 
     my $options_dump = "# This MySQL options file was generated by $innobackup_script.\n\n" .
           "# The MySQL server\n" .
-          "[mysqld]\n" .
-          "datadir=$root\n" .
-          "innodb_data_home_dir=$root\n" .
-          "innodb_log_group_home_dir=$root\n";
+          "[mysqld]\n";
 
     my $option_name;
     foreach $option_name (@option_names) {
-        if (has_option(\%config, 'mysqld', $option_name)) {
-            my $option_value = get_option(\%config, 'mysqld', $option_name);
+        if (has_option(\%config, $option_defaults_group, $option_name)) {
+            my $option_value = get_option(\%config, $option_defaults_group, $option_name);
             $options_dump .= "$option_name=$option_value\n";
         }
     }
@@ -1738,6 +1734,7 @@
                         'user=s' => \$option_mysql_user,
                         'host=s' => \$option_mysql_host,
                         'port=s' => \$option_mysql_port,
+                        'defaults-group=s' => \$option_defaults_group,
                         'slave-info' => \$option_slave_info,
                         'galera-info' => \$option_galera_info,
                         'socket=s' => \$option_mysql_socket,
@@ -1853,7 +1850,7 @@
 sub make_backup_dir {
     my $dir;
     my $innodb_data_file_path = 
-        get_option(\%config, 'mysqld', 'innodb_data_file_path');
+        get_option(\%config, $option_defaults_group, 'innodb_data_file_path');
 
     # create backup directory
     $dir = $backup_root;
@@ -1945,7 +1942,7 @@
 #
 sub backup_files {
     my $prep_mode = shift;
-    my $source_dir = get_option(\%config, 'mysqld', 'datadir');
+    my $source_dir = get_option(\%config, $option_defaults_group, 'datadir');
     my @list;
     my $file;
     my $database;
@@ -2217,8 +2214,15 @@
     my $cmdline = '';
     my $options = '';
 
+
     if ($option_defaults_file) {
         $options = $options . " --defaults-file=\"$option_defaults_file\" ";
+    } elsif ($option_apply_log) {
+        $options = $options . " --defaults-file=\"${backup_dir}/backup-my.cnf\" ";
+    }
+
+    if ($option_defaults_group) {
+        $options = $options . " --defaults-group=\"$option_defaults_group\" ";
     }
 
     $options = $options . "--print-param";
@@ -2304,7 +2308,6 @@
     if (!exists ${$config_ref}{$group}) {
         # no group
         print STDERR "$prefix fatal error: no '$group' group in MySQL options\n";
-        print STDERR "$prefix fatal error: OR no 'datadir' option in group '$group' in MySQL options\n";
         exit(1);
     }
 
@@ -2332,7 +2335,6 @@
     if (!exists $config{$group}) {
         # no group
         print STDERR "$prefix fatal error: no '$group' group in MySQL options\n";
-        print STDERR "$prefix fatal error: OR no 'datadir' option in group '$group' in MySQL options\n";
         exit(1);
     }
     
@@ -2628,7 +2630,7 @@
              [--no-timestamp] [--ibbackup=IBBACKUP-BINARY]
              [--slave-info] [--stream=tar|xbstream]
              [--scpopt=OPTIONS-FOR-SCP]  [--sshopt=OPTIONS-FOR-SSH]
-             [--defaults-file=MY.CNF]
+             [--defaults-file=MY.CNF] [--defaults-group=mysqld]
              [--databases=LIST] [--remote-host=HOSTNAME] [--no-lock] 
              [--tmpdir=DIRECTORY] [--tables-file=FILE]
              [--incremental] [--incremental-basedir]
@@ -2636,7 +2638,7 @@
              BACKUP-ROOT-DIR
 
 innobackupex --apply-log [--use-memory=B] [--uncompress]
-             [--defaults-file=MY.CNF]
+             [--defaults-file=MY.CNF] [--defaults-group=mysqld]
              [--export] [--redo-only] [--ibbackup=IBBACKUP-BINARY]
              BACKUP-DIR
 
@@ -2823,6 +2825,10 @@
 
 This option specifies the MySQL username used when connecting to the server, if that's not the current user. The option accepts a string argument.  It is passed to the mysql child process without alteration. See mysql --help for details.
 
+=item --defaults-group=GROUP
+
+This option specifies the group name in my.cnf which should be used. This is needed for mysqld_multi deployments.
+
 =item --version
 
 This option displays the xtrabackup version and copyright notice and then exits.

=== modified file 'src/xtrabackup.c'
--- src/xtrabackup.c	2012-05-15 16:53:07 +0000
+++ src/xtrabackup.c	2012-05-17 09:29:18 +0000
@@ -769,6 +769,8 @@
 char *opt_mysql_tmpdir = NULL;
 MY_TMPDIR mysql_tmpdir_list;
 
+const char *defaults_group = "mysqld";
+
 /* === static parameters in ha_innodb.cc */
 
 #define HA_INNOBASE_ROWS_IN_TABLE 10000 /* to get optimization right */
@@ -994,7 +996,8 @@
   OPT_INNODB_SYNC_SPIN_LOOPS,
   OPT_INNODB_THREAD_CONCURRENCY,
   OPT_INNODB_THREAD_SLEEP_DELAY,
-  OPT_XTRA_DEBUG_SYNC
+  OPT_XTRA_DEBUG_SYNC,
+  OPT_DEFAULTS_GROUP
 };
 
 static struct my_option my_long_options[] =
@@ -1278,6 +1281,9 @@
    0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0},
 #endif
 
+  {"defaults_group", OPT_DEFAULTS_GROUP, "defaults group in config file (default \"mysqld\").",
+   (G_PTR*) &defaults_group, (G_PTR*) &defaults_group,
+   0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0},
   { 0, 0, 0, 0, 0, 0, GET_NO_ARG, NO_ARG, 0, 0, 0, 0, 0, 0}
 };
 
@@ -1337,7 +1343,7 @@
 #endif
 }
 
-static const char *load_default_groups[]= { "mysqld","xtrabackup",0 };
+static const char *load_default_groups[]= { "mysqld","xtrabackup",0,0 };
 
 static void print_version(void)
 {
@@ -6280,6 +6286,17 @@
 	MY_INIT(argv[0]);
 	xb_regex_init();
 
+	/* scan options for group to load defaults from */
+	{
+		int	i;
+		char*	optend;
+		for (i=1; i < argc; i++) {
+			optend = strcend(argv[i], '=');
+			if (strncmp(argv[i], "--defaults-group", optend - argv[i]) == 0) {
+				load_default_groups[2] = defaults_group = optend + 1;
+			}
+		}
+	}
 	load_defaults("my",load_default_groups,&argc,&argv);
 
 	/* ignore unsupported options */
@@ -6333,7 +6350,7 @@
 	if ((ho_error=handle_options(&argc, &argv, my_long_options, get_one_option)))
 		exit(ho_error);
 
-	if ((!xtrabackup_prepare) && (strcmp(mysql_data_home, "./") == 0)) {
+	if ((!xtrabackup_print_param) && (!xtrabackup_prepare) && (strcmp(mysql_data_home, "./") == 0)) {
 		if (!xtrabackup_print_param)
 			usage();
 		msg("\nxtrabackup: Error: Please set parameter 'datadir'\n");
@@ -6518,7 +6535,7 @@
 			exit(EXIT_FAILURE);
 
 		printf("# This MySQL options file was generated by XtraBackup.\n");
-		printf("[mysqld]\n");
+		printf("[%s]\n", defaults_group);
 		printf("datadir = \"%s\"\n", mysql_data_home);
 		printf("tmpdir = \"%s\"\n", mysql_tmpdir_list.list[0]);
 		printf("innodb_data_home_dir = \"%s\"\n",

=== modified file 'test/t/bug983685.sh'
--- test/t/bug983685.sh	2012-04-17 12:55:16 +0000
+++ test/t/bug983685.sh	2012-05-17 09:29:18 +0000
@@ -6,7 +6,7 @@
 
 init
 
-options="innodb_data_file_path innodb_log_group_home_dir"
+options="innodb_data_file_path"
 
 run_mysqld
 

=== added file 'test/t/bug996493.sh'
--- test/t/bug996493.sh	1970-01-01 00:00:00 +0000
+++ test/t/bug996493.sh	2012-05-17 09:29:18 +0000
@@ -0,0 +1,39 @@
+########################################################################
+# Bug #996493: innobackupex --apply-log doesn't read config from
+# backup-my.cnf
+########################################################################
+
+. inc/common.sh
+
+init
+
+run_mysqld
+
+backup_dir=$topdir/backup
+innobackupex --no-timestamp $backup_dir
+vlog "Backup created in directory $backup_dir"
+
+vlog "Stop mysqld"
+stop_mysqld
+
+vlog "Remove datadir"
+rm -r $mysql_datadir/*
+
+vlog "Remove my.cnf"
+mv $topdir/my.cnf $topdir/my.cnf.bak
+
+trap "vlog restoring $topdir/my.cnf ; \
+mv $topdir/my.cnf.bak $topdir/my.cnf" INT TERM EXIT
+
+vlog "Apply log"
+# Do not run innobackupex, because it pass option --defaults-file
+# which we should avoid. Our goal is to test that backup-my.cnf
+# will be read by default when apply-log is run.
+run_cmd $IB_BIN --ibbackup=$XB_BIN --apply-log $backup_dir
+
+vlog "Get my.cnf back"
+mv $topdir/my.cnf.bak $topdir/my.cnf
+trap "vlog test exit" INT TERM EXIT
+
+vlog "Copy back"
+innobackupex --copy-back $backup_dir