Merge lp:~akopytov/percona-xtrabackup/replication-support-in-test-suite-1.6 into lp:percona-xtrabackup/1.6

[Alexey Kopytov]

[Alexey Kopytov (akopytov)]

http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-1.6-param/136/

[Stewart Smith (stewart)]

The method of trying to find a free TCP port is a concern to me... it looks racey - and in a impossible to determine way.

If we must have TCP, we'll have to do a mechanism similar to MTR and ensure we're the only user of it.

review: Needs Fixing

[Stewart Smith (stewart)]

basically, the racey bit is two test suite runs executing lock step, each will think the same port is free and try to run mysqld on it, causing one of them to fail.

[Alexey Kopytov (akopytov)]

Yes, I was looking at MTR when implementing get_free_port(). It uses absolutely the same probing method, so it has the same race. The only difference is that MTR tries to get a unique ID for "build thread" with the file locking primitives, which are unavailable in shell. So the XB test suite uses random numbers instead.

But anyway, MTR also "pings" the port and then tries starting the server on that port. I think he problem is not that serious in reality, with only a few possible concurrent runs, and only a few seconds between checking a port and using it.

[Stewart Smith (stewart)]

On Tue, 12 Jun 2012 07:42:22 -0000, Alexey Kopytov <email address hidden> wrote:
> Yes, I was looking at MTR when implementing get_free_port(). It uses absolutely the same probing method, so it has the same race. The only difference is that MTR tries to get a unique ID for "build thread" with the file locking primitives, which are unavailable in shell. So the XB test suite uses random numbers instead.
>
> But anyway, MTR also "pings" the port and then tries starting the
> server on that port. I think he problem is not that serious in
> reality, with only a few possible concurrent runs, and only a few
> seconds between checking a port and using it.

The lock in MTR should prevent the race though, as only one process is
ever going to get the lock.

We could do the next best thing by putting pid in a file, and if pid
exists, it is in use (for if we fail to cleanup file).

thoughts?

--
Stewart Smith

[Alexey Kopytov (akopytov)]

Actually, on a closer look, MTR picks unique IDs only for multiple threads within the same MTR run, i.e. for --parallel. It doesn't warrant any uniqueness with multiple concurrent MTR runs. Which means it is absolutely identical to what XB test suite does: just check if a port is open and hope nothing grabs it before we start the server on it.

[Alexey Kopytov (akopytov)]

Implemented "port locking" in free port selection code. It's still
racey, but it's the best thing you can do in bash.

[Alexey Kopytov (akopytov)]

http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-1.6-param/138/

[Stewart Smith (stewart)]

I think this is about as good as we're going to get.

review: Approve

=== modified file 'innobackupex'
--- innobackupex	2012-05-22 14:52:54 +0000
+++ innobackupex	2012-06-12 10:33:17 +0000
@@ -1202,28 +1202,21 @@
 #
 sub write_binlog_info {
     my @lines;
-    my @info_lines = ();
     my $position = '';
     my $filename = '';
 
     # get binlog position
-    mysql_send "SHOW MASTER STATUS;";
+    mysql_send 'SHOW MASTER STATUS\G';
 
-    # get "show master status" output lines (2) from mysql output
+    # get the name of the last binlog file and position in it
+    # from the SHOW MASTER STATUS output
     file_to_array($mysql_stdout, \@lines);
-    foreach my $line (@lines) {
-        if ($line =~ m/xtrabackup ping/) {
-            # this is a hello message, ignore it
-        } else {
-            # this is output line from "show master status"
-            push(@info_lines, $line);
-        }
+    foreach (@lines) {
+        $filename = $1 if /^\s+File:\s(\S+)\s*$/;
+        $position = $1 if /^\s+Position:\s(\d+)\s*$/;
     }
 
     # write binlog info file
-    if (!defined $info_lines[1]) {
-        $info_lines[1] = "";
-    }
     if (!$option_remote_host) {
         open(FILE, ">$binlog_info") || 
             Die "Failed to open file '$binlog_info': $!";
@@ -1231,7 +1224,7 @@
         open(FILE, "| ssh $option_ssh_opt $option_remote_host 'cat > $binlog_info'") ||
             Die "Failed to open file '$option_remote_host:$binlog_info': $!";
     }
-    print FILE  "$info_lines[1]\n";
+    print FILE  "$filename\t$position\t\t\n";
     close(FILE);
 
     if ($option_stream eq 'tar') {
@@ -1240,14 +1233,7 @@
         unlink $binlog_info || Die "Failed to delete '$binlog_info': $!";
     }
 
-    # get the name of the last binlog file and position in it
-    ($filename, $position) = $info_lines[1] =~ /^\s*([^\s]+)\s+(.*)$/;
-
-    if (defined $filename && defined $position) {
-        $mysql_binlog_position = "filename '$filename', position $position";
-    } else {
-        $mysql_binlog_position = "filename '', position ";
-    }        
+    $mysql_binlog_position = "filename '$filename', position $position";
 }
 
 
@@ -1265,7 +1251,7 @@
 	
     # get slave status. Use single quotes here, otherwise
     # \G is evaluated as a control character.
-    mysql_send 'SHOW SLAVE STATUS\G;';
+    mysql_send 'SHOW SLAVE STATUS\G';
 
     # get output of the "show slave status" command from mysql output
     # and extract binlog position of the master server

=== modified file 'test/disabled/ib_include.sh'
--- test/disabled/ib_include.sh	2011-03-16 19:41:17 +0000
+++ test/disabled/ib_include.sh	2012-06-12 10:33:17 +0000
@@ -1,7 +1,7 @@
 . inc/common.sh
 
-init
-run_mysqld
+start_server
+
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "create database include;"
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "create table t1 (a int) ENGINE=MyISAM;" include
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "create table t2 (a int) ENGINE=InnoDB;" include
@@ -15,7 +15,7 @@
 vlog "checksum_t2 is $checksum_t2"
 run_cmd ${IB_BIN} --user=root --socket=$mysql_socket --include="^include[.]t" $topdir/backup > $OUTFILE 2>&1 
 backup_dir=`grep "innobackupex: Backup created in directory" $OUTFILE | awk -F\' '{ print $2}'`
-stop_mysqld
+stop_server
 # Remove datadir
 rm -r $mysql_datadir
 # Restore data
@@ -30,7 +30,7 @@
 echo "# RESTORE #" >> $OUTFILE
 echo "###########" >> $OUTFILE
 run_cmd ${IB_BIN} --copy-back $backup_dir >> $OUTFILE 2>&1
-run_mysqld
+start_server
 checksum_tt1=`${MYSQL} ${MYSQL_ARGS} -Ns -e "checksum table t1" include | awk '{print $2}'`
 checksum_tt2=`${MYSQL} ${MYSQL_ARGS} -Ns -e "checksum table t2" include | awk '{print $2}'`
 vlog "checksum_tt1 is $checksum_tt1"
@@ -42,5 +42,3 @@
         vlog "Checksums are not equal"
         exit -1
 fi
-stop_mysqld
-clean

=== modified file 'test/experimental/xb_race_drop.sh'
--- test/experimental/xb_race_drop.sh	2011-06-10 06:11:56 +0000
+++ test/experimental/xb_race_drop.sh	2012-06-12 10:33:17 +0000
@@ -1,7 +1,31 @@
 . inc/common.sh
 
-init
-run_mysqld --innodb_file_per_table
+function race_create_drop()
+{
+  vlog "Started create/drop table cycle"
+  race_cycle_num=$1
+  if [ -z $race_cycle_num ]
+  then
+    race_cycle_num=1000
+  fi
+  run_cmd ${MYSQL} ${MYSQL_ARGS} -e "create database race;"
+  race_cycle_cnt=0;
+  while [ "$race_cycle_num" -gt "$race_cycle_cnt"]
+  do
+	t1=tr$RANDOM
+	t2=tr$RANDOM
+	t3=tr$RANDOM
+	${MYSQL} ${MYSQL_ARGS} -e "create table $t1 (a int) ENGINE=InnoDB;" race
+	${MYSQL} ${MYSQL_ARGS} -e "create table $t2 (a int) ENGINE=InnoDB;" race
+	${MYSQL} ${MYSQL_ARGS} -e "create table $t3 (a int) ENGINE=InnoDB;" race
+	${MYSQL} ${MYSQL_ARGS} -e "drop table $t1;" race
+	${MYSQL} ${MYSQL_ARGS} -e "drop table $t2;" race
+	${MYSQL} ${MYSQL_ARGS} -e "drop table $t3;" race
+	let "race_cycle_cnt=race_cycle_cnt+1"
+  done
+}
+
+start_server --innodb_file_per_table
 
 run_cmd race_create_drop &
 

=== modified file 'test/inc/common.sh'
--- test/inc/common.sh	2011-12-21 16:40:27 +0000
+++ test/inc/common.sh	2012-06-12 10:33:17 +0000
@@ -17,25 +17,10 @@
     echo "`date +"%F %T"`: `basename "$0"`: $@" >&2
 }
 
-function clean_datadir()
-{
-    vlog "Removing MySQL data directory: $mysql_datadir"
-    rm -rf "$mysql_datadir"
-    vlog "Creating MySQL data directory: $mysql_datadir"
-    mkdir -p "$mysql_datadir"
-    init_mysql_dir
-}
 
 function clean()
 {
-    vlog "Removing temporary $topdir"
-    rm -rf "$topdir"
-}
-function clean_on_error()
-{
-vlog "Exit on error"
-clean
-exit -1
+    rm -rf ${TEST_BASEDIR}/var[0-9]
 }
 
 function die()
@@ -44,86 +29,46 @@
   exit 1
 }
 
-function initdir()
-{
-    if test -d "$topdir"
-    then
-        vlog "Directory $topdir exists. Removing it..."
-        rm -r "$topdir"
-    fi
-    vlog "Creating temporary directory: $topdir"
-    mkdir -p "$topdir/tmp"
-    vlog "Creating MySQL data directory: $mysql_datadir"
-    mkdir -p "$mysql_datadir"
-}
-
 function init_mysql_dir()
 {
-    vlog "Creating MySQL database"
-    $MYSQL_INSTALL_DB --no-defaults --basedir=$MYSQL_BASEDIR --datadir="$mysql_datadir" --tmpdir="$mysql_tmpdir"
-}
-function set_mysl_port()
-{
-    i=$mysql_port
-    while [ $i -lt 65536 ]
-    do
-        # check if port $i is used
-        vlog "Checking port $i"
-        port_status=`netstat -an | grep LISTEN | grep tcp | grep ":$i " || true`
-        if test -z "$port_status"
-        then
-            # port is not used
-            vlog "Port $i is free"
-            mysql_port=$i
-            break
-        else
-            vlog "Port $i is used"
-            let i=$i+1
-        fi
-    done
+    if [ ! -d "$MYSQLD_VARDIR" ]
+    then
+	vlog "Creating server root directory: $MYSQLD_VARDIR"
+	mkdir "$MYSQLD_VARDIR"
+    fi
+    if [ ! -d "$MYSQLD_TMPDIR" ]
+    then
+	vlog "Creating server temporary directory: $MYSQLD_TMPDIR"
+	mkdir "$MYSQLD_TMPDIR"
+    fi
+    if [ ! -d "$MYSQLD_DATADIR" ]
+    then
+	vlog "Creating server data directory: $MYSQLD_DATADIR"
+	mkdir -p "$MYSQLD_DATADIR"
+	vlog "Calling mysql_install_db"
+	$MYSQL_INSTALL_DB --no-defaults --basedir=$MYSQL_BASEDIR --datadir="$MYSQLD_DATADIR" --tmpdir="$MYSQLD_TMPDIR"
+    fi
 }
 
+########################################################################
 # Checks whether MySQL is alive
+########################################################################
 function mysql_ping()
 {
-    local result="0"
-    if test -S ${mysql_socket}
-    then
-        result=`${MYSQL} ${MYSQL_ARGS} -e "SELECT IF(COUNT(*)>=0, 1, 1) FROM user;" -s --skip-column-names mysql 2>/dev/null` || result="0"
-    else
-        result="0"
-    fi
-    echo $result
+    $MYSQLADMIN $MYSQL_ARGS --wait=100 ping >/dev/null 2>&1 || return 1
 }
 
 function kill_leftovers()
 {
-    while test -f "$PWD/mysqld.pid" 
-    do
-	vlog "Found a leftover mysqld processes with PID `cat $PWD/mysqld.pid`, stopping it"
-	stop_mysqld 2> /dev/null
-    done
-}
-
-function run_mysqld()
-{
-    local c=0
-    kill_leftovers
-    ${MYSQLD} ${MYSQLD_ARGS} $* --pid-file="$PWD/mysqld.pid" &
-    while [ "`mysql_ping`" != "1" ]
-    do
-        if [ ${c} -eq 100 ]
-        then
-            vlog "Can't run MySQL!"
-            clean_on_error
-        fi
-        let c=${c}+1
-        sleep 1
-    done
-    if [ -n "$MYSQL_VERSION" -a -n "$INNODB_VERSION" ]
-    then
-	vlog "MySQL $MYSQL_VERSION (InnoDB $INNODB_VERSION) started successfully"
-    fi
+    local file
+    for file in ${TEST_BASEDIR}/mysqld*.pid
+    do
+	if [ -f $file ]
+	then
+	    vlog "Found a leftover mysqld processes with PID `cat $file`, stopping it"
+	    kill -9 `cat $file` 2>/dev/null || true
+	fi
+    done
 }
 
 function run_cmd()
@@ -152,21 +97,6 @@
   fi
 }
 
-function stop_mysqld()
-{
-    if [ -f "$PWD/mysqld.pid" ]
-    then
-	${MYSQLADMIN} ${MYSQL_ARGS} shutdown 
-	vlog "Database server has been stopped"
-	if [ -f "$PWD/mysqld.pid" ]
-	then
-	    sleep 1;
-	    kill -9 `cat $PWD/mysqld.pid`
-	    rm -f $PWD/mysqld.pid
-	fi
-    fi
-}
-
 function load_sakila()
 {
 vlog "Loading sakila"
@@ -196,39 +126,236 @@
   run_cmd ${MYSQL} ${MYSQL_ARGS} -e "DROP DATABASE $1"
 }
 
-function init()
-{
-initdir
-init_mysql_dir
-echo "
+########################################################################
+# Choose a free port for a MySQL server instance
+########################################################################
+function get_free_port()
+{
+    local id=$1
+    local port
+    local lockfile
+
+    for (( port=PORT_BASE+id; port < 65535; port++))
+    do
+	lockfile="/tmp/xtrabackup_port_lock.$port"
+	# Try to atomically lock the current port number
+	if ! set -C > $lockfile
+	then
+	    set +C
+	    # check if the process still exists
+	    if kill -0 "`cat $lockfile`"
+	    then
+		continue;
+	    fi
+	    # stale file, overwrite it with the current PID
+	fi
+	set +C
+	echo $$ > $lockfile
+	if ! nc -z -w1 localhost $port >/dev/null 2>&1
+	then
+	    echo $port
+	    return 0
+	fi
+	rm -f $lockfile
+    done
+
+    die "Could not find a free port for server id $id!"
+}
+
+########################################################################
+# Initialize server variables such as datadir, tmpdir, etc. and store
+# them with the specified index in SRV_MYSQLD_* arrays to be used by
+# switch_server() later
+########################################################################
+function init_server_variables()
+{
+    local id=$1
+
+    if [ -n ${SRV_MYSQLD_IDS[$id]:-""} ]
+    then
+	die "Server with id $id has already been started"
+    fi
+
+    SRV_MYSQLD_IDS[$id]="$id"
+    local vardir="${TEST_BASEDIR}/var${id}"
+    SRV_MYSQLD_VARDIR[$id]="$vardir"
+    SRV_MYSQLD_DATADIR[$id]="$vardir/data"
+    SRV_MYSQLD_TMPDIR[$id]="$vardir/tmp"
+    SRV_MYSQLD_PIDFILE[$id]="${TEST_BASEDIR}/mysqld${id}.pid"
+    SRV_MYSQLD_PORT[$id]=`get_free_port $id`
+    SRV_MYSQLD_SOCKET[$id]=`mktemp -t xtrabackup.mysql.sock.XXXXXX`
+}
+
+########################################################################
+# Reset server variables
+########################################################################
+function reset_server_variables()
+{
+    local id=$1
+
+    if [ -z ${SRV_MYSQLD_VARDIR[$id]:-""} ]
+    then
+	# Variables have already been reset
+	return 0;
+    fi
+
+    SRV_MYSQLD_IDS[$id]=
+    SRV_MYSQLD_VARDIR[$id]=
+    SRV_MYSQLD_DATADIR[$id]=
+    SRV_MYSQLD_TMPDIR[$id]=
+    SRV_MYSQLD_PIDFILE[$id]=
+    SRV_MYSQLD_PORT[$id]=
+    SRV_MYSQLD_SOCKET[$id]=
+}
+
+##########################################################################
+# Change the environment to make all utilities access the server with an
+# id specified in the argument.
+##########################################################################
+function switch_server()
+{
+    local id=$1
+
+    MYSQLD_VARDIR="${SRV_MYSQLD_VARDIR[$id]}"
+    MYSQLD_DATADIR="${SRV_MYSQLD_DATADIR[$id]}"
+    MYSQLD_TMPDIR="${SRV_MYSQLD_TMPDIR[$id]}"
+    MYSQLD_PIDFILE="${SRV_MYSQLD_PIDFILE[$id]}"
+    MYSQLD_PORT="${SRV_MYSQLD_PORT[$id]}"
+    MYSQLD_SOCKET="${SRV_MYSQLD_SOCKET[$id]}"
+
+    MYSQL_ARGS="--no-defaults --socket=${MYSQLD_SOCKET} --user=root"
+    MYSQLD_ARGS="--no-defaults --basedir=${MYSQL_BASEDIR} \
+--socket=${MYSQLD_SOCKET} --port=${MYSQLD_PORT} --server-id=$id \
+--datadir=${MYSQLD_DATADIR} --tmpdir=${MYSQLD_TMPDIR} --log-bin=mysql-bin \
+--relay-log=mysql-relay-bin --pid-file=${MYSQLD_PIDFILE} ${MYSQLD_EXTRA_ARGS}"
+    if [ "`whoami`" = "root" ]
+    then
+	MYSQLD_ARGS="$MYSQLD_ARGS --user=root"
+    fi
+
+    IB_ARGS="--defaults-file=${MYSQLD_VARDIR}/my.cnf --user=root \
+--socket=${MYSQLD_SOCKET} --ibbackup=$XB_BIN"
+    XB_ARGS="--no-defaults"
+
+    # Some aliases for compatibility, as tests use the following names
+    topdir="$MYSQLD_VARDIR"
+    mysql_datadir="$MYSQLD_DATADIR"
+    mysql_tmpdir="$MYSQLD_TMPDIR"
+    mysql_socket="$MYSQLD_SOCKET"
+}
+
+########################################################################
+# Start server with the id specified as the first argument
+########################################################################
+function start_server_with_id()
+{
+    local id=$1
+    shift
+
+    vlog "Starting server with id=$id..."
+
+    init_server_variables $id
+    switch_server $id
+
+    init_mysql_dir
+    cat > ${MYSQLD_VARDIR}/my.cnf <<EOF
 [mysqld]
-datadir=$mysql_datadir
-tmpdir=$mysql_tmpdir" > $topdir/my.cnf
-}
-
-function race_create_drop()
-{
-  vlog "Started create/drop table cycle"
-  race_cycle_num=$1
-  if [ -z $race_cycle_num ]
-  then
-    race_cycle_num=1000
-  fi
-  run_cmd ${MYSQL} ${MYSQL_ARGS} -e "create database race;"
-  race_cycle_cnt=0;
-  while [ "$race_cycle_num" -gt "$race_cycle_cnt"]
-  do
-	t1=tr$RANDOM
-	t2=tr$RANDOM
-	t3=tr$RANDOM
-	${MYSQL} ${MYSQL_ARGS} -e "create table $t1 (a int) ENGINE=InnoDB;" race
-	${MYSQL} ${MYSQL_ARGS} -e "create table $t2 (a int) ENGINE=InnoDB;" race
-	${MYSQL} ${MYSQL_ARGS} -e "create table $t3 (a int) ENGINE=InnoDB;" race
-	${MYSQL} ${MYSQL_ARGS} -e "drop table $t1;" race
-	${MYSQL} ${MYSQL_ARGS} -e "drop table $t2;" race
-	${MYSQL} ${MYSQL_ARGS} -e "drop table $t3;" race
-	let "race_cycle_cnt=race_cycle_cnt+1"
-  done
+datadir=${MYSQLD_DATADIR}
+tmpdir=${MYSQLD_TMPDIR}
+EOF
+
+    # Start the server
+    ${MYSQLD} ${MYSQLD_ARGS} $* &
+    if ! mysql_ping
+    then
+        die "Can't start the server!"
+    fi
+    vlog "Server with id=$id has been started on port $MYSQLD_PORT, \
+socket $MYSQLD_SOCKET"
+}
+
+########################################################################
+# Stop server with the id specified as the first argument and additional
+# command line arguments (if specified)
+########################################################################
+function stop_server_with_id()
+{
+    local id=$1
+    switch_server $id
+
+    vlog "Stopping server with id=$id..."
+
+    if [ -f "${MYSQLD_PIDFILE}" ]
+    then
+	${MYSQLADMIN} ${MYSQL_ARGS} --shutdown-timeout=60 shutdown
+	if [ -f "${MYSQLD_PIDFILE}" ]
+	then
+	    vlog "Could not stop the server with id=$id, using kill -9"
+	    kill -9 `cat ${MYSQLD_PIDFILE}`
+	    rm -f ${MYSQLD_PIDFILE}
+	fi
+	vlog "Server with id=$id has been stopped"
+    else
+	vlog "Server pid file '${MYSQLD_PIDFILE}' doesn't exist!"
+    fi
+
+    # unlock the port number
+    rm -f /tmp/xtrabackup_port_lock.$MYSQLD_PORT
+
+    reset_server_variables $id
+}
+
+########################################################################
+# Start server with id=1 and additional command line arguments
+# (if specified)
+########################################################################
+function start_server()
+{
+    start_server_with_id 1 $*
+}
+
+########################################################################
+# Stop server with id=1
+########################################################################
+function stop_server()
+{
+    stop_server_with_id 1
+}
+
+########################################################################
+# Stop all running servers
+########################################################################
+function stop_all_servers()
+{
+    local id
+    for id in ${SRV_MYSQLD_IDS[*]}
+    do
+	stop_server_with_id ${SRV_MYSQLD_IDS[$id]}
+    done
+}
+
+########################################################################
+# Configure a specified server as a slave
+# Synopsis:
+#   setup_slave <slave_id> <master_id>
+#########################################################################
+function setup_slave()
+{
+    local slave_id=$1
+    local master_id=$2
+
+    vlog "Setting up server #$slave_id as a slave of server #$master_id"
+
+    switch_server $slave_id
+
+    run_cmd $MYSQL $MYSQL_ARGS <<EOF
+CHANGE MASTER TO
+  MASTER_HOST='localhost',
+  MASTER_USER='root',
+  MASTER_PORT=${SRV_MYSQLD_PORT[$master_id]};
+
+START SLAVE
+EOF
 }
 
 ########################################################################
@@ -239,3 +366,6 @@
 {
     $MYSQL $MYSQL_ARGS -Ns -e "CHECKSUM TABLE $2" $1 | awk {'print $2'}
 }
+
+# To avoid unbound variable error when no server have been started
+SRV_MYSQLD_IDS=

=== modified file 'test/run.sh'
--- test/run.sh	2011-07-05 03:02:01 +0000
+++ test/run.sh	2012-06-12 10:33:17 +0000
@@ -1,5 +1,7 @@
 #!/bin/bash
 
+export DEBUG=
+
 . inc/common.sh
 
 set +e
@@ -14,7 +16,7 @@
 Usage: $0 [-f] [-g] [-h] [-s suite] [-t test_name] [-d mysql_basedir] [-c build_conf]
 -f          Continue running tests after failures
 -d path     Server installation directory. Default is './server'.
--g          Output debug information to results/*.out
+-g          Debug mode
 -t path     Run only a single named test
 -h          Print this help message
 -s suite    Select a test suite to run. Possible values: experimental, t. 
@@ -52,13 +54,9 @@
 
 function set_vars()
 {
-    topdir="`pwd`/var"
-    mysql_datadir="$topdir/mysql"
-    mysql_tmpdir="$topdir/tmp"
-    mysql_port="3306"
-    mysql_socket=`mktemp -t xtrabackup.mysql.sock.XXXXXX`
-    IB_ARGS="--defaults-file=$topdir/my.cnf --user=root --socket=$mysql_socket"
-    XB_ARGS="--no-defaults"
+    TEST_BASEDIR="$PWD"
+    MYSQL_BASEDIR=${MYSQL_BASEDIR:-"$PWD/server"}
+    PORT_BASE=$((3306 + $RANDOM))
 
     if gnutar --version > /dev/null 2>&1
     then
@@ -67,16 +65,6 @@
 	TAR=tar
     fi
 
-    MYSQL_BASEDIR=${MYSQL_BASEDIR:-"$PWD/server"}
-
-    MYSQL_ARGS="--no-defaults --socket=${mysql_socket} --user=root"
-
-    MYSQLD_ARGS="--no-defaults --basedir=${MYSQL_BASEDIR} --socket=${mysql_socket} --datadir=$mysql_datadir --tmpdir=$mysql_tmpdir --skip-networking"
-    if [ "`whoami`" = "root" ]
-    then
-	MYSQLD_ARGS="$MYSQLD_ARGS --user=root"
-    fi
-
     find_program MYSQL_INSTALL_DB mysql_install_db $MYSQL_BASEDIR/bin \
 	$MYSQL_BASEDIR/scripts
     find_program MYSQLD mysqld $MYSQL_BASEDIR/bin/ $MYSQL_BASEDIR/libexec
@@ -92,22 +80,26 @@
 
     PATH="${MYSQL_BASEDIR}/bin:$PATH"
 
-    export topdir mysql_datadir mysql_tmpdir mysql_port mysql_socket OUTFILE \
-	IB_ARGS XB_ARGS TAR MYSQL_BASEDIR MYSQL MYSQLADMIN \
-	MYSQL_ARGS MYSQLD_ARGS MYSQL_INSTALL_DB MYSQLD PATH
+    export TEST_BASEDIR PORT_BASE TAR MYSQL_BASEDIR MYSQL MYSQLD MYSQLADMIN \
+MYSQL_INSTALL_DB PATH
 }
 
 function get_version_info()
 {
+    MYSQLD_EXTRA_ARGS=
+
+    XB_BIN=""
+    IB_ARGS="--user=root --ibbackup=$XB_BIN"
+    XB_ARGS="--no-defaults"
+
     if [ "$XB_BUILD" != "autodetect" ]
     then
-	XB_BIN=""
 	case "$XB_BUILD" in
 	    "innodb51_builtin" )
 		XB_BIN="xtrabackup_51";;
 	    "innodb51" )
 		XB_BIN="xtrabackup_plugin"
-		MYSQLD_ARGS="$MYSQLD_ARGS --ignore-builtin-innodb --plugin-load=innodb=ha_innodb_plugin.so";;
+		MYSQLD_EXTRA_ARGS="--ignore-builtin-innodb --plugin-load=innodb=ha_innodb_plugin.so";;
 	    "innodb55" )
 		XB_BIN="xtrabackup_innodb55";;
 	    "xtradb51" )
@@ -122,11 +114,11 @@
 	fi
     fi
 
-    init
-
     MYSQL_VERSION=""
     INNODB_VERSION=""
-    run_mysqld
+
+    start_server
+
     # Get MySQL and InnoDB versions
     MYSQL_VERSION=`$MYSQL ${MYSQL_ARGS} -Nsf -e "SHOW VARIABLES LIKE 'version'"`
     MYSQL_VERSION=${MYSQL_VERSION#"version	"}
@@ -176,17 +168,17 @@
 	vlog "Cannot find 'innobackupex' in PATH"
 	return 1
     fi
-    IB_ARGS="$IB_ARGS --ibbackup=$XB_BIN"
 
     vlog "Using '$XB_BIN' as xtrabackup binary"
 
+    stop_server
+
     export MYSQL_VERSION MYSQL_VERSION_COMMENT INNODB_VERSION XTRADB_VERSION \
-	XB_BIN IB_BIN IB_ARGS
+	XB_BIN IB_BIN IB_ARGS XB_ARGS MYSQLD_EXTRA_ARGS
 }
 
 export SKIPPED_EXIT_CODE=200
 
-
 tname=""
 XTRACE_OPTION=""
 XB_BUILD="autodetect"
@@ -195,7 +187,7 @@
 	case $options in
 	        f ) force="yes";;
 		t ) tname="$OPTARG";;
-		g ) XTRACE_OPTION="-x";;
+		g ) XTRACE_OPTION="-x"; DEBUG=on;;
 		h ) usage; exit;;
 		s ) tname="$OPTARG/*.sh";;
 	        d ) export MYSQL_BASEDIR="$OPTARG";;
@@ -219,6 +211,9 @@
 total_count=0
 
 export OUTFILE="$PWD/results/setup"
+
+clean >>$OUTFILE 2>&1
+
 if ! get_version_info
 then
     echo "get_version_info failed. See $OUTFILE for details."
@@ -253,6 +248,12 @@
    bash $XTRACE_OPTION $t > $OUTFILE 2>&1
    rc=$?
 
+   if [[ -z "$DEBUG" || -n "$force" ]]
+   then
+       kill_leftovers >>$OUTFILE 2>&1
+       clean >>$OUTFILE 2>&1
+   fi
+
    if [ $rc -eq 0 ]
    then
        echo "[passed]"
@@ -279,17 +280,9 @@
        then
  	   break;
        fi
-   fi
-       
-   stop_mysqld >/dev/null 2>&1
-   clean >/dev/null 2>&1
+   fi       
 done
 
-if [ -z "$force" -o $failed_count -eq 0 ]
-then
-    kill_leftovers
-fi
-
 if [ $result -eq 1 ]
 then
     echo

=== modified file 'test/t/bug1002688.sh'
--- test/t/bug1002688.sh	2012-05-22 14:52:54 +0000
+++ test/t/bug1002688.sh	2012-06-12 10:33:17 +0000
@@ -3,8 +3,7 @@
 ############################################################################
 . inc/common.sh
 
-init
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 load_sakila
 
 # Full backup
@@ -44,7 +43,7 @@
 vlog "Data prepared for restore"
 
 # Destroying mysql data
-stop_mysqld
+stop_server
 rm -rf $mysql_datadir/*
 vlog "Data destroyed"
 
@@ -53,7 +52,7 @@
 innobackupex --copy-back $full_backup_dir
 vlog "Data restored"
 
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 
 vlog "Checking checksums"
 checksum_b=`checksum_table newdb actor_copy`

=== modified file 'test/t/bug514068.sh'
--- test/t/bug514068.sh	2011-12-05 12:35:43 +0000
+++ test/t/bug514068.sh	2012-06-12 10:33:17 +0000
@@ -5,12 +5,11 @@
 
 . inc/common.sh
 
-init
-run_mysqld
+start_server
 
 innobackupex  --no-timestamp $topdir/backup >$topdir/stdout 2>$topdir/stderr
 
-stop_mysqld
+stop_server
 # Remove datadir
 rm -r $mysql_datadir
 

=== modified file 'test/t/bug722638.sh'
--- test/t/bug722638.sh	2011-09-14 15:32:21 +0000
+++ test/t/bug722638.sh	2012-06-12 10:33:17 +0000
@@ -3,15 +3,14 @@
 #              created breaks backup
 ########################################################################
 
-if ! `$XB_BIN --help | grep debug-sync > /dev/null`; then
+. inc/common.sh
+
+if ! $XB_BIN --help 2>&1 | grep -q debug-sync; then
     echo "Requires --debug-sync support" > $SKIPPED_REASON
     exit $SKIPPED_EXIT_CODE
 fi
 
-. inc/common.sh
-
-init
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 
 run_cmd $MYSQL $MYSQL_ARGS test <<EOF
 
@@ -88,14 +87,14 @@
 # Prepare
 xtrabackup --datadir=$mysql_datadir --prepare --target-dir=$topdir/backup
 
-stop_mysqld
+stop_server
 
 # Restore
 rm -rf $mysql_datadir/ibdata1 $mysql_datadir/ib_logfile* \
     $mysql_datadir/test/*.ibd
 cp -r $topdir/backup/* $mysql_datadir
 
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 
 # Verify checksums
 checksum_t1_new=`checksum_table test t1`

=== modified file 'test/t/bug723097.sh'
--- test/t/bug723097.sh	2011-12-21 16:40:27 +0000
+++ test/t/bug723097.sh	2012-06-12 10:33:17 +0000
@@ -1,7 +1,6 @@
 . inc/common.sh
 
-init
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 
 vlog "Loading data from sql file"
 run_cmd ${MYSQL} ${MYSQL_ARGS} test < t/bug723097.sql
@@ -17,15 +16,15 @@
 vlog "Backup is done"
 xtrabackup --datadir=$mysql_datadir --prepare --target-dir=$topdir/data/full
 vlog "Data prepared fo restore"
-stop_mysqld
+stop_server
 
 cd $topdir/data/full
 cp -r * $mysql_datadir
 cd -
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 checksum_b=`checksum_table test messages`
 vlog "Checksum after is $checksum_b"
-stop_mysqld
+stop_server
 
 if [ "$checksum_a" -eq "$checksum_b" ]
 then
@@ -34,5 +33,3 @@
 	vlog "Checksums are not equal"
 	exit -1
 fi
-
-clean 

=== modified file 'test/t/bug729843.sh'
--- test/t/bug729843.sh	2011-11-25 18:50:56 +0000
+++ test/t/bug729843.sh	2012-06-12 10:33:17 +0000
@@ -4,8 +4,7 @@
 
 . inc/common.sh
 
-init
-run_mysqld
+start_server
 
 mkdir $topdir/backup
 logfile=$topdir/backup/innobackupex_log

=== modified file 'test/t/bug733651.sh'
--- test/t/bug733651.sh	2012-01-31 11:15:25 +0000
+++ test/t/bug733651.sh	2012-06-12 10:33:17 +0000
@@ -5,14 +5,12 @@
 
 . inc/common.sh
 
-init
-
 options="innodb_log_files_in_group innodb_log_file_size"
 if [ ! -z "$XTRADB_VERSION" ]; then
     options="$options innodb_page_size innodb_fast_checksum innodb_log_block_size"
 fi
 
-run_mysqld
+start_server
 
 mkdir -p $topdir/backup
 innobackupex  $topdir/backup

=== modified file 'test/t/bug759225.sh'
--- test/t/bug759225.sh	2011-09-21 09:00:00 +0000
+++ test/t/bug759225.sh	2012-06-12 10:33:17 +0000
@@ -16,15 +16,14 @@
     exit $SKIPPED_EXIT_CODE
 fi
 
-init
-run_mysqld
+start_server
 load_sakila
 
 # Take backup
 echo "innodb_flush_method=ALL_O_DIRECT" >> $topdir/my.cnf
 mkdir -p $topdir/backup
 innobackupex --stream=tar $topdir/backup > $topdir/backup/out.tar
-stop_mysqld
+stop_server
 
 # See if tar4ibd was using O_DIRECT
 if ! grep "tar4ibd: using O_DIRECT for the input file" $OUTFILE ;
@@ -46,6 +45,6 @@
 mkdir -p $mysql_datadir
 innobackupex --copy-back --defaults-file=$topdir/my.cnf $backup_dir
 
-run_mysqld
+start_server
 # Check sakila
 ${MYSQL} ${MYSQL_ARGS} -e "SELECT count(*) from actor" sakila

=== modified file 'test/t/bug759701.sh'
--- test/t/bug759701.sh	2011-12-21 16:40:27 +0000
+++ test/t/bug759701.sh	2012-06-12 10:33:17 +0000
@@ -4,8 +4,7 @@
 ##########################################################################
 . inc/common.sh
 
-init
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 load_sakila
 
 # Full backup
@@ -45,7 +44,7 @@
 vlog "Data prepared for restore"
 
 # Destroying mysql data
-stop_mysqld
+stop_server
 rm -rf $mysql_datadir/*
 vlog "Data destroyed"
 
@@ -54,7 +53,7 @@
 innobackupex --copy-back $full_backup_dir
 vlog "Data restored"
 
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 
 vlog "Checking checksums"
 checksum_b=`checksum_table newdb actor_copy`

=== modified file 'test/t/bug766607.sh'
--- test/t/bug766607.sh	2011-07-05 03:02:01 +0000
+++ test/t/bug766607.sh	2012-06-12 10:33:17 +0000
@@ -1,7 +1,6 @@
 . inc/common.sh
 
-init
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 load_dbase_schema incremental_sample
 
 # Full backup dir
@@ -18,8 +17,8 @@
 FLUSH LOGS;
 EOF
 
-stop_mysqld
-run_mysqld --innodb_file_per_table
+stop_server
+start_server --innodb_file_per_table
 
 vlog "Making incremental backup"
 xtrabackup --datadir=$mysql_datadir --backup --target-dir=$topdir/data/delta --incremental-basedir=$topdir/data/full
@@ -37,7 +36,7 @@
 xtrabackup --datadir=$mysql_datadir --prepare --target-dir=$topdir/data/full
 vlog "Data prepared for restore"
 
-stop_mysqld
+stop_server
 
 vlog "Copying files"
 
@@ -46,6 +45,6 @@
 cd $topdir
 
 vlog "Data restored"
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 
 run_cmd $MYSQL $MYSQL_ARGS -e "SELECT * FROM t" test

=== modified file 'test/t/bug810269.sh'
--- test/t/bug810269.sh	2011-09-14 12:48:27 +0000
+++ test/t/bug810269.sh	2012-06-12 10:33:17 +0000
@@ -5,14 +5,12 @@
 
 . inc/common.sh
 
-init
-
 if [ -z "$INNODB_VERSION" ]; then
     echo "Requires InnoDB plugin or XtraDB" >$SKIPPED_REASON
     exit $SKIPPED_EXIT_CODE
 fi
 
-run_mysqld "--innodb_strict_mode --innodb_file_per_table \
+start_server "--innodb_strict_mode --innodb_file_per_table \
 --innodb_file_format=Barracuda"
 
 load_dbase_schema incremental_sample
@@ -54,7 +52,7 @@
 
 innobackupex --stream=tar $topdir/backup > $topdir/backup/out.tar
 
-stop_mysqld
+stop_server
 rm -rf $mysql_datadir
 
 vlog "Applying log"
@@ -68,7 +66,7 @@
 mkdir -p $mysql_datadir
 innobackupex --copy-back $topdir/backup
 
-run_mysqld
+start_server
 
 checksum_b=`checksum_table incremental_sample test`
 

=== modified file 'test/t/bug817132.sh'
--- test/t/bug817132.sh	2011-12-07 13:28:47 +0000
+++ test/t/bug817132.sh	2012-06-12 10:33:17 +0000
@@ -12,8 +12,7 @@
     exit $SKIPPED_EXIT_CODE
 fi
 
-init
-run_mysqld
+start_server
 load_dbase_schema sakila
 load_dbase_data sakila
 
@@ -22,7 +21,7 @@
 backup_dir=`grep "innobackupex: Backup created in directory" $OUTFILE | awk -F\' '{ print $2}'`
 vlog "Backup created in directory $backup_dir"
 
-stop_mysqld
+stop_server
 # Remove datadir
 rm -r $mysql_datadir
 
@@ -37,6 +36,6 @@
 # --ibbackup explicitly (see $IB_ARGS).
 run_cmd $IB_BIN --defaults-file=$topdir/my.cnf --user=root --socket=$mysql_socket --copy-back $backup_dir
 
-run_mysqld
+start_server
 # Check sakila
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "SELECT count(*) from actor" sakila

=== modified file 'test/t/bug884737.sh'
--- test/t/bug884737.sh	2011-11-01 16:17:02 +0000
+++ test/t/bug884737.sh	2012-06-12 10:33:17 +0000
@@ -5,8 +5,7 @@
 
 . inc/common.sh
 
-init
-run_mysqld 
+start_server
 
 # Check that --parallel=<negative value> doesn't blow up
 vlog "Creating the backup directory: $topdir/backup"

=== modified file 'test/t/bug891496.sh'
--- test/t/bug891496.sh	2012-01-16 11:49:20 +0000
+++ test/t/bug891496.sh	2012-06-12 10:33:17 +0000
@@ -4,17 +4,14 @@
 
 . inc/common.sh
 
-init
-
 innodb_data_file_path="ibdata1:3M;ibdata2:10M:autoextend"
 
+start_server --innodb_data_file_path=$innodb_data_file_path
+
 cat >> $topdir/my.cnf <<EOF
 innodb_data_file_path=$innodb_data_file_path
 EOF
 
-MYSQLD_ARGS="$MYSQLD_ARGS --innodb_data_file_path=$innodb_data_file_path"
-
-run_mysqld
 load_dbase_schema sakila
 load_dbase_data sakila
 
@@ -22,7 +19,7 @@
 mkdir -p $topdir/backup
 innobackupex --stream=tar $topdir/backup > $topdir/backup/out.tar
 
-stop_mysqld
+stop_server
 # Remove datadir
 rm -r $mysql_datadir
 # Restore sakila
@@ -42,6 +39,6 @@
 vlog "###########"
 innobackupex  --copy-back $backup_dir
 
-run_mysqld
+start_server --innodb_data_file_path=$innodb_data_file_path
 # Check sakila
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "SELECT count(*) from actor" sakila

=== added file 'test/t/bug977101.sh'
--- test/t/bug977101.sh	1970-01-01 00:00:00 +0000
+++ test/t/bug977101.sh	2012-06-12 10:33:17 +0000
@@ -0,0 +1,31 @@
+########################################################################
+# Bug #977101: --safe-slave-backup results in incorrect binlog info
+########################################################################
+
+. inc/common.sh
+
+master_id=1
+slave_id=2
+
+start_server_with_id $master_id
+start_server_with_id $slave_id
+
+setup_slave $slave_id $master_id
+
+# Full backup of the slave server
+switch_server $slave_id
+
+# Check that binlog info is correct with --safe-slave-backup
+innobackupex --no-timestamp --safe-slave-backup $topdir/backup
+egrep -q '^mysql-bin.000001[[:space:]]+[0-9]+[[:space:]]+$' \
+    $topdir/backup/xtrabackup_binlog_info
+
+# Check that both binlog info and slave info are correct with 
+# --safe-slave-backup
+rm -rf $topdir/backup
+innobackupex --no-timestamp --slave-info --safe-slave-backup $topdir/backup
+egrep -q '^mysql-bin.000001[[:space:]]+[0-9]+[[:space:]]+$' \
+    $topdir/backup/xtrabackup_binlog_info
+egrep -q '^CHANGE MASTER TO MASTER_LOG_FILE='\''mysql-bin.000001'\'', MASTER_LOG_POS=[0-9]+$' \
+    $topdir/backup/xtrabackup_slave_info
+

=== modified file 'test/t/bug983685.sh'
--- test/t/bug983685.sh	2012-04-17 12:44:49 +0000
+++ test/t/bug983685.sh	2012-06-12 10:33:17 +0000
@@ -4,12 +4,10 @@
 
 . inc/common.sh
 
-init
+start_server
 
 options="innodb_data_file_path innodb_log_group_home_dir"
 
-run_mysqld
-
 mkdir -p $topdir/backup
 innobackupex  $topdir/backup
 backup_dir=`grep "innobackupex: Backup created in directory" $OUTFILE | awk -F\' '{ print $2}'`

=== modified file 'test/t/ib_binlog_info.sh'
--- test/t/ib_binlog_info.sh	2011-12-21 05:54:55 +0000
+++ test/t/ib_binlog_info.sh	2012-06-12 10:33:17 +0000
@@ -4,8 +4,7 @@
 
 . inc/common.sh
 
-init
-run_mysqld
+start_server
 load_sakila
 
 innobackupex --no-timestamp $topdir/backup

=== modified file 'test/t/ib_csm_csv.sh'
--- test/t/ib_csm_csv.sh	2011-12-07 13:28:47 +0000
+++ test/t/ib_csm_csv.sh	2012-06-12 10:33:17 +0000
@@ -1,7 +1,6 @@
 . inc/common.sh
 
-init
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "create database csv"
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "create table csm (a int NOT NULL ) ENGINE=CSV" csv
 # Adding initial rows
@@ -37,7 +36,7 @@
 vlog "Data prepared for restore"
 
 # Destroying mysql data
-stop_mysqld
+stop_server
 rm -rf $mysql_datadir/*
 vlog "Data destroyed"
 
@@ -49,7 +48,7 @@
 innobackupex --copy-back $full_backup_dir
 vlog "Data restored"
 
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 checksum_b=`checksum_table csv csm`
 vlog "Checking checksums: $checksum_a/$checksum_b"
 

=== modified file 'test/t/ib_empty_dir.sh'
--- test/t/ib_empty_dir.sh	2011-11-28 09:26:22 +0000
+++ test/t/ib_empty_dir.sh	2012-06-12 10:33:17 +0000
@@ -1,14 +1,13 @@
 # Test for bug https://bugs.launchpad.net/percona-xtrabackup/+bug/737569
 . inc/common.sh
 
-init
-run_mysqld
+start_server
 
 innobackupex --no-timestamp $topdir/backup
 backup_dir=$topdir/backup
 vlog "Backup created in directory $backup_dir"
 
-stop_mysqld
+stop_server
 # Remove datadir
 rm -r $mysql_datadir
 # Restore backup

=== modified file 'test/t/ib_incremental.sh'
--- test/t/ib_incremental.sh	2011-12-21 16:40:27 +0000
+++ test/t/ib_incremental.sh	2012-06-12 10:33:17 +0000
@@ -1,7 +1,6 @@
 . inc/common.sh
 
-init
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 load_dbase_schema incremental_sample
 
 # Adding initial rows
@@ -72,7 +71,7 @@
 vlog "Data prepared for restore"
 
 # Destroying mysql data
-stop_mysqld
+stop_server
 rm -rf $mysql_datadir/*
 vlog "Data destroyed"
 
@@ -84,7 +83,7 @@
 innobackupex --copy-back $full_backup_dir
 vlog "Data restored"
 
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 
 vlog "Checking checksums"
 checksum_b=`checksum_table incremental_sample test`

=== modified file 'test/t/ib_rsync.sh'
--- test/t/ib_rsync.sh	2011-11-15 13:01:05 +0000
+++ test/t/ib_rsync.sh	2012-06-12 10:33:17 +0000
@@ -6,13 +6,12 @@
     exit $SKIPPED_EXIT_CODE
 fi
 
-init
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 load_sakila
 
 innobackupex --rsync --no-timestamp $topdir/backup
 
-stop_mysqld
+stop_server
 
 run_cmd rm -r $mysql_datadir
 
@@ -22,5 +21,5 @@
 
 innobackupex --copy-back $topdir/backup
 
-run_mysqld
+start_server
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "SELECT COUNT(*) FROM actor" sakila

=== modified file 'test/t/ib_slave_info.sh'
--- test/t/ib_slave_info.sh	2011-08-29 19:44:10 +0000
+++ test/t/ib_slave_info.sh	2012-06-12 10:33:17 +0000
@@ -1,7 +1,14 @@
 . inc/common.sh
 
-init
-run_mysqld --innodb_file_per_table
+master_id=1
+slave_id=2
+
+start_server_with_id $master_id
+start_server_with_id $slave_id
+
+setup_slave $slave_id $master_id
+
+switch_server $master_id
 load_dbase_schema incremental_sample
 
 # Adding initial rows
@@ -15,14 +22,19 @@
 done
 vlog "Initial rows added"
 
-# Full backup
-# backup root directory
-mkdir -p $topdir/backup
+# Full backup of the slave server
+switch_server $slave_id
 
 vlog "Check that --slave-info with --no-lock and no --safe-slave-backup fails"
-run_cmd_expect_failure $IB_BIN $IB_ARGS --slave-info --no-lock $topdir/backup
+run_cmd_expect_failure $IB_BIN $IB_ARGS --no-timestamp --slave-info --no-lock \
+  $topdir/backup
 
 vlog "Check that --slave-info with --incremental and no --safe-slave-backup fails"
-run_cmd_expect_failure $IB_BIN $IB_ARGS --slave-info --incremental --incremental-basedir=$topdir/backup $topdir/backup
+run_cmd_expect_failure $IB_BIN $IB_ARGS --no-timestamp --slave-info \
+  --incremental --incremental-basedir=$topdir/backup $topdir/backup
 
-# TODO: add positive tests when the test suite is able to setup replication
+innobackupex --no-timestamp --slave-info $topdir/backup
+egrep -q '^mysql-bin.000001[[:space:]]+[0-9]+[[:space:]]+$' \
+    $topdir/backup/xtrabackup_binlog_info
+egrep -q '^CHANGE MASTER TO MASTER_LOG_FILE='\''mysql-bin.000001'\'', MASTER_LOG_POS=[0-9]+$' \
+    $topdir/backup/xtrabackup_slave_info

=== modified file 'test/t/ib_specialchar.sh'
--- test/t/ib_specialchar.sh	2011-12-07 13:28:47 +0000
+++ test/t/ib_specialchar.sh	2012-06-12 10:33:17 +0000
@@ -1,7 +1,7 @@
 . inc/common.sh
 
-init
-run_mysqld
+start_server
+
 load_dbase_schema sakila
 load_dbase_data sakila
 
@@ -20,7 +20,7 @@
 run_cmd ${MYSQLADMIN} ${MYSQL_ARGS} -p'$PASSWD' password ''
 
 vlog "Stopping database server"
-stop_mysqld 
+stop_server
 # Remove datadir
 vlog "Removing data folder"
 rm -r $mysql_datadir
@@ -40,7 +40,7 @@
 
 vlog "Starting database server"
 # using --skip-grant-tables to override root password restored from backup
-run_mysqld --skip-grant-tables
+start_server --skip-grant-tables
 vlog "Database server started"
 # Check sakila
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "SELECT count(*) from actor" sakila 

=== modified file 'test/t/tar4ibd_o_direct.sh'
--- test/t/tar4ibd_o_direct.sh	2012-02-02 13:29:22 +0000
+++ test/t/tar4ibd_o_direct.sh	2012-06-12 10:33:17 +0000
@@ -11,15 +11,15 @@
     exit $SKIPPED_EXIT_CODE
 fi
 
-init
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
+
 load_sakila
 
 # Take backup
 echo "innodb_flush_method=O_DIRECT" >> $topdir/my.cnf
 mkdir -p $topdir/backup
 innobackupex --stream=tar $topdir/backup > $topdir/backup/out.tar
-stop_mysqld
+stop_server
 
 # See if tar4ibd was using O_DIRECT for all InnoDB files
 cnt=`grep "tar4ibd: using O_DIRECT for the input file" $OUTFILE | wc -l`
@@ -42,6 +42,6 @@
 mkdir -p $mysql_datadir
 innobackupex --copy-back --defaults-file=$topdir/my.cnf $backup_dir
 
-run_mysqld
+start_server
 # Check sakila
 ${MYSQL} ${MYSQL_ARGS} -e "SELECT count(*) from actor" sakila

=== modified file 'test/t/tar4ibd_symlink.sh'
--- test/t/tar4ibd_symlink.sh	2011-06-14 13:33:54 +0000
+++ test/t/tar4ibd_symlink.sh	2012-06-12 10:33:17 +0000
@@ -4,37 +4,41 @@
 
 . inc/common.sh
 
-init
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
+
 load_dbase_schema sakila
 load_dbase_data sakila
 
+# Force a checkpoint
+stop_server
+start_server
+
 # Copy some .ibd files to a temporary location and replace them with symlinks
 
-mv $topdir/mysql/sakila/actor.ibd $topdir
-ln -s $topdir/actor.ibd $topdir/mysql/sakila
+mv $MYSQLD_DATADIR/sakila/actor.ibd $MYSQLD_VARDIR/
+ln -s $MYSQLD_VARDIR/actor.ibd $MYSQLD_DATADIR/sakila/actor.ibd
 
-mv $topdir/mysql/sakila/customer.ibd $topdir
-ln -s $topdir/customer.ibd $topdir/customer_link.ibd
-ln -s $topdir/customer_link.ibd $topdir/mysql/sakila/customer.ibd
+mv $MYSQLD_DATADIR/sakila/customer.ibd $MYSQLD_VARDIR/customer.ibd
+ln -s $MYSQLD_VARDIR/customer.ibd $MYSQLD_VARDIR/customer_link.ibd
+ln -s $MYSQLD_VARDIR/customer_link.ibd $MYSQLD_DATADIR/sakila/customer.ibd
 
 # Take backup
-mkdir -p $topdir/backup
-innobackupex --stream=tar $topdir/backup > $topdir/backup/out.tar 
+mkdir -p $MYSQLD_VARDIR/backup
+innobackupex --stream=tar $MYSQLD_VARDIR/backup > $MYSQLD_VARDIR/backup/out.tar 
 
-stop_mysqld
+stop_server
 
 # Remove datadir
 rm -r $mysql_datadir
 
 # Remove the temporary files referenced by symlinks
-rm -f $topdir/actor.ibd
-rm -f $topdir/customer.ibd
-rm -f $topdir/customer_link.ibd
+rm -f $MYSQLD_VARDIR/actor.ibd
+rm -f $MYSQLD_VARDIR/customer.ibd
+rm -f $MYSQLD_VARDIR/customer_link.ibd
 
 # Restore sakila
 vlog "Applying log"
-backup_dir=$topdir/backup
+backup_dir=$MYSQLD_VARDIR/backup
 cd $backup_dir
 $TAR -ixvf out.tar
 cd - >/dev/null 2>&1 
@@ -46,10 +50,8 @@
 
 innobackupex --copy-back $backup_dir
 
-run_mysqld
+start_server
 
 # Check sakila
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "SELECT COUNT(*) FROM actor" sakila
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "SELECT COUNT(*) FROM customer" sakila
-stop_mysqld
-clean

=== modified file 'test/t/xb_basic.sh'
--- test/t/xb_basic.sh	2011-12-07 13:28:47 +0000
+++ test/t/xb_basic.sh	2012-06-12 10:33:17 +0000
@@ -1,16 +1,16 @@
 . inc/common.sh
 
-init
-run_mysqld
+start_server
+
 load_dbase_schema sakila
 load_dbase_data sakila
 
 mkdir -p $topdir/backup
-innobackupex  $topdir/backup
+innobackupex $topdir/backup
 backup_dir=`grep "innobackupex: Backup created in directory" $OUTFILE | awk -F\' '{ print $2}'`
 vlog "Backup created in directory $backup_dir"
 
-stop_mysqld
+stop_server
 # Remove datadir
 rm -r $mysql_datadir
 #init_mysql_dir
@@ -27,6 +27,6 @@
 vlog "###########"
 innobackupex --copy-back $backup_dir
 
-run_mysqld
+start_server
 # Check sakila
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "SELECT count(*) from actor" sakila

=== modified file 'test/t/xb_export.sh'
--- test/t/xb_export.sh	2011-12-21 16:40:27 +0000
+++ test/t/xb_export.sh	2012-06-12 10:33:17 +0000
@@ -1,11 +1,5 @@
 . inc/common.sh
 
-init
-
-backup_dir=$topdir/xb_export_backup
-rm -rf $backup_dir
-mkdir $backup_dir
-
 if [ -z "$XTRADB_VERSION" ]; then
     echo "Requires XtraDB" > $SKIPPED_REASON
     exit $SKIPPED_EXIT_CODE
@@ -18,11 +12,15 @@
     import_option="--innodb_expand_import=1"
 fi
 
-MYSQLD_ARGS="$MYSQLD_ARGS --innodb_file_per_table $import_option \
+mysql_extra_args="--innodb_file_per_table $import_option \
 --innodb_file_format=Barracuda"
 
 # Starting database server
-run_mysqld
+start_server $mysql_extra_args
+
+backup_dir=$topdir/xb_export_backup
+rm -rf $backup_dir
+mkdir $backup_dir
 
 # Loading table schema
 load_dbase_schema incremental_sample
@@ -48,10 +46,9 @@
 vlog "Table was backed up"
 
 vlog "Re-initializing database server"
-stop_mysqld
-# Can't use clean/init because that will remove $backup_dir as well
-clean_datadir
-run_mysqld
+stop_server
+rm -rf ${MYSQLD_DATADIR}
+start_server $mysql_extra_args
 load_dbase_schema incremental_sample
 vlog "Database was re-initialized"
 
@@ -91,8 +88,8 @@
 # consistent backup results. Otherwise we risk ending up with no test.ibd
 # in the backup in case importing has not finished before taking backup
 
-stop_mysqld
-run_mysqld
+stop_server
+start_server $mysql_extra_args
 
 # Some testing queries
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "select count(*) from test;" incremental_sample
@@ -111,13 +108,13 @@
 
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "delete from test;" incremental_sample
 
-stop_mysqld
+stop_server
 
 cd $topdir/backup/full
 cp -r * $mysql_datadir
 cd -
 
-run_mysqld
+start_server $mysql_extra_args
 
 vlog "Cheking checksums"
 checksum_3=`checksum_table incremental_sample test`
@@ -131,6 +128,6 @@
 
 vlog "Checksums are OK"
 
-stop_mysqld
+stop_server
 
 rm -rf $backup_dir

=== modified file 'test/t/xb_incremental.sh'
--- test/t/xb_incremental.sh	2011-12-21 16:40:27 +0000
+++ test/t/xb_incremental.sh	2012-06-12 10:33:17 +0000
@@ -1,7 +1,7 @@
 . inc/common.sh
 
-init
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
+
 load_dbase_schema incremental_sample
 
 # Adding 10k rows
@@ -73,14 +73,14 @@
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "delete from test;" incremental_sample
 
 # Restore backup
-stop_mysqld
+stop_server
 vlog "Copying files"
 cd $topdir/data/full/
 cp -r * $mysql_datadir
 cd $topdir
 
 vlog "Data restored"
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 
 vlog "Checking checksums"
 checksum_b=`checksum_table incremental_sample test`

=== modified file 'test/t/xb_incremental_compressed.sh'
--- test/t/xb_incremental_compressed.sh	2011-12-21 16:40:27 +0000
+++ test/t/xb_incremental_compressed.sh	2012-06-12 10:33:17 +0000
@@ -4,8 +4,6 @@
 
 . inc/common.sh
 
-init
-
 if [ -z "$INNODB_VERSION" ]; then
     echo "Requires InnoDB plugin or XtraDB" >$SKIPPED_REASON
     exit $SKIPPED_EXIT_CODE
@@ -27,7 +25,7 @@
   mysqld_additional_args="--innodb_strict_mode --innodb_file_per_table \
       --innodb_file_format=Barracuda"
   
-  run_mysqld ${mysqld_additional_args}
+  start_server ${mysqld_additional_args}
 
   load_dbase_schema incremental_sample
 
@@ -134,7 +132,7 @@
 
   # Restore backup
 
-  stop_mysqld
+  stop_server
 
   vlog "Copying files"
 
@@ -144,7 +142,7 @@
 
   vlog "Data restored"
 
-  run_mysqld ${mysqld_additional_args}
+  start_server ${mysqld_additional_args}
 
   vlog "Cheking checksums"
   checksum_b=`checksum_table incremental_sample test`
@@ -158,7 +156,7 @@
   vlog "Checksums are OK"
 
   drop_dbase incremental_sample
-  stop_mysqld
+  stop_server
 }
 
 for page_size in 1 2 4 8 16; do

=== modified file 'test/t/xb_parallel.sh'
--- test/t/xb_parallel.sh	2011-12-28 12:52:39 +0000
+++ test/t/xb_parallel.sh	2012-06-12 10:33:17 +0000
@@ -1,7 +1,7 @@
 . inc/common.sh
 
-init
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
+
 load_dbase_schema sakila
 load_dbase_data sakila
 
@@ -13,7 +13,7 @@
 
 #echo "Backup dir in $backup_dir"
 
-stop_mysqld
+stop_server
 # Remove datadir
 rm -r $mysql_datadir
 # Restore sakila
@@ -29,6 +29,6 @@
 vlog "###########"
 innobackupex --copy-back $backup_dir
 
-run_mysqld
+start_server
 # Check sakila
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "SELECT count(*) from actor" sakila

=== modified file 'test/t/xb_parallel_incremental.sh'
--- test/t/xb_parallel_incremental.sh	2011-12-28 13:14:34 +0000
+++ test/t/xb_parallel_incremental.sh	2012-06-12 10:33:17 +0000
@@ -4,8 +4,8 @@
 
 . inc/common.sh
 
-init
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
+
 load_dbase_schema sakila
 load_dbase_data sakila
 
@@ -38,7 +38,7 @@
 innobackupex --incremental --no-timestamp --parallel=8 \
     --incremental-basedir=$topdir/full_backup $topdir/inc_backup
 
-stop_mysqld
+stop_server
 # Remove datadir
 rm -r $mysql_datadir
 
@@ -52,7 +52,7 @@
 mkdir -p $mysql_datadir
 innobackupex --copy-back $topdir/full_backup
 
-run_mysqld
+start_server
 
 # Check sakila
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "SELECT count(*) from actor" sakila

=== modified file 'test/t/xb_part_range.sh'
--- test/t/xb_part_range.sh	2011-12-21 16:40:27 +0000
+++ test/t/xb_part_range.sh	2012-06-12 10:33:17 +0000
@@ -1,7 +1,6 @@
 . inc/common.sh
 
-init
-run_mysqld
+start_server
 
 run_cmd $MYSQL $MYSQL_ARGS test <<EOF
 CREATE TABLE test (
@@ -89,7 +88,7 @@
 
 # Restore backup
 
-stop_mysqld
+stop_server
 
 vlog "Copying files"
 
@@ -99,7 +98,7 @@
 
 vlog "Data restored"
 
-run_mysqld
+start_server
 
 vlog "Cheking checksums"
 checksum_b=`checksum_table test test`

=== modified file 'test/t/xb_partial.sh'
--- test/t/xb_partial.sh	2011-12-21 16:40:27 +0000
+++ test/t/xb_partial.sh	2012-06-12 10:33:17 +0000
@@ -1,7 +1,7 @@
 . inc/common.sh
 
-init
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
+
 load_dbase_schema incremental_sample
 
 # Adding 10k rows
@@ -34,13 +34,13 @@
 vlog "Table cleared"
 
 # Restore backup
-stop_mysqld
+stop_server
 vlog "Copying files"
 cd $topdir/data/parted/
 cp -r * $mysql_datadir
 cd $topdir
 vlog "Data restored"
-run_mysqld --innodb_file_per_table
+start_server --innodb_file_per_table
 vlog "Checking checksums"
 checksum_b=`checksum_table incremental_sample test`
 

=== modified file 'test/t/xb_perm_basic.sh'
--- test/t/xb_perm_basic.sh	2012-01-27 11:21:39 +0000
+++ test/t/xb_perm_basic.sh	2012-06-12 10:33:17 +0000
@@ -1,8 +1,7 @@
 # Test for fix of https://bugs.launchpad.net/percona-xtrabackup/+bug/691090
 . inc/common.sh
 
-init
-run_mysqld
+start_server
 
 chmod -R 500 $mysql_datadir
 trap "vlog restoring permissions on datadir $mysql_datadir ; \

=== modified file 'test/t/xb_perm_stream.sh'
--- test/t/xb_perm_stream.sh	2012-01-27 11:21:39 +0000
+++ test/t/xb_perm_stream.sh	2012-06-12 10:33:17 +0000
@@ -1,8 +1,7 @@
 # Test for fix of https://bugs.launchpad.net/percona-xtrabackup/+bug/691090
 . inc/common.sh
 
-init
-run_mysqld
+start_server
 
 # Take backup
 chmod -R 500 $mysql_datadir

=== modified file 'test/t/xb_stats.sh'
--- test/t/xb_stats.sh	2011-12-07 13:28:47 +0000
+++ test/t/xb_stats.sh	2012-06-12 10:33:17 +0000
@@ -1,7 +1,7 @@
 . inc/common.sh
 
-init
-run_mysqld
+start_server
+
 load_dbase_schema sakila
 load_dbase_data sakila
 

=== modified file 'test/t/xb_stream.sh'
--- test/t/xb_stream.sh	2011-12-07 13:28:47 +0000
+++ test/t/xb_stream.sh	2012-06-12 10:33:17 +0000
@@ -1,7 +1,7 @@
 . inc/common.sh
 
-init
-run_mysqld
+start_server
+
 load_dbase_schema sakila
 load_dbase_data sakila
 
@@ -9,7 +9,7 @@
 mkdir -p $topdir/backup
 innobackupex --stream=tar $topdir/backup > $topdir/backup/out.tar
 
-stop_mysqld
+stop_server
 # Remove datadir
 rm -r $mysql_datadir
 # Restore sakila
@@ -29,6 +29,6 @@
 vlog "###########"
 innobackupex  --copy-back $backup_dir
 
-run_mysqld
+start_server
 # Check sakila
 run_cmd ${MYSQL} ${MYSQL_ARGS} -e "SELECT count(*) from actor" sakila