Merge lp:~gl-az/percona-xtrabackup/BT-32889-history-on-server-2.2 into lp:percona-xtrabackup/2.2

[George Ormond Lorch III]

BT-32889

Implementation of xtrabackup http://blueprints.launchpad.net/percona-xtrabackup/+spec/history-on-the-server and test case.

http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.2-param/8/

matrix reloaded for two failures jenkis had when branching...

http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.2-param/9/

[Alexey Kopytov (akopytov)]

George,

   - I propose we rename ‘mysql_version’ to ‘server_version’, since we
     support not only MySQL.

   - let’s also rename xtrabackup_history to xtrabackup_info. It
     contains meta information about the backup where it’s stored, not
     the history. I think we can even deprecate xtrabackup_checkpoints
     with xtrabackup_info in future.

   - the table does not define the character set as we discussed
     previously. This means the character set will depend on
     server/connection settings.

   - update_history() is vulnerable to SQL injection attacks :) what if
     something like this is passed to --history, for example:

     --history="foo'; DROP DATABASE customers".

     Please use DBI prepared statements. This way you don’t have to roll
     your own quoting in other places.

   - it also appears that xtrabackup_history will not be created if
     --history is not passed to innobackupex. I think it should always
     be created. I.e. update_history() should collect all info
     unconditionally, but just avoid executing the actual INSERT if
     $option_history if false.

   - for start_time / end_time it’s better to use time() and then
     convert values it returns to TIMESTAMP with FROM_UNIXTIME():

  $history_start_time = time();
  ...

  # start_time
  $insert_vals .= ", FROM_UNIXTIME($history_start_time)";

  # end_time
  $tmp = time();
  $insert_vals .= ", FROM_UNIXTIME($tmp)";

    - /proc is only available on Linux, so the following will fail on
      everything else. And there are really no reasons to use it,
      there are many portable ways to get the cmd line from a Perl script

+ $tmp = sprintf("cat /proc/%d/cmdline", $$);
+ $tmp = qx($tmp);
+ system("cp /proc/$$/cmdline /tmp");
+ $tmp =~ s/\0/ /g; # transform embedded nulls between args to spaces

    - system("rm -f $file") is equivalent to unlink($file)

    - do you really want to scrub --encrypt and --encrypt-key-file?

    - here’s a more elegant and correct scrub_option() implementation:

sub scrub_option {
  my $scrub = shift;
  my $command = shift;

  return $command =~ s/--$scrub=[^ ]+ /--$scrub=... /g

}

   - in case --incremental-history-name or --incremental-history-uuid is
     used, it might be useful to have a diagnostic message about the LSN
     value being used and why it is being used.

   - I guess built-in innobackupex help needs some updates (still
     mentions --history-name, --no-history, etc.)

   - in the test case: it is not necessary to include inc/common.sh
     anymore. it is also not necessary to use “shift” if you don’t
     access the parameters array later in a function

review: Needs Fixing

[George Ormond Lorch III (gl-az)]

On 9/23/2013 9:58 AM, Alexey Kopytov wrote:
> Review: Needs Fixing
>
> George,
>
> - I propose we rename ‘mysql_version’ to ‘server_version’, since we
> support not only MySQL.
Makes sense, can do.
> - let’s also rename xtrabackup_history to xtrabackup_info. It
> contains meta information about the backup where it’s stored, not
> the history. I think we can even deprecate xtrabackup_checkpoints
> with xtrabackup_info in future.
Do you want to rename the entire feature and all options (--no-info,
--info=<name>, --incremental-info-name, --incremental-info-uuid) as well
or just the table name?
> - the table does not define the character set as we discussed
> previously. This means the character set will depend on
> server/connection settings.
Very true, forgot about that. Will add utf8 specifier.
> - update_history() is vulnerable to SQL injection attacks :) what if
> something like this is passed to --history, for example:
>
> --history="foo'; DROP DATABASE customers".
>
> Please use DBI prepared statements. This way you don’t have to roll
> your own quoting in other places.
Yes, I actually had a discussion w/ Vadim on whether or not we are
concerned w/ SQL injection at that level and the answer was not really
so I just stuck with the original prototype code I had that used all
literals. I suppose it wouldn't hurt and would be more proper to use DBD
prepare and bind values instead of all the literal bits.
>
> - it also appears that xtrabackup_history will not be created if
> --history is not passed to innobackupex. I think it should always
> be created. I.e. update_history() should collect all info
> unconditionally, but just avoid executing the actual INSERT if
> $option_history if false.
Correct. There were some XB test case failures that would need to be
fixed up and permissions manipulated if we always created the
schema/filled the record. At that point I thought too that it may not be
a great idea to change the default assumption that XB was read only on a
server database by now having it start to write stuff back into it,
possibly requiring a user to change their backup user privileges. So, I
changed it to be entirely optional and any upgrades would be compatible
with any existing backup user privileges without having to explicitly
disable it or alter backup user privileges. I should have checked that
out w/ you first. That being said, if you do still want it on/creating
schema and filling the table and generating the file by default, I can
change it back and go fix up whatever failing tests occur as a result
(just adding --no-history/--no-info) and make sure that I give Hrvoje
enough info doc wise to try to avoid confusion.
> - for start_time / end_time it’s better to use time() and then
> convert values it returns to TIMESTAMP with FROM_UNIXTIME():
>
> $history_start_time = time();
> ...
>
> # start_time
> $insert_vals .= ", FROM_UNIXTIME($history_start_time)";
>
> # end_time
> $tmp = time();
> $insert_vals .= ", FROM_UNIXTIME($tmp)";
OK, will do.
> - /proc is only available on Li...

[Alexey Kopytov (akopytov)]

Hi George,

On Mon, 23 Sep 2013 15:49:15 -0700, George Lorch wrote:
> On 9/23/2013 9:58 AM, Alexey Kopytov wrote:
>> Review: Needs Fixing
>>
>> George,
>>
>> - I propose we rename ‘mysql_version’ to ‘server_version’, since we
>> support not only MySQL.
> Makes sense, can do.
>> - let’s also rename xtrabackup_history to xtrabackup_info. It
>> contains meta information about the backup where it’s stored, not
>> the history. I think we can even deprecate xtrabackup_checkpoints
>> with xtrabackup_info in future.
> Do you want to rename the entire feature and all options (--no-info,
> --info=<name>, --incremental-info-name, --incremental-info-uuid) as well
> or just the table name?

No, sorry I should have been more specific. I was referring to the file
created in the backup directory. 'xtrabackup_info' reflects its content
better than 'xtrabackup_history' IMO.

>> - the table does not define the character set as we discussed
>> previously. This means the character set will depend on
>> server/connection settings.
> Very true, forgot about that. Will add utf8 specifier.
>> - update_history() is vulnerable to SQL injection attacks :) what if
>> something like this is passed to --history, for example:
>>
>> --history="foo'; DROP DATABASE customers".
>>
>> Please use DBI prepared statements. This way you don’t have to roll
>> your own quoting in other places.
> Yes, I actually had a discussion w/ Vadim on whether or not we are
> concerned w/ SQL injection at that level and the answer was not really
> so I just stuck with the original prototype code I had that used all
> literals. I suppose it wouldn't hurt and would be more proper to use DBD
> prepare and bind values instead of all the literal bits.

Yes, it's certainly not a critical issue in the XtraBackup context, but
still opens the way for users shooting themselves in the foot with some
bad argument to --history.

>>
>> - it also appears that xtrabackup_history will not be created if
>> --history is not passed to innobackupex. I think it should always
>> be created. I.e. update_history() should collect all info
>> unconditionally, but just avoid executing the actual INSERT if
>> $option_history if false.
> Correct. There were some XB test case failures that would need to be
> fixed up and permissions manipulated if we always created the
> schema/filled the record. At that point I thought too that it may not be
> a great idea to change the default assumption that XB was read only on a
> server database by now having it start to write stuff back into it,
> possibly requiring a user to change their backup user privileges. So, I
> changed it to be entirely optional and any upgrades would be compatible
> with any existing backup user privileges without having to explicitly
> disable it or alter backup user privileges. I should have checked that
> out w/ you first. That being said, if you do still want it on/creating
> schema and filling the table and generating the file by default, I can
> change it back and go fix up whatever failing tests occur as a result
> (just adding --no-history/--no-inf...

[George Ormond Lorch III (gl-az)]

Hey there Alexey...

On 9/24/2013 4:18 AM, Alexey Kopytov wrote:
> - let’s also rename xtrabackup_history to xtrabackup_info. It
> contains meta information about the backup where it’s stored, not
> the history. I think we can even deprecate xtrabackup_checkpoints
> with xtrabackup_info in future.
>> Do you want to rename the entire feature and all options (--no-info,
>> --info=<name>, --incremental-info-name, --incremental-info-uuid) as well
>> or just the table name?
> No, sorry I should have been more specific. I was referring to the file
> created in the backup directory. 'xtrabackup_info' reflects its content
> better than 'xtrabackup_history' IMO.
>
>>> - it also appears that xtrabackup_history will not be created if
>>> --history is not passed to innobackupex. I think it should always
>>> be created. I.e. update_history() should collect all info
>>> unconditionally, but just avoid executing the actual INSERT if
>>> $option_history if false.
>> Correct. There were some XB test case failures that would need to be
>> fixed up and permissions manipulated if we always created the
>> schema/filled the record. At that point I thought too that it may not be
>> a great idea to change the default assumption that XB was read only on a
>> server database by now having it start to write stuff back into it,
>> possibly requiring a user to change their backup user privileges. So, I
>> changed it to be entirely optional and any upgrades would be compatible
>> with any existing backup user privileges without having to explicitly
>> disable it or alter backup user privileges. I should have checked that
>> out w/ you first. That being said, if you do still want it on/creating
>> schema and filling the table and generating the file by default, I can
>> change it back and go fix up whatever failing tests occur as a result
>> (just adding --no-history/--no-info) and make sure that I give Hrvoje
>> enough info doc wise to try to avoid confusion.
> Sorry, I should have been more specific here to. I just meant to say
> that we have to create the xtrabackup_history (or xtrabackup_info) file
> regardless of whether --history is enabled.
>
> The reason we are implementing that file is
> https://bugs.launchpad.net/percona-xtrabackup/2.2/+bug/1133017
Ahh, OK, Now that makes a lot more sense. Yes, I can easily do this,
just have to shuffle around the means of data collection and writing in
update_history with the new DBD prepare and bind_param stuff so there
are not 20 if (!defined($option_history)).
>>> - here’s a more elegant and correct scrub_option() implementation:
>>>
>>> sub scrub_option {
>>> my $scrub = shift;
>>> my $command = shift;
>>>
>>> return $command =~ s/--$scrub=[^ ]+ /--$scrub=... /g
>>>
>>> }
>> Nice.
> Actually that won't work (I didn't verify it before posting). The
> following will work:
>
> sub scrub_option {
> my $scrub = shift;
> my $command = shift;
>
> $command =~ s/--$scrub=[^ ]+/--$scrub=.../g;
> return $command;
> }
>
>
>
Yup, I figured that out yesterday and fixed it myself. Actually, I just
removed the whole scrub_option call and just us...

[George Ormond Lorch III (gl-az)]

New jenkins for changes based on comments above:
http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.2-param/13/

Don't know what that debian6/innodb51 failure is about in compact.sh. That has nothing to do with this feature.

[Alexey Kopytov (akopytov)]

The following lines:

$tmp =~ s/--password=[^ ]+ /--password=... /g;
$tmp =~ s/--encrypt-key=[^ ]+ /--encrypt-key=... /g;

Should not contain spaces, i.e.:

$tmp =~ s/--password=[^ ]+/--password=.../g;
$tmp =~ s/--encrypt-key=[^ ]+/--encrypt-key=.../g;

That's my fault, it was in the first incorrect version of scrub_option() I proposed, but then fixed in a later version.

The difference is that when one of the options to be scrubbed is at the end of the command line, there will be no space after it, so no substitution would occur with the current code then.

review: Needs Fixing

[George Ormond Lorch III (gl-az)]

> The following lines:
>
> $tmp =~ s/--password=[^ ]+ /--password=... /g;
> $tmp =~ s/--encrypt-key=[^ ]+ /--encrypt-key=... /g;
>
> Should not contain spaces, i.e.:
>
> $tmp =~ s/--password=[^ ]+/--password=.../g;
> $tmp =~ s/--encrypt-key=[^ ]+/--encrypt-key=.../g;
>
> That's my fault, it was in the first incorrect version of scrub_option() I
> proposed, but then fixed in a later version.
>
> The difference is that when one of the options to be scrubbed is at the end of
> the command line, there will be no space after it, so no substitution would
> occur with the current code then.

Yeah, I see that now, good catch. Fixed and went ahead and added a test for the command_tool with encrypt-key at the end of the line to validate that it gets scrubbed correctly.

[George Ormond Lorch III (gl-az)]

new jenkins run http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.2-param/14/

[Alexey Kopytov (akopytov)]

review: Approve

=== modified file 'xtrabackup/innobackupex'
--- xtrabackup/innobackupex	2013-09-23 10:58:53 +0000
+++ xtrabackup/innobackupex	2013-09-25 17:11:22 +0000
@@ -147,6 +147,10 @@
 my %mysql;
 my $option_backup = '';
 
+my $option_history;
+my $option_incremental_history_name = '';
+my $option_incremental_history_uuid = '';
+
 # name of the my.cnf configuration file
 #my $config_file = '';
 
@@ -195,6 +199,9 @@
 # name of the file where slave info is written
 my $slave_info;
 
+# name of the file where version and backup history is written 
+my $backup_history;
+
 # mysql binlog position as given by "SHOW MASTER STATUS" command
 my $mysql_binlog_position = '';
 
@@ -270,6 +277,7 @@
 my $copy_dir_overwrite;
 my $copy_dir_resolve_isl;
 
+<<<<<<< TREE
 # ###########################################################################
 # HTTPMicro package
 # This package is a copy without comments from the original.  The original
@@ -1511,6 +1519,14 @@
 # End VersionCheck package
 # ###########################################################################
 
+=======
+# for history on server record
+my $history_tool_command = "@ARGV";
+my $history_start_time = time();
+my $history_lock_time = 0;
+my $history_delete_checkpoints = 0;
+
+>>>>>>> MERGE-SOURCE
 ######################################################################
 # program execution begins here
 ######################################################################
@@ -1919,6 +1935,43 @@
         detect_mysql_capabilities_for_backup(\%mysql);
     }
 
+    #
+    # if one of the history incrementals is being used, try to grab the
+    # innodb_to_lsn from the history table and set the option_incremental_lsn
+    #
+    if ($option_incremental && !$option_incremental_lsn) {
+        if ($option_incremental_history_name) {
+            my $query = "SELECT innodb_to_lsn ".
+                        "FROM PERCONA_SCHEMA.xtrabackup_history ".
+                        "WHERE name = '$option_incremental_history_name' ".
+                        "AND innodb_to_lsn IS NOT NULL ".
+                        "ORDER BY innodb_to_lsn DESC LIMIT 1";
+
+            eval {
+                $option_incremental_lsn =
+                    $mysql{dbh}->selectrow_hashref($query)->{innodb_to_lsn};
+            } || die("Error while attempting to find history record for name ".
+                     "'$option_incremental_history_name'\n");
+            print STDERR "$prefix --incremental-history-name=".
+                "$option_incremental_history_name specified.".
+                "Found and using lsn: $option_incremental_lsn";
+        } elsif ($option_incremental_history_uuid) {
+            my $query = "SELECT innodb_to_lsn ".
+                        "FROM PERCONA_SCHEMA.xtrabackup_history ".
+                        "WHERE uuid = '$option_incremental_history_uuid' ".
+                        "AND innodb_to_lsn IS NOT NULL";
+
+            eval {
+                $option_incremental_lsn =
+                    $mysql{dbh}->selectrow_hashref($query)->{innodb_to_lsn};
+            } || die("Error while attempting to find history record for uuid ".
+                     "'$option_incremental_history_uuid'\n");
+            print STDERR "$prefix --incremental-history-uuid=".
+                "$option_incremental_history_uuid specified.".
+                "Found and using lsn: $option_incremental_lsn";
+        }
+    }
+
     # start ibbackup as a child process
     start_ibbackup();
 
@@ -1943,6 +1996,9 @@
       # make a prep copy before locking tables, if using rsync
       backup_files(1);
 
+      # start counting how long lock is held for history
+      $history_lock_time = time();
+
       # flush tables with read lock
       mysql_lockall(\%mysql);
 
@@ -1976,6 +2032,13 @@
     # release read locks on all tables
     mysql_unlockall(\%mysql) if !$option_no_lock;
 
+    # calculate how long lock was held for history
+    if ($option_no_lock) {
+      $history_lock_time = 0;
+    } else {
+      $history_lock_time = time() - $history_lock_time;
+    }
+
     my $ibbackup_exit_code = wait_for_ibbackup_finish();
 
     if ( $option_safe_slave_backup && $sql_thread_started) {
@@ -1999,6 +2062,8 @@
         print STDERR "$prefix MySQL slave binlog position: $mysql_slave_position\n";
     }
 
+    write_xtrabackup_info(\%mysql);
+
     return $ibbackup_exit_code;
 }
 
@@ -3510,8 +3575,14 @@
         $binlog_info = $work_dir . '/xtrabackup_binlog_info';
         $galera_info = $work_dir . '/xtrabackup_galera_info';
         $slave_info = $work_dir . '/xtrabackup_slave_info';
+        $backup_history = $work_dir . '/xtrabackup_info';
         write_backup_config_file($backup_config_file);
 
+        if (!$option_extra_lsndir) {
+            $option_extra_lsndir = $option_tmpdir;
+            $history_delete_checkpoints = 1;
+        }
+
         foreach (@xb_suspend_files) {
             my $suspend_file = $work_dir . "$_";
             if ( -e "$suspend_file" ) {
@@ -3615,6 +3686,7 @@
                         'encrypt-threads=i' => \$option_encrypt_threads,
                         'encrypt-chunk-size=s' => \$option_encrypt_chunk_size,
                         'help' => \$option_help,
+                        'history:s' => \$option_history,
                         'version' => \$option_version,
                         'throttle=i' => \$option_throttle,
                         'log-copy-interval=i', \$option_log_copy_interval,
@@ -3642,6 +3714,8 @@
                         'incremental' => \$option_incremental,
                         'incremental-basedir=s' => \$option_incremental_basedir,
                         'incremental-force-scan' => \$option_incremental_force_scan,
+                        'incremental-history-name=s' => \$option_incremental_history_name,
+                        'incremental-history-uuid=s' => \$option_incremental_history_uuid,
                         'incremental-lsn=s' => \$option_incremental_lsn,
                         'incremental-dir=s' => \$option_incremental_dir,
                         'extra-lsndir=s' => \$option_extra_lsndir,
@@ -3750,7 +3824,9 @@
         # we are making a backup, get backup root directory
         $option_backup = "1";
         $backup_root = $ARGV[0];
-        if ($option_incremental && !$option_incremental_lsn) {
+        if ($option_incremental && !$option_incremental_lsn &&
+            !$option_incremental_history_name &&
+            !$option_incremental_history_uuid) {
             if ($option_incremental_basedir) {
                 $incremental_basedir = $option_incremental_basedir;
             } else {
@@ -4717,6 +4793,247 @@
                     'WHERE PLUGIN_NAME LIKE "INNODB_CHANGED_PAGES"')
 }
 
+#
+# Writes xtrabackup_info file and if backup_history is enable creates
+# PERCONA_SCHEMA.xtrabackup_history and writes a new history record to the
+# table containing all the history info particular to the just completed
+# backup.
+#
+sub write_xtrabackup_info()
+{
+  my $con = shift;
+  my $sth;
+  my $uuid;
+  my $tmp;
+  my $file_content;
+
+  eval {
+    $sth = $con->{dbh}->prepare("insert into PERCONA_SCHEMA.xtrabackup_history(".
+                  "uuid, name, tool_name, tool_command, tool_version, ".
+                  "ibbackup_version, server_version, start_time, end_time, ".
+                  "lock_time, binlog_pos, innodb_from_lsn, innodb_to_lsn, ".
+                  "partial, incremental, format, compact, compressed, ".
+                  "encrypted) ".
+                  "values(?,?,?,?,?,?,?,from_unixtime(?),from_unixtime(?),?,?,".
+                  "?,?,?,?,?,?,?,?)");
+  } || die("Eror while preparing history record\n");
+
+
+  # uuid
+  # we use the mysql UUID() function to avoid platform dependencies with uuidgen
+  # and Data::UUID. We select uuid here individually here so that it can be
+  # reported to stderr after successful history record insertion
+  eval {
+    $uuid = $con->{dbh}->selectrow_hashref("SELECT UUID() AS uuid")->{uuid};
+  } || die("Error while attempting to create UUID for history record.\n");
+  $sth->bind_param(1, $uuid);
+  $file_content = "uuid = $uuid";
+
+  # name
+  if ($option_history) {
+    $sth->bind_param(2, $option_history);
+    $file_content .= "\nname = $option_history";
+  } else {
+    $sth->bind_param(2, undef);
+    $file_content .= "\nname = ";
+  }
+
+  # tool_name
+  $tmp = basename($0);
+  $sth->bind_param(3, $tmp);
+  $file_content .= "\ntool_name = $tmp";
+
+  # tool_command
+  # scrub --password and --encrypt-key from tool_command
+  $tmp = $history_tool_command;
+  $tmp =~ s/--password=[^ ]+/--password=.../g;
+  $tmp =~ s/--encrypt-key=[^ ]+/--encrypt-key=.../g;
+  $sth->bind_param(4, $tmp);
+  $file_content .= "\ntool_command = $tmp";
+
+  # tool_version
+  $sth->bind_param(5, $innobackup_version);
+  $file_content .= "\ntool_version = $innobackup_version";
+
+  # ibbackup_version
+  $tmp = "$option_ibbackup_binary -v 2>&1";
+  $tmp = qx($tmp);
+  chomp($tmp);
+  if (length $tmp) {
+    $sth->bind_param(6, $tmp);
+    $file_content .= "\nibbackup_version = $tmp";
+  } else {
+    $sth->bind_param(6, undef);
+    $file_content .= "\nibbackup_version = ";
+  }
+
+  # server_version
+  $tmp =
+    $con->{dbh}->selectrow_hashref("SELECT VERSION() as version")->{version};
+  $sth->bind_param(7, $tmp);
+  $file_content .= "\nserver_version = $tmp";
+
+  # start_time
+  $sth->bind_param(8, $history_start_time);
+  $file_content .= "\nstart_time = " . strftime("%Y-%m-%d %H:%M:%S", localtime($history_start_time));
+
+  # end_time
+  $tmp = time();
+  $sth->bind_param(9, $tmp);
+  $file_content .= "\nend_time = " . strftime("%Y-%m-%d %H:%M:%S", localtime($tmp));
+
+  # lock_time
+  $sth->bind_param(10, $history_lock_time);
+  $file_content .= "\nlock_time = $history_lock_time";
+
+  # binlog_pos
+  if ($mysql_binlog_position) {
+    $sth->bind_param(11, $mysql_binlog_position);
+    $file_content .= "\nbinlog_pos = $mysql_binlog_position";
+  } else {
+    $sth->bind_param(11, undef);
+    $file_content .= "\nbinlog_pos = ";
+  }
+
+  # innodb_from_lsn
+  # grab the from_lsn from the extra checkpoints file, parse and delete it
+  # if necessary. This could generate an error if the checkpoints file
+  # isn't 100% correct in the format.
+  $tmp = "cat $option_extra_lsndir/xtrabackup_checkpoints | grep from_lsn";
+  $tmp = qx($tmp);
+  chomp($tmp);
+  substr($tmp, 0, 11, ""); # strip "from_lsn = "
+  if (length $tmp) {
+    $sth->bind_param(12, $tmp);
+    $file_content .= "\ninnodb_from_lsn = $tmp";
+  } else {
+    print STDERR "WARNING : Unable to obtain from_lsn from " .
+                 "$option_extra_lsndir/xtrabackup_checkpoints, " .
+                 "writing NULL to history record.\n";
+    $sth->bind_param(12, undef);
+    $file_content .= "\ninnodb_from_lsn = ";
+  }
+
+  # innodb_to_lsn
+  # grab the to_lsn from the extra checkpoints file, parse and delete it
+  # if necessary. This could generate an error if the checkpoints file
+  # isn't 100% correct in the format.
+  $tmp = "cat $option_extra_lsndir/xtrabackup_checkpoints | grep to_lsn";
+  $tmp = qx($tmp);
+  chomp($tmp);
+  substr($tmp, 0, 9, ""); # strip "to_lsn = "
+  if (length $tmp) {
+    $sth->bind_param(13, $tmp);
+    $file_content .= "\ninnodb_to_lsn = $tmp";
+  } else {
+    print STDERR "WARNING : Unable to obtain to_lsn from " .
+                 "$option_extra_lsndir/xtrabackup_checkpoints, " .
+                 "writing NULL to history record.\n";
+    $sth->bind_param(13, undef);
+    $file_content .= "\ninnodb_to_lsn = ";
+  }
+
+  # we're finished with the checkpoints file, delete it if it was a temp
+  # only for the history
+  if ($history_delete_checkpoints > 0) {
+    unlink("$option_extra_lsndir/xtrabackup_checkpoints");
+  }
+
+  # partial (Y | N)
+  if ($option_include || $option_databases || $option_tables_file || $option_export) {
+    $sth->bind_param(14, "Y");
+    $file_content .= "\npartial = Y";
+  } else {
+    $sth->bind_param(14, "N");
+    $file_content .= "\npartial = N";
+  }
+
+  # incremental (Y | N)
+  if ($option_incremental) {
+    $sth->bind_param(15, "Y");
+    $file_content .= "\nincremental = Y";
+  } else {
+    $sth->bind_param(15, "N");
+    $file_content .= "\nincremental = N";
+  }
+
+  # format (file | tar | xbsream)
+  if ($option_stream eq 'tar') {
+    $sth->bind_param(16, "tar");
+    $file_content .= "\nformat = tar";
+  } elsif ($option_stream eq 'xbstream') {
+    $sth->bind_param(16, "xbstream");
+    $file_content .= "\nformat = xbstream";
+  } else {
+    $sth->bind_param(16, "file");
+    $file_content .= "\nformat = file";
+  }
+
+  # compact (Y | N)
+  if ($option_compact) {
+    $sth->bind_param(17, "Y");
+    $file_content .= "\ncompact = Y";
+  } else {
+    $sth->bind_param(17, "N");
+    $file_content .= "\ncompact = N";
+  }
+
+  # compressed (Y | N)
+  if ($option_compress) {
+    $sth->bind_param(18, "Y");
+    $file_content .= "\ncompressed = Y";
+  } else {
+    $sth->bind_param(18, "N");
+    $file_content .= "\ncompressed = N";
+  }
+
+  # encrypted (Y | N)
+  if ($option_encrypt) {
+    $sth->bind_param(19, "Y");
+    $file_content .= "\nencrypted = Y";
+  } else {
+    $sth->bind_param(19, "N");
+    $file_content .= "\nencrypted = N";
+  }
+
+  # create the backup history file
+  write_to_backup_file("$backup_history", "$file_content");
+
+  if (!defined($option_history)) {
+    return;
+  }
+
+  mysql_query($con, "CREATE DATABASE IF NOT EXISTS PERCONA_SCHEMA");
+  mysql_query($con, "CREATE TABLE IF NOT EXISTS PERCONA_SCHEMA.xtrabackup_history(".
+                    "uuid VARCHAR(40) NOT NULL PRIMARY KEY,".
+                    "name VARCHAR(255) DEFAULT NULL,".
+                    "tool_name VARCHAR(255) DEFAULT NULL,".
+                    "tool_command TEXT DEFAULT NULL,".
+                    "tool_version VARCHAR(255) DEFAULT NULL,".
+                    "ibbackup_version VARCHAR(255) DEFAULT NULL,".
+                    "server_version VARCHAR(255) DEFAULT NULL,".
+                    "start_time TIMESTAMP NULL DEFAULT NULL,".
+                    "end_time TIMESTAMP NULL DEFAULT NULL,".
+                    "lock_time BIGINT UNSIGNED DEFAULT NULL,".
+                    "binlog_pos VARCHAR(128) DEFAULT NULL,".
+                    "innodb_from_lsn BIGINT UNSIGNED DEFAULT NULL,".
+                    "innodb_to_lsn BIGINT UNSIGNED DEFAULT NULL,".
+                    "partial ENUM('Y', 'N') DEFAULT NULL,".
+                    "incremental ENUM('Y', 'N') DEFAULT NULL,".
+                    "format ENUM('file', 'tar', 'xbstream') DEFAULT NULL,".
+                    "compact ENUM('Y', 'N') DEFAULT NULL,".
+                    "compressed ENUM('Y', 'N') DEFAULT NULL,".
+                    "encrypted ENUM('Y', 'N') DEFAULT NULL".
+                    ") CHARACTER SET utf8 ENGINE=innodb");
+
+  eval {
+    $sth->execute;
+  } || die("Error while attempting to insert history record.\n");
+
+  print STDERR "$prefix Backup history record uuid $uuid successfully written\n";
+}
+
+
 =pod
 
 =head1 NAME
@@ -4735,8 +5052,10 @@
              [--defaults-file=MY.CNF] [--defaults-group=GROUP-NAME]
              [--databases=LIST] [--no-lock] 
              [--tmpdir=DIRECTORY] [--tables-file=FILE]
+             [--history=NAME]
              [--incremental] [--incremental-basedir]
              [--incremental-dir] [--incremental-force-scan] [--incremental-lsn]
+             [--incremental-history-name=NAME] [--incremental-history-uuid=UUID]
              [--compact]     
              BACKUP-ROOT-DIR
 
@@ -4906,6 +5225,10 @@
 
 This option displays a help screen and exits.
 
+=item --history=NAME
+
+This option enables the tracking of backup history in the PERCONA_SCHEMA.xtrabackup_history table. An optional history series name may be specified that will be placed with the history record for the current backup being taken.
+
 =item --host=HOST
 
 This option specifies the host to use when connecting to the database server with TCP/IP.  The option accepts a string argument. It is passed to the mysql child process without alteration. See mysql --help for details.
@@ -4930,6 +5253,14 @@
 
 This option specifies the directory where the incremental backup will be combined with the full backup to make a new full backup.  The option accepts a string argument. It is used with the --incremental option.
 
+=item --incremental-history-name=NAME
+
+This option specifies the name of the backup series stored in the PERCONA_SCHEMA.xtrabackup_history history record to base an incremental backup on. Xtrabackup will search the history table looking for the most recent (highest innodb_to_lsn), successful backup in the series and take the to_lsn value to use as the starting lsn for the incremental backup. This will be mutually exclusive with --incremental-history-uuid, --incremental-basedir and --incremental-lsn. If no valid lsn can be found (no series by that name, no successful backups by that name) xtrabackup will return with an error. It is used with the --incremental option.
+
+=item --incremental-history-uuid=UUID
+
+This option specifies the UUID of the specific history record stored in the PERCONA_SCHEMA.xtrabackup_history to base an incremental backup on. --incremental-history-name, --incremental-basedir and --incremental-lsn. If no valid lsn can be found (no success record with that uuid) xtrabackup will return with an error. It is used with the --incremental option.
+
 =item --incremental-force-scan
 
 This options tells xtrabackup to perform full scan of data files for taking an incremental backup even if full changed page bitmap data is available to enable the backup without the full scan.
 
=== added file 'xtrabackup/test/t/history_on_server.sh'
--- xtrabackup/test/t/history_on_server.sh	1970-01-01 00:00:00 +0000
+++ xtrabackup/test/t/history_on_server.sh	2013-09-25 17:11:22 +0000
@@ -0,0 +1,210 @@
+###############################################################################
+# Test history-on-server feature
+###############################################################################
+
+
+###############################################################################
+# Gets a single column value from the last history record added
+function get_one_value()
+{
+    local column=$1
+    val=`${MYSQL} ${MYSQL_ARGS} -Ns -e "SELECT $column FROM PERCONA_SCHEMA.xtrabackup_history ORDER BY start_time DESC LIMIT 1"`
+}
+
+
+###############################################################################
+# Checks a single column from the last history record added for some value and
+# not NULL.
+function check_for_not_NULL()
+{
+    local column=$1
+    get_one_value "$column"
+    if [ -z "$val" ] || [ "$val" = "NULL" ];
+    then
+        vlog "Error: $column in history record invalid, expected NOT NULL, got \"$val\""
+        exit 1
+    fi
+}
+
+
+###############################################################################
+# Checks a single column from the last history record added to see if is a
+# specific value.
+function check_for_value()
+{
+    local column=$1
+    shift
+    get_one_value "$column"
+    if [ -z "$val" ] || [ "$val" != "$@" ];
+    then
+        vlog "Error: $column in history record invalid, got \"$val\" expected \"$@\""
+        exit 1
+    fi
+}
+
+
+###############################################################################
+vlog "Prepping server"
+start_server
+load_dbase_schema incremental_sample
+multi_row_insert incremental_sample.test \({1..100},100\)
+backup_dir=$topdir/backups
+mkdir $backup_dir
+
+
+###############################################################################
+# This tests the to make sure that no xtrabackup_history unless --history is
+# specified
+#vlog "Testing no --history"
+#innobackupex --stream=tar $backup_dir > /dev/null
+
+#run_cmd_expect_failure get_one_value "uuid"
+
+
+###############################################################################
+# This tests the basic creation of a history record and that fields are
+# populated with some data. It also tests specifically that
+# partial, incremental, compact, compressed, encrypted and format are exactly
+# the correct values after the backup.
+# Missing is a test that binlog_pos is NULL, that would require restarting the
+# server without the log-bin option in the .cnf file but that has been tested
+# manually and doesn't seem to be something that would be required to be
+# validated.
+vlog "Testing basic history record"
+innobackupex --history=test1 --stream=tar $backup_dir > /dev/null
+
+for column in uuid name tool_name tool_command tool_version ibbackup_version \
+server_version start_time end_time lock_time binlog_pos innodb_from_lsn \
+innodb_to_lsn
+do
+    check_for_not_NULL "$column"
+done
+
+for column in partial incremental compact compressed encrypted
+do
+    check_for_value "$column" "N"
+done
+
+check_for_value "format" "tar"
+
+# saving for later
+get_one_value "innodb_to_lsn"
+first_to_lsn=$val
+
+
+###############################################################################
+# This tests the taking of an incremental backup based on the last record
+# of a history series and validates that the lsns in the record are correct.
+# It also tests that format, incremental and compact are exactly the correct
+# values after the backup.
+vlog "Testing incremental based on history name"
+
+multi_row_insert incremental_sample.test \({101..200},100\)
+
+innobackupex --history=test1 --incremental \
+--incremental-history-name=test1 --compact $backup_dir > /dev/null
+
+# saving for later
+get_one_value "uuid"
+second_uuid=$val
+get_one_value "innodb_from_lsn"
+second_from_lsn=$val
+get_one_value "innodb_to_lsn"
+second_to_lsn=$val
+
+check_for_value "format" "file"
+check_for_value "incremental" "Y"
+check_for_value "compact" "Y"
+
+if [ -z "$second_from_lsn" ] || [ "$second_from_lsn" != "$first_to_lsn" ]
+then
+    vlog "Second backup was not properly based on the to_lsn of the first"
+    exit 1
+fi
+
+multi_row_insert incremental_sample.test \({201..300},100\)
+
+# This will be a backup based on the last incremental just done, so, its
+# innodb_from_lsn (third_from_lsn) should be the same as the value in 
+# second_to_lsn. This tests that we find the right record in the test1 series
+# out of the two records that should be present before the backup is done.
+innobackupex --history=test1 --incremental \
+--incremental-history-name=test1 $backup_dir > /dev/null
+
+# saving for later
+get_one_value "uuid"
+third_uuid=$val
+get_one_value "innodb_from_lsn"
+third_from_lsn=$val
+get_one_value "innodb_to_lsn"
+third_to_lsn=$val
+
+if [ -z "$third_from_lsn" ] || [ "$third_from_lsn" != "$second_to_lsn" ]
+then
+    vlog "Third backup was not properly based on the to_lsn of the second"
+    exit 1
+fi
+
+
+###############################################################################
+# This tests that we can base an incremental on a specific history record
+# identified by its uuid that we captured earlier from a history record or it
+# could be scraped from the output of innobackupex at some point in the past.
+# It also tests specifically that incremental, compressed, encrypted and format
+# are exactly the correct values after the backup.
+# It tests that --history can be specified, resulting in a history record with
+# no name
+vlog "Testing incremental based on history uuid"
+multi_row_insert incremental_sample.test \({301..400},100\)
+
+innobackupex --history --incremental --incremental-history-uuid=$third_uuid \
+--stream=xbstream --compress --encrypt=AES256 \
+$backup_dir --encrypt-key=percona_xtrabackup_is_awesome___ > /dev/null
+
+get_one_value "innodb_from_lsn"
+fourth_from_lsn=$val
+
+for column in incremental compressed encrypted
+do
+    check_for_value "$column" "Y"
+done
+
+check_for_value "format" "xbstream"
+check_for_value "name" "NULL"
+
+# validate command tool and encrypt key scrubbibng but need to pop off first
+# two arguments in the result added by test framework function innobackupex
+get_one_value "tool_command"
+val=${val:`expr index "$val" " "`}; val=${val:`expr index "$val" " "`}
+expected_val="--history --incremental "\
+"--incremental-history-uuid=$third_uuid --stream=xbstream --compress "\
+"--encrypt=AES256 $backup_dir --encrypt-key=..."
+
+if [ -z "$val" ] || [ "$val" != "$expected_val" ]
+then
+  vlog "Error: $column in history record invalid, got \"$val\" expected \"$expected_val\""
+  exit 1
+fi
+
+if [ -z "$fourth_from_lsn" ] || [ "$fourth_from_lsn" != "$third_to_lsn" ]
+then
+    vlog "Fourth backup was not properly based on the to_lsn of the third"
+    exit 1
+fi
+
+
+###############################################################################
+# This tests that innobackupex fails when an invalid --incremental-history-name
+# is given.
+vlog "Testing bad --incremental-history-name"
+run_cmd_expect_failure $IB_BIN $IB_ARGS --incremental \
+--incremental-history-name=foo --stream=tar $backup_dir > /dev/null
+
+
+
+###############################################################################
+# This tests that innobackupex fails when an invalid --incremental-history-uuid
+# is given.
+vlog "Testing bad --incremental-history-uuid"
+run_cmd_expect_failure $IB_BIN $IB_ARGS --incremental \
+--incremental-history-uuid=foo --stream=tar $backup_dir > /dev/null