Merge lp:~laurynas-biveinis/percona-xtrabackup/xb-changed-page-bitmap into lp:percona-xtrabackup/2.1

[Laurynas Biveinis]

2nd MP:

http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.1-param/242/

Updated to the current 2.1 trunk.

1st MP:

BT 16274

http://jenkins.percona.com/job/percona-xtrabackup-2.1-param/237/

The MP has a couple of changes to anticipate the BT 28340 5.6 merge in innodb_int.h (introduction of lsn_t and friends). But this MP and the 28340 MP will need some updating depending on their trunk merge order.

Implement bitmap-based incremental backups.
Blueprints:
https://blueprints.launchpad.net/percona-xtrabackup/+spec/changed-page-bmp-inc-backups
and
https://blueprints.launchpad.net/percona-xtrabackup/+spec/multipl-bmp-file-inc-backup
- innobackupex: split the xtrabackup_suspended to three different
  files depending on the context (1st suspend, 2nd suspend, log copy
  end). If backing up, check if server supports changed page bitmaps
  by querying for the presence of I_S.INNODB_CHANGED_PAGES plugin. If
  found, add 1st suspend to xtrabackup invocation and issue FLUSH
  CHANGED_PAGE_BITMAPS during it. Factor out waitpid/sync file
  presence loop out of wait_for_ibbackup_suspend and resume_ibbackup
  into new subroutine wait_for_ibbackup_file_create. Split the log
  copying finish logic out of resume_ibbackup into a new subroutine
  wait_for_ibbackup_log_copy_finish.

- New source files changed_page_bitmap.h and changed_page_bitmap.cc
  containing the types and functions for working with page bitmaps and
  their iterators.

- New source files read_filt.h and read_filt.cc containing the data
  file read filter abstraction. Provide two filters: rf_pass_through
  for no filtering, and rf_bitmap that uses the changed page bitmaps.

- fil_cur.h, fil_cur.cc: xb_fil_cur_t: move offset field to
  xb_read_filt_ctxt_t. Add fields for the read filter and its
  context: read_filter and read_filter_ctxt. xb_fil_cur_open(): new
  arg read_filter. Initialize the read filter context.
  xb_fil_cur_read: use the read filter to get the next read offset and
  length. xb_fil_cur_close(): deinitialize the read filter.

- innodb_int.h: include mysql_version.h to make version checks not
  depend on mysql_version.h having been included before innodb_int.h.

- innodb_int.cc: fix an unrelated warning to make it easier to ensure
  that the feature does not regress in warnings: xb_space_get_by_name:
  mark fold as unused.

- xtrabackup.h: declare checkpoint_lsn_start and changed_page_bitmap.

- xtrabackup.cc: split xtrabackup_suspended to three different files
  as in innobackupex. New functions xb_make_sync_file_name and
  xtrabackup_suspend, rename xb_create_suspend_file to
  xb_create_sync_file. Add new option --suspend-at-start to suspend
  xtrabackup after the log copying thread has started.
  xtrabackup_copy_datafile: set up a bitmap read filter if the bitmap
  has been allocated, a pass through filter otherwise and pass it to
  the file cursor.
  xtrabackup_backup_func(): in case of incremental backups attempt to
  read the changed page bitmap. Write a message if the bitmap was
  found, otherwise that a full scan is going to be used. Free the
  bitmap in the end.

- test/inc/common.sh: init_server_variables(), reset_server_variables:
  set up server error log location in SRV_MYSQLD_ERRFILE[].
  switch_server(): set up MySQL error log.
  check_full_scan_inc_backup(), check_bitmap_inc_backup(): new
  functions for grepping XtraBackup output for the incremental backup
  type used. wait_for_xb_suspend(), resume_suspended_xb(): new helper
  functions for the testcases that use xtrabackup suspend.

- Rename test test/t/ib_incremental.sh to a test include file
  test/inc/ib_incremental_common.sh. Pass additional args in
  $mysqld_extra_args to mysqld.

- Rename test test/t/xb_incremental.sh to an include file
  test/inc/xb_incremental_common.sh. Pass additional args in
  $mysqld_extra_args to mysqld. Adjust server options to have a
  minimum possible log file size (1M) and thread concurrency that can
  support such size for 5.0. Change the schema of test table T2 to
  have large rows. Insert enough rows into a database so that the
  resulting bitmap file spans at least two bitmap pages for the T2
  tablespace.

- Rename test test/t/ib_incremental.sh to an include file
  test/inc/ib_incremental_common.sh. Pass additional args in
  $mysqld_extra_args to mysqld. Tweak it to increase the testing
  coverage. ADd two new testcases ib_incremental_bitmap.sh and
  ib_incremental_full_scan.sh that use it.

- Adjust test/t/xb_incremental_compressed.inc for bitmap backups. New
  tests xb_incremental_compressed_bitmap_(1-16)kb.sh. Rename
  the previous xb_incremental_compressed_*.sh tests to
  xb_incremental_compressed_full_scan_*.sh. Remove the InnoDB version
  check from xb_incremental_compressed.inc as it's always passing with
  the currently supported versions.

- New test xb_incremental_bitmap_missing.sh to test diagnostics for
  partially-missing bitmap data.

- Temporarily disable all the bitmap tests on XtraDB Cluster
  configuration until a supporting version is released.

- Adjust test/t/bug1007446.sh for changed suspend file names.

- Adjust test/t/xb_log_overwraph.sh to use
  wait_for_xb_to_suspend/resume_suspended_xb.

- Makefile: update dependencies.

[Alexey Kopytov (akopytov)]

I have merged the 28340 MP, so this probably needs updating now according to the comments?

review: Needs Information

[Laurynas Biveinis (laurynas-biveinis)]

Yes, I will update it. I expect mostly mechanical changes (innodb_int.h does not need to provide lsn_t, 64-bit args instead of 32-bit pairs for I/O, etc).

[Alexey Kopytov (akopytov)]

Laurynas,

There's no time for a detailed review, but:

   - it would be nice to have an option to force the old .delta
     calculation method even if bitmaps are available; something like
     --incremental-force-scan?
   - there's no need to append ';' to the end of a query in
     mysql_query()
   - the optimization attempt to wrap multiple INSERT statements for
     incremental tests into BEGIN/COMMIT does not work, as we disconnect
     between mysql client invokations; it should be smth like the
     following:
: (echo "BEGIN;"; echo "..."; echo "COMMIT;") | $MYSQL $MYSQL_ARGS ...
   - would updating docs in the same MP be a stretch? Just trying to
     avoid the communication overhead when docs are updated by Hrvoje,
     and the required docs updates seem to be minor

Please let me know if addressing the above would take more than a day. We can then merge the current MP as is, and address the remaining stuff later.

review: Needs Fixing

[Laurynas Biveinis (laurynas-biveinis)]

> - it would be nice to have an option to force the old .delta
> calculation method even if bitmaps are available; something like
> --incremental-force-scan?

Yes.

> - there's no need to append ';' to the end of a query in
> mysql_query()

Ack

> - the optimization attempt to wrap multiple INSERT statements for
> incremental tests into BEGIN/COMMIT does not work, as we disconnect
> between mysql client invokations; it should be smth like the
> following:
> : (echo "BEGIN;"; echo "..."; echo "COMMIT;") | $MYSQL $MYSQL_ARGS ...

Right, I forgot that it's all separate connections, thus separate transactions. I had some sort of persistent connection setup in progress at bug 1119451, comments #2 and #3. I think here we can just go with one statement transactions and optimize later as needed (bug 1119451 stuff is incomplete).

> - would updating docs in the same MP be a stretch? Just trying to
> avoid the communication overhead when docs are updated by Hrvoje,
> and the required docs updates seem to be minor

Will review, should be feasible.

> Please let me know if addressing the above would take more than a day. We can
> then merge the current MP as is, and address the remaining stuff later.

It should take less than a day, and I will work on this today (maybe tomorrow morning will be needed to wait for Jenkins run etc).

=== modified file 'doc/source/innobackupex/how_innobackupex_works.rst'
--- doc/source/innobackupex/how_innobackupex_works.rst	2012-09-07 13:20:26 +0000
+++ doc/source/innobackupex/how_innobackupex_works.rst	2013-04-15 15:08:38 +0000
@@ -15,7 +15,7 @@
 
 If no mode is specified, |innobackupex| will assume the backup mode.
 
-By default, it starts :program:`xtrabackup` with the :option:`--suspend-at-end` option, and lets it copy the InnoDB data files. When :program:`xtrabackup` finishes that, :program:`innobackupex` sees it create the :file:`xtrabackup_suspended` file and executes ``FLUSH TABLES WITH READ LOCK``. Then it begins copying the rest of the files.
+By default, it starts :program:`xtrabackup` with the :option:`--suspend-at-end` option, and lets it copy the InnoDB data files. When :program:`xtrabackup` finishes that, :program:`innobackupex` sees it create the :file:`xtrabackup_suspended_2` file and executes ``FLUSH TABLES WITH READ LOCK``. Then it begins copying the rest of the files.
 
 If the :option:`--ibbackup` is not supplied, |innobackupex| will try to detect it: if the :file:`xtrabackup_binary` file exists on the backup directory, it reads from it which binary of |xtrabackup| will be used. Otherwise, it will try to connect to the database server in order to determine it. If the connection can't be established, |xtrabackup| will fail and you must specify it (see :ref:`ibk-right-binary`).
 
@@ -25,7 +25,7 @@
 
 Once this is done, the backup of the files will begin. It will backup :term:`.frm`, :term:`.MRG`, :term:`.MYD`, :term:`.MYI`, :term:`.TRG`, :term:`.TRN`, :term:`.ARM`, :term:`.ARZ`, :term:`.CSM`, :term:`.CSV` and :term:`.opt` files.
 
-When all the files are backed up, it resumes :program:`ibbackup` and wait until it finishes copying the transactions done while the backup was done. Then, the tables are unlocked, the slave is started (if the option :option:`--safe-slave-backup` was used) and the connection with the server is closed. Then, it removes the :file:`xtrabackup_suspended` file and permits :program:`xtrabackup` to exit.
+When all the files are backed up, it resumes :program:`ibbackup` and wait until it finishes copying the transactions done while the backup was done. Then, the tables are unlocked, the slave is started (if the option :option:`--safe-slave-backup` was used) and the connection with the server is closed. Then, it removes the :file:`xtrabackup_suspended_2` file and permits :program:`xtrabackup` to exit.
 
 It  will also create the following files in the directory of the backup:
 

=== modified file 'doc/source/xtrabackup_bin/incremental_backups.rst'
--- doc/source/xtrabackup_bin/incremental_backups.rst	2013-01-17 15:08:37 +0000
+++ doc/source/xtrabackup_bin/incremental_backups.rst	2013-04-15 15:08:38 +0000
@@ -4,7 +4,7 @@
 
 Both |xtrabackup| and |innobackupex| tools supports incremental backups, which means that it can copy only the data that has changed since the last full backup. You can perform many incremental backups between each full backup, so you can set up a backup process such as a full backup once a week and an incremental backup every day, or full backups every day and incremental backups every hour.
 
-Incremental backups work because each InnoDB page (usually 16kb in size) contains a log sequence number, or :term:`LSN`. The :term:`LSN` is the system version number for the entire database. Each page's :term:`LSN` shows how recently it was changed. An incremental backup copies each page whose :term:`LSN` is newer than the previous incremental or full backup's :term:`LSN`.
+Incremental backups work because each InnoDB page (usually 16kb in size) contains a log sequence number, or :term:`LSN`. The :term:`LSN` is the system version number for the entire database. Each page's :term:`LSN` shows how recently it was changed. An incremental backup copies each page whose :term:`LSN` is newer than the previous incremental or full backup's :term:`LSN`. There are two algorithms in use to find the set of such pages to be copied. The first one, available with all the server types and versions, is to check the page :term:`LSN` directly by reading all the data pages. The second one, available with |Percona Server|, is to enable the changed page tracking feature on the server, which will note the pages as they are being changed. This information will be then written out in a compact separate so-called bitmap file. The |xtrabackup| binary will use that file to read only the data pages it needs for the incremental backup, potentially saving many read requests. The latter algorithm is enabled by default if the |xtrabackup| binary finds the bitmap file. It is possible to specify :option:`--incremental-force-scan` to read all the pages even if the bitmap data is available.
 
 Incremental backups do not actually compare the data files to the previous backup's data files. In fact, you can use :option:`--incremental-lsn` to perform an incremental backup without even having the previous backup, if you know its :term:`LSN`. Incremental backups simply read the pages and compare their :term:`LSN` to the last backup's :term:`LSN`. You still need a full backup to recover the incremental changes, however; without a full backup to act as a base, the incremental backups are useless.
 

=== modified file 'doc/source/xtrabackup_bin/xbk_option_reference.rst'
--- doc/source/xtrabackup_bin/xbk_option_reference.rst	2012-05-18 11:19:17 +0000
+++ doc/source/xtrabackup_bin/xbk_option_reference.rst	2013-04-15 15:08:38 +0000
@@ -65,6 +65,10 @@
 
    When preparing an incremental backup, this is the directory where the incremental backup is combined with the full backup to make a new full backup.
 
+.. option:: --incremental-force-scan
+
+   When creating an incremental backup, force a full scan of the data pages in the instance being backuped even if the complete changed page bitmap data is available.
+
 .. option:: --incremental-lsn=name
 
    When creating an incremental backup, you can specify the log sequence number (:term:`LSN`) instead of specifying :option:`--incremental-basedir`. For databases created by *MySQL* and *Percona Server* 5.0-series versions, specify the :term:`LSN` as two 32-bit integers in high:low format. For databases created in 5.1 and later, specify the :term:`LSN` as a single 64-bit integer.  ##ATTENTION##: If a wrong LSN value is specified, it is impossible to diagnose this, causing the backup to be unusable. Be careful!

=== modified file 'innobackupex'
--- innobackupex	2013-04-09 10:58:46 +0000
+++ innobackupex	2013-04-15 15:08:38 +0000
@@ -71,6 +71,10 @@
 # end of modifiable parameters
 ######################################################################
 
+#######################
+# Server capabilities #
+#######################
+my $have_changed_page_bitmaps = 0;
 
 # command line options
 my $option_help = '';
@@ -144,8 +148,8 @@
 # backup directory pathname
 my $backup_dir = '';
 
-# name of the ibbackup suspend-at-end file
-my $suspend_file = '';
+# directory where innobackupex aux files are cretead
+my $work_dir;
 
 # home directory of innoDB log files
 my $innodb_log_group_home_dir = '';
@@ -221,6 +225,12 @@
 my $xtrabackup_pid_file = 'xtrabackup_pid';
 my %rsync_files_hash;
 
+my $xb_fn_suspended_at_start = "/xtrabackup_suspended_1";
+my $xb_fn_suspended_at_end = "/xtrabackup_suspended_2";
+my $xb_fn_log_copied = "/xtrabackup_log_copied";
+my @xb_suspend_files = ($xb_fn_suspended_at_start, $xb_fn_suspended_at_end,
+                        $xb_fn_log_copied);
+
 my $copy_dir_src;
 my $copy_dir_dst;
 my $copy_dir_exclude_regexp;
@@ -379,12 +389,25 @@
 #
 sub backup {
     my $orig_datadir = get_option(\%config, $option_defaults_group, 'datadir');
+    my $suspend_file;
+
+    if ($option_incremental) {
+        detect_mysql_capabilities_for_backup(\%mysql);
+    }
 
     # start ibbackup as a child process
     start_ibbackup();
 
+    if ($option_incremental && $have_changed_page_bitmaps) {
+        $suspend_file = $work_dir . $xb_fn_suspended_at_start;
+        wait_for_ibbackup_suspend($suspend_file);
+        mysql_query(\%mysql, 'FLUSH NO_WRITE_TO_BINLOG CHANGED_PAGE_BITMAPS');
+        resume_ibbackup($suspend_file);
+    }
+
     # wait for ibbackup to suspend itself
-    wait_for_ibbackup_suspend();
+    $suspend_file = $work_dir . $xb_fn_suspended_at_end;
+    wait_for_ibbackup_suspend($suspend_file);
 
     if ($option_safe_slave_backup) {
         wait_for_safe_slave(\%mysql);
@@ -408,8 +431,10 @@
     # (or finalize the backup by syncing changes if using rsync)
     backup_files(0);
 
-    # resume ibbackup and wait till log copying is finished
-    resume_ibbackup();
+    # resume XtraBackup and wait till it has finished
+    resume_ibbackup($suspend_file);
+    $suspend_file = $work_dir . $xb_fn_log_copied;
+    wait_for_ibbackup_log_copy_finish($suspend_file);
 
     # release read locks on all tables
     mysql_unlockall(\%mysql) if !$option_no_lock;
@@ -887,24 +912,50 @@
 
 }
 
-
-#
-# wait_for_ibbackup_suspend subroutine waits until ibbackup has suspended 
+#
+# Loops until the specified file is created or the ibbackup process dies.
+#
+#
+# If this function detects that the xtrabackup process has terminated, it also
+# sets ibbackup_exit_code and resets ibbackup_pid to avoid trying to reap a
+# non-existing process later
+#
+sub wait_for_ibbackup_file_create {
+
+    my $suspend_file = shift;
+    my $pid = -1;
+
+    for (;;) {
+        # check if the xtrabackup process is still alive _before_ checking if
+        # the file exists to avoid a race condition when the file is created
+        # and the process terminates right after we do the file check
+        $pid = waitpid($ibbackup_pid, &WNOHANG);
+
+        last if -e $suspend_file;
+
+        if ($ibbackup_pid == $pid) {
+            # The file doesn't exist, but the process has terminated
+            Die "ibbackup child process has died";
+        }
+
+        sleep 1;
+    }
+
+    if ($pid == $ibbackup_pid) {
+        $ibbackup_exit_code = $CHILD_ERROR >> 8;
+        $ibbackup_pid = '';
+    }
+}
+
+#
+# wait_for_ibbackup_suspend subroutine waits until ibbackup has suspended
 # itself.
 #
 sub wait_for_ibbackup_suspend {
+    my $suspend_file = shift;
     print STDERR "$prefix Waiting for ibbackup (pid=$ibbackup_pid) to suspend\n";
     print STDERR "$prefix Suspend file '$suspend_file'\n\n";
-    for (;;) {
-        usleep(100000);
-        last if -e $suspend_file;
-
-        # check that ibbackup child process is still alive
-        if ($ibbackup_pid == waitpid($ibbackup_pid, &WNOHANG)) {
-            $ibbackup_pid = '';
-            Die "ibbackup child process has died";
-        }
-    }
+    wait_for_ibbackup_file_create($suspend_file);
     $now = current_time();
     open XTRABACKUP_PID, "> $option_tmpdir/$xtrabackup_pid_file";
     print XTRABACKUP_PID $ibbackup_pid;
@@ -912,43 +963,27 @@
     print STDERR "\n$now  $prefix Continuing after ibbackup has suspended\n";
 }
 
-
 #
 # resume_ibbackup subroutine signals ibbackup to finish log copying by deleting
-# the 'xtrabackup_suspended' file and waits until log copying is stopped,
-# i.e. until 'xtrabackup_suspended' is created again.
-#
-# If this functions detects that the xtrabackup process has terminated, it also
-# sets ibbackup_exit_code and resets ibbackup_pid to avoid trying to reap a
-# non-existing process later
+# $suspend_file.
 #
 sub resume_ibbackup {
-    my $pid = -1;
+    my $suspend_file = shift;
+    unlink $suspend_file || Die "Failed to delete '$suspend_file': $!";
+}
 
-    $now = current_time();
+#
+# Wait until xtrabackup finishes copying log or dies.  The log copying is
+# signaled by a sync file creation.  If this function detects that the
+# xtrabackup process has terminated, it also sets ibbackup_exit_code and resets
+# ibbackup_pid to avoid trying to reap a non-existing process later.
+#
+sub wait_for_ibbackup_log_copy_finish {
+    my $suspend_file = shift;
+    my $now = current_time();
     print STDERR "$now  $prefix Waiting for log copying to finish\n\n";
-    unlink $suspend_file || Die "Failed to delete '$suspend_file': $!";
-
-    for (;;) {
-        # check if the xtrabackup process is still alive _before_ checking if
-        # the file exists to avoid a race condition when the file is created and
-        # the process terminates right after we do the file check
-        $pid = waitpid($ibbackup_pid, &WNOHANG);
-
-        last if -e $suspend_file;
-
-        if ($ibbackup_pid == $pid) {
-            # The file doesn't exist, but the process has terminated
-            Die "ibbackup child process has died";
-        }
-
-        sleep 1;
-    }
-
-    if ($pid == $ibbackup_pid) {
-        $ibbackup_exit_code = $CHILD_ERROR >> 8;
-        $ibbackup_pid = '';
-    }
+
+    wait_for_ibbackup_file_create($suspend_file);
 
     unlink $suspend_file || Die "Failed to delete '$suspend_file': $!";
 }
@@ -972,7 +1007,6 @@
     $ibbackup_pid = '';
     return $CHILD_ERROR >> 8;
 }
-    
 
 #
 # start_ibbackup subroutine spawns a child process running ibbackup
@@ -1051,6 +1085,9 @@
         } else {
           $options = $options . " --incremental-basedir='$incremental_basedir'";
         }
+        if ($have_changed_page_bitmaps) {
+            $options = $options . " --suspend-at-start";
+        }
     }
 
     if ($option_tables_file) {
@@ -1638,28 +1675,31 @@
         $backup_dir = File::Spec->rel2abs(make_backup_dir());
         print STDERR "$prefix Created backup directory $backup_dir\n";
         if (!$option_stream) {
-            $backup_config_file = $backup_dir . '/backup-my.cnf';
-            $suspend_file = $backup_dir . '/xtrabackup_suspended';
-            $binlog_info = $backup_dir . '/xtrabackup_binlog_info';
-            $galera_info = $backup_dir . '/xtrabackup_galera_info';
-            $slave_info = $backup_dir . '/xtrabackup_slave_info';
+            $work_dir = $backup_dir;
         } else {
-            $suspend_file = $option_tmpdir . '/xtrabackup_suspended';
-	    $backup_config_file = $option_tmpdir . '/backup-my.cnf';
-	    $binlog_info = $option_tmpdir . '/xtrabackup_binlog_info';
-	    $galera_info = $option_tmpdir . '/xtrabackup_galera_info';
-	    $slave_info = $option_tmpdir . '/xtrabackup_slave_info';
+            $work_dir = $option_tmpdir;
         }
+        $backup_config_file = $work_dir . '/backup-my.cnf';
+        $binlog_info = $work_dir . '/xtrabackup_binlog_info';
+        $galera_info = $work_dir . '/xtrabackup_galera_info';
+        $slave_info = $work_dir . '/xtrabackup_slave_info';
         write_backup_config_file($backup_config_file);
+
+        foreach (@xb_suspend_files) {
+            my $suspend_file = $work_dir . "$_";
+            if ( -e "$suspend_file" ) {
+                print STDERR
+                    "WARNING : A left over instance of " .
+                    "suspend file '$suspend_file' was found.\n";
+                unlink "$suspend_file"
+                    || Die "Failed to delete '$suspend_file': $!";
+            }
+        }
+
     } elsif ($option_copy_back || $option_move_back) {
         #$backup_config_file = $backup_dir . '/backup-my.cnf';
         #read_config_file($backup_config_file, \%backup_config);
     }
-
-    if ( -e "$suspend_file" ) {
-        print STDERR "WARNING : A left over instance of suspend file '$suspend_file' was found.\n";
-        unlink $suspend_file || Die "Failed to delete '$suspend_file': $!";
-    }
 }
 
 
@@ -2730,6 +2770,14 @@
   return $rcode;
 }
 
+#
+# Query the server to find out what backup capabilities it supports.
+#
+sub detect_mysql_capabilities_for_backup {
+    $have_changed_page_bitmaps =
+        mysql_query($_[0], 'SELECT COUNT(*) FROM INFORMATION_SCHEMA.PLUGINS '.
+                    'WHERE PLUGIN_NAME LIKE "INNODB_CHANGED_PAGES"')
+}
 
 =pod
 

=== modified file 'src/Makefile'
--- src/Makefile	2013-04-09 07:24:21 +0000
+++ src/Makefile	2013-04-15 15:08:38 +0000
@@ -41,7 +41,9 @@
 	datasink.o \
 	xbstream_write.o \
 	quicklz/quicklz.o
-XTRABACKUPCCOBJS = xtrabackup.o innodb_int.o compact.o fil_cur.o write_filt.o
+XTRABACKUPCCOBJS = xtrabackup.o innodb_int.o compact.o fil_cur.o write_filt.o \
+	changed_page_bitmap.o \
+	read_filt.o
 
 XBSTREAMOBJS = xbstream.o xbstream_write.o xbstream_read.o
 
@@ -231,7 +233,14 @@
 xbcrypt: $(XBCRYPTOBJS) $(MYSQLOBJS)
 	$(CXX) $(CXXFLAGS) $^ $(INC) $(MYSQLOBJS) $(LIBS) -o $@
 
-xtrabackup.o: xtrabackup.cc xb_regex.h write_filt.h fil_cur.h xtrabackup.h compact.h
+changed_page_bitmap.o: changed_page_bitmap.cc changed_page_bitmap.h innodb_int.h \
+	common.h xtrabackup.h
+
+read_filt.o: read_filt.cc read_filt.h fil_cur.h xtrabackup.h innodb_int.h \
+	common.h changed_page_bitmap.h
+
+xtrabackup.o: xtrabackup.cc xb_regex.h write_filt.h fil_cur.h xtrabackup.h compact.h \
+	common.h changed_page_bitmap.h read_filt.h innodb_int.h
 
 $(TARGET): $(XTRABACKUPCCOBJS) $(XTRABACKUPCOBJS) $(INNODBOBJS) $(MYSQLOBJS) $(LIBARCHIVE_A)
 	$(CXX) $(CFXXLAGS) $(XTRABACKUPCCOBJS) $(XTRABACKUPCOBJS) $(INNODBOBJS) $(MYSQLOBJS) $(LIBS) \

=== added file 'src/changed_page_bitmap.cc'
--- src/changed_page_bitmap.cc	1970-01-01 00:00:00 +0000
+++ src/changed_page_bitmap.cc	2013-04-15 15:08:38 +0000
@@ -0,0 +1,1013 @@
+/******************************************************
+XtraBackup: hot backup tool for InnoDB
+(c) 2009-2012 Percona Inc.
+Originally Created 3/3/2009 Yasufumi Kinoshita
+Written by Alexey Kopytov, Aleksandr Kuzminsky, Stewart Smith, Vadim Tkachenko,
+Yasufumi Kinoshita, Ignacio Nin and Baron Schwartz.
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*******************************************************/
+
+/* Changed page bitmap implementation */
+
+#include "changed_page_bitmap.h"
+
+#include "common.h"
+#include "innodb_int.h"
+#include "xtrabackup.h"
+
+/* TODO: copy-pasted shared definitions from the XtraDB bitmap write code.
+Remove these on the first opportunity, i.e. single-binary XtraBackup.  */
+
+/* log0online.h */
+
+/** Single bitmap file information */
+struct log_online_bitmap_file_t {
+	char		name[FN_REFLEN];	/*!< Name with full path */
+	os_file_t	file;			/*!< Handle to opened file */
+	ib_uint64_t	size;			/*!< Size of the file */
+	ib_uint64_t	offset;			/*!< Offset of the next read,
+						or count of already-read bytes
+						*/
+};
+
+/** A set of bitmap files containing some LSN range */
+struct log_online_bitmap_file_range_t {
+	size_t	count;				/*!< Number of files */
+	/*!< Dynamically-allocated array of info about individual files */
+	struct files_t {
+		char	name[FN_REFLEN];/*!< Name of a file */
+		lsn_t	start_lsn;	/*!< Starting LSN of data in this
+					file */
+		ulong	seq_num;	/*!< Sequence number of this file */
+	}	*files;
+};
+
+/* log0online.c */
+
+/** File name stem for bitmap files. */
+static const char* bmp_file_name_stem = "ib_modified_log_";
+
+/** The bitmap file block size in bytes.  All writes will be multiples of this.
+ */
+enum {
+	MODIFIED_PAGE_BLOCK_SIZE = 4096
+};
+
+/** Offsets in a file bitmap block */
+enum {
+	MODIFIED_PAGE_IS_LAST_BLOCK = 0,/* 1 if last block in the current
+					write, 0 otherwise. */
+	MODIFIED_PAGE_START_LSN = 4,	/* The starting tracked LSN of this and
+					other blocks in the same write */
+	MODIFIED_PAGE_END_LSN = 12,	/* The ending tracked LSN of this and
+					other blocks in the same write */
+	MODIFIED_PAGE_SPACE_ID = 20,	/* The space ID of tracked pages in
+					this block */
+	MODIFIED_PAGE_1ST_PAGE_ID = 24,	/* The page ID of the first tracked
+					page in this block */
+	MODIFIED_PAGE_BLOCK_UNUSED_1 = 28,/* Unused in order to align the start
+					  of bitmap at 8 byte boundary */
+	MODIFIED_PAGE_BLOCK_BITMAP = 32,/* Start of the bitmap itself */
+	MODIFIED_PAGE_BLOCK_UNUSED_2 = MODIFIED_PAGE_BLOCK_SIZE - 8,
+					/* Unused in order to align the end of
+					bitmap at 8 byte boundary */
+	MODIFIED_PAGE_BLOCK_CHECKSUM = MODIFIED_PAGE_BLOCK_SIZE - 4
+					/* The checksum of the current block */
+};
+
+/** Length of the bitmap data in a block */
+enum { MODIFIED_PAGE_BLOCK_BITMAP_LEN
+       = MODIFIED_PAGE_BLOCK_UNUSED_2 - MODIFIED_PAGE_BLOCK_BITMAP };
+
+/** Length of the bitmap data in a block in page ids */
+enum { MODIFIED_PAGE_BLOCK_ID_COUNT = MODIFIED_PAGE_BLOCK_BITMAP_LEN * 8 };
+
+typedef ib_uint64_t	bitmap_word_t;
+
+/****************************************************************//**
+Calculate a bitmap block checksum.  Algorithm borrowed from
+log_block_calc_checksum.
+@return checksum */
+UNIV_INLINE
+ulint
+log_online_calc_checksum(
+/*=====================*/
+	const byte*	block);	/*!<in: bitmap block */
+
+/****************************************************************//**
+Provide a comparisson function for the RB-tree tree (space,
+block_start_page) pairs.  Actual implementation does not matter as
+long as the ordering is full.
+@return -1 if p1 < p2, 0 if p1 == p2, 1 if p1 > p2
+*/
+static
+int
+log_online_compare_bmp_keys(
+/*========================*/
+	const void* p1,	/*!<in: 1st key to compare */
+	const void* p2)	/*!<in: 2nd key to compare */
+{
+	const byte *k1 = (const byte *)p1;
+	const byte *k2 = (const byte *)p2;
+
+	ulint k1_space = mach_read_from_4(k1 + MODIFIED_PAGE_SPACE_ID);
+	ulint k2_space = mach_read_from_4(k2 + MODIFIED_PAGE_SPACE_ID);
+	if (k1_space == k2_space) {
+
+		ulint k1_start_page
+			= mach_read_from_4(k1 + MODIFIED_PAGE_1ST_PAGE_ID);
+		ulint k2_start_page
+			= mach_read_from_4(k2 + MODIFIED_PAGE_1ST_PAGE_ID);
+		return k1_start_page < k2_start_page
+			? -1 : k1_start_page > k2_start_page ? 1 : 0;
+	}
+	return k1_space < k2_space ? -1 : 1;
+}
+
+/****************************************************************//**
+Calculate a bitmap block checksum.  Algorithm borrowed from
+log_block_calc_checksum.
+@return checksum */
+UNIV_INLINE
+ulint
+log_online_calc_checksum(
+/*=====================*/
+	const byte*	block)	/*!<in: bitmap block */
+{
+	ulint	sum;
+	ulint	sh;
+	ulint	i;
+
+	sum = 1;
+	sh = 0;
+
+	for (i = 0; i < MODIFIED_PAGE_BLOCK_CHECKSUM; i++) {
+
+		ulint	b = block[i];
+		sum &= 0x7FFFFFFFUL;
+		sum += b;
+		sum += b << sh;
+		sh++;
+		if (sh > 24) {
+
+			sh = 0;
+		}
+	}
+
+	return sum;
+}
+
+/****************************************************************//**
+Read one bitmap data page and check it for corruption.
+
+@return TRUE if page read OK, FALSE if I/O error */
+static
+ibool
+log_online_read_bitmap_page(
+/*========================*/
+	log_online_bitmap_file_t	*bitmap_file,	/*!<in/out: bitmap
+							file */
+	byte				*page,	/*!<out: read page.  Must be at
+						least MODIFIED_PAGE_BLOCK_SIZE
+						bytes long */
+	ibool				*checksum_ok)	/*!<out: TRUE if page
+							checksum OK */
+{
+	ulint	checksum;
+	ulint	actual_checksum;
+	ibool	success;
+
+	ut_a(bitmap_file->size >= MODIFIED_PAGE_BLOCK_SIZE);
+	ut_a(bitmap_file->offset
+	     <= bitmap_file->size - MODIFIED_PAGE_BLOCK_SIZE);
+	ut_a(bitmap_file->offset % MODIFIED_PAGE_BLOCK_SIZE == 0);
+
+	success = xb_os_file_read(bitmap_file->file, page, bitmap_file->offset,
+				  MODIFIED_PAGE_BLOCK_SIZE);
+
+	if (UNIV_UNLIKELY(!success)) {
+
+		/* The following call prints an error message */
+		os_file_get_last_error(TRUE);
+		msg("InnoDB: Warning: failed reading changed page bitmap "
+		    "file \'%s\'\n", bitmap_file->name);
+		return FALSE;
+	}
+
+	bitmap_file->offset += MODIFIED_PAGE_BLOCK_SIZE;
+	ut_ad(bitmap_file->offset <= bitmap_file->size);
+
+	checksum = mach_read_from_4(page + MODIFIED_PAGE_BLOCK_CHECKSUM);
+	actual_checksum = log_online_calc_checksum(page);
+	*checksum_ok = (checksum == actual_checksum);
+
+	return TRUE;
+}
+
+/*********************************************************************//**
+Check the name of a given file if it's a changed page bitmap file and
+return file sequence and start LSN name components if it is.  If is not,
+the values of output parameters are undefined.
+
+@return TRUE if a given file is a changed page bitmap file.  */
+static
+ibool
+log_online_is_bitmap_file(
+/*======================*/
+	const os_file_stat_t*	file_info,		/*!<in: file to
+							check */
+	ulong*			bitmap_file_seq_num,	/*!<out: bitmap file
+							sequence number */
+	lsn_t*			bitmap_file_start_lsn)	/*!<out: bitmap file
+							start LSN */
+{
+	char	stem[FN_REFLEN];
+
+	ut_ad (strlen(file_info->name) < OS_FILE_MAX_PATH);
+
+	return ((file_info->type == OS_FILE_TYPE_FILE
+		 || file_info->type == OS_FILE_TYPE_LINK)
+		&& (sscanf(file_info->name, "%[a-z_]%lu_" LSN_PF ".xdb", stem,
+			   bitmap_file_seq_num, bitmap_file_start_lsn) == 3)
+		&& (!strcmp(stem, bmp_file_name_stem)));
+}
+
+/*********************************************************************//**
+List the bitmap files in srv_data_home and setup their range that contains the
+specified LSN interval.  This range, if non-empty, will start with a file that
+has the greatest LSN equal to or less than the start LSN and will include all
+the files up to the one with the greatest LSN less than the end LSN.  Caller
+must free bitmap_files->files when done if bitmap_files set to non-NULL and
+this function returned TRUE.  Field bitmap_files->count might be set to a
+larger value than the actual count of the files, and space for the unused array
+slots will be allocated but cleared to zeroes.
+
+@return TRUE if succeeded
+*/
+static
+ibool
+log_online_setup_bitmap_file_range(
+/*===============================*/
+	log_online_bitmap_file_range_t	*bitmap_files,	/*!<in/out: bitmap file
+							range */
+	lsn_t				range_start,	/*!<in: start LSN */
+	lsn_t				range_end)	/*!<in: end LSN */
+{
+	os_file_dir_t	bitmap_dir;
+	os_file_stat_t	bitmap_dir_file_info;
+	ulong		first_file_seq_num	= ULONG_MAX;
+	ulong		last_file_seq_num	= 0;
+	lsn_t		first_file_start_lsn	= 0;
+
+	xb_ad(range_end >= range_start);
+
+	bitmap_files->count = 0;
+	bitmap_files->files = NULL;
+
+	/* 1st pass: size the info array */
+
+	bitmap_dir = os_file_opendir(srv_data_home, FALSE);
+	if (UNIV_UNLIKELY(!bitmap_dir)) {
+
+		msg("InnoDB: Error: failed to open bitmap directory \'%s\'\n",
+		    srv_data_home);
+		return FALSE;
+	}
+
+	while (!os_file_readdir_next_file(srv_data_home, bitmap_dir,
+					  &bitmap_dir_file_info)) {
+
+		ulong	file_seq_num;
+		lsn_t	file_start_lsn;
+
+		if (!log_online_is_bitmap_file(&bitmap_dir_file_info,
+					       &file_seq_num,
+					       &file_start_lsn)
+		    || file_start_lsn >= range_end) {
+
+			continue;
+		}
+
+		if (file_seq_num > last_file_seq_num) {
+
+			last_file_seq_num = file_seq_num;
+		}
+
+		if (file_start_lsn >= range_start
+		    || file_start_lsn == first_file_start_lsn
+		    || first_file_start_lsn > range_start) {
+
+			/* A file that falls into the range */
+
+			if (file_start_lsn < first_file_start_lsn) {
+
+				first_file_start_lsn = file_start_lsn;
+			}
+			if (file_seq_num < first_file_seq_num) {
+
+				first_file_seq_num = file_seq_num;
+			}
+		} else if (file_start_lsn > first_file_start_lsn) {
+
+			/* A file that has LSN closer to the range start
+			but smaller than it, replacing another such file */
+			first_file_start_lsn = file_start_lsn;
+			first_file_seq_num = file_seq_num;
+		}
+	}
+
+	if (UNIV_UNLIKELY(os_file_closedir(bitmap_dir))) {
+
+		os_file_get_last_error(TRUE);
+		msg("InnoDB: Error: cannot close \'%s\'\n",srv_data_home);
+		return FALSE;
+	}
+
+	if (first_file_seq_num == ULONG_MAX && last_file_seq_num == 0) {
+
+		bitmap_files->count = 0;
+		return TRUE;
+	}
+
+	bitmap_files->count = last_file_seq_num - first_file_seq_num + 1;
+
+	/* 2nd pass: get the file names in the file_seq_num order */
+
+	bitmap_dir = os_file_opendir(srv_data_home, FALSE);
+	if (UNIV_UNLIKELY(!bitmap_dir)) {
+
+		msg("InnoDB: Error: failed to open bitmap directory \'%s\'\n",
+		    srv_data_home);
+		return FALSE;
+	}
+
+	bitmap_files->files =
+		static_cast<log_online_bitmap_file_range_t::files_t *>
+		(ut_malloc(bitmap_files->count
+			   * sizeof(bitmap_files->files[0])));
+	memset(bitmap_files->files, 0,
+	       bitmap_files->count * sizeof(bitmap_files->files[0]));
+
+	while (!os_file_readdir_next_file(srv_data_home, bitmap_dir,
+					  &bitmap_dir_file_info)) {
+
+		ulong	file_seq_num;
+		lsn_t	file_start_lsn;
+		size_t	array_pos;
+
+		if (!log_online_is_bitmap_file(&bitmap_dir_file_info,
+					       &file_seq_num,
+					       &file_start_lsn)
+		    || file_start_lsn >= range_end
+		    || file_start_lsn < first_file_start_lsn) {
+
+			continue;
+		}
+
+		array_pos = file_seq_num - first_file_seq_num;
+		xb_a(array_pos < bitmap_files->count);
+		if (file_seq_num > bitmap_files->files[array_pos].seq_num) {
+
+			bitmap_files->files[array_pos].seq_num = file_seq_num;
+			strncpy(bitmap_files->files[array_pos].name,
+				bitmap_dir_file_info.name, FN_REFLEN);
+			bitmap_files->files[array_pos].name[FN_REFLEN - 1]
+				= '\0';
+			bitmap_files->files[array_pos].start_lsn
+				= file_start_lsn;
+		}
+	}
+
+	if (UNIV_UNLIKELY(os_file_closedir(bitmap_dir))) {
+
+		os_file_get_last_error(TRUE);
+		msg("InnoDB: Error: cannot close \'%s\'\n", srv_data_home);
+		free(bitmap_files->files);
+		return FALSE;
+	}
+
+#ifdef UNIV_DEBUG
+	ut_ad(bitmap_files->files[0].seq_num == first_file_seq_num);
+	ut_ad(bitmap_files->files[0].start_lsn == first_file_start_lsn);
+
+	for (size_t i = 1; i < bitmap_files->count; i++) {
+		if (!bitmap_files->files[i].seq_num) {
+
+			break;
+		}
+		ut_ad(bitmap_files->files[i].seq_num
+		      > bitmap_files->files[i - 1].seq_num);
+		ut_ad(bitmap_files->files[i].start_lsn
+		      >= bitmap_files->files[i - 1].start_lsn);
+	}
+#endif
+
+	return TRUE;
+}
+
+/****************************************************************//**
+Open a bitmap file for reading.
+
+@return TRUE if opened successfully */
+static
+ibool
+log_online_open_bitmap_file_read_only(
+/*==================================*/
+	const char*			name,		/*!<in: bitmap file
+							name without directory,
+							which is assumed to be
+							srv_data_home */
+	log_online_bitmap_file_t*	bitmap_file)	/*!<out: opened bitmap
+							file */
+{
+	ibool	success	= FALSE;
+
+	xb_ad(name[0] != '\0');
+
+	ut_snprintf(bitmap_file->name, FN_REFLEN, "%s%s", srv_data_home, name);
+	bitmap_file->file
+		= xb_file_create_no_error_handling(bitmap_file->name,
+						   OS_FILE_OPEN,
+						   OS_FILE_READ_ONLY,
+						   &success);
+	if (UNIV_UNLIKELY(!success)) {
+
+		/* Here and below assume that bitmap file names do not
+		contain apostrophes, thus no need for ut_print_filename(). */
+		msg("InnoDB: Warning: error opening the changed page "
+		    "bitmap \'%s\'\n", bitmap_file->name);
+		return FALSE;
+	}
+
+	bitmap_file->size = os_file_get_size(bitmap_file->file);
+	bitmap_file->offset = 0;
+
+#ifdef UNIV_LINUX
+	posix_fadvise(bitmap_file->file, 0, 0, POSIX_FADV_SEQUENTIAL);
+	posix_fadvise(bitmap_file->file, 0, 0, POSIX_FADV_NOREUSE);
+#endif
+
+	return TRUE;
+}
+
+/****************************************************************//**
+Diagnose one or both of the following situations if we read close to
+the end of bitmap file:
+1) Warn if the remainder of the file is less than one page.
+2) Error if we cannot read any more full pages but the last read page
+did not have the last-in-run flag set.
+
+@return FALSE for the error */
+static
+ibool
+log_online_diagnose_bitmap_eof(
+/*===========================*/
+	const log_online_bitmap_file_t*	bitmap_file,	/*!< in: bitmap file */
+	ibool				last_page_in_run)/*!< in: "last page in
+							run" flag value in the
+							last read page */
+{
+	/* Check if we are too close to EOF to read a full page */
+	if ((bitmap_file->size < MODIFIED_PAGE_BLOCK_SIZE)
+	    || (bitmap_file->offset
+		> bitmap_file->size - MODIFIED_PAGE_BLOCK_SIZE)) {
+
+		if (UNIV_UNLIKELY(bitmap_file->offset != bitmap_file->size)) {
+
+			/* If we are not at EOF and we have less than one page
+			to read, it's junk.  This error is not fatal in
+			itself. */
+
+			msg("InnoDB: Warning: junk at the end of changed "
+			    "page bitmap file \'%s\'.\n", bitmap_file->name);
+		}
+
+		if (UNIV_UNLIKELY(!last_page_in_run)) {
+
+			/* We are at EOF but the last read page did not finish
+			a run */
+			/* It's a "Warning" here because it's not a fatal error
+			for the whole server */
+			msg("InnoDB: Warning: changed page bitmap "
+			    "file \'%s\' does not contain a complete run "
+			    "at the end.\n", bitmap_file->name);
+			return FALSE;
+		}
+	}
+	return TRUE;
+}
+
+/* End of copy-pasted definitions */
+
+/** Iterator structure over changed page bitmap */
+struct xb_page_bitmap_range_struct {
+	const xb_page_bitmap	*bitmap;	/* Bitmap with data */
+	ulint			space_id;	/* Space id for this
+					        iterator */
+	ulint			bit_i;		/* Bit index of the iterator
+						position in the current page */
+	const ib_rbt_node_t	*bitmap_node;	/* Current bitmap tree node */
+	const byte		*bitmap_page;	/* Current bitmap page */
+	ulint			current_page_id;/* Current page id */
+};
+
+/****************************************************************//**
+Print a diagnostic message on missing bitmap data for an LSN range.  */
+static __attribute__((cold))
+void
+xb_msg_missing_lsn_data(
+/*====================*/
+	lsn_t	missing_interval_start,	/*!<in: interval start */
+	lsn_t	missing_interval_end)	/*!<in: interval end */
+{
+	msg("xtrabackup: warning: changed page data missing for LSNs between "
+	    LSN_PF " and " LSN_PF "\n", missing_interval_start,
+	    missing_interval_end);
+}
+
+/****************************************************************//**
+Scan a bitmap file until data for a desired LSN or EOF is found and check that
+the page before the starting one is not corrupted to ensure that the found page
+indeed contains the very start of the desired LSN data.  The caller must check
+the page LSN values to determine if the bitmap file was scanned until the data
+was found or until EOF.  Page must be at least MODIFIED_PAGE_BLOCK_SIZE big.
+
+@return TRUE if the scan successful without corruption detected
+*/
+static
+ibool
+xb_find_lsn_in_bitmap_file(
+/*=======================*/
+	log_online_bitmap_file_t	*bitmap_file,	/*!<in/out: bitmap
+							file */
+	byte				*page,		/*!<in/out: last read
+							bitmap page */
+	lsn_t				*page_end_lsn,	/*!<out: end LSN of the
+							last read page */
+	lsn_t				lsn)		/*!<in: LSN to find */
+{
+	ibool		last_page_ok		= TRUE;
+	ibool		next_to_last_page_ok	= TRUE;
+
+	xb_ad (bitmap_file->size >= MODIFIED_PAGE_BLOCK_SIZE);
+
+	*page_end_lsn = 0;
+
+	while ((*page_end_lsn <= lsn)
+	       && (bitmap_file->offset
+		   <= bitmap_file->size - MODIFIED_PAGE_BLOCK_SIZE)) {
+
+		next_to_last_page_ok = last_page_ok;
+		if (!log_online_read_bitmap_page(bitmap_file, page,
+						 &last_page_ok)) {
+
+			return FALSE;
+		}
+
+		*page_end_lsn = MACH_READ_64(page + MODIFIED_PAGE_END_LSN);
+	}
+
+	/* We check two pages here because the last read page already contains
+	the required LSN data. If the next to the last one page is corrupted,
+	then we have no way of telling if that page contained the required LSN
+	range data too */
+	return last_page_ok && next_to_last_page_ok;
+}
+
+/****************************************************************//**
+Read the disk bitmap and build the changed page bitmap tree for the
+LSN interval incremental_lsn to checkpoint_lsn_start.
+
+@return the built bitmap tree or NULL if unable to read the full interval for
+any reason. */
+xb_page_bitmap*
+xb_page_bitmap_init(void)
+/*=====================*/
+{
+	log_online_bitmap_file_t	bitmap_file;
+	lsn_t				bmp_start_lsn	= incremental_lsn;
+	lsn_t				bmp_end_lsn	= checkpoint_lsn_start;
+	byte				page[MODIFIED_PAGE_BLOCK_SIZE];
+	lsn_t				current_page_end_lsn;
+	xb_page_bitmap			*result;
+	ibool				last_page_in_run= FALSE;
+	log_online_bitmap_file_range_t	bitmap_files;
+	size_t				bmp_i;
+	ibool				last_page_ok	= TRUE;
+
+	if (UNIV_UNLIKELY(bmp_start_lsn > bmp_end_lsn)) {
+
+		msg("xtrabackup: incremental backup LSN " LSN_PF
+		    " is larger than than the last checkpoint LSN " LSN_PF
+		    "\n", bmp_start_lsn, bmp_end_lsn);
+		return NULL;
+	}
+
+	if (!log_online_setup_bitmap_file_range(&bitmap_files, bmp_start_lsn,
+						bmp_end_lsn)) {
+
+		return NULL;
+	}
+
+	/* Only accept no bitmap files returned if start LSN == end LSN */
+	if (bitmap_files.count == 0 && bmp_end_lsn != bmp_start_lsn) {
+
+		return NULL;
+	}
+
+	result = rbt_create(MODIFIED_PAGE_BLOCK_SIZE,
+			    log_online_compare_bmp_keys);
+
+	if (bmp_start_lsn == bmp_end_lsn) {
+
+		/* Empty range - empty bitmap */
+		return result;
+	}
+
+	bmp_i = 0;
+
+	if (UNIV_UNLIKELY(bitmap_files.files[bmp_i].start_lsn
+			  > bmp_start_lsn)) {
+
+		/* The 1st file does not have the starting LSN data */
+		xb_msg_missing_lsn_data(bmp_start_lsn,
+					bitmap_files.files[bmp_i].start_lsn);
+		rbt_free(result);
+		free(bitmap_files.files);
+		return NULL;
+	}
+
+	/* Skip any zero-sized files at the start */
+	while ((bmp_i < bitmap_files.count - 1)
+	       && (bitmap_files.files[bmp_i].start_lsn
+		   == bitmap_files.files[bmp_i + 1].start_lsn)) {
+
+		bmp_i++;
+	}
+
+	/* Is the 1st bitmap file missing? */
+	if (UNIV_UNLIKELY(bitmap_files.files[bmp_i].name[0] == '\0')) {
+
+		/* TODO: this is not the exact missing range */
+		xb_msg_missing_lsn_data(bmp_start_lsn, bmp_end_lsn);
+		rbt_free(result);
+		free(bitmap_files.files);
+		return NULL;
+	}
+
+	/* Open the 1st bitmap file */
+	if (UNIV_UNLIKELY(!log_online_open_bitmap_file_read_only(
+				  bitmap_files.files[bmp_i].name,
+				  &bitmap_file))) {
+
+		rbt_free(result);
+		free(bitmap_files.files);
+		return NULL;
+	}
+
+	/* If the 1st file is truncated, no data.  Not merged with the case
+	below because zero-length file indicates not a corruption but missing
+	subsequent files instead.  */
+	if (UNIV_UNLIKELY(bitmap_file.size < MODIFIED_PAGE_BLOCK_SIZE)) {
+
+		xb_msg_missing_lsn_data(bmp_start_lsn, bmp_end_lsn);
+		rbt_free(result);
+		free(bitmap_files.files);
+		os_file_close(bitmap_file.file);
+		return NULL;
+	}
+
+	/* Find the start of the required LSN range in the file */
+	if (UNIV_UNLIKELY(!xb_find_lsn_in_bitmap_file(&bitmap_file, page,
+						      &current_page_end_lsn,
+						      bmp_start_lsn))) {
+
+		msg("xtrabackup: Warning: changed page bitmap file "
+		    "\'%s\' corrupted\n", bitmap_file.name);
+		rbt_free(result);
+		free(bitmap_files.files);
+		os_file_close(bitmap_file.file);
+		return NULL;
+	}
+
+	last_page_in_run
+		= mach_read_from_4(page + MODIFIED_PAGE_IS_LAST_BLOCK);
+
+	if (UNIV_UNLIKELY(!log_online_diagnose_bitmap_eof(&bitmap_file,
+							  last_page_in_run))) {
+
+		rbt_free(result);
+		free(bitmap_files.files);
+		os_file_close(bitmap_file.file);
+		return NULL;
+	}
+
+	if (UNIV_UNLIKELY(current_page_end_lsn < bmp_start_lsn)) {
+
+		xb_msg_missing_lsn_data(current_page_end_lsn, bmp_start_lsn);
+		rbt_free(result);
+		free(bitmap_files.files);
+		os_file_close(bitmap_file.file);
+		return NULL;
+	}
+
+	/* 1st bitmap page found, add it to the tree.  */
+	rbt_insert(result, page, page);
+
+	/* Read next pages/files until all required data is read */
+	while (last_page_ok
+	       && (current_page_end_lsn < bmp_end_lsn
+		   || (current_page_end_lsn == bmp_end_lsn
+		       && !last_page_in_run))) {
+
+		ib_rbt_bound_t	tree_search_pos;
+
+		/* If EOF, advance the file skipping over any empty files */
+		while (bitmap_file.size < MODIFIED_PAGE_BLOCK_SIZE
+		       || (bitmap_file.offset
+			   > bitmap_file.size - MODIFIED_PAGE_BLOCK_SIZE)) {
+
+			os_file_close(bitmap_file.file);
+
+			if (UNIV_UNLIKELY(
+				!log_online_diagnose_bitmap_eof(
+					&bitmap_file, last_page_in_run))) {
+
+				rbt_free(result);
+				free(bitmap_files.files);
+				return NULL;
+			}
+
+			bmp_i++;
+
+			if (UNIV_UNLIKELY(bmp_i == bitmap_files.count
+					  || (bitmap_files.files[bmp_i].seq_num
+					      == 0))) {
+
+				xb_msg_missing_lsn_data(current_page_end_lsn,
+							bmp_end_lsn);
+				rbt_free(result);
+				free(bitmap_files.files);
+				return NULL;
+			}
+
+			/* Is the next file missing? */
+			if (UNIV_UNLIKELY(bitmap_files.files[bmp_i].name[0]
+					  == '\0')) {
+
+				/* TODO: this is not the exact missing range */
+				xb_msg_missing_lsn_data(bitmap_files.files
+							[bmp_i - 1].start_lsn,
+							bmp_end_lsn);
+				rbt_free(result);
+				free(bitmap_files.files);
+				return NULL;
+			}
+
+			if (UNIV_UNLIKELY(
+				    !log_online_open_bitmap_file_read_only(
+					    bitmap_files.files[bmp_i].name,
+					    &bitmap_file))) {
+
+				rbt_free(result);
+				free(bitmap_files.files);
+				return NULL;
+			}
+		}
+
+		if (UNIV_UNLIKELY(
+			    !log_online_read_bitmap_page(&bitmap_file, page,
+							 &last_page_ok))) {
+
+			rbt_free(result);
+			free(bitmap_files.files);
+			os_file_close(bitmap_file.file);
+			return NULL;
+		}
+
+		if (UNIV_UNLIKELY(!last_page_ok)) {
+
+			msg("xtrabackup: warning: changed page bitmap file "
+			    "\'%s\' corrupted.\n", bitmap_file.name);
+			rbt_free(result);
+			free(bitmap_files.files);
+			os_file_close(bitmap_file.file);
+			return NULL;
+		}
+
+		/* Merge the current page with an existing page or insert a new
+		page into the tree */
+
+		if (!rbt_search(result, &tree_search_pos, page)) {
+
+			/* Merge the bitmap pages */
+			byte	*existing_page
+				= rbt_value(byte, tree_search_pos.last);
+			bitmap_word_t *bmp_word_1	= (bitmap_word_t *)
+				(existing_page + MODIFIED_PAGE_BLOCK_BITMAP);
+			bitmap_word_t *bmp_end = (bitmap_word_t *)
+				(existing_page + MODIFIED_PAGE_BLOCK_UNUSED_2);
+			bitmap_word_t *bmp_word_2	= (bitmap_word_t *)
+				(page + MODIFIED_PAGE_BLOCK_BITMAP);
+			while (bmp_word_1 < bmp_end) {
+
+				*bmp_word_1++ |= *bmp_word_2++;
+			}
+			xb_a (bmp_word_1 == bmp_end);
+		} else {
+
+			/* Add a new page */
+			rbt_add_node(result, &tree_search_pos, page);
+		}
+
+		current_page_end_lsn
+			= MACH_READ_64(page + MODIFIED_PAGE_END_LSN);
+		last_page_in_run
+			= mach_read_from_4(page + MODIFIED_PAGE_IS_LAST_BLOCK);
+	}
+
+	xb_a (current_page_end_lsn >= bmp_end_lsn);
+
+	free(bitmap_files.files);
+	os_file_close(bitmap_file.file);
+
+	return result;
+}
+
+/****************************************************************//**
+Free the bitmap tree. */
+void
+xb_page_bitmap_deinit(
+/*==================*/
+	xb_page_bitmap*	bitmap)	/*!<in/out: bitmap tree */
+{
+	if (bitmap) {
+
+		rbt_free(bitmap);
+	}
+}
+
+/****************************************************************//**
+Advance to the next bitmap page or setup the first bitmap page for the
+given bitmap range.  Assumes that bitmap_range->bitmap_page has been
+already found/bumped by rbt_search()/rbt_next().
+
+@return FALSE if no more bitmap data for the range space ID */
+static
+ibool
+xb_page_bitmap_setup_next_page(
+/*===========================*/
+	xb_page_bitmap_range*	bitmap_range)	/*!<in/out: the bitmap range */
+{
+	ulint	new_space_id;
+	ulint	new_1st_page_id;
+
+	if (bitmap_range->bitmap_node == NULL) {
+
+		bitmap_range->current_page_id = ULINT_UNDEFINED;
+		return FALSE;
+	}
+
+	bitmap_range->bitmap_page = rbt_value(byte, bitmap_range->bitmap_node);
+
+	new_space_id = mach_read_from_4(bitmap_range->bitmap_page
+					+ MODIFIED_PAGE_SPACE_ID);
+	if (new_space_id != bitmap_range->space_id) {
+
+		/* No more data for the current page id. */
+		xb_a(new_space_id > bitmap_range->space_id);
+		bitmap_range->current_page_id = ULINT_UNDEFINED;
+		return FALSE;
+	}
+
+	new_1st_page_id = mach_read_from_4(bitmap_range->bitmap_page +
+					   MODIFIED_PAGE_1ST_PAGE_ID);
+	xb_a (new_1st_page_id >= bitmap_range->current_page_id
+	      || bitmap_range->current_page_id == ULINT_UNDEFINED);
+
+	bitmap_range->current_page_id = new_1st_page_id;
+	bitmap_range->bit_i = 0;
+
+	return TRUE;
+}
+
+/****************************************************************//**
+Set up a new bitmap range iterator over a given space id changed
+pages in a given bitmap.
+
+@return bitmap range iterator */
+xb_page_bitmap_range*
+xb_page_bitmap_range_init(
+/*======================*/
+	xb_page_bitmap*	bitmap,		/*!< in: bitmap to iterate over */
+	ulint		space_id)	/*!< in: space id */
+{
+	byte			search_page[MODIFIED_PAGE_BLOCK_SIZE];
+	xb_page_bitmap_range	*result
+		= static_cast<xb_page_bitmap_range *>
+		(ut_malloc(sizeof(*result)));
+
+	memset(result, 0, sizeof(*result));
+	result->bitmap = bitmap;
+	result->space_id = space_id;
+	result->current_page_id = ULINT_UNDEFINED;
+
+	/* Search for the 1st page for the given space id */
+	/* This also sets MODIFIED_PAGE_1ST_PAGE_ID to 0, which is what we
+	want. */
+	memset(search_page, 0, MODIFIED_PAGE_BLOCK_SIZE);
+	mach_write_to_4(search_page + MODIFIED_PAGE_SPACE_ID, space_id);
+
+	result->bitmap_node = rbt_lower_bound(result->bitmap, search_page);
+
+	xb_page_bitmap_setup_next_page(result);
+
+	return result;
+}
+
+/****************************************************************//**
+Get the value of the bitmap->range->bit_i bitmap bit
+
+@return the current bit value */
+static inline
+ibool
+is_bit_set(
+/*=======*/
+	const xb_page_bitmap_range*	bitmap_range)	/*!< in: bitmap
+							range */
+{
+	return ((*(((bitmap_word_t *)(bitmap_range->bitmap_page
+				     + MODIFIED_PAGE_BLOCK_BITMAP))
+		  + (bitmap_range->bit_i >> 6)))
+		& (1ULL << (bitmap_range->bit_i & 0x3F))) ? TRUE : FALSE;
+}
+
+/****************************************************************//**
+Get the next page id that has its bit set or cleared, i.e. equal to
+bit_value.
+
+@return page id */
+ulint
+xb_page_bitmap_range_get_next_bit(
+/*==============================*/
+	xb_page_bitmap_range*	bitmap_range,	/*!< in/out: bitmap range */
+	ibool			bit_value)	/*!< in: bit value */
+{
+	if (UNIV_UNLIKELY(bitmap_range->current_page_id
+			  == ULINT_UNDEFINED)) {
+
+		return ULINT_UNDEFINED;
+	}
+
+	do {
+		while (bitmap_range->bit_i < MODIFIED_PAGE_BLOCK_ID_COUNT) {
+
+			while (is_bit_set(bitmap_range) != bit_value
+			       && (bitmap_range->bit_i
+				   < MODIFIED_PAGE_BLOCK_ID_COUNT)) {
+
+				bitmap_range->current_page_id++;
+				bitmap_range->bit_i++;
+			}
+
+			if (bitmap_range->bit_i
+			    < MODIFIED_PAGE_BLOCK_ID_COUNT) {
+
+				ulint result = bitmap_range->current_page_id;
+				bitmap_range->current_page_id++;
+				bitmap_range->bit_i++;
+				return result;
+			}
+		}
+
+		bitmap_range->bitmap_node
+			= rbt_next(bitmap_range->bitmap,
+				   bitmap_range->bitmap_node);
+
+	} while (xb_page_bitmap_setup_next_page(bitmap_range));
+
+	return ULINT_UNDEFINED;
+}
+
+/****************************************************************//**
+Free the bitmap range iterator. */
+void
+xb_page_bitmap_range_deinit(
+/*========================*/
+	xb_page_bitmap_range*	bitmap_range)	/*! in/out: bitmap range */
+{
+	ut_free(bitmap_range);
+}

=== added file 'src/changed_page_bitmap.h'
--- src/changed_page_bitmap.h	1970-01-01 00:00:00 +0000
+++ src/changed_page_bitmap.h	2013-04-15 15:08:38 +0000
@@ -0,0 +1,84 @@
+/******************************************************
+XtraBackup: hot backup tool for InnoDB
+(c) 2009-2012 Percona Inc.
+Originally Created 3/3/2009 Yasufumi Kinoshita
+Written by Alexey Kopytov, Aleksandr Kuzminsky, Stewart Smith, Vadim Tkachenko,
+Yasufumi Kinoshita, Ignacio Nin and Baron Schwartz.
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*******************************************************/
+
+/* Changed page bitmap interface */
+
+#ifndef XB_CHANGED_PAGE_BITMAP_H
+#define XB_CHANGED_PAGE_BITMAP_H
+
+#include "innodb_int.h"
+
+/* The changed page bitmap structure */
+typedef ib_rbt_t xb_page_bitmap;
+
+struct xb_page_bitmap_range_struct;
+
+/* The bitmap range iterator over one space id */
+typedef struct xb_page_bitmap_range_struct xb_page_bitmap_range;
+
+/****************************************************************//**
+Read the disk bitmap and build the changed page bitmap tree for the
+LSN interval incremental_lsn to checkpoint_lsn_start.
+
+@return the built bitmap tree */
+xb_page_bitmap*
+xb_page_bitmap_init(void);
+/*=====================*/
+
+/****************************************************************//**
+Free the bitmap tree. */
+void
+xb_page_bitmap_deinit(
+/*==================*/
+	xb_page_bitmap*	bitmap);	/*!<in/out: bitmap tree */
+
+
+/****************************************************************//**
+Set up a new bitmap range iterator over a given space id changed
+pages in a given bitmap.
+
+@return bitmap range iterator */
+xb_page_bitmap_range*
+xb_page_bitmap_range_init(
+/*======================*/
+	xb_page_bitmap*	bitmap,		/*!< in: bitmap to iterate over */
+	ulint		space_id);	/*!< in: space id */
+
+/****************************************************************//**
+Get the next page id that has its bit set or cleared, i.e. equal to
+bit_value.
+
+@return page id */
+ulint
+xb_page_bitmap_range_get_next_bit(
+/*==============================*/
+	xb_page_bitmap_range*	bitmap_range,	/*!< in/out: bitmap range */
+	ibool			bit_value);	/*!< in: bit value */
+
+/****************************************************************//**
+Free the bitmap range iterator. */
+void
+xb_page_bitmap_range_deinit(
+/*========================*/
+	xb_page_bitmap_range*	bitmap_range);	/*! in/out: bitmap range */
+
+#endif

=== modified file 'src/compact.cc'
--- src/compact.cc	2013-03-22 09:14:31 +0000
+++ src/compact.cc	2013-04-15 15:08:38 +0000
@@ -443,7 +443,7 @@
 
 	ds_set_pipe(ds_buffer, ds_local);
 
-	res = xb_fil_cur_open(&cursor, node, 1);
+	res = xb_fil_cur_open(&cursor, &rf_pass_through, node, 1);
 	xb_a(res == XB_FIL_CUR_SUCCESS);
 
 	snprintf(tmpfile_path, sizeof(tmpfile_path), "%s%s",

=== modified file 'src/fil_cur.cc'
--- src/fil_cur.cc	2013-03-22 09:14:31 +0000
+++ src/fil_cur.cc	2013-04-15 15:08:38 +0000
@@ -26,13 +26,14 @@
 #include "innodb_int.h"
 #include "fil_cur.h"
 #include "common.h"
+#include "read_filt.h"
 #include "xtrabackup.h"
 
 /* Size of read buffer in pages */
 #define XB_FIL_CUR_PAGES 64
 
 /************************************************************************
-Open a source file cursor.
+Open a source file cursor and initialize the associated read filter.
 
 @return XB_FIL_CUR_SUCCESS on success, XB_FIL_CUR_SKIP if the source file must
 be skipped and XB_FIL_CUR_ERROR on error. */
@@ -40,6 +41,7 @@
 xb_fil_cur_open(
 /*============*/
 	xb_fil_cur_t*	cursor,		/*!< out: source file cursor */
+	xb_read_filt_t*	read_filter,	/*!< in/out: the read filter */
 	fil_node_t*	node,		/*!< in: source tablespace node */
 	uint		thread_n)	/*!< thread number for diagnostics */
 {
@@ -131,7 +133,6 @@
 	cursor->buf = static_cast<byte *>
 		(ut_align(cursor->orig_buf, UNIV_PAGE_SIZE));
 
-	cursor->offset = 0;
 	cursor->buf_read = 0;
 	cursor->buf_npages = 0;
 	cursor->buf_offset = 0;
@@ -140,6 +141,10 @@
 
 	cursor->space_size = cursor->statinfo.st_size / page_size;
 
+	cursor->read_filter = read_filter;
+	cursor->read_filter->init(&cursor->read_filter_ctxt, cursor,
+				  node->space->id);
+
 	return(XB_FIL_CUR_SUCCESS);
 }
 
@@ -155,17 +160,16 @@
 	xb_fil_cur_t*	cursor)	/*!< in/out: source file cursor */
 {
 	ibool			success;
-	ulint			page_size;
 	byte*			page;
 	ulint			i;
 	ulint			npages;
 	ulint			retry_count;
 	xb_fil_cur_result_t	ret;
+	ib_int64_t		offset;
 	ib_int64_t		to_read;
 
-	page_size = cursor->page_size;
-
-	to_read = (ib_int64_t) cursor->statinfo.st_size - cursor->offset;
+	cursor->read_filter->get_next_batch(&cursor->read_filter_ctxt,
+					    &offset, &to_read);
 
 	if (to_read == 0LL) {
 		return(XB_FIL_CUR_EOF);
@@ -174,8 +178,8 @@
 	if (to_read > (ib_int64_t) cursor->buf_size) {
 		to_read = (ib_int64_t) cursor->buf_size;
 	}
-	ut_a(to_read > 0 && to_read <= 0xFFFFFFFFLL);
-	ut_a(to_read % page_size == 0);
+	xb_a(to_read > 0 && to_read <= 0xFFFFFFFFLL);
+	xb_a(to_read % cursor->page_size == 0);
 
 	npages = (ulint) (to_read >> cursor->page_size_shift);
 
@@ -187,19 +191,18 @@
 
 	cursor->buf_read = 0;
 	cursor->buf_npages = 0;
-	cursor->buf_offset = cursor->offset;
-	cursor->buf_page_no = (ulint) (cursor->offset >>
-				       cursor->page_size_shift);
+	cursor->buf_offset = offset;
+	cursor->buf_page_no = (ulint) (offset >> cursor->page_size_shift);
 
-	success = xb_os_file_read(cursor->file, cursor->buf, cursor->offset,
-				  to_read);
+	success = xb_os_file_read(cursor->file, cursor->buf, offset, to_read);
 	if (!success) {
 		return(XB_FIL_CUR_ERROR);
 	}
 
 	/* check pages for corruption and re-read if necessary. i.e. in case of
 	partially written pages */
-	for (page = cursor->buf, i = 0; i < npages; page += page_size, i++) {
+	for (page = cursor->buf, i = 0; i < npages;
+	     page += cursor->page_size, i++) {
 		if (xb_buf_page_is_corrupted(page, cursor->zip_size))
 		{
 			ulint page_no = cursor->buf_page_no + i;
@@ -208,7 +211,7 @@
 			    page_no >= FSP_EXTENT_SIZE &&
 			    page_no < FSP_EXTENT_SIZE * 3) {
 				/* skip doublewrite buffer pages */
-				ut_a(page_size == UNIV_PAGE_SIZE);
+				xb_a(cursor->page_size == UNIV_PAGE_SIZE);
 				msg("[%02u] xtrabackup: "
 				    "Page %lu is a doublewrite buffer page, "
 				    "skipping.\n", cursor->thread_n, page_no);
@@ -233,24 +236,25 @@
 				goto read_retry;
 			}
 		}
-		cursor->buf_read += page_size;
+		cursor->buf_read += cursor->page_size;
 		cursor->buf_npages++;
 	}
 
 	posix_fadvise(cursor->file, 0, 0, POSIX_FADV_DONTNEED);
 
-	cursor->offset += page_size * i;
-
 	return(ret);
 }
 
 /************************************************************************
-Close the source file cursor opened with xb_fil_cur_open(). */
+Close the source file cursor opened with xb_fil_cur_open() and its
+associated read filter. */
 void
 xb_fil_cur_close(
 /*=============*/
 	xb_fil_cur_t *cursor)	/*!< in/out: source file cursor */
 {
+	cursor->read_filter->deinit(&cursor->read_filter_ctxt);
+
 	if (cursor->orig_buf != NULL) {
 		ut_free(cursor->orig_buf);
 	}

=== modified file 'src/fil_cur.h'
--- src/fil_cur.h	2013-03-22 09:14:31 +0000
+++ src/fil_cur.h	2013-04-15 15:08:38 +0000
@@ -27,8 +27,9 @@
 
 #include <my_dir.h>
 #include "innodb_int.h"
+#include "read_filt.h"
 
-typedef struct {
+struct xb_fil_cur_t {
 	os_file_t	file;		/*!< source file handle */
 	char		path[FN_REFLEN];/*!< normalized file path */
 	MY_STAT		statinfo;	/*!< information about the file */
@@ -38,9 +39,11 @@
 					UNIV_PAGE_SIZE for uncompressed ones */
 	ulint		page_size_shift;/*!< bit shift corresponding to
 					page_size */
-	ib_int64_t	offset;		/*!< current file offset in bytes */
 	my_bool		is_system;	/*!< TRUE for system tablespace, FALSE
 					otherwise */
+	xb_read_filt_t*	read_filter;	/*!< read filter */
+	xb_read_filt_ctxt_t	read_filter_ctxt;
+					/*!< read filter context */
 	byte*		orig_buf;	/*!< read buffer */
 	byte*		buf;		/*!< aligned pointer for orig_buf */
 	ulint		buf_size;	/*!< buffer size in bytes */
@@ -55,7 +58,7 @@
 	uint		thread_n;	/*!< thread number for diagnostics */
 	ulint		space_id;	/*!< ID of tablespace */
 	ulint		space_size;	/*!< space size in pages */
-} xb_fil_cur_t;
+};
 
 typedef enum {
 	XB_FIL_CUR_SUCCESS,
@@ -65,7 +68,7 @@
 } xb_fil_cur_result_t;
 
 /************************************************************************
-Open a source file cursor.
+Open a source file cursor and initialize the associated read filter.
 
 @return XB_FIL_CUR_SUCCESS on success, XB_FIL_CUR_SKIP if the source file must
 be skipped and XB_FIL_CUR_ERROR on error. */
@@ -73,6 +76,7 @@
 xb_fil_cur_open(
 /*============*/
 	xb_fil_cur_t*	cursor,		/*!< out: source file cursor */
+	xb_read_filt_t*	read_filter,	/*!< in/out: the read filter */
 	fil_node_t*	node,		/*!< in: source tablespace node */
 	uint		thread_n);	/*!< thread number for diagnostics */
 
@@ -88,7 +92,8 @@
 	xb_fil_cur_t*	cursor);	/*!< in/out: source file cursor */
 
 /************************************************************************
-Close the source file cursor opened with xb_fil_cur_open(). */
+Close the source file cursor opened with xb_fil_cur_open() and its
+associated read filter. */
 void
 xb_fil_cur_close(
 /*=============*/

=== modified file 'src/innodb_int.cc'
--- src/innodb_int.cc	2013-03-22 09:14:31 +0000
+++ src/innodb_int.cc	2013-04-15 15:08:38 +0000
@@ -199,7 +199,7 @@
 	const char*	name)	/*!< in: space name */
 {
 	fil_space_t*	space;
-	ulint		fold;
+	ulint		fold __attribute__((unused));
 
 	ut_ad(mutex_own(&fil_system->mutex));
 

=== modified file 'src/innodb_int.h'
--- src/innodb_int.h	2013-03-22 09:14:31 +0000
+++ src/innodb_int.h	2013-04-15 15:08:38 +0000
@@ -85,6 +85,7 @@
 #include <ut0crc32.h>
 #endif
 #include <ut0mem.h>
+#include <ut0rbt.h>
 
 #if MYSQL_VERSION_ID < 50600
 }
@@ -166,6 +167,7 @@
    typedef ib_uint64_t		lsn_t;
    typedef merge_index_def_t	index_def_t;
    typedef merge_index_field_t	index_field_t;
+   typedef struct ib_rbt_struct ib_rbt_t;
 #  define xb_btr_pcur_open_at_index_side(from_left, index, latch_mode, pcur, \
 				       init_pcur, level, mtr)		\
 	btr_pcur_open_at_index_side(from_left, index, latch_mode, pcur,	\

=== added file 'src/read_filt.cc'
--- src/read_filt.cc	1970-01-01 00:00:00 +0000
+++ src/read_filt.cc	2013-04-15 15:08:38 +0000
@@ -0,0 +1,206 @@
+/******************************************************
+XtraBackup: hot backup tool for InnoDB
+(c) 2009-2012 Percona Inc.
+Originally Created 3/3/2009 Yasufumi Kinoshita
+Written by Alexey Kopytov, Aleksandr Kuzminsky, Stewart Smith, Vadim Tkachenko,
+Yasufumi Kinoshita, Ignacio Nin and Baron Schwartz.
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*******************************************************/
+
+/* Data file read filter implementation */
+
+#include "read_filt.h"
+#include "common.h"
+#include "fil_cur.h"
+#include "xtrabackup.h"
+
+/****************************************************************//**
+Perform read filter context initialization that is common to all read
+filters.  */
+static
+void
+common_init(
+/*========*/
+	xb_read_filt_ctxt_t*	ctxt,	/*!<in/out: read filter context */
+	const xb_fil_cur_t*	cursor)	/*!<in: file cursor */
+{
+	ctxt->offset = 0;
+	ctxt->data_file_size = cursor->statinfo.st_size;
+	ctxt->buffer_capacity = cursor->buf_size;
+	ctxt->page_size = cursor->page_size;
+}
+
+/****************************************************************//**
+Initialize the pass-through read filter. */
+static
+void
+rf_pass_through_init(
+/*=================*/
+	xb_read_filt_ctxt_t*	ctxt,	/*!<in/out: read filter context */
+	const xb_fil_cur_t*	cursor,	/*!<in: file cursor */
+	ulint			space_id __attribute__((unused)))
+					/*!<in: space id we are reading */
+{
+	common_init(ctxt, cursor);
+}
+
+/****************************************************************//**
+Get the next batch of pages for the pass-through read filter.  */
+static
+void
+rf_pass_through_get_next_batch(
+/*===========================*/
+	xb_read_filt_ctxt_t*	ctxt,			/*!<in/out: read filter
+							context */
+	ib_int64_t*		read_batch_start,	/*!<out: starting read
+							offset in bytes for the
+							next batch of pages */
+	ib_int64_t*		read_batch_len)		/*!<out: length in
+							bytes of the next batch
+							of pages */
+{
+	*read_batch_start = ctxt->offset;
+	*read_batch_len = ctxt->data_file_size - ctxt->offset;
+
+	if (*read_batch_len > ctxt->buffer_capacity) {
+		*read_batch_len = ctxt->buffer_capacity;
+	}
+
+	ctxt->offset += *read_batch_len;
+}
+
+/****************************************************************//**
+Deinitialize the pass-through read filter.  */
+static
+void
+rf_pass_through_deinit(
+/*===================*/
+	xb_read_filt_ctxt_t*	ctxt __attribute__((unused)))
+					/*!<in: read filter context */
+{
+}
+
+/****************************************************************//**
+Initialize the changed page bitmap-based read filter.  Assumes that
+the bitmap is already set up in changed_page_bitmap.  */
+static
+void
+rf_bitmap_init(
+/*===========*/
+	xb_read_filt_ctxt_t*	ctxt,		/*!<in/out: read filter
+						context */
+	const xb_fil_cur_t*	cursor,		/*!<in: read cursor */
+	ulint			space_id)	/*!<in: space id  */
+{
+	common_init(ctxt, cursor);
+	ctxt->bitmap_range = xb_page_bitmap_range_init(changed_page_bitmap,
+						       space_id);
+	ctxt->filter_batch_end = 0;
+}
+
+/****************************************************************//**
+Get the next batch of pages for the bitmap read filter.  */
+static
+void
+rf_bitmap_get_next_batch(
+/*=====================*/
+	xb_read_filt_ctxt_t*	ctxt,			/*!<in/out: read filter
+							context */
+	ib_int64_t*		read_batch_start,	/*!<out: starting read
+							offset in bytes for the
+							next batch of pages */
+	ib_int64_t*		read_batch_len)		/*!<out: length in
+							bytes of the next batch
+							of pages */
+{
+	ulint	start_page_id;
+
+	start_page_id = ctxt->offset / ctxt->page_size;
+
+	xb_a (ctxt->offset % ctxt->page_size == 0);
+
+	if (start_page_id == ctxt->filter_batch_end) {
+
+		/* Used up all the previous bitmap range, get some more */
+		ulint next_page_id;
+
+		/* Find the next changed page using the bitmap */
+		next_page_id = xb_page_bitmap_range_get_next_bit
+			(ctxt->bitmap_range, TRUE);
+
+		if (next_page_id == ULINT_UNDEFINED) {
+			*read_batch_len = 0;
+			return;
+		}
+
+		ctxt->offset = next_page_id * ctxt->page_size;
+
+		/* Find the end of the current changed page block by searching
+		for the next cleared bitmap bit */
+		ctxt->filter_batch_end
+			= xb_page_bitmap_range_get_next_bit(ctxt->bitmap_range,
+							    FALSE);
+		xb_a(next_page_id < ctxt->filter_batch_end);
+	}
+
+	*read_batch_start = ctxt->offset;
+	if (ctxt->filter_batch_end == ULINT_UNDEFINED) {
+		/* No more cleared bits in the bitmap, need to copy all the
+		remaining pages.  */
+		*read_batch_len = ctxt->data_file_size - ctxt->offset;
+	} else {
+		*read_batch_len = ctxt->filter_batch_end * ctxt->page_size
+			- ctxt->offset;
+	}
+
+	/* If the page block is larger than the buffer capacity, limit it to
+	buffer capacity.  The subsequent invocations will continue returning
+	the current block in buffer-sized pieces until ctxt->filter_batch_end
+	is reached, trigerring the next bitmap query.  */
+	if (*read_batch_len > ctxt->buffer_capacity) {
+		*read_batch_len = ctxt->buffer_capacity;
+	}
+
+	ctxt->offset += *read_batch_len;
+	xb_a (ctxt->offset % ctxt->page_size == 0);
+	xb_a (*read_batch_start % ctxt->page_size == 0);
+	xb_a (*read_batch_len % ctxt->page_size == 0);
+}
+
+/****************************************************************//**
+Deinitialize the changed page bitmap-based read filter.  */
+static
+void
+rf_bitmap_deinit(
+/*=============*/
+	xb_read_filt_ctxt_t*	ctxt)	/*!<in/out: read filter context */
+{
+	xb_page_bitmap_range_deinit(ctxt->bitmap_range);
+}
+
+/* The pass-through read filter */
+xb_read_filt_t rf_pass_through = {
+	&rf_pass_through_init,
+	&rf_pass_through_get_next_batch,
+	&rf_pass_through_deinit
+};
+
+/* The changed page bitmap-based read filter */
+xb_read_filt_t rf_bitmap = {
+	&rf_bitmap_init,
+	&rf_bitmap_get_next_batch,
+	&rf_bitmap_deinit
+};

=== added file 'src/read_filt.h'
--- src/read_filt.h	1970-01-01 00:00:00 +0000
+++ src/read_filt.h	2013-04-15 15:08:38 +0000
@@ -0,0 +1,63 @@
+/******************************************************
+XtraBackup: hot backup tool for InnoDB
+(c) 2009-2012 Percona Inc.
+Originally Created 3/3/2009 Yasufumi Kinoshita
+Written by Alexey Kopytov, Aleksandr Kuzminsky, Stewart Smith, Vadim Tkachenko,
+Yasufumi Kinoshita, Ignacio Nin and Baron Schwartz.
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; version 2 of the License.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+
+*******************************************************/
+
+/* Data file read filter interface */
+
+#ifndef XB_READ_FILT_H
+#define XB_READ_FILT_H
+
+#include "innodb_int.h"
+#include "changed_page_bitmap.h"
+
+struct xb_fil_cur_t;
+
+/* The read filter context */
+struct xb_read_filt_ctxt_t {
+	ib_int64_t		offset;		/*!< current file offset */
+	ib_int64_t		data_file_size;	/*!< data file size */
+	ib_int64_t		buffer_capacity;/*!< read buffer capacity */
+	ulint			space_id;	/*!< space id */
+	/* The following fields used only in bitmap filter */
+	/* Move these to union if any other filters are added in future */
+	xb_page_bitmap_range	*bitmap_range;	/*!< changed page bitmap range
+						iterator for space_id */
+	ulint			page_size;	/*!< page size */
+	ulint			filter_batch_end;/*!< the ending page id of the
+						 current changed page block in
+						 the bitmap */
+};
+
+/* The read filter */
+struct xb_read_filt_t {
+	void (*init)(xb_read_filt_ctxt_t* ctxt,
+		     const xb_fil_cur_t* cursor,
+		     ulint space_id);
+	void (*get_next_batch)(xb_read_filt_ctxt_t* ctxt,
+			       ib_int64_t* read_batch_start,
+			       ib_int64_t* read_batch_len);
+	void (*deinit)(xb_read_filt_ctxt_t* ctxt);
+};
+
+extern xb_read_filt_t rf_pass_through;
+extern xb_read_filt_t rf_bitmap;
+
+#endif

=== modified file 'src/xtrabackup.cc'
--- src/xtrabackup.cc	2013-04-09 07:24:21 +0000
+++ src/xtrabackup.cc	2013-04-15 15:08:38 +0000
@@ -72,6 +72,13 @@
 #include "ds_buffer.h"
 #include "ds_tmpfile.h"
 #include "xbstream.h"
+#include "changed_page_bitmap.h"
+#include "read_filt.h"
+
+/* === File name constants === */
+#define XB_FN_SUSPENDED_AT_START "xtrabackup_suspended_1"
+#define XB_FN_SUSPENDED_AT_END "xtrabackup_suspended_2"
+#define XB_FN_LOG_COPIED "xtrabackup_log_copied"
 
 my_bool innodb_inited= 0;
 
@@ -87,6 +94,7 @@
 my_bool xtrabackup_export = FALSE;
 my_bool xtrabackup_apply_log_only = FALSE;
 
+static my_bool	xtrabackup_suspend_at_start = FALSE;
 my_bool xtrabackup_suspend_at_end = FALSE;
 longlong xtrabackup_use_memory = 100*1024*1024L;
 my_bool xtrabackup_create_ib_logfile = FALSE;
@@ -99,6 +107,7 @@
 lsn_t incremental_lsn;
 lsn_t incremental_to_lsn;
 lsn_t incremental_last_lsn;
+xb_page_bitmap *changed_page_bitmap = NULL;
 
 char *xtrabackup_incremental_basedir = NULL; /* for --backup */
 char *xtrabackup_extra_lsndir = NULL; /* for --backup with --extra-lsndir */
@@ -256,6 +265,8 @@
 static my_bool xtrabackup_compact = FALSE;
 static my_bool xtrabackup_rebuild_indexes = FALSE;
 
+static my_bool xtrabackup_incremental_force_scan = FALSE;
+
 /* Datasinks */
 ds_ctxt_t       *ds_data     = NULL;
 ds_ctxt_t       *ds_meta     = NULL;
@@ -356,6 +367,7 @@
   OPT_XTRA_EXPORT,
   OPT_XTRA_APPLY_LOG_ONLY,
   OPT_XTRA_PRINT_PARAM,
+  OPT_XTRA_SUSPEND_AT_START,
   OPT_XTRA_SUSPEND_AT_END,
   OPT_XTRA_USE_MEMORY,
   OPT_XTRA_THROTTLE,
@@ -430,6 +442,7 @@
   OPT_INNODB_CHECKSUM_ALGORITHM,
   OPT_UNDO_TABLESPACES,
 #endif
+  OPT_XTRA_INCREMENTAL_FORCE_SCAN,
   OPT_DEFAULTS_GROUP
 };
 
@@ -485,9 +498,21 @@
    (G_PTR*) &xtrabackup_use_memory, (G_PTR*) &xtrabackup_use_memory,
    0, GET_LL, REQUIRED_ARG, 100*1024*1024L, 1024*1024L, LONGLONG_MAX, 0,
    1024*1024L, 0},
-  {"suspend-at-end", OPT_XTRA_SUSPEND_AT_END, "creates a file 'xtrabackup_suspended' and waits until the user deletes that file at the end of '--backup'",
+
+  {"suspend-at-start", OPT_XTRA_SUSPEND_AT_START,
+   "creates a file '" XB_FN_SUSPENDED_AT_START "' and waits until the user "
+   "deletes that file after the background log copying thread is started "
+   "during backup",
+   (G_PTR*) &xtrabackup_suspend_at_start,
+   (G_PTR*) &xtrabackup_suspend_at_start, 0, GET_BOOL, NO_ARG,
+   0, 0, 0, 0, 0, 0},
+
+  {"suspend-at-end", OPT_XTRA_SUSPEND_AT_END, "creates a file '"
+   XB_FN_SUSPENDED_AT_END "' and waits until the user deletes that file at "
+   "the end of '--backup'",
    (G_PTR*) &xtrabackup_suspend_at_end, (G_PTR*) &xtrabackup_suspend_at_end,
    0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0},
+
   {"throttle", OPT_XTRA_THROTTLE, "limit count of IO operations (pairs of read&write) per second to IOS values (for '--backup')",
    (G_PTR*) &xtrabackup_throttle, (G_PTR*) &xtrabackup_throttle,
    0, GET_LONG, REQUIRED_ARG, 0, 0, LONG_MAX, 0, 1, 0},
@@ -792,6 +817,13 @@
    0, GET_ULONG, REQUIRED_ARG, 0, 0, 126, 0, 1, 0},
 #endif
 
+  {"incremental-force-scan", OPT_XTRA_INCREMENTAL_FORCE_SCAN,
+   "Perform a full-scan incremental backup even in the presence of changed "
+   "page bitmap data",
+   (G_PTR*)&xtrabackup_incremental_force_scan,
+   (G_PTR*)&xtrabackup_incremental_force_scan, 0, GET_BOOL, NO_ARG,
+   0, 0, 0, 0, 0, 0},
+
   {"defaults_group", OPT_DEFAULTS_GROUP, "defaults group in config file (default \"mysqld\").",
    (G_PTR*) &defaults_group, (G_PTR*) &defaults_group,
    0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0},
@@ -1747,6 +1779,7 @@
 	xb_write_filt_t		*write_filter = NULL;
 	xb_write_filt_ctxt_t	 write_filt_ctxt;
 	const char		*action;
+	xb_read_filt_t		*read_filter;
 
 	if (!trx_sys_sys_space(node->space->id) && /* don't skip system space */
 	    check_if_skip_table(node->name, "ibd")) {
@@ -1754,7 +1787,13 @@
 		return(FALSE);
 	}
 
-	res = xb_fil_cur_open(&cursor, node, thread_n);
+	if (!changed_page_bitmap) {
+		read_filter = &rf_pass_through;
+	}
+	else {
+		read_filter = &rf_bitmap;
+	}
+	res = xb_fil_cur_open(&cursor, read_filter, node, thread_n);
 	if (res == XB_FIL_CUR_SKIP) {
 		goto skip;
 	} else if (res == XB_FIL_CUR_ERROR) {
@@ -2302,7 +2341,9 @@
 		ds_data = ds;
 
 		if (xtrabackup_stream_fmt != XB_STREAM_FMT_XBSTREAM ||
-		     xtrabackup_suspend_at_end) {
+		    xtrabackup_suspend_at_end ||
+		    xtrabackup_suspend_at_start) {
+
 			/* 'xbstream' allow parallel streams, but we
 			still can't stream directly to stdout when
 			xtrabackup is invoked from innobackupex
@@ -2481,20 +2522,34 @@
 }
 
 /***********************************************************************
+Prepare a sync file path. */
+static void
+xb_make_sync_file_name(
+/*===================*/
+	const char*	name,	/*!<in: sync file name */
+	char*		path)	/*!<in/out: path to the sync file */
+{
+	snprintf(path, FN_REFLEN, "%s/%s", xtrabackup_target_dir, name);
+	srv_normalize_path_for_win(path);
+}
+
+/***********************************************************************
 Create an empty file with a given path and close it.
 @return TRUE on succees, FALSE on error. */
 static ibool
-xb_create_suspend_file(const char *path)
+xb_create_sync_file(
+/*================*/
+	const char*	path)	/*!<in: path to the sync file */
 {
-	ibool			success;
-	os_file_t		suspend_file = XB_FILE_UNDEFINED;
+	ibool		success;
+	os_file_t	suspend_file = XB_FILE_UNDEFINED;
 
 	/* xb_file_create reads srv_unix_file_flush_method */
 	suspend_file = xb_file_create(path, OS_FILE_CREATE,
 				      OS_FILE_NORMAL, OS_DATA_FILE,
 				      &success);
 
-	if (success && suspend_file != XB_FILE_UNDEFINED) {
+	if (UNIV_LIKELY(success && suspend_file != XB_FILE_UNDEFINED)) {
 
 		os_file_close(suspend_file);
 
@@ -2641,6 +2696,33 @@
 	}
 }
 
+/***********************************************************************
+Create a specified sync file and wait until it's removed.  */
+static void
+xtrabackup_suspend(
+/*===============*/
+	const char*	sync_fn)	/*!<in: sync file name */
+{
+	char		suspend_path[FN_REFLEN];
+	ibool		success = TRUE;
+	ibool		exists = TRUE;
+	os_file_type_t	type;
+
+	xb_make_sync_file_name(sync_fn, suspend_path);
+
+	if (!xb_create_sync_file(suspend_path)) {
+		return;
+	}
+
+	while (exists && success) {
+		os_thread_sleep(200000); /*0.2 sec*/
+		success = os_file_status(suspend_path, &exists, &type);
+		/* success == FALSE if file exists, but stat() failed.
+		os_file_status() prints an error message in this case */
+		ut_a(success);
+	}
+}
+
 static void
 xtrabackup_backup_func(void)
 {
@@ -2650,8 +2732,6 @@
 	uint			 count;
 	os_ib_mutex_t		 count_mutex;
 	data_thread_ctxt_t 	*data_threads;
-	char			suspend_path[FN_REFLEN];
-	ibool			success;
 
 #ifdef USE_POSIX_FADVISE
 	msg("xtrabackup: uses posix_fadvise().\n");
@@ -2939,6 +3019,25 @@
 
 	os_thread_create(log_copying_thread, NULL, &log_copying_thread_id);
 
+	/* Suspend at start, for the FLUSH CHANGED_PAGE_BITMAPS call */
+	if (xtrabackup_suspend_at_start) {
+		xtrabackup_suspend(XB_FN_SUSPENDED_AT_START);
+	}
+
+	if (xtrabackup_incremental) {
+		if (!xtrabackup_incremental_force_scan) {
+			changed_page_bitmap = xb_page_bitmap_init();
+		}
+		if (!changed_page_bitmap) {
+			msg("xtrabackup: using the full scan for incremental "
+			    "backup\n");
+		} else if (incremental_lsn != checkpoint_lsn_start) {
+			/* Do not print that bitmaps are used when dummy bitmap
+			is build for an empty LSN range. */
+			msg("xtrabackup: using the changed page bitmap\n");
+		}
+	}
+
 	ut_a(xtrabackup_parallel > 0);
 
 	if (xtrabackup_parallel > 1) {
@@ -2982,29 +3081,14 @@
 	ut_free(data_threads);
 	datafiles_iter_free(it);
 
+	if (changed_page_bitmap) {
+		xb_page_bitmap_deinit(changed_page_bitmap);
+	}
 	}
 
 	/* suspend-at-end */
 	if (xtrabackup_suspend_at_end) {
-		ibool		exists;
-		os_file_type_t	type;
-
-		sprintf(suspend_path, "%s%s", xtrabackup_target_dir,
-			"/xtrabackup_suspended");
-		srv_normalize_path_for_win(suspend_path);
-
-		if (!xb_create_suspend_file(suspend_path)) {
-			exit(EXIT_FAILURE);
-		}
-
-		exists = TRUE;
-		while (exists) {
-			os_thread_sleep(200000); /*0.2 sec*/
-			success = os_file_status(suspend_path, &exists, &type);
-			/* success == FALSE if file exists, but stat() failed.
-			os_file_status() prints an error message in this case */
-			ut_a(success);
-		}
+		xtrabackup_suspend(XB_FN_SUSPENDED_AT_END);
 	}
 
 	/* read the latest checkpoint lsn */
@@ -3051,9 +3135,12 @@
 	/* Signal innobackupex that log copying has stopped and it may now
 	unlock tables, so we can possibly stream xtrabackup_logfile later
 	without holding the lock. */
-	if (xtrabackup_suspend_at_end &&
-	    !xb_create_suspend_file(suspend_path)) {
-		exit(EXIT_FAILURE);
+	if (xtrabackup_suspend_at_end) {
+		char	log_copied_sync_file[FN_REFLEN];
+		xb_make_sync_file_name(XB_FN_LOG_COPIED, log_copied_sync_file);
+		if (!xb_create_sync_file(log_copied_sync_file)) {
+			exit(EXIT_FAILURE);
+		}
 	}
 
 	if(!xtrabackup_incremental) {

=== modified file 'src/xtrabackup.h'
--- src/xtrabackup.h	2013-03-22 09:14:31 +0000
+++ src/xtrabackup.h	2013-04-15 15:08:38 +0000
@@ -23,6 +23,7 @@
 
 #include "datasink.h"
 #include "innodb_int.h"
+#include "changed_page_bitmap.h"
 
 typedef struct {
 	ulint	page_size;
@@ -46,6 +47,11 @@
 extern ds_ctxt_t	*ds_meta;
 extern ds_ctxt_t	*ds_data;
 
+/* The last checkpoint LSN at the backup startup time */
+extern lsn_t checkpoint_lsn_start;
+
+extern xb_page_bitmap *changed_page_bitmap;
+
 void xtrabackup_io_throttling(void);
 my_bool xb_write_delta_metadata(const char *filename,
 				const xb_delta_info_t *info);

=== modified file 'test/inc/common.sh'
--- test/inc/common.sh	2013-03-22 09:14:31 +0000
+++ test/inc/common.sh	2013-04-15 15:08:38 +0000
@@ -189,6 +189,7 @@
     SRV_MYSQLD_DATADIR[$id]="$vardir/data"
     SRV_MYSQLD_TMPDIR[$id]="$vardir/tmp"
     SRV_MYSQLD_PIDFILE[$id]="${TEST_BASEDIR}/mysqld${id}.pid"
+    SRV_MYSQLD_ERRFILE[$id]="$vardir/data/mysqld${id}.err"
     SRV_MYSQLD_PORT[$id]=`get_free_port $id`
     SRV_MYSQLD_SOCKET[$id]=`mktemp -t xtrabackup.mysql.sock.XXXXXX`
 }
@@ -211,6 +212,7 @@
     SRV_MYSQLD_DATADIR[$id]=
     SRV_MYSQLD_TMPDIR[$id]=
     SRV_MYSQLD_PIDFILE[$id]=
+    SRV_MYSQLD_ERRFILE[$id]=
     SRV_MYSQLD_PORT[$id]=
     SRV_MYSQLD_SOCKET[$id]=
 }
@@ -227,11 +229,13 @@
     MYSQLD_DATADIR="${SRV_MYSQLD_DATADIR[$id]}"
     MYSQLD_TMPDIR="${SRV_MYSQLD_TMPDIR[$id]}"
     MYSQLD_PIDFILE="${SRV_MYSQLD_PIDFILE[$id]}"
+    MYSQLD_ERRFILE="${SRV_MYSQLD_ERRFILE[$id]}"
     MYSQLD_PORT="${SRV_MYSQLD_PORT[$id]}"
     MYSQLD_SOCKET="${SRV_MYSQLD_SOCKET[$id]}"
 
     MYSQL_ARGS="--no-defaults --socket=${MYSQLD_SOCKET} --user=root"
     MYSQLD_ARGS="--no-defaults --basedir=${MYSQL_BASEDIR} \
+--log-error=${MYSQLD_ERRFILE}
 --socket=${MYSQLD_SOCKET} --port=${MYSQLD_PORT} --server-id=$id \
 --datadir=${MYSQLD_DATADIR} --tmpdir=${MYSQLD_TMPDIR} --log-bin=mysql-bin \
 --relay-log=mysql-relay-bin --pid-file=${MYSQLD_PIDFILE} ${MYSQLD_EXTRA_ARGS}"
@@ -455,5 +459,62 @@
     return ${PIPESTATUS[0]}
 }
 
+readonly xb_performed_bmp_inc_backup="xtrabackup: using the full scan for incremental backup"
+readonly xb_performed_full_scan_inc_backup="xtrabackup: using the changed page bitmap"
+
+####################################################
+# Helper functions for testing incremental backups #
+####################################################
+function check_full_scan_inc_backup()
+{
+    if ! grep -q "$xb_performed_bmp_inc_backup" $OUTFILE ;
+    then
+        vlog "xtrabackup did not perform a full scan for the incremental backup."
+        exit -1
+    fi
+    if grep -q "$xb_performed_full_scan_inc_backup" $OUTFILE ;
+    then
+        vlog "xtrabackup appeared to use bitmaps instead of full scan for the incremental backup."
+        exit -1
+    fi
+}
+
+function check_bitmap_inc_backup()
+{
+    if ! grep -q "$xb_performed_full_scan_inc_backup" $OUTFILE ;
+    then
+        vlog "xtrabackup did not use bitmaps for the incremental backup."
+        exit -1
+    fi
+    if grep -q "$xb_performed_bmp_inc_backup" $OUTFILE ;
+    then
+        vlog "xtrabackup used a full scan instead of bitmaps for the incremental backup."
+        exit -1
+    fi
+}
+
+##############################################################
+# Helper functions for xtrabackup process suspend and resume #
+##############################################################
+function wait_for_xb_to_suspend()
+{
+    local file=$1
+    local i=0
+    echo "Waiting for $file to be created"
+    while [ ! -r $file ]
+    do
+        sleep 1
+        i=$((i+1))
+        echo "Waited $i seconds for xtrabackup_suspended to be created"
+    done
+}
+
+function resume_suspended_xb()
+{
+    local file=$1
+    echo "Removing $file"
+    rm -f $file
+}
+
 # To avoid unbound variable error when no server have been started
 SRV_MYSQLD_IDS=

=== renamed file 'test/t/ib_incremental.sh' => 'test/inc/ib_incremental_common.sh'
--- test/t/ib_incremental.sh	2012-10-15 16:14:59 +0000
+++ test/inc/ib_incremental_common.sh	2013-04-15 15:08:38 +0000
@@ -1,6 +1,14 @@
+# Expects the following variable to be set before including:
+#    mysqld_extra_args: an extra arg to be passed for all mysqld invocations.  
+#                       Use this to set special options that influence 
+#                       incremental backups, e.g. turns on log archiving or 
+#                       changed page bitmap output.
+
 . inc/common.sh
 
-start_server --innodb_file_per_table
+mysqld_extra_args="$mysqld_extra_args --innodb_file_per_table"
+
+start_server $mysqld_extra_args
 load_dbase_schema incremental_sample
 
 # Adding initial rows
@@ -35,6 +43,18 @@
 	${MYSQL} ${MYSQL_ARGS} -e "insert into t2 values ($count, $numrow);" incremental_sample
 	let "count=count+1"
 done
+
+# Rotate bitmap file here and force checkpoint at the same time
+shutdown_server
+start_server $mysqld_extra_args
+
+i=1001
+while [ "$i" -lt "7500" ]
+do
+        ${MYSQL} ${MYSQL_ARGS} -e "insert into t2 values ($i, repeat(\"ab\", 32500));" incremental_sample
+        let "i=i+1"
+done
+
 vlog "Changes done"
 
 # Saving the checksum of original table
@@ -87,7 +107,7 @@
 innobackupex --copy-back $full_backup_dir
 vlog "Data restored"
 
-start_server --innodb_file_per_table
+start_server $mysqld_extra_args
 
 vlog "Checking checksums"
 checksum_test_b=`checksum_table incremental_sample test`

=== modified file 'test/t/bug1007446.sh'
--- test/t/bug1007446.sh	2012-12-26 23:48:57 +0000
+++ test/t/bug1007446.sh	2013-04-15 15:08:38 +0000
@@ -6,7 +6,7 @@
 
 start_server
 
-echo "" > $MYSQLD_TMPDIR/xtrabackup_suspended
+echo "" > $MYSQLD_TMPDIR/xtrabackup_suspended_1
 
 mkdir -p $topdir/backup
 innobackupex --stream=tar $topdir/backup > /dev/null

=== added file 'test/t/ib_incremental_bitmap.sh'
--- test/t/ib_incremental_bitmap.sh	1970-01-01 00:00:00 +0000
+++ test/t/ib_incremental_bitmap.sh	2013-04-15 15:08:38 +0000
@@ -0,0 +1,25 @@
+# Test for incremental backups that use changed page bitmaps
+
+if [ -z "$XTRADB_VERSION" ]; then
+    echo "Requires XtraDB" > $SKIPPED_REASON
+    exit $SKIPPED_EXIT_CODE
+fi
+
+# The test is disabled for Percona XtraDB Cluster until a version
+# with bitmap user requests is released (containing PS 5.5.29-30.0 
+# or later)
+set +e
+${MYSQLD} --basedir=$MYSQL_BASEDIR --user=$USER --help --verbose --wsrep-sst-method=rsync| grep -q wsrep
+probe_result=$?
+if [[ "$probe_result" == "0" ]]
+    then
+        vlog "Incompatible test" > $SKIPPED_REASON
+        exit $SKIPPED_EXIT_CODE
+fi
+set -e
+
+mysqld_extra_args=--innodb-track-changed-pages=TRUE
+
+. inc/ib_incremental_common.sh
+
+check_bitmap_inc_backup

=== added file 'test/t/ib_incremental_full_scan.sh'
--- test/t/ib_incremental_full_scan.sh	1970-01-01 00:00:00 +0000
+++ test/t/ib_incremental_full_scan.sh	2013-04-15 15:08:38 +0000
@@ -0,0 +1,7 @@
+# Test incremental backups that do full data file scans
+
+mysqld_extra_args=
+
+. inc/ib_incremental_common.sh
+
+check_full_scan_inc_backup

=== added file 'test/t/xb_incremental_bitmap_misc.sh'
--- test/t/xb_incremental_bitmap_misc.sh	1970-01-01 00:00:00 +0000
+++ test/t/xb_incremental_bitmap_misc.sh	2013-04-15 15:08:38 +0000
@@ -0,0 +1,76 @@
+# Test diagnostics for missing bitmap data and --incremental-force-scan option
+
+if [ -z "$XTRADB_VERSION" ]; then
+    echo "Requires XtraDB" > $SKIPPED_REASON
+    exit $SKIPPED_EXIT_CODE
+fi
+
+# The test is disabled for Percona XtraDB Cluster until a version
+# with bitmap user requests is released (containing PS 5.5.29-30.0 
+# or later)
+set +e
+${MYSQLD} --basedir=$MYSQL_BASEDIR --user=$USER --help --verbose --wsrep-sst-method=rsync| grep -q wsrep
+probe_result=$?
+if [[ "$probe_result" == "0" ]]
+    then
+        vlog "Incompatible test" > $SKIPPED_REASON
+        exit $SKIPPED_EXIT_CODE
+fi
+set -e
+
+. inc/common.sh
+
+mysqld_extra_args=--innodb-track-changed-pages=TRUE
+
+start_server $mysqld_extra_args
+${MYSQL} ${MYSQL_ARGS} -e "CREATE TABLE t1 (a INT) ENGINE=INNODB" test
+${MYSQL} ${MYSQL_ARGS} -e "INSERT INTO t1 VALUES (1)" test
+
+# Force checkpoint
+shutdown_server
+start_server $mysqld_extra_args
+
+# Full backup
+# mkdir $topdir/data/full
+# xtrabackup --datadir=$mysql_datadir --backup --target-dir=$topdir/data/inc1 --
+
+${MYSQL} ${MYSQL_ARGS} -e "INSERT INTO t1 VALUES (2)" test
+# Force checkpoint
+shutdown_server
+start_server $mysqld_extra_args
+${MYSQL} ${MYSQL_ARGS} -e "INSERT INTO t1 VALUES (3)" test
+# Force checkpoint
+shutdown_server
+start_server $mysqld_extra_args
+${MYSQL} ${MYSQL_ARGS} -e "INSERT INTO t1 VALUES (4)" test
+# Force checkpoint
+shutdown_server
+start_server $mysqld_extra_args
+
+ls -l $mysql_datadir/*.xdb
+
+vlog "Test --incremental-force-scan option"
+xtrabackup --datadir=$mysql_datadir --backup \
+   --target-dir=$topdir/data/inc0 --incremental_lsn=9000 \
+   --incremental-force-scan
+
+vlog "Test nonsensical LSN range (start LSN > end LSN)"
+xtrabackup --datadir=$mysql_datadir --backup \
+   --target-dir=$topdir/data/inc1 --incremental-lsn=500000000000
+
+vlog "Test missing bitmap data diagnostics: middle of range"
+run_cmd rm $mysql_datadir/ib_modified_log_2*.xdb
+xtrabackup --datadir=$mysql_datadir --backup \
+    --target-dir=$topdir/data/inc2 --incremental-lsn=9000
+
+vlog "Test missing bitmap data diagnostics: end of range too"
+run_cmd rm $mysql_datadir/ib_modified_log_3*.xdb
+xtrabackup --datadir=$mysql_datadir --backup \
+    --target-dir=$topdir/data/inc3 --incremental-lsn=9000
+
+vlog "Test missing bitmap data diagnostics: start of range too"
+run_cmd rm $mysql_datadir/ib_modified_log_1*.xdb
+xtrabackup --datadir=$mysql_datadir --backup \
+    --target-dir=$topdir/data/inc4 --incremental-lsn=9000
+
+check_full_scan_inc_backup

=== modified file 'test/t/xb_incremental_compressed.inc'
--- test/t/xb_incremental_compressed.inc	2012-11-08 03:44:14 +0000
+++ test/t/xb_incremental_compressed.inc	2013-04-15 15:08:38 +0000
@@ -1,13 +1,33 @@
 ################################################################################
 # Test incremental backup with InnoDB compression
 ################################################################################
+# Expects the following variable to be set before including:
+#    mysqld_extra_args: an extra arg to be passed for all mysqld invocations.  
+#                       Use this to set special options that influence 
+#                       incremental backups, e.g. turns on log archiving or 
+#                       changed page bitmap output.
+#    first_inc_suspend_command: if non-empty string, passes --suspend-at-start 
+#                               to incremental XB invocation and executes the 
+#                               given command while it's suspended
 
 . inc/common.sh
 
-if [ -z "$INNODB_VERSION" ]; then
-    echo "Requires InnoDB plugin or XtraDB" >$SKIPPED_REASON
-    exit $SKIPPED_EXIT_CODE
-fi
+function add_rows()
+{
+  local table=$1
+  local start=$2
+  local limit=$3
+
+  while [ "$limit" -gt "$start" ]; do
+    sql="INSERT INTO $table VALUES ($start, $limit)"
+    let "start=start+1"
+    for ((i=0; $i<99; i++)); do
+      sql="${sql},($start, $limit)"
+      let "start=start+1"
+    done
+    ${MYSQL} ${MYSQL_ARGS} -e "$sql" incremental_sample
+  done
+}
 
 #
 # Test incremental backup of a compressed tablespace with a specific page size
@@ -22,9 +42,9 @@
 
   # Use innodb_strict_mode so that failure to use compression results in an 
   # error rather than a warning
-  mysqld_additional_args="--innodb_strict_mode --innodb_file_per_table \
-      --innodb_file_format=Barracuda --innodb_max_dirty_pages_pct=0 \
-      --innodb_log_file_size=1M"
+  mysqld_additional_args="$mysqld_extra_args --innodb_strict_mode \
+      --innodb_file_per_table --innodb_file_format=Barracuda \
+      --innodb_max_dirty_pages_pct=0 --innodb_log_file_size=1M"
   
   start_server ${mysqld_additional_args}
 
@@ -40,17 +60,7 @@
 
   vlog "Adding initial rows to database..."
 
-  numrow=10000
-  count=0
-  while [ "$numrow" -gt "$count" ]; do
-    sql="INSERT INTO test VALUES ($count, $numrow)"
-    let "count=count+1"
-    for ((i=0; $i<99; i++)); do
-      sql="${sql},($count, $numrow)"
-      let "count=count+1"
-    done
-    ${MYSQL} ${MYSQL_ARGS} -e "$sql" incremental_sample
-  done
+  add_rows test 0 10000
 
   rows=`${MYSQL} ${MYSQL_ARGS} -Ns -e "SELECT COUNT(*) FROM test" \
 incremental_sample`
@@ -84,18 +94,15 @@
 number INT(11) DEFAULT NULL) ENGINE=INNODB \
 ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=$page_size" incremental_sample
 
-  let "count=numrow+1"
-  let "numrow=15000"
-  while [ "$numrow" -gt "$count" ]; do
-    sql="VALUES ($count, $numrow)"
-    let "count=count+1"
-    for ((i=0; $i<99; i++)); do
-      sql="${sql},($count, $numrow)"
-      let "count=count+1"
-    done
-    ${MYSQL} ${MYSQL_ARGS} -e "INSERT INTO test $sql" incremental_sample
-    ${MYSQL} ${MYSQL_ARGS} -e "INSERT INTO t2 $sql" incremental_sample
-  done
+  add_rows test 10001 12500
+  add_rows t2 10001 12500
+
+  # Rotate bitmap file here and force checkpoint at the same time
+  shutdown_server
+  start_server ${mysqld_additional_args}
+
+  add_rows test 12501 15000
+  add_rows t2 12501 15000
 
   rows=`${MYSQL} ${MYSQL_ARGS} -Ns -e "SELECT COUNT(*) FROM test" \
 incremental_sample`
@@ -122,10 +129,29 @@
 
   vlog "Making incremental backup"
 
+  if [ -n "$first_inc_suspend_command" ]
+  then
+      suspend_arg="--suspend_at_start";
+  else
+      suspend_arg="";
+  fi
+ 
   # Incremental backup
   xtrabackup --datadir=$mysql_datadir --backup \
       --target-dir=$topdir/data/delta --incremental-basedir=$topdir/data/full \
-      ${mysqld_additional_args}
+      ${mysqld_additional_args} $suspend_arg &
+
+  xb_pid=$!
+
+  if [ -n "$first_inc_suspend_command" ]
+  then
+      wait_for_xb_to_suspend $topdir/data/delta/xtrabackup_suspended_1
+      ${MYSQL} ${MYSQL_ARGS} -e "$first_inc_suspend_command"
+
+      resume_suspended_xb $topdir/data/delta/xtrabackup_suspended_1
+  fi
+
+  wait $xb_pid
 
   vlog "Incremental backup done"
   vlog "Preparing backup"
@@ -149,7 +175,8 @@
 
   # Restore backup
 
-  stop_server
+  shutdown_server
+  rm -f $mysql_datadir/ib_modified_log*
 
   vlog "Copying files"
 

=== added file 'test/t/xb_incremental_compressed_bitmap_16kb.sh'
--- test/t/xb_incremental_compressed_bitmap_16kb.sh	1970-01-01 00:00:00 +0000
+++ test/t/xb_incremental_compressed_bitmap_16kb.sh	2013-04-15 15:08:38 +0000
@@ -0,0 +1,28 @@
+# Test incremental backups that use bitmaps with 16KB compressed pages
+
+if [ -z "$XTRADB_VERSION" ]; then
+    echo "Requires XtraDB" > $SKIPPED_REASON
+    exit $SKIPPED_EXIT_CODE
+fi
+
+# The test is disabled for Percona XtraDB Cluster until a version
+# with bitmap user requests is released (containing PS 5.5.29-30.0 
+# or later)
+set +e
+${MYSQLD} --basedir=$MYSQL_BASEDIR --user=$USER --help --verbose --wsrep-sst-method=rsync| grep -q wsrep
+probe_result=$?
+if [[ "$probe_result" == "0" ]]
+    then
+        vlog "Incompatible test" > $SKIPPED_REASON
+        exit $SKIPPED_EXIT_CODE
+fi
+set -e
+
+mysqld_extra_args=--innodb-track-changed-pages=TRUE
+first_inc_suspend_command="FLUSH CHANGED_PAGE_BITMAPS;"
+
+source t/xb_incremental_compressed.inc
+
+test_incremental_compressed 16
+
+check_bitmap_inc_backup

=== added file 'test/t/xb_incremental_compressed_bitmap_1kb.sh'
--- test/t/xb_incremental_compressed_bitmap_1kb.sh	1970-01-01 00:00:00 +0000
+++ test/t/xb_incremental_compressed_bitmap_1kb.sh	2013-04-15 15:08:38 +0000
@@ -0,0 +1,28 @@
+# Test incremental backups that use bitmaps with 1KB compressed pages
+
+if [ -z "$XTRADB_VERSION" ]; then
+    echo "Requires XtraDB" > $SKIPPED_REASON
+    exit $SKIPPED_EXIT_CODE
+fi
+
+# The test is disabled for Percona XtraDB Cluster until a version
+# with bitmap user requests is released (containing PS 5.5.29-30.0 
+# or later)
+set +e
+${MYSQLD} --basedir=$MYSQL_BASEDIR --user=$USER --help --verbose --wsrep-sst-method=rsync| grep -q wsrep
+probe_result=$?
+if [[ "$probe_result" == "0" ]]
+    then
+        vlog "Incompatible test" > $SKIPPED_REASON
+        exit $SKIPPED_EXIT_CODE
+fi
+set -e
+
+mysqld_extra_args=--innodb-track-changed-pages=TRUE
+first_inc_suspend_command="FLUSH CHANGED_PAGE_BITMAPS;"
+
+source t/xb_incremental_compressed.inc
+
+test_incremental_compressed 1
+
+check_bitmap_inc_backup

=== added file 'test/t/xb_incremental_compressed_bitmap_2kb.sh'
--- test/t/xb_incremental_compressed_bitmap_2kb.sh	1970-01-01 00:00:00 +0000
+++ test/t/xb_incremental_compressed_bitmap_2kb.sh	2013-04-15 15:08:38 +0000
@@ -0,0 +1,28 @@
+# Test incremental backups that use bitmaps with 2KB compressed pages
+
+if [ -z "$XTRADB_VERSION" ]; then
+    echo "Requires XtraDB" > $SKIPPED_REASON
+    exit $SKIPPED_EXIT_CODE
+fi
+
+# The test is disabled for Percona XtraDB Cluster until a version
+# with bitmap user requests is released (containing PS 5.5.29-30.0 
+# or later)
+set +e
+${MYSQLD} --basedir=$MYSQL_BASEDIR --user=$USER --help --verbose --wsrep-sst-method=rsync| grep -q wsrep
+probe_result=$?
+if [[ "$probe_result" == "0" ]]
+    then
+        vlog "Incompatible test" > $SKIPPED_REASON
+        exit $SKIPPED_EXIT_CODE
+fi
+set -e
+
+mysqld_extra_args=--innodb-track-changed-pages=TRUE
+first_inc_suspend_command="FLUSH CHANGED_PAGE_BITMAPS;"
+
+source t/xb_incremental_compressed.inc
+
+test_incremental_compressed 2
+
+check_bitmap_inc_backup

=== added file 'test/t/xb_incremental_compressed_bitmap_4kb.sh'
--- test/t/xb_incremental_compressed_bitmap_4kb.sh	1970-01-01 00:00:00 +0000
+++ test/t/xb_incremental_compressed_bitmap_4kb.sh	2013-04-15 15:08:38 +0000
@@ -0,0 +1,28 @@
+# Test incremental backups that use bitmaps with 4KB compressed pages
+
+if [ -z "$XTRADB_VERSION" ]; then
+    echo "Requires XtraDB" > $SKIPPED_REASON
+    exit $SKIPPED_EXIT_CODE
+fi
+
+# The test is disabled for Percona XtraDB Cluster until a version
+# with bitmap user requests is released (containing PS 5.5.29-30.0 
+# or later)
+set +e
+${MYSQLD} --basedir=$MYSQL_BASEDIR --user=$USER --help --verbose --wsrep-sst-method=rsync| grep -q wsrep
+probe_result=$?
+if [[ "$probe_result" == "0" ]]
+    then
+        vlog "Incompatible test" > $SKIPPED_REASON
+        exit $SKIPPED_EXIT_CODE
+fi
+set -e
+
+mysqld_extra_args=--innodb-track-changed-pages=TRUE
+first_inc_suspend_command="FLUSH CHANGED_PAGE_BITMAPS;"
+
+source t/xb_incremental_compressed.inc
+
+test_incremental_compressed 4
+
+check_bitmap_inc_backup

=== added file 'test/t/xb_incremental_compressed_bitmap_8kb.sh'
--- test/t/xb_incremental_compressed_bitmap_8kb.sh	1970-01-01 00:00:00 +0000
+++ test/t/xb_incremental_compressed_bitmap_8kb.sh	2013-04-15 15:08:38 +0000
@@ -0,0 +1,28 @@
+# Test incremental backups that use bitmaps with 8KB compressed pages
+
+if [ -z "$XTRADB_VERSION" ]; then
+    echo "Requires XtraDB" > $SKIPPED_REASON
+    exit $SKIPPED_EXIT_CODE
+fi
+
+# The test is disabled for Percona XtraDB Cluster until a version
+# with bitmap user requests is released (containing PS 5.5.29-30.0 
+# or later)
+set +e
+${MYSQLD} --basedir=$MYSQL_BASEDIR --user=$USER --help --verbose --wsrep-sst-method=rsync| grep -q wsrep
+probe_result=$?
+if [[ "$probe_result" == "0" ]]
+    then
+        vlog "Incompatible test" > $SKIPPED_REASON
+        exit $SKIPPED_EXIT_CODE
+fi
+set -e
+
+mysqld_extra_args=--innodb-track-changed-pages=TRUE
+first_inc_suspend_command="FLUSH CHANGED_PAGE_BITMAPS;"
+
+source t/xb_incremental_compressed.inc
+
+test_incremental_compressed 8
+
+check_bitmap_inc_backup

=== renamed file 'test/t/xb_incremental_compressed_16kb.sh' => 'test/t/xb_incremental_compressed_full_scan_16kb.sh'
--- test/t/xb_incremental_compressed_16kb.sh	2012-11-08 03:44:14 +0000
+++ test/t/xb_incremental_compressed_full_scan_16kb.sh	2013-04-15 15:08:38 +0000
@@ -1,2 +1,10 @@
+# Test incremental backups that do full data scans with 16KB compressed pages
+
+mysqld_extra_args=
+first_inc_suspend_command=
+
 source t/xb_incremental_compressed.inc
+
 test_incremental_compressed 16
+
+check_full_scan_inc_backup

=== renamed file 'test/t/xb_incremental_compressed_1kb.sh' => 'test/t/xb_incremental_compressed_full_scan_1kb.sh'
--- test/t/xb_incremental_compressed_1kb.sh	2012-11-08 03:44:14 +0000
+++ test/t/xb_incremental_compressed_full_scan_1kb.sh	2013-04-15 15:08:38 +0000
@@ -1,2 +1,10 @@
+# Test incremental backups that do full data scans with 1KB compressed pages
+
+mysqld_extra_args=
+first_inc_suspend_command=
+
 source t/xb_incremental_compressed.inc
+
 test_incremental_compressed 1
+
+check_full_scan_inc_backup

=== renamed file 'test/t/xb_incremental_compressed_2kb.sh' => 'test/t/xb_incremental_compressed_full_scan_2kb.sh'
--- test/t/xb_incremental_compressed_2kb.sh	2012-11-08 03:44:14 +0000
+++ test/t/xb_incremental_compressed_full_scan_2kb.sh	2013-04-15 15:08:38 +0000
@@ -1,2 +1,10 @@
+# Test incremental backups that do full data scans with 2KB compressed pages
+
+mysqld_extra_args=
+first_inc_suspend_command=
+
 source t/xb_incremental_compressed.inc
+
 test_incremental_compressed 2
+
+check_full_scan_inc_backup

=== renamed file 'test/t/xb_incremental_compressed_4kb.sh' => 'test/t/xb_incremental_compressed_full_scan_4kb.sh'
--- test/t/xb_incremental_compressed_4kb.sh	2012-11-08 03:44:14 +0000
+++ test/t/xb_incremental_compressed_full_scan_4kb.sh	2013-04-15 15:08:38 +0000
@@ -1,2 +1,10 @@
+# Test incremental backups that do full data scans with 4KB compressed pages
+
+mysqld_extra_args=
+first_inc_suspend_command=
+
 source t/xb_incremental_compressed.inc
+
 test_incremental_compressed 4
+
+check_full_scan_inc_backup

=== renamed file 'test/t/xb_incremental_compressed_8kb.sh' => 'test/t/xb_incremental_compressed_full_scan_8kb.sh'
--- test/t/xb_incremental_compressed_8kb.sh	2012-11-08 03:44:14 +0000
+++ test/t/xb_incremental_compressed_full_scan_8kb.sh	2013-04-15 15:08:38 +0000
@@ -1,2 +1,10 @@
+# Test incremental backups that do full data scans with 8KB compressed pages
+
+mysqld_extra_args=
+first_inc_suspend_command=
+
 source t/xb_incremental_compressed.inc
+
 test_incremental_compressed 8
+
+check_full_scan_inc_backup

=== modified file 'test/t/xb_log_overwrap.sh'
--- test/t/xb_log_overwrap.sh	2013-01-08 10:14:53 +0000
+++ test/t/xb_log_overwrap.sh	2013-04-15 15:08:38 +0000
@@ -20,14 +20,7 @@
 
 pid_file=$topdir/backup/xtrabackup_debug_sync
 
-# Wait for xtrabackup to suspend
-i=0
-while [ ! -r "$pid_file" ]
-do
-    sleep 1
-    i=$((i+1))
-    echo "Waited $i seconds for $pid_file to be created"
-done
+wait_for_xb_to_suspend $pid_file
 
 xb_pid=`cat $pid_file`
 
@@ -41,5 +34,7 @@
 vlog "Resuming xtrabackup"
 kill -SIGCONT $xb_pid
 
+resume_suspended_xb $topdir/backup/xtrabackup_suspended_2
+
 # wait's return code will be the code returned by the background process
 run_cmd wait $job_pid