Merge lp:~vlad-lesin/percona-xtrabackup/2.1-apply-archived-logs-innodb5.6 into lp:percona-xtrabackup/2.1

Proposed by Vlad Lesin on 2013-04-25
Status: Merged
Approved by: Alexey Kopytov on 2013-08-27
Approved revision: 672
Merged at revision: 675
Proposed branch: lp:~vlad-lesin/percona-xtrabackup/2.1-apply-archived-logs-innodb5.6
Merge into: lp:percona-xtrabackup/2.1
Diff against target: 1525 lines (+1383/-17)
3 files modified
patches/innodb56.patch (+822/-0)
src/xtrabackup.cc (+172/-17)
test/t/xb_apply_archived_logs.sh (+389/-0)
To merge this branch: bzr merge lp:~vlad-lesin/percona-xtrabackup/2.1-apply-archived-logs-innodb5.6
Reviewer Review Type Date Requested Status
Alexey Kopytov (community) 2013-04-25 Approve on 2013-08-27
Laurynas Biveinis (community) Needs Fixing on 2013-04-25
Review via email: mp+160804@code.launchpad.net

Description of the change

This is implementation of ability to apply archived logs in xtrabackup which is described here: https://blueprints.launchpad.net/percona-xtrabackup/+spec/log-archiving-restore
See commit comment for more detailed description.

To post a comment you must log in.

A merge conflict, unless spurious, that needs rebasing on the current trunk?

review: Needs Fixing
Vlad Lesin (vlad-lesin) wrote :

> A merge conflict, unless spurious, that needs rebasing on the current trunk?
I have just rebased it on the current trunk.

Alexey Kopytov (akopytov) wrote :
Download full text (3.2 KiB)

Vlad,

    - I don't think the XtraBackup part should pull the entire server
      implementation of log archiving. It looks like only a minor part
      of it should be applied to InnoDB 5.6 code base to be able to
      parse and apply archived logs? As in, do we really need
      fil_space_contains_node(), log_sys->archive_buf initialization,
      changes to log_checkpoint_set_nth_group_info(), etc.?

    - can you clarify the following change? I.e. under what
      circumstances exactly can a tablespace exist when we are trying
      to replay MLOG_FILE_CREATE from the archived logs?

+@@ -3325,9 +3341,13 @@
+ path = fil_make_ibd_name(tablename, false);
+ }
+
++ /* The files can already exist in the case of archived logs applying */
+ file = os_file_create(
+ innodb_file_data_key, path,
+- OS_FILE_CREATE | OS_FILE_ON_ERROR_NO_EXIT,
++ (srv_archive_recovery ?
++ OS_FILE_OVERWRITE :
++ OS_FILE_CREATE) |
++ OS_FILE_ON_ERROR_NO_EXIT,
+ OS_FILE_NORMAL,
+ OS_DATA_FILE,
+ &ret);

    - s/alligned to the up of log file block/aligned up to the log block size/

    - please clarify why the following change is needed. As I
      understand, we don't start the FTS optimize thread in
      XtraBackup. That code is also missing block braces and is using
      spaces instead of tabs for indentation:

+@@ -3032,7 +3032,8 @@
+
+ ib_logf(IB_LOG_LEVEL_INFO, "FTS optimize thread exiting.");
+
+- os_event_set(exit_event);
++ if (exit_event)
++ os_event_set(exit_event);
+
+ /* We count the number of threads in os_thread_exit(). A created
+ thread should always use that to exit and not use return() to exit. */
+

    - please use LSN_PF instead of "%llu"

    - in recv_recovery_from_archive_start() we ignore the first_log_no
      argument and use xtrabackup_arch_first_file_lsn instead. wouldn't
      it be better to pass xtrabackup_arch_first_file_lsn as the
      first_log_no argument to recv_recovery_from_archive_start() and
      get rid of min/max_arch_log_no in
      innobase_start_or_create_for_mysql()?

    - in xb_data_files_init() the patch replaces min/max_flushed_lsn
      declarations with min/max_arch_log_no. Which are not really
      used. but then the call still calls open_or_create_data_files()
      with the (now non-existing) flushed_lsn variables?

    - s/xtrabackup: Note: opening archived log directory %s was
      failed/xtrabackup: error: cannot open archived log directory %s/

    - replace the strstr() call in xtrabackup_arch_search_files() with
      strncmp()

    - xtrabackup_arch_search_files() is declared to return bool, but
      returns xtrabackup_arch_first_file_lsn?

    - we tend to use the InnoDB code style in xtrabackup.cc. In
      particular, the opening brace must be on the same line as the
      'if' statement. And braces should be used even for
      single-statement 'if' blocks;

    - s/trunsactions/transactions/

    - s/undid/rolled back/

    - the check for Percona Server 5.6 doesn't work (try starting it
      with an upstream 5.6 server). The reason is that the syntax for
      string equality comparison is "val1 = val2", not "val1=val2"

    - the test case doesn't wo...

Read more...

review: Needs Fixing

Vlad,

Why don't we use --log-arch-dir option (with same name as server one) instead of --archived-logs-dir?

Vlad Lesin (vlad-lesin) wrote :

On 05/06/2013 06:21 PM, Sergei Glushchenko wrote:
> Vlad,
>
> Why don't we use --log-arch-dir option (with same name as server one) instead of --archived-logs-dir?
>

This is good idea, thanks.

--
Vlad Lesin, Software Engineer, Percona Inc.
<email address hidden>
skype: vlad_lesin
JID: <email address hidden>
ICQ: 204036003
phone: +79051122311
Tula, Russia (GMT +4)

Vlad Lesin (vlad-lesin) wrote :
Download full text (6.8 KiB)

On 05/01/2013 02:40 PM, Alexey Kopytov wrote:
> - I don't think the XtraBackup part should pull the entire server
> implementation of log archiving. It looks like only a minor part
> of it should be applied to InnoDB 5.6 code base to be able to
> parse and apply archived logs? As in, do we really need
> fil_space_contains_node(), log_sys->archive_buf initialization,
> changes to log_checkpoint_set_nth_group_info(), etc.?

Done.

> - can you clarify the following change? I.e. under what
> circumstances exactly can a tablespace exist when we are trying
> to replay MLOG_FILE_CREATE from the archived logs?

Yes. It can be for example in the case when this file did not exist at
the beginning of backup process and was created somewhere in the middle
of backup process. So log file contains "create" command as backup
process started tracking it before the file creation, but data directory
contains this file as it was copied from original directory. I copied
this explanation to code comment.

> +@@ -3325,9 +3341,13 @@
> + path = fil_make_ibd_name(tablename, false);
> + }
> +
> ++ /* The files can already exist in the case of archived logs applying */
> + file = os_file_create(
> + innodb_file_data_key, path,
> +- OS_FILE_CREATE | OS_FILE_ON_ERROR_NO_EXIT,
> ++ (srv_archive_recovery ?
> ++ OS_FILE_OVERWRITE :
> ++ OS_FILE_CREATE) |
> ++ OS_FILE_ON_ERROR_NO_EXIT,
> + OS_FILE_NORMAL,
> + OS_DATA_FILE,
> + &ret);
>
> - s/alligned to the up of log file block/aligned up to the log block size/

Done.

> - please clarify why the following change is needed. As I
> understand, we don't start the FTS optimize thread in
> XtraBackup. That code is also missing block braces and is using
> spaces instead of tabs for indentation:
>
> +@@ -3032,7 +3032,8 @@
> +
> + ib_logf(IB_LOG_LEVEL_INFO, "FTS optimize thread exiting.");
> +
> +- os_event_set(exit_event);
> ++ if (exit_event)
> ++ os_event_set(exit_event);
> +
> + /* We count the number of threads in os_thread_exit(). A created
> + thread should always use that to exit and not use return() to exit. */
> +

I fixed using spaces instead tabs for indentation and missing braces.

As I understuud FTS optimize thread is started in xtrabackup because I
have the following backtrace in the case of missing exit_event condition:
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007ffff633f037 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007ffff6342698 in __GI_abort () at abort.c:90
#2 0x000000000063ac80 in os_event_set (event=0x0)
     at
/home/vlesin/src/work/percona-xtrabackup-2.1-arch-log/mysql-5.6/storage/innobase/os/os0sync.cc:437
#3 0x00000000006063ec in fts_optimize_thread (arg=0x1e604e8)
     at
/home/vlesin/src/work/percona-xtrabackup-2.1-arch-log/mysql-5.6/storage/innobase/fts/fts0opt.cc:3036
#4 0x00007ffff79bcf8e in start_thread (arg=0x7fffd57fa700) at
pthread_create.c:311
#5 0x00007ffff6401e1d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Here is the backtrace of FTS initializati...

Read more...

Vlad Lesin (vlad-lesin) wrote :

On 05/06/2013 06:21 PM, Sergei Glushchenko wrote:
> Vlad,
>
> Why don't we use --log-arch-dir option (with same name as server one) instead of --archived-logs-dir?
>

Done.

--
Vlad Lesin, Software Engineer, Percona Inc.
<email address hidden>
skype: vlad_lesin
JID: <email address hidden>
ICQ: 204036003
phone: +79051122311
Tula, Russia (GMT +4)

Alexey Kopytov (akopytov) wrote :
Download full text (6.3 KiB)

Hi Vlad,

General comments:

     - I think this MP should wait until 5.6.11-60.3 with the fix for bug
       #1172591 is released. Then the test suite should use 5.6.11-60.30
       for testing instead of the custom build

     - there should be a separate fix for bug #1177182 for both 2.0 and
2.1, and again, this branch should be rebased once the fix is merged to
trunks.

     - xtrabackup_arch_search_files() should use msg() instead of
       fprintf(stderr,...)

Other comments/questions:

On Fri, 10 May 2013 09:42:06 +0400, Vlad Lesin wrote:
>> - can you clarify the following change? I.e. under what
>> circumstances exactly can a tablespace exist when we are trying
>> to replay MLOG_FILE_CREATE from the archived logs?
>
> Yes. It can be for example in the case when this file did not exist at
> the beginning of backup process and was created somewhere in the middle
> of backup process. So log file contains "create" command as backup
> process started tracking it before the file creation, but data directory
> contains this file as it was copied from original directory. I copied
> this explanation to code comment.
>

If a tablespace exists on recovery, it is created into fil_system before
the recovery starts. When replaying MLOG_FILE_CREATE,
fil_op_log_parse_or_replay first checks if the tablespace exists in
fil_system and does nothing. So I still don't understand that change in
fil_create_new_single_table_tablespace().

>
>> +@@ -3325,9 +3341,13 @@
>> + path = fil_make_ibd_name(tablename, false);
>> + }
>> +
>> ++ /* The files can already exist in the case of archived logs
>> applying */
>> + file = os_file_create(
>> + innodb_file_data_key, path,
>> +- OS_FILE_CREATE | OS_FILE_ON_ERROR_NO_EXIT,
>> ++ (srv_archive_recovery ?
>> ++ OS_FILE_OVERWRITE :
>> ++ OS_FILE_CREATE) |
>> ++ OS_FILE_ON_ERROR_NO_EXIT,
>> + OS_FILE_NORMAL,
>> + OS_DATA_FILE,
>> + &ret);
>>
>> - s/alligned to the up of log file block/aligned up to the log
>> block size/
>
> Done.
>

Not fully done, it is "aligned" (single "l"), not "alligned".

>> - please clarify why the following change is needed. As I
>> understand, we don't start the FTS optimize thread in
>> XtraBackup. That code is also missing block braces and is using
>> spaces instead of tabs for indentation:
>>
>> +@@ -3032,7 +3032,8 @@
>> +
>> + ib_logf(IB_LOG_LEVEL_INFO, "FTS optimize thread exiting.");
>> +
>> +- os_event_set(exit_event);
>> ++ if (exit_event)
>> ++ os_event_set(exit_event);
>> +
>> + /* We count the number of threads in os_thread_exit(). A created
>> + thread should always use that to exit and not use return() to
>> exit. */
>> +
>
> I fixed using spaces instead tabs for indentation and missing braces.
>
> As I understuud FTS optimize thread is started in xtrabackup because I
>

OK, I see the problem now, reported as bug #1179193. This means we have
to fix that bug separately in both 2.0 and 2.1, and then rebase this
branch on 2.1 trunk with that fix included.

>> - please use LSN_PF instead of "%llu"
>
...

Read more...

Vlad Lesin (vlad-lesin) wrote :

On 05/12/2013 04:48 PM, Alexey Kopytov wrote:
>>> - in recv_recovery_from_archive_start() we ignore the first_log_no
>>> argument and use xtrabackup_arch_first_file_lsn instead. wouldn't
>>> it be better to pass xtrabackup_arch_first_file_lsn as the
>>> first_log_no argument to recv_recovery_from_archive_start() and

As I understood our chat right the main cause of this change is to
remove 'fake' arguments from recv_recovery_from_archive_start().

Why just don't remove first_log_no from
recv_recovery_from_archive_start() arguments list? I don't see any
benefits in passing xtrabackup_arch_first_file_lsn and
xtrabackup_arch_last_file_lsn to recv_recovery_from_archive_start() as
arguments because in this case we just remove using global external
variables from recv_recovery_from_archive_start() to
innobase_start_or_create_for_mysql().

 >>> get rid of min/max_arch_log_no in
 >>> innobase_start_or_create_for_mysql()?

Right. In this case we could remove min/max_arch_log_no from
open_or_create_data_files() arguments list and from xb_data_files_init()
too.

--
Vlad Lesin, Software Engineer, Percona Inc.
<email address hidden>
skype: vlad_lesin
JID: <email address hidden>
ICQ: 204036003
phone: +79051122311
Tula, Russia (GMT +4)

Vlad Lesin (vlad-lesin) wrote :

On 05/12/2013 04:48 PM, Alexey Kopytov wrote:

>
> I tried to repeat the test, but your branch doesn't build for me now, it
> fails as follows:
>
> [ 70%] Building CXX object
> libmysqld/CMakeFiles/sql_embedded.dir/__/sql/field_conv.cc.o
> /Users/kaa/src/launchpad/percona-xtrabackup/2.1-apply-archived-logs-innodb5.6/mysql-5.6/storage/innobase/log/log0recv.cc:3142:3:
> error: no matching function for call to
> 'log_checkpoint_get_nth_group_info'
> log_checkpoint_get_nth_group_info(buf, group->id,
> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> /Users/kaa/src/launchpad/percona-xtrabackup/2.1-apply-archived-logs-innodb5.6/mysql-5.6/storage/innobase/include/log0log.h:286:1:
> note: candidate function not viable: no known
> conversion from 'lsn_t *' (aka 'unsigned long long *') to 'ulint
> *' (aka 'unsigned long *') for 3rd argument;
> log_checkpoint_get_nth_group_info(
> ^
> 1[ 70%] error generated.
>

I am preparing the more detailed answer with code fixes so currently I
don't want to commit unfinished changes. But it is possible just to
comment log_checkpoint_get_nth_group_info() invoking at that place
because the values it reads from buffer are not used for logs applying.

--
Vlad Lesin, Software Engineer, Percona Inc.
<email address hidden>
skype: vlad_lesin
JID: <email address hidden>
ICQ: 204036003
phone: +79051122311
Tula, Russia (GMT +4)

Alexey Kopytov (akopytov) wrote :

Hi Vlad,

On Mon, 13 May 2013 17:44:36 +0400, Vlad Lesin wrote:
> On 05/12/2013 04:48 PM, Alexey Kopytov wrote:
>>>> - in recv_recovery_from_archive_start() we ignore the
>>>> first_log_no
>>>> argument and use xtrabackup_arch_first_file_lsn instead.
>>>> wouldn't
>>>> it be better to pass xtrabackup_arch_first_file_lsn as the
>>>> first_log_no argument to recv_recovery_from_archive_start()
>>>> and
>
> As I understood our chat right the main cause of this change is to
> remove 'fake' arguments from recv_recovery_from_archive_start().
>
> Why just don't remove first_log_no from
> recv_recovery_from_archive_start() arguments list? I don't see any
> benefits in passing xtrabackup_arch_first_file_lsn and
> xtrabackup_arch_last_file_lsn to recv_recovery_from_archive_start() as
> arguments because in this case we just remove using global external
> variables from recv_recovery_from_archive_start() to
> innobase_start_or_create_for_mysql().
>

Either way is fine by me as long as we avoid fake arguments.

> >>> get rid of min/max_arch_log_no in
> >>> innobase_start_or_create_for_mysql()?
>
> Right. In this case we could remove min/max_arch_log_no from
> open_or_create_data_files() arguments list and from xb_data_files_init()
> too.
>

OK.

Vlad Lesin (vlad-lesin) wrote :

> > Why just don't remove first_log_no from
> > recv_recovery_from_archive_start() arguments list? I don't see any
> > benefits in passing xtrabackup_arch_first_file_lsn and
> > xtrabackup_arch_last_file_lsn to recv_recovery_from_archive_start() as
> > arguments because in this case we just remove using global external
> > variables from recv_recovery_from_archive_start() to
> > innobase_start_or_create_for_mysql().
> >
>
> Either way is fine by me as long as we avoid fake arguments.
Done.

> > Right. In this case we could remove min/max_arch_log_no from
> > open_or_create_data_files() arguments list and from xb_data_files_init()
> > too.
> >
>
> OK.
Done.

Vlad Lesin (vlad-lesin) wrote :

> Hi Vlad,
>
>
> General comments:
>
> - I think this MP should wait until 5.6.11-60.3 with the fix for bug
> #1172591 is released. Then the test suite should use 5.6.11-60.30
> for testing instead of the custom build
The fix is released.

> - there should be a separate fix for bug #1177182 for both 2.0 and
> 2.1, and again, this branch should be rebased once the fix is merged to
> trunks.
The fix for 2.1 is merged and for 2.0 is approved for merging.

> - xtrabackup_arch_search_files() should use msg() instead of
> fprintf(stderr,...)
Fixed.

> >> - s/alligned to the up of log file block/aligned up to the log
> >> block size/
> >
> > Done.
> >
>
> Not fully done, it is "aligned" (single "l"), not "alligned".
Done.

> OK, I see the problem now, reported as bug #1179193. This means we have
> to fix that bug separately in both 2.0 and 2.1, and then rebase this
> branch on 2.1 trunk with that fix included.
I explored this problem more deeply and found out that the problem is "apply archived log" specific. See my comment to the bug report and
                if (srv_apply_log_only) {
                        goto skip_processes;
                }
strings in innobase_start_or_create_for_mysql().

> >> - the test case doesn't work for me (fails with "table 'test.t1'
> >> doesn't exist" when run against PS 5.6 with log archiving enabled)
> >>
> >
> > I need your assistance here because I can't reproduce this on my
> > environment and, as I understood, on Jenkins environment too(see
> > http://jenkins.percona.com/view/XtraBackup/job/percona-
> xtrabackup-2.1-param/298/#showFailuresLink).
> >
>
> I tried to repeat the test, but your branch doesn't build for me now, it
> fails as follows:
>
> [ 70%] Building CXX object
> libmysqld/CMakeFiles/sql_embedded.dir/__/sql/field_conv.cc.o
> /Users/kaa/src/launchpad/percona-xtrabackup/2.1-apply-archived-logs-
> innodb5.6/mysql-5.6/storage/innobase/log/log0recv.cc:3142:3:
> error: no matching function for call to
> 'log_checkpoint_get_nth_group_info'
> log_checkpoint_get_nth_group_info(buf, group->id,
> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> /Users/kaa/src/launchpad/percona-xtrabackup/2.1-apply-archived-logs-
> innodb5.6/mysql-5.6/storage/innobase/include/log0log.h:286:1:
> note: candidate function not viable: no known
> conversion from 'lsn_t *' (aka 'unsigned long long *') to 'ulint
> *' (aka 'unsigned long *') for 3rd argument;
> log_checkpoint_get_nth_group_info(
> ^
> 1[ 70%] error generated.
Fixed. The invocation of log_checkpoint_get_nth_group_info() is commented in this place because there is no need to read archived_file_no and archived_offset. They are calculated another way.

Vlad Lesin (vlad-lesin) wrote :

Rebased on newest trunk (lp:percona-xtrabackup revision 650).

Alexey Kopytov (akopytov) wrote :

I don't think XtraDB-related changes in run.sh are necessary now. Check the is_xtradb()/require_xtradb() functions and how they are used in other tests.

Alexey Kopytov (akopytov) wrote :

Oh, and you will also need require_server_version_higher_than()

Vlad Lesin (vlad-lesin) wrote :

On 07/31/2013 04:54 PM, Alexey Kopytov wrote:
> I don't think XtraDB-related changes in run.sh are necessary now. Check the is_xtradb()/require_xtradb() functions and how they are used in other tests.
>
Yes, you are right, there is no need to change run.sh as server and
innodb flavours are detected right now based on "innodb_version" server
variable.

But require_server_version_higher_than() and
require_xtradb()/require_xtradb() does not suite me because
require_server_version_higher_than() checks only server version but I
need to check server flavour too as well as
require_xtradb()/require_xtradb() checks only the fact of xtradb using
but xtradb version must be checked too.

So I reverted run.sh and fixed xb_apply_archived_logs.sh to use
is_server_version_higher_than() for server version checking.

Alexey Kopytov (akopytov) wrote :

Hi Vlad,

On Wed, 31 Jul 2013 21:43:33 -0000, Vlad Lesin wrote:
> But require_server_version_higher_than() and
> require_xtradb()/require_xtradb() does not suite me because
> require_server_version_higher_than() checks only server version but I
> need to check server flavour too as well as
> require_xtradb()/require_xtradb() checks only the fact of xtradb using
> but xtradb version must be checked too.
>

You can do:

require_xtradb
require_server_version_higher_than 5.6.10

Or use is_* functions if you need something complex.

Vlad Lesin (vlad-lesin) wrote :

On 08/01/2013 08:39 AM, Alexey Kopytov wrote:
> Hi Vlad,
>
> On Wed, 31 Jul 2013 21:43:33 -0000, Vlad Lesin wrote:
>> But require_server_version_higher_than() and
>> require_xtradb()/require_xtradb() does not suite me because
>> require_server_version_higher_than() checks only server version but I
>> need to check server flavour too as well as
>> require_xtradb()/require_xtradb() checks only the fact of xtradb using
>> but xtradb version must be checked too.
>>
>
> You can do:
>
> require_xtradb
> require_server_version_higher_than 5.6.10
>
> Or use is_* functions if you need something complex.
>

Yes, but xtradb can be 5.5,5.1 or 5.6. But only 5.6 supports logs applying.

Alexey Kopytov (akopytov) wrote :

Hi Vlad,

On Thu, 01 Aug 2013 11:01:56 +0400, Vlad Lesin wrote:
>> You can do:
>>
>> require_xtradb
>> require_server_version_higher_than 5.6.10
>>
>> Or use is_* functions if you need something complex.
>>
>
> Yes, but xtradb can be 5.5,5.1 or 5.6. But only 5.6 supports logs applying.
>

The above 2 lines will make the test run only on XtraDB > 5.6.10?

Alexey Kopytov (akopytov) wrote :

Vlad,

Do you have any explanations for multiple test failures and crashes in both builds?

review: Needs Information
Vlad Lesin (vlad-lesin) wrote :

> Vlad,
>
> Do you have any explanations for multiple test failures and crashes in both
> builds?

Yes, there were bug concerned with wrong lsn writing to data files on "prepare" step when logs applying is not used. I fixed it, here is the new test results:
http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.1-param/420/

656. By Alexey Kopytov on 2013-08-10

Manual merge from trunk.

657. By Alexey Kopytov on 2013-08-12

Null merge from 2.0.

658. By Alexey Kopytov on 2013-08-14

Merged lp:~akopytov/percona-xtrabackup/bug1210266-2.1.

659. By Alexey Kopytov on 2013-08-14

Empty merge from 2.0.

660. By Alexey Kopytov on 2013-08-16

Merged lp:~gl-az/percona-xtrabackup/bug1211263.

661. By Alexey Kopytov on 2013-08-16

Merged lp:~akopytov/percona-xtrabackup/bug1135441-2.1.

662. By Alexey Kopytov on 2013-08-18

s/Percona Ireland Ltd./Percona LLC and/or its affiliates/

663. By Alexey Kopytov on 2013-08-18

Follow fix for bug #1135441:

Do exit a child process when it's terminated with a fatal signal, even
if we want it to exit silently.

664. By Alexey Kopytov on 2013-08-20

Merged lp:~akopytov/percona-xtrabackup/bug1213036-2.1.

665. By Alexey Kopytov on 2013-08-20

Automerge from 2.0.

666. By Alexey Kopytov on 2013-08-20

Null merge from 2.0.

667. By Alexey Kopytov on 2013-08-20

Merged lp:~akopytov/percona-xtrabackup/bug1213102-2.1.

668. By Alexey Kopytov on 2013-08-20

Manual merge of lp:~akopytov/percona-xtrabackup/bug1194828.

669. By Alexey Kopytov on 2013-08-20

Bug #1204045: Enable xb_incremental_compressed_bitmap* tests for PXC

Enabled back bitmap tests for PXC.

670. By Alexey Kopytov on 2013-08-21

Automerge from 2.0.

671. By Alexey Kopytov on 2013-08-21

Bug #1111380: --rebuild-indexes holds on to disk space during --prepare

Close the file cursor in xb_expand_file() to release the inode for the
original file replaced by the temporary expanded one. Otherwise both
files will be present in the filesystem until the xtrabackup process
finishes.

Alexey Kopytov (akopytov) wrote :

Vlad,

Approved, but please rebase on the current trunk. I get many conflicts in innodb56.patch. The good news is that it's going to be the last rebase for this branch. Sorry about the stretched review process. Please also do another Jenkins param build after rebasing.

Thanks.

review: Approve
672. By Vlad Lesin on 2013-08-26

Apply archived logs.

The following options are used only with --prepare option:
--innodb-arch-log-dir - the path to archived logs directory
--to-archived-lsn - lsn to which logs should be applied

Metadata file contains lsn number to which log file was scanned during backup.
As part of log file that was copied during backup is not applied during
archived logs applying archived directory must contain log files the latest lsn
of whose is greater or equal to scanned lsn otherwise there will be error
message.

If min_flushed_lsn from data files is less than arhived logs start lsn the logs
will not be applied and xtrabackup will exit with error. Otherwise logs
applying will start with min_flushed_lsn.

Vlad Lesin (vlad-lesin) wrote :

> Vlad,
>
> Approved, but please rebase on the current trunk. I get many conflicts in
> innodb56.patch. The good news is that it's going to be the last rebase for
> this branch. Sorry about the stretched review process. Please also do another
> Jenkins param build after rebasing.
>
> Thanks.

Done.

http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.1-param/443/

Alexey Kopytov (akopytov) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'patches/innodb56.patch'
2--- patches/innodb56.patch 2013-08-20 17:43:45 +0000
3+++ patches/innodb56.patch 2013-08-26 07:08:27 +0000
4@@ -1564,3 +1564,825 @@
5
6 buf = group->checkpoint_buf;
7
8+--- a/storage/innobase/fil/fil0fil.cc
9++++ b/storage/innobase/fil/fil0fil.cc
10+@@ -1980,12 +1980,6 @@
11+ contain sensible data */
12+ ulint* flags, /*!< out: tablespace flags */
13+ ulint* space_id, /*!< out: tablespace ID */
14+-#ifdef UNIV_LOG_ARCHIVE
15+- ulint* min_arch_log_no, /*!< out: min of archived
16+- log numbers in data files */
17+- ulint* max_arch_log_no, /*!< out: max of archived
18+- log numbers in data files */
19+-#endif /* UNIV_LOG_ARCHIVE */
20+ lsn_t* min_flushed_lsn, /*!< out: min of flushed
21+ lsn values in data files */
22+ lsn_t* max_flushed_lsn) /*!< out: max of flushed
23+@@ -2014,10 +2008,6 @@
24+ if (!one_read_already) {
25+ *min_flushed_lsn = flushed_lsn;
26+ *max_flushed_lsn = flushed_lsn;
27+-#ifdef UNIV_LOG_ARCHIVE
28+- *min_arch_log_no = arch_log_no;
29+- *max_arch_log_no = arch_log_no;
30+-#endif /* UNIV_LOG_ARCHIVE */
31+ return;
32+ }
33+
34+@@ -2027,14 +2017,6 @@
35+ if (*max_flushed_lsn < flushed_lsn) {
36+ *max_flushed_lsn = flushed_lsn;
37+ }
38+-#ifdef UNIV_LOG_ARCHIVE
39+- if (*min_arch_log_no > arch_log_no) {
40+- *min_arch_log_no = arch_log_no;
41+- }
42+- if (*max_arch_log_no < arch_log_no) {
43+- *max_arch_log_no = arch_log_no;
44+- }
45+-#endif /* UNIV_LOG_ARCHIVE */
46+ }
47+
48+ /*================ SINGLE-TABLE TABLESPACES ==========================*/
49+@@ -3538,9 +3520,6 @@
50+ lsn_t lsn; /*!< Flushed LSN from header page */
51+ ulint id; /*!< Space ID */
52+ ulint flags; /*!< Tablespace flags */
53+-#ifdef UNIV_LOG_ARCHIVE
54+- ulint arch_log_no; /*!< latest archived log file number */
55+-#endif /* UNIV_LOG_ARCHIVE */
56+ };
57+
58+ static
59+@@ -3771,9 +3750,6 @@
60+ if (def.success) {
61+ fil_read_first_page(
62+ def.file, FALSE, &def.flags, &def.id,
63+-#ifdef UNIV_LOG_ARCHIVE
64+- &space_arch_log_no, &space_arch_log_no,
65+-#endif /* UNIV_LOG_ARCHIVE */
66+ &def.lsn, &def.lsn);
67+
68+ /* Validate this single-table-tablespace with SYS_TABLES,
69+@@ -3795,9 +3771,6 @@
70+ if (remote.success) {
71+ fil_read_first_page(
72+ remote.file, FALSE, &remote.flags, &remote.id,
73+-#ifdef UNIV_LOG_ARCHIVE
74+- &remote.arch_log_no, &remote.arch_log_no,
75+-#endif /* UNIV_LOG_ARCHIVE */
76+ &remote.lsn, &remote.lsn);
77+
78+ /* Validate this single-table-tablespace with SYS_TABLES,
79+@@ -3820,9 +3793,6 @@
80+ if (dict.success) {
81+ fil_read_first_page(
82+ dict.file, FALSE, &dict.flags, &dict.id,
83+-#ifdef UNIV_LOG_ARCHIVE
84+- &dict.arch_log_no, &dict.arch_log_no,
85+-#endif /* UNIV_LOG_ARCHIVE */
86+ &dict.lsn, &dict.lsn);
87+
88+ /* Validate this single-table-tablespace with SYS_TABLES,
89+@@ -4054,9 +4024,6 @@
90+ {
91+ fil_read_first_page(
92+ fsp->file, FALSE, &fsp->flags, &fsp->id,
93+-#ifdef UNIV_LOG_ARCHIVE
94+- &fsp->arch_log_no, &fsp->arch_log_no,
95+-#endif /* UNIV_LOG_ARCHIVE */
96+ &fsp->lsn, &fsp->lsn);
97+
98+ if (fsp->id == ULINT_UNDEFINED || fsp->id == 0) {
99+
100+--- a/storage/innobase/handler/ha_innodb.cc
101++++ b/storage/innobase/handler/ha_innodb.cc
102+@@ -2859,12 +2859,9 @@
103+ }
104+
105+ #ifdef UNIV_LOG_ARCHIVE
106+- /* Since innodb_log_arch_dir has no relevance under MySQL,
107+- starting from 4.0.6 we always set it the same as
108+- innodb_log_group_home_dir: */
109+-
110+- innobase_log_arch_dir = innobase_log_group_home_dir;
111+-
112++ if (!innobase_log_arch_dir) {
113++ innobase_log_arch_dir = srv_log_group_home_dir;
114++ }
115+ srv_arch_dir = innobase_log_arch_dir;
116+ #endif /* UNIG_LOG_ARCHIVE */
117+
118+
119+--- a/storage/innobase/include/fil0fil.h
120++++ b/storage/innobase/include/fil0fil.h
121+@@ -374,12 +374,6 @@
122+ contain sensible data */
123+ ulint* flags, /*!< out: tablespace flags */
124+ ulint* space_id, /*!< out: tablespace ID */
125+-#ifdef UNIV_LOG_ARCHIVE
126+- ulint* min_arch_log_no, /*!< out: min of archived
127+- log numbers in data files */
128+- ulint* max_arch_log_no, /*!< out: max of archived
129+- log numbers in data files */
130+-#endif /* UNIV_LOG_ARCHIVE */
131+ lsn_t* min_flushed_lsn, /*!< out: min of flushed
132+ lsn values in data files */
133+ lsn_t* max_flushed_lsn); /*!< out: max of flushed
134+
135+--- a/storage/innobase/include/log0log.h
136++++ b/storage/innobase/include/log0log.h
137+@@ -70,6 +70,10 @@
138+ /** Maximum number of log groups in log_group_t::checkpoint_buf */
139+ #define LOG_MAX_N_GROUPS 32
140+
141++#define IB_ARCHIVED_LOGS_PREFIX "ib_log_archive_"
142++#define IB_ARCHIVED_LOGS_PREFIX_LEN (sizeof(IB_ARCHIVED_LOGS_PREFIX) - 1)
143++#define IB_ARCHIVED_LOGS_SERIAL_LEN 20
144++
145+ /*******************************************************************//**
146+ Calculates where in log files we find a specified lsn.
147+ @return log file number */
148+@@ -340,8 +344,18 @@
149+ log_archived_file_name_gen(
150+ /*=======================*/
151+ char* buf, /*!< in: buffer where to write */
152++ ulint buf_len,/*!< in: buffer length */
153+ ulint id, /*!< in: group id */
154+- ulint file_no);/*!< in: file number */
155++ lsn_t file_no);/*!< in: file number */
156++
157++UNIV_INTERN
158++void
159++log_archived_get_offset(
160++/*====================*/
161++ log_group_t* group, /*!< in: log group */
162++ lsn_t file_no, /*!< in: archive log file number */
163++ lsn_t archived_lsn, /*!< in: last archived LSN */
164++ lsn_t* offset); /*!< out: offset within archived file */
165+ #else /* !UNIV_HOTBACKUP */
166+ /******************************************************//**
167+ Writes info to a buffer of a log group when log files are created in
168+@@ -739,19 +753,19 @@
169+ ulint archive_space_id;/*!< file space which
170+ implements the log group
171+ archive */
172+- ulint archived_file_no;/*!< file number corresponding to
173++ lsn_t archived_file_no;/*!< file number corresponding to
174+ log_sys->archived_lsn */
175+- ulint archived_offset;/*!< file offset corresponding to
176++ lsn_t archived_offset;/*!< file offset corresponding to
177+ log_sys->archived_lsn, 0 if we have
178+ not yet written to the archive file
179+ number archived_file_no */
180+- ulint next_archived_file_no;/*!< during an archive write,
181++ lsn_t next_archived_file_no;/*!< during an archive write,
182+ until the write is completed, we
183+ store the next value for
184+ archived_file_no here: the write
185+ completion function then sets the new
186+ value to ..._file_no */
187+- ulint next_archived_offset; /*!< like the preceding field */
188++ lsn_t next_archived_offset; /*!< like the preceding field */
189+ #endif /* UNIV_LOG_ARCHIVE */
190+ /*-----------------------------*/
191+ lsn_t scanned_lsn; /*!< used only in recovery: recovery scan
192+
193+--- a/storage/innobase/include/log0recv.h
194++++ b/storage/innobase/include/log0recv.h
195+@@ -297,18 +297,13 @@
196+ Recovers from archived log files, and also from log files, if they exist.
197+ @return error code or DB_SUCCESS */
198+ UNIV_INTERN
199+-ulint
200++dberr_t
201+ recv_recovery_from_archive_start(
202+ /*=============================*/
203+ lsn_t min_flushed_lsn,/*!< in: min flushed lsn field from the
204+ data files */
205+- lsn_t limit_lsn, /*!< in: recover up to this lsn if
206++ lsn_t limit_lsn); /*!< in: recover up to this lsn if
207+ possible */
208+- ulint first_log_no); /*!< in: number of the first archived
209+- log file to use in the recovery; the
210+- file will be searched from
211+- INNOBASE_LOG_ARCH_DIR specified in
212+- server config file */
213+ /********************************************************//**
214+ Completes recovery from archive. */
215+ UNIV_INTERN
216+
217+--- a/storage/innobase/include/univ.i
218++++ b/storage/innobase/include/univ.i
219+@@ -46,6 +46,9 @@
220+ #define INNODB_VERSION_MINOR MYSQL_VERSION_MINOR
221+ #define INNODB_VERSION_BUGFIX MYSQL_VERSION_PATCH
222+
223++/* Enable UNIV_LOG_ARCHIVE in XtraDB */
224++#define UNIV_LOG_ARCHIVE 1
225++
226+ /* The following is the InnoDB version as shown in
227+ SELECT plugin_version FROM information_schema.plugins;
228+ calculated in make_version_string() in sql/sql_show.cc like this:
229+
230+--- a/storage/innobase/log/log0log.cc
231++++ b/storage/innobase/log/log0log.cc
232+@@ -896,10 +896,10 @@
233+ mem_zalloc(sizeof(byte**) * n_files));
234+
235+ #ifdef UNIV_LOG_ARCHIVE
236+- group->archive_file_header_bufs_ptr = static_cast<byte*>(
237++ group->archive_file_header_bufs_ptr = static_cast<byte**>(
238+ mem_zalloc( sizeof(byte*) * n_files));
239+
240+- group->archive_file_header_bufs = static_cast<byte*>(
241++ group->archive_file_header_bufs = static_cast<byte**>(
242+ mem_zalloc(sizeof(byte*) * n_files));
243+ #endif /* UNIV_LOG_ARCHIVE */
244+
245+@@ -924,7 +924,7 @@
246+ #ifdef UNIV_LOG_ARCHIVE
247+ group->archive_space_id = archive_space_id;
248+
249+- group->archived_file_no = 0;
250++ group->archived_file_no = LOG_START_LSN;
251+ group->archived_offset = 0;
252+ #endif /* UNIV_LOG_ARCHIVE */
253+
254+@@ -2228,7 +2228,7 @@
255+ fil_io(OS_FILE_READ | OS_FILE_LOG, sync, group->space_id, 0,
256+ (ulint) (source_offset / UNIV_PAGE_SIZE),
257+ (ulint) (source_offset % UNIV_PAGE_SIZE),
258+- len, buf, NULL);
259++ len, buf, (type == LOG_ARCHIVE) ? &log_archive_io : NULL);
260+
261+ start_lsn += len;
262+ buf += len;
263+@@ -2247,12 +2247,68 @@
264+ log_archived_file_name_gen(
265+ /*=======================*/
266+ char* buf, /*!< in: buffer where to write */
267++ ulint buf_len,/*!< in: buffer length */
268+ ulint id __attribute__((unused)),
269+ /*!< in: group id;
270+ currently we only archive the first group */
271+- ulint file_no)/*!< in: file number */
272+-{
273+- sprintf(buf, "%sib_arch_log_%010lu", srv_arch_dir, (ulong) file_no);
274++ lsn_t file_no)/*!< in: file number */
275++{
276++ ulint dirnamelen;
277++
278++ dirnamelen = strlen(srv_arch_dir);
279++
280++ ut_a(buf_len > dirnamelen +
281++ IB_ARCHIVED_LOGS_SERIAL_LEN +
282++ IB_ARCHIVED_LOGS_PREFIX_LEN + 2);
283++
284++ strcpy(buf, srv_arch_dir);
285++
286++ if (buf[dirnamelen-1] != SRV_PATH_SEPARATOR) {
287++ buf[dirnamelen++] = SRV_PATH_SEPARATOR;
288++ }
289++ sprintf(buf + dirnamelen, IB_ARCHIVED_LOGS_PREFIX
290++ "%0" IB_TO_STR(IB_ARCHIVED_LOGS_SERIAL_LEN) "llu",
291++ file_no);
292++}
293++
294++/******************************************************//**
295++Get offset within archived log file to continue to write
296++with. */
297++UNIV_INTERN
298++void
299++log_archived_get_offset(
300++/*=====================*/
301++ log_group_t* group, /*!< in: log group */
302++ lsn_t file_no, /*!< in: archive log file number */
303++ lsn_t archived_lsn, /*!< in: last archived LSN */
304++ lsn_t* offset) /*!< out: offset within archived file */
305++{
306++ char file_name[OS_FILE_MAX_PATH];
307++ ibool exists;
308++ os_file_type_t type;
309++
310++ log_archived_file_name_gen(file_name,
311++ sizeof(file_name), group->id, file_no);
312++
313++ ut_a(os_file_status(file_name, &exists, &type));
314++
315++ if (!exists) {
316++ *offset = 0;
317++ return;
318++ }
319++
320++ *offset = archived_lsn - file_no + LOG_FILE_HDR_SIZE;
321++
322++ if (archived_lsn != IB_ULONGLONG_MAX) {
323++ *offset = archived_lsn - file_no + LOG_FILE_HDR_SIZE;
324++ } else {
325++ /* Archiving was OFF prior startup */
326++ *offset = 0;
327++ }
328++
329++ ut_a(group->file_size >= *offset + LOG_FILE_HDR_SIZE);
330++
331++ return;
332+ }
333+
334+ /******************************************************//**
335+@@ -2290,6 +2346,7 @@
336+ MONITOR_INC(MONITOR_LOG_IO);
337+
338+ fil_io(OS_FILE_WRITE | OS_FILE_LOG, TRUE, group->archive_space_id,
339++ 0,
340+ dest_offset / UNIV_PAGE_SIZE,
341+ dest_offset % UNIV_PAGE_SIZE,
342+ 2 * OS_FILE_LOG_BLOCK_SIZE,
343+@@ -2325,6 +2382,7 @@
344+ MONITOR_INC(MONITOR_LOG_IO);
345+
346+ fil_io(OS_FILE_WRITE | OS_FILE_LOG, TRUE, group->archive_space_id,
347++ 0,
348+ dest_offset / UNIV_PAGE_SIZE,
349+ dest_offset % UNIV_PAGE_SIZE,
350+ OS_FILE_LOG_BLOCK_SIZE,
351+@@ -2379,7 +2437,7 @@
352+ open_mode = OS_FILE_OPEN;
353+ }
354+
355+- log_archived_file_name_gen(name, group->id,
356++ log_archived_file_name_gen(name, sizeof(name), group->id,
357+ group->archived_file_no + n_files);
358+
359+ file_handle = os_file_create(innodb_file_log_key,
360+@@ -2454,6 +2512,7 @@
361+ MONITOR_INC(MONITOR_LOG_IO);
362+
363+ fil_io(OS_FILE_WRITE | OS_FILE_LOG, FALSE, group->archive_space_id,
364++ 0,
365+ (ulint) (next_offset / UNIV_PAGE_SIZE),
366+ (ulint) (next_offset % UNIV_PAGE_SIZE),
367+ ut_calc_align(len, OS_FILE_LOG_BLOCK_SIZE), buf,
368+@@ -2825,7 +2884,6 @@
369+ trunc_len);
370+ if (increment_file_count) {
371+ group->archived_offset = 0;
372+- group->archived_file_no += 2;
373+ }
374+
375+ #ifdef UNIV_DEBUG
376+@@ -3090,7 +3148,6 @@
377+ /*=======================================*/
378+ {
379+ lsn_t lsn;
380+- ulint arch_log_no;
381+ ulint count = 0;
382+ ulint total_trx;
383+ ulint pending_io;
384+@@ -3307,15 +3364,7 @@
385+ goto loop;
386+ }
387+
388+- arch_log_no = 0;
389+-
390+ #ifdef UNIV_LOG_ARCHIVE
391+- UT_LIST_GET_FIRST(log_sys->log_groups)->archived_file_no;
392+-
393+- if (0 == UT_LIST_GET_FIRST(log_sys->log_groups)->archived_offset) {
394+-
395+- arch_log_no--;
396+- }
397+
398+ log_archive_close_groups(TRUE);
399+ #endif /* UNIV_LOG_ARCHIVE */
400+@@ -3374,7 +3423,16 @@
401+ srv_shutdown_lsn = lsn;
402+
403+ if (!srv_read_only_mode) {
404+- fil_write_flushed_lsn_to_data_files(lsn, arch_log_no);
405++ /*
406++ log_sys->lsn is aligned up to the log block size
407++ in recv_reset_logs(), but if archived logs are applied
408++ data files must contain exactly the same flushed_lsn
409++ on which applying was finished
410++ */
411++ fil_write_flushed_lsn_to_data_files(
412++ srv_archive_recovery ? recv_sys->recovered_lsn :
413++ lsn,
414++ 0);
415+
416+ fil_flush_file_spaces(FIL_TABLESPACE);
417+ }
418+@@ -3597,7 +3655,7 @@
419+
420+ #ifdef UNIV_LOG_ARCHIVE
421+ rw_lock_free(&log_sys->archive_lock);
422+- os_event_create();
423++ os_event_free(log_sys->archiving_on);
424+ #endif /* UNIV_LOG_ARCHIVE */
425+
426+ #ifdef UNIV_LOG_DEBUG
427+
428+--- a/storage/innobase/log/log0recv.cc
429++++ b/storage/innobase/log/log0recv.cc
430+@@ -53,6 +53,13 @@
431+ //#else /* !UNIV_HOTBACKUP */
432+ #include "xb0xb.h"
433+
434++/** The size of archived log file */
435++extern ib_uint64_t xtrabackup_arch_file_size;
436++/** The minimal LSN of found archived log files */
437++extern ib_uint64_t xtrabackup_arch_first_file_lsn;
438++/** The maximum LSN of found archived log files */
439++extern ib_uint64_t xtrabackup_arch_last_file_lsn;
440++
441+ /** This is set to FALSE if the backup was originally taken with the
442+ ibbackup --include regexp option: then we do not want to create tables in
443+ directories which were not included */
444+@@ -2866,7 +2873,10 @@
445+ = log_block_get_checkpoint_no(log_block);
446+ }
447+
448+- if (data_len < OS_FILE_LOG_BLOCK_SIZE) {
449++ /* Stop scanning if scanned lsn reaches lsn limit */
450++ if (data_len < OS_FILE_LOG_BLOCK_SIZE ||
451++ (recv_sys->limit_lsn &&
452++ recv_sys->scanned_lsn >= recv_sys->limit_lsn)) {
453+ /* Log data for this group ends here */
454+
455+ finished = TRUE;
456+@@ -3157,10 +3167,14 @@
457+ group = UT_LIST_GET_FIRST(log_sys->log_groups);
458+
459+ while (group) {
460++ /* There is no need to read archived_file_no and
461++ archived_offset because they are calculated another
462++ way */
463++ /*
464+ log_checkpoint_get_nth_group_info(buf, group->id,
465+ &(group->archived_file_no),
466+ &(group->archived_offset));
467+-
468++ */
469+ group = UT_LIST_GET_NEXT(log_groups, group);
470+ }
471+ #endif /* UNIV_LOG_ARCHIVE */
472+@@ -3731,23 +3745,23 @@
473+ ulint len;
474+ ibool ret;
475+ byte* buf;
476+- os_offset_t read_offset;
477++ os_offset_t read_offset = 0;
478+ os_offset_t file_size;
479+ int input_char;
480+- char name[10000];
481++ char name[OS_FILE_MAX_PATH];
482+
483+- ut_a(0);
484+
485+ try_open_again:
486+ buf = log_sys->buf;
487+
488+ /* Add the file to the archive file space; open the file */
489+
490+- log_archived_file_name_gen(name, group->id, group->archived_file_no);
491++ log_archived_file_name_gen(name, sizeof(name),
492++ group->id, group->archived_file_no);
493+
494+ file_handle = os_file_create(innodb_file_log_key,
495+ name, OS_FILE_OPEN,
496+- OS_FILE_LOG, OS_FILE_AIO, &ret);
497++ OS_FILE_AIO, OS_LOG_FILE, &ret);
498+
499+ if (ret == FALSE) {
500+ ask_again:
501+@@ -3792,20 +3806,19 @@
502+
503+ /* Add the archive file as a node to the space */
504+
505+- fil_node_create(name, 1 + file_size / UNIV_PAGE_SIZE,
506+- group->archive_space_id, FALSE);
507+-#if RECV_SCAN_SIZE < LOG_FILE_HDR_SIZE
508+-# error "RECV_SCAN_SIZE < LOG_FILE_HDR_SIZE"
509+-#endif
510++ ut_a(fil_node_create(name, 1 + file_size / UNIV_PAGE_SIZE,
511++ group->archive_space_id, FALSE));
512++ ut_a(RECV_SCAN_SIZE >= LOG_FILE_HDR_SIZE);
513+
514+ /* Read the archive file header */
515+- fil_io(OS_FILE_READ | OS_FILE_LOG, TRUE, group->archive_space_id, 0, 0,
516++ fil_io(OS_FILE_READ | OS_FILE_LOG, TRUE, group->archive_space_id, 0,
517++ 0, 0,
518+ LOG_FILE_HDR_SIZE, buf, NULL);
519+
520+ /* Check if the archive file header is consistent */
521+
522+ if (mach_read_from_4(buf + LOG_GROUP_ID) != group->id
523+- || mach_read_from_4(buf + LOG_FILE_NO)
524++ || mach_read_from_8(buf + LOG_FILE_START_LSN)
525+ != group->archived_file_no) {
526+ fprintf(stderr,
527+ "InnoDB: Archive file header inconsistent %s\n", name);
528+@@ -3834,6 +3847,20 @@
529+ return(TRUE);
530+ }
531+
532++ /*
533++ if this is the first file of scanning set
534++ calculate offset of the first log record
535++ */
536++ if (recv_sys->parse_start_lsn >= start_lsn &&
537++ recv_sys->parse_start_lsn < file_end_lsn) {
538++ read_offset = ut_calc_align_down(
539++ recv_sys->parse_start_lsn - start_lsn,
540++ OS_FILE_LOG_BLOCK_SIZE);
541++ start_lsn = ut_calc_align_down(
542++ recv_sys->parse_start_lsn,
543++ OS_FILE_LOG_BLOCK_SIZE);
544++ }
545++
546+ recv_sys->scanned_lsn = start_lsn;
547+ }
548+
549+@@ -3846,7 +3873,7 @@
550+ return(TRUE);
551+ }
552+
553+- read_offset = LOG_FILE_HDR_SIZE;
554++ read_offset += LOG_FILE_HDR_SIZE;
555+
556+ for (;;) {
557+ len = RECV_SCAN_SIZE;
558+@@ -3872,7 +3899,7 @@
559+ #endif /* UNIV_DEBUG */
560+
561+ fil_io(OS_FILE_READ | OS_FILE_LOG, TRUE,
562+- group->archive_space_id, read_offset / UNIV_PAGE_SIZE,
563++ group->archive_space_id, 0, read_offset / UNIV_PAGE_SIZE,
564+ read_offset % UNIV_PAGE_SIZE, len, buf, NULL);
565+
566+ ret = recv_scan_log_recs(
567+@@ -3887,10 +3914,12 @@
568+ }
569+
570+ if (ret) {
571+- fprintf(stderr,
572+- "InnoDB: Archive log file %s"
573+- " does not scan right\n",
574+- name);
575++ if (recv_sys->scanned_lsn < recv_sys->limit_lsn) {
576++ fprintf(stderr,
577++ "InnoDB: Archive log file %s"
578++ " does not scan right\n",
579++ name);
580++ }
581+ return(TRUE);
582+ }
583+
584+@@ -3907,26 +3936,19 @@
585+ Recovers from archived log files, and also from log files, if they exist.
586+ @return error code or DB_SUCCESS */
587+ UNIV_INTERN
588+-ulint
589++dberr_t
590+ recv_recovery_from_archive_start(
591+ /*=============================*/
592+ ib_uint64_t min_flushed_lsn,/*!< in: min flushed lsn field from the
593+ data files */
594+- ib_uint64_t limit_lsn, /*!< in: recover up to this lsn if
595++ ib_uint64_t limit_lsn) /*!< in: recover up to this lsn if
596+ possible */
597+- ulint first_log_no) /*!< in: number of the first archived
598+- log file to use in the recovery; the
599+- file will be searched from
600+- INNOBASE_LOG_ARCH_DIR specified in
601+- server config file */
602+ {
603+ log_group_t* group;
604+ ulint group_id;
605+ ulint trunc_len;
606+ ibool ret;
607+- ulint err;
608+-
609+- ut_a(0);
610++ dberr_t err;
611+
612+ recv_sys_create();
613+ recv_sys_init(buf_pool_get_curr_size());
614+@@ -3956,7 +3978,7 @@
615+ return(DB_ERROR);
616+ }
617+
618+- group->archived_file_no = first_log_no;
619++ group->archived_file_no = xtrabackup_arch_first_file_lsn;
620+
621+ recv_sys->parse_start_lsn = min_flushed_lsn;
622+
623+@@ -3970,7 +3992,8 @@
624+
625+ mutex_enter(&(log_sys->mutex));
626+
627+- while (!ret) {
628++ while (!ret &&
629++ group->archived_file_no <= xtrabackup_arch_last_file_lsn) {
630+ ret = log_group_recover_from_archive_file(group);
631+
632+ /* Close and truncate a possible processed archive file
633+@@ -3983,9 +4006,9 @@
634+ trunc_len);
635+ }
636+
637+- group->archived_file_no++;
638++ group->archived_file_no += group->file_size - LOG_FILE_HDR_SIZE;
639+ }
640+-
641++/*
642+ if (recv_sys->recovered_lsn < limit_lsn) {
643+
644+ if (!recv_sys->scanned_lsn) {
645+@@ -4006,12 +4029,12 @@
646+
647+ mutex_enter(&(log_sys->mutex));
648+ }
649+-
650++*/
651+ if (limit_lsn != IB_ULONGLONG_MAX) {
652+
653+ recv_apply_hashed_log_recs(FALSE);
654+
655+- recv_reset_logs(0, FALSE, recv_sys->recovered_lsn);
656++ // recv_reset_logs(0, FALSE, recv_sys->recovered_lsn);
657+ }
658+
659+ mutex_exit(&(log_sys->mutex));
660+
661+--- a/storage/innobase/srv/srv0start.cc
662++++ b/storage/innobase/srv/srv0start.cc
663+@@ -641,6 +641,12 @@
664+ }
665+ }
666+
667++#ifdef UNIV_LOG_ARCHIVE
668++ /* Create the file space object for archived logs. */
669++ fil_space_create("arch_log_space", SRV_LOG_SPACE_FIRST_ID + 1,
670++ 0, FIL_LOG);
671++#endif
672++
673+ log_group_init(0, srv_n_log_files,
674+ srv_log_file_size * UNIV_PAGE_SIZE,
675+ SRV_LOG_SPACE_FIRST_ID,
676+@@ -651,7 +657,12 @@
677+ /* Create a log checkpoint. */
678+ mutex_enter(&log_sys->mutex);
679+ ut_d(recv_no_log_write = FALSE);
680+- recv_reset_logs(lsn);
681++ recv_reset_logs(
682++#ifdef UNIV_LOG_ARCHIVE
683++ UT_LIST_GET_FIRST(log_sys->log_groups)->archived_file_no,
684++ TRUE,
685++#endif
686++ lsn);
687+ mutex_exit(&log_sys->mutex);
688+
689+ return(DB_SUCCESS);
690+@@ -738,12 +749,6 @@
691+ /*======================*/
692+ ibool* create_new_db, /*!< out: TRUE if new database should be
693+ created */
694+-#ifdef UNIV_LOG_ARCHIVE
695+- ulint* min_arch_log_no,/*!< out: min of archived log
696+- numbers in data files */
697+- ulint* max_arch_log_no,/*!< out: max of archived log
698+- numbers in data files */
699+-#endif /* UNIV_LOG_ARCHIVE */
700+ lsn_t* min_flushed_lsn,/*!< out: min of flushed lsn
701+ values in data files */
702+ lsn_t* max_flushed_lsn,/*!< out: max of flushed lsn
703+@@ -964,9 +969,6 @@
704+ skip_size_check:
705+ fil_read_first_page(
706+ files[i], one_opened, &flags, &space,
707+-#ifdef UNIV_LOG_ARCHIVE
708+- min_arch_log_no, max_arch_log_no,
709+-#endif /* UNIV_LOG_ARCHIVE */
710+ min_flushed_lsn, max_flushed_lsn);
711+
712+ /* The first file of the system tablespace must
713+@@ -1450,8 +1452,8 @@
714+ lsn_t min_flushed_lsn;
715+ lsn_t max_flushed_lsn;
716+ #ifdef UNIV_LOG_ARCHIVE
717+- ulint min_arch_log_no;
718+- ulint max_arch_log_no;
719++ lsn_t min_arch_log_no;
720++ lsn_t max_arch_log_no;
721+ #endif /* UNIV_LOG_ARCHIVE */
722+ ulint sum_of_new_sizes;
723+ ulint sum_of_data_file_sizes;
724+@@ -1890,17 +1892,6 @@
725+ os_thread_create(io_handler_thread, n + i, thread_ids + i);
726+ }
727+
728+-#ifdef UNIV_LOG_ARCHIVE
729+- if (0 != ut_strcmp(srv_log_group_home_dir, srv_arch_dir)) {
730+- ut_print_timestamp(stderr);
731+- fprintf(stderr, " InnoDB: Error: you must set the log group home dir in my.cnf\n");
732+- ut_print_timestamp(stderr);
733+- fprintf(stderr, " InnoDB: the same as log arch dir.\n");
734+-
735+- return(DB_ERROR);
736+- }
737+-#endif /* UNIV_LOG_ARCHIVE */
738+-
739+ if (srv_n_log_files * srv_log_file_size * UNIV_PAGE_SIZE
740+ >= 512ULL * 1024ULL * 1024ULL * 1024ULL) {
741+ /* log_block_convert_lsn_to_no() limits the returned block
742+@@ -1959,9 +1950,6 @@
743+ }
744+
745+ err = open_or_create_data_files(&create_new_db,
746+-#ifdef UNIV_LOG_ARCHIVE
747+- &min_arch_log_no, &max_arch_log_no,
748+-#endif /* UNIV_LOG_ARCHIVE */
749+ &min_flushed_lsn, &max_flushed_lsn,
750+ &sum_of_new_sizes);
751+ if (err == DB_FAIL) {
752+@@ -1989,7 +1977,6 @@
753+
754+ #ifdef UNIV_LOG_ARCHIVE
755+ srv_normalize_path_for_win(srv_arch_dir);
756+- srv_arch_dir = srv_add_path_separator_if_needed(srv_arch_dir);
757+ #endif /* UNIV_LOG_ARCHIVE */
758+
759+ dirnamelen = strlen(srv_log_group_home_dir);
760+@@ -2068,9 +2055,15 @@
761+ max_flushed_lsn, logfile0);
762+
763+ /* Suppress the message about
764+- crash recovery. */
765+- max_flushed_lsn = min_flushed_lsn
766+- = log_get_lsn();
767++ crash recovery. If archive recovery
768++ is enabled the min/max_flushed_lsn
769++ must point to the position from
770++ which applying is started. */
771++ if (!srv_archive_recovery) {
772++ max_flushed_lsn = min_flushed_lsn
773++ = log_get_lsn();
774++ }
775++
776+ goto files_checked;
777+ #if 0
778+ } else if (i < 2) {
779+@@ -2244,10 +2237,13 @@
780+
781+ ib_logf(IB_LOG_LEVEL_INFO,
782+ " Starting archive recovery from a backup...");
783++ /* Load table spaces before recovery as during recovery
784++ there can be log records that are applied to the spaces
785++ with unknown id's */
786++ fil_load_single_table_tablespaces(NULL);
787+
788+ err = recv_recovery_from_archive_start(
789+- min_flushed_lsn, srv_archive_recovery_limit_lsn,
790+- min_arch_log_no);
791++ min_flushed_lsn, srv_archive_recovery_limit_lsn);
792+ if (err != DB_SUCCESS) {
793+
794+ return(DB_ERROR);
795+@@ -2271,6 +2267,11 @@
796+ srv_startup_is_before_trx_rollback_phase = FALSE;
797+
798+ recv_recovery_from_archive_finish();
799++
800++ if (srv_apply_log_only) {
801++ goto skip_processes;
802++ }
803++
804+ #endif /* UNIV_LOG_ARCHIVE */
805+ } else {
806+
807+@@ -2478,6 +2479,8 @@
808+ if (!srv_log_archive_on) {
809+ ut_a(DB_SUCCESS == log_archive_noarchivelog());
810+ } else {
811++ bool start_archive;
812++
813+ mutex_enter(&(log_sys->mutex));
814+
815+ start_archive = FALSE;
816+
817+--- a/storage/innobase/trx/trx0sys.cc
818++++ b/storage/innobase/trx/trx0sys.cc
819+@@ -1185,7 +1185,8 @@
820+ trx_purge_sys_close();
821+
822+ /* Free the double write data structures. */
823+- buf_dblwr_free();
824++ if (buf_dblwr)
825++ buf_dblwr_free();
826+
827+ mutex_enter(&trx_sys->mutex);
828+
829+
830
831=== modified file 'src/xtrabackup.cc'
832--- src/xtrabackup.cc 2013-08-20 17:43:45 +0000
833+++ src/xtrabackup.cc 2013-08-26 07:08:27 +0000
834@@ -118,6 +118,8 @@
835 char *xtrabackup_extra_lsndir = NULL; /* for --backup with --extra-lsndir */
836 char *xtrabackup_incremental_dir = NULL; /* for --prepare */
837
838+lsn_t xtrabackup_archived_to_lsn = 0; /* for --archived-to-lsn */
839+
840 char *xtrabackup_tables = NULL;
841 int tables_regex_num;
842 xb_regex_t *tables_regex;
843@@ -279,6 +281,17 @@
844
845 static my_bool xtrabackup_incremental_force_scan = FALSE;
846
847+/* The flushed lsn which is read from data files */
848+lsn_t min_flushed_lsn= 0;
849+lsn_t max_flushed_lsn= 0;
850+
851+/* The size of archived log file */
852+size_t xtrabackup_arch_file_size = 0ULL;
853+/* The minimal LSN of found archived log files */
854+lsn_t xtrabackup_arch_first_file_lsn = 0ULL;
855+/* The maximum LSN of found archived log files */
856+lsn_t xtrabackup_arch_last_file_lsn = 0ULL;
857+
858 /* Datasinks */
859 ds_ctxt_t *ds_data = NULL;
860 ds_ctxt_t *ds_meta = NULL;
861@@ -388,6 +401,7 @@
862 OPT_XTRA_INCREMENTAL_BASEDIR,
863 OPT_XTRA_EXTRA_LSNDIR,
864 OPT_XTRA_INCREMENTAL_DIR,
865+ OPT_XTRA_ARCHIVED_TO_LSN,
866 OPT_XTRA_TABLES,
867 OPT_XTRA_TABLES_FILE,
868 OPT_XTRA_CREATE_IB_LOGFILE,
869@@ -551,6 +565,12 @@
870 {"incremental-dir", OPT_XTRA_INCREMENTAL_DIR, "(for --prepare): apply .delta files and logfile in the specified directory.",
871 (G_PTR*) &xtrabackup_incremental_dir, (G_PTR*) &xtrabackup_incremental_dir,
872 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0},
873+#ifdef UNIV_LOG_ARCHIVE
874+ {"to-archived-lsn", OPT_XTRA_ARCHIVED_TO_LSN,
875+ "Don't apply archived logs with bigger log sequence number.",
876+ (G_PTR*) &xtrabackup_archived_to_lsn, (G_PTR*) &xtrabackup_archived_to_lsn, 0,
877+ GET_LL, REQUIRED_ARG, 0, 0, LONGLONG_MAX, 0, 0, 0},
878+#endif /* UNIV_LOG_ARCHIVE */
879 {"tables", OPT_XTRA_TABLES, "filtering by regexp for table names.",
880 (G_PTR*) &xtrabackup_tables, (G_PTR*) &xtrabackup_tables,
881 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0},
882@@ -733,11 +753,11 @@
883 (G_PTR*) &innobase_locks_unsafe_for_binlog,
884 (G_PTR*) &innobase_locks_unsafe_for_binlog, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0},
885 */
886-/*
887+#ifdef UNIV_LOG_ARCHIVE
888 {"innodb_log_arch_dir", OPT_INNODB_LOG_ARCH_DIR,
889 "Where full logs should be archived.", (G_PTR*) &innobase_log_arch_dir,
890 (G_PTR*) &innobase_log_arch_dir, 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0},
891-*/
892+#endif /* UNIV_LOG_ARCHIVE */
893 {"innodb_log_buffer_size", OPT_INNODB_LOG_BUFFER_SIZE,
894 "The size of the buffer which InnoDB uses to write log to the log files on disk.",
895 (G_PTR*) &innobase_log_buffer_size, (G_PTR*) &innobase_log_buffer_size, 0,
896@@ -2573,8 +2593,6 @@
897 ibool create_new_doublewrite_file;
898 #endif
899 ulint err;
900- lsn_t min_flushed_lsn;
901- lsn_t max_flushed_lsn;
902 ulint sum_of_new_sizes;
903
904 for (i = 0; i < srv_n_file_io_threads; i++) {
905@@ -5146,6 +5164,76 @@
906
907 }
908
909+#ifdef UNIV_LOG_ARCHIVE
910+/********************************************************************//**
911+Searches archived log files in archived log directory. The min and max
912+LSN's of found files as well as archived log file size are stored in
913+xtrabackup_arch_first_file_lsn, xtrabackup_arch_last_file_lsn and
914+xtrabackup_arch_file_size respectively.
915+@return true on success
916+*/
917+static
918+bool
919+xtrabackup_arch_search_files(
920+/*=========================*/
921+ ib_uint64_t start_lsn) /*!< in: filter out log files
922+ witch does not contain data
923+ with lsn < start_lsn */
924+{
925+ os_file_dir_t dir;
926+ os_file_stat_t fileinfo;
927+ ut_ad(innobase_log_arch_dir);
928+
929+ dir = os_file_opendir(innobase_log_arch_dir, FALSE);
930+ if (!dir) {
931+ msg("xtrabackup: error: cannot open archived log directory %s\n",
932+ innobase_log_arch_dir);
933+ return false;
934+ }
935+
936+ while(!os_file_readdir_next_file(innobase_log_arch_dir,
937+ dir,
938+ &fileinfo) ) {
939+ lsn_t log_file_lsn;
940+ char* log_str_end_lsn_ptr;
941+
942+ if (strncmp(fileinfo.name,
943+ IB_ARCHIVED_LOGS_PREFIX,
944+ sizeof(IB_ARCHIVED_LOGS_PREFIX) - 1)) {
945+ continue;
946+ }
947+
948+ log_file_lsn = strtoll(fileinfo.name +
949+ sizeof(IB_ARCHIVED_LOGS_PREFIX) - 1,
950+ &log_str_end_lsn_ptr, 10);
951+
952+ if (*log_str_end_lsn_ptr) {
953+ continue;
954+ }
955+
956+ if (log_file_lsn + (fileinfo.size - LOG_FILE_HDR_SIZE) < start_lsn) {
957+ continue;
958+ }
959+
960+ if (!xtrabackup_arch_first_file_lsn ||
961+ log_file_lsn < xtrabackup_arch_first_file_lsn) {
962+ xtrabackup_arch_first_file_lsn = log_file_lsn;
963+ }
964+ if (log_file_lsn > xtrabackup_arch_last_file_lsn) {
965+ xtrabackup_arch_last_file_lsn = log_file_lsn;
966+ }
967+
968+ //TODO: find the more suitable way to extract archived log file
969+ //size
970+ if (fileinfo.size > (ib_int64_t)xtrabackup_arch_file_size) {
971+ xtrabackup_arch_file_size = fileinfo.size;
972+ }
973+ }
974+
975+ return xtrabackup_arch_first_file_lsn != 0;
976+}
977+#endif /* UNIV_LOG_ARCHIVE */
978+
979 static void
980 xtrabackup_prepare_func(void)
981 {
982@@ -5169,14 +5257,18 @@
983 xtrabackup_target_dir[0]=FN_CURLIB; // all paths are relative from here
984 xtrabackup_target_dir[1]=0;
985
986- /* read metadata of target */
987+ /*
988+ read metadata of target, we don't need metadata reading in the case
989+ archived logs applying
990+ */
991+ sprintf(metadata_path, "%s/%s", xtrabackup_target_dir,
992+ XTRABACKUP_METADATA_FILENAME);
993+
994+ if (!xtrabackup_read_metadata(metadata_path))
995+ msg("xtrabackup: error: xtrabackup_read_metadata()\n");
996+
997+ if (!innobase_log_arch_dir)
998 {
999- sprintf(metadata_path, "%s/%s", xtrabackup_target_dir,
1000- XTRABACKUP_METADATA_FILENAME);
1001-
1002- if (!xtrabackup_read_metadata(metadata_path))
1003- msg("xtrabackup: error: xtrabackup_read_metadata()\n");
1004-
1005 if (!strcmp(metadata_type, "full-backuped")) {
1006 msg("xtrabackup: This target seems to be not prepared "
1007 "yet.\n");
1008@@ -5220,7 +5312,7 @@
1009 mem_init(srv_mem_pool_size);
1010 ut_crc32_init();
1011
1012- if(xtrabackup_init_temp_log())
1013+ if(!innobase_log_arch_dir && xtrabackup_init_temp_log())
1014 goto error;
1015
1016 if(innodb_init_param()) {
1017@@ -5248,14 +5340,15 @@
1018
1019 xb_normalize_init_values();
1020
1021- if (xtrabackup_incremental) {
1022+ if (xtrabackup_incremental || innobase_log_arch_dir) {
1023 err = xb_data_files_init();
1024 if (err != DB_SUCCESS) {
1025 msg("xtrabackup: error: xb_data_files_init() failed "
1026 "with error code %lu\n", err);
1027 goto error;
1028 }
1029-
1030+ }
1031+ if (xtrabackup_incremental) {
1032 inc_dir_tables_hash = hash_create(1000);
1033
1034 if(!xtrabackup_apply_deltas()) {
1035@@ -5263,9 +5356,11 @@
1036 xb_tables_hash_free(inc_dir_tables_hash);
1037 goto error;
1038 }
1039-
1040+ }
1041+ if (xtrabackup_incremental || innobase_log_arch_dir) {
1042 xb_data_files_close();
1043-
1044+ }
1045+ if (xtrabackup_incremental) {
1046 /* Cleanup datadir from tablespaces deleted between full and
1047 incremental backups */
1048
1049@@ -5295,6 +5390,55 @@
1050 srv_n_write_io_threads = 4;
1051 }
1052
1053+#ifdef UNIV_LOG_ARCHIVE
1054+ if (innobase_log_arch_dir) {
1055+ srv_arch_dir = innobase_log_arch_dir;
1056+ srv_archive_recovery = TRUE;
1057+ if (xtrabackup_archived_to_lsn) {
1058+ if (xtrabackup_archived_to_lsn < metadata_last_lsn) {
1059+ msg("xtrabackup: warning: logs applying lsn "
1060+ "limit " UINT64PF " is "
1061+ "less than metadata last-lsn " UINT64PF
1062+ " and will be set to metadata last-lsn value\n",
1063+ xtrabackup_archived_to_lsn,
1064+ metadata_last_lsn);
1065+ xtrabackup_archived_to_lsn = metadata_last_lsn;
1066+ }
1067+ if (xtrabackup_archived_to_lsn < min_flushed_lsn) {
1068+ msg("xtrabackup: error: logs applying "
1069+ "lsn limit " UINT64PF " is less than "
1070+ "min_flushed_lsn " UINT64PF
1071+ ", there is nothing to do\n",
1072+ xtrabackup_archived_to_lsn,
1073+ min_flushed_lsn);
1074+ goto error;
1075+ }
1076+ }
1077+ srv_archive_recovery_limit_lsn= xtrabackup_archived_to_lsn;
1078+ /*
1079+ Unfinished transactions are not rolled back during log applying
1080+ as they can be finished at the firther files applyings.
1081+ */
1082+ srv_apply_log_only = TRUE;
1083+
1084+ if (!xtrabackup_arch_search_files(min_flushed_lsn)) {
1085+ goto error;
1086+ }
1087+
1088+ /*
1089+ Check if last log file last lsn is big enough to overlap
1090+ last scanned lsn read from metadata.
1091+ */
1092+ if (xtrabackup_arch_last_file_lsn +
1093+ xtrabackup_arch_file_size -
1094+ LOG_FILE_HDR_SIZE < metadata_last_lsn) {
1095+ msg("xtrabackup: error: there are no enough archived logs "
1096+ "to apply\n");
1097+ goto error;
1098+ }
1099+ }
1100+#endif /* UNIV_LOG_ARCHIVE */
1101+
1102 msg("xtrabackup: Starting InnoDB instance for recovery.\n"
1103 "xtrabackup: Using %lld bytes for buffer pool "
1104 "(set by --use-memory parameter)\n", xtrabackup_use_memory);
1105@@ -5517,7 +5661,10 @@
1106 "'xtrabackup_binlog_pos_innodb'\n");
1107 }
1108 }
1109-
1110+#ifdef UNIV_LOG_ARCHIVE
1111+ if (innobase_log_arch_dir)
1112+ srv_start_lsn = log_sys->lsn = recv_sys->recovered_lsn;
1113+#endif /* UNIV_LOG_ARCHIVE */
1114 /* Check whether the log is applied enough or not. */
1115 if ((xtrabackup_incremental
1116 && ut_dulint_cmp(srv_start_lsn, incremental_last_lsn) < 0)
1117@@ -5829,6 +5976,14 @@
1118 exit(EXIT_FAILURE);
1119 }
1120
1121+ if (!xtrabackup_prepare &&
1122+ (innobase_log_arch_dir || xtrabackup_archived_to_lsn)) {
1123+ msg("xtrabackup: error: "
1124+ "--archived-logs-dir and --to-archived-lsn can be used "
1125+ "only with --prepare\n");
1126+ exit(EXIT_FAILURE);
1127+ }
1128+
1129 /* cannot execute both for now */
1130 {
1131 int num = 0;
1132
1133=== added file 'test/t/xb_apply_archived_logs.sh'
1134--- test/t/xb_apply_archived_logs.sh 1970-01-01 00:00:00 +0000
1135+++ test/t/xb_apply_archived_logs.sh 2013-08-26 07:08:27 +0000
1136@@ -0,0 +1,389 @@
1137+. inc/common.sh
1138+
1139+#The result to return
1140+RESULT=0
1141+
1142+function repeat_until_new_arch_log_created
1143+{
1144+ local arch_log_dir=$1
1145+ local command=$2
1146+ local stop_lsn=`run_cmd $MYSQL $MYSQL_ARGS test -e 'SHOW ENGINE INNODB STATUS\G'|grep 'Log sequence number'|awk '{print $4}'`
1147+ local old_arch_logs_count=`ls -al $arch_log_dir/| wc -l`
1148+ local new_arch_logs_count=0
1149+
1150+ echo $old_arch_logs_count
1151+ # To be sure the data was flushed to archived log wait until new file
1152+ # with START_LSN > CURRENT_LSN is created
1153+ local max_lsn=0;
1154+ while [ $max_lsn -le $stop_lsn ];
1155+ do
1156+ $command
1157+ for i in `ls -1 $arch_log_dir`;
1158+ do
1159+ local lsn=`echo $i | sed -s 's/ib_log_archive_\([0-9]\+\)/\1/'`
1160+ if [ $lsn -gt $max_lsn ];
1161+ then
1162+ max_lsn=$lsn
1163+ fi
1164+ done
1165+ done
1166+}
1167+
1168+function check_if_equal
1169+{
1170+ local NAME=$1
1171+ local VAL1=$2
1172+ local VAL2=$3
1173+ local MSG="$NAME ($VAL1 == $VAL2):"
1174+ if [ $VAL1 -ne $VAL2 ]
1175+ then
1176+ vlog "$MSG failed"
1177+ RESULT=-1
1178+ else
1179+ vlog "$MSG passed"
1180+ fi
1181+}
1182+
1183+function check_if_not_equal
1184+{
1185+ local NAME=$1
1186+ local VAL1=$2
1187+ local VAL2=$3
1188+ local MSG="$NAME ($VAL1 != $VAL2):"
1189+ if [ $VAL1 -eq $VAL2 ]
1190+ then
1191+ vlog "$MSG failed"
1192+ RESULT=-1
1193+ else
1194+ vlog "$MSG passed"
1195+ fi
1196+}
1197+
1198+function fill_tables
1199+{
1200+ local TABLES_COUNT=$1
1201+ local TABLE_NAME=$2
1202+
1203+ for i in `seq 1 $TABLES_COUNT`; do
1204+ local TN=$TABLE_NAME$i
1205+ run_cmd $MYSQL $MYSQL_ARGS test \
1206+<<EOF
1207+ INSERT INTO $TN (B)
1208+ SELECT ${TN}_1.B FROM
1209+ $TN ${TN}_1,
1210+ $TN ${TN}_2,
1211+ $TN ${TN}_3
1212+ LIMIT 10000;
1213+
1214+EOF
1215+ done
1216+}
1217+
1218+function create_and_fill_db
1219+{
1220+ local TABLES_COUNT=$1
1221+ local TABLE_NAME=$2
1222+ local ARCH_LOG_DIR=$3
1223+ local CREATE_TABLE_OPTIONS=${4:-''}
1224+
1225+ for i in `seq 1 $TABLES_COUNT`; do
1226+ local TN=$TABLE_NAME$i
1227+
1228+ run_cmd $MYSQL $MYSQL_ARGS test \
1229+<<EOF
1230+ CREATE TABLE $TN
1231+ (A INT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
1232+ B INT(10) UNSIGNED NOT NULL DEFAULT 0) ENGINE=INNODB $CREATE_TABLE_OPTIONS;
1233+ INSERT INTO $TN (B) VALUES (1),(1),(1),(1),(1),(1),(1),(1),(1),(1),
1234+ (1),(1),(1),(1),(1),(1),(1),(1),(1),(1),
1235+ (1),(1),(1),(1),(1),(1),(1),(1),(1),(1),
1236+ (1),(1),(1),(1),(1),(1),(1),(1),(1),(1),
1237+ (1),(1),(1),(1),(1),(1),(1),(1),(1),(1),
1238+ (1),(1),(1),(1),(1),(1),(1),(1),(1),(1),
1239+ (1),(1),(1),(1),(1),(1),(1),(1),(1),(1),
1240+ (1),(1),(1),(1),(1),(1),(1),(1),(1),(1),
1241+ (1),(1),(1),(1),(1),(1),(1),(1),(1),(1),
1242+ (1),(1),(1),(1),(1),(1),(1),(1),(1),(1);
1243+EOF
1244+ done
1245+
1246+ repeat_until_new_arch_log_created \
1247+ $ARCH_LOG_DIR \
1248+ "fill_tables $TABLES_COUNT $TABLE_NAME"
1249+}
1250+
1251+function make_changes
1252+{
1253+ local START_I=$1
1254+ local TABLES_COUNT=$2
1255+ local TABLE_NAME=$3
1256+ local CLIENT_CMD="${MYSQL} ${MYSQL_ARGS} -Ns test -e"
1257+
1258+ for i in `seq $START_I $(($START_I+$TABLES_COUNT-1))`; do
1259+ local TN=$TABLE_NAME$(($i-$START_I+1))
1260+
1261+ #INSERT
1262+ I_FROM[$i]=`$CLIENT_CMD "select max(a) from $TN"`
1263+ $CLIENT_CMD "insert into $TN (b) select b from $TN limit 1000"
1264+ I_TO[$i]=`$CLIENT_CMD "select max(a) from $TN"`
1265+ vlog "Inserted rows for $i are in the range [${I_FROM[$i]}, ${I_TO[$i]})"
1266+ I_COUNT[$i]=`$CLIENT_CMD "select count(*) from $TN where a >= ${I_FROM[$i]} and a < ${I_TO[$i]} "`
1267+
1268+ #DELETE
1269+ if [ $i -gt $TABLES_COUNT ];
1270+ then
1271+ local START_ROW=${U_TO[$(($i-$TABLES_COUNT))]}
1272+ else
1273+ local START_ROW=1000
1274+ fi
1275+ D_FROM[$i]=`$CLIENT_CMD "select min(a) from (select a from $TN where a >= $START_ROW limit 1000) as temp_table"`
1276+ D_TO[$i]=`$CLIENT_CMD "select max(a) from (select a from $TN where a >= $START_ROW limit 1000) as temp_table"`
1277+ D_COUNT[$i]=`$CLIENT_CMD "select count(*) from $TN where a>=${D_FROM[$i]} and a<${D_TO[$i]}"`
1278+ $CLIENT_CMD "delete from $TN where a >= ${D_FROM[$i]} and a < ${D_TO[$i]}"
1279+ vlog "Deleted rows for $i are in the range [${D_FROM[$i]}, ${D_TO[$i]}), total ${D_COUNT[$i]} are deleted"
1280+
1281+ #UPDATE
1282+ U_FROM[$i]=${D_TO[$i]}
1283+ U_TO[$i]=`$CLIENT_CMD "select max(a) from (select a from $TN where a >= ${U_FROM[$i]} limit 1000) as temp_table"`
1284+ U_COUNT[$i]=`$CLIENT_CMD "select count(*) from $TN where a>=${U_FROM[$i]} and a<${U_TO[$i]}"`
1285+ $CLIENT_CMD "update $TN set b=2 where a >= ${U_FROM[$i]} and a < ${U_TO[$i]}"
1286+ vlog "Updated rows for $i are in the range [${U_FROM[$i]}, ${U_TO[$i]}), total ${U_COUNT[$i]} are updated"
1287+
1288+ done
1289+}
1290+
1291+function get_changes
1292+{
1293+ local START_I=$1
1294+ local TABLES_COUNT=$2
1295+ local TABLE_NAME=$3
1296+ local CLIENT_CMD="${MYSQL} ${MYSQL_ARGS} -Ns test -e"
1297+
1298+ for i in `seq $START_I $(($START_I+$TABLES_COUNT-1))`; do
1299+ local TN=$TABLE_NAME$(($i-$START_I+1))
1300+
1301+ #INSERT
1302+ I_COUNT1[$i]=`$CLIENT_CMD "select count(*) from $TN where a >= ${I_FROM[$i]} and a < ${I_TO[$i]} "`
1303+ #DELETE
1304+ D_COUNT1[$i]=`$CLIENT_CMD "select count(*) from $TN where a >= ${D_FROM[$i]} and a < ${D_TO[$i]} "`
1305+ #UPDATE
1306+ U_COUNT1[$i]=`$CLIENT_CMD "select count(*) from $TN where a >= ${U_FROM[$i]} and a < ${U_TO[$i]} and b = 2"`
1307+ done
1308+}
1309+
1310+function check_changes
1311+{
1312+ local START_I=$1
1313+ local TABLES_COUNT=$2
1314+ local CMD=$3
1315+
1316+ for i in `seq $START_I $(($START_I+$TABLES_COUNT-1))`; do
1317+ $CMD "INSERT TEST for $i" ${I_COUNT[$i]} ${I_COUNT1[$i]}
1318+ $CMD "DELETE TEST for $i" 0 ${D_COUNT1[$i]}
1319+ $CMD "UPDATE TEST for $i" ${U_COUNT[$i]} ${U_COUNT1[$i]}
1320+ done
1321+}
1322+
1323+function unset_global_variables
1324+{
1325+ unset I_FROM
1326+ unset I_TO
1327+ unset I_COUNT
1328+
1329+ unset D_FROM
1330+ unset D_TO
1331+ unset D_COUNT
1332+
1333+ unset U_FROM
1334+ unset U_TO
1335+ unset U_COUNT
1336+
1337+ unset I_FROM1
1338+ unset I_TO1
1339+ unset I_COUNT1
1340+
1341+ unset D_FROM1
1342+ unset D_TO1
1343+ unset D_COUNT1
1344+
1345+ unset U_FROM1
1346+ unset U_TO1
1347+ unset U_COUNT1
1348+}
1349+
1350+function test_archived_logs
1351+{
1352+ local TABLE_NAME=T
1353+ local TABLES_COUNT=4
1354+ local EXTRA_OPTIONS=${1:-''}
1355+ local CREATE_TABLE_OPTIONS=${2:-''}
1356+
1357+ #Setup server environment to get access to some variables
1358+ init_server_variables 1
1359+ switch_server 1
1360+ local BASE_BACKUP_DIR=$topdir/backup_base
1361+ local BACKUP_DIR=$topdir/backup
1362+ local BASE_DATA_DIR=$topdir/base_data
1363+ local ARCHIVED_LOGS_DIR=$topdir/archived_logs
1364+ local XTRABACKUP_OPTIONS="--innodb_log_file_size=2M $EXTRA_OPTIONS"
1365+ #Setup ROW binlog format to supress warnings in result file
1366+ local SERVER_OPTIONS="$XTRABACKUP_OPTIONS --innodb_log_archive=ON --innodb_log_arch_dir=$ARCHIVED_LOGS_DIR --binlog-format=ROW"
1367+ mkdir -p $BASE_BACKUP_DIR $BACKUP_DIR
1368+ mkdir -p $ARCHIVED_LOGS_DIR
1369+ reset_server_variables 1
1370+
1371+ ###################################################################
1372+ # --to-lsn test. It checks the availability to apply logs only to #
1373+ # the certain LSN. #
1374+ ###################################################################
1375+ start_server $SERVER_OPTIONS
1376+ #Create and fill tables to generate log files
1377+ create_and_fill_db $TABLES_COUNT $TABLE_NAME $ARCHIVED_LOGS_DIR $CREATE_TABLE_OPTIONS
1378+ #Backup the data
1379+ xtrabackup --backup --datadir=$mysql_datadir --target-dir=$BASE_BACKUP_DIR $XTRABACKUP_OPTIONS
1380+ #Make some changes in tables after backup is done
1381+ make_changes 1 $TABLES_COUNT $TABLE_NAME
1382+ #Make sure that changes are flushed to archived log
1383+ repeat_until_new_arch_log_created \
1384+ $ARCHIVED_LOGS_DIR \
1385+ "fill_tables $TABLES_COUNT $TABLE_NAME"
1386+ #Remember current LSN
1387+ local LSN=`run_cmd $MYSQL $MYSQL_ARGS test -e 'SHOW ENGINE INNODB STATUS\G'|grep 'Log sequence number'|awk '{print $4}'`
1388+ #Make more changes over remembered LSN
1389+ make_changes $(($TABLES_COUNT+1)) $TABLES_COUNT $TABLE_NAME
1390+ #Make sure the above changes are flushed to archived log
1391+ repeat_until_new_arch_log_created \
1392+ $ARCHIVED_LOGS_DIR \
1393+ "fill_tables $TABLES_COUNT $TABLE_NAME"
1394+ stop_server
1395+ cp -R $mysql_datadir $BASE_DATA_DIR
1396+
1397+ #########################################
1398+ # Apply logs only to the remembered lsn #
1399+ #########################################
1400+ # --apply-logs-only is set implicitly because unfinished transactions
1401+ # can be finished on further logs applying but using this option with
1402+ # --innodb-log-arch-dir is tested here to prove this bug
1403+ # https://bugs.launchpad.net/percona-xtrabackup/+bug/1199555 is not
1404+ # concerned with this case
1405+ cp -R $BASE_BACKUP_DIR/* $BACKUP_DIR
1406+ xtrabackup --prepare \
1407+ --target-dir=$BACKUP_DIR \
1408+ --innodb-log-arch-dir=$ARCHIVED_LOGS_DIR \
1409+ --to-archived-lsn=$LSN \
1410+ --apply-logs-only \
1411+ $XTRABACKUP_OPTIONS
1412+ #Copy prepared data to server data dir
1413+ cp -R $BACKUP_DIR/* $mysql_datadir
1414+ rm $mysql_datadir/ib_*
1415+ #Start server with prepared data
1416+ start_server "--binlog-format=ROW $EXTRA_OPTIONS"
1417+ #Get values from restored data files before remembered LSN
1418+ get_changes 1 $TABLES_COUNT $TABLE_NAME
1419+ #Get values from restored data files after remembered LSN
1420+ get_changes $(($TABLES_COUNT+1)) $TABLES_COUNT $TABLE_NAME
1421+ #We don't need server already
1422+ stop_server
1423+ #Check if the changes which was made before remembered LSN are in the
1424+ #restored databse
1425+ check_changes 1 $TABLES_COUNT 'check_if_equal'
1426+ #Check if the changes which was made after remembered LSN are NOT in the
1427+ #restored databse
1428+ check_changes $(($TABLES_COUNT+1)) $TABLES_COUNT 'check_if_not_equal'
1429+ # Apply the rest of archived logs
1430+ xtrabackup --prepare \
1431+ --target-dir=$BACKUP_DIR \
1432+ --innodb-log-arch-dir=$ARCHIVED_LOGS_DIR \
1433+ --apply-logs-only \
1434+ $XTRABACKUP_OPTIONS
1435+ #Copy prepared data to server data dir
1436+ cp -R $BACKUP_DIR/* $mysql_datadir
1437+ rm $mysql_datadir/ib_*
1438+ #Start server with prepared data
1439+ start_server "--binlog-format=ROW $EXTRA_OPTIONS"
1440+ #Get values from restored data files before remembered LSN
1441+ get_changes 1 $TABLES_COUNT $TABLE_NAME
1442+ #Get values from restored data files after remembered LSN
1443+ get_changes $(($TABLES_COUNT+1)) $TABLES_COUNT $TABLE_NAME
1444+ stop_server
1445+ #Check if the changes which was made before remembered LSN are in the
1446+ #restored databse
1447+ check_changes 1 $TABLES_COUNT 'check_if_equal'
1448+ #Check if the changes which was made after remembered LSN are in the
1449+ #restored databse
1450+ check_changes $(($TABLES_COUNT+1)) $TABLES_COUNT 'check_if_equal'
1451+ rm -rf $mysql_datadir
1452+ rm -rf $BACKUP_DIR
1453+
1454+ ##################################################
1455+ # Check the possibility of applying logs by sets #
1456+ ##################################################
1457+ cp -R $BASE_DATA_DIR $mysql_datadir
1458+ cp -R $BASE_BACKUP_DIR $BACKUP_DIR
1459+ mkdir -p $ARCHIVED_LOGS_DIR/1 $ARCHIVED_LOGS_DIR/2
1460+ #Make two log files sets. The first set contains the first two files,
1461+ #the second set contains the rest and the last log file from the first
1462+ #set.
1463+ pushd .
1464+ cd $ARCHIVED_LOGS_DIR
1465+ for i in `find . -type f -printf "%T+ %p\n" | cut -d' ' -f2`;
1466+ do
1467+ local n=`echo $i | sed 's/.*ib_log_archive_\([0-9]*\).*/\1/'`;
1468+ if [ $n -le $LSN ];
1469+ then
1470+ mv $i 1/;
1471+ else
1472+ mv $i 2/;
1473+ fi;
1474+ done
1475+ cd 1
1476+ find . -type f -printf "%T+ %p\n" | cut -d' ' -f2 | sort -n | tail -1 | \
1477+ xargs -I{} cp {} ../2/
1478+ popd
1479+ #Prepare the first set
1480+ xtrabackup --prepare \
1481+ --target-dir=$BACKUP_DIR \
1482+ --innodb-log-arch-dir=$ARCHIVED_LOGS_DIR/1 \
1483+ $XTRABACKUP_OPTIONS
1484+ #Prepare the second set
1485+ xtrabackup --prepare \
1486+ --target-dir=$BACKUP_DIR \
1487+ --innodb-log-arch-dir=$ARCHIVED_LOGS_DIR/2 \
1488+ $XTRABACKUP_OPTIONS
1489+ #Copy prepared data to server data dir
1490+ cp -R $BACKUP_DIR/* $mysql_datadir
1491+ rm $mysql_datadir/ib_*
1492+ #Start server with prepared data
1493+ start_server "--binlog-format=ROW $EXTRA_OPTIONS"
1494+ #Get values from restored data files before remembered LSN
1495+ get_changes 1 $TABLES_COUNT $TABLE_NAME
1496+ #Get values from restored data files after remembered LSN
1497+ get_changes $(($TABLES_COUNT+1)) $TABLES_COUNT $TABLE_NAME
1498+ stop_server
1499+ #Check all made changes
1500+ check_changes 1 $TABLES_COUNT 'check_if_equal'
1501+ check_changes $(($TABLES_COUNT+1)) $TABLES_COUNT 'check_if_equal'
1502+
1503+ #Clean up dir for the next procedure launch
1504+ rm -rf $BACKUP_DIR/* $BASE_BACKUP_DIR/*
1505+ rm -rf $mysql_datadir $BASE_DATA_DIR
1506+ rm -rf $ARCHIVED_LOGS_DIR/*
1507+
1508+ #Clean up variables
1509+ unset_global_variables
1510+}
1511+
1512+require_xtradb
1513+require_server_version_higher_than '5.6.10'
1514+
1515+
1516+if [ `basename $XB_BIN` = 'xtrabackup_56' ]
1517+then
1518+ test_archived_logs
1519+ test_archived_logs '' 'ROW_FORMAT=COMPRESSED'
1520+else
1521+ echo 'Requires xtrabackup built with XtraDB >= 5.6' > $SKIPPED_REASON
1522+ exit $SKIPPED_EXIT_CODE
1523+fi
1524+
1525+exit $RESULT

Subscribers

People subscribed via source and target branches