Merge lp:~vlad-lesin/percona-xtrabackup/2.1-apply-archived-logs-innodb5.6 into lp:percona-xtrabackup/2.1
- 2.1-apply-archived-logs-innodb5.6
- Merge into 2.1
Status: | Merged |
---|---|
Approved by: | Alexey Kopytov |
Approved revision: | no longer in the source branch. |
Merged at revision: | 675 |
Proposed branch: | lp:~vlad-lesin/percona-xtrabackup/2.1-apply-archived-logs-innodb5.6 |
Merge into: | lp:percona-xtrabackup/2.1 |
Diff against target: |
1525 lines (+1383/-17) 3 files modified
patches/innodb56.patch (+822/-0) src/xtrabackup.cc (+172/-17) test/t/xb_apply_archived_logs.sh (+389/-0) |
To merge this branch: | bzr merge lp:~vlad-lesin/percona-xtrabackup/2.1-apply-archived-logs-innodb5.6 |
Related bugs: | |
Related blueprints: |
Log Archiving for xtrabackup: restoring
(Essential)
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Alexey Kopytov (community) | Approve | ||
Laurynas Biveinis (community) | Needs Fixing | ||
Review via email: mp+160804@code.launchpad.net |
Commit message
Description of the change
This is implementation of ability to apply archived logs in xtrabackup which is described here: https:/
See commit comment for more detailed description.
Laurynas Biveinis (laurynas-biveinis) wrote : | # |
Vlad Lesin (vlad-lesin) wrote : | # |
> A merge conflict, unless spurious, that needs rebasing on the current trunk?
I have just rebased it on the current trunk.
Alexey Kopytov (akopytov) wrote : | # |
Vlad,
- I don't think the XtraBackup part should pull the entire server
implement
of it should be applied to InnoDB 5.6 code base to be able to
parse and apply archived logs? As in, do we really need
fil_
changes to log_checkpoint_
- can you clarify the following change? I.e. under what
circumstances exactly can a tablespace exist when we are trying
to replay MLOG_FILE_CREATE from the archived logs?
+@@ -3325,9 +3341,13 @@
+ path = fil_make_
+ }
+
++ /* The files can already exist in the case of archived logs applying */
+ file = os_file_create(
+ innodb_
+- OS_FILE_CREATE | OS_FILE_
++ (srv_archive_
++ OS_FILE_OVERWRITE :
++ OS_FILE_CREATE) |
++ OS_FILE_
+ OS_FILE_NORMAL,
+ OS_DATA_FILE,
+ &ret);
- s/alligned to the up of log file block/aligned up to the log block size/
- please clarify why the following change is needed. As I
understand, we don't start the FTS optimize thread in
XtraBackup. That code is also missing block braces and is using
spaces instead of tabs for indentation:
+@@ -3032,7 +3032,8 @@
+
+ ib_logf(
+
+- os_event_
++ if (exit_event)
++ os_event_
+
+ /* We count the number of threads in os_thread_exit(). A created
+ thread should always use that to exit and not use return() to exit. */
+
- please use LSN_PF instead of "%llu"
- in recv_recovery_
argument and use xtrabackup_
it be better to pass xtrabackup_
first_log_no argument to recv_recovery_
get rid of min/max_arch_log_no in
innobase_
- in xb_data_
declarations with min/max_
used. but then the call still calls open_or_
with the (now non-existing) flushed_lsn variables?
- s/xtrabackup: Note: opening archived log directory %s was
failed/
- replace the strstr() call in xtrabackup_
strncmp()
- xtrabackup_
returns xtrabackup_
- we tend to use the InnoDB code style in xtrabackup.cc. In
particular, the opening brace must be on the same line as the
'if' statement. And braces should be used even for
single-
- s/trunsactions/
- s/undid/rolled back/
- the check for Percona Server 5.6 doesn't work (try starting it
with an upstream 5.6 server). The reason is that the syntax for
string equality comparison is "val1 = val2", not "val1=val2"
- the test case doesn't wo...
Sergei Glushchenko (sergei.glushchenko) wrote : | # |
Vlad,
Why don't we use --log-arch-dir option (with same name as server one) instead of --archived-
Vlad Lesin (vlad-lesin) wrote : | # |
On 05/06/2013 06:21 PM, Sergei Glushchenko wrote:
> Vlad,
>
> Why don't we use --log-arch-dir option (with same name as server one) instead of --archived-
>
This is good idea, thanks.
--
Vlad Lesin, Software Engineer, Percona Inc.
<email address hidden>
skype: vlad_lesin
JID: <email address hidden>
ICQ: 204036003
phone: +79051122311
Tula, Russia (GMT +4)
Vlad Lesin (vlad-lesin) wrote : | # |
On 05/01/2013 02:40 PM, Alexey Kopytov wrote:
> - I don't think the XtraBackup part should pull the entire server
> implementation of log archiving. It looks like only a minor part
> of it should be applied to InnoDB 5.6 code base to be able to
> parse and apply archived logs? As in, do we really need
> fil_space_
> changes to log_checkpoint_
Done.
> - can you clarify the following change? I.e. under what
> circumstances exactly can a tablespace exist when we are trying
> to replay MLOG_FILE_CREATE from the archived logs?
Yes. It can be for example in the case when this file did not exist at
the beginning of backup process and was created somewhere in the middle
of backup process. So log file contains "create" command as backup
process started tracking it before the file creation, but data directory
contains this file as it was copied from original directory. I copied
this explanation to code comment.
> +@@ -3325,9 +3341,13 @@
> + path = fil_make_
> + }
> +
> ++ /* The files can already exist in the case of archived logs applying */
> + file = os_file_create(
> + innodb_
> +- OS_FILE_CREATE | OS_FILE_
> ++ (srv_archive_
> ++ OS_FILE_OVERWRITE :
> ++ OS_FILE_CREATE) |
> ++ OS_FILE_
> + OS_FILE_NORMAL,
> + OS_DATA_FILE,
> + &ret);
>
> - s/alligned to the up of log file block/aligned up to the log block size/
Done.
> - please clarify why the following change is needed. As I
> understand, we don't start the FTS optimize thread in
> XtraBackup. That code is also missing block braces and is using
> spaces instead of tabs for indentation:
>
> +@@ -3032,7 +3032,8 @@
> +
> + ib_logf(
> +
> +- os_event_
> ++ if (exit_event)
> ++ os_event_
> +
> + /* We count the number of threads in os_thread_exit(). A created
> + thread should always use that to exit and not use return() to exit. */
> +
I fixed using spaces instead tabs for indentation and missing braces.
As I understuud FTS optimize thread is started in xtrabackup because I
have the following backtrace in the case of missing exit_event condition:
56 ../nptl/
(gdb) bt
#0 0x00007ffff633f037 in __GI_raise (sig=sig@entry=6) at
../nptl/
#1 0x00007ffff6342698 in __GI_abort () at abort.c:90
#2 0x000000000063ac80 in os_event_set (event=0x0)
at
/home/vlesin/
#3 0x00000000006063ec in fts_optimize_thread (arg=0x1e604e8)
at
/home/vlesin/
#4 0x00007ffff79bcf8e in start_thread (arg=0x7fffd57f
pthread_
#5 0x00007ffff6401e1d in clone () at
../sysdeps/
Here is the backtrace of FTS initializati...
Vlad Lesin (vlad-lesin) wrote : | # |
On 05/06/2013 06:21 PM, Sergei Glushchenko wrote:
> Vlad,
>
> Why don't we use --log-arch-dir option (with same name as server one) instead of --archived-
>
Done.
--
Vlad Lesin, Software Engineer, Percona Inc.
<email address hidden>
skype: vlad_lesin
JID: <email address hidden>
ICQ: 204036003
phone: +79051122311
Tula, Russia (GMT +4)
Alexey Kopytov (akopytov) wrote : | # |
Hi Vlad,
General comments:
- I think this MP should wait until 5.6.11-60.3 with the fix for bug
#1172591 is released. Then the test suite should use 5.6.11-60.30
for testing instead of the custom build
- there should be a separate fix for bug #1177182 for both 2.0 and
2.1, and again, this branch should be rebased once the fix is merged to
trunks.
- xtrabackup_
Other comments/questions:
On Fri, 10 May 2013 09:42:06 +0400, Vlad Lesin wrote:
>> - can you clarify the following change? I.e. under what
>> circumstances exactly can a tablespace exist when we are trying
>> to replay MLOG_FILE_CREATE from the archived logs?
>
> Yes. It can be for example in the case when this file did not exist at
> the beginning of backup process and was created somewhere in the middle
> of backup process. So log file contains "create" command as backup
> process started tracking it before the file creation, but data directory
> contains this file as it was copied from original directory. I copied
> this explanation to code comment.
>
If a tablespace exists on recovery, it is created into fil_system before
the recovery starts. When replaying MLOG_FILE_CREATE,
fil_op_
fil_system and does nothing. So I still don't understand that change in
fil_create_
>
>> +@@ -3325,9 +3341,13 @@
>> + path = fil_make_
>> + }
>> +
>> ++ /* The files can already exist in the case of archived logs
>> applying */
>> + file = os_file_create(
>> + innodb_
>> +- OS_FILE_CREATE | OS_FILE_
>> ++ (srv_archive_
>> ++ OS_FILE_OVERWRITE :
>> ++ OS_FILE_CREATE) |
>> ++ OS_FILE_
>> + OS_FILE_NORMAL,
>> + OS_DATA_FILE,
>> + &ret);
>>
>> - s/alligned to the up of log file block/aligned up to the log
>> block size/
>
> Done.
>
Not fully done, it is "aligned" (single "l"), not "alligned".
>> - please clarify why the following change is needed. As I
>> understand, we don't start the FTS optimize thread in
>> XtraBackup. That code is also missing block braces and is using
>> spaces instead of tabs for indentation:
>>
>> +@@ -3032,7 +3032,8 @@
>> +
>> + ib_logf(
>> +
>> +- os_event_
>> ++ if (exit_event)
>> ++ os_event_
>> +
>> + /* We count the number of threads in os_thread_exit(). A created
>> + thread should always use that to exit and not use return() to
>> exit. */
>> +
>
> I fixed using spaces instead tabs for indentation and missing braces.
>
> As I understuud FTS optimize thread is started in xtrabackup because I
>
OK, I see the problem now, reported as bug #1179193. This means we have
to fix that bug separately in both 2.0 and 2.1, and then rebase this
branch on 2.1 trunk with that fix included.
>> - please use LSN_PF instead of "%llu"
>
...
Vlad Lesin (vlad-lesin) wrote : | # |
On 05/12/2013 04:48 PM, Alexey Kopytov wrote:
>>> - in recv_recovery_
>>> argument and use xtrabackup_
>>> it be better to pass xtrabackup_
>>> first_log_no argument to recv_recovery_
As I understood our chat right the main cause of this change is to
remove 'fake' arguments from recv_recovery_
Why just don't remove first_log_no from
recv_recovery_
benefits in passing xtrabackup_
xtrabackup_
arguments because in this case we just remove using global external
variables from recv_recovery_
innobase_
>>> get rid of min/max_arch_log_no in
>>> innobase_
Right. In this case we could remove min/max_arch_log_no from
open_or_
too.
--
Vlad Lesin, Software Engineer, Percona Inc.
<email address hidden>
skype: vlad_lesin
JID: <email address hidden>
ICQ: 204036003
phone: +79051122311
Tula, Russia (GMT +4)
Vlad Lesin (vlad-lesin) wrote : | # |
On 05/12/2013 04:48 PM, Alexey Kopytov wrote:
>
> I tried to repeat the test, but your branch doesn't build for me now, it
> fails as follows:
>
> [ 70%] Building CXX object
> libmysqld/
> /Users/
> error: no matching function for call to
> 'log_checkpoint
> log_checkpoint_
> ^~~~~~~
> /Users/
> note: candidate function not viable: no known
> conversion from 'lsn_t *' (aka 'unsigned long long *') to 'ulint
> *' (aka 'unsigned long *') for 3rd argument;
> log_checkpoint_
> ^
> 1[ 70%] error generated.
>
I am preparing the more detailed answer with code fixes so currently I
don't want to commit unfinished changes. But it is possible just to
comment log_checkpoint_
because the values it reads from buffer are not used for logs applying.
--
Vlad Lesin, Software Engineer, Percona Inc.
<email address hidden>
skype: vlad_lesin
JID: <email address hidden>
ICQ: 204036003
phone: +79051122311
Tula, Russia (GMT +4)
Alexey Kopytov (akopytov) wrote : | # |
Hi Vlad,
On Mon, 13 May 2013 17:44:36 +0400, Vlad Lesin wrote:
> On 05/12/2013 04:48 PM, Alexey Kopytov wrote:
>>>> - in recv_recovery_
>>>> first_log_no
>>>> argument and use xtrabackup_
>>>> wouldn't
>>>> it be better to pass xtrabackup_
>>>> first_log_no argument to recv_recovery_
>>>> and
>
> As I understood our chat right the main cause of this change is to
> remove 'fake' arguments from recv_recovery_
>
> Why just don't remove first_log_no from
> recv_recovery_
> benefits in passing xtrabackup_
> xtrabackup_
> arguments because in this case we just remove using global external
> variables from recv_recovery_
> innobase_
>
Either way is fine by me as long as we avoid fake arguments.
> >>> get rid of min/max_arch_log_no in
> >>> innobase_
>
> Right. In this case we could remove min/max_arch_log_no from
> open_or_
> too.
>
OK.
Vlad Lesin (vlad-lesin) wrote : | # |
> > Why just don't remove first_log_no from
> > recv_recovery_
> > benefits in passing xtrabackup_
> > xtrabackup_
> > arguments because in this case we just remove using global external
> > variables from recv_recovery_
> > innobase_
> >
>
> Either way is fine by me as long as we avoid fake arguments.
Done.
> > Right. In this case we could remove min/max_arch_log_no from
> > open_or_
> > too.
> >
>
> OK.
Done.
Vlad Lesin (vlad-lesin) wrote : | # |
> Hi Vlad,
>
>
> General comments:
>
> - I think this MP should wait until 5.6.11-60.3 with the fix for bug
> #1172591 is released. Then the test suite should use 5.6.11-60.30
> for testing instead of the custom build
The fix is released.
> - there should be a separate fix for bug #1177182 for both 2.0 and
> 2.1, and again, this branch should be rebased once the fix is merged to
> trunks.
The fix for 2.1 is merged and for 2.0 is approved for merging.
> - xtrabackup_
> fprintf(stderr,...)
Fixed.
> >> - s/alligned to the up of log file block/aligned up to the log
> >> block size/
> >
> > Done.
> >
>
> Not fully done, it is "aligned" (single "l"), not "alligned".
Done.
> OK, I see the problem now, reported as bug #1179193. This means we have
> to fix that bug separately in both 2.0 and 2.1, and then rebase this
> branch on 2.1 trunk with that fix included.
I explored this problem more deeply and found out that the problem is "apply archived log" specific. See my comment to the bug report and
if (srv_apply_
}
strings in innobase_
> >> - the test case doesn't work for me (fails with "table 'test.t1'
> >> doesn't exist" when run against PS 5.6 with log archiving enabled)
> >>
> >
> > I need your assistance here because I can't reproduce this on my
> > environment and, as I understood, on Jenkins environment too(see
> > http://
> xtrabackup-
> >
>
> I tried to repeat the test, but your branch doesn't build for me now, it
> fails as follows:
>
> [ 70%] Building CXX object
> libmysqld/
> /Users/
> innodb5.
> error: no matching function for call to
> 'log_checkpoint
> log_checkpoint_
> ^~~~~~~
> /Users/
> innodb5.
> note: candidate function not viable: no known
> conversion from 'lsn_t *' (aka 'unsigned long long *') to 'ulint
> *' (aka 'unsigned long *') for 3rd argument;
> log_checkpoint_
> ^
> 1[ 70%] error generated.
Fixed. The invocation of log_checkpoint_
Vlad Lesin (vlad-lesin) wrote : | # |
Rebased on newest trunk (lp:percona-xtrabackup revision 650).
Alexey Kopytov (akopytov) wrote : | # |
I don't think XtraDB-related changes in run.sh are necessary now. Check the is_xtradb(
Alexey Kopytov (akopytov) wrote : | # |
Oh, and you will also need require_
Vlad Lesin (vlad-lesin) wrote : | # |
On 07/31/2013 04:54 PM, Alexey Kopytov wrote:
> I don't think XtraDB-related changes in run.sh are necessary now. Check the is_xtradb(
>
Yes, you are right, there is no need to change run.sh as server and
innodb flavours are detected right now based on "innodb_version" server
variable.
But require_
require_
require_
need to check server flavour too as well as
require_
but xtradb version must be checked too.
So I reverted run.sh and fixed xb_apply_
is_server_
Alexey Kopytov (akopytov) wrote : | # |
Hi Vlad,
On Wed, 31 Jul 2013 21:43:33 -0000, Vlad Lesin wrote:
> But require_
> require_
> require_
> need to check server flavour too as well as
> require_
> but xtradb version must be checked too.
>
You can do:
require_xtradb
require_
Or use is_* functions if you need something complex.
Vlad Lesin (vlad-lesin) wrote : | # |
On 08/01/2013 08:39 AM, Alexey Kopytov wrote:
> Hi Vlad,
>
> On Wed, 31 Jul 2013 21:43:33 -0000, Vlad Lesin wrote:
>> But require_
>> require_
>> require_
>> need to check server flavour too as well as
>> require_
>> but xtradb version must be checked too.
>>
>
> You can do:
>
> require_xtradb
> require_
>
> Or use is_* functions if you need something complex.
>
Yes, but xtradb can be 5.5,5.1 or 5.6. But only 5.6 supports logs applying.
Alexey Kopytov (akopytov) wrote : | # |
Hi Vlad,
On Thu, 01 Aug 2013 11:01:56 +0400, Vlad Lesin wrote:
>> You can do:
>>
>> require_xtradb
>> require_
>>
>> Or use is_* functions if you need something complex.
>>
>
> Yes, but xtradb can be 5.5,5.1 or 5.6. But only 5.6 supports logs applying.
>
The above 2 lines will make the test run only on XtraDB > 5.6.10?
Vlad Lesin (vlad-lesin) wrote : | # |
http://
http://
Alexey Kopytov (akopytov) wrote : | # |
Vlad,
Do you have any explanations for multiple test failures and crashes in both builds?
Vlad Lesin (vlad-lesin) wrote : | # |
> Vlad,
>
> Do you have any explanations for multiple test failures and crashes in both
> builds?
Yes, there were bug concerned with wrong lsn writing to data files on "prepare" step when logs applying is not used. I fixed it, here is the new test results:
http://
Alexey Kopytov (akopytov) wrote : | # |
Vlad,
Approved, but please rebase on the current trunk. I get many conflicts in innodb56.patch. The good news is that it's going to be the last rebase for this branch. Sorry about the stretched review process. Please also do another Jenkins param build after rebasing.
Thanks.
Vlad Lesin (vlad-lesin) wrote : | # |
> Vlad,
>
> Approved, but please rebase on the current trunk. I get many conflicts in
> innodb56.patch. The good news is that it's going to be the last rebase for
> this branch. Sorry about the stretched review process. Please also do another
> Jenkins param build after rebasing.
>
> Thanks.
Done.
http://
Alexey Kopytov (akopytov) : | # |
Preview Diff
1 | === modified file 'patches/innodb56.patch' |
2 | --- patches/innodb56.patch 2013-08-20 17:43:45 +0000 |
3 | +++ patches/innodb56.patch 2013-08-26 07:08:27 +0000 |
4 | @@ -1564,3 +1564,825 @@ |
5 | |
6 | buf = group->checkpoint_buf; |
7 | |
8 | +--- a/storage/innobase/fil/fil0fil.cc |
9 | ++++ b/storage/innobase/fil/fil0fil.cc |
10 | +@@ -1980,12 +1980,6 @@ |
11 | + contain sensible data */ |
12 | + ulint* flags, /*!< out: tablespace flags */ |
13 | + ulint* space_id, /*!< out: tablespace ID */ |
14 | +-#ifdef UNIV_LOG_ARCHIVE |
15 | +- ulint* min_arch_log_no, /*!< out: min of archived |
16 | +- log numbers in data files */ |
17 | +- ulint* max_arch_log_no, /*!< out: max of archived |
18 | +- log numbers in data files */ |
19 | +-#endif /* UNIV_LOG_ARCHIVE */ |
20 | + lsn_t* min_flushed_lsn, /*!< out: min of flushed |
21 | + lsn values in data files */ |
22 | + lsn_t* max_flushed_lsn) /*!< out: max of flushed |
23 | +@@ -2014,10 +2008,6 @@ |
24 | + if (!one_read_already) { |
25 | + *min_flushed_lsn = flushed_lsn; |
26 | + *max_flushed_lsn = flushed_lsn; |
27 | +-#ifdef UNIV_LOG_ARCHIVE |
28 | +- *min_arch_log_no = arch_log_no; |
29 | +- *max_arch_log_no = arch_log_no; |
30 | +-#endif /* UNIV_LOG_ARCHIVE */ |
31 | + return; |
32 | + } |
33 | + |
34 | +@@ -2027,14 +2017,6 @@ |
35 | + if (*max_flushed_lsn < flushed_lsn) { |
36 | + *max_flushed_lsn = flushed_lsn; |
37 | + } |
38 | +-#ifdef UNIV_LOG_ARCHIVE |
39 | +- if (*min_arch_log_no > arch_log_no) { |
40 | +- *min_arch_log_no = arch_log_no; |
41 | +- } |
42 | +- if (*max_arch_log_no < arch_log_no) { |
43 | +- *max_arch_log_no = arch_log_no; |
44 | +- } |
45 | +-#endif /* UNIV_LOG_ARCHIVE */ |
46 | + } |
47 | + |
48 | + /*================ SINGLE-TABLE TABLESPACES ==========================*/ |
49 | +@@ -3538,9 +3520,6 @@ |
50 | + lsn_t lsn; /*!< Flushed LSN from header page */ |
51 | + ulint id; /*!< Space ID */ |
52 | + ulint flags; /*!< Tablespace flags */ |
53 | +-#ifdef UNIV_LOG_ARCHIVE |
54 | +- ulint arch_log_no; /*!< latest archived log file number */ |
55 | +-#endif /* UNIV_LOG_ARCHIVE */ |
56 | + }; |
57 | + |
58 | + static |
59 | +@@ -3771,9 +3750,6 @@ |
60 | + if (def.success) { |
61 | + fil_read_first_page( |
62 | + def.file, FALSE, &def.flags, &def.id, |
63 | +-#ifdef UNIV_LOG_ARCHIVE |
64 | +- &space_arch_log_no, &space_arch_log_no, |
65 | +-#endif /* UNIV_LOG_ARCHIVE */ |
66 | + &def.lsn, &def.lsn); |
67 | + |
68 | + /* Validate this single-table-tablespace with SYS_TABLES, |
69 | +@@ -3795,9 +3771,6 @@ |
70 | + if (remote.success) { |
71 | + fil_read_first_page( |
72 | + remote.file, FALSE, &remote.flags, &remote.id, |
73 | +-#ifdef UNIV_LOG_ARCHIVE |
74 | +- &remote.arch_log_no, &remote.arch_log_no, |
75 | +-#endif /* UNIV_LOG_ARCHIVE */ |
76 | + &remote.lsn, &remote.lsn); |
77 | + |
78 | + /* Validate this single-table-tablespace with SYS_TABLES, |
79 | +@@ -3820,9 +3793,6 @@ |
80 | + if (dict.success) { |
81 | + fil_read_first_page( |
82 | + dict.file, FALSE, &dict.flags, &dict.id, |
83 | +-#ifdef UNIV_LOG_ARCHIVE |
84 | +- &dict.arch_log_no, &dict.arch_log_no, |
85 | +-#endif /* UNIV_LOG_ARCHIVE */ |
86 | + &dict.lsn, &dict.lsn); |
87 | + |
88 | + /* Validate this single-table-tablespace with SYS_TABLES, |
89 | +@@ -4054,9 +4024,6 @@ |
90 | + { |
91 | + fil_read_first_page( |
92 | + fsp->file, FALSE, &fsp->flags, &fsp->id, |
93 | +-#ifdef UNIV_LOG_ARCHIVE |
94 | +- &fsp->arch_log_no, &fsp->arch_log_no, |
95 | +-#endif /* UNIV_LOG_ARCHIVE */ |
96 | + &fsp->lsn, &fsp->lsn); |
97 | + |
98 | + if (fsp->id == ULINT_UNDEFINED || fsp->id == 0) { |
99 | + |
100 | +--- a/storage/innobase/handler/ha_innodb.cc |
101 | ++++ b/storage/innobase/handler/ha_innodb.cc |
102 | +@@ -2859,12 +2859,9 @@ |
103 | + } |
104 | + |
105 | + #ifdef UNIV_LOG_ARCHIVE |
106 | +- /* Since innodb_log_arch_dir has no relevance under MySQL, |
107 | +- starting from 4.0.6 we always set it the same as |
108 | +- innodb_log_group_home_dir: */ |
109 | +- |
110 | +- innobase_log_arch_dir = innobase_log_group_home_dir; |
111 | +- |
112 | ++ if (!innobase_log_arch_dir) { |
113 | ++ innobase_log_arch_dir = srv_log_group_home_dir; |
114 | ++ } |
115 | + srv_arch_dir = innobase_log_arch_dir; |
116 | + #endif /* UNIG_LOG_ARCHIVE */ |
117 | + |
118 | + |
119 | +--- a/storage/innobase/include/fil0fil.h |
120 | ++++ b/storage/innobase/include/fil0fil.h |
121 | +@@ -374,12 +374,6 @@ |
122 | + contain sensible data */ |
123 | + ulint* flags, /*!< out: tablespace flags */ |
124 | + ulint* space_id, /*!< out: tablespace ID */ |
125 | +-#ifdef UNIV_LOG_ARCHIVE |
126 | +- ulint* min_arch_log_no, /*!< out: min of archived |
127 | +- log numbers in data files */ |
128 | +- ulint* max_arch_log_no, /*!< out: max of archived |
129 | +- log numbers in data files */ |
130 | +-#endif /* UNIV_LOG_ARCHIVE */ |
131 | + lsn_t* min_flushed_lsn, /*!< out: min of flushed |
132 | + lsn values in data files */ |
133 | + lsn_t* max_flushed_lsn); /*!< out: max of flushed |
134 | + |
135 | +--- a/storage/innobase/include/log0log.h |
136 | ++++ b/storage/innobase/include/log0log.h |
137 | +@@ -70,6 +70,10 @@ |
138 | + /** Maximum number of log groups in log_group_t::checkpoint_buf */ |
139 | + #define LOG_MAX_N_GROUPS 32 |
140 | + |
141 | ++#define IB_ARCHIVED_LOGS_PREFIX "ib_log_archive_" |
142 | ++#define IB_ARCHIVED_LOGS_PREFIX_LEN (sizeof(IB_ARCHIVED_LOGS_PREFIX) - 1) |
143 | ++#define IB_ARCHIVED_LOGS_SERIAL_LEN 20 |
144 | ++ |
145 | + /*******************************************************************//** |
146 | + Calculates where in log files we find a specified lsn. |
147 | + @return log file number */ |
148 | +@@ -340,8 +344,18 @@ |
149 | + log_archived_file_name_gen( |
150 | + /*=======================*/ |
151 | + char* buf, /*!< in: buffer where to write */ |
152 | ++ ulint buf_len,/*!< in: buffer length */ |
153 | + ulint id, /*!< in: group id */ |
154 | +- ulint file_no);/*!< in: file number */ |
155 | ++ lsn_t file_no);/*!< in: file number */ |
156 | ++ |
157 | ++UNIV_INTERN |
158 | ++void |
159 | ++log_archived_get_offset( |
160 | ++/*====================*/ |
161 | ++ log_group_t* group, /*!< in: log group */ |
162 | ++ lsn_t file_no, /*!< in: archive log file number */ |
163 | ++ lsn_t archived_lsn, /*!< in: last archived LSN */ |
164 | ++ lsn_t* offset); /*!< out: offset within archived file */ |
165 | + #else /* !UNIV_HOTBACKUP */ |
166 | + /******************************************************//** |
167 | + Writes info to a buffer of a log group when log files are created in |
168 | +@@ -739,19 +753,19 @@ |
169 | + ulint archive_space_id;/*!< file space which |
170 | + implements the log group |
171 | + archive */ |
172 | +- ulint archived_file_no;/*!< file number corresponding to |
173 | ++ lsn_t archived_file_no;/*!< file number corresponding to |
174 | + log_sys->archived_lsn */ |
175 | +- ulint archived_offset;/*!< file offset corresponding to |
176 | ++ lsn_t archived_offset;/*!< file offset corresponding to |
177 | + log_sys->archived_lsn, 0 if we have |
178 | + not yet written to the archive file |
179 | + number archived_file_no */ |
180 | +- ulint next_archived_file_no;/*!< during an archive write, |
181 | ++ lsn_t next_archived_file_no;/*!< during an archive write, |
182 | + until the write is completed, we |
183 | + store the next value for |
184 | + archived_file_no here: the write |
185 | + completion function then sets the new |
186 | + value to ..._file_no */ |
187 | +- ulint next_archived_offset; /*!< like the preceding field */ |
188 | ++ lsn_t next_archived_offset; /*!< like the preceding field */ |
189 | + #endif /* UNIV_LOG_ARCHIVE */ |
190 | + /*-----------------------------*/ |
191 | + lsn_t scanned_lsn; /*!< used only in recovery: recovery scan |
192 | + |
193 | +--- a/storage/innobase/include/log0recv.h |
194 | ++++ b/storage/innobase/include/log0recv.h |
195 | +@@ -297,18 +297,13 @@ |
196 | + Recovers from archived log files, and also from log files, if they exist. |
197 | + @return error code or DB_SUCCESS */ |
198 | + UNIV_INTERN |
199 | +-ulint |
200 | ++dberr_t |
201 | + recv_recovery_from_archive_start( |
202 | + /*=============================*/ |
203 | + lsn_t min_flushed_lsn,/*!< in: min flushed lsn field from the |
204 | + data files */ |
205 | +- lsn_t limit_lsn, /*!< in: recover up to this lsn if |
206 | ++ lsn_t limit_lsn); /*!< in: recover up to this lsn if |
207 | + possible */ |
208 | +- ulint first_log_no); /*!< in: number of the first archived |
209 | +- log file to use in the recovery; the |
210 | +- file will be searched from |
211 | +- INNOBASE_LOG_ARCH_DIR specified in |
212 | +- server config file */ |
213 | + /********************************************************//** |
214 | + Completes recovery from archive. */ |
215 | + UNIV_INTERN |
216 | + |
217 | +--- a/storage/innobase/include/univ.i |
218 | ++++ b/storage/innobase/include/univ.i |
219 | +@@ -46,6 +46,9 @@ |
220 | + #define INNODB_VERSION_MINOR MYSQL_VERSION_MINOR |
221 | + #define INNODB_VERSION_BUGFIX MYSQL_VERSION_PATCH |
222 | + |
223 | ++/* Enable UNIV_LOG_ARCHIVE in XtraDB */ |
224 | ++#define UNIV_LOG_ARCHIVE 1 |
225 | ++ |
226 | + /* The following is the InnoDB version as shown in |
227 | + SELECT plugin_version FROM information_schema.plugins; |
228 | + calculated in make_version_string() in sql/sql_show.cc like this: |
229 | + |
230 | +--- a/storage/innobase/log/log0log.cc |
231 | ++++ b/storage/innobase/log/log0log.cc |
232 | +@@ -896,10 +896,10 @@ |
233 | + mem_zalloc(sizeof(byte**) * n_files)); |
234 | + |
235 | + #ifdef UNIV_LOG_ARCHIVE |
236 | +- group->archive_file_header_bufs_ptr = static_cast<byte*>( |
237 | ++ group->archive_file_header_bufs_ptr = static_cast<byte**>( |
238 | + mem_zalloc( sizeof(byte*) * n_files)); |
239 | + |
240 | +- group->archive_file_header_bufs = static_cast<byte*>( |
241 | ++ group->archive_file_header_bufs = static_cast<byte**>( |
242 | + mem_zalloc(sizeof(byte*) * n_files)); |
243 | + #endif /* UNIV_LOG_ARCHIVE */ |
244 | + |
245 | +@@ -924,7 +924,7 @@ |
246 | + #ifdef UNIV_LOG_ARCHIVE |
247 | + group->archive_space_id = archive_space_id; |
248 | + |
249 | +- group->archived_file_no = 0; |
250 | ++ group->archived_file_no = LOG_START_LSN; |
251 | + group->archived_offset = 0; |
252 | + #endif /* UNIV_LOG_ARCHIVE */ |
253 | + |
254 | +@@ -2228,7 +2228,7 @@ |
255 | + fil_io(OS_FILE_READ | OS_FILE_LOG, sync, group->space_id, 0, |
256 | + (ulint) (source_offset / UNIV_PAGE_SIZE), |
257 | + (ulint) (source_offset % UNIV_PAGE_SIZE), |
258 | +- len, buf, NULL); |
259 | ++ len, buf, (type == LOG_ARCHIVE) ? &log_archive_io : NULL); |
260 | + |
261 | + start_lsn += len; |
262 | + buf += len; |
263 | +@@ -2247,12 +2247,68 @@ |
264 | + log_archived_file_name_gen( |
265 | + /*=======================*/ |
266 | + char* buf, /*!< in: buffer where to write */ |
267 | ++ ulint buf_len,/*!< in: buffer length */ |
268 | + ulint id __attribute__((unused)), |
269 | + /*!< in: group id; |
270 | + currently we only archive the first group */ |
271 | +- ulint file_no)/*!< in: file number */ |
272 | +-{ |
273 | +- sprintf(buf, "%sib_arch_log_%010lu", srv_arch_dir, (ulong) file_no); |
274 | ++ lsn_t file_no)/*!< in: file number */ |
275 | ++{ |
276 | ++ ulint dirnamelen; |
277 | ++ |
278 | ++ dirnamelen = strlen(srv_arch_dir); |
279 | ++ |
280 | ++ ut_a(buf_len > dirnamelen + |
281 | ++ IB_ARCHIVED_LOGS_SERIAL_LEN + |
282 | ++ IB_ARCHIVED_LOGS_PREFIX_LEN + 2); |
283 | ++ |
284 | ++ strcpy(buf, srv_arch_dir); |
285 | ++ |
286 | ++ if (buf[dirnamelen-1] != SRV_PATH_SEPARATOR) { |
287 | ++ buf[dirnamelen++] = SRV_PATH_SEPARATOR; |
288 | ++ } |
289 | ++ sprintf(buf + dirnamelen, IB_ARCHIVED_LOGS_PREFIX |
290 | ++ "%0" IB_TO_STR(IB_ARCHIVED_LOGS_SERIAL_LEN) "llu", |
291 | ++ file_no); |
292 | ++} |
293 | ++ |
294 | ++/******************************************************//** |
295 | ++Get offset within archived log file to continue to write |
296 | ++with. */ |
297 | ++UNIV_INTERN |
298 | ++void |
299 | ++log_archived_get_offset( |
300 | ++/*=====================*/ |
301 | ++ log_group_t* group, /*!< in: log group */ |
302 | ++ lsn_t file_no, /*!< in: archive log file number */ |
303 | ++ lsn_t archived_lsn, /*!< in: last archived LSN */ |
304 | ++ lsn_t* offset) /*!< out: offset within archived file */ |
305 | ++{ |
306 | ++ char file_name[OS_FILE_MAX_PATH]; |
307 | ++ ibool exists; |
308 | ++ os_file_type_t type; |
309 | ++ |
310 | ++ log_archived_file_name_gen(file_name, |
311 | ++ sizeof(file_name), group->id, file_no); |
312 | ++ |
313 | ++ ut_a(os_file_status(file_name, &exists, &type)); |
314 | ++ |
315 | ++ if (!exists) { |
316 | ++ *offset = 0; |
317 | ++ return; |
318 | ++ } |
319 | ++ |
320 | ++ *offset = archived_lsn - file_no + LOG_FILE_HDR_SIZE; |
321 | ++ |
322 | ++ if (archived_lsn != IB_ULONGLONG_MAX) { |
323 | ++ *offset = archived_lsn - file_no + LOG_FILE_HDR_SIZE; |
324 | ++ } else { |
325 | ++ /* Archiving was OFF prior startup */ |
326 | ++ *offset = 0; |
327 | ++ } |
328 | ++ |
329 | ++ ut_a(group->file_size >= *offset + LOG_FILE_HDR_SIZE); |
330 | ++ |
331 | ++ return; |
332 | + } |
333 | + |
334 | + /******************************************************//** |
335 | +@@ -2290,6 +2346,7 @@ |
336 | + MONITOR_INC(MONITOR_LOG_IO); |
337 | + |
338 | + fil_io(OS_FILE_WRITE | OS_FILE_LOG, TRUE, group->archive_space_id, |
339 | ++ 0, |
340 | + dest_offset / UNIV_PAGE_SIZE, |
341 | + dest_offset % UNIV_PAGE_SIZE, |
342 | + 2 * OS_FILE_LOG_BLOCK_SIZE, |
343 | +@@ -2325,6 +2382,7 @@ |
344 | + MONITOR_INC(MONITOR_LOG_IO); |
345 | + |
346 | + fil_io(OS_FILE_WRITE | OS_FILE_LOG, TRUE, group->archive_space_id, |
347 | ++ 0, |
348 | + dest_offset / UNIV_PAGE_SIZE, |
349 | + dest_offset % UNIV_PAGE_SIZE, |
350 | + OS_FILE_LOG_BLOCK_SIZE, |
351 | +@@ -2379,7 +2437,7 @@ |
352 | + open_mode = OS_FILE_OPEN; |
353 | + } |
354 | + |
355 | +- log_archived_file_name_gen(name, group->id, |
356 | ++ log_archived_file_name_gen(name, sizeof(name), group->id, |
357 | + group->archived_file_no + n_files); |
358 | + |
359 | + file_handle = os_file_create(innodb_file_log_key, |
360 | +@@ -2454,6 +2512,7 @@ |
361 | + MONITOR_INC(MONITOR_LOG_IO); |
362 | + |
363 | + fil_io(OS_FILE_WRITE | OS_FILE_LOG, FALSE, group->archive_space_id, |
364 | ++ 0, |
365 | + (ulint) (next_offset / UNIV_PAGE_SIZE), |
366 | + (ulint) (next_offset % UNIV_PAGE_SIZE), |
367 | + ut_calc_align(len, OS_FILE_LOG_BLOCK_SIZE), buf, |
368 | +@@ -2825,7 +2884,6 @@ |
369 | + trunc_len); |
370 | + if (increment_file_count) { |
371 | + group->archived_offset = 0; |
372 | +- group->archived_file_no += 2; |
373 | + } |
374 | + |
375 | + #ifdef UNIV_DEBUG |
376 | +@@ -3090,7 +3148,6 @@ |
377 | + /*=======================================*/ |
378 | + { |
379 | + lsn_t lsn; |
380 | +- ulint arch_log_no; |
381 | + ulint count = 0; |
382 | + ulint total_trx; |
383 | + ulint pending_io; |
384 | +@@ -3307,15 +3364,7 @@ |
385 | + goto loop; |
386 | + } |
387 | + |
388 | +- arch_log_no = 0; |
389 | +- |
390 | + #ifdef UNIV_LOG_ARCHIVE |
391 | +- UT_LIST_GET_FIRST(log_sys->log_groups)->archived_file_no; |
392 | +- |
393 | +- if (0 == UT_LIST_GET_FIRST(log_sys->log_groups)->archived_offset) { |
394 | +- |
395 | +- arch_log_no--; |
396 | +- } |
397 | + |
398 | + log_archive_close_groups(TRUE); |
399 | + #endif /* UNIV_LOG_ARCHIVE */ |
400 | +@@ -3374,7 +3423,16 @@ |
401 | + srv_shutdown_lsn = lsn; |
402 | + |
403 | + if (!srv_read_only_mode) { |
404 | +- fil_write_flushed_lsn_to_data_files(lsn, arch_log_no); |
405 | ++ /* |
406 | ++ log_sys->lsn is aligned up to the log block size |
407 | ++ in recv_reset_logs(), but if archived logs are applied |
408 | ++ data files must contain exactly the same flushed_lsn |
409 | ++ on which applying was finished |
410 | ++ */ |
411 | ++ fil_write_flushed_lsn_to_data_files( |
412 | ++ srv_archive_recovery ? recv_sys->recovered_lsn : |
413 | ++ lsn, |
414 | ++ 0); |
415 | + |
416 | + fil_flush_file_spaces(FIL_TABLESPACE); |
417 | + } |
418 | +@@ -3597,7 +3655,7 @@ |
419 | + |
420 | + #ifdef UNIV_LOG_ARCHIVE |
421 | + rw_lock_free(&log_sys->archive_lock); |
422 | +- os_event_create(); |
423 | ++ os_event_free(log_sys->archiving_on); |
424 | + #endif /* UNIV_LOG_ARCHIVE */ |
425 | + |
426 | + #ifdef UNIV_LOG_DEBUG |
427 | + |
428 | +--- a/storage/innobase/log/log0recv.cc |
429 | ++++ b/storage/innobase/log/log0recv.cc |
430 | +@@ -53,6 +53,13 @@ |
431 | + //#else /* !UNIV_HOTBACKUP */ |
432 | + #include "xb0xb.h" |
433 | + |
434 | ++/** The size of archived log file */ |
435 | ++extern ib_uint64_t xtrabackup_arch_file_size; |
436 | ++/** The minimal LSN of found archived log files */ |
437 | ++extern ib_uint64_t xtrabackup_arch_first_file_lsn; |
438 | ++/** The maximum LSN of found archived log files */ |
439 | ++extern ib_uint64_t xtrabackup_arch_last_file_lsn; |
440 | ++ |
441 | + /** This is set to FALSE if the backup was originally taken with the |
442 | + ibbackup --include regexp option: then we do not want to create tables in |
443 | + directories which were not included */ |
444 | +@@ -2866,7 +2873,10 @@ |
445 | + = log_block_get_checkpoint_no(log_block); |
446 | + } |
447 | + |
448 | +- if (data_len < OS_FILE_LOG_BLOCK_SIZE) { |
449 | ++ /* Stop scanning if scanned lsn reaches lsn limit */ |
450 | ++ if (data_len < OS_FILE_LOG_BLOCK_SIZE || |
451 | ++ (recv_sys->limit_lsn && |
452 | ++ recv_sys->scanned_lsn >= recv_sys->limit_lsn)) { |
453 | + /* Log data for this group ends here */ |
454 | + |
455 | + finished = TRUE; |
456 | +@@ -3157,10 +3167,14 @@ |
457 | + group = UT_LIST_GET_FIRST(log_sys->log_groups); |
458 | + |
459 | + while (group) { |
460 | ++ /* There is no need to read archived_file_no and |
461 | ++ archived_offset because they are calculated another |
462 | ++ way */ |
463 | ++ /* |
464 | + log_checkpoint_get_nth_group_info(buf, group->id, |
465 | + &(group->archived_file_no), |
466 | + &(group->archived_offset)); |
467 | +- |
468 | ++ */ |
469 | + group = UT_LIST_GET_NEXT(log_groups, group); |
470 | + } |
471 | + #endif /* UNIV_LOG_ARCHIVE */ |
472 | +@@ -3731,23 +3745,23 @@ |
473 | + ulint len; |
474 | + ibool ret; |
475 | + byte* buf; |
476 | +- os_offset_t read_offset; |
477 | ++ os_offset_t read_offset = 0; |
478 | + os_offset_t file_size; |
479 | + int input_char; |
480 | +- char name[10000]; |
481 | ++ char name[OS_FILE_MAX_PATH]; |
482 | + |
483 | +- ut_a(0); |
484 | + |
485 | + try_open_again: |
486 | + buf = log_sys->buf; |
487 | + |
488 | + /* Add the file to the archive file space; open the file */ |
489 | + |
490 | +- log_archived_file_name_gen(name, group->id, group->archived_file_no); |
491 | ++ log_archived_file_name_gen(name, sizeof(name), |
492 | ++ group->id, group->archived_file_no); |
493 | + |
494 | + file_handle = os_file_create(innodb_file_log_key, |
495 | + name, OS_FILE_OPEN, |
496 | +- OS_FILE_LOG, OS_FILE_AIO, &ret); |
497 | ++ OS_FILE_AIO, OS_LOG_FILE, &ret); |
498 | + |
499 | + if (ret == FALSE) { |
500 | + ask_again: |
501 | +@@ -3792,20 +3806,19 @@ |
502 | + |
503 | + /* Add the archive file as a node to the space */ |
504 | + |
505 | +- fil_node_create(name, 1 + file_size / UNIV_PAGE_SIZE, |
506 | +- group->archive_space_id, FALSE); |
507 | +-#if RECV_SCAN_SIZE < LOG_FILE_HDR_SIZE |
508 | +-# error "RECV_SCAN_SIZE < LOG_FILE_HDR_SIZE" |
509 | +-#endif |
510 | ++ ut_a(fil_node_create(name, 1 + file_size / UNIV_PAGE_SIZE, |
511 | ++ group->archive_space_id, FALSE)); |
512 | ++ ut_a(RECV_SCAN_SIZE >= LOG_FILE_HDR_SIZE); |
513 | + |
514 | + /* Read the archive file header */ |
515 | +- fil_io(OS_FILE_READ | OS_FILE_LOG, TRUE, group->archive_space_id, 0, 0, |
516 | ++ fil_io(OS_FILE_READ | OS_FILE_LOG, TRUE, group->archive_space_id, 0, |
517 | ++ 0, 0, |
518 | + LOG_FILE_HDR_SIZE, buf, NULL); |
519 | + |
520 | + /* Check if the archive file header is consistent */ |
521 | + |
522 | + if (mach_read_from_4(buf + LOG_GROUP_ID) != group->id |
523 | +- || mach_read_from_4(buf + LOG_FILE_NO) |
524 | ++ || mach_read_from_8(buf + LOG_FILE_START_LSN) |
525 | + != group->archived_file_no) { |
526 | + fprintf(stderr, |
527 | + "InnoDB: Archive file header inconsistent %s\n", name); |
528 | +@@ -3834,6 +3847,20 @@ |
529 | + return(TRUE); |
530 | + } |
531 | + |
532 | ++ /* |
533 | ++ if this is the first file of scanning set |
534 | ++ calculate offset of the first log record |
535 | ++ */ |
536 | ++ if (recv_sys->parse_start_lsn >= start_lsn && |
537 | ++ recv_sys->parse_start_lsn < file_end_lsn) { |
538 | ++ read_offset = ut_calc_align_down( |
539 | ++ recv_sys->parse_start_lsn - start_lsn, |
540 | ++ OS_FILE_LOG_BLOCK_SIZE); |
541 | ++ start_lsn = ut_calc_align_down( |
542 | ++ recv_sys->parse_start_lsn, |
543 | ++ OS_FILE_LOG_BLOCK_SIZE); |
544 | ++ } |
545 | ++ |
546 | + recv_sys->scanned_lsn = start_lsn; |
547 | + } |
548 | + |
549 | +@@ -3846,7 +3873,7 @@ |
550 | + return(TRUE); |
551 | + } |
552 | + |
553 | +- read_offset = LOG_FILE_HDR_SIZE; |
554 | ++ read_offset += LOG_FILE_HDR_SIZE; |
555 | + |
556 | + for (;;) { |
557 | + len = RECV_SCAN_SIZE; |
558 | +@@ -3872,7 +3899,7 @@ |
559 | + #endif /* UNIV_DEBUG */ |
560 | + |
561 | + fil_io(OS_FILE_READ | OS_FILE_LOG, TRUE, |
562 | +- group->archive_space_id, read_offset / UNIV_PAGE_SIZE, |
563 | ++ group->archive_space_id, 0, read_offset / UNIV_PAGE_SIZE, |
564 | + read_offset % UNIV_PAGE_SIZE, len, buf, NULL); |
565 | + |
566 | + ret = recv_scan_log_recs( |
567 | +@@ -3887,10 +3914,12 @@ |
568 | + } |
569 | + |
570 | + if (ret) { |
571 | +- fprintf(stderr, |
572 | +- "InnoDB: Archive log file %s" |
573 | +- " does not scan right\n", |
574 | +- name); |
575 | ++ if (recv_sys->scanned_lsn < recv_sys->limit_lsn) { |
576 | ++ fprintf(stderr, |
577 | ++ "InnoDB: Archive log file %s" |
578 | ++ " does not scan right\n", |
579 | ++ name); |
580 | ++ } |
581 | + return(TRUE); |
582 | + } |
583 | + |
584 | +@@ -3907,26 +3936,19 @@ |
585 | + Recovers from archived log files, and also from log files, if they exist. |
586 | + @return error code or DB_SUCCESS */ |
587 | + UNIV_INTERN |
588 | +-ulint |
589 | ++dberr_t |
590 | + recv_recovery_from_archive_start( |
591 | + /*=============================*/ |
592 | + ib_uint64_t min_flushed_lsn,/*!< in: min flushed lsn field from the |
593 | + data files */ |
594 | +- ib_uint64_t limit_lsn, /*!< in: recover up to this lsn if |
595 | ++ ib_uint64_t limit_lsn) /*!< in: recover up to this lsn if |
596 | + possible */ |
597 | +- ulint first_log_no) /*!< in: number of the first archived |
598 | +- log file to use in the recovery; the |
599 | +- file will be searched from |
600 | +- INNOBASE_LOG_ARCH_DIR specified in |
601 | +- server config file */ |
602 | + { |
603 | + log_group_t* group; |
604 | + ulint group_id; |
605 | + ulint trunc_len; |
606 | + ibool ret; |
607 | +- ulint err; |
608 | +- |
609 | +- ut_a(0); |
610 | ++ dberr_t err; |
611 | + |
612 | + recv_sys_create(); |
613 | + recv_sys_init(buf_pool_get_curr_size()); |
614 | +@@ -3956,7 +3978,7 @@ |
615 | + return(DB_ERROR); |
616 | + } |
617 | + |
618 | +- group->archived_file_no = first_log_no; |
619 | ++ group->archived_file_no = xtrabackup_arch_first_file_lsn; |
620 | + |
621 | + recv_sys->parse_start_lsn = min_flushed_lsn; |
622 | + |
623 | +@@ -3970,7 +3992,8 @@ |
624 | + |
625 | + mutex_enter(&(log_sys->mutex)); |
626 | + |
627 | +- while (!ret) { |
628 | ++ while (!ret && |
629 | ++ group->archived_file_no <= xtrabackup_arch_last_file_lsn) { |
630 | + ret = log_group_recover_from_archive_file(group); |
631 | + |
632 | + /* Close and truncate a possible processed archive file |
633 | +@@ -3983,9 +4006,9 @@ |
634 | + trunc_len); |
635 | + } |
636 | + |
637 | +- group->archived_file_no++; |
638 | ++ group->archived_file_no += group->file_size - LOG_FILE_HDR_SIZE; |
639 | + } |
640 | +- |
641 | ++/* |
642 | + if (recv_sys->recovered_lsn < limit_lsn) { |
643 | + |
644 | + if (!recv_sys->scanned_lsn) { |
645 | +@@ -4006,12 +4029,12 @@ |
646 | + |
647 | + mutex_enter(&(log_sys->mutex)); |
648 | + } |
649 | +- |
650 | ++*/ |
651 | + if (limit_lsn != IB_ULONGLONG_MAX) { |
652 | + |
653 | + recv_apply_hashed_log_recs(FALSE); |
654 | + |
655 | +- recv_reset_logs(0, FALSE, recv_sys->recovered_lsn); |
656 | ++ // recv_reset_logs(0, FALSE, recv_sys->recovered_lsn); |
657 | + } |
658 | + |
659 | + mutex_exit(&(log_sys->mutex)); |
660 | + |
661 | +--- a/storage/innobase/srv/srv0start.cc |
662 | ++++ b/storage/innobase/srv/srv0start.cc |
663 | +@@ -641,6 +641,12 @@ |
664 | + } |
665 | + } |
666 | + |
667 | ++#ifdef UNIV_LOG_ARCHIVE |
668 | ++ /* Create the file space object for archived logs. */ |
669 | ++ fil_space_create("arch_log_space", SRV_LOG_SPACE_FIRST_ID + 1, |
670 | ++ 0, FIL_LOG); |
671 | ++#endif |
672 | ++ |
673 | + log_group_init(0, srv_n_log_files, |
674 | + srv_log_file_size * UNIV_PAGE_SIZE, |
675 | + SRV_LOG_SPACE_FIRST_ID, |
676 | +@@ -651,7 +657,12 @@ |
677 | + /* Create a log checkpoint. */ |
678 | + mutex_enter(&log_sys->mutex); |
679 | + ut_d(recv_no_log_write = FALSE); |
680 | +- recv_reset_logs(lsn); |
681 | ++ recv_reset_logs( |
682 | ++#ifdef UNIV_LOG_ARCHIVE |
683 | ++ UT_LIST_GET_FIRST(log_sys->log_groups)->archived_file_no, |
684 | ++ TRUE, |
685 | ++#endif |
686 | ++ lsn); |
687 | + mutex_exit(&log_sys->mutex); |
688 | + |
689 | + return(DB_SUCCESS); |
690 | +@@ -738,12 +749,6 @@ |
691 | + /*======================*/ |
692 | + ibool* create_new_db, /*!< out: TRUE if new database should be |
693 | + created */ |
694 | +-#ifdef UNIV_LOG_ARCHIVE |
695 | +- ulint* min_arch_log_no,/*!< out: min of archived log |
696 | +- numbers in data files */ |
697 | +- ulint* max_arch_log_no,/*!< out: max of archived log |
698 | +- numbers in data files */ |
699 | +-#endif /* UNIV_LOG_ARCHIVE */ |
700 | + lsn_t* min_flushed_lsn,/*!< out: min of flushed lsn |
701 | + values in data files */ |
702 | + lsn_t* max_flushed_lsn,/*!< out: max of flushed lsn |
703 | +@@ -964,9 +969,6 @@ |
704 | + skip_size_check: |
705 | + fil_read_first_page( |
706 | + files[i], one_opened, &flags, &space, |
707 | +-#ifdef UNIV_LOG_ARCHIVE |
708 | +- min_arch_log_no, max_arch_log_no, |
709 | +-#endif /* UNIV_LOG_ARCHIVE */ |
710 | + min_flushed_lsn, max_flushed_lsn); |
711 | + |
712 | + /* The first file of the system tablespace must |
713 | +@@ -1450,8 +1452,8 @@ |
714 | + lsn_t min_flushed_lsn; |
715 | + lsn_t max_flushed_lsn; |
716 | + #ifdef UNIV_LOG_ARCHIVE |
717 | +- ulint min_arch_log_no; |
718 | +- ulint max_arch_log_no; |
719 | ++ lsn_t min_arch_log_no; |
720 | ++ lsn_t max_arch_log_no; |
721 | + #endif /* UNIV_LOG_ARCHIVE */ |
722 | + ulint sum_of_new_sizes; |
723 | + ulint sum_of_data_file_sizes; |
724 | +@@ -1890,17 +1892,6 @@ |
725 | + os_thread_create(io_handler_thread, n + i, thread_ids + i); |
726 | + } |
727 | + |
728 | +-#ifdef UNIV_LOG_ARCHIVE |
729 | +- if (0 != ut_strcmp(srv_log_group_home_dir, srv_arch_dir)) { |
730 | +- ut_print_timestamp(stderr); |
731 | +- fprintf(stderr, " InnoDB: Error: you must set the log group home dir in my.cnf\n"); |
732 | +- ut_print_timestamp(stderr); |
733 | +- fprintf(stderr, " InnoDB: the same as log arch dir.\n"); |
734 | +- |
735 | +- return(DB_ERROR); |
736 | +- } |
737 | +-#endif /* UNIV_LOG_ARCHIVE */ |
738 | +- |
739 | + if (srv_n_log_files * srv_log_file_size * UNIV_PAGE_SIZE |
740 | + >= 512ULL * 1024ULL * 1024ULL * 1024ULL) { |
741 | + /* log_block_convert_lsn_to_no() limits the returned block |
742 | +@@ -1959,9 +1950,6 @@ |
743 | + } |
744 | + |
745 | + err = open_or_create_data_files(&create_new_db, |
746 | +-#ifdef UNIV_LOG_ARCHIVE |
747 | +- &min_arch_log_no, &max_arch_log_no, |
748 | +-#endif /* UNIV_LOG_ARCHIVE */ |
749 | + &min_flushed_lsn, &max_flushed_lsn, |
750 | + &sum_of_new_sizes); |
751 | + if (err == DB_FAIL) { |
752 | +@@ -1989,7 +1977,6 @@ |
753 | + |
754 | + #ifdef UNIV_LOG_ARCHIVE |
755 | + srv_normalize_path_for_win(srv_arch_dir); |
756 | +- srv_arch_dir = srv_add_path_separator_if_needed(srv_arch_dir); |
757 | + #endif /* UNIV_LOG_ARCHIVE */ |
758 | + |
759 | + dirnamelen = strlen(srv_log_group_home_dir); |
760 | +@@ -2068,9 +2055,15 @@ |
761 | + max_flushed_lsn, logfile0); |
762 | + |
763 | + /* Suppress the message about |
764 | +- crash recovery. */ |
765 | +- max_flushed_lsn = min_flushed_lsn |
766 | +- = log_get_lsn(); |
767 | ++ crash recovery. If archive recovery |
768 | ++ is enabled the min/max_flushed_lsn |
769 | ++ must point to the position from |
770 | ++ which applying is started. */ |
771 | ++ if (!srv_archive_recovery) { |
772 | ++ max_flushed_lsn = min_flushed_lsn |
773 | ++ = log_get_lsn(); |
774 | ++ } |
775 | ++ |
776 | + goto files_checked; |
777 | + #if 0 |
778 | + } else if (i < 2) { |
779 | +@@ -2244,10 +2237,13 @@ |
780 | + |
781 | + ib_logf(IB_LOG_LEVEL_INFO, |
782 | + " Starting archive recovery from a backup..."); |
783 | ++ /* Load table spaces before recovery as during recovery |
784 | ++ there can be log records that are applied to the spaces |
785 | ++ with unknown id's */ |
786 | ++ fil_load_single_table_tablespaces(NULL); |
787 | + |
788 | + err = recv_recovery_from_archive_start( |
789 | +- min_flushed_lsn, srv_archive_recovery_limit_lsn, |
790 | +- min_arch_log_no); |
791 | ++ min_flushed_lsn, srv_archive_recovery_limit_lsn); |
792 | + if (err != DB_SUCCESS) { |
793 | + |
794 | + return(DB_ERROR); |
795 | +@@ -2271,6 +2267,11 @@ |
796 | + srv_startup_is_before_trx_rollback_phase = FALSE; |
797 | + |
798 | + recv_recovery_from_archive_finish(); |
799 | ++ |
800 | ++ if (srv_apply_log_only) { |
801 | ++ goto skip_processes; |
802 | ++ } |
803 | ++ |
804 | + #endif /* UNIV_LOG_ARCHIVE */ |
805 | + } else { |
806 | + |
807 | +@@ -2478,6 +2479,8 @@ |
808 | + if (!srv_log_archive_on) { |
809 | + ut_a(DB_SUCCESS == log_archive_noarchivelog()); |
810 | + } else { |
811 | ++ bool start_archive; |
812 | ++ |
813 | + mutex_enter(&(log_sys->mutex)); |
814 | + |
815 | + start_archive = FALSE; |
816 | + |
817 | +--- a/storage/innobase/trx/trx0sys.cc |
818 | ++++ b/storage/innobase/trx/trx0sys.cc |
819 | +@@ -1185,7 +1185,8 @@ |
820 | + trx_purge_sys_close(); |
821 | + |
822 | + /* Free the double write data structures. */ |
823 | +- buf_dblwr_free(); |
824 | ++ if (buf_dblwr) |
825 | ++ buf_dblwr_free(); |
826 | + |
827 | + mutex_enter(&trx_sys->mutex); |
828 | + |
829 | + |
830 | |
831 | === modified file 'src/xtrabackup.cc' |
832 | --- src/xtrabackup.cc 2013-08-20 17:43:45 +0000 |
833 | +++ src/xtrabackup.cc 2013-08-26 07:08:27 +0000 |
834 | @@ -118,6 +118,8 @@ |
835 | char *xtrabackup_extra_lsndir = NULL; /* for --backup with --extra-lsndir */ |
836 | char *xtrabackup_incremental_dir = NULL; /* for --prepare */ |
837 | |
838 | +lsn_t xtrabackup_archived_to_lsn = 0; /* for --archived-to-lsn */ |
839 | + |
840 | char *xtrabackup_tables = NULL; |
841 | int tables_regex_num; |
842 | xb_regex_t *tables_regex; |
843 | @@ -279,6 +281,17 @@ |
844 | |
845 | static my_bool xtrabackup_incremental_force_scan = FALSE; |
846 | |
847 | +/* The flushed lsn which is read from data files */ |
848 | +lsn_t min_flushed_lsn= 0; |
849 | +lsn_t max_flushed_lsn= 0; |
850 | + |
851 | +/* The size of archived log file */ |
852 | +size_t xtrabackup_arch_file_size = 0ULL; |
853 | +/* The minimal LSN of found archived log files */ |
854 | +lsn_t xtrabackup_arch_first_file_lsn = 0ULL; |
855 | +/* The maximum LSN of found archived log files */ |
856 | +lsn_t xtrabackup_arch_last_file_lsn = 0ULL; |
857 | + |
858 | /* Datasinks */ |
859 | ds_ctxt_t *ds_data = NULL; |
860 | ds_ctxt_t *ds_meta = NULL; |
861 | @@ -388,6 +401,7 @@ |
862 | OPT_XTRA_INCREMENTAL_BASEDIR, |
863 | OPT_XTRA_EXTRA_LSNDIR, |
864 | OPT_XTRA_INCREMENTAL_DIR, |
865 | + OPT_XTRA_ARCHIVED_TO_LSN, |
866 | OPT_XTRA_TABLES, |
867 | OPT_XTRA_TABLES_FILE, |
868 | OPT_XTRA_CREATE_IB_LOGFILE, |
869 | @@ -551,6 +565,12 @@ |
870 | {"incremental-dir", OPT_XTRA_INCREMENTAL_DIR, "(for --prepare): apply .delta files and logfile in the specified directory.", |
871 | (G_PTR*) &xtrabackup_incremental_dir, (G_PTR*) &xtrabackup_incremental_dir, |
872 | 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0}, |
873 | +#ifdef UNIV_LOG_ARCHIVE |
874 | + {"to-archived-lsn", OPT_XTRA_ARCHIVED_TO_LSN, |
875 | + "Don't apply archived logs with bigger log sequence number.", |
876 | + (G_PTR*) &xtrabackup_archived_to_lsn, (G_PTR*) &xtrabackup_archived_to_lsn, 0, |
877 | + GET_LL, REQUIRED_ARG, 0, 0, LONGLONG_MAX, 0, 0, 0}, |
878 | +#endif /* UNIV_LOG_ARCHIVE */ |
879 | {"tables", OPT_XTRA_TABLES, "filtering by regexp for table names.", |
880 | (G_PTR*) &xtrabackup_tables, (G_PTR*) &xtrabackup_tables, |
881 | 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0}, |
882 | @@ -733,11 +753,11 @@ |
883 | (G_PTR*) &innobase_locks_unsafe_for_binlog, |
884 | (G_PTR*) &innobase_locks_unsafe_for_binlog, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, |
885 | */ |
886 | -/* |
887 | +#ifdef UNIV_LOG_ARCHIVE |
888 | {"innodb_log_arch_dir", OPT_INNODB_LOG_ARCH_DIR, |
889 | "Where full logs should be archived.", (G_PTR*) &innobase_log_arch_dir, |
890 | (G_PTR*) &innobase_log_arch_dir, 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0}, |
891 | -*/ |
892 | +#endif /* UNIV_LOG_ARCHIVE */ |
893 | {"innodb_log_buffer_size", OPT_INNODB_LOG_BUFFER_SIZE, |
894 | "The size of the buffer which InnoDB uses to write log to the log files on disk.", |
895 | (G_PTR*) &innobase_log_buffer_size, (G_PTR*) &innobase_log_buffer_size, 0, |
896 | @@ -2573,8 +2593,6 @@ |
897 | ibool create_new_doublewrite_file; |
898 | #endif |
899 | ulint err; |
900 | - lsn_t min_flushed_lsn; |
901 | - lsn_t max_flushed_lsn; |
902 | ulint sum_of_new_sizes; |
903 | |
904 | for (i = 0; i < srv_n_file_io_threads; i++) { |
905 | @@ -5146,6 +5164,76 @@ |
906 | |
907 | } |
908 | |
909 | +#ifdef UNIV_LOG_ARCHIVE |
910 | +/********************************************************************//** |
911 | +Searches archived log files in archived log directory. The min and max |
912 | +LSN's of found files as well as archived log file size are stored in |
913 | +xtrabackup_arch_first_file_lsn, xtrabackup_arch_last_file_lsn and |
914 | +xtrabackup_arch_file_size respectively. |
915 | +@return true on success |
916 | +*/ |
917 | +static |
918 | +bool |
919 | +xtrabackup_arch_search_files( |
920 | +/*=========================*/ |
921 | + ib_uint64_t start_lsn) /*!< in: filter out log files |
922 | + witch does not contain data |
923 | + with lsn < start_lsn */ |
924 | +{ |
925 | + os_file_dir_t dir; |
926 | + os_file_stat_t fileinfo; |
927 | + ut_ad(innobase_log_arch_dir); |
928 | + |
929 | + dir = os_file_opendir(innobase_log_arch_dir, FALSE); |
930 | + if (!dir) { |
931 | + msg("xtrabackup: error: cannot open archived log directory %s\n", |
932 | + innobase_log_arch_dir); |
933 | + return false; |
934 | + } |
935 | + |
936 | + while(!os_file_readdir_next_file(innobase_log_arch_dir, |
937 | + dir, |
938 | + &fileinfo) ) { |
939 | + lsn_t log_file_lsn; |
940 | + char* log_str_end_lsn_ptr; |
941 | + |
942 | + if (strncmp(fileinfo.name, |
943 | + IB_ARCHIVED_LOGS_PREFIX, |
944 | + sizeof(IB_ARCHIVED_LOGS_PREFIX) - 1)) { |
945 | + continue; |
946 | + } |
947 | + |
948 | + log_file_lsn = strtoll(fileinfo.name + |
949 | + sizeof(IB_ARCHIVED_LOGS_PREFIX) - 1, |
950 | + &log_str_end_lsn_ptr, 10); |
951 | + |
952 | + if (*log_str_end_lsn_ptr) { |
953 | + continue; |
954 | + } |
955 | + |
956 | + if (log_file_lsn + (fileinfo.size - LOG_FILE_HDR_SIZE) < start_lsn) { |
957 | + continue; |
958 | + } |
959 | + |
960 | + if (!xtrabackup_arch_first_file_lsn || |
961 | + log_file_lsn < xtrabackup_arch_first_file_lsn) { |
962 | + xtrabackup_arch_first_file_lsn = log_file_lsn; |
963 | + } |
964 | + if (log_file_lsn > xtrabackup_arch_last_file_lsn) { |
965 | + xtrabackup_arch_last_file_lsn = log_file_lsn; |
966 | + } |
967 | + |
968 | + //TODO: find the more suitable way to extract archived log file |
969 | + //size |
970 | + if (fileinfo.size > (ib_int64_t)xtrabackup_arch_file_size) { |
971 | + xtrabackup_arch_file_size = fileinfo.size; |
972 | + } |
973 | + } |
974 | + |
975 | + return xtrabackup_arch_first_file_lsn != 0; |
976 | +} |
977 | +#endif /* UNIV_LOG_ARCHIVE */ |
978 | + |
979 | static void |
980 | xtrabackup_prepare_func(void) |
981 | { |
982 | @@ -5169,14 +5257,18 @@ |
983 | xtrabackup_target_dir[0]=FN_CURLIB; // all paths are relative from here |
984 | xtrabackup_target_dir[1]=0; |
985 | |
986 | - /* read metadata of target */ |
987 | + /* |
988 | + read metadata of target, we don't need metadata reading in the case |
989 | + archived logs applying |
990 | + */ |
991 | + sprintf(metadata_path, "%s/%s", xtrabackup_target_dir, |
992 | + XTRABACKUP_METADATA_FILENAME); |
993 | + |
994 | + if (!xtrabackup_read_metadata(metadata_path)) |
995 | + msg("xtrabackup: error: xtrabackup_read_metadata()\n"); |
996 | + |
997 | + if (!innobase_log_arch_dir) |
998 | { |
999 | - sprintf(metadata_path, "%s/%s", xtrabackup_target_dir, |
1000 | - XTRABACKUP_METADATA_FILENAME); |
1001 | - |
1002 | - if (!xtrabackup_read_metadata(metadata_path)) |
1003 | - msg("xtrabackup: error: xtrabackup_read_metadata()\n"); |
1004 | - |
1005 | if (!strcmp(metadata_type, "full-backuped")) { |
1006 | msg("xtrabackup: This target seems to be not prepared " |
1007 | "yet.\n"); |
1008 | @@ -5220,7 +5312,7 @@ |
1009 | mem_init(srv_mem_pool_size); |
1010 | ut_crc32_init(); |
1011 | |
1012 | - if(xtrabackup_init_temp_log()) |
1013 | + if(!innobase_log_arch_dir && xtrabackup_init_temp_log()) |
1014 | goto error; |
1015 | |
1016 | if(innodb_init_param()) { |
1017 | @@ -5248,14 +5340,15 @@ |
1018 | |
1019 | xb_normalize_init_values(); |
1020 | |
1021 | - if (xtrabackup_incremental) { |
1022 | + if (xtrabackup_incremental || innobase_log_arch_dir) { |
1023 | err = xb_data_files_init(); |
1024 | if (err != DB_SUCCESS) { |
1025 | msg("xtrabackup: error: xb_data_files_init() failed " |
1026 | "with error code %lu\n", err); |
1027 | goto error; |
1028 | } |
1029 | - |
1030 | + } |
1031 | + if (xtrabackup_incremental) { |
1032 | inc_dir_tables_hash = hash_create(1000); |
1033 | |
1034 | if(!xtrabackup_apply_deltas()) { |
1035 | @@ -5263,9 +5356,11 @@ |
1036 | xb_tables_hash_free(inc_dir_tables_hash); |
1037 | goto error; |
1038 | } |
1039 | - |
1040 | + } |
1041 | + if (xtrabackup_incremental || innobase_log_arch_dir) { |
1042 | xb_data_files_close(); |
1043 | - |
1044 | + } |
1045 | + if (xtrabackup_incremental) { |
1046 | /* Cleanup datadir from tablespaces deleted between full and |
1047 | incremental backups */ |
1048 | |
1049 | @@ -5295,6 +5390,55 @@ |
1050 | srv_n_write_io_threads = 4; |
1051 | } |
1052 | |
1053 | +#ifdef UNIV_LOG_ARCHIVE |
1054 | + if (innobase_log_arch_dir) { |
1055 | + srv_arch_dir = innobase_log_arch_dir; |
1056 | + srv_archive_recovery = TRUE; |
1057 | + if (xtrabackup_archived_to_lsn) { |
1058 | + if (xtrabackup_archived_to_lsn < metadata_last_lsn) { |
1059 | + msg("xtrabackup: warning: logs applying lsn " |
1060 | + "limit " UINT64PF " is " |
1061 | + "less than metadata last-lsn " UINT64PF |
1062 | + " and will be set to metadata last-lsn value\n", |
1063 | + xtrabackup_archived_to_lsn, |
1064 | + metadata_last_lsn); |
1065 | + xtrabackup_archived_to_lsn = metadata_last_lsn; |
1066 | + } |
1067 | + if (xtrabackup_archived_to_lsn < min_flushed_lsn) { |
1068 | + msg("xtrabackup: error: logs applying " |
1069 | + "lsn limit " UINT64PF " is less than " |
1070 | + "min_flushed_lsn " UINT64PF |
1071 | + ", there is nothing to do\n", |
1072 | + xtrabackup_archived_to_lsn, |
1073 | + min_flushed_lsn); |
1074 | + goto error; |
1075 | + } |
1076 | + } |
1077 | + srv_archive_recovery_limit_lsn= xtrabackup_archived_to_lsn; |
1078 | + /* |
1079 | + Unfinished transactions are not rolled back during log applying |
1080 | + as they can be finished at the firther files applyings. |
1081 | + */ |
1082 | + srv_apply_log_only = TRUE; |
1083 | + |
1084 | + if (!xtrabackup_arch_search_files(min_flushed_lsn)) { |
1085 | + goto error; |
1086 | + } |
1087 | + |
1088 | + /* |
1089 | + Check if last log file last lsn is big enough to overlap |
1090 | + last scanned lsn read from metadata. |
1091 | + */ |
1092 | + if (xtrabackup_arch_last_file_lsn + |
1093 | + xtrabackup_arch_file_size - |
1094 | + LOG_FILE_HDR_SIZE < metadata_last_lsn) { |
1095 | + msg("xtrabackup: error: there are no enough archived logs " |
1096 | + "to apply\n"); |
1097 | + goto error; |
1098 | + } |
1099 | + } |
1100 | +#endif /* UNIV_LOG_ARCHIVE */ |
1101 | + |
1102 | msg("xtrabackup: Starting InnoDB instance for recovery.\n" |
1103 | "xtrabackup: Using %lld bytes for buffer pool " |
1104 | "(set by --use-memory parameter)\n", xtrabackup_use_memory); |
1105 | @@ -5517,7 +5661,10 @@ |
1106 | "'xtrabackup_binlog_pos_innodb'\n"); |
1107 | } |
1108 | } |
1109 | - |
1110 | +#ifdef UNIV_LOG_ARCHIVE |
1111 | + if (innobase_log_arch_dir) |
1112 | + srv_start_lsn = log_sys->lsn = recv_sys->recovered_lsn; |
1113 | +#endif /* UNIV_LOG_ARCHIVE */ |
1114 | /* Check whether the log is applied enough or not. */ |
1115 | if ((xtrabackup_incremental |
1116 | && ut_dulint_cmp(srv_start_lsn, incremental_last_lsn) < 0) |
1117 | @@ -5829,6 +5976,14 @@ |
1118 | exit(EXIT_FAILURE); |
1119 | } |
1120 | |
1121 | + if (!xtrabackup_prepare && |
1122 | + (innobase_log_arch_dir || xtrabackup_archived_to_lsn)) { |
1123 | + msg("xtrabackup: error: " |
1124 | + "--archived-logs-dir and --to-archived-lsn can be used " |
1125 | + "only with --prepare\n"); |
1126 | + exit(EXIT_FAILURE); |
1127 | + } |
1128 | + |
1129 | /* cannot execute both for now */ |
1130 | { |
1131 | int num = 0; |
1132 | |
1133 | === added file 'test/t/xb_apply_archived_logs.sh' |
1134 | --- test/t/xb_apply_archived_logs.sh 1970-01-01 00:00:00 +0000 |
1135 | +++ test/t/xb_apply_archived_logs.sh 2013-08-26 07:08:27 +0000 |
1136 | @@ -0,0 +1,389 @@ |
1137 | +. inc/common.sh |
1138 | + |
1139 | +#The result to return |
1140 | +RESULT=0 |
1141 | + |
1142 | +function repeat_until_new_arch_log_created |
1143 | +{ |
1144 | + local arch_log_dir=$1 |
1145 | + local command=$2 |
1146 | + local stop_lsn=`run_cmd $MYSQL $MYSQL_ARGS test -e 'SHOW ENGINE INNODB STATUS\G'|grep 'Log sequence number'|awk '{print $4}'` |
1147 | + local old_arch_logs_count=`ls -al $arch_log_dir/| wc -l` |
1148 | + local new_arch_logs_count=0 |
1149 | + |
1150 | + echo $old_arch_logs_count |
1151 | + # To be sure the data was flushed to archived log wait until new file |
1152 | + # with START_LSN > CURRENT_LSN is created |
1153 | + local max_lsn=0; |
1154 | + while [ $max_lsn -le $stop_lsn ]; |
1155 | + do |
1156 | + $command |
1157 | + for i in `ls -1 $arch_log_dir`; |
1158 | + do |
1159 | + local lsn=`echo $i | sed -s 's/ib_log_archive_\([0-9]\+\)/\1/'` |
1160 | + if [ $lsn -gt $max_lsn ]; |
1161 | + then |
1162 | + max_lsn=$lsn |
1163 | + fi |
1164 | + done |
1165 | + done |
1166 | +} |
1167 | + |
1168 | +function check_if_equal |
1169 | +{ |
1170 | + local NAME=$1 |
1171 | + local VAL1=$2 |
1172 | + local VAL2=$3 |
1173 | + local MSG="$NAME ($VAL1 == $VAL2):" |
1174 | + if [ $VAL1 -ne $VAL2 ] |
1175 | + then |
1176 | + vlog "$MSG failed" |
1177 | + RESULT=-1 |
1178 | + else |
1179 | + vlog "$MSG passed" |
1180 | + fi |
1181 | +} |
1182 | + |
1183 | +function check_if_not_equal |
1184 | +{ |
1185 | + local NAME=$1 |
1186 | + local VAL1=$2 |
1187 | + local VAL2=$3 |
1188 | + local MSG="$NAME ($VAL1 != $VAL2):" |
1189 | + if [ $VAL1 -eq $VAL2 ] |
1190 | + then |
1191 | + vlog "$MSG failed" |
1192 | + RESULT=-1 |
1193 | + else |
1194 | + vlog "$MSG passed" |
1195 | + fi |
1196 | +} |
1197 | + |
1198 | +function fill_tables |
1199 | +{ |
1200 | + local TABLES_COUNT=$1 |
1201 | + local TABLE_NAME=$2 |
1202 | + |
1203 | + for i in `seq 1 $TABLES_COUNT`; do |
1204 | + local TN=$TABLE_NAME$i |
1205 | + run_cmd $MYSQL $MYSQL_ARGS test \ |
1206 | +<<EOF |
1207 | + INSERT INTO $TN (B) |
1208 | + SELECT ${TN}_1.B FROM |
1209 | + $TN ${TN}_1, |
1210 | + $TN ${TN}_2, |
1211 | + $TN ${TN}_3 |
1212 | + LIMIT 10000; |
1213 | + |
1214 | +EOF |
1215 | + done |
1216 | +} |
1217 | + |
1218 | +function create_and_fill_db |
1219 | +{ |
1220 | + local TABLES_COUNT=$1 |
1221 | + local TABLE_NAME=$2 |
1222 | + local ARCH_LOG_DIR=$3 |
1223 | + local CREATE_TABLE_OPTIONS=${4:-''} |
1224 | + |
1225 | + for i in `seq 1 $TABLES_COUNT`; do |
1226 | + local TN=$TABLE_NAME$i |
1227 | + |
1228 | + run_cmd $MYSQL $MYSQL_ARGS test \ |
1229 | +<<EOF |
1230 | + CREATE TABLE $TN |
1231 | + (A INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, |
1232 | + B INT(10) UNSIGNED NOT NULL DEFAULT 0) ENGINE=INNODB $CREATE_TABLE_OPTIONS; |
1233 | + INSERT INTO $TN (B) VALUES (1),(1),(1),(1),(1),(1),(1),(1),(1),(1), |
1234 | + (1),(1),(1),(1),(1),(1),(1),(1),(1),(1), |
1235 | + (1),(1),(1),(1),(1),(1),(1),(1),(1),(1), |
1236 | + (1),(1),(1),(1),(1),(1),(1),(1),(1),(1), |
1237 | + (1),(1),(1),(1),(1),(1),(1),(1),(1),(1), |
1238 | + (1),(1),(1),(1),(1),(1),(1),(1),(1),(1), |
1239 | + (1),(1),(1),(1),(1),(1),(1),(1),(1),(1), |
1240 | + (1),(1),(1),(1),(1),(1),(1),(1),(1),(1), |
1241 | + (1),(1),(1),(1),(1),(1),(1),(1),(1),(1), |
1242 | + (1),(1),(1),(1),(1),(1),(1),(1),(1),(1); |
1243 | +EOF |
1244 | + done |
1245 | + |
1246 | + repeat_until_new_arch_log_created \ |
1247 | + $ARCH_LOG_DIR \ |
1248 | + "fill_tables $TABLES_COUNT $TABLE_NAME" |
1249 | +} |
1250 | + |
1251 | +function make_changes |
1252 | +{ |
1253 | + local START_I=$1 |
1254 | + local TABLES_COUNT=$2 |
1255 | + local TABLE_NAME=$3 |
1256 | + local CLIENT_CMD="${MYSQL} ${MYSQL_ARGS} -Ns test -e" |
1257 | + |
1258 | + for i in `seq $START_I $(($START_I+$TABLES_COUNT-1))`; do |
1259 | + local TN=$TABLE_NAME$(($i-$START_I+1)) |
1260 | + |
1261 | + #INSERT |
1262 | + I_FROM[$i]=`$CLIENT_CMD "select max(a) from $TN"` |
1263 | + $CLIENT_CMD "insert into $TN (b) select b from $TN limit 1000" |
1264 | + I_TO[$i]=`$CLIENT_CMD "select max(a) from $TN"` |
1265 | + vlog "Inserted rows for $i are in the range [${I_FROM[$i]}, ${I_TO[$i]})" |
1266 | + I_COUNT[$i]=`$CLIENT_CMD "select count(*) from $TN where a >= ${I_FROM[$i]} and a < ${I_TO[$i]} "` |
1267 | + |
1268 | + #DELETE |
1269 | + if [ $i -gt $TABLES_COUNT ]; |
1270 | + then |
1271 | + local START_ROW=${U_TO[$(($i-$TABLES_COUNT))]} |
1272 | + else |
1273 | + local START_ROW=1000 |
1274 | + fi |
1275 | + D_FROM[$i]=`$CLIENT_CMD "select min(a) from (select a from $TN where a >= $START_ROW limit 1000) as temp_table"` |
1276 | + D_TO[$i]=`$CLIENT_CMD "select max(a) from (select a from $TN where a >= $START_ROW limit 1000) as temp_table"` |
1277 | + D_COUNT[$i]=`$CLIENT_CMD "select count(*) from $TN where a>=${D_FROM[$i]} and a<${D_TO[$i]}"` |
1278 | + $CLIENT_CMD "delete from $TN where a >= ${D_FROM[$i]} and a < ${D_TO[$i]}" |
1279 | + vlog "Deleted rows for $i are in the range [${D_FROM[$i]}, ${D_TO[$i]}), total ${D_COUNT[$i]} are deleted" |
1280 | + |
1281 | + #UPDATE |
1282 | + U_FROM[$i]=${D_TO[$i]} |
1283 | + U_TO[$i]=`$CLIENT_CMD "select max(a) from (select a from $TN where a >= ${U_FROM[$i]} limit 1000) as temp_table"` |
1284 | + U_COUNT[$i]=`$CLIENT_CMD "select count(*) from $TN where a>=${U_FROM[$i]} and a<${U_TO[$i]}"` |
1285 | + $CLIENT_CMD "update $TN set b=2 where a >= ${U_FROM[$i]} and a < ${U_TO[$i]}" |
1286 | + vlog "Updated rows for $i are in the range [${U_FROM[$i]}, ${U_TO[$i]}), total ${U_COUNT[$i]} are updated" |
1287 | + |
1288 | + done |
1289 | +} |
1290 | + |
1291 | +function get_changes |
1292 | +{ |
1293 | + local START_I=$1 |
1294 | + local TABLES_COUNT=$2 |
1295 | + local TABLE_NAME=$3 |
1296 | + local CLIENT_CMD="${MYSQL} ${MYSQL_ARGS} -Ns test -e" |
1297 | + |
1298 | + for i in `seq $START_I $(($START_I+$TABLES_COUNT-1))`; do |
1299 | + local TN=$TABLE_NAME$(($i-$START_I+1)) |
1300 | + |
1301 | + #INSERT |
1302 | + I_COUNT1[$i]=`$CLIENT_CMD "select count(*) from $TN where a >= ${I_FROM[$i]} and a < ${I_TO[$i]} "` |
1303 | + #DELETE |
1304 | + D_COUNT1[$i]=`$CLIENT_CMD "select count(*) from $TN where a >= ${D_FROM[$i]} and a < ${D_TO[$i]} "` |
1305 | + #UPDATE |
1306 | + U_COUNT1[$i]=`$CLIENT_CMD "select count(*) from $TN where a >= ${U_FROM[$i]} and a < ${U_TO[$i]} and b = 2"` |
1307 | + done |
1308 | +} |
1309 | + |
1310 | +function check_changes |
1311 | +{ |
1312 | + local START_I=$1 |
1313 | + local TABLES_COUNT=$2 |
1314 | + local CMD=$3 |
1315 | + |
1316 | + for i in `seq $START_I $(($START_I+$TABLES_COUNT-1))`; do |
1317 | + $CMD "INSERT TEST for $i" ${I_COUNT[$i]} ${I_COUNT1[$i]} |
1318 | + $CMD "DELETE TEST for $i" 0 ${D_COUNT1[$i]} |
1319 | + $CMD "UPDATE TEST for $i" ${U_COUNT[$i]} ${U_COUNT1[$i]} |
1320 | + done |
1321 | +} |
1322 | + |
1323 | +function unset_global_variables |
1324 | +{ |
1325 | + unset I_FROM |
1326 | + unset I_TO |
1327 | + unset I_COUNT |
1328 | + |
1329 | + unset D_FROM |
1330 | + unset D_TO |
1331 | + unset D_COUNT |
1332 | + |
1333 | + unset U_FROM |
1334 | + unset U_TO |
1335 | + unset U_COUNT |
1336 | + |
1337 | + unset I_FROM1 |
1338 | + unset I_TO1 |
1339 | + unset I_COUNT1 |
1340 | + |
1341 | + unset D_FROM1 |
1342 | + unset D_TO1 |
1343 | + unset D_COUNT1 |
1344 | + |
1345 | + unset U_FROM1 |
1346 | + unset U_TO1 |
1347 | + unset U_COUNT1 |
1348 | +} |
1349 | + |
1350 | +function test_archived_logs |
1351 | +{ |
1352 | + local TABLE_NAME=T |
1353 | + local TABLES_COUNT=4 |
1354 | + local EXTRA_OPTIONS=${1:-''} |
1355 | + local CREATE_TABLE_OPTIONS=${2:-''} |
1356 | + |
1357 | + #Setup server environment to get access to some variables |
1358 | + init_server_variables 1 |
1359 | + switch_server 1 |
1360 | + local BASE_BACKUP_DIR=$topdir/backup_base |
1361 | + local BACKUP_DIR=$topdir/backup |
1362 | + local BASE_DATA_DIR=$topdir/base_data |
1363 | + local ARCHIVED_LOGS_DIR=$topdir/archived_logs |
1364 | + local XTRABACKUP_OPTIONS="--innodb_log_file_size=2M $EXTRA_OPTIONS" |
1365 | + #Setup ROW binlog format to supress warnings in result file |
1366 | + local SERVER_OPTIONS="$XTRABACKUP_OPTIONS --innodb_log_archive=ON --innodb_log_arch_dir=$ARCHIVED_LOGS_DIR --binlog-format=ROW" |
1367 | + mkdir -p $BASE_BACKUP_DIR $BACKUP_DIR |
1368 | + mkdir -p $ARCHIVED_LOGS_DIR |
1369 | + reset_server_variables 1 |
1370 | + |
1371 | + ################################################################### |
1372 | + # --to-lsn test. It checks the availability to apply logs only to # |
1373 | + # the certain LSN. # |
1374 | + ################################################################### |
1375 | + start_server $SERVER_OPTIONS |
1376 | + #Create and fill tables to generate log files |
1377 | + create_and_fill_db $TABLES_COUNT $TABLE_NAME $ARCHIVED_LOGS_DIR $CREATE_TABLE_OPTIONS |
1378 | + #Backup the data |
1379 | + xtrabackup --backup --datadir=$mysql_datadir --target-dir=$BASE_BACKUP_DIR $XTRABACKUP_OPTIONS |
1380 | + #Make some changes in tables after backup is done |
1381 | + make_changes 1 $TABLES_COUNT $TABLE_NAME |
1382 | + #Make sure that changes are flushed to archived log |
1383 | + repeat_until_new_arch_log_created \ |
1384 | + $ARCHIVED_LOGS_DIR \ |
1385 | + "fill_tables $TABLES_COUNT $TABLE_NAME" |
1386 | + #Remember current LSN |
1387 | + local LSN=`run_cmd $MYSQL $MYSQL_ARGS test -e 'SHOW ENGINE INNODB STATUS\G'|grep 'Log sequence number'|awk '{print $4}'` |
1388 | + #Make more changes over remembered LSN |
1389 | + make_changes $(($TABLES_COUNT+1)) $TABLES_COUNT $TABLE_NAME |
1390 | + #Make sure the above changes are flushed to archived log |
1391 | + repeat_until_new_arch_log_created \ |
1392 | + $ARCHIVED_LOGS_DIR \ |
1393 | + "fill_tables $TABLES_COUNT $TABLE_NAME" |
1394 | + stop_server |
1395 | + cp -R $mysql_datadir $BASE_DATA_DIR |
1396 | + |
1397 | + ######################################### |
1398 | + # Apply logs only to the remembered lsn # |
1399 | + ######################################### |
1400 | + # --apply-logs-only is set implicitly because unfinished transactions |
1401 | + # can be finished on further logs applying but using this option with |
1402 | + # --innodb-log-arch-dir is tested here to prove this bug |
1403 | + # https://bugs.launchpad.net/percona-xtrabackup/+bug/1199555 is not |
1404 | + # concerned with this case |
1405 | + cp -R $BASE_BACKUP_DIR/* $BACKUP_DIR |
1406 | + xtrabackup --prepare \ |
1407 | + --target-dir=$BACKUP_DIR \ |
1408 | + --innodb-log-arch-dir=$ARCHIVED_LOGS_DIR \ |
1409 | + --to-archived-lsn=$LSN \ |
1410 | + --apply-logs-only \ |
1411 | + $XTRABACKUP_OPTIONS |
1412 | + #Copy prepared data to server data dir |
1413 | + cp -R $BACKUP_DIR/* $mysql_datadir |
1414 | + rm $mysql_datadir/ib_* |
1415 | + #Start server with prepared data |
1416 | + start_server "--binlog-format=ROW $EXTRA_OPTIONS" |
1417 | + #Get values from restored data files before remembered LSN |
1418 | + get_changes 1 $TABLES_COUNT $TABLE_NAME |
1419 | + #Get values from restored data files after remembered LSN |
1420 | + get_changes $(($TABLES_COUNT+1)) $TABLES_COUNT $TABLE_NAME |
1421 | + #We don't need server already |
1422 | + stop_server |
1423 | + #Check if the changes which was made before remembered LSN are in the |
1424 | + #restored databse |
1425 | + check_changes 1 $TABLES_COUNT 'check_if_equal' |
1426 | + #Check if the changes which was made after remembered LSN are NOT in the |
1427 | + #restored databse |
1428 | + check_changes $(($TABLES_COUNT+1)) $TABLES_COUNT 'check_if_not_equal' |
1429 | + # Apply the rest of archived logs |
1430 | + xtrabackup --prepare \ |
1431 | + --target-dir=$BACKUP_DIR \ |
1432 | + --innodb-log-arch-dir=$ARCHIVED_LOGS_DIR \ |
1433 | + --apply-logs-only \ |
1434 | + $XTRABACKUP_OPTIONS |
1435 | + #Copy prepared data to server data dir |
1436 | + cp -R $BACKUP_DIR/* $mysql_datadir |
1437 | + rm $mysql_datadir/ib_* |
1438 | + #Start server with prepared data |
1439 | + start_server "--binlog-format=ROW $EXTRA_OPTIONS" |
1440 | + #Get values from restored data files before remembered LSN |
1441 | + get_changes 1 $TABLES_COUNT $TABLE_NAME |
1442 | + #Get values from restored data files after remembered LSN |
1443 | + get_changes $(($TABLES_COUNT+1)) $TABLES_COUNT $TABLE_NAME |
1444 | + stop_server |
1445 | + #Check if the changes which was made before remembered LSN are in the |
1446 | + #restored databse |
1447 | + check_changes 1 $TABLES_COUNT 'check_if_equal' |
1448 | + #Check if the changes which was made after remembered LSN are in the |
1449 | + #restored databse |
1450 | + check_changes $(($TABLES_COUNT+1)) $TABLES_COUNT 'check_if_equal' |
1451 | + rm -rf $mysql_datadir |
1452 | + rm -rf $BACKUP_DIR |
1453 | + |
1454 | + ################################################## |
1455 | + # Check the possibility of applying logs by sets # |
1456 | + ################################################## |
1457 | + cp -R $BASE_DATA_DIR $mysql_datadir |
1458 | + cp -R $BASE_BACKUP_DIR $BACKUP_DIR |
1459 | + mkdir -p $ARCHIVED_LOGS_DIR/1 $ARCHIVED_LOGS_DIR/2 |
1460 | + #Make two log files sets. The first set contains the first two files, |
1461 | + #the second set contains the rest and the last log file from the first |
1462 | + #set. |
1463 | + pushd . |
1464 | + cd $ARCHIVED_LOGS_DIR |
1465 | + for i in `find . -type f -printf "%T+ %p\n" | cut -d' ' -f2`; |
1466 | + do |
1467 | + local n=`echo $i | sed 's/.*ib_log_archive_\([0-9]*\).*/\1/'`; |
1468 | + if [ $n -le $LSN ]; |
1469 | + then |
1470 | + mv $i 1/; |
1471 | + else |
1472 | + mv $i 2/; |
1473 | + fi; |
1474 | + done |
1475 | + cd 1 |
1476 | + find . -type f -printf "%T+ %p\n" | cut -d' ' -f2 | sort -n | tail -1 | \ |
1477 | + xargs -I{} cp {} ../2/ |
1478 | + popd |
1479 | + #Prepare the first set |
1480 | + xtrabackup --prepare \ |
1481 | + --target-dir=$BACKUP_DIR \ |
1482 | + --innodb-log-arch-dir=$ARCHIVED_LOGS_DIR/1 \ |
1483 | + $XTRABACKUP_OPTIONS |
1484 | + #Prepare the second set |
1485 | + xtrabackup --prepare \ |
1486 | + --target-dir=$BACKUP_DIR \ |
1487 | + --innodb-log-arch-dir=$ARCHIVED_LOGS_DIR/2 \ |
1488 | + $XTRABACKUP_OPTIONS |
1489 | + #Copy prepared data to server data dir |
1490 | + cp -R $BACKUP_DIR/* $mysql_datadir |
1491 | + rm $mysql_datadir/ib_* |
1492 | + #Start server with prepared data |
1493 | + start_server "--binlog-format=ROW $EXTRA_OPTIONS" |
1494 | + #Get values from restored data files before remembered LSN |
1495 | + get_changes 1 $TABLES_COUNT $TABLE_NAME |
1496 | + #Get values from restored data files after remembered LSN |
1497 | + get_changes $(($TABLES_COUNT+1)) $TABLES_COUNT $TABLE_NAME |
1498 | + stop_server |
1499 | + #Check all made changes |
1500 | + check_changes 1 $TABLES_COUNT 'check_if_equal' |
1501 | + check_changes $(($TABLES_COUNT+1)) $TABLES_COUNT 'check_if_equal' |
1502 | + |
1503 | + #Clean up dir for the next procedure launch |
1504 | + rm -rf $BACKUP_DIR/* $BASE_BACKUP_DIR/* |
1505 | + rm -rf $mysql_datadir $BASE_DATA_DIR |
1506 | + rm -rf $ARCHIVED_LOGS_DIR/* |
1507 | + |
1508 | + #Clean up variables |
1509 | + unset_global_variables |
1510 | +} |
1511 | + |
1512 | +require_xtradb |
1513 | +require_server_version_higher_than '5.6.10' |
1514 | + |
1515 | + |
1516 | +if [ `basename $XB_BIN` = 'xtrabackup_56' ] |
1517 | +then |
1518 | + test_archived_logs |
1519 | + test_archived_logs '' 'ROW_FORMAT=COMPRESSED' |
1520 | +else |
1521 | + echo 'Requires xtrabackup built with XtraDB >= 5.6' > $SKIPPED_REASON |
1522 | + exit $SKIPPED_EXIT_CODE |
1523 | +fi |
1524 | + |
1525 | +exit $RESULT |
A merge conflict, unless spurious, that needs rebasing on the current trunk?