Merge lp:~sergei.glushchenko/percona-xtrabackup/buffer_pool_dump into lp:percona-xtrabackup/2.0

Proposed by Sergei Glushchenko
Status: Merged
Approved by: Alexey Kopytov
Approved revision: 542
Merged at revision: 546
Proposed branch: lp:~sergei.glushchenko/percona-xtrabackup/buffer_pool_dump
Merge into: lp:percona-xtrabackup/2.0
Diff against target: 291 lines (+194/-19)
5 files modified
innobackupex (+28/-19)
src/xtrabackup.cc (+11/-0)
test/t/ib_buffer_pool.sh (+57/-0)
test/t/ib_buffer_pool_dump_incremental.sh (+58/-0)
test/t/ib_buffer_pool_rsync.sh (+40/-0)
To merge this branch: bzr merge lp:~sergei.glushchenko/percona-xtrabackup/buffer_pool_dump
Reviewer Review Type Date Requested Status
Alexey Kopytov (community) Approve
Review via email: mp+160825@code.launchpad.net

Description of the change

Support for buffer pool dump MySQL 5.6

   - MySQL 5.6 adds an ability to dump the buffer pool into file for it to be
     possible to load it into memory at startup or on demand. The location of
     buffer pool file is set by global variable innodb_buffer_pool_filename
     and is relative to MySQL data directory.
   - The patch adds an ability to take a backup of buffer pool dump. It has been
     made exactly the same way as XtraDB LRU dump is backed up.
   - Two tests have been added for this feature. One takes regular backup and
     after takes streaming backup. Another one takes backup with --remote-host
     option. Second test has not been ported to 2.1 because --remote-host
     option is not supported in 2.1.
   - Also test for these features to work correctly with incremental backups
     is implemented.

http://jenkins.percona.com/view/XtraBackup/job/percona-xtrabackup-2.0-param/419/

To post a comment you must log in.
Revision history for this message
Alexey Kopytov (akopytov) wrote :

Sergei,

  - string values for most options in --print-param are enclosed in
    double quotes. It only matters when path or file names contain
    spaces.
  - MP and revision comments refer to a --remote-host test, but I don't
    see it in the code (would also be rather tricky to test in the test
    suite). I thought it was a typo for --rsync, but the rsync test is
    also present in the 2.1 MP as it should be.
  - 5.6 version checks are different in test case. I suggest to use
    "${MYSQL_VERSION:0:3}" != "5.6", the other one looks a bit
    unreliable
  - please rebase it on current trunks. my.cnf is handled differently
    after recent merges, and the way it is done in these tests will
    conflict. You can now specify MYSQLD_EXTRA_MY_CNF_OPTS and it will
    be used by both server (no need for extra start_server arguments)
    and innobackupex/xtrabackup

review: Needs Fixing
Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :

Alexei,

  - string values for most options in --print-param are enclosed in
    double quotes. It only matters when path or file names contain
    spaces.

> I added quotes

  - MP and revision comments refer to a --remote-host test, but I don't
    see it in the code (would also be rather tricky to test in the test
    suite). I thought it was a typo for --rsync, but the rsync test is
    also present in the 2.1 MP as it should be.

> Indeed, I mixed many things in one. And also forgot to add test for incremental backup.

  - 5.6 version checks are different in test case. I suggest to use
    "${MYSQL_VERSION:0:3}" != "5.6", the other one looks a bit
    unreliable

> I modified version check to be comparison with "5.6"

  - please rebase it on current trunks. my.cnf is handled differently
    after recent merges, and the way it is done in these tests will
    conflict. You can now specify MYSQLD_EXTRA_MY_CNF_OPTS and it will
    be used by both server (no need for extra start_server arguments)
    and innobackupex/xtrabackup

> done.

Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :

Alexey, sorry for typo in name.

Revision history for this message
Alexey Kopytov (akopytov) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'innobackupex'
2--- innobackupex 2013-04-25 08:27:18 +0000
3+++ innobackupex 2013-04-25 14:34:24 +0000
4@@ -392,6 +392,8 @@
5 #
6 sub backup {
7 my $orig_datadir = get_option(\%config, $option_defaults_group, 'datadir');
8+ my $buffer_pool_filename = get_option(\%config, $option_defaults_group,
9+ 'innodb_buffer_pool_filename');
10
11 # check that we can connect to the database. This done by
12 # connecting, issuing a query, and closing the connection.
13@@ -447,21 +449,24 @@
14 # Close the DB connection
15 mysql_close();
16
17- # copy ib_lru_dump
18- if (-e "$orig_datadir/ib_lru_dump") {
19- if ($option_remote_host) {
20- print STDERR "$prefix Backing up file 'ib_lru_dump'\n";
21- system("scp $option_scp_opt '$orig_datadir/ib_lru_dump' '$option_remote_host:$backup_dir/ib_lru_dump'")
22- and Die "Failed to scp file 'ib_lru_dump': $!";
23- } elsif ($option_stream) {
24- print STDERR "$prefix Backing up as tar stream 'ib_lru_dump'\n";
25- system("cd $orig_datadir; $stream_cmd ib_lru_dump")
26- and Die "Failed to stream 'ib_lru_dump': $!";
27- } elsif (!$option_rsync) {
28- my $src_name = escape_path("$orig_datadir/ib_lru_dump");
29- my $dst_name = escape_path("$backup_dir/ib_lru_dump");
30- system("$CP_CMD \"$src_name\" \"$dst_name\"")
31- and Die "Failed to copy file 'ib_lru_dump': $!";
32+ # Copy buffer poll dump and/or LRU dump
33+ foreach my $dump_name ($buffer_pool_filename, 'ib_lru_dump') {
34+ if (-e "$orig_datadir/$dump_name") {
35+ if ($option_remote_host) {
36+ print STDERR "$prefix Backing up file '$dump_name'\n";
37+ system("scp $option_scp_opt '$orig_datadir/$dump_name' " .
38+ "'$option_remote_host:$backup_dir/$dump_name'")
39+ and Die "Failed to scp file '$dump_name': $!";
40+ } elsif ($option_stream) {
41+ print STDERR "$prefix Backing up as tar stream '$dump_name'\n";
42+ system("cd $orig_datadir; $stream_cmd $dump_name")
43+ and Die "Failed to stream '$dump_name': $!";
44+ } elsif (!$option_rsync) {
45+ my $src_name = escape_path("$orig_datadir/$dump_name");
46+ my $dst_name = escape_path("$backup_dir/$dump_name");
47+ system("$CP_CMD \"$src_name\" \"$dst_name\"")
48+ and Die "Failed to copy file '$dump_name': $!";
49+ }
50 }
51 }
52
53@@ -2204,6 +2209,8 @@
54 sub backup_files {
55 my $prep_mode = shift;
56 my $source_dir = get_option(\%config, $option_defaults_group, 'datadir');
57+ my $buffer_pool_filename = get_option(\%config, $option_defaults_group,
58+ 'innodb_buffer_pool_filename');
59 my @list;
60 my $file;
61 my $database;
62@@ -2338,10 +2345,12 @@
63 closedir(DIR);
64
65 if ($option_rsync) {
66- if (-e "$source_dir/ib_lru_dump") {
67- print RSYNC "ib_lru_dump\n";
68- if (!$prep_mode) {
69- $rsync_files_hash{"ib_lru_dump"} = 1;
70+ foreach my $dump_name ($buffer_pool_filename, 'ib_lru_dump') {
71+ if (-e "$source_dir/$dump_name") {
72+ print RSYNC "$dump_name\n";
73+ if (!$prep_mode) {
74+ $rsync_files_hash{"$dump_name"} = 1;
75+ }
76 }
77 }
78 close(RSYNC);
79
80=== modified file 'src/xtrabackup.cc'
81--- src/xtrabackup.cc 2013-04-24 14:09:42 +0000
82+++ src/xtrabackup.cc 2013-04-25 14:34:24 +0000
83@@ -1436,6 +1436,7 @@
84 my_bool innobase_fast_checksum = FALSE;
85 my_bool innobase_extra_undoslots = FALSE;
86 char* innobase_doublewrite_file = NULL;
87+char* innobase_buffer_pool_filename = NULL;
88
89 longlong innobase_buffer_pool_size = 8*1024*1024L;
90 longlong innobase_log_file_size = DEFAULT_LOG_FILE_SIZE;
91@@ -1633,6 +1634,7 @@
92 OPT_INNODB_EXTRA_UNDOSLOTS,
93 OPT_INNODB_DOUBLEWRITE_FILE,
94 #endif
95+ OPT_INNODB_BUFFER_POOL_FILENAME,
96 OPT_INNODB_FORCE_RECOVERY,
97 OPT_INNODB_LOCK_WAIT_TIMEOUT,
98 OPT_INNODB_LOG_BUFFER_SIZE,
99@@ -1961,6 +1963,11 @@
100 (G_PTR*) &innobase_doublewrite_file, (G_PTR*) &innobase_doublewrite_file,
101 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0},
102 #endif
103+ {"innodb_buffer_pool_filename", OPT_INNODB_BUFFER_POOL_FILENAME,
104+ "Filename to/from which to dump/load the InnoDB buffer pool",
105+ (G_PTR*) &innobase_buffer_pool_filename,
106+ (G_PTR*) &innobase_buffer_pool_filename,
107+ 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0},
108
109 #ifndef __WIN__
110 {"debug-sync", OPT_XTRA_DEBUG_SYNC,
111@@ -8040,6 +8047,10 @@
112 }
113 printf("innodb_undo_tablespaces = %lu\n", srv_undo_tablespaces);
114 #endif
115+ printf("innodb_buffer_pool_filename = \"%s\"\n",
116+ innobase_buffer_pool_filename ?
117+ innobase_buffer_pool_filename :
118+ "ib_buffer_pool");
119 exit(EXIT_SUCCESS);
120 }
121
122
123=== added file 'test/t/ib_buffer_pool.sh'
124--- test/t/ib_buffer_pool.sh 1970-01-01 00:00:00 +0000
125+++ test/t/ib_buffer_pool.sh 2013-04-25 14:34:24 +0000
126@@ -0,0 +1,57 @@
127+########################################################################
128+# Blueprint: Support InnoDB buffer pool dumps in MySQL 5.6
129+########################################################################
130+
131+. inc/common.sh
132+
133+if [ "${MYSQL_VERSION:0:3}" != "5.6" ]; then
134+ echo "Requires MySQL 5.6" > $SKIPPED_REASON
135+ exit $SKIPPED_EXIT_CODE
136+fi
137+
138+MYSQLD_EXTRA_MY_CNF_OPTS="
139+innodb_buffer_pool_filename=pool/dump
140+"
141+
142+start_server
143+
144+# test presence of innodb_buffer_pool_filename
145+${XB_BIN} --defaults-file=$topdir/my.cnf --print-param | \
146+ grep -q "innodb_buffer_pool_filename"
147+
148+mkdir $mysql_datadir/pool
149+
150+# produce buffer pool dump
151+${MYSQL} ${MYSQL_ARGS} -e "SET GLOBAL innodb_buffer_pool_dump_now=ON;"
152+
153+# take a backup
154+innobackupex --no-timestamp $topdir/backup
155+
156+if [ -f $topdir/backup/pool/dump ] ; then
157+ vlog "Buffer pool dump has been backed up"
158+else
159+ vlog "Buffer pool dump has not been backed up"
160+ exit -1
161+fi
162+
163+# take streaming backup
164+mkdir -p $topdir/backup
165+innobackupex --stream=tar $topdir/backup > $topdir/backup/stream.tar
166+
167+if $TAR itf $topdir/backup/stream.tar | grep 'pool/dump' ; then
168+ vlog "Buffer pool dump has been restored"
169+else
170+ vlog "Buffer pool dump has not been restored"
171+ exit -1
172+fi
173+
174+# restore from backup
175+rm -rf $mysql_datadir/*
176+innobackupex --copy-back $topdir/backup
177+
178+if [ -f $mysql_datadir/pool/dump ] ; then
179+ vlog "Buffer pool dump has been restored"
180+else
181+ vlog "Buffer pool dump has not been restored"
182+ exit -1
183+fi
184
185=== added file 'test/t/ib_buffer_pool_dump_incremental.sh'
186--- test/t/ib_buffer_pool_dump_incremental.sh 1970-01-01 00:00:00 +0000
187+++ test/t/ib_buffer_pool_dump_incremental.sh 2013-04-25 14:34:24 +0000
188@@ -0,0 +1,58 @@
189+########################################################################
190+# Blueprint: Support InnoDB buffer pool dumps in MySQL 5.6
191+########################################################################
192+
193+. inc/common.sh
194+
195+if [ "${MYSQL_VERSION:0:3}" != "5.6" ]; then
196+ echo "Requires MySQL 5.6" > $SKIPPED_REASON
197+ exit $SKIPPED_EXIT_CODE
198+fi
199+
200+MYSQLD_EXTRA_MY_CNF_OPTS="
201+innodb_buffer_pool_filename=pool/dump
202+"
203+
204+start_server
205+
206+# test presence of innodb_buffer_pool_filename
207+${XB_BIN} --defaults-file=$topdir/my.cnf --print-param | \
208+ grep -q "innodb_buffer_pool_filename"
209+
210+mkdir $mysql_datadir/pool
211+
212+# take a backup
213+innobackupex --no-timestamp $topdir/backup
214+
215+# produce buffer pool dump
216+${MYSQL} ${MYSQL_ARGS} -e "SET GLOBAL innodb_buffer_pool_dump_now=ON;"
217+
218+# incremental backup
219+innobackupex --no-timestamp --incremental --incremental-basedir=$topdir/backup \
220+ $topdir/incremental
221+
222+if [ -f $topdir/incremental/pool/dump ] ; then
223+ vlog "Buffer pool dump has been backed up"
224+else
225+ vlog "Buffer pool dump has not been backed up"
226+ exit -1
227+fi
228+
229+# prepare
230+innobackupex --apply-log --redo-only $topdir/backup
231+
232+innobackupex --apply-log --redo-only --incremental-dir=$topdir/incremental \
233+ $topdir/backup
234+
235+innobackupex --apply-log $topdir/backup
236+
237+# restore from backup
238+rm -rf $mysql_datadir/*
239+innobackupex --copy-back $topdir/backup
240+
241+if [ -f $mysql_datadir/pool/dump ] ; then
242+ vlog "Buffer pool dump has been restored"
243+else
244+ vlog "Buffer pool dump has not been restored"
245+ exit -1
246+fi
247
248=== added file 'test/t/ib_buffer_pool_rsync.sh'
249--- test/t/ib_buffer_pool_rsync.sh 1970-01-01 00:00:00 +0000
250+++ test/t/ib_buffer_pool_rsync.sh 2013-04-25 14:34:24 +0000
251@@ -0,0 +1,40 @@
252+########################################################################
253+# Blueprint: Support InnoDB buffer pool dumps in MySQL 5.6
254+########################################################################
255+
256+. inc/common.sh
257+
258+if ! which rsync > /dev/null 2>&1 ; then
259+ echo "Requires rsync to be installed" > $SKIPPED_REASON
260+ exit $SKIPPED_EXIT_CODE
261+fi
262+
263+if [ "${MYSQL_VERSION:0:3}" != "5.6" ]; then
264+ echo "Requires MySQL 5.6" > $SKIPPED_REASON
265+ exit $SKIPPED_EXIT_CODE
266+fi
267+
268+MYSQLD_EXTRA_MY_CNF_OPTS="
269+innodb_buffer_pool_filename=pool/dump
270+"
271+
272+start_server
273+
274+# test presence of innodb_buffer_pool_filename
275+${XB_BIN} --defaults-file=$topdir/my.cnf --print-param | \
276+ grep -q "innodb_buffer_pool_filename"
277+
278+mkdir $mysql_datadir/pool
279+
280+# produce buffer pool dump
281+${MYSQL} ${MYSQL_ARGS} -e "SET GLOBAL innodb_buffer_pool_dump_now=ON;"
282+
283+# take a backup with rsync mode
284+innobackupex --rsync --no-timestamp $topdir/backup
285+
286+if [ -f $topdir/backup/pool/dump ] ; then
287+ vlog "Buffer pool dump has been backed up"
288+else
289+ vlog "Buffer pool dump has not been backed up"
290+ exit -1
291+fi

Subscribers

People subscribed via source and target branches