Merge lp:~vlad-lesin/percona-server/5.6-mysqlbinlog-replacedb into lp:percona-server/5.6

Proposed by Vlad Lesin on 2013-11-25
Status: Merged
Approved by: Laurynas Biveinis on 2014-02-24
Approved revision: 541
Merged at revision: 544
Proposed branch: lp:~vlad-lesin/percona-server/5.6-mysqlbinlog-replacedb
Merge into: lp:percona-server/5.6
Diff against target: 1011 lines (+850/-2)
15 files modified
client/CMakeLists.txt (+2/-1)
client/client_priv.h (+1/-0)
client/mysqlbinlog.cc (+160/-0)
mysql-test/extra/rpl_tests/grep_pattern.inc (+22/-0)
mysql-test/suite/binlog/r/binlog_mysqlbinlog_rewrite_db.result (+33/-0)
mysql-test/suite/binlog/r/binlog_rewrite_db_noleak.result (+29/-0)
mysql-test/suite/binlog/r/binlog_rewrite_suppress_use.result (+29/-0)
mysql-test/suite/binlog/r/percona_mysqlbinlog_rewritedb.result (+152/-0)
mysql-test/suite/binlog/t/binlog_mysqlbinlog_rewrite_db.test (+95/-0)
mysql-test/suite/binlog/t/binlog_rewrite_db_noleak.test (+79/-0)
mysql-test/suite/binlog/t/binlog_rewrite_suppress_use.test (+61/-0)
mysql-test/suite/binlog/t/percona_mysqlbinlog_rewritedb-master.opt (+1/-0)
mysql-test/suite/binlog/t/percona_mysqlbinlog_rewritedb.test (+80/-0)
sql/log_event.cc (+102/-0)
sql/log_event.h (+4/-1)
To merge this branch: bzr merge lp:~vlad-lesin/percona-server/5.6-mysqlbinlog-replacedb
Reviewer Review Type Date Requested Status
Laurynas Biveinis (community) Approve on 2014-02-24
Stewart Smith (community) Needs Fixing on 2013-12-20
George Ormond Lorch III g2 2013-11-25 Approve on 2013-11-27
Review via email: mp+196583@code.launchpad.net

Description of the change

Merged from 5.5.

Port --rewrite-db mysqlbinlog option from 5.1.

--rewrite-db is a string option which allows to replace one db-name to another
in mysqlbinlog output. The format of the string value is the following:
"from1->to1,from2->to2,from3->to3". Where "to[123]" are the names to which
the correspondent "from[123]" names must be replaced.

The original implementation does not replace db names in statements for
statement-based binlog, it just replaces db name of binlog event and issues
"USE" statement with replaced db name if necessary. I think this restriction
concerned with unwillingness to implement complex logic for parsing db names
from statements for such simple feature.

But for row-based binlog it replaces db names in statements correctly because
each table name is mapped to some number in binlog and it is quite easy to
extract such table ids from binlog event and replace one name by another.

http://jenkins.percona.com/job/percona-server-5.6-param/408/

The following changes are between 5.5 and 5.6:
1) Define "CHARSET_INFO* table_alias_charset= &my_charset_utf8_general_ci;" in 5.6 instead of "CHARSET_INFO* system_charset_info= &my_charset_utf8_general_ci;" in 5.5. There is no need to define system_charset_info in 5.6 to link mysqlbinlog with rpl_filter.o, but table_alias_charset is necessary.

2) process_event() differs because the original code differs too.

To post a comment you must log in.
review: Approve (g2)
Stewart Smith (stewart) wrote :

It states that you can do "a->b,c->d" but I don't see the handling for the comma in the mysqlbinlog.cc command line parsing. Am I missing something? We should also test this.

review: Needs Fixing

See 5.5 comments.

review: Needs Fixing
Vlad Lesin (vlad-lesin) wrote :

> See 5.5 comments.
Fixed, http://jenkins.percona.com/view/PS%205.6/job/percona-server-5.6-param/467/.

Also there is a memory leak in mysql-5.7.3 when --rewrite-db option is used. Here is the bug report http://bugs.mysql.com/bug.php?id=71283 .

Vlad Lesin (vlad-lesin) wrote :

See 5.5 comments.

http://jenkins.percona.com/view/PS 5.6/job/percona-server-5.6-param/499

review: Needs Fixing
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'client/CMakeLists.txt'
2--- client/CMakeLists.txt 2013-10-08 06:03:07 +0000
3+++ client/CMakeLists.txt 2014-02-15 00:30:14 +0000
4@@ -61,7 +61,8 @@
5 MYSQL_ADD_EXECUTABLE(mysql_plugin mysql_plugin.c)
6 TARGET_LINK_LIBRARIES(mysql_plugin perconaserverclient)
7
8-MYSQL_ADD_EXECUTABLE(mysqlbinlog mysqlbinlog.cc sql_string.cc)
9+MYSQL_ADD_EXECUTABLE(mysqlbinlog mysqlbinlog.cc sql_string.cc
10+ ../sql/rpl_filter.cc ../sql/sql_list.cc)
11 TARGET_LINK_LIBRARIES(mysqlbinlog perconaserverclient)
12
13 MYSQL_ADD_EXECUTABLE(mysqladmin mysqladmin.cc)
14
15=== modified file 'client/client_priv.h'
16--- client/client_priv.h 2013-06-25 13:13:06 +0000
17+++ client/client_priv.h 2014-02-15 00:30:14 +0000
18@@ -104,6 +104,7 @@
19 OPT_SERVER_PUBLIC_KEY,
20 OPT_ENABLE_CLEARTEXT_PLUGIN,
21 OPT_INNODB_OPTIMIZE_KEYS,
22+ OPT_REWRITE_DB,
23 OPT_MAX_CLIENT_OPTION
24 };
25
26
27=== modified file 'client/mysqlbinlog.cc'
28--- client/mysqlbinlog.cc 2013-10-31 18:42:25 +0000
29+++ client/mysqlbinlog.cc 2014-02-15 00:30:14 +0000
30@@ -60,6 +60,18 @@
31 using std::min;
32 using std::max;
33
34+/* Needed for Rpl_filter */
35+CHARSET_INFO *table_alias_charset= &my_charset_bin;
36+
37+#include "rpl_filter.h"
38+
39+/*
40+ True if obsolette syntax warning has been already shown
41+ during parsing --rewrite-db command line option
42+*/
43+bool rewrite_db_obs_syn_warn= false;
44+Rpl_filter *binlog_filter= NULL;
45+
46 #define BIN_LOG_HEADER_SIZE 4U
47 #define PROBE_HEADER_LEN (EVENT_LEN_OFFSET+4)
48 #define INTVAR_DYNAMIC_INIT 16
49@@ -712,6 +724,35 @@
50 strcmp(log_dbname, database);
51 }
52
53+/**
54+ Rewrites db name in T instance if binlog_filter contains
55+ name for replacement(see --rewrite-db option).
56+
57+ T::db must be weak pointer and can point to the memory owned by
58+ "binlog_filter" after this function execution, that is why "ev" must be
59+ destroyed before "binlog_filter".
60+
61+ @param ev Event to process
62+*/
63+template <typename T>
64+static void rewrite_db(T &ev)
65+{
66+ size_t len_to= 0;
67+ const char* db_to;
68+
69+ DBUG_ASSERT(binlog_filter);
70+
71+ if (!ev.db)
72+ return;
73+
74+ db_to= binlog_filter->get_rewrite_db(ev.db, &len_to);
75+
76+ if (!len_to)
77+ return;
78+
79+ ev.db= db_to;
80+ ev.db_len= len_to;
81+}
82
83 /**
84 Checks whether the given event should be filtered out,
85@@ -898,6 +939,13 @@
86 switch (ev_type) {
87 case QUERY_EVENT:
88 {
89+ /*
90+ ev is deleted at the end of this function(before binlog_filter deletion)
91+ so it is safe to set ev->db to some memory owned by binlog_filter here.
92+ */
93+ if (binlog_filter)
94+ rewrite_db(*static_cast<Query_log_event *>(ev));
95+
96 bool parent_query_skips=
97 !((Query_log_event*) ev)->is_trans_keyword() &&
98 shall_skip_database(((Query_log_event*) ev)->db);
99@@ -989,6 +1037,12 @@
100 {
101 Create_file_log_event* ce= (Create_file_log_event*)ev;
102 /*
103+ ev is deleted at the end of this function(before binlog_filter deletion)
104+ so it is safe to set ev->db to some memory owned by binlog_filter here.
105+ */
106+ if (binlog_filter)
107+ rewrite_db(*ce);
108+ /*
109 We test if this event has to be ignored. If yes, we don't save
110 this event; this will have the good side-effect of ignoring all
111 related Append_block and Exec_load.
112@@ -1115,6 +1169,12 @@
113 case EXECUTE_LOAD_QUERY_EVENT:
114 {
115 Execute_load_query_log_event *exlq= (Execute_load_query_log_event*)ev;
116+ /*
117+ ev is deleted at the end of this function(before binlog_filter deletion)
118+ so it is safe to set ev->db to some memory owned by binlog_filter here.
119+ */
120+ if (binlog_filter)
121+ rewrite_db(*exlq);
122 char *fname= load_processor.grab_fname(exlq->file_id);
123
124 if (shall_skip_database(exlq->db))
125@@ -1144,6 +1204,19 @@
126 case TABLE_MAP_EVENT:
127 {
128 Table_map_log_event *map= ((Table_map_log_event *)ev);
129+ // Rewrite db name here (see --rewrite-db option)
130+ if (binlog_filter)
131+ {
132+ size_t len_to= 0;
133+ const char* db_to= binlog_filter->get_rewrite_db(map->get_db_name(),
134+ &len_to);
135+ if (len_to && map->rewrite_db(db_to, len_to, glob_description_event))
136+ {
137+ error("Could not rewrite database name");
138+ goto err;
139+ }
140+ }
141+
142 if (shall_skip_database(map->get_db_name()))
143 {
144 print_event_info->skipped_event_in_transaction= true;
145@@ -1589,6 +1662,10 @@
146 "Identifiers were provided.",
147 &opt_exclude_gtids_str, &opt_exclude_gtids_str, 0,
148 GET_STR_ALLOC, REQUIRED_ARG, 0, 0, 0, 0, 0, 0},
149+ {"rewrite-db", OPT_REWRITE_DB,
150+ "Updates to a database with a different name than the original. "
151+ "Example: rewrite-db='from->to'.",
152+ 0, 0, 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0},
153 {0, 0, 0, 0, 0, 0, GET_NO_ARG, NO_ARG, 0, 0, 0, 0, 0, 0}
154 };
155
156@@ -1674,6 +1751,7 @@
157 delete_dynamic(&buff_ev);
158
159 delete glob_description_event;
160+ delete binlog_filter;
161 if (mysql)
162 mysql_close(mysql);
163 }
164@@ -1773,6 +1851,88 @@
165 opt_base64_output_mode= (enum_base64_output_mode)
166 (find_type_or_exit(argument, &base64_output_mode_typelib, opt->name)-1);
167 break;
168+ case OPT_REWRITE_DB: // db_from->db_to
169+ {
170+ /* See also handling of OPT_REPLICATE_REWRITE_DB in sql/mysqld.cc */
171+ char* ptr= argument;
172+ do {
173+ char* key= ptr; // db-from
174+ char* val; // db-to
175+
176+ // Where key begins
177+ while (*key && my_isspace(&my_charset_latin1, *key))
178+ key++;
179+
180+ // Where val begins
181+ if (!(ptr= strstr(key, "->")))
182+ {
183+ sql_print_error("Bad syntax in rewrite-db: missing '->'!\n");
184+ return 1;
185+ }
186+ val= ptr + 2;
187+ while (*val && my_isspace(&my_charset_latin1, *val))
188+ val++;
189+
190+ // Write \0 and skip blanks at the end of key
191+ *ptr-- = 0;
192+ while (my_isspace(&my_charset_latin1, *ptr) && ptr > argument)
193+ *ptr-- = 0;
194+
195+ if (!*key)
196+ {
197+ sql_print_error("Bad syntax in rewrite-db: empty db-from!\n");
198+ return 1;
199+ }
200+
201+ // Skip blanks at the end of val
202+ ptr= val;
203+ while (*ptr && !my_isspace(&my_charset_latin1, *ptr) && *ptr != ',')
204+ ++ptr;
205+
206+ if (my_isspace(&my_charset_latin1, *ptr))
207+ {
208+ *(ptr++)= 0;
209+ while (*ptr && *ptr != ',')
210+ {
211+ if (!my_isspace(&my_charset_latin1, *ptr)) {
212+ sql_print_error("Bad syntax in rewrite-db: db-to must contain db "
213+ "name without spaces!\n");
214+ return 1;
215+ }
216+ ++ptr;
217+ }
218+ }
219+
220+ if (*ptr == ',')
221+ {
222+ if (!rewrite_db_obs_syn_warn)
223+ {
224+ warning("The comma-separated list of rewritings syntax is obsolete and "
225+ "discarded in 5.7\n");
226+ rewrite_db_obs_syn_warn= true;
227+ }
228+ *(ptr++)= 0;
229+ }
230+ else
231+ *ptr= 0;
232+
233+ if (!*val)
234+ {
235+ sql_print_error("Bad syntax in rewrite-db: empty db-to!\n");
236+ return 1;
237+ }
238+
239+ if (!binlog_filter &&
240+ !(binlog_filter= new Rpl_filter))
241+ {
242+ sql_print_error("Failed to create Rpl_filter\n");
243+ return 1;
244+ }
245+
246+ binlog_filter->add_db_rewrite(key, val);
247+ } while (*ptr);
248+ break;
249+ }
250 case 'v':
251 if (argument == disabled_my_option)
252 verbose= 0;
253
254=== added file 'mysql-test/extra/rpl_tests/grep_pattern.inc'
255--- mysql-test/extra/rpl_tests/grep_pattern.inc 1970-01-01 00:00:00 +0000
256+++ mysql-test/extra/rpl_tests/grep_pattern.inc 2014-02-15 00:30:14 +0000
257@@ -0,0 +1,22 @@
258+# Please set GREP_FILE and GREP_PATTERN environment variables
259+# to work this file properly.
260+--perl
261+ use strict;
262+ my $file= $ENV{'GREP_FILE'} or die "grep file not set";
263+ my $pattern= $ENV{'GREP_PATTERN'} or die "pattern is not set";
264+ open(FILE, "$file") or die("Unable to open $file: $!\n");
265+ my $count = 0;
266+ print "Matching lines are:\n";
267+ while (<FILE>) {
268+ my $line = $_;
269+ if ($line =~ /$pattern/) {
270+ print "$line\n";
271+ $count++;
272+ }
273+ }
274+ if ($count == 0) {
275+ print "None\n";
276+ }
277+ print "Occurrences of the $pattern in the input file : $count\n";
278+ close(FILE);
279+EOF
280
281=== added file 'mysql-test/suite/binlog/r/binlog_mysqlbinlog_rewrite_db.result'
282--- mysql-test/suite/binlog/r/binlog_mysqlbinlog_rewrite_db.result 1970-01-01 00:00:00 +0000
283+++ mysql-test/suite/binlog/r/binlog_mysqlbinlog_rewrite_db.result 2014-02-15 00:30:14 +0000
284@@ -0,0 +1,33 @@
285+RESET MASTER;
286+CREATE DATABASE db1;
287+USE db1;
288+CREATE TABLE t1 (i INT);
289+INSERT INTO t1 VALUES(1);
290+INSERT INTO t1 VALUES(2);
291+UPDATE t1 SET i= i+1;
292+DELETE FROM t1 WHERE i=2;
293+[Syntax error in the use of the new option: The from database name is missing]
294+[Syntax error in the use of the new option: The '->' is missing]
295+[Syntax error in the use of the new option: The to database name is missing]
296+[VALID SYNTAX,The from->to database names are correctly mentioned, but there is obsolete syntax warning 1]
297+[Test that warning about obsolete syntax is shown only once for --rewrite-db="db1->db2,db3->db4,db5->db6"]
298+[log_grep.inc] file: mysqltest.log pattern: The comma-separated list of rewritings syntax is obsolete and discarded
299+[log_grep.inc] lines: 1
300+[VALID SYNTAX,The from->to database names are correctly mentioned, but there is obsolete syntax warning 2]
301+[Test that warning about obsolete syntax is shown only once for --rewrite-db="db1->db2,db3->db4" --rewrite-db="db5->db6,db6->db7"]
302+[log_grep.inc] file: mysqltest.log pattern: The comma-separated list of rewritings syntax is obsolete and discarded
303+[log_grep.inc] lines: 2
304+[VALID SYNTAX,The from->to database names are correctly mentioned]
305+#Dropping the database db1 and creating the table in the new database db2.
306+CREATE DATABASE db2;
307+DROP DATABASE db1;
308+RESET MASTER;
309+SELECT * FROM db2.t1;
310+ERROR 42S02: Table 'db2.t1' doesn't exist
311+DROP DATABASE db1;
312+RESET MASTER;
313+[The event of table db1.t1 has been successfully applied to db2.t1]
314+include/assert.inc [Assert that table db2.t1 has one row after applying the sql file.]
315+CLEANUP
316+DROP DATABASE db1;
317+DROP DATABASE db2;
318
319=== added file 'mysql-test/suite/binlog/r/binlog_rewrite_db_noleak.result'
320--- mysql-test/suite/binlog/r/binlog_rewrite_db_noleak.result 1970-01-01 00:00:00 +0000
321+++ mysql-test/suite/binlog/r/binlog_rewrite_db_noleak.result 2014-02-15 00:30:14 +0000
322@@ -0,0 +1,29 @@
323+RESET MASTER;
324+CREATE DATABASE db1;
325+USE db1;
326+CREATE TABLE t1 (i INT);
327+INSERT INTO db1.t1 VALUES(1);
328+INSERT INTO db1.t1 VALUES(2);
329+UPDATE t1 SET i= i+1;
330+DELETE FROM t1 WHERE i=2;
331+CREATE DATABASE db2;
332+CREATE TABLE db2.t1 (i INT);
333+INSERT INTO db2.t1 VALUES(3);
334+INSERT INTO db2.t1 VALUES(4);
335+INSERT INTO db2.t1 VALUES(5);
336+UPDATE db2.t1 SET i= i+1;
337+DELETE FROM db2.t1 WHERE i=4;
338+call mtr.add_suppression("Slave SQL: Error executing row event:*");
339+Dropping the database db1 creating the new database db3.
340+DROP DATABASE db1;
341+CREATE DATABASE db3;
342+DROP DATABASE db2;
343+RESET MASTER;
344+[The sql file will be applied on the current database]
345+[The content of table db3.t1 and db2.t1 will be different confirming no leak]
346+include/assert.inc [The content of the table t1 in database db3 and db2 is different]
347+include/assert.inc [Table t1 in db3 have different row count than t1 in db2]
348+CLEANUP
349+DROP DATABASE db1;
350+DROP DATABASE db2;
351+DROP DATABASE db3;
352
353=== added file 'mysql-test/suite/binlog/r/binlog_rewrite_suppress_use.result'
354--- mysql-test/suite/binlog/r/binlog_rewrite_suppress_use.result 1970-01-01 00:00:00 +0000
355+++ mysql-test/suite/binlog/r/binlog_rewrite_suppress_use.result 2014-02-15 00:30:14 +0000
356@@ -0,0 +1,29 @@
357+RESET MASTER;
358+CREATE DATABASE db1;
359+USE db1;
360+CREATE TABLE t1 (i INT);
361+INSERT INTO t1 VALUES(1);
362+INSERT INTO t1 VALUES(2);
363+UPDATE t1 SET i= i+1;
364+DELETE FROM t1 WHERE i=2;
365+[The use <db_name> is not suppressed in the general use of mysqlbinlog]
366+Matching lines are:
367+use `db1`/*!*/;
368+
369+Occurrences of the use `db1` in the input file : 1
370+[The use <db_name> is suppressed on using rewrite-db option of mysqlbinlog]
371+Matching lines are:
372+None
373+Occurrences of the use `db1` in the input file : 0
374+CREATE DATABASE db2;
375+DROP DATABASE db1;
376+RESET MASTER;
377+SELECT * FROM db2.t1;
378+ERROR 42S02: Table 'db2.t1' doesn't exist
379+DROP DATABASE db1;
380+RESET MASTER;
381+[The event of table db1.t1 has been successfully applied to db2.t1]
382+include/assert.inc [Assert that table db2.t1 has no rows after applying the sql file.]
383+[CLEANUP]
384+DROP DATABASE db1;
385+DROP DATABASE db2;
386
387=== added file 'mysql-test/suite/binlog/r/percona_mysqlbinlog_rewritedb.result'
388--- mysql-test/suite/binlog/r/percona_mysqlbinlog_rewritedb.result 1970-01-01 00:00:00 +0000
389+++ mysql-test/suite/binlog/r/percona_mysqlbinlog_rewritedb.result 2014-02-15 00:30:14 +0000
390@@ -0,0 +1,152 @@
391+#
392+# Apply log with ''
393+#
394+# use a
395+f
396+0
397+0
398+# use bbbbbbbbbbbbbbbbbbbb
399+f
400+1
401+1
402+# use cccccccccccccccc
403+f
404+2
405+2
406+# use d
407+f
408+#
409+# Apply log with '--rewrite-db="a->bbbbbbbbbbbbbbbbbbbb" --rewrite-db="cccccccccccccccc->d" --database="d"'
410+#
411+# use a
412+f
413+# use bbbbbbbbbbbbbbbbbbbb
414+f
415+# use cccccccccccccccc
416+f
417+# use d
418+f
419+2
420+2
421+#
422+# Apply log with '--rewrite-db="a->bbbbbbbbbbbbbbbbbbbb" --rewrite-db="cccccccccccccccc->d" --database="cccccccccccccccc"'
423+#
424+# use a
425+f
426+# use bbbbbbbbbbbbbbbbbbbb
427+f
428+# use cccccccccccccccc
429+f
430+# use d
431+f
432+#
433+# Apply log with '--rewrite-db="a->bbbbbbbbbbbbbbbbbbbb" --rewrite-db="cccccccccccccccc->d" --database="bbbbbbbbbbbbbbbbbbbb"'
434+#
435+# use a
436+f
437+# use bbbbbbbbbbbbbbbbbbbb
438+f
439+0
440+0
441+1
442+1
443+# use cccccccccccccccc
444+f
445+# use d
446+f
447+#
448+# Apply log with '--rewrite-db="a->bbbbbbbbbbbbbbbbbbbb" --rewrite-db="cccccccccccccccc->d" --database="a"'
449+#
450+# use a
451+f
452+# use bbbbbbbbbbbbbbbbbbbb
453+f
454+# use cccccccccccccccc
455+f
456+# use d
457+f
458+#
459+# Apply log with '--rewrite-db="a->bbbbbbbbbbbbbbbbbbbb" --rewrite-db="cccccccccccccccc->d"'
460+#
461+# use a
462+f
463+# use bbbbbbbbbbbbbbbbbbbb
464+f
465+0
466+0
467+1
468+1
469+# use cccccccccccccccc
470+f
471+# use d
472+f
473+2
474+2
475+#
476+# Apply log with '--rewrite-db=" a -> bbbbbbbbbbbbbbbbbbbb , cccccccccccccccc -> d "'
477+#
478+# use a
479+f
480+# use bbbbbbbbbbbbbbbbbbbb
481+f
482+0
483+0
484+1
485+1
486+# use cccccccccccccccc
487+f
488+# use d
489+f
490+2
491+2
492+#
493+# Apply log with '--rewrite-db="a->bbbbbbbbbbbbbbbbbbbb,cccccccccccccccc->d"'
494+#
495+# use a
496+f
497+# use bbbbbbbbbbbbbbbbbbbb
498+f
499+0
500+0
501+1
502+1
503+# use cccccccccccccccc
504+f
505+# use d
506+f
507+2
508+2
509+#
510+# Apply log with '--rewrite-db="a->d"'
511+#
512+# use a
513+f
514+# use bbbbbbbbbbbbbbbbbbbb
515+f
516+1
517+1
518+# use cccccccccccccccc
519+f
520+2
521+2
522+# use d
523+f
524+0
525+0
526+#
527+# Apply log with '--rewrite-db="a->bbbbbbbbbbbbbbbbbbbb"'
528+#
529+# use a
530+f
531+# use bbbbbbbbbbbbbbbbbbbb
532+f
533+0
534+0
535+1
536+1
537+# use cccccccccccccccc
538+f
539+2
540+2
541+# use d
542+f
543
544=== added file 'mysql-test/suite/binlog/t/binlog_mysqlbinlog_rewrite_db.test'
545--- mysql-test/suite/binlog/t/binlog_mysqlbinlog_rewrite_db.test 1970-01-01 00:00:00 +0000
546+++ mysql-test/suite/binlog/t/binlog_mysqlbinlog_rewrite_db.test 2014-02-15 00:30:14 +0000
547@@ -0,0 +1,95 @@
548+# WL#6404 - Add rewrite-db option to mysqlbinlog on RBR
549+#
550+# The test checks the new rewrite-db option for the mysqlbinlog.
551+#
552+# The test checks the following aspects of the new option:
553+
554+# 1. The valid syntax for the use of the new option and the errors in
555+# case the usage is not correct.
556+# 2. Apply the new binlog file ( with database rewritten ) to a new database
557+# and check if it works.
558+#
559+--source include/have_binlog_format_row.inc
560+
561+RESET MASTER;
562+CREATE DATABASE db1;
563+USE db1;
564+CREATE TABLE t1 (i INT);
565+
566+# Get some INSERT, UPDATE and DELETE ROW events.
567+INSERT INTO t1 VALUES(1);
568+INSERT INTO t1 VALUES(2);
569+UPDATE t1 SET i= i+1;
570+DELETE FROM t1 WHERE i=2;
571+
572+--let $MYSQLD_DATADIR= `select @@datadir`
573+--exec $MYSQL_BINLOG --force-if-open $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/row_event.sql
574+
575+# Using the new option to apply the row event on some other database (from db1 -> db2 in the current case)
576+
577+--echo [Syntax error in the use of the new option: The from database name is missing]
578+--error 1
579+--exec $MYSQL_BINLOG --force-if-open --rewrite-db="->db2" $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/row_event_rewrite.sql
580+
581+--echo [Syntax error in the use of the new option: The '->' is missing]
582+--error 1
583+--exec $MYSQL_BINLOG --force-if-open --rewrite-db="db1 db2" $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/row_event_rewrite.sql
584+
585+--echo [Syntax error in the use of the new option: The to database name is missing]
586+--error 1
587+--exec $MYSQL_BINLOG --force-if-open --rewrite-db="db1->" $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/row_event_rewrite.sql
588+
589+--echo [VALID SYNTAX,The from->to database names are correctly mentioned, but there is obsolete syntax warning 1]
590+--exec $MYSQL_BINLOG --force-if-open --rewrite-db="db1->db2,db3->db4,db5->db6" $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/row_event_rewrite.sql
591+
592+--echo [Test that warning about obsolete syntax is shown only once for --rewrite-db="db1->db2,db3->db4,db5->db6"]
593+--let log_file=mysqltest.log
594+--let log_file_full_path=$MYSQLTEST_VARDIR/log/current_test
595+--let grep_pattern=The comma-separated list of rewritings syntax is obsolete and discarded
596+--source include/log_grep.inc
597+
598+--echo [VALID SYNTAX,The from->to database names are correctly mentioned, but there is obsolete syntax warning 2]
599+--exec $MYSQL_BINLOG --force-if-open --rewrite-db="db1->db2,db3->db4" --rewrite-db="db5->db6,db6->db7" $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/row_event_rewrite.sql
600+
601+--echo [Test that warning about obsolete syntax is shown only once for --rewrite-db="db1->db2,db3->db4" --rewrite-db="db5->db6,db6->db7"]
602+--let log_file=mysqltest.log
603+--let log_file_full_path=$MYSQLTEST_VARDIR/log/current_test
604+--let grep_pattern=The comma-separated list of rewritings syntax is obsolete and discarded
605+--source include/log_grep.inc
606+
607+--echo [VALID SYNTAX,The from->to database names are correctly mentioned]
608+--exec $MYSQL_BINLOG --force-if-open --rewrite-db="db1->db2" $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/row_event_rewrite.sql
609+
610+--echo #Dropping the database db1 and creating the table in the new database db2.
611+
612+CREATE DATABASE db2;
613+DROP DATABASE db1;
614+
615+# The SQL file will be applied but nothing is applied on Database db2 since the row event was
616+# generated for database db1 and table t1.
617+
618+# With gtid-mode=on we need purge gtid_executed, if not transactions
619+# replayed through mysqlbinlog will be skipped.
620+RESET MASTER;
621+--exec $MYSQL --database=db2 --local-infile=1 < $MYSQLTEST_VARDIR/tmp/row_event.sql
622+--error ER_NO_SUCH_TABLE
623+SELECT * FROM db2.t1;
624+
625+# The SQL file should be applied since the row event was extracted using the new mysqlbinlog option.
626+
627+DROP DATABASE db1;
628+# With gtid-mode=on we need purge gtid_executed, if not transactions
629+# replayed through mysqlbinlog will be skipped.
630+RESET MASTER;
631+--echo [The event of table db1.t1 has been successfully applied to db2.t1]
632+--exec $MYSQL --database=db2 --local-infile=1 < $MYSQLTEST_VARDIR/tmp/row_event_rewrite.sql
633+--let $assert_text= Assert that table db2.t1 has one row after applying the sql file.
634+--let $assert_cond= `SELECT COUNT(*)=1 from db2.t1`
635+--source include/assert.inc
636+
637+--echo CLEANUP
638+
639+--remove_file $MYSQLTEST_VARDIR/tmp/row_event.sql
640+--remove_file $MYSQLTEST_VARDIR/tmp/row_event_rewrite.sql
641+DROP DATABASE db1;
642+DROP DATABASE db2;
643
644=== added file 'mysql-test/suite/binlog/t/binlog_rewrite_db_noleak.test'
645--- mysql-test/suite/binlog/t/binlog_rewrite_db_noleak.test 1970-01-01 00:00:00 +0000
646+++ mysql-test/suite/binlog/t/binlog_rewrite_db_noleak.test 2014-02-15 00:30:14 +0000
647@@ -0,0 +1,79 @@
648+# WL#6404 - Add rewrite-db option to mysqlbinlog on RBR
649+#
650+# The test checks that there is no leak of the global variables used
651+# to decide whether rewrite-db option is ON or OFF.
652+#
653+# The test checks this by creating two tables in two different databases.
654+# insert rows in both of them.
655+#
656+# Read the binlog file using the new option --rewrite-db="db1->db3" and
657+# writing in the .sql file.
658+#
659+# On applying the .sql file on database db3, we get an error since not
660+# all events are converted for database db3. Only DB1 is converted to db3.
661+# The db2 event cause the error.
662+#
663+
664+--source include/have_binlog_format_row.inc
665+RESET MASTER;
666+CREATE DATABASE db1;
667+USE db1;
668+CREATE TABLE t1 (i INT);
669+
670+# Get some INSERT, UPDATE and DELETE ROW events.
671+INSERT INTO db1.t1 VALUES(1);
672+INSERT INTO db1.t1 VALUES(2);
673+UPDATE t1 SET i= i+1;
674+DELETE FROM t1 WHERE i=2;
675+
676+CREATE DATABASE db2;
677+CREATE TABLE db2.t1 (i INT);
678+
679+# Get some INSERT, UPDATE and DELETE ROW events.
680+INSERT INTO db2.t1 VALUES(3);
681+INSERT INTO db2.t1 VALUES(4);
682+INSERT INTO db2.t1 VALUES(5);
683+UPDATE db2.t1 SET i= i+1;
684+DELETE FROM db2.t1 WHERE i=4;
685+
686+--let $MYSQLD_DATADIR= `select @@datadir`
687+
688+# Using the new option to apply the row event on some other database (from db1 -> db3 in the current case)
689+
690+--exec $MYSQL_BINLOG --force-if-open --rewrite-db="db1->db3" $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/rewrite_noleak.sql
691+call mtr.add_suppression("Slave SQL: Error executing row event:*");
692+
693+--echo Dropping the database db1 creating the new database db3.
694+DROP DATABASE db1;
695+CREATE DATABASE db3;
696+
697+# The SQL file will be applied but only db1 events will be replayed on db3.
698+# Database db2 will not be applied on db3 confirming there is no leak.
699+DROP DATABASE db2;
700+
701+# With gtid-mode=on we need purge gtid_executed, if not transactions
702+# replayed through mysqlbinlog will be skipped.
703+RESET MASTER;
704+--echo [The sql file will be applied on the current database]
705+--exec $MYSQL --database=db3 --local-infile=1 < $MYSQLTEST_VARDIR/tmp/rewrite_noleak.sql
706+
707+--echo [The content of table db3.t1 and db2.t1 will be different confirming no leak]
708+
709+--let $db3_max= `SELECT MAX(i) FROM db3.t1`
710+--let $db2_max= `SELECT MAX(i) FROM db2.t1`
711+--let $assert_text= The content of the table t1 in database db3 and db2 is different
712+--let $assert_cond= $db3_max <> $db2_max
713+--source include/assert.inc
714+
715+--let $db3_count= `SELECT COUNT(*) FROM db3.t1`
716+--let $db2_count= `SELECT COUNT(*) FROM db2.t1`
717+--let $assert_text= Table t1 in db3 have different row count than t1 in db2
718+--let $assert_cond= $db3_count=1 AND $db2_count<>1
719+--source include/assert.inc
720+
721+--echo CLEANUP
722+
723+#--remove_file $MYSQLTEST_VARDIR/tmp/rewrite_noleak.sql
724+DROP DATABASE db1;
725+DROP DATABASE db2;
726+DROP DATABASE db3;
727
728=== added file 'mysql-test/suite/binlog/t/binlog_rewrite_suppress_use.test'
729--- mysql-test/suite/binlog/t/binlog_rewrite_suppress_use.test 1970-01-01 00:00:00 +0000
730+++ mysql-test/suite/binlog/t/binlog_rewrite_suppress_use.test 2014-02-15 00:30:14 +0000
731@@ -0,0 +1,61 @@
732+# WL#6404 - Add rewrite-db option to mysqlbinlog on RBR
733+#
734+# The test aims to check that the use of rewrite-db option of
735+# mysqlbinlog suppresses the USE DATABASE command logged in
736+# binlog file.
737+#
738+
739+--source include/have_binlog_format_row.inc
740+RESET MASTER;
741+CREATE DATABASE db1;
742+USE db1;
743+CREATE TABLE t1 (i INT);
744+
745+# Get some INSERT, UPDATE and DELETE ROW events.
746+INSERT INTO t1 VALUES(1);
747+INSERT INTO t1 VALUES(2);
748+UPDATE t1 SET i= i+1;
749+DELETE FROM t1 WHERE i=2;
750+
751+--let $MYSQLD_DATADIR= `select @@datadir`
752+
753+# Checking for the suppression of the USE DATABASE command on using the new option.
754+# Reading binlog file without the rewrite-db option.
755+--echo [The use <db_name> is not suppressed in the general use of mysqlbinlog]
756+--exec $MYSQL_BINLOG --force-if-open $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/row_event.sql
757+--let GREP_FILE=$MYSQLTEST_VARDIR/tmp/row_event.sql
758+--let GREP_PATTERN=use `db1`
759+--source extra/rpl_tests/grep_pattern.inc
760+
761+# Reading binlog file with the rewrite-db option.
762+--echo [The use <db_name> is suppressed on using rewrite-db option of mysqlbinlog]
763+--exec $MYSQL_BINLOG --force-if-open --rewrite-db="db1->db2" $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/row_event_rewrite.sql
764+--let GREP_FILE=$MYSQLTEST_VARDIR/tmp/row_event_rewrite.sql
765+--let GREP_PATTERN=use `db1`
766+--source extra/rpl_tests/grep_pattern.inc
767+
768+CREATE DATABASE db2;
769+DROP DATABASE db1;
770+# With gtid-mode=on we need purge gtid_executed, if not transactions
771+# replayed through mysqlbinlog will be skipped.
772+RESET MASTER;
773+--exec $MYSQL --database=db2 --local-infile=1 < $MYSQLTEST_VARDIR/tmp/row_event.sql
774+--error ER_NO_SUCH_TABLE
775+SELECT * FROM db2.t1;
776+
777+DROP DATABASE db1;
778+# With gtid-mode=on we need purge gtid_executed, if not transactions
779+# replayed through mysqlbinlog will be skipped.
780+RESET MASTER;
781+--echo [The event of table db1.t1 has been successfully applied to db2.t1]
782+--exec $MYSQL --database=db2 --local-infile=1 < $MYSQLTEST_VARDIR/tmp/row_event_rewrite.sql
783+--let $assert_text= Assert that table db2.t1 has no rows after applying the sql file.
784+--let $assert_cond= `SELECT COUNT(*)=1 from db2.t1`
785+--source include/assert.inc
786+
787+--echo [CLEANUP]
788+--remove_file $MYSQLTEST_VARDIR/tmp/row_event.sql
789+--remove_file $MYSQLTEST_VARDIR/tmp/row_event_rewrite.sql
790+
791+DROP DATABASE db1;
792+DROP DATABASE db2;
793
794=== added file 'mysql-test/suite/binlog/t/percona_mysqlbinlog_rewritedb-master.opt'
795--- mysql-test/suite/binlog/t/percona_mysqlbinlog_rewritedb-master.opt 1970-01-01 00:00:00 +0000
796+++ mysql-test/suite/binlog/t/percona_mysqlbinlog_rewritedb-master.opt 2014-02-15 00:30:14 +0000
797@@ -0,0 +1,1 @@
798+--local-infile=1
799
800=== added file 'mysql-test/suite/binlog/t/percona_mysqlbinlog_rewritedb.test'
801--- mysql-test/suite/binlog/t/percona_mysqlbinlog_rewritedb.test 1970-01-01 00:00:00 +0000
802+++ mysql-test/suite/binlog/t/percona_mysqlbinlog_rewritedb.test 2014-02-15 00:30:14 +0000
803@@ -0,0 +1,80 @@
804+--source include/have_log_bin.inc
805+#
806+# WL #36: http://askmonty.org/worklog/Server-Sprint/?tid=36
807+# This is basic test for --rewrite-db option.
808+# mysqlbinlog --rewrite-db="from->to" should rewrite database name
809+#
810+
811+--disable_query_log
812+
813+# Because RBR event DB rewrite requires shifting of
814+# Table_map_log_event contents around if the DB name length
815+# changes, this case is tested for both shorter and longer
816+# names.
817+
818+--let DB1=a
819+--let DB2=bbbbbbbbbbbbbbbbbbbb
820+--let DB3=cccccccccccccccc
821+--let DB4=d
822+
823+eval CREATE DATABASE $DB1; eval USE $DB1; CREATE TABLE t1(f INT);
824+eval CREATE DATABASE $DB2; eval USE $DB2; CREATE TABLE t1(f INT);
825+eval CREATE DATABASE $DB3; eval USE $DB3; CREATE TABLE t1(f INT);
826+eval CREATE DATABASE $DB4; eval USE $DB4; CREATE TABLE t1(f INT);
827+
828+RESET MASTER;
829+--let MYSQLD_DATA_DIR=`select @@datadir`
830+--let LOAD_DATA_FILE=$MYSQLD_DATA_DIR/table.values
831+--let BIN_LOG_FILE_NAME=query_get_value(show master status, File, 1)
832+--let BIN_LOG_START_POSITION=query_get_value(show master status, Position, 1)
833+eval USE $DB1; INSERT INTO t1 VALUES(0); INSERT INTO t1 VALUES(0);
834+eval USE $DB2; INSERT INTO t1 VALUES(1); INSERT INTO t1 VALUES(1);
835+write_file $LOAD_DATA_FILE;
836+2
837+2
838+EOF
839+eval USE $DB3;
840+--eval LOAD DATA INFILE "$LOAD_DATA_FILE" INTO TABLE t1
841+--let BIN_LOG_STOP_POSITION=query_get_value(show master status, Position, 1)
842+FLUSH LOGS;
843+
844+--let BIN_LOG_FULL_PATH=`SELECT CONCAT("$MYSQLD_DATA_DIR", "$BIN_LOG_FILE_NAME")`
845+
846+--let i=10
847+
848+while ($i)
849+{
850+# Test for different combinations of syntax and db name filtering and rewriting
851+--let REWRITE=`SELECT CASE $i WHEN 10 THEN '' WHEN 9 THEN '--rewrite-db="$DB1->$DB2" --rewrite-db="$DB3->$DB4" --database="$DB4"' WHEN 8 THEN '--rewrite-db="$DB1->$DB2" --rewrite-db="$DB3->$DB4" --database="$DB3"' WHEN 7 THEN '--rewrite-db="$DB1->$DB2" --rewrite-db="$DB3->$DB4" --database="$DB2"' WHEN 6 THEN '--rewrite-db="$DB1->$DB2" --rewrite-db="$DB3->$DB4" --database="$DB1"' WHEN 5 THEN '--rewrite-db="$DB1->$DB2" --rewrite-db="$DB3->$DB4"' WHEN 4 THEN '--rewrite-db=" $DB1 -> $DB2 , $DB3 -> $DB4 "' WHEN 3 THEN '--rewrite-db="$DB1->$DB2,$DB3->$DB4"' WHEN 2 THEN '--rewrite-db="$DB1->$DB4"' WHEN 1 THEN '--rewrite-db="$DB1->$DB2"' END`
852+
853+eval USE $DB1; DELETE FROM t1;
854+eval USE $DB2; DELETE FROM t1;
855+eval USE $DB3; DELETE FROM t1;
856+eval USE $DB4; DELETE FROM t1;
857+
858+--echo #
859+--echo # Apply log with '$REWRITE'
860+--echo #
861+
862+--exec $MYSQL_BINLOG $REWRITE --start-position=$BIN_LOG_START_POSITION --stop-position=$BIN_LOG_STOP_POSITION $BIN_LOG_FULL_PATH | $MYSQL --local-infile=1
863+
864+--echo # use $DB1
865+eval USE $DB1; SELECT * FROM t1 ORDER BY f;
866+--echo # use $DB2
867+eval USE $DB2; SELECT * FROM t1 ORDER BY f;
868+--echo # use $DB3
869+eval USE $DB3; SELECT * FROM t1 ORDER BY f;
870+--echo # use $DB4
871+eval USE $DB4; SELECT * FROM t1 ORDER BY f;
872+
873+dec $i;
874+}
875+
876+eval USE $DB1; DROP TABLE t1; eval DROP DATABASE $DB1;
877+eval USE $DB2; DROP TABLE t1; eval DROP DATABASE $DB2;
878+eval USE $DB3; DROP TABLE t1; eval DROP DATABASE $DB3;
879+eval USE $DB4; DROP TABLE t1; eval DROP DATABASE $DB4;
880+
881+--remove_file $LOAD_DATA_FILE
882+
883+--enable_query_log
884
885=== modified file 'sql/log_event.cc'
886--- sql/log_event.cc 2013-12-16 08:45:31 +0000
887+++ sql/log_event.cc 2014-02-15 00:30:14 +0000
888@@ -11862,6 +11862,108 @@
889 my_free(m_memory);
890 }
891
892+#ifdef MYSQL_CLIENT
893+
894+/*
895+ Rewrite database name for the event to name specified by new_db
896+ SYNOPSIS
897+ new_db Database name to change to
898+ new_len Length
899+ desc Event describing binlog that we're writing to.
900+
901+ DESCRIPTION
902+ Reset db name. This function assumes that temp_buf member contains event
903+ representation taken from a binary log. It resets m_dbnam and m_dblen and
904+ rewrites temp_buf with new db name.
905+
906+ RETURN
907+ 0 - Success
908+ other - Error
909+*/
910+
911+int Table_map_log_event::rewrite_db(const char* new_db, size_t new_len,
912+ const Format_description_log_event* desc)
913+{
914+ DBUG_ENTER("Table_map_log_event::rewrite_db");
915+ DBUG_ASSERT(temp_buf);
916+
917+ uint header_len= min((unsigned)desc->common_header_len,
918+ (unsigned)LOG_EVENT_MINIMAL_HEADER_LEN) + TABLE_MAP_HEADER_LEN;
919+ int len_diff;
920+
921+ if (!(len_diff= new_len - m_dblen))
922+ {
923+ memcpy((void*) (temp_buf + header_len + 1), new_db, m_dblen + 1);
924+ memcpy((void*) m_dbnam, new_db, m_dblen + 1);
925+ DBUG_RETURN(0);
926+ }
927+
928+ // Create new temp_buf
929+ ulong event_cur_len= uint4korr(temp_buf + EVENT_LEN_OFFSET);
930+ ulong event_new_len= event_cur_len + len_diff;
931+ char* new_temp_buf= (char*) my_malloc(event_new_len, MYF(MY_WME));
932+
933+ if (!new_temp_buf)
934+ {
935+ sql_print_error("Table_map_log_event::rewrite_db: "
936+ "failed to allocate new temp_buf (%d bytes required)",
937+ event_new_len);
938+ DBUG_RETURN(-1);
939+ }
940+
941+ // Rewrite temp_buf
942+ char* ptr= new_temp_buf;
943+ ulong cnt= 0;
944+
945+ // Copy header and change event length
946+ memcpy(ptr, temp_buf, header_len);
947+ int4store(ptr + EVENT_LEN_OFFSET, event_new_len);
948+ ptr += header_len;
949+ cnt += header_len;
950+
951+ // Write new db name length and new name
952+ *ptr++ = new_len;
953+ memcpy(ptr, new_db, new_len + 1);
954+ ptr += new_len + 1;
955+ cnt += m_dblen + 2;
956+
957+ // Copy rest part
958+ memcpy(ptr, temp_buf + cnt, event_cur_len - cnt);
959+
960+ // Reregister temp buf
961+ free_temp_buf();
962+ register_temp_buf(new_temp_buf);
963+
964+ // Reset m_dbnam and m_dblen members
965+ m_dblen= new_len;
966+
967+ // m_dbnam resides in m_memory together with m_tblnam and m_coltype
968+ uchar* memory= m_memory;
969+ char const* tblnam= m_tblnam;
970+ uchar* coltype= m_coltype;
971+
972+ m_memory= (uchar*) my_multi_malloc(MYF(MY_WME),
973+ &m_dbnam, (uint) m_dblen + 1,
974+ &m_tblnam, (uint) m_tbllen + 1,
975+ &m_coltype, (uint) m_colcnt,
976+ NullS);
977+
978+ if (!m_memory)
979+ {
980+ sql_print_error("Table_map_log_event::rewrite_db: "
981+ "failed to allocate new m_memory (%d + %d + %d bytes required)",
982+ m_dblen + 1, m_tbllen + 1, m_colcnt);
983+ DBUG_RETURN(-1);
984+ }
985+
986+ memcpy((void*)m_dbnam, new_db, m_dblen + 1);
987+ memcpy((void*)m_tblnam, tblnam, m_tbllen + 1);
988+ memcpy(m_coltype, coltype, m_colcnt);
989+
990+ my_free(memory);
991+ DBUG_RETURN(0);
992+}
993+#endif /* MYSQL_CLIENT */
994 /*
995 Return value is an error code, one of:
996
997
998=== modified file 'sql/log_event.h'
999--- sql/log_event.h 2013-10-23 08:48:28 +0000
1000+++ sql/log_event.h 2014-02-15 00:30:14 +0000
1001@@ -3861,7 +3861,10 @@
1002 const Table_id& get_table_id() const { return m_table_id; }
1003 const char *get_table_name() const { return m_tblnam; }
1004 const char *get_db_name() const { return m_dbnam; }
1005-
1006+#ifdef MYSQL_CLIENT
1007+ int rewrite_db(const char* new_name, size_t new_name_len,
1008+ const Format_description_log_event*);
1009+#endif
1010 virtual Log_event_type get_type_code() { return TABLE_MAP_EVENT; }
1011 virtual bool is_valid() const { return m_memory != NULL; /* we check malloc */ }
1012

Subscribers

People subscribed via source and target branches