Merge lp:~percona-toolkit-dev/percona-toolkit/pt-fingerprint-broken-fingerprinting-of-specific-INSERT-REPLACE-queries-1289516 into lp:percona-toolkit/2.2

Proposed by Frank Cizmich
Status: Rejected
Rejected by: Frank Cizmich
Proposed branch: lp:~percona-toolkit-dev/percona-toolkit/pt-fingerprint-broken-fingerprinting-of-specific-INSERT-REPLACE-queries-1289516
Merge into: lp:percona-toolkit/2.2
Diff against target: 434 lines (+103/-34)
8 files modified
bin/pt-fingerprint (+18/-6)
bin/pt-index-usage (+18/-6)
bin/pt-kill (+18/-6)
bin/pt-query-digest (+2/-2)
bin/pt-table-usage (+18/-6)
bin/pt-upgrade (+18/-6)
lib/QueryRewriter.pm (+2/-2)
t/lib/QueryRewriter.t (+9/-0)
To merge this branch: bzr merge lp:~percona-toolkit-dev/percona-toolkit/pt-fingerprint-broken-fingerprinting-of-specific-INSERT-REPLACE-queries-1289516
Reviewer Review Type Date Requested Status
David Bennett (community) Needs Fixing
Daniel Nichter Pending
Review via email: mp+220869@code.launchpad.net

Description of the change

Changed QueryRewrite's function fingerprint so it isn't confused by string values with parentheses inside.
eg:
insert into foo values(10,"(1),(2)")

To post a comment you must log in.
Revision history for this message
Daniel Nichter (daniel-nichter) wrote :

The test does not seem to address the issue originally reported:

$ pt-fingerprint --query "REPLACE INTO a VALUES('INSERT INTO foo VALUES (1),(2)')"

replace into a values('insert into foo values(?+)

That query is going to be very difficult to fingerprinted, perhaps not even possible because fingerprinting is currently heuristic, pattern-based.

Revision history for this message
Frank Cizmich (frank-cizmich) wrote :

Agreed. The fix only addresses a related subset of the original issue. A
perfect fix might be impossible with regexp, or at least quite complex
and costly.

On 05/27/2014 10:37 PM, Daniel Nichter wrote:
> The test does not seem to address the issue originally reported:
>
> $ pt-fingerprint --query "REPLACE INTO a VALUES('INSERT INTO foo VALUES (1),(2)')"
>
> replace into a values('insert into foo values(?+)
>
> That query is going to be very difficult to fingerprinted, perhaps not even possible because fingerprinting is currently heuristic, pattern-based.

--
Frank Cizmich, Software Engineer, Percona
Tel: +1-888-401-3401 Ext: 630
Skype: percona.fcizmich
Montevideo, Uruguay (UTC -3)
www.percona.com
www.mysqlperformanceblog.com

Revision history for this message
David Bennett (dbpercona) wrote :

Size of duplicate full text indexes is reported as 0.

CREATE TABLE `ai5` (
  `id` int(11) NOT NULL,
  `c1` int(11) DEFAULT NULL,
  `c2` int(11) DEFAULT NULL,
  `c3` varchar(64) DEFAULT NULL,
  PRIMARY KEY (`id`),
  FULLTEXT KEY `c3` (`c3`),
  FULLTEXT KEY `c3_2` (`c3`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1

$ ./pt-duplicate-key-checker --verbose --tables=test.ai5
# ########################################################################
# test.ai5
# ########################################################################

# c3_2 (`c3`)
# PRIMARY (`id`)
# c3 (`c3`)

# c3_2 is a duplicate of c3
# Key definitions:
# FULLTEXT KEY `c3_2` (`c3`)
# FULLTEXT KEY `c3` (`c3`),
# Column types:
# `c3` varchar(64) default null
# To remove this duplicate index, execute:
ALTER TABLE `test`.`ai5` DROP INDEX `c3_2`;

# ########################################################################
# Summary of indexes
# ########################################################################

# Size Duplicate Indexes 0
# Total Duplicate Indexes 1
# Total Indexes 3

review: Needs Fixing
Revision history for this message
David Bennett (dbpercona) wrote :

Comment 630864 made on wrong merge proposal, disregard.

Unmerged revisions

597. By Frank Cizmich

fixes query digest for inserts with values with parentheses inside - issue 1289516

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'bin/pt-fingerprint'
--- bin/pt-fingerprint 2014-02-20 08:10:16 +0000
+++ bin/pt-fingerprint 2014-05-25 01:57:50 +0000
@@ -1529,8 +1529,8 @@
15291529
1530my $olc_re = qr/(?:--|#)[^'"\r\n]*(?=[\r\n]|\Z)/; # One-line comments1530my $olc_re = qr/(?:--|#)[^'"\r\n]*(?=[\r\n]|\Z)/; # One-line comments
1531my $mlc_re = qr#/\*[^!].*?\*/#sm; # But not /*!version */1531my $mlc_re = qr#/\*[^!].*?\*/#sm; # But not /*!version */
1532my $vlc_re = qr#/\*.*?[0-9+].*?\*/#sm; # For SHOW + /*!version */1532my $vlc_re = qr#/\*.*?[0-9]+.*?\*/#sm; # For SHOW + /*!version */
1533my $vlc_rf = qr#^(SHOW).*?/\*![0-9+].*?\*/#sm; # Variation for SHOW1533my $vlc_rf = qr#^(?:SHOW).*?/\*![0-9]+(.*?)\*/#sm; # Variation for SHOW
15341534
15351535
1536sub new {1536sub new {
@@ -1545,7 +1545,8 @@
1545 $query =~ s/$mlc_re//go;1545 $query =~ s/$mlc_re//go;
1546 $query =~ s/$olc_re//go;1546 $query =~ s/$olc_re//go;
1547 if ( $query =~ m/$vlc_rf/i ) { # contains show + version1547 if ( $query =~ m/$vlc_rf/i ) { # contains show + version
1548 $query =~ s/$vlc_re//go;1548 my $qualifier = $1 || '';
1549 $query =~ s/$vlc_re/$qualifier/go;
1549 }1550 }
1550 return $query;1551 return $query;
1551}1552}
@@ -1607,8 +1608,8 @@
1607 && return $query;1608 && return $query;
1608 $query =~ m/\A\s*(call\s+\S+)\(/i1609 $query =~ m/\A\s*(call\s+\S+)\(/i
1609 && return lc($1); # Warning! $1 used, be careful.1610 && return lc($1); # Warning! $1 used, be careful.
1610 if ( my ($beginning) = $query =~ m/\A((?:INSERT|REPLACE)(?: IGNORE)?\s+INTO.+?VALUES\s*\(.*?\))\s*,\s*\(/is ) {1611 if ( my ($beginning) = $query =~ m/\A((?:INSERT|REPLACE)(?: IGNORE)?\s+INTO.+?VALUES\s*\("[^"]*?"|[^"]+?\))\s*,\s*\(/is ) {
1611 $query = $beginning; # Shorten multi-value INSERT statements ASAP1612 $query = $beginning; # Shorten multi-value INSERT statements ASAP
1612 }1613 }
16131614
1614 $query =~ s/$mlc_re//go;1615 $query =~ s/$mlc_re//go;
@@ -1668,6 +1669,13 @@
1668 $query =~ m/\A\s*UNLOCK TABLES/i && return "UNLOCK";1669 $query =~ m/\A\s*UNLOCK TABLES/i && return "UNLOCK";
1669 $query =~ m/\A\s*xa\s+(\S+)/i && return "XA_$1";1670 $query =~ m/\A\s*xa\s+(\S+)/i && return "XA_$1";
16701671
1672 if ( $query =~ m/\A\s*LOAD/i ) {
1673 my ($tbl) = $query =~ m/INTO TABLE\s+(\S+)/i;
1674 $tbl ||= '';
1675 $tbl =~ s/`//g;
1676 return "LOAD DATA $tbl";
1677 }
1678
1671 if ( $query =~ m/\Aadministrator command:/ ) {1679 if ( $query =~ m/\Aadministrator command:/ ) {
1672 $query =~ s/administrator command:/ADMIN/;1680 $query =~ s/administrator command:/ADMIN/;
1673 $query = uc $query;1681 $query = uc $query;
@@ -1680,7 +1688,7 @@
1680 PTDEBUG && _d($query);1688 PTDEBUG && _d($query);
16811689
1682 $query = uc $query;1690 $query = uc $query;
1683 $query =~ s/\s+(?:GLOBAL|SESSION|FULL|STORAGE|ENGINE)\b/ /g;1691 $query =~ s/\s+(?:SESSION|FULL|STORAGE|ENGINE)\b/ /g;
1684 $query =~ s/\s+COUNT[^)]+\)//g;1692 $query =~ s/\s+COUNT[^)]+\)//g;
16851693
1686 $query =~ s/\s+(?:FOR|FROM|LIKE|WHERE|LIMIT|IN)\b.+//ms;1694 $query =~ s/\s+(?:FOR|FROM|LIKE|WHERE|LIMIT|IN)\b.+//ms;
@@ -1695,6 +1703,7 @@
1695 eval $QueryParser::tbl_ident;1703 eval $QueryParser::tbl_ident;
1696 my ( $dds ) = $query =~ /^\s*($QueryParser::data_def_stmts)\b/i;1704 my ( $dds ) = $query =~ /^\s*($QueryParser::data_def_stmts)\b/i;
1697 if ( $dds) {1705 if ( $dds) {
1706 $query =~ s/\s+IF(?:\s+NOT)?\s+EXISTS/ /i;
1698 my ( $obj ) = $query =~ m/$dds.+(DATABASE|TABLE)\b/i;1707 my ( $obj ) = $query =~ m/$dds.+(DATABASE|TABLE)\b/i;
1699 $obj = uc $obj if $obj;1708 $obj = uc $obj if $obj;
1700 PTDEBUG && _d('Data def statment:', $dds, 'obj:', $obj);1709 PTDEBUG && _d('Data def statment:', $dds, 'obj:', $obj);
@@ -1761,6 +1770,9 @@
1761 map { $verbs =~ s/$_/$alias_for{$_}/ } keys %alias_for;1770 map { $verbs =~ s/$_/$alias_for{$_}/ } keys %alias_for;
1762 $query = $verbs;1771 $query = $verbs;
1763 }1772 }
1773 elsif ( $verbs && $verbs =~ m/^LOAD DATA/ ) {
1774 return $verbs;
1775 }
1764 else {1776 else {
1765 my @tables = $self->__distill_tables($query, $table, %args);1777 my @tables = $self->__distill_tables($query, $table, %args);
1766 $query = join(q{ }, $verbs, @tables); 1778 $query = join(q{ }, $verbs, @tables);
17671779
=== modified file 'bin/pt-index-usage'
--- bin/pt-index-usage 2014-02-20 08:10:16 +0000
+++ bin/pt-index-usage 2014-05-25 01:57:50 +0000
@@ -2374,8 +2374,8 @@
23742374
2375my $olc_re = qr/(?:--|#)[^'"\r\n]*(?=[\r\n]|\Z)/; # One-line comments2375my $olc_re = qr/(?:--|#)[^'"\r\n]*(?=[\r\n]|\Z)/; # One-line comments
2376my $mlc_re = qr#/\*[^!].*?\*/#sm; # But not /*!version */2376my $mlc_re = qr#/\*[^!].*?\*/#sm; # But not /*!version */
2377my $vlc_re = qr#/\*.*?[0-9+].*?\*/#sm; # For SHOW + /*!version */2377my $vlc_re = qr#/\*.*?[0-9]+.*?\*/#sm; # For SHOW + /*!version */
2378my $vlc_rf = qr#^(SHOW).*?/\*![0-9+].*?\*/#sm; # Variation for SHOW2378my $vlc_rf = qr#^(?:SHOW).*?/\*![0-9]+(.*?)\*/#sm; # Variation for SHOW
23792379
23802380
2381sub new {2381sub new {
@@ -2390,7 +2390,8 @@
2390 $query =~ s/$mlc_re//go;2390 $query =~ s/$mlc_re//go;
2391 $query =~ s/$olc_re//go;2391 $query =~ s/$olc_re//go;
2392 if ( $query =~ m/$vlc_rf/i ) { # contains show + version2392 if ( $query =~ m/$vlc_rf/i ) { # contains show + version
2393 $query =~ s/$vlc_re//go;2393 my $qualifier = $1 || '';
2394 $query =~ s/$vlc_re/$qualifier/go;
2394 }2395 }
2395 return $query;2396 return $query;
2396}2397}
@@ -2452,8 +2453,8 @@
2452 && return $query;2453 && return $query;
2453 $query =~ m/\A\s*(call\s+\S+)\(/i2454 $query =~ m/\A\s*(call\s+\S+)\(/i
2454 && return lc($1); # Warning! $1 used, be careful.2455 && return lc($1); # Warning! $1 used, be careful.
2455 if ( my ($beginning) = $query =~ m/\A((?:INSERT|REPLACE)(?: IGNORE)?\s+INTO.+?VALUES\s*\(.*?\))\s*,\s*\(/is ) {2456 if ( my ($beginning) = $query =~ m/\A((?:INSERT|REPLACE)(?: IGNORE)?\s+INTO.+?VALUES\s*\("[^"]*?"|[^"]+?\))\s*,\s*\(/is ) {
2456 $query = $beginning; # Shorten multi-value INSERT statements ASAP2457 $query = $beginning; # Shorten multi-value INSERT statements ASAP
2457 }2458 }
24582459
2459 $query =~ s/$mlc_re//go;2460 $query =~ s/$mlc_re//go;
@@ -2513,6 +2514,13 @@
2513 $query =~ m/\A\s*UNLOCK TABLES/i && return "UNLOCK";2514 $query =~ m/\A\s*UNLOCK TABLES/i && return "UNLOCK";
2514 $query =~ m/\A\s*xa\s+(\S+)/i && return "XA_$1";2515 $query =~ m/\A\s*xa\s+(\S+)/i && return "XA_$1";
25152516
2517 if ( $query =~ m/\A\s*LOAD/i ) {
2518 my ($tbl) = $query =~ m/INTO TABLE\s+(\S+)/i;
2519 $tbl ||= '';
2520 $tbl =~ s/`//g;
2521 return "LOAD DATA $tbl";
2522 }
2523
2516 if ( $query =~ m/\Aadministrator command:/ ) {2524 if ( $query =~ m/\Aadministrator command:/ ) {
2517 $query =~ s/administrator command:/ADMIN/;2525 $query =~ s/administrator command:/ADMIN/;
2518 $query = uc $query;2526 $query = uc $query;
@@ -2525,7 +2533,7 @@
2525 PTDEBUG && _d($query);2533 PTDEBUG && _d($query);
25262534
2527 $query = uc $query;2535 $query = uc $query;
2528 $query =~ s/\s+(?:GLOBAL|SESSION|FULL|STORAGE|ENGINE)\b/ /g;2536 $query =~ s/\s+(?:SESSION|FULL|STORAGE|ENGINE)\b/ /g;
2529 $query =~ s/\s+COUNT[^)]+\)//g;2537 $query =~ s/\s+COUNT[^)]+\)//g;
25302538
2531 $query =~ s/\s+(?:FOR|FROM|LIKE|WHERE|LIMIT|IN)\b.+//ms;2539 $query =~ s/\s+(?:FOR|FROM|LIKE|WHERE|LIMIT|IN)\b.+//ms;
@@ -2540,6 +2548,7 @@
2540 eval $QueryParser::tbl_ident;2548 eval $QueryParser::tbl_ident;
2541 my ( $dds ) = $query =~ /^\s*($QueryParser::data_def_stmts)\b/i;2549 my ( $dds ) = $query =~ /^\s*($QueryParser::data_def_stmts)\b/i;
2542 if ( $dds) {2550 if ( $dds) {
2551 $query =~ s/\s+IF(?:\s+NOT)?\s+EXISTS/ /i;
2543 my ( $obj ) = $query =~ m/$dds.+(DATABASE|TABLE)\b/i;2552 my ( $obj ) = $query =~ m/$dds.+(DATABASE|TABLE)\b/i;
2544 $obj = uc $obj if $obj;2553 $obj = uc $obj if $obj;
2545 PTDEBUG && _d('Data def statment:', $dds, 'obj:', $obj);2554 PTDEBUG && _d('Data def statment:', $dds, 'obj:', $obj);
@@ -2606,6 +2615,9 @@
2606 map { $verbs =~ s/$_/$alias_for{$_}/ } keys %alias_for;2615 map { $verbs =~ s/$_/$alias_for{$_}/ } keys %alias_for;
2607 $query = $verbs;2616 $query = $verbs;
2608 }2617 }
2618 elsif ( $verbs && $verbs =~ m/^LOAD DATA/ ) {
2619 return $verbs;
2620 }
2609 else {2621 else {
2610 my @tables = $self->__distill_tables($query, $table, %args);2622 my @tables = $self->__distill_tables($query, $table, %args);
2611 $query = join(q{ }, $verbs, @tables); 2623 $query = join(q{ }, $verbs, @tables);
26122624
=== modified file 'bin/pt-kill'
--- bin/pt-kill 2014-02-20 08:10:16 +0000
+++ bin/pt-kill 2014-05-25 01:57:50 +0000
@@ -4652,8 +4652,8 @@
46524652
4653my $olc_re = qr/(?:--|#)[^'"\r\n]*(?=[\r\n]|\Z)/; # One-line comments4653my $olc_re = qr/(?:--|#)[^'"\r\n]*(?=[\r\n]|\Z)/; # One-line comments
4654my $mlc_re = qr#/\*[^!].*?\*/#sm; # But not /*!version */4654my $mlc_re = qr#/\*[^!].*?\*/#sm; # But not /*!version */
4655my $vlc_re = qr#/\*.*?[0-9+].*?\*/#sm; # For SHOW + /*!version */4655my $vlc_re = qr#/\*.*?[0-9]+.*?\*/#sm; # For SHOW + /*!version */
4656my $vlc_rf = qr#^(SHOW).*?/\*![0-9+].*?\*/#sm; # Variation for SHOW4656my $vlc_rf = qr#^(?:SHOW).*?/\*![0-9]+(.*?)\*/#sm; # Variation for SHOW
46574657
46584658
4659sub new {4659sub new {
@@ -4668,7 +4668,8 @@
4668 $query =~ s/$mlc_re//go;4668 $query =~ s/$mlc_re//go;
4669 $query =~ s/$olc_re//go;4669 $query =~ s/$olc_re//go;
4670 if ( $query =~ m/$vlc_rf/i ) { # contains show + version4670 if ( $query =~ m/$vlc_rf/i ) { # contains show + version
4671 $query =~ s/$vlc_re//go;4671 my $qualifier = $1 || '';
4672 $query =~ s/$vlc_re/$qualifier/go;
4672 }4673 }
4673 return $query;4674 return $query;
4674}4675}
@@ -4730,8 +4731,8 @@
4730 && return $query;4731 && return $query;
4731 $query =~ m/\A\s*(call\s+\S+)\(/i4732 $query =~ m/\A\s*(call\s+\S+)\(/i
4732 && return lc($1); # Warning! $1 used, be careful.4733 && return lc($1); # Warning! $1 used, be careful.
4733 if ( my ($beginning) = $query =~ m/\A((?:INSERT|REPLACE)(?: IGNORE)?\s+INTO.+?VALUES\s*\(.*?\))\s*,\s*\(/is ) {4734 if ( my ($beginning) = $query =~ m/\A((?:INSERT|REPLACE)(?: IGNORE)?\s+INTO.+?VALUES\s*\("[^"]*?"|[^"]+?\))\s*,\s*\(/is ) {
4734 $query = $beginning; # Shorten multi-value INSERT statements ASAP4735 $query = $beginning; # Shorten multi-value INSERT statements ASAP
4735 }4736 }
47364737
4737 $query =~ s/$mlc_re//go;4738 $query =~ s/$mlc_re//go;
@@ -4791,6 +4792,13 @@
4791 $query =~ m/\A\s*UNLOCK TABLES/i && return "UNLOCK";4792 $query =~ m/\A\s*UNLOCK TABLES/i && return "UNLOCK";
4792 $query =~ m/\A\s*xa\s+(\S+)/i && return "XA_$1";4793 $query =~ m/\A\s*xa\s+(\S+)/i && return "XA_$1";
47934794
4795 if ( $query =~ m/\A\s*LOAD/i ) {
4796 my ($tbl) = $query =~ m/INTO TABLE\s+(\S+)/i;
4797 $tbl ||= '';
4798 $tbl =~ s/`//g;
4799 return "LOAD DATA $tbl";
4800 }
4801
4794 if ( $query =~ m/\Aadministrator command:/ ) {4802 if ( $query =~ m/\Aadministrator command:/ ) {
4795 $query =~ s/administrator command:/ADMIN/;4803 $query =~ s/administrator command:/ADMIN/;
4796 $query = uc $query;4804 $query = uc $query;
@@ -4803,7 +4811,7 @@
4803 PTDEBUG && _d($query);4811 PTDEBUG && _d($query);
48044812
4805 $query = uc $query;4813 $query = uc $query;
4806 $query =~ s/\s+(?:GLOBAL|SESSION|FULL|STORAGE|ENGINE)\b/ /g;4814 $query =~ s/\s+(?:SESSION|FULL|STORAGE|ENGINE)\b/ /g;
4807 $query =~ s/\s+COUNT[^)]+\)//g;4815 $query =~ s/\s+COUNT[^)]+\)//g;
48084816
4809 $query =~ s/\s+(?:FOR|FROM|LIKE|WHERE|LIMIT|IN)\b.+//ms;4817 $query =~ s/\s+(?:FOR|FROM|LIKE|WHERE|LIMIT|IN)\b.+//ms;
@@ -4818,6 +4826,7 @@
4818 eval $QueryParser::tbl_ident;4826 eval $QueryParser::tbl_ident;
4819 my ( $dds ) = $query =~ /^\s*($QueryParser::data_def_stmts)\b/i;4827 my ( $dds ) = $query =~ /^\s*($QueryParser::data_def_stmts)\b/i;
4820 if ( $dds) {4828 if ( $dds) {
4829 $query =~ s/\s+IF(?:\s+NOT)?\s+EXISTS/ /i;
4821 my ( $obj ) = $query =~ m/$dds.+(DATABASE|TABLE)\b/i;4830 my ( $obj ) = $query =~ m/$dds.+(DATABASE|TABLE)\b/i;
4822 $obj = uc $obj if $obj;4831 $obj = uc $obj if $obj;
4823 PTDEBUG && _d('Data def statment:', $dds, 'obj:', $obj);4832 PTDEBUG && _d('Data def statment:', $dds, 'obj:', $obj);
@@ -4884,6 +4893,9 @@
4884 map { $verbs =~ s/$_/$alias_for{$_}/ } keys %alias_for;4893 map { $verbs =~ s/$_/$alias_for{$_}/ } keys %alias_for;
4885 $query = $verbs;4894 $query = $verbs;
4886 }4895 }
4896 elsif ( $verbs && $verbs =~ m/^LOAD DATA/ ) {
4897 return $verbs;
4898 }
4887 else {4899 else {
4888 my @tables = $self->__distill_tables($query, $table, %args);4900 my @tables = $self->__distill_tables($query, $table, %args);
4889 $query = join(q{ }, $verbs, @tables); 4901 $query = join(q{ }, $verbs, @tables);
48904902
=== modified file 'bin/pt-query-digest'
--- bin/pt-query-digest 2014-02-20 08:10:16 +0000
+++ bin/pt-query-digest 2014-05-25 01:57:50 +0000
@@ -2878,8 +2878,8 @@
2878 && return $query;2878 && return $query;
2879 $query =~ m/\A\s*(call\s+\S+)\(/i2879 $query =~ m/\A\s*(call\s+\S+)\(/i
2880 && return lc($1); # Warning! $1 used, be careful.2880 && return lc($1); # Warning! $1 used, be careful.
2881 if ( my ($beginning) = $query =~ m/\A((?:INSERT|REPLACE)(?: IGNORE)?\s+INTO.+?VALUES\s*\(.*?\))\s*,\s*\(/is ) {2881 if ( my ($beginning) = $query =~ m/\A((?:INSERT|REPLACE)(?: IGNORE)?\s+INTO.+?VALUES\s*\("[^"]*?"|[^"]+?\))\s*,\s*\(/is ) {
2882 $query = $beginning; # Shorten multi-value INSERT statements ASAP2882 $query = $beginning; # Shorten multi-value INSERT statements ASAP
2883 }2883 }
28842884
2885 $query =~ s/$mlc_re//go;2885 $query =~ s/$mlc_re//go;
28862886
=== modified file 'bin/pt-table-usage'
--- bin/pt-table-usage 2014-02-20 08:10:16 +0000
+++ bin/pt-table-usage 2014-05-25 01:57:50 +0000
@@ -2112,8 +2112,8 @@
21122112
2113my $olc_re = qr/(?:--|#)[^'"\r\n]*(?=[\r\n]|\Z)/; # One-line comments2113my $olc_re = qr/(?:--|#)[^'"\r\n]*(?=[\r\n]|\Z)/; # One-line comments
2114my $mlc_re = qr#/\*[^!].*?\*/#sm; # But not /*!version */2114my $mlc_re = qr#/\*[^!].*?\*/#sm; # But not /*!version */
2115my $vlc_re = qr#/\*.*?[0-9+].*?\*/#sm; # For SHOW + /*!version */2115my $vlc_re = qr#/\*.*?[0-9]+.*?\*/#sm; # For SHOW + /*!version */
2116my $vlc_rf = qr#^(SHOW).*?/\*![0-9+].*?\*/#sm; # Variation for SHOW2116my $vlc_rf = qr#^(?:SHOW).*?/\*![0-9]+(.*?)\*/#sm; # Variation for SHOW
21172117
21182118
2119sub new {2119sub new {
@@ -2128,7 +2128,8 @@
2128 $query =~ s/$mlc_re//go;2128 $query =~ s/$mlc_re//go;
2129 $query =~ s/$olc_re//go;2129 $query =~ s/$olc_re//go;
2130 if ( $query =~ m/$vlc_rf/i ) { # contains show + version2130 if ( $query =~ m/$vlc_rf/i ) { # contains show + version
2131 $query =~ s/$vlc_re//go;2131 my $qualifier = $1 || '';
2132 $query =~ s/$vlc_re/$qualifier/go;
2132 }2133 }
2133 return $query;2134 return $query;
2134}2135}
@@ -2190,8 +2191,8 @@
2190 && return $query;2191 && return $query;
2191 $query =~ m/\A\s*(call\s+\S+)\(/i2192 $query =~ m/\A\s*(call\s+\S+)\(/i
2192 && return lc($1); # Warning! $1 used, be careful.2193 && return lc($1); # Warning! $1 used, be careful.
2193 if ( my ($beginning) = $query =~ m/\A((?:INSERT|REPLACE)(?: IGNORE)?\s+INTO.+?VALUES\s*\(.*?\))\s*,\s*\(/is ) {2194 if ( my ($beginning) = $query =~ m/\A((?:INSERT|REPLACE)(?: IGNORE)?\s+INTO.+?VALUES\s*\("[^"]*?"|[^"]+?\))\s*,\s*\(/is ) {
2194 $query = $beginning; # Shorten multi-value INSERT statements ASAP2195 $query = $beginning; # Shorten multi-value INSERT statements ASAP
2195 }2196 }
21962197
2197 $query =~ s/$mlc_re//go;2198 $query =~ s/$mlc_re//go;
@@ -2251,6 +2252,13 @@
2251 $query =~ m/\A\s*UNLOCK TABLES/i && return "UNLOCK";2252 $query =~ m/\A\s*UNLOCK TABLES/i && return "UNLOCK";
2252 $query =~ m/\A\s*xa\s+(\S+)/i && return "XA_$1";2253 $query =~ m/\A\s*xa\s+(\S+)/i && return "XA_$1";
22532254
2255 if ( $query =~ m/\A\s*LOAD/i ) {
2256 my ($tbl) = $query =~ m/INTO TABLE\s+(\S+)/i;
2257 $tbl ||= '';
2258 $tbl =~ s/`//g;
2259 return "LOAD DATA $tbl";
2260 }
2261
2254 if ( $query =~ m/\Aadministrator command:/ ) {2262 if ( $query =~ m/\Aadministrator command:/ ) {
2255 $query =~ s/administrator command:/ADMIN/;2263 $query =~ s/administrator command:/ADMIN/;
2256 $query = uc $query;2264 $query = uc $query;
@@ -2263,7 +2271,7 @@
2263 PTDEBUG && _d($query);2271 PTDEBUG && _d($query);
22642272
2265 $query = uc $query;2273 $query = uc $query;
2266 $query =~ s/\s+(?:GLOBAL|SESSION|FULL|STORAGE|ENGINE)\b/ /g;2274 $query =~ s/\s+(?:SESSION|FULL|STORAGE|ENGINE)\b/ /g;
2267 $query =~ s/\s+COUNT[^)]+\)//g;2275 $query =~ s/\s+COUNT[^)]+\)//g;
22682276
2269 $query =~ s/\s+(?:FOR|FROM|LIKE|WHERE|LIMIT|IN)\b.+//ms;2277 $query =~ s/\s+(?:FOR|FROM|LIKE|WHERE|LIMIT|IN)\b.+//ms;
@@ -2278,6 +2286,7 @@
2278 eval $QueryParser::tbl_ident;2286 eval $QueryParser::tbl_ident;
2279 my ( $dds ) = $query =~ /^\s*($QueryParser::data_def_stmts)\b/i;2287 my ( $dds ) = $query =~ /^\s*($QueryParser::data_def_stmts)\b/i;
2280 if ( $dds) {2288 if ( $dds) {
2289 $query =~ s/\s+IF(?:\s+NOT)?\s+EXISTS/ /i;
2281 my ( $obj ) = $query =~ m/$dds.+(DATABASE|TABLE)\b/i;2290 my ( $obj ) = $query =~ m/$dds.+(DATABASE|TABLE)\b/i;
2282 $obj = uc $obj if $obj;2291 $obj = uc $obj if $obj;
2283 PTDEBUG && _d('Data def statment:', $dds, 'obj:', $obj);2292 PTDEBUG && _d('Data def statment:', $dds, 'obj:', $obj);
@@ -2344,6 +2353,9 @@
2344 map { $verbs =~ s/$_/$alias_for{$_}/ } keys %alias_for;2353 map { $verbs =~ s/$_/$alias_for{$_}/ } keys %alias_for;
2345 $query = $verbs;2354 $query = $verbs;
2346 }2355 }
2356 elsif ( $verbs && $verbs =~ m/^LOAD DATA/ ) {
2357 return $verbs;
2358 }
2347 else {2359 else {
2348 my @tables = $self->__distill_tables($query, $table, %args);2360 my @tables = $self->__distill_tables($query, $table, %args);
2349 $query = join(q{ }, $verbs, @tables); 2361 $query = join(q{ }, $verbs, @tables);
23502362
=== modified file 'bin/pt-upgrade'
--- bin/pt-upgrade 2014-02-20 08:10:16 +0000
+++ bin/pt-upgrade 2014-05-25 01:57:50 +0000
@@ -4559,8 +4559,8 @@
45594559
4560my $olc_re = qr/(?:--|#)[^'"\r\n]*(?=[\r\n]|\Z)/; # One-line comments4560my $olc_re = qr/(?:--|#)[^'"\r\n]*(?=[\r\n]|\Z)/; # One-line comments
4561my $mlc_re = qr#/\*[^!].*?\*/#sm; # But not /*!version */4561my $mlc_re = qr#/\*[^!].*?\*/#sm; # But not /*!version */
4562my $vlc_re = qr#/\*.*?[0-9+].*?\*/#sm; # For SHOW + /*!version */4562my $vlc_re = qr#/\*.*?[0-9]+.*?\*/#sm; # For SHOW + /*!version */
4563my $vlc_rf = qr#^(SHOW).*?/\*![0-9+].*?\*/#sm; # Variation for SHOW4563my $vlc_rf = qr#^(?:SHOW).*?/\*![0-9]+(.*?)\*/#sm; # Variation for SHOW
45644564
45654565
4566sub new {4566sub new {
@@ -4575,7 +4575,8 @@
4575 $query =~ s/$mlc_re//go;4575 $query =~ s/$mlc_re//go;
4576 $query =~ s/$olc_re//go;4576 $query =~ s/$olc_re//go;
4577 if ( $query =~ m/$vlc_rf/i ) { # contains show + version4577 if ( $query =~ m/$vlc_rf/i ) { # contains show + version
4578 $query =~ s/$vlc_re//go;4578 my $qualifier = $1 || '';
4579 $query =~ s/$vlc_re/$qualifier/go;
4579 }4580 }
4580 return $query;4581 return $query;
4581}4582}
@@ -4637,8 +4638,8 @@
4637 && return $query;4638 && return $query;
4638 $query =~ m/\A\s*(call\s+\S+)\(/i4639 $query =~ m/\A\s*(call\s+\S+)\(/i
4639 && return lc($1); # Warning! $1 used, be careful.4640 && return lc($1); # Warning! $1 used, be careful.
4640 if ( my ($beginning) = $query =~ m/\A((?:INSERT|REPLACE)(?: IGNORE)?\s+INTO.+?VALUES\s*\(.*?\))\s*,\s*\(/is ) {4641 if ( my ($beginning) = $query =~ m/\A((?:INSERT|REPLACE)(?: IGNORE)?\s+INTO.+?VALUES\s*\("[^"]*?"|[^"]+?\))\s*,\s*\(/is ) {
4641 $query = $beginning; # Shorten multi-value INSERT statements ASAP4642 $query = $beginning; # Shorten multi-value INSERT statements ASAP
4642 }4643 }
46434644
4644 $query =~ s/$mlc_re//go;4645 $query =~ s/$mlc_re//go;
@@ -4698,6 +4699,13 @@
4698 $query =~ m/\A\s*UNLOCK TABLES/i && return "UNLOCK";4699 $query =~ m/\A\s*UNLOCK TABLES/i && return "UNLOCK";
4699 $query =~ m/\A\s*xa\s+(\S+)/i && return "XA_$1";4700 $query =~ m/\A\s*xa\s+(\S+)/i && return "XA_$1";
47004701
4702 if ( $query =~ m/\A\s*LOAD/i ) {
4703 my ($tbl) = $query =~ m/INTO TABLE\s+(\S+)/i;
4704 $tbl ||= '';
4705 $tbl =~ s/`//g;
4706 return "LOAD DATA $tbl";
4707 }
4708
4701 if ( $query =~ m/\Aadministrator command:/ ) {4709 if ( $query =~ m/\Aadministrator command:/ ) {
4702 $query =~ s/administrator command:/ADMIN/;4710 $query =~ s/administrator command:/ADMIN/;
4703 $query = uc $query;4711 $query = uc $query;
@@ -4710,7 +4718,7 @@
4710 PTDEBUG && _d($query);4718 PTDEBUG && _d($query);
47114719
4712 $query = uc $query;4720 $query = uc $query;
4713 $query =~ s/\s+(?:GLOBAL|SESSION|FULL|STORAGE|ENGINE)\b/ /g;4721 $query =~ s/\s+(?:SESSION|FULL|STORAGE|ENGINE)\b/ /g;
4714 $query =~ s/\s+COUNT[^)]+\)//g;4722 $query =~ s/\s+COUNT[^)]+\)//g;
47154723
4716 $query =~ s/\s+(?:FOR|FROM|LIKE|WHERE|LIMIT|IN)\b.+//ms;4724 $query =~ s/\s+(?:FOR|FROM|LIKE|WHERE|LIMIT|IN)\b.+//ms;
@@ -4725,6 +4733,7 @@
4725 eval $QueryParser::tbl_ident;4733 eval $QueryParser::tbl_ident;
4726 my ( $dds ) = $query =~ /^\s*($QueryParser::data_def_stmts)\b/i;4734 my ( $dds ) = $query =~ /^\s*($QueryParser::data_def_stmts)\b/i;
4727 if ( $dds) {4735 if ( $dds) {
4736 $query =~ s/\s+IF(?:\s+NOT)?\s+EXISTS/ /i;
4728 my ( $obj ) = $query =~ m/$dds.+(DATABASE|TABLE)\b/i;4737 my ( $obj ) = $query =~ m/$dds.+(DATABASE|TABLE)\b/i;
4729 $obj = uc $obj if $obj;4738 $obj = uc $obj if $obj;
4730 PTDEBUG && _d('Data def statment:', $dds, 'obj:', $obj);4739 PTDEBUG && _d('Data def statment:', $dds, 'obj:', $obj);
@@ -4791,6 +4800,9 @@
4791 map { $verbs =~ s/$_/$alias_for{$_}/ } keys %alias_for;4800 map { $verbs =~ s/$_/$alias_for{$_}/ } keys %alias_for;
4792 $query = $verbs;4801 $query = $verbs;
4793 }4802 }
4803 elsif ( $verbs && $verbs =~ m/^LOAD DATA/ ) {
4804 return $verbs;
4805 }
4794 else {4806 else {
4795 my @tables = $self->__distill_tables($query, $table, %args);4807 my @tables = $self->__distill_tables($query, $table, %args);
4796 $query = join(q{ }, $verbs, @tables); 4808 $query = join(q{ }, $verbs, @tables);
47974809
=== modified file 'lib/QueryRewriter.pm'
--- lib/QueryRewriter.pm 2013-09-27 02:16:19 +0000
+++ lib/QueryRewriter.pm 2014-05-25 01:57:50 +0000
@@ -164,8 +164,8 @@
164 # mysqldump's INSERT statements will have long values() lists, don't waste164 # mysqldump's INSERT statements will have long values() lists, don't waste
165 # time on them... they also tend to segfault Perl on some machines when you165 # time on them... they also tend to segfault Perl on some machines when you
166 # get to the "# Collapse IN() and VALUES() lists" regex below!166 # get to the "# Collapse IN() and VALUES() lists" regex below!
167 if ( my ($beginning) = $query =~ m/\A((?:INSERT|REPLACE)(?: IGNORE)?\s+INTO.+?VALUES\s*\(.*?\))\s*,\s*\(/is ) {167 if ( my ($beginning) = $query =~ m/\A((?:INSERT|REPLACE)(?: IGNORE)?\s+INTO.+?VALUES\s*\("[^"]*?"|[^"]+?\))\s*,\s*\(/is ) {
168 $query = $beginning; # Shorten multi-value INSERT statements ASAP168 $query = $beginning; # Shorten multi-value INSERT statements ASAP
169 }169 }
170170
171 $query =~ s/$mlc_re//go;171 $query =~ s/$mlc_re//go;
172172
=== modified file 't/lib/QueryRewriter.t'
--- t/lib/QueryRewriter.t 2013-09-27 02:16:19 +0000
+++ t/lib/QueryRewriter.t 2014-05-25 01:57:50 +0000
@@ -285,6 +285,15 @@
285 'union all fingerprints together',285 'union all fingerprints together',
286);286);
287287
288# Issue 1289516
289is(
290 $qr->fingerprint(q[insert into foo values(1,"(2),(3)",4,"(5),(6)")]),
291 'insert into foo values(?+)',
292 'not confused by parentheses inside a quoted string',
293);
294
295
296
288# Issue 322: mk-query-digest segfault before report297# Issue 322: mk-query-digest segfault before report
289is(298is(
290 $qr->fingerprint( load_file('t/lib/samples/huge_replace_into_values.txt') ),299 $qr->fingerprint( load_file('t/lib/samples/huge_replace_into_values.txt') ),

Subscribers

People subscribed via source and target branches