Merge lp:~percona-toolkit-dev/percona-toolkit/fix-bug-903379 into lp:percona-toolkit/2.0

[Brian Fraser]

The fix itself is very simple, so I've probably overcomplicated things. Still, I think this approach has its advantages, since it doesn't punish users unlucky enough not to have Encode, but thorough enough to be using standardized charset names.

[lp:~percona-toolkit-dev/percona-toolkit/fix-bug-903379]

209. By Daniel Nichter on 2012-03-07

Fix samples for MySQL 5.0.

210. By Daniel Nichter on 2012-03-07

Fix trailing blank lines in samples.

211. By Daniel Nichter on 2012-03-07

Set TZ to make tests work on other boxes.

212. By Daniel Nichter on 2012-03-07

Update Daemon in all tools (bug 944420).

213. By Daniel Nichter on 2012-03-07

Update Changelog for 2.0.4.

214. By Daniel Nichter on 2012-03-07

Fix quoting in util/build-packages.

215. By Daniel Nichter on 2012-03-07

Fix more quoting in util/build-packages.

216. By Daniel Nichter on 2012-03-07

Fix 'Bad substitution' error in util/build-packages.

217. By Daniel Nichter on 2012-03-07

Build percona-toolkit-2.0.4

218. By Daniel Nichter on 2012-03-26

Merge lp:~percona-toolkit-dev/percona-toolkit/doc-spelling-fixes-2.1-bug-896553.

219. By Daniel Nichter on 2012-03-27

Merge lp:~percona-toolkit-dev/percona-toolkit/update-relay-log-space-limit-docs-fix-bug-949154.

220. By Daniel Nichter on 2012-03-30

Merge fix-risks-docs-bug-949653.

221. By Daniel Nichter on 2012-03-30

Merge fix-quoting-bug-967451.

222. By Daniel Nichter on 2012-03-30

Merge fix-pt-upgrade-docs-bug-953461.

223. By Daniel Nichter on 2012-03-30

Merge fix-no-stalk-bug-955860.

224. By Daniel Nichter on 2012-03-30

Add pt-fingerprint.

225. By Daniel Nichter on 2012-03-30

Add pt-table-usage and update SQLParser.pm from Maatkit.

226. By Daniel Nichter on 2012-04-02

Merge pt-table-usage-docs.

227. By Daniel Nichter on 2012-04-03

Merge fix-table-status-bug-960513.

228. By Daniel Nichter on 2012-04-03

Merge pt-osc-2.1. Fix NibbleItertor.pm to work with OobNibbleIterator when resuming at oob boundaries (t/pt-table-checksum/resume.t was failing).

229. By Daniel Nichter on 2012-04-03

Merge fix-doc-bug-972399.

230. By Daniel Nichter on 2012-04-03

Merge fix-log-parser-writer-bug-963225.

231. By Daniel Nichter on 2012-04-03

Merge summary-tools-2.1.

232. By Daniel Nichter on 2012-04-03

Remove a bzr backup file.

233. By Daniel Nichter on 2012-04-03

Convert TableUsage.t from Maatkit to Percona Toolkit.

234. By Daniel Nichter on 2012-04-03

Add func to col ident struct to fix QueryAdvisorRules.t.

235. By Daniel Nichter on 2012-04-03

Fix SchemaIterator.t by updating and adding 5.0 samples.

236. By Daniel Nichter on 2012-04-03

Update modules in pt-query-advisor, replacing MySQLDump with TableParser.

237. By Daniel Nichter on 2012-04-03

Make pt-stalk.t a little more reliable.

238. By Daniel Nichter on 2012-04-03

Remove unused developer test file.

239. By Daniel Nichter on 2012-04-03

Remove useless var in SchemaIterator::_iterate_files() and update module in tools.

240. By Daniel Nichter on 2012-04-03

Update Changelog for 2.1.1.

241. By Daniel Nichter on 2012-04-03

Build percona-toolkit-2.1.1

242. By Daniel Nichter on 2012-04-17

Merged fix-sleep-bug-979092.

243. By Daniel Nichter on 2012-04-26

Merge fix-debug-bug-989227.

244. By Daniel Nichter on 2012-05-04

Merge ../fix-ptc-innodb-bug-994010.

245. By Daniel Nichter on 2012-05-07

Merge ../dis-fk-chk-bug-976108.

246. By Daniel Nichter on 2012-05-07

Merge ../doc-osc-limitation-bug-976109.

247. By Daniel Nichter on 2012-05-07

Merge ../fix-summay-size-bug-993436.

248. By Daniel Nichter on 2012-05-08

Merge ../fix-ptc-pk-bug-978432.

249. By Daniel Nichter on 2012-05-08

Merge fix-undef-arrayref-bug-995274.

250. By Daniel Nichter on 2012-05-09

Merge change-diskstats-group-by-default-fix-bug-952727.

251. By Daniel Nichter on 2012-05-10

Merge fix-empty-table-bug-987393.

252. By Daniel Nichter on 2012-05-11

Merge fix-diskstats-hdr-bug-994176.

253. By Daniel Nichter on 2012-05-15

Merge fix-dupe-key-bug-894140.

254. By Daniel Nichter on 2012-05-15

Merge fix-stalk-cycles-bug-994947.

255. By Daniel Nichter on 2012-05-16

Merge fix-osc-index-bug-994002.

256. By Daniel Nichter on 2012-05-16

Merge fix-mktemp-bug-986151.

257. By Daniel Nichter on 2012-05-16

Merge fix-osc-innodb-bug-996110.

258. By Daniel Nichter on 2012-05-17

Merge recursion-method-none-bug-987694.

259. By Daniel Nichter on 2012-05-18

Merge fix-ptc-mysql-5.5-bug-919499.

260. By Daniel Nichter on 2012-05-21

Merge fix-sync-ignore-bug-1002365.

261. By Daniel Nichter on 2012-05-22

Merge fix-sync-index-bug-1003014.

262. By Daniel Nichter on 2012-05-23

Merge fix-pqd-infinite-loop-bug-888114.

263. By Daniel Nichter on 2012-05-24

Merge fix-password-comma-bug-886077.

264. By Daniel Nichter on 2012-05-24

Merge fix-pva-mysql-5.5-bug-898138.

265. By Daniel Nichter on 2012-05-25

Merge handle-deadlocks-pt-osc.

266. By Daniel Nichter on 2012-05-25

Merge fix-osc-no-swap-tables-bug-1004551.

267. By Daniel Nichter on 2012-05-25

Merge test-table-usage-with-fingerprints.

268. By Daniel Nichter on 2012-05-25

Merge validate-load-options-bug-996915.

269. By Brian Fraser <email address hidden> on 2012-05-28

Fix for 903379

[Daniel Nichter (daniel-nichter)]

Whenever the 2.1 merge is done, you can do the magic to backport to 2.0.

review: Approve

[Brian Fraser (fraserbn)]

review: Approve

=== modified file 'Changelog'
--- Changelog	2012-03-07 23:41:54 +0000
+++ Changelog	2012-05-28 08:02:19 +0000
@@ -1,5 +1,26 @@
 Changelog for Percona Toolkit
 
+v2.1.1 released 2012-04-03
+
+  * Completely redesigned pt-online-schema-change
+  * Completely redesigned pt-mysql-summary
+  * Completely redesigned pt-summary
+  * Added new tool: pt-table-usage
+  * Added new tool: pt-fingerprint
+  * Fixed bug 955860: pt-stalk doesn't run vmstat, iostat, and mpstat for --run-time
+  * Fixed bug 960513: SHOW TABLE STATUS is used needlessly
+  * Fixed bug 969726: pt-online-schema-change loses foreign keys
+  * Fixed bug 846028: pt-online-schema-change does not show progress until completed
+  * Fixed bug 898695: pt-online-schema-change add useless ORDER BY
+  * Fixed bug 952727: pt-diskstats shows incorrect wr_mb_s
+  * Fixed bug 963225: pt-query-digest fails to set history columns for disk tmp tables and disk filesort
+  * Fixed bug 967451: Char chunking doesn't quote column name
+  * Fixed bug 972399: pt-table-checksum docs are not rendered right
+  * Fixed bug 896553: Various documentation spelling fixes
+  * Fixed bug 949154: pt-variable-advisor advice for relay-log-space-limit
+  * Fixed bug 953461: pt-upgrade manual broken 'output' section
+  * Fixed bug 949653: pt-table-checksum docs don't mention risks posed by inconsistent schemas
+
 v2.0.4 released 2012-03-07
 
   * Added --filter to pt-kill to allow arbitrary --group-by

=== modified file 'MANIFEST'
--- MANIFEST	2012-02-03 23:25:29 +0000
+++ MANIFEST	2012-05-28 08:02:19 +0000
@@ -12,6 +12,7 @@
 bin/pt-duplicate-key-checker
 bin/pt-fifo-split
 bin/pt-find
+bin/pt-fingerprint
 bin/pt-fk-error-logger
 bin/pt-heartbeat
 bin/pt-index-usage
@@ -33,6 +34,7 @@
 bin/pt-summary
 bin/pt-table-checksum
 bin/pt-table-sync
+bin/pt-table-usage
 bin/pt-tcp-model
 bin/pt-trend
 bin/pt-upgrade

=== modified file 'Makefile.PL'
--- Makefile.PL	2012-03-07 23:41:54 +0000
+++ Makefile.PL	2012-05-28 08:02:19 +0000
@@ -2,7 +2,7 @@
 
 WriteMakefile(
     NAME      => 'percona-toolkit',
-    VERSION   => '2.0.4',
+    VERSION   => '2.1.1',
     EXE_FILES => [ <bin/*> ],
     MAN1PODS  => {
       'docs/percona-toolkit.pod' => 'blib/man1/percona-toolkit.1',

=== modified file 'bin/pt-align'
--- bin/pt-align	2012-03-07 23:41:54 +0000
+++ bin/pt-align	2012-05-28 08:02:19 +0000
@@ -218,6 +218,6 @@
 
 =head1 VERSION
 
-pt-align 2.0.4
+pt-align 2.1.1
 
 =cut

=== modified file 'bin/pt-archiver'
--- bin/pt-archiver	2012-05-24 17:52:01 +0000
+++ bin/pt-archiver	2012-05-28 08:02:19 +0000
@@ -3962,18 +3962,30 @@
    # Open the file and print the header to it. 
    if ( $archive_file ) { 
       my $need_hdr = $o->get('header') && !-f $archive_file; 
-      my $charset  = lc $o->get('charset'); 
-      if ( $charset ) { 
-          $archive_fh = IO::File->new($archive_file, ">>:$charset") 
-             or die "Cannot open $charset $archive_file: $OS_ERROR\n"; 
-      } 
-      else { 
-          $archive_fh = IO::File->new($archive_file, ">>") 
-             or die "Cannot open $archive_file: $OS_ERROR\n"; 
-      } 
+      my $charset  = $o->get('charset') || '';
+      if ($charset eq 'utf8') {
+         $charset = ":$charset";
+      }
+      elsif ($charset) {
+         eval { require Encode }
+            or (PTDEBUG &&
+               _d("Couldn't load Encode: ", $EVAL_ERROR,
+                  "Going to try using the charset ",
+                  "passed in without checking it."));
+         # No need to punish a user if they did their
+         # homework and passed in an official charset,
+         # rather than an alias.
+         $charset = ":encoding("
+                  . (defined &Encode::resolve_alias
+                     ? Encode::resolve_alias($charset) || $charset
+                     : $charset)
+                  . ")";
+      }
+      $archive_fh = IO::File->new($archive_file, ">>$charset") 
+         or die "Cannot open $charset $archive_file: $OS_ERROR\n"; 
       $archive_fh->autoflush(1) unless $o->get('buffer'); 
       if ( $need_hdr ) { 
-         print $archive_fh '', escape(\@sel_cols), "\n" 
+         print { $archive_fh } '', escape(\@sel_cols), "\n" 
             or die "Cannot write to $archive_file: $OS_ERROR\n"; 
       } 
    } 
@@ -4183,6 +4195,16 @@
          commit($o, 1) if $commit_each;
          $get_sth = $get_next;
 
+         # Sleep between fetching the next chunk of rows.
+         if( my $sleep_time = $o->get('sleep') ) {
+            $sleep_time = $last_select_time * $o->get('sleep-coef')
+               if $o->get('sleep-coef');
+            PTDEBUG && _d('Sleeping', $sleep_time);
+            trace('sleep', sub {
+               sleep($sleep_time);
+            });
+         }
+
          PTDEBUG && _d('Fetching rows in next chunk');
          trace('select', sub {
             my $select_start = time;
@@ -4215,16 +4237,6 @@
             $lag = $ms->get_slave_lag($lag_dbh);
          }
       }
-
-      # Sleep between rows.
-      if( my $sleep_time = $o->get('sleep') ) {
-         $sleep_time = $last_select_time * $o->get('sleep-coef')
-            if $o->get('sleep-coef');
-         PTDEBUG && _d('Sleeping', $sleep_time);
-         trace('sleep', sub {
-            sleep($sleep_time);
-         });
-      }
    }  # ROW 
    PTDEBUG && _d('Done fetching rows');
 
@@ -5731,6 +5743,6 @@
 
 =head1 VERSION
 
-pt-archiver 2.0.4
+pt-archiver 2.1.1
 
 =cut

=== modified file 'bin/pt-config-diff'
--- bin/pt-config-diff	2012-05-24 17:52:01 +0000
+++ bin/pt-config-diff	2012-05-28 08:02:19 +0000
@@ -3408,6 +3408,6 @@
 
 =head1 VERSION
 
-pt-config-diff 2.0.4
+pt-config-diff 2.1.1
 
 =cut

=== modified file 'bin/pt-deadlock-logger'
--- bin/pt-deadlock-logger	2012-05-24 17:52:01 +0000
+++ bin/pt-deadlock-logger	2012-05-28 08:02:19 +0000
@@ -2745,6 +2745,6 @@
 
 =head1 VERSION
 
-pt-deadlock-logger 2.0.4
+pt-deadlock-logger 2.1.1
 
 =cut

=== modified file 'bin/pt-diskstats'
--- bin/pt-diskstats	2012-05-11 14:28:14 +0000
+++ bin/pt-diskstats	2012-05-28 08:02:19 +0000
@@ -4106,6 +4106,6 @@
 
 =head1 VERSION
 
-pt-diskstats 2.0.4
+pt-diskstats 2.1.1
 
 =cut

=== modified file 'bin/pt-duplicate-key-checker'
--- bin/pt-duplicate-key-checker	2012-05-24 17:52:01 +0000
+++ bin/pt-duplicate-key-checker	2012-05-28 08:02:19 +0000
@@ -199,19 +199,58 @@
    return bless $self, $class;
 }
 
+sub get_create_table {
+   my ( $self, $dbh, $db, $tbl ) = @_;
+   die "I need a dbh parameter" unless $dbh;
+   die "I need a db parameter"  unless $db;
+   die "I need a tbl parameter" unless $tbl;
+   my $q = $self->{Quoter};
+
+   my $new_sql_mode
+      = '/*!40101 SET @OLD_SQL_MODE := @@SQL_MODE, '
+      . q{@@SQL_MODE := REPLACE(REPLACE(@@SQL_MODE, 'ANSI_QUOTES', ''), ',,', ','), }
+      . '@OLD_QUOTE := @@SQL_QUOTE_SHOW_CREATE, '
+      . '@@SQL_QUOTE_SHOW_CREATE := 1 */';
+
+   my $old_sql_mode = '/*!40101 SET @@SQL_MODE := @OLD_SQL_MODE, '
+                     . '@@SQL_QUOTE_SHOW_CREATE := @OLD_QUOTE */';
+
+   PTDEBUG && _d($new_sql_mode);
+   eval { $dbh->do($new_sql_mode); };
+   PTDEBUG && $EVAL_ERROR && _d($EVAL_ERROR);
+
+   my $use_sql = 'USE ' . $q->quote($db);
+   PTDEBUG && _d($dbh, $use_sql);
+   $dbh->do($use_sql);
+
+   my $show_sql = "SHOW CREATE TABLE " . $q->quote($db, $tbl);
+   PTDEBUG && _d($show_sql);
+   my $href;
+   eval { $href = $dbh->selectrow_hashref($show_sql); };
+   if ( $EVAL_ERROR ) {
+      PTDEBUG && _d($EVAL_ERROR);
+
+      PTDEBUG && _d($old_sql_mode);
+      $dbh->do($old_sql_mode);
+
+      return;
+   }
+
+   PTDEBUG && _d($old_sql_mode);
+   $dbh->do($old_sql_mode);
+
+   my ($key) = grep { m/create (?:table|view)/i } keys %$href;
+   if ( !$key ) {
+      die "Error: no 'Create Table' or 'Create View' in result set from "
+         . "$show_sql: " . Dumper($href);
+   }
+
+   return $href->{$key};
+}
+
 sub parse {
    my ( $self, $ddl, $opts ) = @_;
    return unless $ddl;
-   if ( ref $ddl eq 'ARRAY' ) {
-      if ( lc $ddl->[0] eq 'table' ) {
-         $ddl = $ddl->[1];
-      }
-      else {
-         return {
-            engine => 'VIEW',
-         };
-      }
-   }
 
    if ( $ddl !~ m/CREATE (?:TEMPORARY )?TABLE `/ ) {
       die "Cannot parse table definition; is ANSI quoting "
@@ -518,41 +557,31 @@
    return $ddl;
 }
 
-sub remove_secondary_indexes {
-   my ( $self, $ddl ) = @_;
-   my $sec_indexes_ddl;
-   my $tbl_struct = $self->parse($ddl);
-
-   if ( ($tbl_struct->{engine} || '') =~ m/InnoDB/i ) {
-      my $clustered_key = $tbl_struct->{clustered_key};
-      $clustered_key  ||= '';
-
-      my @sec_indexes   = map {
-         my $key_def = $_->{ddl};
-         $key_def =~ s/([\(\)])/\\$1/g;
-         $ddl =~ s/\s+$key_def//i;
-
-         my $key_ddl = "ADD $_->{ddl}";
-         $key_ddl   .= ',' unless $key_ddl =~ m/,$/;
-         $key_ddl;
-      }
-      grep { $_->{name} ne $clustered_key }
-      values %{$tbl_struct->{keys}};
-      PTDEBUG && _d('Secondary indexes:', Dumper(\@sec_indexes));
-
-      if ( @sec_indexes ) {
-         $sec_indexes_ddl = join(' ', @sec_indexes);
-         $sec_indexes_ddl =~ s/,$//;
-      }
-
-      $ddl =~ s/,(\n\) )/$1/s;
-   }
-   else {
-      PTDEBUG && _d('Not removing secondary indexes from',
-         $tbl_struct->{engine}, 'table');
-   }
-
-   return $ddl, $sec_indexes_ddl, $tbl_struct;
+sub get_table_status {
+   my ( $self, $dbh, $db, $like ) = @_;
+   my $q = $self->{Quoter};
+   my $sql = "SHOW TABLE STATUS FROM " . $q->quote($db);
+   my @params;
+   if ( $like ) {
+      $sql .= ' LIKE ?';
+      push @params, $like;
+   }
+   PTDEBUG && _d($sql, @params);
+   my $sth = $dbh->prepare($sql);
+   eval { $sth->execute(@params); };
+   if ($EVAL_ERROR) {
+      PTDEBUG && _d($EVAL_ERROR);
+      return;
+   }
+   my @tables = @{$sth->fetchall_arrayref({})};
+   @tables = map {
+      my %tbl; # Make a copy with lowercased keys
+      @tbl{ map { lc $_ } keys %$_ } = values %$_;
+      $tbl{engine} ||= $tbl{type} || $tbl{comment};
+      delete $tbl{type};
+      \%tbl;
+   } @tables;
+   return @tables;
 }
 
 sub _d {
@@ -3200,7 +3229,7 @@
 
 sub new {
    my ( $class, %args ) = @_;
-   my @required_args = qw(OptionParser Quoter);
+   my @required_args = qw(OptionParser TableParser Quoter);
    foreach my $arg ( @required_args ) {
       die "I need a $arg argument" unless $args{$arg};
    }
@@ -3209,8 +3238,19 @@
    die "I need either a dbh or file_itr argument"
       if (!$dbh && !$file_itr) || ($dbh && $file_itr);
 
+   my %resume;
+   if ( my $table = $args{resume} ) {
+      PTDEBUG && _d('Will resume from or after', $table);
+      my ($db, $tbl) = $args{Quoter}->split_unquote($table);
+      die "Resume table must be database-qualified: $table"
+         unless $db && $tbl;
+      $resume{db}  = $db;
+      $resume{tbl} = $tbl;
+   }
+
    my $self = {
       %args,
+      resume  => \%resume,
       filters => _make_filters(%args),
    };
 
@@ -3271,9 +3311,19 @@
    return \%filters;
 }
 
-sub next_schema_object {
+sub next {
    my ( $self ) = @_;
 
+   if ( !$self->{initialized} ) {
+      $self->{initialized} = 1;
+      if ( $self->{resume}->{tbl}
+           && !$self->table_is_allowed(@{$self->{resume}}{qw(db tbl)}) ) {
+         PTDEBUG && _d('Will resume after',
+            join('.', @{$self->{resume}}{qw(db tbl)}));
+         $self->{resume}->{after} = 1;
+      }
+   }
+
    my $schema_obj;
    if ( $self->{file_itr} ) {
       $schema_obj= $self->_iterate_files();
@@ -3283,19 +3333,13 @@
    }
 
    if ( $schema_obj ) {
-      if ( $schema_obj->{ddl} && $self->{TableParser} ) {
-         $schema_obj->{tbl_struct}
-            = $self->{TableParser}->parse($schema_obj->{ddl});
-      }
-
-      delete $schema_obj->{ddl} unless $self->{keep_ddl};
-
       if ( my $schema = $self->{Schema} ) {
          $schema->add_schema_object($schema_obj);
       }
+      PTDEBUG && _d('Next schema object:',
+         $schema_obj->{db}, $schema_obj->{tbl});
    }
 
-   PTDEBUG && _d('Next schema object:', $schema_obj->{db}, $schema_obj->{tbl});
    return $schema_obj;
 }
 
@@ -3321,7 +3365,8 @@
          my $db = $1; # XXX
          $db =~ s/^`//;  # strip leading `
          $db =~ s/`$//;  # and trailing `
-         if ( $self->database_is_allowed($db) ) {
+         if ( $self->database_is_allowed($db)
+              && $self->_resume_from_database($db) ) {
             $self->{db} = $db;
          }
       }
@@ -3334,21 +3379,22 @@
          my ($tbl) = $chunk =~ m/$tbl_name/;
          $tbl      =~ s/^\s*`//;
          $tbl      =~ s/`\s*$//;
-         if ( $self->table_is_allowed($self->{db}, $tbl) ) {
+         if ( $self->_resume_from_table($tbl)
+              && $self->table_is_allowed($self->{db}, $tbl) ) {
             my ($ddl) = $chunk =~ m/^(?:$open_comment)?(CREATE TABLE.+?;)$/ms;
             if ( !$ddl ) {
                warn "Failed to parse CREATE TABLE from\n" . $chunk;
                next CHUNK;
             }
             $ddl =~ s/ \*\/;\Z/;/;  # remove end of version comment
-
-            my ($engine) = $ddl =~ m/\).*?(?:ENGINE|TYPE)=(\w+)/;   
-
-            if ( !$engine || $self->engine_is_allowed($engine) ) {
+            my $tbl_struct = $self->{TableParser}->parse($ddl);
+            if ( $self->engine_is_allowed($tbl_struct->{engine}) ) {
                return {
-                  db  => $self->{db},
-                  tbl => $tbl,
-                  ddl => $ddl,
+                  db         => $self->{db},
+                  tbl        => $tbl,
+                  name       => $self->{Quoter}->quote($self->{db}, $tbl),
+                  ddl        => $ddl,
+                  tbl_struct => $tbl_struct,
                };
             }
          }
@@ -3365,6 +3411,7 @@
 sub _iterate_dbh {
    my ( $self ) = @_;
    my $q   = $self->{Quoter};
+   my $tp  = $self->{TableParser};
    my $dbh = $self->{dbh};
    PTDEBUG && _d('Getting next schema object from dbh', $dbh);
 
@@ -3378,7 +3425,9 @@
    }
 
    if ( !$self->{db} ) {
-      $self->{db} = shift @{$self->{dbs}};
+      do {
+         $self->{db} = shift @{$self->{dbs}};
+      } until $self->_resume_from_database($self->{db});
       PTDEBUG && _d('Next database:', $self->{db});
       return unless $self->{db};
    }
@@ -3391,8 +3440,9 @@
       }
       grep {
          my ($tbl, $type) = @$_;
-         $self->table_is_allowed($self->{db}, $tbl)
-            && (!$type || ($type ne 'VIEW'));
+         (!$type || ($type ne 'VIEW'))
+         && $self->_resume_from_table($tbl)
+         && $self->table_is_allowed($self->{db}, $tbl);
       }
       @{$dbh->selectall_arrayref($sql)};
       PTDEBUG && _d('Found', scalar @tbls, 'tables in database', $self->{db});
@@ -3400,27 +3450,15 @@
    }
 
    while ( my $tbl = shift @{$self->{tbls}} ) {
-      my $engine;
-      if ( $self->{filters}->{'engines'}
-           || $self->{filters}->{'ignore-engines'} ) {
-         my $sql = "SHOW TABLE STATUS FROM " . $q->quote($self->{db})
-                 . " LIKE \'$tbl\'";
-         PTDEBUG && _d($sql);
-         $engine = $dbh->selectrow_hashref($sql)->{engine};
-         PTDEBUG && _d($tbl, 'uses', $engine, 'engine');
-      }
-
-
-      if ( !$engine || $self->engine_is_allowed($engine) ) {
-         my $ddl;
-         if ( my $du = $self->{MySQLDump} ) {
-            $ddl = $du->get_create_table($dbh, $q, $self->{db}, $tbl)->[1];
-         }
-
+      my $ddl        = $tp->get_create_table($dbh, $self->{db}, $tbl);
+      my $tbl_struct = $tp->parse($ddl);
+      if ( $self->engine_is_allowed($tbl_struct->{engine}) ) {
          return {
-            db  => $self->{db},
-            tbl => $tbl,
-            ddl => $ddl,
+            db         => $self->{db},
+            tbl        => $tbl,
+            name       => $q->quote($self->{db}, $tbl),
+            ddl        => $ddl,
+            tbl_struct => $tbl_struct,
          };
       }
    }
@@ -3481,6 +3519,10 @@
 
    my $filter = $self->{filters};
 
+   if ( $db eq 'mysql' && ($tbl eq 'general_log' || $tbl eq 'slow_log') ) {
+      return 0;
+   }
+
    if ( $filter->{'ignore-tables'}->{$tbl}
         && ($filter->{'ignore-tables'}->{$tbl} eq '*'
             || $filter->{'ignore-tables'}->{$tbl} eq $db) ) {
@@ -3520,7 +3562,11 @@
 
 sub engine_is_allowed {
    my ( $self, $engine ) = @_;
-   die "I need an engine argument" unless $engine;
+
+   if ( !$engine ) {
+      PTDEBUG && _d('No engine specified; allowing the table');
+      return 1;
+   }
 
    $engine = lc $engine;
 
@@ -3540,6 +3586,40 @@
    return 1;
 }
 
+sub _resume_from_database {
+   my ($self, $db) = @_;
+
+   return 1 unless $self->{resume}->{db};
+
+   if ( $db eq $self->{resume}->{db} ) {
+      PTDEBUG && _d('At resume db', $db);
+      delete $self->{resume}->{db};
+      return 1;
+   }
+
+   return 0;
+}
+
+sub _resume_from_table {
+   my ($self, $tbl) = @_;
+
+   return 1 unless $self->{resume}->{tbl};
+
+   if ( $tbl eq $self->{resume}->{tbl} ) {
+      if ( !$self->{resume}->{after} ) {
+         PTDEBUG && _d('Resuming from table', $tbl);
+         delete $self->{resume}->{tbl};
+         return 1;
+      }
+      else {
+         PTDEBUG && _d('Resuming after table', $tbl);
+         delete $self->{resume}->{tbl};
+      }
+   }
+
+   return 0;
+}
+
 sub _d {
    my ($package, undef, $line) = caller 0;
    @_ = map { (my $temp = $_) =~ s/\n/\n# /g; $temp; }
@@ -3649,11 +3729,10 @@
       MySQLDump    => $du,
       TableParser  => $tp,
       Schema       => $schema,
-      keep_ddl     => 1,
    );
    TABLE:
-   while ( my $tbl = $schema_itr->next_schema_object() ) {
-      $tbl->{engine} = $tp->get_engine($tbl->{ddl});
+   while ( my $tbl = $schema_itr->next() ) {
+      $tbl->{engine} = $tbl->{tbl_struct}->{engine};
 
       my ($keys, $clustered_key, $fks);
       if ( $get_keys ) {
@@ -4279,6 +4358,6 @@
 
 =head1 VERSION
 
-pt-duplicate-key-checker 2.0.4
+pt-duplicate-key-checker 2.1.1
 
 =cut

=== modified file 'bin/pt-fifo-split'
--- bin/pt-fifo-split	2012-03-07 23:41:54 +0000
+++ bin/pt-fifo-split	2012-05-28 08:02:19 +0000
@@ -1547,6 +1547,6 @@
 
 =head1 VERSION
 
-pt-fifo-split 2.0.4
+pt-fifo-split 2.1.1
 
 =cut

=== modified file 'bin/pt-find'
--- bin/pt-find	2012-05-24 17:52:01 +0000
+++ bin/pt-find	2012-05-28 08:02:19 +0000
@@ -3827,6 +3827,6 @@
 
 =head1 VERSION
 
-pt-find 2.0.4
+pt-find 2.1.1
 
 =cut

=== added file 'bin/pt-fingerprint'
--- bin/pt-fingerprint	1970-01-01 00:00:00 +0000
+++ bin/pt-fingerprint	2012-05-28 08:02:19 +0000
@@ -0,0 +1,2143 @@
+#!/usr/bin/env perl
+
+# This program is part of Percona Toolkit: http://www.percona.com/software/
+# See "COPYRIGHT, LICENSE, AND WARRANTY" at the end of this file for legal
+# notices and disclaimers.
+
+use strict;
+use warnings FATAL => 'all';
+use constant MKDEBUG => $ENV{MKDEBUG} || 0;
+
+# ###########################################################################
+# OptionParser package
+# This package is a copy without comments from the original.  The original
+# with comments and its test file can be found in the Bazaar repository at,
+#   lib/OptionParser.pm
+#   t/lib/OptionParser.t
+# See https://launchpad.net/percona-toolkit for more information.
+# ###########################################################################
+{
+package OptionParser;
+
+use strict;
+use warnings FATAL => 'all';
+use English qw(-no_match_vars);
+use constant PTDEBUG => $ENV{PTDEBUG} || 0;
+
+use List::Util qw(max);
+use Getopt::Long;
+
+my $POD_link_re = '[LC]<"?([^">]+)"?>';
+
+sub new {
+   my ( $class, %args ) = @_;
+   my @required_args = qw();
+   foreach my $arg ( @required_args ) {
+      die "I need a $arg argument" unless $args{$arg};
+   }
+
+   my ($program_name) = $PROGRAM_NAME =~ m/([.A-Za-z-]+)$/;
+   $program_name ||= $PROGRAM_NAME;
+   my $home = $ENV{HOME} || $ENV{HOMEPATH} || $ENV{USERPROFILE} || '.';
+
+   my %attributes = (
+      'type'       => 1,
+      'short form' => 1,
+      'group'      => 1,
+      'default'    => 1,
+      'cumulative' => 1,
+      'negatable'  => 1,
+   );
+
+   my $self = {
+      head1             => 'OPTIONS',        # These args are used internally
+      skip_rules        => 0,                # to instantiate another Option-
+      item              => '--(.*)',         # Parser obj that parses the
+      attributes        => \%attributes,     # DSN OPTIONS section.  Tools
+      parse_attributes  => \&_parse_attribs, # don't tinker with these args.
+
+      %args,
+
+      strict            => 1,  # disabled by a special rule
+      program_name      => $program_name,
+      opts              => {},
+      got_opts          => 0,
+      short_opts        => {},
+      defaults          => {},
+      groups            => {},
+      allowed_groups    => {},
+      errors            => [],
+      rules             => [],  # desc of rules for --help
+      mutex             => [],  # rule: opts are mutually exclusive
+      atleast1          => [],  # rule: at least one opt is required
+      disables          => {},  # rule: opt disables other opts 
+      defaults_to       => {},  # rule: opt defaults to value of other opt
+      DSNParser         => undef,
+      default_files     => [
+         "/etc/percona-toolkit/percona-toolkit.conf",
+         "/etc/percona-toolkit/$program_name.conf",
+         "$home/.percona-toolkit.conf",
+         "$home/.$program_name.conf",
+      ],
+      types             => {
+         string => 's', # standard Getopt type
+         int    => 'i', # standard Getopt type
+         float  => 'f', # standard Getopt type
+         Hash   => 'H', # hash, formed from a comma-separated list
+         hash   => 'h', # hash as above, but only if a value is given
+         Array  => 'A', # array, similar to Hash
+         array  => 'a', # array, similar to hash
+         DSN    => 'd', # DSN
+         size   => 'z', # size with kMG suffix (powers of 2^10)
+         time   => 'm', # time, with an optional suffix of s/h/m/d
+      },
+   };
+
+   return bless $self, $class;
+}
+
+sub get_specs {
+   my ( $self, $file ) = @_;
+   $file ||= $self->{file} || __FILE__;
+   my @specs = $self->_pod_to_specs($file);
+   $self->_parse_specs(@specs);
+
+   open my $fh, "<", $file or die "Cannot open $file: $OS_ERROR";
+   my $contents = do { local $/ = undef; <$fh> };
+   close $fh;
+   if ( $contents =~ m/^=head1 DSN OPTIONS/m ) {
+      PTDEBUG && _d('Parsing DSN OPTIONS');
+      my $dsn_attribs = {
+         dsn  => 1,
+         copy => 1,
+      };
+      my $parse_dsn_attribs = sub {
+         my ( $self, $option, $attribs ) = @_;
+         map {
+            my $val = $attribs->{$_};
+            if ( $val ) {
+               $val    = $val eq 'yes' ? 1
+                       : $val eq 'no'  ? 0
+                       :                 $val;
+               $attribs->{$_} = $val;
+            }
+         } keys %$attribs;
+         return {
+            key => $option,
+            %$attribs,
+         };
+      };
+      my $dsn_o = new OptionParser(
+         description       => 'DSN OPTIONS',
+         head1             => 'DSN OPTIONS',
+         dsn               => 0,         # XXX don't infinitely recurse!
+         item              => '\* (.)',  # key opts are a single character
+         skip_rules        => 1,         # no rules before opts
+         attributes        => $dsn_attribs,
+         parse_attributes  => $parse_dsn_attribs,
+      );
+      my @dsn_opts = map {
+         my $opts = {
+            key  => $_->{spec}->{key},
+            dsn  => $_->{spec}->{dsn},
+            copy => $_->{spec}->{copy},
+            desc => $_->{desc},
+         };
+         $opts;
+      } $dsn_o->_pod_to_specs($file);
+      $self->{DSNParser} = DSNParser->new(opts => \@dsn_opts);
+   }
+
+   if ( $contents =~ m/^=head1 VERSION\n\n^(.+)$/m ) {
+      $self->{version} = $1;
+      PTDEBUG && _d($self->{version});
+   }
+
+   return;
+}
+
+sub DSNParser {
+   my ( $self ) = @_;
+   return $self->{DSNParser};
+};
+
+sub get_defaults_files {
+   my ( $self ) = @_;
+   return @{$self->{default_files}};
+}
+
+sub _pod_to_specs {
+   my ( $self, $file ) = @_;
+   $file ||= $self->{file} || __FILE__;
+   open my $fh, '<', $file or die "Cannot open $file: $OS_ERROR";
+
+   my @specs = ();
+   my @rules = ();
+   my $para;
+
+   local $INPUT_RECORD_SEPARATOR = '';
+   while ( $para = <$fh> ) {
+      next unless $para =~ m/^=head1 $self->{head1}/;
+      last;
+   }
+
+   while ( $para = <$fh> ) {
+      last if $para =~ m/^=over/;
+      next if $self->{skip_rules};
+      chomp $para;
+      $para =~ s/\s+/ /g;
+      $para =~ s/$POD_link_re/$1/go;
+      PTDEBUG && _d('Option rule:', $para);
+      push @rules, $para;
+   }
+
+   die "POD has no $self->{head1} section" unless $para;
+
+   do {
+      if ( my ($option) = $para =~ m/^=item $self->{item}/ ) {
+         chomp $para;
+         PTDEBUG && _d($para);
+         my %attribs;
+
+         $para = <$fh>; # read next paragraph, possibly attributes
+
+         if ( $para =~ m/: / ) { # attributes
+            $para =~ s/\s+\Z//g;
+            %attribs = map {
+                  my ( $attrib, $val) = split(/: /, $_);
+                  die "Unrecognized attribute for --$option: $attrib"
+                     unless $self->{attributes}->{$attrib};
+                  ($attrib, $val);
+               } split(/; /, $para);
+            if ( $attribs{'short form'} ) {
+               $attribs{'short form'} =~ s/-//;
+            }
+            $para = <$fh>; # read next paragraph, probably short help desc
+         }
+         else {
+            PTDEBUG && _d('Option has no attributes');
+         }
+
+         $para =~ s/\s+\Z//g;
+         $para =~ s/\s+/ /g;
+         $para =~ s/$POD_link_re/$1/go;
+
+         $para =~ s/\.(?:\n.*| [A-Z].*|\Z)//s;
+         PTDEBUG && _d('Short help:', $para);
+
+         die "No description after option spec $option" if $para =~ m/^=item/;
+
+         if ( my ($base_option) =  $option =~ m/^\[no\](.*)/ ) {
+            $option = $base_option;
+            $attribs{'negatable'} = 1;
+         }
+
+         push @specs, {
+            spec  => $self->{parse_attributes}->($self, $option, \%attribs), 
+            desc  => $para
+               . (defined $attribs{default} ? " (default $attribs{default})" : ''),
+            group => ($attribs{'group'} ? $attribs{'group'} : 'default'),
+         };
+      }
+      while ( $para = <$fh> ) {
+         last unless $para;
+         if ( $para =~ m/^=head1/ ) {
+            $para = undef; # Can't 'last' out of a do {} block.
+            last;
+         }
+         last if $para =~ m/^=item /;
+      }
+   } while ( $para );
+
+   die "No valid specs in $self->{head1}" unless @specs;
+
+   close $fh;
+   return @specs, @rules;
+}
+
+sub _parse_specs {
+   my ( $self, @specs ) = @_;
+   my %disables; # special rule that requires deferred checking
+
+   foreach my $opt ( @specs ) {
+      if ( ref $opt ) { # It's an option spec, not a rule.
+         PTDEBUG && _d('Parsing opt spec:',
+            map { ($_, '=>', $opt->{$_}) } keys %$opt);
+
+         my ( $long, $short ) = $opt->{spec} =~ m/^([\w-]+)(?:\|([^!+=]*))?/;
+         if ( !$long ) {
+            die "Cannot parse long option from spec $opt->{spec}";
+         }
+         $opt->{long} = $long;
+
+         die "Duplicate long option --$long" if exists $self->{opts}->{$long};
+         $self->{opts}->{$long} = $opt;
+
+         if ( length $long == 1 ) {
+            PTDEBUG && _d('Long opt', $long, 'looks like short opt');
+            $self->{short_opts}->{$long} = $long;
+         }
+
+         if ( $short ) {
+            die "Duplicate short option -$short"
+               if exists $self->{short_opts}->{$short};
+            $self->{short_opts}->{$short} = $long;
+            $opt->{short} = $short;
+         }
+         else {
+            $opt->{short} = undef;
+         }
+
+         $opt->{is_negatable}  = $opt->{spec} =~ m/!/        ? 1 : 0;
+         $opt->{is_cumulative} = $opt->{spec} =~ m/\+/       ? 1 : 0;
+         $opt->{is_required}   = $opt->{desc} =~ m/required/ ? 1 : 0;
+
+         $opt->{group} ||= 'default';
+         $self->{groups}->{ $opt->{group} }->{$long} = 1;
+
+         $opt->{value} = undef;
+         $opt->{got}   = 0;
+
+         my ( $type ) = $opt->{spec} =~ m/=(.)/;
+         $opt->{type} = $type;
+         PTDEBUG && _d($long, 'type:', $type);
+
+
+         $opt->{spec} =~ s/=./=s/ if ( $type && $type =~ m/[HhAadzm]/ );
+
+         if ( (my ($def) = $opt->{desc} =~ m/default\b(?: ([^)]+))?/) ) {
+            $self->{defaults}->{$long} = defined $def ? $def : 1;
+            PTDEBUG && _d($long, 'default:', $def);
+         }
+
+         if ( $long eq 'config' ) {
+            $self->{defaults}->{$long} = join(',', $self->get_defaults_files());
+         }
+
+         if ( (my ($dis) = $opt->{desc} =~ m/(disables .*)/) ) {
+            $disables{$long} = $dis;
+            PTDEBUG && _d('Deferring check of disables rule for', $opt, $dis);
+         }
+
+         $self->{opts}->{$long} = $opt;
+      }
+      else { # It's an option rule, not a spec.
+         PTDEBUG && _d('Parsing rule:', $opt); 
+         push @{$self->{rules}}, $opt;
+         my @participants = $self->_get_participants($opt);
+         my $rule_ok = 0;
+
+         if ( $opt =~ m/mutually exclusive|one and only one/ ) {
+            $rule_ok = 1;
+            push @{$self->{mutex}}, \@participants;
+            PTDEBUG && _d(@participants, 'are mutually exclusive');
+         }
+         if ( $opt =~ m/at least one|one and only one/ ) {
+            $rule_ok = 1;
+            push @{$self->{atleast1}}, \@participants;
+            PTDEBUG && _d(@participants, 'require at least one');
+         }
+         if ( $opt =~ m/default to/ ) {
+            $rule_ok = 1;
+            $self->{defaults_to}->{$participants[0]} = $participants[1];
+            PTDEBUG && _d($participants[0], 'defaults to', $participants[1]);
+         }
+         if ( $opt =~ m/restricted to option groups/ ) {
+            $rule_ok = 1;
+            my ($groups) = $opt =~ m/groups ([\w\s\,]+)/;
+            my @groups = split(',', $groups);
+            %{$self->{allowed_groups}->{$participants[0]}} = map {
+               s/\s+//;
+               $_ => 1;
+            } @groups;
+         }
+         if( $opt =~ m/accepts additional command-line arguments/ ) {
+            $rule_ok = 1;
+            $self->{strict} = 0;
+            PTDEBUG && _d("Strict mode disabled by rule");
+         }
+
+         die "Unrecognized option rule: $opt" unless $rule_ok;
+      }
+   }
+
+   foreach my $long ( keys %disables ) {
+      my @participants = $self->_get_participants($disables{$long});
+      $self->{disables}->{$long} = \@participants;
+      PTDEBUG && _d('Option', $long, 'disables', @participants);
+   }
+
+   return; 
+}
+
+sub _get_participants {
+   my ( $self, $str ) = @_;
+   my @participants;
+   foreach my $long ( $str =~ m/--(?:\[no\])?([\w-]+)/g ) {
+      die "Option --$long does not exist while processing rule $str"
+         unless exists $self->{opts}->{$long};
+      push @participants, $long;
+   }
+   PTDEBUG && _d('Participants for', $str, ':', @participants);
+   return @participants;
+}
+
+sub opts {
+   my ( $self ) = @_;
+   my %opts = %{$self->{opts}};
+   return %opts;
+}
+
+sub short_opts {
+   my ( $self ) = @_;
+   my %short_opts = %{$self->{short_opts}};
+   return %short_opts;
+}
+
+sub set_defaults {
+   my ( $self, %defaults ) = @_;
+   $self->{defaults} = {};
+   foreach my $long ( keys %defaults ) {
+      die "Cannot set default for nonexistent option $long"
+         unless exists $self->{opts}->{$long};
+      $self->{defaults}->{$long} = $defaults{$long};
+      PTDEBUG && _d('Default val for', $long, ':', $defaults{$long});
+   }
+   return;
+}
+
+sub get_defaults {
+   my ( $self ) = @_;
+   return $self->{defaults};
+}
+
+sub get_groups {
+   my ( $self ) = @_;
+   return $self->{groups};
+}
+
+sub _set_option {
+   my ( $self, $opt, $val ) = @_;
+   my $long = exists $self->{opts}->{$opt}       ? $opt
+            : exists $self->{short_opts}->{$opt} ? $self->{short_opts}->{$opt}
+            : die "Getopt::Long gave a nonexistent option: $opt";
+
+   $opt = $self->{opts}->{$long};
+   if ( $opt->{is_cumulative} ) {
+      $opt->{value}++;
+   }
+   else {
+      $opt->{value} = $val;
+   }
+   $opt->{got} = 1;
+   PTDEBUG && _d('Got option', $long, '=', $val);
+}
+
+sub get_opts {
+   my ( $self ) = @_; 
+
+   foreach my $long ( keys %{$self->{opts}} ) {
+      $self->{opts}->{$long}->{got} = 0;
+      $self->{opts}->{$long}->{value}
+         = exists $self->{defaults}->{$long}       ? $self->{defaults}->{$long}
+         : $self->{opts}->{$long}->{is_cumulative} ? 0
+         : undef;
+   }
+   $self->{got_opts} = 0;
+
+   $self->{errors} = [];
+
+   if ( @ARGV && $ARGV[0] eq "--config" ) {
+      shift @ARGV;
+      $self->_set_option('config', shift @ARGV);
+   }
+   if ( $self->has('config') ) {
+      my @extra_args;
+      foreach my $filename ( split(',', $self->get('config')) ) {
+         eval {
+            push @extra_args, $self->_read_config_file($filename);
+         };
+         if ( $EVAL_ERROR ) {
+            if ( $self->got('config') ) {
+               die $EVAL_ERROR;
+            }
+            elsif ( PTDEBUG ) {
+               _d($EVAL_ERROR);
+            }
+         }
+      }
+      unshift @ARGV, @extra_args;
+   }
+
+   Getopt::Long::Configure('no_ignore_case', 'bundling');
+   GetOptions(
+      map    { $_->{spec} => sub { $self->_set_option(@_); } }
+      grep   { $_->{long} ne 'config' } # --config is handled specially above.
+      values %{$self->{opts}}
+   ) or $self->save_error('Error parsing options');
+
+   if ( exists $self->{opts}->{version} && $self->{opts}->{version}->{got} ) {
+      if ( $self->{version} ) {
+         print $self->{version}, "\n";
+      }
+      else {
+         print "Error parsing version.  See the VERSION section of the tool's documentation.\n";
+      }
+      exit 0;
+   }
+
+   if ( @ARGV && $self->{strict} ) {
+      $self->save_error("Unrecognized command-line options @ARGV");
+   }
+
+   foreach my $mutex ( @{$self->{mutex}} ) {
+      my @set = grep { $self->{opts}->{$_}->{got} } @$mutex;
+      if ( @set > 1 ) {
+         my $err = join(', ', map { "--$self->{opts}->{$_}->{long}" }
+                      @{$mutex}[ 0 .. scalar(@$mutex) - 2] )
+                 . ' and --'.$self->{opts}->{$mutex->[-1]}->{long}
+                 . ' are mutually exclusive.';
+         $self->save_error($err);
+      }
+   }
+
+   foreach my $required ( @{$self->{atleast1}} ) {
+      my @set = grep { $self->{opts}->{$_}->{got} } @$required;
+      if ( @set == 0 ) {
+         my $err = join(', ', map { "--$self->{opts}->{$_}->{long}" }
+                      @{$required}[ 0 .. scalar(@$required) - 2] )
+                 .' or --'.$self->{opts}->{$required->[-1]}->{long};
+         $self->save_error("Specify at least one of $err");
+      }
+   }
+
+   $self->_check_opts( keys %{$self->{opts}} );
+   $self->{got_opts} = 1;
+   return;
+}
+
+sub _check_opts {
+   my ( $self, @long ) = @_;
+   my $long_last = scalar @long;
+   while ( @long ) {
+      foreach my $i ( 0..$#long ) {
+         my $long = $long[$i];
+         next unless $long;
+         my $opt  = $self->{opts}->{$long};
+         if ( $opt->{got} ) {
+            if ( exists $self->{disables}->{$long} ) {
+               my @disable_opts = @{$self->{disables}->{$long}};
+               map { $self->{opts}->{$_}->{value} = undef; } @disable_opts;
+               PTDEBUG && _d('Unset options', @disable_opts,
+                  'because', $long,'disables them');
+            }
+
+            if ( exists $self->{allowed_groups}->{$long} ) {
+
+               my @restricted_groups = grep {
+                  !exists $self->{allowed_groups}->{$long}->{$_}
+               } keys %{$self->{groups}};
+
+               my @restricted_opts;
+               foreach my $restricted_group ( @restricted_groups ) {
+                  RESTRICTED_OPT:
+                  foreach my $restricted_opt (
+                     keys %{$self->{groups}->{$restricted_group}} )
+                  {
+                     next RESTRICTED_OPT if $restricted_opt eq $long;
+                     push @restricted_opts, $restricted_opt
+                        if $self->{opts}->{$restricted_opt}->{got};
+                  }
+               }
+
+               if ( @restricted_opts ) {
+                  my $err;
+                  if ( @restricted_opts == 1 ) {
+                     $err = "--$restricted_opts[0]";
+                  }
+                  else {
+                     $err = join(', ',
+                               map { "--$self->{opts}->{$_}->{long}" }
+                               grep { $_ } 
+                               @restricted_opts[0..scalar(@restricted_opts) - 2]
+                            )
+                          . ' or --'.$self->{opts}->{$restricted_opts[-1]}->{long};
+                  }
+                  $self->save_error("--$long is not allowed with $err");
+               }
+            }
+
+         }
+         elsif ( $opt->{is_required} ) { 
+            $self->save_error("Required option --$long must be specified");
+         }
+
+         $self->_validate_type($opt);
+         if ( $opt->{parsed} ) {
+            delete $long[$i];
+         }
+         else {
+            PTDEBUG && _d('Temporarily failed to parse', $long);
+         }
+      }
+
+      die "Failed to parse options, possibly due to circular dependencies"
+         if @long == $long_last;
+      $long_last = @long;
+   }
+
+   return;
+}
+
+sub _validate_type {
+   my ( $self, $opt ) = @_;
+   return unless $opt;
+
+   if ( !$opt->{type} ) {
+      $opt->{parsed} = 1;
+      return;
+   }
+
+   my $val = $opt->{value};
+
+   if ( $val && $opt->{type} eq 'm' ) {  # type time
+      PTDEBUG && _d('Parsing option', $opt->{long}, 'as a time value');
+      my ( $prefix, $num, $suffix ) = $val =~ m/([+-]?)(\d+)([a-z])?$/;
+      if ( !$suffix ) {
+         my ( $s ) = $opt->{desc} =~ m/\(suffix (.)\)/;
+         $suffix = $s || 's';
+         PTDEBUG && _d('No suffix given; using', $suffix, 'for',
+            $opt->{long}, '(value:', $val, ')');
+      }
+      if ( $suffix =~ m/[smhd]/ ) {
+         $val = $suffix eq 's' ? $num            # Seconds
+              : $suffix eq 'm' ? $num * 60       # Minutes
+              : $suffix eq 'h' ? $num * 3600     # Hours
+              :                  $num * 86400;   # Days
+         $opt->{value} = ($prefix || '') . $val;
+         PTDEBUG && _d('Setting option', $opt->{long}, 'to', $val);
+      }
+      else {
+         $self->save_error("Invalid time suffix for --$opt->{long}");
+      }
+   }
+   elsif ( $val && $opt->{type} eq 'd' ) {  # type DSN
+      PTDEBUG && _d('Parsing option', $opt->{long}, 'as a DSN');
+      my $prev = {};
+      my $from_key = $self->{defaults_to}->{ $opt->{long} };
+      if ( $from_key ) {
+         PTDEBUG && _d($opt->{long}, 'DSN copies from', $from_key, 'DSN');
+         if ( $self->{opts}->{$from_key}->{parsed} ) {
+            $prev = $self->{opts}->{$from_key}->{value};
+         }
+         else {
+            PTDEBUG && _d('Cannot parse', $opt->{long}, 'until',
+               $from_key, 'parsed');
+            return;
+         }
+      }
+      my $defaults = $self->{DSNParser}->parse_options($self);
+      $opt->{value} = $self->{DSNParser}->parse($val, $prev, $defaults);
+   }
+   elsif ( $val && $opt->{type} eq 'z' ) {  # type size
+      PTDEBUG && _d('Parsing option', $opt->{long}, 'as a size value');
+      $self->_parse_size($opt, $val);
+   }
+   elsif ( $opt->{type} eq 'H' || (defined $val && $opt->{type} eq 'h') ) {
+      $opt->{value} = { map { $_ => 1 } split(/(?<!\\),\s*/, ($val || '')) };
+   }
+   elsif ( $opt->{type} eq 'A' || (defined $val && $opt->{type} eq 'a') ) {
+      $opt->{value} = [ split(/(?<!\\),\s*/, ($val || '')) ];
+   }
+   else {
+      PTDEBUG && _d('Nothing to validate for option',
+         $opt->{long}, 'type', $opt->{type}, 'value', $val);
+   }
+
+   $opt->{parsed} = 1;
+   return;
+}
+
+sub get {
+   my ( $self, $opt ) = @_;
+   my $long = (length $opt == 1 ? $self->{short_opts}->{$opt} : $opt);
+   die "Option $opt does not exist"
+      unless $long && exists $self->{opts}->{$long};
+   return $self->{opts}->{$long}->{value};
+}
+
+sub got {
+   my ( $self, $opt ) = @_;
+   my $long = (length $opt == 1 ? $self->{short_opts}->{$opt} : $opt);
+   die "Option $opt does not exist"
+      unless $long && exists $self->{opts}->{$long};
+   return $self->{opts}->{$long}->{got};
+}
+
+sub has {
+   my ( $self, $opt ) = @_;
+   my $long = (length $opt == 1 ? $self->{short_opts}->{$opt} : $opt);
+   return defined $long ? exists $self->{opts}->{$long} : 0;
+}
+
+sub set {
+   my ( $self, $opt, $val ) = @_;
+   my $long = (length $opt == 1 ? $self->{short_opts}->{$opt} : $opt);
+   die "Option $opt does not exist"
+      unless $long && exists $self->{opts}->{$long};
+   $self->{opts}->{$long}->{value} = $val;
+   return;
+}
+
+sub save_error {
+   my ( $self, $error ) = @_;
+   push @{$self->{errors}}, $error;
+   return;
+}
+
+sub errors {
+   my ( $self ) = @_;
+   return $self->{errors};
+}
+
+sub usage {
+   my ( $self ) = @_;
+   warn "No usage string is set" unless $self->{usage}; # XXX
+   return "Usage: " . ($self->{usage} || '') . "\n";
+}
+
+sub descr {
+   my ( $self ) = @_;
+   warn "No description string is set" unless $self->{description}; # XXX
+   my $descr  = ($self->{description} || $self->{program_name} || '')
+              . "  For more details, please use the --help option, "
+              . "or try 'perldoc $PROGRAM_NAME' "
+              . "for complete documentation.";
+   $descr = join("\n", $descr =~ m/(.{0,80})(?:\s+|$)/g)
+      unless $ENV{DONT_BREAK_LINES};
+   $descr =~ s/ +$//mg;
+   return $descr;
+}
+
+sub usage_or_errors {
+   my ( $self, $file, $return ) = @_;
+   $file ||= $self->{file} || __FILE__;
+
+   if ( !$self->{description} || !$self->{usage} ) {
+      PTDEBUG && _d("Getting description and usage from SYNOPSIS in", $file);
+      my %synop = $self->_parse_synopsis($file);
+      $self->{description} ||= $synop{description};
+      $self->{usage}       ||= $synop{usage};
+      PTDEBUG && _d("Description:", $self->{description},
+         "\nUsage:", $self->{usage});
+   }
+
+   if ( $self->{opts}->{help}->{got} ) {
+      print $self->print_usage() or die "Cannot print usage: $OS_ERROR";
+      exit 0 unless $return;
+   }
+   elsif ( scalar @{$self->{errors}} ) {
+      print $self->print_errors() or die "Cannot print errors: $OS_ERROR";
+      exit 0 unless $return;
+   }
+
+   return;
+}
+
+sub print_errors {
+   my ( $self ) = @_;
+   my $usage = $self->usage() . "\n";
+   if ( (my @errors = @{$self->{errors}}) ) {
+      $usage .= join("\n  * ", 'Errors in command-line arguments:', @errors)
+              . "\n";
+   }
+   return $usage . "\n" . $self->descr();
+}
+
+sub print_usage {
+   my ( $self ) = @_;
+   die "Run get_opts() before print_usage()" unless $self->{got_opts};
+   my @opts = values %{$self->{opts}};
+
+   my $maxl = max(
+      map {
+         length($_->{long})               # option long name
+         + ($_->{is_negatable} ? 4 : 0)   # "[no]" if opt is negatable
+         + ($_->{type} ? 2 : 0)           # "=x" where x is the opt type
+      }
+      @opts);
+
+   my $maxs = max(0,
+      map {
+         length($_)
+         + ($self->{opts}->{$_}->{is_negatable} ? 4 : 0)
+         + ($self->{opts}->{$_}->{type} ? 2 : 0)
+      }
+      values %{$self->{short_opts}});
+
+   my $lcol = max($maxl, ($maxs + 3));
+   my $rcol = 80 - $lcol - 6;
+   my $rpad = ' ' x ( 80 - $rcol );
+
+   $maxs = max($lcol - 3, $maxs);
+
+   my $usage = $self->descr() . "\n" . $self->usage();
+
+   my @groups = reverse sort grep { $_ ne 'default'; } keys %{$self->{groups}};
+   push @groups, 'default';
+
+   foreach my $group ( reverse @groups ) {
+      $usage .= "\n".($group eq 'default' ? 'Options' : $group).":\n\n";
+      foreach my $opt (
+         sort { $a->{long} cmp $b->{long} }
+         grep { $_->{group} eq $group }
+         @opts )
+      {
+         my $long  = $opt->{is_negatable} ? "[no]$opt->{long}" : $opt->{long};
+         my $short = $opt->{short};
+         my $desc  = $opt->{desc};
+
+         $long .= $opt->{type} ? "=$opt->{type}" : "";
+
+         if ( $opt->{type} && $opt->{type} eq 'm' ) {
+            my ($s) = $desc =~ m/\(suffix (.)\)/;
+            $s    ||= 's';
+            $desc =~ s/\s+\(suffix .\)//;
+            $desc .= ".  Optional suffix s=seconds, m=minutes, h=hours, "
+                   . "d=days; if no suffix, $s is used.";
+         }
+         $desc = join("\n$rpad", grep { $_ } $desc =~ m/(.{0,$rcol})(?:\s+|$)/g);
+         $desc =~ s/ +$//mg;
+         if ( $short ) {
+            $usage .= sprintf("  --%-${maxs}s -%s  %s\n", $long, $short, $desc);
+         }
+         else {
+            $usage .= sprintf("  --%-${lcol}s  %s\n", $long, $desc);
+         }
+      }
+   }
+
+   $usage .= "\nOption types: s=string, i=integer, f=float, h/H/a/A=comma-separated list, d=DSN, z=size, m=time\n";
+
+   if ( (my @rules = @{$self->{rules}}) ) {
+      $usage .= "\nRules:\n\n";
+      $usage .= join("\n", map { "  $_" } @rules) . "\n";
+   }
+   if ( $self->{DSNParser} ) {
+      $usage .= "\n" . $self->{DSNParser}->usage();
+   }
+   $usage .= "\nOptions and values after processing arguments:\n\n";
+   foreach my $opt ( sort { $a->{long} cmp $b->{long} } @opts ) {
+      my $val   = $opt->{value};
+      my $type  = $opt->{type} || '';
+      my $bool  = $opt->{spec} =~ m/^[\w-]+(?:\|[\w-])?!?$/;
+      $val      = $bool              ? ( $val ? 'TRUE' : 'FALSE' )
+                : !defined $val      ? '(No value)'
+                : $type eq 'd'       ? $self->{DSNParser}->as_string($val)
+                : $type =~ m/H|h/    ? join(',', sort keys %$val)
+                : $type =~ m/A|a/    ? join(',', @$val)
+                :                    $val;
+      $usage .= sprintf("  --%-${lcol}s  %s\n", $opt->{long}, $val);
+   }
+   return $usage;
+}
+
+sub prompt_noecho {
+   shift @_ if ref $_[0] eq __PACKAGE__;
+   my ( $prompt ) = @_;
+   local $OUTPUT_AUTOFLUSH = 1;
+   print $prompt
+      or die "Cannot print: $OS_ERROR";
+   my $response;
+   eval {
+      require Term::ReadKey;
+      Term::ReadKey::ReadMode('noecho');
+      chomp($response = <STDIN>);
+      Term::ReadKey::ReadMode('normal');
+      print "\n"
+         or die "Cannot print: $OS_ERROR";
+   };
+   if ( $EVAL_ERROR ) {
+      die "Cannot read response; is Term::ReadKey installed? $EVAL_ERROR";
+   }
+   return $response;
+}
+
+sub _read_config_file {
+   my ( $self, $filename ) = @_;
+   open my $fh, "<", $filename or die "Cannot open $filename: $OS_ERROR\n";
+   my @args;
+   my $prefix = '--';
+   my $parse  = 1;
+
+   LINE:
+   while ( my $line = <$fh> ) {
+      chomp $line;
+      next LINE if $line =~ m/^\s*(?:\#|\;|$)/;
+      $line =~ s/\s+#.*$//g;
+      $line =~ s/^\s+|\s+$//g;
+      if ( $line eq '--' ) {
+         $prefix = '';
+         $parse  = 0;
+         next LINE;
+      }
+      if ( $parse
+         && (my($opt, $arg) = $line =~ m/^\s*([^=\s]+?)(?:\s*=\s*(.*?)\s*)?$/)
+      ) {
+         push @args, grep { defined $_ } ("$prefix$opt", $arg);
+      }
+      elsif ( $line =~ m/./ ) {
+         push @args, $line;
+      }
+      else {
+         die "Syntax error in file $filename at line $INPUT_LINE_NUMBER";
+      }
+   }
+   close $fh;
+   return @args;
+}
+
+sub read_para_after {
+   my ( $self, $file, $regex ) = @_;
+   open my $fh, "<", $file or die "Can't open $file: $OS_ERROR";
+   local $INPUT_RECORD_SEPARATOR = '';
+   my $para;
+   while ( $para = <$fh> ) {
+      next unless $para =~ m/^=pod$/m;
+      last;
+   }
+   while ( $para = <$fh> ) {
+      next unless $para =~ m/$regex/;
+      last;
+   }
+   $para = <$fh>;
+   chomp($para);
+   close $fh or die "Can't close $file: $OS_ERROR";
+   return $para;
+}
+
+sub clone {
+   my ( $self ) = @_;
+
+   my %clone = map {
+      my $hashref  = $self->{$_};
+      my $val_copy = {};
+      foreach my $key ( keys %$hashref ) {
+         my $ref = ref $hashref->{$key};
+         $val_copy->{$key} = !$ref           ? $hashref->{$key}
+                           : $ref eq 'HASH'  ? { %{$hashref->{$key}} }
+                           : $ref eq 'ARRAY' ? [ @{$hashref->{$key}} ]
+                           : $hashref->{$key};
+      }
+      $_ => $val_copy;
+   } qw(opts short_opts defaults);
+
+   foreach my $scalar ( qw(got_opts) ) {
+      $clone{$scalar} = $self->{$scalar};
+   }
+
+   return bless \%clone;     
+}
+
+sub _parse_size {
+   my ( $self, $opt, $val ) = @_;
+
+   if ( lc($val || '') eq 'null' ) {
+      PTDEBUG && _d('NULL size for', $opt->{long});
+      $opt->{value} = 'null';
+      return;
+   }
+
+   my %factor_for = (k => 1_024, M => 1_048_576, G => 1_073_741_824);
+   my ($pre, $num, $factor) = $val =~ m/^([+-])?(\d+)([kMG])?$/;
+   if ( defined $num ) {
+      if ( $factor ) {
+         $num *= $factor_for{$factor};
+         PTDEBUG && _d('Setting option', $opt->{y},
+            'to num', $num, '* factor', $factor);
+      }
+      $opt->{value} = ($pre || '') . $num;
+   }
+   else {
+      $self->save_error("Invalid size for --$opt->{long}: $val");
+   }
+   return;
+}
+
+sub _parse_attribs {
+   my ( $self, $option, $attribs ) = @_;
+   my $types = $self->{types};
+   return $option
+      . ($attribs->{'short form'} ? '|' . $attribs->{'short form'}   : '' )
+      . ($attribs->{'negatable'}  ? '!'                              : '' )
+      . ($attribs->{'cumulative'} ? '+'                              : '' )
+      . ($attribs->{'type'}       ? '=' . $types->{$attribs->{type}} : '' );
+}
+
+sub _parse_synopsis {
+   my ( $self, $file ) = @_;
+   $file ||= $self->{file} || __FILE__;
+   PTDEBUG && _d("Parsing SYNOPSIS in", $file);
+
+   local $INPUT_RECORD_SEPARATOR = '';  # read paragraphs
+   open my $fh, "<", $file or die "Cannot open $file: $OS_ERROR";
+   my $para;
+   1 while defined($para = <$fh>) && $para !~ m/^=head1 SYNOPSIS/;
+   die "$file does not contain a SYNOPSIS section" unless $para;
+   my @synop;
+   for ( 1..2 ) {  # 1 for the usage, 2 for the description
+      my $para = <$fh>;
+      push @synop, $para;
+   }
+   close $fh;
+   PTDEBUG && _d("Raw SYNOPSIS text:", @synop);
+   my ($usage, $desc) = @synop;
+   die "The SYNOPSIS section in $file is not formatted properly"
+      unless $usage && $desc;
+
+   $usage =~ s/^\s*Usage:\s+(.+)/$1/;
+   chomp $usage;
+
+   $desc =~ s/\n/ /g;
+   $desc =~ s/\s{2,}/ /g;
+   $desc =~ s/\. ([A-Z][a-z])/.  $1/g;
+   $desc =~ s/\s+$//;
+
+   return (
+      description => $desc,
+      usage       => $usage,
+   );
+};
+
+sub _d {
+   my ($package, undef, $line) = caller 0;
+   @_ = map { (my $temp = $_) =~ s/\n/\n# /g; $temp; }
+        map { defined $_ ? $_ : 'undef' }
+        @_;
+   print STDERR "# $package:$line $PID ", join(' ', @_), "\n";
+}
+
+if ( PTDEBUG ) {
+   print '# ', $^X, ' ', $], "\n";
+   if ( my $uname = `uname -a` ) {
+      $uname =~ s/\s+/ /g;
+      print "# $uname\n";
+   }
+   print '# Arguments: ',
+      join(' ', map { my $a = "_[$_]_"; $a =~ s/\n/\n# /g; $a; } @ARGV), "\n";
+}
+
+1;
+}
+# ###########################################################################
+# End OptionParser package
+# ###########################################################################
+
+# ###########################################################################
+# QueryParser package
+# This package is a copy without comments from the original.  The original
+# with comments and its test file can be found in the Bazaar repository at,
+#   lib/QueryParser.pm
+#   t/lib/QueryParser.t
+# See https://launchpad.net/percona-toolkit for more information.
+# ###########################################################################
+{
+package QueryParser;
+
+use strict;
+use warnings FATAL => 'all';
+use English qw(-no_match_vars);
+use constant PTDEBUG => $ENV{PTDEBUG} || 0;
+
+our $tbl_ident = qr/(?:`[^`]+`|\w+)(?:\.(?:`[^`]+`|\w+))?/;
+our $tbl_regex = qr{
+         \b(?:FROM|JOIN|(?<!KEY\s)UPDATE|INTO) # Words that precede table names
+         \b\s*
+         \(?                                   # Optional paren around tables
+         ($tbl_ident
+            (?: (?:\s+ (?:AS\s+)? \w+)?, \s*$tbl_ident )*
+         )
+      }xio;
+our $has_derived = qr{
+      \b(?:FROM|JOIN|,)
+      \s*\(\s*SELECT
+   }xi;
+
+our $data_def_stmts = qr/(?:CREATE|ALTER|TRUNCATE|DROP|RENAME)/i;
+
+our $data_manip_stmts = qr/(?:INSERT|UPDATE|DELETE|REPLACE)/i;
+
+sub new {
+   my ( $class ) = @_;
+   bless {}, $class;
+}
+
+sub get_tables {
+   my ( $self, $query ) = @_;
+   return unless $query;
+   PTDEBUG && _d('Getting tables for', $query);
+
+   my ( $ddl_stmt ) = $query =~ m/^\s*($data_def_stmts)\b/i;
+   if ( $ddl_stmt ) {
+      PTDEBUG && _d('Special table type:', $ddl_stmt);
+      $query =~ s/IF\s+(?:NOT\s+)?EXISTS//i;
+      if ( $query =~ m/$ddl_stmt DATABASE\b/i ) {
+         PTDEBUG && _d('Query alters a database, not a table');
+         return ();
+      }
+      if ( $ddl_stmt =~ m/CREATE/i && $query =~ m/$ddl_stmt\b.+?\bSELECT\b/i ) {
+         my ($select) = $query =~ m/\b(SELECT\b.+)/is;
+         PTDEBUG && _d('CREATE TABLE ... SELECT:', $select);
+         return $self->get_tables($select);
+      }
+      my ($tbl) = $query =~ m/TABLE\s+($tbl_ident)(\s+.*)?/i;
+      PTDEBUG && _d('Matches table:', $tbl);
+      return ($tbl);
+   }
+
+   $query =~ s/ (?:LOW_PRIORITY|IGNORE|STRAIGHT_JOIN)//ig;
+
+   if ( $query =~ /^\s*LOCK TABLES/i ) {
+      PTDEBUG && _d('Special table type: LOCK TABLES');
+      $query =~ s/^(\s*LOCK TABLES\s+)//;
+      $query =~ s/\s+(?:READ|WRITE|LOCAL)+\s*//g;
+      PTDEBUG && _d('Locked tables:', $query);
+      $query = "FROM $query";
+   }
+
+   $query =~ s/\\["']//g;                # quoted strings
+   $query =~ s/".*?"/?/sg;               # quoted strings
+   $query =~ s/'.*?'/?/sg;               # quoted strings
+
+   my @tables;
+   foreach my $tbls ( $query =~ m/$tbl_regex/gio ) {
+      PTDEBUG && _d('Match tables:', $tbls);
+
+      next if $tbls =~ m/\ASELECT\b/i;
+
+      foreach my $tbl ( split(',', $tbls) ) {
+         $tbl =~ s/\s*($tbl_ident)(\s+.*)?/$1/gio;
+
+         if ( $tbl !~ m/[a-zA-Z]/ ) {
+            PTDEBUG && _d('Skipping suspicious table name:', $tbl);
+            next;
+         }
+
+         push @tables, $tbl;
+      }
+   }
+   return @tables;
+}
+
+sub has_derived_table {
+   my ( $self, $query ) = @_;
+   my $match = $query =~ m/$has_derived/;
+   PTDEBUG && _d($query, 'has ' . ($match ? 'a' : 'no') . ' derived table');
+   return $match;
+}
+
+sub get_aliases {
+   my ( $self, $query, $list ) = @_;
+
+   my $result = {
+      DATABASE => {},
+      TABLE    => {},
+   };
+   return $result unless $query;
+
+   $query =~ s/ (?:LOW_PRIORITY|IGNORE|STRAIGHT_JOIN)//ig;
+
+   $query =~ s/ (?:INNER|OUTER|CROSS|LEFT|RIGHT|NATURAL)//ig;
+
+   my @tbl_refs;
+   my ($tbl_refs, $from) = $query =~ m{
+      (
+         (FROM|INTO|UPDATE)\b\s*   # Keyword before table refs
+         .+?                       # Table refs
+      )
+      (?:\s+|\z)                   # If the query does not end with the table
+      (?:WHERE|ORDER|LIMIT|HAVING|SET|VALUES|\z) # Keyword after table refs
+   }ix;
+
+   if ( $tbl_refs ) {
+
+      if ( $query =~ m/^(?:INSERT|REPLACE)/i ) {
+         $tbl_refs =~ s/\([^\)]+\)\s*//;
+      }
+
+      PTDEBUG && _d('tbl refs:', $tbl_refs);
+
+      my $before_tbl = qr/(?:,|JOIN|\s|$from)+/i;
+
+      my $after_tbl  = qr/(?:,|JOIN|ON|USING|\z)/i;
+
+      $tbl_refs =~ s/ = /=/g;
+
+      while (
+         $tbl_refs =~ m{
+            $before_tbl\b\s*
+               ( ($tbl_ident) (?:\s+ (?:AS\s+)? (\w+))? )
+            \s*$after_tbl
+         }xgio )
+      {
+         my ( $tbl_ref, $db_tbl, $alias ) = ($1, $2, $3);
+         PTDEBUG && _d('Match table:', $tbl_ref);
+         push @tbl_refs, $tbl_ref;
+         $alias = $self->trim_identifier($alias);
+
+         if ( $tbl_ref =~ m/^AS\s+\w+/i ) {
+            PTDEBUG && _d('Subquery', $tbl_ref);
+            $result->{TABLE}->{$alias} = undef;
+            next;
+         }
+
+         my ( $db, $tbl ) = $db_tbl =~ m/^(?:(.*?)\.)?(.*)/;
+         $db  = $self->trim_identifier($db);
+         $tbl = $self->trim_identifier($tbl);
+         $result->{TABLE}->{$alias || $tbl} = $tbl;
+         $result->{DATABASE}->{$tbl}        = $db if $db;
+      }
+   }
+   else {
+      PTDEBUG && _d("No tables ref in", $query);
+   }
+
+   if ( $list ) {
+      return \@tbl_refs;
+   }
+   else {
+      return $result;
+   }
+}
+
+sub split {
+   my ( $self, $query ) = @_;
+   return unless $query;
+   $query = $self->clean_query($query);
+   PTDEBUG && _d('Splitting', $query);
+
+   my $verbs = qr{SELECT|INSERT|UPDATE|DELETE|REPLACE|UNION|CREATE}i;
+
+   my @split_statements = grep { $_ } split(m/\b($verbs\b(?!(?:\s*\()))/io, $query);
+
+   my @statements;
+   if ( @split_statements == 1 ) {
+      push @statements, $query;
+   }
+   else {
+      for ( my $i = 0; $i <= $#split_statements; $i += 2 ) {
+         push @statements, $split_statements[$i].$split_statements[$i+1];
+
+         if ( $statements[-2] && $statements[-2] =~ m/on duplicate key\s+$/i ) {
+            $statements[-2] .= pop @statements;
+         }
+      }
+   }
+
+   PTDEBUG && _d('statements:', map { $_ ? "<$_>" : 'none' } @statements);
+   return @statements;
+}
+
+sub clean_query {
+   my ( $self, $query ) = @_;
+   return unless $query;
+   $query =~ s!/\*.*?\*/! !g;  # Remove /* comment blocks */
+   $query =~ s/^\s+//;         # Remove leading spaces
+   $query =~ s/\s+$//;         # Remove trailing spaces
+   $query =~ s/\s{2,}/ /g;     # Remove extra spaces
+   return $query;
+}
+
+sub split_subquery {
+   my ( $self, $query ) = @_;
+   return unless $query;
+   $query = $self->clean_query($query);
+   $query =~ s/;$//;
+
+   my @subqueries;
+   my $sqno = 0;  # subquery number
+   my $pos  = 0;
+   while ( $query =~ m/(\S+)(?:\s+|\Z)/g ) {
+      $pos = pos($query);
+      my $word = $1;
+      PTDEBUG && _d($word, $sqno);
+      if ( $word =~ m/^\(?SELECT\b/i ) {
+         my $start_pos = $pos - length($word) - 1;
+         if ( $start_pos ) {
+            $sqno++;
+            PTDEBUG && _d('Subquery', $sqno, 'starts at', $start_pos);
+            $subqueries[$sqno] = {
+               start_pos => $start_pos,
+               end_pos   => 0,
+               len       => 0,
+               words     => [$word],
+               lp        => 1, # left parentheses
+               rp        => 0, # right parentheses
+               done      => 0,
+            };
+         }
+         else {
+            PTDEBUG && _d('Main SELECT at pos 0');
+         }
+      }
+      else {
+         next unless $sqno;  # next unless we're in a subquery
+         PTDEBUG && _d('In subquery', $sqno);
+         my $sq = $subqueries[$sqno];
+         if ( $sq->{done} ) {
+            PTDEBUG && _d('This subquery is done; SQL is for',
+               ($sqno - 1 ? "subquery $sqno" : "the main SELECT"));
+            next;
+         }
+         push @{$sq->{words}}, $word;
+         my $lp = ($word =~ tr/\(//) || 0;
+         my $rp = ($word =~ tr/\)//) || 0;
+         PTDEBUG && _d('parentheses left', $lp, 'right', $rp);
+         if ( ($sq->{lp} + $lp) - ($sq->{rp} + $rp) == 0 ) {
+            my $end_pos = $pos - 1;
+            PTDEBUG && _d('Subquery', $sqno, 'ends at', $end_pos);
+            $sq->{end_pos} = $end_pos;
+            $sq->{len}     = $end_pos - $sq->{start_pos};
+         }
+      }
+   }
+
+   for my $i ( 1..$#subqueries ) {
+      my $sq = $subqueries[$i];
+      next unless $sq;
+      $sq->{sql} = join(' ', @{$sq->{words}});
+      substr $query,
+         $sq->{start_pos} + 1,  # +1 for (
+         $sq->{len} - 1,        # -1 for )
+         "__subquery_$i";
+   }
+
+   return $query, map { $_->{sql} } grep { defined $_ } @subqueries;
+}
+
+sub query_type {
+   my ( $self, $query, $qr ) = @_;
+   my ($type, undef) = $qr->distill_verbs($query);
+   my $rw;
+   if ( $type =~ m/^SELECT\b/ ) {
+      $rw = 'read';
+   }
+   elsif ( $type =~ m/^$data_manip_stmts\b/
+           || $type =~ m/^$data_def_stmts\b/  ) {
+      $rw = 'write'
+   }
+
+   return {
+      type => $type,
+      rw   => $rw,
+   }
+}
+
+sub get_columns {
+   my ( $self, $query ) = @_;
+   my $cols = [];
+   return $cols unless $query;
+   my $cols_def;
+
+   if ( $query =~ m/^SELECT/i ) {
+      $query =~ s/
+         ^SELECT\s+
+           (?:ALL
+              |DISTINCT
+              |DISTINCTROW
+              |HIGH_PRIORITY
+              |STRAIGHT_JOIN
+              |SQL_SMALL_RESULT
+              |SQL_BIG_RESULT
+              |SQL_BUFFER_RESULT
+              |SQL_CACHE
+              |SQL_NO_CACHE
+              |SQL_CALC_FOUND_ROWS
+           )\s+
+      /SELECT /xgi;
+      ($cols_def) = $query =~ m/^SELECT\s+(.+?)\s+FROM/i;
+   }
+   elsif ( $query =~ m/^(?:INSERT|REPLACE)/i ) {
+      ($cols_def) = $query =~ m/\(([^\)]+)\)\s*VALUE/i;
+   }
+
+   PTDEBUG && _d('Columns:', $cols_def);
+   if ( $cols_def ) {
+      @$cols = split(',', $cols_def);
+      map {
+         my $col = $_;
+         $col = s/^\s+//g;
+         $col = s/\s+$//g;
+         $col;
+      } @$cols;
+   }
+
+   return $cols;
+}
+
+sub parse {
+   my ( $self, $query ) = @_;
+   return unless $query;
+   my $parsed = {};
+
+   $query =~ s/\n/ /g;
+   $query = $self->clean_query($query);
+
+   $parsed->{query}   = $query,
+   $parsed->{tables}  = $self->get_aliases($query, 1);
+   $parsed->{columns} = $self->get_columns($query);
+
+   my ($type) = $query =~ m/^(\w+)/;
+   $parsed->{type} = lc $type;
+
+
+   $parsed->{sub_queries} = [];
+
+   return $parsed;
+}
+
+sub extract_tables {
+   my ( $self, %args ) = @_;
+   my $query      = $args{query};
+   my $default_db = $args{default_db};
+   my $q          = $self->{Quoter} || $args{Quoter};
+   return unless $query;
+   PTDEBUG && _d('Extracting tables');
+   my @tables;
+   my %seen;
+   foreach my $db_tbl ( $self->get_tables($query) ) {
+      next unless $db_tbl;
+      next if $seen{$db_tbl}++; # Unique-ify for issue 337.
+      my ( $db, $tbl ) = $q->split_unquote($db_tbl);
+      push @tables, [ $db || $default_db, $tbl ];
+   }
+   return @tables;
+}
+
+sub trim_identifier {
+   my ($self, $str) = @_;
+   return unless defined $str;
+   $str =~ s/`//g;
+   $str =~ s/^\s+//;
+   $str =~ s/\s+$//;
+   return $str;
+}
+
+sub _d {
+   my ($package, undef, $line) = caller 0;
+   @_ = map { (my $temp = $_) =~ s/\n/\n# /g; $temp; }
+        map { defined $_ ? $_ : 'undef' }
+        @_;
+   print STDERR "# $package:$line $PID ", join(' ', @_), "\n";
+}
+
+1;
+}
+# ###########################################################################
+# End QueryParser package
+# ###########################################################################
+
+# ###########################################################################
+# QueryRewriter package
+# This package is a copy without comments from the original.  The original
+# with comments and its test file can be found in the Bazaar repository at,
+#   lib/QueryRewriter.pm
+#   t/lib/QueryRewriter.t
+# See https://launchpad.net/percona-toolkit for more information.
+# ###########################################################################
+{
+package QueryRewriter;
+
+use strict;
+use warnings FATAL => 'all';
+use English qw(-no_match_vars);
+use constant PTDEBUG => $ENV{PTDEBUG} || 0;
+
+our $verbs   = qr{^SHOW|^FLUSH|^COMMIT|^ROLLBACK|^BEGIN|SELECT|INSERT
+                  |UPDATE|DELETE|REPLACE|^SET|UNION|^START|^LOCK}xi;
+my $quote_re = qr/"(?:(?!(?<!\\)").)*"|'(?:(?!(?<!\\)').)*'/; # Costly!
+my $bal;
+$bal         = qr/
+                  \(
+                  (?:
+                     (?> [^()]+ )    # Non-parens without backtracking
+                     |
+                     (??{ $bal })    # Group with matching parens
+                  )*
+                  \)
+                 /x;
+
+my $olc_re = qr/(?:--|#)[^'"\r\n]*(?=[\r\n]|\Z)/;  # One-line comments
+my $mlc_re = qr#/\*[^!].*?\*/#sm;                  # But not /*!version */
+my $vlc_re = qr#/\*.*?[0-9+].*?\*/#sm;             # For SHOW + /*!version */
+my $vlc_rf = qr#^(SHOW).*?/\*![0-9+].*?\*/#sm;     # Variation for SHOW
+
+
+sub new {
+   my ( $class, %args ) = @_;
+   my $self = { %args };
+   return bless $self, $class;
+}
+
+sub strip_comments {
+   my ( $self, $query ) = @_;
+   return unless $query;
+   $query =~ s/$olc_re//go;
+   $query =~ s/$mlc_re//go;
+   if ( $query =~ m/$vlc_rf/i ) { # contains show + version
+      $query =~ s/$vlc_re//go;
+   }
+   return $query;
+}
+
+sub shorten {
+   my ( $self, $query, $length ) = @_;
+   $query =~ s{
+      \A(
+         (?:INSERT|REPLACE)
+         (?:\s+LOW_PRIORITY|DELAYED|HIGH_PRIORITY|IGNORE)?
+         (?:\s\w+)*\s+\S+\s+VALUES\s*\(.*?\)
+      )
+      \s*,\s*\(.*?(ON\s+DUPLICATE|\Z)}
+      {$1 /*... omitted ...*/$2}xsi;
+
+   return $query unless $query =~ m/IN\s*\(\s*(?!select)/i;
+
+   my $last_length  = 0;
+   my $query_length = length($query);
+   while (
+      $length          > 0
+      && $query_length > $length
+      && $query_length < ( $last_length || $query_length + 1 )
+   ) {
+      $last_length = $query_length;
+      $query =~ s{
+         (\bIN\s*\()    # The opening of an IN list
+         ([^\)]+)       # Contents of the list, assuming no item contains paren
+         (?=\))           # Close of the list
+      }
+      {
+         $1 . __shorten($2)
+      }gexsi;
+   }
+
+   return $query;
+}
+
+sub __shorten {
+   my ( $snippet ) = @_;
+   my @vals = split(/,/, $snippet);
+   return $snippet unless @vals > 20;
+   my @keep = splice(@vals, 0, 20);  # Remove and save the first 20 items
+   return
+      join(',', @keep)
+      . "/*... omitted "
+      . scalar(@vals)
+      . " items ...*/";
+}
+
+sub fingerprint {
+   my ( $self, $query ) = @_;
+
+   $query =~ m#\ASELECT /\*!40001 SQL_NO_CACHE \*/ \* FROM `# # mysqldump query
+      && return 'mysqldump';
+   $query =~ m#/\*\w+\.\w+:[0-9]/[0-9]\*/#     # pt-table-checksum, etc query
+      && return 'percona-toolkit';
+   $query =~ m/\Aadministrator command: /
+      && return $query;
+   $query =~ m/\A\s*(call\s+\S+)\(/i
+      && return lc($1); # Warning! $1 used, be careful.
+   if ( my ($beginning) = $query =~ m/\A((?:INSERT|REPLACE)(?: IGNORE)?\s+INTO.+?VALUES\s*\(.*?\))\s*,\s*\(/is ) {
+      $query = $beginning; # Shorten multi-value INSERT statements ASAP
+   }
+  
+   $query =~ s/$olc_re//go;
+   $query =~ s/$mlc_re//go;
+   $query =~ s/\Ause \S+\Z/use ?/i       # Abstract the DB in USE
+      && return $query;
+
+   $query =~ s/\\["']//g;                # quoted strings
+   $query =~ s/".*?"/?/sg;               # quoted strings
+   $query =~ s/'.*?'/?/sg;               # quoted strings
+
+   if ( $self->{match_md5_checksums} ) { 
+      $query =~ s/([._-])[a-f0-9]{32}/$1?/g;
+   }
+
+   if ( !$self->{match_embedded_numbers} ) {
+      $query =~ s/[0-9+-][0-9a-f.xb+-]*/?/g;
+   }
+   else {
+      $query =~ s/\b[0-9+-][0-9a-f.xb+-]*/?/g;
+   }
+
+   if ( $self->{match_md5_checksums} ) {
+      $query =~ s/[xb+-]\?/?/g;                
+   }
+   else {
+      $query =~ s/[xb.+-]\?/?/g;
+   }
+
+   $query =~ s/\A\s+//;                  # Chop off leading whitespace
+   chomp $query;                         # Kill trailing whitespace
+   $query =~ tr[ \n\t\r\f][ ]s;          # Collapse whitespace
+   $query = lc $query;
+   $query =~ s/\bnull\b/?/g;             # Get rid of NULLs
+   $query =~ s{                          # Collapse IN and VALUES lists
+               \b(in|values?)(?:[\s,]*\([\s?,]*\))+
+              }
+              {$1(?+)}gx;
+   $query =~ s{                          # Collapse UNION
+               \b(select\s.*?)(?:(\sunion(?:\sall)?)\s\1)+
+              }
+              {$1 /*repeat$2*/}xg;
+   $query =~ s/\blimit \?(?:, ?\?| offset \?)?/limit ?/; # LIMIT
+
+   if ( $query =~ m/\bORDER BY /gi ) {  # Find, anchor on ORDER BY clause
+      1 while $query =~ s/\G(.+?)\s+ASC/$1/gi && pos $query;
+   }
+
+   return $query;
+}
+
+sub distill_verbs {
+   my ( $self, $query ) = @_;
+
+   $query =~ m/\A\s*call\s+(\S+)\(/i && return "CALL $1";
+   $query =~ m/\A\s*use\s+/          && return "USE";
+   $query =~ m/\A\s*UNLOCK TABLES/i  && return "UNLOCK";
+   $query =~ m/\A\s*xa\s+(\S+)/i     && return "XA_$1";
+
+   if ( $query =~ m/\Aadministrator command:/ ) {
+      $query =~ s/administrator command:/ADMIN/;
+      $query = uc $query;
+      return $query;
+   }
+
+   $query = $self->strip_comments($query);
+
+   if ( $query =~ m/\A\s*SHOW\s+/i ) {
+      PTDEBUG && _d($query);
+
+      $query = uc $query;
+      $query =~ s/\s+(?:GLOBAL|SESSION|FULL|STORAGE|ENGINE)\b/ /g;
+      $query =~ s/\s+COUNT[^)]+\)//g;
+
+      $query =~ s/\s+(?:FOR|FROM|LIKE|WHERE|LIMIT|IN)\b.+//ms;
+
+      $query =~ s/\A(SHOW(?:\s+\S+){1,2}).*\Z/$1/s;
+      $query =~ s/\s+/ /g;
+      PTDEBUG && _d($query);
+      return $query;
+   }
+
+   eval $QueryParser::data_def_stmts;
+   eval $QueryParser::tbl_ident;
+   my ( $dds ) = $query =~ /^\s*($QueryParser::data_def_stmts)\b/i;
+   if ( $dds) {
+      my ( $obj ) = $query =~ m/$dds.+(DATABASE|TABLE)\b/i;
+      $obj = uc $obj if $obj;
+      PTDEBUG && _d('Data def statment:', $dds, 'obj:', $obj);
+      my ($db_or_tbl)
+         = $query =~ m/(?:TABLE|DATABASE)\s+($QueryParser::tbl_ident)(\s+.*)?/i;
+      PTDEBUG && _d('Matches db or table:', $db_or_tbl);
+      return uc($dds . ($obj ? " $obj" : '')), $db_or_tbl;
+   }
+
+   my @verbs = $query =~ m/\b($verbs)\b/gio;
+   @verbs    = do {
+      my $last = '';
+      grep { my $pass = $_ ne $last; $last = $_; $pass } map { uc } @verbs;
+   };
+
+   if ( ($verbs[0] || '') eq 'SELECT' && @verbs > 1 ) {
+      PTDEBUG && _d("False-positive verbs after SELECT:", @verbs[1..$#verbs]);
+      my $union = grep { $_ eq 'UNION' } @verbs;
+      @verbs    = $union ? qw(SELECT UNION) : qw(SELECT);
+   }
+
+   my $verb_str = join(q{ }, @verbs);
+   return $verb_str;
+}
+
+sub __distill_tables {
+   my ( $self, $query, $table, %args ) = @_;
+   my $qp = $args{QueryParser} || $self->{QueryParser};
+   die "I need a QueryParser argument" unless $qp;
+
+   my @tables = map {
+      $_ =~ s/`//g;
+      $_ =~ s/(_?)[0-9]+/$1?/g;
+      $_;
+   } grep { defined $_ } $qp->get_tables($query);
+
+   push @tables, $table if $table;
+
+   @tables = do {
+      my $last = '';
+      grep { my $pass = $_ ne $last; $last = $_; $pass } @tables;
+   };
+
+   return @tables;
+}
+
+sub distill {
+   my ( $self, $query, %args ) = @_;
+
+   if ( $args{generic} ) {
+      my ($cmd, $arg) = $query =~ m/^(\S+)\s+(\S+)/;
+      return '' unless $cmd;
+      $query = (uc $cmd) . ($arg ? " $arg" : '');
+   }
+   else {
+      my ($verbs, $table)  = $self->distill_verbs($query, %args);
+
+      if ( $verbs && $verbs =~ m/^SHOW/ ) {
+         my %alias_for = qw(
+            SCHEMA   DATABASE
+            KEYS     INDEX
+            INDEXES  INDEX
+         );
+         map { $verbs =~ s/$_/$alias_for{$_}/ } keys %alias_for;
+         $query = $verbs;
+      }
+      else {
+         my @tables = $self->__distill_tables($query, $table, %args);
+         $query     = join(q{ }, $verbs, @tables); 
+      } 
+   }
+
+   if ( $args{trf} ) {
+      $query = $args{trf}->($query, %args);
+   }
+
+   return $query;
+}
+
+sub convert_to_select {
+   my ( $self, $query ) = @_;
+   return unless $query;
+
+   return if $query =~ m/=\s*\(\s*SELECT /i;
+
+   $query =~ s{
+                 \A.*?
+                 update(?:\s+(?:low_priority|ignore))?\s+(.*?)
+                 \s+set\b(.*?)
+                 (?:\s*where\b(.*?))?
+                 (limit\s*[0-9]+(?:\s*,\s*[0-9]+)?)?
+                 \Z
+              }
+              {__update_to_select($1, $2, $3, $4)}exsi
+      || $query =~ s{
+                    \A.*?
+                    (?:insert(?:\s+ignore)?|replace)\s+
+                    .*?\binto\b(.*?)\(([^\)]+)\)\s*
+                    values?\s*(\(.*?\))\s*
+                    (?:\blimit\b|on\s+duplicate\s+key.*)?\s*
+                    \Z
+                 }
+                 {__insert_to_select($1, $2, $3)}exsi
+      || $query =~ s{
+                    \A.*?
+                    (?:insert(?:\s+ignore)?|replace)\s+
+                    (?:.*?\binto)\b(.*?)\s*
+                    set\s+(.*?)\s*
+                    (?:\blimit\b|on\s+duplicate\s+key.*)?\s*
+                    \Z
+                 }
+                 {__insert_to_select_with_set($1, $2)}exsi
+      || $query =~ s{
+                    \A.*?
+                    delete\s+(.*?)
+                    \bfrom\b(.*)
+                    \Z
+                 }
+                 {__delete_to_select($1, $2)}exsi;
+   $query =~ s/\s*on\s+duplicate\s+key\s+update.*\Z//si;
+   $query =~ s/\A.*?(?=\bSELECT\s*\b)//ism;
+   return $query;
+}
+
+sub convert_select_list {
+   my ( $self, $query ) = @_;
+   $query =~ s{
+               \A\s*select(.*?)\bfrom\b
+              }
+              {$1 =~ m/\*/ ? "select 1 from" : "select isnull(coalesce($1)) from"}exi;
+   return $query;
+}
+
+sub __delete_to_select {
+   my ( $delete, $join ) = @_;
+   if ( $join =~ m/\bjoin\b/ ) {
+      return "select 1 from $join";
+   }
+   return "select * from $join";
+}
+
+sub __insert_to_select {
+   my ( $tbl, $cols, $vals ) = @_;
+   PTDEBUG && _d('Args:', @_);
+   my @cols = split(/,/, $cols);
+   PTDEBUG && _d('Cols:', @cols);
+   $vals =~ s/^\(|\)$//g; # Strip leading/trailing parens
+   my @vals = $vals =~ m/($quote_re|[^,]*${bal}[^,]*|[^,]+)/g;
+   PTDEBUG && _d('Vals:', @vals);
+   if ( @cols == @vals ) {
+      return "select * from $tbl where "
+         . join(' and ', map { "$cols[$_]=$vals[$_]" } (0..$#cols));
+   }
+   else {
+      return "select * from $tbl limit 1";
+   }
+}
+
+sub __insert_to_select_with_set {
+   my ( $from, $set ) = @_;
+   $set =~ s/,/ and /g;
+   return "select * from $from where $set ";
+}
+
+sub __update_to_select {
+   my ( $from, $set, $where, $limit ) = @_;
+   return "select $set from $from "
+      . ( $where ? "where $where" : '' )
+      . ( $limit ? " $limit "      : '' );
+}
+
+sub wrap_in_derived {
+   my ( $self, $query ) = @_;
+   return unless $query;
+   return $query =~ m/\A\s*select/i
+      ? "select 1 from ($query) as x limit 1"
+      : $query;
+}
+
+sub _d {
+   my ($package, undef, $line) = caller 0;
+   @_ = map { (my $temp = $_) =~ s/\n/\n# /g; $temp; }
+        map { defined $_ ? $_ : 'undef' }
+        @_;
+   print STDERR "# $package:$line $PID ", join(' ', @_), "\n";
+}
+
+1;
+}
+# ###########################################################################
+# End QueryRewriter package
+# ###########################################################################
+
+# ###########################################################################
+# This is a combination of modules and programs in one -- a runnable module.
+# http://www.perl.com/pub/a/2006/07/13/lightning-articles.html?page=last
+# Or, look it up in the Camel book on pages 642 and 643 in the 3rd edition.
+#
+# Check at the end of this package for the call to main() which actually runs
+# the program.
+# ###########################################################################
+package pt_fingerprint;
+
+use English qw(-no_match_vars);
+use Data::Dumper;
+$Data::Dumper::Indent = 1;
+$OUTPUT_AUTOFLUSH     = 1;
+
+use constant MKDEBUG => $ENV{MKDEBUG} || 0;
+
+sub main {
+   @ARGV    = @_;  # set global ARGV for this package
+
+   # ##########################################################################
+   # Get configuration information.
+   # ##########################################################################
+   my $o = new OptionParser();
+   $o->get_specs();
+   $o->get_opts();
+   $o->usage_or_errors();
+
+   my $qp = new QueryParser();
+   my $qr = new QueryRewriter(
+      QueryParser            => $qp,
+      match_md5_checksums    => $o->get('match-md5-checksums'),
+      match_embedded_numbers => $o->get('match-embedded-numbers'),
+   );
+
+   if ( $o->got('query') ) {
+      print $qr->fingerprint($o->get('query')), "\n";
+   }
+   else {
+      local $INPUT_RECORD_SEPARATOR = ";\n";
+      while ( <> ) {
+         my $query = $_;
+         chomp $query;
+         $query =~ s/^#.+$//mg;
+         $query =~ s/^\s+//;
+         next unless $query =~ m/^\w/;
+         print $qr->fingerprint($query), "\n";
+      }
+   }
+}
+
+# ############################################################################
+# Run the program.
+# ############################################################################
+if ( !caller ) { exit main(@ARGV); }
+
+1; # Because this is a module as well as a script.
+
+# #############################################################################
+# Documentation.
+# #############################################################################
+
+=pod
+
+=head1 NAME
+
+pt-fingerprint - Convert queries into fingerprints.
+
+=head1 SYNOPSIS
+
+Usage: pt-fingerprint [OPTIONS] [FILES]
+
+pt-fingerprint converts queries into fingerprints.  With the --query
+option, converts the option's value into a fingerprint.  With no options, treats
+command-line arguments as FILEs and reads and converts semicolon-separated
+queries from the FILEs. When FILE is -, it read standard input.
+
+Convert a single query:
+
+  pt-fingerprint --query "select a, b, c from users where id = 500"
+
+Convert a file full of queries:
+
+  pt-fingerprint /path/to/file.txt
+
+=head1 RISKS
+
+The following section is included to inform users about the potential risks,
+whether known or unknown, of using this tool.  The two main categories of risks
+are those created by the nature of the tool (e.g. read-only tools vs. read-write
+tools) and those created by bugs.
+
+The pt-fingerprint tool simply reads data and transforms it, so risks are
+minimal.
+
+See also L<"BUGS"> for more information on filing bugs and getting help.
+
+=head1 DESCRIPTION
+
+A query fingerprint is the abstracted form of a query, which makes it possible
+to group similar queries together.  Abstracting a query removes literal values,
+normalizes whitespace, and so on.  For example, consider these two queries:
+
+  SELECT name, password FROM user WHERE id='12823';
+  select name,   password from user
+     where id=5;
+
+Both of those queries will fingerprint to
+
+  select name, password from user where id=?
+
+Once the query's fingerprint is known, we can then talk about a query as though
+it represents all similar queries.
+
+Query fingerprinting accommodates a great many special cases, which have proven
+necessary in the real world.  For example, an IN list with 5 literals is really
+equivalent to one with 4 literals, so lists of literals are collapsed to a
+single one.  If you want to understand more about how and why all of these cases
+are handled, please review the test cases in the Subversion repository.  If you
+find something that is not fingerprinted properly, please submit a bug report
+with a reproducible test case.  Here is a list of transformations during
+fingerprinting, which might not be exhaustive:
+
+=over
+
+=item *
+
+Group all SELECT queries from mysqldump together, even if they are against
+different tables.  Ditto for all of pt-table-checksum's checksum queries.
+
+=item *
+
+Shorten multi-value INSERT statements to a single VALUES() list.
+
+=item *
+
+Strip comments.
+
+=item *
+
+Abstract the databases in USE statements, so all USE statements are grouped
+together.
+
+=item *
+
+Replace all literals, such as quoted strings.  For efficiency, the code that
+replaces literal numbers is somewhat non-selective, and might replace some
+things as numbers when they really are not.  Hexadecimal literals are also
+replaced.  NULL is treated as a literal.  Numbers embedded in identifiers are
+also replaced, so tables named similarly will be fingerprinted to the same
+values (e.g. users_2009 and users_2010 will fingerprint identically).
+
+=item *
+
+Collapse all whitespace into a single space.
+
+=item *
+
+Lowercase the entire query.
+
+=item *
+
+Replace all literals inside of IN() and VALUES() lists with a single
+placeholder, regardless of cardinality.
+
+=item *
+
+Collapse multiple identical UNION queries into a single one.
+
+=back
+
+=head1 OPTIONS
+
+This tool accepts additional command-line arguments.  Refer to the
+L<"SYNOPSIS"> and usage information for details.
+
+=over
+
+=item --config
+
+type: Array
+
+Read this comma-separated list of config files; if specified, this must be the
+first option on the command line.
+
+=item --help
+
+Show help and exit.
+
+=item --match-embedded-numbers
+
+Match numbers embedded in words and replace as single values.  This option
+causes the tool to be more careful about matching numbers so that words
+with numbers, like C<catch22> are matched and replaced as a single C<?>
+placeholder.  Otherwise the default number matching pattern will replace
+C<catch22> as C<catch?>.
+
+This is helpful if database or table names contain numbers.
+
+=item --match-md5-checksums
+
+Match MD5 checksums and replace as single values.  This option causes
+the tool to be more careful about matching numbers so that MD5 checksums
+like C<fbc5e685a5d3d45aa1d0347fdb7c4d35> are matched and replaced as a
+single C<?> placeholder.  Otherwise, the default number matching pattern will
+replace C<fbc5e685a5d3d45aa1d0347fdb7c4d35> as C<fbc?>.
+
+=item --query
+
+type: string
+
+The query to convert into a fingerprint.
+
+=item --version
+
+Show version and exit.
+
+=back
+
+=head1 ENVIRONMENT
+
+The environment variable C<PTDEBUG> enables verbose debugging output to STDERR.
+To enable debugging and capture all output to a file, run the tool like:
+
+   PTDEBUG=1 pt-fingerprint ... > FILE 2>&1
+
+Be careful: debugging output is voluminous and can generate several megabytes
+of output.
+
+=head1 SYSTEM REQUIREMENTS
+
+You need Perl, DBI, DBD::mysql, and some core packages that ought to be
+installed in any reasonably new version of Perl.
+
+=head1 BUGS
+
+For a list of known bugs, see L<http://www.percona.com/bugs/pt-fingerprint>.
+
+Please report bugs at L<https://bugs.launchpad.net/percona-toolkit>.
+Include the following information in your bug report:
+
+=over
+
+=item * Complete command-line used to run the tool
+
+=item * Tool L<"--version">
+
+=item * MySQL version of all servers involved
+
+=item * Output from the tool including STDERR
+
+=item * Input files (log/dump/config files, etc.)
+
+=back
+
+If possible, include debugging output by running the tool with C<PTDEBUG>;
+see L<"ENVIRONMENT">.
+
+=head1 DOWNLOADING
+
+Visit L<http://www.percona.com/software/percona-toolkit/> to download the
+latest release of Percona Toolkit.  Or, get the latest release from the
+command line:
+
+   wget percona.com/get/percona-toolkit.tar.gz
+
+   wget percona.com/get/percona-toolkit.rpm
+
+   wget percona.com/get/percona-toolkit.deb
+
+You can also get individual tools from the latest release:
+
+   wget percona.com/get/TOOL
+
+Replace C<TOOL> with the name of any tool.
+
+=head1 AUTHORS
+
+Baron Schwartz and Daniel Nichter
+
+=head1 ABOUT PERCONA TOOLKIT
+
+This tool is part of Percona Toolkit, a collection of advanced command-line
+tools developed by Percona for MySQL support and consulting.  Percona Toolkit
+was forked from two projects in June, 2011: Maatkit and Aspersa.  Those
+projects were created by Baron Schwartz and developed primarily by him and
+Daniel Nichter, both of whom are employed by Percona.  Visit
+L<http://www.percona.com/software/> for more software developed by Percona.
+
+=head1 COPYRIGHT, LICENSE, AND WARRANTY
+
+This program is copyright 2011-2012 Percona Inc.
+Feedback and improvements are welcome.
+
+THIS PROGRAM IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED
+WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.
+
+This program is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free Software
+Foundation, version 2; OR the Perl Artistic License.  On UNIX and similar
+systems, you can issue `man perlgpl' or `man perlartistic' to read these
+licenses.
+
+You should have received a copy of the GNU General Public License along with
+this program; if not, write to the Free Software Foundation, Inc., 59 Temple
+Place, Suite 330, Boston, MA  02111-1307  USA.
+
+=head1 VERSION
+
+pt-fingerprint 2.1.1
+
+=cut

=== modified file 'bin/pt-fk-error-logger'
--- bin/pt-fk-error-logger	2012-05-24 17:52:01 +0000
+++ bin/pt-fk-error-logger	2012-05-28 08:02:19 +0000
@@ -2465,6 +2465,6 @@
 
 =head1 VERSION
 
-pt-fk-error-logger 2.0.4
+pt-fk-error-logger 2.1.1
 
 =cut

=== modified file 'bin/pt-heartbeat'
--- bin/pt-heartbeat	2012-05-24 17:52:01 +0000
+++ bin/pt-heartbeat	2012-05-28 08:02:19 +0000
@@ -73,6 +73,9 @@
          dsn_table_dsn => $dsn_table_dsn,
       );
    }
+   elsif ( $method =~ m/none/i ) {
+      PTDEBUG && _d('Not getting to slaves');
+   }
    else {
       die "Invalid --recursion-method: $method.  Valid values are: "
         . "dsn=DSN, hosts, or processlist.\n";
@@ -87,6 +90,11 @@
    my $dp   = $args->{dsn_parser};
    my $dsn  = $args->{dsn};
 
+   if ( lc($args->{method} || '') eq 'none' ) {
+      PTDEBUG && _d('Not recursing to slaves');
+      return;
+   }
+
    my $dbh;
    eval {
       $dbh = $args->{dbh} || $dp->get_dbh(
@@ -4136,9 +4144,10 @@
 Possible methods are:
 
   METHOD       USES
-  ===========  ================
+  ===========  ==================
   processlist  SHOW PROCESSLIST
   hosts        SHOW SLAVE HOSTS
+  none         Do not find slaves
 
 The processlist method is preferred because SHOW SLAVE HOSTS is not reliable.
 However, the hosts method is required if the server uses a non-standard
@@ -4400,6 +4409,6 @@
 
 =head1 VERSION
 
-pt-heartbeat 2.0.4
+pt-heartbeat 2.1.1
 
 =cut

=== modified file 'bin/pt-index-usage'
--- bin/pt-index-usage	2012-05-24 17:52:01 +0000
+++ bin/pt-index-usage	2012-05-28 08:02:19 +0000
@@ -2674,19 +2674,58 @@
    return bless $self, $class;
 }
 
+sub get_create_table {
+   my ( $self, $dbh, $db, $tbl ) = @_;
+   die "I need a dbh parameter" unless $dbh;
+   die "I need a db parameter"  unless $db;
+   die "I need a tbl parameter" unless $tbl;
+   my $q = $self->{Quoter};
+
+   my $new_sql_mode
+      = '/*!40101 SET @OLD_SQL_MODE := @@SQL_MODE, '
+      . q{@@SQL_MODE := REPLACE(REPLACE(@@SQL_MODE, 'ANSI_QUOTES', ''), ',,', ','), }
+      . '@OLD_QUOTE := @@SQL_QUOTE_SHOW_CREATE, '
+      . '@@SQL_QUOTE_SHOW_CREATE := 1 */';
+
+   my $old_sql_mode = '/*!40101 SET @@SQL_MODE := @OLD_SQL_MODE, '
+                     . '@@SQL_QUOTE_SHOW_CREATE := @OLD_QUOTE */';
+
+   PTDEBUG && _d($new_sql_mode);
+   eval { $dbh->do($new_sql_mode); };
+   PTDEBUG && $EVAL_ERROR && _d($EVAL_ERROR);
+
+   my $use_sql = 'USE ' . $q->quote($db);
+   PTDEBUG && _d($dbh, $use_sql);
+   $dbh->do($use_sql);
+
+   my $show_sql = "SHOW CREATE TABLE " . $q->quote($db, $tbl);
+   PTDEBUG && _d($show_sql);
+   my $href;
+   eval { $href = $dbh->selectrow_hashref($show_sql); };
+   if ( $EVAL_ERROR ) {
+      PTDEBUG && _d($EVAL_ERROR);
+
+      PTDEBUG && _d($old_sql_mode);
+      $dbh->do($old_sql_mode);
+
+      return;
+   }
+
+   PTDEBUG && _d($old_sql_mode);
+   $dbh->do($old_sql_mode);
+
+   my ($key) = grep { m/create (?:table|view)/i } keys %$href;
+   if ( !$key ) {
+      die "Error: no 'Create Table' or 'Create View' in result set from "
+         . "$show_sql: " . Dumper($href);
+   }
+
+   return $href->{$key};
+}
+
 sub parse {
    my ( $self, $ddl, $opts ) = @_;
    return unless $ddl;
-   if ( ref $ddl eq 'ARRAY' ) {
-      if ( lc $ddl->[0] eq 'table' ) {
-         $ddl = $ddl->[1];
-      }
-      else {
-         return {
-            engine => 'VIEW',
-         };
-      }
-   }
 
    if ( $ddl !~ m/CREATE (?:TEMPORARY )?TABLE `/ ) {
       die "Cannot parse table definition; is ANSI quoting "
@@ -2993,41 +3032,31 @@
    return $ddl;
 }
 
-sub remove_secondary_indexes {
-   my ( $self, $ddl ) = @_;
-   my $sec_indexes_ddl;
-   my $tbl_struct = $self->parse($ddl);
-
-   if ( ($tbl_struct->{engine} || '') =~ m/InnoDB/i ) {
-      my $clustered_key = $tbl_struct->{clustered_key};
-      $clustered_key  ||= '';
-
-      my @sec_indexes   = map {
-         my $key_def = $_->{ddl};
-         $key_def =~ s/([\(\)])/\\$1/g;
-         $ddl =~ s/\s+$key_def//i;
-
-         my $key_ddl = "ADD $_->{ddl}";
-         $key_ddl   .= ',' unless $key_ddl =~ m/,$/;
-         $key_ddl;
-      }
-      grep { $_->{name} ne $clustered_key }
-      values %{$tbl_struct->{keys}};
-      PTDEBUG && _d('Secondary indexes:', Dumper(\@sec_indexes));
-
-      if ( @sec_indexes ) {
-         $sec_indexes_ddl = join(' ', @sec_indexes);
-         $sec_indexes_ddl =~ s/,$//;
-      }
-
-      $ddl =~ s/,(\n\) )/$1/s;
-   }
-   else {
-      PTDEBUG && _d('Not removing secondary indexes from',
-         $tbl_struct->{engine}, 'table');
-   }
-
-   return $ddl, $sec_indexes_ddl, $tbl_struct;
+sub get_table_status {
+   my ( $self, $dbh, $db, $like ) = @_;
+   my $q = $self->{Quoter};
+   my $sql = "SHOW TABLE STATUS FROM " . $q->quote($db);
+   my @params;
+   if ( $like ) {
+      $sql .= ' LIKE ?';
+      push @params, $like;
+   }
+   PTDEBUG && _d($sql, @params);
+   my $sth = $dbh->prepare($sql);
+   eval { $sth->execute(@params); };
+   if ($EVAL_ERROR) {
+      PTDEBUG && _d($EVAL_ERROR);
+      return;
+   }
+   my @tables = @{$sth->fetchall_arrayref({})};
+   @tables = map {
+      my %tbl; # Make a copy with lowercased keys
+      @tbl{ map { lc $_ } keys %$_ } = values %$_;
+      $tbl{engine} ||= $tbl{type} || $tbl{comment};
+      delete $tbl{type};
+      \%tbl;
+   } @tables;
+   return @tables;
 }
 
 sub _d {
@@ -3917,7 +3946,7 @@
 
 sub new {
    my ( $class, %args ) = @_;
-   my @required_args = qw(OptionParser Quoter);
+   my @required_args = qw(OptionParser TableParser Quoter);
    foreach my $arg ( @required_args ) {
       die "I need a $arg argument" unless $args{$arg};
    }
@@ -3926,8 +3955,19 @@
    die "I need either a dbh or file_itr argument"
       if (!$dbh && !$file_itr) || ($dbh && $file_itr);
 
+   my %resume;
+   if ( my $table = $args{resume} ) {
+      PTDEBUG && _d('Will resume from or after', $table);
+      my ($db, $tbl) = $args{Quoter}->split_unquote($table);
+      die "Resume table must be database-qualified: $table"
+         unless $db && $tbl;
+      $resume{db}  = $db;
+      $resume{tbl} = $tbl;
+   }
+
    my $self = {
       %args,
+      resume  => \%resume,
       filters => _make_filters(%args),
    };
 
@@ -3988,9 +4028,19 @@
    return \%filters;
 }
 
-sub next_schema_object {
+sub next {
    my ( $self ) = @_;
 
+   if ( !$self->{initialized} ) {
+      $self->{initialized} = 1;
+      if ( $self->{resume}->{tbl}
+           && !$self->table_is_allowed(@{$self->{resume}}{qw(db tbl)}) ) {
+         PTDEBUG && _d('Will resume after',
+            join('.', @{$self->{resume}}{qw(db tbl)}));
+         $self->{resume}->{after} = 1;
+      }
+   }
+
    my $schema_obj;
    if ( $self->{file_itr} ) {
       $schema_obj= $self->_iterate_files();
@@ -4000,19 +4050,13 @@
    }
 
    if ( $schema_obj ) {
-      if ( $schema_obj->{ddl} && $self->{TableParser} ) {
-         $schema_obj->{tbl_struct}
-            = $self->{TableParser}->parse($schema_obj->{ddl});
-      }
-
-      delete $schema_obj->{ddl} unless $self->{keep_ddl};
-
       if ( my $schema = $self->{Schema} ) {
          $schema->add_schema_object($schema_obj);
       }
+      PTDEBUG && _d('Next schema object:',
+         $schema_obj->{db}, $schema_obj->{tbl});
    }
 
-   PTDEBUG && _d('Next schema object:', $schema_obj->{db}, $schema_obj->{tbl});
    return $schema_obj;
 }
 
@@ -4038,7 +4082,8 @@
          my $db = $1; # XXX
          $db =~ s/^`//;  # strip leading `
          $db =~ s/`$//;  # and trailing `
-         if ( $self->database_is_allowed($db) ) {
+         if ( $self->database_is_allowed($db)
+              && $self->_resume_from_database($db) ) {
             $self->{db} = $db;
          }
       }
@@ -4051,21 +4096,22 @@
          my ($tbl) = $chunk =~ m/$tbl_name/;
          $tbl      =~ s/^\s*`//;
          $tbl      =~ s/`\s*$//;
-         if ( $self->table_is_allowed($self->{db}, $tbl) ) {
+         if ( $self->_resume_from_table($tbl)
+              && $self->table_is_allowed($self->{db}, $tbl) ) {
             my ($ddl) = $chunk =~ m/^(?:$open_comment)?(CREATE TABLE.+?;)$/ms;
             if ( !$ddl ) {
                warn "Failed to parse CREATE TABLE from\n" . $chunk;
                next CHUNK;
             }
             $ddl =~ s/ \*\/;\Z/;/;  # remove end of version comment
-
-            my ($engine) = $ddl =~ m/\).*?(?:ENGINE|TYPE)=(\w+)/;   
-
-            if ( !$engine || $self->engine_is_allowed($engine) ) {
+            my $tbl_struct = $self->{TableParser}->parse($ddl);
+            if ( $self->engine_is_allowed($tbl_struct->{engine}) ) {
                return {
-                  db  => $self->{db},
-                  tbl => $tbl,
-                  ddl => $ddl,
+                  db         => $self->{db},
+                  tbl        => $tbl,
+                  name       => $self->{Quoter}->quote($self->{db}, $tbl),
+                  ddl        => $ddl,
+                  tbl_struct => $tbl_struct,
                };
             }
          }
@@ -4082,6 +4128,7 @@
 sub _iterate_dbh {
    my ( $self ) = @_;
    my $q   = $self->{Quoter};
+   my $tp  = $self->{TableParser};
    my $dbh = $self->{dbh};
    PTDEBUG && _d('Getting next schema object from dbh', $dbh);
 
@@ -4095,7 +4142,9 @@
    }
 
    if ( !$self->{db} ) {
-      $self->{db} = shift @{$self->{dbs}};
+      do {
+         $self->{db} = shift @{$self->{dbs}};
+      } until $self->_resume_from_database($self->{db});
       PTDEBUG && _d('Next database:', $self->{db});
       return unless $self->{db};
    }
@@ -4108,8 +4157,9 @@
       }
       grep {
          my ($tbl, $type) = @$_;
-         $self->table_is_allowed($self->{db}, $tbl)
-            && (!$type || ($type ne 'VIEW'));
+         (!$type || ($type ne 'VIEW'))
+         && $self->_resume_from_table($tbl)
+         && $self->table_is_allowed($self->{db}, $tbl);
       }
       @{$dbh->selectall_arrayref($sql)};
       PTDEBUG && _d('Found', scalar @tbls, 'tables in database', $self->{db});
@@ -4117,27 +4167,15 @@
    }
 
    while ( my $tbl = shift @{$self->{tbls}} ) {
-      my $engine;
-      if ( $self->{filters}->{'engines'}
-           || $self->{filters}->{'ignore-engines'} ) {
-         my $sql = "SHOW TABLE STATUS FROM " . $q->quote($self->{db})
-                 . " LIKE \'$tbl\'";
-         PTDEBUG && _d($sql);
-         $engine = $dbh->selectrow_hashref($sql)->{engine};
-         PTDEBUG && _d($tbl, 'uses', $engine, 'engine');
-      }
-
-
-      if ( !$engine || $self->engine_is_allowed($engine) ) {
-         my $ddl;
-         if ( my $du = $self->{MySQLDump} ) {
-            $ddl = $du->get_create_table($dbh, $q, $self->{db}, $tbl)->[1];
-         }
-
+      my $ddl        = $tp->get_create_table($dbh, $self->{db}, $tbl);
+      my $tbl_struct = $tp->parse($ddl);
+      if ( $self->engine_is_allowed($tbl_struct->{engine}) ) {
          return {
-            db  => $self->{db},
-            tbl => $tbl,
-            ddl => $ddl,
+            db         => $self->{db},
+            tbl        => $tbl,
+            name       => $q->quote($self->{db}, $tbl),
+            ddl        => $ddl,
+            tbl_struct => $tbl_struct,
          };
       }
    }
@@ -4198,6 +4236,10 @@
 
    my $filter = $self->{filters};
 
+   if ( $db eq 'mysql' && ($tbl eq 'general_log' || $tbl eq 'slow_log') ) {
+      return 0;
+   }
+
    if ( $filter->{'ignore-tables'}->{$tbl}
         && ($filter->{'ignore-tables'}->{$tbl} eq '*'
             || $filter->{'ignore-tables'}->{$tbl} eq $db) ) {
@@ -4237,7 +4279,11 @@
 
 sub engine_is_allowed {
    my ( $self, $engine ) = @_;
-   die "I need an engine argument" unless $engine;
+
+   if ( !$engine ) {
+      PTDEBUG && _d('No engine specified; allowing the table');
+      return 1;
+   }
 
    $engine = lc $engine;
 
@@ -4257,6 +4303,40 @@
    return 1;
 }
 
+sub _resume_from_database {
+   my ($self, $db) = @_;
+
+   return 1 unless $self->{resume}->{db};
+
+   if ( $db eq $self->{resume}->{db} ) {
+      PTDEBUG && _d('At resume db', $db);
+      delete $self->{resume}->{db};
+      return 1;
+   }
+
+   return 0;
+}
+
+sub _resume_from_table {
+   my ($self, $tbl) = @_;
+
+   return 1 unless $self->{resume}->{tbl};
+
+   if ( $tbl eq $self->{resume}->{tbl} ) {
+      if ( !$self->{resume}->{after} ) {
+         PTDEBUG && _d('Resuming from table', $tbl);
+         delete $self->{resume}->{tbl};
+         return 1;
+      }
+      else {
+         PTDEBUG && _d('Resuming after table', $tbl);
+         delete $self->{resume}->{tbl};
+      }
+   }
+
+   return 0;
+}
+
 sub _d {
    my ($package, undef, $line) = caller 0;
    @_ = map { (my $temp = $_) =~ s/\n/\n# /g; $temp; }
@@ -5149,13 +5229,11 @@
       dbh          => $dbh,
       OptionParser => $o,
       Quoter       => $q,
-      MySQLDump    => $du,
       TableParser  => $tp,
       Schema       => $schema,
-      keep_ddl     => 1,
    );
    TALBE:
-   while ( my $tbl = $schema_itr->next_schema_object() ) {
+   while ( my $tbl = $schema_itr->next() ) {
       eval {
          my ($indexes) = $tp->get_keys($tbl->{ddl}, {version => $version});
          $iu->add_indexes(%$tbl, indexes=>$indexes);
@@ -6191,6 +6269,6 @@
 
 =head1 VERSION
 
-pt-index-usage 2.0.4
+pt-index-usage 2.1.1
 
 =cut

=== modified file 'bin/pt-ioprofile'
--- bin/pt-ioprofile	2012-05-16 17:31:10 +0000
+++ bin/pt-ioprofile	2012-05-28 08:02:19 +0000
@@ -1037,7 +1037,7 @@
 
 =head1 VERSION
 
-pt-ioprofile 2.0.4
+pt-ioprofile 2.1.1
 
 =cut
 

=== modified file 'bin/pt-kill'
--- bin/pt-kill	2012-05-24 17:52:01 +0000
+++ bin/pt-kill	2012-05-28 08:02:19 +0000
@@ -4860,6 +4860,6 @@
 
 =head1 VERSION
 
-pt-kill 2.0.4
+pt-kill 2.1.1
 
 =cut

=== modified file 'bin/pt-log-player'
--- bin/pt-log-player	2012-05-24 17:52:01 +0000
+++ bin/pt-log-player	2012-05-28 08:02:19 +0000
@@ -3609,6 +3609,6 @@
 
 =head1 VERSION
 
-pt-log-player 2.0.4
+pt-log-player 2.1.1
 
 =cut

=== modified file 'bin/pt-mext'
--- bin/pt-mext	2012-05-16 17:31:10 +0000
+++ bin/pt-mext	2012-05-28 08:02:19 +0000
@@ -279,7 +279,7 @@
 
 =head1 VERSION
 
-pt-mext 2.0.4
+pt-mext 2.1.1
 
 =cut
 

=== modified file 'bin/pt-mysql-summary'
--- bin/pt-mysql-summary	2012-05-16 17:31:10 +0000
+++ bin/pt-mysql-summary	2012-05-28 08:02:19 +0000
@@ -4,14 +4,421 @@
 # See "COPYRIGHT, LICENSE, AND WARRANTY" at the end of this file for legal
 # notices and disclaimers.
 
+set -u
+
+# ###########################################################################
+# log_warn_die package
+# This package is a copy without comments from the original.  The original
+# with comments and its test file can be found in the Bazaar repository at,
+#   lib/bash/log_warn_die.sh
+#   t/lib/bash/log_warn_die.sh
+# See https://launchpad.net/percona-toolkit for more information.
+# ###########################################################################
+
+
+set -u
+
+PTFUNCNAME=""
+PTDEBUG="${PTDEBUG:-""}"
+EXIT_STATUS=0
+
+log() {
+   TS=$(date +%F-%T | tr :- _);
+   echo "$TS $*"
+}
+
+warn() {
+   log "$*" >&2
+   EXIT_STATUS=1
+}
+
+die() {
+   warn "$*"
+   exit 1
+}
+
+_d () {
+   [ "$PTDEBUG" ] && echo "# $PTFUNCNAME: $(log "$*")" >&2
+}
+
+# ###########################################################################
+# End log_warn_die package
+# ###########################################################################
+
+# ###########################################################################
+# parse_options package
+# This package is a copy without comments from the original.  The original
+# with comments and its test file can be found in the Bazaar repository at,
+#   lib/bash/parse_options.sh
+#   t/lib/bash/parse_options.sh
+# See https://launchpad.net/percona-toolkit for more information.
+# ###########################################################################
+
+
+
+
+
+set -u
+
+ARGV=""           # Non-option args (probably input files)
+EXT_ARGV=""       # Everything after -- (args for an external command)
+HAVE_EXT_ARGV=""  # Got --, everything else is put into EXT_ARGV
+OPT_ERRS=0        # How many command line option errors
+OPT_VERSION=""    # If --version was specified
+OPT_HELP=""       # If --help was specified
+PO_DIR=""         # Directory with program option spec files
+
 usage() {
-   if [ "${OPT_ERR}" ]; then
-      echo "${OPT_ERR}" >&2
-   fi
-   echo "Usage: pt-mysql-summary [MYSQL-OPTIONS]" >&2
-   echo "For more information, 'man pt-mysql-summary' or 'perldoc $0'" >&2
-   exit 1
-}
+   local file="$1"
+
+   local usage=$(grep '^Usage: ' "$file")
+   echo $usage
+   echo
+   echo "For more information, 'man $TOOL' or 'perldoc $file'."
+}
+
+usage_or_errors() {
+   local file="$1"
+
+   if [ "$OPT_VERSION" ]; then
+      local version=$(grep '^pt-[^ ]\+ [0-9]' "$file")
+      echo "$version"
+      return 1
+   fi
+
+   if [ "$OPT_HELP" ]; then
+      usage "$file"
+      echo
+      echo "Command line options:"
+      echo
+      perl -e '
+         use strict;
+         use warnings FATAL => qw(all);
+         my $lcol = 20;         # Allow this much space for option names.
+         my $rcol = 80 - $lcol; # The terminal is assumed to be 80 chars wide.
+         my $name;
+         while ( <> ) {
+            my $line = $_;
+            chomp $line;
+            if ( $line =~ s/^long:/  --/ ) {
+               $name = $line;
+            }
+            elsif ( $line =~ s/^desc:// ) {
+               $line =~ s/ +$//mg;
+               my @lines = grep { $_      }
+                           $line =~ m/(.{0,$rcol})(?:\s+|\Z)/g;
+               if ( length($name) >= $lcol ) {
+                  print $name, "\n", (q{ } x $lcol);
+               }
+               else {
+                  printf "%-${lcol}s", $name;
+               }
+               print join("\n" . (q{ } x $lcol), @lines);
+               print "\n";
+            }
+         }
+      ' "$PO_DIR"/*
+      echo
+      echo "Options and values after processing arguments:"
+      echo
+      for opt in $(ls "$PO_DIR"); do
+         local varname="OPT_$(echo "$opt" | tr a-z- A-Z_)"
+         local varvalue="${!varname}"
+         printf -- "  --%-30s %s" "$opt" "${varvalue:-(No value)}"
+         echo
+      done
+      return 1
+   fi
+
+   if [ $OPT_ERRS -gt 0 ]; then
+      echo
+      usage "$file"
+      return 1
+   fi
+
+   return 0
+}
+
+option_error() {
+   local err="$1"
+   OPT_ERRS=$(($OPT_ERRS + 1))
+   echo "$err" >&2
+}
+
+parse_options() {
+   local file="$1"
+   shift
+
+   ARGV=""
+   EXT_ARGV=""
+   HAVE_EXT_ARGV=""
+   OPT_ERRS=0
+   OPT_VERSION=""
+   OPT_HELP=""
+   PO_DIR="$TMPDIR/po"
+
+   if [ ! -d "$PO_DIR" ]; then
+      mkdir "$PO_DIR"
+      if [ $? -ne 0 ]; then
+         echo "Cannot mkdir $PO_DIR" >&2
+         exit 1
+      fi
+   fi
+
+   rm -rf "$PO_DIR"/*
+   if [ $? -ne 0 ]; then
+      echo "Cannot rm -rf $PO_DIR/*" >&2
+      exit 1
+   fi
+
+   _parse_pod "$file"  # Parse POD into program option (po) spec files
+   _eval_po            # Eval po into existence with default values
+
+   if [ $# -ge 2 ] &&  [ "$1" = "--config" ]; then
+      shift  # --config
+      local user_config_files="$1"
+      shift  # that ^
+      local IFS=","
+      for user_config_file in $user_config_files; do
+         _parse_config_files "$user_config_file"
+      done
+   else
+      _parse_config_files "/etc/percona-toolkit/percona-toolkit.conf" "/etc/percona-toolkit/$TOOL.conf" "$HOME/.percona-toolkit.conf" "$HOME/.$TOOL.conf"
+   fi
+
+   _parse_command_line "$@"
+}
+
+_parse_pod() {
+   local file="$1"
+
+   cat "$file" | PO_DIR="$PO_DIR" perl -ne '
+      BEGIN { $/ = ""; }
+      next unless $_ =~ m/^=head1 OPTIONS/;
+      while ( defined(my $para = <>) ) {
+         last if $para =~ m/^=head1/;
+         chomp;
+         if ( $para =~ m/^=item --(\S+)/ ) {
+            my $opt  = $1;
+            my $file = "$ENV{PO_DIR}/$opt";
+            open my $opt_fh, ">", $file or die "Cannot open $file: $!";
+            print $opt_fh "long:$opt\n";
+            $para = <>;
+            chomp;
+            if ( $para =~ m/^[a-z ]+:/ ) {
+               map {
+                  chomp;
+                  my ($attrib, $val) = split(/: /, $_);
+                  print $opt_fh "$attrib:$val\n";
+               } split(/; /, $para);
+               $para = <>;
+               chomp;
+            }
+            my ($desc) = $para =~ m/^([^?.]+)/;
+            print $opt_fh "desc:$desc.\n";
+            close $opt_fh;
+         }
+      }
+      last;
+   '
+}
+
+_eval_po() {
+   local IFS=":"
+   for opt_spec in "$PO_DIR"/*; do
+      local opt=""
+      local default_val=""
+      local neg=0
+      local size=0
+      while read key val; do
+         case "$key" in
+            long)
+               opt=$(echo $val | sed 's/-/_/g' | tr [:lower:] [:upper:])
+               ;;
+            default)
+               default_val="$val"
+               ;;
+            "short form")
+               ;;
+            type)
+               [ "$val" = "size" ] && size=1
+               ;;
+            desc)
+               ;;
+            negatable)
+               if [ "$val" = "yes" ]; then
+                  neg=1
+               fi
+               ;;
+            *)
+               echo "Invalid attribute in $opt_spec: $line" >&2
+               exit 1
+         esac 
+      done < "$opt_spec"
+
+      if [ -z "$opt" ]; then
+         echo "No long attribute in option spec $opt_spec" >&2
+         exit 1
+      fi
+
+      if [ $neg -eq 1 ]; then
+         if [ -z "$default_val" ] || [ "$default_val" != "yes" ]; then
+            echo "Option $opt_spec is negatable but not default: yes" >&2
+            exit 1
+         fi
+      fi
+
+      if [ $size -eq 1 -a -n "$default_val" ]; then
+         default_val=$(size_to_bytes $default_val)
+      fi
+
+      eval "OPT_${opt}"="$default_val"
+   done
+}
+
+_parse_config_files() {
+
+   for config_file in "$@"; do
+      test -f "$config_file" || continue
+
+      while read config_opt; do
+
+         echo "$config_opt" | grep '^[ ]*[^#]' >/dev/null 2>&1 || continue
+
+         config_opt="$(echo "$config_opt" | sed -e 's/^ *//g' -e 's/ *$//g' -e 's/[ ]*=[ ]*/=/' -e 's/[ ]*#.*$//')"
+
+         [ "$config_opt" = "" ] && continue
+
+         if ! [ "$HAVE_EXT_ARGV" ]; then
+            config_opt="--$config_opt"
+         fi
+
+         _parse_command_line "$config_opt"
+
+      done < "$config_file"
+
+      HAVE_EXT_ARGV=""  # reset for each file
+
+   done
+}
+
+_parse_command_line() {
+   local opt=""
+   local val=""
+   local next_opt_is_val=""
+   local opt_is_ok=""
+   local opt_is_negated=""
+   local real_opt=""
+   local required_arg=""
+   local spec=""
+
+   for opt in "$@"; do
+      if [ "$opt" = "--" -o "$opt" = "----" ]; then
+         HAVE_EXT_ARGV=1
+         continue
+      fi
+      if [ "$HAVE_EXT_ARGV" ]; then
+         if [ "$EXT_ARGV" ]; then
+            EXT_ARGV="$EXT_ARGV $opt"
+         else
+            EXT_ARGV="$opt"
+         fi
+         continue
+      fi
+
+      if [ "$next_opt_is_val" ]; then
+         next_opt_is_val=""
+         if [ $# -eq 0 ] || [ $(expr "$opt" : "-") -eq 1 ]; then
+            option_error "$real_opt requires a $required_arg argument"
+            continue
+         fi
+         val="$opt"
+         opt_is_ok=1
+      else
+         if [ $(expr "$opt" : "-") -eq 0 ]; then
+            if [ -z "$ARGV" ]; then
+               ARGV="$opt"
+            else
+               ARGV="$ARGV $opt"
+            fi
+            continue
+         fi
+
+         real_opt="$opt"
+
+         if $(echo $opt | grep '^--no-' >/dev/null); then
+            opt_is_negated=1
+            opt=$(echo $opt | sed 's/^--no-//')
+         else
+            opt_is_negated=""
+            opt=$(echo $opt | sed 's/^-*//')
+         fi
+
+         if $(echo $opt | grep '^[a-z-][a-z-]*=' >/dev/null 2>&1); then
+            val="$(echo $opt | awk -F= '{print $2}')"
+            opt="$(echo $opt | awk -F= '{print $1}')"
+         fi
+
+         if [ -f "$TMPDIR/po/$opt" ]; then
+            spec="$TMPDIR/po/$opt"
+         else
+            spec=$(grep "^short form:-$opt\$" "$TMPDIR"/po/* | cut -d ':' -f 1)
+            if [ -z "$spec"  ]; then
+               option_error "Unknown option: $real_opt"
+               continue
+            fi
+         fi
+
+         required_arg=$(cat "$spec" | awk -F: '/^type:/{print $2}')
+         if [ "$required_arg" ]; then
+            if [ "$val" ]; then
+               opt_is_ok=1
+            else
+               next_opt_is_val=1
+            fi
+         else
+            if [ "$val" ]; then
+               option_error "Option $real_opt does not take a value"
+               continue
+            fi 
+            if [ "$opt_is_negated" ]; then
+               val=""
+            else
+               val="yes"
+            fi
+            opt_is_ok=1
+         fi
+      fi
+
+      if [ "$opt_is_ok" ]; then
+         opt=$(cat "$spec" | grep '^long:' | cut -d':' -f2 | sed 's/-/_/g' | tr [:lower:] [:upper:])
+
+         if grep "^type:size" "$spec" >/dev/null; then
+            val=$(size_to_bytes $val)
+         fi
+
+         eval "OPT_$opt"="'$val'"
+
+         opt=""
+         val=""
+         next_opt_is_val=""
+         opt_is_ok=""
+         opt_is_negated=""
+         real_opt=""
+         required_arg=""
+         spec=""
+      fi
+   done
+}
+
+size_to_bytes() {
+   local size="$1"
+   echo $size | perl -ne '%f=(B=>1, K=>1_024, M=>1_048_576, G=>1_073_741_824, T=>1_099_511_627_776); m/^(\d+)([kMGT])?/i; print $1 * $f{uc($2 || "B")};'
+}
+
+# ###########################################################################
+# End parse_options package
+# ###########################################################################
 
 # ###########################################################################
 # tmpdir package
@@ -22,6 +429,9 @@
 # See https://launchpad.net/percona-toolkit for more information.
 # ###########################################################################
 
+
+set -u
+
 TMPDIR=""
 
 mk_tmpdir() {
@@ -51,27 +461,68 @@
 # End tmpdir package
 # ###########################################################################
 
-# ########################################################################
-# Some global setup is necessary for cross-platform compatibility, even
-# when sourcing this script for testing purposes.
-# ########################################################################
-AP_AWK="$(which awk)"
-which gawk >/dev/null 2>&1 && AP_AWK="$(which gawk)"
-AP_SED="$(which sed)"
-which gsed >/dev/null 2>&1 && AP_SED="$(which gsed)"
-AP_GREP="$(which grep)"
-which ggrep >/dev/null 2>&1 && AP_GREP="$(which ggrep)"
-
-# ########################################################################
-# Globals, helper functions
-# ########################################################################
-
-# The awk code for fuzzy rounding.  (It's used in a few places, so makes sense
-# not to duplicate).  It fuzzy-rounds the variable named fuzzy_var.  It goes in
-# steps of 5, 10, 25, then repeats by a factor of 10 larger (50, 100, 250), and
-# so on, until it finds a number that's large enough.  The pattern is slightly
-# broken between the initial 1 and 50, because rounding to the nearest 2.5
-# doesn't seem right to me.
+# ###########################################################################
+# alt_cmds package
+# This package is a copy without comments from the original.  The original
+# with comments and its test file can be found in the Bazaar repository at,
+#   lib/bash/alt_cmds.sh
+#   t/lib/bash/alt_cmds.sh
+# See https://launchpad.net/percona-toolkit for more information.
+# ###########################################################################
+
+
+set -u
+
+_seq() {
+   local i="$1"
+   awk "BEGIN { for(i=1; i<=$i; i++) print i; }"
+}
+
+_pidof() {
+   local cmd="$1"
+   if ! pidof "$cmd" 2>/dev/null; then
+      ps -eo pid,ucomm | awk -v comm="$cmd" '$2 == comm { print $1 }'
+   fi
+}
+
+_lsof() {
+   local pid="$1"
+   if ! lsof -p $pid 2>/dev/null; then
+      /bin/ls -l /proc/$pid/fd 2>/dev/null
+   fi
+}
+
+
+
+_which() {
+   if [ -x /usr/bin/which ]; then
+      /usr/bin/which "$1" 2>/dev/null | awk '{print $1}'
+   elif which which 1>/dev/null 2>&1; then
+      which "$1" 2>/dev/null | awk '{print $1}'
+   else
+      echo "$1"
+   fi
+}
+
+# ###########################################################################
+# End alt_cmds package
+# ###########################################################################
+
+# ###########################################################################
+# report_formatting package
+# This package is a copy without comments from the original.  The original
+# with comments and its test file can be found in the Bazaar repository at,
+#   lib/bash/report_formatting.sh
+#   t/lib/bash/report_formatting.sh
+# See https://launchpad.net/percona-toolkit for more information.
+# ###########################################################################
+
+
+set -u
+
+POSIXLY_CORRECT=1
+export POSIXLY_CORRECT
+
 fuzzy_formula='
    rounded = 0;
    if (fuzzy_var <= 10 ) {
@@ -94,108 +545,458 @@
       factor = factor * 10;
    }'
 
-# The temp files are for storing working results so we don't call commands many
-# times (gives inconsistent results, maybe adds load on things I don't want to
-# such as RAID controllers).  They must not exist -- if they did, someone would
-# symlink them to /etc/passwd and then run this program as root.  Call this
-# function with "rm" or "touch" as an argument.
-temp_files() {
-   for file in $TMPDIR/percona-toolkit{,-mysql-variables,-mysql-status,-innodb-status} \
-         $TMPDIR/percona-toolkit{2,-mysql-databases,-mysql-processlist,-noncounters} \
-         $TMPDIR/percona-toolkit-mysql{dump,-slave};
-   do
-      case "$1" in
-      touch)
-         if ! touch "${file}"; then
-            echo "I can't make my temp file ${file}";
-            exit 1;
-         fi
-         ;;
-      rm)
-         rm -f "${file}"
-         ;;
-      esac
-   done
-}
-
-# Print a space-padded string into $line.  Then translate spaces to hashes, and
-# underscores to spaces.  End result is a line of hashes with words at the
-# start.
+fuzz () {
+   awk -v fuzzy_var="$1" "BEGIN { ${fuzzy_formula} print fuzzy_var;}"
+}
+
+fuzzy_pct () {
+   local pct="$(awk -v one="$1" -v two="$2" 'BEGIN{ if (two > 0) { printf "%d", one/two*100; } else {print 0} }')";
+   echo "$(fuzz "${pct}")%"
+}
+
 section () {
-   line="$(printf '#_%-60s' "$1_")"
-   line="${line// /#}"
-   printf "%s\n" "${line//_/ }"
-}
-
-# Print a "name | value" line.
-name_val() {
-   printf "%20s | %s\n" "$1" "$2"
-}
-
-# Converts a value to units of power of 2.  Optional precision is $2.
+   local str="$1"
+   awk -v var="${str} _" 'BEGIN {
+      line = sprintf("# %-60s", var);
+      i = index(line, "_");
+      x = substr(line, i);
+      gsub(/[_ \t]/, "#", x);
+      printf("%s%s\n", substr(line, 1, i-1), x);
+   }'
+}
+
+NAME_VAL_LEN=12
+name_val () {
+   printf "%+*s | %s\n" "${NAME_VAL_LEN}" "$1" "$2"
+}
+
 shorten() {
-   unit=k
-   size=1024
-   if [ $1 -ge 1099511627776 ] ; then
-      size=1099511627776
-      unit=T
-   elif [ $1 -ge 1073741824 ] ; then
-      size=1073741824
-      unit=G
-   elif [ $1 -ge 1048576 ] ; then
-      size=1048576
-      unit=M
-   fi
-   result=$(echo "$1 $size ${2:-0}" | $AP_AWK '{printf "%." $3 "f", $1 / $2}')
-   echo "${result}${unit}"
+   local num="$1"
+   local prec="${2:-2}"
+   local div="${3:-1024}"
+
+   echo "$num" | awk -v prec="$prec" -v div="$div" '
+      {
+         num  = $1;
+         unit = num >= 1125899906842624 ? "P" \
+              : num >= 1099511627776    ? "T" \
+              : num >= 1073741824       ? "G" \
+              : num >= 1048576          ? "M" \
+              : num >= 1024             ? "k" \
+              :                           "";
+         while ( num >= div ) {
+            num /= div;
+         }
+         printf "%.*f%s", prec, num, unit;
+      }
+   '
 }
 
-# Collapse a file into an aggregated list; file must be created with 'sort |
-# uniq -c'.  This function is copy-pasted from 'summary' so see there for full
-# docs and tests.
-# ##############################################################################
 group_concat () {
-   sed -e '{H; $!d}' -e 'x' -e 's/\n[[:space:]]*\([[:digit:]]*\)[[:space:]]*/, \1x/g' -e 's/[[:space:]][[:space:]]*/ /g' -e 's/, //' ${1}
-}
-
-# Accepts a number of seconds, and outputs a d+h:m:s formatted string
+   sed -e '{H; $!d;}' -e 'x' -e 's/\n[[:space:]]*\([[:digit:]]*\)[[:space:]]*/, \1x/g' -e 's/[[:space:]][[:space:]]*/ /g' -e 's/, //' "${1}"
+}
+
+# ###########################################################################
+# End report_formatting package
+# ###########################################################################
+
+# ###########################################################################
+# summary_common package
+# This package is a copy without comments from the original.  The original
+# with comments and its test file can be found in the Bazaar repository at,
+#   lib/bash/summary_common.sh
+#   t/lib/bash/summary_common.sh
+# See https://launchpad.net/percona-toolkit for more information.
+# ###########################################################################
+
+
+set -u
+
+CMD_FILE="$( _which file 2>/dev/null )"
+CMD_NM="$( _which nm 2>/dev/null )"
+CMD_OBJDUMP="$( _which objdump 2>/dev/null )"
+
+get_nice_of_pid () {
+   local pid="$1"
+   local niceness="$(ps -p $pid -o nice | awk '$1 !~ /[^0-9]/ {print $1; exit}')"
+
+   if [ -n "${niceness}" ]; then
+      echo $niceness
+   else
+      local tmpfile="$TMPDIR/nice_through_c.tmp.c"
+      _d "Getting the niceness from ps failed, somehow. We are about to try this:"
+      cat <<EOC > "$tmpfile"
+
+int main(void) {
+   int priority = getpriority(PRIO_PROCESS, $pid);
+   if ( priority == -1 && errno == ESRCH ) {
+      return 1;
+   }
+   else {
+      printf("%d\\n", priority);
+      return 0;
+   }
+}
+
+EOC
+      local c_comp=$(_which gcc)
+      if [ -z "${c_comp}" ]; then
+         c_comp=$(_which cc)
+      fi
+      _d "$tmpfile: $( cat "$tmpfile" )"
+      _d "$c_comp -xc \"$tmpfile\" -o \"$tmpfile\" && eval \"$tmpfile\""
+      $c_comp -xc "$tmpfile" -o "$tmpfile" 2>/dev/null && eval "$tmpfile" 2>/dev/null
+      if [ $? -ne 0 ]; then
+         echo "?"
+         _d "Failed to get a niceness value for $pid"
+      fi
+   fi
+}
+
+get_oom_of_pid () {
+   local pid="$1"
+   local oom_adj=""
+
+   if [ -n "${pid}" -a -e /proc/cpuinfo ]; then
+      if [ -s "/proc/$pid/oom_score_adj" ]; then
+         oom_adj=$(cat "/proc/$pid/oom_score_adj" 2>/dev/null)
+         _d "For $pid, the oom value is $oom_adj, retreived from oom_score_adj"
+      else
+         oom_adj=$(cat "/proc/$pid/oom_adj" 2>/dev/null)
+         _d "For $pid, the oom value is $oom_adj, retreived from oom_adj"
+      fi
+   fi
+
+   if [ -n "${oom_adj}" ]; then
+      echo "${oom_adj}"
+   else
+      echo "?"
+      _d "Can't find the oom value for $pid"
+   fi
+}
+
+has_symbols () {
+   local executable="$(_which "$1")"
+   local has_symbols=""
+
+   if    [ "${CMD_FILE}" ] \
+      && [ "$($CMD_FILE "${executable}" | grep 'not stripped' )" ]; then
+      has_symbols=1
+   elif    [ "${CMD_NM}" ] \
+        || [ "${CMD_OBJDMP}" ]; then
+      if    [ "${CMD_NM}" ] \
+         && [ !"$("${CMD_NM}" -- "${executable}" 2>&1 | grep 'File format not recognized' )" ]; then
+         if [ -z "$( $CMD_NM -- "${executable}" 2>&1 | grep ': no symbols' )" ]; then
+            has_symbols=1
+         fi
+      elif [ -z "$("${CMD_OBJDUMP}" -t -- "${executable}" | grep '^no symbols$' )" ]; then
+         has_symbols=1
+      fi
+   fi
+
+   if [ "${has_symbols}" ]; then
+      echo "Yes"
+   else
+      echo "No"
+   fi
+}
+
+setup_data_dir () {
+   local existing_dir="$1"
+   local data_dir=""
+   if [ -z "$existing_dir" ]; then
+      mkdir "$TMPDIR/data" || die "Cannot mkdir $TMPDIR/data"
+      data_dir="$TMPDIR/data"
+   else
+      if [ ! -d "$existing_dir" ]; then
+         mkdir "$existing_dir" || die "Cannot mkdir $existing_dir"
+      elif [ "$( ls -A "$existing_dir" )" ]; then
+         die "--save-samples directory isn't empty, halting."
+      fi
+      touch "$existing_dir/test" || die "Cannot write to $existing_dir"
+      rm "$existing_dir/test"    || die "Cannot rm $existing_dir/test"
+      data_dir="$existing_dir"
+   fi
+   echo "$data_dir"
+}
+
+get_var () {
+   local varname="$1"
+   local file="$2"
+   awk -v pattern="${varname}" '$1 == pattern { if (length($2)) { len = length($1); print substr($0, len+index(substr($0, len+1), $2)) } }' "${file}"
+}
+
+# ###########################################################################
+# End summary_common package
+# ###########################################################################
+
+# ###########################################################################
+# collect_mysql_info package
+# This package is a copy without comments from the original.  The original
+# with comments and its test file can be found in the Bazaar repository at,
+#   lib/bash/collect_mysql_info.sh
+#   t/lib/bash/collect_mysql_info.sh
+# See https://launchpad.net/percona-toolkit for more information.
+# ###########################################################################
+
+
+
+CMD_MYSQL="${CMD_MYSQL:-""}"
+CMD_MYSQLDUMP="${CMD_MYSQLDUMP:-""}"
+
+collect_mysqld_instances () {
+   local variables_file="$1"
+
+   local pids="$(_pidof mysqld)"
+
+   if [ -n "$pids" ]; then
+
+      for pid in $pids; do
+         local nice="$( get_nice_of_pid $pid )"
+         local oom="$( get_oom_of_pid $pid )"
+         echo "internal::nice_of_$pid    $nice" >> "$variables_file"
+         echo "internal::oom_of_$pid    $oom" >> "$variables_file"
+      done
+
+      pids="$(echo $pids | sed -e 's/ /,/g')"
+      ps ww -p "$pids" 2>/dev/null
+   else
+      echo "mysqld doesn't appear to be running"
+   fi
+
+}
+
+find_my_cnf_file() {
+   local file="$1"
+   local port="${2:-""}"
+
+   local cnf_file=""
+   if test -n "$port" && grep -- "/mysqld.*--port=$port" "${file}" >/dev/null 2>&1 ; then
+      cnf_file="$(grep -- "/mysqld.*--port=$port" "${file}" \
+         | awk 'BEGIN{RS=" "; FS="=";} $1 ~ /--defaults-file/ { print $2; }' \
+         | head -n1)"
+   else
+      cnf_file="$(grep '/mysqld' "${file}" \
+         | awk 'BEGIN{RS=" "; FS="=";} $1 ~ /--defaults-file/ { print $2; }' \
+         | head -n1)"
+   fi
+
+   if [ ! -n "${cnf_file}" ]; then
+      cnf_file="/etc/my.cnf";
+      if [ ! -e "${cnf_file}" ]; then
+         cnf_file="/etc/mysql/my.cnf";
+      fi
+      if [ ! -e "${cnf_file}" ]; then
+         cnf_file="/var/db/mysql/my.cnf";
+      fi
+   fi
+
+   echo "$cnf_file"
+}
+
+collect_mysql_variables () {
+   $CMD_MYSQL $EXT_ARGV -ss  -e 'SHOW /*!40100 GLOBAL*/ VARIABLES'
+}
+
+collect_mysql_status () {
+   $CMD_MYSQL $EXT_ARGV -ss -e 'SHOW /*!50000 GLOBAL*/ STATUS'
+}
+
+collect_mysql_databases () {
+   $CMD_MYSQL $EXT_ARGV -ss -e 'SHOW DATABASES' 2>/dev/null
+}
+
+collect_mysql_plugins () {
+   $CMD_MYSQL $EXT_ARGV -ss -e 'SHOW PLUGINS' 2>/dev/null
+}
+
+collect_mysql_slave_status () {
+   $CMD_MYSQL $EXT_ARGV -ssE -e 'SHOW SLAVE STATUS' 2>/dev/null
+}
+
+collect_mysql_innodb_status () {
+   $CMD_MYSQL $EXT_ARGV -ssE -e 'SHOW /*!50000 ENGINE*/ INNODB STATUS' 2>/dev/null
+}
+
+collect_mysql_processlist () {
+   $CMD_MYSQL $EXT_ARGV -ssE -e 'SHOW FULL PROCESSLIST' 2>/dev/null
+}
+
+collect_mysql_users () {
+   $CMD_MYSQL $EXT_ARGV -ss -e 'SELECT COUNT(*), SUM(user=""), SUM(password=""), SUM(password NOT LIKE "*%") FROM mysql.user' 2>/dev/null
+}
+
+collect_master_logs_status () {
+   local master_logs_file="$1"
+   local master_status_file="$2"
+   $CMD_MYSQL $EXT_ARGV -ss -e 'SHOW MASTER LOGS' > "$master_logs_file" 2>/dev/null
+   $CMD_MYSQL $EXT_ARGV -ss -e 'SHOW MASTER STATUS' > "$master_status_file" 2>/dev/null
+}
+
+collect_mysql_deferred_status () {
+   local status_file="$1"
+   collect_mysql_status > "$TMPDIR/defer_gatherer"
+   join "$status_file" "$TMPDIR/defer_gatherer"
+}
+
+collect_internal_vars () {
+   local mysqld_executables="${1:-""}"
+
+   local FNV_64=""
+   if $CMD_MYSQL $EXT_ARGV -e 'SELECT FNV_64("a")' >/dev/null 2>&1; then
+      FNV_64="Enabled";
+   else
+      FNV_64="Unknown";
+   fi
+
+   local now="$($CMD_MYSQL $EXT_ARGV -ss -e 'SELECT NOW()')"
+   local user="$($CMD_MYSQL $EXT_ARGV -ss -e 'SELECT CURRENT_USER()')"
+   local trigger_count=$($CMD_MYSQL $EXT_ARGV -ss -e "SELECT COUNT(*) FROM INFORMATION_SCHEMA.TRIGGERS" 2>/dev/null)
+
+   echo "pt-summary-internal-mysql_executable    $CMD_MYSQL"
+   echo "pt-summary-internal-now    $now"
+   echo "pt-summary-internal-user   $user"
+   echo "pt-summary-internal-FNV_64   $FNV_64"
+   echo "pt-summary-internal-trigger_count   $trigger_count"
+
+   if [ -e "$mysqld_executables" ]; then
+      local i=1
+      while read executable; do
+         echo "pt-summary-internal-mysqld_executable_${i}   $(has_symbols "$executable")"
+         i=$(($i + 1))
+      done < "$mysqld_executables"
+   fi
+}
+
+get_mysqldump_for () {
+   local args="$1"
+   local dbtodump="${2:-"--all-databases"}"
+
+   $CMD_MYSQLDUMP $EXT_ARGV --no-data --skip-comments \
+      --skip-add-locks --skip-add-drop-table --compact \
+      --skip-lock-all-tables --skip-lock-tables --skip-set-charset \
+      ${args} --databases $( local IFS=,; echo ${dbtodump})
+}
+
+get_mysqldump_args () {
+   local file="$1"
+   local trg_arg=""
+
+   if $CMD_MYSQLDUMP --help --verbose 2>&1 | grep triggers >/dev/null; then
+      trg_arg="--routines"
+   fi
+
+   if [ "${trg_arg}" ]; then
+      local triggers="--skip-triggers"
+      local trg=$(get_var "pt-summary-internal-trigger_count" "$file" )
+      if [ -n "${trg}" ] && [ "${trg}" -gt 0 ]; then
+         triggers="--triggers"
+      fi
+      trg_arg="${trg_arg} ${triggers}";
+   fi
+   echo "${trg_arg}"
+}
+
+collect_mysqld_executables () {
+   local mysqld_instances="$1"
+
+   for pid in $( grep '/mysqld' "$mysqld_instances" | awk '/^ .*[0-9]/{print $1}' ); do
+      ps -o cmd -p $pid | sed -e 's/^\(.*mysqld\) .*/\1/' | grep -v '^CMD$'
+   done | sort -u
+}
+
+collect_mysql_info () {
+   local dir="$1"
+
+   collect_mysql_variables     > "$dir/mysql-variables"
+   collect_mysql_status        > "$dir/mysql-status"
+   collect_mysql_databases     > "$dir/mysql-databases"
+   collect_mysql_plugins       > "$dir/mysql-plugins"
+   collect_mysql_slave_status  > "$dir/mysql-slave"
+   collect_mysql_innodb_status > "$dir/innodb-status"
+   collect_mysql_processlist   > "$dir/mysql-processlist"   
+   collect_mysql_users         > "$dir/mysql-users"
+
+   collect_mysqld_instances   "$dir/mysql-variables"  > "$dir/mysqld-instances"
+   collect_mysqld_executables "$dir/mysqld-instances" > "$dir/mysqld-executables"
+
+   local binlog="$(get_var log_bin "$dir/mysql-variables")"
+   if [ "${binlog}" ]; then
+      collect_master_logs_status "$dir/mysql-master-logs" "$dir/mysql-master-status"
+   fi
+
+   local uptime="$(get_var Uptime "$dir/mysql-status")"
+   local current_time="$($CMD_MYSQL $EXT_ARGV -ss -e \
+                         "SELECT LEFT(NOW() - INTERVAL ${uptime} SECOND, 16)")"
+
+   local port="$(get_var port "$dir/mysql-variables")"
+   local cnf_file="$(find_my_cnf_file "$dir/mysqld-instances" ${port})"
+
+   cat "$cnf_file" > "$dir/mysql-config-file"
+
+   local pid_file="$(get_var "pid_file" "$dir/mysql-variables")"
+   local pid_file_exists=""
+   [ -e "${pid_file}" ] && pid_file_exists=1
+   echo "pt-summary-internal-pid_file_exists    $pid_file_exists" >> "$dir/mysql-variables"
+
+   echo "pt-summary-internal-current_time    $current_time" >> "$dir/mysql-variables"
+   echo "pt-summary-internal-Config_File_path    $cnf_file" >> "$dir/mysql-variables"
+   collect_internal_vars "$dir/mysqld-executables" >> "$dir/mysql-variables"
+
+   if [ -n "${OPT_DATABASES}" ]; then
+      local trg_arg="$( get_mysqldump_args "$dir/mysql-variables" )"
+      get_mysqldump_for "${trg_arg}" "${OPT_DATABASES}" > "$dir/mysqldump"
+   fi
+
+   (
+      sleep $OPT_SLEEP
+      collect_mysql_deferred_status "$dir/mysql-status" > "$dir/mysql-status-defer"
+   ) &
+   _d "Forked child is $!"
+}
+
+# ###########################################################################
+# End collect_mysql_info package
+# ###########################################################################
+
+# ###########################################################################
+# report_mysql_info package
+# This package is a copy without comments from the original.  The original
+# with comments and its test file can be found in the Bazaar repository at,
+#   lib/bash/report_mysql_info.sh
+#   t/lib/bash/report_mysql_info.sh
+# See https://launchpad.net/percona-toolkit for more information.
+# ###########################################################################
+
+
+set -u
+POSIXLY_CORRECT=1
+
 secs_to_time () {
-   echo "$1" | $AP_AWK '{
-      printf( "%d+%02d:%02d:%02d", $1 / 86400, ($1 % 86400) / 3600, ($1 % 3600) / 60, $1 % 60);
+   awk -v sec="$1" 'BEGIN {
+      printf( "%d+%02d:%02d:%02d", sec / 86400, (sec % 86400) / 3600, (sec % 3600) / 60, sec % 60);
    }'
 }
 
-# gets a value from $TMPDIR/percona-toolkit-mysql-variables.  Returns zero if it doesn't
-# exist.
-get_var () {
-   v="$($AP_AWK "\$1 ~ /^$1$/ { print \$2 }" $TMPDIR/percona-toolkit-mysql-variables)"
-   echo "${v:-0}"
-}
-
-# Returns true if a variable exists
-var_exists () {
-   $AP_GREP "$1" $TMPDIR/percona-toolkit-mysql-variables >/dev/null 2>&1;
-}
-
-# Returns "Enabled", "Disabled", or "Not Supported" depending on whether the
-# variable exists and is ON or enabled.  You can pass 2nd and 3rd variables to
-# control whether the variable should be 'gt' (numeric greater than) or 'eq'
-# (string equal) to some value.
 feat_on() {
-   if var_exists $1 ; then
-      var="$($AP_AWK "\$1 ~ /^$1$/ { print \$2 }" $TMPDIR/percona-toolkit-mysql-variables)"
+   local file="$1"
+   local varname="$2"
+   [ -e "$file" ] || return
+
+   if [ "$( get_var "$varname" "${file}" )" ]; then
+      local var="$(awk "\$1 ~ /^$2$/ { print \$2 }" $file)"
       if [ "${var}" = "ON" ]; then
          echo "Enabled"
       elif [ "${var}" = "OFF" -o "${var}" = "0" -o -z "${var}" ]; then
          echo "Disabled"
-      elif [ "$2" = "ne" ]; then
-         if [ "${var}" != "$3" ]; then
+      elif [ "$3" = "ne" ]; then
+         if [ "${var}" != "$4" ]; then
             echo "Enabled"
          else
             echo "Disabled"
          fi
-      elif [ "$2" = "gt" ]; then
-         if [ "${var}" -gt "$3" ]; then
+      elif [ "$3" = "gt" ]; then
+         if [ "${var}" -gt "$4" ]; then
             echo "Enabled"
          else
             echo "Disabled"
@@ -210,128 +1011,124 @@
    fi
 }
 
-# gets a value from $TMPDIR/percona-toolkit-mysql-status.  Returns zero if it doesn't
-# exist.
-get_stat () {
-   v="$($AP_AWK "\$1 ~ /^$1$/ { print \$2 }" $TMPDIR/percona-toolkit-mysql-status)"
-   echo "${v:-0}"
-}
-
-# Does fuzzy rounding: rounds to nearest interval, but the interval gets larger
-# as the number gets larger.  This is to make things easier to diff.
-fuzz () {
-   echo $1 | $AP_AWK "{fuzzy_var=\$1; ${fuzzy_formula} print fuzzy_var;}"
-}
-
-# Fuzzy computes the percent that $1 is of $2
-fuzzy_pct () {
-   pct=$(echo $1 $2 | $AP_AWK '{ if ($2 > 0) { printf "%d", $1/$2*100; } else {print 0} }');
-   echo "$(fuzz ${pct})%"
-}
-
-# ##############################################################################
-# Functions for parsing specific files and getting desired info from them.
-# These are called from within main() and are separated so they can be tested
-# easily.  The calling convention is that the data they need to run is prepared
-# first by putting it into $TMPDIR/percona-toolkit.  Then code that's testing
-# just needs to put sample data into $TMPDIR/percona-toolkit and call it.
-# ##############################################################################
-
-# Parses the output of 'ps -e -o args | $AP_GREP mysqld' or 'ps auxww...'
-# which should be in $TMPDIR/percona-toolkit.
+get_table_cache () {
+   local file="$1"
+
+   [ -e "$file" ] || return
+
+   local table_cache=""
+   if [ "$( get_var table_open_cache "${file}" )" ]; then
+      table_cache="$(get_var table_open_cache "${file}")"
+   else
+      table_cache="$(get_var table_cache "${file}")"
+   fi
+   echo ${table_cache:-0}
+}
+
+get_plugin_status () {
+   local file="$1"
+   local plugin="$2"
+
+   local status="$(grep -w "$plugin" "$file" | awk '{ print $2 }')"
+
+   echo ${status:-"Not found"}
+}
+
+
+_NO_FALSE_NEGATIVES=""
 parse_mysqld_instances () {
-   local file=$1
+   local file="$1"
+   local variables_file="$2"
+
    local socket=${socket:-""}
    local port=${port:-""}
-   local datadir=${datadir:-""}
-   echo "  Port  Data Directory             Socket"
-   echo "  ===== ========================== ======"
-   $AP_GREP '/mysqld ' $file | while read line; do
+   local datadir="${datadir:-""}"
+
+   [ -e "$file" ] || return
+
+   echo "  Port  Data Directory             Nice OOM Socket"
+   echo "  ===== ========================== ==== === ======"
+
+   grep '/mysqld ' "$file" | while read line; do
+      local pid=$(echo "$line" | awk '{print $1;}')
       for word in ${line}; do
-         # Some grep doesn't have -o, so I have to pull out the words I want by
-         # looking at each word
-         if echo "${word}" | $AP_GREP -- "--socket=" > /dev/null; then
+         if echo "${word}" | grep -- "--socket=" > /dev/null; then
             socket="$(echo "${word}" | cut -d= -f2)"
          fi
-         if echo "${word}" | $AP_GREP -- "--port=" > /dev/null; then
+         if echo "${word}" | grep -- "--port=" > /dev/null; then
             port="$(echo "${word}" | cut -d= -f2)"
          fi
-         if echo "${word}" | $AP_GREP -- "--datadir=" > /dev/null; then
+         if echo "${word}" | grep -- "--datadir=" > /dev/null; then
             datadir="$(echo "${word}" | cut -d= -f2)"
          fi
       done
-      printf "  %5s %-26s %s\n" "${port}" "${datadir}" "${socket}"
+      local nice="$(get_var "internal::nice_of_$pid" "$variables_file")"
+      local oom="$(get_var "internal::oom_of_$pid" "$variables_file")"
+      if [ -n "${_NO_FALSE_NEGATIVES}" ]; then
+         nice="?"
+         oom="?"
+      fi
+      printf "  %5s %-26s %-4s %-3s %s\n" "${port}" "${datadir}" "${nice:-"?"}" "${oom:-"?"}" "${socket}"
    done
 }
 
-# Tries to find the my.cnf file by examining 'ps' output, which should be in
-# $TMPDIR/percona-toolkit.  You have to specify the port for the instance you are
-# interested in, in case there are multiple instances.
-find_my_cnf_file() {
-   local file=$1
-   local port=${2:-""}
-   if test -n "$port" && $AP_GREP -- "/mysqld.*--port=$port" $file >/dev/null 2>&1 ; then
-      $AP_GREP -- "/mysqld.*--port=$port" $file \
-         | $AP_AWK 'BEGIN{RS=" "; FS="=";} $1 ~ /--defaults-file/ { print $2; }' \
-         | head -n1
-   else
-      $AP_GREP '/mysqld' $file \
-         | $AP_AWK 'BEGIN{RS=" "; FS="=";} $1 ~ /--defaults-file/ { print $2; }' \
-         | head -n1
-   fi
-}
-
-# Gets the MySQL system time.  Uses input from $TMPDIR/percona-toolkit-mysql-variables.
 get_mysql_timezone () {
-   tz="$(get_var time_zone)"
+   local file="$1"
+
+   [ -e "$file" ] || return
+
+   local tz="$(get_var time_zone "${file}")"
    if [ "${tz}" = "SYSTEM" ]; then
-      tz="$(get_var system_time_zone)"
+      tz="$(get_var system_time_zone "${file}")"
    fi
    echo "${tz}"
 }
 
-# Gets the MySQL system version.  Uses input from $TMPDIR/percona-toolkit-mysql-variables.
 get_mysql_version () {
-   name_val Version "$(get_var version) $(get_var version_comment)"
-   name_val "Built On" "$(get_var version_compile_os) $(get_var version_compile_machine)"
+   local file="$1"
+
+   name_val Version "$(get_var version "${file}") $(get_var version_comment "${file}")"
+   name_val "Built On" "$(get_var version_compile_os "${file}") $(get_var version_compile_machine "${file}")"
 }
 
-# Gets the system start and uptime in human readable format.  Last restart date
-# should be in $TMPDIR/percona-toolkit.
 get_mysql_uptime () {
-   local file=$1
-   restart="$(cat $file)"
-   uptime="$(get_stat Uptime)"
+   local uptime="$1"
+   local restart="$2"
    uptime="$(secs_to_time ${uptime})"
    echo "${restart} (up ${uptime})"
 }
 
-# Summarizes the output of SHOW MASTER LOGS, which is in $TMPDIR/percona-toolkit
 summarize_binlogs () {
-   local file=$1
-   name_val "Binlogs" $(wc -l $file)
-   name_val "Zero-Sized" $($AP_GREP -c '\<0$' $file)
-   size=$($AP_AWK '{t += $2} END{printf "%0.f\n", t}' $file)
+   local file="$1"
+
+   [ -e "$file" ] || return
+
+   local size="$(awk '{t += $2} END{printf "%0.f\n", t}' "$file")"
+   name_val "Binlogs" $(wc -l "$file")
+   name_val "Zero-Sized" $(grep -c '\<0$' "$file")
    name_val "Total Size" $(shorten ${size} 1)
 }
 
-# Print out binlog_do_db and binlog_ignore_db
+format_users () {
+   local file="$1"
+   [ -e "$file" ] || return
+   awk '{printf "%d users, %d anon, %d w/o pw, %d old pw\n", $1, $2, $3, $4}' "${file}"
+}
+
 format_binlog_filters () {
-   local file=$1
-   name_val "binlog_do_db" $(cut -f3 $file)
-   name_val "binlog_ignore_db" $(cut -f4 $file)
+   local file="$1"
+   [ -e "$file" ] || return
+   name_val "binlog_do_db" "$(cut -f3 "$file")"
+   name_val "binlog_ignore_db" "$(cut -f4 "$file")"
 }
 
-# Takes as input a file that has two samples of SHOW STATUS, columnized next to
-# each other.  These should be in $TMPDIR/percona-toolkit.  Outputs fuzzy-ed numbers:
-# absolute, all-time per second, and per-second over the interval between the
-# samples.  Omits any rows that are all zeroes.
 format_status_variables () {
-   local file=$1
-   # First, figure out the intervals.
-   utime1=$($AP_AWK '/Uptime /{print $2}' $file);
-   utime2=$($AP_AWK '/Uptime /{print $3}' $file);
-   ${AP_AWK} "
+   local file="$1"
+   [ -e "$file" ] || return
+
+   utime1="$(awk '/Uptime /{print $2}' "$file")";
+   utime2="$(awk '/Uptime /{print $3}' "$file")";
+   awk "
    BEGIN {
       utime1 = ${utime1};
       utime2 = ${utime2};
@@ -367,28 +1164,22 @@
             printf(format, \$1, perday, persec, nowsec);
          }
       }
-   }" $file
+   }" "$file"
 }
 
-# Slices the processlist a bunch of different ways.  The processlist should be
-# created with the \G flag so it's vertical.
-# The parsing is a bit awkward because different
-# versions of awk have limitations like "too many fields on line xyz".  So we
-# use 'cut' to shorten the lines.  We count all things into temporary variables
-# for each process in the processlist, and when we hit the Info: line which
-# ought to be the last line in the process, we decide what to do with the temp
-# variables.  If we're summarizing Command, we count everything; otherwise, only
-# non-Sleep processes get counted towards the sum and max of Time.
 summarize_processlist () {
-   local file=$1
+   local file="$1"
+
+   [ -e "$file" ] || return
+
    for param in Command User Host db State; do
       echo
       printf '  %-30s %8s %7s %9s %9s\n' \
          "${param}" "COUNT(*)" Working "SUM(Time)" "MAX(Time)"
       echo "  ------------------------------" \
          "-------- ------- --------- ---------"
-      cut -c1-80 $file \
-         | $AP_AWK "
+      cut -c1-80 "$file" \
+         | awk "
          \$1 == \"${param}:\" {
             p = substr(\$0, index(\$0, \":\") + 2);
             if ( index(p, \":\") > 0 ) {
@@ -428,22 +1219,21 @@
    echo
 }
 
-# Pretty-prints the my.cnf file, which should be in $TMPDIR/percona-toolkit.  It's super
-# annoying, but some *modern* versions of awk don't support POSIX character
-# sets in regular expressions, like [[:space:]] (looking at you, Debian).  So
-# the below patterns contain [<space><tab>] and must remain that way.
 pretty_print_cnf_file () {
-   local file=$1
-   $AP_AWK '
+   local file="$1"
+
+   [ -e "$file" ] || return
+
+   awk '
    BEGIN {
       FS="="
    }
-   /^ *[a-zA-Z[]/ {
-      if ($2) {
-         gsub(/^[ 	]*/, "", $1);
-         gsub(/^[ 	]*/, "", $2);
-         gsub(/[ 	]*$/, "", $1);
-         gsub(/[ 	]*$/, "", $2);
+   /^[ \t]*[a-zA-Z[]/ {
+      if (length($2)) {
+         gsub(/^[ \t]*/, "", $1);
+         gsub(/^[ \t]*/, "", $2);
+         gsub(/[ \t]*$/, "", $1);
+         gsub(/[ \t]*$/, "", $2);
          printf("%-35s = %s\n", $1, $2);
       }
       else if ( $0 ~ /\[/ ) {
@@ -453,11 +1243,12 @@
       else {
          print $1;
       }
-   }' $file
+   }' "$file"
 }
 
 find_checkpoint_age() {
-   $AP_AWK '
+   local file="$1"
+   awk '
    /Log sequence number/{
       if ( $5 ) {
          lsn = $5 + ($4 * 4294967296);
@@ -474,11 +1265,15 @@
          print lsn - $4;
       }
    }
-   ' "$@"
+   ' "$file"
 }
 
 find_pending_io_reads() {
-   $AP_AWK '
+   local file="$1"
+
+   [ -e "$file" ] || return
+
+   awk '
    /Pending normal aio reads/ {
       normal_aio_reads  = substr($5, 1, index($5, ","));
    }
@@ -495,11 +1290,15 @@
       printf "%d buf pool reads, %d normal AIO", reads, normal_aio_reads;
       printf ", %d ibuf AIO, %d preads", ibuf_aio_reads, preads;
    }
-   ' "${1}"
+   ' "${file}"
 }
 
 find_pending_io_writes() {
-   $AP_AWK '
+   local file="$1"
+
+   [ -e "$file" ] || return
+
+   awk '
    /aio writes/ {
       aio_writes = substr($NF, 1, index($NF, ","));
    }
@@ -522,11 +1321,15 @@
    END {
       printf "%d buf pool (%d LRU, %d flush list, %d page); %d AIO, %d sync, %d log IO (%d log, %d chkp); %d pwrites", lru + flush_list + single_page, lru, flush_list, single_page, aio_writes, sync_ios, log_ios, log_writes, chkp_writes, pwrites;
    }
-   ' "${1}"
+   ' "${file}"
 }
 
 find_pending_io_flushes() {
-   $AP_AWK '
+   local file="$1"
+
+   [ -e "$file" ] || return
+
+   awk '
    /Pending flushes/ {
       log_flushes = substr($5, 1, index($5, ";"));
       buf_pool = $NF;
@@ -534,13 +1337,17 @@
    END {
       printf "%d buf pool, %d log", buf_pool, log_flushes;
    }
-   ' "${1}"
+   ' "${file}"
 }
 
 summarize_undo_log_entries() {
-   $AP_GREP 'undo log entries' "$1" \
-      | $AP_SED -e 's/^.*undo log entries \([0-9]*\)/\1/' \
-      | $AP_AWK '
+   local file="$1"
+
+   [ -e "$file" ] || return
+
+   grep 'undo log entries' "${file}" \
+      | sed -e 's/^.*undo log entries \([0-9]*\)/\1/' \
+      | awk '
       {
          count++;
          sum += $1;
@@ -554,7 +1361,11 @@
 }
 
 find_max_trx_time() {
-   $AP_AWK '
+   local file="$1"
+
+   [ -e "$file" ] || return
+
+   awk '
    BEGIN {
       max = 0;
    }
@@ -570,59 +1381,69 @@
    }
    END {
       print max;
-   }' "$@"
-}
-
-# Summarizes various things about InnoDB status that are not easy to see by eye.
+   }' "${file}"
+}
+
+find_transation_states () {
+   local file="$1"
+   local tmpfile="$TMPDIR/find_transation_states.tmp"
+
+   [ -e "$file" ] || return
+
+   awk -F, '/^---TRANSACTION/{print $2}' "${file}"   \
+                        | sed -e 's/ [0-9]* sec.*//' \
+                        | sort                       \
+                        | uniq -c > "${tmpfile}"
+   group_concat "${tmpfile}"
+}
+
 format_innodb_status () {
    local file=$1
-   name_val "Checkpoint Age"       $(shorten $(find_checkpoint_age "${file}"))
+
+   [ -e "$file" ] || return
+
+   name_val "Checkpoint Age"      "$(shorten $(find_checkpoint_age "${file}") 0)"
    name_val "InnoDB Queue"        "$(awk '/queries inside/{print}' "${file}")"
    name_val "Oldest Transaction"  "$(find_max_trx_time "${file}") Seconds";
-   name_val "History List Len"     $(awk '/History list length/{print $4}' "${file}")
-   name_val "Read Views"           $(awk '/read views open inside/{print $1}' "${file}")
+   name_val "History List Len"    "$(awk '/History list length/{print $4}' "${file}")"
+   name_val "Read Views"          "$(awk '/read views open inside/{print $1}' "${file}")"
    name_val "Undo Log Entries"    "$(summarize_undo_log_entries "${file}")"
    name_val "Pending I/O Reads"   "$(find_pending_io_reads "${file}")"
    name_val "Pending I/O Writes"  "$(find_pending_io_writes "${file}")"
    name_val "Pending I/O Flushes" "$(find_pending_io_flushes "${file}")"
-   $AP_AWK -F, '/^---TRANSACTION/{print $2}' "${file}" \
-      | $AP_SED -e 's/ [0-9]* sec.*//' | sort | uniq -c > $TMPDIR/percona-toolkit2
-   name_val "Transaction States" "$(group_concat $TMPDIR/percona-toolkit2)"
-   if $AP_GREP 'TABLE LOCK table' "${file}" >/dev/null ; then
+   name_val "Transaction States"  "$(find_transation_states "${file}" )"
+   if grep 'TABLE LOCK table' "${file}" >/dev/null ; then
       echo "Tables Locked"
-      $AP_AWK '/^TABLE LOCK table/{print $4}' "${file}" \
+      awk '/^TABLE LOCK table/{print $4}' "${file}" \
          | sort | uniq -c | sort -rn
    fi
-   if $AP_GREP 'has waited at' "${file}" > /dev/null ; then
+   if grep 'has waited at' "${file}" > /dev/null ; then
       echo "Semaphore Waits"
-      $AP_GREP 'has waited at' "${file}" | cut -d' ' -f6-8 \
+      grep 'has waited at' "${file}" | cut -d' ' -f6-8 \
          | sort | uniq -c | sort -rn
    fi
-   if $AP_GREP 'reserved it in mode' "${file}" > /dev/null; then
+   if grep 'reserved it in mode' "${file}" > /dev/null; then
       echo "Semaphore Holders"
-      $AP_AWK '/has reserved it in mode/{
+      awk '/has reserved it in mode/{
          print substr($0, 1 + index($0, "("), index($0, ")") - index($0, "(") - 1);
       }' "${file}" | sort | uniq -c | sort -rn
    fi
-   if $AP_GREP -e 'Mutex at' -e 'lock on' "${file}" >/dev/null 2>&1; then
+   if grep -e 'Mutex at' -e 'lock on' "${file}" >/dev/null 2>&1; then
       echo "Mutexes/Locks Waited For"
-      $AP_GREP -e 'Mutex at' -e 'lock on' "${file}" | $AP_SED -e 's/^[XS]-//' -e 's/,.*$//' \
+      grep -e 'Mutex at' -e 'lock on' "${file}" | sed -e 's/^[XS]-//' -e 's/,.*$//' \
          | sort | uniq -c | sort -rn
    fi
 }
 
-# Summarizes per-database statistics for a bunch of different things: count of
-# tables, views, etc.  $1 is the file name.  $2 is the database name; if none,
-# then there should be multiple databases.
 format_overall_db_stats () {
-   local file=$1
+   local file="$1"
+   local tmpfile="$TMPDIR/format_overall_db_stats.tmp"
+
+   [ -e "$file" ] || return
+
    echo
-   # We keep counts of everything in an associative array keyed by db name, and
-   # what it is.  The num_dbs counter is to ensure sort order is consistent when
-   # we run the awk commands following this one.
-   $AP_AWK '
+   awk '
       BEGIN {
-         # In case there is no USE statement in the file.
          db      = "{chosen}";
          num_dbs = 0;
       }
@@ -634,7 +1455,6 @@
          }
       }
       /^CREATE TABLE/ {
-         # Handle single-DB dumps, where there is no USE statement.
          if (num_dbs == 0) {
             num_dbs     = 1;
             db_seen[db] = 1;
@@ -674,15 +1494,13 @@
             printf fmt, db, counts[db ",tables"], counts[db ",views"], counts[db ",sps"], counts[db ",trg"], counts[db ",func"], counts[db ",fk"], counts[db ",partn"];
          }
       }
-   ' $file > $TMPDIR/percona-toolkit
-   head -n2 $TMPDIR/percona-toolkit
-   tail -n +3 $TMPDIR/percona-toolkit | sort
+   ' "$file" > "$tmpfile"
+   head -n2 "$tmpfile"
+   tail -n +3 "$tmpfile" | sort
 
    echo
-   # Now do the summary of engines per DB
-   $AP_AWK '
+   awk '
       BEGIN {
-         # In case there is no USE statement in the file.
          db          = "{chosen}";
          num_dbs     = 0;
          num_engines = 0;
@@ -695,7 +1513,6 @@
          }
       }
       /^\) ENGINE=/ {
-         # Handle single-DB dumps, where there is no USE statement.
          if (num_dbs == 0) {
             num_dbs     = 1;
             db_seen[db] = 1;
@@ -734,16 +1551,13 @@
             print "";
          }
       }
-   ' $file > $TMPDIR/percona-toolkit
-   head -n1 $TMPDIR/percona-toolkit
-   tail -n +2 $TMPDIR/percona-toolkit | sort
+   ' "$file" > "$tmpfile"
+   head -n1 "$tmpfile"
+   tail -n +2 "$tmpfile" | sort
 
    echo
-   # Now do the summary of index types per DB. Careful -- index is a reserved
-   # word in awk.
-   $AP_AWK '
+   awk '
       BEGIN {
-         # In case there is no USE statement in the file.
          db        = "{chosen}";
          num_dbs   = 0;
          num_idxes = 0;
@@ -756,7 +1570,6 @@
          }
       }
       /KEY/ {
-         # Handle single-DB dumps, where there is no USE statement.
          if (num_dbs == 0) {
             num_dbs     = 1;
             db_seen[db] = 1;
@@ -807,15 +1620,13 @@
             print "";
          }
       }
-   ' $file > $TMPDIR/percona-toolkit
-   head -n1 $TMPDIR/percona-toolkit
-   tail -n +2 $TMPDIR/percona-toolkit | sort
+   ' "$file" > "$tmpfile"
+   head -n1 "$tmpfile"
+   tail -n +2 "$tmpfile" | sort
 
    echo
-   # Now do the summary of datatypes per DB
-   $AP_AWK '
+   awk '
       BEGIN {
-         # In case there is no USE statement in the file.
          db          = "{chosen}";
          num_dbs     = 0;
          num_types = 0;
@@ -828,7 +1639,6 @@
          }
       }
       /^  `/ {
-         # Handle single-DB dumps, where there is no USE statement.
          if (num_dbs == 0) {
             num_dbs     = 1;
             db_seen[db] = 1;
@@ -898,106 +1708,195 @@
             print "";
          }
       }
-   ' $file > $TMPDIR/percona-toolkit
-   hdr=$($AP_GREP -n Database $TMPDIR/percona-toolkit | cut -d: -f1);
-   head -n${hdr} $TMPDIR/percona-toolkit
-   tail -n +$((${hdr} + 1)) $TMPDIR/percona-toolkit | sort
+   ' "$file" > "$tmpfile"
+   local hdr=$(grep -n Database "$tmpfile" | cut -d: -f1);
+   head -n${hdr} "$tmpfile"
+   tail -n +$((${hdr} + 1)) "$tmpfile" | sort
    echo
 }
 
-# ##############################################################################
-# The main() function is called at the end of the script.  This makes it
-# testable.  Major bits of parsing are separated into functions for testability.
-# ##############################################################################
-main() {
-
-   # Begin by setting the $PATH to include some common locations that are not
-   # always in the $PATH, including the "sbin" locations.  On SunOS systems,
-   # prefix the path with the location of more sophisticated utilities.
-   export PATH="${PATH}:/usr/local/bin:/usr/bin:/bin:/usr/libexec"
-   export PATH="${PATH}:/usr/mysql/bin/:/usr/local/sbin:/usr/sbin:/sbin"
-   export PATH="/usr/gnu/bin/:/usr/xpg4/bin/:${PATH}"
-
-   # Set up temporary files.
-   mk_tmpdir
-   temp_files "rm"
-   temp_files "touch"
-
-   # ########################################################################
-   # Header for the whole thing, table of discovered instances
-   # ########################################################################
-   section Percona_Toolkit_MySQL_Summary_Report
-   name_val "System time" "`date -u +'%F %T UTC'` (local TZ: `date +'%Z %z'`)"
-   section Instances
-   ps auxww 2>/dev/null | $AP_GREP mysqld > $TMPDIR/percona-toolkit
-   parse_mysqld_instances $TMPDIR/percona-toolkit
-
-   # ########################################################################
-   # Fetch some basic info so we can start
-   # ########################################################################
-   mysql "$@" -ss -e 'SELECT CURRENT_USER()' > $TMPDIR/percona-toolkit
-   if [ "$?" != "0" ]; then
-      echo "Cannot connect to mysql, please specify command-line options."
-      temp_files "rm"
-      rm_tmpdir
-      exit 1
-   fi
-   user="$(cat $TMPDIR/percona-toolkit)";
-   mysql "$@" -ss  -e 'SHOW /*!40100 GLOBAL*/ VARIABLES' > $TMPDIR/percona-toolkit-mysql-variables
-   mysql "$@" -ss  -e 'SHOW /*!50000 GLOBAL*/ STATUS' > $TMPDIR/percona-toolkit-mysql-status
-   mysql "$@" -ss  -e 'SHOW DATABASES' > $TMPDIR/percona-toolkit-mysql-databases 2>/dev/null