Percona Toolkit moved to https://jira.percona.com/projects/PT

pt-table-checksum: Deep recursion on subroutine "SchemaIterator::_iterate_dbh"

Bug #1136559 reported by Steven Barre
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Percona Toolkit moved to https://jira.percona.com/projects/PT
Fix Released
High
Daniel Nichter
Percona Toolkit moved to https://jira.percona.com/projects/PT 2.2.4
2.1
Fix Released
High
Daniel Nichter
Percona Toolkit moved to https://jira.percona.com/projects/PT 2.1.10
2.2
Fix Released
High
Daniel Nichter
Percona Toolkit moved to https://jira.percona.com/projects/PT 2.2.4

Bug Description

Seems related to https://bugs.launchpad.net/percona-toolkit/+bug/1003323 but I wanted to post a new bug since that one is quite old. Feel free to close as dupe

# pt-table-checksum --version
pt-table-checksum 2.1.9

# PTDEBUG=1 pt-table-checksum --tables nwmls.properties_tmp --chunk-size-limit 4 h=localhost,u=XXXX,p=XXXX
<snip output>
# SchemaIterator:6888 7610 Next database: ftmyers_far
# SchemaIterator:6894 7610 SHOW /*!50002 FULL*/ TABLES FROM `ftmyers_far`
# SchemaIterator:7014 7610 Table city_codes is not in --tables list, ignoring
# SchemaIterator:6905 7610 Found 0 tables in database ftmyers_far
# SchemaIterator:6933 7610 No more tables in database ftmyers_far
# SchemaIterator:6872 7610 Getting next schema object from dbh DBI::db=HASH(0xe5d3cd0)
# SchemaIterator:6888 7610 Next database: fwamls
# SchemaIterator:6894 7610 SHOW /*!50002 FULL*/ TABLES FROM `fwamls`
# SchemaIterator:7014 7610 Table sold_property_mult is not in --tables list, ignoring
# SchemaIterator:7014 7610 Table sold_property_parr is not in --tables list, ignoring
# SchemaIterator:7014 7610 Table sold_property_resi is not in --tables list, ignoring
# SchemaIterator:7014 7610 Table sold_property_resr is not in --tables list, ignoring
# SchemaIterator:7014 7610 Table virtual_media is not in --tables list, ignoring
# SchemaIterator:6905 7610 Found 0 tables in database fwamls
# SchemaIterator:6933 7610 No more tables in database fwamls
Deep recursion on subroutine "SchemaIterator::_iterate_dbh" at /root/bin/pt-table-checksum line 6937.
# Cxn:3349 7610 Disconnecting dbh DBI::db=HASH(0xe5dec30) undef
# Cxn:3349 7610 Disconnecting dbh DBI::db=HASH(0xe5e8d10) undef
# Cxn:3349 7610 Disconnecting dbh DBI::db=HASH(0xe5d3cd0) undef

Related branches

lp:~percona-toolkit-dev/percona-toolkit/fix-1136559-deep-recursion-SchemaIterator
lp:~percona-toolkit-dev/percona-toolkit/fix-deep-recursion-bug-1136559
Daniel Nichter: Approve
Diff: 854 lines (+510/-268)
8 files modified
bin/pt-duplicate-key-checker (+57/-52)
bin/pt-index-usage (+57/-52)
bin/pt-table-checksum (+57/-52)
bin/pt-table-sync (+57/-52)
lib/SchemaIterator.pm (+61/-58)
t/lib/SchemaIterator.t (+16/-2)
t/lib/samples/100-dbs-drop.sql (+100/-0)
t/lib/samples/100-dbs.sql (+105/-0)
lp:~percona-toolkit-dev/percona-toolkit/fix-deep-recursion-bug-1136559-2.1
Daniel Nichter: Approve
Diff: 854 lines (+510/-268)
8 files modified
bin/pt-duplicate-key-checker (+57/-52)
bin/pt-index-usage (+57/-52)
bin/pt-table-checksum (+57/-52)
bin/pt-table-sync (+57/-52)
lib/SchemaIterator.pm (+61/-58)
t/lib/SchemaIterator.t (+16/-2)
t/lib/samples/100-dbs-drop.sql (+100/-0)
t/lib/samples/100-dbs.sql (+105/-0)
Revision history for this message
Daniel Nichter (daniel-nichter) wrote :

Steven, can you create a test case to reproduce this? I.e., dump some dbs and tables that would cause this to happen on any system? As in bug 1003323, we'll need some way to reproduce this issue because it seems pretty rare and it has never happened in testing.

tags: added: crash pt-table-checksum schema-iter
Changed in percona-toolkit:
status: New → Incomplete
Revision history for this message
Steven Barre (slashterix) wrote :

It seems to be just having too many databases causes the issue. The attached sql creates 100 databases and adds a table to the first one.

# cat /etc/redhat-release
CentOS release 5.9 (Final)

# perl -v
This is perl, v5.8.8 built for x86_64-linux-thread-multi

# rpm -qa perl*
perl-DBI-1.52-2.el5
perl-5.8.8-38.el5_8
perl-TermReadKey-2.30-4.el5
perl-String-CRC32-1.4-2.fc6
perl-DBD-MySQL-3.0007-2.el5

# mysql -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 26
Server version: 5.5.30-log Distributed by The IUS Community Project

# PTDEBUG=1 ./pt-table-checksum --tables foo0001.bla01 h=localhost,u=root --ask-pass
<snip>
# SchemaIterator:6888 3789 Next database: foo0097
# SchemaIterator:6894 3789 SHOW /*!50002 FULL*/ TABLES FROM `foo0097`
# SchemaIterator:6905 3789 Found 0 tables in database foo0097
# SchemaIterator:6933 3789 No more tables in database foo0097
# SchemaIterator:6872 3789 Getting next schema object from dbh DBI::db=HASH(0x134adcc0)
# SchemaIterator:6888 3789 Next database: foo0098
# SchemaIterator:6894 3789 SHOW /*!50002 FULL*/ TABLES FROM `foo0098`
# SchemaIterator:6905 3789 Found 0 tables in database foo0098
# SchemaIterator:6933 3789 No more tables in database foo0098
# SchemaIterator:6872 3789 Getting next schema object from dbh DBI::db=HASH(0x134adcc0)
# SchemaIterator:6888 3789 Next database: foo0099
# SchemaIterator:6894 3789 SHOW /*!50002 FULL*/ TABLES FROM `foo0099`
# SchemaIterator:6905 3789 Found 0 tables in database foo0099
# SchemaIterator:6933 3789 No more tables in database foo0099
Deep recursion on subroutine "SchemaIterator::_iterate_dbh" at ./pt-table-checksum line 6937.
# Cxn:3349 3789 Disconnecting dbh DBI::db=HASH(0x134adcc0) undef

Brian Fraser (fraserbn)
Changed in percona-toolkit:
status: Incomplete → Confirmed
Brian Fraser (fraserbn)
Changed in percona-toolkit:
assignee: nobody → Brian Fraser (fraserbn)
Brian Fraser (fraserbn)
Changed in percona-toolkit:
status: Confirmed → In Progress
Revision history for this message
Brian Fraser (fraserbn) wrote :

Steven: Many, many thanks for that .sql file. I had taken some stabs at this bug previously, but turns out I was looking for the wrong sort of problem.

The triage for this is simple: The "deep recursion" warning triggers if a sub recurses 100 times (that value is hardcoded in Perl, although greater in upcoming versions). The solution is also simple: Just don't recurse at all.

Changed in percona-toolkit:
milestone: none → 2.2.3
Daniel Nichter (daniel-nichter)
Changed in percona-toolkit:
assignee: Brian Fraser (fraserbn) → nobody
Daniel Nichter (daniel-nichter)
Changed in percona-toolkit:
status: In Progress → Triaged
Daniel Nichter (daniel-nichter)
Changed in percona-toolkit:
importance: Undecided → High
Revision history for this message
Daniel Nichter (daniel-nichter) wrote :

I can reproduce this now in 2.2.3:

Deep recursion on subroutine "SchemaIterator::_iterate_dbh" at bin/pt-table-checksum line 7398.

Revision history for this message
Steven Barre (slashterix) wrote :

Thanks! @daniel-nichter ! Tested on my systems and its working great now.

Revision history for this message
Daniel Nichter (daniel-nichter) wrote :

Thanks Steven, glad to hear it!

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PT-355

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.